SDE - 1 Data Engineering

We are hiring an SDE-1 Data Engineer (Individual Contributor) to execute high-quality data engineering work across ingestion pipelines, data quality, monitoring, curated datasets, and heavy third-party/vendor integrations. This role is pure execution: writing code, fixing issues, adding validations, and ensuring reliable, timely data delivery. You will work hands-on with Spark, Databricks, Python, Kafka, AWS (S3/EC2/Lambda) and internal CDC + ingestion frameworks.

The core responsibilities for the job include the following:

Build and Enhance Data Pipelines (Internal + External Ingestion):

• Develop ingestion pipelines for internal data (CDC, service DBs).
• Build and maintain ingestion from external vendors and third parties, including the following: Custody providers, Trading partners (TPE), Banking partners, External APIs (REST-based integrations)
• Handle pagination, rate limits, incremental loads, retries, and backoffs.
• Implement Spark-based transformations on Databricks.

Implement Data Quality Checks:

• Add schema validations, field-level checks, and null/boundary checks.
• Maintain 99% data quality for assigned datasets.
• Quickly identify and fix data mismatches caused by source/vendor changes.

Monitoring, Alerts, and Observability:

• Configure alerts for: Freshness, Latency, Data quality, Pipeline failures
• Add logs and metrics to improve troubleshooting.
• Ensure MTTR < 4 hours for failures.

Vendor and Third-Party Data Reliability:

• Monitor vendor API health, schema changes, and data drift.
• Add defensive coding, retries, and fallback logic for unstable third-party feeds.
• Ensure < 15 min data lag for assigned vendor connectors (where applicable).
• Keep documentation updated for all vendor integrations.

Curated Datasets Execution:

• Build/modify curated datasets under guidance.
• Ensure 0 metric mismatches and correct business logic.
• Maintain documentation for dataset transformations.

Engineering Discipline and On-Time Delivery:

• Maintain > 80% test coverage and 0 PR hygiene rejections.
• Execute tasks with 95%+ on-time delivery.
• Provide crisp updates with minimal follow-ups.

Requirements:

• 3-5 pipelines owned independently with 99.9% uptime.
• External vendor integrations functioning reliably (< 15 min freshness).
• 99% DQ on all assigned datasets.
• Full alerting coverage on all owned pipelines.
• MTTR consistently < 4 hours.
• 1-2 curated datasets delivered without metric errors.
• 95%+ tasks delivered on-time, < 10% rollovers.
• Ability to independently debug Spark jobs, API integrations, schema mismatches, and ingestion failures.
• Good-to-Have: Databricks experience, Kafka basics, Financial/transaction data exposure, and AWS fundamentals.
• Experience integrating with third-party APIs.

Must-Have:

• Python + SQL proficiency.
• Basic to intermediate Spark/PySpark knowledge.
• Experience with ETL / ingestion pipelines.
• Understanding of APIs (GET/POST, tokens, pagination).
• Strong debugging skills.
• Fast execution and high ownership.