Data Engineer

We are seeking a skilled Data Engineer to join a specialized team focused on inbound data acquisition. You will be responsible for designing and maintaining robust ingestion pipelines that power a modern Lakehouse architecture, ensuring high-quality data delivery for downstream AI and analytics initiatives.

Our client is a prominent global financial services institution currently undergoing a significant digital transformation. They are building a cloud-native, AI-ready data ecosystem to support advanced analytics and decision-making across their international operations.

Responsibilities

• Design and maintain scalable ingestion pipelines for REST/SOAP APIs, SFTP/FTP, and cloud-based secure file drops.
• Implement connector frameworks that handle complex authentication, rate limiting, and retry logic.
• Develop ingestion components using Python, PySpark, and Azure Data Factory with orchestration via Databricks Workflows.
• Execute automated data quality checks including schema conformance, freshness monitoring, and anomaly detection.
• Maintain a comprehensive feed registry and data catalogue within Databricks Unity Catalog.
• Collaborate with internal stakeholders and external providers to define and enforce data contracts.
• Support the ingestion of unstructured data to enable downstream RAG and GenAI pipelines.
• Manage CI/CD processes using Azure DevOps and contribute to incident response and root cause analysis.

Requirements

• 3-7 years of professional experience in data engineering or a related technical field.
• Strong proficiency in Python and SQL for data manipulation and pipeline development.
• Hands-on experience with Azure Data Factory, Azure Databricks, and Delta Lake.
• Deep understanding of Azure Data Lake Storage Gen2 and Azure DevOps for CI/CD.
• Experience with Databricks Unity Catalog, Delta Live Tables, and dbt.
• Familiarity with event-driven architecture and modern file transfer protocols.
• Knowledge of infrastructure-as-code tools such as Terraform or Bicep.
• Ability to monitor system health using Azure Monitor and Log Analytics.