Data Ingestion Engineer (Informatica IDMC, Azure & Databricks) - 7300

Data Ingestion Engineer (Informatica IDMC, Azure & Databricks) - 7300

Duration: 3 Months (extensions likely)

Location: Toronto or London (Hybrid, 3x/week)

On-call Support: Yes - Rotational

Overview:

Our client is seeking two Data Ingestion Engineers to design, develop and support reliable, scalable ingestion pipelines within its Azure data platform. Working with Informatica IDMC, Azure Data Factory and Databricks, you will enable the ingestion of structured and unstructured data while maintaining data quality, monitoring performance and supporting production operations.

Responsibilities

• Design, develop, test and support end-to-end ingestion pipelines using Informatica IDMC, Azure Data Factory and Databricks.
• Collaborate with cross-functional teams and enterprise and portfolio architects to translate data requirements into efficient ingestion patterns and workflows.
• Implement and maintain pipeline monitoring, controls and alerts to proactively identify issues and maintain data quality and integrity.
• Work with support teams to triage and resolve major incidents, system outages and pipeline disruptions, including participation in rotational on-call support.
• Automate operational processes using Azure tools to improve pipeline reliability, efficiency and scalability.
• Assess performance and capacity, identify bottlenecks and implement improvements to optimize processing and reduce operational costs.
• Implement data governance, compliance and security controls to protect sensitive data.
• Apply best practices for scalable, reusable and reliable data pipelines.
• Maintain documentation covering system configurations, operational processes and troubleshooting procedures.

Must-Haves

• 5+ years of experience in data operations or development, such as ETL or data warehousing.
• 2+ years of recent development experience with Informatica IDMC, Microsoft Azure and Databricks.
• Demonstrated experience designing and implementing data ingestion systems using Microsoft Azure and Databricks.
• Solid programming experience with Python or SQL.
• Experience setting up pipeline monitoring and controls. Relevant experience with data warehousing or other data technologies is also accepted.
• Strong knowledge of service management, including incident and change management procedures.
• Ability to troubleshoot complex data pipeline and operational issues.
• Availability to work onsite 3 days per week in Toronto or London and participate in rotational on-call support.
• Valid Reliability Clearance or eligibility to obtain it, including meeting the stated five-year residency requirement.

Nice-to-Haves

• Experience with Kafka.
• Experience with continuous integration and continuous delivery (CI/CD).
• Experience with change data capture (CDC) tools.