Lead Software Engineer - Data Ingestion

Responsibilities

Data Ingestion Specialist

Role Summary

We are looking for a Medior Data Ingestion Specialist to support development, operation, and maintenance of data ingestion platform orchestrated by Apache Airflow. The role focuses on onboarding and maintaining data integrations from multiple source systems, troubleshooting operational issues, and supporting release and governance processes

Key Responsibilities

Data Integration & Pipeline Development

• Configure and maintain data ingestion pipelines

• Develop and support Python-based ingestion components

• Integrate data:

  • Relational databases (Oracle, PostgreSQL, MS SQL, MySQL, DB2, etc)

  • Apache Kafka

  • AWS S3, flat files

  • Elasticsearch, Hive, Impala, hBase

• Troubleshoot data flow, connectivity, and integration issues

• Ingestion to Databricks, Databricks loader

Platform Operations

• Support incident, problem, and change management activities

• Maintain Python libraries and dependencies

• Create ETL/ELT pipelines focused on source ingestion

• Monitor pipeline execution and resolve operational issues

• Assist with platform upgrades, patching, and maintenance activities

Release & Governance

• Support Dev/Test/Prod deployment processes

• Review configuration and template changes

• Ensure compliance with governance, security, and operational standards

• Maintain technical and operational documentation

Collaboration

• Work closely with architects, engineering teams, and platform owners

• Collaborate with security, audit, and infrastructure teams

• Participate in release planning and communication activities

Nice-to-Have Skills

• Knowledge of ETL Python frameworks (e.g. Meltano)

• Experience with other ETL/ELT tools

• Experience with Git, CI/CD pipelines, and DevOps practices

• Knowledge of Linux environments

• Experience with Jira and ITSM tooling

• Exposure to cloud-based data platforms

Candidate Profile

• Medior-level engineer capable of working independently on standard tasks

• Able to create and resolve customers' requirements

• Able to analyse integration issues and identify root causes

• Comfortable working in an enterprise environment with governance and security requirements

• Good communication skills and ability to collaborate with multiple stakeholders

Profile required

Must-Have Skills

• 2+ years of experience with Python development

• Practical (hands on) experience with Apache NiFi

• Practical (hands on) experience with Apache Airflow

• Experience integrating data from relational databases

• ETL/ELT performance tuning, optimalizations

• Basic experience with Apache Kafka

• Basic experience with Elasticsearch

• SQL knowledge and understanding of data integration concepts

• Experience troubleshooting technical and connectivity issues

• Understanding of incident and release management processes

• Ability to create and maintain technical documentation

• Strong analytical and problem-solving skills