Data Engineer

The candidate will have responsibilities across the following functions:

ETL/ELT Pipeline Architecture and Development:

• Architect, design, and develop robust, scalable ETL/ELT pipelines for ingesting, transforming, and loading high-volume data from diverse sources, including relational databases, REST APIs, streaming platforms, flat files, and cloud storage.
• Lead the end-to-end design of data flows from raw ingestion layers through to curated, consumption-ready data products across data warehouse, data lake, and lakehouse architectures.
• Build and maintain pipelines using industry-standard frameworks, including Apache Spark, Apache Airflow, dbt, AWS Glue, Azure Data Factory, or Google Dataflow.
• Implement incremental, change data capture (CDC), and real-time streaming ingestion patterns using tools such as Apache Kafka or AWS Kinesis, where applicable.
• Define and enforce pipeline reliability standards, SLA monitoring, alerting, retry logic, idempotency, and graceful failure handling across all data engineering workloads.
• Own pipeline performance tuning, query optimisation, partitioning strategies, caching, and resource management to ensure efficient processing of large-scale datasets.

Data Modelling and Warehouse Design:

• Lead the design and implementation of dimensional and relational data models, including star schema, snowflake schema, and medallion (Bronze/Silver/Gold) architecture for data warehouse and lakehouse environments.
• Define physical data models, partitioning schemes, and storage strategies optimised for query performance and cost efficiency across cloud platforms.
• Collaborate with data architects to evolve the enterprise data model, ensuring alignment with business domain definitions and downstream consumption requirements.
• Document and maintain entity-relationship diagrams (ERDs), data dictionaries, and schema version history for all data models under ownership.
• Drive adoption of data modelling best practices and standards across the engineering team.

Data Quality, Governance and Lineage:

• Design and implement comprehensive data quality frameworks, including validation rules, reconciliation checks, anomaly detection, and automated alerting
• to ensure data reliability at every layer.
• Establish and maintain end-to-end data lineage tracking to provide full visibility of data origin, transformation, and movement across systems.
• Lead metadata management and data cataloguing initiatives, ensuring all datasets are well-documented, discoverable, and governed in line with enterprise data standards.
• Define and enforce data retention, archiving, and access control policies in collaboration with data governance and security teams.
• Champion DataOps practices CI/CD for data pipelines, automated testing, version control, and environment management to improve pipeline reliability and deployment velocity.

Cloud Infrastructure and Platform Engineering:

• Design and manage scalable data infrastructure on cloud platforms AWS (S3 Redshift, Glue, EMR, Kinesis), GCP (BigQuery, Dataflow, Pub/Sub), or Azure (Synapse, Data Factory, ADLS) based on project requirements.
• Work with DevOps and cloud infrastructure teams on infrastructure-as-code (Terraform, CloudFormation) for provisioning and managing data platform components.
• Optimise cloud data platform costs through right-sizing, storage tiering, query optimisation, and workload management strategies.
• Evaluate and introduce new tools, frameworks, and cloud-native services to improve pipeline efficiency, scalability, and maintainability.
• Ensure all data infrastructure meets security, compliance, and data residency requirements applicable to the organisation and its clients.

Technical Leadership and Mentoring:

• Serve as a technical lead for the data engineering team, providing architectural guidance, conducting design reviews, and setting coding and documentation standards.
• Mentor and coach junior and mid-level data engineers through hands-on guidance, code reviews, and structured feedback to accelerate their technical growth.
• Lead technical discussions, architecture design sessions, and proof-of-concept evaluations for new data engineering initiatives.
• Contribute to engineering roadmap planning, sprint estimation, and technical backlog prioritisation in collaboration with the engineering manager.
• EngiNeo Solutions: Drive adoption of best practices in pipeline testing, version control, peer review, and automated quality checks across the team.

Collaboration and Stakeholder Engagement:

• Partner closely with data architects, data scientists, ML engineers, BI developers, and business analysts to deliver high-quality, consumption-ready data products.
• Engage directly with business stakeholders to understand data requirements, translate them into technical specifications, and manage expectations on delivery timelines.
• Collaborate with data governance, security, and compliance teams to ensure all data engineering deliverables meet organisational policy requirements.
• Participate in Agile ceremonies, sprint planning, standups, retrospectives, and reviews and contribute to continuous improvement of the team's delivery processes.

The core requirements for the job include the following:

Education and Technical Skills:

• Bachelor's or Master's degree in Computer Science, Information Technology, Electronics, or a related engineering discipline.
• 7-10 years of hands-on experience in data engineering, with a strong focus on ETL/ELT pipeline development and data warehouse or lakehouse architecture.
• Expert-level proficiency in Python and SQL, including advanced query writing, window functions, query optimisation, and stored procedures.
• Deep, production-grade experience with at least two of: Apache Spark, Apache Airflow, dbt, AWS Glue, Azure Data Factory, Google Dataflow, or equivalent pipeline frameworks.
• Proven experience designing and implementing dimensional data models and warehouse architectures for large-scale analytical workloads.
• Hands-on experience with at least one major cloud data platform: AWS (Redshift, S3 Glue, EMR), GCP (BigQuery, Dataflow), or Azure (Synapse Analytics, ADLS, ADF).
• Strong understanding of data quality, data lineage, metadata management, and data governance principles.

Good to Have:

• Experience with modern table formats and lakehouse technologies, such as Delta Lake, Apache Iceberg, or Apache Hudi.
• Familiarity with real-time and streaming data ingestion using Apache Kafka, Apache Flink, AWS Kinesis, or Google Pub/Sub.
• Exposure to infrastructure-as-code tools such as Terraform or CloudFormation for data platform provisioning.
• Knowledge of data ontology concepts, knowledge graphs, or graph databases (Neo4j, Amazon Neptune).
• Experience with DataOps practices, CI/CD pipelines for data, automated pipeline testing frameworks, and Git-based workflow management.
• Familiarity with data observability tools such as Monte Carlo, Great Expectations, or Soda.
• Cloud certifications in AWS, GCP, or Azure data or analytics specialisations.

Soft Skills and Leadership Qualities:

• Strong ownership mindset takes end-to-end accountability for data engineering deliverables and pipeline reliability.
• Ability to communicate complex technical concepts clearly to both technical peers and non-technical business stakeholders.
• Collaborative and team-oriented, works effectively across data, analytics, ML, and business teams in a fast-paced environment.
• Structured problem-solver who can diagnose and resolve complex data pipeline issues under time pressure.
• Proactive learner who stays current with evolving data engineering tools, frameworks, and best practices.
• Experience working in Agile delivery environments with a continuous improvement mindset.

Skills

Apache Spark, ELT, ETL, Python, SQL, Spark, dbt