Data Engineer

We're seeking a Data Engineer with 3-4 years of experience to join our growing tech team. The ideal candidate will have hands-on experience in building and managing scalable data systemson AWS, with strong expertise in data pipelines, real-time streaming, and modern data frameworks in building and managing scalable data systems on AWS using Spark and modern data frameworks.

Responsibilities:

• Design, build, and maintain end-to-end data pipelines for ingestion, transformation, and delivery of high-volume data.
• Develop Spark-based ETL/ELT workflows for both batch and real-time streaming data.
• Core Skills: Spark, AWS (S3 Glue, Redshift, Kinesis), Kafka, Airflow, Python, SQL.
• Integrate data from multiple internal and external systems using Kafka, Kinesis, or other streaming frameworks.
• Build and manage data models, warehouses, and lakehouses using AWS services such as S3 Glue, Redshift, Athena, etc.
• Implement data quality checks, validation rules, and monitoring to ensure reliability and consistency.
• Collaborate with data analysts and scientists to provide clean, structured datasets optimised for analytics and ML.
• Work with orchestration tools (Airflow, MWAA, Step Functions, etc. ) for automated workflow scheduling.
• Continuously optimise data pipelines for cost, scalability, and performance.

Requirements:

• Strong programming skills in Python for data manipulation and automation.
• Hands-on expertise in Apache Spark (PySpark or Spark SQL) for large-scale data processing.
• Deep understanding of the AWS data ecosystem: S3 Glue, Lambda, Redshift, Athena, EMR, Kinesis, IAM.
• Experience with real-time streaming platforms such as Kafka, Kinesis, or Flink.
• Strong command of SQL and data modelling (star schema, dimensional modelling, partitioning).
• Proficiency with data orchestration and workflow management tools (Airflow, Step Functions, etc. ).
• Familiarity with Git, CI/CD, and modern development best practices.
• Experience working in Linux/Unix environments and handling large datasets efficiently.

Good to Have:

• Exposure to data lakehouse technologies (Delta Lake, Iceberg, Hudi).
• Understanding of data governance, cataloguing, and lineage tools (Glue Data Catalogue, Amundsen, DataHub).
• Familiarity with containerization and deployment (Docker, ECS, EKS).
• Basic understanding of AI/ML.
• Bachelor's degree in Computer Science, IT, Engineering, or related technical field.
• 3-4 years of experience in data engineering, big data, or analytics infrastructure.

Skills

Apache Spark, Data Lake, Data bricks, Data engineer, Data warehouse, Data warehousing, Databricks, Datalake, Date engineering, ELT, ETL, Pipeline, PySpark, Python, Spark, Spark SQL, data pipeline, data pipelines, data processing