Data Engineer

Responsibilities:

• Design, build, and maintain scalable ETL/ELT pipelines and data workflows on AWS.
• Develop and optimize data models (OLTP/OLAP), schemas, and warehousing solutions (e. g., Redshift, Snowflake, or similar).
• Build data ingestion frameworks for batch and streaming data from diverse sources (APIs, RDBMS, files, event streams).
• Implement data quality checks, validation rules, and monitoring for data reliability and SLAs.
• Write efficient, production-grade Python and SQL for data transformations and orchestration.
• Manage and optimize data storage, partitioning, and performance (query tuning, indexing, cost optimization).
• Implement orchestration and workflow management (e. g., Airflow/Step Functions).
• Collaborate with stakeholders to translate business requirements into technical specs and data contracts.
• Ensure data security, governance, lineage, and compliance (IAM, encryption, PII handling).
• Set up observability for pipelines (logging, metrics, alerting) and support on-call rotations.
• Contribute to CI/CD for data workloads, infrastructure-as-code, and best practices documentation.

Requirements:

• AWS: S3 Glue, Athena, Lambda, Step Functions, EMR or Glue Spark, Redshift, IAM, CloudWatch, CloudFormation/Terraform.
• Programming: Python (pandas, PySpark), SQL (advanced).
• Orchestration: Apache Airflow (or MWAA), Step Functions.
• Data Warehousing/Lakes: Redshift, Lake Formation, Parquet/Delta/Iceberg.
• Streaming (nice-to-have): Kinesis, MSK (Kafka).
• Containers (nice-to-have): Docker, ECS/EKS.

Preferred:

• 4-5 years of hands-on data engineering experience in production environments.
• Strong expertise in AWS data services and cost/performance trade-offs.
• Advanced SQL skills (window functions, CTEs, query tuning) and Python for data engineering.
• Solid experience with ETL/ELT design patterns, data modeling (star/snowflake), and schema evolution.
• Experience with Spark (Glue/EMR) or similar distributed processing frameworks.
• Proficiency with workflow orchestration (Airflow or equivalent).
• Practical knowledge of data quality frameworks and automated testing for data pipelines.
• Experience with version control (Git) and CI/CD for data jobs.
• Strong understanding of security best practices (IAM roles/policies, encryption, VPC, Secrets Manager).