Data Engineer

Data Engineer (AWS & Databricks)

Experience : 3 - 5 Years

Role Summary :

We are looking for a Data Engineer with 3 - 5 years of experience specializing in Databricks and AWS to design, build, and optimize scalable data pipelines for analytical and BI workloads.

The ideal candidate brings hands-on expertise in PySpark, Delta Lake, and cloud-native integration between AWS and Databricks.

Key Responsibilities :

- Develop and maintain batch and streaming ETL/ELT data pipelines using Databricks and PySpark on AWS.

- Implement Lakehouse architectures using Delta Lake on Amazon S3 following the Medallion architecture (Bronze/Silver/Gold).

- Orchestrate data workflows using Databricks Workflows, AWS Step Functions, or Apache Airflow (MWAA).

- Write, tune, and optimize complex SQL queries for Amazon Redshift, Athena, and Databricks SQL.

- Configure secure data ingestion, storage, and cataloging leveraging AWS IAM, S3, and Databricks Unity Catalog.

- Build automated data validation checks and unit tests to ensure high data quality and pipeline reliability.

- Implement CI/CD deployment workflows using Git, Databricks Asset Bundles (DABs), or AWS CI/CD tools.

- Collaborate with data analysts, data scientists, and downstream stakeholders to deliver performant data models.

Technical Skills Required :

- Big Data & Compute: Databricks, Apache Spark, PySpark, Delta Lake

- AWS Services: Amazon S3, Redshift, Athena, IAM, CloudWatch

- Programming & Querying: Python, PySpark, Advanced SQL

- Orchestration: Databricks Workflows, Apache Airflow, or AWS Step Functions

- Streaming & Messaging: Apache Kafka, AWS Kinesis, or Spark Structured Streaming

- Data Governance & Formats: Unity Catalog, Parquet, Delta format

- Version Control & DevOps: Git, GitHub/GitLab, Docker basics

Good to Have :

- Databricks Certified Data Engineer Associate/Professional certification.

- Familiarity with AWS Glue, EMR, or DynamoDB.

- Experience with dbt (data build tool) for SQL transformations.

- Exposure to Generative AI integration, vector databases, or AWS Bedrock.

Skills

Databricks, PySpark, Delta Lake, AWS, Python, SQL, Apache Airflow, Data Build Tool, Data Engineering, Data Management