Data Engineer

Role Overview :

We are looking for a highly skilled Databricks PySpark Developer to join our data platform implementation team. In this role, you will be responsible for designing, developing, and optimizing scalable ETL pipelines and data workflows using Databricks and Apache Spark.

Key Responsibilities :

1. ETL Development & Data Engineering :

- Design, develop, and maintain scalable ETL processes using Databricks PySpark.

- Extract, transform, and load data from heterogeneous sources into Data Lake and Data Warehouse environments.

- Optimize ETL workflows for performance, scalability, and cost efficiency using Spark SQL and PySpark.

- Design and implement data solutions following Medallion Architecture (Bronze, Silver, Gold layers).

2. Data Pipeline Management :

- Hands-on experience in building and managing advanced data pipelines using Databricks Workflows.

- Develop and maintain reliable, reusable, and scalable pipelines ensuring data quality and integrity.

3. Data Analysis & Query Optimization :

- Write, review, and optimize complex SQL queries for data transformation, aggregation, and analysis.

4. Project Coordination & Continuous Improvement :

- Document ETL processes, data lineage, metadata, and workflows.

- Mentor junior developers and contribute to team knowledge sharing.

Required Qualifications :

- Bachelor's degree in Computer Science, Engineering, or a related field.

- 5+ years of experience in ETL/Data Engineering roles with strong focus on Databricks PySpark.

- Strong proficiency in Python and SQL.

- In-depth understanding of Apache Spark architecture.

Preferred Qualifications :

- Experience working on cloud platforms, preferably AWS.

- Hands-on experience with tools such as Snowflake or Tableau.

- Relevant certifications in Databricks or cloud technologies.

Skills

Data Engineering, Databricks, Python, PySpark, SQL, Apache Spark, ETL, Data Warehousing, AWS, Snowflake DB, Data Pipeline