Data Engineer - ETL/PySpark

Role overview :

- 2 - 6 years of experience in data engineering or analytics engineering roles.

- Strong problem-solving and ability to work independently with guidance from senior engineers.

- Good communication skills to translate business requirements into technical data solutions.

- Ability to work in Agile environments and collaborate with global teams across time zones.

- Hands-on experience with Azure Databricks and Apache Spark (PySpark or Scala) for large-scale data processing.

- Strong SQL skills, including complex queries, window functions, CTEs, and query optimization for both transactional and analytical workloads.

- Experience with SQL Server: data warehousing concepts, schema design, indexing, and optimization for analytics.

- Familiarity with Azure data services: Data Lake Storage, Synapse Analytics, Data Factory, or Blob Storage.

- Understanding of ETL/ELT concepts, data modeling, dimensional modeling (fact/dimension tables), and slowly changing dimensions.

- Experience with batch and/or stream processing concepts; familiarity with event-driven architectures.

- Knowledge of data quality frameworks, testing strategies, and monitoring/alerting for data pipelines.

- Basic understanding of relational and NoSQL databases (Cosmos DB exposure is beneficial).

What would you do here :

- The Data Engineering team builds and operates robust, scalable data platforms on Azure and Databricks, closely partnering with domain PODs, Risk, Finance, and Analytics teams across the globe.

- You'll work on end-to-end data pipelines that ingest, transform, and serve data to multiple downstream consumers with strict quality and SLA requirements.

- Build and maintain ETL/ELT pipelines on Azure Databricks using PySpark or Scala to ingest, transform, and deliver data products at scale.

- Ingest data from SQL Server, Cosmos DB, event streams, APIs, and external sources into cloud data lakes and Databricks lakehouses.

- Design and implement data quality checks, validation rules, and monitoring to ensure reliability and early detection of anomalies.

- Optimize Spark jobs for performance and cost, including cluster sizing, partitioning strategies, and caching mechanisms.

- Document data pipelines, lineage, and schemas; maintain data dictionaries and metadata catalogs for consumer discovery and understanding.

- Collaborate with data analysts, scientists, and domain engineers to understand requirements and refine data products.

- Contribute to infrastructure and tooling for data workflows (scheduling, orchestration, monitoring, alerting).

Skills

Data Engineering, Azure Databricks, Apache Spark, SQL, Python, Scala, ETL, PySpark