Senior Data Engineer

We're looking for a Senior Data Engineer with 3+ years of hands-on experience in Apache Spark and Databricks to design and build scalable data pipelines and distributed data processing solutions on Azure. If you're strong in PySpark, Spark optimization, Delta Lake, Databricks, and modern data engineering, and enjoy working with large-scale data, this role is for you.

Responsibilities:

• Design, develop, and maintain high-performance batch and streaming data pipelines using Databricks, Apache Spark, PySpark, SQL, Python, and Delta Lake.
• Build scalable data solutions within the Azure ecosystem, leveraging services such as Azure Blob Storage, Azure Data Factory, and Event Hubs.
• Implement Medallion Architecture (Bronze/Silver/Gold) and robust ETL/ELT frameworks.
• Develop incremental processing and CDC pipelines for large-scale datasets.
• Design and optimize Spark workloads handling TB+ datasets.
• Apply Spark performance optimization techniques, including partitioning and repartitioning, efficient joins and broadcast joins, caching and persistence, adaptive query execution (AQE), data skew handling, Query and job optimization.
• Build and manage Delta Lake solutions, including ACID transactions, schema evolution, optimization, and reliable data pipelines.
• Develop Delta Live Tables (DLT) pipelines with strong data quality and pipeline design practices.
• Use Databricks Jobs and Workflows for production-grade orchestration.
• Implement data governance, access control, security, and lineage using Unity Catalog and RBAC.
• Establish strong practices around data quality, testing, monitoring, maintainability, and reusability.
• Integrate the data platform with Power BI, Tableau, APIs, and custom applications.
• Contribute to CI/CD, infrastructure-as-code, automated testing, monitoring, and alerting using tools such as Git, Terraform, Azure DevOps, GitHub Actions, Jenkins, Azure Monitor, or Datadog.
• Work closely with engineering, product, and business teams to deliver reliable data solutions that support reporting and analytics.
• Contribute to the development of a self-service data platform that enables teams to discover, access, and use trusted data efficiently.

Requirements:

• 3+ years of strong hands-on experience with Apache Spark and Databricks.
• Expert-level knowledge of PySpark / Spark, with Python or Scala.
• Strong understanding of Spark DataFrames, Spark SQL, Structured Streaming, Distributed Data Processing, and Large-Scale Data Engineering.
• Deep Databricks experience across Delta Lake, Delta Live Tables (DLT), Unity Catalog, Databricks Jobs & Workflows, and Databricks Notebooks.
• Proven experience designing and tuning high-performance Spark jobs.
• Hands-on experience working with large-scale datasets, preferably TB+.
• Strong understanding of batch and streaming architectures.
• Experience implementing Medallion Architecture.
• Practical experience with incremental processing and CDC patterns.
• Strong SQL and data engineering fundamentals.
• Experience working with Azure cloud services, particularly Azure Blob Storage.

Good to Have:

• Experience integrating Databricks with Azure Data Factory.
• Experience with Azure Event Hubs and streaming pipelines.
• Experience with external orchestration tools such as Apache Airflow.
• Experience with NoSQL databases.
• Experience with MS SQL, PostgreSQL, or MySQL.
• Knowledge of data governance, security, compliance, RBAC, and data lineage.
• Experience with Terraform / Infrastructure as Code.
• Experience with Azure DevOps, GitHub Actions, or Jenkins.
• Experience with Azure Monitor or Datadog.
• Exposure to Power BI, Tableau, APIs, or custom applications.

You'll be expected to:

• Build for scale: Design Spark pipelines that perform reliably across TB+ datasets.
• Build for reliability: Create pipelines with strong data quality, testing, monitoring, and failure-handling practices.
• Build for change: Handle schema evolution, incremental loads, CDC, and continuously changing data sources.
• Build for governance: Use Unity Catalog, RBAC, lineage, and security best practices to make data discoverable and trustworthy.
• Build for reuse: Create maintainable frameworks and components that can be used across multiple data products.
• Build for production: Follow CI/CD, Infrastructure as Code, deployment, observability, and operational best practices.

Technology Stack:

• Cloud: Azure.
• Data Platform: Databricks.
• Processing: Apache Spark, PySpark, Spark SQL, Structured Streaming.
• Storage: Delta Lake, Azure Blob Storage.
• Pipelines: DLT, Databricks Jobs, and Workflows.
• Governance: Unity Catalog, RBAC, Data Lineage.
• Integration: Azure Data Factory, Event Hubs, Airflow.
• Databases: SQL Server, PostgreSQL, MySQL, NoSQL.
• DevOps: Git, Terraform, Azure DevOps, GitHub Actions, Jenkins.
• Monitoring: Azure Monitor, Datadog.
• Analytics: Power BI, Tableau.