Senior Azure Databricks Engineer

We are seeking a highly skilled Senior Azure Databricks Engineer to design, develop, and optimize scalable data platforms and analytics solutions. You will play a pivotal role in building robust data pipelines, implementing Lakehouse architectures, and ensuring high-performance data processing to support enterprise-wide AI and analytics initiatives. This role offers the opportunity to work on complex, large-scale data ecosystems within a hybrid work environment.

Our client is a leading global computer software organization known for delivering innovative digital transformation and analytics solutions. They operate at a significant scale, supporting enterprise-level clients across various industries with cutting-edge technology and data-driven insights. The company fosters a collaborative, high-performance culture that encourages technical excellence and continuous professional growth.

Responsibilities

• Design, develop, and maintain high-performance data pipelines using Databricks, Apache Spark, and cloud-native services.
• Build and optimize complex ETL/ELT workflows for large-scale structured and unstructured datasets.
• Develop sophisticated data models and implement comprehensive data quality, validation, and governance frameworks.
• Integrate data from diverse sources into a unified, scalable Lakehouse architecture.
• Optimize Spark jobs and Databricks workloads to ensure maximum performance, scalability, and cost efficiency.
• Implement robust security controls, access management, and data governance policies using Unity Catalog.
• Collaborate cross-functionally with business, analytics, and AI/ML teams to deliver trusted, high-impact data products.
• Monitor, troubleshoot, and resolve complex data pipeline issues to ensure system reliability.
• Support CI/CD, DevOps, and infrastructure automation practices to streamline deployment cycles.

Requirements

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.
• 9-15 years of total experience in Data Engineering and Data Warehousing.
• Minimum 5+ years of hands-on experience with Azure Data Engineering technologies.
• Minimum 4+ years of hands-on experience with the Azure Databricks and Spark ecosystem.
• Proficiency in Databricks Lakehouse Platform, Apache Spark, PySpark, Delta Lake, SQL, and Python.
• Strong understanding of data lake, lakehouse, and cloud-native architecture patterns.
• Experience with Microsoft Fabric, Power BI, Azure Event Hubs, Kafka, and Microsoft Purview.
• Proven ability to build and maintain Machine Learning data pipelines.
• Strong analytical, problem-solving, and communication skills with the ability to lead technical initiatives.