Senior Databricks Data Engineer

We are looking for a highly skilled Senior Databricks Data Engineer to design, develop, and optimize scalable data platforms using the Databricks Lakehouse architecture. You will play a pivotal role in building robust ETL/ELT pipelines, integrating AI/ML workflows, and ensuring high-performance data processing for enterprise-level analytics initiatives.

Our client is a prominent global leader in the computer software industry, known for delivering innovative digital solutions and advanced analytics services. They foster a culture of continuous learning and technical excellence, supporting large-scale enterprise transformations for clients worldwide.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks, Apache Spark, and cloud-native services.
• Build and optimize complex ETL/ELT workflows to handle large-scale structured and unstructured data.
• Develop sophisticated data models and implement rigorous data quality, validation, and governance frameworks.
• Optimize Spark jobs and Databricks workloads to ensure maximum performance, scalability, and cost efficiency.
• Implement robust security controls, access management, and data governance policies using Unity Catalog.
• Collaborate with cross-functional business, analytics, and AI/ML teams to deliver trusted data products.
• Monitor, troubleshoot, and resolve data pipeline issues to ensure high availability and reliability.
• Support CI/CD, DevOps, and infrastructure automation practices to streamline deployment cycles.

Requirements

• 8-10 years of professional experience in Data Engineering and Data Warehousing.
• Expertise in Databricks, Apache Spark, PySpark, and Spark SQL.
• Strong proficiency in Python and SQL for data manipulation and pipeline development.
• Hands-on experience with Delta Lake, Unity Catalog, and modern ETL/ELT frameworks.
• Proven ability to integrate AI/ML pipelines and implement MLOps best practices.
• Experience with data governance tools, data lineage, and quality frameworks like Great Expectations or Deequ.
• Familiarity with Snowflake, Kafka, and event streaming architectures.
• Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.