Databricks Consultant - Python/PySpark

Yoda Tech is looking for an experienced Databricks Consultant to design, develop, and optimize scalable data engineering solutions using the Databricks platform.

The ideal candidate will have strong expertise in data engineering, Apache Spark, Python, SQL, and cloud-based data architectures.

The candidate will work with business stakeholders, data architects, and engineering teams to build reliable data pipelines, improve data processing performance, and deliver high-quality data solutions.

Key Responsibilities :

- Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.

- Build and optimize ETL/ELT workflows for processing structured and semi-structured data.

- Develop data transformation logic using Python, PySpark, and SQL.

- Implement data ingestion, cleansing, transformation, and integration processes.

- Work with the Lakehouse architecture and Delta Lake for reliable data storage and processing.

- Optimize Spark jobs, cluster configurations, and data workflows to improve performance and cost efficiency.

- Develop and manage batch and, where required, streaming data pipelines.

- Implement data quality checks, error handling, logging, and monitoring mechanisms.

- Work with cloud-based data services and integrate Databricks with existing data platforms.

- Follow data security, access control, governance, and compliance best practices.

- Collaborate with architects, analysts, and cross-functional teams to understand requirements and deliver solutions.

- Troubleshoot production issues and ensure the reliability and scalability of data pipelines.

- Participate in technical discussions, code reviews, solution design, and documentation.

Required Skills & Qualifications :

- 8 - 11 years of experience in data engineering, data development, or related roles.

- Strong hands-on experience with Databricks and its data engineering capabilities.

- Proficiency in Python, PySpark, and SQL.

- Strong understanding of Apache Spark, distributed data processing, and performance optimization.

- Experience designing and implementing ETL/ELT pipelines.

- Knowledge of Delta Lake, lakehouse architecture, and data lake concepts.

- Experience working with cloud-based data platforms such as AWS, Microsoft Azure, or Google Cloud.

- Understanding of data modelling, data integration, and data quality management.

- Experience with version control, deployment processes, and data engineering best practices.

- Strong analytical, troubleshooting, and problem-solving skills.

- Excellent communication and collaboration skills.

Preferred Skills :

- Experience with Databricks Workflows and Delta Live Tables/Lakeflow capabilities.

- Familiarity with Azure Data Factory, AWS Glue, or comparable data integration tools.

- Experience with CI/CD practices for data engineering solutions.

- Knowledge of data governance, cataloguing, and access management.

- Exposure to streaming technologies such as Kafka or Structured Streaming.

- Experience handling large-scale enterprise data platforms and complex data transformations.

Ideal Candidate Profile :

- The ideal candidate will be an experienced data engineering professional with strong hands-on Databricks expertise, capable of designing scalable data pipelines, optimizing distributed processing, and delivering reliable cloud-based data solutions.

Skills

Data Engineering, Databricks, Data Pipeline, Apache Spark, Python, SQL, PySpark, ETL, AWS, Azure Data Factory