[Remote] Lead Data Engineer

Note: The job is a remote job and is open to candidates in USA. Celebal Technologies is seeking a Lead Data Engineer to lead data engineering initiatives and architecture decisions for enterprise-grade solutions on Azure or AWS. The role focuses on designing and optimizing scalable Databricks data pipelines and lakehouse platforms while guiding engineers and collaborating with stakeholders.

Responsibilities

• Lead the design, development, and implementation of scalable and high-performance data engineering solutions using Databricks
• Design and implement end-to-end data pipelines based on Medallion Architecture (Bronze, Silver, and Gold layers)
• Develop and optimize data ingestion and transformation pipelines using Delta Live Tables (DLT) and Auto Loader
• Build and manage data lakehouse solutions using Delta Lake and Apache Spark
• Design, develop, and optimize ETL/ELT workflows using PySpark, Python, and SQL
• Work with Azure or AWS cloud services to build and manage scalable data platforms
• Implement data quality, validation, governance, and monitoring frameworks
• Optimize Spark workloads, cluster configurations, and query performance
• Establish best practices for data engineering, coding standards, CI/CD, and deployment
• Lead technical discussions, mentor data engineers, and provide technical guidance to the team
• Collaborate with architects, business stakeholders, and other technical teams to translate business requirements into technical solutions

Skills

• We are looking for candidates who are Green Card holders or US Citizens (USC) only
• 10+ years of overall experience
• The ideal candidate should have hands-on experience designing and implementing scalable data pipelines using Databricks, Medallion Architecture, Delta Live Tables (DLT), and Auto Loader
• 10+ years of overall experience in Data Engineering, Big Data, and Data Platform development
• Strong hands-on experience with Databricks and its Lakehouse Platform
• Extensive experience implementing Medallion Architecture (Bronze, Silver, Gold)
• Strong experience with Delta Live Tables (DLT)
• Hands-on experience with Databricks Auto Loader for incremental and continuous data ingestion
• Strong programming experience in PySpark, Python, and SQL
• Experience with Delta Lake , Apache Spark, and distributed data processing
• Strong cloud experience in either Microsoft Azure or AWS
• Experience designing and implementing scalable ETL/ELT pipelines
• Good understanding of data modeling, data governance, security, and data quality frameworks
• A strong technical leader with excellent hands-on data engineering capabilities
• Ability to independently drive complex data engineering initiatives
• Experience designing scalable, reliable, and cost-effective data solutions
• Databricks certifications are an added advantage
• Experience with Unity Catalog and data governance
• Knowledge of CI/CD, Git, and DevOps practices
• Experience with Infrastructure as Code (Terraform)
• Experience leading teams and managing end-to-end data engineering projects
• Strong communication, stakeholder management, and problem-solving skills

Benefits

• Remote work (must be open to travel to client locations based on project requirements)

Company Overview