Lead Data Engineer
Job Description
Position Summary:
We are looking for a hands-on Senior Data Engineer with a strong DevOps mindset to design, build, and operate reliable, scalable, and observable data pipelines that power business functions across the enterprise. This is a senior individual-contributor role - you'll independently own the delivery of complex pipelines, uphold engineering standards, deploy via CI/CD, support the operational health of the platform, and mentor junior engineers through reviews and collaboration.
Core Skills:
Databricks · Python (PySpark) · SQL · Data Pipelines · CI/CD
Key Responsibilities:
Engineering & Delivery:
Independently design, build, and maintain complex, production-grade data pipelines on Databricks.
Develop efficient ETL/ELT processes with a strong focus on data quality, consistency, and scalability.
Build reusable frameworks for ingestion, transformation, and reconciliation across enterprise source systems.
Apply and help improve engineering standards - pipeline architecture, coding standards, and ETL/ELT best practices.
Technical Mentorship:
Mentor junior engineers through code reviews, design reviews, and pair-programming on complex problems.
Share best practices in Databricks/PySpark, coding standards, and engineering discipline.
Contribute to a culture of ownership, automation, and continuous improvement.
Operations & DevOps:
Deploy changes through CI/CD and the Change Request (CR) lifecycle, including validation, release management, and ticket closure.
Participate in problem management and root-cause analysis - driving permanent fixes and automation over recurring firefighting.
Support the operational health of business-critical data workloads - monitoring, alerting, and incident response.
Collaboration:
Partner with Reporting, Visualization, Platform, and Business teams to expose curated datasets for downstream analytics consumers.
Communicate technical trade-offs, progress, and risks clearly to technical and non-technical stakeholders across geographies.
Document workflows, standards, and runbooks to ensure reproducibility and knowledge continuity.
What Success Looks Like (First 6-12 Months):
In your first 6-12 months, you'll independently deliver key data pipelines to a high standard, strengthen data quality and CI/CD practices in your area, reduce recurring incidents through problem management, and become a go-to technical resource for the team.
Required Qualifications:
Bachelor's or Master's degree in Computer Science, Information Technology, or equivalent relevant experience.
6+ years of experience in data engineering.
Strong hands-on background in Databricks, Python (PySpark), and SQL for large-scale data processing.
Proven experience designing and delivering production data pipelines (ETL/ELT) at enterprise scale.
Working knowledge of CI/CD pipelines, Git-based branching strategies, and DevOps practices.
Experience with cloud platforms (AWS preferred) and core data services.
Experience supporting production data pipelines, including monitoring, alerting, and incident response.
Strong communication skills across engineering and business audiences.
Preferred Qualifications:
Experience with orchestration frameworks and streaming technologies.
Exposure to Infrastructure-as-Code and modern deployment tooling.
Familiarity with observability tooling for data platforms.
Background in semiconductor manufacturing or large-scale industrial data processing.
Databricks Certified Data Engineer Associate or Professional certification is a plus.
Competencies:
Ownership and accountability - end-to-end responsibility for your pipelines, from design to production support.
Problem-solving orientation - bias toward permanent fixes and automation.
Technical depth - leads by example through hands-on engineering and high standards.
Collaboration - works well with Reporting, Platform, and Business teams across geographies.
Clear communication - articulates technical trade-offs to non-technical stakeholders.