Principal Data Engineer

We are seeking a highly experienced Principal Data Engineer to spearhead the technical direction of a next-generation data platform. In this hands-on leadership role, you will design sophisticated lakehouse architectures, drive the migration of legacy workloads, and establish engineering standards to ensure high-performance, secure, and scalable data delivery across the organization.

Our client is a leading global computer software organization known for delivering innovative data-driven solutions. They operate at a significant scale, supporting complex enterprise environments with a focus on cutting-edge technology and engineering excellence. The company fosters a collaborative culture that prioritizes technical growth and high-impact problem solving.

Responsibilities

• Own the end-to-end design of the Databricks lakehouse, including ingestion, medallion architecture, and Unity Catalog implementation.
• Lead the hands-on migration of legacy Hadoop, Hive, and Spark workloads to Databricks while ensuring data parity.
• Define and enforce engineering standards for SQL, PySpark, and Delta Live Tables.
• Oversee CI/CD, automated testing, and observability frameworks to maintain high system reliability.
• Optimize performance and DBU spend through cluster right-sizing, liquid clustering, and advanced query tuning.
• Implement robust data security measures, including PII protection, encryption, and row/column-level security.
• Lead technical design reviews and author architecture decision records.
• Mentor senior and mid-level engineers to elevate overall engineering quality and technical strategy.

Requirements

• 12-16 years of experience in data or software engineering, with at least 5 years in a technical lead, staff, or principal capacity.
• Expert-level proficiency in SQL with a focus on query tuning for massive datasets.
• Deep experience with Apache Spark, including both PySpark and Scala.
• Proven track record of managing Databricks in production, including Unity Catalog, Workflows, and cost management.
• Strong expertise in large-scale batch and streaming pipeline design using Structured Streaming, Delta Live Tables, and Kafka.
• Proficiency in data modeling techniques such as Dimensional, Medallion, and Data Vault.
• Hands-on experience with CI/CD tools including Git, Terraform, and Databricks Asset Bundles.
• Graduate degree in Computer Science, Data Science, or a related field.