Principal Data Engineer

We are looking for an experienced Principal Data Engineer to lead the design, development, and optimization of large-scale, cloud-native data platforms that power business-critical analytics, AI/ML, personalization, supply chain, merchandising, and customer experiences across Walmart's global ecosystem. As a principal engineer, you will drive technical strategy, establish engineering best practices, mentor senior engineers, and collaborate with cross-functional teams to build highly scalable, secure, and reliable data solutions capable of processing petabytes of structured and unstructured data.

Responsibilities:

• Architect enterprise-scale data platforms and distributed data processing systems.
• Design and build scalable batch and real-time data pipelines.
• Lead data architecture decisions across multiple engineering teams.
• Develop high-performance ETL/ELT frameworks using modern big data technologies.
• Build cloud-native data solutions leveraging Azure, GCP, or AWS.
• Design robust streaming architectures using Kafka and Spark Streaming.
• Optimize data storage, partitioning, indexing, and query performance.
• Build reusable data engineering frameworks and platform services.
• Ensure data quality, governance, security, lineage, and compliance.
• Partner with product, data science, ML engineers, and business stakeholders.
• Define engineering standards, coding practices, and architecture guidelines.
• Mentor senior engineers and provide technical leadership across organizations.
• Drive platform modernization initiatives and cloud migration programs.
• Conduct architecture reviews and production readiness assessments.
• Improve system observability, monitoring, scalability, and resiliency.
• Lead incident resolution and performance optimization efforts.
• What You'll Build: Enterprise Data Lake Platforms, Real-time Streaming Pipelines, Customer Data Platforms, Supply Chain Analytics Platforms, Pricing and Merchandising Data Systems, AI/ML Feature Engineering Pipelines, Retail Analytics Platforms, Recommendation and Personalization Data Infrastructure, and Data Governance Platforms.

Requirements:

• Bachelor's or master's degree in computer science, engineering, or a related field.
• 10+ years of experience in data engineering or big data platform development.
• Strong expertise in Python and Java or Scala.
• Extensive experience with Apache Spark.
• Strong knowledge of distributed computing architectures.
• Experience designing large-scale data lakes and data warehouses.
• Hands-on expertise in SQL and NoSQL databases.
• Experience with Apache Kafka and event-driven architectures.
• Expertise in Airflow or enterprise workflow orchestration tools.
• Strong understanding of data modeling (dimensional, star, snowflake, and data vault).
• Experience building cloud-native data platforms.
• Excellent understanding of data governance and metadata management.
• Strong software engineering fundamentals, including OOP, design patterns, and system design.
• Experience working with CI/CD pipelines and Infrastructure as Code.
• Excellent stakeholder management and leadership skills.

Technical Skills:

• Programming Languages: Python, Java, Scala, SQL.
• Big Data Technologies: Apache Spark, Hadoop, Hive, HDFS, Apache Kafka, Spark Streaming, Delta Lake, Apache Iceberg, Apache Hudi.
• Cloud Platforms: Microsoft Azure, Google Cloud Platform (GCP), and Amazon Web Services (AWS).
• Data Warehousing: Snowflake, BigQuery, Azure Synapse, Amazon Redshift.
• Workflow and Orchestration: Apache Airflow, Azure Data Factory, Databricks Workflows.
• Databases: PostgreSQL, MySQL, Cassandra, MongoDB, Redis.
• DevOps and Infrastructure: Docker, Kubernetes, Terraform, GitHub Actions, Jenkins.
• Monitoring: Prometheus, Grafana, Splunk.

Preferred Qualifications:

• Experience building enterprise data platforms serving billions of records daily.
• Strong expertise in lakehouse architecture.
• Experience with the Databricks ecosystem.
• Exposure to AI/ML feature engineering pipelines.
• Experience implementing Data Mesh or Data Fabric architectures.
• Knowledge of privacy, governance, and regulatory compliance.
• Experience working in large-scale retail or e-commerce environments.
• Strong understanding of distributed systems and high-availability architectures.

Leadership Expectations:

• Provide technical vision across multiple engineering organizations.
• Influence architecture decisions and engineering roadmaps.
• Mentor Staff and Senior Engineers.
• Drive engineering excellence and operational maturity.
• Lead design reviews and technical governance.
• Champion innovation and adoption of modern data technologies.
• Collaborate effectively with Product, Platform, Security, and Infrastructure teams.
• Nice to have: Databricks, Unity Catalog, dbt, MLflow, Feature Store, Kubernetes Operators, Data Contracts, OpenLineage, Great Expectations, Iceberg Catalog.