Lead Data Engineer

The core responsibilities for the job include the following:

Data Platform (Core):

• Architect and scale a modern data lake supporting both batch and real-time (streaming) workloads.
• Design and manage reliable data ingestion pipelines.
• Data modeling and governance.
• Data quality and lineage.
• Build and maintain a scalable feature store for ML and AI systems.
• Develop a robust semantic layer that enables analytics, experimentation, and AI use cases.

ML and Data Science:

• Own the complete ML lifecycle from training and validation to deployment, monitoring, and retraining.
• Build and operationalize predictive and optimization models.
• Establish experimentation frameworks and model evaluation standards.
• Implement drift detection, monitoring, and performance tracking to ensure model reliability.

AI Readiness:

• Design and deploy RAG (Retrieval-Augmented Generation) pipelines.
• Build and manage embeddings infrastructure.
• Structure and maintain AI-ready datasets optimized for LLMs and agent-based systems.
• Implement feedback loops that enable continuous learning and system improvement.

What You'll Build:

• Central data lake (events, operational, external).
• Real-time pipelines (Kafka / PubSub / streaming).
• ML pipelines (forecasting, anomaly detection, optimization), feature stores, experiment tracking, data observability, decision intelligence dashboards, and AI-ready datasets.

Architecture First:

• Define the complete data architecture: source, ingestion, storage, transformation, and serving.
• Design and optimize offline vs. real-time processing strategies.
• Make informed cost vs. performance trade-offs.

ML in Production:

• Model deployment pipelines.
• Versioning.
• Monitoring.
• Retraining strategies.

Cross-Functional:

• Work with the product to define signals.
• Work with engineering to integrate pipelines.
• Working with AI teams enables agent intelligence.

Requirements:

• 6+ years of experience across data engineering, data science, or machine learning roles.
• Proven experience designing and building data lake or data warehouse architectures in production.
• Demonstrated ability to ship ML models to production.
• Strong understanding of data modeling, feature engineering, ML lifecycle, experimentation, and data governance.
• Hands-on programming experience with Python and SQL.
• Experience working with modern cloud data stacks on GCP or AWS.