Lead Software Engineer, Data Engineering

Responsibilities

• Architect, Design, develop, and maintain data ingestion, transformation, and orchestration pipelines (batch and real-time).
• Build and optimize data Lakehouse architectures using Azure Synapse, Delta Lake, or similar frameworks.
• Integrate and manage structured and unstructured data sources (SQL/NoSQL, files, documents, IoT streams).
• Develop and operationalize ETL/ELT pipelines using Azure Data Factory, Databricks, or Apache Spark.
• Collaborate with Data Scientists to prepare and serve ML-ready datasets for model training and inference.
• Implement data quality, lineage, and governance frameworks across pipelines and storage layers.
• Work with BI tools (Power BI, Superset, Tableau) to enable self-service analytics for business teams.
• Deploy and maintain data APIs and ML models in production using Azure ML, Kubernetes, and CI/CD pipelines.
• Ensure scalability, performance, and observability of data workflows through effective monitoring and automation.
• Collaborate cross-functionally with engineering, product, and business teams to translate insights into action.
• Mentor junior team members and review their work

Requirements

• Experience: 6 to 12 years in Data Engineering or equivalent roles.
• Programming: Strong in Python, SQL, and at least one compiled language (C#, Java, or Scala).
• Databases: Experience with relational (SQL Server, PostgreSQL, MySQL) and NoSQL (MongoDB, Cosmos DB) systems.
• Data Platforms: Hands-on experience with Azure Data Lake, Databricks.
• ETL/ELT Tools: Azure Data Factory, Apache Airflow, or dbt.
• Messaging & Streaming: Kafka, Event Hubs, or Service Bus for real-time data processing.
• AI/ML Exposure: Familiarity with ML frameworks (TensorFlow, PyTorch) and MLOps concepts.
• Visualization & Analytics: Power BI, Apache Superset, or Tableau.
• Cloud & DevOps: Azure, Docker, Kubernetes, GitHub Actions/Azure DevOps.
• Best Practices: Solid understanding of data modeling, version control, and CI/CD for data systems.

Preferred skills

• Preferred Skills (Nice to Have):
• Experience in knowledge graph or semantic search solutions.
• Understanding of LLM-based data retrieval (RAG) patterns.
• Exposure to data mesh, data fabric, or domain-oriented data architecture.
• Familiarity with MLflow, Delta Live Tables, or Data Bricks Unity Catalog.

Soft skills

• Strong analytical and problem-solving ability.
• Excellent communication and collaboration skills.
• Attention to detail and ability to work with large, complex datasets.
• Creativity and ability to automate repetitive workflows.
• Passion for continuous learning and innovation in data and AI technologies.