Lead DBT Engineer

About the role

In this role, you will help transform a data platform that currently relies on Spark and Python, orchestrated with Airflow DAGs and stored in Apache Iceberg, into a more modular target architecture centered on dbt and Lakekeeper.

Responsibilities

• Support the migration of the Agora data platform to a cloud-first architecture
• Develop and maintain data transformation logic using dbt as the central business logic layer
• Refactor and consolidate logic currently distributed across Spark, Python, and Airflow into a more standardized dbt-based approach
• Work with Apache Iceberg as the core storage layer in both current and target architecture
• Contribute to the implementation of an open and modular platform built around Lakekeeper
• Support data ingestion workflows across Starburst, Microsoft Fabric, and Snowflake
• Help build and maintain scheduling and orchestration processes using dbt and/or Microsoft Fabric
• Enable and improve data quality, lineage, documentation, and CI/CD practices across the platform
• Collaborate on the transition of the semantic layer from Dremio to Starburst
• Support semantic modeling for analytics using Iceberg views stored in Lakekeeper
• Work closely with cross-functional stakeholders to reduce operational complexity and improve platform maintainability
• Contribute to the development of a flexible data foundation prepared for future analytics and AI-driven use cases

Requirements

• Strong experience in data engineering or analytics engineering
• Hands-on experience with dbt
• Solid knowledge of SQL and SQL-based transformation design
• Experience with modern data platform and warehouse/lakehouse concepts
• Practical experience with Apache Iceberg
• Experience with at least some of the following technologies: Spark, Python, Airflow, Starburst, Microsoft Fabric, Snowflake
• Understanding of data modeling, semantic layers, and analytics-focused data design
• Experience with CI/CD, testing, and data quality practices in data platforms
• Familiarity with lineage and technical documentation standards
• Ability to work in an open, modular architecture with separated storage and compute principles
• Understanding of cloud-first platform migration challenges and best practices
• Strong collaboration and communication skills

Nice to have

• Experience with Lakekeeper
• Experience migrating from legacy or fragmented data transformation patterns to dbt-centered architectures
• Familiarity with Dremio and migration to Starburst
• Experience with ODBC/JDBC-based BI connectivity
• Exposure to AI-ready data platform design principles

SoftServe is an equal opportunity employer. Qualified applicants will receive consideration regardless of race, color, ancestry, ethnicity, national origin, religion, sex, sexual orientation, gender identity or expression, age, citizenship, disability, health condition, marital or family status, veteran status, or any other characteristic protected by applicable law.