Senior Data Engineer (Apache Kafka)

ABOUT THE ROLE

In this role, you will work as a Senior Data Engineer on a next-generation data platform supporting a broad portfolio of energy-sector products.

You will design canonical and semantic data models, integrate fragmented data sources, build real-time pipelines, and take ownership of the Apache Kafka platform. The role is focused on creating reliable, reusable data foundations for cross-product reporting, analytics, AI/ML, and data observability.

RESPONSIBILITIES

• Design and maintain canonical and semantic data models across multiple products and data domains
• Define data contracts, standards, and integration patterns for heterogeneous systems
• Build and maintain batch and real-time data pipelines
• Architect, deploy, and administer Apache Kafka clusters
• Own Kafka topic and partition design, replication, security, monitoring, capacity planning, and performance tuning
• Create data foundations for reporting, analytics, machine learning, and LLM-based capabilities
• Implement data lineage, usage tracking, telemetry, observability, and data quality monitoring
• Collaborate with product managers, data scientists, and application engineering teams
• Define and maintain data dictionaries, integration specifications, and architecture documentation
• Drive cross-product data integration in complex and evolving environments

REQUIREMENTS

• 5+ years of professional data engineering experience
• Expert-level hands-on Apache Kafka experience, including production cluster setup and administration
• Strong experience with Kafka broker configuration, topics, partitions, replication, SASL/SSL, ACLs, schema registry, monitoring, alerting, and performance tuning
• Deep expertise in canonical and semantic data modelling
• Strong experience integrating data across heterogeneous systems with different schemas, frequencies, and business contexts
• Strong SQL and Python skills for production data engineering
• Experience designing data platforms for reporting, analytics, AI, and machine learning use cases
• Strong understanding of data contracts, integration standards, and reusable data architectures
• Excellent technical documentation and written communication skills
• Ability to work autonomously and drive solutions in ambiguous environments
• Experience with data observability, lineage, telemetry, and data quality monitoring is strongly preferred
• Experience with Databricks, Microsoft Fabric, Snowflake, SageMaker, or similar platforms is an advantage
• Experience supporting AI/ML pipelines, feature stores, training datasets, or embedding-ready data structures is an advantage
• Knowledge of Kimball, Data Vault, and medallion architecture approaches is preferred
• Experience with energy, Oil & Gas, or industrial operational data is a strong advantage

SoftServe is an equal opportunity employer. Qualified applicants will receive consideration regardless of race, color, ancestry, ethnicity, national origin, religion, sex, sexual orientation, gender identity or expression, age, citizenship, disability, health condition, marital or family status, veteran status, or any other characteristic protected by applicable law.