DE&A -Data Engineering Apache

Design, develop, and implement scalable, resilient, and high-performance data pipelines and data architecturesfor enterprise data platforms.

· Build and optimize ETL/ELT pipelines supporting batch, near-real-time, and streaming data processing requirements.

· Design and implement enterprise data ingestion and integration workflows using Apache NiFi, including data routing, transformation, orchestration, monitoring, and integration across heterogeneous data sources and platforms.

· Develop distributed data processing solutions using Apache Spark with Scala and/or Python (PySpark).

· Design and implement data solutions across relational database management systems (RDBMS) and modern data lakehouse technologies such as Apache Iceberg.

· Build and optimize federated and distributed SQL capabilities using Starburst and Trino query engines.

· Develop event-driven and real-time data processing solutions using Apache Kafka and Spark Streaming / Structured Streaming.

· Lead end-to-end delivery of enterprise-scale data engineering initiatives involving multiple engineering, product, architecture, and business teams.

· Strengthen platform reliability through automation, CI/CD, testing, monitoring, observability, documentation, and operational excellence.

· Improve data quality, performance, scalability, resiliency, and cost efficiency across data platforms and pipelines.

· Support enterprise data governance by maintaining metadata, technical and business documentation, data lineage, data quality controls, and secure access to data assets.

· Partner with security and governance teams to ensure data solutions meet enterprise requirements for privacy, security, access control, compliance, and data lifecycle management.

· Troubleshoot complex production issues, perform root-cause analysis, and implement sustainable solutions to improve platform stability.

· Collaborate effectively with stakeholders across engineering, architecture, analytics, AI/ML, governance, product, and business organizations.

7+ years of experience in Data Engineering, Software Engineering, Enterprise Data Platforms, or a related technical discipline.

· Strong hands-on experience designing and delivering enterprise-scale data pipelines and data architectures.

· Strong understanding of ETL and ELT patterns, data integration, data modeling, and distributed data processing.

· Hands-on development experience with Apache Spark using Scala and/or Python/PySpark.

· Experience working with RDBMS technologies and modern lakehouse/table formats, preferably Apache Iceberg.

· Experience with distributed SQL/query technologies such as Starburst and Trino.

· Experience designing real-time and streaming data solutions using technologies such as Apache Kafka and Spark Streaming.

· Understanding of integration patterns across NiFi, Kafka, Spark, RDBMS, and lakehouse platforms for batch and real-time data processing.

· Strong knowledge of software engineering practices including CI/CD, automated testing, source control, infrastructure/platform automation, observability, and production support.

· Experience with data governance, metadata management, data lineage, data quality, and secure data access in enterprise environments.

· Proven ability to lead end-to-end delivery of enterprise-scale data engineering initiatives across multiple teams.

· Ability to lead technical discussions, influence architectural decisions, establish engineering standards, and mentor engineers.

· Excellent problem-solving, communication, stakeholder management, and cross-functional collaboration skills.

· Ability to translate complex business and technical requirements into scalable, maintainable, and production-ready data engineering solutions.

· Demonstrated ownership mindset with a strong focus on engineering quality, platform reliability, security, scalability, and operational excellence.

NA