Lead BigData Engineer (Databricks and AWS)

ABOUT THE ROLE

In this role, you will lead the development of scalable data platforms and processing solutions on AWS, while remaining actively involved in hands-on engineering. You will work with batch and real-time data architectures, guide technical implementation, and collaborate with engineers, architects, and business stakeholders to deliver reliable, production-ready data solutions.

RESPONSIBILITIES

• Lead the design and implementation of scalable data pipelines and platform components on AWS
• Drive end-to-end delivery of data engineering solutions, ensuring quality, performance, and maintainability
• Provide technical guidance and support to data engineers within the team
• Build and maintain batch and streaming data processing solutions using modern big data technologies
• Develop data ingestion processes, source-to-target mappings, and integrations with new data sources
• Build and orchestrate workflows using Apache Airflow or MWAA
• Implement real-time data processing using Apache Kafka, Amazon MSK, or Kinesis
• Contribute to data warehousing and analytics solutions using Snowflake or Amazon Redshift
• Collaborate with architects and stakeholders to translate business requirements into effective technical solutions
• Promote engineering best practices, code quality, and continuous improvement across the team

REQUIREMENTS

• Strong hands-on experience in Big Data / Data Engineering, including building scalable data pipelines and platforms
• Proven experience providing technical leadership and guiding engineering teams
• Strong expertise with AWS services for data engineering and analytics
• Proficiency in Python, Scala, or Java, with strong SQL skills
• Hands-on experience with Apache Spark, Databricks, Flink, or similar big data technologies
• Practical experience with Apache Airflow or MWAA
• Experience with streaming technologies such as Apache Kafka, Amazon MSK, or Kinesis
• Familiarity with Snowflake or Amazon Redshift
• Strong understanding of data engineering best practices, performance optimization, and production-grade development
• Strong problem-solving, stakeholder collaboration, and communication skills
• Upper-intermediate or higher level of English

SoftServe is an equal opportunity employer. Qualified applicants will receive consideration regardless of race, color, ancestry, ethnicity, national origin, religion, sex, sexual orientation, gender identity or expression, age, citizenship, disability, health condition, marital or family status, veteran status, or any other characteristic protected by applicable law.