Data Engineer (Real - Time Streaming & Big Data)

We are looking for a hands-on Data Engineer with strong experience in big data engineering and real-time streaming systems. You will work closely with Data Platform, Analytics, and Machine Learning teams to design and build scalable, reliable, and low-latency data pipelines that enable real-time analytics and decision-making.

Responsibilities:

• Design, develop, deploy, and own real-time streaming applications using Apache Flink.
• Build and maintain Spark Structured Streaming pipelines for high-volume, real-time data processing.
• Tune and troubleshoot Flink and Spark workloads in production, including: Checkpointing, State management, Backpressure handling, Resource and performance optimization.
• Develop and optimize batch and streaming ETL/ELT pipelines using PySpark and Python.
• Write efficient and optimized SQL for data transformation, validation, and downstream analytics consumption.
• Build and deploy cloud-native data pipelines on AWS, leveraging services such as EMR, S3 AWS Glue, and Lambda.
• Design scalable solutions capable of meeting defined latency, throughput, reliability, and data accuracy SLAs.
• Implement monitoring, schema validation, error handling, and data quality checks to ensure pipeline reliability.
• Apply strong programming, data structures, and algorithms fundamentals to build performant, maintainable, production-grade systems.
• Work effectively across a complex ecosystem involving technology teams, vendors, business stakeholders, and other engineering teams.
• Contribute to Agile projects spanning multiple teams, business units, and organizations.

Requirements:

• 3-5 years of professional experience in Data Engineering / Big Data Engineering.
• Strong hands-on experience with Apache FlinkMandatory.
• Experience building and supporting production-grade real-time streaming applications.
• Strong experience with Apache Spark / Spark Structured Streaming.
• Hands-on expertise in PySpark and Python.
• Strong understanding of streaming architectures, distributed data processing, and large-scale data systems.
• Strong working knowledge of SQL and data transformation.
• Experience building data pipelines on AWS, particularly with: EMR, S3 AWS Glue, Lambda.
• Experience with performance tuning, debugging, monitoring, and optimization of production data pipelines.
• Ability to work across multiple stakeholders, teams, and technology ecosystems.

Good to Have:

• Programming experience in Java or Scala.
• Strong understanding of Data Structures and Algorithms (DSA).
• Experience working in Agile environments across multiple organizations or business units.
• Key Skills: Apache Flink, Spark Structured Streaming, Apache Spark, PySpark, Python, SQL, AWS, EMR, S3 AWS Glue, Lambda, Real-Time Streaming, ETL/ELT, distributed systems, and big data.