Senior Software Engineer

Job description:

We are seeking a highly skilled and analytical Senior Data Engineer with deep expertise in designing, building, and maintaining end-to-end distributed data pipelines within the Apache Hadoop ecosystem. In this role,

A Day in the Life of Senior Data Engineer:

You will lead the architecture and implementation of scalable batch and real-time streaming pipelines.

You will be responsible for ingest, storage, large-scale processing, and serving layers, transforming massive volumes of unstructured, semi-structured, and structured data into high-performance datasets ready for downstream analytics and machine learning applications.

So, how are we going to keep you busy?:

• Pipeline Architecture: Design, build, and optimize scalable end-to-end data pipelines (ETL/ELT) using Hadoop ecosystem components.
• Data Ingestion: Implement high-throughput ingestion pipelines utilizing Apache Nifi or Apache Kafka or Flume to capture streaming event data and transactional batch files.
• Distributed Processing: Develop robust data transformation workflows using Apache Spark (Core, SparkSQL, Structured Streaming) and MapReduce.
• Data Storage & Modeling: Optimize data layout in HDFS utilizing columnar formats (Parquet, ORC) and manage NoSQL persistence layers using HBase or Cassandra.
• Query Performance Optimization: Author and tune highly complex distributed queries in Apache Hive to meet tight SLAs.
• Orchestration & Workflow Management: Automate and monitor multi-stage pipeline DAGs using orchestration tools like Apache Airflow or Oozie.
• Cluster Performance Tuning: Partner with DevOps/Infrastructure teams to debug issues related to data skew, resource allocation (YARN), cluster memory management, and Spark join strategies.

What will make you eligible for this role?:

• Relevant bachelor's degree or master's degree.
• 6-8 years of relevant experience in IT industry (Telecom preferred but not mandatory).
• 3 to 7+ years of dedicated data engineering experience, with at least 3 years actively managing end-to-end production pipelines on Hadoop clusters.
• Proficiency working in Linux/Unix operating systems, shell scripting, and using code versioning tools like Git.

Experience, that will be an advantage:

• Core Big Data Frameworks: Hands-on mastery of HDFS, MapReduce, YARN, Hive.
• Distributed Compute: Advanced proficiency in Apache Spark using Python (PySpark) or Scala.
• Streaming & Ingestion: Deep operational knowledge of Apache Nifi or Kafka (brokers, producers, consumers, and schema registry).
• Programming Languages: Strong foundations in Python, SQL, and Scala or Java.
• NoSQL Database Systems: Direct experience handling unstructured/semi-structured data architectures in HBase, MongoDB, or Cassandra.
• Data Warehouse & Modelling: Strong grasp of dimensional data modelling, star/snowflake schemas, and data governance practices.

Skills

Airflow, ELT, ETL, Hadoop, Hive, Java, Kafka, PySpark, Scala, Spark