Big Data Developer

Job Description :

We are looking for an experienced Big Data Developer with 7-11 years of experience to design, develop, and optimize large-scale data processing solutions. The ideal candidate should have strong expertise in Apache Spark, Python, Hadoop, distributed computing, streaming technologies, and cloud-based data platforms, along with the ability to provide technical guidance to development teams.

Key Responsibilities :

- Design, develop, and maintain scalable Big Data solutions for high-volume and complex data workloads.

- Develop distributed data processing applications using Apache Spark and Python.

- Design and implement batch and real-time data processing pipelines.

- Work with Hadoop ecosystem technologies including HDFS, MapReduce, Sqoop, Hive, and Impala.

- Develop stream-processing solutions using Spark Streaming, Apache Storm, or equivalent technologies.

- Design and integrate messaging and event-streaming solutions using Apache Kafka, RabbitMQ, or similar platforms.

- Build data pipelines integrating data from multiple sources including RDBMS, ERP systems, APIs, and files.

- Develop and optimize SQL queries, joins, stored procedures, and relational data models.

- Work with NoSQL databases such as HBase, Cassandra, or MongoDB based on application requirements.

- Implement ETL processes and data integration frameworks for large-scale data environments.

- Perform Spark job optimization and performance tuning to improve processing efficiency and resource utilization.

- Design and implement cloud-based Big Data solutions using Azure Databricks and related Azure data services.

- Troubleshoot production issues and ensure reliability, scalability, and performance of data platforms.

- Collaborate with data engineers, architects, analysts, and business stakeholders to translate requirements into technical solutions.

- Provide technical guidance and mentor team members while contributing to design and architecture decisions.

Required Skills :

- 7+ years of experience in Big Data and data engineering technologies.

- Strong understanding of distributed computing principles and architectures.

- Expert-level hands-on experience with Apache Spark.

- Strong programming experience in Python.

- Proficiency in Hadoop v2, HDFS, MapReduce, and Sqoop.

- Experience with Spark Streaming, Apache Storm, or equivalent stream-processing technologies.

- Hands-on experience with Kafka, RabbitMQ, or similar messaging platforms.

- Strong knowledge of Hive, Impala, and Big Data querying technologies.

- Strong SQL skills, including complex joins, stored procedures, and relational schemas.

- Experience working with NoSQL databases such as HBase, Cassandra, or MongoDB.

- Strong understanding of ETL techniques and data integration frameworks.

- Experience in Spark performance tuning and optimization.

- Hands-on experience with Azure Databricks and cloud-based data services.

- Strong problem-solving, analytical, and communication skills.

- Experience leading or technically guiding a team is required.

- Ability to design and implement scalable, reliable, and high-performance data solutions.

Skills

Data Engineering, Apache Spark, Hadoop, Python, Azure Databricks, Big Data, HDFS, Hive, RDBMS, SQL