Big Data Developer

Responsibilities:

• Create new and maintain existing Scala/Spark jobs for data transformation and aggregation from simple to complex data transformations involving structured and unstructured data.
• Produce unit tests for Spark transformations and helper performance optimisation methods.
• Develop data processing pipelines, data storage, and management architecture.
• Define scalable calculation logic for interactive and batch use cases
• Interact with infrastructure and data teams to produce complex analysis across data
• You'll be working on a unique and challenging big data ecosystem with a focus on storage efficiency, data security and privacy, scalable and performant queries, expandability and flexibility, etc., and to help better measure the quality of map data.
• You will work with engineers to build a big data platform that processes and manages exabytes of data and enables efficient access to that data.

Requirements:

• Minimum 7 years of working experience in Big Data and Hadoop platforms, and building large-scale distributed systems with high availability.
• Experience in developing Spark Applications using Spark RDD API, Hadoop, Spark SQL, Spark Streaming API, Spark MLlib API, and DataFrames APIs
• Should have broad knowledge of Spark advantages, Spark workflows, how to write Spark jobs, Spark query tuning and performance optimisation.
• Solid in data structures and algorithm basics.
• Should have good hands-on experience with any one programming language (Scala/Java 8 - 1st preference, OR Python - 2nd preference OR Java - 3rd preference).
• Strong investigative and problem-solving skills.
• Data ingestion, optimisation techniques, data transformation and aggregation pipeline design/development knowledge are required.
• Experience working on cutting-edge Big Data storage systems and technologies like Hadoop, HDFS, AWS S3 AWS Lambda, Storm/Heron, Cassandra, Apache Kafka, Solr/ElasticSearch, MongoDB, DynamoDB, Postgres, MySQL, etc.

Skills

spark, apache spark, hadoop, hdfs, java, python, scala