Scala + Spark Engineer

Responsibilities:

• Create and maintain existing Scala/Spark jobs for data transformation and aggregation, from simple to complex data transformations involving structured and unstructured data.
• Produce unit tests for Spark transformations and helper performance optimization methods.
• Develop data processing pipelines, data storage, and management architecture.
• Define scalable calculation logic for interactive and batch use cases.
• Interact with infrastructure and data teams to produce complex analysis across data.
• You will be working on a unique and challenging big data ecosystem with a focus on storage efficiency, data security and privacy, scalable and performant queries, expandability and flexibility, etc., and to help better measure the quality of map data.
• You will work with engineers to build a big data platform that processes and manages exabytes of data and enables efficient access to that data.

Requirements:

• Minimum of 5 years of work experience in Big Data platforms and building large-scale distributed systems with high availability.
• Experience in developing Spark applications using Spark RDD API, Spark-SQL, Spark GraphX API, Spark Streaming API, Spark MLlib API, and DataFrames APIs.
• Should have broad knowledge of Spark advantages, Spark workflows, how to write Spark jobs, Spark workflows, Spark query tuning, and performance optimization.
• Solid in data structures and algorithm basics and should have good hands-on experience with the Scala programming language and strong investigative and problem-solving skills.
• Data ingestion, optimization techniques, data transformation, and aggregation pipeline design/development knowledge are required.
• Experience working on cutting-edge big data storage systems and technologies like Hadoop, HDFS, AWS S3, AWS Lambda, Storm/Heron, Cassandra, Apache Kafka, Solr/ElasticSearch, MongoDB, DynamoDB, Postgres, and/or MySQL, etc.

Skills

Scala