Data Engineer III

As our Data Engineer III, you will be responsible for designing and developing highly scalable data pipeline solutions and collectively owning their data integrity and consistency. You must be able to see the big picture of the company's data situation and be an expert in building distributed data solutions. You need to take the challenges and ownership that come with developing a complex system. You will be working in a fast-paced and agile work environment, delivering quality and innovative solutions in highly collaborative environments that have immediate business impact. As a DE-III, you will also get to work with Senior Engineers at Junglee Games to evolve the design and architecture of the system.

Responsibilities:

• Hands-on - Responsible for writing code primarily in Spark, Python, and/or Scala.
• Hands-on - Responsible for developing streaming applications like Spark structured streaming and Kafka streams.
• Scale our current data infrastructure to handle millions of events/transactions per day.
• Own data integrity and consistency of our Data lake and Data warehouse.
• Working with AWS-managed services and managing NoSQL stores like Cassandra, ElasticSearch, Redis, MongoDB, ScyllaDB, etc.
• Monitor and improve the best practices and processes to manage the data pipeline and ETL operations.
• Work closely with Senior engineers to design, implement, and deploy Data pipelines that impact the Junglee Games business with an emphasis on the gaming industry.
• Solve complex problems innovatively and deliver quality solutions while taking ownership and accountability for assigned tasks.
• Demonstrate good learnability and adopt technologies that help build large-scale, performant, reliable, and sustainable systems.
• The ideal candidate will be a leader, a quick learner, and able to work independently.
• Collaborating with peers on all elements of the development process as per industry coding standards and creating appropriate technical documentation.

Requirements:

• A solid engineer at heart with strong abstraction, coding, and distributed system development experience.
• Excellent programming skills in Python/Scala and expertise in multi-threading and object-oriented programming.
• In-depth knowledge of Spark, Kafka, Streaming Systems, AWS, and Cloud Data Warehouse like Trino/Snowflake, and Docker/Kubernetes.
• A proven track record of leading a team and developing in a startup/e-commerce ecosystem within a high-growth and matrix environment.
• Empathetic, motivated, and a Good team player.
• Excellent listener and interpreter.

Special consideration was given for:

• Knowledge in Big Data, Amazon Web Services(AWS), and Apache Spark.
• Knowledge of Cloud Data warehouses such as Redshift, Snowflake, and Trino.
• Knowledge of streaming applications such as Spark Streaming and Kafka.
• Knowledge of writing complex SQL procedures.

Skills

Apache Kafka, Apache Spark, Kafka, Python, Scala, Spark