Data Engineer

Responsibilities:

• Designing data systems.
• Reviewing code.
• Developing/Implementing Part of Systems.
• Develop a comprehensive understanding of the team's projects and interdependencies, ensuring alignment across initiatives.
• Partially own team-wide and cross-team projects, driving the definition, scoping, and delivery of features.
• Mentoring people in the team. On-call monitoring and support.
• Working across engineering and business teams to clarify requirements.
• Building strong technical ownership (Build, Ship and Own) and driving technical vision for the team.
• Should own the product from requirement till maintenance.
• Build strong ownership on ACR-related data pipeline development. Contribute to governance initiatives and migration of on-premise infrastructure to the cloud (Databricks).
• Scalable Data Models Pipeline processing TBs of data daily. Data Quality and Monitoring Challenges at Scale.

Requirements:

• Cross-engineering team. Operations. Data Analysts, Product. Engineering. Ops. External Clients.
• Spark (PySpark/Scala Spark), Databricks, S3 Git, IAC, Pulumi, Kafka, Airflow, and AWS.
• People should not limit themselves to the tech and should be able to pick up new stuff based on the team's needs.
• Understand and build ownership of big complex technical components.
• Should start deriving vision and development within a month.
• Exposure to ad systems like DSP and SSP through data pipelines and modelling.
• Exposure to state-of-the-art ACR systems through data pipelines. Work with a cutting-edge technology stack.
• Strong ownership-driven culture. High-Growth Company.

Core Skills:

• Technical requirements (please indicate must-have vs. nice-to-have requirements): Spark (Scala Spark/PySpark), Git, Airflow (must), Kafka (nice), SQL (must), DBMS (must), S3 (nice), ElasticSearch/OpenSearch (nice), IAC (must), and AWS (nice). Databricks (Big Plus) Data Modelling and Client Management Requirements.