Data Engineer IV

We are hiring a Tech Lead / SDE IV to anchor and grow our data engineering capability within our Demand-Side Platform (DSP). This is a high-impact, technically hands-on leadership role at the intersection of real-time data infrastructure, programmatic advertising, and scalable backend systems. You will own the design and delivery of pipelines that power bidding intelligence, audience targeting, campaign analytics, and attribution, processing billions of bid-stream events daily. Alongside your data engineering mandate, you will architect and contribute to backend microservices that sit at the core of our DSP stack, including integrations with SSPs, ad exchanges, and identity resolution platforms. You will mentor a team of engineers, drive technical direction, and collaborate closely with product, ML, and growth functions to turn data into a true competitive advantage.

The core responsibilities for the job include the following:

Data Stack:

• Design and own end-to-end data pipelines, batch and real-time, handling bid requests, win/loss events, impression logs, click streams, and conversion signals at scale.
• Build and maintain our data lakehouse architecture (ingestion, storage, and serving layer) optimised for low-latency DSP analytics and ML feature generation.
• Define standards for data quality, lineage, observability, and SLA adherence across all data products.
• Own and evolve our StarRocks deployment for real-time analytical queries, schema design, ingestion patterns, and query optimisation for DSP reporting workloads.
• Drive FinOps discipline across the data platform, track compute and storage costs, enforce resource quotas, identify optimisation opportunities, and report on unit economics.
• Establish and lead data governance practices: metadata management, data cataloguing, access control policies, PII classification, and data retention standards compliant with GDPR and CCPA.

Backend Microservices:

• Contribute to and review the design of backend services that power DSP functionality, bid shading, pacing, budget management, and reporting APIs.
• Architect data contracts and event schemas between microservices, ensuring consistency across the bid request lifecycle.
• Champion best practices in API design, service reliability (SLOs, circuit breakers, retries), and observability (distributed tracing, structured logging, metrics).
• Collaborate with the bidder team on low-latency data reads (Aerospike, Cassandra, or similar) for real-time targeting signal lookups.

Technical Leadership:

• Lead architecture reviews, set engineering standards, and drive adoption of best practices across the data and backend engineering teams.
• Mentor SDE II/III engineers through code reviews, design sessions, and pair programming; grow the next generation of technical leaders.
• Partner with ML engineers on the full model lifecycle feature engineering pipelines, offline training data generation, experiment tracking, and model serving infrastructure.
• Translate complex business requirements (campaign KPIs, ROAS, viewability, brand safety) into robust technical designs.
• Own the technical roadmap for data infrastructure capacity planning, migration strategies, and build vs. buy decisions.

Requirements:

• 8+ years of software engineering experience, with at least 4 years focused on data engineering or large-scale data systems.
• Prior experience in ad-tech, programmatic advertising, or real-time bidding (RTB) environments strongly preferred.
• Experience in ML engineering, feature engineering, building offline training pipelines, or collaborating closely on model productionisation is preferred.
• A bachelor's or master's degree in computer science, engineering, or a related technical field.

Data Engineering (Core):

• Deep expertise in distributed stream processing frameworks Apache Kafka, Apache Flink, or Apache Spark Structured Streaming.
• Strong command of SQL and experience optimising complex analytical queries at petabyte scale.
• Hands-on experience with real-time analytical databases, particularly StarRocks; familiarity with Snowflake or ClickHouse is a plus.
• Proficiency with workflow orchestration tools: Apache Airflow, Prefect, or Dagster.
• Solid understanding of data modelling, dimensional modelling, OBT patterns, and event-driven schemas relevant to ad impression and attribution data.
• Familiarity with open table formats: Apache Iceberg, Delta Lake, or Apache Hudi.

Backend and Systems:

• Proficiency in Python (primary) and/or Java/Scala for building production-grade services and data processing jobs.
• Experience designing and building RESTful APIs and/or gRPC-based microservices.
• Working knowledge of low-latency data stores (Aerospike, Cassandra) used for real-time lookups in the bidding path.
• Comfortable with containerised deployments on Kubernetes and CI/CD pipelines (GitHub Actions, ArgoCD, or similar).

Ad-Tech Domain Knowledge:

• Understanding of the RTB ecosystem: DSPs, SSPs, ad exchanges, bid request/response lifecycle, and OpenRTB protocol.
• Familiarity with identity resolution approaches: cookie IDs, device IDs, UID2 / RampID, and privacy-preserving identity.
• Exposure to audience segmentation, frequency capping, pacing algorithms, or attribution models is a significant plus.
• Awareness of privacy regulations (GDPR, CCPA) and their implications for data pipelines and user-level data handling.

Leadership and Collaboration:

• Demonstrated experience leading technical squads or functioning as a technical anchor on cross-functional projects.
• Strong written and verbal communication skills; ability to translate technical complexity for non-engineering stakeholders.
• Track record of driving projects from ambiguous requirements to production with high engineering quality.