Staff / Lead Data Engineer

Our client is seeking multiple technical leaders to contribute at both the Staff and Lead Engineer level. These are high-visibility, high-impact opportunities with a talent-dense, homegrown, bootstrapped, globally competitive Canadian tech success story. The company finds itself squarely in the middle of the disruption of search by AI/LLMs, the fragmentation of the media landscape and in a chapter of rapid, transformational growth.

The shared mandate is to modernize how massive volumes of data are processed, modeled, and made available to products and customers. The work spans petabyte-scale batch and streaming pipelines, low-latency APIs, analytical platforms, and the architecture that connects raw event data to reliable, production-grade data products.

Staff Engineer candidates may bring a stronger centre of gravity in backend and API systems or in data processing and platform engineering. For the Lead Engineer search, the client is seeking a hands-on technical and people leader with the affinity and aptitude to guide a distributed team through transformation.

The client

Our client builds a high throughput system powering hundreds of billions of daily transactions, each completed within milliseconds, across globally distributed infrastructure designed for reliability and efficiency. They have been quietly bootstrapping and growing in line with revenue for over 2 decades. They operate at a planetary scale with an exchange platform with thousands of nodes that handles nearly 500+ billion daily auctions and is trending towards 1 trillion daily auctions in the next ~4 years. To the present day, the company has not raised money from venture capitalists. It maintains its independence from external stakeholders, enabling it to chart its course, maintain a long-term perspective and build an enduring, sustainable business that currently employs ~600 team members globally.

Data is central to nearly every part of its platform. High-volume event and transactional data powers customer reporting, billing, marketplace analytics, experimentation, machine-learning workflows, and product experiences. The organization is evolving from centralized, batch-oriented reporting toward a platform-driven architecture combining batch, streaming, and asynchronous processing, with APIs becoming a primary interface to data.

This is not conventional business-intelligence work. The challenge is to process and serve enormous datasets with strong correctness guarantees, predictable latency, and disciplined infrastructure economics. Engineers work across the full path from edge and transport through compute, storage, query execution, and serving, balancing latency, cost, complexity, and reliability at every layer.

Our client has been stubbornly racking and stacking infrastructure around the world for the duration of their existence, a habit that allows for them to price themselves at the cost of electricity while their competitors are mired in rising cloud infrastructure costs. This long-term, somewhat contrarian, thinking puts them in a position to offer stability and career longevity evidenced by robust benefits that include RRSP matching.

If you’re looking for a role that offers an opportunity to innovate and optimize systems at a massive scale, creating a lasting impact in an environment that values technical excellence and resilience, this could be for you.

Role overviews

In our client’s context, both Staff Engineers and Leads remain hands-on. This is an intentional and informed cultural practice tailored to the challenge ahead of them. Leads are expected to understand the systems deeply, make informed architectural decisions, and help their teams execute.

As a Staff Engineer, you will be a primary owner and technical leader for data-intensive systems spanning processing, platform, and product-facing services. You will drive cross-team architecture, define reusable platform patterns, lead complex multi-system initiatives, and influence the long-term evolution of data processing and access across the organization. Depending on your background, your work may lean toward one or both of the following domains:

Data products and APIs: This team builds and operates high-scale services exposing aggregated and near real-time data for reporting, analytics, and customer-facing products; shaping API contracts, query abstraction and execution, caching, response design, performance, scalability, and reliability. This team operates as a full-stack data engineering group, owning the path from data generation and aggregation through query execution, APIs, and user-facing delivery. On this team you can expect to:

• Design high-scale APIs for multidimensional reporting and analytical workloads
• Define contracts for metrics, dimensions, filters, schemas, and aggregations
• Improve dynamic query generation, caching, response shaping, and execution performance
• Connect APIs to batch, streaming, and near real-time data sources
• Build primarily in Go or a comparable backend language such as Java
• Improve observability, incident response, reliability, and consumer experience
• Lead initiatives involving Product, Data Platform, ML, and Application Engineering

Data processing and platform: This team builds and evolves large-scale batch and streaming pipelines that transform raw event data into clean, canonical, business-ready datasets; improving data models, workflow orchestration, correctness, observability, runtime, and compute efficiency. On this team you can expect to:

• Build large-scale Spark pipelines and streaming systems using Kafka and Flink
• Transform raw event data into canonical, production-grade datasets
• Design partitioning, storage, aggregation, and query strategies for enormous datasets
• Support historical backfills, incremental processing, and real-time availability
• Help decompose centralized processing into domain-oriented streams and workloads
• Move appropriate services and processing patterns from Hadoop toward Kubernetes
• Improve data quality, correctness, observability, throughput, and compute efficiency
• Participate in the long-term design of globally distributed data infrastructure

As a Lead Engineer you will lead a distributed team of data and software engineers responsible for high-scale, low-latency data processing pipelines, analytical platforms, and reporting products. In practice this means:

• Set the technical direction for the team’s products, systems, and architectures while remaining comfortable working directly with the code and technology
• Hire, mentor, and develop engineers, creating an environment of accountability, collaboration, and meaningful technical growth
• Partner with Product to establish roadmaps, goals, forecasts, and measurable technical and business outcomes
• Manage scope and sequencing across more opportunities than available resources, communicating progress and risks early and coordinating dependencies across teams
• Own the quality, health, and outcomes of the team’s platforms, including operational response, technical debt, reliability, and continuous improvement

The shared mandate

• Platform architecture: Define and evolve patterns across edge, transport, compute, storage, modeling, query execution, and serving
• Batch and streaming systems: Build systems supporting historical processing, backfills, incremental updates, and near real-time availability without creating unnecessary architectural complexity
• Data products and interfaces: Provide stable, flexible access to metrics, dimensions, filters, aggregations, and canonical datasets through well-designed APIs and analytical platforms
• Performance and economics: Optimize query planning, data layout, partitioning, caching, execution, throughput, and infrastructure use so capacity does not need to grow linearly with data volume
• Correctness and trust: Design for deduplication, late-arriving data, schema evolution, data contracts, reconciliation, and the integrity of business-critical reporting and billing datasets
• Reliability and observability: Own monitoring, alerting, incident response, root-cause analysis, workflow health, and the mechanisms that make system behaviour visible
• Cross-functional influence: Work across Data Engineering, Data Systems, Application Engineering, Product, analytics, machine learning, and experimentation teams to turn platform capabilities into business outcomes

Tech stack

• Processing and streaming: Spark, Scala, Kafka, Flink, Airflow
• Data services and APIs: Go, Java, REST, gRPC, SQL
• Compute and storage: Kubernetes, Hadoop/HDFS, Ceph
• Query and analytics: Trino, Vertica, Druid and StarRocks
• Reporting and BI: Looker, Superset, Redash

The precise mix will vary by team and assignment. Depth in comparable technologies and the ability to reason from first principles matter more than matching every tool.

Key responsibilities

• Vision and strategic direction: Identify high-leverage technical opportunities and translate them into a coherent direction for the platform and business
• System design and delivery: Lead the design and implementation of distributed data systems operating under heavy workloads and demanding latency requirements
• Data platform evolution: Help move the organization from centralized, batch-oriented systems toward workload-aware architectures combining batch, streaming, and asynchronous processing
• API and query architecture: Define durable interfaces, schemas, query abstractions, and serving patterns for flexible access to large-scale multidimensional data
• Pipeline and model ownership: Build canonical datasets and processing workflows supporting reporting, billing, products, analytics, experimentation, and machine-learning use cases
• Operational excellence: Improve observability, service health, incident response, workflow reliability, technical debt management, and post-incident learning
• Organizational leadership: Coordinate across teams, unblock dependencies, communicate trade-offs clearly, and build alignment around technical and product outcomes

Your know-how

• You have significant experience designing, building, and operating data-intensive systems at scale
• You have a deep understanding of distributed-systems fundamentals, including partitioning, consistency, failure modes, state, throughput, latency, and cost trade-offs
• You have experience with large-scale data processing and modeling (the client leans on Spark currently)
• You have experience with streaming technologies such as Kafka and Flink, including incremental processing, late data, deduplication, and replay
• You have experience designing stable data contracts, schemas, canonical datasets, and transformation layers for business-critical use cases
• You have experience with large-scale storage, data warehouse, or analytical query systems and a strong grasp of query performance, data layout, and workload-aware optimization
• You have experience with workflow orchestration and the production deployment or operation of services on Kubernetes
• You have a track record of end-to-end ownership: navigating ambiguity, debugging complex cross-system issues, making sound trade-offs, and improving systems after they enter production
• You have an excellent command of English and experience collaborating with neighbouring engineering disciplines and stakeholders beyond R&D

For API-oriented work, you will call backend engineering skills (ideally but not necessarily with Go) and experience with REST/gRPC design, versioning, performance, and high-concurrency services.

Additional know-how for the Lead Engineer roles

• You have 3+ years of experience leading and managing distributed teams
• You have an affinity for mentorship and creating an environment that nurtures a balance of innovation and accountability
• You have experience partnering with Product Management and internal stakeholders to shape roadmaps, forecasts, deliverables, timelines, and measurable outcomes
• You are experienced making tradeoffs and experience exercising judgment to prioritize a broad portfolio when initiatives outnumber resources, including thoughtful sequencing and proactive communication of progress, delays, and risk