{"id":4615567,"url":"https://alion.io/job/the-trade-desk-software-engineer-ii-dataproc","title":"Software Engineer II, DataProc","company":{"id":684619,"name":"The Trade Desk","domain":"thetradedesk.com","url":"https://alion.io/company/thetradedesk","size_band":"1001-5000","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Greenhouse","truth_index":{"grade":"A","score":92,"open_postings":72,"ghost_share":0,"stale_share":0.111,"repost_share":0,"time_to_fill_p50_days":85,"computed_at":"2026-11-19T09:08:57Z"}},"role":"Backend","role_family":"Backend","seniority":"middle","employment_type":null,"work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["London, United Kingdom"],"countries":["GB"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":2,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"AWS","optional":false},{"name":"CI/CD","optional":false},{"name":"Databricks","optional":false},{"name":"Hadoop","optional":false},{"name":"Java","optional":false},{"name":"Kubernetes","optional":false},{"name":"OpenTelemetry","optional":false},{"name":"Python","optional":false},{"name":"Scala","optional":false},{"name":"Spark","optional":false},{"name":"Grafana","optional":true},{"name":"Machine Learning","optional":true},{"name":"Prometheus","optional":true},{"name":"Snowflake","optional":true},{"name":"SQL","optional":true},{"name":"Vertica","optional":true}],"status":"live","first_seen_at":"2026-11-19T14:22:38Z","employer_posted_date":"2026-11-19","last_verified_at":"2026-11-19T16:10:25Z","board_verified":true,"closed_at":null,"days_open":0,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":0},"description":"The Trade Desk is a global technology company and the world’s leading independent platform for digital advertising, with over 3,000 employees across more than 30 offices. Our technology helps advertisers reach the right audiences across the open internet - from streaming TV and podcasts to mobile apps, news, and more. \nAdvertising powers the content people love. By making it more transparent, effective, and responsible, we help support trusted journalism, quality entertainment, and creators worldwide. The world’s brands and agencies rely on us to reach their customers and grow their businesses responsibly.\nThe scale of our platform brings unique technical challenges - from processing massive datasets in real time to building systems that operate reliably on a global scale. When you work here, your impact is worldwide. We welcome diverse perspectives, encourage curiosity, and build teams that learn from one another. If you’re driven to solve meaningful challenges, we’d love to meet you.\nWhat wedo \nOur Software Engineers are end-to-end owners who have the opportunity to participate in many aspects of designing, building, and delivering data-focused products for our stakeholders. \nAt The Trade Desk, we believe in building the back-end infrastructure of our platform with a focus on quality at scale. This means whether we are building parts of our client-facing applications, internal custom solutions for our own team members, or model pipelines for bidding optimizations, we are making sure that we have infrastructure, development, and tools to do it right and efficiently. \nOur system performs all day, every day serving global traffic. We build a distributed system in a highly collaborative environment, utilizing a broad range of technologies. Our back-end developers work on finding solutions to algorithmic, optimization and scale challenges in everything we do. \nThe DataProc team owns the platform that orchestrates and processes The Trade Desk's data at scale - from workflow scheduling (Airflow, TaskService) to distributed compute (Spark on Amazon EMR and Kubernetes, Databricks). Nearly every engineering team at the company builds on top of what we ship, so the reliability, performance, and usability of our platform has an outsized impact across the business. \nWhat you’lldo: \nSome of the work that you will be doing to help us deliver on our mission is: \nDesign, build, and operate the services and libraries that make up our data orchestration and workflow platform (Airflow, TaskService), used by teams across the company.\nDevelop, scale, and optimize distributed data processing jobs running on Spark across Amazon EMR, Kubernetes, and Databricks.\nPartner with teams across engineering to migrate and modernize big data workloads - for example, moving jobs onto Databricks or from EMR onto Spark-on-Kubernetes.\nImprove the reliability and observability of our platform: metrics, alerting, tracing (OpenTelemetry), and on-call tooling.\nBuild and maintain internal tools and lightweight UIs that help engineers across the company monitor, debug, and operate their data pipelines.\nOwn your projects end-to-end - from design and code review through deployment, rollout, and production support.\nDebug and resolve complex issues in large-scale, distributed production systems.\nContribute to our technical roadmap and help define best practices for data pipeline development company-wide.\nWe are a global team with different backgrounds, experiences and perspectives. To complement this team, you will welcome ideas that are different from your own and be well-versed in building from common ground to value, seek out, and foster invisible and visible dimensions of diversity. \nWho you are: \nOur culture is much deeper than just having fun together (though, we do that well too...). We take pride in our engineers being trust-builders, generous givers, scrappy problem solvers, and gritty pursuers of excellence. You do not have to meet all the requirements below, but we believe that people who meet most of them will have a higher likelihood of succeeding in this role: \nStrong software engineering fundamentals, with production experience in Python, Scala, or Java.\nExperience building or operating distributed data processing systems (e.g., Spark, Databricks, Hadoop) and/or workflow orchestration tools (e.g., Airflow).\nComfortable working with cloud infrastructure (AWS, Kubernetes) and CI/CD pipelines.\nSolid grounding in data structures, algorithms, and distributed systems design.\nStrong debugging and troubleshooting skills, with a track record of resolving issues in large-scale production systems.\nSome experience with, or interest in, building internal tools, dashboards, or lightweight UIs that make platform capabilities more usable.\nA clear communicator who enjoys partnering with engineering teams across the company to understand their needs and unblock them.\nA pragmatic, ownership-driven approach to building and operating software.\nYou have 2+ years of software development/engineering experience\nYou have a Bachelor’s/Master’s level degree in a computer-science or relevant engineering related field or equivalent experience. Not everyone has the same level of access to opportunities. What is most important to us is what and how you can contribute, which is why our consideration is not limited by the level of education you have.\nYou have additional nice-to-have experience in the following areas: SQL and data warehousing (e.g., Snowflake, Vertica)\nKubernetes operators and cluster administration\nObservability tooling (OpenTelemetry, Grafana, Prometheus)\nCarrying production on-call responsibilities for critical systems\nExposure to machine learning pipelines or ad-tech / real-time bidding systems\n\nVariety of technical opportunity is one of the best things about working at The Trade Desk as a software engineer which is why we do not expect you to know every technology we use when you start. What we care about is that you can learn quickly and find solutions to complex problems using the optimum tools for the job.What you know is less important than how well you learn and innovate. We don't need engineers who know all the answers; we need engineers who can invent the answers no one has thought of yet, to the questions yet to be asked. \nThe Trade Desk does not accept unsolicited resumes from search firm recruiters. Fees will not be paid in the event a candidate submitted by a recruiter without an agreement in place is hired; such resumes will be deemed the sole property of The Trade Desk. The Trade Desk is an equal opportunity employer. All aspects of employment will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, veteran status, or any other status protected under federal, state, or local law.\nAs an Equal Opportunity Employer, The Trade Desk is committed to creating an inclusive hiring experience where everyone has the opportunity to thrive.\nPlease reach out to us at  to request an accommodation or discuss any accessibility needs you may require to access our Company Website or navigate any part of the hiring process. \nWhen you contact us, please include your preferred contact details and specify the nature of your accommodation request or questions. Any information you share will be handled confidentially and will not impact our hiring decisions.","description_format":"text","description_chars":7635,"description_truncated":false,"requirements":{"experience_years_min":2,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"bachelor","optional":false},"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Advertising"],"lifecycle":[{"event":"open","at":"2026-11-19T16:10:25Z"}],"visa":[{"country":"GB","licensed_sponsor":true,"adverts_offering_sponsorship_24m":0,"sanctioned_now":false,"evidence":"Licensed UK visa sponsor (Senior or Specialist Worker, Skilled Worker)","filings_12m":null,"filings_prev_12m":null,"green_card_filings_12m":null,"median_offered_wage_usd":null,"route":"Global Business Mobility: Senior or Specialist Worker; Skilled Worker","cap_exempt":false,"checked_at":"2026-11-19T08:39:47+00:00","sources":["UK Home Office: register of licensed sponsors (workers)"],"filings_for_role_12m":0}],"liveness":{"score":86,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.86,"p_room":1,"age_days":0,"expected_fill_days":85,"reasons":["conf:3","win:early"],"computed_at":"2026-11-19T19:35:26Z"},"pay":null,"html_url":"https://alion.io/job/the-trade-desk-software-engineer-ii-dataproc","json_url":"https://alion.io/job/the-trade-desk-software-engineer-ii-dataproc.json","meta":{"generated_at":"2026-11-19T19:35:26Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":1987,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-20T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":684619},"rest":"https://alion.io/mcp/rest/get_company?id=684619"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fthe-trade-desk-software-engineer-ii-dataproc"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"free","price":{"credits":0,"usd":0},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fthe-trade-desk-software-engineer-ii-dataproc"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fthe-trade-desk-software-engineer-ii-dataproc"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/the-trade-desk-software-engineer-ii-dataproc\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fthe-trade-desk-software-engineer-ii-dataproc"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":4615567}},{"tool":"search_jobs","arguments":{"company":"thetradedesk.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":4615567},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}