{"id":3289617,"url":"https://alion.io/job/firmable-lead-data-engineer","title":"Lead Data Engineer","company":{"id":14688,"name":"Firmable","domain":"firmable.ai","url":"https://alion.io/company/firmable","size_band":"1001-5000","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Teamtailor","truth_index":null},"role":"Data Science","role_family":"Data Science","seniority":"lead","employment_type":"full_time","work_mode":"remote","remote_scope":"stated_countries","remote_scope_basis":"inferred_company_offices","remote_working_hours":null,"hiring_geo_confidence":"inferred","locations":[],"countries":[],"hiring_countries":["IN"],"hiring_countries_total":1,"salary":null,"salary_estimate":{"min_usd":114000,"max_usd":235000,"period":"year","method":"global_role_seniority_cell","sample_n":771},"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"AI Agents","optional":false},{"name":"Amazon ECS","optional":false},{"name":"Amazon Redshift","optional":false},{"name":"Amazon S3","optional":false},{"name":"AWS","optional":false},{"name":"AWS Lambda","optional":false},{"name":"Braintrust","optional":false},{"name":"Claude Code","optional":false},{"name":"Cursor","optional":false},{"name":"Embeddings","optional":false},{"name":"ETL/ELT","optional":false},{"name":"GDPR","optional":false},{"name":"LLM","optional":false},{"name":"LLM Evaluation","optional":false},{"name":"OpenTelemetry","optional":false},{"name":"Promptfoo","optional":false},{"name":"Python","optional":false},{"name":"Snowflake","optional":false},{"name":"SQL","optional":false},{"name":"Structured Outputs","optional":false}],"status":"live","first_seen_at":"2026-10-10T17:47:13Z","employer_posted_date":null,"last_verified_at":"2026-10-10T17:47:13Z","board_verified":false,"closed_at":null,"days_open":21,"trust":{"level":"not_scored","repost_count":null,"flags":[],"days_open":21},"description":"Firmable is the market-leading B2B sales intelligence platform in Asia-Pacific. Our competitive moat is our data: the deepest, most localised company and people dataset in every market we operate in. We've proven the model in ANZ. Now we're scaling it across Southeast Asia and the US.\nEvery record Firmable sells starts in a sourcing pipeline. This role architects that pipeline.\nThe RoleLead Data Engineer: Sourcing designs and owns the end-to-end extraction and ETL pipeline that turns unstructured web data into the world's most accurate B2B dataset across 13 markets. You set the architectural standards: extractor patterns, proxy strategy, LLM infrastructure as production systems, agentic escalation workflows, and cost-aware orchestration.\nThis is not a pipeline maintenance or incremental optimisation role. You're architecting the framework other engineers ship into, building for the hardest extraction problems (anti-bot defences, JS-heavy rendering, schema drift), and owning the sourcing layer end to end.\n~80% hands-on architecture and reference implementation: designing extractor frameworks, shipping the hardest extractors, building agentic pipelines, writing eval harnesses and labelled datasets. ~20% cross-functional voice: technical input on sourcing decisions with data platform, product, and analytics; setting standards for coverage, schema, and accuracy.\nWhat You'll OwnData Sourcing and ETL architecture\nEnd-to-end pipeline design: extraction, normalisation, deduplication, validation, and load; plus cost, performance, and reliability of the whole layer\n\nSourcing layer standards: coverage, accuracy, schema design; the rule-vs-LLM standard codified so the team applies it without you\n\nCost and performance optimisation: cost ceilings with the token math behind them, incremental processing, recovery design, scheduling\n\nExtraction systems and frameworks\nExtractor framework: patterns, abstractions, and tooling other engineers ship into; new extractors are fast to build and reliable to run\n\nHard-source extractors: anti-bot defences, JS-heavy rendering, schema drift, low-quality structure; plus the proxy and IP rotation strategy behind them\n\nAgentic extraction pipelines: rule-based triage, LLM escalation, structured-output validation, retries, human-review queues\n\nLLM infrastructure and observability\nLLMs as production systems: versioned prompts, labelled eval sets, measured precision and recall, prompt versioning you can roll back, judges debugged on real data\n\nEval and observability scaffolding: eval frameworks, prompt versioning, traces, drift detection when a vendor silently updates a model\n\nModel-choice playbook: cheap models for classification, stronger models for nuanced extraction, frontier models for hard edge cases; revised as model economics shift\n\nSkills library and orchestration\nSkills library: versioned specs for recurring extraction patterns, invocable by engineers and agents alike\n\nOrchestration: Airflow or equivalent patterns that scale with data volumes and source counts; dependency management, recovery, cost-aware scheduling\n\nWhat We're Looking ForMust Haves\n6-10 years shipping production extraction, ETL, or data pipeline systems in business-critical environments; deep web extraction at scale with experience designing around anti-bot defences, proxy architecture, JS rendering, schema drift, and recovery\n\nShipped LLMs inside extraction pipelines as production systems: structured outputs, versioned prompts, labelled eval sets, logged traces, judges debugged on precision/recall, drift detected on vendor model updates. You can show the repo.\n\nBuilt agents and tool-calling pipelines: architected agent workflows, written specs others depend on, run tool-calling systems in production\n\nSharp judgement on rules vs. LLMs: reach for deterministic logic first, defend the call either way, and have codified the standard for a team\n\nExpert Python: production-grade, performance-aware, comfortable with concurrency and scale; plus advanced SQL for complex transformations and performance optimisation\n\nExtensive Airflow or equivalent: orchestration, dependency management, recovery patterns, cost optimisation at production scale\n\nShipped with agentic IDEs: Claude Code, Cursor, or equivalent as your default mode, with shipped extraction systems to show for it\n\nArchitecture judgement and a product mindset: when to refactor, optimise, ship, or start again; you care about coverage and accuracy landing with customers, not uptime metrics\n\nYou live and breathe AI tools. Structured outputs, evals, traces, and LLM tracing (Logfire, OpenTelemetry, similar) as your default way of working, not a productivity experiment. In 2026, this is how data engineers build extraction at scale and you need to already be doing it.\n\nHighly Valued\nCloud data platforms: Snowflake, Redshift; AWS for pipeline deployment (Lambda, S3, ECS, Glue)\n\nVector databases, embeddings, or retrieval patterns for matching and deduplication\n\nEval frameworks like Braintrust, Promptfoo, or Inspect; LLM tracing tooling\n\nData quality frameworks with automated testing and anomaly detection at scale\n\nB2B data: firmographics, people data, company registries across markets\n\nData privacy and compliance: GDPR, CCPA\n\nStartup or scaleup experience where you shipped fast and owned outcomes end to end\n\nThe EnvironmentFirmable runs lean and ships fast: small senior teams, no layers, minimal process. Sourcing is a core competitive advantage; this role sets the pace for how fast and accurate the entire data pipeline moves. Weekly releases moving toward daily; no fixed hours, full autonomy on architecture choices.\nWe are an AI-native organisation. That means AI isn't a tool we reach for: it's the default operating mode. Agentic extraction pipelines, LLM-powered classification with eval sets, structured-output validation, labelled datasets versioned with prompt runs, traces logged from day one with cost and latency baked in. If you're not already working this way, this role isn't right for you.\nWhy This Role\nOwn the sourcing engine: every record Firmable sells starts in the pipeline you architect; your design shapes the accuracy and cost of every customer interaction\n\nGreenfield AI-native architecture: the extractor framework, skills library, eval harnesses, agentic orchestration, and production LLM infrastructure are largely unbuilt; you'll architect them from first principles\n\nFrontier technical problems: agentic extraction at production scale, LLM-as-extractor with full eval coverage, vendor model drift detection, rule-vs-LLM orchestration across 13 markets\n\nSmall team, massive leverage: your architecture reaches every Firmable customer, every day; you write the reference implementation daily and own outcomes end to end\n\nReal career runway: this role is a bottleneck role; progress is measured by pipeline speed and accuracy, not tenure\n\nWho This Role is NOT For\nAnyone looking to optimise an existing pipeline rather than architect from first principles\n\nAnyone who treats infrastructure as someone else's problem; sourcing architecture lives in your code every day\n\nAnyone who sees LLMs as a parsing shortcut rather than production systems that need eval sets, versioning, and drift detection\n\nAnyone uncomfortable shipping with minimal process or without an established playbook to follow\n\nAnyone who treats AI as something they'll learn on the job rather than already use daily in infrastructure work\n\nAnyone coming from a pure ops or data analytics background without shipped systems engineering at scale\n\nOriginally posted on Himalayas","description_format":"text","description_chars":7601,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[{"name":"India","iso":"IN","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Sales Enablement","Lead Generation","Marketing & Sales AI"],"lifecycle":[{"event":"open","at":"2026-10-26T19:11:04Z"}],"visa":[],"liveness":{"score":53,"band":"ok","label":"Likely open","p_open":0.85,"p_active":0.839,"p_room":0.75,"age_days":20,"expected_fill_days":29,"reasons":["seen:20","velocity","win:late"],"computed_at":"2026-10-31T05:58:55Z"},"pay":null,"html_url":"https://alion.io/job/firmable-lead-data-engineer","json_url":"https://alion.io/job/firmable-lead-data-engineer.json","meta":{"generated_at":"2026-11-01T01:48:52Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":2669,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-02T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":14688},"rest":"https://alion.io/mcp/rest/get_company?id=14688"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Ffirmable-lead-data-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Ffirmable-lead-data-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Ffirmable-lead-data-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/firmable-lead-data-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Ffirmable-lead-data-engineer"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":3289617}},{"tool":"search_jobs","arguments":{"company":"firmable.ai","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":3289617},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}