{"id":3308850,"url":"https://alion.io/job/factorin-senior-data-engineer","title":"Senior Data Engineer","company":{"id":3818111,"name":"Factorin","domain":"factorin.ru","url":"https://alion.io/company/faktorin","size_band":"11-50","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":null,"truth_index":null},"role":"Data Science","role_family":"Data Science","seniority":"senior","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Bengaluru, India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":21000,"max_usd":47000,"period":"year","method":"role_seniority_country_remote_cell","sample_n":75},"experience_years_min":6,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Airflow","optional":false},{"name":"Apache Iceberg","optional":false},{"name":"Apache Kafka","optional":false},{"name":"CI/CD","optional":false},{"name":"Delta Lake","optional":false},{"name":"FinOps","optional":false},{"name":"GCP","optional":false},{"name":"Google BigQuery","optional":false},{"name":"Google Bigtable","optional":false},{"name":"Google GKE","optional":false},{"name":"Java","optional":false},{"name":"Kubernetes","optional":false},{"name":"LLM","optional":false},{"name":"pySpark","optional":false},{"name":"Scala","optional":false},{"name":"Shuffle","optional":false},{"name":"Spark","optional":false},{"name":"SQL","optional":false},{"name":"Terraform","optional":false},{"name":"Apache Hudi","optional":true},{"name":"Claude","optional":true},{"name":"ClickHouse","optional":true},{"name":"dbt","optional":true},{"name":"Docker","optional":true},{"name":"ElasticSearch","optional":true},{"name":"Flink","optional":true},{"name":"Gemini","optional":true},{"name":"LangChain","optional":true},{"name":"OpenAI","optional":true},{"name":"PostgreSQL","optional":true},{"name":"Presto","optional":true},{"name":"Python","optional":true},{"name":"RAG","optional":true},{"name":"Trino","optional":true},{"name":"Vertex AI","optional":true}],"status":"live","first_seen_at":"2026-10-27T00:58:14Z","employer_posted_date":null,"last_verified_at":"2026-10-27T00:58:14Z","board_verified":false,"closed_at":null,"days_open":7,"trust":{"level":"not_scored","repost_count":null,"flags":[],"days_open":7},"description":"About us\nFactori is a rapidly growing location and consumer intelligence company built for an AI-first world. Each day we ingest billions of privacy-compliant signals and maintain the world's most comprehensive consumer, mobility and point-of-Interest database.\n\nOur AI-driven insights help Fortune 500 brands, global media agencies and technology innovators decide where to invest, build and grow. Headquartered in New York City with hubs across APAC, EMEA and the Americas, we operate with the ownership mindset of a startup and the global footprint of an enterprise data leader.\n\nAbout the Role:\nWe are hiring a Senior Data Engineer to own and scale a petabyte-scale data platform on Google Cloud Platform (GCP), processing TB scale of data every day with Apache Spark. You will lead a small team of data engineers, drive cloud cost optimisation across our data workloads, and build AI-driven workflows that change how we support and operate the platform.\n\nThis is a product engineering role, not a services or consulting engagement. You will own systems end to end, live with your design decisions in production, and help define data engineering culture in India.\n\nKey Responsibilities:\nBuild and scale the platform\nDesign, build and operate highly available, fault-tolerant batch and streaming pipelines on GCP that ingest hundreds of terabytes a day into a petabyte-scale data lake and warehouse.\nWrite and tune large Spark workloads (PySpark, Spark SQL, Scala/Java) on Dataproc or Spark on Kubernetes (GKE): partitioning, shuffle and join optimization, skew handling, memory and executor sizing.\nBuild real-time and near-real-time pipelines with Pub/Sub or Kafka and Spark Structured Streaming or Dataflow (Apache Beam), with clear guarantees on latency, ordering and exactly-once or idempotent processing.\nModel data in BigQuery and the lakehouse (Parquet, Avro, Delta Lake or Apache Iceberg) into clean, simplified datasets that teams can self-serve.\n\nOwn data quality\nBuild data cleaning, validation and reconciliation pipelines that stay correct when source data is late, duplicated, malformed or missing.\nPut data contracts, schema evolution rules, quality checks and alerting in place, and own data mapping, business logic and transformations end to end.\n\nLead cloud cost optimization\nOwn the cost of our GCP data workloads: measure cost per pipeline and per customer, then reduce it through right-sizing, autoscaling, preemptible/Spot VMs, storage tiering, BigQuery slot and query optimization, and job consolidation.\nSet up cost visibility (billing export, labels, dashboards) and FinOps practices so cost is tracked and owned, not discovered after the fact.\n\nLead and mentor\nLead and mentor 2–4 junior and mid-level data engineers: plan and break down work, review code and designs, and raise the team's engineering bar.\nLead system design and architecture discussions, write design docs, and make trade-offs on scalability, reliability, latency and cost.\n\nInnovate with AI\nDesign AI-driven workflows for technical support and operations: LLM-assisted triage of pipeline failures and data incidents, automated root-cause analysis, runbook automation, and agents that answer data questions from internal teams.\nTurn repeated support work into automated, process-driven workflows, and measure the time and tickets they save\n\nRun it in production\nOwn production reliability: on-call, monitoring and observability, incident response, postmortems, and low-level debugging and performance tuning on large clusters.\nOrchestrate pipelines with Airflow / Cloud Composer and ship through CI/CD and infrastructure as code (Terraform).\n\nWhat we are looking for :\nExperience:\n6 plus years in data engineering or backend/software engineering, with at least 4 years building big data pipelines in production.\nMinimum 1–2 years as a tech lead or team lead, mentoring junior engineers and owning delivery for a team.\nProduct background:\nMost of your experience at product or SaaS companies, owning platforms you built and ran in production, rather than consulting or IT services delivery.\nApache Spark at scale:\nDeep hands-on PySpark and Spark SQL; Spark internals (DAG, stages, shuffle, catalyst, AQE), performance tuning, and handling data skew on multi-terabyte jobs. Scala or Java Spark is a strong plus.\nGoogle Cloud Platform:\nProduction experience with BigQuery, Dataproc, Cloud Storage (GCS), Pub/Sub, and Cloud Composer or Airflow. Dataflow, GKE and Bigtable are a plus.\nBatch and stream processing:\nDesigned and run both, with a solid grasp of windowing, watermarks, late data, checkpointing, backfills and idempotency. Kafka or Pub/Sub plus Spark Structured Streaming, Flink or Dataflow.\nSQL:\nExpert-level, including window functions, complex joins and query optimization on large analytical warehouses.\nProgramming and OOP:\nStrong Python and Java with object-oriented design, design patterns and clean, testable, reusable code.\nData quality:\nBuilt data cleaning, validation, deduplication and reconciliation pipelines, with automated quality checks and alerting.\nSystem design:\nCan design a scalable, fault-tolerant distributed data system and defend the trade-offs in consistency, latency, cost and failure handling.\nCloud cost optimization:\nA track record of measurably reducing cloud or data platform costs (state the numbers on your CV).\nDatabases and storage:\nParquet, Avro and ORC; partitioning and clustering; production experience with PostgreSQL, Elasticsearch and Bigtable (or comparable relational, search and NoSQL stores).\n\nGood to have :\nGoogle Cloud Professional Data Engineer or Professional Cloud Architect certification.\nLakehouse table formats: Delta Lake, Apache Iceberg or Apache Hudi. dbt, Trino/Presto or ClickHouse.\nTerraform, Docker and Kubernetes (GKE) for data infrastructure.\nData observability and lineage tools such as Monte Carlo, OpenLineage or DataHub.\nBuilding with LLM APIs, RAG or agent frameworks (Vertex AI, Gemini, OpenAI, Claude, LangChain).\nDomain experience in ad tech, martech, marketing attribution or measurement.\n\nWho thrives here :\nYou own outcomes. When a pipeline breaks at 2 a.m. or a number looks wrong, you treat it as yours until it is fixed and the root cause is gone.\nYou are accountable. You make commitments you can keep, flag risks early, and close the loop without being chased.\nYou think like a product engineer. You care about the customer impact of data quality, latency and cost, not just whether the job finished.\nYou raise the people around you. You give clear reviews, share context, and help junior engineers grow into owners.\nYou simplify. You prefer fewer, well-understood systems and automate repeated work, increasingly with AI.\n\nHow to apply :\nIf this sounds like you, apply now at [HIDDEN TEXT]. We'd love to hear from you! For more information visit - https://factori.ai/","description_format":"text","description_chars":6855,"description_truncated":false,"requirements":{"experience_years_min":6,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Factoring & Trade Finance"],"lifecycle":[{"event":"open","at":"2026-10-27T02:09:45Z"}],"visa":[],"liveness":{"score":86,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.86,"p_room":1,"age_days":7,"expected_fill_days":30,"reasons":["seen:7","win:early"],"computed_at":"2026-11-03T05:54:40Z"},"pay":null,"html_url":"https://alion.io/job/factorin-senior-data-engineer","json_url":"https://alion.io/job/factorin-senior-data-engineer.json","meta":{"generated_at":"2026-11-03T12:02:20Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":16576,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-04T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":3818111},"rest":"https://alion.io/mcp/rest/get_company?id=3818111"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Ffactorin-senior-data-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Ffactorin-senior-data-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Ffactorin-senior-data-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/factorin-senior-data-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Ffactorin-senior-data-engineer"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":3308850}},{"tool":"search_jobs","arguments":{"company":"factorin.ru","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":3308850},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}