{"id":4548851,"url":"https://alion.io/job/cybage-software-data-engineer","title":"Data Engineer","company":{"id":59343,"name":"Cybage Software","domain":"cybage.com","url":"https://alion.io/company/cybage-software","size_band":"51-200","is_staffing_agency":false,"employer_type":"services","is_intermediary":false,"listed_via":null,"ats_vendor":null,"truth_index":null},"role":"Data Science","role_family":"Data Science","seniority":null,"employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Hyderabad, India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Python","optional":false},{"name":"Scrapy","optional":false},{"name":"Chrome DevTools","optional":true},{"name":"ETL/ELT","optional":true},{"name":"Git","optional":true},{"name":"JavaScript","optional":true},{"name":"Playwright","optional":true},{"name":"Selenium","optional":true}],"status":"live","first_seen_at":"2026-11-18T17:23:52Z","employer_posted_date":null,"last_verified_at":"2026-11-18T17:23:52Z","board_verified":false,"closed_at":null,"days_open":3,"trust":{"level":"not_scored","repost_count":null,"flags":[],"days_open":3},"description":"Job Description \n\nMust Have :\n● Strong Python experience required.\n● Experience building and operating web scraping (with Scrapy) and data engineering systems in\nproduction is a plus - Preferable\n● Demonstrated debugging strength: a methodical approach to reproducing, isolating, and root-causing\nfailures in code you did not write, against sources you do not control.\n● Chrome DevTools used as a reverse-engineering tool: finding the API behind the page, replaying requests,\nand understanding network and auth flows - not just inspecting elements.\n● Hands-on MongoDB: queries, aggregation, indexes, and ETL on large document collections.\n● Strong understanding of ETL and data engineering concepts, including extraction, transformation,\nvalidation, deduplication, incremental processing, backfills, and data-quality checks.\n● Production crawl operations: scheduling, retries, rate limiting, logging, and recovery when a source breaks\novernight.\n● Multi-format extraction, particularly PDF, where the structure is never where you want it.\n● Git and collaborative development workflows, including code review.\n Good to Have :\n● Browser automation for JS-rendered sources (Playwright, Selenium).\n● Crawl orchestration: ScrapyKeeper, Scrapyd, or Airflow.\n● Hierarchical or citation-style document modelling.\n● Proxy and anti-bot handling within legal and contractual limits.\n\nRole in brief: A hands-on Scrapy and data engineer who builds, runs and debugs production pipelines that ingest global regulatory content into MongoDB, working closely with a legal research team on data quality.\nBuild\nDesign and own Scrapy spiders and ETL pipelines for global regulatory sources.\nExtract and normalize HTML, JSON/XML APIs, PDFs and RTF/DOC into versioned datasets, keeping document relationships, metadata and stable identifiers.\nBuild item pipelines that persist to MongoDB and object storage.\nReverse engineer hard sources (hidden APIs, JavaScript flows, tokens, anti-bot measures) using Chrome DevTools and logs.\nOperate\nRun scheduled crawls through ScrapyKeeper/Scrapyd, and resolve stuck or failed jobs.\nImprove reliability with incremental and full loads, output validation, and alerts that catch silent failures.\nUse MongoDB queries to inspect, backfill and correct data safely.\nDebug\nOwn triage when a source breaks: reproduce, compare against the live page, and find the root cause before changing code.\nTackle the hard bugs where a crawl succeeds but produces wrong content (truncated documents, duplicate or unstable IDs, missing versions).\nWrite short postmortems and turn each into a validation check, test or fix.\nImprove speed\nBuild reusable base spiders and parsers so a new source takes days, not weeks.\nOwn metrics such as time-to-launch, failure rate and manual intervention hours.\nUse AI-assisted development tools responsibly, and evaluate new techniques with measurable before-and-after results.\nCollaborate\nWork with the legal research team on schema, coverage and content quality, and treat their QA feedback as the main correctness signal.\nDocument spider designs, source quirks and runbooks, and help raise code review and testing standards.","description_format":"text","description_chars":3157,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["IT Consulting & Digital Transformation","IT Outsourcing & Dedicated Teams"],"lifecycle":[{"event":"open","at":"2026-11-18T17:59:00Z"}],"visa":[],"liveness":{"score":90,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.903,"p_room":1,"age_days":1,"expected_fill_days":27,"reasons":["seen:1","velocity","win:early"],"computed_at":"2026-11-20T05:46:30Z"},"pay":null,"html_url":"https://alion.io/job/cybage-software-data-engineer","json_url":"https://alion.io/job/cybage-software-data-engineer.json","meta":{"generated_at":"2026-11-22T08:59:21Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":14439,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-23T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":59343},"rest":"https://alion.io/mcp/rest/get_company?id=59343"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fcybage-software-data-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"free","price":{"credits":0,"usd":0},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fcybage-software-data-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fcybage-software-data-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/cybage-software-data-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fcybage-software-data-engineer"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":4548851}},{"tool":"search_jobs","arguments":{"company":"cybage.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":4548851},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}