2,461,267open jobs
149,950companies
317,727added this week
Browse all
Location
In office (Hyderabad)
Visa
Sponsors visas in United States; no register of this country names it
Employment
Full-Time

First seen by Alion on Nov 18, 2026.

Overview
Company
Impact
Profile match
Cybage Software provides technology consulting and product engineering services. Its delivery model applies data-driven process analytics to project execution. The company serves clients in media, travel, healthcare and retail technology.

Job Description

Must Have :

● Strong Python experience required.

● Experience building and operating web scraping (with Scrapy) and data engineering systems in

production is a plus - Preferable

● Demonstrated debugging strength: a methodical approach to reproducing, isolating, and root-causing

failures in code you did not write, against sources you do not control.

● Chrome DevTools used as a reverse-engineering tool: finding the API behind the page, replaying requests,

and understanding network and auth flows - not just inspecting elements.

● Hands-on MongoDB: queries, aggregation, indexes, and ETL on large document collections.

● Strong understanding of ETL and data engineering concepts, including extraction, transformation,

validation, deduplication, incremental processing, backfills, and data-quality checks.

● Production crawl operations: scheduling, retries, rate limiting, logging, and recovery when a source breaks

overnight.

● Multi-format extraction, particularly PDF, where the structure is never where you want it.

● Git and collaborative development workflows, including code review.

Good to Have :

● Browser automation for JS-rendered sources (Playwright, Selenium).

● Crawl orchestration: ScrapyKeeper, Scrapyd, or Airflow.

● Hierarchical or citation-style document modelling.

● Proxy and anti-bot handling within legal and contractual limits.

Role in brief: A hands-on Scrapy and data engineer who builds, runs and debugs production pipelines that ingest global regulatory content into MongoDB, working closely with a legal research team on data quality.

Build

  • Design and own Scrapy spiders and ETL pipelines for global regulatory sources.
  • Extract and normalize HTML, JSON/XML APIs, PDFs and RTF/DOC into versioned datasets, keeping document relationships, metadata and stable identifiers.
  • Build item pipelines that persist to MongoDB and object storage.
  • Reverse engineer hard sources (hidden APIs, JavaScript flows, tokens, anti-bot measures) using Chrome DevTools and logs.

Operate

  • Run scheduled crawls through ScrapyKeeper/Scrapyd, and resolve stuck or failed jobs.
  • Improve reliability with incremental and full loads, output validation, and alerts that catch silent failures.
  • Use MongoDB queries to inspect, backfill and correct data safely.

Debug

  • Own triage when a source breaks: reproduce, compare against the live page, and find the root cause before changing code.
  • Tackle the hard bugs where a crawl succeeds but produces wrong content (truncated documents, duplicate or unstable IDs, missing versions).
  • Write short postmortems and turn each into a validation check, test or fix.

Improve speed

  • Build reusable base spiders and parsers so a new source takes days, not weeks.
  • Own metrics such as time-to-launch, failure rate and manual intervention hours.
  • Use AI-assisted development tools responsibly, and evaluate new techniques with measurable before-and-after results.

Collaborate

  • Work with the legal research team on schema, coverage and content quality, and treat their QA feedback as the main correctness signal.
  • Document spider designs, source quirks and runbooks, and help raise code review and testing standards.

Free account
Stop reading job ads. Get the ones that fit.
One free account turns this page into a shortlist built around your stack, your level and your pay.
Match on every job. Stack, seniority, pay and location, scored against your profile.
2,461,267 open roles. Read straight off company career pages, refreshed every day.
Unlimited applications. Every one you send is tracked in one place, on-site or on a company board.
3 tailored CVs a month. Rewritten for the exact job you are applying to. Included free.
Create a free account Continue with Google
Free forever. No card. Under a minute.

Your match

How well do you fit this role?
Two answers are enough for a real match. No account needed.
Check my fit
Answers stay in this browser until you create an account.

Recommended for you based on this role

Data Science
Similar stack
Same company
Hyderabad
In office · Full-Time · 15+ years exp · Bengaluru
Python
SQL
Python
pySpark
Databases
Databricks
Oracle
AI/ML
Spark
Apply
Data Engineer 2 days ago
In office · Full-Time · 3+ years exp · Chennai
Python
SQL
Analytics
ETL/ELT
Apply
Data Engineer 2 days ago
Hybrid · Full-Time · 5+ years exp · Mumbai
AI/ML
Hadoop
Analytics
ETL/ELT
Apply
Data Engineer 2 days ago
In office · Full-Time · 3+ years exp · Bengaluru
SQL
Databases
Teradata
Analytics
ETL/ELT
Apply
In office · 6+ years exp · Bengaluru
Python
JavaScript
TypeScript
SQL
Python
SQLAlchemy
FastAPI
Django
Databases
MySQL
PostgreSQL
AI/ML
Copilot
Cursor
LangGraph
LangChain
Claude
Embeddings
Prompt Engineering
AI Agents
RAG
OpenAI
Frontend
Redux
Tailwind CSS
Next.js
Bootstrap
React.js
React Query
Material UI
Ant Design
Mobile
State Management
DevOps
Rest API
Terraform
GCP
Azure DevOps
GitHub Actions
Prometheus
Azure
CI/CD
Jenkins
Git
AWS
Docker
Kubernetes
Grafana
Analytics
Tableau
Power BI
QA
Swagger
Jest
Pytest
Apply
Hybrid · Full-Time · 10+ years exp · Bachelor's Degree · Hyderabad
Python
SQL
Python
FastAPI
pySpark
Databases
PostgreSQL
Redis
Snowflake
Databricks
Oracle
Apache Kafka
Google BigQuery
Google Bigtable
AI/ML
Spark
Airflow
DevOps
GCP
Azure
CI/CD
AWS
Kubernetes
Google GKE
Google Cloud Run
GitHub
IAM
Cybersecurity
DLP
Analytics
ETL/ELT
Azure Data Factory
Apply
Hybrid · 8+ years exp · Hyderabad · Bengaluru · Chennai · Pune · Nagpur
SQL
Databases
Snowflake
AI/ML
dbt
DevOps
GCP
Azure
CI/CD
Git
AWS
Analytics
Informatica
Matillion
Fivetran
Azure Data Factory
Dimensional Modeling
Management
Agile
Apply
In office · Hyderabad
Analytics
Microsoft Excel
Apply
Team Lead 1 day ago
In office · 1+ year exp · Hyderabad
Apply
In office · Hyderabad
Apply
See all jobs
This is one of many
2,461,267 more open roles from verified company boards, updated every day.