Data Engineer (Web Scraping/Crawling)

OSI Digital • IT Consulting & Digital Transformation • Enterprise Apps Implementation Partners • Python • SQL • ETL/ELT • 2 days ago
Smart Apply
Similar jobs
1,970,010open jobs
116,594companies
254,125added this week
Browse all
Salary
≈ $14k – $36k per year (Estimated)
Location
In office (Hyderabad)
Experience
3+ years exp
Employment
Full-Time

First seen by Alion on Oct 23, 2026.

Overview
Company
Impact
Profile match

OSI Digital

Founded in 1993, OSI Digital is a global IT consulting and technology firm specializing in digital transformation, cloud integration, and enterprise software engineering. Headquartered in Calabasas, California, United States, the company maintains extensive offshore delivery centers and strategic regional offices across North America, India, and the Philippines. Through its comprehensive suite of data analytics, application management, and cybersecurity services, it enables enterprise clients across diverse industries to optimize performance and drive business growth.
osidigital.com • HQ: Calabasas, United States • Analytics & BI Consulting • Enterprise Apps Implementation Partners • IT Consulting & Digital Transformation • 51-200 employees • Est. 1993
Tech stack of OSI Digital
Job Title: Web Crawling Engineer

Experience:3-5 Years

Location: Hyderabad

We are looking for a skilled Web Crawling Engineer with strong expertise in Python and hands-on experience building scalable crawlers and scrapers. The ideal candidate should have deep knowledge of crawling frameworks, anti-blocking techniques, and proxy management, along with experience solving challenges like captchas and rate-limiting.

Key Responsibilities

  • Design, develop, and maintain scalable and efficient web crawlers for data extraction.
  • Work with Scrapy, Requests, and other Python libraries to implement crawling solutions.
  • Handle anti-scraping measures, including IP blocks, rate limits, and captchas.
  • Implement and manage proxy rotation and session management strategies.
  • Ensure high-quality and structured data extraction, cleaning, and storage.
  • Monitor crawler performance, troubleshoot issues, and optimize for reliability and efficiency.
  • Collaborate with the data engineering team to integrate crawled data into pipelines.
  • Stay updated with the latest tools, techniques, and best practices in web crawling.

Required Skills & Qualifications

  • Strong programming skills in Python.
  • Proven experience with Scrapy, Requests, BeautifulSoup, Selenium (if required).
  • Hands-on experience solving blocking issues, handling
  • Good understanding of proxy usage, rotation, and fingerprinting avoidance techniques.
  • Knowledge of HTTP, cookies, headers, sessions, and request/response cycles.
  • Experience with databases (SQL/NoSQL) for storing crawled data.
  • Strong debugging, problem-solving, and analytical skills.

Good to Have

  • Experience with cloud environments (AWS, GCP, Azure).
  • Knowledge of data pipelines and ETL processes.

Education & Experience

  • Bachelor's degree in Computer Science, Information Technology, or equivalent practical experience.
  • 3+ years of experience in web crawling or related fields
Free account
Stop reading job ads. Get the ones that fit.
One free account turns this page into a shortlist built around your stack, your level and your pay.
Match on every job. Stack, seniority, pay and location, scored against your profile.
1,970,010 open roles. Read straight off company career pages, refreshed every day.
Unlimited applications. Every one you send is tracked in one place, on-site or on a company board.
3 tailored CVs a month. Rewritten for the exact job you are applying to. Included free.
Create a free account Continue with Google
Free forever. No card. Under a minute.

Your match

How well do you fit this role?
Two answers are enough for a real match. No account needed.
Check my fit
Answers stay in this browser until you create an account.

Recommended for you based on this role

Data Science
Similar stack
Same company
Hyderabad
≈ $20k – $41k per year (Estimated) • Remote (India) • Full-Time • 5+ years exp • Bachelor's Degree • Hyderabad
SQL
Databases
Databricks
Azure SQL Database
AI/ML
Copilot
DevOps
Azure DevOps
Azure
Git
Analytics
ETL/ELT
Azure Data Factory
Dimensional Modeling
Management
Agile
UML
BPMN
Apply
≈ $18k – $36k per year (Estimated) • In office • Full-Time • 5+ years exp • Bachelor's Degree • Bengaluru
Python
Databases
Neo4j
AI/ML
DGL
Fine-tuning
Embeddings
Scikit-learn
AI Agents
NLP
TensorFlow
PyTorch
LLM
RAG
Anomaly Detection
Semantic Search
Knowledge Graph
Machine Learning
DevOps
GCP
Azure
AWS
Platform Engineering
Apply
Data Engineer III 2 months ago
≈ $13k – $32k per year (Estimated) • In office • Full-Time • 3+ years exp • Bachelor's Degree • Chennai
Python
SQL
Scala
Databases
Databricks
DynamoDB
Apache Kafka
Amazon Redshift
Amazon Aurora
AI/ML
Hadoop
Spark
DevOps
Terraform
CloudFormation
CI/CD
Git
AWS
AWS Lambda
Amazon EC2
Amazon S3
IAM
Amazon CloudWatch
Amazon Kinesis
AWS Step Functions
Cybersecurity
GDPR
Analytics
ETL/ELT
AWS Glue
Management
Agile
Apply
CPW Data Steward I 3 days ago
≈ $12k – $28k per year (Estimated) • Hybrid • Full-Time • 2+ years exp • Bachelor's Degree • Mumbai
SQL
DevOps
Azure
Analytics
Microsoft Excel
Apply
$6.3k – $17k per year (gross) • In office • Full-Time • 5+ years exp • Gurgaon
Python
SQL
Python
pySpark
Databases
PostgreSQL
Apache Iceberg
Delta Lake
Apache Kafka
Amazon Redshift
Apache Hudi
AI/ML
Spark
Airflow
dbt
Frontend
GraphQL
DevOps
AWS
AWS Lambda
Amazon S3
IAM
Apache HTTP Server
Analytics
ETL/ELT
AWS Glue
Dimensional Modeling
Apply
≈ $15k – $38k per year (Estimated) • In office • 4+ years exp • Bengaluru • Gurgaon • Hyderabad
Databases
Oracle
DevOps
Incident Management
IAM
Apply
≈ $20k – $42k per year (Estimated) • In office • 5+ years exp • Hyderabad
Python
SQL
Python
pySpark
Databases
Databricks
AI/ML
Spark
DevOps
Azure DevOps
Azure
CI/CD
Apply
≈ $23k – $46k per year (Estimated) • In office • 8+ years exp • Bengaluru • Chennai • Hyderabad
Python
Java
SQL
Scala
Python
pySpark
Databases
Databricks
Azure Cosmos DB
Azure SQL Database
AI/ML
Spark
Machine Learning
DevOps
Terraform
Azure DevOps
Azure
CI/CD
Git
Docker
Kubernetes
Analytics
ETL/ELT
Azure Data Factory
Dimensional Modeling
Apply
≈ $28k – $61k per year (Estimated) • In office • 9+ years exp • Hyderabad
Python
C++
DevOps
Linux
Apply
≈ $13k – $30k per year (Estimated) • In office • 3+ years exp • Hyderabad
DevOps
Windows
TCP/IP
VPN
Cybersecurity
Zscaler
Management
ServiceNow
Apply
See all jobs
This is one of many
1,970,010 more open roles from verified company boards, updated every day.
Browse all jobs Get matched roles