Python Developer - Web Scraping
First seen by Alion on Sep 9, 2026.
Key Responsibilities :
- Develop and maintain robust web scraping and web crawling solutions using Python.
- Build data extraction pipelines to collect, clean, transform, and store information from multiple web sources.
- Develop and optimize scraping solutions for both static and dynamic websites.
- Work with frameworks and libraries such as Requests, Scrapy, BeautifulSoup, Selenium, Puppeteer, or Playwright.
- Design efficient solutions for handling large-scale data extraction and crawling requirements.
- Integrate scraped data with SQL databases and data warehousing platforms.
- Perform data validation, cleansing, and quality checks to ensure accuracy and consistency.
- Troubleshoot scraping failures, website changes, anti-bot challenges, and performance issues.
- Optimize crawlers for scalability, reliability, and execution speed.
- Collaborate with data, engineering, and business teams to understand data requirements and deliver effective solutions.
Required Skills :
- 5+ years of experience in Python development with strong hands-on programming expertise.
- Proven experience in Web Scraping / Web Crawling.
- Strong knowledge of Python programming and scripting.
- Hands-on experience with SQL and any database or data warehousing technology.
- Experience with one or more of: Requests, Scrapy.
- Experience with one or more of: Puppeteer, Playwright, BeautifulSoup, Selenium.
- Knowledge of PySpark is preferred.
- Strong understanding of HTML, CSS, DOM, HTTP requests, and web technologies.
- Good debugging, analytical, and problem-solving skills.
Preferred Candidate Profile :
- Strong hands-on experience in building production-grade web scraping and crawling solutions.
- Experience working with large datasets and high-volume data extraction.
- Ability to work independently and handle complex scraping requirements.
- Strong understanding of data processing, database integration, and automation.
Skills
Python, Web Scraping, Scrapy, SQL, Backend Architecture, Web Crawling, Data Pipeline, Data Integration