Data Acquisition Engineer

The role involves building and scaling automated web crawlers and data extraction systems using Python and Scrapy. You'll also be working on anti-bot and WAF challenges, optimising proxies and crawler configurations, and building distributed systems using tools like Redis, Kafka, and Celery. Additionally, you will work on mobile API analysis, including understanding how iOS and Android applications communicate with backend systems and extracting data through APIs.

Responsibilities:

• Scale and Architecture: Architect, build, and optimise high-throughput crawler networks using Python and the Scrapy framework.
• Anti-Bot and WAF Defence: Develop robust, automated mechanisms to bypass advanced anti-bot protections, rate limits, enterprise WAFs (e. g., Akamai Bot Manager, Cloudflare, Datadome), and complex CAPTCHA challenges.
• Infrastructure Optimisation: Implement proxy rotation, custom Scrapy middleware, request fingerprinting, header emulation, and TLS/JA3 fingerprint spoofing.
• Distributed Orchestration: Build distributed queuing and scraping architectures using Scrapy, Redis, Kafka, and Celery for seamless execution at scale.
• Mobile Reverse Engineering: Analyse mobile network traffic (iOS/Android), inspect API endpoints, perform certificate unpinning, and extract data directly from native mobile backend APIs.

Requirements:

• Scrapy Mastery: Deep internal knowledge of Scrapy (Twisted engine, asynchronous networking, custom pipelines, spider middleware, and extensions).
• Anti-Bot Infra Expertise: Proven track record bypassing enterprise anti-scraping systems (Akamai Bot Detection, Cloudflare, Imperva, Datadome, PerimeterX).
• Network and Protocol Mastery: Deep understanding of HTTP/HTTPS, HTTP/2, WebSocket protocols, TLS signatures/fingerprinting (JA3/JA4), request headers, and cookie management.
• Scale Experience: Direct experience running distributed crawling infrastructure handling millions of requests daily.

Preferred Qualifications:

• Mobile Application Reverse Engineering: Experience with tools like Frida, Charles Proxy, Mitmproxy, or Burp Suite to intercept, analyse, and bypass mobile SSL pinners.
• Device Farm Management: Hands-on experience working with mobile device farms (Real device or Emulated Android/iOS farms) for app automation.
• Browser Automation and Bypassing: Experience integrating Scrapy with undetected headless engines (Playwright, Selenium-Stealth, Puppeteer-Extra).
• Data Pipelines and Cloud Infrastructure: Familiarity with AWS/GCP, Docker, Kubernetes, and streaming message queues (Kafka, Redis, RabbitMQ).