Data & Crawler Engineer
First seen by Alion on Sep 11, 2026.
Company Overview :
Kaam Job Career Pvt Ltd is a dynamic recruitment and talent solutions firm based in Mumbai, dedicated to bridging the gap between high-caliber professionals and industry-leading organizations. We operate at the intersection of technology and human capital, leveraging data-driven insights to streamline hiring processes across diverse sectors. Our culture is built on efficiency, precision, and a commitment to empowering businesses with the right technical expertise to scale their operations effectively.
Role Overview :
As a Data and Crawler Engineer, you will be the architect of our data acquisition infrastructure, responsible for building robust systems that harvest and process vast amounts of information from the web. You will work closely with our engineering and product teams to transform raw, unstructured web data into clean, actionable insights that drive our internal decision-making and client-facing platforms. Your daily contributions will directly influence the quality of our data products, ensuring that our pipelines remain resilient, scalable, and capable of handling complex extraction challenges in a fast-paced environment.
Key Responsibilities :
- Design and maintain high-performance web crawling architectures to extract structured data from complex, dynamic websites at scale.
- Develop and optimize end-to-end ETL pipelines to ensure seamless data flow from source to storage, maintaining high standards of data integrity.
- Implement advanced anti-blocking strategies and proxy management techniques to ensure uninterrupted data collection across diverse web environments.
- Collaborate with cross-functional stakeholders to define data requirements and translate them into efficient data models that support business objectives.
- Monitor and troubleshoot data pipelines to proactively identify and resolve bottlenecks, ensuring high availability and performance of our big data systems.
- Automate data extraction workflows to reduce manual intervention and improve the overall efficiency of our data acquisition lifecycle.
Required Skillset :
- Demonstrated expertise in building scalable web crawlers using Python, Scrapy, Selenium, and Puppeteer to handle large-scale data extraction tasks.
- Strong proficiency in data engineering principles, including the design and management of SQL and NoSQL databases to store and query complex datasets.
- Proven ability to architect and maintain robust data pipelines and ETL processes that handle high-volume data ingestion with minimal latency.
- Deep understanding of data modeling techniques to organize and structure unstructured web data for downstream analytical consumption.
- Ability to work effectively in an on-site environment in Mumbai, collaborating with technical teams to solve complex data challenges.
- Excellent communication skills to articulate technical concepts to non-technical stakeholders and work effectively within a team-oriented culture.
- A minimum of 5 - 10 years of experience in data engineering or crawler development, with a track record of delivering production-grade data solutions.
Skills
Python, Scrapy, Puppeteer, ETL, Web Scraping, Data Mining, SQL, NoSQL, Data Engineering