{"id":3418963,"url":"https://alion.io/job/forage-senior-software-engineer-2","title":"Senior Software Engineer","company":{"id":26589,"name":"Forage","domain":"forage.ai","url":"https://alion.io/company/forage-ai","size_band":null,"is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":null,"truth_index":null},"role":"Backend","role_family":"Backend","seniority":"senior","employment_type":"full_time","work_mode":"hybrid","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":16000,"max_usd":38000,"period":"year","method":"role_seniority_country_remote_cell","sample_n":168},"experience_years_min":4,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"AI Agents","optional":false},{"name":"Amazon CloudWatch","optional":false},{"name":"Amazon ECS","optional":false},{"name":"Amazon EKS","optional":false},{"name":"Amazon S3","optional":false},{"name":"AWS","optional":false},{"name":"AWS Lambda","optional":false},{"name":"Azure","optional":false},{"name":"Beautiful Soup","optional":false},{"name":"DynamoDB","optional":false},{"name":"ETL/ELT","optional":false},{"name":"GCP","optional":false},{"name":"Git","optional":false},{"name":"IAM","optional":false},{"name":"Least Privilege","optional":false},{"name":"Linux","optional":false},{"name":"Pandas","optional":false},{"name":"Playwright","optional":false},{"name":"PostgreSQL","optional":false},{"name":"Python","optional":false},{"name":"RAG","optional":false},{"name":"Scrapy","optional":false},{"name":"Selenium","optional":false},{"name":"SQL","optional":false},{"name":"Apache Kafka","optional":true},{"name":"CI/CD","optional":true},{"name":"CloudFormation","optional":true},{"name":"Docker","optional":true},{"name":"GitHub Actions","optional":true},{"name":"JavaScript","optional":true},{"name":"Jenkins","optional":true},{"name":"Kubernetes","optional":true},{"name":"LLM","optional":true},{"name":"Spark","optional":true},{"name":"Terraform","optional":true}],"status":"live","first_seen_at":"2026-10-28T17:19:52Z","employer_posted_date":null,"last_verified_at":"2026-10-28T17:19:52Z","board_verified":false,"closed_at":null,"days_open":2,"trust":{"level":"not_scored","repost_count":null,"flags":[],"days_open":2},"description":"Location: Remote (Work from Home)\n\nAbout Forage AI\nForage AI builds next-generation systems for large-scale data collection and processing — including web crawling, document parsing, enrichment pipelines, and automation. We primarily work in Python, design cloud-native systems (AWS-first, with exposure to GCP/Azure), and increasingly integrate GenAI and agent-based workflows into our stack.\nOur engineering culture emphasizes ownership, clarity, and reliability. Every developer owns their module end-to-end and collaborates closely in a high-trust, high-impact environment.\n\nRole Overview\n\nThis role offers a unique blend of advanced web scraping and GenAI/LLM-driven capabilities to build intelligent, scalable data collection systems and pipelines. You will work on designing and operating generic, reusable scrapers, while also leveraging GenAI and AI agents to enhance extraction, enrichment, validation, and automation workflows.\nBeyond building crawlers, you will use GenAI to help create stable, production-grade data products and services aligned with real-world business use cases — transforming raw web and document data into reliable, market-ready solutions. The work sits at the intersection of deep technical problem-solving and market orientation, in a fast-moving, high-tech, and cutting-edge engineering environment.\nThe role is highly hands-on and emphasizes clean design, reliability, scalability, and impact, with strong ownership from design through deployment and operation.\n\nKey Responsibilities\n\n●Develop and maintain Python-based systems for large-scale crawling, parsing, enrichment, and processing of structured and unstructured data.\n● Build generic, reusable crawling frameworks capable of extracting data from thousands of websites and documents using shared, configurable codebases.\n● Design and implement GenAI-assisted data extraction and enrichment workflows, including:\n○ Using LLMs to interpret semi-structured or unstructured content (HTML, PDFs, text-heavy pages).\n○ Applying prompt-driven logic for classification, normalization, entity extraction, and validation.\n● Implement RAG (Retrieval-Augmented Generation) patterns by combining crawled data with vector databases to improve accuracy, consistency, and explainability.\n● Handle complex anti-crawling challenges, including IP rotation, retries, throttling, headers, fingerprinting, and bot-detection mechanisms.\n● Derive common patterns from semi-structured data, build resilient parsing logic, and gracefully manage edge cases and failures.\n● Build and operate end-to-end automated pipelines (crawl → process → enrich → validate → store → deliver), including AI-powered enrichment stages.\n● Design and maintain ETL/ELT workflows with strong validation, monitoring, error-handling, and auditability.\n● Work with SQL, NoSQL, and vector databases, contributing to data modeling, storage, and retrieval strategies.\n● Implement and consume APIs and microservices, including services that expose AI-powered enrichment or extraction capabilities.\n● Contribute to cloud-native system design on AWS (S3, Lambda, ECS/EKS, SQS/SNS, RDS/DynamoDB, CloudWatch).\n● Own live execution of crawlers and pipelines, managing turnaround times, exceptions, QA checks, and delivery SLAs.\n● Write unit and integration tests, debug production issues, profile performance, and participate in code reviews.\n● Implement observability (logging, metrics, tracing) and follow security best practices (secrets management, IAM, least privilege).\n● Collaborate closely with Dev, QA, and Ops teams; ship incrementally using small PRs, design docs, and measurable outcomes.\n\nRequired Qualifications:\n\n● 4+ years of professional experience as a Software Engineer.\n● Strong proficiency in Python, with solid understanding of data structures, algorithms, and clean software design\n● Hands-on experience with web crawling and scraping, including:\n○ Requests, Scrapy, BeautifulSoup (BS4), Pandas, Urllib\n○ Selenium / Playwright (or similar browser automation tools)\n● Proven experience scraping large-scale or complex websites.\n● Strong understanding of anti-bot measures and resilient crawling strategies.\n● Working knowledge of SQL and experience with at least one RDBMS (PostgreSQL, SQL Server, etc.).\n● Exposure to AWS services and cloud-native concepts.\n● Comfortable working on Linux and using Git for version control.\n● Practical understanding of system design and distributed systems basics.\n\nPreferred / Good to Have (Prioritized)\n\n● Containers & CI/CD:\n○ Docker, GitHub Actions / Jenkins\n○ Basic exposure to Kubernetes\n● Data Infrastructure:\n○ Airflow, Spark, Kafka, or large-scale ETL systems\n● Infrastructure as Code:\n○ Terraform or CloudFormation\n○ Basic cloud cost and performance optimization\n● Frontend / JavaScript:\n○ basic familiarity is a nice-to-have\n● Exposure to GCP or Azure\n\nHow We Work:\n● Strong ownership: design → build → deploy → operate.\n● Pragmatic engineering with small PRs and incremental delivery.\n● Emphasis on clear communication, documentation, and reliability.\n● Engineering decisions guided by scale, cost, and long-term maintainability.\n\nWork-from-Home Requirements\n● Reliable high-speed internet for calls and collaboration.\n● A capable computer (modern CPU, 8GB+ RAM).\n● Headphones with clear audio quality.\n● Stable power and backup arrangements.\n\nForage AI is an equal-opportunity employer. We value curiosity, craftsmanship, and collaboration, and we look for engineers who enjoy solving hard problems at scale.","description_format":"text","description_chars":5513,"description_truncated":false,"requirements":{"experience_years_min":4,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":[],"lifecycle":[{"event":"open","at":"2026-10-28T20:09:00Z"}],"visa":[],"liveness":{"score":86,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.86,"p_room":1,"age_days":1,"expected_fill_days":28,"reasons":["seen:1","win:early"],"computed_at":"2026-10-30T05:48:23Z"},"pay":null,"html_url":"https://alion.io/job/forage-senior-software-engineer-2","json_url":"https://alion.io/job/forage-senior-software-engineer-2.json","meta":{"generated_at":"2026-10-30T22:28:08Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":13290,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-31T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":26589},"rest":"https://alion.io/mcp/rest/get_company?id=26589"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fforage-senior-software-engineer-2"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fforage-senior-software-engineer-2"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fforage-senior-software-engineer-2"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/forage-senior-software-engineer-2\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fforage-senior-software-engineer-2"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":3418963}},{"tool":"search_jobs","arguments":{"company":"forage.ai","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":3418963},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}