Senior Consultant - Agentic AI & Web Data Engineering
First seen by Alion on Aug 13, 2026.
Job Summary :
We are looking for a highly skilled Senior Consultant Agentic AI & Web Data Engineer with strong expertise in Python, AWS Bedrock, LLM agents, web crawling, and intelligent data extraction.
The ideal candidate will design and develop production-grade Agentic AI solutions using AWS Bedrock, LangChain/LangGraph, and Claude, while also building resilient web data acquisition and document extraction pipelines.
The role requires strong engineering fundamentals and an ability to design deterministic-first, LLM-second pipelines, using traditional software techniques wherever they provide greater reliability, accuracy, and predictability.
Key Responsibilities :
Agentic AI & LLM Engineering :
- Design and develop production-ready Agentic AI applications and LLM-powered workflows using AWS Bedrock.
- Build intelligent agents and multi-step workflows using LangChain and LangGraph.
- Integrate and orchestrate Claude and other LLM models through AWS Bedrock.
- Design agent workflows involving tool calling, state management, context handling, validation, and error recovery.
- Implement appropriate guardrails, retries, fallbacks, and output validation mechanisms.
- Develop reliable AI pipelines following a deterministic-first, LLM-second architecture.
- Evaluate and optimize LLM applications for accuracy, latency, scalability, reliability, and cost.
- Stay current with emerging developments in Agentic AI, LLM frameworks, and AI engineering.
Python & AWS Engineering :
- Develop scalable and maintainable backend applications using Python.
- Build integrations with AWS services using boto3.
- Implement secure AWS authentication and authorization using AWS STS and IAM.
- Design cloud-native solutions with appropriate security, scalability, observability, and fault tolerance.
- Develop APIs and services required to support AI agents and data processing pipelines.
- Perform performance optimization, debugging, and production issue resolution.
Web Crawling & Data Engineering :
- Design and implement resilient web crawling and data acquisition pipelines.
- Develop crawlers using httpx, Playwright, and related Python technologies.
- Handle dynamic web applications, JavaScript-rendered content, pagination, retries, timeouts, rate limits, and session management.
- Work with robots.txt and sitemaps to identify and efficiently navigate publicly available web content.
- Build robust mechanisms for content discovery, extraction, normalization, deduplication, and validation.
- Design crawling systems that can handle changes in website structure and temporary failures.
- Ensure crawling implementations respect applicable website policies, access restrictions, and legal requirements.
PDF & Document Extraction :
- Design and implement content-anchored PDF extraction solutions.
- Extract structured and semi-structured information from complex PDF documents.
- Combine deterministic parsing techniques with LLM-based extraction where appropriate.
- Implement validation and confidence checks to improve extraction accuracy.
- Handle different document layouts, tables, sections, and content structures.
Architecture & Engineering Excellence :
- Design scalable, reliable, maintainable, and production-grade data and AI solutions.
- Establish appropriate boundaries between deterministic processing and LLM reasoning.
- Write clean, modular, reusable, and testable Python code.
- Develop unit, integration, and functional tests for AI, crawling, and extraction workflows.
- Participate in architecture reviews, code reviews, technical discussions, and solution design activities.
- Mentor junior engineers and contribute to technical best practices.
- Troubleshoot complex engineering challenges and perform root-cause analysis.
Skills
Agentic AI, LLM, AWS, AWS Bedrock, LangChain, LangGraph, Data Engineering, Python, Web Crawling