Senior Data Engineer - Agentic Data Infrastructure
First seen by Alion on Nov 2, 2026.
Job Overview :
We are seeking a highly skilled Senior Data Engineer to architect and build the foundational data infrastructure that powers enterprise-grade Generative AI and autonomous agent systems. Rather than simply building traditional batch-oriented data warehouses, you will be designing real-time, federated data substrates that allow AI agents to securely access, reason over, and orchestrate complex enterprise data in milliseconds.
In this role, you will solve systemic data fragmentation problems by building zero-copy architectures, low-latency event streams, and dynamic access layers across heterogeneous enterprise platforms. You will work closely with AI Architects to ensure data structures (whether relational, graph, or vector) are optimized specifically for LLM tool-calling and agentic reasoning under strict regulatory constraints.
Key Responsibilities :
- Agent-Ready Data Pipelines: Design and deploy low-latency, event-driven data pipelines (GCP Pub/Sub, Dataflow) that feed real-time context to stateless AI agents, shifting away from traditional nightly batch ETLs.
- Federated Data Architecture: Build read-only integration layers and API wrappers that allow AI models to query disparate operational systems (legacy ledgers, risk engines, CRMs) in parallel without massive data duplication.
- Optimized Data Modeling for AI: Design sophisticated BigQuery schemas, Spanner/AlloyDB relational models, and potentially Graph or Vector structures specifically tuned for high-concurrency, low-latency LLM tool-calling and context injection.
- Identity Resolution & Data Joining: Solve complex cross-system identity mapping challenges, ensuring AI agents have a unified, accurate "360-degree" view of entities across fragmented enterprise brands and databases.
- Governance & Pipeline Security: Implement automated data quality checks, PII redaction layers (Cloud DLP), and Attribute-Based Access Controls (ABAC) to ensure agents never ingest or expose restricted regulatory data.
- Pipeline Orchestration & Cost Tuning: Orchestrate complex DAGs using Apache Airflow (Cloud Composer) while aggressively monitoring and optimizing compute resources (minimizing shuffle in Dataflow/Spark) to control cloud costs at enterprise scale.
Required Qualifications :
- Data Engineering Experience: 6+ years of hands-on experience building robust, scalable data pipelines and modern data warehouses in enterprise environments.
- Google Cloud Platform (GCP) Mastery: Strong operational experience with GCP data services, specifically BigQuery, Dataflow (Apache Beam), Pub/Sub, and Cloud Storage.
- Real-Time & Event-Driven Systems: Proven ability to design streaming architectures and asynchronous data feeds, moving beyond traditional batch processing.
- Advanced SQL & Data Modeling: Expert-level SQL optimization (joins, window functions) and experience designing schemas for both analytical (OLAP) and transactional/operational (OLTP) access patterns.
- Pipeline Orchestration: Hands-on experience with Apache Airflow (Cloud Composer) or similar modern orchestration tools.
- Distributed Frameworks: Working knowledge of distributed processing engines like Spark or Hadoop for heavy lifting when required.
- Cross-Functional Problem Solving: Ability to tackle ambiguous data problems (e.g., identity resolution, federated querying) across siloed enterprise teams.
What Makes You Stand Out :
- Experience building data layers specifically designed for consumption by LLMs or autonomous agents (e.g., semantic layers, feature stores for AI).
- Familiarity with low-latency operational databases on GCP (Cloud Spanner, AlloyDB, or Memorystore/Redis).
- Experience in heavily regulated industries (FinTech, Banking, Healthcare) handling sensitive PII/financial data with strict compliance mandates.
- Understanding of API integration and how to expose data assets as tools for Model Context Protocol (MCP) or similar agent frameworks.
Skills
Data Engineering, Google Cloud Platform, BigQuery, Apache Airflow, SQL, Spark, Agentic AI, Data Infrastructure