Data & AI Platform Engineer

Job Description – Data & AI Platform Engineer

Location: Remote (Full time)

Experience: 4+ years

The client is building enterprise AI products and shared platform capabilities that combine modern data engineering with AI, cloud and platform engineering. The Data & AI Platform Engineering job family builds the dependable data and intelligence services used across products, client deployments and Catalyst platform capabilities.

Responsibilities

The balance of work will depend on the assigned product, project or Catalyst capability. Each engineer will contribute across the common foundation and lead or specialize in one or more areas below.Responsibilities

1) Data Platform & Pipeline Engineering (20%)

• Design, build and operate reliable batch, streaming and event-driven data pipelines using Python, SQL, Spark or PySpark and fit-for-purpose cloud data services.
• Implement governed storage, schema evolution, transformation, metadata and data-product patterns for structured and unstructured data.
• Build reusable components that support multiple products and tenants rather than creating one-off pipelines for each delivery.

2) Enterprise Connectors & Governed Access (20%)

• Build and maintain connectors for enterprise systems, APIs, files, databases, event streams and external data providers.
• Implement secure authentication, credential handling, identity propagation, tenant isolation and access controls for connector use.
• Expose consistent application and agent interfaces through APIs and, where appropriate, MCP tools with clear health, lifecycle and audit controls.

3) RAG, Enterprise Knowledge & Memory (20%)

• Engineer ingestion, extraction, chunking, enrichment, embedding, indexing and retrieval flows for enterprise RAG and knowledge services.
• Implement retrieval metadata, provenance, lineage, citations, permissions and quality signals from source content through to application responses.
• Contribute to governed Memory services that capture, qualify, retrieve and remove user or application context across sessions and agents.

4) Evaluation, AI Policy & Quality Engineering (15%)

• Build governed datasets, evaluators, quality checks, regression runs and release evidence for data, retrieval and AI capabilities.
• Implement measurable controls for accuracy, safety, policy compliance, traceability and production readiness.
• Support AI Policy and business-rule enforcement through versioned, testable and explainable service contracts.

5) Cloud Platform, Kubernetes & Operations (15%)

• Package and operate services using containers, Kubernetes, Helm, GitOps and CI/CD across local, SaaS and customer-managed deployment profiles.
• Implement configuration, secrets, identity, tenancy, scaling, resilience, backup and recovery controls appropriate to the service.
• Instrument services with logs, metrics and traces, and use operational evidence to improve performance, reliability and cost.

6) Technical Delivery & Engineering Leadership (10%)

• Translate product and architecture requirements into small, testable and supportable increments with clear acceptance criteria.
• Contribute to architecture decisions, technical design documents, code reviews, testing standards, runbooks and onboarding guidance.
• Work across product, data, AI, platform, security and operations teams to resolve dependencies and deliver reusable outcomes.

Specialization Areas

Data Integration & Enterprise Connectors: APIs, event-driven integration, enterprise systems, credential flows, identity-aware access and connector lifecycle engineering.

RAG & Enterprise Knowledge: document processing, embeddings, vector and keyword search, retrieval quality, metadata, lineage and citations.

Memory, Evaluation, AI Policy & Ontology: context capture and retrieval, governed evaluation, quality gates, policy enforcement, business meaning and explainable outcomes.

Cloud Data & Platform Engineering: cloud data services, Kubernetes, GitOps, platform APIs, security, tenancy, observability, reliability and deployment automation.