Data & AI Platform Engineer - Python/PySpark
First seen by Alion on Oct 9, 2026.
About the Role :
We are looking for Data & AI Platform Engineers to build scalable, secure, and reusable enterprise data and AI capabilities. The role involves data engineering, enterprise integrations, RAG, knowledge and memory services, AI evaluation, and cloud platform engineering. Candidates should have strong hands-on experience in Python, SQL, production data pipelines, APIs, and cloud-based engineering.
Key Responsibilities :
- Design and maintain scalable batch, streaming, and event-driven data pipelines using Python, SQL, Spark, or PySpark.
- Build enterprise connectors and integrations for APIs, databases, files, and external systems with secure authentication and access controls.
- Develop RAG pipelines covering document processing, chunking, embeddings, indexing, and retrieval.
- Implement data governance, metadata management, lineage, access controls, and data quality checks.
- Build evaluation frameworks, automated testing, regression checks, and quality controls for AI-powered applications.
- Deploy and operate services using Docker, Kubernetes, Helm, CI/CD, and GitOps.
- Implement monitoring, logging, security, scalability, and reliability across production environments.
- Collaborate with product, AI, data, security, and platform teams to deliver reusable enterprise solutions.
Required Skills:
- Strong hands-on proficiency in Python and SQL.
- Practical experience with Spark/PySpark and production-grade data pipelines.
- Experience building APIs, backend services, data integrations, or reusable platform components.
- Knowledge of cloud platforms and modern data architectures; Azure is preferred.
- Experience with Git, automated testing, CI/CD, and containerized deployments.
- Understanding of data security, identity management, access controls, and production operations.
- Kubernetes experience is expected for Senior and Lead-level positions.
Good to Have :
- RAG, LLMs, embeddings, vector databases, OpenSearch, or Azure AI Search.
- MCP, agent frameworks, AI gateways, Memory or Evaluation platforms.
- Databricks, Delta Lake, Kafka, Azure Event Hubs, PostgreSQL, or pgvector.
- Kubernetes, Helm, Argo CD, GitOps, and OpenTelemetry.
- Experience with enterprise, customer-managed cloud, or on-premises deployments.
Skills
Data Engineering, Python, SQL, Spark, PySpark, Cloud, Generative AI, Artificial Intelligence, RAG, LLM