Data Platform Architect - Python
First seen by Alion on Oct 23, 2026.
Data Platform Architect
Location : Hyderabad
Experience : 10+ Years
Role Overview :
We are looking for an experienced Data Platform Architect with strong hands-on expertise in Python, Apache Airflow, AWS, Terraform, and GitHub Actions.
The ideal candidate will be responsible for designing, developing, and managing scalable, reliable, and efficient data platforms and pipelines while ensuring automation, observability, and operational excellence.
Key Responsibilities :
- Design, develop, and maintain scalable data architectures and data pipelines using Python.
- Deploy, configure, and manage Apache Airflow on Amazon EKS at scale.
- Develop and maintain custom Airflow DAGs, Operators, Hooks, and Sensors to support complex data orchestration requirements.
- Work extensively with AWS services, including EKS, Glue, Firehose, S3, AWS Batch, Lambda, ECS, RDS, and DynamoDB.
- Implement Infrastructure as Code (IaC) using Terraform for infrastructure provisioning and management.
- Build and maintain CI/CD pipelines using GitHub Actions to automate deployments and streamline development workflows.
- Establish platform monitoring, logging, and observability practices to ensure reliability, scalability, and performance.
- Troubleshoot complex platform and pipeline issues, identify root causes, and implement long-term solutions.
- Follow best practices for security, performance optimization, fault tolerance, and cloud resource management.
- Collaborate with engineering, data, and DevOps teams to deliver robust and production-ready data platform solutions.
Required Skills and Qualifications :
- 10+ years of overall experience in data engineering, platform engineering, cloud architecture, or related domains.
- Strong hands-on programming experience in Python.
- Extensive experience with Apache Airflow, including DAG development and custom Operators, Hooks, and Sensors.
- Proven experience deploying and managing Airflow on Amazon EKS in large-scale environments.
- Strong knowledge of AWS services, particularly EKS, Glue, Firehose, S3, AWS Batch, Lambda, ECS, RDS, and DynamoDB.
- Hands-on expertise in Terraform and Infrastructure as Code practices.
- Experience designing and implementing CI/CD pipelines using GitHub Actions.
- Strong understanding of data pipeline architecture, orchestration, distributed systems, and cloud-native technologies.
- Experience with monitoring, observability, production troubleshooting, and platform reliability.
- Strong problem-solving skills and the ability to work independently in complex technical environments.
Preferred Qualifications :
- Experience building and operating enterprise-scale data platforms.
- Knowledge of cloud security, access management, networking, and cost optimization.
- Experience implementing automated testing, deployment strategies, and infrastructure monitoring.
- Excellent communication and collaboration skills.
Ideal Candidate :
- A hands-on technical architect who can design scalable data platforms while actively contributing to Python development, Airflow orchestration, AWS infrastructure, and deployment automation.
Skills
Data Architect, Python, AWS, IAC Terraform, Elastic Kubernetes Service, CI/CD Pipeline, Monitoring Tools, Log Management Tools