Data Solution Engineer
First seen by Alion on Nov 12, 2026.
Data Solution Engineer:
- Core experience: Minimum 5 years of hands-on experience in data processing pipeline and API development, with strong proficiency in Python (PySpark) and Scala a must
- Big data & distributed systems: Apache Spark, Databricks, Blob Storage, BigQuery and managed Spark (Dataproc) on GCP; processing structured, semi-structured and multi-modal data (text, image, audio, video)
- Lakehouse & table formats: Delta Lake, Apache Iceberg and Apache Hudi
- Data modelling & stores: Designing data schemas, tables and partitioning strategies across SQL and NoSQL databases such as Cosmos DB and Redis
- Pipelines & orchestration: Building batch and streaming ETL/ELT processes over pub/sub platforms (Kafka, Azure Event Hubs, Google Cloud Pub/Sub), orchestrated with Airflow on Kubernetes, in containerized, scalable and serverless architectures
- Infrastructure as Code: Terraform or equivalent
- Performance & cost optimization: Spark tuning, shuffle optimization, caching, cluster sizing, query optimization, debugging long-running jobs and cloud cost control
- Data quality & lineage: Defining data quality rules and data lineage definitions, and creating monitoring alerts out of them
- Testing for data pipelines: Data validation, regression tests, reconciliation and end-to-end pipeline testing
- Observability & reliability: Centralized logging, metrics, tracing, job monitoring, SLAs/SLOs and incident troubleshooting
- ML engineering: Feature store, feature and training pipelines, model deployment and serving, model monitoring with drift detection and retraining (MLflow / MLOps)
- AI-assisted development: Hands-on experience using tools such as GitHub Copilot, Claude Code or equivalent for code generation, debugging, testing and developer productivity
Skills
Data Engineering, Python, Scala, Apache Spark, Databricks, BigQuery, Delta Lake, Terraform, MLOps