First seen by Alion on Nov 18, 2026. Tesco scores A on the Alion truth index.
We are looking for an experienced Data Software Engineer (SDE 2) with strong expertise in Python, Scala, Apache Spark and Big Data technologies.
The ideal candidate should have hands-on experience building and designing large-scale distributed data platforms, data lakes and real-time data processing solutions.
You will be responsible for developing scalable data engineering solutions, optimising distributed processing systems, implementing data architecture best practices and taking ownership of end-to-end technical deliverables.
Key Responsibilities:
- Design, develop and maintain large-scale distributed data platforms and services.
- Build scalable data ingestion and processing pipelines using Python, Scala and Apache Spark.
- Develop batch and real-time streaming applications using Spark Streaming and Kafka.
- Design and implement Data Lake and Medallion Architecture solutions.
- Optimise Spark jobs, SQL queries and distributed processing workloads for performance and scalability.
- Implement data validation, data quality and automated testing frameworks.
- Develop reusable components, frameworks and automation solutions.
- Drive technical design, architecture decisions and end-to-end delivery.
- Collaborate with Engineering Managers, Product Managers, TPMs and cross-functional stakeholders.
- Mentor junior engineers, conduct code reviews and promote software engineering best practices.
- Work with CI/CD pipelines, DevOps tools and Agile development processes.
- Participate in production support activities, including 24x7 support on a rotational basis.
Mandatory Skills & Experience:
- 5-10 years of overall software development / Big Data engineering experience.
- Strong hands-on experience in Python and advanced Scala programming.
- Strong understanding of Functional Programming, Case Classes, Complex Data Structures and Algorithms.
- Experience building large-scale distributed systems and data platforms.
- Strong experience with Apache Spark, Spark Streaming and SQL.
- Hands-on experience with Hadoop, Sqoop and Big Data processing frameworks.
- Experience with Kafka and real-time data streaming.
- Strong understanding of Data Lakes, Data Warehousing and Medallion Architecture.
- Experience with data ingestion from diverse sources and file formats.
- Knowledge of performance optimisation and query tuning.
- Experience with ORC, Avro, Parquet, JSON and CSV file formats.
- Good understanding of SOLID principles, DRY principles and software architecture.
- Experience with unit, integration, system and regression testing.
- Exposure to DevOps and CI/CD tools such as Jenkins, Maven, GitHub and JFrog/Artifactory.
- Experience with workflow orchestration tools such as NiFi, Airflow, Oozie, Azkaban or Spark on Kubernetes.
- Shell scripting and automation experience using Ansible or similar tools.
- Strong communication, documentation and technical ownership skills.
Good to Have:
- Experience managing large-scale on-premises Hadoop clusters or cloud-based Big Data platforms.
- Cloud experience with AWS, Azure or GCP.
- Experience with Kubernetes, Docker and container technologies.
- Knowledge of Hive, Cassandra, Dremio and Presto.
- Exposure to Apache Iceberg, Flink, Apache Arrow and Kafka Connect.
- Experience with cloud-native storage such as S3 or Ceph.
- Knowledge of Data Catalog, Data Governance and Data Security.
- Understanding of GDPR and regulatory compliance.
- Experience with security and secrets management tools such as HashiCorp Vault, Ansible Vault and Snyk.
Leadership & Collaboration:
- Experience working in a Technical Lead or Module Lead capacity.
- Ability to mentor engineers and guide technical implementation.
- Strong problem-solving and architectural design capabilities.
- Ability to translate business requirements into high-quality technical solutions.
- Strong ownership, proactive approach and ability to work across multiple engineering teams.
- Experience working in Agile Data Engineering environments.
Note:
The role involves 24x7 production support as per the rotational schedule.
Skills
Airflow, Apache Spark, Hadoop, Kafka, Python, Scala, Spark Streaming

