Big Data Engineer
First seen by Alion on Sep 29, 2026.
Role Overview :
We are looking for an experienced Big Data Engineer with strong hands-on expertise in Apache Spark and Scala to design, develop, and optimize large-scale data processing solutions. The role will involve building scalable data pipelines, processing high-volume datasets, optimizing distributed workloads, and collaborating with data and technology teams to deliver reliable data solutions.
Key Responsibilities :
- Design, develop, and maintain scalable big data processing applications using Apache Spark and Scala.
- Build data processing and transformation pipelines for large and complex datasets.
- Develop efficient distributed data-processing solutions with a strong focus on performance, scalability, and reliability.
- Optimize Spark jobs, transformations, queries, memory utilization, and cluster workloads.
- Analyze data-processing requirements and translate them into scalable technical solutions.
- Work with structured and unstructured data and implement appropriate processing and transformation techniques.
- Troubleshoot data pipeline failures, performance bottlenecks, and distributed processing issues.
- Develop reusable components and follow software engineering and data engineering best practices.
- Collaborate with Data Engineers, Architects, Analysts, and business stakeholders to understand requirements and deliver data solutions.
- Participate in code reviews, testing, deployment, and production support activities.
- Maintain technical documentation covering data pipelines, processing logic, and operational procedures.
- Contribute to continuous improvement of data processing frameworks, performance, and engineering practices.
Required Skills & Experience :
- 6 - 11 years of experience in Big Data, Data Engineering, or a related technology domain.
- Strong hands-on expertise in Apache Spark and Scala.
- Strong understanding of Big Data concepts, distributed computing, and parallel data processing.
- Experience developing large-scale data processing, transformation, and ingestion solutions.
- Strong understanding of Spark architecture, execution models, partitioning, transformations, actions, and performance optimization.
- Experience troubleshooting and tuning distributed data-processing workloads.
- Good knowledge of SQL and relational data concepts.
- Strong analytical and problem-solving skills.
- Good understanding of software development lifecycle, testing, version control, and deployment practices.
- Ability to work effectively in a collaborative, fast-paced engineering environment.
- Strong communication and stakeholder-management skills.
Skills
Apache Spark, Scala, Big Data, Data Engineering, SQL, ETL