Technical Lead

Job Summary

• 4+ years of software engineering experience developing big data pipelines.
• 3+ years of hands-on experience programming in Scala and PySpark.
• 3+ years of experience building solutions on AWS, including AWS Glue.
• 3+ years of experience working with big data file formats, including Parquet.
• 1+ years of experience with data validation, data profiling, or building rule-based data engines.
• 1+ years of experience leveraging AI coding assistants or code-generation tools (e.g., GitHub Copilot, Codex) in daily software development tasks.

Key Responsibilities

• Design, build, and optimize a highly scalable, rule-based data validation engine capable of validating data for any data pipeline.
• Develop robust, high-performance big data pipelines and platforms using Scala, PySpark, and AWS Glue.
• Write clean, well-tested code utilizing optimized formats like Parquet for highly efficient storage and retrieval.
• Perform detailed data validation and profiling to ensure quality and consistency across various data systems.
• Use enterprise-approved AI tools to streamline software development workflows, automate tasks, and drive continuous improvement.
• Collaborate with cross-functional teams to translate business requirements into efficient, scalable technical solutions.
• Evaluate emerging trends and AI technologies to inform solution design and strategic innovation in data validation and big data.
• Participate in code reviews, testing, debugging, and profiling to ensure code meets quality and performance standards.

Skill Requirements

Other Requirements