Data Engineer - PySpark

We are seeking a highly motivated and technically skilled Senior Data Engineer with strong expertise in data pipeline development and advanced data modelling to join our data team. The ideal candidate will have extensive experience in designing, building, and refactoring scalable data pipelines and data models within a modern data engineering ecosystem using Palantir Foundry, AWS, and Snowflake, along with broader expertise in data architecture, governance, and processing. This role offers the opportunity to work on cutting-edge data platforms and deliver impactful solutions for enterprise data integration, transformation, and advanced analytics.

Key Responsibilities:

• Design, implement, and maintain scalable and reliable data pipelines using Palantir Foundry and AWS.
• Solid experience in web front-end development and web services.
• Some exposure to business analytics, financial forecasting, or a related domain would be preferable.
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and translate them into technical solutions.
• Build reusable data assets, integrate data from multiple sources, and ensure data quality, integrity, and security.
• Develop and maintain transformation workflows, ontology models, and data governance frameworks within Foundry.
• Optimize data pipelines for performance, scalability, and cost efficiency.
• Troubleshoot, monitor, and resolve data issues and pipeline failures in real time.
• Implement best practices in data versioning, lineage, and observability.
• Assist in defining data architecture strategies and tooling roadmaps.
• Mentor junior engineers and contribute to team knowledge sharing and documentation.

Required Qualifications:

• 5-12 years of experience in data engineering, analytics, or software engineering roles.
• Strong hands-on experience with PySpark, Palantir Foundry, and AWS, including ontology modelling, pipelines, data fusion, and transformation logic.
• Proficient in Python, SQL, Snowflake, and data processing frameworks.
• Solid understanding of data integration, ETL workflows, and data modelling principles.
• Experience working with relational databases, data lakes, cloud data warehouses (AWS, Azure, GCP, Snowflake), and web services.
• Strong problem-solving skills and ability to debug complex data workflows.
• Familiarity with CI/CD pipelines and version control tools like Git.
• Excellent communication skills and ability to work collaboratively with cross-functional teams.

Skills

AWS, Foundry, Palantir, Palantir Foundry, PySpark, Snowflake