Data Engineer

We are looking for a Data Engineer to join our Data Platform team to partner with our product and business stakeholders across risk, operations, and other domains. We are looking for someone who thrives in taking on complex data infrastructure problems with the initiative, problem-solving, and curiosity needed to build reliable and scalable solutions. This role will focus on building robust data pipelines and engineering foundations by ingesting data from disparate sources, ensuring data quality and consistency, and enabling better business decisions through reliable data infrastructure across core product areas.

This is the perfect opportunity for an individual passionate about data engineering, who has the drive to solve complex technical problems and has the ability to grow alongside our team and our company. The ideal candidate can quickly identify data quality issues and bottlenecks, develop creative engineering solutions, and demonstrate the ability to effectively communicate technical decisions to stakeholders. We are seeking someone who is comfortable collaborating directly with data analysts, data scientists, and business stakeholders to understand requirements and strategically architect and implement data solutions - especially when the path forward isn't fully defined. This will include data pipeline development, data modeling, orchestration design, infrastructure automation, and data quality monitoring.

What you will be doing:

• Building and maintaining scalable data pipelines using Airflow to orchestrate data workflows that ingest, transform, and deliver data from various sources into Snowflake and Databricks.
• Designing and implementing data models in Snowflake that support analytics, reporting, and ML use cases with a focus on performance, reliability, and scalability.
• Developing infrastructure as code using Terraform to automate and manage cloud resources in AWS, ensuring consistent and reproducible deployments.
• Monitoring data pipeline health and implementing data quality checks to ensure accuracy, completeness, and timeliness of data as business needs evolve.
• Optimizing data processing workflows to improve performance, reduce costs, and handle growing data volumes efficiently.
• Troubleshooting and resolving data pipeline issues, working through ambiguity to get to the root cause and implementing long-term fixes.
• Bridging gaps between data and the business by working with cross-functional teams across the US and India office to understand requirements and translate them into robust technical solutions.
• Creating comprehensive documentation on data pipelines, data models, and infrastructure, keeping documentation up to date and facilitating knowledge transfer across the team.

What you should bring:

• 2+ years of data engineering experience with strong technical skills and the ability to architect scalable data solutions.
• Hands-on experience with Python for data processing, automation, and building data pipelines.
• Proficiency with workflow orchestration tools, preferably Airflow, including DAG development, task dependencies, and monitoring.
• Strong SQL skills and experience with cloud data warehouses like Snowflake, including performance optimization and data modeling.
• Experience with cloud platforms, preferably AWS (S3, Lambda, EC2, IAM, etc.), and understanding of cloud-based data architectures.
• Experience working cross-functionally with data analysts, analytics engineers, data scientists, and business stakeholders to understand requirements and deliver solutions.
• An ownership mentality - this engineer will be responsible for the reliability and performance of their data pipelines and expected to fully understand data flows, dependencies, and their implications on downstream users.
• A proactive mindset. While work is assigned, engineers are expected to independently drive their work forward, ask thoughtful questions, and bring structure to ambiguous technical problems.

Nice to have:

• Experience with dbt for transformation logic and analytics engineering workflows integrated with data pipelines.
• Familiarity with Databricks for large-scale data processing, including Spark optimization and Delta Lake.
• Experience with Infrastructure as Code (IaC) tools like Terraform for managing cloud resources and data infrastructure.
• Knowledge of data modeling concepts (e.g., dimensional modeling, star/snowflake schemas, slowly changing dimensions).
• Experience with CI/CD practices for data pipelines and automated testing frameworks.
• Experience with streaming data and real-time processing frameworks