Senior Data Engineer

What you will Do

• Software Engineering

• Architects, deploys, and maintains production-grade Python microservices, serverless components (AWS Lambda), batch workers, web services, and shared core libraries across the analytics ecosystem

• Engineers RESTful interfaces and application integrations to expose dataset capabilities to platform applications, encompassing secure object storage exchange, query orchestration, automated report delivery, and embedded analytics infrastructure

• Constructs backend infrastructure powering AI analytics solutions, incorporating LLM and natural language query translation engines like Snowflake Cortex, controlling dialog state, guaranteeing multi-tenant data isolation before execution, and optimizing throughput and resource utilization

• Authors clean, resilient codebase backed by comprehensive test suites while directing GitHub Actions continuous integration and continuous deployment pipelines for automated compilation, verification, security inspection, and production releases

• Safeguards system configuration and secrets using key-pair authentication, parameter management, and service credentials while proactively updating software dependencies based on vulnerability scanning metrics

• Data Engineering

• Constructs and manages ingestion and transformation workflows utilizing Airflow, AWS Glue, Spark, and Snowflake dynamic tables to feed multi-tier lakehouse environments from diverse sources

• Engineers automated data cleansing and enrichment routines integrating machine learning models and record reconciliation methods, while safely executing massive historical data re-processing

• Architects robust multi-tenant security frameworks in Snowflake employing row-level policies, secure functions, and isolated access controls across global cloud infrastructure regions

• Implements metadata-driven automation to dynamically generate client-specific semantic layers from templates, incorporating state tracking, automated recovery, and version rollbacks

• Executes data governance protocols and privacy compliance workflows, including systematic tenant data purging across all analytical storage tiers

• Develops internal platform analytics modules to track, aggregate, and report infrastructure consumption and operational expenditures across cloud vendor ecosystems

• Establishes proactive telemetry and alerting, drives resolution for system incidents, conducts thorough root-cause investigations, and deploys long-term technical remediation

• Process

• Follows SDLC processes, including adopting agile-based processes, peer code-reviews, and technical preparations required for scheduled releases and demos.

• Partners with product, analytics and data science team to drive requirements that take into account all parties' needs

• Establishes monitoring, responds to alerts and resolves issues within data pipelines

• Uses and helps shape AI-assisted development practices and tooling for the team.

• Impact

• Executes intricate, multi-faceted capabilities spanning microservices, automated workflows, and analytical storage infrastructure with high autonomy while providing technical leadership across mid-scale initiatives

• Assumes direct accountability for the reliability, security architecture, and long-term viability of deployed systems while systematically uncovering technical remediation opportunities

• Pinpoints architectural risks across peer technical deliverables and formulates pragmatic, alternative implementation strategies

• Streamlines operational processes, resolves critical system defects, performs advanced codebase refactoring, and exhibits expert troubleshooting capabilities across application code, cloud infrastructure, and analytical queries

• Authors comprehensive technical documentation, operational runbooks, and detailed data dictionaries to support platform capabilities

• Drives system architecture discussions and fulfills expanded peer code review obligations across the engineering organization

• Spearheads onboarding initiatives and provides technical coaching and mentorship to junior staff members

What you will bring

Experience:

• Demonstrated background of at least six years architecting enterprise software solutions and big data platforms in operational environments.

• Required: Bachelor’s degree in Computer Science or related field, or equivalent years’ experience

• Advanced proficiency in Python engineering, spanning distributed microservices, internal modules, RESTful interface design, unit testing frameworks, and GitHub Actions release pipelines rather than ad-hoc scripting.

• Expert SQL query optimization, data modeling techniques, and extensive background delivering cloud warehouse architectures using Snowflake.

• Proven mastery deploying and orchestrating data processing flows in live production through Apache Airflow.

• Hands-on expertise utilizing Amazon Web Services ecosystems, including AWS Lambda, S3 storage, Glue jobs, IAM policies, Key Management/Secrets Manager, and EventBridge integration.

• Deep comprehension of multi-tenant security architecture-incorporating row-level access policies, fine-grained access governance, strict tenant isolation, and compliant handling of restricted financial assets across international cloud regions.

• Track record of effective cross-functional partnership in product-driven organizations alongside engineering peers and product managers.

Preferred:

• Snowflake advanced features: row access policies, secure views, dynamic tables, stored procedures, Cortex AI / Cortex Analyst, semantic views.

• Experience building LLM-powered or text-to-SQL features, or other GenAI applications in production.

• Spark (AWS Glue or Databricks).

• Fivetran or an equivalent ELT tool; Looker or an equivalent BI platform, including embedded analytics.

• Infrastructure as code and GitHub Actions.

• Experience with ML-assisted data matching or entity resolution.

• Financial, expense or travel industry experience.