Senior Data Engineer

Overview

Sr Data Engineer

Position Summary

We are looking for a self-sufficient BI Analyst / Data Engineer to join a small, high-impact analytics team. In this role you will own the full lifecycle of business intelligence delivery-from building and maintaining ETL pipelines in AWS to designing interactive dashboards in Amazon QuickSight. You will work closely with onshore stakeholders to translate business requirements into reliable, well-documented data products that drive operational and strategic decision-making.

The ideal candidate is comfortable operating independently across the data stack, takes ownership of their work, and communicates proactively with team members across time zones.

Core Responsibilities

ETL / Data Pipeline Development

• Design, build, and maintain AWS Glue (PySpark) ETL jobs that ingest data from multiple source systems such as Salesforce, JIRA, Azure DevOps, and Zendesk into a layered data lake architecture (raw → core → presentation).
• Develop and optimize AWS Step Functions workflows for pipeline orchestration, including parallel execution, error handling, retry logic, and concurrency management.
• Implement incremental loading strategies, schema evolution handling, and data quality checks within Glue scripts.
• Work with Apache Iceberg table format on S3 for ACID-compliant analytics tables, including MERGE operations, compaction, snapshot expiration, and orphan file cleanup.
• Integrate supporting AWS services such as EventBridge, SQS, SNS, Athena, and CloudWatch into pipeline architectures.
• Troubleshoot production pipeline failures, diagnose performance bottlenecks (OOM errors, API timeouts, DPU tuning), and implement fixes with minimal guidance.
• Write and maintain clear technical documentation for all pipeline components.

Dashboard & Reporting Development

• Build, publish, and maintain Amazon QuickSight dashboards and analyses that serve operational and executive audiences.
• Create well-modeled QuickSight datasets backed by Athena queries over Iceberg tables, applying appropriate joins, calculated fields, and filters.
• Translate business requirements and existing Tableau/Excel reports into QuickSight visuals, ensuring accuracy and usability.
• Optimize dashboard performance, evaluating SPICE vs. Direct Query modes and managing refresh schedules.
• Collaborate with stakeholders to iterate on dashboard designs, incorporating feedback and refining KPIs.

General Team Responsibilities

• Participate in code reviews, sprint planning, and knowledge-sharing sessions with the broader BI and engineering teams.
• Follow established Git branching strategies and CI/CD practices for infrastructure and code deployment.
• Monitor data freshness and pipeline health; respond to alerts and escalate issues as appropriate.
• Proactively identify opportunities to improve data quality, reduce pipeline run times, and enhance reporting capabilities.

Required Qualifications

• 5-10+ years of professional experience in data engineering, BI development, or analytics engineering.
• Strong hands-on experience with AWS Glue (PySpark), Step Functions, S3, Athena, and IAM.
• Proficiency in Python and SQL; experience writing PySpark transformations and Athena queries.
• Experience building and maintaining ETL pipelines that ingest data from REST APIs or SaaS platforms (e.g., Salesforce, JIRA, Zendesk).
• Working knowledge of Amazon QuickSight, including dataset creation, calculated fields, parameters, and dashboard publishing.
• Familiarity with columnar/lakehouse table formats such as Apache Iceberg, Delta Lake, or Apache Hudi.
• Solid understanding of data warehouse concepts: dimensional modeling, slowly changing dimensions, incremental vs. full loads.
• Comfortable working independently with minimal day-to-day supervision, managing your own task priorities, and communicating status asynchronously.
• Strong written and verbal English communication skills; able to participate in cross-timezone meetings and write clear documentation.

Preferred Qualifications

• Experience with infrastructure-as-code tools such as AWS CDK, CloudFormation, or Terraform.
• Familiarity with CI/CD pipelines (GitHub Actions, CodePipeline) for data engineering workflows.
• Exposure to PII detection/redaction practices or data governance frameworks.
• Prior experience translating Tableau workbooks or Excel-based reports into another BI platform.
• Knowledge of EventBridge-driven architectures and SQS/SNS messaging patterns.
• AWS certifications (e.g., Data Analytics Specialty, Solutions Architect Associate) are a plus.

What We Offer

• The opportunity to work on a greenfield BI platform with modern cloud-native tooling.
• A collaborative, small-team environment where your contributions have direct and visible impact.
• Exposure to a broad technology stack spanning data engineering, analytics, and cloud infrastructure.

• Mentorship and growth opportunities within a growing data organization.