Sr. Data Engineer

About the Job

As a Data Engineer at Aspire, you will work

closely with the data engineering and data platform teams to support a

large-scale data platform modernization initiative. You will play a key role in

migrating approximately 1,000 datasets to Snowflake, modernizing existing data

pipelines, and improving data processing reliability and performance.

You will focus on executing established technical migration plans, adapting existing pipelines, and supporting orchestration using Apache Airflow 3. This role offers the

opportunity to contribute to critical data platform initiatives, including Airflow upgrades, Confluent regional migrations, and the transition from Hive to Apache Iceberg, with a target completion date of March 2027.

What you’ll do

• Migrate existing data

  pipelines from Apache Airflow 2 to Apache Airflow 3.

• Migrate Confluent connections

  from the USE1 region to the USE2 region.

• Migrate data pipelines from

  Hive to Apache Iceberg using Snowpark Connect.

• Adapt and migrate existing

  data pipelines to Snowflake as part of a large-scale data platform

  modernization effort.

• Develop, test, and maintain

  data pipelines using Python, SQL, Apache Airflow, and Apache Spark.

• Ensure data quality,

  accuracy, reliability, and consistency throughout the migration process.

• Support workflow scheduling,

  monitoring, troubleshooting, and performance optimization in Apache

  Airflow.

• Work with large-scale

  datasets and AWS services, particularly Amazon S3, to support data lake

  operations and data processing.

• Implement testing and

  debugging strategies to identify and resolve issues across data pipelines

  and processing workflows.

• Collaborate with data

  engineering and data platform teams to execute migration plans and meet

  project deadlines.

• Contribute to the successful

  delivery of time-sensitive platform modernization initiatives targeted for

  completion by the end of March 2027.

What you’ll need

• 5+ years of experience in

  data engineering or software engineering.

• Strong software engineering

  fundamentals and hands-on Python development experience.

• Advanced SQL skills,

  including complex joins, common table expressions (CTEs), window

  functions, and query performance tuning.

• Experience with Snowflake

  data modeling and data warehouse concepts.

• Hands-on experience

  developing, testing, and maintaining data pipelines using Apache Airflow

  and Python.

• Experience working with

  Apache Kafka and distributed data streaming platforms.

• Strong experience with Apache

  Spark, including hands-on PySpark development.

• Experience with Apache

  Iceberg and Parquet data formats.

• AWS experience, particularly

  Amazon S3, including data lakes, partitioning strategies, and lifecycle

  policies.

• At least 2 years of industry

  experience working with large-scale datasets using technologies such as

  Apache Airflow and Apache Spark.

• Strong testing, debugging,

  and troubleshooting skills, with the ability to investigate and resolve

  issues independently.

• Excellent analytical,

  problem-solving, and communication skills.

• Ability to work

  independently, adapt to established technical architectures, and deliver

  results within tight deadlines.

• Strong collaboration skills

  and the ability to work effectively with distributed engineering teams.

• Experience with IT security

  best practices and standards such as ISO or SOC.

• Hands-on experience with Snowflake

  migrations, including data warehouse modernization, performance

  optimization, and cost optimization.

• Experience migrating Apache

  Airflow environments, including version upgrades, executor changes, or

  platform migrations.

• Experience with Confluent and

  Kafka infrastructure migrations across regions.

• Experience migrating data

  pipelines from Hive to Apache Iceberg.

• Experience using AI-powered

  tools to support software development, testing, and debugging.

Why Aspire

In addition to a competitive

long-term total compensation package with salary and performance-based bonus,

we have a reward philosophy that goes beyond compensation.

• Be part of a remote-first

  organization where flexibility is embraced.

• Work and learn alongside

  talented engineers and technology leaders.

• Explore opportunities to

  learn and grow through technical and non-technical training programs.

• Gain global exposure by

  working on products with international teams and clients.

• Attend virtual and

  in-person international technology conferences to expand your knowledge and

  network.

• Exposure to working in an

  IT environment that adheres to rigorous security and compliance standards

  defined by ISO / SOC.