Sr. Data Engineer

Responsibilities:

• Contribute to the architecture, design, and growth of our Data Products and Data pipelines in Scala and Python / Spark while maintaining uptime SLAs.
• Develop scalable solutions, building ETL pipelines in Big Data environments (cloud, on-prem, hybrid).
• Implement the Lakehouse architecture, working with key partners to shift towards a Lakehouse-centric data platform.
• Collaborate with Data Product Managers, Data Architects, and Data Engineers to design, implement, and deliver successful data solutions.
• Maintain detailed documentation of your work and changes to support data quality and data governance.
• Ensure high operational efficiency and quality of your solutions to meet SLAs and support commitment to our customers (Data Science, Data Analytics teams).
• Be an active participant and advocate of agile/scrum practice to ensure health and process improvements for your team.
• Be a problem solver; when presented with new challenges, you are expected to research and network to find solutions.
• Seek out answers to business problems and look for opportunities to automate processes and optimize costs.
• Engage with and understand our customers, forming relationships that allow us to understand and prioritize both innovative new offerings and
• incremental platform improvements.

Requirements:

• Bachelor's degree in computer science, Information Systems, Software, Electrical or Electronics Engineering, or a comparable field of study, and/or equivalent work experience.
• At least 5 years of data engineering experience developing large data pipelines.
• Strong hands-on Experience with Cloud technologies like AWS (S3 EMR, EC2) and building data pipelines.
• Strong algorithmic problem-solving expertise.
• Strong SQL skills and ability to create queries to extract data and build performant datasets.
• Hands-on experience with distributed systems such as Spark, Hadoop (HDFS, Hive, Presto, PySpark) to query and process data.
• Strong fundamental Scala and Python programming skills.
• Experience with at least one major MPP or cloud database technology (Snowflake, Redshift, BigQuery).
• Solid experience with data integration toolsets (i. e, Airflow) and writing and maintaining Data Pipelines with Databricks.
• Experience in Data Modeling techniques and Data Warehousing standard methodologies and practices.
• Familiar with Scrum and Agile methodologies.
• Demonstrated excellent interpersonal and communication skills, including the ability to partner with others and build consensus in a cross-functional team toward a desired outcome.

Preferred Qualifications:

• Experience with at least one major Massively Parallel Processing (MPP) or cloud database technology (Snowflake, Redshift, Big Query).
• Experience with Software Application development.
• Good data acumen to understand domain /functional knowledge and to be able to articulate analysis of data points in a business context for stakeholders.
• Experience working with a test-first mentality.
• Deep Understanding of AWS or other cloud providers, as well as infrastructure as code.

Tech stack:

• AWS
• Snowflake
• Spark
• Databricks
• Delta Lake
• Airflow
• Python
• Scala