Data Resiliency Engineer - Data Lake

SUMMARY OF RESPONSIBILITIES

As a Data Resiliency Engineer, you will be at the forefront of maintaining and enhancing the robustness of our data infrastructure. You will analyze issues to identify root causes and implement solutions that ensure a seamless user experience. Your expertise in data ecosystem monitoring, incident management, and data quality enhancement will play a pivotal role in maintaining operational excellence. You will utilize advanced tools like Datadog and Opsgenie to proactively monitor and resolve issues, act as a subject matter expert for data-related inquiries, and champion data quality initiatives. Additionally, you will work closely with the team to ensure our suite of payments reporting systems for participants as well as PayNet internal reports, are accurate and aligned with evolving organizational needs, driving continuous improvement in our data-driven operations.

KEY AREAS OF RESPONSIBILITIES

• Root Cause Analysis & Bug Fixing: Analyze issues to uncover root causes and implement effective solutions, ensuring a smooth user experience.
• Data Ecosystem Monitoring: Oversee the data environment using advanced monitoring tools like Datadog, proactively identifying and addressing issues before they escalate.
• Alert System Management: Collaborate with the team managing Opsgenie and other alert systems, ensuring timely responses to critical incidents and maintaining operational excellence.
• Data Lake Support: Serve as the primary contact for data-related inquiries, including ETL incident management, reporting challenges, and providing actionable insights.
• Data Quality Monitoring: Own and enhance data quality monitoring tools, designing robust pipelines and frameworks to maintain the highest data integrity standards.
• Reporting Management: Lead report amendments and ensure reporting processes are accurate and meet the evolving needs of the organization.

QUALIFICATIONS & EXPERIENCE

Minimum Qualifications

• Bachelor's degree in Computer Science, Engineering, Information Systems, or a related field.
• Strong experience with AWS data services (e.g., S3, Glue, Athena, Quicksight, Lambda).
• Strong experience in Python programming language
• Proficiency in PySpark for large-scale data processing.
• Familiarity with workflow management tools (e.g., Apache Airflow)
• Familiarity with operating data tools on Kubernetes and EKS
• Expertise in monitoring tools (e.g., Datadog, CloudWatch) to proactively track and resolve issues.
• Experience with alert management systems like Opsgenie or similar platforms.
• Strong understanding of data lakes and managing large-scale data environments.
• Experience in performing root cause analysis for complex data issues and implementing effective bug fixes.
• Terraform or other IaC tools for infrastructure provisioning
• Relevant certifications in AWS and data engineering

PERSONAL QUALITIES

• Self-motivated problem solver who can work with minimal guidance
• Excellent communication skills to articulate technical details clearly to non-technical stakeholders.
• Detail-oriented with a focus on data quality and reliability
• Proven ability to work cross-functionally with multiple teams (e.g., Data Engineering, Operations, Analytics).
• Passionate engineer looking to learn new technologies