Senior Data Engineer

This role focuses on building and maintaining scalable data pipelines and ETL processes to support our company-wide Data and Analytics solutions, with particular emphasis on our OpenSearch (AWS) and Elasticsearch (Azure) environments. You'll be responsible for designing and developing performant, maintainable data processing systems that handle large volumes of data efficiently across our multi-cloud search platforms.

Responsibilities:

• Design, develop, and maintain scalable ETL pipelines that process data from various sources into our OpenSearch (AWS) and Elasticsearch (Azure) platforms.
• Implement and optimize indexing strategies, mapping configurations, and query patterns for search performance.
• Build reliable data workflows with appropriate error handling, monitoring, and alerting specific to search platforms.
• Configure and maintain OpenSearch/Elasticsearch clusters, including sharding strategies, replication, and security settings.
• Create data transformation processes that prepare data for optimal search indexing and retrieval.
• Develop data validation and quality control procedures to ensure accuracy and consistency in search indices.
• Implement and manage data refresh processes to maintain index currency while minimizing performance impact.
• Design and implement proper data retention policies and archiving solutions for search data.
• Ensure search data is optimized for their analytical needs.
• Create and maintain documentation for search platform architecture, data models, and pipeline processes.
• Provide production support for search infrastructure and troubleshoot performance issues.
• Mentor team members on best practices for working with search technologies.

Requirements:

• Bachelor's degree in computer science, engineering, or a related field (or equivalent experience).
• Proven experience with OpenSearch on AWS and Elasticsearch on Azure.
• Proficiency in Python and search query DSL.
• Experience building ETL pipelines that feed search platforms at scale.
• Understanding of search indexing concepts and optimization techniques.
• Experience with version control systems (Git) and CI/CD for data pipelines.