Senior Data Engineer

Note: looking for candidate with Energy background 

Senior Data Engineer – Forecasting Platforms & Operations

Position Specific Description

The Senior Data Engineer – Forecasting Platforms & Operations will support the development, integration, operational reliability, and continued modernization of data pipelines and applications within the AURA energy forecasting ecosystem.

This role combines strong data engineering capabilities with hands-on production application support. The position will focus primarily on building and maintaining scalable ETL/ELT pipelines, data integrations, databases, APIs, and automated data workflows supporting load, wind, and solar forecasting, while also providing secondary operational support for the cloud-based platforms and services responsible for producing and delivering forecasts to business customers.

The ideal candidate has a strong data engineering foundation but is also comfortable troubleshooting production applications, following data and application dependencies across systems, supporting cloud-hosted services, and responding to operational issues when necessary.

The position is intended to strengthen both the data engineering capacity and operational resiliency of the AURA forecasting team by developing broader technical ownership across the environment.

Key Responsibilities

• Data Engineering & Integration
• Design, develop, maintain, and optimize ETL/ELT pipelines supporting forecasting, telemetry, weather, market, asset, and operational data.
• Develop and maintain automated data ingestion and transformation processes across APIs, databases, flat files, SFTP, cloud storage, and external data providers.
• Build scalable and reusable Python and SQL-based data processing frameworks.
• Support PostgreSQL and other relational database environments used by forecasting applications and analytical workflows.
• Improve data pipeline monitoring, logging, restart ability, validation, and failure recovery.
• Implement data-quality checks, reconciliation processes, and source-to-target validation.
• Support onboarding and integration of new wind, solar, storage, and other generation assets into forecasting data pipelines.
• Partner with Data Scientists to ensure reliable, timely, and well-structured data is available for model development, training, production forecasting, and performance analysis.
• Support modernization and standardization of legacy data workflows and reduce manual or hard-coded processing wherever practical.
• Forecasting Platform & Operational Support
• Provide secondary technical support for production load, wind, and solar forecasting applications and associated data-delivery processes.
• Troubleshoot production issues across application, data, database, cloud infrastructure, integration, and delivery layers.
• Support AWS-hosted forecasting workloads and services such as S3, Lambda, ECS, EC2, CloudWatch, and related cloud resources.
• Assist with application deployments, release validation, configuration changes, operational readiness, and post-deployment monitoring.
• Support scheduled jobs, automated forecast processes, reporting services, and customer-facing forecast delivery workflows.
• Assist with SFTP services, secure file transfers, SSH keys, directory configurations, and customer data-exchange processes.
• Participate in incident response, root-cause analysis, change management, service-ticket resolution, and production support.
• Maintain sufficient cross-functional knowledge to provide backup coverage for key forecasting platform responsibilities when the primary platform resource is unavailable.
• Reliability, Automation & Documentation
• Identify recurring manual processes and develop automation to improve reliability and reduce operational effort.
• Establish effective monitoring and alerting for pipelines, scheduled processes, data feeds, and production forecasting applications.
• Document system dependencies, troubleshooting procedures, support processes, recovery steps, and operational handoffs.
• Participate in cross-training and knowledge-sharing activities to ensure critical systems have appropriate primary and backup technical coverage.
• Support continuous improvement of the team's operational support model and technical resiliency.

Required Technical Experience

• Strong experience designing, developing, and supporting enterprise ETL/ELT or data-integration pipelines.
• Advanced SQL skills and experience with relational databases such as PostgreSQL, Oracle, SQL Server, or similar platforms.
• Proficiency with Python for data processing, automation, integration, or application-support activities.
• Experience integrating data from APIs, databases, flat files, cloud storage, SFTP, or other enterprise data sources.
• Experience supporting production data pipelines, scheduled jobs, and troubleshooting failed processing workflows.
• Experience working with AWS or comparable cloud environments.
• Strong understanding of logging, monitoring, troubleshooting, root-cause analysis, and production-support practices.
• Experience with Git-based source control and modern software-development practices.
• Preferred Qualifications
• Experience with AWS services including S3, Lambda, ECS, EC2, CloudWatch, Step Functions, or related services.
• Experience with PostgreSQL/Aurora, Redshift, Databricks, Spark/PySpark, or similar data platforms.
• Experience supporting CI/CD pipelines, application deployments, or cloud-hosted production applications.
• Experience with SFTP administration, secure data exchange, SSH key management, or automated file-transfer processes.
• Experience building data-quality, monitoring, reconciliation, and automated alerting frameworks.
• Familiarity with energy markets, utility operations, renewable generation, SCADA/telemetry data, weather data, or load/wind/solar forecasting.
• Experience supporting machine-learning or forecasting data pipelines is strongly preferred.
• Experience working in Agile environments using tools such as Jira and Confluence.

Candidate Profile

• The successful candidate should be a hands-on engineer who naturally follows problems across system boundaries rather than limiting troubleshooting to a single application or technology.
• The strongest candidate will be able to move comfortably between:
• source data → pipeline → database → forecasting application → cloud service → delivery → customer
• and determine where a failure has occurred, how it should be corrected, and how recurrence can be prevented.
• This role requires someone who can operate independently, document clearly, learn unfamiliar systems quickly, and balance planned engineering work with occasional production-support priorities