Senior Data/ML Engineer - Databricks Forecasting Platform | Remote
Toptal
About the Role
We're looking for a Senior Data/ML Engineer to refactor, operationalize, and improve an existing time series forecasting platform that's already live and driving real business value. This is not a greenfield build — the focus is modernizing a Databricks-based forecasting system that's been maintained primarily by a single developer for years. You'll reduce technical debt, strengthen testing and observability, and raise the engineering bar on a system the business already depends on. If you'd rather bring discipline and maturity to an existing production system than start from a blank slate, this is built for that.
What You'll Do
Review the current forecasting platform architecture and identify areas for improvement
Refactor existing Databricks, Python, and PySpark implementations
Move business logic out of Databricks notebooks and into reusable Python modules or packages
Improve separation of concerns between orchestration and core business logic
Establish stronger engineering standards and help define what "good" looks like for the platform
Implement or improve automated testing practices and validation mechanisms for forecasting workflows
Build or improve monitoring and observability, increasing visibility into how predictions are generated
Help monitor model behavior and operational health over time
Improve reliability of scheduled training workflows, reducing manual intervention on failure
Improve failure handling, retries, and overall workflow resilience
Maintain and extend existing forecasting capabilities as needed
What You Bring
Strong professional experience with Databricks, including workspaces, notebooks, scheduled workflows, and CI/CD processes
Strong Python engineering experience, including designing reusable modules or packages
Strong PySpark experience with production data pipelines or distributed data processing
Experience refactoring production code and improving maintainability
Familiarity with time series forecasting concepts and workflows
Ability to understand and work effectively within an existing, unfamiliar codebase
Experience improving software quality, testing strategy, and engineering standards
Experience implementing automated testing practices
Experience improving monitoring, observability, or operational visibility for production systems
Strong judgment around technical debt, refactoring priorities, and maintainable architecture
Ability to work with existing systems rather than only building from scratch
Why This Role
Real production impact: Improve a system the business already relies on, not a proof-of-concept
Engineering maturity focus: Bring testing, observability, and maintainability to a platform that's outgrown its current state
Meaningful ownership: Help define engineering standards for the forecasting platform going forward
Flexible location: Preference for Toronto or St. Louis, but open to remote consultants globally with North American working-hours overlap
How to Apply
Ready to bring engineering rigor to a production forecasting platform? Apply through Toptal here: https://www.toptal.com/talent/apply