Data Engineer (Databricks) | Specialist

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Data Engineer (Databricks) | Specialist based in Brazil.

This is a specialist-level data engineering role focused on building scalable, reliable, and high-performance data solutions.

You will work extensively with Databricks and modern distributed data technologies to support enterprise data and machine learning initiatives.

The role combines data engineering, cloud platforms, MLOps, and model lifecycle automation.

You will contribute from technical discovery and architecture through implementation, deployment, monitoring, and continuous improvement.

A key focus will be establishing standardized MLOps pipelines and progressively migrating existing models into a robust, reusable framework.

You will work in an agile environment alongside multidisciplinary teams, contributing to technical decisions and delivery.

This is an opportunity to work with modern AI and data technologies while helping build solutions designed for scale, reliability, and long-term business impact.

Accountabilities:

• Build and maintain scalable, reliable data pipelines using modern distributed processing technologies to support high-quality data ingestion, transformation, and delivery.
• Organize and manage data tables using Delta Lake and Unity Catalog, ensuring effective data management and governance.
• Design and implement an automated, native MLOps pipeline on Databricks and Google Cloud Platform (GCP) covering the complete machine learning model lifecycle.
• Develop processes for data preparation, feature engineering, model training, validation, registration, deployment, serving, monitoring, and automated retraining.
• Participate in technical discovery activities, including inventorying existing machine learning models and assessing their migration requirements.
• Develop and validate a standardized MLOps pipeline template through a pilot implementation, followed by progressive migration of models in waves based on business and technical criticality.
• Collaborate with engineering and data teams to ensure solutions are scalable, maintainable, reliable, and aligned with technical standards.
• Work within an agile delivery model, actively participating in sprints, refinement sessions, reviews, retrospectives, and other team rituals.
• Continuously identify opportunities to improve data pipeline performance, automation, reliability, and operational efficiency.

Requirements

• Demonstrated professional experience working with Databricks and modern data engineering environments.
• Strong hands-on experience with PySpark and Apache Spark for distributed data processing.
• Experience developing and orchestrating workflows using Apache Airflow.
• Practical experience with Google BigQuery and cloud-based data platforms.
• Experience integrating and using MLflow for machine learning lifecycle management.
• Knowledge of AWS Glue and its application within data integration and processing workflows.
• Experience working with both SQL and NoSQL databases, including technologies such as PostgreSQL, MongoDB, and Cassandra.
• Ability to design and maintain scalable data pipelines with a strong focus on data quality, performance, automation, and reliability.
• Experience working in Agile/Scrum environments, including sprint planning, refinement, reviews, and retrospectives.
• Strong analytical and problem-solving abilities, with the capacity to work independently and collaboratively on complex technical challenges.
• Nice to have: Knowledge of Apache Kafka for event streaming and real-time data architectures.
• Nice to have: Experience with dbt for data transformation and analytics engineering.

Benefits

• Opportunity to work with modern data engineering, AI, cloud, and MLOps technologies.
• Exposure to large-scale Databricks and GCP-based data environments.
• Opportunity to contribute to end-to-end machine learning lifecycle automation and reusable technical solutions.
• Collaborative and agile working environment.
• Continuous learning and professional development opportunities.
• Exposure to emerging trends in Artificial Intelligence, Generative AI, and advanced technology.
• Opportunity to work on technically challenging projects with meaningful business impact.
• Career growth within a technology-focused and innovation-driven environment.
• Compensation and benefits package aligned with the role and local market