Databricks/Python/AWS

Job Summary

Sr Developer will design and optimize scalable data solutions on cloud platforms using Databricks Pyspark and Azure services for business critical analytics. The role focuses on creating secure data products with Unity Catalog managing workflows and improving data reliability. The Sr Developer will collaborate across teams in a hybrid day shift model to deliver trusted insights that support strategic decisions and responsible innovation.

Responsibilities

• Design and implement scalable data pipelines on Databricks that transform raw data into curated datasets enabling analytics teams to generate timely insights that improve business decision making and operational efficiency.
• Develop reusable Python and PySpark components that standardize data processing logic reduce code duplication and enhance maintainability for complex data engineering solutions across multiple projects.
• Configure and manage Azure Data Lake Store structures and access patterns to ensure secure organized and performant storage that supports large volume data workloads and reliable downstream consumption.
• Apply Databricks Unity Catalog to define and enforce governance policies including fine grained data permissions and standardized metadata so that data assets remain compliant and consistently discoverable across domains.
• Author and optimize Databricks SQL queries that deliver efficient reporting datasets and analytical views focusing on performance tuning and resource utilization to reduce compute costs while preserving responsiveness.
• Orchestrate Databricks Workflows to schedule monitor and recover critical data jobs ensuring end to end pipeline reliability predictable delivery times and robust handling of failures and dependencies.
• Integrate data engineering solutions with Azure DevOps by configuring repositories branching strategies and automated deployment pipelines that promote continuous integration practices and reduce manual release risks.
• Collaborate with data analysts and product stakeholders to translate analytical requirements into technical designs validating that delivered data models and pipelines align with business intent and provide measurable value.
• Implement data quality checks validation rules and logging frameworks within pipelines to detect anomalies early improve trust in data products and support transparent incident analysis and remediation efforts.
• Document data flows transformation logic and operational procedures in clear technical artifacts that help other team members understand system behavior and support sustainable long term maintenance of solutions.
• Optimize compute cluster configurations and job parameters within Databricks to balance performance and cost ensuring that data workloads run effectively during day shift windows and align with capacity planning goals.
• Coordinate effectively in a hybrid work setting using collaboration tools and scheduled on site interactions maintaining consistent communication on progress risks and dependencies without requiring travel commitments.
• Contribute to continuous improvement initiatives by reviewing existing data engineering practices proposing enhancements and implementing changes that elevate the reliability and societal impact of data driven products.
  
  

Qualifications

• Demonstrate strong hands on experience in Databricks Unity Catalog including designing catalog structures and permission models that support secure multi domain data governance in large scale environments.
• Show advanced proficiency with Azure Data Lake Store covering data layout strategies security controls and performance considerations that enable high throughput ingestion and processing patterns.
• Apply solid expertise in Azure DevOps for version control and pipeline automation including configuration of build and release workflows that support data engineering lifecycles and quality gates.
• Exhibit deep practical knowledge of Python and PySpark for data transformation including writing efficient code handling schema evolution and optimizing distributed processing logic on big data workloads.
• Use Databricks SQL fluently to design analytical views parameterized queries and performance tuned datasets that serve reporting and dashboard needs for business and operations stakeholders.
• Operate Databricks Workflows with confidence setting up job dependencies monitoring execution results and implementing retry and alerting strategies that maintain reliable day to day data operations.
  
  

Certifications Required

Preferred certifications Azure Data Engineer Associate and Databricks Certified Data Engineer Professional