Senior Data Engineer

Job Description:

Responsibilities:

• Develop & Optimize Data Pipelines
  • Build, test, and maintainETL/ELT data pipelines using Azure Databricks & Apache Spark (PySpark).
  • Optimizeperformance and cost-efficiency of Spark jobs.
  • Ensure data quality through validation, monitoring, and alerting mechanisms.
  • Understand cluster types, configuration, and use-case for serverless
• Implement Unity Catalog for Data Governance
  • Design and enforceaccess control policies using Unity Catalog.
  • Managedata lineage, auditing, and metadata governance.
  • Enable secure data sharing across teams and external stakeholders.
• Integrate with Cloud Data Platforms
  • Work withAzure Data Lake Storage / Azure Blob Storage/ Azure Event Hub to integrate Databricks with cloud-baseddata lakes, data warehouses, and event streams.
  • ImplementDelta Lake for scalable, ACID-compliant storage.
• Automate & Orchestrate Workflows
  • DevelopCI/CD pipelines for data workflows usingAzureDatabricks Workflows or Azure Data Factory.
  • Monitor and troubleshoot failures injob execution and cluster performance.
• Collaborate with Stakeholders
  • Work withData Analysts, Scientists, and Business Teams to understand requirements.
  • Translate business needs intoscalable data engineering solutions.
• API expertise
  • Ability to pull data from a wide variety of APIs using different strategies and methods

Required Skills & Experience:

• Azure Databricks & Apache Spark (PySpark) - Strong experience in buildingdistributed data pipelines.
• Python - Proficiency in writing optimized and maintainable Python code for data engineering.
• Unity Catalog - Hands-on experience implementingdata governance, access controls, and lineage tracking.
• SQL - Strong knowledge of SQL for data transformations and optimizations.
• Delta Lake - Understanding oftime travel, schema evolution, and performance tuning.
• Workflow Orchestration - Experience withAzureDatabricks Jobs or Azure Data Factory.
• CI/CD & Infrastructure as Code (IaC) - Familiarity withDatabricks CLI, Databricks DABs, and DevOps principles.
• Security & Compliance - Knowledge of IAM, role-based access control (RBAC), and encryption.

Preferred Qualifications:

• Experience withMLflow for model tracking & deployment in Databricks.
• Familiarity with streaming technologies (Kafka, Delta Live Tables, Azure Event Hub, Azure Event Grid).
• Hands-on experience with dbt (Data Build Tool) for modular ETL development.
• Certification inDatabricks, Azure is a plus.
• Experience with Azure Databricks Lakehouse connectors for SalesForce and SQL Server
• Experience with Azure Synapse Link for Dynamics, dataverse
• Familiarity with other data pipeline strategies, like Azure Functions, Fabric, ADF, etc

Soft Skills:

• Strongproblem-solving and debugging skills.
• Ability towork independently and in teams.
• Excellent communication and documentation skills.