Databricks Data Architect/Lead Data Engineer

We are looking for an experienced Databricks Data Architect / Lead Data Engineer with strong expertise in Data Warehousing, ETL, Databricks, Cloud Data Platforms, and Data Engineering. The ideal candidate should have extensive experience delivering large-scale data and analytics solutions, with strong hands-on expertise in Databricks, SQL, Python, PySpark, AWS/Azure, and modern data architecture.

The candidate must have strong experience in the BFSI (Banking, Financial Services & Insurance) domain, preferably with hands-on experience working on Banking projects, financial data platforms, regulatory reporting, customer/account data, transactions, risk, compliance, or related banking data solutions.

Roles and Responsibilities :

- Design and develop modern Data Warehouse and Data Lakehouse solutions using Databricks and cloud platforms such as AWS and Azure.

- Define and implement scalable, high-performance data architecture and data engineering solutions aligned with business and analytical requirements.

- Provide forward-thinking and innovative solutions in the Data Engineering, Data Analytics, and Data Platform space.

- Collaborate with Data Warehouse, BI, Business, and Technical Leads to understand requirements for new ETL/data pipeline development.

- Design, develop, and maintain robust ETL/ELT pipelines for batch and near-real-time/streaming data processing.

- Develop data transformation processes using SQL, Python, PySpark, Spark, and Databricks.

- Build and optimize Delta Lake/Lakehouse architectures, including ingestion, transformation, storage, and consumption layers.

- Work closely with business stakeholders to understand reporting requirements and translate them into effective data models and reporting-layer solutions.

- Analyze and triage production issues, identify gaps in existing pipelines, perform root-cause analysis, and implement permanent fixes.

- Monitor and troubleshoot ETL/data pipelines, ensuring data quality, availability, reliability, and performance.

- Orchestrate data pipelines using Apache Airflow and integrate workflows with cloud and Databricks platforms.

- Implement data engineering solutions using batch and streaming technologies, including Kafka and AWS Kinesis, where applicable.

- Drive technical discussions with client architects, business stakeholders, engineering teams, and other technical leads.

- Participate in solution architecture, technical design, code reviews, performance optimization, and implementation discussions.

- Mentor and support junior/entry-level team members in resolving technical challenges and production issues.

- Provide technical guidance and help establish engineering best practices, coding standards, and reusable frameworks.

- Work within a DevOps/CI-CD environment using tools such as Git, Terraform, CircleCI, and related technologies.

- Implement and support data governance, data management, security, metadata, and data quality practices.

- Work with modern Databricks capabilities including Unity Catalog, Delta Lake, data sharing, and related Data & AI platform capabilities.

- Collaborate with cross-functional Agile teams and actively participate in Scrum ceremonies, sprint planning, estimation, technical discussions, and retrospectives.

- Support performance tuning of complex SQL queries, Spark jobs, ETL pipelines, and data warehouse processes.

- Ensure solutions adhere to enterprise architecture, security, compliance, and data governance standards, particularly within the BFSI/Banking environment.

Experience :

- 13+ years of overall experience in Data & Analytics, with strong experience in Data Engineering/Data Warehousing.

- Experience delivering at least 2 large-scale, end-to-end Data Warehouse/Data Engineering implementations.

- Strong experience in BFSI/Banking domain projects is mandatory.

- Proven experience in technical leadership, solution design, architecture, and client-facing technical discussions.

- Strong communication, stakeholder management, presentation, and technical leadership skills.

Education :

- Bachelor's and/or Master's degree in Computer Science, Information Technology, Engineering, Data Science, or a related field.

- Equivalent professional experience may be considered.

Skills

Data Architect, Data Engineering, Databricks, ETL, AWS, SQL, Python, Data Warehousing, PySpark, Apache Airflow