[Job- 29791] Senior Data Engineer [Databricks required] (Hybrid 3x/wk)

Senior Data Engineer

Job Purpose

As a Senior Data Engineer, you will serve as a technical leader and mentor within cross-functional project teams, taking ownership of complex data solutions and architectural decisions within your area of expertise. You will be responsible for designing and implementing high-performance data systems, including data pipelines, storage solutions, and processing frameworks, while mentoring junior and middle-level colleagues and ensuring technical excellence through comprehensive code reviews and testing practices.

In this role, you will contribute to data strategy discussions, lead the implementation of critical data workflows, and bridge the gap between technical execution and business objectives. You will also maintain strong client relationships and support pre-sales activities when needed.

Required Technical Stack

Core Tech Stack: SQL | Python | PySpark | Databricks | AWS

• SQL - Strong proficiency in writing complex queries, data transformation, optimization, and working with large datasets
• Python - Strong experience in Python for data engineering, automation, data processing, and pipeline development
• PySpark - Hands-on experience developing and optimizing distributed data processing workflows using PySpark
• Databricks - Strong hands-on experience with Databricks for data engineering, data processing, pipeline development, and analytics
• AWS - Hands-on experience designing, implementing, and supporting cloud-based data solutions using AWS services

Key Accountabilities

Technical Leadership & Engineering Excellence

• Lead the design and implementation of features, including complex data processing workflows and pipelines, with high attention to detail and quality standards
• Design, develop, and optimize scalable data solutions using SQL, Python, PySpark, Databricks, and AWS
• Contribute to architecture decisions within project scope and provide technical input for broader data strategy discussions
• Establish and maintain engineering standards, best practices, and comprehensive testing strategies within development teams
• Conduct thorough code reviews and drive adoption of a strong peer review culture for continuous improvement
• Lead troubleshooting of complex technical issues and provide innovative solutions to challenging data engineering problems
• Drive performance optimization initiatives for data systems and ensure scalability and reliability in technical implementations
• Evaluate data processing frameworks, cloud technologies, and emerging tools for potential adoption within projects
• Lead proof-of-concept development and technical risk assessments for new data initiatives
• Ensure data solutions are secure, maintainable, reliable, and aligned with business and technical requirements

Team Development & Mentorship

• Mentor and develop junior and middle-level colleagues across different technical areas and specializations
• Provide technical guidance, knowledge sharing, and support for the career progression of team members
• Support technical hiring processes through candidate evaluation, interviewing, and technical assessments
• Contribute performance evaluation input and provide constructive feedback for team members
• Develop and deliver technical training sessions to elevate team capabilities and foster a culture of continuous learning
• Lead by example in adopting engineering best practices, including test-driven development and automated testing approaches
• Support team collaboration and knowledge transfer across different technical domains and projects
• Share expertise in SQL, Python, PySpark, Databricks, AWS, and other relevant data engineering technologies

Project Execution & Delivery

• Take ownership of complex technical tasks and ensure timely, high-quality delivery within project timelines
• Design and implement reliable and scalable data pipelines and processing workflows
• Provide accurate technical estimations and planning input for development tasks and project milestones
• Coordinate technical dependencies and collaborate effectively across different organizational units
• Contribute to Agile development practices and ensure technical considerations are represented in sprint planning
• Support release management activities and participate in deployment processes with comprehensive testing and validation
• Balance technical debt management with feature delivery to maintain sustainable development practices
• Monitor and optimize data workflows to ensure performance, scalability, reliability, and maintainability

Client & Stakeholder Engagement

• Participate in client interactions and technical discussions to understand requirements and provide appropriate data engineering solutions
• Contribute to technical documentation, solution design, and clear communication of complex technical concepts to stakeholders
• Support pre-sales activities through technical expertise, solution demonstrations, and client consultations when needed
• Assist in translating business requirements into technical specifications and implementation approaches
• Provide technical input on project feasibility, resource requirements, and timeline estimations for stakeholder planning
• Explain technical solutions and recommendations clearly to both technical and non-technical stakeholders
• Maintain professional relationships with clients and contribute to long-term client satisfaction through technical excellence

Business Adaptability & Professional Growth

• Demonstrate Technical Leadership: Lead technical initiatives with confidence, make informed decisions, and take ownership of complex data engineering challenges while mentoring others
• Drive Adaptability & Continuous Growth: Execute seamless transitions between different projects, technologies, and client requirements while continuously upskilling in emerging data engineering technologies and methodologies
• Execute Quality-Focused Development: Apply analytical thinking with attention to detail, prioritize security and maintainability, and ensure comprehensive testing coverage in all deliverables
• Practice Effective Communication: Communicate complex technical concepts clearly to various stakeholders, collaborate effectively across teams, and maintain high ethical standards with transparency
• Stay Current with Technology: Continuously evaluate advancements in data engineering, cloud computing, distributed processing, and analytics technologies to identify opportunities for improvement
• Promote Engineering Excellence: Advocate for scalable, reliable, and maintainable data solutions while continuously improving development and delivery practices

Core Technology Requirements

SQL | Python | PySpark | Databricks | AWS

Strong hands-on experience across the core technology stack is expected, particularly in building and optimizing scalable data pipelines, distributed data processing solutions, and cloud-based data platforms.