Senior Engineer - Data Engineering

Company: Altimetrik

Location: Hyderabad preferred/ Any location

Job Title: L4 Databricks Platform Administrator (AWS)

About Altimetrik:

Altimetrik is a digital business services company that focuses on delivering tangible business outcomes through an agile, product-driven approach. Their mission revolves around enabling enterprises to achieve digital success by leveraging their unique Digital Business Methodology and crafting customized, innovative digital strategies. Essentially, they aim to help businesses modernize their technology, optimize performance, and build new business models in the digital age.

Role Overview:

We are seeking a highly skilled and experienced L4 Databricks Platform Administrator with deep expertise in managing Databricks environments hosted on AWS. The ideal candidate will be responsible for ensuring the stability, performance, security, and scalability of our Databricks platform. You will play a critical role in supporting our data science and engineering teams by providing expert-level administration, troubleshooting, and optimization of the Databricks platform.

Responsibilities:

• Platform Administration:
• Administer and maintain Databricks workspaces deployed on AWS, including user management, cluster configuration, and resource allocation.
• Monitor platform performance, identify bottlenecks, and implement optimization strategies.
• Manage and troubleshoot Databricks clusters, jobs, and notebooks.
• Implement and enforce best practices for Databricks usage and governance.
• Manage Databricks SQL warehouses and jobs.
• AWS Integration:
• Leverage AWS services (e.g., S3, IAM, EC2, VPC) to ensure seamless integration with the Databricks platform.
• Configure and manage AWS networking and security for Databricks environments.
• Optimize data storage and retrieval using AWS S3 and other relevant services.
• Understand and implement AWS security best practices related to Databricks.
• Security and Compliance:
• Implement and maintain security policies and procedures for the Databricks platform.
• Ensure compliance with relevant industry standards and regulations (e.g., GDPR, HIPAA).
• Manage access control and permissions using Databricks and AWS IAM.
• Conduct regular security audits and vulnerability assessments.
• Troubleshooting and Support:
• Provide expert-level troubleshooting for Databricks platform issues.
• Diagnose and resolve performance, connectivity, and security problems.
• Support data science and engineering teams with platform-related inquiries and requests.
• Create and maintain documentation for platform configurations and procedures.
• Automation and Optimization:
• Develop and implement automation scripts and tools to streamline platform administration tasks.
• Optimize Databricks cluster configurations for performance and cost efficiency.
• Implement infrastructure-as-code (IaC) using tools like Terraform or CloudFormation.
• Implement CI/CD pipelines for Databricks deployments.
• Collaboration:
• Work closely with data engineers, data scientists, and other stakeholders to ensure platform alignment with business needs.
• Collaborate with AWS support and Databricks support to resolve complex technical issues.
• Participate in on-call rotations as needed.

Qualifications:

• Education: Bachelor's degree in Computer Science, Information Technology, or a related field.
• Experience:
• Minimum 10+ years of experience in data platform administration, with a focus on Databricks.
• Extensive experience managing Databricks environments on AWS.
• Strong understanding of AWS services and architecture.
• Proven experience with cluster management, performance tuning, and security best practices.
• Experience with infrastructure-as-code (IaC) tools (e.g., Terraform, CloudFormation).
• Experience with CI/CD pipelines.
• Technical Skills:
• Proficiency in Databricks administration and management.
• Strong knowledge of AWS services (S3, IAM, EC2, VPC, etc.).
• Experience with scripting languages (e.g., Python, Bash).
• Knowledge of SQL and data warehousing concepts.
• Familiarity with data governance and security principles.
• Familiarity with Spark and related technologies.
• Soft Skills:
• Excellent problem-solving and analytical skills.
• Strong communication and collaboration skills.
• Ability to work independently and as part of a team.
• Strong attention to detail and a commitment to quality.
• Ability to manage multiple priorities and meet deadlines.
• Certifications (Preferred):
• AWS Certified Solutions Architect - Associate or Professional.
• Databricks Certified Associate Developer for Apache Spark.
• Databricks Certified Data Engineer Associate.