Big Data Platform Engineer

Experience Requirements

• 8+ years of overall IT experience
• 5+ years of experience in Cloud Infrastructure Engineering
• 4+ years of experience working with Kubernetes platforms
• 3+ years of experience supporting enterprise-scale Data Platforms
• 3+ years of experience in DevOps/SRE practices
• 2+ years of experience working with Azure Data Services

Mandatory Skills

Candidates must possess strong hands-on experience in the following areas:

• Apache NiFi
• Azure Event Hub
• Azure Functions
• Enterprise-scale Data Platform Administration These are core and non-negotiable requirements. Profiles without hands-on experience in Apache NiFi and Azure Event Hub will not be considered.

Requirements

1. Azure Kubernetes Service - AKS

Strong Expertise

• AKS cluster deployment and administration
• Cluster upgrades and patch management
• Node pool management
• Ingress Controllers, including NGINX and Application Gateway
• RBAC implementation
• AKS security hardening
• Autoscaling using HPA, VPA, and Cluster Autoscaler
• Persistent storage management
• Troubleshooting cluster issues
• Network policies

Experience

• Production support of AKS environments
• Multi-node and multi-environment AKS setups
• High availability and disaster recovery implementation

2. Kubernetes and Azure Networking

Kubernetes Networking

• Kubernetes networking concepts
• Services, Ingress, and Load Balancers
• DNS resolution
• Pod-to-Pod communication
• Pod-to-Service communication
• Network Policies
• Service Mesh concepts, preferably Istio
• CNI Architecture

Azure Networking

• Virtual Networks
• Subnets
• Private Endpoints
• Peering
• Network Security Groups
• User Defined Routes
• Azure Firewall
• VPN connectivity

3. Docker and Containerization

• Docker image creation and optimization
• Container lifecycle management
• Registry management
• Security scanning
• Multi-stage builds
• Container troubleshooting
• Azure Container Registry
• Image versioning and governance

4. Azure DevOps and CI/CD

CI/CD Expertise

• Azure DevOps Pipelines
• YAML pipelines
• Build and Release pipelines
• Git repositories
• Branching strategies

Infrastructure Deployment

• Infrastructure as Code
• ARM Templates
• Bicep
• Terraform

Automation

• Deployment automation
• Environment promotion
• Release management

5. Monitoring and Observability

Azure Monitor

• Monitoring strategy implementation
• Alert configuration
• Dashboard creation

Log Analytics

• KQL query development
• Performance analysis
• Troubleshooting

Application Monitoring

• Azure Application Insights
• Distributed tracing
• Metrics collection

AKS Monitoring

• Cluster health monitoring
• Pod performance monitoring
• Resource optimization

6. Azure Event Hub Candidates must have strong hands-on experience in:

• Event Hub architecture
• Namespace management
• Consumer groups
• Scaling and throughput units
• Monitoring and troubleshooting
• Security and access control 7. Azure Functions Candidates must have hands-on experience in:
• Deployment and operations support
• Function App management
• Scaling configuration
• Managed Identity integration
• Monitoring and troubleshooting
• CI/CD deployment 8. Apache NiFi Candidates must have strong hands-on experience in:
• NiFi cluster setup and maintenance
• Flow deployment
• Security configuration
• Performance tuning
• Monitoring
• Troubleshooting Preferred
• NiFi Registry
• Cluster scaling
• High availability implementation 9. Infrastructure as Code Must-Have Skills
• Terraform
• Bicep
• ARM Templates

Required Capabilities

• Platform provisioning •Environment consistency •Version-controlled infrastructure Good-to-Have Skills Platform Engineering •GitOps using ArgoCD or Flux •Platform automation •Self-service platform concepts •Internal Developer Platform Observability •Grafana •Prometheus •OpenTelemetry Security •Microsoft Defender for Cloud
• Kubernetes security best practices •Container security •Secrets management •Vulnerability management Data Platform •Azure Data Factory •Synapse Analytics Cloud •Multi-region architecture •Disaster recovery planning •Landing Zone implementation Scripting •PowerShell •Bash •Python

Key Responsibilities

• Manage and support AKS-based Big Data platform infrastructure.
• Administer Apache NiFi, Azure Event Hub, and Azure Functions.
• Implement and maintain CI/CD pipelines using Azure DevOps.
• Monitor platform health using Azure Monitor and Log Analytics.
• Troubleshoot platform, networking, and infrastructure issues.
• Implement infrastructure automation through Terraform and Bicep.
• Ensure platform security, governance, and compliance.
• Collaborate with Data Engineering teams for platform enablement.
• Perform upgrades, patching, capacity planning, and optimization.
• Support production incidents and platform reliability initiatives.