Infrastructure Engineer

About aion

Aion is the enterprise AI platform, a full-stack solution for building, fine-tuning, and deploying AI at scale. Whether an organization is modernizing internal operations, launching AI-powered products, or transforming customer experiences, Aion takes them from concept to production on a single, unified platform.

We work differently than most AI companies: our teams deploy alongside our customers, turning production-ready AI into real business outcomes in weeks, not quarters.

We’re a fast-growing, VC-backed startup led by founders with a track record of successful exits. With teams across the US, UK, and India, we’re building the next generation of enterprise AI and we’re looking for exceptional people to help us scale.

Who You Are

You are a visionary infrastructure architect passionate about democratizing AI compute at global scale. You thrive on solving complex technical challenges that create elegant, accessible systems from intricate infrastructure. With deep expertise in secure multi-tenancy environments, you understand how to design and implement comprehensive isolation guarantees across hardware, network, and storage layers for both VM and container workloads.

You're excited to join an ambitious AI infrastructure startup at the ground floor, where your work will directly unlock siloed compute resources and remove barriers limiting AI advancement. You have the technical depth to architect platform systems that seamlessly connect compute providers with AI engineers while maintaining robust security foundations that scale to serve diverse client requirements and compliance needs.

You're motivated by the opportunity to build something transformative-creating the infrastructure that will make high-performance compute more accessible, affordable, and user-friendly for the next generation of AI innovation.

What You'll Do

• Observability Systems: Build and deploy comprehensive monitoring for GPU infrastructure using DCGM, NVML, and custom exporters; design metrics collection pipelines that track GPU health, utilization, thermal management, and performance across heterogeneous providers
• LGTM Stack Architecture & Operations: Deploy and manage production-scale Loki, Grafana, Tempo, and Mimir alongside Prometheus and Thanos/VictoriaMetrics; design retention strategies, aggregation rules, and query patterns that scale to thousands of GPUs
• Custom Exporter Development: Write Prometheus exporters in Go or Python for GPU metrics, platform services, and infrastructure components; implement proper metric naming, labeling strategies, and follow OpenMetrics standards
• Kubernetes Controller Development: Build custom controllers and operators for GPU workload management, scheduling, and resource allocation; instrument controllers with comprehensive metrics and tracing for observability
• Training & Inference Monitoring: Design and implement specialized observability for AI training workloads (GPU efficiency, distributed training performance, resource utilization) and inference services (latency percentiles, throughput, cost analytics)
• SLURM Integration & Monitoring: Deploy and manage SLURM clusters for HPC workloads, build observability for batch jobs, create bridges between SLURM and Kubernetes, and design unified monitoring across orchestrators
• Systemd Service Development: Write and deploy systemd services for bare-metal GPU nodes including monitoring agents, metric collectors, and platform daemons; implement proper logging and error handling
• GitOps Platform Engineering: Manage infrastructure using ArgoCD and GitOps workflows, create observable platform abstractions that teams consume declaratively, build self-service capabilities with integrated monitoring
• Alerting & SRE: Design intelligent, actionable alerting systems for GPU failures, thermal throttling, performance degradation, and workload anomalies; define platform SLOs and implement comprehensive monitoring to track reliability
• Multi-Tenant Monitoring Isolation: Build secure observability isolation ensuring customers access only their metrics and logs while maintaining platform-wide visibility for operations; implement query-time filtering and RBAC
• Metrics Pipeline Engineering: Implement automated collection of GPU and platform telemetry; integrate with OpenTelemetry for unified observability; manage cardinality and storage costs through intelligent aggregation
• Cost & Utilization Analytics: Build monitoring pipelines that track GPU utilization, idle time, efficiency metrics, and cost allocation per tenant; create dashboards for platform economics and provider payout calculations
• Dynamic Workload Management: Design systems that use observability data to detect hardware failures, trigger workload migration, and handle graceful degradation; ensure monitoring survives infrastructure changes
• Self-Service Observability Platforms: Build intuitive dashboards, APIs, and alerting interfaces enabling providers to monitor hardware contributions and customers to track workload performance in real-time
• Incident Response & Debugging: Use observability systems to rapidly identify root causes during production issues - GPU hardware failures, network bottlenecks, or workload problems; build tooling that reduces MTTD and MTTR
• Performance Optimization: Leverage observability data to identify bottlenecks in distributed training, inference latency issues, networking inefficiencies, and opportunities for GPU utilization improvement

Requirements

Technical Skills & Experience

• 6-10 years of experience in infrastructure engineering with strong focus on observability, monitoring systems, and production service development (exceptional candidates with different experience profiles will be considered)
• GPU Observability expertise with production experience monitoring NVIDIA GPUs using DCGM, NVML, and nvidia-smi; understanding GPU-specific metrics (utilization, memory, temperature, power, ECC errors) and failure modes
• LGTM Stack proficiency deploying and operating Loki, Grafana, Tempo, and Mimir alongside Prometheus in production; experience with at least one long-term storage solution (Thanos, VictoriaMetrics, or Mimir)
• Systems Programming in Go or Rust for building production services including custom Prometheus exporters, monitoring agents, systemd daemons, and instrumented controllers
• Kubernetes controller development building custom controllers and operators using controller-runtime or client-go; implementing proper instrumentation and observability for custom resources
• Active & Passive Monitoring designing SLO/SLI frameworks, implementing intelligent alerting strategies, building health check systems, and creating anomaly detection for distributed workloads
• HPC Systems experience deploying and managing SLURM clusters, understanding job schedulers, and monitoring batch workloads with observability requirements different from typical web services
• Advanced Kubernetes expertise including custom resource definitions, admission controllers, scheduling extensions, and cluster-wide monitoring architectures
• AI Workload Observability monitoring ML training jobs (GPU efficiency, distributed training metrics, NCCL performance) and inference workloads (latency, throughput, batch processing, cost per inference)
• Metrics Architecture design including cardinality management, aggregation strategies, recording rules, retention policies, and balancing observability costs with data granularity at scale
• GitOps & ArgoCD experience managing observable infrastructure declaratively, building platform abstractions, and creating self-service systems with integrated monitoring
• Systems expertise writing unit files, managing service dependencies, deploying monitoring agents as systemd services, and debugging service failures on bare-metal hosts
• Multi-tenant observability implementing secure metrics and log isolation, RBAC policies for monitoring data, and ensuring proper isolation while maintaining platform-wide visibility
• Networking proficiency with CNI plugins, understanding network observability, and monitoring distributed training communication patterns
• Infrastructure as Code using Terraform or similar tools to deploy observable infrastructure, building monitoring into provisioning workflows

Benefits

Preferred Attributes:

• Founder-level ownership and bias for action.
• Strong strategic thinking and ability to connect technical decisions to business impact.
• Excellent communication and mentoring skills.
• Thrives in ambiguity, fast-paced environments, and early-stage startup culture.

Why Join aion?

• Work directly with high-pedigree founders shaping technical and product strategy.
• Build infrastructure powering the future of AI compute globally.
• Significant ownership and impact with equity reflective of your contributions.
• Competitive compensation, flexible work options, and wellness benefits