{"id":1029051,"url":"https://alion.io/job/safe-security-sde-ii-ai-platform","title":"SDE II - AI Platform","company":{"id":48325,"name":"Safe Security","domain":"safe.security","url":"https://alion.io/company/safe-security","size_band":"51-200","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Lever","truth_index":{"grade":"A","score":85,"open_postings":10,"ghost_share":0,"stale_share":0.4,"repost_share":0,"time_to_fill_p50_days":111,"computed_at":"2026-10-14T06:34:16Z"}},"role":"Backend","role_family":"Backend","seniority":"middle","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Bengaluru, India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":2,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Agile","optional":false},{"name":"AI Agents","optional":false},{"name":"Amazon EC2","optional":false},{"name":"Amazon S3","optional":false},{"name":"Amazon SageMaker","optional":false},{"name":"API Gateway","optional":false},{"name":"AWS","optional":false},{"name":"AWS Bedrock","optional":false},{"name":"AWS Lambda","optional":false},{"name":"Azure","optional":false},{"name":"CI/CD","optional":false},{"name":"CUDA","optional":false},{"name":"CUDA Toolkit","optional":false},{"name":"Docker","optional":false},{"name":"GCP","optional":false},{"name":"Go","optional":false},{"name":"Hallucination","optional":false},{"name":"KServe","optional":false},{"name":"Kubernetes","optional":false},{"name":"Linux","optional":false},{"name":"LLM","optional":false},{"name":"LLM Evaluation","optional":false},{"name":"LLMOps","optional":false},{"name":"NCCL","optional":false},{"name":"Platform Engineering","optional":false},{"name":"Python","optional":false},{"name":"Quantization","optional":false},{"name":"Ray","optional":false},{"name":"SQL","optional":false},{"name":"Terraform","optional":false},{"name":"Triton","optional":false},{"name":"Vertex AI","optional":false},{"name":"vLLM","optional":false},{"name":"AWQ","optional":true},{"name":"GPTQ","optional":true},{"name":"Grafana","optional":true},{"name":"Human-in-the-Loop","optional":true},{"name":"KV Cache","optional":true},{"name":"Langfuse","optional":true},{"name":"OpenTelemetry","optional":true},{"name":"Prometheus","optional":true},{"name":"PyTorch","optional":true},{"name":"RAG","optional":true},{"name":"Speculative Decoding","optional":true},{"name":"TypeScript","optional":true}],"status":"live","first_seen_at":"2026-09-18T11:31:28Z","employer_posted_date":"2026-09-18","last_verified_at":"2026-10-15T00:19:16Z","board_verified":true,"closed_at":null,"days_open":26,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":26},"description":"AI is not a side project at Safe - it is the engine behind how we quantify cyber risk for the world's largest enterprises. Agentic services, LLM analytics, and inference workloads run in production, on real customer data, under enterprise security constraints.\nYou will own the platform underneath all of it: GPU clusters, inference serving, model lifecycle, and the developer-facing abstractions on top, so every engineering and AI/ML team at Safe ships AI workloads without rebuilding infrastructure each time.\nThis is a platform and infrastructure engineering role, not an ML research role. We want an engineer who has operated GPUs in production, not only consumed them.\nAI platform is your primary charter. As an SDE II on the platform team, you will also contribute to core platform engineering, multi-tenant microservices, APIs, and cloud architecture, and carry the same review, mentoring, and delivery ownership as every SDE II at Safe.\nWhat You'll Do:\nBuild the AI platform: APIs, SDKs, and self-service workflows so AI/ML engineers deploy, version, evaluate, and monitor workloads without touching raw infrastructure.\n\nOwn GPU infrastructure: Cluster design, provisioning, scheduling, isolation, and utilisation optimisation across shared multi-team demand.\n\nRun inference in production: vLLM, Triton, KServe, or Ray - continuous batching, autoscaling, model routing, and multi-model serving against real latency and throughput SLOs.\n\nOptimize relentlessly: Drive tokens/sec, p99 latency, GPU utilization, and cost per token through quantization, batching strategy, and capacity planning.\n\nEngineer GPU-aware Kubernetes: GPU Operator, device plugins, GPU-aware scheduling, MIG partitioning, and distributed workloads over NCCL.\n\nDebug the hard layer: GPU OOMs, driver and CUDA runtime mismatches, interconnect bottlenecks, throughput regressions.\n\nBuild the MLOps backbone: Model registry and versioning, CI/CD for AI workloads, and safe rollout/rollback.\n\nOwn the evaluation layer: Offline and online eval harnesses, golden datasets, LLM-as-a-judge scoring, accuracy and hallucination metrics, and automated regression gates so no model, prompt, or quantisation change ships without a measured quality verdict.\n\nMake it observable: Response quality and accuracy drift, latency, throughput, GPU utilisation, and per-tenant cost - with SLOs, alerting, and incident response.\n\nSecure and multi-tenant by default: AuthN/AuthZ, secrets, data protection, tenant isolation, and governed access to models and GPUs.\n\nCodify and lead: Terraform for everything, HA and DR designed in, plus architecture direction and mentoring across teams.\n\nContribute beyond AI: Design and build secure, multi-tenant microservices and APIs on AWS, run thorough code reviews, and own feature delivery end-to-end with Product and Design.\n\nWhat We're Looking For:\n2-4 years building and operating production software and infrastructure, with senior ownership of systems end to end.\n\nBachelor's or Master's in Computer Science, Engineering, or equivalent practical experience.\n\nStrong Python and/or Go - you write and review production services, not just scripts and manifests.\n\nDeep Kubernetes and Docker: scheduling, resource management, operators, networking, debugging under load.\n\nStrong Linux, networking, storage, and distributed systems fundamentals.\n\nProduction AWS/Azure/GCP experience - Lambda, API Gateway, EC2, S3, RDS and equivalents - with Terraform as your default way of working.\n\nWorking knowledge of SQL and NoSQL databases, including schema design and performance tuning.\n\nExperience building and operating backend services and APIs in a multi-tenant SaaS product.\n\nLeadership, code review, and mentoring skills, with end-to-end ownership of delivery in an agile environment.\n\nHands-on with a managed ML/AI platform - AWS SageMaker, Bedrock, GCP Vertex AI, or Azure ML - including where it fits and where self-managed infrastructure wins on cost or control.\n\nTrack record on HA, scalability, and DR in multi-tenant environments.\n\nSolid observability and CI/CD practice - metrics, traces, SLOs, automated delivery.\n\nExperience building or operating AI evaluation systems - accuracy and quality measurement, LLM-as-a-judge pipelines, eval datasets, and regression testing for models and prompts.\n\nBackground in platform engineering, infrastructure, SRE, distributed systems, AI infrastructure, or MLOps/LLMOps.\n\nGPU & AI Experience (Required)\nHands-on GPU cluster operations: provisioning, capacity planning, and day-2 ownership.\n\nNVIDIA ecosystem and CUDA - drivers, container runtime, toolkit compatibility, and their failure modes.\n\nGPU scheduling, allocation, isolation, and utilisation optimisation across competing workloads.\n\nKubernetes GPU workloads: GPU Operator, device plugins, GPU-aware scheduling.\n\nGPU troubleshooting and tuning: memory/OOM, driver faults, interconnect and throughput bottlenecks.\n\nLLM inference and model serving in production (vLLM, Triton, KServe, Ray or equivalent) with demonstrated cost and performance gains.\n\nManaged AI platform experience - SageMaker (training jobs, endpoints, inference components) or equivalent - alongside self-managed GPU serving.\n\nNice to Have:\nSageMaker HyperPod, Bedrock, or Vertex AI at scale · DCGM and Prometheus/Grafana for GPU fleets · MIG and GPU partitioning in multi-tenant setups · NCCL and distributed GPU workloads · quantization and inference optimization (FP8/INT8, AWQ/GPTQ, speculative decoding, KV-cache tuning) · LLM observability and tracing (Langfuse, OpenTelemetry for LLMs) · eval frameworks and human-in-the-loop labeling · PyTorch profiling · GPU cluster schedulers and queueing · vector databases, RAG, LLM gateways, agentic systems · TypeScript/Express API development · React or other modern frontends · security or compliance-bound environments.\nWhat Success Looks Like:\nAI teams deploy and roll back models through self-service workflows, with no manual infrastructure work per deployment.\n\nGPU utilisation is measured, forecast, and consistently optimised across the shared fleet.\n\nCost per token falls quarter over quarter, with the numbers on a dashboard.\n\nInference SLOs hold under peak enterprise load, and GPU incidents drop in frequency and time-to-resolve.\n\nNo model, prompt, or optimisation change reaches production without passing automated accuracy and quality evaluation.","description_format":"text","description_chars":6376,"description_truncated":false,"requirements":{"experience_years_min":2,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"bachelor","optional":false},"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Artificial Intelligence","Threat Intelligence","Information Security","Cybersecurity AI"],"lifecycle":[{"event":"open","at":"2026-09-18T13:02:06Z"}],"visa":[],"liveness":{"score":90,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.901,"p_room":1,"age_days":25,"expected_fill_days":111,"reasons":["conf:1","velocity","win:early"],"computed_at":"2026-10-14T06:34:16Z"},"pay":null,"html_url":"https://alion.io/job/safe-security-sde-ii-ai-platform","json_url":"https://alion.io/job/safe-security-sde-ii-ai-platform.json","meta":{"generated_at":"2026-10-15T04:06:27Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":5887,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-16T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":48325},"rest":"https://alion.io/mcp/rest/get_company?id=48325"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fsafe-security-sde-ii-ai-platform"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fsafe-security-sde-ii-ai-platform"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fsafe-security-sde-ii-ai-platform"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/safe-security-sde-ii-ai-platform\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fsafe-security-sde-ii-ai-platform"}]}