{"id":1962617,"url":"https://alion.io/job/appnovation-site-reliability-engineer-ai-observability","title":"Site Reliability Engineer, AI Observability","company":{"id":58181,"name":"Appnovation","domain":"appnovation.com","url":"https://alion.io/company/appnovation","size_band":"201-500","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Greenhouse","truth_index":null},"role":"DevOps","role_family":"DevOps","seniority":null,"employment_type":null,"work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["New York, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":116000,"max_usd":260000,"period":"year","method":"role_country_seniority_unknown","sample_n":3941},"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Amazon EKS","optional":false},{"name":"Amazon S3","optional":false},{"name":"ArgoCD","optional":false},{"name":"AWS","optional":false},{"name":"ClickHouse","optional":false},{"name":"Helm","optional":false},{"name":"Kubernetes","optional":false},{"name":"Langfuse","optional":false},{"name":"LLM","optional":false},{"name":"PostgreSQL","optional":false},{"name":"Redis","optional":false},{"name":"Arize Phoenix","optional":true},{"name":"GitHub Actions","optional":true},{"name":"Grafana","optional":true},{"name":"LangSmith","optional":true},{"name":"OpenTelemetry","optional":true}],"status":"live","first_seen_at":"2026-10-06T14:09:49Z","employer_posted_date":"2026-10-06","last_verified_at":"2026-10-15T18:25:13Z","board_verified":true,"closed_at":null,"days_open":9,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":9},"description":"About us\nAppnovation is the AI Product Company. Headquartered in Vancouver, Canada, with teams around the world, Appnovation builds AI products, delivers enterprise AI and digital solutions, and co-invests with clients to bring new AI products to market.\nWe’re looking for a Site Reliability Engineer to keep a shared observability platform for LLM-based applications running for a global life sciences client. The platform is built on Langfuse and self-hosted on Kubernetes on AWS, with ClickHouse as the analytical store, PostgreSQL for metadata and Redis as the ingestion queue, all delivered through Argo CD.\nTwo things need building rather than maintaining. The platform has no monitoring, alerting or defined service levels today, and you will own putting them in place. Infrastructure is defined as code throughout, in Kubernetes manifests, Helm values and Argo CD applications.\nAlongside the platform itself, you will own the runbooks that make an incident survivable by someone other than the author, and the onboarding and support path for the internal teams that depend on the platform.\nROLE RESPONSIBILITIES\nPlatform Operations: Diagnose and resolve failures across ClickHouse, PostgreSQL, Redis, the ingestion workers and the Kubernetes layer beneath them, including ingest backpressure from queue depth and worker drain behaviour.\nClickHouse Operations: Own ClickHouse under the operator model, including Keeper quorum, replication, shard and replica topology, and S3 storage tiering.\nProvable Backup and Restore: Rehearse the restore, time it, document it and test it against its failure modes, rather than assuming a successful backup job means a recoverable system.\nSafe Upgrades: Plan and rehearse upgrades in a lower environment, with a rollback plan that works even if a migration is only partly complete.\nMonitoring and Service Levels: Build monitoring, alerting and service levels from scratch, so problems are found here before a user reports them.\nInfrastructure as Code: Maintain Kubernetes manifests, Helm values and Argo CD applications so every change goes through the delivery pipeline.\nRunbooks and SOPs: Write and maintain runbooks and SOPs that let a colleague resolve an incident without the author present.\nOnboarding and Support: Run the onboarding and support path for internal teams that depend on the platform, and triage what they bring.\nAutomation: Turn recurring operational work into automation.\n Upgrade Partnership: Work with the platform engineer who owns what the platform offers. They decide what to adopt and how it is configured; you own the migration and its rollback.\nQUALIFICATIONS\nHands-on experience with Kubernetes on AWS (managed EKS), with routine work done through Helm values and Argo CD applications.\nHands-on experience running ClickHouse in production, including replication and Keeper quorum, shard and replica topology, and backup and restore, ideally run through an operator.\nExperience building monitoring and alerting from scratch, including service levels that reflect what users actually experience rather than what is easy to measure.\nExperience upgrading self-hosted software safely, including schema migrations, rehearsal in a lower environment and a rollback plan for a partly completed migration.\nPostgreSQL and Redis operations deep enough to debug metadata-store and queue problems, including backpressure and worker drain.\nStrong operational writing: runbooks, SOPs and post-incident reviews that a colleague can follow unaided during an incident.\nPREFERRED QUALIFICATIONS\nObservability engineering, including OpenTelemetry Collector pipelines, alerting design and Grafana dashboards.\nOIDC or enterprise SSO integration with a corporate identity provider.\nGitHub Actions for plan and apply pipelines with approval gates.\nExperience running LLM observability tools such as Langfuse, LangSmith or Arize Phoenix.\nExperience in pharma, life sciences or another regulated industry.\nWHO YOU ARE\nYou don’t trust a backup until you’ve restored from it\nYou want to find problems before users do\nYou write runbooks for the person on call at 3am, not for yourself\nYou stay calm in incidents and focus on fixing the process afterward\nYou automate anything you have to do twice\nYou work well inside a client team and build trust quickly\nYou have prior experience in consulting\nPrior experience and connections in the Life Sciences industry is preferred\nThank you for your interest in a career with Appnovation Technologies! Please note that only those selected for an interview will be contacted.\nAt Appnovation, we recognize that diverse teams are the strongest teams. Diversity, Equity & Inclusion is not only something that we embrace - we celebrate it! We are proud to be an Equal Opportunity Employer and we encourage applicants from all backgrounds, lived experiences and industries to apply. Come join us at Appnovation, and learn more about how we stay true to our company values as we build better lives through better digital. \n\nAccommodations are available upon request throughout the recruitment process.","description_format":"text","description_chars":5080,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":["Equity"],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Biotechnology","Web Design","Web Development Services","Custom Software Development"],"lifecycle":[{"event":"open","at":"2026-10-06T16:12:29Z"}],"visa":[],"liveness":{"score":79,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.833,"p_room":0.945,"age_days":8,"expected_fill_days":24,"reasons":["conf:3","velocity","win:mid"],"computed_at":"2026-10-15T05:46:00Z"},"pay":null,"html_url":"https://alion.io/job/appnovation-site-reliability-engineer-ai-observability","json_url":"https://alion.io/job/appnovation-site-reliability-engineer-ai-observability.json","meta":{"generated_at":"2026-10-15T21:49:09Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":33087,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-16T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":58181},"rest":"https://alion.io/mcp/rest/get_company?id=58181"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fappnovation-site-reliability-engineer-ai-observability"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fappnovation-site-reliability-engineer-ai-observability"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fappnovation-site-reliability-engineer-ai-observability"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/appnovation-site-reliability-engineer-ai-observability\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fappnovation-site-reliability-engineer-ai-observability"}]}