{"id":603622,"url":"https://alion.io/job/bybit-senior-principal-site-reliability-engineer","title":"Senior Principal Site Reliability Engineer","company":{"id":47455,"name":"Bybit","domain":"bybit.com","url":"https://alion.io/company/bybit","size_band":"501-1000","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Greenhouse","truth_index":{"grade":"A","score":93,"open_postings":22,"ghost_share":0,"stale_share":0.273,"repost_share":0,"time_to_fill_p50_days":53,"computed_at":"2026-10-14T06:34:16Z"}},"role":"DevOps","role_family":"DevOps","seniority":"lead","employment_type":null,"work_mode":"hybrid","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Kuala Lumpur, Malaysia"],"countries":["MY"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":20000,"max_usd":50000,"period":"year","method":"global_role_cell_scaled_by_country","sample_n":830},"experience_years_min":8,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Amazon EC2","optional":false},{"name":"Chaos Engineering","optional":false},{"name":"Go","optional":false},{"name":"Kubernetes","optional":false},{"name":"SLI/SLO/SLA","optional":false},{"name":"Amazon EKS","optional":true},{"name":"AWS","optional":true},{"name":"Error Budget","optional":true},{"name":"Grafana","optional":true},{"name":"OpenTelemetry","optional":true},{"name":"Prometheus","optional":true},{"name":"Self-Healing","optional":true},{"name":"Thanos","optional":true}],"status":"live","first_seen_at":"2026-08-07T13:05:24Z","employer_posted_date":"2026-08-31","last_verified_at":"2026-10-14T16:24:12Z","board_verified":true,"closed_at":null,"days_open":68,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":68},"description":"About Us\nEstablished in 2018, Bybit is one of the world’s leading cryptocurrency exchanges and digital financial platforms, serving over 80 million users across more than 200 countries and regions. Powered by world-class technology and a user-first mindset, Bybit delivers a seamless ecosystem across trading, payments, wealth management, custody, institutional services, and Web3 - connecting users to the future of digital finance.\nOur core values define how we build. We listen, care and improve to create products and experiences that put users first. Backed by a global team of ambitious builders, problem-solvers, and innovators, we foster a high-performance and fast-moving environment where talent is empowered to drive real impact at the global scale. Supported by 24/7 multilingual customer service and a strong commitment to innovation, we are shaping the future of finance through technology, collaboration, and bold execution.\nToday, Bybit is recognized as one of the most trusted and transparent platforms in the digital asset industry, continuing to expand its global presence while building the infrastructure for the next generation of financial services.\nCore Responsibilities\nChaos Engineering Platform Architecture & Development (50%)\nDesign and build an enterprise-grade chaos engineering platform supporting multi-cluster (K8s + EC2 hybrid), multi-region, and multi-environment (testnet/mainnet) deployments\nCore capability development:\nFault Injection Engine: Pod-level / Node-level / AZ-level fault simulation, network latency / packet loss / partition, dependency timeout / error injection\nProduction Safety Assurance: Blast radius control, one-click Kill Switch, automatic rollback, real-time impact monitoring\nTraffic Isolation: Experiment traffic tagging and isolation to ensure fault injection does not impact real users\nFault Isolation: Precise impact scoping at service / cluster / AZ granularity\nDesign experiment orchestration capabilities supporting complex fault scenario composition (e.g., simultaneous network latency + downstream timeout + cache invalidation)\nDeep integration with existing monitoring, alerting, and SLO systems to achieve an automated closed loop: inject fault → observe impact → determine pass/fail\nProduction Resilience Validation Framework (30%)\nDefine safety standards and approval workflows for mainnet fault injection\nDesign and drive routine chaos experiments:\nDaily patrol-level experiments: Low-risk experiments executed automatically on a daily/weekly basis\nPeriodic validation experiments: Monthly/quarterly resilience verification of critical paths\nLarge-scale drills: Cross-AZ / cross-region disaster recovery failover validation\nEstablish a resilience scoring system to quantify system health based on experiment results\nDeliver improvement recommendations and drive business teams to remediate identified weaknesses\n3. Technology Selection & Team Enablement (20%)\nEvaluate and select the technology foundation (Chaos Mesh / Litmus / custom components - hybrid strategy)\nDevelop chaos engineering best practices and playbooks to enable SRE teams and application developers\nMentor and grow the team (2-3 engineers) in chaos engineering capabilities\nStay current with industry developments and introduce cutting-edge practices (e.g., AI-driven fault scenario discovery)\n──────\nRequirements\nMust-Have:\n8+ years of backend / infrastructure engineering experience, with 3+ years dedicated to chaos engineering or stability engineering\nHands-on experience with large-scale fault injection in production environments (not just test environments), with deep understanding of production safety constraints\nExpert-level proficiency in Kubernetes fault injection (Chaos Mesh / Litmus / custom solutions), familiar with CRD / Operator development\nProficient in at least one backend language (Go preferred), with platform-level system architecture design capability\nDeep understanding of distributed system failure modes (network partitions, split-brain, cascading failures, data inconsistency, etc.)\nFamiliarity with observability tech stack (Prometheus / Grafana / Thanos / OpenTelemetry)\nExcellent technical documentation and solution design skills\nNice-to-Have:\nExperience in financial / trading system stability (understanding of transaction consistency and fund safety constraints)\nExperience building SLO / Error Budget frameworks\nExperience building automated fault recovery (self-healing) systems\nFamiliarity with AWS infrastructure (EC2 / EKS / Multi-AZ / Multi-Region)\nKnowledge of Netflix Chaos Engineering / AWS Fault Injection Simulator / Gremlin\nOpen-source community contributions (Chaos Mesh / Litmus or similar projects)\nSoft Skills:\nAbility to balance \"safety\" and \"validation depth\" - not afraid of production injection, while maintaining strict risk control\nStrong cross-team collaboration and influence - chaos engineering requires buy-in from business teams; this role demands persuasion skills\nSelf-driven, capable of independently planning and executing in ambiguous situations\nWhy Join Us\nAt Bybit, we are committed to fostering a supportive and enriching work environment. \nOur benefits include:\n- Study Growth Fund: We support your professional development and continuous learning.\n- Internal Events: Participate in regular team-building activities, workshops, and events designed to promote collaboration and innovation.\n- Global Collaboration: Be part of a diverse, international team, working alongside colleagues from around the world.\n- Career Advancement: Access opportunities for growth and advancement within a rapidly expanding global company.\n- Internal Mobility: Grow with us- Your long-term development is important to us. We offer internal job opportunities to help build your career path.","description_format":"text","description_chars":5787,"description_truncated":false,"requirements":{"experience_years_min":8,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":["Continuous learning","Professional development"],"hiring_locations":[{"name":"Malaysia","iso":"MY","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Cryptocurrencies","Crypto Exchanges"],"lifecycle":[{"event":"open","at":"2026-09-10T17:51:37Z"}],"visa":[],"liveness":{"score":32,"band":"fade","label":"Fading","p_open":1,"p_active":0.738,"p_room":0.44,"age_days":67,"expected_fill_days":53,"reasons":["conf:2","velocity","win:tail","crowd:brand"],"computed_at":"2026-10-14T06:34:16Z"},"pay":null,"html_url":"https://alion.io/job/bybit-senior-principal-site-reliability-engineer","json_url":"https://alion.io/job/bybit-senior-principal-site-reliability-engineer.json","meta":{"generated_at":"2026-10-14T18:44:45Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":24305,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-15T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":47455},"rest":"https://alion.io/mcp/rest/get_company?id=47455"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fbybit-senior-principal-site-reliability-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fbybit-senior-principal-site-reliability-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fbybit-senior-principal-site-reliability-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/bybit-senior-principal-site-reliability-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fbybit-senior-principal-site-reliability-engineer"}]}