{"id":720852,"url":"https://alion.io/job/samsungsemiconductor-technical-director-large-scale-ai-model-inferencing","title":"Technical Director, Large-Scale AI Model Inferencing","company":{"id":684996,"name":"Samsung Semiconductor","domain":"samsungsemiconductor.com","url":"https://alion.io/company/samsungsemiconductor","size_band":"501-1000","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Greenhouse","truth_index":{"grade":"B","score":84,"open_postings":7,"ghost_share":0,"stale_share":0.429,"repost_share":0,"time_to_fill_p50_days":70,"computed_at":"2026-11-12T10:38:37Z"}},"role":"Leadership","role_family":"Leadership","seniority":"head","employment_type":null,"work_mode":"hybrid","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["San Jose, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":{"min":219000,"max":351000,"currency":"USD","period":"year","gross":null,"usd_annual":351000},"salary_estimate":null,"experience_years_min":12,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Cognos","optional":false},{"name":"CUDA","optional":false},{"name":"DeepSeek","optional":false},{"name":"KV Cache","optional":false},{"name":"Llama","optional":false},{"name":"llama.cpp","optional":false},{"name":"LLM","optional":false},{"name":"Mamba","optional":false},{"name":"Mixture of Experts","optional":false},{"name":"Multimodal AI","optional":false},{"name":"PyTorch","optional":false},{"name":"Qwen","optional":false},{"name":"RAG","optional":false},{"name":"SGLang","optional":false},{"name":"SLI/SLO/SLA","optional":false},{"name":"Speculative Decoding","optional":false},{"name":"TensorRT-LLM","optional":false},{"name":"TPOT","optional":false},{"name":"Transformers","optional":false},{"name":"vLLM","optional":false},{"name":"TensorRT","optional":true}],"status":"live","first_seen_at":"2026-09-10T00:13:35Z","employer_posted_date":"2026-10-16","last_verified_at":"2026-11-13T12:36:31Z","board_verified":true,"closed_at":null,"days_open":64,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":64},"description":"Please Note:\nTo provide the best candidate experience amidst our high application volumes, each candidate is limited to 10 applications across all open jobs within a 6-month period. \nAdvancing the World’s Technology Together\nOur technology solutions power the tools you use every day--including smartphones, electric vehicles, hyperscale data centers, IoT devices, and so much more. Here, you’ll have an opportunity to be part of a global leader whose innovative designs are pushing the boundaries of what’s possible and powering the future. \nWe believe innovation and growth are driven by an inclusive culture and a diverse workforce. We’re dedicated to empowering people to be their true selves. Together, we’re building a better tomorrow for our employees, customers, partners, and communities.\n What You’ll Do\nInference is becoming a memory-bandwidth business. As models scale past what any single GPU can hold - KV caches grow with context, MoE expert weights spill beyond HBM, and new architectures change the rules of what \"model state\" even means - the winners will be the companies that treat memory as the core product of AI inference, not an afterthought.\nWe are looking for a Hands-on Principal Engineer who combines deep, first-principles knowledge of AI model architectures (dense Transformers, Mixture-of-Experts, State Space Models, and hybrids) with production-scale inference expertise, to own the requirement for full-stack AI memory solutions at scale - spanning GPU HBM, host DRAM, CXL-attached memory pools, and NVMe/SSD tiers and Samsung Cognos, AI memory software that moves model state intelligently across them.\nThis person will be the technical authority who connects model behavior to memory-system design: someone who can explain why an MoE router's activation pattern dictates an LRU expert cache policy, why a Mamba state cache breaks the assumptions of PagedAttention, and why disaggregated prefill/decode changes the required memory bandwidth per token by an order of magnitude - and then build the products that exploit those facts.\nLevel: Principal Engineer \nTeam: Memory Solutions Lab / Data Fabric Solutions\nReports to: Chief Technologist, Memory Solutions Lab\nLocation: Daily onsite presence preferred at our San Jose office/headquarters in alignment with our Flexible Work policy; remote/hybrid option available. \nJob ID: 43027\nModel Architecture Expertise - The Foundation\nServe as expert on how different model families consume and move memory, and translate that into memory-product requirements:Dense Transformers: MHA/MQA/GQA/MLA attention, KV-cache growth characteristics, long-context behaviors, attention sinks and prefix locality.\nMixture-of-Experts: routed vs. shared experts, expert-parallel execution, routing skew and hot-expert locality, expert-weight offloading and cache-admission policies, per-token weight-read economics.\n State Space Models (Mamba/Mamba-2) and hybrid SSM-attention architectures: recurrent state vs. KV cache semantics, state size per sequence and per layer, cache-swapping behavior for context switching and batching, and what \"cache-aware scheduling\" means when the state is a fixed-size tensor instead of a token-indexed table.\n Emerging architectures: linear attention, sliding-window/hybrid layers, diffusion and multimodal transformers - and how each changes the memory hierarchy math.\n\nModel the memory footprint, bandwidth demand, and access patterns of frontier open-weight models (e.g., Llama/Qwen-class dense, DeepSeek/Kimi-class MoE, Jamba-class hybrids) and publish internal reference architectures for each.\nTrack the model landscape as a roadmap input: anticipate what coming architectures (longer contexts, agentic multi-session reuse, reasoning-loop workloads, speculative decoding drafts) will demand from memory systems 12-24 months out.\nLarge-Scale Inference Expertise\nOwn deep expertise in production inference stacks - SGLang (HiCache), vLLM (PagedAttention, LMCache integration), NVIDIA Dynamo, TensorRT-LLM, llama.cpp-class engines - including their memory-management internals, not just their flags.\nDrive inference performance engineering: continuous batching, chunked prefill, disaggregated prefill/decode, prefix and radix caching, speculative decoding, CUDA Graphs, and their interactions with memory tiering.\nOwn the latency/throughput/cost envelope: TTFT and TBT/TPOT SLOs, tokens-per-second per dollar, GPU memory utilization as the binding constraint, and the tradeoff curves between cache hit rate, memory capacity, and bandwidth.\nDefine benchmarking and characterization methodology: realistic agentic and long-context workloads (multi-turn reuse, session persistence, RAG prefixes), KV-cache reuse-rate measurement, and bandwidth-latency profiling across the full hierarchy (Nsight, PyTorch Profiler, vendor memory tools).\nFull-Stack AI Memory Solutions - The Core Mandate\nDefine engineering requirements, with proof, for tiered memory systems for inference at fleet scale: HBM as L1, host DRAM (pinned, NUMA-aware pools) as L2, CXL-attached memory pools as an elastic tier, and NVMe/SSD as capacity tier - with the policies (admission, eviction, prefetch, placement) that make the hierarchy behave like one memory.\nDesign expert-weight offloading solutions for MoE serving: host-resident expert pools, GPU-resident expert caches with bandwidth-adaptive fill/evict policies, and CPU/CXL-execution hybrid paths - informed by the routing statistics of real models.\nTranslate model knowledge into product: write the requirements, reference architectures, and performance models that guide memory hardware and firmware roadmaps (HBM capacity/bandwidth, CXL device behavior, SSD QoS for cache tiers), and validate with end-to-end prototypes on real inference workloads.\nDevelop and Deliver POCs: demos and published benchmarks showing inference TCO improvement from the memory stack - e.g., context capacity multiplied at constant GPU count, or cost-per-token reduced through cache-hit-rate gains - credible to both CTOs and PhD researchers.\nTechnical Leadership\nSet multi-year technical strategy for AI memory solutions; own build-vs-adopt-vs-contribute decisions across the open-source inference and caching ecosystem (vLLM, SGLang, LMCache, Cognos-style KV stores) and drive upstream contributions where strategic.\nLead architecture reviews and deep-dive design sessions; write the documents that become the company's standard for how we talk about memory for AI.\nRepresent the company with customers and partners at the deepest technical level: serve as the expert voice in CTO-to-CTO conversations, design wins, and standards discussions.\nMentor senior engineers and grow a bench of architecture talent across the model-to-memory boundary.\nWhat You Bring\nBS in Computer/Electrical/Electronic Engineering or Computer Science, and 10+ years of relevant experience MS in Computer/Electrical/Electronic Engineering or Computer Science with 8 years of relevant experience preferred.\n12+ years in systems engineering, with 4+ years hands-on in large-scale LLM inference or GPU systems performance - you have personally profiled, diagnosed, and fixed memory bottlenecks in production serving, not just read about them.\nFirst-principles understanding of transformer-class model internals: you can derive KV-cache size formulas from attention math, explain MQA/GQA/MLA tradeoffs, and reason about activation-memory peaks during prefill.\nWorking expertise with MoE model behavior: routing, expert parallelism, load skew, and the weight-memory economics of serving models larger than GPU capacity.\nDirect experience with at least one major inference stack's memory-management internals (vLLM PagedAttention/block manager, SGLang HiCache/token pools, TensorRT-LLM KV manager, or llama.cpp compute buffers) - code-level, not configuration-level.\nStrong performance-engineering skills: bandwidth-bound vs. compute-bound analysis, NUMA and PCIe topology reasoning, RDMA basics, and fluency with GPU/CPU profilers.\nTrack record of building systems software at the memory/storage/IO layer - caches, tiering, paging, or storage engines - with production deployments.\nAbility to write models and simulators, not just measure: analytical queueing, cache-hit-rate, and bandwidth models that predict system behavior before hardware exists.\nExcellent written and verbal communication, including executive-level technical narrative; comfort being the technical face of the company in front of customers.\nPreferred\nExperience with State Space Model or hybrid SSM-attention serving (Mamba-class state management, cache swapping for recurrent models) - rare and highly valued.\nContributions to open-source inference/caching projects (vLLM, SGLang, LMCache, HiCache, Mooncake, KTransformers, llama.cpp).\nExperience with CXL memory pooling, CXL-attached tiering, or near-memory processing in real deployments or serious prototypes.\nExperience with SSD/NVMe as a KV or expert cache tier, including QoS engineering for inference-grade latency.\nBackground in memory/storage product companies bringing hardware-software co-designed solutions to market.\nYou’re inclusive, adapting your style to the situation and diverse global norms of our people. \nAn avid learner, you approach challenges with curiosity and resilience, seeking data to help build understanding. \nYou’re collaborative, building relationships, humbly offering support and openly welcoming approaches. \nInnovative and creative, you proactively explore new ideas and adapt quickly to change.\nWhat We Offer\n The pay range below is for all roles at this level across all US locations and functions. Pay within this range varies by work location and may also depend on job-related knowledge, skills, and experience. We also offer incentive opportunities that reward employees based on individual and company performance. \nThis is in addition to our diverse package of benefits centered around the wellbeing of our employees and their loved ones. In addition to the usual Medical/Dental/Vision/401k, our inclusive rewards plan empowers our people to care for their whole selves. An investment in your future is an investment in ours.\nGive Back With a charitable giving match and frequent opportunities to get involved, we take an active role in supporting the community.\nEnjoy Time Away You’ll start with 4+ weeks of paid time off a year, plus holidays and sick leave, to rest and recharge.\nCare for Family Whatever family means to you, we want to support you along the way-including a stipend for fertility care or adoption, caregiver support, and medical travel support.\nPrioritize Emotional Wellness With on-demand apps and free confidential therapy sessions, you’ll have support no matter where you are.\nStay Fit Eating well and being active are important parts of a healthy life. Our onsite Café and gym, plus virtual classes, make it easier.\nEmbrace Flexibility Benefits are best when you have the space to use them. That’s why we facilitate a flexible environment so you can find the right balance for you.\nBase Pay Range\n$219,000—$351,000 USD\nEqual Opportunity Employment Policy \nSamsung Semiconductor takes pride in being an equal opportunity workplace dedicated to fostering an environment where all individuals feel valued and empowered to excel, regardless of race, religion, color, age, disability, sex, gender identity, sexual orientation, ancestry, genetic information, marital status, national origin, political affiliation, or veteran status.\nWhen selecting team members, we prioritize talent and qualities such as humility, kindness, and dedication. We extend comprehensive accommodations throughout our recruiting processes for candidates with disabilities, long-term conditions, neurodivergent individuals, or those requiring pregnancy-related support. All candidates scheduled for an interview will receive guidance on requesting accommodations.\nOur Commitment to Innovation and Fairness\nAt Samsung Semiconductor, we use Artificial Intellige...","description_format":"text","description_chars":13380,"description_truncated":true,"requirements":{"experience_years_min":12,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"bachelor","optional":false},"security_clearance":false,"languages":[]},"benefits":["401k plan","Flexible schedule"],"hiring_locations":[{"name":"United States","iso":"US","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Processors, MCUs & AI Chips","Memory & Storage Chips"],"lifecycle":[{"event":"open","at":"2026-09-11T07:11:29Z"}],"visa":[{"country":"US","licensed_sponsor":true,"adverts_offering_sponsorship_24m":0,"sanctioned_now":false,"evidence":"H-1B filings in 12 months: 147 · green card filings: 45","filings_12m":147,"filings_prev_12m":123,"green_card_filings_12m":45,"median_offered_wage_usd":168672,"route":null,"cap_exempt":false,"checked_at":"2026-11-12T07:36:15+00:00","sources":["US Department of Labor: LCA disclosure data (H-1B, H-1B1, E-3)","US Department of Labor: PERM disclosure data (green cards)"],"filings_for_role_12m":5}],"liveness":{"score":54,"band":"ok","label":"Likely open","p_open":1,"p_active":0.903,"p_room":0.6,"age_days":63,"expected_fill_days":70,"reasons":["conf:0","velocity","win:late","crowd:"],"computed_at":"2026-11-12T10:38:37Z"},"pay":{"stated_usd_annual":351000,"is_top_pay":true},"html_url":"https://alion.io/job/samsungsemiconductor-technical-director-large-scale-ai-model-inferencing","json_url":"https://alion.io/job/samsungsemiconductor-technical-director-large-scale-ai-model-inferencing.json","meta":{"generated_at":"2026-11-13T17:48:57Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":35602,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-14T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":684996},"rest":"https://alion.io/mcp/rest/get_company?id=684996"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fsamsungsemiconductor-technical-director-large-scale-ai-model-inferencing"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"free","price":{"credits":0,"usd":0},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fsamsungsemiconductor-technical-director-large-scale-ai-model-inferencing"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fsamsungsemiconductor-technical-director-large-scale-ai-model-inferencing"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/samsungsemiconductor-technical-director-large-scale-ai-model-inferencing\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fsamsungsemiconductor-technical-director-large-scale-ai-model-inferencing"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":720852}},{"tool":"search_jobs","arguments":{"company":"samsungsemiconductor.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":720852},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}