{"id":1277247,"url":"https://alion.io/job/blackline-staff-ii-software-engineer-aiml-ops","title":"Staff II Software Engineer AI/ML Ops","company":{"id":1833518,"name":"BlackLine","domain":"blackline.com","url":"https://alion.io/company/blackline-com","size_band":"1001-5000","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Workday","truth_index":null},"role":"AI/ML","role_family":"AI/ML","seniority":"staff","employment_type":"full_time","work_mode":"hybrid","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Seattle, United States","New York, United States","Los Angeles, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":{"min":245000,"max":307000,"currency":"USD","period":"year","gross":null,"usd_annual":307000},"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"AI Agents","optional":false},{"name":"Airflow","optional":false},{"name":"AWS","optional":false},{"name":"Azure","optional":false},{"name":"CI/CD","optional":false},{"name":"Docker","optional":false},{"name":"Edge AI","optional":false},{"name":"ETL/ELT","optional":false},{"name":"Feature Store","optional":false},{"name":"Fivetran","optional":false},{"name":"GCP","optional":false},{"name":"Google ADK","optional":false},{"name":"Grafana","optional":false},{"name":"Java","optional":false},{"name":"Kubeflow","optional":false},{"name":"Kubernetes","optional":false},{"name":"LangChain","optional":false},{"name":"LangGraph","optional":false},{"name":"LLM","optional":false},{"name":"LLM Guardrails","optional":false},{"name":"Machine Learning","optional":false},{"name":"MLFlow","optional":false},{"name":"Model Context Protocol","optional":false},{"name":"Prometheus","optional":false},{"name":"pySpark","optional":false},{"name":"Python","optional":false},{"name":"PyTorch","optional":false},{"name":"Scala","optional":false},{"name":"Scikit-learn","optional":false},{"name":"TensorFlow","optional":false},{"name":"TPU","optional":false},{"name":"Vertex AI","optional":false},{"name":"Weights & Biases","optional":false},{"name":"Spark","optional":true}],"status":"live","first_seen_at":"2026-07-20T00:00:00Z","employer_posted_date":"2026-07-20","last_verified_at":"2026-10-14T13:36:51Z","board_verified":true,"closed_at":null,"days_open":87,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":87},"description":"Make Your Mark:\nWe’re looking for a Lead Data Engineer to design, build, and optimize data pipelines that power our next-generation AI-driven accounting agents. You’ll lead the development of scalable, high-performance data infrastructure while collaborating closely across teams. Responsibilities: - Lead data pipeline development: Build and maintain PySpark ETL pipelines with high data quality and performance - Manage integrations: Establish robust connections to client data sources via APIs and tools like FiveTran, Plaid, and BlackLine’s own internal connector ecosystem - Ensure reliability: Monitor pipeline performance, automate testing, and validate data accuracy - Optimize for scale: Implement performance improvements (e.g., CDC mechanisms, indexing strategies) for large-scale datasets - Collaborate & innovate: Work with business stakeholders to refine data requirements and integrate cutting-edge AI and big data technologiesYou'll Get To:\nLeadership and Strategy\nPartner with data science, security, and product teams to set evaluation and governance standards (Guardrails, Bias, Drift, Latency SLAs).\nMentor senior engineers and drive design reviews for ML pipelines, model registries, and agentic runtime environments.\nLead incident response and reliability strategies for ML/AI systems. AI System Deployment and Integration:\nCollaborate with development teams to integrate AI solutions into existing workflows and applications.\nEnsure seamless integration with different platforms and technologies.\nDefine and manage MCP Registry for agentic component onboarding, lifecycle versioning, and dependency governance.\nBuild CI/CD pipelines automating LLM agent deployment, policy validation, and prompt evaluation of workflows.\nDevelop and operationalize experimentation frameworks for agent evaluations, scenario regression, and performance analytics.\nImplement logging, metering, and auditing for agent behavior, function calls, and compliance alignment.\nCreate scalable observability systems-tracking conversation outcomes, factual accuracy, latency, escalation patterns, and safety events.\nArchitect end-to-end guardrails for AI agents including prompt injection protection, identity-aware routing, and tool usage authorization.\nCollaborate cross-functionally to standardize authentication, authorization, and session governance for multi-agent runtimes. Model Deployment and Integration:\nArchitect and standardize model registries and feature stores to support version tracking, lineage, and reproducibility across environments.\nLead the deployment of machine learning models into production environments, ensuring scalability, reliability, and efficiency.\nCollaborate with software engineers to integrate machine learning models into existing applications and systems.\nImplement and maintain APIs for model inference. Infrastructure and Environment Management:\nDesign and manage training infrastructure including distributed training orchestration, GPU/TPU resource allocation, and automatic scaling.\nImplement CI/CD for model workflows using pipelines integrated with model validation, bias checks, and rollback automation.\nBuild standardized experimentation frameworks for reproducible training, tuning, and deployment cycles (MLflow, W&B, Kubeflow).\nManage and optimize the infrastructure required for machine learning operations in cloud.\nWork closely with other teams to ensure the availability, security, and performance of machine learning systems. Monitoring and Maintenance:\nImplement robust monitoring solutions for deployed machine learning models to detect issues and ensure performance.\nCollaborate with data scientists and engineers to address and resolve model performance and data quality issues.\nConduct regular system maintenance, updates, and optimizations to ensure optimal performance of machine learning solutions. Automation and Orchestration:\nDevelop and maintain automation scripts and tools for managing machine learning workflows.\nImplement orchestration systems to streamline the end-to-end machine learning lifecycle, from data preparation to model deployment. Collaboration with Data Science Teams:\nCollaborate with data scientists to understand model requirements and constraints for deployment.\nFacilitate the transition of machine learning models from research to production, ensuring scalability and efficiency. Performance Optimization:\nIdentify and implement optimizations to enhance the performance and efficiency of machine learning models in production.\nConduct performance analysis and implement improvements based on resource utilization of metrics. Security and Compliance:\nImplement security measures to protect machine learning systems and data.\nEnsure compliance with regulatory requirements and industry standards related to machine learning and data privacy.\nIntegrate audit controls, metadata storage, and lineage tracking across ML and AI workflows.\nEnsure complete monitoring and feedback loops including event logs, evaluations, and automated retraining triggers.\nEnforce secure deployment patterns with Infrastructure-as-Code and cloud-native secrets management.\nDefine SLAs, error budgets, and compliance reporting mechanisms for ML and AI systems.\nWhat You'll Bring:\nKnowledge: Typically possesses extensive practical experience with consistent, demonstrated success developing effective business solutions/applications for products or services that may effect broad areas of the org | Expert solution builder Competencies: Recognized expert within and outside of the organization Possesses industry expertise as an individual contributor to operations | Sets objectives and delivers results that have an impact within the department or division | Provides advice, counsel and thought leadership within the department | Influencer/architect/orchestrator High level strategic influence | Decisions impact business unit's or department's strategic direction | Anticipates emerging trends | Accountable to 3+ year horizon | Futurist mindset | Expert operator High level of unprecedented work or experience | High degree of autonomy and exercises independent discretion | Accountable for complex, highly strategic duties requiring functional expertise | Develops path through org's most ambiguous endeavors | Developer of innovation or adaptation\nWe’re Even More Excited If You Have:\nEducation and Experience:\nBachelor's or Master's degree in Computer Science, Machine Learning, Data Science, or a related field.\nTechnical Skills:\nStrong programming skills in languages such as Python, Java, or Scala.\nExpertise in ML frameworks (TensorFlow, PyTorch, scikit-learn) and orchestration tools (Airflow, Kubeflow, Vertex AI, MLflow).\nProven experience operating production pipelines for ML and LLM-based systems across cloud ecosystems (GCP, AWS, Azure).\nDeep familiarity with LangChain, LangGraph, ADK or similar agentic system runtime management.\nStrong competencies in CI/CD, IaC, and DevSecOps pipelines integrating testing, compliance, and deployment automation.\nHands-on with observability stacks (Prometheus, Grafana, Newrelic) for model and agent performance tracking.\nUnderstanding of governance frameworks for Responsible AI, auditability, and cost metering across training and inference workloads.\nProficiency in containerization technologies (e.g., Docker, Kubernetes).\nOperations and Infrastructure:\nProficient in scripting languages (e.g., Bash, python) for automation.\nExperience with workflow orchestration tools (e.g., Apache Airflow).\nExpertise in managing and optimizing cloud-based infrastructure.\nFamiliarity with DevOps practices and tools for automated deployment.\nUnderstanding of network configurations and security protocols.\nProblem-solving and Critical Thinking:\nAbility to define problems, collect and analyze data, and propose innovative solutions. Strong critical thinking skills to evaluate models, identify limitations, and\nAdaptability and Learning Agility:\nComfortable working in a fast-paced, rapidly evolving environment. Proactive in staying up to date with the latest trends, techniques, and technologies in AI/data science\nThrive at BlackLine Because You Are Joining:\nA technology-based company with a sense of adventure and a vision for the future. Every door at BlackLine is open. Just bring your brains, your problem-solving skills, and be part of a winning team at the world's most trusted name in Finance Automation!\nA culture that is kind, open, and accepting. It's a place where people can embrace what makes them unique, and the mix of cultural backgrounds and varying interests cultivates diverse thought and perspectives.\nA culture where BlackLiner's continued growth and learning is empowered. BlackLine offers a wide variety of professional development seminars and inclusive affinity groups to celebrate and support our diversity.\nBlackLine is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to sex, gender identity or expression, race, ethnicity, age, religious creed, national origin, physical or mental disability, ancestry, color, marital status, sexual orientation, military or veteran status, status as a victim of domestic violence, sexual assault or stalking, medical condition, genetic information, or any other protected class or category recognized by applicable equal employment opportunity or other similar laws.\nBlackLine recognizes that the ways we work and the workplace itself has shifted. We innovate in a workplace that optimizes a combination of virtual and in-person interactions to maximize collaboration and nurture our culture. Candidates who live within a reasonable commute to one of our offices will work in the office at least 3 days a week.\nSalary Range:\n$245,000.00-$307,000.00Pay Transparency Statement:\nPlacement within this range depends upon several factors, including the applicant's prior relevant job experience, skill set, and geographic location.\nIn addition to base pay, BlackLine also offers short-term and long-term incentive programs, based on eligibility, along with a robust offering of benefit and wellness plans.\nWe are committed to pay transparency and ensuring candidates have clear information about compensation expectations. For roles that include variable incentive components such as an Incentive Compensation Plan (ICP) or On-Target Earnings (OTE), the compensation structure may follow a split model - for example, a 50/50, 70/30, or 60/40 ratio between base salary and variable incentive.\nBlackLine is committed to creating an inclusive and accessible experience for all candidates. If you require a reasonable accommodation that would better enable your success during the application or interview process, please complete this form.\nAccommodations:\nBlackLine is committed to creating an inclusive and accessible experience for all candidates. If you require a reasonable accommodation that would better enable your success during the application or interview process, please complete this form.","description_format":"text","description_chars":11038,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"bachelor","optional":false},"security_clearance":false,"languages":[]},"benefits":["Professional development"],"hiring_locations":[{"name":"United States","iso":"US","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":[],"lifecycle":[{"event":"open","at":"2026-09-26T02:06:21Z"}],"visa":[],"liveness":{"score":27,"band":"fade","label":"Fading","p_open":1,"p_active":0.747,"p_room":0.36,"age_days":87,"expected_fill_days":57,"reasons":["conf:16","velocity","win:tail","crowd:brand"],"computed_at":"2026-10-15T05:46:00Z"},"pay":{"stated_usd_annual":307000,"is_top_pay":true},"html_url":"https://alion.io/job/blackline-staff-ii-software-engineer-aiml-ops","json_url":"https://alion.io/job/blackline-staff-ii-software-engineer-aiml-ops.json","meta":{"generated_at":"2026-10-15T06:10:32Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":9284,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-16T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":1833518},"rest":"https://alion.io/mcp/rest/get_company?id=1833518"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fblackline-staff-ii-software-engineer-aiml-ops"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fblackline-staff-ii-software-engineer-aiml-ops"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fblackline-staff-ii-software-engineer-aiml-ops"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/blackline-staff-ii-software-engineer-aiml-ops\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fblackline-staff-ii-software-engineer-aiml-ops"}]}