{"id":3721573,"url":"https://alion.io/job/shield-ai-senior-staff-data-engineer-databricks-r6508","title":"Senior Staff Data Engineer, Databricks (R6508)","company":{"id":50,"name":"Shield AI","domain":"shield.ai","url":"https://alion.io/company/shield-ai","size_band":"1001-5000","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Lever","truth_index":{"grade":"B","score":74,"open_postings":89,"ghost_share":0,"stale_share":0.854,"repost_share":0,"time_to_fill_p50_days":68,"computed_at":"2026-11-06T05:58:05Z"}},"role":"Data Science","role_family":"Data Science","seniority":"staff","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":{"min":145000,"max":220000,"currency":"USD","period":"year","gross":null,"usd_annual":220000},"salary_estimate":null,"experience_years_min":12,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"CI/CD","optional":false},{"name":"Databricks","optional":false},{"name":"Delta Lake","optional":false},{"name":"Dimensional Modeling","optional":true},{"name":"pySpark","optional":true},{"name":"Python","optional":true},{"name":"Spark","optional":true},{"name":"SQL","optional":true}],"status":"live","first_seen_at":"2026-11-03T22:57:50Z","employer_posted_date":"2026-11-03","last_verified_at":"2026-11-06T12:29:46Z","board_verified":true,"closed_at":null,"days_open":2,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":2},"description":"Role Summary\nThe Senior Staff Data Engineer will help build and operate the enterprise lakehouse on Databricks, creating the governed data foundation that supports multiple business domains and downstream analytics. This is a hands-on role, responsible for scalable ingestion, reliable data processing, and strong technical controls across the Bronze and Silver layers of the medallion architecture.\nThis role is not limited to moving data from point A to point B. The Data Engineer is expected to understand the meaning, sensitivity, classification, and intended use of the data flowing through the pipelines they build, and to apply that understanding when designing controls, access patterns, data quality checks, and segregation boundaries appropriate for a highly regulated environment.\nWhat You'll Do:\nDesign and build ingestion pipelines, batch and streaming where appropriate, from enterprise source systems into the Databricks lakehouse using Delta Lake.\nOwn Bronze-layer ingestion, including raw landing patterns, metadata capture, load traceability, and recoverable ingestion design.\nBuild Silver-layer pipelines for cleansing, standardization, deduplication, conformance, and quality enforcement without embedding unauthorized or undocumented KPI logic.\nDefine and evolve reusable ingestion and transformation patterns, templates, and engineering standards that other domains can adopt as they onboard to the platform.\nImplement and maintain Databricks platform constructs needed for secure delivery, including catalogs, schemas, service principals, job orchestration, and environment-aware deployment patterns.\nBuild and maintain CI/CD pipelines for data platform assets, including ingestion code, transformation logic, workflow definitions, tests, and environment promotion across dev, test, and prod.\nApply data classification, segregation, and handling requirements within the pipeline design, ensuring sensitive and regulated data is processed in accordance with enterprise controls and access policies.\nBuild data quality controls that do more than detect technical failures, including checks that reflect actual business meaning, record integrity, completeness, and expected domain behavior.\nQuarantine, flag, and route problematic records or datasets according to defined quality and compliance rules rather than silently dropping or obscuring issues.\nMaintain documentation for source objects, ingestion logic, applied transformations, data quality rules, and known limitations so downstream teams can trust and use the data correctly.\nPartner with the Analytics Engineer and domain teams to ensure Silver-layer data is reliable, well-governed, and suitable for trusted Gold-layer modeling.\nCollaborate with domain engineering teams to align on ownership boundaries, onboarding patterns, data contracts, and support expectations as new domains are enabled onto the platform.\nPreferred Qualifications:\nDatabricks certification such as Data Engineer Associate or Professional.\nExperience standing up or maturing Unity Catalog structures and access models in an enterprise setting.\nExperience with CI/CD, infrastructure-as-code, and deployment automation for data platforms.\nExperience in defense, aerospace, financial services, healthcare, or another highly regulated environment.\nExperience enabling multiple business domains on a shared data platform while maintaining strong governance and ownership boundaries.\nRequired Qualifications:\n12+ years of data engineering experience, including hands-on ownership of production data pipelines.\nStrong Databricks experience, including Delta Lake, Databricks Workflows or Jobs, and Spark with PySpark and/or Spark SQL.\nWorking knowledge of Unity Catalog, including catalogs, schemas, tables, lineage, and access control concepts.\nExperience with batch, CDC, and/or streaming ingestion patterns and the operational trade-offs associated with each.\nExperience with CI/CD and deployment automation for data pipelines and platform assets, including version control, testing, and controlled promotion across environments.\nStrong SQL skills and solid grounding in data modeling fundamentals, even if dimensional modeling is not the primary responsibility of this role.\nDemonstrated ability to understand the business and regulatory context of the data being processed, not just the mechanics of pipeline development.\nExperience applying data classification, segregation, security, retention, or compliance requirements in data engineering workflows within a regulated or security-sensitive environment.\nAbility to design pipelines with awareness of the actual data domains involved, including sensitivity, ownership, permitted use, and downstream impact.\nComfort operating in a fast-moving platform build where patterns are still being established and engineers are expected to shape standards, not just follow them.","description_format":"text","description_chars":4874,"description_truncated":false,"requirements":{"experience_years_min":12,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"phd","optional":false},"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Military Drones","Defense Manufacturing","Defense AI","Drone AI"],"lifecycle":[{"event":"open","at":"2026-11-04T02:51:31Z"}],"visa":[{"country":"US","licensed_sponsor":true,"evidence":"H-1B filings in 12 months: 2","filings_12m":2,"filings_prev_12m":2,"green_card_filings_12m":0,"median_offered_wage_usd":132500,"route":null,"cap_exempt":false,"checked_at":"2026-10-29T16:27:21+00:00","sources":["US Department of Labor: LCA disclosure data (H-1B, H-1B1, E-3)","US Department of Labor: PERM disclosure data (green cards)"],"filings_for_role_12m":0}],"liveness":{"score":63,"band":"ok","label":"Likely open","p_open":1,"p_active":0.632,"p_room":1,"age_days":2,"expected_fill_days":68,"reasons":["conf:2","stale_co","velocity","win:early","comp:brand"],"computed_at":"2026-11-06T05:58:05Z"},"pay":{"stated_usd_annual":220000,"is_top_pay":true},"html_url":"https://alion.io/job/shield-ai-senior-staff-data-engineer-databricks-r6508","json_url":"https://alion.io/job/shield-ai-senior-staff-data-engineer-databricks-r6508.json","meta":{"generated_at":"2026-11-06T14:18:57Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":26509,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-07T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":50},"rest":"https://alion.io/mcp/rest/get_company?id=50"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fshield-ai-senior-staff-data-engineer-databricks-r6508"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fshield-ai-senior-staff-data-engineer-databricks-r6508"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fshield-ai-senior-staff-data-engineer-databricks-r6508"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/shield-ai-senior-staff-data-engineer-databricks-r6508\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fshield-ai-senior-staff-data-engineer-databricks-r6508"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":3721573}},{"tool":"search_jobs","arguments":{"company":"shield.ai","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":3721573},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}