{"id":1093322,"url":"https://alion.io/job/brillio-lead-data-engineer-with-databricks-r01570796","title":"Lead Data Engineer with Databricks - R01570796","company":{"id":3063,"name":"Brillio","domain":"brillio.com","url":"https://alion.io/company/brillio","size_band":"501-1000","is_staffing_agency":false,"employer_type":"services","is_intermediary":false,"listed_via":null,"ats_vendor":"Lever","truth_index":{"grade":"B","score":76,"open_postings":43,"ghost_share":0,"stale_share":0.977,"repost_share":0,"time_to_fill_p50_days":28,"computed_at":"2026-10-23T05:54:15Z"}},"role":"Data Science","role_family":"Data Science","seniority":"lead","employment_type":null,"work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Saint Louis, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":{"min":140000,"max":150000,"currency":"USD","period":"year","gross":null,"usd_annual":150000},"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Amazon S3","optional":false},{"name":"AWS","optional":false},{"name":"AWS Lambda","optional":false},{"name":"Azure","optional":false},{"name":"Azure Data Factory","optional":false},{"name":"CI/CD","optional":false},{"name":"Databricks","optional":false},{"name":"Delta Lake","optional":false},{"name":"ETL/ELT","optional":false},{"name":"GitHub","optional":false},{"name":"IAM","optional":false},{"name":"Jenkins","optional":false},{"name":"pySpark","optional":false},{"name":"Python","optional":false},{"name":"Spark","optional":false},{"name":"SQL","optional":false},{"name":"Terraform","optional":false}],"status":"live","first_seen_at":"2026-09-04T11:28:04Z","employer_posted_date":"2026-09-04","last_verified_at":"2026-10-23T06:49:08Z","board_verified":true,"closed_at":null,"days_open":48,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":48},"description":"Lead Data Engineer\nJob requirements\nKey Responsibilities\nDesign, develop, and maintain ETL/ELT pipelines using PySpark and Python in Databricks (including notebooks, jobs, Delta Lake tables, and Unity Catalog for governance).\nImplement medallion architecture (bronze/silver/gold layers) and optimize Spark jobs for performance, cost, scalability, and reliability (handling partitioning, skew, caching, adaptive query execution, etc.).\nWrite efficient SQL queries for data transformation, validation, and analytics within Databricks.\nProvision and manage cloud infrastructure using Terraform (IaC) for Databricks workspaces, clusters, jobs, storage (ADLS/S3), networking, IAM roles/permissions, and related resources on Azure and/or AWS.\nImplement and maintain CI/CD pipelines using Jenkins, GitHub (Actions/Repositories), and branching strategies for automated testing, deployment of notebooks, jobs, Delta Live Tables, and Terraform configurations.\nIntegrate data from diverse sources (databases, APIs, streaming, files) into cloud storage and processing layers.\nEnsure data quality, lineage, security, and compliance (including Delta Lake ACID transactions, schema evolution, time travel, and access controls).\nMonitor pipeline performance, troubleshoot failures, and implement alerting/observability (using Databricks tools, cloud monitoring services, or third-party solutions).\nOptimize cloud costs through auto-scaling clusters, spot instances, job scheduling, and efficient resource usage.\nCollaborate in agile teams, participate in code reviews, and contribute to best practices for data engineering.\nRequired Skills & Experience\nStrong proficiency in Python and PySpark for distributed data processing and ETL.\nAdvanced SQL skills with experience in complex querying, window functions, and optimization.\nHands-on experience with Databricks (clusters, notebooks, Delta Lake, Unity Catalog, Delta Live Tables, workflows/jobs).\nProficiency in Terraform for infrastructure provisioning and management (Databricks resources, cloud storage, IAM, networking).\nExperience with GitHub for version control and collaboration (branching, pull requests, code reviews).\nSolid knowledge of CI/CD practices and tools, particularly Jenkins (pipelines, plugins for Databricks/GitHub/Terraform).\nWorking experience on Azure (Data Lake, Data Factory, Synapse, Key Vault, etc.) and/or AWS (S3, Glue, EMR, IAM, Lambda, etc.).\nUnderstanding of big data concepts, data modeling (star/snowflake, dimensional), and lakehouse principles.\nFamiliarity with performance tuning in Spark/Databricks environments.\nAdditional Information:\nThis role requires presence in St. Louis and attendance at the client office three days a week.\nKnow more about DAE: https://www.brillio.com/services-data-analytics/\nKnow what it’s like to work and grow at Brillio: https://www.brillio.com/join-us/\nEqual Employment Opportunity Declaration\nBrillio is an equal opportunity employer to all, regardless of age, ancestry, colour, disability (mental and physical), exercising the right to family care and medical leave, gender, gender expression, gender identity, genetic information, marital status, medical condition, military or veteran status, national origin, political affiliation, race, religious creed, sex (includes pregnancy, childbirth, breastfeeding, and related medical conditions), and sexual orientation.","description_format":"text","description_chars":3376,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["IT Consulting & Digital Transformation","AI Consulting & Integration","Cloud Consulting & Migration","Data Engineering & Migration Services"],"lifecycle":[{"event":"open","at":"2026-09-21T16:05:30Z"}],"visa":[{"country":"US","licensed_sponsor":true,"evidence":"H-1B filings in 12 months: 201 · green card filings: 28","filings_12m":201,"filings_prev_12m":220,"green_card_filings_12m":28,"median_offered_wage_usd":132413,"route":null,"cap_exempt":false,"checked_at":"2026-10-15T11:24:14+00:00","sources":["US Department of Labor: LCA disclosure data (H-1B, H-1B1, E-3)","US Department of Labor: PERM disclosure data (green cards)"],"filings_for_role_12m":37}],"liveness":{"score":18,"band":"cold","label":"Long shot","p_open":1,"p_active":0.494,"p_room":0.36,"age_days":48,"expected_fill_days":28,"reasons":["conf:3","stale_co","velocity","win:tail","crowd:brand"],"computed_at":"2026-10-23T05:54:15Z"},"pay":{"stated_usd_annual":150000,"is_top_pay":false},"html_url":"https://alion.io/job/brillio-lead-data-engineer-with-databricks-r01570796","json_url":"https://alion.io/job/brillio-lead-data-engineer-with-databricks-r01570796.json","meta":{"generated_at":"2026-10-23T08:08:40Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":12987,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-24T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":3063},"rest":"https://alion.io/mcp/rest/get_company?id=3063"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fbrillio-lead-data-engineer-with-databricks-r01570796"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fbrillio-lead-data-engineer-with-databricks-r01570796"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fbrillio-lead-data-engineer-with-databricks-r01570796"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/brillio-lead-data-engineer-with-databricks-r01570796\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fbrillio-lead-data-engineer-with-databricks-r01570796"}]}