{"id":4384881,"url":"https://alion.io/job/gulf-coast-automation-group-senior-data-engineer-databricks","title":"Senior Data Engineer (Databricks)","company":{"id":4431485,"name":"Gulf Coast Automation Group","domain":"gulfcoastautomationgroup.com","url":"https://alion.io/company/gulf-coast-automation-group","size_band":null,"is_staffing_agency":false,"employer_type":"staffing","is_intermediary":false,"listed_via":null,"ats_vendor":"Manatal","truth_index":null},"role":"Data Science","role_family":"Data Science","seniority":"senior","employment_type":null,"work_mode":"hybrid","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"explicit","locations":["Chicago, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":{"min":150000,"max":164000,"currency":"USD","period":"year","gross":null,"usd_annual":164000},"salary_estimate":null,"experience_years_min":6,"visa_sponsorship":true,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Agile","optional":false},{"name":"AWS","optional":false},{"name":"Azure","optional":false},{"name":"Azure DevOps","optional":false},{"name":"Bicep","optional":false},{"name":"CI/CD","optional":false},{"name":"Databricks","optional":false},{"name":"Delta Lake","optional":false},{"name":"ETL/ELT","optional":false},{"name":"GCP","optional":false},{"name":"Git","optional":false},{"name":"GitHub","optional":false},{"name":"Hadoop","optional":false},{"name":"LLM","optional":false},{"name":"Python","optional":false},{"name":"Spark","optional":false},{"name":"SQL","optional":false},{"name":"Terraform","optional":false}],"status":"live","first_seen_at":"2026-11-16T17:23:40Z","employer_posted_date":"2026-11-16","last_verified_at":"2026-11-17T03:21:25Z","board_verified":true,"closed_at":null,"days_open":0,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":0},"description":"Job Title: Senior Data Engineer (Databricks)\nPrimary Location: Chicago, IL (Hybrid - 3 days onsite / 2 days remote)\nPosition Type: Full-Time, Direct Hire\nOVERVIEW TalentFish is casting a line for a Senior Data Engineer. This is a full-time, direct hire role in Chicago, IL, working a hybrid schedule with a minimum of three days onsite per week. This role exists as part of a multi-year initiative to transform our client's Data & AI function, building a modern data platform that enables speed-to-insight and makes the organization truly data-driven.\nYou'll be one of the core builders of a brand-new Azure Databricks platform, designing and delivering the pipelines, transformations, and models that everything else runs on. The environment is greenfield: Databricks is stood up, but no production pipelines exist yet, so you'll help set the standards from day one. This is a hands-on senior individual contributor role, not a people management, architecture, AI/LLM, or BI/reporting role. The architecture is defined; your job is to build it well and raise the technical bar of a growing team (expanding to 25-30) that is largely BI-focused today.\nThe work carries strong executive sponsorship and enterprise-wide visibility, with the chance to influence engineering practices across the organization. The interview process includes a live technical assessment covering Python, SQL, and practical data pipeline logic. WHAT YOU BRING TO THE ROLE. (IDEAL EXPERIENCE)\nBachelor's degree in Computer Science, Engineering, Data Science, or a related field\n6+ years of hands-on data engineering experience, with a track record of owning pipelines end to end: ingestion, validation, transformation, and governance\nReal, hands-on Databricks experience (Azure preferred; AWS or GCP acceptable), including Delta Lake, Unity Catalog, Databricks SQL, Spark clusters, and Workflows/Jobs\nStrong coding skills in Python, SQL, and Apache Spark that go well beyond basic scripting\nProven experience designing and building scalable ETL/ELT pipelines, including reusable, metadata-driven pipelines\nExperience with object storage, lakehouse engines, orchestration tools, and streaming or CDC pipelines\nSolid data modeling, schema design, and performance tuning experience\nWorking knowledge of Git-based workflows, CI/CD, test automation, and monitoring and alerting\nStrong troubleshooting skills and a solid grasp of engineering fundamentals: scalability, reliability, and maintainability\nThe ability to clearly explain what you built, why it mattered, and how it solved a business or platform problem\nPreferred\nExperience building or migrating a data platform from the ground up\nInfrastructure as Code (Terraform, Bicep, ARM) and Governance as Code experience\nMulti-cloud background with depth in Azure\nLegacy data warehousing and/or Hadoop experience\nDeep Databricks exposure beyond basic pipelines, such as enterprise-scale work and optimization\nA track record of mentoring and upskilling peers\nAgile delivery experience\nWHAT YOU'LL DO. (SKILLS USED IN THIS POSITION)\nLead the design and build of scalable data pipelines and models in Azure Databricks on a brand-new platform with no production code yet\nOwn pipelines end to end, from ingesting source systems to transforming and modeling data to producing outputs for downstream analytics, BI, and AI/ML use cases\nRefactor and migrate multiple legacy data warehouses into the centralized Databricks lakehouse\nWork at the code level every day in Python, SQL, and Spark, building pipelines with Databricks Asset Bundles, Spark workflows, and API-driven jobs\nEstablish engineering standards for CI/CD (GitHub/Azure DevOps), automated data testing, monitoring, and alerting\nHelp shape and enforce standards for data modeling and governance\nTroubleshoot and optimize performance across the platform\nMentor and coach junior engineers, conduct code reviews, and lead by example in design, coding, and testing practices\nPartner with Data Engineering, BI, AI, Finance, and product owners, and help translate technical work into business impact\nCOMPENSATION INFORMATION\nThe expected salary range for this position i s $150,000-$164,000 per year, depending on experience and qualifications. This role also qualifies for comprehensive benefits such as health insurance, 401(k), and paid time off. TalentFish is committed to pay transparency and equal opportunity. The salary range provided is in compliance with applicable state and federal regulations.\nThis role requires authorization to work in the U.S. without current or future visa sponsorship.\nAll offers are contingent upon the completion of a background check, which may include but is not limited to reference checks, education verification, employment verification, drug testing, criminal records checks, and any required certifications or compliance requirements based on the end client's background check policies and applicable laws.\nTalentFish is an employee-owned company pioneering a new realm in talent acquisition. We are redefining IT staffing by evolving AI, video screening, and our unique platform. TalentFish focuses on providing the best employee, consultant, and client experience possible. At TalentFish we are an Equal Opportunity Employer; we embrace and encourage diversity!","description_format":"text","description_chars":5278,"description_truncated":false,"requirements":{"experience_years_min":6,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"bachelor","optional":false},"security_clearance":false,"languages":[]},"benefits":["Health insurance"],"hiring_locations":[{"name":"United States","iso":"US","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["IT Staffing & Staff Augmentation"],"lifecycle":[{"event":"open","at":"2026-11-16T17:23:40Z"}],"visa":[],"liveness":{"score":90,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.903,"p_room":1,"age_days":0,"expected_fill_days":19,"reasons":["conf:2","velocity","win:early"],"computed_at":"2026-11-17T05:47:56Z"},"pay":{"stated_usd_annual":164000,"is_top_pay":true},"html_url":"https://alion.io/job/gulf-coast-automation-group-senior-data-engineer-databricks","json_url":"https://alion.io/job/gulf-coast-automation-group-senior-data-engineer-databricks.json","meta":{"generated_at":"2026-11-17T06:36:13Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":11968,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-18T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":4431485},"rest":"https://alion.io/mcp/rest/get_company?id=4431485"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fgulf-coast-automation-group-senior-data-engineer-databricks"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"free","price":{"credits":0,"usd":0},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fgulf-coast-automation-group-senior-data-engineer-databricks"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fgulf-coast-automation-group-senior-data-engineer-databricks"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/gulf-coast-automation-group-senior-data-engineer-databricks\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fgulf-coast-automation-group-senior-data-engineer-databricks"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":4384881}},{"tool":"search_jobs","arguments":{"company":"gulfcoastautomationgroup.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":4384881},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}