{"id":3165226,"url":"https://alion.io/job/abacus-service-data-engineer","title":"Data Engineer","company":{"id":4443625,"name":"Abacus Service","domain":"abacusservice.com","url":"https://alion.io/company/abacus-service","size_band":null,"is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":null,"truth_index":null},"role":"Data Science","role_family":"Data Science","seniority":null,"employment_type":"contractor","work_mode":"hybrid","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Indianapolis, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":89000,"max_usd":195000,"period":"year","method":"role_country_seniority_unknown","sample_n":3718},"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Azure","optional":false},{"name":"Azure Data Factory","optional":false},{"name":"Azure DevOps","optional":false},{"name":"CI/CD","optional":false},{"name":"Databricks","optional":false},{"name":"Delta Lake","optional":false},{"name":"Dimensional Modeling","optional":false},{"name":"Git","optional":false},{"name":"GitHub","optional":false},{"name":"Microsoft Entra ID","optional":false},{"name":"pySpark","optional":false},{"name":"Spark","optional":false},{"name":"SQL","optional":false},{"name":"Python","optional":true}],"status":"live","first_seen_at":"2026-10-22T21:29:35Z","employer_posted_date":null,"last_verified_at":"2026-10-22T21:29:35Z","board_verified":false,"closed_at":null,"days_open":5,"trust":{"level":"ok","repost_count":1,"flags":[],"days_open":5},"description":"Position: Data Engineer\nLocation: Indianapolis, IN (Hybrid- 1 day in a week onsite)\nDuration: 12+ Months and Extendable\nClient: State of IN (Indiana Department of Health) \n\nMUST BE in INDIANA Location\nDescription: \nThis role supports data integration, pipeline development, data modeling, governance, and documentation while collaborating with cross-functional teams to deliver high‐quality, reliable data products for the agency.\nPlays a key role in implementing the agency's data strategy by developing batch and streaming pipelines, optimizing data warehouse workloads, and enforcing data governance standards. The position assists with requirements gathering, solution reviews, and technical support, and works closely with data architects, scientists, analysts, and project managers to translate business needs into technical designs. This role supports operational processes, contributes to policy and procedure improvements, and may provide mentorship or supervision based on agency needs.\nJob Duties \n\n·Assists with the data technology strategy, including resolvingstrategic data integration issues, such as data quality and stewardship,real-time,\n·Event-based data integration, and crafting a universal vision for data integration\n·Provides direction and oversight for the design and implementation of data solutions for integration and synthesis across multiple secure systems\n·Assists with the definition, implementation, and continued adherence of new data management policies and design standards for data engineering\n·Provides advice, training, and technical support for various projects\n·Participates in project activities such as requirements gathering, solution reviews, and explaining technical complexities to non-technical personnel\n·Work closely with the solution leads, project managers, data architects, data scientists, and data analysts on solution design, architecture, and implementation\n·Assists with extraction, transformation, and loading of data from a wide variety of data sources using various data engineering tools and methods.\n·Designs, builds, and maintains batch and streaming data pipelines in Azure Databricks using Py Spark, Spark SQL, and Delta Lake, following a medallion (bronze/silver/gold) lake house architecture\n·Develops, loads, and tunes dimensional data models and SQL workloads in Azure Synapse Analytics (dedicated and serverless SQL pools) to support reporting and dashboard needs\n·Orchestrates and monitors data workflows using Databricks Workflows/Jobs and Synapse or Azure Data Factory pipelines, including scheduling, alerting, and failure recovery\n·Applies data governance and security controls (e.g., Unity Catalog permissions, role-based access, row- and column-level security, data masking) to protect sensitive health data (PHI/PII) in accordance with HIPAA and agency policy\n·Assists with documenting and demonstrating solutions by developing documentation, flowcharts, layouts, diagrams, charts, code comments and clear code.\n·Assists in operations by conducting systems analysis; recommendingchanges in policies and procedures.\n·The job description is not designed to cover or contain a comprehensive listing of activities, duties or responsibilities that are required of the employee. Other duties, responsibilities and activities may change or be assigned at any time.\n\nSkill\nRequired / Desired\nMaster's degree in data science, analytics, informatics, computer science, mathematics, engineering, or related field; OR....\nRequired 4 Years\nBachelor's degree in a related field plus 2+ years of relevant experience; or 4+ years of hands-on experience in a similar data engineering role; OR..\nRequired 4 Years\nRelevant certifications and coursework may substitute for formal education/experience requirements\nRequired 4 Years\nDeveloping with Databricks notebooks in PySpark, Spark SQL, and R, including transformations, joins, aggregations, and window functions\nRequired 4 Years\nACID transactions, MERGE upserts, schema enforcement/evolution, time travel, and table maintenance (OPTIMIZE, Z-order, VACUUM)\nRequired 4 Years\nWorking with medallion (bronze/silver/gold) architecture and incremental ingestion via Auto Loader, Structured Streaming, or Lakeflow/DLT\nRequired 4 Years\nUsing Unity Catalog for Workspace Catalogs, schemas, permissions, lineage, external locations, and storage credentials\nRequired 4 Years\nRight-sizing clusters and SQL warehouses and diagnose partitioning, shuffle, skew, and caching issues via Spark UI\nRequired 4 Years\nScheduling and managing dependencies with ADF Pipelines, Databricks Workflows (Jobs), and using Git folders (Repos) for source control\nRequired 4 Years\nDesigning MPP warehouse tables, including distribution (hash, round-robin, replicated), columnstore indexes, and partitioning\nRequired 4 Years\nQuerying Parquet, Delta, and CSV files in ADLS Gen2 using OPENROWSET, external tables, and views\nRequired 4 Years\nBuilding star-schema facts, dimensions, stored procedures, and views, including SCD Type 1 and Type 2\nRequired 4 Years\nDeveloping with COPY INTO, PolyBase, and CETAS, and building parameterized Synapse/ADF pipelines (copy, data flows, triggers)\nRequired 4 Years\nWorkload management, resource classes, statistics, materialized views, result-set caching, and pool scaling/pausing\nRequired 4 Years\nADLS Gen2 folder/file-format design (Parquet/Delta), Entra ID access control, managed identities, and Key Vault\nRequired 4 Years\nGit and CI/CD in Azure DevOps or GitHub, including branching, PRs, and deploying Synapse artifacts and Databricks Asset Bundles\nRequired 4 Years\nMonitoring pipelines and workloads with Azure Monitor/Log Analytics and alerting, and troubleshooting failed jobs\nRequired 4 Years\nStrong written and verbal communication skills\nRequired\nUsing Microsoft lakehouse patterns, including interoperability and migration paths between Synapse and Databricks\nRequired 4 Years\nDatabricks Certified Data Engineer Associate (Databricks Certified Data Engineer Professional a plus)\nHighly desired\nMicrosoft Certified: Azure Data Engineer Associate (DP-203) if already held\nHighly desired\nPower BI connectivity to Synapse (Import vs. DirectQuery) and how model design affects dashboard performance\nDesired","description_format":"text","description_chars":6230,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":[],"lifecycle":[{"event":"open","at":"2026-10-23T16:52:30Z"},{"event":"close","at":"2026-10-26T17:01:24Z"},{"event":"reopen","at":"2026-10-28T04:52:14Z"}],"visa":[],"liveness":{"score":85,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.855,"p_room":1,"age_days":5,"expected_fill_days":23,"reasons":["seen:5","win:early"],"computed_at":"2026-10-28T05:59:10Z"},"pay":null,"html_url":"https://alion.io/job/abacus-service-data-engineer","json_url":"https://alion.io/job/abacus-service-data-engineer.json","meta":{"generated_at":"2026-10-28T07:36:27Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":9703,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-29T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":4443625},"rest":"https://alion.io/mcp/rest/get_company?id=4443625"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fabacus-service-data-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fabacus-service-data-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fabacus-service-data-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/abacus-service-data-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fabacus-service-data-engineer"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":3165226}},{"tool":"search_jobs","arguments":{"company":"abacusservice.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":3165226},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}