{"id":3973645,"url":"https://alion.io/job/digicomplish-data-engineering-tech-lead","title":"Data Engineering Tech Lead","company":{"id":3801214,"name":"Digicomplish","domain":"digicomplish.com","url":"https://alion.io/company/digicomplish","size_band":null,"is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":null,"truth_index":null},"role":"Leadership","role_family":"Leadership","seniority":"lead","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Indianapolis, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Azure","optional":false},{"name":"Azure Data Factory","optional":false},{"name":"Azure DevOps","optional":false},{"name":"Azure SQL Database","optional":false},{"name":"Bicep","optional":false},{"name":"CI/CD","optional":false},{"name":"Databricks","optional":false},{"name":"Git","optional":false},{"name":"MS SQL","optional":false},{"name":"Poetry","optional":false},{"name":"pylint","optional":false},{"name":"pySpark","optional":false},{"name":"Pytest","optional":false},{"name":"Python","optional":false},{"name":"SonarQube","optional":false},{"name":"Spark","optional":false},{"name":"SQL","optional":false},{"name":"Terraform","optional":false}],"status":"live","first_seen_at":"2026-11-05T13:31:26Z","employer_posted_date":null,"last_verified_at":"2026-11-05T13:31:26Z","board_verified":false,"closed_at":null,"days_open":4,"trust":{"level":"not_scored","repost_count":null,"flags":[],"days_open":4},"description":"Job Description:\nWe are looking for an experienced Data Engineering Tech Lead with strong expertise in Azure Databricks, PySpark, Azure Data Factory (ADF), ADLS Gen2, Azure SQL, Python, and Azure Cloud. The role will lead the design, development, optimization, and maintenance of scalable enterprise data engineering and Lakehouse solutions, including batch and near-real-time data processing.\n4+ years of experience in Azure Databricks with PySpark.\n2+ years of experience in Databricks workflow & Unity catalog.\n3+ years of experience in ADF (Azure Data Factory).\n3+ years of experience in ADLS Gen 2.\n3+ years of experience in Azure SQL.\n5+ years of experience in Azure Cloud platform.\n2+ years of experience in Python programming & package builds.\nThe Azure Data Engineering platform is a critical backbone for scalable, secure, and reliable enterprise data processing and analytics.\n\nThis role ensures:\nReliable and scalable data pipelines across enterprise data sources\nHigh-performance data processing using Databricks, PySpark, and ADF\nStrong data governance, security, and quality across the Lakehouse\nContinuous technical leadership, optimization, and modernization of the data platform\"\n\nKey technical skills:\nData management experience handling Analytics workload covering design, development, and maintenance of lakehouse solutions sourcing data from platforms such as ERP sources, API sources, Relational stores, NoSQL and on-prem sources using Databricks/PySpark as distributed /big data management service, supporting batch and near-real-time ingestion, transformation, and processing.\nAbility to optimize Spark jobs and manage large-scale data processing using RDD/DataFrame APIs.\nDemonstrated expertise in partitioning strategies, file format optimization (Parquet/Delta), and Spark SQL tuning.\nFamiliarity with Databricks runtime versions, cluster policies, libraries, and workspace management.\nSkilled in governing and manage data access for Azure Data lakehouse with Unity Catalog. \nExperience in configuring data permissions, object lineage, and access policies with Unity Catalog.\nUnderstanding of integrating Unity Catalog with Azure AD, external metastores, and audit trails.\n\nExperience in building efficient orchestration solutions using Azure data factory, Databricks Workflows.\nAbility to design modular, reusable workflows using tasks, triggers, and dependencies.\nSkilled in using dynamic expressions, parameterized pipelines, custom activities, and triggers.\nFamiliarity with integration runtime configurations, pipeline performance tuning, and error handling strategies.\n\nStrong experience in implementing secure, hierarchical namespace-based data lake storage for structured/semi-structured data, aligned to bronze-silver-gold layers with ADLS Gen2. \nHands-on experience with lifecycle policies, access control (RBAC/ACLs), and folder-level security. Understanding of best practices in file partitioning, retention management, and storage performance optimization.\n\nCapable of developing T-SQL queries, stored procedures, and managing metadata layers on Azure SQL.\n\nComprehensive experience working across the Azure ecosystem, including networking, security, monitoring, and cost management relevant to data engineering workloads.\nUnderstanding of VNets, Private Endpoints, Key Vaults, Managed Identities, and Azure Monitor.\nExposure to DevOps tools for deployment automation (e.g., Azure DevOps, ARM/Bicep/Terraform).\n\nExperience in writing modular, testable Python code used in data transformations, utility functions, and packaging reusable components.\nFamiliarity with Python environments, dependency management (pip/Poetry/Conda), and packaging libraries.\nAbility to write unit tests using PyTest/unittest and integrate with CI/CD pipelines.\n\nLead solution design discussions, mentor junior engineers, and ensure adherence to coding guidelines, design patterns, and peer review processes.\nAble to prepare Design documents for development and guiding the team technically. Experience preparing technical design documents, HLD/LLDs, and architecture diagrams.\nFamiliarity with code quality tools (e.g., SonarQube, pylint), and version control workflows (Git).\n\nDemonstrates strong verbal and written communication, proactive stakeholder engagement, and a collaborative attitude in cross-functional teams.\nAbility to articulate technical concepts clearly to both technical and business audiences.\nExperience in working with product owners, QA, and business analysts to translate requirements into deliverables.","description_format":"text","description_chars":4545,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Corporate Training","Human Resources"],"lifecycle":[{"event":"open","at":"2026-11-09T16:53:55Z"}],"visa":[],"liveness":{"score":86,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.86,"p_room":1,"age_days":4,"expected_fill_days":28,"reasons":["seen:4","win:early"],"computed_at":"2026-11-09T18:30:41Z"},"pay":null,"html_url":"https://alion.io/job/digicomplish-data-engineering-tech-lead","json_url":"https://alion.io/job/digicomplish-data-engineering-tech-lead.json","meta":{"generated_at":"2026-11-09T18:30:41Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":3265,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-10T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":3801214},"rest":"https://alion.io/mcp/rest/get_company?id=3801214"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fdigicomplish-data-engineering-tech-lead"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":2,"usd":0.2},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fdigicomplish-data-engineering-tech-lead"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fdigicomplish-data-engineering-tech-lead"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/digicomplish-data-engineering-tech-lead\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fdigicomplish-data-engineering-tech-lead"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":3973645}},{"tool":"search_jobs","arguments":{"company":"digicomplish.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":3973645},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}