{"id":1809189,"url":"https://alion.io/job/kanerika-lead-data-engineer-databricks","title":"Lead Data Engineer - Databricks","company":{"id":177954,"name":"Kanerika","domain":"kanerika.com","url":"https://alion.io/company/kanerika","size_band":null,"is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Zoho Recruit","truth_index":null},"role":"Data Science","role_family":"Data Science","seniority":"lead","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"explicit","locations":["Hyderabad, India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":27000,"max_usd":47000,"period":"year","method":"role_seniority_country_remote_cell","sample_n":16},"experience_years_min":2,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"AWS","optional":false},{"name":"Azure","optional":false},{"name":"CI/CD","optional":false},{"name":"Databricks","optional":false},{"name":"Delta Lake","optional":false},{"name":"ETL/ELT","optional":false},{"name":"GCP","optional":false},{"name":"GDPR","optional":false},{"name":"Git","optional":false},{"name":"LLM","optional":false},{"name":"pySpark","optional":false},{"name":"Python","optional":false},{"name":"SOC 2","optional":false},{"name":"Spark","optional":false},{"name":"SQL","optional":false},{"name":"Apache Kafka","optional":true},{"name":"Azure Data Factory","optional":true},{"name":"Fivetran","optional":true},{"name":"MLFlow","optional":true}],"status":"live","first_seen_at":"2026-10-03T19:56:08Z","employer_posted_date":"2026-10-03","last_verified_at":"2026-10-11T18:15:56Z","board_verified":true,"closed_at":null,"days_open":7,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":7},"description":"About\nKanerika\nWho\nwe are:\nKanerika\nInc. is a premier global software products and services firm that specializes\nin providing innovative solutions and services for data-driven enterprises. Our\nfocus is to empower businesses to achieve their digital transformation goals\nand maximize their business impact through the effective use of data and AI. We leverage cutting-edge technologies in data\nanalytics, data governance, AI-ML, GenAI/ LLM and industry best practices to\ndeliver custom solutions that help organizations optimize their operations,\nenhance customer experiences, and drive growth.\nAwards and Recognitions\nKanerika has won several awards over the years,\nincluding:\n·CMMI\nLevel 3 Appraised in 2024.\n·Best\nPlace to Work 2023 byGreat Place to Work\n·Top\n10 Most Recommended RPA Start-Ups in 2022 by RPA today.\n·Frost\n& Sullivan India 2021 Technology Innovation Award for its Kompass\ncomposable solution architecture.\n·Kanerika\nhas also been recognized for its commitment to customer privacy and data\nsecurity, having achieved, ISO 9001, ISO 27701, SOC2, and GDPR compliances.\nWorking\nfor us\nKanerika\nis rated 4.6/5 on Glassdoor, for many good reasons. We truly value our\nemployees' growth, well-being, and diversity, and people’s experiences bear\nthis out. At Kanerika, we offer a host of enticing benefits that create an\nenvironment where you can thrive both personally and professionally. From our\ninclusive hiring practices and mandatory training on creating a safe work\nenvironment to our flexible working hours and generous parental leave, we\nprioritize the well-being and success of our employees. Our commitment to\nprofessional development is evident through our mentorship programs, job training\ninitiatives, and support for professional certifications. Additionally, our\ncompany-sponsored outings and various time-off benefits ensure a healthy\nwork-life balance. Join us at Kanerika and become part of a vibrant and diverse\ncommunity where your talents are recognized, your growth is nurtured, and your\ncontributions make a real impact. See the benefits section below for the perks\nyou’ll get while working for Kanerika.\nLocations\nWe\nare located in Austin (USA), Singapore, Hyderabad, Indore and Ahmedabad\n(India).\nJob\nLocation: Hyderabad, Indore and Ahmedabad (India)\nRequirements\nRole:\nAs\none of the founding members of Kanerika's Databricks delivery practice, you\nwill be a hands-on builder responsible for designing, developing, and\noptimizing data pipelines on Databricks for client engagements. You'll work\nclosely with the Practice Lead to translate architecture into working\nsolutions, build reusable accelerators ahead of client demand, and help\nestablish engineering best practices for the growing team.\nKey\nResponsibilities\n·Design,\nbuild, and optimize ETL/ELT data pipelines on Databricks using PySpark and\nSpark SQL.\n·Build\nand maintain incremental ingestion pipelines using Auto Loader and structured\nstreaming, including checkpoint management and schema evolution handling.\n·Implement\nand maintain Delta Lake tables (including Change Data Feed and liquid\nclustering), Unity Catalog structures, and Databricks Workflows for client and\ninternal projects.\n·Build\nreusable accelerators, templates, and demo environments to support pre-sales\nand speed up future client delivery.\n·Collaborate\nwith the Practice Lead/Architect on solution design for client engagements,\nproviding engineering-level input on feasibility and effort.\n·Perform\ndata quality checks, performance tuning, and cost optimization on Databricks\nclusters and jobs.\n·Participate\nin client-facing technical discussions as needed, including discovery sessions\nand technical walkthroughs.\n·Write\nclean, well-documented, and testable code following team engineering standards.\n·Mentor\njunior engineers as the team scales, and contribute to internal\nknowledge-sharing and best-practice documentation.\n·Troubleshoot\nand resolve production data pipeline issues across client environments.\n·Stay\ncurrent with Databricks platform releases across Unity Catalog, Lakeflow\nDeclarative Pipelines, and share learnings with the team through internal\nknowledge sessions and documentation.\nRequired\nSkills & Experience\n·5-9\nyears of data engineering experience, with at least 2 years of hands-on\nDatabricks experience in production.\n·Strong\nproficiency in PySpark and/or Spark SQL for large-scale data processing.\n·Practical\nexperience with Delta Lake, Unity Catalog, and Databricks Workflows/job\norchestration.\n·Solid\nSQL skills and experience with data modeling for analytics/lakehouse\narchitectures.\n·Experience\nwith at least one major cloud platform (Azure, AWS, or GCP); Azure strongly\npreferred.\n·Experience\nwith Python for data engineering tasks beyond Spark (scripting, automation,\ntesting).\n·Familiarity\nwith CI/CD practices for data pipelines (Git-based workflows, automated\ntesting/deployment).\n·Strong\ndebugging and performance-tuning skills for Spark jobs (partitioning, caching,\ncluster sizing).\n·Good\ncommunication skills and comfort working directly with client stakeholders when\nneeded.\nPEFERRED/\nNICE TO HAVE\n·Databricks\nCertified Data Engineer Associate/Professional certification.\n·Experience\nwith Lakeflow Declarative Pipelines (formerly Delta Live Tables), Lakeflow\nConnect for managed ingestion, MLflow for experiment tracking and model\nregistry, Databricks SQL, and AI/BI Genie.\n·Exposure\nto data governance and security frameworks (row/column-level security, data\nmasking).\n·Prior\nexperience in a consulting/IT services environment delivering to multiple\nclients.\n·Familiarity\nwith orchestration tools (Airflow) and ingestion tools (Fivetran, Kafka, Azure\nData Factory).\nWhat\nSuccess Looks Like\n·Within\n3 months: Comfortable with Kanerika's delivery standards; has built or\ncontributed to at least one reusable accelerator/demo asset.\n·Within\n6 months: Independently delivering core engineering work on client\nengagement(s) with minimal oversight.\n·Within\n12 months: Recognized as a go-to senior engineer on the team, mentoring newer\nhires and contributing to architecture decisions.\n·Leading\ndatabricks partnership upgrade to Gold level.\nBenefits\nWhy\njoin us?\n·Work\nwith a passionate and innovative team in a fast-paced, growth-oriented\nenvironment.\n·Gain\nhands-on experience in content marketing with exposure to real-world projects.\n·Opportunity\nto learn from experienced professionals and enhance your marketing skills.\n·Contribute\nto exciting initiatives and make an impact from day one.\n·Competitive\nstipend and potential for growth within the company.\n·Recognized\nfor excellence in data and AI solutions with industry awards and accolades.\nEmployee Benefits:\n1. Culture:\ni.Open\nDoor Policy: Encourages open communication and accessibility to management.\nii.Open\nOffice Floor Plan: Fosters a collaborative and interactive work environment.\niii.Flexible\nWorking Hours: Allows employees to have flexibility in their work schedules.\niv.Employee\nReferral Bonus: Rewards employees for referring qualified candidates.\nv.Appraisal\nProcess Twice a Year: Provides regular performance evaluations and feedback.\n2. Inclusivity and Diversity:\na.Hiring practices that promote diversity: Ensures a\ndiverse and inclusive workforce.\nb.Mandatory POSH training: Promotes a safe and\nrespectful work environment.\n3. Health Insurance and Wellness Benefits:\na.GMC and Term Insurance: Offers medical coverage\nand financial protection.\nb.Health Insurance: Provides coverage for medical\nexpenses.\nc.Disability Insurance: Offers financial support in\ncase of disability.\n4. Child Care & Parental Leave Benefits:\na.Company-sponsored family events: Creates\nopportunities for employees and their families to bond.\nb.Generous Parental Leave: Allows parents to take\ntime off after the birth or adoption of a child.\nc.Family Medical Leave: Offers leave for employees\nto take care of family members' medical needs.\n5. Perks and Time-Off Benefits:\na.Company-sponsored outings: Organizes recreational\nactivities for employees.\nb.Gratuity: Provides a monetary benefit as a token\nof appreciation.\nc.Provident Fund: Helps employees save for\nretirement.\nd.Generous PTO: Offers more than the industry\nstandard for paid time off.\ne.Paid sick days: Allows employees to take paid time\noff when they are unwell.\nf.Paid\nholidays: Gives employees paid time off for designated holidays.\ng.Bereavement Leave: Provides time off for employees\nto grieve the loss of a loved one.\n6. Professional Development Benefits:\na.L&D with FLEX- Enterprise Learning Repository:\nProvides access to a learning repository for professional development.\nb.Mentorship Program: Offers guidance and support\nfrom experienced professionals.\nc.Job Training: Provides training to enhance\njob-related skills.\nd.Professional Certification Reimbursements: Assists\nemployees in obtaining professional certifications.\ne.Promote from Within: Encourages internal growth\nand advancement opportunities.","description_format":"text","description_chars":8927,"description_truncated":false,"requirements":{"experience_years_min":2,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":["Certification reimbursement","Flexible schedule","Health insurance","Parental leave","Professional development"],"hiring_locations":[{"name":"United States","iso":"US","kind":"country"},{"name":"India","iso":"IN","kind":"country"},{"name":"Singapore","iso":"SG","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["AI Consulting & Integration","Data Engineering & Migration Services"],"lifecycle":[{"event":"open","at":"2026-10-03T19:56:08Z"}],"visa":[],"liveness":{"score":87,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.872,"p_room":1,"age_days":6,"expected_fill_days":23,"reasons":["conf:2","velocity","win:early"],"computed_at":"2026-10-10T05:45:15Z"},"pay":null,"html_url":"https://alion.io/job/kanerika-lead-data-engineer-databricks","json_url":"https://alion.io/job/kanerika-lead-data-engineer-databricks.json","meta":{"generated_at":"2026-10-11T19:50:36Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":7098,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-12T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":177954},"rest":"https://alion.io/mcp/rest/get_company?id=177954"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fkanerika-lead-data-engineer-databricks"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fkanerika-lead-data-engineer-databricks"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fkanerika-lead-data-engineer-databricks"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/kanerika-lead-data-engineer-databricks\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fkanerika-lead-data-engineer-databricks"}]}