{"id":3107382,"url":"https://alion.io/job/databeat-databricks-developer","title":"Databricks Developer","company":{"id":3806738,"name":"DataBeat","domain":"databeat.io","url":"https://alion.io/company/databeat","size_band":"1001-5000","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Keka","truth_index":{"grade":"C","score":68,"open_postings":17,"ghost_share":0.529,"stale_share":0,"repost_share":0,"time_to_fill_p50_days":null,"computed_at":"2026-11-03T05:54:40Z"}},"role":"Data Science","role_family":"Data Science","seniority":"senior","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":[],"countries":[],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":5,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Airflow","optional":false},{"name":"Amazon S3","optional":false},{"name":"AWS","optional":false},{"name":"AWS Glue","optional":false},{"name":"AWS Lambda","optional":false},{"name":"Azure","optional":false},{"name":"Azure Data Factory","optional":false},{"name":"CI/CD","optional":false},{"name":"Databricks","optional":false},{"name":"Delta Lake","optional":false},{"name":"ETL/ELT","optional":false},{"name":"GitHub Actions","optional":false},{"name":"GitLab","optional":false},{"name":"MLFlow","optional":false},{"name":"Power BI","optional":false},{"name":"pySpark","optional":false},{"name":"Python","optional":false},{"name":"Spark","optional":false},{"name":"SQL","optional":false},{"name":"Tableau","optional":false},{"name":"ISO 27001","optional":true}],"status":"live","first_seen_at":"2026-09-21T06:08:03Z","employer_posted_date":"2026-09-21","last_verified_at":"2026-11-03T17:12:08Z","board_verified":true,"closed_at":null,"days_open":43,"trust":{"level":"ok","repost_count":0,"flags":["company_stale"],"days_open":42},"description":"Job Title: Databricks Developer\nExperience Required: 5+ Years\nTechnical Knowledge:\nDatabricks (PySpark, SQL, Notebooks), Delta Lake, Apache Spark, Python, Azure Data Factory (ADF), Azure Synapse Analytics, AWS S3, Glue, Lambda, CI/CD Pipelines, GitLab, Data Quality Tools, Power BI/Tableau, Data Cataloging, Data Governance, Unity Catalog, MLflow, Job Clusters\nRole Summary:\nWe are seeking a Databricks Developer with over 5 years of experience in building and optimizing scalable data pipelines using Databricks and Apache Spark. This role requires strong expertise in PySpark, Delta Lake, and cloud-native data engineering across Azure and/or AWS. You will lead the development of real-time and batch data solutions, contribute to data architecture, and ensure high performance, cost efficiency, and reliability across platforms. The ideal candidate is passionate about distributed computing, clean code, and modern data stack automation.\nKey Responsibilities:\nDesign, build, and optimize robust ETL/ELT pipelines on Databricks using PySpark, SQL, and Delta Lake.\nLead the development of reusable and modular Notebooks, Jobs, and Workflows for both batch and streaming workloads.\nCollaborate with data architects and engineers to build scalable data lakes and data warehouse layers on cloud platforms.\nImplement best practices for coding standards, version control, testing, and performance tuning in Databricks.\nUse Unity Catalog to manage metadata, lineage, access policies, and data governance across workspaces.\nIntegrate Databricks with orchestration tools like ADF, AWS Glue, or Airflow, depending on the environment.\nBuild and manage CI/CD pipelines using GitLab, GitHub Actions, or other DevOps tools for automated deployments.\nWork with business users and data scientists to develop high-performance pipelines for analytics and ML models.\nMonitor and troubleshoot Databricks Jobs, Job Clusters, and compute usage, optimizing for cost and speed.\nEnsure security and compliance across data assets, leveraging row-level security, tokenization, and audit logging.\nMaintain comprehensive documentation of pipelines, job configs, and data processing logic.\nRequirements:\n5+ years of hands-on experience working with Databricks, building scalable, distributed data pipelines using PySpark and SQL.\nStrong understanding of Apache Spark internals, cluster tuning, job performance optimization, and partitioning strategies.\nExpertise in working with Delta Lake, schema evolution, and ACID-compliant data lakes.\nProficient in Python for scripting and data transformation, and SQL for data wrangling and analysis.\nExperience integrating Databricks with Azure (ADF, Synapse) and/or AWS (S3, Glue, Lambda) environments.\nKnowledge of Unity Catalog or similar governance tools for managing access, lineage, and policies.\nSolid understanding of data modeling principles, especially for lakehouse and medallion architecture.\nExperience implementing CI/CD pipelines and DevOps practices in a Databricks environment.\nStrong communication skills to work with stakeholders, analysts, and engineers across teams.\nBachelor's or Master’s degree in Computer Science, Engineering, or a related technical field.\nPreferred Skills:\nExperience with Databricks Workflows, MLflow, and Job Clusters for advanced data science and automation use cases.\nFamiliarity with data observability tools and performance monitoring dashboards.\nKnowledge of Power BI/Tableau and building semantic layers on top of Databricks SQL endpoints.\nInformation Security & ISO 27001 Compliance\nSecurity is at the heart of everything we do. In this role, you will be strictly required to uphold our Information Security Management System (ISMS) policies in alignment with ISO/IEC 27001 standards. Responsibilities include safeguarding sensitive asset data, completing mandatory security awareness training, reporting potential security incidents or vulnerabilities immediately, and ensuring that daily operations comply with our rigorous data protection protocols.","description_format":"text","description_chars":4017,"description_truncated":false,"requirements":{"experience_years_min":5,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"bachelor","optional":false},"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Marketing Analytics & Attribution","Analytics & BI Consulting"],"lifecycle":[{"event":"open","at":"2026-10-22T19:41:40Z"}],"visa":[],"liveness":{"score":29,"band":"fade","label":"Fading","p_open":1,"p_active":0.521,"p_room":0.55,"age_days":42,"expected_fill_days":30,"reasons":["conf:2","stale_co","win:tail"],"computed_at":"2026-11-03T05:54:40Z"},"pay":null,"html_url":"https://alion.io/job/databeat-databricks-developer","json_url":"https://alion.io/job/databeat-databricks-developer.json","meta":{"generated_at":"2026-11-03T21:27:32Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":650,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-04T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":3806738},"rest":"https://alion.io/mcp/rest/get_company?id=3806738"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fdatabeat-databricks-developer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fdatabeat-databricks-developer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fdatabeat-databricks-developer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/databeat-databricks-developer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fdatabeat-databricks-developer"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":3107382}},{"tool":"search_jobs","arguments":{"company":"databeat.io","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":3107382},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}