{"id":896748,"url":"https://alion.io/job/jpmorganchase-lead-software-engineer-databricks-52","title":"Lead Software Engineer - Databricks","company":{"id":257,"name":"JPMorganChase","domain":"jpmorganchase.com","url":"https://alion.io/company/jpmorganchase","size_band":"5000+","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Oracle","truth_index":{"grade":"A","score":100,"open_postings":535,"ghost_share":0.011,"stale_share":0.123,"repost_share":0.114,"time_to_fill_p50_days":8,"computed_at":"2026-10-17T05:46:00Z"}},"role":"Data Science","role_family":"Data Science","seniority":"lead","employment_type":null,"work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Wilmington, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":153000,"max_usd":300000,"period":"year","method":"role_seniority_country_remote_cell","sample_n":390},"experience_years_min":5,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Agile","optional":false},{"name":"Airflow","optional":false},{"name":"AWS","optional":false},{"name":"AWS Glue","optional":false},{"name":"CI/CD","optional":false},{"name":"Databricks","optional":false},{"name":"Delta Lake","optional":false},{"name":"Dimensional Modeling","optional":false},{"name":"ETL/ELT","optional":false},{"name":"Git","optional":false},{"name":"Java","optional":false},{"name":"Python","optional":false},{"name":"Shuffle","optional":false},{"name":"Spark","optional":false},{"name":"SQL","optional":false},{"name":"Terraform","optional":false},{"name":"Streamlit","optional":true}],"status":"live","first_seen_at":"2026-07-08T00:00:00Z","employer_posted_date":"2026-07-08","last_verified_at":"2026-10-17T10:05:06Z","board_verified":true,"closed_at":null,"days_open":101,"trust":{"level":"stale","repost_count":0,"flags":["stale"],"days_open":101},"description":"Be an integral part of an agile team that’s constantly pushing the envelope to enhance, build, and deliver top-notch technology products.\nAs a Lead Software Engineer-Databricks at JPMorgan Chase within our Corporate Sector’s Enterprise Technology team, you are an integral part of an agile team that works to enhance, build, and deliver trusted market-leading technology products in a secure, stable, and scalable way. As a core technical contributor, you are responsible for conducting critical technology solutions across multiple technical areas within various business functions in support of the firm’s business objectives.\nJob responsibilities\nLead the architecture and delivery of high-throughput, low-latency data pipelines on Databricks using Apache Spark (Core, SQL, Structured Streaming), driving performance, reliability, and scalability.\nEstablish and evolve Lakehouse patterns with Delta Lake (ACID transactions, schema evolution, time travel, Z-ordering, compaction) to ensure performant, maintainable data platforms at scale.\nOwn Databricks cluster strategy and configuration, including runtime selection, autoscaling, driver/executor sizing, Spark configurations, init scripts, cluster policies, pools, and instance profiles.\nOrchestrate and automate pipelines and jobs using Databricks Workflows, integrating with AWS eventing and orchestration services as needed.\nDesign secure ingestion and transformation frameworks leveraging Databricks services, including Delta or unmanaged table design, ingestion task creation, and Airflow DAGs to produce trusted and refined datasets.\nEnforce data quality, lineage, and governance using Unity Catalog and/or AWS Glue Catalog, embedding expectations and validation directly into pipelines.\nDrive Spark and Databricks performance engineering and tuning (partitioning and file sizing, AQE, broadcast joins, shuffle tuning, caching, spill/memory control, job right-sizing, and liquid clustering/partitioning keys) to optimize cost and throughput.\nBuild and maintain reusable libraries, frameworks, and APIs in Python and/or Java, ensuring strong unit, integration, and data validation test coverage.\nImplement CI/CD for data projects using Git-based workflows, Terraform-based infrastructure deployments and environment promotion, and automated releases; champion engineering standards, code reviews, and enterprise-authorized AI-assisted engineering practices (e.g., code review/refactoring, test acceleration, and incident/root-cause analysis) with consistent validation (secure coding, peer review, automated testing) and reuse of proven patterns.\nDrives team adoption of enterprise-authorized AI-assisted engineering practices within the work environment to improve code quality, delivery speed, and operational outcomes (e.g., AI-assisted code review/refactoring, test strategy acceleration, incident/root-cause analysis support), while establishing consistent validation standards (secure coding, peer review, automated testing) and promoting reuse of effective patterns across the team.\nApplies knowledge of tools within the Software Development Life Cycle toolchain, including enterprise-authorized AI-assisted development and automation capabilities, to improve the value realized by automation.\nRequired qualifications, capabilities, and skills\n\nFormal training or certification on software engineering concepts and 5+ years applied experience. \nAdvanced experience in software engineering and data engineering, including significant production delivery with Apache Spark on Databricks and/or AWS EMR.\nAdvanced hands-on Databricks expertise across Delta Lake, Unity Catalog, Workflows, Repos/notebooks, and SQL Warehouses, including cluster configuration and optimization.\nProven ability to architect, build, and operate reliable ETL/ELT data pipelines (batch and streaming), including schema design/evolution, SLAs, and reliability engineering practices.\nDeep Spark performance tuning skills, with experience diagnosing bottlenecks and optimizing jobs for scalability, cost, and runtime efficiency.\nStrong programming proficiency in Python and/or Java for data processing, platform tooling, and automation.\nStrong SQL and analytics data modeling expertise, including dimensional/star schema design and Lakehouse best practices.\nDemonstrated experience leading effective use of approved AI-assisted software development tools (coding, code review, test acceleration, troubleshooting), including setting team expectations and validation standards for correctness, performance, and security of AI outputs.\nStrong responsible-AI and security-first engineering mindset, including data sensitivity awareness, secure handling of inputs/outputs, roles/instance profiles, secrets management, encryption at rest/in transit, network controls, and adherence to resiliency and security expectations; experience coaching teams on safe, compliant adoption within delivery practices.\nDemonstrated experience leading effective use of approved AI-assisted software development tools (e.g., for coding, code review, test acceleration, troubleshooting) with the ability to set team expectations for validating AI outputs for correctness, performance, and security.\nStrong understanding of responsible AI use in engineering workflows, including data sensitivity considerations, secure handling of inputs/outputs, and adherence to resiliency and security expectations; experience coaching engineers on safe, compliant adoption within delivery practicesPreferred qualifications, capabilities, and skills\n\nExperience with Delta Live Tables and advanced governance (catalogs, grants, auditing) in Databricks.\nAWS networking knowledge (VPC, subnets, routing, security groups) and data egress controls.\nExperience with Terraform for Infra deployments\nCost optimization experience: autoscaling strategies, spot vs on-demand, auto-termination, storage layouts and compaction.\nFamiliarity with Airflow, Genie, Streamlit and React\nObservability for data systems (freshness/completeness metrics, lineage, SLAs, alerting).\nDemonstrated leadership in code quality, reviews, testing strategy, CI/CD, and technical mentorship; excellent communication with stakeholders.","description_format":"text","description_chars":6200,"description_truncated":false,"requirements":{"experience_years_min":5,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"phd","optional":false},"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Commercial & Retail Banks","Wealth Management & Financial Advisors","Asset Management & Funds","Investment Banking & M&A Advisory"],"lifecycle":[{"event":"open","at":"2026-09-14T09:49:40Z"},{"event":"close","at":"2026-09-14T09:49:40Z"},{"event":"reopen","at":"2026-09-14T10:54:24Z"}],"visa":[{"country":"US","licensed_sponsor":true,"evidence":"H-1B filings in 12 months: 3,429 · green card filings: 933","filings_12m":3429,"filings_prev_12m":3601,"green_card_filings_12m":933,"median_offered_wage_usd":160366,"route":null,"cap_exempt":false,"checked_at":"2026-10-15T11:24:14+00:00","sources":["US Department of Labor: LCA disclosure data (H-1B, H-1B1, E-3)","US Department of Labor: PERM disclosure data (green cards)"],"filings_for_role_12m":451}],"liveness":{"score":7,"band":"cold","label":"Long shot","p_open":1,"p_active":0.245,"p_room":0.28,"age_days":101,"expected_fill_days":8,"reasons":["conf:0","velocity","win:tail","crowd:brand"],"computed_at":"2026-10-17T05:46:00Z"},"pay":null,"html_url":"https://alion.io/job/jpmorganchase-lead-software-engineer-databricks-52","json_url":"https://alion.io/job/jpmorganchase-lead-software-engineer-databricks-52.json","meta":{"generated_at":"2026-10-17T10:58:32Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":14346,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-18T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":257},"rest":"https://alion.io/mcp/rest/get_company?id=257"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fjpmorganchase-lead-software-engineer-databricks-52"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fjpmorganchase-lead-software-engineer-databricks-52"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fjpmorganchase-lead-software-engineer-databricks-52"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/jpmorganchase-lead-software-engineer-databricks-52\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fjpmorganchase-lead-software-engineer-databricks-52"}]}