{"id":1904754,"url":"https://alion.io/job/chase-uk-lead-software-engineer-pyspark-aws-python","title":"Lead Software Engineer - Pyspark, AWS, Python","company":{"id":1854819,"name":"Chase UK","domain":"chase.co.uk","url":"https://alion.io/company/chase-co-uk","size_band":"1001-5000","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Oracle","truth_index":{"grade":"A","score":100,"open_postings":269,"ghost_share":0.011,"stale_share":0,"repost_share":0.097,"time_to_fill_p50_days":4,"computed_at":"2026-10-10T05:45:15Z"}},"role":"Backend","role_family":"Backend","seniority":"lead","employment_type":null,"work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Pune, India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":24000,"max_usd":51000,"period":"year","method":"role_seniority_country_remote_cell","sample_n":54},"experience_years_min":5,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Agile","optional":false},{"name":"Airflow","optional":false},{"name":"Apache Iceberg","optional":false},{"name":"Apache Kafka","optional":false},{"name":"AWS","optional":false},{"name":"AWS Step Functions","optional":false},{"name":"CI/CD","optional":false},{"name":"Data Vault","optional":false},{"name":"Databricks","optional":false},{"name":"Dimensional Modeling","optional":false},{"name":"Docker","optional":false},{"name":"Hadoop","optional":false},{"name":"Java","optional":false},{"name":"Kubernetes","optional":false},{"name":"Machine Learning","optional":false},{"name":"Python","optional":false},{"name":"Scala","optional":false},{"name":"Spark","optional":false},{"name":"SQL","optional":false},{"name":"CloudFormation","optional":true},{"name":"Snowflake","optional":true},{"name":"Spinnaker","optional":true},{"name":"Terraform","optional":true}],"status":"closed","first_seen_at":"2026-10-05T09:46:59Z","employer_posted_date":"2026-10-05","last_verified_at":"2026-10-10T12:58:07Z","board_verified":false,"closed_at":"2026-10-10T12:58:07Z","days_open":5,"trust":{"level":"not_scored","repost_count":null,"flags":[],"days_open":5},"description":"We have an opportunity to impact your career and provide an adventure where you can push the limits of what's possible.\nAs a Lead Software Engineer at JPMorganChase within the Consumer and Community Banking, you are an integral part of an agile team that works to enhance, build, and deliver trusted market-leading technology products in a secure, stable, and scalable way. As a core technical contributor, you are responsible for conducting critical technology solutions across multiple technical areas within various business functions in support of the firm’s business objectives.\nJob responsibilities\nLead the design, development, and maintenance of robust, scalable cloud-based data processing pipelines and infrastructure, ensuring adherence to engineering standards, governance frameworks, and industry best practices.\nArchitect and refine data models for large-scale datasets, optimizing for efficient storage, high-performance retrieval, and advanced analytics while upholding data integrity and quality.\nPartner with cross-functional teams to translate complex business requirements into effective, scalable data engineering solutions that drive organizational value.\nChampion a culture of innovation and continuous improvement, proactively identifying and implementing enhancements to data infrastructure, processing workflows, and analytics capabilities.\nDefine and execute data strategy, including the development of enterprise data models and the management of end-to-end data infrastructure-from design and construction to installation and ongoing maintenance of large-scale processing systems.\nDrive data quality initiatives, ensure seamless data accessibility for analysts and data scientists, and maintain strict compliance with data governance and regulatory requirements.\nAlign data engineering practices with business objectives, ensuring solutions are both technically sound and strategically relevant.\nAuthor, review, and approve technical requirements and architectural designs, and lead process re-engineering efforts to deliver cost-effective, high-impact business solution\nDrives team adoption of enterprise-authorized AI-assisted engineering practices within the work environment to improve code quality, delivery speed, and operational outcomes (e.g., AI-assisted code review/refactoring, test strategy acceleration, incident/root-cause analysis support), while establishing consistent validation standards (secure coding, peer review, automated testing) and promoting reuse of effective patterns across the team.\nApplies knowledge of tools within the Software Development Life Cycle toolchain, including enterprise-authorized AI-assisted development and automation capabilities, to improve the value realized by automation.\nRequired qualifications, capabilities, and skills\nFormal training or certification on software engineering concepts and 5+ years applied experience\nExpert in at least one distributed data processing framework (Spark). Expert in at least one cloud data Lakehouse platforms (AWS Data lake services or Databricks, if not Hadoop),\nExpert in at least one scheduling/orchestration tools ( Airflow, alternatively AWS Step Functions or similar) & Expert with relational and NoSQL databases. Expert in data structures, data serialization formats (JSON, AVRO, Protobuf, or similar), and big-data storage formats (Parquet, Iceberg, or similar)\nHands-on professional experience in one or more programming language(s), including Java or Python, proficiency in Python, SQL, and at least one additional language (e.g. Java or Scala) for data engineering tasks\nHands-on experience utilizing Apache Spark for large-scale data processing, including developing and optimizing data pipelines, performing real-time and batch analytics, and leveraging Spark’s libraries for machine learning and data transformation to drive actionable business insights.\nProficiency in microservices architecture, serverless computing and distributed cluster computing tools such as Docker, Kubernetes etc. Experience in one or more data modelling techniques (Dimensional, Data Vault, Kimball, Inmon, etc.)\nExperience with test-driven development (TDD) or behavior-driven development (BDD) practices, as well as working with continuous integration and continuous deployment (CI/CD) tools.\nExperience organizing and leading design workshops, coding sessions, and hackathons to promote a culture of excellence and innovation in data engineering. Expertise in architecting reusable, future-ready design patterns that address diverse use cases across the organization.\nExpertise in working with streaming platforms like Kafka, MQ etc.\nDemonstrated experience leading effective use of approved AI-assisted software development tools (e.g., for coding, code review, test acceleration, troubleshooting) with the ability to set team expectations for validating AI outputs for correctness, performance, and security.\nStrong understanding of responsible AI use in engineering workflows, including data sensitivity considerations, secure handling of inputs/outputs, and adherence to resiliency and security expectations; experience coaching engineers on safe, compliant adoption within delivery practices\nPreferred qualifications, capabilities, and skills\nHands-on experience with Infrastructure as Code (IaC) tools, preferably Terraform; experience with AWS CloudFormation is also valued.\nProficiency in cloud-based data pipeline technologies such as Spinnaker or similar platforms.\nStrong working knowledge of the Snowflake data platform.\nExperience in budgeting and resource allocation for data engineering projects.\nProven ability to manage vendor relationships effectively.","description_format":"text","description_chars":5682,"description_truncated":false,"requirements":{"experience_years_min":5,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Smartphones"],"lifecycle":[{"event":"open","at":"2026-10-05T12:26:16Z"},{"event":"close","at":"2026-10-10T12:58:07Z"}],"visa":[],"liveness":null,"pay":null,"html_url":"https://alion.io/job/chase-uk-lead-software-engineer-pyspark-aws-python","json_url":"https://alion.io/job/chase-uk-lead-software-engineer-pyspark-aws-python.json","meta":{"generated_at":"2026-10-11T05:15:51Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler","counted_by":"address","units_charged":1,"used_today":4807,"day_limit":5000,"remaining_today":193,"minute_limit":60,"resets_at":"2026-10-12T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":1854819},"rest":"https://alion.io/mcp/rest/get_company?id=1854819"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fchase-uk-lead-software-engineer-pyspark-aws-python"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fchase-uk-lead-software-engineer-pyspark-aws-python"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fchase-uk-lead-software-engineer-pyspark-aws-python"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/chase-uk-lead-software-engineer-pyspark-aws-python\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fchase-uk-lead-software-engineer-pyspark-aws-python"}]}