{"id":3451578,"url":"https://alion.io/job/luxoft-lead-pyspark-developer","title":"Lead Pyspark Developer","company":{"id":171185,"name":"Luxoft","domain":"luxoft.com","url":"https://alion.io/company/luxoft","size_band":null,"is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Career sitemap","truth_index":{"grade":"B","score":83,"open_postings":13,"ghost_share":0,"stale_share":0.769,"repost_share":0.154,"time_to_fill_p50_days":18,"computed_at":"2026-10-29T05:49:05Z"}},"role":"Data Science","role_family":"Data Science","seniority":"lead","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Singapore"],"countries":["SG"],"hiring_countries":[],"hiring_countries_total":0,"salary":{"min":9000,"max":15000,"currency":"SGD","period":"month","gross":true,"usd_annual":140352},"salary_estimate":null,"experience_years_min":8,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Git","optional":false},{"name":"NumPy","optional":false},{"name":"Pandas","optional":false},{"name":"pySpark","optional":false},{"name":"Python","optional":false},{"name":"Spark","optional":false},{"name":"Airflow","optional":true},{"name":"Django","optional":true},{"name":"Machine Learning","optional":true},{"name":"PostgreSQL","optional":true},{"name":"SQL","optional":true}],"status":"live","first_seen_at":"2026-10-29T00:00:00Z","employer_posted_date":null,"last_verified_at":"2026-10-29T00:00:00Z","board_verified":false,"closed_at":null,"days_open":0,"trust":{"level":"not_scored","repost_count":null,"flags":[],"days_open":0},"description":"About the Role\nSurveillance Enhancement Program (SEP) is a multi-year program (MYP) to build and enhance the surveillance platform to detect potential instances of market misconduct, market manipulation and market abuse. The Bank met its commitment to FCA (Financial Conduct Authority, FCA) to meet Market Abuse Regulation (\"MAR\") requirements for critical data sources in September 2017. Following that, subsequent projects were initiated to further enhance, expand and complement the current coverage of the automated surveillance platform. This project will focus on Alerts enhancements, Strategic data sourcing from FM order and trade lakes, UI / workflow enhancements to meet regulatory obligations (i.e. Market Abuse Regulation, Dodd Frank). This also includes Control Room & Suitability Monitoring related data, UI and scenario enhancements.\n\nResponsibilities\nDevelop software applications based on business requirementsMaintain software applications and make enhancements according to project specifications\nParticipate in requirement analysis, design, development, testing, and implementation activitiesPropose new techniques and technologies for software development.\nPerform unit testing and user acceptance testing to evaluate application functionality\nEnsure to complete the assigned development tasks within the deadlines\nWork in compliance with coding standards and best practicesProvide assistance to Junior Developers when needed.\nPerform code reviews and recommend improvements.Review business requirements and recommend changes to develop reliable applications.\nDevelop coding documentation and other technical specifications for assigned projects.\nAct as primary contact for development queries and concerns.\nAnalyze and resolve development issues accurately.\n\nRequired Skills\n15+ years of experience in data intensive Pyspark development ,Window Functions , Performance Optimization ,Spark Architecture,Salting ,Column Pruning\nMust have experience in leading a Development TeamProven experience as a Core Python developer\nGood understanding of Gen AIExperience developing Classes, OOPS, exception handling, parallel processing\nStrong knowledge of DB connectivity, data loading , transformation, calculation\nExtensive experience in Pandas/Numpy dataframes, slicing, data wrangling, aggregations\nLambda Functions, Decorators\nVector operations on Pandas dataframes /seriesApplication of apply map, apply, map functions\nConcurrency and error handling data pipeline batch of size [1-10 gb]\nAbility to understand business requirements and translate them into technical requirementsAbility to design architecture of data pipeline for concurrent data processing\nWorking with Git source control Service-orientated architecture, including the ability to consider integrations with other applications and services\nDebugging application\n\nPreferred Skills\nKnowledge of web backend technology - Django, Python, PostgreSQL.\nApache Airflow\nAtlassian\nJira\nUnderstanding of Financial Markets Asset Classes (FX, FI, Equities, Rates, Commodities & Credit), various trade types (OTC, exchange traded, Spot, Forward, Swap, Options) and related systems is a plus\nSurveillance domain knowledge, regulations (MAR, MIFID, CAT, Dodd Frank) and related Systems knowledge is certainly a plus\nSkills\nProject Specification, Machine Learning, meeting project deadlines, Design, Airflow, Development, Django web framework, Software Implementation, Requirement Reviews, Develop Business Applications, Software Engineering, Systems Knowledge, Top Systems, SQL, JIRA, Python, User Acceptance Testing, interaction with developers, support to junior staff, Equities","description_format":"text","description_chars":3648,"description_truncated":false,"requirements":{"experience_years_min":8,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["IT Consulting & Digital Transformation","IT Outsourcing & Dedicated Teams","Custom Software Development"],"lifecycle":[{"event":"open","at":"2026-10-29T10:32:45Z"}],"visa":[],"liveness":{"score":86,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.86,"p_room":1,"age_days":0,"expected_fill_days":18,"reasons":["seen:0","win:early","comp:brand"],"computed_at":"2026-10-29T14:19:17Z"},"pay":{"stated_usd_annual":140352,"is_top_pay":false},"html_url":"https://alion.io/job/luxoft-lead-pyspark-developer","json_url":"https://alion.io/job/luxoft-lead-pyspark-developer.json","meta":{"generated_at":"2026-10-29T14:19:17Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":21534,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-30T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":171185},"rest":"https://alion.io/mcp/rest/get_company?id=171185"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fluxoft-lead-pyspark-developer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fluxoft-lead-pyspark-developer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fluxoft-lead-pyspark-developer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/luxoft-lead-pyspark-developer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fluxoft-lead-pyspark-developer"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":3451578}},{"tool":"search_jobs","arguments":{"company":"luxoft.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":3451578},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}