{"id":2924061,"url":"https://alion.io/job/scrumconnect-data-engineer","title":"Data Engineer","company":{"id":3861039,"name":"Scrumconnect Consulting","domain":"scrumconnect.com","url":"https://alion.io/company/scrumconnect","size_band":"51-200","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Zoho Recruit","truth_index":{"grade":"B","score":80,"open_postings":25,"ghost_share":0,"stale_share":1,"repost_share":0,"time_to_fill_p50_days":9,"computed_at":"2026-10-27T05:46:45Z"}},"role":"Data Science","role_family":"Data Science","seniority":null,"employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Newcastle upon Tyne, United Kingdom"],"countries":["GB"],"hiring_countries":[],"hiring_countries_total":0,"salary":{"min":null,"max":65000,"currency":"GBP","period":"year","gross":null,"usd_annual":86499},"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Agile","optional":false},{"name":"Airflow","optional":false},{"name":"Amazon CloudWatch","optional":false},{"name":"Amazon EC2","optional":false},{"name":"Amazon S3","optional":false},{"name":"AWS","optional":false},{"name":"CI/CD","optional":false},{"name":"Docker","optional":false},{"name":"GitLab","optional":false},{"name":"GitLab CI","optional":false},{"name":"IAM","optional":false},{"name":"NLP","optional":false},{"name":"Power BI","optional":false},{"name":"pySpark","optional":false},{"name":"Python","optional":false},{"name":"Spark","optional":false},{"name":"SQL","optional":false},{"name":"Terraform","optional":false}],"status":"live","first_seen_at":"2026-08-28T00:00:00Z","employer_posted_date":"2026-08-28","last_verified_at":"2026-10-27T07:14:23Z","board_verified":true,"closed_at":null,"days_open":60,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":60},"description":"Data Engineer \nUp to £65k per annum\nApache Spark Python AWS Cloud Data Pipelines\nA hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong analytical and engineering skills to deliver trusted, well-governed data assets in a modern, cloud-native environment.\nAbout Scrumconnect\nScrumconnect is a leading UK technology consultancy delivering digital transformation across public and private sectors, contributing to over 20% of the UK's major citizen-facing public services. We specialise in cloud engineering, data platforms, and agile delivery, helping clients build scalable, secure, and user-centred digital solutions that create real impact.\nWorking arrangement:\nThis role is hybrid. Candidates must be willing and able to travel to the Newcastle office once per week. Remaining days may be worked remotely from anywhere in the UK.\nAbout the role\nYou will work as a Data Engineer on a complex, cloud-based data programme - designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics, you will help deliver reliable, well-governed data assets to downstream users.\nYou will apply strong data analysis skills to identify root causes of data issues, work with dimensional data models and slowly changing dimensions, and implement infrastructure as code using Terraform. Familiarity with engineering best practices and the ability to translate customer expectations into applied technical functionality are key to success in this role.\nKey responsibilitiesData pipeline development\nBuild and maintain scalable data pipelines using Apache Spark and PySpark, processing and transforming large datasets across distributed cloud infrastructure.\nWorkflow orchestration\nConfigure and manage Apache Airflow DAGs for task orchestration, ensuring reliable scheduling, monitoring, and execution of data processing workflows.\nRoot cause analysis\nPerform data analysis to identify and resolve root causes of pipeline failures and data quality issues - including reviewing EMR output logs and CloudWatch metrics.\nData modelling\nApply understanding of dimensional data models and slowly changing dimensions (SCD) to design and maintain well-structured, analytically trusted data assets.\nInfrastructure as code\nProvision and manage cloud infrastructure using Terraform. Containerise solutions using Docker and manage deployments through GitLab CI/CD pipelines and release tagging.\nSecurity & encryption\nApply understanding of both Server Side and client-side encryption patterns within AWS. Work within IAM policies and data governance standards appropriate to a regulated government environment.\nTechnical skills requiredLanguages & analytics\nPython - primary language for pipeline development and data processing\nSQL - used for querying, transformation, and validation across data stores\nPySpark - Power BI for distributed data processing using Apache Spark on AWS EMR\nFamiliarity with basic data structures for constructing robust, scalable solutions\nData processing & orchestration\nApache Spark - understanding of distributed data processing architecture and execution\nApache Airflow - configuring DAGs and managing task orchestration at scale\nJupyter Notebooks - for exploratory data analysis and pipeline prototyping\nUnderstanding of dimensional data models and slowly changing dimensions (SCD Types 1, 2, 3)\nData analysis skills to identify root cause of issues within pipelines and data assets\nAWS services\nAmazon EMR - running Spark workloads and reviewing output logs\nAmazon Athena - ad hoc querying of data in S3\nAmazon Textract and Comprehend - familiarity with AI/ML document extraction and NLP services\nAWS S3, IAM, CloudWatch, EC2, ECR - core platform services used day-to-day\nAWS console proficiency - navigating, configuring, and monitoring services\nUnderstanding of Server Side and client-side encryption within AWS\nInfrastructure, DevOps & delivery\nTerraform - Infrastructure as Code for provisioning and managing AWS environments\nDocker - containerisation of data engineering solutions\nGitLab - source code management, CI/CD pipeline configuration, release tagging, and component versioning\nFamiliarity with engineering best practices\nAbility to translate customer expectations into applied, functional technical solutions\nTechnology stack at a glance\nPythonPySparkSQLApache, Power BI SparkApache AirflowJupyter NotebooksDimensional modelling/SCDAWS EMRAmazon AthenaAWS S3AWS IAMAWS CloudWatchAWS EC2/ECRAmazon TextractAmazon ComprehendTerraformDockerGitLab CI/CDGitLab Tags","description_format":"text","description_chars":4894,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[{"name":"United Kingdom","iso":"GB","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Professional Services","Travel & Tourism","E-learning"],"lifecycle":[{"event":"open","at":"2026-10-20T13:00:11Z"}],"visa":[{"country":"GB","licensed_sponsor":true,"evidence":"Licensed UK visa sponsor (Skilled Worker)","filings_12m":null,"filings_prev_12m":null,"green_card_filings_12m":null,"median_offered_wage_usd":null,"route":"Skilled Worker","cap_exempt":false,"checked_at":"2026-10-27T06:21:15+00:00","sources":["UK Home Office: register of licensed sponsors (workers)"],"filings_for_role_12m":0}],"liveness":{"score":4,"band":"cold","label":"Long shot","p_open":1,"p_active":0.153,"p_room":0.28,"age_days":60,"expected_fill_days":9,"reasons":["conf:2","stale_co","velocity","win:tail","crowd:brand"],"computed_at":"2026-10-27T05:46:45Z"},"pay":{"stated_usd_annual":86499,"is_top_pay":false},"html_url":"https://alion.io/job/scrumconnect-data-engineer","json_url":"https://alion.io/job/scrumconnect-data-engineer.json","meta":{"generated_at":"2026-10-27T10:25:01Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":11950,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-28T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":3861039},"rest":"https://alion.io/mcp/rest/get_company?id=3861039"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fscrumconnect-data-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fscrumconnect-data-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fscrumconnect-data-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/scrumconnect-data-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fscrumconnect-data-engineer"}]}