{"id":4016750,"url":"https://alion.io/job/cloudandthings-senior-data-engineer","title":"Senior Data Engineer","company":{"id":3881362,"name":"cloudandthings","domain":"cloudandthings.io","url":"https://alion.io/company/cloudandthings","size_band":null,"is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":null,"truth_index":null},"role":"Data Science","role_family":"Data Science","seniority":"senior","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Cape Town, South Africa"],"countries":["ZA"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Airflow","optional":false},{"name":"Amazon Aurora","optional":false},{"name":"Amazon Kinesis","optional":false},{"name":"Amazon Redshift","optional":false},{"name":"Amazon S3","optional":false},{"name":"Apache HTTP Server","optional":false},{"name":"Apache Hudi","optional":false},{"name":"Apache Iceberg","optional":false},{"name":"Apache Kafka","optional":false},{"name":"AWS","optional":false},{"name":"AWS Glue","optional":false},{"name":"AWS Lambda","optional":false},{"name":"AWS Step Functions","optional":false},{"name":"Azure","optional":false},{"name":"CI/CD","optional":false},{"name":"CloudFormation","optional":false},{"name":"Databricks","optional":false},{"name":"Delta Lake","optional":false},{"name":"Docker","optional":false},{"name":"DynamoDB","optional":false},{"name":"ElasticSearch","optional":false},{"name":"ETL/ELT","optional":false},{"name":"Git","optional":false},{"name":"Hadoop","optional":false},{"name":"IAM","optional":false},{"name":"Linux","optional":false},{"name":"Machine Learning","optional":false},{"name":"Memcached","optional":false},{"name":"Microsoft Fabric","optional":false},{"name":"MySQL","optional":false},{"name":"OpenSearch","optional":false},{"name":"PostgreSQL","optional":false},{"name":"pySpark","optional":false},{"name":"Python","optional":false},{"name":"Redis","optional":false},{"name":"Spark","optional":false},{"name":"SQL","optional":false},{"name":"Terraform","optional":false}],"status":"live","first_seen_at":"2026-11-10T11:51:55Z","employer_posted_date":null,"last_verified_at":"2026-11-10T11:51:55Z","board_verified":false,"closed_at":null,"days_open":0,"trust":{"level":"not_scored","repost_count":null,"flags":[],"days_open":0},"description":"Introduction\nWe aim to build one of the strongest Engineering capabilities within Africa. Our Data Engineers are key to help us achieve this.\n\nData Engineers at cloudandthings.io are responsible for building and operating data platforms anddata pipelines, primarily using the AWS, Microsoft Fabric, or Databricks.\n\nWe are seeking highly skilled and experienced Data Engineers to join our team.\n\nThis role involves designing, building, and optimising data architecture and data pipelines. The ideal candidate will have a strong background in data processing technologies and experience in cloud environments.\n\nWhile this list is long, an ideal candidate should have working knowledge and experience covering many of the tools and services. The requirements for each project differ over time, and these skills provide an overview of what may typically be required of a Data Engineer.\n\nDuties & Responsibilities\nThe following outlines the primary responsibilities for an Data Engineer, but the role will require flexibility and adaptation as the business expands.\n\nSoftware Engineering - Fundamentals\n\nA fundamental software engineering skill set underpins all engineering work at cloudandthings.io.\n\n Experience with modern operating systems, particularly Linux.\n\n Experience working with terminals / cli.\n\n Experience with version control software, particularly Git.\n\n Software fundamentals, such as Problem-solving, data structures and algorithms, software development methodologies, common design patterns and best practices.\n\n Experience with the following programming languages, preferably more. Python and SQL.\n\nCloud - Core\n\n Ability to identify serverless options, managed options, and roll-your-own options, strengths and weaknesses.\n\n Development experience working with Terraform or Cloud Formation IAC to provision and maintain data infrastructure.\n\n Familiarity with AWS well architected principles and experience implementing them.\n\nData Engineering - General\n\n Working knowledge of Big Data: Volume, Variety, Velocity, etc.\n\nData Engineering - Collection\n\n Good experience collecting data in hybrid environments: on-premise to cloud, and cloud to on-premise.\n\n Real-time: AWS Kinesis Data Streams (KDS), Kafka / MSK.\n\n Near Real-time: AWS Kinesis Data Firehose (KDF).\n\n Batch: AWS DataSync, StorageGateway, TransferFamily (FTP / SFTP / MFT), Snowball.\n\n Databases: ODBC / JDBC, database replicas and replication tools, migration tools such as Database Migration Service (DMS) and SCT.\n\nData Engineering - Storage\n\n Basic experience working with on-premise storage solutions: NFS / SMB, NAS / DAS, etc.\n\n Cloud Storage: Amazon S3.\n\n Data Formats: Parquet, CSV, Avro, JSON etc., compression, partitioning.\n\n NoSQL Databases: AWS DynamoDB, MongoDB, etc.\n\n Relational Databases: AWS RDS or similar, MySQL / PostgreSQL, Aurora.\n\n Massively Parallel Processing: Redshift.\n\n Search Databases: AWS Elasticsearch / OpenSearch.\n\n Caching: Redis / Memcached.\n\nData Engineering - Processing\n\n Strong experience developing ETL processes, and integrating with source and destination systems.\n\n Strong experience developing using Python, Spark (e.g. PySpark), and SQL to work with data.\n\n Basic experience with Lakehouse technologies such as Apache Hudi, Apache Iceberg or Databricks DeltaLake.\n\n AWS Lambda for file / stream / event processing, ETL, and triggers.\n\n General ETL and cataloguing of data, access control: AWS Glue ETL, Glue catalog, LakeFormation.\n\n Hadoop-like processing: Mainly Spark and Hive, Instance types and cluster and job sizing; AWS Elastic MapReduce (EMR).\n\nData Engineering - Analysis\n\n Basic understanding of cloud data warehouse architecture and data integration: AWS Redshift and Redshift spectrum.\n\n Data modelling skills, normalisation, facts and dimensions.\n\n Experience in data quality.\n\n On-object-store querying: AWS Athena, Glue Crawlers.\n\nData Engineering - Security\n\n Basic experience with authentication and identity federation, authorisation and RBAC pertaining to data.\n\n Basic knowledge of cloud network security: AWS VPC, VPC endpoints, Subnets, DirectConnect.\n\n Identity and Access Management: AWS IAM, STS and cross-account access.\n\n Encryption for data at rest, and data in motion for all services used: AWS KMS / SSE, TLS, etc.\n\nData Engineering - Operations\n\n Orchestration of data pipelines: AWS Step Functions, Managed Apache Airflow, Glue, etc.\n\n Basic knowledge of good architecture pillars and how to apply them:\n\n Operational excellence, Security, and Reliability.\n\n Performance Efficiency, Cost Optimisation, and Sustainability.\n\nAdvantageous technical skills\n\n Any other data-related experience, e.g. working with Hadoop, databases, analytics software, etc.\n\n Experience working with a second cloud vendor, e.g. both AWS and Azure.\n\n Experience working with Docker / Containers / CICD pipelines for data.\n\n Experience working with and contributing to open-source projects.\n\nDesired Experience & Qualification\n\n Bachelor’s or Master's degree in Computer Science, Engineering, Information Technology, or related field.\n\n 5+ years of experience in a data engineering role with a focus on cloud technologies, specifically Amazon Web Services, Microsoft Azure, or Databricks.\n\n Knowledge of data related Amazon Web Services and understanding of best practices in Data Architecture and Data Modelling.\n\n Strong proficiency in SQL and experience with scripting languages such as Python.\n\n Experience with big data technologies (e.g., Hadoop, Spark) and familiarity with machine learning algorithms.\n\n Experience in real-time data processing and analytics.\n\n Knowledge of data integration and enterprise integration patterns.\n\n Familiarity with DevOps practices and tools related to data management (CI / CD for data pipelines, version control, etc.).\n\n Excellent problem-solving skills and the ability to work collaboratively in a team environment.\n\n Strong communication and interpersonal skills to interact with team members and stakeholders.\n\n Proven track record of designing and implementing data solutions.\n\n Knowledge of and experience with Cloud infrastructure and services.\n\n Understanding of Cloud Security best practices.\n\n Willingness to learn and expand knowledge related to Cloud and Data Technologies.\n\n Strong problem-solving and analytical skills.\n\n Self-organising with the ability to prioritise and manage multiple tasks simultaneously.\n\n Willingness to travel to clients as and when required.\n\n Any of the following certifications:\n\n AWS Certified Data Engineer - Associate\n\n AWS Certified Machine Learning – Specialty\n\n AWS Solutions Architect - Associate\n\n AWS Solutions Architect - Professional\n\n Databricks Certified Data Analyst Associate\n\n Databricks Certified: Data Engineer Associate\n\n Databricks Certified: Data Engineer Professional\n\n Databricks Certified Machine Learning Associate\n\n Databricks Certified Machine Learning Professional\n\n Microsoft Certified: Azure Data Engineer Associate (DP-203)\n\n Microsoft Certified: Azure AI Engineer Associate (AI-102)\n\n Microsoft Certified: Fabric Analytics Engineer Associate (DP-600)\n\n Microsoft Certified: Fabric Data Engineer Associate (DP-700)\n\n Microsoft Certified: Azure Data Scientist Associate (DP-100)\n\n Microsoft Certified: Azure Solutions Architect Expert (AZ-305)\n\nPackage & Remuneration\n\n A culture of engineering and an environment where ideas are heard and builders can build.\n\n Competitive salary, bonus, and incentive structure.\n\n A flexible and supportive work environment that values diversity, work-life balance, and personal growth.\n\n Opportunities for career advancement and ongoing professional development.\n\n The chance to work on cutting-edge products and technologies that make a real impact on people's lives.\n\n Mentoring and knowledge-sharing opportunities that foster personal and professional growth, with access to experienced leaders and a supportive peer network.","description_format":"text","description_chars":7916,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":["Professional development"],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Predictive Analytics","Loyalty Programs","Industrial Real Estate"],"lifecycle":[{"event":"open","at":"2026-11-10T12:57:53Z"}],"visa":[],"liveness":{"score":86,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.86,"p_room":1,"age_days":0,"expected_fill_days":29,"reasons":["seen:0","win:early"],"computed_at":"2026-11-10T15:19:53Z"},"pay":null,"html_url":"https://alion.io/job/cloudandthings-senior-data-engineer","json_url":"https://alion.io/job/cloudandthings-senior-data-engineer.json","meta":{"generated_at":"2026-11-10T15:19:53Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":1367,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-11T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":3881362},"rest":"https://alion.io/mcp/rest/get_company?id=3881362"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fcloudandthings-senior-data-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":2,"usd":0.2},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fcloudandthings-senior-data-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fcloudandthings-senior-data-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/cloudandthings-senior-data-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fcloudandthings-senior-data-engineer"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":4016750}},{"tool":"search_jobs","arguments":{"company":"cloudandthings.io","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":4016750},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}