{"id":3974100,"url":"https://alion.io/job/andi-shi-website-senior-azure-data-engineering-expert","title":"Senior Azure Data Engineering Expert","company":{"id":4489316,"name":"Andi Shi Website","domain":"andishi.com","url":"https://alion.io/company/andishi","size_band":null,"is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":null,"truth_index":null},"role":"DevOps","role_family":"DevOps","seniority":"senior","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Nairobi, Kenya"],"countries":["KE"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Azure","optional":false},{"name":"Azure Data Factory","optional":false},{"name":"Azure DevOps","optional":false},{"name":"CI/CD","optional":false},{"name":"Databricks","optional":false},{"name":"ETL/ELT","optional":false},{"name":"Git","optional":false},{"name":"GitHub","optional":false},{"name":"Machine Learning","optional":false},{"name":"pySpark","optional":false},{"name":"Rest API","optional":false},{"name":"Spark","optional":false},{"name":"SQL","optional":false},{"name":"Unstructured.io","optional":false},{"name":".NET","optional":true},{"name":"Azure Cosmos DB","optional":true},{"name":"Azure SQL Database","optional":true},{"name":"C#","optional":true},{"name":"Java","optional":true},{"name":"JavaScript","optional":true},{"name":"MS SQL","optional":true},{"name":"Node JS","optional":true},{"name":"Python","optional":true},{"name":"Unix","optional":true}],"status":"live","first_seen_at":"2026-11-09T15:04:52Z","employer_posted_date":null,"last_verified_at":"2026-11-09T15:04:52Z","board_verified":false,"closed_at":null,"days_open":1,"trust":{"level":"not_scored","repost_count":null,"flags":[],"days_open":1},"description":"Project Overview\n\n We are seeking an experienced Senior Data Engineer with strong Azure cloud, data engineering, and AI/ML platform development expertise to independently design, develop, and deliver a scalable enterprise-grade data platform.\n\n The project involves building and improving data processing solutions, developing reliable data pipelines, integrating cloud-based services, and preparing data infrastructure to support AI, machine learning, and advanced analytics applications.\n\n This is an independent contract engagement. The selected developer will take end-to-end responsibility for technical execution, from architecture and implementation through testing, documentation, and final delivery.\n\n We are looking for someone who can work autonomously, make sound technical decisions, troubleshoot complex problems, and deliver production-ready solutions.\n\nKey Responsibilities\n\nData Platform Architecture and Development\n\n Design and implement scalable Azure-based data engineering architectures.\n\n Develop enterprise-grade data platforms supporting structured and unstructured datasets.\n\n Build batch and real-time data processing solutions.\n\n Design efficient data storage, processing, and integration architectures.\n\n Identify opportunities to improve platform performance, reliability, scalability, and cost efficiency.\n\nData Engineering and ETL Development\n\n Build, maintain, and optimize end-to-end ETL/ELT pipelines.\n\n Develop data ingestion and transformation workflows.\n\n Implement data integration between internal systems, external APIs, and cloud services.\n\n Develop advanced SQL queries, stored procedures, database functions, and data models.\n\n Implement data validation, error handling, and data quality controls.\n\nAzure Cloud and Distributed Processing\n\n Develop solutions using Azure Data Factory, Azure Functions, Event Hubs, and related cloud-native services.\n\n Implement event-driven architectures using Azure Event Grid and Service Bus.\n\n Work with Azure Databricks and distributed processing technologies.\n\n Optimize data workflows for high-volume and real-time processing.\n\n Support cloud migration, integration, and infrastructure improvements.\n\nAI and Machine Learning Data Infrastructure\n\n Develop data processing frameworks supporting AI/ML applications.\n\n Prepare and transform datasets for machine learning and advanced analytics.\n\n Implement reliable data workflows throughout the AI/ML data lifecycle.\n\n Develop data algorithms and processing logic for analytical applications.\n\n Ensure data pipelines are scalable, consistent, and suitable for downstream AI/ML workloads.\n\nSoftware Development and Integration\n\n Develop backend services and RESTful APIs.\n\n Integrate databases, cloud applications, and third-party systems.\n\n Implement secure and maintainable application components.\n\n Follow software engineering best practices, including version control, code reviews, and automated testing.\n\nDeployment, Testing, and Documentation\n\n Configure and maintain CI/CD pipelines using Azure DevOps and GitHub.\n\n Perform unit, integration, and end-to-end testing.\n\n Identify and resolve technical issues independently.\n\n Produce technical documentation, architecture diagrams, and implementation guides.\n\n Provide a complete handover of the developed solution.\n\nRequired Technical Stack\n\n Cloud and Infrastructure\n\n Microsoft Azure\n\n Azure Data Factory\n\n Azure Databricks\n\n Azure Functions\n\n Azure Event Hubs\n\n Azure Event Grid\n\n Azure Service Bus\n\n Azure Cosmos DB\n\n Azure SQL Data Warehouse / Azure SQL\n\n Programming and Development\n\n Python\n\n SQL / T-SQL\n\n C# / .NET\n\n Java\n\n Node.js\n\n Unix Shell Scripting\n\n Data Engineering and Processing\n\n ETL / ELT\n\n Apache Spark / PySpark\n\n Spark SQL\n\n Hive SQL\n\n Data Modeling\n\n RDBMS\n\n Structured and Unstructured Data Processing\n\n Batch and Stream Processing\n\n Event-Driven Architecture\n\n DevOps and Integration\n\n Azure DevOps\n\n Git / GitHub\n\n CI/CD Pipelines\n\n RESTful APIs\n\n JSON / JSON-LD / XML\n\n Version Control and Automated Deployment","description_format":"text","description_chars":4024,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Psychotherapy & Counseling","Coaching & Mentoring"],"lifecycle":[{"event":"open","at":"2026-11-09T17:15:00Z"}],"visa":[],"liveness":{"score":86,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.86,"p_room":1,"age_days":0,"expected_fill_days":29,"reasons":["seen:0","win:early"],"computed_at":"2026-11-10T06:52:35Z"},"pay":null,"html_url":"https://alion.io/job/andi-shi-website-senior-azure-data-engineering-expert","json_url":"https://alion.io/job/andi-shi-website-senior-azure-data-engineering-expert.json","meta":{"generated_at":"2026-11-11T03:50:54Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":3549,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-12T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":4489316},"rest":"https://alion.io/mcp/rest/get_company?id=4489316"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fandi-shi-website-senior-azure-data-engineering-expert"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":2,"usd":0.2},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fandi-shi-website-senior-azure-data-engineering-expert"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fandi-shi-website-senior-azure-data-engineering-expert"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/andi-shi-website-senior-azure-data-engineering-expert\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fandi-shi-website-senior-azure-data-engineering-expert"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":3974100}},{"tool":"search_jobs","arguments":{"company":"andishi.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":3974100},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}