{"id":2568958,"url":"https://alion.io/job/cloudary-data-platform-engineer-102-08seng-01","title":"Data Platform Engineer (102-08SENG-01)","company":{"id":4417501,"name":"Cloudary","domain":"cloudary.io","url":"https://alion.io/company/cloudary-2","size_band":null,"is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Recruitee","truth_index":null},"role":"Data Science","role_family":"Data Science","seniority":null,"employment_type":"contractor","work_mode":"remote","remote_scope":"stated_regions","remote_scope_basis":"board_field","remote_working_hours":{"label":"EST","utc_offset_min":-5,"utc_offset_max":-5},"hiring_geo_confidence":"structured","locations":[],"countries":[],"hiring_countries":["AR","BR","MX","AG","AW","BS","BB","BZ","BM","BO","VG","BQ","KY","CL","CR","CW","DM","DO","EC","SV","FK","GF","GD","GP","GT","GY","HT","HN","JM","MQ","MS","NI","PA","PY","PE","PR","KN","LC","PM","SX"],"hiring_countries_total":45,"salary":null,"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Amazon S3","optional":false},{"name":"AWS","optional":false},{"name":"AWS Lambda","optional":false},{"name":"AWS Step Functions","optional":false},{"name":"Azure","optional":false},{"name":"Azure Data Factory","optional":false},{"name":"Databricks","optional":false},{"name":"pySpark","optional":false},{"name":"Python","optional":false},{"name":"Spark","optional":false},{"name":"SQL","optional":false},{"name":"Apache Kafka","optional":true},{"name":"dbt","optional":true},{"name":"Delta Lake","optional":true},{"name":"Great Expectations","optional":true},{"name":"SAS","optional":true},{"name":"Terraform","optional":true}],"status":"live","first_seen_at":"2026-08-20T19:13:01Z","employer_posted_date":"2026-08-20","last_verified_at":"2026-10-15T10:46:08Z","board_verified":true,"closed_at":null,"days_open":55,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":55},"description":"This role converts a large Azure Data Factory estate into Databricks workflows on AWS. The scope for year one: 2,094 ADF pipelines to migrate - built as reusable templates rather than one-by-one - 9,859 pipeline activities to translate (some map directly, others need rewriting as Lambda or Step Functions), 471 Spark dataflows to move onto Databricks on AWS, and 4 Databricks workspaces (Dev, QA, Pre-prod, Prod) to rehost, including notebook paths and Unity Catalog rewiring. This is a regulated environment, so reconciling migrated data against source systems is part of the definition of done, not an afterthought.\nRequirements\nWhat you will do\nConvert Azure Data Factory pipelines into Databricks workflows on AWS, building reusable templates rather than migrating one at a time.\n\nRehost Databricks workspaces onto AWS and migrate ADLS Gen2 storage to S3.\n\nRewrite ADF Web Activities as Lambda functions or Step Functions tasks, and replace ADF-specific scaling with native Databricks mechanisms.\n\nBuild and tune PySpark transformations for production data volumes.\n\nReplace Azure Synapse Serverless with Databricks SQL Warehouse.\n\nReconcile migrated data against source systems as part of the definition of done.\n\nRequired\nProduction experience with Databricks: workspaces, jobs and workflows. The central skill for this role.\n\nStrong Spark and PySpark experience for real data volumes, including tuning.\n\nProduction-grade Python.\n\nExperience building or migrating Azure Data Factory pipelines, with a solid understanding of the ADF activity model.\n\nAWS data services: S3, Glue, Athena, Lambda and Step Functions.\n\nAdvanced SQL, including reading and reasoning about stored procedures.\n\nProfessional written and spoken English.\n\nNice to have\nDelta Lake, Unity Catalog, Azure Synapse, Terraform, Airflow/MWAA, dbt, Kafka, Databricks certification, data modeling, Great Expectations, SAS/analytics platform integration, CRM data.\nEngagement details\nEstimated start date: December 2026\n\nDuration of project: 9 months\n\nEST Time Zone\n\nOpen to candidates from all LATAM","description_format":"text","description_chars":2069,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[{"language":"English","level":"All levels","optional":false}]},"benefits":[],"hiring_locations":[{"name":"Argentina","iso":"AR","kind":"region"},{"name":"Brazil","iso":"BR","kind":"country"},{"name":"Colombia","iso":null,"kind":"region"},{"name":"Mexico","iso":"MX","kind":"region"},{"name":"Anguilla","iso":null,"kind":"region"},{"name":"Antigua and Barbuda","iso":"AG","kind":"region"},{"name":"Aruba","iso":"AW","kind":"region"},{"name":"Bahamas","iso":"BS","kind":"region"},{"name":"Barbados","iso":"BB","kind":"region"},{"name":"Belize","iso":"BZ","kind":"region"},{"name":"Bermuda","iso":"BM","kind":"region"},{"name":"Bolivia","iso":"BO","kind":"region"},{"name":"British Virgin Islands","iso":"VG","kind":"region"},{"name":"Bonaire, Sint Eustatius and Saba","iso":"BQ","kind":"region"},{"name":"Cayman Islands","iso":"KY","kind":"region"},{"name":"Chile","iso":"CL","kind":"region"},{"name":"Costa Rica","iso":"CR","kind":"region"},{"name":"Curaçao","iso":"CW","kind":"region"},{"name":"Dominica","iso":"DM","kind":"region"},{"name":"Dominican Republic","iso":"DO","kind":"region"},{"name":"Ecuador","iso":"EC","kind":"region"},{"name":"El Salvador","iso":"SV","kind":"region"},{"name":"Falkland Islands","iso":"FK","kind":"region"},{"name":"French Guiana","iso":"GF","kind":"region"},{"name":"Grenada","iso":"GD","kind":"region"},{"name":"Guadeloupe","iso":"GP","kind":"region"},{"name":"Guatemala","iso":"GT","kind":"region"},{"name":"Guyana","iso":"GY","kind":"region"},{"name":"Haiti","iso":"HT","kind":"region"},{"name":"Honduras","iso":"HN","kind":"region"},{"name":"Jamaica","iso":"JM","kind":"region"},{"name":"Martinique","iso":"MQ","kind":"region"},{"name":"Montserrat","iso":"MS","kind":"region"},{"name":"Nicaragua","iso":"NI","kind":"region"},{"name":"Panama","iso":"PA","kind":"region"},{"name":"Paraguay","iso":"PY","kind":"region"},{"name":"Peru","iso":"PE","kind":"region"},{"name":"Puerto Rico","iso":"PR","kind":"region"},{"name":"Saint Kitts and Nevis","iso":"KN","kind":"region"},{"name":"Saint Lucia","iso":"LC","kind":"region"},{"name":"Saint Vincent and the Grenadines","iso":null,"kind":"region"},{"name":"Saint-Pierre and Miquelon","iso":"PM","kind":"region"},{"name":"Sint Maarten","iso":"SX","kind":"region"},{"name":"Suriname","iso":"SR","kind":"region"},{"name":"Trinidad and Tobago","iso":"TT","kind":"region"},{"name":"Turks and Caicos Islands","iso":"TC","kind":"region"},{"name":"Uruguay","iso":"UY","kind":"region"},{"name":"United States Virgin Islands","iso":"VI","kind":"region"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Information Technology","Managed IT Services (MSP)","Cloud Consulting & Migration"],"lifecycle":[{"event":"open","at":"2026-10-14T22:30:26Z"}],"visa":[],"liveness":{"score":17,"band":"cold","label":"Long shot","p_open":1,"p_active":0.473,"p_room":0.35,"age_days":55,"expected_fill_days":27,"reasons":["conf:3","win:tail"],"computed_at":"2026-10-15T05:46:00Z"},"pay":null,"html_url":"https://alion.io/job/cloudary-data-platform-engineer-102-08seng-01","json_url":"https://alion.io/job/cloudary-data-platform-engineer-102-08seng-01.json","meta":{"generated_at":"2026-10-15T14:01:19Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":21483,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-16T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":4417501},"rest":"https://alion.io/mcp/rest/get_company?id=4417501"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fcloudary-data-platform-engineer-102-08seng-01"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fcloudary-data-platform-engineer-102-08seng-01"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fcloudary-data-platform-engineer-102-08seng-01"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/cloudary-data-platform-engineer-102-08seng-01\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fcloudary-data-platform-engineer-102-08seng-01"}]}