{"id":3793245,"url":"https://alion.io/job/cloudthat-sr-data-engineer-aws-multi-cloud-databricks","title":"Sr Data Engineer (AWS | Multi-Cloud | Databricks)","company":{"id":2540181,"name":"CloudThat","domain":"cloudthat.com","url":"https://alion.io/company/cloudthat","size_band":"1001-5000","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Keka","truth_index":{"grade":"C","score":59,"open_postings":81,"ghost_share":0.691,"stale_share":0,"repost_share":0,"time_to_fill_p50_days":null,"computed_at":"2026-11-07T05:47:08Z"}},"role":"Data Science","role_family":"Data Science","seniority":"senior","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Bengaluru, India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":5,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Airflow","optional":false},{"name":"Amazon S3","optional":false},{"name":"Apache Iceberg","optional":false},{"name":"AWS","optional":false},{"name":"AWS Step Functions","optional":false},{"name":"Azure","optional":false},{"name":"Databricks","optional":false},{"name":"Delta Lake","optional":false},{"name":"ETL/ELT","optional":false},{"name":"GCP","optional":false},{"name":"pySpark","optional":false},{"name":"Python","optional":false},{"name":"Spark","optional":false},{"name":"SQL","optional":false},{"name":"Apache Kafka","optional":true},{"name":"CI/CD","optional":true},{"name":"CloudFormation","optional":true},{"name":"Terraform","optional":true}],"status":"live","first_seen_at":"2026-11-05T05:55:47Z","employer_posted_date":"2026-11-05","last_verified_at":"2026-11-08T03:19:14Z","board_verified":true,"closed_at":null,"days_open":2,"trust":{"level":"ok","repost_count":0,"flags":["company_stale"],"days_open":1},"description":"Job Description - Data Engineer (AWS | Multi-Cloud | Databricks)\nExperience: 5+ Years\nRole: Senior Data Engineer\nPrimary Cloud Platform: AWS\nSecondary Cloud Platforms: GCP and Microsoft Azure\nCore Technology: Databricks, Apache Spark, PySpark, SQL, Data Engineering\nJob Overview\nWe are looking for an experienced Senior Data Engineer with 5+ years of hands-on experience in designing, developing, and implementing scalable data engineering solutions across cloud platforms.\nThe ideal candidate should have strong expertise in AWS as the primary cloud platform, along with working knowledge and exposure to data engineering services across Google Cloud Platform (GCP) and Microsoft Azure.\nThe candidate must possess excellent data engineering fundamentals, strong programming and problem-solving skills, and very good hands-on experience with Databricks and Apache Spark.\nThe role involves building modern data pipelines, implementing data lakes and lakehouse architectures, optimizing large-scale data processing workloads, and developing reliable, scalable, and cost-efficient data platforms.\nKey Responsibilities\nDesign, develop, and maintain scalable end-to-end data pipelines using AWS and Databricks.\nBuild and optimize batch and real-time data processing solutions using Apache Spark, PySpark, and SQL.\nDesign and implement modern data lake and lakehouse architectures using S3, Delta Lake, Apache Iceberg, and Databricks.\nWork with AWS data engineering services for data ingestion, transformation, orchestration, storage, cataloging, and analytics.\nDevelop ETL/ELT workflows integrating structured, semi-structured, and unstructured data from multiple sources.\nImplement data transformation, data quality, validation, and reconciliation frameworks.\nOptimize Spark jobs, SQL queries, cluster configurations, and distributed processing workloads.\nImplement workflow orchestration using Airflow, AWS Step Functions, Databricks Workflows, or equivalent services.\nWork with streaming technologies and services for real-time data ingestion and processing.\nDevelop reusable frameworks, automation scripts, and standardized data engineering components.\nImplement security, access control, encryption, monitoring, logging, and governance across data platforms.\nCollaborate with architects, analysts, data scientists, and business stakeholders to translate requirements into scalable technical solutions.\nParticipate in architecture discussions, technical design, code reviews, and performance optimization initiatives.\n\nRequired Experience & Qualifications\n5+ years of experience in Data Engineering.\nStrong hands-on experience with AWS data engineering services.\nVery good practical experience with Databricks and Apache Spark.\nStrong programming skills in Python, PySpark, and SQL.\nPractical exposure to at least one of GCP or Azure, with familiarity with the data engineering ecosystem of both.\nExperience designing and implementing end-to-end data pipelines.\nUnderstanding of modern data lakehouse architecture and distributed data processing.\nExperience working with large-scale datasets and performance optimization.\n\nGood to Have\nExperience working on multi-cloud data engineering architectures.\nExposure to AWS, Azure, and GCP migrations.\nExperience with Apache Kafka, Amazon MSK, or other streaming platforms.\nKnowledge of Apache Iceberg, Delta Lake, and open table formats.\nExposure to data governance, lineage, and metadata management.\nKnowledge of DevOps practices and CI/CD automation.\nExperience with Infrastructure as Code (Terraform / CloudFormation).\nExposure to Generative AI data pipelines and vector databases.\n\nIdeal Candidate Profile\nThe ideal candidate should be a hands-on Data Engineer who can independently design, develop, troubleshoot, and optimize complex data engineering pipelines.\nWe are specifically looking for someone who:\nHas strong practical expertise in AWS data engineering services.\nHas very good hands-on experience with Databricks and PySpark.\nUnderstands data engineering ecosystems across AWS, GCP, and Azure.\nHas strong fundamentals in distributed data processing, SQL, Python, and data architecture.\nCan work independently on end-to-end data engineering implementations.\nCan contribute to technical architecture and engineering best practices.","description_format":"text","description_chars":4287,"description_truncated":false,"requirements":{"experience_years_min":5,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["IT Consulting & Digital Transformation","Engineering Services","Cloud Consulting & Migration","IT Training & Certification"],"lifecycle":[{"event":"open","at":"2026-11-05T07:36:51Z"}],"visa":[],"liveness":{"score":63,"band":"ok","label":"Likely open","p_open":1,"p_active":0.632,"p_room":1,"age_days":1,"expected_fill_days":30,"reasons":["conf:3","stale_co","velocity","win:early"],"computed_at":"2026-11-07T05:47:08Z"},"pay":null,"html_url":"https://alion.io/job/cloudthat-sr-data-engineer-aws-multi-cloud-databricks","json_url":"https://alion.io/job/cloudthat-sr-data-engineer-aws-multi-cloud-databricks.json","meta":{"generated_at":"2026-11-08T05:16:09Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":11417,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-09T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":2540181},"rest":"https://alion.io/mcp/rest/get_company?id=2540181"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fcloudthat-sr-data-engineer-aws-multi-cloud-databricks"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fcloudthat-sr-data-engineer-aws-multi-cloud-databricks"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fcloudthat-sr-data-engineer-aws-multi-cloud-databricks"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/cloudthat-sr-data-engineer-aws-multi-cloud-databricks\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fcloudthat-sr-data-engineer-aws-multi-cloud-databricks"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":3793245}},{"tool":"search_jobs","arguments":{"company":"cloudthat.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":3793245},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}