{"id":4334178,"url":"https://alion.io/job/teamstation-data-engineer-lakehouse-spark-nifi","title":"Data Engineer – Lakehouse / Spark / NiFi","company":{"id":2889395,"name":"TeamStation AI","domain":"teamstation.dev","url":"https://alion.io/company/teamstation","size_band":"51-200","is_staffing_agency":true,"employer_type":"agency","is_intermediary":false,"listed_via":null,"ats_vendor":null,"truth_index":{"grade":"D","score":41,"open_postings":148,"ghost_share":0.986,"stale_share":0,"repost_share":0,"time_to_fill_p50_days":null,"computed_at":"2026-11-20T05:46:30Z"}},"role":"Data Science","role_family":"Data Science","seniority":"senior","employment_type":"full_time","work_mode":"remote","remote_scope":"stated_regions","remote_scope_basis":"board_field","remote_working_hours":null,"hiring_geo_confidence":"structured","locations":[],"countries":[],"hiring_countries":["US","AR","BR","CO","MX","AI","AG","AW","BS","BB","BZ","BM","BO","VG","BQ","KY","CL","CR","CU","CW","DM","DO","EC","SV","FK","GF","GD","GP","GT","GY","HT","HN","JM","MQ","MS","NI","PA","PY","PE","PR"],"hiring_countries_total":51,"salary":null,"salary_estimate":null,"experience_years_min":5,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Airflow","optional":false},{"name":"Amazon S3","optional":false},{"name":"Apache Hudi","optional":false},{"name":"Apache Iceberg","optional":false},{"name":"Apache NiFi","optional":false},{"name":"Azure","optional":false},{"name":"CI/CD","optional":false},{"name":"Delta Lake","optional":false},{"name":"ETL/ELT","optional":false},{"name":"Git","optional":false},{"name":"HIPAA","optional":false},{"name":"Presto","optional":false},{"name":"Python","optional":false},{"name":"Scala","optional":false},{"name":"Spark","optional":false},{"name":"SQL","optional":false},{"name":"Superset","optional":false},{"name":"Trino","optional":false},{"name":"Looker","optional":true},{"name":"Open Policy Agent","optional":true},{"name":"Power BI","optional":true}],"status":"live","first_seen_at":"2025-05-28T00:00:00Z","employer_posted_date":"2026-11-17","last_verified_at":"2026-11-19T11:59:46Z","board_verified":false,"closed_at":null,"days_open":543,"trust":{"level":"ghost","repost_count":0,"flags":["stale","company_stale"],"days_open":542},"description":"At TeamStation AI, we are on a mission to bring together the brightest minds to solve tomorrow’s toughest technology challenges.\nOur work is about more than just AI-it’s about building the future through collaboration and innovation.\nWe believe that the key to solving the world’s most complex problems lies in aligning diverse talents and perspectives. Our AI-powered platform enables cutting-edge scientific and technical teams to work smarter, faster, and together. By joining us, you’ll help unlock new technological breakthroughs and drive innovation where it matters most.\nJoin the Mission at TeamStation AI!\nWhere do we come from?\nWe are seeking visionaries, innovators, and problem solvers who thrive in fast-paced, collaborative environments. If you’re passionate about AI, technology, and solving critical challenges, we want to hear from you. Come be part of a team where your ideas can drive the future.\nData Engineer - Lakehouse / Spark / NiFi: Architecting Next-Gen Data Platforms\nLocation: Remote (Latin America)\nWe are seeking a highly skilled and experienced Data Engineer to join our team. In this pivotal role, you will be instrumental in designing and implementing a scalable, open-source-driven data lakehouse platform. This is a hands-on position focused on building high-performance pipelines, optimizing schema evolution, and supporting sophisticated analytical workloads.\nKey Responsibilities\nSchema Management: Lead the design and management of Apache Iceberg schemas across Bronze, Silver, and Gold data layers, with a strong emphasis on partitioning strategies, metadata optimization, and time-travel capabilities.\nETL Pipeline Development: Develop and orchestrate robust Bronze-to-Silver and Silver-to-Gold data pipelines utilizing Apache NiFi, Apache Airflow, and/or Apache Spark.\nData Integration & Tuning: Integrate and fine-tune ETL processes for diverse structured and semi-structured data formats, including CSV, JSON, and EDI.\nData Quality & Governance: Implement comprehensive data quality, deduplication, and change detection logic, employing strategies such as hash-based methodologies.\nAnalytics Collaboration: Collaborate effectively with analytics teams to ensure performant Trino queries and support dashboarding initiatives within Superset.\nTech Stack & Tools\nTable Formats:Apache Iceberg, Delta Lake, Apache Hudi, Hive external tables\nETL & Transformation: Apache Spark (SQL/DataFrame API), Apache NiFi, Apache Airflow\nProgramming Languages: Python, Scala\nDistributed Query Engines:Trino, Presto, Athena\nCloud & Storage:Azure Data Lake Storage Gen2, S3, GCS\nMonitoring & Version Control:Git\nDesired Qualifications\nData Engineering Experience: 5+ years of hands-on experience in Data Engineering with a proven track record of developing robust ETL pipelines.\nTable Format Expertise: Solid experience with table formats like Apache Iceberg (preferred!), Delta Lake, Apache Hudi, or Hive external tables with ORC/Parquet. Proficiency in managing ACID tables and schema evolution in a data lake context is essential.\nETL & Orchestration: Demonstrated proficiency with Apache Spark (SQL/DataFrame API) for building large-scale batch and streaming pipelines. Experience with Apache NiFi, Apache Airflow, or similar orchestration frameworks is required.\nProgramming Prowess: Expert-level proficiency in Python and/or Scala.\nQuery Engine Proficiency: Hands-on experience with distributed query engines such as Trino, Presto, or Athena, combined with strong SQL skills and an understanding of query optimization, predicate pushdown, and partition pruning.\nCloud & Storage Acumen: Practical knowledge of Azure Data Lake Storage Gen2, S3, or GCS, including object storage layout and file formats (Parquet, Avro). Understanding of access control mechanisms (e.g., Azure RBAC, managed identity).\nData Architecture & Governance: Familiarity with Bronze/Silver/Gold layer design in lakehouse environments. Experience with hashing strategies for change detection and time-travel queries. Comfort working in HIPAA/PII-sensitive environments, with a focus on de-identification and auditability.\nDevOps & CI/CD: Hands-on experience with Git for managing data schema and script assets.\nNice-to-Have (Bonus Points!)\nExperience with Apache Iceberg utilizing a JDBC or Nessie catalog.\nFamiliarity with Open Policy Agent (OPA) or column-/row-level security.\nPrior experience in highly regulated environments such as healthcare or financial services.\nExposure to Superset, Looker, or Power BI on top of data lake engines.\nWhat We Offer\nCompetitive Compensation: A highly competitive salary package reflecting your expertise and contributions.\nFully Remote Work: Enjoy the flexibility of a fully remote work environment within the US.\nCareer Growth & Development: Opportunities for continuous learning, professional development, mentorship, and upskilling in cutting-edge technologies.\nImpactful Projects: Contribute to high-impact data workflows and help shape a modern, open-source lakehouse architecture from the ground up as part of a TeamStation AI partnership.\nCollaborative Environment: Work alongside a cross-functional team of talented engineers and analysts.\nLearn more about TeamStation AI Careers and our mission: https://teamstation.dev","description_format":"text","description_chars":5268,"description_truncated":false,"requirements":{"experience_years_min":5,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":["Continuous learning","Professional development"],"hiring_locations":[{"name":"United States","iso":"US","kind":"country"},{"name":"Argentina","iso":"AR","kind":"country"},{"name":"Brazil","iso":"BR","kind":"country"},{"name":"Colombia","iso":"CO","kind":"country"},{"name":"Mexico","iso":"MX","kind":"country"},{"name":"Anguilla","iso":"AI","kind":"region"},{"name":"Antigua and Barbuda","iso":"AG","kind":"region"},{"name":"Aruba","iso":"AW","kind":"region"},{"name":"Bahamas","iso":"BS","kind":"region"},{"name":"Barbados","iso":"BB","kind":"region"},{"name":"Belize","iso":"BZ","kind":"region"},{"name":"Bermuda","iso":"BM","kind":"region"},{"name":"Bolivia","iso":"BO","kind":"country"},{"name":"British Virgin Islands","iso":"VG","kind":"region"},{"name":"Bonaire, Sint Eustatius and Saba","iso":"BQ","kind":"region"},{"name":"Cayman Islands","iso":"KY","kind":"region"},{"name":"Chile","iso":"CL","kind":"country"},{"name":"Costa Rica","iso":"CR","kind":"country"},{"name":"Cuba","iso":"CU","kind":"country"},{"name":"Curaçao","iso":"CW","kind":"region"},{"name":"Dominica","iso":"DM","kind":"region"},{"name":"Dominican Republic","iso":"DO","kind":"country"},{"name":"Ecuador","iso":"EC","kind":"country"},{"name":"El Salvador","iso":"SV","kind":"country"},{"name":"Falkland Islands","iso":"FK","kind":"region"},{"name":"French Guiana","iso":"GF","kind":"region"},{"name":"Grenada","iso":"GD","kind":"region"},{"name":"Guadeloupe","iso":"GP","kind":"region"},{"name":"Guatemala","iso":"GT","kind":"country"},{"name":"Guyana","iso":"GY","kind":"region"},{"name":"Haiti","iso":"HT","kind":"region"},{"name":"Honduras","iso":"HN","kind":"country"},{"name":"Jamaica","iso":"JM","kind":"region"},{"name":"Martinique","iso":"MQ","kind":"region"},{"name":"Montserrat","iso":"MS","kind":"region"},{"name":"Nicaragua","iso":"NI","kind":"country"},{"name":"Panama","iso":"PA","kind":"country"},{"name":"Paraguay","iso":"PY","kind":"country"},{"name":"Peru","iso":"PE","kind":"country"},{"name":"Puerto Rico","iso":"PR","kind":"country"},{"name":"Saint Kitts and Nevis","iso":"KN","kind":"region"},{"name":"Saint Lucia","iso":"LC","kind":"region"},{"name":"Saint Vincent and the Grenadines","iso":"VC","kind":"region"},{"name":"Saint-Pierre and Miquelon","iso":"PM","kind":"region"},{"name":"Sint Maarten","iso":"SX","kind":"region"},{"name":"Suriname","iso":"SR","kind":"region"},{"name":"Trinidad and Tobago","iso":"TT","kind":"region"},{"name":"Turks and Caicos Islands","iso":"TC","kind":"region"},{"name":"Uruguay","iso":"UY","kind":"country"},{"name":"United States Virgin Islands","iso":"VI","kind":"region"},{"name":"Venezuela","iso":"VE","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Artificial Intelligence","Information Technology","Machine Learning"],"lifecycle":[{"event":"open","at":"2026-11-15T10:08:53Z"}],"visa":[],"liveness":{"score":3,"band":"cold","label":"Long shot","p_open":1,"p_active":0.118,"p_room":0.28,"age_days":541,"expected_fill_days":38,"reasons":["conf:17","agency","stale_co","ghost","win:tail","crowd:brand"],"computed_at":"2026-11-20T05:46:30Z"},"pay":null,"html_url":"https://alion.io/job/teamstation-data-engineer-lakehouse-spark-nifi","json_url":"https://alion.io/job/teamstation-data-engineer-lakehouse-spark-nifi.json","meta":{"generated_at":"2026-11-22T02:21:31Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":4055,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-23T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":2889395},"rest":"https://alion.io/mcp/rest/get_company?id=2889395"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fteamstation-data-engineer-lakehouse-spark-nifi"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"free","price":{"credits":0,"usd":0},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fteamstation-data-engineer-lakehouse-spark-nifi"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fteamstation-data-engineer-lakehouse-spark-nifi"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/teamstation-data-engineer-lakehouse-spark-nifi\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fteamstation-data-engineer-lakehouse-spark-nifi"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":4334178}},{"tool":"search_jobs","arguments":{"company":"teamstation.dev","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":4334178},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}