{"id":1472925,"url":"https://alion.io/job/delan-associates-aws-lakehouse-data-engineer","title":"AWS Lakehouse Data Engineer","company":{"id":690244,"name":"Delan Associates","domain":"delanhq.com","url":"https://alion.io/company/delanhq","size_band":null,"is_staffing_agency":true,"employer_type":"agency","is_intermediary":false,"listed_via":null,"ats_vendor":"Breezy","truth_index":{"grade":"B","score":80,"open_postings":44,"ghost_share":0,"stale_share":1,"repost_share":0,"time_to_fill_p50_days":21,"computed_at":"2026-10-20T05:45:00Z"}},"role":"Data Science","role_family":"Data Science","seniority":"senior","employment_type":"contractor","work_mode":"remote","remote_scope":"stated_countries","remote_scope_basis":"board_field","remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Atlanta, United States"],"countries":["US"],"hiring_countries":["US"],"hiring_countries_total":1,"salary":null,"salary_estimate":{"min_usd":114000,"max_usd":208000,"period":"year","method":"role_seniority_country_remote_cell","sample_n":365},"experience_years_min":6,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Amazon Redshift","optional":false},{"name":"Amazon S3","optional":false},{"name":"Anomaly Detection","optional":false},{"name":"Apache Iceberg","optional":false},{"name":"AWS","optional":false},{"name":"AWS Glue","optional":false},{"name":"CI/CD","optional":false},{"name":"Databricks","optional":false},{"name":"ETL/ELT","optional":false},{"name":"IAM","optional":false},{"name":"Least Privilege","optional":false},{"name":"Machine Learning","optional":false},{"name":"pySpark","optional":false},{"name":"Python","optional":false},{"name":"SQL","optional":false},{"name":"Airflow","optional":true},{"name":"Amazon Kinesis","optional":true},{"name":"AWS CDK","optional":true},{"name":"AWS Lambda","optional":true},{"name":"AWS Step Functions","optional":true},{"name":"ChatGPT","optional":true},{"name":"CloudFormation","optional":true},{"name":"Copilot","optional":true},{"name":"Cursor","optional":true},{"name":"Delta Lake","optional":true},{"name":"Docker","optional":true},{"name":"Git","optional":true},{"name":"GitHub Actions","optional":true},{"name":"GraphRAG","optional":true},{"name":"Jenkins","optional":true},{"name":"Knowledge Graph","optional":true},{"name":"Power BI","optional":true},{"name":"Spark","optional":true},{"name":"Tableau","optional":true},{"name":"Terraform","optional":true}],"status":"live","first_seen_at":"2026-09-29T12:21:31Z","employer_posted_date":"2026-09-29","last_verified_at":"2026-10-20T10:33:43Z","board_verified":true,"closed_at":null,"days_open":21,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":21},"description":"Position: AWS Lakehouse Data Engineer\nClearance:Ability to Obtain Public Trust\nLocation: 100% Remote (prefer DMV)\n\nAbility to Obtain Public Trust\nAWS Lakehouse Data Engineer\nWe are seeking an AWS Lakehouse Data Engineer to design, implement, and operate the cloud-native data platform that powers AI/ML, analytics, reporting, and data visualization. You will build a modern lakehouse on Amazon S3 using AWS-native services and open table formats, providing Databricks-like capabilities while maintaining portability, strong governance, cost efficiency, and operational control. You will also develop scalable batch and streaming ingestion, Python and PySpark ETL/ELT pipelines, metadata and governance services, and automated cloud provisioning and CI/CD across environments.\nThis role is ideal for an engineer who enjoys platform building, automation, performance optimization, and enabling advanced analytics through trusted, secure, and well-governed data.\nWhat You Will Do\nBuild and Operate Data Pipelines (Batch and Streaming)Design and implement batch and streaming ingestion from APIs, relational databases, file drops, event streams, and external partners.\nImplement, test, and optimize ETL/ELT pipelines using Python and PySpark to produce curated, analytics-ready datasets for reporting, visualization, and machine learning.\nImplement incremental processing, change data capture (CDC), data contracts, schema validation, and reusable transformation frameworks.\nImprove pipeline reliability through automated testing, orchestration, monitoring, retry handling, and operational runbooks.\nDeliver an AWS-Native Lakehouse Data Platform\nDesign and implement a Delta Lakehouse-style data platform using AWS-native services to provide Databricks-like capabilities for data engineering, analysis, and data visualization.\nBuild and manage a scalable lakehouse on Amazon S3 using Apache Iceberg and open columnar formats such as Apache Parquet.\nImplement SQL-like table reliability for data stored in Amazon S3, including ACID transactions, schema evolution, partition evolution, snapshot isolation, time travel, and rollback capabilities using Apache Iceberg.\nEnable fast, interactive querying of lakehouse data using AWS-native query and compute services such as Amazon Athena, Amazon EMR, AWS Glue, and Amazon Redshift where appropriate.\nOptimize performance and cost through partitioning, compaction, file sizing, statistics, caching, lifecycle policies, and efficient separation of compute and storage.\nEstablish standardized development, test, and production environments with consistent configuration and controlled promotion across stages.\nMetadata, Governance, Access Control, Lineage, and Quality\nImplement data governance and fine-grained access control using AWS-native services, including AWS Lake Formation, AWS Glue Data Catalog, AWS Identity and Access Management (IAM), AWS Key Management Service (KMS), and related security services.\nImplement a managed metadata repository for dataset cataloging, ownership, business definitions, tagging, classification, and discoverability.\nEnable end-to-end lineage from source through transformation and consumption to support auditability, impact analysis, and regulatory requirements.\nApply policy-based access, least-privilege permissions, row-, column-, and cell-level controls where required, data classification, retention, encryption, and secure data handling.\nBuild operational data quality checks for freshness, completeness, uniqueness, validity, consistency, and anomaly detection, and publish measurable SLAs/SLOs.\nAWS Automation, CI/CD, and Operations\nImplement automated AWS provisioning using Infrastructure as Code (IaC) to create consistent environments and secure-by-default baselines.\nBuild and enhance CI/CD for data pipelines and lakehouse components, including automated tests, security checks, validation gates, packaging, deployment, promotion, and rollback strategies.\nImplement observability with centralized metrics, logs, traces, alerts, dashboards, runbooks, and incident-response procedures.\nContinuously evaluate platform performance, scalability, reliability, security, and cost, and implement measurable improvements.\nCross-Team Collaboration and Documentation\nWork closely with data, application, analytics, AI/ML, security, networking, and cloud platform teams to support mission needs and delivery timelines.\nMaintain high-quality engineering documentation, including architecture diagrams, data models, SOPs, interface specifications, operational runbooks, and secure configuration baselines.\nPresent technical findings, trade-offs, risks, and recommendations clearly to technical and non-technical stakeholders.\nWhat You Will Need:\n Bachelor's degree in Engineering, Information Technology, Computer Science, Data Engineering, or a related field, or FOUR (4) years equivalent practical experience in leu of degree.\n SIX (6) years of relevant experience.\nHands-on experience implementing AWS-native data lake or lakehouse architectures using Amazon S3 and services such as AWS Glue, Amazon Athena, Amazon EMR, AWS Lake Formation, and Amazon Redshift.\nStrong experience developing production ETL/ELT pipelines using Python and PySpark, including data modeling, transformation, testing, performance tuning, and error handling.\nHands-on experience with Apache Iceberg, including ACID transactions, snapshots, schema and partition evolution, time travel, table maintenance, and query optimization.\nAdvanced SQL skills and experience supporting analytical queries, semantic layers, reporting tools, and data visualization workloads.\nExperience implementing metadata management and governance capabilities, including cataloging, lineage, ownership, classification, policy enforcement, and fine-grained access controls.\nExperience with AWS security fundamentals, including IAM and least privilege, KMS encryption, secrets management, network security, logging, and secure SDLC practices.\nExperience provisioning AWS resources using IaC and operating data platforms across multiple environments.\nExperience building or operating CI/CD pipelines for data workflows, including testing, packaging, deployment automation, environment promotion, and rollback.\nAbility to troubleshoot distributed data-processing workloads and optimize performance, reliability, and cost.\nWhat Would Be Nice to Have\nHands-on experience with Databricks, Delta Lake, or migrating Databricks workloads to AWS-native services and Apache Iceberg.\nExperience with AWS Step Functions, Amazon Managed Workflows for Apache Airflow (MWAA), Amazon Kinesis, AWS Database Migration Service (DMS), AWS Lambda, Amazon MSK, or similar ingestion and orchestration services.\nExperience with modern DevOps practices and tools such as Git, Terraform, AWS CloudFormation or AWS CDK, Jenkins, AWS CodePipeline, GitHub Actions, and Docker.\nExperience integrating lakehouse data with business intelligence and visualization tools such as Amazon QuickSight, Tableau, or Power BI.\nExperience using AI-assisted coding tools, such as GitHub Copilot, ChatGPT, Cursor, or Kiro, to accelerate implementation while maintaining code quality, testing, review, privacy, and security controls.\nKnowledge graph and Graph RAG experience, including graph modeling, ontology and taxonomy alignment, entity resolution, relationship extraction, and hybrid retrieval that combines graph traversal with semantic or vector search.","description_format":"text","description_chars":7447,"description_truncated":false,"requirements":{"experience_years_min":6,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"bachelor","optional":false},"security_clearance":true,"languages":[]},"benefits":[],"hiring_locations":[{"name":"United States","iso":"US","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":[],"lifecycle":[{"event":"open","at":"2026-09-29T17:19:56Z"}],"visa":[],"liveness":{"score":24,"band":"cold","label":"Long shot","p_open":1,"p_active":0.317,"p_room":0.75,"age_days":20,"expected_fill_days":21,"reasons":["conf:0","agency","stale_co","velocity","win:late","comp:brand"],"computed_at":"2026-10-20T05:45:00Z"},"pay":null,"html_url":"https://alion.io/job/delan-associates-aws-lakehouse-data-engineer","json_url":"https://alion.io/job/delan-associates-aws-lakehouse-data-engineer.json","meta":{"generated_at":"2026-10-20T13:01:21Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":14430,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-21T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":690244},"rest":"https://alion.io/mcp/rest/get_company?id=690244"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fdelan-associates-aws-lakehouse-data-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fdelan-associates-aws-lakehouse-data-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fdelan-associates-aws-lakehouse-data-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/delan-associates-aws-lakehouse-data-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fdelan-associates-aws-lakehouse-data-engineer"}]}