{"id":3554281,"url":"https://alion.io/job/persistent-systems-data-engineer-7","title":"Data Engineer","company":{"id":2073156,"name":"Persistent Systems","domain":"persistentsystems.com","url":"https://alion.io/company/persistentsystems","size_band":"201-500","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"iCIMS","truth_index":null},"role":"Data Science","role_family":"Data Science","seniority":"middle","employment_type":null,"work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["New York, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":{"min":133000,"max":166000,"currency":"USD","period":"year","gross":null,"usd_annual":166000},"salary_estimate":null,"experience_years_min":3,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Airflow","optional":false},{"name":"Apache Kafka","optional":false},{"name":"Azure","optional":false},{"name":"Azure Cosmos DB","optional":false},{"name":"Azure Data Factory","optional":false},{"name":"Azure DevOps","optional":false},{"name":"Azure SQL Database","optional":false},{"name":"Bicep","optional":false},{"name":"Bitbucket","optional":false},{"name":"CI/CD","optional":false},{"name":"Claude Code","optional":false},{"name":"Copilot","optional":false},{"name":"Databricks","optional":false},{"name":"dbt","optional":false},{"name":"Delta Lake","optional":false},{"name":"Dimensional Modeling","optional":false},{"name":"ETL/ELT","optional":false},{"name":"Git","optional":false},{"name":"GitLab","optional":false},{"name":"Microsoft Entra ID","optional":false},{"name":"Microsoft Fabric","optional":false},{"name":"MS SQL","optional":false},{"name":"OpenAI Codex","optional":false},{"name":"Power Apps","optional":false},{"name":"Power Automate","optional":false},{"name":"Power BI","optional":false},{"name":"PowerShell","optional":false},{"name":"pySpark","optional":false},{"name":"Python","optional":false},{"name":"RAG","optional":false},{"name":"Rest API","optional":false},{"name":"Snowflake","optional":false},{"name":"Spark","optional":false},{"name":"SQL","optional":false},{"name":"Terraform","optional":false}],"status":"live","first_seen_at":"2026-10-31T00:04:33Z","employer_posted_date":"2026-10-31","last_verified_at":"2026-10-31T22:49:23Z","board_verified":true,"closed_at":null,"days_open":0,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":0},"description":"Job Description & Responsibilities\nPersistent Systems is seeking a Data Engineer to join our team in our New York City headquarters. The Data Engineer will design, build, and operate the pipelines, lakehouses, and data models that deliver trusted, well-governed data for analytics, reporting, and AI solutions, with a primary focus on Microsoft Fabric and the Azure data platform.\nThis role sits between source systems and the people and applications that depend on them: ingesting operational, file, API, and streaming data; transforming it through medallion layers in OneLake; and publishing curated, secure datasets for Power BI, data science, and retrieval-augmented generation (RAG) workloads, while integrating tools from other providers where solutions require them.\nThe ideal candidate is a pragmatic builder who treats data pipelines as production software. They write clean, testable SQL and PySpark, care about data quality and lineage as much as throughput, and can explain tradeoffs among cost, latency, data freshness, and complexity to both technical and nontechnical stakeholders.\nPosition Responsibilities\nOwn data engineering work from source-system discovery and requirements through design, build, testing, deployment, monitoring, and ongoing support\nPartner with analysts, BI developers, AI engineers, architects, security, and business stakeholders to define data requirements, freshness targets, and acceptance criteria\nBuild reusable ingestion patterns, transformation frameworks, and data contracts so that pipelines, definitions, and lineage are consistent and maintainable across teams\nEvaluate Microsoft Fabric, Azure, and complementary tools against requirements for scale, performance, security, and total cost of ownership, and track platform changes and new features\nDocument designs, data flows, and operating procedures clearly, review peers’ work, and contribute to a culture of engineering discipline\nDesign and implement medallion (bronze, silver, gold) architectures in Microsoft Fabric Lakehouse and Warehouse on OneLake using Delta Lake tables, OneLake shortcuts, and appropriate partitioning and table optimization strategies\nBuild batch and incremental ingestion with Fabric Data Factory pipelines, Copy job, Dataflow Gen2, and Azure Data Factory, and replicate operational databases into OneLake using mirroring and change data capture (CDC)\nDevelop transformation logic in Spark notebooks (PySpark and Spark SQL), T-SQL, and KQL; handle duplicate, missing, and late-arriving data; and implement slowly changing dimensions and incremental merge patterns \nBuild and maintain pipelines in Azure Databricks using Lakeflow Jobs, Lakeflow Spark Declarative Pipelines, Auto Loader, and Unity Catalog, and integrate them with OneLake and Azure Data Lake Storage Gen2\nIngest and process streaming data with Fabric Real-Time Intelligence (Eventstreams, Eventhouse, and Activator), Azure Event Hubs, and Spark Structured Streaming\nDesign star-schema fact and dimension models and collaborate with BI developers on Power BI semantic models, including Direct Lake, to deliver performant and consistent reporting\nImplement validation rules, schema enforcement, reconciliations, and automated data quality tests; monitor pipelines with the Fabric Monitoring hub, Azure Monitor, and Log Analytics; and resolve failures through root-cause analysis\nApply OneLake security, row- and column-level security, dynamic data masking, sensitivity labels, and Microsoft Purview cataloging and lineage; secure access with Microsoft Entra ID, managed identities, and Azure Key Vault\nManage notebooks, pipelines, and SQL database projects in Git-based version control (for example, Azure Repos); promote changes through Fabric deployment pipelines and Azure Pipelines with automated tests and approvals; and provision Azure resources with Bicep or Terraform\nPrepare curated datasets and document-processing pipelines that feed Azure AI Search, Microsoft Foundry, and Copilot solutions, including chunking, metadata enrichment, and freshness monitoring for RAG\nTune Spark jobs, Delta tables (OPTIMIZE, V-Order, and liquid clustering or Z-ordering), warehouse queries, and pipeline schedules, and monitor Fabric capacity consumption and Azure spend\nTools and Systems You’ll Use\nMicrosoft Fabric\nOneLake, Lakehouse, Warehouse, Data Factory (pipelines, Copy job, Dataflow Gen2), mirroring, shortcuts, Spark notebooks, Real-Time Intelligence (Eventstream, Eventhouse, Activator), Direct Lake\nAzure data services\nAzure Data Factory, Azure Databricks (Unity Catalog, Lakeflow), Azure Data Lake Storage Gen2, Azure SQL Database, SQL Server, Azure Event Hubs, Azure Functions, Azure Cosmos DB\nLanguages and processing\nSQL and T-SQL, Python, PySpark, Spark SQL, KQL, Delta Lake, Parquet, JSON, REST APIs, PowerShell\nDataOps and automation\nGit-based version control (for example, Azure Repos, GitLab, or Bitbucket), Azure Pipelines, Azure Boards, Fabric deployment pipelines, SQL database projects, Bicep, Terraform, Azure CLI, automated data testing\nGovernance and operations\nMicrosoft Purview, Microsoft Entra ID, managed identities, Azure Key Vault, OneLake security, Fabric Monitoring hub, Fabric Capacity Metrics app, Azure Monitor, Log Analytics\nValuable adjacent skills\nPower BI semantic models, Power Platform (Power Apps, Power Automate, Dataverse), Azure AI Search, Microsoft Foundry, Copilot in Fabric, dbt, Apache Airflow, Apache Kafka, Snowflake, Claude Code or Codex for development\nJob Qualifications & Experience\nMinimum Qualifications\n3-5 years of professional experience in data engineering, ETL/ELT development, or data warehousing, including building and supporting production data pipelines\nBachelor’s degree in computer science, information systems, engineering, mathematics, or a related field, or an equivalent combination of education and relevant professional experience \nAdvanced SQL proficiency, including complex joins, common table expressions, window functions, MERGE and upsert logic, query tuning, and validation against source systems\nStrong Python skills for data engineering, including PySpark or Spark SQL for distributed data processing\nHands-on experience building pipelines with Fabric Data Factory, Azure Data Factory, or Azure Databricks, including parameterization, scheduling, error handling, and incremental loads\nExperience designing lakehouse or data warehouse solutions using Delta Lake or comparable table formats, medallion architecture, and dimensional modeling\nWorking knowledge of Azure storage, identity, networking, and RBAC, including secure data access with Microsoft Entra ID\nExperience with Git-based version control and CI/CD practices for data workloads, including branching, code review, and environment promotion\nDemonstrated ability to troubleshoot pipeline failures and data quality issues, communicate clearly, and document data flows and lineage\nPreferred Qualifications\nProduction experience with Microsoft Fabric, including OneLake, Lakehouse, Warehouse, mirroring, shortcuts, and Fabric deployment pipelines\nExperience with real-time data using Fabric Real-Time Intelligence, Eventstreams, Eventhouse and KQL, Azure Event Hubs, or Spark Structured Streaming\nExperience with Azure Databricks, Unity Catalog, and Lakeflow, including integration with Microsoft Fabric\nExperience building release pipelines in Azure DevOps (Azure Repos and Azure Pipelines) and provisioning data infrastructure with Bicep or Terraform\nExperience with Microsoft Purview data governance, including data cataloging, lineage, sensitivity labels, and data quality\nExperience preparing data for AI workloads, including Azure AI Search indexing, Microsoft Foundry solutions, or RAG pipelines\nExperience with complementary non-Microsoft tools such as dbt, Apache Airflow, Apache Kafka, or Snowflake\nExperience with Microsoft Power Platform, including Power Apps, Power Automate, and Dataverse, and integrating low-code solutions with governed enterprise data\nExperience using AI coding agents such as Claude Code or Codex to accelerate pipeline development safely\nRelevant certifications or credentials, such as: \nMicrosoft Certified: Fabric Data Engineer Associate (DP-700) \nMicrosoft Certified: Azure Databricks Data Engineer Associate (DP-750) \nMicrosoft Certified: Fabric Analytics Engineer Associate (DP-600) \nMicrosoft Certified: Power Platform Fundamentals (PL-900) \nDatabricks Certified Data Engineer Associate or Professional \nExperience operating in a regulated, security-sensitive, government, or defense-adjacent environment\nThe total compensation for this role can vary from $133,000 to $166,000 a year based on individual qualifications. Eligibility for yearly discretionary bonus. Persistent Systems, LLC offers a comprehensive benefits package including medical, dental, vision, life, and disability insurance; paid time off (sick time and vacation time); flexible spending accounts; 401(k) plan with company match; fitness membership reimbursement; tuition assistance; mental health benefits; and pet discounts.\nCompany Overview\nHeadquartered in New York City since 2007, Persistent Systems, LLC is a global communications technology company which develops, manufactures, and integrates a patented and secure Mobile Ad Hoc Networking (MANET) system: Wave Relay®. The company’s industry leading R&D team has designed wireless networking protocols to support their cutting-edge Wave Relay® system and technology. Wave Relay® is capable of running data, video, voice and other applications under the most difficult and unpredictable conditions. Their suite of products is field proven and utilized in Commercial, Military, Government, Industrial, Agriculture, Robotics, and Unmanned System markets.","description_format":"text","description_chars":9756,"description_truncated":false,"requirements":{"experience_years_min":3,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"bachelor","optional":false},"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Artificial Intelligence","Military","Drones & UAVs"],"lifecycle":[{"event":"open","at":"2026-10-31T00:04:33Z"}],"visa":[],"liveness":{"score":90,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.903,"p_room":1,"age_days":0,"expected_fill_days":14,"reasons":["conf:1","velocity","win:early"],"computed_at":"2026-10-31T05:58:55Z"},"pay":{"stated_usd_annual":166000,"is_top_pay":true},"html_url":"https://alion.io/job/persistent-systems-data-engineer-7","json_url":"https://alion.io/job/persistent-systems-data-engineer-7.json","meta":{"generated_at":"2026-10-31T23:40:18Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":37077,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-01T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":2073156},"rest":"https://alion.io/mcp/rest/get_company?id=2073156"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fpersistent-systems-data-engineer-7"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fpersistent-systems-data-engineer-7"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fpersistent-systems-data-engineer-7"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/persistent-systems-data-engineer-7\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fpersistent-systems-data-engineer-7"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":3554281}},{"tool":"search_jobs","arguments":{"company":"persistentsystems.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":3554281},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}