{"id":4587825,"url":"https://alion.io/job/atlas-systems-p1-gp-senior-data-engineer-data-architect","title":"P1 - GP - Senior Data Engineer / Data Architect","company":{"id":2282715,"name":"Atlas Systems •","domain":"atlassystems.com","url":"https://alion.io/company/atlassystems","size_band":"201-500","is_staffing_agency":false,"employer_type":"services","is_intermediary":false,"listed_via":null,"ats_vendor":"BambooHR","truth_index":{"grade":"B","score":83,"open_postings":3,"ghost_share":0,"stale_share":0.667,"repost_share":0,"time_to_fill_p50_days":42,"computed_at":"2026-11-18T05:48:20Z"}},"role":"Data Science","role_family":"Data Science","seniority":"staff","employment_type":null,"work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Bengaluru, India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Databricks","optional":false},{"name":"Delta Lake","optional":false},{"name":"Dimensional Modeling","optional":false},{"name":"MS SQL","optional":false},{"name":"PostgreSQL","optional":false},{"name":"Power BI","optional":false},{"name":"pySpark","optional":false},{"name":"Python","optional":false},{"name":"Spark","optional":false},{"name":"SQL","optional":false},{"name":"Azure","optional":true},{"name":"Azure SQL Database","optional":true},{"name":"CI/CD","optional":true}],"status":"live","first_seen_at":"2026-11-19T07:33:37Z","employer_posted_date":"2026-11-19","last_verified_at":"2026-11-19T07:33:37Z","board_verified":true,"closed_at":null,"days_open":0,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":0},"description":"Job Description: Senior Data Engineer / Data Architect\nAbout the role\nWe are hiring a Senior Data Engineer to build and run the pipelines that bring portfolio company data into one business-aligned model on Databricks.\nThe data comes from a third-party portfolio monitoring platform, with history carried over once from a legacy portfolio system. You will take it through Bronze, Silver and Gold layers and serve it to Power BI reporting and internal applications. You will own pipelines end to end: ingestion, transformation, data quality, performance and day-to-day operation.\nWhat you'll do\nIngestion:build incremental loads from the source platform into Bronze using change timestamps and watermarks, with safe handling of late-arriving rows and full retention of raw history.\nSilver:model source data close to its native structure, with deterministic deduplication, type-aware unit conversion, sign rules and quarantine of suspect rows rather than silent fixes.\nGold:implement a source-agnostic business model of dimensions, fact tables with different history rules and reference data, built as performant Delta tables for Power BI and application use.\nMigration:support a single cutover from the legacy system, and load and freeze the history of companies that will not move.\nData modeling:work with the data model owner on the item catalog, dimensions (period, currency, fund and entity), cash transactions and derived measures such as multiples and FX conversion.\nData quality and reconciliation:build checks and quality flags, reconcile new figures against the legacy system before the cutover and run regression checks when the model changes.\nServing:publish Gold to downstream consumers, including Lakebase (Postgres) synced tables for applications and direct Delta access for BI.\nOperations:orchestrate and monitor the pipelines, manage access in Unity Catalog, and keep documentation, lineage and runbooks current.\nCollaboration:work closely with the vendor's data team, reporting and application teams, and finance and portfolio operations users on definitions and issues.\nWhat you'll bring\n5+ years building production data pipelines, including 2+ years on Databricks with Delta Lake.\nStrong SQL (Spark SQL and T-SQL) and Python or PySpark.\nHands-on experience with a medallion (Bronze, Silver, Gold) architecture and incremental processing: watermarks, MERGE, change data feed.\nDimensional modeling: facts at a clear grain, dimensions, surrogate keys, slowly changing dimensions.\nA track record of data quality work: deduplication, reconciliation, and tracing a wrong number back to its source.\nExperience with Unity Catalog or a comparable governance layer: permissions, lineage, documentation.\nPerformance tuning for BI workloads, such as file layout, clustering and query plans.\nClear written communication, including explaining data issues and trade-offs to non-technical users.\nNice to have\nPrivate equity, asset management or fund administration data: portfolio monitoring, valuations, cash flows, fund and co-invest structures.\nFinancial reporting concepts: LTM and YTD periods, fiscal year ends, budgets and forecasts, restatements, currency conversion.\nDatabricks Lakebase or PostgreSQL, including serving data to applications.\nLakeflow (Delta Live Tables) or Databricks Workflows orchestration.\nAzure SQL and migrating SQL Server logic to Databricks.\nPower BI data modeling, or working closely with BI developers.\nInfrastructure as code and CI/CD for data pipelines, for example Databricks Asset Bundles.\nOur environment\nArea\nTools\nLakehouse\nDatabricks, Delta Lake, Unity Catalog\nLanguages\nSpark SQL, Python, PySpark, T-SQL\nApplication serving\nDatabricks Lakebase (Postgres) synced tables\nLegacy and transition\nAzure SQL Database\nReporting\nPower BI\nCloud\nMicrosoft Azure","description_format":"text","description_chars":3796,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Managed IT Services (MSP)","EHR & Clinical Software"],"lifecycle":[{"event":"open","at":"2026-11-19T07:33:37Z"}],"visa":[],"liveness":{"score":86,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.86,"p_room":1,"age_days":0,"expected_fill_days":42,"reasons":["conf:1","win:early"],"computed_at":"2026-11-19T09:18:41Z"},"pay":null,"html_url":"https://alion.io/job/atlas-systems-p1-gp-senior-data-engineer-data-architect","json_url":"https://alion.io/job/atlas-systems-p1-gp-senior-data-engineer-data-architect.json","meta":{"generated_at":"2026-11-19T09:18:41Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":15781,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-20T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":2282715},"rest":"https://alion.io/mcp/rest/get_company?id=2282715"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fatlas-systems-p1-gp-senior-data-engineer-data-architect"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"free","price":{"credits":0,"usd":0},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fatlas-systems-p1-gp-senior-data-engineer-data-architect"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fatlas-systems-p1-gp-senior-data-engineer-data-architect"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/atlas-systems-p1-gp-senior-data-engineer-data-architect\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fatlas-systems-p1-gp-senior-data-engineer-data-architect"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":4587825}},{"tool":"search_jobs","arguments":{"company":"atlassystems.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":4587825},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}