Solution Architect

Job Description :

About Quantzig :

Quantzig is a global analytics and advisory firm with offices in the US, UK, Canada, China, and India. We have assisted our clients across the globe with end-to-end advanced analytics, visual storyboarding, Machine Learning and data engineering solutions implementation for prudent decision making. We are a rapidly growing organization that is built and operated by high-performance champions.

If you have what it takes to be the champion with business and functional skills to take ownership of an entire project end-to-end, help build a team with great work ethic and a drive to learn, you are the one we are looking for. The clients love us for our solutioning capability, our enthusiasm and we expect you to be a part of our growth story.

Solution Architect Azure Databricks Data & ML Platform Role Overview :

We are seeking an experienced Solution Architect with strong expertise in Azure Databricks, enterprise data platforms, edge-to-cloud integration, and MLOps to lead architecture design and implementation across large-scale industrial and analytics ecosystems. The role involves designing scalable data platforms, enabling production-grade machine learning environments, and integrating OT systems with enterprise cloud data infrastructure. This role is ideal for architects working in client-facing consulting environments, delivering secure, governed, and scalable solutions across multi-layer data platforms and edge-connected intelligent systems.

Key Responsibilities :

1. Architect Integrated Data & ML Solutions :

- Lead architecture design across edge, cloud, and ML platform layers.

- Define scalable architectures supporting telemetry ingestion, operational analytics, and ML-driven insights.

- Design zone-based lakehouse architectures aligned with enterprise governance standards.

2. Build and Govern Enterprise Data Platform :

- Architect ingestion and transformation pipelines across Raw - Bronze - Silver - Gold layers.

- Implement enterprise-grade Unity Catalog-based governance frameworks.

- Define consumption-layer strategies supporting BI, ML, and operational workloads.

3. Enable Scalable Machine Learning Platform :

- Architect infrastructure supporting feature engineering, training, validation, versioning, and deployment.

- Enable Feature Store and Model Store implementations using Azure Databricks ML workspace.

- Support cloud and edge inference deployment strategies.

- Define lifecycle cadence models : 1. Weekly refresh, 2. Monthly retraining, 3. Quarterly model revamp.

4. Edge Integration and Automation :

- Architect secure OT/IT integration pipelines.

- Design edge-to-cloud ingestion using Docker, Portainer, RabbitMQ, and OPC UA interfaces for PLC connectivity.

- Implement IDMZ-compliant deployment patterns.

5. Monitor and Optimize Data & ML Pipelines :

- Define monitoring frameworks for ETL and ML workloads.

- Enable near real-time monitoring (e.g., 1-minute intervals for model telemetry).

- Optimize performance, reliability, and cost efficiency across workloads.

6. Governance and Security Compliance :

- Define enterprise governance standards using Unity Catalog.

- Implement access policies, metadata tagging, and lineage tracking.

- Ensure compliance across multi-zone data lake environments.

7. Lead CI/CD Automation :

- Architect CI/CD pipelines using Azure DevOps, GitHub Actions, and GitHub self-hosted runners.

- Enable automation for ML workflows and platform deployments.

Technical Expertise :

Azure Cloud Stack & DevOps :

- Azure Databricks (including ML workspace, Feature Store, Model Store).

- Azure Data Factory (ADF) orchestration.

- Azure Data Lake Storage (ADLS) with Medallion architecture.

- Azure Event Hub (topics, consumer groups, event-driven integration).

- Azure Stream Analytics for real-time telemetry processing.

- Azure Key Vault, Azure App Service, Azure Container Registry (ACR).

- Azure IoT Hub for edge device connectivity.

- Azure DevOps, GitHub Actions for CI/CD automation, GitHub self-hosted runners.

Edge and On-Prem Integration :

- OT/IT integration architecture and data extraction from industrial/PLC environments.

- Edge VM deployment using Docker, Portainer, and RabbitMQ messaging services.

- OPC UA integration with PLC systems (e.g., compressors, filtration systems).

- IDMZ deployment strategies and secure edge-to-cloud service integration.

Machine Learning Platform & MLOps :

- End-to-end ML lifecycle : feature engineering, training, validation, versioning, deployment.

- Model export and versioning strategies; production deployment architectures.

- Monitoring deployed models at frequent intervals; training vs inference architecture design.

- Cloud vs edge inference strategies; GitHub monorepo-based ML workflow management.

- Hands-on expertise : Azure Databricks ML workspace, Feature Store, Model Store.

Data Architecture & Integration :

- Medallion architecture (Raw - Bronze - Silver - Gold) across enterprise lakehouse environments.

- Unity Catalog integration for governance and sharing.

- CDC ingestion tools (e.g., Aecorsoft for SAP ingestion); streaming ingestion frameworks.

- API-based ingestion architectures; template-driven ingestion with configuration-based mapping.

Governance and Data Modeling :

- Define enterprise data governance frameworks; access control strategies across DLZ zones.

- Design scalable dimensional and feature-driven data models.

- Enable metadata tagging and lineage tracking.

- Support analytics and ML feature readiness through optimized modeling.

Qualifications :

Required Qualifications :

- 10+ years in data platform architecture or enterprise analytics.

- 5+ years on Azure-based data platforms.

- Strong expertise in Azure Databricks Lakehouse architecture.

- Experience designing ML platform infrastructure.

- Proven edge/industrial data system integration with cloud.

- Experience implementing Unity Catalog governance frameworks.

- Strong stakeholder management in enterprise delivery.

Preferred Qualifications :

- Industrial analytics or manufacturing environments.

- Exposure to IoT telemetry architectures.

- SAP CDC ingestion framework experience.

- Near real-time streaming analytics platform design.

- Consulting or client-facing enterprise program experience.

Soft Skills :

- Strong architectural leadership.

- Excellent client communication skills.

- Translate business problems into platform architecture.

- Cross-team collaboration (Data Eng, ML Eng, DevOps).

- Multi-vendor enterprise delivery experience.

Engagement Model :

- Client-facing architecture role.

- Works with enterprise stakeholders, ML teams, platform engineers, OT system teams.

- Leads technical governance & architecture decision-making across platform layers.

Primary Technology Stack :

Azure Databricks, ADLS, ADF, Unity Catalog, Event Hub, Stream Analytics, Azure IoT Hub, Docker, RabbitMQ, OPC UA, GitHub Actions, Azure DevOps, Feature Store, Model Store, Edge ML Deployment, CDC (SAP), Medallion Architecture.

Skills

Solution Architect, Azure Databricks, Solution Integration, Azure, DevOps, ADLS, MLOps, Google Cloud Platform