Senior Databricks Engineer

Sharp Decisions is seeking a Senior Databricks Engineer to support a cybersecurity data initiative and build a security and risk data platform for Information Security. The role focuses on designing, developing, testing, and deploying cloud data solutions, developing cybersecurity data feeds, establishing data governance, and collaborating with technical and business stakeholders.

Responsibilities

• Design, develop, test, and support cloud data solutions focusing on data ingestion, data quality, tuning, and performance of upstream cybersecurity data sources using Databricks
• Provide indirect leadership and mentorship to junior team members; lead data feed development efforts and design initiatives
• Participate in development meetings to align priorities and objectives, assign tasks, and share experiences across applications under development
• Consult with technology and development teams to coordinate integration of applications within the broader software ecosystem
• Capture and document metadata for Key Data Elements (KDEs) to ensure accuracy and completeness for Data Quality (DQ) rules and daily dataset processing
• Work with the data architecture team to align KDEs to logical data models, develop physical data structures, and document physical data names, definitions, and types
• Partner with data owners and stakeholders to create technical requirements and DQ rules around data elements needed in the data warehouse
• Partner with Business Management and Data Owners to identify critical metrics and data fields required for metric dashboards
• Establish views to encapsulate data for downstream consumption and ensure alignment with DQ rules established on metadata
• Utilize IBM TWS production job scheduling and adhere to standards around daily scheduling and batch monitoring of production jobs
• Identify and resolve data quality issues including inaccuracies and incomplete information; implement improved procedures and processes

Skills

• 8+ Years Experience
• Azure Databricks, Python, PySpark, SQL
• Azure Databricks & Unity Catalog
• Azure Data Factory
• Azure Data Lake Storage
• Azure Functions
• Azure Event Grid
• Azure Log Analytics & Monitor
• Python / Java
• SQL Server, Oracle / PL-SQL, T-SQL
• ETL / ELT
• REST API & Microservices
• CI/CD — Git, Jenkins, Azure DevOps
• Collibra DQIM/DQE, IBM Infosphere DA
• * Strong knowledge of enterprise Information Security data domains: Phishing, Identity Management, Privileged Access, Cloud Security, Incident Response, Vulnerability Management, and Threat Detection
• * Data knowledge of PaaS/SaaS products including ServiceNow, CrowdStrike, MS Purview, Proofpoint, WIZ.IO, JIRA, SharePoint, and Azure Active Directory
• * Understanding of information security frameworks (NIST, CIS, CRI Profile) and regulatory compliance (NYSDFS, GDPR, CCPA)
• * Financial services or similarly regulated environment experience strongly preferred
• * Knowledge of Microsoft Sentinel for SIEM
• * Knowledge of IBM Tivoli Workload Scheduler
• * Exposure to Power BI for data visualization and reporting

Benefits

• Hybrid work arrangement: 3 days onsite
• Contract to Hire
• W2 employment only

Company Overview

Company H1B Sponsorship