Data Engineer / Data Architect

Job Summary

Design and optimize cloud based data platforms that power analytics and decision making for a global property and casualty insurance business using AWS PySpark Python and Databricks within a hybrid work model ensuring resilient data pipelines reliable governance and scalable architectures that enable accurate risk assessment and faster claims processing.

Responsibilities

• Design and implement scalable data architectures on AWS that support end to end data flows for property and casualty insurance underwriting pricing claims and policy administration use cases
• Build robust PySpark and Python based data pipelines in Databricks that ingest transform and curate structured and semi structured insurance data from multiple internal and external sources
• Optimize Databricks workflows and Spark jobs to achieve predictable performance cost efficiency and reliability for large volumes of policy premium exposure and claims data
• Develop reusable data models and canonical data sets that standardize property and casualty insurance entities such as policies coverages claims and reinsurance treaties across business lines
• Establish and maintain data quality rules data validation checks and reconciliation processes that improve accuracy timeliness and completeness of critical regulatory and financial reporting datasets
• Collaborate with actuaries underwriters claims analysts and product teams to translate analytical and reporting requirements into well defined data engineering solutions that create measurable business value
• Implement secure data access patterns on AWS using appropriate services and configurations that protect sensitive customer and financial information while enabling governed self service analytics
• Create and maintain technical documentation including data flow diagrams data dictionaries and architecture blueprints that provide clear traceability from source systems to curated consumption layers
• Automate testing monitoring and alerting for data pipelines using Python and cloud native tools to proactively identify data anomalies and pipeline failures before they affect downstream consumers
• Partner with enterprise and solution architects to align data engineering solutions with target cloud architecture standards helping to modernize legacy insurance platforms and decommission redundant assets
• Drive continuous improvement by profiling datasets identifying performance bottlenecks and implementing tuning strategies that shorten data delivery cycles for reporting dashboards and advanced analytics
• Support business continuity by implementing backup recovery and environment management strategies in AWS and Databricks that reduce operational risk and minimize downtime for critical data services
• Mentor peers and junior engineers by sharing best practices in PySpark coding standards data modeling for insurance and cloud based engineering patterns that raise the overall quality of team deliverables
  
  

Qualifications

• Possess eight to twelve years of experience in data engineering or data architecture roles with a consistent focus on delivering production grade solutions in complex enterprise environments
• Demonstrate strong hands on expertise with AWS data and compute services including data storage processing orchestration and security components used to run modern analytical workloads
• Exhibit advanced proficiency in PySpark and Python for building scalable transformation logic unit tests and utility frameworks that can be reused across multiple insurance data pipelines
• Bring proven experience working with Databricks to develop optimize and operate collaborative notebooks jobs and clusters tailored for large scale property and casualty data processing
• Show deep functional understanding of property and casualty insurance concepts including policies coverages rating factors claims lifecycle and regulatory reporting and apply this knowledge to data designs
• Apply solid knowledge of dimensional and relational data modeling practices to design curated data marts and semantic layers that serve analytics actuarial studies and operational reporting
• Utilize strong skills in SQL data profiling and data quality assessment to diagnose source system issues and drive corrective actions with business and technology stakeholders
• Demonstrate effective communication and collaboration abilities to work in a hybrid model with distributed teams conveying complex data topics in clear and concise business language
• Display familiarity with modern DevOps and DataOps practices such as version control continuous integration automated testing and environment promotion for data engineering solutions
• Show commitment to continual learning by staying updated on evolving AWS Databricks and open source data engineering capabilities and by actively incorporating relevant innovations into project delivery

Skills

Pyspark, Databricks, Python, Sql, AWS