Lead Data Engineer

Discover Your Career at Emory University

Emory University is a leading research university that fosters excellence and attracts world-class talent to innovate today and prepare leaders for the future. We welcome candidates who can contribute to the excellence of our academic community.

Description

The Lead Data Engineer provides technical leadership and hands-on engineering expertise for the development and operation of Emory University’s Unified Data Platform (UDP), built on Microsoft Fabric. The role leads to the integration of data from HR, Finance, Student, Research, and other enterprise applications to support analytics, reporting, research, and institutional decision-making.

Working closely with architects, researchers, business stakeholders, and cross-functional teams, the lead translates requirements into reliable, scalable, and governed data solutions. This position guides engineering delivery, mentors team members, establishes development standards, and resolves complex technical issues while contributing directly to pipeline development and production support. Microsoft Fabric experience is preferred. Candidates with strong experience in Azure Synapse or Databricks with PySpark, and the ability to apply that experience to Microsoft Fabric, will also be considered. The role requires sound judgment in data quality, security, governance, and the handling of sensitive university and research data. KEY RESPONSIBILITIES:

• Lead the technical delivery of source system onboarding and data integration initiatives, from source analysis and design through development, validation, deployment, and production support.
• Partner with architects to implement platform designs, reusable ingestion frameworks, and engineering standards aligned with the UDP architecture and institutional priorities.
• Design, develop, and optimize ETL and ELT pipelines using SQL, Python, PySpark, and Microsoft Fabric or comparable cloud technologies.
• Analyze source schemas, business rules, and dependencies across enterprise applications; define source-to-target mappings, incremental load strategies, and reconciliation requirements.
• Guide implementation of bronze, silver, and gold data layers, including data transformation, dimensional modeling, and curated datasets for reporting and analytics.
• Establish and review standards for coding, testing, documentation, version control, and deployment; conduct code reviews and mentor data engineers.
• Coordinate engineering tasks and technical dependencies with Business Analysts, Project Managers, Data Analysts, Architects, and source system owners; communicate delivery progress, risks, and resource needs.
• Implement data quality checks, reconciliation controls, monitoring, alerting, error handling, and recovery procedures to support reliable production processing.
• Troubleshoot complex pipeline and data issues; tune SQL, Spark workloads, and orchestration to improve performance, scalability, and efficient use of platform capacity.
• Apply institutional data governance and security standards, including access controls, metadata, lineage, and appropriate handling of sensitive data; support HIPAA requirements where applicable.
• Work with researchers and business stakeholders to translate requirements into technical specifications and explain design choices and data limitations clearly.
• Create and maintain technical documentation, operational runbooks, source mappings, and support procedures; coordinate production deployments and knowledge transfer.
• Support existing UDP and Enterprise Business Intelligence (EBI) integrations and services, balancing operational needs with project delivery priorities.
• Evaluate emerging technologies and develop focused proofs of concept that address identified business or engineering needs.

KNOWLEDGE, SKILLS, AND ABILITIES:

• Strong experience designing and maintaining data pipelines and ETL/ELT processes across multiple source systems.
• Proficiency in Python, SQL, and Apache Spark (PySpark)
• Experience with modern data platforms such as Azure Fabric (preferred), Azure Synapse Serverless, or Databricks
• Ability to work with large, complex, and distributed datasets
• Solid understanding of data modeling concepts and best practices
• Familiarity with data governance, metadata management, and data quality frameworks
• Ability to lead technical design discussions, review code, mentor engineers, and guide delivery across multiple workstreams
• Experience with CI/CD practices and proficiency with version control systems like Git
• Familiarity with data pipeline orchestration and monitoring tools Like Airflow, Azure Data Factory or similar frameworks
• Understanding of biomedical informatics methodologies is a plus
• Strong analytical, problem-solving, and critical-thinking abilities
• Strong analytical, problem-solving, communication, and collaboration skills, with the ability to explain technical concepts to diverse audiences.
• Ability to prioritize engineering work, identify delivery risks, and adapt to changing business and platform requirements.

ADDITIONAL JOB DETAILS:

• Works as a positive team member of a project that may consist of Business Analysts, Project Managers, Information Architects, Data Analysts, and/or Database Administrators to deliver quality applications and components within scope, on time, within budget, and ensures compliance.
• Manages workload effectively and report status of tasks in a timely manner.
• Is able to guide stakeholders on solutions and support structure.
• Ensures proper governance of prioritizing work from partners.
• Forms tactical strategies to support multiple clients.
• Leads and proactively identifies emerging technologies; develops proof-of-concepts, and promotes the usage of these emerging technologies.
• Leads small to medium size technical teams, and mentors other analysts on regulation adherence.
• Follows standard operational procedures and HIPAA regulations.
• Develops strategies for managing complex data sets through maintaining data standards and metadata.
• Applies biomedical informatics technical standards, methodologies, and principles to research-specific program needs, objectives, and outcomes.
• Develops reporting infrastructure to meet multiple client needs.
• Generates standard templates for architecture documentation related to current and proposed informatic solutions.
• Performs other related duties as required.

MINIMUM QUALIFICATIONS:

• A bachelor's degree in a related field and seven years of related experience, OR an equivalent combination of education, training, and experience.

PREFERRED QUALIFICATIONS:

• Experience with Microsoft Fabric Lakehouse, Warehouse, notebooks, and data pipelines.
• Experience in integrating higher education and Healthcare Enterprise HR, Finance, Student, and Research applications.

NOTE: Tasks related to this position can be performed remotely with only occasional visits to an Emory University location. Eastern (EST) time zone business hours may apply. Emory reserves the right to change this status with notice to employee. Emory does not approve as a primary work location in the following states; NJ, AK, and HI, any U.S. Territories or outside of the United States.

Additional Details

Emory is an equal opportunity employer, and qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, disability, protected veteran status or other characteristics protected by state or federal law. Emory University does not discriminate in admissions, educational programs, or employment, including recruitment, hiring, promotions, transfers, discipline, terminations, wage and salary administration, benefits, and training. Students, faculty, and staff are assured of participation in university programs and in the use of facilities without such discrimination. Emory University complies with Section 503 of the Rehabilitation Act of 1973, the Vietnam Era Veteran's Readjustment Assistance Act, and applicable executive orders, federal and state regulations regarding nondiscrimination, equal opportunity, and affirmative action (for protected veterans and individuals with disabilities). Inquiries regarding this policy should be directed to the Emory University Department of Equity and Civil Rights Compliance, 201 Dowman Drive, Administration Building, Atlanta, GA 30322. Telephone: 404-727-9867 (V) | 404-712-2049 (TDD). Emory University is committed to ensuring equal access and providing reasonable accommodations to qualified individuals with disabilities upon request. To request this document in an alternate format or to seek a reasonable accommodation, please contact the Department of Accessibility Services at [email protected] or call 404-727-9877 (Voice) | 404-712-2049 (TDD). We kindly ask that requests be made at least seven business days in advance to allow adequate time for coordination.