Program Analyst Sr – Generative AI, Prompt Engineering & Validation

Company:

Job Area:

General Summary:

Assists with program development and implementation through managing processes, procedures, and tools that improve efficiencies. A Program Analyst coordinates across teams and monitors timelines, budgets, risks, and priorities to achieve program progress. Typically, a program needing a Program Analyst will be of significant size and will require expertise related to the development of project management mechanisms.

Minimum Qualifications:

OR

High School Diploma or equivalent and 2+ years of relevant work experience.

Job Title: Program Analyst Sr - Generative AI, Prompt Engineering & Validation

Location: Bangalore

Experience: 2-6 years

Employment Type: Full-time

About the Role

We are looking for a proactive and detail-oriented Program Analyst to support Generative AI initiatives focused on prompt engineering, feature validation, test execution, output quality evaluation, and program tracking. The role will support internal GenAI platforms and tools, including search, summarization, RAG-based workflows, analytical agents, and domain-specific AI assistants.

The ideal candidate should combine strong program coordination skills with a practical understanding of GenAI systems, structured prompt-based testing, multi-source data validation, and clear reporting for engineering and leadership stakeholders.

Key Responsibilities

1. GenAI Program Tracking & Coordination

• Track GenAI feature validation plans, deliverables, owners, dependencies, and closure status across teams.
• Maintain dashboards, trackers, action item logs, risk registers, and validation status reports.
• Coordinate with engineering, product, platform, and domain expert teams to drive timely completion of testing and feedback cycles.
• Consolidate weekly progress updates, blockers, risks, and recommendations for program reviews.
• Ensure validation documentation, prompt libraries, and issue logs are accurate, current, and accessible.

2. Prompt Engineering & Prompt Test Design

• Create, refine, and maintain prompts for GenAI features, search workflows, summarization use cases, and analytical agents.
• Develop zero-shot, few-shot, role-based, multi-turn, edge-case, and negative test prompts.
• Build reusable prompt templates and feature-wise prompt libraries.
• Compare prompt variants and document impact on accuracy, relevance, completeness, consistency, and latency.
• Translate domain expert feedback into improved prompts and updated test scenarios.

3. Feature Validation & Test Execution

• Design and execute structured test cases for GenAI features, including functional, sanity, regression, negative, and exploratory testing.
• Validate end-to-end behavior from user prompt to retrieved context, generated response, and final output format.
• Map feature requirements to prompts, expected outputs, acceptance criteria, and validation evidence.
• Identify defects, usability gaps, hallucinations, incomplete answers, inconsistent responses, and low-confidence outputs.
• Support release readiness assessment using clear status indicators such as Go / Watch / Risk.

4. Data, Search & RAG Validation

• Validate AI outputs against structured, semi-structured, and unstructured data sources such as SharePoint, Confluence, Jira, documents, JSON/XML, and internal knowledge bases.
• Check search relevance, context quality, source completeness, and data freshness before summarization.
• Support validation of RAG-based workflows, semantic search, keyword search, global filters, and domain filters.
• Identify data gaps, indexing issues, retrieval failures, and source-quality problems impacting AI response accuracy.
• Maintain golden datasets, expected-answer sets, and benchmark queries for repeatable validation.

5. Output Quality Evaluation & Defect Analysis

• Evaluate GenAI responses for factual correctness, relevance, completeness, clarity, format quality, and consistency across repeated runs.
• Log defects with clear reproduction steps, prompt used, expected output, actual output, severity, and owner.
• Perform first-level root cause analysis to distinguish prompt issues, data issues, retrieval issues, model limitations, and product defects.
• Work with SMEs to validate technical correctness and define acceptance thresholds for complex responses.
• Track quality improvements over time and report trends in accuracy, hallucination reduction, and feature readiness.

6. Reporting, Documentation & Stakeholder Communication

• Prepare weekly validation summaries, risk updates, action item trackers, and executive-ready status reports.
• Document prompt libraries, test suites, validation evidence, known issues, and improvement logs.
• Facilitate review meetings, capture minutes of meeting, document decisions, and follow up on action items.
• Communicate clearly with cross-functional stakeholders across engineering, platform, product, and program management teams.
• Support leadership reviews with concise summaries, dashboards, and recommendations.

7. Automation & Productivity Support

• Use Excel, Power BI, Jira, Confluence, SharePoint, or similar tools to maintain trackers, dashboards, and reporting views.
• Support basic scripting or automation for batch prompt execution, response capture, comparison, and reporting where applicable.
• Help improve validation efficiency through reusable templates, structured checklists, and repeatable test processes.
• Track productivity metrics such as test coverage, closure rate, defect aging, prompt reuse, and feature readiness.

Required Skills & Competencies

• 2-6 years of experience in program analysis, project coordination, validation, QA, data analysis, or technical operations.
• Hands-on exposure to Generative AI tools, LLM-based applications, prompt engineering, or AI-assisted workflows.
• Strong ability to create structured test cases, validation reports, action trackers, and executive summaries.
• Good understanding of prompt-response behavior, hallucinations, context limitations, and output evaluation methods.
• Working knowledge of search, summarization, RAG concepts, information retrieval, or knowledge-base validation.
• Strong Excel skills; Power BI, Jira, Confluence, SharePoint, or dashboarding experience is preferred.
• Basic Python, SQL, or scripting knowledge is preferred but not mandatory.
• Excellent written and verbal communication skills with strong attention to detail.

Preferred Qualifications

Education & Background

• Bachelor’s degree in Engineering, Computer Science, Electronics, Data Science, Information Systems, or a related discipline.
• Prior experience in semiconductor, SoC, embedded systems, platform engineering, enterprise AI, or technical program management is preferred.
• Experience supporting validation of internal tools, AI platforms, search systems, summarization tools, or workflow automation platforms is a plus.
• Exposure to responsible AI, data privacy, model evaluation, or enterprise GenAI governance is desirable.

Success Metrics

• Completeness and accuracy of GenAI feature validation coverage.
• Quality, reusability, and effectiveness of prompt test suites.
• Timely closure of validation defects, risks, action items, and stakeholder feedback.
• Improvement in output accuracy, relevance, consistency, and hallucination reduction.
• Clarity and usefulness of weekly reports, dashboards, executive summaries, and readiness recommendations.

Applicants: Qualcomm is an equal opportunity employer. If you are an individual with a disability and need an accommodation during the application/hiring process, rest assured that Qualcomm is committed to providing an accessible process. You may e-mail [email protected] or call Qualcomm's toll-free number found here. Upon request, Qualcomm will provide reasonable accommodations to support individuals with disabilities to be able participate in the hiring process. Qualcomm is also committed to making our workplace accessible for individuals with disabilities. (Keep in mind that this email address is used to provide reasonable accommodations for individuals with disabilities. We will not respond here to requests for updates on applications or resume inquiries).

Qualcomm expects its employees to abide by all applicable policies and procedures, including but not limited to security and other requirements regarding protection of Company confidential information and other confidential and/or proprietary information, to the extent those requirements are permissible under applicable law.

To all Staffing and Recruiting Agencies:Our Careers Site is only for individuals seeking a job at Qualcomm. Staffing and recruiting agencies and individuals being represented by an agency are not authorized to use this site or to submit profiles, applications or resumes, and any such submissions will be considered unsolicited. Qualcomm does not accept unsolicited resumes or applications from agencies. Please do not forward resumes to our jobs alias, Qualcomm employees or any other company location. Qualcomm is not responsible for any fees related to unsolicited resumes/applications.

If you would like more information about this role, please contact Qualcomm Careers.