Product Analyst, AI Solutions

Product Analyst, AI Solutions

The Product Analyst, AI Solutions is responsible for shaping, optimizing, and maintaining advanced AI-driven features and user experiences within our product suite. This role sits at the intersection of business analysis, AI tuning, and user experience—combining expertise in software requirements gathering, prompt engineering, and AI model evaluation. The Analyst is heavily focused on refining how AI systems behave in real-world scenarios, ensuring outputs are accurate, trustworthy, and closely aligned with client needs.

You partner with AI Engineers, Product Managers, UX/UI Designers, QA, and Subject Matter Experts (SMEs) to extract user requirements, translate those into actionable User Stories and evaluation metrics, and continuously tune AI systems for best-in-class performance and usability.

This role is pivotal in ensuring our AI solutions provide maximum business value, aligning advanced technology with end-user needs while driving continuous improvement and reliability. The Analyst produces high-quality deliverables using systematic processes, with impact across multiple projects, deadlines, and teams.

Principal Duties

AI Tuning & Output Quality

• Design, evaluate, and refine prompts, templates, and conversation flows for reliability, cost effectiveness, and user alignment.
• Maintain prompt libraries, track changes, and manage versioning.
• Systematically test AI model outputs for accuracy, consistency, compliance, and tone, applying rerankers, guardrails, and validation layers to ensure business rules are met.
• Collaborate with engineering and QA to promptly integrate feedback and resolve model drift or new scenarios.

Requirements Gathering & Documentation

• Proactively gather requirements of moderate scope and complexity from diverse sources (clients, market analysis, SMEs) to drive product development and AI enhancements.
• Document software and system requirements, including user stories, program functions, test cases, and steps required for new features or modifications.
• Ensure all documentation supports future maintainability and growth of AI-enabled features.

Evaluation, Metrics, & Continuous Improvement

• Establish systematic evaluation frameworks for AI model outputs, leveraging quantitative and qualitative benchmarks.
• Run experiments, A/B tests, and structured evaluations to compare tuning strategies and track performance metrics (accuracy, coverage, hallucination rates, latency).
• Translate business and user feedback into actionable improvements, owning continuous iteration for released solutions.

Collaboration & UX Alignment

• Act as a bridge between technical and non-technical stakeholders, ensuring clear, effective communication and alignment.
• Work closely with product, development, and UX/UI teams to ensure AI-driven features fit seamlessly into business workflows, are intuitive and user-friendly, and meet client needs.
• Help shape the overall user experience for AI interactions, promoting explainability and transparency.

Education, Experience and Special Skills

• Bachelor’s degree in Computer Science, Management Information Science, Business, or related field OR equivalent work experience.
• 1+ years’ experience in software, AI/ML, or related product analysis roles.
• Experience with AI/ML applications (LLMs, NLP, machine learning pipelines).
• Strong understanding of prompt engineering and generative AI interaction design.
• Basic knowledge of data modeling, relational database concepts, and SQL.
• Familiarity with model evaluation techniques and performance metrics.
• Proficiency with Python or JavaScript for prototyping and evaluation.
• Experience with Agile SCRUM development processes.
• Excellent planning, organization, and interpersonal communication skills.
• Ability to work independently and collaboratively across functional groups.

• Experience in ERP systems, enterprise software, or complex business domains.
• Familiarity with RAG (retrieval-augmented generation), embeddings, MLOps, and responsible AI practices.
• Experience with fine-tuning methods (LoRA, QLoRA, PEFT), running structured evaluation series, or human-in-the-loop systems.