QA Engineer (AI)

QA Engineer (AI)

Location: Montreal, QC (4 Days Onsite/Week)
Interview: Face-to-Face Required
Employment Type: Contract

Description du poste

We are looking for an experienced QA Engineer (AI) to lead testing initiatives for AI/LLM-powered applications. The ideal candidate will have expertise in Generative AI testing, RAG pipelines, prompt engineering, automation, and API testing, along with strong experience in QA automation and enterprise application testing.

Responsabilités clés

• Design and execute comprehensive test strategies for AI/LLM applications.
• Validate prompt quality, hallucination detection, bias, safety, and response accuracy.
• Test Retrieval-Augmented Generation (RAG) pipelines, vector databases, embeddings, and retrieval accuracy.
• Build and maintain automated test frameworks using Python and Java.
• Perform API, integration, regression, functional, and performance testing.
• Validate LangChain/LangGraph workflows and AI agent integrations.
• Collaborate with developers, product owners, and business stakeholders.
• Lead defect triage, quality reporting, and production support.
• Mentor QA team members and promote QA best practices.

Exigences

• 7+ years of QA/Quality Engineering experience.
• 3+ years in a Senior or Lead QA role.
• Strong experience with:
  • AI/LLM Testing
  • Prompt Engineering
  • Hallucination & Bias Testing
  • RAG Testing
  • Vector Databases
  • LangChain / LangGraph
  • Python & Java
  • API Automation
  • Selenium, Playwright, Robot Framework, or similar
  • CI/CD (Jenkins, GitHub, etc.)
• Experience with SQL, database validation, and reconciliation testing.
• Knowledge of LLM architecture, embeddings, tokenization, and inference.

Compétences préférées

• Model Context Protocol (MCP).
• AI Safety, Red Teaming, and Responsible AI testing.
• Capital Markets or Securitization domain experience.
• DevOps and cloud-based testing.
• Agile/Scrum methodologies.

eye