A
Hiring companyAnaplan
Active listing
Engineer II - QA Engineer
Gurugram, India Full-time On-site
About the role
AI Quality Assurance team at Anaplan builds testing frameworks for GenAI-powered features. The QA Engineer designs and implements comprehensive test strategies, automated prompt tests, and evaluation metrics to ensure reliability, safety, and performance of LLM applications. Based in Gurugram with an on-site work model, the role collaborates closely with engineers, data scientists, and product teams to define quality gates and monitor AI feature health.
What you’ll do
- Design and implement testing strategies for GenAI features
- Develop automated test suites for prompt and regression testing
- Create evaluation frameworks and quality metrics for LLM applications
- Build and maintain test datasets and golden examples
- Implement monitoring and alerting for AI quality degradation
- Conduct adversarial testing to identify hallucinations and biases
- Collaborate with engineers to define acceptance criteria and quality gates
- Document testing procedures and known issues
What you’ll bring
- 3+ years QA or test engineering experience
- Experience with AI/ML systems, especially GenAI
- Proficiency in Python and JavaScript for test automation
- Knowledge of software testing methodologies and API testing tools
- Familiarity with CI/CD pipelines and test integration
Nice to have
- Experience testing conversational AI or chatbots
- Understanding of LLM evaluation frameworks (LangSmith, PromptFoo)
- Knowledge of responsible AI principles
Skills
Python automationAPI testingCI/CD integrationPrompt engineeringLLM evaluationAdversarial testingTest data managementQuality metrics