Responsibilities / Job Description

• Design and execute test strategies for machine learning models (classification, regression, NLP, CV, etc.).
• Validate model accuracy, fairness, bias detection, explainability, and robustness under varying datasets.
• Perform adversarial testing and edge-case validation for AI systems.
• Test autonomous agents, multi-step reasoning paths, and state transitions.
• Validate correctness, grounding, consistency, and safety of LLM outputs.
• Evaluate prompt robustness and behavioral variations across scenarios.
• Validate retrieval accuracy, grounding quality, and hallucination reduction.
• Test vector store behavior, document chunking logic, and retriever configurations.
• Validate fallback behaviors when tools or external services fail.
• Execute adversarial, prompt-injection, and red team style testing.
• Validate compliance with data privacy, business rules, and insurance guidelines.
• Benchmark LLM latency, throughput, and multi-agent performance.
• Validate concurrency handling, degradation logic, and retry mechanisms.
• Document test cases, behavior maps, prompt variations, and evaluation reports.
• Work closely with AI engineers and SMEs to refine agent workflows.
• Functional & Non-Functional Testing:

o Conduct API testing for AI services and model endpoints.
o Validate performance, scalability, and latency of AI inference services.
o Perform security testing for data pipelines and model-serving endpoints.

• Quality Governance & Compliance:

o Ensure compliance with AI ethics, data privacy, and regulatory standards.
o Establish KPIs for model quality (precision, recall, F1-score, drift detection).
o Document test strategies, results, and provide audit-ready evidence.

• Collaboration & Leadership:

o Work closely with data scientists, ML engineers, and DevOps teams to integrate testing into the AI lifecycle.
o Mentor junior testers and contribute to building reusable test accelerators.
o Represent the testing function in client discussions, providing insights on AI quality assurance.

• Develop automated test frameworks for AI/ML pipelines.
• Implement synthetic data generation for edge-case testing.
• Conduct bias and fairness testing across diverse datasets.
• Validate monitoring dashboards for model drift and performance degradation.
• Perform regression testing for retrained models.
• Create reusable test assets for AI projects (scripts, datasets, frameworks).
• Provide detailed defect analysis and SLA-driven closure.

Required Skills

• Strong experience in software testing (manual + automation) with focus on AI systems.
• Hands-on expertise in Python for test automation, data validation, and AI testing scripts.
• Knowledge of Gen AI frameworks (Langchain, Langgraph, CrewAI).
• Familiarity with CI/CD pipelines (Jenkins, GitHub Actions, GitLab CI).
• Strong understanding of data validation, model evaluation metrics, and drift detection.
• API testing experience (REST, GraphQL) with tools like Postman or REST Assured.
• Solid knowledge of cloud platforms (AWS, Azure, GCP) for AI deployments.
• Exposure to AI ethics and compliance testing.
• Knowledge of containerization & orchestration (Docker, Kubernetes).
• Experience with Big Data ecosystems (Spark, Hadoop) for large-scale AI testing.
• Familiarity with observability tools (Prometheus, Grafana, ELK stack).
• Understanding of security testing for AI pipelines (data poisoning, adversarial attacks).