Senior Test Analyst
Centorrino Technologies
Senior Test Analyst
Smartstream Limited· Quality Assurance
About this role
The Senior QA Engineer will drive the quality strategy for agentic AI systems, including applications powered by LLMs, autonomous agents, RAG pipelines, tool-calling, memory modules, and complex AI decisioning flows.
The role ensures that AI-driven capabilities meet strict enterprise-grade standards of accuracy, safety, reliability, and compliance, while collaborating closely with engineering, product, and AI/ML teams globally.
Job Responsibilities
- Write automation alongside development to test unit level, API and e2e UI.
- Implement backend services supporting agent runtimes, tool execution, and memory layers
- Integrate LLMs into business workflows and internal systems
- Contribute to orchestration logic connecting tools, APIs, and external services
- Write clean, maintainable, well-tested code
- Participate in architectural discussions
- Collaborate with AI engineers to productionize experimental features
- Contribute to monitoring, logging, and performance optimization
- Strong hands-on experience testing LLM-based or agentic AI applications.
- Expertise in Python automation (PyTest, Playwright, Postman).
- Understanding of prompts, embeddings, vector stores, tokenization. • Experience testing RAG pipelines and grounding data workflows.
- Proficiency with APIs, microservices, and cloud environments (Azure/AWS/GCP).
- Ability to evaluate unstructured AI output using structured scoring techniques.
- Strong analytical and debugging skills in complex AI-driven workflows.
Desirable Skills
- Experience with AI evaluation tools: TruLens, PromptLayer, MLFlow, W&B.;
- Familiarity with agentic frameworks: LangChain, AutoGen, Semantic Kernel, CrewAI.
- Understanding of LLM safety concepts, adversarial testing, guardrails, red teaming.
- Knowledge of model fine-tuning, RLHF, and observability pipelines.
- Exposure to distributed architecture and enterprise platforms.
- Ability to mentor QA engineers on AI testing methods.
Qualifications
- 5+ years of professional software engineering experience
- Hands-on work with generative AI and agentic systems.
- Experience working in cross-functional product teams
- Degree in Computer Science, Engineering, or related field (or equivalent experience)
- Strong problem-solving skills and ability to work in fast-evolving technical domains
- Interest in AI-native software architectures and continuous learning mindset
- Certifications such as ISTQB Advanced, Azure/AWS/Google AI certifications (optional).
Equality Statement
SmartStream is an equal opportunities employer. We are committed to promoting equality of opportunity and following practices which are free from unfair and unlawful discrimination.
Frequently Asked Questions
Is the salary disclosed for the Senior Test Analyst position at Smartstream Limited?
The salary for this Senior Test Analyst role at Smartstream Limited is not publicly listed. Click "Apply Now" to learn more about the compensation package on their official careers page.
Where is the Senior Test Analyst position at Smartstream Limited located?
This Senior Test Analyst role at Smartstream Limited is based in Mumbai, Maharashtra, India. The position is listed as on-site or hybrid. Check the full job description or apply directly to confirm the work arrangement.
Is the Senior Test Analyst role at Smartstream Limited full-time or part-time?
This is listed as a Full time position. It is posted as a Senior Test Analyst role in the Quality Assurance department at Smartstream Limited.
Which team or department does the Senior Test Analyst at Smartstream Limited belong to?
This Senior Test Analyst position is part of the Quality Assurance department at Smartstream Limited. See the full job description for more information about the team structure and responsibilities.
How do I apply for the Senior Test Analyst position at Smartstream Limited?
Click the "Apply Now" button on this page. You will be redirected to Smartstream Limited's official application portal hosted on workable where you can submit your application directly.
When was the Senior Test Analyst job at Smartstream Limited posted?
This Senior Test Analyst position at Smartstream Limited was posted on Jul 31, 2026. Apply as soon as possible — early applications are often reviewed first.
Senior Test Analyst
Smartstream Limited
You'll be redirected to Smartstream Limited's official application page on workable.