The Forrester Wave™: Continuous Automation And Testing Services, Q2 2024

Continuous Automation And Testing Services Vendor Product Comparisons 2024

Key features and pros and cons of top Continuous Automation And Testing Services products:

IBM - Leaders

Pros:

AI capabilities and innovation bringing research prowess to CAT services
Flagship testing accelerator IGNITE with comprehensive AI-cycle testing approach
Robust global delivery organization with extensive regional presence and diverse ecosystem
At the forefront of application modernization testing using genAI LLMs and Watson.ai
Strong support for large-scale legacy application migration (COBOL to modern stacks)

Cons:

Testing in production is below par in leveraging modern practices like A/B testing, feature flags, blue-green deployment, and crowdsource testing
Scale challenges prevent consistent delivery of IBM's depth and experience to all clients
Reference customers noted they don't consistently get the benefit of IBM's full capabilities

Infosys - Leaders

Pros:

Excellent AI talent strategy focusing on AI-ready workforce for increased automation productivity
Innovation approach paying off with AI integration into all software testing accelerators
Global delivery strategy spanning geographies with resident workforce beyond India
Excellent cloud testing services with comprehensive solutions, partnerships, and automation frameworks
Strong differentiation in testing AI-infused applications through proprietary IP
Flexible outcome contracts with transparent behavior and billing

Cons:

Reference customers noted the firm should improve its ability to communicate the value it delivers

Qualitest - Leaders

Pros:

Prominent global pure-play testing vendor with strategic acquisitions in AI, blockchain, physical-digital, telco, healthcare, and life sciences
Compelling vision of 'zero-touch quality orchestration' with strong AI foundation
Open ecosystem with niche partnerships to transform workforce offerings
High scores in testing in production, especially AI-infused criteria
Strong capabilities in testing AI-infused applications and genAI
Effective talent sourcing, recruitment, retention, and management strategies

Cons:

Test environment and test data management services are below par due to heavier focus on in-house accelerators
Reference customers expressed concerns about pricing flexibility as services scale

Accenture - Strong Performers

Pros:

Plans to double AI talent to 80,000 with investments in assets, industry solutions, and acquisitions
Shifted skill sets toward cloud and engineering capabilities in testing
Numerous AI-infused accelerators available
Strong global services with growing AI and engineering capabilities
Cloud-first AI strategy alignment

Cons:

Below-par talent strategy and average staff retention rates compared to peers
Not transparent about investments in AI and genAI capabilities
Does not state level of investment in developing testing organization skills
Customers did not show strong utilization or appreciation of AI-infused accelerators
Declined to participate in full Forrester Wave evaluation process

Capgemini - Strong Performers

Pros:

Vast CAT services clients and project credentials with extensive experience
Differentiated market vision with robust innovation efforts
Several testing labs worldwide
Strong thought leadership in the testing space
On-par position across advisory, consulting, execution models, automation, cloud, test data, AI-infused practices, accelerators, and employee retention

Cons:

Partnership strategy needs fine-tuning to shift from scale to qualitative and selective approach
Should prioritize partnership outcomes based on client impact rather than opportunities and deals closed
Scored lower in leveraging AI and genAI for legacy application modernization
Reference customers noted need for improvement in offshore talent retention

Cognizant - Strong Performers

Pros:

Automation strategy expanding across various markets including manufacturing and physical-digital space
Strong emphasis on partnerships as key component of testing strategy with cocreation focus
Outstanding scores in CAT execution models, test design/optimization automation, automation engineering, and atomic accelerators
Extensive infusion of AI, ML, and genAI in accelerators
Strong knowledge base with continuous evolution of ideas to address market trends
Specialized expertise in technologies and vertical markets

Cons:

Below par scores in methods and approaches for testing genAI and LLMs
Some reference customers faced challenges acquiring niche skills for specialized testing projects

EPAM - Strong Performers

Pros:

Bold stance on disruptive impact of AI and genAI on testing and development services
Innovative approach enabling cost-effective services using AI accelerators with less senior professionals
Differentiated talent strategy
Strong reliance on in-house resources including open-source accelerators and AI-driven platforms
Excels in test design/optimization automation
Solid test automation engineering services

Cons:

Partnership strategy falls short and needs to focus on measuring customer over partner benefits
Weight on CAT test scenarios doesn't align to customer and wider market needs
Reference customers emphasized need for improved communication on status updates

EY - Strong Performers

Pros:

Nimble testing services organization that can pivot quickly to embrace emerging technologies
Vision to replace labor-intensive testing with intelligent technology-driven quality services
Innovative testing accelerators and practices
Robust talent management program and culture
Integrates testers into multidisciplinary development and delivery teams
High employee retention rate
Flexible and transparent pricing approach to building partnerships

Cons:

Ongoing communication and feedback with clients can improve
Offers insights platform with metrics but reference customers need more direct feedback on internal skill gaps
Performance in testing AI-infused applications falls below expectations
Gaps in measurement capabilities

TCS - Strong Performers

Pros:

Significantly global large-scale testing delivery with modern Chennai campus
Competitive vision fostering innovation and strategic talent strategy
Product-led services vision using AI and genAI in platform accelerators
Excels in test environment and test data management services
Rich accelerator licensing model and integrations with own and third-party tools
Strong capabilities and strategic vision engendering trust
Proficiency in developing and implementing defined solutions

Cons:

Test design and optimization solutions are below par with less genAI innovation
Use of dated model-driven approaches in some areas
Reference customers highlighted room for improvement in consulting space

Wipro - Strong Performers

Pros:

Strong global presence as well-established India-based systems integrator
Solid strategy with pricing flexibility and transparency
Excels in nonfunctional testing with outstanding test environments and data management services
Structured programs and insightful customer case descriptions
Successful genAI test data management pilots
Comprehensive cloud migration and testing services
Advanced approaches for testing in production
Flexible and transparent pricing described as 'reasonable'

Cons:

Performance in testing AI-infused applications is below par
Can improve in automating end-to-end testing and test orchestration across models and software
Testing accelerators presented more à la carte lacking unified platform view
Vision, innovation, partnerships, and global delivery competitive but not differentiated

Amdocs - Contenders

Pros:

Thriving global testing services business rooted in telco quality engineering
On-par vision and innovation particularly in AI with corporate platform brAIn
Integration of AI, ML, and genAI into long-standing Ginger testing platform
Scored well in CAT testing scenarios
Excels in employee retention
Reference customers praised desire to listen to client needs and address them

Cons:

Talent strategy is below par with relatively lower planned investments
Can enhance upskilling efforts across professionals
Global delivery reach requires improvement in hub-and-spoke organization
Test automation and design optimization beginning to trail peers
Cloud and AI-infused application testing fall short of wider market
Reference customers noted uneven skills among team members
Poor communication regarding talent strategy with clients

HCLTech - Contenders

Pros:

Well-established global testing services company with software products expansion
Valuable experience from partnership with IBM in DevOps and rational testing products
Excels in accelerator licensing and integration approach
On-par approach in partnering, flexible and transparent pricing, talent strategy, and global delivery
Balanced approach to genAI testing with manual teams, automation, techniques, and commercial tools

Cons:

Vision and innovation scores below average with no clear differentiating vision statement
Lowest number of patents for innovation and low R&D investment
AI and genAI platform capabilities are below par
Test automation engineering, testing in production, cloud testing services, AI-infused application testing, and AI testing for legacy migration all scored below par
Testing services have high aspirations but not quite there yet
Provided only two reference customers for evaluation

LTIMindtree - Contenders

Pros:

Merger expanded global testing services with offices worldwide
Excels in testing production techniques with strong proof points and wide range of methods
Improved in testing AI-infused applications and genAI
Reference customers appreciated flexibility, support, and receptiveness to feedback

Cons:

Lags in effectiveness and reach of global delivery model compared to peers
Innovation falls short particularly in leveraging genAI
Below par in utilizing genAI in accelerators and applying genAI to legacy application migration testing
AI- and ML-infused accelerators in à la carte and platform approaches need enhancement
Reference customers wanted more references for genAI testing and a stronger global delivery model

FAQs

Q: Which vendors are Leaders in 2024?

A: IBM, Infosys, Qualitest

Q: What does "Leader" mean in Forrester Wave terms?

A: Leaders in the Forrester Wave for Continuous Automation and Testing Services demonstrate the strongest current offering and strategy. These vendors excel across multiple dimensions including AI and genAI capabilities, global delivery models, innovation in testing accelerators, talent management, and the ability to deliver comprehensive testing services for modern application development. Leaders show strong vision for the future of testing services, transparent pricing and client relationships, and proven capabilities in emerging areas such as testing AI-infused applications and leveraging generative AI to enhance testing productivity. They typically have extensive client references, significant market presence, and demonstrate both strategic thinking and execution excellence.

Q: How do I interpret a Forrester Wave?

A: https://www.forrester.com/policies/forrester-wave-methodology/

Report:

Top Vendor Products for Continuous Automation And Testing Services in 2024

Continuous Automation And Testing Services Vendor Product Comparisons 2024

Pros:

Cons:

Pros:

Cons:

Pros:

Cons:

Pros:

Cons:

Pros:

Cons:

Pros:

Cons:

Pros:

Cons:

Pros:

Cons:

Pros:

Cons:

Pros:

Cons:

Pros:

Cons:

Pros:

Cons:

Pros:

Cons:

FAQs

Q: Which vendors are Leaders in 2024?

Q: What does "Leader" mean in Forrester Wave terms?

Q: How do I interpret a Forrester Wave?

Reference