Automated testing remains a significant challenge, finds Keysight commissioned Forrester research

28 June 2022

What test approach does your company use today? From the report, Conquer Testing Complexities with Automation & Artificial Intelligence (AI)
What test approach does your company use today? From the report, Conquer Testing Complexities with Automation & Artificial Intelligence (AI)

Few firms employ fully automated testing or use AI, despite 84% stating that the majority of testing involves complex systems, found electronics design, validation & test firm, Keysight Technologies in its global study, conducted by Forrester, that surveyed test operations decision-makers to understand the state of test strategies & technologies.

The research report, titled Conquer Testing Complexities with Automation & Artificial Intelligence (AI), identified that automation is gaining inroads, with 75% of organisations using a combination of automated and manual testing. However, only 11% have a fully automated strategy. With growing complexity, the number of tests is increasing (77%) and without automation, product development slows.

According to Jeff Harris, Vice President, Portfolio & Global Marketing at Keysight Technologies, "We learned in the study that companies feel pressure to do more test automation, especially when asked about the future. Manual or partially automated strategies simply can't keep up with the needs of organisations today, and without AI-powered automation, they will struggle to conquer testing complexities. Additionally, we expect the COVID pandemic to accelerate adoption of remote development for automated test sequencing. We also expect a much higher use of digital twins as development teams strive to continue working together, but from different locations. At Keysight, we’re building this automation of intelligence across the workflow into our entire software product portfolio to address these needs and drive test and validation automation forward."

The primary technical challenges resulting from testing complexities are long cycles, accurately capturing bugs and issues, and then fixing them. These problems directly impact business results and the product development process. Respondents ranked the outcomes as:

•    51% - Security breach risk

•    48% - Increased expenses

•    42% - Slower time to market

•    36% - Defective product

•    34% - Loss of revenue

However, enterprises recognise that manual and partially automated testing strategies cannot keep pace with the increasing complexity of products. As a result, 45% will consider using a fully automated approach in the next three years, a 409% growth. Over half (52%) stated they would look at AI for integrating complex test suites, a 325% growth.    

By shifting to intelligent automated testing strategies, organisations expect benefits. Operational improvements include increased productivity (59%), ability to simulate product function or performance (54%) and bug fix automation or simulation to save time on fix-retest cycles (53%). The anticipated business impact spans higher quality product that increases customer satisfaction (59%), the ability to reduce time to market (50%) and more agile product development cycles (50%).


As described in the survey, the most anticipated technology or operations improvements from automation and AI are increased productivity, ability to simulate product function/performance and bug fix automation/simulation.

But a better metric for success is the business impact that organisations expect from the investment in new testing technologies such as Keysight's Automating Intelligence, which offers customers:

•    A higher-quality product that increases customer satisfaction

•    The ability to reduce product time-to-market

•    More agile product development cycles


In this study, research & advisory firm, Forrester conducted an online survey of 406 test operations decision-makers at organisations in North America, EMEA and APAC to evaluate current testing capabilities and to hear their thoughts on investing in automation (including AI). Questions provided to the participants asked about their organisations' current testing environments, future investments, challenges and expected outcomes from testing automation.

Additional information:

•    Research summary is located at:

•    Q&A with Jeff Harris, Vice President, Portfolio & Global Marketing at Keysight Technologies can be found at:

•    More information about Keysight’s software test automation tools & solutions can be found here

Contact Details and Archive...

Print this page | E-mail this page

This website uses cookies primarily for visitor analytics. Certain pages will ask you to fill in contact details to receive additional information. On these pages you have the option of having the site log your details for future visits. Indicating you want the site to remember your details will place a cookie on your device. To view our full cookie policy, please click here. You can also view it at any time by going to our Contact Us page.