Monday, April 26, 2010

Software Test Automation tool evaluation


In this post I will explain about evaluating Automation test tool and selecting appropriate tool suitable for our requirements.

Before start evaluating tool we should analyze whether automating software testing will really give any benefit over manual testing for your needs.


Actually, Software Test Automation is a good way to cut down time and cost.

But, it will reduce the cost and time only when it is really necessary or it is used effectively.

Test Automation is not required if you are going to use your application one time only or for short period only. For example, assume that you are having a website developed in ASP, and you are making some changes in this website. And, assume you are having solid plan for converting/migrating this ASP site into either ASP.NET or PHP in near future.

In this situation, it is not advisable to automate the testing of the new changes done in the ASP site.
In this case, simply you can complete the testing manually and then you can start your automation testing preparation once after the migration is done.

So, basically we need to automate our testing procedure when we have lot of regression work.


Once after taking decision to do the test automation, the next step is selecting appropriate automation tool.

There are hundreds of tools available for automating the software testing and some of them are Free.
Test complete, SilkTest, SilkPerformer, QARun, QALoad, TestPartner, WinRunner, LoadRunner, QTP, Rational Robot,Selenium,WATIR and openSTA are some of them.

Some of these Tools (e.g Selenium) are open-sourced.


We need to select appropriate tool based on below factors.

Budget allocated for the Testing process - Price for each automation tool will vary. Some of them are costly, and some of them are even free.
License pattern will be varying for each tool. License cost of some tools will vary according to geography location also. And, some tool vendors will fix different price for seat license and floating license.

So, first we need to decide about our licensing needs. i-e Ask below questions,
- In case the tool price changes according to geographic location, whether it will be cost effective for your location.

- How many Automation Test engineers will simultaneously work in your automation project?

- Whether you need separate set up for developing the scripts and for executing the scripts?

- Whether you are having plan to automate your any other testing activities? Whether the selected tool can be used for other projects also?


Support available for the Automation Tool. We need to evaluate whether the Tool provider will provide enough support and bug fix releases. And, we need to think about the support provided by the Forum community also.

Analyze whether the execution speed of the automation tool matches with your requirements.


Check the installation requirements (both Software and Hardware) for installing the automation tool in your test script development environment.

List down current skill set of your testers and check whether the tool can be effectively used by your testers. For example QTP will support vbscript, if your testers know vbscript they can easily learn using QTP.



Feasibility study is very important before finalizing the Tool. Most of Tools will provide evaluation or Trail offer. For example QTP can be downloaded from HP site and we can use it for 14 days. During this trial period try to automate different portions of your application to make sure that the Tool can be used for automating your Testing needs.

Analyze the Market Share and financial Stability of the vendor of the tool. It will get significance if you are going to use the Automation tool for long term regression testing purpose.


Check whether the Tool can be easily integrated with bug tracking tools. For example, QTP will be closely integrated with Quality Center (Test Director) which is a Test Management Tool.


The best approach is, we can prepare a list with all these factors and add remarks for each tool. And, we can select the tool by analyzing this list.

And, you can share you experience with Automation tool selection.

Software Testing Terms

eBook for learning Software Testing and QTP Automation

More Articles...
You can bookmark this blog for further reading, or you can subscribe to our blog feed.

No comments:

Search This Blog