Types of Testing

Volume Testing

Volume Testing, as its name implies, is testing that purposely subjects a system (both hardware and software) to a series of tests where the volume of data being processed is the subject of the test. Such systems can be transactions processing systems capturing real time sales or could be database updates and or data retrieval.

Volume testing will seek to verify the physical and logical limits to a system's capacity and ascertain whether such limits are acceptable to meet the projected capacity of the organisation's business processing.

Load / Systems Loading

The 'load' on a system refers to the demands placed upon it. The overall load combines many factors and includes :-

Total storage capacity for programs and data

• Number of applications being run concurrently
• Number of concurrent users, peaks, troughs and average
• Number of peripherals: e.g. using a file server as a print server increases demand, as each printed document is 'spooled' to the server's disk before being queued to the printer.

Whilst the sizing of hardware can become complex, once the above points are clear, other factors, such as expected response times / performance can be considered.

.

Stress Testing

Stress Testing is a form of testing which purposely attempts to identify the weakest link of a system. Stress testing will seek to verify that, following any abnormal conditions, the system can revert quickly to normal operation. Such conditions might include : data processing immediately after system downtime, after a network failure, or during peak activity periods.

Performance testing

Performance testing answers the question: How much can you get out of a given product? At VeriTest, we'll take the product you care about—whether it's one you make or one you're considering installing in your line of business—and push it to its limits. This knowledge is valuable at many points in the product life cycle, from development to marketing to technical support. Knowing how a product really performs also gives you the confidence necessary to install a third-party product in a mission-critical environment.

The first step in the performance testing process is determining what factors matter most to you: the number of transactions per second? The speed at which the product it can execute a list of tasks? The number of hours a battery lasts?

Next we determine the best way to measure those factors. Sometimes we'll use a benchmark tool, either from the industry-standard PC Magazine benchmarks that we develop here at VeriTest, or from one of the other test tools we regularly use, such as RadView's WebLoad. Sometimes it may be as simple as a using a stopwatch—physical or in software—to clock performance. More often than not, each performance test—even if it involves a standard test tool—requires a unique methodology, and this is where our years of experience in all sorts of performance measurement and test plan creation give us a big advantage over other testing labs.

Many performance tests also require an underlying testbed, which can be as simple as a single PC or as complex as hundreds of network clients and servers connected over a gigabit backbone with active network monitors. Whatever the appropriate testbed, we're likely to have it—and to have the experience to use it to best effect.

To get accurate performance results, you need testers with the right expertise and the right equipment. With years of experience in performance testing and in creating industry standards for performance measurement tools, VeriTest will take your product—and your profits—to the limit.

Accessibility testing

Examines how accessible your hardware, software, Web site, or internal application is to users with disabilities

Capacity planning

Helps determine the resources your company will need to provide a given level of performance

Capacity testing

Determines the capacity of a system, helping you set expectations and determine when the deployment/production cycle should start over

Certification

Assures your potential customers that your product will integrate effectively into their existing infrastructure

Compatibility testing

Examines whether your hardware, software, Web site, or internal application works as expected on the wide range of components and systems available in the homes and offices of your end users

Competitive analysis testing

Examines how your hardware, software, Web site, or internal application fits into the competitive landscape

Custom testing

Addresses testing needs you don't see represented on our site

Functionality testing

Examines the extent to which your hardware, software, Web site, or internal application meets expected functional requirements

Globalization testing

Helps in your efforts to prepare your hardware, software, Web site, or internal application for global release

Internationalization ("I18N") testing

Examines the extent to which your hardware, software, Web site, or internal application is ready to be customized for global release

Internet BenchMark Service

Provides a wealth of reliable, up-to-the-minute data on vital ISP performance areas

Interoperability testing

Helps when you're introducing a new hardware or software item or system into an existing environment

Load and stress testing

Measures server performance under the load of a specific number of users and the deterioration of server performance under a heavy transaction load over time

Localization ("L10N") testing

Examines the extent to which your hardware, software, Web site, or internal application is ready for global release

Market readiness testing

Helps determine whether or not you're prepared to take your product live

Migration testing

Examines the extent to which your applications, infrastructure, and data will remain intact as your company migrates from one platform or operating system to another

Multi-tier testing

Finds the weak links in environments with several levels of applications and/or servers working together in a complex relationship

Performance testing

Examines how much you can get out of a given product

QA testing

Ensures that your hardware, software, Web site, or internal application is free from bugs, glitches, and oversights

Scalability testing

Examines the performance of your Web sites, hardware and software products, and internal applications at all the stages from minimum to maximum load

Scale-up testing for Windows applications

Examines how well your applications scale-up to enterprise-level server configurations of Windows 2000 Datacenter Server

Security testing

Examines how well you're protected against denial-of-service attacks, hacking, network-sniffs, and more

Test plan development

Focuses our expertise on developing the right program to meet your needs

Usability testing

Helps keep support costs down and user satisfaction high by making sure your user interface is intuitive and easy-to-use

User-preference testing

Provides independent, third-party verification that users prefer your product to that of your competitor(s)

Windows 2000 migration testing

Helps you reduce the time, cost and effort of moving to the low-TCO platform

Load Testing Guide

The following table is a high level guide to help with delivering an effective and efficient load/performance testing project:-

Term	Description
Like Live	Always ensure that the test environment is a mirror of the live. If this is not possible then ensure that the servers, etc are scalable or have a quantifiable performance measure compared to the live boxes or components. Run one functional test machine at the same time the load is being generated.
Workload Model	Ensure that the workload model is derived from business users, transaction logs or similar ‘real life’ activities. Take account of peak times and transactions including end of year/period. Capture any background processes that the user experiences (whether knowingly or not)
Virtual Users	Ensure the Load/Performance test tool is able to simulate 125% of the anticipated user community
Load Test clients	Typical per virtual (NT box) memory requirement is 1mb. Try not to go much above 200 virtual users a box (however this depends on the Load Test product and test machine configuration). This would require 200mb to service the virtual users, at least 32mb for the operating system and more for any other applications that may reside on the box (this should be kept to a minimum). Ensure the test clients will not be a bottleneck by checking the CPU utilisation per test client. If the average is above 75% then the test client will be a bottle neck, reduce the amount of virtual users that client runs. Dual processors will typically give a 60 – 70% performance gain
Controller	Try to have a 19” monitor and a graphics card with 8mb+. This helps with monitoring several machines, reports, graphs, etc. 128mb of memory should be sufficient. Ensure that the PC has adequate hard disk space due to the amount of data that can be collected during tests. Don’t run virtual users on the controller PC.
Bandwidth	Ensure that there is sufficient bandwidth on the LAN to run all the virtual users. An example would be if the average amount of data being submitted (including images) by a virtual user was 25k and the test had to simulate a user community of 250 (concurrently), then the bandwidth requirement would be a minimum of in excess of 6mb/s. Note no network Topology will be 100% efficient in fact as clients increase on a LAN segment the available bandwidth can drop to below 50%. So in this case a 10mb hub would not be sufficient. It is preferable to use a switch as this does not share the available bandwidth between clients. In most cases a 100mb switch will be sufficient. Ensure that all test machines have 100mb Network Interface Cards (NIC).
Monitors	There are various resources that may be used to monitor system performance. All load test tools will usually have an array of performance monitors or have access to the servers under tests own monitors. However to reduce network traffic it is best to only use basic monitors as supplied by the load tools. These include response times (transactions), this has no network overhead as it measures the round-trip time of selected transactions. Note: Response times do not usually include the time it takes the browser to render the returned response (which can include several images). To measure this at least one test client should be included in the test to carry out functional tests while the system is under load. This client should be built to the same specification as those in the live environment. Also network/node-based monitors should be used to give a general indication of which segment of the system caused the most delay. Monitors should be set up on the local machine(s) as prescribed by system specialists. In some cases the tester may want to remotely monitor these on their machine, this can be achieved by using emulation software such as Exceed or VNU (VNU is free). Each monitor that you run will degrade the performance of the machine it is run on. This performance hit will be small but it should be known so that an account can be made of it in the test results. You may get an estimate of this by closing all non-critical processes on the server and then run the monitor and see what the CPU utilisation is with the monitor on and then what it is with it off. The hit is (approximately) the difference.
Load Test Scripts	When a client opens a connection to a web server this is known as a session. While the client continues to send and receive responses from the server, the session is maintained by a session ID. Sessions can be maintained by a variety of methods. It is crucial to performance testing web-based applications to ascertain how the web server/application identifies the session as this will need to be used in subsequent tests. Don’t use random number generators to produce unique id’s for log on, etc as inevitably (especially where extended tests are concerned), it will generate two numbers that are the same. In any HTTP traffic between client and server the GET and POST functions are the most important.
Test Results	Use Mean and Standard Deviation in preference to Min and Max values when analysing results. Min and Max values are normally known as outliners and can skew results. Users are not normally concerned if once in a while they get a response time that is less than expected. However what does concern users is what the performance is ‘normally’. Get as much information as is possible. Take your time to analyse the results with system specialists and rerun tests to confirm your findings.