Load
testing covers more than just performance testing. It includes stress
testing, endurance testing, reliability testing and concurrency
testing. The “its just performance testing” view results in most
projects doing too little load testing leading to extra costs and
unhappy customers. This article presents a different approach which
avoids these pitfalls and generates additional business benefits.
People often equate load testing with performance testing. Load testing
is seen as a way of answering the question “How fast does the system
respond?” This view then tends to mean that load testing is seen as an
end of project activity. Only at the end of development will we have
the final implementation for performance testing and so we can confirm
only then that it performs quickly enough in the real world and
smoothly transition into live service.
Wrong approach! This is extremely risky and misses out on the many
benefits of starting load testing early and applying it throughout the
project. With this approach does the system sail through load testing
and transition smoothly into service? Occasionally yes. But more
frequently the system starts to fail as load starts to be applied, even
with small increases in volume.. For the first time there are
concurrent demands on the system and arbitration over resources is
required. Paths through code that have never been executed are
triggered, situations arise that nobody really thought through.
Transactions fail. Systems crash. After these problems are fixed and
more load is applied in a test, we then encounter problems like
resource exhaustion, buffer overflows, timeouts and inconsistent
behaviour. The real work needed to turn a functional pre-production
system into a robust solution has only just begun.
Examples abound of products that failed when load testing started and,
after lots of effort, stress and expenditure, have been shelved. Worse
still are the ones that missed load testing altogether and failed
dramatically during live operation. An internet portal developer
recently stopped development of a new service, one that had completed
functional development, when load testing revealed fundamental
structural problems and inefficient coding which led to a poorly
performing and unstable system.
So what should you do to avoid these risks? We all know it is better to
find faults early when they cost far less to fix yet load testing is
still left until as late as possible. The types of faults it finds
frequently need architectural changes and major rewrites which are by
then are hugely expensive to implement. The answer is that you should
start early. Different forms of load testing should be repeatedly
applied throughout the project to identify problems early and to check
that the system is not going off track.
This is a natural extension of the practice of test led development.
Test led development, where automated tests are written first and code
must pass these tests as it is developed, offers major benefits.
However, in its current form, the focus of this testing is on
functionality. As it evolves the functional status of the software is
always known and hence manageable, functional faults are nipped in the
bud avoiding high cost fixes, the functional risk is greatly reduced.
Not so other risks. If a project performs early and continuous load
testing it gets much wider and comprehensive risk reduction. To make
this effective:
1. Study the system and perform a risk analysis to help to order the
threats to the system, this will help you to prioritise load testing
activities.
2. Collect data to allow comparison of the efficiency of different
builds. This permits monitoring of the long term trend, “Is the system
using more and more processor time to do the same work?” This data can
be used to predict resource requirements at different levels of demand
and so support scalability predictions.
3. Execute tests that aim to assess the behaviour of the system and to
trigger faults under load. Use workloads that simulate expected
patterns of demand to observe the aggregate behaviour of the system.
Use specially targeted extreme workloads to probe the vulnerabilities
of the system.
4. Include the full spectrum of load tests into the test suite. This
means performance testing with typical and busy period work loads;
stress testing to check both atypical demand spikes and resource
exhaustion impacts; endurance testing that uses both operational period
and cumulative operation tests; reliability testing that runs lots of
transactions and then checks whether occasional transactions fail;
concurrency testing of two users working on the same account at the
same time.
5. Design measurement activities as scientists would design an
experiment, design them to provide data that can be analysed. Sample
the system under different steady state workloads to provide multiple
data sets to support interpolation. Chose the workloads to permit
estimation of the resource costs for each transaction type.
6. Target the middleware first with generic activities and evolve the
suite as functionality is developed. Start early and then test each
incremental release of the system, firstly with the previous suite and
then with a modified suite that addresses new functionality.
7. Invest the time and resources to work at a representative scale.
Maybe the test bed can’t be full scale but it should not be two orders
of magnitude smaller than the intended system. Be smart and innovative
to use resources effectively to provide an appropriate scale test bed.
The costs that will be incurred if this is not done will far exceed the
cost of providing the test bed.
8. Don’t delay; test an increment as soon as possible. Don’t skip one
or you’ll end up skipping them all. Compare the measurements and
behaviour with the previous one, is it better or worse?
9. Provide a background load for functional testing. Features that work
offload may fail when the system has other things to think about.
10. Consider occasional events such as server failures and reconfiguration of the system. Do these need to be tested under load?
In conclusion, you need to incorporate load testing throughout the
development process. Leaving load testing until the final run in to
live service is a recipe for disaster. If this became common practice
then a lot more applications and systems that work would be delivered
on time and to budget.
© Acutest Ltd 2005 – http://www.acutest.co.uk
