| Jc,
What are the objectives of the benchmark? Defining crisp objectives will
lead you to choose the right tool and approach.
You have several options:
A. emulate users on physical clients (8000 of 'em!)
B. emulate users on physical clients and eliminate think times and
intertransaction delays to reduce the number of clients needed, but present
the servers with the same volume of work
C. emulate the clients
D. take a hybrid approach by using some number of physical clients running
at maximum speed, and emulate the rest of the clients
E. Write a driver tool that emulates the various interactions between the
clients and the servers.
F. benchmark a smaller number of clients using any of the approaches above
and use analytic or simulation modeling to extrapolate the performance
to the desired number of users.
You can trade off between cost, complexity, and accuracy depending on what
aspects of the system are of critical importance or simply of passing interest.
There are many tools available. The ones we use in-house are:
Type A: Softbridge ATF is the primary tool we use. Other tools are MS Test,
Visual Test, Mercury Interactive WinRunner and LoadRunner, Segue QA-Partner,
and SQA Robot.
Type B: Same as type A.
Type C: We use Pure-Atria PerformixCS. Other tools are SQA Robot and Mercury
Interactive WinRunner and LoadRunner. This approach is best suited to
loading database engines since most of these tools provide ODBC interfaces
that will connect to the database and submit SQL statements to the server.
Do not overlook the infrastructure needed to drive this number of users. For
a recent client/server characterization project, we needed an AlphaServer
8400 to reach 1600 on-line users. We have 100 PC clients on racks. If we
need more, the client server center in Salem NH has 250 PC clients and the
capability to add many more.
Enjoy,
Marc
|