Conference ulysse::rdb_vms_competition

83.0. "Debit Credit Comparisons" by HPSVAX::KASTNER (Peter Kastner CSG 491-0364) Fri Mar 04 1988 15:42

A PRIMER ON COMPARING DEBIT CREDIT TESTS

BACKGROUND 
As debit credit has emerged as the defacto commercial/TP benchmark, more
business and engineering decisions have been made based on comparisons of
various vendor's debit credit reported results.  This makes sense because debit
credit is supposed to be a well-defined benchmark [Datamation, April, 1985, "A
Measure of Transaction processing performance"] where vendor results can be
compared.  The reality is that every vendor has reported debit credit results
under different conditions, sometimes unspecified.  

This primer is designed to level the playing field a bit by providing some
conditions which readers might apply to reported debit credit results.  Serious
miscalculations in engineering specifications, market requirements, cost goals
and other activities can result if debit credit results are compared between
vendors without some consideration for the differences between the vendor's
testing and reporting methodology. 

While the author is not recommending the 'normalization' of debit credit
results, I strongly urge anyone considering the comparison of vendor's products
based on debit credit to read on.  

What you won't find here is results. This is not the forum to publish Digital
debit credit results. 

DIGITAL EQUIPMENT
DEC uses three Styles of debit credit.  A full description of Digital debit
credit is available from the author.

Style 1 is a fully qualified debit credit where all presentation services
and database activity is handled by the system under test.  We believe Digital
Style 1 to be the most rigorous implementation of debit credit.  Style 1
results cannot be directly compared with any competitor's results.  As a rule,
Digital engineering uses Style 1 as the metric for projecting database and
system performance.

Style 2 offloads forms and character processing to a front end VAX.  The
front end cost is included in cost of ownership results.  Because VAX
architecture, ACMS, DECintact etc. support such distribution, this is a
viable model and as rigorous as Style 1.

Style 3 assumes MicroVAXes or intelligent controllers are in the 'bank
branches' of the debit credit.  This offloads character and forms presentation
services from the system under test and appreciably improves throughput,
and hence price/performance.  In general, Digital Style 3 results should
be used to compare DEC with other vendors since no other vendor tests the
costs of presentation services (which use 40% of the VUPS in a Style 1 test
of an 8700).

IBM
IBM has not generally reported debit credit results, depending on RAMP-C
to obscure comparative performance.

System/88
Tests based on Stratus ET-1, which is not debit credit.  IBM results were
obtained at 70% cpu utilization (an in-house rule), and do not (in spite
of published reports in FTSN) use data integrity features.  Otherwise, see
the Stratus results for comments.

370 architecture

Testing on the 9370, 4381 and 3090 were conducted by DEC (CSG and HPS) using an
experienced contractor with extensive IBM experience.  The results were
obtained over a period of years using a consistent methodology.  The results
can be directly compared between the 370 architecture machines, and probably
can be extrapolated to other similar models not tested.  Note that various
versions of layered software were used, and that such layered software tends to
get better over time. 

The DEC-sponsored 370 architecture tests were run against 'qualified' (eg the
tests followed the letter and spirit of the Datamation debit credit benchmark
description)debit credit standards. Note that because IBM uses intelligent
terminals and controllers (3274/3270), IBM does not pay the async and forms
overhead that Digital does with Style 1 debit credit.  Although somewhat
controversial within DEC, the 370 results can be compared to Digital Style 2 or
3 with distributed processing on MicroVAXes.

OneKay, a 1000 tps benchmark reported by IBM in late 1987 has been used
to extrapolate some 750 debit credit tps under IMS Fastpath.  The author
believes this to be a bad conclusion.  Since IMS Fastpath uses in memory
databases with limited access capabilities, it is not reasonable to compare
such results with fully journaled disk-based debit credit.  Second, no
presentation services were used.  At 750 tps, some 22,500 field maps per
second are necessary for fully qualified debit credit.  

System/3X
No debit credit results reported.  Digital is sponsoring tests of the System/38
and Silverlake.

TANDEM
Tandem's results are fully documented in a Tandem white paper on what they
called the  TopGun benchmark (March 1987).  Tandem disclosed the faults
mentioned below, which goes to prove that non-conformance to the debit credit
standard is OK as long as you tell people about it...just don't compare
the results to anything else.

Tandem reports from 2.5 tps per processor on the CLX to 6.5 tps per processor
on the VLX using the TopGun methodology commented on below.

Tandem uses 90%-2 second response times instead of 95%-1 second standard.
Divide Tandem's reported results in half to get to 95%-1 second.

Tandem simulated only 1000 tellers per 100 tps, not 10,000.  This saved memory,
communications buffer overhead and other unspecified costs.  Digital always
simulates the proper number of tellers and database sizes.

Tandem published the source code for their benchmark.  It shows that no
presentation services were used (because Tandem's screen COBOL is interpreted
and performance would have died).  Therefore, Digital Style 3 is an
approximation of Tandem's tests.

The TopGun file distribution was highly artificial but completely balanced
for all-out performance.  Be careful to ascertain the structure of Digital
tests on VAXclusters before comparing to Tandem's tests.  Tandem uses physical
and logical partitioning by processor (which their software supports
completely.  Digital tests which partition files on the Cluster so as to
minimize distributed lock manager overhead are the best comparisons to what
Tandem did.                                                

The NonStop SQL code Tandem used had proprietary extensions which treated
the teller and branch files as having relative record keys.  Presumably,
it's much easier to do a relative file I/O than a relational row retrieval.

In reality, a Style 1 qualified test of Tandem will likely show performance at
one-quarter of Tandem's published numbers. roughly comparable to Tandem's
tests.
 

From a price/performance standpoint, Tandem has used debit credit performance
as a CPU metric when deriving newly announced product price/performance. Tandem
will quote debit credit performance on new product entry configurations not
capable of handling the I/O of debit credit.  Tandem derives
it's CLX price/performance as measured in K$/tps by dividing the CLX CPU debit
credit throughput (under unspecified conditions) into the entry system package
price.  Therefore, the reader should avoid comparing Digital five year costs
of ownership for a full debit credit configuration against Tandem's
price/performance for an "entry" system. 
                                        

STRATUS

Stratus ET-1 should not be confused with debit credit.  Stratus, for obvious
marketing reasons, must appear to beat Tandem, and their numbers reflect
this war.  It is valid, however, to compare various Stratus machines using
ET-1 as a guideline.

Stratus does not use database journaling.  Any long power failure, system
software crash or other non-hardware fault will lose transactions and leave
the databases in an inconsistent state.  Stratus does not report journaling
numbers because their journaling is single threaded and peaks out at about
15-17 tps, killing the Stratus linear growth story.  A Model 120  gets about
15 tps (Style 3 with some non-qualified areas as noted below) with journaling
for transaction protection.  A model 140 doesn't do much better.

Like Tandem, Stratus ignores the presentation services part of debit credit.

Stratus also does injustice to the definition of the word random.  Although
physical I/O's are done, there is extensive caching including even the account
file.

The author estimates a Stratus midrange Model 120 (2 MC68020's) will do
less than  8.5 Style 1 debit credit tps.


Peter S. Kastner
Corporate Systems Group
Competitive Marketing Programs
PDM1-2(E1)
DTN 291-0364

March 4, 1988

Peter is incorrect in his assumption that engineering uses Style 1.  In
fact, the whole Style business is new.  Database Systems has, for the
last 2 years, used a matrix.  Experiments are run either BATCH (just the
database systems being exercised in batch) or END-TO-END (ACMS et al,
with the forms work assumed to not be on the same machine as the
database work).  The results can either be PEAK (maximum TPS achievable)
or QUALIFIED (95% 1 Second Response Time).

Our official goals are stated END-TO-END QUALIFIED.  Our internal
engineering effort is mostly driven off of BATCH.

So, basically, Digital assumes STYLE 2.  I don't know where this
holyier-than-thou attitude about STYLE 1 came from, since Digital
decided 3 years ago that STYLE 1 was uninteresting and actually
CONFLICTS with the definition of DebitCredit as it appeared in
Datamation.  The purest Datamation implementation would perform STYLE 2
or STYLE 3 with X.25 connecting the front-end/branch-controllers to the
backend.

Every time we switch the players, we switch the story.

    Hal,
    
    Do your comments in .1 still hold? I note that in RDB 5.4 you say
    that "Digital engineering typically uses Style 1 to project database
    and system performance."

My comments still hold.  All of our performance projections (vis a vi
DebitCredit) assume that the terminal work is offloaded.

    As an interesting aside in the Debit-Credit arena...
    
    It was reported in the November 7 edition of Computerworld (page 6) 
    that IBM has joined the Transaction Processing Performance Council:
    
    "...releasing an independent auditor's report of its Debit/Credit
    benchmark testing of the 4381 and Enterprise System/9370.  The IBM
    benchmark shows results that are three times greater than those claimed
    by Digital Equipment Corp. when it tested those IBM systems earlier
    this year.
    
    "...IBM's auditor, Tom Sawyer of the Codd and Date Consulting Group
    in San Jose, Calif., said the difference could have been related
    to the implementation of ONE-TENTH the terminals and shorter "think
    times" in the IBM configuration.
    
    "In calculating cost per transaction, IBM used annual maintenance
    charges based on...service discount plans.
    
    "The IBM system tested also used a 3725 communications controller,
    the cost of which was not included in the configuration."

    
    Hopefully, the vendors who join the TPC will be required to submit
    Debit/Credit benchmarks which follow the rules, once the rules have 
    been finalized!
    
    Otherwise, vendors will use membership in the TPC as a marketing 
    ploy, while continuing to use their own flavor of the Debit/Credit 
    benchmark to make their TPS and $/TPS look good.
    
    The net result will be a TPC organization which is as much a joke
    as the world of benchmarks they are trying to standardize.

    Rick C