Conference ulysse::rdb_vms_competition

1057.0. "TPC-B and RDBMSs" by DATABS::NEEDLEMAN (today nas/is, tomorrow...) Thu Jan 09 1992 17:21

    Oracle has been using a TPC-B benchmark run by Database Associates to
    discredit Rdb. This note will be reserved for factual ammunition to
    respond. Please do not advise or argue within this entry. I am working
    on other sections but here are some facts that several of us have
    agreed upon. If you have other facts you think should be added, please
    mail them to me and I will see if they can be included in part II.
    
    
    
    Barry
    
The position of the Database Systems Group on the TPC-B Benchmark


Authored by: Barry Needleman, Mike Brey, and Evan Bauer 

One of our competitors has directed a misinformation campaign against
Rdb using TPC-B. Here is some factual data to use. 


         In the words of the Transaction Processing Council,

              "...there is no single TP benchmark that can
              adequately measure performance in all TP
              environments.  . . . TPC benchmarks should not be
              used as a substitute for a specific customer
              application benchmark when critical capacity
              planning and/or product evaluation decisions are
              contemplated."

         The TPC describes the difference between TPC-A and TPC-B 
	 in the following words:
         
         The TPC-A is a benchmark that 

              "measures performance in update-intensive database
              environments typical in on-line transaction
              processing (OLTP) applications. Such environments
              are characterized by:

              -Multiple on-line terminal sessions 
              -Significant disk input/output 
              -Moderate system and application execution time 
              -Transaction integrity "

         On the other hand, the TPC-B wording is 

              "In contrast to the TPC-A, TPC-B is not an OLTP
              benchmark. Rather, TPC-B can be looked at as a
              database stress test characterized by :

              -Significant disk input/output 
              -Moderate system and application execution time 
              -Transaction integrity"


Digital is in the production systems business and a leading system
supplier. As such, we in Database systems participate in running a
standard price/performance benchmark that measures our system. We
believe that the TPC Benchmark A holds the greater validity until
Benchmark C is available. It tests the efficiency of interfaces
between terminal users, network, message passing, TP monitor,
database, and operating system -- an important measurement point even
if the database workload is low and unrealistic. 

What exactly is wrong with TPC-B?

The TPC-B test does not in any way look at the features of a DBMS
system that provide benefits to its users. No relational function is
stressed. No real-world scenario is simulated. What bank would
implement a 10,000 cage teller system without a TP monitor or
transaction router?. Database performance should be measured in a
complex workload against a realistic database design. Database
performance testing should include multi-table joins, complex WHERE
clauses, incomplete transactions that need to be rolled back, and
complex updates.


The TPC-B does not stress the components of a database management
system that are critical to real application performance: the
optimizer, data space recovery and extension, triggers, referential
integrity and more complex constraints, uneven key distributions,
rollback of failed transactions, and the ability to compress data. 
Probably the most critical untested feature is the optimizer -- if you
didn't want to handle complex boolean conditions, partial key
searches, and multi-row operations, you wouldn't use a relational
database in the first place. 
 
There are no heavy I/O requirements (we average less than 3 I/Os per 
transaction). This is very small and unrealistic for a business
transaction. for example, in some actual banking applications, the
most update intensive do 10 reads per write -- complex integrity
checking, auditing and cash monitoring activities require it.

The data placement is artificial and is done with high performance in
mind and not user consideration. Submitted TPC-B benchmark designs use
record clustering to reduce I/O. Much of this clustering would be
unrealistic in an environment with multiple transaction types. Only
the deliberate narrow focus of this benchmark affords this placement.
For example, under normal circumstances, end of day processing (batch
jobs summarizing positions across accounts) are one of the major
performance choke points for banking systems. TPC-B physical designs
de-optimize for this workload -- the benchmark does not check
performance on this "de-optimized" query.

The TPC-B starts with a tiny database, with no deletion or archiving 
of data. This is a unrealistic condition in most production databases
as they hardly ever run in this mode all the time. Only one table
grows and importantly, there is only growth and no shrinkage.
Performance with record deletes and use of reclaimed space is not
measured. Having no space management capability is actually an asset
when running TPC-B.  Rdb reclaims deleted space while the database is
open for update, unlike our ISV database competitors.

The transactions are too short and do not represent reality. 

Real world DB's have to balance update and read transactions and
optimize between the two of them, TPC-B only has update transactions
and these have limited interaction with each other, no read only type
transactions. This does not match real customer transaction
environments.
	
No human intervention/interaction. The transaction requires no forms,
no human think time. This goes against the norm of practical
applications and again reinforces the artificiality of this test.

More critically, the small number of transaction streams used to
optimize TPC-B performance do not take into account a mechanism for
concentrating thousands of terminals into a dozen (or less) streams. 
None of the third party database products on VMS or Unix have a
mechanism for doing the transaction routing.

The benchmark does not enforce a specific degree of consistency and
therefore anything is acceptable (no degree 3 consistency requirement
- dirty reads are permissible). This is an unlikely environment for
high volume productions applications. The consistency test is weak as
it only runs single user transactions and check the balances at the
end. Even the ACID test in a loosely coupled environment does not
require stopping a single processor, Oracle shuts all CPUs of together
(VAXcluster, NCube we believe). We have no idea how they would do if
only a single cpu failed. In addition, no banking system would be
passed by the auditors if degree 2 integrity was all it provided --
the potential for fraud or error is too great. Rdb enforces degree-3
consistency throughout the transaction at the cost of additional
overhead. Our competitors tune to the benchmark and allow for dirty
reads.


No relational DBMS evaluation should be complete if it does not
include validation of the majority of the relational rules as espoused
by Dr. Codd. We looked at TPC-B to see which rules are evaluated. Only
two out of the twelve rules were tested and an index file system
could meet one of these.


	Codd's Rules				Evaluated in the TPC-B ?
        -----------				------------------------

Rule 1, "the information rule"				No 

Rule 2, "guaranteed access rule"    			Yes 

Rule 3, "systematic treatment of null values"		No

Rule 4, "dynamic on-line catalog based on the 		No
         relational model" 

Rule 5, "comprehensive data sublanguage rule"   	No

Rule 6, "view updating rule"   				No

Rule 7, "high level insert, update, and delete"		Yes

Rule 8, "physical data independence"   			No

Rule 9, "logical data independence"   			No

Rule 10, "integrity independence"			No

Rule 11, "distribution independence"   			No

Rule 12, "nonsubversion rule"   			No




In summary, the TPC Benchmark B is simply a relative scale and one
point of comparison. Any other use of it should be considered
misleading. Doing well on TPC-B, in no way implies success on a
customers application or in providing a production quality database.