Conference ulysse::rdb_vms_competition

    The official reply is that we don't mingle with people that does TP1
    benchmarks (sorry do not comment on benchmarks done uisng TP1).
    
    TP1 is not well described and it open for almost any cheat you might
    think off.
    Only TPC-A (online benchmark) and
    TPC-B (Batch) numbers can be compared and discussed.
    
    Another thing is that Oracle as usual has cheated like h...
    1) Rdb is running with SNAPS enabled.
    2) Rdb is running with AIJ enabled, Oracle is not
    3) Oracle has set it's REDO flush interval so long that it didn't
    needed to write anything to the database while the test ran (they
    deferred comming the real stuff to teh database until the end of the
    benchmark), handy aehh, for benchmarks that is, but real applications
    with get sick if this is done, just think of the time to ROLLBACK a
    transaction...
    
    AND alot of other things can be notes aswell.
    
    But let's stick to the fact that they haven'ty usded an official
    benchmark. The other inforamtion was just to calm you, Rdb is FASTER
    than O in the real world.
    
    Lars
    
    Ps. In Rdb V2.3 I tried a little benchmark cheat.
        I loaded a database into memory (LOTS OF BUFFERS and a small
    database, and ran a test on a 3600 MVax. I got 36 TPS out of it, quite
    amasing what you can do if you avoid using the disks for anything.

From:	NAME: HANS GYLLSTROM @ALF           
	FUNC: DATABASE SYSTEMS GROUP          
	TEL: 343-0272                         <GYLLSTROM.HANS AT OA at CSCOAC at ALF>
To:	See Below


              ORACLE'S MISLEADING ADVERTISING CLAIM vs. Rdb


SUBJ:   Oracle's Advertising Claims re: Rdb Performance

In recent ads, Oracle Corp. is claiming that their Oracle relational database 
achieved over two times the performance of Rdb on a VAX 6360.  They claim:

        Oracle - 66 TPS (Transactions per second)
        Rdb    - 30.6 TPS

The ad cites the fact that the performance test comparing the two products, 
run by Oracle Corp., was audited by Tom Sawyer of Codd and Date Consulting
Group, noted database experts.  The ad also states:

   "Rdb is slow on VAXs."

   "Rdb does less work on two computers than Oracle does on one."

   "The choice is yours.  Stay with Rdb and look into the price of a second 
    VAX.  Or call Oracle and save more than money."

Careful scrutiny of the detailed performance report, released in September 
reveals the following:

1 - It is not an "apples-to-apples" comparison.  Oracle performance features 
such as checkpointing (not present in Rdb) are used.  Rdb performance
features such as hash keys (not present in Oracle) are NOT used.  In fact, 
Rdb is not tuned at all.  The application is designed to use more I/Os to
write the data than Rdb would actually require.  A TRUE comparison should
feature an efficiently tuned Oracle implementation vs. an efficiently tuned 
Rdb implementation.

2 - Oracle ran the tests on HSC disk controllers, but substituted less 
expensive KDBs when calculating price/performance.  In fact, the 
configuration they priced is not a valid orderable configuration.  This 
calls into question the overall approach.  At Digital, all configurations 
are run through the XCON configurator to verify their validity, before 
releasing price/performance results.

3 - To perform the benchmark, Oracle required an additional 64MB of memory 
($85,600).  This was also added to the Rdb configuration, even though Rdb
is less memory-intensive and could run the test without the additional 
memory.

4 - The TP1 benchmark used is not approved by the Transaction Processing 
Performance Council (TPC), does not have detailed specifications, and is 
therefore, subject to various interpretations on how to run the test.   
There is a benchmark which has been approved by the TPC and which is 
carefully specified, so that the results can be used in comparing 
different vendors' system performance.  It has been known for several years 
as DebitCredit.  The final approved version is called TPC-A.  It is Digital's 
position that only approved, specified benchmarks should be used for 
reporting system performance.  To do otherwise, just confuses the issue.

5 - Furthermore,  the DebitCredit (now TPC-A) benchmark is far more 
rigorous than TP1 used by Oracle.  It measures the performance of an entire
system, including communication, transaction submission, and involves many 
more system requests than TP1.  Digital is releasing DebitCredit performance 
results for Rdb V3.1 on a series of VAX 6400 processors.  Oracle cannot and 
will not be able to match those results.  Our challenge to Oracle is to run 
that approved industry standard benchmark as we have done.

6 - Oracle's license agreement prohibits the release of any Oracle benchmark 
results.  Therefore, no one can dispute Oracle's claims.  Oracle has run 
similar performance comparisons against the Microsoft OS/2 server.  Like
Digital, Microsoft is prohibited from releasing comparative test results.

    I hope that someone from DEC has explained to Sawyer, and Codd and
    Date, that "auditing" comparisons like this without the participation
    of all vendors being compared damages their good name...

    sorry- but as auditors, they are certifying to what they see. They are
    not claiming TP1 = TPCB. No damage is sticking to them.

    Tom is paid to observe and report. He does a very good job. It is our
    job to explain the difference in benchmarking from benchmarketing to
    the buyers.

    sorry,

    Barry

    I just wish the auditor's attestation read more like
    
    "We have observed that it is possible to arrange Rdb/VMS so that
    it runs even slower than Oracle..."

    [TP1 is] worthless- no one should ever suggest that TP1 results [have]
    value greater than the paper on which they are written," charged Omri
    Serlin....."every vendor fixes the test results to show [results] their
    own way, and I would strongly recommend users to disregard them."

    
    Computerworld 1-/1/90

I don't recall seeing Oracle's TPC-A benchmark report.  Was one published? 
If so, what were their numbers? 

    As far as I know, Oracle has only published TPC-B numbers until now.
    And even those numbers are still not accepted by the TPC as they did
    not pass all the test they need to (like ACID, clause 2).
    
    Henny Olthof, TP-DB Netherlands