Conference ulysse::rdb_vms_competition

    Inside cover of the 12/18/89 Digital Review issue.  Apparently the
    'no-Rdb' ad has ben replaced with one that says ..
    
    'ORACLE certified over twice as fast as Rdb'...
    
    And today I received a survey asking if we should be partnering with
    these folks by making them a CMP......unbelievable!
    
    (And yes, Virginia, I sent in my $.02)

    Interesting comparison....
    
    deferred writes once again...
    
    And I've heard that the Oracle portion of their latest test was run on
    a sequent.  Naturally the rdb portion was run on a vax...is this what
    Oracle calls a fair comparison?
    
    The meek (with no advertising budget) shall inherit the earth!

Actually its not just deferred writes (which are valid), its that they failed
to tune the Rdb database design.

    ORACLE says both of these benchmarks were run on a VAX-6360 and
    audited by Tom Sawyer of Codd & Date. 
    
    ORACLE V6.0 66 TPS   -   Rdb V?.? 30.6
                                          
    You can get a copy of the benchmark report by calling...
                                          
    1-800-Rdb-lies or 1-800-345-DBMS x1083

    I called and asked for the reports and a few other brochures
    (SQL*connect for DB2, SQL*connect for RMS).  When I got my package,
    guess what .... no reports !!  You tell me ....

    re: Hal,  talked to someone the other day that told me that the
    recovery checkpoint in the deferred write was actually set for longer
    than the performance sampling.  IF (which I have not verified) is the
    case deferred writes are not acceptable.
    
    -Jay
    

    The copy I briefly saw was at a customer.
    
    My package from O has not arrived yet. Maybe I will have better luck, 
    I used by buddy's consulting firm as the mailing address. I will
    keep you advised, or send you a copy if/when I get it.
    
    _Paul

There is nothing inherently wrong with deferred writes, which is quite different
than running a benchmark that defers all writes until after the measurement
period ends.  What I'm concerned about, and what I occasionally detect in this
conference, is a desire to say "defered writes are bad.  period.".  Deferred
writes are perfectly good (assuming proper design) are just a tradeoff
between cold-restart times and performance.  The fact that Oracle misused a
perfectly good feature to make benchmark results look screwy is the only problem.

    Agree on the feature and the misuse in their design.  Just trying to
    point it out.
    
    -Jay
    

    Please excuse the ignorance, but I am confused.  From the previous
    replies I understand that the 66 TPS is questionable.  My confusion
    lies in understanding what the knock-offs are.  I think they are:
    	_ 1.  Oracle did not tune the Rdb db
    		-- The debit/credit nos. I have show 33.4 TPS on a 6360 w/
    		   Rdb 3.0......so we are at 33.4 TPS......
    	_ 2.  Oracle did not write any updates to disk until after the
    		benchmark was complete.
    
    I have other info that indicates that Oracle tends to do the test using
    files which are 10% of full size, uses 0 terminals, and measures
    response time internally.  Are these applicable to the 66 TPS no.?  Are
    there any other knock-offs I am missing?
    
    Thanks for any help.  Rusty.

    Oracle uses large databases. But they do testing with no simulated
    terminals, no think time to simulate terminal overhead, no networking
    since no terminals are attached. The measurement period is generally
    3-5 minutes. This would suggest that during the measurement period all
    writes are "deferred". That is, all the writing is done to memory cache
    rather than disk.
    
    While none of this is unethical, neither is it representative of a
    real-world commercial implementation in even the most liberal sense.
    
    ---- Michael Booth

There are three issues here:

1) What benchmark is being run.  At the current time people always run a
benchmark whose database work is nominally the same, but they vary:
	- Database Size
	- Method of running (TP-style, batch, funny drivers, different think
	  times, etc.)
	- Method of measurement (1 vs 2 second response time, 90 vs 95%, etc.)
	- Recovery implications (ie, there is nothing to prevent you from
	  making the transactions go fast at the expense of very long recovery
	  times.
They then make quotes like "our system does xx TPS", and when you compare that 
with a competitors number you make inaccurate conclusions.  Although the
comparison isn't quite as different as Apples and Oranges, it is like
comparing Apples and Pears!

2) Having an "audited" benchmark is meaningless.  The auditors (particularly 
Sawyer) seem to take the approach that as long as Oracle did what they said
they did, then its ok.  In other words, the audit only indicates that the
benchmark was run, and the results obtained, just as claimed.  The auditor
makes no claims as to the validity of the benchmark itself.  So, if
you create a benchmark for Apples with the following criteria:  (1) Sweet,
(2) Yellow or Red, (3) Firm Flesh, etc.  and then claimed that a
Pear was an Apple according to your benchmark, the auditor would attest to
the accuracy of your statement!

3) When Digital run's a benchmark on another vendor's product, we do everything
possible to make the competitor look good.  By that I mean that we try to
apply the same level of expertise to running the benchmark on the other vendors
product that we put into our own benchmark.  So, for example, to obtain IBM
performance numbers we hire outside consultants with extensive IBM expertise
to perform the configuration design, tuning, etc.  It is apparent that Oracle
did not do this, instead choosing to compare an untuned Rdb/VMS against a tuned
Oracle.

A few questions and a comment.

Q:  Do we know *for sure* that Rdb was not tuned ??  If so, how ?  Please
	be factual eg NO "well, ... I heard ...".

Q:  Where did Oracle get their copy of Rdb ?

C:  Deferred writes are made possible by setting parameters in the INIT.ORA
    startup file.  While there are a number of these parameters having to do
    with global memory (SGA), there are probably three that are used to "turn
    on" deferred writes and to control dio performance for benchmarks. 

    DB_BLOCK_BUFFERS specifies the number of database blocks cached in memory in
    the SGA.  This parameter is the most significant determinant of the SGA size
    and database performance and probably enables Oracle to load in the entire
    database with one (large and long) dio.  Since you can have a maximum of
    65535 buffers (each = 4 vms disk blocks), the largest database size you
    could have would be approx. 134mb.  Setting this parameter gets the database
    in memory. 

    LOG_CHECKPOINT_INTERVAL specifies the number of newly filled redo log file
    blocks (vms blocks) needed to trigger a checkpoint.  (Remember redo logs are
    pieces of database pages that have been changed via a commit but not yet
    written to the physical database pages.) So if the redo log file is large
    enough to hold the entire benchmark, setting this parameter high (there is
    no upper limit) would reduce, if not eliminate, the number of dio's
    associated with the RUJ/AIJ activity. 

    PRE_PAGE_SGA causes all pages of the SGA to be paged into each user's
    working set.  Setting this parameter to true would cause all SGA memory
    pages to be duplicated in each batch process running as a part of the TP1
    suite.  This would provide high throughput for read-only tables since each
    process would have copies of the data.  I'm not sure how updates to table
    would be handled. 


-- gerry

    Do we know it is true...
    
    Well, for starters, the entire database definition for both Rdb and
    Oracle is supplied in the benchmark. We know, for instance, that no
    Hashing or DBKeys were used in the Rdb database. I find that very
    suspicious in an exect match benchmark like TP1. So, yes, on that point
    alone, we KNOW that Rdb was not tuned. We also know that Rdb was using
    snapshots. Is that something one does in benchmarking? NO.
    
    So, yes, we know for sure that Rdb was not tuned.
    
    ---- Michael Booth

    It doesn't matter how Oracle ran their benchmark; customers only get to
    know the comparison -- the 'Oracle runs twice as fast as Rdb' myth is
    indelibly etched in their minds. I've had a couple of customers say
    that they have heard Rdb has 'poor' performance. When I guess that
    Oracle told them that they just look down at the floor, shuffle their
    feet & mumble & look embarressed.
    
    But the FUD is there & its a job removing it.

At least that should help.  I understand why we can't/won't publish alternate
numbers.

    I just received my package of benchmark reports from ORACLE. There
    was a copy of the ORACLE report with Tom Sawyers report attached.
    But guess what? No mention of Rdb at all. It looks to me that Tom
    Sawyer may have only audited the ORACLE numbers, and not Rdb. Only
    ORACLE knows for sure.
    
    _Paul

    I have access to the complete Tom Sawyer report, with the Rdb numbers.
    
    In summary he reports the following:
    
    ORACLE Version 6 with the transaction processing option and VAX Rdb 3.0
    executed the TP1 benchmark on the Digital VAX 6360.  The measured
    configuration and the results were:
    
    System   Memory  Disks    % Under 1     tps        Full
              (MB)   Measured   second             Development
                                                      K$/tps
    ORACLE     192     12         99.0      66.0       28.47
    Rdb        192     12         97.5      30.6       42.68
    
    If you have detailed questions about this report, please let me know
    and I will try to get answers to you.
    
    The report is a document published and copyrighted by Oracle, although
    it refers to Tom Sawyer constantly.  There is no attestation letter
    with his signature.
    
    Michael