[Search for users]
[Overall Top Noters]
[List of all Conferences]
[Download this site]
| Title: | DEC Rdb against the World |
|
| Moderator: | HERON::GODFRIND |
|
| Created: | Fri Jun 12 1987 |
| Last Modified: | Thu Feb 23 1995 |
| Last Successful Update: | Fri Jun 06 1997 |
| Number of topics: | 1348 |
| Total number of notes: | 5438 |
220.0. "Review of Oracle's 6.0 Benchmark" by BROKE::DREYFUS () Tue Oct 18 1988 16:38
David Dreyfus
Database Systems Group
DTN 381-2893
DEBIT::DREYFUS
18-October-1988
Review of Oracle's Benchmark Report
NOTE
The enclosed information is based upon Oracle's
"Oracle Performance Report" and "ORACLE TP1
Benchmark Methodology" reports dated July, 1988.
On July 18, 1988, Oracle announced version 6.0 of the Oracle
relational DBMS and TPS (Transaction Processing Subsystem).
According to Oracle, the new and upgraded products set speed
records on multiple platforms.
Oracle's benchmarking techniques allowed them to show 49 trans-
actions per second (TPS) on a VAX 6240 with a 0.1 million record
database. Oracle also showed 265 TPS on an IBM MVS 3090 Model
600E with a 0.1 million record database. These tests used a
heavily modified version of the TP1 benchmark. The results bear
no relation to any number Digital, or any other vendor, has
produced.
The press and consultants have had a field-day discrediting
Oracle's benchmark results. Oracle has had a very difficult time
finding auditors who would verify the results (Computerworld,
July 18, 1988). In fact, they searched for multiple auditors in
an attempt to get their tests verified. One member of the press,
referring to the material, says, "Yes, ladies and gentlemen,
it was a thick document, but we should read it as well as weigh
it."
Well, the report has been read and here are the findings.
Digital Internal Use Only 1
1 Confusing the Market
One of Oracle's first goals in releasing their benchmark results
seems to be to confuse the market. Lawrence Ellison, President
of Oracle, claims that their TP1 is a "Debit/Credit benchmark,
absolutely the same as Tandem's." (Computerworld, August 1,
1988). However, unlike Tandem, the article says, Oracle did no
front-end communication, no mirrored journaling, no simulated
terminal networking (transactions initiated from main memory),
etc. According to consultant Omri Serlin, "Oracle has gone to
such an extreme in what they have done. It had no on-line flavor
whatsoever."
Although Oracle states that TP1 and Debit/Credit are the same,
they are not (See Appendix A and the July 12, 1988, and August
22, 1988, editions of Competitive Update). The only distinction
that Oracle draws between tests is between their version of TP1
and Sybase's version (which Oracle calls the banking benchmark).
The difference between these two is database size.
Oracle tries to dismiss all Debit/Credit requirements for online
users, think time, communications, and user-interface management
as 'hardware-oriented' factors. Therefore, they removed them
from their benchmark. This has the very predictable effect of
inflating Oracle's benchmark results.
2 The Test Results
Oracle's benchmark results are questionable and must be highly
qualified. Oracle's report acknowledges that a benchmark run
has a ramp-up time (during which data is loaded into memory),
a peak performance time (a high transaction rate prior to in-
ternal buffers filling up), and a sustainable transaction rate.
The reported numbers are under the caption "Peak Performance."
The first question is: Are the reported numbers sustainable?
The report and auditor fail to answer this question. Subsequent
investigation has found that Oracle ran the benchmark for five
2 Digital Internal Use Only
minutes. This mitigated the need for any significant I/O be-
cause database changes were held in memory until after the test
completed. A longer test run would have forced much greater I/O.
If Oracle is given the benefit of the doubt (that the results
are sustainable), the next issue is the TPS rate itself. Oracle
reports a TPS rate of 42.6 on a VAX 6240 with 99.9% of all
transaction completing in under a second. Average response time
is 0.4 seconds.
This rate was achieved by not complying to the Debit/Credit
benchmark standard. Oracle uses just enough batch jobs to reach
their maximum TPS rate. That is, no more batch jobs are added
after the CPU, memory, or disks become a bottleneck. In this
scenario, minimal queue delays occur. One would expect all
transactions in such a scenario to complete in less than one
second. After all, Oracle is doing 42.6 TPS. In addition, re-
sponse time is measured from when the transaction is submitted
to the database, not from when the user (which they don't simu-
late) initiated a request.
Systems using real, online, Debit/Credit benchmarks have a more
difficult time meeting the subsecond response time rules because
they are simulating thousands of users with communication and
networking overhead. The system has bottlenecks. In order to
meet the response time rule, these systems demonstrate lower
TPS. Thus, Oracle's batch mode TP1 test is not comparable to the
standard, online, Debit/Credit benchmark Digital uses.
On a VAX 8820, Oracle demonstrated 32.8 TPS using this same
batch methodology. Digital has done some testing on a VAX 8820
with VAX Rdb V3.0 in batch mode and has achieved 36 TPS. Digi-
tal's results have not been released because the testing has not
been rigorously audited (by internal auditors).
It appears that VAX Rdb V3.0 is conservatively 10% faster in
batch tests. This is conservative because of the questions sur-
rounding the reporting of peak or sustainable TPS rates for
Digital Internal Use Only 3
Oracle's tests. VAX Rdb V3.0 may have an even greater perfor-
mance advantage in online, Debit/Credit benchmarks because of
VAX Rdb's closer connection to DECnet, VMS, and TP monitors.
3 Price Performance
Oracle tries to use its batch TP1 tests to show greater price
performance than Digital. However, Oracle's price/performance
quotes are very confusing. Sometimes maintenance is included,
sometimes disks, and sometimes both. Although Oracle says that
they have included software costs in their cost-of-ownership
model, they have, in fact, only included the cost of VMS. They
have not included the cost of the database management software -
their software.
The five year cost of hardware and VMS is $1,161,543 (accord-
ing to Oracle) on a VAX 6240 configured to run Oracle. Using
Oracle's list price for software, the five year cost of a C pre-
compiler, the DBMS kernel with TPS, and DECnet with SQL*Net is
$459,375. The equivalent cost for VAX Rdb (which includes per-
formance, precompilers, and DECnet support) is $68,178. Notice
that the software cost of Oracle is 40% of the cost of the hard-
ware. The price difference between Oracle and Rdb could be used
to add additional processing power (hardware).
In order to compare $/TPS for Oracle and VAX Rdb, Oracle's
numbers have to be revised to produce Debit/Credit benchmark
equivalents. Because Oracle doesn't offer a TP monitor, it is
very difficult to estimate Oracle's Debit/Credit performance.
In order to maintain conservative estimates, it is assumed that
Oracle's performance doesn't drop below its 10% disadvantage to
VAX Rdb.
VAX Rdb achieved 18.4 TPS on a VAX 6240. Estimated Oracle per-
formance is therefore 16.7 TPS (10% less). This puts Oracle's
$/TPS at $97,060. Note that this does not include the cost of
a TP monitor since Oracle doesn't have one. The comparable VAX
Rdb cost of ownership is $56,000/TPS. This figure does include
4 Digital Internal Use Only
the cost of a TP monitor (VAX ACMS) and a data dictionary (VAX
CDD/Plus). Note that a Digital solution is less than 60% of the
cost of an Oracle solution on the same hardware.
The difference in cost of ownership has two factors. First, as
noted, there is a large cost of software difference. The second
reason is that Oracle requires nineteen (19) RA82 disk drives to
implement their test and record the required 90 days of history
data. Digital achieves its performance from twelve (12) disk
drives.
4 Technology Leadership
Oracle states that their performance is due to their technology
leadership. Oracle is not a technology leader. Oracle states
that they are the first relational DBMS with unrestricted row-
level locking (introduced with V6.0 in 1988). VAX Rdb has had
unrestricted row-level locking since V1.0 in 1984. Oracle states
that they are the only high performance DBMS with multi-version
read consistency (introduced with V6.0 in 1988). VAX Rdb has
had a much more flexible and potentially higher performance
implementation of this functionality through snapshot files
introduced in V1.0 in 1984.
4.1 Fast Commit and deferred writes
Oracle states that they use a technology called 'fast commit'
in order to reduce I/O and improve throughput. Fast commit
allows the recovery log to be written while the data pages
stay in memory when a transaction is committed. The expectation
is that multiple transactions can modify the same data pages
before they are actually written to disk. Writes to the database
are deferred until a checkpoint or until internal buffers are
filled.
Digital Internal Use Only 5
During a checkpoint, all modified data pages are written to
disk and the recovery journal is reset. No mention of delays
caused by checkpointing are made in Oracle's benchmark report.
This is because they don't run their tests long enough to reach
the point at which a checkpoint must occur. That is, they run
benchmarks by keeping all database changes in memory until the
test is complete.
There are a number of pitfalls with this technology. Recovery
time increases because the recovery log must contain a lot more
data (all the unwritten, but committed, data pages) and be-
cause this extra data must be re-written to the appropriate data
pages. Another problem with this technology is its incompati-
bility with VAXclusters. Fast commit relies upon shared memory.
Since VAXclusters do not share memory, this feature doesn't
help, or may be incompatible with, VAXclusters. In fact, the use
of VAXclusters seriously degrades Oracle's performance.
Oracle may try to make a big case for their fast commit technol-
ogy. However, it is not clear that it really helps.
4.2 Piggy-backed writes
Oracle may also make a case for a technology called piggy-
backed writes. When multiple transactions require I/O to the
same recovery unit pages, the database software attempts to
satisfy all the I/O requests at the same time. This economizes
on I/O and improves performance.
VAX Rdb has had this capability, called group commit, since
version 2.2 in 1986. Group commit applies to root and after-
image journal file pages.
6 Digital Internal Use Only
5 Auditor Verification
A very simple letter of verification is supplied by Tom Sawyer
of the Codd and Date consulting group. It is not nearly as ex-
tensive as the one provided for Tandem's TopGun benchmark. For
Oracle, Sawyer only attested to the attributes of the benchmark
that were followed. He did not discuss any of the numerous de-
viations from the Debit/Credit benchmark or the implications of
the deviations. Sawyer verified that the transaction was imple-
mented correctly, that the database was sized correctly, that
the transactions were randomly distributed, that updates caused
the correct locking, that transaction consistency with rollback
was implemented, that response time was measured correctly, and
that the database records were correctly sized.
Sawyer did not verify that the following Debit/Credit benchmark
standards were followed: that 15% of all transactions used
accounts from remote branches, that the database was journaled
for rollforward recovery capability, that users were simulated,
that network overhead was simulated, and that the TPS rates were
steady-state. Each of these factors has a significant impact on
performance: it is unfortunate that Tom Sawyer was not able to
comment on them.
6 Batch TP1
Oracle has gone to the extreme in TP1 performance testing by
using a batch-mode benchmark. The Debit/Credit benchmark is
used to measure online transaction processing performance.
The differences between batch testing and online testing are
enormous, as one might expect.
The Debit/Credit standard calls for 100 users per demonstrated
TPS. 42.6 TPS would require a simulation of 4,260 users. This
many users requires memory, communication, and TP monitor re-
sources that significantly impact performance. Instead of
simulating users, Oracle uses an undisclosed number of batch
Digital Internal Use Only 7
programs linked with their database kernel subroutines to con-
tinuously generate transactions. Batch mode processing allows
Oracle to show high performance without paying the overhead of
online users.
One impact of a few batch-mode transaction generators is that
there are never more than a few transaction requests outstand-
ing. This tends to keep response times low. When Oracle in-
creases the number of transaction generators (more batch jobs),
throughput declines and response times increase. If Oracle tried
to maintain subsecond response time while increasing the number
of transaction generators, performance would drop off rapidly.
In addition, by placing both the transaction generators and the
database software on the same machine, Oracle avoids network
overheads and delays. The cost of generating the transactions on
the VAX managing the database is marginal because the transac-
tion generation routines are linked into the Oracle DBMS kernel.
Contrary to Oracle's statements, using the network would proba-
bly reduce their performance. The network requires both CPU and
memory resources that would otherwise be used by the database
software.
The Debit/Credit benchmark standard states that applications
should read 200 bytes for form display and write 100 bytes from
the form to the TP application. It appears that Oracle only
sends 40 bytes from their application generator to the database.
This difference also reduces memory and CPU overhead by reducing
message packaging and transmission costs. The importance of this
deviation would increase if Oracle used the network.
8 Digital Internal Use Only
7 Summary
On July 18, 1988, Oracle announced a number of new features
that improve Oracle's DBMS performance. The performance of their
product has, in fact, improved. However, the benchmark results
that Oracle produced used a batch mode version of TP1. The
results are not comparable to Digital's Debit/Credit benchmark
results.
Any direct comparison between the two tests is meaningless.
After Oracle's numbers are adjusted to reflect the differences
between benchmarks, VAX Rdb is the better performer with a
substantially lower cost of ownership. The cost factor can be
used to sell additional hardware, software, and services.
The new features that Oracle introduced have generally either
been in Rdb for years or have been added to Rdb in version 3.0.
In addition, Rdb generally has a more complete and flexible
implementation that should provide users with higher performance
across a wide variety of system configurations than Oracle.
Digital Internal Use Only 9
APPENDIX A
COMPARING DEBIT/CREDIT AND TP1
Originally, Debit/Credit and TP1 were synonymous. Since the
original definition of Debit/Credit, TP1 has taken on different
meanings.
Unlike TP1, the requirements for Debit/Credit are well defined
as shown in the April 1, 1985, Datamation article (v31 p112(5)).
Debit/Credit is becoming an industry standard benchmark for OLTP
(On-Line Transaction Processing) applications. Details of the
benchmark are explained in great detail in the July, 1988, issue
of Competitive Update. Briefly, for each transaction per second
(TPS) the system must simulate:
The Debit/Credit Standard
o 100 Tellers (users) executing one transaction each 100 sec-
onds.
o A database with:
o 100,000 Account records
o 100 Teller records
o 10 Branch records
o 2,590,000 History records
o Undo/Redo recovery (Rollback/Rollforward)
o Form presentation services
o 15% of the transactions go against an account that is not in
the teller's branch.
Comparing Debit/Credit and TP1 11
o 95% of all transactions must complete in under one second.
o Duplexed recovery log
Thus, in order to measure 20 TPS the database must have 2 mil-
lion account records (20*100,000), 2 thousand teller records,
2 thousand users (tellers), 2 hundred branches (20*10), etc.
The size of the system scales with the desired throughput. This
tends to make the Debit/Credit benchmark more realistic than
those that show high throughput on minimal data with few users
(i.e. TP1).
Debit/Credit measures two important variables - performance and
cost. Performance has already been described as throughput (TPS)
and response time (95% of all transactions complete in 1 second
or less). Cost is determined by adding the five year costs of
hardware, software, and maintenance and dividing by TPS. For
example, a million dollar configuration producing 20 TPS costs
$50K/TPS.
Changing any of these parameters can have a profound impact
on performance. We know through our own testing efforts that
performance is easily doubled by reducing the database size,
number of users, and response time requirements.
The TP1 Benchmark
Vendors who run TP1 will often make substantial changes to the
Debit/Credit benchmark. The most common changes are:
o Reducing the number of Tellers (users).
For example, using only 100 Tellers for 30 TPS.
o Reducing the database size and not scaling size with through-
put.
For example, using a 100,000 Account record database for 30
TPS.
12 Comparing Debit/Credit and TP1
o Not using an After Image journal for recovery.
o Avoiding the cost of form presentation services.
o Measuring average response time instead of 95% 1 second
response time.
o Not measuring network overhead.
TP1 does a poor job characterizing database performance be-
cause it uses unrealistic database sizes, numbers of users, and
performance monitoring. TP1 does an even worse job in character-
izing total system performance. Database applications generally
have multiple users who require presentation services and net-
work services (DECnet). Since each vendor's product uses these
services differently, they must be included in the benchmark.
Debit/Credit tests these aspects of system performance; TP1 does
not.
Comparing Debit/Credit and TP1 13
| T.R | Title | User | Personal
Name | Date | Lines |
|---|
| 220.1 | Comparable TPS: Rdb/Oracle ? | CUJO::RUOFF | | Tue Oct 18 1988 20:38 | 16 |
| Nice summary, David. But what is the bottom line comparison between
a comparable tp test ? Can I assume that the 10% Rdb advantage
can be applied against Oracle's TP1 results on all 62xx platform?
If this be true, then is the following correct ?
Rdb tps Oracle tps
VAX 6210 5.7 5.13
VAX 6220 11.0 9.9
VAX 6230 ? ? *.9
VAX 6240 18.4 16.56
(by the way, what is the tps for the 6230 ?)
|
| 220.2 | Estimating Oracle TPS | BROKE::DREYFUS | | Wed Oct 19 1988 19:42 | 48 |
| >> But what is the bottom line comparison between
>> a comparable tp test ? Can I assume that the 10% Rdb advantage
>> can be applied against Oracle's TP1 results on all 62xx platform?
The bottom line is that Oracle has not demonstrated a performance
advantage. We have internally shown that in similar tests on a 8820
we are 10% faster. On legitimate Debit/Credit tests we should be
faster since Oracle doesn't have a TP monitor like ACMS. The
forms processing alone would kill them.
Because of the complexities of benchmarking and making projections
(different hardware, software, testing methods, etc), I would not
take the 10% advantage that we enjoy as a rule. In many cases it could be
more, in some cases it could be less. Also note that we were not optimized
for performance in our tests. Rather, we were optimized for price/performance.
We can go faster on a 6240 than the reports have indicated (about 10%).
Oracle's tests using the TP1 benchmark on a 62xx showed good
scaling from 11 TPS for a 6210 to 42 for a 6240. They showed
about 32 on a 6230 and 22 for a 6220. It is not clear that this
scaling will be enjoyed if they run a real debit/credit test.
There are a few main points that I tried to get across.
- Oracle did not follow the rules of Debit/Credit benchmarking
- There numbers are not comparable to ours
They are comparing their batch tests to our online tests.
- existing information shows we are faster
- they did not perform an online TP test
- they don't have the software to do TP
- We should always run TP applications and benchmarks with VAX ACMS.
Additionally, if INIT.ORA (oracle startup file) is told that another node
in a cluster may want to access the database, performance drops significantly.
If the other cluster node actually attaches to the database, performance
drops off even more.
If we only test released products, we have to test against Oracle 5.1 which
only does table level locking on update and which we should win easily.
Look at the Oracle User Group reports (previous note topics)
for other hints on competing with Oracle.
Hope this helps. Give me a call if you have more questions.
--david
dtn 381-2893
|