[Search for users]
[Overall Top Noters]
[List of all Conferences]
[Download this site]
| Title: | DEC Rdb against the World |
|
| Moderator: | HERON::GODFRIND |
|
| Created: | Fri Jun 12 1987 |
| Last Modified: | Thu Feb 23 1995 |
| Last Successful Update: | Fri Jun 06 1997 |
| Number of topics: | 1348 |
| Total number of notes: | 5438 |
176.0. "Review of Audited Sybase TP1 Report" by DEBIT::DREYFUS () Thu Aug 18 1988 20:39
*************** THIS REPORT IS INTERNAL USE ONLY ***************
Enclosed is a Review of the Sybase TP1 Benchmark as audited by AIM.
I went through the report and:
noted the deficiencies from the Debit/Credit standard
noted product deficiencies against Rdb
made some general comments on how their report impacts us.
Please contact me with any questions or comments on the report.
I would like to know what you like, don't like, and want to see more of.
Regards,
David
DIGITAL INTEROFFICE MEMORANDUM
TO: DBS Competition Inter- DATE: August 18, 1988
est List FROM: David Dreyfus
DEPT: Database Systems
Group
EXT: 381-2893
ENET: DEBIT::DREYFUS
SUBJECT: Review of the Audited SYBASE TP1 BENCHMARK
This report is based upon my analysis of the audited SYBASE
TP1 benchmark, dated 12-MAY-1988. It is intended for in-
ternal use only.
The Sybase benchmark results were audited by AIM Technol-
ogy, Palo Alto, CA 94303, on 14-MAR-1988. AIM provided a
one page cover letter to a Sybase report dated 12-MAY-1988.
The cover letter states that Sybase conformed to the TP1
benchmark standard.
Promoting TP1
SYBASE considers TP1 a de facto standard. They state that
TP1 and Debit/Credit are used interchangeably (by them) and
tend to blur the distinction between the two in the remain-
der of the report.
To my knowledge, there is no TP1 standard. The only stan-
dard transaction processing (TP) benchmark is Debit/Credit.
TP1 is a derivative of that benchmark. A comparison of the
two is provided in Appendix A.
Using Release 3.0, Sybase achieved 29 TPS with an average
response time of .68 on a VAX 8700 simulating a workload
of 600 users with a TP1 benchmark.
Page 2
The report talks about the need to fully specify how the
tests were run - including type of logging, buffer sizes,
and system parameters. However, this information is not in-
cluded in their report. Some, but not all, of the missing
information is included in the separately requested appendixes.
Although SYBASE talks about the importance of a standard
TP1, they fail to follow the Debit/Credit standard in a num-
ber of ways: Their database is not big enough for the through-
put they demonstrate, they don't have enough users on the
system for the throughput they demonstrate, they report av-
erage response times instead of 95% 1 second response times,
they use suspect methods of file partitioning, they don't
follow the transaction dispersion rules, and others.
The most obvious deviation from the standard is their sim-
ulation of 600 users on a 100,000 account record database.
Sybase should be using 2900 users and 2.9 million account
records in order to comply with the Debit/Credit standard
(based upon their demonstrated transaction rate).
Transaction Generation
Sybase's simulation of 600 users is suspect. What they re-
ally do is have 20 processes on a remote machine (VAX 8800)
continuously submit transactions to the system under test.
This is very similar to our batch tests where 15-20 batch
jobs continuously submit transactions. It supports our con-
tention that Sybase's TP1 is very much like our batch tests
and is not a measure of OLTP capability.
Sybase says that they simulate 600 users by assuming that
20 processes going at one transaction per second is the same
as 600 users which wait 30 seconds between transactions.
It is my belief that this logic is faulty. There are a con-
siderable number of resources consumed in managing hundreds
or thousands of users. Specifically, there are large mem-
ory and communication costs in addition to system overhead
Page 3
that must be paid by each user. Having only 20 processes
generating the transactions avoids this problem.
Sybase acknowledges that the standard calls for 100 sec-
ond delays between transactions. We can only assume that
there is a big performance hit as you start to have many
users with long delays. Otherwise, why would they pick '600
users with a 30 second delay.'
When we run our tests on Rdb/VMS we support the correct num-
ber of users. Digital's DECtp numbers reflect the overhead
of thousands of users. In order to handle these users, our
tests use TP monitors (like VAX ACMS). The overhead of the
monitor is reflected in our Debit/Credit results and is gen-
erally required for TP applications.
Sybase does not demonstrate the capability to support large
numbers of users. They do not provide or use a TP monitor.
Almost any system can handle 20 users, but TP systems must
handle many more. When we use batch testing methodologies
(20 batch streams, no monitors), we double our OLTP TPS re-
sults.
Transaction Dispersion
Debit/Credit requires that 15% of all transactions go to
a branch that doesn't own the required account. This forces
systems that use multiple databases (distributed databases)
to incorporate access to multiple databases 15% of the time.
It also forces systems that can cluster data to bypass the
clustering optimization 15% of the time.
By not following the 15% dispersion rule, Sybase can im-
prove its performance through clustering and caching tech-
niques.
Digital follows the Debit/Credit specification, including
the 15% dispersion rule.
Page 4
The Database
Using a database with only 100,000 account records allows
the database to, in effect, reside in real memory on a VAX
8700. This, too, is not realistic. Although Sybase states
that the database should be big enough not to fit into real
memory, theirs does not appear to be big enough.
By placing the database in memory, they achieve a high de-
gree of cache hits. Many of Sybase's performance strate-
gies operate optimally in small database configurations.
For example, Sybase uses a capability called 'Fast Commit'.
With this technique only the log file is written when the
transaction is committed. The data pages stay in memory in
case someone else needs the page. The effect is that many
transactions can share a single IO. If the database is large
enough, the odds of getting such cache hits are small. With
a small database the odds of taking advantage of the Fast
Commit feature are great. Sybase gives a lot of credit to
Fast Commit for their performance.
Another method that database managers use to reduce IO and
CPU costs is data-file partitioning. The ability to par-
tition data-files across multiple disks can drastically im-
prove performance. We use Rdb/VMS's multifile database ca-
pability to partition tables for high performance. Although
the data is spread over multiple disks, there is still one
logical table. This is very important for the maintenance
of database consistency as well as reporting tools. You don't
want users to have to search multiple physical tables for
their records.
In order to improve performance when accessing the History
tables, they create multiple History tables and use their
stored procedures to multiplex between them. This violates
the notion that the History table is a single logical ta-
ble and violates the terms of the TP1 and (more emphati-
cally) the Debit/Credit benchmark. Because Sybase uses mul-
tiple History tables in the benchmark, it would seem that
Page 5
they don't have the capability to support multidisk tables
and provide the associated benefits.
While their stored procedure multiplexes writes to the ta-
ble, it doesn't handle reads. Each user would have to de-
termine how to find their data from one or more different
tables and could not take advantage of common indexing strate-
gies. Moreover, if the partitioning changes, the stored pro-
cedure is not changed automatically. Thus, the use of stored
procedures to maintain partitioned tables can easily lead
to corrupt databases.
Configuration Details
The tests were run on a VAX 8700 driven by a VAX 8800. This
is similar to style 3 Debit/Credit tests. The VAX 8700 had
32 megabytes of real memory. They used dual HSC 70's with
4 RA81's spread across 4 controllers.
Sybase also ran tests on a MicroVAX II with 16 megabytes
of real memory, two Fujitsu 2333 disks, and one Fujitsu 2351
disk on 2 controllers.
Response time in both tests was linear as the number of trans-
action generators increased. On the 8700, 50 generators has
an average response time of 1.78 seconds.
Logging information is not clearly disclosed. From a care-
ful reading of their test results, it seems that they do
not use an after-image log and thereby reduce IO and CPU
utilization as well as reduce response times.
I could not find the schema for the benchmark database in
their report. There is, however, great detail about the ta-
bles used to collect performance information (which makes
the report thicker but of no greater value).
On the MicroVAX II, 25 generators had an average response
time of 3.9 seconds. The MicroVAX II tests had an average
of 6.5 TPS (100,000 account records and 10 generators) with
an average response time of 1.5 seconds.
Page 6
Testing Methodology
With Sybase's Fast Commit feature only the transaction log
needs to be written at transaction commit time. At a user
controllable time all the dirty-pages (those with new data)
are written to disk. Through this mechanism, users can con-
trol how much work is required to recover the database in
the event of system failure. For these tests, Sybase set
the value at 5 minutes.
HOWEVER, THEIR TESTS RUN FOR ONLY THREE MINUTES. This means
that they do almost no IO during their test runs. Obviously,
this greatly reduces CPU utilization and dramatically im-
proves throughput. I assume that had they ran the tests for
more than three minutes, IO would have increased and per-
formance would have dropped considerably.
At 30 TPS for 180 seconds, Sybase only touches 5.4% of the
accounts in the database. In addition, it is highly prob-
able that Sybase runs some tests prior to running the bench-
mark that loads the database into the server's caches. Gen-
erating a test that touches a small part of an in-memory
database fails to follow the spirit of TP1 or Debit/Credit
and definitely fails to model most user applications.
Response Times
Sybase measures average response time. They claim that this
is because their driver program doesn't measure 95% data.
Since they wrote their driver, why didn't they capture the
standard data? By recording average response time, trans-
actions with great delays are masked by those with low re-
sponse times.
Debit/Credit requires that the transaction rate be lowered
until 95% of all transactions complete in less than one sec-
ond. By reporting average response times, Sybase is able
to report higher TPS rates than would be achievable under
Debit/Credit. Moreover, average response times allow Sybase
to show linearity of growth where none may exist.
Page 7
Sybase makes a big deal about their average response time
growing linearly with the number of users. This is precisely
the result you would expect when using average response times.
Their results show that they saturate the throughput of the
machine (either the 8700 or MicroVAX) with five transac-
tion generators. Then, as users are added, the database server
time-slices between each user, increasing average response
time. The same effect can be achieved with any TP monitor
(such as VAX ACMS).
A much more useful measure of linear response time is mea-
sured by following the Debit/Credit standard and measur-
ing response time as the transaction load increases. In-
creasing the transaction load requires that an increased
number of users are supported. Rather than hold through-
put constant and increase the number of users, the Debit/Credit
standard requires both throughput and the number of users
to increase.
The purpose of this type of scaling is to show that addi-
tional users can be added to a system, increasing overall
throughput while maintaining adequate response time. Sybase's
tests show that as more users are added to the system, no
more work (throughput) is achieved; but, overall response
time increases linearly. This is not an acceptable solu-
tion for most TP applications.
Linear Growth
It seems that what Sybase is trying to demonstrate is lin-
ear growth. An important aspect of system performance is
the impact of adding additional CPUs to the TP solution.
The goal is increased performance from increased CPUs. It
is my understanding that a single database can only be served
from a single Sybase server. This means that adding CPUs
to a VAXcluster or to an SMP machine does not provide ad-
ditional performance. Note that this information may change
with the next release.
Page 8
VAX Rdb/VMS provides additional performance in both of these
cases. Perhaps this is why Sybase concentrates on linear
user growth rather than linear performance growth.
Comments and Considerations
Sybase should be treated cautiously in performance eval-
uations. Their 29 TPS can predispose prospects towards them.
It should also be clear, however, that Sybase's 29 TPS can-
not be compared to the DECtp performance results (11 TPS
on an 8700).
The difference in performance numbers puts us at a compet-
itive disadvantage. We need to stress that Sybase modified
the database schema, database size, and test methods to the
point that they effectively tested a small, memory-based
database in batch mode. This does not demonstrate any ca-
pability to support OLTP applications. You can be sure that
if we placed our database in memory and broke all the rules
for testing performance, we could demonstrate much higher
performance results.
Another interesting point that the benchmark reveals is that
while Sybase claims support for standards, Digital supports
standards. To what extent is Sybase's support for other stan-
dards like SQL like their support for Debit/Credit?
Page 9
Appendix A - Comparing Debit/Credit and TP1
Originally, Debit/Credit and TP1 were synonymous. Since the
original definition of Debit/Credit, TP1 has taken on dif-
ferent meanings.
Unlike TP1, the requirements for Debit/Credit are well de-
fined as shown in the April 1, 1985, Datamation article (v31
p112(5)). Debit/Credit is becoming an industry standard bench-
mark for OLTP (On-Line Transaction Processing) applications.
Details of the benchmark are explained in great detail in
the July, 1988, issue of Competitive Update. Briefly, for
each transaction per second (TPS) the system must simulate:
The Debit/Credit Standard
o 100 Tellers (users) executing one transaction each 100
seconds.
o A database with:
o 100,000 Account records
o 100 Teller records
o 10 Branch records
o 2,590,000 History records
o Undo/Redo recovery (Rollback/Rollforward)
o Form presentation services
o 15% of the transactions go against an account that is
not in the teller's branch.
o 95% of all transactions must complete in under one sec-
ond.
o Duplexed recovery log
Page 10
Thus, in order to measure 20 TPS the database must have 2
million account records (20*100,000), 2 thousand teller records,
2 thousand users (tellers), 2 hundred branches (20*10), etc.
The size of the system scales with the desired throughput.
This tends to make the Debit/Credit benchmark more real-
istic than those that show high throughput on minimal data
with few users (i.e. TP1).
Debit/Credit measures two important variables - performance
and cost. Performance has already been described as through-
put (TPS) and response time (95% of all transactions com-
plete in 1 second or less). Cost is determined by adding
the five year costs of hardware, software, and maintenance
and dividing by TPS. For example, a million dollar config-
uration producing 20 TPS costs $50K/TPS.
Changing any of these parameters can have a profound im-
pact on performance. We know through our own testing ef-
forts that performance is easily doubled by reducing the
database size, number of users, and response time require-
ments.
The TP1 Benchmark
Vendors who run TP1 will often make substantial changes to
the Debit/Credit benchmark. The most common changes are:
o Reducing the number of Tellers (users).
For example, using only 100 Tellers for 30 TPS.
o Reducing the database size and not scaling size with through-
put.
For example, using a 100,000 Account record database for
30 TPS.
o Not using an After Image journal for recovery.
o Avoiding the cost of form presentation services.
Page 11
o Measuring average response time instead of 95% 1 sec-
ond response time.
o Not measuring network overhead.
TP1 does a poor job characterizing database performance be-
cause it uses unrealistic database sizes, numbers of users,
and performance monitoring. TP1 does an even worse job in
characterizing total system performance. Database appli-
cations generally have multiple users who require presen-
tation services and network services (DECnet). Since each
vendor's product uses these services differently, they must
be included in the benchmark. Debit/Credit tests these as-
pects of system performance; TP1 does not.
Digital runs the Debit/Credit benchmark.
| T.R | Title | User | Personal
Name | Date | Lines
|
|---|