Quest Software Benchmark Factory

We mentioned that the benchmarks we previously used were no longer useful, as we did not have the I/O capacity required to support them. We went looking for alternative benchmarks, and stumbled upon Benchmark Factory from Quest Software. Below, is a description of the product and the benchmarks we used in this article.

Benchmark Factory for Databases is a performance and code scalability testing tool that simulates users and transactions on the database and replays a production or synthetic workload in non-production environments. This enables organizations to validate database scalability as user loads increase, application changes are made, and platform changes are implemented. Benchmark Factory is available for Oracle, SQL Server, DB2, Sybase, MySQL and other databases via ODBC & Native connectivity.

Benchmark Factory provides many tests you can run, and has a very nice and customizable metric reporting engine. We decided to run the AS3AP test, and the Scalable Hardware CPU, Reads, and Mixed tests. Here is what Quest's help file says about these tests:

AS3AP

The AS3AP benchmark is an American National Standards Institute (ANSI) Structured Query Language (SQL) relational database benchmark. The AS3AP benchmark provides the following features:
  • Tests database processing power
  • Built-in scalability and portability that tests a broad range of database systems
  • Minimizes effort in implementing and running benchmark tests
  • Provides a uniform metric and straightforward interpretation of benchmark results
Systems tested with the AS3AP benchmark must support common data types and provide a complete relational interface with basic integrity, consistency, and recovery mechanisms. The AS3AP tests systems ranging from a single-user microcomputer Database Management System (DBMS) to a high-performance parallel or distributed database.

Scalable Hardware

The Scalable Hardware benchmark measures relational database systems. This benchmark is a subset of the AS3AP benchmark and tests the following:
  • CPU
  • Disk
  • Network
  • Any combination of the above three entities
We run three iterations of each load point, and then average the results. We also monitor deviations to ensure they are within an acceptable range. We like to see a max deviation of +/- 3%.

Benchmarking Clovertown Test Setup
Comments Locked

56 Comments

View All Comments

  • yyrkoon - Monday, April 2, 2007 - link

    You can not read, and understand what I am writting, and I am the dolt or moron . . .

    Interresting that . . . interresting indeed. I think what I will do, is just ignore whatever else you have to say, just like the majority of other readers seemingly have done.
  • archcommus - Friday, March 30, 2007 - link

    However if Barcelona comes out and then Penryn smashes it just a few months later, yeah, then I'm gonna be worried about them. :(
  • Griswold - Saturday, March 31, 2007 - link

    Say no to drugs.
  • anony - Friday, March 30, 2007 - link

    This is for the authors. Sorry if I missed it, but do the power measurements
    include chipset power? AMD processors include the memory controller as well,
    right? Do the performance/watt take this into account?
  • Ross Whitehead - Friday, March 30, 2007 - link

    We measured power at the wall, but we do not include the power for the disk chassis.

    Thus, performance/watt takes all of your mentioned items into account.
  • blckgrffn - Friday, March 30, 2007 - link

    I am guessing Pernyn will be different enough from Clovertown to make using vmotion (and many other enterprise features) impossible. It sucks enough that we already have two processor families in our Dell 2950's, and here comes one more.

    I am all for progress, it just looks like this might be something VMware has to address at some point.

    Nat
  • Beenthere - Friday, March 30, 2007 - link

    ...the industry. As usual Intel's "glueblob" is another rushed-out-the-door, knee-jerk reaction to AMD supplying superior CPU products. AMD is really gonna hurt Intel with Barcelona and friends.
  • johnsonx - Friday, March 30, 2007 - link

    Beenthere + Cornfedone = Cramitpal
  • Griswold - Saturday, March 31, 2007 - link

    You forgot to add some "fine-ass".
  • Phynaz - Friday, March 30, 2007 - link

    Wow, you really are a moron.

Log in

Don't have an account? Sign up now