Index of /~farrell/cc11/reference/hint
_ _
| | _ _ _ _____ TM
|-- | | |\ | | | |
| --| | | \ | |
| | | | \| |
^ ^ ^ ^ ^ ^
*** The HINT PERFORMANCE ANALYZER ***
HINT reveals many aspects of computer performance: operation speed,
precision, memory bandwidth, and usable memory size. It is completely
unlike conventional computer benchmarks. HINT measures "QUIPS"
(QUality Improvement Per Second) as a function of time. "Net QUIPS" is a
single number that summarizes the QUIPS over time. This file explains how
to run HINT, how parameters may be adjusted, and what may NOT be adjusted.
The faster the computer, the greater the need for high precision, fast access,
and large usable storage at all levels. Unlike conventional benchmarks,
shortcomings in any of these will be reflected in the figure of merit
(Net QUIPS).
ADJUSTING PRECISION
The HINT performance analyzer is designed to compile and run with
user-selected data types, via C preprocessor directives and defines. DSIZE
sets the data type for computation and ISIZE sets the type for the indexing.
The -D option of C compilers performs the substitution. One method
is to set the data types on the compile line:
cc -DDSIZE=double -DISIZE=int hint.c hkernel.c -o hint
^^^^^^^^^^^^^^ ^^^^^^^^^^^
Or, the data types can be set by placing two of the defines found in
typedefs.h on the compile line:
cc -DDOUBLE -DIINT hint.c hkernel.c -o hint
^^^^^^^^ ^^^^^^
Or, insert
#define DSIZE double
#define ISIZE int
at the top of the hint.h file.
These examples cause the computation variables to be the C "double" type,
which typically will give enough precision to exercise workstation-speed
computers. Integers work as well as floating-point numbers, but unsigned
integers are currently not allowed. The tradeoff is that low precision
might run faster initially but with lower answer quality. The Net QUIPS
rating reflects both answer quality and speed, so experiment with different
types to maximize Net QUIPS as reported by the program. It is useful to
know both the best integer Net QUIPS and the best floating point Net QUIPS.
The examples above also show the indexing variables set to the C "int" type.
HINT will automatically ensure that indexing variables can represent integers
at least as large as the square root of the largest whole number used in a
computational variable. To make full use of DSIZE precision, the number of
bits in the ISIZE type should be at least half that of the DSIZE variable type.
PORTING THE CODE
HINT is written in ANSI C to provide for easy porting. HINT is basically
two parts, a driver and a kernel. The driver performs some self diagnostics
for the data types being used and then runs the loop which calls the kernel.
The running of the kernel is the only part of the code that is timed so
emphasis on optimization should be placed on the file "hkernel.c". We
recommend that porting the code proceed in the following steps.
* Modify the source code following the commented locations for vendor
specific code in "hint.c" and "hkernel.c".
* Modify the Makefile appropriately as shown in its comments.
* Make any modifications to the "ADJUSTABLE DEFINES" in "hint.h" that
you feel are necessary for your machine. Be sure to follow the rules
when you do these modifications.
We recommend that you first download and port the serial version of the code
and familiarize yourself with its workings on your machine. The serial version
of HINT should be fairly easy to port. Most Unix workstations require
only modifications to the Makefile. Good Luck! Any questions should be
directed to hint@scl.ameslab.gov.
GETTING ACCURATE TIMINGS
The philosophy is as follows: There is an ultimate QUIPS curve for a
given computer. Do your best to get close to it, within the limits of your
patience. By taking many data points and many repetitions per data
point, you might increase Net QUIPS by a few percent but find yourself
running HINT for hours. We envision a QUIPS curve measurement taking 10 to 20
minutes, usually. Comments in the hint.h file under "Adjustable Defines"
indicate how to adjust the way the QUIPS curve is sampled.
HINT is designed to filter out bad results due to spurious interrupts.
The best numbers will be attained, and attained with fewer trials, when
interrupts can be avoided.
UNDERSTANDING AND VISUALIZING THE OUTPUT
The results of the benchmark are written to mass storage. Our makefile
creates an executable program named "hint". The output data file is named
after the name of the data type being tested like "DOUBLE" or "LONGLONG".
Other diagnostic information is written to the screen. This information
includes
* the number of bits of accuracy of each data type used,
* the resulting grid size used for HINT,
* a series of periods (.) indicating that HINT is running, with "#"
used to indicate a retrial of an apparently inaccurate measurement,
* the reason for quitting the run, and
* the Net QUIPS.
The Net QUIPS represents the integral of the Quality step function divided
by time squared. This is the area under a graph of QUIPS versus the log of
the time. This simply means that the QUIPS at a given time is weighted lower
in proportion to how long it takes to get to that QUIPS rate. For more
information see "HINT: A New Way To Measure Computer Performance",
Ames Laboratory Technical Report Number IS-5109, Category UC-405.
The data file is organized in five columns: time, QUIPS, Quality,
subintervals, and memory use in bytes. The first two columns are used to
produce the QUIPS graph. The third and fourth columns are Quality and
subintervals. These columns can be used to check for loss of quality due to
insufficient precision and poor choices of which rectangle to split. The fifth
is useful for finding the sizes of memory regimes, like where primary cache
saturates or where virtual memory is first invoked.
The file is set up in a format suitable for gnuplot or similar plotting
packages. QUIPS graphs should be plotted on a semi-log scale with log of
time as the horizontal axis and QUIPS as the vertical axis. The following
gnuplot commands would be suitable to display the QUIPS graph from the file
"DOUBLE":
> set logscale x
> plot "DOUBLE"
QUIPS vs. memory use can also be plotted to reveal memory regime sizes,
by using the following gnuplot command:
> plot "DOUBLE" using 5:2
Although the human tendency is to try to turn performance into a single
number, our hope is that machines will be compared using the entire QUIPS
curve and not just the Net QUIPS rating. Just as the evaluation of cars
involves many variables other than 0-60 acceleration time, computers can
never be judged by one number alone. These QUIPS graphs reveal many
traits of a particular computer, such as
* The "startup time" to the first data point, a measure of nimbleness
* The cost for a sum collapse for parallel computers
* The number of distinct memory regimes
* The "usable" size of each regime
* The relative speed of access of each regime
* Tendency to be erratic because of interrupts
Plotting the QUIPS curves of multiple computers on the same graph
reveals the relative strengths and weaknesses of the machines. For example,
a computer might be fast only on small problems (like 100 by 100 LINPACK)
but much slower on real applications that do not fit in cache. Using the
knowledge gained from the QUIPS graph coupled with that of the expected
workload will help to judge which computer is best suited for the workload.
REPORTING RESULTS
We are trying to create an atlas of HINT results. We would appreciate
all information and results sent to us. Please include the following
information as well as any output data files generated by HINT. Also include
the modified code and an explanation of your modifications. Note that your
name is required as well as the machine statistics. If there is any information
that you are unable or unwilling to stand behind, please leave the entry blank.
Also, if the information was not personally measured, like power consumption,
please indicate by placing an asterisk (*) at the end of the entry. Send all
information and data files to:
hint@ameslab.gov
REPORT FORM
Modify one of the following examples to match your computer.
Serial Computer: (typical workstation)
Personal Information
Name: Quinn O. Snell
Company: Ames Laboratory
Address: 236 Wilhelm
Iowa State University
Ames, Iowa 50011 - 3020
Email: snell@scl.ameslab.gov
Date: June 20, 1994
Computer Architecture
Computer Vendor: Silicon Graphics Inc.
Computer Model name/number: Indy SC
Operating System: IRIX 5.2
Graphics: Indy 24-bit graphics
Standalone or Multiuser: Multiuser
Processor: MIPS R4000
Floating Point Coprocessor: MIPS R4010
Clock Rate: 100 MHZ *(Not personally measured)
# of Processors: 1
# used: 1
Communication Architecture Not applicable
Topology:
Interprocessor Latency:
Interprocessor Bandwidth:
* Note: If communication time is not equal to
latency + bandwidth * size, please explain.
Physical Statistics
Serial Number: 08006906E67A
Physical Dimensions: 41cm x 35cm x 8cm
Weight:
Power Consumption:
Purchase Date: 9/30/93
List Price at Purchase Date:
Memory Architecture
Memory Size: 48 MB
Memory Interleave:
Memory Bandwidth: 267 MB / sec *
Primary Cache Size: 8 KB data / 8 KB instruction *
Secondary Cache Size: 1 MB *
Cache Associativity:
Type of Hard Disk(Virtual Mem): SCSI 2
Hard Disk brand name: IBM
Hard Disk Size: 1.2 GB *
Hard Disk access time:
HINT Executable Statistics and Results
HINT version: 1.0
Language Used: C (ANSI)
Compiler: cc
Options Used: -O3 -non_shared
Data Type for Computation: double
Data Type for Indexing: int
Net QUIPS: 2.32 MQUIPS
Parallel Computer:
Personal Information
Name: Quinn O. Snell
Company: Ames Laboratory
Address: 236 Wilhelm
Iowa State University
Ames, Iowa 50011 - 3020
Email: snell@scl.ameslab.gov
Date: June 20, 1994
Computer Architecture
Computer Vendor: Intell
Computer Model name/number: Paragon
Operating System: OSF/1 Release 1.0.4 Server 1.2 R1.2
Graphics:
Standalone or Multiuser: Standalone
Processor: i860 xp
Floating Point Coprocessor: none
Clock Rate: 50 MHZ *
# of Processors: 32
# used: 32
Communication Architecture
Topology: Mesh
Interprocessor Latency:
Interprocessor Bandwidth:
* Note: If communication time is not equal to
latency + bandwidth * size, please explain.
Physical Statistics
Serial Number:
Physical Dimensions: 56cm x 156cm x 107cm *
Weight: 1,000 lbs *
Power Consumption: 11,000 Watts *
Purchase Date: 3/93
List Price at Purchase Date:
Memory Architecture
Memory Size: 32 MB / node *
Memory Interleave:
Memory Bandwidth: 400 MB / sec *
Primary Cache Size: 16 KB data / 16 KB instruction *
Secondary Cache Size: none
Cache Associativity:
Type of Hard Disk(Virtual Mem): RAID SCSI 2
Hard Disk brand name: Maxtor
Hard Disk Size: 4 GB *
Hard Disk access time:
HINT Executable Statistics and Results
HINT version: 1.0
Language Used: C (ANSI)
Compiler: cc
Options Used: -O3 -Knoieee
Data Type for Computation: double
Data Type for Indexing: int
Net QUIPS: