First we need to have an understanding of the input, processing, and output of the benchmark run. When a benchmark gets started, the driver framework reads the benchmark configuration file and merges the configuration information with the annotations in the driver. Then it starts the load as specified by the drivers. At the end of the run, the driver framework will create result files in the output directory as specified in the configuration file.
The driver configuration file acts as the glue betweenFaban and the benchmarks. It is an xml file which may contain any number of sections defining configurations of other components, even the benchmark application. The driver framework requires the presence of a runConfig section and one driverConfig section for each driver. Faban specifies the structure of this section. The driver framework usually skips all other sections which may be used by the driver class itself, the Faban harness, the benchmark class, and other components of the benchmark. Appendix C lists a sample driver configuration file consisting of only the driverConfig section.
At the end of a completed run, the
master outputs one or more summary files and a detail file, as follows:
For a single driver rig, Faban only outputs one grand summary called summary.xml. For larger rigs with 2 or more driver systems, Faban automatically generates the per-driver system summary file in addition to the grand summary. The per-driver summary is named summary.xml.<hostname>. This can give us an indication how much of the load was generated by each of the driver systems, and also helps us detect driver-side anomalies such as a bad network interface on one of the driver systems.
The summary file is in xml format with
a concrete schema. It contains a header
section that describes the aggregated results, and has sections showing
results for each driver. This file can be
translated into other more readable formats. The driver provides a
style sheet named summary_report.xsl
to translate the summary to readable html format. Appendix D shows an
example summary file and a HTML translation of the summary file.
For a single driver rig, Faban outputs
one detail file named detail.xan.
For larger rigs with 2 or more driver systems, Faban automatically
generates the per-driver system detail file in addition to the grand
summary. The per-driver detail is named detail.xan.<hostname>.
The detail files are in human-readable text
format compatible with FenXi, the postprocessing tool integrated with
Faban. The detail files contain the
information representing the throughput over time, response time over
time, and histograms representing the distribution of
response times and cycle or think times of each driver and operation
collected from the run and each driver system.
The preferred way to run a benchmark is to deploy it in the harness. The harness will be able to manage dependencies and ensure that all needed classes are available to all applicable systems in the rig. In addition, the harness can take care of configuring the rig, starting/restarting server processes, load data, post-processing, and others.
To deploy the benchmark in the harness, you may need 3 additional components:
Details for benchmark deployment is
discussed in the Faban
Harness Developers
Guide and the tutorials.
The remainder of this secion discusses how you would go about running the benchmark by hand, mainly for testing and debuging.
A
Faban driver can be run in two modes: The restricted single-process
mode and the full fledged distributed mode. The single-process mode
is very useful when testing drivers. It provides for a simple
environment to run the driver from a script and debug the driver in
your IDE. The distributed driver is
more suitable for testing full-fledged benchmarks, the same way the
benchmark would be run in the harness. The commands and sequences to
run in distributed mode are more complicated and make scripting more
difficult as – generally –
they will need to be executed on many machines. Most developers do not
need to test in this mode and the harness is clearly the suggested way
for running distributed drivers.
Running the single process driver is simple. We just need to edit the configuration file and execute the following command:
java -cp <path>/fabandriver.jar:<myclasspath> -Dbenchmark.config=<config_file_path>
com.sun.faban.driver.engine.MasterImpl
Alternatively, you can use the run task from the ant build
file build.xml
or run sbin/master.sh
in all the samples to start the driver.
Starting the distributed driver involves starting all three components of the Faban driver framework in order. First we need to start the registry. The registry registers the agents as they come up. Then we need to start all the agents on all the driver machines. Once all the agents are successfully started, the master will need to be started. The master's job is to read the configuration, coordinate the start of the whole benchmark, and collect the metrices after the benchmark run has ended.
We do need special security policies to run the distributed driver. Since we are not concerned about security in a benchmark environment, we can have a very broad “allow-all” type of security policy in the policy file as follows:
grant {
permission java.security.AllPermission
}
The following command starts the registry.
java -cp <path>/fabandriver.jar -Djava.security.policy=<policy_file_path>
com.sun.faban.common.RegistryImpl
Alternatively, you can use the registry task from the ant build file build.xml or run sbin/registry.sh in the samples to start the driver.
Next we start all the agents. Note that the <driver_name> represents the annotated driver name, not the driver class name. The <agent_id> is a unique numeric id for each agent running the same driver, starting with 1.
java -cp <path>/fabandriver.jar:<myclasspath> -Djava.security.policy=<policy_file_path>
com.sun.faban.driver.engine.AgentImpl <driver_name> <agent_id> <master_host_name>
Alternatively, you can use the agent task from the ant build file build.xml or run sbin/agent.sh in the samples to start the driver.
After all the agents are up, we then start the master with this command. The master is just started like the single-process driver, with a few additional but necessary arguments.
java -cp <path>/fabandriver.jar:<myclasspath> -Djava.security.policy=<policy_file_path>
-Dbenchmark.config=<config_file_path> com.sun.faban.driver.engine.MasterImpl
Alternatively, you can use the run task from the ant build file build.xml or run sbin/master.sh in all the samples to start the driver.
These represent the minimum needed to start the Faban driver framework processes. For proper tuning and configurations, the next section will discuss the common JVM tuning and system properties used for running the driver framework.
All
properties are passed to Faban using the Java standard -D command line
argument. There must be no space between the -D and the property name,
i.e. -Djava.util.logging.config.file=myLogConfig.properties.
The way java parameters are passed to the JVM depends on the specific
parameter and will be discussed separately for each specific option
below:
java.util.logging.config.file
Scope: All Faban Driver Framework processes.
Faban
utilizes the java logging APIs and therefore allow explicit logging
configuration utilizing the standard java logging API standard. The
property is needed to set the log level for each package and the log
target. If the Faban Harness is used, this will be set
automatically.
java.security.policy
Scope: All Faban Driver processes, distributed mode only.
This
is a mandatory property for the distributed driver. Without this
property and a proper security policy file (discussed above), the
distributed driver will seize to function.
benchmark.config
Scope: Master or single-process driver.
This
is a mandatory property for the master or single-process driver. It
points the master to the benchmark config file and uses this config
file to configure and read the developer-implemented driver.
faban.outputdir.unique
Scope: Master or single-process driver.
This
property tells the master not to generate it's own run identifier. It
is generally used by Harnesses telling the Faban Driver framework to
target the run output to a certain directory instead of creating a
sub-directory. The name of the directory isuse as the run
identifier.
faban.registry.port
Scope: All processes, distributed mode only.
This
property overrides the default registry port 9998 to use a different
port. This property is used to avoid conflict between multiple Faban
Driver framework instances installed in the same network.
faban.ssl.autotiming
Scope: All Faban Driver processes, when driving ssl traffic over the Apache HttpClient 3.1 transport provider.
Attaches the socket auto-timer above
or below the ssl stack.
The two
options are "above"
and "below."
The default is attaching the auto-timer above the ssl stack. Attaching
it below excludes the client-side cryptographic processing from the
reported times.
faban.xsl.path
Scope: Master or single-process driver.
This
property specifies the stylesheet paths to use for displaying results.
It defaults to the harness' xsl path which is a relative path ../../xslt.
DisableExplicitGC
Scope: All processes, distributed mode only.
This option is
recommended for all drivers running in distributed mode. This
is a JVM option, not a property. It disables the regular garbage
collection calls and avoid long pauses in the driver. To use
this
option, specify -XX:+DisableExplicitGC
in the java command line.