PreviousBeginningNext

Making Benchmark Runs

First we need to have an understanding of the input, processing, and output of the benchmark run. When a benchmark gets started, the driver framework reads the benchmark configuration file and merges the configuration information with the annotations in the driver. Then it starts the load as specified by the drivers. At the end of the run, the driver framework will create result files in the output directory as specified in the configuration file. 

Benchmark Configuration File

The driver configuration file acts as the glue betweenFaban and the benchmarks. It is an xml file which may contain any number of sections defining configurations of other components, even the benchmark application. The driver framework requires the presence of a runConfig section and one driverConfig section for each driver. Faban specifies the structure of this section. The driver framework usually skips all other sections which may be used by the driver class itself, the Faban harness, the benchmark class, and other components of the benchmark. Appendix C lists a sample driver configuration file consisting of only the driverConfig section.

Benchmark Output Files

At the end of a completed run, the master outputs one or more summary files and a detail file, as follows:
 

Summary File

For a single driver rig, Faban only outputs one grand summary called summary.xml. For larger rigs with 2 or more driver systems, Faban automatically generates the per-driver system summary file in addition to the grand summary. The per-driver summary is named summary.xml.<hostname>. This can give us an indication how much of the load was generated by each of the driver systems, and also helps us detect driver-side anomalies such as a bad network interface on one of the driver systems.

The summary file is in xml format with a concrete schema. It contains a header section that describes the aggregated results, and has sections showing results for each driver. This file can be translated into other more readable formats. The driver provides a style sheet named summary_report.xsl to translate the summary to readable html format. Appendix D shows an example summary file and a HTML translation of the summary file.

Detail File

For a single driver rig, Faban outputs one detail file named detail.xan. For larger rigs with 2 or more driver systems, Faban automatically generates the per-driver system detail file in addition to the grand summary. The per-driver detail is named detail.xan.<hostname>. The detail files are in human-readable text format compatible with FenXi, the postprocessing tool integrated with Faban. The detail files contain the information representing the throughput over time, response time over time, and histograms representing the distribution of response times and cycle or think times of each driver and operation collected from the run and each driver system.

Deploying the Benchmark

The preferred way to run a benchmark is to deploy it in the harness. The harness will be able to manage dependencies and ensure that all needed classes are available to all applicable systems in the rig. In addition, the harness can take care of configuring the rig, starting/restarting server processes, load data, post-processing, and others.

To deploy the benchmark in the harness, you may need 3 additional components:

  1. The benchmark descriptor - benchmark.xml
    This file is needed for any non-standard deployment or if any deployment parameters are needed. Otherwise it is optional.
  2. A configuration form
    By default, this is named config.xhtml. Faban will auto-generate the form if it does not exist. Alternatively, you can use the genxform ant task to generate a form from your configuration file and edit it as you prefer.
  3. The benchmark class
    If you have any custom rig setup/configuration or post-processing, you will want to subclass DefaultFabanBenchmark2 and define those in methods annotated by @Configure and @PostRun.

Details for benchmark deployment is discussed in the Faban Harness Developers Guide and the tutorials.

Testing the Benchmark

The remainder of this secion discusses how you would go about running the benchmark by hand, mainly for testing and debuging.

A Faban driver can be run in two modes: The restricted single-process mode and the full fledged distributed mode. The single-process mode is very useful when testing drivers. It provides for a simple environment to run the driver from a script and debug the driver in your IDE. The distributed driver is more suitable for testing full-fledged benchmarks, the same way the benchmark would be run in the harness. The commands and sequences to run in distributed mode are more complicated and make scripting more difficult as – generally – they will need to be executed on many machines. Most developers do not need to test in this mode and the harness is clearly the suggested way for running distributed drivers.

Running The Single-Process Driver

Running the single process driver is simple. We just need to edit the configuration file and execute the following command:

 java -cp <path>/fabandriver.jar:<myclasspath> -Dbenchmark.config=<config_file_path>
com.sun.faban.driver.engine.MasterImpl

Alternatively, you can use the run task from the ant build file build.xml or run sbin/master.sh in all the samples to start the driver.

Running The Distributed Driver

Starting the distributed driver involves starting all three components of the Faban driver framework in order. First we need to start the registry. The registry registers the agents as they come up. Then we need to start all the agents on all the driver machines. Once all the agents are successfully started, the master will need to be started. The master's job is to read the configuration, coordinate the start of the whole benchmark, and  collect the metrices after the benchmark run has ended.

We do need special security policies to run the distributed driver. Since we are not concerned about security in a benchmark environment, we can have a very broad “allow-all” type of security policy in the policy file as follows:

 grant {
permission java.security.AllPermission
}

The following command starts the registry. 

 java -cp <path>/fabandriver.jar -Djava.security.policy=<policy_file_path>
com.sun.faban.common.RegistryImpl

Alternatively, you can use the registry task from the ant build file build.xml or run sbin/registry.sh in the samples to start the driver.

Next we start all the agents. Note that the <driver_name> represents the annotated driver name, not the driver class name. The <agent_id> is a unique numeric id for each agent running the same driver, starting with 1.

 java -cp <path>/fabandriver.jar:<myclasspath> -Djava.security.policy=<policy_file_path>
com.sun.faban.driver.engine.AgentImpl <driver_name> <agent_id> <master_host_name>

Alternatively, you can use the agent task from the ant build file build.xml or run sbin/agent.sh in the samples to start the driver.

After all the agents are up, we then start the master with this command. The master is just started like the single-process driver, with a few additional but necessary arguments.

 java -cp <path>/fabandriver.jar:<myclasspath> -Djava.security.policy=<policy_file_path>
-Dbenchmark.config=<config_file_path> com.sun.faban.driver.engine.MasterImpl

Alternatively, you can use the run task from the ant build file build.xml or run sbin/master.sh in all the samples to start the driver.

These represent the minimum needed to start the Faban driver framework processes. For proper tuning and configurations, the next section will discuss the common JVM tuning and  system properties used for running the driver framework.

Properties and Java Parameters

All properties are passed to Faban using the Java standard -D command line argument. There must be no space between the -D and the property name, i.e. -Djava.util.logging.config.file=myLogConfig.properties. The way java parameters are passed to the JVM depends on the specific parameter and will be discussed separately for each specific option below:

java.util.logging.config.file 

Scope: All Faban Driver Framework processes. 

Faban utilizes the java logging APIs and therefore allow explicit logging configuration utilizing the standard java logging API standard. The property is needed to set the log level for each package and the log target. If the Faban Harness is used, this will be set automatically.

java.security.policy 

Scope: All Faban Driver processes, distributed mode only.

This is a mandatory property for the distributed driver. Without this property and a proper security policy file (discussed above), the distributed driver will seize to function.

benchmark.config

Scope: Master or single-process driver.

This is a mandatory property for the master or single-process driver. It points the master to the benchmark config file and uses this config file to configure and read the developer-implemented driver.

faban.outputdir.unique

Scope: Master or single-process driver.

This property tells the master not to generate it's own run identifier. It is generally used by Harnesses telling the Faban Driver framework to target the run output to a certain directory instead of creating a sub-directory. The name of the directory isuse as the run identifier.

faban.registry.port

Scope: All processes, distributed mode only.

This property overrides the default registry port 9998 to use a different port. This property is used to avoid conflict between multiple Faban Driver framework instances installed in the same network.

faban.ssl.autotiming

Scope: All Faban Driver processes, when driving ssl traffic over the Apache HttpClient 3.1 transport provider.

Attaches the socket auto-timer above or below the ssl stack. The two options are "above" and "below." The default is attaching the auto-timer above the ssl stack. Attaching it below excludes the client-side cryptographic processing from the reported times.

faban.xsl.path

Scope: Master or single-process driver. 

This property specifies the stylesheet paths to use for displaying results. It defaults to the harness' xsl path which is a relative path ../../xslt.

DisableExplicitGC

Scope: All processes, distributed mode only. 

This option is recommended for all drivers running in distributed mode. This is a JVM option, not a property. It disables the regular garbage collection calls and avoid long pauses in the driver.  To use this option, specify -XX:+DisableExplicitGC in the java command line.

PreviousBeginningNext