Faban Harness Development Overview

The Faban harness is a tool to help automate running benchmarks. It does this by a well-defined process. The Faban Driver Framework Developers Guide explains how to develop a benchmark to run in Faban. That framework provides a way to define the workload logic, but we still need something to glue the benchmark to the harness . This glue is described in this guide. Benchmarks developed using the Faban driver framework (hereinafter referred to simply as Faban benchmark) can take advantage of the built-in process logic for the Faban Driver Framework. Since this is generally standard, you may not need to implement anything additional. And even if you find that the default process doesn't work for you, you can easily extend it. For benchmarks developed outside the faban framework, you will have to implement additional process glue to link the benchmark to the harness. In the best case you may be able to copy the logic from one of the sample benchmarks or an existing benchmark and adapt it.

Faban framework also provides for pluggable services and tools. A service is a module in Faban that allows interactions with specific server software. Common services used for many tests are OracleService, Apache2HttpdService, MemcachedService, MySQLService, GlassFishAppServerService, etc. Many of such server software provide observerability interfaces and tools that can tell users what was happening in the server during the benchmark run. For example, awr is a tool to look into the Oracle database, MySQL provides a query interface to read certain MySQL counters, etc. The pluggable tools interface allows for writing accessors or wrappers for these tools in a standard way, allowing the Faban harness to invoke them and collect information at proper times during the experiments.  Some of the services and tools examples are packaged with the Faban kit which are located in samples/services directory. Services and tools are enabled by certain entries in the run configuration file. There is no direct API call into the services and tools from the benchmark class.

Benchmark Components

The following components are required to run a benchmark in the Faban harness:

1. The benchmark deployment descriptor
2. The default configuration file
3. The XForm for editing the benchmark configuration
4. The harness hook (or benchmark class) that defines the process for each benchmark run.

The first three files reside in the deploy sub-directory of the benchmark directory (as for e.g. samples/benchmarks/web1/deploy). The harness hook is Java code. The benchmark deployment descriptor references all files as well as the harness hook's fully qualified  class name.

Benchmark Run Process

The entire benchmark run process and how the components described above fit in is described in the figure below. The harness uses the current (or default if this is the first run) run configuration file and the submission form to generate the web form for the run. Using the user-entered values, it then generates a new run configuration file for this run. The run is then placed on the run-queue to be executed by the run daemon. When ready to run, the Benchmark class processes the run and if defined, uses the harness hook.

Pluggable Service/Tools Components

Services and tools are Faban components representing certain server software and the observerability interfaces for such software. If services are configured in the run configuration file and restart for the services are enabled, they will get started/restarted by the Faban harness before the run starts. The same services will get stopped by Faban after the run ends.  Similarly, tools get configured before the run starts. However, they actually collect information during the steady state. Tools usually need to access service configuration information in order to connect  to the right server and collect statistics from or for the given service instance. Multiple services and tools may be bundled in a single package. To implement services and/or tools, the following components are needed..

1. The services-tools.xml deployment descriptor
2. The toolsets.xml file
3. The service implementations or wrappers
4. The tool implementations or wrappers
   

The xml files reside in the deploy sub-directory of the services project (as for e.g. samples/services/MySQLService/deploy). Alternatively, you may also provide services and tools as part of a benchmark, sharing the directories and class structure with your benchmark. This will be discussed further in Services/Tools Deployment.

Implementing a Benchmark, Service, or Tool Module

Implementing a benchmark, service or tool module involves coming up with all the components described, respectively, and packaging them correctly before deploying them into the Faban harness. Luckily, the sample provided with the Faban package provides a good starting point for a benchmark/service project structure. It has a very generic ant build file that works for most basic benchmarks, services, and tools, and can be extended for even more complex ones. Also it provides a good starting point for all the components mentioned above. We will use the provided samples in most of the following discussions. It is a good idea now to copy the samples/benchmarks/web101(for benchmarks) and samples/services/MySQLService(for services) directory and create a project using your IDE of choice with this sample.

Benchmark, Services, and Tools Operation Sequence

In order to understand how the benchmarks, services, and tools fit together, the following describe the call sequence for annotated methods in the benchmark, services, and tools. You can then decide based on the sequence what infrastructures are available and what can be done from what annotated operaion.

1. Benchmark @Validate - this allows validating and editing the configuration before any infrastructure bringup.
2. Remote infrastructure bringup
3. Benchmark @Configure
4. Services @Configure
5. Services @Stop - only if restart is set to 'true' in the configuration
6. Services @ClearLogs
7. Services @Start - only if restart is set to 'true' in the configuration
8. Services @GetConfig - obtains the service configuration
9. Tools @Configure
10. Benchmark @PreRun
11. Benchmark @StartRun
12. At steady state begin - Tools @Start
13. At end of steady state (if > 0) - Tools @Stop
14. Benchmark @EndRun
15. If steady state undefined (<= 0) - Tools @Stop
16. Tools @PostProcess
17. Tools transfer output files to master
18. Benchmark @PostRun
19. Services @Stop
20. Services @GetLogs

Please see The Benchmark Class and The Service and Tool Classes for more information on these annotations.

Structure of a Faban Benchmark and Service Bundle Project

Before we start getting our hands dirty, we'll understand a generic Faban project structure. Part of it is well reflected in the web101 sample. However, the structure below also shows optional paths not in the sample:

MyBenchmarkProject
bin
build
classes
lib
config
security
driver.policy
logging.properties
run.xml
deploy
benchmark.xml
config.xhtml
run.xml
services-tools.xml
toolset.xml
lib
public_html
resources
sbin
agent.sh
master.sh
multi.sh
registry.sh
setenv.sh
src
build.properties
build.xml
README

Let's start with the most significant directories first. 

• The src directory is the root of your Java source tree. This is where you want to place the benchmark class, under an appropriate package under the source tree. If you develop using the Faban driver framework, your driver classes and any support class would also be hosted under this source tree.
  
• The build directory and the subdirectories underneath are automatically created by the build script - build.xml. The compiled Java classes are organized according to the source tree in the build/classes directory under the build directory. The packaged jar libraries are placed under the build/lib directory. Although the deployable image is named a jar file, it includes a set of jar libraries and other binaries. It is not a java library by itself. It is more analogous to a war file (web archive).
  
• The deploy directory contains the deploy metadata. It will get packaged into the META-INF directory of the deployable image. The contents of this directory is discussed later under Benchmark Deployment and Service Deployment.
  
• The bin directory is the place for all benchmark-specific binaries and scripts that are called by the benchmark. You may want to create subdirectories named after the OS, and next level subdirectories named after the processor's architecture to support binaries for multiple OSs and instruction set architectures (ISAs). Names currently in use are SunOS, Linux, Windows, and Mac. The name of the ISAs are OS and JVM dependent. Faban will search for the most specific binary or script (the deepest directory in the hierarchy) to the least specific when trying to execute a binary or script. Also, the benchmark-specific binaries supercede the Faban binaries and the OS path, in order, if there are commands with the same name.
  
• The lib directory is the place for all 3rd-party libraries, both Java jars and native, shared libraries. All jar files are automatically included in the classpath when Faban executes a Java command. The native libraries can be placed into OS and ISA-specific subdirectories to avoid naming conflicts. The structure is just like the bin directory.
  
• The resources directory is used for other files that both the benchmark class and Faban drivers may need in order to conduct a successful benchmark run. Faban provides APIs for the driver framework to obtain a reference from this directory directly.
  
• The public_html directory allows the benchmark to include any relevant documentation. This along with any custom banners will be served by the harness.
  
• The config and sbin directories are only used for and by the Faban driver framework. The config directory contains configuration files needed for testing a benchmark driver locally. The bin directory contains scripts to assist in starting the Faban driver framework. Both directories are not used for a deployed benchmark.
  
• The build.xml file is the ant build file used for building and deploying a Faban project. Most if not all Java IDEs include ant and allow you to execute the relevant ant targets directly from the IDE. But if needed, you can also obtain ant directly from http://ant.apache.org/. This file can be copied from any of the sample benchmarks and is a standard build script that does not need any modification.
• The build.properties file contains important build and deploy time properties. This is the file you want to edit to fit the build process to your build environment. We will discuss the build and deployment process later under Deployment.
  
• It is always advised to provide a README file to your project to provide basic information about the benchmark and refer to the documents under public_html.