While we do not encourage writing your benchmark from scratch, doing so is certainly simple enough. The easier method is just to use one of the samples Faban provided by Faban as a template. This guide should just be a reference to better understand the benchmark class as you'll find in the samples.
Benchmarks are relatively simple. You only need to provide at
least one
driver class and a driver configuration file. The driver class
represents a
plain old Java object (POJO) containing annotations defining the load
model of the driver. The Faban driver framework will use this load
model to call your driver class. The followings show the steps of
developing a
benchmark.
While most benchmarks define only one
driver, more complex benchmarks may use multiple driver classes. The
first driver class usually contains the benchmark definition - the
@BenchmarkDefinition annotation. We also call this driver the "defining
driver." Other drivers need not have the @BenchmarkDefinition
annotation.
The Faban driver framework organizes the common driver API inside a single package: com.sun.faban.driver. Since we'll require many of the annotations and support classes in this package, it is useful to do a wildcard import at the top of your Java class file, as follows:
import com.sun.faban.driver.*;
In the simplest case, annotate the defining driver class with an annotation as follows:
@BenchmarkDefinition (
name = "Sample Web Workload 1",
version = "0.2",
drivers = "WebDriver.class" // Optional
)
A benchmark is defined using the @BenchmarkDefinition
annotation. The configuration file refers to a class with this
annotation. Only one class need to be annotated with the
BenchmarkDefinition annotation.The drivers attribute lists all the
other drivers you may have. It is optional to list the defining
driver as it is automatically added to the set of drivers. For a simple
benchmark with only one driver and the benchmark
definition in the driver, the drivers
attribute can be omitted. Please see the @BenchmarkDefinition
annotation
reference for complete detail of this annotation.
While many benchmarks may have only one driver, there are more complex benchmarks that have multiple drivers. Each driver class needs to have the @BenchmarkDriver annotation. From the implementation standpoint, a driver is a user-defined class of an arbitrary name. It is a plain old Java object (POJO) annotated as a BenchmarkDriver as follows:
@BenchmarkDriver (
name = "WebDriver",
threadPerScale = 1 // Optional, defaults to 1.
)
Operations define the logic to generate load to the severs. All operations for a driver are defined as methods inside the respective driver class. The method defining each individual operation must be be declared as public void, must not take any arguments, and must be annotated as a BenchmarkOperation. The following is an example of a valid operation declaration:
@BenchmarkOperation (
name = "MyOperation1",
max90th = 2,
timing = Timing.AUTO
)
public void doMyOperation1() {
....
}
A mix tells the driver framework the policy or algorithm it should use to select operations. While the Faban driver framework may be extended to support additional mixes over time, currently it supports four mixes:
The mix annotations have a class scope. You'll need to annotate the driver class with the mix. Only one mix type may be specified for a certain driver or a DefinitionException will be thrown on driver startup causing the benchmark run to terminate immediately. The following example shows a matrix mix ratio and the translation of that into the @MatrixMix annotation:
| From/To
⇒ |
Operation1 |
Operation2 |
Operation3 |
|---|---|---|---|
| Operation1 |
10 |
50 |
40 |
| Operation2 |
30 |
10 |
60 |
| Operation3 |
45 |
45 |
10 |
@MatrixMix(
operations = { “Operation1”, “Operation2”, “Operation3” },
mix = { @Row({10, 50, 40}),
@Row({30, 10, 60}),
@Row({45, 45, 10}) },
deviation = 2
)
Note: For stateful mixes (all mixes except FlatMix), you may need to reset the state if an error occurred in the operation. Please see Error Handling and the DriverContext.resetMix() method for details.
Cycle timings tell the driver framework the policy or algorithm to be used for inter-operation times. Currently, three timing characteristics are supported:
Unlike the mix annotations, the cycle timings can have a class or a method scope. You can specify the cycles as an operation annotation or a class annotation. The class level annotations covers all operations in that driver class. The method level annotation specifies the cycle timings for only that operation. It also overrides the class level annotation if there is one, just for the given operation.
Also it is important to note that if specified at the operation level, these annotations specify the policy for calculating the cycle or sleep time after the invocation of the operation. The followings example shows specifying the negative exponential disctibution for cycle time:
@NegativeExponential (
cycleType = CycleType.CYCLETIME,
cycleMin = 500,
cycleMean = 5000,
cycleDeviation = 2
)
In certain circumstances, we need to simulate client applications interacting with the SUT and some of the interactions are not controlled by the user but are executed regularly in the background. A common example is a POP mail client which needs to download the new mail every preset number of minutes. This is the basis of the additional annotation type @Background. The following example shows the background annotation:
@Background (
operations = "Synchronize",
timings = @FixedTime (
cycleTime = 60000,
cycleDeviation = 2
)
)
Usually, there is only one background operation type, but the Faban driver framework allows for more than one operation type if necessary. The cycle, however, must be FixedTime. Please also note that while the background operation is scheduled separate from the foreground operations, it actually uses the same physical thread to call the background operation. So, background operations may cause a larger deviation of the cycle time from the targeted cycle time. Use the background characteristics with care, and only for an exact match to the usage scenario.
All Java classes have constructors. Some Java
classes that chose to not implement any constructor implicitly have a
default constructor. The constructor for a driver class is responsible
for obtaining the driver context by calling DriverContext.getContext().
It is also responsible for reading the any necessary configuration
parameter required for the load and store them in instance variables.
As instance variables are not shared across instances, they do not
incur any concurrency or synchronization frequently accessing them.
There are two special annotations that you can add for specific one-time tasks. These are @OnceBefore and @OnceAfter. An @OnceBefore operation gets called by global thread 0 of the driver only once and before any other driver thread gets instantiated. This will guarantee that no operation is called while the @OnceBefore operation is running.
The second annotation type is @OnceAfter. The @OnceAfter operation is called only once after the run by thread 0 after all other threads have terminated. Note that both run on thread 0. So any state saved by the @OnceBefore operation can be used by the @OnceAfter operation. This functionality is frequently used for auditing purposes, such as counting the number of records in the database before and after the run to ensure that the benchmark run really interacted with the database. These operations are not designed to run lengthy tasks such as data loading.
The operation is where you will implement the client logic for interacting with the servers. This usually includes preparing data for submitting to the server, invoking the server one or more times, and optionally validating the result from the server. As the driver instance is never concurrently accessed, the operation implementation need not be thread safe. Just implement the client interaction with the server and the driver framework should take care of most tasks.