Metrics, Data Collection, and Reporting

The key to running any performance benchmark is - measuring performance. In order to measure performance, timing data for each individual operation needs to be collected, organized, and reported. The Faban driver framework has a rich set of data collection and reporting facilities. And the good news is - most or all of it happens automatically.

General Metrics

Collecting and reporting of general metrics happens automatically. For automatic timing, the operation does not need to deal with data collection at all. With manual timing, the operation will need to call DriverContext.recordTime() and optionally, DriverContext.pauseTime(). All data collection is then done automatically by the driver framework.

The Faban driver framework automatically collects the following data:

Count of successful operations
Count of failed operations
Response times of each operation
Cycle time of each operation
Targeted cycle time of each operation

After the run, the master will automatically postprocess the data and generate a summary report and a detail report. The summary report is in XML format and can be easily read and reformatted to fit specific reporting needs. However, Faban provides a stylesheet for formatting the summary report in a web browser. The detail report is in a text format compatible with FenXi, the tool we use for analysis and graph generation. It is also extremely readable by a human eye.

CustomMetrics

While most of the standard data collection and reporting is handled by Faban, it is not uncommon for the driver to collect additional data. The Faban driver framework provides a mechanism to collect such additional metrics by attaching a CustomMetrics implementation to the DriverContext. This is typically done in the driver's constructor. There are two method signatures for attaching the custom metrics:

DriverContext.attachMetrics(CustomMetrics)
Attaches the CustomMetrics titled "Miscellaneous Statistics." This is the same as calling attachMetrics("Miscellaneous Statistics", CustomMetrics).
DriverContext.attachMetrics(String, CustomMetrics)
Attaches the CustomMetrics with a given title. Only one CustomMetrics can be attached using a given title. Each metrics is reported in its own table in the summary report with the title used for attaching. The metrics are reported in the sequence they were attached.

Each CustomMetrics can represent a multitude of data the particular driver needs to keep track. At reporting time, each of the line items are reported using a CustomMetrics.Element instance. The results are then reported in the given section of the summary report identified by the title.

CustomTableMetrics

Besides the standard reporting format for CustomMetrics, some workloads may need to report arbitrary tables in the summary report. This can be achieved by attaching CustomTableMetrics with DriverContext.attachMetrics(String, CustomTableMetrics). The CustomTableMetrics is an open interface used for collecting stats just like CustomMetrics while enforcing the result to be a TableModel. The TableModel will then be used for reporting the result table.

While very flexible, the CustomTableMetrics has some limitation. It does not have the ability to render a run passed or failed as the TableModel does not have a way to express such results explicitly. It is merely used for reporting that does not impact the pass/fail status of the run.

Implementing CustomMetrics

Implementing the CustomMetrics is rather straightforward. In Java terms, it is done by creating a class implementing the CustomMetrics interface. The class can have any name, be in any package, and can have any kind of constructor. The driver will need to instantiate the CustomMetrics at construction time.

A typical CustomMetrics implementation will keep the collected data in its instance variables. The driver needs to update the data during the run. The driver can both check the time by calling DriverContext.getTime() and check whether it is in the steady state by calling DriverContext.isTxSteadyState() methods to know whether or not to record data.

The form of keeping data is not prescribed and can vary from individual instance variables to arrays to collections, etc. Since a CustomMetrics instance is attached to each driver instance, thread, and simulated user, it needs to be aggregated at the run. The implementor will need to implement the methods add() and clone() which are used for aggregation which means he/she will need to know the characteristics of the data. Counts and times are generally added directly, min and max values are compared and selected. Averages and variances will need some more complex processing, accordingly.

Once all CustomMetrics instances are aggregated, the driver framework will call getResults() on the aggregated instance to obtain the final results. This method returns an array of Elements, each describing the metric to report. At the minimum, the description field must not be null. This is for just reporting metrics that have no effect on the pass or fail status of the run. If the metric has an impact on the pass or fail status of the run, the passed field need to be set. The target and allowedDeviation fields are optional and are useful to report such deviations. All fields except passed are of type java.lang.String to allow for flexibility of data type and formatting. The data will be represented in the summary report exactly the same way it is formatted in the string.

Implementing CustomTableMetrics

Similar to the CustomMetrics implementation, we subclass the CustomTableMetrics class and are allowed to do all kinds of data collection and keeping the data in the in the instance. It also gets aggregated similar to CustomMetrics. The only difference is that at reporting time, it has to produce a TableModel instead of an array of CustomMetrics.Element objects. This table will be placed in the summary report with the given title used when attaching the CustomTableMetrics.

ContentSizeStats

The com.sun.faban.driver.util.ContentSizeStats is a convenient CustomMetrics implementation provided for collecting and averaging content sizes. It is one of the simplest cases of collecting custom metrics and is provided as part of the fhb facility and and some Faban sample code. If you just want to know the average content size, ContentSizeStats is right for you. To use it, just do the followings:

Declare an instance variable to hold the reference to ContentSizeStats.
```
    ContentSizeStats contentStats = null;
```

In the driver constructor, just create the ContentSizeStats as follows:

 ctx = DriverContext.getContext();
 contentStats = new ContentSizeStats(ctx.getOperationCount());
 ctx.attachMetrics(contentStats);

At the end of each operation, add up the content size:

    if (ctx.isTxSteadyState())
     contentStats.sumContentSize[ctx.getOperationId()] += contentSize;

You'll find the content sizes for each operation reported in the miscellaneous stats section of the summary result.

Accessing General Results

In many cases the results need to be calculated based on the run results. For instance, the percentage of cancelled purchase orders (custom metrics) is calculated from the number of cancelled purchase orders and the number of total purchase orders (run results). So the getResults() method implementation will need to have a way to know the number of purchase order transactions during steady state.

The Result class is provided for this purpose. It provides the CustomMetrics.getResults() with access to the run results and metrics. It is only available to the getResults() method at reporting time as this gets called after the results are established. The getResults() method may access the Result object by calling Result.getInstance() inside the method implementation.