| Module: | Driver Framework |
|---|
For long running, monitored benchmarks, it is desirable to learn early on whether the run meets the expectations. Faban has a runtime reporting feature that calculates the results of the run based on a time period immediately preceding the report as well as the overall results up to the time of the report. The metrics reported in the runtime report generally converge towards the final report in a steady (non-load varying) run.
Currently, if enabled, the runtime metrics are reported into the run log, one entry each time, for each benchmark driver type. The reports lend themselves very well to runtime graphing. This is, however, not available at this time.
Once enabled and configured, Faban will write a report as in the following example to the run log for each time period:
| 65.00s
- WebDriver: MyOperation1/MyOperation2/MyOperation3 CThru=4.799/3.799/2.200 OThru=4.804/3.803/2.002 CErr=0.000/0.000/0.000 CResp=0.009/0.001/0.002 OResp=0.009/0.001/0.001 CSD=0.030/0.000/0.002 OSD=0.021/0.000/0.000 C90%Resp=0.010/0.010/0.010 O90%Resp=0.010/0.010/0.010 |
The following describes each of the output fields:
65.00s - WebDriver: MyOperation1/MyOperation2/MyOperation3
The displayed stats are displayed for each operation and are labeled as follows:
| Cthru | current thruput (as of the last interval period) | |
| Othru | overall measured steady state thruput, not available before steady state | |
| CErr | current error rate | |
| CResp | current response time (avg of the last interval period) | |
| OResp | overall avg steady state response time, 0 before steady state | |
| CSD | current standard deviation of response time | |
| OSD | overall standard deviation of response time, not available before steady state | |
| C90%Resp | current 90th% response time as of last interval period, not available before steady state | |
| O90%Resp | overall steady state 90th% response time, not available before steady state |
To enable runtime reporting, simply set the runtime stats to true in your runConfig part of the configuration file:
| <fa:runConfig
definition="sample.driver.WebDriver" xmlns="http://faban.sunsource.net/ns/fabandriver" xmlns:fa="http://faban.sunsource.net/ns/faban" xmlns:fh="http://faban.sunsource.net/ns/fabanharness"> .... <runtimeStats enabled="true"> <interval>30</interval> </runtimeStats> .... </fa:runConfig> |
Once the run ramp up starts, a stats record will be written to the log every number of seconds specified by the interval.
There are no restrictions or limitations for enabling the runtime reporting. However, there may be performance implications. With the runtime reporting disabled, there is absolutely no crosstalk between the agents. Enabling the runtime reporting causes each stat from each thread to be read for each of the given interval. This will create some crosstalk, not much - but may be enough to impact scalability of large scale runs with very many driver agents. increasing the inteval time should reduce the impact.