The Tucson Timing Protocol

The Tucson Timing Protocol (TTP) is a computer software protocol for measuring various aspects of a computer program as it executes. The most important thing it measures is the running time of that program.

This protocol arose out of the study of "ergalic metrology". Metrology is "the science of measurement" (wikipedia article). Ergalics is "the science of computational tools and of computation itself." So ergalic metrology is "the science of measuring computational tools and computation itself".

There are currently two versions of the Tucson Timing Protocol, each described in a paper.

Tucson Timing Protocol Version One (TTPv1)

TTPv1 is described in the following conference article.

Sabah Currim, Richard T. Snodgrass, Young-Kyoon Suh, Rui Zhang, Matthew Wong, and Cheng Yi, "DBMS Metrology: Measuring Query Time," Proceedings of the SIGMOD International Conference, pp. 421-432, New York City, June 2013. (pdf).

For each query and each desired cardinality (termed a "query at cardinality" or Q@C), TTPv1 executes the query at that cardinality ten times, collecting all sorts of information about each query execution. The protocol then performs the following steps on that data.
Step 1: Perform a dozen sanity checks to ensure the validity of the data
Step 2: Perhaps drop some query executions via seven clearly motivated predicates
Step 3: Perhaps drop some entire Q@Cs, again via three clearly motivated predicates
Step 4: For those Q@Cs that remain, for each compute a single measured time by a carefully-justified formula over the underlying measures (DBMS user time, DBMS system time, and a portion of IOWait time due to DBMS user time, via a regression equation), of the remaining query executions, and
Step 5: Perform three post-analysis sanity checks.

Tucson Timing Protocol Version Two (TTPv2)

TTPv2 is described in the following journal article.

Sabah Currim, Richard T. Snodgrass, Young-Kyoon Suh, and Rui Zhang, "DBMS Metrology: Measuring Query Time," ACM Transactions on Database Systems, to appear, 41+8 pages, 2016.

For each query at cardinality (Q@C), TTPv2 executes the query at that cardinality ten times, collecting all sorts of information about each query execution. TTPv2 utilizes an additional thread that gathers taskstats for those processes that stopped during the timing; TTPv1 had no way of recording measures for such processes.

The protocol then performs the following steps on that data.
Step 1: Perform sanity checks (three classes comprising 25 sanity checks) to ensure the validity of the data
Step 2: Perhaps drop some query executions via a total of 13 clearly motivated predicates
Step 3: Perhaps drop some entire Q@Cs, again via a total of four clearly motivated predicates
Step 4: For those Q@Cs that remain, for each compute a single measured time by a carefully-justified formula over the underlying measures of query BlockIO ticks, IOWait ticks, query user time, query system time, and query IO Wait time, the last calculated from query BLockIO ticks and overall IOWait ticks, of the remaining query executions, and
Step 5: Perform six post-analysis sanity checks.

TTPv2 is a mature, robust, self-checking protocol that provides a more precise and more accurate timing of the query, reducing variance from 2.2% for TTPv1 to 2.1% for calculated time. TTPv2 also discards only 4.0% of original Q@Cs, while TTPv1 discarded 24% of the Q@Cs, in a biased way, in that longer-running queries were much more likely to be discarded, because of the increased chance of ephemeral processes, which are handled better now.

AZDBLab supports TTPv2.