Simple macro for measuring algorithm's running time

Recently I'm spending a lot of time on my master's thesis. I'm working on algorithm for automatic number plates recognition using image segmentation. I'm trying to achieve high performance, so I need to measure execution time of my algorithms.
A few weeks ago I've written about google benchmark [1]. It's quite powerful library, and it's easy to use even in simple case, but sometimes we don't want to depend on an external library. So is it in my case too.
I've created simple macro for measuring running time:

#include <chrono>

#define MEASURE_TIME(unit, ...)    \
  [&] {         \
    using namespace std::chrono;     \
    auto start = high_resolution_clock::now ();    \
    __VA_ARGS__;       \
    auto time = high_resolution_clock::now () - start;   \
    return duration_cast<unit> (time).count ();    \
  } ();

Arguments:

• unit - std::chrono::duration time interval. You can simply pass defined in standard library types, e.g. predefined types:
  • std::chrono::nanoseconds
  • std::chrono::microseconds
  • std::chrono::milliseconds
  • std::chrono::seconds
  • std::chrono::minutes
  • std::chrono::hours
• ... - code for measurement

And example usage:

  int value = 0;

  auto duration = MEASURE_TIME(std::chrono::milliseconds, 
    int arg0, arg1;
    arg0 = run_time_consuming_algorithm ();
    arg1 = run_another_one(arg0);
    value = run_third_algorithm(arg1);
  );

  cout << "Execution time: " << duration << std::endl
       << "Computed value: " << value;

Note, that you can use earlier declared values (value variable, in my case), because all the values are captured by reference in lambda.

Feel free to use it ;)

Links
[1] http://cookandcommit.blogspot.com/2014/09/tiny-c-benchmark-framework_29.html 

Tiny C++ Benchmark Framework

Lately I had to improve a little some of my algorithms. I wanted to check, how changes affect to time of algorithms execution. So I started to search benchmark framework for C++. After a while, I have found quite small, but powerful framework - google benchmark [1].

Simple usage

The simplest usage of google benchmark is shown below:

#include <benchmark/benchmark.h>
#include <algorithm>

int complex_computation(int n)
{
  if (n == 0) 
    return 0;

  unsigned int a = 1, b = 1;

  for (unsigned int i=0; i < n-1; i++) 
    {
      std::swap(a, b);
      b += a;
    }

  return b;
}

static void BM_Fibonacci(benchmark::State& state)
{
  int ret;

  while (state.KeepRunning())
    ret |= complex_computation(500);

  CHECK(ret != 0);
}

BENCHMARK(BM_Fibonacci);

int main(int argc, const char* argv[]) 
{
  benchmark::Initialize(&argc, argv);
  benchmark::RunSpecifiedBenchmarks();

  return 0;
}

At the beginning, we have to define function, which will be measured (complex_computation, in my case). Next, we're defining benchmark method - BM_Fibonacci. It has one argument - state (I will talk about it later). In while loop we call our method, until benchmark's working. ret variable is used only because of compiler optimizations (we'd like to avoid removing dummy code). Benchmark method should be later registered (using BENCHMARK macro). In main function, framework should be initialized, and after that, we can run all our benchmark methods.

loganek@hf-gcs-computer:~/Documents/blog-benchmark$ g++ benchmark.cpp -std=c++11 -lbenchmark -lpthread -O2 -o benchmark
loganek@hf-gcs-computer:~/Documents/blog-benchmark$ ./benchmark 
Reading /proc/self/cputime_ns failed. Using getrusage().
Benchmarking on 4 X 2701 MHz CPUs
2014/09/29-00:49:29
CPU scaling is enabled: Benchmark timings may be noisy.
DEBUG: Benchmark      Time(ns)    CPU(ns) Iterations
----------------------------------------------------
DEBUG: BM_Fibonacci        208        225    2222788                                  
loganek@hf-gcs-computer:~/Documents/blog-benchmark$ 

As you can see, there is computation time, cpu and iterations count - everything what we need :) Moreover, we can also get a few informations about machine.

Arguments, range of arguments

Snippet shown above is very simple, but this framework offers users much more. We can define a set of arguments for benchmark. Let's remind previous example. We tried to measure fibonacci implementation. But I hardcoded an argument. Sometimes we'd like to measure one method with different arguments. We don't have to define BM_Fibonacci_5, BM_Fibonacci_42, BM_Fibonacci_87 functions for different arguments. Let's look at the improvement of previous code:

static void BM_Fibonacci(benchmark::State& state)
{
  int ret;
  while (state.KeepRunning())
    ret |= complex_computation(state.range_x());
  CHECK(ret != 0);
}

BENCHMARK(BM_Fibonacci)->Arg(5)->Arg(42)->Arg(87);

and the output is also quite different:

loganek@hf-gcs-computer:~/Documents/blog-benchmark$ ./benchmark 
Reading /proc/self/cputime_ns failed. Using getrusage().
Benchmarking on 4 X 2701 MHz CPUs
2014/09/29-01:20:03
CPU scaling is enabled: Benchmark timings may be noisy.
DEBUG: Benchmark         Time(ns)    CPU(ns) Iterations
-------------------------------------------------------
DEBUG: BM_Fibonacci/5           3         19   25832109                                  
DEBUG: BM_Fibonacci/42         18         35   14398500                                  
DEBUG: BM_Fibonacci/87         44         61    8224413   

We've access for results of every argument, which I passed to a benchmark method.
It is also possible to pass range of arguments to our benchmark method.

BENCHMARK(BM_Fibonacci)->Range(0, 1024);

Defined range runs benchmark for min, max in this range, and every power of 8 from this range. So in my case: 0, 1, 8, 64, 512 and 1024.

Multithreading, other features

Google Benchmark framework gives a few other features for developers:

• multithreading
• benchmark templates
• pair of arguments
• custom reporter class

You may see usage of other features in google benchmark example code [2] or on README page[3].

Summary

That's one of first test framework, which I used for now. Previously I wrote simple methods for measurements, but it wasn't so convenient, and in the long run, it's just waste of time. Moreover, benchmark helps me keep in order my tests, so my code is much cleaner than before.
In the future, I'd like to check also other frameworks; I heard about Celero [4] and hayai [5]. But for now, google benchmark fits my needs.

Links
[1] https://github.com/google/benchmark
[2] https://github.com/google/benchmark/blob/master/test/benchmark_test.cc
[3] https://github.com/google/benchmark/blob/master/README.md
[4] https://github.com/DigitalInBlue/Celero
[5] https://github.com/nickbruun/hayai