Welcome to simple_benchmark’s documentation!

Installation

Using pip:

pip install simple_benchmark

Or the manual installation using git:

git clone https://github.com/MSeifert04/simple_benchmark.git
cd simple_benchmark
python setup.py install

Getting started

Suppose you want to compare how NumPys sum and Pythons sum perform on lists of different sizes:

>>> from simple_benchmark import benchmark
>>> import numpy as np
>>> funcs = [sum, np.sum]
>>> arguments = {i: [1]*i for i in [1, 10, 100, 1000, 10000, 100000]}
>>> argument_name = 'list size'
>>> aliases = {sum: 'Python sum', np.sum: 'NumPy sum'}
>>> b = benchmark(funcs, arguments, argument_name, function_aliases=aliases)

The result can be visualized with pandas (needs to be installed):

>>> b
        NumPy sum    Python sum
1        0.000003  1.032715e-07
10       0.000004  1.569619e-07
100      0.000007  7.155641e-07
1000     0.000042  6.153851e-06
10000    0.000382  6.030774e-05
100000   0.004034  6.026672e-04

Or with matplotlib (has to be installed too):

>>> b.plot()
_images/sum_example.png

Warning

This package is under active development. API changes are very likely.

This package aims to give an easy way to benchmark several functions for different inputs and provide ways to visualize the benchmark results.

To utilize the full features (visualization and post-processing) you need to install the optional dependencies:

  • NumPy
  • Pandas
  • Matplotlib
simple_benchmark.benchmark(funcs, arguments, argument_name='', warmups=None, time_per_benchmark=0.1, function_aliases=None, estimator=<built-in function min>)[source]

Create a benchmark suite for different functions and for different arguments.

Parameters:
  • funcs (iterable of callables) – The functions to benchmark.
  • arguments (dict) – A dictionary containing where the key represents the reported value (for example an integer representing the list size) as key and the argument for the functions (for example the list) as value. In case you want to plot the result it should be sorted and ordered (e.g. an collections.OrderedDict or a plain dict if you are using Python 3.7 or later).
  • argument_name (str, optional) – The name of the reported value. For example if the arguments represent list sizes this could be “size of the list”. Default is an empty string.
  • warmups (None or iterable of callables, optional) – If not None it specifies the callables that need a warmup call before being timed. That is so, that caches can be filled or jitters to kick in. Default is None.
  • time_per_benchmark (float, optional) – Each benchmark should take approximately this value in seconds. However the value is ignored for functions that take very little time or very long. Default is 0.1 (seconds).
  • function_aliases (None or dict, optional) – If not None it should be a dictionary containing the function as key and the name of the function as value. The value will be used in the final reports and plots. Default is None.
  • estimator (callable, optional) – Each function is called with each argument multiple times and each timing is recorded. The benchmark_estimator (by default min()) is used to reduce this list of timings to one final value. The minimum is generally a good way to estimate how fast a function can run (see also the discussion in timeit.Timer.repeat()).
Returns:

benchmark – The result of the benchmarks.

Return type:

BenchmarkResult

Examples

For example to benchmark different sum functions on a Python list.

The setup:

>>> from simple_benchmark import benchmark
>>> import numpy as np
>>> funcs = [sum, np.sum]
>>> arguments = {i: [1]*i for i in [1, 10, 100, 1000, 10000, 100000]}
>>> argument_name = 'list size'
>>> aliases = {sum: 'Python sum', np.sum: 'NumPy sum'}
>>> b = benchmark(funcs, arguments, argument_name, function_aliases=aliases)

Inspecting the results:

>>> b.to_pandas_dataframe()

Plotting the results:

>>> b.plot()
>>> b.plot(relative_to=np.sum)
>>> b.plot_both(relative_to=sum)

It’s also possible to pass multiple arguments to the functions being benchmarked by using MultiArgument:

>>> def func(a, b):
...     return a + b
>>> funcs = [func]
>>> arguments = {10: MultiArgument([10, 10])}
>>> argument_name = "something"
>>> benchmark(funcs, arguments, argument_name)

The multi-argument is simply unpacked when the function is called for that particular benchmark.

simple_benchmark.benchmark_random_array(funcs, sizes, warmups=None, time_per_benchmark=0.1, function_aliases=None)[source]

A shortcut for benchmark() if a random array is wanted.

The arguments arguments and argument_name of the normal constructor are replaced with a simple size argument.

Parameters:
  • funcs (iterable of callables) – The functions to benchmark.
  • sizes (iterable of int) – The different size values for arrays. In case you want to plot the result it should be sorted.
  • warmups (None or iterable of callables, optional) – If not None it specifies the callables that need a warmup call before being timed. That is so, that caches can be filled or jitters to kick in. Default is None.
  • time_per_benchmark (float, optional) – Each benchmark should take approximately this value in seconds. However the value is ignored for functions that take very little time or very long. Default is 0.1 (seconds).
  • function_aliases (None or dict, optional) – If not None it should be a dictionary containing the function as key and the name of the function as value. The value will be used in the final reports and plots. Default is None.
Returns:

benchmark – The result of the benchmarks.

Return type:

BenchmarkResult

Raises:

ImportError – If NumPy isn’t installed.

Examples

In case the arguments are NumPy arrays containing random floats this function allows for a more concise benchmark:

>>> from simple_benchmark import benchmark_random_array
>>> import numpy as np
>>> funcs = [sum, np.sum]
>>> sizes = [i ** 4 for i in range(20)]
>>> aliases = {sum: 'Python sum', np.sum: 'NumPy sum'}
>>> b = benchmark_random_array(funcs, sizes, function_aliases=aliases)
simple_benchmark.benchmark_random_list(funcs, sizes, warmups=None, time_per_benchmark=0.1, function_aliases=None)[source]

A shortcut for benchmark() if a random list is wanted.

The arguments arguments and argument_name of the normal constructor are replaced with a simple size argument.

Parameters:
  • funcs (iterable of callables) – The functions to benchmark.
  • sizes (iterable of int) – The different size values for list. In case you want to plot the result it should be sorted.
  • warmups (None or iterable of callables, optional) – If not None it specifies the callables that need a warmup call before being timed. That is so, that caches can be filled or jitters to kick in. Default is None.
  • time_per_benchmark (float, optional) – Each benchmark should take approximately this value in seconds. However the value is ignored for functions that take very little time or very long. Default is 0.1 (seconds).
  • function_aliases (None or dict, optional) – If not None it should be a dictionary containing the function as key and the name of the function as value. The value will be used in the final reports and plots. Default is None.
Returns:

benchmark – The result of the benchmarks.

Return type:

BenchmarkResult

Examples

In case the arguments are lists containing random floats this function allows for a more concise benchmark:

>>> from simple_benchmark import benchmark_random_list
>>> import numpy as np
>>> funcs = [sum, np.sum]
>>> sizes = [i ** 4 for i in range(20)]
>>> aliases = {sum: 'Python sum', np.sum: 'NumPy sum'}
>>> b = benchmark_random_list(funcs, sizes, function_aliases=aliases)
class simple_benchmark.BenchmarkResult(timings, function_aliases, arguments, argument_name)[source]

A class holding a benchmarking result that provides additional printing and plotting functions.

plot(relative_to=None, ax=None)[source]

Plot the benchmarks, either relative or absolute.

Parameters:
  • relative_to (callable or None, optional) – If None it will plot the absolute timings, otherwise it will use the given relative_to function as reference for the timings.
  • ax (matplotlib.Axes or None, optional) – The axes on which to plot. If None plots on the currently active axes.
Raises:

ImportError – If matplotlib isn’t installed.

plot_both(relative_to)[source]

Plot both the absolute times and the relative time.

Parameters:relative_to (callable or None) – If None it will plot the absolute timings, otherwise it will use the given relative_to function as reference for the timings.
Raises:ImportError – If matplotlib isn’t installed.
plot_difference_percentage(relative_to, ax=None)[source]

Plot the benchmarks relative to one of the benchmarks with percentages on the y-axis.

Parameters:
  • relative_to (callable) – The benchmarks are plotted relative to the timings of the given function.
  • ax (matplotlib.Axes or None, optional) – The axes on which to plot. If None plots on the currently active axes.
Raises:

ImportError – If matplotlib isn’t installed.

to_pandas_dataframe()[source]

Return the timing results as pandas Dataframe. This is the preferred way of accessing the text form of the timings.

Returns:The timings as DataFrame.
Return type:pandas.DataFrame
Raises:ImportError – If pandas isn’t installed.
class simple_benchmark.MultiArgument[source]

Class that behaves like a tuple but signals to the benchmark that it should pass multiple arguments to the function to benchmark.

Indices and tables