Welcome to simple_benchmark’s documentation!¶
Installation¶
Using pip
:
pip install simple_benchmark
Or the manual installation using git
:
git clone https://github.com/MSeifert04/simple_benchmark.git
cd simple_benchmark
python setup.py install
Getting started¶
Suppose you want to compare how NumPys sum and Pythons sum perform on lists of different sizes:
>>> from simple_benchmark import benchmark
>>> import numpy as np
>>> funcs = [sum, np.sum]
>>> arguments = {i: [1]*i for i in [1, 10, 100, 1000, 10000, 100000]}
>>> argument_name = 'list size'
>>> aliases = {sum: 'Python sum', np.sum: 'NumPy sum'}
>>> b = benchmark(funcs, arguments, argument_name, function_aliases=aliases)
The result can be visualized with pandas
(needs to be installed):
>>> b
NumPy sum Python sum
1 0.000003 1.032715e07
10 0.000004 1.569619e07
100 0.000007 7.155641e07
1000 0.000042 6.153851e06
10000 0.000382 6.030774e05
100000 0.004034 6.026672e04
Or with matplotlib
(has to be installed too):
>>> b.plot()
Warning
This package is under active development. API changes are very likely.
This package aims to give an easy way to benchmark several functions for different inputs and provide ways to visualize the benchmark results.
To utilize the full features (visualization and postprocessing) you need to install the optional dependencies:
 NumPy
 Pandas
 Matplotlib

simple_benchmark.
benchmark
(funcs, arguments, argument_name='', warmups=None, time_per_benchmark=0.1, function_aliases=None, estimator=<builtin function min>)[source]¶ Create a benchmark suite for different functions and for different arguments.
Parameters:  funcs (iterable of callables) – The functions to benchmark.
 arguments (dict) – A dictionary containing where the key represents the reported value
(for example an integer representing the list size) as key and the argument
for the functions (for example the list) as value.
In case you want to plot the result it should be sorted and ordered
(e.g. an
collections.OrderedDict
or a plain dict if you are using Python 3.7 or later).  argument_name (str, optional) – The name of the reported value. For example if the arguments represent list sizes this could be “size of the list”. Default is an empty string.
 warmups (None or iterable of callables, optional) – If not None it specifies the callables that need a warmup call before being timed. That is so, that caches can be filled or jitters to kick in. Default is None.
 time_per_benchmark (float, optional) – Each benchmark should take approximately this value in seconds. However the value is ignored for functions that take very little time or very long. Default is 0.1 (seconds).
 function_aliases (None or dict, optional) – If not None it should be a dictionary containing the function as key and the name of the function as value. The value will be used in the final reports and plots. Default is None.
 estimator (callable, optional) – Each function is called with each argument multiple times and each
timing is recorded. The benchmark_estimator (by default
min()
) is used to reduce this list of timings to one final value. The minimum is generally a good way to estimate how fast a function can run (see also the discussion intimeit.Timer.repeat()
).
Returns: benchmark – The result of the benchmarks.
Return type: Examples
For example to benchmark different sum functions on a Python list.
The setup:
>>> from simple_benchmark import benchmark >>> import numpy as np >>> funcs = [sum, np.sum] >>> arguments = {i: [1]*i for i in [1, 10, 100, 1000, 10000, 100000]} >>> argument_name = 'list size' >>> aliases = {sum: 'Python sum', np.sum: 'NumPy sum'} >>> b = benchmark(funcs, arguments, argument_name, function_aliases=aliases)
Inspecting the results:
>>> b.to_pandas_dataframe()
Plotting the results:
>>> b.plot() >>> b.plot(relative_to=np.sum) >>> b.plot_both(relative_to=sum)
It’s also possible to pass multiple arguments to the functions being benchmarked by using
MultiArgument
:>>> def func(a, b): ... return a + b >>> funcs = [func] >>> arguments = {10: MultiArgument([10, 10])} >>> argument_name = "something" >>> benchmark(funcs, arguments, argument_name)
The multiargument is simply unpacked when the function is called for that particular benchmark.

simple_benchmark.
benchmark_random_array
(funcs, sizes, warmups=None, time_per_benchmark=0.1, function_aliases=None)[source]¶ A shortcut for
benchmark()
if a random array is wanted.The arguments arguments and argument_name of the normal constructor are replaced with a simple size argument.
Parameters:  funcs (iterable of callables) – The functions to benchmark.
 sizes (iterable of int) – The different size values for arrays. In case you want to plot the result it should be sorted.
 warmups (None or iterable of callables, optional) – If not None it specifies the callables that need a warmup call before being timed. That is so, that caches can be filled or jitters to kick in. Default is None.
 time_per_benchmark (float, optional) – Each benchmark should take approximately this value in seconds. However the value is ignored for functions that take very little time or very long. Default is 0.1 (seconds).
 function_aliases (None or dict, optional) – If not None it should be a dictionary containing the function as key and the name of the function as value. The value will be used in the final reports and plots. Default is None.
Returns: benchmark – The result of the benchmarks.
Return type: Raises: ImportError
– If NumPy isn’t installed.Examples
In case the arguments are NumPy arrays containing random floats this function allows for a more concise benchmark:
>>> from simple_benchmark import benchmark_random_array >>> import numpy as np >>> funcs = [sum, np.sum] >>> sizes = [i ** 4 for i in range(20)] >>> aliases = {sum: 'Python sum', np.sum: 'NumPy sum'} >>> b = benchmark_random_array(funcs, sizes, function_aliases=aliases)
See also

simple_benchmark.
benchmark_random_list
(funcs, sizes, warmups=None, time_per_benchmark=0.1, function_aliases=None)[source]¶ A shortcut for
benchmark()
if a random list is wanted.The arguments arguments and argument_name of the normal constructor are replaced with a simple size argument.
Parameters:  funcs (iterable of callables) – The functions to benchmark.
 sizes (iterable of int) – The different size values for list. In case you want to plot the result it should be sorted.
 warmups (None or iterable of callables, optional) – If not None it specifies the callables that need a warmup call before being timed. That is so, that caches can be filled or jitters to kick in. Default is None.
 time_per_benchmark (float, optional) – Each benchmark should take approximately this value in seconds. However the value is ignored for functions that take very little time or very long. Default is 0.1 (seconds).
 function_aliases (None or dict, optional) – If not None it should be a dictionary containing the function as key and the name of the function as value. The value will be used in the final reports and plots. Default is None.
Returns: benchmark – The result of the benchmarks.
Return type: Examples
In case the arguments are lists containing random floats this function allows for a more concise benchmark:
>>> from simple_benchmark import benchmark_random_list >>> import numpy as np >>> funcs = [sum, np.sum] >>> sizes = [i ** 4 for i in range(20)] >>> aliases = {sum: 'Python sum', np.sum: 'NumPy sum'} >>> b = benchmark_random_list(funcs, sizes, function_aliases=aliases)
See also

class
simple_benchmark.
BenchmarkResult
(timings, function_aliases, arguments, argument_name)[source]¶ A class holding a benchmarking result that provides additional printing and plotting functions.

plot
(relative_to=None, ax=None)[source]¶ Plot the benchmarks, either relative or absolute.
Parameters: Raises: ImportError
– If matplotlib isn’t installed.

plot_both
(relative_to)[source]¶ Plot both the absolute times and the relative time.
Parameters: relative_to (callable or None) – If None it will plot the absolute timings, otherwise it will use the given relative_to function as reference for the timings. Raises: ImportError
– If matplotlib isn’t installed.

plot_difference_percentage
(relative_to, ax=None)[source]¶ Plot the benchmarks relative to one of the benchmarks with percentages on the yaxis.
Parameters: Raises: ImportError
– If matplotlib isn’t installed.

to_pandas_dataframe
()[source]¶ Return the timing results as pandas Dataframe. This is the preferred way of accessing the text form of the timings.
Returns: The timings as DataFrame. Return type: pandas.DataFrame Raises: ImportError
– If pandas isn’t installed.


class
simple_benchmark.
MultiArgument
[source]¶ Class that behaves like a tuple but signals to the benchmark that it should pass multiple arguments to the function to benchmark.