The Mir Blog

Writing efficient numerical code in D

Mon, 12 Dec 2016 00:00:00 +0000

Introduction

This post gives a brief overview of some of mir-algorithm’s modules, which provide iteration tools allowing convenient and fast processing of multidimensional data. It also presents some examples of usage, to demonstrate how this module can be easily utilized to significantly improve performance of numerical code written in D.

This article assumes the reader has a cursory understanding of the ndslice package. If not, please consult this before returning here.

What does mir-algorithm offer?

mir-algorithm is a suite packed with structures and algorithms for multidimensional processing, amongst which are some of the algorithms often seen in functional style programming, such as map, reduce, etc. There is a good amount of such algorithms already implemented in Phobos, D’s standard library, that are being used with great success for some time now. What is special about mir-algorithm’s variant, is that it is integrated seamlessly with rest of the ndslice package, which makes for elegantly flowing processing pipelines.

LLVM acceleration

One key component that makes code based on mir-algorithm blazingly fast, is to compile it with LDC, the LLVM based D compiler^[1]. Iteration algorithms in ndslice have been specially tailored to help LDC auto-vectorize computation kernels written by the end user, and also to apply unsafe floating point operations, turned on with @fastmath attribute in LDC. For more info on LDC’s optimization techniques, you can check out this great article by Johan Engelen.

Application

In past few months, Mir team has been actively refactoring implementation details of DCV, computer vision library written in D, by replacing critical processing parts written in loops, with mir-algorithm equivalents. With minimal effort, we’ve managed to make code slightly cleaner, but more importantly - a lot faster!

All measuring presented in this post are made using following configuration:

Model Identifier	MacBookPro13,1
CPU	Intel Core i5-6360U @ 2.0 GHz
L3 Cache	4 MB
Memory	8 GB 1867 MHz LPDDR3
OS	macOS Sierra 10.12.1

And here are the highlights from benchmarking comparison between before and after these refactorings:

Algorithm	Previous Runtime [s]	Current Runtime [s]	Speedup [%]
harrisCorners (3x3)	1.62469	0.278579	483
harrisCorners (5x5)	4.40641	0.328159	1242
shiTomasiCorners (3x3)	1.5839	0.223794	607
shiTomasiCorners (5x5)	4.42253	0.297106	1388
extractCorners	3.16413	0.355564	789
gray2rgb	0.441354	0.008918	4849
hsv2rgb	0.433122	0.051392	742
rgb2gray	0.262186	0.031813	724
rgb2hsv	0.365969	0.065572	458
convolution 1D (3)	0.124888	0.067486	85
convolution 1D (5)	0.159795	0.068881	131
convolution 1D (7)	0.206059	0.075361	173
convolution 2D (3x3)	0.767058	0.120216	538
convolution 2D (5x5)	1.94106	0.360809	437
convolution 2D (7x7)	3.71955	0.865524	329
convolution 3D (3x3)	2.09103	0.374006	459
convolution 3D (5x5)	5.54736	1.07421	416
bilateralFilter (3x3)	6.11875	1.77848	244
bilateralFilter (5x5)	16.7187	4.59703	263
calcGradients	2.2448	0.506101	343
calcPartialDerivatives	0.428318	0.14152	202
canny	4.10899	0.75824	441
filterNonMaximum	0.477543	0.038968	1125
nonMaximaSupression	0.588455	0.084436	596
remap	0.22578	0.062089	263
warp	0.235169	0.063821	268

Speedups are massive - average in this set is 676.7%, or if written as multiplier: mean(previous / current) = 7.7x. But as shown below, changes made to the algorithm implementations were trivial. For the complete benchmarking results, please take a look at the pull request implementing these changes.

Disclaimer: Please keep in mind the DCV project is still far too young to be compared against proven computer vision toolkits such as OpenCV. Optimizations done here are showing the power of mir-algorithm, but if you dive into the implementation of these algorithms, you’ll notice most of them are implemented naively, without extensive optimizations. A future post will focus on separable filtering, followed by cache locality improvement.

Examples

We’d like to show few examples of mir-algorithm, but first let’s take a look at the basic principle of replacing loop-based code with pipelines efficiently. And later on we’ll see how it can be used in a bit more complex algorithms.

Basics

So, let’s first examine the basic principle of utilizing iteration algorithms. This principle is also the basis of that DCV refactoring we’ve mentioned. Say we have following code, written plainly in C-style loops:

@fastmath void someFunc(Slice!(Contiguous, [2], float*) image) {
    for(size_t r; r < image.length!0; ++r) {
        for(size_t c; c < image.length!1; ++c) {
            // perform some processing on image pixel at [r, c]
        }
    }
}

This code can be rewritten like so:

import mir.ndslice.algorithm : each;

@fastmath void kernel(ref float e)
{
    // perform that processing from inside those loops
}

image.each!(kernel);

So, instead of writing a function over the whole image, we could utilize each to apply the given kernel function to each pixel. As said in the docs, each iterates eagerly over the data. If processing should be rather evaluated lazily, we could utilize map.

Convolution

To make the example more concrete, let’s examine how we would implement classic image convolution with these algorithms. We’ll write classic, C-style implementation, and its analogue with mir-algorithm. We will wrap both variants with @fastmath attribute, to be as fair as possible. Here is the most trivial C-style implementation:

@fastmath
void convLoop
(
    Slice!(Contiguous, [2], float*) input,
    Slice!(Contiguous, [2], float*) output,
    Slice!(Contiguous, [2], float*) kernel
)
{
    auto kr = kernel.length!0; // kernel row size
    auto kc = kernel.length!1; // kernel column size
    foreach (r; 0 .. output.length!0)
        foreach (c; 0 .. output.length!1)
        {
            // take window to input at given pixel coordinate
            Slice!(Canonical, [2], float*) window = input[r .. r + kr, c .. c + kc];

            // calculate result for current pixel
            float v = 0.0f;
            foreach (cr; 0 .. kr)
                foreach (cc; 0 .. kc)
                    v += window[cr, cc] * kernel[cr, cc];
            output[r, c] = v;
        }
}

Now let’s examine how this would be implemented using mir-algorithm:

static @fastmath float kapply(float v, float e, float k) @safe @nogc nothrow pure
{
    return v + (e * k);
}

void convAlgorithm
(
    Slice!(Contiguous, [2], float*) input,
    Slice!(Contiguous, [2], float*) output,
    Slice!(Contiguous, [2], float*) kernel
)
{
    import mir.ndslice.algorithm : reduce;
    import mir.ndslice.topology: windows, map;

    auto mapping = input
        // look at each pixel through kernel-sized window
        .windows(kernel.shape)
        // map each window to resulting pixel using convolution function
        .map!((window) {
            return reduce!(kapply)(0.0f, window, kernel);
        });

    // assign mapped results to the output buffer.
    output[] = mapping[];
}

The pipeline version replaces two double loops with a few magic calls:

windows: Convenient selector, allows us to look at each pixel through kernel-sized window. It is effectively replacing first two loops in c-style function, automatically giving us the window slice.
map: mapping multidimensional slice by given lambda.
reduce: apply reduce algorithm on each element of the window, multiplying it with convolution kernel (mask) values. This is replacing third and fourth loop from first function. This could also be the key for performance improvement, since its well suited to be auto-vectorized by LLVM.

Lazy evaluation?

As it is previously noted, map is evaluated lazily. At the end of the convAlgorithm function, we are evaluating the mapping function to the data, and assigning resulting values to the output buffer. If instead we had needed lazy evaluated convolution, we could have just returned mapping value from the function, so we could evaluate it lazily afterwards.

Comparison

Complete benchmark source is located in the Mir benchmark section. Let’s compile this program, and make a comparison:

dub run --build=release-nobounds --compiler=ldmd2 --single convolution.d

Output is:

Running ./convolution
                     loops = 1 sec, 649 ms, 99 μs, and 6 hnsecs
     mir.ndslice.algorithm = 159 ms, 392 μs, and 9 hnsecs

So, for as little effort as this, we get ~10x speedup! And hopefully many would agree that the variant written with mir-algorithm is much cleaner and less error prone!

Zipped tensors

D offers two ways to zip multiple ranges: zip and lockstep. Both of these functions provide easy-to-use syntax and are very useful for common usage. Unfortunately, those are not so performance rewarding for multidimensional processing with ndslice. Instead, mir.ndslice.topology.zip should be used. This function zips two slices of the same structure (shape and strides). It offers same dimension-wise range interface as Slice does (and by that is compatible with the rest of the mir-algorithm), and can perform faster than general purpose utilities Phobos’ zip and lockstep.

To explain this concept further, let’s examine following function:

@fastmath
void binarizationLockstep
(
    Slice!(Contiguous, [2], float*) input,
    float threshold,
    Slice!(Contiguous, [2], float*) output
)
in
{
    assert(output.shape == input.shape);
}
body
{
    import mir.ndslice.topology : flattened;
    import std.range : lockstep;
    foreach(i, ref o; lockstep(input.flattened, output.flattened))
    {
        o = (i > threshold) ? float(1) : float(0);
    }
}

So, this is a most basic binarization, based on given threshold. To zip those matrices using std.range.lockstep, we first had to flatten them with mir.ndslice.topology.flattened, to construct a vector, i.e. one-dimensional array, so it can be inserted into lockstep, as classic D range.

Let’s replace std.range.lockstep with mir.ndslice.topology.zip, and see how that affects the implementation:

@fastmath @nogc nothrow @safe
void binarizationZip
(
    Slice!(Contiguous, [2], float*) input,
    float threshold,
    Slice!(Contiguous, [2], float*) output
)
{
    import mir.ndslice.algorithm : each;
    import mir.ndslice.topology : zip;

    zip(input, output).each!( (z) {
        z.b = z.a > threshold ? 1.0f : 0.0f;
    });
}

This one looks pretty much the same as previous one, with slight difference of utilizing mir.ndslice.toplogy.zip instead of std.range.lockstep. Also, now there’s no need to flatten slices, since mir.ndslice.algorithm.each works on multidimensional data. But if you take one more look at the signature of these two functions you’ll notice a difference in annotated attibutes:

@fastmath
void binarizationLockstep(...);

@fastmath @nogc nothrow @safe
void binarizationZip(...);

std.range.lockstep is not nothrow, not @nogc, and not @safe, but mir.ndslice.topology.zip is!

Now, let’s compare the impact of these changes on performance by running the benchmark program. The results are as follows:

Running ./binarization
                  lockstep = 169 ms, 871 μs, and 4 hnsecs
                       zip = 39 ms, 105 μs, and 9 hnsecs

So, mir.ndslice.topology.zip gives us about 4.5x speedup. Also it is important to note this gives us interface compatible with ndslice iteration algorithms, and also gives us freedom to write nothrow @nogc @safe code!

Conclusion

In these two examples we’ve achieved some nice performance improvements with very little effort by using mir-algorithm suite. We have also seen the improvement could be even better if mir-algorithm solutions are applied to more complex code you might encounter in numerical library such as DCV. We would argue that every newcomer to D, having an interest in numerical computing, should take a close look at ndslice and its submodules. And, we hope this post will inspire people to give it a spin, so we would have some more projects built on top of it, growing our young scientific ecosystem in D!

Acknowledgements

Thanks to Ilya Yaroshenko, Sebastian Wilzbach, Andrei Alexandrescu and Johan Engelen for helping out with truly informative reviews!

[1] Mir works with LDC compilers of version 1.1.0 beta 5 and later.

Numeric age for D: Mir GLAS is faster than OpenBLAS and Eigen

Fri, 23 Sep 2016 00:00:00 +0000

This post presents performance benchmarks for general matrix-matrix multiplication between Mir GLAS, OpenBLAS, Eigen, and two closed source BLAS implementations from Intel and Apple.

OpenBLAS is the default BLAS implementation for most numeric and scientific projects, for example the Julia Programing Language and NumPy. The OpenBLAS Haswell computation kernels were written in assembler.

Mir is an LLVM-Accelerated Generic Numerical Library for Science and Machine Learning. It requires LDC (LLVM D Compiler) for compilation. Mir GLAS (Generic Linear Algebra Subprograms) has a single generic kernel for all CPU targets, all floating point types, and all complex types. It is written completely in D, without any assembler blocks. In addition, Mir GLAS Level 3 kernels are not unrolled and produce tiny binary code, so they put less pressure on the instruction cache in large applications.

Mir GLAS is truly generic comparing with C++ Eigen. To add a new architecture or target, an engineer just needs to extend one small GLAS configuration file. As of October 2016 configurations are available for the X87, SSE2, AVX, and AVX2 instruction sets.

Machine and software

CPU	2.2 GHz Core i7 (I7-4770HQ)
L3 Cache	6 MB
RAM	16 GB of 1600 MHz DDR3L SDRAM
Model Identifier	MacBookPro11,2
OS	OS X 10.11.6
Mir GLAS	0.18.0, single thread
OpenBLAS	0.2.18, single thread
Eigen	3.3-rc1, single thread (sequential configurations)
Intel MKL	2017.0.098, single thread (sequential configurations)
Apple Accelerate	OS X 10.11.6, single thread (sequential configurations)

Source code

The benchmark source code can be found here. It contains Mir vs a CBLAS implementation benchmark.

Mir GLAS has a native mir.ndslice interface. mir.ndslice is a development version of std.experimental.ndslice, which provides an N-dimensional equivalent of D’s built-in array slicing. GLAS uses Slice!(2, T*) for matrix representation. It is a plain structure composed of two lengths, two strides, and a pointer type of T*. GLAS calling convention can be easily used in any programming language with C ABI support.

// Performs: c := alpha a x b + beta c
gemm(alpha, a, b, beta, c);

On the other hand, CBLAS interface is unwieldy

void cblas_sgemm (
	const CBLAS_LAYOUT layout,
	const CBLAS_TRANSPOSE TransA,
	const CBLAS_TRANSPOSE TransB,
	const int M,
	const int N,
	const int K,
	const float alpha,
	const float *A,
	const int lda,
	const float *B,
	const int ldb,
	const float beta,
	float *C,
	const int ldc)

Environment variables to set single thread for cblas

openBLAS	OPENBLAS_NUM_THREADS=1
Accelerate (Apple)	VECLIB_MAXIMUM_THREADS=1
Intel MKL	MKL_NUM_THREADS=1

OpenBLAS and Intel MKL have sequential configurations. Sequential configuration is preferred for benchmarks.

Building Eigen

Eigen should be built with EIGEN_TEST_AVX and EIGEN_TEST_FMA flags:

mkdir build_dir
cd build_dir
cmake -DCMAKE_BUILD_TYPE=Release -DEIGEN_TEST_AVX=ON -DEIGEN_TEST_FMA=ON ..
make blas

Eigen 3.3-rc1 provides the Fortran BLAS interface. Netlib’s CBLAS library can be used for the benchmark to provide CBLAS interface on top of Eigen.

Results

There are four benchmarks, two charts per benchmark. The first chart represents absolute values, the second chart represents normalised values.

single precision numbers
double precision numbers
single precision complex numbers
double precision complex numbers

Higher is better.

Conclusion

Mir GLAS is significantly faster than OpenBLAS and Apple Accelerate for virtually all benchmarks and parameters, two times faster than Eigen and Apple Accelerate for complex matrix multiplication. Mir GLAS average performance equals to Intel MKL, which is the best for Intel CPUs. Due to its simple and generic architecture it can be easily configured for new targets.

Acknowledgements

Andrei Alexandrescu, Martin Nowak, Mike Parker, Johan Engelen, Ali Çehreli, Joseph Rushton Wakeling.

Transformed density rejection sampling

Mon, 22 Aug 2016 03:20:19 +0000

In the first post a short introduction to non-uniform sampling was given. This post explains the Tinflex algorithm, which is implemented in mir.random.flex.

Idea

The main idea of the Flex algorithm is to split the density distribution into multiple intervals and sample from these constructed intervals using the composition method. For each interval a hat and squeeze function can be constructed. Thus with the approximated hat and squeeze function of a selected interval a value $x$ can be sampled using the rejection with inversion method.

For example, with efficiency $\rho = 1.8$ the normal distribution is split into six intervals with different hat and squeeze functions. The efficiency of the sampling can iteratively be improved by splitting the composition into more intervals and thus yielding a better approximation to the density function.

\rho = 1.8

, 6 intervals constructed

\rho = 1.25

, 10 intervals constructed

\rho = 1.1

, 14 intervals constructed

\rho = 1.01

, 44 intervals constructed

In the following we will quickly dive into the essential basics behind the Tinflex algorithm. In case your last math lesson was long ago, feel free to jump directly to the examples and come back when you want to understand the inner workings of the Tinflex algorithm in more details.

Tinflex transformations

Remember that the Tinflex algorithm splits the density distribution into intervals and constructs a hat and squeeze function for each interval. For the approximation of a hat or squeeze function, the Tinflex algorithm constructs linear functions for concave, convex intervals (or both if the interval contains only one inflection point). However as most distributions aren’t concave, but log-concave, Tinflex uses a family of transformations for the probability density function (pdf) as it operates in logspace and can thus “force” most distributions to be concave. The transformations can be grouped in two classes: (1) $c \neq 0$ and (2) $c = 0$ , which will subsequently be analyzed.

$c$ -transformations with $c \neq 0$

The general transformation function is $T(x) = x^c$ , however as its inverse function is only defined in $\mathbb{C}$ , its needs to be restricted to positive numbers only:

\displaystyle T_c(x) = sgn(c) \cdot x^c

With the usual inversion rules and if $c > 0$ we get:

\displaystyle \begin{aligned} y &= x^c \\ log(y) &= log(x^c) \\ log(y) &= c \cdot log(x) \\ \frac{log(y)}{c} &= log(x) \\ exp\left(\frac{log(y)}{c}\right) &= x \end{aligned}

which is $y^{1 / c}$ and if $c < 0$ we obtain:

\displaystyle \begin{aligned} y &= -x^c \\ log(y) &= log(-x^c) \\ log(y) &= -c \cdot log(x) \\ \frac{log(y)}{c} &= -log(x) \\ exp\left(\frac{log(y)}{c}\right) &= -x \end{aligned}

which is $(-y)^{1 / c}$ .

Thus in general the inverse function is defined as:

\displaystyle T_c^{-1}(x) = (sgn(c) \cdot x)^{\frac{1}{c}}

Two examples of $T_c$ transformations can be seen below:

x^2

(blue) and it’s inverse

x^{1/2}

(green)

-x^{-0.5}

(blue) and it’s inverse

(-x)^{1/-0.5}

(green)

Please note that (1) in the Tinflex paper and Mir implementation this case is split into multiple common cases to obtain simpler formulas and thus reduce numerical errors. Furthermore, (2) only with $c = 0$ the transformation is two-sided, thus for $c < 0$ the inverse transformation $T^{-1}(x)$ in $\mathbb{R}$ is only defined for $x < 0$ .

$c$ -transformations with $c = 0$

The special case for $c = 0$ needs to be made as division by zero isn’t defined, however for this case the natural logarithm and it’s inverse the exponential function can be used:

\displaystyle T_c(x) = log(x) \quad T_c^{-1}(x) = exp(x)

Exponential function (green) and it’s inverse the natural log (blue)

Input of Tinflex

Remember that $f(x)$ is the pdf, and is its transformation $\tilde{f}(x)$ is:

\displaystyle \tilde{f}(x) = T_c(f(x))

The Tinflex algorithm expects the log-density, which means that for $c = 0$ , no transformation needs to be applied. However, for $c \neq 0$ the inverse is needed. This mean we first need to apply the inverse $T_c^{-1}(x) = exp(x)$ and then apply the other $T_c$ transformation:

\displaystyle \begin{aligned} \tilde{f}(x) &= T_{c \neq 0}(T_0^{-1}(x))) \\ &= T_{c \neq 0}(exp(x))) \\ &= sgn(c) \cdot exp(x)^c \\ &= sgn(c) \cdot exp(x \cdot c) \end{aligned}

Linear functions

Construction of linear functions

For every interval with function $\tilde{f}$ (black), we define the three linear functions: its left tangent (orange), its right tangent (red), and its secant (blue):

Concave interval

Convex interval

As it can be seen in the graphs of the two major case categories (concave, convex) at least one linear function is majorizing $\tilde{f}$ (e.g. both tangents in the concave case), and at least one linear function is majorized by $\tilde{f}$ (e.g. the secant in the concave case). Botts et. al. (2011) distinguish between eight cases and prove that this observation of finding at least one linear hat and squeeze function is valid as long as there is at most one inflection point in the interval.

Definition of the linear functions

The following definition for linear functions is used:

\displaystyle \tilde{f}(x) = \alpha + \beta \cdot (x - x_0)

Remember that the transformation is applied with $T_c(x)$ and that the hat function majorizes the transformed pdf, whereas the transformed pdf majorizes the squeeze function.

Both the left and right tangent can be defined as follows, where $x_0$ is either $l$ or $r$ :

\displaystyle tan_{x_0}(x) = \tilde{f}(x_0) + \tilde{f}'(x_0) \cdot (x - x_0)

Moreover for the secant we can pick either $\ell$ or $r$ as root point, and thus only the definition of the slope $\beta$ is different:

\displaystyle sec(x) = \tilde{f}(\ell) + \frac{\tilde{f}(r) - \tilde{f}(\ell)}{r - \ell} \cdot (x - \ell)

With these definition we can integrate the linear function to calculate the area below the hat and squeeze function. Furthermore, using the their integral the inverse function for the cumulative density function can be calculated, which is used to map a uniform point $u$ to a point $x$ which is proportional to the density of $h(x)$ . These details can be found in Tinflex paper and in the Mir implementation. For mir.random.flex additional supplementary material is provided which explains the used integrations, inversions and approximations. In the following a more practical side of the Tinflex algorithm will be given by showing two examples for the construction of a random distribution with the Tinflex method.

Example 1: Dagum distribution

Assume you would want to draw random values from a non-common random distribution like e.g. the Dagum distribution which has the following probability density function:

\displaystyle f(x) = \frac{ap}{x} \left( \frac{ (\frac{x}{b})^{ap}}{\left((\frac{x}{a})^a + 1\right)^{p+1}} \right)

For simplicity of this example we fix $a$ , $b$ and $p$ to $1$ and thus obtain:

\displaystyle f(x) = \frac{1}{(x + 1)^2}

For the Tinflex algorithm the pdf function and its first and second derivative need to be given in log-space:

\displaystyle \tilde{f}(x) = log \left( \frac{1}{(x + 1)^2} \right)

Its first and second derivative after $x$ are:

\displaystyle \tilde{f}'(x) = - \frac{2}{x + 1} \quad \tilde{f}''(x) = \frac{2}{(x + 1)^2}

In case you haven’t used the differentiation syntax on Wolfram Alpha, you should familiarize yourself with it. For example, the correctness of the first derivate and second derivative can be quickly checked. Last but not least, the Tinflex algorithm requires that each interval can have at most one inflection point. Hence we need to check the plot of $\tilde{f}(x)$ for inflection points. However as there none if $x > 0$ , we can freely pick any interval as starting intervals and can thus construct the random distribution:

import std.math : log, pow;

alias S = double;

// log-transformed pdf + first two derivatives
auto f0 = (S x) => cast(S) log(1 / pow(x + 1, 2));
auto f1 = (S x) => -2 / (1 + x);
auto f2 = (S x) => 2 / pow(1 + x, 2);

// which c-transformation function to use
S c = 0.5;

// the range of the interval that should be sampled from
// if there are multiple inflection points, more points need to be given
S[] points = [0, 10, S.max];

// target efficiency of hatArea / squeezeArea
S rho = 1.1;

flex(f0, f1, f2, c, points, rho);

Consecutive histogram

Cumulative histogram

We can also have a look under the hood and inspect hat / squeeze plot of the Dagum distribution (hat: red, squeeze: green). For a clearer visibility only the range $[0, 3]$ is shown:

\rho = 1.5

, 4 intervals

\rho = 1.1

, 8 intervals

As it can be seen $\rho$ has a huge impact on the resulting speed, but not on the statistical quality. Similarly the predefined intervals have only a slight impact on the algorithm itself as e.g. they limit the left and right start border. Picking a $c$ -transformation can influence the numerical errors, construction and sampling performance and even the number of intervals. Hence usually a pre-definded transformation like $0$ (the “pure” log transformation) or $1$ is used. However there are some restrictions, for example for unbounded domains $c > -1$ is required, but $c$ also needs to be sufficiently small, so usually $-0.5$ is used.

Example 2: Gompertz distribution

Assume you would want to sample from the Gompertz distribution which has the following probability density function:

\displaystyle f(x) = b \eta e^{bx} \ e^{\eta} \ exp(-\eta e^{bx})

For simplicity, we set $\eta = 0.005$ and $b = 1.5$ and thus get

\displaystyle f(x) = 0.00753759 e^{-0.005 e^{1.5 x} + 1.5 x}

And consequently in log-space:

\displaystyle \tilde{f}(x) = log \left( 0.00753759 e^{-0.005 e^{1.5 x} + 1.5 x} \right)

Its first and second derivative after $x$ are:

\displaystyle \tilde{f}'(x) = 1.5-0.0075 e^{1.5 x} \quad \tilde{f}''(x) = -0.01125 e^{1.5 x}

Note that for log-density also no inflection point exists and thus we can define our sampling procedure:

import std.math : log, pow;

alias S = double;

// log-transformed pdf + first two derivatives
auto f0 = (S x) => cast(S) log( S(0.00753759) * exp(S(-0.005) * exp(S(1.5) * x) + S(1.5) * x));
auto f1 = (S x) => S(1.5) - S(0.0075) * exp(1.5 * x);
auto f2 = (S x) => S(-0.01125) * exp(S(1.5) * x);

// which c-transformation function to use
S c = 1.5;

// the range of the interval that should be sampled from
S[] points = [0, 6, S.max];

// target efficiency of hatArea / squeezeArea
S rho = 1.1;

flex(f0, f1, f2, c, points, rho);

Consecutive histogram

Cumulative histogram

We can also have a look under the hood and inspect hat / squeeze plot of the Gompertz distribution (hat: red, squeeze: green). For a clearer visibility only the range $[0, 6]$ is shown:

\rho = 1.5

, 8 intervals

\rho = 1.1

, 14 intervals

Benchmarks

Constructing intervals

In the following the R Tinflex implementation and the Mir Flex implementation are compared. For the benchmark the time to construct the (Tin)flex intervals with efficiency $\rho$ for $n$ iterations has been measured:

$\rho$	n	Flex	Tinflex
1.1	10K	122 ms	79.473s
1.01	1K	46 ms	21.229s
1.001	1K	169 ms	79.047s

Due to the enormous time consumption of the R implementation, the number of iterations has been reduced from 10K to 1K for higher $\rho$ . With the Mir Flex implementation a performance boost for the construction phase of the factor 500-600 in comparison to the R Tinflex implementation can be observed.

Not only has the D implementation been engineered very carefully with performance in mind, but it also uses several tricks like a BinaryHeap to pick the best interval to split, so that the number of constructed intervals is also lower:

$\rho$	Flex	Tinflex
1.5	8	9
1.1	15	20
1.01	45	49
1.001	141	175
1.0001	452	496
1.00001	1361	1824

The source of this benchmark is available online.

Sampling

A benchmark between multiple existing normal distribution samplers of sampling 10M random values has been made. All existing libraries (dstats, hap, atmosphere) use the Box-Muller transformation to sample random values. Moreover intervals with mir.random.flex with three different efficiencies $\rho$ ( $1.3, 1.1, 1.001$ ) were constructed. The last method is based on the WIP implementation of the Ziggurat algorithm, which is a specialized algorithm for monotone decreasing distributions.

Method	Avg. time	Std.dev. time
boxMuller.naive	850 ms	82 ms
boxMuller.dstats	801 ms	2 ms
boxMuller.hap	3618 ms	4 ms
boxMuller.atmos	934 ms	2 ms
flexNormal.slow	1406 ms	4 ms
flexNormal.medium	1206 ms	1 ms
flexNormal.fast	1124 ms	0 ms
ziggurat	357 ms	5 ms
clang++	2216 ms	6 ms
g++	941 ms	11 ms

While one should look with care at performance benchmarks as statistical quality is usually a higher goal than speed, it still can be seen that (1) investing more time in constructing better approximated intervals is worth the investment if more values need to be drawn and (2) there are specialized algorithm for very common random distributions that of course should be preferred over generalized methods.

However, after all the goal of the Flex algorithm isn’t to create the fastest random method, but a general applicable method that can be used to create samplers for new distributions quickly.

The source of this benchmark is directly available within the Mir library.

Where to go from here

The Tinflex algorithm is available via the Mir library in mir.random.flex. Apart from the examples listed in this blog, more examples are provided as part of Mir and can be conveniently executed with D’s package manager dub. If you want to dive deeper, you can also read the Tinflex paper, browse the Mir implementation or see the supplementary material for the Mir implementation in which the used equations will be explained.

Literature references

Devroye, Luc. “Sample-based non-uniform random variate generation.” Proceedings of the 18th conference on Winter simulation. ACM, 1986.
Hörmann, Wolfgang. “A rejection technique for sampling from T-concave distributions.” ACM Transactions on Mathematical Software (TOMS) 21.2 (1995): 182-193.
Leydold, Josef, et al. “An automatic code generator for nonuniform random variate generation.” Mathematics and Computers in Simulation 62.3 (2003): 405-412.
Botts, Carsten, Wolfgang Hörmann, and Josef Leydold. “Transformed density rejection with inflection points.” Statistics and Computing 23.2 (2013): 251-260.
Hörmann, Wolfgang, Josef Leydold, and Gerhard Derflinger. Automatic nonuniform random variate generation. Springer Science & Business Media, 2013.

An introduction to sampling from non-uniform random distributions

Fri, 19 Aug 2016 01:29:44 +0000

The problem

This post will dive into the topic of sampling of non-uniform random numbers. The problem statement is: given a uniform random generator, sample non-uniform random values.

The following subsections will introduce some of the basic methods of non-uniform random sampling, which are also used by the Tinflex algorithm implemented in Mir.

The inversion method

The underlying idea of non-uniform random sampling is that given an inverse function $F^{-1}$ for the cumulative density function (CDF) of a target density $f(x)$ , random values can be mapped to a distribution.

More visually one can imagine this with the histogram and cumulative histogram of a random distribution:

Exponential distribution

Exponential distribution, cumulative

To sample a variable distributed according to the exponential distribution, a point $y$ on the CDF graph can be sampled and then using the inverse function its matching value $x$ can be obtained, e.g. if $0.4$ is sampled (first brown line) the inverse yields $f^{-1}(0.4) = -log(1 - 0.4) = 0.51$ , for $0.6$ (yellow) it is $0.91$ and for $0.9$ (last brown line) it is $2.30$ , respectively.

Hence, given $F^{-1}$ , values from the density can be sampled by using the given uniform random generator of the interval $[0, 1]$ . For example, for the sampling procedure for the exponential distribution is:

import std.math : log;
import std.random : rndGen, uniform;

S sample(S, RNG, FInv)(ref RNG gen, FInv finv)
{
    S u = uniform!("[]", S)(0, 1);
    return finv(u);
}

auto fInvExp = (S x) => -log(S(1) - x);
sample!double(rndGen, fInvExp)

However there are two big problems of the inversion method: (1) for most densities the inverse CDF is not known or can’t be determined or (2) if it can be determined it’s usually very computationally expensive function (e.g. an iterative numeric approximation needs to be used if no exact form can be found).

Play yourself

All code listed in this blog post is available online and can be conveniently executed:

wget https://raw.githubusercontent.com/wilzbach/flex-paper/master/samplers/inverse_expo.d
dub inverse_expo.d

The plots above were generated with the inverse_expo.d snippet, so give it a try and play with them.

The rejection method

A very popular alternative is the rejection method - often known as acceptance-rejection sampling. It only requires one to know the density function of a distribution. The general idea is that if a random variable $(x, y)$ is distributed within the density $f$ , then it has the density $f$ (blue curve in Figure 2).

This means $x$ and $y$ can be uniformly sampled within an area that is strictly larger than $f$ and check whether the generated point is in the area covered by the density function. If it’s within the area the point is accepted (black in Figure 2), otherwise it is rejected (brown in Figure 2) and new points are sampled until a point, which is within the area and thus can be accepted, is drawn.

More formally, a hat function $h(x)$ that majorizes the density function $f(x)$ is needed. In other words $\forall x \in [l, r]: f(x) < h(x)$ , where $l$ and $r$ are the left and right boundaries of an interval. The hat function (green in Figure 2) is the boundary function for our target density sampling. In this simple example the hat function is the green, horizontal line $x = 1$ .

Rejection sampling of

sin(x)

(blue). Hat function

(x = 1)

is drawn in green. Points are marked black when accepted, and brown when rejected. The area of the distribution density is colored in grey.

It is important to see that for every point $x$ , it needs to be evaluated whether $x$ is within the density area $f(x)$ . As $h(x)$ is by definition always larger than $f(x)$ the formula $y \cdot h(x) \leq f(x)$ can be used to programmatically check whether a generated point is within the target density as it covers the entire density of $f(x)$ (as $f(x) \in [0, 1]\ \forall x \in [\ell, r],\ y \in [0, 1]$ )

Thus the basic rejection method is:

import std.random : rndGen, uniform;
import std.math : PI, sin;
alias T = double;

S sample(RNG, Pdf, Hat, S)(ref RNG gen, Pdf pdf, Hat hat, S left, S right)
{
    for (;;)
    {
        // generate x with density proportional to hat(x)
        S x = uniform!("[]", S)(left, right, gen);

        // generate "vertical" variable y to evaluate x
        S y = uniform!("[]", S)(0, 1, gen);

        // check whether the sampled point is within the density
        if (y * hat(x) <= pdf(x))
            return x;
    }
}
auto pdf = (T) => sin(x);
auto hat = (T x) => 1;
sample(rndGen, pdf, hat, T(0.0), PI);

Furthermore the performance of this method depends heavily on the ratio of $f(x) / h(x) = \alpha$ , where $1 / \alpha$ is the average number of needed iterations to sample one value.

Rejection with inversion

With just a straight line as upper bound a high percentage of the sampled points fall in uncovered areas and thus need to be sampled again. Therefore a more generic hat function is necessary. If the inverse of the hat function is known, the inversion method can be used to sample from an arbitrary hat function using its inverse of the cumulative density function within the interval boundaries. For example for $h(x) = 1$ the integral is trivially $H(x) = x$ and thus the cumulative density in the interval $[0, \pi]$ is $H_{CDF}(x) = \frac{1}{\pi} x$ (the highest value of $h(x)$ needs to yield 1). Hence the inverse is $H_{CDF}^{-1} = \pi x$ and the sampling procedure can be generalized:

import std.math : PI, sin;
import std.random : rndGen, uniform;
alias T = double;

S sample(S, RNG, Pdf, Hat, HatInv)(ref RNG gen, Pdf pdf, Hat hat,
                                   HatInv hatInvCDF)
{
    for (;;)
    {
        // generate x with density proportional to hat(x) by inversing u
        S u = uniform!("[]", S)(0, 1, gen);
        S x = hatInvCDF(u);

        // generate "vertical" variable y to evaluate x
        S y = uniform!("[]", S)(0, 1, gen);

        // check whether the sampled point is within the density
        if (y * hat(x) <= pdf(x))
            return x;
    }
}
auto pdf = (T x) => sin(x);
auto hat = (T x) => 1;
auto hatInvCDF = (T u) => u * PI;
sample!T(rndGen, pdf, hat, hatInvCDF);

The Tinflex algorithm, which will be explained in a later post, can automatically construct a hat function for any differentiable density function.

Squeeze functions

Calculating the probability density function is often expensive, thus defining a lower bound that can be evaluated much faster yields a performance boost. This lower bound is called $s(x)$ which is majorized by $f(x)$ , i.e. $s(x) \leq f(x) \ \forall x \in [l, r]$ . For the previous example the squeeze function is $1 - |1 - 2x / \pi |$ . If $x$ is below the squeeze function, it can be accepted without the need to calculate the density function as by definition every point in the squeeze function $s(x)$ is also below $f(x)$ .

Rejection sampling of

sin(x)

(blue). Hat function

(x = 1)

is drawn in green, whereas the squeeze function

1 - |1 - 2x / \pi |

is drawn in red. Points are marked black when accepted, green when accepted directly with the squeeze function, and brown if rejected. The area of the distribution density is colored in grey.

Furthermore, the sample routine can be adapted:

import std.math : abs, PI, sin;
import std.random : rndGen, uniform;
alias T = double;

S sample(S, RNG, Pdf, Hat, HatInv, Squeeze)(ref RNG gen, Pdf pdf, Hat hat,
                                            HatInv hatInvCDF, Squeeze sq)
{
    import std.random : uniform;
    for (;;)
    {
        // generate x with density proportional to hat(x) by inversing u
        S u = uniform!("[]", S)(0, 1, gen);
        S x = hatInvCDF(u);

        // generate "vertical" variable y to evaluate x
        S y = uniform!("[]", S)(0, 1, gen);
        S t = y * hat(x);

        // check whether the sampled point is below the squeeze
        if (t <= sq(x))
            return x;

        // check whether the sampled point is within the density
        if (y * hat(x) <= pdf(x))
            return x;
    }
}

auto pdf = (T) => sin(x);
auto hat = (T x) => 1;
auto hatInvCDF = (T u) => u * PI;
auto sq = (T x) => 1 - abs(1 - 2 * x / PI);
sample!S(rndGen, pdf, hat, hatInvCDF, sq);

Composition

A density function might be split into multiple parts with each having it’s own hat and squeeze. Given multiple densities, one can sample from the overall density by picking from each sampler according to a given probability $p_i$ which is defined by their hat area. The following figure shows a distribution that is composed out of multiple hat and squeeze parts:

Distribution split into multiple hat (red) and squeeze (green) functions.

A simple example of composing distributions is illustrated below. The density function is composed out of an exponential distribution (left) and a uniform distribution (right) and features a gap in the middle.


import std.random: Mt19937, uniform, rndGen;
alias T = double;

S sample(S, RNG, Sampler)(ref RNG gen, Sampler[] samplers, S[] probs)
{
    import mir.random.discrete : discrete;
    // pick a sampler with prob_i
    auto ds = discrete(probs);
    // sample with the chosen sampler
    return samplers[ds(gen)](gen);
}

// for simplicity we setup only two different samplers
alias Sampler = T delegate(ref typeof(rndGen) gen);
S[] probs = [0.7, 0.3];
Sampler[] samplers = new Sampler[probs.length];

// a part of the exponential distribution on the left
samplers[0] = (ref typeof (gen) gen) {
    import std.math : log;
    auto finv = (S x) => -log(S(1) - x);
    S u = uniform!("[]", S)(0, 0.8);
    return finv(u);
};

// a uniform sampler on the right half
samplers[1] = (ref typeof (gen) gen) {
    return uniform!("[]", S)(2, 3, gen);
};

sample!T(gen, samplers, probs);

Drawing a value from a discrete distribution can be done in $\mathcal{O}(1)$ with the mir.random.discrete package, which implements the Alias method.

The result of our basic composition with different target probabilities for the samples can be seen in the diagrams below. The Tinflex algorithm will automatically generate intervals with hat and squeeze function for the desired distribution that are connected with such a composition sampler.

p = [0.7, 0.3]

p = [0.7, 0.3]

, cumulative

p = [0.3, 0.7]

p = [0.3, 0.7]

, cumulative

Where to go from here

All example snippets are available online and can be run directly with dub - the D package manager. In the next post the Tinflex algorithm will be explained, The Tinflex algorithm automatically splits a differentiable random density function into intervals and constructs hat and squeeze function for each interval.

Literature references

Devroye, Luc. “Sample-based non-uniform random variate generation.” Proceedings of the 18th conference on Winter simulation. ACM, 1986.
Vose, Michael D. “A linear algorithm for generating random numbers with a given distribution.” IEEE Transactions on software engineering 17.9 (1991): 972-975.
Hörmann, Wolfgang, Josef Leydold, and Gerhard Derflinger. Automatic nonuniform random variate generation. Springer Science & Business Media, 2013.

The Mir Blog

Writing efficient numerical code in D

Introduction

What does mir-algorithm offer?

LLVM acceleration

Application

Examples

Basics

Convolution

Lazy evaluation?

Comparison

Zipped tensors

Conclusion

Acknowledgements

Numeric age for D: Mir GLAS is faster than OpenBLAS and Eigen

Machine and software

Source code

Environment variables to set single thread for cblas

Building Eigen

Results

Conclusion

Acknowledgements

Transformed density rejection sampling

Idea

Tinflex transformations

ccc-transformations with c≠0c \neq 0c≠0

ccc-transformations with c=0c = 0c=0

Input of Tinflex

Linear functions

Construction of linear functions

Definition of the linear functions

Example 1: Dagum distribution

Example 2: Gompertz distribution

Benchmarks

Constructing intervals

Sampling

Where to go from here

Literature references

An introduction to sampling from non-uniform random distributions

The problem

The inversion method

Play yourself

The rejection method

Rejection with inversion

Squeeze functions

Composition

Where to go from here

Literature references

$c$ -transformations with $c \neq 0$

$c$ -transformations with $c = 0$