One of my Julia packages is Deldir. It is a wrapper around the Fortran code from the deldir package for R.

Until now, the procedure has been that when Deldir is installed, Julia runs the associated build script that attempts to compile the Fortran code using gfortran. This works for me on both Linux and Mac when gfortran is installed.

But it would nice to not require gfortran and be able to provide Deldir on Windows. This is exactly the purpose of the BinaryBuilder package. Here I am going through the steps I took to get BinaryBuilder to work for Deldir.

Local compilation

The first thing to do is to figure out how to compile the code on Linux, as this is the platform used for doing cross compilation for all other platforms.

When installing deldir in R on Linux the compiler commands are all printed:

> install.packages("deldir")
* installing *source* package ‘deldir’ ...
** package ‘deldir’ successfully unpacked and MD5 sums checked
** libs
f95   -fpic  -g -O2  -c acchk.f -o acchk.o
f95   -fpic  -g -O2  -c addpt.f -o addpt.o
f95   -fpic  -g -O2  -c adjchk.f -o adjchk.o
f95   -fpic  -g -O2  -c binsrt.f -o binsrt.o
f95   -fpic  -g -O2  -c circen.f -o circen.o
f95   -fpic  -g -O2  -c cross.f -o cross.o
f95   -fpic  -g -O2  -c delet.f -o delet.o
f95   -fpic  -g -O2  -c delet1.f -o delet1.o
f95   -fpic  -g -O2  -c delout.f -o delout.o
f95   -fpic  -g -O2  -c delseg.f -o delseg.o
f95   -fpic  -g -O2  -c dirout.f -o dirout.o
f95   -fpic  -g -O2  -c dirseg.f -o dirseg.o
f95   -fpic  -g -O2  -c dldins.f -o dldins.o
clang -I"/opt/R/3.5.1/lib/R/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2 -Wall -pedantic -c init.c -o init.o
f95   -fpic  -g -O2  -c initad.f -o initad.o
f95   -fpic  -g -O2  -c insrt.f -o insrt.o
f95   -fpic  -g -O2  -c insrt1.f -o insrt1.o
f95   -fpic  -g -O2  -c intri.f -o intri.o
f95   -fpic  -g -O2  -c locn.f -o locn.o
f95   -fpic  -g -O2  -c master.f -o master.o
f95   -fpic  -g -O2  -c mnnd.f -o mnnd.o
f95   -fpic  -g -O2  -c pred.f -o pred.o
f95   -fpic  -g -O2  -c qtest.f -o qtest.o
f95   -fpic  -g -O2  -c qtest1.f -o qtest1.o
f95   -fpic  -g -O2  -c stoke.f -o stoke.o
f95   -fpic  -g -O2  -c succ.f -o succ.o
f95   -fpic  -g -O2  -c swap.f -o swap.o
f95   -fpic  -g -O2  -c testeq.f -o testeq.o
f95   -fpic  -g -O2  -c triar.f -o triar.o
f95   -fpic  -g -O2  -c trifnd.f -o trifnd.o
clang -shared -L/opt/R/3.5.1/lib/R/lib -L/usr/local/lib -o deldir.so acchk.o addpt.o adjchk.o binsrt.o circen.o cross.o delet.o delet1.o delout.o delseg.o dirout.o dirseg.o dldins.o init.o initad.o insrt.o insrt1.o intri.o locn.o master.o mnnd.o pred.o qtest.o qtest1.o stoke.o succ.o swap.o testeq.o triar.o trifnd.o -lgfortran -lm -lquadmath -L/opt/R/3.5.1/lib/R/lib -lR
installing to /home/robert/R/x86_64-pc-linux-gnu-library/3.5.1/deldir/libs
** R
** data
*** moving datasets to lazyload DB
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (deldir)

A side note: Many of the Julia packages using BinaryBuilder rely on a Makefile to compile the binary dependencies, but this is not available for deldir (at least not in a form I understand). This is why I have to be more explicit here.

What essentially happens with the Fortran code is that each file is compiled into an object file (the .o) and in the end these are all linked together into the shared object file deldir.so. A number of libraries and library paths are specified with the -l and -L flags to make deldir.so work with R.

The linking is handled with clang because my ~/.R/Makevars contains the following:

CC=clang
CXX=clang++

But clang could be replaced with gfortran or gcc in the linking step.

In BinaryBuilder we have to end up with deldir.so using only bash. To compile the all the Fortran we can execute the following loop:

for f in *.f; do
    gfortran -fPIC -O2 -pipe -g -c "${f}" -o "$(basename "${f}" .f).o"
done

It turns out that I don’t need all the libraries in the linking, so the shared object file can be created with this command:

clang -shared -o deldir.so *.o

Compilation with BinaryBuilder

BinaryBuilder’s README is good to get started. In particular, we can use the wizard with BinaryBuilder.run_wizard().

BinaryBuilder’s wizard first demands to get a URL where the source code can be obtained. A hash is computed of the downloaded file to ensure that subsequent runs actually use the same source code. To download a fixed version of the Fortran code I link to specific date on MRAN’s time machine. There is nothing particular about the date I use – it was just around that time I made the first version of Deldir, so I am pretty confident that the compiled code works as expected.

Update: Instead choosing an arbitrary MRAN date I now use the archive on CRAN’s deldir page. The reason is that the archive include the release dates of the different versions.

When running the wizard we soon enter a bash shell. But there are differences from the regular bash shell where the above commands worked. For instance, it is not safe to use gfortran and gcc since the specific versions depend on the environment where the commands are executed – and the environment may change. Instead, we rely on environment variables specifying the compilers with CC for the C compiler and FC for the Fortran compiler. Another thing is that the shared object file has to be in a specific folder to be found.

My script is available in the DeldirBuilder repo. The main difficulty for me was finding the right environment variables to use. I found those by snooping through other builder projects (found in the JuliaPackaging project on GitHub) and writing out all environment variables in the wizard’s bash shell with the command env. However, we have to complete the guide to actually run the bash commands and see the output.

The bash script works on all platforms except FreeBSD. I have not pursued FreeBSD further as I do not have a computer with FreeBSD installed.

Deployment

A super nice idea in BinaryBuilder is to use Travis to build the binaries and provide a smooth process to make them available. To achieve this, BinaryBuilder sets up a personal access token on GitHub to upload the binaries to DeldirBuilder’s releases.

Note that by default Travis does not build for macOS. This can easily be turned on by consciously changing this line in the .travis.yml file:

- BINARYBUILDER_AUTOMATIC_APPLE=false

There is also the little trick that Travis only uploads binaries to GitHub if a commit is tagged in Git.

Changes in Deldir

After building the binaries a few things had to be changed in Deldir. This is also documented in BinaryBuilder’s README.

Part of the release from Travis is a build.jl file that goes into Deldir’s deps folder. This deps/build.jl used to contain compilation commands and now it uses the BinaryProvider package to utilize the binaries created with BinaryBuilder. This package has to be included as a dependency.

In the the main package file src/Deldir.jl the following lines are included:

depsfile = joinpath(@__DIR__, "..", "deps", "deps.jl")
if isfile(depsfile)
     include(depsfile)
end

This provides the constant libdeldir with the path to the downloaded deldir.so (or deldir.dll on Windows).

Experimenting with BinaryBuilder

Once the wizard completes a build_tarballs.jl is generated. This can be run from the shell as a program. To see its options run

julia build_tarballs.jl --help

One possibility is to generate binaries for a specific platform:

julia --color=yes build_tarballs.jl --verbose x86_64-linux-gnu

It is even possible to use specific versions of the the C compiler, e.g., x86_64-linux-gnu-gcc7.