HPC / dietslurm

dietslurm

Run h2o on a SLURM cluster using the h2o R package.

To use h2o on a SLURM cluster, one needs to spawn multiple instances of java across several compute nodes, specifying details like threads, memory, IP addresses, etc.

While the h2o docs suggest launching the cluster from inside R, it makes suboptimal use of memory and CPUs, and does a poor job of validating the cluster resources. We instead use bash to make the h2o cluster:

1. Language independent: one can use the script to launch h2o clusters for languages besides R.

2. Use SLURM's srun process manager to log resource usage which helps troubleshoot performance issues, etc.

3. Additional checks and unit tests.

The name "dietslurm" is a play on the slurm drink from Futurama, and adding water or "h2o" to make a diet version. Thanks to Luke Malinowski for the name.

Installation

Install the "h2o" R package for your preferred version of R. See "Install in R" installation instructions in the h2o docs.

At the time of writing, on the Storrs HPC cluster we have installed the "h2o" package for r/3.3.3

Usage

Try the example submission script:

sbatch h2o.slurm

See the example submission script h2o.slurm

Tests

We use the bats unit test suite.

Run the unit tests on the cluster with:

module load bats
tests/*.bats

Some tests take about ~1 second to run because their functionality is hard to replace with fixtures.

If you have done any unit testing before, the format is similar and the documentation is simple and concise. Learn to use bats from reading the man pages, or GitHub. The GitHub link includes links to projects using bats as well as setting up your favorite text editor for Syntax highlighting, etc.

man bats
man 7 bats
# Or see https://github.com/sstephenson/bats

Hacking

1. Read the dirclean script. Many stylistic elements can be intuitively determined by reading and copying what's already there. Some of the style guides below are not strictly followed. Use your noodle ;)

2. As far as possible follow Jeff Lindsay

3. When in doubt, follow Greg Wooledge

4. Only for documenting functions, follow the Google style guide