dietslurm
Run h2o on a SLURM cluster using the h2o R package.
To use h2o on a SLURM cluster, one needs to spawn multiple instances of java across several compute nodes, specifying details like threads, memory, IP addresses, etc.
While the h2o docs suggest launching the cluster from
inside R,
it makes suboptimal use of memory and CPUs,
and does a poor job of validating the cluster resources.
We instead use bash
to make the h2o cluster:
-
Language independent: one can use the script to launch h2o clusters for languages besides R.
-
Use SLURM's
srun
process manager to log resource usage which helps troubleshoot performance issues, etc. -
Additional checks and unit tests.
The name "dietslurm" is a play on the slurm drink from Futurama, and adding water or "h2o" to make a diet version. Thanks to Luke Malinowski for the name.
Installation
Install the "h2o" R package for your preferred version of R. See "Install in R" installation instructions in the h2o docs.
At the time of writing, on the Storrs HPC cluster we have installed the "h2o" package for r/3.3.3
Usage
Try the example submission script:
sbatch h2o.slurm
See the example submission script h2o.slurm
Tests
We use the bats unit test suite.
Run the unit tests on the cluster with:
module load bats
tests/*.bats
Some tests take about ~1 second to run because their functionality is hard to replace with fixtures.
If you have done any unit testing before, the format is similar and the documentation is simple and concise. Learn to use bats from reading the man pages, or GitHub. The GitHub link includes links to projects using bats as well as setting up your favorite text editor for Syntax highlighting, etc.
man bats
man 7 bats
# Or see https://github.com/sstephenson/bats
Hacking
-
Read the dirclean script. Many stylistic elements can be intuitively determined by reading and copying what's already there. Some of the style guides below are not strictly followed. Use your noodle ;)
-
As far as possible follow Jeff Lindsay
-
When in doubt, follow Greg Wooledge
-
Only for documenting functions, follow the Google style guide