Run h2o on a SLURM cluster using the h2o R package.
To use h2o on a SLURM cluster, one needs to spawn multiple instances of java across several compute nodes, specifying details like threads, memory, IP addresses, etc.
While the h2o docs suggest launching the cluster from
it makes suboptimal use of memory and CPUs,
and does a poor job of validating the cluster resources.
We instead use
bash to make the h2o cluster:
Language independent: one can use the script to launch h2o clusters for languages besides R.
srunprocess manager to log resource usage which helps troubleshoot performance issues, etc.
Additional checks and unit tests.
The name "dietslurm" is a play on the slurm drink from Futurama, and adding water or "h2o" to make a diet version. Thanks to Luke Malinowski for the name.
Install the "h2o" R package for your preferred version of R. See "Install in R" installation instructions in the h2o docs.
At the time of writing, on the Storrs HPC cluster we have installed the "h2o" package for r/3.3.3
Try the example submission script:
See the example submission script h2o.slurm
We use the bats unit test suite.
Run the unit tests on the cluster with:
module load bats tests/*.bats
Some tests take about ~1 second to run because their functionality is hard to replace with fixtures.
If you have done any unit testing before, the format is similar and the documentation is simple and concise. Learn to use bats from reading the man pages, or GitHub. The GitHub link includes links to projects using bats as well as setting up your favorite text editor for Syntax highlighting, etc.
man bats man 7 bats # Or see https://github.com/sstephenson/bats
Read the dirclean script. Many stylistic elements can be intuitively determined by reading and copying what's already there. Some of the style guides below are not strictly followed. Use your noodle ;)
As far as possible follow Jeff Lindsay
When in doubt, follow Greg Wooledge
Only for documenting functions, follow the Google style guide