Using the Oregon Neuroscience Grid
The Oregon Neuroscience Grid (ONG) is a computational instrument available for researchers using the Center. Your LCNI account will permit access to this instrument.
We maintain a number of software packages commonly used in neuroimaging analysis. These are primarily installed from the NeuroDebian repository. If you would like to use software not already installed, let us know and we will do our best to accommodate your needs.
Connecting to the Grid
ONG supports command-line and graphical program interfaces. Start by choosing a node between ong0 and ong11. Using a terminal program, connect to the command-line interface using ssh, for example:
joec@torrontes:~$ ssh chuck@ong5
. . . <lots of output here> . . .
chuck@ong5 ~ $
This initiates a connection to the grid node. You can then run command-line programs, for instance, MATLAB:
chuck@ong5 ~ $ matlab
Warning: No display specified. You will not be able to display graphics on the screen.
< M A T L A B (R) >
Copyright 1984-2011 The MathWorks, Inc.
R2011b (188.8.131.524) 64-bit (glnxa64)
August 13, 2011
To get started, type one of these: helpwin, helpdesk, or demo.
For product information, visit www.mathworks.com.
Note that programs lacking a command-line interface, such as the graphical interface to the FSL Suite, fsl, will not be able to run. For this, we need to be able to tunnel graphical (X11) traffic back to our local X11 server. First, see the knowledge base article Using X11 to set up your machine as an X11 server.
With that done, you can add the -X command-line option to enable tunneling X11 traffic. Now, running matlab from the command-line will produce the graphical environment. Note that the About window is helpfully tagged with the machine it is running on.
Some packages are managed through the Environment Modules system. This offers great flexibility as it permits installation of different versions of the same software and allows you to choose which version you wish to use. This comes in handy when you want to have a work flow on a set of data whose results are invariant despite version upgrades. Simply set up your work flow scripts to select the relevant versions of the software packages used in your analysis.
Working with Environment Modules is done with the module command, specifying the particular action with a sub-command. Commonly used sub-commands are: avail to see what is available, load to load a program module, and list to see what you have already loaded.
chuck@ong8 ~ $ module avail
--------------------- /usr/local/packages/Modules/versions ---------------------
---------------- /usr/local/packages/Modules/3.2.10/modulefiles ----------------
chuck@ong8 ~ $
Here you see we have various versions of fsl, matlab, and mriconvert installed. To load the default version of a module, simply invoke module load <module name>, for instance module load matlab. Running matlab would then launch version R2013a. If you want to use a non-default version, you can specify that with an additional qualifier, e.g. module load fsl/fsl-5.0.4 or module load matlab/R2011a.
Note that we periodically update software on the grid. Once done, the latest version becomes the default, so if you have an ongoing analysis, you may want to specify the version to the module command to be sure you get the same version with each analysis run. We will always announce any upgrades on the lcni-users mailing list.
Use of the grid can be as simple as running programs and scripts directly from the command-line. Doing so will make use of the node to which you are connected, though the command prompt will not return until the program is complete. To regain the command prompt and still allow the job to run, press Ctrl-Z, then run the bg (background) command. Ctrl-Z suspends the current foreground process and returns you to the command prompt, and bg will resume the suspended process in the background, permitting you to issue more commands while the process runs.
Note that, at this point, if you simply close your terminal window, all background processes will be killed, which can leave the output in an indeterminate state. To avoid having your background processes terminated, you can exit your terminal session gracefully, either with the exit command or by pressing Ctrl-D. Your terminal program can then be stopped, but any background processes will continue to normal completion.
The above techniques will process your data, but will typically not make use of other grid nodes. A noted exception is FSL, which has tight integration with the grid computing software.
There are a few options to make use of the processing cycles the grid has to offer. You can make connections to multiple grid nodes and run multiple jobs on each, and this works well for interactive programs in particular. However this can become tedious and difficult to manage when used for batch jobs. There is a better way. If the programs you are using are not natively grid-aware, you can launch them in such a way so they are assigned to a grid node implicitly. The qsub program and the fsl_sub script both serve this purpose. While fsl_sub is slightly easier to use, both will enable submission of programs and scripts to the grid.
Basic use is to simply precede your command line with qsub or fsl_sub. Below are examples of both, each of which accomplishes the same thing:
qsub -b y /usr/local/packages/mriconvert/2.1.0/mcverter -F out-%SQ%PR-put -o ~/MRI/FisherNormal/1732/converted -f fsl -n -x -d ~/MRI/FisherNormal/1732/184.108.40.206.1220.127.116.11.20418.30000009120218043109300000067/
fsl_sub mcverter -F out-%SQ%PR-put -o MRI/FisherNormal/1732/converted -f fsl -n -x -d MRI/FisherNormal/1732/18.104.22.168.122.214.171.124.20418.30000009120218043109300000067/
- qsub requires the "-b y" option to supply command-line arguments to your program.
- qsub is not aware of your environment or current directory, the full path to your command and any pathname arguments should be supplied.
- qsub puts its process output and error files in the root of your home directory. This can be changed with the -e and -o options.
The fsl_sub script is a wrapper around qsub, and supplies options that make the script slightly easier to use. You will need to run "module load fsl" for it to work, but it can run any program or script, not just FSL commands. If you are submitting programs that are under Environment Modules control, you will also need to run the module load command for them as well.
The steps to connect to and use the ONG are:
- Use a terminal program to ssh into a grid node, e.g. ong9.uoregon.edu.
- For a graphical interface, be sure your local machine has an X11 server active and the terminal client is set up to receive X11 traffic.
- Run the programs of your choice. Use the module command to load environments for some software.
- Use qsub or fsl_sub to submit jobs for grid processing.
Software Available on the ONG
As noted above, we install software from the main Debian stable repositories and the NeuroDebian repository, so if a package is in these repositories, it is already installed or readily available.
Some software is not available through NeuroDebian, or alternate versions are installed. These may be found in the /usr/local/bin directory. You can show these with ls /usr/local/bin. In most cases, these are available without any additional action. Simply run them from the command line, e.g. peate25.pl.
Here is a short list of software you will find on ONG machines:
- SPM8 and SPM12b
- Python 2.7.x
The simplest way to determine if something is in your reach is to try to run it. If you get "command not found", then the program was not found in your search path.