CAHL Cluster WikiPage


About

cahl_cluster CAHL Cluster is a parallel computing facility originally housed at the Computational Aero & Hydrodynamics Laboratory (CAHL) in the Department of Mechanical and Aerospace Engineering at the George Washington University. It was initially deployed in 2011, and since then primarily managed by team members of the lab. In 2017, the team rebuilt the cluster to update its operating and provisioning systems. Several hard drive failures led to another rebuild in 2019 using OpenHPC's collection of open-source packages in a "stateless" way (OS runs in RAMs instead of on hard drives). The cluster now runs CentOS 7, employs Warewulf as the provisioning system, Slurm as the job management tool, and Lmod as the package management tool. The cluster serves as the main testing and debugging platform of the lab for developing parallel flow and structure solvers. In the Fall of 2019, the cluster was relocated to Clarkson University following the move of the lab.


Hardware

The cluster features a total number of 252 CPU cores, with 12 on the master node and 240 on the compute nodes. The detailed configurations are:


User's Guide

To run jobs on the cluster, a user must have basic knowledge in at least three aspects: logging in the system, accessing preinstalled packages, and managing jobs.

Logging in

To login to the system, use the following command in your local terminal:

[user@localhost ~]$ ssh your_user_name@cluster_hostname

For Windows users, PuTTY is a good choice to serve as your local terminal. Alternatively, enable the Windows Subsystem for Linux (WSL) feature that comes with Windows for the purpose.

If one wants to use remote software (such as Tecplot) with graphic user interface (GUI), use the -X option to login:

[user@localhost ~]$ ssh -X your_user_name@cluster_hostname

Note that, the local system must have X-Window support for the remote GUI to display correctly. Mac and Linux come with X-Window support naturally. For Windows, third-party software, such as Xming, is required.

Finally, to transfer files between local and remote systems, use the scp or sftp command. One can also use GUI-based sftp clients, such as WinSCP or FileZilla, to transfer or sync files.

Package Management

This section describes how to access preinstalled packages, such as compilers, numerical libraries, and many other tools.

Show available packages

[user@cluster ~]$ module avail

----------------------------------- /opt/ohpc/pub/moduledeps/intel ------------------------------------
   hdf5/1.10.4        metis/5.1.0         ocr/1.0.1             pdtoolkit/3.25      py3-numpy/1.15.3
   impi/2019.3.199    mpich/3.3           openmpi/1.10.7        plasma/2.8.0        scotch/6.0.6
   likwid/4.3.3       mvapich2/2.3        openmpi3/3.1.3        py2-numpy/1.15.3    superlu/5.2.1

-------------------------------------- /opt/ohpc/pub/modulefiles --------------------------------------
   EasyBuild/3.8.1           clustershell/1.8        gnu8/8.3.0              pmix/2.2.2
   Tecplot/2018r1     (L)    cmake/3.13.4            hwloc/2.0.3             prun/1.3
   autotools                 gnu/5.4.0               intel/19.0.3.199        singularity/3.1.0
   charliecloud/0.9.7        gnu7/7.3.0              papi/5.6.0              valgrind/3.14.0

  Where:
   L:  Module is loaded

Use "module spider" to find all possible modules.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".

Note that, this command does not necessarily display all the preinstalled packages. As the output suggests, one can use module spider to see a complete list.

Load a package

To compile, debug, or run a code, a user must first load the necessary packages. For example, one can use the following command to load the Intel compilers:

[user@cluster ~]$ module load intel/19.0.3.199

With that, the Intel Fortran compiler ifort, Intel C/C++ compiler icc, and the related Intel libraries are available for use.

To compile an MPI code, one must load one of the preinstalled MPI packages (mpich, mvapich2, openmpi, openmpi3). We recommend using the openmpi package on CAHL Cluster (for the best performance). The following command will load it:

[user@cluster ~]$ module load openmpi/1.10.7

After that, one can compile MPI Fortran code using mpifort or mpif90, MPI C code using mpicc, and MPI C++ code using mpicxx. It is worth noting is that, once an MPI package is loaded, an environment variable MPI_DIR is also created. This is helpful if one wants to manually link the MPI libraries to some applications.

The widely used domain decomposition package Metis is also preinstalled, and is loadable through:

[user@cluster ~]$ module load metis/5.1.0

Once loaded, commands such as gpmetis and mpmetis are available for use. Meanwhile, four environmental variables METIS_BIN, METIS_DIR, METIS_INC, and METIS_LIB are also created.

To automatically load these packages the next time a user logs into the system, one can add the above commands to the .bashrc file in his/her home directory. Alternatively, one can use the module save command to save the loaded module list.

Show loaded packages

[user@cluster ~]$ module list

Currently Loaded Modules:
  1) Tecplot/2018r1   2) intel/19.0.3.199   3) openmpi/1.10.7   4) metis/5.1.0

Unload a package

For example, if a user wants to use the GNU compiler instead of the Intel compiler, he/she must first unload the Intel compiler (if it's already loaded) using the following command:

[user@cluster ~]$ module unload intel/19.0.3.199

and then load the GNU compiler using the usual module load command.

Job Management

This section gives information on how to submit jobs to the compute nodes and how to manage jobs. Note that, it is generally discouraged to run long-time jobs on the master (login) node.

Show available resources

To see how busy or idle the cluster is, use the sinfo command:

[user@cluster ~]$ sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
defq*        up   infinite     12  alloc node[01-12]
defq*        up   infinite      8   idle node[13-20]

The output simply says that the defq (default queue) partition currently has 12 busy (allocated) nodes and 8 idle nodes. Note that, there is currently no enforced time limit for a job in this queue, but it is suggested that a user limit his/her run time to one week for the fairness of all users.

Submit a job

There are a couple of ways to run jobs on the compute nodes. But the most common way is to use a job script. Here is a sample script called jobsub.sh for submitting an MPI job. The contents of the script are also listed below with comments:

#!/bin/bash                             # shell environment
#SBATCH -J YourJobName                  # job name
#SBATCH -t 7-00:00:00                   # requested maximum run time (day-hour:min:sec)
#SBATCH -p defq                         # requested partition (queue)
#SBATCH -n 60                           # requested number of CPU cores
#SBATCH -e slurm%j.err                  # system error  file
#SBATCH -o slurm%j.out                  # system output file
#SBATCH --mail-type=ALL                 # email notification type
#SBATCH --mail-user=xxx@xxx.edu         # user's email address to recieve notifications

mpirun ./YourCode > output.out 2>&1     # mpi command to run the job

Do not forget to modify items in the script where necessary before submitting your job. Note that, the requested number of cores usually should be multiples of 12, because each compute node has 12 cores. In the rare case that a user has a series of jobs each reuire just 1 core, use the srun command with for loops in your script.

To submit a job, use the following sbatch command:

[user@cluster rundir]$ sbatch jobsub.sh
Submitted batch job 58

The output tells the user that the submitted job has a job ID 58 in this example.

One can also submit interactive jobs using the salloc command. More information regarding this command can be found in Slurm's manual.

Check the status of jobs

[user@cluster ~]$ squeue -u username
JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
   58      defq     prop     user PD       0:00     12 (Resources)
   57      defq     prop     user  R       0:48     12 node[01-12]

The output says that the user has one running job and one pending job in the queue, with job ID 57 and 58, respectively.

Cancel a job

[user@cluster ~]$ scancel 58

The above command will cancel/terminate the job with job ID 58.


Admin's Guide

Add a new user

To add a new user whose preferred user name is xxx:

[root@cluster ~]# adduser xxx

Then set up a password for the user:

[root@cluster ~]# passwd xxx

And, that's it, no more action is needed. Note that, the cluster is configured to automatically synchronize users' passwords to all compute nodes every five minutes using the cron service and Warewulf's file sync function. For this reason, a new user might have to wait for up to five minutes before he/she can run a job.

Delete a user

To delete the user xxx together with his/her home directory:

[root@cluster ~]# userdel -r xxx

Omit the -r if want to keep the user's home directory.

Install packages

All packages should be installed into the best category (e.g., apps, compiler, libs, utils, etc.) under the /opt/ohpc/pub directory (which is an NFS directory shared to all compute nodes).

Once a package is installed, the admin must set up a module file for this package under the /opt/ohpc/pub/modulefiles directory. In this way, a user can access this package through module load. Sample module files are found in any sub-directories of /opt/ohpc/pub/modulefiles, and also in /opt/ohpc/pub/examples.

Intel compiler license

The Intel compiler is free for students and educators. Its license is valid for one year but is renewable here. Once a new license file is received, go to /opt/intel/licenses to replace the expired license file with the new one.

Reboot the cluster

The master node can be rebooted when the compute nodes are either on or off, but not when any of them are in the process of booting up.

The compute nodes can be rebooted only when the master node is on.

More information

For more information, please consult BZ or Google.



Last updated 01/13/2020 by BZ@CAHL