Category Archives: HPCC

Setup EMSoft on HPCC@MSU

EMSoft is an open source package that allows users to simulate EBSD and ECCI images. The public repository can be found via the following link:

https://github.com/marcdegraef/EMsoftPublic.git

Currently, only GNU compiler is officially supported and tested by the developer. To avoid any conflict with the DAMASK settings on HPCC (Intel bases system), it is recommended to install EMsoft to your own HOME directory. Before starting the installation, you need to make sure you are on a dev-node with GPU (see HPCC website for more details) and switch to GNU environment with the following script,

#!/bin/sh

# Switch environment
module swap Intel GNU/4.9
module load CUDA/6.0
module load BLAS/3.6.0
module load LAPACK/3.6.0

# export EMSoft binary location
export PATH=${HOME}/EMsoft/Build/Bin:$PATH

Save the script above to useEMsoft.env so that in the future​ you can easily toggle back to EMsoft environment by

source useEMsoft.env

The environment is configured with the assumption that you are going to install EMsoft in your home directory under the folder EMsoft.

To install EMsoft, clone the public repository and change its name to EMsoft

git clone https://github.com/marcdegraef/EMsoftPublic.git
mv EMsoftPublic EMsoft
cd ${HOME}/EMsoft/Support/SDK_Build_Scripts/Linux_Build_Scripts

Inside this folder, you will find the installation script, Build_SDK.sh, and a configuration file, SDK_Configuration.conf. Nothing needs to be done for the build script as long as you are planning to let EMsoft to built all 3rd-party library for you, which is recommended. However, if you already have your own libraries ready and want EMsoft to link to existing libraries, make sure they have CMake configuration files, which are required by EMsoft during compiling. Open SDK_Configuration.conf with a text editor and change the following lines to match your own settings,

SDK_PARENT=${HOME}
SDK_FOLDER_NAME=${HOME}/EMsoft_SDK
SDK_INSTALL=${HOME}/EMsoft_SDK

Then execute the build script let the system take care all the 3rd-party library installation for you.

bash Build_SDK.sh

After all 3rd-party library installed, change back to the EMsoft root directory and open  CMakeLists.txt. Locate the following compiler setting block

if (Fortran_COMPILER_NAME MATCHES "gfortran.*")
set (CMAKE_Fortran_FLAGS "${CMAKE_Fortran_FLAGS} -fopenmp -fno-sign-zero")
set (CMAKE_Fortran_FLAGS_RELEASE "${CMAKE_Fortran_FLAGS_RELEASE} -funroll-all-loops -fno-f2c -O3 -fno-sign-zero -std=gnu")
set (CMAKE_Fortran_FLAGS_DEBUG "${CMAKE_Fortran_FLAGS_DEBUG} -fcheck=bounds -fno-f2c -O0 -g -fno-sign-zero -std=gnu")
get_filename_component (Fortran_COMPILER_PATH ${CMAKE_Fortran_COMPILER} DIRECTORY)
set(GFortran_LIB_DIR "${Fortran_COMPILER_PATH}/../lib")

and insert

set(EXE_LINK_LIBRARIES ${EXE_LINK_LIBRARIES} lapack blas gomp gcc_eh)

in the block above to force the system to link to LAPACK, BLASm and use OpenMP.

After editing the CMake file, create a build folder and start building EMsoft

mkdir Build
cd Build
${HOME}/EMsoft_SDK/cmake-3.4.3-Linux-x86_64/bin/cmake -DEMsoft_SDK=${HOME}/EMsoft_SDK/ -DCMAKE_BUILD_TYPE=Debug  ..
make -j

The building time should be fairly short, after which all the EMsoft binary will be available in the build/Bin folder. Since we did not specify the install directory, CMake will try to install them to /opt, which regular users do not have access to. Hence, we are going to leave the built binary here and move the necessary file into the build folder.

First of all, export the build/Bin folder to your path or just source the environment setting at the very top. Then copy the two folders in the EMsoft to build, i.e.

cd ${HOME}/EMsoft
cp -R opencl Build/
cp -R resources Build/

Execute EMsoftInit to generate the configuration file (EMsoftConfig.json) in ${HOME}/.config/EMsoft. Manualy edit it as follows

{
"EMsoftpathname": "${HOME}/EMsoft/Build/",
"EMdatapathname": "${HOME}/EMsoftWorkbench/",
"EMtmppathname": "${HOME}/.config/EMsoft/tmp/",
"EMsoftLibraryLocation": ${HOME}/EMsoft/Build",
"Release": "Yes",
"Develop": "No",
"UserName": "Yourname",
"UserEmail": "yourEmail"
"UserLocation": "yourOrganization"
}

As always, remember to replace ${HOME} with the absolute path. Also, do not forget to create the workbench folder as all of the EMsoft simulation will take place in that particular folder.

For the EBSD example in the manual, there are few things need to be modified in order to make it work on HPCC.
1. A XtalFolder needs to be created in the EMsoftWorkbench. EMsoft cannot automatic generate folders on HPCC.
2. Change the top line of the data file EMMCexample.nml (required by EMMCOopenCL) into “&MCCLdata” as this is the name hardwired in the Fortran code.
3. All strings in the .nml files need to be in double quotes and all path need to be a relative path.
4. Generally speaking, the simulation is short enough for a developing node. However, if you want finer data, you can edit your input file (
.nml) and submit the job to HPCC queue. More information can be found at​ the link below:
https://wiki.hpcc.msu.edu/display/hpccdocs/GPU+Computing#GPUComputing-JobScheduling

PETSc 3.7.4 on HPCC

DAMASK is influenced by API changes in PETSc from 3.6.x to 3.7.x. For more details, see . A particular source code of DAMASK will either work with PETSc 3.6.x or 3.7.x, but not both. Therefore, an intermediate solution before the official transition of DAMASK to PETSc 3.7.x is to apply a patch that ensures compatibility with the current version 3.7.x of PETSc.

To switch your version of DAMASK to utilize PETSc 3.7.4 on HPCC, try:

damask
patch -p1 installation/patch/petsc3.7
export PETSC_DIR=/mnt/research/CMM/opt/petsc/3.7.4--Intel-16.3
make clean spectral

This should set your environment on HPCC to use PETSc 3.7.4 with the patched petsc3.7 version of DAMASK. If you want to use PETSc 3.7 by default, make a permanent change of $PETSC_DIR  in your bash profile (done easiest by putting the export command into $HOME/.environment).