Installing xSDK 0.4.0

This page discusses the platforms on which the xSDK 0.4.0 release has been tested and contains general instructions for building, as well as more specific instructions for select high-end computing systems. See also details about obtaining the xSDK.

As more information becomes available for building the xSDK 0.4.0 release on different platforms, that information will be posted here..  Check back for updates.

xSDK 0.4.0 general build instructions

1. After cloning spack git repo, setup spack environment
# For bash users
$ export SPACK_ROOT=/path/to/spack
$ . $SPACK_ROOT/share/spack/setup-env.sh

# For tcsh or csh users (note you must set SPACK_ROOT)
$ setenv SPACK_ROOT /path/to/spack
$ source $SPACK_ROOT/share/spack/setup-env.csh
1.1 Make sure proxy settings are set – if needed.

If a web proxy is required for internet access on the install machine, please set up proxy settings appropriately. Otherwise, Spack will fail to “fetch” the packages of your interest.

# For bash users
$ export http_proxy=<your proxy URL>
$ export https_proxy=<your proxy URL>

# For tcsh or csh users
$ setenv http_proxy <your proxy URL>
$ setenv https_proxy <your proxy URL>
2. Setup spack compilers 
spack compiler find

Spack compiler configuration is stored in $HOME/.spack/$UNAME/compilers.yaml and can be checked with

spack compiler list
3. Edit/update packages.yaml file to specify any system/build tools needed for xSDK installation.

Although Spack can install required build tools, it can be convenient to use preinstalled  tools – if already installed.  On macOS, in addition to Xcode, we’ve used packages from Homebrew.  Such preinstalled packages can be specified to spack in $HOME/.spack/packages.yaml config file.    The following is an example for macOS

# -------------------------------------------------------------------------
# This file controls default concretization preferences for Spack.
#
# Settings here are versioned with Spack and are intended to provide
# sensible defaults out of the box. Spack maintainers should edit this
# file to keep it current.
#
# Users can override these settings by editing the following files.
#
# Per-spack-instance settings (overrides defaults):
#   $SPACK_ROOT/etc/spack/packages.yaml
#
# Per-user settings (overrides default and site settings):
#   ~/.spack/packages.yaml
# -------------------------------------------------------------------------
packages:
  autoconf:
    paths:
      autoconf@2.69: /usr/local
    buildable: False
  automake:
    paths:
      automake@1.15.1: /usr/local
    buildable: False
  libtool:
    paths:
      libtool@2.4.6: /usr/local
    buildable: False
  m4:
    paths:
      m4@1.4.18: /usr/local
    buildable: False
  cmake:
    paths:
      cmake@3.9.3: /usr/local
    buildable: False
  pkg-config:
    paths:
      pkg-config@1.3.9: /usr
    buildable: False
  python:
    paths:
      python@2.7.13: /usr
    buildable: False
  all:
    providers:
      mpi: [mpich]
    compiler: [clang@9.0.0-apple]
4. Install xSDK

After the edit, xSDK packages and external dependencies can be installed with a single command:

 spack install xsdk
5. Install environment modules.

Optionally one can install  environment-modules package to access xsdk packages as modules.

 spack install environment-modules

After installation, the modules can be enabled with the following command line.

For bash:

# For bash users
$ source  `spack location -i environment-modules`/Modules/init/bash
# For tcsh or csh users
$ source  `spack location -i environment-modules`/Modules/init/tcsh
6. Load xSDK module and its sub-modules.

Now you can load xSDK environment. Try Spack’s load command with the -r (resolve all dependencies) option:

spack load -r xsdk

Then, module list generates the following output, for example:

Currently Loaded Modulefiles:
 1) mpich-3.3b2-gcc-8.2.1-hexpnjh
 2) numactl-2.0.12-gcc-8.2.1-iazvfzp
 3) zlib-1.2.11-gcc-8.2.1-cwve3xs
 4) hdf5-1.10.4-gcc-8.2.1-pryqzbp
 5) openblas-0.3.4-gcc-8.2.1-thneimy
 6) hypre-2.15.1-gcc-8.2.1-c7h4gtr
 7) metis-5.1.0-gcc-8.2.1-f6wzzhi
 8) parmetis-4.0.3-gcc-8.2.1-gwbojrk
 9) superlu-dist-6.1.0-gcc-8.2.1-p2olnk3
 10) bzip2-1.0.6-gcc-8.2.1-dmheqyf
 11) boost-1.69.0-gcc-8.2.1-tpes32u
 12) glm-0.9.7.1-gcc-8.2.1-ozt2r2z
 13) matio-1.5.13-gcc-8.2.1-3oidyje
 14) netcdf-4.6.2-gcc-8.2.1-u7ywiqz
 15) trilinos-12.14.0-rc1-gcc-8.2.1-kass6i7
 16) petsc-3.10.3-gcc-8.2.1-eko2h5x
 17) pflotran-xsdk-0.4.0-gcc-8.2.1-hemk2tj
 18) alquimia-xsdk-0.4.0-gcc-8.2.1-oerexyg
 19) amrex-18.10.1-gcc-8.2.1-r5f4jnv
 20) adol-c-develop-gcc-8.2.1-bjm4wr2
 21) arpack-ng-3.6.3-gcc-8.2.1-wyahvso
 22) gsl-2.5-gcc-8.2.1-qzonakz
 23) intel-tbb-2019.2-gcc-8.2.1-spudoqk
 24) muparser-2.2.6.1-gcc-8.2.1-mf6kph5
 25) nanoflann-1.2.3-gcc-8.2.1-vivp3qi
 26) netlib-scalapack-2.0.2-gcc-8.2.1-45so534
 27) oce-0.18.3-gcc-8.2.1-o4dtwgt
 28) p4est-2.0-gcc-8.2.1-5ks4hsq
 29) suite-sparse-5.3.0-gcc-8.2.1-zrlpqbs
 30) sundials-3.2.1-gcc-8.2.1-bt6b3j5
 31) dealii-9.0.1-gcc-8.2.1-pkrq3ll
 32) mfem-3.4.0-gcc-8.2.1-xq55orp
 33) phist-1.7.5-gcc-8.2.1-z6bvhby
 34) plasma-18.11.1-gcc-8.2.1-g4kvnvg
 35) pumi-2.2.0-gcc-8.2.1-dd674x6
 36) slepc-3.10.1-gcc-8.2.1-w7otlnl
 37) strumpack-3.1.1-gcc-8.2.1-zcntogv
 38) tasmanian-6.0-gcc-8.2.1-qsse4mb
 39) xsdk-0.4.0-gcc-8.2.1-vq54mbz

xSDK 0.4.0 platform testing

xSDK 0.4.0 has been updated/fixed on a regular basis on various workstations (and more)

In collaboration with ALCF, NERSC, and OLCF xSDK packages are tested on key machines at these DOE computing facilities.

  • ALCF: Theta: Cray XC40 with Intel compilers [in KNL mode]
    • Theta front end nodes use Xeon processors and the compute nodes use KNL processors. Due to this difference – usually the builds on the compile/front-end nodes are done in cross compile mode – this does not work well with  all xSDK packages. xSDK build on Theta is done in stages:
    • download package sources on the front end node 
      module load cce
spack fetch --dependencies xsdk ^adol-c@develop~examples
spack stage adol-c
    • apply work-around patch for Intel compiler issue with petsc+mfem
    • build the packages on the compute node – by allocating a long enough 1 node job (if possible – say 24h) and run the following script 
      #!/bin/bash -x
module load cce
aprun -cc none -n 1 python /home/balay/spack/bin/spack install --dont-restage -j 16 xsdk ^adol-c@develop~examples
    • Relevant .spack config files for this build are at:
      cray-cnl6-mic_knl / intel@18.0.0.128
  • NERSC: Cori: Cray XC40 with Intel compilers [in Haswell mode]
    • build packages on the compile/fronet-end node with: 
      spack install --dirty -j8 xsdk
    • Relevant .spack config files for this build are at:
      cray-cnl6-haswell / intel@18.0.3.222
  • OLCF: Titan: Cray XK7 with GNU compilers (in Interlagos mode).
    • Titan’s login/service nodes use AMD Istanbul processors and the compute nodes use AMD Interlagos processors. The code compiled for login/service nodes will run on compute nodes as long as the dynamic libraries are accessible (some file systems might not be mounted). The code compiled for compute nodes crashes on login/service nodes with “Illegal instruction” exception.
    • There are many GNU compilers available on Titan. The successfully tested one was version 7.2.0. Version 6.3.0 generates a compiler bug for some of the xSDK packages. Other versions were not extensively tested.
    • deal.II had to be disabled when building xSDK 0.4.0 due to an issue with building SuiteSparse: 
      spack install xsdk~dealii
    • Some xSDK 0.4.0 packages have to be installed with modified dependencies: 
      depends_on('trilinos@12.14.0-rc1+hypre+superlu-dist+metis~mumps+nox~dtk~intrepid2~shards~suite-sparse+amesos~amesos2~anasazi+aztec~belos~boost+ifpack~ifpack2~fortran~exodus~gtest~hdf5+tpetra+kokkos+ml+muelu+sacado+zoltan~zoltan2',when='@0.4.0')
depends_on('petsc@3.10.3+trilinos+mpi+hypre+superlu-dist+metis+hdf5~mumps+double~int64', when='@0.4.0')
    • Some system modules have be disabled and built from scratch on login/service nodes (Perl and Python) because of issues with location of installed files. Some more details are covered in PETSc issue 208. These packages have to be built for login/service nodes.
    • Some system modules have to be modified by changing the environment variables they insert into the build environment. This is described in an issue 7629 
      module load cmake3/3.9.0
unset CMAKE_PREFIX_PATH
    • Relevant Spack config files for this build are at cray-cnl5-interlagos / gcc@7.2.0