Installing the Software

This page discusses the platforms on which the xSDK 0.3.0 release has been tested and contains general instructions for building, as well as more specific instructions for select high-end computing systems. See also details about obtaining the xSDK.

As more information becomes available for building the xSDK 0.3.0 release on different platforms, that information will be posted here..  Check back for updates.

xSDK 0.3.0 general build instructions

1. Setup spack compilers
spack compiler find

Spack compiler configuration is stored in $HOME/.spack/$UNAME/compilers.yaml and can be checked with

spack compiler list
2. Edit packages.yaml file to specify the system-build software packages for xSDK installation.

Although Spack is updated regularly, we recommend using preinstalled build tools like compilers, python, cmake.  On macOS, in addition to Xcode, we’ve used packages from Homebrew.  Such preinstalled packages can be specified to spack in $HOME/.spack/packages.yaml or  $SPACK_ROOT/etc/spack/defaults/packages.yaml config files.    The following is an example for macOS

# -------------------------------------------------------------------------
# This file controls default concretization preferences for Spack.
#
# Settings here are versioned with Spack and are intended to provide
# sensible defaults out of the box. Spack maintainers should edit this
# file to keep it current.
#
# Users can override these settings by editing the following files.
#
# Per-spack-instance settings (overrides defaults):
#   $SPACK_ROOT/etc/spack/packages.yaml
#
# Per-user settings (overrides default and site settings):
#   ~/.spack/packages.yaml
# -------------------------------------------------------------------------
packages:
  autoconf:
    paths:
      autoconf@2.69: /usr/local
    buildable: False
  automake:
    paths:
      automake@1.15.1: /usr/local
    buildable: False
  libtool:
    paths:
      libtool@2.4.6: /usr/local
    buildable: False
  m4:
    paths:
      m4@1.4.18: /usr/local
    buildable: False
  cmake:
    paths:
      cmake@3.9.3: /usr/local
    buildable: False
  pkg-config:
    paths:
      pkg-config@1.3.9: /usr
    buildable: False
  python:
    paths:
      python@2.7.13: /usr
    buildable: False
  all:
    providers:
      mpi: [mpich]
    compiler: [clang@9.0.0-apple]
3. Install xSDK

After the edit, xSDK packages and external dependencies (as shown in the above figure) can be installed with a single command:

 spack install xsdk
4. Install environment modules.

Environment modules are not typically pre-installed in a generic Linux client or macOS environment. Skip this process if your environment already has modules installed.   Otherwise, Spack manages the installation for you.

 spack install environment-modules

After the installation, the module is enabled by the following command line.

For bash:

source  `spack location -i environment-modules`/Modules/init/bash

For tcsh:

source  `spack location -i environment-modules`/Modules/init/tcsh
5. Load xSDK module and its sub-modules.

Now you can load xSDK environment. Try Spack’s load command with the -r (resolve all dependencies) option:

spack load -r xsdk

Then, module list generates the following output, for example:

Currently Loaded Modulefiles:
  1) mpich-3.2-clang-9.0.0-apple-gea6brx
  2) zlib-1.2.11-clang-9.0.0-apple-t74siw3
  3) hdf5-1.10.1-clang-9.0.0-apple-gvokovx
  4) openblas-0.2.20-clang-9.0.0-apple-ub6yqod
  5) hypre-2.12.1-clang-9.0.0-apple-f7cpjmd
  6) metis-5.1.0-clang-9.0.0-apple-yfvcoac
  7) parmetis-4.0.3-clang-9.0.0-apple-djkgfcb
  8) superlu-dist-5.2.2-clang-9.0.0-apple-xwp4jhs
  9) bzip2-1.0.6-clang-9.0.0-apple-vfmdedx
 10) boost-1.65.1-clang-9.0.0-apple-lkttyum
 11) glm-0.9.7.1-clang-9.0.0-apple-mwipot5
 12) matio-1.5.9-clang-9.0.0-apple-sibb6bn
 13) netcdf-4.4.1.1-clang-9.0.0-apple-bxqwcqn
 14) trilinos-12.12.1-clang-9.0.0-apple-buov2ak
 15) petsc-3.8.2-clang-9.0.0-apple-6g5mown
 16) pflotran-xsdk-0.3.0-clang-9.0.0-apple-2cgk44d
 17) alquimia-xsdk-0.3.0-clang-9.0.0-apple-dvkcatz
 18) sundials-3.1.0-clang-9.0.0-apple-4dppkej
 19) mfem-3.3.2-clang-9.0.0-apple-rjxuwlu
 20) xsdk-0.3.0-clang-9.0.0-apple-gaiex6h

xSDK 0.3.0 platform testing

xSDK 0.3.0 has been updated/fixed on a regular basis in collaboration with ALCF, NERSC, and OLCF to fully test xSDK packages on key machines at these DOE computing facilities.  The Spack xSDK installer has been tested on the following machines:

  • Linux with GNU (4.9, 5,3, 5.4- with CUDA-9, 6.3, and 7.2) compilers.
  • macOS (excludes magma) – El Capitan with Clang-8.0.0, and High Sierra with Clang-9.0.0 and GNU 7.2  compilers.
  • DOE high-end computing systems at ALCF, OLCF, NERSC:
    • ALCF: Theta: Cray XC40 with Intel compilers
      • Theta front end nodes use xeon processors and the compute nodes use KNL processors. Due to this difference – the builds on the front end nodes are done in cross compile mode – this does not work well with  all xSDK packages. xSDK build on Theta is done in two stages:
      • download package sources on the front end node
        module load cce
        spack fetch --dependencies xsdk
      • build the pacakges on the compute node – by allocating a long enough 1 node job (if possible – say 24h) and run the following script
        #!/bin/bash -x
        module load cce
        aprun -n 1 python /home/balay/spack/bin/spack install -j 50 xsdk
        
      • compilers.yaml
        compilers:
        - compiler:
            environment: {}
            extra_rpaths: []
            flags: {}
            modules:
            - PrgEnv-intel
            - intel/18.0.0.128
            operating_system: CNL
            paths:
              cc: cc
              cxx: CC
              f77: ftn
              fc: ftn
            spec: intel@18.0.0.128
            target: any
      • packages.yaml
        packages:
          cmake:
            paths:
              cmake@3.5.2%intel@18.0.0.128 arch=cray-CNL-mic_knl: /usr
            buildable: False
          python:
            paths:
              python@2.7.13%intel@18.0.0.128 arch=cray-CNL-mic_knl: /usr
            buildable: False
          pkg-config:
            paths:
              pkg-config@0.28%intel@18.0.0.128 arch=cray-CNL-mic_knl: /usr
            buildable: False
          autoconf:
            paths:
              autoconf@2.69%intel@18.0.0.128 arch=cray-CNL-mic_knl: /usr
            buildable: False
          automake:
            paths:
              automake@1.13.4%intel@18.0.0.128 arch=cray-CNL-mic_knl: /usr
            buildable: False
          libtool:
            paths:
              libtool@2.4.2%intel@18.0.0.128 arch=cray-CNL-mic_knl: /usr
            buildable: False
          m4:
            paths:
              m4@1.4.16%intel@18.0.0.128 arch=cray-CNL-mic_knl: /usr
            buildable: False
          intel-mkl:
            paths:
              intel-mkl@18.0.0.128%intel@18.0.0.128 arch=cray-CNL-mic_knl: /opt/intel
            buildable: False
          mpich:
            modules:
              mpich@7.6.3%intel@18.0.0.128 arch=cray-CNL-mic_knl: cray-mpich/7.6.3
            buildable: False
          boost:
            paths:
              boost@1.64.0%intel@18.0.0.128 arch=cray-CNL-mic_knl: /soft/libraries/boost/1.64.0/intel
            buildable: False
          hdf5:
            modules:
              hdf5@1.8.16%intel@18.0.0.128+hl+fortran arch=cray-CNL-mic_knl: cray-hdf5-parallel/1.8.16
            buildable: False
          all:
            providers:
              mpi: [mpich]
              blas: [intel-mkl]
              lapack: [intel-mkl]
            compiler: [intel@18.0.0.128]
    • NERSCEdison: Cray with Gnu compilers
      • Installed with:
        spack install xsdk%gcc@5.2.0
        spack install xsdk%intel@16.0.3.210
        
      • compilers.yaml
        compilers:
        - compiler:
            environment: {}
            extra_rpaths: []
            flags: {}
            modules:
            - PrgEnv-gnu
            - gcc/5.2.0
            operating_system: CNL
            paths:
              cc: cc
              cxx: CC
              f77: ftn
              fc: ftn
            spec: gcc@5.2.0
            target: any
        - compiler:
            environment: {}
            extra_rpaths: []
            flags: {}
            modules:
            - PrgEnv-intel
            - intel/16.0.3.210
            operating_system: CNL
            paths:
              cc: cc
              cxx: CC
              f77: ftn
              fc: ftn
            spec: intel@16.0.3.210
            target: any
        
      • packages.yaml
        packages:
          python:
            paths:
              python@2.7.14%gcc@5.2.0 arch=cray-CNL-ivybridge: /usr
              python@2.7.14%intel@16.0.3.210 arch=cray-CNL-ivybridge: /usr
              python@2.7.14%cce@8.6.1 arch=cray-CNL-ivybridge: /usr
            buildable: False
          perl:
            paths:
              perl@2.7.14%gcc@5.2.0 arch=cray-CNL-ivybridge: /usr
              perl@2.7.14%intel@16.0.3.210 arch=cray-CNL-ivybridge: /usr
              perl@2.7.14%cce@8.6.1 arch=cray-CNL-ivybridge: /usr
            buildable: False
          mpich:
            modules:
              mpich@7.6.0%gcc@5.2.0 arch=cray-CNL-ivybridge: cray-mpich
              mpich@7.6.0%intel@16.0.3.210 arch=cray-CNL-ivybridge: cray-mpich
              mpich@7.6.0%cce@8.6.1 arch=cray-CNL-ivybridge: cray-mpich
            buildable: False
          hdf5:
            modules:
              hdf5@1.10.0.3%gcc@5.2.0+hl+fortran arch=cray-CNL-ivybridge: cray-hdf5-parallel
              hdf5@1.10.0.3%intel@16.0.3.210+hl+fortran arch=cray-CNL-ivybridge: cray-hdf5-parallel
              hdf5@1.10.0.3%cce@8.6.1+hl+fortran arch=cray-CNL-ivybridge: cray-hdf5-parallel
            buildable: False
          cmake:
            modules:
              cmake@3.8.2%gcc@5.2.0 arch=cray-CNL-ivybridge: cmake
              cmake@3.8.2%intel@16.0.3.210 arch=cray-CNL-ivybridge: cmake
              cmake@3.8.2%cce@8.6.1 arch=cray-CNL-ivybridge: cmake
            buildable: False
          boost:
            modules:
              boost@1.65.1%gcc@5.2.0 arch=cray-CNL-ivybridge: boost
              boost@1.65.1%intel@16.0.3.210 arch=cray-CNL-ivybridge: boost
              boost@1.65.1%cce@8.6.1 arch=cray-CNL-ivybridge: boost
            buildable: False
          all:
            compiler: [gcc@5.2.0, intel/16.0.3.210, cce/8.6.1]
            providers:
              mpi: [mpich]
        
    • OLCF: Titan: Cray with gnu compilers.
      • Load the appropriate modules:
        module load modules
        module swap PrgEnv-pgi PrgEnv-gnu
        module swap gcc gcc/5.3.0
        module load cmake3/3.6.1
        module load cray-hdf5-parallel/1.10.0.1
        module load mercurial
        module load python/2.7.9
        module load cudatoolkit
      • Install:
        spack install xsdk arch=cray-CNL-interlagos
        
      • compilers.yaml
      • packages.yaml
        packages:
          mpich:
            modules:
              mpich@7.5.2%gcc@5.3.0 arch=cray-CNL-interlagos: cray-mpich/7.5.2
            buildable: False
          cmake:
            paths:
              cmake@3.6.1%gcc@5.3.0 arch=cray-CNL-interlagos: /autofs/nccs-svm1_sw/titan/.swci/0-login/opt/spack/20170612/linux-suse_linux11-x86_64/gcc-4.3.4/cmake-3.9.0-owxiriblogovogl5zbrg45ulm3ln34cx/bin/cmake
            buildable: False
          hdf5:
            paths:
              hdf5@1.10.0%gcc@5.3.0+hl+fortran arch=cray-CNL-interlagos: /opt/cray/hdf5-parallel/1.10.0.1/GNU/5.1
            buildable: False
          boost:
            modules:
              boost@1.60.0%gcc@5.3.0 arch=cray-CNL-interlagos: boost/1.60.0
            buildable: False
          perl:
            modules:
              perl@5.20.2%gcc@5.3.0 arch=cray-CNL-interlagos: perl/5.20.2
            buildable: False
          python:
            modules:
              python@2.7.9%gcc@5.3.0 arch=cray-CNL-interlagos: python/2.7.9
              python@3.5.1%gcc@5.3.0 arch=cray-CNL-interlagos: python/3.5.1
            buildable: False
          mercurial:
            modules:
              mercurial@2.6.3%gcc@5.3.0 arch=cray-CNL-interlagos: mercurial/2.6.3
            buildable: False
          magma:
            modules:
              magma@1.6.2%gcc@5.3.0 arch=cray-CNL-interlagos: magma/1.6.2
            buildable: False
          all:
            providers:
              mpi: [mpich]
              compiler: [gcc@5.3.0]