Installing xSDK 0.3.0

This page discusses the platforms on which the xSDK 0.3.0 release has been tested and contains general instructions for building, as well as more specific instructions for selected high-end computing systems. See also details about obtaining the xSDK.

As more informational becomes available about building the xSDK 0.3.0 release on different platforms, that information will be posted here.  Check back for updates.

xSDK 0.3.0 general build instructions

1. Edit packages.yaml file to specify the system-build software packages for the xSDK installation.

Although Spack is updated regularly, we recommend using preinstalled build tools like compilers, python, and cmake. On macOS, in addition to Xcode, we’ve used packages from Homebrew. Such preinstalled packages can be specified to spack in $HOME/.spack/packages.yaml or $SPACK_ROOT/etc/spack/defaults/packages.yaml config files. The following is an example for macOS:

# -------------------------------------------------------------------------
# This file controls default concretization preferences for Spack.
#
# Settings here are versioned with Spack and are intended to provide
# sensible defaults out of the box. Spack maintainers should edit this
# file to keep it current.
#
# Users can override these settings by editing the following files.
#
# Per-spack-instance settings (overrides defaults):
#   $SPACK_ROOT/etc/spack/packages.yaml
#
# Per-user settings (overrides default and site settings):
#   ~/.spack/packages.yaml
# -------------------------------------------------------------------------
packages:
  autoconf:
    paths:
      autoconf@2.69: /usr/local
    buildable: False
  automake:
    paths:
      automake@1.15.1: /usr/local
    buildable: False
  libtool:
    paths:
      libtool@2.4.6: /usr/local
    buildable: False
  m4:
    paths:
      m4@1.4.18: /usr/local
    buildable: False
  cmake:
    paths:
      cmake@3.9.3: /usr/local
    buildable: False
  pkg-config:
    paths:
      pkg-config@1.3.9: /usr
    buildable: False
  python:
    paths:
      python@2.7.13: /usr
    buildable: False
  all:
    providers:
      mpi: [mpich]
    compiler: [clang@9.0.0-apple]

After the edit, xSDK packages and external dependencies (as shown in the above figure) can be installed with a single command:

 spack install xsdk
2. Install environment modules.

Environment modules are not typically pre-installed in a generic Linux client or Mac OS X environment. Skip this process if your environment already has modules installed.   Otherwise, Spack manages the installation for you.

 spack install environment-modules

After the installation, the module is enabled by the following command line.

For bash:

source  `spack location -i environment-modules`/Modules/init/bash

For tcsh:

source  `spack location -i environment-modules`/Modules/init/tcsh
3. Load xSDK module and its sub-modules.

Now you can load the xSDK environment. Try Spack’s load command with the option “-r” (resolve all dependencies):

spack load -r xsdk

Then, module list generates the following output, for example:

Currently Loaded Modulefiles:
  1) mpich-3.2-clang-9.0.0-apple-gea6brx
  2) zlib-1.2.11-clang-9.0.0-apple-t74siw3
  3) hdf5-1.10.1-clang-9.0.0-apple-gvokovx
  4) openblas-0.2.20-clang-9.0.0-apple-ub6yqod
  5) hypre-2.12.1-clang-9.0.0-apple-f7cpjmd
  6) metis-5.1.0-clang-9.0.0-apple-yfvcoac
  7) parmetis-4.0.3-clang-9.0.0-apple-djkgfcb
  8) superlu-dist-5.2.2-clang-9.0.0-apple-xwp4jhs
  9) bzip2-1.0.6-clang-9.0.0-apple-vfmdedx
 10) boost-1.65.1-clang-9.0.0-apple-lkttyum
 11) glm-0.9.7.1-clang-9.0.0-apple-mwipot5
 12) matio-1.5.9-clang-9.0.0-apple-sibb6bn
 13) netcdf-4.4.1.1-clang-9.0.0-apple-bxqwcqn
 14) trilinos-12.12.1-clang-9.0.0-apple-buov2ak
 15) petsc-3.8.2-clang-9.0.0-apple-6g5mown
 16) pflotran-xsdk-0.3.0-clang-9.0.0-apple-2cgk44d
 17) alquimia-xsdk-0.3.0-clang-9.0.0-apple-dvkcatz
 18) sundials-3.1.0-clang-9.0.0-apple-4dppkej
 19) mfem-3.3.2-clang-9.0.0-apple-rjxuwlu
 20) xsdk-0.3.0-clang-9.0.0-apple-gaiex6h

xSDK 0.3.0 platform testing

xSDK 0.3.0 has been updated/fixed on a regular basis in collaboration with ALCF, NERSC, and OLCF to fully test xSDK packages on key machines at these DOE computing facilities.  The Spack xSDK installer has been tested on the following machines:

  • Linux with GNU (4.9, 5,3, 5.4- with CUDA-9, 6.3, and 7.2) compilers.
  • Mac OS X – El Capitan with Clang-8.0.0 and High Sierra with Clang-9.0.0 and GNU 7.2  compilers (excludes MAGMA package)
  • DOE high-end computing systems at ALCF, OLCF, NERSC:
    • ALCF: Theta: Cray XC40 with Intel compilers.
    • Theta front-end nodes use xeon processors and the compute nodes use KNL processors. Due to this difference, the builds on the front-end nodes are done in cross-compile mode, which does not currently work well for all xSDK packages. The xSDK build on Theta takes place in two stages:
      • Download sources on the front end node:
        spack fetch --dependencies xsdk
      • Build the packages on the compute node — by allocating a long enough 1 node job (if possible – say 24 hours) and run the following script:
        #!/bin/bash -x
        module load cce
        aprun -n 1 python /home/balay/spack/bin/spack install -j 50 xsdk
        

        Due to node allocation restrictions, for this test we built some packages on the front-end node and some on a compute node.

    • compilers.yaml
    • compilers:
      - compiler:
          environment: {}
          extra_rpaths: []
          flags: {}
          modules:
          - PrgEnv-intel
          - intel/18.0.0.128
          operating_system: CNL
          paths:
            cc: cc
            cxx: CC
            f77: ftn
            fc: ftn
          spec: intel@18.0.0.128
          target: any
    • packages.yaml
    • packages:
        cmake:
          paths:
            cmake@3.5.2%intel@18.0.0.128 arch=cray-CNL-mic_knl: /usr
          buildable: False
        python:
          paths:
            python@2.7.13%intel@18.0.0.128 arch=cray-CNL-mic_knl: /usr
          buildable: False
        pkg-config:
          paths:
            pkg-config@0.28%intel@18.0.0.128 arch=cray-CNL-mic_knl: /usr
          buildable: False
        autoconf:
          paths:
            autoconf@2.69%intel@18.0.0.128 arch=cray-CNL-mic_knl: /usr
          buildable: False
        automake:
          paths:
            automake@1.13.4%intel@18.0.0.128 arch=cray-CNL-mic_knl: /usr
          buildable: False
        libtool:
          paths:
            libtool@2.4.2%intel@18.0.0.128 arch=cray-CNL-mic_knl: /usr
          buildable: False
        m4:
          paths:
            m4@1.4.16%intel@18.0.0.128 arch=cray-CNL-mic_knl: /usr
          buildable: False
        intel-mkl:
          paths:
            intel-mkl@18.0.0.128%intel@18.0.0.128 arch=cray-CNL-mic_knl: /opt/intel
          buildable: False
        mpich:
          modules:
            mpich@7.6.3%intel@18.0.0.128 arch=cray-CNL-mic_knl: cray-mpich/7.6.3
          buildable: False
        boost:
          paths:
            boost@1.64.0%intel@18.0.0.128 arch=cray-CNL-mic_knl: /soft/libraries/boost/1.64.0/intel
          buildable: False
        hdf5:
          modules:
            hdf5@1.8.16%intel@18.0.0.128+hl+fortran arch=cray-CNL-mic_knl: cray-hdf5-parallel/1.8.16
          buildable: False
        all:
          providers:
            mpi: [mpich]
            blas: [intel-mkl]
            lapack: [intel-mkl]
          compiler: [intel@18.0.0.128]
    • NERSCEdison: Cray with Gnu compilers
        • Exact options used on Edison:
          module load mercurial
          spack install xsdk%gcc@6.1.0

           

      • packages.yaml
        packages:
          mpich:
            modules:
              mpich@7.4.1%gcc@6.1.0 arch=cray-CNL-ivybridge : cray-mpich/7.4.1
            buildable: False
          hdf5:
            modules:
              hdf5@1.8.12%gcc@6.1.0 arch=cray-CNL-ivybridge : cray-hdf5-parallel/1.8.12
            buildable: False
          netcdf:
            modules:
               netcdf@4.3.1%gcc@6.1.0 arch=cray-CNL-ivybridge : cray-netcdf-hdf5parallel/4.3.1
            buildable: False
          zlib:
            modules:
              zlib@1.2.7%gcc@6.1.0 arch=cray-CNL-ivybridge : zlib/1.2.7
            buildable: False
          boost:
            modules:
              boost@1.54%gcc@6.1.0 arch=cray-CNL-ivybridge : boost/1.54
            buildable: False
          cmake:
            modules:
              cmake@2.8.12.2%gcc@6.1.0 arch=cray-CNL-ivybridge : cmake/2.8.12.2
            buildable: False
          python:
            paths:
              python@2.7.9%gcc@6.1.0 arch=cray-CNL-ivybridge : /usr/common/software/python/2.7.9
            buildable: False
          all:
            providers:
              mpi: [mpich]
    • OLCF: Titan: Cray with gnu compilers.
      • Load the appropriate modules:
      • module load modules
        module swap PrgEnv-pgi PrgEnv-gnu
        module swap gcc gcc/5.3.0
        module load cmake3/3.6.1
        module load cray-hdf5-parallel/1.10.0.1
        module load mercurial
        module load python/2.7.9
        module load cudatoolkit

        Clone spack

        cd ${ROOT_DIR}
        git clone https://github.com/llnl/spack.git
        cd ${ROOT_DIR}/spack
        git checkout xsdk-0.3.0
      • Create packages.yaml
        packages:
          mpich:
            modules:
              mpich@7.4.0%gcc@5.3.0 arch=cray: cray-mpich/7.4.0
              mpich@7.4.0%gcc@5.3.0 arch=cray-CNL-interlagos: cray-mpich/7.4.0
            buildable: False
          cmake:
            paths:
              cmake@3.6.1%gcc@5.3.0 arch=cray-CNL-interlagos: /autofs/nccs-svm1_sw/titan/.swci/0-login/opt/spack/20170612/linux-suse_linux11-x86_64/gcc-4.3.4/cmake-3.9.0-owxiriblogovogl5zbrg45ulm3ln34cx/bin/cmake
            buildable: False
          hdf5:
            paths:
              hdf5@1.10.0%gcc@5.3.0+hl+fortran arch=cray-CNL-interlagos: /opt/cray/hdf5-parallel/1.10.0.1/GNU/5.1
            buildable: False
          boost:
            modules:
              boost@1.60.0%gcc@5.3.0 arch=cray-CNL-interlagos: boost/1.60.0
            buildable: False
          perl:
            modules:
              perl@5.20.2%gcc@5.3.0 arch=cray-CNL-interlagos: perl/5.20.2
            buildable: False
          python:
            modules:
              python@2.7.9%gcc@5.3.0 arch=cray-CNL-interlagos: python/2.7.9
              python@3.5.1%gcc@5.3.0 arch=cray-CNL-interlagos: python/3.5.1
            buildable: False
          mercurial:
            modules:
              mercurial@2.6.3%gcc@5.3.0 arch=cray-CNL-interlagos: mercurial/2.6.3
            buildable: False
          magma:
            modules:
              magma@1.6.2%gcc@5.3.0 arch=cray-CNL-interlagos: magma/1.6.2
            buildable: False
          all:
            providers:
              mpi: [mpich]
              compiler: [gcc@5.3.0]
        
      • Create compilers.yaml (optional, spack will generate, see link)
      • Install:
        ./spack -vd install xsdk@xsdk-0.3.0 arch=cray-CNL-interlagos