This page discusses the platforms on which the xSDK 0.3.0 release has been tested and contains general instructions for building, as well as more specific instructions for selected high-end computing systems. See also details about obtaining the xSDK.
As more informational becomes available about building the xSDK 0.3.0 release on different platforms, that information will be posted here. Check back for updates.
xSDK 0.3.0 general build instructions
1. Edit packages.yaml file to specify the system-build software packages for the xSDK installation.
Although Spack is updated regularly, we recommend using preinstalled build tools like compilers, python, and cmake. On macOS, in addition to Xcode, we’ve used packages from Homebrew. Such preinstalled packages can be specified to spack in $HOME/.spack/packages.yaml or $SPACK_ROOT/etc/spack/defaults/packages.yaml config files. The following is an example for macOS:
# -------------------------------------------------------------------------
# This file controls default concretization preferences for Spack.
#
# Settings here are versioned with Spack and are intended to provide
# sensible defaults out of the box. Spack maintainers should edit this
# file to keep it current.
#
# Users can override these settings by editing the following files.
#
# Per-spack-instance settings (overrides defaults):
# $SPACK_ROOT/etc/spack/packages.yaml
#
# Per-user settings (overrides default and site settings):
# ~/.spack/packages.yaml
# -------------------------------------------------------------------------
packages:
autoconf:
paths:
autoconf@2.69: /usr/local
buildable: False
automake:
paths:
automake@1.15.1: /usr/local
buildable: False
libtool:
paths:
libtool@2.4.6: /usr/local
buildable: False
m4:
paths:
m4@1.4.18: /usr/local
buildable: False
cmake:
paths:
cmake@3.9.3: /usr/local
buildable: False
pkg-config:
paths:
pkg-config@1.3.9: /usr
buildable: False
python:
paths:
python@2.7.13: /usr
buildable: False
all:
providers:
mpi: [mpich]
compiler: [clang@9.0.0-apple]
After the edit, xSDK packages and external dependencies (as shown in the above figure) can be installed with a single command:
spack install xsdk
2. Install environment modules.
Environment modules are not typically pre-installed in a generic Linux client or Mac OS X environment. Skip this process if your environment already has modules installed. Otherwise, Spack manages the installation for you.
spack install environment-modules
After the installation, the module is enabled by the following command line.
For bash:
source `spack location -i environment-modules`/Modules/init/bash
For tcsh:
source `spack location -i environment-modules`/Modules/init/tcsh
3. Load xSDK module and its sub-modules.
Now you can load the xSDK environment. Try Spack’s load command with the option “-r” (resolve all dependencies):
spack load -r xsdk
Then, module list generates the following output, for example:
Currently Loaded Modulefiles: 1) mpich-3.2-clang-9.0.0-apple-gea6brx 2) zlib-1.2.11-clang-9.0.0-apple-t74siw3 3) hdf5-1.10.1-clang-9.0.0-apple-gvokovx 4) openblas-0.2.20-clang-9.0.0-apple-ub6yqod 5) hypre-2.12.1-clang-9.0.0-apple-f7cpjmd 6) metis-5.1.0-clang-9.0.0-apple-yfvcoac 7) parmetis-4.0.3-clang-9.0.0-apple-djkgfcb 8) superlu-dist-5.2.2-clang-9.0.0-apple-xwp4jhs 9) bzip2-1.0.6-clang-9.0.0-apple-vfmdedx 10) boost-1.65.1-clang-9.0.0-apple-lkttyum 11) glm-0.9.7.1-clang-9.0.0-apple-mwipot5 12) matio-1.5.9-clang-9.0.0-apple-sibb6bn 13) netcdf-4.4.1.1-clang-9.0.0-apple-bxqwcqn 14) trilinos-12.12.1-clang-9.0.0-apple-buov2ak 15) petsc-3.8.2-clang-9.0.0-apple-6g5mown 16) pflotran-xsdk-0.3.0-clang-9.0.0-apple-2cgk44d 17) alquimia-xsdk-0.3.0-clang-9.0.0-apple-dvkcatz 18) sundials-3.1.0-clang-9.0.0-apple-4dppkej 19) mfem-3.3.2-clang-9.0.0-apple-rjxuwlu 20) xsdk-0.3.0-clang-9.0.0-apple-gaiex6h
xSDK 0.3.0 platform testing
xSDK 0.3.0 has been updated/fixed on a regular basis in collaboration with ALCF, NERSC, and OLCF to fully test xSDK packages on key machines at these DOE computing facilities. The Spack xSDK installer has been tested on the following machines:
- Linux with GNU (4.9, 5,3, 5.4- with CUDA-9, 6.3, and 7.2) compilers.
- Mac OS X – El Capitan with Clang-8.0.0 and High Sierra with Clang-9.0.0 and GNU 7.2 compilers (excludes MAGMA package)
- DOE high-end computing systems at ALCF, OLCF, NERSC:
- ALCF: Theta: Cray XC40 with Intel compilers.
- Theta front-end nodes use xeon processors and the compute nodes use KNL processors. Due to this difference, the builds on the front-end nodes are done in cross-compile mode, which does not currently work well for all xSDK packages. The xSDK build on Theta takes place in two stages:
- Download sources on the front end node:
spack fetch --dependencies xsdk
- Build the packages on the compute node — by allocating a long enough 1 node job (if possible – say 24 hours) and run the following script:
#!/bin/bash -x module load cce aprun -n 1 python /home/balay/spack/bin/spack install -j 50 xsdk
Due to node allocation restrictions, for this test we built some packages on the front-end node and some on a compute node.
- Download sources on the front end node:
- compilers.yaml
-
compilers: - compiler: environment: {} extra_rpaths: [] flags: {} modules: - PrgEnv-intel - intel/18.0.0.128 operating_system: CNL paths: cc: cc cxx: CC f77: ftn fc: ftn spec: intel@18.0.0.128 target: any - packages.yaml
-
packages: cmake: paths: cmake@3.5.2%intel@18.0.0.128 arch=cray-CNL-mic_knl: /usr buildable: False python: paths: python@2.7.13%intel@18.0.0.128 arch=cray-CNL-mic_knl: /usr buildable: False pkg-config: paths: pkg-config@0.28%intel@18.0.0.128 arch=cray-CNL-mic_knl: /usr buildable: False autoconf: paths: autoconf@2.69%intel@18.0.0.128 arch=cray-CNL-mic_knl: /usr buildable: False automake: paths: automake@1.13.4%intel@18.0.0.128 arch=cray-CNL-mic_knl: /usr buildable: False libtool: paths: libtool@2.4.2%intel@18.0.0.128 arch=cray-CNL-mic_knl: /usr buildable: False m4: paths: m4@1.4.16%intel@18.0.0.128 arch=cray-CNL-mic_knl: /usr buildable: False intel-mkl: paths: intel-mkl@18.0.0.128%intel@18.0.0.128 arch=cray-CNL-mic_knl: /opt/intel buildable: False mpich: modules: mpich@7.6.3%intel@18.0.0.128 arch=cray-CNL-mic_knl: cray-mpich/7.6.3 buildable: False boost: paths: boost@1.64.0%intel@18.0.0.128 arch=cray-CNL-mic_knl: /soft/libraries/boost/1.64.0/intel buildable: False hdf5: modules: hdf5@1.8.16%intel@18.0.0.128+hl+fortran arch=cray-CNL-mic_knl: cray-hdf5-parallel/1.8.16 buildable: False all: providers: mpi: [mpich] blas: [intel-mkl] lapack: [intel-mkl] compiler: [intel@18.0.0.128] - NERSC: Edison: Cray with Gnu compilers
-
-
-
- Exact options used on Edison:
module load mercurial spack install xsdk%gcc@6.1.0
- Exact options used on Edison:
- packages.yaml
packages: mpich: modules: mpich@7.4.1%gcc@6.1.0 arch=cray-CNL-ivybridge : cray-mpich/7.4.1 buildable: False hdf5: modules: hdf5@1.8.12%gcc@6.1.0 arch=cray-CNL-ivybridge : cray-hdf5-parallel/1.8.12 buildable: False netcdf: modules: netcdf@4.3.1%gcc@6.1.0 arch=cray-CNL-ivybridge : cray-netcdf-hdf5parallel/4.3.1 buildable: False zlib: modules: zlib@1.2.7%gcc@6.1.0 arch=cray-CNL-ivybridge : zlib/1.2.7 buildable: False boost: modules: boost@1.54%gcc@6.1.0 arch=cray-CNL-ivybridge : boost/1.54 buildable: False cmake: modules: cmake@2.8.12.2%gcc@6.1.0 arch=cray-CNL-ivybridge : cmake/2.8.12.2 buildable: False python: paths: python@2.7.9%gcc@6.1.0 arch=cray-CNL-ivybridge : /usr/common/software/python/2.7.9 buildable: False all: providers: mpi: [mpich]
-
-
-
- OLCF: Titan: Cray with gnu compilers.
- Load the appropriate modules:
-
module load modules module swap PrgEnv-pgi PrgEnv-gnu module swap gcc gcc/5.3.0 module load cmake3/3.6.1 module load cray-hdf5-parallel/1.10.0.1 module load mercurial module load python/2.7.9 module load cudatoolkit
Clone spack
cd ${ROOT_DIR} git clone https://github.com/llnl/spack.git cd ${ROOT_DIR}/spack git checkout xsdk-0.3.0 - Create packages.yaml
packages: mpich: modules: mpich@7.4.0%gcc@5.3.0 arch=cray: cray-mpich/7.4.0 mpich@7.4.0%gcc@5.3.0 arch=cray-CNL-interlagos: cray-mpich/7.4.0 buildable: False cmake: paths: cmake@3.6.1%gcc@5.3.0 arch=cray-CNL-interlagos: /autofs/nccs-svm1_sw/titan/.swci/0-login/opt/spack/20170612/linux-suse_linux11-x86_64/gcc-4.3.4/cmake-3.9.0-owxiriblogovogl5zbrg45ulm3ln34cx/bin/cmake buildable: False hdf5: paths: hdf5@1.10.0%gcc@5.3.0+hl+fortran arch=cray-CNL-interlagos: /opt/cray/hdf5-parallel/1.10.0.1/GNU/5.1 buildable: False boost: modules: boost@1.60.0%gcc@5.3.0 arch=cray-CNL-interlagos: boost/1.60.0 buildable: False perl: modules: perl@5.20.2%gcc@5.3.0 arch=cray-CNL-interlagos: perl/5.20.2 buildable: False python: modules: python@2.7.9%gcc@5.3.0 arch=cray-CNL-interlagos: python/2.7.9 python@3.5.1%gcc@5.3.0 arch=cray-CNL-interlagos: python/3.5.1 buildable: False mercurial: modules: mercurial@2.6.3%gcc@5.3.0 arch=cray-CNL-interlagos: mercurial/2.6.3 buildable: False magma: modules: magma@1.6.2%gcc@5.3.0 arch=cray-CNL-interlagos: magma/1.6.2 buildable: False all: providers: mpi: [mpich] compiler: [gcc@5.3.0] - Create compilers.yaml (optional, spack will generate, see link)
- Install:
./spack -vd install xsdk@xsdk-0.3.0 arch=cray-CNL-interlagos
- OLCF: Titan: Cray with gnu compilers.