Installing with Spack

Motivation

Combinations of multiple software packages developed by different groups are essential for large-scale computational science, where the capabilities needed for modeling, simulation, and analysis are broader than any single team has resources to address. The often tedious trial-and-error process of obtaining, configuring, and installing any single tool may arguably be manageable. However, from the perspective of an end-user application scientist, handling each tool’s installation idiosyncrasies can easily become overwhelming when dealing with several packages in combination. Worse, such problems are compounded by the need for consistency among packages to be used within the same application in terms of compiler, compiler versions and options, and common third-party packages such as BLAS and HDF5.

Spack

A goal of the xSDK is to simplify the process of working with various xSDK packages in combination. We thus provide an xSDK Spack metapackage that includes, by default, the four initial xSDK numerical libraries (hypre, PETSc,SuperLU_dist, and Trilinos) as well as some commonly needed external packages.

Obtaining and installing the xSDK using Spack

See details about obtaining and using spack.

Once spack has been installed, the xSDK can be installed in the simplest case using:

./spack install xsdk

xSDK alpha release 0.2.0

Release date: February 28, 2017

The xSDK spack metapackage (version 0.2.0) includes tested development versions of the following xSDK packages:

  • hypre
  • PETSc
  • SuperLU, SuperLU_DIST
  • Trilinos
  • Alquimia
  • PFLOTRAN

Note: Chombo is not currently included in the 0.2.0 alpha version of the xSDK, but has satisfied all xSDK required policies.

xSDK 0.2.0 platform testing

Work is currently in progress in collaboration with ALCF, NERSC, and OLCF to fully test xSDK packages on key machines at these DOE computing facilities.  Below is the current status.  Check back for updates.

The spack xSDK installer has been tested on the following machines:

      • Linux with Gnu compilers, both static and shared libraries.
      • Mac OS X, both static and shared libraries.
        • The xSDKTrilinos package fails to build with shared libraries (reported to developers).
        • One ctest example fails with static libraries (reported to developers).
      • DOE high-end computing systems at ALCF, OLCF, NERSC:
              • ALCF: Mira: IBM BlueGene with Gnu compilers
                • Cannot build with IBM compilers or gcc-4.4.7 since they do not support C++11 needed by Trilinos.
                • The hdf5 supplied for gcc-4.8.4 does not have the fortran .mod files needed for some packages
                •  Set up the gcc-4.8.4 toolchain
                • soft add +bgqtoolchain-gcc484
                •  compilers.yaml
                • compilers:
- compiler:
    modules: []
    operating_system: cnk
    paths:
       cc: /soft/compilers/gcc/4.8.4/bin/powerpc64-bgq-linux-gcc
       cxx: soft/compilers/gcc/4.8.4/bin/powerpc64-bgq-linux-g++
       f77: soft/compilers/gcc/4.8.4/bin/powerpc64-bgq-linux-gfortran
       fc: /soft/compilers/gcc/4.8.4/bin/powerpc64-bgq-linux-gfortran
    spec: gcc@4.8.4
                • packages.yaml
                • packages:
  mpich:
    paths:     mpich@3%gcc@4.8.4:bgsys/drivers/ppcfloor/comm/gcc
    buildable: False

  netcdf:
    paths:
      netcdf@4.3.3%gcc@4.8.4: /soft/libraries/netcdf/4.3.3-f4.4.1/cnk-gcc/V1R2M2-20150514
      buildable: False

  hdf5:
    paths:
      hdf5@1.8.12%gcc@4.8.4: /home/sarich/software/hdf5/1.8.12/gnu484
    buildable: False

  cmake:
    paths:
      cmake@3: /soft/buildtools/cmake/current
    buildable: False

  zlib:
    paths:
      zlib@1.2.11: /soft/libraries/alcf/current/gcc/ZLIB
    buildable: False

  python:
    paths:
      python@2.6: /usr
    buildable: False

  m4:
    paths:
      m4@1.4.17%gcc@4.8.4+sigsegv: /usr
    buildable: False

  all:
    compiler: [gcc@4.8.4]
    variants: ~shared
    providers:
      mpi: [mpich@3]
      blas: [netlib-lapack]
      lapack: [netlib-lapack]

                   

              • NERSC: Edison: Cray with Gnu compilers
                    • Exact options used on Edison: 
                      module load mercurial
module load python
spack install xsdk
                    • compilers.yaml 
                      compilers:
- compiler:
    environment: {}
    extra_rpaths: []
    flags: {}
    modules:
    - PrgEnv-gnu
    - gcc/6.1.0
    operating_system: CNL
    paths:
      cc: /opt/cray/craype/2.5.5/bin/cc
      cxx: /opt/cray/craype/2.5.5/bin/CC
      f77: /opt/cray/craype/2.5.5/bin/ftn
      fc: /opt/cray/craype/2.5.5/bin/ftn
    spec: gcc@6.1.0
                    • packages.yaml 
                      packages:
  mpich:
    modules:
      mpich@7.4.1%gcc@6.1.0 arch=cray-CNL-ivybridge : cray-mpich/7.4.1
    buildable: False
  hdf5:
    modules:
      hdf5@1.8.12%gcc@6.1.0 arch=cray-CNL-ivybridge : cray-hdf5-parallel/1.8.12
    buildable: False
  netcdf:
    modules:
       netcdf@4.3.1%gcc@6.1.0 arch=cray-CNL-ivybridge : cray-netcdf-hdf5parallel/4.3.1
    buildable: False
  zlib:
    modules:
      zlib@1.2.7%gcc@6.1.0 arch=cray-CNL-ivybridge : zlib/1.2.7
    buildable: False
  boost:
    modules:
      boost@1.54%gcc@6.1.0 arch=cray-CNL-ivybridge : boost/1.54
    buildable: False
  cmake:
    modules:
      cmake@2.8.12.2%gcc@6.1.0 arch=cray-CNL-ivybridge : cmake/2.8.12.2
    buildable: False
  python:
    paths:
      python@2.7.9%gcc@6.1.0 arch=cray-CNL-ivybridge : /usr/common/software/python/2.7.9
    buildable: False
  all:
    providers:
      mpi: [mpich]
        • OLCF: Titan: Cray with gnu compilers.
          • Exact options used on Titan:
          • module swap PrgEnv-pgi PrgEnv-gnu
module load cray-hdf5-parallel
module load cray-netcdf-hdf5parallel
module load cmake3
            More information to come

Users who prefer to work with just one xSDK package rather than several can use spack to install only that particular package.  Alternatively, they can bypass spack and instead obtain the package directly from its website and install according to instructions there.

Open source packages installed with xSDK packages

Users can view a tree of dependencies for any spack package, including xSDK, by running spack spec.  This is the dependency tree for the xSDK metapackage:

xsdk
    ^hypre@develop~internal-superlu
        ^blas
        ^lapack
        ^mpi
    ^petsc@develop~boost+hdf5+hypre+metis+mpi~mumps+superlu-dist+trilinos
        ^hdf5@1.8.12:+fortran+mpi
            ^zlib@1.2.5:
        ^parmetis
            ^cmake@2.8:
            ^metis@5:
        ^python@2.6:2.7
            ^bzip2
            ^ncurses
            ^openssl
                ^perl@5.14.0:
            ^readline
            ^sqlite
        ^sowing
        ^trilinos@develop+boost+hdf5+hypre+metis~mumps~suite-sparse+superlu-dist+xsdkflags
            ^boost
            ^glm
            ^matio
            ^netcdf+mpi
                ^m4
            ^superlu-dist@develop
    ^pflotran@develop
    ^xsdktrilinos@develop

The following open source HPC packages are installed by default with the xSDK.  Note that you may override these and use a preinstalled version if you wish.

          • Boost C++ headers and libraries
          • HDF5 low-level parallel IO library: version 1.8.12
          • NetCDF high-level parallel IO library
          • METIS graph partitioner: version 5.X
          • ParMETIS parallel graph partitioner

If Alquimia (an xSDK biogeochemistry package) is installed with the option “+alquimia”, then PFLOTRAN also needs to be installed, which is used by Alquimia as one of its chemistry engines.