Motivation
Combinations of multiple software packages developed by different groups are essential for large-scale computational science, where the capabilities needed for modeling, simulation, and analysis are broader than any single team has resources to address. The often tedious trial-and-error process of obtaining, configuring, and installing any single tool may arguably be manageable. However, from the perspective of an end-user application scientist, handling each tool’s installation idiosyncrasies can easily become overwhelming when dealing with several packages in combination. Worse, such problems are compounded by the need for consistency among packages to be used within the same application in terms of compiler, compiler versions and options, and common third-party packages such as BLAS and HDF5.
Spack
A goal of xSDK is to simplify the process of working with various xSDK packages in combination. Spack is a package manager for high-performance computing. We thus provide an xSDK Spack metapackage that includes many xSDK numerical libraries (AMReX, ArborX*, ButterflyPACK, Data Transfer Toolkit (DTK), deal.ii, Ginkgo, HeFFTe, Hypre, libEnsemble, MAGMA, MFEM, Omega_h, PETSc/TAO, PHIST, PLASMA, preCICE, PUMI, SLATE, SLEPc, STRUMPACK, SUNDIALS, SuperLU_dist, Tasmanian, and Trilinos ) as well as some commonly needed external packages.
Spack allows the xSDK packages to be deployed with a single command. The user can optionally choose compilers, MPI implementation, and build options. We are working toward support of combinatorial test dashboards for all xSDK packages.
*Denotes a package released as part of the xSDK for the first time in version 0.7.0.
xSDK 0.7.0 release components
Final release date: November 12, 2021
The xSDK spack metapackage (version 0.7.0) includes tested release versions of the following xSDK packages:
- AMReX
- ArborX
- ButterflyPACK
- Data Transfer Toolkit (DTK)
- deal.ii
- Ginkgo
- HeFFTe
- Hypre
- libEnsemble
- MAGMA
- MFEM
- Omega_h
- PETSc/TAO
- PHIST
- PLASMA
- preCICE
- PUMI
- SLATE
- SLEPc
- STRUMPACK
- SUNDIALS
- SuperLU
- Tasmanian
- Trilinos
- Alquimia
- PFLOTRAN
Users who prefer to work with just one xSDK package rather than several can use Spack to install only that particular package. Alternatively, they can bypass Spack and instead obtain the package directly from its website and install according to instructions there.
Packages using CUDA
The following packages can be installed with CUDA feature enabled
- PETSc
- SUNDIALS
- MAGMA
- STRUMPACK
- Tasmanian
- Ginkgo
- HeFFTe
- SLATE
- Hypre
- MFEM
- AMReX
Packages using ROCM
The following packages can be installed with ROCM feature enabled, but require special handling, see the installxSDK repository for further information
- AMReX
- MFEM
- SUNDIALS
- MAGMA
- Tasmanian
- Ginkgo
- HeFFTe
Open source packages installed with xSDK packages
Users can view a tree of dependencies for any Spack package, including xSDK, by running ‘spack spec xsdk’. This is an example of the dependency tree for the xSDK metapackage (with cuda enabled) as of version 0.7.0:
Input spec
--------------------------------
xsdk+cuda cuda_arch=70 arch=linux-None-skylake_avx512
^cuda@11.2.0
^openmpi
Concretized
--------------------------------
xsdk@0.7.0%gcc@9.2.0+alquimia+arborx+butterflypack+cuda+datatransferkit+dealii+ginkgo+heffte+libensemble+omega-h+phist+precice~rocm+slate+strumpack+trilinos amdgpu_target=none cuda_arch=70 arch=linux-centos7-skylake_avx512
^alquimia@1.0.9%gcc@9.2.0~ipo+shared build_type=RelWithDebInfo arch=linux-centos7-skylake_avx512
^cmake@3.19.4%gcc@9.2.0~doc+ncurses+openssl+ownlibs~qt build_type=Release patches=b48396c0e4f61756248156b6cebe9bc0d7a22228639b47b5aa77c9330588ce88 arch=linux-centos7-skylake_avx512
^hdf5@1.10.7%gcc@9.2.0~cxx+fortran+hl~ipo~java+mpi+shared~szip~threadsafe+tools api=default build_type=RelWithDebInfo arch=linux-centos7-skylake_avx512
^numactl@2.0.14%gcc@9.2.0 patches=4e1d78cbbb85de625bad28705e748856033eaafab92a66dffd383a3d7e00cc94,62fc8a8bf7665a60e8f4c93ebbd535647cebf74198f7afafec4c085a8825c006,ff37630df599cfabf0740518b91ec8daaf18e8f288b19adaae5364dc1f6b2296 arch=linux-centos7-skylake_avx512
^autoconf@2.69%gcc@9.2.0 patches=7793209b33013dc0f81208718c68440c5aae80e7a1c4b8d336e382525af791a7 arch=linux-centos7-skylake_avx512
^automake@1.16.3%gcc@9.2.0 arch=linux-centos7-skylake_avx512
^libtool@2.4.6%gcc@9.2.0 arch=linux-centos7-skylake_avx512
^m4@1.4.18%gcc@9.2.0+sigsegv patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 arch=linux-centos7-skylake_avx512
^openmpi@4.1.1%gcc@9.2.0~atomics+cuda~cxx~cxx_exceptions+gpfs~internal-hwloc~java~legacylaunchers~lustre~memchecker~pmi~pmix~singularity~sqlite3+static~thread_multiple+vt+wrapper-rpath fabrics=none schedulers=none arch=linux-centos7-skylake_avx512
^cuda@11.2.0%gcc@9.2.0~dev arch=linux-centos7-skylake_avx512
^hwloc@2.6.0%gcc@9.2.0~cairo+cuda~gl~libudev+libxml2~netloc~nvml~opencl+pci~rocm+shared arch=linux-centos7-skylake_avx512
^libpciaccess@0.16%gcc@9.2.0 arch=linux-centos7-skylake_avx512
^pkgconf@1.8.0%gcc@9.2.0 arch=linux-centos7-skylake_avx512
^util-macros@1.19.3%gcc@9.2.0 arch=linux-centos7-skylake_avx512
^libxml2@2.9.12%gcc@9.2.0~python arch=linux-centos7-skylake_avx512
^libiconv@1.16%gcc@9.2.0 libs=shared,static arch=linux-centos7-skylake_avx512
^xz@5.2.2%gcc@9.2.0~pic libs=shared,static arch=linux-centos7-skylake_avx512
^zlib@1.2.7%gcc@9.2.0+optimize+pic+shared arch=linux-centos7-skylake_avx512
^ncurses@6.2%gcc@9.2.0~symlinks+termlib abi=none arch=linux-centos7-skylake_avx512
^libevent@2.1.12%gcc@9.2.0+openssl arch=linux-centos7-skylake_avx512
^openssl@1.1.1l%gcc@9.2.0~docs certs=system arch=linux-centos7-skylake_avx512
^perl@5.16.3%gcc@9.2.0+cpanm+shared+threads patches=0eac10ed90aeb0459ad8851f88081d439a4e41978e586ec743069e8b059370ac,3bbd7d6f9933d80b9571533867b444c6f8f5a1ba0575bfba1fba4db9d885a71a arch=linux-centos7-skylake_avx512
^openssh@8.7p1%gcc@9.2.0 arch=linux-centos7-skylake_avx512
^libedit@3.1-20210216%gcc@9.2.0 arch=linux-centos7-skylake_avx512
^pflotran@3.0.2%gcc@9.2.0 arch=linux-centos7-skylake_avx512
^petsc@3.16.1%gcc@9.2.0~X~batch~cgns~complex+cuda~debug+double~exodusii~fftw~giflib+hdf5~hpddm~hwloc+hypre~int64~jpeg~knl~libpng~libyaml~memkind+metis~mkl-pardiso~mmg~moab~mpfr+mpi~mumps~openmp~p4est~parmmg~ptscotch~random123~rocm~saws~scalapack+shared~strumpack~suite-sparse+superlu-dist~tetgen~trilinos~valgrind amdgpu_target=none clanguage=C cuda_arch=70 arch=linux-centos7-skylake_avx512
^diffutils@3.3.5%gcc@9.2.0 arch=linux-centos7-skylake_avx512
^hypre@2.23.0%gcc@9.2.0~complex+cuda~debug+fortran~int64~internal-superlu~mixedint+mpi~openmp+shared+superlu-dist~unified-memory cuda_arch=70 arch=linux-centos7-skylake_avx512
^openblas@0.3.18%gcc@9.2.0~bignuma~consistent_fpcsr~ilp64+locking+pic+shared threads=openmp arch=linux-centos7-skylake_avx512
^superlu-dist@7.1.1%gcc@9.2.0~cuda~int64~ipo~openmp+shared build_type=RelWithDebInfo cuda_arch=none arch=linux-centos7-skylake_avx512
^metis@5.1.0%gcc@9.2.0~gdb~int64~real64+shared build_type=Release patches=4991da938c1d3a1d3dea78e49bbebecba00273f98df2a656e38b83d55b281da1,b1225da886605ea558db7ac08dd8054742ea5afe5ed61ad4d0fe7a495b1270d2 arch=linux-centos7-skylake_avx512
^parmetis@4.0.3%gcc@9.2.0~gdb~int64~ipo+shared build_type=RelWithDebInfo patches=4f892531eb0a807eb1b82e683a416d3e35154a455274cf9b162fb02054d11a5b,50ed2081bc939269689789942067c58b3e522c269269a430d5d34c00edbc5870,704b84f7c7444d4372cb59cca6e1209df4ef3b033bc4ee3cf50f369bce972a9d arch=linux-centos7-skylake_avx512
^python@3.8.7%gcc@9.2.0+bz2+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib patches=0d98e93189bc278fbc37a50ed7f183bd8aaf249a8e1670a465f0db6bb4f8cf87,4c2457325f2b608b1b6a2c63087df8c26e07db3e3d493caf36a56f0ecf6fb768,f2fd060afc4b4618fe8104c4c5d771f36dc55b1db5a4623785a4ea707ec72fb4 arch=linux-centos7-skylake_avx512
^amrex@21.10%gcc@9.2.0~amrdata+cuda~eb~fortran~hdf5~hypre~ipo+linear_solvers+mpi~openmp~particles~petsc~pic~plotfile_tools~rocm~shared+sundials amdgpu_target=none build_type=RelWithDebInfo cuda_arch=70 dimensions=3 precision=double arch=linux-centos7-skylake_avx512
^sundials@5.8.0%gcc@9.2.0+ARKODE+CVODE+CVODES+IDA+IDAS+KINSOL+cuda+examples+examples-install~f2003~fcmix+generic-math+hypre~int64~ipo~klu~lapack~monitoring+mpi~openmp+petsc~pthread~raja~rocm+shared+static+superlu-dist~superlu-mt~sycl+trilinos amdgpu_target=none build_type=RelWithDebInfo cuda_arch=70 precision=double arch=linux-centos7-skylake_avx512
^trilinos@13.2.0%gcc@9.2.0~adios2+amesos+amesos2+anasazi+aztec~basker+belos+boost~chaco~complex~cuda~cuda_rdc~debug~dtk+epetra+epetraext~epetraextbtf~epetraextexperimental~epetraextgraphreorderings~exodus+explicit_template_instantiation~float+fortran~gtest+hdf5+hypre+ifpack+ifpack2~intrepid+intrepid2~ipo~isorropia+kokkos~mesquite~minitensor+ml+mpi+muelu~mumps+nox~openmp~phalanx~piro~python~rol~rythmos+sacado~scorec+shards+shared~shylu~stk~stokhos+stratimikos~strumpack~suite-sparse~superlu+superlu-dist~teko~tempus+tpetra~trilinoscouplings~wrapper~x11+zoltan+zoltan2 build_type=RelWithDebInfo cuda_arch=none cxxstd=14 gotype=int arch=linux-centos7-skylake_avx512
^boost@1.76.0%gcc@9.2.0+atomic+chrono~clanglibcpp~container~context~coroutine+date_time~debug+exception~fiber+filesystem+graph~icu+iostreams+locale+log+math~mpi+multithreaded~numpy~pic+program_options~python+random+regex+serialization+shared+signals~singlethreaded+system~taggedlayout+test+thread+timer~versionedlayout+wave cxxstd=98 visibility=hidden arch=linux-centos7-skylake_avx512
^bzip2@1.0.6%gcc@9.2.0~debug~pic+shared arch=linux-centos7-skylake_avx512
^arborx@1.1%gcc@9.2.0~cuda~ipo+mpi~openmp~rocm+serial~sycl~trilinos build_type=RelWithDebInfo arch=linux-centos7-skylake_avx512
^kokkos@3.4.01%gcc@9.2.0~aggressive_vectorization~compiler_warnings~cuda~cuda_constexpr~cuda_lambda~cuda_ldg_intrinsic~cuda_relocatable_device_code~cuda_uvm~debug~debug_bounds_check~debug_dualview_modify_check~deprecated_code~examples~explicit_instantiation~hpx~hpx_async_dispatch~hwloc~ipo~memkind~numactl~openmp~pic+profiling~profiling_load_print~pthread~qthread~rocm+serial+shared~sycl~tests~tuning~wrapper amdgpu_target=none build_type=RelWithDebInfo cuda_arch=none std=14 arch=linux-centos7-skylake_avx512
^butterflypack@2.0.0%gcc@9.2.0~ipo+shared build_type=RelWithDebInfo arch=linux-centos7-skylake_avx512
^arpack-ng@3.8.0%gcc@9.2.0+mpi+shared arch=linux-centos7-skylake_avx512
^netlib-scalapack@2.1.0%gcc@9.2.0~ipo~pic+shared build_type=Release patches=1c9ce5fee1451a08c2de3cc87f446aeda0b818ebbce4ad0d980ddf2f2a0b2dc4,f2baedde688ffe4c20943c334f580eb298e04d6f35c86b90a1f4e8cb7ae344a2 arch=linux-centos7-skylake_avx512
^datatransferkit@3.1-rc3%gcc@9.2.0~external-arborx~ipo~openmp+serial+shared build_type=RelWithDebInfo arch=linux-centos7-skylake_avx512
^dealii@9.3.2%gcc@9.2.0~adol-c~arborx+arpack~assimp~cuda~doc+examples~ginkgo~gmsh+gsl+hdf5~int64~ipo+metis+mpi+muparser~nanoflann~netcdf+oce~optflags+p4est+petsc~python+scalapack~simplex+slepc~sundials~symengine+threads+trilinos build_type=DebugRelease cuda_arch=none cxxstd=default arch=linux-centos7-skylake_avx512
^gsl@2.7%gcc@9.2.0~external-cblas arch=linux-centos7-skylake_avx512
^intel-tbb@2020.3%gcc@9.2.0~ipo+shared+tm build_type=RelWithDebInfo cxxstd=default patches=62ba015ebd1819c45bef47411540b789b493e31ca668c4ff4cb2afcbc306b476,ce1fb16fb932ce86a82ca87cf0431d1a8c83652af9f552b264213b2ff2945d73,d62cb666de4010998c339cde6f41c7623a07e9fc69e498f2e149821c0c2c6dd0 arch=linux-centos7-skylake_avx512
^muparser@2.2.6.1%gcc@9.2.0 arch=linux-centos7-skylake_avx512
^ninja@1.10.2%gcc@9.2.0 arch=linux-centos7-skylake_avx512
^oce@0.18.3%gcc@9.2.0~X11+tbb arch=linux-centos7-skylake_avx512
^p4est@2.8%gcc@9.2.0+mpi~openmp arch=linux-centos7-skylake_avx512
^slepc@3.16.0%gcc@9.2.0+arpack~blopex~cuda~rocm amdgpu_target=none cuda_arch=none arch=linux-centos7-skylake_avx512
^suite-sparse@5.10.1%gcc@9.2.0~cuda~graphblas~openmp+pic~tbb arch=linux-centos7-skylake_avx512
^gmp@6.2.1%gcc@9.2.0 arch=linux-centos7-skylake_avx512
^mpfr@4.1.0%gcc@9.2.0 arch=linux-centos7-skylake_avx512
^autoconf-archive@2019.01.06%gcc@9.2.0 arch=linux-centos7-skylake_avx512
^texinfo@6.5%gcc@9.2.0 patches=12f6edb0c6b270b8c8dba2ce17998c580db01182d871ee32b7b6e4129bd1d23a,1732115f651cff98989cb0215d8f64da5e0f7911ebf0c13b064920f088f2ffe1 arch=linux-centos7-skylake_avx512
^ginkgo@1.4.0%gcc@9.2.0+cuda~develtools~full_optimizations~hwloc~ipo~openmp~rocm+shared amdgpu_target=none build_type=Release cuda_arch=70 patches=43e571a49ea774793d89d145571c95ae7cdf991f5220d8fe84db217a3c613a32 arch=linux-centos7-skylake_avx512
^heffte@2.2.0%gcc@9.2.0+cuda+fftw~fortran~ipo+magma~mkl~python~rocm+shared amdgpu_target=none build_type=RelWithDebInfo cuda_arch=70 arch=linux-centos7-skylake_avx512
^fftw@3.3.10%gcc@9.2.0+mpi~openmp~pfft_patches precision=double,float arch=linux-centos7-skylake_avx512
^magma@2.6.1%gcc@9.2.0+cuda+fortran~ipo~rocm+shared amdgpu_target=none build_type=RelWithDebInfo cuda_arch=70 arch=linux-centos7-skylake_avx512
^mfem@4.3.0%gcc@9.2.0~amgx~conduit+cuda~debug+examples~gnutls~gslib~lapack~libceed~libunwind+metis+miniapps~mpfr+mpi~netcdf~occa~openmp+petsc~pumi~raja~rocm~shared~slepc+static~strumpack~suite-sparse+sundials+superlu-dist~threadsafe~umpire+zlib amdgpu_target=none cuda_arch=70 patches=5d0451e9ad09fdfc6a8114604a1a9eeef78fe5d8b12b139514a1fe7c2b07f3d1,ccc98d6d874a760218c9f6e4bcf3a97ea79746b0f28997f4312fcf02a0beae6e timer=auto arch=linux-centos7-skylake_avx512
^omega-h@9.34.1%gcc@9.2.0~examples~ipo+mpi+optimize+shared+symbols~throw+trilinos~warnings+zlib build_type=RelWithDebInfo arch=linux-centos7-skylake_avx512
^phist@1.9.5%gcc@9.2.0~fortran~host~int64~ipo+mpi~openmp~parmetis~scamac+shared~trilinos build_type=RelWithDebInfo kernel_lib=tpetra outlev=2 arch=linux-centos7-skylake_avx512
^plasma@21.8.29%gcc@9.2.0~ipo~lua+shared build_type=RelWithDebInfo arch=linux-centos7-skylake_avx512
^precice@2.3.0%gcc@9.2.0~ipo+mpi+petsc~python+shared build_type=RelWithDebInfo arch=linux-centos7-skylake_avx512
^eigen@3.4.0%gcc@9.2.0~ipo build_type=RelWithDebInfo arch=linux-centos7-skylake_avx512
^pumi@2.2.6%gcc@9.2.0~fortran~int64~ipo~shared+simmodsuite_version_check~testing~zoltan build_type=RelWithDebInfo simmodsuite=none arch=linux-centos7-skylake_avx512
^py-libensemble@0.8.0%gcc@9.2.0~deap~mpi~mpmath~nlopt+petsc4py~pyyaml~scipy~tasmanian arch=linux-centos7-skylake_avx512
^py-numpy@1.21.4%gcc@9.2.0+blas+lapack patches=873745d7b547857fcfec9cae90b09c133b42a4f0c23b6c2d84cf37e2dd816604 arch=linux-centos7-skylake_avx512
^py-cython@0.29.24%gcc@9.2.0 arch=linux-centos7-skylake_avx512
^py-setuptools@58.2.0%gcc@9.2.0 arch=linux-centos7-skylake_avx512
^py-petsc4py@3.16.1%gcc@9.2.0+mpi patches=d344e0e8b4c5ebb96961e1905bdd749162c52b151538238609f4dec77e2a6360 arch=linux-centos7-skylake_avx512
^py-mpi4py@3.1.2%gcc@9.2.0 arch=linux-centos7-skylake_avx512
^py-psutil@5.8.0%gcc@9.2.0 arch=linux-centos7-skylake_avx512
^slate@2021.05.02%gcc@9.2.0+cuda~ipo+mpi+openmp~rocm+shared amdgpu_target=none build_type=RelWithDebInfo cuda_arch=70 arch=linux-centos7-skylake_avx512
^blaspp@2021.04.01%gcc@9.2.0+cuda~ipo+openmp~rocm+shared amdgpu_target=none build_type=RelWithDebInfo cuda_arch=none arch=linux-centos7-skylake_avx512
^lapackpp@2021.04.00%gcc@9.2.0~ipo+shared build_type=RelWithDebInfo arch=linux-centos7-skylake_avx512
^strumpack@6.1.0%gcc@9.2.0+butterflypack+c_interface~count_flops~cuda~ipo+mpi~openmp+parmetis~rocm~scotch+shared~slate~task_timers+zfp amdgpu_target=none build_type=RelWithDebInfo cuda_arch=none arch=linux-centos7-skylake_avx512
^zfp@0.5.5%gcc@9.2.0~aligned~c~cuda~fasthash~fortran~ipo~openmp~profile~python+shared~strided~twoway bsws=64 build_type=RelWithDebInfo cuda_arch=none arch=linux-centos7-skylake_avx512
^tasmanian@7.7%gcc@9.2.0+blas+cuda~fortran~ipo+magma+mpi+openmp~python~rocm+xsdkflags amdgpu_target=none build_type=Release cuda_arch=70 arch=linux-centos7-skylake_avx512
The following open source HPC packages are installed by default with the xSDK. Note that you may override these and use a preinstalled version if you wish.