klepo/k-Wave-Fluid-OMP

Overview

k-Wave is an open source MATLAB toolbox designed for the time-domain simulation
of propagating acoustic waves in 1D, 2D, or 3D. The toolbox has a wide range of
functionality, but at its heart is an advanced numerical model that can account
for both linear or nonlinear wave propagation, an arbitrary distribution of
weakly heterogeneous material parameters, and power law acoustic absorption,
see the http://www.k-wave.org [1].

This project is a part of the k-Wave toolbox accelerating 2D/3D simulations
using an optimized C++ implementation to run moderate to big grid sizes (128x128
to 10,000x10,000) or (64x64x64 to 512x512x512).

This repository contains a modified C++ implementation of kpsaceFirstOrder-OMP
version 2.16 (04 September 2017) - release 1.2, which is extended by on-the-fly
compression and calculation of time-averaged acoustic intensity in time-domain
ultrasound simulations. These approaches were published in [2].

See also a modified CUDA/C++ implementation of kpsaceFirstOrder-CUDA
https://github.com/klepo/k-Wave-Fluid-CUDA or
https://github.com/klepo/Visualisation-of-the-outputs-of-k-Wave-ultrasound-simulations

Repository structure

.
+--Compression         - Helper class for the on-the-fly compression
+--Containers          - Matrix and output stream containers
+--Data                - Small test data
+--GetoptWin64         - Windows version of the getopt routine
+--Hdf5                - HDF5 classes (file access)
+--KSpaceSolver        - Solver classes with all the kernels
+--Logger              - Logger class to report progress and errors
+--MatrixClasses       - Matrix classes to hold data
+--Makefiles           - GNU makefiles for different systems
+--OutputHDF5Streams   - Output streams to sample data
+--Parameters          - Parameters of the simulation
+--Utils               - Utility routines
+--nbproject           - NetBeans IDE 8.2 project
+--qtproject           - QtCreator IDE project
+--visualstudioproject - Visual Studio IDE project
.clang-format          - Clang-Format config file
Changelog.md           - Changelog
License.md             - License file
Makefile               - NetBeans 8.2 makefile
Readme.md              - Readme
Doxyfile               - Doxygen generator file
header_bg.png          - Doxygen logo
main.cpp               - Main file of the project

Compilation

The source codes of kpsaceFirstOrder-OMP are written using the C++-11
standard, the OpenMP 4.0 library, FFTW 3.3.8 or MKL 11 and HDF5 1.8.x library

There are variety of different C++ compilers that can be used to compile the
source codes. We recommend using either the GNU C++ compiler (gcc/g++) version
7.3 and higher, or the Intel C++ compiler version 2018 and higher. Please note
that Visual Studio compilers do not support OpenMP 4.0 standard and cannot be
used thus. Be also aware of Intel compilers 2017 and their MKL bug producing
incorrect results when AVX2 is enabled. The codes can be compiled on 64-bit
Linux and Windows. 32-bit systems are not supported!

This section describes the compilation procedure using GNU and Intel compilers
on Linux. (The Windows users are encouraged to download the Visual Studio 2015
project and compile it using Intel Compiler from within Visual Studio.)

Before compiling the code, it is necessary to install a C++ compiler and several
libraries. The GNU compiler is usually part of Linux distributions and
distributed as open source. It can be downloaded from http://gcc.gnu.org/)
if necessary.

The Intel compiler can be downloaded from
http://software.intel.com/en-us/intel-composer-xe/. This package also includes
the Intel MKL (Math Kernel Library)library that contains FFT. The Intel compiler
is only free for non-commercial use.

The code also relies on several libraries that are to be installed before
compiling:

HDF5 library - Mandatory I/O library, version 1.8.x,
https://support.hdfgroup.org/HDF5/release/obtain518.html.
FFTW library - Optional library for FFT, version 3.3.x,
http://www.fftw.org/.
MKL library - Optional library for FFT, version 2015 or higher
http://software.intel.com/en-us/intel-composer-xe/.

Although it is possible to use any combination of the FFT library and the
compiler, the best performance is observed when using GNU compiler and FFTW,
or Intel Compiler and Intel MKL.

The HDF5 library installation procedure

Download the 64-bit HDF5 library
https://support.hdfgroup.org/HDF5/release/obtain518.html. Please use version
1.8.x, the version 1.10.x is not compatible with MATLAB yet.
Configure the HDF5 distribution. Enable the high-level library and specify
an installation folder by typing:
```
./configure --enable-hl --prefix=folder_to_install
```
Make the HDF library by typing:
```
make -j
```
Install the HDF5 library by typing:
```
make install
```

The FFTW library installation procedure

Download the FFTW library package for your platform,
http://www.fftw.org/download.html).
Configure the FFTW distribution. Enable OpenMP support, and desired SIMD
instruction set, single precision floating point arithmetic, and specify an
installation folder:
```
./configure --enable-single --enable-sse --enable-openmp --enable-shared \
            --prefix=folder_to_install
```
if you intend to use the FFTW library (and the C++ code) only on a selected
machine and want to get the best possible performance, you may also add
processor specific optimisations and AVX or AVX2 instructions set. Note,
the compiled binary code is not likely to be portable on different CPUs.
SSE2 version will work on any processor, AVX on Intel Sandy Bridge and
newer, AVX2 on Intel Haswell and newer.
```
./configure --enable-single --enable-avx --enable-openmp  --enable-shared \
            --with-gcc-arch=native --prefix=folder_to_install
```
More information about the installation and customization can be found at
http://www.fftw.org/fftw3_doc/Installation-and-Customization.htm.
For recent CPUs based on Sandy Bridge, Ivy Bridge, Haswell and Broadwell with
strongly recommend to use the AVX and AVX2 support.
Make the FFTW library by typing:
```
make -j
```
Install the FFTW library by typing:
```
make install
```

The Intel Compiler and MKL installation procedure

Download the Intel Composer XE package for your platform
http://software.intel.com/en-us/intel-compilers.
Run the installation script and follow the procedure by typing:
```
./install.sh
```

Compiling the C++ code on Linux

After the libraries and the compiler have been installed, you are ready to
compile the kspaceFirstOrder-OMP code.

Select the most appropriate makefile.
We recommend Makefiles/Release/Makefile
The Makefile supports code compilation under GNU compiler and FFTW, or Intel
compiler with MKL. Uncomment the desired compiler by removing character #.
```
#COMPILER = GNU
#COMPILER = Intel
```
Select how to link the libraries. Static linking is preferred as it may be
a bit faster, however, on some systems (HPC clusters) it may be better to
use dynamic linking and use the system specific libraries at runtime.
```
#LINKING = STATIC
#LINKING = DYNAMIC
```
Select the instruction set and the CPU architecture.
For users who will only use the binary on the same machine as compiled,
the best choice is CPU_ARCH=native. If you are about to run the same
binary on different machines or you want to cross-compile the code, you are
free to use any of the possible choices, where SSE 3 is the most general but
slowest and AVX2 is the most recent instruction set while believed to be the
fastest one.
```
#CPU_ARCH = native
#CPU_ARCH = SSE3
#CPU_ARCH = SSE4
#CPU_ARCH = AVX
#CPU_ARCH = AVX2
```
Set installation paths of the libraries (an example is shown bellow). Zlib
and SZIP may be required if the compression is switched on.
```
MKL_DIR=
FFT_DIR=
HDF5_DIR=
ZLIB_DIR=
SZIP_DIR=
```
Compile the source code by typing:
```
make -j
```
If you want to clean the distribution, type:
```
make clean
```

Alternatively, the code can be compiled using Netbeans IDE and attached project.

Usage

The C++ codes offers a lot of parameters and output flags to be used. For more
information, please type:

./kspaceFirstOrder-OMP --help

References

[1]
B. E. Treeby and B. T. Cox, "k-Wave: MATLAB toolbox for the simulation and
reconstruction of photoacoustic wave-fields," J. Biomed. Opt., vol. 15, no. 2,
p. 021314, 2010.

[2]
P. Kleparnik, P. Zemcik, B. E. Treeby and J. Jaros, "On-the-Fly Calculation of
Time-Averaged Acoustic Intensity in Time-Domain Ultrasound Simulations Using
a k-Space Pseudospectral Method," in IEEE Transactions on Ultrasonics,
Ferroelectrics, and Frequency Control, vol. 69, no. 10, pp. 2917-2929, Oct. 2022,
doi: 10.1109/TUFFC.2022.3199173.