Documentation du code de simulation numérique SUNFLUIDH

The code sunfluidh is programmed in fortran 95 and 2003. A makefile (named makefile) is present in the directory 'TRUNK/DATA/MAKEFILE_DIR' in order to help you to compile the code. Two compilers are available :

• the inlet compiler (ifort)
• the GNU compiler (gfortran)

Common information on fortran compiler's options can be found here.

You have to set up the makefile in accordance with the computing environment and the simulation type. In the file 'makefile', two sets of parameters are proposed in two sections named:

• Compiler options
• Code configuration options

These parameters define the compiler and compilation options used.

• COMPILER_TYPE=IFORT or GNU : compiler choice between ifort and gfortran. Default value is GNU
• LEVEL_OPT= DEBUG, OPT_O2, OPT_O3 : select the level of compilation, debug level, medium optimization level or high optimization level. Default value is OPT_O3
• SKYLAKE_OPT=YES or NO : enable (or not) some additional optimization options for skylake Intel architecture (Intel CPU). Options are : -axAVX,SSE4.2 -xCORE-AVX512 -qopt-zmm-usage=high. Default value is NO.
• REPORT_OPT=NO : Create a report about compiler's vectorization (optons= -qopt-report=3 -qopt-report-phase=vec). For Intel CPU only. Default value is NO
• AMD_OPT=YES or NO : enable (or not) some additional optimization options for AMD architecture (Intel CPU). Options are : -mavx2. Default value is NO
• ADD_USER_OPT= …. . The user can add its own compiler's options
• HOSTNAME=LOCAL, DEFAULT or GRAPPE : define how to access to external libraries used by the code (i.e lapack) in regard to the host environment (the computer). At present, three cases are predefined :
  • LOCAL (the default value) : the access path to external libraries is implicitly defined in the Operating System of the host. The usual fortran option “-l” is used.
  • DEFAULT : the access path to external libraries is enabled by means of the “module” command (see the host's documentation about “module”). No specific action in the makefile is required.
  • 'hostname' (i.e. GRAPPE) : the access path must be explicitly stated in the makefile. This procedure is defined for the cluster Grappe at present.

In order to configure the code in the suitable form (2D, 3D, parallelisation mode,…), labels are used to set the variable “CODE_CONFIG” defined inside the makefile. So, this variable must just be set as CODE_CONFIG= LABEL1 LABEL2 … The common labels to use are :

• -DDIM3D : for 3D geometrical configuration
• -DDIM2D : for 2D geometrical configuration
• -DMPI : for MPI parallelisation with domain docomposition approach
• -DOPENMP : Multithreading parallelisation using OpenMP
• -DSHARED_DISK : files are stored on disks shared by all processors (this default option must always be enable if possible)
• -DNOLIBS : Special flag to disable any compilation entailing the external libraries

Some other labels are available and can be used in specific cases

• -DHYPRE : link the HYPRE scientific Library
• -DHDF5 : output files formatted with hdf5 format (for Radiation DOM only)
• -DGET_MPI_GRAPHICAL_CONFIG : only to build the file that describes the MPI subdomain distribution according to a MPI graphical topology

The code works with some specific blocks of instructions selected during the compilation in respect with the simulation setup (2D or 3D, parallel computation, etc …). This works according to the principle of conditional compilation : instructions are selected and compiled by means of the compiler's option “-cpp” (with ifort and gfortran) and specific labels linked to the configuration wanted. For example :

   ifort  ... -cpp -DLABEL1 -DLABEL2  ...
   gfortran  ... -cpp -DLABEL1 -DLABEL2   ...
   

The available labels in the code sunfluidh are :

• DIM2D or DIM3D associated to the 2D or 3D configuration. One of these labels must be necessary present for the compilation.
• MPI : MPI parallelisation (based on the domain decomposition approach). The compilation must be performed with the mpi wrapper instead of the usual compiler command (for instance mpiifort in place of ifort, mpif90 in place of pgif95, mpifort in place of gfortran, …). Please contact the system administrator if you have any doubt.
• OPENMP : The OpenMP parallelization of the code is performed during the compilation. Don't forget to set also the usual OpenMP option for the fortran compiler : -fopenmp with gfortran, -openmp with ifort, -mp with pgf95, -qsmp=omp with xlf95.
• SHARED_DISK : This label is used in order to handle some backup/restart files of the code in the particular context of parallel computing on distributed systems. By default this label must be present (even if the computation is not parallel or parallel on shared systems). If all the hard disks of the computing system can be seen by every cores, this label must also be present. Otherwise, when each group of hard disk is only plugged into a group of cores (each core sees only a group of hard disk), this label must be removed. Please contact the code's developer (Yann Fraigneau) if you have any doubt on this part.
• HYPRE : HYPRE Library (with MPI)

These labels must be placed in the variable CODE_CONFIG (see examples in the makefile).

Set the different variables related to the paths in order to access to the suitable libraries. The path variables depend on our system environment. Sometimes they can be leave empty (library not used, access path already defined by the system)

• For LAPACK : PATH_LAPACK= …
• For SCALAPACK : PATH_SCALAPACK= …
• For HYPRE : PATH_HYPRE= … ; INCLUDE_HYPRE=…
• etc …

You can define new access path variables if needed (use of other libraries not explicitly mentioned here) and add them in the variables LDFLAGS_LIB (for the compile process) and/or LDFLAGS_EXE (for the link-edit process).