# Using HONEE :::{note} Options in the documentation are specified by command-line flag and YAML format interchangeably. See {ref}`yaml-options` for details. ::: Setting up a problem in HONEE requires a few different things: - {ref}`opt-mesh` - {ref}`opt-governing-equation` - {ref}`opt-restart` - {ref}`bc-flags` - {ref}`opt-discretization` - {ref}`opt-solvers` - {ref}`opt-solution-output` Below we detail how to tell HONEE these individual pieces. (opt-mesh)= ## Mesh HONEE uses PETSc's DMPlex module for handling the mesh and its related data structures, and so is the interface point for this area. The mesh can either be generated by PETSc on the fly or input via a file. To use a mesh from a file, HONEE supports reading in meshes supported by DMPlex using `-dm_plex_filename`. Often, this is either a CGNS file or a GMSH file. :::{list-table} Mesh File Options :header-rows: 1 * - Option - Description * - `-dm_plex_filename` - Path to the mesh file to be read in (usually a CGNS or GMSH file) ::: To have PETSc generate a mesh, DMPlex offers many runtime options to determine the shape and parameters of the mesh. Shapes maybe a box, sphere, or many [other options](https://petsc.org/release/manualpages/DMPlex/DMPlexShape) and are specified with `-dm_plex_shape`. The dimensionality of the shape must be set with `-dm_plex_dim`. We focus on the box option here, specifically recommending `-dm_plex_shape zbox`[^1]. The number of elements in each direction is set via `-dm_plex_box_faces`, which is a comma-delimited list of the number of elements in the x, y, and z directions. To set the size of the box, use `-dm_plex_box_lower` and `-dm_plex_box_upper` to set the coordinates of the lower and upper corner of the box. :::{list-table} PETSc Generated Box Mesh Options :header-rows: 1 * - Option - Description * - `-dm_plex_shape` - Shape for mesh to be generated (usually `zbox`) * - `-dm_plex_dim` - Topological dimension for mesh to be generated (usually `zbox`) * - `-dm_plex_box_faces` - Number of elements in each direction * - `-dm_plex_box_lower` - Coordinates of the lower corner of the box * - `-dm_plex_box_upper` - Coordinates of the upper corner of the box * - `-dm_plex_box_bd` - Set periodic boundary conditions (i.e. `none,periodic,periodic`). See {ref}`bc-periodicity`. ::: The box meshes that PETSc generates are equispaced in every direction. HONEE offers some limited options of warping or transforming the mesh to better suit certain problems. Currently, only `platemesh` is supported, see {ref}`example-platemesh` for more details. :::{list-table} Mesh Transform Options :header-rows: 1 * - Option - Description * - `-mesh_transform` - Transformation to apply to the mesh `platemesh` ::: (opt-governing-equation)= ## Governing Equations The governing equations that HONEE will solve on the given mesh is determined by the `-problem` parameter. In addition to governing equation, `-problem` may also specify boundary conditions and initial conditions available to use for specific example problems (see {ref}`examples`). There are currently two primary governing equations supported: - Compressible Navier-Stokes (`newtonian`) - Advection-diffusion (`advection`) Using either of these options in `-problem` ignores any example-specific additions. While not always strictly governing equations, turbulence models may also be specified using `-sgs_model_type` (see {ref}`theory-sgs`). :::{list-table} Governing Equation Runtime Options :header-rows: 1 * - Option - Description - Default value * - `-problem` - Problem to solve (`advection`, `newtonian`, `density_current`, `euler_vortex`, `shocktube`, `blasius`, `channel`, `gaussian_wave`, and `taylor_green`) - `density_current` * - `-sgs_model_type` - Subgrid stress model to use (`none`, `data-driven`) - `none` ::: ### Compressible Navier-Stokes, Newtonian viscosity, Ideal Gas For problems that use the compressible Navier-Stokes governing equations (`newtonian`, `density_current`, `blasius`, `channel`, `gaussian_wave`, and `taylor_green`), the following options are universal for setting the parameters of them :::{list-table} Newtonian Ideal Gas Problems Runtime Options :header-rows: 1 * - Option - Description - Default value - Unit * - `-state_var` - State variables to solve solution with. `conservative` ($\rho, \rho \bm{u}, \rho e$), `primitive` ($P, \bm{u}, T$), or `entropy` ($\frac{\gamma - s}{\gamma - 1} - \frac{\rho}{P} (e - c_v T),\ \frac{\rho}{P} \bm{u},\ -\frac{\rho}{P}$) where $s = \ln(P\rho^{-\gamma})$ - `conservative` - string * - `-cv` - Heat capacity at constant volume - `717` - `J/(kg K)` * - `-cp` - Heat capacity at constant pressure - `1004` - `J/(kg K)` * - `-gravity` - Gravitational acceleration vector - `0,0,0` - `m/s^2` * - `-lambda` - Stokes hypothesis second viscosity coefficient - `-2/3` - * - `-mu` - Shear dynamic viscosity coefficient - `1.8e-5` - `Pa s` * - `-k` - Thermal conductivity - `0.02638` - `W/(m K)` * - `-stab` - Stabilization method (`none`, `su`, or `supg`) - `none` - * - `-Ctau_t` - Stabilization time constant, $C_t$. See {ref}`theory-ns-tau` - `1.0` - * - `-Ctau_v` - Stabilization viscous constant, $C_v$. See {ref}`theory-ns-tau` - `36, 60, 128 for degree = 1, 2, 3` - * - `-Ctau_C` - Stabilization continuity constant, $C_c$. See {ref}`theory-ns-tau` - `1.0` - * - `-Ctau_M` - Stabilization momentum constant, $C_m$. See {ref}`theory-ns-tau` - `1.0` - * - `-Ctau_E` - Stabilization energy constant, $C_E$. See {ref}`theory-ns-tau` - `1.0` - * - `-div_diff_flux_projection_method` - Method used to calculate divergence of diffusive flux projection (`none`, `direct`, or `indirect`). See {ref}`theory-divdiff` - `none` - * - `-div_diff_flux_projection_ksp*` - Control the KSP object for the projection of the divergence of diffusive flux. See {ref}`theory-divdiff` - N/A - ::: #### Newtonian Reference State Options These options set the Newtonian reference state. :::{list-table} Newtonian Reference State Runtime Options :header-rows: 1 * - Option - Description - Default value * - `-reference_pressure` - Reference/initial pressure. Also used as the default for `-ic_boundary_layer_wall_pressure`. - `1.01e5` * - `-reference_velocity` - Reference/initial velocity (3 values). Also used as the default for `-ic_boundary_layer_wall_velocity`. - `0,0,0` * - `-reference_temperature` - Reference/initial temperature. Also used as the default for `-ic_boundary_layer_wall_temperature`. - `288.15` ::: (opt-restart)= ## Initial Conditions / Restart Initial conditions are either created from an analytical definition or from a solution file. All the example problems have their own custom analytical initial conditions. See those examples for their custom IC options. The `newtonian` initial condition simply sets everything equal to the reference state given by `-reference*`. For `newtonian`, {ref}`aux-ics-boundary-layer` can also be used. To set initial conditions from a file, simply pass the path to the file to `-continue_filename`. This must either be a binary or CGNS file. Restarting from CGNS is the preferred method as it may be done in parallel. Loading from binary is used exclusively for testing purposes. To use the CGNS file as input, `-dm_plex_filename` must load the same file and `-dm_plex_cgns_parallel` must also be given. Restarting from a CGNS file created by HONEE will also load in the step number and solution time of the solution being loaded (see {ref}`opt-solution-output` for how to write out CGNS solution files). :::{list-table} File I/O Options :header-rows: 1 * - Option - Description * - `-continue_filename` - Path to file from which to continue from. Either binary file or CGNS * - `-dm_plex_cgns_parallel` - Turn on the parallel CGNS reader in DMPlex. Required if `-continue_filename` is a CGNS file. ::: (bc-flags)= ## Boundary Conditions Boundary conditions (BCs) are defined by a name and a list of face IDs that said BC applies to. These are collectively known as a `BCDefinition`. A list of `BCDefinition` names is specified by the user using `-bc_names`. The `BCDefinition` name will be used in the flag prefix `-bc_{name}_*` for every option pertaining to that `BCDefinition`. The list of face_ids used for a `BCDefinition` is specified using `-bc_{name}_face_ids`. For example: ```yaml bc_names: airfoil,inflow bc_airfoil: face_ids: 2,5 bc_inflow: face_ids: 1,3,4 ``` Essential boundary conditions (i.e. strong, Dirichlet, etc.) are defined by the component indices they constrain in the solution vector. They are specified by `-bc_{name}_essential_comps`. :::{note} The essential boundary conditions are specified by component indices rather than by their solution component name to inhibit invalid states. For example, if the user specifies an essential constraint on velocity and is running with entropy solution variables, that constraint cannot be set, and is thus an invalid state. Additionally, this more generic interface allows for expansion to other governing equations (e.g. RANS turbulence models, chemistry transport, etc.). ::: Natural boundary conditions (i.e. weak, Neumann, etc.) are specified by type (a string), from a list of possibilities given by the given governing equation and example problem. These possibilities are registered by the problem/governing equation and are specified by `-bc_{name}_natural_type`. :::{list-table} `BCDefinition` Options :header-rows: 1 * - Option - Description * - `-bc_names` - List of names used to reference boundary conditions * - `-bc_{name}_face_ids` - List of face label values to define BC * - `-bc_{name}_essential_comps` - Component indices to apply essential BC on * - `-bc_{name}_natural_type` - Type of natural BC to apply (`none`, `freestream`, `outflow`, etc.) ::: The `face_ids` are simply the label values of the `Face Sets` label on the DM. If the mesh is read in from file, the label values are determined from the file. For the case of PETSc-generated box meshes, those labels correspond to the following faces: :::{list-table} 2D Face ID Labels :header-rows: 1 * - PETSc Face Name - Cartesian direction - Face ID * - faceMarkerBottom - -z - 1 * - faceMarkerRight - +x - 2 * - faceMarkerTop - +z - 3 * - faceMarkerLeft - -x - 4 ::: :::{list-table} 3D Face ID Labels :header-rows: 1 * - PETSc Face Name - Cartesian direction - Face ID * - faceMarkerBottom - -z - 1 * - faceMarkerTop - +z - 2 * - faceMarkerFront - -y - 3 * - faceMarkerBack - +y - 4 * - faceMarkerRight - +x - 5 * - faceMarkerLeft - -x - 6 ::: (opt-discretization)= ## Discretization The order of the spatial discretization can be specified by `-degree`. HONEE also allows for using more than the required number of quadrature points when computing the finite element operators. This is controlled by `-q_extra`. :::{list-table} Discretization Runtime Options :header-rows: 1 * - Option - Description - Default value * - `-degree` - Polynomial degree of tensor product basis (must be >= 1) - `1` * - `-q_extra` - Number of extra quadrature points - `0` ::: (opt-solvers)= ## Solvers HONEE uses the PETSc solver stack for solving the defined problem, which is described below. :::{list-table} Solver Stack :header-rows: 1 * - Solver Name - Description - Options Prefix - Options Documentation - User Documentation * - TS - Timestepping - `-ts_*` - [TSSetFromOptions](https://petsc.org/release/manualpages/TS/TSSetFromOptions) - [PETSc Documentation](https://petsc.org/release/manual/ts/) * - SNES - Non-linear solver - `-snes_*` - [SNESSetFromOptions](https://petsc.org/release/manualpages/SNES/SNESSetFromOptions) - [PETSc Documentation](https://petsc.org/release/manual/snes/) * - KSP - Linear solver - `-ksp_*` - [KSPSetFromOptions](https://petsc.org/release/manualpages/KSP/KSPSetFromOptions) - [PETSc Documentation](https://petsc.org/release/manual/ksp/) * - PC - Preconditioner (for linear solver) - `-pc_*` - [PCSetFromOptions](https://petsc.org/release/manualpages/PC/PCSetFromOptions) - [PETSc Documentation](https://petsc.org/release/manual/ksp/#preconditioners) ::: The prefix options listed above will apply for the primary solve that HONEE performs. Other auxillary solves and functionality will use the same options, but with an additional prefix. For example, `-auxillary_ksp_*`. In addition to these options, there are a few HONEE-specific solver options as well. The method of timestepping is controlled by `-implicit`. Note this must match with the choice of timestepper set by `-ts_type`. :::{list-table} HONEE-specific Solver Runtime Options :header-rows: 1 * - Option - Description - Default value * - `-ceed` - CEED resource specifier - `/cpu/self/opt/blocked` * - `-implicit` - Use implicit time integrator formulation - `false` * - `-amat_type` - `shell` for the analytical matrix-free Jacobian - ::: ### Mass Matrix and Explicit Timestepping When using explicit timestepping, the inverted mass matrix must be applied to the residual. This matrix inversion is implemented as a KSP solve and is controlled by the `-mass_ksp*` prefix. By default, a lumped mass matrix is used, but modifying the KSP options can make it into a consistent mass matrix (specifically, setting `-mass_ksp_type` to something other than `pconly`). :::{list-table} Mass Matrix Runtime Options :header-rows: 1 * - Option - Description * - `-mass_ksp*` - Prefix for giving specific KSP options for the mass matrix inversion * - `-mass_pc*` - Prefix for giving specific PC options for the mass matrix inversion * - `-mass_ksp_view_pre_ts_solve` - View mass KSP once before `TSSolve()` is called ::: ### Nondimensionalization Problems may have solution values or parameters vary by orders of magnitude between them. For example, low-Mach flow using sea-level air properties would have velocities at $O(10)$, while pressures would be $O(10^5)$. These large differences in scales can lead to poorly-conditioned problems. To ease these issues, the values solved can be nondimensionalized. :::{caution} This feature may be broken for certain use cases. If you discover a bug related to nondimensionalization, please submit an issue to the HONEE repo so that we can address it. ::: :::{list-table} Nondimensionalization Options :header-rows: 1 * - Option - Description - Default value * - `-units_meter` - 1 meter in scaled length units - `1` * - `-units_second` - 1 second in scaled time units - `1` * - `-units_kilogram` - 1 kilogram in scaled mass units - `1` * - `-units_Kelvin` - 1 Kelvin in scaled temperature units - `1` ::: (opt-solution-output)= ## Solution Output HONEE offers the following options for writing out solution information. We divide this into complete solution output and post-processed output. ### Complete Solution Output :::{list-table} Solution Output Options :header-rows: 1 * - Option - Description - Default value * - `-ts_monitor_solution` - PETSc output format, such as `cgns:output-%d.cgns` (requires PETSc `--download-cgns`) - * - `-ts_monitor_solution_interval` - Number of time steps between visualization output frames. - `1` * - `-viewer_cgns_batch_size` - Number of frames written per CGNS file if the CGNS file name includes a format specifier (`%d`). - `20` * - `-ts_eval_times` - Sets intermediate time points to evaluate the solution at. See [PETSc documentation](https://petsc.org/main/manualpages/TS/TSSetEvaluationTimes/) for more details. - * - `-ts_eval_solutions_view` - PETSc output format for `-ts_eval_times` solutions to be written to - ::: The following options are options using the legacy output system. It is not advised to use them. :::{list-table} Legacy Solution Output Options :header-rows: 1 * - Option - Description - Default value * - `-checkpoint_interval` - Number of steps between writing binary checkpoints. `0` has no output, `-1` outputs final state only - `0` * - `-checkpoint_vtk` - Checkpoints include VTK (`*.vtu`) files for visualization. Consider `-ts_monitor_solution`instead. - `false` * - `-viz_refine` - Use regular refinement for VTK visualization - `0` * - `-output_dir` - Output directory for binary checkpoints and VTK files (if enabled). - `.` * - `-output_add_stepnum2bin` - Whether to add step numbers to output binary files - `false` ::: ### Post-Processed Solution Output Here are the options for monitoring specific quantities of interest of the solution as the solver progresses. Note that all of these options have an addition `*_interval` option which specifies the :::{list-table} Post-Processed Solution Output Options :header-rows: 1 * - Option - Description - Default value * - `-ts_monitor_cfl` - Viewer for the min/max CFL in the domain e.g., `ascii:cfl.csv:ascii_csv` to write a CSV file. - * - `-ts_monitor_wall_force` - Viewer for the force on each no-slip wall, e.g., `ascii:force.csv:ascii_csv` to write a CSV file. - * - `-ts_monitor_smartsim_solution` - Puts the solution into a SmartSim database. This does not take "normal" viewer inputs. - * - `-ts_monitor_total_kinetic_energy` - Viewer for the total kinetic energy in the domain and other terms, e.g., `ascii:total_ke.csv:ascii_csv` to write a CSV file. - * - `-ts_monitor_*_interval` - Number of timesteps between executing the monitor - `1` ::: ## Common Options HONEE is controlled via command-line options. The following options are common among all problem types: :::{list-table} Common Runtime Options :header-rows: 1 * - Option - Description - Default value * - `-help` - View comprehensive information about run-time options - * - `-honee_check_step_interval` - Number of time steps between checking the solution for Nans. Negative interval indicates it will not run. - `-1` * - `-honee_max_wall_time_duration` - Wall clock duration of simulation before it should be stopped. Acceptable formats are `hh`, `hh:mm`, and `hh:mm:ss`. Simulation is stopped at `start_time + duration - buffer` - `'0'` * - `-honee_max_wall_time_buffer` - Approximate time required to exit simulation cleanly (write checkpoints, etc.) - `'00:01'` * - `-honee_max_wall_time_interval` - Number of time steps between checking whether simulation should stop based on `-honee_max_wall_time_duration` - `1` ::: ### Logging Options Some of these are PETSc options here as reference, while others are custom to HONEE. :::{list-table} Logging Options :header-rows: 1 * - Option - Description * - `-ts_pre_view` - View PETSc `TS` solver configuration before it begins it's solve * - `-ts_monitor` - View log for every timestep taken by the `TS` solver * - `-snes_monitor` - View log for every iteration taken by the `SNES` solver * - `-snes_converged_reason` - View convergence reason for every iteration taken by the `SNES` solver * - `-ksp_converged_reason` - View convergence reason for every iteration taken by the `KSP` solver * - `-log_view` - View PETSc performance log * - `-ksp_post_solve_residual` - Print KSP residual summary information after each ::: ### Testing Options :::{list-table} Testing Options :header-rows: 1 * - Option - Description - Default value * - `-test_type` - Run in test mode and specify whether solution (`solver`) or turbulent statistics (`turb_spanstats`) output should be verified - `none` * - `-compare_final_state_atol` - Test absolute tolerance - `1E-11` * - `-compare_final_state_filename` - Test filename - * - `-newtonian_unit_tests` - Run unit tests of Newtonian state variable transformation functions - `false` * - `-riemann_solver_unit_tests` - Run unit tests of Riemann problem solver and it's Jacobian - `false` ::: (yaml-options)= ## YAML vs Command line options Options for HONEE and PETSc can be given by either command line options or via a YAML file. Command-line flags can be represented in a YAML file by removing the leading `-` and by allowing underscores (`_`) to be converted into a hierarchy of options (so that options with redundant prefixes can be more easily grouped together). For example, ``` -dm_plex_shape zbox -dm_plex_box_faces 10,12,4 -dm_plex_box_bd none,none,periodic ``` could be represented as ```yaml dm_plex: shape: zbox box: faces: 10,12,4 bd: none,none,periodic ``` A YAML file of options can be passed to HONEE via `-options_file `. Overriding of options are done in the order that HONEE/PETSc reads them. Given the following YAML file: ```yaml # ./options.yaml option_a: 1 ``` Running ``` navierstokes -option_a 2 -options_file options.yaml ``` will result in `option_a == 1`, but ``` navierstokes -option_a 2 -options_file options.yaml -option_a 3 ``` will result in `option_a == 3`. See [PETSc documentation](https://petsc.org/release/manual/other/#adding-options-from-a-file) for more advanced uses of the yaml file system. [^1]: `zbox` differs from `box` in two key ways: 1. When specifying periodicity via `-dm_plex_box_bd`, `zbox` uses isoperiodicity to, while `box` uses topological periodicity (see {ref}`bc-periodicity`). 2. `zbox` uses a fully parallel algorithm for building the mesh in parallel, while `box` builds in serial and then distributes the mesh.