Using HONEE¶

Note

Options in the documentation are specified by command-line flag and YAML format interchangeably. See YAML vs Command line options for details.

Setting up a problem in HONEE requires a few different things:

Mesh
Governing Equations
Initial Conditions / Restart
Boundary Conditions
Discretization
Solvers
Solution Output

Below we detail how to tell HONEE these individual pieces.

Mesh¶

HONEE uses PETSc’s DMPlex module for handling the mesh and its related data structures, and so is the interface point for this area. The mesh can either be generated by PETSc on the fly or input via a file.

To use a mesh from a file, HONEE supports reading in meshes supported by DMPlex using -dm_plex_filename. Often, this is either a CGNS file or a GMSH file.

Table 1 Mesh File Options¶
Option	Description
`-dm_plex_filename`	Path to the mesh file to be read in (usually a CGNS or GMSH file)

To have PETSc generate a mesh, DMPlex offers many runtime options to determine the shape and parameters of the mesh. Shapes maybe a box, sphere, or many other options and are specified with -dm_plex_shape. The dimensionality of the shape must be set with -dm_plex_dim. We focus on the box option here, specifically recommending -dm_plex_shape zbox[1]. The number of elements in each direction is set via -dm_plex_box_faces, which is a comma-delimited list of the number of elements in the x, y, and z directions. To set the size of the box, use -dm_plex_box_lower and -dm_plex_box_upper to set the coordinates of the lower and upper corner of the box.

Table 2 PETSc Generated Box Mesh Options¶
Option	Description
`-dm_plex_shape`	Shape for mesh to be generated (usually `zbox`)
`-dm_plex_dim`	Topological dimension for mesh to be generated (usually `zbox`)
`-dm_plex_box_faces`	Number of elements in each direction
`-dm_plex_box_lower`	Coordinates of the lower corner of the box
`-dm_plex_box_upper`	Coordinates of the upper corner of the box
`-dm_plex_box_bd`	Set periodic boundary conditions (i.e. `none,periodic,periodic`). See Periodicity.

The box meshes that PETSc generates are equispaced in every direction. HONEE offers some limited options of warping or transforming the mesh to better suit certain problems. Currently, only platemesh is supported, see Platemesh Transform for more details.

Table 3 Mesh Transform Options¶
Option	Description
`-mesh_transform`	Transformation to apply to the mesh `platemesh`

Governing Equations¶

The governing equations that HONEE will solve on the given mesh is determined by the -problem parameter. In addition to governing equation, -problem may also specify boundary conditions and initial conditions available to use for specific example problems (see Examples).

There are currently two primary governing equations supported:

Compressible Navier-Stokes (newtonian)
Advection-diffusion (advection)

Using either of these options in -problem ignores any example-specific additions.

While not always strictly governing equations, turbulence models may also be specified using -sgs_model_type (see Subgrid Stress Modeling).

Table 4 Governing Equation Runtime Options¶
Option	Description	Default value
`-problem`	Problem to solve (`advection`, `newtonian`, `density_current`, `euler_vortex`, `shocktube`, `blasius`, `channel`, `gaussian_wave`, and `taylor_green`)	`density_current`
`-sgs_model_type`	Subgrid stress model to use (`none`, `data-driven`)	`none`

Compressible Navier-Stokes, Newtonian viscosity, Ideal Gas¶

For problems that use the compressible Navier-Stokes governing equations (newtonian, density_current, blasius, channel, gaussian_wave, and taylor_green), the following options are universal for setting the parameters of them

Table 5 Newtonian Ideal Gas Problems Runtime Options¶
Option	Description	Default value	Unit
`-state_var`	State variables to solve solution with. `conservative` (\(\rho, \rho \bm{u}, \rho e\)), `primitive` (\(P, \bm{u}, T\)), or `entropy` (\(\frac{\gamma - s}{\gamma - 1} - \frac{\rho}{P} (e - c_v T),\ \frac{\rho}{P} \bm{u},\ -\frac{\rho}{P}\)) where \(s = \ln(P\rho^{-\gamma})\)	`conservative`	string
`-cv`	Heat capacity at constant volume	`717`	`J/(kg K)`
`-cp`	Heat capacity at constant pressure	`1004`	`J/(kg K)`
`-gravity`	Gravitational acceleration vector	`0,0,0`	`m/s^2`
`-lambda`	Stokes hypothesis second viscosity coefficient	`-2/3`
`-mu`	Shear dynamic viscosity coefficient	`1.8e-5`	`Pa s`
`-k`	Thermal conductivity	`0.02638`	`W/(m K)`
`-stab`	Stabilization method (`none`, `su`, or `supg`)	`none`
`-Ctau_t`	Stabilization time constant, \(C_t\). See Navier-Stokes \tau definition	`1.0`
`-Ctau_v`	Stabilization viscous constant, \(C_v\). See Navier-Stokes \tau definition	`36, 60, 128 for degree = 1, 2, 3`
`-Ctau_C`	Stabilization continuity constant, \(C_c\). See Navier-Stokes \tau definition	`1.0`
`-Ctau_M`	Stabilization momentum constant, \(C_m\). See Navier-Stokes \tau definition	`1.0`
`-Ctau_E`	Stabilization energy constant, \(C_E\). See Navier-Stokes \tau definition	`1.0`
`-div_diff_flux_projection_method`	Method used to calculate divergence of diffusive flux projection (`none`, `direct`, or `indirect`). See Divergence of Diffusive Flux Projection	`none`
`-div_diff_flux_projection_ksp*`	Control the KSP object for the projection of the divergence of diffusive flux. See Divergence of Diffusive Flux Projection	N/A

Newtonian Reference State Options¶

These options set the Newtonian reference state.

Table 6 Newtonian Reference State Runtime Options¶
Option	Description	Default value
`-reference_pressure`	Reference/initial pressure. Also used as the default for `-ic_boundary_layer_wall_pressure`.	`1.01e5`
`-reference_velocity`	Reference/initial velocity (3 values). Also used as the default for `-ic_boundary_layer_wall_velocity`.	`0,0,0`
`-reference_temperature`	Reference/initial temperature. Also used as the default for `-ic_boundary_layer_wall_temperature`.	`288.15`

Initial Conditions / Restart¶

Initial conditions are either created from an analytical definition or from a solution file.

All the example problems have their own custom analytical initial conditions. See those examples for their custom IC options. The newtonian initial condition simply sets everything equal to the reference state given by -reference*. For newtonian, IC Boundary Layer can also be used.

To set initial conditions from a file, simply pass the path to the file to -continue_filename. This must either be a binary or CGNS file. Restarting from CGNS is the preferred method as it may be done in parallel. Loading from binary is used exclusively for testing purposes.

To use the CGNS file as input, -dm_plex_filename must load the same file and -dm_plex_cgns_parallel must also be given. Restarting from a CGNS file created by HONEE will also load in the step number and solution time of the solution being loaded (see Solution Output for how to write out CGNS solution files).

Table 7 File I/O Options¶
Option	Description
`-continue_filename`	Path to file from which to continue from. Either binary file or CGNS
`-dm_plex_cgns_parallel`	Turn on the parallel CGNS reader in DMPlex. Required if `-continue_filename` is a CGNS file.

Boundary Conditions¶

Boundary conditions (BCs) are defined by a name and a list of face IDs that said BC applies to. These are collectively known as a BCDefinition. A list of BCDefinition names is specified by the user using -bc_names. The BCDefinition name will be used in the flag prefix -bc_{name}_* for every option pertaining to that BCDefinition. The list of face_ids used for a BCDefinition is specified using -bc_{name}_face_ids. For example:

bc_names: airfoil,inflow
bc_airfoil:
    face_ids: 2,5
bc_inflow:
    face_ids: 1,3,4

Essential boundary conditions (i.e. strong, Dirichlet, etc.) are defined by the component indices they constrain in the solution vector. They are specified by -bc_{name}_essential_comps.

Note

The essential boundary conditions are specified by component indices rather than by their solution component name to inhibit invalid states. For example, if the user specifies an essential constraint on velocity and is running with entropy solution variables, that constraint cannot be set, and is thus an invalid state. Additionally, this more generic interface allows for expansion to other governing equations (e.g. RANS turbulence models, chemistry transport, etc.).

Natural boundary conditions (i.e. weak, Neumann, etc.) are specified by type (a string), from a list of possibilities given by the given governing equation and example problem. These possibilities are registered by the problem/governing equation and are specified by -bc_{name}_natural_type.

Table 8 `BCDefinition` Options¶
Option	Description
`-bc_names`	List of names used to reference boundary conditions
`-bc_{name}_face_ids`	List of face label values to define BC
`-bc_{name}_essential_comps`	Component indices to apply essential BC on
`-bc_{name}_natural_type`	Type of natural BC to apply (`none`, `freestream`, `outflow`, etc.)

The face_ids are simply the label values of the Face Sets label on the DM. If the mesh is read in from file, the label values are determined from the file. For the case of PETSc-generated box meshes, those labels correspond to the following faces:

Table 9 2D Face ID Labels¶
PETSc Face Name	Cartesian direction	Face ID
faceMarkerBottom	-z	1
faceMarkerRight	+x	2
faceMarkerTop	+z	3
faceMarkerLeft	-x	4

Table 10 3D Face ID Labels¶
PETSc Face Name	Cartesian direction	Face ID
faceMarkerBottom	-z	1
faceMarkerTop	+z	2
faceMarkerFront	-y	3
faceMarkerBack	+y	4
faceMarkerRight	+x	5
faceMarkerLeft	-x	6

Discretization¶

The order of the spatial discretization can be specified by -degree. HONEE also allows for using more than the required number of quadrature points when computing the finite element operators. This is controlled by -q_extra.

Table 11 Discretization Runtime Options¶
Option	Description	Default value
`-degree`	Polynomial degree of tensor product basis (must be >= 1)	`1`
`-q_extra`	Number of extra quadrature points	`0`

Solvers¶

HONEE uses the PETSc solver stack for solving the defined problem, which is described below.

Table 12 Solver Stack¶
Solver Name	Description	Options Prefix	Options Documentation	User Documentation
TS	Timestepping	`-ts_*`	TSSetFromOptions	PETSc Documentation
SNES	Non-linear solver	`-snes_*`	SNESSetFromOptions	PETSc Documentation
KSP	Linear solver	`-ksp_*`	KSPSetFromOptions	PETSc Documentation
PC	Preconditioner (for linear solver)	`-pc_*`	PCSetFromOptions	PETSc Documentation

The prefix options listed above will apply for the primary solve that HONEE performs. Other auxillary solves and functionality will use the same options, but with an additional prefix. For example, -auxillary_ksp_*.

In addition to these options, there are a few HONEE-specific solver options as well. The method of timestepping is controlled by -implicit. Note this must match with the choice of timestepper set by -ts_type.

Table 13 HONEE-specific Solver Runtime Options¶
Option	Description	Default value
`-ceed`	CEED resource specifier	`/cpu/self/opt/blocked`
`-implicit`	Use implicit time integrator formulation	`false`
`-amat_type`	`shell` for the analytical matrix-free Jacobian

Mass Matrix and Explicit Timestepping¶

When using explicit timestepping, the inverted mass matrix must be applied to the residual. This matrix inversion is implemented as a KSP solve and is controlled by the -mass_ksp* prefix. By default, a lumped mass matrix is used, but modifying the KSP options can make it into a consistent mass matrix (specifically, setting -mass_ksp_type to something other than pconly).

Table 14 Mass Matrix Runtime Options¶
Option	Description
`-mass_ksp*`	Prefix for giving specific KSP options for the mass matrix inversion
`-mass_pc*`	Prefix for giving specific PC options for the mass matrix inversion
`-mass_ksp_view_pre_ts_solve`	View mass KSP once before `TSSolve()` is called

Nondimensionalization¶

Problems may have solution values or parameters vary by orders of magnitude between them. For example, low-Mach flow using sea-level air properties would have velocities at \(O(10)\), while pressures would be \(O(10^5)\). These large differences in scales can lead to poorly-conditioned problems. To ease these issues, the values solved can be nondimensionalized.

Caution

This feature may be broken for certain use cases. If you discover a bug related to nondimensionalization, please submit an issue to the HONEE repo so that we can address it.

Table 15 Nondimensionalization Options¶
Option	Description	Default value
`-units_meter`	1 meter in scaled length units	`1`
`-units_second`	1 second in scaled time units	`1`
`-units_kilogram`	1 kilogram in scaled mass units	`1`
`-units_Kelvin`	1 Kelvin in scaled temperature units	`1`

Solution Output¶

HONEE offers the following options for writing out solution information. We divide this into complete solution output and post-processed output.

Complete Solution Output¶

Table 16 Solution Output Options¶
Option	Description	Default value
`-ts_monitor_solution`	PETSc output format, such as `cgns:output-%d.cgns` (requires PETSc `--download-cgns`)
`-ts_monitor_solution_interval`	Number of time steps between visualization output frames.	`1`
`-viewer_cgns_batch_size`	Number of frames written per CGNS file if the CGNS file name includes a format specifier (`%d`).	`20`
`-ts_eval_times`	Sets intermediate time points to evaluate the solution at. See PETSc documentation for more details.
`-ts_eval_solutions_view`	PETSc output format for `-ts_eval_times` solutions to be written to

The following options are options using the legacy output system. It is not advised to use them.

Table 17 Legacy Solution Output Options¶
Option	Description	Default value
`-checkpoint_interval`	Number of steps between writing binary checkpoints. `0` has no output, `-1` outputs final state only	`0`
`-checkpoint_vtk`	Checkpoints include VTK (`*.vtu`) files for visualization. Consider `-ts_monitor_solution`instead.	`false`
`-viz_refine`	Use regular refinement for VTK visualization	`0`
`-output_dir`	Output directory for binary checkpoints and VTK files (if enabled).	`.`
`-output_add_stepnum2bin`	Whether to add step numbers to output binary files	`false`

Post-Processed Solution Output¶

Here are the options for monitoring specific quantities of interest of the solution as the solver progresses. Note that all of these options have an addition *_interval option which specifies the

Table 18 Post-Processed Solution Output Options¶
Option	Description	Default value
`-ts_monitor_cfl`	Viewer for the min/max CFL in the domain e.g., `ascii:cfl.csv:ascii_csv` to write a CSV file.
`-ts_monitor_wall_force`	Viewer for the force on each no-slip wall, e.g., `ascii:force.csv:ascii_csv` to write a CSV file.
`-ts_monitor_smartsim_solution`	Puts the solution into a SmartSim database. This does not take “normal” viewer inputs.
`-ts_monitor_total_kinetic_energy`	Viewer for the total kinetic energy in the domain and other terms, e.g., `ascii:total_ke.csv:ascii_csv` to write a CSV file.
`-ts_monitor_*_interval`	Number of timesteps between executing the monitor	`1`

Common Options¶

HONEE is controlled via command-line options. The following options are common among all problem types:

Table 19 Common Runtime Options¶
Option	Description	Default value
`-help`	View comprehensive information about run-time options
`-honee_check_step_interval`	Number of time steps between checking the solution for Nans. Negative interval indicates it will not run.	`-1`
`-honee_max_wall_time_duration`	Wall clock duration of simulation before it should be stopped. Acceptable formats are `hh`, `hh:mm`, and `hh:mm:ss`. Simulation is stopped at `start_time + duration - buffer`	`'0'`
`-honee_max_wall_time_buffer`	Approximate time required to exit simulation cleanly (write checkpoints, etc.)	`'00:01'`
`-honee_max_wall_time_interval`	Number of time steps between checking whether simulation should stop based on `-honee_max_wall_time_duration`	`1`

Logging Options¶

Some of these are PETSc options here as reference, while others are custom to HONEE.

Table 20 Logging Options¶
Option	Description
`-ts_pre_view`	View PETSc `TS` solver configuration before it begins it’s solve
`-ts_monitor`	View log for every timestep taken by the `TS` solver
`-snes_monitor`	View log for every iteration taken by the `SNES` solver
`-snes_converged_reason`	View convergence reason for every iteration taken by the `SNES` solver
`-ksp_converged_reason`	View convergence reason for every iteration taken by the `KSP` solver
`-log_view`	View PETSc performance log
`-ksp_post_solve_residual`	Print KSP residual summary information after each

Testing Options¶

Table 21 Testing Options¶
Option	Description	Default value
`-test_type`	Run in test mode and specify whether solution (`solver`) or turbulent statistics (`turb_spanstats`) output should be verified	`none`
`-compare_final_state_atol`	Test absolute tolerance	`1E-11`
`-compare_final_state_filename`	Test filename
`-newtonian_unit_tests`	Run unit tests of Newtonian state variable transformation functions	`false`
`-riemann_solver_unit_tests`	Run unit tests of Riemann problem solver and it’s Jacobian	`false`

YAML vs Command line options¶

Options for HONEE and PETSc can be given by either command line options or via a YAML file. Command-line flags can be represented in a YAML file by removing the leading - and by allowing underscores (_) to be converted into a hierarchy of options (so that options with redundant prefixes can be more easily grouped together). For example,

-dm_plex_shape zbox -dm_plex_box_faces 10,12,4 -dm_plex_box_bd none,none,periodic

could be represented as

dm_plex:
  shape: zbox
  box:
    faces: 10,12,4
    bd: none,none,periodic

A YAML file of options can be passed to HONEE via -options_file <yaml file>. Overriding of options are done in the order that HONEE/PETSc reads them. Given the following YAML file:

# ./options.yaml
option_a: 1

Running

navierstokes -option_a 2 -options_file options.yaml

will result in option_a == 1, but

navierstokes -option_a 2 -options_file options.yaml -option_a 3

will result in option_a == 3.

See PETSc documentation for more advanced uses of the yaml file system.