Skip to content

Conducting a Grid Refinement Study

Creating a set of grids for a convergence study is standard procedure to bound a grid's truncation error. Possible options within the context of the LAVA project file system are:

  • Incorporate the refinement factors within the grid growth scripts. This requires the grid parameters in parm/MakeParm.tcl to be dependent on grid level (already inherent), and for a set of starting surface grids to exist within each component's geom sub-directory. Likely, conditional grid growth arguments such as splay have to be implemented iteratively in the components' tcl scripts in order for the now denser grids to have proper overlap, non-folded elements, etc.
  • Build a new volume grid from scratch using EPOGS for each refinement level desired with appropriate refinement of grid metrics. This may result in needing to manually correct individual grids for each level (see the EPOGS tutorial for a brief discussion on this).
  • Use LAVA's refine utility to take existing volumes and refine them by a factor. If refinement issues arise, the growth parameters of the level one grids (i.e. the starting grids) must be adjusted to allow for a well-posed starting point. This generally doesn't require much effort.

We chose to adopt the last option for this tutorial. As show in the figure below, LAVA's refine executable takes each block and directionally parameterizes each block curve with a spline, redistributes it with a refined point set in parameter space, and then maps that redistribution back to physical space.

method

There is currently no projection back to CAD, so one caveat is if significant c1-discontinuity or repeated alternations between concavity and convexity exists within the parameterized curve curvature then the refined curves may no longer well-represent the true model geometry (this is no worse than refining grids using existing CGT functionality such as SrapRedist). For this reason the surface grids were built a-priori to delineate feature lines that should be respected.

Building the Refined Grid Set

To run the refinement procedure first navigate to the desired run sub-folder (pick the one you created for the scripting tutorial). If you do not have the grid.x and grid.bc files required as a starting point for this tutorial, either do the project grid scripting tutorial or download them from that tutorial. Once those files exist, run the run_refineGrid.sh script, shown below, making sure the GROUP_ID, LAVA_EXE_DIR, and LAVA_EXE_SUFFIX, and OMP_NUM_THREADS variables are set in envVars.sh.

run_refineGrid.sh
# Define a number of refinements (1.414214 and 2.0 here), then
# perform the refinements in appropriate sub directory.
# Use sed to alter the yaml files to have the right refinement factor.
# This script assumes that we will be performing a RANS convergence study,
# and so also copies in and appropriately alters the files needed to run
# a RANS solution on NAS.
# ---------------------------------------------------------------------------- #
factOnes=(1      2)
factDecs=(414214 0)

baseDir=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )

source $baseDir/envVars.sh

mkdir -p soln_RANS

cp $baseDir/auxFiles/LAVA_solver_RANS.yaml       ./soln_RANS/
cp $baseDir/auxFiles/LAVA_solver_RANS_athena.pbs ./soln_RANS/

sed -i "/#PBS -W/c\\#PBS -W group_list=$GROUP_ID" \
       LAVA_solver_RANS_athena.pbs

for ((factI=0; factI<${#factOnes[@]}; factI++)); do

    # Make and enter refined grid folder.
    # ------------------------------------------------------------------------ #
    refFolder="refinedGrid_factor${factOnes[$factI]}p${factDecs[$factI]}"

    mkdir -p $refFolder
    cd       $refFolder
    # ------------------------------------------------------------------------ #

    # Refine the grid.
    # ------------------------------------------------------------------------ #
    cp ../grid.bc .
    cp $baseDir/auxFiles/LAVA_refine.yaml .

    sed -i "/  factor : 1.0/c\  factor : ${factOnes[$factI]}.${factDecs[$factI]}" \
           LAVA_refine.yaml

    $LAVA_EXE_DIR/refine.$LAVA_EXE_SUFFIX LAVA_refine.yaml

    mv surface.x grid.sur
    rm LAVA_refine.yaml
    # ------------------------------------------------------------------------ #

    # Get and update the files needed to run RANS on NAS.
    # ------------------------------------------------------------------------ #
    jobName="LAVA_solver_OGSCRM_AOA2p5_L${factOnes[$factI]}p${factDecs[$factI]}"
    factor="${factOnes[$factI]}.${factDecs[$factI]}"
    nNode=$(echo "(($factor * $factor * $factor) + 0.5) / 1" | bc)

    mkdir -p soln_RANS

    cp $baseDir/auxFiles/LAVA_solver_RANS.yaml       ./soln_RANS/
    cp $baseDir/auxFiles/LAVA_solver_RANS_athena.pbs ./soln_RANS/

    sed -i "/#PBS -l select/c\\#PBS -l select=$nNode:ncpus=256:mpiprocs=4:model=tur_ath" \
           soln_RANS/LAVA_solver_RANS_athena.pbs
    sed -i "/#PBS -N/c\\#PBS -N $jobName" \
           soln_RANS/LAVA_solver_RANS_athena.pbs
    sed -i "/#PBS -W/c\\#PBS -W group_list=$GROUP_ID" \
           soln_RANS/LAVA_solver_RANS_athena.pbs
    # ------------------------------------------------------------------------ #

    # Back to base grid folder.
    # ------------------------------------------------------------------------ #
    cd ..
    # ------------------------------------------------------------------------ #
done
# ---------------------------------------------------------------------------- #

For each grid level defined with a ones and decimals place, a level-named sub-folder is created and the refine executable is called within it to create a local grid. The grid within these sub-folders is named the same as the base grid level so that other run scripts can be called on them (e.g. connectivity - see below). The BC file is copied in because the BCs shouldn't have changed. In this case we are creating grids that are a factor of 1.414214 and 2.0 denser than the original.

The input file to the refinement utility is shown below. Note how the grid to be refined is one level up from the sub-directory we have created. When it is imported by run_refineGrid.sh, the factor entry is modified appropriately. The surface grid is the portion of the refined grid defined by the bc file. Note that if partial BCs existed (the ranges were not [1,-1]) then the bc file would have to be altered prior to refine being called to ensure accurate surface extraction.

refine:
  factor : 1.0
  coarseGridFile : ../grid.x
  refineGridFile : grid.x
  nSmooth : 0
  writerecordmarkersingrids: false

surface :
  extract : True
  surfaceGridFile : surface.x
  bcFile : grid.bc
  surfaceBCNames : [fuse,wing,tail]

Slices and surfaces of the refined grid set can be seen below. The entire grid system has been refined, but note that there are options to refine individual grids to different levels, or directionally.

L1p0 L1p4 L2p0

Connecting and Assessing the Refined Grids

Run the run_connectivity.sh script within each refined grid folder (refinedGrid_1p414214, refinedGrid_2p0), or run on NAS as described in the connectivity tutorial: no changes need to be made to the connectivity files themselves because the grid files are named as expected. The level 1.414214 grid takes 12 seconds with 56 threads to run the connectivity script and has ~68 million nodes, while the level 2.0 grid takes 30 seconds and has ~189 million.

There are generally three things to check for issues on a refined grid:

  1. Negative Jacobians: Open the grid.x.ihc file in overgrid and click DIAGNOSE and then the show box next to Jacobian in the bottom right. If any of the table entries have non-zero count then view them in the grid. Look at your level zero grids in the same locations (likely they will be almost folded in the now problem areas) and alter the grid scripts to smooth or splay them more to prevent refinement issues, then rerun refinement.
  2. Orphans: It is unlikely that orphans will appear due to overlap issues that didn't already exist in the starting grid because the overlapped region isn't reduced. However, the shape of the geometry surface may have slightly changed if the spline across a large portion of a body which, for instance, undulates. More laxity in projection may be needed. Clicking DIAGNOSE within overgrid and then All under orphan analysis in the mid right will count and highlight orphans.
  3. Commensurate B2Bs: The connectivity process assesses and writes a new B2B file. One indication of refinement going wrong is if the B2B file created doesn't match that of the starting grid. Diff the two to double-check.

The level 1.414214 grid does not show any issues for these checks. The Level 2.0 however, shows the orphans on zone 5: the inboard wing trailing edge. Zooming in and cycling through constant L of grid 5 shows that the orphans surround a hole that has been created in grid 5 from [J 17->18, K 47, L 8->9]:

Block 5 Orphans Zoomed in Block 5 Orphans

These kind of localized 'hash' patterns of orphans generally mean an edge or two has been cut, but we will re-run connectivity with a debug node to be sure. Alter the LAVA_connectivity.yaml file to have the following additional debugNode entry. The run_connectivity.sh script is designed so that alterations to the yaml file will not be overwritten.

overset :
  debugNode : [5, 17, 47, 8]

When connectivity is re-run now, we see output with a 'Debug Node' prefix. Assessing the output from debug nodes can be complex and the manual exists in part to make this output less opaque. We focus in on the following lines, which state that the portion of block 1 (the fuse) which intersects one of the edges of this node is too far away to project to given the relative size of the elements, which means connectivity doesn't think they are the 'same' surface.

[[Debug Node (Blk,j,k,l)                5   17   47    8]] (thread    27): Checking edge [   17   47    8] to [   17   47    9] for surface intersection ...
[[Debug Node (Blk,j,k,l)                5   17   47    8]] (thread    27):     Start node closest surfNode Blk/i/j/k:     5   17   47    1
[[Debug Node (Blk,j,k,l)                5   17   47    8]] (thread    27):     End node closest surfNode Blk/i/j/k:     5   17   47    1
[[Debug Node (Blk,j,k,l)                5   17   47    8]] (thread    27):     Considering intersection     1 of     1 with block     1...
[[Debug Node (Blk,j,k,l)                5   17   47    8]] (thread    27):         Found surface relation between intersected face  114099 and edge start node.
[[Debug Node (Blk,j,k,l)                5   17   47    8]] (thread    27):         Edge start node dist from wall node to cut node:   0.00001302
[[Debug Node (Blk,j,k,l)                5   17   47    8]] (thread    27):         Edge start node max local deviation offset lim:   0.00000378
[[Debug Node (Blk,j,k,l)                5   17   47    8]] (thread    27):         Start dist from wall outside of surface deviation limit: cut edge.
[[Debug Node (Blk,j,k,l)                5   17   47    8]] (thread    27):         At least one edge is linked to the block surface: checking unrelated interpolated intersection overlap ...
[[Debug Node (Blk,j,k,l)                5   17   47    8]] (thread    27):         Edge node dist from interpolated wall node to cut node:   4.05455871
[[Debug Node (Blk,j,k,l)                5   17   47    8]] (thread    27):         Edge average node max local deviation offset lim:   0.00000378
[[Debug Node (Blk,j,k,l)                5   17   47    8]] (thread    27):         Interpolated intersection is also outside of surface deviation limit: cut edge.
[[Debug Node (Blk,j,k,l)                5   17   47    8]] (thread    27):     Intersected     1 times with surface projs, at least one of which can't be treated as the same surface: blanking both edge nodes.

Taking an x location near the orphans (4.053641), we take a slice of the un-cut grid.x file and zoom into trailing edge intersection with the fuse to find that there is indeed a slightly larger offset of the fuse on the refined grid (right) than was on the original grid (left):

Grid 1.0 x Slice Grid 2.0 x Slice

Opening both the original and refined grid.sur files with the Flat Shaded Display Option checked shows that splining the long fuse undulations has caused some minor differences in the fuse which are mostly irrelevant except for this problem location:

Surface Overlays

The fix for this issue is simple: slightly increase the distance surfaces on block 5 are allowed to accept projection onto other surfaces. This is done through the edgeFactor option (defaulted to 1) assigned to block 5 through a zoneSpecific option by adding the following lines to the LAVA_connectivity.yaml file and re-running run_connectivity.sh:

overset :
  zoneSpecific:
   -
     # Allow block 5 to project just a bit further (onto block 1).
     zones :      [5]
     edgeFactor : 1.2

One could also argue that the fuselage component should be split into smaller grids along its length near the fairing, but the changes in shape are so minor that this was deemed unnecessary. One non-hard-coded strategy for this would be to use the GetLimitedBBox CGT utility to find the limits of the fairing within the fuse.tcl script, and then split at the appropriate j indices.

Run RANS Solutions and Assess Grid Convergence

The purpose of building a family of refined grids is to truncate the grid discretization error - i.e. find the coarsest grid which still captures the flow physics. In preparation for running a RANS simulation on each of these grids, the run_refineGrid.sh made a solution sub directory and copied in and modified a solver input and job submission (LAVA_solver_RANS.yaml and LAVA_solver_RANS_athena.pbs) file to each case, then modifies them as needed to account for increased resource need and job name. To run these cases on NAS, do the following for each:

  • Run the run_connectivity.sh script within the grid's directory to connect the grid.
  • If you aren't building on NAS, push the following files to the grid's folder on NAS:
    • grid.x.ihc
    • grid.bc
    • grid.x.b2b
    • INTOUT
    • soln_RANS
  • ssh into athfe01.
  • cd into soln_RANS
  • submit the pbs job with qsub LAVA_solver_RANS_athena.pbs.

Note that you could use the run_RANS.sh script to run these solutions locally, as is done in the RANS tutorial, but especially for higher grid level refinements this can become untenable.

As shown below, all simulations' flow/turbulence residual converge 7+/5+ orders of magnitude, and their standard deviations of lift and drag coefficient over the last 100 iterations are well below a count. The three levels required ~77, ~294, ~3890 CPU-hrs of computational time, respectively. For a discussion on the RANS solution settings choices and analysis, see the RANS tutorial.

Residuals Grid 1.0 clcd Grid 1.4 clcd Grid 2.0 clcd

These plots were generated using LAVA's plotHist tool by running the following commands in the run sub-directory:

plotHist.py -F                      data.iter iter cl cd -showStd -xRange  230  330
plotHist.py -F refinedGrid_1p414214/data.iter iter cl cd -showStd -xRange  300  400
plotHist.py -F      refinedGrid_2p0/data.iter iter cl cd -showStd -xRange 1040 1140
plotHist.py -F data.iter refinedGrid_1p414214/data.iter refinedGrid_2p0/data.iter iter flowres turres

If we plot a trend of cd vs a linearized representation of 3D grid size increase such that an infinitely fine grid is located on the x axis, we see a very flat trend with an asymptotic cd estimate of 0.028193. Since all grids are well within a defined linear trend and we can quantify the bounds of integrated loads, it is safe to say that the initial grid is fine enough to resolve the physics of the RANS case at AOA of 2.5 degrees.

cd Trend