TensorBoard is a tool for providing the measurements and visualizations needed during the machine learning workflow. It enables tracking of experiment metrics such as loss and accuracy, visualizing the model graph, projecting embeddings to a lower dimensional space, and much more.

This article explains how to quickly get started with TensorBoard. You will need to load and activate a NAS conda environment that includes TensorFlow.

Before You Begin: In order to use the methods described in this article, you must set up SSH Passthrough.

Creating Data for TensorBoard

Within your TensorFlow application there is a piece of code that does the model fitting, as shown below. Here, we create a TensorBoard callback:

log_dir = "logs/fit/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
tensorboard_callback= tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1)

Add the argument callbacks=[tensorboard_callback]), which accesses the function:

model.fit(train_ds,
    epochs=EPOCHS,
    steps_per_epoch=int(image_count/BATCH),
    verbose=verbose,
    callbacks=[tensorboard_callback])

Once your application is finished, you will see a directory called logs/.

Using TensorBoard

Complete these steps to use TensorBoard.

1. Start a PBS interactive section:

   pfe20 % qsub -I -q devel -lselect=1:ncpus=16:model=bro,walltime=1:00:00

   Note: You don't need a GPU node to look at the TensorBoard data.

2. On the compute node, load the miniconda3 module from the /swbuild/analytix directory:

   r601i1n1 % module use -a /swbuild/analytix/tools/modulefiles
   r601i1n1 % module load miniconda3/v4

   where r601i1n1 is the name of a compute node (the one you use will have a different name). Be sure to note the name of the compute node, as you will need it to access the TensorBoard on your local machine in step 5.

3. Activate a conda environment to use the proper libraries. For example, where env_name is the name of the environment:
   • For bash:
     r601i1n1 % source activate env_name
   • For csh (on one line):
     r601i1n1 % source /swbuild/analytix/tools/miniconda3_220407/bin/activate.csh env_name

4. Start a TensorBoard on the compute node:

   (env_name)r601i1n1 % tensorboard --logdir=logs
   Serving TensorBoard on localhost; to expose to the network, 
   use a proxy or pass --bind_all
   TensorBoard 2.2.1 at http://localhost:6006/ (Press CTRL+C to quit)
   

5. On your local workstation, open a terminal and type the following (on one command line) to connect via SSH to the compute node:

   your_local_system% ssh -o "StrictHostKeyChecking ask" -L 
   8080:localhost:6006 -o ProxyJump=sfe,pfe20 r601i1n1

   Note: You must specify the PFE you used to launch the compute node, and the name of the compute node. In this example, we specify pfe20 and r601i1n1.

6. You can now access JupyterLab from a browser on your workstation by connecting to:

   http://localhost:8080/ or http://127.0.0.1:8080

   Note: If your browser objects to the self-signed certificate used by this instance of TensorBoard, allow it to continue.

Once you have completed these steps, you will get this interactive screen to create customized measurements and views of your data:

References

For more information, see the following documentation:

• Get Started with TensorBoard
• TensorBoard Tutorial