3 Analyzing training using Weights and Biases¶

Weights and biases (wandb) is an effective tool for training machine learning models (free for academic use). The Runner has an argument called use_wandb, which when set to True will use weights and biases. In this notebook we show how wandb can be used effectively. First we show how it can help, after we show how to set it up for yourself.

3.1 Experimental set-up¶

We run the same experiment as in notebook 2, now with more data and seeds. We compare DFL methods SPO+ [1] and PFYL [2] to baseline PFL on the shortest path problem. Below is the code required to run it, but this run is already stored in wandb.

[1]:

rerun_experiment = False
if rerun_experiment:
    import yaml
    from pydflt.utils.experiments import run, update_config

    experiment_kwargs = {
        "SPO+": {},
        "PFYL": {
            "decision_maker": {
                "loss_function_str": "perturbedFenchelYoung",
            },
        },
        "PFL": {
            "decision_maker": {
                "loss_function_str": "mse",
            }
        },
    }

    seeds = list(range(5))
    for experiment_name, kwargs in experiment_kwargs.items():
        for seed in seeds:
            yaml_dir = "configs/shortest_path_wandb.yml"
            config = yaml.safe_load(open(yaml_dir))
            config["runner"]["experiment_name"] = experiment_name
            for key in config:
                if isinstance(config[key], dict) and "seed" in config[key]:
                    config[key]["seed"] = seed
            updated_config = update_config(config, kwargs)
            run(updated_config)

We display the wandb dashboard below, alternatively one can go to the link

[2]:

import wandb
from IPython.display import IFrame

# Display it inline
IFrame("https://wandb.ai/noahschutte-tudelft/shortest_path/runs/", width=1000, height=800)

[2]:

3.2 Analyzing using wandb¶

Above (or through this link) a wandb dashboard is shown.

On the left all runs are listed, grouped by name. Runs can be viewed individually as well, where the runs config is also listed so it is clear what all the parameters are.

The main group of panels is broken down into test, train, validation and system. The non-system panels include multiple metrics. In general, methods as defined in the PyDFLT registries include metrics that do not increase runtime. For example, training with PFL but recording regret on the train or validation set would increase training time, so this is non-default. Outside of regular quality metrics, some direct prediction and decision metrics are also recorded. In the above panels for example one can see validation/arc_costs_mean and validation/select_arc_mean, which show the average predicted costs and the average number of selected arcs at each epoch.

Additional panels can be added and customized on the top right.

3.3 Setting up wandb¶

To set up wandb, you can make an account here. After this, you can go here to copy your api key. There are a few ways to make sure you can use wandb:

1. Set an environment variable named WANDB_KEY with your key, or create a .env file with WANDB_KEY = 'your-key-here'.

2. Using your api key you can run the following cell (or use the terminal command: wandb login). If you have logged in using a certain api key once, you do not have to log in again, as it automatically stores the key locally.

[3]:

login_wandb = False
if login_wandb:
    wandb.login(key="YOUR_API_KEY")