2 Running experiments using registries

PyDFLT includes a few tools that make it easier to run experiments. This notebook introduces registries that have predefined configurations of methods, models and data from literature. In 2.2 the usage of configs and utils is shown to be able to easily run more extended experiments.

2.1 Registries

To make it easy to use different models, data and DFL methods from literature, we created different registries in pydflt.registries. These 3 categories are stored here with the settings as they are relevant and appear in literature. It also makes it easy to specify a base version of a model, data or decision maker. Below are the currently stored registries. In the files themselves there are some comments regarding works that use about the same configuration, with slightly different parameters

[1]:

from pydflt.registries.decision_makers import decision_maker_registry
from pydflt.utils.load import print_registry

print_registry(decision_maker_registry)

Auto-Sklearn cannot be imported.
Name: differentiable
Class/function: <class 'pydflt.decision_makers.differentiable_decision_maker.DifferentiableDecisionMaker'>
Parameters: {'learning_rate': 0.001, 'device_str': 'cpu', 'loss_function_str': 'regret', 'predictor_str': 'MLP'}

Name: PFL linear
Class/function: <class 'pydflt.decision_makers.differentiable_decision_maker.DifferentiableDecisionMaker'>
Parameters: {'learning_rate': 0.001, 'device_str': 'cpu', 'loss_function_str': 'mse', 'predictor_str': 'MLP', 'predictor_kwargs': {'num_hidden_layers': 0}}

Name: SPO+ linear
Class/function: <class 'pydflt.decision_makers.differentiable_decision_maker.DifferentiableDecisionMaker'>
Parameters: {'learning_rate': 0.001, 'device_str': 'cpu', 'loss_function_str': 'SPOPlus', 'predictor_str': 'MLP', 'predictor_kwargs': {'num_hidden_layers': 0}}

Name: SFGE
Class/function: <class 'pydflt.decision_makers.sfge_decision_maker.SFGEDecisionMaker'>
Parameters: {'learning_rate': 0.01, 'batch_size': 32, 'device_str': 'cpu', 'loss_function_str': 'regret', 'predictor_str': 'MLP', 'noisifier_kwargs': {'sigma_setting': 'independent'}}

Name: Lancer
Class/function: <class 'pydflt.decision_makers.lancer_decision_maker.LancerDecisionMaker'>
Parameters: {'learning_rate': 0.01, 'batch_size': 32, 'device_str': 'cpu', 'predictor_str': 'MLP'}


[2]:

from pydflt.registries.models import model_registry

print_registry(model_registry)

Name: knapsack_2D_Tang2022
Class/function: <class 'pydflt.concrete_models.grbpy_knapsack.GRBPYKnapsackModel'>
Parameters: {'num_decisions': 32, 'capacity': 20, 'weights_lb': 3, 'weights_ub': 8, 'dimension': 2, 'seed': 5}

Name: shortest_path
Class/function: <class 'pydflt.concrete_models.grbpy_shortest_path.ShortestPath'>
Parameters: {'grid': (5, 5)}

Name: tsp
Class/function: <class 'pydflt.concrete_models.grbpy_tsp.TravelingSalesperson'>
Parameters: {'num_nodes': 20}

Name: WSMC_Silvestri2024
Class/function: <class 'pydflt.concrete_models.grbpy_two_stage_weighted_set_multi_cover.WeightedSetMultiCover'>
Parameters: {'num_items': 5, 'num_covers': 25, 'penalty': 5, 'cover_costs_lb': 1, 'cover_costs_ub': 100, 'Silvestri2024': True, 'seed': 5}

Name: knapsack_continuous
Class/function: <class 'pydflt.concrete_models.cvxpy_knapsack.CVXPYDiffKnapsackModel'>
Parameters: {'num_decisions': 10, 'capacity': 20, 'weights_lb': 3, 'weights_ub': 8, 'dimension': 1, 'seed': 5}

Name: knapsack_2_stage
Class/function: <class 'pydflt.concrete_models.grbpy_two_stage_knapsack.TwoStageKnapsack'>
Parameters: {'num_decision': 10, 'capacity': 20, 'penalty_add': 0.1, 'penalty_remove': 10, 'values_lb': 3, 'values_ub': 8, 'seed': 5}

Name: WSMC_Schutte2025
Class/function: <class 'pydflt.concrete_models.grbpy_two_stage_weighted_set_multi_cover.WeightedSetMultiCover'>
Parameters: {'num_items': 5, 'num_covers': 25, 'penalty': 5, 'cover_costs_lb': 5, 'cover_costs_ub': 50, 'recovery_ratio': 0.8, 'seed': 5}


[3]:

from pydflt.registries.data import data_registry

print_registry(data_registry)

Name: load_data_from_dict
Class/function: <function load_data_from_dict at 0x15a703ec0>
Parameters: {'path': None}

Name: knapsack
Class/function: <function gen_data_knapsack at 0x131d0a520>
Parameters: {'seed': 5, 'num_data': 2000, 'num_features': 5, 'num_items': 10, 'dimension': 2, 'polynomial_degree': 6, 'noise_width': 0.5}

Name: shortest_path
Class/function: <function gen_data_shortest_path at 0x131d0a340>
Parameters: {'seed': 5, 'num_data': 2000, 'num_features': 5, 'grid': (5, 5), 'polynomial_degree': 6, 'noise_width': 0.5}

Name: tsp
Class/function: <function gen_data_traveling_salesperson at 0x14a387f60>
Parameters: {'seed': 5, 'num_data': 2000, 'num_features': 5, 'num_nodes': 20, 'polynomial_degree': 6, 'noise_width': 0.5}

Name: WSMC_Silvestri2024
Class/function: <function gen_data_wsmc at 0x15a703d80>
Parameters: {'seed': 5, 'num_data': 2500, 'num_features': 5, 'num_items': 10, 'degree': 5, 'noise_width': 0.5}

With these registries one can simply load a certain model, data generation and method. For example, if we want to apply DFL ot a shortest path problem using SPO+ [1] with a linear predictor, one can use:

[4]:

from pydflt.problem import Problem
from pydflt.registries.data import get_data
from pydflt.registries.decision_makers import make_decision_maker
from pydflt.registries.models import make_model
from pydflt.runner import Runner

model, _ = make_model("shortest_path")
data, _ = get_data("shortest_path")
problem = Problem(data_dict=data, opt_model=model)
decision_maker, _ = make_decision_maker(problem, "SPO+ linear", learning_rate=0.1)  # adjust learning rate
runner = Runner(decision_maker, num_epochs=3, use_wandb=False)
result = runner.run()

Set parameter Username
Set parameter LicenseID to value 2612263
Academic license - for non-commercial use only - expires 2026-01-20
Set parameter FeasibilityTol to value 1e-06
Generating data using shortest_path
Computing optimal decisions for the entire dataset...
Optimal decisions computed and added to dataset.
Computing optimal objectives for the entire dataset...
Optimal objectives computed and added to dataset.
Shuffling indices before splitting...
Dataset split completed: Train=1400, Validation=300, Test=300
Problem mode set to: train
Problem mode set to: train
Num of cores: 1
Epoch 0/3: Starting initial validation...
Problem mode set to: validation
Epoch Results:
validation/objective_mean: 7.2163
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.3727
validation/rel_regret_mean: 1.5864
validation/arc_costs_mean: 0.8190
validation/mse_mean: 2.5531
validation/abs_regret_mean: 4.1574
Initial best validation metric (abs_regret): 4.157423496246338
Starting training...
Epoch: 1/3
Problem mode set to: train
Epoch Results:
validation/objective_mean: 7.2163
train/rel_regret_mean: 0.1963
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.3727
validation/rel_regret_mean: 1.5864
validation/arc_costs_mean: 0.8190
train/abs_regret_mean: 0.5537
train/sym_rel_regret_mean: 0.0731
validation/mse_mean: 2.5531
train/solver_calls_mean: 1019.8182
train/objective_mean: 3.6200
validation/abs_regret_mean: 4.1574
train/loss_mean: 3.9228
train/grad_norm_mean: 3.4966
Problem mode set to: validation
Epoch Results:
validation/objective_mean: 5.3228
train/rel_regret_mean: 0.1963
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.2166
validation/rel_regret_mean: 0.8717
validation/arc_costs_mean: 0.7986
train/abs_regret_mean: 0.5537
train/sym_rel_regret_mean: 0.0731
validation/mse_mean: 2.8506
train/solver_calls_mean: 1019.8182
train/objective_mean: 3.6200
validation/abs_regret_mean: 2.2639
train/loss_mean: 3.9228
train/grad_norm_mean: 3.4966
Validation evaluation (abs_regret): 0.37041616439819336
New best validation evaluation (abs_regret): 0.37041616439819336 (was 4.157423496246338)
Epoch: 2/3
Problem mode set to: train
Epoch Results:
validation/objective_mean: 5.3228
train/rel_regret_mean: 0.1720
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.2166
validation/rel_regret_mean: 0.8717
validation/arc_costs_mean: 0.7986
train/abs_regret_mean: 0.4696
train/sym_rel_regret_mean: 0.0655
validation/mse_mean: 2.8506
train/solver_calls_mean: 1869.8182
train/objective_mean: 3.5379
validation/abs_regret_mean: 2.2639
train/loss_mean: 3.4466
train/grad_norm_mean: 3.3807
Problem mode set to: validation
Epoch Results:
validation/objective_mean: 4.7273
train/rel_regret_mean: 0.1720
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.1683
validation/rel_regret_mean: 0.6445
validation/arc_costs_mean: 0.7887
train/abs_regret_mean: 0.4696
train/sym_rel_regret_mean: 0.0655
validation/mse_mean: 2.8355
train/solver_calls_mean: 1869.8182
train/objective_mean: 3.5379
validation/abs_regret_mean: 1.6685
train/loss_mean: 3.4466
train/grad_norm_mean: 3.3807
Validation evaluation (abs_regret): 0.4776097238063812
Epoch: 3/3
Problem mode set to: train
Epoch Results:
validation/objective_mean: 4.7273
train/rel_regret_mean: 0.1609
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.1683
validation/rel_regret_mean: 0.6445
validation/arc_costs_mean: 0.7887
train/abs_regret_mean: 0.4381
train/sym_rel_regret_mean: 0.0627
validation/mse_mean: 2.8355
train/solver_calls_mean: 2719.8182
train/objective_mean: 3.5069
validation/abs_regret_mean: 1.6685
train/loss_mean: 3.2599
train/grad_norm_mean: 3.3489
Problem mode set to: validation
Epoch Results:
validation/objective_mean: 4.4265
train/rel_regret_mean: 0.1609
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.1416
validation/rel_regret_mean: 0.5204
validation/arc_costs_mean: 0.7806
train/abs_regret_mean: 0.4381
train/sym_rel_regret_mean: 0.0627
validation/mse_mean: 2.8306
train/solver_calls_mean: 2719.8182
train/objective_mean: 3.5069
validation/abs_regret_mean: 1.3676
train/loss_mean: 3.2599
train/grad_norm_mean: 3.3489
Validation evaluation (abs_regret): 0.46507906913757324
Training finished. Evaluating on the test set...
Problem mode set to: test
Epoch Results:
validation/rel_regret_mean: 0.5204
test/abs_regret_mean: 0.3703
train/sym_rel_regret_mean: 0.0627
train/loss_mean: 3.2599
validation/objective_mean: 4.4265
test/arc_costs_mean: 0.7668
validation/arc_costs_mean: 0.7806
test/objective_mean: 3.5663
test/mse_mean: 3.0618
test/sym_rel_regret_mean: 0.0591
test/rel_regret_mean: 0.1471
train/rel_regret_mean: 0.1609
validation/mse_mean: 2.8306
train/solver_calls_mean: 2719.8182
train/objective_mean: 3.5069
train/grad_norm_mean: 3.3489
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.1416
train/abs_regret_mean: 0.4381
test/select_arc_mean: 0.2000
validation/abs_regret_mean: 1.3676

The make_decision_maker, make_model and get_data methods above return the used configuration as a second argument, such that it can be saved with the results.

2.2 Configs and utils

Configs are most easily defined using as a .yml file (though it is also possible to directly define dictionaries). Below we load an example config that for the model, data and decision_maker specifies a name from the registries.

[5]:

import yaml

yaml_dir = "configs/shortest_path.yml"
config = yaml.safe_load(open(yaml_dir))
for key, value in config.items():
    print(f"{key}: {value}")

model: {'name': 'shortest_path'}
data: {'name': 'shortest_path', 'num_data': 250}
runner: {'num_epochs': 5, 'use_wandb': False, 'save_best': True, 'experiments_folder': 'results/', 'seed': 5}
problem: {'train_ratio': 0.75, 'val_ratio': 0.15, 'seed': 5}
decision_maker: {'name': 'SPO+ linear', 'learning_rate': 0.005}

Using configs we can directly run experiments with the run method from pydflt.utils.experiments.

[6]:

from pydflt.utils.experiments import run

result = run(config)

Set parameter FeasibilityTol to value 1e-06
Generating data using shortest_path
Computing optimal decisions for the entire dataset...
Optimal decisions computed and added to dataset.
Computing optimal objectives for the entire dataset...
Optimal objectives computed and added to dataset.
Shuffling indices before splitting...
Dataset split completed: Train=187, Validation=37, Test=26
Problem mode set to: train
Problem mode set to: train
Num of cores: 1
Epoch 0/5: Starting initial validation...
Problem mode set to: validation
Epoch Results:
validation/objective_mean: 8.8257
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.3757
validation/rel_regret_mean: 1.4209
validation/arc_costs_mean: 0.7233
validation/mse_mean: 4.1249
validation/abs_regret_mean: 4.9402
Initial best validation metric (abs_regret): 4.940208911895752
Starting training...
Epoch: 1/5
Problem mode set to: train
Epoch Results:
validation/objective_mean: 8.8257
train/rel_regret_mean: 1.2780
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.3757
validation/rel_regret_mean: 1.4209
validation/arc_costs_mean: 0.7233
train/abs_regret_mean: 3.0089
train/sym_rel_regret_mean: 0.3273
validation/mse_mean: 4.1249
train/solver_calls_mean: 148.1667
train/objective_mean: 6.0990
validation/abs_regret_mean: 4.9402
train/loss_mean: 11.3349
train/grad_norm_mean: 5.3897
Problem mode set to: validation
Epoch Results:
validation/objective_mean: 8.3676
train/rel_regret_mean: 1.2780
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.3454
validation/rel_regret_mean: 1.2636
validation/arc_costs_mean: 0.7235
train/abs_regret_mean: 3.0089
train/sym_rel_regret_mean: 0.3273
validation/mse_mean: 4.0777
train/solver_calls_mean: 148.1667
train/objective_mean: 6.0990
validation/abs_regret_mean: 4.4821
train/loss_mean: 11.3349
train/grad_norm_mean: 5.3897
Validation evaluation (abs_regret): 4.023902416229248
New best validation evaluation (abs_regret): 4.023902416229248 (was 4.940208911895752)
Epoch: 2/5
Problem mode set to: train
Epoch Results:
validation/objective_mean: 8.3676
train/rel_regret_mean: 1.2206
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.3454
validation/rel_regret_mean: 1.2636
validation/arc_costs_mean: 0.7235
train/abs_regret_mean: 2.8474
train/sym_rel_regret_mean: 0.3153
validation/mse_mean: 4.0777
train/solver_calls_mean: 260.1667
train/objective_mean: 5.9390
validation/abs_regret_mean: 4.4821
train/loss_mean: 10.6778
train/grad_norm_mean: 5.2712
Problem mode set to: validation
Epoch Results:
validation/objective_mean: 7.9068
train/rel_regret_mean: 1.2206
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.3204
validation/rel_regret_mean: 1.1314
validation/arc_costs_mean: 0.7235
train/abs_regret_mean: 2.8474
train/sym_rel_regret_mean: 0.3153
validation/mse_mean: 4.0331
train/solver_calls_mean: 260.1667
train/objective_mean: 5.9390
validation/abs_regret_mean: 4.0212
train/loss_mean: 10.6778
train/grad_norm_mean: 5.2712
Validation evaluation (abs_regret): 3.0996029376983643
New best validation evaluation (abs_regret): 3.0996029376983643 (was 4.023902416229248)
Epoch: 3/5
Problem mode set to: train
Epoch Results:
validation/objective_mean: 7.9068
train/rel_regret_mean: 1.1591
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.3204
validation/rel_regret_mean: 1.1314
validation/arc_costs_mean: 0.7235
train/abs_regret_mean: 2.6625
train/sym_rel_regret_mean: 0.3034
validation/mse_mean: 4.0331
train/solver_calls_mean: 372.1667
train/objective_mean: 5.7577
validation/abs_regret_mean: 4.0212
train/loss_mean: 10.0913
train/grad_norm_mean: 5.2871
Problem mode set to: validation
Epoch Results:
validation/objective_mean: 7.6373
train/rel_regret_mean: 1.1591
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.3052
validation/rel_regret_mean: 1.0553
validation/arc_costs_mean: 0.7239
train/abs_regret_mean: 2.6625
train/sym_rel_regret_mean: 0.3034
validation/mse_mean: 3.9899
train/solver_calls_mean: 372.1667
train/objective_mean: 5.7577
validation/abs_regret_mean: 3.7517
train/loss_mean: 10.0913
train/grad_norm_mean: 5.2871
Validation evaluation (abs_regret): 2.943209648132324
New best validation evaluation (abs_regret): 2.943209648132324 (was 3.0996029376983643)
Epoch: 4/5
Problem mode set to: train
Epoch Results:
validation/objective_mean: 7.6373
train/rel_regret_mean: 1.0797
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.3052
validation/rel_regret_mean: 1.0553
validation/arc_costs_mean: 0.7239
train/abs_regret_mean: 2.4905
train/sym_rel_regret_mean: 0.2886
validation/mse_mean: 3.9899
train/solver_calls_mean: 484.1667
train/objective_mean: 5.5862
validation/abs_regret_mean: 3.7517
train/loss_mean: 9.5175
train/grad_norm_mean: 5.1662
Problem mode set to: validation
Epoch Results:
validation/objective_mean: 7.4342
train/rel_regret_mean: 1.0797
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.2892
validation/rel_regret_mean: 0.9740
validation/arc_costs_mean: 0.7243
train/abs_regret_mean: 2.4905
train/sym_rel_regret_mean: 0.2886
validation/mse_mean: 3.9491
train/solver_calls_mean: 484.1667
train/objective_mean: 5.5862
validation/abs_regret_mean: 3.5487
train/loss_mean: 9.5175
train/grad_norm_mean: 5.1662
Validation evaluation (abs_regret): 2.736553430557251
New best validation evaluation (abs_regret): 2.736553430557251 (was 2.943209648132324)
Epoch: 5/5
Problem mode set to: train
Epoch Results:
validation/objective_mean: 7.4342
train/rel_regret_mean: 0.9906
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.2892
validation/rel_regret_mean: 0.9740
validation/arc_costs_mean: 0.7243
train/abs_regret_mean: 2.3385
train/sym_rel_regret_mean: 0.2712
validation/mse_mean: 3.9491
train/solver_calls_mean: 596.1667
train/objective_mean: 5.4348
validation/abs_regret_mean: 3.5487
train/loss_mean: 8.9871
train/grad_norm_mean: 5.0426
Problem mode set to: validation
Epoch Results:
validation/objective_mean: 7.2499
train/rel_regret_mean: 0.9906
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.2749
validation/rel_regret_mean: 0.9101
validation/arc_costs_mean: 0.7247
train/abs_regret_mean: 2.3385
train/sym_rel_regret_mean: 0.2712
validation/mse_mean: 3.9103
train/solver_calls_mean: 596.1667
train/objective_mean: 5.4348
validation/abs_regret_mean: 3.3643
train/loss_mean: 8.9871
train/grad_norm_mean: 5.0426
Validation evaluation (abs_regret): 2.442441463470459
New best validation evaluation (abs_regret): 2.442441463470459 (was 2.736553430557251)
Training finished. Evaluating on the test set...
Problem mode set to: test
Epoch Results:
validation/rel_regret_mean: 0.9101
test/abs_regret_mean: 2.1883
train/sym_rel_regret_mean: 0.2712
train/loss_mean: 8.9871
validation/objective_mean: 7.2499
test/arc_costs_mean: 0.7306
validation/arc_costs_mean: 0.7247
test/objective_mean: 5.1237
test/mse_mean: 2.0621
test/sym_rel_regret_mean: 0.2125
test/rel_regret_mean: 0.6114
train/rel_regret_mean: 0.9906
validation/mse_mean: 3.9103
train/solver_calls_mean: 596.1667
train/objective_mean: 5.4348
train/grad_norm_mean: 5.0426
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.2749
train/abs_regret_mean: 2.3385
test/select_arc_mean: 0.2000
validation/abs_regret_mean: 3.3643

To set up a more extended set of experiments, one can use the base config and update only parts of the config by using update_config. Below we compare two DFL methods to a PFL baseline: The Smart “Predict-then-Optimize”+ loss [1] and perturbed Fenchel-Young loss [2].

[7]:

from pydflt.utils.experiments import update_config

experiment_kwargs = {
    "SPO+": {},
    "PFYL": {
        "decision_maker": {
            "loss_function_str": "perturbedFenchelYoung",
        },
    },
    "PFL": {
        "decision_maker": {
            "loss_function_str": "mse",
        }
    },
}

seeds = list(range(1))
for experiment_name, kwargs in experiment_kwargs.items():
    for seed in seeds:
        yaml_dir = "configs/shortest_path.yml"
        config = yaml.safe_load(open(yaml_dir))
        config["runner"]["experiment_name"] = experiment_name
        for key in config:
            if isinstance(config[key], dict) and "seed" in config[key]:
                config[key]["seed"] = seed
        updated_config = update_config(config, kwargs)
        run(updated_config)

Set parameter FeasibilityTol to value 1e-06
Generating data using shortest_path
Computing optimal decisions for the entire dataset...
Optimal decisions computed and added to dataset.
Computing optimal objectives for the entire dataset...
Optimal objectives computed and added to dataset.
Shuffling indices before splitting...
Dataset split completed: Train=187, Validation=37, Test=26
Problem mode set to: train
Problem mode set to: train
Num of cores: 1
Epoch 0/5: Starting initial validation...
Problem mode set to: validation
Epoch Results:
validation/objective_mean: 7.9157
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.3156
validation/rel_regret_mean: 1.0661
validation/arc_costs_mean: 0.9063
validation/mse_mean: 2.4028
validation/abs_regret_mean: 3.9976
Initial best validation metric (abs_regret): 3.997610330581665
Starting training...
Epoch: 1/5
Problem mode set to: train
Epoch Results:
validation/objective_mean: 7.9157
train/rel_regret_mean: 1.1059
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.3156
validation/rel_regret_mean: 1.0661
validation/arc_costs_mean: 0.9063
train/abs_regret_mean: 2.7159
train/sym_rel_regret_mean: 0.2918
validation/mse_mean: 2.4028
train/solver_calls_mean: 148.1667
train/objective_mean: 5.8617
validation/abs_regret_mean: 3.9976
train/loss_mean: 10.8307
train/grad_norm_mean: 6.2947
Problem mode set to: validation
Epoch Results:
validation/objective_mean: 7.8570
train/rel_regret_mean: 1.1059
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.3108
validation/rel_regret_mean: 1.0444
validation/arc_costs_mean: 0.9040
train/abs_regret_mean: 2.7159
train/sym_rel_regret_mean: 0.2918
validation/mse_mean: 2.3619
train/solver_calls_mean: 148.1667
train/objective_mean: 5.8617
validation/abs_regret_mean: 3.9389
train/loss_mean: 10.8307
train/grad_norm_mean: 6.2947
Validation evaluation (abs_regret): 3.880136489868164
New best validation evaluation (abs_regret): 3.880136489868164 (was 3.997610330581665)
Epoch: 2/5
Problem mode set to: train
Epoch Results:
validation/objective_mean: 7.8570
train/rel_regret_mean: 1.0102
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.3108
validation/rel_regret_mean: 1.0444
validation/arc_costs_mean: 0.9040
train/abs_regret_mean: 2.5431
train/sym_rel_regret_mean: 0.2753
validation/mse_mean: 2.3619
train/solver_calls_mean: 260.1667
train/objective_mean: 5.6869
validation/abs_regret_mean: 3.9389
train/loss_mean: 10.1121
train/grad_norm_mean: 6.2201
Problem mode set to: validation
Epoch Results:
validation/objective_mean: 7.6323
train/rel_regret_mean: 1.0102
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.2916
validation/rel_regret_mean: 0.9680
validation/arc_costs_mean: 0.9021
train/abs_regret_mean: 2.5431
train/sym_rel_regret_mean: 0.2753
validation/mse_mean: 2.3254
train/solver_calls_mean: 260.1667
train/objective_mean: 5.6869
validation/abs_regret_mean: 3.7142
train/loss_mean: 10.1121
train/grad_norm_mean: 6.2201
Validation evaluation (abs_regret): 3.264850378036499
New best validation evaluation (abs_regret): 3.264850378036499 (was 3.880136489868164)
Epoch: 3/5
Problem mode set to: train
Epoch Results:
validation/objective_mean: 7.6323
train/rel_regret_mean: 0.9116
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.2916
validation/rel_regret_mean: 0.9680
validation/arc_costs_mean: 0.9021
train/abs_regret_mean: 2.3044
train/sym_rel_regret_mean: 0.2568
validation/mse_mean: 2.3254
train/solver_calls_mean: 372.1667
train/objective_mean: 5.4557
validation/abs_regret_mean: 3.7142
train/loss_mean: 9.4832
train/grad_norm_mean: 6.0468
Problem mode set to: validation
Epoch Results:
validation/objective_mean: 7.1838
train/rel_regret_mean: 0.9116
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.2695
validation/rel_regret_mean: 0.8808
validation/arc_costs_mean: 0.9003
train/abs_regret_mean: 2.3044
train/sym_rel_regret_mean: 0.2568
validation/mse_mean: 2.2927
train/solver_calls_mean: 372.1667
train/objective_mean: 5.4557
validation/abs_regret_mean: 3.2657
train/loss_mean: 9.4832
train/grad_norm_mean: 6.0468
Validation evaluation (abs_regret): 1.9202847480773926
New best validation evaluation (abs_regret): 1.9202847480773926 (was 3.264850378036499)
Epoch: 4/5
Problem mode set to: train
Epoch Results:
validation/objective_mean: 7.1838
train/rel_regret_mean: 0.8349
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.2695
validation/rel_regret_mean: 0.8808
validation/arc_costs_mean: 0.9003
train/abs_regret_mean: 2.0938
train/sym_rel_regret_mean: 0.2415
validation/mse_mean: 2.2927
train/solver_calls_mean: 484.1667
train/objective_mean: 5.2419
validation/abs_regret_mean: 3.2657
train/loss_mean: 8.8925
train/grad_norm_mean: 5.9122
Problem mode set to: validation
Epoch Results:
validation/objective_mean: 6.7817
train/rel_regret_mean: 0.8349
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.2467
validation/rel_regret_mean: 0.7923
validation/arc_costs_mean: 0.8986
train/abs_regret_mean: 2.0938
train/sym_rel_regret_mean: 0.2415
validation/mse_mean: 2.2633
train/solver_calls_mean: 484.1667
train/objective_mean: 5.2419
validation/abs_regret_mean: 2.8636
train/loss_mean: 8.8925
train/grad_norm_mean: 5.9122
Validation evaluation (abs_regret): 1.2551634311676025
New best validation evaluation (abs_regret): 1.2551634311676025 (was 1.9202847480773926)
Epoch: 5/5
Problem mode set to: train
Epoch Results:
validation/objective_mean: 6.7817
train/rel_regret_mean: 0.7617
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.2467
validation/rel_regret_mean: 0.7923
validation/arc_costs_mean: 0.8986
train/abs_regret_mean: 1.9082
train/sym_rel_regret_mean: 0.2256
validation/mse_mean: 2.2633
train/solver_calls_mean: 596.1667
train/objective_mean: 5.0547
validation/abs_regret_mean: 2.8636
train/loss_mean: 8.3514
train/grad_norm_mean: 5.8002
Problem mode set to: validation
Epoch Results:
validation/objective_mean: 6.4851
train/rel_regret_mean: 0.7617
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.2287
validation/rel_regret_mean: 0.7240
validation/arc_costs_mean: 0.8967
train/abs_regret_mean: 1.9082
train/sym_rel_regret_mean: 0.2256
validation/mse_mean: 2.2373
train/solver_calls_mean: 596.1667
train/objective_mean: 5.0547
validation/abs_regret_mean: 2.5670
train/loss_mean: 8.3514
train/grad_norm_mean: 5.8002
Validation evaluation (abs_regret): 1.0841915607452393
New best validation evaluation (abs_regret): 1.0841915607452393 (was 1.2551634311676025)
Training finished. Evaluating on the test set...
Problem mode set to: test
Epoch Results:
validation/rel_regret_mean: 0.7240
test/abs_regret_mean: 0.9632
train/sym_rel_regret_mean: 0.2256
train/loss_mean: 8.3514
validation/objective_mean: 6.4851
test/arc_costs_mean: 0.8605
validation/arc_costs_mean: 0.8967
test/objective_mean: 3.9039
test/mse_mean: 1.7596
test/sym_rel_regret_mean: 0.1545
test/rel_regret_mean: 0.4607
train/rel_regret_mean: 0.7617
validation/mse_mean: 2.2373
train/solver_calls_mean: 596.1667
train/objective_mean: 5.0547
train/grad_norm_mean: 5.8002
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.2287
train/abs_regret_mean: 1.9082
test/select_arc_mean: 0.2000
validation/abs_regret_mean: 2.5670
Set parameter FeasibilityTol to value 1e-06
Generating data using shortest_path
Computing optimal decisions for the entire dataset...
Optimal decisions computed and added to dataset.
Computing optimal objectives for the entire dataset...
Optimal objectives computed and added to dataset.
Shuffling indices before splitting...
Dataset split completed: Train=187, Validation=37, Test=26
Problem mode set to: train
Problem mode set to: train
Num of cores: 1
Epoch 0/5: Starting initial validation...
Problem mode set to: validation
Epoch Results:
validation/objective_mean: 7.9080
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.3020
validation/rel_regret_mean: 1.0252
validation/arc_costs_mean: 0.8379
validation/mse_mean: 2.3372
validation/abs_regret_mean: 3.9899
Initial best validation metric (abs_regret): 3.9898881912231445
Starting training...
Epoch: 1/5
Problem mode set to: train
Epoch Results:
validation/objective_mean: 7.9080
train/rel_regret_mean: 1.1443
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.3020
validation/rel_regret_mean: 1.0252
validation/arc_costs_mean: 0.8379
train/abs_regret_mean: 2.8377
train/sym_rel_regret_mean: 0.2979
validation/mse_mean: 2.3372
train/solver_calls_mean: 148.1667
train/objective_mean: 5.9835
validation/abs_regret_mean: 3.9899
train/loss_mean: 6.8692
train/grad_norm_mean: 2.2794
Problem mode set to: validation
Epoch Results:
validation/objective_mean: 7.7732
train/rel_regret_mean: 1.1443
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.2924
validation/rel_regret_mean: 0.9810
validation/arc_costs_mean: 0.8396
train/abs_regret_mean: 2.8377
train/sym_rel_regret_mean: 0.2979
validation/mse_mean: 2.2990
train/solver_calls_mean: 148.1667
train/objective_mean: 5.9835
validation/abs_regret_mean: 3.8552
train/loss_mean: 6.8692
train/grad_norm_mean: 2.2794
Validation evaluation (abs_regret): 3.7204201221466064
New best validation evaluation (abs_regret): 3.7204201221466064 (was 3.9898881912231445)
Epoch: 2/5
Problem mode set to: train
Epoch Results:
validation/objective_mean: 7.7732
train/rel_regret_mean: 1.0915
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.2924
validation/rel_regret_mean: 0.9810
validation/arc_costs_mean: 0.8396
train/abs_regret_mean: 2.6506
train/sym_rel_regret_mean: 0.2897
validation/mse_mean: 2.2990
train/solver_calls_mean: 260.1667
train/objective_mean: 5.7944
validation/abs_regret_mean: 3.8552
train/loss_mean: 6.6645
train/grad_norm_mean: 2.2079
Problem mode set to: validation
Epoch Results:
validation/objective_mean: 7.3671
train/rel_regret_mean: 1.0915
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.2736
validation/rel_regret_mean: 0.8980
validation/arc_costs_mean: 0.8412
train/abs_regret_mean: 2.6506
train/sym_rel_regret_mean: 0.2897
validation/mse_mean: 2.2640
train/solver_calls_mean: 260.1667
train/objective_mean: 5.7944
validation/abs_regret_mean: 3.4490
train/loss_mean: 6.6645
train/grad_norm_mean: 2.2079
Validation evaluation (abs_regret): 2.6368212699890137
New best validation evaluation (abs_regret): 2.6368212699890137 (was 3.7204201221466064)
Epoch: 3/5
Problem mode set to: train
Epoch Results:
validation/objective_mean: 7.3671
train/rel_regret_mean: 1.0401
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.2736
validation/rel_regret_mean: 0.8980
validation/arc_costs_mean: 0.8412
train/abs_regret_mean: 2.4822
train/sym_rel_regret_mean: 0.2798
validation/mse_mean: 2.2640
train/solver_calls_mean: 372.1667
train/objective_mean: 5.6335
validation/abs_regret_mean: 3.4490
train/loss_mean: 6.5385
train/grad_norm_mean: 2.1539
Problem mode set to: validation
Epoch Results:
validation/objective_mean: 7.0815
train/rel_regret_mean: 1.0401
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.2560
validation/rel_regret_mean: 0.8291
validation/arc_costs_mean: 0.8427
train/abs_regret_mean: 2.4822
train/sym_rel_regret_mean: 0.2798
validation/mse_mean: 2.2318
train/solver_calls_mean: 372.1667
train/objective_mean: 5.6335
validation/abs_regret_mean: 3.1635
train/loss_mean: 6.5385
train/grad_norm_mean: 2.1539
Validation evaluation (abs_regret): 2.3066790103912354
New best validation evaluation (abs_regret): 2.3066790103912354 (was 2.6368212699890137)
Epoch: 4/5
Problem mode set to: train
Epoch Results:
validation/objective_mean: 7.0815
train/rel_regret_mean: 0.9910
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.2560
validation/rel_regret_mean: 0.8291
validation/arc_costs_mean: 0.8427
train/abs_regret_mean: 2.3052
train/sym_rel_regret_mean: 0.2692
validation/mse_mean: 2.2318
train/solver_calls_mean: 484.1667
train/objective_mean: 5.4533
validation/abs_regret_mean: 3.1635
train/loss_mean: 6.3538
train/grad_norm_mean: 2.1029
Problem mode set to: validation
Epoch Results:
validation/objective_mean: 6.8533
train/rel_regret_mean: 0.9910
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.2392
validation/rel_regret_mean: 0.7577
validation/arc_costs_mean: 0.8442
train/abs_regret_mean: 2.3052
train/sym_rel_regret_mean: 0.2692
validation/mse_mean: 2.2020
train/solver_calls_mean: 484.1667
train/objective_mean: 5.4533
validation/abs_regret_mean: 2.9352
train/loss_mean: 6.3538
train/grad_norm_mean: 2.1029
Validation evaluation (abs_regret): 2.02203631401062
New best validation evaluation (abs_regret): 2.02203631401062 (was 2.3066790103912354)
Epoch: 5/5
Problem mode set to: train
Epoch Results:
validation/objective_mean: 6.8533
train/rel_regret_mean: 0.9298
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.2392
validation/rel_regret_mean: 0.7577
validation/arc_costs_mean: 0.8442
train/abs_regret_mean: 2.1462
train/sym_rel_regret_mean: 0.2555
validation/mse_mean: 2.2020
train/solver_calls_mean: 596.1667
train/objective_mean: 5.2927
validation/abs_regret_mean: 2.9352
train/loss_mean: 6.1865
train/grad_norm_mean: 2.0330
Problem mode set to: validation
Epoch Results:
validation/objective_mean: 6.6156
train/rel_regret_mean: 0.9298
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.2234
validation/rel_regret_mean: 0.6922
validation/arc_costs_mean: 0.8456
train/abs_regret_mean: 2.1462
train/sym_rel_regret_mean: 0.2555
validation/mse_mean: 2.1750
train/solver_calls_mean: 596.1667
train/objective_mean: 5.2927
validation/abs_regret_mean: 2.6975
train/loss_mean: 6.1865
train/grad_norm_mean: 2.0330
Validation evaluation (abs_regret): 1.5090945959091187
New best validation evaluation (abs_regret): 1.5090945959091187 (was 2.02203631401062)
Training finished. Evaluating on the test set...
Problem mode set to: test
Epoch Results:
validation/rel_regret_mean: 0.6922
test/abs_regret_mean: 1.0047
train/sym_rel_regret_mean: 0.2555
train/loss_mean: 6.1865
validation/objective_mean: 6.6156
test/arc_costs_mean: 0.9208
validation/arc_costs_mean: 0.8456
test/objective_mean: 3.9453
test/mse_mean: 1.8451
test/sym_rel_regret_mean: 0.2012
test/rel_regret_mean: 0.6255
train/rel_regret_mean: 0.9298
validation/mse_mean: 2.1750
train/solver_calls_mean: 596.1667
train/objective_mean: 5.2927
train/grad_norm_mean: 2.0330
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.2234
train/abs_regret_mean: 2.1462
test/select_arc_mean: 0.2000
validation/abs_regret_mean: 2.6975
Set parameter FeasibilityTol to value 1e-06
Generating data using shortest_path
Computing optimal decisions for the entire dataset...
Optimal decisions computed and added to dataset.
Computing optimal objectives for the entire dataset...
Optimal objectives computed and added to dataset.
Shuffling indices before splitting...
Dataset split completed: Train=187, Validation=37, Test=26
Problem mode set to: train
Problem mode set to: train
Epoch 0/5: Starting initial validation...
Problem mode set to: validation
Epoch Results:
validation/objective_mean: 7.9080
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.3020
validation/rel_regret_mean: 1.0252
validation/arc_costs_mean: 0.8379
validation/mse_mean: 2.3372
validation/abs_regret_mean: 3.9899
Initial best validation metric (abs_regret): 3.9898881912231445
Starting training...
Epoch: 1/5
Problem mode set to: train
Epoch Results:
validation/objective_mean: 7.9080
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.3020
validation/rel_regret_mean: 1.0252
validation/arc_costs_mean: 0.8379
validation/mse_mean: 2.3372
train/solver_calls_mean: 37.0000
validation/abs_regret_mean: 3.9899
train/loss_mean: 2.8444
train/grad_norm_mean: 0.9549
Problem mode set to: validation
Epoch Results:
validation/objective_mean: 7.8244
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.2955
validation/rel_regret_mean: 1.0024
validation/arc_costs_mean: 0.8454
validation/mse_mean: 2.2571
train/solver_calls_mean: 37.0000
validation/abs_regret_mean: 3.9063
train/loss_mean: 2.8444
train/grad_norm_mean: 0.9549
Validation evaluation (abs_regret): 3.8226499557495117
New best validation evaluation (abs_regret): 3.8226499557495117 (was 3.9898881912231445)
Epoch: 2/5
Problem mode set to: train
Epoch Results:
validation/objective_mean: 7.8244
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.2955
validation/rel_regret_mean: 1.0024
validation/arc_costs_mean: 0.8454
validation/mse_mean: 2.2571
train/solver_calls_mean: 55.5000
validation/abs_regret_mean: 3.9063
train/loss_mean: 2.7419
train/grad_norm_mean: 0.9360
Problem mode set to: validation
Epoch Results:
validation/objective_mean: 7.7703
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.2914
validation/rel_regret_mean: 0.9837
validation/arc_costs_mean: 0.8513
validation/mse_mean: 2.1812
train/solver_calls_mean: 55.5000
validation/abs_regret_mean: 3.8522
train/loss_mean: 2.7419
train/grad_norm_mean: 0.9360
Validation evaluation (abs_regret): 3.7440361976623535
New best validation evaluation (abs_regret): 3.7440361976623535 (was 3.8226499557495117)
Epoch: 3/5
Problem mode set to: train
Epoch Results:
validation/objective_mean: 7.7703
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.2914
validation/rel_regret_mean: 0.9837
validation/arc_costs_mean: 0.8513
validation/mse_mean: 2.1812
train/solver_calls_mean: 74.0000
validation/abs_regret_mean: 3.8522
train/loss_mean: 2.6594
train/grad_norm_mean: 0.9133
Problem mode set to: validation
Epoch Results:
validation/objective_mean: 7.6018
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.2837
validation/rel_regret_mean: 0.9439
validation/arc_costs_mean: 0.8579
validation/mse_mean: 2.1092
train/solver_calls_mean: 74.0000
validation/abs_regret_mean: 3.6837
train/loss_mean: 2.6594
train/grad_norm_mean: 0.9133
Validation evaluation (abs_regret): 3.1782002449035645
New best validation evaluation (abs_regret): 3.1782002449035645 (was 3.7440361976623535)
Epoch: 4/5
Problem mode set to: train
Epoch Results:
validation/objective_mean: 7.6018
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.2837
validation/rel_regret_mean: 0.9439
validation/arc_costs_mean: 0.8579
validation/mse_mean: 2.1092
train/solver_calls_mean: 92.5000
validation/abs_regret_mean: 3.6837
train/loss_mean: 2.5935
train/grad_norm_mean: 0.8948
Problem mode set to: validation
Epoch Results:
validation/objective_mean: 7.3846
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.2745
validation/rel_regret_mean: 0.9040
validation/arc_costs_mean: 0.8638
validation/mse_mean: 2.0417
train/solver_calls_mean: 92.5000
validation/abs_regret_mean: 3.4665
train/loss_mean: 2.5935
train/grad_norm_mean: 0.8948
Validation evaluation (abs_regret): 2.59779691696167
New best validation evaluation (abs_regret): 2.59779691696167 (was 3.1782002449035645)
Epoch: 5/5
Problem mode set to: train
Epoch Results:
validation/objective_mean: 7.3846
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.2745
validation/rel_regret_mean: 0.9040
validation/arc_costs_mean: 0.8638
validation/mse_mean: 2.0417
train/solver_calls_mean: 111.0000
validation/abs_regret_mean: 3.4665
train/loss_mean: 2.5165
train/grad_norm_mean: 0.8880
Problem mode set to: validation
Epoch Results:
validation/objective_mean: 7.1984
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.2673
validation/rel_regret_mean: 0.8707
validation/arc_costs_mean: 0.8700
validation/mse_mean: 1.9777
train/solver_calls_mean: 111.0000
validation/abs_regret_mean: 3.2803
train/loss_mean: 2.5165
train/grad_norm_mean: 0.8880
Validation evaluation (abs_regret): 2.3491334915161133
New best validation evaluation (abs_regret): 2.3491334915161133 (was 2.59779691696167)
Training finished. Evaluating on the test set...
Problem mode set to: test
Epoch Results:
validation/objective_mean: 7.1984
test/arc_costs_mean: 0.8639
validation/select_arc_mean: 0.2000
validation/sym_rel_regret_mean: 0.2673
validation/rel_regret_mean: 0.8707
validation/arc_costs_mean: 0.8700
test/objective_mean: 4.9862
test/mse_mean: 1.3423
test/abs_regret_mean: 2.0456
test/sym_rel_regret_mean: 0.3036
validation/mse_mean: 1.9777
train/solver_calls_mean: 111.0000
test/select_arc_mean: 0.2000
test/rel_regret_mean: 1.1638
validation/abs_regret_mean: 3.2803
train/loss_mean: 2.5165
train/grad_norm_mean: 0.8880

To be able to analyze the results of multiple runs like these properly, we suggest using Weight & Biases. We explain how to use this tool in the next notebook.

References

[1] Adam N. Elmachtoub and Paul Grigas. Smart “predict, then optimize”’. Management Science, 68:9–26, 2022. doi:10.1287/mnsc.2020.3922.

[2] Quentin Berthet, Mathieu Blondel, Olivier Teboul, Marco Cuturi, Jean-Philippe Vert, and Francis Bach. Learning with differentiable perturbed optimizers. Advances in Neural Information Processing Systems, 33:9508–9519, 2020. doi:10.48550/arXiv.2002.08676.