4 Hyperparamater tuning using Optuna¶

Optuna is a hyperparameter tuning package that is integrated in PyDFLT. In this notebook we describe how to use it.

4.1 Prepare basic config¶

We first define a base config with the basic parameter configuration. Specifically include parameters here that you want to remain fixed.

[1]:

import yaml

yaml_dir = "configs/knapsack.yml"
base_config = yaml.safe_load(open(yaml_dir))  # base_config is a dictionary, so does not have to be loaded from a .yml

for key, value in base_config.items():
    print(f"{key}: {value}")

model: {'name': 'knapsack_continuous', 'seed': 5}
data: {'name': 'knapsack', 'num_data': 500, 'seed': 5}
runner: {'num_epochs': 5, 'use_wandb': False, 'experiment_name': 'tuning', 'experiments_folder': 'results/', 'seed': 5}
problem: {'train_ratio': 0.75, 'val_ratio': 0.15, 'seed': 5}
decision_maker: {'name': 'differentiable', 'learning_rate': 0.05, 'batch_size': 32, 'seed': 5}

4.2 Define search spaces¶

Now we define the parameters we want to search over. In this example we will search the best learning_rate and batch_size.

[2]:

import os

from pydflt.utils.optuna import SearchSpaceConfig

# To conduct a hyperparameter tuning experiment, we first need to define the search spaces
path_to_project = os.path.dirname(os.path.abspath("")) + "/"
search_space = SearchSpaceConfig(path_to_project + "examples/hparams_search_spaces/test_search_config.yaml")

for key, value in search_space.config.items():  # search_space.config is a nested dictionary
    print(f"{key}: {value}")

Auto-Sklearn cannot be imported.
decision_maker: {'learning_rate': {'type': 'float', 'low': 1e-05, 'high': 0.001, 'log': True}, 'batch_size': {'type': 'int', 'low': 1, 'high': 512, 'log': True}}

4.3 Create pruner¶

A pruner can be used to prune trials that are not promising. This can greatly reduce the time to find good parameters.

[3]:

import optuna

pruner = optuna.pruners.MedianPruner(
    n_startup_trials=10,  # Number of trials to run before pruning
    n_warmup_steps=15,  # Number of epochs to wait before pruning
    interval_steps=1,  # Interval between pruning checks
    n_min_trials=1,  # Minimum trials required for pruning
)

4.4 Specify study¶

Specify study name and the folder where the results are stored. A database.db file will be created in the folder OUTPUT_DIR. While different problems/methods can be all inside the same database, it can be convenient to separate it by problem or method.

[4]:

import os

from pydflt.utils.optuna import create_study

STUDY_NAME = "test_study"  # Note that optuna will continue with an existing study if the study already exists
OUTPUT_DIR = "hparam_optimization_results/"  # Folder where the study database is stored (at OUTPUT_DIR/STUDY_NAME)
os.makedirs(f"{OUTPUT_DIR}/{STUDY_NAME}", exist_ok=True)  # Ensure that the folder exists
study = create_study(
    STUDY_NAME,
    storage_url=f"sqlite:///{OUTPUT_DIR}/{STUDY_NAME}/database.db",
    prunner=pruner,
)

Storage url: sqlite:///hparam_optimization_results//test_study/database.db

[I 2025-10-21 11:40:15,217] A new study created in RDB with name: test_study

4.5 Set up dashboard¶

We can set up a dashboard that visualizes results when the study is running, opened in the background (using package optuna-dashboard). Alternatively open the dashboard through the terminal using: optuna-dashboard sqlite:///examples/hparam_optimization_results//test_study//database.db

[5]:

import os
import socket
import subprocess

# Suppress dashboard warning noise about experimental PedAnova importance evaluator
warning_filters = [
    "ignore::optuna.exceptions.ExperimentalWarning",
    "ignore:PedAnovaImportanceEvaluator computes the importances of params to achieve low `target` values.:UserWarning",
]
env = {
    **os.environ,
    "PYTHONWARNINGS": ",".join(filter(None, [os.environ.get("PYTHONWARNINGS", ""), *warning_filters])),
}


# Find a free port to host the dashboard
def find_free_port():
    with socket.socket() as sock:
        sock.bind(("", 0))
        return sock.getsockname()[1]


port = str(find_free_port())
print(f"Found free port: {port}")

# Set-up dashboard
subprocess.Popen(
    [
        "optuna-dashboard",
        "sqlite:///hparam_optimization_results//test_study//database.db",
        "--port",
        port,
    ],
    stdout=subprocess.DEVNULL,
    stderr=subprocess.DEVNULL,
    env=env,
)

print(f"See dashboard here: http://localhost:{port}")

Found free port: 51336
See dashboard here: http://localhost:51336

Below we show how the dashboard looks. It updates when the study is running.

[6]:

from IPython.display import IFrame

IFrame(src=f"http://localhost:{port}", width="100%", height=600)

[6]:

4.6 Define trial¶

To run hyperparameters tuning with Optuna, we need to define what a ‘trial’ looks like. We run each configuration for multiple seeds using run_trial. Finally, we define objective_function, denoting what we use as evaluation metric. In this case this is the validation objective as returned by running the experiment through run_trial.

[7]:

import numpy as np
from pydflt.utils.experiments import run, update_config


# from pydflt.utils.optuna import run_trial
def run_trial(trial, search_space: SearchSpaceConfig, base_config, seeds: list):
    assert len(seeds) > 0, "Provide at least one seed!"

    trial_config = search_space.get_trial_config(trial)
    per_seed_results: list[float] = []

    for seed_idx, seed in enumerate(seeds, start=1):
        config = update_config(base_config=base_config, updates_config=trial_config)

        config["seed"] = seed
        for key in config:
            if isinstance(config[key], dict) and "seed" in config[key]:
                config[key]["seed"] = seed

        result = run(config, optuna_trial=None)
        per_seed_results.append(float(result))

        if trial is not None:
            intermediate = float(np.mean(per_seed_results))
            trial.report(intermediate, seed_idx)
            if trial.should_prune():
                raise optuna.TrialPruned()

    return float(np.mean(per_seed_results))


def objective_fn(trial):
    return run_trial(trial, search_space, base_config, seeds=list(range(3)))

4.7 Run tuning¶

Now we run the hyperparameter tuning.

[8]:

study.optimize(objective_fn, n_trials=3, timeout=None, catch=(Exception,), show_progress_bar=True)

Generating data using knapsack
Computing optimal decisions for the entire dataset...
Optimal decisions computed and added to dataset.
Computing optimal objectives for the entire dataset...
Optimal objectives computed and added to dataset.
Shuffling indices before splitting...
Dataset split completed: Train=375, Validation=75, Test=50
Problem mode set to: train
Problem mode set to: train
Epoch 0/5: Starting initial validation...
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.4107
validation/mse_mean: 41.6570
validation/sym_rel_regret_mean: 0.2772
validation/item_value_mean: 4.3376
validation/select_item_mean: 0.3410
validation/objective_mean: 15.7564
validation/abs_regret_mean: 8.9304
Initial best validation metric (abs_regret): 8.930413246154785
Starting training...
Epoch: 1/5
Problem mode set to: train
Epoch Results:
validation/rel_regret_mean: 0.4107
validation/mse_mean: 41.6570
train/loss_mean: 8.1462
validation/sym_rel_regret_mean: 0.2772
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.1462
train/rel_regret_mean: 0.3777
validation/item_value_mean: 4.3376
validation/select_item_mean: 0.3410
train/objective_mean: 17.2348
train/sym_rel_regret_mean: 0.2514
validation/objective_mean: 15.7564
validation/abs_regret_mean: 8.9304
train/solver_calls_mean: 375.0000
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.4108
validation/mse_mean: 41.6462
train/loss_mean: 8.1462
validation/sym_rel_regret_mean: 0.2772
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.1462
train/rel_regret_mean: 0.3777
validation/item_value_mean: 4.3365
validation/select_item_mean: 0.3410
train/objective_mean: 17.2348
train/sym_rel_regret_mean: 0.2514
validation/objective_mean: 15.7562
validation/abs_regret_mean: 8.9307
train/solver_calls_mean: 375.0000
Validation evaluation (abs_regret): 8.93091869354248
Epoch: 2/5
Problem mode set to: train
Epoch Results:
validation/rel_regret_mean: 0.4108
validation/mse_mean: 41.6462
train/loss_mean: 8.1025
validation/sym_rel_regret_mean: 0.2772
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.1025
train/rel_regret_mean: 0.3768
validation/item_value_mean: 4.3365
validation/select_item_mean: 0.3410
train/objective_mean: 17.3211
train/sym_rel_regret_mean: 0.2509
validation/objective_mean: 15.7562
validation/abs_regret_mean: 8.9307
train/solver_calls_mean: 600.0000
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.4109
validation/mse_mean: 41.6355
train/loss_mean: 8.1025
validation/sym_rel_regret_mean: 0.2773
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.1025
train/rel_regret_mean: 0.3768
validation/item_value_mean: 4.3354
validation/select_item_mean: 0.3410
train/objective_mean: 17.3211
train/sym_rel_regret_mean: 0.2509
validation/objective_mean: 15.7535
validation/abs_regret_mean: 8.9333
train/solver_calls_mean: 600.0000
Validation evaluation (abs_regret): 8.938689231872559
Epoch: 3/5
Problem mode set to: train
Epoch Results:
validation/rel_regret_mean: 0.4109
validation/mse_mean: 41.6355
train/loss_mean: 8.1587
validation/sym_rel_regret_mean: 0.2773
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.1587
train/rel_regret_mean: 0.3763
validation/item_value_mean: 4.3354
validation/select_item_mean: 0.3410
train/objective_mean: 17.5386
train/sym_rel_regret_mean: 0.2505
validation/objective_mean: 15.7535
validation/abs_regret_mean: 8.9333
train/solver_calls_mean: 825.0000
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.4110
validation/mse_mean: 41.6270
train/loss_mean: 8.1587
validation/sym_rel_regret_mean: 0.2773
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.1587
train/rel_regret_mean: 0.3763
validation/item_value_mean: 4.3348
validation/select_item_mean: 0.3410
train/objective_mean: 17.5386
train/sym_rel_regret_mean: 0.2505
validation/objective_mean: 15.7523
validation/abs_regret_mean: 8.9346
train/solver_calls_mean: 825.0000
Validation evaluation (abs_regret): 8.938183784484863
Epoch: 4/5
Problem mode set to: train
Epoch Results:
validation/rel_regret_mean: 0.4110
validation/mse_mean: 41.6270
train/loss_mean: 8.1555
validation/sym_rel_regret_mean: 0.2773
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.1555
train/rel_regret_mean: 0.3754
validation/item_value_mean: 4.3348
validation/select_item_mean: 0.3410
train/objective_mean: 17.6058
train/sym_rel_regret_mean: 0.2497
validation/objective_mean: 15.7523
validation/abs_regret_mean: 8.9346
train/solver_calls_mean: 1050.0000
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.4110
validation/mse_mean: 41.6210
train/loss_mean: 8.1555
validation/sym_rel_regret_mean: 0.2773
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.1555
train/rel_regret_mean: 0.3754
validation/item_value_mean: 4.3343
validation/select_item_mean: 0.3410
train/objective_mean: 17.6058
train/sym_rel_regret_mean: 0.2497
validation/objective_mean: 15.7516
validation/abs_regret_mean: 8.9353
train/solver_calls_mean: 1050.0000
Validation evaluation (abs_regret): 8.938183784484863
Epoch: 5/5
Problem mode set to: train
Epoch Results:
validation/rel_regret_mean: 0.4110
validation/mse_mean: 41.6210
train/loss_mean: 8.1425
validation/sym_rel_regret_mean: 0.2773
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.1425
train/rel_regret_mean: 0.3746
validation/item_value_mean: 4.3343
validation/select_item_mean: 0.3410
train/objective_mean: 17.6185
train/sym_rel_regret_mean: 0.2491
validation/objective_mean: 15.7516
validation/abs_regret_mean: 8.9353
train/solver_calls_mean: 1275.0000
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.4111
validation/mse_mean: 41.6161
train/loss_mean: 8.1425
validation/sym_rel_regret_mean: 0.2774
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.1425
train/rel_regret_mean: 0.3746
validation/item_value_mean: 4.3341
validation/select_item_mean: 0.3410
train/objective_mean: 17.6185
train/sym_rel_regret_mean: 0.2491
validation/objective_mean: 15.7500
validation/abs_regret_mean: 8.9369
train/solver_calls_mean: 1275.0000
Validation evaluation (abs_regret): 8.944727897644043
Training finished. Evaluating on the test set...
Problem mode set to: test
Epoch Results:
validation/rel_regret_mean: 0.4111
validation/sym_rel_regret_mean: 0.2774
test/item_value_mean: 4.3211
validation/item_value_mean: 4.3341
train/sym_rel_regret_mean: 0.2491
test/mse_mean: 27.6233
validation/select_item_mean: 0.3410
validation/mse_mean: 41.6161
train/loss_mean: 8.1425
train/rel_regret_mean: 0.3746
test/rel_regret_mean: 0.4275
validation/objective_mean: 15.7500
test/abs_regret_mean: 9.4049
validation/abs_regret_mean: 8.9369
test/select_item_mean: 0.3407
train/abs_regret_mean: 8.1425
train/grad_norm_mean: 0.0000
train/objective_mean: 17.6185
test/sym_rel_regret_mean: 0.2862
test/objective_mean: 14.6595
train/solver_calls_mean: 1275.0000
Generating data using knapsack
Computing optimal decisions for the entire dataset...
Optimal decisions computed and added to dataset.
Computing optimal objectives for the entire dataset...
Optimal objectives computed and added to dataset.
Shuffling indices before splitting...
Dataset split completed: Train=375, Validation=75, Test=50
Problem mode set to: train
Problem mode set to: train
Epoch 0/5: Starting initial validation...
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.2468
validation/mse_mean: 36.8984
validation/sym_rel_regret_mean: 0.1521
validation/item_value_mean: 4.3190
validation/select_item_mean: 0.4999
validation/objective_mean: 21.8615
validation/abs_regret_mean: 7.4671
Initial best validation metric (abs_regret): 7.467118263244629
Starting training...
Epoch: 1/5
Problem mode set to: train
Epoch Results:
validation/rel_regret_mean: 0.2468
validation/mse_mean: 36.8984
train/loss_mean: 8.9851
validation/sym_rel_regret_mean: 0.1521
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.9851
train/rel_regret_mean: 0.2638
validation/item_value_mean: 4.3190
validation/select_item_mean: 0.4999
train/objective_mean: 22.5571
train/sym_rel_regret_mean: 0.1617
validation/objective_mean: 21.8615
validation/abs_regret_mean: 7.4671
train/solver_calls_mean: 375.0000
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.2455
validation/mse_mean: 36.8992
train/loss_mean: 8.9851
validation/sym_rel_regret_mean: 0.1511
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.9851
train/rel_regret_mean: 0.2638
validation/item_value_mean: 4.3194
validation/select_item_mean: 0.4999
train/objective_mean: 22.5571
train/sym_rel_regret_mean: 0.1617
validation/objective_mean: 21.8799
validation/abs_regret_mean: 7.4487
train/solver_calls_mean: 375.0000
Validation evaluation (abs_regret): 7.430324554443359
New best validation evaluation (abs_regret): 7.430324554443359 (was 7.467118263244629)
Epoch: 2/5
Problem mode set to: train
Epoch Results:
validation/rel_regret_mean: 0.2455
validation/mse_mean: 36.8992
train/loss_mean: 9.0195
validation/sym_rel_regret_mean: 0.1511
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 9.0195
train/rel_regret_mean: 0.2638
validation/item_value_mean: 4.3194
validation/select_item_mean: 0.4999
train/objective_mean: 22.5863
train/sym_rel_regret_mean: 0.1619
validation/objective_mean: 21.8799
validation/abs_regret_mean: 7.4487
train/solver_calls_mean: 600.0000
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.2451
validation/mse_mean: 36.9009
train/loss_mean: 9.0195
validation/sym_rel_regret_mean: 0.1507
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 9.0195
train/rel_regret_mean: 0.2638
validation/item_value_mean: 4.3196
validation/select_item_mean: 0.4999
train/objective_mean: 22.5863
train/sym_rel_regret_mean: 0.1619
validation/objective_mean: 21.8860
validation/abs_regret_mean: 7.4426
train/solver_calls_mean: 600.0000
Validation evaluation (abs_regret): 7.430324554443359
Epoch: 3/5
Problem mode set to: train
Epoch Results:
validation/rel_regret_mean: 0.2451
validation/mse_mean: 36.9009
train/loss_mean: 8.9642
validation/sym_rel_regret_mean: 0.1507
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.9642
train/rel_regret_mean: 0.2649
validation/item_value_mean: 4.3196
validation/select_item_mean: 0.4999
train/objective_mean: 22.3901
train/sym_rel_regret_mean: 0.1628
validation/objective_mean: 21.8860
validation/abs_regret_mean: 7.4426
train/solver_calls_mean: 825.0000
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.2448
validation/mse_mean: 36.9027
train/loss_mean: 8.9642
validation/sym_rel_regret_mean: 0.1505
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.9642
train/rel_regret_mean: 0.2649
validation/item_value_mean: 4.3197
validation/select_item_mean: 0.4999
train/objective_mean: 22.3901
train/sym_rel_regret_mean: 0.1628
validation/objective_mean: 21.8895
validation/abs_regret_mean: 7.4391
train/solver_calls_mean: 825.0000
Validation evaluation (abs_regret): 7.428503513336182
New best validation evaluation (abs_regret): 7.428503513336182 (was 7.430324554443359)
Epoch: 4/5
Problem mode set to: train
Epoch Results:
validation/rel_regret_mean: 0.2448
validation/mse_mean: 36.9027
train/loss_mean: 8.9285
validation/sym_rel_regret_mean: 0.1505
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.9285
train/rel_regret_mean: 0.2641
validation/item_value_mean: 4.3197
validation/select_item_mean: 0.4999
train/objective_mean: 22.4179
train/sym_rel_regret_mean: 0.1623
validation/objective_mean: 21.8895
validation/abs_regret_mean: 7.4391
train/solver_calls_mean: 1050.0000
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.2446
validation/mse_mean: 36.9044
train/loss_mean: 8.9285
validation/sym_rel_regret_mean: 0.1504
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.9285
train/rel_regret_mean: 0.2641
validation/item_value_mean: 4.3198
validation/select_item_mean: 0.4998
train/objective_mean: 22.4179
train/sym_rel_regret_mean: 0.1623
validation/objective_mean: 21.8908
validation/abs_regret_mean: 7.4378
train/solver_calls_mean: 1050.0000
Validation evaluation (abs_regret): 7.432821273803711
Epoch: 5/5
Problem mode set to: train
Epoch Results:
validation/rel_regret_mean: 0.2446
validation/mse_mean: 36.9044
train/loss_mean: 8.8991
validation/sym_rel_regret_mean: 0.1504
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.8991
train/rel_regret_mean: 0.2641
validation/item_value_mean: 4.3198
validation/select_item_mean: 0.4998
train/objective_mean: 22.4133
train/sym_rel_regret_mean: 0.1623
validation/objective_mean: 21.8908
validation/abs_regret_mean: 7.4378
train/solver_calls_mean: 1275.0000
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.2445
validation/mse_mean: 36.9053
train/loss_mean: 8.8991
validation/sym_rel_regret_mean: 0.1503
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.8991
train/rel_regret_mean: 0.2641
validation/item_value_mean: 4.3199
validation/select_item_mean: 0.4998
train/objective_mean: 22.4133
train/sym_rel_regret_mean: 0.1623
validation/objective_mean: 21.8916
validation/abs_regret_mean: 7.4370
train/solver_calls_mean: 1275.0000
Validation evaluation (abs_regret): 7.432821273803711
Training finished. Evaluating on the test set...
Problem mode set to: test
Epoch Results:
validation/rel_regret_mean: 0.2445
validation/sym_rel_regret_mean: 0.1503
test/item_value_mean: 4.2988
validation/item_value_mean: 4.3199
train/sym_rel_regret_mean: 0.1623
test/mse_mean: 21.6812
validation/select_item_mean: 0.4998
validation/mse_mean: 36.9053
train/loss_mean: 8.8991
train/rel_regret_mean: 0.2641
test/rel_regret_mean: 0.2393
validation/objective_mean: 21.8916
test/abs_regret_mean: 7.5399
validation/abs_regret_mean: 7.4370
test/select_item_mean: 0.5009
train/abs_regret_mean: 8.8991
train/grad_norm_mean: 0.0000
train/objective_mean: 22.4133
test/sym_rel_regret_mean: 0.1470
test/objective_mean: 22.2512
train/solver_calls_mean: 1275.0000
Generating data using knapsack
Computing optimal decisions for the entire dataset...
Optimal decisions computed and added to dataset.
Computing optimal objectives for the entire dataset...
Optimal objectives computed and added to dataset.
Shuffling indices before splitting...
Dataset split completed: Train=375, Validation=75, Test=50
Problem mode set to: train
Problem mode set to: train
Epoch 0/5: Starting initial validation...
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.2326
validation/mse_mean: 23.9004
validation/sym_rel_regret_mean: 0.1394
validation/item_value_mean: 3.8620
validation/select_item_mean: 0.4642
validation/objective_mean: 21.4090
validation/abs_regret_mean: 6.3541
Initial best validation metric (abs_regret): 6.3540873527526855
Starting training...
Epoch: 1/5
Problem mode set to: train
Epoch Results:
validation/rel_regret_mean: 0.2326
validation/mse_mean: 23.9004
train/loss_mean: 9.6785
validation/sym_rel_regret_mean: 0.1394
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 9.6785
train/rel_regret_mean: 0.2783
validation/item_value_mean: 3.8620
validation/select_item_mean: 0.4642
train/objective_mean: 20.7949
train/sym_rel_regret_mean: 0.1739
validation/objective_mean: 21.4090
validation/abs_regret_mean: 6.3541
train/solver_calls_mean: 375.0000
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.2331
validation/mse_mean: 23.8987
train/loss_mean: 9.6785
validation/sym_rel_regret_mean: 0.1396
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 9.6785
train/rel_regret_mean: 0.2783
validation/item_value_mean: 3.8625
validation/select_item_mean: 0.4641
train/objective_mean: 20.7949
train/sym_rel_regret_mean: 0.1739
validation/objective_mean: 21.3956
validation/abs_regret_mean: 6.3676
train/solver_calls_mean: 375.0000
Validation evaluation (abs_regret): 6.381021499633789
Epoch: 2/5
Problem mode set to: train
Epoch Results:
validation/rel_regret_mean: 0.2331
validation/mse_mean: 23.8987
train/loss_mean: 9.5462
validation/sym_rel_regret_mean: 0.1396
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 9.5462
train/rel_regret_mean: 0.2787
validation/item_value_mean: 3.8625
validation/select_item_mean: 0.4641
train/objective_mean: 20.6851
train/sym_rel_regret_mean: 0.1739
validation/objective_mean: 21.3956
validation/abs_regret_mean: 6.3676
train/solver_calls_mean: 600.0000
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.2332
validation/mse_mean: 23.8994
train/loss_mean: 9.5462
validation/sym_rel_regret_mean: 0.1397
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 9.5462
train/rel_regret_mean: 0.2787
validation/item_value_mean: 3.8628
validation/select_item_mean: 0.4641
train/objective_mean: 20.6851
train/sym_rel_regret_mean: 0.1739
validation/objective_mean: 21.3911
validation/abs_regret_mean: 6.3720
train/solver_calls_mean: 600.0000
Validation evaluation (abs_regret): 6.381021499633789
Epoch: 3/5
Problem mode set to: train
Epoch Results:
validation/rel_regret_mean: 0.2332
validation/mse_mean: 23.8994
train/loss_mean: 9.5115
validation/sym_rel_regret_mean: 0.1397
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 9.5115
train/rel_regret_mean: 0.2785
validation/item_value_mean: 3.8628
validation/select_item_mean: 0.4641
train/objective_mean: 20.6462
train/sym_rel_regret_mean: 0.1739
validation/objective_mean: 21.3911
validation/abs_regret_mean: 6.3720
train/solver_calls_mean: 825.0000
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.2333
validation/mse_mean: 23.9017
train/loss_mean: 9.5115
validation/sym_rel_regret_mean: 0.1398
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 9.5115
train/rel_regret_mean: 0.2785
validation/item_value_mean: 3.8629
validation/select_item_mean: 0.4641
train/objective_mean: 20.6462
train/sym_rel_regret_mean: 0.1739
validation/objective_mean: 21.3888
validation/abs_regret_mean: 6.3743
train/solver_calls_mean: 825.0000
Validation evaluation (abs_regret): 6.381021499633789
Epoch: 4/5
Problem mode set to: train
Epoch Results:
validation/rel_regret_mean: 0.2333
validation/mse_mean: 23.9017
train/loss_mean: 9.4595
validation/sym_rel_regret_mean: 0.1398
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 9.4595
train/rel_regret_mean: 0.2781
validation/item_value_mean: 3.8629
validation/select_item_mean: 0.4641
train/objective_mean: 20.5825
train/sym_rel_regret_mean: 0.1735
validation/objective_mean: 21.3888
validation/abs_regret_mean: 6.3743
train/solver_calls_mean: 1050.0000
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.2333
validation/mse_mean: 23.9021
train/loss_mean: 9.4595
validation/sym_rel_regret_mean: 0.1398
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 9.4595
train/rel_regret_mean: 0.2781
validation/item_value_mean: 3.8631
validation/select_item_mean: 0.4641
train/objective_mean: 20.5825
train/sym_rel_regret_mean: 0.1735
validation/objective_mean: 21.3875
validation/abs_regret_mean: 6.3756
train/solver_calls_mean: 1050.0000
Validation evaluation (abs_regret): 6.381021499633789
Epoch: 5/5
Problem mode set to: train
Epoch Results:
validation/rel_regret_mean: 0.2333
validation/mse_mean: 23.9021
train/loss_mean: 9.5189
validation/sym_rel_regret_mean: 0.1398
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 9.5189
train/rel_regret_mean: 0.2790
validation/item_value_mean: 3.8631
validation/select_item_mean: 0.4641
train/objective_mean: 20.5816
train/sym_rel_regret_mean: 0.1741
validation/objective_mean: 21.3875
validation/abs_regret_mean: 6.3756
train/solver_calls_mean: 1275.0000
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.2334
validation/mse_mean: 23.9016
train/loss_mean: 9.5189
validation/sym_rel_regret_mean: 0.1398
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 9.5189
train/rel_regret_mean: 0.2790
validation/item_value_mean: 3.8634
validation/select_item_mean: 0.4641
train/objective_mean: 20.5816
train/sym_rel_regret_mean: 0.1741
validation/objective_mean: 21.3861
validation/abs_regret_mean: 6.3770
train/solver_calls_mean: 1275.0000
Validation evaluation (abs_regret): 6.384034633636475
Training finished. Evaluating on the test set...
Problem mode set to: test
Epoch Results:
validation/rel_regret_mean: 0.2334
validation/sym_rel_regret_mean: 0.1398
test/item_value_mean: 3.8784
validation/item_value_mean: 3.8634
train/sym_rel_regret_mean: 0.1741
test/mse_mean: 44.9416
validation/select_item_mean: 0.4641
validation/mse_mean: 23.9016
train/loss_mean: 9.5189
train/rel_regret_mean: 0.2790
test/rel_regret_mean: 0.2576
validation/objective_mean: 21.3861
test/abs_regret_mean: 8.6495
validation/abs_regret_mean: 6.3770
test/select_item_mean: 0.4629
train/abs_regret_mean: 9.5189
train/grad_norm_mean: 0.0000
train/objective_mean: 20.5816
test/sym_rel_regret_mean: 0.1573
test/objective_mean: 20.6775
train/solver_calls_mean: 1275.0000
[I 2025-10-21 11:40:26,187] Trial 0 finished with value: 7.571001370747884 and parameters: {'learning_rate': 2.3175836771328023e-05, 'batch_size': 225}. Best is trial 0 with value: 7.571001370747884.
Generating data using knapsack
Computing optimal decisions for the entire dataset...
Optimal decisions computed and added to dataset.
Computing optimal objectives for the entire dataset...
Optimal objectives computed and added to dataset.
Shuffling indices before splitting...
Dataset split completed: Train=375, Validation=75, Test=50
Problem mode set to: train
Problem mode set to: train
Epoch 0/5: Starting initial validation...
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.4107
validation/mse_mean: 41.6570
validation/sym_rel_regret_mean: 0.2772
validation/item_value_mean: 4.3376
validation/select_item_mean: 0.3410
validation/objective_mean: 15.7564
validation/abs_regret_mean: 8.9304
Initial best validation metric (abs_regret): 8.930413246154785
Starting training...
Epoch: 1/5
Problem mode set to: train
Epoch Results:
validation/rel_regret_mean: 0.4107
validation/mse_mean: 41.6570
train/loss_mean: 8.1276
validation/sym_rel_regret_mean: 0.2772
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.1276
train/rel_regret_mean: 0.3731
validation/item_value_mean: 4.3376
validation/select_item_mean: 0.3410
train/objective_mean: 17.7197
train/sym_rel_regret_mean: 0.2477
validation/objective_mean: 15.7564
validation/abs_regret_mean: 8.9304
train/solver_calls_mean: 450.0000
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.4119
validation/mse_mean: 41.7070
train/loss_mean: 8.1276
validation/sym_rel_regret_mean: 0.2774
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.1276
train/rel_regret_mean: 0.3731
validation/item_value_mean: 4.3263
validation/select_item_mean: 0.3411
train/objective_mean: 17.7197
train/sym_rel_regret_mean: 0.2477
validation/objective_mean: 15.7483
validation/abs_regret_mean: 8.9386
train/solver_calls_mean: 450.0000
Validation evaluation (abs_regret): 8.946749687194824
Epoch: 2/5
Problem mode set to: train
Epoch Results:
validation/rel_regret_mean: 0.4119
validation/mse_mean: 41.7070
train/loss_mean: 8.2948
validation/sym_rel_regret_mean: 0.2774
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.2948
train/rel_regret_mean: 0.3737
validation/item_value_mean: 4.3263
validation/select_item_mean: 0.3411
train/objective_mean: 17.5525
train/sym_rel_regret_mean: 0.2480
validation/objective_mean: 15.7483
validation/abs_regret_mean: 8.9386
train/solver_calls_mean: 675.0000
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.4160
validation/mse_mean: 41.7356
train/loss_mean: 8.2948
validation/sym_rel_regret_mean: 0.2804
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.2948
train/rel_regret_mean: 0.3737
validation/item_value_mean: 4.3242
validation/select_item_mean: 0.3410
train/objective_mean: 17.5525
train/sym_rel_regret_mean: 0.2480
validation/objective_mean: 15.6654
validation/abs_regret_mean: 9.0214
train/solver_calls_mean: 675.0000
Validation evaluation (abs_regret): 9.187128067016602
Epoch: 3/5
Problem mode set to: train
Epoch Results:
validation/rel_regret_mean: 0.4160
validation/mse_mean: 41.7356
train/loss_mean: 8.4206
validation/sym_rel_regret_mean: 0.2804
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.4206
train/rel_regret_mean: 0.3750
validation/item_value_mean: 4.3242
validation/select_item_mean: 0.3410
train/objective_mean: 17.4267
train/sym_rel_regret_mean: 0.2487
validation/objective_mean: 15.6654
validation/abs_regret_mean: 9.0214
train/solver_calls_mean: 900.0000
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.4189
validation/mse_mean: 41.7582
train/loss_mean: 8.4206
validation/sym_rel_regret_mean: 0.2828
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.4206
train/rel_regret_mean: 0.3750
validation/item_value_mean: 4.3254
validation/select_item_mean: 0.3410
train/objective_mean: 17.4267
train/sym_rel_regret_mean: 0.2487
validation/objective_mean: 15.6051
validation/abs_regret_mean: 9.0817
train/solver_calls_mean: 900.0000
Validation evaluation (abs_regret): 9.262625694274902
Epoch: 4/5
Problem mode set to: train
Epoch Results:
validation/rel_regret_mean: 0.4189
validation/mse_mean: 41.7582
train/loss_mean: 8.5036
validation/sym_rel_regret_mean: 0.2828
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.5036
train/rel_regret_mean: 0.3763
validation/item_value_mean: 4.3254
validation/select_item_mean: 0.3410
train/objective_mean: 17.3437
train/sym_rel_regret_mean: 0.2498
validation/objective_mean: 15.6051
validation/abs_regret_mean: 9.0817
train/solver_calls_mean: 1125.0000
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.4200
validation/mse_mean: 41.7798
train/loss_mean: 8.5036
validation/sym_rel_regret_mean: 0.2839
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.5036
train/rel_regret_mean: 0.3763
validation/item_value_mean: 4.3274
validation/select_item_mean: 0.3409
train/objective_mean: 17.3437
train/sym_rel_regret_mean: 0.2498
validation/objective_mean: 15.5807
validation/abs_regret_mean: 9.1062
train/solver_calls_mean: 1125.0000
Validation evaluation (abs_regret): 9.203877449035645
Epoch: 5/5
Problem mode set to: train
Epoch Results:
validation/rel_regret_mean: 0.4200
validation/mse_mean: 41.7798
train/loss_mean: 8.5967
validation/sym_rel_regret_mean: 0.2839
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.5967
train/rel_regret_mean: 0.3785
validation/item_value_mean: 4.3274
validation/select_item_mean: 0.3409
train/objective_mean: 17.2506
train/sym_rel_regret_mean: 0.2516
validation/objective_mean: 15.5807
validation/abs_regret_mean: 9.1062
train/solver_calls_mean: 1350.0000
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.4209
validation/mse_mean: 41.7884
train/loss_mean: 8.5967
validation/sym_rel_regret_mean: 0.2847
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.5967
train/rel_regret_mean: 0.3785
validation/item_value_mean: 4.3273
validation/select_item_mean: 0.3409
train/objective_mean: 17.2506
train/sym_rel_regret_mean: 0.2516
validation/objective_mean: 15.5238
validation/abs_regret_mean: 9.1630
train/solver_calls_mean: 1350.0000
Validation evaluation (abs_regret): 9.447274208068848
Training finished. Evaluating on the test set...
Problem mode set to: test
Epoch Results:
validation/rel_regret_mean: 0.4209
validation/sym_rel_regret_mean: 0.2847
test/item_value_mean: 4.3211
validation/item_value_mean: 4.3273
train/sym_rel_regret_mean: 0.2516
test/mse_mean: 27.6233
validation/select_item_mean: 0.3409
validation/mse_mean: 41.7884
train/loss_mean: 8.5967
train/rel_regret_mean: 0.3785
test/rel_regret_mean: 0.4275
validation/objective_mean: 15.5238
test/abs_regret_mean: 9.4049
validation/abs_regret_mean: 9.1630
test/select_item_mean: 0.3407
train/abs_regret_mean: 8.5967
train/grad_norm_mean: 0.0000
train/objective_mean: 17.2506
test/sym_rel_regret_mean: 0.2862
test/objective_mean: 14.6595
train/solver_calls_mean: 1350.0000
Generating data using knapsack
Computing optimal decisions for the entire dataset...
Optimal decisions computed and added to dataset.
Computing optimal objectives for the entire dataset...
Optimal objectives computed and added to dataset.
Shuffling indices before splitting...
Dataset split completed: Train=375, Validation=75, Test=50
Problem mode set to: train
Problem mode set to: train
Epoch 0/5: Starting initial validation...
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.2468
validation/mse_mean: 36.8984
validation/sym_rel_regret_mean: 0.1521
validation/item_value_mean: 4.3190
validation/select_item_mean: 0.4999
validation/objective_mean: 21.8615
validation/abs_regret_mean: 7.4671
Initial best validation metric (abs_regret): 7.467118263244629
Starting training...
Epoch: 1/5
Problem mode set to: train
Epoch Results:
validation/rel_regret_mean: 0.2468
validation/mse_mean: 36.8984
train/loss_mean: 8.8952
validation/sym_rel_regret_mean: 0.1521
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.8952
train/rel_regret_mean: 0.2634
validation/item_value_mean: 4.3190
validation/select_item_mean: 0.4999
train/objective_mean: 22.4465
train/sym_rel_regret_mean: 0.1617
validation/objective_mean: 21.8615
validation/abs_regret_mean: 7.4671
train/solver_calls_mean: 450.0000
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.2445
validation/mse_mean: 36.9268
train/loss_mean: 8.8952
validation/sym_rel_regret_mean: 0.1505
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.8952
train/rel_regret_mean: 0.2634
validation/item_value_mean: 4.3223
validation/select_item_mean: 0.5007
train/objective_mean: 22.4465
train/sym_rel_regret_mean: 0.1617
validation/objective_mean: 21.8909
validation/abs_regret_mean: 7.4377
train/solver_calls_mean: 450.0000
Validation evaluation (abs_regret): 7.408243179321289
New best validation evaluation (abs_regret): 7.408243179321289 (was 7.467118263244629)
Epoch: 2/5
Problem mode set to: train
Epoch Results:
validation/rel_regret_mean: 0.2445
validation/mse_mean: 36.9268
train/loss_mean: 8.8818
validation/sym_rel_regret_mean: 0.1505
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.8818
train/rel_regret_mean: 0.2621
validation/item_value_mean: 4.3223
validation/select_item_mean: 0.5007
train/objective_mean: 22.4599
train/sym_rel_regret_mean: 0.1607
validation/objective_mean: 21.8909
validation/abs_regret_mean: 7.4377
train/solver_calls_mean: 675.0000
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.2440
validation/mse_mean: 36.9325
train/loss_mean: 8.8818
validation/sym_rel_regret_mean: 0.1501
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.8818
train/rel_regret_mean: 0.2621
validation/item_value_mean: 4.3261
validation/select_item_mean: 0.5010
train/objective_mean: 22.4599
train/sym_rel_regret_mean: 0.1607
validation/objective_mean: 21.8888
validation/abs_regret_mean: 7.4398
train/solver_calls_mean: 675.0000
Validation evaluation (abs_regret): 7.444068431854248
Epoch: 3/5
Problem mode set to: train
Epoch Results:
validation/rel_regret_mean: 0.2440
validation/mse_mean: 36.9325
train/loss_mean: 8.8853
validation/sym_rel_regret_mean: 0.1501
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.8853
train/rel_regret_mean: 0.2617
validation/item_value_mean: 4.3261
validation/select_item_mean: 0.5010
train/objective_mean: 22.4564
train/sym_rel_regret_mean: 0.1603
validation/objective_mean: 21.8888
validation/abs_regret_mean: 7.4398
train/solver_calls_mean: 900.0000
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.2427
validation/mse_mean: 36.9519
train/loss_mean: 8.8853
validation/sym_rel_regret_mean: 0.1491
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.8853
train/rel_regret_mean: 0.2617
validation/item_value_mean: 4.3280
validation/select_item_mean: 0.5013
train/objective_mean: 22.4564
train/sym_rel_regret_mean: 0.1603
validation/objective_mean: 21.9130
validation/abs_regret_mean: 7.4155
train/solver_calls_mean: 900.0000
Validation evaluation (abs_regret): 7.342764377593994
New best validation evaluation (abs_regret): 7.342764377593994 (was 7.408243179321289)
Epoch: 4/5
Problem mode set to: train
Epoch Results:
validation/rel_regret_mean: 0.2427
validation/mse_mean: 36.9519
train/loss_mean: 8.9087
validation/sym_rel_regret_mean: 0.1491
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.9087
train/rel_regret_mean: 0.2609
validation/item_value_mean: 4.3280
validation/select_item_mean: 0.5013
train/objective_mean: 22.4331
train/sym_rel_regret_mean: 0.1597
validation/objective_mean: 21.9130
validation/abs_regret_mean: 7.4155
train/solver_calls_mean: 1125.0000
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.2415
validation/mse_mean: 24.4883
train/loss_mean: 9.5644
validation/sym_rel_regret_mean: 0.1464
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 9.5644
train/rel_regret_mean: 0.2804
validation/item_value_mean: 3.8154
validation/select_item_mean: 0.4608
train/objective_mean: 20.6071
train/sym_rel_regret_mean: 0.1749
validation/objective_mean: 21.0447
validation/abs_regret_mean: 6.7184
train/solver_calls_mean: 1125.0000
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.2423
validation/mse_mean: 24.5902
train/loss_mean: 9.5644
validation/sym_rel_regret_mean: 0.1471
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 9.5644
train/rel_regret_mean: 0.2804
validation/item_value_mean: 3.8068
validation/select_item_mean: 0.4604
train/objective_mean: 20.6071
train/sym_rel_regret_mean: 0.1749
validation/objective_mean: 21.0151
validation/abs_regret_mean: 6.7480
train/solver_calls_mean: 1125.0000
Validation evaluation (abs_regret): 6.866395473480225
Epoch: 5/5
Problem mode set to: train
Epoch Results:
validation/rel_regret_mean: 0.2423
validation/mse_mean: 24.5902
train/loss_mean: 9.5869
validation/sym_rel_regret_mean: 0.1471
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 9.5869
train/rel_regret_mean: 0.2806
validation/item_value_mean: 3.8068
validation/select_item_mean: 0.4604
train/objective_mean: 20.5846
train/sym_rel_regret_mean: 0.1752
validation/objective_mean: 21.0151
validation/abs_regret_mean: 6.7480
train/solver_calls_mean: 1350.0000
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.2428
validation/mse_mean: 24.6599
train/loss_mean: 9.5869
validation/sym_rel_regret_mean: 0.1475
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 9.5869
train/rel_regret_mean: 0.2806
validation/item_value_mean: 3.8018
validation/select_item_mean: 0.4602
train/objective_mean: 20.5846
train/sym_rel_regret_mean: 0.1752
validation/objective_mean: 20.9964
validation/abs_regret_mean: 6.7667
train/solver_calls_mean: 1350.0000
Validation evaluation (abs_regret): 6.860617637634277
Training finished. Evaluating on the test set...
Problem mode set to: test
Epoch Results:
validation/rel_regret_mean: 0.2428
validation/sym_rel_regret_mean: 0.1475
test/item_value_mean: 3.8784
validation/item_value_mean: 3.8018
train/sym_rel_regret_mean: 0.1752
test/mse_mean: 44.9416
validation/select_item_mean: 0.4602
validation/mse_mean: 24.6599
train/loss_mean: 9.5869
train/rel_regret_mean: 0.2806
test/rel_regret_mean: 0.2576
validation/objective_mean: 20.9964
test/abs_regret_mean: 8.6495
validation/abs_regret_mean: 6.7667
test/select_item_mean: 0.4629
train/abs_regret_mean: 9.5869
train/grad_norm_mean: 0.0000
train/objective_mean: 20.5846
test/sym_rel_regret_mean: 0.1573
test/objective_mean: 20.6775
train/solver_calls_mean: 1350.0000
[I 2025-10-21 11:40:36,546] Trial 1 finished with value: 7.542421658833821 and parameters: {'learning_rate': 0.0006581524232209884, 'batch_size': 498}. Best is trial 1 with value: 7.542421658833821.
Generating data using knapsack
Computing optimal decisions for the entire dataset...
Optimal decisions computed and added to dataset.
Computing optimal objectives for the entire dataset...
Optimal objectives computed and added to dataset.
Shuffling indices before splitting...
Dataset split completed: Train=375, Validation=75, Test=50
Problem mode set to: train
Problem mode set to: train
Epoch 0/5: Starting initial validation...
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.4311
validation/mse_mean: 34.8658
validation/sym_rel_regret_mean: 0.2933
validation/item_value_mean: 4.3363
validation/select_item_mean: 0.3412
validation/objective_mean: 13.9579
validation/abs_regret_mean: 8.5578
Initial best validation metric (abs_regret): 8.55784797668457
Starting training...
Epoch: 1/5
Problem mode set to: train
Epoch Results:
validation/rel_regret_mean: 0.4311
validation/mse_mean: 34.8658
train/loss_mean: 8.0180
validation/sym_rel_regret_mean: 0.2933
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.0180
train/rel_regret_mean: 0.3677
validation/item_value_mean: 4.3363
validation/select_item_mean: 0.3412
train/objective_mean: 18.0093
train/sym_rel_regret_mean: 0.2441
validation/objective_mean: 13.9579
validation/abs_regret_mean: 8.5578
train/solver_calls_mean: 296.8750
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.4324
validation/mse_mean: 34.8662
train/loss_mean: 8.0180
validation/sym_rel_regret_mean: 0.2946
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.0180
train/rel_regret_mean: 0.3677
validation/item_value_mean: 4.3361
validation/select_item_mean: 0.3411
train/objective_mean: 18.0093
train/sym_rel_regret_mean: 0.2441
validation/objective_mean: 13.9493
validation/abs_regret_mean: 8.5664
train/solver_calls_mean: 296.8750
Validation evaluation (abs_regret): 8.575040817260742
Epoch: 2/5
Problem mode set to: train
Epoch Results:
validation/rel_regret_mean: 0.4324
validation/mse_mean: 34.8662
train/loss_mean: 7.9754
validation/sym_rel_regret_mean: 0.2946
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 7.9754
train/rel_regret_mean: 0.3686
validation/item_value_mean: 4.3361
validation/select_item_mean: 0.3411
train/objective_mean: 17.8123
train/sym_rel_regret_mean: 0.2446
validation/objective_mean: 13.9493
validation/abs_regret_mean: 8.5664
train/solver_calls_mean: 521.8750
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.4323
validation/mse_mean: 34.8643
train/loss_mean: 7.9754
validation/sym_rel_regret_mean: 0.2945
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 7.9754
train/rel_regret_mean: 0.3686
validation/item_value_mean: 4.3351
validation/select_item_mean: 0.3411
train/objective_mean: 17.8123
train/sym_rel_regret_mean: 0.2446
validation/objective_mean: 13.9489
validation/abs_regret_mean: 8.5669
train/solver_calls_mean: 521.8750
Validation evaluation (abs_regret): 8.567665100097656
Epoch: 3/5
Problem mode set to: train
Epoch Results:
validation/rel_regret_mean: 0.4323
validation/mse_mean: 34.8643
train/loss_mean: 8.0722
validation/sym_rel_regret_mean: 0.2945
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.0722
train/rel_regret_mean: 0.3705
validation/item_value_mean: 4.3351
validation/select_item_mean: 0.3411
train/objective_mean: 17.7830
train/sym_rel_regret_mean: 0.2458
validation/objective_mean: 13.9489
validation/abs_regret_mean: 8.5669
train/solver_calls_mean: 746.8750
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.4319
validation/mse_mean: 34.8624
train/loss_mean: 8.0722
validation/sym_rel_regret_mean: 0.2942
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.0722
train/rel_regret_mean: 0.3705
validation/item_value_mean: 4.3338
validation/select_item_mean: 0.3411
train/objective_mean: 17.7830
train/sym_rel_regret_mean: 0.2458
validation/objective_mean: 13.9538
validation/abs_regret_mean: 8.5619
train/solver_calls_mean: 746.8750
Validation evaluation (abs_regret): 8.547199249267578
New best validation evaluation (abs_regret): 8.547199249267578 (was 8.55784797668457)
Epoch: 4/5
Problem mode set to: train
Epoch Results:
validation/rel_regret_mean: 0.4319
validation/mse_mean: 34.8624
train/loss_mean: 8.0540
validation/sym_rel_regret_mean: 0.2942
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.0540
train/rel_regret_mean: 0.3710
validation/item_value_mean: 4.3338
validation/select_item_mean: 0.3411
train/objective_mean: 17.7011
train/sym_rel_regret_mean: 0.2462
validation/objective_mean: 13.9538
validation/abs_regret_mean: 8.5619
train/solver_calls_mean: 971.8750
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.4314
validation/mse_mean: 34.8620
train/loss_mean: 8.0540
validation/sym_rel_regret_mean: 0.2936
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.0540
train/rel_regret_mean: 0.3710
validation/item_value_mean: 4.3331
validation/select_item_mean: 0.3411
train/objective_mean: 17.7011
train/sym_rel_regret_mean: 0.2462
validation/objective_mean: 13.9607
validation/abs_regret_mean: 8.5550
train/solver_calls_mean: 971.8750
Validation evaluation (abs_regret): 8.527207374572754
New best validation evaluation (abs_regret): 8.527207374572754 (was 8.547199249267578)
Epoch: 5/5
Problem mode set to: train
Epoch Results:
validation/rel_regret_mean: 0.4314
validation/mse_mean: 34.8620
train/loss_mean: 8.0799
validation/sym_rel_regret_mean: 0.2936
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.0799
train/rel_regret_mean: 0.3718
validation/item_value_mean: 4.3331
validation/select_item_mean: 0.3411
train/objective_mean: 17.6700
train/sym_rel_regret_mean: 0.2467
validation/objective_mean: 13.9607
validation/abs_regret_mean: 8.5550
train/solver_calls_mean: 1196.8750
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.4314
validation/mse_mean: 34.8632
train/loss_mean: 8.0799
validation/sym_rel_regret_mean: 0.2936
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.0799
train/rel_regret_mean: 0.3718
validation/item_value_mean: 4.3326
validation/select_item_mean: 0.3411
train/objective_mean: 17.6700
train/sym_rel_regret_mean: 0.2467
validation/objective_mean: 13.9601
validation/abs_regret_mean: 8.5556
train/solver_calls_mean: 1196.8750
Validation evaluation (abs_regret): 8.55860710144043
Training finished. Evaluating on the test set...
Problem mode set to: test
Epoch Results:
validation/rel_regret_mean: 0.4314
validation/sym_rel_regret_mean: 0.2936
test/item_value_mean: 4.3144
validation/item_value_mean: 4.3326
train/sym_rel_regret_mean: 0.2467
test/mse_mean: 27.6208
validation/select_item_mean: 0.3411
validation/mse_mean: 34.8632
train/loss_mean: 8.0799
train/rel_regret_mean: 0.3718
test/rel_regret_mean: 0.4249
validation/objective_mean: 13.9601
test/abs_regret_mean: 9.2402
validation/abs_regret_mean: 8.5556
test/select_item_mean: 0.3410
train/abs_regret_mean: 8.0799
train/grad_norm_mean: 0.0000
train/objective_mean: 17.6700
test/sym_rel_regret_mean: 0.2841
test/objective_mean: 14.8243
train/solver_calls_mean: 1196.8750
Generating data using knapsack
Computing optimal decisions for the entire dataset...
Optimal decisions computed and added to dataset.
Computing optimal objectives for the entire dataset...
Optimal objectives computed and added to dataset.
Shuffling indices before splitting...
Dataset split completed: Train=375, Validation=75, Test=50
Problem mode set to: train
Problem mode set to: train
Epoch 0/5: Starting initial validation...
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.2465
validation/mse_mean: 43.8282
validation/sym_rel_regret_mean: 0.1501
validation/item_value_mean: 4.3194
validation/select_item_mean: 0.4999
validation/objective_mean: 22.1269
validation/abs_regret_mean: 7.8851
Initial best validation metric (abs_regret): 7.885140419006348
Starting training...
Epoch: 1/5
Problem mode set to: train
Epoch Results:
validation/rel_regret_mean: 0.2465
validation/mse_mean: 43.8282
train/loss_mean: 8.7369
validation/sym_rel_regret_mean: 0.1501
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.7369
train/rel_regret_mean: 0.2660
validation/item_value_mean: 4.3194
validation/select_item_mean: 0.4999
train/objective_mean: 21.9465
train/sym_rel_regret_mean: 0.1635
validation/objective_mean: 22.1269
validation/abs_regret_mean: 7.8851
train/solver_calls_mean: 296.8750
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.2455
validation/mse_mean: 43.8287
train/loss_mean: 8.7369
validation/sym_rel_regret_mean: 0.1493
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.7369
train/rel_regret_mean: 0.2660
validation/item_value_mean: 4.3198
validation/select_item_mean: 0.4999
train/objective_mean: 21.9465
train/sym_rel_regret_mean: 0.1635
validation/objective_mean: 22.1433
validation/abs_regret_mean: 7.8688
train/solver_calls_mean: 296.8750
Validation evaluation (abs_regret): 7.8523736000061035
New best validation evaluation (abs_regret): 7.8523736000061035 (was 7.885140419006348)
Epoch: 2/5
Problem mode set to: train
Epoch Results:
validation/rel_regret_mean: 0.2455
validation/mse_mean: 43.8287
train/loss_mean: 9.0459
validation/sym_rel_regret_mean: 0.1493
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 9.0459
train/rel_regret_mean: 0.2661
validation/item_value_mean: 4.3198
validation/select_item_mean: 0.4999
train/objective_mean: 22.3657
train/sym_rel_regret_mean: 0.1637
validation/objective_mean: 22.1433
validation/abs_regret_mean: 7.8688
train/solver_calls_mean: 521.8750
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.2452
validation/mse_mean: 43.8298
train/loss_mean: 9.0459
validation/sym_rel_regret_mean: 0.1491
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 9.0459
train/rel_regret_mean: 0.2661
validation/item_value_mean: 4.3201
validation/select_item_mean: 0.4999
train/objective_mean: 22.3657
train/sym_rel_regret_mean: 0.1637
validation/objective_mean: 22.1462
validation/abs_regret_mean: 7.8658
train/solver_calls_mean: 521.8750
Validation evaluation (abs_regret): 7.859895706176758
Epoch: 3/5
Problem mode set to: train
Epoch Results:
validation/rel_regret_mean: 0.2452
validation/mse_mean: 43.8298
train/loss_mean: 9.0263
validation/sym_rel_regret_mean: 0.1491
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 9.0263
train/rel_regret_mean: 0.2662
validation/item_value_mean: 4.3201
validation/select_item_mean: 0.4999
train/objective_mean: 22.3473
train/sym_rel_regret_mean: 0.1638
validation/objective_mean: 22.1462
validation/abs_regret_mean: 7.8658
train/solver_calls_mean: 746.8750
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.2450
validation/mse_mean: 43.8315
train/loss_mean: 9.0263
validation/sym_rel_regret_mean: 0.1489
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 9.0263
train/rel_regret_mean: 0.2662
validation/item_value_mean: 4.3202
validation/select_item_mean: 0.4999
train/objective_mean: 22.3473
train/sym_rel_regret_mean: 0.1638
validation/objective_mean: 22.1477
validation/abs_regret_mean: 7.8643
train/solver_calls_mean: 746.8750
Validation evaluation (abs_regret): 7.859895706176758
Epoch: 4/5
Problem mode set to: train
Epoch Results:
validation/rel_regret_mean: 0.2450
validation/mse_mean: 43.8315
train/loss_mean: 8.9924
validation/sym_rel_regret_mean: 0.1489
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.9924
train/rel_regret_mean: 0.2651
validation/item_value_mean: 4.3202
validation/select_item_mean: 0.4999
train/objective_mean: 22.4252
train/sym_rel_regret_mean: 0.1630
validation/objective_mean: 22.1477
validation/abs_regret_mean: 7.8643
train/solver_calls_mean: 971.8750
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.2447
validation/mse_mean: 43.8340
train/loss_mean: 8.9924
validation/sym_rel_regret_mean: 0.1486
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.9924
train/rel_regret_mean: 0.2651
validation/item_value_mean: 4.3205
validation/select_item_mean: 0.4999
train/objective_mean: 22.4252
train/sym_rel_regret_mean: 0.1630
validation/objective_mean: 22.1405
validation/abs_regret_mean: 7.8715
train/solver_calls_mean: 971.8750
Validation evaluation (abs_regret): 7.90017032623291
Epoch: 5/5
Problem mode set to: train
Epoch Results:
validation/rel_regret_mean: 0.2447
validation/mse_mean: 43.8340
train/loss_mean: 8.9693
validation/sym_rel_regret_mean: 0.1486
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.9693
train/rel_regret_mean: 0.2649
validation/item_value_mean: 4.3205
validation/select_item_mean: 0.4999
train/objective_mean: 22.3722
train/sym_rel_regret_mean: 0.1628
validation/objective_mean: 22.1405
validation/abs_regret_mean: 7.8715
train/solver_calls_mean: 1196.8750
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.2445
validation/mse_mean: 43.8359
train/loss_mean: 8.9693
validation/sym_rel_regret_mean: 0.1484
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 8.9693
train/rel_regret_mean: 0.2649
validation/item_value_mean: 4.3207
validation/select_item_mean: 0.4999
train/objective_mean: 22.3722
train/sym_rel_regret_mean: 0.1628
validation/objective_mean: 22.1357
validation/abs_regret_mean: 7.8763
train/solver_calls_mean: 1196.8750
Validation evaluation (abs_regret): 7.90017032623291
Training finished. Evaluating on the test set...
Problem mode set to: test
Epoch Results:
validation/rel_regret_mean: 0.2445
validation/sym_rel_regret_mean: 0.1484
test/item_value_mean: 4.2984
validation/item_value_mean: 4.3207
train/sym_rel_regret_mean: 0.1628
test/mse_mean: 21.6649
validation/select_item_mean: 0.4999
validation/mse_mean: 43.8359
train/loss_mean: 8.9693
train/rel_regret_mean: 0.2649
test/rel_regret_mean: 0.2349
validation/objective_mean: 22.1357
test/abs_regret_mean: 7.5118
validation/abs_regret_mean: 7.8763
test/select_item_mean: 0.5017
train/abs_regret_mean: 8.9693
train/grad_norm_mean: 0.0000
train/objective_mean: 22.3722
test/sym_rel_regret_mean: 0.1443
test/objective_mean: 22.2793
train/solver_calls_mean: 1196.8750
Generating data using knapsack
Computing optimal decisions for the entire dataset...
Optimal decisions computed and added to dataset.
Computing optimal objectives for the entire dataset...
Optimal objectives computed and added to dataset.
Shuffling indices before splitting...
Dataset split completed: Train=375, Validation=75, Test=50
Problem mode set to: train
Problem mode set to: train
Epoch 0/5: Starting initial validation...
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.2338
validation/mse_mean: 25.3590
validation/sym_rel_regret_mean: 0.1395
validation/item_value_mean: 3.8675
validation/select_item_mean: 0.4642
validation/objective_mean: 21.7255
validation/abs_regret_mean: 6.4268
Initial best validation metric (abs_regret): 6.42680549621582
Starting training...
Epoch: 1/5
Problem mode set to: train
Epoch Results:
validation/rel_regret_mean: 0.2338
validation/mse_mean: 25.3590
train/loss_mean: 9.4179
validation/sym_rel_regret_mean: 0.1395
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 9.4179
train/rel_regret_mean: 0.2774
validation/item_value_mean: 3.8675
validation/select_item_mean: 0.4642
train/objective_mean: 20.5956
train/sym_rel_regret_mean: 0.1733
validation/objective_mean: 21.7255
validation/abs_regret_mean: 6.4268
train/solver_calls_mean: 296.8750
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.2338
validation/mse_mean: 25.3414
train/loss_mean: 9.4179
validation/sym_rel_regret_mean: 0.1394
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 9.4179
train/rel_regret_mean: 0.2774
validation/item_value_mean: 3.8692
validation/select_item_mean: 0.4641
train/objective_mean: 20.5956
train/sym_rel_regret_mean: 0.1733
validation/objective_mean: 21.7263
validation/abs_regret_mean: 6.4260
train/solver_calls_mean: 296.8750
Validation evaluation (abs_regret): 6.425119400024414
New best validation evaluation (abs_regret): 6.425119400024414 (was 6.42680549621582)
Epoch: 2/5
Problem mode set to: train
Epoch Results:
validation/rel_regret_mean: 0.2338
validation/mse_mean: 25.3414
train/loss_mean: 9.2864
validation/sym_rel_regret_mean: 0.1394
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 9.2864
train/rel_regret_mean: 0.2764
validation/item_value_mean: 3.8692
validation/select_item_mean: 0.4641
train/objective_mean: 20.4727
train/sym_rel_regret_mean: 0.1726
validation/objective_mean: 21.7263
validation/abs_regret_mean: 6.4260
train/solver_calls_mean: 521.8750
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.2337
validation/mse_mean: 25.3295
train/loss_mean: 9.2864
validation/sym_rel_regret_mean: 0.1393
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 9.2864
train/rel_regret_mean: 0.2764
validation/item_value_mean: 3.8703
validation/select_item_mean: 0.4643
train/objective_mean: 20.4727
train/sym_rel_regret_mean: 0.1726
validation/objective_mean: 21.7310
validation/abs_regret_mean: 6.4213
train/solver_calls_mean: 521.8750
Validation evaluation (abs_regret): 6.411908149719238
New best validation evaluation (abs_regret): 6.411908149719238 (was 6.425119400024414)
Epoch: 3/5
Problem mode set to: train
Epoch Results:
validation/rel_regret_mean: 0.2337
validation/mse_mean: 25.3295
train/loss_mean: 9.3091
validation/sym_rel_regret_mean: 0.1393
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 9.3091
train/rel_regret_mean: 0.2763
validation/item_value_mean: 3.8703
validation/select_item_mean: 0.4643
train/objective_mean: 20.4893
train/sym_rel_regret_mean: 0.1725
validation/objective_mean: 21.7310
validation/abs_regret_mean: 6.4213
train/solver_calls_mean: 746.8750
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.2332
validation/mse_mean: 25.3174
train/loss_mean: 9.3091
validation/sym_rel_regret_mean: 0.1390
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 9.3091
train/rel_regret_mean: 0.2763
validation/item_value_mean: 3.8714
validation/select_item_mean: 0.4643
train/objective_mean: 20.4893
train/sym_rel_regret_mean: 0.1725
validation/objective_mean: 21.7498
validation/abs_regret_mean: 6.4025
train/solver_calls_mean: 746.8750
Validation evaluation (abs_regret): 6.346304416656494
New best validation evaluation (abs_regret): 6.346304416656494 (was 6.411908149719238)
Epoch: 4/5
Problem mode set to: train
Epoch Results:
validation/rel_regret_mean: 0.2332
validation/mse_mean: 25.3174
train/loss_mean: 9.3790
validation/sym_rel_regret_mean: 0.1390
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 9.3790
train/rel_regret_mean: 0.2772
validation/item_value_mean: 3.8714
validation/select_item_mean: 0.4643
train/objective_mean: 20.4860
train/sym_rel_regret_mean: 0.1731
validation/objective_mean: 21.7498
validation/abs_regret_mean: 6.4025
train/solver_calls_mean: 971.8750
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.2330
validation/mse_mean: 25.3091
train/loss_mean: 9.3790
validation/sym_rel_regret_mean: 0.1388
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 9.3790
train/rel_regret_mean: 0.2772
validation/item_value_mean: 3.8722
validation/select_item_mean: 0.4644
train/objective_mean: 20.4860
train/sym_rel_regret_mean: 0.1731
validation/objective_mean: 21.7610
validation/abs_regret_mean: 6.3913
train/solver_calls_mean: 971.8750
Validation evaluation (abs_regret): 6.346304416656494
Epoch: 5/5
Problem mode set to: train
Epoch Results:
validation/rel_regret_mean: 0.2330
validation/mse_mean: 25.3091
train/loss_mean: 9.4203
validation/sym_rel_regret_mean: 0.1388
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 9.4203
train/rel_regret_mean: 0.2774
validation/item_value_mean: 3.8722
validation/select_item_mean: 0.4644
train/objective_mean: 20.5266
train/sym_rel_regret_mean: 0.1732
validation/objective_mean: 21.7610
validation/abs_regret_mean: 6.3913
train/solver_calls_mean: 1196.8750
Problem mode set to: validation
Epoch Results:
validation/rel_regret_mean: 0.2328
validation/mse_mean: 25.3039
train/loss_mean: 9.4203
validation/sym_rel_regret_mean: 0.1387
train/grad_norm_mean: 0.0000
train/abs_regret_mean: 9.4203
train/rel_regret_mean: 0.2774
validation/item_value_mean: 3.8727
validation/select_item_mean: 0.4644
train/objective_mean: 20.5266
train/sym_rel_regret_mean: 0.1732
validation/objective_mean: 21.7685
validation/abs_regret_mean: 6.3838
train/solver_calls_mean: 1196.8750
Validation evaluation (abs_regret): 6.346304416656494
Training finished. Evaluating on the test set...
Problem mode set to: test
Epoch Results:
validation/rel_regret_mean: 0.2328
validation/sym_rel_regret_mean: 0.1387
test/item_value_mean: 3.8848
validation/item_value_mean: 3.8727
train/sym_rel_regret_mean: 0.1732
test/mse_mean: 44.7985
validation/select_item_mean: 0.4644
validation/mse_mean: 25.3039
train/loss_mean: 9.4203
train/rel_regret_mean: 0.2774
test/rel_regret_mean: 0.2537
validation/objective_mean: 21.7685
test/abs_regret_mean: 8.4771
validation/abs_regret_mean: 6.3838
test/select_item_mean: 0.4635
train/abs_regret_mean: 9.4203
train/grad_norm_mean: 0.0000
train/objective_mean: 20.5266
test/sym_rel_regret_mean: 0.1547
test/objective_mean: 20.8499
train/solver_calls_mean: 1196.8750
[I 2025-10-21 11:40:48,217] Trial 2 finished with value: 7.575295130411784 and parameters: {'learning_rate': 1.3672881452765286e-05, 'batch_size': 50}. Best is trial 1 with value: 7.542421658833821.

4.8 Retrieve and save results¶

The dashboard summarizes the results. Alternatively results can be retrieved and saved as follows.

[9]:

import optuna
from pydflt.utils.optuna import save_progress

path = f"sqlite:///{OUTPUT_DIR}/{STUDY_NAME}/database.db"
studies = optuna.study.get_all_study_summaries(storage=path)

for study_summary in studies:
    study_name = study_summary.study_name
    study = optuna.load_study(study_name=study_name, storage=path)

    print(
        f"Results study: {study_name}\n"
        f"Completed trials: {len(study.trials)}\n"
        f"Best value: {study.best_value:.4f}\n"
        f"Best parameters: {study.best_params}"
    )

save_progress(study, search_space, OUTPUT_DIR)

Results study: test_study
Completed trials: 3
Best value: 7.5424
Best parameters: {'learning_rate': 0.0006581524232209884, 'batch_size': 498}
Best configuration saved to hparam_optimization_results/best_config.yaml