T7 - Calibration

We saw in Tutorial 4 how to load and plot data. But the next step is to actually calibrate the model to the data, i.e. find the model parameters that are the most likely explanation for the observed data. This tutorial gives an introduction to the Fit object and some recipes for optimization approaches.

Click here to open an interactive version of this notebook.

The Fit object

The Fit object is responsible for quantifying how well a given model run matches the data. Let’s consider a simple example, building on Tutorial 4:

import covasim as cv
cv.options(jupyter=True, verbose=0)

pars = dict(
    pop_size  = 20_000,
    start_day = '2020-02-01',
    end_day   = '2020-04-11',
)
sim = cv.Sim(pars=pars, datafile='example_data.csv', interventions=cv.test_num(daily_tests='data'))
sim.run()
sim.plot(to_plot=['cum_tests', 'cum_diagnoses', 'cum_deaths'])

Covasim 4.0.1 (2026-10-16) — © 2020-2026 by IDM
  Automatically adding cumulative column cum_diagnoses from new_diagnoses
  Automatically adding cumulative column cum_tests from new_tests
  Automatically adding cumulative column cum_deaths from new_deaths

We can see that tests match extremely well (they’re input data!), diagnoses match reasonably well, and deaths match poorly. Can the Fit object capture our intuition about this?

fit = sim.compute_fit()
fit.summarize()

Mismatch values for:
#0. 'cum_deaths':    49.565217391304344
#1. 'cum_tests':     0.0
#2. 'cum_diagnoses': 44.96169560776302

Total mismatch value:
94.52691299906736

So the results seem to match our intuition. (Note that by default the Fit object uses normalized absolute difference, but other estimates, such as mean squared error, are also possible.)

What if we improve the fit? Does the mismatch reduce?

sim['rel_death_prob'] = 2 # Increase the death rate since deaths were too low
sim.initialize(reset=True) # Reinitialize the sim

# Rerun and compute fit
sim.run()
fit = sim.compute_fit()

# Output
sim.plot(to_plot=['cum_tests', 'cum_diagnoses', 'cum_deaths'])

fit.plot()
fit.summarize()

Mismatch values for:
#0. 'cum_deaths':    73.04347826086956
#1. 'cum_tests':     0.0
#2. 'cum_diagnoses': 44.95658835546476

Total mismatch value:
118.00006661633432

As expected, the fit is slightly improved. By now, you may be wondering how the mistmatch is actually calculated in Covasim. It follows more or less this expression:

\(\text{mismatch} = \sum_i\left(\sum_t\dfrac{|d_i(t) - m_i(t)|}{\text{max}(d_i(t))}\right)\)

where \(i\) is over different data quantities (eg, cum_tests, cum_deaths, cum_diagnoses in this example); \(t\) is time; \(d_i(t)\) is the time series of data for quantity \(i\) (the lines with square markers in the figures); and, \(m_i(t)\) is the time series of the model for quantity \(i\) (solid lines).

Each data quantity \(i\) has very different scales. Cumulative deaths could be in the few hundreds, while cumulative diagnoses could be in the tens of thousands (or more). For that reason we use the term \(\text{max}(d_i(t))\), so we can reasonably add individual mismatches of different quantities like cumulative deaths and diagnoses, to produce an overall mismatch, which is a ‘normalized’ absolute error of sorts.

Calibration approaches

Calibration is a complex and dark art and cannot be covered fully here; many books have been written about it and it continues to be an area of active research. A good review article about calibrating agent-based models like Covasim is available here. Calibration is usually expressed as an optimization problem: specifically, find a vector of parameters θ that minimizes the mismatch between the data D and the model M(θ).

In practice, most calibration is done simply by hand, as in the example above. Once deaths are “calibrated”, the user might modify testing assumptions so that the diagnoses match. Since we are only fitting to deaths and diagnoses, the model is then “calibrated”.

However, automated approaches to calibration are possible as well. The simplest is probably the built-in SciPy optimization functions, e.g. scipy.optimize. A wrinkle here is that normal gradient descent methods will not work with Covasim or other agent-based models, due to the stochastic variability between model runs that makes the landscape very “bumpy”. One way of getting around this is to use many different runs and take the average, e.g.:

import covasim as cv
import numpy as np
from scipy import optimize

def objective(x, n_runs=10):
    print(f'Running sim for beta={x[0]}, rel_death_prob={x[1]}')
    pars = dict(
        pop_size       = 20_000,
        start_day      = '2020-02-01',
        end_day        = '2020-04-11',
        beta           = x[0],
        rel_death_prob = x[1],
        verbose        = 0,
    )
    sim = cv.Sim(pars=pars, datafile='example_data.csv', interventions=cv.test_num(daily_tests='data'))
    msim = cv.MultiSim(sim)
    msim.run(n_runs=n_runs)
    mismatches = []
    for sim in msim.sims:
        fit = sim.compute_fit()
        mismatches.append(fit.mismatch)
    mismatch = np.mean(mismatches)
    return mismatch

guess = [0.015, 1] # Initial guess of parameters -- beta and relative death probability
pars = optimize.minimize(objective, x0=guess, method='nelder-mead') # Run the optimization

This should converge after roughly 3-10 minutes, although you will likely find that the improvement is minimal.

What’s happening here? Trying to overcome the limitations of an algorithm that expects deterministic results simply by running more sims is fairly futile – if you run N sims and average them together, you’ve only reduced noise by √N, i.e. you have to average together 100 sims to reduce noise by a factor of 10, and even that might not be enough. Clearly, we need a more powerful approach.

Built-in calibration

One such package we have found works reasonably well is called Optuna. It is built into Covasim as sim.calibrate() (it’s not installed by default, so please install it first with pip install optuna). Do not expect this to be a magic bullet solution: you will likely still need to try out multiple different parameter sets for calibration, manually update the values of uncalibrated parameters, check if the data actually make sense, etc. Even once all these things are in place, it still needs to be run for enough iterations, which might be a few hundred iterations for 3-4 calibrated (free) parameters or tens of thousands of iterations for 10 or more free parameters. The example below should get you started, but best to expect that it will not work for your particular use case without significant modification!

'''
Example for running built-in calibration with Optuna
'''

import sciris as sc
import covasim as cv

# Create default simulation
pars = sc.objdict(
    pop_size       = 20_000,
    start_day      = '2020-02-01',
    end_day        = '2020-04-11',
    beta           = 0.015,
    rel_death_prob = 1.0,
    interventions  = cv.test_num(daily_tests='data'),
    verbose        = 0,
)
sim = cv.Sim(pars=pars, datafile='example_data.csv')

# Parameters to calibrate -- format is best, low, high
calib_pars = dict(
    beta           = [pars.beta, 0.005, 0.20],
    rel_death_prob = [pars.rel_death_prob, 0.5, 3.0],
)

if __name__ == '__main__':

    # Run the calibration
    calib = sim.calibrate(calib_pars=calib_pars, total_trials=100)

  Automatically adding cumulative column cum_diagnoses from new_diagnoses
  Automatically adding cumulative column cum_tests from new_tests
  Automatically adding cumulative column cum_deaths from new_deaths
Removed existing calibration file /tmp/tmp9jvhewug/covasim_calibration.db
sqlite:////tmp/tmp9jvhewug/covasim_calibration.db

[I 2026-10-16 16:11:40,121] A new study created in RDB with name: covasim_calibration

[I 2026-10-16 16:11:42,871] Trial 2 finished with value: 369.09868099658036 and parameters: {'beta': 0.023002552719921552, 'rel_death_prob': 0.6933501380034991}. Best is trial 2 with value: 369.09868099658036.

[I 2026-10-16 16:11:42,918] Trial 1 finished with value: 1269.0815828041036 and parameters: {'beta': 0.09121114883429964, 'rel_death_prob': 1.187838853392936}. Best is trial 2 with value: 369.09868099658036.

[I 2026-10-16 16:11:43,036] Trial 3 finished with value: 2788.0433672336458 and parameters: {'beta': 0.17739840909469856, 'rel_death_prob': 1.966369937428822}. Best is trial 2 with value: 369.09868099658036.

[I 2026-10-16 16:11:43,061] Trial 0 finished with value: 2818.1138695208065 and parameters: {'beta': 0.17174901621978053, 'rel_death_prob': 2.176279954416776}. Best is trial 2 with value: 369.09868099658036.

[I 2026-10-16 16:11:44,599] Trial 4 finished with value: 1255.6648310165651 and parameters: {'beta': 0.03715171112381674, 'rel_death_prob': 2.2932086372723868}. Best is trial 2 with value: 369.09868099658036.

[I 2026-10-16 16:11:44,603] Trial 5 finished with value: 1354.80947728383 and parameters: {'beta': 0.06214218388195482, 'rel_death_prob': 1.7166643899018346}. Best is trial 2 with value: 369.09868099658036.

[I 2026-10-16 16:11:44,769] Trial 7 finished with value: 1643.1932539858772 and parameters: {'beta': 0.07290762868031589, 'rel_death_prob': 1.7891976282825433}. Best is trial 2 with value: 369.09868099658036.

[I 2026-10-16 16:11:44,820] Trial 6 finished with value: 1626.531953635031 and parameters: {'beta': 0.16245133024704195, 'rel_death_prob': 1.101701937085906}. Best is trial 2 with value: 369.09868099658036.

[I 2026-10-16 16:11:46,307] Trial 9 finished with value: 1165.2250521827953 and parameters: {'beta': 0.12502828730360843, 'rel_death_prob': 0.969912425156491}. Best is trial 2 with value: 369.09868099658036.

[I 2026-10-16 16:11:46,317] Trial 8 finished with value: 2548.800794954923 and parameters: {'beta': 0.150719071893702, 'rel_death_prob': 2.1889500990297743}. Best is trial 2 with value: 369.09868099658036.

[I 2026-10-16 16:11:46,458] Trial 11 finished with value: 719.2449038504242 and parameters: {'beta': 0.023649458367440448, 'rel_death_prob': 2.2598912071377244}. Best is trial 2 with value: 369.09868099658036.

[I 2026-10-16 16:11:46,482] Trial 10 finished with value: 3208.024714660035 and parameters: {'beta': 0.16972955175571622, 'rel_death_prob': 2.9946738130569925}. Best is trial 2 with value: 369.09868099658036.

[I 2026-10-16 16:11:47,710] Trial 12 finished with value: 101.61156015454989 and parameters: {'beta': 0.01140587478555763, 'rel_death_prob': 0.6205338185147005}. Best is trial 12 with value: 101.61156015454989.

[I 2026-10-16 16:11:47,736] Trial 14 finished with value: 129.50859350712796 and parameters: {'beta': 0.008173421711085689, 'rel_death_prob': 2.8578779526952642}. Best is trial 12 with value: 101.61156015454989.

[I 2026-10-16 16:11:48,014] Trial 13 finished with value: 2661.305347071102 and parameters: {'beta': 0.13575935827703134, 'rel_death_prob': 2.7383382447944373}. Best is trial 12 with value: 101.61156015454989.

[I 2026-10-16 16:11:48,111] Trial 15 finished with value: 679.6368299507039 and parameters: {'beta': 0.027746493198058345, 'rel_death_prob': 1.374267086477777}. Best is trial 12 with value: 101.61156015454989.

[I 2026-10-16 16:11:48,881] Trial 16 finished with value: 134.44231025447442 and parameters: {'beta': 0.005943599564144072, 'rel_death_prob': 2.9589712722934913}. Best is trial 12 with value: 101.61156015454989.

[I 2026-10-16 16:11:49,408] Trial 17 finished with value: 2300.56812630457 and parameters: {'beta': 0.08213204880615743, 'rel_death_prob': 2.8808409594157767}. Best is trial 12 with value: 101.61156015454989.

[I 2026-10-16 16:11:49,672] Trial 18 finished with value: 645.4431096504862 and parameters: {'beta': 0.08174451866089266, 'rel_death_prob': 0.5240268786032249}. Best is trial 12 with value: 101.61156015454989.

[I 2026-10-16 16:11:49,781] Trial 19 finished with value: 1994.634942487898 and parameters: {'beta': 0.08368759637018261, 'rel_death_prob': 2.3878420385345573}. Best is trial 12 with value: 101.61156015454989.

[I 2026-10-16 16:11:50,527] Trial 20 finished with value: 1720.831482879602 and parameters: {'beta': 0.05562186637644186, 'rel_death_prob': 2.833754355363955}. Best is trial 12 with value: 101.61156015454989.

[I 2026-10-16 16:11:50,803] Trial 21 finished with value: 93.66378736066083 and parameters: {'beta': 0.010630614665097168, 'rel_death_prob': 1.7377783788091319}. Best is trial 21 with value: 93.66378736066083.

[I 2026-10-16 16:11:51,056] Trial 22 finished with value: 106.22385308877736 and parameters: {'beta': 0.010454693481184524, 'rel_death_prob': 1.191398942272785}. Best is trial 21 with value: 93.66378736066083.

[I 2026-10-16 16:11:51,398] Trial 23 finished with value: 537.7268286183772 and parameters: {'beta': 0.05840520356086936, 'rel_death_prob': 0.5367106767484588}. Best is trial 21 with value: 93.66378736066083.

[I 2026-10-16 16:11:51,996] Trial 24 finished with value: 54.138206688279965 and parameters: {'beta': 0.014395790675966972, 'rel_death_prob': 2.727698502640594}. Best is trial 24 with value: 54.138206688279965.

[I 2026-10-16 16:11:52,272] Trial 26 finished with value: 140.97182129058044 and parameters: {'beta': 0.006932494501818487, 'rel_death_prob': 0.5894529826749788}. Best is trial 24 with value: 54.138206688279965.

[I 2026-10-16 16:11:52,379] Trial 25 finished with value: 242.31092063774037 and parameters: {'beta': 0.01773462358059827, 'rel_death_prob': 1.827997229785084}. Best is trial 24 with value: 54.138206688279965.

[I 2026-10-16 16:11:52,537] Trial 27 finished with value: 143.1296353865968 and parameters: {'beta': 0.005540469083275237, 'rel_death_prob': 1.3575443392922217}. Best is trial 24 with value: 54.138206688279965.

[I 2026-10-16 16:11:53,456] Trial 28 finished with value: 67.28904827463694 and parameters: {'beta': 0.012539847752352961, 'rel_death_prob': 2.5404478992900295}. Best is trial 24 with value: 54.138206688279965.

[I 2026-10-16 16:11:53,920] Trial 29 finished with value: 1145.2320468978994 and parameters: {'beta': 0.03743013618728536, 'rel_death_prob': 1.829627930391857}. Best is trial 24 with value: 54.138206688279965.

[I 2026-10-16 16:11:54,000] Trial 30 finished with value: 677.4497046675845 and parameters: {'beta': 0.046866306091883095, 'rel_death_prob': 0.8162514062911292}. Best is trial 24 with value: 54.138206688279965.

[I 2026-10-16 16:11:54,178] Trial 31 finished with value: 896.9129990673712 and parameters: {'beta': 0.025802743825255416, 'rel_death_prob': 2.684025793550218}. Best is trial 24 with value: 54.138206688279965.

[I 2026-10-16 16:11:54,731] Trial 32 finished with value: 129.7945996358307 and parameters: {'beta': 0.008022158154520634, 'rel_death_prob': 2.457228427705118}. Best is trial 24 with value: 54.138206688279965.

[I 2026-10-16 16:11:55,357] Trial 35 finished with value: 134.22269840564906 and parameters: {'beta': 0.006100524180838024, 'rel_death_prob': 1.9302709666596103}. Best is trial 24 with value: 54.138206688279965.

[I 2026-10-16 16:11:55,544] Trial 33 finished with value: 1521.451347870498 and parameters: {'beta': 0.04537130466457549, 'rel_death_prob': 2.555006355012783}. Best is trial 24 with value: 54.138206688279965.

[I 2026-10-16 16:11:55,643] Trial 34 finished with value: 1145.2461473553317 and parameters: {'beta': 0.03152046858560473, 'rel_death_prob': 2.9297943073755066}. Best is trial 24 with value: 54.138206688279965.

[I 2026-10-16 16:11:56,438] Trial 36 finished with value: 1958.6497979304527 and parameters: {'beta': 0.14057725229874507, 'rel_death_prob': 1.6223968553942199}. Best is trial 24 with value: 54.138206688279965.

[I 2026-10-16 16:11:56,982] Trial 37 finished with value: 1043.5601989607853 and parameters: {'beta': 0.0463183317259629, 'rel_death_prob': 1.3844050455411625}. Best is trial 24 with value: 54.138206688279965.

[I 2026-10-16 16:11:57,167] Trial 38 finished with value: 548.5734112004263 and parameters: {'beta': 0.026450889564245524, 'rel_death_prob': 0.9784722726808347}. Best is trial 24 with value: 54.138206688279965.

[I 2026-10-16 16:11:57,274] Trial 39 finished with value: 1555.2802327130614 and parameters: {'beta': 0.05410381520190981, 'rel_death_prob': 2.2255040285503616}. Best is trial 24 with value: 54.138206688279965.

[I 2026-10-16 16:11:58,130] Trial 40 finished with value: 1549.147755029533 and parameters: {'beta': 0.09737263625160689, 'rel_death_prob': 1.5071491555535297}. Best is trial 24 with value: 54.138206688279965.

[I 2026-10-16 16:11:58,635] Trial 41 finished with value: 1851.8075232046897 and parameters: {'beta': 0.06532233796181112, 'rel_death_prob': 2.5855845930747807}. Best is trial 24 with value: 54.138206688279965.

[I 2026-10-16 16:11:58,761] Trial 42 finished with value: 1648.7658213794025 and parameters: {'beta': 0.043255786572857315, 'rel_death_prob': 2.827175662039319}. Best is trial 24 with value: 54.138206688279965.

[I 2026-10-16 16:11:58,863] Trial 43 finished with value: 443.70042634453966 and parameters: {'beta': 0.02103837172737795, 'rel_death_prob': 1.6851122269565306}. Best is trial 24 with value: 54.138206688279965.

[I 2026-10-16 16:11:59,665] Trial 44 finished with value: 123.78458498023717 and parameters: {'beta': 0.01682607561018154, 'rel_death_prob': 1.2502249147657176}. Best is trial 24 with value: 54.138206688279965.

[I 2026-10-16 16:12:00,189] Trial 45 finished with value: 147.4802371541502 and parameters: {'beta': 0.016471723828901923, 'rel_death_prob': 2.080017737652131}. Best is trial 24 with value: 54.138206688279965.

[I 2026-10-16 16:12:00,334] Trial 47 finished with value: 78.918261757783 and parameters: {'beta': 0.014278697809554177, 'rel_death_prob': 0.8144283823087446}. Best is trial 24 with value: 54.138206688279965.

[I 2026-10-16 16:12:00,401] Trial 46 finished with value: 838.7136385841808 and parameters: {'beta': 0.0329829492430634, 'rel_death_prob': 1.5691467526701992}. Best is trial 24 with value: 54.138206688279965.

[I 2026-10-16 16:12:01,284] Trial 48 finished with value: 475.9529466625217 and parameters: {'beta': 0.032002845332538464, 'rel_death_prob': 0.5117003116408295}. Best is trial 24 with value: 54.138206688279965.

[I 2026-10-16 16:12:01,788] Trial 51 finished with value: 116.49309410667495 and parameters: {'beta': 0.009929410409925456, 'rel_death_prob': 0.962054398917388}. Best is trial 24 with value: 54.138206688279965.

[I 2026-10-16 16:12:01,797] Trial 49 finished with value: 403.1434027623573 and parameters: {'beta': 0.02012968852110064, 'rel_death_prob': 2.412011634578883}. Best is trial 24 with value: 54.138206688279965.

[I 2026-10-16 16:12:01,948] Trial 50 finished with value: 343.5639738863969 and parameters: {'beta': 0.01866519705818423, 'rel_death_prob': 2.9922131154763956}. Best is trial 24 with value: 54.138206688279965.

[I 2026-10-16 16:12:02,950] Trial 52 finished with value: 1187.4711329217923 and parameters: {'beta': 0.06471657716992726, 'rel_death_prob': 1.2695312689410456}. Best is trial 24 with value: 54.138206688279965.

[I 2026-10-16 16:12:03,293] Trial 53 finished with value: 53.792800994803926 and parameters: {'beta': 0.014174820466868463, 'rel_death_prob': 2.718748728914293}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:03,429] Trial 54 finished with value: 276.47066660745213 and parameters: {'beta': 0.020714129434252368, 'rel_death_prob': 0.5846630687527455}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:03,621] Trial 55 finished with value: 1102.9519918283963 and parameters: {'beta': 0.03365104742498976, 'rel_death_prob': 2.525877825060907}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:04,453] Trial 56 finished with value: 54.643824665808054 and parameters: {'beta': 0.01461148211366993, 'rel_death_prob': 2.5586951945948044}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:04,825] Trial 57 finished with value: 132.3296842385753 and parameters: {'beta': 0.016260213502637302, 'rel_death_prob': 2.7960264732922475}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:05,027] Trial 59 finished with value: 104.88286627881155 and parameters: {'beta': 0.009850498696612184, 'rel_death_prob': 2.729481326327466}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:05,047] Trial 58 finished with value: 1581.284807034685 and parameters: {'beta': 0.04618506383760847, 'rel_death_prob': 2.707181341539466}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:05,727] Trial 60 finished with value: 129.4728427410401 and parameters: {'beta': 0.008361511989071452, 'rel_death_prob': 2.2334981175315693}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:06,475] Trial 61 finished with value: 1650.642847626238 and parameters: {'beta': 0.04193466613562529, 'rel_death_prob': 2.9669248937523958}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:06,689] Trial 62 finished with value: 1492.4047386419147 and parameters: {'beta': 0.04684574510126778, 'rel_death_prob': 2.3847079489455174}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:06,713] Trial 63 finished with value: 969.5721010791847 and parameters: {'beta': 0.031041360617729732, 'rel_death_prob': 2.138755861197031}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:07,349] Trial 64 finished with value: 643.4185282231203 and parameters: {'beta': 0.03670663625134973, 'rel_death_prob': 0.7214141180952288}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:07,942] Trial 65 finished with value: 54.65914642270285 and parameters: {'beta': 0.014641364147826847, 'rel_death_prob': 2.586424166468532}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:08,257] Trial 67 finished with value: 122.78922591819514 and parameters: {'beta': 0.016841446744865374, 'rel_death_prob': 1.0024347308235837}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:08,323] Trial 66 finished with value: 772.3477816760669 and parameters: {'beta': 0.025083343667972227, 'rel_death_prob': 2.0231970031490127}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:08,983] Trial 68 finished with value: 1517.1344983790027 and parameters: {'beta': 0.05924579096748124, 'rel_death_prob': 2.0649487306199616}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:09,401] Trial 69 finished with value: 54.20904205711241 and parameters: {'beta': 0.014059113673456249, 'rel_death_prob': 2.5870360579164866}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:09,705] Trial 71 finished with value: 105.82648665452768 and parameters: {'beta': 0.009838174199236221, 'rel_death_prob': 1.613189009050247}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:09,880] Trial 70 finished with value: 1388.367011591242 and parameters: {'beta': 0.03932934051925689, 'rel_death_prob': 2.627412091775503}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:10,617] Trial 72 finished with value: 1704.2364657814096 and parameters: {'beta': 0.05743075006743683, 'rel_death_prob': 2.703477044714136}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:10,993] Trial 73 finished with value: 304.3931251942976 and parameters: {'beta': 0.018376529944979503, 'rel_death_prob': 2.5432518190774926}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:11,335] Trial 74 finished with value: 1148.8930585779633 and parameters: {'beta': 0.032986855955735896, 'rel_death_prob': 2.7935823096286527}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:11,409] Trial 75 finished with value: 136.3702979970689 and parameters: {'beta': 0.016376182726632535, 'rel_death_prob': 2.6606074612143202}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:12,288] Trial 76 finished with value: 2091.071634764844 and parameters: {'beta': 0.06700528769679938, 'rel_death_prob': 2.8323275717595995}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:12,444] Trial 77 finished with value: 67.59548341253276 and parameters: {'beta': 0.012519039423267441, 'rel_death_prob': 2.355847429945413}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:12,973] Trial 78 finished with value: 856.7452369320956 and parameters: {'beta': 0.024305111422712605, 'rel_death_prob': 2.883499898630114}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:13,052] Trial 79 finished with value: 1573.8104543234 and parameters: {'beta': 0.05545522262146624, 'rel_death_prob': 2.381795538033999}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:13,678] Trial 80 finished with value: 104.95692143713637 and parameters: {'beta': 0.009836277353145552, 'rel_death_prob': 2.5460664558551467}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:14,078] Trial 81 finished with value: 864.282431052094 and parameters: {'beta': 0.026617079010794437, 'rel_death_prob': 2.364444369167474}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:14,526] Trial 83 finished with value: 56.63698538881734 and parameters: {'beta': 0.013786475698010503, 'rel_death_prob': 2.882156774014804}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:14,640] Trial 82 finished with value: 810.4216813962782 and parameters: {'beta': 0.08350578435935582, 'rel_death_prob': 0.7617606228106107}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:15,725] Trial 84 finished with value: 668.5054625394148 and parameters: {'beta': 0.02901766350570941, 'rel_death_prob': 1.2047539335928619}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:16,220] Trial 85 finished with value: 1759.294088910601 and parameters: {'beta': 0.05671936933079073, 'rel_death_prob': 2.984293168839965}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:16,238] Trial 86 finished with value: 131.84138650797175 and parameters: {'beta': 0.007216891806234477, 'rel_death_prob': 2.640692102627852}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:16,739] Trial 87 finished with value: 856.0080161655637 and parameters: {'beta': 0.02505340634418405, 'rel_death_prob': 2.617691591505065}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:17,399] Trial 88 finished with value: 635.3769374250566 and parameters: {'beta': 0.022467927010085306, 'rel_death_prob': 2.730151751737642}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:17,634] Trial 89 finished with value: 98.22345339077143 and parameters: {'beta': 0.010352494188373649, 'rel_death_prob': 2.373131024577989}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:17,727] Trial 90 finished with value: 58.26875249811254 and parameters: {'beta': 0.013424315490844859, 'rel_death_prob': 2.95439465013449}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:18,403] Trial 91 finished with value: 1654.0530488075676 and parameters: {'beta': 0.05320304035722009, 'rel_death_prob': 2.5973399110642283}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:18,979] Trial 92 finished with value: 195.23271306124263 and parameters: {'beta': 0.017550035162294655, 'rel_death_prob': 1.5321195626566348}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:19,284] Trial 93 finished with value: 852.3521117377982 and parameters: {'beta': 0.028941734428934676, 'rel_death_prob': 1.865169932970896}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:19,353] Trial 94 finished with value: 418.2580938846205 and parameters: {'beta': 0.020085905797139075, 'rel_death_prob': 2.9209632857048433}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:19,948] Trial 95 finished with value: 124.46440467202558 and parameters: {'beta': 0.015956032604575246, 'rel_death_prob': 2.4432595201832306}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:20,612] Trial 96 finished with value: 895.5911089399121 and parameters: {'beta': 0.027468272840415447, 'rel_death_prob': 2.5133953561271185}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:20,729] Trial 97 finished with value: 86.14868765821379 and parameters: {'beta': 0.011060455965640374, 'rel_death_prob': 2.9211605138036356}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:20,928] Trial 99 finished with value: 134.8687658213794 and parameters: {'beta': 0.005526180445360026, 'rel_death_prob': 2.852835916886544}. Best is trial 53 with value: 53.792800994803926.

[I 2026-10-16 16:12:20,931] Trial 98 finished with value: 1168.4680463649686 and parameters: {'beta': 0.034752362672698, 'rel_death_prob': 2.424697633756763}. Best is trial 53 with value: 53.792800994803926.

Making results structure...
Processed 100 trials; 0 failed
Best pars: #0. 'beta':           0.014174820466868463
#1. 'rel_death_prob': 2.718748728914293
Removed existing calibration file /tmp/tmp9jvhewug/covasim_calibration.db
Calibration for 100 total trials completed in 41.5 s.

Initial parameter values:
#0. 'beta':           0.015
#1. 'rel_death_prob': 1.0

Best parameter values:
#0. 'beta':           0.014174820466868463
#1. 'rel_death_prob': 2.718748728914293

Mismatch before calibration: 69.5043
Mismatch after calibration:  53.7928
Percent improvement:         22.6%

So it improved the fit (see above), but let’s visualize this as a plot:

# Plot the results
calib.plot_trend()
calib.plot_sims(to_plot=['cum_tests', 'cum_diagnoses', 'cum_deaths'])

Compared to scipy.optimize.minimize(), Optuna took less time and produced a better fit. Specifically, the mismatch obtained with Optuna is almost half the smallest mismatch achieved with scipy.optimize. However, Optuna’s results are still far from perfect – running calibration runs, or more iterations, and calibrating more parameters beyond just these two, would still be required before the model could be considered “calibrated”.

Sometimes you want to calibrate a parameter that isn’t a built-in parameter – for example, part of an intervention. You can do this using the custom_fn keyword argument. In this example, we calibrate the test probability by modifying the test_prob intervention directly.

pars = dict(
    verbose = 0,
    start_day = '2020-02-05',
    pop_size = 1000,
    pop_scale = 4,
    interventions = cv.test_prob(symp_prob=0.01),
)

sim = cv.Sim(pars, datafile='example_data.csv')

calib_pars = dict(
    beta      = [0.013, 0.005, 0.020],
    test_prob = [0.01, 0.00, 0.30]
)

def set_test_prob(sim, calib_pars):
    tp = sim.get_intervention(cv.test_prob)
    tp.symp_prob = calib_pars['test_prob']
    return sim

calib = sim.calibrate(calib_pars=calib_pars, custom_fn=set_test_prob, total_trials=60)
calib.plot_all()
calib.plot_sims(to_plot=['cum_deaths', 'cum_diagnoses'])

[I 2026-10-16 16:12:24,309] A new study created in RDB with name: covasim_calibration

  Automatically adding cumulative column cum_diagnoses from new_diagnoses
  Automatically adding cumulative column cum_tests from new_tests
  Automatically adding cumulative column cum_deaths from new_deaths
Removed existing calibration file /tmp/tmpxq18xs1n/covasim_calibration.db
sqlite:////tmp/tmpxq18xs1n/covasim_calibration.db

[I 2026-10-16 16:12:25,178] Trial 0 finished with value: 145.82646214330205 and parameters: {'beta': 0.005504491005049062, 'test_prob': 0.05166773266305171}. Best is trial 0 with value: 145.82646214330205.

[I 2026-10-16 16:12:25,286] Trial 2 finished with value: 143.9018619933265 and parameters: {'beta': 0.010908996050339667, 'test_prob': 0.2790364347045143}. Best is trial 2 with value: 143.9018619933265.

[I 2026-10-16 16:12:25,401] Trial 3 finished with value: 137.06500214468718 and parameters: {'beta': 0.012409618878393303, 'test_prob': 0.25758191247310475}. Best is trial 3 with value: 137.06500214468718.

[I 2026-10-16 16:12:25,416] Trial 1 finished with value: 104.6631036819623 and parameters: {'beta': 0.01361869555393476, 'test_prob': 0.24096438335085807}. Best is trial 1 with value: 104.6631036819623.

[I 2026-10-16 16:12:25,809] Trial 4 finished with value: 131.85164572557423 and parameters: {'beta': 0.009202286140883443, 'test_prob': 0.14850857252691888}. Best is trial 1 with value: 104.6631036819623.

[I 2026-10-16 16:12:26,038] Trial 5 finished with value: 117.27471868171762 and parameters: {'beta': 0.013669274134687558, 'test_prob': 0.12717526156148992}. Best is trial 1 with value: 104.6631036819623.

[I 2026-10-16 16:12:26,139] Trial 6 finished with value: 140.63206840213942 and parameters: {'beta': 0.01123670852809908, 'test_prob': 0.04689898515521013}. Best is trial 1 with value: 104.6631036819623.

[I 2026-10-16 16:12:26,156] Trial 7 finished with value: 140.91392906503114 and parameters: {'beta': 0.011249744491860563, 'test_prob': 0.0407193758562861}. Best is trial 1 with value: 104.6631036819623.

[I 2026-10-16 16:12:26,560] Trial 8 finished with value: 108.18612142658694 and parameters: {'beta': 0.0184419493659384, 'test_prob': 0.24750110606803546}. Best is trial 1 with value: 104.6631036819623.

[I 2026-10-16 16:12:26,617] Trial 9 finished with value: 145.86548878606916 and parameters: {'beta': 0.006752717934249219, 'test_prob': 0.12460655076275873}. Best is trial 1 with value: 104.6631036819623.

[I 2026-10-16 16:12:26,691] Trial 11 finished with value: 145.89512102222102 and parameters: {'beta': 0.005126124176252769, 'test_prob': 0.1879652314765682}. Best is trial 1 with value: 104.6631036819623.

[I 2026-10-16 16:12:26,894] Trial 10 finished with value: 113.72056710442197 and parameters: {'beta': 0.016638398127028, 'test_prob': 0.11731114368666697}. Best is trial 1 with value: 104.6631036819623.

[I 2026-10-16 16:12:27,289] Trial 12 finished with value: 122.55811142816259 and parameters: {'beta': 0.013956326301889671, 'test_prob': 0.09296397188871741}. Best is trial 1 with value: 104.6631036819623.

[I 2026-10-16 16:12:27,412] Trial 13 finished with value: 108.15248975592324 and parameters: {'beta': 0.018792380134720375, 'test_prob': 0.03998810934013455}. Best is trial 1 with value: 104.6631036819623.

[I 2026-10-16 16:12:27,492] Trial 14 finished with value: 107.76059439491708 and parameters: {'beta': 0.019722946179406002, 'test_prob': 0.2345245286223155}. Best is trial 1 with value: 104.6631036819623.

[I 2026-10-16 16:12:27,687] Trial 15 finished with value: 74.01713555871177 and parameters: {'beta': 0.015937190918692733, 'test_prob': 0.2265235963069397}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:28,053] Trial 16 finished with value: 116.07651436325628 and parameters: {'beta': 0.016946990896098288, 'test_prob': 0.29965631631342116}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:28,180] Trial 17 finished with value: 112.36346343692901 and parameters: {'beta': 0.018360860776404204, 'test_prob': 0.017971262911604908}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:28,284] Trial 18 finished with value: 116.32475789943281 and parameters: {'beta': 0.01883394426636677, 'test_prob': 0.18736388418115255}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:28,481] Trial 19 finished with value: 85.40567840901804 and parameters: {'beta': 0.016399459647427558, 'test_prob': 0.20337812155631685}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:28,787] Trial 20 finished with value: 139.99404812335038 and parameters: {'beta': 0.011203148507054613, 'test_prob': 0.20717520121421212}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:28,941] Trial 21 finished with value: 100.91797799245904 and parameters: {'beta': 0.01611281473364096, 'test_prob': 0.25046388901534433}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:29,077] Trial 22 finished with value: 84.64956037895999 and parameters: {'beta': 0.014421690695594673, 'test_prob': 0.20730874478372344}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:29,246] Trial 23 finished with value: 85.44740281238334 and parameters: {'beta': 0.016243763929219186, 'test_prob': 0.20502012580083223}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:29,366] Trial 24 finished with value: 133.46683028544848 and parameters: {'beta': 0.009154127709012324, 'test_prob': 0.22589001628128894}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:29,709] Trial 25 finished with value: 116.80407926807052 and parameters: {'beta': 0.016747795265581556, 'test_prob': 0.24621345982017034}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:29,842] Trial 26 finished with value: 88.2626542867416 and parameters: {'beta': 0.015378897952740733, 'test_prob': 0.16167957688642653}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:30,000] Trial 27 finished with value: 116.0124148984927 and parameters: {'beta': 0.013533002201392603, 'test_prob': 0.19983378709286478}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:30,145] Trial 28 finished with value: 95.77179594693594 and parameters: {'beta': 0.018215915457251833, 'test_prob': 0.12270872952489263}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:30,485] Trial 29 finished with value: 115.04880891491906 and parameters: {'beta': 0.017135462433068883, 'test_prob': 0.17697232693419568}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:30,605] Trial 30 finished with value: 113.78082475225725 and parameters: {'beta': 0.018616879888035755, 'test_prob': 0.2939375885096959}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:30,768] Trial 31 finished with value: 89.94239991755107 and parameters: {'beta': 0.018666772362519388, 'test_prob': 0.2140946647274676}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:30,923] Trial 32 finished with value: 111.11688763352805 and parameters: {'beta': 0.019780220603411766, 'test_prob': 0.2723624222282836}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:31,256] Trial 33 finished with value: 98.07669277984058 and parameters: {'beta': 0.014992808439941501, 'test_prob': 0.2767551574849445}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:31,377] Trial 34 finished with value: 85.76509537437241 and parameters: {'beta': 0.01552551037413872, 'test_prob': 0.20708228273470689}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:31,528] Trial 35 finished with value: 106.09607912483722 and parameters: {'beta': 0.014555571339048438, 'test_prob': 0.24199221900453896}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:31,704] Trial 36 finished with value: 113.84830185165971 and parameters: {'beta': 0.017472727810825317, 'test_prob': 0.1499360668869229}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:32,034] Trial 37 finished with value: 116.59360914092838 and parameters: {'beta': 0.017493661665113642, 'test_prob': 0.22330274602442135}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:32,161] Trial 38 finished with value: 118.53474391294625 and parameters: {'beta': 0.01763110924713132, 'test_prob': 0.27477140978192693}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:32,289] Trial 39 finished with value: 134.9227957268452 and parameters: {'beta': 0.012257139951464307, 'test_prob': 0.15321489632669524}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:32,401] Trial 40 finished with value: 141.85265946639265 and parameters: {'beta': 0.009830429639071952, 'test_prob': 0.18911997602744562}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:32,824] Trial 41 finished with value: 110.04155636064422 and parameters: {'beta': 0.019628664752808528, 'test_prob': 0.13920966797812234}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:32,952] Trial 42 finished with value: 85.23394203340277 and parameters: {'beta': 0.016353886761835128, 'test_prob': 0.1918710755869819}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:33,075] Trial 43 finished with value: 117.01391780223064 and parameters: {'beta': 0.013517064853643979, 'test_prob': 0.17660242756237318}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:33,177] Trial 44 finished with value: 81.85286087604237 and parameters: {'beta': 0.014858451713165683, 'test_prob': 0.1980609066110709}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:33,597] Trial 45 finished with value: 85.60304676271429 and parameters: {'beta': 0.01617681164995593, 'test_prob': 0.21000351665960185}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:33,739] Trial 46 finished with value: 90.50499460771042 and parameters: {'beta': 0.016635291071669077, 'test_prob': 0.2294787072091887}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:33,857] Trial 47 finished with value: 85.49084063258378 and parameters: {'beta': 0.016490921596807158, 'test_prob': 0.18743204539518346}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:33,948] Trial 48 finished with value: 98.6608710106324 and parameters: {'beta': 0.015048896072679153, 'test_prob': 0.23045965926973544}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:34,364] Trial 49 finished with value: 98.39405823860572 and parameters: {'beta': 0.015705432270013545, 'test_prob': 0.0916506766568425}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:34,456] Trial 50 finished with value: 141.8260448465674 and parameters: {'beta': 0.01159094691479332, 'test_prob': 0.09729995198144448}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:34,634] Trial 51 finished with value: 82.533575905118 and parameters: {'beta': 0.014609115391600458, 'test_prob': 0.20863642413747738}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:34,689] Trial 52 finished with value: 116.69105764562849 and parameters: {'beta': 0.01313364120202744, 'test_prob': 0.2732419569729677}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:35,142] Trial 53 finished with value: 83.13091568266323 and parameters: {'beta': 0.014638449321000422, 'test_prob': 0.18304714117544235}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:35,254] Trial 54 finished with value: 103.68178899165733 and parameters: {'beta': 0.014167953791125946, 'test_prob': 0.22347524907594818}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:35,442] Trial 55 finished with value: 118.44885351926158 and parameters: {'beta': 0.019605392163649482, 'test_prob': 0.20582936633301005}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:35,464] Trial 56 finished with value: 132.99044697957714 and parameters: {'beta': 0.012532883611748556, 'test_prob': 0.21789158328945282}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:35,893] Trial 57 finished with value: 117.29802421602145 and parameters: {'beta': 0.01310327866111346, 'test_prob': 0.24990485475791693}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:35,929] Trial 59 finished with value: 134.98671768192358 and parameters: {'beta': 0.012461814936340448, 'test_prob': 0.18732687929787428}. Best is trial 15 with value: 74.01713555871177.

[I 2026-10-16 16:12:35,980] Trial 58 finished with value: 91.47257268472296 and parameters: {'beta': 0.014364793720207023, 'test_prob': 0.14356013782970978}. Best is trial 15 with value: 74.01713555871177.

Making results structure...
Processed 60 trials; 0 failed
Best pars: #0. 'beta':      0.015937190918692733
#1. 'test_prob': 0.2265235963069397
Removed existing calibration file /tmp/tmpxq18xs1n/covasim_calibration.db
Calibration for 60 total trials completed in 11.9 s.

Initial parameter values:
#0. 'beta':      0.013
#1. 'test_prob': 0.01

Best parameter values:
#0. 'beta':      0.015937190918692733
#1. 'test_prob': 0.2265235963069397

Mismatch before calibration: 143.586
Mismatch after calibration:  74.0171
Percent improvement:         48.5%