We saw in Tutorial 4 how to load and plot data. But the next step is to actually calibrate the model to the data, i.e. find the model parameters that are the most likely explanation for the observed data. This tutorial gives an introduction to the Fit object and some recipes for optimization approaches.
Click here to open an interactive version of this notebook.
The Fit object is responsible for quantifying how well a given model run matches the data. Let’s consider a simple example, building on Tutorial 4:
import covasim as cv
cv.options(jupyter=True, verbose=0)
pars = dict(
pop_size = 20_000,
start_day = '2020-02-01',
end_day = '2020-04-11',
)
sim = cv.Sim(pars=pars, datafile='example_data.csv', interventions=cv.test_num(daily_tests='data'))
sim.run()
sim.plot(to_plot=['cum_tests', 'cum_diagnoses', 'cum_deaths'])Covasim 3.1.8 (2026-05-29) — © 2020-2026 by IDM
We can see that tests match extremely well (they’re input data!), diagnoses match reasonably well, and deaths match poorly. Can the Fit object capture our intuition about this?
fit = sim.compute_fit()
fit.summarize()Mismatch values for:
#0. 'cum_deaths': 55.65217391304348
#1. 'cum_tests': 0.0
#2. 'cum_diagnoses': 22.428498467824312
Total mismatch value:
78.08067238086778So the results seem to match our intuition. (Note that by default the Fit object uses normalized absolute difference, but other estimates, such as mean squared error, are also possible.)
What if we improve the fit? Does the mismatch reduce?
sim['rel_death_prob'] = 2 # Increase the death rate since deaths were too low
sim.initialize(reset=True) # Reinitialize the sim
# Rerun and compute fit
sim.run()
fit = sim.compute_fit()
# Output
sim.plot(to_plot=['cum_tests', 'cum_diagnoses', 'cum_deaths'])
fit.plot()
fit.summarize()
Mismatch values for:
#0. 'cum_deaths': 43.913043478260875
#1. 'cum_tests': 0.0
#2. 'cum_diagnoses': 23.240551583248212
Total mismatch value:
67.1535950615091As expected, the fit is slightly improved. By now, you may be wondering how the mistmatch is actually calculated in Covasim. It follows more or less this expression:
\(\text{mismatch} = \sum_i\left(\sum_t\dfrac{|d_i(t) - m_i(t)|}{\text{max}(d_i(t))}\right)\)
where \(i\) is over different data quantities (eg, cum_tests, cum_deaths, cum_diagnoses in this example); \(t\) is time; \(d_i(t)\) is the time series of data for quantity \(i\) (the lines with square markers in the figures); and, \(m_i(t)\) is the time series of the model for quantity \(i\) (solid lines).
Each data quantity \(i\) has very different scales. Cumulative deaths could be in the few hundreds, while cumulative diagnoses could be in the tens of thousands (or more). For that reason we use the term \(\text{max}(d_i(t))\), so we can reasonably add individual mismatches of different quantities like cumulative deaths and diagnoses, to produce an overall mismatch, which is a ‘normalized’ absolute error of sorts.
Calibration is a complex and dark art and cannot be covered fully here; many books have been written about it and it continues to be an area of active research. A good review article about calibrating agent-based models like Covasim is available here. Calibration is usually expressed as an optimization problem: specifically, find a vector of parameters θ that minimizes the mismatch between the data D and the model M(θ).
In practice, most calibration is done simply by hand, as in the example above. Once deaths are “calibrated”, the user might modify testing assumptions so that the diagnoses match. Since we are only fitting to deaths and diagnoses, the model is then “calibrated”.
However, automated approaches to calibration are possible as well. The simplest is probably the built-in SciPy optimization functions, e.g. scipy.optimize. A wrinkle here is that normal gradient descent methods will not work with Covasim or other agent-based models, due to the stochastic variability between model runs that makes the landscape very “bumpy”. One way of getting around this is to use many different runs and take the average, e.g.:
import covasim as cv
import numpy as np
from scipy import optimize
def objective(x, n_runs=10):
print(f'Running sim for beta={x[0]}, rel_death_prob={x[1]}')
pars = dict(
pop_size = 20_000,
start_day = '2020-02-01',
end_day = '2020-04-11',
beta = x[0],
rel_death_prob = x[1],
verbose = 0,
)
sim = cv.Sim(pars=pars, datafile='example_data.csv', interventions=cv.test_num(daily_tests='data'))
msim = cv.MultiSim(sim)
msim.run(n_runs=n_runs)
mismatches = []
for sim in msim.sims:
fit = sim.compute_fit()
mismatches.append(fit.mismatch)
mismatch = np.mean(mismatches)
return mismatch
guess = [0.015, 1] # Initial guess of parameters -- beta and relative death probability
pars = optimize.minimize(objective, x0=guess, method='nelder-mead') # Run the optimizationThis should converge after roughly 3-10 minutes, although you will likely find that the improvement is minimal.
What’s happening here? Trying to overcome the limitations of an algorithm that expects deterministic results simply by running more sims is fairly futile – if you run N sims and average them together, you’ve only reduced noise by √N, i.e. you have to average together 100 sims to reduce noise by a factor of 10, and even that might not be enough. Clearly, we need a more powerful approach.
One such package we have found works reasonably well is called Optuna. It is built into Covasim as sim.calibrate() (it’s not installed by default, so please install it first with pip install optuna). Do not expect this to be a magic bullet solution: you will likely still need to try out multiple different parameter sets for calibration, manually update the values of uncalibrated parameters, check if the data actually make sense, etc. Even once all these things are in place, it still needs to be run for enough iterations, which might be a few hundred iterations for 3-4 calibrated (free) parameters or tens of thousands of iterations for 10 or more free parameters. The example below should get you started, but best to expect that it will not work for your particular use case without significant modification!
'''
Example for running built-in calibration with Optuna
'''
import sciris as sc
import covasim as cv
# Create default simulation
pars = sc.objdict(
pop_size = 20_000,
start_day = '2020-02-01',
end_day = '2020-04-11',
beta = 0.015,
rel_death_prob = 1.0,
interventions = cv.test_num(daily_tests='data'),
verbose = 0,
)
sim = cv.Sim(pars=pars, datafile='example_data.csv')
# Parameters to calibrate -- format is best, low, high
calib_pars = dict(
beta = [pars.beta, 0.005, 0.20],
rel_death_prob = [pars.rel_death_prob, 0.5, 3.0],
)
if __name__ == '__main__':
# Run the calibration
calib = sim.calibrate(calib_pars=calib_pars, total_trials=100)[I 2026-05-29 13:25:38,692] A new study created in RDB with name: covasim_calibration
Could not delete study, skipping...
'Record does not exist.'
Removed existing calibration covasim_calibration.db[I 2026-05-29 13:25:39,759] Trial 0 finished with value: 1151.3410978371899 and parameters: {'beta': 0.03209207289428316, 'rel_death_prob': 2.387762339549171}. Best is trial 0 with value: 1151.3410978371899. [I 2026-05-29 13:25:39,792] Trial 1 finished with value: 2220.028422969312 and parameters: {'beta': 0.10465024339161452, 'rel_death_prob': 2.9319581838780686}. Best is trial 0 with value: 1151.3410978371899. [I 2026-05-29 13:25:39,828] Trial 3 finished with value: 1832.6920770973043 and parameters: {'beta': 0.0619906900924722, 'rel_death_prob': 2.8328090609561642}. Best is trial 0 with value: 1151.3410978371899. [I 2026-05-29 13:25:39,847] Trial 2 finished with value: 1599.0967935337746 and parameters: {'beta': 0.12591747822325416, 'rel_death_prob': 1.6624925195126468}. Best is trial 0 with value: 1151.3410978371899. [I 2026-05-29 13:25:40,575] Trial 5 finished with value: 1600.4862992405738 and parameters: {'beta': 0.04478004596996837, 'rel_death_prob': 2.8875864223694716}. Best is trial 0 with value: 1151.3410978371899. [I 2026-05-29 13:25:40,659] Trial 4 finished with value: 2626.2052005151663 and parameters: {'beta': 0.16931298493579286, 'rel_death_prob': 2.3594553366511026}. Best is trial 0 with value: 1151.3410978371899. [I 2026-05-29 13:25:40,666] Trial 7 finished with value: 1368.4184171958964 and parameters: {'beta': 0.08076377114743104, 'rel_death_prob': 1.3838989561050687}. Best is trial 0 with value: 1151.3410978371899. [I 2026-05-29 13:25:40,714] Trial 6 finished with value: 1151.9857218990096 and parameters: {'beta': 0.19016354111234704, 'rel_death_prob': 0.946725890947004}. Best is trial 0 with value: 1151.3410978371899. [I 2026-05-29 13:25:41,410] Trial 11 finished with value: 225.84780388151177 and parameters: {'beta': 0.019927974620019148, 'rel_death_prob': 0.7446367149157386}. Best is trial 11 with value: 225.84780388151177. [I 2026-05-29 13:25:41,434] Trial 8 finished with value: 2582.1016343207357 and parameters: {'beta': 0.1536189550770978, 'rel_death_prob': 2.863002239703054}. Best is trial 11 with value: 225.84780388151177. [I 2026-05-29 13:25:41,451] Trial 10 finished with value: 1025.7765244037837 and parameters: {'beta': 0.037460413572388045, 'rel_death_prob': 1.8084577955851862}. Best is trial 11 with value: 225.84780388151177. [I 2026-05-29 13:25:41,484] Trial 9 finished with value: 1215.4741972731713 and parameters: {'beta': 0.067459500486679, 'rel_death_prob': 1.5768506510423312}. Best is trial 11 with value: 225.84780388151177. [I 2026-05-29 13:25:41,963] Trial 13 finished with value: 144.1627437047564 and parameters: {'beta': 0.006114433335209302, 'rel_death_prob': 0.5773012925313374}. Best is trial 13 with value: 144.1627437047564. [I 2026-05-29 13:25:41,972] Trial 12 finished with value: 110.37771461562374 and parameters: {'beta': 0.012554646560207255, 'rel_death_prob': 0.6156456948287303}. Best is trial 12 with value: 110.37771461562374. [I 2026-05-29 13:25:41,995] Trial 15 finished with value: 144.1627437047564 and parameters: {'beta': 0.00608495503100036, 'rel_death_prob': 0.5358864808663785}. Best is trial 12 with value: 110.37771461562374. [I 2026-05-29 13:25:42,184] Trial 14 finished with value: 301.78509570546703 and parameters: {'beta': 0.021660745102746595, 'rel_death_prob': 0.5073000382982942}. Best is trial 12 with value: 110.37771461562374. [I 2026-05-29 13:25:42,470] Trial 17 finished with value: 145.42301372296487 and parameters: {'beta': 0.00589071448064904, 'rel_death_prob': 1.0654093512805618}. Best is trial 12 with value: 110.37771461562374. [I 2026-05-29 13:25:42,488] Trial 16 finished with value: 133.5264466847271 and parameters: {'beta': 0.007370637654597948, 'rel_death_prob': 0.6219728797951443}. Best is trial 12 with value: 110.37771461562374. [I 2026-05-29 13:25:42,527] Trial 18 finished with value: 128.68166274370475 and parameters: {'beta': 0.011887530664799282, 'rel_death_prob': 1.1748361533738227}. Best is trial 12 with value: 110.37771461562374. [I 2026-05-29 13:25:43,007] Trial 19 finished with value: 1411.2116400941509 and parameters: {'beta': 0.09629279410361108, 'rel_death_prob': 1.1424593053274026}. Best is trial 12 with value: 110.37771461562374. [I 2026-05-29 13:25:43,301] Trial 20 finished with value: 1321.5717013811786 and parameters: {'beta': 0.09480390238610642, 'rel_death_prob': 1.2072184458941546}. Best is trial 12 with value: 110.37771461562374. [I 2026-05-29 13:25:43,313] Trial 21 finished with value: 1211.6086734467292 and parameters: {'beta': 0.08982477706168261, 'rel_death_prob': 1.184892857217235}. Best is trial 12 with value: 110.37771461562374. [I 2026-05-29 13:25:43,351] Trial 22 finished with value: 1292.9572101079184 and parameters: {'beta': 0.09481228033035495, 'rel_death_prob': 1.1934747408119444}. Best is trial 12 with value: 110.37771461562374. [I 2026-05-29 13:25:43,792] Trial 23 finished with value: 1057.4863436514634 and parameters: {'beta': 0.05269317395051476, 'rel_death_prob': 1.2962943017248199}. Best is trial 12 with value: 110.37771461562374. [I 2026-05-29 13:25:44,094] Trial 25 finished with value: 591.5744770617755 and parameters: {'beta': 0.052540786733767816, 'rel_death_prob': 0.8287782412405171}. Best is trial 12 with value: 110.37771461562374. [I 2026-05-29 13:25:44,094] Trial 24 finished with value: 884.9844561886574 and parameters: {'beta': 0.05152131725051655, 'rel_death_prob': 0.850920494432317}. Best is trial 12 with value: 110.37771461562374. [I 2026-05-29 13:25:44,134] Trial 26 finished with value: 764.8061464671138 and parameters: {'beta': 0.0540247803506136, 'rel_death_prob': 0.8748965192653938}. Best is trial 12 with value: 110.37771461562374. [I 2026-05-29 13:25:44,512] Trial 27 finished with value: 418.6240396145135 and parameters: {'beta': 0.024386205448893885, 'rel_death_prob': 0.8369820590922862}. Best is trial 12 with value: 110.37771461562374. [I 2026-05-29 13:25:44,818] Trial 28 finished with value: 413.140182972865 and parameters: {'beta': 0.023648560152337534, 'rel_death_prob': 0.7310868585018244}. Best is trial 12 with value: 110.37771461562374. [I 2026-05-29 13:25:44,832] Trial 29 finished with value: 446.5905760092375 and parameters: {'beta': 0.023726822575391115, 'rel_death_prob': 0.7008608657072444}. Best is trial 12 with value: 110.37771461562374. [I 2026-05-29 13:25:44,841] Trial 30 finished with value: 541.3190034196384 and parameters: {'beta': 0.0231118888222256, 'rel_death_prob': 1.9329699276211019}. Best is trial 12 with value: 110.37771461562374. [I 2026-05-29 13:25:45,272] Trial 31 finished with value: 738.0603543988987 and parameters: {'beta': 0.02861840506465791, 'rel_death_prob': 1.4235453751236795}. Best is trial 12 with value: 110.37771461562374. [I 2026-05-29 13:25:45,351] Trial 33 finished with value: 138.1714926499978 and parameters: {'beta': 0.005738466109795195, 'rel_death_prob': 0.5658986468629223}. Best is trial 12 with value: 110.37771461562374. [I 2026-05-29 13:25:45,373] Trial 34 finished with value: 140.81815961273702 and parameters: {'beta': 0.0070597033439661495, 'rel_death_prob': 0.6154898025370633}. Best is trial 12 with value: 110.37771461562374. [I 2026-05-29 13:25:45,604] Trial 32 finished with value: 1209.2535639738865 and parameters: {'beta': 0.03745891754915898, 'rel_death_prob': 2.1111578767661756}. Best is trial 12 with value: 110.37771461562374. [I 2026-05-29 13:25:45,766] Trial 35 finished with value: 143.5205622418617 and parameters: {'beta': 0.0064679421996081685, 'rel_death_prob': 0.6043132297826945}. Best is trial 12 with value: 110.37771461562374. [I 2026-05-29 13:25:46,125] Trial 36 finished with value: 739.8792023804237 and parameters: {'beta': 0.036760700688077234, 'rel_death_prob': 1.0543736810228452}. Best is trial 12 with value: 110.37771461562374. [I 2026-05-29 13:25:46,210] Trial 37 finished with value: 874.5327974419328 and parameters: {'beta': 0.11821180116736783, 'rel_death_prob': 0.6705882725116388}. Best is trial 12 with value: 110.37771461562374. [I 2026-05-29 13:25:46,504] Trial 38 finished with value: 1310.9976906337433 and parameters: {'beta': 0.1213307067710734, 'rel_death_prob': 1.0668575610989945}. Best is trial 12 with value: 110.37771461562374. [I 2026-05-29 13:25:47,214] Trial 41 finished with value: 1882.8646134032065 and parameters: {'beta': 0.13484933432880455, 'rel_death_prob': 1.5518281921240855}. Best is trial 12 with value: 110.37771461562374. [I 2026-05-29 13:25:47,232] Trial 39 finished with value: 1354.8170271350536 and parameters: {'beta': 0.14187243887634388, 'rel_death_prob': 1.0214238347742997}. Best is trial 12 with value: 110.37771461562374. [I 2026-05-29 13:25:47,590] Trial 40 finished with value: 1074.1303237553848 and parameters: {'beta': 0.11221191454067747, 'rel_death_prob': 0.9577142182109176}. Best is trial 12 with value: 110.37771461562374. [I 2026-05-29 13:25:47,792] Trial 43 finished with value: 50.78773815339521 and parameters: {'beta': 0.015663771761687623, 'rel_death_prob': 2.5360450379756676}. Best is trial 43 with value: 50.78773815339521. [I 2026-05-29 13:25:47,816] Trial 44 finished with value: 65.72611804414441 and parameters: {'beta': 0.014698294392563128, 'rel_death_prob': 2.561682176065935}. Best is trial 43 with value: 50.78773815339521. [I 2026-05-29 13:25:47,885] Trial 42 finished with value: 905.8589066038992 and parameters: {'beta': 0.06595087274127606, 'rel_death_prob': 0.9591089607113388}. Best is trial 43 with value: 50.78773815339521. [I 2026-05-29 13:25:48,178] Trial 45 finished with value: 108.88673002620243 and parameters: {'beta': 0.015030383104071504, 'rel_death_prob': 0.5044609375645066}. Best is trial 43 with value: 50.78773815339521. [I 2026-05-29 13:25:48,516] Trial 48 finished with value: 87.51598792023805 and parameters: {'beta': 0.014976200995626264, 'rel_death_prob': 2.583149341106556}. Best is trial 43 with value: 50.78773815339521. [I 2026-05-29 13:25:48,667] Trial 47 finished with value: 1889.380357063552 and parameters: {'beta': 0.07072088508587582, 'rel_death_prob': 2.6074277625719815}. Best is trial 43 with value: 50.78773815339521. [I 2026-05-29 13:25:48,867] Trial 46 finished with value: 51.22252076209086 and parameters: {'beta': 0.015659777237578637, 'rel_death_prob': 2.6389740309524994}. Best is trial 43 with value: 50.78773815339521. [I 2026-05-29 13:25:49,030] Trial 49 finished with value: 1483.4800373051473 and parameters: {'beta': 0.04013831555415317, 'rel_death_prob': 2.620761296345172}. Best is trial 43 with value: 50.78773815339521. [I 2026-05-29 13:25:49,246] Trial 51 finished with value: 61.539281431807076 and parameters: {'beta': 0.015399872349426368, 'rel_death_prob': 2.7281184228361943}. Best is trial 43 with value: 50.78773815339521. [I 2026-05-29 13:25:49,317] Trial 50 finished with value: 1660.4881867033796 and parameters: {'beta': 0.04404666592442259, 'rel_death_prob': 2.6293433000981135}. Best is trial 43 with value: 50.78773815339521. [I 2026-05-29 13:25:49,650] Trial 52 finished with value: 1265.2600257583158 and parameters: {'beta': 0.03315637640750525, 'rel_death_prob': 2.722902459739992}. Best is trial 43 with value: 50.78773815339521. [I 2026-05-29 13:25:49,870] Trial 54 finished with value: 178.33015055291557 and parameters: {'beta': 0.016294044065832767, 'rel_death_prob': 2.692016834154217}. Best is trial 43 with value: 50.78773815339521. [I 2026-05-29 13:25:49,930] Trial 55 finished with value: 114.11366967180352 and parameters: {'beta': 0.01631934304877503, 'rel_death_prob': 2.450624817299867}. Best is trial 43 with value: 50.78773815339521. [I 2026-05-29 13:25:49,936] Trial 53 finished with value: 2381.5292889816583 and parameters: {'beta': 0.19404523932060247, 'rel_death_prob': 2.3564424219750624}. Best is trial 43 with value: 50.78773815339521. [I 2026-05-29 13:25:50,276] Trial 56 finished with value: 50.624750188746276 and parameters: {'beta': 0.015817495584203735, 'rel_death_prob': 2.4698850891142277}. Best is trial 56 with value: 50.624750188746276. [I 2026-05-29 13:25:50,439] Trial 57 finished with value: 76.46200648398988 and parameters: {'beta': 0.0144148788771224, 'rel_death_prob': 2.44354747177611}. Best is trial 56 with value: 50.624750188746276. [I 2026-05-29 13:25:50,622] Trial 58 finished with value: 191.36907669760626 and parameters: {'beta': 0.01766569011398751, 'rel_death_prob': 2.9839794596227476}. Best is trial 56 with value: 50.624750188746276. [I 2026-05-29 13:25:50,741] Trial 59 finished with value: 1120.208176044766 and parameters: {'beta': 0.03175477043681932, 'rel_death_prob': 2.4388095543103923}. Best is trial 56 with value: 50.624750188746276. [I 2026-05-29 13:25:51,044] Trial 60 finished with value: 1099.3840209619398 and parameters: {'beta': 0.029660280581477745, 'rel_death_prob': 2.987802110012928}. Best is trial 56 with value: 50.624750188746276. [I 2026-05-29 13:25:51,197] Trial 61 finished with value: 1075.0834924723542 and parameters: {'beta': 0.028229761161991225, 'rel_death_prob': 2.935887849473884}. Best is trial 56 with value: 50.624750188746276. [I 2026-05-29 13:25:51,396] Trial 62 finished with value: 1113.2669760625306 and parameters: {'beta': 0.03049754630032437, 'rel_death_prob': 2.264210631504208}. Best is trial 56 with value: 50.624750188746276. [I 2026-05-29 13:25:51,614] Trial 63 finished with value: 2642.7498112537196 and parameters: {'beta': 0.1656640909921746, 'rel_death_prob': 2.8080303979531647}. Best is trial 56 with value: 50.624750188746276. [I 2026-05-29 13:25:51,923] Trial 64 finished with value: 2357.8828218679223 and parameters: {'beta': 0.17910845936082073, 'rel_death_prob': 2.2938733166623897}. Best is trial 56 with value: 50.624750188746276. [I 2026-05-29 13:25:51,968] Trial 66 finished with value: 46.08851090287338 and parameters: {'beta': 0.013234829425226262, 'rel_death_prob': 2.8042253055941613}. Best is trial 66 with value: 46.08851090287338. [I 2026-05-29 13:25:52,090] Trial 65 finished with value: 2512.590487187458 and parameters: {'beta': 0.1780005634299735, 'rel_death_prob': 2.22555093466894}. Best is trial 66 with value: 46.08851090287338. [I 2026-05-29 13:25:52,156] Trial 67 finished with value: 90.377492561176 and parameters: {'beta': 0.014106581350032248, 'rel_death_prob': 2.520431778487569}. Best is trial 66 with value: 46.08851090287338. [I 2026-05-29 13:25:52,465] Trial 68 finished with value: 101.33166052316028 and parameters: {'beta': 0.012113742096009076, 'rel_death_prob': 2.5420287246206428}. Best is trial 66 with value: 46.08851090287338. [I 2026-05-29 13:25:52,496] Trial 69 finished with value: 104.15319536350313 and parameters: {'beta': 0.01198712444811926, 'rel_death_prob': 2.494629412967191}. Best is trial 66 with value: 46.08851090287338. [I 2026-05-29 13:25:52,845] Trial 71 finished with value: 343.278967002709 and parameters: {'beta': 0.021403939850474153, 'rel_death_prob': 2.7965160806661564}. Best is trial 66 with value: 46.08851090287338. [I 2026-05-29 13:25:52,883] Trial 70 finished with value: 1539.5676599902295 and parameters: {'beta': 0.04528376932633679, 'rel_death_prob': 2.529153869093958}. Best is trial 66 with value: 46.08851090287338. [I 2026-05-29 13:25:53,248] Trial 72 finished with value: 1523.5568681440686 and parameters: {'beta': 0.04418223581169416, 'rel_death_prob': 2.804018979709172}. Best is trial 66 with value: 46.08851090287338. [I 2026-05-29 13:25:53,290] Trial 73 finished with value: 1544.322622907137 and parameters: {'beta': 0.04554653801538596, 'rel_death_prob': 2.799542060916669}. Best is trial 66 with value: 46.08851090287338. [I 2026-05-29 13:25:53,568] Trial 75 finished with value: 366.24494826131365 and parameters: {'beta': 0.021206250656355022, 'rel_death_prob': 2.7498162875674685}. Best is trial 66 with value: 46.08851090287338. [I 2026-05-29 13:25:53,627] Trial 74 finished with value: 1449.3893502686856 and parameters: {'beta': 0.04437417230914571, 'rel_death_prob': 2.7316028040017653}. Best is trial 66 with value: 46.08851090287338. [I 2026-05-29 13:25:53,948] Trial 76 finished with value: 438.9777723497802 and parameters: {'beta': 0.02018920824021038, 'rel_death_prob': 2.713186562084037}. Best is trial 66 with value: 46.08851090287338. [I 2026-05-29 13:25:53,988] Trial 77 finished with value: 420.2428165386152 and parameters: {'beta': 0.020731779789863458, 'rel_death_prob': 2.88338044470933}. Best is trial 66 with value: 46.08851090287338. [I 2026-05-29 13:25:54,074] Trial 78 finished with value: 131.92265843584846 and parameters: {'beta': 0.010385737748931438, 'rel_death_prob': 2.122316055311007}. Best is trial 66 with value: 46.08851090287338. [I 2026-05-29 13:25:54,372] Trial 79 finished with value: 859.3403872629569 and parameters: {'beta': 0.025938347891191014, 'rel_death_prob': 2.3709787205753843}. Best is trial 66 with value: 46.08851090287338. [I 2026-05-29 13:25:54,466] Trial 80 finished with value: 116.31955855575787 and parameters: {'beta': 0.010492460281660444, 'rel_death_prob': 2.144736274897943}. Best is trial 66 with value: 46.08851090287338. [I 2026-05-29 13:25:54,743] Trial 81 finished with value: 725.2433716747346 and parameters: {'beta': 0.026224597219054357, 'rel_death_prob': 2.1151640295847756}. Best is trial 66 with value: 46.08851090287338. [I 2026-05-29 13:25:54,818] Trial 82 finished with value: 866.9084469511924 and parameters: {'beta': 0.02585288877263619, 'rel_death_prob': 2.3784181766089882}. Best is trial 66 with value: 46.08851090287338. [I 2026-05-29 13:25:55,140] Trial 83 finished with value: 1280.389039392459 and parameters: {'beta': 0.036605869207684695, 'rel_death_prob': 2.5750449128725403}. Best is trial 66 with value: 46.08851090287338. [I 2026-05-29 13:25:55,237] Trial 84 finished with value: 1279.0996802415953 and parameters: {'beta': 0.035640937681644384, 'rel_death_prob': 2.4428034917177626}. Best is trial 66 with value: 46.08851090287338. [I 2026-05-29 13:25:55,354] Trial 86 finished with value: 100.4098014833237 and parameters: {'beta': 0.014889344438645064, 'rel_death_prob': 2.5806633648687876}. Best is trial 66 with value: 46.08851090287338. [I 2026-05-29 13:25:55,523] Trial 85 finished with value: 1286.0855131678288 and parameters: {'beta': 0.0347858803240704, 'rel_death_prob': 2.575671537416596}. Best is trial 66 with value: 46.08851090287338. [I 2026-05-29 13:25:55,684] Trial 87 finished with value: 92.63989430208287 and parameters: {'beta': 0.014836785959824125, 'rel_death_prob': 2.6539199399173006}. Best is trial 66 with value: 46.08851090287338. [I 2026-05-29 13:25:55,722] Trial 88 finished with value: 146.65386152684638 and parameters: {'beta': 0.005483592195535151, 'rel_death_prob': 2.6653518792563453}. Best is trial 66 with value: 46.08851090287338. [I 2026-05-29 13:25:55,831] Trial 89 finished with value: 146.65386152684638 and parameters: {'beta': 0.005419582005089817, 'rel_death_prob': 2.634334587772729}. Best is trial 66 with value: 46.08851090287338. [I 2026-05-29 13:25:56,015] Trial 90 finished with value: 146.65386152684638 and parameters: {'beta': 0.005450422438572159, 'rel_death_prob': 2.653785002653335}. Best is trial 66 with value: 46.08851090287338. [I 2026-05-29 13:25:56,191] Trial 91 finished with value: 139.53279744193276 and parameters: {'beta': 0.008787630984488391, 'rel_death_prob': 2.666629273962717}. Best is trial 66 with value: 46.08851090287338. [I 2026-05-29 13:25:56,230] Trial 92 finished with value: 141.52995514500157 and parameters: {'beta': 0.00920822767347531, 'rel_death_prob': 2.493435071709483}. Best is trial 66 with value: 46.08851090287338. [I 2026-05-29 13:25:56,500] Trial 93 finished with value: 323.0909979126882 and parameters: {'beta': 0.018686193298792394, 'rel_death_prob': 2.4758900440949327}. Best is trial 66 with value: 46.08851090287338. [I 2026-05-29 13:25:56,678] Trial 94 finished with value: 214.09013190034196 and parameters: {'beta': 0.017229011087926713, 'rel_death_prob': 2.487160099977696}. Best is trial 66 with value: 46.08851090287338. [I 2026-05-29 13:25:56,827] Trial 95 finished with value: 124.68257316694053 and parameters: {'beta': 0.016092574173911137, 'rel_death_prob': 2.4871566424544955}. Best is trial 66 with value: 46.08851090287338. [I 2026-05-29 13:25:56,873] Trial 96 finished with value: 189.19005640182974 and parameters: {'beta': 0.01766637517703349, 'rel_death_prob': 2.8611097648578063}. Best is trial 66 with value: 46.08851090287338. [I 2026-05-29 13:25:57,097] Trial 98 finished with value: 82.65754763067905 and parameters: {'beta': 0.014289016387880294, 'rel_death_prob': 1.9866303664294203}. Best is trial 66 with value: 46.08851090287338. [I 2026-05-29 13:25:57,123] Trial 97 finished with value: 161.326553270862 and parameters: {'beta': 0.016801171107593518, 'rel_death_prob': 2.3384080246512413}. Best is trial 66 with value: 46.08851090287338. [I 2026-05-29 13:25:57,295] Trial 99 finished with value: 49.74297197672869 and parameters: {'beta': 0.015289376696570245, 'rel_death_prob': 1.9639635777318745}. Best is trial 66 with value: 46.08851090287338.
Making results structure...
Processed 100 trials; 0 failed
Deleted study covasim_calibration in sqlite:///covasim_calibration.db
Removed existing calibration covasim_calibration.db
Calibration for 100 total trials completed in 19.2 s.
Initial parameter values:
#0. 'beta': 0.015
#1. 'rel_death_prob': 1.0
Best parameter values:
#0. 'beta': 0.013234829425226262
#1. 'rel_death_prob': 2.8042253055941613
Mismatch before calibration: 101.465
Mismatch after calibration: 46.0885
Percent improvement: 54.6%So it improved the fit (see above), but let’s visualize this as a plot:
# Plot the results
calib.plot_trend()
calib.plot_sims(to_plot=['cum_tests', 'cum_diagnoses', 'cum_deaths'])
Compared to scipy.optimize.minimize(), Optuna took less time and produced a better fit. Specifically, the mismatch obtained with Optuna is almost half the smallest mismatch achieved with scipy.optimize. However, Optuna’s results are still far from perfect – running calibration runs, or more iterations, and calibrating more parameters beyond just these two, would still be required before the model could be considered “calibrated”.
Sometimes you want to calibrate a parameter that isn’t a built-in parameter – for example, part of an intervention. You can do this using the custom_fn keyword argument. In this example, we calibrate the test probability by modifying the test_prob intervention directly.
pars = dict(
verbose = 0,
start_day = '2020-02-05',
pop_size = 1000,
pop_scale = 4,
interventions = cv.test_prob(symp_prob=0.01),
)
sim = cv.Sim(pars, datafile='example_data.csv')
calib_pars = dict(
beta = [0.013, 0.005, 0.020],
test_prob = [0.01, 0.00, 0.30]
)
def set_test_prob(sim, calib_pars):
tp = sim.get_intervention(cv.test_prob)
tp.symp_prob = calib_pars['test_prob']
return sim
calib = sim.calibrate(calib_pars=calib_pars, custom_fn=set_test_prob, total_trials=60)
calib.plot_all()
calib.plot_sims(to_plot=['cum_deaths', 'cum_diagnoses'])[I 2026-05-29 13:25:59,574] A new study created in RDB with name: covasim_calibration
Could not delete study, skipping...
'Record does not exist.'
Removed existing calibration covasim_calibration.db[I 2026-05-29 13:26:00,014] Trial 0 finished with value: 137.1989021381298 and parameters: {'beta': 0.010457176921730586, 'test_prob': 0.15645223776665906}. Best is trial 1 with value: 132.0757312533293. [I 2026-05-29 13:26:00,016] Trial 1 finished with value: 132.0757312533293 and parameters: {'beta': 0.019868254154857344, 'test_prob': 0.07925122966585484}. Best is trial 2 with value: 118.40084962687826. [I 2026-05-29 13:26:00,017] Trial 3 finished with value: 145.48811364134866 and parameters: {'beta': 0.006301876098526713, 'test_prob': 0.089991840658284}. Best is trial 2 with value: 118.40084962687826. [I 2026-05-29 13:26:00,025] Trial 2 finished with value: 118.40084962687826 and parameters: {'beta': 0.01049512852651999, 'test_prob': 0.01349697391771626}. Best is trial 2 with value: 118.40084962687826. [I 2026-05-29 13:26:00,209] Trial 6 finished with value: 141.85320847005153 and parameters: {'beta': 0.011795531949896822, 'test_prob': 0.2779747655091742}. Best is trial 2 with value: 118.40084962687826. [I 2026-05-29 13:26:00,219] Trial 7 finished with value: 145.31209455024015 and parameters: {'beta': 0.006117185496086811, 'test_prob': 0.16212756292203492}. Best is trial 4 with value: 109.50713835767935. [I 2026-05-29 13:26:00,221] Trial 4 finished with value: 109.50713835767935 and parameters: {'beta': 0.009572180474251537, 'test_prob': 0.16124351575361948}. Best is trial 4 with value: 109.50713835767935. [I 2026-05-29 13:26:00,238] Trial 5 finished with value: 61.97985766457559 and parameters: {'beta': 0.01844799374856563, 'test_prob': 0.12329231069630693}. Best is trial 5 with value: 61.97985766457559. [I 2026-05-29 13:26:00,422] Trial 8 finished with value: 125.901676835941 and parameters: {'beta': 0.011312891669307795, 'test_prob': 0.29078221556968825}. Best is trial 5 with value: 61.97985766457559. [I 2026-05-29 13:26:00,433] Trial 9 finished with value: 137.05841178465835 and parameters: {'beta': 0.011378278243329182, 'test_prob': 0.0569577257692895}. Best is trial 5 with value: 61.97985766457559. [I 2026-05-29 13:26:00,435] Trial 11 finished with value: 144.79629044523006 and parameters: {'beta': 0.007580335052529119, 'test_prob': 0.09600666366082301}. Best is trial 5 with value: 61.97985766457559. [I 2026-05-29 13:26:00,444] Trial 10 finished with value: 116.47308725408499 and parameters: {'beta': 0.01867195833195315, 'test_prob': 0.25034225261806214}. Best is trial 5 with value: 61.97985766457559. [I 2026-05-29 13:26:00,648] Trial 12 finished with value: 103.04619880252525 and parameters: {'beta': 0.01727388237861289, 'test_prob': 0.2140349909647608}. Best is trial 5 with value: 61.97985766457559. [I 2026-05-29 13:26:00,654] Trial 14 finished with value: 94.26497171740735 and parameters: {'beta': 0.01596401145066296, 'test_prob': 0.20406672227022626}. Best is trial 5 with value: 61.97985766457559. [I 2026-05-29 13:26:00,659] Trial 15 finished with value: 94.90390136079643 and parameters: {'beta': 0.015909249637381793, 'test_prob': 0.21804897424170855}. Best is trial 5 with value: 61.97985766457559. [I 2026-05-29 13:26:00,676] Trial 13 finished with value: 66.45210028379132 and parameters: {'beta': 0.01621002537364602, 'test_prob': 0.1818426701695777}. Best is trial 5 with value: 61.97985766457559. [I 2026-05-29 13:26:00,864] Trial 16 finished with value: 121.69245588877806 and parameters: {'beta': 0.015541126036509013, 'test_prob': 0.19645208906702621}. Best is trial 5 with value: 61.97985766457559. [I 2026-05-29 13:26:00,874] Trial 17 finished with value: 129.8363483904854 and parameters: {'beta': 0.014219730803606397, 'test_prob': 0.19453867013739493}. Best is trial 5 with value: 61.97985766457559. [I 2026-05-29 13:26:00,884] Trial 18 finished with value: 101.99542544343309 and parameters: {'beta': 0.014551160532827952, 'test_prob': 0.12615229180744564}. Best is trial 5 with value: 61.97985766457559. [I 2026-05-29 13:26:00,887] Trial 19 finished with value: 105.70024064100089 and parameters: {'beta': 0.014482144121869952, 'test_prob': 0.12301586318355569}. Best is trial 5 with value: 61.97985766457559. [I 2026-05-29 13:26:01,086] Trial 21 finished with value: 126.80660044490494 and parameters: {'beta': 0.01791097759217656, 'test_prob': 0.04520690798568254}. Best is trial 5 with value: 61.97985766457559. [I 2026-05-29 13:26:01,088] Trial 20 finished with value: 113.02777673821478 and parameters: {'beta': 0.017855812780021622, 'test_prob': 0.03871922508389536}. Best is trial 5 with value: 61.97985766457559. [I 2026-05-29 13:26:01,107] Trial 22 finished with value: 131.51249616800573 and parameters: {'beta': 0.017926637582518726, 'test_prob': 0.03432582677803603}. Best is trial 5 with value: 61.97985766457559. [I 2026-05-29 13:26:01,112] Trial 23 finished with value: 108.25688868471767 and parameters: {'beta': 0.018827646817093358, 'test_prob': 0.029652181820540857}. Best is trial 5 with value: 61.97985766457559. [I 2026-05-29 13:26:01,291] Trial 24 finished with value: 109.71022979865901 and parameters: {'beta': 0.013215703171180133, 'test_prob': 0.24284248436886535}. Best is trial 5 with value: 61.97985766457559. [I 2026-05-29 13:26:01,302] Trial 25 finished with value: 138.57232183436867 and parameters: {'beta': 0.013203483669553127, 'test_prob': 0.24476342558709757}. Best is trial 5 with value: 61.97985766457559. [I 2026-05-29 13:26:01,316] Trial 26 finished with value: 140.29812065737565 and parameters: {'beta': 0.012843632632888072, 'test_prob': 0.12979046349908158}. Best is trial 5 with value: 61.97985766457559. [I 2026-05-29 13:26:01,326] Trial 27 finished with value: 99.26468210991851 and parameters: {'beta': 0.016372160578021194, 'test_prob': 0.25222619066430485}. Best is trial 5 with value: 61.97985766457559. [I 2026-05-29 13:26:01,512] Trial 28 finished with value: 130.03360072872283 and parameters: {'beta': 0.016057988365588666, 'test_prob': 0.12626180725994302}. Best is trial 5 with value: 61.97985766457559. [I 2026-05-29 13:26:01,533] Trial 29 finished with value: 115.38424394132298 and parameters: {'beta': 0.019599344331873318, 'test_prob': 0.18328077144323307}. Best is trial 5 with value: 61.97985766457559. [I 2026-05-29 13:26:01,533] Trial 30 finished with value: 135.25761719753154 and parameters: {'beta': 0.016569914977525697, 'test_prob': 0.18760597251712433}. Best is trial 5 with value: 61.97985766457559. [I 2026-05-29 13:26:01,548] Trial 31 finished with value: 106.7653147945028 and parameters: {'beta': 0.01989619040825877, 'test_prob': 0.1917271699669035}. Best is trial 5 with value: 61.97985766457559. [I 2026-05-29 13:26:01,724] Trial 33 finished with value: 92.99400011888706 and parameters: {'beta': 0.015964867687519653, 'test_prob': 0.2171655964585061}. Best is trial 5 with value: 61.97985766457559. [I 2026-05-29 13:26:01,725] Trial 32 finished with value: 118.84779061165291 and parameters: {'beta': 0.015304148652336138, 'test_prob': 0.2206433108355492}. Best is trial 5 with value: 61.97985766457559. [I 2026-05-29 13:26:01,763] Trial 34 finished with value: 75.04898721619413 and parameters: {'beta': 0.01523906981520133, 'test_prob': 0.2172293628678592}. Best is trial 5 with value: 61.97985766457559. [I 2026-05-29 13:26:01,794] Trial 35 finished with value: 120.00608501582352 and parameters: {'beta': 0.015324722890282666, 'test_prob': 0.21892349831468821}. Best is trial 5 with value: 61.97985766457559. [I 2026-05-29 13:26:01,942] Trial 37 finished with value: 66.61197577204138 and parameters: {'beta': 0.01695726773104013, 'test_prob': 0.15426725053807447}. Best is trial 5 with value: 61.97985766457559. [I 2026-05-29 13:26:01,947] Trial 36 finished with value: 111.58375726604835 and parameters: {'beta': 0.017371033795776054, 'test_prob': 0.15381359148247759}. Best is trial 5 with value: 61.97985766457559. [I 2026-05-29 13:26:01,989] Trial 38 finished with value: 55.58951462443305 and parameters: {'beta': 0.017223123708084303, 'test_prob': 0.14598393211741523}. Best is trial 38 with value: 55.58951462443305. [I 2026-05-29 13:26:02,012] Trial 39 finished with value: 59.03095157824809 and parameters: {'beta': 0.017030206839442896, 'test_prob': 0.1503183038044989}. Best is trial 38 with value: 55.58951462443305. [I 2026-05-29 13:26:02,157] Trial 40 finished with value: 74.44628697470752 and parameters: {'beta': 0.01897471707926785, 'test_prob': 0.17196642322971092}. Best is trial 38 with value: 55.58951462443305. [I 2026-05-29 13:26:02,177] Trial 41 finished with value: 112.00156495289578 and parameters: {'beta': 0.019051188455340977, 'test_prob': 0.171257302717275}. Best is trial 38 with value: 55.58951462443305. [I 2026-05-29 13:26:02,203] Trial 42 finished with value: 74.75126299422311 and parameters: {'beta': 0.018974479600230087, 'test_prob': 0.17069745776427536}. Best is trial 38 with value: 55.58951462443305. [I 2026-05-29 13:26:02,230] Trial 43 finished with value: 41.444407760975864 and parameters: {'beta': 0.018716237635956007, 'test_prob': 0.10445955823336331}. Best is trial 43 with value: 41.444407760975864. [I 2026-05-29 13:26:02,375] Trial 44 finished with value: 84.26921128678988 and parameters: {'beta': 0.016811356068338574, 'test_prob': 0.14210751774908192}. Best is trial 43 with value: 41.444407760975864. [I 2026-05-29 13:26:02,404] Trial 45 finished with value: 93.70910900752665 and parameters: {'beta': 0.016981446635514055, 'test_prob': 0.1440945419338408}. Best is trial 43 with value: 41.444407760975864. [I 2026-05-29 13:26:02,419] Trial 46 finished with value: 60.961037276931805 and parameters: {'beta': 0.018117501131757183, 'test_prob': 0.10730312812449264}. Best is trial 43 with value: 41.444407760975864. [I 2026-05-29 13:26:02,428] Trial 47 finished with value: 145.17283752809215 and parameters: {'beta': 0.009270285779414456, 'test_prob': 0.10622512462493391}. Best is trial 43 with value: 41.444407760975864. [I 2026-05-29 13:26:02,585] Trial 48 finished with value: 126.20611688664603 and parameters: {'beta': 0.00941759056243948, 'test_prob': 0.0977442865154786}. Best is trial 43 with value: 41.444407760975864. [I 2026-05-29 13:26:02,637] Trial 49 finished with value: 54.24894738189754 and parameters: {'beta': 0.018077997496311837, 'test_prob': 0.07953528250409904}. Best is trial 43 with value: 41.444407760975864. [I 2026-05-29 13:26:02,638] Trial 50 finished with value: 59.68259838972003 and parameters: {'beta': 0.01811769082625173, 'test_prob': 0.06551498673715289}. Best is trial 43 with value: 41.444407760975864. [I 2026-05-29 13:26:02,648] Trial 51 finished with value: 66.04668729581272 and parameters: {'beta': 0.0181037844734396, 'test_prob': 0.07180546080474137}. Best is trial 43 with value: 41.444407760975864. [I 2026-05-29 13:26:02,761] Trial 52 finished with value: 145.4321594445076 and parameters: {'beta': 0.005206075024522764, 'test_prob': 0.07296700172192659}. Best is trial 43 with value: 41.444407760975864. [I 2026-05-29 13:26:02,856] Trial 53 finished with value: 128.43963889221732 and parameters: {'beta': 0.01954022774388474, 'test_prob': 0.08205757496082765}. Best is trial 43 with value: 41.444407760975864. [I 2026-05-29 13:26:02,861] Trial 54 finished with value: 66.76196358242221 and parameters: {'beta': 0.017502575480544, 'test_prob': 0.07344081683856779}. Best is trial 43 with value: 41.444407760975864. [I 2026-05-29 13:26:02,874] Trial 55 finished with value: 65.34309728969211 and parameters: {'beta': 0.018430210476745514, 'test_prob': 0.07926703790455344}. Best is trial 43 with value: 41.444407760975864. [I 2026-05-29 13:26:02,978] Trial 56 finished with value: 120.404319793701 and parameters: {'beta': 0.019466861015471158, 'test_prob': 0.0859028966125277}. Best is trial 43 with value: 41.444407760975864. [I 2026-05-29 13:26:03,057] Trial 58 finished with value: 65.56437044368244 and parameters: {'beta': 0.018319585088089185, 'test_prob': 0.1105718524782084}. Best is trial 43 with value: 41.444407760975864. [I 2026-05-29 13:26:03,076] Trial 57 finished with value: 61.45319425090234 and parameters: {'beta': 0.01826127979245502, 'test_prob': 0.054494404100793044}. Best is trial 43 with value: 41.444407760975864. [I 2026-05-29 13:26:03,092] Trial 59 finished with value: 116.17868539762858 and parameters: {'beta': 0.018382185385046948, 'test_prob': 0.10574560055367432}. Best is trial 43 with value: 41.444407760975864.
Making results structure...
Processed 60 trials; 0 failed
Deleted study covasim_calibration in sqlite:///covasim_calibration.db
Removed existing calibration covasim_calibration.db
Calibration for 60 total trials completed in 3.8 s.
Initial parameter values:
#0. 'beta': 0.013
#1. 'test_prob': 0.01
Best parameter values:
#0. 'beta': 0.018716237635956007
#1. 'test_prob': 0.10445955823336331
Mismatch before calibration: 141.953
Mismatch after calibration: 41.4444
Percent improvement: 70.8%
