Continue an Experiment¶
This page shows you how you can resume logging to an experiment that already exists in Comet.
There are scenarios where you need to continue logging data to an existing experiment run.
Here are some:
- The run is interrupted due to an error or a loss of internet connectivity.
- Your training and evaluation operations happen in different scripts.
To handle these, it is best to use Comet's ExistingExperiment
object.
- You need to be able to get access to the current experiment created in another part of the code but can't pass the experiment object directly.
To handle this scenario, it is best to use Comet's get_global
method.
Use the ExistingExperiment
method¶
Comet provides an ExistingExperiment
object that allows you to continue logging from where you left off.
ExistingExperiment
is particularly useful when you need to continue logging system metrics from your resumed run, or would like to pass in different command-line arguments to the resumed experiment. By default, these flags will be turned off when trying to resume an experiment run.
The following snippets illustrate how you would structure your code to continue experiments.
Create an Experiment¶
experiment = comet_ml.Experiment(
api_key="<Your API Key>",
project_name="<Your Project Name>"
)
experiment.log_metric('my-metric', 1)
Note
There are alternatives to setting the API key programatically. See more here.
Resume an Experiment¶
resume = True
if resume:
experiment = comet_ml.ExistingExperiment(
api_key="<Your API Key>",
experiment_key="<Key of experiment to resume>",
)
else:
experiment = comet_ml.Experiment(
api_key="<Your API Key>",
project_name="<Your Project Name>",
)
# your model training or evaluation code
There are two ways to set the experiment_key
parameter. The first is by setting it directly in the source code. The second, by setting the following environment variable:
export COMET_EXPERIMENT_KEY=<The experiment key of the existing experiment>
When resuming logging, note that ExistingExperiment
does not alter or destroy previously logged information. To override or add to previous information, you must set the following parameters in the ExistingExperiment
object to True
:
log_code | log_graph | parse_args |
log_env_details | log_git_metadata | log_git_patch |
log_env_gpu | log_env_cpu | log_env_host |
Use the get_global
method¶
The get_global
method lets you get access to the current experiment created in another part of the code. So:
import comet_ml
experiment = comet_ml.get_global_experiment()
Info
The details of comet_ml.get_global_experiment()
may change in the future.
If you are trying to continue an experiment from a different session, worker, process, or thread, it is recommended you use the method described in the previous section.
Try it out!¶
Try our example for Comet in Jupyter Notebooks: