Experiments

Patch experiment

PATCH
/api/v2/projects/{projectKey}/environments/{environmentKey}/experiments/{experimentKey}

Update an experiment. Updating an experiment uses the semantic patch format.

To make a semantic patch request, you must append domain-model=launchdarkly.semanticpatch to your Content-Type header. To learn more, read Updates using semantic patch.

Instructions

Semantic patch requests support the following kind instructions for updating experiments.

updateName

Deprecated: updateName will be removed in a future version. Use updateExperimentFields instead.

Updates the experiment name.

Parameters
  • value: The new name.

Here's an example:

{  "instructions": [{    "kind": "updateName",    "value": "Example updated experiment name"  }]}

updateDescription

Deprecated: updateDescription will be removed in a future version. Use updateExperimentFields instead.

Updates the experiment description.

Parameters
  • value: The new description.

Here's an example:

{  "instructions": [{    "kind": "updateDescription",    "value": "Example updated description"  }]}

updateExperimentFields

Updates one or more fields on an experiment or its current iteration. Each field update specifies an operation (add, update, or remove) and an optional value.

Which fields are mutable depends on the current iteration status. To discover which fields and operations are allowed, expand mutableFieldsByStatus on the Get experiment response.

Parameters
  • value: An object mapping field names to field updates. Each field update has the following properties:
    • operation: The operation to perform. One of add, update, or remove.
    • value: The new value for the field. Required for add and update operations.

To find which fields are supported and which operations are allowed for each iteration status, expand mutableFieldsByStatus on the Get experiment response.

Here's an example:

{  "instructions": [{    "kind": "updateExperimentFields",    "value": {      "name": {        "operation": "update",        "value": "Updated experiment name"      },      "tags": {        "operation": "add",        "value": ["tag1", "tag2"]      }    }  }]}

saveAndStartNewIteration

Stops the current running iteration, creates a new iteration from it, optionally applies field updates, and starts the new iteration. This is a convenience instruction that combines stopping, updating, and starting in a single operation.

Parameters
  • changeJustification: (Optional) The reason for stopping and starting a new iteration.
  • value: (Optional) An object mapping field names to field updates, using the same format as updateExperimentFields. These updates are applied to the new iteration before it is started.

Here's an example:

{  "instructions": [{    "kind": "saveAndStartNewIteration",    "changeJustification": "Adjusting hypothesis based on early results",    "value": {      "hypothesis": {        "operation": "update",        "value": "Updated hypothesis text"      }    }  }]}

startIteration

Starts a new iteration for this experiment. You must create a new iteration before calling this instruction.

An iteration may not be started until it meets the following criteria:

  • Its associated flag is toggled on and is not archived
  • Its randomizationUnit is set
  • At least one of its treatments has a non-zero allocationPercent
Parameters
  • changeJustification: The reason for starting a new iteration. Required when you call startIteration on an already running experiment, otherwise optional.

Here's an example:

{  "instructions": [{    "kind": "startIteration",    "changeJustification": "It's time to start a new iteration"  }]}

stopIteration

Stops the current iteration for this experiment.

Parameters
  • winningTreatmentId: The ID of the winning treatment. Treatment IDs are returned as part of the Get experiment response. They are the _id of each element in the treatments array.
  • winningReason: The reason for the winner

Here's an example:

{  "instructions": [{    "kind": "stopIteration",    "winningTreatmentId": "3a548ec2-72ac-4e59-8518-5c24f5609ccf",    "winningReason": "Example reason to stop the iteration"  }]}

archiveExperiment

Archives this experiment. Archived experiments are hidden by default in the LaunchDarkly user interface. You cannot start new iterations for archived experiments.

Here's an example:

{  "instructions": [{ "kind": "archiveExperiment" }]}

restoreExperiment

Restores an archived experiment. After restoring an experiment, you can start new iterations for it again.

Here's an example:

{  "instructions": [{ "kind": "restoreExperiment" }]}

Authorization

ApiKey read, write
Authorization<token>

In: header

Scope: read, write

Path Parameters

projectKey*string

The project key

Formatstring
environmentKey*string

The environment key

Formatstring
experimentKey*string

The experiment key

Formatstring

Request Body

application/json

comment?string

Optional comment describing the update

instructions*array<>

The instructions to perform when updating. This should be an array with objects that look like {"kind": "update_action"}. Some instructions also require a value field in the array element.

Response Body

application/json

application/json

application/json

application/json

application/json

application/json

application/json

application/json

curl -X PATCH "https://example.com/api/v2/projects/string/environments/string/experiments/string" \  -H "Content-Type: application/json" \  -d '{    "comment": "Example comment describing the update",    "instructions": [      {        "kind": "updateName",        "value": "Updated experiment name"      }    ]  }'
{  "_id": "12ab3c45de678910fgh12345",  "key": "experiment-key-123abc",  "name": "Example experiment",  "description": "An example experiment, used in testing",  "_maintainerId": "12ab3c45de678910fgh12345",  "_creationDate": "1654104600000",  "environmentKey": "string",  "methodology": "bayesian",  "dataSource": "snowflake",  "archivedDate": "1654104600000",  "tags": [    "experiment",    "feature"  ],  "_links": {    "parent": {      "href": "/api/v2/projects/my-project/environments/my-environment",      "type": "application/json"    },    "self": {      "href": "/api/v2/projects/my-project/environments/my-environment/experiments/my-experiment",      "type": "application/json"    }  },  "holdoutId": "f3b74309-d581-44e1-8a2b-bb2933b4fe40",  "currentIteration": {    "_id": "12ab3c45de678910fgh12345",    "hypothesis": "The new button placement will increase conversion",    "status": "running",    "createdAt": "1654104600000",    "startedAt": "1655314200000",    "endedAt": "1656610200000",    "winningTreatmentId": "122c9f3e-da26-4321-ba68-e0fc02eced58",    "winningReason": "We ran this iteration for two weeks and the winning variation was clear",    "canReshuffleTraffic": true,    "flags": {      "property1": {        "targetingRule": "fallthrough",        "targetingRuleDescription": "Customers who live in Canada",        "targetingRuleClauses": [          null        ],        "flagConfigVersion": 12,        "notInExperimentVariationId": "e432f62b-55f6-49dd-a02f-eb24acf39d05",        "_links": {          "self": {            "href": "/api/v2/flags/my-project/my-flag",            "type": "application/json"          }        }      },      "property2": {        "targetingRule": "fallthrough",        "targetingRuleDescription": "Customers who live in Canada",        "targetingRuleClauses": [          null        ],        "flagConfigVersion": 12,        "notInExperimentVariationId": "e432f62b-55f6-49dd-a02f-eb24acf39d05",        "_links": {          "self": {            "href": "/api/v2/flags/my-project/my-flag",            "type": "application/json"          }        }      }    },    "reallocationFrequencyMillis": 3600000,    "version": 0,    "primaryMetric": {      "key": "metric-key-123abc",      "_versionId": "string",      "name": "My metric",      "kind": "custom",      "isNumeric": true,      "eventKey": "event-key-123abc",      "_links": {        "self": {          "href": "/api/v2/metrics/my-project/my-metric",          "type": "application/json"        }      },      "isGroup": true,      "metrics": [        {          "key": "metric-key-123abc",          "_versionId": "version-id-123abc",          "name": "Example metric",          "kind": "custom",          "isNumeric": true,          "unitAggregationType": "sum",          "analysisType": "mean",          "eventKey": "event-key-123abc",          "analysisUnit": "string",          "_links": {            "self": {              "href": "/api/v2/metrics/my-project/my-metric",              "type": "application/json"            }          },          "nameInGroup": "Step 1",          "randomizationUnits": [            "user"          ],          "analysisUnits": [            "user"          ]        }      ],      "analysisType": "mean",      "analysisUnit": "user"    },    "primarySingleMetric": {      "key": "metric-key-123abc",      "_versionId": "version-id-123abc",      "name": "Example metric",      "kind": "custom",      "isNumeric": true,      "unitAggregationType": "sum",      "analysisType": "mean",      "eventKey": "event-key-123abc",      "analysisUnit": "string",      "_links": {        "self": {          "href": "/api/v2/metrics/my-project/my-metric",          "type": "application/json"        }      }    },    "primaryFunnel": {      "key": "metric-group-key-123abc",      "name": "My metric group",      "kind": "funnel",      "_links": {        "parent": {          "href": "/api/v2/projects/my-project",          "type": "application/json"        },        "self": {          "href": "/api/v2/projects/my-project/metric-groups/my-metric-group",          "type": "application/json"        }      },      "metrics": [        {          "key": "metric-key-123abc",          "_versionId": "version-id-123abc",          "name": "Example metric",          "kind": "custom",          "isNumeric": true,          "unitAggregationType": "sum",          "analysisType": "mean",          "eventKey": "event-key-123abc",          "analysisUnit": "string",          "_links": {            "self": {              "href": "/api/v2/metrics/my-project/my-metric",              "type": "application/json"            }          },          "nameInGroup": "Step 1",          "randomizationUnits": [            "user"          ],          "analysisUnits": [            "user"          ]        }      ]    },    "randomizationUnit": "user",    "attributes": [      "string"    ],    "treatments": [      {        "_id": "122c9f3e-da26-4321-ba68-e0fc02eced58",        "name": "Treatment 1",        "allocationPercent": "10",        "baseline": true,        "parameters": [          {            "variationId": "string",            "flagKey": "string"          }        ]      }    ],    "secondaryMetrics": [      {        "key": "metric-key-123abc",        "_versionId": "version-id-123abc",        "name": "Example metric",        "kind": "custom",        "isNumeric": true,        "unitAggregationType": "sum",        "analysisType": "mean",        "eventKey": "event-key-123abc",        "analysisUnit": "string",        "_links": {          "self": {            "href": "/api/v2/metrics/my-project/my-metric",            "type": "application/json"          }        }      }    ],    "metrics": [      {        "key": "metric-key-123abc",        "_versionId": "string",        "name": "My metric",        "kind": "custom",        "isNumeric": true,        "eventKey": "event-key-123abc",        "_links": {          "self": {            "href": "/api/v2/metrics/my-project/my-metric",            "type": "application/json"          }        },        "isGroup": true,        "metrics": [          {            "key": "metric-key-123abc",            "_versionId": "version-id-123abc",            "name": "Example metric",            "kind": "custom",            "isNumeric": true,            "unitAggregationType": "sum",            "analysisType": "mean",            "eventKey": "event-key-123abc",            "analysisUnit": "string",            "_links": {              "self": {                "href": "/api/v2/metrics/my-project/my-metric",                "type": "application/json"              }            },            "nameInGroup": "Step 1",            "randomizationUnits": [              "user"            ],            "analysisUnits": [              "user"            ]          }        ],        "analysisType": "mean",        "analysisUnit": "user"      }    ],    "layerSnapshot": {      "key": "checkout-flow",      "name": "Checkout Flow",      "reservationPercent": 10,      "otherReservationPercent": 70    },    "covariateInfo": {      "id": "74a49a2b-4834-4246-917e-5d85231d8c2a",      "fileName": "covariate.csv",      "createdAt": "1654104600000"    }  },  "type": "experiment",  "_access": {    "denied": [      {        "action": "string",        "reason": {          "resources": [            "proj/*:env/*;qa_*:/flag/*"          ],          "notResources": [            "string"          ],          "actions": [            "*"          ],          "notActions": [            "string"          ],          "effect": "allow",          "role_name": "string"        }      }    ],    "allowed": [      {        "action": "string",        "reason": {          "resources": [            "proj/*:env/*;qa_*:/flag/*"          ],          "notResources": [            "string"          ],          "actions": [            "*"          ],          "notActions": [            "string"          ],          "effect": "allow",          "role_name": "string"        }      }    ]  },  "draftIteration": {    "_id": "12ab3c45de678910fgh12345",    "hypothesis": "The new button placement will increase conversion",    "status": "running",    "createdAt": "1654104600000",    "startedAt": "1655314200000",    "endedAt": "1656610200000",    "winningTreatmentId": "122c9f3e-da26-4321-ba68-e0fc02eced58",    "winningReason": "We ran this iteration for two weeks and the winning variation was clear",    "canReshuffleTraffic": true,    "flags": {      "property1": {        "targetingRule": "fallthrough",        "targetingRuleDescription": "Customers who live in Canada",        "targetingRuleClauses": [          null        ],        "flagConfigVersion": 12,        "notInExperimentVariationId": "e432f62b-55f6-49dd-a02f-eb24acf39d05",        "_links": {          "self": {            "href": "/api/v2/flags/my-project/my-flag",            "type": "application/json"          }        }      },      "property2": {        "targetingRule": "fallthrough",        "targetingRuleDescription": "Customers who live in Canada",        "targetingRuleClauses": [          null        ],        "flagConfigVersion": 12,        "notInExperimentVariationId": "e432f62b-55f6-49dd-a02f-eb24acf39d05",        "_links": {          "self": {            "href": "/api/v2/flags/my-project/my-flag",            "type": "application/json"          }        }      }    },    "reallocationFrequencyMillis": 3600000,    "version": 0,    "primaryMetric": {      "key": "metric-key-123abc",      "_versionId": "string",      "name": "My metric",      "kind": "custom",      "isNumeric": true,      "eventKey": "event-key-123abc",      "_links": {        "self": {          "href": "/api/v2/metrics/my-project/my-metric",          "type": "application/json"        }      },      "isGroup": true,      "metrics": [        {          "key": "metric-key-123abc",          "_versionId": "version-id-123abc",          "name": "Example metric",          "kind": "custom",          "isNumeric": true,          "unitAggregationType": "sum",          "analysisType": "mean",          "eventKey": "event-key-123abc",          "analysisUnit": "string",          "_links": {            "self": {              "href": "/api/v2/metrics/my-project/my-metric",              "type": "application/json"            }          },          "nameInGroup": "Step 1",          "randomizationUnits": [            "user"          ],          "analysisUnits": [            "user"          ]        }      ],      "analysisType": "mean",      "analysisUnit": "user"    },    "primarySingleMetric": {      "key": "metric-key-123abc",      "_versionId": "version-id-123abc",      "name": "Example metric",      "kind": "custom",      "isNumeric": true,      "unitAggregationType": "sum",      "analysisType": "mean",      "eventKey": "event-key-123abc",      "analysisUnit": "string",      "_links": {        "self": {          "href": "/api/v2/metrics/my-project/my-metric",          "type": "application/json"        }      }    },    "primaryFunnel": {      "key": "metric-group-key-123abc",      "name": "My metric group",      "kind": "funnel",      "_links": {        "parent": {          "href": "/api/v2/projects/my-project",          "type": "application/json"        },        "self": {          "href": "/api/v2/projects/my-project/metric-groups/my-metric-group",          "type": "application/json"        }      },      "metrics": [        {          "key": "metric-key-123abc",          "_versionId": "version-id-123abc",          "name": "Example metric",          "kind": "custom",          "isNumeric": true,          "unitAggregationType": "sum",          "analysisType": "mean",          "eventKey": "event-key-123abc",          "analysisUnit": "string",          "_links": {            "self": {              "href": "/api/v2/metrics/my-project/my-metric",              "type": "application/json"            }          },          "nameInGroup": "Step 1",          "randomizationUnits": [            "user"          ],          "analysisUnits": [            "user"          ]        }      ]    },    "randomizationUnit": "user",    "attributes": [      "string"    ],    "treatments": [      {        "_id": "122c9f3e-da26-4321-ba68-e0fc02eced58",        "name": "Treatment 1",        "allocationPercent": "10",        "baseline": true,        "parameters": [          {            "variationId": "string",            "flagKey": "string"          }        ]      }    ],    "secondaryMetrics": [      {        "key": "metric-key-123abc",        "_versionId": "version-id-123abc",        "name": "Example metric",        "kind": "custom",        "isNumeric": true,        "unitAggregationType": "sum",        "analysisType": "mean",        "eventKey": "event-key-123abc",        "analysisUnit": "string",        "_links": {          "self": {            "href": "/api/v2/metrics/my-project/my-metric",            "type": "application/json"          }        }      }    ],    "metrics": [      {        "key": "metric-key-123abc",        "_versionId": "string",        "name": "My metric",        "kind": "custom",        "isNumeric": true,        "eventKey": "event-key-123abc",        "_links": {          "self": {            "href": "/api/v2/metrics/my-project/my-metric",            "type": "application/json"          }        },        "isGroup": true,        "metrics": [          {            "key": "metric-key-123abc",            "_versionId": "version-id-123abc",            "name": "Example metric",            "kind": "custom",            "isNumeric": true,            "unitAggregationType": "sum",            "analysisType": "mean",            "eventKey": "event-key-123abc",            "analysisUnit": "string",            "_links": {              "self": {                "href": "/api/v2/metrics/my-project/my-metric",                "type": "application/json"              }            },            "nameInGroup": "Step 1",            "randomizationUnits": [              "user"            ],            "analysisUnits": [              "user"            ]          }        ],        "analysisType": "mean",        "analysisUnit": "user"      }    ],    "layerSnapshot": {      "key": "checkout-flow",      "name": "Checkout Flow",      "reservationPercent": 10,      "otherReservationPercent": 70    },    "covariateInfo": {      "id": "74a49a2b-4834-4246-917e-5d85231d8c2a",      "fileName": "covariate.csv",      "createdAt": "1654104600000"    }  },  "previousIterations": [    {      "_id": "12ab3c45de678910fgh12345",      "hypothesis": "The new button placement will increase conversion",      "status": "running",      "createdAt": "1654104600000",      "startedAt": "1655314200000",      "endedAt": "1656610200000",      "winningTreatmentId": "122c9f3e-da26-4321-ba68-e0fc02eced58",      "winningReason": "We ran this iteration for two weeks and the winning variation was clear",      "canReshuffleTraffic": true,      "flags": {        "property1": {          "targetingRule": "fallthrough",          "targetingRuleDescription": "Customers who live in Canada",          "targetingRuleClauses": [            null          ],          "flagConfigVersion": 12,          "notInExperimentVariationId": "e432f62b-55f6-49dd-a02f-eb24acf39d05",          "_links": {            "self": {              "href": "/api/v2/flags/my-project/my-flag",              "type": "application/json"            }          }        },        "property2": {          "targetingRule": "fallthrough",          "targetingRuleDescription": "Customers who live in Canada",          "targetingRuleClauses": [            null          ],          "flagConfigVersion": 12,          "notInExperimentVariationId": "e432f62b-55f6-49dd-a02f-eb24acf39d05",          "_links": {            "self": {              "href": "/api/v2/flags/my-project/my-flag",              "type": "application/json"            }          }        }      },      "reallocationFrequencyMillis": 3600000,      "version": 0,      "primaryMetric": {        "key": "metric-key-123abc",        "_versionId": "string",        "name": "My metric",        "kind": "custom",        "isNumeric": true,        "eventKey": "event-key-123abc",        "_links": {          "self": {            "href": "/api/v2/metrics/my-project/my-metric",            "type": "application/json"          }        },        "isGroup": true,        "metrics": [          {            "key": "metric-key-123abc",            "_versionId": "version-id-123abc",            "name": "Example metric",            "kind": "custom",            "isNumeric": true,            "unitAggregationType": "sum",            "analysisType": "mean",            "eventKey": "event-key-123abc",            "analysisUnit": "string",            "_links": {              "self": {                "href": "/api/v2/metrics/my-project/my-metric",                "type": "application/json"              }            },            "nameInGroup": "Step 1",            "randomizationUnits": [              "user"            ],            "analysisUnits": [              "user"            ]          }        ],        "analysisType": "mean",        "analysisUnit": "user"      },      "primarySingleMetric": {        "key": "metric-key-123abc",        "_versionId": "version-id-123abc",        "name": "Example metric",        "kind": "custom",        "isNumeric": true,        "unitAggregationType": "sum",        "analysisType": "mean",        "eventKey": "event-key-123abc",        "analysisUnit": "string",        "_links": {          "self": {            "href": "/api/v2/metrics/my-project/my-metric",            "type": "application/json"          }        }      },      "primaryFunnel": {        "key": "metric-group-key-123abc",        "name": "My metric group",        "kind": "funnel",        "_links": {          "parent": {            "href": "/api/v2/projects/my-project",            "type": "application/json"          },          "self": {            "href": "/api/v2/projects/my-project/metric-groups/my-metric-group",            "type": "application/json"          }        },        "metrics": [          {            "key": "metric-key-123abc",            "_versionId": "version-id-123abc",            "name": "Example metric",            "kind": "custom",            "isNumeric": true,            "unitAggregationType": "sum",            "analysisType": "mean",            "eventKey": "event-key-123abc",            "analysisUnit": "string",            "_links": {              "self": {                "href": "/api/v2/metrics/my-project/my-metric",                "type": "application/json"              }            },            "nameInGroup": "Step 1",            "randomizationUnits": [              "user"            ],            "analysisUnits": [              "user"            ]          }        ]      },      "randomizationUnit": "user",      "attributes": [        "string"      ],      "treatments": [        {          "_id": "122c9f3e-da26-4321-ba68-e0fc02eced58",          "name": "Treatment 1",          "allocationPercent": "10",          "baseline": true,          "parameters": [            {              "variationId": "string",              "flagKey": "string"            }          ]        }      ],      "secondaryMetrics": [        {          "key": "metric-key-123abc",          "_versionId": "version-id-123abc",          "name": "Example metric",          "kind": "custom",          "isNumeric": true,          "unitAggregationType": "sum",          "analysisType": "mean",          "eventKey": "event-key-123abc",          "analysisUnit": "string",          "_links": {            "self": {              "href": "/api/v2/metrics/my-project/my-metric",              "type": "application/json"            }          }        }      ],      "metrics": [        {          "key": "metric-key-123abc",          "_versionId": "string",          "name": "My metric",          "kind": "custom",          "isNumeric": true,          "eventKey": "event-key-123abc",          "_links": {            "self": {              "href": "/api/v2/metrics/my-project/my-metric",              "type": "application/json"            }          },          "isGroup": true,          "metrics": [            {              "key": "metric-key-123abc",              "_versionId": "version-id-123abc",              "name": "Example metric",              "kind": "custom",              "isNumeric": true,              "unitAggregationType": "sum",              "analysisType": "mean",              "eventKey": "event-key-123abc",              "analysisUnit": "string",              "_links": {                "self": {                  "href": "/api/v2/metrics/my-project/my-metric",                  "type": "application/json"                }              },              "nameInGroup": "Step 1",              "randomizationUnits": [                "user"              ],              "analysisUnits": [                "user"              ]            }          ],          "analysisType": "mean",          "analysisUnit": "user"        }      ],      "layerSnapshot": {        "key": "checkout-flow",        "name": "Checkout Flow",        "reservationPercent": 10,        "otherReservationPercent": 70      },      "covariateInfo": {        "id": "74a49a2b-4834-4246-917e-5d85231d8c2a",        "fileName": "covariate.csv",        "createdAt": "1654104600000"      }    }  ],  "analysisConfig": {    "bayesianThreshold": "string",    "significanceThreshold": "string",    "testDirection": "string",    "multipleComparisonCorrectionMethod": "bonferroni",    "multipleComparisonCorrectionScope": "variations",    "sequentialTestingEnabled": true  },  "mutableFieldsByStatus": {    "not_started": {      "property1": [        "string"      ],      "property2": [        "string"      ]    },    "running": {      "property1": [        "string"      ],      "property2": [        "string"      ]    },    "stopped": {      "property1": [        "string"      ],      "property2": [        "string"      ]    }  }}

Get experiment GET

Get details about an experiment. ### Expanding the experiment response LaunchDarkly supports five fields for expanding the "Get experiment" response. By default, these fields are **not** included in the response. To expand the response, append the `expand` query parameter and add a comma-separated list with any of the following fields: - `previousIterations` includes all iterations prior to the current iteration. By default only the current iteration is included in the response. - `draftIteration` includes the iteration which has not been started yet, if any. - `secondaryMetrics` includes secondary metrics. By default only the primary metric is included in the response. - `treatments` includes all treatment and parameter details. By default treatment data is not included in the response. - `analysisConfig` includes the analysis configuration for the experiment, such as the Bayesian threshold or significance threshold. For example, `expand=draftIteration,treatments` includes the `draftIteration` and `treatments` fields in the response. If fields that you request with the `expand` query parameter are empty, they are not included in the response.

Create iteration POST

> **Deprecated**: This endpoint will be removed in a future version. Use the `updateExperimentFields` and `saveAndStartNewIteration` instructions on [Update experiment](https://launchdarkly.com/docs/api/experiments/patch-experiment) instead. Create an experiment iteration. Experiment iterations let you record experiments in individual blocks of time. Initially, iterations are created with a status of `not_started` and appear in the `draftIteration` field of an experiment. To start or stop an iteration, [update the experiment](https://launchdarkly.com/docs/api/experiments/patch-experiment) with the `startIteration` or `stopIteration` instruction. To learn more, read [Start experiment iterations](https://launchdarkly.com/docs/home/experimentation/create#start-an-experiment-iteration).