Key stats
- 10 authoritative HTTPS sources
- 16 preregistered protocol stages
- 0 completed client-outcome claims
Key takeaways
- Freeze the population, versions, and definitions before reviewing outcomes.
- Preserve missingness, disagreement, reversals, reverts, and residual cases.
- Keep analysis separate from deployment authority and risk acceptance.
Research question and boundary
Deployment Rollback Verification Delay Study asks whether rollback duration measures include destination verification and residual-state reconciliation rather than only the rollback command. Stage 1, research question and boundary, declares one production change linked to trigger, rollback decision, execution events, health observations, residual checks, and verified recovery as the observation unit and freezes population, system boundaries, local and UTC times, source versions, inclusion rules, exclusions, and intended decision before outcomes are reviewed. Record DO-R-1013-01 distinguishes system events, engineering interpretations, owner decisions, implementation evidence, and unresolved hypotheses. This prospective protocol reports no completed client outcomes and defines no universal performance benchmark.
At stage 1 for Deployment Rollback Verification Delay Study, include routine, missing-source, wrong-identity, conflicting-version, late-arriving, inaccessible-system, unavailable-owner, reversed, corrected, reverted, and reopened cases. Two reviewers classify a stratified subset independently using written anchors, preserve first-pass disagreement, and adjudicate against retained evidence. Report numerator, denominator, exclusions, null values, age bands, correction rate, disagreement, and residual cases together. Alternative explanations include configuration, case mix, cache state, clock drift, synchronization, sampling, ownership changes, release versions, seasonality, and selection effects. The ten public sources guide control design but do not inspect this system, establish causation, certify a provider, or authorize deployment. Preserve protocol, queries, codebook, fixtures, sample receipt, chronology, reviewer notes, decision log, approved action, verification, limitations, and refresh trigger so another authorized engineer can reproduce the bounded result without retaining secrets or unnecessary production data.
Population and observation window
Deployment Rollback Verification Delay Study asks whether rollback duration measures include destination verification and residual-state reconciliation rather than only the rollback command. Stage 2, population and observation window, declares one production change linked to trigger, rollback decision, execution events, health observations, residual checks, and verified recovery as the observation unit and freezes population, system boundaries, local and UTC times, source versions, inclusion rules, exclusions, and intended decision before outcomes are reviewed. Record DO-R-1013-02 distinguishes system events, engineering interpretations, owner decisions, implementation evidence, and unresolved hypotheses. This prospective protocol reports no completed client outcomes and defines no universal performance benchmark.
At stage 2 for Deployment Rollback Verification Delay Study, include routine, missing-source, wrong-identity, conflicting-version, late-arriving, inaccessible-system, unavailable-owner, reversed, corrected, reverted, and reopened cases. Two reviewers classify a stratified subset independently using written anchors, preserve first-pass disagreement, and adjudicate against retained evidence. Report numerator, denominator, exclusions, null values, age bands, correction rate, disagreement, and residual cases together. Alternative explanations include configuration, case mix, cache state, clock drift, synchronization, sampling, ownership changes, release versions, seasonality, and selection effects. The ten public sources guide control design but do not inspect this system, establish causation, certify a provider, or authorize deployment. Preserve protocol, queries, codebook, fixtures, sample receipt, chronology, reviewer notes, decision log, approved action, verification, limitations, and refresh trigger so another authorized engineer can reproduce the bounded result without retaining secrets or unnecessary production data.
Source hierarchy and provenance
Deployment Rollback Verification Delay Study asks whether rollback duration measures include destination verification and residual-state reconciliation rather than only the rollback command. Stage 3, source hierarchy and provenance, declares one production change linked to trigger, rollback decision, execution events, health observations, residual checks, and verified recovery as the observation unit and freezes population, system boundaries, local and UTC times, source versions, inclusion rules, exclusions, and intended decision before outcomes are reviewed. Record DO-R-1013-03 distinguishes system events, engineering interpretations, owner decisions, implementation evidence, and unresolved hypotheses. This prospective protocol reports no completed client outcomes and defines no universal performance benchmark.
At stage 3 for Deployment Rollback Verification Delay Study, include routine, missing-source, wrong-identity, conflicting-version, late-arriving, inaccessible-system, unavailable-owner, reversed, corrected, reverted, and reopened cases. Two reviewers classify a stratified subset independently using written anchors, preserve first-pass disagreement, and adjudicate against retained evidence. Report numerator, denominator, exclusions, null values, age bands, correction rate, disagreement, and residual cases together. Alternative explanations include configuration, case mix, cache state, clock drift, synchronization, sampling, ownership changes, release versions, seasonality, and selection effects. The ten public sources guide control design but do not inspect this system, establish causation, certify a provider, or authorize deployment. Preserve protocol, queries, codebook, fixtures, sample receipt, chronology, reviewer notes, decision log, approved action, verification, limitations, and refresh trigger so another authorized engineer can reproduce the bounded result without retaining secrets or unnecessary production data.
Unit of analysis
Deployment Rollback Verification Delay Study asks whether rollback duration measures include destination verification and residual-state reconciliation rather than only the rollback command. Stage 4, unit of analysis, declares one production change linked to trigger, rollback decision, execution events, health observations, residual checks, and verified recovery as the observation unit and freezes population, system boundaries, local and UTC times, source versions, inclusion rules, exclusions, and intended decision before outcomes are reviewed. Record DO-R-1013-04 distinguishes system events, engineering interpretations, owner decisions, implementation evidence, and unresolved hypotheses. This prospective protocol reports no completed client outcomes and defines no universal performance benchmark.
At stage 4 for Deployment Rollback Verification Delay Study, include routine, missing-source, wrong-identity, conflicting-version, late-arriving, inaccessible-system, unavailable-owner, reversed, corrected, reverted, and reopened cases. Two reviewers classify a stratified subset independently using written anchors, preserve first-pass disagreement, and adjudicate against retained evidence. Report numerator, denominator, exclusions, null values, age bands, correction rate, disagreement, and residual cases together. Alternative explanations include configuration, case mix, cache state, clock drift, synchronization, sampling, ownership changes, release versions, seasonality, and selection effects. The ten public sources guide control design but do not inspect this system, establish causation, certify a provider, or authorize deployment. Preserve protocol, queries, codebook, fixtures, sample receipt, chronology, reviewer notes, decision log, approved action, verification, limitations, and refresh trigger so another authorized engineer can reproduce the bounded result without retaining secrets or unnecessary production data.
Sampling and exclusions
Deployment Rollback Verification Delay Study asks whether rollback duration measures include destination verification and residual-state reconciliation rather than only the rollback command. Stage 5, sampling and exclusions, declares one production change linked to trigger, rollback decision, execution events, health observations, residual checks, and verified recovery as the observation unit and freezes population, system boundaries, local and UTC times, source versions, inclusion rules, exclusions, and intended decision before outcomes are reviewed. Record DO-R-1013-05 distinguishes system events, engineering interpretations, owner decisions, implementation evidence, and unresolved hypotheses. This prospective protocol reports no completed client outcomes and defines no universal performance benchmark.
At stage 5 for Deployment Rollback Verification Delay Study, include routine, missing-source, wrong-identity, conflicting-version, late-arriving, inaccessible-system, unavailable-owner, reversed, corrected, reverted, and reopened cases. Two reviewers classify a stratified subset independently using written anchors, preserve first-pass disagreement, and adjudicate against retained evidence. Report numerator, denominator, exclusions, null values, age bands, correction rate, disagreement, and residual cases together. Alternative explanations include configuration, case mix, cache state, clock drift, synchronization, sampling, ownership changes, release versions, seasonality, and selection effects. The ten public sources guide control design but do not inspect this system, establish causation, certify a provider, or authorize deployment. Preserve protocol, queries, codebook, fixtures, sample receipt, chronology, reviewer notes, decision log, approved action, verification, limitations, and refresh trigger so another authorized engineer can reproduce the bounded result without retaining secrets or unnecessary production data.
Independent classification
Deployment Rollback Verification Delay Study asks whether rollback duration measures include destination verification and residual-state reconciliation rather than only the rollback command. Stage 6, independent classification, declares one production change linked to trigger, rollback decision, execution events, health observations, residual checks, and verified recovery as the observation unit and freezes population, system boundaries, local and UTC times, source versions, inclusion rules, exclusions, and intended decision before outcomes are reviewed. Record DO-R-1013-06 distinguishes system events, engineering interpretations, owner decisions, implementation evidence, and unresolved hypotheses. This prospective protocol reports no completed client outcomes and defines no universal performance benchmark.
At stage 6 for Deployment Rollback Verification Delay Study, include routine, missing-source, wrong-identity, conflicting-version, late-arriving, inaccessible-system, unavailable-owner, reversed, corrected, reverted, and reopened cases. Two reviewers classify a stratified subset independently using written anchors, preserve first-pass disagreement, and adjudicate against retained evidence. Report numerator, denominator, exclusions, null values, age bands, correction rate, disagreement, and residual cases together. Alternative explanations include configuration, case mix, cache state, clock drift, synchronization, sampling, ownership changes, release versions, seasonality, and selection effects. The ten public sources guide control design but do not inspect this system, establish causation, certify a provider, or authorize deployment. Preserve protocol, queries, codebook, fixtures, sample receipt, chronology, reviewer notes, decision log, approved action, verification, limitations, and refresh trigger so another authorized engineer can reproduce the bounded result without retaining secrets or unnecessary production data.
Adverse and negative controls
Deployment Rollback Verification Delay Study asks whether rollback duration measures include destination verification and residual-state reconciliation rather than only the rollback command. Stage 7, adverse and negative controls, declares one production change linked to trigger, rollback decision, execution events, health observations, residual checks, and verified recovery as the observation unit and freezes population, system boundaries, local and UTC times, source versions, inclusion rules, exclusions, and intended decision before outcomes are reviewed. Record DO-R-1013-07 distinguishes system events, engineering interpretations, owner decisions, implementation evidence, and unresolved hypotheses. This prospective protocol reports no completed client outcomes and defines no universal performance benchmark.
At stage 7 for Deployment Rollback Verification Delay Study, include routine, missing-source, wrong-identity, conflicting-version, late-arriving, inaccessible-system, unavailable-owner, reversed, corrected, reverted, and reopened cases. Two reviewers classify a stratified subset independently using written anchors, preserve first-pass disagreement, and adjudicate against retained evidence. Report numerator, denominator, exclusions, null values, age bands, correction rate, disagreement, and residual cases together. Alternative explanations include configuration, case mix, cache state, clock drift, synchronization, sampling, ownership changes, release versions, seasonality, and selection effects. The ten public sources guide control design but do not inspect this system, establish causation, certify a provider, or authorize deployment. Preserve protocol, queries, codebook, fixtures, sample receipt, chronology, reviewer notes, decision log, approved action, verification, limitations, and refresh trigger so another authorized engineer can reproduce the bounded result without retaining secrets or unnecessary production data.
Missingness analysis
Deployment Rollback Verification Delay Study asks whether rollback duration measures include destination verification and residual-state reconciliation rather than only the rollback command. Stage 8, missingness analysis, declares one production change linked to trigger, rollback decision, execution events, health observations, residual checks, and verified recovery as the observation unit and freezes population, system boundaries, local and UTC times, source versions, inclusion rules, exclusions, and intended decision before outcomes are reviewed. Record DO-R-1013-08 distinguishes system events, engineering interpretations, owner decisions, implementation evidence, and unresolved hypotheses. This prospective protocol reports no completed client outcomes and defines no universal performance benchmark.
At stage 8 for Deployment Rollback Verification Delay Study, include routine, missing-source, wrong-identity, conflicting-version, late-arriving, inaccessible-system, unavailable-owner, reversed, corrected, reverted, and reopened cases. Two reviewers classify a stratified subset independently using written anchors, preserve first-pass disagreement, and adjudicate against retained evidence. Report numerator, denominator, exclusions, null values, age bands, correction rate, disagreement, and residual cases together. Alternative explanations include configuration, case mix, cache state, clock drift, synchronization, sampling, ownership changes, release versions, seasonality, and selection effects. The ten public sources guide control design but do not inspect this system, establish causation, certify a provider, or authorize deployment. Preserve protocol, queries, codebook, fixtures, sample receipt, chronology, reviewer notes, decision log, approved action, verification, limitations, and refresh trigger so another authorized engineer can reproduce the bounded result without retaining secrets or unnecessary production data.
Timeline reconstruction
Deployment Rollback Verification Delay Study asks whether rollback duration measures include destination verification and residual-state reconciliation rather than only the rollback command. Stage 9, timeline reconstruction, declares one production change linked to trigger, rollback decision, execution events, health observations, residual checks, and verified recovery as the observation unit and freezes population, system boundaries, local and UTC times, source versions, inclusion rules, exclusions, and intended decision before outcomes are reviewed. Record DO-R-1013-09 distinguishes system events, engineering interpretations, owner decisions, implementation evidence, and unresolved hypotheses. This prospective protocol reports no completed client outcomes and defines no universal performance benchmark.
At stage 9 for Deployment Rollback Verification Delay Study, include routine, missing-source, wrong-identity, conflicting-version, late-arriving, inaccessible-system, unavailable-owner, reversed, corrected, reverted, and reopened cases. Two reviewers classify a stratified subset independently using written anchors, preserve first-pass disagreement, and adjudicate against retained evidence. Report numerator, denominator, exclusions, null values, age bands, correction rate, disagreement, and residual cases together. Alternative explanations include configuration, case mix, cache state, clock drift, synchronization, sampling, ownership changes, release versions, seasonality, and selection effects. The ten public sources guide control design but do not inspect this system, establish causation, certify a provider, or authorize deployment. Preserve protocol, queries, codebook, fixtures, sample receipt, chronology, reviewer notes, decision log, approved action, verification, limitations, and refresh trigger so another authorized engineer can reproduce the bounded result without retaining secrets or unnecessary production data.
Alternative explanations
Deployment Rollback Verification Delay Study asks whether rollback duration measures include destination verification and residual-state reconciliation rather than only the rollback command. Stage 10, alternative explanations, declares one production change linked to trigger, rollback decision, execution events, health observations, residual checks, and verified recovery as the observation unit and freezes population, system boundaries, local and UTC times, source versions, inclusion rules, exclusions, and intended decision before outcomes are reviewed. Record DO-R-1013-10 distinguishes system events, engineering interpretations, owner decisions, implementation evidence, and unresolved hypotheses. This prospective protocol reports no completed client outcomes and defines no universal performance benchmark.
At stage 10 for Deployment Rollback Verification Delay Study, include routine, missing-source, wrong-identity, conflicting-version, late-arriving, inaccessible-system, unavailable-owner, reversed, corrected, reverted, and reopened cases. Two reviewers classify a stratified subset independently using written anchors, preserve first-pass disagreement, and adjudicate against retained evidence. Report numerator, denominator, exclusions, null values, age bands, correction rate, disagreement, and residual cases together. Alternative explanations include configuration, case mix, cache state, clock drift, synchronization, sampling, ownership changes, release versions, seasonality, and selection effects. The ten public sources guide control design but do not inspect this system, establish causation, certify a provider, or authorize deployment. Preserve protocol, queries, codebook, fixtures, sample receipt, chronology, reviewer notes, decision log, approved action, verification, limitations, and refresh trigger so another authorized engineer can reproduce the bounded result without retaining secrets or unnecessary production data.
Sensitivity analysis
Deployment Rollback Verification Delay Study asks whether rollback duration measures include destination verification and residual-state reconciliation rather than only the rollback command. Stage 11, sensitivity analysis, declares one production change linked to trigger, rollback decision, execution events, health observations, residual checks, and verified recovery as the observation unit and freezes population, system boundaries, local and UTC times, source versions, inclusion rules, exclusions, and intended decision before outcomes are reviewed. Record DO-R-1013-11 distinguishes system events, engineering interpretations, owner decisions, implementation evidence, and unresolved hypotheses. This prospective protocol reports no completed client outcomes and defines no universal performance benchmark.
At stage 11 for Deployment Rollback Verification Delay Study, include routine, missing-source, wrong-identity, conflicting-version, late-arriving, inaccessible-system, unavailable-owner, reversed, corrected, reverted, and reopened cases. Two reviewers classify a stratified subset independently using written anchors, preserve first-pass disagreement, and adjudicate against retained evidence. Report numerator, denominator, exclusions, null values, age bands, correction rate, disagreement, and residual cases together. Alternative explanations include configuration, case mix, cache state, clock drift, synchronization, sampling, ownership changes, release versions, seasonality, and selection effects. The ten public sources guide control design but do not inspect this system, establish causation, certify a provider, or authorize deployment. Preserve protocol, queries, codebook, fixtures, sample receipt, chronology, reviewer notes, decision log, approved action, verification, limitations, and refresh trigger so another authorized engineer can reproduce the bounded result without retaining secrets or unnecessary production data.
Authority and access boundary
Deployment Rollback Verification Delay Study asks whether rollback duration measures include destination verification and residual-state reconciliation rather than only the rollback command. Stage 12, authority and access boundary, declares one production change linked to trigger, rollback decision, execution events, health observations, residual checks, and verified recovery as the observation unit and freezes population, system boundaries, local and UTC times, source versions, inclusion rules, exclusions, and intended decision before outcomes are reviewed. Record DO-R-1013-12 distinguishes system events, engineering interpretations, owner decisions, implementation evidence, and unresolved hypotheses. This prospective protocol reports no completed client outcomes and defines no universal performance benchmark.
At stage 12 for Deployment Rollback Verification Delay Study, include routine, missing-source, wrong-identity, conflicting-version, late-arriving, inaccessible-system, unavailable-owner, reversed, corrected, reverted, and reopened cases. Two reviewers classify a stratified subset independently using written anchors, preserve first-pass disagreement, and adjudicate against retained evidence. Report numerator, denominator, exclusions, null values, age bands, correction rate, disagreement, and residual cases together. Alternative explanations include configuration, case mix, cache state, clock drift, synchronization, sampling, ownership changes, release versions, seasonality, and selection effects. The ten public sources guide control design but do not inspect this system, establish causation, certify a provider, or authorize deployment. Preserve protocol, queries, codebook, fixtures, sample receipt, chronology, reviewer notes, decision log, approved action, verification, limitations, and refresh trigger so another authorized engineer can reproduce the bounded result without retaining secrets or unnecessary production data.
Privacy and retention
Deployment Rollback Verification Delay Study asks whether rollback duration measures include destination verification and residual-state reconciliation rather than only the rollback command. Stage 13, privacy and retention, declares one production change linked to trigger, rollback decision, execution events, health observations, residual checks, and verified recovery as the observation unit and freezes population, system boundaries, local and UTC times, source versions, inclusion rules, exclusions, and intended decision before outcomes are reviewed. Record DO-R-1013-13 distinguishes system events, engineering interpretations, owner decisions, implementation evidence, and unresolved hypotheses. This prospective protocol reports no completed client outcomes and defines no universal performance benchmark.
At stage 13 for Deployment Rollback Verification Delay Study, include routine, missing-source, wrong-identity, conflicting-version, late-arriving, inaccessible-system, unavailable-owner, reversed, corrected, reverted, and reopened cases. Two reviewers classify a stratified subset independently using written anchors, preserve first-pass disagreement, and adjudicate against retained evidence. Report numerator, denominator, exclusions, null values, age bands, correction rate, disagreement, and residual cases together. Alternative explanations include configuration, case mix, cache state, clock drift, synchronization, sampling, ownership changes, release versions, seasonality, and selection effects. The ten public sources guide control design but do not inspect this system, establish causation, certify a provider, or authorize deployment. Preserve protocol, queries, codebook, fixtures, sample receipt, chronology, reviewer notes, decision log, approved action, verification, limitations, and refresh trigger so another authorized engineer can reproduce the bounded result without retaining secrets or unnecessary production data.
Reperformance test
Deployment Rollback Verification Delay Study asks whether rollback duration measures include destination verification and residual-state reconciliation rather than only the rollback command. Stage 14, reperformance test, declares one production change linked to trigger, rollback decision, execution events, health observations, residual checks, and verified recovery as the observation unit and freezes population, system boundaries, local and UTC times, source versions, inclusion rules, exclusions, and intended decision before outcomes are reviewed. Record DO-R-1013-14 distinguishes system events, engineering interpretations, owner decisions, implementation evidence, and unresolved hypotheses. This prospective protocol reports no completed client outcomes and defines no universal performance benchmark.
At stage 14 for Deployment Rollback Verification Delay Study, include routine, missing-source, wrong-identity, conflicting-version, late-arriving, inaccessible-system, unavailable-owner, reversed, corrected, reverted, and reopened cases. Two reviewers classify a stratified subset independently using written anchors, preserve first-pass disagreement, and adjudicate against retained evidence. Report numerator, denominator, exclusions, null values, age bands, correction rate, disagreement, and residual cases together. Alternative explanations include configuration, case mix, cache state, clock drift, synchronization, sampling, ownership changes, release versions, seasonality, and selection effects. The ten public sources guide control design but do not inspect this system, establish causation, certify a provider, or authorize deployment. Preserve protocol, queries, codebook, fixtures, sample receipt, chronology, reviewer notes, decision log, approved action, verification, limitations, and refresh trigger so another authorized engineer can reproduce the bounded result without retaining secrets or unnecessary production data.
Corrective-action mapping
Deployment Rollback Verification Delay Study asks whether rollback duration measures include destination verification and residual-state reconciliation rather than only the rollback command. Stage 15, corrective-action mapping, declares one production change linked to trigger, rollback decision, execution events, health observations, residual checks, and verified recovery as the observation unit and freezes population, system boundaries, local and UTC times, source versions, inclusion rules, exclusions, and intended decision before outcomes are reviewed. Record DO-R-1013-15 distinguishes system events, engineering interpretations, owner decisions, implementation evidence, and unresolved hypotheses. This prospective protocol reports no completed client outcomes and defines no universal performance benchmark.
At stage 15 for Deployment Rollback Verification Delay Study, include routine, missing-source, wrong-identity, conflicting-version, late-arriving, inaccessible-system, unavailable-owner, reversed, corrected, reverted, and reopened cases. Two reviewers classify a stratified subset independently using written anchors, preserve first-pass disagreement, and adjudicate against retained evidence. Report numerator, denominator, exclusions, null values, age bands, correction rate, disagreement, and residual cases together. Alternative explanations include configuration, case mix, cache state, clock drift, synchronization, sampling, ownership changes, release versions, seasonality, and selection effects. The ten public sources guide control design but do not inspect this system, establish causation, certify a provider, or authorize deployment. Preserve protocol, queries, codebook, fixtures, sample receipt, chronology, reviewer notes, decision log, approved action, verification, limitations, and refresh trigger so another authorized engineer can reproduce the bounded result without retaining secrets or unnecessary production data.
Limitations and refresh rule
Limitations. Deployment Rollback Verification Delay Study asks whether rollback duration measures include destination verification and residual-state reconciliation rather than only the rollback command. Stage 16, limitations and refresh rule, declares one production change linked to trigger, rollback decision, execution events, health observations, residual checks, and verified recovery as the observation unit and freezes population, system boundaries, local and UTC times, source versions, inclusion rules, exclusions, and intended decision before outcomes are reviewed. Record DO-R-1013-16 distinguishes system events, engineering interpretations, owner decisions, implementation evidence, and unresolved hypotheses. This prospective protocol reports no completed client outcomes and defines no universal performance benchmark.
At stage 16 for Deployment Rollback Verification Delay Study, include routine, missing-source, wrong-identity, conflicting-version, late-arriving, inaccessible-system, unavailable-owner, reversed, corrected, reverted, and reopened cases. Two reviewers classify a stratified subset independently using written anchors, preserve first-pass disagreement, and adjudicate against retained evidence. Report numerator, denominator, exclusions, null values, age bands, correction rate, disagreement, and residual cases together. Alternative explanations include configuration, case mix, cache state, clock drift, synchronization, sampling, ownership changes, release versions, seasonality, and selection effects. The ten public sources guide control design but do not inspect this system, establish causation, certify a provider, or authorize deployment. Preserve protocol, queries, codebook, fixtures, sample receipt, chronology, reviewer notes, decision log, approved action, verification, limitations, and refresh trigger so another authorized engineer can reproduce the bounded result without retaining secrets or unnecessary production data.
Evidence table
| Signal | What to inspect | Owner |
|---|---|---|
| Outcome | Acceptance evidence for the bounded task | Task reviewer |
| Control | Access, test, and approval boundary | Internal owner |
| Handoff | Open risks and next decision | Next owner |
Good distributed work is observable at the handoff: the result, evidence, limitations, and next owner are explicit.
Frequently asked questions
Does this report claim measured production outcomes?
No. It specifies a prospective protocol and does not present completed field results.
Why preserve reviewer disagreement?
Disagreement exposes ambiguous definitions before consensus hides them.
What authority does this study transfer?
None. Named client-side technical, security, privacy, and release owners retain consequential decisions.
Sources
Build the evidence packet
Freeze the decision, population, versions, and owners before collecting outcomes.
