Developer Offshore logoDeveloper OffshorePowered by StealthAgentsScope a developer
Table of contents

Developer Offshore research

Change failure rate as a contract health signal in offshore delivery

An analysis of change failure rate in offshore delivery: what the metric captures and hides, how to define the denominator, attribution across the boundary, contract use, gaming risk, and limits.

Use this report with the Research library and related developer guides to turn evidence into a bounded work brief.

Change failure rate as a contract health signal in offshore delivery

Key stats

  • A change failure rate is only meaningful with a written definition of what counts as failed
  • Change volume and backlog age must be read alongside the rate to detect avoidance
  • Attribute failures to contributing conditions on both sides of the engagement
  • Verify a sample of reported events against release notes and incident logs
  • Use the metric to open a process review, never as a standalone penalty

Key takeaways

  • Write down what counts as a failed change and keep the definition stable.
  • Count only changes that reached production and record the source of the list.
  • Review two real failures with the provider and name both sides' contributions.
  • Watch change volume and backlog age so a low rate does not hide avoidance.
  • End each review with one process change and a date to verify it.

Question and scope

This report examines change failure rate as a contract health signal in outsourced software delivery. The unit is one production change delivered by an offshore pod for a buyer-owned application. The scope covers application releases, configuration changes, and rollback events, and the teams that review them across time zones. It does not define a universal target or replace the buyer's own reliability goals. The boundary matters because the metric is often quoted without saying what was counted.

The question is whether change failure rate tells a buyer something useful about the delivery relationship. A low rate can reflect stable engineering or a team that avoids risky work. A high rate can reflect a fragile system or a change process that never matured. The analysis treats the metric as a conversation starter rather than a score. Used carefully, it can reveal where the change process needs attention on both sides of the engagement.

What the metric captures

Change failure rate is the share of changes that require remediation, such as a rollback, a hotfix, or an emergency patch. It captures the outcome after a change reaches production, not the effort spent preparing it. That focus makes it useful for buyers who care about disruption rather than activity. It also connects the metric to events the buyer can observe directly, such as a release that had to be undone. The metric is therefore easier to verify than a self-reported measure of quality.

The metric works best when the definition of a failed change is written down. Teams disagree about whether a small follow-up fix counts as a failure. Some count only rollbacks, while others include any urgent correction within a fixed window. Write the rule once and apply it consistently across releases. A clear rule makes the trend meaningful even when the absolute number is small. It also makes the metric comparable between two review periods.

What the metric hides

A change failure rate says nothing about the size or importance of the changes being measured. A team that ships one trivial change a month can look better than a team that ships a complex migration. The metric also hides near misses that were caught before customers noticed. Those near misses often carry the most useful learning because the team can still see the warning signal. A rate without context can reward caution and punish useful progress.

The rate can also be distorted by how failures are recorded. A rollback that nobody labels as a rollback disappears from the numerator. A hotfix that is recorded as a normal release does the same. Buyers should ask to see the raw event list rather than only the summary. Sampling a few events against the release notes is a quick way to test whether the record is complete. A metric that depends on self-reporting needs a verification step.

Defining the denominator

The denominator matters as much as the numerator. Count only changes that reached production, and exclude routine restarts or scheduled maintenance that carries no code change. Decide whether configuration changes belong in the same population as application releases. Mixing unlike changes makes the rate hard to compare over time. The definition should be short enough to apply consistently and specific enough to reproduce.

Keep the population stable for at least two quarters before drawing a trend. A change in the definition can create a false improvement or decline. Record the definition beside the number so a later reviewer can see what was counted. Include the source of the release list, because a manually maintained list will drift. A metric without its definition is a number without a meaning.

Attribution across the boundary

A failed change is rarely the responsibility of one person or one team. The change may have been specified by the buyer, implemented by the provider, reviewed by both, and deployed through a shared pipeline. Attribution should describe the contributing conditions rather than assign blame. That framing keeps the review honest and makes corrective action possible. It also reflects the reality that the buyer controls many of the constraints the pod works within.

Separate the cause from the trigger. A missing test may be the cause, while a specific release was the trigger. The provider may own the test gap, and the buyer may own the acceptance criteria that omitted the case. Naming both sides makes the next change safer. A review that stops at the provider's name will fix one instance and leave the underlying condition in place.

Using it in a contract review

Bring the metric to a quarterly review with the raw events and the agreed definition. Ask the provider to walk through two recent failures and what changed afterward. Look for patterns across events rather than a single dramatic incident. A provider that can explain the pattern is easier to trust than one that only reports a rate. The conversation should also include the buyer's own contribution to the conditions.

Connect the metric to the change process rather than to a penalty. A contract that punishes every failure encourages teams to hide them. A contract that requires disclosure and follow-through encourages learning. The review should end with one or two process changes and a date to check them. That follow-through is the part that turns a metric into a control.

Avoiding gaming

Any metric can be gamed when it becomes a target. A team can reduce reported failures by redefining a rollback as a planned release. It can also avoid risky changes and let the backlog grow. Neither behavior improves the product. The buyer should watch the change volume and the backlog age alongside the failure rate. A sudden drop in failures with a falling change volume deserves a question.

Sample the release notes against the incident log. If the notes are much cleaner than the log, the metric is probably incomplete. Ask the provider how failures are detected and who records them. A metric that depends on self-reporting needs a verification step. The verification does not need to be exhaustive, but it should be regular enough to keep the record honest.

Limitations

This analysis cannot set a target rate for a specific application. The acceptable rate depends on the system, the customer promise, and the cost of a failed change. It also cannot prove that a low rate reflects good engineering. The useful contribution is a structure for defining, verifying, and discussing the metric. A buyer should adapt the structure to the risk profile of the application.

Small teams and low release volumes make rates unstable. A single failure can move the number sharply, and a quiet quarter can look like improvement. Buyers should use the metric as one input among several. It should never be the only evidence in a contract decision. The surrounding evidence matters more than a precise-looking percentage.

A worked review agenda

A useful quarterly review starts with the agreed definition and the list of production changes. The provider presents two failures, one that was caught quickly and one that caused customer impact. For each, the team describes the contributing conditions, the detection signal, and the corrective action. The buyer then describes any constraint it owns that made the change harder, such as a late acceptance decision or an unclear owner. This structure keeps the review specific and shared.

The second half of the review looks at the trend rather than the individual events. Compare the rate with the change volume and the backlog age for the same period. Ask whether a lower rate came from fewer risky changes or from better preparation. Ask whether a higher rate came from a fragile area that deserves investment. End with one process change, one owner, and a date to check the result. That ending is what separates a review from a status meeting.

Conclusion

Change failure rate is useful when the definition is explicit, the events are visible, and the review focuses on patterns. It gives a buyer a shared language for disruption without turning every failure into a dispute. Pair it with change volume and backlog age to avoid rewarding caution over progress. The metric then supports a real conversation about delivery health rather than a scorekeeping exercise.

The strongest signal is a provider that can explain a failure, show the corrective action, and describe what changed in the process. That behavior is more valuable than a perfect rate. Use the metric to start the review, not to end it. A contract health signal is only as good as the conversation it produces. The conversation is where the buyer and provider can actually improve the system.

Evidence table

Signals, inspection points, and accountable owners
SignalWhat to inspectOwner
OutcomeAcceptance evidence for the bounded taskTask reviewer
ControlAccess, test, and approval boundaryInternal owner
HandoffOpen risks and next decisionNext owner
Good distributed work is observable at the handoff: the result, evidence, limitations, and next owner are explicit.

Frequently asked questions

What is a good change failure rate?

There is no universal target. The acceptable rate depends on the system, the customer promise, and the cost of a failed change. Compare the trend against your own baseline.

Should configuration changes count?

Decide explicitly and apply the rule consistently. If configuration changes can disrupt customers, they belong in the population or in a separate parallel measure.

How do we stop the provider from hiding failures?

Ask for the raw event list, sample it against release notes and incident logs, and make disclosure part of the review rather than a penalty.

Sources

  1. DORA, Research program
  2. NIST SP 800-40 Rev. 4, Guide to Enterprise Patch Management Planning
  3. CISA Known Exploited Vulnerabilities Catalog
  4. NIST SP 800-218, Secure Software Development Framework

Related research