How does a bad release reach production when tests are green?

What does a green test result actually establish?

A green test result establishes that specified checks passed for particular inputs and conditions. It does not independently establish that production will run the same files, use appropriate configuration or encounter only covered traffic. Some incidents reflect missing tests; others reflect a broken connection between evidence and deployment. Adding more tests alone does not fix deploying a different package. First identify the object the test result describes, the conditions under which it was produced and the deployment claim the team is trying to make from it.

Consider a hypothetical workflow that builds a commit in a test environment and rebuilds it on the production host. Dependency resolution, build-time configuration or toolchain differences may change the second output. A shared commit identity does not make the artifacts identical. This article is not an incident report or a description of Velocize's current deployment machinery. It examines how to connect source changes, observed test outcomes and the executable output actually used in production through evidence a reviewer can check.

Why distinguish commit, artifact and environment identity?

A commit identifies source state, an artifact digest identifies output bytes and an environment identity identifies the deployment target. A release record can also include build inputs, tool versions, the tested artifact digest and configuration revision. An immutable digest makes approval more concrete than a movable 'latest' tag. Actually verify the bytes against that digest: trusting a filename is insufficient when another file can be uploaded under the same name or a tag can be moved after approval.

This does not mean identical bytes necessarily behave identically everywhere. Runtime settings, data, external services and permissions can still differ. Building once and promoting the same bytes addresses output identity; runtime suitability needs additional evidence. If an environment requires rebuilding, expose that as a new artifact with a new identity and state which checks were repeated for it. Do not automatically attach the earlier artifact's test result to changed output merely because the source revision has the same label.

What does build provenance establish, and what does it not establish?

SLSA provenance is a reference for structured information about how an artifact was produced. Its verification guidance goes beyond checking that a provenance document exists. A design can validate the artifact digest, trusted builder identity and expected source relationship. The example here is not a SLSA implementation or a claim of achieving a SLSA level. Correctly established origin does not establish correct application behavior; an authentic build can still contain a functional defect. Provenance and quality evaluation answer different questions.

Evidence needs its own trust arrangement. A freely editable JSON file beside a freely editable package is a circular authority. Identify the evidence producer, who can change its records and which acceptance policy applies. Bind approval to the artifact and destination it was intended to authorize; later commits or packages should not silently inherit it. A reviewer can then inspect which checks applied to which immutable output, rather than treating the color of a pipeline badge as a complete deployment authorization.

Section references: SLSA v1.2 — Provenance ↗SLSA v1.2 — Verifying artifacts ↗

How can the artifact-to-evidence relationship be tested?

The standalone Node.js example checks agreement between artifact bytes, the approved digest, the tested digest and the target configuration revision. Negative cases independently change the artifact, test evidence and configuration. It generates trusted inputs in one file for demonstration convenience, which does not provide real signing or identity verification. A production gate must obtain evidence from a trusted source, identify its policy version and retain failure details for review. The sample isolates identity binding instead of claiming to implement a complete secure release service.

Not every check runs at the same stage. Unit tests may be associated with source and build inputs, package-level tests with an artifact digest and production health checks with a running release identity. Keep those subjects explicit. Rebranding source-only unit tests as artifact tests creates misleading evidence. The simplified example assumes an actual package-level acceptance check exists and prevents its result being silently attached to a different package. A real evidence model should preserve this distinction rather than forcing every test into an inaccurate common label.

const { createHash } = require('node:crypto');
const assert = require('node:assert/strict');
const digest = (bytes) => createHash('sha256').update(bytes).digest('hex');
const artifact = Buffer.from('illustrative-build-output-v1');
// These inputs must come from trusted evidence in a real deployment.
const approved = {
  artifactDigest: digest(artifact),
  environment: 'production',
  configRevision: 'config-7'
};
const evidence = { testedDigest: digest(artifact), testsPassed: true };
function gate(bytes, approval, tests, target) {
  assert.equal(digest(bytes), approval.artifactDigest, 'artifact mismatch');
  assert.equal(tests.testedDigest, approval.artifactDigest, 'untested artifact');
  assert.equal(tests.testsPassed, true, 'tests failed');
  assert.equal(target.environment, approval.environment, 'wrong environment');
  assert.equal(target.configRevision, approval.configRevision, 'config changed');
}
const target = { environment: 'production', configRevision: 'config-7' };
gate(artifact, approved, evidence, target);
assert.throws(() => gate(Buffer.from('different-build'), approved, evidence, target));
assert.throws(() => gate(artifact, approved, { ...evidence, testedDigest: 'other' }, target));
assert.throws(() => gate(artifact, approved, evidence, { ...target, configRevision: 'config-8' }));
console.log('Approved tuple accepted; changed artifact, evidence and config rejected.');

How can the same artifact fail with different configuration?

An incorrect service endpoint, missing feature flag, different timeout or insufficient permission can send tested code down a different production path. Bind a non-secret configuration revision and destination to the release record. There is no need to publish secret values in reports or public hashes; use access-controlled references and version identities. Startup validation catches missing required fields, but does not independently prove that a dependency is reachable or that its credentials authorize the intended operation.

GitHub's environment documentation provides one example of deployment protection and target association, subject to account and configuration conditions. An approval button cannot replace review of technical evidence. Make the artifact, destination, configuration and migration plan visible to the reviewer. Define what happens when configuration changes while a release is waiting. Otherwise an approval given hours earlier can be used for an installation whose effective settings have changed without the release record acknowledging the difference.

Section references: GitHub — Deployments and environments ↗

Why does rolling back the application not roll back the database?

Restarting old application code does not restore a column removed by a newer migration. New-format records and external side effects likewise cannot be reversed by switching application files. A migration can separate expansion, coexistence and contraction: add backward-compatible structure, introduce new code and backfill, then remove old structures after confirming the state of running versions and the rollback window. The key question is compatibility over time, not whether a migration command completed successfully on one empty test database.

For a new status field, define how old and new applications read the same record and how conflicting updates are handled if dual writes are used. Backfill should have an explicit repeatability policy and expose partial failure. This is not a universal automatic migration recipe: transaction behavior, locking and volume constraints vary by database. Acceptance should exercise representative populated data, mixed application versions and interrupted migration rather than testing only a clean schema with one application instance.

Which metrics matter during staged rollout?

Comparing error counts alone can be misleading. In a hypothetical control group, 40 errors among 20,000 requests is 0.2%. Eight errors among 1,000 requests in the new-release group is 0.8%. Fewer errors in absolute terms does not mean a lower failure rate; the difference is 0.6 percentage points. These are illustrative numbers, not Gokbilge production measurements or a claim of statistical significance. A decision needs traffic composition, observation window, sample size and an agreed operating threshold.

Break out critical operations rather than relying only on a global average. A rarely used endpoint might receive ten requests and fail one, with the overall rate concealing that path. Observe latency distribution, queue buildup and background-job success as well. Define a hold state when evidence is insufficient instead of advancing solely because a timer expired. Staged traffic routing must actually exist in the infrastructure; displaying a monitoring graph does not establish that a canary rollout has been implemented.

What should a real rollback exercise cover?

A rollback target should include the previous artifact digest, compatible configuration, database state, queued work and external effects. If the old worker cannot read jobs created by the new release, reverting web traffic may leave background processing broken. Queue formats and task versions therefore belong in the compatibility contract. After rollback, verify representative end-to-end work rather than only checking that the homepage loads. Define which effects can be reversed, which require compensation and which remain outside an automatic rollback procedure.

Measure recovery duration, but do not make it the only success criterion. Verify the running artifact identity, cessation of new failures, progress of queued work and data consistency separately. Derive time targets from actual operational requirements. After an incompatible schema change, automatically starting old code may be worse than halting or performing a controlled forward repair. Decide the conditions for each recovery path before the release begins, and preserve the exercise results so the plan can be checked when dependencies or migration patterns change.

• Running artifact, approval and package-test evidence refer to the same output.
• Configuration changes and schema transitions are explicitly versioned.
• Rollback is exercised with data, queue and external-effect boundaries.
• Insufficient evidence leads to a defined hold or stop decision.

Which evidence should a Velocize evaluation request?

Velocize is publicly positioned as a beta product for lifecycle traceability, evidence-centered delivery and readiness governance. The gate, SLSA references and staged-rollout scenario here are not claims of ready-made integrations or a verified assurance level. Ask which relationships can be demonstrated from requirement to test, test to artifact and artifact to deployment. Collecting existing evidence, producing additional evidence and changing deployment infrastructure may be distinct work packages, each with its own acceptance conditions.

A bounded first exercise can use one service and one release type. Review a sample artifact, test output, approval record and failed recovery scenario to identify missing links. Industrial software engineering is relevant to integrations, while Batur is relevant to collecting release-related operational events; no ready-made connector between them is assumed. The useful outcome is a reproducible explanation of what was deployed, which evidence supported it and which limits remained. That makes the next release review more concrete without confusing additional paperwork with demonstrated readiness.