A new version can be built, be ready, carry the right bytes, and still not get all the traffic, for example when traffic is pinned to an older revision. From the terminal that can look like a successful deploy.
So our deploy script checks two things before it calls a deploy done. If the running revision doesn't reference the exact image it just built, it stops: "Do not report this as shipped." If the new revision isn't serving 100% of traffic, it stops and prints: "Nothing shipped to users; do not report this as deployed."
Every day a separate job also checks that production runs exactly what master says. A red run opens an issue.
How code reaches the live store, step by step: x.com/con_golias/status/2121352219635159234
