How do you tell a broken process from a bad run?
Why this matters. Both look identical from inside, the responses are opposite, and getting it wrong is expensive in either direction.
Something has stopped working. Either the approach no longer fits conditions, or it fits fine and you are in the variance any approach produces. Change a working process and you have destroyed something; keep a broken one and you are paying to learn nothing.
Why the question is genuinely hard
Practitioner testimony, and the most useful in this course: Michael Mauboussin's work on separating skill from luck argues for measurable proxies rather than outcomes, precisely because outcome sequences in high-variance domains cannot distinguish the two at small samples. Your sample is small. It will always be small.
And Nassim Taleb's Fooled by Randomness is shelved beside every practitioner in this domain for the same reason — a run that looks like a verdict is usually a path.
What can actually be checked
Not the outcomes. The inputs:
- Are the conditions the process assumed still present, stated as observations rather than impressions?
- Has your execution drifted — is the process being run as written?
- Has anything changed that you decided in advance would matter?
A process failing while being executed as designed under the conditions it assumed is a different object from a process that has quietly stopped being followed. The second is far more common and is invisible without a record.
A fourth input, written before the run. The hit rate and sizing the process was designed with imply a worst ordinary losing run and a worst ordinary drawdown; Technical Analysis computes both, in How can a system that wins 40% of the time make money? and Why is a 50% loss not undone by a 50% gain?. Write the two numbers beside the process, and what crossing them triggers: a dated review of the three inputs above, not a redesign. Without that line a drawdown the process was built to survive reads as a verdict. Crossing the line opens the question, and the count decides how much outcomes can add to it: at twenty decisions a year, almost nothing for years; at five hundred, something within a quarter.
The artefact
A change log: every modification to your approach, dated, with the reason and what you expected it to do.
Without it, a slow accumulation of small adjustments is indistinguishable from a stable process — and you will attribute results to a method you are no longer running. With it, "did I change something?" is a question with an answer.
Try it now
Write down every change you have made to how you decide in the last six months. Whatever you cannot date, you cannot attribute.