Doctors often claim eloquently and pretentiously to be “having great results” with their treatment choices. However, they do not realize it is impossible to perceive effectiveness from clinical experience in many of the circumstances. Experience is blind to prognostic effectiveness.
The theoretical framework for this observation comes from Daniel Kahneman’s discussion about prediction under uncertainty. Previously, Meehel demonstrated that statistical predictions beat expert opinion in 60% of professional situations. Kahneman proposed that expert opinion is mediated by heuristics, therefore vulnerable to cognitive bias.
As opposed, the work by psychologist Gary Klein with firefighters recognized the value of experience in building accurate expert opinion. To reconcile both views, Kahneman and Klein worked together and proposed that in situations of large, consistent and immediate effects, professionals are able to accumulate experience to predict consequences of their choices.
It is the case for pain control by morphine. There is a large, consistent and immediate effectiveness, easily perceived. Reproducibility and time proximity between cause and effect makes it easier to link treatment and improvement. It is how players learn the game. A wrong move in chess almost always results in a quick defeat.
As opposed, prognostic effectiveness is a typical situation of uncertainty. First, it refers to the future, so the result is not immediate. Second, the result of a prognostic treatment is probabilistic, not deterministic. Thus, it is inconsistent and of modest effect size. To understand it we should review the philosophical concept of number needed to treat: and the sense of humility provided by it.
The Impossible to Perceive NNT
The argument based on “clinical experience” is often used in cases of great uncertainty and questionable scientific evidence. And it is precisely in those situations that it is impossible to notice the benefit (or harm) from clinical observation.
The blindness clinical perception occurs when intervention happens in the present in order to reduce risk in the future: it is not an immediate effect. Second, when it comes to the future, the benefit is much more uncertain. These are probabilistic situations that suffer from the uncertainty of the “number needed to be treated”.
In these circumstances, practice does not increase our decision-making ability. In contrast, trying to create experience-based concepts of effectiveness is a good example of confirmation bias. It is about using experience is a way of unlearning on the basis of practical illusion.
We must understand the difference between the “prevalent NNT” and “incident NNT”. In prevalent situations (symptom improvement), the NNT tends to approach 1. In incident situations (future improvement), the NNT tries to distance itself from 1 towards infinity. The incident NNT is the one hard to perceive.
It happens for two reasons. In prevalent situations all patients are in need of treatment. But in preventing future events, only a small portion of patients really need treatment: those who will suffer the future outcome. But as we do not know who they are, we treat them all and many who would not need to end up being treated, increasing NNT to get a benefit. Second, relieving a symptom is usually easier than preventing an outcome, so the effect size of present symptoms treatments is greater than preventing future events.
A great current example is the (increasingly common) claim by cardiologists that their patients with chronic heart failure have benefited from the new medication called "Entresto." This clinical argument is usually brought when one questions the validity of that trial based on the asymmetric control group of the PARADIGM trial.
That study concluded that a new drug (sacubitril) was effective from a severe adjunctive therapy asymmetry between the drug and placebo groups. The study showed a reduction in the combined death or hospitalization due to heart failure, two future outcomes.
The methodological fallacy of the study generates uncertainty. To compensate for this uncertainty, faithful cardiologists have used their eloquence, and use the statement, "In my experience, I've had a great response to Entresto."
Statement Mathematical Analysis
The NNT derived from the PARADIGM-HF trial is 21 to reduce the combined death or hospitalization, a benefit that would be of great magnitude. But how could a doctor perceive an NNT of 21 in clinical practice?
Suppose she had 21 patients using Entresto and 21 patients without using Entresto. In 20 of each group the evolution would be the same, only the difference of evolution in the twenty-first patient of each group would take place. It is just imperceptible in everyday life.
This is the “fallacy of the prognostic clinical impression”. It is impossible to perceive 1 in 21 “with the naked eye” or “with the clinical eye”.
If we think of 100 patients treated in each group, the difference between the groups would be only 4 patients. 4 out of 100 patients: how to perceive the phenomenon depicted in the figure below?
So let's stop using this argument, which borders on ridiculousness.
One statistician once told the story of his two adopted children. The daughter was a child who was adopted in China. The son was American. One day the girl said, “Girls come from China, boys come from the United States.” The innocence of the child shows a trace of the human mind: to conclude from small samples.
Does it seem cartoonish for the girl to have completed this? But that's exactly what doctors do when, after three successful consecutive experiments, they find that something works.
Behind this is the confirmation bias. Because clinical practice is not an experimental scientific environment, any choice is based on the belief in benefit. If I prescribe something, it is because I believe in it. In fact, it would be unethical to prescribe something I do not believe. Therefore, clinical practice is a naturally believing environment, predisposed to confirmation bias. Starting from belief and looking at the world around us, we will fall into the cognitive trap of seeking evidence for what we believe. We will record in our memory patients who evolved according to our belief and validate our conduct without symmetrically computing patients who rejected our belief.
Complicating matters, clinical practice is fraught with performance bias. The tendency of the concerned physician who changes the patient's treatment to Entresto is to make further improvements in his conduct, to adjust the diuretic, to better guide the diet. Therefore, even if it were possible to perceive the result, it would be impossible to know what caused that result.
It is different in the scientific environment in which we start from skepticism and reject the null hypothesis only when the evidence is far beyond chance and bias-mediated effects.
Conclusion
Experience is essential for the individual application of a scientific concept, in the perception of patient values, in shared-decision making, in the generation of a diagnostic hypothesis. But we cannot trivialize and undermine the value of clinical experience by its caricature and inappropriate use.
Clinical experience is blind to the effectiveness of prognostic management. We are blind to the prognostic NNT, but even blinder to the limitation of our own experience.
No comments:
Post a Comment