NNT Invisibility to the Clinical Eye

Doctors often claim eloquently and pretentiously to be “having great results” with their treatment choices. However, they do not realize it is impossible to perceive effectiveness from clinical experience in many of the circumstances. Experience is blind to prognostic effectiveness.

The theoretical framework for this observation comes from Daniel Kahneman’s discussion about prediction under uncertainty. Previously, Meehel demonstrated that statistical predictions beat expert opinion in 60% of professional situations. Kahneman proposed that expert opinion is mediated by heuristics, therefore vulnerable to cognitive bias.

As opposed, the work by psychologist Gary Klein with firefighters recognized the value of experience in building accurate expert opinion. To reconcile both views, Kahneman and Klein worked together and proposed that in situations of large, consistent and immediate effects, professionals are able to accumulate experience to predict consequences of their choices. 

It is the case for pain control by morphine. There is a large, consistent and immediate effectiveness, easily perceived. Reproducibility and time proximity between cause and effect makes it easier to link treatment and improvement. It is how players learn the game. A wrong move in chess almost always results in a quick defeat. 

As opposed, prognostic effectiveness is a typical situation of uncertainty. First, it refers to the future, so the result is not immediate. Second, the result of a prognostic treatment is probabilistic, not deterministic. Thus, it is inconsistent and of modest effect size. To understand it we should review the philosophical concept of number needed to treat: and the sense of humility provided by it.

The Impossible to Perceive NNT

The argument based on “clinical experience” is often used in cases of great uncertainty and questionable scientific evidence. And it is precisely in those situations that it is impossible to notice the benefit (or harm) from clinical observation.

The blindness clinical perception occurs when intervention happens in the present in order to reduce risk in the future: it is not an immediate effect. Second, when it comes to the future, the benefit is much more uncertain. These are probabilistic situations that suffer from the uncertainty of the “number needed to be treated”.

In these circumstances, practice does not increase our decision-making ability. In contrast, trying to create experience-based concepts of effectiveness is a good example of confirmation bias. It is about using experience is a way of unlearning on the basis of practical illusion.

We must understand the difference between the “prevalent NNT” and “incident NNT”. In prevalent situations (symptom improvement), the NNT tends to approach 1. In incident situations (future improvement), the NNT tries to distance itself from 1 towards infinity. The incident NNT is the one hard to perceive. 

It happens for two reasons. In prevalent situations all patients are in need of treatment. But in preventing future events, only a small portion of patients really need treatment: those who will suffer the future outcome. But as we do not know who they are, we treat them all and many who would not need to end up being treated, increasing NNT to get a benefit. Second, relieving a symptom is usually easier than preventing an outcome, so the effect size of present symptoms treatments is greater than preventing future events.

A great current example is the (increasingly common) claim by cardiologists that their patients with chronic heart failure have benefited from the new medication called "Entresto." This clinical argument is usually brought when one questions the validity of that trial based on the asymmetric control group of the PARADIGM trial.

That study concluded that a new drug (sacubitril) was effective from a severe adjunctive therapy asymmetry between the drug and placebo groups. The study showed a reduction in the combined death or hospitalization due to heart failure, two future outcomes.

The methodological fallacy of the study generates uncertainty. To compensate for this uncertainty,  faithful cardiologists have used their eloquence, and use the statement, "In my experience, I've had a great response to Entresto."

Statement Mathematical Analysis

The NNT derived from the PARADIGM-HF trial is 21 to reduce the combined death or hospitalization, a benefit that would be of great magnitude. But how could a doctor perceive an NNT of 21 in clinical practice?

Suppose she had 21 patients using Entresto and 21 patients without using Entresto. In 20 of each group the evolution would be the same, only the difference of evolution in the twenty-first patient of each group would take place. It is just imperceptible in everyday life.

This is the “fallacy of the prognostic clinical impression”. It is impossible to perceive 1 in 21 “with the naked eye” or “with the clinical eye”.

If we think of 100 patients treated in each group, the difference between the groups would be only 4 patients. 4 out of 100 patients: how to perceive the phenomenon depicted in the figure below?

So let's stop using this argument, which borders on ridiculousness.

One statistician once told the story of his two adopted children. The daughter was a child who was adopted in China. The son was American. One day the girl said, “Girls come from China, boys come from the United States.” The innocence of the child shows a trace of the human mind: to conclude from small samples.

Does it seem cartoonish for the girl to have completed this? But that's exactly what doctors do when, after three successful consecutive experiments, they find that something works.

Behind this is the confirmation bias. Because clinical practice is not an experimental scientific environment, any choice is based on the belief in benefit. If I prescribe something, it is because I believe in it. In fact, it would be unethical to prescribe something I do not believe. Therefore, clinical practice is a naturally believing environment, predisposed to confirmation bias. Starting from belief and looking at the world around us, we will fall into the cognitive trap of seeking evidence for what we believe. We will record in our memory patients who evolved according to our belief and validate our conduct without symmetrically computing patients who rejected our belief.

Complicating matters, clinical practice is fraught with performance bias. The tendency of the concerned physician who changes the patient's treatment to Entresto is to make further improvements in his conduct, to adjust the diuretic, to better guide the diet. Therefore, even if it were possible to perceive the result, it would be impossible to know what caused that result.

It is different in the scientific environment in which we start from skepticism and reject the null hypothesis only when the evidence is far beyond chance and bias-mediated effects.

Conclusion

Experience is essential for the individual application of a scientific concept, in the perception of patient values, in shared-decision making, in the generation of a diagnostic hypothesis. But we cannot trivialize and undermine the value of clinical experience by its caricature and inappropriate use.

Clinical experience is blind to the effectiveness of prognostic management. We are blind to the prognostic NNT, but even blinder to the limitation of our own experience.

The story of how Entresto got approved for the Brazilian Public Health System

In one of the most skillful evidence-based medicine hijacking movements, Entresto is approved to be financed by the government for use in patients with heart failure in the Brazilian Unified Health System.

The hijacking of a concept occurs when it is transformed into its own caricature, to the point of breaking the barrier of rationality, supporting paradoxical application of the original concept itself.

Evidence-based medicine proposes that reliable scientific concepts be used in the reasoning process of medical decision. In the process of “hijacking", the evidence dominates rationality. The recommendation becomes hostage of the evidence. And by mastering the process, the evidence is no longer questioned as to its reliability.

We will explain in this post how the hijacking occurred in the case of Entresto and our public health system. Entresto is the trade name that contains Sacubitril. Sacubitril inhibits the enzyme neprilysin, which is responsible for the breakdown of good molecules such as natriuretic peptide and bradykinin. Thus, by inhibiting neprilysin, sacubritil increases the concentration of these good molecules, which have vasodilating and natriuretic action.

Therefore, there is biological plausibility for benefit, being justifiable to test for clinical efficacy. So there comes the PARADIGM-HF randomized clinical trial, published in 2014 in the New England Journal of Medicine.

PARADIGM-HF, a disingenuous asymmetry of the control group

Following the successful demonstration of efficacy of angiotensin inhibitors, aldosterone antagonists and beta-blockers in heart failure, molecules with different neurohumoral effects over the past decade have failed to demonstrate efficacy, leading to successive negative clinical trials. Amid the perception that we could have reached a pharmacological "sealing" in heart failure, an idea seemed to have sprung up in the minds of "researchers": inventing a new "scientific" method.

The original scientific method was proposed by Ronald Fisher and consists of an experimental innovation: the existence of a control group. This method was translated into biomedicine by statistician Bradford Hill. In his seminal Lancet article, Hill wrote:

“The essence of the method lies in the determination that we are really comparing like with like. We must endeavor to equalize the groups we compare in every possibly influential respect, except the one factor at issue. ”

Following this logic, clinical trials compare new drug versus placebo with an equal treatment background in both groups. This allows testing of the intrinsic efficacy of the new molecule, as nothing other than the mais treatment is different between the two groups.

The design of the PARADIGM-HF study violates the very definition of control group, as if it created a “new” scientific method. And this method brings the advent of a double molecule: the new drug is combined with a proven effective molecule, resulting in a name that connotes pharmacological ingenuity. LCZ696 is born, the fusion of sacubitril with a proven vasodilator at full dose (valsartan 320 mg daily). In this new method, instead of testing the new molecule, we are testing the combination of a proven effective molecule with a molecule of unknown benefit. And now comes the greatest idea: the control group consisted of another vasodilator, enalapril at a more modest dose.

We need to understand the difference between what was done and the traditional scientific method: sacubitril versus placebo was not compared taking into account the rest of the treatment being equal between groups, as Bradford Hill (like with like) suggested. In reality, sacubitril versus placebo was compared, being the best adjuvant treatment in the sacubitril group. And to ensure that the adjuvant was indeed better, the maximum dose valsartan molecule was glued to Sacubitril (LCZ 696) and the enalapril dose in the control group was frozen at 20 mg daily.

If sacubitril-valsartan was compared with placebo-valsartan there would be no bias. 

So why didn't they do this and prefer the freakish idea of ​​setting sacubitril-valsartan maximum dose versus enalapril half dose?

The violation of the scientific method of the PARADIGM-HF case is so gross that it is not on any clinical trial bias risk checklist. For this reason, if we apply any critical appraisal tool to the PARADIGM-HF study, this bias will not be detected. Thus, a pseudo low risk of bias study was created.

The commotion

A commotion improves the predisposition of people believing in the unreal.

Following successive reductions in morbidity and mortality in heart failure, such as vasodilation and blockage of the angiotensin-aldosterone system in the 1980s and 1990s, beta-adrenergic blockers in the 1990s, resynchronizers and implantable defibrillators in the 2000s, cardiologists expressed concern and frustration about the absence of novelty in recent years. This background that I call “commotion-induced belief”.

Even better if the commotion is also triggered in the patients. The best scenario is a synergism between the physician's enthusiasm for prescribing a medication and the patient's desire to receive that prescription. Therefore, there was a need to convince patients of the importance of fighting with new weapons for heart failure.

Suddenly, Novartis, maker of Entresto, develop a desire to educate the public about the problem of heart failure. Folders were prepared and placed in the offices waiting rooms, making the public aware of the seriousness of the problem. The strategy of causing fear is used, a feeling that makes any kind of solution desired.

A caricature of the strategy was the event called Weak Heart, sponsored by Novartis, and organized by nations' main newspaper, Folha de São Paulo. The event was open to the general public. In that event, heart failure was characterized as an "epidemic" by some of the experts. Strange to use the word epidemic, which means "transient disease that simultaneously attacks large numbers of individuals in a particular locality." Sounds sensational.

In fact, heart failure is a serious problem, but epidemiologically morbidity and mortality of heart failure has been decreasing over the last decades in Brazil and worldwide. The question remains: why wake up to this problem right now? Doesn’t it seem unusual the temporal coincidence with the public consultation of Entresto? Or is it no coincidence?

The Recommendation

CONITEC is the technical body that evaluates requests for drug cover by our universal health system. Novartis requested the approval of Entresto and CONITEC recommended not to implement Entresto.

But CONITEC bases the recommendation on the cost of the drug, while recognizing the efficacy "demonstrated" by PARADIGM-HF. Failing to offer something very good for lack of money is different from not to offer something of dubious benefit.

The Public Consultation

After the technical rejection of the incorporation of Entresto, nothing like a public consultation that democratizes the discussion. Consultation with the public is in the protocol of CONITEC. In summary, out of 185 technical-scientific manifestations, in which health professionals predominate, there was no record of opinion against Entresto. Almost all opinions were positive and a minority were neutral. But the good thing is the 1,956 contributions based on "experience or opinion”: 1,797 in favor. Regardless of a possible selection bias, the public consultation was clearly in favor of Entresto.

However, a question remains: does democracy apply to drug incorporation?

The Big Turn

After the public consultation, the contrary opinion of CONITEC becomes a favorable opinion.

Apparently, based on the interaction between a technical opinion that recognizes its efficacy (although not recommending) and a democratic manifestation in favor of the drug, CONITEC position was reversed. 

To mitigate, this decision brought with it the restriction of use in patients over 75 years, simulating a judicious decision.

It is noteworthy that this restriction is not based on a correct analysis of the evidence, since in the subgroup analysis there was no interaction of age and drug efficacy. The scientific concept that “interaction is a rare phenomenon” and that subgroups should not be analyzed for the significance of interaction was not considered. This restriction promotes a scientific injustice. If it is to be released for use by a 74-year-old, it should be  released for the 76-year-old as well.

Epilogue

A hijacking of evidence-based medicine. A paradigm that proposes evidence as a guiding concept is used to promote the influence of pseudo-evidence on recommendation of a new drug. A clinical trial that violates the essence of the scientific method (control group) promotes the approval of a high-cost drug by the Brazilian Unified Health System.

It should be emphasized that our system universally supply more than 200 million people in a poor country. We need to be proud of this system, try to preserve it and increase its efficiency.

But what I am discussing here is not just about the monetary cost of this drug. There is a greater cost, which is the transformation of medicine into a profession guided by studies of questionable validity. It is one thing to say that we need evidence, but another is to have evidence fabricated and published based on conflicts of interest.

We cannot say that it is proved that Entresto is not superior to traditional treatment. But we cannot say it is either. It was just not tested.

The paradigm is hijacked… patients have their drugs switched on a questionable basis… an optimism bias predominates, followed by the confirmation bias that makes us realize “how well the drug is working”, although an NNT of 20 should be imperceptible to the clinical eye. And so we live in an immediate, fantasy world that neglects a careful analysis of the evidence as the basis of scientific thinking. The damage to the scientific culture is greater than the monetary waste.

We need nonconformity with tribal positions, where prevalence of opinion is confused with evidence. Rationality must prevail over the opinion of the masses who stand up for modest sized results in late-breaking clinical trial sessions, turning medical specialties into religion cults.

We need to realize the difference between invention and innovation.