• English
  • Español

Testing for excess statistical significance

Topic category Statistical methods and meta-analysis
Date and Location
Date: 
Saturday 22 October 2011 - 14:00 - 15:30
Location: 
Methods Group
Methods Group: 
None
Contact person
Contact person: 
Marta Roque (Contact this person)
Facilitators
First nameLast nameAffiliation and Country
First name: 
John
Last name: 
Ioannidis
Affiliation and Country: 
University of Ioannina, Greece
Target audience
Target audience: 
Anyone interested in the topic
Is your workshop restricted to a specific audience or open to all Colloquium participants?: 
Open
Level of knowledge required: 
Advanced
Type of workshop
Type of workshop: 
Discussion
Abstract text
Abstract: 
Almost all biomedical studies currently are highlighting some statistically significant results with p-values <0.05. However, the large majority of these statistically significant claims fail to get replicated when larger and better studies are conducted. Publication bias and other types of selection biases are often thought to be the explanation. Most methods to detect publication and related biases have serious problems at the conceptual and inferential level and are commonly misused. When properly used, these methods may be applicable to about 10% of the current meta-analyses and evidence synthesis. The workshop will discuss a different approach that tests for excess significance in wider domains or whole fields of research. The concept tries to model the difference in observed versus expected results that pass different thresholds of statistical significance under different assumptions about the true effect sizes. One can use a simplified model where the p=0.05 threshold is considered to be a major attractor, or extend the concept to include more complex dynamics in the generation and publication of evidence, including the Proteus phenomenon and differential preferences for specific results based on what prior results in the field have been. Examples will be provided from diverse fields.