Meta-analysis
Meta-analysis is the statistical combination of the results of multiple studies addressing a similar research question. An important part of this method involves computing an effect size across all of the studies; this involves extracting effect sizes and variance measures from various studies. Meta-analyses are integral in supporting research grant proposals, shaping treatment guidelines, and influencing health policies. They are also pivotal in summarizing existing research to guide future studies, thereby cementing their role as a fundamental methodology in metascience. Meta-analyses are often, but not always, important components of a systematic review procedure. For instance, a meta-analysis may be conducted on several clinical trials of a medical treatment, in an effort to obtain a better understanding of how well the treatment works.
For the process in historical linguistics known as metanalysis, see Rebracketing.History[edit]
The term "meta-analysis" was coined in 1976 by the statistician Gene Glass,[1][2] who stated "Meta-analysis refers to the analysis of analyses".[3] Glass's work aimed at describing aggregated measures of relationships and effects.[4] While Glass is credited with authoring the first modern meta-analysis, a paper published in 1904 by the statistician Karl Pearson in the British Medical Journal[5] collated data from several studies of typhoid inoculation and is seen as the first time a meta-analytic approach was used to aggregate the outcomes of multiple clinical studies.[6][7] Numerous other examples of early meta-analyses can be found including occupational aptitude testing,[8][9] and agriculture.[10]
The first model meta-analysis was published in 1978 on the effectiveness of psychotherapy outcomes by Mary Lee Smith and Gene Glass.[2][11] After publication of their article there was pushback on the usefulness and validity of meta-analysis as a tool for evidence synthesis. The first example of this was by Han Eysenck who in an 1978 article in response to the work done by Mary Lee Smith and Gene Glass called meta-analysis an "exercise in mega-silliness".[12][13] Later Eysenck would refer to meta-analysis as "statistical alchemy".[14] Despite these criticisms the use of meta-analysis has only grown since its modern introduction. By 1991 there were 334 published meta-analyses;[13] this number grew to 9,135 by 2014.[1][15]
The field of meta-analysis expanded greatly since the 1970s and touches multiple disciplines including psychology, medicine, and ecology.[1] Further the more recent creation of evidence synthesis communities has increased the cross pollination of ideas, methods, and the creation of software tools across disciplines.[16][17][18]
Literature Search[edit]
One of the most important steps of a meta-analysis is data collection. For an efficient database search, appropriate keywords and search limits need to be identified.[21] The use of Boolean operators and search limits can assist the literature search.[22][23] A number of databases are available (e.g., PubMed, Embase, PsychInfo), however, it is up to the researcher to choose the most appropriate sources for their research area.[24] Indeed, many scientists use duplicate search terms within two or more databases to cover multiple sources. The reference lists of eligible studies can also be searched for eligible studies (i.e., snowballing). The initial search may return a large volume of studies. Quite often, the abstract or the title of the manuscript reveals that the study is not eligible for inclusion, based on the pre-specified criteria. These studies can be discarded. However, if it appears that the study may be eligible (or even if there is some doubt) the full paper can be retained for closer inspection. The references lists of eligible articles can also be searched for any relevant articles. These search results need to be detailed in a PRIMSA flow diagram[25] which details the flow of information through all stages of the review. Thus, it is important to note how many studies were returned after using the specified search terms and how many of these studies were discarded, and for what reason.[24] The search terms and strategy should be specific enough for a reader to reproduce the search. The date range of studies, along with the date (or date period) the search was conducted should also be provided.[26]
A data collection form provides a standardized means of collecting data from eligible studies. For a meta-analysis of correlational data, effect size information is usually collected as Pearson's r statistic. Partial correlations are often reported in research, however, these may inflate relationships in comparison to zero-order correlations.[27] Moreover, the partialed out variables will likely vary from study-to-study. As a consequence, many meta-analyses exclude partial correlations from their analysis.[24] As a final resort, plot digitizers can be used to scrape data points from scatterplots (if available) for the calculation of Pearson's r.[28][29] Data reporting important study characteristics that may moderate effects, such as the mean age of participants, should also be collected.[30] A measure of study quality can also be included in these forms to assess the quality of evidence from each study.[31] There are more than 80 tools available to assess the quality and risk of bias in observational studies reflecting the diversity of research approaches between fields.[31][32][33] These tools usually include an assessment of how dependent variables were measured, appropriate selection of participants, and appropriate control for confounding factors. Other quality measures that may be more relevant for correlational studies include sample size, psychometric properties, and reporting of methods.[24]
A final consideration is whether to include studies from the gray literature, which is defined as research that has not been formally published.[34] This type of literature includes conference abstracts,[35] dissertations,[36] and pre-prints.[37] While the inclusion of gray literature reduces the risk of publication bias, the methodological quality of the work is often (but not always) lower than formally published work.[38][39] Reports from conference proceedings, which are the most common source of gray literature,[40] are poorly reported[41] and data in the subsequent publication is often inconsistent, with differences observed in almost 20% of published studies.[42]
Methods and assumptions[edit]
Approaches[edit]
In general, two types of evidence can be distinguished when performing a meta-analysis: individual participant data (IPD), and aggregate data (AD). The aggregate data can be direct or indirect.
AD is more commonly available (e.g. from the literature) and typically represents summary estimates such as odds ratios or relative risks. This can be directly synthesized across conceptually similar studies using several approaches (see below). On the other hand, indirect aggregate data measures the effect of two treatments that were each compared against a similar control group in a meta-analysis. For example, if treatment A and treatment B were directly compared vs placebo in separate meta-analyses, we can use these two pooled results to get an estimate of the effects of A vs B in an indirect comparison as effect A vs Placebo minus effect B vs Placebo.
IPD evidence represents raw data as collected by the study centers. This distinction has raised the need for different meta-analytic methods when evidence synthesis is desired, and has led to the development of one-stage and two-stage methods.[43] In one-stage methods the IPD from all studies are modeled simultaneously whilst accounting for the clustering of participants within studies. Two-stage methods first compute summary statistics for AD from each study and then calculate overall statistics as a weighted average of the study statistics. By reducing IPD to AD, two-stage methods can also be applied when IPD is available; this makes them an appealing choice when performing a meta-analysis. Although it is conventionally believed that one-stage and two-stage methods yield similar results, recent studies have shown that they may occasionally lead to different conclusions.[44][45]
Applications in modern science[edit]
Modern statistical meta-analysis does more than just combine the effect sizes of a set of studies using a weighted average. It can test if the outcomes of studies show more variation than the variation that is expected because of the sampling of different numbers of research participants. Additionally, study characteristics such as measurement instrument used, population sampled, or aspects of the studies' design can be coded and used to reduce variance of the estimator (see statistical models above). Thus some methodological weaknesses in studies can be corrected statistically. Other uses of meta-analytic methods include the development and validation of clinical prediction models, where meta-analysis may be used to combine individual participant data from different research centers and to assess the model's generalisability,[117][118] or even to aggregate existing prediction models.[119]
Meta-analysis can be done with single-subject design as well as group research designs.[120] This is important because much research has been done with single-subject research designs.[121] Considerable dispute exists for the most appropriate meta-analytic technique for single subject research.[122]
Meta-analysis leads to a shift of emphasis from single studies to multiple studies. It emphasizes the practical importance of the effect size instead of the statistical significance of individual studies. This shift in thinking has been termed "meta-analytic thinking". The results of a meta-analysis are often shown in a forest plot.
Results from studies are combined using different approaches. One approach frequently used in meta-analysis in health care research is termed 'inverse variance method'. The average effect size across all studies is computed as a weighted mean, whereby the weights are equal to the inverse variance of each study's effect estimator. Larger studies and studies with less random variation are given greater weight than smaller studies. Other common approaches include the Mantel–Haenszel method[123] and the Peto method.[124]
Seed-based d mapping (formerly signed differential mapping, SDM) is a statistical technique for meta-analyzing studies on differences in brain activity or structure which used neuroimaging techniques such as fMRI, VBM or PET.
Different high throughput techniques such as microarrays have been used to understand Gene expression. MicroRNA expression profiles have been used to identify differentially expressed microRNAs in particular cell or tissue type or disease conditions or to check the effect of a treatment. A meta-analysis of such expression profiles was performed to derive novel conclusions and to validate the known findings.[125]
Meta-analysis of whole genome sequencing studies provides an attractive solution to the problem of collecting large sample sizes for discovering rare variants associated with complex phenotypes. Some methods have been developed to enable functionally informed rare variant association meta-analysis in biobank-scale cohorts using efficient approaches for summary statistic storage.[126]
Sweeping meta-analyses can also be used to estimate a network of effects. This allows researchers to examine patterns in the fuller panorama of more accurately estimated results and draw conclusions that consider the broader context (e.g., how personality-intelligence relations vary by trait family).[127]
