False discovery rate

m is the total number hypotheses tested

${\displaystyle m_{0}}$ is the number of true , an unknown parameter

null hypotheses

${\displaystyle m-m_{0}}$ is the number of true

alternative hypotheses

V is the number of (also called "false discoveries")

false positives (Type I error)

S is the number of (also called "true discoveries")

true positives

T is the number of

false negatives (Type II error)

U is the number of

true negatives

${\displaystyle R=V+S}$ is the number of rejected null hypotheses (also called "discoveries", either true or false)

If the tests are independent or positively correlated (as in Benjamini–Hochberg procedure): ${\displaystyle c(m)=1}$

Under arbitrary dependence (including the case of negative correlation), c(m) is the : ${\displaystyle c(m)=\sum _{i=1}^{m}{\frac {1}{i}}}$.

Note that ${\displaystyle c(m)}$ can be approximated by using the Taylor series expansion and the Euler–Mascheroni constant (${\displaystyle \gamma =0.57721...}$):

$${\displaystyle \sum _{i=1}^{m}{\frac {1}{i}}\approx \ln(m)+\gamma +{\frac {1}{2m}}.}$$

harmonic number

PCER () is defined as: ${\displaystyle \mathrm {PCER} =E\left[{\frac {V}{m}}\right]}$. Testing individually each hypothesis at level α guarantees that ${\displaystyle \mathrm {PCER} \leq \alpha }$ (this is testing without any correction for multiplicity)

per-comparison error rate

FWER (the ) is defined as: ${\displaystyle \mathrm {FWER} =P(V\geq 1)}$. There are numerous procedures that control the FWER.

family-wise error rate

${\displaystyle k{\text{-FWER}}}$ (The tail probability of the False Discovery Proportion), suggested by Lehmann and Romano, van der Laan at al, is defined as: ${\displaystyle k{\text{-FWER}}=P(V\geq k)\leq q}$.

${\displaystyle k{\text{-FDR}}}$ (also called the generalized FDR by Sarkar in 2007^[20]) is defined as: ${\displaystyle k{\text{-FDR}}=E\left({\frac {V}{R}}I_{(V>k)}\right)\leq q}$.

[19]

${\displaystyle Q'}$ is the proportion of false discoveries among the discoveries", suggested by Soric in 1989, and is defined as: ${\displaystyle Q'={\frac {E[V]}{R}}}$. This is a mixture of expectations and realizations, and has the problem of control for ${\displaystyle m_{0}=m}$.^[1]

[9]

${\displaystyle \mathrm {FDR} _{-1}}$(or Fdr) was used by Benjamini and Hochberg, and later called "Fdr" by Efron (2008) and earlier.^[21] It is defined as: ${\displaystyle \mathrm {FDR} _{-1}=Fdr={\frac {E[V]}{E[R]}}}$. This error rate cannot be strictly controlled because it is 1 when ${\displaystyle m=m_{0}}$.

[3]

${\displaystyle \mathrm {FDR} _{+1}}$ was used by Benjamini and Hochberg, and later called "pFDR" by Storey (2002).^[22] It is defined as: ${\displaystyle \mathrm {FDR} _{+1}=pFDR=E\left[\left.{\frac {V}{R}}\right|R>0\right]}$. This error rate cannot be strictly controlled because it is 1 when ${\displaystyle m=m_{0}}$. JD Storey promoted the use of the pFDR (a close relative of the FDR), and the q-value, which can be viewed as the proportion of false discoveries that we expect in an ordered table of results, up to the current line. Storey also promoted the idea (also mentioned by BH) that the actual number of null hypotheses, ${\displaystyle m_{0}}$, can be estimated from the shape of the probability distribution curve. For example, in a set of data where all null hypotheses are true, 50% of results will yield probabilities between 0.5 and 1.0 (and the other 50% will yield probabilities between 0.0 and 0.5). We can therefore estimate ${\displaystyle m_{0}}$ by finding the number of results with ${\displaystyle P>0.5}$ and doubling it, and this permits refinement of our calculation of the pFDR at any particular cut-off in the data-set.^[22]

[3]

False exceedance rate (the tail probability of FDP), defined as: ${\displaystyle \mathrm {P} \left({\frac {V}{R}}>q\right)}$

[23]

${\displaystyle W{\text{-FDR}}}$ (Weighted FDR). Associated with each hypothesis i is a weight ${\displaystyle w_{i}\geq 0}$, the weights capture importance/price. The W-FDR is defined as: ${\displaystyle W{\text{-FDR}}=E\left({\frac {\sum w_{i}V_{i}}{\sum w_{i}R_{i}}}\right)}$.

FDCR (False Discovery Cost Rate). Stemming from : associated with each hypothesis i is a cost ${\displaystyle \mathrm {c} _{i}}$ and with the intersection hypothesis ${\displaystyle H_{00}}$ a cost ${\displaystyle c_{0}}$. The motivation is that stopping a production process may incur a fixed cost. It is defined as: ${\displaystyle \mathrm {FDCR} =E\left(c_{0}V_{0}+{\frac {\sum c_{i}V_{i}}{c_{0}R_{0}+\sum c_{i}R_{i}}}\right)}$

statistical process control

PFER (per-family error rate) is defined as: ${\displaystyle \mathrm {PFER} =E(V)}$.

FNR (False non-discovery rates) by Sarkar; Genovese and Wasserman is defined as: ${\displaystyle \mathrm {FNR} =E\left({\frac {T}{m-R}}\right)=E\left({\frac {m-m_{0}-(R-V)}{m-R}}\right)}$

${\displaystyle \mathrm {FDR} (z)}$ is defined as: ${\displaystyle \mathrm {FDR} (z)={\frac {p_{0}F_{0}(z)}{F(z)}}}$

${\displaystyle \mathrm {FDR} }$ The local fdr is defined as: ${\displaystyle \mathrm {FDR} ={\frac {p_{0}f_{0}(z)}{f(z)}}}$

Positive predictive value

– Lists links with popular R packages

False Discovery Rate Analysis in R

– Python implementations of false discovery rate procedures

False Discovery Rate Analysis in Python

- MATLAB/GNU Octave implementation and discussion on the difference between corrected and adjusted FDR p-values.

False Discovery Rate: Corrected & Adjusted P-values

- blog post

Understanding False Discovery Rate

on YouTube

StatQuest: FDR and the Benjamini-Hochberg Method clearly explained

- Includes Excel VBA code to implement it, and an example in cell line development

Understanding False Discovery Rate