Signal Over Noise

1. The Core Idea: Signal Over Noise

Every statistical test computes some version of this ratio:

Signal

effect size

Noise

chance variation

test statistic

Signal is the pattern you care about: the difference between group means, the slope of a regression line, the change in frequency across categories.

Noise is the variation that exists even when nothing interesting is happening: individual differences between organisms, measurement error, spatial and temporal heterogeneity, the irreducible randomness of the natural world.

A test statistic close to zero means signal and noise are the same size — the pattern is indistinguishable from background variation. A large test statistic means the signal is much larger than expected from noise alone — the pattern is rare under the assumption that nothing is happening.

Three scenarios with different signal-to-noise ratios. Left: large signal, small noise — the groups are clearly different. Middle: moderate signal, moderate noise — uncertain. Right: small signal, large noise — the groups are indistinguishable. The ratio determines what a test can detect, not either quantity alone.

Signal-to-noise in the t-statistic

For a two-group comparison, the signal-to-noise ratio is the t-statistic:

\[t = \frac{\underbrace{\bar{x}_A - \bar{x}_B}_{\text{signal: observed difference}}} {\underbrace{SE_{\text{diff}}}_{\text{noise: expected sampling variation}}}\]

For the F-statistic in ANOVA:

\[F = \frac{\underbrace{MS_{\text{between}}}_{\text{signal: variance explained by groups}}} {\underbrace{MS_{\text{within}}}_{\text{noise: variance within groups}}}\]

Both are signal-over-noise ratios. They differ only in what “signal” and “noise” mean geometrically. The core question is always the same: how often would we see a ratio this large if there were actually no effect?

2. The Null World — What If Nothing Were Happening?

To decide whether an observed signal-to-noise ratio is large, we need to know the distribution of that ratio under the null hypothesis — i.e., in a world where the groups have identical population means and any apparent difference is pure chance.

There are two ways to construct this null distribution:

Simulation (Monte Carlo / permutation): Shuffle the data to destroy any real group structure. Compute the test statistic many times. The histogram of those values is the empirical null distribution.
Mathematics (t, F distributions): Mathematicians derived the exact shape of these null distributions analytically. We compare our observed statistic to the theoretical curve rather than simulating.

Both approaches answer the same question. Simulation is slower but requires no assumptions beyond exchangeability. Theory is instant but requires the data to satisfy certain conditions (normality, equal variance, independence). We build the simulation first because it makes the logic completely transparent.

3. Monte Carlo Simulation: Building Intuition

Before looking at real data, let us build intuition by watching what happens when we know for certain that the null hypothesis is true.

Generating null data

We will create two groups drawn from identical populations. Any difference between their means is pure sampling noise — there is no signal by construction.

set.seed(7)

# Two groups from the SAME population — null is true by construction
n_per_group  <- 20
true_mean    <- 10
true_sd      <- 3

group_A_null <- rnorm(n_per_group, mean = true_mean, sd = true_sd)
group_B_null <- rnorm(n_per_group, mean = true_mean, sd = true_sd)

cat("Group A mean:", round(mean(group_A_null), 2), "\n")

## Group A mean: 11.33

cat("Group B mean:", round(mean(group_B_null), 2), "\n")

## Group B mean: 10.3

cat("Observed difference:", round(mean(group_A_null) - mean(group_B_null), 2), "\n")

## Observed difference: 1.03

Even though both groups were drawn from the same distribution, we observe a non-zero difference. This is sampling variation — noise. The question is: how often does sampling produce a difference at least this large?

Repeating the experiment 5,000 times

We simulate 5,000 “experiments” under the null (both groups from the same population), record the mean difference each time, and build the null distribution.

set.seed(99)
n_sims       <- 5000
null_diffs   <- numeric(n_sims)

for (i in seq_len(n_sims)) {
  A <- rnorm(n_per_group, mean = true_mean, sd = true_sd)
  B <- rnorm(n_per_group, mean = true_mean, sd = true_sd)
  null_diffs[i] <- mean(A) - mean(B)
}

cat("Null distribution summary:\n")

## Null distribution summary:

round(quantile(null_diffs, c(0.025, 0.05, 0.25, 0.50, 0.75, 0.95, 0.975)), 2)

##  2.5%    5%   25%   50%   75%   95% 97.5% 
## -1.87 -1.58 -0.64  0.00  0.65  1.59  1.92

The null distribution of mean differences when both groups are drawn from the same population (n = 20 each, 5,000 simulations). The distribution is centred on zero and roughly normal. The amber shaded tails contain 5% of simulations — our conventional Type I error threshold.

📌 What the null distribution tells us

The amber tails contain exactly 5% of all simulated differences — values so extreme they would arise by chance less than 1 in 20 times when the null is true. This 5% is our conventional α = 0.05 threshold. An observed difference that falls in the tail is called “statistically significant” — not because it is important, but because it is rare under the null hypothesis.

4. Permutation Test on Real Data

Now we apply the same logic to real data — but instead of generating new data from a known distribution, we shuffle the existing observations to destroy the real group structure. This is a permutation test, and it is the most assumption-free way to construct a null distribution.

The dataset: lead in drinking water by pipe material

set.seed(3)
# Lead concentration (µg/L) sampled from buildings with different pipe material
# EPA action level: 15 µg/L
lead <- data.frame(
  pipe    = rep(c("Lead/Lead-solder", "Copper/PVC"), c(28, 32)),
  lead_ppb = c(
    round(rlnorm(28, meanlog = 2.6, sdlog = 0.55)),   # older lead pipes
    round(rlnorm(32, meanlog = 1.6, sdlog = 0.50))    # modern copper/PVC
  )
)

ggplot(lead, aes(pipe, lead_ppb, fill = pipe)) +
  geom_boxplot(width = 0.45, outlier.shape = NA,
               colour = "#1e3a2f", alpha = 0.75, linewidth = 0.6) +
  geom_jitter(aes(colour = pipe), width = 0.18,
              size = 1.8, alpha = 0.5) +
  geom_hline(yintercept = 15, linetype = "dashed",
             colour = "#bf360c", linewidth = 0.9) +
  annotate("text", x = 2.48, y = 17,
           label = "EPA action level (15 µg/L)",
           colour = "#bf360c", size = 3.2, hjust = 1, fontface = "italic") +
  scale_fill_manual(values  = c("Lead/Lead-solder" = "#7a3a2f",
                                "Copper/PVC"       = "#4a7c5e")) +
  scale_colour_manual(values = c("Lead/Lead-solder" = "#7a3a2f",
                                 "Copper/PVC"       = "#4a7c5e")) +
  labs(x = NULL, y = "Lead concentration (µg/L)",
       title = "Tap Water Lead Concentration by Pipe Material") +
  theme_minimal(base_size = 12) +
  theme(
    plot.title         = element_text(family = "serif", face = "bold", colour = "#1e3a2f"),
    legend.position    = "none",
    panel.grid.major.x = element_blank()
  )

Lead concentration by pipe material. Each point is one building sample. The EPA action level (15 µg/L) is shown as a dashed line. Lead/lead-solder pipes produce substantially higher concentrations.

# The observed difference in means — our "signal"
obs_diff <- mean(lead$lead_ppb[lead$pipe == "Lead/Lead-solder"]) -
            mean(lead$lead_ppb[lead$pipe == "Copper/PVC"])
cat("Observed mean difference:", round(obs_diff, 2), "µg/L\n")

## Observed mean difference: 7.71 µg/L

Step 1 Understand what shuffling does

The key insight: if pipe material has no effect on lead concentration, then the labels “Lead/Lead-solder” and “Copper/PVC” are arbitrary — we could randomly reassign them and the difference in means would look just like our real difference.

Shuffling the labels destroys any real association while preserving the marginal distribution of the data. The collection of mean differences from many shuffles is the null distribution.

# One single shuffle — what does a "null" difference look like?
set.seed(42)
shuffled_labels <- sample(lead$pipe)    # randomly reassign pipe labels
one_null_diff   <- mean(lead$lead_ppb[shuffled_labels == "Lead/Lead-solder"]) -
                   mean(lead$lead_ppb[shuffled_labels == "Copper/PVC"])
cat("One null difference (shuffled):", round(one_null_diff, 2), "µg/L\n")

## One null difference (shuffled): -1.12 µg/L

cat("Observed difference:           ", round(obs_diff, 2),      "µg/L\n")

## Observed difference:            7.71 µg/L

Step 2 Repeat 9,999 times

set.seed(1234)
n_perms     <- 9999
perm_diffs  <- numeric(n_perms)

for (i in seq_len(n_perms)) {
  shuffled        <- sample(lead$pipe)
  perm_diffs[i]   <- mean(lead$lead_ppb[shuffled == "Lead/Lead-solder"]) -
                     mean(lead$lead_ppb[shuffled == "Copper/PVC"])
}

# p-value: proportion of null differences >= observed (one-tailed)
# Two-tailed: proportion of |null differences| >= |observed|
p_perm <- mean(abs(perm_diffs) >= abs(obs_diff))
cat("Permutation p-value (two-tailed):", p_perm, "\n")

## Permutation p-value (two-tailed): 0

cat("Observed difference falls in the",
    round(mean(perm_diffs < obs_diff) * 100, 1),
    "th percentile of the null distribution.\n")

## Observed difference falls in the 100 th percentile of the null distribution.

Step 3 Visualise the observed statistic against the null

The permutation null distribution of mean differences (9,999 shuffles). The observed difference of ~9 µg/L (red line) sits far into the right tail — it would almost never arise by chance if pipe material had no effect. The permutation p-value is the proportion of null differences exceeding the red line.

⚙️ Computing the permutation p-value
p_value <- mean(abs(perm_diffs) >= abs(obs_diff))

This counts the fraction of shuffled experiments that produced a difference at least as extreme as the one we observed. That fraction is the probability of seeing our data — or something more extreme — if the null were true. No formula required. The simulation is the calculation.

5. From Simulation to Theory: The t-Distribution

The permutation test is the most honest approach, but it requires computing thousands of shuffles. Mathematicians asked: what shape does the null distribution of a signal-to-noise ratio always take, if the data are normally distributed? The answer is the t-distribution.

Watching the t-statistic accumulate under the null

We repeat the same logic as the Monte Carlo section, but now we standardise the mean difference by its standard error — computing the t-statistic each time.

set.seed(77)
n_sims  <- 10000
t_nulls <- numeric(n_sims)

for (i in seq_len(n_sims)) {
  A <- rnorm(20, mean = 0, sd = 1)   # both from N(0,1) — null true
  B <- rnorm(20, mean = 0, sd = 1)
  t_nulls[i] <- t.test(A, B)$statistic
}

Histogram of 10,000 simulated t-statistics when the null is true (both groups from N(0,1), n = 20 each). The theoretical t-distribution with df = 38 (amber curve) fits perfectly. The simulation and the formula describe the same null distribution — the formula is just much faster.

Applying the t-test to the lead data

t_result <- t.test(lead_ppb ~ pipe, data = lead)
print(t_result)

## 
##  Welch Two Sample t-test
## 
## data:  lead_ppb by pipe
## t = -5.9361, df = 34.673, p-value = 9.699e-07
## alternative hypothesis: true difference in means between group Copper/PVC and group Lead/Lead-solder is not equal to 0
## 95 percent confidence interval:
##  -10.353401  -5.075171
## sample estimates:
##       mean in group Copper/PVC mean in group Lead/Lead-solder 
##                        5.50000                       13.21429

The observed t-statistic from the lead data (red line) placed on the theoretical t-distribution under the null. The shaded amber region is the two-tailed rejection area (α = 0.05). Our observed t sits far into the tail — the same conclusion reached by the permutation test, now from a formula.

The t-distribution family: degrees of freedom

The t-distribution is actually a family of curves, indexed by degrees of freedom (df). With small samples, the tails are heavier — reflecting greater uncertainty. As df → ∞, the t-distribution converges to the standard normal.

The t-distribution for different degrees of freedom. Small df produces heavy tails — small samples generate more extreme statistics by chance, so the critical value must be larger to maintain α = 0.05. With df ≥ 30, the t-distribution is almost indistinguishable from the normal.

📌 Why degrees of freedom shrink the distribution’s tails

With only a few observations, your sample standard deviation is an unreliable estimate of the population SD. Sometimes it underestimates by a lot — which makes the t-statistic artificially large even when the null is true. The t-distribution with small df compensates by having heavier tails: it expects to see large values occasionally just from estimation error, so it demands a larger observed t before calling a result “rare.” As sample size grows, the SD estimate becomes reliable and the t-distribution narrows toward the normal.

6. The F-Distribution: Signal Over Noise for Multiple Groups

When comparing three or more groups, the t-statistic generalises to the F-statistic, which is the ratio of variance between groups to variance within groups. The same signal-over-noise logic applies.

Building the F null distribution by simulation

set.seed(55)
n_sims  <- 10000
f_nulls <- numeric(n_sims)

# Null: three groups from identical populations
for (i in seq_len(n_sims)) {
  grp <- factor(rep(c("A","B","C"), each = 15))
  y   <- rnorm(45, mean = 0, sd = 1)
  f_nulls[i] <- summary(aov(y ~ grp))[[1]]["grp", "F value"]
}

Simulated F-statistics when all three groups are drawn from the same population (n = 15 per group). The theoretical F-distribution F(2, 42) (amber curve) fits the simulation perfectly. The F-distribution is right-skewed and bounded below by zero — unlike the symmetric t-distribution.

Applying the F-test: nitrogen across land-use types

set.seed(42)
nitrogen <- data.frame(
  landuse = rep(c("Forest", "Grassland", "Cropland"), each = 25),
  N_gkg   = c(rnorm(25, 4.8, 0.7),
              rnorm(25, 3.2, 0.9),
              rnorm(25, 1.9, 0.6))
)

model_aov <- aov(N_gkg ~ landuse, data = nitrogen)
aov_table <- summary(model_aov)[[1]]
F_obs     <- aov_table["landuse", "F value"]
df1       <- aov_table["landuse", "Df"]
df2       <- aov_table["Residuals", "Df"]

cat("Between-group MS (signal):", round(aov_table["landuse",   "Mean Sq"], 3), "\n")

## Between-group MS (signal): 56.242

cat("Within-group MS  (noise): ", round(aov_table["Residuals", "Mean Sq"], 3), "\n")

## Within-group MS  (noise):  0.634

cat("F = signal / noise =",       round(F_obs, 2), "\n")

## F = signal / noise = 88.71

cat("p-value:                 ",  round(aov_table["landuse", "Pr(>F)"], 6), "\n")

## p-value:                  0

The observed F-statistic (red line) from the soil nitrogen ANOVA placed on the F(2, 72) null distribution. The observed F is so large it falls entirely beyond the plotted rejection region — correspondingly, p < 0.0001. The ratio of between-group to within-group variance is far larger than chance produces.

Decomposing the variance in a table

# Manual variance decomposition — showing the signal/noise arithmetic
grand_mean  <- mean(nitrogen$N_gkg)
group_means <- tapply(nitrogen$N_gkg, nitrogen$landuse, mean)
group_sizes <- tapply(nitrogen$N_gkg, nitrogen$landuse, length)

SS_between <- sum(group_sizes * (group_means - grand_mean)^2)
SS_within  <- sum(tapply(seq_along(nitrogen$N_gkg), nitrogen$landuse,
                  function(i) sum((nitrogen$N_gkg[i] - mean(nitrogen$N_gkg[i]))^2)))
SS_total   <- sum((nitrogen$N_gkg - grand_mean)^2)

df_between <- length(group_means) - 1
df_within  <- nrow(nitrogen) - length(group_means)

MS_between <- SS_between / df_between
MS_within  <- SS_within  / df_within

knitr::kable(
  data.frame(
    Source     = c("Between groups (signal)", "Within groups (noise)", "Total"),
    SS         = round(c(SS_between, SS_within, SS_total), 2),
    df         = c(df_between, df_within, df_between + df_within),
    MS         = round(c(MS_between, MS_within, NA), 3),
    F          = c(round(MS_between / MS_within, 2), NA, NA),
    `p-value`  = c(round(pf(MS_between/MS_within, df_between, df_within,
                            lower.tail = FALSE), 6), NA, NA)
  ),
  col.names = c("Source", "Sum of Squares", "df", "Mean Square", "F", "p-value"),
  caption = "ANOVA decomposition: the F-statistic is the ratio of between-group mean square (signal) to within-group mean square (noise).",
  na.encode = FALSE
)

ANOVA decomposition: the F-statistic is the ratio of between-group mean square (signal) to within-group mean square (noise).
Source	Sum of Squares	df	Mean Square	F	p-value
Between groups (signal)	112.48	2	56.242	88.71	0
Within groups (noise)	45.65	72	0.634	NA	NA
Total	158.14	74	NA	NA	NA

7. Two Roads to the Same Answer

The permutation test and the t/F-test give the same conclusion because they are asking the same question in different ways. Here we verify this side-by-side.

set.seed(888)

# ── Observed F for the nitrogen data ──────────────────────────────────────────
F_obs_check <- summary(aov(N_gkg ~ landuse, data = nitrogen))[[1]][1, "F value"]

# ── Permutation F-distribution ────────────────────────────────────────────────
n_perm   <- 9999
perm_F   <- numeric(n_perm)

for (i in seq_len(n_perm)) {
  shuffled_groups <- sample(nitrogen$landuse)
  perm_F[i] <- summary(
    aov(nitrogen$N_gkg ~ shuffled_groups)
  )[[1]][1, "F value"]
}

p_permutation <- mean(perm_F >= F_obs_check)

# ── Parametric p-value ────────────────────────────────────────────────────────
p_parametric <- summary(model_aov)[[1]]["landuse", "Pr(>F)"]

cat("─── Permutation test ────────────────────────────────\n")

## ─── Permutation test ────────────────────────────────

cat("Permutation p-value:", p_permutation, "\n\n")

## Permutation p-value: 0

cat("─── Parametric F-test ───────────────────────────────\n")

## ─── Parametric F-test ───────────────────────────────

cat("Parametric  p-value:", round(p_parametric, 6), "\n")

## Parametric  p-value: 0

Left: the permutation null distribution of F (grey bars) with the observed F marked in red. Right: the theoretical F(2, 72) null distribution with the same observed statistic. Both methods place the observed F far into the tail. The tiny difference between the two p-values is pure Monte Carlo sampling noise.

💡 When the two approaches disagree

Permutation and parametric tests give almost identical p-values when the parametric assumptions (normality, equal variances) are satisfied — as they do here. They diverge when the data are strongly non-normal or when group variances differ dramatically. In those cases, the permutation p-value is more trustworthy because it makes no distributional assumptions. The practical rule: if shapiro.test() or a Q-Q plot raises concerns, check your parametric p-value against a permutation or bootstrap equivalent.

🔑 Signal over noise — key takeaways

Every test statistic is a signal-to-noise ratio. Signal is the pattern you care about (difference in means, slope, variance between groups). Noise is the variation that exists even when nothing is happening (within-group spread, estimation uncertainty).
The p-value is a rarity statement: “If the null hypothesis were true, data as extreme as mine would arise p × 100% of the time.” It is not the probability that the null is true, nor a measure of effect size.
Monte Carlo simulation makes the logic transparent. Shuffle the labels, compute the statistic, repeat 10,000 times. The histogram you get is the null distribution. The p-value is just the fraction of bars to the right of your observed value.
The t and F distributions are mathematical shortcuts to the same null distribution, valid when data are approximately normal with equal group variances. Use t.test() and aov() — they are fast and well-studied. Use permutation tests when assumptions are violated.
Degrees of freedom control tail heaviness. Small samples produce larger test statistics by chance alone, so the critical value must be larger — which is why a t-test with n = 5 requires a bigger observed difference than one with n = 50 to reach the same α threshold.
Statistical significance ≠ practical importance. A very large sample can make a tiny, ecologically irrelevant difference highly significant. Always report effect sizes (Cohen’s d, η², odds ratio) alongside p-values.

What’s next?

Four Frameworks — which test to use for each data type
Describing Data — mean, SD, SE, and confidence intervals
t-test in depth — full worked example with course data

✏️ Practice Exercises

Exercise 1 — Computing F by hand

Three soil plots received different nitrogen treatments. Bulk density (g/cm³) measurements: Control: c(1.4, 1.5, 1.3), Low N: c(1.2, 1.1, 1.3), High N: c(0.9, 1.0, 0.8). (a) Compute the grand mean. (b) Compute SS_between and SS_within by hand. (c) Compute the F-ratio. (d) Verify with aov().

Show solution

ctrl <- c(1.4, 1.5, 1.3); lown <- c(1.2, 1.1, 1.3); highn <- c(0.9, 1.0, 0.8)
all_data <- c(ctrl, lown, highn)
grand_mean <- mean(all_data)

# SS_between
ss_b <- 3 * sum((c(mean(ctrl), mean(lown), mean(highn)) - grand_mean)^2)
# SS_within
ss_w <- sum((ctrl - mean(ctrl))^2) + sum((lown - mean(lown))^2) + sum((highn - mean(highn))^2)

ms_b <- ss_b / 2    # df_between = k-1 = 2
ms_w <- ss_w / 6    # df_within  = N-k = 6
F_ratio <- ms_b / ms_w

# Verify
bd <- data.frame(density = all_data, trt = rep(c("ctrl","low","high"), each=3))
summary(aov(density ~ trt, data = bd))

Exercise 2 — Power thinking

You want to detect a difference in mean leaf area between two forest types. Your pilot data suggest SD ≈ 12 cm². You need power = 0.80 at α = 0.05. Use power.t.test() to find the sample size needed to detect a difference of (a) 5 cm², (b) 10 cm², (c) 15 cm². Describe the pattern you see.

Show solution

for (delta in c(5, 10, 15)) {
  p <- power.t.test(delta = delta, sd = 12, power = 0.80, sig.level = 0.05)
  cat(sprintf("delta = %d cm2 → n = %.0f per group
", delta, p$n))
}

As the effect size doubles, required n drops by roughly four-fold — because power scales with (δ/σ)². Detecting a small 5 cm² difference requires far larger samples than a 15 cm² difference.

Four Frameworks ↑ Module Index Hypothesis Testing

Signal Over Noise

Marc Los Huertos

EA030 · Compiled 2026-04-14

Signal Over Noise

1. The Core Idea: Signal Over Noise

Signal-to-noise in the t-statistic

2. The Null World — What If Nothing Were Happening?

3. Monte Carlo Simulation: Building Intuition

Generating null data

Repeating the experiment 5,000 times

4. Permutation Test on Real Data

The dataset: lead in drinking water by pipe material

Step 1 Understand what shuffling does

Step 2 Repeat 9,999 times

Step 3 Visualise the observed statistic against the null

5. From Simulation to Theory: The t-Distribution

Watching the t-statistic accumulate under the null

Applying the t-test to the lead data

The t-distribution family: degrees of freedom

6. The F-Distribution: Signal Over Noise for Multiple Groups

Building the F null distribution by simulation

Applying the F-test: nitrogen across land-use types

Decomposing the variance in a table

7. Two Roads to the Same Answer

🔑 Signal over noise — key takeaways

What’s next?