Home / Lean Six Sigma / Statistical analysis

Hypothesis tests: means, proportions, variances and rates

Is the new supplier better? Did the new method shorten the cycle? Did the poka-yoke cut scrap? Paste the measurements for A and B and see whether the difference is real, whether it matters in practice, whether the sample was large enough and what it is worth per year.

Pre-sizing and estimating tool: results do not replace design, calculations or reports signed by a licensed professional, nor verification against the applicable codes and standards.

CheckRevised 11/03/2026
Analysis
What is being compared
Data
See what was read

Measurements: one value per cell. Proportion: each cell is the result of one item (OK/NOK, 0/1). Rate: each cell is the count of occurrences in one observation. In the paired test, each row is the same part.

Hypothesis and options
Translation into parts and money (optional)

With LSL/USL, the page estimates from the normal curve how many parts per year fall outside specification in A and in B. With the value of 1 unit (e.g. hourly cost ÷ 3,600 for seconds), it translates the difference in means into money per year.

How the data were collected

Your inputs stay only on this browser and are not sent anywhere. Untick to delete them.

—A × B comparison
—

Test results

Method

Descriptive statistics

Estimation

Test

Difference with CI

Dot = estimate; bar = confidence interval; red line = H₀; green band = difference that does not matter in practice.

Tap the chart to see the values.

Individual values, boxplot and CI for the mean

Dots = each measurement; box = middle 50% and median; diamond with bar = estimate and CI; dashed lines = H₀ or specification.

Tap the chart to see the values.

Histogram by sample

Distribution of each sample’s values, with each mean.

Tap the chart to see the values.

Power curve

Chance of detecting each true difference with the current sample (solid) and with the sample that gives 80% at the difference that matters (dashed).

Tap the chart to see the values.

Reference distribution and P-value area

How the test statistic is distributed if H₀ is true; the red area is the P-value.

Tap the chart to see the values.

Next step

Take this page’s results to another calculator with the fields already filled in.

The calculation report gathers this page's inputs, results, equations, assumptions and references, with fields for project, engineer and revision. The full report also includes the charts and interpretations, in color. In the print dialog, choose Save as PDF.

Method, assumptions and references

Equations used

1-sample t: t = (x̄ − μ₀)/(s/√N), DF = N − 1; paired t: the same on the differences d = B − A (or 1 − 2); CI: estimate ± t(1 − α/2; DF)·SE
2-sample t (Welch): SE = √(s₁²/N₁ + s₂²/N₂), DF = SE⁴/[(s₁²/N₁)²/(N₁ − 1) + (s₂²/N₂)²/(N₂ − 1)]; equal variances: s_p = √{[(N₁ − 1)s₁² + (N₂ − 1)s₂²]/(N₁ + N₂ − 2)}, SE = s_p·√(1/N₁ + 1/N₂)
1 proportion: exact binomial (Blaker two-sided P-value and CI; one-sided and per-sample CI by Clopper-Pearson) or Z = (p̂ − p₀)/√[p₀(1 − p₀)/N]; 2 proportions: Z = (p̂₁ − p̂₂)/√[p̂₁(1 − p̂₁)/N₁ + p̂₂(1 − p̂₂)/N₂] and Fisher’s exact (hypergeometric)
1 variance: χ² = (N − 1)s²/σ₀², CI for σ by χ²; Bonett: CI for σ² = c·s²·exp(±c·z·se), c = N/(N − z), se = √{[γ̂ − (N − 3)/N]/(N − 1)}, γ̂ = kurtosis with trimmed mean (proportion 1/[2√(N − 4)])
2 variances: F = s₁²/s₂² (DF N₁ − 1 and N₂ − 1); Levene = ANOVA of |x − median|; Bonett: pooled kurtosis γ̂_P and CI for σ₁/σ₂ from the roots of [ln x − ln(c·ρ̂²)]² = z²·se²(x)
Poisson: λ̂ = occurrences/(N·length); exact CI χ²(α/2; 2s)/(2T) to χ²(1 − α/2; 2s + 2)/(2T); 2 samples: conditional exact (binomial of the total) and Z = (λ̂₁ − λ̂₂)/√(λ̂₁/T₁ + λ̂₂/T₂)
Power (t): 1 − T′(t_c; DF, δ) + T′(−t_c; DF, δ), noncentral t with δ = Δ/SE; proportions and rates: normal approximation; repeated looks: real α from Armitage et al. and per-look α from Pocock

Assumptions and limits

  • Random, independent samples (except in the paired test, where each pair is the same part). The t-tests tolerate non-normality with 15 to 20 or more values per sample; with fewer and skewed data, confirm with a nonparametric test.
  • In A × B mode the difference is always B − A, the test is two-sided and the type of result decides the test: measurement → Welch t (or paired), defective/not → 2 proportions (Fisher with few events), count → 2 Poisson rates.
  • The “no real difference” band compares the CI with the difference that matters entered; formal equivalence (TOST) would use the 1 − 2α CI. Parts and money per year assume the measured difference holds in production.

Standards and technical documents

  • Montgomery, D. C.; Runger, G. C. Applied Statistics and Probability for Engineers, 7th ed. Wiley, 2018, cap. 9–10
  • Welch, B. L. The generalization of “Student’s” problem when several different population variances are involved. Biometrika 34:28–35, 1947
  • Clopper, C. J.; Pearson, E. S. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika 26:404–413, 1934; Blaker, H. Confidence curves and improved exact confidence intervals for discrete distributions. Canadian Journal of Statistics 28:783–798, 2000
  • Bonett, D. G. Approximate confidence interval for standard deviation of nonnormal distributions. Computational Statistics & Data Analysis 50:775–782, 2006
  • Banga, S. J.; Fox, G. D. Bonett’s method (white paper, 2013) e Brown, M. B.; Forsythe, A. B. Robust tests for the equality of variances. JASA 69:364–367, 1974
  • Armitage, P.; McPherson, C. K.; Rowe, B. C. Repeated significance tests on accumulating data. JRSS A 132:235–244, 1969; Pocock, S. J. Group sequential methods in the design and analysis of clinical trials. Biometrika 64:191–199, 1977
  • Ortiz, R. J. F.; Toscano, E. M. M. Estatística usando Minitab, RTE-03-2010, Departamento de Estatística, UFMG, 2010 (exemplos 5.2 a 6.4, usados na conferência)

U.S. regulations

  • ISO 9001:2015, 9.1.3: analysis and evaluation
  • ISO 2854:1976 Statistical interpretation of data — Techniques of estimation and tests relating to means and variances
  • ISO 11453:1996 Statistical interpretation of data — Tests and confidence intervals relating to proportions

Frequently asked questions

How do I know whether supplier (or method) B is better than A?

Use the "Compare A × B" mode: paste the measurements for A and B and say which is better (higher or lower). The page picks the test, shows the difference B − A with its confidence interval and tells you whether it is real, large enough to matter and whether the sample was big enough. In the cycle-time example, the new method is about 2.2 s faster and the whole interval lies beyond 1 s, so the improvement is real and relevant.

What does a P-value below 0.05 mean?

That a difference as large as the one observed would be rare (under 5% chance) if A and B were the same. It does not say whether the difference is large: with huge samples, tiny differences become significant. That is why the page also shows the confidence interval and the band of the difference that matters.

When should I use the paired test?

When the same part, lot or operator appears in both conditions, such as the same part measured on two gauges or the same lot before and after a treatment. Pairing removes part-to-part variation and often detects differences the 2-sample test misses. Different parts in A and B: use the 2-sample test.

The test says "no difference". Can I conclude that A and B are equal?

Only if the whole confidence interval fits inside the green band of the difference that matters. If the interval is wide, the result is inconclusive: the sample was too small. The power curve shows the chance the test had of seeing the difference and how many more samples are needed for 80%.

Which test should I use for scrap or defective parts?

To compare two proportions defective (before × after, shift × shift), use 2 proportions: the page runs the Z test and Fisher’s exact test, which holds even with few defects. For counts of defects per part, failures per month or stops per shift, use the Poisson rates.

What are Bonett and Levene for?

To compare the spread (stability) of two processes without assuming normal data. The F and chi-square tests are exact only for normal data and can be badly off with heavy tails. A process B with the same mean but a smaller standard deviation is also an improvement: fewer parts out of specification.

Why can’t I stop collecting as soon as the result is significant?

Because each look is a new chance of a false alarm. Looking 5 times at 5%, the real chance of “finding” a difference that does not exist reaches about 14%. Set the sample size beforehand, or enter how many times you looked: the page shows the real α and the corrected p threshold for each look.