Your Sample Size Calculator Is Answering a Survey Question

Type "sample size calculator" into a search box and the first several results will ask you for a confidence level, a margin of error, and a population size. Enter 95% and ±5% and you get 385.

That number is correct. It answers the question how many people do I need to poll so that my estimate of a proportion is within five points. It is a survey question, and it is the question almost every general-purpose sample size calculator on the internet was built for.

It is not the question a quality engineer is being asked when someone says "how many parts do we need to run?"

There are at least three different questions hiding inside that sentence, they have different formulas, and they give answers that differ by an order of magnitude. Picking the wrong one does not produce an error message. It produces a number, and the number looks reasonable.

Question 1 — Can I detect a difference?

You changed a supplier, a tool, a setting. You want to know whether the mean moved. This is a power question, and it is the one the statistics education system prepares you for.

The inputs are the effect size you care about, the significance level, and the power you want. For a difference of half a standard deviation, at α = 0.05 and 80% power:

n = 63 per group

For a full standard deviation, it drops to 16 per group. That sensitivity is the whole point — the smaller the shift you insist on catching, the more parts it costs, and the relationship is steep.

The important discipline here is stating the effect size before collecting data. An effect size chosen after seeing the result is not a design, it is a justification.

Question 2 — How precise is my capability estimate?

You ran a capability study. It came back Cpk = 1.33, which clears the usual 1.33 requirement exactly. How much do you actually know?

Cpk is an estimate from a sample, so it has a confidence interval like anything else. For n parts, its standard error is approximately

SE(Cpk) ≈ √( 1 / (9·n·Cpk²)  +  1 / (2(n−1)) )

Run that out and the results are uncomfortable:

A capability study of thirty parts reporting Cpk 1.33 is consistent with a true Cpk of 1.06 — comfortably failing — and equally consistent with 1.60. Thirty parts is a very common study size.

To pin Cpk 1.33 to ±0.10, you need 218 parts.

Nobody runs 218 parts for a capability study, and this article is not suggesting you start. The point is narrower and more useful: when a customer asks you to demonstrate Cpk ≥ 1.33 and your study returns 1.35 from thirty parts, you have not demonstrated it. You have failed to rule it out. Those are different claims and only one of them survives a second study.

(If this is new, the point estimate problem goes further into what to do about it.)

Question 3 — Can I demonstrate reliability?

This is the validation question, and it is the one the survey calculators answer worst.

A validation protocol usually asks you to demonstrate that some proportion of parts conform, at some confidence. No failures allowed. That is not a margin-of-error question and not a power question — it is a reliability demonstration, and it has its own closed form:

n = ln(1 − C) / ln(R)

  C = confidence
  R = reliability you want to demonstrate

Which gives the numbers that appear all over validation protocols, and now they have a derivation attached:

The 59 is the one people recognize. It is where "run 59 and if none fail you are done" comes from, and it is worth knowing that it is a statement about 95% reliability at 95% confidence rather than a universal constant.

Two things follow immediately. The cost of confidence is mild — dropping from 95% to 90% saves you 14 parts. The cost of reliability is brutal: each additional nine multiplies the sample by roughly five. If a protocol asks for 99.9% at 95%, it is asking for three thousand parts, and somebody should establish that is what they meant before the run starts rather than after.

Note also what the zero-failure rule costs you: a single failure does not reduce your demonstrated reliability a little. It invalidates the run. Plans that tolerate failures exist and need more parts.

The failure mode

All three questions are asked with the same words. The answers for one situation range from 16 to 2,995 parts.

The specific trap is using a survey or power calculator for question 3. A validation protocol says "95% confidence" and a survey calculator has a field labeled confidence, so the numbers go in and 385 comes out. It has the right shape, it is in the right range to be plausible, and it is the answer to a question nobody asked.

The tell is the second field. If the calculator wants a margin of error, it is estimating a proportion. If it wants power and an effect size, it is testing a hypothesis. If it wants neither, and asks for reliability and confidence, it is the one you want for validation.

Which question are you actually being asked?

A short version worth keeping:

Comparing two states — before and after, supplier A and supplier B, machine 1 and machine 2 — is question 1. You need an effect size before you start.

Reporting a capability index — Cpk, Ppk, a process performance number that goes on a certificate — is question 2. Report the interval alongside the point estimate, and state the sample size that produced it.

Demonstrating conformance for a protocol — PPAP, IQ/OQ/PQ, a customer validation, a design verification — is question 3. Use the reliability form, agree the reliability and confidence in writing first, and expect the number of parts to be driven almost entirely by the reliability figure.

None of these is difficult arithmetic. The expensive mistake is upstream of the arithmetic, in deciding which question is on the table, and it is usually made in the first two minutes of a meeting rather than in the calculator.

Run the numbers

The free sample size calculator handles question 1 — one-sample, two-sample, proportions and ANOVA, with the power curve drawn so you can see how the requirement moves as the effect size changes. No account needed.

Question 3 needs no tool at all. n = ln(1−C) / ln(R) fits in a spreadsheet cell, and now you know what the 59 in your validation protocol is made of.