How do you test significance in an unreplicated factorial design?

An unreplicated factorial spends every degree of freedom on effects, so there is no residual left to estimate error from and no t-test to run. The standard procedure is graphical: plot the absolute effects on a half-normal probability scale, where inactive effects fall on a line through the origin and active ones depart from it. Lenth's method puts a margin of error on that judgment, derived from the median of the small effects rather than from a replicate. SVEND draws the half-normal plot with Lenth's margin on it, so the selection procedure and the picture are the same object rather than an illustration drawn afterwards.

What is design resolution and why does it matter?

A fractional factorial buys fewer runs by confounding effects with each other, and resolution names what is confounded with what. In a resolution III design main effects are aliased with two-factor interactions, so an effect that looks large may be either. Resolution IV keeps main effects clear of two-factor interactions but aliases those interactions in pairs. Resolution V separates both. The alias structure is a property of the design you chose, not of the data you collected, which is why it is worth reading before the runs are made rather than after the result is surprising.

How many runs does an experiment need?

That depends on the smallest effect worth detecting, the noise in the response, and how often you are willing to miss a real effect. SVEND answers it in both directions: given an effect size it reports the power of a design, and given a design it reports the smallest effect that design can resolve. The second is the one to use when a plant cannot state an effect size in advance, which is common, because a run sheet that cannot resolve anything is cheaper to discover before the runs than after.

I — Datum

Design of Experiments

An experiment is a plan before it is a dataset. The runs are what cost money, and they are made once — so the design, the order they are run in, and the effect the study is powered to resolve are all decided before the first part is made, and are worth recording as decisions rather than reconstructing afterwards.

Ref: Montgomery · Box, Hunter & Hunter · Daniel (1959) · Lenth (1989)

How a DOE study runs → Start Free → Pricing →

II — What Runs Today

This table is generated from the analysis catalog when the page is requested, so it cannot drift from the product. The right-hand column says which surface runs a method: the bench issues a design and takes the numbers back later, and the workbench runs the three that need only a dataset you already have.

Method	What it does	Runs on
Full Factorial	Every combination of 2-level factors, optional center points	bench
Fractional Factorial	2^(k−p) screening design that states its confounding	bench
Plackett-Burman	Run-efficient main-effects screening	bench
Definitive Screening	Three-level screening; main effects clear of two-way interactions	bench
Central Composite	Response-surface design with axial and center points	bench
Optimal (D/I)	The best design a run budget allows, when the budget is not a catalog size	bench
Optimize	The setting the fitted model expects to be best, over the region explored	bench
Predict	What the fitted model expects at a setting, with its interval	bench
Design Power	Per-term power for a design, before it is run	bench
Detectable Effect	The smallest effect this design could resolve at a stated power	bench
Optimize (multi-response)	The settings that best satisfy several responses at once, each with its own goal	bench
Box-Behnken	Three-level response surface without corner runs	bench
Taguchi Array	Orthogonal-array run sheet (2-level); analyze responses with Model Fit	bench
Model Fit	Fit a main-effects, interaction, or quadratic model to experiment results	workbench
Split-Plot Fit	Two-stratum ANOVA for hard-to-change factors — whole plots earn their own error term	workbench
Split-Plot Fit (unbalanced, REML)	Restricted maximum likelihood for split-plots that did not come out balanced — unequal whole plots, missing runs	workbench

13 on the bench, 3 in the workbench picker.

III — Where This Is Unusual

Most packages treat a design as a worksheet you generate and then own. Two things here are not the norm.

The design is issued, not generated. A run sheet leaves the bench as a document with a fixed run order and a recorded custody, because the randomization is the defense against drift and it only defends anything if the floor works the order it was given. Changing the design under an issued sheet is refused rather than merged.

Responses are keyed by standard order. The sheet is worked in run order and the responses are recorded against the run they belong to, so the fit reads them back by standard order without anyone re-sorting a spreadsheet. Section IV explains why that is the part worth getting right.

IV — Where DOE Studies Go Wrong

Four failure modes, all of which produce a result that looks complete.

Standard order against run order. A design is generated in standard order and run in a randomized order. If the responses are read back in sheet order without being re-keyed, every value is attributed to the wrong set of factor levels. The worksheet is fully populated, the fit converges, the effects are wrong, and nothing about the output looks unusual.

An unreplicated design has no error term. With every degree of freedom spent on effects there is no residual to test against. The half-normal plot with Lenth's margin is the selection procedure in that case, not a picture of one — which is why it is drawn rather than described.

Aliasing read after the fact. Resolution is a property of the design, decided when it is chosen. A resolution III screen confounds main effects with two-factor interactions, so a large effect is two candidate explanations rather than one, and no amount of care in the running recovers the difference.

A plane fitted to a curve. A two-level factorial can only fit a plane. Center points are how curvature is detected before a response surface is fitted to data that never had the runs to support one.

V — Boundaries

The bench issues a plan and records what came back. It is not a data acquisition system and does not talk to equipment while an experiment runs.

It does not choose your factors. Which variables matter, what range is safe to move them over, and what the response should be are engineering judgments the software has no basis to make and does not pretend to.

A design that has been issued is a record. Editing it is refused rather than versioned, because a run sheet somebody worked from cannot be retroactively changed into a different one.

VI — Where It Sits

The DOE bench is part of the paid bench alongside SPC, capability, measurement system analysis and reliability — see pricing for what each tier carries. Measurement system analysis is the study worth running first: an experiment resolves effects against measurement noise, so a gauge that cannot tell two parts apart sets the floor on what any design can detect. That bench is documented here.