Designing a Factorial Experiment

Capstone 7 analyzed a factorial experiment somebody else had designed. This one is about the design itself: how many runs you can afford, what a cheaper design stops being able to tell you, and how to find that out before the runs rather than after.

Five factors at two levels each is 2⁵ = 32 distinct settings. Replicated twice, so that noise can be estimated without assuming the model is right, that is 64 runs. Each run needs a line changeover, which is why this is a design problem before it is an analysis problem.

The budget allowed 32 runs. The interesting question is what the quarter fraction, at half that cost, stops being able to tell you, and the honest answer is that it stops being able to tell you what it is telling you.

Last quarter the team ran a quarter fraction. With three factors you get eight settings for free; the other two have to be built out of the first three, and the team chose D = AB and E = AC. Eight settings, two replicates, sixteen runs. Five factors, five effects, all estimated. It looks complete.

The standard error of an effect here is about 0.86 N, so line speed at +6.22 is more than seven standard errors from zero. By any conventional reading that is a solid, actionable finding, and the recommendation wrote itself: slow the line down.

The right-hand column of that table is not a theoretical caution. It is a statement about arithmetic that you can check in a few lines: build the column of signs for the interaction A×D, and compare it element by element with the column for B.

The same check across all five factors gives the whole picture: A is indistinguishable from B×D and C×E, B from A×D, C from A×E, D from A×B, and E from A×C. That is what resolution III means, and it is a property of the design, decided before any panel was coated. No analysis recovers information the design never collected.

The line was slowed from 12 to 8 meters per minute. Forty panels were measured, twenty before and twenty after. The design had promised about six newtons.

A confirmation run is the cheapest insurance in experimental work, and this is what it buys. The interval runs from about one newton down to two thirds of a newton up, and excludes the promised +6 many times over. Something was wrong with the design, not with the process, and forty panels were enough to find out.

The half fraction uses a single generator, E = ABCD, giving the defining relation I = ABCDE. Main effects are now aliased only with four-factor interactions, and two-factor interactions only with three-factor ones. Under the usual assumption that high-order interactions in a physical process are negligible, all five main effects and all ten two-factor interactions are separately estimable. That is resolution V, and the notebook confirms it the same way as before: among those fifteen columns, no two are identical.

The lost run leaves the design very slightly unbalanced. Least squares handles that without comment; the hand-computed contrast of the previous section would not, which is a good practical reason to fit a model rather than average columns once a design stops being perfectly balanced.

Line speed is +0.17 newtons on a standard error of 0.38. It was never there. The +6.22 the eight-run design reported belonged to oven temperature crossed with cure time, which appears here at +6.47 and could not have appeared at all in a design where its column and line speed's column were the same numbers.

Thirty-two runs need two batches of primer, and batches differ. The plan put replicate 1 entirely on batch 1 and replicate 2 on batch 2, so batch is the block. It enters the model as a term, which takes the batch difference out of the error rather than leaving it there to inflate every standard error.

Note what blocking did not do. The batch difference is still there in the process; it has not been removed from the world, only from the yardstick. And note the choice it represents. Batch was known in advance, so it became a block. Had the batches instead been randomized across runs, the same variation would have gone into the error term and every interval would have been a third wider for no benefit. The rule is short: block what you can predict, randomize what you cannot.

The oven creeps upward across a shift. Nobody measured that and nobody had to, because run order within each batch was randomized. The value of randomizing is easiest to see by simulating the alternative: running the settings in a convenient order, all the low-pressure runs first, which on a real line saves genuine time.

Four tenths of a newton is a five percent distortion of the temperature effect, which nobody would ever notice. On nozzle pressure it is half the effect, and it changes the conclusion about whether pressure matters at all. You cannot know in advance which of your effects are small, which is the entire argument for randomizing: it converts a bias you cannot see into noise you can measure.

The recommendation is not a factor. It is a combination, and the interaction is the reason.

Read the bottom row against the top. Raising the temperature at short cure gains 2.3 N. Raising it at long cure gains 14.9 N. And at the low temperature, extending the cure actually makes the coating worse, from 60.89 down to 56.73. A recommendation that named either factor alone would be wrong in a way that a recommendation naming both is not.

The design that cost twice as much found the lever the cheap design pointed away from, and the confirmation run is what turns an estimate into a change somebody is willing to sign.

• ✓Report the resolution next to the results, every time. A reader cannot infer it from an effect table, and a resolution III design produces sharp numbers that invite action rather than vague ones that invite caution.
• ✓Aliasing is a property of the design, not of the data. The only honest options are to run the extra points that break the alias, or to print the alias chain beside the estimate and let the reader see the ambiguity.
• ✓Say which interactions you assumed away. Resolution V is safe on the assumption that three-factor and higher interactions are negligible. That is usually reasonable in a physical process, and it is an assumption, so it belongs in the write-up.
• ✓Fifteen effects is fifteen tests. Six terms here clear p = 0.05, and on a pure-noise design you would expect about one to. Rank by magnitude against the standard error rather than reading down a column of p-values, which is the lesson of Capstone 4 arriving in new clothes.
• ✓Randomize even when it is inconvenient. Sorting runs by a factor saves real time on a production line and buys it by confounding that factor with everything that drifts. The cost never appears in the output.
• ✓Confirm before you commit. Both recommendations here were tested on fresh panels first. The first confirmation is the only reason anyone discovered the screening design had misled them, and it cost forty panels.
• ✓A designed experiment on a production line has consequences. Running deliberately poor settings makes genuinely worse product, and somebody has to decide what happens to it. That belongs in the plan, not in the write-up.

Factorial thinking is older than computing and it keeps being rediscovered wherever runs are expensive.