Quote a survey figure with its weights and margin of error
Why it matters
Strategy papers lean on survey figures about customers, staff and AI adoption, and one national health survey shows how far such a figure can move, from 11.29% of its respondents to 8.44% of adults once its weights are applied.1 Before a survey number supports a decision, ask whether it was weighted and how wide its margin of error is.
Once you know a survey’s design, weighting it and stating an honest margin of error take a few lines of statsmodels.2 The margin needs the most care, because unequal weights make it wider than the textbook formula gives, and it covers sampling error only, so it says nothing about a biased sample or a leading question.2,3
What you will do
A simulated city wants to know whether a parks levy has majority support. Its survey of 1,200 households was written for the book, so the true answer is known and every estimate can be checked against it.2
- Lay out the design. The four districts are sampled at different rates, so each household carries a weight equal to its district’s population divided by the number sampled there.2
- Plot support by district before averaging anything. Oldtown, the district least in favor, was over-sampled on purpose and supplies a third of the respondents from under a fifth of the city’s households.2
- Estimate support twice with
DescrStatsWfrom statsmodels. The raw responses give about 55%, and the weighted estimate is 63%, close to the city’s true 62%.2 - Give the weighted figure a margin of error that allows for the weights. Unequal weights cut the effective sample from 932 respondents to about 660, and the margin comes to about 4 points either side.2
- Size the next survey. Holding the margin near 3 points takes roughly 1,000 respondents under simple random sampling and about 1,400 with this design.2
Before you start
A free Google account is all you need to run the notebook in Colab. It reads the survey from the book’s repository and uses pandas, statsmodels and seaborn, which Colab already has, so there is nothing to install and nothing to pay. If margins of error are new to you, chapter 74 of the Visual Handbook explains what sets their size.
Articles it puts into practice
- The same survey says 11.29% or 8.44% applies weights and design-based standard errors to a national survey of 134,897 adults.
- Adoption is outrunning governance reads four recent surveys about AI agents one at a time, because their samples and measures differ.
Found this useful? Follow me on LinkedIn for the next one.
Working on this? Connect and tell me what you are seeing.