How do you account for ACS margin of error when ranking areas?

Keep the margin-of-error columns through every step of the pipeline and make the ranking account for them, so a small area with a noisy estimate cannot outrank areas with reliable ones. Then test the output against what the source data can support.

Why did the tests not catch the defect?

They checked that the answer looked right, and it did. Nothing checked whether the source data could support the answer.

Geospatial risk scoring for a cancer-screening program, and the margin-of-error defect every test missed.

A county cancer-screening program now ranks 70 ZIP areas by where its uninsured, screening-age residents live. Program staff use the ranking to decide where outreach goes.

Fig. C2 · ranked estimates with their error bars · schematic

Results

Before and after.

Before

After

No ZIP-level view of the cohort

948,704 residents scored across 70 ZIP areas

Small areas ranked on noise

Rankings account for each estimate’s margin of error

Priority cohort unmeasured

An estimated 7,638 uninsured residents aged 45 to 64, located by ZIP area

The work

How the work went.

The problem

The program needed to know where its uninsured, screening-age residents lived, across a county of nearly a million people. Outreach hours are limited, so the order of the list matters as much as the list.

What we built

We built a scoring service over U.S. Census American Community Survey (ACS) tables with a weighted risk model. Program staff see its output on a map and use it to plan outreach. The final dataset had no missing values, no duplicates and no integrity errors.

The defect every test missed

Every ACS estimate comes with a margin of error. The data-preparation step was dropping those columns, and every test still passed, because the tests checked that the answer looked right. As a result, an area of 84 people ranked second on an estimate its own margin of error swamped. We restored the margins, made the ranking account for them, and now check output against the limits of the source data, not against the answer we expect.

Keep reading.

Platform used

Platform 02Data platformSee the platform →

Other case studies

Production AI · National wireless carrierA production AI agent that reconciles six systems with no shared customer keyRead the case study →Security · Law firmStopping a credential-stuffing attack on four law-firm websitesRead the case study →

Questions · Sheet C-12

Questions about this case study.

Not covered here? Send us your question and an engineer will answer it.

Ask a question

How do you account for ACS margin of error when ranking areas?: Keep the margin-of-error columns through every step of the pipeline and make the ranking account for them, so a small area with a noisy estimate cannot outrank areas with reliable ones. Then test the output against what the source data can support.
Why did the tests not catch the defect?: They checked that the answer looked right, and it did. Nothing checked whether the source data could support the answer.