Community survey mistakes that mute citizen feedback

So were the consultants. So were the people who organized the push. The trouble is that the headline number tells you almost nothing about what came back. A pile of reply slips is not the same thing as a representative read of the people who live, work, and pay taxes here. It's a sample of the people who happened to hear about it, cared to click, and had thirty minutes to spare. The infrastructure that's supposed to carry citizen input into the planning process has a load rating — and most municipal surveys are running well over it.

This is not a critique of Somerset County alone. The same load-bearing failures show up in surveys from Crisfield to Princess Anne, from Westover to the rest of the Eastern Shore. What follows is a field inspection of the most common methodological mistakes that quietly drain the signal out of citizen feedback — and what residents and officials can actually do about them before the next planning cycle treats bad data as a mandate.

The Convenience Sample Trap

The first mistake is the most common one, and it sits at the foundation of nearly every other problem downstream: treating an opt-in survey as if it were a referendum. When a county office posts a link on its website, pushes it through a few social channels, and runs a newspaper notice, the responses that come back are a convenience sample. They reflect the people who saw the link, felt motivated, and had the bandwidth to fill out the form. They do not reflect the population as a whole.

This is not a minor footnote. A convenience sample by definition cannot tell you what the wider community thinks. It can only tell you what the self-selected respondents think. Any policy conclusion drawn from that data should carry the same weight as a weather forecast based on one thermometer bolted to a south-facing porch — directionally useful, structurally meaningless.

Somerset County is not a huge jurisdiction. The kind of rigorous probability sampling that would produce a representative read is expensive and slow, and nobody expects a county planning office to commission a phone survey of two thousand residents every time the Comprehensive Plan comes up for review. But there is a difference between acknowledging the limits of your data and pretending the limits don't exist. The first line of any public report should say, clearly, that the survey was an open participation tool — not a measurement instrument. The moment staff start quoting the 360-response figure as evidence of community consensus, the load-bearing claim is being placed on a foundation that can't hold it.

An opt-in survey measures who showed up. It does not measure who lives here.

Length Kills the Signal

Industry guidance on municipal surveys is pretty consistent: anything beyond fifteen to twenty minutes triggers survey fatigue. Once a respondent crosses that threshold, two things happen, and both of them are bad for the data. Either they drop out entirely, which means every later question is answered by a smaller and less representative subset, or they switch to straight-lining — the practice of clicking the same answer down the page because they stopped reading the questions. Either way, the drain on data quality is significant.

A question set that takes thirty-five minutes to finish is not a survey. It is a homework assignment, and most people don't do homework for free. The shorter, sharper version of the same questionnaire — the one that respects the respondent's time — will almost always produce more honest answers than the comprehensive version that exhausts the participant halfway through.

This is a particularly common mistake in comprehensive planning work, because the planners want to ask about everything — land use, housing, transportation, economic development, environmental quality, public facilities, agricultural preservation, intergovernmental coordination. That is a long list. The temptation to put all of it in one instrument is real, and the consequences are predictable. The fix is ruthless prioritization. A focused twenty-minute survey on three or four core issues will return more usable signal than a forty-minute omnibus that tries to capture the entire planning horizon in one pass.

The Digital Divide Nobody Maps

Relying on a single distribution channel — particularly a digital-only online form — creates a coverage gap that the planners never see, because the gap is defined by the people who didn't hear about the survey in the first place.

Somerset County has pockets of low broadband access. Not everyone has reliable home internet. Some households are mobile-only. A meaningful number of older residents do not interact with the county's website or social media accounts at all. When the survey is posted exclusively online, those residents are functionally excluded from the participation process. The 360 responses that came back represent the people who could reach the form — not the people who couldn't.

The fix is not complicated, but it does require labor. Paper copies at the public library, the senior center, the post office, and town halls. A mail-in option with a prepaid return envelope. In-person tabling at community events, churches, and farmers markets. Phone calls for residents who prefer to talk through questions. None of this is glamorous. All of it is necessary if the goal is actually to hear from the community rather than from the community's most connected subset.

The Demographic Skew

Traditional public engagement methods — and online surveys are part of that lineage — tend to over-represent older, higher-income, home-owning residents. Renters show up less. Young adults show up less. Non-English speakers show up less. Lower-income families show up less. The result is a feedback portrait that looks more like the county's homeowner associations than like the county itself.

This is not a complaint about who chooses to participate. People have busy lives, work shifts, child care responsibilities, and limited patience for government forms. The skew is a structural property of the engagement method, not a moral failing of the residents. But unless the planners know the skew is there and correct for it — or at minimum flag it openly in the final report — the conclusions drawn from the survey will be skewed along with the data.

The February 2024 update to the Somerset County Zoning Ordinance and the Comprehensive Plan update process both required meaningful public input. The surveys were one channel. The public hearings under Maryland Land Use Code §3-203 were the statutory channel. The risk is that the easier-to-collect survey data ends up carrying more weight in the final write-up than the harder-to-collect hearing testimony, simply because the survey came in a spreadsheet and the testimony came in scribbled notes. That would be a methodological error with real consequences.

Surveys are a straw. Public hearings are the load-bearing wall. Don't confuse the two.

Surveys Cannot Replace Hearings

This is the part of the conversation that gets glossed over in a lot of planning documents, and it shouldn't be. Maryland Land Use Code §3-203 is clear: the local planning commission must hold at least one formal public hearing before recommending a comprehensive plan. The statute does not say the commission can substitute a survey for that hearing. It does not say a survey can stand in for the public record. The hearing is the legal mechanism. The survey is an outreach tool.

Conflating the two is a mistake that happens in two directions. Sometimes staff present survey results as if they were the public hearing, and the formal hearing becomes a procedural afterthought. Sometimes commissioners cite the survey response count as evidence that they've already heard from the community, which makes the upcoming hearing feel like a formality rather than a substantive step.

Neither of those postures survives scrutiny. Surveys and hearings capture different things. Surveys capture structured preferences at a moment in time. Hearings capture testimony, argument, cross-questioning, and the back-and-forth that comes from having actual people in a room explaining why they hold a particular position. The Comprehensive Plan process needs both. Eliding one in favor of the other is a structural failure of the engagement plan, and it tends to happen quietly enough that nobody catches it until the adoption vote is on the table.

The Optimism Bias in Reporting

The habit of citing the response count as a success metric is its own kind of error. "We got 360 responses, exceeding our goal of 300" reads as a win, and it is a win in terms of participation effort. But the goal of 300 was presumably set because someone, at some point, decided that 300 responses would be sufficient to characterize community sentiment. Once the survey closes, the response count tells you nothing about whether 300 was actually enough, or whether the people who answered were the people you most needed to hear from.

The more useful reporting framework separates three numbers:

A response count without a demographic profile is a headcount, not a survey result. A headcount can confirm a turnout effort. It cannot confirm a community read.

Phone Bias and Social Desirability

For jurisdictions that do use phone interviews alongside online instruments, there's a separate methodological wrinkle worth naming. Live phone interviews introduce social desirability bias — the tendency of respondents to give answers they think the interviewer wants to hear, particularly on politically sensitive topics like zoning changes, housing density, or environmental regulation. Self-administered web and mail surveys tend to produce more candid answers because the respondent is alone with the form.

This doesn't mean phone outreach is useless. It means phone results should be weighted differently from anonymous web results, and the two should not be averaged together as if they were equivalent measures. Somerset County's process leaned primarily on digital channels, which mostly sidesteps this issue but inherits the digital divide problem instead. There is no clean channel. There is only the channel whose limitations you are willing to document.

The Twenty-Year Horizon

Somerset County's Comprehensive Plan operates on a roughly twenty-year horizon. The previous baseline plan dates to 1996. That means the survey results gathered in mid-2024 are being used to inform decisions that will define land use, infrastructure investment, and zoning policy through the middle of the 2040s. A twenty-year horizon is a long time to be wrong about what the community actually wants.

The structural risk is that the survey produced a snapshot of who answered in 2024, and that snapshot will be treated as representative of the community for the next two decades. By the time the next major update rolls around, the demographic distribution of the county will have shifted, the priorities will have shifted, and the survey instrument that captured the 2024 sentiment will be a historical artifact rather than a current read. Building a twenty-year plan on convenience-sample data is a deferral of maintenance that compounds annually.

What Actually Works

The fix list is not exotic. It is mostly labor, sequencing, and honesty about what the data can support.

First, cap the survey at twenty minutes. If the questionnaire runs longer, cut questions. A focused survey on three or four priorities beats a sprawling one with eight priority areas and signal-to-noise problems.

Second, distribute through multiple channels. Online, paper, in-person, and phone. Tabling at community events. Drop-off points at the library and senior centers. Translated materials for the county's Spanish-speaking residents. The point is to reach people where they already are, not to wait for them to come to you.

Third, report demographic data alongside response counts. If the county doesn't know who answered, it has no business claiming to know what the community thinks.

Fourth, keep the formal public hearing central. The hearing is the legal mechanism and the substantive mechanism. Surveys support the hearing. They do not replace it.

Fifth, treat the survey as the beginning of a conversation, not the conclusion. Follow-up workshops, focus groups, and citizen advisory committees can fill in the gaps that the survey left open. The Somerset County planning process has chapter-level comment opportunities that remain active after the headline survey closes. That follow-on structure is the right instinct — it just needs to be staffed and resourced.

Sixth, be honest in the write-up. State the methodological limits plainly. Identify the demographic skew. Acknowledge the digital divide. Quote the survey results as one input among several, not as the voice of the community. A planning document that admits its own data limits is more defensible, not less — because it shows the planners understood what they were working with.

The Position

The 360 responses that came in for Somerset County's Comprehensive Plan update represent real effort. They represent real opinions from real residents who took the time to participate. The mistake is not in collecting the data. The mistake is in treating the collection effort as a measurement of consensus when it was a measurement of turnout. Those are different things, and the difference matters when the resulting plan is going to govern zoning, infrastructure, and land use for the next twenty years.

Surveys are useful. They are also limited. They are a straw, not a load-bearing wall. Citizens who want their feedback to count should fill out the survey, show up at the hearing, and follow the chapter-level comments as the plan draft moves through the process. Officials who want the plan to hold up should design the engagement to reach the people who don't usually show up, report the limits of the data honestly, and stop citing response counts as evidence of consensus. The infrastructure for citizen input has a load rating. Right now, most municipal surveys are running over it.