Skip to content

Appendix C — Sampling, Response Rates, and Statistical Confidence

C.1 What the Research Needs to Support

The starting point for sampling is not statistical theory. It is the level of confidence ARDC needs in order to make useful decisions. In this proposal, the research is intended to support two kinds of judgment:

  • a reliable top-line view of the licensed population and the Entry system
  • meaningful analysis of selected subgroups that are strategically important, even if they are smaller parts of the population

That leads to two practical questions. First, how many completed responses are needed to support a credible overall read? Second, how large do important subgroup cells need to be in order to support useful comparison?

In this appendix:

  • Population means the full group we want to understand
  • Frame means the list or source used to reach that population
  • Cell means the number of completed responses in a particular subgroup, such as a license class or lifecycle stage

C.2 Initial Planning Targets

At a high level, the project can be thought of in terms of three illustrative survey scales:

  • Top-line design — target about 400 completes
  • Segment-aware design — target about 800 completes
  • Expanded design — target about 1,200 completes

These are not final recommendations. They are practical planning anchors that help connect research ambitions to confidence, invitation volume, and cost.

A useful rule of thumb is:

  • 400 completes supports a strong overall read of a large population, typically around ±5% margin of error at 95% confidence
  • 800 completes gives more room for meaningful subgroup analysis
  • 1,200 completes provides more flexibility for crossed-segment questions and emerging research questions that were not fully anticipated at the start

For subgroup work:

  • 100 respondents in a subgroup is a strong target for useful comparison
  • 30 respondents in a subgroup is better treated as directional only

C.3 Response Rates

Response rate is one of the most important drivers of survey design. It determines how many invitations must be sent, how much fieldwork will cost, and how realistic it is to achieve the number of completed responses required by the research questions.

In a mail-to-web design, response rates are often modest. A practical working range is roughly 5% to 15%, with 10% to 12% a reasonable planning target for a motivated affinity community such as amateur radio when the outreach is credible and well executed.

Several factors can improve response rate:

  • a credible mailed invitation rather than a less formal ccontact method, like a postcard
  • clear explanation of purpose and relevance
  • a simple URL and QR code
  • one or more reminder contacts
  • a small incentive, especially a prepaid token incentive
  • careful branding and language that signal legitimacy to the target population

The proposal should not assume best-case response, but it is reasonable to plan around the upper part of the normal range for a niche technical community if the survey design and outreach are done well.

C.4 Invitations, Cost, and Cost per Complete

A large FCC license pool does not automatically translate into a large survey. What matters in practice is how many completed responses can be achieved at reasonable cost.

For planning purposes, a reasonable working design is a mailed letter invitation followed by a postcard reminder. That keeps the outreach simple, credible, and practical while giving the survey more than one chance to be seen. The exact cost will depend on printing, postage, mail handling, and whether incentives are used, but this kind of two-touch design is a useful basis for early planning.

Using a 10% response assumption for illustration:

  • 4,000 invitations may yield about 400 completes
  • 8,000 invitations may yield about 800 completes
  • 12,000 invitations may yield about 1,200 completes

The cost per invited sampled individual in a design like this will depend on the final production and mailing choices, but the main point is straightforward: response rate drives how many invitations must be sent, and invitation volume is a major driver of survey cost. This is why sample design, response assumptions, and budget need to be considered together in the planning process.

C.5 Why Subgroup Cells Matter

Top-line confidence is only part of the story. The more detailed the research questions become, the more important it is to understand the size of the resulting subgroup cells.

A good example is the current national mix of license classes. As of March 9, 2026, the U.S. individual license population is approximately:

  • Technician: 362,140 (49.2%)
  • General: 185,193 (25.1%)
  • Extra: 157,911 (21.4%)
  • Advanced: 27,049 (3.7%)
  • Novice: 4,400 (0.6%)

This distribution also has practical meaning for the research: the Technician class is not just the largest class, but the primary gateway into amateur radio today. The vast majority of new licensees enter through Technician, which means that any serious effort to understand Entry and early-stage participation must pay particular attention to that segment.

If a survey achieved 400 completed responses, and those responses reflected the national population mix reasonably well, the expected class counts would be approximately:

  • Technician: 197
  • General: 101
  • Extra: 86
  • Advanced: 15
  • Novice: 2

At 800 completed responses, the expected counts would be approximately:

  • Technician: 393
  • General: 201
  • Extra: 171
  • Advanced: 29
  • Novice: 5

At 1,200 completed responses, the expected counts would be approximately:

  • Technician: 590
  • General: 301
  • Extra: 257
  • Advanced: 44
  • Novice: 7

This example shows why a sample design that looks adequate at the overall population level may still be thin for smaller but strategically important groups.

C.6 Why Higher Sample Levels Can Matter

ARDC is unlikely to care only about the population as a whole. More useful questions often involve crossed segments such as:

  • license class by lifecycle stage
  • license class by engagement segment
  • geography by lifecycle stage
  • interest area by engagement risk

Once one dimension is crossed with another, the cells shrink quickly. That is one of the main reasons a design closer to 800 or 1,200 completes can become attractive. It does not simply make the top line more precise. It creates more room for the kinds of segment analysis that are often more useful for strategy and grantmaking.

A simple way to think about this is:

  • 400 is the practical target for a strong overall picture
  • 800 is the practical target when ARDC wants a stronger read on important subgroups
  • 1,200 is the practical target when ARDC wants more freedom to answer crossed-segment questions and investigate unexpected but important findings

C.7 Strategic Oversampling

Even with a larger overall sample, some groups may still be too important to leave to chance within the general sample. In those cases, the right answer may be strategic oversampling.

A likely example in this proposal is the population of individuals who received their first amateur radio license within roughly the 30 to 60 days prior to survey fielding. That group is especially important because it can inform Track 2’s understanding of feeder-system mix, motivating interests, barriers, and pathway patterns, while also supporting a possible cohort handoff into Track 1’s early integration period.

Another practical option may be to oversample Technicians, reflecting the reality that Technician is the main gateway into amateur radio and is likely to be central to many of the early participation and retention questions ARDC most wants to understand.

When a subgroup is oversampled, those respondents do not need to be removed from the full dataset. Instead:

  • the oversample can be used directly for subgroup analysis
  • overall population analysis can still include those respondents, provided the data are properly weighted so that the subgroup contributes at its true population share

C.8 What Early Planning Needs to Decide

One of the early responsibilities of the project will be to turn ARDC’s most important research questions into a practical sampling strategy. That means deciding:

  • what level of confidence is needed for top-line findings
  • which subgroup questions are most important
  • which crossed segments matter enough to justify a larger sample
  • which rare but strategic groups should be oversampled
  • what response-rate assumptions are realistic
  • what level of investment is appropriate to support those goals

Those decisions should drive the final sample plan.

Source for license-class counts: American Radio Relay League, individual licensees only, including FCC actions through Monday, March 9, 2026.