A medical journal used a questionnaire survey to determine whether a particular change in its format would increase its readership. Sixty-two percent of those who returned the questionnaire supported that change. On the basis made to introduce the new format.
What this question is testing
Conclusion
The journal's decision (introduce the new format) will increase readership.
Evidence
62% of returners supported the change.
Evaluate
This is a sampling issue. The decision was made based on a self-selected group — people who bothered to return the questionnaire. To trust the result, we need that group to actually represent the entire potential readership. Otherwise, the journal might have reshaped the magazine for the people who like writing back, but lost everyone else.
Goal
The right answer will tell us the responders' preferences mirror the preferences of the larger population the journal is trying to reach.
Reading along? Open the full official question in LawHub — we show a fragment here and keep the reasoning in our own words.