forked from saylordotorg/text_principles-of-sociological-inquiry-qualitative-and-quantitative-methods
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy paths10-04-a-word-of-caution-questions-to.html
87 lines (71 loc) · 12.4 KB
/
s10-04-a-word-of-caution-questions-to.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<link href="shared/bookhub.css" rel="stylesheet" type="text/css">
<title>A Word of Caution: Questions to Ask About Samples</title>
</head>
<body>
<div id=navbar-top class="navbar">
<div class="navbar-part left">
<a href="s10-03-sampling-in-quantitative-resea.html"><img src="shared/images/batch-left.png"></a> <a href="s10-03-sampling-in-quantitative-resea.html">Previous Section</a>
</div>
<div class="navbar-part middle">
<a href="index.html"><img src="shared/images/batch-up.png"></a> <a href="index.html">Table of Contents</a>
</div>
<div class="navbar-part right">
<a href="s11-00-survey-research-a-quantitative.html">Next Section</a> <a href="s11-00-survey-research-a-quantitative.html"><img src="shared/images/batch-right.png"></a>
</div>
</div>
<div id="book-content">
<div class="section" id="blackstone_1.0-ch07_s04" condition="start-of-chunk" version="5.0" lang="en">
<h2 class="title editable block">
<span class="title-prefix">7.4</span> A Word of Caution: Questions to Ask About Samples</h2>
<div class="learning_objectives editable block" id="blackstone_1.0-ch07_s04_n01">
<h3 class="title">Learning Objectives</h3>
<ol class="orderedlist" id="blackstone_1.0-ch07_s04_l01">
<li>Identify several questions we should ask about samples when reading the results of research.</li>
<li>Name some tenets worth keeping in mind with respect to responsibly reading research findings.</li>
</ol>
</div>
<p class="para editable block" id="blackstone_1.0-ch07_s04_p01">We read and hear about research results so often that we might overlook the need to ask important questions about where research participants come from and how they are identified for inclusion in a research project. It is easy to focus only on findings when we’re busy and when the really interesting stuff is in a study’s conclusions, not its procedures. But now that you have some familiarity with the variety of procedures for selecting study participants, you are equipped to ask some very important questions about the findings you read and to be a more responsible consumer of research.</p>
<div class="section" id="blackstone_1.0-ch07_s04_s01">
<h2 class="title editable block">Who Sampled, How Sampled, and for What Purpose?</h2>
<p class="para editable block" id="blackstone_1.0-ch07_s04_s01_p01">Have you ever been a participant in someone’s research? If you have ever taken an introductory psychology or sociology class at a large university, that’s probably a silly question to ask. Social science researchers on college campuses have a luxury that researchers elsewhere may not share—they have access to a whole bunch of (presumably) willing and able human guinea pigs. But that luxury comes at a cost—sample representativeness. One study of top academic journals in psychology found that over two-thirds (68%) of participants in studies published by those journals were based on samples drawn in the United States (Arnett, 2008).<span class="footnote" id="blackstone_1.0-fn07_009">Arnett, J. J. (2008). The neglected 95%: Why American psychology needs to become less American. <em class="emphasis">American Psychologist, 63</em>, 602–614.</span> Further, the study found that two-thirds of the work that derived from US samples published in the <em class="emphasis">Journal of Personality and Social Psychology</em> was based on samples made up entirely of American undergraduates taking psychology courses.</p>
<p class="para editable block" id="blackstone_1.0-ch07_s04_s01_p02">These findings certainly beg the question: What do we actually learn from social scientific studies and about whom do we learn it? That is exactly the concern raised by Joseph Henrich and colleagues (Henrich, Heine, & Norenzayan, 2010),<span class="footnote" id="blackstone_1.0-fn07_010">Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world? <em class="emphasis">Behavioral and Brain Sciences, 33</em>, 61–135.</span> authors of the article “The Weirdest People in the World?” In their piece, Henrich and colleagues point out that behavioral scientists very commonly make sweeping claims about human nature based on samples drawn only from WEIRD (Western, educated, industrialized, rich, and democratic) societies, and often based on even narrower samples, as is the case with many studies relying on samples drawn from college classrooms. As it turns out, many robust findings about the nature of human behavior when it comes to fairness, cooperation, visual perception, trust, and other behaviors are based on studies that excluded participants from outside the United States and sometimes excluded anyone outside the college classroom (Begley, 2010).<span class="footnote" id="blackstone_1.0-fn07_011"><em class="emphasis">Newsweek</em> magazine published an interesting story about Henrich and his colleague’s study: Begley, S. (2010). What’s really human? The trouble with student guinea pigs. Retrieved from <a class="link" target="_blank" href="http://www.newsweek.com/2010/07/23/what-s-really-human.html">http://www.newsweek.com/2010/07/23/what-s-really-human.html</a></span> This certainly raises questions about what we really know about human behavior as opposed to US resident or US undergraduate behavior. Of course not all research findings are based on samples of WEIRD folks like college students. But even then it would behoove us to pay attention to the population on which studies are based and the claims that are being made about to whom those studies apply.</p>
<p class="para editable block" id="blackstone_1.0-ch07_s04_s01_p03">In the preceding discussion, the concern is with researchers making claims about populations other than those from which their samples were drawn. A related, but slightly different, potential concern is <span class="margin_term"><a class="glossterm">sampling bias</a><span class="glossdef">Occurs when the elements selected for inclusion in a study do not represent the larger population from which they were drawn.</span></span>. Bias in sampling occurs when the elements selected for inclusion in a study do not represent the larger population from which they were drawn. For example, a poll conducted online by a newspaper asking for the public’s opinion about some local issue will certainly not represent the public since those without access to computers or the Internet, those who do not read that paper’s website, and those who do not have the time or interest will not answer the question.</p>
<p class="para editable block" id="blackstone_1.0-ch07_s04_s01_p04">Another thing to keep in mind is that just because a sample may be representative in all respects that a researcher thinks are relevant, there may be aspects that are relevant that didn’t occur to the researcher when she was drawing her sample. You might not think that a person’s phone would have much to do with their voting preferences, for example. But had pollsters making predictions about the results of the 2008 presidential election not been careful to include both cell phone–only and landline households in their surveys, it is possible that their predictions would have underestimated Barack Obama’s lead over John McCain because Obama was much more popular among cell-only users than McCain (Keeter, Dimock, & Christian, 2008).<span class="footnote" id="blackstone_1.0-fn07_012">Keeter, S., Dimock, M., & Christian, L. (2008). Calling cell phones in ’08 pre-election polls. <em class="emphasis">The Pew Research Center for the People and the Press</em>. Retrieved from <a class="link" target="_blank" href="http://people-press.org/http://people-press.org/files/legacy-pdf/cell-phone-commentary.pdf">http://people-press.org/http://people-press.org/files/legacy-pdf/cell-phone-commentary.pdf</a></span></p>
<p class="para editable block" id="blackstone_1.0-ch07_s04_s01_p05">So how do we know when we can count on results that are being reported to us? While there might not be any magic or always-true rules we can apply, there are a couple of things we can keep in mind as we read the claims researchers make about their findings. First, remember that sample quality is determined only by the sample actually obtained, not by the sampling method itself. A researcher may set out to administer a survey to a representative sample by correctly employing a random selection technique, but if only a handful of the people sampled actually respond to the survey, the researcher will have to be very careful about the claims he can make about his survey findings. Another thing to keep in mind, as demonstrated by the preceding discussion, is that researchers may be drawn to talking about implications of their findings as though they apply to some group other than the population actually sampled. Though this tendency is usually quite innocent and does not come from a place of malice, it is all too tempting a way to talk about findings; as consumers of those findings, it is our responsibility to be attentive to this sort of (likely unintentional) bait and switch.</p>
<p class="para editable block" id="blackstone_1.0-ch07_s04_s01_p06">Finally, keep in mind that a sample that allows for comparisons of theoretically important concepts or variables is certainly better than one that does not allow for such comparisons. In a study based on a nonrepresentative sample, for example, we can learn about the strength of our social theories by comparing relevant aspects of social processes. Klawiter’s previously mentioned study (1999)<span class="footnote" id="blackstone_1.0-fn07_013">Klawiter, M. (1999). Racing for the cure, walking women, and toxic touring: Mapping cultures of action within the Bay Area terrain of breast cancer. <em class="emphasis">Social Problems, 46</em>, 104–126.</span> of three carefully chosen breast cancer activist groups allowed her to contribute to our understandings of activism by addressing potential weaknesses in theories of social change.</p>
<p class="para editable block" id="blackstone_1.0-ch07_s04_s01_p07">At their core, questions about sample quality should address who has been sampled, how they were sampled, and for what purpose they were sampled. Being able to answer those questions will help you better understand, and more responsibly read, research results.</p>
<div class="key_takeaways editable block" id="blackstone_1.0-ch07_s04_s01_n01">
<h3 class="title">Key Takeaways</h3>
<ul class="itemizedlist" id="blackstone_1.0-ch07_s04_s01_l01">
<li>Sometimes researchers may make claims about populations other than those from whom their samples were drawn; other times they may make claims about a population based on a sample that is not representative. As consumers of research, we should be attentive to both possibilities.</li>
<li>A researcher’s findings need not be generalizable to be valuable; samples that allow for comparisons of theoretically important concepts or variables may yield findings that contribute to our social theories and our understandings of social processes.</li>
</ul>
</div>
<div class="exercises editable block" id="blackstone_1.0-ch07_s04_s01_n02">
<h3 class="title">Exercise</h3>
<ol class="orderedlist" id="blackstone_1.0-ch07_s04_s01_l02">
<li>Find any news story or blog entry that describes results from any social scientific study. How much detail is reported about the study’s sample? What sorts of claims are being made about the study’s findings, and to whom do they apply?</li>
</ol>
</div>
</div>
</div>
</div>
<div id=navbar-bottom class="navbar">
<div class="navbar-part left">
<a href="s10-03-sampling-in-quantitative-resea.html"><img src="shared/images/batch-left.png"></a> <a href="s10-03-sampling-in-quantitative-resea.html">Previous Section</a>
</div>
<div class="navbar-part middle">
<a href="index.html"><img src="shared/images/batch-up.png"></a> <a href="index.html">Table of Contents</a>
</div>
<div class="navbar-part right">
<a href="s11-00-survey-research-a-quantitative.html">Next Section</a> <a href="s11-00-survey-research-a-quantitative.html"><img src="shared/images/batch-right.png"></a>
</div>
</div>
</div>
<script type="text/javascript" src="shared/book.js"></script>
</body>
</html>