forked from saylordotorg/text_principles-of-sociological-inquiry-qualitative-and-quantitative-methods
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy paths14-unobtrusive-research-qualitati.html
482 lines (462 loc) · 71.1 KB
/
s14-unobtrusive-research-qualitati.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<link href="shared/bookhub.css" rel="stylesheet" type="text/css">
<title>Unobtrusive Research: Qualitative and Quantitative Approaches</title>
</head>
<body>
<div id=navbar-top class="navbar">
<div class="navbar-part left">
<a href="s13-field-research-a-qualitative-t.html"><img src="shared/images/batch-left.png"></a> <a href="s13-field-research-a-qualitative-t.html">Previous Chapter</a>
</div>
<div class="navbar-part middle">
<a href="index.html"><img src="shared/images/batch-up.png"></a> <a href="index.html">Table of Contents</a>
</div>
<div class="navbar-part right">
<a href="s15-other-methods-of-data-collecti.html">Next Chapter</a> <a href="s15-other-methods-of-data-collecti.html"><img src="shared/images/batch-right.png"></a>
</div>
</div>
<div id="book-content">
<div class="chapter" id="blackstone_1.0-ch11" version="5.0" lang="en">
<h1 class="title editable block">
<span class="title-prefix">Chapter 11</span> Unobtrusive Research: Qualitative and Quantitative Approaches</h1>
<div class="section" id="blackstone_1.0-ch11_s00">
<h2 class="title editable block">Why Unobtrusive Research?</h2>
<p class="para editable block" id="blackstone_1.0-ch11_s00_p01">Are female and male athletes at the professional and college levels treated equally? You might think, 40 years since the passing of Title IX (the civil rights law that prohibits sex discrimination in education including athletics) and with the growing visibility of women athletes in sports such as golf, basketball, hockey, and tennis, that the answer would be an easy yes. But Professor Michael Messner’s (2002)<span class="footnote" id="blackstone_1.0-fn11_001">Messner, M. A. (2002). <em class="emphasis">Taking the field: Women, men, and sports</em>. Minneapolis: University of Minnesota Press.</span> unobtrusive research shows otherwise, as does Professors Jo Ann M. Buysse and Melissa Sheridan Embser-Herbert’s (2004)<span class="footnote" id="blackstone_1.0-fn11_002">Buysse, J. A. M., & Embser-Herbert, M. S. (2004). Constructions of gender in sport: An analysis of intercollegiate media guide cover photographs. <em class="emphasis">Gender & Society, 18,</em> 66–81.</span> content analysis of college athletics media guide photographs. In fact, Buysse and Embser-Herbert’s unobtrusive research shows that traditional definitions of femininity are fiercely maintained through colleges’ visual representations of women athletes as passive and overtly feminine (as opposed to strong and athletic). In addition, Messner and colleagues’ (Messner, Duncan, & Jensen, 1993)<span class="footnote" id="blackstone_1.0-fn11_003">Messner, M. A., Duncan, M. C., & Jensen, K. (1993). Separating the men from the girls: The gendered language of televised sports. <em class="emphasis">Gender & Society, 7,</em> 121–137.</span> content analysis of verbal commentary in televised coverage of men’s and women’s sports shows that announcers’ comments vary depending on an athlete’s gender identity. Such commentary not only infantilizes women athletes but also asserts an ambivalent stance toward their accomplishments.</p>
<p class="para editable block" id="blackstone_1.0-ch11_s00_p02">Without unobtrusive research we might be inclined to think that more has changed for women athletes over the past 40 years than actually has changed.</p>
</div>
</div>
<div class="section" id="blackstone_1.0-ch11_s01" condition="start-of-chunk" version="5.0" lang="en">
<h2 class="title editable block">
<span class="title-prefix">11.1</span> Unobtrusive Research: What Is It and When to Use It?</h2>
<div class="learning_objectives editable block" id="blackstone_1.0-ch11_s01_n01">
<h3 class="title">Learning Objectives</h3>
<ol class="orderedlist" id="blackstone_1.0-ch11_s01_l01">
<li>Define unobtrusive research.</li>
<li>Define historical comparative research.</li>
</ol>
</div>
<p class="para editable block" id="blackstone_1.0-ch11_s01_p01">In this chapter, we explore unobtrusive methods of collecting data. <span class="margin_term"><a class="glossterm">Unobtrusive research</a><span class="glossdef">Methods of collecting data that don’t interfere with the subjects under study.</span></span> refers to methods of collecting data that don’t interfere with the subjects under study (because these methods are not <em class="emphasis">obtrusive</em>). Both qualitative and quantitative researchers use unobtrusive research methods. Unobtrusive methods share the unique quality that they do not require the researcher to interact with the people he or she is studying. It may seem strange that sociology, a discipline dedicated to understanding human social behavior, would employ a methodology that requires no interaction with human beings. But humans create plenty of evidence of their behaviors—they write letters to the editor of their local paper, they create various sources of entertainment for themselves such as movies and televisions shows, they consume goods, they walk on sidewalks, they lie on the grass in public parks. All these activities leave something behind—worn paths, trash, recorded shows, and printed papers. These are all potential sources of data for the unobtrusive researcher.</p>
<p class="para editable block" id="blackstone_1.0-ch11_s01_p02">Sociologists interested in history are likely to use unobtrusive methods, which are also well suited to comparative research. <span class="margin_term"><a class="glossterm">Historical comparative research</a><span class="glossdef">Research that focuses on more than one location or time period.</span></span> is “research that focuses either on one or more cases over time (the historical part) or on more than one nation or society at one point in time (the comparative part)” (Esterberg, 2002, p. 129).<span class="footnote" id="blackstone_1.0-fn11_004">Esterberg, K. G. (2002). <em class="emphasis">Qualitative methods in social research</em>. Boston, MA: McGraw-Hill.</span> While not all unobtrusive researchers necessarily conduct historical, comparative, or even some combination of historical and comparative work, unobtrusive methods are well suited to such work. As an example, Melissa Weiner (2010)<span class="footnote" id="blackstone_1.0-fn11_005">Weiner, M. (2010). <em class="emphasis">Power, protest, and the public schools: Jewish and African American struggles in New York City</em>. Piscataway, NJ: Rutgers University Press.</span> used a historical comparative approach to study racial barriers historically experienced by Jews and African Americans in New York City public schools. Weiner analyzed public records from several years of newspapers, trial transcripts, and several organizations as well as private manuscript collections to understand how parents, children, and other activists responded to inequality and worked to reform schools. Not only did this work inform readers about the little-known similarities between Jewish and African American experiences, but it also informs current debates over inequalities experienced in public schools today.</p>
<p class="para editable block" id="blackstone_1.0-ch11_s01_p03">In this chapter, we’ll examine content analysis as well as analysis of data collected by others. Both types of analysis have in common their use of data that do not require direct interaction with human subjects, but the particular type and source of data for each type of analysis differs. We’ll explore these similarities and differences in the following sections, after we look at some of the pros and cons of unobtrusive research methods.</p>
<div class="key_takeaways editable block" id="blackstone_1.0-ch11_s01_n02">
<h3 class="title">Key Takeaways</h3>
<ul class="itemizedlist" id="blackstone_1.0-ch11_s01_l02">
<li>Unobtrusive methods allow researchers to collect data without interfering with the subjects under study.</li>
<li>Historical comparative methods, which are unobtrusive, focus on changes in multiple cases over time or on more than one nation or society at a single point in time.</li>
</ul>
</div>
<div class="exercises editable block" id="blackstone_1.0-ch11_s01_n03">
<h3 class="title">Exercises</h3>
<ol class="orderedlist" id="blackstone_1.0-ch11_s01_l03">
<li>What are some additional sources of unobtrusive data, aside from those already mentioned, that sociologists might take interest in examining?</li>
<li>What locations might you like to compare in your own research? Are there changes that occur over time that you might be interested in exploring? Explain.</li>
</ol>
</div>
</div>
<div class="section" id="blackstone_1.0-ch11_s02" condition="start-of-chunk" version="5.0" lang="en">
<h2 class="title editable block">
<span class="title-prefix">11.2</span> Pros and Cons of Unobtrusive Research</h2>
<div class="learning_objectives editable block" id="blackstone_1.0-ch11_s02_n01">
<h3 class="title">Learning Objectives</h3>
<ol class="orderedlist" id="blackstone_1.0-ch11_s02_l01">
<li>Identify the major strengths of unobtrusive research.</li>
<li>Identify the major weaknesses of unobtrusive research.</li>
<li>Define researcher bias.</li>
<li>Define the Hawthorne effect.</li>
</ol>
</div>
<p class="para editable block" id="blackstone_1.0-ch11_s02_p01">As is true of the other research types examined in this text, unobtrusive research has a number of strengths and several weaknesses.</p>
<div class="section" id="blackstone_1.0-ch11_s02_s01">
<h2 class="title editable block">Strengths of Unobtrusive Research</h2>
<p class="para editable block" id="blackstone_1.0-ch11_s02_s01_p01">Researchers who seek evidence of what people actually <em class="emphasis">do</em>, as opposed to what they say they do (as in survey and interview research), might wish to consider using unobtrusive methods. Field researchers may also claim this advantage over interview and survey research, but field researchers cannot be certain about what effect their presence in the field may have on the people and the interactions that they observe. While unobtrusive research projects, like all research projects, face the risk of introducing <span class="margin_term"><a class="glossterm">researcher bias</a><span class="glossdef">Bias that occurs when the preconceptions of a researcher—either intentionally or, more typically, unintentionally—shape her or his findings.</span></span> into the work, researchers employing unobtrusive methods do not need to be concerned about the effect of the research <em class="emphasis">on their subjects</em>. This effect, known as the <span class="margin_term"><a class="glossterm">Hawthorne effect</a><span class="glossdef">A situation that occurs when research subjects, either intentionally or unintentionally, alter their behaviors because they know they are being studied.</span></span>, is not a concern for unobtrusive researchers because they do not interact directly with their research participants. In fact, this is one of the major strengths of unobtrusive research.</p>
<p class="para editable block" id="blackstone_1.0-ch11_s02_s01_p02">Another benefit of unobtrusive research is that it <em class="emphasis">can</em> be relatively low-cost compared to some of the other methods we’ve discussed. Because “participants” are generally inanimate objects as opposed to human beings, researchers may be able to access data without having to worry about paying participants for their time (though certainly travel to or access to some documents and archives can be costly).</p>
<p class="para editable block" id="blackstone_1.0-ch11_s02_s01_p03">Unobtrusive research is also pretty forgiving. It is far easier to correct mistakes made in data collection when conducting unobtrusive research than when using any of the other methods described in this text. Imagine what you would do, for example, if you realized at the end of conducting 50 in-depth interviews that you’d accidentally omitted two critical questions from your interview guide. What are your options? Reinterview all 50 participants? Try to figure out what they might have said based on their other responses? Reframe your research question? Scratch the project entirely? Obviously none of these options is ideal. The same problems arise if a mistake is made in survey research. For field researchers, the consequences of “messing up” during data collection can be even more disastrous. Imagine discovering after tagging along on a political candidate’s campaign that you needed a “do-over.” In this case, that simply isn’t an option. The campaign is over, and you’d need to find a new source of data. Fortunately for unobtrusive researchers, going back to the source of the data to gather more information or correct some problem in the original data collection is a relatively straightforward prospect.</p>
<p class="para editable block" id="blackstone_1.0-ch11_s02_s01_p04">Finally, as described in <a class="xref" href="blackstone_1.0-ch11_s01#blackstone_1.0-ch11_s01">Section 11.1 "Unobtrusive Research: What Is It and When to Use It?"</a>, unobtrusive research is well suited to studies that focus on processes that occur over time. While longitudinal surveys and long-term field observations are also suitable ways of gathering such information, they cannot examine processes that occurred decades before data collection began, nor are they the most cost-effective ways to examine long-ranging processes. Unobtrusive methods, on the other hand, enable researchers to investigate events and processes that have long since passed. They also do not rely on retrospective accounts, which may be subject to errors in memory, as some longitudinal surveys do.</p>
<p class="para editable block" id="blackstone_1.0-ch11_s02_s01_p05">In sum, the strengths of unobtrusive research include the following:</p>
<ol class="orderedlist editable block" id="blackstone_1.0-ch11_s02_s01_l01">
<li>There is no possibility for the Hawthorne effect.</li>
<li>The method is cost effective</li>
<li>It is easier in unobtrusive research than with other methods to correct mistakes.</li>
<li>Unobtrusive methods are conducive to examining processes that occur over time or in the past.</li>
</ol>
</div>
<div class="section" id="blackstone_1.0-ch11_s02_s02">
<h2 class="title editable block">Weaknesses of Unobtrusive Research</h2>
<p class="para editable block" id="blackstone_1.0-ch11_s02_s02_p01">While there are many benefits to unobtrusive research, this method also comes with a unique set of drawbacks. Because unobtrusive researchers analyze data that may have been created or gathered for purposes entirely different from the researcher’s aim, problems of validity sometimes arise in such projects. It may also be the case that data sources measuring whatever a researcher wishes to examine simply do not exist. This means that unobtrusive researchers may be forced to tweak their original research interests or questions to better suit the data that are available to them. Finally, it can be difficult in unobtrusive research projects to account for context. In a field research project, for example, the researcher is able to see what events lead up to some occurrence and observe how people respond to that occurrence. What this means for unobtrusive research is that while it can be difficult to ascertain <em class="emphasis">why</em> something occurred, we can gain a good understanding of <em class="emphasis">what</em> has occurred.</p>
<p class="para editable block" id="blackstone_1.0-ch11_s02_s02_p02">In sum, the weaknesses of unobtrusive research include the following:</p>
<ol class="orderedlist editable block" id="blackstone_1.0-ch11_s02_s02_l01">
<li>There may be potential problems with validity.</li>
<li>The topics or questions that can be investigated are limited by data availability.</li>
<li>It can be difficult to see or account for social context.</li>
</ol>
<div class="key_takeaways editable block" id="blackstone_1.0-ch11_s02_s02_n01">
<h3 class="title">Key Takeaways</h3>
<ul class="itemizedlist" id="blackstone_1.0-ch11_s02_s02_l02">
<li>Unobtrusive research is cost effective and allows for easier correction of mistakes than other methods of data collection do.</li>
<li>The Hawthorne effect, which occurs when research subjects alter their behaviors because they know they are being studied, is not a risk in unobtrusive research as it is in other methods of data collection.</li>
<li>Weaknesses of unobtrusive research include potential problems with validity, limitations in data availability, and difficulty in accounting for social context.</li>
</ul>
</div>
<div class="exercises editable block" id="blackstone_1.0-ch11_s02_s02_n02">
<h3 class="title">Exercises</h3>
<ol class="orderedlist" id="blackstone_1.0-ch11_s02_s02_l03">
<li>
<p class="para">Want to see the Hawthorne effect in action? Check out this totally nonscientific yet wholly entertaining application of the principle from Korea:</p>
<p class="simpara"></p>
<div class="mediaobject">
<a data-iframe-code='<iframe src="http://www.youtube.com/v/b_YAJtJmPLE" condition="http://img.youtube.com/vi/b_YAJtJmPLE/0.jpg" vendor="youtube" width="450" height="340" scalefit="1"></iframe>' href="http://www.youtube.com/v/b_YAJtJmPLE" class="replaced-iframe" onclick="return replaceIframe(this)">(click to see video)</a>
</div>
<p class="para" id="blackstone_1.0-ch11_s02_s02_p03">What evidence of the Hawthorne effect do you see in the video?</p>
</li>
<li>What do you view as the most important strength and the most important weakness of unobtrusive research? Why?</li>
</ol>
</div>
</div>
</div>
<div class="section" id="blackstone_1.0-ch11_s03" condition="start-of-chunk" version="5.0" lang="en">
<h2 class="title editable block">
<span class="title-prefix">11.3</span> Unobtrusive Data Collected by You</h2>
<div class="learning_objectives editable block" id="blackstone_1.0-ch11_s03_n01">
<h3 class="title">Learning Objectives</h3>
<ol class="orderedlist" id="blackstone_1.0-ch11_s03_l01">
<li>Define content analysis.</li>
<li>Describe the kinds of texts that content analysts analyze.</li>
<li>Name at least two examples of content analysis research.</li>
<li>Define primary and secondary sources, describe their differences, and provide an example of each.</li>
<li>Define physical traces and compare them to material artifacts.</li>
<li>Outline the differences between manifest content and latent content.</li>
<li>Discuss the differences between qualitative and quantitative content analysis.</li>
<li>Describe code sheets and their purpose.</li>
</ol>
</div>
<p class="para editable block" id="blackstone_1.0-ch11_s03_p01">This section focuses on how to gather data unobtrusively and what to do with those data once they have been collected. There are two main ways of gathering data unobtrusively: conducting a content analysis of existing texts and analyzing physical traces of human behavior. We’ll explore both approaches.</p>
<div class="section" id="blackstone_1.0-ch11_s03_s01">
<h2 class="title editable block">Content Analysis</h2>
<p class="para editable block" id="blackstone_1.0-ch11_s03_s01_p01">One way of conducting unobtrusive research is to analyze texts. Texts come in all kinds of formats. At its core, content analysis addresses the questions of “Who says what, to whom, why, how, and with what effect?” (Babbie, 2010, pp. 328–329).<span class="footnote" id="blackstone_1.0-fn11_006">Babbie, E. (2010). <em class="emphasis">The practice of social research</em> (12th ed.). Belmont, CA: Wadsworth.</span> <span class="margin_term"><a class="glossterm">Content analysis</a><span class="glossdef">A type of unobtrusive research that involves the study of human communications.</span></span> is a type of unobtrusive research that involves the study of human communications. Another way to think of content analysis is as a way of studying texts and their meaning. Here we use a more liberal definition of <em class="emphasis">text</em> than you might find in your dictionary. The text that content analysts investigate includes such things as actual written copy (e.g., newspapers or letters) and content that we might see or hear (e.g., speeches or other performances). Content analysts might also investigate more visual representations of human communication such as television shows, advertisements, or movies. The following table provides a few specific examples of the kinds of data that content analysts have examined in prior studies. Which of these sources of data might be of interest to you?</p>
<div class="table block" id="blackstone_1.0-ch11_s03_s01_t01" frame="all">
<p class="title"><span class="title-prefix">Table 11.1</span> Content Analysis Examples</p>
<table cellpadding="0" cellspacing="0">
<thead>
<tr>
<th>Data</th>
<th>Research question</th>
<th>Author(s) (year)</th>
</tr>
</thead>
<tbody>
<tr>
<td>Spam e-mails</td>
<td>What is the form, content, and quantity of unsolicited e-mails?</td>
<td>Berzins (2009)<span class="footnote" id="blackstone_1.0-fn11_007">Berzins, M. (2009). Spams, scams, and shams: Content analysis of unsolicited email. <em class="emphasis">International Journal of Technology, Knowledge, and Society, 5,</em> 143–154.</span>
</td>
</tr>
<tr>
<td>James Bond films</td>
<td>How are female characters portrayed in James Bond films, and what broader lessons can be drawn from these portrayals?</td>
<td>Neuendorf, Gore, Dalessandro, Janstova, and Snyder-Suhy (2010)<span class="footnote" id="blackstone_1.0-fn11_008">Neuendorf, K. A., Gore, T. D., Dalessandro, A., Janstova, P., & Snyder-Suhy, S. (2010). Shaken and stirred: A content analysis of women’s portrayals in James Bond films. <em class="emphasis">Sex Roles, 62,</em> 747–761.</span>
</td>
</tr>
<tr>
<td>Console video games</td>
<td>How is male and female sexuality portrayed in the best-selling console video games?</td>
<td>Downs and Smith (2010)<span class="footnote" id="blackstone_1.0-fn11_009">Downs, E., & Smith, S. L. (2010). Keeping abreast of hypersexuality: A video game character content analysis. <em class="emphasis">Sex Roles, 62,</em> 721–733.</span>
</td>
</tr>
<tr>
<td>Newspaper articles</td>
<td>How do newspapers cover closed-circuit television surveillance in Canada, and what are the implications of coverage for public opinion and policymaking?</td>
<td>Greenberg and Hier (2009)<span class="footnote" id="blackstone_1.0-fn11_010">Greenberg, J., & Hier, S. (2009). CCTV surveillance and the poverty of media discourse: A content analysis of Canadian newspaper coverage. <em class="emphasis">Canadian Journal of Communication, 34,</em> 461–486.</span>
</td>
</tr>
<tr>
<td>Pro-eating disorder websites</td>
<td>What are the features of pro-eating disorder websites, and what are the messages to which users may be exposed?</td>
<td>Borzekowski, Schenk, Wilson, and Peebles (2010)<span class="footnote" id="blackstone_1.0-fn11_011">Borzekowski, D. L. G., Schenk, S., Wilson, J. L., & Peebles, R. (2010). e-Ana and e-Mia: A content analysis of pro-eating disorder Web sites. <em class="emphasis">American Journal of Public Health, 100,</em> 1526–1534.</span>
</td>
</tr>
</tbody>
</table>
</div>
<p class="para editable block" id="blackstone_1.0-ch11_s03_s01_p02">One thing you might notice about <a class="xref" href="#blackstone_1.0-ch11_s03_s01_t01">Table 11.1 "Content Analysis Examples"</a> is that the data sources represent <span class="margin_term"><a class="glossterm">primary sources</a><span class="glossdef">Original sources of data that have not already been analyzed; “raw” sources.</span></span>. That is, they are original. <span class="margin_term"><a class="glossterm">Secondary sources</a><span class="glossdef">Sources of data that have been analyzed or otherwise tweaked for presentation by others; “cooked” sources.</span></span>, on the other hand, are those that have already been analyzed. Shulamit Reinharz offers a helpful way of distinguishing between these two types of sources in her methods text. She explains that while primary sources represent the “‘raw’ materials of history,” secondary sources are the “‘cooked’ analyses of those materials” (1992, p. 155).<span class="footnote" id="blackstone_1.0-fn11_012">Reinharz, S. (1992). <em class="emphasis">Feminist methods in social research</em>. New York, NY: Oxford University Press.</span> The distinction between primary and secondary sources is important for many aspects of social science, but it is <em class="emphasis">especially</em> important to understand when conducting content analysis. While there are certainly instances of content analysis in which secondary sources are analyzed, I think it is safe to say that it is more common for content analysts to analyze primary sources.</p>
<p class="para editable block" id="blackstone_1.0-ch11_s03_s01_p03">In those instances where secondary sources are analyzed, the researcher’s focus is usually on the process by which the original analyst or presenter of data reached his conclusions or on the choices that were made in terms of how and in what ways to present the data. For example, Ferree and Hall (1990)<span class="footnote" id="blackstone_1.0-fn11_013">Ferree, M. M., & Hall, E. J. (1990). Visual images of American society: Gender and race in introductory sociology textbooks. <em class="emphasis">Gender & Society, 4</em>(4), 500–533.</span> conducted a content analysis of introductory sociology textbooks, but their aim was not to learn about the content of sociology as a discipline. Instead, the researchers sought to learn how students are <em class="emphasis">taught</em> the subject of sociology and understand what images are presented to students as representative of sociology as a discipline.</p>
<p class="para editable block" id="blackstone_1.0-ch11_s03_s01_p04">Sometimes students new to research methods struggle to grasp the difference between a content analysis of secondary sources and a review of literature, which is discussed in <a class="xref" href="blackstone_1.0-ch05#blackstone_1.0-ch05">Chapter 5 "Research Design"</a>. In a review of literature, researchers analyze secondary materials to try to understand what we know, and what we don’t know, about a particular topic. The sources used to conduct a scholarly review of the literature are typically peer-reviewed sources, written by trained scholars, published in some academic journal or press, and based on empirical research that has been conducted using accepted techniques of data collection for the discipline (scholarly theoretical pieces are included in literature reviews as well). These sources are culled in a review of literature in order to arrive at some conclusion about our overall knowledge about a topic. Findings are generally taken at face value.</p>
<p class="para editable block" id="blackstone_1.0-ch11_s03_s01_p05">Conversely, a content analysis of scholarly literature would raise questions not raised in a literature review. A content analyst might examine scholarly articles to learn something about the authors (e.g., Who publishes what, where?), publication outlets (e.g., How well do different journals represent the diversity of the discipline?), or topics (e.g., How has the popularity of topics shifted over time?). A content analysis of scholarly articles would be a “study of the studies” as opposed to a “review of studies.” Perhaps, for example, a researcher wishes to know whether more men than women authors are published in the top-ranking journals in the discipline. The researcher could conduct a content analysis of different journals and count authors by gender (though this may be a tricky prospect if relying only on names to indicate gender). Or perhaps a researcher would like to learn whether or how various topics of investigation go in and out of style. She could investigate changes over time in topical coverage in various journals. In these latter two instances, the researcher is not aiming to summarize the content of the articles but instead is looking to learn something about how, why, or by whom particular articles came to be published.</p>
<p class="para editable block" id="blackstone_1.0-ch11_s03_s01_p06">Content analysis can be qualitative or quantitative, and often researchers will use both strategies to strengthen their investigations. In qualitative content analysis the aim is to identify themes in the text being analyzed and to identify the underlying meaning of those themes. A graduate student colleague of mine once conducted qualitative content analysis in her study of national identity in the United States. To understand how the boundaries of citizenship were constructed in the United States, Alyssa Goolsby (2007)<span class="footnote" id="blackstone_1.0-fn11_014">Goolsby, A. (2007). <em class="emphasis">U.S. immigration policy in the regulatory era: Meaning and morality in state discourses of citizenship</em> (Unpublished master’s thesis). Department of Sociology, University of Minnesota, Minneapolis, MN.</span> conducted a qualitative content analysis of key historical congressional debates focused on immigration law. Quantitative content analysis, on the other hand, involves assigning numerical values to raw data so that it can be analyzed using various statistical procedures. One of my research collaborators, Jason Houle, conducted a quantitative content analysis of song lyrics. Inspired by an article on the connections between fame, chronic self-consciousness (as measured by frequent use of first-person pronouns), and self-destructive behavior (Schaller, 1997),<span class="footnote" id="blackstone_1.0-fn11_015">Schaller, M. (1997). The psychological consequences of fame: Three tests of the self-consciousness hypothesis. <em class="emphasis">Journal of Personality, 65,</em> 291–309.</span> Houle counted first-person pronouns in Elliott Smith song lyrics. Houle found that Smith’s use of self-referential pronouns increased steadily from the time of his first album release in 1994 until his suicide in 2003 (2008).<span class="footnote" id="blackstone_1.0-fn11_016">Houle, J. (2008). Elliott Smith’s self referential pronouns by album/year. Prepared for teaching SOC 207, Research Methods, at Pennsylvania State University, Department of Sociology.</span> We’ll elaborate on how qualitative and quantitative researchers collect, code, and analyze unobtrusive data in the final portion of this section.</p>
</div>
<div class="section" id="blackstone_1.0-ch11_s03_s02">
<h2 class="title editable block">Indirect Measures</h2>
<p class="para editable block" id="blackstone_1.0-ch11_s03_s02_p01">Texts are not the only sort of data that researchers can collect unobtrusively. Unobtrusive researchers might also be interested in analyzing the evidence that humans leave behind that tells us something about who they are or what they do. This kind evidence includes the <span class="margin_term"><a class="glossterm">physical traces</a><span class="glossdef">Evidence left by people that tells us something about their behaviors or interactions.</span></span> left by humans and the <span class="margin_term"><a class="glossterm">material artifacts</a><span class="glossdef">Objects created by humans that tell us something about their beliefs, values, or norms.</span></span> that tell us something about their beliefs, values, or norms. Physical traces include such things as worn paths across campus, the materials in a landfill or in someone’s trash can (a data source William Rathje and colleagues [Rathje, 1992; Rathje & Murthy, 1992]<span class="footnote" id="blackstone_1.0-fn11_017">Rathje, W. (1992). How much alcohol do we drink? It’s a question…so to speak. <em class="emphasis">Garbage, 4,</em> 18–19; Rathje, W., & Murthy, C. (1992). Garbage demographics. <em class="emphasis">American Demographics, 14,</em> 50–55.</span> have used), indentations in furniture, or empty shelves in the grocery store. Examples of material artifacts include video games and video game equipment, sculptures, mementos left on gravestones, housing structures, or even kitchen utensils. What kinds of physical traces or material artifacts might be of interest to you?</p>
<p class="para editable block" id="blackstone_1.0-ch11_s03_s02_p02">I recently visited the National Museum of American History in Washington, DC. While there I saw an exhibit displaying chef Julia Child’s home kitchen, where she filmed many of her famous cooking shows. Seeing the kitchen made me wonder how cooking has changed over the past few decades since Child’s shows were on air. I wondered how the layout of our kitchens and the utensils and appliances they contain might influence how we entertain guests, how much time we spend preparing meals, and how much time we spend cleaning up afterward. Our use of particular kitchen gadgets and utensils might even indicate something about our social class identities.<span class="footnote" id="blackstone_1.0-fn11_018">Watch the following clip, featuring satirist Joe Queenan, from the PBS documentary <em class="emphasis">People Like Us</em> on social class in the United States: <a class="link" target="_blank" href="http://www.youtube.com/watch?v=j_Rtl3Y4EuI">http://www.youtube.com/watch?v=j_Rtl3Y4EuI</a>. The clip aptly demonstrates the sociological relevance of kitchen gadgets.</span> Answers to these questions have bearing on our norms and interactions as humans; thus they are just the sorts of questions sociologists using unobtrusive methods might be interested in answering. I snapped a few photos of the kitchen while at the museum. Though the glass surrounding the exhibit prevents ideal picture taking, I hope the photos in <a class="xref" href="#blackstone_1.0-ch11_s03_s02_f01">Figure 11.4</a> give you an idea of what I saw. Might the organizational scheme used in this kitchen, or the appliances that are either present or missing from it, shape the answers to the questions I pose above about human behaviors and interactions?</p>
<div class="figure medium editable block" id="blackstone_1.0-ch11_s03_s02_f01">
<p class="title"><span class="title-prefix">Figure 11.4</span> </p>
<img src="section_14/b440ad850de1db0c01f050808c60afd6.jpg">
<p class="para">A visit to Chef Julia Child’s kitchen at the National Museum of American History inspired the author to think about how kitchens, kitchen utensils, and kitchen appliances have changed over time.</p>
</div>
<p class="para editable block" id="blackstone_1.0-ch11_s03_s02_p03">One challenge with analyzing physical traces and material artifacts is that you generally don’t have access to the people who left the traces or created the artifacts that you are analyzing. (And if you did find a way to contact them, then your research would no longer qualify as unobtrusive!) It can be especially tricky to analyze meanings of these materials if they come from some historical or cultural context other than your own. Situating the traces or artifacts you wish to analyze both in their original contexts and in your own is not always easy and can lead to problems related to validity and reliability. How do you know that you are viewing an object or physical trace in the way that it was intended to be viewed? Do you have the necessary understanding or knowledge about the background of its original creators or users to understand where they were coming from when they created it?</p>
<p class="para editable block" id="blackstone_1.0-ch11_s03_s02_p04">Imagine an alien trying to understand some aspect of Western human culture simply by examining our artifacts. Cartoonist Mark Parisi demonstrates the misunderstanding that could ensue in his drawing featuring three very small aliens standing atop a toilet. One alien says, “Since water is the life-blood on this planet, this must be a temple of some sort.…Let’s stick around and see how they show their respect” (1989).<span class="footnote" id="blackstone_1.0-fn11_019">Parisi, M. (1989). Alien cartoon 6. <em class="emphasis">Off the Mark</em>. Retrieved from <a class="link" target="_blank" href="http://www.offthemark.com/aliens/aliens06.htm">http://www.offthemark.com/aliens/aliens06.htm</a></span> Without a contextual understanding of Western human culture, the aliens have misidentified the purpose of the toilet, and they will be in for quite a surprise when someone shows up to use it!</p>
<p class="para editable block" id="blackstone_1.0-ch11_s03_s02_p05">The point is that while physical traces and material artifacts make excellent sources of data, analyzing their meaning takes more than simply trying to understand them from your own contextual position. You must also be aware of who caused the physical trace or created the artifact, when they created it, why they created, and for whom they created it. Answering these questions will require accessing materials in addition to the traces or artifacts themselves. It may require accessing historical documents or, if a contemporary trace or artifact, perhaps another method of data collection such as interviews with its creators.</p>
</div>
<div class="section" id="blackstone_1.0-ch11_s03_s03">
<h2 class="title editable block">Analysis of Unobtrusive Data Collected by You</h2>
<p class="para editable block" id="blackstone_1.0-ch11_s03_s03_p01">Once you have identified the set of texts, physical traces, or artifacts that you would like to analyze, the next step is to figure out <em class="emphasis">how</em> you’ll analyze them. This step requires that you determine your procedures for coding, understand the difference between manifest and latent content, and understand how to identify patterns across your coded data. We’ll begin by discussing procedures for coding.</p>
<p class="para editable block" id="blackstone_1.0-ch11_s03_s03_p02">You might recall being introduced to coding procedures in <a class="xref" href="blackstone_1.0-ch09#blackstone_1.0-ch09">Chapter 9 "Interviews: Qualitative and Quantitative Approaches"</a>, where we discussed the coding of qualitative interview data. While the coding procedures used for written documents obtained unobtrusively may resemble those used to code interview data, many sources of unobtrusive data differ dramatically from written documents or transcripts. What if your data are sculptures or worn paths, or perhaps kitchen utensils, as in the previously discussed example? The idea of conducting open coding and focused coding on these sources as you would for a written document sounds a little silly, not to mention impossible. So how do we begin to identify patterns across the sculptures or worn paths or utensils we wish to analyze? One option is to take field notes as we observe our data and then code patterns in those notes. Let’s say, for example, that we’d like to analyze kitchen utensils. Taking field notes might be a useful approach were we conducting observations of people actually using utensils in a documentary or on a television program. (Remember, if we’re observing people <em class="emphasis">in person</em> then our method is no longer unobtrusive.)</p>
<p class="para editable block" id="blackstone_1.0-ch11_s03_s03_p03">If rather than observing people in documentaries or television shows our data include a collection of actual utensils, note taking may not be the most effective way to record our observations. Instead, we could create a <span class="margin_term"><a class="glossterm">code sheet</a><span class="glossdef">The instrument used by an unobtrusive researcher to record observations.</span></span> to record details about the utensils in our sample. A code sheet, sometimes referred to as a tally sheet in quantitative coding, is the instrument an unobtrusive researcher uses to record observations.</p>
<p class="para editable block" id="blackstone_1.0-ch11_s03_s03_p04">In the example of kitchen utensils, perhaps we’re interested in how utensils have changed over time. If we had access to sales records for utensils over the past 50 years, we could analyze the top-selling utensil for each year. To do so, we’d want to make some notes about each of the 50 utensils included in our sample. For each top-rated utensil, we might note its name, its purpose, and perhaps its price in current dollar amounts. We might also want to make some assessment about how easy or difficult it is to use or some other qualitative assessment about the utensil and its use or purpose. To rate the difficulty of use we could use a 5-point scale, with 1 being very easy to use and 5 being very difficult to use. We could even record other notes or observations about the utensils that may not occur to us until we actually see the utensils. Our code sheet might look something like the sample shown in <a class="xref" href="#blackstone_1.0-ch11_s03_s03_t01">Table 11.2 "Sample Code Sheet for Study of Kitchen Utensil Popularity Over Time"</a>. Note that the sample sheet contains columns only for 10 years’ worth of utensils. If you were to conduct this project, obviously you’d need to create a code sheet that allows you to record observations for each of the 50 items in your sample.</p>
<div class="table block" id="blackstone_1.0-ch11_s03_s03_t01" frame="all">
<p class="title"><span class="title-prefix">Table 11.2</span> Sample Code Sheet for Study of Kitchen Utensil Popularity Over Time</p>
<table cellpadding="0" cellspacing="0">
<thead>
<tr>
<th></th>
<th>1961</th>
<th>1962</th>
<th>1963</th>
<th>1964</th>
<th>1965</th>
<th>1966</th>
<th>1967</th>
<th>1968</th>
<th>1969</th>
<th>1970</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong class="emphasis bold">Utensil name</strong></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td><strong class="emphasis bold">Utensil purpose</strong></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td><strong class="emphasis bold">Price (in 2011 $)</strong></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td><strong class="emphasis bold">Ease of use (1–5 scale)</strong></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td><strong class="emphasis bold">Other notes</strong></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
</tr>
</tbody>
</table>
</div>
<p class="para editable block" id="blackstone_1.0-ch11_s03_s03_p05">As you can see, our code sheet will contain both qualitative and quantitative data. Our “ease of use” rating is a quantitative assessment; we can therefore conduct some statistical analysis of the patterns here, perhaps noting the mean value on ease of use for each decade we’ve observed. We could do the same thing with the data collected in the row labeled <em class="emphasis">Price</em>, which is also quantitative. The final row of our sample code sheet, containing notes about our impressions of the utensils we observe, will contain qualitative data. We may conduct open and focused coding on these notes to identify patterns across those notes. In both cases, whether the data being coded are quantitative or qualitative, the aim is to identify patterns across the coded data.</p>
<p class="para editable block" id="blackstone_1.0-ch11_s03_s03_p06">The <em class="emphasis">Purpose</em> row in our sample code sheet provides an opportunity for assessing both manifest and latent content. <span class="margin_term"><a class="glossterm">Manifest content</a><span class="glossdef">Content that is most visible, obvious, or at the surface.</span></span> is the content we observe that is most apparent; it is the surface content. This is in contrast to <span class="margin_term"><a class="glossterm">latent content</a><span class="glossdef">The underlying meaning of the surface content we observe.</span></span>, which is less obvious. Latent content refers to the underlying meaning of the surface content we observe. In the example of utensil purpose, we might say a utensil’s manifest content is the stated purpose of the utensil. The latent content would be our assessment of what it means that a utensil with a particular purpose is top rated. Perhaps after coding the manifest content in this category we see some patterns that tell us something about the meanings of utensil purpose. Perhaps we conclude, based on the meanings of top-rated utensils across five decades, that the shift from an emphasis on utensils designed to facilitate entertaining in the 1960s to those designed to maximize efficiency and minimize time spent in the kitchen in the 1980s reflects a shift in how (and how much) people spend time in their homes.</p>
<p class="para editable block" id="blackstone_1.0-ch11_s03_s03_p07">Kathleen Denny’s (2011)<span class="footnote" id="blackstone_1.0-fn11_020">Denny, K. (2011). Gender in context, content, and approach: Comparing gender messages in Girl Scout and Boy Scout handbooks. <em class="emphasis">Gender & Society, 25,</em> 27–47.</span> recent study of scouting manuals offers another excellent example of the differences between manifest and latent content. Denny compared Boy Scout and Girl Scout handbooks to understand gender socializing among scouts. By counting activity types described in the manuals, Denny learned from this <em class="emphasis">manifest content</em> that boys are offered more individual-based and more scientific activities while girls are offered more group-based and more artistic activities. Denny also analyzed the <em class="emphasis">latent meaning</em> of the messages that scouting handbooks portray about gender; she found that girls were encouraged to become “up-to-date traditional women” while boys were urged to adopt “an assertive heteronormative masculinity” (p. 27).</p>
<div class="key_takeaways editable block" id="blackstone_1.0-ch11_s03_s03_n01">
<h3 class="title">Key Takeaways</h3>
<ul class="itemizedlist" id="blackstone_1.0-ch11_s03_s03_l01">
<li>Content analysts study human communications.</li>
<li>The texts that content analysts analyze include actual written texts such as newspapers or journal entries as well as visual and auditory sources such as television shows, advertisements, or movies.</li>
<li>Content analysts most typically analyze primary sources, though in some instances they may analyze secondary sources.</li>
<li>Indirect measures that content analysts examine include physical traces and material artifacts.</li>
<li>Manifest content is apparent; latent content is underlying.</li>
<li>Content analysts use code sheets to collect data.</li>
</ul>
</div>
<div class="exercises editable block" id="blackstone_1.0-ch11_s03_s03_n02">
<h3 class="title">Exercises</h3>
<ol class="orderedlist" id="blackstone_1.0-ch11_s03_s03_l02">
<li>Identify a research question you could answer using unobtrusive research. Now state a testable hypothesis having to do with your research question. Next identify at least two potential sources of data you might analyze to answer your research question and test your hypothesis.</li>
<li>Create a code sheet for each of the two potential sources of data that you identified in the preceding exercise.</li>
</ol>
</div>
</div>
</div>
<div class="section" id="blackstone_1.0-ch11_s04" condition="start-of-chunk" version="5.0" lang="en">
<h2 class="title editable block">
<span class="title-prefix">11.4</span> Analyzing Others’ Data</h2>
<div class="learning_objectives editable block" id="blackstone_1.0-ch11_s04_n01">
<h3 class="title">Learning Objectives</h3>
<ol class="orderedlist" id="blackstone_1.0-ch11_s04_l01">
<li>Name at least two sources of publicly available quantitative data.</li>
<li>Name at least two sources of publicly available qualitative data.</li>
</ol>
</div>
<p class="para editable block" id="blackstone_1.0-ch11_s04_p01">One advantage (or disadvantage, depending on which parts of the research process you most enjoy) of unobtrusive research is that you may be able to skip the data collection phase altogether. Whether you wish to analyze qualitative data or quantitative data sources, there are a number of free data sets available to social researchers. This section introduces you to several of those sources.</p>
<p class="para editable block" id="blackstone_1.0-ch11_s04_p02">Many sources of quantitative data are publicly available. The General Social Survey (GSS), which was discussed in <a class="xref" href="blackstone_1.0-ch08#blackstone_1.0-ch08">Chapter 8 "Survey Research: A Quantitative Technique"</a>, is one of the most commonly used sources of publicly available data among quantitative researchers (<a class="link" target="_blank" href="http://www.norc.uchicago.edu/GSS+Website">http://www.norc.uchicago.edu/GSS+Website</a>). Data for the GSS have been collected regularly since 1972, thus offering social researchers the opportunity to investigate changes in Americans’ attitudes and beliefs over time. Questions on the GSS cover an extremely broad range of topics, from family life to political and religious beliefs to work experiences.</p>
<p class="para editable block" id="blackstone_1.0-ch11_s04_p03">Other sources of quantitative data include Add Health (<a class="link" target="_blank" href="http://www.cpc.unc.edu/projects/addhealth">http://www.cpc.unc.edu/projects/addhealth</a>), a study that was initiated in 1994 to learn about the lives and behaviors of adolescents in the United States, and the Wisconsin Longitudinal Study (<a class="link" target="_blank" href="http://www.ssc.wisc.edu/wlsresearch">http://www.ssc.wisc.edu/wlsresearch</a>), a study that has, for over 40 years, surveyed 10,000 women and men who graduated from Wisconsin high schools in 1957. Quantitative researchers interested in studying social processes outside of the United States also have many options when it comes to publicly available data sets. Data from the British Household Panel Study (<a class="link" target="_blank" href="http://www.iser.essex.ac.uk/bhps">http://www.iser.essex.ac.uk/bhps</a>), a longitudinal, representative survey of households in Britain, are freely available to those conducting academic research (private entities are charged for access to the data). The International Social Survey Programme (<a class="link" target="_blank" href="http://www.issp.org">http://www.issp.org</a>) merges the GSS with its counterparts in other countries around the globe. These represent just a few of the many sources of publicly available quantitative data.</p>
<p class="para editable block" id="blackstone_1.0-ch11_s04_p04">Unfortunately for qualitative researchers, far fewer sources of free, publicly available qualitative data exist. This is slowly changing, however, as technical sophistication grows and it becomes easier to digitize and share qualitative data. Despite comparatively fewer sources than for quantitative data, there are still a number of data sources available to qualitative researchers whose interests or resources limit their ability to collect data on their own. The Murray Research Archive Harvard, housed at the Institute for Quantitative Social Science at Harvard University, offers case histories and qualitative interview data (<a class="link" target="_blank" href="http://dvn.iq.harvard.edu/dvn/dv/mra">http://dvn.iq.harvard.edu/dvn/dv/mra</a>). The Global Feminisms project at the University of Michigan offers interview transcripts and videotaped oral histories focused on feminist activism; women’s movements; and academic women’s studies in China, India, Poland, and the United States.<span class="footnote" id="blackstone_1.0-fn11_021">These data are not free, though they are available at a reasonable price. See the Global Feminism’s order site for more on pricing: <a class="link" target="_blank" href="http://www.umich.edu/~glblfem/dvd.html">http://www.umich.edu/~glblfem/dvd.html</a>; <a class="link" target="_blank" href="http://www.umich.edu/~glblfem/index.html">http://www.umich.edu/~glblfem/index.html</a>.</span> At the University of Connecticut, the Oral History Office provides links to a number of other oral history sites (<a class="link" target="_blank" href="http://www.oralhistory.uconn.edu/links.html">http://www.oralhistory.uconn.edu/links.html</a>). Not all the links offer publicly available data, but many do. Finally, the Southern Historical Collection at University of North Carolina–Chapel Hill offers digital versions of many primary documents online such as journals, letters, correspondence, and other papers that document the history and culture of the American South (<a class="link" target="_blank" href="http://dc.lib.unc.edu/ead/archivalhome.php?CISOROOT=/ead">http://dc.lib.unc.edu/ead/archivalhome.php?CISOROOT=/ead</a>).</p>
<p class="para editable block" id="blackstone_1.0-ch11_s04_p05">Keep in mind that the resources mentioned here represent just a snapshot of the many sources of publicly available data that can be easily accessed via the web. <a class="xref" href="#blackstone_1.0-ch11_s04_t01">Table 11.3 "Sources of Publicly Available Data"</a> summarizes the data sources discussed in this section.</p>
<div class="table block" id="blackstone_1.0-ch11_s04_t01" frame="all">
<p class="title"><span class="title-prefix">Table 11.3</span> Sources of Publicly Available Data</p>
<table cellpadding="0" cellspacing="0">
<thead>
<tr>
<th>Organizational home</th>
<th>Focus/topic</th>
<th>Data</th>
<th>Web address</th>
</tr>
</thead>
<tbody>
<tr>
<td>National Opinion Research Center</td>
<td>General Social Survey; demographic, behavioral, attitudinal, and special interest questions; national sample</td>
<td>Quantitative</td>
<td><a class="link" target="_blank" href="http://www.norc.uchicago.edu/GSS+Website/">http://www.norc.uchicago.edu/GSS+Website/</a></td>
</tr>
<tr>
<td>Carolina Population Center</td>
<td>Add Health; longitudinal social, economic, psychological, and physical well-being of cohort in grades 7–12 in 1994</td>
<td>Quantitative</td>
<td><a class="link" target="_blank" href="http://www.cpc.unc.edu/projects/addhealth">http://www.cpc.unc.edu/projects/addhealth</a></td>
</tr>
<tr>
<td>Center for Demography of Health and Aging</td>
<td>Wisconsin Longitudinal Study; life course study of cohorts who graduated from high school in 1957</td>
<td>Quantitative</td>
<td><a class="link" target="_blank" href="http://www.ssc.wisc.edu/wlsresearch/">http://www.ssc.wisc.edu/wlsresearch/</a></td>
</tr>
<tr>
<td>Institute for Social & Economic Research</td>
<td>British Household Panel Survey; longitudinal study of British lives and well-being</td>
<td>Quantitative</td>
<td><a class="link" target="_blank" href="http://www.iser.essex.ac.uk/bhps">http://www.iser.essex.ac.uk/bhps</a></td>
</tr>
<tr>
<td>International Social Survey Programme</td>
<td>International data similar to GSS</td>
<td>Quantitative</td>
<td><a class="link" target="_blank" href="http://www.issp.org/">http://www.issp.org/</a></td>
</tr>
<tr>
<td>The Institute for Quantitative Social Science at Harvard University</td>
<td>Large archive of written data, audio, and video focused on many topics</td>
<td>Quantitative and qualitative</td>
<td><a class="link" target="_blank" href="http://dvn.iq.harvard.edu/dvn/dv/mra">http://dvn.iq.harvard.edu/dvn/dv/mra</a></td>
</tr>
<tr>
<td>Institute for Research on Women and Gender</td>
<td>Global Feminisms Project; interview transcripts and oral histories on feminism and women’s activism</td>
<td>Qualitative</td>
<td><a class="link" target="_blank" href="http://www.umich.edu/~glblfem/index.html">http://www.umich.edu/~glblfem/index.html</a></td>
</tr>
<tr>
<td>Oral History Office</td>
<td>Descriptions and links to numerous oral history archives</td>
<td>Qualitative</td>
<td><a class="link" target="_blank" href="http://www.oralhistory.uconn.edu/links.html">http://www.oralhistory.uconn.edu/links.html</a></td>
</tr>
<tr>
<td>UNC Wilson Library</td>
<td>Digitized manuscript collection from the Southern Historical Collection</td>
<td>Qualitative</td>
<td><a class="link" target="_blank" href="http://dc.lib.unc.edu/ead/archivalhome.php?CISOROOT=/ead">http://dc.lib.unc.edu/ead/archivalhome.php?CISOROOT=/ead</a></td>
</tr>
</tbody>
</table>
</div>
<p class="para editable block" id="blackstone_1.0-ch11_s04_p06">While the public and free sharing of data has become increasingly common over the years, and it is an increasingly common requirement of those who fund research, Harvard researchers recently learned of the potential dangers of making one’s data available to all (Parry, 2011).<span class="footnote" id="blackstone_1.0-fn11_022">Parry, M. (2011, July 10). Harvard researchers accused of breaching students’ privacy. <em class="emphasis">The Chronicle of Higher Education</em>. Retrieved from <a class="link" target="_blank" href="http://chronicle.com/article/Harvards-Privacy-Meltdown/128166">http://chronicle.com/article/Harvards-Privacy-Meltdown/128166</a></span> In 2008, Professor Nicholas Christakis, Jason Kaufman, and colleagues, of Harvard’s Berkman Center for Internet & Society, rolled out the first wave of their data collected from the profiles of 1,700 Facebook users (2008).<span class="footnote" id="blackstone_1.0-fn11_023">Berkman Center for Internet & Society. (2008, September 25). Tastes, ties, and time: Facebook data release. Retrieved from <a class="link" target="_blank" href="http://cyber.law.harvard.edu/node/4682">http://cyber.law.harvard.edu/node/4682</a></span> But shortly thereafter, the researchers were forced to deny public access to the data after it was discovered that subjects could easily be identified with some careful mining of the data set. Perhaps only time and additional experience will tell what the future holds for increased access to data collected by others.</p>
<div class="key_takeaways editable block" id="blackstone_1.0-ch11_s04_n02">
<h3 class="title">Key Takeaways</h3>
<ul class="itemizedlist" id="blackstone_1.0-ch11_s04_l02">
<li>Previously collected data sources enable researchers to conduct analysis without having to collect any of their own data.</li>
<li>Some publicly available data sources are quantitative; others are qualitative.</li>
</ul>
</div>
<div class="exercises editable block" id="blackstone_1.0-ch11_s04_n03">
<h3 class="title">Exercises</h3>
<ol class="orderedlist" id="blackstone_1.0-ch11_s04_l03">
<li>If you’re interested in learning about additional sources of publicly available data, check out the American Sociological Association’s lengthy and thorough list of public data resources: <a class="link" target="_blank" href="http://www2.asanet.org/student/pubdata00a.html">http://www2.asanet.org/student/pubdata00a.html</a>. Which of those that you read about on the site are of most interest to you? Why?</li>
<li>Read <em class="emphasis">The Chronicle of Higher Education</em> article on Harvard’s recent “privacy meltdown”: <a class="link" target="_blank" href="http://chronicle.com/article/Harvards-Privacy-Meltdown/128166/">http://chronicle.com/article/Harvards-Privacy-Meltdown/128166/</a>. What do you think the future holds for the public sharing of data? How might the incident described in the <em class="emphasis">Chronicle</em> article shape how, when, and whether public sharing of social scientific data occurs?</li>
</ol>
</div>
</div>
<div class="section" id="blackstone_1.0-ch11_s05" condition="start-of-chunk" version="5.0" lang="en">
<h2 class="title editable block">
<span class="title-prefix">11.5</span> Reliability in Unobtrusive Research</h2>
<div class="learning_objectives editable block" id="blackstone_1.0-ch11_s05_n01">
<h3 class="title">Learning Objectives</h3>
<ol class="orderedlist" id="blackstone_1.0-ch11_s05_l01">
<li>Define <em class="emphasis">stability</em> and describe strategies for overcoming problems of stability.</li>
<li>Define <em class="emphasis">reproducibility</em> and describe strategies for overcoming problems of reproducibility.</li>
<li>Define <em class="emphasis">accuracy</em> and describe strategies for overcoming problems of accuracy.</li>
</ol>
</div>
<p class="para editable block" id="blackstone_1.0-ch11_s05_p01">This final section of the chapter investigates a few particularities related to reliability in unobtrusive research projects (Krippendorff, 2009)<span class="footnote" id="blackstone_1.0-fn11_024">Krippendorff, K. (2009). Testing the reliability of content analysis data: What is involved and why. In K. Krippendorff & M. A. Bock (Eds.), <em class="emphasis">The content analysis reader</em> (pp. 350–357). Thousand Oaks, CA: Sage.</span> that warrant our attention. These particularities have to do with how and by whom the coding of data occurs. Issues of stability, reproducibility, and accuracy all speak to the unique problems—and opportunities—with establishing reliability in unobtrusive research projects.</p>
<p class="para editable block" id="blackstone_1.0-ch11_s05_p02"><span class="margin_term"><a class="glossterm">Stability</a><span class="glossdef">In terms of reliability, refers to the extent to which the results of coding vary across different time periods.</span></span> refers to the extent to which the results of coding vary across different time periods. If stability is a problem, it will reveal itself when the same person codes the same content at different times and comes up with different results. Coding is said to be stable when the same content has been coded multiple times by the same person with the same result each time. If you discover problems of instability in your coding procedures, it is possible that your coding rules are ambiguous and need to clarified. Ambiguities in the text itself might also contribute to problems of stability. While you cannot alter your original textual data sources, simply being aware of possible ambiguities in the data as you code may help reduce the likelihood of problems with stability. It is also possible that problems with stability may result from a simple coding error, such as inadvertently jotting a <em class="emphasis">1</em> instead of a <em class="emphasis">10</em> on your code sheet.</p>
<p class="para editable block" id="blackstone_1.0-ch11_s05_p03"><span class="margin_term"><a class="glossterm">Reproducibility</a><span class="glossdef">The extent to which one’s coding procedures will garner the same results when the same text is coded by different people.</span></span>, sometimes referred to as intercoder reliability (Lombard, Snyder-Duch, & Campanella Bracken, 2010),<span class="footnote" id="blackstone_1.0-fn11_025">Lombard, M., Snyder-Duch, J., & Campanella Bracken, C. (2004). Practical resources for assessing and reporting intercoder reliability in content analysis research projects. Retrieved from <a class="link" target="_blank" href="http://astro.temple.edu/~lombard/reliability">http://astro.temple.edu/~lombard/reliability</a></span> is the extent to which one’s coding procedures will result in the same results when the same text is coded by different people. Cognitive differences among the individuals coding data may result in problems with reproducibility, as could ambiguous coding instructions. Random coding errors might also cause problems. One way of overcoming problems of reproducibility is to have coders code together. While working as a graduate research assistant, I participated in a content analysis project in which four individuals shared the responsibility for coding data. To reduce the potential for reproducibility problems with our coding, we conducted our coding at the same time in the same room, while sitting around a large, round table. We coded at the same time in the same room so that we could consult one another when we ran into problems or had questions about what we were coding. Resolving those ambiguities together meant that we grew to have a shared understanding of how to code various bits of data.</p>
<p class="para editable block" id="blackstone_1.0-ch11_s05_p04">Finally, <span class="margin_term"><a class="glossterm">accuracy</a><span class="glossdef">The extent to which one’s coding procedures correspond to some preexisting standard.</span></span> refers to the extent to which one’s coding procedures correspond to some preexisting standard. This presumes that a standard coding strategy has already been established for whatever text you’re analyzing. It may not be the case that official standards have been set, but perusing the prior literature for the collective wisdom on coding on your particular area is time well spent. Scholarship focused on similar data or coding procedures will no doubt help you to clarify and improve your own coding procedures.</p>
<div class="key_takeaways editable block" id="blackstone_1.0-ch11_s05_n02">
<h3 class="title">Key Takeaways</h3>
<ul class="itemizedlist" id="blackstone_1.0-ch11_s05_l02">
<li>Stability can become an issue in unobtrusive research project when the results of coding by the same person vary across different time periods.</li>
<li>Reproducibility has to do with multiple coders’ results being the same for the same text.</li>
<li>Accuracy refers to the extent to which one’s coding procedures correspond to some preexisting standard.</li>
</ul>
</div>
<div class="exercises editable block" id="blackstone_1.0-ch11_s05_n03">
<h3 class="title">Exercise</h3>
<ol class="orderedlist" id="blackstone_1.0-ch11_s05_l03">
<li>With a peer, create a code sheet that you can use to record instances of violence on the television program of your choice. Now, on your own, watch two or three episodes of that program, coding for instances of violence as you go along. Your peer should do the same. Finally, compare your code sheet to that of your peer. How similar were you in your coding? Where do your coding results differ, and why? Which issues of reliability may be relevant?</li>
</ol>
</div>
</div>
</div>
<div id=navbar-bottom class="navbar">
<div class="navbar-part left">
<a href="s13-field-research-a-qualitative-t.html"><img src="shared/images/batch-left.png"></a> <a href="s13-field-research-a-qualitative-t.html">Previous Chapter</a>
</div>
<div class="navbar-part middle">
<a href="index.html"><img src="shared/images/batch-up.png"></a> <a href="index.html">Table of Contents</a>
</div>
<div class="navbar-part right">
<a href="s15-other-methods-of-data-collecti.html">Next Chapter</a> <a href="s15-other-methods-of-data-collecti.html"><img src="shared/images/batch-right.png"></a>
</div>
</div>
</div>
<script type="text/javascript" src="shared/book.js"></script>
</body>
</html>