forked from saylordotorg/text_principles-of-sociological-inquiry-qualitative-and-quantitative-methods
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy paths14-05-reliability-in-unobtrusive-res.html
82 lines (66 loc) · 8.19 KB
/
s14-05-reliability-in-unobtrusive-res.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<link href="shared/bookhub.css" rel="stylesheet" type="text/css">
<title>Reliability in Unobtrusive Research</title>
</head>
<body>
<div id=navbar-top class="navbar">
<div class="navbar-part left">
<a href="s14-04-analyzing-others-data.html"><img src="shared/images/batch-left.png"></a> <a href="s14-04-analyzing-others-data.html">Previous Section</a>
</div>
<div class="navbar-part middle">
<a href="index.html"><img src="shared/images/batch-up.png"></a> <a href="index.html">Table of Contents</a>
</div>
<div class="navbar-part right">
<a href="s15-01-focus-groups.html">Next Section</a> <a href="s15-01-focus-groups.html"><img src="shared/images/batch-right.png"></a>
</div>
</div>
<div id="book-content">
<div class="section" id="blackstone_1.0-ch11_s05" condition="start-of-chunk" version="5.0" lang="en">
<h2 class="title editable block">
<span class="title-prefix">11.5</span> Reliability in Unobtrusive Research</h2>
<div class="learning_objectives editable block" id="blackstone_1.0-ch11_s05_n01">
<h3 class="title">Learning Objectives</h3>
<ol class="orderedlist" id="blackstone_1.0-ch11_s05_l01">
<li>Define <em class="emphasis">stability</em> and describe strategies for overcoming problems of stability.</li>
<li>Define <em class="emphasis">reproducibility</em> and describe strategies for overcoming problems of reproducibility.</li>
<li>Define <em class="emphasis">accuracy</em> and describe strategies for overcoming problems of accuracy.</li>
</ol>
</div>
<p class="para editable block" id="blackstone_1.0-ch11_s05_p01">This final section of the chapter investigates a few particularities related to reliability in unobtrusive research projects (Krippendorff, 2009)<span class="footnote" id="blackstone_1.0-fn11_024">Krippendorff, K. (2009). Testing the reliability of content analysis data: What is involved and why. In K. Krippendorff & M. A. Bock (Eds.), <em class="emphasis">The content analysis reader</em> (pp. 350–357). Thousand Oaks, CA: Sage.</span> that warrant our attention. These particularities have to do with how and by whom the coding of data occurs. Issues of stability, reproducibility, and accuracy all speak to the unique problems—and opportunities—with establishing reliability in unobtrusive research projects.</p>
<p class="para editable block" id="blackstone_1.0-ch11_s05_p02"><span class="margin_term"><a class="glossterm">Stability</a><span class="glossdef">In terms of reliability, refers to the extent to which the results of coding vary across different time periods.</span></span> refers to the extent to which the results of coding vary across different time periods. If stability is a problem, it will reveal itself when the same person codes the same content at different times and comes up with different results. Coding is said to be stable when the same content has been coded multiple times by the same person with the same result each time. If you discover problems of instability in your coding procedures, it is possible that your coding rules are ambiguous and need to clarified. Ambiguities in the text itself might also contribute to problems of stability. While you cannot alter your original textual data sources, simply being aware of possible ambiguities in the data as you code may help reduce the likelihood of problems with stability. It is also possible that problems with stability may result from a simple coding error, such as inadvertently jotting a <em class="emphasis">1</em> instead of a <em class="emphasis">10</em> on your code sheet.</p>
<p class="para editable block" id="blackstone_1.0-ch11_s05_p03"><span class="margin_term"><a class="glossterm">Reproducibility</a><span class="glossdef">The extent to which one’s coding procedures will garner the same results when the same text is coded by different people.</span></span>, sometimes referred to as intercoder reliability (Lombard, Snyder-Duch, & Campanella Bracken, 2010),<span class="footnote" id="blackstone_1.0-fn11_025">Lombard, M., Snyder-Duch, J., & Campanella Bracken, C. (2004). Practical resources for assessing and reporting intercoder reliability in content analysis research projects. Retrieved from <a class="link" target="_blank" href="http://astro.temple.edu/~lombard/reliability">http://astro.temple.edu/~lombard/reliability</a></span> is the extent to which one’s coding procedures will result in the same results when the same text is coded by different people. Cognitive differences among the individuals coding data may result in problems with reproducibility, as could ambiguous coding instructions. Random coding errors might also cause problems. One way of overcoming problems of reproducibility is to have coders code together. While working as a graduate research assistant, I participated in a content analysis project in which four individuals shared the responsibility for coding data. To reduce the potential for reproducibility problems with our coding, we conducted our coding at the same time in the same room, while sitting around a large, round table. We coded at the same time in the same room so that we could consult one another when we ran into problems or had questions about what we were coding. Resolving those ambiguities together meant that we grew to have a shared understanding of how to code various bits of data.</p>
<p class="para editable block" id="blackstone_1.0-ch11_s05_p04">Finally, <span class="margin_term"><a class="glossterm">accuracy</a><span class="glossdef">The extent to which one’s coding procedures correspond to some preexisting standard.</span></span> refers to the extent to which one’s coding procedures correspond to some preexisting standard. This presumes that a standard coding strategy has already been established for whatever text you’re analyzing. It may not be the case that official standards have been set, but perusing the prior literature for the collective wisdom on coding on your particular area is time well spent. Scholarship focused on similar data or coding procedures will no doubt help you to clarify and improve your own coding procedures.</p>
<div class="key_takeaways editable block" id="blackstone_1.0-ch11_s05_n02">
<h3 class="title">Key Takeaways</h3>
<ul class="itemizedlist" id="blackstone_1.0-ch11_s05_l02">
<li>Stability can become an issue in unobtrusive research project when the results of coding by the same person vary across different time periods.</li>
<li>Reproducibility has to do with multiple coders’ results being the same for the same text.</li>
<li>Accuracy refers to the extent to which one’s coding procedures correspond to some preexisting standard.</li>
</ul>
</div>
<div class="exercises editable block" id="blackstone_1.0-ch11_s05_n03">
<h3 class="title">Exercise</h3>
<ol class="orderedlist" id="blackstone_1.0-ch11_s05_l03">
<li>With a peer, create a code sheet that you can use to record instances of violence on the television program of your choice. Now, on your own, watch two or three episodes of that program, coding for instances of violence as you go along. Your peer should do the same. Finally, compare your code sheet to that of your peer. How similar were you in your coding? Where do your coding results differ, and why? Which issues of reliability may be relevant?</li>
</ol>
</div>
</div>
</div>
<div id=navbar-bottom class="navbar">
<div class="navbar-part left">
<a href="s14-04-analyzing-others-data.html"><img src="shared/images/batch-left.png"></a> <a href="s14-04-analyzing-others-data.html">Previous Section</a>
</div>
<div class="navbar-part middle">
<a href="index.html"><img src="shared/images/batch-up.png"></a> <a href="index.html">Table of Contents</a>
</div>
<div class="navbar-part right">
<a href="s15-01-focus-groups.html">Next Section</a> <a href="s15-01-focus-groups.html"><img src="shared/images/batch-right.png"></a>
</div>
</div>
</div>
<script type="text/javascript" src="shared/book.js"></script>
</body>
</html>