One of my favorite blogs recently brought my attention to an interesting editorial in the Oklahoman: Bogus “fact checking” undermines media credibility
The piece, written by the entire editorial board of the newspaper, begins with a “We wish it wasn’t so” fact:
THIS year Gallup found trust in the media “to report the news fully, accurately and fairly” has fallen to the lowest level in polling history. Just 32 percent voice a great deal or fair amount of trust.
Following this, they go on to talk about the new study from Chloe Lim, a Ph.D. student at Stanford University, called “Checking How Fact-checkers Check,” which “evaluates the performance of two major online fact-checkers, Politfact at the Tampa Bay Times and Fact Checker at The Washington Post, by comparing instances in which they analyze the same statements.”
Interestingly, only one in ten statements is covered by both organizations.* And as the Oklahoman sums things up:
…when fact checkers do examine the same statement, Lim says “there is little agreement in their ratings.”….
Of 70 statements evaluated by both organizations, Lim notes, 14 “received two completely opposite ratings from the fact-checkers.”
“While 14 may not seem as big a number, it implies that 1 out of every 5 times, one fact-checker considers a statement true while the other fact-checker flags the same statement as a lie,” Lim writes.
The agreement rate of fact checkers is “much lower than what is acceptable for social scientific coding.”
As Mark Hemingway, writing at The Weekly Standard, noted, “if you know what a garbage fire the issue of accuracy in social science is, that is really saying something.” Yet Lim’s findings will surprise few who have read media fact checks closely.
And the Oklahoman also speaks about other studies as well, leading to their conclusion:
Two studies, conducted by officials at the University of Minnesota and George Mason University, both determined PolitiFact was far more likely to rate Republican statements false than Democrat statements.
Put simply, media fact checkers routinely contradict one another, issue self-contradictory statements, and do so in ways that favor one political party over another. Based on Gallup polling, it seems those facts haven’t escaped public notice.
What about a site like Snopes? A fact-checking sites like Snopes has, historically, been generally respected because they were able to attain a reputation, across the spectrum, for real fairness and reliability. That said, things like this piece from Molly Ziegler Hemingway (yes, the wife of Mark H., pictured above), a conservative reporter I trust, make me less inclined to trust Snopes.
The fact that the Snopes article changed the day after the Molly Ziegler Hemingway article is a good sign. That said, that Snopes would have written what they did in the first place does not engender trust at all. Nor does the fact that there is no explanation on their page regarding a retraction or why the page was updated, etc.**
Of course, there is this question: Are you going to trust the Oklahoman editorial board? Are you going to trust that they gave an accurate summary of those studies? Are you going to trust the authors of the studies? If you are interested in this question, how much are you inclined to dig here? Are these rocks that you are willing to turn over yourself? Do you feel equipped to do so? Are you really?
We all live by trust of course. We have to make hard choices about when its appropriate not to “trust but verify,” an oxymoronic statement if there ever was one, but rather “be skeptical (yes, distrust!) and verify.”
*from the study, quoted in the article:
“Given the similarity between the criteria adopted by both fact-checkers, one would expect that a large number of statements will be fact-checked by both websites,” Lim writes. “However, among the 1,135 statements fact-checked by Politifact, only 6 percent (70 statements) were also evaluated by Fact Checker. … Of the 240 statements evaluated by Fact Checker, 175 statements were fact-checked by Fact Checker alone.”
**I lifted the text of the last two paragraphs preceding this footnote from my past piece here.