Wikipedia:Reliable sources checklist

From Wikipedia, the free encyclopedia
Jump to: navigation, search

Here's a checklist to help organize your evaluation of a reference.

Annotated checklist[edit]

The goal

  • What are we trying to do here?
    • A succinct description of what we want to find out, e.g. "We want to find out what the actual area of Baltimore is" or whatever.

The material

  • What's the material that the ref supports?
  • Is it contentious or contended?
    • Contentious material is material that people might take a position on for ideological reasons. Contended material is just material that another editor has made a reasonable challenge to or where sources disagree. (Whether a person was born on April 19 or April 20 might be contended but its not an ideological issue.) If it's not contentious, the source is only going to be wrong because of failure of diligence. If it's contentious, we also have to be aware of the possibility of deliberate bias.
  • Does the ref indeed support the material?
    • For instance, if it supports a quote, does the quote indeed appear in the ref, and so forth.

The author

  • Who is the author?
  • Does the author have a Wikipedia article?
    • This provides a quick-and-dirty (albeit imperfect) gauge of notability, which is not the same as veracity but is a data point. And the article and its links provide a good start in figuring out who the person is.
  • What are the author's academic credentials and professional experience?
  • What else has the author published?
  • Is the author, or this work, cited in other reliable sources? In academic works?
    • This is a rough indicator of post-publication peer review and acceptance. (Pro tips: if there is a DOI link, this will often show "where cited" for scientific publications. If it's a book, searching at Google Books (enclose the book title in quotes) can bring up cites.) WP:NJournals describes some other citation-finding tools and methods.)
  • How does the author make a living?
    • If he works for a salary, he has an incentive to not get fired. That means if he's a professor at an established university and that's his main source of income, he has an incentive to avoid outright mendacity, since that'll get you fired. Newspaper reporter, same thing. But other entities might encourage mendacity if it supports their mission. If he makes a living writing books or whatnot, his main financial incentive is to increase sales rather than necessarily get his facts right.
  • What about reputation? Are there any big character markers?
    • If he's been fired for plagiarism or indicted for perjury or successfully sued for libel or whatever, those are data points. It's probably best to be skeptical about other markers. Matters unrelated to his writing such as sex scandals might or might not indicate anything. Awards and accolades might matter some, depending on the source, but it's probably best to avoid giving much importance to man-of-the-year type logrolling. Most everyone has enemies, so a lot of people have been called a liar by somebody. If there's a pattern of disinterested people doing that then it might indicate a problem.
  • Does the author have an opinion on the matter? On the continuum running from "utterly disinterested investigator or reporter" to "complete polemicist", where does this person fit?
    • Even if he's utterly disinterested, he can still be inaccurate, of course. But if he's well to the right end of this continuum, that's a big red flag. It doesn't mean everything he says is inaccurate, of course, but it's an important data point. You have to be honest here – if he's a polemicist who supports your version of things, he's still a polemicist.
  • Anything else?

The publication

  • What is it?
  • Is it a peer-reviewed scholarly journal, or a magazine (or newspaper) known to have an effective fact-checking operation?
    • WP:RS, in its sections WP:SCHOLARSHIP and WP:NEWSORG, strongly (and sensibly) indicates that these are the only sources that are assumed to be reliable. (This doesn't prove that they are reliable in a given case, just that the assumption that they are is your starting point.) Everything else is up for debate. WP:RS is lengthy and there's a lot of hedging, but that's a reasonable summary.
  • If not, is there any reason to believe that anyone has checked the author's facts?
  • What's their circulation?
    • Size doesn't prove anything, but it's a data point. The New England Journal of Medicine and the North Carolina Literary Review are both scholarly journals, but they're not equal. Ditto the New York Times and the Easton (Maryland) Gazette. A bigger operation means more resources for fact-checking, a bigger reputation to uphold, and greater liklihood of employing top-tier people.
  • What about the publisher? What kind of outfit are they? What's their reputation?
  • Do they have an agenda?
  • What's their business incentive for veracity?
    • Some magazines and newspapers rely on their reputation for veracity as part of their marketing model. If they don't pay attention to that they're eventually out of business. Others, not so much.
  • Anything else?

Other

  • Does the source have standing to address the material?
    • If the Easton Gazette is used to reference a claim that Easton High School opened in 1989, that's one thing. If it's used to reference a claim that Hitler died in Brazil, that's probably outside their area of competence. The Lancet is a good source, but if used to support material on the origins of American Baseball, maybe not so much.
  • Anything else?

Summary

Blank checklist (to copy and use)[edit]

Examples[edit]

Example #1[edit]

Supplementary material[edit]

About fact-checking[edit]

Most large magazines employ fact-checkers. Books publishers and most newspapers don't. They employ copy editors whose main brief is fixing grammar and style. Copy editors may check facts, but only on an ad hoc basis.

When you cite a book, you are relying almost entirely on the author. Book publishers have little incentive to worry about facts since people generally buy books based on the author rather than the publisher. For this reason books are seldom very reliable sources.

When you cite a newspaper, you are also relying on the author but mainly on the publication. Rather than checking facts, newspaper editors will expect reporters to check their own facts and they'll fire them if they don't and reporters know this. Newspapers do have an incentive to worry about facts since people do generally buy newspapers based (partly) on the paper's general reputation for veracity and not on the names of particular reporters. It depends a great deal on the newspaper, of course, and business incentives to get facts right varies a lot among newspapers, and so does editorial rigor.

Journalistic entities known to have good fact-checking operations[edit]

But don't just throw your favorite paper into this list. A good citation describing their fact-checking operation would be helpful.
  • Der Spiegel. According to the Columbia Journalism Review (2010) they have about eighty full time fact-checkers – more than any other publication on the planet – and a very rigorous and well-organized operation.[1]
  • The New Yorker is famous for the rigor of its editing (including fact checking). According to the Columbia Journalism Review (2010) they employ 16 fact-checkers.[1] That's a lot for small weekly that doesn't specialize in hard news. Here's a New Yorker fact-checker recounting her service: "[A] long piece... received her full-time attention for three or four weeks... Each word in the piece that has even a shred of fact clinging to it is scrutinized... After an error gets into The New Yorker, heat-seeking missiles rise off the earth and home in on the author, the fact-checker, and the editor"[2] Look at this: "Sometimes a phrase would contain hidden facts, as in 'Jane’s youngest son.' You’d have to check maternity and birth order, but you’d also have to confirm that Jane had at least three sons for one to be considered 'youngest.'"[3] (If only two, it would be "younger", get it? These people are serious.) And "[T]he material originally appeared in The New Yorker, which, along with Time magazine, originated the practice of fact checking and has for many years been famous for the reliability of its content." - Ben Yagoda[4]
  • The Economist. (The Economist has a good reputation and a strong business incentive to get their facts straight since their core constituency includes high-powered decision-maker types who require reliable data and pay a lot for it, but so far we don't have a cite describing their operation, so... [citation needed]) There have been sporadic complaints about fact-checking at The Economist,[5][6] but as noted in The Atlantic, the Economist seems to employ fact checking in the book review process.[7]

See also[edit]

Further reading[edit]

  • Smith, Sarah Harrison (2004). The Fact Checker's Bible. Anchor. ISBN 978-0385721066. 
  • D'Agata, John; Fingal, Jim (February 27, 2012). The Lifespan of a Fact. W. W. Norton & Company. ISBN 978-0393340730. Retrieved February 2012. "An innovative essayist and his fact-checker do battle about the use of truth and the definition of nonfiction." 

References[edit]

  1. ^ a b Craig Silverman (April 9, 2010). "Inside the World’s Largest Fact Checking Operation". Columbia Journalism Review. Retrieved March 5, 2013. 
  2. ^ John McPhee (February 9, 2009). "Checkpoints (abstract)". The New Yoker. Retrieved September 2, 2011. 
  3. ^ Virginia Heffernan (August 20, 2010). "What 'Fact-Checking' Means Online". New York Times. Retrieved September 2, 2011. 
  4. ^ Ben Yagoda (March 20, 2013). "Fact Checking 'In Cold Blood'". Slate. Retrieved August 21, 2012. 
  5. ^ Portes, Jonathan (15 March 2012). "Not the Treasury view...: The Economist: fact check fail....". National Institute of Economic and Social Research blog. Retrieved March 5, 2013. 
  6. ^ Cox, Wendell (April 20, 2010). "Portland Myths & The Economist’s Need for Fact Checking". Demographia Observations (blog). Retrieved March 5, 2013. 
  7. ^ McArdle, Megan (January 24, 2011). "Why Don't Publishers Check Facts". The Atlantic. Retrieved March 5, 2013. "The Economist pens one of its customarily acerbic book reviews in which it notes an extraordinary number of basic errors" 

External links[edit]