Wikipedia:Reliable sources checklist
|This page is an essay, containing the advice or opinions of one or more Wikipedia contributors. Essays are not Wikipedia policies or guidelines. Some essays represent widespread norms; others only represent minority viewpoints.||
Here's a checklist to help organize your evaluation of a reference.
- 1 Annotated checklist
- 2 Blank checklist (to copy and use)
- 3 Examples
- 4 Supplementary material
- What are we trying to do here?
- A succinct description of what we want to find out, e.g. "We want to find out what the actual area of Baltimore is" or whatever.
- What's the material that the ref supports?
- Is it contentious or contended?
- Contentious material is material that people might take a position on for ideological reasons. Contended material is just material that another editor has made a reasonable challenge to or where sources disagree. (Whether a person was born on April 19 or April 20 might be contended but it's not an ideological issue.) If it's not contentious, the source is only going to be wrong because of failure of diligence. If it's contentious, we also have to be aware of the possibility of deliberate bias.
- Does the ref indeed support the material?
- For instance, if it supports a quote, does the quote indeed appear in the ref, and so forth.
- Who is the author?
- Does the author have a Wikipedia article?
- This provides a quick-and-dirty (albeit imperfect) gauge of notability, which is not the same as veracity but is a data point. And the article and its links provide a good start in figuring out who the person is.
- What are the author's academic credentials and professional experience?
- What else has the author published?
- Is the author, or this work, cited in other reliable sources? In academic works?
- This is a rough indicator of post-publication peer review and acceptance. (Pro tips: if there is a DOI link, this will often show "where cited" for scientific publications. If it's a book, searching at Google Books (enclose the book title in quotes) can bring up cites.) WP:NJournals describes some other citation-finding tools and methods.)
- How does the author make a living?
- If he works for a salary, he has an incentive to not get fired. That means if he's a professor at an established university and that's his main source of income, he has an incentive to avoid outright mendacity, since that'll get you fired. Newspaper reporter, same thing. But other entities might encourage mendacity if it supports their mission. If he makes a living writing books or whatnot, his main financial incentive is to increase sales rather than necessarily get his facts right.
- What about reputation? Are there any big character markers?
- If he's been fired for plagiarism or indicted for perjury or successfully sued for libel or whatever, those are data points. It's probably best to be skeptical about other markers. Matters unrelated to his writing such as sex scandals might or might not indicate anything. Awards and accolades might matter some, depending on the source, but it's probably best to avoid giving much importance to man-of-the-year type logrolling. Most everyone has enemies, so a lot of people have been called a liar by somebody. If there's a pattern of disinterested people doing that then it might indicate a problem.
- Does the author have an opinion on the matter? On the continuum running from "utterly disinterested investigator or reporter" to "complete polemicist", where does this person fit?
- Even if he's utterly disinterested, he can still be inaccurate, of course. But if he's well to the right end of this continuum, that's a big red flag. It doesn't mean everything he says is inaccurate, of course, but it's an important data point. You have to be honest here – if he's a polemicist who supports your version of things, he's still a polemicist.
- Anything else?
- What is it?
- Is it a peer-reviewed scholarly journal, or a magazine (or newspaper) known to have an effective fact-checking operation?
- WP:RS, in its sections WP:SCHOLARSHIP and WP:NEWSORG, strongly (and sensibly) indicates that these are the only sources that are assumed to be reliable. (This doesn't prove that they are reliable in a given case, just that the assumption that they are is your starting point.) Everything else is up for debate. WP:RS is lengthy and there's a lot of hedging, but that's a reasonable summary.
- If not, is there any reason to believe that anyone has checked the author's facts?
- What's their circulation?
- Size doesn't prove anything, but it's a data point. The New England Journal of Medicine and the North Carolina Literary Review are both scholarly journals, but they're not equal. Ditto the New York Times and the Easton (Maryland) Gazette. A bigger operation means more resources for fact-checking, a bigger reputation to uphold, and greater likelihood of employing top-tier people.
- What about the publisher? What kind of outfit are they? What's their reputation?
- Do they have an agenda?
- What's their business incentive for veracity?
- Some magazines and newspapers rely on their reputation for veracity as part of their marketing model. If they don't pay attention to that they're eventually out of business. Others, not so much.
- Anything else?
- Does the source have standing to address the material?
- If the Easton Gazette is used to reference a claim that Easton High School opened in 1989, that's one thing. If it's used to reference a claim that Hitler died in Brazil, that's probably outside their area of competence. The Lancet is a good source, but if used to support material on the origins of American Baseball, maybe not so much.
- Anything else?
Blank checklist (to copy and use)
Most large magazines employ fact-checkers. Book publishers and most newspapers don't. They employ copy editors whose main brief is fixing grammar and style. Copy editors may check facts, but only on an ad hoc basis.
When you cite a book, you are relying almost entirely on the author. Book publishers have little incentive to worry about facts since people generally buy books based on the author rather than the publisher. For this reason books are seldom very reliable sources.
When you cite a newspaper, you are also relying on the author but mainly on the publication. Rather than checking facts, newspaper editors will expect reporters to check their own facts and they'll fire them if they don't and reporters know this. Newspapers do have an incentive to worry about facts since people do generally buy newspapers based (partly) on the paper's general reputation for veracity and not on the names of particular reporters. It depends a great deal on the newspaper, of course, and business incentives to get facts right varies a lot among newspapers, and so does editorial rigor.
Journalistic entities known to have good fact-checking operations
But don't just throw your favorite paper into this list. A good citation describing their fact-checking operation would be helpful.
- Der Spiegel. According to the Columbia Journalism Review (2010) they have about eighty full-time fact-checkers – more than any other publication on the planet – and a very rigorous and well-organized operation.
- The New Yorker is famous for the rigor of its editing (including fact checking). According to the Columbia Journalism Review (2010) they employ 16 fact-checkers. That's a lot for small weekly that doesn't specialize in hard news. Here's a New Yorker fact-checker recounting her service: "[A] long piece... received her full-time attention for three or four weeks... Each word in the piece that has even a shred of fact clinging to it is scrutinized... After an error gets into The New Yorker, heat-seeking missiles rise off the earth and home in on the author, the fact-checker, and the editor" Look at this: "Sometimes a phrase would contain hidden facts, as in 'Jane’s youngest son.' You’d have to check maternity and birth order, but you’d also have to confirm that Jane had at least three sons for one to be considered 'youngest.'" (If only two, it would be "younger", get it? These people are serious.) And "[T]he material originally appeared in The New Yorker, which, along with Time magazine, originated the practice of fact checking and has for many years been famous for the reliability of its content." - Ben Yagoda Here's from a 2012 piece in the Columbia Journalism Review: "To start checking a nonfiction piece, you begin by consulting the writer about how the piece was put together and using the writer’s sources as well as our own departmental sources. We then essentially take the piece apart and put it back together again. You make sure that the names and dates are right, but then if it is a John McPhee piece, you make sure that the USGS report that he read, he read correctly... Or if we describe the basis on which the FDA approved or disapproved the medical tests that ImClone used for Erbitux, then you need to find out what the complexities of that whole situation were."
- The Economist. (The Economist has a good reputation and a strong business incentive to get their facts straight since their core constituency includes high-powered decision-maker types who require reliable data and pay a lot for it, but so far we don't have a cite describing their operation, so... ) There have been sporadic complaints about fact-checking at The Economist, but as noted in The Atlantic, the Economist seems to employ fact checking in the book review process.
- British Broadcasting Corporation. The BBC has a rigorous set of guidelines including for accuracy, impartiality, fairness, and the reporting of controversial events such as war, terror and emergencies.
- PolitiFact deals with political statements. It has a business model of 1) checking facts – that's what they do and it's all they do, and 2) being strictly non-partisan. So based on this the assumption would be that they are fairly reliable. But we need more proof, so... 
Journalistic entities known to have bad (or no) fact-checking operations
- Some Forbes material, specifically that found at "www.forbes.com/sites/". Despite the Forbes nameplate, these are blog posts and are not fact-checked at all. According to the Columbia Journalism Review, "Around 250 to 300 stories go up on the forbes site each day. ... No matter their background or compensation, all contributors can publish their own work without so much as a cursory edit.
- Newsweek is an old name in American journalism, but as their print operation died they fell upon hard times; the 2012 Niall Ferguson fiasco showed that, at least as of 2012, they weren't fact-checking at all. (Poynter,NYTimes) Since 2013 they are owned by IBT Media and we're not sure of if their reliability has recovered any or if so how much.
- Beall's List is a list kept by Jeffrey Beall of questionable and shady open-access scientific journals. Articles in journals on Beall's List are presumed not reviewed and should usually be treated as blog posts.
- Esquire, an established American magazine, does not come off too well in this anecdote (from Jacobs, A. J. (2005). The Know-It-All. Simon & Schuster. p. 48. ISBN 978-0743250627.): "I was helping edit an article... my boss had sent it back to me, suggesting it was a little bland... I simply tacked on [a] sentence [to a quote]... And it worked. My boss liked it better. The problem was, I completely forgot to send the piece back to the [person quoted] to see if she was okay with my little addition. I had meant to -- I know you can't just insert something without the writer's approval. But I forgot... [and later] the [subject] is complaining that Esquire put words in her mouth... I feel terrible... I'm hoping this little scandal will blow over, and I think it will."
- On the one hand, the person knows they messed up and feels bad, and there was a "little scandal". On the other hand... why did this person even consider for one moment tacking on an extra sentence to a quote? And where was the fact-checker in all this -- where is the person taking the article from the editor and making sure the subject is called to verify the quote? There isn't one. And what is the meaning of "you can't just insert something without the writer's approval"? (If the article writer says "Sure, I don't care", it's OK then?) Where is the indication that person was put in fear for their career over this? I don't see that either. It's only one anecdote, but this is not a good look for Esquire.
- Smith, Sarah Harrison (2004). The Fact Checker's Bible. Anchor. ISBN 978-0385721066.
- D'Agata, John; Fingal, Jim (February 27, 2012). The Lifespan of a Fact. W. W. Norton & Company. ISBN 978-0393340730. Retrieved February 1, 2012.
An innovative essayist and his fact-checker do battle about the use of truth and the definition of nonfiction.
- Excerpt: John D'Agata and Jim Fingal (February 2012). "What Happened in Vegas". Harper's. Retrieved February 12, 2012.
- Review: Laura Miller (February 9, 2012). "In Defense of Fact Checking". Slate. Retrieved February 12, 2012.
- Review: Hannah Goldfield (February 9, 2012). "The Art of Fact-Checking". The New Yorker. Retrieved February 12, 2012.
- Craig Silverman (April 9, 2010). "Inside the World's Largest Fact Checking Operation". Columbia Journalism Review. Retrieved March 5, 2013.
- John McPhee (February 9, 2009). "Checkpoints (abstract)". The New Yoker. Retrieved September 2, 2011.
- Virginia Heffernan (August 20, 2010). "What 'Fact-Checking' Means Online". New York Times. Retrieved September 2, 2011.
- Ben Yagoda (March 20, 2013). "Fact Checking 'In Cold Blood'". Slate. Retrieved August 21, 2012.
- Portes, Jonathan (15 March 2012). "Not the Treasury view...: The Economist: fact check fail....". National Institute of Economic and Social Research blog. Retrieved March 5, 2013.
- Cox, Wendell (April 20, 2010). "Portland Myths & The Economist's Need for Fact Checking". Demographia Observations (blog). Retrieved March 5, 2013.
- McArdle, Megan (January 24, 2011). "Why Don't Publishers Check Facts". The Atlantic. Retrieved March 5, 2013.
The Economist pens one of its customarily acerbic book reviews in which it notes an extraordinary number of basic errors
- "Editorial Guidelines". BBC. Retrieved 19 December 2015.
- "Fact-Checking – Fakten finden, Fehler vermeiden" [Fact-Checking - find the facts, avoid mistakes]. Netzwerk Recherche. 2010. Retrieved September 2, 2011. (German)
- Craig Silverman (April 9, 2010). "Top fact checkers and news accuracy experts gather in Germany". Regret the Error. Retrieved September 2, 2011.
- Kevin Drum (January 4, 2011). "The Mystery of Fact Checking". Mother Jones. Retrieved September 2, 2011. It's a blog, so it's not a very reliable source itself... and the comments are completely unreliable of course, but interesting and probably true. We couldn't even think about using those in an article, but this isn't one, so read away.
- Brendan I. Koerner (May 23, 2003). "Who Uses Fact Checkers, Anyway?". Slate. Retrieved September 2, 2011.
- Katherine Goldstein (June 6, 2011). "Orgasm Guaranteed". Slate. Retrieved September 2, 2011. Fact-checking at Cosmopolitan.
- Ta-Nehisi Coates (August 21, 2012). "In Praise of Fact-Checkers". The Atlantic. Retrieved August 21, 2012. Coates comments on the Niall Ferguson affair and the downfall of Newsweek, and other matters.