User:Herostratus/Deep vetting

Everything you know is wrong

So you know, criminal trials have long depended on eyewitnesses. Well but in fact eyewitness reports are really unreliable (And there are lots of sources saying this, google if you like). I imagine they're still used, because you have do something. You've got a guy, your spidey-sense tells you he's probably the perp, all you really have that's solid is a couple eyewitness accounts -- "I was there. I saw him. That's the guy". Well what are you going to do, nothing? Not use that? And not get a conviction?

Well, so, our "reliable source", you know, are a lot less reliable than you think. I get that there's big pressure to use them -- your spidey-sense tells you the statement is true, and here you've the AAA-level New York Times to back you up. Heck when an article go goes to WP:AFD there's the Times has it's own special button for you to check.

Well I mean here's the NY Times with "violent crime is up sharply [in LA, last year]". But here we have that's dead wrong -- Murders were up. But violent crime was down, 2%. Assault, rape, mugging, all down. Murders and violent crimes, very different things. (Source is FBI Uniform Crime Reporting). So if you use the Times to ref "In 2020, violent crime was up sharply in LA", exactly the statement the Times makes, you're telling the reader false things.

"I am so tired of this crap. How hard is it for reporters to look up the latest crime data?" says Kevin Drum. He's tired of this crap because the Times does this now and then. Well the Times has to put out a huge paper every single day, and has deadlines, and that's probably why it is hard, for reporters to always look up the latest crime data or whatnot. Was the statement fact-checked, either by the reporter herself or an independent fact-checker? Not in this instance apparently.

All your sources are less reliable than you think.

Moving on... at a lot of sources, the fact-checker (if there even is one) is an intern or low-level flunky. It's not a profession, it's an entry-level job (often).

Esquire, big glossy mag, Hearst mag, founded 1933, 629,949 total circulation, 2007 National Magazine Award for reporting, 1989 and 2009 for feature writing, Certificate of Special Recognition in 1968, bunch of others. Scores of international editions. Category:Works originally published in Esquire (magazine) has 26 articles, and Category:Esquire (magazine) people has 24.

Sound pretty good? OK to assume it's reliable? What do you think? If a publication like that can't be used, we'll have to stop using hella other publications.

Well here's a quote from an Esquire copy editor (if they even have an independent fact-checking operation, they didn't in this case (copy editors are entirely different beast)):

“ I was helping edit an article... my boss had sent it back to me, suggesting it was a little bland... I simply tacked on [a] sentence [to a quote]... And it worked. My boss liked it better. The problem was, I completely forgot to send the piece back to the [person quoted] to see if she was okay with my little addition. I had meant to -- I know you can't just insert something without the writer's approval. But I forgot... [and later] the [subject] is complaining that Esquire put words in her mouth... I feel terrible... I'm hoping this little scandal will blow over, and I think it will. ”

Simply tacked on a sentence to a quote. Well boy howdy. Slap my ass and call me Susan. And the guy didn't get fired. (He did "feel terrible" tho).

OK? All your sources are less reliable than you think.

Deep vetting is your friend

OK, but there are ways to deal with this, we can call that "deep vetting".

For instance, if you have two separate sources saying the same thing, and they clearly are working from the source material (assuming no elaborate hoax), and they are clearly not copying each other (assuming no elaborate hoax), that is different. You've got two separate one-person film-fan blogs saying that they saw Kevin McCarthy make a brief cameo in Invasion of the Body Snatchers, you can have a high degree of confidence that he did. Higher than one observer for a fancy magazine. The two bloggers are independently supporting each other, fact-checking each other in a sense.

Heck to this day 2019–2020 Hong Kong protests has the statement "A Guardian article dated 22 October 2019 reported that 'protesters have tracked at least nine cases of suicides that appear to be linked directly to the demonstrations' since June". But there's no such nine suicides. It's made up. I know this because I read the source very carefully (and the Guardian is the sole source) and checked their sources . But, you know, when I pointed this out I got outnumbered -- "The Guardian is a reliable source, so we can include the statement, periodt". So it's in the article (it does say "the Guardian reported..." but that's a fig leaf. It's not true so it shouldn't be near the article.)

All your sources are less reliable than you think.

Anyway, this sort of checking is what I mean by "deep vetting". There's a lot of ways to deep vet -- consider the publication's business model, how the author makes a living, and plenty more. Analysis of the language used in the article can give a sense of how rigorous the author was. More than one source is really good, as we said.

WP:RS supports this to a degree. It's a pretty good rule, mostly.

Uh-oh

That's right. None of this is touched on WP:RS.

WP:RS does, correctly, say that there are two sources that are presumed to be reliable: peer-reviewed academic journals, and publications known to have a vigorous independent fact-checking operation. ("Known to have" should include detailed knowledge of their operation (ideally; we grant that it's hard to get that info.)

There's nothing written in WP:RS that mention this at all. Didn't occur the writers probably. I've never seen "these two sources are unreliable, but taken together the make a reliable whole" I don't think. I don't think it would be given long shrift. The Wikipedia is very rule-bound compared to other big publications (sorry, but it is true).

I get it

I get it. It's impossible to do this kind of forensic analysis on all sources, or even on more than a tiny handful of sources. So, we go with "well this publication is big and famous and fancy people read it and it looks reliable and other editors agree, so it's OK". We have to. Just like DAs have to use eyewitnesses. Otherwise, nothing would get done, and we'd have a much smaller encyclopedia. I get it, and it's fine. I'm not suggesting we make a major change.

Fine. But just recognize that it's kabuki to a degree, and if deep analysis puts the source in doubt, remember that you're not a robot and the rules were to made for the editors, not the editors for the rules.

All your sources are less reliable than you think. Use two. If you don't have two, and the fact isn't self-evidently true, and you don't have the time to do deep vetting... you've got a potential problem. Herostratus (talk) 18:53, 10 June 2021 (UTC)