Wikipedia:Wikipedia Signpost/2012-04-23/Investigative report

From Wikipedia, the free encyclopedia
Jump to: navigation, search
I agree with Tgr. The way that argument is presented in the article, with the implication that 41% would be a more accurate value, is not mathematically sound. JN466 13:23, 25 April 2012 (UTC)
HaeB, the article suggests that 60% is somehow artificial and even manipulative ("This cleverly allows DiStaso to exclude...", "boosts the factual error rate"), while 41% is presented without any such concerns. This is simply untrue: ignoring the "don't know" answers is a completely natural assumption, which is regularly used in statistics (leaving out uncertain voters in election opinion polls, for example). Assuming all "don't know" answers are in reality "no error" is, on the other hand, completely artificial. Even if the article is merely equating the two practices, as you suggest, that's still severely misleading. DiStaso was justified to use that number, there is nothing unethical or unprofessional in it, and there is no good reason to suspect the real number would be lower. (What was not at all justified is pretending that number is the percentage of Wikipedia articles, when it is in reality the percentage of PR people, and given that they were notified of the survey through channels which strongly favored those who had conflicts with Wikipedia, it is very likely that problematic articles are strongly overrepresented. The whole "don't knows" issue is just a red herring, as I said.) --Tgr (talk) 07:44, 26 April 2012 (UTC)
"Including the don't knows would yield 41% (406 / [406 + 310 + 273])."—that's all it says; would. I don't know where JN466 gets the idea that the story, as published, privileges any number. The point made is that the blanket assertion of 60% by DiStaso is not defensible. Tony (talk) 09:46, 26 April 2012 (UTC)
It privileges one end of a distribution. Including the don't knows does not yield 41%; it yields the 41%-72% range. You get 41% by including the don't knows and assuming they all have error-free articles; the article conveniently does not mention that (rather unlikely) assumption. And, as I tried to argue in some length above, "the blanket assertion of 60%" is completely defensible (well, probably not when it is asserted to be the ratio of all Wikipedia company articles which have errors, but that is for different reasons). Here is a random study about Wikipedia, for example, which does the same; the press release says 45 percent of toxicologists find Wikipedia accurate; if you read the article, you will find that the actual number is 23%, with 54% having no opinion. Reliability of Wikipedia reports the 45% number; so did Signpost. Do you think this was "not defensible", too?
I'm fairly sure if you checked more Wikipedia research, you would see the same treatment of "don't know"/"not sure" answers; it is the common thing to do. I'm completely sure you will not find a single study where "don't know" answers are treated the same way as "accurate" answers. Sorry, but the Signpost article is way more dishonest with the numbers than the study it is trying to criticize. --Tgr (talk) 12:08, 26 April 2012 (UTC)
I really don't like being called "dishonest" even if only indirectly. As far as the other study cited, it's a completely different methodology that compares different sources' general accuracy, without attempting to count errors or "inaccurate articles." It has two sides that are being considered "overstated" and "understated" as well as "accurate". And Wikipedia is considered to be one of the most accurate sources, no matter which measure you use. DiStaso could learn a lot by studying this methodology. It's very important in this discussion to remember that the type of error, including errors of omission, and bias are extremely important. Frankly, I have to say that 100% of Wikipedia articles have some type of error, but I'd say the same about Encyclopedia Britannica, and even the most highly respected journal articles (and I hope nobody misquotes me on this!) Until we have a direct pipeline to the "truth," any study of accuracy has to be a study of comparative or relative accuracy. Ultimately there is no verifiable measure of "absolute accuracy" any more than of "absolute truth." Smallbones (talk) 16:22, 26 April 2012 (UTC)
I am missing the Facebook Like button here. :) Tgr made a typo (the actual figure that underlay the 45% in the Signpost report was 21%, not 23% – see last page of the original document), but apart from that hit the nail squarely on the head, and gave a beautiful example. The toxicology study is the exact same thing. The Signpost was happy to have don't knows discounted in the toxicology study when the resulting figure was in Wikipedia's favour, accepting the study authors' summary that 45% of toxicologists considered Wikipedia accurate, when in fact it was only 21% of respondents, due to the large number of don't knows. But the same standard procedure in DiStaso's study is pilloried as dishonest. That's just not good behaviour, and misleads readers. Never let fear cloud your view of the data. --JN466 16:17, 27 April 2012 (UTC)

Independent research needed

The frustrating thing is that we don't seem to have a recent reputable study into the accuracy of the pedia. Without that we are vulnerable to this sort of exercise. A more open response to this would be for the WMF to commission a trustworthy third party to quality check a random set of facts and articles and produce a report on it. If this was done as an annual or even biannual exercise then the press would have something to check against, we would have an interesting benchmark, and if and when "studies" like this emerged the press could ask the researchers would done the benchmark study to comment on the competing study. ϢereSpielChequers 17:16, 24 April 2012 (UTC)

I'm completely agnostic about whether this would be a good use of WMF funds or not, but perhaps, for reasons of perceived bias, it would be better for independent researchers to do it.
  • If it were to be done, following the suggestions in the section above "No real data given" would be a good start. Graduate students would likely do the actual error evaluation, and they could not be expected to research each error on the spot. Rather they would have to be given a detailed instruction sheet on what constitutes a major error, and minor error, etc. Given all the fuss about "errors of omission" above there should also be questions regarding omitted material and neutrality of presentation. While those aren't black and white calls, mechanical instructions could at least lead to fairly consistent results. The evaluators and the instructions would have to first be evaluated to see if there were internally consistent results.
  • This type of test wouldn't be very meaningful for the full set of Wikipedia articles. Rather subsets should be separately examined, e.g. articles on Fortune 500 businesses, social science, mathematics, or even popular culture.
  • These results by themselves wouldn't be very meaningful - perhaps we could use them comparing 2 time periods to see if accuracy has changed over time. But there will be possible perceived error in any material - the real question is how does the accuracy compare to something else, e.g. to Encyclopedia Britannica, or maybe corporate websites or annual reports. The comparison would have to be something that has similar goals to an encyclopedia. Comparing a Wikipedia article to a sales brochure would be a meaningless exercise.
So this would be a challenging and expensive task. Not one to be taken lightly. Smallbones (talk) 19:02, 24 April 2012 (UTC)
I agree that it isn't to be taken lightly, this would be a big investment, but I think a worthwhile one - even if the results aren't as positive as some of us expect. However I'm not convinced that we currently have a competitor worth benchmarking against. If we used the Britannica I fear that we'd get a false confidence as the articles that have analogues on the Britannica will be skewed to our higher quality content. The Britannica is much smaller than we are and perforce their notability criteria are more stringent. Since we rely on crowd sourcing, those articles which are less notable will tend to have fewer readers and generally fewer editors. Better in my view to have a rigourous process based on random sampling - even though each year's sample will of necessity be different as I can't imagine we could identify a bunch of errors without fixing them. ϢereSpielChequers 17:09, 25 April 2012 (UTC)
The Foundation has initiated exactly that, see [2]. I have been following this kind of research for several years now, and while there are actually quite a few systematic studies examining Wikipedia's accuracy by now (see e.g. those listed under reliability of Wikipedia#Comparative studies, or in these slides of a talk I gave in January), it is obvious that it is hard to do such studies on a big scale, if one aims at maintaining good research standards - a solid evaluation of factual accuracy is a lot of work (to quote from a review I wrote half a year ago: "[The paper] first gives an overview of existing literature about the information quality of Wikipedia, and of encyclopedias in general, identifying four main criteria that several pre-2002 works about the quality of reference works agreed on. Interestingly, 'accuracy' was not among them, an omission explained by the authors by the difficulty of fact-checking an entire encyclopedia.") This is for the most infuriating aspect of the DiStaso's press release claims - pretending to have done a solid evaluation of the factual accuracy of many hundred Wikipedia articles, where more conscientious researchers have had to content themselves with much smaller numbers and less far reaching claims.
By the way, DiStaso's article says (p.2) that a study "to analyze the accuracy of corporate articles ... is underway as a CREWE initiative".
Regards, HaeB (talk) 00:53, 25 April 2012 (UTC)
The Oxford/Epic study design looks awesome. Are they on schedule? There shouldn't be any perceived bias issues there. The reliability of Wikipedia#Comparative studies, and these slides of a talk YOU gave in January are also quite informative. What strikes me first is the small sample sizes - which result because of the detailed work that needs to be done. But the "specific fact - large N" approach (did Wikipedia get all 280 US gubernatorial candidates in a given period?) looks promising. All the best. Smallbones (talk) 03:30, 25 April 2012 (UTC)

PRSA has changed/updated their headline

Arthur Yann of PRSA said in the CREWE Facebook group:

PRSA acknowledges that the headline of its news release announcing the publication of Marcia DiStaso’s research study in PR Journal oversimplified the study’s results. To help prevent any further misinterpretation of the findings that our release may have caused, we have updated its headline, subhead and lead. In doing so, we hope the focus of this discussion can remain on the gap that exists between public relations professionals and Wikipedia concerning the proper protocol for editing entries, and the profession’s desire for clear, consistent rules that will ease the learning process for public relations professionals of how factual corrections can properly be made.”


April 17, 2012
Survey Finds Majority of Wikipedia Entries Contain Factual Errors
Public relations professionals cite issues with Wikipedia’s accuracy and editing process


April 24, 2012
Wikipedia & Public Relations: Survey Shows Gaps in Accuracy, Understanding (UPDATED*)
60% of respondents say articles on their companies, clients contained factual errors; 25% say they are unfamiliar with such articles

-- Fuzheado | Talk 19:15, 24 April 2012 (UTC)

This has gone beyond the point where quibbling over mere words makes sense, but the numbers still don't add up. I could live with "41% of respondents say articles on their companies, clients contain factual errors, 31% say they are unfamiliar with such articles, and 28% can't find any errors" These numbers come from the same question and are calculated as 406/989, 310/989, and 273/989. The headline's numbers appear to be calculated as 406/679 and 310/1260. Switching the denominators like this just doesn't make any sense. For example it could be inferred from the headline's numbers that the % who can't find errors is 100% - 60% - 25% = 15%. It just ain't right. Smallbones (talk) 20:24, 24 April 2012 (UTC)
You have a point. --JN466 13:29, 25 April 2012 (UTC)

Thank you!

There are lots of people on the in the corporate and PR community, lobbyists and others who would like to use Wikipedia for promotion. I cannot imagine a stronger COI than a person who is paid to make their client look good on the internet (or a person or company editing an article about him, her or itself). For an academic to put together a biased opinion poll of these people with strong COIs, and then to publish their poll answers as if what they said is somehow objective is astonishing. Thanks, Signpost, for alerting us to this travesty. The proof of the pudding is in the eating: What this "study" led to is asinine headlines like this: NYDailyNews: "Wikipedia entries full of factual errors". No, what the "study" found is that 60% of paid shills who were asked ambiguous questions, if they had any opinion at all, felt that one or more articles on their clients had an error, including, possibly, a spelling error. -- Ssilvers (talk) 01:23, 25 April 2012 (UTC)

According to the report 82 "errors" were spelling, though without knowing the alleged errors we don't know whether these are genuine typos or simply examples of us using a different variety of English. A further 152 were "leadership or board information" but this wasn't subdivided into examples where we are incomplete, out of date, or have information which was never correct. ϢereSpielChequers 23:14, 25 April 2012 (UTC)

What the report tells me

The report is intended to convince Wikipedians to openly allow PR people, but the report itself seems to demonstrate a laundry list of reasons NOT to. It's not in encyclopedic tone, doesn't represent all majority and minority viewpoints, uses misinformation to support an agenda and so on. It even demonstrates an ability to corrupt the balance of trusted sources from the real-world equivalent of the Talk page and create one-sided stories in independent sources through the availability of resources.

In other words, editors like Smallbones weren't given a voice in these media articles, because he doesn't have a PR person pitching him to the media. Data to support their POV was presented, but what about data like this[3] showing the edit histories associated with the top ten PR agencies by revenue. If the same behavior and dynamics we see with the report were brought to Wikipedia, it would certainly be a bad thing for the pedia, more so than factual errors.

While I don't believe this to actually be the case, the report seems to communicate to me a need to outright ban PR people. Additionally, I find it difficult for anyone who cares about Wikipedia to consider an open collaboration with a group that publicly assaults the website's credibility in such a manner. All I can do is invite PRSA/IPR/etc. to humble themselves and commit to learn how to meet Wikipedia's content needs and collaboration style, but I don't expect such an invitation to be met. User:King4057 (COI Disclosure on User Page) 01:51, 25 April 2012 (UTC)

I first took "The report" to mean this "Investigative report". Were you referring to the DiStaso journal article? Tony (talk) 02:02, 25 April 2012 (UTC)

A practical response to PR complaints

Let's move on from the hyped and flakey 60% claim. Whether the current low-tolerance policy remains—which looks likely for the time being—or whether it's loosened, it's hard to ignore the perceptions among PR and communications professionals of long waits or no response at all to open requests for changes to articles on companies. These are the good guys, the ones who do the right thing by asking for editorial mediation; yet the message is that they're routinely discouraged. Perhaps this is a collision between the volunteer culture on the foundation's sites ("there's no deadline") and the rigours of turbo-charged capitalism, where I tend to agree with DiStaso's point that five days is a long time for professionals and their clients to sit in silence ("is anyone at home?"). Yet volunteers appear to have done reasonably well in managing serious and complex issues such as quick action on copyright and BLP issues: we've shown that dynamic management is possible, and isn't it part of the cost of doing business on a big, powerful wiki?

Personally, I've found it difficult and time-consuming to navigate through the maze of CoI-related pages on the English WP. Some are tired, moribund, or confused, bloat abounds, and there seems to be no centre of gravity. No wonder a lot of PR professionals and company reps throw up their hands and edit under the radar, when the radar resembles a low-wattage flickering street light in bad weather.

The head of communications, Jay Walsh, sees the response problem and has acknowledged it publicly, as reported at the end of the story. So leadership is in order from the foundation—the cross-wiki implications alone suggest that it's a matter in which the foundation should take a more active, practical role: god knows what tangled webs or straight-out neglect are the norm on the other 280 WPs (including the smaller, outlying language WPs, largely impenetrable to the movement).

If it's good enough for the foundation to create a summer fellowship to revamp our help pages (see the Signpost report this week), it's good enough to consider employing a fellow to work with the community to revamp the speed and efficiency with which we respond to PR requests and queries—to see things from the perspective of incoming PR professionals and to create an easy system to tempt them away from subterfuge. Good openers would be to create a template for posting on company-article talk pages with a link to a convenient, one-stop noticeboard, and working out how to attract volunteers into a response team that involves personal stimulation and social reward. And there's the possibility of sending pro-active messages out to the PR/communications/corporate community about working with them to ensure balance and neutrality; that would be good for the movement's image, wouldn't it. Tony (talk) 04:14, 25 April 2012 (UTC)

I've made similar proposals on the CREWE Facebook page, but using a community-based rather than Foundation-led process:
  1. collaborative creation of a guideline or policy for company articles that defines
    1. what sort of information self-disclosed PR professionals are not just welcome, but requested to add and keep up to date, based on company sources – things like the name of the current CEO, location of the company headquarters, officially reported financial figures etc.
    2. what sort of information generally may be added based on primary sources, and what requires secondary sources (court cases for example should require secondary sources, as in BLPs)
    3. general content expectations, i.e. what any article on a company should contain
    4. guidelines on neutrality, balance, coatracks, attack pages
  2. institution of a noticeboard where PR professionals can flag articles that have gone wrong, and help to work out fixes, to then be implemented by another Wikipedian
Having said that, Foundation support would of course be welcome. --JN466 12:55, 25 April 2012 (UTC)
Whether the current policy "remains or is loosened" leaves out a third option. If this is how the PR industry behaves then perhaps we should tighten and more strongly enforce policies on COI? ϢereSpielChequers 23:18, 25 April 2012 (UTC)
The comments apply even if the guidelines are strengthened. Tony (talk) 01:18, 26 April 2012 (UTC)

Lose the drama, read the study

The DiStasso paper is publicly available:

It is not hard reading.

There are certainly a couple of structural problems with the survey, nicely pointed out above: (1) There is no quantification of the magnitude of error, minor errors and major catastrophes are both considered the same; (2) Respondents were not asked to answer about a single client, so some may be venting about one client and being counted for it, but having no problems with other pages and not having those "good" pages tallied; (3) The paper pretends there is something called a "bright line" policy about paid COI editing and spends a lot of time studying respondent understanding of this incorrect interpretation of actual WP policy.

There were also tactical errors: (1) It was a mistake to try to come up with a sensational high error number and to make that the hook of the piece. The takeaway should be "Most PR people who deal with clients that have Wikipedia pages feel that there are significant errors on those pages, and they are confused about Wikipedia's practices for getting those corrected." Instead we've got a bunch of people yelling about whether 41% or 60% are more accurate quantifications of the problem; (2) It was a very big mistake taking the results to the press and trying to make a news story out of it, rather than quietly bringing the findings to WP directly. Bad blood resulted.

We've just had an RfC on COI editing, now running out of gas. As one might have predicted, opinions vary widely and there is no consensus for any approach to clarification of the matter. What's pretty clear is that as long as there are pages about large corporations on Wikipedia, there will be paid PR people with a professional interest in making sure that those Wikipedia pages are fair, neutral, and error free. That does not describe the current state of many of these pages, I think we all can agree — whether 41% are screwed up, or 21%, or 60%, or some other is absolutely irrelevant. The fact is that there is a problem of some magnitude. How this is resolved is ultimately up to us as a community.

I am very disappointed in this piece, my comments above were written about a late draft, which changed little. It is not journalism, it is an opinion piece disguised as journalism, and a very one-sided and shrill piece of work. Done's done. The issue isn't going to go away. I just urge people to actually read the report and to see what it says and what it does not say directly before they fly off the handle being all too sure about how to resolve a complex problem. Carrite (talk) 04:14, 25 April 2012 (UTC)

Wait we are meant to think that that headline was an ah "tactical error"? Seriously? No it was moderately competent PR and blatant statistical abuse to the extent that whoever did it can lacks intellectual integrity to the point where there is little reason to further consider any of their claims.©Geni 09:49, 25 April 2012 (UTC)
Have you read the study from beginning to end, or just this hatchet job? JN466 12:46, 25 April 2012 (UTC)
I, for one, have read it from beginning to end; and I do understand what things like chi square mean. I still found the big fat lie, "In other words, 60% of the Wikipedia articles for respondents who were familiar with their company or recent client’s article contained factual errors" right there in the text. Who passed this person's Ph.D. thesis at the U. of Miami? Was the statistics work that shoddy in said thesis? Are sentences like, "Also, by disallowing public relations/communications professionals to make edits while allowing competitors, activists and anyone else who wants to chime in, is simply asking of misinformation." [sic] considered acceptable English by Public Relations Journal or the College of Communications at Pennsylvania State University???? --Orange Mike | Talk 14:04, 25 April 2012 (UTC)
Did you take similar issue when the Signpost reported that 45% of toxicologists found Wikipedia accurate? Would you insist that that figure should be revised down to 21%? For further background, see Tgr's post above, 12:08, 26 April 2012 (UTC). Same maths. Goose, gander. Will you be going to Reliability_of_wikipedia#Expert_opinion and rewrite the Science and medicine section to revise the percentage of toxicologists down from 45% to 21%? Because you cannot with intellectual honesty maintain the complaint you are making here while letting those 45% stand. JN466 21:54, 27 April 2012 (UTC)
I've also read the report, and even looked up the study she cites for her figure that only 23% of Wikipedians have college degrees. Checking Glott, Schmidt &Ghosh (2010) they actually said that in their sample 49% of Wikipedia contributors are graduates of whom 23% have a masters or a PhD. Now perhaps College degree is an American English term for Masters and PhDs that I wasn't aware of, but unless that's the case this seems like an error that makes Wikipedians in general look rather less educated than the study indicated. I haven't checked everything in her report that triggered my bullshit detector, the bit about us having "more than 82 hundred contributors" is of course technically correct, as it would be to say more than 82,000 or indeed more than 82 contributors. But it is misleading, shows that Wikipedia has had nearly 1.5 million contributors who've made over ten edits each, and the figure for the English language wikipedia alone is over three quarters of a million, with over thirty thousand making more than 5 edits a month. Combined with the discredited 60% stat I think I might be detecting a pattern here. ϢereSpielChequers 22:52, 25 April 2012 (UTC)
And, just to prove how easy it is to argue and fight about numbers rather than actually discuss the underlying problems, I'll note that you aren't talking about 1.5M contributors at all, you're talking about 1.5M usernames, which includes multiple names for single individuals and quite possibly — for all I or you know — every single IP address for dynamic IP users. Bottom line: we can carp about numbers all day. That's not the issue here, the question is whether a deeply divided WP community can get together well enough to come up with a set of mutually satisfactory "best practices" for COI editors that addresses the needs of both The Project and the PR pros seeking to make sure that WP delivers on its claim of truthful NPOV coverage of their clients. I'm starting to think that we'll chase our tails for five years on this without any progress forward... Carrite (talk) 06:16, 26 April 2012 (UTC) Last edit: Carrite (talk) 06:18, 26 April 2012 (UTC)
The difficulty is that I think everyone acknowledges that WP has problems with errors in company articles. However, part of that process is to understand the nature and scope of the problem - if research being presented is flawed, we can't use that to understand the issues. Errors in a peer-reviewed paper raise flags which risk hiding the genuinely valuable points in the paper, which is what I feel happened here. That doesn't mean we shouldn't be addressing the problem, but that this paper probably isn't the best means of raising it. I gather that there is some good research underway looking at the Fortune 500 companies, and that will be interesting to see. - Bilby (talk) 07:36, 26 April 2012 (UTC)
@Carrite, Well I know it doesn't include IPs because I chose a stat that excluded IP contributors altogether - as well as any account with fewer than ten edits. Yes there will be humans with multiple Socks in that list, but there are plenty of contributors who are IP editors or have fewer than ten edits, the key point is that "82 hundred contributors" is wildly out.
@Bilby. In the absence of any examples of these alleged errors I for one am not yet convinced that the error rate in company articles is any different to the general Wikipedia error rate. My suspicion is that our concept of an error is so different to those of PR writers that the error rate is not something we will be able to agree on, it must be very difficult for an article on a company to be simultaneously accurate both by the standards of Wikipedia and in the minds of those paid to spin for that company. ϢereSpielChequers 09:14, 26 April 2012 (UTC)
Sorry, my assumption without any other data was that the error rate would be the same as on other pages. Hence there would be an error rate - that was what I was assuming was a generally acknowledged issue, not that company articles were especially problematic. :) My concern is that we need solid research looking at this particular domain, and that the red flags with this paper lead me to want to look elsewhere for that data. And I agree with the problem of identifying those errors through PR people who represent the companies - that's one of the red flags. - Bilby (talk) 09:22, 26 April 2012 (UTC)
  • RE: "I for one am not yet convinced that the error rate in company articles is any different to the general Wikipedia error rate." NOW we get to the point of the entire exercise — what the PR people are (clumsily) attempting to express is that in their view there is a significant (perhaps massive) problem with the content of business pages as compared with "average content" on WP. (I think they are right.) There is also study going on about how highly WP page results figure on a Google search — showing up on the front results page, even high on the front results page, for very major corporate entities. In short, there is a sense by the "designated representatives" of those entities that there is a fairly huge problem here.
Maybe 6 months ago I was of the belief that we should be hunting down and wiping out all paid editors/PR people editing at WP and backtracking to neutralize or eliminate their edits. I was as hardline as anybody on this. I've come to a new understanding though, one that I think most hardline opponents eventually will come to once they really start to ponder the reality of the situation. There is ALWAYS going to be pressure over content as long as (a) Wikipedia remains important; (b) big corporations remain big. That is to say: there is ALWAYS going to be such pressure... We all know what bad, horrible spammy pages look like. Somehow in our minds' eye we think that these are the product of PR flacks, doing their dirty work. And some of them are. More, however, fly under the radar — because they are TRYING to stay within our rules and to produce neutral and encyclopedic content.
There needs to be a formal set of "best practices" for these Under The Radar Because They Are Doing It Right editors — in the hopes that the Doing It Wrong PR types will join them in doing it right. Just arguing as the Co-Founder does — "don't do it because don't do it" — is not a tenable situation because there are no extant reliable mechanisms for necessary changes to be rendered. We work on the principle of BE BOLD at WP and that will always be the best way for changes to be rendered. Failing that, there needs to be something else that actually works. Suggesting changes on talk pages is like writing a note, stuffing it in a bottle, and throwing it in the ocean. We can't even keep up with complaints filed on BLP pages... The solution realistically needs to involve direct editing by interested parties. But under what parameters? That's the question.
I think the hardliners seeking to ban off paid editors are in a substantial minority, although recent comments by Jimmy Wales indicate he thinks otherwise. That needs to be the first fundamental decision made: are we going to attempt editor-based or editing-based guidelines to COI editing? If the former, where are those lines to be drawn — what precisely constitutes bannable COI and how are we to make this determination in an encyclopedia in which no formal registration and signing in is required to edit? If the latter, what exactly do we expect paid editors to do to meet our highest expectations?
Everything flows from that fundamental question. The recent RFC on conflict of interest editing was a mess. ArbCom needs to start another, proper RFC consisting of one question: "Is paid COI editing to be bannable on a per se basis?" If yes, that line needs to be carefully drawn — who exactly is a "paid employee" with a COI to be banned if caught? (A teacher writing about their school? A factory worker about the corporation for which they work? A grocery store owner about a product which they sell? A minister writing about the history of his church?)
If we decide to focus upon the edits and not the editor — I believe this to be the majority view, judging by opinions expressed in the failed RFC — then what EXACTLY do we expect paid COI editors to do or not do?
Anyone who thinks this is a simple issue is wrong. Sorry for using so much space. Carrite (talk) 20:36, 26 April 2012 (UTC) Last edit: Carrite (talk) 20:59, 26 April 2012 (UTC)

Utter nonsense

It is impossible to give a sensible critique of, or response to, something that is utter nonsense to begin with.

  • We start with a survey—that is, an opinion poll—about whether Wikipedia's articles about companies each contain at least one factual error. There is no necessary correlation between opinion and fact. To put it differently, an individual's opinion about a fact may say something about the individual, but it says nothing about the fact. Suppose one were to conduct an opinion poll of a random sample of American adults on this question: Is there a factual error in Wikipedia's List of Presidents of the United States? The results, whatever they might be, would say nothing about the list's accuracy. Likewise, the results of this survey say nothing about the factual accuracy of Wikipedia's articles about companies. Indeed, if 100% of the survey respondents had said that there was at least one factual error in the article about the respondent's company or PR client, that would not prove any error in any of the articles. Criticizing the statistical methodology or conclusions is beside the point, since the individual survey responses are worthless to begin with.
  • This otherwise meaningless survey's respondents all have conflicts of interests, which makes their responses—that is, their professed opinions—even less meaningful than would be the respopnses of unbiasred responsents. It was in their respective interests for the survey to "prove" that Wikipedia's articles about companies were inaccurate, so these self-interested respondents would be allowed a freer hand in determining content of articles about which they have a self-interested point of view. Indeed, the respondents were chosen because of their conflct of interest. Normal survey design neutralizes bias; this survey guaranties that 100% of the respondents are biased.
  • At the other end, suppose the statistically flawed conclusion drawn from the meaningless, biased data were true. Suppose 60% of Wikipedia's articles about companies contain at least one factual error. What would that prove about the articles' factual accuracy? Almost nothing.
    • It says nothing about the ratio of inaccurate to accurate facts. Surely, the error rate does not approach 60%, as the "headline" might imply to the average (careless, innumerate) reader. I would be concerned it the error rate approached 5%, or even 3%. However, given the number of facts in typical articles about companies, the survey's conclusion (even if taken seriously) does not imply an error rate that approaches ½% (by my utterly baseless guesstimate).
    • It says nothing about the importance of any errors. At one exterme, an article might place a company in the wrong industry or misstate its annual revenue by a factor of 100; I doubt that Wikipedia has any errors of that magnitude, but almost anything is possible. At the other exterme, an article might have the wrong middle initial for some individual.
    • It says nothing about the source or nature of any inaccuracy. If an incorrect fact is based on a reliable source, and no reliable source has the correct fact, there is little that Wikipedia can, or should, do.

While this piece is a noble effort to put the survey in some perspective, its flaw is treating the the survey as though it means anything in the first place.—Finell 18:35, 28 April 2012 (UTC)


This article has prompted me to make a proposal at Wikipedia:Village_pump_(policy)#BizProd. ϢereSpielChequers 16:31, 29 April 2012 (UTC)