User talk:TidyCat/Achieving validation on Wikipedia

This is the Achieving validation on Wikipedia Talk page. I have added a copy of the essay here so users may comment on it point-by-point. You may also add your comments to the general comments section at the end. --TidyCat

Hi. I've been editing Wikipedia for a few months now (and reading it for far longer), and it is a wonderful project. The Wiki model is astonishingly effective, and I am one of many people who believe in its world-changing potential.

But like many others, I've also encountered Wikipedia's key shortcoming: in its present form, it is of questionable use as a reference source. It is free, it is large, and it is an excellent source of information on many subjects. But the information it contains can be so variable in quality that Wikipedia does not, and cannot have, authority. At any given moment, any article on Wikipedia can be incomprehensible, inaccurate, one-sided, vandalized, or even wholly fictional, possibly with malicious intent. This devastates Wikipedia's central goal: to be an encyclopedia.

Initial research

I started studying the problem in depth; for several weeks I've been poring over the encyclopedia, contributing, learning about its working methods, and reviewing its history. I learned that the concept of article validation goes back almost to Wikipedia's inception -- and that many proposals to implement it have been submitted, but none have yet taken hold.

In the midst of my research, the Seigenthaler incident occurred, and since then, criticism has been coming at Wikipedia fast and hard. These incidents have dramatically underscored the need for article validation on Wikipedia -- no longer as a "should," but as a "must."

There have been several validation schemes suggested on Wikipedia, and many of the ideas have been quite good. I think their failure to be adopted to date has been a combination of a lack of urgency -- which is now upon us -- and the absence of certain key elements in their specification.

Since the Seigenthaler incident, there has been a renewed interest in validation, and there are several proposals being actively discussed. I have attempted to draw the best elements of each of these proposals together into a full and practical specification which still aims to keep the Wiki as open as possible. I hope the advocates of these various proposals can see that I am neither ignoring nor taking credit for their ideas.

It is my goal to rally the community to adopt a workable and full solution in the very near future. If there is any one project that needs prioritizing on Wikipedia, this is it. Let us make this long-sought feature a reality this time.

Notable proposals

Most of the proposals I've seen fall into a few major categories:

• Expert review
• Voting/rating of individual articles/versions
• Community review (à la Peer review or Featured article candidates)
• Voting/rating of individual editors (including "trust metrics")
• Stable/locked versions

All of these approaches face the challenge of needing to satisfy two seemingly-opposed goals: to keep Wikipedia's openness intact -- anyone may edit Wikipedia -- while enacting a process that will weed out misinformation.

Any approach to article validation must thus satisfy the following criteria:

• Avoid the "gatekeeper" effect -- there should be no privileged users whose ability to shape the content of Wikipedia far exceeds others';
• Scale to accommodate potentially hundreds of editors' inputs without descending into chaos or inaction;
• Be implementable upon tens or hundreds of thousands of articles within a reasonable time frame;
• Be resistant to abuses by individuals or groups trying to subvert the system; and
• Produce articles which are reliably factual, consistent in style, and neutral in tone.

Expert review

Expert review -- the appointment of credentialed academics to shape Wikipedia's content -- fails on several of these points. There are many active Wikipedians who are experts in a number of fields, but it seems unlikely that they, working alone, could keep up with the huge volume of edits and new articles on the site without becoming a bottleneck to improvements. Hiring reviewers would likely be prohibitively expensive.

A greater danger is the possibility that such experts would edit articles according to what they know or believe, and discard whatever falls outside of their personal knowledge.

Because the standard embraced by Wikipedia is to only include statements based on independently verifiable sources, expert review is in fact not even necessary. The only expertise reviewers would need is the ability to understand the article they are reviewing -- and the sources from which it is drawn. Wikipedia's expert contributors have been invaluable, and will continue to be so, but the project has been built on far more than their efforts.

Voting on articles

Article voting or rating is a passive process, and it seems unlikely that it could ensure Wikipedia articles reliably met specific objective standards. Rating also is an inherently subjective process: while I am sure many Wikipedians will vote thoughtfully, I suspect most will simply vote along the lines of "I like this," "I hate this," "This seems correct," and so on.

It is my understanding that a system of this type is close to being implemented on Wikipedia. I do believe it will be of use in evaluating certain aspects of articles' quality, and it will probably help draw attention to a variety of articles which could use improvement. But it is at best an indirect mechanism for rooting out misinformation in the encyclopedia, and it is conceivable that it will be little better than the existing mechanism -- "You can edit this page right now!" I mean all due respect to its proponents -- but an article rating system should not be considered a true mechanism for validation.

Comment. Well, voteing is a form of "The Wisdom of Crowds" which is very powerful. Google search algorithms are based on WOTC concepts. Flikr and any number of popular "tagging"-based services use it. I honestly think this is the way forward for Wikipedia; the other methods listed here are all essentially gatekeeper and dont scale well. Perhaps WOTC can be done via an algorithm like Google, analysing how many articles link to an article (and the value of those linked articles), scoreing and ranking using a number of metrics.. Thats just one idea, but the concept of allowing users to naturally and effortlessly score and rank articles (directly or indirectly) is the only way to deal with 1 million plus articles (it could be 100 million some day sooner than we think). --Stbalbach 07:25, 12 January 2006 (UTC)

Well, as mentioned above, an article-rating system is in the works, and might be released soon. A test version of it can be found here. I would be pleased if a voting system could actually make most of Wikipedia's articles reliably factual and well-written; it's certainly likely to be a simpler approach than I have described.

But, particularly where factuality is concerned, an "effortless" (and unexplained) assessment by an anonymous user is not likely to be a reliable assessment. You don't need thousands of votes to find out if an article is factual or not; you only need one person who can demonstrate that it is wrong. That's why I have far greater confidence in a formal peer-review system. Voting is an extremely indirect approach to fact-checking -- and fact-checking is the single most important task article validation should achieve.

As for my proposal's scalability, the I expect the number of qualified reviewers on Wikipedia to scale with its number of editors (and, in turn, its number of readers). I don't know what the proportion of reviewers to editors might end up being, but I would anticipate at least as many reviewers as administrators, and hopefully far more.

Even if there weren't enough reviewers to keep all validated articles on Wikipedia up-to-date, I would still consider validation an absolute success if Wikipedia's 5% most-important (or most-viewed) articles had a decent validated version. That 5% would be 45,000 articles at the present time, or 5 million articles out of a projected 100 million. If a review system such as I have described were integrated into Wikipedia, I think 5% would not be out of the question. --TidyCat 08:45, 13 January 2006 (UTC)

Wisdom of the crowds certainly makes me nervous on any articles dealing with religion, politics, science and pseudoscience. For the later two, the wisdom of the crowd is often wrong, whereas actual facts can actually be validated objectively. linas 01:18, 14 January 2006 (UTC)

But.. Wikipedia is wisdom of the crowd. See this aricle:

For Kendall Whitehouse, senior director of information technology at Wharton, the most interesting question is "whether the wisdom of the crowd is ultimately a better approach compared to scholarly review and edited content." As Whitehouse observes, "Wikipedia's strength is that it has thousands of eyes looking at it. The hope is that errors will be quickly caught and corrected.".. Marketing professor Peter Fader notes that Wikipedia shows there is wisdom in crowds, but a better user rating system would filter out those who post bogus information. Joel Waldfogel, professor of business and public policy, agrees that much of the concern about Wikipedia is just a new spin on whether old media (printed encyclopedias, in this case) stand a chance against the new breed of instantly updatable online media. (quoted from article). In short, no matter what system you come up with here, it is going to be wisdom of the crowd, by design, that is what Wiki is. So you might as well design a system around that concept, go with the natural design of the system, work with it to all its advantages, not work against it and try to limit it with "old media" concepts. --Stbalbach 19:31, 15 January 2006 (UTC)

Community review

A formal review process is probably the most direct way to evaluate and standardize articles according to objective criteria. Any article on Wikipedia could be brought up to encyclopedic quality by a team of motivated Wikipedians. But the question is how to implement it: Who reviews? Who edits? What standards should be applied? What procedures should be followed?

Featured Article Candidates is an excellent example of a process by which Wikipedians organize themselves into review teams and turn out excellent articles. However, the challenges that an official validation process would face are likely to be different from the Featured Article process, so certain changes would need to be made.

Voting on editors

If a review process were adopted on Wikipedia, those performing the reviews would have a strong shaping influence over article content. Wikipedia would need to be able to trust the reviewers to ensure that articles met all quality standards, as well as to ensure that articles incorporated community views.

A "trusted-user" system already exists on Wikipedia: the promotion of certain editors to adminship, based on their contributions to the project and to the community. It is not a perfect system, but it seems to have worked pretty well and produced a large staff of admins who are for the most part conscientious and able.

A similar procedure could be adopted to create a broader staff of "reviewers": individuals who could be trusted to carry out the task of evaluating each article according to objective, community-developed standards -- and who can serve as effective mediators of the review process, making sure that all viewpoints are considered, and that consensus is achieved.

Stable/locked versions

No matter which validation method is ultimately adopted, Wikipedia should provide a way to view the exact version of the article that passed validation. This is critical: there is no point to offering readers a validated version if we can't guarantee that the version they view is the version that was actually validated.

This could be accomplished by providing a link to a specific version in the article's history, or by placing validated versions on a separate page which is protected from edits. I favor the separate-page approach, as it will place validated versions in a consistent, easy-to-find location, and allow for greater versatility in the presentation of validated versions.

The Featured Article review process

A typical Featured Article candidate is nominated and sponsored by a single editor. Several self-appointed reviewers offer comments and suggestions, and the article's sponsor(s) do what they can to bring the article in line with the reviewers' suggestions. On occasion, reviewers will help rewrite parts of the article.

This system works well for a variety of reasons: the participants in Featured Article reviews are generally like-minded individuals; they avoid reviewing articles with a contentious history (including edit wars); there is usually only one sponsoring editor, avoiding differences over style or content; and the end result has no particular "official" status -- it is featured on the Main Page for one day, but the overall result is merely a good article having the same official status as every other version of every article on Wikipedia.

FA status is prestigious, and a Featured Article is generally well written and researched. But the FA process, as it currently works, doesn't satisfy all of Wikipedia's needs for article validation. For one thing, readers of a given featured article are not informed that it has passed the FA process (and thus has a higher likelihood of being a good source of information). FA status is noted on articles' Talk pages and on the master list of Featured Articles, but neither of these will come to the attention of most readers.

More importantly, after an article has become featured, the article is not stabilized in any way -- a link to the reviewed version in the article's history is not even usually provided. This means that even a Featured Article may be a vandalized article at any given moment that a reader encounters it.

Therefore, the FA process would need to be altered in a handful of ways in order to cover all the requirements that an "official" validation process would need to satisfy.

Proposed article validation process

An official validation process on Wikipedia would need to address the following issues which the Featured Article process currently sidesteps:

• It must be able to produce balanced and reasonably well-written articles even on contentious subjects;
• The process must be open to the input of a potentially unlimited number of editors; and
• The end result should be a stable, although not final, version.

The validation process must thus be part standardization and part negotiation. There will be many differing views on how to change each article to bring them in line with community standards, particularly as regards the requirement of neutrality.

The FA process offers clues as to how this might be accomplished. FA reviewers are generally not the authors of the article, nor do they perform whatever rewrites are needed. They state their opinions as to where each article falls short of established standards, and leave the actual rewriting to others.

This approach usually allows them to evaluate the article more objectively than any of its authors could. Furthermore, they are compelled to make whatever objections they might have clear, specific, and actionable -- "I don't like it" is not actionable, whereas "This section is unclear/unsupported by references" is.

In order to broaden this approach to accommodate multiple parties actively writing the article, only one simple change would need to be made. Instead of reviewers making their recommendations to a single editor (the article's nominator/sponsor), they should make their recommendations to all editors with an interest in the article. Any editor could submit to the reviewers their version of how they would rewrite a given section. Reviewers would then decide amongst themselves which version was the most readable, factual, or neutral (depending on which objections needed to be satisfied).

Editors, in turn, could offer their own suggestions as to what changes they felt the article needed. Reviewers would be compelled to incorporate these suggestions, as long as they fit within established criteria and met with few objections from participating editors.

A negotiation process of this type is already applied throughout Wikipedia as the basic mechanism for achieving consensus. By combining it with a "mediator class" (the reviewers), it ought be possible to achieve consensus (or as close to it as possible) on every article on Wikipedia.

Proposed validation criteria

Featured Article candidates are evaluated according to a set of standard criteria. These criteria are, at the present time:

• The article should be well written;
• comprehensive;
• factually accurate (with included references);
• neutral;
• stable (not subject to ongoing edit wars);
• compliant with community standards of style;
• have legally-usable images, where appropriate; and
• be of appropriate length, remaining focused on its main subject (referencing "daughter" articles where necessary).

Most of these criteria should also be applied to articles undergoing validation. However, where Featured Articles are intended to meet (or exceed) standards for "Wikipedia 1.0," validated articles should merely be required to meet the "0.5" standard (usable articles) mentioned on the Wikipedia 1.0 page.

I thus propose that validation criteria should be modified from the FA criteria in the following ways:

• Validated articles should be reasonably well written. As long as they are readable, reasonably structured, and cast in an encyclopedic "voice," that should be sufficient.
• Articles should not have to be comprehensive. As long as the article is not completely one-sided in its coverage of a subject, it should be acceptable. More roundly-written articles, however, might be higher-priority candidates for validation.
• Articles should not have to be stable. A lot of articles have been the focus of bitter edit wars, vandalism, and bias. Many of these articles need a validated (i.e., stable) version more badly than less controversial subjects.

In essence, validated articles should not strive for perfection. The primary purpose of validation should be to prevent the dissemination of misinformation. As articles improve through user contributions and successive reviews, they should be held to higher and higher standards -- each new validated version should be an improvement on the prior one. But the initial standard for validation should be a relatively relaxed one (except where factuality is concerned). The best mechanism for improving articles' overall completeness and quality is the Wiki process itself.

Nevertheless, there will probably be cases where articles will be too incomplete or otherwise flawed to be meaningfully validated. Reviewers in these cases could decline to review an article and instead suggest it be listed on the improvement drive or peer review pages. Featured Article reviewers commonly do this, directing articles that fall far short of the FA standard to Peer Review first.

In addition to the above relaxations of the FA criteria, I would suggest the following additions:

• References should not just be required, but explicitly checked by at least one reviewer. I am sure many FA reviewers do check the references provided in the articles they review; however, it is not presently a requirement that these sources be checked, and just as a statement can be false or erroneous, so can a citation. Explicit fact-checking is possibly the most important aspect of article validation.
  This requirement might be difficult to meet in cases where a reference cannot be checked online. If an alternative reference can be found which supports the statement(s), it should be cited instead, or in addition.
• Plagiarism should also be checked for explicitly. FA reviewers commonly do this, particularly with images; however, if feasible, simple checks for plagiarism should be performed on the text as well (at the very least, a Web search).

The Featured Article process, meanwhile, can remain on Wikipedia without modification. Where validation should strive to give us many articles meeting the 0.5 standard ("usable"), Featured Articles should continue to push articles to the 1.0 standard and beyond ("our best work").

Post-validation

After an article passes review and is formally validated, it should be placed on a "Validated" page, accessible via a tab placed beside the current article tab. The Validated and Current articles should have a prominent link to each other at the top of the article, with a note explaining the difference.

The Validated version should be locked from edits. This is critical: there is no point to offering a reader a validated version if we can't guarantee that the version they view is the version that was actually validated. Ongoing improvements to the article, however, should continue on the Current page in normal, unrestricted Wiki fashion.

It might be useful to give the Validated page its own "Talk" page, where users could list specific errors they believe exist in the validated article, possibly prompting a re-review. (Since it would have this specialized purpose, the page should perhaps be called "Errata," and its tab could be labeled "Report an error." The top of the Errata page could additionally suggest "If you like, you may edit the current draft of the article and fix it yourself," with a link to the Current article.)

At any time, any article on Wikipedia could be nominated by any editor for review (or re-review). However, there should probably be a moratorium of perhaps a month between reviews to keep ideological battles from being waged constantly over certain articles. (Editors could continue to tilt at each other on Current and Talk pages.)

In cases where a user could demonstrate a specific error in a recently-validated article, the review process could be reopened to correct that error only. If the review team can be shown to have been grossly negligent in enacting community standards or incorporating editors' inputs (hopefully this will be rare), the article could be opened for a full re-review. And in cases where substantially new material were added to the current article (particularly material covering current events), the review could be reopened to include the new material only. Outside the moratorium period, however, all articles should be fair game for re-review.

I'm not sure what mechanism should be used to determine if an article in moratorium merited re-review: some editors might constantly submit material of low relevance and insist that it be included. One possibility might be a short-duration straw poll of reviewers only (possibly requiring a supermajority). This could permit the inclusion of new material without keeping the article in constant dispute. Disputed material could still be added to the Current page, and if moratorium periods were kept short, a full re-review could start in the near future anyway.

A similar straw poll approach might be necessary during a full review to determine whether disputed material should be included in the main article, moved to a subarticle, or discarded altogether. I am not advocating voting over each minute aspect of each article, but there will probably be issues over which consensus through discussion cannot be achieved.

Default display version

Having both Validated and Current versions of articles available on Wikipedia raises an issue: which one should users be shown, by default?

There are sound reasons for each version to be the default shown to Wikipedia visitors. The Validated version will have a high likelihood of being of good quality -- at a minimum, factual and free of vandalism. However, the Current article may contain far more information than the most recent Validated version. Much more importantly, Wikipedia should do everything it can to attract new editors to the encyclopedia. If visitors encountered a "working draft," that might inspire more of them to contribute than if they saw a stable version.

Comment - I don't think reading a validated version need discourage editing in any way. On a validated article there should still be an edit tab, which should lead to the latest working draft (rather than the validated version per se). It might be labeled "Edit latest draft" or some such instead of "Edit this page", but the essence is that one can still create an improvement to the current article.

Also, we should be putting a higher emphasis on reliability in Wikipedia, so the fact that the current draft might have "far more information" should not be the controlling aspect — presenting factual unvandalized material first (with a link to the current draft) rather than the free-for-all who-knows current version seems to be a big improvement. -R. S. Shaw 20:50, 22 January 2006 (UTC)

However, I feel the potentially edit-suppressing effects of stable versions could be mitigated in several ways. For one thing, links to view and to edit the Current version should always be provided, with a prominent banner explaining the difference between the versions offered to unregistered users. Registered users should be given the links between versions but would not need the explanatory text. Registered users should also be able to set whether they want to see Current or Validated versions by default.

Comment - Although I see no need to mitigate "edit-suppression", I agree with the desire for these two characteristics: prominent article-status flagging and the ability of the logged-on user to set his default version (most wiki-addicts usually want to see the current version, but occasional editors probably don't, and non-logged-on users should get the known-good stuff, the stable version). We should consider making the article-status flagging only on the current version, with a prominent message, almost a warning, like "The contents of this page may be erroneous since it may have been recently changed without any review." Opposed to this, the stable version might need no prominent message (although it should be marked). This is a change from today, but would very usefully emphasize the reliability/unreliability of the classes of article versions which will be popping up in Wikipedia. -R. S. Shaw 20:50, 22 January 2006 (UTC)

Another mitigating factor is that there are only two cases where a "default version" would need to be shown -- when a user types in an article name and presses the 'Go' button, or types its URL directly into the address bar. In cases where a user performs a search (either from within Wikipedia or via an external search site), both versions should have equal standing, and both versions could appear in the search results, depending on how closely they matched the search.

A third method of visiting articles -- clicking on wikilinks found in other articles -- would not need to favor a version either. If the user is presently viewing the Current version of an article, its wikilinks should take the user to the Current version of linked articles. If the user is viewing the Validated version, its wikilinks should load the Validated version of the linked article, if one is available.

Comment - I think this is wrong in that following a link to another article should go to the version of the users default, regardless of the status of the page containing the link. This is because naive visitors will be seeing both kinds of pages (after all, the Wikipedia isn't going to get completely validated in one night). The naive user should always see the reliable stuff, if available (otherwise the current version) whether by search or link. Once on a current/draft article, clicking a link should take him to the stable version for the new article so that he gets the most reliable/factual/nonvandalized version available. One hop through a single no-stable-version-available article should not forever condemn the reader to unreliable versions from then on while surfing into further articles. -R. S. Shaw 20:50, 22 January 2006 (UTC)

The election of reviewers

The Requests for Adminship process works fairly well in identifying editors who can be counted on to carry out administrative and regulatory tasks according to established rules. A similar system could be created to elect reviewers -- individuals who could be counted on to carry out reviews competently and impartially.

The standards for reviewership could probably be far less restrictive than the standards for adminship. For one thing, reviewers will have a very limited, although important power: the ability to evaluate whether an article meets community standards, and make recommendations as to how it should be changed to meet those standards. For another thing, reviewers will never act alone, but instead as part a team of reviewers, who must achieve consensus in their evaluation of articles.

The standards for achieving adminship are not formalized, although most voters on admin candidates apply a high standard. The standards for reviewership would also not need to be formalized -- editors may apply their own standards for whom they think would make a good reviewer.

Admins should be allowed to be reviewers, but should not be granted reviewership automatically. The duties of admins and reviewers are very different, and a person who is suited to one role may not be suited to the other. In keeping with the existing policy that discourages admins from exercising admin powers upon articles they actively edit, they should also be discouraged from performing admin actions on articles they actively review.

It is inevitable that some reviewers will prove to be bad mediators, possibly ignoring others' views or applying personal (not community) standards of style. There will thus need to be some regulatory mechanism for demoting reviewers who abuse their powers. I feel that a remedy which would not require admin or arbitrator involvement would be preferable.

One possible approach could be to have reviewers elected to finite terms of service, of perhaps 3 to 6 months. Reviewers could apply for re-election so as to serve continuously, if they so wished. I feel this would be a good mechanism by which editors' suitability as reviewers could be periodically re-evaluated, without burdening administrators or arbitrators with the problematic duty of deciding "who's in" and "who's out." I hope that only extreme cases of abuse would necessitate the attention of administrators or arbitrators.

The proposed review process, step-by-step

A lot of the working details of how a review should proceed will probably emerge in response to practical concerns. To get the ball rolling, however, I would like to propose an initial model for the validation process.

Initiating a review

Any editor may nominate any article for validation review. There are several ways the nomination procedure could be handled. One simple way of handling it could be to add a template to the article. The template could do any or all of the following things:

• Announce to readers of the article that it has been nominated for review;
• Provide a link to a "Validation review candidates" page where they may second the nomination by signing under a heading containing the article's name;
• Add the article to a category, "Validation review candidates"

Reviewers could then sign on to review the article on the "review candidates" page. Alternatively, they could leave comments there -- for instance, to suggest that an article needs to cite its sources, or that it is too short to be a high-priority candidate for review.

Once enough reviewers agree to review the article (at least two, but perhaps more), the reviewers should open a "review" page specific to the article. The simplest way to implement this would probably be to add a "Review:" namespace to Wikipedia, which could be accessed via a tab at the top of each article. This would also make it easy to let interested editors know when a review of a particular article has begun: they could simply keep its Review page on their watchlist.

Once a review has begun, the article should be removed from the "review candidates" page and added to a "current reviews" page.

Opening steps of a review

To open the review, reviewers should copy the current article (or a suitable recent version) to the Review page. The Review page should serve as a repository for agreed-upon article changes and should be editable by reviewers only. The discussion of what changes should be made should take place on a different page -- most likely another namespace, "Review Talk."

The "Review Talk" page is where reviewers should discuss their objections to any part of the article, or other changes they wish to see. All editors may respond on this page with their own opinions, or by offering proposed implementations of the changes.

The "Current" article page, meanwhile, should be locked from edits for the duration of the review. The reason for this is to keep the current article and validated article from getting out of sync while the review is being performed. If the end result of the review -- the validated article -- were copied to the Validated page but not the Current page, that would produce a content fork between the two pages, which is highly undesirable. The validated version should be used as the starting point for all edits which follow the most recent review.

However, if the validated article does get copied to the Current page, then any changes editors make to the Current page during the review will get discarded. Rather than allow them to waste their time, a template should be added to the Current article directing them to the Review Talk page, where they may contribute to the active review.

Because the opening of a review would involve the locking of an article's Current page, an admin should be the official initiator of the review proceedings. This would also ensure that the criteria for review are met: namely, a minimum number of committed reviewers, and the article not in "moratorium" from a recent review. To commence a review, an admin should add the "review in progress" template to the Current page and protect it. The review can then proceed.

Sequence of review

To avoid performing work that might be discarded, the article should probably be reviewed according to a specific sequence. The presence of plagiarism should be checked for first; then citations checked and factuality disputes resolved; then major structural issues tackled, such as the regrouping of passages in new sections, or the removal or addition of material as necessary to keep the article balanced or focused on its core topic.

General issues may then be addressed -- essentially, any objection a reviewer wishes to raise. Non-reviewers may also raise objections -- if they are reasonable objections, reviewers should attempt to address them; otherwise, they may be disregarded. It may be useful to address articles on a section-by-section basis until every issue is finally resolved.

Following such a sequence should be a practical guideline, not a requirement, as every article will potentially have unique issues that need resolving in different ways.

Closing the review

It might be necessary to halt or abandon a review -- if the reviewers fail to actually conduct the review, for instance, or if events of any other sort impede its completion. If that happens, the review should be cancelled, and an admin should reopen the Current page to edits.

However, provided the review successfully addresses all of the major issues raised, the revision assembled on the Review page should be given one last fact-checking and proofreading, and an admin should be notified of the successful conclusion of the review. The admin should briefly review the proceedings to confirm they were conducted properly, then copy the Review version to the Validated page as well as to the Current page. On the Validated page, all templates should be subst'ed, and all images used should be locked from edits (if necessary, a copy specific to the article could be used). Finally, the article listing should be removed from the "current reviews" page. The Current page can then be reopened to edits by all users.

Technical changes

In order to implement the proposals I have listed above, the following technical changes would need to be made to the Wikipedia site and/or the MediaWiki software:

• The addition of Validated, Review, Review Talk, and Errata namespaces;
• The addition of tabs and messages to the basic interface which direct users between each of these namespaces;
• The possible renaming of the Article tab to "Current" or something similar;
• New code to load either Current or Validated pages via wikilinks, depending on whether the user is currently viewing a Current or Validated page;
• The addition of a user option to view Current or Validated pages by default; and
• The addition of a "Reviewer" user class which can edit the Review page.

There might be other necessary technical changes which I have not considered.

Closing thoughts

I do feel that a true system of article validation can be implemented on Wikipedia without disturbing the openness of the Wiki. Not all of the solutions I've presented here may be the best ones, and not everyone may agree with them. However, I do feel that validation's time has come, and that we can fit together all the necessary pieces in the near future, allowing Wikipedia to become not only the world's most comprehensive source of information, but its best.

--TidyCat 05:06, 8 January 2006 (UTC)

---

General comments

Hey, this looks like a very nice, and very well thought-out proposal. You should post a comment linking to this page from here, and perhaps you could join us on WP1.0? We need people who care a lot about these issues getting involved in WP 1.0. Keep up the good work, Walkerma 18:35, 10 January 2006 (UTC)

Who sees which version?

I like the look of this a lot. The only bit I disagree with is the "so which version do we show users?" bit. In order for Wikipedia to remain Wikipedia, I think it is critical that favour the latest draft wherever possible. I think many people take into account the visibility of their changes, and will be much less motivated to work on "some 'latest draft' that no one will ever see". So, if we imagine a scale of users from least committed to Wikipedia to most committed:

1. User reading some offline derivation (eg, printed encyclopaedia)
2. User reading some online derivation (eg, about.com)
3. User stumbles on WP page from Google
4. User arrives at WP and searches for a page
5. User reports errata or interacts with more advanced features
6. User has a registered account
7. User has made a significant number of changes before
8. User is an admin etc

I would propose that user 1 and arguably 2 probably don't even need to know about the existence of "draft" pages unless there is no validated version of a page they're visiting. User 3 should see a fairly prominent "This page may be aout of date" indicator or a link to the "latest draft". They should be enticed to visit it, if with a disclaimer of possible vandalism etc. User 4 should probably land on the last draft with a warning, and a link to visit the validated page. "Doing research? You may want to try the more reliable version." 5 I'm not sure. For users 6,7, and 8, they should only be working with the latest draft version, and the 'stable version' should simply be another link they can go to, akin to the history, if they have the time to validate pages or if they suspect the page is crooked.

I agree with your most of your points here, including the warnings and links to/from Current and Validated. They closely match what I propose.

User 3 (coming in via Google) might arrive at either the Validated or Current version depending on which Google ranked as the best match for the search.

User 4 (Wikipedia search box) is two different scenarios, depending on whether they hit 'Search' or 'Go'. I think 'Go' should fetch the Current (i.e., wiki) version. The wiki version should remain in its current place (wikipedia.org/wiki/Article Name), where the Validated version would be in a separate namespace (wikipedia.org/wiki/Validated:Article Name). Search could return either or both pages, depending on which namespaces the user included in the search.

User 5: The way I envision it, the Errata tab will only be shown to those already viewing the Validated version. On the Errata page, users would be given instructions on how to report errata (an "add new section" button, basically), or directed to the Current page where they could fix the error there.

Users 6,7,8: Any registered user should have the option to choose which to view by default (or "no preference"). User 7, if unregistered, might have an additional enticement to register in order to set their default view.--TidyCat 01:20, 12 January 2006 (UTC)

Aesthetically speaking, I would imagine the "stable version" as appearing as a prominent horizontal line in the history, with perhaps a warning appearing when a page has been modified more than X times since the stable version, or more than Y months have passed. (x=30, Y=2?) Stevage 22:39, 11 January 2006 (UTC)

As proposed, the Validated page will have its own history. Every newly-validated version will also be copied to the Current page (probably with a special edit comment like "Validated version"). It ought to be easy to add features to MediaWiki to have it compare the difference in age or edit count between the current version and the last Validated version. The header which links back and forth between versions could simply note the difference in time and # of edits between the two, if there were any.--TidyCat 01:20, 12 January 2006 (UTC)

Validation tags

In all this discussion it is presumed that articles have two states: validated (meets all sorts of minimum standards), and unvalidated (everything else). This would prevent a short, but correct article from being distinguished from a long but heavily biased article (both would be 'unvalidated'). Similarly a horribly written but correctly cited article would be in the same cess pool as a wonderfully written piece of propaganda with false sources. Why not have tags to cover a couple of degrees of quality along several different dimensions? For example, quality ratings of A,B,C,D,F. For example, normal might be D, with F being actually worse than normal. For dimensions, how about:

• Writing:'A' being encyclopaedia standard, F being unstructured and full of grammatical errors.
• Sourcing:'A' being better than even our featured articles now, F containing only a few false citations. (D would be uncited, say)
• POV:I'm not sure you could really grade it that finely from A to F, but F would be blatant propaganda, say.
• Comprehensiveness:A would be an article which could not be realistically expanded. D would be a stub, F might be something that pretended to be comprehensive

IMHO, there is nothing wrong with showing a non-comprehensive page with bad writing and slightly substandard sourcing to the outside world - as long as we tell them exactly what it is.

I'm presuming here that these ratings would come from a formal review process as described above, and not from a random poll of internet visitors. Rather, a group of people would submit opinions on an article, and an experienced reviewer would make a final determination and stamp the page, possibly with some overall "fit for the outside world" or "not fit for the outside world" stamp. But it would be better to just have the ratings and leave it to derivative projects to decide what level of quality they want. Stevage 22:39, 11 January 2006 (UTC)

Essentially what you're proposing is a rating system, albeit one based on the review process I've proposed. I don't feel it's as important to inform users that various articles might be Cs, Ds, or Fs, as it is to let them know This article is reliable vs. Its reliability is unknown (which is the current state of all Wikipedia articles, even the good ones).

I see validation as strictly a pass/fail standard: a way distinguish the A and B articles from the Cs, Ds, and Fs. (A "C" article would be almost ready for validation and would be a good candidate for review.) Its primary purpose should be to let visitors to Wikipedia know which articles contain reliable information without forcing them to "check it themselves."

That said, a greater specificity of evaluation could be added to the system I proposed. No article falling below a "B" in most, if not all categories, should be validated -- but reviewers could always add templates to an article which failed review (accuracy, NPOV, etc.) They could also take addtional actions on articles which did pass review, such as adding a Stub template to it (I feel stub articles can be meaningfully validated: "Correct, but please add material!"), or nominating it for Collaboration of the week or Featured article candidates. Featured Article status can serve as the remaining distinction between "B" and "A" articles.

As for distinguishing how well each article meets each criteria, that's probably a distinction more important to Wikipedia's editors than to its readers. I feel adding templates drawing attention to specific problems would be the best way to address that issue.--TidyCat 04:19, 12 January 2006 (UTC)

The problem I see is very few articles would pass. So what to do with the rest? I suggest that it's more useful to include some of the shorter (but factually correct) articles, rather than to throw out the whole lot. It also strikes me that if you're doing a review anyway, and are explicitly checking for compliance in these various areas, it is no extra work to note your final thoughts on how the article performs in those areas. Stevage 07:09, 12 January 2006 (UTC)

Well, it's hard to know how many would pass. For comparison, 0.1% of Wikipedia's articles have passed Featured review. FAC's standards are more stringent than validation's minimum standards should be, and FAC at present is still a relatively small project maintained by a modest number of dedicated Wikipedians. If a less-ambitious standard were applied to articles via a system that was implemented site-wide, I think validation could be achieved on a high proportion of articles.

I also agree with including short-but-correct articles -- as I mentioned above, "I feel stub articles can be meaningfully validated." And I agree with the idea of making use of review findings even when articles fail review. The review proceedings would by default remain on the Review Talk page, where people could see all the specific objections reviewers raised.

For greater visibility, reviewers could copy their unresolved objections to the article's Talk page. And as I mentioned before, they could add various specific "needs fixing" templates to the article. I think these steps would be sufficient to make every review improve an article, validated or not. But I'm certainly open to further ideas. --TidyCat 09:12, 13 January 2006 (UTC)

Thinking about your suggestions a little bit more -- perhaps a "Failed validation" template could be added to articles which failed validation which would direct readers to the Review Talk page. Something similar to Template:Oldpeerreview could be used. --TidyCat 09:36, 13 January 2006 (UTC)

I tend to think also that validation is more pass/fail - it is either "valid" or it is not. On the other hand, assessment is an attempt to "grade" the article according to various criteria. On WP:1.0, we currently only do assessment, and that only crudely, but ultimately we will also need to have a validation mechanism. Walkerma 21:25, 10 February 2006 (UTC)

Article Rating

I have read your proposal, so here are my thoughts. Article validation is NOT the act of improving an article to a certain quality standard. Rather, article validation is simply a verification of the quality of an article (whether it be good or bad). It is not possible to include every article in 1.0, so we will use an article's validated rating to determine which articles to weed out. The methods by which we achieve this rating are what differentiates your proprosal from mine (the WikiSort project).

You have basically dismissed a voting system in favor of a review process, but you have grossy underestimated the former. First and foremore, (and you touched on this a bit) a voting system is the only scalable system. You are overestimating the scalability of anything else. Your concerns with a voting system are primarily erroneous ratings, which are either vandalism (intentional), or misinformed people (unintentional). As outlined in the vandalism section in WikiSort, there are many ways to deal with this. Basically, the wiki software would keep a short rating history (maybe 50-100 votes), and if a new vote fell a certain distance from the average, it can safely be considered either vandalism or a misinformed vote. To further quell your fears, there are various ways to make said exclusionary process deadly accurate. The number of votes to average, as well as the deviation limit would most likely vary by subject. Furthermore, "even noise is data". That is, if the votes are all over the map, that tells you the article is controversial. In short, I think you should reconsider the power and potential reliability of a voting system.the1physicist 19:36, 14 January 2006 (UTC)

Some comments

There may be problems locking an article. It may cover a current event in which case an update is very desirable. There may be solutions for this.

We do want a gatekeeper effect. We just need dependable gatekeepers doing their job.

I think that validation is a good idea in order to try to avoid future criticism. Just recently, the Tony Blair article got a poor review because the reviewer reviewed it when it was in an inferior state.

Has a review process been used in other wikis? What were the results?

It may be a good idea to develop some kind of emergency procedure to get rid of bad claims on a validated version. This would be valuable in a future Seigenthaler incident. The procedure could be invoked and the offending claim neutralised.

I have been interested in validation for a while. Please let me know (via my talk page) if there is anything I can do to help you move things forward. Eiler7 21:54, 23 February 2006 (UTC)

I don't see why the current version would need to be locked during the review process at all, especially since reviewers would have their own protected copy of the version under review. Given your method, the "Current" and "Validated" articles are inevitably going to diverge anyway, so what's it matter if that happens the day before the review's complete or the day after?

As to "who gets what", I think the best idea would be:

-Someone who's not logged in gets "Validated" articles by default-likely, they're here looking for useful information rather then to edit, and an easy "See the version of this page you can edit" and "Report a suspected error in the Validated version" would accommodate those who do have something to say.

-I think an error-tracking system would be useful for the Validated pages, with several functions:

Keeping track of reports of errors and their resolutions (hasn't been re-reviewed yet, re-review was done and found no error, re-review has been done and found an error which needs to be corrected, re-review has been done and found an error which was corrected.) This would also easily allow reviewers to tell when an error is reported in one of their articles.

I would also suggest a "Corrections" area-if an error is found in a Validated article, a notice of what the error was and what the correct information is should be placed there as well as in the article itself.

There would need to be a "Contains errors" template, to be placed on articles for which errors have been verified (not just reported) but haven't yet been corrected yet, and possibly also a "Recently Corrected" one for Validated pages which have had a correction made within a certain time frame.

-I think that the "This page may contain errors" etc. for the current versions might be going a little far, especially if non-logged-in users will be seeing validated by default. Logged-in users will presumably know the difference if they have gone to the trouble of changing to "Always see current page by default." It could easily be read as demeaning to the work of editors of the "Current" pages, and they will continue to be an extremely valuable resource.

Various criticisms/suggestions aside, I think your idea is about the first I've seen that would actually address all of the concerns you listed, and prevent too much power from accumulating with any one "gatekeeper". It would really make a great way for Wikipedia to continue evolving without losing what makes it special, and I hope to see it or something like it implemented. seraphimblade 04:31, 16 June 2006 (UTC)

Libya

Hi,

I've recently added Libya to the list of featured article candidates. Overall the candidature is going well with many of the objections now sorted out. The final concrete objection is with the article's prose. I have been the main contributor to the article and have been looking at it for the previous 9 - 10 months. My eyes no longer see it freshly, so I am not a suitable copy-editor!

To meet the final demand of copy editing, I have been advised to ask different people to edit parts of the article.

I would really love to get this article featured as you can probably see from the page's history! I've worked very hard on it and I see this as possibly being the final hurdle.

You can see the prose objections, mostly raised by Sandy, on the candidature page. If you have the time, please choose a section (Politics, Religion, Culture etc.) and copyedit, perfect, ace it! I would be very grateful with any help I can get.

Thanks a lot,

--Jaw101ie 17:06, 20 July 2006 (UTC)

Validated articles

Some way must be found to create validated article-versions, subject to periodic updating. We can ill afford to keep thousands of capable people out on perpetual anti-vandalism patrol. Nihil novi 03:01, 7 May 2007 (UTC)