Wikipedia:Requests for comment/Pending changes trial

The following is a request for comment on the upcoming trial of flagged protection.

Background

• The poll approving the trial of flagged protection and patrolled revisions, a specific implementation of flagged revisions, was closed on 1 April 2009.
• The Wikimedia Foundation created a team for the rolling out of the trial, a testing site was created to test the configuration, it is used since September 2009.

Situation

• The team in charge of rolling out the trial has announced they were in pre-rollout phase. The rollout is expected in the next few weeks.
• Only flagged protection will be part of the trial, patrolled revisions has not been developed yet.
• The configuration as requested, flagged protection with reviewer usergroup and option to deactivate autoreview, is currently enabled on the testing site and fairly stable.
• Despite this, it seems that based on 'private conversations' the team wants to deploy for the trial a version with no reviewer usergroup and give to autoconfirmed users review rights while the community had repeatedly rejected this possibility.

This RFC

To determine how the community should react to this, hopefully find agreement then work out other open issues with the trial.

Reviewer usergroup

Flagged protection without reviewer usergroup was the original version of flagged protection, discussed here, consensus was not found and it was finally rejected in favor of the new version. Autopromotion of reviewers, and a fortiori giving review rights to all autoconfirmed users was rejected at Wikipedia talk:Reviewers#New discussion and poll: reviewer criteria and in a prior poll.

Reasons for using a reviewer usergroup

• high risk for good-faith autoconfirmed users to be sanctioned for doing bad reviews due to inexperience (and not malice)
• very high risk for malicious users to compromise the system with bad reviews
• lack of confidence in the reviewing system
• no possibility of protection against sockpuppeters and major disruption as afforded the intermediate flagged protection level, need to use full protection as now
• deemed untenable by FlaggedRevs main developer Aaron Schultz
• need to double check reviews which in turn may affect the efficiency in the processing of edits

Reasons against using a reviewer usergroup

• less complicated
• more users with the ability to review - less likelihood of backlog

Discussion

• I'm not convinced that this would reduce backlog since there will be a need to double check reviews and deal with bad reviews. We had planned on a liberal granting of the reviewer group so most users interested in reviewing would probably be granted the rights anyway. With database reports identifying candidates likely experienced enough for the right, it would be unlikely to have shortages in reviewers. Further, edits by autoconfirmed users are automatically reviewed when the previous revision is reviewed (except when specifically disabled to deal with socks/major disruption but those instances would be rare) so the number of edits to review is considerably less important that in other type of FlaggedRevs implementation, and also obviously because it isn't used on that many pages. Cenarium (talk) 14:30, 22 May 2010 (UTC)
• The wiki way is to be as open as possible for as long as possible. This is a trial period. Therefore, it seems to me that the best way to find out if fears about autoconfirmed users gaming the system or causing trouble in some way is to start out with granting the right to autoconfirmed users and then scaling it back after the test if we have empirical evidence that they are doing something wrong.--Jimbo Wales (talk) 15:10, 22 May 2010 (UTC)

I would agree with that if it were for a passive right, and it's already the case that on flagged protected pages all autoconfirmed users are automatically reviewed if the previous revision is, but here we're talking about reviewing other user' edits. It is unlikely that inexperienced users even know how to read a diff, it will most certainly lead to usability issues by confusing them, etc. I think an autopromotion would be a much better compromise, so we can at least remove the right from good-faith users misusing the right instead of having to block them, and allow to raise the bar a little. Another point, if the community isn't satisfied with the trial, which is most likely if all autoconfirmed users can review and its consequences, then the community simply won't want FlaggedRevs any more. Cenarium (talk) 16:23, 22 May 2010 (UTC)

I don't understand this logic, so the community will prefer the far less restrictive status quo over FlaggedRevs because FlaggedRevs is not restrictive enough? Sole Soul (talk) 18:09, 22 May 2010 (UTC)

That's really not a question of being more or less restrictive. All autoconfirmed users' edits are already automatically reviewed if the previous revision is reviewed, and cases where an autoconfirmed user edits a semi flagged protected page with a last revision which is not reviewed should be quite rare. Cenarium (talk) 18:23, 24 May 2010 (UTC)

I expect it to be rare during the trial, but I'm not sure what will happen later when the enthusiasm for active reviewing subside. There is also something that cannot be measured easily, and that is the psychological barrieir that may prevent an autoconfirmed users from editing a page in the first place when they know that their edits will not appear immediately. Sole Soul (talk) 19:58, 24 May 2010 (UTC)

• Unless there's some technical reason behind the switch that would result in significant delays (i.e. more than a month) or make deployment actually impossible, the technical staff should not be overriding community consensus. I agree with Cenarium that if the trial goes poorly for any reason, the community will almost certainly not accept FlaggedRevs in any form. Mr.Z-man 16:40, 22 May 2010 (UTC)
  • Hi there. The problem is manifold: 1. there's a usability issue. the knobs and widgets necessary to implement the full proposal as written need a lot more thought toward usability than we have time to implement. 2. defaults matter. if people are, by default, not included in the review group, there will be a lot of very qualified editors who would not participate due to status quo bias. 3. a primary target for this feature is articles currently under semi-protected status. There will be rightful hesitation to use this as a replacement for semi-protection if the people who currently have the right to have their edits show up immediately lose that ability.
    I'm not wedded to the configuration that's listed in the Wikipedia:Flagged protection and patrolled revisions/Trial, and I'm happy to represent some other viewpoint to other WMF staff if you all can convince me that what's up there is wrong, but whatever we come up with needs to be simpler than what's on Wikipedia:Flagged protection, and needs to be nailed down very quickly to ensure this feature gets launched in a timely fashion. -- RobLa (talk) 17:47, 22 May 2010 (UTC)
    • As for #1, I disagree. This aspect of the FlaggedRevs feature is targeted toward admins and experienced users. Usability is a major issue for things targeted toward new users and readers. But for those groups, the result will be basically the same either way. As for number 2, our experience with rollback, autoreview, and accountcreator suggest otherwise. That could also be mitigated using an autopromotion system more stringent than autoconfirmed. What you should be wedded to is the configuration that the community agreed upon. Unless the board agrees to change things or an authorized staff member changes it as an office action or due to actual technical problems (i.e. things that may crash the site, not just assumed usability issues that the WMF has had no problem sticking on other wikis) the staff should implement what the community agreed on. That's how things have worked for years, and I see no reason to change that. You do not run this project. You do not get to override community policies. Mr.Z-man 18:21, 22 May 2010 (UTC)
    • Usability is a secondary concern at this point. I'm willing to entertain a healthy amount of debate here, but the primary concern should be to get a working implementation of the software that the community has requested. I'm already disappointed that there is yet no implementation of patrolled revisions (which should have been far simpler to implement than flagged protection), but I think it's totally unreasonable for usability to be a blocking issue for this software at this point in time. {{Nihiltres|talk|edits|⚡}} 19:02, 22 May 2010 (UTC)
• Can I ask, (as someone who's only started editing since the discussion, poll etc.) under this system (ie the one proposed for the trial here, with autoconfirmed users being reviewers), will every edit by an autoconfirmed used mark the preceding version as patrolled? It would, wouldn't it? Surely this would lead to large numbers of versions being marked as patrolled by autoconfirmed users who didn't mean to, but were simply trying to make an edit? -- Lear's Fool (talk | contribs) 19:21, 22 May 2010 (UTC)
  • No, if the previous edit (i.e. the current version) was not reviewed, then reviewing an edit is an active process (it requires checking a box or an extra button), not a passive one. This is independent of the actual system used and will be the same for any usage of FR. Mr.Z-man 21:26, 22 May 2010 (UTC)
• I oppose FlaggedRevs, as it suddenly disconnects the readers from the editors. On Wikibooks, logged in users see "the latest revision" and the public sees an old revision. For current events, it would be another step to get a needed change (a fact fixed). WP:RFA-like (approval) system would be torture; might as well stop editing for all users except admins. The idea of usability and "anyone can edit" would be destroyed; the editing box is confusing enough without confusing "Revision 5428294 rejected by Admin". There are too many autoconfirmed users for this to work; too few users would be chosen as reviewers. A possible way to use the groups would be to tie rollback permission to FlaggedRevs; I think the community should decide, not Mr. Wales. Also, the other projects (Wikibooks, etc.) are nothing like Wikipedia! (see how WN is different from WP; it frustrates me to discover that content between WP and WN cannot be copied.) Finally, WP:TROLLs, disruptive editors, page-move vandals, etc. would have a new way to mess up the system without a ton of people knowing (most people of this nature are autoconfirmed).--moɳo 20:30, 22 May 2010 (UTC)
  • Followup: I noticed a comment about patrolled revs would be so much easier to implement (WP:HG, WP:IG, WP:LUPIN) than FR and wouldn't confuse anyone. Instead of redefining the system, you could just add something new and not that different. --moɳo 20:35, 22 May 2010 (UTC)
  • You appear to be confusing FlaggedRevs (the MediaWiki extension) with Flagged Revisions (a particular implementation of that extension). The plan on the English Wikipedia is to use a different system of flagged protection and patrolled revisions. Flagged protection is essentially using Flagged Revisions only on pages which would otherwise be protected to some degree. Patrolled revisions (which you seem to understand better) is simply passive "This revision has been reviewed" flags for revisions, which will likely be useful for maintenance and removal of vandalism. Please take these differences into account when commenting on these discussions. Cheers, {{Nihiltres|talk|edits|⚡}} 15:33, 23 May 2010 (UTC)
• Giving the review rights to autoreviewers seems like a sensible idea. But we should change the criteria for autoreviewer qualification. As not many have made around 75 articles. We could change the criteria to say 100 good edits. Graeme Bartlett (talk) 03:57, 23 May 2010 (UTC)
  • It does sound like a good idea. But I think 100 good edits is not quite enough, because autoreviewer will still have the same function it has now, I assume. So at least 5 new articles that are referenced and (at least borderline) notable, plus the 100 good edits. I'm saying this as someone who has 2k edits but only four articles, three of which are unsourced. However, if it is possible, I'd like the two groups to be separate. I would like to be a reviewer, that's the kind of job I like, but I'd be a nightmare autoreviewer because I haven't worked much with actual content-building. Maybe, as mono said, the type that would make good reviewers are the people who are already "approving" edits with Huggle- the rollbackers. {{Sonia|talk|simple}} 08:29, 23 May 2010 (UTC)
    • Well as someone who makes a large amount of edits but rarely start articles. I can tell you are immediately going to hit a wall with that. Perhaps you would wish to review my 500 typo edits this day? Yes??? No, thought not. ;) Regards, SunCreator ^(talk) 09:43, 23 May 2010 (UTC)
      • In the spirit of SunCreator's objection, I'm not sure "x new referenced articles" would be an easy enough criteria to meet, specially considering how often people argue that, with over 3 million articles, the number of notable topics in Wikipedia is approaching saturation... --Duplode (talk) 16:46, 23 May 2010 (UTC)
  • I believe giving reviewer rights to autoreviewers was always planned. However, the WMF FlaggedRevs people have decided to change the plan and give it to all autoconfirmed users – users who have made 10 edits and have been here for 4 days. Mr.Z-man 14:12, 23 May 2010 (UTC)
    • That seems sensible, both in method and scope. It's somewhat like the wikibooks criteria. Four days and 10 edits works well to stop reaction response and vandals; confirmed so that you know they have email account and can be reached. Giving reviewer rights to all confirmed users because otherwise Wikipedia will grind to a halt editing wise at the beginning. Regards, SunCreator ^(talk) 15:51, 23 May 2010 (UTC)
      • Autoconfirmed does not require an email address. Wikibooks requires significantly more experience before reviewer rights are given automatically - 100 total non-deleted edits, 30 days, at least 50 content edits on 10 unique pages, no blocks, and an email address set. Mr.Z-man 16:44, 23 May 2010 (UTC)
  • Here is the rationale for choosing autoconfirmed as the first level of flagged protection to roll out. The primary candidates for flagged protection are articles that are currently under semi-protection. Articles that are under semi-protection are currently fully editable by autoconfirmed users. If we use any other level other than autoconfirmed, then putting them under flagged protection actually takes away rights that those users previously held. We can always add more levels later, but the idea is to start off with the most inclusive level so that people can get comfortable with the feature itself, and then expand based on community demand. -- RobLa (talk) 17:27, 23 May 2010 (UTC)
    • I understand your rationale for preferring it, but what is your rationale for ignoring the community consensus and substituting your own opinion? (And I thought the rationale was usability?) It has taken more than a year to get the trial that we got consensus for last April. Now you're changing things based on your own opinion. Forgive me if I have absolutely zero confidence in the FlaggedRevs team to do anything based on "community demand." Mr.Z-man 17:43, 23 May 2010 (UTC)
      • Jimbo has been disputing the media's claim that the new system is less inclusive. When you look at the original proposal, the media seems partially correct, as the proposal takes away rights from autoconfirmed users as RobLa said. Sole Soul (talk) 17:53, 23 May 2010 (UTC)
        • I don't care what the media says. I want to know the WMF staff's rationale for giving themselves the power to overrule the community on matters that are neither serious technical nor legal issues. Up until now, the WMF has limited their involvement in setting local policies and procedures to things that pose a serious problem to the site (technical changes needed to keep the site running, DMCA notices, etc.) or things that there isn't an explicit consensus on (the recent skin change, most new features). Here they appear to simply be substituting their own preferences because they simply don't like what the community decided on. This represents a major shift and I am frankly shocked that they are doing it so lightly. Mr.Z-man
      • Hi Mr.Z-man. I know you are frustrated with how we got where we are, but I'd ask that you please assume good faith.

        We didn't ignore community consensus. As near as I can tell, the feedback from the community on these specific issues was ambiguous. For example, the first poll on the issue over a year ago was split 22 support, 32 oppose. While that may be a relatively strong majority, that's not consensus. In a community the size of the English Wikipedia editors community, that doesn't even seem like a quorum. The later poll about auto-promotion was similarly small, with 13 support, 19 oppose, and 2 neutral. My reading of the polls and the other discussion is that the typical community member isn't tracking things at the level of detail we're discussing here.

        What I did at that point was to create a configuration and post on wikien-l notifying everyone that I was making changes, and that things would be in flux. I later found out that there had been earlier conversations at WMF, and something even simpler than what I first posted was what many folks agreed to there. After talking it over with them, I became convinced myself that they were right, and changed the page accordingly.

        In answer to your question about rationale, the rationale for simplifying it down to one option was usability and the ability to explain how the feature works as we introduce it. The rationale for picking the option we did is explained above.

        We're committed to making this trial successful. We want the editing community to like this feature. We will be receptive to the reasons why what we proposed will not work for the short duration of the trial. Let's focus on the proposal so that we can get something implemented. Thanks! -- RobLa (talk) 21:49, 23 May 2010 (UTC)

        • What proposal? The option to chose between 2 predetermined names for the system? I don't see anything here being proposed, I see the foundation saying "Well, you asked for that, but we decided to give you this instead, we hope you like it." Mr.Z-man 22:55, 23 May 2010 (UTC)
        • There's been no proposal, I had to reveal that now as you didn't bring this for comment publicly. Foundation staff discussed this since the beginning of April; a person of influence (not Jimbo) advanced the opinion that we should give review rights to all autoconfirmed users, Aaron was asked to comment on the technical aspect and he raised a few objections, some of the reasons advanced here, he then suggested to ask me. I gave the community position as I see it, and afterwards there has been apparently an overall agreement that autopromotion would be a good compromise. My last email didn't receive a reply and now I learned that there were no more questions, it had been decided to give review rights to all autoconfirmed users, so why ignoring Aaron and the community's concerns, why having decided this with minimal community input ? Cenarium (talk) 03:10, 25 May 2010 (UTC)

• Endorse RFC concern (if technically accurate) - I would not be comfortable with reviewer rights being auto-distributed, to autoconfirmed users or any other way. Reasons that speak to me:

  1. It's "just a trial". But it's a high profile trial and the entire efficacy of the Flagged Revisions approach will be carefully scrutinized, here and with commentary in the media. A weakened version that allows anyone - including vandals and users who have not shown they understand our policies and criteria (what "vandalism is and isn't") - to get reviewer rights anyway, won't help us evaluate the tool and will increase the chances of negative evaluations.
  2. We expect many people to get the tool. Many people get rollback too. But there is a difference between "many" and "indiscriminately anybody". Support "many", oppose "indiscriminate/automated", for reasons the community has agreed as well.
  3. A tool that requires double checking in this way is too different from that discussed. The aim is to save work by having users with some basis of trust and some element of scrutiny to have that access. Not "everyone".

  If the RFC statement is accurate then I would agree with it as being a concern. FT2 ^{(Talk | email)} 00:36, 24 May 2010 (UTC)

• Endorse separate user group. However, admission to the user group should be like rollback, readily granted on request by any administrator. It should be a separate group because of the potential for good-faith abuse, not to mention the other kind of abuse; it could then readily be revoked by any admin without blocking, pending discussion, etc. It is also possible for the right to be automatic, with autoconfirmation, but separately revocable. As a trial, it seems safest to have it be like rollback. --Abd (talk) 14:42, 24 May 2010 (UTC)

• 4 days and 10 edits to review? I'm not sure that's a good idea. A separate group is needed. ++Lar: t/c 01:36, 25 May 2010 (UTC)

• Addressing the points by RobLa:

1. Usability issues would come with the reviewers group removed, not the reverse. Admins and experienced users are already accustomed to relatively complex tools, they won't be afraid by two new levels of protection, while readers and inexperienced users would not be bothered. Further, the configuration as requested is already up and running on the testing site and it works well enough; usability is not synonym with 'indefinite polish up'. Plenty of admins have tested the interface and didn't fin any substantial usability issues in the interface, they can use it, and even then they know where to find help if needed, there's no issue. However, there would be a usability issue, and a big one, if inexperienced autoconfirmed users were faced with strange things named 'diffs' to 'review'.
2. Alleged status quo bias: we're going to use database reports to identify users who are likely experienced enough to be granted review right or as second option an autopromotion so it's addressed.
3. At the risk of repeating it again and again, autoconfirmed users have their edits automatically reviewed if the prior revision is reviewed, cases where an autoconfirmed user edits a flagged protected page with unreviewed latest revision should be quite rare. On the other hand if experienced users have to double check reviews, then there will be less time to review new edits by IPs and non-autoconfirmed users.
4. What's on WP:Flagged protection is old, the community request is at WP:FPPR and you know this is exactly the configuration which is on the testing site. I don't get the 'it's simpler' argument, readers and most users won't know the internal, it's going to matter only to admins and the users who are going to review edits, I assure you they can handle this level of complexity.

If we don't get what the community wants of the implementation, then what's the point of having a trial, so far remote of what the community could support using on longer terms ? You can think the two polls on autopromotion are anecdotal, but very clearly the community rejected the first version of flagged protection, without reviewer group, in favor of this one, with reviewer group. Cenarium (talk) 03:10, 25 May 2010 (UTC)

• I'm generally with Z-man and Cenarium. Can we please just get on with the trial as discussed, and not talk this to death any further? It's a trial, for crying out loud: If we run into a noteworthy backlog, we then change the process.
  To keep the concept somewhat useful, the group of reviewers must be large and active enough to keep the backlog small, but small enough to only contain reasonably trustworthy users. If we manage that, editing remains as open for autoconfirmeds as it used to, and is more open for IPs.
  Four days/ten edits are *far* from what's required to keep the reviewe process meaningful. Not being able to revoke reviewer status and having to fall back to blocks in case makes that change even worse. Amalthea 13:51, 25 May 2010 (UTC)
  • Amalthea, the reason why I started down the path I did was because I was trying to align the configuration with what was on the web page. After figuring out that I didn't have enough information on the web page to do that, I decided to scratch that, and go with a simplified version. Then, after talking with people at WMF about the conversations they had had before I even arrived, they had something even simpler in mind. So, that's what I went with. -- RobLa (talk) 16:33, 25 May 2010 (UTC)