Wikipedia talk:Peer review

From Wikipedia, the free encyclopedia
Jump to: navigation, search
Main Unanswered Instructions Discussion Tools Archive

Why not Peer Review the new Peer Reviewers?[edit]

In the hope of getting my own nomination peer reviewed sometime, I decided to do some QPQ and started responding to other nominations. I found this an interesting and enjoyable experience which highlighted a few issues which I thought worth raising here.

  • It struck me that the nominee can't tell how 'experienced' (=competent?) the person doing the review is, and that any edit to a nomination (no matter how brief or useless it may be) causes the article to disappear forever from the Unanswered list. That can be frustrating for the nominator, and worrying for the newbie helper (i.e. me) if a small edit only leaves it showing on the 'Main List'
  • How does a nominator know whether they've had a visit from an expert reviewer, or from someone helpfully throwing in a few drive-by suggestions? Will they think a full review has been offered, when it's only a few helpful remarks from a newbie to the process?
  • The Peer Review team have been crying out for volunteers, it seems - but, having signed up to assist a couple of months back, no-one came along and said 'hello' or offered guidance in assisting. I know you don't want it to seem like a closed shop here at PR Towers, but that's rather how it looks from the outside, I'm afraid.
  • I'd be happy to wear an L-plate when commenting, and I'm sure article nominators would welcome knowing at what level a particular reviewer has experience in, too. If I could tell whether a reviewer is a newbie, or has experience up to GA or to FA level, then I'd be able to compare their comments with their abilities, and learn how best to do things.
  • As someone willing to volunteer, shouldn't my first few attempts be scrutinised to make sure I'm not messing things up for everyone? It's also good to have a friendly and welcoming arm around one's shoulder when starting out. Maybe I've missed an existing page of guidance for new PR volunteers?
  • To that end, I'd welcome some feedback on whether my own first attempts at leaving constructive criticism for others have set the right tone or not, and some guidance from my peers if I could have done it better. In order these are: Cambridge, then Accokeek, Maryland then Furry fandom.
  • Finally, the Unanswered List Tab seems to miss off so many genuinely unanswered articles. Whilst I fell foul of editing my own PR nomination when it reached GA level shortly after being posted, spare a thought for the nominators of Randeep Hooda and Final Fantasy Type-0 Online, both of whom made minor typo corrections a few minutes after posting their nominations. It seems these will now remain off the Unanswered Reviews List and be missed unless someone checks the Main List and looks for discrepancies between the two. To that end, I have updated the list of reviews with minimal or no feedback (9 articles up to 9th May 2016) and have taken the liberty of making that section more obvious on the Unanswered Tab in the hope that more people might spot them. Parkywiki (talk) 01:44, 10 May 2016 (UTC)
Thanks Parkywiki for starting this conversation. I can sense your frustration. It is fairly lonely up here in "peer review towers" and there are only a few of us present - and short of us manually monitoring the reviews (which is very time consuming) there's not even a real way to work out who's reviewing. (and PS - join usssss!!!!) Maybe you could create a "welcome" template we can use to greet new reviewers? With regard to "reviewing the reviewers" - great idea, but there are so few of us I worry it will deter the few remaining reviewers. I suppose we could have a mentoring programme as good articles have done, however that was limited by a small uptake and was also very time consuming. --Tom (LT) (talk) 08:16, 10 May 2016 (UTC)
Something you should realise is that reviews come in different types - there are a whole group of articles that want pre-FA reviews. You can post at the featured article talk page to get some experienced reviewers. Most people here are not familiar with that process so don't comment on those reviews. Then there are another set of reviews from new editors. Most of those reviews need reassurance and general help with wikipedia, but don't usually respond well (or at all) to comprehensive reviews - hence most regulars here don't give them. Another subset of reviews wants a comprehensive review but not for any particular purpose. The problem is (which you may find out with time here) that if you spend your time giving a comprehensive review, invariably it is ignored and not attended to. Those reviews are better directed to a good article or FA review where the editor is invested and more likely to respond to the review.--Tom (LT) (talk) 08:16, 10 May 2016 (UTC)
I am getting the feeling you are worried about the giving of reviews? I would say don't worry about this - an essential part of the Wiki culture is to dive in and give it a go (WP:BOLD).--Tom (LT) (talk) 08:16, 10 May 2016 (UTC)
Lastly, the frustrating "Unanswered reviews" tab. Even being on this tab doesn't guarantee a response. Being on the backlog doesn't guarantee a response. There is no satisfactory solution to the problem of identifying which reviews need attention. Such a list is either automated or manual. A manual list need constant attention from an editor to add and remove articles. It is impossible to maintain, as we've seen over the last few years, because editors simply don't give day in and day out attention to monitoring the reviews and manually adding them to the list (who would?) Hence an automated list. I see what you're saying about problems with the automated list those. I'll create a separate topic below so we can talk separately about a solution. --Tom (LT) (talk) 08:16, 10 May 2016 (UTC)
I'll second what Tom has posted. Moreover, you don't need to be experienced to give a PR, everyone is welcome. If you're worried that you're not experienced enough, you can always add a disclaimer while reviewing ("I don't know much about FAs but..') etc. In fact, that's what I first did while reviewing and in spite of that, helped someone get a FA. Ugog Nizdast (talk) 14:54, 10 May 2016 (UTC)

Thanks for your responses. OK, so, rising to the challenge posed by Tom (LT), I've attempted to create my first ever template - intended as a welcome message for new users. (See {{Peer review/welcome}}) It would presuppose that a user has already added their name to the Volunteer list, and that one of the watchers will detect the change and act to place the template on the new volunteer's talk page.(does that sound right?)
I recommend that:

  1. A 'Volunteers' Tab is added to the PR page;
  2. The text on the volunteer page is changed so that it also encourages new volunteers to sign up;
  3. A 'new helpers' or 'new helpers/generalists' section is added so that newbies (like me) can sign up, and not be put off by feeling they have to declare a specific interest or skill straight away in order to take part.
  4. Requesters who arrive with good-quality articles, and are helped by reviewers, are approached directly towards the end of the review to give their support/QPQ to the PR process, and are encouraged to sign up as newbie reviewers themselves.

How does that look? (for some reason I can't get the wikilink to 'discussion page' to function correctly) Parkywiki (talk) 00:40, 11 May 2016 (UTC)

It looks quite good. The discussion link works, it showing as bold here because it's on the very same page - if I understand what you're referring to. I can envision different types of requests, some just want outsite feedback, some are more specific and want to reach GA/FA standards and, finally, a bit more rare are those who seek expert attention (usually they think this is like the real world Peer review where experts review their colleague's work). Similarly those who answer the reviews can also be classified. Ugog Nizdast (talk) 02:18, 11 May 2016 (UTC)
That looks great, thanks for creating it! (plop it on the 'tools' page for future users, too). We can identify new reviewers by those who add themselves to the 'volunteers' list, and also by skimming the long list of reviews for newish reviewers. --Tom (LT) (talk) 06:38, 12 May 2016 (UTC)
@Parkywiki, @Ugog Nizdast added to the templates page! Now we can 'watch' the volunteers page and send people the template as they add themselves. --Tom (LT) (talk) 02:09, 7 August 2016 (UTC)

Volunteers list[edit]

As Parkywiki noted, we have this volunteers list. I had a quite check through it and, as I suspected, about half the listed editors on it are inactive. I have a couple of questions / thoughts for you guys:

  1. We should do a census of the list and move all the inactive editors to a section at the bottom "Inactive editors"
  2. We have two relatively active Wikiprojects, guild of copy editors and military history... instead of listing editors here, would we be better to direct editors straight to those Wikiprojects to a review?
  3. We do state "editors: yet we also hold lists of related Wikiprojects (which is probably useful), just a little inconsistent.

Thoughts? --Tom (LT) (talk) 07:33, 11 May 2016 (UTC)

This list has to be updated but how do we go about it? I hear there is this mass message sending right which a user can have. How do these other active projects maintain their list or do they even at all? Ugog Nizdast (talk) 13:33, 11 May 2016 (UTC)
I would think you could remove inactive editors without much fanfare and message active editors to remove their name if they are no longer taking requests. A-class reviews at Wikiprojects that have them are great to direct people to, but I think GOCE is a different process from PR. PR should dig more into the meat of the article (sourcing, comprehensiveness, etc) while GOCE seems more like surface-level copyediting and proofreading. --Laser brain (talk) 13:44, 11 May 2016 (UTC)
Maybe now is the time to:
  1. remove inactive editor (let's say, no editors in > 12 months on WP), and
  2. send a mass message to the remaining editors, reminding them about PR and updating them on changes in the last 1-2 years. We might get a few more regulars doing that... and we can even think about those messages to the teahouse, village pump / signpost etc to attract some new ones.--Tom (LT) (talk) 07:14, 12 May 2016 (UTC)

Timeline for auto-closing peer reviews[edit]

I've had a few editors express confusion or revert my bot's closure of peer reviews because they've received no responses yet. Should we rethink the timeline for auto-closing? It's currently set at 30 days, but I could move it back to 45 days or 60 days if necessary. We have to balance two things here - leaving unanswered reviews open and closing answered reviews in a timely manner. I'm unable to differentiate between a review that's been answered and one that hasn't using AWB, which is what the bot is based on. ~ RobTalk 20:09, 16 May 2016 (UTC)

Thanks for raising this issue here; I was the one who reverted a bot closure earlier, but I was unaware that this was the correct forum to raise the issue. I am unaware of the technical issues here, but a review closed with no response is not helpful to anybody. Would it be possible to examine them on the basis of size (Something like "close every review which has not been edited in 15 days that is >20kb in size" or something like that) or on the basis of the number of editors who have edited the review? Regards, Vanamonde93 (talk) 21:39, 16 May 2016 (UTC)
Possibly, but not using AWB. Another bot operator might be able to do something like that, but I cannot. AWB is very good at making find-and-replace edits (and basically any type of repetitive edit can be configured as find-and-replace), but it doesn't do a very good job of generating a list of pages on which to edit. The way I've worked around that is to create Category:Current peer reviews pending closure, which is generated when the last revision of a page that contains {{Peer review page}} is older than a certain amount of time (30 days, currently). I use the magic word REVISIONTIMESTAMP to do that. REVISIONUSER shows the current revision's editor, but there is no magic word to show editors before the current revision, unfortunately. It would be great to have a magic word that returns the number of editors who have edited a page, but no such feature exists in the current software. Without any method of placing pages that need archiving into a category, my bot can't work. I'm proficient with the regex used in AWB and various programming languages used for statistical analysis, but I don't code in any language that would work here (such as Perl). There are basically three routes we can go.
  1. Do nothing.
  2. Change the time after which unanswered reviews are archived, keeping in mind that this will also push back the time after which answered reviews are archived.
  3. Search for another bot operator who can take over this task using Perl or a similar language. I could reach out to a few who might be willing to do so. It's not a hard task to do, and the only reason I tried to do it with AWB is because no bot operator had picked up the task after several months. In the meantime, I could move the time for my bot back to 60 days to be very safe on what's being archived.
~ RobTalk 21:57, 16 May 2016 (UTC)
I think extending the duration seems reasonable. Thanks for helping out with this, BU Rob 13. Unfortunately this problem is not new and also used to occur with the prior bot :(. --Tom (LT) (talk) 22:27, 16 May 2016 (UTC)
Hmm. That technical issue does not appear to have an easy solution. I think the question then becomes "what does more damage?" At 30 days, we're more likely to have unanswered reviews being archived, which need to be manually unarchived; at 60 days, we're more likely to have too many open reviews, which leads to some technical issues with the main PR page (right?) and so some reviews might have to be manually archived. I actually don't know enough to make an informed decision about this, but my instinct is to err on the side of "reviews should be answered" and moving it to 60 days. Vanamonde93 (talk) 22:35, 16 May 2016 (UTC)
The issue with the PR pages is actually worse than that. As soon as additional edits are made to a PR page, they no longer show up as unanswered in the unanswered tab. So anyone who unarchives after the bot comes through would have to manually list their review on that tab or it wouldn't appear there. I think it's important that PRs aren't unarchived. Whether that means moving the time back or encouraging editors to just start a new PR page if theirs is archived - I'll leave that to the regular participants to sort out. Ping me when a decision is made and I'll make sure my bot abides by it. ~ RobTalk 23:26, 16 May 2016 (UTC)
I was unaware of that. Could somebody with the technical skills help me add Wikipedia:Peer review/The Left Hand of Darkness/archive1 back to the list of unanswered reviews? Vanamonde93 (talk) 04:30, 17 May 2016 (UTC)
@Vanamonde93: Done, see here. It's easy enough to do. The problem is that most editors have no clue they need to do that, and we can't reasonably expect them to know that. ~ RobTalk 05:55, 17 May 2016 (UTC)
Ah okay thanks. Given what you just pointed out, I am leaning towards extending the automated time-period, perhaps to 60 days. Vanamonde93 (talk) 17:22, 17 May 2016 (UTC)

──────────────────────────────────────────────────────────────────────────────────────────────────── @Vanamonde93, BU Rob13, Ugog Nizdast, Parkywiki sorry, have been busy for the last two months. Pinging this discussion + other active users on the page. I'd support a change to 60 days for autoarchiving of unanswered reviews. Other thoughts? --Tom (LT) (talk) 01:59, 7 August 2016 (UTC)

I'm 90% sure I already unilaterally did this because I saw a non-trivial number of people unarchiving the discussions once they were archived, sometimes doing it incorrectly. This was causing problems, so I pushed the date out to 2 months. ~ Rob13Talk 02:03, 7 August 2016 (UTC)
@BU Rob13 seems to be quite a lot of answered old reviews not closed... time to run the bot again :)?

Peer Review Reform Proposal[edit]

There is Wikipedia_talk:Featured_article_candidates#Peer_Review_Reform_Proposal by another editor. It's a more broader proposal actually but still does concern us. Ugog Nizdast (talk) 19:56, 11 August 2016 (UTC)

It is not a proposal yet, but I am developing a proposal, and for that purpose your comments and suggestions and criticisms are welcome at the discussion page.·maunus · snunɐɯ· 06:41, 12 August 2016 (UTC)

Iazyges' Peer Review Reform Proposal[edit]

I don't have quite the 95 thesis of martin luther, but my proposition is this, remove FA and GA things entirely, while this may sound radical, I believe it will allow the PR to function better, that rather than the PR checking things against a criteria checklist, we give suggestions to or improve the article itself. The second part of this is obviously that another body will need to be formed, perhaps the CR, the criteria review. The two being seperate entirely, the PR giving suggestions on improving it, the CR given suggestions on how to make it fit the criteria of either GA or FA. The PR would likely be the first step, after it is improved by the PR, it can be sent to the CR to be check for criteria, after which it can be nominated. By splitting the two, you not only lessen the backlog, but you allow people who only want to give suggestions to help it meet the criteria to do just that at the CR, and people who just want to give suggestions for improving the article to be able to just that at the PR, people who want to do both may do both. This change would allow for streamlining of cases, as the PR would no longer be involved in criteria, as i which would split the workload, and likely the backlog in half. This reform is obviously open to change, and will likely be denied, but I thought it might help. TL;DR: split the peer review process and the criteria review process. Iazyges (talk) 17:34, 12 August 2016 (UTC)

This proposal is actually very similar to my proposal, except that I don't think there is any reason to distinguish between your PR and CR. The objective of any review should be to improve the article - in my model the team working on the article review when they feel like it can make the decision to nominate for GA or FA which is then put to a !vote - they don't have to nominate to vote that is something they decide together, hence there is no need for the PR/CR difference.·maunus · snunɐɯ· 07:41, 13 August 2016 (UTC)

PR bot?[edit]

Suggestion: Make a PR bot that acts like the bot for RFC, if you've signed up for a category, like history, have the bot give you a link every time something of that category is added. Iazyges Consermonor Opus meum 00:37, 19 September 2016 (UTC)

@Iazyges great suggestion! It'd also get a lot more interested readers. What's the bot's name / who created it? Maybe we can ask around and see if anyone's interested in making such a bot? --Tom (LT) (talk) 09:36, 19 September 2016 (UTC)
@LT910001:, The bot is User:Legobot, owned by User:Legoktm, I will ask him if he is willing to make it. Iazyges Consermonor Opus meum 11:40, 19 September 2016 (UTC)