User talk:Nettrom

From Wikipedia, the free encyclopedia
Jump to: navigation, search


Hello Nettrom, Eduemoni↑talk↓ has given you a shinning smiling star! You see, these things promote WikiLove and hopefully this has made your day better. Spread the Shinning Smiling Star whether it be someone you have had disagreements with in the past or someone putting up with some stick at this time. Enjoy!

Wikipedia:Community portal/Opentask[edit]

Heya; apparently SuggestBot should be updating Wikipedia:Community portal/Opentask, but doesn't seem to be :/. Any chance you could hit it with a stick? :). Ironholds (talk) 19:34, 18 January 2014 (UTC)

I should probably update the template while this part of SuggestBot is on leave, awaiting being moved to Tool Labs. With the shutdown of the Toolserver, I expired my account there and packed things up, so it's currently not running. At the moment I'm busy chasing a conference submission deadline, so it'll have to wait until after the 23rd. I will have some time to make some adjustments to the code and get it running on Tools Labs then, though, so it should be back in operation before the end of the month. Hope that's not a problem? Cheers, Nettrom (talk) 23:56, 19 January 2014 (UTC)
Naw, that sounds good :). Let me know if you have any difficulty with Tool Labs - I know people who know people on ;p. Ironholds (talk) 18:54, 20 January 2014 (UTC)

Hello, Nettrom. Any news on this front, please? Thanks. --PFHLai (talk) 11:33, 1 July 2014 (UTC)

@PFHLai: I'm sorry for not getting this done. The main reason is that I'm conflicted about the Community Portal, as it is right now it does not appear to be an effective way to engage editors. Both my own research (which I hope to have published soon) and WMF's click data from the 2012 redesign (meta:Research:Community portal redesign/Opentask#Results) indicates that the portal doesn't really lead to saved edits. The WMF data suggests about 2-3 edits per day come through the portal, which is not a lot given that it gets about 8-10,000 views per day (ref Wiki ViewStats graph of WP:CP).
I'm not sure I have the time to lead a redesign effort of the portal, but I'd be happy to get the conversation started by writing something on the talk page. Then we could see where we go from there? Regards, Nettrom (talk) 16:46, 2 July 2014 (UTC)
That's alright, Nettrom. I'll take care of the template manually for now. Thank you for looking into this. Cheers! --PFHLai (talk) 03:26, 3 July 2014 (UTC)
@PFHLai: I wasn't happy about the answer I gave you and having you doing all this manually, so I went ahead and edited the script that updates the task list. It's a simpler version than what we had before in that it only picks articles randomly whereas the previous version used data on article popularity & quality for some of the articles. I've got it ready to run on Tool Labs now, so just let me know and I'll set it to update Wikipedia:Community portal/Opentask again. Regards, Nettrom (talk) 20:38, 3 July 2014 (UTC)
That's very nice of you, Nettrom. Please feel free to unleash your bot on WP:OT anytime. Thank you very much. --PFHLai (talk) 10:11, 5 July 2014 (UTC)
No problem, PFHLai! I've now set the bot up to update the list twice every hour. Thanks for taking care of the list manually while SuggestBot was away, and for nudging me to get it working again! Regards, Nettrom (talk) 11:05, 6 July 2014 (UTC)

Another barnstar for you![edit]

Vitruvian Barnstar Hires.png The Technical Barnstar
Thank you so much for your work on m:Research:Ideas/Screening WikiProject Medicine articles for quality/Prediction table. This is going to save me so much time and effort. I really appreciate it! Thank you, WhatamIdoing (talk) 20:33, 7 August 2014 (UTC)
@WhatamIdoing: Thanks so much! I was tremendously happy to get this started for WP Medicine since doing something like this has been an idea for a long, long time. Glad you like it, am looking forward to learning about your experiences and how we can make it work better! Cheers, Nettrom (talk) 16:50, 8 August 2014 (UTC)
I checked the bottom ten articles yesterday—the ones rated right around 50%—and they were really right about 50% in my opinion. The higher rated ones were of course definitely not stubs.
Would it be both useful and not too much work to see which ones of these get changed (over the next couple of weeks), as a means of validating the probability levels? WhatamIdoing (talk) 18:15, 8 August 2014 (UTC)
@WhatamIdoing: I'm pleasantly surprised that it seems that even a low probability of stub (e.g. around 50%) is still often right. It might be that we can improve the software, I have better data available but did not have the time to use it before the Hackathon. Next time we run it the results should be better (although exactly how much better I do not know).
And yes, I have been thinking about following up on this, both to understand which articles were reassessed, but I think it would also be great if I could discuss some of the articles that weren't reassessed with the WikiProject members, as I'm really curious to understand where the software failed. We could use that feedback to hopefully improve the training of the software as well. And it shouldn't be too much work to do, I have some code from a different project that looks for article reassessments, it should be easy to modify that. Cheers, Nettrom (talk) 17:39, 10 August 2014 (UTC)
I still do a lot of the article assessment for WPMED, and you can always talk to me. WT:MEDA is the official page for such discussions; you're welcome at the main WT:MED page, too. Wouterstomp does a lot of this work, too, and so does CFCF. WhatamIdoing (talk) 00:20, 11 August 2014 (UTC)

I've got a false positive for you and User:EpochFail to consider: Henry Ford Health System rated above 95%. It has a lot of tables, but almost no text. It's possible that someone else would rate it as Start-class, but I think I'm leaving it as a Stub. WhatamIdoing (talk) 22:44, 12 August 2014 (UTC)

Shuvo Roy is another false positive (same type: lots of lists, almost no actual sentences). WhatamIdoing (talk) 22:24, 15 August 2014 (UTC)
Also Monica Lin Brown. It might be appropriate to ignore anything inside blockquotes. WhatamIdoing (talk) 22:32, 15 August 2014 (UTC)

Thanks for doing this! The predictions seem pretty accurate. It would be interesting to see the actually predicted class as well. Can you also do this for other classes? (I assume stubs that aren't stubs were the easiest to start with). Also could you provide the list of redirects that were classified as stubs so we can correct those? --WS (talk) 07:29, 13 August 2014 (UTC)

Ooh! Good idea. I've added a new column to the table to include the predicted assessment class. I'll look into making another run to show articles that need re-assessment in other classes. Maybe Nettrom will beat me to it. ;) --EpochFail (talkcontribs) 09:33, 13 August 2014 (UTC)
I think this would be interesting, but it does seem a lot less likely to be accurate. WhatamIdoing (talk) 22:24, 15 August 2014 (UTC)

Automation and improving the slush pile on ENWIKI[edit]

I watched caught the Research showcase video today, and it sort of got me thinking. Just some brainstorming here...

One of the places I do quite a bit of editing and reviewing is ENWIKI's article for creation process. If you are not familiar with it, in broad terms, it is an optional path that new editors can use to create article drafts, and wait to have them reviewed before being subjected the the full onslaught of Wikipedia's gatekeeping. Severe problems tend to be caught relatively quickly, but more modest, arguably correctable problems, in theory have a chance to survive.

It is also a somewhat ... err, problematic process. At present, there are roughly two thousand, six hundred new editors who have submitted new articles waiting for some feedback. A third of those have been waiting three weeks or more without having had a word said back to them. Many of the people who wait that long ... will never come back. I often worry that the process is a whisper-chipper for potential new editors.

Most new articles.. the vast majority of new articles, do not survive the process. It is my guess without evidence that most of the associated editors... same result. For the promotional ones, many of which are self-promoting or otherwise interest conflicted, that is probably an okay result. But as a total process result, it concerns me.

Here are some angles on where SuggestBot or similar ML techniques could (I say, handwaving feverishly) perhaps help things a bit:

  1. Identify interesting drafts to SuggestBot users and include them in Suggest results, drawing additional eyes to article drafts. Difficulty: Picking out the minority of salvageable drafts.
  2. Identify "articles more likely to survive" for the small cadre of AfC reviewers, letting reviewers focus more on articles with survival potential, and less on those with little survival potential.
  3. Identify cases where ML could actually provide constructive advice prior to the arrival of a trusted reviewer. (Probably hard.)
  4. More general understanding of the AfC slush pile.

Anyway, those are some thoughts that came to mind. Thanks for publicizing your research, and for SuggestBot. Cheers, --j⚛e deckertalk 01:02, 21 August 2014 (UTC)

Update, please?[edit]

Could you or User:EpochFail run an update on m:Research:Screening WikiProject Medicine articles for quality/Stub prediction table? I've personally gone through the top half or so, and I know a couple of other people have been through parts of the list as well. It would be a little more efficient for me if the ones that they've already updated were taken off the list.

If it's a hassle–really, if it's more than about ten minutes work–then it might not be worth your time, but if it's easy, then I'd appreciate it.

By the way, your predicted percentages are still holding approximately true. WhatamIdoing (talk) 22:25, 9 September 2014 (UTC)

I had this on my calendar yesterday, but failed to get around to it. Will take care of it later today. Cheers, Nettrom (talk) 14:46, 11 September 2014 (UTC)
@WhatamIdoing: The table is now updated. I also created a page with the list of redirects that were identified, in case WP Medicine wants to fix those too, see m:Research:Screening WikiProject Medicine articles for quality. Regards, Nettrom (talk) 23:28, 11 September 2014 (UTC)
Thanks! I'm starting down the new list. I see that a few more have been added. I have found one copyvio among the recently enlarged articles. Thanks, too, for the redirects list. Cleaning out mislabeled redirects is something I haven't done for over a year, and I can see that it's high time.
I've been thinking about the idea of producing a predicted status for all articles. The line between a long-ish Stub and a weak-ish Start is pretty fuzzy. What might be most useful as a starting point for that is a report only on the articles that are wrong by two classes–not Stubs that are probably Starts, but Stubs that are probably C-class or better, or, in the other direction, C-class articles that are probably Stubs. That would clear out the most significant outdated (or vandalized; it happens) assessments before going on to the much longer list of slightly-off ratings. WhatamIdoing (talk) 00:09, 12 September 2014 (UTC)
I've cleared the entire redirects list, and blanked it so that anyone visiting the page will know that it's done. Feel free to ping me if future runs find more. WhatamIdoing (talk) 00:47, 12 September 2014 (UTC)
One more update, please? I've reached the end of the list. The page currently lists 490 articles, and while some of these are still stubs, more than half of them will go away. WhatamIdoing (talk) 00:02, 28 October 2014 (UTC)
Hi WhatamIdoing, I'll take care of the update tomorrow (Oct 30). Cheers, Nettrom (talk) 22:38, 29 October 2014 (UTC)
@WhatamIdoing: I have updated both the main table as well as the redirect table, there were ten redirects this time around that you might want to have a look at. Let me know if you there's anything else! Cheers, Nettrom (talk) 20:06, 30 October 2014 (UTC)
Thanks! I'm halfway through the new ones.
In terms of turning this into a sustainable ongoing task (e.g., the bot updates it periodically and editors go over the ratings), it would be useful to know which ones are newly added and which ones are old. I can look at the color of the link and know that I visited this page recently, but not that one, but nobody except me can know which ones I've already reviewed vs which ones still need to be done. I don't know how much work that would be, or if it's worth it.
User:Dank was interested in seeing how this worked out, with an eye towards seeing whether MILHIST would benefit from a similar re-assessment drive. I think that I've reassessed about 1% of WPMED's articles (about 400) as a result of this list. As I said earlier, the ones that are probably very wrong (rated as stub but predicted as C or higher) are probably the most important ones to review, but the predicted levels were reasonably close to my experience. WhatamIdoing (talk) 22:31, 30 October 2014 (UTC)

Does SuggestBot have a redlink pool?[edit]

Hi, there are some high-value redlinks at Wikipedia:WikiProject Missing encyclopedic articles/Thompson-Reuters most cited scientists which I've suggested for Wikipedia talk:Community portal/Opentask#Scientist redlinks. Please see and please do it because starting new articles is fun. Do you have a redlink pool somewhere for SuggestBot? EllenCT (talk) 16:41, 26 September 2014 (UTC)

SuggestBot and HTML5[edit]

Hello there! I was wondering if you could update SuggestBot to have the tables it posts on user's talk page be HTML5 compliant. I'd be happy to help if you point me to the source code for the script or templates used to make those tables. Currently, I'm seeing a lot of align='right'align='center'align='left' where style="text-align: right;"style="text-align: center;"style="text-align: left;" would be up-to-date and appropriate. If you could do this, it would fix a huge eyesore for me and a couple other users. Thanks! — {{U|Technical 13}} (etc) 05:40, 27 November 2014 (UTC)

@Technical 13: Thanks for letting me know about this. I'm a little swamped at the moment, but I've made a note to look into this, should be done some time next week. I suspect it's only a matter of minor changes to SuggestBot's templates to get it taken care of. Thanks again! Cheers, Nettrom (talk) 15:44, 29 November 2014 (UTC)
  • If you want to point me to those templates, I'd be happy to make the needed changes. I can edit on-wiki templates (even if I need to submit a specific edit request to a Template editor/Administrator protected page) and I could submit a pull request if it is on BitBucket. I don't mind. If not and you'd rather just do it yourself, I understand and that's not a problem either. Happy editing and thanks for your time! :) — {{U|Technical 13}} (etc) 16:45, 29 November 2014 (UTC)
@Technical 13: Thanks so much for volunteering, that's much appreciated! I would've pointed you to the templates if it didn't mean I first had to go poke around in SuggestBot's configuration to check which ones are in use, and also inspect the other content elements that are in the configuration. Since that's about 75% of the work to fix the issue, once it was done I might as well go do some copy & paste on the templates and test them.
If you have time to go have a look at the templates and see if there's anything I've missed, that would be great, another pair of eyes is always helpful! There are two templates currently in use: the first one is used for one-time requests, and the other one is used for our subscribers (those who get suggestions periodically). The resulting posts from SuggestBot can be seen here in my sandbox, they're in the same order. Looks like the vertical alignment in the big table is slightly off, that might be due to how images and text is aligned in CSS, if I remember correctly, but as it looks like it's a really small difference I haven't yet allocated time to look into it (if I'll bother). Anyways... thanks again for helping me get these upgraded! Cheers, Nettrom (talk) 22:10, 1 December 2014 (UTC)

A barnstar for you![edit]

Administrator Barnstar Hires.png The Admin's Barnstar
I'v done and people got credit but now ? Zaed.zaman (talk) 04:59, 6 December 2014 (UTC)

Stub predictions[edit]

Hi Nettrom,

I wanted to give you and User:EpochFail an update on m:Research:Screening WikiProject Medicine articles for quality/Stub prediction table. I'm "done", in the sense that I've checked every page on the list at least once. Overall, I think it was useful. I personally reassessed the class on about 450 articles, and in other cases I kept the class rating but made other adjustments, like adding task forces or removing the WPMED banner altogether. I really appreciated the list of pages that were redirects.

Here are a few complications:

  • Does your algorithm use raw file size (vs readable content)? It seems to give high ratings to articles that have a dozen footnotes, but very few sentences.
  • Imagine that I checked an article yesterday, and decided that it's still a stub. User:Dank expanded it this morning. The bot runs tonight. When I look at the list tomorrow, there is no way for me to know that it needs to be re-checked. What I see, basically, is that I have visited that URL recently.
  • It's fun when you start. It gets boring after a while. Using a rater script is probably essential to getting the job done for mid-size or larger projects.
  • Updates don't work as well as the original list. With the updated list, you run into the "I already checked that one" problem, but you also get articles that are unstable. On the most recent list, several of the highest rated articles were being expanded now, appeared spammy or smelled of copyvios, didn't belong to WPMED, or were at AFD. These are articles that are more likely than average to get stubbified, which would render my "new" rating out of date.

I think it might be useful to run through a list like this maybe a couple of times a year, but I don't think that it's a good candidate for an everyday maintenance task. If you were setting up a bot to do this, then you might consider having it run once a month. Every day would be too often to be useful.

And, finally, I could see some value in two other lists:

  1. An off-by-two list, to find things rated as Start (for example) but that are at least two classes higher (or lower, I suppose).
  2. A list of pages that are probably over-rated (for example, listed as B, but having a greater than 50% chance of being C class or lower).

Thank you both again for doing this. Between all of us, we have probably corrected the ratings on 2% of WPMED's articles this way. I really appreciate it.

Are you interested in doing this for other groups? I don't want to go out and volunteer your time, but a few of the more organized projects might benefit from a list. WhatamIdoing (talk) 00:45, 11 December 2014 (UTC)

Hi WhatamIdoing, thank you so much for this great feedback! Sorry about the delay in getting back to you. I'm of course happy to hear this has been a useful experiment, but also that there's improvements left to do. Let me respond to a question you had and then I'll move on to future plans, I have some ideas that can hopefully result in some good support tools for WikiProjects.
You asked whether the algorithm uses raw file size. It's a random forest classifier, which we give eleven inputs to (here's a perhaps rather cryptic list of them). We do use the raw length of an article’s wikitext as one of the inputs, it might be better to use an approximation of readable content, I’ll make a note to test that. If my memory of the classifier's scoring of importance is correct, the raw length of the article's wikitext is the #1 feature when it comes to predicting quality, and the number of footnoted references is #2. We did some investigation into the classifier's performance earlier this year for a research paper that'll be presented at a conference in 2015, and one thing I for instance noticed is that it'll flag some articles as C-class that some Wikipedians will rate Start-class. At that time the Start-class criteria contained the phrase "most notably, lacks adequate reliable sources", which I see you changed on Nov 17 (ref this revision). The classifier is a bit of a black box and there will definitely be edge cases it gets wrong, short articles with a good number of references is likely one of them. Hopefully the Revision scoring as a service IEG results in a gold standard labelled set of revisions that we can use to further improve the classifier.
I hope that answers the question. As for future plans and interest in doing this for other groups, the answer is “yes, definitely!” I am currently working on designing a research experiment to finish up my PhD and based on the previous work we have done WikiProjects is probably the right place to build some support tools. A tool that will help projects keep track of candidates for reassessment is one part of that, but I’m also very much interested in understanding more about how projects (or groups of project members) make decisions about which articles to work on, and if there’s an opportunity to build tools to support that. We’ve been experimenting with information about article viewership and quality in SuggestBot’s suggestions since 2011, and my latest research found some interesting results with regards to how article viewership and quality is related to WikiProjects (unfortunately this work isn’t published yet, but I'd be happy to discuss it off-wiki).
Would that be something that WikiProject Medicine might be interested in as well? And are there other projects that you’re in touch with that I could also reach out to? Given its size, I’m quite curious about what MILHIST participants are thinking about this, but of course, only if they’re interested. And I'd be happy to hear about other potential projects I could get in touch with, if you have ideas! I’m thinking I would first reach out with a small set of questions to learn more about the decision process and whether article viewership is something they’re concerned with (or would like to be informed about).
Let me know if this sounds interesting, and please do ask questions or let me know if I’m off on the wrong path. Regards, Nettrom (talk) 22:00, 16 December 2014 (UTC)
Half an answer, so that I can get you some reply this week:
I suspect that readable content will be a better measurement, especially for stubs. The difference may be greater for pharma stubs than anything else, because there are many stubs that have a large number of refs (sometimes more than a dozen) behind just a couple of sentences. The difference may be less obvious in other subjects.
I changed the Start-class description because of a complaint: Article ratings are being sent to new editors via WP:Echo, and "most notably, lacks adequate citations" was being read as "if you have citations, this class should not have been assigned to your article". In broad theory, you must have one (1) citation (need not be inline) per article to leave Start-class. In theory, you need (almost always) one (1) inline citation per ==section== to leave C-class. In practice, some reviewers are much stricter.
More later, WhatamIdoing (talk) 03:47, 21 December 2014 (UTC)