Wikipedia:Bots/Noticeboard

Bots noticeboard

Here we coordinate and discuss Wikipedia issues related to bots and other programs interacting with the MediaWiki software. Bot operators are the main users of this noticeboard, but even if you are not one, your comments will be welcome. Just make sure you are aware about our bot policy and know where to post your issue. Do not post here if you came to discuss non-urgent bot issues, bugs and suggestions for improvement. Do that at the bot operator's talk page discuss urgent/major bot issues. Do that according to instructions at WP:BOTISSUE discuss general questions about the MediaWiki software and syntax. We have the village pump's technical section for that request approval for your new bot. Here is where you should do it request new functionality for bots. Share your ideas at the dedicated page

INterwikis

I'm sure I remember reading that commented out interwikis aren't re-added. Is this correct? Rich Farmbrough, 20:20 7 September 2008 (GMT).

No of transclusions

Is there a(n easier) way to get the number of transclusions a template has?

My php-based bot could load a webpage, so that's not out of the question if a suitable web page does exist. (I did ask on the WP help desk, but I think the people watching this page will be better/quicker at making suggestions.) If worst comes to worst, it could include links to the template as well; redirects should be included (if this is possible). At the moment, you see, I'm having to resort to loading a list of transclusions, and then counting them, which is pretty resource intensive. Cheers! - Jarry1250 ^{(t, c)} 17:13, 31 January 2009 (UTC)

AWB can do that, if I get what you're saying. §hep • ^Talk 04:12, 5 February 2009 (UTC)

Counting the number of elements in a list isn't *that* resource intensive. How are you loading the list? Q ^{T C} 04:18, 5 February 2009 (UTC)

For posterity's sake, I got a suitable function from Cobi/Chris' PHP bot framework. - Jarry1250 ^{(t, c)} 09:50, 18 February 2009 (UTC)

Unauthorized bot?

User:LilHelpa claims to not use a bot for reverting, but this mistake seems so blatant that it appears likely to have been made in an automted manner. I think this should be investigated. TubeToner (talk) 23:23, 7 February 2009 (UTC)

Doesn't look like a bot to me, maybe AWB or something else. Give him a barnstar or something. BJ^Talk 23:38, 7 February 2009 (UTC)

I note that LilHelpa isn't listed at Wikipedia:AutoWikiBrowser/CheckPage and has no user javascript, so if it's anything it's "something else". My guess is that it's just someone with a lot of time on their hands. I suggest a polite note on their talk page pointing out the mistake. Anomie⚔ 23:51, 7 February 2009 (UTC)

JS can be run locally, too. Just because it isn't on site, doesn't mean it isn't running. — neuro^(talk) 00:05, 11 February 2009 (UTC)

WP:BLP/N

There's a big backlog at WP:BLP/N, and the bot is archiving four-day-old threads that haven't been resolved. THF (talk) 13:55, 14 February 2009 (UTC)

You need to tweak the MiszaBot template at the top of the page. I changed it from "4d" (four-day) to "7d" (seven-day) as a starting point, but you may like to increase it further. - Jarry1250 ^{(t, c)} 15:01, 14 February 2009 (UTC)

PHP Bot Frameworks

May I tentatively suggest that some consultation is required about the variety of PHP frameworks currently available. I know of many that exist; they range from the bad (e.g. my own) to the very, very good and yet we do not have even a list of frameworks that is even half-complete! This contrasts with other languages, such as perl, which have a handful of well documented, well supported solutions.

Whilst I admit that it is naive to assume that all the existing PHP bots could be made to work from the same framework, I think a possible starting point for helping newcomers develop bots in PHP might be to draw up a comprehensive list of solutions that worked with the present API. Then they could at least choose the one that best suited their needs.

The ones I know of:

1. Chris G has one
2. Cobi has one (has also contributed to Chris G's)
3. I have Wikibot, built from Kaspo's phpwikibot framework, though it's very basic.
4. SQL operates SxWiki
5. Foxy Loxy is developing PHPediaWiki
6. The old BasicBot framework (I don't think it still works though)
7. GeorgeMoney's (et al) Bot Framework
8. Edward Z. Yang's (last update February 2007)

I know the above list is pretty shoddy, so please correct the glaring mistakes with it. - Jarry1250 ^{(t, c)} 16:15, 14 February 2009 (UTC)

Chris G's is wikibot.classes (mine) with a few modifications (admin functions, mainly). Also, as far as I know, SxWiki is not really being developed much anymore. -- Cobi^(t|c|b) 18:05, 14 February 2009 (UTC)

I have created a rough table to try to clarify the matter, but there are still large gaps. Could knowledgeable people expand it / correct the falsehoods. - Jarry1250 ^{(t, c)} 11:49, 16 February 2009 (UTC)

Proposal: Redesign and cleanup of WP:BRFA

Note: This should really go on WT:BRFA, but seeing as no-one reads that page, I thought I'd get a more meaningful response here!

The instructions and documentation that currently exist at Requests for bot approval are very out-of-date, and in some cases, confusing. The 'Organisation' section was virtually identical to its current state way back in 2006. Practices have moved on by light years from then. I propose that something like this is implemented to make the process more accessible and the approvals page less cluttered. I have also added brief guidance for BAG members on how to close BRFAs (based on SQL's How to close a BRFA). All the essential information is still there, just in a different, more user-friendly format.

Comments, suggestions, ideas?

P.S. I know that the colour scheme isn't particularly brilliant, I chose colours that wouldn't clash with/obscure the links. If anyone with more... aesthetic abilities than me has a better colour scheme: by all means implement it! Richard⁰⁶¹² 17:51, 14 February 2009 (UTC)

It definitely needed tidying up; I did like the 1-2-3 nature of the instruction system though. Could that be preserved somehow? - Jarry1250 ^{(t, c)} 18:46, 14 February 2009 (UTC)

Short answer: yes, like this. I just thought that the big table was rather clunky. However, it looks a lot cleaner inside the collapsible table, and I suppose it does help to divide up the process and make things easier to follow. Richard⁰⁶¹² 19:46, 14 February 2009 (UTC)

Looks good to me. I don't particularly like the version with the numbers due to the table borders; IMO this modification doesn't look too bad. Anomie⚔ 22:11, 14 February 2009 (UTC)

That's certainly an improvement over my tables (much cleaner design). In keeping with other pages that use the I/II/III system (AfD, TfD, etc), how about this this progression of colours from light-dark? Richard⁰⁶¹² 22:35, 14 February 2009 (UTC)

Looks good to me. Anomie⚔ 01:13, 15 February 2009 (UTC)

I'd say that's an improvement. Nice work. §hep^Talk 01:53, 15 February 2009 (UTC)

Given that no-one has said that the redesign is a bad idea, I'll be bold and implement it. Feel free to revert me if I acted with too much haste. Richard⁰⁶¹² 21:43, 15 February 2009 (UTC)

Actually done, it does look very good! Nice job Richard. IMHO the TOC should go left of the table, but I'm not really bothered so I'll leave it. - Jarry1250 ^{(t, c)} 09:24, 16 February 2009 (UTC)

I boldly moved the TOC back to above the table, having it to the side makes the screen scroll horizontally and makes most TOC lines wrap. Anomie⚔ 12:19, 16 February 2009 (UTC)

Now I have whitespace taking up half of my screen... If it helps IE7/Vista, 1440x900, 22" §hep^Talk 12:26, 16 February 2009 (UTC)

I see what you mean about the whitespace (1440x900, FF3), but having the TOC above the stats table does stop either of them from gettting 'squashed'. Richard⁰⁶¹² 12:31, 16 February 2009 (UTC)

800x600 browser window here (on a 1024x768 screen, I like to be able to see things besides my browser window so I seldom maximize). If you can figure out a way to rearrange things that doesn't force horizontal scrolling, feel free. Anomie⚔ 22:34, 16 February 2009 (UTC)

Wikipedia:Bot Approvals Group/nominations/Jarry1250

I have accepted a nomination to join the Bot Approvals Group - relevant discussion is just a click on the link above away. - Jarry1250 ^{(t, c)} 20:56, 16 February 2009 (UTC)

SmackBot must fail-safe

SmackBot (BRFA · contribs · actions log · block log · flag log · user rights) is an ambitious effort to make detailed corrections to tags and text of articles. It performs useful tasks well. However, in its current state, SmackBot fails the do-no-harm test. It needs testing on degenerate cases, and a more complete understanding of the grammar rules of the tags it edits.

The critical failure with SmackBot is that it doesn't fail-safe when it encounters an unexpected condition. Before SmackBot is reactivated, it must demonstrate that it does no harm in degenerate test cases. In the software development sense, SmackBot needs Verification and Validation quality control.

I've communicated my concern to SmackBot's developer/owner. He's working on a problem with damage to tags. The larger problem is fail-safe and do no harm.

--Mtd2006 (talk) 06:03, 18 February 2009 (UTC)

Moved from Adins Noticeboard per author. Rich User talk:Rich Farmbrough, 09:07 18 February 2009 (UTC).

It would greatly help if you gave links to the issues you see and (if it's on-wiki) the discussion you've had with Rich Farmbrough. Otherwise, this is just a vague rant with nothing to comment on. Anomie⚔ 12:37, 18 February 2009 (UTC)

No problem. The gory details are at SmackBot's talk history page. Some of them have been moved to User talk:Rich Farmbrough. That's where he answers questions and reports his fixes. We've been discussing the problem at Thank you for the message. He explained how SmackBot works. I've also e-mailed Rich.

The problem that triggered this "rant" of mine is SmackBot damages tags. Some similar problems where SmackBot has tripped up are:

• SmackBot problem (fixed)
• SmackBot- Cybernetic revolt (fixed)
• SmackBot moved tags from within the article to the very top (fixed)
• SmackBot bug report (open, I think, but disabled)

SmackBot does well with problems that fit its view of reality (aside from some annoying habits, #1 and #2). It fails on degenerate cases. It's hard to catch SmackBot in the act. An alert editor can track a problem; others see errors, but mistake the damage for vandalism. We can't know how many pages are damaged before a problem is accurately reported.

SmackBot is leveraging the capabilities of AWB. I'd like to give Rich time to investigate the problem before I say more. He's been very responsive to fixing bugs. Mtd2006 (talk) 16:09, 18 February 2009 (UTC)

Rich seems to be quite responsive to bug reports, and frankly you overreacted to the South Korea edit. I don't see anything that needs wider attention at the moment. Best would be to continue with the discussion at User talk:Rich Farmbrough. Anomie⚔ 20:14, 18 February 2009 (UTC)

• For what it's worth, I don't think SmackBot should bother making changes such as this. It's a wasteful edit. –xeno (talk) 16:24, 18 February 2009 (UTC)

You are quite correct, and genreally it won't make white space only edits. Rich Farmbrough, 19:51 18 February 2009 (UTC).

SmackBot overview

This overview explains how SmackBot works. The bot leverages AWB to edit articles. It combines WP templates with exception rules in a automated process to generate its ruleset, a list of edit rules for AWB. Rich has said, the ruleset is generated from "over a thousand templates, each of which can be formatted in hundreds of ways, it is necessary to canonicalise templates to make the problem tractable." This as another way to say that SmackBot is complex and it makes arbitrary assumptions to simplify its ruleset. Arbitrary choices in software design is the opposite of a deterministic algorithm. Arbitrary decision making implies unpredictable failure modes. SmackBot requires Verification and Validation too prove it does no harm, see WP:BOTPOL#Bot_requirements.

Because of SmackBot's complexity and the potential for damage to live articles caused by unpredictable failure modes, I propose some guidelines for its use.

Test and debug for SmackBot

A test standard should apply to all bots.

• No test runs on live articles. Testing should happen on local machine or in a WP:SANDBOX (same rule for human editors),
• Changes should be tested against a test suite that exercises all of SmackBot's edits (software design and testing) with known and degenerate test cases (that is, situations that SmackBot expects and those that are outside its assumptions about a normal article),
• Changes to SmackBot's ruleset (templates or bug fixes) must be regression tested before it's released on live articles.

This warning is found on complex templates. Templates are, in effect, complex software. {{intricate template}} As the template warning says, a flaw in the template can appear on a large number of articles. The nice thing about templates is that if a flaw is repaired, the errors are automatically reverted. This is not the case with SmackBot. Flaws in SmackBot are permanently applied until reverted. Because SmackBot's ruleset is generated from many templates, it's vulnerable to errors and conflicts in them. These problems are "fixed" in SmackBot with exception rules. However, when a problem is fixed in SmackBot, its incorrect edits are not automatically reverted. They remain until a human editor or an automated process repairs them.

Design pitfalls for SmackBot

While there are applications for non-deterministic algorithms, a bot should be strictly deterministic. That is, a bot must do what it's designed to do without side effects. The tasks that a bot performs are its design specification. SmackBot's design specification is User:SmackBot#Tasks. One of SmackBot's design specs (or tasks) is to add missing dates or repair incorrect dates in various tags, e.g., {{fact}} to {{fact|date=February 2009}}. However, when SmackBot fixes a date in an article, it also makes trivial changes. AWB users are cautioned not to make trivial edits.

SmackBot makes trivial edits because it does not conform to its own design specification. There is no SmackBot task that says "change the case of every tag to upper case". Human editors are not allowed to hide substantive changes in a long set of trivial edits (changing case, removing blanks or new lines, etc.) Humans are cautioned for not using a sandbox for experiments and tests. A bot must follow the same rules because it may perform these disruptive edits to thousands of articles. A bot can waste resources, be disruptive and cause damage much faster than the fastest human editor.

SmackBot problems:

• Failure to do no harm (SmackBot damages tags)
• Side effects of a non-deterministic algorithm

• Inverting upper/lower case of tags (dead link vs Dead link)
• Converting new-line to blank (SmackBot and templates)
• Page blanking (SmackBot- Cybernetic revolt)
• Unpredictable side effects

Summary

SmackBot's bugs can be fixed, up to a point. SmackBot's owner is responsive to trouble reports. But until a bug is reported, SmackBot has made changes that are difficult to find, because of trivial edits, and hard to revert, again because it makes too many trivial edits. Beyond this, the most critical flaw of SmackBot is its failure to be harmless.

Mtd2006 (talk) 03:34, 19 February 2009 (UTC)

Bot categories: Grand plan

In my opinion, Category: Wikipedia bots could do with some substantial re-organisation. Rather than doing it all bit-by-bit, I think it might be best to start with an adventurous design, which then gets modified until consensus is reached. To reiterate, feasibility was not considered when drawing up this design. At the moment, we have this:

The category system, as it is. "An effing mess."

Wikipedia bots (all bots are members) Wikipedia bot requests for approval (members are "open" requests) Wikipedia approved bot requests (also a direct subcategory of Wikipedia bots) Wikipedia withdrawn bot requests (also a direct subcategory of Wikipedia bots) Wikipedia failed bot requests (also a direct subcategory of Wikipedia bots) Wikipedia expired bot requests (also a direct subcategory of Wikipedia bots) Wikipedia bot owners Exclusion compliant bots Perlwikipedia bots Wikipedia pages monitored by bots Various subcategories Wikipedia bots running on the Wikimedia Toolserver Active bots on Wikipedia Wikipedia anti-vandal bots Autoassessment bots Wikipedia adminbots Bots not currently active Categories owned by bots (empty) Various subcategories (containing members) Dead Links Found by BezkingBot-Link Newsletter delivery bots Pages archived by Werdnabot Template substitution bots WikiProject tagging bots Wikipedia archive bots Wikipedia bots with source code published (3 members) Wikipedia bots with PHP source code published Wikipedia bots with Perl source code published

I would propose a system more like this:

My grand plan

Wikipedia bots (empty) Wikipedia bot requests for approval (empty or containing undetermined, perhaps) Approved Wikipedia bot requests for approval (also a direct subcategory of Wikipedia bots) Withdrawn Wikipedia bot requests for approval (also a direct subcategory of Wikipedia bots) Denied Wikipedia bot requests for approval (takes over from "Failed") (also a direct subcategory of Wikipedia bots) Expired Wikipedia bot requests for approval (also a direct subcategory of Wikipedia bots) Open Wikipedia bot requests for approval Wikipedia bots by status (empty or containing undetermined, perhaps) Active Wikipedia Bots (takes over from "Active bots on Wikipedia") Inactive Wikipedia Bots (takes over from "Bots not currently active") Unapproved Wikipedia Bots Wikipedia bots by purpose (empty or containing undetermined, perhaps) Wikipedia bots for autoassessment Wikipedia bots for anti-vandalism Wikipedia bots for template substituion Wikipedia bots for WikiProject tagging Wikipedia bots for archiving Wikipedia bots for newsletter delivery Other sub-cats as appropriate Pages monitored by Wikipedia bots (if it is possible to merge with "Categories owned by bots") Various subcategories Dead Links Found by BezkingBot-Link (or delete) Pages archived by Werdnabot These may just have to be left as-is/renamed: Wikipedia bot owners Wikipedia bots which are exclusion compliant (takes over from "Exclusion compliant bots") Wikipedia bots using Perlwikipedia (takes over from "Perlwikipedia bots") Wikipedia bots running on the Wikimedia Toolserver Wikipedia adminbots Wikipedia bots with source code published Wikipedia bots with PHP source code published Wikipedia bots with Perl source code published

I know there are bots which rely on these categories, so it would be good to get everyone's view on this. Another obstacle may lie with the {{Bot}} template, which is compulsory for all bots, because it adds all bots to "Wikipedia bots". With some tweaking, however, it too could become a useful tool in the categorisation process - simply asking for a status and a purpose would help enormously. Also, that reminds me - if this were to be implemented, we'd need to work out what to do with bots with many different tasks (when it came down to "purpose") - multiple progress categories per bot, perhaps? Anyhow, let's see how far we can get.

- Jarry1250 ^{(t, c)} 20:18, 18 February 2009 (UTC)

Sounds good to me, it is a effing mess. As for multitasking bots, just slap 'em into each relavent category seems perfectly logical to me. Q ^{T C} 23:29, 18 February 2009 (UTC)

Looks good to me (categories are long overdue an overhaul), although I would rename 'Failed Wikipedia bot requests for approval' to 'Denied Wikipedia bot requests for approval', as the template is {{BotDenied}}, and the requests are listed on the main BRFA page under 'Denied', not 'Failed'. Richard⁰⁶¹² 23:53, 18 February 2009 (UTC)

I've changed it - the more logical, the better, IMHO. - Jarry1250 ^{(t, c)} 09:24, 19 February 2009 (UTC)

Doesnt trouble my brain. You have my support ! -- Tinu Cherian - 09:42, 19 February 2009 (UTC)

I've add "Unapproved" as a sub- of "by status", for bots which have not been approved for any task. - Jarry1250 ^{(t, c)} 13:07, 19 February 2009 (UTC)

If anyone has any worries/criticisms - however minor - please shout; I'm about to contact some experts on category naming to check the exact wording of the categories and to see how we can move this along. (I'm sorry, that's just my way - I hate doing nothing when something can be done.) - Jarry1250 ^{(t, c)} 14:40, 20 February 2009 (UTC)

This standardization looks great to me. Perhaps a group CFD to get more input from those who don't watch teh BON? –xeno (talk) 14:44, 20 February 2009 (UTC)

I agree - CfD is probably a good place to root out any problems so yes, I think I'll file one later today. - Jarry1250 ^{(t, c)} 14:50, 20 February 2009 (UTC)

n.b. I've made some tweaks to the grand plan (not major, just wording issues).

CfD filed here, links back to this discussion. - Jarry1250 ^{(t, c)} 15:59, 21 February 2009 (UTC)

Revised grand plan

Wikipedia bots (empty) Wikipedia bot requests for approval (empty or containing undetermined, perhaps) Approved Wikipedia bot requests for approval (also a direct subcategory of Wikipedia bots) Withdrawn Wikipedia bot requests for approval (also a direct subcategory of Wikipedia bots) Denied Wikipedia bot requests for approval (takes over from "Failed") (also a direct subcategory of Wikipedia bots) Expired Wikipedia bot requests for approval (also a direct subcategory of Wikipedia bots) Open Wikipedia bot requests for approval Wikipedia bots by status (empty or containing undetermined, perhaps) Active Wikipedia Bots (takes over from "Active bots on Wikipedia") Inactive Wikipedia Bots (takes over from "Bots not currently active") Unapproved Wikipedia Bots Wikipedia bots by purpose (empty or containing undetermined, perhaps) Wikipedia bots for [...] Pages monitored by Wikipedia bots (if it is possible to merge with "Categories owned by bots") Various subcategories Dead Links Found by BezkingBot-Link (or delete) Pages archived by Werdnabot These may just have to be left as-is/renamed: Wikipedia bot owners Wikipedia bots which are exclusion compliant (takes over from "Exclusion compliant bots") Wikipedia bots using MediaWiki::Bot (takes over from "Perlwikipedia bots") Wikipedia bots running on the Wikimedia Toolserver Wikipedia adminbots Wikipedia bots with source code published Wikipedia bots with PHP source code published Wikipedia bots with Perl source code published

I've removed the purpose categories and renamed perlwikipedia in line with the CfD discussion. - Jarry1250 ^{(t, c)} 18:59, 21 February 2009 (UTC)

Something that does need doing is generating a list of templates that categorize into these templates so if this ever passes they can be updated. Q ^{T C} 05:08, 23 February 2009 (UTC)

Template that will need changing

• {{Bot}}
• {{Infobox Bot}}
• More

I'll check out the rest when I get the chance. - Jarry1250 ^{(t, c)} 07:50, 23 February 2009 (UTC)

Would {{AWB bot}} need to be on this list? §hep^Talk 16:41, 23 February 2009 (UTC)

Squid Changes

Just an FYI for bot owners/writers. The recent squid rollout changed the behavior when encountering Expect headers. So either have to be sure to not send them, or correctly handle now recieving a 'HTTP/1.0 417 Expectation failed' response instead of the 100 Continue. Q ^{T C} 22:22, 18 February 2009 (UTC)

Thehelpfulbot - Deleting Broken Redirects Task (Admin bot) BRFA

Hi all,

I have put a request in at Bot Requests for Approval for my bot, Thehelpfulbot to be able to use pywikipedia's Python script of redirect.py to delete broken redirects. pywikipedia has been extensively tested and the bot has already been speedily approved for using the same script, but fixing double redirects. As far as I can tell, no other bot is running this task, as User:RedirectCleanupBot is no longer in use as WJBscribe left Wikipedia. This bot will require the admin flag to run this task, which is why I am posting on this board - to let you know about the bot.

If you wish to comment, please do so at Wikipedia:Bots/Requests for approval/Thehelpfulbot 5.

Thanks,

The Helpful One 20:45, 19 February 2009 (UTC)

Thehelpfulbot - Deleting Local Images that are also on Commons Task (Admin bot) BRFA

Hi again all,

Thehelpfulbot now has another request, using pywikipedia's Python script nowcommons.py to delete local images that are also on Commons. You can have a look at the code if you wish, by seeing the pywikipedia library here.

This task will also require the admin flag to run, which is why I am posting on this board again, to let you know about the second admin bot task.

If you wish to comment, please do so at Wikipedia:Bots/Requests for approval/Thehelpfulbot 6.

Thanks,

The Helpful One 20:45, 19 February 2009 (UTC)

Note: This task has now been withdrawn. The Helpful One 22:11, 19 February 2009 (UTC)

Notice re: proposal to give BAG the ability to (de)flag bots

Just a quick notice to inform you that I have posted a proposal at the Proposals Village Pump regarding giving BAG the bot-flagging right. Comments, questions, trouts, etc welcome there! Richard⁰⁶¹² 11:11, 20 February 2009 (UTC)

Erik9bot's editing speed

Could another bot operator voice an opinion here please?--Rockfang (talk) 16:41, 23 February 2009 (UTC)

New BAG member

Just to let everyone know that my BAG nomination was successful one - so yes, I am now a member of the BAG! Thanks to everyone who showed their support, I hope to now show that your trust was correctly placed. However, in the unlikely event that I do get something wrong - however small - I hope you all will put me right ASAP ;) - Jarry1250 ^{(t, c)} 20:31, 23 February 2009 (UTC)

Good Good :) Now get to work! :P --·Add§hore· ^{Talk To Me!} 20:32, 23 February 2009 (UTC)

Intricate templates

A template on this page (the header I think) is putting this page into the Intricate Templates category. Is there a way to fix this seeing as how this page itself isn't a template? :) Rockfang (talk) 20:41, 23 February 2009 (UTC)