Wikipedia:Bots/Requests for approval/DannyS712 bot 10

The following discussion is an archived debate. Please do not modify it. To request review of this BRFA, please start a new section at WT:BRFA. The result of the discussion was

Withdrawn by operator.

DannyS712 bot 10

Operator: DannyS712 (talk · contribs · SUL · edit count · logs · page moves · block log · rights log · ANI search)

Time filed: 23:47, Saturday, March 2, 2019 (UTC)

Automatic, Supervised, or Manual: supervised

Programming language(s): Javascript

Source code available: User:DannyS712 test/links.js

Function overview: Update a (to be created) database report with articles tagged as orphans with incoming links

Links to relevant discussions (where appropriate):

Edit period(s): Ad hoc

Estimated number of pages affected: 1

Exclusion compliant (Yes/No): No

Already has a bot flag (Yes/No): Yes

Function details: Scan through all of the pages in a category and update a report with which have incoming links. Currently the report is set to be User:DannyS712 test/sandbox - eg this edit. The updates would be manually triggered from my bot account, but I'm requesting to run this as a bot to access apihighlimits, given that many of the updates could involved more than 500 pages being analyzed.

Discussion

Why aren't you running SQL queries against the toolforge replica? -FASTILY 07:31, 3 March 2019 (UTC)[reply]

@Fastily: I couldn't figure out how to quickly search for links to pages in a specific category using sql, and I was struggling to figure out how to set up toolforge --DannyS712 (talk) 07:44, 3 March 2019 (UTC)[reply]

I think you should use toolforge to avoid clobbering the API. Follow the instructions at wikitech:Help:Getting Started and wikitech:Help:Toolforge/Database to get started. Alternatively, use quarry. For SQL, filter by categorylinks, join with pagelinks, and do a count on the pagelinks. -FASTILY 07:56, 3 March 2019 (UTC)[reply]

@Fastily: I looked at both of those, I spent hours on this, and I still can't get past the step of creating a repository (No Phabricator accounts found for tool maintainers.) --DannyS712 (talk) 08:00, 3 March 2019 (UTC)[reply]

What about quarry? -FASTILY 08:15, 3 March 2019 (UTC)[reply]

@Fastily: Its a lot harder to then set up the table, because my upload speed is pretty slow. I wouldn't run this every day, nor do all of the monthly orphan categories at once; more like (at most) 1 monthly category in a single day --DannyS712 (talk) 08:21, 3 March 2019 (UTC)[reply]

That makes no sense; your upload speed has no effect on whether you can use quarry. You should always use SQL queries instead of mass API calls so as to conserve server resources/bandwidth for readers/human editors. -FASTILY 08:39, 3 March 2019 (UTC)[reply]

@Fastily: no I mean how to convert the quarry to a table here on enwiki. Its not "mass API calls" its less than the 5000 bots make, usually less than 1000. I can do it at server downtimes? --DannyS712 (talk) 08:42, 3 March 2019 (UTC)[reply]

m:Research:Quarry#Downloading a resultset? -FASTILY 08:44, 3 March 2019 (UTC)[reply]

@Fastily: And that is where we get to the upload speed issue - how to download and then upload a table with hundreds of rows --DannyS712 (talk) 08:56, 3 March 2019 (UTC)[reply]

With all due respect, that's ridiculous. When you run this code on your computer, you will be downloading whatever the API spits out, processing it locally, and then uploading it! That workflow sounds identical to submitting SQL to be run by the quarry service on toolforge and then simply retrieving and uploading the results, but with the additional cost of all the API calls. -FASTILY 09:14, 3 March 2019 (UTC)[reply]

@Fastily: running the api downloads and then uploads the results? Huh, good to know. In that case, for now this is

Withdrawn by operator.. If I ever figure out how to use toolforge, and want to run it automatically, I'll re-open this or make a new brfa, but as it currently stands this isn't something I need a bot for. Thanks for explaining --DannyS712 (talk) 09:16, 3 March 2019 (UTC)[reply]

No, that's not what I meant by 'uploading'; in your code, you eventually edit User:DannyS712 test/sandbox, in which you 'upload' your report to Wikipedia. And for the record, my goal isn't to make your life difficult. It's honestly fine if you don't know SQL and/or linux, the technical-minded folks around here are more than happy to help if you ask. -FASTILY 09:29, 3 March 2019 (UTC)[reply]

@Fastily: That would explain a lot (and/or linux) - I use windows --DannyS712 (talk) 09:36, 3 March 2019 (UTC)[reply]

@Fastily: Any chance you can write a query to check incoming links to orphans? I couldn't figure it out. Thanks, --DannyS712 (talk) 04:38, 4 March 2019 (UTC)[reply]

Sure, although I can't promise I'll get around to it immediately. In the meantime, you can try yourself - some modifications to wikitech:Help:MySQL queries#Pages containing 0 page links should get you what you need. If you still find yourself struggling, then let me know and I'll toss it on my todo list -FASTILY 09:03, 4 March 2019 (UTC)[reply]

@Fastily: I did try it myself, for like 2 hours, and couldn't figure it out. Thanks, --DannyS712 (talk) 18:14, 4 March 2019 (UTC)[reply]

The above discussion is preserved as an archive of the debate. Please do not modify it. To request review of this BRFA, please start a new section at WT:BRFA.