User:EarwigBot/Copyvios/Tests/3

What we’re really doing here is looking at the line number table of the code object contained within the function. Since it’s anonymous, there are no line numbers, so the string is empty. Replace the 0 with _ to make it more confusing (it doesn’t matter, since the function’s not being called), and stick it in. We’ll also refactor out the 256 into an argument that gets passed to our obfuscated convert() along with the number. This requires adding an argument to the combinator:

To use the search engine, we must first break the article text up into plain text search queries, or “chunks”. This involves some help from mwparserfromhell, which is used to strip out non-text wikicode from the article, and the Python Natural Language Toolkit, which is then used to split this up into sentences, of which we select a few medium-sized ones to search for. mwparserfromhell is also used to extract the external links.

The argument to range(), span, represents the width of the search space. This can’t be too large, or we’ll end getting num as our base and 0 as our shift (because diff is zero), and since base can’t be represented as a single variable, it’ll repeat, recursing infinitely. If it’s too small, we’ll end up with something like the (1 << 0) + (1 << 0) + ... mentioned above. In practice, we want span to get smaller as the recursion depth increases. Through trial and error, I found this equation to work well:

Here, id(f) is the memory location of our method, which refers to the start of the C struct from above. offset is the number of bytes between this memory location and the start of the im_self field. We use ctypes.byref() to create a reference to the replacement object, b, which will be copied over the existing reference to a. Finally, field_size is the number of bytes we’re copying, equal to the size of the im_self field.

Of course, a Wiki Toolset would be nothing without login! Our username and password are stored (encrypted with Blowfish) in the bot’s config.json file, and we login automatically whenever we create a new Site object – unless we’re already logged in, of course, and we know that based on whether we have valid login cookies.

All of the images I tested looked decent when displayed under this method, some better than others, but all acceptable. I figured this code provided a nice touch to an otherwise drab webpage (like the one you’re viewing now, it wouldn’t have been very pretty), which is why I did it, but I couldn’t help but wonder if there was an… easier… method that still saved bandwidth and didn’t resort to ugly scaling/cropping/repeating/whatever, but I could come up with nothing. It was a fun project in a language I almost never use, though, so worth it in the end.

Some other kinds of replacements are known, but impossible. For example, replacing a class object that uses __slots__ with another class will not work if the replacement class has a different slot layout and instances of the old class exist. More generally, replacing a class with a non-class object won’t work if instances of the class exist. Furthermore, references stored in data structures managed by C extensions cannot be changed, since there’s no good way for us to track these.

Unfortunately, there’s still a bit more work to do on EarwigBot before he’s ready for his first release (0.1!). Aside from the copyvio stuff above, which is integrated directly as a function of Page, I want to finish porting over the remaining tasks from old EarwigBot that are still running via cron, improve the Wiki Toolset such that new sites can be added programmatically, and improve config such that it can be created by the bot and not only by hand. This is the main barrier stopping other people from running EarwigBot, and thus the primary concern before v0.1 is good. Of course, none of this urgent; getting copyvio detection finished is my primary concern.