On the Wikimedia Techblog, Mark Bergsma explained the root of problems users encountered the morning of Monday, July 5:
Starting at 0:10 UTC on July 5th, the Wikimedia Foundation suffered from intermittent, partial power failures in the internal power network of one of its main data centers in Tampa, Florida. Due to the temporary unavailability of several critical systems and the large impact on the available systems capacity, all Wikimedia projects went down. The power situation stabilized at 1:12 UTC, and systems and services recovery has been taking place since.
Fortunately, the technicians managed to get everything back up and running shortly afterwards.
Following a blog post by Phoebe about Wikimedia's future strategies, Brion Vibber left a detailed comment giving his thoughts. As a former Chief Technical Officer (CTO) and all-round maintainer of Wikimedia projects and the software behind them for a number of years – not to mention the namesake of this report – his thoughts provided a useful insight into the past, present and future of the Wikimedia Foundation's "tech" department:
Tech of course is the area I know best, and I can assure you that there will be plenty of work for [the projected] ~75 tech folks to do in 2015; even with the major expansion over the last couple years there’s still wayyy more stuff to do than time to do it. Purely on that end, there are a few rough tasks that need to get taken care of all the time:
keeping the actual site running smoothly – we’re still understaffed on ops, and relatively low-hanging fruit remains in improving proactive & reactive system administration procedures ...
keeping the software development going smoothly – this includes making sure that volunteer devs are getting the attention they need on getting patches cleaned up and accepted, bugs fixed, relevant feature requests implemented and live
minor feature development – things like improving how we can show images, or adding a new file format, or supporting a nicer way to upload a file, whatever. Lots of little things!
major feature development & near-term research – all the awesome work that’s been coming in from the UX team is a big example now....
long-term research – do we even know what needs to be created to make things as kick-ass as they’re going to need to be five or ten years down the pipe?
... A strong “MediaWiki Labs” [as a parallel to Mozilla Labs] research program would be *extremely* awesome, and well worth the money spent, once the resources and organization are all in there to get all the basics past "we keep the site up" and "we’re fixing things we’ve known suck" into really future-focused areas too. From what I see and hear, Danese [Cooper, current CTO] is doing a smashing job at getting tech organized for its big-enough-to-need-real-management present and future, so as long as the rest of the system is working I feel pretty good about the next few years of Wikimedia tech.
The full comment is available here. Regular Signpost readers will be pleased to know that an interview with Danese Cooper – to whom Brion referred – should be available shortly.
Wikimania 2010, in Gdansk, Poland, opens on July 9. It is scheduled to include a number of talks, workshops and tutorials useful to developers. One session From WMF with love, expressly aimed at the technically minded, is scheduled for Sunday. Its programme comprises:
Note: not all fixes may have gone live to WMF sites at the time of writing; some may not be scheduled to go live for many weeks. Wikimedia sites now have their own "branch", meaning that it is no longer possible to say that they are running revision X; rather, they are running a mix of different revisions prioritised by importance.
Bug #24167 has been fixed, preventing multiple "warn" edit filters from conflicting with each other and thus preventing an edit from getting through at all.
The 'Languages' section of the sidebar, used for interwiki links, will now stay collapsed if so desired across pages (bug #24140).
The "Peachy" bot framework for PHP moves into official beta mode, with PHP bot operators required to put it through its paces.