First, please be aware that we are already testing MediaWiki. We have parser tests, unit tests, and browser-based Javascript tests. We can still improve, though. So, on WP:VPT, I asked for ideas about how the foundation could better test MediaWiki before deploying it to the projects. Below are the responses. — ☠MarkAHershberger☣ 19:10, 7 October 2011 (UTC)[reply]

First theres no doubt that testing was done and I am sure that many things were caught. I also agree that you can't reasonably be expected to test all the gadgets that are not part of the core software. Additionally these were just some of the things that I thought should have stood out. This is a very short and very watered down answer to your question but there are several things that I try to do when doing testing:

Make sure that everyone knows its coming. I typically shotgun it multiple times. They might get tired of seeing it but they definately know its coming. This might include a message on the Wikipedia landing page and/or a generic banner message includeing the blog, facebook, Village pump, etc.
Solicite some testers in a developement environment. I assume there is a dev environment but if not there should be. I also guess that there were some testers but I don't know and I am not a fan of the go live and let the users tell us what the problems are approach. Several big software companies do this a lot and its extremely frustrating and annoying.
Create a one for one comparison mapping of how things are to how the new changes will be. Basically, try and line up new and old functionality and build some test cases to check for how they line up.

Some things more specific to WP and there should be a standard grouping of things we check for each release

Make sure that test cases are built that account for all namespaces, major templates, special use things like tables and how the formatting looks.
Make sure that these things are viewed on a regular monitor and not just the giant development screen we IT types use.
Make sure to build test cases for items on the Main page that accounts for the various "Featured" types.
Send a notification to the app and script developers to tell them to stand by. I realize that WP isn't "Responsible" for these apps but it would be the neighborly thing to do to give them a heads up. I am also guessing that given the widespread use of some applications and how they function that it was known ahead of time that certain ones would break because of code changes or the implementation of new functionality. This type of information shouldn't be kept secret.
As for the helping to test comment I would be glad too and I am confident others would as well if we know its coming. But if I have to watch a couple of remote blogs and the technical page of the Village pump theres a good chance I'm gonna miss it.
In regards to a general gadgets comment. Many of the gadgets are very very widely used and should be reviewed for inclusion in the core software anyway at least as a preference the user can select if they want it so although I agree that its impossible to test every gadget, some of them should have been added to the software long ago (I.e. HotCat, Improved Diff view, Add an edit link to the lede section of a page, open external links in a new tab, etc). Since the software is used by other things I don't think that Wiked and Twinkle would be well suited for inclusion in the core software but im not sure.

I hope this helps. --Kumioko (talk) 18:42, 6 October 2011 (UTC)[reply]

I agree with pretty much all of Kumioko's comments. Sure, deal with the odd test case, but we must consider day-to-day usage, i.e. does the content on the mainpage (which changes every day) still appear and function correctly? Do the so-called gadgets (that are used by 10s perhaps 100s of 1000s of editors) still work? Have we invited the main stakeholders directly to engage in the upgrade rather than rely on them spotting a Village pump or Signpost notice which may not even give the whole truth, e.g. that Wikipedia will be broken until further notice. While I've heard moving to 1.18 is a "success" (compared to the move to 1.17), I've been around a while and never noticed so many mainstream issues arise. Something has definitely not gone quite right. The Rambling Man (talk) 21:30, 6 October 2011 (UTC)[reply]

Wikipedia:Village pump (technical)/Preparing for MediaWiki deployments

See also