MAssaf

Using web analytics to test effects of user interface changes.

• With JavaScript-based web analytics software it is easy to measure the effect of changes such as adding diacritics to page titles—it is easy to measure important metrics such as the percentage of people who bounce (give up on the page virtually immediately), easy to measure engagement (time on page/time on site) and retention (does the user come back to the same page?) With web analytics, numbers would quickly show that the majority of users react unfavorably to editors adding diacritics to article titles.
• Wikipedia is surely unique among major web sites in apparently not having web analytics that make it easy to get such metrics. No other website of Wikipedia's size would make major user interface changes without A/B and multivariate testing using analytics—Google even enables switching between old and new versions of the user interface when Gmail or Google Groups interfaces are changed. Google Analytics is surely the most widely used, but open-source web analytics software is available.
• You have probably heard the story of Coca Cola's switch to a new formula. It had been tested—in blind taste tests the majority preferred the lower-sugar new Coke. But when Coca Cola announced that the original Coke was being phased out, customers were angry—the perception was that they were losing something good, rather than gaining something better. So when I hear talk about WP's Great Leap Forward (with a new interface) I am worried about whether people will be offered a choice. Most large web sites make changes gradually—take small steps, and use analytics to test the effect—rather than taking big leaps like Coca Cola did.
• I wonder if there's an open place on Wikipedia where informed techies brainstorm about such issues and Wikipedia? I have asked around, but haven't found anybody who knows. Maybe Wikipedia techies don't communicate very well with each other, and with the Wikipedia community :-( LittleBen (talk) 04:46, 16 September 2012 (UTC)

Hey! We actually do have a proper ClickTracking infrastructure slowly being built as part of the E3 team's work, and we're using it for all of our experiments. As for communication, most of it is in IRC or mailing lists, which community members are welcome to join (and they do!). Our work is formally documented on MediaWiki, since it doesn't strictly relate to Wikipedia in many cases: http://www.mediawiki.org/wiki/E3. There are a lot of links there to our various communication streams. We love constructive feedback from community members, so please feel free to get involved! E3 definitely does not making unilateral, sweeping changes. We make smaller ones, test them carefully under proper experimental conditions, and implement their best parts based on objective data. If your fears are related to the Athena skin: that's a speculative design, not a skin which will be shoved down anyone's throat. AFAIK, we have no immediate plans to make large changes to the Wikipedia web interface. E3 is mainly concerned with improving the current editor experience and engaging new editors. Hope this helps! — Preceding unsigned comment added by MAssaf (WMF) (talk • contribs) 17:10, 17 September 2012 (UTC)

BTW, sorry for the account name switch. I had MAssaf (WMF) issued to me by the foundation, so I'm using that as my "official" account now. MAssaf (WMF) (talk) 17:16, 17 September 2012 (UTC)