Wikipedia talk:Wikipedia Signpost/2024-03-29/Recent research

← Back to Recent research

Discuss this story

• Looking at just the first 15 days for a new editor seems short. Many people dip their toes in the water and then return to editing more intensely months or even years later. —Ganesha811 (talk) 23:37, 29 March 2024 (UTC)

  Hi @Ganesha811, thanks for reading and commenting! I’m the first author on that paper, and you’re right that 15 days appears a bit short. In the paper we describe how we chose two weeks rather than four weeks due to the marginal gains you get from extending the window. We also investigate two longer-term retention windows that were used in the Teahouse paper (1–2 months and 2–6 months), and again find no significant impact. One methodological note about retention is that we control for first-day activity in our models. This means that we can have a “no impact” result even if the Newcomer Homepage increases first-day activity, because the method’s comparing a more active newcomer in the treatment group to a similarly active one in the control group. Understanding or increasing retention is a challenging area, it’s often a “needle in a haystack” kind of problem because the retention rates are very low. Hope this helps explain things, thanks again! Cheers, Nettrom (talk) 19:58, 30 March 2024 (UTC)

  Interesting, thanks for the reply! —Ganesha811 (talk) 20:25, 30 March 2024 (UTC)

• Just want to comment that I was interested in what was going on with The Wikipedia Adventure and that "blog post" where the whole thing was discussed at length (I believe) is linked to a parked domain now unfortunately. Don't know if the expiration of communitydata.cc came up before... Reconrabbit 04:05, 31 March 2024 (UTC)
  • The domain moved from .cc to .science and the old .cc domain expired and appears to have been lost/parked. That said, the post itself is still online (at the slightly different location). I've updated the link in the old WRN report. You can find the blog post here: https://blog.communitydata.science/adventures-in-onboarding-new-users-on-wikipedia/ —mako๛ 13:45, 31 March 2024 (UTC)

    Thank you very much! I didn't know about the migration. Reconrabbit 14:46, 31 March 2024 (UTC)

• There's an interesting conclusion to the paper on Newcomer tasks that I think the Signpost write-up missed: context matters. The strange results from the beginning of the paper (lots of nulls, weak and contradictory postiive results) make more sense if you consider the contexts in which the users are making account. In Section 4.4 they cover this in detail, but to summarize: editors make accounts because (1) they want to make an edit or (2) they want to read. The authors replicate a finding from this 2014 working paper that when an editor receives an intervention after opening the edit window they are less likely to make the edit they were planning to make as an unregistered user---the homepage and newcomer tasks distract the would-be-editor from actually making the edit they started. For the second group, who make an account to read, the authors find the opposite effect, with readers who make an account to read are more likely to make an edit (activate) if they are given the homepage and newcomer tasks.
  This is a cool finding! It suggests that the features need a more complicated roll-out system than "give to everyone" and instead take into account what the user was doing at the time they created an account. It seems like an effective first pass would be just providing the newcomer features based on whether an edit window was open or not when the create an account link was clicked, but further research might show better ways to decide what kind of welcome would be most effect for a given account creation context. — Wug·a·po·des 04:34, 31 March 2024 (UTC)

  There are surely lots of interesting remarks in the paper that had to be left out in the review (I do encourage folks to read the whole thing). But this particular conclusion was actually already summarized in the overall findings as quoted in the review: "Our intervention appears to distract newcomers who were already in the process of contributing, but seems to support those who were not, and in particular those who did not create an account with an intention to contribute."

  Either way, I agree that this is an interesting takeaway. I guess from an UX design perspective it's a good reminder that one should always be aware of the possible downsides and opportunity costs of a particular design feature (here: distracting users from their intended task).

  It's interesting that the team appears to have previous have worked from somewhat different assumptions, if I read this 2021 post correctly:

  The Newcomer homepage was the second project built as a result of the usability testing of our earlier work, where we noticed a lot of test participants expected a dashboard or some homepage to orient themselves.
  [...]
  Based on foundational research and cumulative feedback from previous experiments, we knew that many newcomers either arrive with a very challenging edit in mind (e.g., write a new article from scratch) from which failure becomes a demoralising dead-end, or else they don’t have any edits in mind at all, and never find their footing as contributors. The “Suggested edits” module offers tasks ranging from easy tasks (like copyediting or adding links), to harder tasks (like adding citations or expanding articles) that newcomers can filter to based on their specific interests (e.g., copyedit articles about “Food and drink”). This feature is intended to cater to both groups – a clear way to start editing for those who don’t know what they want to do, and for those who do have a difficult edit in mind, it provides a learning pathway to build up their skills first.

  I.e. in 2021 they assumed that there basically exist two relevant groups of newbies, those who need be encouraged to edit, and those who already want to make an edit but need to be discouraged from overly ambitious tasks and steered to easier edits first, both of which would be helped by the Newcomer Homepage. Per earlier parts of the same post, the foundational research that this had been based on was qualitative in nature, focusing on user interviews and constructing user personas. The quantitative results in the paper show that this was not the full picture, and vindicate the notion that such assumptions must be tested with the actual user base. The 2021 post also highlighted, commendably, that A very important work principle we try to keep in mind is that experiments fail. When we see something is not working, we try to make a clean stop and take learnings from that failure to make the next experiment better.

  PS: It's also interesting that the 2021 post reported quantitative conclusions that are very different from those of the later paper:

  Since its launch, and as we added more editing guidance and improvements to the homepage design over multiple iterations and variant tests (a subject for another post), we’ve seen steady increases in edits and number of editors using it. In other words, the release of this experimental feature ["Suggested edits"] saw us succeed in increasing editor activation and retention. 🙌 🙌

  HaeB (talk) 06:06, 31 March 2024 (UTC)

• If you are curious of the Homepage, you can test it at special:Homepage (which might require to activate it in your Preferences). The coordination around Growth features at English Wikipedia is at Wikipedia:Growth Team features. Trizek_(WMF) (talk) 07:58, 2 April 2024 (UTC)
• Thanks, @HaeB, for summarizing the Increasing Participation in Peer Production Communities with the Newcomer Homepage paper for the Signpost. That paper was published in 2023, but from experiments run in early 2022. In the last couple of years the Growth team has completed several other projects and research experiments, many of which have had more promising results: Add a Link, Add an Image, and Leveling Up.
  I think it's worth highlighting that the Growth team almost always makes improvements to features once we have initial experiment findings.  Sometimes we run a secondary experiment after making those improvements, but sometimes we don’t have the time and resources to run a complete follow-up experiment. For example, in this experiment we saw that Growth features were clearly helping certain user groups, but seemed to distract people who were already mid-edit when they created an account.  We then implemented a change so these users no longer receive a Welcome survey or prompt to visit their Newcomer Homepage while mid-edit (T310320).  So, as Wugapodes suggests, we now take into account what the user was doing at the time they created an account. We’ve always hoped to eventually personalize onboarding further, for example, the onboarding process for someone who signed up to “create a new article” should likely look different than for someone who signed up to “fix a typo or error in a Wikipedia article.” That work isn’t on our immediate roadmap, but we have many ideas shared in Phabricator (T353301 & T229865). We welcome further ideas about new editor onboarding and retention on the Growth team’s Talk page or join the English Wikipedia conversation at Wikipedia:Growth_Team_features. Thanks, - KStoller-WMF (talk) 17:51, 2 April 2024 (UTC)