Wikipedia talk:Research help

This page is for discussing the deployment of WP:Research help following the proposal following the process outlined here. Please create a new section to add new feedback.

Comment

Came here after seeing this template added to debut novel. Honestly this doesn't seem like a good idea. If the goal is to have it on every page (which the project page says is the goal), then using templates in article space is a bad idea. It should be in the left hand column under "Interaction" so it doesn't clutter up article space. There are many many things editors should be aware of, that's why there is a "About Wikipeage" page linked to the left, where you will find Using Wikipedia as a Research Tool. -- GreenC 00:02, 21 December 2015 (UTC)

Hi @Green Cardamom:, I added that template using my volunteer account Sadads (talk · contribs) in my volunteer time, because I was excited about the page, and piloting it. I shouldn't have done that: we are trying to work systematically with community consensus on the pilot -- and I shouldn't be mixing my work/volunteer time.

As for the concept: we are responding to a particular ask/concern of the libraries community: how do we get readers (not editors) who are using Wikipedia for academic or other kinds of research do so with critical awareness. Our theory, is that the page will meet readers best when its concise and it helps them do the thing they are doing in the moment: research. There are particular questions that researchers, and teachers of students doing research, ask in every outreach situation (from library conferences, to GLAM-Wiki events, to WEP recruitment). The questions are designed to meet our ideal audience, which can be split into two buckets: 1) students using Wikipedia in research and 2) people who teach research and literacy (librarians and teachers). That we reach potential new editors, is a secondary feature, and we would welcome links to more learning tools: but something we have learned, is that new users are easy to overwhelm and pages like WP:About, are just that: overwhelming. We want something a researcher can engage with in a very short window of time.

We don't plan on the template being a permanent solution -- if consensus is for the link to be in the reference section we are hoping to integrate it into {{reflist}} or <references/> as a variable turn on/off (likely, with a default to on). The template is a short term solutions: I will make sure the proposal page better reflects this.Astinson (WMF) (talk) 16:57, 28 December 2015 (UTC)

WP:About may be overwhelming to some but adding yet more links competing for space and attention at the bottom of every article is contributing not solving information overload. There is already too much meta information in the way of templates, categories etc.. in my opinion. -- GreenC 17:10, 28 December 2015 (UTC)

Thank you for the feedback! The response to this particular page has been overwhelmingly positive in being concise and meeting the researcher needs with the libraries community, the reader opt-in survey, and sharing with community members who focus on outreach. Part of the pilot is figuring out if it meets a significant enough critical need -- in that we have enough evidence that it should be included. We also have a track record: thus far almost all of our WP:TWL research tools have been well used, even the ones that are relatively new -- and have hardly been linked. I will make sure to bring you into the future conversations, once we have more of the data, so that your opinion can be included. We want to make sure that the consensus is thoroughly probed. In the meantime, feel free to remove the link from pages you feel its is inappropriate that aren't part of the MILHIST and MED pilots. Astinson (WMF) (talk) 18:08, 28 December 2015 (UTC)

Well, thanks for offering to keep me in the loop. I would accept whatever decision is made but believe it would be healthy to voice other points of view outside those immediate parties who this is benefiting. And maybe the new reflink system will be qualitatively different. -- GreenC 18:27, 28 December 2015 (UTC)

Thinking about redesigning the entry

I anticipate some pushback from the community re: the idea of [eventually] deploying on every article page via the reflist. While I understand the design/presentation isn't necessarily finalized, it does jut out at debut novel. On one hand, people might not see it if it doesn't jut out, but on the other, each person only has to encounter it once before [in most cases] never needing it again and I'm fairly certain people who care a great deal about article aesthetics are going to have problems. I like the idea of having a useful link that isn't the much more broad "About Wikipedia", but I do feel like the question currently posed is whether that's worth sacrificing prime, visible article real estate for -- and for something to be rolled on everywhere it seems like there must be ways to make it so that it doesn't feel like a sacrifice has to be made. In the next steps, it may be useful to explore (a) alternative placement/presentation, such as in the sidebar or further down the bottom, and more importantly (b) user-specific presentation, for example having the message visible for unregistered users or a user's first hundred edits and/or using an enabled-by-default gadget that people can opt out of/hide. — Rhododendrites ^talk \\ 12:52, 3 February 2016 (UTC)

@Rhododendrites: We realize that resistance based on design is likely the case (and its part of our project plan), but I would disagree with you on the one-time use page: a big part of making the page was adding a bunch of useful tools that aren't otherwise visible for readers, such as easy Wikipedia tutorials and pages like WP:Find your library, which we have been creating to fill gaps in reader-friendly research support. We also anticipate the page to be heavily used by educators, who are default repeat users.

As for other designs, we proposed a few of the potential other positions in the survey (and responses so far are about a third split between "this should be at the top of articles", "this should be in the left hand side of the page", and "this should be in the reference section" ). One option that we didn't include in the survey and plan to investigate more thorughly: integrate into the popups in Reference Tooltips (which I didn't know were turned on for all readers by default, and would be at the point of user research-- and explaining the importance of the footnotes -- which is the main assumption we are battling here). Sadads (talk) 14:49, 3 February 2016 (UTC)

@Sadads: Thanks for the reply. I admit that I'm ignorant as to the background of this project. I'll take some time to look more closely a bit later. For now, I just want to ask to what extent you've solicited survey responses from different populations? You've talked about librarians, educators, and they are important, but what about other Wikipedians? It's probably not the right time now, but it might be good to advertise this on e.g. the village pump and elsewhere (and this is operating under a rather large assumption that you've not already done so :) ). — Rhododendrites ^talk \\ 15:31, 3 February 2016 (UTC)

Just a note while I'm looking at the survey. The relevant question asks "Where on a Wikipedia page, would Wikipedia readers most benefit from a link to the "Research Help" page?". It asks this whether or not the answer to the previous question was yes or no ("Do you think a small link to the "Research Help" page on Wikipedia articles would help Wikipedia readers?"). It also doesn't ask where it should go or where it shouldn't go -- just where readers would most benefit. So even if I didn't think it should be in the article, and even if I think it certainly shouldn't go at the top of the article, as the top of the article is the most visible, that would also be where it would be of most benefit to readers. In other words, I think there's a difference between asking people where readers would benefit most from a link (which will, logically, always be the most visible/obvious place) and asking people where, if they think it would be helpful, a link should be displayed. — Rhododendrites ^talk \\ 15:40, 3 February 2016 (UTC)

(edit conflict) @Rhododendrites: We had open conversations at both MILHIST and MED for the test, and notified VPT that we had discussed using them for pilots per the proposal page. We are piloting with the BAG approved limit of 10,000 articles so that we have enough evidence to have an evidence-based RFC, rather than one based on purely personal and/or subjective beliefs about design. The survey is open (and linked), on WP:Research help page itself and is catching mostly 0, or <1000 edit folks, with research careers (students, professors, and professional researchers).

We also have colloquial evidence that suggests very few people scroll down to the bottom of the page, much less read the references -- so even though the link is catchy, its only visible to a small part of our readership anyway (and pageviews from template links, suggest that at least a minimum number of people find the call to learn compelling ~5 click throughs a day for ~50 low-view articles in med and ~50 low-view articles in mil ). We are getting much higher click throughs on the GEN link, which includes a much smaller handful of articles that volunteers were enthusiastic about.

Also, in response to your critique of the survey I added a question about "too obtrusive", which we can see how it goes.

We are also working with a couple library-community researchers to see if this meets a very basic information-literacy level knowledge that they would find effective in their context, Astinson (WMF) (talk) 16:18, 3 February 2016 (UTC)

The content on the linked page seems reasonable and it says the same things I tell people when discussing the academic value of wikipedia, however the location doesn't work well as its intended audience may not get that far - an additional tab at the top may be better, or an automatic screen if anyone attempts to copy a block of text. It is also disruptive mid-article - such links should be at the top, bottom or side of the page.NiD.29 (talk) 16:10, 4 February 2016 (UTC)

@NiD.29: I recently discovered that Reference Tooltips has been enabled for enwiki readers. Would integration into that be more useful? Having a clear link? Astinson (WMF) (talk) 16:51, 4 February 2016 (UTC)

Not really sure - it should appear for people who are not logged in or are not heavy editors, but at the same time it does not belong within the article space as it isn't directly about any specific article, and does not assist editing in any way. You might condense the page down a bit as well as it needs to be as short as possible to get the message across - many people will just skip it and move on if it is too wordy.NiD.29 (talk) 04:17, 5 February 2016 (UTC)

@NiD.29: We tried breaking the page into smaller chunks of reader using the prompt questions early in the page, to help figure out what the readers need help and focus their attention space. We have several academics (and likely an library school intern), who will be doing more user-studies, to make sure this strategy is actually working. That being said: we wanted to catch the main concerns that both librarians and teachers find when discussing research with us in outreach situations (talk to any contributor who has done Education or GLAM-Wiki outreach, and they will have spent 30-60 minutes at EVERY event discussing these issues -- because our community fails to share this core assumed knowledge).

As for not assisting editing in any way: I am not completely understanding that assumption. Knowing how Wikipedia's footnotes and research process works is 100% fundamental to new editors realizing that they can contribute -- and research rather than copy editing is really the kind of work I would wish people would start on as a volunteer (FYI, I am User:Sadads in volunteer time). WP:TWL has distributed over 5000 accounts to 2600+ editors because understanding research is a core part of editing. Moreover, when we talk to librarians, one of their biggest concerns is that they aren't told where their research expertise can help; in the programs community more generally, program leaders spend most of their time "sharing awareness of research needs" + "call to action", and their is a pretty clear consensus that exposure to both is the biggest barriers to a lot of different expert communities -- they see the edit button but don't know that the policies can enable their expertise in research to be shared. Its part of the reason #1lib1ref worked so well. Can you help me understand your concern better? Astinson (WMF) (talk) 18:05, 9 February 2016 (UTC)

It doesn't help because if they are looking at the refs, they have probably figured that out. Also, the button is too large and is extremely intrusive - although perhaps it could be on the right opposite where the edit buttons are, not in line with the headings.

Perhaps instead, before someone edits a page from an IP or new user account, they should be run through a quick course on Wikipedia and how it works - including references (with periodic repeats if editing frequency is low or there are gaps). We get a lot of people who come onto wikipedia with no understanding of how it works, they make one or two bad edits and get driven off - edits they probably wouldn't have made if they had even a basic intro. I have noticed a pattern especially with the right wing/religious crowd who view wikipedia as a left wing conspiracy partly because they can't understand why their edits get rejected - if they are forced to read a short intro and take a quiz before continuing, I suspect more might understand wikipedias positions better, and it might even sway some of them back to reality.NiD.29 (talk) 22:16, 9 February 2016 (UTC)

Spam or Help Links

Not sure who though of spamming article pages with this but it really needs to be thought about first, clearly obtrusive surely a help function for the menu bar. MilborneOne (talk) 19:59, 3 February 2016 (UTC)

Hi @MilborneOne:, thank you for the feedback! Those pages, the ones around aviation, are ones that overlap with WP:MILHIST, as part of our pilot of the tool (see the project plan). I don't understand how something can be both helpful and spam, especially when it focuses on the community's own policies (WP:SPAM talks about "advertisements masquerading as content"). Can you please explain why its a "clearly" a function for the sidebar? We are collecting user and editor feedback and want to understand how we might integrate this kind of basic literacy for readers. Astinson (WMF) (talk) 15:30, 4 February 2016 (UTC)

I can see that an effort is being made to help the reader but a distracting notice breaking the flow of the article is not the way to go, I can see it being a link on the side bar. I used the term SPAM because adding a pet project in an obvious way to every article are "advertisements masquerading as content" as I see it. Perhaps I am trying to say it is in the wrong place. MilborneOne (talk) 15:37, 4 February 2016 (UTC)

@MilborneOne: Can you look at this section of the proposal and help us figure out where in the UX/UI that you would rather see a link to the page. In part, we are testing in the reference sections, because that is where a certain kind of reader (someone already doing research), will see the page. However, we also want to test it in a couple other places (like in the ).

Really, our goal is to get the best information in front of the best audience, researchers, in the place where they are most likely too see it, article space: the greatest critiques of Wikipedia from teachers and librarians (who influence many of our future and current readers) are ones around understanding how our content is created (basic literacy). Consultations with outreach leaders in the GLAM and Education programs, and both of the WikiProjects we are testing with were overwhelmingly positive (see the Med and Mil conversations)

We also ideated a couple more ways to represent the link on this subpage, but we opted for the slightly bigger message, because we wanted to ensure that it got in-front of readers. Astinson (WMF) (talk) 16:01, 4 February 2016 (UTC)

I would endorse MilborneOne's judgement that this is basically spam. Limited use for a trial is sort of acceptable if it generates useful data towards doing things a better way. The way I see it, this is a link that a researcher will only follow once or twice until they have absorbed all they need from the Project info. Nearly everybody who checks out references, for whatever reason, will not need this link 99.999 percent of the time. Forcing its eye candy past our eyes all the times we don't need it is just spamming us. For the same reason Template:reflist - or anything else that appears in articles - is the wrong place to put this link. I'd suggest you add a simple link on the Interaction menu, where we will only take our eyes, as we already do, when we do actually want something. — Cheers, Steelpillow (Talk) 17:11, 4 February 2016 (UTC)

@Steelpillow: The issue with the left hand column or the top of the page, is that we know that readers rarely see these. When I do outreach in my volunteer time, people are always amazed when I click on the "Recent changes" button in the left hand column and the edit and history buttons at the top of the page. I always get 1/2 or more of the audience saying "Oh! Thats amazing, I didn't know those existed." Part of our theory in this, is that we need to meet more of our readers where they are performing a behaviour (in this case research): in the article itself. The section edit buttons are a good example of something that radically changed IP and other new user editing behaviours (In my volunteer time, I have seen much less mess created with Infoboxes by inexpereinced users, for example). We may be entirely wrong though, and will probably have to test a bunch of different positions in the UI/UX, Astinson (WMF) (talk) 19:11, 4 February 2016 (UTC)

Well, look at it this way. The logic of your argument is that the top and side menu links need better focus too. They should all be pushed under the nose of the reader - say embedded in every section heading along with the [edit] link. Let's do that. then. Oh, wait.... No, the truth is that Wikipedia is not a click-through app, it is a complex and sophisticated environment for not only editing an encyclopedia but also for managing its community. Anything as complex as this needs a complex UI (user interface) and most people are going to have trouble. Any tool with three or four menus scattered about is going to take a lot of getting into. Have you ever tried to use professional graphics software or to pilot an aeroplane without having the common sense to hunt down and read the manual first? In both cases I have known surprisingly experienced learners exclaim over a widget in plain sight, "Oh, I never realised that was there!" In other words, this Project's problem is endemic to much of Wikipedia and needs to be attacked at that level. For example, below the Wikipedia logo one could head the side menu Main menu. That alone would make it a hundred times more noticeable but barely more intrusive. — Cheers, Steelpillow (Talk) 10:59, 5 February 2016 (UTC)

Agreed - this is a band-aid to a haphazard systembut what is needed is a comprehensive clean sheet approach to create a logical well thought out interface that is easily understood, and especially by newbies - but that will take one person who understands it well enough, who sees the need and has a lot of time on their hands, followed by a lot of arguments over details.NiD.29 (talk) 18:13, 5 February 2016 (UTC)

Accelerated idioventricular rhythm

Why was this added to the article Accelerated idioventricular rhythm? This is pointless clutter in my opinion. I'd bet there is much pent up demand for people to add links to their various pet projects much like the editors who like to litter banners at the top of pages. I don't want the reference section to become another part of the page where people feel entitled to add offtopic links. Pgcudahy (talk) 16:31, 4 February 2016 (UTC)

@Pgcudahy: I am not fully understanding why you find this an "offtopic" link? When readers are reading pages created using Wikipedia's policies and process, shouldn't they know how that information is created? Similarly, based on your editing background, you probably have a minimum amount of training in Medicine: don't you need a minimal education about how medicine experiments for and creates peer reviewed knowledge, so as to understand the scholarship there? Without that knowledge, wouldn't you be at a severe disadvantage in interpreting that information?

WP:Research help, as NiD.29 points out above, explains something that all of our readers need to understand to intelligibly engage our content (and our hope, is provide the foundation for more of our readers to understand our project sufficiently to realize that they can and should become editors). Please help me understand what about the link is off topic? Does the page need to be more tailored to doing medical research to help make it more valuable to you?

Thank you so much for your feedback, Astinson (WMF) (talk) 18:55, 4 February 2016 (UTC)

People all over this page have been saying the same thing as me so I think you understand it, you just think that we're wrong. Educating users is a fine goal but here it bumps up with the goal of keeping pages uncluttered and focused on the actual topic involved. I think clarity is more important since there are many people out there who would like to put banners and links all over pages. That means there should be very strict criteria for doing so and this just doesn't cut it.

Your reasoning for this being on-topic is also suspect. Sure this information is derived from research so information about the peer review process could be helpful. But by that reasoning why not have a link to how research is funded, or to how physicians are trained to read EKGs, or to how medical devices are certified, or every time that an EKG has been seen in an episode of family guy. If people want to learn about how research is conducted let google help them find it. This is unnecessary. Pgcudahy (talk) 19:13, 4 February 2016 (UTC)

An idea - perhaps set it so the tag only appears if the page is viewed when not signed in, or signed in from an ip or new account? Those are the users who would most benefit from it - while long time editors would not need to see it after the first time - just as we don't see the banner asking for donations. I still think that the reference section is not the place for it - perhaps a shorter name for the link and have it at the start of the page? Alternatively on the left side bar add a section on getting the most from wikipedia? I think most of the resistance is coming from the fact that it was put in the middle of the article space where it should never have been placed - put it somewhere else and resistance will disappear.NiD.29 (talk) 04:12, 5 February 2016 (UTC)

The statement "something that all of our readers need to understand" is not accurate. It would be nice if they understood it for their own benefit, but there are 5000 things on Wikipedia like that. We don't spam main article space with general things our readers need would benefit to understand about Wikipedia in general. -- GreenC 17:59, 9 February 2016 (UTC)

@Green Cardamom: "Need" is a little strong, you are correct. Consider myself redacted -- its an "ought to have" as part of larger literacy. Astinson (WMF) (talk) 18:14, 9 February 2016 (UTC)

Spam! Spam!

Not sure why this link is being spammed in the ref sections of articles all over and in some-cases 2 times on 1 page..project essay spam at its worst. We call this a run-around link here... people think it leads to sources about the topic at hand because its in the ref section...but all they get is an essay on research. Spamming a link to an essay that all could look for if they like is a bit much. Firstly i dont believe a link to a project essay in the content of a main space article is a good idea ever (we have recommendations as to this point WP:LINKSTYLE). This info in this link can already be found under "About Wikipedia" ...at best they should be a proposal to have it in the left side bar. Having a link of this nature that the average reader will not use in the ref section is odd. Many other reasons I dont think this is a good idea. Just dont see how this will help our readers or editors...so many other essays could be linked like this. e.g for med articles...i would love a link to Wikipedia:Medical disclaimer. Would love to see Wikipedia:Contributing to Wikipedia to attract new editors..but ......we dont spam essays in this fashion. --Moxy (talk) 02:49, 8 February 2016 (UTC)

Hi @Moxy: Thank you so much for the feedback! We plan on testing other placements of the link as well per this outline of project steps. However, we wanted some baseline data from a pilot, so that we can help guide community conversation and decision making -- and better understanding of how user behavior might be effected. For example, we are learning that there is more interest in clicking through from the initial sampling of Medical articles than the MILHIST ones: this is not something we would have gotten from a general pilot in the interface.

I want to better understand two of your concerns:

• Why you think this particular effort is akin to other WP:Essays, with highly subjective analysis/opinions? We worked to ensure the content reflects both a) existing guidance, tutorials, and policies, b) the best practices already used in the outreach community and c) consistently identified as a problem by professionals in research process. Nothing that we talk about is new, or framed differently than it is elsewhere (its just simplified for a broad audience). We also welcome tweaks, and specific concerns about the content itself.
• Secondly, I am confused how this is "Wikipedia:Spam"? How are we pressing a particular position/opinion/type of resource on readers? Moreover, I want to highlight that this is a pilot with clear limits, that we hope to use for larger community consensus on this particular intervention to deal with the broader issues at hand.

Again, thank you so much for the feedback: you have quite a history contributing to our various help pages, and we welcome ideas based on your experience (I am also collecting the data to respond to the comment on my talk page). Astinson (WMF) (talk) 17:39, 9 February 2016 (UTC)

It would be better on talk pages

Moved from WikiProject Medicine ....

If it belongs anywhere, it would be on the Talk pages, definitely not in main space - but this is a much wider problem as other subject versions have been spammed much more widely. Guy (Help!) 10:47, 8 February 2016 (UTC)

This is a self-reference and does not belong in the article space. I would support its integration in the {{WPMED}} template on the talk page, but not as is being done currently.

@Barbara (WVS): (is Bfpage) might have established consensus for this in another forum - comments appreciated. JFW | T@lk 15:04, 8 February 2016 (UTC)

@JZG: Thanks for the feedback. I am not understanding "Other subject versions"? Can you point to what you mean?

@Jfdwolff: Thank you for the feedback. But talk pages very much defeat the purpose: most of our readers aren't even aware those exist. The primary audiences for this page are a) general researchers and b) people who teach general researchers (i.e. teachers, librarians), with the secondary impact on broader readership and editors. More discussion of the platforms own process is something that is well documented concern raised by professionals who study research process and our readership. Can you please explain why this is an issue?

Thanks much, Astinson (WMF) (talk) 17:39, 9 February 2016 (UTC)

I might just be resistant to change, but I think this is the kind of link that either belongs on every page in the side bar, or nowhere. It is not part of the article content or even of supporting information - it just links to a general resource. Finally, as I mentioned, it would need to be suppressed in sites reusing our content because they would not necessarily be hosting the page with the "using Wikipedia for research" article from that namespace. JFW | T@lk 06:51, 17 February 2016 (UTC)

3 requests in the essay to complete a survey is a bit much

that's all i have to say. Jytdog (talk) 02:48, 9 February 2016 (UTC)

@Jytdog: Thanks for the feedback JYTdog. We wanted to make sure that we caught people at the various points of reading termination. We haven't received too many responses yet (n=12). Is there one that you don't thing we should have a link? Astinson (WMF) (talk) 17:07, 9 February 2016 (UTC)

The BOT isn't perfect.

First, consider me in the group that feels placing this under references is the wrong place. My vote is for either the left side bar or top of the page. Part of my issue is related to the way the BOT currently places this template. The BOT apparently places this above the template:reflist entry, which leads me to believe your goal was to place it as the first line under the Reference section. In most articles, this will be the case. However, considering you've targeted MilHist articles, there will be many times when it will not. Many older articles place bulletized references before reflist template. Some articles place the reflist template under a different section. At some point, the BOT will target U.S. ship articles, many of which use information from DANFS. Often these entries use text from the public domain (i.e., DANFS), and when template:DANFS is used, it is most often placed before reflist, this template will be placed after the DANFS template. This makes the BOT, as it currently set up, appear to place this template at random for many articles. To help clarify, here's a small sample:

• template placed in the reference section, but not the first entry of that section: Arthur Thomas Drinkwater; Assistant Commandant of the Marine Corps
• template placed under a section other than the reference section: Arrogant-class cruiser; Battle of Dogger Bank (1915)

If this is to meet the intended goal, it will help if its placement in each article is consistent. As currently set up, the BOT will not achieve that.--KMJKWhite (talk) 15:45, 19 February 2016 (UTC)

Thanks for the great feedback! We are in the pilot stage around this, and this is part of the reason we are doing it in small batches in the pilot (see the proposed timeline). One consideration, was that, though not perfect, the top of reflists meets more people looking at the references, then if we put it above the PD-Text templates (and, because we have inconsistent labeling of reference sections in the community, its hard to teach machine logic to find the right section header). @Hazard-SJ: Can you puzzle out how to address this concern. In the short term KMJWhite, feel free to move/change the inconsistencies on articles you notice. Astinson (WMF) (talk) 17:35, 19 February 2016 (UTC)

@KMJKWhite:, Astinson (WMF) (talk) 17:36, 19 February 2016 (UTC)

@Astinson (WMF) and KMJKWhite: Is there any issue with it being the first item in the section?  Hazard SJ  21:57, 19 February 2016 (UTC)

I am okay with either placement, as long as its largely consistent. The vast majority of the articles are going to be true to both rules (before reflist, directly after the section header) both, and since our demo does have more consistent: having it directly after the section my be more visually consistent (but it needs to be a section with {{reflist}} in it). Astinson (WMF) (talk) 22:06, 19 February 2016 (UTC)

Yes, the script does check for the presence of {{reflist}} (or <references>...</references>), so the only change would be the exact placement. Also, there's only one run left (for insertion), and that's going to be 8000 edits, so we need to be certain before we run ahead with this. @KMJKWhite: How does that sound to you?  Hazard SJ  00:24, 20 February 2016 (UTC)

Consistent placement should be the goal, and placing it directly under the section header will achieve that better than the current set up. If you can make the BOT do this, count me in as supporting it.--KMJKWhite (talk) 04:51, 20 February 2016 (UTC)

@Hazard-SJ and KMJKWhite: if we can make a consistent rule for all of them to be directly below the section header, that sounds fine. Let me know if I can help, Astinson (WMF) (talk) 21:54, 22 February 2016 (UTC)

@Hazard-SJ: Did you take a look at this? Do you have a solution so we can do another pilot group, to fill out the full population? Astinson (WMF) (talk) 18:21, 29 February 2016 (UTC)

@Astinson (WMF): In a dry-run, I stumbled upon a few cases where the references section (level 2) of the page is divided into sub-sections (level 3), with all the references/notes etc. within those sub-sections. Should I simply insert them at the beginning of the level 2 section or at the beginning of the (possibly level 3) subsection in such cases?  Hazard SJ  21:05, 1 March 2016 (UTC)

Hazard-SJ, I'll let Alex confirm but I'd assume we want it in the same place under the level 2 header always, for consistency, even with level 3 headers present. Thanks so much for your help testing this out! Jake Ocaasi (WMF) (talk) 19:40, 2 March 2016 (UTC)

@Hazard-SJ: Yep I agree. For right now, lets work with the most consistent (under the Level 2 section headers). This is an interesting MOS questions that will definitely have to be part of the community consultation, once we have more evidence. Astinson (WMF) (talk) 19:47, 2 March 2016 (UTC)

@Astinson (WMF): All set! The next/final batch should be 8000 edits, and I've set the bot to specifically ignore other pages that start with A and B for that run. Let me know when you're ready.  Hazard SJ  05:34, 3 March 2016 (UTC)

@Hazard-SJ: All good for any time weekend, I will plan on being on hand for a good chunk of Monday to respond to concerns :) Astinson (WMF) (talk) 16:18, 4 March 2016 (UTC)

I had started the script yesterday, only to realize upon review earlier that I hadn't pulled my latest changes, so it only did 900 edits (which explains why there was no error in the log!). The other 7100 are underway now, though.  Hazard SJ  10:50, 7 March 2016 (UTC)

Pageviews

Despite this being on so many pages the view count is very low...I think this is an indicator that the location of the link is not all that good. Left side bar would be much better place even though others think the leftside bar is not used.....this is wrong POV as per our stats - Help:Contents pages views per day 5000- Wikipedia:About 10,000- Wikipedia:Community portal - Recent changes 20,000- Wikipedia:Contact us 6000. Just an observation. -- Moxy (talk) 20:26, 2 March 2016 (UTC)

Thanks @Moxy: Actually there is substantially more traffic going through the page. The template points to redirects which, on less than 2,000 pages is generating 250 page views a day, half of which are coming from 1000 Medical articles.

More generally: the information from the pilot is proving interesting so far: the spike in pages views after the first addition of medical articles, but subsequent flat in the next additions suggests that part of interest in research might be linked to specific topics. Moreover, the lower participation in Military History articles, might indicates a broader public interest in the health material as "researchable". The feedback on the survey is indicating value/appreciation, and we have some library practicum students doing assessment of the literacy impact -- so I hope to be able to share more information in the coming months -- ideally we would be able to test links to the page in a limited number of article sidebars, and similarly with a limited number of articles in the Reference Tooltips (the two other proposed locations).Astinson (WMF) (talk) 22:36, 2 March 2016 (UTC)

A large number of page views for at least the first week of this project are likely to be active editors with watchlists who are concerned about a somewhat cryptic bot addition. Unless you have a way to filter out logged in users, or better, those with the article watch listed, clickthroughs probably shouldn't be interpreted as interest in research.Dialectric (talk) 07:05, 8 March 2016 (UTC)

I'd agree with this. Despite Wiki's best efforts over the years, no-one uses portals and, to my mind at least, this will have exactly the same problem. Frankly, if a user can be bothered to click on the link, they can also be bothered to do their own research on google. I have no doubt the project was well-intentioned, but I'm not optimistic for its success.—Brigade Piron (talk) 09:14, 8 March 2016 (UTC)

If they can't filter logged in users, as a control for their study, the people behind this project might have added a 'Click on this meaningless button that does not link anywhere' template to the refs section of 10,000 other pages, to see how much of the clickthrough could reasonably be interpreted as interest.Dialectric (talk) 20:27, 8 March 2016 (UTC)

This is not article content and doesn't belong in articles

There's lot of things we might like to tell Wikipedia readers but the article content isn't the place. What next? Can I add a sentence saying "Please try the Visual Editor?" Or "Please donate to Wikipedia?". Or "please wash your hands after using a public computer". Why not run a banner campaign instead of reducing the signal-to-noise ratio of the articles themselves.

If people are using Wikipedia for research, then the fact that they are reading the References section is a very good start (they presumably grasp the importance of references). The readers who just read the article body and then perhaps copy-and-paste it into their homework are more worrying because they never scroll down long enough to see the References and this irritating message

And for those of us who do understand research and references, where's the opt-out? Kerry (talk) 13:51, 6 March 2016 (UTC)

I'm not sure the people responsible for this care that much about your or my opinion, they simply brush all the complaints aside and just keep doing what they want to do. Hzh (talk) 14:02, 6 March 2016 (UTC)

I would just add that information on how to use Wikipedia is not limited to those doing research, and if the information is important enough, then it should be on the left hand bar under a general heading to all users on how to use Wikipedia and understand its limitations. I would certainly support that. I also know for a fact that many lecturers in universities actually forbid citing Wikipedia as a source (or copying and pasting its content), therefore a special note in the article for those doing research is somewhat superfluous, given that those doing research should already know what not to do with Wikipedia. Hzh (talk) 17:35, 6 March 2016 (UTC)

Kerry, this seems to be a well-thought-out study with measurable goals. None of the tags added to articles are "article content", whether they're {{fact}} or this template. (There actually have been site banners that ask people both to try the visual editor and to donate to Wikipedia.) Do you have other ideas about how to get this information to non-editors?

This is precisely my point. We use banners to run campaigns about various things. Why not use a banner for how-to-research too? Then people like me (who is someone with a PhD who has engaged in serious academic work for pay) doesn't have to keep seeing it after one click to make it go away. While I agree that some of our quality tags (particularly whole-of-article ones) are way too large, nonetheless at least they are providing some warning *specific* to this article or specific claims within this article (e.g. citation needed)? Whereas this how-to-research message is not specifically about this article but general advice about any Wikipedia article. I'm not disagreeing that we might like to educate our readers in relation to their use of Wikipedia, just the means being used for doing it. Kerry (talk) 23:32, 6 March 2016 (UTC)

Hzh, if by "researcher" you mean "someone with a PhD who is engaged in serious academic work, probably for pay", e.g., lab work or trying to write peer-reviewed academic articles, then I also hope that they know how to do a proper literature search. However, it's also true that we've seen some sources develop a sudden spike in their popularity in peer-reviewed journal articles after they've been cited in Wikipedia articles, which suggests that not all professional researchers do their own literature searches. We've also seen multiple professional researchers plagiarize our work, which again is something that "those doing research should already know" is a bad idea. However, the audience for this isn't professional researchers; it's students and other non-experts who are trying to figure out more about a subject – people whom we can reasonably assume do not know how to use Wikipedia, and maybe not anything else. Directing teenagers and interested non-professional adults to a page like this seems to be a reasonable way to educate them. WhatamIdoing (talk) 22:39, 6 March 2016 (UTC)

Well, I have already said I would support some general information to all users on the side bar, so you are not addressing the question of why it needs to be in the article. Hzh (talk) 11:23, 7 March 2016 (UTC)

But no-one looks at the sidebar! That would make the entire venture pointless... [1] CFCF 💌 📧 11:53, 7 March 2016 (UTC) 

Really, you can't make that argument, or you could dump all the links on the side bar into the article (your link in any case says "users scan the content's left side in a vertical movement."). You might then even make the argument that no one reads the reference section, therefore it should go into the lede because you want to make sure everyone reads it. Having one on the side bar would indicate that it is meant for all users, and all readers could benefit from understanding how to use Wikipedia, for example learning about sourcing of information, assessing its reliability, avoiding POV content, what its limitations are, etc. Information on an article should not be targeted at only just a specific set of users such as those engaging in research, professional or otherwise. What's next, special note to dyslexic readers? Or children? Or professors? Or any other specific groups of people? Hzh (talk) 12:17, 7 March 2016 (UTC)

So, who by your reading does not engage in research on Wikipedia? Following your logic we shouldn't have the even more prominent edit button — because its presence only assists editors. CFCF 💌 📧 16:12, 7 March 2016 (UTC)

Odd someone would say no ones uses the side bar. We just talked about how much people use them. -- Moxy (talk) 19:47, 7 March 2016 (UTC)

Moxy, the left side bar doesn't exist at all on the mobile web site, and that's 45% of page views. We can therefore absolutely guarantee that the left side bar is not used 45% of the time. WhatamIdoing (talk) 01:20, 8 March 2016 (UTC)

Then that's something developers of the mobile site should fix. It doesn't mean we should start shoving in generic unrelated helper text into the article content. Wugapodes (talk) 02:09, 8 March 2016 (UTC)

I don't think so. If you tried to read an article on a smartphone, would you really want the left half of the screen covered up by that sidebar? WhatamIdoing (talk) 03:37, 8 March 2016 (UTC)

There are other methods than half the screen being taken up by a sidebar. There can be footers (like we have for getting to talk pages). There can be a side bar that you toggle by a button or swiping like with bootstrap or Google's material design. Mobile not having a sidebar isn't because it's impossible, tons of apps and websites have implemented them. I'm assuming the people paid to do the UX design for the mobile site are smart enough to figure out something similar. Time would be better spent developing the solution people seem to be wanting rather than trying to circumvent guidelines like WP:SELFREFERENCE. Wugapodes (talk) 04:12, 8 March 2016 (UTC)

Thank you so much for the feedback. We are trying to meet readers who already are using Wikipedia to do deeper learning/research, hence why the reference section.

This is, of course, a pilot in an attempt to deal with a very clear and well understood digital literacy/knowledge issue, that we are trying to test: WP:Research help/Proposal. This last set of additions to articles, is currently happening right now. In about a month, we are going to assess the data, and also look for piloting 2 or 3 UX/UI positions. However, as we discuss in the top section -- the left hand bar doesn't show on Mobile, and over half of our readership is coming from mobile now, so any additions in the desktop interface, are guaranteed to miss half of our readers (mobile web is only ~1/6th of the pageviews from this link, so its telling us something about that readership).

As for the concept of opt-out raised by Kerry -- there certainly could be a script that removes the template from your visible content. But, since this is a pilot, I would wait until this is something embedded more permanently.Astinson (WMF) (talk) 16:50, 7 March 2016 (UTC)

@Hzh and Kerry Raymond: Repinging, above is a response to this conversation. First ping was bad, Astinson (WMF) (talk) 16:54, 7 March 2016 (UTC)

As expected, invite comments, dismiss all criticisms, and do what you want to do anyway. Hzh (talk) 21:34, 7 March 2016 (UTC)

OK, let me make a more positive suggestion if you insist on the notice. Incorporate it into boxes you sometimes see at the bottom right of articles, for example here Prokaryote on Wikimedia, Protease about library resources, Charles Dickens with Wikiquotes and and Wikisource, etc. It might be a good idea to collate all these and put them into one box. Hzh (talk) 22:12, 7 March 2016 (UTC)

Thanks for the productive comments @Hzh: we fully acknowledge that this might not be the best solution, and that there are probably a lot of reasons to think our approach isn't going to work -- but we are really interested in getting past assumptions about if it will or won't and understanding what the approach does do. The feedback that we have solicited from the survey has been incredibly positive -- in that we have as many readers who don't give feedback on Wiki and some prolific editors, liking this extremely -- both the idea of the template, and the content on the page. In part we were inspired by the library resource tag as you note, but we know that that location doesn't get clicked on often (per portals), and it doesn't meet many readers where they need the help most (when they get down to the reference section), Astinson (WMF) (talk) 16:56, 8 March 2016 (UTC)

The thing you should realized is that it is not about the clicks you get - you could argue a case for it being in the lede if the number of click is what you want, and everyone wants to put things (content or otherwise) in the lede, however inappropriate they may be. It's about doing what is appropriate, and if you don't take other people's opinions on board (and the overwhelming opinion of those not involved in the project here is negative), then the whole project will get sunk. Hzh (talk) 17:11, 8 March 2016 (UTC)

Thank you Hzh, and I hope I am demonstrating I am listening (I really am, and with this growing interest in the sidebar, I am beginning to think that might be a good intermediary solution -- or one of the several ways to place the link to good effect). We have been making lots of tweaks to our plans: and we plan to test the link in the sidebar and in the reference tooltips. Its interesting that a couple of readers, in our survey, have suggested we should be displaying this information much higher in the article. However, we recognize that would be a bad placement strategy, because of its link baitiness. We are exploring other options (personally, I think the Reference tooltips might be a good compromise or baked into the UX/UI for reflists). But this needs to all come through consensus and discussion, which we are use this pilot to collect data to present in that context. Astinson (WMF) (talk) 18:03, 8 March 2016 (UTC)

Arbitrary break

I really don't find this addition to be a good thing. The article space is for content. That's why we have guidelines like WP:NOTHOWTO and WP:SELFREF. This isn't content, it's simply a link to a generic project space page. I strongly oppose this non article content being placed in the article, and I don't understand the pushback from the project participants about putting it in the sidebar, especially considering it is used and would go on every page which is pretty much the ideal situation for something like this. Wugapodes (talk) 00:28, 8 March 2016 (UTC)

@Wugapodes: See the comments by CFCF above F shaped reading patterns. Also, the sidebar doesn't reach mobile users. Astinson (WMF) (talk) 17:02, 8 March 2016 (UTC)

I concur that this information should appear elsewhere. Article space is for content. The purpose seems to be to tell readers how to incorporate information from Wikipedia into their project rather than how to do academic research on Wikipedia itself (which is what I initially took it to mean). The advice given is entirely generic and doesn't relate to the article itself at all. A dedicated link in the content area of each article is not in my view a good way to get a general message across. --MichaelMaggs (talk) 10:25, 8 March 2016 (UTC)

@MichaelMaggs and Wugapodes: Thanks for the feedback. I am not sure why the page didn't meet your expectations about research: The page has advice for both editors to find research materials (in the right hand column), and to introduce readers to how Wikipedia contributions work. How would you communicate this basic message? Do you think its one we should be sharing with our readers more often? I look forward to learning more about your thoughts, Astinson (WMF) (talk) 17:02, 8 March 2016 (UTC)

Appearances

I don't like this either, for reasons that others have stated, but that's not why I'm here. I find the appearance of this notice very odd, to the point that I think I've been parsing it wrong. Having the question mark as an icon at the end of the line makes it appear to be a separate UI element, not part of a sentence (or sentence fragment in this case). So at first I thought this was a link labelled "Using Wikipedia for research" followed by a "help" icon; I expected if I clicked on the question mark I'd get a popup explaining what would happen if I clicked on the text. Since they both go the same place, I guess I don't see the point of the icon. Also note that at least for me the icon does not scale with the text, which can result in it appearing huge depending on prefs. Kendall-K1 (talk) 19:29, 7 March 2016 (UTC)

@Kendall-K1: Thanks for the nuanced feedback. Thats a great recommendation, and definitely a concern. I wonder if there is a good css way to ensure that they are in the same scale. I can also add some alt-text to see if that helps with the hover issue. If you know how to tweak it, we would welcome consistency tweaks to Template:Research help, Astinson (WMF) (talk) 21:33, 7 March 2016 (UTC)

@Kendall-K1: I tweaked the template a bit, so that the question mark stays even with the line break, and is a little bit smaller: see the diff. Does that make it more approachable?

I tried adjusting the size of the image to em's (which would scale the image to each users default font size), but I don't think our images can be scaled in relationship to font, Astinson (WMF) (talk) 21:59, 7 March 2016 (UTC)

Incorrect parameter

Why is the template being added with the "Med" parameter to non-medical articles such as Disability and Deaf rights movement? I've changed it to the "Gen" (General) parameter in those two articles. If such tagging were done by a human editor it could be seen as an intentional ableist insult, so please teach the bot to do better. Roger (Dodger67) (talk) 20:40, 7 March 2016 (UTC)

Probably because both those articles are tagged on their talk pages as being part of WP:WikiProject Medicine. Kendall-K1 (talk) 20:45, 7 March 2016 (UTC)

That is not a sufficient indication that the article topic is specifically medical. If it has only the medical project's tag it is probably correct, but when other projects have also tagged an article it might not be an actual medical article. The bot should only use the Med parameter if the medical project tag is the only one. BTW I see no valid reason for the medical project tag to be on Deaf rights movement which is a political/civil rights/legal article - clearly not medical at all so I'm removing it. Roger (Dodger67) (talk) 21:06, 7 March 2016 (UTC)

@Dodger67: The template does not change the content of the article or link to a different version of the page, it just redirects readers through which pilot is relevant (WikiProject Medicine, Military History, or not part of one of the targeted pilots). WikiProject Tagging is largely a matter of volunteer maintenance. I am sorry that you found this problematic, Astinson (WMF) (talk) 21:28, 7 March 2016 (UTC)

Roger, you should go revert those changes. "Gen" means that an editor voluntarily chose to add this template to the article. "Med" means that the template was added by a bot as part of a structured research study, based on the list of pages that WikiProject Medicine has tagged. None of the options mean "this is a general subject" or "this is a medical subject". By changing the tag mid-study, you may be accidentally screwing up the data. WhatamIdoing (talk) 01:27, 8 March 2016 (UTC)

Nomination for deletion of Template:Research help

Template:Research help has been nominated for deletion. You are invited to comment on the discussion at the template's entry on the Templates for discussion page. --Dirk Beetstra ^{T C} 06:29, 8 March 2016 (UTC)

Three WikiProjects do not speak for the community and does not have consensus

I was just about to revert several of the recently added spam templates and thought I would comment here first. These templates are not appropriate for inclusion on article pages and should either be on the talk pages or on the left hand navigation menu. The left hand nav menu has a lot of useful links, so if it is not appearing on Mobile platforms then that is a problem that should be addressed.

Also and more importantly, changes like this that affect the entire community should be discussed appropriately. Three WikiProjects do not have the authority to speak for the entire project for changes like this. Since I see the template has been submitted for deletion I will vote to support that in lieu of a better plan.

It's also worth noting that adding these does precisely zero for researchers. It may help children if that is your target audience. Wikipedia has been around for more than 15 years. It's a household name and virtually everyone in the world knows what it is. Adding spam to the article with the argument that it's to help researchers is ridiculous. Anyone who is high school level or higher certainly knows what Wikipedia is, what it is not and what it's worth from a research standpoint (Zero). Navyvet2016 (talk) 14:23, 8 March 2016 (UTC)

@Navyvet2016: Thank you so much for your thoughts. I agree Three WikiProjects is not consensus, and we definitely don't plan to deploy it any further (we have maxed out the number of transclusions allowed for our pilot).

Actually, we have really good evidence that both academics and librarians really appreciate this link (both in our outreach, and in the survey linked on the page). Because of the rocky reception Wikipedia received for many years, there are a number of misconceptions about Wikipedia, and we have evidence in multiple contexts, and documented well on the WP:Research help/Proposal that most of our readers don't understand how its made, how it should be used (by verifying information through sources), or how to answer the question about Wikipedia's credibility. Even though we have seen a lot of press in North America, many of our readers have bad information, and in parts of the world where Wikipedia and internet access are relatively new, there is even more missing information. For example, the communications department at WMF did a study of how South African's perceived Wikipedia, and some of them thought it was a corporate effort run by Google and Microsoft. If we expect more people to contribute to our community, we need to make sure that these very basic assumptions get shared with them -- when they want to think about it, which is when they are trying to understand if they should trust Wikipedia or not during research. We are listening to better plans as they are proposed, and hope to synthesize more of the feedback soon, and when we have the data from this initial research. I hope this helps you understand why we framed it this way, Astinson (WMF) (talk) 17:44, 8 March 2016 (UTC)

"(we have maxed out the number of transclusions allowed for our pilot)." @Astinson (WMF): that sounds like the usual WMF doublespeak crap to avoid stating "we were only in the midst of adding it to another 10,000 pages when the bot who added these was blocked for this". Fram (talk) 19:39, 8 March 2016 (UTC)

Astinson (WMF). Sorry Alex but I have a few problems with your statement. First, what evidence? This "pilot" just started, so how can there be "good evidence" on something that is less than a week old? Second, Librarians and researchers are not the only demographic that uses Wikipedia. In fact, IMO, they are not likely to even care about an insignificant help link buried on the bottom of the article. If they are librarians and researchers then they have enough experience that they already know how to use Wikipedia. If not, then they likely need to find a new line of work. The best way to improve people's perceptions of Wikipedia is to treat it like an encyclopedia with literary and formatting standards that we follow. Not adding unrelated spam links to the bottom of articles. Lastly, this is, according to comments above, focused on Mobile edits. Mobile is important, don't get me wrong, but we shouldn't sacrifice the quality and integrity of the content of the encyclopedia just to cater to one niche because our Mobile app was poorly designed. Navyvet2016 (talk) 20:27, 8 March 2016 (UTC)

Feedback

I have some concerns about the implementation of this project and the page itself:

• Firstly, WikiProjects do not have absolute dominion over their topic areas; placing a template that would affect the reader experience based on a consensus of three WikiProjects is probably poor consensus. Consensus represents the entire community. "Inferred" consensus from a small group of our general readership (i.e. librarians and researchers) is not consensus.Esquivalience ^t 02:22, 10 March 2016 (UTC)
  • Agreed, this is why it was only a local pilot -- we didn't expand to a big project.Astinson (WMF) (talk) 20:22, 10 March 2016 (UTC)
• This template is like the pesky donation banners that disregard the cookies that it places in readers' browsers: readers may be interested once but, once they have made use of the page, will probably use is less and less and the template begins to be clutter. This is unlike templates such as {{Citation needed}} that inform the reader about unverified statements, which number thousands, and etc. Readers that will make repeated use of this page will probably bookmark it, and almost all desktop and mobile browsers have bookmarking ability. Perhaps only show the template once?Esquivalience ^t 02:22, 10 March 2016 (UTC)
  • I am not comfortable assuming anyone will bookmark the link on first usage. One of the potentially usage stories, is that a student discovers the link while writing a paper, uses it to create a justification for editing the page, writes a paper using Wikipedia, and then the teacher asks about what they read, and the student needs to find the link again. Similarly, teachers showing students basic literacy about Wikipedia, will want to be able to rely on finding that page. Now, we are investigating creating a user script to remove it for logged in editors -- we would love your help working on that. Astinson (WMF) (talk) 20:22, 10 March 2016 (UTC).
• Additionally, Wikipedia:Research help#How reliable is Wikipedia? disregards the fact that Wikipedia makes no guarantee of validity and there is a considerable amount of content that contain errors, misinformation, original research, or otherwise mislead by misrepresenting viewpoints, especially on topics such as Indian businesses and so on.Esquivalience ^t 02:22, 10 March 2016 (UTC)
  • Linked to the General disclaimer, about information potentially being wrong. We also talk about systematic bias in other sections. We are doing research with academics, to make sure that the page actually covers the critical thinking skills needed to be a discerning reader of Wikipedia. Astinson (WMF) (talk) 20:22, 10 March 2016 (UTC)
• Few readers seem to have interest in this page. For comparison (as of 8 March 2016), there are about 551 million page views per day (excluding crawlers, [2]) for the English Wikipedia, spread across 5.1 million articles. This equates to 108 page views per day. Note that this is a very conservative estimate, as this includes the permastubs and non-notable subjects as a result of generally poor new pages patrolling. The template has 10,653 transclusions. Within the articles that the template is transcluded in, the potential audience is about 178,000 readers. Yet the page only received 672 views. That is about 0.3% of the audience was supposed to reach. Additionally, it appears that the page receives only about 0.05 new daily page views per transclusion based on a rough comparison between transclusions and page views. If the template is placed on every English Wikipedia article, then it may receive a burst of 300,000 page views but because readers probably will not refer to this frequently, will become a nuisance to most soon after.Esquivalience ^t 02:22, 10 March 2016 (UTC)
  • But that is much higher than the click through ratio for Help:Contents and WP:About linked in the left sidebar (which is one of the suggested (we think that is because the question is asked directly in a location where someone is doing research -- rather than in a arbitrary point elsewhere in the interface. I think judging whether this is enough, is going to have to be a concensus issue, and probably compared to tests in other parts of the interface. Astinson (WMF) (talk) 20:22, 10 March 2016 (UTC)

- Esquivalience ^t 02:22, 10 March 2016 (UTC)

@Esquivalience: Thanks for the great feedback, some responses inline above, Astinson (WMF) (talk) 20:22, 10 March 2016 (UTC)

• Who is saying what here ...looks like one person replying to themselves.--Moxy (talk) 02:11, 12 March 2016 (UTC)
• @Astinson (WMF): (specific note for the "removal" script): One-liner is $(".research-help").remove();. Esquivalience ^t 02:47, 14 March 2016 (UTC)
  • Brilliant! Thanks, going to add that to the FAQ, Astinson (WMF) (talk) 18:23, 14 March 2016 (UTC)

Research methodology

Taking off my "grumpy old Wikipedian" hat and donning my academic bonnet of a professional researcher, I am curious how this trial is to be evaluated in terms of the success of its mission.

The commentary here suggests this is a trial on 10,000 pages. What is the duration of this trial? I saw some mention of one month. Is that correct or is there some other criteria that ends the trial?

Yes, we had intended 1 month -- but we didn't label it as such, because we have seen a number of small failures in maintaining schedules -- so didn't want to overcommit. We planned on withdrawing the template with noinlcudes at 1 month or when a concensus emerged that it was done. Because of delays in implementing it and in our other research, our preferred timeline has become mismatched when we are getting which data.Astinson (WMF) (talk) 00:06, 11 March 2016 (UTC)

After the trial is completed, how is the success or failure of the trial evaluated? I saw a mention of hits on the link to the Research Help page. Is that the criteria? If so, how will hits by people actually in need of this information be distinguished from those who were (like me) curious/irritated/enraged? I saw a comment about number of survey completions, but given the call to action is to "help improve this page", then it suggests that responders may well be people who want more information about the topic or those who have come to this page feeling they understand the topic but think the advice could be improved in some way. It seems neither of these have any real potential as a measure of success.Kerry (talk) 22:14, 10 March 2016 (UTC)

We are doing several things at once: collecting the data from source, getting consultant research in digital literacy from both a library practicum student and a couple of outside experts (included, the Rutgers study librarian), and doing some reader walk through UX/UI conversations. We hoped to analyze all these to see if there is a better decision and provide that data to the volunteer community to get wider consensus, rather than one point of failure. We honestly didn't know what the results would look like, because we aren't aware of a similar intervention in the way readers interact with our content (most interventions have been purely design oriented or interface oriented). We also have a body of colloquial knowledge collected from the GLAM, Education and Library outreach communities suggesting this kind of basic literacy is huge barrier to new reader conversion to editors (and is something being validated by Aaron Halfaker's work). Our hope was to examine this as a programmatic intervention -- as part of the Wikipedia Library's domain of research outreach, and generating editors who do research. This means its less rigorous -- because it operates more within our theory of change and programmatic opportunity. That being said, we are trying to collect as much learning as we can, with the connections/support we do have. Astinson (WMF) (talk) 00:04, 11 March 2016 (UTC)

Is it intended to run another Twitter survey? If so, what level of improvement in the responses is desired as a metric of success? And do we believe that the audience of Twitter (generally) or the followers of Wikipedia's twitter account (particularly) are representative of Wikipedia readers generally or of the Wikipedia readers who we think are in need of the information in the Research Help page. FWIW, Twitter has a relatively low following compared to (say) Facebook and with a different demographic. See here for some discussion of demographics. One specific point raised there is that Twitter is much more "male" and Facebook more appealing to "women". Given Wikipedia's somewhat notorious gender gap, I would be very careful about taking input from a predominantly male source for fear of increasing the systemic bias.Kerry (talk) 22:14, 10 March 2016 (UTC)

The Communications team was using that survey as a test, in how to engage its audience. I would love to do something similar again as well in other contexts. I do know a few things about our social media audience. From the #1lib1ref campaign we were able to get over 50% women engaging on twitter, so at least the @WikiLibrary handle has a strong female following, and the biggest audiences for both Facebook and Twitter for the Wikipedia handle are in India. For Wikipedia 15, they have done some initial analysis of the comments in both mediums, and also found a wide range of comments about educational use, student research, and "credibility questions". The reading and communications team are also doing broader studies, which haven't been released yet, that suggest non-Western audiences don't have even as much understanding of Wikipedia (waiting for those studies to be published, but I promise they exist). I think of the twitter survey, as one of a constellation of signals that we have a core literacy problem -- that we need to tackle in multiple ways.Astinson (WMF) (talk) 00:04, 11 March 2016 (UTC)

As per the mentioned poster, is it intended to grab the usual "convenient sample" of undergraduate students keen to make their teacher happy by being research subjects (university students are the best studied species on the planet) to be interviewed about whether they have seen these links to the Research Help page and whether they have followed the link and felt better informed because of it or changed their behaviours as a result of it? I note that a sample of 30 students (as used on the poster) are probably statistically unlikely to have even seen the link given it's on 10,000 out of 5M pages. So I think you'd have to survey a lot more readers to find ones that had genuinely seen it in their random browsing rather than construct a survey in which the student keen to please their teacher is shown the link first and then asked about it.Kerry (talk) 22:14, 10 March 2016 (UTC)

I agree, we need bigger studies. That something we would love to see, and something TWL wants to encourage among our partners, but we don't have the bandwidth to persue it actively. We do have some research from years ago (for example) which talks about research usage/literacy, but those data points are really old -- I compiled the proposal material using much newer data points. Astinson (WMF) (talk) 00:04, 11 March 2016 (UTC)

I ask these questions because I do believe it is genuinely difficult to judge the success of a Wikipedia-based program aimed at unknown anonymous readers behaving differently in contexts outside of Wikipedia (e.g. students doing better research and writing better assignments) using any kinds of metrics that we can gather from Wikipedia itself. If you look at the 1Lib1Ref program, there are ways you can use Wikipedia metrics to evaluate it -- you can look for the hashtage #1Lib1Ref that they were asked to add in the edit summary and you compare the number/proportion of edits that add a ref tag before/during/after the campaign to see if the campaign made a visible impact and to what extent it may have been sustained after the campaign.Kerry (talk) 22:14, 10 March 2016 (UTC)

I would love feedback on our #1lib1ref campaign analysis at meta:The Wikipedia Library/1Lib1Ref/Lessons. It was a small scale program, and operates under much the same theory as this intervention: that the biggest barrier to contribution in our community, is a lack of deep understanding of how editors identify what needs more research and verification.Astinson (WMF) (talk) 00:04, 11 March 2016 (UTC)

So, for this trial, is it possible to measure the rate of click-through of external links in citations? If the campaign is successful, we would expect readers to be more interested in reading citations, which should manifest in higher click-through rates in the references section. So, in theory, we could measure:

• any changes in the rate of click-through to citations in the articles with the Research Help link
• ... in other articles within the scope of the Wiki Projects participating
• ... in articles generally

Kerry (talk) 22:14, 10 March 2016 (UTC)

Unfortunately, I think this kind of change is going to be almost impossible to measure in the short term. We do have some evidence from the Wikipedia Education program, that students that understand how to edit, are more articulate and self aware about their own research habits (I have a 2 term study of different groups of students I taught at K-State that will be published soon which verifies this)-- and how they use Wikipedia. We have also so seen the rate of DOI referrals increase from 8th to 5th. Moreover, WMF Research and TWL are working to document how referrals work to academic works as part of research on how the HTTPS change effected referral information. Because the TWL team is largely experienced in programatic support, we are approaching this pilot as a programmatic activity, which is more based on our audience needs and seeing if we can generate community support for a theory of change.Astinson (WMF) (talk) 00:04, 11 March 2016 (UTC)

Is such data available? Lifts in the click-through rates of citations would demonstrate a measure of success -- it might even convince this grumpy old Wikipedian :-) Kerry (talk) 22:14, 10 March 2016 (UTC)

@Kerry Raymond: I did some responses above, in line--- feel free to break them out, or respond there as well, Astinson (WMF) (talk) 00:06, 11 March 2016 (UTC)

Thinking a little more about it, we possibly can get a tighter set of data around student behaviour from Wikipedia itself. We have 85000+ IP users which are tagged as being educational institutions (Template:Shared IP edu). So we could add to my list above:

• any changes in the rate of click-through to citations by educational-institution IP addresses

Is this data obtainable? Kerry (talk) 22:36, 10 March 2016 (UTC)

I would love to help someone obtain that data, and I am sure its something that WMF Research would be interested in helping someone acquiring this aggregated anonomously. If you are interested, I would love to connect you -- go ahead and email me: astinsonwikimedia.org, Astinson (WMF) (talk) 00:04, 11 March 2016 (UTC)

• Who is saying what here ...looks like one person replying to themselves.--Moxy (talk) 02:10, 12 March 2016 (UTC)

@Moxy: like in the above section, I responded to the bulleted questions by Kerry, with indented responses directly after the question, rather than have hard to read responses that are mismatched. Astinson (WMF) (talk) 18:22, 14 March 2016 (UTC)

While I agree the structure of talk pages isn't helpful here, if you are going to respond in-line in this way, I think you need to copy my signature over to the ends of each chunk before your reponse, so others can clearly distinguish what I said and what you said. As it stands, it looks as though you have written much of what I wrote based on the positioning of the signatures. Kerry (talk) 20:54, 14 March 2016 (UTC)

 Done and will do so with the other section, Astinson (WMF) (talk) 22:19, 14 March 2016 (UTC)

Some important points

1. You should get approval from WP:VP(T)WP:VP before imposing this on 10,633 prominent pages.
2. The name is misleading - the project is mainly about "How to use references"
3. WikiProjects do not "own" articles that are within their scope.

All the best: Rich Farmbrough, 11:24, 22 March 2016 (UTC).

Hi @Rich Farmbrough: Thanks for the feedback. This is a pilot - we sought concensus from the editors most effected by the change and notified village pump about the conversations. We plan to pull down the pilot after it has been on article content for 1 month, to make sure we get enough data and feedback to determine if it continues, and to get sufficient feedback on the page proper to give us a good point of reference. The no consensus at this TFD suggests that there is at least significant enough support for this pilot, though also significant concern about the template, to better assess this and have a full concensus (we plan to have a full scale VPT conversation before any other scaling).

As for misleading title: is it? How to use references, and how to check if Wikipedia content is verifiable, is an important part of anyone's research process if they are consulting Wikipedia. Also, our feedback on the survey strongly suggests that the page answers most of those questions, and that students and educators are finding it useful (over half of the survey respondents are related to teaching/universities). If you would like to suggest some other titles, we would be happy to test them. Astinson (WMF) (talk) 17:01, 22 March 2016 (UTC)

It would be useful to have a definition of "research" in this context. We have the following (at least):

1. People who publish "research papers"
2. People who are researching for other published material, which may or may not include references
3. People who are doing "research" to show some didact they have learned how to do it.
4. People who are finding out about something "researching it" for a practical purpose
5. People who are adding to their store of knowledge - "researching" for their own understanding

It is far from clear which of these is the target audience, or how the message is relevant to other audiences.

(Now if WMF wants to trial something like this, why not propose it to the community (and have it put into the sidebar)? Has WMF learnt nothing from Lila's departure? From Visual Editor? From Flagged revisions? From Superprotect?)

All the best: Rich Farmbrough, 17:30, 22 March 2016 (UTC).

Hi Rich: this fell off my radar in tracking the other conversations, many apologies: "research" for the vast majority of people in the public, means learning and collecting information for synthesizing that information for another audience (whether in school, hobby or as part of a job). In developing this, our ideal audiences were 1)Librarians and educators attuned to this kind of conversation already, 2) folks in educational and research setting which robustly use reference sections and other parts of Wikipedia, and 3) potential contributors and/or new contributors who could imagine themselves as "doing research" -- a skill we need very broadly in the community, because it provides the foundations of core policies . I will say that all of the feedback we received in the survey from readers did not challenge or question this. We believe its sufficiently clear in many contexts, but would be open to a better and/or simpler language.

As for the meta question of "hasn't WMF learned". Quite frankly, we are very different teams, working in very different areas, and this was not a software change, so doesn't follow the same kinds of processes as VE, etc: we were doing a programmatic reader-facing experiment relevant too our mission and scope as a program (programatically exploring how to improve the entire Research process on Wikipedia). During the time we were working on this page, two seperate reader surveys and interviews conducted by two seperate teams at the foundation in South Africa and Mexico identified the exact same weakness in reader literacy that we are trying to examine: this isn't just a pet project, but a global issue of readers not understanding how Wikipedia is created, or knowing how much (or if) to trust that process -- and subsequently the content and platform. Moreover, without this knowledge, you are defeating the broader editor pipeline (you need to know how Wikipedia works, before you can edit it). We have pretty substantial research in the Western libraries and education communities to support this as well-- a part of our next step in the process, is to better understand what we didn't present effectively to the community about this. I appreciate the feedback and thorough thinking, and I would greatly appreciate your input/feedback on next steps, once we do a "after action" review of our process. Astinson (WMF) (talk) 23:41, 4 April 2016 (UTC)

What are we trying to solve?

I have read this section four times. It explains the authors' opinions on several matters, with some references, which might provide the context of the problems, but does not itemise the problem or problems to be solved.

All the best: Rich Farmbrough, 14:37, 25 March 2016 (UTC).

Hi Rich: we are trying to solve a broad illiteracy amongst our global readers about the process by which our content is created, and to provide tangible language and stories that help readers, esp. educator, librarians and student but also more broadly partially "bought-in" readers, to understand how and why Wikipedia works the way it does, with a wide range of quality in content. For Wikipedia Education Programs, Library outreach, and other GLAM outreach the world over: this is a oft repeated set of knowledge, and often provides the first barrier for creating new editors. We believe that this will help fill the trust gap, by informing readers where they are at in the process of reading and researching new knowledge on Wikipedia. See above and the data in the Proposal page for more information of why we believe this is a key problem for our community and readership. Astinson (WMF) (talk) 23:47, 4 April 2016 (UTC)

Courtesy notice

There has been posted at WP:VPR#"Research help" proposal a proposal to terminate the trial. BethNaught (talk) 22:26, 25 March 2016 (UTC)

I pinged everyone. All the best: Rich Farmbrough, 05:09, 26 March 2016 (UTC).

Rich, I didn't get your ping. Maybe they don't work when they're inside a box? WhatamIdoing (talk) 20:35, 1 April 2016 (UTC)

Curious. There may be a limit to the number of accounts you can ping from one message. All the best: Rich Farmbrough, 20:41, 1 April 2016 (UTC).

Noinclude

I have noincluded the template just as the WMF staffers said it could be if there was any objection.

I therefore think it unreasonable for said WMF staffers to revert, especially since we have consensus.

Nor is it concomitant with Wiki culture to filibuster, to get the "required" outcome.

All the best: Rich Farmbrough, 01:27, 3 April 2016 (UTC).

@Rich Farmbrough: FYI, none of the staff involved in this project reverted the consensus change, that was Ed17 -- during his personal, volunteer time. He has not been involved in our programmatic work -- its two different parts of the organization. Astinson (WMF) (talk) 18:08, 4 April 2016 (UTC)

Yes, Ed has made that clear. And I made clear why I had thought he had his WMF hat on. It's interesting how people believe that their innate honesty and ability to compartmentalize without being affected by COI, shines through the written word.

Do you have any response to sections 15 and 16?

All the best: Rich Farmbrough, 18:55, 4 April 2016 (UTC).

Sorry for not responding sooner: too many moving projects at once, and lost track of these messages. I hope the above explanations help. I would really like to partner with you to see if we can find a better way to approach and communicate how a project like this might work -- or what the better options might be for approaching the broader problems of our readers not understanding how Wikipedia's content is tied to sources, thus able to use it effectively in research. Astinson (WMF) (talk) 23:49, 4 April 2016 (UTC)

Alternative

Here is a possible alternative

A box would be nice of course.

All the best: Rich Farmbrough, 19:12, 14 April 2016 (UTC).

@Astinson (WMF): All the best: Rich Farmbrough, 11:45, 6 May 2016 (UTC).

Hey Rich, thanks for the idea. I have been traveling a fair bit recently, so sorry for not responding sooner. We are analyzing the survey and pilot data right now, so I will have a better response to you on the alternative and where this fits into the larger research that is going on right now, in terms of helping our readers understand our content (it appears that several other teams were thinking about similar things at WMF, and we have begun talking to them as well).

Personally, and based on the research we are seeing come out of other teams at the Foundation, I don't think adding a link on the left hand side of the page is a good solution (though the way you insert the text right now is very interesting -- and definitely something that I might apply in other contexts). I really want to make sure we are proposing solutions that don't just shuffle the link into part of the interface that we do not have good evidence that its used very frequently. Thanks for staying engaged! Astinson (WMF) (talk) 21:52, 10 May 2016 (UTC)

I think the standard "tool-box" is something people are acclimatised to, and therefore ignore. Something like this, standing on its on would draw attention, whilst still maintaining the division between content and instruction.

I may propose a weeks trial, if I have the energy.

All the best: Rich Farmbrough, 16:54, 1 June 2016 (UTC).

@Rich Farmbrough: Would love your feedback on the evaluation of the outcomes described in the section below. Sorry for not getting the information published sooner, Astinson (WMF) (talk) 14:39, 14 September 2016 (UTC)

Input on Research Help Pilot evaluation

The Wikipedia Library has now posted a report on its spring pilot test of a Research Help portal. As the report outlines, our target audience of readers and new editors generally reacted more positively to the pilot than experienced editors, who raised important critiques for discussion. The report provides more details on the results and some proposed next steps for the project. Your input is welcome on the report talk page.Astinson (WMF) (talk) 14:38, 14 September 2016 (UTC)