Wikipedia

Wikipedia:Bots/Requests for approval/ExpertIdeasBot

< Wikipedia:Bots | Requests for approval
The following discussion is an archived debate. Please do not modify it. To request review of this BRFA, please start a new section at WT:BRFA. The result of the discussion was  Approved.

ExpertIdeasBot

edit

Operator: I.yeckehzaare (talk · contribs · SUL · edit count · logs · page moves · block log · rights log · ANI search)

Time filed: 19:03, Thursday July 31, 2014 (UTC)

Automatic, Supervised, or Manual: Supervised

Programming language(s): Python

Source code available: ExpertIdeasBot on GitHub

Function overview: We are a group of researchers at the University of Michigan, Carnegie Mellon University and the University of Pittsburgh. Our research aims at discovering ways to motivate academic researchers who can be considered domain experts to provide feedback about Wikipedia articles in their area of expertise. The goal of the bot is to make the process of posting comments on article talk pages easy for users who are not familiar with Wikipedia markup language. The users who have access to this bot are experts in different scientific fields such as psychology and Economics and the purpose of the comments left on the article talk pages are providing information on how the article can be improved. To avoid the abuse of the bot, the bot is not allowed to post comments more than one in 2 minutes per single user. Any attempt to abuse the bot will alert the system administrators and they will follow up on the account which has been making the attempt.

Since the messages from this bot are individual messages from real people, we do not want to skip any message even if it is related to template pages, we would like all messages to be be delivered.

Moreover, for the same reason and the fact that these edits represents the comments of a real person we would like those post *not* to be flagged as by a bot to make sure that they are not going to be filtered out of watchlists and recent changes Links to relevant discussions (where appropriate):

Edit period(s): Continuous over the period of study.

Estimated number of pages affected: We are going to add new sections to at most 3000 talk pages.

Exclusion compliant (Yes/No):Yes

Already has a bot flag (Yes/No): No (new bot)

Function details: The bot is part of an experiment in which we are going to send emails to academic researchers who have published a number of research papers in various domains to review specific Wikipedia articles related to their domain of expertise. Once the experts provide us with their comments about the content of these articles, our ExpertIdeasBot will post these comments as new sections to the corresponding talk pages. All the new sections which are going to be created will have the following format:

Title: Professor ...'s comment on this article

Professor ... has recently published the following research publications which are related to this Wikipedia article:

Reference 1: ... , Number of Ciations: ...

Reference 2: ... , Number of Ciations: ...

Reference 3: ... , Number of Ciations: ...

...

Professor ... has reviewed this Wikipedia page, and provided us with the following comments to improve its quality:

... ... ...

We hope Wikipedians on this talk page can take advantage of these comments and improve the quality of this page accordingly.


Then the bot will add the Talk page and also the corresponding Wikipedia page to my watchlist, so we will be able to observe the active Wikipedians’ reactions to the experts’ comments, which will help us to realize:

  1. if these comments really help Wikipedians to improve the quality of these pages or not.
  2. other factors that we can add to the experts’ comments to help Wikipeidians.

I.yeckehzaare (talk) 17:20, 4 August 2014 (UTC)[reply]

Discussion

edit

I changed the name of the bot as advised by xaosflux. I.yeckehzaare (talk) 02:38, 4 August 2014 (UTC)[reply]

Thank you so much for your comments. I revised the code on GitHub. This new version only uses PyWikiBot. I will appreciate your comments and advise to improve our bot and the experiment. I.yeckehzaare (talk) 04:50, 16 August 2014 (UTC)[reply]
  • I like this concept. Do you have a test plan? Also, remember that due to the small and declining number of Wikipedia editors, professors should understand that their comments may take a long time to be noticed by editors if the professors comment on low traffic articles. --Pine✉ 19:30, 13 August 2014 (UTC)[reply]
Thank you so much for your nice comment. Yes, we have a project plan. As mentioned above, we are a group of researchers at the University of Michigan, Carnegie Mellon University and the University of Pittsburgh. For the first step, we want to identify factors which can motivate experts to go through specific Wikipedia articles which are related to a number of their publications and provide us with their feedback about the accuracy and completeness of the content and references of the article. If we were successful in the first step, then we will try to identify ways which can help domain experts to directly edit Wikipedia pages, and how to facilitate communication between Wikipedians who are interested in a specific domain and experts in that domain. We will really appreciate your help and advise in this regard. I.yeckehzaare (talk) 20:14, 13 August 2014 (UTC)[reply]
Hi,
  • What is your test plan for the bot software, especially prior to testing it on English Wikipedia with live researchers?
  • Having domain "experts" directly edit articles has caused problems in the past like POV-pushing, so attempting to encourage "expert" article editing on a large scale is likely to require very careful management. I would treat that as a separate project.
  • Encouraging researchers to post comments on talk pages is less risky and may be very favorably recieved if the information provided is highly relevant, brief, and well sourced to freely available docunents.
  • Are you coordinating with professional groups like the APA, and relevant Wikiprojects?
  • Proceeding gradually is important so that early problems are at small scale.
--Pine✉ 05:33, 14 August 2014 (UTC)[reply]
Hello,
Thank you so much for your helpful comments and suggestions.
1- According to MediaWiki guidelines, unfortunately we are not able to test the bot now, but as soon as receiving approval for a trial period, we will test the Bot on our own research team. Our research team includes 4 professors from 3 universities, and they have a number of highly cited publications. In the trial period, the program will send emails to these professors asking them to review related Wikipedia pages to their recent publications, and will post the comments to the corresponding Talk pages. In this way, we will test both the Bot and the Wikipedians' reactions to these posts, especially those Wikipedians who are active on corresponding pages, having them in their watchlists.
2- This is one of the reasons why we don't want to ask experts to directly edit Wikipedia pages at the moment. Our most recent vision is to find factors which can motivate experts to just review specific Wikipedia pages, and provide us with their feedback. Then our program sends the feedback to corresponding Talk pages, which will help active Wikipedians on those Talk pages to take advantage of those comments and edit the main articles based on their own conventions.
3- Unfortunately at the moment we cannot predict the experts' reactions to our emails. After the trial period and finalizing the tests on the functionality of the program and the Bot, we will start our "pilot study" which will be on about 1000 experts in fields of Economics, Psychology and Information Sciences. The objective of the pilot study is to identify: a) How to communicate with experts via email to receive the most number of responses from them; b) Which factors in the experts' feedback are more helpful for Wikipedians; c) How to format the study page to receive the most appropriate feedback from the experts which can be helpful for Wikipedians.
4- Two of the professors on our team, Professor Robert Kraut from Carnegie Mellon University and Professor Rosta Farzan from the University of Pittsburgh, have collaborated with the APA in their previous research on Wikipedia. We plan to continue working with APA for our project. Professor Yan Chen and Professor Qiaozhu Mei are in the process of seeking support from AEA and ACM communities.
All Wikipedia pages that our pilot program selects to propose to the experts are categorized under Economics and Psychology WikiProjects. In this way, we are trying to ask experts' feedback only on those pages which are important for members of these WikiProjects.
5- We have defined a timeline of 3 years for this research project, and we are going to spend a fair amount of time and effort in the pilot study period to investigate the most appropriate factors that we should incorporate in our main study.
Thank you so much for your concern about this project. We will really appreciate your comments and advice in this regard.
I.yeckehzaare (talk) 04:50, 16 August 2014 (UTC)[reply]


I was alerted about this BRFA from a fellow researcher, and thought I'd voice my support (as a fellow researcher and bot operator). The follow up on this discussion from I.yeckehzaare is a positive sign, and I appreciate that the code has been put on GitHub. In light of the question above about bot frameworks, I went and reviewed the code, and have a few thoughts:

  • Are we certain that the break criteria in the while-loop in the AddNewSectionToTalkPage() function does not cause an infinite loop? (meaning the bot gets stuck trying to log in)
  • What happens if the bot fails to log in correctly?
  • There is no code to handle throttle requests from the API.

The reason I bring up the latter is that I do not know what the magnitude of the scholarsList list, will it try to post a handful of comments at a time? On the order of "tens"? If it's more than that, a bot framework might be useful since at least some of them transparently handle API throttling. I use pywikibot myself, but there are others as APerson referenced. Regards, Nettrom (talk) 19:34, 14 August 2014 (UTC)[reply]

Thank you so much for your comments. I revised the code on GitHub. This new version only uses PyWikiBot. I will appreciate your comments and advise to improve our bot and the experiment. I.yeckehzaare (talk) 04:50, 16 August 2014 (UTC)[reply]
Updating here after I did a closer review of the code. I submitted a pull request with some changes, it's mostly Unicode strings to not trigger any Unicode errors, and proposes a few other changes. Apart from those the code looked good to go. With regards to the research it sounds like you'll have a test run with the PIs, so that should give some indication of how things go. I'm still supportive of this project and hope someone from BAG can stop by and comment soon so things can get moving. Regards, Nettrom (talk) 16:47, 26 August 2014 (UTC)[reply]
I really appreciate your help with improving my code. I.yeckehzaare (talk) 18:22, 26 August 2014 (UTC)[reply]

@Xaosflux: I have not followed the discussion of this one. You take over here :) -- Magioladitis (talk) 08:53, 4 September 2014 (UTC)[reply]

@I.yeckehzaare: is the bot able to run now? Is the bot ready for a trial? -- Magioladitis (talk) 14:40, 14 September 2014 (UTC)[reply]

Thank you for your concern about our bot. As Nettrom mentioned "the code looked good to go." The only thing we need is the permission to be able to run the bot, and start the trial. I.yeckehzaare (talk) 15:31, 14 September 2014 (UTC)[reply]

@I.yeckehzaare: is there going to be a limit of emails sent per day? -- Magioladitis (talk) 08:23, 15 September 2014 (UTC)[reply]

No. We are going to send many emails, because previous studies showed that as most researchers are too busy, response rate is very low, or no response in many cases. The bot will submit the response on the corresponding Talk page as soon as receiving it, however because of low response rate and the fact that we are gathering email addresses gradually, I don't think if the bot will send more than 50 responses per day. It's noteworthy to mention that over the trial period, we are not going to send emails to study subjects, but we will test the bot only on the Professors and researchers in our own team.

((BotTrial|days=4)) @I.yeckehzaare: OK. I give 4 days to demonstrate how this things works. -- Magioladitis (talk) 12:22, 17 September 2014 (UTC)[reply]

Hi Magioladitis I really appreciate your help with giving us 4 days of trial period. However, I was not able to test the bot before having the trial permission. As soon as receiving the permission, I tried to incorporate the bot in the rest of the project, and deploy it on Google App Engines. Unfortunately I was not able to install PyWikibot on Google App Engine. Then I spent two nights to migrate the whole project to Django framework and deploy it on Heroku or AWS, but I definitely need more time to migrate the whole project, install PyWikibot and other required packages and test everything. Then, as planned, I will ask the professors in our team to test the program. I will really appreciate it if you give me more trial time to fix all the problems and test the bot in action. I.yeckehzaare (talk) 05:26, 20 September 2014 (UTC)[reply]
I.yeckehzaare tell me when you are ready to start demonstration and I ll give permission to start. After the trial period we will need some statistics from you. Thanks, Magioladitis (talk) 13:41, 22 September 2014 (UTC)[reply]
Magioladitis thank you so much. I will really appreciate it if you give me permission to test the bot on a sample talk page to make sure everything works before asking professors to start using the trial version, and post comments on real talk pages. I tested the bot on my own talk page, and it works, but as I am meticulous about any possibility of bug in my probgram or the bot, I will really appreciate it if you let me test it on a sample talk page, not a user talk page, to test it in different conditions, to make sure everything is OK before the trial test. I.yeckehzaare (talk) 16:48, 22 September 2014 (UTC)[reply]
I.yeckehzaare Permission granted. Please notify me about the results. -- Magioladitis (talk) 16:49, 22 September 2014 (UTC)[reply]
Magioladitis This is a great help. I really appreciate it. I.yeckehzaare (talk) 17:42, 22 September 2014 (UTC)[reply]
  • The bot should make useful citations, at least a journal or chapter title, plus the publication and date. The title alone is not sufficient, a reader with some familiarity on the topic can have an idea about the professor's contributions to the field and topic with the journal title. 75.166.207.124 (talk) 11:25, 12 November 2014 (UTC)[reply]
    Basically citations that our bot adds to each post are just to show related publications by the domain expert who has reviewed the page. They are not the main part of the post. The main part is the comment that the domain expert has provided for us to inform Wikipedians about. I think adding complete citations will result in long posts which might not be useful. A more efficient method can be linking the title of the citation to the Google Scholar page of the publication. This way, those who want to know more about each publication, can easily go to the Google Scholar page by clicking the link. I will be thankful if you make me aware of your ideas about this proposal or any kind of improvement to the bot. I.yeckehzaare (talk) 14:36, 12 November 2014 (UTC)[reply]
    Then add article title, year, and publication. But to make it so unacademic as to require so much work to learn about the articles, when a year and journal or book title, so little information that would provide so much, defeats the purpose of treating people as experts in their area. Then link it to Google scholar, but don't make it so obscure and disrespectful of academia when it is academics contributing. Wikipedia has enough problems with citations without creating a bot to correspond with academic experts that makes them appear in agreement with lame citations. --71.212.238.206 (talk) 19:49, 12 November 2014 (UTC)[reply]
    Sure. In the future posts, the bot will add the appropriate citations. I.yeckehzaare (talk) 04:11, 13 November 2014 (UTC)[reply]
  • I.yeckehzaare, I love the idea of the bot but think that the messages it leaves on the article talk pages could use a bit of formatting. For example, the quote from the academic expert could go inside a box or something (e.g. {{Quote}} or {{Quotation}}). Also, the list of references should probably be formatted like a list (with *s). APerson (talk!) 01:48, 1 December 2014 (UTC)[reply]
    APerson Thank you so much for your constructive suggestions. We will definitely use quote for the experts' comments. In terms of the list of references, do you mean the citations that our system proposes automatically, or the references that experts cite in their comments? In case of the latter one, it is very difficult to ask them to format their references. Basically we are trying to make edit cost as low as possible for the experts. If an expert knows and is willing to use appropriate formatting for their citations, they will definitely edit Wikipedia articles directly without being asked by our bot. Our interviews showed that many domain experts including professors don't like to spend time dealing with the Wikipedia modeling language, or even activating the visual editor. Asking them to type their feedback in a simple textbox increases the probability of their contribution. I.yeckehzaare (talk) 01:06, 2 December 2014 (UTC)[reply]
    I.yeckehzaare, I was talking about only the former. I completely agree that the burden on the experts should be as low as possible. APerson (talk!) 01:40, 2 December 2014 (UTC)[reply]
  • @I.yeckehzaare: Is the plan to post to all kinds of articles or just to articles with a high edit rate or a high talk page edit rate? Alternatively, would there be a minimum time between posts to the same article's talk page? I am a bit concerned that if this ran long-term, it could clutter barely-used Talk: pages in the same way that the Singpost clutters some barely-used User talk: pages. – Philosopher Let us reason together. 23:22, 1 December 2014 (UTC)[reply]
    Philosopher Thank you for your comment. In our pilot study, we have not defined any restriction on the Wikipedia pages in terms of number of editors or length. Also, similar previous studies show that the response rate is very low, which by itself, obviates the need to restrict the speed or number of posts to Talk pages. However, we have restricted the bot from posting the comments automatically, and all posts are going to be checked by our administrators before being posted to Talk pages. I.yeckehzaare (talk) 01:06, 2 December 2014 (UTC)[reply]
  • This bot really needs a better edit summary. The current one is way too generic and says absolutely nothing of substance. It really doesn't have to repeat the bot's name, and needs to actually describe what it is doing:Jay8g [V•T•E] 01:48, 5 December 2014 (UTC)[reply]
    Jay8g What do you think about copying the title of the post to the edit summary? Should I get rid of the bot name from the edit summary? I.yeckehzaare (talk) 17:33, 5 December 2014 (UTC)[reply]
    Definitely get rid of the bot's name. Copying the title would be OK, but not ideal. Perhaps Adding comment from [name] on this article or similar?:Jay8g [V•T•E] 01:05, 6 December 2014 (UTC)[reply]

@I.yeckehzaare: have the issues raised been fixed? -- Magioladitis (talk) 08:09, 24 December 2014 (UTC)[reply]

@Magioladitis: So sorry for my very late response. Although I was checking this page everyday, I did not see your message. Yes, it's changed based on the feedback received from people in this forum. However, I am not going to change the 3 posts which have already been posted to talkpages during our pilot study, but those messages that we are going to send in the future will be generated by the updated program. I.yeckehzaare (talk) 23:07, 15 January 2015 (UTC)[reply]

Trial complete. -- Magioladitis (talk) 21:43, 20 January 2015 (UTC)[reply]

 Approved. -- Magioladitis (talk) 20:56, 28 January 2015 (UTC)[reply]

I.yeckehzaare bot approved. -- Magioladitis (talk) 20:59, 28 January 2015 (UTC)[reply]

The above discussion is preserved as an archive of the debate. Please do not modify it. To request review of this BRFA, please start a new section at WT:BRFA.
Retrieved from "https://en.wikipedia.org/w/index.php?title=Wikipedia:Bots/Requests_for_approval/ExpertIdeasBot&oldid=1083678544"