Extension:QuickSurveys/Deploying surveys

The process to get a survey from idea to deployed can seem a bit complicated at first, so this page is here to explain things step-by-step. Additionally, a proposal for a new deployment process is included below.

Current process

The current method for running a survey has 8 steps:

1. Create a task in Phabricator describing the survey and its intended date range for running. Include the configuration parameters for the survey with the names of message keys. Up-to-date configuration options can be found in code. Example:

   'wgQuickSurveysConfig' => [
   	'enwiki' => [
   		'name' => 'example-survey',
   		'enabled' => true,
   		// question on top of form
   		// e.g. Answer three questions and help us improve Wikipedia
   		'question'=> 'example-survey-message',
   
   		// description bellow question (optional)
   		// e.g. Visit the link to access the form. Your opinion matters. 
   		'description' => 'example-survey-description',
   		
   		// external link (must be https)
   		// e.g. https://docs.google.com/forms/...
   		'link' => 'example-survey-link',
   
   		// privacy policy
   		// e.g. [https://wikimediafoundation.org/wiki/Survey_Privacy_Statement Our Privacy Statement]. Survey data handled by a third party. 
   		'privacyPolicy' => 'example-survey-privacy',
   
   		// percentage of users that will see the survey
   		'coverage' => '0.00005', // 5 out 100K (0.005%)
   		'platforms' => [
   			'desktop' => [ 'stable, beta, alpha' ],
   			'mobile' => [ 'stable' ]
   		]
   	]
   ]
   

1. Create one wiki page for each message key within the MediaWiki namespace of the project on which the survey will run. For example, "Reader-segmentation-1-description" corresponds to MediaWiki:Reader-segmentation-1-description. In order to do this, you'll need someone with permission to edit pages within the MediaWiki namespace (usually someone with "staff" rights across the projects. We think this is the list of such people.).
   1. Alternatively, you can update Extension:WikimediaMessages. This is particularly useful if you need to send the survey to multiple languages and leverage the translatewiki community for translations.
2. To test the survey, write a patch (or coordinate with an engineer who can write a patch) to the "mediawiki-config" repository adding the survey configuration to the "wgQuickSurveysConfig" variable. We suggest enabling the survey on the beta cluster for 100% of users to check it's working before enabling everywhere. Schedule the patch for a backport. See https://wikitech.wikimedia.org/wiki/Deployments for more information on how to deploy.
3. To enable the survey in production, write another patch in the "mediawiki-config" repository. See https://gerrit.wikimedia.org/r/#/c/254908/2/wmf-config/InitialiseSettings.php for an example.
4. Ping relevant teams to make sure they know about a QuickSurveys deploy. QuickSurveys have a high JavaScript penalty so it's important to ping performance team and reading web so that they are not surprised by a deploy. You can ping them via email, scrum of scrums or phabricator (tag "readers-web-backlog" and "performance-team")!
5. Be present during the deploy and verify the survey went live by checking the wgEnabledQuickSurveys variable in the JavaScript console.
   • Surveys can also be manually triggered for verification. See Extension:QuickSurveys#How_to_load_a_specific_survey
6. Write a patch to disable the survey by removing the previously-added config.
7. Schedule the disable patch for a backport on the proposed ending date.
8. Be present during the deployment window. Verify the survey has been disabled via the "wgEnabledQuickSurveys" variable once again.

Proposed process

Background

Since the above process can be quite cumbersome, requires engineer involvement, necessitates deployments, and is resistant to survey tweaks, some of the Reading Web team met up with some engineers from Fundraising Tech in order to discuss the possibility of using CentralNotice (or some other service) for QuickSurveys in the future. The meeting went extremely well and the result was a general consensus that CentralNotice could easily serve the needs of QuickSurveys with very few engineering changes required.

One of the first topics discussed was a sort of checklist of all the areas that needed consideration:

1. Deployment
2. Translations
3. Show survey on first impression
4. Alleviate "on-wiki pain" (staff rights, one page per message per project, etc.)

QuickSurveys in CentralNotice

Before delving into the topics, a brief overview of QuickSurveys was presented. One of the immediate highlights was the fact that it's currently only enabled on English Wikipedia due to the painful creation process. After the process had been fully explained and questions answered, Fundraising Tech gave an overview of CentralNotice, at which point each concern from the list was discussed. The proposed premise was to define surveys as CentralNotice banners and programmatically grab the content from the banner to generate a survey at runtime. Since arbitrary code can be entered into banners, this would involve simply including a JSON blob and updating the QuickSurveys extension to read from said JSON instead of MediaWiki PHP.

Deployment

All CentralNotice banner campaigns are run through the CentralNotice Meta Admin UI (https://meta.wikimedia.org/wiki/Special:CentralNotice). Anyone can apply for the necessary rights to run campaigns and no deploys are needed; a time range is specified for each campaign within the interface and the rest is handled automatically.

Translations

The CentralNotice Admin UI has complete integration with the Translate extension on Meta-wiki, meaning surveys could easily be run across multiple projects without manually defining each one.

Show survey on first impression

CentralNotice is able to decide whether to show a banner (and which banner to show) based on

• Sampling rate
• Picking randomly from a set of defined banners
• Device target (mobile vs. desktop)
• Displaying only once (on first impression) or more often
• Logged-in status

QuickSurveys currently only handles the first three (first impression is also supported, but the survey will continue to show on every page until dismissed or completed). Research has already expressed interest in running surveys that would require the last two criteria.

Alleviate "on-wiki pain"

Most importantly, use of the CN Admin UI means no more creating wiki pages for every single message key; a cumbersome process that not only pollutes the MediaWiki namespace, but also just doesn't quite make sense (even blank messages require pages). MediaWiki template pages are still usable for defining banners, however, which would enable support for more customizeable QuickSurveys in the future, if ever desired.

Concerns

CentralNotice for QuickSurveys is not without its drawbacks, naturally. Please feel free to add to this section if more concerns arise.

• CN can only display one banner from one campaign at a time. The banner selected is based on priority (and then selected at random based on the sampling rate of all campaigns/banners with the same priority). The other side of this coin, however, is that blackout periods for campaigns such as fundraising would be handled automatically.
• The process for getting "Central-Notice-Admin" rights on metawiki is not very well-documented.
• A user can at max only ever see one banner (survey) from a campaign, no matter how many are defined. Basically, once they're bucketed into one survey, that's the only one they're getting. This is a deviation from the status quo, which allows users to potentially be bucketed into multiple surveys. This is easily circumvented by limiting campaigns to only having one banner (survey).

Process

Creating a new survey with CentralNotice would involve the following steps:

1. Garner "Central-Notice-Admin" rights on metawiki. This would only need to be done once per person.
2. Create a new campaign on the CN Meta Admin UI.
3. Define a banner (survey). This would most likely be a JSON blob containing the information listed in step 1 of the current process.
4. Set the metadata for the campaign (sampling rate, runtime, etc.) and enable it.

Conclusion

CentralNotice can help drastically simplify the process of getting a QuickSurvey from "idea" to "live." Engineer involvement would be reduced to a minimum and SWAT deployments would be completely eliminated. This would hopefully encourage more surveys so we can gather more information about how our users use the Wikimedia projects and specific features therein. There are some drawbacks, but most seem to have easy workarounds.

Next Steps

• Reading Web team (maintainers of the extension) discuss the proposal over email
• If there is general consensus, carve time out for updating the extension in line with yearly and quarterly goals
• If not, reject the proposal and document the rationalization on-wiki