View source for Extension:SpamBlacklist/hi

<languages/>
{{bundled|1.21}}
{{Extension
|status    = stable 
|type      = page action
|username  = Tim Starling
|author    = Tim Starling
|image     = SpamBlacklist.gif
|description = <span lang="en" dir="ltr" class="mw-content-ltr">Provides a regex-based spam filter</span>
|version   = <span lang="en" dir="ltr" class="mw-content-ltr">Continuous updates</span>
|php =
|update    = 
|mediawiki = 1.31+
|compatibility policy = rel
|download  = {{WikimediaDownload|SpamBlacklist|phab=ESPB}}
|readme    = {{git file|action=raw|project=mediawiki/extensions/SpamBlacklist|file=README|text=README}}
|changelog = 
|phabricator = MediaWiki-extensions-SpamBlacklist
|pagedrive1 = true
|rights =  * sboverride
* spamblacklistlog
}}
{{Note|1=<span lang="en" dir="ltr" class="mw-content-ltr">A proposal to rename this extension is discussed at {{Phab|T254649}}.</span>}}

<div lang="en" dir="ltr" class="mw-content-ltr">
The '''SpamBlacklist''' extension prevents edits that contain URLs whose domains match regular expression patterns defined in specified files or wiki pages and registration by users using specified email addresses.
</div>

<span lang="en" dir="ltr" class="mw-content-ltr">When someone tries to save a page, this extension checks the text against a (potentially very large) list of illegal host names.</span>
<span lang="en" dir="ltr" class="mw-content-ltr">If there is a match, the extension displays an error message to the user and refuses to save the page.</span>

<div lang="en" dir="ltr" class="mw-content-ltr">
==Installation and setup==
</div>

<div lang="en" dir="ltr" class="mw-content-ltr">
===Installation===
</div>
{{ExtensionInstall
|repo-name=SpamBlacklist
|custom-steps=* [[#Setting the block list|<span lang="en" dir="ltr" class="mw-content-ltr">Configure the block list at your convenience</span>]]
|registration=required
}}

<div lang="en" dir="ltr" class="mw-content-ltr">
===Setting the block list===
</div>

<div lang="en" dir="ltr" class="mw-content-ltr">
The following local pages are always used, whatever additional sources are listed:
</div>
* [[MediaWiki:Spam-blacklist]]
* [[MediaWiki:Spam-whitelist]]
* [[MediaWiki:Email-blacklist]]
* [[MediaWiki:Email-whitelist]]

<span lang="en" dir="ltr" class="mw-content-ltr">The default additional source for a block list of forbidden URLs is the Wikimedia spam block list on Meta-Wiki, at '''[[m:Special:MyLanguage/Spam blacklist|m:Spam blacklist]]'''.</span>
<span lang="en" dir="ltr" class="mw-content-ltr">By default, the extension uses this list, and reloads it once every 10-15 minutes.</span>
<span lang="en" dir="ltr" class="mw-content-ltr">For many wikis, using this list will be enough to block most spamming attempts.</span>
<span lang="en" dir="ltr" class="mw-content-ltr">However, since the Wikimedia block list is used by a diverse group of large wikis with hundreds of thousands of external links, it is comparatively conservative in the links it blocks.</span>

<div lang="en" dir="ltr" class="mw-content-ltr">
The Wikimedia spam block list can only be edited by administrators; but you can suggest modifications to the block list at [[m:Talk:Spam blacklist]].
</div>

<div lang="en" dir="ltr" class="mw-content-ltr">
You can add other bad URLs on your own wiki. List them in the global variable <code>$wgBlacklistSettings</code> in {{ll|Manual:LocalSettings.php|LocalSettings.php}}. See examples below.
</div>  

<div lang="en" dir="ltr" class="mw-content-ltr">
<code>$wgBlacklistSettings</code> is an two level array. Top level key is <code>spam</code> or <code>email</code>. They take an array with each value containing either a URL, a filename or a database location.
</div>

<div lang="en" dir="ltr" class="mw-content-ltr">
If you use <code>$wgBlacklistSettings</code> in "LocalSettings.php", the default value of "<nowiki>[[m:Spam blacklist]]</nowiki>" will no longer be used - if you want that block list to be accessed, you will have to add it in manually, see examples below.
</div>

<div lang="en" dir="ltr" class="mw-content-ltr">
Specifying a database location allows you to draw the block list from a page on your wiki.
</div>

<div lang="en" dir="ltr" class="mw-content-ltr">
The format of the database location specifier is ">DB: [db name] [title]". [db name] should exactly match the value of <code>$wgDBname</code> in {{ll|Manual:LocalSettings.php|LocalSettings.php}}.
</div>
<span lang="en" dir="ltr" class="mw-content-ltr">You should create the required page name [title] in the default namespace of your wiki.</span>
<span lang="en" dir="ltr" class="mw-content-ltr">If you do this, it is strongly recommended that you protect the page from general editing.</span>
<span lang="en" dir="ltr" class="mw-content-ltr">Besides the obvious danger that someone may add a regex that matches everything, please note that an attacker with the ability to input arbitrary regular expressions may be able to generate [[w:Segmentation fault|segfaults]] in the [[w:PCRE|PCRE library]].</span>

; <span lang="en" dir="ltr" class="mw-content-ltr">Examples</span>

<div lang="en" dir="ltr" class="mw-content-ltr">
If you want to, for instance, use the English-language Wikipedia's spam block list in addition to the standard Meta-Wiki one, you could call the following in {{ll|Manual:LocalSettings.php|LocalSettings.php}}, AFTER <code>wfLoadExtension( 'SpamBlacklist' );</code> call:
</div>

<syntaxhighlight lang="php">
$wgBlacklistSettings = [
	'spam' => [
		'files' => [
			"https://meta.wikimedia.org/w/index.php?title=Spam_blacklist&action=raw&sb_ver=1",
			"https://en.wikipedia.org/w/index.php?title=MediaWiki:Spam-blacklist&action=raw&sb_ver=1"
		],
	],
];
</syntaxhighlight>
<div lang="en" dir="ltr" class="mw-content-ltr">
Here is an example of an entirely local set of block lists: the administrator is using the {{ll|Extension:SpamBlacklist/update script|update script}} to generate a local file called "wikimedia_blacklist" that holds a copy of the Meta-Wiki blacklist, and has an additional block list on the wiki page "My spam block list":
</div>

<syntaxhighlight lang="php">
$wgBlacklistSettings = [
	'spam' => [
		'files' => [
			"$IP/extensions/SpamBlacklist/wikimedia_blacklist", // Wikimedia's list
			// database, title
			'DB: wikidb My_spam_block_list',    
		],
	],
];
</syntaxhighlight>

<div lang="en" dir="ltr" class="mw-content-ltr">
===Logging===
</div>
<span lang="en" dir="ltr" class="mw-content-ltr">By default, the extension does not log hits to a spam blacklist log.</span>
<span lang="en" dir="ltr" class="mw-content-ltr">To enable logging set <syntaxhighlight lang=php inline>$wgLogSpamBlacklistHits = true;</syntaxhighlight>.</span>
<span lang="en" dir="ltr" class="mw-content-ltr">You can use the <code>spamblacklist</code> user right to control access to the logs.</span>
<span lang="en" dir="ltr" class="mw-content-ltr">Every signed-in user can view the logs by default.</span>


<div lang="en" dir="ltr" class="mw-content-ltr">
===Issues===
</div>

<div lang="en" dir="ltr" class="mw-content-ltr">
====Backtrack Limit====
</div>

<div lang="en" dir="ltr" class="mw-content-ltr">
If you encounter issues with the block list, you may want to increase the backtrack limit.
</div>
<span lang="en" dir="ltr" class="mw-content-ltr">However on the other hand, this can reduce your security against DOS <ref name="dos">[[w:en:Denial-of-service_attack|Denial of Service]] (DOS)</ref> attacks, as the backtrack limit is a performance limit:</span>

<syntaxhighlight lang="php">
// Bump the Perl Compatible Regular Expressions backtrack memory limit
// (PHP 5.3.x default, 1000K, is too low for SpamBlacklist)
ini_set( 'pcre.backtrack_limit', '8M' );
</syntaxhighlight>

<div lang="en" dir="ltr" class="mw-content-ltr">
====Hardened Wikis====
</div>
<span lang="en" dir="ltr" class="mw-content-ltr">The SpamBlacklist will not allow editing if the wiki is hardened.</span>
<span lang="en" dir="ltr" class="mw-content-ltr">Hardening includes limiting <code>open_basedir</code> so that <code>curl</code> is not on-path, and setting <code>allow_url_fopen=Off</code> in <code>php.ini</code>.</span>

<span lang="en" dir="ltr" class="mw-content-ltr">In the hardened case, SpamBlacklist will cause an exception when Guzzle attempts to make a network request.</span>
<span lang="en" dir="ltr" class="mw-content-ltr">The Guzzle exception message is, ''GuzzleHttp requires cURL, the allow_url_fopen ini setting, or a custom HTTP handler.''</span>

<div lang="en" dir="ltr" class="mw-content-ltr">
===Safe list===
</div>
<span lang="en" dir="ltr" class="mw-content-ltr">A corresponding safe list can be maintained by editing the [[MediaWiki:Spam-whitelist]] page.</span>
<span lang="en" dir="ltr" class="mw-content-ltr">This is useful if you would like to override certain entries from another wiki's block list that you are using.</span>
<span lang="en" dir="ltr" class="mw-content-ltr">Wikimedia wikis, for instance, sometimes use the spam block list for purposes other than combating spam.</span>

<span lang="en" dir="ltr" class="mw-content-ltr">It is questionable how effective the Wikimedia spam block lists are at keeping spam off of third-party wikis.</span>
<span lang="en" dir="ltr" class="mw-content-ltr">Some spam might be targeted only at Wikimedia wikis, or only at third-party wikis, which would make Wikimedia's blacklist of little help to said third-party wikis in those cases.</span>
<span lang="en" dir="ltr" class="mw-content-ltr">Also, some third-party wikis might prefer that users be allowed to cite sources that Wikipedia does not allow. Sometimes what one wiki considers useless spam, another wiki might consider useful.</span>

<span lang="en" dir="ltr" class="mw-content-ltr">Users may not always realize that, when a link is rejected as spammy, it does not necessarily mean that the individual wiki they are editing has specifically chosen to ban that URL.</span>
<span lang="en" dir="ltr" class="mw-content-ltr">Therefore, wiki system administrators may want to edit the {{ll|Manual:System messages}} at [[MediaWiki:Spamprotectiontext]] and/or [[MediaWiki:Spamprotectionmatch]] on your wiki to invite users to make suggestions at [[MediaWiki talk:Spam-whitelist]] for pages that should be added by a {{ll|Manual:Administrators}} to the safe list. For example, you could put, for MediaWiki:Spamprotectiontext:</span>

:<span lang="en" dir="ltr" class="mw-content-ltr">The text you wanted to save was blocked by the spam filter.</span> <span lang="en" dir="ltr" class="mw-content-ltr">This is probably caused by a link to a blacklisted external site.</span> <span lang="en" dir="ltr" class="mw-content-ltr"><nowiki>{{SITENAME}}</nowiki> maintains <nowiki>[[MediaWiki:Spam-blacklist|its own block list]]</nowiki>; however, most blocking is done by means of <nowiki>[[metawikimedia:Spam-blacklist|Meta-Wiki's block list]]</nowiki>, so this block should not necessarily be construed as an indication that <nowiki>{{SITENAME}}</nowiki> made a decision to block this particular text (or URL).</span> <span lang="en" dir="ltr" class="mw-content-ltr">If you would like this text (or URL) to be added to <nowiki>[[MediaWiki:Spam-whitelist|the local spam safe list]]</nowiki>, so that <nowiki>{{SITENAME}}</nowiki> users will not be blocked from adding it to pages, please make a request at <nowiki>[[MediaWiki talk:Spam-whitelist]]</nowiki>.</span> <span lang="en" dir="ltr" class="mw-content-ltr">A <nowiki>[[Project:Sysops|sysop]]</nowiki> will then respond on that page with a decision as to whether it should be listed as safe.</span>

<div lang="en" dir="ltr" class="mw-content-ltr">
==Notes==
</div>

<div lang="en" dir="ltr" class="mw-content-ltr">
* This extension examines only new external links added by wiki editors.
</div> <span lang="en" dir="ltr" class="mw-content-ltr">To check user agents, add {{ll|Extension:AkismetKlik|Akismet}}</span> <span lang="en" dir="ltr" class="mw-content-ltr">As the various tools for [[Special:MyLanguage/Manual:combating spam|combating spam]] on MediaWiki use different methods to spot [[Special:MyLanguage/Extension:AbuseFilter|abuse]], the safeguards are best used in combination.</span>
<div lang="en" dir="ltr" class="mw-content-ltr">
* Users with the <code>sboverride</code> can override the block list and add blocked links to pages. By default this right is only given to bots.
</div>

<div lang="en" dir="ltr" class="mw-content-ltr">
==Usage==
</div>

<div lang="en" dir="ltr" class="mw-content-ltr">
===Syntax===
</div>

<div lang="en" dir="ltr" class="mw-content-ltr">
If you would like to create a block list of your own, or modify an existing one, here is the syntax:
</div>

<span lang="en" dir="ltr" class="mw-content-ltr">Everything on a line after a '#' character is ignored (for comments).</span>
<span lang="en" dir="ltr" class="mw-content-ltr">All other strings are regex fragments which will only match inside URLs.</span>

<div lang="en" dir="ltr" class="mw-content-ltr">
;Notes:
</div>

<div lang="en" dir="ltr" class="mw-content-ltr">
* Do not add "http://"; this would fail, since the regex will match ''after'' "http://" (or "https://") inside URLs.
</div>
<div lang="en" dir="ltr" class="mw-content-ltr">
* Furthermore "www" is unneeded, since the regex will match any subdomains.
</div> <span lang="en" dir="ltr" class="mw-content-ltr">By giving "www\." explicitly one can match specific subdomains.</span>
<div lang="en" dir="ltr" class="mw-content-ltr">
* The <code>(?<=//|\.)</code> and <code>$</code> anchors match the beginning and end of the domain name, ''not'' the beginning and end of the URL.
</div> <span lang="en" dir="ltr" class="mw-content-ltr">The regular anchor <code>^</code> won't be of any use.</span>
<div lang="en" dir="ltr" class="mw-content-ltr">
* Slashes don't need to be escaped by backslashes, this will be done automatically by the script.
</div>
* <span lang="en" dir="ltr" class="mw-content-ltr">The spam blacklist functions prior to abuse filters, so blacklisted domains will not show in the entries in abuse filter log ([[special:abuselog]]), and will only show in ([[special:log/spamblacklist]]).</span>

; <span lang="en" dir="ltr" class="mw-content-ltr">Example</span>:

<div lang="en" dir="ltr" class="mw-content-ltr">
The following line will block all URLs that contain the string "example.com", except where it is immediately preceded or followed by a letter or a number.
</div>

 <nowiki>\bexample\.com\b</nowiki>

<div lang="en" dir="ltr" class="mw-content-ltr">
These are blocked:
</div>

* <nowiki>http://www.example.com</nowiki>
* <nowiki>http://www.this-example.com</nowiki>
* <nowiki>http://www.google.de/search?q=example.com</nowiki>

<div lang="en" dir="ltr" class="mw-content-ltr">
These are not blocked:
</div>

* <nowiki>http://www.goodexample.com</nowiki>
* <nowiki>http://www.google.de/search?q=example.commodity</nowiki>

<div lang="en" dir="ltr" class="mw-content-ltr">
===Performance===
</div>
<span lang="en" dir="ltr" class="mw-content-ltr">The extension creates a single regex statement which looks like <code><nowiki>/https?:\/\/[a-z0-9\-.]*(line 1|line 2|line 3|....)/Si</nowiki></code> (where all slashes within the lines are escaped automatically).</span>
<span lang="en" dir="ltr" class="mw-content-ltr">It saves this in a small "loader" file to avoid loading all the code on every page view.</span>
<span lang="en" dir="ltr" class="mw-content-ltr">Page view performance will not be affected even if you're not using a [[w:Byte-code|bytecode]] cache although using a cache is strongly recommended for any MediaWiki installation.</span>

<span lang="en" dir="ltr" class="mw-content-ltr">The regex match itself generally adds an insignificant overhead to page saves (on the order of 100ms in our experience).</span>
<span lang="en" dir="ltr" class="mw-content-ltr">However, loading the spam file from disk or the database, and constructing the regex, may take a significant amount of time depending on your hardware.</span>
<span lang="en" dir="ltr" class="mw-content-ltr">If you find that enabling this extension slows down saves excessively, try installing a supported bytecode cache.</span>
<span lang="en" dir="ltr" class="mw-content-ltr">This extension will cache the constructed regex if such a system is present.</span>

<span lang="en" dir="ltr" class="mw-content-ltr">If you're sharing a server and cache with several wikis, you may improve your cache performance by modifying getSharedBlacklists and clearCache in SpamBlacklist_body.php to use {{wg|SharedUploadDBname}} (or a specific DB if you do not have a shared upload DB) rather than {{wg|DBname}}.</span>
<span lang="en" dir="ltr" class="mw-content-ltr">Be sure to get all references! The regexes from the separate MediaWiki:Spam-blacklist and MediaWiki:Spam-whitelist pages on each wiki will still be applied.</span>

<div lang="en" dir="ltr" class="mw-content-ltr">
=== External block list servers (RBL's) ===
</div>

<div lang="en" dir="ltr" class="mw-content-ltr">
In its standard form, this extension requires that the block list be constructed manually. While regular expression wildcards are permitted, and a block list originated on one wiki may be re-used by many others, there is still some effort required to add new patterns in response to spam or remove patterns which generate false-positives.
</div>

<span lang="en" dir="ltr" class="mw-content-ltr">Much of this effort may be reduced by supplementing the spam regex with lists of known domains advertised in spam email.</span>
<span lang="en" dir="ltr" class="mw-content-ltr">The regex will catch common patterns (like "casino-" or "-viagra") while the external block list server will automatically update with names of specific sites being promoted through spam.</span>

<div lang="en" dir="ltr" class="mw-content-ltr">
In the filter() function in includes/SpamBlacklist.php, approximately halfway between the file start and end, are the lines:
</div>

<syntaxhighlight lang="php">
       # Do the match
       wfDebugLog( 'SpamBlacklist', "Checking text against " . count( $blacklists ) .
           " regexes: " . implode( ', ', $blacklists ) . "\n" );
</syntaxhighlight>

<div lang="en" dir="ltr" class="mw-content-ltr">
Directly above this section (which does the actual regex test on the extracted links), one could add additional code to check the external RBL servers <ref name="rbl">[[w:en:Domain_Name_System_blocklist|Realtime Blackhole List]] (RBL)</ref>:
</div>

<syntaxhighlight lang="php">
        # Do RBL checks
        $retVal = false;
        $wgAreBelongToUs = ['l1.apews.org.', 'multi.surbl.org.', 'multi.uribl.com.'];
        foreach( $addedLinks as $link ) {
              $link_url=parse_url($link);
              $link_url=$link_url['host'];
              if ($link_url) {
                   foreach( $wgAreBelongToUs as $base ) {
                        $host = "$link_url.$base";
                        $ipList = gethostbynamel( $host );
                        if( $ipList ) {
                           wfDebug( "RBL match: Hostname $host is {$ipList[0]}, it's spam says $base!\n" );
                           $ip = wfGetIP();
                           wfDebugLog( 'SpamBlacklistHit', "$ip caught submitting spam: {$link_url} per RBL {$base}\n" );
                           $retVal = $link_url . ' (blacklisted by ' . $base .')';
                           wfProfileOut( $fname );
                           return $retVal;
                        }
                   }
              }
        }

        # if no match found on RBL server, continue normally with regex tests...
</syntaxhighlight>

<span lang="en" dir="ltr" class="mw-content-ltr">This ensures that, if an edit contains URLs from already blocked spam domains, an error is returned to the user indicating which link cannot be saved due to its appearance on an external spam block list.</span>
<span lang="en" dir="ltr" class="mw-content-ltr">If nothing is found, the remaining regex tests are allowed to run normally, so that any manually-specified 'suspicious pattern' in the URL may be identified and blocked.</span>

<span lang="en" dir="ltr" class="mw-content-ltr">Note that the RBL servers list just the base domain names - not the full URL path - so <nowiki>http://example.com/casino-viagra-lottery.html</nowiki> will trigger RBL only if "example.com" itself were blocked by name by the external server.</span>
<span lang="en" dir="ltr" class="mw-content-ltr">The regex, however, would be able to block on any of the text in the URL and path, from "example" to "lottery" and everything in between.</span>
<span lang="en" dir="ltr" class="mw-content-ltr">Both approaches carry some risk of false-positives - the regex because of the use of wildcard expressions, and the external RBL as these servers are often created for other purposes - such as control of abusive spam email - and may include domains which are not engaged in forum, wiki, blog or guestbook comment spam ''per se''.</span>

<div lang="en" dir="ltr" class="mw-content-ltr">
==Other spam-fighting tools==
</div>

<div lang="en" dir="ltr" class="mw-content-ltr">
There are various helpful manuals on mediawiki.org on combating spam and other vandalism:
</div>

* {{ll|Anti-spam features}} &mdash; <span lang="en" dir="ltr" class="mw-content-ltr">includes link to the built-in {{ll|Manual:$wgSpamRegex|$wgSpamRegex}} anti-spam mechanism.</span>
* {{ll|Manual:Combating spam|nsp=0}}
* {{ll|Manual:Combating vandalism|nsp=0}}

<div lang="en" dir="ltr" class="mw-content-ltr">
Other anti-spam, anti-vandalism extensions include:
</div>

* {{ll|Extension:ConfirmEdit|nsp=0}}
* [[m:Special:Prefixindex/MediaWiki talk:Spam|<span lang="en" dir="ltr" class="mw-content-ltr">System messages</span>]]

<div lang="en" dir="ltr" class="mw-content-ltr">
==References==
</div>
<references />

{{OnWikimedia}}
{{Used by}}

[[Category:API extensions{{#translation:}}]]
[[Category:Edit extensions{{#translation:}}]]
[[Category:Spam management extensions{{#translation:}}]]
[[Category:Extensions for data exchange with other wikis{{#translation:}}]]