Manual:File cache

From Linux Web Expert

MediaWiki has an optional simplistic scheme for caching the rendered HTML of article pages.

Operation and usage

The file cache serves anonymous users only and they see the same HTML rendered. Logged-in users don't benefit from this cache because their page contains user name, selected skin, etc.

The file cache is enabled by setting $wgUseFileCache and $wgFileCacheDirectory variables in LocalSettings.php: 

$wgUseFileCache = true; // default: false
$wgFileCacheDirectory = "$IP/images/cache"; // default: "{$wgUploadDirectory}/cache" which equals to "$IP/images/cache"

This causes the rendered HTML webpage for each page of the wiki to be stored in an individual file on the hard disk. Any subsequent requests from anonymous users are met not by rendering the page again, but by simply sending the stored HTML version which is on the disk. This saves time.

The generated HTML web text is stored to disk in directories under the $wgFileCacheDirectory and looks somewhat like: 

-rw-r--r-- 1 apache apache 57421 Jan  7 01:58 cache/0/04/P%C3%A1gina_principal.html
-rw-r--r-- 1 apache apache 29608 Jan  4 16:37 cache/1/17/P%C3%A1gina_Riscada.html
-rw-r--r-- 1 apache apache 21592 Jan  3 07:27 cache/1/1c/P%C3%A1gina_Duplicada.html
-rw-r--r-- 1 apache apache 36088 Jan  7 02:03 cache/2/24/P%C3%A1gina_principal_alternativa.html
-rw-r--r-- 1 apache apache 44205 Jan  7 06:10 cache/a/a4/P%C3%A1gina_linkada.html
-rw-r--r-- 1 apache apache 24686 Jan  3 07:27 cache/d/db/P%C3%A1gina_Invertida.html
-rw-r--r-- 1 apache apache 17222 Jan  3 06:28 cache/f/f0/P%C3%A1gina_n%C3%A3o_encontrada.html

In this example, the file cache directory is cache/ and the stored wiki pages are named Página ..., yielding one corresponding Página_....html file for each. These files will be created as users visit articles on the wiki; you can create them all in one go with the rebuildFileCache.php maintenance script.

The file cache tends to cache aggressively; there is no set expiry date for the cached pages and pages are cached unconditionally even if they contain variables, extensions and other changeable output. Some extensions disable file cache for pages with dynamic content.

Consider using $wgUseGzip instead or in combination for much more effective results.

For larger sites, the use of an external cache such as Squid or Varnish is preferable to enabling the file cache.

If the file cache appears not to be working, make sure that the webserver has permission to write to the directory you chose.

Limitations

The file names of the cache files are determined by the according page titles. Depending on the language, which is used in the page titles, some characters in the file names will have to be encoded in order to make up a valid file name for the according file system. For languages like Arabic, Chinese and so on this may result in the following PHP warning: 

PHP Warning:  file_put_contents(cache/......html.gz): failed to open stream: File name too long in includes/cache/FileCacheBase.php on line 171

This is a known limitation. See <translate> task <tvar name=1>T148684</tvar></translate>.

Category and image description pages aren't purged from file cache. For example, adding or removing a page from a category doesn't update the category page, causing not logged in users to not see the changes in the category page. This is a known limitation. See <translate> task <tvar name=1>T26575</tvar></translate>

Domain and range

Caching is only done for users who:

  • are not logged in.
  • do not have their user_newtalk flag active.

Caching is only done for pages which:

  • are not special pages.
  • are not redirects.
  • are being viewed in current version, plain view, no url parameters

This covers the majority of requests to the wiki, but it is still important to set up bytecode and application caching to handle the rest.

Validation

The modification time of the cached file is compared with the global $wgCacheEpoch timestamp set in LocalSettings.php on each view.

If the file is at least as new as both of these, it is considered valid and is sent directly to the client. If it is older or does not exist, parsing and rendering continues and the results are saved for future use.

Invalidation

The entire cache can be invalidated by setting $wgCacheEpoch to the current time, or by deleting all files in the cache.

Individual pages are invalidated by updating their page.page_touched fields, conveniently done by calling Title::invalidateCache(). This should be done on article creation, edit saves, renames, and creation and deletion of linked articles (in order to update edit links). Invalidation operations that occur when editing templates, for example, check pages that use template to compare the cache date against page_touched.

Some cases are not yet handled properly, which probably includes:

  • browser 'reload' or 'refresh' (just reloads same cached page without updates)
  • output variables from extensions such as Extension:DynamicPageList disable filecache for those pages to avoid stale data. Those that don't will return stale data even on browser refresh.
  • certain long error messages may be cached indefinitely as if they were valid page content; only ?action=purge will remove these from the file cache once stored. A minimum output size threshold is used to avoid caching most errors, like fatal PHP errors.

Also see: Manual:$wgInvalidateCacheOnLocalSettingsChange .

Expiration

There should probably be some method of expiration of cache pages, particularly for pages containing variables (it is X date, we have X articles, etc).

If $parserOutput->getCacheTime() is -1, file caching will be disabled for the output of the page. There is no provision to set an expiry time, so all HTML for all pages is cached forever. An explicit ?action=purge command (or an edit to the page) will regenerate that one page, but neither the MediaWiki internal no-cache flags nor the browser refresh will remove outdated extension output once it has been stored as part of a file-cached page.

Refresh tab

It is possible to add a tab to force one individual page to be invalidated and refreshed by using ?action=purge using the Purge page extension.

Compression

Optionally, the cache may be compressed to save space and bandwidth. (This requires that zlib be enabled in the PHP config.) 

$wgUseGzip = true;

If compression is enabled, the cache files are saved as .html.gz. Browsers that advertise support for gzip in their Accept-Encoding field will be given the gzipped version straight; for those browsers that don't, we unzip the data on the fly and send them the plaintext.

A "Vary: User-agent" header will be sent to tell proxy caches to be more careful about who it resends data to. ("Vary: Accept-encoding" would be more appropriate, but Internet Explorer refuses to cache pages so marked.)

File:OOjs UI icon notice-destructive.svg <translate> Warning:</translate> Do not enable $wgUseGzip if you have output buffering enabled that gzips the content (via ob_gzhandler or zlib.output_compression). Doing so will result in gibberish being shown to users.

Emergency fallback

If the wiki can't contact the database server, it will try to show the cached version of whatever page was requested, regardless of whether it may be current or not, with a "database is down" message tacked into it.

This has some limitations:

  • special pages are not covered in any way, there's just a warning message
  • redirect pages are not cached, so clicking a link to a redirect doesn't go through to the final destination
  • pages with colons in the title (other than for a valid namespace) may still fail due to MediaWiki checking the DB to see if the title has a valid interwiki prefix. This occurs only if there is no interwiki cache entry for the page's prefix (see $wgInterwikiExpiry ) or $wgMainCacheType is falling back (or set to) DB_CACHE.
  • attempts to use non-view actions result in a plain page view, which may be confusing
  • there may be issues with the MySQL connection timeout which make it take prohibitively long before giving up, particularly if using persistent connections and the db dies later. Adjust mysql.connect_timeout (set to 3 or more) in php.ini to deal with this. Edit the webserver's php.ini not the cli php.ini.
  • fallback may fail if $wgStatsMethod is set to cache and $wgMainCacheType is falling back (or set to) DB_CACHE


Serving cached pages directly

By default, MediaWiki passes cache page with php. Here is how we use webserver configuration tricks to serve cached files directly without invoking MediaWiki or PHP at all.

First, setting $wgFileCacheDepth = 0;. Then we add rewrite rules. See below.

Apache

The following mod_rewrite rule has been found to work in an .htaccess file on a shared webhost running Apache 2.2: 

RewriteBase /
# If a cached page exists under /w/html_cache, do an internal redirect to it:
RewriteCond %{HTTP_COOKIE} !UserID=
RewriteCond %{QUERY_STRING} !.
RewriteCond %{DOCUMENT_ROOT}/w/html_cache/$1.html -s
RewriteRule ^wiki/(.+)$ /w/html_cache/$1.html [B,L,NS]

However, this rewrite rule won't work properly for pages whose titles contain punctuation or non-ASCII characters. A tidy solution would be to use a RewriteMap, but this is not supported in .htaccess context. Instead, the following trick can be used to handle non-ASCII titles by pulling the URL-escaped title directly from the raw request line: 

RewriteBase /
# If a cached page exists under /w/html_cache, do an internal redirect to it: 
RewriteCond %{HTTP_COOKIE} !UserID=
RewriteCond %{QUERY_STRING} !.
ReWriteCond %{THE_REQUEST} ^GET\x20/wiki/([^\x20/]+)\x20HTTP
RewriteCond %{DOCUMENT_ROOT}/w/html_cache/%1.html -s
RewriteRule ^wiki/(.+)$ /w/html_cache/%1.html [B,L,NS]

The previous rewrite rules will handle requests of the form 

https://site/wiki/Page_Name

To handle requests where the Page_Name is in the QUERY_STRING like 

https://site/w/index.php?title=Page_Name

the following works: 

RewriteCond %{HTTP_COOKIE} !UserID=
RewriteCond %{QUERY_STRING} ^title=([^&]+)$
RewriteCond %{DOCUMENT_ROOT}/images/cache/ns0\%3A%1.html -s
RewriteRule ^ /images/cache/ns0\%253A%1.html? [B,L,NS]

This will still not match titles containing periods, slashes or other punctuation which the file cache escapes but browsers don't (or vice versa). One should also ensure that the appropriate Vary header gets sent: 

Header append Vary Cookie

Also note that, since this redirect trick bypasses MediaWiki entirely, it won't respect $wgCacheEpoch . You may need to clear or rebuild the cache manually as needed instead.

Nginx

Since the nginx configuration language is pretty limited (it doesn't allow nested if statements, for example), it requires the ngx_lua module.

The following conf should be integrated inside your server block, and it assumes that:

  • you're using fancy urls like example.com/Page;
  • your cache folder is inside your MediaWiki folder (default);
  • you're already proxying *.php files to php-fpm (or whatever) in the last location block.

And here are the limitations:

  • as with the Apache solution, you have to run a cron job or manually rebuild the cache files when they change;
  • pages whose title include non-ASCII characters are served through PHP (which should be transparent to the user anyway).

Change paths as needed. 

 location ~ /cache/(.*) {
     internal;
 }
 
 location / {
     default_type 'text/html';
     rewrite_by_lua '
         local cookie = ngx.req.get_headers()["Cookie"] or ""
         if string.match(cookie, "UserID") then  -- if we are logged in...
             ngx.exec("@rewrite")
         else
             local page = string.sub(ngx.var.request_uri, 2, -1)  -- get the page name without the leading slash
             local filename = page .. ".html"
             local f = io.open("/var/www/w/cache/" .. filename, "r")
             if f ~= nil then  -- if the file exists, close it and serve it directly
                 io.close(f)
                 ngx.header["x-wiki-cache"] = "via nginx"  -- headers to help you debug
                 ngx.exec("/cache/" .. filename)
             else
                 ngx.header["x-wiki-cache"] = "via mediawiki"
                 ngx.exec("@rewrite")
             end
         end
     ';
 }
  
 location @rewrite {
     rewrite ^/([^?]*)(?:\?(.*))? /index.php?title=$1&$2 last;
 }

Code stewardship

See also

Retrieved from "https://www.linuxwebexpert.com/wiki/index.php?title=Manual:File_cache&oldid=2728"