View source for Manual:Importing XML dumps

<languages/>
{{RightTOC}}
<translate>
<!--T:20-->
This page describes methods to '''import XML dumps'''.
XML Dumps contain the content of a wiki (wiki pages with all their revisions), without the site-related data.
A XML dump does not create a full backup of the wiki database, the dump does not contain user accounts, images, edit logs, etc.

<!--T:21-->
The <tvar name=1>[[Special:Export]]</tvar> page of any MediaWiki site, including any Wikimedia site and Wikipedia, creates an XML file (content dump).
See [[<tvar name=2>m:Special:MyLanguage/Data dumps</tvar>|meta:Data dumps]] and <tvar name=3>{{ll|Manual:DumpBackup.php}}</tvar>.
XML files are  explained more on [[<tvar name=4>Special:MyLanguage/Help:Export#Export format</tvar>|meta:Help:Export]].

==How to import?== <!--T:2-->

<!--T:7-->
There are several methods for importing these XML dumps.

===Importing large XML dumps (such as English Wikipedia) === <!--T:11-->
</translate>
====Guide to Importing the English Wikipedia XML File====

Importing the English Wikipedia files can be a complex task, but with the right approach, it can be achieved reliably.
This guide outlines the process using the importDump.php script located in the maintenance folder of your MediaWiki installation.

=====Prepare your environment=====

Before starting the import process, ensure that your server meets the necessary [[Manual:Installation_requirements|requirements]], including sufficient disk space and system resources.

=====Obtain the Wikipedia XML Dump=====

The English Wikipedia XML dump can be downloaded from the Wikimedia Foundation's website.
At the time of this writing, the file size is approximately 91GB.

=====Importing the XML Dump=====
There are several methods to import the XML dump into your MediaWiki installation:

a. Direct Import: You can use the importDump.php script provided in the maintenance folder. This method is reliable but may be resource-intensive.

b. Individual File Import: Alternatively, you can download and import each multipart XML file individually. While this approach allows for more manageable imports, it requires additional effort.

=====Managing Import Errors=====

Importing such a large file may result in errors, leading to interruptions in the process.
To mitigate this, consider setting up cron jobs to import the XML dump in smaller batches over time.
This approach ensures smoother processing and reduces the risk of failures.

Simple Cron Schedule Example

Notes:
# The Wiki files are assumed to be located in the main Web Server folder, typically /var/www/html/.
# The XML dump file is also assumed to be located in the same directory.
# This cron job is executed by the root user, and log files are created in the /root/wikilogs folder.
# Replace {your-server} with the URL of your server in the --server argument.
# This schedule runs every 10 minutes, allowing for different import sections to be executed in different hours.

Please make changes to point to your specific folders, or create any folders you wish to use.

<syntaxhighlight lang="Bash">
*/15 4 * * * /root/wikilogs/10-cronjob.sh > ~/wikilogs/import-$(date +\%Y\%m\%d_\%H\%M\%S)-cron.log 2>&1
*/15 5 * * * /root/wikilogs/10-cronjob.sh > ~/wikilogs/import-$(date +\%Y\%m\%d_\%H\%M\%S)-cron.log 2>&1
*/15 6 * * * /root/wikilogs/10-cronjob.sh > ~/wikilogs/import-$(date +\%Y\%m\%d_\%H\%M\%S)-cron.log 2>&1
*/15 7 * * * /root/wikilogs/10-cronjob.sh > ~/wikilogs/import-$(date +\%Y\%m\%d_\%H\%M\%S)-cron.log 2>&1
*/15 8 * * * /root/wikilogs/10-cronjob.sh > ~/wikilogs/import-$(date +\%Y\%m\%d_\%H\%M\%S)-cron.log 2>&1
*/15 9 * * * /root/wikilogs/10-cronjob.sh > ~/wikilogs/import-$(date +\%Y\%m\%d_\%H\%M\%S)-cron.log 2>&1
*/15 10 * * * /root/wikilogs/10-cronjob.sh > ~/wikilogs/import-$(date +\%Y\%m\%d_\%H\%M\%S)-cron.log 2>&1
*/15 11 * * * /root/wikilogs/10-cronjob.sh > ~/wikilogs/import-$(date +\%Y\%m\%d_\%H\%M\%S)-cron.log 2>&1
*/15 12 * * * /root/wikilogs/10-cronjob.sh > ~/wikilogs/import-$(date +\%Y\%m\%d_\%H\%M\%S)-cron.log 2>&1
*/15 13 * * * /root/wikilogs/10-cronjob.sh > ~/wikilogs/import-$(date +\%Y\%m\%d_\%H\%M\%S)-cron.log 2>&1
*/15 14 * * * /root/wikilogs/10-cronjob.sh > ~/wikilogs/import-$(date +\%Y\%m\%d_\%H\%M\%S)-cron.log 2>&1
*/15 15 * * * /root/wikilogs/10-cronjob.sh > ~/wikilogs/import-$(date +\%Y\%m\%d_\%H\%M\%S)-cron.log 2>&1
*/15 16 * * * /root/wikilogs/10-cronjob.sh > ~/wikilogs/import-$(date +\%Y\%m\%d_\%H\%M\%S)-cron.log 2>&1
*/15 17 * * * /root/wikilogs/10-cronjob.sh > ~/wikilogs/import-$(date +\%Y\%m\%d_\%H\%M\%S)-cron.log 2>&1
*/15 18 * * * /root/wikilogs/10-cronjob.sh > ~/wikilogs/import-$(date +\%Y\%m\%d_\%H\%M\%S)-cron.log 2>&1
*/15 19 * * * /root/wikilogs/10-cronjob.sh > ~/wikilogs/import-$(date +\%Y\%m\%d_\%H\%M\%S)-cron.log 2>&1
*/15 20 * * * /root/wikilogs/10-cronjob.sh > ~/wikilogs/import-$(date +\%Y\%m\%d_\%H\%M\%S)-cron.log 2>&1
*/15 21 * * * /root/wikilogs/10-cronjob.sh > ~/wikilogs/import-$(date +\%Y\%m\%d_\%H\%M\%S)-cron.log 2>&1
*/15 22 * * * /root/wikilogs/10-cronjob.sh > ~/wikilogs/import-$(date +\%Y\%m\%d_\%H\%M\%S)-cron.log 2>&1
0 0 * * * /var/www/html/maintenance/run updateArticleCount.php --use-master --globals --memory-limit default --profiler json --update > ~/wikilogs/articleCount-`date +\%Y\%m\%d\%H\%M\%S`-cron.log 2>&1
30 1 * * * /var/www/html/maintenance/run update --quick > ~/wikilogs/update-`date +\%Y\%m\%d\%H\%M\%S`-cron.log 2>&1
30 2 * * * /var/www/html/maintenance/run rebuildrecentchanges.php --globals --memory-limit default --profiler json > ~/wikilogs/recentChanges-`date +\%Y\%m\%d\%H\%M\%S`-cron.log 2>&1
30 3 * * * /var/www/html/maintenance/run cleanupInvalidDbKeys.php --globals --memory-limit default --profiler json > ~/wikilogs/cleanupInvalidDbKeys-`date +\%Y\%m\%d\%H\%M\%S`-cron.log 2>&1
0 4 * * * /var/www/html/maintenance/run rebuildall.php --use-master --globals --memory-limit default --profiler json --update > ~/wikilogs/rebuildAll-`date +\%Y\%m\%d\%H\%M\%S`-cron.log 2>&1
0 5 * * * /var/www/html/maintenance/run cleanupUsersWithNoId.php --use-master --globals --memory-limit default --profiler json --update > ~/wikilogs/cleanupUsersWithNoId-`date +\%Y\%m\%d\%H\%M\%S`-cron.log 2>&1
</syntaxhighlight>

In the /root/wikilogs folder, I have created a simple shell script to help import the dump.

<syntaxhighlight lang="bash">
#!/bin/bash

# Get the Wiki version to test for compatibility
# Get the BASH version to test for compatibility
# Get the current number of pages, round down to the nearest 1000's or 0 if less than 1000
# Get the server URL
# Start the import to the number of Articles
# Show total pages imported

# Function to convert HH:MM:SS to seconds
function time_to_seconds() {
    local time=$1
    local seconds=0
    IFS=":" read -r hours minutes seconds <<< "$time"
    seconds=$((hours*3600 + minutes*60 + seconds))
    echo "$seconds"
}

# Function to compare version numbers
function version_compare() {
    local version1="$1"
    local version2="$2"
    if [[ "$(printf '%s\n' "$version1" "$version2" | sort -V | head -n1)" == "$version1" ]]; then
        return 0
    else
        return 1
    fi
}

wikilocation="/var/www/html"

# Get the currently installed Wiki Version
rawWikiVersion=$(${wikilocation}/maintenance/run version)
wikiVersion=$(echo "$rawWikiVersion" | awk -F': ' '{split($2,a," "); print a[1]}')

# Compare the version with 1.4.0
# if ! version_compare "$wikiVersion" "1.40.0"; then
#     echo "MediaWiki version is not greater than 1.40.0. Aborting import process."
#     exit 1
# fi

# Get the latest number of pages in the stats
totalStartPages=$(${wikilocation}/maintenance/run showSiteStats | awk '/Total pages/ {print $NF}')

# Round down
articleCount=$(echo "scale=0; (${totalStartPages} / 1000) * 1000" | bc)

# The URL to the wiki
server="<YOUR HTTPS://WEBSITE HERE>"

# Location of the xml file
inputFile="/var/www/html/enwiki-20240120-pages-articles-multistream.xml"

start_time=$(date +%s)
echo "Starting on page ${articleCount} at $(date '+%m-%d-%y %I:%M %p') - Wiki Version ${wikiVersion} with ${server} for the URL"
${wikilocation}/maintenance/run importDump --quiet --memory-limit default --skip-to "${articleCount}" --server "${server}" < ${inputFile} > /dev/null
end_time=$(date +%s)

# Calculate the total number of pages imported
totalEndPages=$(${wikilocation}/maintenance/run showSiteStats | awk '/Total pages/ {print $NF}')
pages_imported=$((totalEndPages - totalStartPages))

# Calculate the duration of the import process in seconds
duration=$((end_time - start_time))

# Convert duration to minutes
duration_minutes=$((duration / 60))

# Calculate pages per minute
pages_per_minute=$(awk "BEGIN {printf \"%.2f\", ${pages_imported} / ${duration_minutes}}")

echo "Completed ${pages_imported} pages in ${duration_minutes} minutes (${pages_per_minute} pages/minute) on $(date '+%m-%d-%y %I:%M %p')"
</syntaxhighlight>

=====Monitoring and Troubleshooting=====

During the import process, monitor the server logs for any errors or warnings. Common issues may include database connection errors, memory exhaustion, or file permissions issues. Troubleshoot any encountered problems promptly to ensure a successful import.

=====Post-Import Tasks=====

Once the import is complete, perform any necessary post-import tasks, such as rebuilding the search index or updating site configurations. Verify the integrity of the imported content to ensure consistency with the original Wikipedia articles.

=====Conclusion=====

Utilizing a server equipped with 16 cores, 16GB of RAM, and a 2TB SSD, the average import rate over a two-week period stands at 96 pages per minute. This performance metric provides valuable insight into the expected efficiency of the import process, allowing for effective planning and scheduling of import tasks. By leveraging the capabilities of such a server configuration, you can successfully import the English Wikipedia XML dump into your MediaWiki installation, enriching your platform with a wealth of knowledge and resources for users to explore.

To calculate the approximate time it will take to import 59,795,345 pages at an average rate of 96 pages per minute, we can use the following formula:

Total pages ÷ Pages per minute ÷ Minutes per hour ÷ Hours per day ÷ Days per week

'''Plugging in the values''':
59,795,345÷96÷(60×24)÷7 ≈11.87

Rounding up, it will take approximately 12 weeks to import 59,795,345 pages.

<translate>
===Using Special:Import=== <!--T:3-->

<!--T:22-->
<tvar name=1>[[Special:Import]]</tvar> can be used by wiki users with <tvar name=2><code>import</code></tvar> permission (by default this is users in the <tvar name=3><code>sysop</code></tvar> group) to import a small number of pages (about 100 should be safe).

<!--T:23-->
''Trying to import large dumps this way may result in timeouts or connection failures''.
</translate>

* <translate><!--T:24--> See [[<tvar name=1>m:Special:MyLanguage/Help:Import</tvar>|meta:Help:Import]] for a detailed description.</translate><ref><translate><!--T:25--> See <tvar name=1>{{ll|Manual:XML Import file manipulation in CSharp}}</tvar> for a C# code sample that manipulates an XML import file.</translate></ref>
* <translate><!--T:26--> Large files might be rejected due to limits in PHP configuration, see [[<tvar name=1>m:Special:MyLanguage/Help:Import#Large-scale_transfer</tvar>|meta:Help:Import#Large-scale_transfer]]</translate>

<translate>
<!--T:27-->
You are asked to give an interwiki prefix.
For instance, if you exported from the English Wikipedia, you have to type 'en'.

====Changing permissions==== <!--T:4-->

<!--T:10-->
See <tvar name=1>{{ll|Manual:User rights}}</tvar>.

<!--T:28-->
To allow all registered editors to import (not recommended) the line added to "LocalSettings.php" would be:
</translate>

<syntaxhighlight lang="php">
$wgGroupPermissions['user']['import'] = true;
$wgGroupPermissions['user']['importupload'] = true;
</syntaxhighlight>

<translate>
==== Possible problems ==== <!--T:5-->

<!--T:29-->
For using Transwiki-Import PHP safe_mode must be off and "open_basedir" must be empty (both of them are variables in php.ini).
Otherwise the import fails.

<!--T:8-->
If you get error like this:
</translate>

 Warning: XMLReader::open(): Unable to open source data in /.../wiki/includes/Import.php on line 53
 Warning: XMLReader::read(): Load Data before trying to read in /.../wiki/includes/Import.php on line 399

<translate>
<!--T:30-->
And Special:Import shows: "''Import failed: Expected &lt;mediawiki> tag, got ''", this may be a problem caused by a fatal error on a previous import, which leaves libxml in a wrong state across the entire server, or because another PHP script on the same server disabled entity loader ([<tvar name=1>https://bugs.php.net/bug.php?id=62577</tvar> PHP bug]).
This happens on MediaWiki versions prior to MediaWiki 1.26, and the solution is to restart the webserver service (apache, etc), or write and execute a script that calls <tvar name=2><code>libxml_disable_entity_loader(false);</code></tvar> (see <tvar name=3>{{phabricator|T86036}}</tvar>).

===Using <tvar name=1>importDump.php</tvar>, if you have shell access=== <!--T:6-->
</translate>
:<translate><!--T:31--> ''Recommended method for general use, but slow for very big data sets.''</translate>
: <translate><!--T:32--> ''See: <tvar name=1>{{ll|Manual:importDump.php}}</tvar>, including tips on how to use it for large wikis.''</translate>

<translate>
===Using <tvar name=1>importTextFiles.php</tvar> maintenance Script=== <!--T:19-->
</translate>
{{MW 1.23|and before}}
{{MW 1.27|+}}
<translate>
<!--T:33-->
If you have a lot of content converted from another source (several word processor files, content from another wiki, etc), you may have several files that you would like to import into your wiki.
In MediaWiki 1.27 and later, you can use the [[<tvar name=1>Special:MyLanguage/Manual:ImportTextFiles.php</tvar>|importTextFiles.php]] maintenance script.

<!--T:34-->
You can also use the [[<tvar name=1>Special:MyLanguage/Manual:Edit.php</tvar>|edit.php]] maintenance script for this purpose.

===rebuildall.php=== <!--T:35-->

<!--T:36-->
For large XML dumps, you can run <tvar name=1><code>[[Special:MyLanguage/Manual:rebuildall.php|rebuildall.php]]</code></tvar>, but, it will take a  ''long'' time, because it has to parse all pages.
This is not recommended for large data sets.

=== Using pywikibot, pagefromfile.py and Nokogiri === <!--T:37-->

<!--T:38-->
<tvar name=1>{{ll|pywikibot}}</tvar> is a collection of tools written in python that automate work on Wikipedia or other MediaWiki sites.
Once installed on your computer, you can use the specific tool '[[<tvar name=2>Special:MyLanguage/Manual:Pywikibot/pagefromfile.py</tvar>|pagefromfile.py]]' which lets you upload a wiki file on Wikipedia or MediaWiki sites.
The xml file created by dumpBackup.php can be transformed into a wiki file suitable to be processed by 'pagefromfile.py' using a simple Ruby program similar to the following (here the program will transform all xml files which are on the current directory which is needed if your MediaWiki site is a family):
</translate>

<syntaxhighlight lang="ruby">
# -*- coding: utf-8 -*-
# dumpxml2wiki.rb

require 'rubygems'
require 'nokogiri'

# This program dumpxml2wiki reads MediaWiki xml files dumped by dumpBackup.php
# on the current directory and transforms them into wiki files which can then 
# be modified and uploaded again by pywikipediabot using pagefromfile.py on a MediaWiki family site.
# The text of each page is searched with xpath and its title is added on the first line as
# an html comment: this is required by pagefromfile.py.
# 
Dir.glob("*.xml").each do |filename|
  input = Nokogiri::XML(File.new(filename), nil, 'UTF-8')

  puts filename.to_s  # prints the name of each .xml file

  File.open("out_" + filename + ".wiki", 'w') {|f| 
    input.xpath("//xmlns:text").each {|n|
      pptitle = n.parent.parent.at_css "title" # searching for the title
      title = pptitle.content
      f.puts "\n{{-start-}}<!--'''" << title.to_s << "'''-->" << n.content  << "\n{{-stop-}}"
    }
  }
end
</syntaxhighlight>

<translate>
<!--T:39-->
For example, here is an excerpt of a wiki file output by the command 'ruby dumpxml2wiki.rb' (two pages can then be uploaded by pagefromfile.py, a Template and a second page which is a redirect):
</translate>

<syntaxhighlight lang="html4strict">
{{-start-}}<!--'''Template:Lang_translation_-pl'''--><includeonly>Tłumaczenie</includeonly>
{{-stop-}}

{{-start-}}#REDIRECT[[badania demograficzne]]<!--'''ilościowa demografia'''-->
<noinclude>
[[Category:Termin wielojęzycznego słownika demograficznego (pierwsze wydanie)|ilościowa demografia]]
[[Category:Termin wielojęzycznego słownika demograficznego (pierwsze wydanie) (redirect)]]
[[Category:10]]</noinclude>
{{-stop-}}
</syntaxhighlight>

<translate>
<!--T:40-->
The program accesses each xml file, extracts the texts within <tvar name=1><nowiki><text> </text></nowiki></tvar> markups of each page, searches the corresponding title as a parent and enclosed it with the paired <nowiki>{{-start-}}<!--'''Title of the page'''--> {{-stop-}}</nowiki> commands used by 'pagefromfile' to create or update a page.
The name of the page is in an html comment and separated by three quotes on the same first start line.
Please notice that the name of the page can be written in Unicode.
Sometimes it is important that the page starts directly with the command, like for a <tvar name=2><nowiki>#REDIRECT</nowiki></tvar>; thus the comment giving the name of the page must be ''after'' the command but still on the first line.

<!--T:41-->
Please remark that the xml dump files produced by dumpBackup.php are prefixed by a namespace:
</translate>

<syntaxhighlight lang="html4strict"><mediawiki xmlns="http://www.mediawiki.org/xml/export-0.4/"></syntaxhighlight>

<translate>
<!--T:42-->
In order to access the text node using Nokogiri, you need to prefix your path with 'xmlns':
</translate>

<syntaxhighlight lang="html4strict">input.xpath("//xmlns:text")</syntaxhighlight> 

<translate>
<!--T:12-->
Nokogiri is an HTML, XML, SAX, & Reader parser with the ability to search documents via XPath or CSS3 selectors from the last generation of XML parsers using Ruby.

<!--T:43-->
Example of the use of 'pagefromfile' to upload the output wiki text file:
</translate>

<syntaxhighlight lang="bash">
python pagefromfile.py -file:out_filename.wiki -summary:"Reason for changes" -lang:pl -putthrottle:01
</syntaxhighlight>

<translate>
=== How to import logs? === <!--T:13-->

<!--T:44-->
Exporting and importing logs with the standard MediaWiki scripts often proves very hard; an alternative for import is the script <tvar name=1><code>pages_logging.py</code></tvar> in the [<tvar name=2>https://github.com/glimmerphoenix/WikiDAT</tvar> WikiDAT tool], as [[<tvar name=3>mailarchive:xmldatadumps-l/2013-July/000876.html</tvar>|suggested by Felipe Ortega]].

== Troubleshooting == <!--T:14-->

=== Merging histories, revision conflict, edit summaries, and other complications === <!--T:15-->
</translate>
{{see also|meta:Help:Import#Merging_histories_and_other_complications}}

<translate>
=== Interwikis === <!--T:16-->

<!--T:45-->
If you get the message
 Page "meta:Blah blah" is not imported because its name is reserved for external linking (interwiki).
the problem is that some pages to be imported have a prefix that is used for <tvar name=1>{{ll|interwiki}}</tvar> linking.
For example, ones with a prefix of 'Meta:' would conflict with the interwiki prefix <tvar name=2><code>meta:</code></tvar> which by default links to <tvar name=3><nowiki>https://meta.wikimedia.org</nowiki></tvar>.

<!--T:46-->
You can do any of the following.
</translate>

*<translate><!--T:47--> Remove the prefix from the {{<tvar name=1>ll|Manual:Interwiki table</tvar>|interwiki}} table. This will preserve page titles, but prevent interwiki linking through that prefix.</translate>
*:<translate><!--T:48--> Example: you will preserve page titles 'Meta:Blah blah' but will not be able to use the prefix 'meta:' to link to meta.wikimedia.org (although it will be possible through a different prefix).</translate>
*:<translate><!--T:49--> How to do it: before importing the dump, run the query <tvar name=1><code>DELETE FROM interwiki WHERE iw_prefix='prefix'</code></tvar> (note: do not include the colon in the <code>prefix</code>). Alternatively, if you have [[<tvar name=2>Special:MyLanguage/Manual:Guide to setting up interwiki linking#Adding interwiki prefixes through Special:Interwiki</tvar>|enabled editing the interwiki table]], you can simply go to <tvar name=3>[[Special:Interwiki]]</tvar> and click the 'Delete' link on the right side of the row belonging to that prefix.</translate>
*<translate><!--T:50--> Replace the unwanted prefix in the XML file with "Project:" before importing. This will preserve the functionality of the prefix as an interlink, but will replace the prefix in the page titles with the name of the wiki where they're imported into, and might be quite a pain to do on large dumps.</translate>
*:<translate><!--T:51--> Example: replace all 'Meta:' with 'Project:' in the XML file. MediaWiki will then replace 'Project:' with the name of your wiki during importing.</translate>

<translate>
==See also== <!--T:17-->
</translate>
*{{ll|Manual:DumpBackup.php}}
*[[meta:Special:MyLanguage/Data dumps/Other tools|meta:Data dumps/Other tools]]
*{{ll|Manual:System administration#Getting data}}
*<translate><!--T:52--> <tvar name=1>{{ll|Manual:Configuring file uploads#Set maximum size for file uploads}}</tvar> &ndash; May come in handy if you are doing massive imports</translate>
*<translate><!--T:53--> <tvar name=1>{{ll|Manual:Errors and Symptoms#Fatal error: Allowed memory size of nnnnnnn bytes exhausted .28tried to allocate nnnnnnnn bytes.29}}</tvar> &ndash; Settings that may need to be changed if you are doing massive imports</translate>
* <translate><!--T:54--> <tvar name=1>{{ll|Manual:ImportImages.php}}</tvar> - for importing images.</translate>
* {{ll|Manual:Importing external content}}
* {{ll|Manual:Importing Wikipedia infoboxes tutorial}}

<translate>
==References== <!--T:18-->
</translate>
<references/>

[[Category:Import/Export{{#translation:}}|{{PAGENAME}}]]