Extension:Math/RESTBase
From Linux Web Expert
Configure Restbase to work with the Math extension
To install mathoid and restbase to work with the math extension the following steps are required:
Mathoid
npm i mathoid
cd node_modules/mathoid
edit config.yaml as follows
# Number of worker processes to spawn.
# Set to 0 to run everything in a single process without clustering.
# Use 'ncpu' to run as many workers as there are CPU units
num_workers: ncpu
# Log error messages and gracefully restart a worker if v8 reports that it
# uses more heap (note: not RSS) than this many mb.
worker_heap_limit_mb: 500
# Logger info
logging:
level: warn
# Statsd metrics reporter
metrics:
type: log
#host: localhost
#port: 8125
services:
- name: mathoid
# a relative path or the name of an npm package, if different from name
module: ./app.js
# optionally, a version constraint of the npm package
# version: ^0.4.0
# per-service config
conf:
port: 10042
# interface: localhost # uncomment to only listen on localhost
# more per-service config settings
svg: true
img: true
png: true
texvcinfo: true
speech: true
speech_on: false
chem: true
no_check: true
run mathoid as a screen session
screen
node server.js -c config.yaml
testing mathoid
Now the following test
curl -d 'q=E=mc^2' localhost:10042/mml
should return
<math xmlns="http://www.w3.org/1998/Math/MathML" display="block">
<semantics>
<mrow>
<mi>E</mi>
<mo>=</mo>
<mi>m</mi>
<msup>
<mi>c</mi>
<mn>2</mn>
</msup>
</mrow>
<annotation encoding="application/x-tex">E=mc^2</annotation>
</semantics>
</math>
RESTBase
npm i restbase
cd node_modules/restbase
edit config.yaml as follows
paths:
/{api:v1}:
x-modules:
# swagger options, overriding the shared ones from the merged specs (?)
- spec:
info:
version: 1.0.0-beta
title: Formulasearchengine REST API
description: >
This API aims to provide a preview of new features reagarding
Math rendering and search, supposed to be integrated to
the [wikimedia restbase api](https://wikimedia.org/api/rest_v1/?doc).
### High-volume access
- Don't perform more than 500 requests/s to this API.
- Set a unique `User-Agent` header that allows us to contact you
quickly. Email addresses or URLs of contact pages work well.
termsOfService: https://wikimediafoundation.org/wiki/Terms_of_Use
contact:
name: the Wikimedia Services team
url: http://mediawiki.org/wiki/RESTBase
license:
name: Apache2
url: http://www.apache.org/licenses/LICENSE-2.0
securityDefinitions: &wp/content-security/1.0.0
mediawiki_auth:
description: Checks permissions using MW api
type: apiKey
in: header
name: cookie
x-internal-request-whitelist:
- /http:\/\/[a-zA-Z0-9\.]+\/w\/api\.php/
# Override the base path for host-based (proxied) requests. In our case,
# we proxy https://{domain}/api/rest_v1/ to the API.
x-host-basePath: /api/rest_v1
x-route-filters:
- path: ./lib/normalize_title_filter.js
options:
redirect_cache_control: '{{options.purged_cache_control}}'
paths:
/media:
x-modules:
- path: v1/mathoid.yaml
options: '{{options.mathoid}}'
/testing:
x-modules:
- path: ../mathpipe.yaml
# options: '{{options.mathoid}}'
options: '{{options}}'
/{api:sys}:
x-modules:
- spec:
paths:
/mathoid:
x-modules:
- path: sys/mathoid.js
options: '{{options.mathoid}}'
/table:
x-modules:
- path: sys/table.js
options:
conf: '{{options.table}}'
/key_value:
x-modules:
- path: sys/key_value.js
/key_rev_value:
x-modules:
- path: sys/key_rev_value.js
/key_rev_latest_value:
x-modules:
- path: sys/key_rev_latest_value.js
/post_data: &sys_post_data
x-modules:
- path: sys/post_data.js
/events:
x-modules:
- path: sys/events.js
options: '{{options.events}}'
options: '{{options}}'
run restbase as a screen session
screen
node server.js -c config.yaml
testing restbase
Navigate to http://localhost:8081/wikimedia.org/v1/?doc you should be able to enter a texvc formula to the check command POST endpoing i.e. /media/math/check/{type} (See screenshot 1)[1] In the response headers you should find something like
"x-resource-location": "4c0004393a88f350a93bcef62106d556c7fc827b"
If you copy the value 4c0004393a88f350a93bcef62106d556c7fc827b and paste it to the GET endpoint /media/math/render/{format}/{hash} as value you should be able to see the PNG image if you select that as an output format. (See screenshot 2)[2]
Configuration Example for arbitrary domain in RESTBase setup
If you want to use domain other than wikimedia.org, you can follow below example;
- mathoid/config.yaml
# Number of worker processes to spawn.
# Set to 0 to run everything in a single process without clustering.
# Use 'ncpu' to run as many workers as there are CPU units
num_workers: 0
# Log error messages and gracefully restart a worker if v8 reports that it
# uses more heap (note: not RSS) than this many mb.
worker_heap_limit_mb: 250
# Logger info
logging:
level: trace
# streams:
# # Use gelf-stream -> logstash
# - type: gelf
# host: logstash1003.eqiad.wmnet
# port: 12201
# Statsd metrics reporter
metrics:
#type: log
#host: localhost
#port: 8125
services:
- name: mathoid
# a relative path or the name of an npm package, if different from name
module: ./app.js
# optionally, a version constraint of the npm package
# version: ^0.4.0
# per-service config
conf:
port: 10044
# interface: localhost # uncomment to only listen on localhost
# more per-service config settings
# the location of the spec, defaults to spec.yaml if not specified
# spec: ./spec.template.yaml
# allow cross-domain requests to the API (default '*')
#cors: '*'
# to disable use:
# cors: false
# to restrict to a particular domain, use:
# cors: restricted.domain.org
# content for the CSP headers
# csp: false # uncomment this line to disable sending them
# URL of the outbound proxy to use (complete with protocol)
# proxy: http://my.proxy.org:8080
# the list of domains for which not to use the proxy defined above
# no_proxy_list:
# - domain1.com
# - domain2.org
# the list of incoming request headers that can be logged; if left empty,
# the following headers are allowed: cache-control, content-length,
# content-type, if-match, user-agent, x-request-id
# log_header_whitelist:
# - cache-control
# - content-length
# - content-type
# - if-match
# - user-agent
# - x-request-id
# list of enabled renders
svg: true
img: true
png: true #new feature
speech: true #new feature
texvcinfo: true
speech_on: true
no_check: false
dpi: 180
svgo: false
# the user agent to use when issuing requests
# user_agent: service-template-node
# the template used for contacting the MW API
#mwapi_req:
# method: post
# uri: https://{{domain}}/w/api.php
# headers:
# user-agent: '{{user-agent}}'
# body: '{{ default(request.query, {}) }}'
# the template used for contacting RESTBase
#restbase_req:
# method: '{{request.method}}'
# uri: https://{{domain}}/api/rest_v1/{+path}
# query: '{{ default(request.query, {}) }}'
# headers: '{{request.headers}}'
# body: '{{request.body}}'
- restbase/config.yaml
services:
- name: restbase
module: hyperswitch
conf:
port: 7231
salt: secret
default_page_size: 125
user_agent: RESTBase
ui_name: RESTBase
ui_url: https://www.mediawiki.org/wiki/RESTBase
ui_title: RESTBase docs
spec:
x-request-filters:
- path: lib/security_response_header_filter.js
x-sub-request-filters:
- type: default
name: http
options:
allow:
- pattern: http://localhost/api.php
forward_headers: true
- pattern: http://localhost:8000
forward_headers: true
- pattern: http://localhost:10044
forward_headers: true
- pattern: /^https?:\/\//
paths:
/{domain:localhost}: &arbitary_domain
x-modules:
- path: projects/MR.yaml
options:
action:
# XXX Check API URL!
apiUriTemplate: http://localhost/api.php
# XXX Check the base RESTBase URI
baseUriTemplate: "{{'http://localhost:7231/{domain}/v1'}}"
parsoid:
# XXX Check Parsoid URL!
host: http://localhost:8000
mathoid:
# XXX Check Mathoid URL!
host: http://localhost:10044
table:
backend: sqlite
dbname: db.sqlite3
pool_idle_timeout: 20000
retry_delay: 250
retry_limit: 10
show_sql: false
/{domain:your.some.domain}: *arbitary_domain
# Finally, a standard service-runner config.
info:
name: restbase
logging:
name: restbase
level: info
- restbase/project/MR.yaml
# will work with Mathoid on any {domain}
paths:
/{api:v1}:
x-modules:
# swagger options, overriding the shared ones from the merged specs (?)
- spec:
info:
version: 1.0.0
title: Wikimedia REST API
description: Welcome to your RESTBase API.
#securityDefinitions: &wp/content-security/1.0.0
# mediawiki_auth:
# description: Checks permissions using MW api
# type: apiKey
# in: header
# name: cookie
# x-internal-request-whitelist:
# - /http:\/\/[a-zA-Z0-9\.]+\/w\/api\.php/
# Override the base path for host-based (proxied) requests. In our case,
# we proxy https://{domain}/api/rest_v1/ to the API.
#x-host-basePath: /api/rest_v1
x-route-filters:
- path: ./lib/normalize_title_filter.js
options:
redirect_cache_control: '{{options.purged_cache_control}}'
paths:
/media:
x-modules:
- path: v1/mathoid.yaml
options: '{{options.mathoid}}'
#/testing:
# x-modules:
# - path: ../mathpipe.yaml
# options: '{{options.mathoid}}'
/page:
x-modules:
- path: v1/content.yaml
options:
purged_cache_control: '{{options.purged_cache_control}}'
- path: v1/common_schemas.yaml # Doesn't really matter where to mount it.
/transform:
x-modules:
- path: v1/transform.yaml
options: '{{options}}'
/{api:sys}:
x-modules:
- spec:
paths:
/mathoid:
x-modules:
- path: sys/mathoid.js
options: '{{options.mathoid}}'
/table:
x-modules:
- path: sys/table.js
options:
conf: '{{options.table}}'
/key_value:
x-modules:
- path: sys/key_value.js
/key_rev_value:
x-modules:
- path: sys/key_rev_value.js
/key_rev_latest_value:
x-modules:
- path: sys/key_rev_latest_value.js
/post_data: &sys_post_data
x-modules:
- path: sys/post_data.js
/events:
x-modules:
- path: sys/events.js
options: '{{options.events}}'
/page_revisions:
x-modules:
- path: sys/page_revisions.js
/action:
x-modules:
- path: sys/action.js
options: '{{options.action}}'
/page_save:
x-modules:
- path: sys/page_save.js
/parsoid:
x-modules:
- path: sys/parsoid.js
options:
parsoidHost: '{{options.parsoid.host}}'
response_cache_control: '{{options.purged_cache_control}}'
options: '{{options}}'
- restbase/v1/mathoid.yaml
# Mathoid - math formula rendering service :simply replaced /wikimedia.org/sys to /{domain}/sys
tags:
- name: Math
description: formula rendering
paths:
/math/check/{type}:
post:
tags: ['Math']
summary: Check and normalize a TeX formula.
description: |
Checks the supplied TeX formula for correctness and returns the
normalised formula representation as well as information about
identifiers. Available types are tex and inline-tex. The response
contains the `x-resource-location` header which can be used to retrieve
the render of the checked formula in one of the supported rendering
formats. Just append the value of the header to `/media/math/{format}/`
and perform a GET request against that URL.
Stability: [stable](https://www.mediawiki.org/wiki/API_versioning#Stable).
produces:
- application/json
parameters:
- name: type
in: path
description: The input type of the given formula; can be tex or inline-tex
type: string
required: true
enum:
- tex
- inline-tex
- chem
- name: q
in: formData
description: The formula to check
type: string
required: true
responses:
'200':
description: Information about the checked formula
'400':
description: Invalid type
schema:
$ref: '#/definitions/problem'
default:
description: Error
schema:
$ref: '#/definitions/problem'
x-monitor: true
x-amples:
- title: Mathoid - check test formula
request:
params:
domain: wikimedia.org
type: tex
body:
q: E=mc^{2}
response:
status: 200
headers:
content-type: /^application\/json/
x-resource-location: /.+/
cache-control: 'no-cache'
body:
success: true
checked: /.+/
x-request-handler:
- get_from_sys:
request:
method: post
uri: /{domain}/sys/mathoid/check/{type}
headers: '{{ request.headers }}'
body: '{{ request.body }}'
/math/formula/{hash}:
get:
tags: ['Math']
summary: Get a previously-stored formula
description: |
Returns the previously-stored formula via `/media/math/check/{type}` for
the given hash.
Stability: [stable](https://www.mediawiki.org/wiki/API_versioning#Stable).
produces:
- application/json
parameters:
- name: hash
in: path
description: The hash string of the previous POST data
type: string
required: true
minLength: 1
responses:
'200':
description: Information about the checked formula
'404':
description: Data for the given hash cannot be found
schema:
$ref: '#/definitions/problem'
default:
description: Error
schema:
$ref: '#/definitions/problem'
x-monitor: false
x-request-handler:
- get_from_sys:
request:
method: get
uri: /{domain}/sys/mathoid/formula/{hash}
headers: '{{ request.headers }}'
/math/render/{format}/{hash}:
get:
tags: ['Math']
summary: Get rendered formula in the given format.
description: |
Given a request hash, renders a TeX formula into its mathematic
representation in the given format. When a request is issued to the
`/media/math/check/{format}` POST endpoint, the response contains the
`x-resource-location` header denoting the hash ID of the POST data. Once
obtained, this endpoint has to be used to obtain the actual render.
Stability: [stable](https://www.mediawiki.org/wiki/API_versioning#Stable).
produces:
- image/svg+xml
- application/mathml+xml
- image/png
parameters:
- name: format
in: path
description: The output format; can be svg or mml
type: string
required: true
enum:
- svg
- mml
- png
- name: hash
in: path
description: The hash string of the previous POST data
type: string
required: true
minLength: 1
responses:
'200':
description: The rendered formula
'404':
description: Unknown format or hash ID
schema:
$ref: '#/definitions/problem'
default:
description: Error
schema:
$ref: '#/definitions/problem'
x-monitor: false
x-setup-handler:
- init_svg:
uri: /{domain}/sys/key_value/mathoid.svg
body:
keyType: string
valueType: string
- init_mml:
uri: /{domain}/sys/key_value/mathoid.mml
body:
keyType: string
valueType: string
- init_png:
uri: /{domain}/sys/key_value/mathoid.png
body:
keyType: string
valueType: blob
x-request-handler:
- check_storage:
request:
method: get
uri: /{domain}/sys/key_value/mathoid.{$.request.params.format}/{$.request.params.hash}
headers:
cache-control: '{{ cache-control }}'
catch:
status: 404
return_if:
status: '2xx'
return:
status: 200
headers: "{{ merge({ 'cache-control': options.cache-control }, check_storage.headers) }}"
body: '{{ check_storage.body }}'
- postdata:
request:
uri: /{domain}/sys/mathoid/formula/{request.params.hash}
- mathoid:
request:
method: post
uri: /{domain}/sys/mathoid/render/{request.params.format}
headers:
content-type: application/json
x-resource-location: '{{ request.params.hash }}'
body: '{{postdata.body}}'
- Apache proxy config : /etc/httpd/conf.d/restbase.conf
#
# Restbase
#
<VirtualHost *:80>
ProxyPreserveHost On
ProxyPass /your.some.domain/ http://localhost:7231/your.some.domain/
ProxyPassReverse /your.some.domain/ http://localhost:7231/your.some.domain/
ProxyPass /localhost/ http://localhost:7231/localhost/
ProxyPassReverse /localhost/ http://localhost:7231/localhost/
# MediaWiki is located here:
DocumentRoot /var/www/html
</VirtualHost>
you can test the APIs at http://your.some.domain/your.some.domain/v1/
setup apache proxy
To load the images from the same location where the main wiki site is hosted a simple apache proxy can be used. Enable apache mod proxy ... and then add a config file along the lines of this file
cat /etc/apache2/sites-available/restbase.conf
<VirtualHost *:80>
ServerName restbase.formulasearchengine.com
ServerAlias api.formulasearchengine.com png.formulasearchengine.com
ServerAdmin wiki@physikerwelt.de
ProxyPreserveHost On
ProxyPass /v1/ http://localhost:8081/wikimedia.org/v1/
ProxyPass /wikimedia.org/v1/ http://localhost:8081/wikimedia.org/v1/
</VirtualHost>
a2ensite restbase.conf
You can use letsencrypt to also generate a HTTPS version for that site.
Helpful commands to check the state of the database
Clean cache
run cqlsh
use "local_group_default_T_mathoid_svg";
truncate data;
truncate meta;
use "local_group_default_T_mathoid_mml";
truncate data;
truncate meta;
use "local_group_default_T_mathoid_input";
truncate data;
truncate meta;
use "local_group_default_T_mathoid_png";
truncate data;
truncate meta;
Browse the data
Task retrieve the input given a key ... in this example we'll use 0f7fa7bb7d214c60f8d2f0b8d356c41bfe5c5d1d as key
run cqlsh
use "local_group_test_T_mathoid_input";
SELECT * from data WHERE key = '0f7fa7bb7d214c60f8d2f0b8d356c41bfe5c5d1d' and "_domain" = 'wikimedia.org';
_domain | key | tid | _del | content-location | content-sha256 | content-type | headers |latestTid | tags | value
---------------+------------------------------------------+--------------------------------------+------+------------------+----------------+------------------+-------------------------------------+-----------+------+----------------------------------------------------------------------------------------------------------------------------------------------------------------
wikimedia.org | 0f7fa7bb7d214c60f8d2f0b8d356c41bfe5c5d1d | e736aba4-9edc-11e5-927a-301f88a60cb9 | null | null | null | application/json | {"content-type":"application/json"} | null | null | {"q":"=nf(x)\\left({n-1 \\choose k-1}F(x)^{k-1}(1-F(x))^{(n-1)-(k-1)}-\\underbrace {{n-1 \\choose n}F(x)^{n}(1-F(x))^{(n-1)-n}} \\right)","type":"inline-tex"}
