Manual:Robot policy

From Linux Web Expert

The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

Robot policy is used to tell web crawlers and other tools what to exclude.

Configuration settings