Manual:Robot policy

From Linux Web Expert

Robot policy is used to tell web crawlers and other tools what to exclude.

Configuration settings