John Xina a8853cdc59 add robots.txt support, default to strict policy 1 gadu atpakaļ
..
robots_relaxed.txt a8853cdc59 add robots.txt support, default to strict policy 1 gadu atpakaļ
robots_strict.txt a8853cdc59 add robots.txt support, default to strict policy 1 gadu atpakaļ