User-agent: * Disallow: /Pages/ # robots.txt created 4/19/06 by Judy Schaefer to prevent spiders from indexing # duplicate content on x-cart sites that also have the static html catalog # generated. this prevents all spiders from crawling the dynamically generated # x-cart pages. they will get to all your catalog (and other) pages via # index.html. place this file at the root level in your website. if you'd like # to suggest changes or improvements to this please email judy@i-stuff.com. User-agent: * # this means ALL spiders Disallow: /home.php # these are your dynamic category pages Disallow: /product.php # these are your dynamic product pages