# /robots.txt file for http://webcrawler.com/ # mail webmaster@webcrawler.com for constructive criticism User-agent: * Allow: / Disallow: /_Dev Disallow: /_fpclass Disallow: /_private Disallow: /aspnet_client Disallow: /fpdb Disallow: /images Disallow: /redux Disallow: /ssfm Disallow: /stats Disallow: /THARONGAMES Disallow: /VHeyes Disallow: /Babylon-A-D_Milwaukee_1992.zip Disallow: /Wimpy Disallow: /flash Disallow: /syrym Disallow: /syrym_pics Disallow: /_wimpybutton Disallow: /_songs