Robots, spiders, and search engines use this file to detmine which 
content they should *not* crawl while indexing your website.
This system is called "The Robots Exclusion Standard."
It is strongly encouraged to use a robots.txt validator to check
for valid syntax before any robots read it!
Examples:
Restrict Google and MSN from indexing your images.
User-agent: Googlebot
Disallow: /images/
User-agent: MSNBot
Disallow: /images/
User-agent: Baidu Spider
Disallow: /images/
User-agent: proximic
Disallow: /registration.php/
User-agent: proximic
Disallow: /online.php/
User-agent: proximic
Disallow: /vbsuper_pm.php/
User-agent: google
Disallow: /registration.php/
User-agent: google
Disallow: /online.php/
User-agent: google
Disallow: /vbsuper_pm.php/
User-agent: google
Disallow: /vbsuper_pm.php?do=auto_reply&action=get_autoreply/
User-agent: proximic
Disallow: /vbsuper_pm.php?do=auto_reply&action=get_autoreply/
User-agent: proximic
Disallow: /thanks.php?do=statistics/
User-agent: google
Disallow: /thanks.php?do=statistics/
User-agent: proximic
Disallow: /private.php/
User-agent: google
Disallow: /private.php/

User-agent: BoardTracker
Disallow: /
User-agent: Gigabot
Disallow: /
User-agent: Twiceler
Disallow: /
User-agent: Slurp
Crawl-delay: 2
User-agent: msnbot
Crawl-delay: 2
User-agent: *
User-agent: *
Disallow: *.js
Disallow: /clientscript/
Disallow: /cpstyles/
Disallow: /customavatars/
Disallow: /customprofilepics/
Disallow: /images/
Disallow: /forum/ajax.php
Disallow: /forum/attachment.php
Disallow: /forum/album.php
Disallow: /forum/thanks.php
Disallow: /forum/thanks.php?do=statistics
Disallow: /forum/calendar.php
Disallow: /forum/cron.php
Disallow: /forum/editpost.php
Disallow: /forum/global.php
Disallow: /forum/image.php
Disallow: /forum/inlinemod.php
Disallow: /forum/joinrequests.php
Disallow: /forum/login.php
Disallow: /forum/member.php
Disallow: /forum/memberlist.php
Disallow: /forum/misc.php
Disallow: /forum/private.php
Disallow: /forum/online.php
Disallow: /forum/online.php?s=
Disallow: /forum/moderator.php
Disallow: /forum/newattachment.php
Disallow: /forum/newreply.php
Disallow: /forum/newthread.php
Disallow: /forum/poll.php
Disallow: /forum/post.php
Disallow: /forum/postings.php
Disallow: /forum/printthread.php
Disallow: /forum/private.php
Disallow: /forum/profile.php
Disallow: /forum/register.php
Disallow: /forum/report.php
Disallow: /forum/reputation.php
Disallow: /forum/search.php
Disallow: /forum/sendmessage.php
Disallow: /forum/showgroups.php
Disallow: /forum/subscription.php
Disallow: /forum/showpost.php
Disallow: /forum/threadrate.php
Disallow: /forum/usercp.php
Disallow: /forum/usernote.php