Adding properly written robots.txt file to your website is one of the most important seo tasks. It is basically an instruction to search engines, on how to crawl a particular website. When a search engine decide to crawl a website, it first looks for robots.txt file. In this video you can learn to create a perfect robots.txt file for a website.

Useful Links

Learn more about robots.txt file –
Reference –
Testing & validating- (need to sign up for webmaster tools)

Below is an example robots.txt file i use for a wordpress website.

User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /xmlrpc.php
Disallow: /wp-content/plugins/
Disallow: /wp-content/themes/
Disallow: /*.php$
Disallow: /*.inc$

Allow: /wp-content/uploads/
Allow: /wp-content/themes/storefront/
Allow: /wp-includes/js/