I was searching the other day for a robots.txt file example and it took me a few minutes to find some fragmented samples in forums. So here is a quick post for you wordpress bloggers who are managing your own Search Engine Optimization..SEO.
Joomla fortunately, automatically sets up a robot.txt file when you do your install through cpanel.
With wordpress for some reason, you need to do it manually. I find this a bit bizarre especially since WP espouses to be very search engine friendly right out of the gate. Anyways here is a a basic robot.txt sample.
Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /category/*/* Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /*?* Disallow: /*? Allow: /wp-content/uploads # Google Image User-agent: Googlebot-Image Disallow: Allow: /* # Google AdSense User-agent: Mediapartners-Google* Disallow: Allow: /* # digg mirror User-agent: duggmirror Disallow: / Sitemap: http://www.example.com/sitemap.xml
Copy and paste this into a file named robots.txt;
You can use notepad, word or any text program will work. Then, upload the this to the root level of your domain. Another way is to just add a plugin to your wordpress blog. There are several robot.txt plugins to choose from, some will add a meta robots tag to your wordpress pages.
Here is another more comprehensive wordpress robots.txt example.
For more tips on robot.txt and search engine optimization you can post a question in the comments section.
Note, if you are a building blog, wordpress has the best user interface and is great for fast preformatted content development. However if you are building a website with several different types of content that need function, I would check out Joomla 1.5 and it’s joomla seo extensions.
What CMS platform do you prefer for building the best search engine optimization into your website, Joomla, Drupal or WordPress?
