Website At 100: Robots

<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="background-color: white; color: #666666; font-family: 'Droid Sans'; font-size: 12px; line-height: 22px; margin-bottom: 18px; text-align: justify;">
Did you know that through <code style="font-family: monospace, monospace; font-size: 1em; white-space: pre-wrap; word-wrap: break-word;">robots.txt</code> you can control which post or page of your site to crawl or not to crawl? and if you’re currently worried about duplicate content on your site, well not a problem folks, <code style="font-family: monospace, monospace; font-size: 1em; white-space: pre-wrap; word-wrap: break-word;">robots.txt</code> is also helpful to avoid duplicate content, yes it is possible to avoid duplicate content via <code style="font-family: monospace, monospace; font-size: 1em; white-space: pre-wrap; word-wrap: break-word;">robots.txt</code> by not crawling site directories, like tag, archive and category these can cause duplicate content.</div>
<div style="background-color: white; color: #666666; font-family: 'Droid Sans'; font-size: 12px; line-height: 22px; margin-bottom: 18px; text-align: justify;">
Implementing good <a href="http://websiteat100.com/blog/category/seo/" style="-webkit-transition: background-color 0.2s ease 0s, border 0.2s ease 0s, color 0.2s ease 0s, opacity 0.2s ease-in-out 0s; color: black; line-height: inherit; text-decoration: none; transition: background-color 0.2s ease 0s, border 0.2s ease 0s, color 0.2s ease 0s, opacity 0.2s ease-in-out 0s;" title="Search engine optimization blog by websiteat100.com">SEO </a><code style="font-family: monospace, monospace; font-size: 1em; white-space: pre-wrap; word-wrap: break-word;">robots.txt</code> will help you get high traffic in most <a href="http://websiteat100.com/blog/category/seo/" style="-webkit-transition: background-color 0.2s ease 0s, border 0.2s ease 0s, color 0.2s ease 0s, opacity 0.2s ease-in-out 0s; color: black; line-height: inherit; text-decoration: none; transition: background-color 0.2s ease 0s, border 0.2s ease 0s, color 0.2s ease 0s, opacity 0.2s ease-in-out 0s;" title="Search engine blog by websiteat100.com">search engines</a> or even receive more paying in relevant ads.</div>
<h3 style="background-color: white; color: #333333; font-family: 'Droid Sans'; font-size: 16px; font-weight: 200; letter-spacing: -0.1px; line-height: 20px; margin-top: 0px; padding: 0px; text-align: justify;">
robots.txt</h3>
<div style="background-color: white; color: #666666; font-family: 'Droid Sans'; font-size: 12px; line-height: 22px; margin-bottom: 18px; text-align: justify;">
Copy and paste the below snippet into <code style="font-family: monospace, monospace; font-size: 1em; white-space: pre-wrap; word-wrap: break-word;">robots.txt</code> of your site.</div>
<div data-settings=" minimize scroll-mouseover" id="crayon-52a6b30b49906821597345" style="background-color: white; color: #666666; font-family: 'Droid Sans'; font-size: 12px; line-height: 22px; text-align: justify;">
<div data-settings=" show">
Apache</div>
<div>
</div>
<div>
<table style="border-collapse: collapse; border-spacing: 0px;"><tbody>
<tr><td data-settings="show" style="border: none; color: #333333; font-size: 12px; line-height: 18px; padding: 9px 10px; vertical-align: top;"><div data-line="crayon-52a6b30b49906821597345-1">
1</div>
<div data-line="crayon-52a6b30b49906821597345-2">
2</div>
<div data-line="crayon-52a6b30b49906821597345-3">
3</div>
<div data-line="crayon-52a6b30b49906821597345-4">
4</div>
<div data-line="crayon-52a6b30b49906821597345-5">
5</div>
<div data-line="crayon-52a6b30b49906821597345-6">
6</div>
<div data-line="crayon-52a6b30b49906821597345-7">
7</div>
<div data-line="crayon-52a6b30b49906821597345-8">
8</div>
<div data-line="crayon-52a6b30b49906821597345-9">
9</div>
<div data-line="crayon-52a6b30b49906821597345-10">
10</div>
<div data-line="crayon-52a6b30b49906821597345-11">
11</div>
<div data-line="crayon-52a6b30b49906821597345-12">
12</div>
<div data-line="crayon-52a6b30b49906821597345-13">
13</div>
<div data-line="crayon-52a6b30b49906821597345-14">
14</div>
<div data-line="crayon-52a6b30b49906821597345-15">
15</div>
<div data-line="crayon-52a6b30b49906821597345-16">
16</div>
<div data-line="crayon-52a6b30b49906821597345-17">
17</div>
<div data-line="crayon-52a6b30b49906821597345-18">
18</div>
<div data-line="crayon-52a6b30b49906821597345-19">
19</div>
<div data-line="crayon-52a6b30b49906821597345-20">
20</div>
<div data-line="crayon-52a6b30b49906821597345-21">
21</div>
<div data-line="crayon-52a6b30b49906821597345-22">
22</div>
<div data-line="crayon-52a6b30b49906821597345-23">
23</div>
<div data-line="crayon-52a6b30b49906821597345-24">
24</div>
<div data-line="crayon-52a6b30b49906821597345-25">
25</div>
<div data-line="crayon-52a6b30b49906821597345-26">
26</div>
<div data-line="crayon-52a6b30b49906821597345-27">
27</div>
<div data-line="crayon-52a6b30b49906821597345-28">
28</div>
<div data-line="crayon-52a6b30b49906821597345-29">
29</div>
<div data-line="crayon-52a6b30b49906821597345-30">
30</div>
<div data-line="crayon-52a6b30b49906821597345-31">
31</div>
<div data-line="crayon-52a6b30b49906821597345-32">
32</div>
<div data-line="crayon-52a6b30b49906821597345-33">
33</div>
<div data-line="crayon-52a6b30b49906821597345-34">
34</div>
<div data-line="crayon-52a6b30b49906821597345-35">
35</div>
</td><td style="border: none; color: #333333; font-size: 12px; line-height: 18px; padding: 9px 10px; vertical-align: top;"><div id="crayon-52a6b30b49906821597345-1">
# Now most search engine support auto discovery of your sitemap.xml via robots.txt</div>
<div id="crayon-52a6b30b49906821597345-2">
Sitemap: http://www.example.com/sitemap.xml</div>
<div id="crayon-52a6b30b49906821597345-3">
</div>
<div id="crayon-52a6b30b49906821597345-4">
# Crawls pages for the image index</div>
<div id="crayon-52a6b30b49906821597345-5">
# Google Image</div>
<div id="crayon-52a6b30b49906821597345-6">
User-agent: Googlebot-Image</div>
<div id="crayon-52a6b30b49906821597345-7">
Disallow:</div>
<div id="crayon-52a6b30b49906821597345-8">
Allow: /*</div>
<div id="crayon-52a6b30b49906821597345-9">
</div>
<div id="crayon-52a6b30b49906821597345-10">
#  Crawls website pages to determine AdSense content</div>
<div id="crayon-52a6b30b49906821597345-11">
# Google AdSense</div>
<div id="crayon-52a6b30b49906821597345-12">
User-agent: Mediapartners-Google*</div>
<div id="crayon-52a6b30b49906821597345-13">
Disallow:</div>
<div id="crayon-52a6b30b49906821597345-14">
</div>
<div id="crayon-52a6b30b49906821597345-15">
# digg mirror</div>
<div id="crayon-52a6b30b49906821597345-16">
User-agent: duggmirror</div>
<div id="crayon-52a6b30b49906821597345-17">
Disallow: /</div>
<div id="crayon-52a6b30b49906821597345-18">
</div>
<div id="crayon-52a6b30b49906821597345-19">
# global</div>
<div id="crayon-52a6b30b49906821597345-20">
User-agent: *</div>
<div id="crayon-52a6b30b49906821597345-21">
Disallow: /cgi-bin/</div>
<div id="crayon-52a6b30b49906821597345-22">
Disallow: /wp-admin/</div>
<div id="crayon-52a6b30b49906821597345-23">
Disallow: /wp-includes/</div>
<div id="crayon-52a6b30b49906821597345-24">
Disallow: /wp-content/plugins/</div>
<div id="crayon-52a6b30b49906821597345-25">
Disallow: /wp-content/cache/</div>
<div id="crayon-52a6b30b49906821597345-26">
Disallow: /wp-content/themes/</div>
<div id="crayon-52a6b30b49906821597345-27">
Disallow: /trackback/</div>
<div id="crayon-52a6b30b49906821597345-28">
Disallow: /feed/</div>
<div id="crayon-52a6b30b49906821597345-29">
Disallow: /comments/</div>
<div id="crayon-52a6b30b49906821597345-30">
Disallow: /category/*/*</div>
<div id="crayon-52a6b30b49906821597345-31">
Disallow: */trackback/</div>
<div id="crayon-52a6b30b49906821597345-32">
Disallow: */feed/</div>
<div id="crayon-52a6b30b49906821597345-33">
Disallow: */comments/</div>
<div id="crayon-52a6b30b49906821597345-34">
Disallow: /*?</div>
<div id="crayon-52a6b30b49906821597345-35">
Allow: /wp-content/uploads/</div>
</td></tr>
</tbody></table>
</div>
</div>
<h3 style="background-color: white; color: #333333; font-family: 'Droid Sans'; font-size: 16px; font-weight: 200; letter-spacing: -0.1px; line-height: 20px; margin-top: 0px; padding: 0px; text-align: justify;">
header.php trick</h3>
<div style="background-color: white; color: #666666; font-family: 'Droid Sans'; font-size: 12px; line-height: 22px; margin-bottom: 18px; text-align: justify;">
Add this snippet in <code style="font-family: monospace, monospace; font-size: 1em; white-space: pre-wrap; word-wrap: break-word;">header.php</code> of your current theme, we simply add conditional statement which robots to display in our blog.</div>
<div data-settings=" minimize scroll-mouseover" id="crayon-52a6b30b49910680535660" style="background-color: white; color: #666666; font-family: 'Droid Sans'; font-size: 12px; line-height: 22px; text-align: justify;">
<div data-settings=" show">
PHP</div>
<div>
</div>
<div>
<table style="border-collapse: collapse; border-spacing: 0px;"><tbody>
<tr><td data-settings="show" style="border: none; color: #333333; font-size: 12px; line-height: 18px; padding: 9px 10px; vertical-align: top;"><div data-line="crayon-52a6b30b49910680535660-1">
1</div>
<div data-line="crayon-52a6b30b49910680535660-2">
2</div>
<div data-line="crayon-52a6b30b49910680535660-3">
3</div>
<div data-line="crayon-52a6b30b49910680535660-4">
4</div>
<div data-line="crayon-52a6b30b49910680535660-5">
5</div>
<div data-line="crayon-52a6b30b49910680535660-6">
6</div>
<div data-line="crayon-52a6b30b49910680535660-7">
7</div>
<div data-line="crayon-52a6b30b49910680535660-8">
8</div>
<div data-line="crayon-52a6b30b49910680535660-9">
9</div>
<div data-line="crayon-52a6b30b49910680535660-10">
10</div>
<div data-line="crayon-52a6b30b49910680535660-11">
11</div>
<div data-line="crayon-52a6b30b49910680535660-12">
12</div>
<div data-line="crayon-52a6b30b49910680535660-13">
13</div>
<div data-line="crayon-52a6b30b49910680535660-14">
14</div>
</td><td style="border: none; color: #333333; font-size: 12px; line-height: 18px; padding: 9px 10px; vertical-align: top;"><div id="crayon-52a6b30b49910680535660-1">
<?php //for home, post, page and category ?></div>
<div id="crayon-52a6b30b49910680535660-2">
<?php if(is_single() || is_page() || is_category() || is_home()) { ?></div>
<div id="crayon-52a6b30b49910680535660-3">
  <meta name=”robots” content=”all,noodp” /></div>
<div id="crayon-52a6b30b49910680535660-4">
<?php } ?></div>
<div id="crayon-52a6b30b49910680535660-5">
</div>
<div id="crayon-52a6b30b49910680535660-6">
<?php //for archive page ?></div>
<div id="crayon-52a6b30b49910680535660-7">
<?php if(is_archive()) { ?></div>
<div id="crayon-52a6b30b49910680535660-8">
  <meta name=”robots” content=”noarchive,noodp” /></div>
<div id="crayon-52a6b30b49910680535660-9">
<?php } ?></div>
<div id="crayon-52a6b30b49910680535660-10">
</div>
<div id="crayon-52a6b30b49910680535660-11">
<?php //for search and 404 page ?></div>
<div id="crayon-52a6b30b49910680535660-12">
<?php if(is_search() || is_404()) { ?></div>
<div id="crayon-52a6b30b49910680535660-13">
  <meta name=”robots” content=”noindex,noarchive” /></div>
<div id="crayon-52a6b30b49910680535660-14">
<?php } ?></div>
</td></tr>
</tbody></table>
</div>
</div>
<h3 style="background-color: white; color: #333333; font-family: 'Droid Sans'; font-size: 16px; font-weight: 200; letter-spacing: -0.1px; line-height: 20px; margin-top: 0px; padding: 0px; text-align: justify;">
Don’t want to get your hand dirty?</h3>
<div style="background-color: white; color: #666666; font-family: 'Droid Sans'; font-size: 12px; line-height: 22px; margin-bottom: 18px; text-align: justify;">
Don’t worry folks as always I will provide plugin from other developer, if there’s any </div>
</div>

ROBOTS.TXT SEO FOR YOUR WORDPRESS BLOG

Did you know that through robots.txt you can control which post or page of your site to crawl or not to crawl? and if you’re currently w...