Weebly is a website builder platform just like blogger or wordpress. You can design an ecommerce site, or business sites, portfolios or personal sites using this platform.
Here is a tip on how to use Robots.txt and Robots meta tags.
Robots.txt and Robots Meta Tags
The robots exclusion standard, also known as robots exclusion protocol lets you control what pages can be crawled and indexed by search engines. There are two main ways for you to use make use of this standard, one is through the robots.txt and the other is through robots meta tags.
Using Robots.txt
The robots.txt file is a page published on your site that gives search engine bots crawling directions. This file is often used to protect private pages from getting crawled by search engines. Your robots.txt file will automatically be created when you publish your site.
To customize it and exclude content from search engine crawling go to the Pages tab > Page you want to edit > SEO Settings > Hide this page from search engines and it will add the page to the robots.txt file and it will not be crawled by search engines. You can see your robots.txt file by adding robots.txt to your domain. For example www.yourdomain.com/robots.txt
Robots Meta Tags
Robots meta tags also let you control how search engines access your site content. While robots.txt controls the actual ability to crawl your page, the robots meta tag tells search engines whether to index your site after crawling. This is an important distinction because if another site links to a page blocked by robots.txt it could still get indexed. You can use robots meta tags as a way to make sure that whenever a search bot tries to crawl your page, even from a link, it will be told not to index the page.
Let's take a look at the two most important robots meta tags and how to add them to your site in Weebly.
No Index Meta Tag
This tag tells search engines not to index the page it's on. This is the best way to keep your page out of search results. Adding in Weebly is easy:
No Follow Meta Tag
This tag tells search engines not to follow any links that you've placed on the page. You can add it using the same steps as the no index tag but with slightly different html.
Here is a tip on how to use Robots.txt and Robots meta tags.
Robots.txt and Robots Meta Tags
The robots exclusion standard, also known as robots exclusion protocol lets you control what pages can be crawled and indexed by search engines. There are two main ways for you to use make use of this standard, one is through the robots.txt and the other is through robots meta tags.
Using Robots.txt
The robots.txt file is a page published on your site that gives search engine bots crawling directions. This file is often used to protect private pages from getting crawled by search engines. Your robots.txt file will automatically be created when you publish your site.To customize it and exclude content from search engine crawling go to the Pages tab > Page you want to edit > SEO Settings > Hide this page from search engines and it will add the page to the robots.txt file and it will not be crawled by search engines. You can see your robots.txt file by adding robots.txt to your domain. For example www.yourdomain.com/robots.txt
Robots Meta Tags
Robots meta tags also let you control how search engines access your site content. While robots.txt controls the actual ability to crawl your page, the robots meta tag tells search engines whether to index your site after crawling. This is an important distinction because if another site links to a page blocked by robots.txt it could still get indexed. You can use robots meta tags as a way to make sure that whenever a search bot tries to crawl your page, even from a link, it will be told not to index the page.
Let's take a look at the two most important robots meta tags and how to add them to your site in Weebly.
No Index Meta Tag
- Go to the Pages Tab > Select Your Page > SEO Settings > Header
- Copy paste the following html <meta name="robots" content="noindex" />
No Follow Meta TagThis tag tells search engines not to follow any links that you've placed on the page. You can add it using the same steps as the no index tag but with slightly different html.
- Go to the Pages Tab > Select Your Page > SEO Settings > Header
- Copy paste the following html <meta name="robots" content="nofollow" />
source: WeeblyBlogs/PNGehow