Please Support Us, By White-Listing/Unblock us from your Ads-Block Programs so We Could keep Run This site For Free. We Promise that Our Advertisement Never Block Your Ways. Thank You So Much !!!


Robots.txt File Undercover

05/31/2017 2:30 PM by Ellys Kho in Tools

After creating a website there must be some version or some sensitive data of your web page that you will not want the search engine to crawl through and index those pages. So, it is needed to instruct the spiders not to crawl through the pages. Here comes the word, Robot Meta tags that are used to tell the search engine which folders and files on your website to avoid. Often this Robot Meta tags gets unnoticed by the search engine. So, here comes robots.txt file that is created to inform the search engine about the pages that you want them to view.

What is a Robots.txt file?

Don’t be mistaken it as an Html file. Robots.txt file is a text file the contains the list of the pages that you don’t want the search engine to visit. It tells the search engine which page not to visit. Robots.txt file is not compulsory for SEO. A question must arise in your mind that do search engine obey the rules? Yes, search engine obeys each and everything they asked not to do. Robots.txt file does not act as a firewall or not a password protection. Robots.txt file is very reliable and preventing the search engine from entering the page which you do not want them to crawl through and also avoid them from indexing and listing in the search engine. So, what is important in Robots.txt? The location in which the file is located is very important. The search engine will not search the whole site for the Robots.txt file. So, how to made it view to the search engine? Make sure that it is located in the main directory say namely "". And if they don’t find the file in the desired location then they will consider that they can crawl in each and every page and also index each page. So, be very careful and don’t forget to place the robots.txt file in the main directory.

how robots.txt file working

Robots.txt File Disallow

The structure of a robots.txt file record is exceptionally straightforward and adaptable as well. It contains a rundown of the client specialists and disallowed documents and catalogs or directories. User_agent are the search engine crawlers whereas disallow contains the list of files and directory that the maker does not the search engine to be viewed. The syntax of a Robots.txt file is as follows;



Do you want to add a comment line then just put #sign at the beginning of the structure? Then

here is the syntax for the same;





Whenever a search engine visits your website it will first search for the robots.txt file where the User_agent give a list of the entire agent. Suppose if you want to address all the robots then just place a * after the User_agent (User_agent: *). Here it will also come across disallow part that tells the search engine which pages are not meant for their visit. Disallow perform a great task on the website. A present just after the Disallow (as such; Disallow :/) can exclude the whole server whereas Disallow without a / can denote a free access to all the robots. If you want that the robots can view a part of the server then just type the name of the files after each Disallow syntax.

Robots.txt File Generator

Are you afraid of creating a robots.txt file of your own? Thinking of consulting any specialist? Then I can help you out with Robots.txt generator. It is a tool that can create robots.txt directives for your website. This generator will make your work easier. As stated earlier that the website spiders first crawl in your website and look for the Robot.txt directives. After finding the same, they read the file and identify those files and directories that are strictly prohibited to be accessed. Robots.txt generator cannot create the blocked files. Robots.txt generator can edit as well as create the robots.txt file for your website. So, what to do in the robots.txt file. It is very simple as you have to just enter the existing file in the space provided. You can also use the copy–paste option too. The tool is dealt with Allow so it is needed to enter the values. For the purpose of customizing the robots.txt file use of Allow and Disallow function is the most. You can also add a new directive to the list just by clicking on the Add directive option. Do you want to remove or edit? Then that is also possible with the option remove directive. Robots.txt files are very important as they ensure that Google and other existing search engine are indexing your website properly or not. If you are not sure that you can create an effective robots.txt file by your own then you can take help of the robots.txt generator. There are many such generators that can help you with the same. You can search on the search engine by the name of the robots.txt generator. There will be lists of many. Some of them come absolutely free whereas some charge you a little. These are not at all time-consuming site and can do your work in a fraction of a second. So, what to wait for and further why taking the risk? Visit the search engine and find out one that can be of your help.


leave a comment
Please post your comments here.

The Influence of the Search Engine Optimization on your moving business
Posted By Monica Smithers | Oct 12, 2017
Important Element Before you Designing your Web
Posted By Ellys Kho | Jul 19, 2017
Rewrite Articles with Article Rewriter tool?
Posted By Ellys Kho | Jul 19, 2017
Free Classifieds Sites List Without Registration
Posted By Ellys Kho | Jul 19, 2017
Installing WordPress & phpMyAdmin on Ubuntu 14.04 server
Posted By Ellys Kho | Jul 19, 2017
Search Engines Submission Sites List 2017
Posted By Ellys Kho | Jul 17, 2017
Why use Md5 hash generator?
Posted By Ellys Kho | Jul 10, 2017
Is Checking The Domain Name Age something important?
Posted By Ellys Kho | Jul 10, 2017

Ellys's Sponsors

Blog Directory

Web Statistics

Get your Website Analysis and Statistics, just input your URL below to get it free.

Powered by Web Statistics