PDF Publication Title:
Text from PDF Page: 087
Figure 4.30: Sample configuration of robots.txt file and its location. If I had wanted to be more specific as to what web crawlers I wanted to grant access to crawl my site, I could have configured the settings of the robots.txt as seen in Figure 4.31 below. Using this configuration I have specifically stated that I’m only allowing Google, Yahoo! and Microsoft’s web crawlers to index the entire site, and I have disallowed Baidu’s web crawler (China’s most popular search engine) from indexing my site. In addition, I have disallowed all access to the /secure/ folder to all crawlers. Figure 4.31: Sample configuration of the robots.txt file. To summarize this section on effective use and configuration of the robots.txt file, below are some best practices to keep in mind: • The file must be named robots.txt. • Place robots.txt in the root directory of the website. • Make sure proper configuration is in place to allow or disallow selected directories to get crawled; having the incorrect configuration may prevent your site from getting indexed, which means none of your page will appear in the search results. 76PDF Image | ANALYSIS OF THE APPLICATION OF SELECTED SEARCH ENGINE OPTIMIZATION EFFECTIVENESS ON GOOGLE’S SEARCH RANK
PDF Search Title:
ANALYSIS OF THE APPLICATION OF SELECTED SEARCH ENGINE OPTIMIZATION EFFECTIVENESS ON GOOGLE’S SEARCH RANKOriginal File Name Searched:
An_Analysis_of_the_Application_of_Selected_SEO_Techniques.pdfDIY PDF Search: Google It | Yahoo | Bing
Cruise Ship Reviews | Luxury Resort | Jet | Yacht | and Travel Tech More Info
Cruising Review Topics and Articles More Info
Software based on Filemaker for the travel industry More Info
The Burgenstock Resort: Reviews on CruisingReview website... More Info
Resort Reviews: World Class resorts... More Info
The Riffelalp Resort: Reviews on CruisingReview website... More Info
CONTACT TEL: 608-238-6001 Email: greg@cruisingreview.com (Standard Web Page)