To avoid undesirable content in the search indexes, webmasters can instruct spiders not to crawl certain files or directories through the standard robots.txt file in the root directory of the domain. Additionally, a page can be explicitly excluded from a search engine's database by using a meta tag specific to robots (usually ). When a search engine visits a site, the robots.txt located in the root directory is the first file crawled. The robots.txt file is then parsed and will instruct the robot as to which pages are not to be crawled. As a search engine crawler may keep a cached copy of this file, it may on occasion crawl pages a webmaster does not wish crawled. Pages typically prevented from being crawled include login specific pages such as shopping carts and user-specific content such as search results from internal searches. In March 2007, Google warned webmasters that they should prevent indexing of internal search results because those pages are considered search spam.
23snaps Amikumu aNobii AsianAve Ask.fm Badoo Cloob Cyworld Diaspora Draugiem.lv Ello Facebook Foursquare Gab Hello Hi5 Highlight Houseparty Idka Instagram IGTV IRC-Galleria Keek LiveJournal Lifeknot LockerDome Marco Polo Mastodon MeetMe Meetup Miaopai micro.blog Minds MixBit Mixi Myspace My World Nasza-klasa.pl Nextdoor OK.ru Path Peach Periscope Pinterest Pixnet Plurk Qzone Readgeek Renren Sina Weibo Slidely Snapchat SNOW Spaces Streetlife StudiVZ Swarm Tagged Taringa! Tea Party Community TikTok Tinder Tout Tuenti TV Time Tumblr Twitter Untappd Vero VK Whisper Xanga Yo
As an Internet marketing strategy, SEO considers how search engines work, the computer programmed algorithms which dictate search engine behavior, what people search for, the actual search terms or keywords typed into search engines, and which search engines are preferred by their targeted audience. SEO is performed because a website will receive more visitors from a search engine the higher the website ranks in the search engine results page (SERP). These visitors can then be converted into customers.