To avoid undesirable content in the search indexes, webmasters can instruct spiders not to crawl certain files or directories through the standard robots.txt file in the root directory of the domain. Additionally, a page can be explicitly excluded from a search engine's database by using a meta tag specific to robots (usually ). When a search engine visits a site, the robots.txt located in the root directory is the first file crawled. The robots.txt file is then parsed and will instruct the robot as to which pages are not to be crawled. As a search engine crawler may keep a cached copy of this file, it may on occasion crawl pages a webmaster does not wish crawled. Pages typically prevented from being crawled include login specific pages such as shopping carts and user-specific content such as search results from internal searches. In March 2007, Google warned webmasters that they should prevent indexing of internal search results because those pages are considered search spam.[47]
In March 2006, KinderStart filed a lawsuit against Google over search engine rankings. KinderStart's website was removed from Google's index prior to the lawsuit, and the amount of traffic to the site dropped by 70%. On March 16, 2007, the United States District Court for the Northern District of California (San Jose Division) dismissed KinderStart's complaint without leave to amend, and partially granted Google's motion for Rule 11 sanctions against KinderStart's attorney, requiring him to pay part of Google's legal expenses.[70][71]
Social networking sites such as Facebook, Instagram, Twitter, MySpace etc. have all influenced the buzz of word of mouth marketing. In 1999, Misner said that word-of mouth marketing is, "the world's most effective, yet least understood marketing strategy" (Trusov, Bucklin, & Pauwels, 2009, p. 3).[79] Through the influence of opinion leaders, the increased online "buzz" of "word-of-mouth" marketing that a product, service or companies are experiencing is due to the rise in use of social media and smartphones. Businesses and marketers have noticed that, "a persons behaviour is influenced by many small groups" (Kotler, Burton, Deans, Brown, & Armstrong, 2013, p. 189). These small groups rotate around social networking accounts that are run by influential people (opinion leaders or "thought leaders") who have followers of groups. The types of groups (followers) are called:[80] reference groups (people who know each other either face-to-face or have an indirect influence on a persons attitude or behaviour); membership groups (a person has a direct influence on a person's attitude or behaviour); and aspirational groups (groups which an individual wishes to belong to).