To avoid undesirable content in the search indexes, webmasters can instruct spiders not to crawl certain files or directories through the standard robots.txt file in the root directory of the domain. Additionally, a page can be explicitly excluded from a search engine's database by using a meta tag specific to robots (usually  ). When a search engine visits a site, the robots.txt located in the root directory is the first file crawled. The robots.txt file is then parsed and will instruct the robot as to which pages are not to be crawled. As a search engine crawler may keep a cached copy of this file, it may on occasion crawl pages a webmaster does not wish crawled. Pages typically prevented from being crawled include login specific pages such as shopping carts and user-specific content such as search results from internal searches. In March 2007, Google warned webmasters that they should prevent indexing of internal search results because those pages are considered search spam.[47]

An SEO technique is considered white hat if it conforms to the search engines' guidelines and involves no deception. As the search engine guidelines[18][19][52] are not written as a series of rules or commandments, this is an important distinction to note. White hat SEO is not just about following guidelines but is about ensuring that the content a search engine indexes and subsequently ranks is the same content a user will see. White hat advice is generally summed up as creating content for users, not for search engines, and then making that content easily accessible to the online "spider" algorithms, rather than attempting to trick the algorithm from its intended purpose. White hat SEO is in many ways similar to web development that promotes accessibility,[53] although the two are not identical.
Many brands are now heavily using this mobile app to boost their marketing strategy. Instagram can be used to gain the necessary momentum needed to capture the attention of the market segment that has an interest in the product offering or services.[54] As Instagram is supported by Apple and android system, it can be easily accessed by smartphone users. Moreover, it can be accessed by the Internet as well. Thus, the marketers see it as a potential platform to expand their brands exposure to the public, especially the younger target group. On top of this, marketers do not only use social media for traditional Internet advertising, but they also encourage users to create attention for a certain brand. This generally creates an opportunity for greater brand exposure.[55] Furthermore, marketers are also using the platform to drive social shopping and inspire people to collect and share pictures of their favorite products. Many big names have already jumped on board: Starbucks, MTV, Nike, Marc Jacobs, and Red Bull are a few examples of multinationals that adopted the mobile photo app early. Fashion blogger Danielle Bernstein, who goes by @weworewhat on Instagram, collaborated with Harper's Bazaar to do a piece on how brands are using Instagram to market their products, and how bloggers make money from it. Bernstein, who currently has one and a half million followers on Instagram, and whose "outfit of the day" photos on Snapchat get tens of thousands of screenshots, explained that for a lot of her sponsored posts, she must feature the brand in a certain number of posts, and often cannot wear a competitor's product in the same picture. According to Harper's Bazaar, industry estimates say that brands are spending more than $1 billion per year on consumer-generated advertising.[56] Founder of Instagram Kevin Systrom even went to Paris Fashion week, going to couture shows and meeting with designers to learn more about how style bloggers, editors, and designers are currently dominating much of the content on his application.[57]
Search engines use complex mathematical algorithms to interpret which websites a user seeks. In this diagram, if each bubble represents a website, programs sometimes called spiders examine which sites link to which other sites, with arrows representing these links. Websites getting more inbound links, or stronger links, are presumed to be more important and what the user is searching for. In this example, since website B is the recipient of numerous inbound links, it ranks more highly in a web search. And the links "carry through", such that website C, even though it only has one inbound link, has an inbound link from a highly popular site (B) while site E does not. Note: Percentages are rounded.
A navigational page is a simple page on your site that displays the structure of your website, and usually consists of a hierarchical listing of the pages on your site. Visitors may visit this page if they are having problems finding pages on your site. While search engines will also visit this page, getting good crawl coverage of the pages on your site, it's mainly aimed at human visitors.
With the development of this system, the price is growing under the high level of competition. Many advertisers prefer to expand their activities, including increasing search engines and adding more keywords. The more advertisers are willing to pay for clicks, the higher the ranking for advertising, which leads to higher traffic.[15] PPC comes at a cost. The higher position is likely to cost $5 for a given keyword, and $4.50 for a third location. A third advertiser earns 10% less than the top advertiser, while reducing traffic by 50%.[15] The investors must consider their return on investment and then determine whether the increase in traffic is worth the increase.
Website owners recognized the value of a high ranking and visibility in search engine results,[6] creating an opportunity for both white hat and black hat SEO practitioners. According to industry analyst Danny Sullivan, the phrase "search engine optimization" probably came into use in 1997. Sullivan credits Bruce Clay as one of the first people to popularize the term.[7] On May 2, 2007,[8] Jason Gambert attempted to trademark the term SEO by convincing the Trademark Office in Arizona[9] that SEO is a "process" involving manipulation of keywords and not a "marketing service."
×