Typically, you want search engines to crawl and index all the pages on your site. This ensures that your site shows up during searches.
Indexing – The collecting, parsing, and storage of data to facilitate fast and accurate information. Gathering information (crawling) by itself is useful, but it is not useful if it cannot be displayed at the right time with the right results. Indexing is the art that accomplishes this.
Crawling – Search engine bots systematically browse all of the content on the World Wide Web. Crawlers (such as Googlebot) follow known links, crawl for new links, and update dead links. They go from link to link and bring data back to the search engine's servers.
In some cases you may not want your content to be found by search engines and users. Examples include:
- Draft content
- Sensitive or confidential information
- One-time event pages
In these cases, you can easily modify your settings.
Modifying Crawling and Indexing Settings in Site Manager
You can manage your Crawling and Indexing Settings in the Site Manager. These settings control how and what Google and other search engines do with your page by inserting specific instructions in the page code that search engines can follow.
- In Page Builder, go to the category of the page you'd like to change.
- Select the gear icon to the far right of the page title under Settings.
- Click the SEO tab.
- Make your selection from the Crawling and Indexing Setting menu:
If you have any questions or need help, send us a note at firstname.lastname@example.org!