We use cookies
We use cookies to optimize our website. By continuing to browse the site, you agree to our use of cookies.
New design
Control panel
  • Русский
  • Українська
  • English
  • UAH
  • USD
  • RUB
  • EUR
  • 0-800-307-307 Hotline
  • +38 (044) 392-74-33 Kiev
  • +38 (057) 728-39-00 Kharkiv
  • +38 (056) 794-38-31 Dnipro
  • +38 (032) 229-58-93 Lviv
  • +38 (048) 738-57-70 Odessa
  • +38(093) 170-15-42  Life
  • +38 (067) 400-88-44 Kievstar
  • +1(888)393-24-51  USA, Toll free
  • +44(131)507-01-14  Great Britain
  • +7 (499) 348-28-61 Moscow

2.39. Limiting indexing frequency

Indexing bots make frequent requests to site pages to update information and save it in their databases for subsequent issuance in search engines. These bots are useful, but too high a scan rate can cause unnecessary useless load.

For some indexing bots, you can configure restrictions:

To set the page scan interval, do the following:
  1. Log in or register, if you do not have an account yet, in Google Search Console.
  2. Add and confirm domain in this system, if it has not been verified.
  3. Go to the management page scan rate settings and select the resource you want:
  4. Set the radio button next to "Limit Google's maximum crawl speed", Set the desired value and click"Save»:
To configure the interval for scanning pages, do the following: - Log in or register, if you do not have an account yet, in Yandex.Webmaster.
  1. Add and confirm domain in this system, if it has not been verified.
  2. Pleaseselect the desired site and go to the section "Indexing → Bypass speed»:
  3. Set the radio button next to "Set manually", Set the desired value and click"Save»:
To set the page scan interval, do the following:
  1. Log in or register using social networks in Bing Webmaster.
  2. Add and confirm domain in this system, if it has not been verified.
  3. Select the site you want, go to the section "Configuration → Bypass control", Select the required site visit frequency template, or select"Customizable"And specify the frequency of indexing by hour yourself, then click on the button"Save changes»:

Attention!

Some bots may completely ignore the Crawl-delay directive.

As a rule, indexing bots follow the directives specified in robots.txtwhere you can set the indexing interval. To do this, do the following:

  1. Create or edit a file if it already exists, robots.txt in root directory of the site.
  2. In file robots.txt specify the directive crawl-delay:
    User-Agent: *
    Disallow:
    Crawl-delay: 3

    Instead * you can specify a specific User-Agent for which the indexing frequency should be limited. The Crawl-delay parameter sets the interval in seconds between requests from the bot.