Setting up the LiteSpeed Cache crawler
Sometimes when you visit your site, you may notice that the initial load takes significantly longer than normal, even though you're already using a caching plugin.
The cache is designed in a way so that the first user that hits an un-cached page will generate a fresh copy that will then be served to all subsequent visitors. During quiet periods however, there may be situations where a page is only visited frequently enough that each user encounters an expired cache and has to generate a fresh copy when they load the page, increasing the load times.
To get around this, we can utilise the LiteSpeed crawler to hit each page and generate a cached copy and then re-crawl the site every few hours replacing the cache.
Please Note: The LiteSpeed Crawler is only available on our Business hosting plans (Emerald and above).
Log in to your WordPress Dashboard
Navigate to the LiteSpeed Cache plugin
Select the "**Crawle**r" option
Navigate to the "**Sitemap Settings**" tab and enter your sitemap URL.
Tip: This will normally be "https://yourdomainname.tld/wp-sitemap.xml", however, this can change depending on the plugins you're using:All in one SEO : https://yourdomainname.tld/sitemap.xmlRankMath SEO : https://yourdomainname.tld/sitemap_index.xml
Once this has been added, you can navigate to "**General settings**" and set the crawler to "**ON**".
Then navigate to the "**Map**" tab and click the "**Refresh Crawler Map**" button.
Lastly, head to the "**Summary**" tab and select "**Manually Run**" to start the first crawl straight away.
Done!
The LiteSpeed crawler will initially do a first pass of every page on the site and ensure that this is cached. The first pass will normally take upwards of a few hours so you won't see the benefit immediately.
Once the first pass is complete, any further passes will only crawl pages that are un-cached or due to expire to ensure that every single page has a pre warmed cache to be served to human visitors.
*
*
The cache is designed in a way so that the first user that hits an un-cached page will generate a fresh copy that will then be served to all subsequent visitors. During quiet periods however, there may be situations where a page is only visited frequently enough that each user encounters an expired cache and has to generate a fresh copy when they load the page, increasing the load times.
To get around this, we can utilise the LiteSpeed crawler to hit each page and generate a cached copy and then re-crawl the site every few hours replacing the cache.
Setting up the LiteSpeed Crawler
Please Note: The LiteSpeed Crawler is only available on our Business hosting plans (Emerald and above).
Log in to your WordPress Dashboard
Navigate to the LiteSpeed Cache plugin
Select the "**Crawle**r" option
Navigate to the "**Sitemap Settings**" tab and enter your sitemap URL.
Tip: This will normally be "https://yourdomainname.tld/wp-sitemap.xml", however, this can change depending on the plugins you're using:All in one SEO : https://yourdomainname.tld/sitemap.xmlRankMath SEO : https://yourdomainname.tld/sitemap_index.xml
Once this has been added, you can navigate to "**General settings**" and set the crawler to "**ON**".
Then navigate to the "**Map**" tab and click the "**Refresh Crawler Map**" button.
Lastly, head to the "**Summary**" tab and select "**Manually Run**" to start the first crawl straight away.
Done!
The LiteSpeed crawler will initially do a first pass of every page on the site and ensure that this is cached. The first pass will normally take upwards of a few hours so you won't see the benefit immediately.
Once the first pass is complete, any further passes will only crawl pages that are un-cached or due to expire to ensure that every single page has a pre warmed cache to be served to human visitors.
*
*
Updated on: 24/01/2024
Thank you!