WordPress Sitemap Slow? Cache Your Exclusion List With a Transient

A WordPress sitemap can quietly become the slowest page on your site. Mine got there because I excluded thin posts from it with a filter that did its checking on every request. This is the cached version I ended up with: about 40 lines of PHP, a transient, and a hook that clears it when you publish. I ran it on a site with 31,957 posts and on a small one, and the numbers below are what those runs printed.

The problem with checking every post on every request

Say you want posts under 100 words kept out of the sitemap. The obvious way is a filter that looks at each post when the sitemap is built. That works fine on 50 posts. On a large site, one slow filter gets run again for every sitemap page, and whatever is crawling your sitemap (a search engine, an audit tool, a monitoring service) asks for all of them in a row.

To measure the cost, I wrote the list-building part on its own and timed it on a live site with 31,957 published posts, reading only. Three runs in a row gave 3.67s, 3.60s and 3.82s, to find 7,516 posts under 100 words. A single scan like that, added to each of the 13 sitemap pages, is about 48 seconds of extra PHP per full crawl (3.7 x 13, so arithmetic rather than something I timed). Mine was never that tidy: the version I started with was slower still, and a crawl that fetched every page at once took the site down.

The fix: build the list once and keep it

Work out which post IDs to exclude, store the list in a transient for 12 hours, and hand it to the sitemap query. Here’s the whole thing, which goes in your theme’s functions.php or a small plugin:

function cb_build_sitemap_exclude_ids() {
    $ids  = array();
    $page = 1;
    do {
        $q = new WP_Query(array(
            'post_type'              => 'post',
            'post_status'            => 'publish',
            'posts_per_page'         => 500,
            'paged'                  => $page,
            'no_found_rows'          => true,
            'update_post_meta_cache' => false,
            'update_post_term_cache' => false,
        ));
        foreach ($q->posts as $p) {
            if (str_word_count(wp_strip_all_tags($p->post_content)) < 100) {
                $ids[] = $p->ID;
            }
        }
        $count = count($q->posts);
        $page++;
    } while ($count === 500);
    return $ids;
}

function cb_get_sitemap_exclude_ids() {
    $ids = get_transient('cb_sitemap_exclude_ids');
    if ($ids === false) {
        $ids = cb_build_sitemap_exclude_ids();
        set_transient('cb_sitemap_exclude_ids', $ids, 12 * HOUR_IN_SECONDS);
    }
    return $ids;
}

add_filter('wp_sitemaps_posts_query_args', function ($args) {
    $args['post__not_in'] = array_merge(
        isset($args['post__not_in']) ? (array) $args['post__not_in'] : array(),
        cb_get_sitemap_exclude_ids()
    );
    return $args;
});

add_action('save_post_post', function () {
    delete_transient('cb_sitemap_exclude_ids');
});
add_action('deleted_post', function () {
    delete_transient('cb_sitemap_exclude_ids');
});

Swap the word-count test for whatever your own rule is (a category, a custom field, a date). The 500 at a time, no_found_rows and the two cache flags are what keep the scan quick, because they stop WordPress running extra queries per post.

This uses WordPress’s built-in sitemaps (the wp-sitemap.xml ones). If you use Yoast, the same cached list can go through its wpseo_exclude_from_sitemap_by_post_ids filter instead, which is what I do on another site. I haven’t tested the Yoast route here, so check yours.

What I tested

The exclusion works, and the sitemap count adjusts. On the 31,957-post site, the unfiltered sitemap had 16 pages. With the 7,516 IDs passed in as post__not_in it had 13, and none of a sample of 300 excluded posts turned up in page 1.

Using the list costs almost nothing. Page 1 (2,000 URLs) took 0.15s without the filter and 0.15s with the ID list. Over three runs the range was 0.14s to 0.24s both ways, so I’d call it no difference. One caveat: in that benchmark I passed the ID list straight in, so it doesn’t include reading it back from the transient.

The transient itself works. I tested that part on a small site with the live code above. The first call built the list in 0.0028s and the second came straight from the transient, which showed as 0.0000s. Calling the save_post_post action emptied it, and the sitemap still produced its 13 URLs. A small site proves the mechanism, not the saving, which is why the big-site figures above are separate.

Things to watch

The first request after the cache expires pays the full cost. On my big site that’s the 3.7s scan, once every 12 hours, instead of on every request. If that bothers you, rebuild it from a cron job instead.
Clearing on save_post_post means every edit rebuilds the list on the next sitemap request. If you edit a lot of posts in a row, that’s still just one scan after the last one.
Don’t cache what depends on the visitor. A sitemap list is the same for everyone, which is why a transient is safe here.

The scan went from something I had to worry about to something I’d forgotten I had, which is the best thing a cache can do (the worst thing a cache can do is show you last week’s sitemap, hence the hook).