An index bloat occurs when a website has many low-value pages being indexed by the search engine. Often, such pages are generated automatically with little or no distinctive content at all. The presence of such URLs tends to have a snowballing effect on the entire SEO process. Some of the common URLs that lead to index bloats include pagination pages, disorganized archive pages, filter combinations drawn from faceted navigation, disorderly parameter pages and expired pages. Others are uncontrolled tag pages, auto-generated profiles that have minimal content, non-optimized on-site result pages, and inconsistency between www and non-www.
The Effects
Index bloats are capable of reducing the crawl efficiency of search engines as Google bots have to slum through low-value paths. This, in turn, reduces the indexing speed of new content as well as re-crawling of content that is updated and has SEO value. Index bloats have duplicate content and are known to cause cannibalization of keywords.
As multiple pages compete for similar search intent on one site, search engines get confused as they try to figure out which is the most relevant page as ranking signals get split across multiple URLs. This has a negative effect on a site’s ranking ability. Where low-quality pages get high ranking, users are disappointed, this also hurts the brand of the company that owns the website. If your website is experiencing index bloat, here are 7 ways that Joel House SEO Expert can help deindex your website pages:
1. Use Rel=Canonical Links
If your website has duplicate URL content, a rel=canonical link will suggest to the search engine which duplicate URLs should be indexed. Should the tag be accepted, alternative pages – which are the duplicates with lower value, get crawled but this happens less frequently. It is also excluded from the main index which means their ranking signals are passed on to the preferred indexed page. However, for these pages to be accepted, their content should be highly similar and their URLs will have to be crawled then be processed by the search engine. This process can be quite slow.
2. Use Password Protection
Protecting the files you have in a server using passwords can keep search engines from indexing them. This is because their URLs cannot be indexed, passed, or crawled on any search engine ranking signal. However, this blocks users as well. This means deindexing is limited to the content that you decide to push behind a log-in.
A requirement for deindexing is search engines make an attempt to crawl URL paths, verify that they are no longer welcome then subsequently do away with the content. This is a time consuming process because the higher the number of URLs crawled, the more the search engines note that no value is returned from the craw and the less it the crawl queue will prioritize URLs of a similar nature.
3. Use the 301 Redirect Approach
If you identify that the index bloat in your website is as a result of having multiple pages that target the same topic, you can use 301 redirects approach to merge all of them into just one page and consolidate all their ranking signals.
For search engines to deindex redirected pages, they have to crawl their original URL, take note of the 301 status code, including the destination URL to their crawl queue then process the content to establish its equivalence in nature. Once this is confirmed, the ranking signals are passed on without any dilution. This process can be slow where the destination URL has a low priority on the crawl queue.
The process becomes even much slower where there are redirect chains. Of importance to note is that pages that are redirected to irrelevant pages like the homepage are treated as soft 404 by search engines such as Google and are not passed on the ranking signals.
4. Use of URL Parameter Tools
Search engines handle URL parameters differently. For instance, in Google’s Search Console, it is possible to specify how the Googlebot ought to handle URL parameters. But there are three key disadvantages of the URL parameter tool to be aware of.
The tool works only when the URL is parameter-based, it only controls crawling and it does not address other search engines other than Google. Though the URL parameter tool does not directly control indexing, is a user specifies ‘No Crawl’ on the parameter, the URLs are ultimately dropped from the index.
However, this comes with a cost – if the Googlebot is not able to crawl, then the signals cannot be processed. As a result, ranking is affected or internal links may be extracted and added to the craw queue – leading to a slowdown in site indexing.
5. Do away with URLs Tool
If you urgently want to deindex pages from Google, your first option should be to do away with URLs tools. Requests for this are often processed the same day they get submitted. The only challenge with this approach is that it is only a temporary solution. Removal requests that are successful last for about 90 days before content reappears in SERPs. Because of this, the best use case is when you need to block a page urgently but have no resources to do so.
6. Use Noindex Tags
To completely block a web page from getting indexed, applying noindex Meta tags or x-robots is the best approach. However, it is advisable not to use noindex directives in robots.txt because these are not respected by search engines. Noindex directives have cascading impacts in that they prevent additions and deindexing from search engines once they are processed. They also cause noindexed URLs to get crawled fewer times, they stop attribution of ranking signals to URLs and if they present for extended periods of time, they result in a ‘nofollow’ of page links. This means Google does not add such links to crawl queues and that ranking signals are not passed to the linked pages.
7. Use Robots.txt
A disallow directive present in the robots.txt file informs search engines the pages they should not crawl. Just like the URL parameter, this approach does not directly control indexing. Where a page has been linked from other locations on the web, search engines may find it necessary to index it. In addition, the use of robots.txt to block pages does not provide an indication of how Google will treat URLs that are currently indexed.