JournalTOCs Blog

News and Opinions about current awareness on new research

Why publishers should never NOINDEX their RSS feeds

without comments


Last week, JournalTOCs stopped indexing all of the 40 journals published by OA Publishing London because this publisher took the unusual and illogical measure of requesting aggregators not to index (aggregate) the RSS feeds for the current issues of its journals. Tables of Contents from the OA Publishing London journals will no longer be updated at JournalTOCs. Those who have been following any of the 40 journals will not be able to keep up with new issues.

Why would OA Publishing London want to stop aggregators and search engines from crawling and collecting its RSS feeds? Years ago, it might just have made some sense using the noindex meta-tag for RSS feeds, but nowadays there is no need to noindex such feeds. Google and the rest of modern search engines can easily identify RSS feeds and they act on that by not including RSS feeds in web search results.

Publishers should, in reality, very much want their RSS feeds to be indexed, because it can help aggregators and search engines to direct users to where the newest content is. Search engines are smart enough to understand the difference between a feed and webpage, and use the feed as a pointer to the webpage where the real source of the content resides. Allowing search engines to index RSS feeds is therefore an important way to drive traffic to the webpages of the actual content.

There is no scenario in which a publisher is not interested in having their latest content indexed. Old feeds generators, such as the deprecated Feedburner, still provide users with the outdated option to noindex feeds to prevent them from being penalized by search engines. Publishers need to be reassured that that it is no longer an issue, and indexed feeds do not create penalty situations. Google itself will normally not show RSS feeds in search results.

The noindex meta-tag is not good for publishers. Any publisher who wants to enable RSS readers, aggregators and APIs to reuse details of their content should make sure to remove the noindex meta-tag from their RSS pages and from their software that generates RSS feeds.

The noindex meta-tag to be removed looks like this:

<meta name="robots" content="noindex">

This code tells search engines and aggregators that they should not index or crawl the content of the RSS feeds.

So, if you want the abstracts of your latest publications to be indexed by JournalTOCs, search engine, aggregator or any web service, and thus ensure that hundreds of thousands of potential readers can discover your content, you should make sure you ARE NOT using the noindex meta-tag.

The noindex meta-tag can help in search engine optimization (SOA) but it should be used wisely, rather than simply assuming that it’s always a good idea to use it. noindex should only be used for web pages you don’t want showing up in search results or want to hide from the external world. For example a test page, archive page, or something similar that is not relevant for the publisher’s business; these should have the noindex tag, so that they don’t end up taking the place of the real important pages in search results (Google’s algorithm tends to avoid placing multiple links from the same domain on the front page (unless the website has a good ranking)).

For optimal crawling, Google recommends using also RSS/Atom feeds

RSS pages (feeds) are not only relevant pages; they are used by the search engines and aggregators to redirect users to your relevant webpages! They help to market your real content. They are good for everyone, including readers, authors, end users and for your business.

Written by Santiago Chumbe

January 26th, 2015 at 5:07 pm