How to stop indexing feed urls in search console?: A Comprehensive Guide

Having a smooth, hassle-free online presence is crucial for every website owner. One such nuisance that many website owners face, especially those using WordPress, is the unexpected “/feed/” URLs appearing in Google’s Search Console. This article delves deep into understanding this issue and presents a straightforward solution for those battling it.

What is this “/feed/” URL?

A “/feed/” URL is an RSS feed automatically generated by certain Content Management Systems (CMS) like WordPress. RSS (Really Simple Syndication) is a type of web feed that allows users and applications to access updates to websites in a standardized, computer-readable format. These feeds can, at times, get indexed by search engines like Google, leading to potential visibility issues.

How to fix the “/feed/” issue via robots.txt?

Steps to Block “/feed/” URLs in Google Search Console using robots.txt:

1. Access Your robots.txt File: This file is typically located in the root directory of your website, e.g., “https://example.com/robots.txt”.
2. Add the Disallow Directive: To block all “/feed/” URLs, add the following line to your robots.txt file:Disallow: /*/feed/
3. Save and Upload: Once you’ve made the changes, save the robots.txt file and re-upload it to your website’s root directory.
4. Verify in Google Search Console: After updating the robots.txt file, head over to Google Search Console. Use the ‘robots.txt Tester’ tool to ensure your “/feed/” URLs are blocked.
5. Wait: Google will eventually re-crawl your website and adhere to the new robots.txt directives.

Here is prepared for you ready to copy and paste robots.txt content that block /feed/ URL/s.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /*/feed/
Disallow: /*/rss/

Accessing robots.txt via Rank Math SEO

1. Log in to WordPress: Navigate to your WordPress dashboard by entering your credentials.
2. Go to Rank Math: On the left-hand sidebar, hover over “Rank Math.” From the dropdown menu, click on “General Settings.”
3. Access robots.txt Editor: In the settings, find and click on the “Edit robots.txt” tab or option.
4. Edit as Needed: Here, you can view, edit, or add rules to your robots.txt file. Once done with the changes, make sure to save them.

Accessing robots.txt via FTP

FTP allows you to directly interact with the files on your website server. This method requires you to have FTP credentials and an FTP client (like FileZilla, Cyberduck, etc.).

Steps:

1. Open Your FTP Client: Launch your preferred FTP client. For this example, we’ll assume you’re using FileZilla.
2. Connect to Your Website Server: Enter your FTP credentials – typically, this includes the host (often your website URL), username, password, and port (usually 21 for FTP). Then, click “Connect.”
3. Navigate to Root Directory: Once connected, on the right-hand side of the FTP client, navigate to the root directory of your website. This is often named public_html, www, or named directly after your website.
4. Find the robots.txt File: Within the root directory, look for the robots.txt file. It should be easily visible as this file resides in the root.
5. Edit the File: Right-click on the robots.txt file and choose the edit option. Make the necessary changes and save the file. If you’re making changes outside the FTP client (e.g., in a text editor), make sure to upload the modified file back to the root directory, replacing the old one.
6. Disconnect: Once done, safely disconnect from the server using your FTP client.

Note: Always backup your website before making changes, especially when working directly with server files through FTP. This ensures that you can quickly restore your site to its original state if something goes wrong.

By understanding both methods, you can choose the one that best suits your comfort and needs. Whether you prefer the user-friendly interface of a plugin or the direct control provided by FTP, you now have the knowledge to access and edit your robots.txt file with confidence.

How to Disable Generating RSS Feeds in WordPress

While WordPress does not have a built-in option to disable RSS feeds, you can achieve this using simple code or a plugin.

a) Using Code:

Insert the following code into the functions.php file of your theme:

function itsme_disable_feed() {
    wp_redirect( home_url(), 302 );
    exit;
}
 
add_action('do_feed', 'itsme_disable_feed', 1);
add_action('do_feed_rdf', 'itsme_disable_feed', 1);
add_action('do_feed_rss', 'itsme_disable_feed', 1);
add_action('do_feed_rss2', 'itsme_disable_feed', 1);
add_action('do_feed_atom', 'itsme_disable_feed', 1);
add_action('do_feed_rss2_comments', 'itsme_disable_feed', 1);
add_action('do_feed_atom_comments', 'itsme_disable_feed', 1);

After adding the aforementioned code, any user attempting to access the RSS feed URLs will be redirected to the main page of your website.

b) Using a Plugin:

There are plugins, such as “Disable Feeds”, that you can install and activate to turn off RSS feeds.

What does the feed content comprise?

The feed typically contains:

• Metadata about the feed itself (like the encoding, version, etc.)
• Details about the website or specific webpage the feed is related to (title, link, and sometimes a brief description)
• Timestamps indicating when the feed was last updated
• Information on how frequently the feed is updated

In our given example, it’s an RSS feed containing comments for an article on ‘Search Experience Optimization (SXO)’.

Why is it an issue?

While RSS feeds are useful for certain applications and platforms, they might not be content that website owners want to be indexed by search engines for various reasons:

• Irrelevance for Typical Searchers: The feed’s content may not provide valuable or relevant information to typical users searching on Google.
• Potential for Duplicate Content: The feed might contain snippets or full versions of content already present on the main website, leading to potential duplicate content issues.
• Aesthetic Concerns: Feeds are meant to be machine-readable and might not be visually appealing or user-friendly when accessed directly through search results.

Is blocking “/feed/” URLs harmful to my website?

No. Blocking these URLs simply prevents search engines from indexing them. It doesn’t affect the functionality of the RSS feeds for those who use them.

Can I selectively block certain feeds but allow others?

Absolutely. Modify the “Disallow” directive in the robots.txt to target specific feed URLs you wish to block.

What if I want to use my feed for marketing or other purposes?

If you intend to use your feed actively, consider keeping it open but monitor its performance and relevance in search results.

Are there other ways to block “/feed/” URLs besides using robots.txt?

Yes, other methods involve meta tags, HTTP headers, or plugins specific to your CMS. But using robots.txt is the most straightforward approach.

Will removing feed and Modifying robots.txt Affect My Site’s Visibility and SEO Negatively?

It’s a common concern among website owners to wonder if certain changes can have unintended consequences on their site’s SEO. When considering the addition of code to redirect RSS feed URLs and modifying the robots.txt file, here’s what you should know:

1. Redirecting RSS Feed URLs: By redirecting RSS feed URLs to your homepage, you’re essentially preventing these feeds from being indexed by search engines. For most websites, RSS feed URLs aren’t primary sources of organic traffic, so redirecting them usually doesn’t have a significant impact on SEO. However, if you have a substantial number of subscribers or third-party applications relying on these feeds, you should inform them of the changes.
2. Modifying robots.txt: The primary purpose of the robots.txt file is to guide search engine bots on which pages or directories of your website should not be crawled. As long as you’re specific in your directives (e.g., disallowing only the /feed/ path), it shouldn’t harm your overall site visibility. Ensure you don’t accidentally block essential parts of your site that you want to be indexed.
3. General SEO Considerations: Always monitor your site’s performance using tools like Google Search Console after making significant changes. This will help you detect any potential issues early on and take corrective actions if needed.

/feed/ URL’s – conclusion

The appearance of “/feed/” URLs in Google Search Console might be puzzling at first, but with a clear understanding and a few steps, it’s a challenge easily overcome. By optimizing your website’s visibility and ensuring only relevant content is indexed, you’re on the path to a more streamlined and effective online presence.

If you’re still facing issues with this or prefer not to make these changes yourself, we’re here to assist. Let us handle it for you!