php - Wordpress remove Robots Meta Tag noindex

Question

were experiencing a strange issue with a wordpress sites meta robots tag. All pages have the following meta tag and we cant seem to remove it

<meta name="robots" content="noindex,follow"/>

We have unchecked "Discourage search engines from indexing this site" in Settings > Reading > Search Engine Visibility but it does nothing.

We are using the Yoast SEO plugin but even when this is disabled the still remains. In fact, we have tried disabling all plugins to check nothing was interfering with it.

We have setup our Robots.txt file as follows:

User-agent: *
Disallow: 
Sitemap: http://speysidedistillery.co.uk/sitemap.xml

Im not sure if the Robots.txt takes precedent over the robots meta tag or not and there doesn't seem to be a definitive answer, as far as i can gather the most restrictive one will take precedent i.e in our case the meta tag.

This is mainly giving us issues with our google listing with the warning "A description for this result is not available because of this site's robots.txt" appearing instead of our sites description.

If worse comes to worst we can edit the wp_no_robots function in wp-includes/general-templates.php but would prefer to resolve this without editing the wp core files.

Any light anyone could shone on this would be great as we are at a loss, cheers

The site can be found at http://speysidedistillery.co.uk/

Answer 1

i Developed an E commerce site for a client and i was stacked with this issue for more than a week, all pages were indexed except for the Home Page.

After a long debugging of Yoast Seo Plugin, i got a solution for you guys.

SOLUTION The Homepage has a search page on it (for searching for products). By default, Yoast disables 'index' meta tag for Search Page and 404 Page. funny enough if your homepage has a search page or search functionality on it, your site may not be indexed by google or any other search engine.

To fix this default by Yoast SEO Plugin, follow the following steps:

STEP 1: Log into your cpanel and navigate to /public_html/yourwebsite.com/wp-content/themes/nameOfTheTheme/functions.php add this line of code to your theme's function.php file (at the top of the page)

 /**
  * EDITED BY OBOT ERNEST
  * REMOVES NOINDEX META TAG FROM HOME PAGE  & SEARCH PAGE IMPLEMENTED BY YOAST
 */


    add_filter('wpseo_robots', 'yoast_no_home_noindex', 999);
    function yoast_no_home_noindex($string= "") {
        if (is_home() || is_front_page() || is_search()) {
            $string= "index,follow";
        }
        return $string;
    }

After pasting the code, save and close file.

STEP 2 Note this second step is for those that are using any of the following SEO plugins (Yoast Premium, Yoast Free or wordpress-seo-premium plugin)

Navigate to: /public_html/youwebsite.com/wp-content/plugins/wordpress-seo-premium/frontend/class-frontend.php

replace the code at line 713 with this one below here:

    if ( is_search() ) {
$robots['index'] = 'index';
}
    if ( is_404() ) {
$robots['index'] = 'noindex';
}

Save file and close

Congratulations! you just made it. It working perfectly guys!