The SGE Can Still Use Your Content Despite the Blocking of Google-Extended

A few weeks ago, Google released a new robots.txt directive to block Google-Extended to tell its algorithms not to use your content for Bard or other Google AI projects. 

However, Google recently announced that blocking Google-Extended does not prevent SGE’s AI from using your content. This means that your content may continue to appear in SGE-generated responses unless you completely block Googlebot.

In this article, we’ll delve deeper into this topic and explore the options available to you.

What is Google-Extended?

Google-Extended, Google’s new standalone product, allows you to tell the search engine via your robots.txt file not to use your site’s content for Bard and Vertex AI as well as other AI projects. 

However, it does not work for AI responses and snapshots provided in Search Generative Experience.

What is the link between SGE and Google-Extended?

If you search on Google, you might find that the Google Search Generative Experience (SGE) has transformed the way results are presented. 

Instead of seeing a simple website link at the top of the results, SGE can now provide you with answers directly on the search engine results page (SERP). This innovation is based on the integration of generative artificial intelligence.

SGE leverages advanced technologies such as Google Bard AI and Google Vertex AI to generate these instant responses. This improves the search experience by providing directly accessible information, which can be particularly convenient for simple, common queries.

However, it is important to note that the use of Google Bard AI to publicly index user conversations has raised data protection concerns. 

Some users have expressed concerns about the privacy of their information when processed by AI engines. This has led to debates about the need to ensure data security in this context.

To address this issue , Google recently introduced a new tool called “Google Extended” to give website owners greater control over how Google Bard and Google Vertex AI use their content. 

As mentioned in the first section, with Google-Extended you can prevent these tools from indexing and using your website content by adding a NOINDEX meta tag to your robots.txt file. 

Technical SEO: Technical SEO ensures that your local SEO company website is accessible to search engines. It involves optimizing site speed, mobile-friendliness, and other technical aspects.

This makes it clear to Google that you do not want your content to be included in the responses generated by Google’s AI tools except Search Generative Experience (SGE).

In fact, even if a website disallows Google-Extended in its robots.txt file, the content will still be used in SGE’s AI-generated responses. 

For example, here is an SGE AI-generated response that includes a map of the Rolling Stones website, despite excluding Google-Extended:

Here is a screenshot of their robots.txt file where Google-Extended is blocked:

These captures are proof that blocking Google-Extended has no impact on the use of your content by the SGE AI.

Google’s response 

Following this revelation, Google stated that the Google Search Generative Experience (SGE) is currently an experimental feature book market agency in USA is independent of Google Extended. 

In other words, EMS is considered a separate search feature, subject to the same guidelines as normal search results. 

So, if you prefer that your content not be included in the responses generated by the SGE, you only have one solution. This is to completely block indexing by Googlebot using the robots.txt file and the NOINDEX meta tag, which is probably not desirable. 

In summary

In conclusion, website administrators should be aware that using Google-Extended does not guarantee that their content will be excluded from AI-generated responses in Search Generative Experience. 

To prevent your content from appearing in the results provided by SGE, you should block the Googlebot user agent using the NOINDEX meta tag.

However, this action may prevent your content from appearing in traditional Google search results.

So you have no choice but to allow Google SGE to use your website content, potentially reducing direct organic traffic to it. Hopefully the situation changes in the future.