add gptbot to robots.txt

Amolith created 2 years ago

Change summary

robots.txt | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)

Detailed changes

robots.txt

@@ -3,10 +3,10 @@ layout: null
 sitemap:
   exclude: 'yes'
 ---
-User-agent: *
-Disallow: /search
-
 User-agent: ia_archiver
 Disallow: /
 
+User-agent: GPTBot
+Disallow: /
+
 Sitemap: {{ site.url }}/sitemap.xml