modify robots.txt

Amolith created 7 years ago

Change summary

robots.txt | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)

Detailed changes

robots.txt

@@ -4,6 +4,9 @@ sitemap:
   exclude: 'yes'
 ---
 User-agent: *
-Disallow:
+Disallow: /search
+
+User-agent: ia_archiver
+Disallow: /
 
 Sitemap: {{ site.url }}/sitemap.xml