http://3oamv2de6ooacy72hv3ulym3i3vvlprlkqas7hhzo4pu5nyxnji37zad.onion/projects/micro-search
You can add the following to your robots.txt file: User-Agent: IW-microsearch
Disallow: / I want avoid indexing specific pages This bot finds all pages that are withing the sitemap.xml , and will NEVER crawl through other links in your webpage. If you want to not index a page that is in your sitemap.xml file, do one of the following: Add <meta name="robots" content="noindex"> tag. Add <meta name="IW-microsearch" content="noindex"> tag. ( IW-microsearch is our user agent)...