For website owners

About CGMSearchBot

If you found CGMSearchBot in your server logs, this is CGMIMM's search-index crawler. Here's exactly what it does and how to control it.

How to identify it

Our crawler always sends this user-agent string:

CGMSearchBot/1.0 (+https://www.cgmimm.com/bot)

The +https://www.cgmimm.com/bot portion links back to this page so you can verify who's visiting.

What it does

  • •Crawls pages so they can be indexed and surfaced in CGMIMM search results — typically pages a site owner has submitted by URL or via an XML sitemap.
  • •Reads each page's title, meta description, canonical URL, main heading, Open Graph image, language, and structured data (JSON-LD) types.
  • •Stores a short excerpt of the visible text so the page can be indexed.
  • •Does not run JavaScript — it reads the HTML your server returns, the same as a sitemap-style fetch.
  • •Does not submit forms, log in, or follow links into checkout, account, or admin areas.

How it behaves

  • •Respects robots.txt. If you disallow it, it won't crawl those paths.
  • •Honors a 10-second crawl delay by default, so it won't hammer your server.
  • •Obeys noindex robots meta tags — pages marked noindex are skipped from indexing.
  • •Identifies itself honestly. We never disguise the bot as a regular browser.

Allow or block it

To allow the bot everywhere (the default — no action needed), or to be explicit:

User-agent: CGMSearchBot
Allow: /

To block it entirely, add this to your robots.txt:

User-agent: CGMSearchBot
Disallow: /

To block a specific section only:

User-agent: CGMSearchBot
Disallow: /private/
Disallow: /drafts/

Questions or problems

If the bot is causing issues on your site, or you have questions about how your pages are crawled, reach us through our contact page. Include the affected URLs and a snippet of your server logs and we'll take a look.