robots.txt analysis
The Crawlability tab shows every AI crawler in Mavel’s bot catalog and each bot’s current status against your domain’s robots.txt:
For each bot, Mavel shows the matched rule: the exact directive in your
robots.txt that produced the verdict. This makes it straightforward to identify
and fix accidental blocks.
Three filter dropdowns let you narrow the catalog by platform (OpenAI,
Anthropic, Google, Perplexity, Microsoft, Meta, Apple, Amazon, Bytedance,
Cohere, Mistral, X, You.com, Common Crawl, Baidu, and others), bot type (Training,
Search, User query, Other), and status (Allowed, Blocked, Partial).
The Reload robots.txt button re-fetches and re-analyses your domain’s
robots.txt on demand. The View robots.txt link opens the raw file in a new
tab so you can inspect the directives directly.
URL Tester
The URL Tester tab lets you test any specific URL against any bot in the catalog. Enter a URL and select a bot; Mavel fetches the robots.txt for that URL’s domain and reports whether the selected bot is allowed to crawl that path, including the matching rule. This is useful for checking specific pages (a product page, a comparison article, a press release) rather than relying on the domain-wide summary.Crawl Insights: measured bot visits
The Crawl Insights page shows what AI bots are actually doing on your site, not just what your robots.txt says they are allowed to do. This requires connecting a log provider. Once connected, Crawl Insights shows:- Total bot hits over your selected time range (7, 30, or 90 days), with a delta against the prior period of the same length.
- Unique bots: how many distinct crawlers visited.
- Top bot: the single highest-volume crawler in the period.
- Last hit: when the most recent crawl event was recorded.
- A daily bot hits chart showing visit volume over time, broken down by bot.
- A top 20 bots table showing each bot’s hit count, last seen date, and share of total traffic.
Connecting a log provider
Crawl Insights is powered by crawl events forwarded from your server logs. Supported providers include Cloudflare Workers, Vercel Log Drains, and CSV upload. The Log provider tile on the Crawl Insights page shows your connection status. Until a provider is connected, the Crawl Insights page shows no data; robots.txt analysis works immediately without any integration.robots.txt analysis tells you what crawlers are instructed to do; Crawl
Insights tells you what they actually do. A bot marked Allowed may never
visit if your pages are not discoverable, and a blocked bot may crawl
anyway. Use both views together.
What to fix first
If you see a Blocked verdict for a major AI crawler:1
Open your robots.txt
Use the View robots.txt button to see the raw file. Look for a
Disallow: / under the bot’s user-agent, or under User-agent: * if
there is no specific rule.2
Add an explicit allow rule
Add a
User-agent: block for the specific crawler (e.g. GPTBot) with
Allow: /. This overrides the wildcard without changing your rules for
other bots.3
Reload and verify
Use the Reload robots.txt button on the Crawlability page to re-analyse
after your changes deploy. Confirm the status changes to Allowed.