> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mavel.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Crawlability

> Whether AI crawlers can actually read your site: robots.txt analysis and measured bot visit tracking.

Being cited starts with being crawlable. If an AI engine's crawler is blocked
by your robots.txt (even by a wildcard rule that was never meant to exclude
AI), it cannot read your pages, and your content cannot be cited in answers.

Mavel's crawlability tools cover two distinct questions: what your robots.txt
rules say AI crawlers are allowed to do, and what AI bots are actually doing
on your site.

## robots.txt analysis

The **Crawlability** tab shows every AI crawler in Mavel's bot catalog and
each bot's current status against your domain's robots.txt:

| Status  | Meaning                                                                                                  |
| ------- | -------------------------------------------------------------------------------------------------------- |
| Allowed | The bot can crawl your site (explicit allow rule, wildcard, or no rule, which is the robots.txt default) |
| Blocked | The bot is explicitly excluded from crawling your site                                                   |
| Partial | Some paths are disallowed; the bot can reach some content but not all                                    |

For each bot, Mavel shows the matched rule: the exact directive in your
robots.txt that produced the verdict. This makes it straightforward to identify
and fix accidental blocks.

Three filter dropdowns let you narrow the catalog by **platform** (OpenAI,
Anthropic, Google, Perplexity, Microsoft, Meta, Apple, Amazon, Bytedance,
Cohere, Mistral, X, You.com, Common Crawl, Baidu, and others), **bot type** (Training,
Search, User query, Other), and **status** (Allowed, Blocked, Partial).

The **Reload robots.txt** button re-fetches and re-analyses your domain's
robots.txt on demand. The **View robots.txt** link opens the raw file in a new
tab so you can inspect the directives directly.

## URL Tester

The **URL Tester** tab lets you test any specific URL against any bot in the
catalog. Enter a URL and select a bot; Mavel fetches the robots.txt for that
URL's domain and reports whether the selected bot is allowed to crawl that
path, including the matching rule.

This is useful for checking specific pages (a product page, a comparison
article, a press release) rather than relying on the domain-wide summary.

## Crawl Insights: measured bot visits

The **Crawl Insights** page shows what AI bots are actually doing on your
site, not just what your robots.txt says they are allowed to do. This requires
connecting a log provider.

Once connected, Crawl Insights shows:

* **Total bot hits** over your selected time range (7, 30, or 90 days), with
  a delta against the prior period of the same length.
* **Unique bots**: how many distinct crawlers visited.
* **Top bot**: the single highest-volume crawler in the period.
* **Last hit**: when the most recent crawl event was recorded.
* A **daily bot hits chart** showing visit volume over time, broken down by bot.
* A **top 20 bots table** showing each bot's hit count, last seen date, and
  share of total traffic.

## Connecting a log provider

Crawl Insights is powered by crawl events forwarded from your server logs.
Supported providers include Cloudflare Workers, Vercel Log Drains, and CSV
upload. The Log provider tile on the Crawl Insights page shows your connection
status. Until a provider is connected, the Crawl Insights page shows no data;
robots.txt analysis works immediately without any integration.

<Note>
  robots.txt analysis tells you what crawlers are instructed to do; Crawl
  Insights tells you what they actually do. A bot marked Allowed may never
  visit if your pages are not discoverable, and a blocked bot may crawl
  anyway. Use both views together.
</Note>

## What to fix first

If you see a **Blocked** verdict for a major AI crawler:

<Steps>
  <Step title="Open your robots.txt">
    Use the View robots.txt button to see the raw file. Look for a
    `Disallow: /` under the bot's user-agent, or under `User-agent: *` if
    there is no specific rule.
  </Step>

  <Step title="Add an explicit allow rule">
    Add a `User-agent:` block for the specific crawler (e.g. `GPTBot`) with
    `Allow: /`. This overrides the wildcard without changing your rules for
    other bots.
  </Step>

  <Step title="Reload and verify">
    Use the Reload robots.txt button on the Crawlability page to re-analyse
    after your changes deploy. Confirm the status changes to Allowed.
  </Step>
</Steps>

A **Partial** verdict usually means a path-level disallow. Use the URL Tester
to check specific pages and identify exactly which paths are blocked.