> ## Documentation Index
> Fetch the complete documentation index at: https://docs.palagus.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# AI Traffic

> See which AI crawlers visit your website, how many visitors AI assistants send you, and what your robots.txt lets AI bots read.

**AI Traffic** is in the sidebar under **Analytics**. It has two tabs:

* **AI crawlers**: which AI bots visit your website, from your Cloudflare zone. Below it, the robots.txt check.
* **AI referrals**: how many visits reach your website from AI assistants such as ChatGPT, Perplexity and Gemini, from Google Analytics 4.

Each tab is connected on the tab itself. Workspace owners and admins connect and disconnect. Everyone else in the workspace sees the numbers, and the tab says who can connect.

AI Traffic is included in every plan unless your plan leaves it out. Then the sidebar entry stays, locked, and the page shows the way to Plans.

## Connect Cloudflare

Palagus reads how often known AI bots requested pages of your website. It needs a Cloudflare API token that can **read the analytics of one zone**, nothing more.

> [!NOTE]
>
> The token reads. It cannot change your zone, your DNS or your settings. Palagus needs no write access of any kind, and installs nothing in your Cloudflare account.

1. In the Cloudflare dashboard, go to **My Profile** → **API Tokens** and select **Create Token**.
2. Give the token a name, for example `Palagus AI Traffic`.
3. Under **Policies**, open **Start from scratch** and select **Start with a Zone policy**. Not a template: each template grants more than Palagus needs.
4. In the zone policy, keep **Include**, select **Specific zone**, and select your website's zone. Not **All zones**.
5. In the list of permission groups, open **Analytics & Logs** and select **Read** next to **Analytics**. Nothing else.
6. Select **Continue to summary**, check the summary, and select **Create Token**.
7. Copy the token. Cloudflare shows it only once.
8. Copy your **Zone ID**: go to **Domains**, select your domain, and scroll to the **API** section at the bottom of its **Overview** page.

Then open **AI Traffic** in Palagus. On the **AI crawlers** tab, select **Connect Cloudflare**, and enter the Zone ID, your domain and the token. Palagus checks the token before it stores it. A token that cannot read this zone is refused, and nothing is stored.

<video autoPlay loop muted playsInline controls aria-label="Creating a Cloudflare API token that reads one zone's analytics, copying the Zone ID, and connecting Cloudflare in Palagus" poster="https://media.palagus.ai/mcp/recordings/cloudflare-connect.poster.9af31ae37a4b.webp">
  <source src="https://media.palagus.ai/mcp/recordings/cloudflare-connect.43c372363419.webm" type="video/webm" />

  <source src="https://media.palagus.ai/mcp/recordings/cloudflare-connect.43c372363419.mp4" type="video/mp4" />
</video>

The token gets a zone policy with **Analytics** **Read** for one zone. The Zone ID comes from the domain's **Overview** page. Back in Palagus, the tab shows **Cloudflare connected** and when the first numbers come.

## Connect Google Analytics

Palagus reads how many sessions of your Google Analytics 4 property came from an AI assistant.

1. Open **AI Traffic** and go to the **AI referrals** tab.
2. Select **Connect Google Analytics**.
3. Sign in with a Google account that can view your GA4 property, and allow read access to Google Analytics.
4. Back in Palagus, pick your property and select **Connect property**.

If Google warns you that it has not verified Palagus, you can still continue. Palagus asks only for read access to your analytics. It cannot change your property.

## When the numbers come

Palagus fetches new numbers once a day. Right after you connect, it also fetches the last 30 days, which takes about twenty minutes, and the tab says so until they are in. After that, it shows when the numbers were last updated.

If a fetch finds nothing, the tab says so: the connection works, there were simply no visits in that period.

## What the numbers mean

**AI crawlers** counts requests from known AI bots, per bot and day. Each bot shows its vendor and its type:

| Type | What the bot does |
| :- | :- |
| Training | Collects pages to train AI models. |
| Search | Builds the index an AI assistant searches when it answers. |
| User request | Fetches a page right now, because someone asked an AI assistant about it. |
| Other | Another kind of AI bot, or one whose vendor does not say what it is for. |

<video autoPlay loop muted playsInline controls aria-label="The AI crawlers tab: the total, the chart per day, and each bot with its vendor and type" poster="https://media.palagus.ai/mcp/recordings/ai-traffic-bots.poster.b39c66e80936.webp">
  <source src="https://media.palagus.ai/mcp/recordings/ai-traffic-bots.62c830ec7550.webm" type="video/webm" />

  <source src="https://media.palagus.ai/mcp/recordings/ai-traffic-bots.62c830ec7550.mp4" type="video/mp4" />
</video>

**AI referrals** counts sessions in Google Analytics whose source is an AI assistant, per assistant and day. The days are the days of your GA4 property's time zone.

Each total is compared with the period before it, of the same number of days.

The periods are whole days, up to yesterday: from the last 7 days to all time, no hours. Today is still running, so it is not counted yet. In the chart, a day without visits shows as 0. A day Palagus did not fetch, for example one more than 30 days before you connected, has no point, and the line breaks there.

## Where the numbers are incomplete

* **AI referrals is a lower bound.** Google Analytics knows where a visit came from only when the app passes that on. Many apps, and many links copied out of a chat, do not. Those visits count as direct traffic in Google Analytics, and Palagus cannot see them.
* **AI crawlers counts what passes through Cloudflare.** Requests to parts of your site that Cloudflare does not proxy are not counted.
* **A bot is recognised by the name it sends.** A bot that does not send its own name is not counted, and new bots appear only once Palagus knows them.

## Check robots.txt

Below the bot numbers on the **AI crawlers** tab, the **Access for AI bots** card reads the public `robots.txt` of your project's website. It shows, for each AI bot:

* **Allowed**: the bot may read every page.
* **Partially blocked**: some pages are closed to it.
* **Blocked**: the whole site is closed to it.

It also shows which rule decides: a group for that bot, the group for all bots (`*`), or no rule at all. This check needs no connection.

<video autoPlay loop muted playsInline controls aria-label="The Access for AI bots check: each bot Allowed, Partially blocked or Blocked, with the rule that decides" poster="https://media.palagus.ai/mcp/recordings/ai-traffic-access.poster.0d76b5c5eb99.webp">
  <source src="https://media.palagus.ai/mcp/recordings/ai-traffic-access.4f39ac54c099.webm" type="video/webm" />

  <source src="https://media.palagus.ai/mcp/recordings/ai-traffic-access.4f39ac54c099.mp4" type="video/mp4" />
</video>

If your website has no `robots.txt`, every bot may read every page. If the file cannot be read, the page says so. If the project has no website, or its website is an IP address rather than a domain, there is nothing to check.

## Disconnect

On the tab, select **Disconnect** and confirm. Palagus stops fetching and deletes the stored access. The numbers fetched so far stay on the page, marked as **not connected anymore**. You can connect again at any time.

You can always disconnect, even when your plan no longer includes AI Traffic.

If Cloudflare or Google stops accepting the connection, for example because the token was deleted or access was revoked, the tab says **Connection expired – reconnect**. Connect again to keep the numbers up to date.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.