Your AI app slows down seconds after you share a link. Did customers arrive—or did machines fetch it first? Here's a runnable way to separate the signals before deciding what to fix.
levelsio reported a revealing test: a URL buried in a reply drew the first non-X crawler 8.40 seconds later. He had noticed pages slowing after sharing new features and asked Claude to investigate. He described thousands of crawlers, including news-alert and social-listening services, alongside X's near-immediate preview fetch.
That's his account; the underlying logs and crawler ownership aren't independently verified here. The useful distinction survives: preview fetching, ongoing monitoring and customer activity serve different purposes. Request counts alone don't tell you which one grew.
Here's a completed SYNTHETIC example for one URL, with equal 60-second windows before/after sharing. A tiny local Python run produced:
Before: unknown UA → 2 requests, 0 errors, p95 100ms.
After: preview UA → 2 requests, 0 errors, p95 100ms.
After: crawler UA → 3 requests, 0 errors, p95 140ms.
After: unknown UA → 2 requests, 2 server errors, p95 2100ms.
In this toy case, inspect the failing requests first. It doesn't demonstrate crawler-caused overload. A quiet crawler bucket and a failing app path call for different fixes.
Copy the code below into a file named triage. Python 3, standard library only; no account, package install or API call. Input fields: t = seconds relative to sharing; status = HTTP status; ms = response time; ua = user-agent. Export one URL's requests in this CSV shape.
Python (save as triage):
import csv, math, sys
from collections import defaultdict
groups = defaultdict(list)
with open(sys.argv[1], newline="", encoding="utf-8-sig") as f:
for r in csv.DictReader(f):
t, ms = float(r["t"]), float(r["ms"])
status = int(r["status"])
if not (math.isfinite(t) and math.isfinite(ms)
and ms >= 0 and 100 <= status <= 599):
raise ValueError("Invalid time, latency or status")
if not -60 <= t < 60:
continue
ua = r["ua"].lower()
if any(s in ua for s in ("twitterbot", "slackbot",
"discordbot", "facebookexternalhit")):
signal = "preview-UA"
elif any(s in ua for s in ("bot", "crawler", "spider")):
signal = "crawler-UA"
else:
signal = "unknown"
window = "before" if t < 0 else "after"
groups[window, signal].append((status, ms))
print("window,signal,requests,5xx,p95_ms")
for window in ("before", "after"):
for signal in ("preview-UA", "crawler-UA", "unknown"):
rows = groups[window, signal]
if not rows:
continue
p95 = sorted(ms for _, ms in rows)[math.ceil(.95*len(rows))-1]
errors = sum(status >= 500 for status, _ in rows)
print(f"{window},{signal},{len(rows)},{errors},{p95:g}")
CSV (save as synthetic; all rows invented):
t,status,ms,ua
-30,200,80,Mozilla/5.0
-5,200,100,Mozilla/5.0
0,200,90,Twitterbot/1.0
1,200,100,Slackbot-LinkExpanding
9,200,120,ExampleCrawler/1.0
10,200,130,ExampleCrawler/1.0
11,200,140,ExampleCrawler/1.0
15,502,1800,Mozilla/5.0
16,504,2100,Mozilla/5.0
Run: python triage synthetic
The script prints request counts, 5xx counts and nearest-rank p95 per window/signal. It keeps -60 ≤ t < 60; t=0 belongs to after.
These are UA string signals. They don't authenticate a crawler or prove that an unknown request is human; a browser-looking UA can be automated. Cloudflare's verified-bot process uses additional verification:
developers.cloudflare.com/bo…
For a real launch, compare the same URL before/after and inspect status/latency with cache and application logs. Keep signups or completed actions separate from fetches. Don't block every crawler because a link got busy: previews and monitoring have legitimate uses.
If you ask an AI to investigate, give it redacted status/latency summaries first. Keep credentials, cookies and personal data out. The result to ask for is the failing path and supporting evidence—not an invented identity behind a request.