Salience

Most SEOs have never looked at their server logs. We’re all talking about AI visibility, LLM citations and whether AI systems are using our content. But how many of us actually know which AI bots are visiting our websites? Google Search Console, analytics and third-party crawlers are useful, but none gives you the complete picture of requests hitting your infrastructure. Server and CDN logs show you which URLs were requested, by which user-agents, when, and what response they received. And with streaming, you can see that activity within seconds. Some of the things worth looking at: AI and LLMs • Which pages GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot and other AI crawlers are requesting, and how often. • The difference between AI training crawlers, search crawlers and user-triggered fetches. They’re not all doing the same thing. • Whether AI crawlers are being blocked, rate-limited or receiving 403s, 429s and 5xx errors. • Which AI bots are genuine and which are scrapers pretending to be recognised crawlers. • Which sections of your website AI crawlers visit most, and which important pages they haven’t requested. • How AI crawler activity changes over time, particularly after publishing new content or making technical changes. • How AI crawler activity compares with actual referral traffic from ChatGPT, Perplexity and other assistants. Traditional technical #SEO • Which important pages Googlebot is crawling, how frequently it returns and how quickly it discovers new content. • Where crawl activity is being wasted on parameters, faceted navigation, redirects and low-value URLs. • Pages Googlebot keeps requesting but Google hasn’t indexed, or indexed pages it hasn’t revisited. • Intermittent 404s, 5xx errors and unexpected redirects that regular website crawls might miss. • Whether Googlebot is discovering new URLs and following the correct redirects after a migration or release. • Slow response times, rate limiting and other issues affecting search engine crawlers. • Orphan pages that bots continue requesting despite no longer being linked internally. • Broken external links that people are actually clicking, only to land on a 404. Of course, an AI crawler requesting a page doesn’t mean the content was used for training or cited in an AI answer. But at least you know what the bots are actually doing, rather than guessing. Most websites have access to valuable first-party data that rarely gets looked at. @saliencecom

Saw Quentin Yacoub's findings on the rapid growth of ExaSearchBot across Adobe customer websites, so thought I’d compare them with what we're seeing in Salience's server logs. Turns out we're seeing a 5,271% increase against its August baseline. Comparing 12–18 October against average weekly crawler activity in August: 📈 ExaSearchBot: +5,271% 📈 OAI-SearchBot: +782% 📈 PerplexityBot: +339% Worth remembering Exa started from a much smaller baseline, so this isn't a comparison of total traffic. I was curious about what might be driving the increase, so had a look at who's actually using Exa. Turns out they've got some pretty significant customers and integrations, including Firefox, Cursor, Cognition, HubSpot, Monday, OpenRouter and Google Cloud. They also recently announced 2 billion requests a month and have been massively expanding their own search index. I wonder how much of the crawler growth we're seeing is Exa building out and refreshing that index, particularly ahead of the Exa 3.0 launch. We also saw a pullback in activity over the latest matched days, so it's certainly not consistent growth every week. Still, interesting to see similar patterns across two independent datasets. As more AI products rely on third-party search providers, understanding who's actually crawling your website is becoming rather important.

We've been reviewing around 1.5 billion server requests, so I thought I'd share a few observations. There's been some interesting discussion recently about how much traffic AI assistants actually send to websites. Worth checking out the research and perspectives from @Marie_Haynes , Kevin Indig, The SEO Works, @randfish and Michael King. Different datasets, methodologies and opinions, all worth reading. Firstly, AI referral traffic remains small compared with Google organic. Our findings broadly agree with the other studies. However, server logs reveal another layer of activity that traditional analytics doesn't capture. Some interesting changes we've observed since mid-August: ✳️ ChatGPT live reads: Verified ChatGPT-User requests have fallen by approximately 80%. ✳️ OpenAI crawling: Verified OAI-SearchBot requests have more than doubled over the same period. ✳️ Claude Code: We recorded approximately 26x growth in distinct sources accessing websites between June and September, predominantly technical documentation. The change in ChatGPT's behaviour is particularly interesting. It's consistent with a shift towards crawling and indexing content in advance, rather than fetching pages during individual conversations. Of course, an AI agent fetching a page doesn't mean someone saw a recommendation, let alone made a purchase. But that doesn't make the activity irrelevant either. It highlights why referral traffic, agent activity and indexing need to be considered separately. Interested to hear whether others analysing server logs are seeing similar patterns.

A good example of why we built Salience.com and why real-time server log monitoring matters. Following the recent PHP2Shell vulnerability, we spotted suspicious scanning activity across several websites were monitoring. Around 20% of monitored sites received exploit-shaped requests, with some activity starting before a patch was even available. Most requests were blocked at the edge, but a small number of sites warranted further investigation. We've already contacted the affected clients. Its a useful reminder that server logs reveal activity you wont find in Google Analytics or Search Console, including requests blocked before they ever reach your application. Security wasn't the primary reason we built Salience, but its another practical benefit of making first-party server data accessible and actionable.

Salience is now live: connect your website for free and start measuring first-party data that most teams never see, let alone act on. Your server and CDN logs show what is actually reaching your site, what gets requested, how often crawlers return, what your infrastructure serves them and where things go wrong. Built alongside my co-founders @ndredding and @channel5 , Salience is designed to make that raw log data genuinely useful and actionable in real time. That includes seeing how search engines and AI crawlers such as ChatGPT and Claude interact with your site: which pages they visit, how frequently they return, what they request and whether they encounter errors. We originally built it off the back of problems we were solving for clients, particularly around crawling, indexing, and AI visibility across products, collections and editorial content, and large marketplace listing websites while giving teams access to a layer of insight they’d rarely had before. It’s designed for a broad range of sites, from Shopify and BigCommerce stores through to publishers, marketplaces and larger businesses running custom technology stacks. It’s also useful beyond #SEO and search, with insights for development, content and security teams, whether you’re in-house, an agency or a consultant. We’ve already had some great feedback from beta users, so thank you to everyone who has tested it so far. We’d love more of it: what’s useful, what’s missing and what could be better. There’s a working demo to explore, or you can connect a production site for free and see what your own data tells you. Please share it with anyone on your team who might find it useful. Visit: salience.com for more detail.