
🕵️♂️ Founder of @VitalSentinel 👨💻 Developer-led Enterprise Technical SEO Consultant ➡️ https://nitter.gq/t.co/VfYeyH1MwA - https://nitter.gq/t.co/2SUOjj7jZL
Joined November 2023
- Tweets1.9K
- Following357
- Followers219
- Likes1.8K
Martin Stepanek retweeted
🟣 Underrated #TechSEO Tip
Your robots.txt response time matters more than you think. If it takes too long to load or returns a 5xx error, Google may temporarily stop crawling your entire site.
Google caches robots.txt for up to 24 hours. Monitor its uptime the same way you monitor your homepage. A slow or broken robots.txt blocks all crawling.
Server logs are one of the most underused free resources in technical SEO. They show exactly how your site responds to crawlers. No guessing, just data.
Your uptime monitor can stay green while Googlebot gets 502s on part of your site.
A monitor checks one URL, usually the home page. Googlebot requests thousands. When one app pool dies or one route fails, the home page keeps answering 200.
Google slows crawling on 5xx errors, in proportion to how many URLs return them. A 502 on robots.txt stops crawling of the whole site for the first 12 hours. Errors that persist eventually drop pages from the index.
Your access log shows the gap. Count verified Googlebot's 502, 503, 504 and 499 lines by hour, and the outage window stands out. Server Log Monitoring keeps that split by bot and hour for you.
➡️ vitalsentinel.com/blog/502-b…
1
9
🟣 Underrated #TechSEO Tip
Your robots.txt response time matters more than you think. If it takes too long to load or returns a 5xx error, Google may temporarily stop crawling your entire site.
Google caches robots.txt for up to 24 hours. Monitor its uptime the same way you monitor your homepage. A slow or broken robots.txt blocks all crawling.
Love it 😍😂😂
Thank you Profound for showing us the error of our ways 🙏
queryburst.com/blog/aeo-vs-s…
15
🟣 Underrated #TechSEO Tip
TTFB is the foundation of every performance metric.
If your server takes 2 seconds to respond, nothing else matters. LCP, FCP, everything starts late.
Check TTFB in your CrUX data. Above 800ms? Fix your server before you touch anything else.
1
1
20
Frequent problem. Easy to prevent.
Robots.txt monitoring is one of the most useful features I've built.
A common way robots.txt breaks: the staging copy with Disallow: / ships with a release. It can stay live for days before anyone notices.
Robots.txt Monitoring checks the file every hour, compares it with the last saved copy and records every line added or removed. It flags errors against RFC 9309 and Google's rules, and alerts when the file stops loading.
Before a deploy you can paste the draft. "What this changes" lists the pages it would block and the Google clicks they had.
AI bots are grouped into training, AI search and user-requested fetches, with ready-made rules if you only want to block training.
vitalsentinel.com/features/r…
11
🟣 Underrated #TechSEO Tip
Preconnect to third-party origins you know you'll need.
DNS lookup, TCP handshake, TLS negotiation – these add 100–300ms per origin on mobile.
Add `<link rel="preconnect" href="example.com">` and the browser starts the connection early.
Do this for your 2–4 most critical third-party domains.
But keep in mind, over-preconnecting hurts.
16
🟣 Underrated #TechSEO Tip
Your JavaScript bundle size directly impacts INP.
Every kilobyte the browser parses and executes blocks the main thread.
Users click a button and nothing happens. That's what a blocked main thread feels like.
Code-split your JS. Load only what the current page needs. Let the rest load on demand.
25
I recently found a severe bug that would have been almost impossible to catch manually 😰
An enterprise site with hundreds of thousands of URLs. Two replicas behind a load balancer were serving two completely different versions of the sitemap.
Depending on which replica answered, you’d get a different set of URLs. Check it in the browser, everything looks fine. Meanwhile Googlebot keeps getting the other version. At that scale, you can’t spot this by eye.
Sitemap monitoring in @VitalSentinel flagged the inconsistency within minutes. A few days later, the fix was live.
If your site runs on multiple replicas, don’t assume they all serve the same thing.
A deploy can change how a sitemap is built and drop a thousand URLs overnight, including ones that still bring in Google clicks. A normal deploy check won't flag it.
Sitemap Monitoring finds every sitemap from robots.txt, Google Search Console, common locations and index files, reads them all once a day and reports what was added, removed or came back.
Removed URLs with Google clicks go to the top, with a suggested fix: add it back or redirect it. A big drop is downloaded a second time before it's reported, so a one-off timeout doesn't page you at 3 a.m.
No install and no limits on sitemaps or URLs, on every plan.
vitalsentinel.com/features/s…
25
Martin Stepanek retweeted
JavaScript often rewrites titles, canonicals and robots tags after the raw HTML has loaded. A tag manager, a consent script or a framework that hydrates late is enough for the rendered page to say something different from what your server sent.
Site Health Monitoring compares the two. It works through your pages one at a time, around the clock, and flags tags rewritten by JavaScript, text and internal links that only exist after JS runs, blank renders, HTML over the 2 MB Googlebot reads, and search or AI bots that can't reach the page.
Every issue shows the Google clicks at stake, so the fix list starts with the pages that cost you the most.
vitalsentinel.com/features/s…
🟣 #TechSEO Myth Debunked
"If I add structured data, I'll get rich results."
Wrong. Schema markup is necessary but not sufficient. Google only shows rich results when it trusts your site, validates the data, and decides a rich result fits the query.
Structured data makes you eligible. It doesn't guarantee anything.
You still need quality and trust.
37
🟣 Underrated #TechSEO Tip
The biggest opportunity in AI visibility right now is dead simple.
Just be crawlable.
Most sites block AI crawlers in robots.txt, serve JS that bots can't render, or have zero structured data.
Fix those three things and you're ahead of almost every competitor fighting for AI citations.
1
25
🟣 #TechSEO Myth Debunked
"Google respects the canonical tag 100% of the time."
Google treats the canonical tag as a hint, not a directive. If it points to a noindexed page, a 404, or something completely different, Google ignores it.
Canonicals only work when they make sense. Check yours in URL Inspection to confirm Google is actually following them.
13
🟣 Underrated #TechSEO Tip
I see developers slapping `will-change` or `transform: translateZ(0)` on everything for "performance."
It does the opposite. It eats memory and causes rendering problems on low-end devices.
Only use GPU acceleration hints on animations and transitions. Not on every element.
28
I can feel it's beta... Except DNS verification everything else failed with error 😅
Now in beta: Businesses in ChatGPT.
Claim your business, manage your information, and get insights on how people engage with you.
Eligible businesses can also build a Business Agent to answer customers’ questions and help them explore what you offer.
openai.com/index/introducing…
18
🟣 Underrated #TechSEO Tip
Without `srcset`, mobile users download the same 2000px desktop image scaled down to 400px.
That's 5x more data than needed on every image.
Multiply that across your page and you see why mobile LCP tanks.
Use responsive images with `srcset` so the browser picks the right size for each device.
23
🟣 Underrated #TechSEO Tip
Redirect chains burn crawl budget and slow down every visit.
Each hop adds latency (and risks hitting Google's redirect-following limit).
I audit these quarterly. I always find chains that built up over time without anyone noticing.
Flatten them.
Every old URL should point straight to the final destination. No exceptions (ideally 😅).
1
1
25
Baidu gave me 10 URLs a day.
Not 10k. Ten. No sitemap submission either.
Getting more took months of paperwork: a Chinese phone number, real-name verification, an ICP filing.
On Google, indexing is something you get. On Baidu, you apply for it.
TechSEO Vitals #49 is out ⤵️
1
53
🟣 Underrated #TechSEO Tip
TTFB on an idle server tells you very little. What matters is response time under load.
When your server slows down or throws errors, Google lowers its crawl rate. Your users wait too.
Run a load test. Stop benchmarking empty servers.
1
1
20









