Technical SEO: indexing, crawling and Core Web Vitals — where to start

What is technical SEO and why start with it?

Technical SEO covers everything that lets Google (and, more and more, AI crawlers) access your pages, understand them and display them properly. It isn't reserved for huge websites: on a small business site, four or five checks are enough to catch the essentials. Great content on an inaccessible page simply doesn't exist for a search engine. That's why it's almost always my first step, before I even talk about keywords.

How to run a technical SEO check: the 4 checks, in order

Step 1: check that a page is indexable (noindex, robots.txt, canonical)

I start with the simplest question: can Google index this page? Three blockers come up all the time: a noindex tag left behind after a launch or a migration, a robots.txt file blocking an entire section, and a canonical tag pointing to a different URL. One point is often misunderstood: robots.txt prevents crawling, not indexing (Google Search Central). And for a noindex tag to be read, the page must stay crawlable, so blocking a page in robots.txt while also adding a noindex is contradictory.

Step 2: read the Search Console indexing report (Discovered / Crawled, not indexed)

Search Console's page indexing report tells you, URL by URL, why a page is or isn't indexed. Two statuses deserve your attention: "Discovered, currently not indexed" (Google knows the URL but hasn't crawled it yet) and "Crawled, currently not indexed" (Google has seen it and decided not to keep it, often because the content is thin, duplicated or not very useful). They are two different problems: the first is often about internal linking, the second about content quality.

Step 3: check the XML sitemap and internal linking

An XML sitemap helps Google discover your pages, but it doesn't replace internal links: a page listed in the sitemap with no internal link pointing to it sends a weak signal. I check that the sitemap only contains useful URLs (indexable, returning a 200 code, no duplicates) and that every important page is reachable in a few clicks from the homepage. It's also the foundation of semantic clusters: technical structure and editorial structure are built together.

Step 4: measure Core Web Vitals (LCP, INP, CLS) on real-world data

Core Web Vitals measure real-world loading experience. Google looks at three metrics: LCP (time to display the largest element, aim for under 2.5 seconds), INP (responsiveness to interactions, under 200 milliseconds) and CLS (visual stability, under 0.1). These thresholds are assessed on field data (the Core Web Vitals report in Search Console, CrUX data), not only on a lab test like Lighthouse. A site that scores well in the lab can still feel slow to real visitors on mobile.

What are the most common causes of poor Core Web Vitals?

Without chasing perfection, three causes explain most poor scores: oversized or undimensioned images (LCP and CLS), stacked third-party scripts such as chat widgets, cookie banners or ad pixels (INP), and fonts or banners that shift the page while it loads (CLS). Fixing these three often gets a site back under the thresholds, with no technical overhaul.

Should you allow GPTBot, ClaudeBot and PerplexityBot in robots.txt?

AI crawlers (GPTBot, ClaudeBot, PerplexityBot…) crawl the web independently of Googlebot, with rules described by OpenAI, Anthropic and Perplexity. A robots.txt file written years ago may block them without anyone having decided it. Whether you allow them or not, it should be a conscious choice, and worth checking in the file, because it conditions your visibility in AI answers.

Comparison table — the 5 technical checks, in order

Where to start a technical audit in one hour: a 5-point checklist

1. Inspect the five pages that matter most to the business in Search Console.
2. Open the indexing report and note the non-indexed URLs and their status.
3. Open the sitemap and check it only contains useful pages.
4. Run PageSpeed Insights on the homepage, on mobile.
5. Read the robots.txt file from top to bottom.

Keep reading: SEO audit, on-page SEO and semantic clusters

→ What an SEO audit actually looks like — the full audit walkthrough, technical part included.

→ SEO for beginners — the basics of crawling and indexing, without jargon.

→ On-page SEO and semantic clusters — structuring content once the technical foundations are in place.

→ SEO, my method and services — the overview page on the topic.