You’ve done everything right. Your website is fast and looks great, your content is hitting all the right notes, and you’ve got a solid SEO strategy in place, so why isn’t your traffic growing? The problem may not be with all you’re doing, but with what’s not happening.

Before Google serves up search results, it crawls and indexes billions of pages to find the most relevant matches. If your pages aren’t making it into Google’s database, they may as well not exist at all. In this post, we’ll break down how indexing works, why it matters, and what might be stopping your content from getting found.

Not Indexed Equals Not Seen

When your goal is to get your content in front of the right audience, it first needs to be crawled, indexed, and ranked by search engines. SEO tends to focus on the last part of this equation, but ensuring search engines can discover and understand your content is equally important.

How Search Engines Work

Search engine bots (aka crawlers or spiders) are constantly scouring the internet looking for new and updated content, including text, images, video, and PDFs. In technical terms, this process is called URL Discovery. These pages may be from previously known URLs that have been updated with fresh content, or discovered by following links, submitted sitemaps, or manual URL indexing requests.

Once URLs have been crawled, Googlebot will then parse the content to understand what the page is all about. This is what we refer to as indexing. Think of it as a giant Google library, filled with information on every webpage Google has catalogued. However, not every page will make it into the Google Search Index; pages can be excluded due to crawl errors, duplicate content, blocked resources, or simply being deemed low-quality.

The final step in this process is ordering and ranking the search results users see. A lot of SEO is focused on this stage, leveraging strategies like keyword research, link building, content optimisation, and user experience to influence where a page lands in the SERPs. However, before you can worry about boosting your ranking, your content needs to be findable. If your pages are not indexed, it doesn’t matter how good your SEO strategy is that they won’t show up in search results.

Why Isn’t Your Content Getting Indexed by Google?

OK, so now that we understand how Google finds, indexes, and ranks search results, let’s dive into some common challenges that might be impacting or preventing your content from being indexed and, ultimately, discovered.

An audit of your website is the logical place to start to find out if your URLs are being indexed, and if they’re not, what the potential issue is. There are several tools you can use, but one of the easiest and most comprehensive is Google Search Console.

Start by logging into Google Search Console and scrolling down to your Indexing section, then click Pages. Here, a chart will appear highlighting your indexed pages as well as your non-indexed pages. It’s a good idea to review your indexed pages to ensure there’s nothing there you don’t want Google indexing (for example, staging environments or outdated content), but you’re probably more interested in scrolling down to see the list of URLs not currently indexed and the reasons why.

Excluded by ‘Noindex’ Tag

A ‘noindex’ tag tells search engines not to include the page in their index. Sometimes this is intentional (like with login pages, pages under development, or pages irrelevant to your SEO), but in some cases, this tag can be mistakenly left on pages that you do want indexed.

To fix this error, filter your not-indexed pages for Excluded by ‘noindex’ tag. After you have made the changes, use the URL Inspection Tool in Google Search Console to request indexing of the updated page.

Canonical Issues

Duplicate or nearly identical content can be very confusing for search engines. If you have multiple pages that contain very similar or duplicate content, Google may choose to index only one version, and it might not be the page you want. This scenario is fairly common, especially for e-commerce sellers who might, for example, have one sweater available in several different colours that change depending on the season. To ensure Google indexes your core product page, make sure to add a canonical tag to the page that you want Google to prioritise.

Crawled – Not Currently Indexed

This one can be frustrating. It means Google knows your page exists and may have even crawled it, but hasn’t added it to the index. There are a number of reasons for this type of error, and Google doesn’t specify the cause. That said, the exclusion of crawled pages from the Search Index most often relates to performance or content quality issues.

Start by reviewing your content to ensure that it’s relevant and provides genuine value to your audience. You can also improve the internal linking structure to give Googlebot more signals about the importance of the page. Adding more relevant content or user-focused information could increase the chances of indexing.

Your website architecture could also be to blame for not being indexed. Earlier, we discussed how Googlebot (aka spiders) crawls your website looking for new content; if the structure of your site makes it difficult to navigate, or no internal links are pointing to a specific page, then crawlers might assume that page has less value than other pages on your site. If you find an important page is being overlooked, make sure to create internal links from high-authority pages to the one that’s being neglected.

Blocked by Robots.txt

When you add a robots.txt file to your site’s root directory, you are telling crawlers they can’t access those specific pages. People intentionally use robots.txt to block crawlers in cases where they want to keep private or sensitive pages (like login pages) from being crawled, or they don’t want duplicate content to be indexed. However, if you are getting this error message and blocking crawlers’ access was not your objective, you’ll have to dig into your robots.txt file to find the source of the problem.

To fix the issue, open the robots.txt file through your website’s root directory. Look for any "Disallow" directives that might be blocking access to important pages. If you find any pages that should be accessible to crawlers but aren’t, remove or adjust the relevant directives in the file that are causing the problem. Once the changes are made, use Google Search Console’s URL Inspection Tool to resubmit the affected pages for indexing.

Error 404 – Not Found

This error is just as aggravating for users as it is for the bots trying to crawl your pages. It can also be one of the most damaging for your SEO. Error 404 signifies a broken link; you have likely removed, renamed, or changed a page on your website but failed to update your sitemap or implement a redirect. In either case, broken links mean Googlebot can’t crawl and index the page, and users will land on an empty page. If you find this error, it’s important to immediately locate and rectify the 404 pages to avoid losing potential organic traffic and harming user experience.

Soft 404 Error

A soft 404 occurs when a page loads but doesn’t have enough meaningful content for Google to consider it valuable. Google will still index the page, but it won’t rank well. We often see this issue when a page has thin content or leads to a 404-like experience with no real information.

If low-quality content is the issue, work to improve the depth and relevance of the content. If the page is intentionally empty or irrelevant, it’s better to return a proper 404 or 410 HTTP status code to signal to Google that the page no longer exists, rather than allowing it to remain indexed with no valuable content.

Page with Redirect

It’s not uncommon for pages with redirects to show up in your not-indexed list. This status means that some of your pages aren’t indexed because crawlers are being redirected to other pages, and Google is indexing those URLs instead. As long as your target URLs are indexed, this isn’t an issue. However, if you find an important page in your not-indexed list, you need to address it.

To fix the issue, you can manually inspect problematic redirects or use the URL Inspection Tool in Google Search Console to check the status of the redirected page. Once you've identified the redirects, ensure that they are set up correctly. It's important to use the appropriate redirect type, such as 301 for permanent redirects and 302 for temporary redirects. After addressing any issues, you can request indexing of the updated page through Google Search Console to ensure it gets priority indexing.

Don’t Let Indexing Issues Hold You Back

If it isn’t on your radar already, indexing needs to be a top priority. Remember, if your content isn’t indexed, it’s like it doesn’t exist online, so everything else you’re doing to grow traffic to those pages is wasted effort.

If you want to improve your rankings, it starts by making sure your content is properly indexed!