Technical SEO, A 2024 Guide

Let's clear any confusion at the beginning: Technical SEO is anything not having to do with your content (on-page SEO) or backlinks (off-page SEO).

This means that Crawling, Ranking, Indexing, Page Experience, Design, User Experience, User Interface, (among others) are all aspects of Technical SEO.

Technical SEO is the result of collaboration between different sectors of marketing: your SEOs and your Website Developers & Designers.

When these teams are working in tandem with one another, there are great results in your typical SEO KPIs, such as: website traffic, keyword attribution, & time-on-page.

Table of Contents

• What is Technical SEO?

• Crawling and Indexing for Search Engine Visibility

• How To Upload Your Sitemap to Bing Webmaster Tools

• What is a Robots.txt file?

• Canonicalization

• Indexing controls (noindex, etc)

• Crafting a Search-Friendly Site Architecture

• Advanced Technical Factors for SEO Wins

What is Technical SEO?

Technical SEO refers to the parameters of your website that define the user's experience & the search engine spider's ability crawl and index.

The easiest way to think about it is: this is the part of SEO where a forgotten checkbox is the usual culprit.

Some of the most important functions of a website determine the ability for it to be recognized by search engines. Instructions for these processes are often overlooked by CMS providers & are hard to notice for the average marketing professional to catch.

Make it & Break it: When Developers Effect Technical SEO

The website parameters that effect technical SEO can greatly impact a website's performance in search engines.

If pages are inaccessible to search engines, they won't appear in results regardless of the value of the content.

This leads to lost traffic and potential revenue.

Situations that are typical for development intiatives (like setting a page collection to 'noindex' while it is being built out), often become the exact reason why a collection of pages is no longer receiving traffic after a content migration or website redesign.

Mobile Considerations for B2B & Enterprise

Page speed and mobile-friendliness are confirmed Google ranking factors. Mobile-first indexing has been the standard for many years at this point. We will cover more of these mobile-friendliness standards later in this guide.

To keep short: B2B & Enterprise websites that don't get with the times are known to be hit with penalties in their keywords and traffic.

The trouble with being online for so many years is becoming complacent to what search engine standards used to be.

The internet is changing from underneath your website (and subsequently, your business). SEO is one of the most dynamic fields of marketing that exist.

Stop catering to desktop-only users. Their experience of your website is not considered.

Slow pages frustrate users who may leave the site, signaling poor user experience to Google.

Crawling and Indexing for Search Engine Visibility

There are different functions present within a website that are intended to instruct crawlers with information regarding which parts of your website to:

• Crawl, or not crawl

• Index, but not index

• Other nuanced situations we won't get into in this section.

The generic name for Google's crawlers is 'Googlebot'. There is one for desktop, one for mobile, and other versions that support features such as their experimental Search-integrated large language model, Bard.

Ignoring these bots could leave critical content unseen by Google or other search engines.

Worse yet, it could put on public display pages that are intended for targeted traffic, such as PPC or other landing pages.

The functions we cover in this section:

These tools at your disposal help to inform crawlers of their instructions and get your website in agreement with Google & other search engines.

What is a Sitemap.xml file?

A sitemap is an XML file that contains a list of the important pages on your website.

The purpose of a sitemap is to help search engines like Google and Bing discover new and updated content on your site.

Having a sitemap is especially useful for sites that have a large number of pages, dynamically generated content, or poor internal linking between pages.

In these cases, a sitemap acts as a roadmap for search engine crawlers to efficiently crawl your site.

A standard XML sitemap includes:

• A list of all your website's pages that you want indexed

• The date each page was last updated

• How frequently each page is updated (always, hourly, daily, weekly, monthly, yearly, never)

• The priority of each page (0.0 to 1.0)

You can break down your sitemap by page type as well.

For example: having separate sitemaps for blog posts, categories, tags, static pages, etc.

The sitemap file is typically located at yourdomain.com/sitemap.xml or yourdomain.com/sitemap_index.xml.

How To Upload Your Sitemap to Google Search Console

Once you've created a sitemap, you need to submit it to Google Search Console. This tells Google where to find your sitemap file. To submit:

1. Go to Google Search Console

2. Click on "Indexing" in the left menu

3. Select "Sitemaps"

4. Click "Add/Test Sitemap"

5. Enter the path to your sitemap and hit submit

After processing your sitemap, Google will provide info on the number of pages submitted, indexed, and any errors. Check this report periodically to ensure your new content is being indexed properly.

There are also sitemap generator tools that can create and update XML sitemaps automatically as your site changes. Using a generator is recommended to save time compared to manually maintaining a sitemap.

How To Upload Your Sitemap to Bing Webmaster Tools

Bing Webmaster Tools allows you to submit your XML sitemap so Bing can more efficiently crawl and index the pages on your site. Here are the steps to upload your sitemap to Bing:

1. Go to Bing Webmaster Tools and either sign in or register for an account.

2. Click "Add site" and verify ownership of your website.

3. Once your site is added, go to the Sitemaps section under Site Configuration.

4. Click "Add/Edit Sitemap" and enter the full URL path to your sitemap (e.g. https://www.example.com/sitemap.xml).

5. Click Submit to add your sitemap.

6. Bing will crawl your sitemap and provide details on the number of pages submitted, indexed, and any errors.

7. You can add multiple sitemaps here, like separate ones for blog posts, products, categories etc.

8. Check back periodically to re-submit updated sitemaps and confirm new pages are being indexed.

9. Use the Sitemap Status tool to see indexing stats per URL and identify any crawling issues.

Submitting your XML sitemap is crucial for Bing visibility. It ensures your important pages are discovered, crawled, and indexed efficiently. Keep your sitemaps updated as your site grows.

The Other Sitemap: A Page On Your Website

Sometimes, it is the choice of developers to include a version of the sitemap that is designed to the same standard as every other page.

This is meant to be used for visitors of the website that are looking for a quick index (we used to refer to these as 'Glossaries' back when Phone Books were still a thing), in order to find the category and page being sought after.

While these pages do serve up internal linking opportunities, they are not conducive for search engine crawlers, just the users of the website themselves.

What is a Robots.txt file?

The robots.txt file gives instructions to search engine crawlers about which pages or folders on your site they should not access. This file needs to be handled carefully, as any mistakes can inadvertently block entire sections of your site from being indexed.

The robots.txt file goes in the root directory of your website and can contain the following directives:

• User-agent - Specifies which crawlers should follow the rules, often "User-agent: *" to target all crawlers

• Disallow - Tells crawlers not to access the specified paths

• Allow - Undoes a Disallow rule for a specific path

• Sitemap - Indicates the location of your XML sitemap

For example:

User-agent: * 
Disallow: /thank-you/
Disallow: /members/
Allow: /members/login/

Sitemap: https://www.example.com/sitemap_index.xml

This would block all crawlers from the /thank-you/ folder and the /members/ folder except for the /members/login/ page.

When determining what to block, avoid removing key pages like product, category, tag, author, etc. pages. Generally only block:

• Thank you / order received pages

• Account or checkout pages

• Pages with duplicate content

• Pages with thin content

• Pages still under development

To test your robots.txt, you can use tools like the Google Robots Testing Tool which shows you what Googlebot can and cannot crawl on your site based on your directives.

It's recommended to use the robots.txt sparingly and with caution. The preferred approach is to use meta robots noindex tags on specific pages you want to remove rather than blocking entire site sections.

Canonicalization

Canonical tags are a way to solve duplicate content issues by specifying the "canonical" or preferred URL that should represent a page.

Duplicate content occurs when the same or very similar content exists on multiple URLs on your site. This confuses search engines about which page to index and rank.

Some common causes of duplicate content include:

• HTTP vs HTTPS pages

• www vs non-www versions

• Parameter URLs like ?sort=price vs ?sort=popular

• Pagination pages

• Category, tag, author archives

To deal with duplicate content, you need to choose the one canonical URL and add a rel="canonical" link tag to the head section of the other pages pointing back to original.

For example:

<link rel="canonical" href="https://www.example.com/product-page">

This would be added to duplicate product pages such as:

• http://example.com/product-page

• https://example.com/product-page

• https://www.example.com/product-page?sort=price

The canonical tag tells search engines to index and rank the URL specified in the href attribute rather than the current page's URL.

Things to note about canonical tags:

• The canonical URL should point to the page you want to prioritize for SEO

• Use self-referencing canonicals on all pages pointing to themselves

• Only use noindex on pages you want removed entirely, not canonical tags

• Check for proper implementation with Google Search Console

Adding canonical tags is crucial for preventing duplicate content from diluting rankings. This technical SEO tactic should be implemented on any site with multiple similar URLs.

Indexing controls (noindex, etc)

Noindex and other indexing control meta tags allow you to selectively exclude pages from being indexed and shown in search engine results.

The most common indexing control meta tag is noindex, which looks like this:

<meta name="robots" content="noindex">

When search engine crawlers come across this tag in the head section of a page, they will exclude that page from their index. This prevents it from ranking in results.

Some common use cases for noindex include:

• Thank you pages after form submissions

• Login and account pages

• Duplicate content pages

• Pages with thin, low-value content

• Pages under development or maintenance

Other indexing control meta tag options:

• nofollow - Prevents crawling links on a page

• noarchive - Prevents caching in search engine archives

• nosnippet - Blocks description snippet in search results

Noindexing pages is preferable to using robots.txt to block pages, as it only removes specific pages rather than full sections of a site.

To verify your noindex tags are working properly, do a "site:" search for the URL in Google, or inspect the page in Google Search Console for the "Indexed, not submitted in sitemap" status.

Things to note:

• Add noindex tags to the live page, not a 410/404 redirect

• Use self-referencing canonicals if you want a page indexed

• Google can take time to drop pages from the index after finding a noindex tag

Noindex and other indexing control tags are useful for cleaning up specific issues pages that you don't want cluttering up search results and distracting from more important pages. Use judiciously.

Crafting a Search-Friendly Site Architecture

URL structure

The structure of URLs on a website can have an impact on both user experience as well as SEO. Well-structured URLs make it easier for visitors to understand where they are on your site. They also help search engines properly crawl and index your pages.

Here are some best practices for creating SEO-friendly URL structures:

• Use descriptive keywords - Include relevant keywords in a natural, readable order. For example, "blue-suede-shoes" rather than just numbers and special characters.

• Keep URLs short and simple - Concise URLs with 60 characters or less are preferred. Avoid overly long, complex parameter-heavy URLs.

• Create logical hierarchy - Organize URLs into a sensible folder structure. For example, "domain.com/category/subcategory/page"

• Consistency - Use consistent URL patterns and naming conventions across your site. Don't mix things like underscores vs hyphens or singular vs plural words.

• Lowercase - Use only lowercase letters in URLs. Avoid uppercase, as some servers treat uppercase and lowercase URLs differently.

• Static not dynamic - Choose static URLs over dynamic URLs whenever possible, as static pages tend to rank better.

• 301 redirect old URLs - When restructuring URLs or moving content, use 301 redirects to pass link equity to new URLs.

• Avoid stop words - Don't include stop words like "a", "the", "and" in URLs.

Properly structured URLs enhance user experience, improve click-through rates, and support technical SEO efforts. Auditing your site's URLs periodically helps spot any issues. Tools like Screaming Frog can crawl your site and identify non-SEO friendly URLs.

Site speed

Site speed refers to how fast your web pages load. It's a very important ranking factor and website performance metric. Faster sites tend to rank higher and provide better user experience than slow loading sites.

There are a few key reasons why site speed matters for SEO:

• Direct ranking factor - Google has explicitly stated faster sites will be favored in rankings. Improving speed can directly boost rankings.

• User engagement - Faster pages encourage more engagement like lower bounce rates and higher pages per session. Better engagement signals lead to higher rankings.

• Mobile friendliness - Slow pages frustrate mobile users who will quickly leave. Google factors mobile friendliness into rankings, so improving speed improves mobile experience.

There are many factors that influence page load times including:

• Server location and performance

• Code optimization and weight - HTML, CSS, JavaScript

• Image compression

• Caching and CDNs to distribute content

• Limited redirects

• Minimized plugins/trackers/analytics

Tools to measure current site speed:

• PageSpeed Insights - Shows desktop and mobile speed scores.

• WebPageTest - Provides detailed speed optimization recommendations.

• Pingdom - Easy to use from multiple geographic locations.

Some ways to improve site speed:

• Optimize images - Compress and resize, utilize newer formats like WebP.

• Minify CSS/JS - Remove unnecessary code characters to reduce file size.

• Use a CDN - Distribute static assets globally.

• Upgrade hosting - More resources (CPU, memory) and better technology.

• Cache dynamic content - Store cached versions to serve faster.

• Lazy load media - Images/videos load only when visible on the page.

Site speed should be regularly monitored and optimized. Aim for a PageSpeed score of at least 90 on mobile and desktop. Fast sites equal happy users, lower bounce rates, and higher conversions, revenue, and rankings.

Mobile-friendliness

With more and more searches happening on mobile devices, having a mobile-friendly website is essential for SEO in today's landscape.

Google has stated that mobile-friendliness is a ranking factor for mobile searches. So optimizing your site for mobile can directly improve visibility and traffic.

Some key elements of a mobile-friendly site include:

• Responsive design - Content adapts and responds to fit any screen size. Text and images resize as needed.

• Tap targets - Buttons and links are spaced appropriately so they are easy to tap on a touchscreen.

• Readability - Font sizes and spacing make text readable without zooming. Short paragraphs are used.

• Page speed - Site loads fast on mobile networks and devices. Caching, compression, and other speed optimizations.

• Minimal horizontal scrolling - Page content fits screen width without too much sideways scrolling.

• Avoid interstitials - Interstitial popups or full page overlays that block content are prohibited.

• Structured data - Markup helps search engines understand page content. Can enable rich result formatting.

To test your site, use Google's Mobile-Friendly Test. It will analyze each page and identify any issues that should be fixed.

For WordPress sites, plugins like WPTouch can help convert sites to be mobile-ready. Responsive design, proper sizing of elements, and fast loading times are key.

Prioritizing the mobile experience pays dividends in higher mobile rankings, lower bounce rates, and increased conversions. Users expect sites to work flawlessly on mobile.

Duplicate content elimination

Duplicate content refers to identical or very similar content that exists on multiple pages on a website. This causes issues with search engine indexing, as engines can't determine which page to show in results.

Some common causes of duplicate content include:

• Same content published across multiple URLs

• Similar category, tag, author archive pages

• Pagination creating multiple similar URLs

• Print and mobile versions of pages

• Scraped or copied content

Consequences of duplicate content include:

• Dilution of page authority and link equity between duplicates

• Landing pages ranking instead of main pages

• Lower visibility as duplicates compete against each other

There are two main ways to fix duplicate content:

1. Canonical tags

Canonical tags specify the one preferred or canonical URL that search engines should index and rank:

html

<link rel="canonical" href="https://www.example.com/page">

The canonical tag goes on all duplicate pages pointing back to original.

2. Noindex tags

Noindex prevents duplicate pages from being indexed. This removes them from competing with the original:

html

<meta name="robots" content="noindex">

Noindex is best for removing pages entirely vs just consolidating authority.

Check for duplicate title and meta descriptions as well. Use Google Search Console to identify duplication issues. Eliminate and consolidate around a single URL version whenever possible.

Advanced Technical Factors for SEO Wins

Structured data enhancement

Adding structured data markup to your website helps search engines better understand your content. This can result in your pages appearing in rich search results like featured snippets, knowledge panels, or enhanced product listings.

Relevant schema markup should be implemented based on your site and content types. Ecommerce sites can markup product pages with Product schema, for example. Article schema is useful for blog posts and news articles. Event schema can highlight key event details like dates, location, and registration.

Search engines recommend JSON-LD format for structured data. There are many types of schema available from Schema.org to choose from. Proper implementation requires valid markup according to the schema specifications. Test your structured data using Google's Structured Data Testing Tool.

Focus on high-value pages first when adding markup. Product category pages, individual product pages, homepage, contact page, and blog posts offer good opportunities for rich results. Just be sure to use markup judiciously - don't overload pages with irrelevant schema. Monitor search appearance to see which markup is triggering rich results.

Some common schema types to consider:

• Product schema for ecommerce product pages

• Article schema for blog posts and news articles

• FAQ schema for FAQ and Q&A pages

• Event schema for events pages

• Breadcrumb schema for site navigation

• Review schema for product/service reviews

• Site navigation schema for menus and footers

Prioritize pages that answer common user questions and searches. Adding the right schema can help searchers quickly find the information they need in rich results.

Core web vitals optimization

Google's Core Web Vitals measure key aspects of user experience and page speed. Optimizing for these metrics can improve your site's search ranking and performance.

The three Core Web Vitals are:

• Largest Contentful Paint (LCP) - measures loading speed and marks when main content appears. Aim for LCP under 2.5 seconds.

• First Input Delay (FID) - measures responsiveness and quantifies delay when users click or tap. Target FID under 100 milliseconds.

• Cumulative Layout Shift (CLS) - measures visual stability and sums unexpected layout shifts. Strive for 0.1 or lower CLS.

There are various ways to optimize for these metrics:

• Minify HTML, CSS, JavaScript and images to improve page load speed. This directly improves LCP.

• Lazy load non-critical resources like images below the fold. Also boosts LCP.

• Reduce render-blocking resources by deferring non-critical CSS/JS. Helps LCP.

• Preconnect to third-party origins like APIs and CDNs. Also improves LCP.

• Code efficiently and size JavaScript bundles appropriately to prevent FID issues.

• Avoid cumbersome layout shifts, like ads or images loading in later. Fixes CLS problems.

Tools like PageSpeed Insights, Search Console, and Lighthouse provide Core Vitals reports to help diagnose optimization opportunities. Aim to have most pages scoring in the green for all three Core Vitals.

Hreflang implementation

Hreflang tags specify the target language and regional URLs for a page. Implementing hreflang helps search engines serve users the most relevant localized content for their location.

The basic format for a hreflang tag in HTML is:

html

<link rel="alternate" href="URL" hreflang="language_code">

The key components are:

• rel="alternate" indicates it is an alternate URL for the same content

• href is the target localized URL

• hreflang is the language/region code like "en-US"

Some tips for proper implementation:

• Add a hreflang tag in the HTML head for each translated version of a URL

• Use the correct ISO language/region codes

• Point to the equivalent localized URL in each hreflang href

• Make sure hreflang URLs actually exist, otherwise they are useless

• Include a self-referential tag pointing back to the current URL

Test your hreflang tags using Google's Rich Results Test to validate proper syntax and configuration. Monitor search appearance and analytics to confirm you are serving visitors appropriate localized content.

Boosting security with HTTPS

HTTPS encrypts data transferred between a website and visitors' browsers. It is increasingly becoming an important ranking factor as Google pushes for an all-HTTPS web.

To implement HTTPS:

• Obtain an SSL certificate from a trusted Certificate Authority like Let's Encrypt.

• Install the SSL certificate on your web server. This enables encryption and activates HTTPS.

• Redirect all HTTP requests to HTTPS using server directives or a plugin. This ensures all traffic is secure.

• Update sitemaps and canonical tags to use HTTPS URLs.

• Replace all hardcoded HTTP links and resources with protocol-relative or HTTPS versions.

With HTTPS enabled, verify that:

• Browsers show a padlock icon and https:// in the URL bar. No security warnings should display.

• Pages load without any mixed content errors for insecure resources.

• Site speed is not negatively impacted by SSL overhead.

Migrating to HTTPS demonstrates to Google and visitors that security is a priority. It also unlocks SEO benefits like higher rankings, more clicks, and increased user trust.