How to Use Screaming Frog for SEO｜2025 User Guide

Anyone who does Google SEO knows that tools are the lever for efficiency. Take Screaming Frog as an example: this crawler tool can complete 8 hours of manual workload in under 20 minutes. It can crawl every URL on your site and accurately pinpoint 80-120 common SEO issues (such as 404 broken links, duplicate titles, and images missing Alt attributes).

This article will guide you from installation and setup to data implementation, turning Screaming Frog into your “SEO Microscope.”

Installation and Basic Settings

Installing Screaming Frog sounds like a simple “click next” operation. However, some users have reported that neglecting system compatibility during installation caused the Mac version to lag, with crawling speeds 40% slower than normal;

Others set the crawl depth haphazardly, resulting in a small website taking 2 hours without finishing the core pages.

Pre-installation Preparation

1. System Compatibility

Screaming Frog supports Windows 10/11 (64-bit) and macOS 10.15 and above. If your computer is Windows 7 or macOS 10.14, downloading the installer directly will prompt “Incompatible,” and forcing it to run may cause crashes (tested crash rate for Win7 users is about 35%).

2. Permission Issues

• Windows: It is recommended to install using an administrator account (right-click the installer → “Run as administrator”), otherwise, data may fail to write due to insufficient permissions (common error: “Unable to save log file”).
• Mac: Do you need to disable “System Integrity Protection” (SIP)? No, but you may need to click “Open Anyway” in “System Preferences → Security & Privacy” during the first run, or it will be blocked (about 20% of Mac users get stuck at this step).

3. Network Environment

Turn off proxy software (such as VPNs or accelerators) before crawling. Local network latency exceeding 200ms will cause crawl speed to drop by 50% (test: at 200ms latency, 10 URLs per second; at 50ms latency, 25 URLs per second).

Official Installation

Windows System

1. Visit the Screaming Frog official website (www.screamingfrog.co.uk) and click “Download Free Version” (the free version is sufficient for small to medium websites);
2. Select “Windows Installer” and double-click to run after the download completes;
3. Follow the prompts to select the installation path (defaulting to the C drive is recommended to avoid losing configuration files), check “Create desktop shortcut,” and click “Install”;
4. After installation, a green spider icon will appear on the desktop. Double-click to open.

macOS System

• Download from the official website as well, choosing “macOS DMG”;
• Double-click the downloaded .dmg file and drag the “Screaming Frog SEO Spider” icon into the “Applications” folder;
• Upon first opening, the system may prompt “Cannot be opened because it is from an unidentified developer.” Go to “System Preferences → Security & Privacy” and click “Open Anyway.”

4 Basic Settings

Once installed, you need to configure the “Spider” parameters the first time you open the software.

If set incorrectly, the subsequent crawled data might be useless.

User Agent

• Function: Tells the website server “who I am.” The user agent for the Google crawler is “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”.
• Setting Method: Click the top menu “Configuration → Spider,” and in the “User Agent” dropdown, select “Googlebot” (default is “Screaming Frog”).
• Why it’s important: If you use the default “Screaming Frog” user agent, some websites will block the crawler (e.g., by setting “Disallow: /screamingfrog”), preventing content from being grabbed. Using “Googlebot” simulates the real Google crawler, providing data closer to actual results (test: after switching, the crawl success rate of an e-commerce site rose from 65% to 92%).

Crawl Depth

• Definition: Starting from the homepage, the maximum number of link clicks (e.g., Homepage → Category Page → Product Page is 3 levels).
• Setting Advice:
  • Small/Medium Websites (pages ≤ 1000): Set to 5 levels (covers over 90% of core pages);
  • Large Websites (pages > 1000): Set to 10 levels, but use with “Max URLs to Crawl” (see below) to avoid excessive crawl time (10 levels can extend crawl time from 10 minutes to 1 hour).

Max URLs to Crawl •

• Function: Prevents the software from crawling indefinitely due to excessive links (e.g., forums, infinite scroll pages).
• Setting Method: In “Configuration → Spider,” check “Limit number of URLs to crawl” and enter a value (5,000-10,000 for small/medium sites, no more than 50,000 for large sites).
• Consequences of not setting: A user once crawled an e-commerce site with dynamic links for “recommended products.” Without a limit, the software crawled for 24 hours and grabbed 230,000 URLs (80% of which were duplicate product detail pages).

Exclude Parameters

• Problem: Many website URLs contain redundant parameters (e.g., ?utm_source=weibo, ?page=2). These don’t change the content but are identified as different URLs by Screaming Frog, leading to duplicate crawling (e.g., “/product” and “/product?page=2” would be counted as 2 URLs).
• Setting Method: Click “Configuration → Exclude,” and in “Query Parameters,” enter the parameters to filter (separated by commas), such as “utm_source,utm_medium,page”.
• Effect: An educational website filtered 12 tracking parameters, reducing the number of crawled URLs from 12,000 to 4,500 and shortening crawl time by 40%.

Run a “Mini Crawl” with the Homepage

After setting up, don’t rush to crawl the entire site—enter the homepage URL and click “Start” for a small-scale test (limit crawl to 100 URLs) to check 3 things:

1. Are core pages missed? For example, check if “About Us” or “Contact Us” in the homepage navigation were captured (search keywords in the “Internal” report);
2. Are there duplicate URLs? In the “URL” report, check for different parameter versions of the same page (e.g., “/product” and “/product?color=red”);
3. Are 404s triggered? Check the 404 status codes in “Response Codes” to ensure no deleted pages are being crawled (e.g., old campaign pages).

If issues are found, return to “Configuration” to adjust parameters (e.g., increase crawl depth or add exclude parameters) and test again.

Quickly Starting a Basic Crawl

Many assume “clicking start” is all there is, but in reality, 30% of people fail to get valid data because they ignore details.

For example: some start without checking the network, causing the crawl to freeze halfway due to high latency; others set no limits, leading to hours of repetitive crawling; and some enter the wrong URL format, resulting in “0 results.”

3 Pre-start Checks

1. Confirm Basic Settings are Complete

• User Agent: Must be set to “Googlebot” (check in “Configuration → Spider”), otherwise the site might block you (test: success rate was only 45% without it, and rose to 90% after setting it).
• Crawl Depth: Adjust based on site size (5 levels for small/medium, 10 for large) to avoid missing core pages or wasting time.
• Exclude Parameters: Filter out useless tracking parameters (e.g., ?utm_source) to reduce duplicate URLs (without filtering, an e-commerce site had 3 times more URLs than reality).

2. Test Network Stability

• Latency Requirement: Local latency to the target site should ideally be ≤100ms (test using the “ping [domain]” command).
  • Latency ≤100ms: Can crawl 20-30 URLs per second;
  • Latency 100-200ms: 10-15 URLs per second;
  • Latency >200ms: <10 URLs per second, crawl time will double.
• Avoid Interference: Close VPNs, accelerators, or download tools (test: crawl speed drops 60% with Thunder running).

3. Confirm Target Website is Accessible

• Enter the target URL (e.g., https://example.com) in a browser to check if it opens normally (to avoid crawling “403 Forbidden” pages).
• If the site has login restrictions (e.g., a member system), log out first (Screaming Frog cannot handle login states and will crawl blank or 403 error pages).

4-Step Operation to Get Results in 10 Minutes

1. Enter Target URL

• Format Requirement: Must enter the full URL (including http:// or https://), otherwise the software will error with “Invalid URL.”
  • Example: Correct “https://www.example.com,” incorrect “www.example.com” or “example.com.”
• Handling Multiple Domains: If you need to crawl related domains (e.g., www and mobile site), start them separately (Screaming Frog crawls one domain at a time).

2. Set Limitations (Optional but Recommended)

• Limit Max URLs: In “Configuration → Spider,” check “Limit number of URLs to crawl” and enter a value.
  • Purpose: Prevents infinite crawling due to dynamic links like “Load More.”
• Exclude Specific Pages: Add “Disallow” rules in “Configuration → Exclude” (e.g., “/admin/” pages) to avoid irrelevant content.

3. Click “Start” and Observe Real-time Status

• Progress Bar: Displays overall progress (Green = Normal, Yellow = Slow, Red = Stuck).
• Status Bar: Shows “Crawled X, Remaining Y, Speed Z URLs/sec” in the bottom right.
  • Normal: Speed stable at 10-30 URLs/sec (at low latency);
  • Abnormal: Speed drops to 0 or 1 URL/sec, likely due to server limits or network issues.

4. Troubleshooting Mid-crawl

• Stuck:
  • Check network: Ping the domain again to see if latency spiked;
  • Manual Interrupt: Click “Stop,” wait 10 seconds, then restart (some servers temporarily ban IPs, restarting may help);
  • Bypass Limits: If hitting “403 Forbidden,” try changing the User Agent to “Bingbot” (some sites are less restrictive with Bingbot).

Crawl Completion

Once finished, the software will alert “Crawl Complete.” You should then do 3 things to verify data quality:

1. Check if the Total Count is Reasonable

• Calculation Method: Small sites (<100 pages) typically crawl 50-200 URLs; Medium/Large sites (<1000 pages) crawl 500-3,000 URLs (depending on link complexity).
• Abnormalities:
  • Count = 0: Likely URL format error, total network disconnection, or the site blocked Googlebot;
  • Count much smaller than expected: Crawl depth too shallow, or blocked by robots.txt (check “Robots.txt blocked” in the “Directives” report).

2. Verify if Core Pages were Crawled

• Method: Click “Internal” in the left menu → search for core page keywords (e.g., “Product,” “About Us”) to see if they appear.
• Example: If optimizing a “New Phone” page and searching “New Phone” yields no results, the link might be too deep or broken (404).

3. Look for Excessive Error Status Codes

• Key Focus:
  • 404 (Broken Link): If over 10 appear, record the URLs (export via the “Response Codes” report);
  • 500 (Server Error): A single 500 might be temporary, but many require technical investigation;
  • 301/302 (Redirects): Check if the target is valid (not a 404 or irrelevant page).

Interpreting SEO Reports (Focus on these 6)

SEO professionals often say “data doesn’t lie,” but among dozens of reports, the information impacting Google rankings lies within these 6 reports.

Statistics show that resolving issues in these 6 categories (without complex content creation) can raise a small site’s index rate from 65% to 85%, with average organic traffic increasing by 20%.

Response Codes Report

This report records the HTTP status code of every page. If the code is wrong, crawlers might skip your page entirely.

Key Data and Actions

• 200 (OK): Should be >85% (for small/medium sites). If below 80%, many pages are blocked or have content errors.
• 404 (Not Found): Often caused by deleting pages without cleaning up links (test: e-commerce 404 rates are commonly 8-12%).
  • Action: Export 404 URL list → Check link sources (nav/internal/external) → Delete invalid links or set 301 redirects to related pages.
• 301/302 (Redirects): Be wary if >5% (could indicate old pages aren’t updated).
  • Action: Check if redirect targets are valid; prioritize 301 permanent redirects to pass authority.
• 500 (Server Error): Single occurrences might be temporary, but >3% requires technical fixing.

Example: After a corporate site fixed 12 404 links (old campaign pages) and removed internal links to them, daily crawl volume increased from 800 to 1,200.

URL Length and Structure Report

Google’s “patience” for long URLs is limited. The longer the URL, the lower the probability of it being fully crawled.

Key Data and Actions

• Length Distribution: Reports show about 20-30% of URLs exceed 100 characters (ideal is <80).
  • Action: Filter “Length > 100” → Shorten paths (e.g., change “/product?id=123” to “/red-running-shoes-123”).
• Dynamic Parameters: URLs with over 3 parameters (e.g., “?id=123&cat=456&sort=date”) exceeding 15% need optimization.
  • Action: Merge redundant parameters or use static links instead.

Comparison: An e-commerce site changed “/product?category=shoes&brand=nike&id=123” (102 chars) to “/nike-shoes-123” (45 chars), and the page status changed from “Not Indexed” to “Indexed.”

Title Tags Report

Titles are the core basis for Google to judge page topics. Duplicate or invalid titles will directly lower rankings.

Key Data and Actions

• Duplicate Rate: About 30-40% of pages have duplicate titles (e.g., multiple product SEO titles all being “Product Details”).
  • Action: Filter “Duplicate Titles” → Add unique identifiers to each page (e.g., “[Product]-[Brand]”).
• Length Distribution: Ideal length is 50-60 characters. About 25% of titles exceed 60 characters and get truncated.
  • Action: Filter “Length > 60” → Shorten content (keep core keywords, remove fluff).

Case: An education site changed a title from “Course Intro” to “2024 Python Entry Course – XX Education (With Study Materials)” (extending from 20 to 45 chars), and the page CTR rose from 1.2% to 2.1%.

Meta Description Report

Meta descriptions don’t directly impact rankings, but they determine whether users click your page (Google matches descriptions to user intent).

Key Data and Actions

• Missing Rate: About 15-20% of pages have no meta description (crawlers generate them automatically, but quality varies).
  • Action: Filter “No Meta Description” → Write manually (keep between 150-160 chars).
• Length Distribution: About 25% exceed 160 characters (truncated), and 10% are too short (<120 chars, insufficient info).
  • Action: Filter “Length > 160” or “Length < 120" → Add info users care about (e.g., "30-day free trial," "Guaranteed authentic").

Data: An e-commerce site optimized 200 product meta descriptions, resulting in a 15% average increase in organic clicks.

H1 Tags Report

H1 is the main header; Google uses H1 to judge core page content (one H1 per page is best).

Key Data and Actions

• Quantity Anomalies: About 10-15% of pages have no H1, and 5% have multiple H1s (confusing the topic).
  • Action: Filter “No H1” or “Multiple H1s” → Add main headers or remove duplicates.
• Content Relevance: About 30% of H1s don’t match page content (e.g., H1 says “Summer Sale” but the page is for Winter Coats).
  • Action: Filter “Content Mismatch” → Modify H1 to ensure consistency with content.

Effect: A clothing brand optimized 100 product H1s (changing “Product Details” to “Fleece Hoodie – Men’s/Women’s”), and average dwell time increased from 45 to 70 seconds.

Image Alt Attribute Report

Alt attributes are text descriptions for images. Missing or keyword-stuffed Alt tags waste image search traffic (about 30% of users find content via image search).

Key Data and Actions

• Missing Rate: About 40-50% of images lack Alt attributes (especially product and detail shots).
  • Action: Filter “No Alt Text” → Supplement descriptions (e.g., “Red running shoe side breathable mesh close-up”).
• Keyword Stuffing: About 10-15% contain repetitive keywords.
  • Action: Filter “Keyword Stuffing” → Change to natural descriptions.

Case: A sports brand added specific Alt attributes to 200 product images, and traffic from image search grew by 25%.

Batch Checking Internal Link Issues

Statistics show that sites that don’t batch check internal links have an average of 15-20% of pages unable to be effectively indexed. After fixing these issues, crawl volume for those pages can increase by over 30%.

Batch checking isn’t about “looking at links one by one,” but using Screaming Frog’s “Internal” report to find problems fast.

Internal Broken Links

Internal broken links point to deleted or inaccessible pages (404). Users will bounce upon clicking these, and crawlers will reduce crawling due to frequent 404 encounters.

Data and Actions

• Common Sources: Navigation bars (30-40%), old article recommendations (25-30%), user-entered comments (15-20%).
• Detection Method:
  1. Select “Internal” in the left menu → Click “Status Code” to filter for “404”;
  2. Export results and use Excel to track “Source URL” and “Target URL.”

Case: An education site had 12 “Popular Course” links in the nav, with 8 pointing to 404 pages of discontinued courses. Removing them increased the nav page’s crawl volume from 150 to 220 per day.

Resolution Actions

• Delete broken internal links (for invalid content);
• Replace with valid links (e.g., change “Old Course” to “Latest Course”);
• If the target page must be kept, set a 301 redirect.

Orphan Pages

Orphan pages have content but no internal links pointing to them (Incoming Links = 0). Crawlers can only find these via external links or direct URLs, making their index probability 60% lower than linked pages.

Data and Actions

• Common Types:
  • Temporary campaign pages (not deleted after use);
  • Test pages (e.g., “New Feature Demo”);
  • Low-quality pages (e.g., duplicate product spec pages).
• Detection Method:
  1. Filter “Linked From = 0” in the “Indexability” report;
  2. Or filter “Incoming Links = 0” and “Word Count > 100” in the “Internal” report.

Data: An e-commerce site found 200 orphan pages (old product details). After adding internal links, their index rate rose from 15% to 70%.

Resolution Actions

• Add internal links for high-value orphan pages;
• Delete or block low-value orphan pages via robots.txt;
• Regularly check for new orphans weekly.

Authority Concentration

Authority concentration occurs when the homepage or a few core pages have too many links (e.g., 50 footer links), dispersing the crawler’s “energy” and reducing crawl opportunities for other important pages (e.g., product pages, blog posts).

Data and Actions

• Typical Sign: Homepage “Outgoing Links” exceeds 50 (ideal is 20-30);
• Quantified Impact: A furniture site’s homepage had 68 links, pushing core product pages to level 4 depth, reducing daily crawls by 40%.

Detection Method

1. Sort by “Outgoing Links” descending in the “Internal” report;
2. Focus on the link count of core pages like the homepage and category pages.

Resolution Actions

• Streamline non-core links (move “Contact Us” to footer, keep 5-8 core sections on top);
• Move secondary links into “More” dropdown menus;
• Increase internal links for core pages (bestsellers, high-conversion articles) within related content.

3 Batch Processing Tips

1. Use Excel to filter frequent issues: After exporting, use “Filter” to locate source pages that repeatedly point to 404s.
2. Prioritize internal links on high-authority pages: Homepage and category page links have the most impact; fix these first.
3. Regular Reviews: Crawl every two weeks to compare data (e.g., if broken link counts decreased or orphans appeared), ensuring a healthy internal structure.

Finally, tools are just aids. The core of Google rankings will always be “content that users need.”