Crawl Budget = Jumlah URL yang Googlebot akan crawl
               dalam periode waktu tertentu

Ditentukan Oleh:
1. Crawl Rate Limit
   - Seberapa cepat Googlebot bisa crawl tanpa overload server

2. Crawl Demand
   - Seberapa banyak Google "ingin" crawl site Anda
   - Berdasarkan popularitas dan freshness need

CRAWL RATE LIMIT:
"How fast can Google crawl without breaking your server?"

Factors:
- Server response time
- Server capacity
- Error rates
- Googlebot's politeness settings

If Server is Fast:
→ Higher crawl rate possible
→ More pages crawled per day

If Server is Slow:
→ Googlebot backs off
→ Fewer pages crawled
→ Crawl budget wasted on slow responses


CRAWL DEMAND:
"How much does Google want to crawl your site?"

Factors:
- Site popularity (links, traffic)
- Content freshness needs
- Historical crawl patterns
- New content discovery rate

High Demand Sites:
→ News sites (need fresh content)
→ E-commerce (inventory changes)
→ High authority sites

Low Demand Sites:
→ Static sites
→ Infrequently updated
→ Low authority/traffic

Googlebot Crawl Process:

1. URL DISCOVERY
   ├── From sitemaps
   ├── From internal links
   ├── From external links
   └── From previous crawls

2. QUEUE PRIORITIZATION
   ├── Popular pages first
   ├── Fresh/updated content
   ├── Pages linked from important sources
   └── New URLs

3. CRAWLING
   ├── Download page
   ├── Parse content
   ├── Discover new links
   └── Move to next URL

4. RENDERING (for JS)
   ├── Execute JavaScript
   ├── Generate final DOM
   └── Additional resource usage

Budget Consumed By:
- Every URL crawled
- Failed requests (404s, 500s)
- Redirects (follows chain)
- Slow responses
- Rendering JavaScript

Crawl Stats Report:

Location: Settings → Crawl stats

Metrics Shown:
- Total crawl requests
- Total download size
- Average response time

Breakdowns:
- By response code
- By file type
- By Googlebot type
- By purpose (Discovery vs Refresh)

What to Look For:
✓ Consistent or growing crawl requests
✓ Low error rates (< 5%)
✓ Fast average response time (< 500ms)
✓ Appropriate file type distribution

Red Flags:
✗ Declining crawl requests
✗ High error percentage
✗ Slow response times
✗ Too many non-HTML crawls

Server Log Analysis:

What to Extract:
- Googlebot requests by URL
- Response codes returned
- Response times
- Crawl frequency per URL

Tools:
- Screaming Frog Log Analyzer
- Semrush Log File Analyzer
- JetOctopus
- Botify
- OnCrawl

Key Insights:
1. Which URLs are crawled most?
2. Which important URLs NOT crawled?
3. What's the crawl distribution?
4. Are soft 404s being crawled?
5. How much time on low-value pages?

Example Log Entry:
66.249.66.1 - - [28/Dec/2026:10:15:32] "GET /products/shoes HTTP/1.1" 200 45678 "-" "Googlebot/2.1"

Speed = More Crawling Possible

Optimization:
☐ Fast server response (TTFB < 200ms ideal)
☐ Enable compression (Brotli/Gzip)
☐ Efficient database queries
☐ CDN for global performance
☐ Adequate server resources
☐ HTTP/2 enabled

Impact:
Faster server → Googlebot crawls more
Slow server → Googlebot backs off

Monitoring:
- Search Console: Average response time
- Real user metrics
- Synthetic monitoring (uptime tools)

What Wastes Crawl Budget:

1. SOFT 404s
   Pages returning 200 but essentially empty
   Fix: Return proper 404 status

2. DUPLICATE CONTENT
   Multiple URLs, same content
   Fix: Canonicalization, redirects

3. FACETED NAVIGATION
   Endless URL combinations
   Fix: noindex or parameter handling

4. SESSION IDs IN URLs
   example.com/page?sessionid=abc123
   Fix: Remove or use canonical

5. INFINITE SPACES
   Calendar pages, sort/filter combinations
   Fix: Block in robots.txt

6. REDIRECT CHAINS
   A → B → C → D
   Fix: Direct to final destination

7. LOW-VALUE PAGES
   Tag pages, author archives, search results
   Fix: noindex or consolidate

Tell Google What Matters:

INTERNAL LINKING:
- Important pages linked from many pages
- Homepage links to key sections
- Silo structure for topic clusters

XML SITEMAP:
- Only include indexable, valuable pages
- Update lastmod when content changes
- Prioritize within sitemap

ROBOTS.TXT:
- Block low-value sections
- Don't block important content
- Don't block CSS/JS needed for rendering

PAGE IMPORTANCE SIGNALS:
- Backlinks indicate importance
- Traffic signals value
- Fresh content gets priority

Strategic Blocking:

# robots.txt
User-agent: *

# Block low-value pages
Disallow: /search/
Disallow: /filter/
Disallow: /sort/
Disallow: /*?sort=
Disallow: /*?filter=
Disallow: /tag/
Disallow: /author/
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/

# Allow important sections
Allow: /products/
Allow: /blog/
Allow: /

# Sitemap location
Sitemap: https://example.com/sitemap.xml

Important:
- Blocking doesn't prevent indexing (only crawling)
- Use noindex for true de-indexing
- Don't block CSS/JS
- Test with robots.txt tester

Sitemap Best Practices for Crawl Budget:

INCLUDE:
✅ All indexable pages
✅ Canonical versions only
✅ Updated lastmod dates
✅ Pages you WANT crawled

EXCLUDE:
❌ Noindexed pages
❌ Redirected URLs
❌ Blocked by robots.txt
❌ Duplicate content
❌ Low-value pages

STRUCTURE:
<sitemapindex>
  <sitemap>
    <loc>sitemap-products.xml</loc>
  </sitemap>
  <sitemap>
    <loc>sitemap-blog.xml</loc>
  </sitemap>
  <sitemap>
    <loc>sitemap-categories.xml</loc>
  </sitemap>
</sitemapindex>

Benefits:
- Organize by content type
- Easier management
- Clear priority signals
- Identify crawl patterns

Parameter Problems:

example.com/products?color=red
example.com/products?color=blue
example.com/products?color=red&size=large
example.com/products?size=large&color=red
...infinite combinations

Solutions:

1. CANONICAL TAGS
   All variations point to base URL
   <link rel="canonical" href="/products/" />

2. ROBOTS.TXT
   Disallow: /*?color=
   Disallow: /*?size=

3. NOINDEX TAG
   <meta name="robots" content="noindex,follow">

4. URL PARAMETER TOOL (Legacy)
   Previously in Search Console
   Now deprecated - use other methods

Best Practice:
Use clean URLs without parameters for important pages
Handle filtering via JavaScript (not URL changes)
OR use canonical consistently

E-commerce Challenges:
- Thousands/millions of product pages
- Faceted navigation creates infinite URLs
- Product variations (size, color)
- Out-of-stock pages
- Price sort, review sort combinations

Strategy:

1. PRODUCT PAGES
   - Include in sitemap
   - Strong internal linking
   - Canonical to main variant

2. CATEGORY PAGES
   - Top categories: crawlable
   - Deep filters: noindex or block

3. FACETED NAVIGATION
   - Base category: indexable
   - Filtered views: canonical to base
   - OR: Block parameters in robots.txt

4. OUT OF STOCK
   - Keep page if temporary
   - 301 redirect if permanent
   - Don't create thin pages

5. PAGINATION
   - page 1: canonical to self
   - page 2+: allow crawl, consider noindex
   - Load more: ensure links discoverable

News Site Challenges:
- High content velocity
- Freshness critical
- Archive pages
- Tag/category proliferation

Strategy:

1. NEW CONTENT
   - Publish to sitemap immediately
   - Ping Google (sitemap resubmit)
   - Link from homepage
   - Push via Google News

2. ARCHIVE CONTENT
   - Lower crawl priority natural
   - Ensure still crawlable
   - Update timestamps when refreshed

3. TAXONOMY PAGES
   - Limit tag pages
   - Consolidate similar topics
   - noindex low-value archives

4. PAGINATION
   - Don't over-paginate
   - Infinite scroll with fallback links

SERVER HEALTH:
☐ Average response time < 500ms
☐ Error rate < 5%
☐ No server crashes during crawls
☐ Adequate bandwidth

URL EFFICIENCY:
☐ No duplicate content issues
☐ No parameter URL explosions
☐ Redirect chains resolved
☐ 404s handled properly

SITEMAP HEALTH:
☐ Only indexable URLs
☐ No redirects in sitemap
☐ lastmod accurate
☐ Under 50,000 URLs per sitemap

ROBOTS.TXT:
☐ Low-value pages blocked
☐ Important pages allowed
☐ CSS/JS accessible
☐ No accidental blocks

CRAWL PATTERNS:
☐ Important pages crawled frequently
☐ No important pages orphaned
☐ Crawl distribution logical
☐ No wasted crawls on junk

DON'T WORRY If:
- Site under 10,000 pages
- Content doesn't change rapidly
- No major technical issues
- Server performs well

SHOULD CARE If:
- 10,000+ pages
- E-commerce with many products
- News/content site with high velocity
- Site with many URL parameters
- Experiencing indexing issues

Signs of Crawl Budget Problems:
- New pages not indexed for weeks
- Important pages rarely refreshed
- Search Console shows declining crawl rate
- Many pages "Discovered - not indexed"

For Small Sites:
Focus on other SEO factors
Crawl budget rarely the bottleneck

Improve Crawl Rate Limit:
- Faster server response
- Better hosting
- CDN implementation
- Efficient code

Improve Crawl Demand:
- Build more backlinks
- Create more valuable content
- Increase site popularity
- Update content regularly

Reduce Budget Waste:
- Fix technical issues
- Remove duplicate content
- Block low-value pages
- Fix redirect chains

Make Budget More Effective:
- Prioritize important pages
- Better internal linking
- Clean sitemap
- Strategic robots.txt

Result:
Same budget, but used on important pages
= Better indexing of valuable content

robots.txt Blocking:
✅ Prevents crawling of blocked URLs
✅ Saves crawl budget for other pages
✅ Good for low-value pages

BUT:
❌ Doesn't prevent INDEXING
❌ URLs can still appear in search (if linked)
❌ Shows as "Blocked by robots.txt" in Search Console

If You Want to:
- Save crawl budget only → robots.txt
- Prevent indexing → noindex meta tag
- Both → noindex (Google won't crawl noindexed pages much)

Common Mistake:
Blocking page in robots.txt but wanting it de-indexed
Google can't see noindex tag if blocked!

Correct Approach:
- Don't block, add noindex
- Wait for de-indexing
- Then optionally block

Factors Affecting Crawl Frequency:

Site-Level:
- Site authority/popularity
- Update frequency
- Site size
- Server capacity

Page-Level:
- Page importance
- Update frequency
- Internal/external links
- Historical patterns

Typical Ranges:
- Homepage: Daily to hourly
- Category pages: Daily to weekly
- Blog posts: Weekly to monthly
- Deep pages: Monthly or less

Check in Search Console:
URL Inspection → Last crawl date
Crawl stats → Overall patterns

Increase Crawl Frequency:
- Update content regularly
- Build more links
- Improve internal linking
- Keep server fast
- Submit updated sitemap

How It Actually Works:
- No hard "budget" number
- Googlebot decides dynamically
- Balances across all sites

What Happens with Limited Budget:
- Some pages not crawled
- Less frequent refreshes
- New pages indexed slowly
- Updates not reflected quickly

Impact on SEO:
- Important content may not rank
- Fresh content advantage lost
- Competitive disadvantage
- Index not reflecting site reality

Not a "Penalty":
Just resource allocation
Improve signals = more resources
Fix issues = better efficiency

Immediate:
☐ Check crawl stats in Search Console
☐ Identify error spikes
☐ Review response time trends

This Week:
☐ Audit robots.txt
☐ Check sitemap for non-indexable URLs
☐ Identify URL parameter issues

This Month:
☐ Implement log file analysis
☐ Fix major crawl waste issues
☐ Optimize server performance

Ongoing:
☐ Monthly crawl budget review
☐ Monitor new page indexing speed
☐ Track important page crawl frequency