Cara Menganalisis Log File untuk SEO

Log files:
- Server records of all requests
- Every visit logged
- Bot visits included
- Technical details recorded

Analysis = Understanding crawl behavior.

Benefits:
1. See real Googlebot activity
2. Find crawl issues
3. Discover orphan pages
4. Understand crawl budget usage
5. Verify bot access
6. Technical troubleshooting

Standard log entry:
66.249.66.1 - - [07/Jan/2026:10:15:30 +0000]
"GET /page/ HTTP/1.1" 200 15234 "-"
"Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)"

Contains:
- IP address
- Timestamp
- Request method/URL
- Status code
- Response size
- User agent

Important data:
1. IP address (identify bot)
2. URL requested
3. Status code (200, 404, 301, etc.)
4. User agent (bot identifier)
5. Timestamp (crawl timing)
6. Referrer (where bot came from)

Common formats:
- Combined (most common)
- Common
- W3C Extended
- Custom formats

Check your server config.

Location:
File Manager → /logs/

Or:
Metrics → Raw Access

Download access_log files.

Logs & Statistics → Logs

Download access logs.

Default location:
/var/log/apache2/access.log
/var/log/httpd/access_log

May need server access.

Default location:
/var/log/nginx/access.log

Configure in nginx.conf.

Cloudflare Dashboard → Analytics → Logs

Or:
Enterprise: Log Export to storage.

1. Screaming Frog Log Analyzer
   - Parse and analyze
   - Bot identification
   - Visual reports

2. GoAccess
   - Real-time analysis
   - Command line
   - HTML reports

3. Excel/Google Sheets
   - Manual analysis
   - Custom filtering

1. Oncrawl
   - Advanced log analysis
   - Combined with crawl data

2. Botify
   - Enterprise log analysis
   - Deep insights

3. JetOctopus
   - Log analyzer included
   - Visual reports

4. SEMrush Log Analyzer
   - Integrated with SEMrush

Process:
1. Download log file
2. Open Screaming Frog Log Analyzer
3. Import log file
4. Select log format
5. Analyze reports

User agent contains:
"Googlebot"
"Googlebot-Image"
"Googlebot-News"
"Googlebot-Video"
"AdsBot-Google"

IP ranges: Verify at google.com/bot.html

Reverse DNS lookup:
host 66.249.66.1

Should return:
*.googlebot.com or *.google.com

Fake bots won't pass this.

Types to track:
- Googlebot (main crawler)
- Googlebot-Mobile (mobile)
- Googlebot-Image (images)
- AdsBot (ad landing pages)
- APIs-Google (APIs)

Questions:
- How often is site crawled?
- Which pages crawled most?
- Crawl trends over time?

High-value pages should be crawled often.

Monitor:
- 200s (success)
- 301/302 (redirects)
- 404 (not found)
- 500s (server errors)

Googlebot hitting errors = bad.

Analyze:
- Total requests per day
- URLs per crawl session
- Time between visits
- Resource allocation

Is budget spent on important pages?

Check:
- Average response time
- Slow-loading pages
- Timeout errors

Slow responses = fewer pages crawled.

Find:
- Pages crawled but not in sitemap
- Pages not internally linked
- Unexpected URLs being crawled

May indicate site structure issues.

Compare bots:
+-------------+--------+--------+
| Bot         | Hits   | %      |
+-------------+--------+--------+
| Googlebot   | 50,000 | 60%    |
| Bingbot     | 20,000 | 24%    |
| Others      | 13,000 | 16%    |
+-------------+--------+--------+

Most crawled URLs:
1. /homepage/ - 5,000 hits
2. /products/ - 3,000 hits
3. /blog/ - 2,500 hits

Are important pages being crawled?

Status breakdown:
- 200 OK: 85%
- 301/302: 8%
- 404: 5%
- 500: 2%

High error rates = problem.

Issue:
Googlebot spending budget on:
- Parameter URLs
- Faceted navigation
- Sorted pages
- Search result pages

Solution:
- Robots.txt rules
- Meta noindex
- Canonical tags

Issue:
Key pages rarely/never crawled.

Causes:
- Deep in structure
- Few internal links
- Orphan pages
- Technical blocks

Solution:
- Improve internal linking
- Add to sitemap
- Reduce crawl depth

Issue:
Googlebot getting many 4xx/5xx.

Impact:
- Wastes crawl budget
- Pages not indexed
- Poor user experience signal

Solution:
- Fix errors
- Set up redirects
- Monitor server health

Issue:
High TTFB for Googlebot requests.

Impact:
- Fewer pages crawled
- Indexing delays
- Potential ranking impact

Solution:
- Server optimization
- Caching
- CDN
- Better hosting

From log analysis:
1. Fix server errors
2. Reduce crawl waste
3. Improve internal linking
4. Update sitemap
5. Optimize page speed

Schedule:
- Weekly: Quick check
- Monthly: Full analysis
- After changes: Verify impact

Track trends over time.

Setup:
☐ Access to log files
☐ Analysis tool chosen
☐ Bot verification method

Analysis:
☐ Identify Googlebot visits
☐ Check crawl frequency
☐ Review status codes
☐ Analyze crawl budget
☐ Find orphan pages
☐ Check response times

Action:
☐ Document findings
☐ Prioritize issues
☐ Implement fixes
☐ Monitor changes