1. What is NovaBot?
NovaBot is an automated web crawler operated by Novaverb. It discovers, fetches, and analyzes web pages to provide site audit, keyword intelligence, and AI Search Visibility signals for website owners and digital marketing professionals.
2. User-Agent Strings
NovaBot identifies itself using a stable, transparent User-Agent header on all HTTP requests:
Mozilla/5.0 (compatible; NovaBot/1.0; +https://novaverb.com/bot; bot@novaverb.com)
3. robots.txt & Meta Directive Compliance
NovaBot follows robots.txt (RFC 9309) and standard robots directives. The controls below explain how website owners can manage NovaBot when Novaverb analyzes a project or discovers public links.
Use the product token NovaBot in your robots.txt. NovaBot honors User-agent: NovaBot and User-agent: * groups. To block all NovaBot crawling, add:
User-agent: NovaBot Disallow: /
| Control | Project analysis | Public link discovery |
|---|---|---|
| User-agent: NovaBot | Supported | Supported |
| Allow / Disallow | Supported | Supported |
| Group merge (RFC 9309 §2.2.1) | Supported | Supported |
| robots 5xx / network → fail-closed | Supported | Supported |
| Crawl-delay | Fixed concurrency limits | Minimum default delay |
| 429 / 503 Retry-After | Partial | Not yet |
| Meta / X-Robots nofollow | Partial | Partial |
| FCrDNS verification | Verified | Verified |
The most reliable way to control NovaBot is robots.txt - it applies to every path above. We do not claim noindex/noarchive/nosnippet support that we do not enforce.
A second, separate identity: NovaAI (Reserved - not currently crawling)
Measures and optimizes how pages appear in AI answer engines (NovaCite, AI Search Visibility) and fetches on behalf of NovaAssistant when a user asks it to browse. Separate from SEO crawling so you can allow or deny AI use independently. NovaAI is not crawling today - Novaverb's AI features currently run through third-party AI APIs, not a crawler on your site. We publish the identity so you can control it in advance.
To allow SEO crawling but deny AI use (or block both), control each token independently:
User-agent: NovaBot Allow: / User-agent: NovaAI Disallow: /
4. Verifying NovaBot Traffic & Published IPs
Verify NovaBot the same way you verify Googlebot: confirm the connecting IP against the published list below, and confirm the User-Agent. Where a crawler path has forward-confirmed reverse DNS (FCrDNS), you can also verify it cryptographically via DNS.
| Crawler path | Egress IP | Verify by |
|---|---|---|
| Web-graph miner (discovery) | 167.233.209.111 |
FCrDNS → novabot.novaverb.com |
| Workspace crawler (audit) | 178.105.213.71 |
FCrDNS → novabot.novaverb.com |
| Workspace crawler (audit, IPv6) | 2a01:4f8:c013:8a60::1 |
FCrDNS → novabot.novaverb.com |
A machine-readable copy of this list is published at /bot/ips.json.
- For an FCrDNS path: run a reverse DNS (PTR) lookup on the connecting IP, confirm it resolves under novaverb.com, then forward-resolve that hostname back to the same IP. An impersonator cannot make both directions agree.
- Also confirm the User-Agent matches the exact string in section 2.
5. Politeness & Server Load Protection
NovaBot is designed to minimize server load. Support differs by crawler path (see the table in section 3):
- Concurrent connections per origin: 2.
- Minimum delay between requests: 2 seconds. Additional safeguards limit request volume during project analysis.
- robots.txt is cached at most 24h (errors far shorter: 5m).
- On 429/503 without Retry-After, NovaBot backs off progressively: 5m → 30m → 2h → 24h. NovaBot never rotates IPs to bypass a 403, 429 or CAPTCHA.
6. Domain Opt-Out Request
Two levers are available. The most reliable is robots.txt: add User-agent: NovaBot / Disallow: / to your /robots.txt and NovaBot applies it on its next scan of your host. The form below is a second channel: your request is recorded and reviewed by our crawler-operations team; once approved, it is enforced at crawl time.
Service window: a case ID / confirmation within 5 minutes; discovery crawling paused within 15 minutes; an in-progress scan cancelled within 5 minutes. A site owner who has verified ownership can always audit their own property.
7. Crawler Changelog
- 2026-07-19 - robots.txt fail-closed on 5xx/network; exact product-token match + group merge (RFC 9309); JS renderer sends the NovaBot UA; miner re-validates every redirect hop against SSRF; edge records stamp first_seen/last_seen.
8. Contact Us
If you have questions, feedback, or notice unexpected behavior from NovaBot on your website, please email our webmaster operations team at: