Help us fix this page
If you found a broken link, missing page, or incorrect redirect, please let us know. Your report helps us improve the website for everyone.
robotstxt.js – robots.txt Parser for JavaScript
robotstxt.js is a lightweight JavaScript library for parsing and interpreting robots.txt files in both browser and Node.js environments. It provides a standards-compliant implementation for evaluating crawler permissions, parsing directives, discovering XML sitemaps, and inspecting user-agent specific rules. Designed for SEO tools, crawlers, validators, and web applications, the library makes it easy to work with robots.txt programmatically while following established robots exclusion standards.
Key Features
- Standards-compliant robots.txt parser
- Browser and Node.js support
- User-agent rule evaluation
- Allow and Disallow directive parsing
- Crawl-delay, Cache-delay and Request-rate support
- Sitemap and Host directive discovery
- Wildcard (
*) and end-of-path ($) matching - Case-insensitive user-agent matching
- Parsing reports, warnings and diagnostics
Typical Applications
- SEO auditing tools
- robots.txt validators
- Web crawlers and spiders
- Search engine utilities
- Website monitoring platforms
- Technical SEO applications
- Website migration tools
- Custom indexing solutions
Installation
Install the latest version from NPM.
npm install @playfulsparkle/robotstxt-jsor
yarn add @playfulsparkle/robotstxt-jsDownload Details
- GitHub URL: https://github.com/playfulsparkle/robotstxt.js
- npm JavaScript Package Manager: https://www.npmjs.com/package/@playfulsparkle/robotstxt-js