2 |
3 | **Latest Updates! 🎉 See the [change log](./CHANGELOG.md) for details.**
4 |
5 | 
6 |
7 | # astro-robots
8 |
9 | It simplifies SEO management with a reliable robots.txt generator for Astro projects, offering zero-config setup and [Verified Bots](https://radar.cloudflare.com/traffic/verified-bots) support.
10 |
11 | [](https://github.com/ACP-CODE/astro-robots/actions/workflows/release.yaml)
12 | 
13 | 
14 |
15 |
16 |
17 | ## Installation
18 |
19 | > The package compatible with Astro 4.0.0 and later.
20 |
21 | Quickly install with the `astro add` command:
22 |
23 | ```sh
24 | npx astro add astro-robots
25 | ```
26 |
27 | If you run into issues, try with [Manual Setup](#setup) guide.
28 |
29 | ## Usage
30 |
31 |
6 |
7 | # astro-robots
8 |
9 | It simplifies SEO management with a reliable robots.txt generator for Astro projects, offering zero-config setup and [Verified Bots](https://radar.cloudflare.com/traffic/verified-bots) support.
10 |
11 | [](https://github.com/ACP-CODE/astro-robots/actions/workflows/release.yaml)
12 | 
13 | 
14 |
15 |
32 |
77 |
78 | Manual Setup
33 | 34 | #### Setup 35 | 36 | To install manually, run: 37 | 38 | ```sh 39 | npm install astro-robots 40 | ``` 41 | 42 | Add the integration to your Astro config file (`astro.config.*`): 43 | 44 | ```ts 45 | // @ts-check 46 | import { defineConfig } from "astro/config"; 47 | import robots from "astro-robots"; // Add code manually 48 | 49 | export default defineConfig({ 50 | site: "https://example.com"; // If you want to support `@astrojs/sitemap` please provide this value 51 | integrations: [robots()], // Add code manually 52 | }); 53 | ``` 54 | 55 | After installing, run `npm run build` or `yarn build` in terminal: 56 | 57 | ```sh 58 | npm run build 59 | ``` 60 | 61 | This will output `robots.txt` to the `dist` folder with default rules: 62 | 63 | ```yaml 64 | User-agent: * 65 | Allow: / 66 | 67 | # crawling rule(s) for above bots 68 | Sitemap: https://example.com/sitemap-index.xml 69 | ``` 70 | 71 | #### Live Access 72 | 73 | Start the server with `npm run dev`, then access the virtual `robots.txt` at `http://localhost:4321/robots.txt`. 74 | 75 | 76 |
79 |
80 |
81 | 82 | To configure the integration, pass an object to the `robots()` function in your `astro.config.*` file: 83 | 84 | ```ts 85 | // @ts-check 86 | import { defineConfig } from "astro/config"; 87 | import robots from "astro-robots"; 88 | 89 | export default defineConfig({ 90 | //... 91 | integrations: [ 92 | robots({ 93 | host: "https://example.com";, 94 | sitemap: [ 95 | "https://example.com/sitemap.xml", 96 | "https://www.example.com/sitemap.xml", 97 | ], 98 | policy: [ 99 | { 100 | userAgent: [ 101 | "Applebot", 102 | "Googlebot", 103 | "bingbot", 104 | "Yandex", 105 | "Yeti", 106 | "Baiduspider", 107 | "360Spider", 108 | "*", 109 | ], 110 | allow: ["/"], 111 | disallow: ["/admin", "/login"], 112 | crawlDelay: 5, 113 | cleanParam: ["sid /", "s /forum/showthread"], 114 | }, 115 | { 116 | userAgent: "BLEXBot", 117 | disallow: ["/assets", "/uploades/1989-08-21/*jpg$"], 118 | }, 119 | ], 120 | }), 121 | ], 122 | }); 123 | ``` 124 | 125 | With the above configuration, the generated `robots.txt` file will look like this: 126 | 127 | ```yaml 128 | User-agent: Applebot 129 | User-agent: Googlebot 130 | User-agent: bingbot 131 | User-agent: Yandex 132 | User-agent: Yeti 133 | User-agent: Baiduspider 134 | User-agent: 360Spider 135 | User-agent: * 136 | Allow: / 137 | Disallow: /admin 138 | Disallow: /login 139 | Crawl-delay: 5 140 | Clean-param: sid / 141 | Clean-param: s /forum/showthread 142 | 143 | User-agent: BLEXBot 144 | Disallow: /assets 145 | Disallow: /uploades/1989-08-21/*jpg$ 146 | 147 | # crawling rule(s) for above bots 148 | Sitemap: https://example.com/sitemap.xml 149 | Sitemap: https://www.example.com/sitemap.xml 150 | Host: example.com 151 | ``` 152 | 153 | > **Note:** Some directives like `Host`, `Clean-param`, and `Crawl-delay` may not be supported by all crawlers. For example, Yandex has ignored `Crawl-delay` since February 2018. To control Yandex's crawl rate, use the [Site crawl rate setting](https://yandex.com/support/webmaster/service/crawl-rate.html#crawl-rate) in Yandex Webmaster. 154 | 155 |
156 |
157 |
158 | ## Contributing
159 |
160 | Submit your issues or feedback on our [GitHub](https://github.com/ACP-CODE/astro-robots/issues) channel.
161 |
162 | ## License
163 |
164 | MIT
165 |
--------------------------------------------------------------------------------
/src/core.ts:
--------------------------------------------------------------------------------
1 | import type { RobotsOptions } from ".";
2 | import type { Policy } from "./types";
3 | import type { AstroIntegrationLogger } from "astro";
4 | import path from "node:path";
5 |
6 | function validatePolicy(policy: Policy[], logger: AstroIntegrationLogger) {
7 | if (policy.length === 0) {
8 | throwMsg("Policy array must contain at least one entry.", "error", logger);
9 | }
10 |
11 | policy.forEach((policy, index) => {
12 | if (!policy.userAgent) {
13 | throwMsg(`policy[${index}].userAgent is required.`, "error", logger);
14 | }
15 |
16 | if (
17 | (!policy.allow && !policy.disallow) ||
18 | (policy.allow?.length === 0 && policy.disallow?.length === 0)
19 | ) {
20 | throwMsg(
21 | `policy[${index}] must have at least one 'disallow' or 'allow' entry.`,
22 | "error",
23 | logger,
24 | );
25 | }
26 |
27 | if (policy.crawlDelay) {
28 | if (
29 | typeof policy.crawlDelay !== "number" ||
30 | policy.crawlDelay < 0.1 ||
31 | policy.crawlDelay > 60
32 | ) {
33 | throwMsg(
34 | `policy[${index}].crawlDelay must be between 0.1 and 60 seconds.`,
35 | "error",
36 | logger,
37 | );
38 | }
39 | }
40 | });
41 | }
42 |
43 | function generatePolicyContent(policy: Policy[]): string {
44 | return policy
45 | .map((rule) => {
46 | let content = `User-agent: ${Array.isArray(rule.userAgent) ? rule.userAgent.join("\nUser-agent: ") : rule.userAgent || "*"}\n`;
47 |
48 | if (rule.allow) {
49 | const allowPaths = Array.isArray(rule.allow)
50 | ? rule.allow
51 | : [rule.allow];
52 | allowPaths.forEach((path: any) => (content += `Allow: ${path}\n`));
53 | }
54 |
55 | if (rule.disallow) {
56 | const disallowPaths = Array.isArray(rule.disallow)
57 | ? rule.disallow
58 | : [rule.disallow];
59 | disallowPaths.forEach(
60 | (path: any) => (content += `Disallow: ${path}\n`),
61 | );
62 | }
63 |
64 | if (rule.crawlDelay) content += `Crawl-delay: ${rule.crawlDelay}\n`;
65 |
66 | if (rule.cleanParam) {
67 | const cleanParams = Array.isArray(rule.cleanParam)
68 | ? rule.cleanParam
69 | : [rule.cleanParam];
70 | cleanParams.forEach(
71 | (param: any) => (content += `Clean-param: ${param}\n`),
72 | );
73 | }
74 |
75 | return content;
76 | })
77 | .join("\n");
78 | }
79 |
80 | function validateSitemapUrl(url: string): boolean {
81 | const urlPattern =
82 | /^(https?:\/\/)[^\s/$.?#].[^\s]*\.(xml|xml\.gz|txt|txt\.gz|json|xhtml)$/i;
83 | return urlPattern.test(url);
84 | }
85 |
86 | function generateSitemapContent(
87 | options: RobotsOptions,
88 | siteHref: string,
89 | logger: AstroIntegrationLogger,
90 | ): string {
91 | if (!options.sitemap) return "";
92 |
93 | if (options.sitemap === true) {
94 | if (siteHref) {
95 | const defaultSitemap = `${siteHref}sitemap-index.xml`;
96 | return `Sitemap: ${defaultSitemap}`;
97 | } else {
98 | logger.warn(
99 | "`sitemap` is configured as true, but `AstroConfig.site` is not provided. Failed to generate default sitemap URL.",
100 | );
101 | return "";
102 | }
103 | }
104 |
105 | const sitemaps = Array.isArray(options.sitemap)
106 | ? options.sitemap
107 | : [options.sitemap];
108 | const validSitemaps = sitemaps.filter((url) => {
109 | if (validateSitemapUrl(url)) {
110 | return true;
111 | } else {
112 | logger.warn(`\x1b[33mInvalid Sitemap URL: ${url}\x1b[33m`);
113 | return false;
114 | }
115 | });
116 |
117 | return validSitemaps.map((url) => `Sitemap: ${url}`).join("\n") + "\n";
118 | }
119 |
120 | function validateHost(host: string, logger: AstroIntegrationLogger) {
121 | const hostPattern =
122 | /^(?=.{1,253}$)(?:(?!-)[a-zA-Z0-9-]{1,63}(? logger.warn(`\x1b[33mWarning: ${msg}\x1b[0m`),
151 | error: () => {
152 | logger.error(`\x1b[31m${msg}\x1b[0m`);
153 | throw new Error(msg);
154 | },
155 | };
156 |
157 | formattedMsg[type]?.();
158 | }
159 |
160 | export function generate(
161 | options: RobotsOptions,
162 | siteMapHref: string,
163 | logger: AstroIntegrationLogger,
164 | ): string {
165 | if (!options.policy || options.policy.length === 0) {
166 | throwMsg("Policy configuration is required.", "error", logger);
167 | }
168 |
169 | validatePolicy(options.policy, logger);
170 |
171 | const content = [
172 | generatePolicyContent(options.policy),
173 | generateSitemapContent(options, siteMapHref, logger),
174 | generateHostContent(options, logger),
175 | ].join("\n");
176 |
177 | return content.trim();
178 | }
179 |
180 | export function logInfo(
181 | fileSize: number,
182 | logger: AstroIntegrationLogger,
183 | destDir: string,
184 | ) {
185 | const sizeLimit = 10;
186 | if (fileSize > sizeLimit) {
187 | console.log(
188 | `\n\x1b[42m\x1b[30m Generating 'robots.txt' file \x1b[39m\x1b[0m`,
189 | );
190 | const warnMsg = [
191 | `\x1b[33m(!) 'robots.txt' file size is ${fileSize} KB.`,
192 | `- Keep it under ${sizeLimit} KB for best results.\x1b[0m\n`,
193 | ];
194 | console.log(warnMsg.join("\n"));
195 | }
196 |
197 | logger.info(
198 | `\`robots.txt\` (${fileSize}KB) created at \`${path.relative(process.cwd(), destDir)}\``,
199 | );
200 | }
201 |
--------------------------------------------------------------------------------
Getting Started with Reference
81 | 82 | To configure the integration, pass an object to the `robots()` function in your `astro.config.*` file: 83 | 84 | ```ts 85 | // @ts-check 86 | import { defineConfig } from "astro/config"; 87 | import robots from "astro-robots"; 88 | 89 | export default defineConfig({ 90 | //... 91 | integrations: [ 92 | robots({ 93 | host: "https://example.com";, 94 | sitemap: [ 95 | "https://example.com/sitemap.xml", 96 | "https://www.example.com/sitemap.xml", 97 | ], 98 | policy: [ 99 | { 100 | userAgent: [ 101 | "Applebot", 102 | "Googlebot", 103 | "bingbot", 104 | "Yandex", 105 | "Yeti", 106 | "Baiduspider", 107 | "360Spider", 108 | "*", 109 | ], 110 | allow: ["/"], 111 | disallow: ["/admin", "/login"], 112 | crawlDelay: 5, 113 | cleanParam: ["sid /", "s /forum/showthread"], 114 | }, 115 | { 116 | userAgent: "BLEXBot", 117 | disallow: ["/assets", "/uploades/1989-08-21/*jpg$"], 118 | }, 119 | ], 120 | }), 121 | ], 122 | }); 123 | ``` 124 | 125 | With the above configuration, the generated `robots.txt` file will look like this: 126 | 127 | ```yaml 128 | User-agent: Applebot 129 | User-agent: Googlebot 130 | User-agent: bingbot 131 | User-agent: Yandex 132 | User-agent: Yeti 133 | User-agent: Baiduspider 134 | User-agent: 360Spider 135 | User-agent: * 136 | Allow: / 137 | Disallow: /admin 138 | Disallow: /login 139 | Crawl-delay: 5 140 | Clean-param: sid / 141 | Clean-param: s /forum/showthread 142 | 143 | User-agent: BLEXBot 144 | Disallow: /assets 145 | Disallow: /uploades/1989-08-21/*jpg$ 146 | 147 | # crawling rule(s) for above bots 148 | Sitemap: https://example.com/sitemap.xml 149 | Sitemap: https://www.example.com/sitemap.xml 150 | Host: example.com 151 | ``` 152 | 153 | > **Note:** Some directives like `Host`, `Clean-param`, and `Crawl-delay` may not be supported by all crawlers. For example, Yandex has ignored `Crawl-delay` since February 2018. To control Yandex's crawl rate, use the [Site crawl rate setting](https://yandex.com/support/webmaster/service/crawl-rate.html#crawl-rate) in Yandex Webmaster. 154 | 155 |
156 |