├── docs
├── account
│ ├── reset-password.png
│ ├── add-email-address.png
│ ├── add-phone-number.png
│ ├── change-primary-email.png
│ ├── verify-phone-number.png
│ ├── referral-program.md
│ ├── password-recovery.md
│ ├── password-change.md
│ ├── sign-up.md
│ ├── change-primary-email.md
│ ├── add-notification-email.md
│ ├── add-notification-phone-number.md
│ ├── README.md
│ ├── account-removal.md
│ └── tips-and-tricks.md
├── monitoring
│ ├── custom-path.png
│ ├── custom-monitoring.png
│ ├── availability-monitoring.png
│ ├── monitoring-security-http.png
│ ├── custom-path.md
│ ├── with-auth.md
│ ├── basics.md
│ ├── README.md
│ ├── nginx-stats.md
│ └── snmp.md
├── prerendering
│ ├── prerendering-es6.png
│ ├── prerendering-cache.png
│ ├── prerendering-stripjs.png
│ ├── prerendering-cache-id.png
│ ├── prerendering-cache-purge.png
│ ├── prerendering-cache-purge-single.png
│ ├── prerendering-genuine-status-code.png
│ ├── strip-javascript.md
│ ├── es6-support.md
│ ├── amp-support.md
│ ├── prerendering-custom-status-codes.md
│ ├── cache.md
│ ├── cloudflare-worker.md
│ ├── meteor-atmosphere.md
│ ├── genuine-status-code.md
│ ├── nextjs-prerendering.md
│ ├── detect-prerendering-meteor.md
│ ├── rendering-endpoints.md
│ ├── detect-prerendering.md
│ ├── node-npm.md
│ ├── cache-purge.md
│ ├── optimization.md
│ ├── apache.md
│ ├── examples
│ │ ├── nginx
│ │ │ ├── phusion-passenger.conf
│ │ │ ├── php-fpm.conf
│ │ │ └── upstream.conf
│ │ └── cloudflare-worker
│ │ │ ├── cloudflare-listener.worker.js
│ │ │ └── coudflare-worker-guide.md
│ ├── netlify-prerendering.md
│ └── README.md
├── SUPPORT.md
├── web-cron
│ └── README.md
├── analytics
│ ├── README.md
│ ├── detect-adblock.md
│ └── track-amp.md
└── websec
│ └── README.md
├── wallpapers
├── ostrio-logo-iphone5s-640x1136.jpg
├── ostrio-logo-iphone5s-640x1136.png
├── ostrio-logo-iphoneXr-828x1792.jpg
├── ostrio-logo-iphoneXr-828x1792.png
├── ostrio-logo-iphone6_7_8-750x1334.jpg
├── ostrio-logo-iphone6_7_8-750x1334.png
├── ostrio-logo-iphoneXs-1125x2436.jpg
├── ostrio-logo-iphoneXs-1125x2436.png
├── ostrio-logo-iphoneXs_max-1242x2688.jpg
├── ostrio-logo-iphoneXs_max-1242x2688.png
├── ostrio-desktop-background-2880x1800.jpg
├── ostrio-desktop-background-2880x1800.png
├── ostrio-logo-iphone6_7_8plus-1242x2208.jpg
└── ostrio-logo-iphone6_7_8plus-1242x2208.png
├── tutorials
├── nginx
│ ├── ssl.conf
│ └── hacking-nginx-with-default.md
├── linux
│ ├── change-timezone.md
│ ├── users
│ │ ├── user-passwd.md
│ │ ├── root-passwd.md
│ │ ├── create-user.md
│ │ └── rbash.md
│ ├── security
│ │ ├── lynis-security-audit.md
│ │ ├── change-ssh-port.md
│ │ ├── sshd_config
│ │ ├── disable-ssh-root.md
│ │ ├── rootkit-hunter.md
│ │ ├── iptables-firewall.md
│ │ ├── iptables-firewall-ipv6.md
│ │ ├── use-ssh-keys.md
│ │ └── automated-backups.md
│ └── bash_profile-tuning.md
├── website
│ └── social-networking
│ │ ├── twitter-cards-basics.md
│ │ ├── open-graph-basics.md
│ │ ├── pinterest-rich-pins-basics.md
│ │ └── structured-data-basics.md
├── mongodb
│ ├── enable-oplog.md
│ ├── use-ssl-http-encryption.md
│ ├── single-server-replica-set.md
│ └── multiple-server-replica-set.md
├── README.md
└── ssl
│ ├── ssl-letsencrypt-dns-validation.md
│ └── ssl-letsencrypt.md
├── LICENSE
├── README.md
└── SITEMAP.md
/docs/account/reset-password.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/veliovgroup/ostrio/HEAD/docs/account/reset-password.png
--------------------------------------------------------------------------------
/docs/monitoring/custom-path.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/veliovgroup/ostrio/HEAD/docs/monitoring/custom-path.png
--------------------------------------------------------------------------------
/docs/account/add-email-address.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/veliovgroup/ostrio/HEAD/docs/account/add-email-address.png
--------------------------------------------------------------------------------
/docs/account/add-phone-number.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/veliovgroup/ostrio/HEAD/docs/account/add-phone-number.png
--------------------------------------------------------------------------------
/docs/account/change-primary-email.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/veliovgroup/ostrio/HEAD/docs/account/change-primary-email.png
--------------------------------------------------------------------------------
/docs/account/verify-phone-number.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/veliovgroup/ostrio/HEAD/docs/account/verify-phone-number.png
--------------------------------------------------------------------------------
/docs/monitoring/custom-monitoring.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/veliovgroup/ostrio/HEAD/docs/monitoring/custom-monitoring.png
--------------------------------------------------------------------------------
/docs/prerendering/prerendering-es6.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/veliovgroup/ostrio/HEAD/docs/prerendering/prerendering-es6.png
--------------------------------------------------------------------------------
/docs/prerendering/prerendering-cache.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/veliovgroup/ostrio/HEAD/docs/prerendering/prerendering-cache.png
--------------------------------------------------------------------------------
/docs/prerendering/prerendering-stripjs.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/veliovgroup/ostrio/HEAD/docs/prerendering/prerendering-stripjs.png
--------------------------------------------------------------------------------
/docs/monitoring/availability-monitoring.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/veliovgroup/ostrio/HEAD/docs/monitoring/availability-monitoring.png
--------------------------------------------------------------------------------
/docs/monitoring/monitoring-security-http.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/veliovgroup/ostrio/HEAD/docs/monitoring/monitoring-security-http.png
--------------------------------------------------------------------------------
/docs/prerendering/prerendering-cache-id.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/veliovgroup/ostrio/HEAD/docs/prerendering/prerendering-cache-id.png
--------------------------------------------------------------------------------
/wallpapers/ostrio-logo-iphone5s-640x1136.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/veliovgroup/ostrio/HEAD/wallpapers/ostrio-logo-iphone5s-640x1136.jpg
--------------------------------------------------------------------------------
/wallpapers/ostrio-logo-iphone5s-640x1136.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/veliovgroup/ostrio/HEAD/wallpapers/ostrio-logo-iphone5s-640x1136.png
--------------------------------------------------------------------------------
/wallpapers/ostrio-logo-iphoneXr-828x1792.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/veliovgroup/ostrio/HEAD/wallpapers/ostrio-logo-iphoneXr-828x1792.jpg
--------------------------------------------------------------------------------
/wallpapers/ostrio-logo-iphoneXr-828x1792.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/veliovgroup/ostrio/HEAD/wallpapers/ostrio-logo-iphoneXr-828x1792.png
--------------------------------------------------------------------------------
/docs/prerendering/prerendering-cache-purge.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/veliovgroup/ostrio/HEAD/docs/prerendering/prerendering-cache-purge.png
--------------------------------------------------------------------------------
/wallpapers/ostrio-logo-iphone6_7_8-750x1334.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/veliovgroup/ostrio/HEAD/wallpapers/ostrio-logo-iphone6_7_8-750x1334.jpg
--------------------------------------------------------------------------------
/wallpapers/ostrio-logo-iphone6_7_8-750x1334.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/veliovgroup/ostrio/HEAD/wallpapers/ostrio-logo-iphone6_7_8-750x1334.png
--------------------------------------------------------------------------------
/wallpapers/ostrio-logo-iphoneXs-1125x2436.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/veliovgroup/ostrio/HEAD/wallpapers/ostrio-logo-iphoneXs-1125x2436.jpg
--------------------------------------------------------------------------------
/wallpapers/ostrio-logo-iphoneXs-1125x2436.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/veliovgroup/ostrio/HEAD/wallpapers/ostrio-logo-iphoneXs-1125x2436.png
--------------------------------------------------------------------------------
/wallpapers/ostrio-logo-iphoneXs_max-1242x2688.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/veliovgroup/ostrio/HEAD/wallpapers/ostrio-logo-iphoneXs_max-1242x2688.jpg
--------------------------------------------------------------------------------
/wallpapers/ostrio-logo-iphoneXs_max-1242x2688.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/veliovgroup/ostrio/HEAD/wallpapers/ostrio-logo-iphoneXs_max-1242x2688.png
--------------------------------------------------------------------------------
/wallpapers/ostrio-desktop-background-2880x1800.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/veliovgroup/ostrio/HEAD/wallpapers/ostrio-desktop-background-2880x1800.jpg
--------------------------------------------------------------------------------
/wallpapers/ostrio-desktop-background-2880x1800.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/veliovgroup/ostrio/HEAD/wallpapers/ostrio-desktop-background-2880x1800.png
--------------------------------------------------------------------------------
/wallpapers/ostrio-logo-iphone6_7_8plus-1242x2208.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/veliovgroup/ostrio/HEAD/wallpapers/ostrio-logo-iphone6_7_8plus-1242x2208.jpg
--------------------------------------------------------------------------------
/wallpapers/ostrio-logo-iphone6_7_8plus-1242x2208.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/veliovgroup/ostrio/HEAD/wallpapers/ostrio-logo-iphone6_7_8plus-1242x2208.png
--------------------------------------------------------------------------------
/docs/prerendering/prerendering-cache-purge-single.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/veliovgroup/ostrio/HEAD/docs/prerendering/prerendering-cache-purge-single.png
--------------------------------------------------------------------------------
/docs/prerendering/prerendering-genuine-status-code.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/veliovgroup/ostrio/HEAD/docs/prerendering/prerendering-genuine-status-code.png
--------------------------------------------------------------------------------
/tutorials/nginx/ssl.conf:
--------------------------------------------------------------------------------
1 | # Adding STS security HTTP response header is optional, but recommended
2 | # do your own research before enabling this header, as it's easy to make
3 | # websites unaccessible once wrong header is received by a browser
4 | # add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload";
5 | ssl on;
--------------------------------------------------------------------------------
/docs/account/referral-program.md:
--------------------------------------------------------------------------------
1 | # Referral Program
2 |
3 | __Give $50. Get $50.__ Share a personal referral link. Each invited account receives $50 off the second purchase; the referrer earns $50 after the invitee's first purchase.
4 |
5 | Go to [your account](https://ostr.io/account#ref) to obtain personalized invitation link and learn more.
6 |
--------------------------------------------------------------------------------
/docs/monitoring/custom-path.md:
--------------------------------------------------------------------------------
1 | # Custom Path
2 |
3 | To avoid caching on `http(s)` endpoints - use `{{rand}}` placeholder.
4 |
5 | Example: `/path/?noCahe={{rand}}`.
6 |
7 | Where `{{rand}}` will be replaced with random *String*.
8 |
9 | 
10 |
--------------------------------------------------------------------------------
/tutorials/linux/change-timezone.md:
--------------------------------------------------------------------------------
1 | Permanently change server timezone
2 | ======
3 |
4 | Often you may want to set your server(s) to some exact time zone, and exclude cases of time shift when daylight saving time, winter time, summer time, etc. occurs.
5 |
6 | We suggest always to use UTC time zone (`+0:00`)
7 |
8 | ```shell
9 | dpkg-reconfigure tzdata
10 | # in the list select "etc" or "none of the above"
11 | # than select "UTC"
12 | ```
13 |
--------------------------------------------------------------------------------
/docs/account/password-recovery.md:
--------------------------------------------------------------------------------
1 | Password Recovery
2 | ======
3 |
4 | - [Direct link to Password Recovery](https://ostr.io/account/forgot)
5 |
6 | We have made password recovery process as simple as possible, you need to fill only single field - "Email Address". We will send you recovery link shortly.
7 |
8 | If you do not remember your email address, contact us directly via email (*our email address can be obtained in a footer of [ostr.io](https://ostr.io)*).
--------------------------------------------------------------------------------
/tutorials/linux/users/user-passwd.md:
--------------------------------------------------------------------------------
1 | # Change user password
2 |
3 | __NOTE:__ change `[username]` to actual users' name
4 |
5 | As `root` or `sudo` user (*to other user*):
6 |
7 | ```shell
8 | passwd [username]
9 |
10 | # this command will ask to
11 | # prompt new password twice
12 | ```
13 |
14 | To change password of a current user:
15 |
16 | ```shell
17 | passwd
18 |
19 | # this command will ask to
20 | # prompt new password twice
21 | ```
22 |
--------------------------------------------------------------------------------
/docs/SUPPORT.md:
--------------------------------------------------------------------------------
1 | ### General User Support
2 |
3 | - [Fill out an issue](https://github.com/veliovgroup/ostrio/issues) - Quickly solve your issue with the power of our customer support and open source community
4 | - __Pro__ Plan Support - Contact us directly via email (*found at the bottom of every page or in the sidebar menu*)
5 | - __Business__ Plan Support - Contact us directly via email or messenger of your choice (*found at the bottom of every page or in the sidebar menu*)
6 |
--------------------------------------------------------------------------------
/docs/account/password-change.md:
--------------------------------------------------------------------------------
1 | # Change Password
2 |
3 | - [Direct link to password change](https://ostr.io/account/pwd)
4 |
5 | Whenever you decide to change your current password, for example right after registration process where the system generates default password for your account, go to ["Change Password" panel](https://ostr.io/account/pwd) ("My Account" -> "Change Password"), enter your current (*old*) password, new password, and prompt new password once again.
6 |
7 | 
8 |
--------------------------------------------------------------------------------
/docs/prerendering/strip-javascript.md:
--------------------------------------------------------------------------------
1 | Prerendering: Strip JavaScript
2 | ======
3 |
4 | By default, after the page is rendered - all JavaScript blocks and file references will be removed from the final result. This is done to avoid unpredictable behavior and results on the web crawler end. We have seen some weird situations with content being rendered multiple times. Although it's recommended to keep this option ON, you're free to turn it off.
5 |
6 | 
--------------------------------------------------------------------------------
/docs/prerendering/es6-support.md:
--------------------------------------------------------------------------------
1 | # Enable ES6 support for Prerendering
2 |
3 | Love bleeding edge technology? Using ES6 and ES7 at your website? Great! We are happy to introduce ES6 and partial ES7 syntax support. This feature is not enabled by default and can be turned on individually for each host in "Prerendering" panel.
4 |
5 | It the same as default Prerendering, but uses more modern JavaScript engine, which may be less stable.
6 |
7 | ## Enable ECMAScript 2015 in one click:
8 |
9 | 
10 |
--------------------------------------------------------------------------------
/docs/monitoring/with-auth.md:
--------------------------------------------------------------------------------
1 | # Protected Endpoints Monitoring
2 |
3 | Every monitoring endpoint may be secured with basic Web Authentication (username and password pair).
4 |
5 | Links:
6 |
7 | - Read how to [generate `.htpasswd` file](https://github.com/veliovgroup/ostrio/blob/master/docs/monitoring/nginx-stats.md#first-create-htpasswd-file)
8 | - [Custom data Monitoring](https://github.com/veliovgroup/ostrio/blob/master/docs/monitoring/basics.md#custom-data-monitoring)
9 |
10 | 
11 |
--------------------------------------------------------------------------------
/docs/account/sign-up.md:
--------------------------------------------------------------------------------
1 | # Sign Up
2 |
3 | We have made the registration process as simple as possible, you need to fill the only single field - "Email Address". Where we will send a verification email.
4 |
5 | Every new account comes with an initial ["free tier"](https://ostr.io/info/pricing), amount of free goodies depends from your billing plan. "Free tier" is made to let you test all ostr.io services and features for free.
6 |
7 | ## Sign Up Now
8 |
9 | Use [this registration link](https://ostr.io/signup) to join ostr.io now.
10 |
11 | ## $50 off your second purchase
12 |
13 | Sign up using [this special link](https://ostr.io/signup/gCZWjiBScePWrnnDr) to get $50 off your second purchase.
14 |
--------------------------------------------------------------------------------
/docs/web-cron/README.md:
--------------------------------------------------------------------------------
1 | # Web CRON: Serverless automation
2 |
3 | Efficient task automation for serverless platforms and hosting providers without native CRON support. Execute scripts reliably across any infrastructure with real-time status tracking and automated failure alerts for full operational transparency.
4 |
5 | > [!TIP]
6 | > Use `{{rand}}` placeholder (*including curly brackets*) in endpoint URL to generate unique random URI for every CRON request. Note: `{{rand}}` can be used multiple times in a single URL, example: `/path/{{rand}}/part?nc={{rand}}`
7 |
8 | - [Learn more about Web-CRON](https://ostr.io/info/web-cron)
9 |
10 |
13 |
--------------------------------------------------------------------------------
/docs/account/change-primary-email.md:
--------------------------------------------------------------------------------
1 | # Change Primary Email Address
2 |
3 | - [Direct link to Contacts panel](https://ostr.io/account/contacts)
4 |
5 | "Primary Email" is the email you are using to login into account. To change "Primary Email" address you need to have at least one more verified email address in the [contacts panel](https://ostr.io/account/contacts).
6 |
7 | Once another email address is added and verified, click on the *star* icon next to it. After you confirm this action you will be logged out, and since then you have to use new "Primary Email" to login into your account.
8 |
9 | 
10 |
--------------------------------------------------------------------------------
/docs/account/add-notification-email.md:
--------------------------------------------------------------------------------
1 | # Add an Email Address for Notifications
2 |
3 | - [Direct link to Contacts panel](https://ostr.io/account/contacts)
4 |
5 | It is a good idea to separate email notifications across teams, team members, or to have two or more notification email addresses for critical events. Go to [contacts panel](https://ostr.io/account/contacts) ("Account" -> "Manage Contacts" -> "Add an Email Address"), enter the email address you wish to add. Then click "Verify" link to send the verification email with a link. Once you visit a link from sent email, the new email address can be used for notifications
6 |
7 | 
8 |
--------------------------------------------------------------------------------
/docs/prerendering/amp-support.md:
--------------------------------------------------------------------------------
1 | # AMP Support
2 |
3 | To properly serve pages for [Accelerated Mobile Pages](https://www.ampproject.org) (AMP) we support following URI schemes:
4 |
5 | ```shell
6 | # Regular URIs (not optimized for AMP):
7 | https://example.com/index.html
8 | https://example.com/articles/article-title.html
9 | https://example.com/articles/article-uniq-id/article-slug
10 |
11 | # AMP optimized URIs (prefix):
12 | https://example.com/amp/index.html
13 | https://example.com/amp/articles/article-title.html
14 | https://example.com/amp/articles/article-uniq-id/article-slug
15 |
16 | # AMP optimized URIs (extension):
17 | https://example.com/amp/index.amp.html
18 | https://example.com/amp/articles/article-title.amp.html
19 | ```
20 |
21 | All URLs with `.amp.` extension and `/amp/` prefix will be optimized for AMP.
22 |
--------------------------------------------------------------------------------
/docs/prerendering/prerendering-custom-status-codes.md:
--------------------------------------------------------------------------------
1 | # Custom status codes
2 |
3 | The "status codes" chart in pre-rendering panel displays origin response codes from rendered pages. In addition in may display custom `7**` codes that intended to help in debugging and improving pre-rendering integration, here's custom codes meanings:
4 |
5 | - `701` and `702` — Credentials error
6 | - `703` — Insufficient credits; Origin page is returned bypassing pre-rendering
7 | - `704` — Pre-rendering requesting features not available in current plan/tariff (e.g. subdomains, load-images, etc)
8 | - `705` — Unexpected error
9 | - `707` — Too many errors on the original page (unrecoverable JS or HTML errors); Origin page is returned bypassing pre-rendering
10 | - `708` — Origin takes too long to respond 90+ seconds; Origin page is returned bypassing pre-rendering
11 |
--------------------------------------------------------------------------------
/docs/prerendering/cache.md:
--------------------------------------------------------------------------------
1 | # Prerendering Caching
2 |
3 | All Prerendering engine results are cached by default. Cache TTL can be changed in the Prerendering settings, individually per each host.
4 |
5 | ## Why cache results?
6 |
7 | Caching allows to offload servers, save bandwidth and speed up sequential requests to the same pages by web crawlers, mean better SEO score for your website.
8 |
9 | We do not charge for requests to the cached results, mean cheaper Prerendering service for you.
10 |
11 | ## Cache TTL
12 |
13 | Cache TTL can be set per each host individually starting from 2 hours and ending with 744 hours (31 days).
14 |
15 | 
16 |
17 | ## Cache purging
18 |
19 | The cache may be fully purged once per 2 hours, but the specific page can be purged individually at any moment. Read more in [cache purging docs](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/cache-purge.md).
20 |
--------------------------------------------------------------------------------
/docs/prerendering/cloudflare-worker.md:
--------------------------------------------------------------------------------
1 | # CloudFlare Worker integration
2 |
3 |
4 |
5 | CloudFlare Workers is the great alternative to server-specific proxy (Nginx, Apache, HAProxy) and app-specific middleware (NPM, PHP, Node.js, Django, and other)
6 |
7 | - [Step-by-step Tutorial](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/examples/cloudflare-worker/coudflare-worker-guide.md)
8 | - **Shopify** [CloudFlare Worker Integration](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/shopify-seo-integration.md#seo-middleware-worker-for-shopify)
9 | - *Modern and recommended*: [ES Module Worker](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/examples/cloudflare-worker/cloudflare.worker.js)
10 | - *Legacy*: [Event Listener Worker](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/examples/cloudflare-worker/cloudflare-listener.worker.js)
11 |
--------------------------------------------------------------------------------
/docs/account/add-notification-phone-number.md:
--------------------------------------------------------------------------------
1 | # Add a Phone Number for Notifications
2 |
3 | - [Direct link to Contacts panel](https://ostr.io/account/contacts)
4 |
5 | Add a mobile phone number to receive SMS notification. Go to [contacts panel](https://ostr.io/account/contacts) ("Account" -> "Manage Contacts" -> "Add a Phone Number"), enter a phone number in [international format](https://en.wikipedia.org/wiki/E.164) you wish to add.
6 |
7 | 
8 |
9 | To verify phone number two options are available:
10 |
11 | - __Short Code__ - Short digit code will be sent to the phone number, you will have to prompt it in Accounts panel. Good for simple phones, of if the phone is physically accessible
12 | - __Link__ - A verification link will be sent to a phone number. Good for smart-phones or if you add phone number of another person.
13 |
14 | 
15 |
--------------------------------------------------------------------------------
/docs/analytics/README.md:
--------------------------------------------------------------------------------
1 | # Web Analytics
2 |
3 | Lightweight, privacy-focused web analytics with real-time metrics and built-in GDPR/CCPA compliance. Offers a simple opt-out for end users and runs by 100% open-source tracking code!
4 |
5 | Web Analytics is a fully-featured and drop-in web-tracking solution with no configuration hassle. Our solution is ready out-of-the-box and fully compatible with all modern and legacy websites.
6 |
7 | ## How to's
8 |
9 | - [Detect AdBlock usage](https://github.com/veliovgroup/ostrio/blob/master/docs/analytics/detect-adblock.md)
10 | - [Track Accelerated Mobile Pages (AMP)](https://github.com/veliovgroup/ostrio/blob/master/docs/analytics/track-amp.md)
11 |
12 | > [!TIP]
13 | > Enable "Development Mode" to test web analytics integration and data-reports right from `localhost`
14 |
15 | ## Further reading
16 |
17 | - 📦 [Read Open Source Library Docs](https://github.com/veliovgroup/ostrio-analytics)
18 | - 📦 [NPM `ostrio-analytics` JavaScript package](https://www.npmjs.com/package/ostrio-analytics)
19 | - 📦 [Atmosphere `ostrio:analytics` Meteor.js package](https://atmospherejs.com/ostrio/analytics)
20 | - 📔 [Learn more about Web-Analytics](https://ostr.io/info/web-analytics)
21 |
--------------------------------------------------------------------------------
/tutorials/linux/security/lynis-security-audit.md:
--------------------------------------------------------------------------------
1 | # Security audit with Lynis
2 |
3 | Lynis is an open source security auditing tool. Lynis is light-weight and easy to use.
4 |
5 | ## References:
6 |
7 | - [Official project page](https://cisofy.com/lynis/)
8 | - [GitHub project page](https://github.com/CISOfy/Lynis)
9 |
10 | ## Installation
11 |
12 | Although Lynis is available in most of Linux package managers - its installation is not required, as it written in shell script.
13 |
14 | Download script (clone from github):
15 |
16 | ```shell
17 | git clone https://github.com/CISOfy/lynis
18 | ```
19 |
20 | Go to cloned directory:
21 |
22 | ```shell
23 | cd lynis/
24 | ```
25 |
26 | ## Run script:
27 |
28 | ```shell
29 | ./lynis audit system
30 | ```
31 |
32 | ### To run security audit against docker, run:
33 |
34 | ```shell
35 | lynis audit dockerfile
36 | ```
37 |
38 | ## Further reading:
39 |
40 | - [Use RKHunter (Rootkit Hunter) to scan OS for vulnerabilities](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/security/rootkit-hunter.md)
41 | - [Basic firewall with `iptables`](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/security/iptables-firewall.md)
42 |
--------------------------------------------------------------------------------
/docs/websec/README.md:
--------------------------------------------------------------------------------
1 | # WebSec: Domains Protection
2 |
3 | Be aware of data substitution, traffic redirection, theft, and hijacking of a domain name, DNS, Whois and SSL/TLS certificate. Nowadays protection and security are number one priority for IT industry.
4 |
5 | Protect domains from hijacking and unauthorized changes — __no setup or installation required__. Get real-time alerts on any modifications to DNS records, Whois data, or SSL/TLS certificates. Full domain security in a single click
6 |
7 | Whenever you are involved in IT as a hobbyist or as a business of any size, your data and data of your visitors must be well protected. __And we made it easy__, no setup is required our famous "*One Click Setup*" will configure everything for you in a matter of seconds
8 |
9 | ## Available WebSec Services
10 |
11 | - **DNS** — *Substitution monitoring and alerts for all or specific types of DNS records* `A`, `AAAA`, `NS`, `TXT`, `SOA`, `CNAME`, `MX`
12 | - **SSL/TLS Certificate** — *Substitution alerts and expiration notifications*
13 | - **Whois Record** — *Substitution alerts and domain name expiration notifications*
14 |
15 | ## Further reading
16 |
17 | - [Learn more about __free__ Domains Protection](https://ostr.io/info/domain-names-protection)
18 |
--------------------------------------------------------------------------------
/tutorials/linux/users/root-passwd.md:
--------------------------------------------------------------------------------
1 | # Change root password
2 |
3 | When it comes to purchasing virtual or dedicated Linux-based servers - its credentials is sent somehow over the wire (http(s), email, sms, push-notification and etc.). Any data sent over the wire - can not be trusted.
4 |
5 | So after the first login to your new and shiny server - __change default passwords__.
6 |
7 | Let's change `root` (the very powerful user) password:
8 |
9 | ```shell
10 | # make sure we are root
11 | whoami
12 |
13 | # if output is: root
14 | # run passwd command
15 | passwd
16 |
17 | # after starting passwd command
18 | # it will ask to prompt new password twice
19 | # yes, NOTHING WOULD GET TYPED it's fine!
20 | # symbols just hidden as it's password
21 | ```
22 |
23 | After new password is set logout from server with `exit` command. And login again with new password.
24 |
25 | ## Further reading:
26 |
27 | - [Change default SSH port](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/security/change-ssh-port.md)
28 | - [Disable root login via SSH](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/security/disable-ssh-root.md)
29 | - [Use SSH keys to login](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/security/use-ssh-keys.md)
30 |
--------------------------------------------------------------------------------
/tutorials/linux/security/change-ssh-port.md:
--------------------------------------------------------------------------------
1 | # Change default SSH port
2 |
3 | The best practice of the "*best practices*" 😅 is to change default SSH port. As the Internet full of automated scanners and bots — most vulnerable are widely used default applications and ports.
4 |
5 | Edit `/etc/ssh/sshd_config` file, — find and update value of the line staring with `Port` as a new value for port pick a number between `1024` and `65535`:
6 |
7 | ```shell
8 | # nano /etc/ssh/sshd_config
9 |
10 | # Find line starting with "Port":
11 | Port 64537 # CHANGE TO PORT OF YOUR CHOICE 1024-65535
12 |
13 | # Save file with: "ctrl + o" combination
14 | # Close file with: "ctrl + x" combination
15 | ```
16 |
17 | Restart SSH service:
18 |
19 | ```shell
20 | service sshd restart
21 | ```
22 |
23 | To connect again via SSH explicitly set port via `-p` flag:
24 |
25 | ```shell
26 | ssh -p 64537 username@host
27 | ```
28 |
29 | ## Further reading:
30 |
31 | - [Disable root login via SSH](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/security/disable-ssh-root.md)
32 | - [Use SSH keys to login](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/security/use-ssh-keys.md)
33 | - [Sample `sshd_config`](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/security/sshd_config)
34 |
--------------------------------------------------------------------------------
/tutorials/linux/security/sshd_config:
--------------------------------------------------------------------------------
1 | # /etc/ssh/sshd_config
2 |
3 | Port 22 # CHANGE TO RANDOM PORT 1024-65535
4 |
5 | AcceptEnv LANG LC_*
6 | AllowAgentForwarding no
7 | AllowTcpForwarding no
8 | # AllowUsers [username] # Enable to allow login only for a singe "special" user
9 | Banner /etc/issue.net # Make sure this file exists
10 | ChallengeResponseAuthentication no
11 | ClientAliveCountMax 2
12 | Compression no
13 | HostbasedAuthentication no
14 | HostKey /etc/ssh/ssh_host_dsa_key
15 | HostKey /etc/ssh/ssh_host_ecdsa_key
16 | HostKey /etc/ssh/ssh_host_ed25519_key
17 | HostKey /etc/ssh/ssh_host_rsa_key
18 | IgnoreRhosts yes
19 | LoginGraceTime 15
20 | LogLevel VERBOSE
21 | MaxAuthTries 2
22 | MaxSessions 2
23 | PasswordAuthentication yes # change to no to disable login with paswords
24 | PermitEmptyPasswords no
25 | PermitRootLogin yes # no|yes|without-password recommended: no and without-password to allow login for root using SSH-key
26 | PrintLastLog yes
27 | PrintMotd no
28 | Protocol 2
29 | PubkeyAuthentication no # change to yes to enable SSH-key based authentication
30 | StrictModes yes
31 | SyslogFacility AUTH
32 | TCPKeepAlive no
33 | ClientAliveInterval 5
34 | ClientAliveCountMax 2160
35 | UseDNS no
36 | UsePAM yes
37 | UsePrivilegeSeparation sandbox
38 | X11DisplayOffset 10
39 | X11Forwarding no
40 | DisableForwarding yes
41 |
--------------------------------------------------------------------------------
/tutorials/nginx/hacking-nginx-with-default.md:
--------------------------------------------------------------------------------
1 | # Hacking Nginx
2 |
3 | Here's a little-known tunning option via setting `server_name` to `_`
4 |
5 | ```nginx
6 | server {
7 | listen 80 default_server fastopen=256 deferred reuseport so_keepalive=on;
8 | listen [::]:80 default_server fastopen=256 deferred reuseport so_keepalive=on ipv6only=on;
9 | server_name _;
10 |
11 | # [optionally] include other configuration files here
12 | # to extend default server functionality with for example
13 | # best practices security check for access and permissions
14 | # include /path/to/file.conf
15 | return 444;
16 | }
17 |
18 | # Enable for SSL/TLS/HTTPS enabled websites
19 | #
20 | # server {
21 | # listen 443 default_server ssl http2 fastopen=256 deferred reuseport so_keepalive=on;
22 | # listen [::]:443 default_server ssl http2 fastopen=256 deferred reuseport so_keepalive=on ipv6only=on;
23 | # server_name _;
24 |
25 | # # [optionally] include other configuration files here
26 | # # to extend default server functionality with for example
27 | # # best practices security check for access and permissions
28 | # # include /path/to/file.conf
29 | # #
30 | # # In SSL/HTTPS server a TLS certificate definition
31 | # # should be included here:
32 | # include /etc/nginx/ssl.conf
33 |
34 | # return 444;
35 | # }
36 | ```
37 |
--------------------------------------------------------------------------------
/docs/prerendering/meteor-atmosphere.md:
--------------------------------------------------------------------------------
1 | # Meteor.js Integration via Atmosphere
2 |
3 | - Learn more - [what is Prerendering and why you need it](https://ostr.io/info/prerendering)
4 | - For more info see [`ostrio:spiderable-middleware` package on Atmosphere](https://atmospherejs.com/ostrio/spiderable-middleware).
5 |
6 | ## Installation
7 |
8 | ```shell
9 | meteor add webapp
10 | meteor add ostrio:spiderable-middleware
11 | ```
12 |
13 | ## Update HTML Markup
14 |
15 | To cause the special behavior of web crawlers on JavaScript powered websites use `fragment` meta tag. Although it's officially deprecated by Google search engine, it's may be used by other search engines and web crawlers. [Learn more](https://developers.google.com/webmasters/ajax-crawling/docs/specification):
16 |
17 | ```html
18 |
19 |
20 |
21 |
22 |
23 |
24 |
25 |
26 |
27 | ```
28 |
29 | ## Middleware integration
30 |
31 | [See more examples here](https://github.com/veliovgroup/spiderable-middleware/tree/master/examples).
32 |
33 | ```js
34 | import Spiderable from 'meteor/ostrio:spiderable-middleware';
35 |
36 | WebApp.connectHandlers.use(new Spiderable({
37 | rootURL: 'http://example.com',
38 | serviceURL: 'https://render.ostr.io',
39 | auth: 'APIUser:APIPass'
40 | }));
41 | ```
42 |
--------------------------------------------------------------------------------
/docs/prerendering/genuine-status-code.md:
--------------------------------------------------------------------------------
1 | # Return Genuine Status Code
2 |
3 | - For more detailed documentation see [`spiderable-middleware` package repository](https://github.com/veliovgroup/spiderable-middleware#return-genuine-status-code)
4 |
5 | Returning genuine status code is very important for indexing by search engines. Most of the front-end JavaScript frameworks returns the same page to every request with `200 OK` status code, mean `404` pages rendered in a browser is returned with `200`status to the web crawlers.
6 |
7 | To return expected response code — use this HTML comment. This comment __can be placed in any part of HTML-page__. `head` or `body` tag is the best place for it.
8 |
9 | ## Format:
10 |
11 | __html:__
12 |
13 | ```html
14 |
15 | ```
16 |
17 | __jade:__
18 |
19 | ```jade
20 | // response:status-code=404
21 | ```
22 |
23 | Use __any__ standard or custom status codes:
24 |
25 | - `201` - ``
26 | - `401` - ``
27 | - `403` - ``
28 | - `500` - ``
29 | - `514` - `` (*non-standard*)
30 |
31 | ## Result:
32 |
33 | 
34 |
--------------------------------------------------------------------------------
/docs/prerendering/nextjs-prerendering.md:
--------------------------------------------------------------------------------
1 | # NEXT.JS
2 |
3 | [ostr.io pre-rendering](https://ostr.io/info/prerendering) integration with Next.js apps
4 |
5 |
6 |
7 | ## Integration Options
8 |
9 | - 👨💻 [Manually Creating `middleware.ts`](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/examples/next.js/middleware.ts) in root of the project or `/src/` directory
10 | - 📦 [Using `seo-middleware-nextjs` NPM package](https://github.com/veliovgroup/seo-middleware-nextjs?tab=readme-ov-file#seo-middleware-for-nextjs)
11 | - 👷♂️ [Via Nginx](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/nginx.md) without changes in codebase
12 | - 🌤️ [Via CloudFlare](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/cloudflare-worker.md) without changes in codebase nor server environment
13 | - ✨ [Via Netlify](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/netlify-prerendering.md) enable in the settings without changes in codebase nor server environment
14 | - ▲ Via Vercel (*coming soon*)
15 | - 🥞 Via Supabase (*coming soon*)
16 |
17 | ## References
18 |
19 | - [Official Middleware Next.JS docs](https://nextjs.org/docs/app/getting-started/route-handlers-and-middleware#middleware)
20 |
--------------------------------------------------------------------------------
/docs/prerendering/detect-prerendering-meteor.md:
--------------------------------------------------------------------------------
1 | # Detect request from Prerendering engine in Meteor.js
2 |
3 | When requests are coming from pre-rendering engine it can be detected in JS-runtime (e.g. front-end code)
4 |
5 | ## Detect that request is coming from pre-rendering engine
6 |
7 | Pre-rendering engine will set `window.IS_PRERENDERING` global variable to `true`. As in Meteor everything should be reactive, let's bound it with `ReactiveVar`:
8 |
9 | ```js
10 | const IS_PRERENDERING = new ReactiveVar(window.IS_PRERENDERING || false);
11 | Object.defineProperty(window, 'IS_PRERENDERING', {
12 | set(val) {
13 | IS_PRERENDERING.set(val);
14 | },
15 | get() {
16 | return IS_PRERENDERING.get();
17 | }
18 | });
19 |
20 | // Make globally available Blaze helper,
21 | // Feel free to omit this line in case if you're not using Blaze
22 | // or going to handle logic in JavaScript part
23 | Template.registerHelper('IS_PRERENDERING', () => IS_PRERENDERING.get());
24 | ```
25 |
26 | __Note__: `window.IS_PRERENDERING` might be `undefined` on initial page load, and may change during runtime.
27 |
28 | ## Detect type of the pre-rendering engine
29 |
30 | Like browsers, crawler, and bots come as "mobile" (small screen touch-devices) or as "desktop" (large screens without touch-events) the pre-rendering engine has the same two types. For cases when content needs to get optimized for different screens pre-rendering engine will set `window.IS_PRERENDERING_TYPE` global variable to `desktop` or `mobile`
31 |
--------------------------------------------------------------------------------
/docs/prerendering/rendering-endpoints.md:
--------------------------------------------------------------------------------
1 | Rendering Endpoints
2 | ======
3 |
4 | - __render (*default*)__ - `https://render.ostr.io` - This endpoint has "optimal" settings, and should fit 98% cases. Respects cache headers of both Crawler and your server;
5 | - __render-bypass (*devel/debug*)__ - `https://render-bypass.ostr.io` - This endpoint has bypass caching mechanism (*almost no cache*). Use it if you're experiencing an issue, or during development, to make sure you're not running into the intermediate cache. You're safe to use this endpoint in production, but it may result in higher usage and response time.
6 | - __render-cache (*under attack*)__ - `https://render-cache.ostr.io` - This endpoint has the most aggressive caching mechanism. Use it if you're looking for the shortest response time, and don't really care about outdated pages in cache for 6-12 hours
7 |
8 | To change default endpoint, grab [integration examples code](https://github.com/veliovgroup/spiderable-middleware/tree/master/examples) and replace `render.ostr.io`, with endpoint of your choice. For NPM/Meteor integration change value of [`serviceURL`](https://github.com/veliovgroup/spiderable-middleware#basic-usage) option.
9 |
10 | __Note:__ Described differences in caching behavior is only about intermediate proxy caching, `Cache-Control` header will be always set to the value defined in "Cache TTL". Cached results at the "Prerendering Engine" end can be [purged at any time](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/cache-purge.md).
11 |
--------------------------------------------------------------------------------
/docs/account/README.md:
--------------------------------------------------------------------------------
1 | # Account Basics ToC:
2 |
3 | Learn how to manage your account, contact, and other ostr.io features
4 |
5 | ## ostr.io UI and account features
6 |
7 | General account features
8 |
9 | - [Registration](https://github.com/veliovgroup/ostrio/blob/master/docs/account/sign-up.md)
10 | - [Password Recovery](https://github.com/veliovgroup/ostrio/blob/master/docs/account/password-recovery.md)
11 | - [Password Change](https://github.com/veliovgroup/ostrio/blob/master/docs/account/password-change.md)
12 | - [Change login (primary email)](https://github.com/veliovgroup/ostrio/blob/master/docs/account/change-primary-email.md)
13 | - [Add extra email(s) for notifications](https://github.com/veliovgroup/ostrio/blob/master/docs/account/add-notification-email.md)
14 | - [Add phone number(s) for SMS notifications](https://github.com/veliovgroup/ostrio/blob/master/docs/account/add-notification-phone-number.md)
15 | - [Account Removal](https://github.com/veliovgroup/ostrio/blob/master/docs/account/account-removal.md)
16 |
17 | ## Additional account features
18 |
19 | - [Tips'n'Tricks](https://github.com/veliovgroup/ostrio/blob/master/docs/account/tips-and-tricks.md)
20 | - [Referral Program](https://github.com/veliovgroup/ostrio/blob/master/docs/account/referral-program.md)
21 |
22 |
--------------------------------------------------------------------------------
/LICENSE:
--------------------------------------------------------------------------------
1 | The ostr.io Documentation and Tutorials
2 | Copyright (c) 2025, Veliov Group, LLC
3 | All rights reserved.
4 |
5 | Redistribution and use in source and binary forms,
6 | with or without modification, are permitted provided
7 | that the following conditions are met:
8 |
9 | 1. Redistributions of source code must retain the
10 | above copyright notice, this list of conditions
11 | and the following disclaimer.
12 |
13 | 2. Redistributions in binary form must reproduce the
14 | above copyright notice, this list of conditions and
15 | the following disclaimer in the documentation and/or
16 | other materials provided with the distribution.
17 |
18 | 3. Neither the name of the copyright holder nor the
19 | names of its contributors may be used to endorse or
20 | promote products derived from this software without
21 | specific prior written permission.
22 |
23 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
24 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
25 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
26 | ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
27 | LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
28 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
29 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
30 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
31 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
32 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
33 |
--------------------------------------------------------------------------------
/docs/account/account-removal.md:
--------------------------------------------------------------------------------
1 | # Permanent account removal
2 |
3 | - [Direct link to account panel](https://ostr.io/account)
4 |
5 | __Important notice__: All account data, all personal data, __including all items (emails, messages, payments) in communication and billing history__. All collected data for all services including servers, domains, hostnames, email addresses, phone numbers, SMS history, notifications, email history, purchase history, billing history, communication emails, cart items, pending and sent emails. All services will be terminated immediately, including — newsletter, pre-rendering, web-CRON, web-analytics, monitoring, and domain protection.
6 |
7 | > As per our [GDPR](https://ostr.io/info/gdpr#therighttobeforgotten) and our beliefs in digital freedom — you have the right to be [permanently forgotten](https://ostr.io/info/gdpr#therighttobeforgotten).
8 |
9 | Steps:
10 |
11 | 1. Login to your account [here](https://ostr.io/login)
12 | 2. Go to "[Account Settings](https://ostr.io/account)"
13 | 3. Click on "Remove Account" (*red link at the middle of a page*)
14 | 4. Click on Remove my account and erase all data button
15 | 5. Read two messages and confirm you understand the removal process
16 | 6. Enter you login to confirm and start removal process
17 | 7. Wait patiently. __Do not reload or close the page!__
18 | 8. You will be logged out from account upon successful account removal
19 | 9. You may or *may not\** receive confirmatory email about successful account removal
20 |
21 | \*Email may not be delivered due to invalid or incorrect email address provided upon registration. Or due to technical issues and service interruptions on intermediate and primary email servers.
--------------------------------------------------------------------------------
/docs/monitoring/basics.md:
--------------------------------------------------------------------------------
1 | # Monitoring Basics
2 |
3 | ## ToC:
4 |
5 | - [Availability Monitoring](https://github.com/veliovgroup/ostrio/blob/master/docs/monitoring/basics.md#availability-httphttps-monitoring)
6 | - [Custom Data Monitoring](https://github.com/veliovgroup/ostrio/blob/master/docs/monitoring/basics.md#custom-data-monitoring)
7 | - [SNMP (Health) Monitoring](https://github.com/veliovgroup/ostrio/blob/master/docs/monitoring/basics.md#snmp-monitoring)
8 |
9 | ## Availability HTTP/HTTPS Monitoring
10 |
11 | This is the most basic type of monitoring. Only most basic data is collected and analyzed - "Response Time", "Status Code", "Uptime".
12 |
13 | 
14 |
15 | ## Custom Data Monitoring
16 |
17 | Return custom data from endpoint as JSON or XML to parse it into charts with notifications triggers.
18 |
19 | ### As JSON:
20 |
21 | ```json
22 | {
23 | "active": 98,
24 | "reading": 0,
25 | "writing": 55,
26 | "waiting":43
27 | }
28 | ```
29 |
30 | ### As XML:
31 |
32 | ```xml
33 |
34 | 98
35 | 0
36 | 55
37 | 43
38 |
39 | ```
40 |
41 | #### Result:
42 |
43 | 
44 |
45 | ## SNMP Monitoring
46 |
47 | SNMP monitoring also known as "Health Monitoring", where a lot of data is collected, including but not limited to: "SSH Sessions", "CPU Consumption", "Storage Space Utilization", "Network Consumption", "RAM Usage", "System Load", "Uptime" and etc. [Read how to configure and enable SNMP monitoring](https://github.com/veliovgroup/ostrio/blob/master/docs/monitoring/snmp.md).
48 |
--------------------------------------------------------------------------------
/tutorials/linux/security/disable-ssh-root.md:
--------------------------------------------------------------------------------
1 | # Disable SSH login for root user
2 |
3 | ## Why?
4 |
5 | - Most of the automatic brute-force software starts with attacking SSH service on `22` port and try to match a password for the `root` user
6 | - `root` user is not the only user under risk, brute-force software has abundant dictionaries of most common used user-names, so it's a good idea to [create a user](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/users/create-user.md) with some random-gibberish name, like: `QBHZEEQXT`
7 |
8 | ## Before you go:
9 |
10 | - [Create Linux user](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/users/create-user.md)
11 | - [Create restricted Linux user](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/users/rbash.md)
12 |
13 | Edit `/etc/ssh/sshd_config` file:
14 |
15 | ```shell
16 | # nano /etc/ssh/sshd_config
17 |
18 | # Find line starting with "PermitRootLogin"
19 | # change "yes" to "no":
20 | PermitRootLogin no
21 |
22 | # Save file with: "ctrl + o" combination
23 | # Close file with: "ctrl + x" combination
24 | ```
25 |
26 | Add extra security layer, with limiting login only to a single user (*optional*):
27 |
28 | ```shell
29 | # nano /etc/ssh/sshd_config
30 |
31 | # Add new line, where [username]
32 | # is the name of ONLY allowed to
33 | # login over SSH user:
34 | AllowUsers [username]
35 |
36 | # Save file with: "ctrl + o" combination
37 | # Close file with: "ctrl + x" combination
38 | ```
39 |
40 | Restart SSH service:
41 |
42 | ```shell
43 | service sshd restart
44 | ```
45 |
46 | ## Further reading:
47 |
48 | - [Change Default SSH port](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/security/change-ssh-port.md)
49 | - [Use SSH keys to login](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/security/use-ssh-keys.md)
50 |
--------------------------------------------------------------------------------
/tutorials/linux/users/create-user.md:
--------------------------------------------------------------------------------
1 | # Create new system user
2 |
3 | ## Why create another system users in Linux?
4 |
5 | - Web-applications should run under its own users, to isolate system files in case of the breach
6 | - Databases should run under its own user
7 | - SSH login should be allowed to the user with limited permissions on the server and disabled for `root` user
8 | - Other security and usage reasons
9 |
10 | ## Two common Linux users types:
11 |
12 | - With login permissions (password and home folder are not set)
13 | - Without login permissions (*services and applications*)
14 |
15 | __NOTE:__ change `[username]` to actual users' name
16 |
17 | ### Password-less user
18 |
19 | Let's start with login-less (*or password-less*) users (most secure, best for running isolated web-apps):
20 |
21 | ```shell
22 | useradd [username]
23 |
24 | # that's it
25 | ```
26 |
27 | To allow login as `user` without password, just set new password for such user:
28 |
29 | ```shell
30 | passwd [username]
31 |
32 | # this command will ask
33 | # to prompt new password twice
34 | ```
35 |
36 | To add a home folder for such limited user, run:
37 |
38 | ```shell
39 | # Create folder
40 | mkdir -p /home/[username]
41 | # Set home folder
42 | usermod -m -d /home/[username] [username]
43 | ```
44 |
45 | ### Full-featured user
46 |
47 | Fully capable user (with password, home folder and SSH access) can be created with single command:
48 |
49 | ```shell
50 | adduser [username]
51 |
52 | # this command will ask
53 | # to fill additional info about user
54 | # it can be safely skipped with hitting ENTER
55 | # But do not skip password, and prompt it twice
56 | # Finally, to confirm provided info - press "Y"
57 | ```
58 |
59 | ### Further reading:
60 |
61 | - [rbash - restricted users](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/users/rbash.md)
62 |
--------------------------------------------------------------------------------
/docs/prerendering/detect-prerendering.md:
--------------------------------------------------------------------------------
1 | # Detect request from Prerendering engine during runtime
2 |
3 | When requests are coming from pre-rendering engine it can be detected in JS-runtime (e.g. front-end code)
4 |
5 | ## Detect that request is coming from pre-rendering engine
6 |
7 | Prerendering engine will set `window.IS_PRERENDERING` global variable to `true`. Detecting request from prerendering engine might be as easy as:
8 |
9 | ```js
10 | if (window.IS_PRERENDERING) {
11 | // This request is coming from Prerendering engine
12 | }
13 | ```
14 |
15 | __Note__: `window.IS_PRERENDERING` might be `undefined` on initial page load, and may change during runtime. That's why we recommend to pre-define a setter for `IS_PRERENDERING`:
16 |
17 | ```js
18 | let _is_prerendering = window.IS_PRERENDERING || false;
19 | Object.defineProperty(window, 'IS_PRERENDERING', {
20 | set(val) {
21 | _is_prerendering = val;
22 | if (_is_prerendering === true) {
23 | // This request is coming from Prerendering engine
24 | }
25 | },
26 | get() {
27 | return _is_prerendering;
28 | }
29 | });
30 | ```
31 |
32 | ## Detect type of the pre-rendering engine
33 |
34 | Like browsers, crawler, and bots come as "mobile" (small screen touch-devices) or as "desktop" (large screens without touch-events) the pre-rendering engine has the same two types. For cases when content needs to get optimized for different screens pre-rendering engine will set `window.IS_PRERENDERING_TYPE` global variable to `desktop` or `mobile`
35 |
36 | ```js
37 | if (window.IS_PRERENDERING_TYPE === 'mobile') {
38 | // This request is coming from "mobile" web crawler and "mobile" pre-rendering engine
39 | } else if (window.IS_PRERENDERING_TYPE === 'desktop') {
40 | // This request is coming from "desktop" web crawler and "desktop" pre-rendering engine
41 | } else {
42 | // This request is coming from user
43 | }
44 | ```
45 |
--------------------------------------------------------------------------------
/docs/prerendering/node-npm.md:
--------------------------------------------------------------------------------
1 | # Pre-rendering: Node.js Integration via NPM
2 |
3 | - Learn more - [what is Prerendering and why you need it](https://ostr.io/info/prerendering)
4 | - For more detailed info, examples and API see [`spiderable-middleware` package repository](https://github.com/veliovgroup/spiderable-middleware)
5 |
6 | This integration will work with any Node.js solution which follows Node's middleware convention, like: [express](https://www.npmjs.com/package/express), [connect](https://www.npmjs.com/package/connect), [vanilla Node.js server](https://nodejs.org/api/http.html), and others
7 |
8 | ### Installation
9 |
10 | ```shell
11 | npm install spiderable-middleware --save
12 | ```
13 |
14 | ### Update HTML Markup
15 |
16 | To cause the special behavior of web crawlers on JavaScript powered websites use `fragment` meta tag. Although it's officially deprecated by Google search engine, it's may be used by other search engines and web crawlers. [Learn more](https://developers.google.com/webmasters/ajax-crawling/docs/specification)
17 |
18 | ```html
19 |
20 |
21 |
22 |
23 |
24 |
25 |
26 |
27 |
28 | ```
29 |
30 | ### Middleware integration
31 |
32 | [See more detailed examples examples here](https://github.com/veliovgroup/spiderable-middleware/tree/master/examples)
33 |
34 | ```js
35 | 'use strict';
36 | import http from 'http';
37 | import Spiderable from 'spiderable-middleware';
38 |
39 | const spiderable = new Spiderable({
40 | rootURL: 'http://example.com',
41 | auth: 'APIUser:APIPass',
42 | });
43 |
44 | const requestListener = function (req, res) {
45 | spiderable.handler(req, res, function () {
46 | res.writeHead(200, {'Content-Type': 'text/plain; charset=UTF-8'});
47 | res.end('Hello vanilla NodeJS!');
48 | });
49 | };
50 |
51 | http.createServer(requestListener).listen(3000);
52 | ```
53 |
--------------------------------------------------------------------------------
/tutorials/linux/security/rootkit-hunter.md:
--------------------------------------------------------------------------------
1 | # Rootkit Hunter (RKHunter)
2 |
3 | ## References:
4 |
5 | - [RKHunter project page](http://rkhunter.sourceforge.net/)
6 | - Article by [Digital Ocean](https://www.digitalocean.com/community/tutorials/how-to-use-rkhunter-to-guard-against-rootkits-on-an-ubuntu-vps)
7 | - For latest release [go here](https://sourceforge.net/projects/rkhunter/files/)
8 |
9 | On the moment of writing latest release is: `v1.4.6 2018-02-24`
10 |
11 | ## Installation:
12 |
13 | Download RKHunter script
14 |
15 | ```shell
16 | wget https://downloads.sourceforge.net/project/rkhunter/rkhunter/1.4.6/rkhunter-1.4.6.tar.gz
17 | ```
18 |
19 | Unpack RKHunter archive and install:
20 |
21 | ```shell
22 | tar xzvf rkhunter-1.4.6.tar.gz
23 | cd rkhunter-1.4.6
24 | ./installer.sh --install
25 | cd ../
26 | rm -Rf rkhunter-1.4.6*
27 | ```
28 |
29 | ### Install prerequisites:
30 |
31 | ```shell
32 | apt-get install binutils libreadline5 ssl-cert unhide
33 | ```
34 |
35 | RKHunter also requires `ruby` to be installed. Read more about [ruby installation](https://www.ruby-lang.org/en/documentation/installation).
36 |
37 | ## Update RKHunter database and generate file properties database used by RKHunter:
38 |
39 | ```shell
40 | rkhunter --update --propupd
41 | ```
42 |
43 | ## Run full system check:
44 |
45 | ```shell
46 | rkhunter -c --enable all --disable none
47 | ```
48 |
49 | Read more detailed setup at [Digital Ocean article](https://www.digitalocean.com/community/tutorials/how-to-use-rkhunter-to-guard-against-rootkits-on-an-ubuntu-vps)
50 |
51 | ## Further reading:
52 |
53 | - [Use Lynis for security audit](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/security/lynis-security-audit.md)
54 | - [Basic firewall with `iptables`](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/security/iptables-firewall.md)
55 | - [Digital Ocean article](https://www.digitalocean.com/community/tutorials/how-to-use-rkhunter-to-guard-against-rootkits-on-an-ubuntu-vps)
56 |
--------------------------------------------------------------------------------
/docs/prerendering/cache-purge.md:
--------------------------------------------------------------------------------
1 | # Prerendering Cache Purging
2 |
3 | ## Full cache purging:
4 |
5 | In the case of significant changes on a website, cached results can be purged. Full cache purging available once per 2 hours.
6 |
7 | 
8 |
9 | ## Individual page cache purge:
10 |
11 | While full cache purging is limited by time-frame. The individually cached page can be purged anytime without limits. To purge single page from cache "Cache ID" has to be obtained in response headers as `X-Prerender-Id`.
12 |
13 |
14 |
15 |
16 |
17 |
18 |
19 |
20 |
21 |
22 | ## Purge aggressively cached pages
23 |
24 | The reason why you may not found `x-prerender-id` header in DevTools is aggressive caching headers or usage of ServiceWorker or AppCache, to avoid this issue please follow next steps (*in Chrome, Opera, Brave, and other Chromium-based browser*):
25 |
26 | 1. Open new **Private** window
27 | 2. Open URL with appended `?_escaped_fragment_=`
28 | 3. Open "Developer Tools"
29 | 4. In "Developer Tools" go to "Network" tab
30 | 5. Check "Disable Cache" checkbox
31 | 6. Reload/Refresh page
32 | 7. In "Network" tab select `/?_escaped_fragment_=` document (*should be the first*)
33 | 8. In opened area make sure "Header" tab is selected
34 | 9. In "Response Headers" section find `x-prerender-id` header
35 |
36 | ## Notes:
37 |
38 | While cache is purged from Pre-rendering engine it may remain cached in intermediate caching layers from few minutes to several days, it depends on selected "Cache TTL" and [rendering endpoint](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/rendering-endpoints.md).
39 |
--------------------------------------------------------------------------------
/tutorials/linux/bash_profile-tuning.md:
--------------------------------------------------------------------------------
1 | .bash_profile tuning
2 | ======
3 |
4 | Create a file named `.bash_profile` in home folder of Linux user.
5 |
6 | To switch to users' home folder use:
7 | ```shell
8 | cd ~
9 | ```
10 |
11 | ### Enable history completion with up and down arrows:
12 | ```bash
13 | bind '"\e[A":history-search-backward'
14 | bind '"\e[B":history-search-forward'
15 | ```
16 |
17 | ### Enable case-insensitive completions (tab key):
18 | ```bash
19 | bind "set completion-ignore-case on"
20 | bind "set show-all-if-ambiguous on"
21 | ```
22 |
23 | ### Enable xterm colors:
24 | ```bash
25 | export TERM='xterm-256color'
26 |
27 | export CLICOLOR=1
28 | export LSCOLORS='BxBxhxDxfxhxhxhxhxcxcx'
29 | ```
30 |
31 | ### Add even more colors (using famous [solarized theme](http://git.io/solarized-colors)):
32 | ```bash
33 | if tput setaf 1 &> /dev/null; then
34 | tput sgr0;
35 | bold=$(tput bold);
36 | reset=$(tput sgr0);
37 | black=$(tput setaf 0);
38 | blue=$(tput setaf 33);
39 | cyan=$(tput setaf 37);
40 | green=$(tput setaf 64);
41 | orange=$(tput setaf 166);
42 | purple=$(tput setaf 125);
43 | red=$(tput setaf 124);
44 | violet=$(tput setaf 61);
45 | white=$(tput setaf 15);
46 | yellow=$(tput setaf 136);
47 | else
48 | bold='';
49 | reset="\e[0m";
50 | black="\e[1;30m";
51 | blue="\e[1;34m";
52 | cyan="\e[1;36m";
53 | green="\e[1;32m";
54 | orange="\e[1;33m";
55 | purple="\e[1;35m";
56 | red="\e[1;31m";
57 | violet="\e[1;35m";
58 | white="\e[1;37m";
59 | yellow="\e[1;33m";
60 | fi;
61 | ```
62 |
63 | ### Set terminal title and prompt format to `username at hostname in current/path`:
64 | ```bash
65 | userStyle="${orange}";
66 | hostStyle="${yellow}";
67 |
68 | # Set the terminal title and prompt.
69 | PS1="\[\033]0;\W\007\]"; # working directory base name
70 | PS1+="\[${bold}\]\n"; # newline
71 | PS1+="\[${userStyle}\]\u"; # username
72 | PS1+="\[${white}\] at ";
73 | PS1+="\[${hostStyle}\]\h"; # hostname
74 | PS1+="\[${white}\] in ";
75 | PS1+="\[${green}\]\w"; # current path
76 | PS1+="\n";
77 | PS1+="\[${white}\]\$ \[${reset}\]"; # `$` (and reset color)
78 | export PS1;
79 |
80 | # Move prompt to second line
81 | PS2="\[${yellow}\]→ \[${reset}\]";
82 | export PS2;
83 | ```
84 |
85 | ### Apply changes:
86 | ```shell
87 | source ~/.bash_profile
88 | ```
89 |
90 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # ostr.io
2 |
3 | Web services for modern and legacy websites, web apps, e-commerce shops, IoT devices, social and corporate portals.
4 |
5 | [ostr.io](https://ostr.io) is a unified web-services platform delivering uptime monitoring, proactive WebSec protection, web analytics, serverless CRON scheduler and SEO-boosting pre-rendering for websites, web apps, e-commerce shops and corporate portals. Every service is security-first, privacy-by-design and fully GDPR and CCPA compliant, with seamless usage on cloud, virtual or bare-metal environments.
6 |
7 | ## Services
8 |
9 | - [__Pre-rendering SEO Middleware__](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/README.md) — *Improve SEO and Web Vitals scores. Increase organic traffic from SERPs and AI agents/bots*
10 | - [__HTTP/SNMP Monitoring__](https://github.com/veliovgroup/ostrio/blob/master/docs/monitoring/README.md) — *Proactive monitoring for websites, APIs, microservices, servers, hardware, and IoT. Preventing incidents and reducing Mean Time to Repair*
11 | - [__Web Analytics__](https://github.com/veliovgroup/ostrio/blob/master/docs/analytics/README.md) — *Lightweight, GDPR/CCPA compliant, and real-time metrics with opensource code*
12 | - [__Web-CRON__](https://github.com/veliovgroup/ostrio/blob/master/docs/web-cron/README.md) — *Serverless Automation & Tasks Scheduler*
13 | - [__WebSec__](https://github.com/veliovgroup/ostrio/blob/master/docs/websec/README.md) — *Domains, DNS, SSL, and Whois Monitoring*
14 |
15 | ## Account & Help
16 |
17 | - 👤 [Account Basics](https://github.com/veliovgroup/ostrio/blob/master/docs/account/README.md) — *Learn how to manage your account, contact, and other ostr.io features*
18 | - 🤝 [Get Help & Support](https://github.com/veliovgroup/ostrio/blob/master/docs/SUPPORT.md) — *General User Support*
19 |
20 | ## Additional Info
21 |
22 | - [CHANGELOG](https://github.com/veliovgroup/ostrio/blob/master/HISTORY.md) — *Learn what's new*
23 | - [Sitemap](https://github.com/veliovgroup/ostrio/blob/master/SITEMAP.md)
24 |
25 |
26 | ## Tutorials
27 |
28 | - [Tutorials](https://github.com/veliovgroup/ostrio/blob/master/tutorials/README.md) — *Dedicated tutorials focused on improvements and optimizations of SSL/TLS certificates, Web Development, SEO, Linux, and MongoDB*
29 |
30 | ----
31 |
32 | > ▲ created and maintained by humans with ♥
33 |
--------------------------------------------------------------------------------
/tutorials/linux/security/iptables-firewall.md:
--------------------------------------------------------------------------------
1 | # Basic firewall with iptables
2 |
3 | __NOTE: IT'S A DANGER ZONE.__ This is expert-level settings setup. You must know what you're doing and do not blindly copy-paste commands and rules described below, otherwise, you may end up with the unaccessible server.
4 |
5 | This tutorial will set iptables rules to accept traffic only on `http (80)`, `https (443)` and `ssh (22)` ports, which is the fine setup for most basic applications. Using suggested rules you're free to add more udp/tcp ports on demand.
6 |
7 | See also the great article on iptables rules by [Digital Ocean](https://www.digitalocean.com/community/tutorials/iptables-essentials-common-firewall-rules-and-commands) and [VPS Cheap](https://crm.vpscheap.net/knowledgebase.php?action=displayarticle&id=29)
8 |
9 | Start with creating the blank iptables file:
10 |
11 | ```shell
12 | iptables-save > /etc/firewall.conf
13 | ```
14 |
15 | Now edit exported rules:
16 |
17 | ```shell
18 | # nano /etc/firewall.conf
19 | # You should end up with something like:
20 | :INPUT ACCEPT [0:0]
21 | :FORWARD ACCEPT [0:0]
22 | :OUTPUT ACCEPT [0:0]
23 | # Allow everything on localhost (loopback)
24 | -A INPUT -i lo -j ACCEPT
25 | -A OUTPUT -o lo -j ACCEPT
26 |
27 | # Allow all outgoing connections
28 | -A OUTPUT -j ACCEPT
29 |
30 | # Allow all active incoming connection to continue
31 | -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
32 | # Allow all active outgoing connection to continue
33 | -A OUTPUT -m conntrack --ctstate ESTABLISHED -j ACCEPT
34 | # Drop all INVALID incoming connections
35 | -A INPUT -m conntrack --ctstate INVALID -j DROP
36 |
37 | # Main incoming connection rules
38 | -A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
39 | -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
40 | -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
41 |
42 | # Drop all other incoming connections
43 | -A INPUT -j DROP
44 | # Reject any forwarding
45 | -A FORWARD -j REJECT
46 | COMMIT
47 | ```
48 |
49 | To test rules run:
50 |
51 | ```shell
52 | iptables-restore < /etc/firewall.conf
53 | ```
54 |
55 | To make created rules persistent, create file `/etc/network/if-up.d/firewall`
56 |
57 | ```shell
58 | #!/bin/sh
59 | iptables-restore < /etc/firewall.conf
60 | ```
61 |
62 | Make it executable:
63 |
64 | ```shell
65 | chmod +x /etc/network/if-up.d/firewall
66 | ```
67 |
68 | ## Further reading:
69 |
70 | - [Basic ipv6 firewall with `ip6tables`](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/security/iptables-firewall-ipv6.md)
71 |
--------------------------------------------------------------------------------
/tutorials/linux/security/iptables-firewall-ipv6.md:
--------------------------------------------------------------------------------
1 | # Basic ipv6 firewall with ip6tables
2 |
3 | __NOTE: IT'S A DANGER ZONE.__ This is expert-level settings setup. You must know what you're doing and do not blindly copy-paste commands and rules described below, otherwise, you may end up with the unaccessible server
4 |
5 | Use ipv6 setting only if your server has assigned the IPv6 address. To check assigned IPv6 addresses use: `/sbin/ifconfig | grep inet6`
6 |
7 | This tutorial will set ip6tables rules to accept traffic only on `http (80)`, `https (443)`, which is fine setup for most basic web applications. Using suggested rules you're free to add more udp/tcp ports on demand.
8 |
9 | See also the great article on ip6tables rules by [linode](https://www.linode.com/docs/security/firewalls/control-network-traffic-with-iptables).
10 |
11 | Start with creating the blank ip6tables file:
12 |
13 | ```shell
14 | ip6tables-save > /etc/firewallv6.conf
15 | ```
16 |
17 | Now edit exported rules:
18 |
19 | ```shell
20 | # nano /etc/firewallv6.conf
21 | # You should end up with something like:
22 | :INPUT ACCEPT [0:0]
23 | :FORWARD ACCEPT [0:0]
24 | :OUTPUT ACCEPT [0:0]
25 | # Allow everything on localhost (loopback)
26 | -A INPUT -i lo -j ACCEPT
27 | -A OUTPUT -o lo -j ACCEPT
28 |
29 | # Allow all outgoing connections
30 | -A OUTPUT -j ACCEPT
31 |
32 | # Allow all active incoming connection to continue
33 | -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
34 | # Allow all active outgoing connection to continue
35 | -A OUTPUT -m conntrack --ctstate ESTABLISHED -j ACCEPT
36 | # Drop all INVALID incoming connections
37 | -A INPUT -m conntrack --ctstate INVALID -j DROP
38 |
39 | # Main incoming connection rules
40 | -A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
41 | -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
42 |
43 | # Allow everything on link-local (interface)
44 | -A INPUT -s fe80::/10 -j ACCEPT
45 |
46 | # Drop all other incoming connections
47 | -A INPUT -j DROP
48 | # Reject any forwarding
49 | -A FORWARD -j REJECT
50 | COMMIT
51 | ```
52 |
53 | To test rules run:
54 |
55 | ```shell
56 | ip6tables-restore < /etc/firewallv6.conf
57 | ```
58 |
59 | To make created rules persistent, create the file `/etc/network/if-up.d/firewall`
60 |
61 | ```shell
62 | #!/bin/sh
63 | ip6tables-restore < /etc/firewallv6.conf
64 | ```
65 |
66 | Make it executable:
67 |
68 | ```shell
69 | chmod +x /etc/network/if-up.d/firewall
70 | ```
71 |
72 | ## Further reading:
73 |
74 | - [Basic firewall with `iptables`](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/security/iptables-firewall.md)
75 |
--------------------------------------------------------------------------------
/tutorials/website/social-networking/twitter-cards-basics.md:
--------------------------------------------------------------------------------
1 | # Twitter Card basics
2 |
3 | ## About:
4 |
5 | __TL;TR;__ Adding Twitter Card tags (*a.k.a. meta `twitter:` tags*) to your website will make it appear more user-friendly and eye-catching in Twitter feed, and as a consequence - will cause more click and traffic.
6 |
7 | With Twitter Card you can add specific image, title and text to each Twitted link. Twitter supports 4 types of cards: "[Summary](https://dev.twitter.com/cards/types/summary)", "[Summary with large image](https://dev.twitter.com/cards/types/summary-large-image)", "[Mobile Application](https://dev.twitter.com/cards/types/app)" and "[Audio/Video/Media Player](https://dev.twitter.com/cards/types/player)". In this article we will cover most used card - "Summary with large image".
8 |
9 | ## Basic structure:
10 |
11 | This example is very basic usage, but will cover most of the cases. For more info and examples - *see links below*.
12 |
13 | ```html
14 |
15 |
16 |
17 |
18 |
19 | Traveler: Norway
20 |
21 |
22 |
23 |
24 |
25 |
26 |
27 |
28 |
29 |
30 |
31 |
32 |
33 |
34 |
35 |
36 |
37 |
38 |
39 |
40 |
41 |
42 |
43 |
44 |
45 | ```
46 |
47 | ## Further reading:
48 |
49 | - [Documentation by Twitter](https://dev.twitter.com/cards/overview)
50 | - [Test your Twitter Card (validator)](https://cards-dev.twitter.com/validator)
51 |
--------------------------------------------------------------------------------
/SITEMAP.md:
--------------------------------------------------------------------------------
1 | # Sitemap
2 |
3 | - 🗺 [Sitemap](https://ostr.io/info/sitemap)
4 |
5 | ## Info
6 |
7 | - 🏠 [Homepage landing](https://ostr.io)
8 | - 🤖 [Pre-rendering](https://ostr.io/info/prerendering)
9 | - 🔭 [Monitoring](https://ostr.io/info/monitoring)
10 | - 🛡 [Web Security](https://ostr.io/info/domain-names-protection)
11 | - 📈 [Website Analytics](https://ostr.io/info/web-analytics)
12 | - ⏰ [Web-CRON](https://ostr.io/info/web-cron)
13 | - 👨💻 [Built by Developers for Developers](https://ostr.io/info/built-by-developers-for-developers)
14 | - 📋 [Docs and how-tos](https://ostr.io/info/docs)
15 | - 🏷 [Pricing & Plans](https://ostr.io/info/pricing)
16 | - 📔 [ToS](https://ostr.io/info/terms-and-conditions)
17 | - 📔 [Privacy Policy](https://ostr.io/info/privacy-policy)
18 | - 📔 [GDPR](https://ostr.io/info/gdpr)
19 | - 👷 [Changelog](https://github.com/veliovgroup/ostrio/blob/master/HISTORY.md)
20 |
21 | ## Services
22 |
23 | These pages exist only for logged-in users.
24 |
25 | - 🎛 [Servers Control Panel](https://ostr.io/service/servers)
26 | - 🌎 [Domains Control Panel](https://ostr.io/service/domains)
27 | - 🔭 [Monitoring Control Panel](https://ostr.io/service/monitoring)
28 | - 📈 [Website Analytics Control Panel](https://ostr.io/service/analytics)
29 | - 🤖 [Pre-rendering Control Panel](https://ostr.io/service/prerender)
30 | - ⏰ [Web-CRON Tasks Control Panel](https://ostr.io/service/crons)
31 | - 📲 [SMS Alerts](https://ostr.io/service/sms)
32 |
33 | ## Account
34 |
35 | These pages exist only for logged-in users.
36 |
37 | - ⚙️ [Account Setting](https://ostr.io/account)
38 | - 🔔 [Notifications](https://ostr.io/account/notifications)
39 | - 📓 [Contacts](https://ostr.io/account/contacts)
40 | - 🛒 [Cart](https://ostr.io/account/cart)
41 | - 🤑 [Billing & Bonuses](https://ostr.io/account/billing)
42 |
43 | ## Open Source
44 |
45 | - 📦 [`seo-middleware-nextjs`](https://github.com/veliovgroup/seo-middleware-nextjs)
46 | - 📦 [`spiderable-middleware`](https://github.com/veliovgroup/spiderable-middleware#spiderable-middleware)
47 | - 📦 [`ostrio-analytics`](https://github.com/veliovgroup/ostrio-analytics/#analytics-for-ostrio)
48 | - 📋 [Documentation repository](https://github.com/veliovgroup/ostrio)
49 |
50 | ## Domains
51 |
52 | List of related domain names.
53 |
54 | - 🤖 [prerendering.com](https://prerendering.com)
55 | - 🤖 [prerendering.info](https://prerendering.info)
56 | - 🤖 [prerender.info](https://prerender.info)
57 | - 🔭 [snmp-monitoring.com](https://snmp-monitoring.com)
58 | - 🔭 [snmp-monitoring.info](https://snmp-monitoring.info)
59 | - 🛡 [domain-protection.info](https://domain-protection.info)
60 | - ⏰ [web-cron.info](https://web-cron.info)
61 |
--------------------------------------------------------------------------------
/tutorials/mongodb/enable-oplog.md:
--------------------------------------------------------------------------------
1 | Enable OpLog
2 | ======
3 |
4 | For more info see [original article](https://veliovgroup.com/article/2qsjtNf8NSB9XxZDh/mongodb-replica-set-with-oplog).
5 |
6 | ### Before you go:
7 | - Read [what is OpLog](http://www.briancarpio.com/2012/04/21/mongodb-the-oplog-explain/)
8 | - Read [official MongoDB reference](https://docs.mongodb.com/manual/core/replica-set-oplog/)
9 | - Existing ReplicaSet is required for OpLog, read how to create one: [on single server](https://github.com/veliovgroup/ostrio/blob/master/tutorials/mongodb/single-server-replica-set.md) or [multiple (cluster) servers](https://github.com/veliovgroup/ostrio/blob/master/tutorials/mongodb/multiple-server-replica-set.md)
10 |
11 | ### Definitions:
12 | - `oplogger` user - User with `read` access to `local` database, where is OpLog is stored
13 | - `oplogger` role - Role which grants `read` access to `local` database
14 | - `` - placeholder, should be changed to strong password. __Always placed in double quotes__
15 |
16 | ### 1. In Mongo Shell on PRIMARY: Create `oplogger` role:
17 | ```shell
18 | # Mongo Shell:
19 | use admin
20 | db.runCommand({createRole:"oplogger", privileges:[{resource: {db:"local", collection:"system.replset"}, actions: ["find"]}], roles:[{role:"read", db:"local"}]})
21 | ```
22 |
23 | ### 2. In Mongo Shell on PRIMARY: Create `oplogger` user and grant `oplogger` role:
24 | ```javascript
25 | // Mongo Shell:
26 | use admin
27 | // For MongoDB 2.4
28 | db.createUser({user:"oplogger", pwd:, roles:[], otherDBRoles:{local:["read"]}})
29 | // For MongoDB >= 2.6
30 | db.createUser({user:"oplogger", pwd:, roles:[{role: "read", db: "local"}]})
31 | db.runCommand({grantRolesToUser:"oplogger", roles:["oplogger"]})
32 | ```
33 |
34 | ### 3. In Mongo Shell on PRIMARY: Make sure user is properly created:
35 | ```shell
36 | # Mongo Shell:
37 | use admin
38 | show users
39 | ```
40 |
41 | ### 4. Limit OpLog size:
42 | By default OpLog will take 5% of __total__ disk size. It's a good idea to limit it.
43 |
44 | If your application relies on OpLog and will read/watch for changes - OpLog size should fit in RAM, we recommend to set it to 25%-50% of RAM.
45 |
46 | In examples below we set OpLog size to `8192` MB or 8Gb.
47 |
48 | Via `/etc/mongod.conf`:
49 | ```yaml
50 | replication:
51 | oplogSizeMB: 8192
52 | replSetName: rs0
53 | ```
54 |
55 | Via `mongod` command flag:
56 | ```shell
57 | mongod --oplogSize 8192 --config /etc/mongod.conf
58 | ```
59 |
60 | __Note__: This won't work on existing OpLog, to change size of existing OpLog [read this article](https://docs.mongodb.com/manual/tutorial/change-oplog-size/).
61 |
62 | ### 5. Connection string:
63 | ```plain
64 | mongodb://oplogger:@:,:,:/local?authSource=admin&replicaSet=rs0
65 | ```
66 |
--------------------------------------------------------------------------------
/docs/account/tips-and-tricks.md:
--------------------------------------------------------------------------------
1 | # Tips and Tricks
2 |
3 | ostr.io usage tips and tricks
4 |
5 | ## ⌨️ Key bindings and shortcuts
6 |
7 | - Pagination: Arrow keys ← and → (*works on every page and modals with pagination*);
8 | - Activate Search: ctrl+f (⌘+f on MacOS);
9 | - Start Searching: return (⏎) key;
10 | - Cancel Search: esc key.
11 |
12 | ## 🤑 Free credits
13 |
14 | - Create an account using [this special link](https://ostr.io/signup/gCZWjiBScePWrnnDr) to get $50 off your second purchase;
15 | - Invite friends! Obtain [unique referral link](https://ostr.io/account#ref) and share it with your friends or colleagues to earn free credits;
16 | - Subscribe on our social accounts ([Twitter](https://twitter.com/ostrio_service), [Facebook](https://www.facebook.com/ostrio.service), [Instagram](https://www.instagram.com/ostr.io/), [Medium](https://medium.com/@ostr.io)) where we post promo-codes;
17 | - Subscribe on our newsletter in [account settings](https://ostr.io/account).
18 |
19 | ## 🤖 Pre-rendering Tips
20 |
21 | - Pass `/sitemap.xml` to Pre-Render (*Pre-render a website*) in __Pre-rendering Panel__ to instantly render whole website;
22 | - Use [alternative *rendering endpoint*](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/rendering-endpoints.md) to reach best performance results.
23 |
24 | ### Running pre-rendering integration tests:
25 |
26 | - Use `cURL` and `Authorization` header (*see nginx integration*): `curl -v -H "Authorization: Basic TOKEN" https://render.ostr.io/?url=https://your-website-domain.com`;
27 | - Use `cURL` and authentication credentials (*see node.js integration*): `curl -v https://auth:string@render-bypass.ostr.io/?url=https://your-website-domain.com`;
28 | - Use `test:test` credentials to test general integration (*to make sure web application server can reach pre-rendering service*);
29 | - Use `Authorization: Basic dGVzdDp0ZXN0` header to test general integration (*to make sure web application server can reach pre-rendering service*);
30 | - Learn more about running tests in [`spiderable-middleware` package documentation](https://github.com/veliovgroup/spiderable-middleware#running-tests).
31 |
32 | ## 📈 Web Analytics Tips
33 |
34 | - Enable "Development Mode" to test web analytics integration and data-reports right from `localhost`.
35 |
36 | ## 🔭 Monitoring Tips
37 |
38 | - Use `{{rand}}` placeholder (*including curly brackets*) in endpoint URL to generate unique random URI for every checking request. Note: `{{rand}}` can be used multiple times in a single URL, example: `/path/{{rand}}/part?nc={{rand}}`.
39 |
40 | ## ⏰ Web-CRON Tips
41 |
42 | - Use `{{rand}}` placeholder (*including curly brackets*) in endpoint URL to generate unique random URI for every CRON request. Note: `{{rand}}` can be used multiple times in a single URL, example: `/path/{{rand}}/part?nc={{rand}}`.
43 |
--------------------------------------------------------------------------------
/tutorials/website/social-networking/open-graph-basics.md:
--------------------------------------------------------------------------------
1 | # Open Graph basics
2 |
3 | ## About:
4 |
5 | __TL;TR;__ Adding Open Graph tags (*a.k.a. meta `og:` tags*) to your website will make it appear more user-friendly in social networks feed, and as a consequence - will cause more click and traffic.
6 |
7 | __Description by [Redclay Interactive](https://www.redclayinteractive.com/what-are-open-graph-tags/):__
8 | > The open graph protocol allows developers to leverage Facebook in new and exciting ways. One of the easiest ways to venture into the world of Open Graph is to add the open graph meta tags to your site. Open graph meta tags allow you to control what content shows up when a page is shared on Facebook.
9 |
10 | ## Basic structure:
11 |
12 | This example is very basic usage, but will cover most of the cases. For more info and examples - *see links below*.
13 |
14 | ```html
15 |
16 |
17 |
18 |
19 |
20 | Traveler: Norway
21 |
22 |
23 |
24 |
25 |
26 |
27 |
28 |
29 |
30 |
31 |
32 |
33 |
34 |
35 |
36 |
37 |
38 |
39 |
40 |
41 |
42 |
43 |
44 |
45 |
46 |
47 |
48 |
49 |
50 |
51 |
52 | ```
53 |
54 | ## Further reading:
55 |
56 | - [Open Graph Protocol specs](http://ogp.me/)
57 | - [Documentation by Facebook](https://developers.facebook.com/docs/sharing/webmasters)
58 | - [Test your markup](https://developers.facebook.com/tools/debug/sharing/)
59 |
--------------------------------------------------------------------------------
/tutorials/linux/users/rbash.md:
--------------------------------------------------------------------------------
1 | # Restricted users (`rbash`)
2 |
3 | Based on [`rbash` article](https://veliovgroup.com/article/BmtWycSfZL37zXMZc/how-to-rbash)
4 |
5 | ## When to use `rbash`?
6 |
7 | - SSH login-only user
8 | - Other users accounts
9 | - Application users
10 | - Other security and permission restriction cases
11 |
12 | Example below creates `ruser`, with only access to `su` command. SSH login-only user is the great use case for `rbash`.
13 |
14 | ### Create a user
15 |
16 | Create user named `ruser`:
17 |
18 | ```shell
19 | useradd ruser
20 | ```
21 |
22 | Set password for new user (*only users with password can login to a server*):
23 |
24 | ```shell
25 | passwd ruser
26 | # then type-in new password twice
27 | ```
28 |
29 | ### Confirm `rbash` is installed
30 |
31 | `rbash` shell is preinstalled on most of Linux distributions, however, to confirm `rbash` shell is installed, execute next lines:
32 |
33 | ```shell
34 | cat /etc/shells
35 | # and:
36 | which rbash
37 | # both commands should have /bin/rbash in return
38 | ```
39 |
40 | If `rbash` is not listed in `/etc/shells` install it via `agt-get` or `aptitude`. After installation, check `/etc/shells` again to make sure it is activated.
41 |
42 | ### Change user's shell
43 |
44 | To change user's default shell use `chsh` (*__ch__ ange __sh__ ell*):
45 |
46 | ```shell
47 | chsh -s /bin/rbash ruser
48 | ```
49 |
50 | ### Create user's home directory
51 |
52 | ```shell
53 | # Create directory
54 | mkdir -p /home/ruser
55 |
56 | # Set home directory
57 | usermod -m -d /home/ruser ruser
58 | ```
59 |
60 | ### Create shell profiles
61 |
62 | ```shell
63 | # Go to ruser home directory
64 | cd /home/ruser
65 | # Create profile
66 | touch .bashrc
67 | echo "export PATH=/home/ruser/usr/bin" >> .bashrc
68 | for i in .bash_login .bash_profile .bash_logout .bash_profile .profile; do cp .bashrc $i; done
69 | ```
70 |
71 | ### Create user's bin directory
72 |
73 | ```shell
74 | mkdir -p /home/ruser/usr/bin
75 | ```
76 |
77 | *From now, user named* `ruser` *after login via* `su`*,* `sudo` *or* `ssh` *can not run any commands.*
78 |
79 | ### Link allowed commands
80 |
81 | For example, we would like to add `su`, as allowed command for `ruser`
82 |
83 | ```shell
84 | which su
85 | # returns /bin/su, copy it
86 | ln -s /bin/su /home/ruser/usr/bin
87 | # │ └─ symlink to ruser's bin
88 | # └─ output from `which`
89 | ```
90 |
91 | Repeat the same procedure for all commands (programs, packages) which should be allowed to run by `ruser`
92 |
93 | ### Set right permissions
94 |
95 | ```shell
96 | chmod -R 750 /home/ruser
97 | chown -R ruser:ruser /home/ruser
98 | ```
99 |
100 | ### Further reading:
101 |
102 | - [Change default SSH port](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/security/change-ssh-port.md)
103 | - [Disable root login via SSH](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/security/disable-ssh-root.md)
104 | - [Use SSH keys to login](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/security/use-ssh-keys.md)
105 |
--------------------------------------------------------------------------------
/docs/prerendering/optimization.md:
--------------------------------------------------------------------------------
1 | # Prerendering Optimization
2 |
3 | ## ToC:
4 |
5 | - [Speed up rendering](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/optimization.md#speed-up-rendering)
6 | - [Nginx](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/optimization.md#nginx)
7 |
8 | ### Speed up rendering
9 |
10 | You should tell to the Prerendering engine when the page is fully rendered. It's can be easily done via `IS_RENDERED` variable located in the global scope.
11 | `window.IS_RENDERED` should be defined and set to `false` at the very beginning of your page or application bundle script file, for example right in `head` tag:
12 |
13 | ```html
14 |
15 |
16 |
19 |
20 |
21 | ```
22 |
23 | After the page is fully rendered and filled with data, set `IS_RENDERED` to `true`:
24 |
25 | ```html
26 |
27 |
28 |
31 |
32 |
33 |
39 |
40 |
41 | ```
42 |
43 | It is a good idea to set a timeout, for the case if the page wasn't rendered in an expected period of time (*due to an error or bad application logic*):
44 |
45 | ```js
46 | setTimeout(function () {
47 | window.IS_RENDERED = true;
48 | }, 6000);
49 | ```
50 |
51 | ### Nginx
52 |
53 | If you're using Nginx integration for Prerendering and default settings doesn't satisfy you, you may want to do precise tuning with next directives:
54 |
55 | ```nginx
56 | http {
57 | # Enable caching
58 | proxy_cache_path /var/lib/nginx/cache levels=1:2 keys_zone=backcache:10m max_size=5G inactive=60m use_temp_path=off;
59 |
60 | location @prerendering {
61 | # Enlarge this directives in case of 408 errors
62 | proxy_read_timeout 25s;
63 | proxy_send_timeout 35s;
64 | proxy_connect_timeout 45s;
65 |
66 | # Disable buffering
67 | proxy_request_buffering off;
68 | proxy_buffering off;
69 | proxy_buffers 256 8k;
70 | proxy_buffer_size 8k;
71 | proxy_busy_buffers_size 64k;
72 |
73 | # Enable caching
74 | proxy_cache backcache;
75 | proxy_cache_use_stale error timeout http_404 http_500 http_502 http_503 http_504;
76 | proxy_cache_key "$scheme$request_method$host$request_uri";
77 | proxy_cache_valid 200 301 302 10m;
78 | proxy_cache_valid 404 2m;
79 | proxy_cache_valid any 1m;
80 | proxy_cache_lock on;
81 | proxy_cache_revalidate on;
82 | proxy_cache_bypass $http_pragma $http_upgrade;
83 | proxy_cache_min_uses 2;
84 |
85 | # Cache for descriptors
86 | open_file_cache max=1024 inactive=12h;
87 | open_file_cache_valid 12h;
88 | open_file_cache_min_uses 2;
89 | open_file_cache_errors off;
90 | }
91 | }
92 | ```
93 |
94 | ## Further reading
95 |
96 | - [Content optimization for different crawlers](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/detect-prerendering.md)
97 |
--------------------------------------------------------------------------------
/docs/analytics/detect-adblock.md:
--------------------------------------------------------------------------------
1 | # Track and Detect AdBlock usage
2 |
3 | AdBlock is the great add-on in most of the cases as the Internet full of scam, spam, and inappropriate ads. Sadly for other, good side of the Internet, publishers are losing revenue. Track how many of your website visitors are using AdBlock browser extension with ostr.io web analytics.
4 |
5 | __Two options:__
6 |
7 | - [Vanilla solution without 3rd party libraries](https://github.com/veliovgroup/ostrio/blob/master/docs/analytics/detect-adblock.md#vanilla-solution-without-3rd-party-libraries)
8 | - [Detect using NPM library](https://github.com/veliovgroup/ostrio/blob/master/docs/analytics/detect-adblock.md#with-npm-library)
9 |
10 | ## Vanilla solution without 3rd party libraries
11 |
12 | Install [`ostrio-analytics`](https://github.com/veliovgroup/ostrio-analytics#analytics-for-ostrio) library:
13 |
14 | ```shell
15 | # For CommonJS (Browser/Node.js):
16 | npm install ostrio-analytics --save
17 |
18 | # For Meteor.js
19 | meteor add ostrio:analytics
20 | ```
21 |
22 | Run test and report AdBlock usage to analytics:
23 |
24 | ```js
25 | // For CommonJS (Browser/Node.js):
26 | const analyticsTracker = new (require('ostrio-analytics'))('trackingId');
27 |
28 | // For Meteor.js:
29 | // import Analytics from 'meteor/ostrio:analytics';
30 | // import adblockDetect from 'adblock-detect';
31 | // const analyticsTracker = new Analytics('trackingId');
32 |
33 | let adBlockEnabled = false;
34 | const testAd = document.createElement('div');
35 | testAd.innerHTML = ' ';
36 | testAd.className = 'adsbox';
37 | document.body.appendChild(testAd);
38 | setTimeout(function() {
39 | if (testAd.offsetHeight === 0) {
40 | adBlockEnabled = true;
41 | }
42 |
43 | testAd.remove();
44 | if (adBlockEnabled) {
45 | analyticsTracker.pushEvent('AdBlock', 'active');
46 | }
47 | }, 128);
48 | ```
49 |
50 | ## With NPM library
51 |
52 | Install [`adblock-detect`](https://www.npmjs.com/package/adblock-detect) and [`ostrio-analytics`](https://github.com/veliovgroup/ostrio-analytics#analytics-for-ostrio) NPM packages:
53 |
54 | ```shell
55 | # For CommonJS (Browser/Node.js):
56 | npm install ostrio-analytics --save
57 | npm install adblock-detect --save
58 |
59 | # For Meteor.js
60 | meteor add ostrio:analytics
61 | meteor npm install adblock-detect --save
62 | ```
63 |
64 | Check AdBlock usage with `adblock-detect` and send result to analytics:
65 |
66 | ```js
67 | // For CommonJS (Browser/Node.js):
68 | const analyticsTracker = new (require('ostrio-analytics'))('trackingId');
69 | const adblockDetect = require('adblock-detect');
70 |
71 | adblockDetect(function(adblockDetected) {
72 | if (adblockDetected) {
73 | analyticsTracker.pushEvent('AdBlock', 'active');
74 | }
75 | });
76 | ```
77 |
78 | ```js
79 | // For Meteor.js:
80 | import { Meteor } from 'meteor/meteor';
81 | import Analytics from 'meteor/ostrio:analytics';
82 | import adblockDetect from 'adblock-detect';
83 |
84 | const analyticsTracker = new Analytics('trackingId');
85 | Meteor.startup(() => {
86 | adblockDetect(function(adblockDetected) {
87 | if (adblockDetected) {
88 | analyticsTracker.pushEvent('AdBlock', 'active');
89 | }
90 | });
91 | });
92 | ```
93 |
94 | ## Further reading:
95 |
96 | - [web-analytics docs](https://github.com/veliovgroup/ostrio-analytics#analytics-for-ostrio)
97 |
--------------------------------------------------------------------------------
/docs/monitoring/README.md:
--------------------------------------------------------------------------------
1 | # Monitoring
2 |
3 | __Fully SaaS with zero setup__. Get real-time email and SMS alerts for website or server downtime, slow responses, unexpected content changes, SSH logins, and resource spikes. Covers everything from HTTP errors to CPU, RAM, and disk usage
4 |
5 | ## Intro
6 |
7 | Downtime costs revenue. With most business online, rapid detection of server, hardware, and software failures is critical. ostr.io Monitoring provides continuous, real-time oversight with instant alerts and predictive insights—preventing incidents and reducing Mean Time to Repair.
8 |
9 | ## Supported Protocols
10 |
11 | `http` and `https` protocols can be used for availability checks, perfect for API endpoints, websites, and web servers. Although it doesn't require any setup for simple availability monitoring it can be easily extended. If an endpoint returns a response with plain one-level `XML` or `JSON` - first-level numerical options will be parsed with an option to create alert triggers.
12 |
13 | `SNMP` protocol more complex and requires additional setup on the server side. It is the industry standard for hardware monitoring. With SNMP we can collect data, monitor and warn about CPU and RAM utilization, storage space, network load, SSH sessions, server reboots, and other critical data.
14 |
15 | ## Documentation
16 |
17 | Learn about Monitoring Service features.
18 |
19 | ### HTTP and SNMP monitoring
20 |
21 | - [Basic HTTP(S) Monitoring](https://github.com/veliovgroup/ostrio/blob/master/docs/monitoring/basics.md)
22 | - [HTTP(S) Monitoring with web-authentication](https://github.com/veliovgroup/ostrio/blob/master/docs/monitoring/with-auth.md)
23 | - Advanced: [SNMP (CPU, RAM, HDD, SSD, Bandwidth, Processes, SSH Sessions, Server Reboot) monitoring](https://github.com/veliovgroup/ostrio/blob/master/docs/monitoring/snmp.md)
24 |
25 | ### Tutorials and how to's
26 |
27 | - [Nginx stats and usage Monitoring](https://github.com/veliovgroup/ostrio/blob/master/docs/monitoring/nginx-stats.md)
28 | - [Build "uncacheable" path to endpoint, prevent cached results](https://github.com/veliovgroup/ostrio/blob/master/docs/monitoring/custom-path.md)
29 |
30 | > [!TIP]
31 | > Use `{{rand}}` placeholder (*including curly brackets*) in endpoint URL to generate unique random URI for every checking request. Note: `{{rand}}` can be used multiple times in a single URL, example: `/path/{{rand}}/part?nc={{rand}}`
32 |
33 | ## Further reading
34 |
35 | - [Learn more about Monitoring Service by ostr.io](https://ostr.io/info/monitoring)
36 |
37 |
39 |
40 |
56 |
--------------------------------------------------------------------------------
/docs/monitoring/nginx-stats.md:
--------------------------------------------------------------------------------
1 | Basic stats for Nginx
2 | ======
3 |
4 | This tutorial will show how to collect Nginx real-time connection metrics, like `active`, `reading`, `writing`, `waiting`. Including protection with simple `auth_basic`.
5 |
6 | __Do not forget to change the value in between `<` and `>` symbols to your own value!__
7 |
8 | #### First create `.htpasswd` file
9 | ```shell
10 | # Go to nginx directory
11 | cd /etc/nginx
12 |
13 | # There are various online websites
14 | # where you can easily generate
15 | # login / password pair
16 | # if you don't trust them
17 | # use htpasswd CLI as shown below
18 | htpasswd .htpasswd
19 | # You will be asked to prompt a password for the user.
20 | ```
21 |
22 | #### Nginx configuration
23 | Let's assume you're editing `/etc/nginx/sites-available/default`
24 | ```shell
25 | # Go to the sites-available directory
26 | cd /etc/nginx/sites-available/
27 | # Edit `default` file, we using nano in the example below, you're free to use your favorite editor
28 | nano default
29 | ```
30 |
31 | Configuration:
32 | ```nginx
33 | server {
34 | listen 80 default_server deferred;
35 | server_name example.com;
36 |
37 | location / {
38 | # Return empty response or do something else here
39 | return 204;
40 | }
41 |
42 | location ~ ^/ngstat/ {
43 | # Use auth_basic with .htpasswd created above
44 | auth_basic "Show your pass please";
45 | auth_basic_user_file /etc/nginx/.htpasswd;
46 |
47 | try_files $uri @stat;
48 | }
49 |
50 | location @stat {
51 | # Internal redirect
52 | rewrite ^(.*)$ /__$1 last;
53 | }
54 |
55 | # Return as JSON
56 | location /__/ngstat/json {
57 | # This block is accessible only
58 | # after successful authentication
59 | internal;
60 | default_type "application/json";
61 | return 200 "{\"active\":$connections_active, \"reading\":$connections_reading, \"writing\":$connections_writing, \"waiting\":$connections_waiting}";
62 | }
63 |
64 | # Return as XML
65 | location /__/ngstat/xml {
66 | # This block is accessible only
67 | # after successful authentication
68 | internal;
69 | default_type "text/xml";
70 | return 200 "$connections_active$connections_reading$connections_writing$connections_waiting";
71 | }
72 | }
73 | ```
74 |
75 | #### Make sure configuration syntax is valid
76 | ```shell
77 | service nginx configtest
78 | ```
79 |
80 | #### Reload configuration to apply changes
81 | ```shell
82 | service nginx reload
83 | ```
84 |
85 | #### Test endpoints
86 | Next URLs will ask for login and password and return Nginx stats upon successful authentication:
87 | ```
88 | http://example.com/ngstat/json
89 | http://example.com/ngstat/xml
90 | ```
91 |
92 | Add endpoint at [ostr.io](https://ostr.io):
93 | 1. Go to [Monitoring](https://ostr.io/service/monitoring)
94 | 2. Click on "*Add Endpoint*"
95 | 3. To select server type-in its name
96 | 4. Select "*HTTP*" (or *HTTPS*) in "*Protocol*" drop-down list
97 | 5. Select desired "*Check Frequency*"
98 | 6. Click on "*Add authentication*"
99 | 7. Enter username provided to `htpasswd` command
100 | 8. Enter password provided to `htpasswd` command
101 | 9. Click on "*Add*" button
102 | 10. Done!
--------------------------------------------------------------------------------
/tutorials/website/social-networking/pinterest-rich-pins-basics.md:
--------------------------------------------------------------------------------
1 | # Pinterest Rich Pins basics
2 |
3 | ## About:
4 |
5 | __Description by Pinterest:__
6 | > Rich Pins show metadata right on the Pin itself, giving Pinners a richer experience and increasing engagement. Information in a Rich Pin is independent of the Pin description, ensuring that important information is always tied to the Pin. There are 4 types of Rich Pins: app, article, product and recipe Pins.
7 | >
8 | > Rich Pins work by displaying metadata from marked up pages on your website. Once you've applied for Rich Pins, any content on your site with metadata will turn into a Rich Pin when a user saves it.
9 |
10 | Needs to be mentioned: ["Structured Data"](https://github.com/veliovgroup/ostrio/blob/master/tutorials/website/social-networking/structured-data-basics.md) and [Open Graph](https://github.com/veliovgroup/ostrio/blob/master/tutorials/website/social-networking/open-graph-basics.md) is supported by Pinterest too, basically Pinterest Rich Pins is based on combination of Open Graph and "Structured Data" with some extra tags extension `article:`.
11 |
12 | ## Basic structure:
13 |
14 | This example is very basic usage, but will cover most of the cases. For more info and examples - *see links below*.
15 |
16 | ```html
17 |
18 |
19 |
20 |
21 |
22 | Traveler: Norway
23 |
24 |
25 |
26 |
27 |
28 |
29 |
30 |
31 |
32 |
33 |
34 |
35 |
36 |
37 |
38 |
39 |
40 |
41 |
42 |
43 |
44 |
45 |
46 |
47 |
48 |
49 |
50 |
51 |
52 |
53 |
54 |
55 | ```
56 |
57 | ## Further reading:
58 |
59 | - [Instructions by Pinterest](https://help.pinterest.com/en/articles/enable-rich-pins-your-site)
60 | - [Pinterest Pins Validator](https://developers.pinterest.com/tools/url-debugger/)
61 | - [Docs by Pinterest](https://developers.pinterest.com/docs/rich-pins/articles/)
62 | - [Specs by Pinterest](https://developers.pinterest.com/docs/rich-pins/reference/)
63 |
--------------------------------------------------------------------------------
/tutorials/mongodb/use-ssl-http-encryption.md:
--------------------------------------------------------------------------------
1 | SSL/TLS/HTTPS Encrypted connection
2 | ======
3 |
4 | Assuming you have SSL/TLS certificate and private key, if not [read how to generate one for free](https://github.com/veliovgroup/ostrio/blob/master/tutorials/ssl/ssl-letsencrypt.md). All examples below for case when certificate, chain and private key at `/etc/nginx/ssl/` directory.
5 |
6 | ### 1. Get Root Certificate
7 | Get one [from here](https://www.identrust.com/certificates/trustid/root-download-x3.html), or copy-paste from below to `/etc/nginx/ssl/ca.crt`:
8 | ```plain
9 | -----BEGIN CERTIFICATE-----
10 | MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
11 | MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
12 | DkRTVCBSb290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow
13 | PzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD
14 | Ew5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
15 | AN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O
16 | rz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq
17 | OLl5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b
18 | xiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw
19 | 7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD
20 | aeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
21 | HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG
22 | SIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69
23 | ikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr
24 | AvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz
25 | R8srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5
26 | JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo
27 | Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ
28 | -----END CERTIFICATE-----
29 |
30 | ```
31 | __Note: new line "\n" at the end of the file is important__
32 |
33 | Append let's encrypt chain:
34 | ```shell
35 | # At /etc/nginx/ssl/ directory
36 | # cd /etc/nginx/ssl/
37 | cat chain.pem >> ca.crt
38 | ```
39 |
40 | ### 2. Concatenate Key and Certificate
41 | ```shell
42 | # Go to folder with certificates
43 | # for example: /etc/nginx/ssl/
44 | cat privkey.pem cert.pem > mongodb.pem
45 | ```
46 |
47 | Verify:
48 | ```shell
49 | openssl verify -CAfile ca.crt mongodb.pem
50 | ```
51 |
52 | ```shell
53 | # Move concatenated cert & chain to
54 | # mongodb owned directory,
55 | # for example to: /data/mongo/
56 | mv /etc/nginx/ssl/mongodb.pem /data/mongo/
57 | mv /etc/nginx/ssl/ca.crt /data/mongo/
58 |
59 | # Secure files:
60 | chown mongodb:mongodb /data/mongo/mongodb.pem
61 | chmod 400 /data/mongo/mongodb.pem
62 | chown mongodb:mongodb /data/mongo/ca.crt
63 | chmod 400 /data/mongo/ca.crt
64 | ```
65 |
66 | ### 3. Edit MongoDB Configuration file
67 | Assuming main mongo configuration at `/etc/mongod.conf`:
68 | ```shell
69 | # nano /etc/mongod.conf
70 | net:
71 | ssl:
72 | mode: requireSSL
73 | PEMKeyFile: /data/mongo/mongodb.pem
74 | CAFile: /data/mongo/ca.crt
75 | ```
76 |
77 | Note: same can be accomplished with `mongod` command flags:
78 | ```shell
79 | mongod --sslMode requireSSL --sslPEMKeyFile
80 | ```
81 |
82 | Check logs for errors:
83 |
84 | ```shell
85 | tail -n 50 -f /var/log/mongodb/mongod.log
86 | # ctrl + c to exit tail command
87 | ```
88 |
89 | ### Connect:
90 |
91 | ```shell
92 | mongo mongodb://[hostname]:[PORT]/ --ssl --sslPEMKeyFile=/etc/nginx/ssl/mongodb.pem
93 | ```
94 |
95 | ### Connect in app (node.js):
96 |
97 | ```js
98 | // Note: File "mongodb.pem" must accessible
99 | // make sure parent folder and file has 404 or higher permissions
100 |
101 | const fs = require('fs');
102 | MongoClient.connect('mongodb://[hostname]:[PORT]?ssl=true', {
103 | ssl: true,
104 | sslKey: fs.readFileSync('/path/to/mongodb.pem'),
105 | sslCert: fs.readFileSync('/path/to/mongodb.pem'),
106 | sslValidate: true
107 | }, (mongoErr, db) => {
108 | /* .... */
109 | });
110 | ```
111 |
112 | ## Further reading:
113 |
114 | - [gist](https://gist.github.com/leommoore/1e773a7d230ca4bbe1c2) by @leommoore
115 | - Read [official mongodb reference](https://docs.mongodb.com/manual/tutorial/configure-ssl/) for more info.
116 |
--------------------------------------------------------------------------------
/docs/analytics/track-amp.md:
--------------------------------------------------------------------------------
1 | # Track AMP pages
2 |
3 | [AMP project](https://www.ampproject.org/) aims to speed up and enhance the user experience, and it's open source! API and markup are minimal and strict, and not easy to find a way to track and build analytics from views of AMP-powered pages.
4 |
5 | Below we suggest using custom events of ostr.io web analytics to track views on Accelerated Mobile Pages (AMP) within `amp-img` tag:
6 |
7 | ```html
8 |
9 |
10 |
11 |
12 |
13 |
14 |
15 |
16 | Title
17 |
18 |
19 |
20 |
21 |
22 |
23 |
Title
24 |
Headline, sub-title
25 |
26 |
27 |
....Main Text Here....
28 |
29 |
30 |
31 |
32 |
33 |
41 |
42 |
43 | ```
44 |
45 | As alternative (*or addition*) to track a visit only, without reporting to custom events use following snippet:
46 |
47 | ```html
48 |
49 |
50 |
51 |
52 |
53 |
54 |
55 |
56 |
57 |
58 |
66 |
67 |
68 | ```
69 |
70 | Track page visit and track event when visitor reach end of the AMP page:
71 |
72 | ```html
73 |
74 |
75 |
76 |
77 |
78 |
79 |
80 |
81 |
82 |
90 |
91 |
92 |
93 |
94 |
102 |
103 |
104 | ```
105 |
106 | ## Further reading:
107 |
108 | - [web-analytics docs](https://github.com/veliovgroup/ostrio-analytics#analytics-for-ostrio)
109 |
--------------------------------------------------------------------------------
/tutorials/linux/security/use-ssh-keys.md:
--------------------------------------------------------------------------------
1 | # SSH key-based authentication
2 |
3 | ## Why?
4 |
5 | - No chance to brute-force SSH service if password-based login is disabled
6 | - No need to type-in password on every login via SSH
7 |
8 | ## Disadvantages:
9 |
10 | - No way to login if key-file is lost
11 | - No way to login if passphrase from key-file is forgotten
12 |
13 | ## Create SSH keys
14 |
15 | On __local__ linux-based (mac or linux) machine:
16 |
17 | ```shell
18 | ssh-keygen
19 |
20 | # "Enter file in which to save the key"
21 | # it's a good idea to have separate
22 | # keys for every host.
23 | # Create a key with the name related to host
24 | # ~/.ssh/my-app-proj
25 |
26 | # "Enter passphrase"
27 | # Skip "empty for no passphrase"
28 | # or set passphrase to encrypt key
29 | # so, even if it's leaked or stolen
30 | # it will be useless without passphrase
31 | ```
32 |
33 | ### Add to keychain (*mac os*)
34 |
35 | ```shell
36 | # To add keys permanently, use -K flag
37 |
38 | ssh-add -K ~/.ssh/my-app-proj
39 |
40 | # If the key file is created with the passphrase
41 | # You will be asked to enter it
42 | ```
43 |
44 | ### Add key to server
45 |
46 | On __local__ machine copy __public__ key:
47 |
48 | ```shell
49 | # Copy contents of ~/.ssh/my-app-proj.pub
50 | cat ~/.ssh/my-app-proj.pub
51 | ```
52 |
53 | On a Server, to login as root (*less secure*):
54 |
55 | ```shell
56 | # Paste copied public key to new line at:
57 | nano ~/.ssh/authorized_keys
58 | # Create file/folder if not exists
59 | # Save file with: "ctrl + o" combination
60 | # Close file with: "ctrl + x" combination
61 | ```
62 |
63 | *Skip this step if root user will be used for login*. On a Server, to login as any other user (*secure*, [read how to create a user](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/users/create-user.md))
64 |
65 | __Note:__ Do not forget to change `username` to actual users' name below:
66 |
67 | ```shell
68 | # Go to users' home directory
69 | cd /home/username
70 |
71 | # Create .ssh directory (if not yet exists)
72 | mkdir -p /home/username/.ssh
73 |
74 | # Paste copied public key to the new line at:
75 | nano /home/username/.ssh/authorized_keys
76 | # Save file with: "ctrl + o" combination
77 | # Close file with: "ctrl + x" combination
78 |
79 | # Set right permissions
80 | chmod -R 700 /home/username/.ssh
81 | chmod -R 640 /home/username/.ssh/authorized_keys
82 | chown -R username:username /home/username/.ssh
83 | ```
84 |
85 | ### Disable password auth (extra security)
86 |
87 | If SSH keys are 100% secure and won't be lost. Disabling password authentication would add an extra layer of security:
88 |
89 | ```shell
90 | # nano /etc/ssh/sshd_config
91 | # find the line with PasswordAuthentication
92 | # remove "#" if presented at the beginning of the line
93 |
94 | PasswordAuthentication no
95 | ```
96 |
97 | Restart SSH service:
98 |
99 | ```shell
100 | service sshd restart
101 | ```
102 |
103 | ### Issue with multiple keys (mac os)
104 |
105 | Having multiple keys in the system keychain would result SSH command to try login with each of added "identity file" keys in alphabet order. By default, Linux server will accept only 5 login attempts per connection, so having more than 5 SSH keys in the keychain will result in rejected connection. To solve this issue use `-i` flag to explicitly set "identity file":
106 |
107 | ```shell
108 | ssh -i ~/.ssh/my-app-proj username@host -o PubkeyAuthentication=no
109 | ```
110 |
111 | ### Login with password (mac os)
112 |
113 | To force login with password, use `-o PubkeyAuthentication=no` and/or `-o PreferredAuthentications="password"` flags:
114 |
115 | ```shell
116 | ssh username@host -o PubkeyAuthentication=no
117 | # or
118 | ssh username@host -o PreferredAuthentications="password"
119 | # or use both flags:
120 | ssh username@host -o PreferredAuthentications="password" -o PubkeyAuthentication=no
121 | ```
122 |
123 | "Too many authentication failures" error can get fixed with `-o IdentitiesOnly=yes` flag:
124 |
125 | ```shell
126 | ssh username@host -o IdentitiesOnly=yes
127 | ```
128 |
129 | ### Further reading:
130 |
131 | - [SSH Key-Based Authentication by DigitalOcean](https://www.digitalocean.com/community/tutorials/how-to-configure-ssh-key-based-authentication-on-a-linux-server)
132 | - [Change Default SSH port](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/security/change-ssh-port.md)
133 | - [Disable root login via SSH](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/security/disable-ssh-root.md)
134 | - [Sample `sshd_config`](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/security/sshd_config)
135 |
--------------------------------------------------------------------------------
/docs/monitoring/snmp.md:
--------------------------------------------------------------------------------
1 | # Monitoring via SNMP
2 |
3 | ## Why SNMP?
4 |
5 | Monitoring via SNMP is beneficial for double durability. While the server is down, build-in server's tools for notifying the administrator about high server usage, low HDD space, fraud logins, and connectivity issues may not work. With double durability in mind, — [ostr.io service](https://ostr.io) check servers from its end, from outside of local network and infrastructure, independently.
6 |
7 | > It's better to receive notifications from multiple sources, than none.
8 |
9 | SNMP protocol is capable of collecting data about active SSH sessions and logins, server's uptime, high memory (RAM) and CPU consumption, low storage space, high network traffic.
10 |
11 | Sever monitoring requirements via SNMP-endpoint:
12 |
13 | - Installed and running SNMPv2c service (*on a server*);
14 | - Public "Community" with random name (*password*).
15 |
16 | Collected data and its OIDs (*allow/deny any of list below to fit your needs*):
17 |
18 | - Host uptime (`1.3.6.1.2.1.25.1.1.0`)
19 | - SSH logins (`1.3.6.1.2.1.25.1.5.0`)
20 | - System processes (`1.3.6.1.2.1.25.4.2.1.7`)
21 | - System load (`1.3.6.1.4.1.2021.10.1.3`)
22 | - CPU utilization (`1.3.6.1.2.1.25.3.3.1.2`)
23 | - HDD/SSD/RAM/Swap names (`1.3.6.1.2.1.25.2.3.1.3`)
24 | - HDD/SSD/RAM/Swap capacity (`1.3.6.1.2.1.25.2.3.1.5`)
25 | - HDD/SSD/RAM/Swap usage (`1.3.6.1.2.1.25.2.3.1.6`)
26 | - Network Interfaces info (`1.3.6.1.2.1.31.1.1.1`, `1.3.6.1.2.1.2.2.1`)
27 |
28 | ## Install and configure SNMP
29 |
30 | All installation and setup instructions and examples are for Debian/Ubuntu Linux distributions. Instructions for other systems are very similar.
31 |
32 | ### Installation
33 |
34 | ```shell
35 | # Install snmpd
36 | apt-get update
37 | apt-get install snmpd
38 |
39 | # Edit configuration file:
40 | nano /etc/snmp/snmpd.conf
41 | ```
42 |
43 | ### Configuration
44 |
45 | __Do not forget to change value in between `<` and `>` symbols to your own value!__
46 |
47 | ```bash
48 | # /etc/snmp/snmpd.conf
49 | # Change 161 to any other random port for higher security
50 | agentAddress udp:161,udp6:[::1]:161
51 |
52 | # sec.name source community
53 | com2sec ReadUser default
54 |
55 | # sec.model sec.name
56 | group ReadGroup v2c ReadUser
57 |
58 | # incl/excl subtree
59 | view ReadData included .1.3.6.1.2.1.25.1.1.0
60 | view ReadData included .1.3.6.1.2.1.25.1.5.0
61 | view ReadData included .1.3.6.1.2.1.25.4.2.1.7
62 | view ReadData included .1.3.6.1.4.1.2021.10.1.3
63 | view ReadData included .1.3.6.1.2.1.25.3.3.1.2
64 | view ReadData included .1.3.6.1.2.1.25.2.3.1.3
65 | view ReadData included .1.3.6.1.2.1.25.2.3.1.5
66 | view ReadData included .1.3.6.1.2.1.25.2.3.1.6
67 | view ReadData included .1.3.6.1.2.1.31.1.1.1
68 | view ReadData included .1.3.6.1.2.1.2.2.1
69 |
70 | # context model level prefix read write notify
71 | access ReadGroup "" any noauth exact ReadData none none
72 | ```
73 |
74 | ### Edit SNMP daemon settings:
75 |
76 | ```shell
77 | nano /etc/default/snmpd
78 | ```
79 |
80 | ```bash
81 | # /etc/default/snmpd
82 | # This file controls the activity of snmpd
83 | # Don't load any MIBs by default.
84 | #export MIBS=/usr/share/snmp/mibs
85 |
86 | # snmpd control (yes means start daemon).
87 | SNMPDRUN=yes
88 |
89 | # snmptrapd control (yes means start daemon).
90 | TRAPDRUN=no
91 |
92 | # snmpd options
93 | SNMPDOPTS='-Lsd -Lf /dev/null -u snmp -g snmp -I -smux,mteTrigger,mteTriggerConf -p /run/snmpd.pid'
94 | ```
95 |
96 | ### Restart SNMP to apply changes
97 |
98 | ```shell
99 | service snmpd restart
100 | ```
101 |
102 | ### Check SNMP daemon status
103 |
104 | ```shell
105 | service snmpd status # should return OK (active (running))
106 | ```
107 |
108 | ## Add SNMP endpoint at [ostr.io](https://ostr.io):
109 |
110 | 1. Go to [Monitoring](https://ostr.io/service/monitoring)
111 | 2. Click on "*Add Endpoint*"
112 | 3. To select server type-in its name, and hit Enter
113 | 4. Select "*SNMP*" in "*Protocol*" drop-down list
114 | 5. Select desired "*Check Frequency*". We recommend to add same SNMP endpoint twice - one with high frequency (*less than 5 mins*), and second with low frequency (1-4 times per day). This will give the better overview of server resources usage
115 | 6. Enter "*SNMP port*" (from `/etc/snmp/snmpd.conf` file)
116 | 7. Enter "*SNMP Community*" (a.k.a. `password` from `/etc/snmp/snmpd.conf` file)
117 | 8. Click on "*Add*" button
118 | 9. Done!
119 |
--------------------------------------------------------------------------------
/tutorials/README.md:
--------------------------------------------------------------------------------
1 | # Tutorials
2 |
3 | Set of tutorials dedicated to improvements and optimizations of MongoDB, SSL/TLS certificates, Web Development, SEO, Linux.
4 |
5 | ## SSL/TLS certificates Tutorials
6 |
7 | - [Free letsencrypt certificates](https://github.com/veliovgroup/ostrio/blob/master/tutorials/ssl/ssl-letsencrypt.md)
8 | - [Verify letsencrypt certificate via DNS](https://github.com/veliovgroup/ostrio/blob/master/tutorials/ssl/ssl-letsencrypt-dns-validation.md)
9 |
10 | ## Web Development Tutorials
11 |
12 | - Social Networks Integration:
13 | - [Open Graph basics](https://github.com/veliovgroup/ostrio/blob/master/tutorials/website/social-networking/open-graph-basics.md) - For Facebook and other platforms with Open Graph support
14 | - [Pinterest Rich Pins basics](https://github.com/veliovgroup/ostrio/blob/master/tutorials/website/social-networking/pinterest-rich-pins-basics.md)
15 | - [Structured Data basics](https://github.com/veliovgroup/ostrio/blob/master/tutorials/website/social-networking/structured-data-basics.md) - Schema.org for Google rich snippets
16 | - [Twitter Card basics](https://github.com/veliovgroup/ostrio/blob/master/tutorials/website/social-networking/twitter-cards-basics.md) - Nice shared URLs with image, title and text preview
17 |
18 | ## Linux Tutorials
19 |
20 | - [Change timezone](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/change-timezone.md)
21 | - [`.bash_profile` tuning](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/bash_profile-tuning.md)
22 | - Users:
23 | - [Change root password](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/users/root-passwd.md)
24 | - [Change user password](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/users/user-passwd.md)
25 | - [Create new system user](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/users/create-user.md)
26 | - [rbash - restricted users](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/users/rbash.md)
27 | - Security:
28 | - [Change Default SSH port](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/security/change-ssh-port.md)
29 | - [Disable SSH login as root](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/security/disable-ssh-root.md)
30 | - [Disable SSH login with password (passwordless authentication), use SSH Keys](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/security/use-ssh-keys.md)
31 | - [Use RKHunter (Rootkit Hunter) to scan OS for vulnerabilities](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/security/rootkit-hunter.md)
32 | - [Use Lynis for security audit](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/security/lynis-security-audit.md)
33 | - [Basic firewall with `iptables`](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/security/iptables-firewall.md)
34 | - [Basic ipv6 firewall with `ip6tables`](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/security/iptables-firewall-ipv6.md)
35 | - [Scheduled and Automated Backups](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/security/automated-backups.md)
36 |
37 | ## MongoDB Tutorials
38 |
39 | - Security:
40 | - [Use SSL certificates](https://github.com/veliovgroup/ostrio/blob/master/tutorials/mongodb/use-ssl-http-encryption.md)
41 | - Performance & Maintenance:
42 | - [Create Single Server ReplicaSet](https://github.com/veliovgroup/ostrio/blob/master/tutorials/mongodb/single-server-replica-set.md)
43 | - [Create Multiple (Custer) Server ReplicaSet](https://github.com/veliovgroup/ostrio/blob/master/tutorials/mongodb/multiple-server-replica-set.md)
44 | - [Use OpLog](https://github.com/veliovgroup/ostrio/blob/master/tutorials/mongodb/enable-oplog.md)
45 |
46 |
58 |
60 |
61 |
62 |
64 |
65 |
66 |
67 |
71 |
72 |
76 |
77 |
83 |
84 |
86 |
87 |
90 |
91 |
--------------------------------------------------------------------------------
/tutorials/ssl/ssl-letsencrypt-dns-validation.md:
--------------------------------------------------------------------------------
1 | Let's Encrypt / Validation via DNS
2 | ======
3 |
4 | ### Alternative:
5 | - [Validation via HTTP](https://github.com/veliovgroup/ostrio/blob/master/tutorials/ssl/ssl-letsencrypt.md)
6 |
7 | In order to make World Wide Web safer and faster, we strongly recommend to use HTTPS protocol for your website, and add H2 protocol support to your web-server.
8 |
9 | This tutorial will show how to use [Let's Encrypt](https://letsencrypt.org) project. Which let's you to generate fully qualified SSL (TLS) certificate for free.
10 |
11 | ### Notes:
12 | - Examples is given for Debian/Ubuntu Linux and Nginx. Instructions for other platforms can be easily googled
13 | - We are not somehow affiliated with "Let's Encrypt", we just love to use this project. You can [support](https://letsencrypt.org/donate/) "Let's Encrypt" project if you like it too
14 |
15 |
16 | ### Install Certbot:
17 | Clone Certbot from its [GitHub repository](https://github.com/certbot/certbot)
18 | ```shell
19 | git clone https://github.com/certbot/certbot.git
20 | cd certbot
21 | ```
22 |
23 | ### Generate certificate:
24 | ```shell
25 | # --email admin@example.com <- Email for important notifications
26 | # -d example.com <- Domain name
27 | # --preferred-challenges dns <- DNS validation
28 | ./certbot-auto --email admin@example.com -d example.com -d www.example.com --manual --preferred-challenges dns certonly
29 |
30 | # This command will prompt you
31 | # to create TXT records
32 | # for each specified domain.
33 |
34 | # After domains is validated this command will return
35 | # path to directory with certificates
36 | # like: /etc/letsencrypt/live/example.com/
37 | ```
38 |
39 | ### Generate `dhparam`:
40 | ```shell
41 | # Go to /etc/nginx/ssl/
42 | mkdir -p /etc/nginx/ssl/
43 | cd /etc/nginx/ssl/
44 |
45 | # Generate dhparam
46 | # Note: it may take up to few hours
47 | openssl dhparam -out dhparam.pem 4096
48 | ```
49 |
50 | ### Update Nginx configuration:
51 | ```nginx
52 | # /etc/nginx/sites-available/example.conf
53 | server{
54 | listen 80;
55 | listen [::]:80;
56 | server_name example.com;
57 |
58 | # Redirect all requests to HTTPS
59 | return 301 https://$http_host$request_uri;
60 | }
61 |
62 | server {
63 | listen 443 ssl http2;
64 | listen [::]:443 ssl http2;
65 | server_name example.com;
66 |
67 | ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
68 | ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
69 | ssl_trusted_certificate /etc/letsencrypt/live/example.com/chain.pem;
70 |
71 | ssl on;
72 |
73 | # Everything below can be moved to nginx.conf
74 | # To apply to all hosts with SSL
75 |
76 | ssl_protocols TLSv1.3 TLSv1.2 TLSv1.1;
77 | ssl_ecdh_curve secp384r1;
78 | ssl_prefer_server_ciphers on;
79 | ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:RSA-AES128-SHA256:RSA-AES256-SHA256:RSA-AES128-SHA:RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA;
80 | ssl_session_cache shared:SSL:32m;
81 | ssl_session_timeout 4h;
82 | ssl_buffer_size 1400;
83 | ssl_session_tickets on;
84 | ssl_dhparam /etc/nginx/ssl/dhparam.pem;
85 |
86 | ssl_stapling on;
87 | ssl_stapling_verify on;
88 |
89 | # ... Other settings of your web-app
90 | }
91 | ```
92 |
93 | ### Enable example.conf:
94 | ```shell
95 | ln -s /etc/nginx/sites-available/example.conf /etc/nginx/sites-enabled/example.conf
96 | ```
97 |
98 | ### Enable http2 (H2):
99 | ```nginx
100 | # create or edit /etc/nginx/sites-available/default
101 | server {
102 | listen 443 ssl http2;
103 | listen [::]:443 ssl http2 ipv6only=on;
104 | server_name _;
105 |
106 | ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
107 | ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
108 | ssl_trusted_certificate /etc/letsencrypt/live/example.com/chain.pem;
109 |
110 | ssl on;
111 |
112 | return 444;
113 | }
114 | ```
115 |
116 | ### Set permissions:
117 | If you've been generating certificates remotely (*not on the server*).
118 | You may copy-paste all generated files to `/etc/nginx/ssl/` directory.
119 |
120 | To provide security after copying files - we should set owner and restrict access.
121 |
122 | Use `chown` to set files owner, usually `www-data` for Nginx
123 | ```shell
124 | # Run only if nginx operates under www-data user
125 | # chown -R www-data:www-data /etc/nginx
126 | chmod -R 644 /etc/nginx
127 | find /etc/nginx -type d -exec chmod 700 {} \;
128 | chmod -R 600 /etc/nginx/ssl
129 |
130 | # Run only if nginx operates under www-data user
131 | # chown -R www-data:www-data /var/www/example
132 | ```
133 |
134 | ### Test configuration:
135 | ```shell
136 | service nginx configtest
137 | nginx -t
138 | ```
139 |
140 | ### Restart Nginx to apply changes:
141 | ```shell
142 | service nginx restart
143 | ```
144 |
145 | ### Test SSL (TLS) setup:
146 | - Go to [ssllabs.com](https://www.ssllabs.com/ssltest/index.html)
147 | - Enter your domain
148 | - You should get A+ rating with this setup
149 |
--------------------------------------------------------------------------------
/docs/prerendering/apache.md:
--------------------------------------------------------------------------------
1 | # Pre-rendering via Apache htaccess
2 |
3 | Apache web server is efficient point to redirect requests from crawlers and bots to pre-rendering engine before it will reach the origin server
4 |
5 | ## Prerequisites
6 |
7 | Enable required modules via `.htaccess` file or `CLI` commands
8 |
9 | ### Enable required modules via htaccess
10 |
11 | Add the next lines into `.htaccess` file:
12 |
13 | ```apacheconf
14 | LoadModule headers_module libexec/apache2/mod_headers.so
15 | LoadModule proxy_module libexec/apache2/mod_proxy.so
16 | LoadModule proxy_http_module libexec/apache2/mod_proxy_http.so
17 | LoadModule ssl_module libexec/apache2/mod_ssl.so
18 | LoadModule rewrite_module libexec/apache2/mod_rewrite.so
19 | ```
20 |
21 | ### Enable required modules via CLI
22 |
23 | Run the next commands in CLI/terminal/shell:
24 |
25 | ```bash
26 | sudo a2enmod headers
27 | sudo a2enmod proxy
28 | sudo a2enmod proxy_http
29 | sudo a2enmod ssl
30 | sudo a2enmod rewrite
31 | ```
32 |
33 | ## Apache config
34 |
35 | Add the next `RewriteCond` and `RewriteRule` into existing `.htaccess` file to redirect bot's traffic to pre-rendering engine
36 |
37 | ```apacheconf
38 | # IF SSL MANAGED BY APACHE, ADD NEXT LINE:
39 | # SSLProxyEngine on
40 |
41 |
42 | ########
43 | # Do not forget to change _YOUR_AUTH_TOKEN_ to token you get from ostr.io
44 | ########
45 | RequestHeader set Authorization "Basic _YOUR_AUTH_TOKEN_"
46 | RequestHeader set User-Agent "%{User-Agent}i"
47 |
48 |
49 |
50 | RewriteEngine On
51 |
52 | RewriteCond %{REQUEST_URI} !\.(?:3ds|3g2|3gp|3gpp|7z|a|aac|aaf|adp|ai|aif|aiff|alz|ape|apk|appcache|ar|arj|asf|asx|atom|au|avchd|avi|bak|bbaw|bh|bin|bk|bmp|btif|bz2|bzip2|cab|caf|cco|cgm|class|cmx|cpio|cr2|crt|crx|css|csv|cur|dat|deb|der|dex|djvu|dll|dmg|dng|doc|docm|docx|dot|dotm|dra|drc|DS_Store|dsk|dts|dtshd|dvb|dwg|dxf|ear|ecelp4800|ecelp7470|ecelp9600|egg|eol|eot|eps|epub|exe|f4a|f4b|f4p|f4v|fbs|fh|fla|flac|fli|flv|fpx|fst|fvt|g3|geojson|gif|graffle|gz|gzip|h261|h263|h264|hqx|htc|ico|ief|img|ipa|iso|jad|jar|jardiff|jng|jnlp|jpeg|jpg|jpgv|jpm|js|jxr|key|kml|kmz|ktx|less|lha|lvp|lz|lzh|lzma|lzo|m2v|m3u|m4a|m4p|m4v|map|manifest|mar|markdown|md|mdi|mdown|mdwn|mht|mid|midi|mj2|mka|mkd|mkdn|mkdown|mkv|mml|mmr|mng|mobi|mov|movie|mp2|mp3|mp4|mp4a|mpe|mpeg|mpg|mpga|mpv|msi|msm|msp|mxf|mxu|nef|npx|nsv|numbers|o|oex|oga|ogg|ogv|opus|otf|pages|pbm|pcx|pdb|pdf|pea|pem|pgm|pic|pl|pm|png|pnm|pot|potm|potx|ppa|ppam|ppm|pps|ppsm|ppsx|ppt|pptm|pptx|prc|ps|psd|pya|pyc|pyo|pyv|qt|ra|rar|ras|raw|rdf|rgb|rip|rlc|rm|rmf|rmvb|ron|roq|rpm|rss|rtf|run|rz|s3m|s7z|safariextz|scpt|sea|sgi|shar|sil|sit|slk|smv|so|sub|svg|svgz|svi|swf|tar|tbz|tbz2|tcl|tga|tgz|thmx|tif|tiff|tk|tlz|topojson|torrent|ttc|ttf|txt|txz|udf|uvh|uvi|uvm|uvp|uvs|uvu|vcard|vcf|viv|vob|vtt|war|wav|wax|wbmp|wdp|weba|webapp|webm|webmanifest|webp|whl|wim|wm|wma|wml|wmlc|wmv|wmx|woff|woff2|wvx|xbm|xif|xla|xlam|xloc|xls|xlsb|xlsm|xlsx|xlt|xltm|xltx|xm|xmind|xpi|xpm|xsl|xwd|xz|yuv|z|zip|zipx)$ [NC]
53 |
54 | RewriteCond "%{HTTP_USER_AGENT}" "\.net crawler|360spider|50\.nu|8bo crawler bot|aboundex|accoona|adldxbot|ahrefsbot|altavista|appengine-google|applebot|archiver|arielisbot|ask jeeves|auskunftbot|baidumobaider|baiduspider|becomebot|bingbot|bingpreview|bitbot|bitlybot|blitzbot|blogbridge|boardreader|botseer|catchbot|catchpoint bot|charlotte|checklinks|cliqzbot|clumboot|coccocbot|converacrawler|crawl-e|crawlconvera|dataparksearch|daum|deusu|discordbot|dotbot|duckduckbot|elefent|embedly|evernote|exabot|facebookbot|facebookexternalhit|meta-external|fatbot|fdse robot|feed seeker bot|feedfetcher|femtosearchbot|findlinks|flamingo_searchengine|flipboard|followsite bot|furlbot|fyberspider|gaisbot|galaxybot|geniebot|genieo|gigablast|gigabot|girafabot|gomezagent|gonzo1|googlebot|google sketchup|adsbot-google|google-structured-data-testing-tool|google-extended|developers\.google\.com/+/web/snippet|haosouspider|heritrix|holmes|hoowwwer|htdig|ia_archiver|idbot|infuzapp|innovazion crawler|instagram|internetarchive|iqdb|iskanie|istellabot|izsearch\.com|kaloogabot|kaz\.kz_bot|kd bot|konqueror|kraken|kurzor|larbin|leia|lesnikbot|linguee bot|linkaider|linkapediabot|linkedinbot|lite bot|llaut|lookseek|lycos|mail\.ru_bot|masidani_bot|masscan|mediapartners-google|metajobbot|mj12bot|mnogosearch|mogimogi|mojeekbot|motominerbot|mozdex|msiecrawler|msnbot|msrbot|netpursual|netresearch|netvibes|newsgator|ng-search|nicebot|nutchcvs|nuzzel|nymesis|objectssearch|odklbot|omgili|oovoo|oozbot|openfosbot|orangebot|orbiter|org_bot|outbrain|pagepeeker|pagesinventory|parsijoobot|paxleframework|peeplo screenshot bot|pinterest|plantynet_webrobot|plukkie|pompos|psbot|quora link preview|qwantify|read%20later|reaper|redcarpet|redditbot|retreiver|riddler|rival iq|rogerbot|saucenao|scooter|scrapy|scrubby|searchie|searchsight|seekbot|semanticdiscovery|seznambot|showyoubot|simplepie|simpy|sitelockspider|skypeuripreview|slackbot|slack-imgproxy|slurp|snappy|sogou|solofield|speedyspider|speedy spider|sputnikbot|stackrambler|teeraidbot|teoma|theusefulbot|thumbshots\.ru|thumbshotsbot|tineye|toweya\.com|toweyabot|tumblr|tweetedtimes|tweetmemebot|twitterbot|url2png|vagabondo|vebidoobot|viber|visionutils|vkshare|voilabot|vortex|votay bot|voyager|w3c_validator|wasalive\.bot|web-sniffer|websquash\.com|webthumb|whatsapp|whatweb|wire|wotbox|yacybot|yahoo|yandex|yeti|yisouspider|yodaobot|yooglifetchagent|yoozbot|yottaamonitor|yowedo|zao-crawler|zebot_www\.ze\.bz|zooshot|zyborgi|ai2bot|amazonbot|anthropic\.com|bard|bytespider|ccbot|chatgpt-user|claude-web|claudebot|cohere-ai|deepseek|diffbot|duckassistbot|gemini|gptbot|grok|mistralai|oai-searchbot|omgili|openai\.com|perplexity\.ai|perplexitybot|xai|youbot" [NC,OR]
55 |
56 | RewriteCond "%{QUERY_STRING}" "_escaped_fragment_" [NC]
57 |
58 | RewriteRule ^(.*)$ "https://render.ostr.io/render/%{REQUEST_SCHEME}://%{HTTP_HOST}/$1" [P, END]
59 |
60 |
61 | ```
62 |
--------------------------------------------------------------------------------
/tutorials/ssl/ssl-letsencrypt.md:
--------------------------------------------------------------------------------
1 | # Let's Encrypt / Free SSL (TLS) certificates
2 |
3 | In this tutorial we use HTTP for domain ownership validation, as alternative see:
4 |
5 | - [Validation via DNS](https://github.com/veliovgroup/ostrio/blob/master/tutorials/ssl/ssl-letsencrypt-dns-validation.md)
6 |
7 | In order to make World Wide Web safer and faster, we strongly recommend to use HTTPS protocol for your website, and add H2 protocol support to your web-server.
8 |
9 | This tutorial will show how to use [Let's Encrypt](https://letsencrypt.org) project. Which let's you to generate fully qualified SSL (TLS) certificate for free.
10 |
11 | ## Notes:
12 |
13 | - Examples is given for Debian/Ubuntu Linux and Nginx. Instructions for other platforms can be easily googled
14 | - We are not somehow affiliated with "Let's Encrypt", we just love to use this project. You can [support](https://letsencrypt.org/donate/) "Let's Encrypt" project if you like it too
15 |
16 | ## Install Certbot:
17 |
18 | Clone Certbot from its [GitHub repository](https://github.com/certbot/certbot)
19 |
20 | ```shell
21 | git clone https://github.com/certbot/certbot.git
22 | cd certbot
23 | ```
24 |
25 | ## Set Nginx configuration:
26 |
27 | ```nginx
28 | # /etc/nginx/sites-available/example.conf
29 | server{
30 | listen 80;
31 | listen [::]:80;
32 | server_name example.com;
33 | root /var/www/example;
34 | location /.well-known/acme-challenge/ {
35 | try_files $uri =404;
36 | }
37 | }
38 | ```
39 |
40 | ## Make sure root exists:
41 |
42 | ```shell
43 | mkdir -p /var/www/example
44 | ```
45 |
46 | ## Generate certificate:
47 |
48 | ```shell
49 | # --email admin@example.com <- Email for important notifications
50 | # -w /var/www/example/ <- Nginx host "root"
51 | # -d example.com <- Domain name
52 | ./certbot-auto certonly --email admin@example.com --webroot -w /var/www/example/ -d example.com -d www.example.com
53 |
54 | # This command will return
55 | # path to directory with certificates
56 | # like: /etc/letsencrypt/live/example.com/
57 | ```
58 |
59 | ## Generate `dhparam`:
60 |
61 | ```shell
62 | # Go to /etc/nginx/ssl/
63 | mkdir -p /etc/nginx/ssl/
64 | cd /etc/nginx/ssl/
65 |
66 | # Generate dhparam
67 | # Note: it may take up to few hours
68 | openssl dhparam -out dhparam.pem 4096
69 | ```
70 |
71 | ## Update Nginx configuration:
72 |
73 | ```nginx
74 | # /etc/nginx/sites-available/example.conf
75 | server{
76 | listen 80;
77 | listen [::]:80;
78 | server_name example.com;
79 |
80 | # Redirect all requests to HTTPS
81 | return 301 https://$http_host$request_uri;
82 | }
83 |
84 | server {
85 | listen 443 ssl http2;
86 | listen [::]:443 ssl http2;
87 | server_name example.com;
88 |
89 | ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
90 | ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
91 | ssl_trusted_certificate /etc/letsencrypt/live/example.com/chain.pem;
92 |
93 | ssl on;
94 |
95 | # Everything below can be moved to nginx.conf
96 | # To apply to all hosts with SSL
97 |
98 | ssl_protocols TLSv1.3 TLSv1.2 TLSv1.1;
99 | ssl_ecdh_curve secp384r1;
100 | ssl_prefer_server_ciphers on;
101 | ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:RSA-AES128-SHA256:RSA-AES256-SHA256:RSA-AES128-SHA:RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA;
102 | ssl_session_cache shared:SSL:32m;
103 | ssl_session_timeout 4h;
104 | ssl_buffer_size 1400;
105 | ssl_session_tickets on;
106 | ssl_dhparam /etc/nginx/ssl/dhparam.pem;
107 |
108 | ssl_stapling on;
109 | ssl_stapling_verify on;
110 |
111 | # ... Other settings of your web-app
112 | }
113 | ```
114 |
115 | ## Enable example.conf:
116 |
117 | ```shell
118 | ln -s /etc/nginx/sites-available/example.conf /etc/nginx/sites-enabled/example.conf
119 | ```
120 |
121 | ## Enable http2 (H2):
122 |
123 | ```nginx
124 | # create or edit /etc/nginx/sites-available/default
125 | server {
126 | listen 443 ssl http2;
127 | listen [::]:443 ssl http2 ipv6only=on;
128 | server_name _;
129 |
130 | ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
131 | ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
132 | ssl_trusted_certificate /etc/letsencrypt/live/example.com/chain.pem;
133 |
134 | ssl on;
135 |
136 | return 444;
137 | }
138 | ```
139 |
140 | ## Set permissions:
141 |
142 | If you've been generating certificates remotely (*not on the server*).
143 | You may copy-paste all generated files to `/etc/nginx/ssl/` directory.
144 |
145 | To provide security after copying files - we should set owner and restrict access.
146 |
147 | Use `chown` to set files owner, usually `www-data` for Nginx
148 |
149 | ```shell
150 | # Run only if nginx operates under www-data user
151 | # chown -R www-data:www-data /etc/nginx
152 | chmod -R 644 /etc/nginx
153 | find /etc/nginx -type d -exec chmod 700 {} \;
154 | chmod -R 600 /etc/nginx/ssl
155 |
156 | # Run only if nginx operates under www-data user
157 | # chown -R www-data:www-data /var/www/example
158 | ```
159 |
160 | ## Test configuration:
161 |
162 | ```shell
163 | service nginx configtest
164 | nginx -t
165 | ```
166 |
167 | ## Restart Nginx to apply changes:
168 |
169 | ```shell
170 | service nginx restart
171 | ```
172 |
173 | ## Test SSL (TLS) setup:
174 |
175 | - Go to [ssllabs.com](https://www.ssllabs.com/ssltest/index.html)
176 | - Enter your domain
177 | - You should get A+ rating with this setup
178 |
--------------------------------------------------------------------------------
/tutorials/mongodb/single-server-replica-set.md:
--------------------------------------------------------------------------------
1 | # Single Server / Replica Set
2 |
3 | For more info see [original article](https://veliovgroup.com/article/2qsjtNf8NSB9XxZDh/mongodb-replica-set-with-oplog).
4 |
5 | We will use wiredTiger as engine, you can read [its release notes](https://www.mongodb.com/blog/post/whats-new-mongodb-30-part-3-performance-efficiency-gains-new-storage-architecture) to find out more about its benefits.
6 |
7 | ### Definitions:
8 |
9 | - `admin` user - User with full `root` access to MongoDB features and commands
10 | - `appUser` - User with `readWrite` access to database used for your application
11 | - `appDB` - Database used for application
12 | - `` - placeholder, should be changed to strong password. __Always placed in double quotes__
13 |
14 | ### 1. Installation:
15 | Follow steps described in [installation guide](https://docs.mongodb.com/manual/tutorial/install-mongodb-on-debian). During installation `mongodb` user in `mongodb` group will be created automatically
16 |
17 | ### 2. Create and set right permissions for storage folders:
18 | ```shell
19 | mkdir -p /data/mongos/one /data/mongos/two /data/mongos/three
20 | chmod 755 /data
21 | chown -R mongodb:mongodb /data/mongos
22 | ```
23 |
24 | ### 3. Create and set permissions for log folders:
25 | ```shell
26 | mkdir -p /var/log/mongodb/one /var/log/mongodb/two /var/log/mongodb/three
27 | chown -R mongodb:mongodb /var/log/mongodb
28 | ```
29 |
30 | ### 4. Remove `mongod` service scripts (*if exists*):
31 | ```shell
32 | rm /etc/init.d/mongod
33 | ```
34 |
35 | ### 5. Edit MongoDB configuration file:
36 | For first member of Replica Set (`nano /etc/mongod-one.conf`):
37 | ```yaml
38 | storage:
39 | dbPath: /data/mongos/one
40 | journal:
41 | enabled: true
42 | engine: wiredTiger
43 | systemLog:
44 | destination: file
45 | logAppend: true
46 | path: /var/log/mongodb/one/mongod.log
47 | net:
48 | port: 27017
49 | bindIp: 127.0.0.1
50 | replication:
51 | replSetName: rs0
52 | ```
53 |
54 | ### 6. Edit MongoDB configuration:
55 | For second member of Replica Set (`nano /etc/mongod-two.conf`):
56 | ```yaml
57 | storage:
58 | dbPath: /data/mongos/two
59 | journal:
60 | enabled: true
61 | engine: wiredTiger
62 | systemLog:
63 | destination: file
64 | logAppend: true
65 | path: /var/log/mongodb/two/mongod.log
66 | net:
67 | port: 27018
68 | bindIp: 127.0.0.1
69 | replication:
70 | replSetName: rs0
71 | ```
72 |
73 | ### 7. Edit MongoDB configuration:
74 | For third member of Replica Set (`nano /etc/mongod-three.conf`):
75 | ```yaml
76 | storage:
77 | dbPath: /data/mongos/three
78 | journal:
79 | enabled: true
80 | engine: wiredTiger
81 | systemLog:
82 | destination: file
83 | logAppend: true
84 | path: /var/log/mongodb/three/mongod.log
85 | net:
86 | port: 27019
87 | bindIp: 127.0.0.1
88 | replication:
89 | replSetName: rs0
90 | ```
91 |
92 | ### 8. Create shared auth key:
93 | For secure communication between Replica Set members ([more info](https://docs.mongodb.com/v3.2/tutorial/enforce-keyfile-access-control-in-existing-replica-set/)):
94 | ```shell
95 | openssl rand -base64 741 > /data/mongos/key
96 | chown mongodb:mongodb /data/mongos/key
97 | chmod 400 /data/mongos/key
98 | ```
99 |
100 | ### 9. Create CRON job
101 | To start MongoDB as `mongodb` user (`crontab -u mongodb -e`):
102 | ```cron
103 | @reboot /usr/bin/mongod --config /etc/mongod-one.conf --fork
104 | @reboot /usr/bin/mongod --config /etc/mongod-two.conf --fork
105 | @reboot /usr/bin/mongod --config /etc/mongod-three.conf --fork
106 | ```
107 |
108 | ### 10. Reboot the machine
109 |
110 | ### 11. Configure and initiate Replica Set:
111 | ```shell
112 | $ mongo # <-- Mongo Shell
113 | var conf = {
114 | "_id" : "rs0",
115 | "members" : [
116 | {
117 | "_id" : 0,
118 | "host" : "127.0.0.1:27017"
119 | },
120 | {
121 | "_id" : 1,
122 | "host" : "127.0.0.1:27018"
123 | },
124 | {
125 | "_id" : 2,
126 | "host" : "127.0.0.1:27019"
127 | }
128 | ]
129 | }
130 | rs.initiate(conf)
131 | ```
132 |
133 | ### 12. Define PRIMARY member:
134 | Next steps should be performed only on PRIMARY member. To find out which member is primary, run:
135 | ```shell
136 | # Mongo Shell
137 | rs.status()
138 | ```
139 |
140 | ### 13. On PRIMARY: Create `admin` user:
141 | With `root` privileges:
142 | ```shell
143 | # Mongo Shell:
144 | $ mongo --port 27017
145 | use admin
146 | db.createUser({user:"admin", pwd:, roles:[{role:"root", db:"admin"}]})
147 | ```
148 |
149 | ### 14. Update CRON job:
150 | To start MongoDB with `--auth` option (`crontab -u mongodb -e`) this will protect MongoDB from unauthorized access (for higher security read [this article](https://docs.mongodb.com/manual/administration/security-checklist/)):
151 | ```cron
152 | @reboot /usr/bin/mongod --config /etc/mongod-one.conf --auth --fork
153 | @reboot /usr/bin/mongod --config /etc/mongod-two.conf --auth --fork
154 | @reboot /usr/bin/mongod --config /etc/mongod-three.conf --auth --fork
155 | ```
156 |
157 | ### 15. Add next lines:
158 | To __each__ configuration of Replica Set members:
159 | ```yaml
160 | # nano /etc/mongod-one.conf
161 | # nano /etc/mongod-two.conf
162 | # nano /etc/mongod-three.conf
163 | security:
164 | keyFile: /data/mongos/key
165 | ```
166 |
167 | ### 16. Reboot the machine
168 |
169 | ### 17. Create application user:
170 | With `readWrite` permissions for db of your app (*you will use this user to access MongoDB from your application*):
171 | ```shell
172 | # Mongo Shell `mongo -u "admin" -p --authenticationDatabase "admin"`
173 | use admin
174 | db.createUser({user:"appUser", pwd:, roles:[{role:"readWrite", db:"appDB"}]})
175 | ```
176 |
177 | ### 19. Make sure all users are properly created:
178 | ```shell
179 | # Mongo Shell `mongo -u "admin" -p --authenticationDatabase "admin"`
180 | use admin
181 | show users
182 | ```
183 |
184 | From this point MongoDB is available on loopback and localhost.
185 |
186 | ### 20. Connection strings:
187 | ```plain
188 | mongodb://appUser:@127.0.0.1:27017,127.0.0.1:27018,127.0.0.1:27019/appDB?authSource=admin&replicaSet=rs0
189 | ```
190 |
191 | ### Further reading:
192 | - [Use and configure OpLog](https://github.com/veliovgroup/ostrio/blob/master/tutorials/mongodb/enable-oplog.md)
193 |
--------------------------------------------------------------------------------
/docs/prerendering/examples/nginx/phusion-passenger.conf:
--------------------------------------------------------------------------------
1 | map $http_user_agent $is_webbot {
2 | default 0;
3 | "~*(?:\.net crawler|360spider|50\.nu|8bo crawler bot|aboundex|accoona|adldxbot|ahrefsbot|altavista|appengine-google|applebot|archiver|arielisbot|ask jeeves|auskunftbot|baidumobaider|baiduspider|becomebot|bingbot|bingpreview|bitbot|bitlybot|blitzbot|blogbridge|boardreader|botseer|catchbot|catchpoint bot|charlotte|checklinks|cliqzbot|clumboot|coccocbot|converacrawler|crawl-e|crawlconvera|dataparksearch|daum|deusu|discordbot|dotbot|duckduckbot|elefent|embedly|evernote|exabot|facebookbot|facebookexternalhit|meta-external|fatbot|fdse robot|feed seeker bot|feedfetcher|femtosearchbot|findlinks|flamingo_searchengine|flipboard|followsite bot|furlbot|fyberspider|gaisbot|galaxybot|geniebot|genieo|gigablast|gigabot|girafabot|gomezagent|gonzo1|googlebot|google sketchup|adsbot-google|google-structured-data-testing-tool|google-extended|developers\.google\.com/+/web/snippet|haosouspider|heritrix|holmes|hoowwwer|htdig|ia_archiver|idbot|infuzapp|innovazion crawler|instagram|internetarchive|iqdb|iskanie|istellabot|izsearch\.com|kaloogabot|kaz\.kz_bot|kd bot|konqueror|kraken|kurzor|larbin|leia|lesnikbot|linguee bot|linkaider|linkapediabot|linkedinbot|lite bot|llaut|lookseek|lycos|mail\.ru_bot|masidani_bot|masscan|mediapartners-google|metajobbot|mj12bot|mnogosearch|mogimogi|mojeekbot|motominerbot|mozdex|msiecrawler|msnbot|msrbot|netpursual|netresearch|netvibes|newsgator|ng-search|nicebot|nutchcvs|nuzzel|nymesis|objectssearch|odklbot|omgili|oovoo|oozbot|openfosbot|orangebot|orbiter|org_bot|outbrain|pagepeeker|pagesinventory|parsijoobot|paxleframework|peeplo screenshot bot|pinterest|plantynet_webrobot|plukkie|pompos|psbot|quora link preview|qwantify|read%20later|reaper|redcarpet|redditbot|retreiver|riddler|rival iq|rogerbot|saucenao|scooter|scrapy|scrubby|searchie|searchsight|seekbot|semanticdiscovery|seznambot|showyoubot|simplepie|simpy|sitelockspider|skypeuripreview|slackbot|slack-imgproxy|slurp|snappy|sogou|solofield|speedyspider|speedy spider|sputnikbot|stackrambler|teeraidbot|teoma|theusefulbot|thumbshots\.ru|thumbshotsbot|tineye|toweya\.com|toweyabot|tumblr|tweetedtimes|tweetmemebot|twitterbot|url2png|vagabondo|vebidoobot|viber|visionutils|vkshare|voilabot|vortex|votay bot|voyager|w3c_validator|wasalive\.bot|web-sniffer|websquash\.com|webthumb|whatsapp|whatweb|wire|wotbox|yacybot|yahoo|yandex|yeti|yisouspider|yodaobot|yooglifetchagent|yoozbot|yottaamonitor|yowedo|zao-crawler|zebot_www\.ze\.bz|zooshot|zyborgi|ai2bot|amazonbot|anthropic\.com|bard|bytespider|ccbot|chatgpt-user|claude-web|claudebot|cohere-ai|deepseek|diffbot|duckassistbot|gemini|gptbot|grok|mistralai|oai-searchbot|omgili|openai\.com|perplexity\.ai|perplexitybot|xai|youbot)" 1;
4 | }
5 |
6 | # CHECK AND SET ESCAPED FRAGMENT
7 | # For more info read: https://developers.google.com/webmasters/ajax-crawling/docs/specification
8 | map $arg__escaped_fragment_ $fragment {
9 | "" "";
10 | default "/$arg__escaped_fragment_";
11 | }
12 |
13 | # FILTER OUT _escaped_fragment_ FROM GET QUERY
14 | map $args $filtered_args {
15 | ~(^|&)_escaped_fragment_=[^&]*&?(.*) $1$2;
16 | default $args;
17 | }
18 |
19 | server {
20 | listen 80;
21 | listen [::]:80;
22 | server_name example.com;
23 | root /path/to/public;
24 | passenger_app_root /path/to/public;
25 |
26 | passenger_enabled on;
27 | passenger_app_type node;
28 | passenger_startup_file index.js;
29 |
30 | location ~* \.(?:3ds|3g2|3gp|3gpp|7z|a|aac|aaf|adp|ai|aif|aiff|alz|ape|apk|appcache|ar|arj|asf|asx|atom|au|avchd|avi|bak|bbaw|bh|bin|bk|bmp|btif|bz2|bzip2|cab|caf|cco|cgm|class|cmx|cpio|cr2|crt|crx|css|csv|cur|dat|deb|der|dex|djvu|dll|dmg|dng|doc|docm|docx|dot|dotm|dra|drc|DS_Store|dsk|dts|dtshd|dvb|dwg|dxf|ear|ecelp4800|ecelp7470|ecelp9600|egg|eol|eot|eps|epub|exe|f4a|f4b|f4p|f4v|fbs|fh|fla|flac|fli|flv|fpx|fst|fvt|g3|geojson|gif|graffle|gz|gzip|h261|h263|h264|hqx|htc|ico|ief|img|ipa|iso|jad|jar|jardiff|jng|jnlp|jpeg|jpg|jpgv|jpm|js|jxr|key|kml|kmz|ktx|less|lha|lvp|lz|lzh|lzma|lzo|m2v|m3u|m4a|m4p|m4v|map|manifest|mar|markdown|md|mdi|mdown|mdwn|mht|mid|midi|mj2|mka|mkd|mkdn|mkdown|mkv|mml|mmr|mng|mobi|mov|movie|mp2|mp3|mp4|mp4a|mpe|mpeg|mpg|mpga|mpv|msi|msm|msp|mxf|mxu|nef|npx|nsv|numbers|o|oex|oga|ogg|ogv|opus|otf|pages|pbm|pcx|pdb|pdf|pea|pem|pgm|pic|pl|pm|png|pnm|pot|potm|potx|ppa|ppam|ppm|pps|ppsm|ppsx|ppt|pptm|pptx|prc|ps|psd|pya|pyc|pyo|pyv|qt|ra|rar|ras|raw|rdf|rgb|rip|rlc|rm|rmf|rmvb|ron|roq|rpm|rss|rtf|run|rz|s3m|s7z|safariextz|scpt|sea|sgi|shar|sil|sit|slk|smv|so|sub|svg|svgz|svi|swf|tar|tbz|tbz2|tcl|tga|tgz|thmx|tif|tiff|tk|tlz|topojson|torrent|ttc|ttf|txt|txz|udf|uvh|uvi|uvm|uvp|uvs|uvu|vcard|vcf|viv|vob|vtt|war|wav|wax|wbmp|wdp|weba|webapp|webm|webmanifest|webp|whl|wim|wm|wma|wml|wmlc|wmv|wmx|woff|woff2|wvx|xbm|xif|xla|xlam|xloc|xls|xlsb|xlsm|xlsx|xlt|xltm|xltx|xm|xmind|xml|xpi|xpm|xsl|xwd|xz|yuv|z|zip|zipx)$ {
31 | # UNCOMMENT THE NEXT 2 LINES TO ENABLE CACHING FOR STATIC FILES
32 | # expires 30d;
33 | # add_header Cache-Control "public";
34 |
35 | # REDUCE LOAD BY DISABLING LOGGING FOR STATIC FILES
36 | access_log off;
37 | log_not_found off;
38 | try_files $uri =404;
39 | }
40 |
41 | location / {
42 | try_files $uri @prerendering;
43 | }
44 |
45 | location @prerendering {
46 | internal;
47 | # SET RENDERING ENDPOINT
48 | # read more at: https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/rendering-endpoints.md
49 | set $renderer_domain "render.ostr.io";
50 | set $orig_uri $request_uri;
51 |
52 | if ($orig_uri ~ "^(.*)\?(.*)$") {
53 | set $orig_uri $1;
54 | }
55 |
56 | # DO NOT RENDER WEBSOCKETS AND OTHER STREAMS
57 | if ($http_upgrade) {
58 | set $is_webbot 0;
59 | }
60 |
61 | proxy_pass_request_headers off;
62 | proxy_ssl_server_name on;
63 | proxy_http_version 1.0;
64 | proxy_hide_header WWW-Authenticate;
65 | proxy_hide_header Set-Cookie;
66 | proxy_set_header Host $renderer_domain;
67 | proxy_set_header User-Agent $http_user_agent;
68 | proxy_set_header Connection "close";
69 |
70 | ########
71 | # Do not forget to change _YOUR_AUTH_TOKEN_ to token you get from ostr.io
72 | ########
73 | proxy_set_header Authorization "Basic _YOUR_AUTH_TOKEN_";
74 |
75 | resolver 1.1.1.1 8.8.4.4 8.8.8.8 1.0.0.1 valid=300s;
76 | resolver_timeout 15s;
77 |
78 | if ($is_webbot = 1) {
79 | sendfile off;
80 | proxy_pass https://$renderer_domain/render/https://$host$orig_uri$fragment$is_args$filtered_args;
81 | break;
82 | }
83 | }
84 | }
--------------------------------------------------------------------------------
/docs/prerendering/examples/cloudflare-worker/cloudflare-listener.worker.js:
--------------------------------------------------------------------------------
1 | const BOT_AGENTS = ['\\.net crawler', '360spider', '50\\.nu', '8bo crawler bot', 'aboundex', 'accoona', 'adldxbot', 'adsbot-google', 'ahrefsbot', 'altavista', 'appengine-google', 'applebot', 'archiver', 'arielisbot', 'ask jeeves', 'auskunftbot', 'baidumobaider', 'baiduspider', 'becomebot', 'bingbot', 'bingpreview', 'bitbot', 'bitlybot', 'blitzbot', 'blogbridge', 'boardreader', 'botseer', 'catchbot', 'catchpoint bot', 'charlotte', 'checklinks', 'cliqzbot', 'clumboot', 'coccocbot', 'converacrawler', 'crawl-e', 'crawlconvera', 'dataparksearch', 'daum', 'deusu', 'developers\\.google\\.com/+/web/snippet', 'discordbot', 'dotbot', 'duckduckbot', 'elefent', 'embedly', 'evernote', 'exabot', 'facebookbot', 'facebookexternalhit', 'fatbot', 'fdse robot', 'feed seeker bot', 'feedfetcher', 'femtosearchbot', 'findlinks', 'flamingo_searchengine', 'flipboard', 'followsite bot', 'furlbot', 'fyberspider', 'gaisbot', 'galaxybot', 'geniebot', 'genieo', 'gigablast', 'gigabot', 'girafabot', 'gomezagent', 'gonzo1', 'google sketchup', 'google-structured-data-testing-tool', 'googlebot', 'haosouspider', 'heritrix', 'holmes', 'hoowwwer', 'htdig', 'ia_archiver', 'idbot', 'infuzapp', 'innovazion crawler', 'internetarchive', 'iqdb', 'iskanie', 'istellabot', 'izsearch\\.com', 'kaloogabot', 'kaz\\.kz_bot', 'kd bot', 'konqueror', 'kraken', 'kurzor', 'larbin', 'leia', 'lesnikbot', 'linguee bot', 'linkaider', 'linkapediabot', 'linkedinbot', 'lite bot', 'llaut', 'lookseek', 'lycos', 'mail\\.ru_bot', 'masidani_bot', 'masscan', 'mediapartners-google', 'metajobbot', 'mj12bot', 'mnogosearch', 'mogimogi', 'mojeekbot', 'motominerbot', 'mozdex', 'msiecrawler', 'msnbot', 'msrbot', 'netpursual', 'netresearch', 'netvibes', 'newsgator', 'ng-search', 'nicebot', 'nutchcvs', 'nuzzel', 'nymesis', 'objectssearch', 'odklbot', 'omgili', 'oovoo', 'oozbot', 'openfosbot', 'orangebot', 'orbiter', 'org_bot', 'outbrain', 'pagepeeker', 'pagesinventory', 'parsijoobot', 'paxleframework', 'peeplo screenshot bot', 'pinterest', 'plantynet_webrobot', 'plukkie', 'pompos', 'psbot', 'quora link preview', 'qwantify', 'read%20later', 'reaper', 'redcarpet', 'redditbot', 'retreiver', 'riddler', 'rival iq', 'rogerbot', 'saucenao', 'scooter', 'scrapy', 'scrubby', 'searchie', 'searchsight', 'seekbot', 'semanticdiscovery', 'seznambot', 'showyoubot', 'simplepie', 'simpy', 'sitelockspider', 'skypeuripreview', 'slack-imgproxy', 'slackbot', 'slurp', 'snappy', 'sogou', 'solofield', 'speedy spider', 'speedyspider', 'sputnikbot', 'stackrambler', 'teeraidbot', 'teoma', 'theusefulbot', 'thumbshots\\.ru', 'thumbshotsbot', 'tineye', 'toweya\\.com', 'toweyabot', 'tumblr', 'tweetedtimes', 'tweetmemebot', 'twitterbot', 'url2png', 'vagabondo', 'vebidoobot', 'viber', 'visionutils', 'vkshare', 'voilabot', 'vortex', 'votay bot', 'voyager', 'w3c_validator', 'wasalive\\.bot', 'web-sniffer', 'websquash\\.com', 'webthumb', 'whatsapp', 'whatweb', 'wire', 'wotbox', 'yacybot', 'yahoo', 'yandex', 'yeti', 'yisouspider', 'yodaobot', 'yooglifetchagent', 'yoozbot', 'yottaamonitor', 'yowedo', 'zao-crawler', 'zebot_www\\.ze\\.bz', 'zooshot', 'zyborg', 'ai2bot', 'amazonbot', 'anthropic\\.com', 'bard', 'bytespider', 'ccbot', 'chatgpt-user', 'claude-web', 'claudebot', 'cohere-ai', 'deepseek', 'diffbot', 'duckassistbot', 'gemini', 'google-extended', 'gptbot', 'grok', 'meta-external', 'mistralai', 'oai-searchbot', 'omgili', 'openai\\.com', 'perplexity\\.ai', 'perplexitybot', 'xai', 'youbot'];
2 | const BOT_AGENTS_RE = new RegExp(BOT_AGENTS.join('|'), 'i');
3 | const IGNORE_EXTENSIONS = ['3ds', '3g2', '3gp', '3gpp', '7z', 'a', 'aac', 'aaf', 'adp', 'ai', 'aif', 'aiff', 'alz', 'ape', 'apk', 'appcache', 'ar', 'arj', 'asf', 'asx', 'atom', 'au', 'avchd', 'avi', 'bak', 'bbaw', 'bh', 'bin', 'bk', 'bmp', 'btif', 'bz2', 'bzip2', 'cab', 'caf', 'cco', 'cgm', 'class', 'cmx', 'cpio', 'cr2', 'crt', 'crx', 'css', 'csv', 'cur', 'dat', 'deb', 'der', 'dex', 'djvu', 'dll', 'dmg', 'dng', 'doc', 'docm', 'docx', 'dot', 'dotm', 'dra', 'drc', 'DS_Store', 'dsk', 'dts', 'dtshd', 'dvb', 'dwg', 'dxf', 'ear', 'ecelp4800', 'ecelp7470', 'ecelp9600', 'egg', 'eol', 'eot', 'eps', 'epub', 'exe', 'f4a', 'f4b', 'f4p', 'f4v', 'fbs', 'fh', 'fla', 'flac', 'fli', 'flv', 'fpx', 'fst', 'fvt', 'g3', 'geojson', 'gif', 'graffle', 'gz', 'gzip', 'h261', 'h263', 'h264', 'hqx', 'htc', 'ico', 'ief', 'img', 'ipa', 'iso', 'jad', 'jar', 'jardiff', 'jng', 'jnlp', 'jpeg', 'jpg', 'jpgv', 'jpm', 'js', 'jxr', 'key', 'kml', 'kmz', 'ktx', 'less', 'lha', 'lvp', 'lz', 'lzh', 'lzma', 'lzo', 'm2v', 'm3u', 'm4a', 'm4p', 'm4v', 'map', 'manifest', 'mar', 'markdown', 'md', 'mdi', 'mdown', 'mdwn', 'mht', 'mid', 'midi', 'mj2', 'mka', 'mkd', 'mkdn', 'mkdown', 'mkv', 'mml', 'mmr', 'mng', 'mobi', 'mov', 'movie', 'mp2', 'mp3', 'mp4', 'mp4a', 'mpe', 'mpeg', 'mpg', 'mpga', 'mpv', 'msi', 'msm', 'msp', 'mxf', 'mxu', 'nef', 'npx', 'nsv', 'numbers', 'o', 'oex', 'oga', 'ogg', 'ogv', 'opus', 'otf', 'pages', 'pbm', 'pcx', 'pdb', 'pdf', 'pea', 'pem', 'pgm', 'pic', 'pl', 'pm', 'png', 'pnm', 'pot', 'potm', 'potx', 'ppa', 'ppam', 'ppm', 'pps', 'ppsm', 'ppsx', 'ppt', 'pptm', 'pptx', 'prc', 'ps', 'psd', 'pya', 'pyc', 'pyo', 'pyv', 'qt', 'ra', 'rar', 'ras', 'raw', 'rdf', 'rgb', 'rip', 'rlc', 'rm', 'rmf', 'rmvb', 'ron', 'roq', 'rpm', 'rss', 'rtf', 'run', 'rz', 's3m', 's7z', 'safariextz', 'scpt', 'sea', 'sgi', 'shar', 'sil', 'sit', 'slk', 'smv', 'so', 'sub', 'svg', 'svgz', 'svi', 'swf', 'tar', 'tbz', 'tbz2', 'tcl', 'tga', 'tgz', 'thmx', 'tif', 'tiff', 'tk', 'tlz', 'topojson', 'torrent', 'ttc', 'ttf', 'txt', 'txz', 'udf', 'uvh', 'uvi', 'uvm', 'uvp', 'uvs', 'uvu', 'vcard', 'vcf', 'viv', 'vob', 'vtt', 'war', 'wav', 'wax', 'wbmp', 'wdp', 'weba', 'webapp', 'webm', 'webmanifest', 'webp', 'whl', 'wim', 'wm', 'wma', 'wml', 'wmlc', 'wmv', 'wmx', 'woff', 'woff2', 'wvx', 'xbm', 'xif', 'xla', 'xlam', 'xloc', 'xls', 'xlsb', 'xlsm', 'xlsx', 'xlt', 'xltm', 'xltx', 'xm', 'xmind', 'xml', 'xpi', 'xpm', 'xsl', 'xwd', 'xz', 'yuv', 'z', 'zip', 'zipx'];
4 |
5 | addEventListener('fetch', (e) => {
6 | e.passThroughOnException();
7 | const { request } = e;
8 | const url = new URL(request.url);
9 | const userAgent = request.headers.get('User-Agent')?.toLowerCase?.() || '';
10 | const pathName = url.pathname.toLowerCase();
11 | const extension = pathName.substring(pathName.lastIndexOf('.') + 1 || pathName.length)?.toLowerCase?.() || '';
12 |
13 | if (!url.origin.includes('ostr.io') && !url.pathname.includes('/.well-known/') && (userAgent && BOT_AGENTS_RE.test(userAgent)) && !(extension && IGNORE_EXTENSIONS.includes(extension))) {
14 | const headers = new Headers(request.headers);
15 | headers.set('Authorization', OSTR_AUTH);
16 |
17 | e.respondWith(fetch(new Request(`https://render.ostr.io/?url=${request.url}&bot=${userAgent}`, {
18 | headers: headers,
19 | redirect: 'manual',
20 | })));
21 | }
22 | });
23 |
--------------------------------------------------------------------------------
/tutorials/linux/security/automated-backups.md:
--------------------------------------------------------------------------------
1 | # Scheduled and Automated Backups
2 |
3 | In this tutorial (ToC):
4 |
5 | - [Create private GitHub Repository](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/security/automated-backups.md#create-private-github-repository)
6 | - [Add SSH Keys to private GitHub Repository](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/security/automated-backups.md#add-ssh-keys-to-private-github-repository)
7 | - [Enable LFS](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/security/automated-backups.md#enable-lfs)
8 | - [Select resources to backup](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/security/automated-backups.md#select-resources-to-backup)
9 | - [Create a bash script](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/security/automated-backups.md#create-the-bash-script)
10 | - [Schedule backups with CRON](https://github.com/veliovgroup/ostrio/blob/master/tutorials/linux/security/automated-backups.md#schedule-backups-with-cron)
11 |
12 | ## Create private GitHub Repository
13 |
14 | - Go to [create new repository](https://github.com/new)
15 | - Enter its name, `backups` for example
16 | - Select "Private"
17 | - Hit Create Repository button
18 |
19 | ## Add SSH Keys to private GitHub Repository
20 |
21 | Read [official tutorial by GitHub](https://help.github.com/articles/adding-a-new-ssh-key-to-your-github-account/).
22 |
23 | - Check for existing ssh keys in `~/.ssh` if directory is empty, generate SSH key:
24 | - Read [official instructions by GitHub](https://help.github.com/articles/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent/#platform-linux)
25 | - `ssh-keygen -t rsa -b 4096 -C "john@example.com"` where `john@example.com` is email used to sign up on GitHub
26 | - Create identity record in `~/.ssh/config`:
27 |
28 | ```bash
29 | # ~/.ssh/config
30 | Host github.com
31 | Hostname github.com
32 | User git
33 | IdentityFile ~/.ssh/id_rsa
34 | IdentitiesOnly yes
35 | ```
36 |
37 | - Read SSH key: `cat ~/.ssh/id_rsa.pub`
38 | - Go to [Keys management page on GitHub](https://github.com/settings/keys)
39 | - Click on New SSH key button
40 | - Enter the "title" describing host
41 | - Copy paste contents of `~/.ssh/id_rsa.pub` into "Key" text area
42 | - Click on Add SSH key button
43 |
44 | ## Enable LFS
45 |
46 | - LFS is highly recommended if the final backup archive is larger than 50 MBs
47 | - Read [official docs](https://help.github.com/articles/configuring-git-large-file-storage/)
48 | - Track `7z` archives with: `git lfs track "*.7z"` for our case
49 |
50 | ## Select resources to backup
51 |
52 | Think about what are you planning to backup, here are some paths and files for inspiration:
53 |
54 | - `/var/www`
55 | - `/var/log/syslog`
56 | - `/var/log/auth.log`
57 | - `/var/log/debug`
58 | - `/var/log/messages`
59 | - `/var/log/nginx/error.log`
60 | - `/var/log/mongodb/mongod.log`
61 | - Mongo DB with `mongodump`
62 | - Databases in general
63 |
64 | ## Create the bash script
65 |
66 | Before you go:
67 |
68 | - Clone repository into `~/` (home directory) `git clone git://url-to-repository`
69 | - It will create directory with repository name, `backups` in our case
70 | - Install 7z, with: `apt-get install p7zip-full`
71 |
72 | In this file:
73 |
74 | - Keep backups only for last 7 days (*change to meet your needs*)
75 | - Every backup has unique name with a date stamp
76 | - Backup Mongo DB (*change to DB of your choice*)
77 | - Backup `/var/www/` directory
78 | - Backup most useful log files
79 | - Archive every file into `.7z` with a password
80 |
81 | ```bash
82 | #!/bin/bash
83 | # Change Directory to repository
84 | cd /root/backups/
85 |
86 | DATE=$(date +%Y%m%d)
87 | DATEOLD=`date -d "${DATE} -7 day" +%Y%m%d`
88 |
89 | mongodump \
90 | --host rs0/1.example.com,2.example.com,3.example.com \
91 | --gzip \
92 | --archive="/root/backups/mongodb.$DATE.gz"
93 |
94 | tar --exclude='node_modules' -zcf "/root/backups/www.$DATE.tar.gz" /var/www
95 | tar -zcf "/root/backups/syslog.$DATE.tar.gz" /var/log/syslog
96 | tar -zcf "/root/backups/auth.$DATE.tar.gz" /var/log/auth.log
97 | faillog -a > /root/backups/faillog.log
98 | tar -zcf "/root/backups/faillog.$DATE.tar.gz" /root/backups/faillog.log
99 | lastlog > /root/backups/lastlog.log
100 | tar -zcf "/root/backups/lastlog.$DATE.tar.gz" /root/backups/lastlog.log
101 | tar -zcf "/root/backups/messages.$DATE.tar.gz" /var/log/messages
102 | tar -zcf "/root/backups/mongod.$DATE.tar.gz" /var/log/mongodb/mongod.log
103 | tar -zcf "/root/ostrio-b/nginx-access.$DATE.tar.gz" /var/log/nginx/access.log
104 | tar -zcf "/root/ostrio-b/nginx-error.$DATE.tar.gz" /var/log/nginx/error.log
105 |
106 | # CHANGE TO THE PASSWORD OF YOUR CHOICE
107 | 7z a "/root/backups/b.$DATE.7z" "/root/backups/mongodb.$DATE.gz" "/root/backups/www.$DATE.tar.gz" "/root/backups/syslog.$DATE.tar.gz" "/root/backups/auth.$DATE.tar.gz" "/root/backups/faillog.$DATE.tar.gz" "/root/backups/lastlog.$DATE.tar.gz" "/root/backups/messages.$DATE.tar.gz" "/root/backups/mongod.$DATE.tar.gz" "/root/backups/nginx-error.$DATE.tar.gz" "/root/backups/nginx-access.$DATE.tar.gz" -p
108 |
109 | rm "/root/backups/mongodb.$DATE.gz"
110 | rm "/root/backups/www.$DATE.tar.gz"
111 | rm "/root/backups/syslog.$DATE.tar.gz"
112 | rm "/root/backups/auth.$DATE.tar.gz"
113 | rm "/root/backups/faillog.$DATE.tar.gz"
114 | rm "/root/backups/faillog.log"
115 | rm "/root/backups/lastlog.$DATE.tar.gz"
116 | rm "/root/backups/lastlog.log"
117 | rm "/root/backups/messages.$DATE.tar.gz"
118 | rm "/root/backups/mongod.$DATE.tar.gz"
119 | rm "/root/backups/nginx-error.$DATE.tar.gz"
120 | rm "/root/backups/nginx-access.$DATE.tar.gz"
121 |
122 | # Remove backup older than 7 days:
123 | rm "/root/backups/b.$DATEOLD.7z"
124 | # Remove old backup from repository history (save LFS storage space):
125 | git rm --cached "b.$DATEOLD.7z"
126 |
127 | # Optional for rare cases (save LFS storage space):
128 | # git filter-branch --force --index-filter 'git rm --cached --ignore-unmatch b.$DATEOLD.7z' --prune-empty --tag-name-filter cat -- --all
129 |
130 | git pull
131 | git add .
132 | git commit -m "bckp $DATE"
133 | git push origin master
134 | ```
135 |
136 | Create file `backup.sh`, with contents of script above:
137 |
138 | ```shell
139 | nano ~/backup.sh
140 | ```
141 |
142 | Give execution permissions:
143 |
144 | ```shell
145 | chmod +x ~/backup.sh
146 | ```
147 |
148 | ## Schedule backups with CRON
149 |
150 | Open CRON editor:
151 |
152 | ```shell
153 | crontab -e
154 | ```
155 |
156 | Enter next rule:
157 |
158 | ```crontab
159 | # Daily at 00:00, saving logs to /var/log/backup.log
160 | 0 0 * * * /root/backup.sh > /var/log/backup.log 2>&1
161 | ```
162 |
--------------------------------------------------------------------------------
/docs/prerendering/netlify-prerendering.md:
--------------------------------------------------------------------------------
1 | # Netlify Integration: Pre-rendering with ostr.io
2 |
3 | This guide explains how to enable [ostr.io pre-rendering SEO MIddleware](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/README.md) for sites deployed on Netlify.
4 |
5 |
6 |
7 | ## Overview
8 |
9 | ostr.io pre-rendering SEO Middleware delivers fully rendered HTML to search engines and modern AI crawlers, improving index coverage and discoverability while reducing origin load. The Netlify integration routes crawler traffic through ostr.io pre-rendering engine.
10 |
11 | ## Availability
12 |
13 | * **Plans:** Available to **PRO** and **ENTERPRISE** Netlify subscriptions
14 | * **Enablement:** Netlify must enable the integration for a site/account after receiving request with **Auth Token** (that begins with `Basic ...`)
15 | * **SLA:** Netlify support team typically fulfills requests **within 24 hours**
16 |
17 | > [!IMPORTANT]
18 | > **Security note:** Treat the Auth Token as a secret. Do not share it publicly or commit it to source control.
19 |
20 |
21 | ## Prerequisites
22 |
23 | 1. An active ostr.io account
24 | 2. An active Netlify account on **PRO** or **ENTERPRISE** plans
25 | 3. A site deployed on Netlify
26 | 4. An ostr.io **Pre-rendering Auth Header/Token** issued for specific host (string that starts with `Basic ...`).
27 |
28 |
29 | ## Enablement Steps (via Netlify Support)
30 |
31 | 1. **Prepare the Auth Token**
32 |
33 | * Ensure the value starts with `Basic ...` (for example: `Basic dXNlcm5hbWU6c2VjcmV0`).
34 |
35 | 2. **Open a ticket with Netlify Support**
36 |
37 | * Subject: “Enable ostr.io pre-rendering integration”
38 | * Include:
39 |
40 | * Site domain name or site IDs on Netlify
41 | * The **exact** ostr.io Auth Token (the full `Basic ...` string)
42 | * Confirmation that ostr.io pre-rendering should be enabled for crawler traffic
43 |
44 | 3. **Wait for confirmation**
45 |
46 | * Netlify enables the integration and replies in the ticket/email (typically within **24 hours**).
47 |
48 | ### Example — Message to Netlify Support
49 |
50 | ```text
51 | Subject: Enable ostr.io pre-rendering integration
52 |
53 | Hello Netlify Support,
54 |
55 | Please enable the ostr.io pre-rendering integration for the following Netlify site:
56 | - Site: example-site.netlify.app (Production domain: example.com)
57 |
58 | X-Prerender-Token: Basic xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
59 |
60 | This integration should route crawler traffic through ostrio's pre-rendering engine.
61 |
62 | Thank you.
63 | ```
64 |
65 |
66 | ## Verifying the Integration
67 |
68 | After Netlify confirms enablement: **Test with a crawler User-Agent** to confirm a fully rendered HTML response is served without requiring client-side JavaScript. Example checks:
69 |
70 | ```bash
71 | # Simulate Googlebot; Should include X-Prerender-Id header
72 | curl --head -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "https://example.com/"
73 |
74 | # Simulate a standard browser (should receive standard response without X-Prerender-Id header)
75 | curl --head -A "Mozilla/5.0" "https://www.example.com/"
76 | ```
77 |
78 | Expected result: the response includes `X-Prerender-Id` header with server-rendered content (meta tags, structured data, critical content in source). The standard browser request remains unchanged.
79 |
80 | > [!TIP]
81 | > Try different tools and real crawler fetch tools (e.g., Search Console’s URL inspection). Include "escaped fragment" query at the end in tools like "page speed", for example: `https://example.com/?_escaped_fragment_=/`
82 |
83 |
84 | ## Common Issues & Fixes
85 |
86 | * **Invalid or missing Auth Token**
87 |
88 | * Symptoms: crawler requests not pre-rendered or return 401/403 errors.
89 | * Action: generate/verify the ostr.io token and re-submit the token to Netlify Support.
90 |
91 | * **Error: wrong domain; check auth creds**
92 |
93 | * Reason: Provided token belongs to other domain name or has a typo
94 | * Action: Double-check of provided Token has typos; If you have multiple domain names added to ostr.io — Ensure provided Auth/X-Prerender-Token belongs to the correct website
95 |
96 | * **Error: domain mismatch; check auth creds**
97 |
98 | * Reason: Provided token belongs to TLD, but used for subdomain
99 | * Action: Upgrade ostr.io account to PRO or BUSINESS plans to support subdomains pre-rendering; Or add subdomain as separate pre-rendering entity
100 |
101 | * **Site not present/verified in ostr.io**
102 |
103 | * Symptoms: requests rejected by the pre-rendering layer.
104 | * Action: add and verify the production domain in ostr.io before testing.
105 |
106 | * **Plan limitation**
107 |
108 | * Symptoms: integration requests declined or integration inactive in Netlify.
109 | * Action: upgrade Netlify account to **PRO** or **ENTERPRISE** plans.
110 |
111 | * **Caching delay**
112 |
113 | * Symptoms: inconsistent behavior immediately after enablement.
114 | * Action: wait for edge propagation, redeploy the site, or purge caches.
115 |
116 | * **IP allowlists / firewall**
117 |
118 | * Symptoms: blocked crawler traffic through the pre-rendering layer.
119 | * Action: ensure any origin firewall/CDN rules allow Netlify and ostr.io traffic.
120 |
121 |
122 | ## Disable Pre-rendering integration
123 |
124 | Open new Netlify Support ticket requesting removal of the ostr.io pre-rendering integration for the site (list site(s)).
125 |
126 |
127 | ## FAQ
128 |
129 | * **Is this automatic?**
130 | * No. Netlify must enable the integration after receiving the `Basic ...` Auth Token.
131 |
132 | * **How long does enablement take?**
133 | * Netlify typically completes enablement within **24 hours** of receiving the request.
134 |
135 | * **Which requests are pre-rendered?**
136 | * Crawler traffic (search engines and modern AI crawlers/agents). Regular visitor traffic is unaffected.
137 |
138 | * **Does this impact Core Web Vitals?**
139 | * Yes, it improves Web Vitals Scores for crawlers and bots with positive impact on crawler index, SEO scores, and organic traffic. Visitor-side Core Web Vitals remain unaffected by the crawler integration.
140 |
141 |
142 | ## Support
143 |
144 | * **ostr.io support:** account, token, and pre-rendering configuration questions.
145 | * **Netlify Support:** enable/disable the integration for specific sites and apply/update the Auth Token.
146 |
147 | If additional details are needed for enterprise scenarios (multi-site enablement, staged rollouts, or custom crawler policies), contact ostr.io support with the target domains and desired policy.
148 |
--------------------------------------------------------------------------------
/tutorials/website/social-networking/structured-data-basics.md:
--------------------------------------------------------------------------------
1 | # Structured data basics
2 |
3 | ## About:
4 |
5 | __TL;TR;__ Adding structured data (*a.k.a. "special" tags and attributes*) to your website will make it appear more user-friendly in search results, and as a consequence - will cause more click and traffic.
6 |
7 | __Description by Google:__
8 | > The structured data markup you provide for your pages helps Google algorithms better understand the kinds of things you offer your users so that their attributes can be featured in a visually compelling way—from ratings for recipe content, to structured layouts for events, to carousels for a related collection of items such as recipes from a specific host. These experiences are also streamlined for the user’s device and context, enhancing your content previews and making it easier for users to take action on your content wherever they are.
9 |
10 | ## Basic structure:
11 |
12 | Code-sample below is showcasing the most used case - *Website + Article*. There are many more types to use, like - reviews, video, audio, software, applications, recipes, movies, albums, etc. (*see links at the end of the article*).
13 |
14 | ```html
15 |
16 |
17 |
18 |
19 |
20 | Traveler: Norway
21 |
22 |
23 |
24 |
25 |
26 |
27 |
28 |
29 |
30 |
31 |
32 |
33 |
34 |
70 |
71 |
72 |
73 |
74 |
75 |
76 |
77 |
78 |
79 |
80 |
81 |
82 |
83 |
84 |
85 |
86 |
87 |
88 |
89 |
90 |
91 |
117 | Planning trip to Norway - here's bucket-list for thing to do during your trip.
118 |
119 |
Tips and hints for travelers to Norway
120 |
121 | The little village of Flåm offers a tranquil spot of natural beauty all year. It is located at the far end of the majestic Aurlandsfjord, with the stunning valley Flåmsdalen taking you up to the mountain areas of Myrdal. It is perfect for great outdoor experiences all year around..... See - https://www.visitnorway.com/flam/
122 |
123 |
124 |
125 |
126 | ```
127 |
128 | ## Further reading:
129 |
130 | - [Test your markup with testing-tool by Google](https://search.google.com/structured-data/testing-tool)
131 | - [Structured Data guidelines by Google](https://developers.google.com/search/docs/guides/search-features)
132 | - [Structured Data specs at schema.org](https://schema.org/)
133 |
--------------------------------------------------------------------------------
/docs/prerendering/examples/nginx/php-fpm.conf:
--------------------------------------------------------------------------------
1 | map $http_user_agent $is_webbot {
2 | default 0;
3 | "~*(?:\.net crawler|360spider|50\.nu|8bo crawler bot|aboundex|accoona|adldxbot|ahrefsbot|altavista|appengine-google|applebot|archiver|arielisbot|ask jeeves|auskunftbot|baidumobaider|baiduspider|becomebot|bingbot|bingpreview|bitbot|bitlybot|blitzbot|blogbridge|boardreader|botseer|catchbot|catchpoint bot|charlotte|checklinks|cliqzbot|clumboot|coccocbot|converacrawler|crawl-e|crawlconvera|dataparksearch|daum|deusu|discordbot|dotbot|duckduckbot|elefent|embedly|evernote|exabot|facebookbot|facebookexternalhit|meta-external|fatbot|fdse robot|feed seeker bot|feedfetcher|femtosearchbot|findlinks|flamingo_searchengine|flipboard|followsite bot|furlbot|fyberspider|gaisbot|galaxybot|geniebot|genieo|gigablast|gigabot|girafabot|gomezagent|gonzo1|googlebot|google sketchup|adsbot-google|google-structured-data-testing-tool|google-extended|developers\.google\.com/+/web/snippet|haosouspider|heritrix|holmes|hoowwwer|htdig|ia_archiver|idbot|infuzapp|innovazion crawler|instagram|internetarchive|iqdb|iskanie|istellabot|izsearch\.com|kaloogabot|kaz\.kz_bot|kd bot|konqueror|kraken|kurzor|larbin|leia|lesnikbot|linguee bot|linkaider|linkapediabot|linkedinbot|lite bot|llaut|lookseek|lycos|mail\.ru_bot|masidani_bot|masscan|mediapartners-google|metajobbot|mj12bot|mnogosearch|mogimogi|mojeekbot|motominerbot|mozdex|msiecrawler|msnbot|msrbot|netpursual|netresearch|netvibes|newsgator|ng-search|nicebot|nutchcvs|nuzzel|nymesis|objectssearch|odklbot|omgili|oovoo|oozbot|openfosbot|orangebot|orbiter|org_bot|outbrain|pagepeeker|pagesinventory|parsijoobot|paxleframework|peeplo screenshot bot|pinterest|plantynet_webrobot|plukkie|pompos|psbot|quora link preview|qwantify|read%20later|reaper|redcarpet|redditbot|retreiver|riddler|rival iq|rogerbot|saucenao|scooter|scrapy|scrubby|searchie|searchsight|seekbot|semanticdiscovery|seznambot|showyoubot|simplepie|simpy|sitelockspider|skypeuripreview|slackbot|slack-imgproxy|slurp|snappy|sogou|solofield|speedyspider|speedy spider|sputnikbot|stackrambler|teeraidbot|teoma|theusefulbot|thumbshots\.ru|thumbshotsbot|tineye|toweya\.com|toweyabot|tumblr|tweetedtimes|tweetmemebot|twitterbot|url2png|vagabondo|vebidoobot|viber|visionutils|vkshare|voilabot|vortex|votay bot|voyager|w3c_validator|wasalive\.bot|web-sniffer|websquash\.com|webthumb|whatsapp|whatweb|wire|wotbox|yacybot|yahoo|yandex|yeti|yisouspider|yodaobot|yooglifetchagent|yoozbot|yottaamonitor|yowedo|zao-crawler|zebot_www\.ze\.bz|zooshot|zyborgi|ai2bot|amazonbot|anthropic\.com|bard|bytespider|ccbot|chatgpt-user|claude-web|claudebot|cohere-ai|deepseek|diffbot|duckassistbot|gemini|gptbot|grok|mistralai|oai-searchbot|omgili|openai\.com|perplexity\.ai|perplexitybot|xai|youbot)" 1;
4 | }
5 |
6 | # CHECK AND SET ESCAPED FRAGMENT
7 | # For more info read: https://developers.google.com/webmasters/ajax-crawling/docs/specification
8 | map $arg__escaped_fragment_ $fragment {
9 | "" "";
10 | default "/$arg__escaped_fragment_";
11 | }
12 |
13 | # FILTER OUT _escaped_fragment_ FROM GET QUERY
14 | map $args $filtered_args {
15 | ~(^|&)_escaped_fragment_=[^&]*&?(.*) $1$2;
16 | default $args;
17 | }
18 |
19 | server {
20 | listen 80;
21 | listen [::]:80;
22 | server_name example.com;
23 |
24 | # DEFINE ROOT AND INDEX
25 | root /path/to/public;
26 | index index.php index.html;
27 |
28 | recursive_error_pages on;
29 | # CUSTOM 454 CODE FOR INTERNAL REDIRECT TO @prerendering
30 | error_page 454 = @prerendering;
31 |
32 | location ~* \.(?:3ds|3g2|3gp|3gpp|7z|a|aac|aaf|adp|ai|aif|aiff|alz|ape|apk|appcache|ar|arj|asf|asx|atom|au|avchd|avi|bak|bbaw|bh|bin|bk|bmp|btif|bz2|bzip2|cab|caf|cco|cgm|class|cmx|cpio|cr2|crt|crx|css|csv|cur|dat|deb|der|dex|djvu|dll|dmg|dng|doc|docm|docx|dot|dotm|dra|drc|DS_Store|dsk|dts|dtshd|dvb|dwg|dxf|ear|ecelp4800|ecelp7470|ecelp9600|egg|eol|eot|eps|epub|exe|f4a|f4b|f4p|f4v|fbs|fh|fla|flac|fli|flv|fpx|fst|fvt|g3|geojson|gif|graffle|gz|gzip|h261|h263|h264|hqx|htc|ico|ief|img|ipa|iso|jad|jar|jardiff|jng|jnlp|jpeg|jpg|jpgv|jpm|js|jxr|key|kml|kmz|ktx|less|lha|lvp|lz|lzh|lzma|lzo|m2v|m3u|m4a|m4p|m4v|map|manifest|mar|markdown|md|mdi|mdown|mdwn|mht|mid|midi|mj2|mka|mkd|mkdn|mkdown|mkv|mml|mmr|mng|mobi|mov|movie|mp2|mp3|mp4|mp4a|mpe|mpeg|mpg|mpga|mpv|msi|msm|msp|mxf|mxu|nef|npx|nsv|numbers|o|oex|oga|ogg|ogv|opus|otf|pages|pbm|pcx|pdb|pdf|pea|pem|pgm|pic|pl|pm|png|pnm|pot|potm|potx|ppa|ppam|ppm|pps|ppsm|ppsx|ppt|pptm|pptx|prc|ps|psd|pya|pyc|pyo|pyv|qt|ra|rar|ras|raw|rdf|rgb|rip|rlc|rm|rmf|rmvb|ron|roq|rpm|rss|rtf|run|rz|s3m|s7z|safariextz|scpt|sea|sgi|shar|sil|sit|slk|smv|so|sub|svg|svgz|svi|swf|tar|tbz|tbz2|tcl|tga|tgz|thmx|tif|tiff|tk|tlz|topojson|torrent|ttc|ttf|txt|txz|udf|uvh|uvi|uvm|uvp|uvs|uvu|vcard|vcf|viv|vob|vtt|war|wav|wax|wbmp|wdp|weba|webapp|webm|webmanifest|webp|whl|wim|wm|wma|wml|wmlc|wmv|wmx|woff|woff2|wvx|xbm|xif|xla|xlam|xloc|xls|xlsb|xlsm|xlsx|xlt|xltm|xltx|xm|xmind|xml|xpi|xpm|xsl|xwd|xz|yuv|z|zip|zipx)$ {
33 | # UNCOMMENT THE NEXT 2 LINES TO ENABLE CACHING FOR STATIC FILES
34 | # expires 30d;
35 | # add_header Cache-Control "public";
36 |
37 | # REDUCE LOAD BY DISABLING LOGGING FOR STATIC FILES
38 | access_log off;
39 | log_not_found off;
40 | try_files $uri =404;
41 | }
42 |
43 | location / {
44 | # IF REQUEST RECEIVED FROM BOT OR
45 | # WITH "FRAGMENT" REDIRECT TO @prerendering
46 | if ($is_webbot = 1) {
47 | return 454;
48 | }
49 | if ($args ~ _escaped_fragment_) {
50 | return 454;
51 | }
52 |
53 | # TRY STATIC FILE, THEN PASS TO THE APPLICATION
54 | try_files $uri $uri/ /index.php?$args;
55 | }
56 |
57 | location ~ \.php$ {
58 | # IF REQUEST RECEIVED FROM BOT OR
59 | # WITH "FRAGMENT" REDIRECT TO @prerendering
60 | if ($is_webbot = 1) {
61 | return 454;
62 | }
63 | if ($args ~ _escaped_fragment_) {
64 | return 454;
65 | }
66 |
67 | include /etc/nginx/fastcgi_params;
68 | fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
69 | fastcgi_param DOCUMENT_ROOT $realpath_root;
70 | fastcgi_pass unix:/var/run/example.com.sock;
71 | }
72 |
73 | location @prerendering {
74 | internal;
75 | # SET RENDERING ENDPOINT
76 | # read more at: https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/rendering-endpoints.md
77 | set $renderer_domain "render.ostr.io";
78 | set $orig_uri $request_uri;
79 |
80 | # Avoid dead loop and drop get query
81 | if ($orig_uri ~ "^(.*)\?(.*)$") {
82 | set $orig_uri $1;
83 | }
84 |
85 | # Minimize amount of forwarded headers
86 | proxy_pass_request_headers off;
87 | proxy_hide_header WWW-Authenticate;
88 | proxy_hide_header Set-Cookie;
89 |
90 | # !!Very important and required:
91 | proxy_set_header Host $renderer_domain;
92 | proxy_set_header User-Agent $http_user_agent;
93 | proxy_set_header Connection "close";
94 |
95 | ########
96 | # Do not forget to change _YOUR_AUTH_TOKEN_ to token obtained via ostr.io
97 | ########
98 | proxy_set_header Authorization "Basic _YOUR_AUTH_TOKEN_";
99 |
100 | # Optional headers, pass if necessary for backend:
101 | # proxy_set_header Pragma $http_pragma;
102 | # proxy_set_header Cache-Control $http_cache_control;
103 | # proxy_set_header Accept $http_accept;
104 | # proxy_set_header Accept-Encoding $http_accept_encoding;
105 | # proxy_set_header Accept-Language $http_accept_language;
106 |
107 | # Disable keepalive support, it's not needed for Pre-rendering
108 | # For more info read: http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_http_version
109 | proxy_http_version 1.0;
110 |
111 | # You can change resolver to local DNS server
112 | # And resolver can be moved to nginx.conf
113 | # Add ipv6=off if your server has no IPv6 support, ex.: `resolver 1.1.1.1 8.8.4.4 8.8.8.8 1.0.0.1 valid=300s ipv6=off;`
114 | resolver 1.1.1.1 8.8.4.4 8.8.8.8 1.0.0.1 valid=300s;
115 | resolver_timeout 15s;
116 |
117 | # Sendfile is useless for proxied requests, disable it
118 | # For more info read: http://nginx.org/en/docs/http/ngx_http_core_module.html#sendfile
119 | sendfile off;
120 | proxy_pass https://$renderer_domain/render/https://$host$orig_uri$fragment$is_args$filtered_args;
121 | }
122 | }
--------------------------------------------------------------------------------
/docs/prerendering/examples/nginx/upstream.conf:
--------------------------------------------------------------------------------
1 | map $http_user_agent $is_webbot {
2 | default 0;
3 | "~*(?:\.net crawler|360spider|50\.nu|8bo crawler bot|aboundex|accoona|adldxbot|ahrefsbot|altavista|appengine-google|applebot|archiver|arielisbot|ask jeeves|auskunftbot|baidumobaider|baiduspider|becomebot|bingbot|bingpreview|bitbot|bitlybot|blitzbot|blogbridge|boardreader|botseer|catchbot|catchpoint bot|charlotte|checklinks|cliqzbot|clumboot|coccocbot|converacrawler|crawl-e|crawlconvera|dataparksearch|daum|deusu|discordbot|dotbot|duckduckbot|elefent|embedly|evernote|exabot|facebookbot|facebookexternalhit|meta-external|fatbot|fdse robot|feed seeker bot|feedfetcher|femtosearchbot|findlinks|flamingo_searchengine|flipboard|followsite bot|furlbot|fyberspider|gaisbot|galaxybot|geniebot|genieo|gigablast|gigabot|girafabot|gomezagent|gonzo1|googlebot|google sketchup|adsbot-google|google-structured-data-testing-tool|google-extended|developers\.google\.com/+/web/snippet|haosouspider|heritrix|holmes|hoowwwer|htdig|ia_archiver|idbot|infuzapp|innovazion crawler|instagram|internetarchive|iqdb|iskanie|istellabot|izsearch\.com|kaloogabot|kaz\.kz_bot|kd bot|konqueror|kraken|kurzor|larbin|leia|lesnikbot|linguee bot|linkaider|linkapediabot|linkedinbot|lite bot|llaut|lookseek|lycos|mail\.ru_bot|masidani_bot|masscan|mediapartners-google|metajobbot|mj12bot|mnogosearch|mogimogi|mojeekbot|motominerbot|mozdex|msiecrawler|msnbot|msrbot|netpursual|netresearch|netvibes|newsgator|ng-search|nicebot|nutchcvs|nuzzel|nymesis|objectssearch|odklbot|omgili|oovoo|oozbot|openfosbot|orangebot|orbiter|org_bot|outbrain|pagepeeker|pagesinventory|parsijoobot|paxleframework|peeplo screenshot bot|pinterest|plantynet_webrobot|plukkie|pompos|psbot|quora link preview|qwantify|read%20later|reaper|redcarpet|redditbot|retreiver|riddler|rival iq|rogerbot|saucenao|scooter|scrapy|scrubby|searchie|searchsight|seekbot|semanticdiscovery|seznambot|showyoubot|simplepie|simpy|sitelockspider|skypeuripreview|slackbot|slack-imgproxy|slurp|snappy|sogou|solofield|speedyspider|speedy spider|sputnikbot|stackrambler|teeraidbot|teoma|theusefulbot|thumbshots\.ru|thumbshotsbot|tineye|toweya\.com|toweyabot|tumblr|tweetedtimes|tweetmemebot|twitterbot|url2png|vagabondo|vebidoobot|viber|visionutils|vkshare|voilabot|vortex|votay bot|voyager|w3c_validator|wasalive\.bot|web-sniffer|websquash\.com|webthumb|whatsapp|whatweb|wire|wotbox|yacybot|yahoo|yandex|yeti|yisouspider|yodaobot|yooglifetchagent|yoozbot|yottaamonitor|yowedo|zao-crawler|zebot_www\.ze\.bz|zooshot|zyborgi|ai2bot|amazonbot|anthropic\.com|bard|bytespider|ccbot|chatgpt-user|claude-web|claudebot|cohere-ai|deepseek|diffbot|duckassistbot|gemini|gptbot|grok|mistralai|oai-searchbot|omgili|openai\.com|perplexity\.ai|perplexitybot|xai|youbot)" 1;
4 | }
5 |
6 | # CHECK AND SET ESCAPED FRAGMENT
7 | # For more info read: https://developers.google.com/webmasters/ajax-crawling/docs/specification
8 | map $arg__escaped_fragment_ $fragment {
9 | "" "";
10 | default "/$arg__escaped_fragment_";
11 | }
12 |
13 | # FILTER OUT _escaped_fragment_ FROM GET QUERY
14 | map $args $filtered_args {
15 | ~(^|&)_escaped_fragment_=[^&]*&?(.*) $1$2;
16 | default $args;
17 | }
18 |
19 | upstream app {
20 | ip_hash;
21 | server 127.0.0.1:3001;
22 | server 127.0.0.1:3002;
23 | server 127.0.0.1:3003;
24 | }
25 |
26 | server {
27 | listen 80;
28 | listen [::]:80;
29 | server_name example.com;
30 | root /path/to/public;
31 |
32 | recursive_error_pages on;
33 | # CUSTOM 454 CODE FOR INTERNAL REDIRECT TO @prerendering
34 | error_page 454 = @prerendering;
35 | # CUSTOM 450 CODE FOR INTERNAL REDIRECT TO @application
36 | error_page 450 = @application;
37 | # HANDLE 404s BY THE APP
38 | error_page 404 = @application;
39 |
40 | location ~* \.(?:3ds|3g2|3gp|3gpp|7z|a|aac|aaf|adp|ai|aif|aiff|alz|ape|apk|appcache|ar|arj|asf|asx|atom|au|avchd|avi|bak|bbaw|bh|bin|bk|bmp|btif|bz2|bzip2|cab|caf|cco|cgm|class|cmx|cpio|cr2|crt|crx|css|csv|cur|dat|deb|der|dex|djvu|dll|dmg|dng|doc|docm|docx|dot|dotm|dra|drc|DS_Store|dsk|dts|dtshd|dvb|dwg|dxf|ear|ecelp4800|ecelp7470|ecelp9600|egg|eol|eot|eps|epub|exe|f4a|f4b|f4p|f4v|fbs|fh|fla|flac|fli|flv|fpx|fst|fvt|g3|geojson|gif|graffle|gz|gzip|h261|h263|h264|hqx|htc|ico|ief|img|ipa|iso|jad|jar|jardiff|jng|jnlp|jpeg|jpg|jpgv|jpm|js|jxr|key|kml|kmz|ktx|less|lha|lvp|lz|lzh|lzma|lzo|m2v|m3u|m4a|m4p|m4v|map|manifest|mar|markdown|md|mdi|mdown|mdwn|mht|mid|midi|mj2|mka|mkd|mkdn|mkdown|mkv|mml|mmr|mng|mobi|mov|movie|mp2|mp3|mp4|mp4a|mpe|mpeg|mpg|mpga|mpv|msi|msm|msp|mxf|mxu|nef|npx|nsv|numbers|o|oex|oga|ogg|ogv|opus|otf|pages|pbm|pcx|pdb|pdf|pea|pem|pgm|pic|pl|pm|png|pnm|pot|potm|potx|ppa|ppam|ppm|pps|ppsm|ppsx|ppt|pptm|pptx|prc|ps|psd|pya|pyc|pyo|pyv|qt|ra|rar|ras|raw|rdf|rgb|rip|rlc|rm|rmf|rmvb|ron|roq|rpm|rss|rtf|run|rz|s3m|s7z|safariextz|scpt|sea|sgi|shar|sil|sit|slk|smv|so|sub|svg|svgz|svi|swf|tar|tbz|tbz2|tcl|tga|tgz|thmx|tif|tiff|tk|tlz|topojson|torrent|ttc|ttf|txt|txz|udf|uvh|uvi|uvm|uvp|uvs|uvu|vcard|vcf|viv|vob|vtt|war|wav|wax|wbmp|wdp|weba|webapp|webm|webmanifest|webp|whl|wim|wm|wma|wml|wmlc|wmv|wmx|woff|woff2|wvx|xbm|xif|xla|xlam|xloc|xls|xlsb|xlsm|xlsx|xlt|xltm|xltx|xm|xmind|xml|xpi|xpm|xsl|xwd|xz|yuv|z|zip|zipx)$ {
41 | # UNCOMMENT THE NEXT 2 LINES TO ENABLE CACHING FOR STATIC FILES
42 | # expires 30d;
43 | # add_header Cache-Control "public";
44 |
45 | # REDUCE LOAD BY DISABLING LOGGING FOR STATIC FILES
46 | access_log off;
47 | log_not_found off;
48 | try_files $uri =404;
49 | }
50 |
51 | location / {
52 | # IF REQUEST RECEIVED FROM BOT OR
53 | # WITH "FRAGMENT" REDIRECT TO @prerendering
54 | if ($is_webbot = 1) {
55 | return 454;
56 | }
57 | if ($args ~ _escaped_fragment_) {
58 | return 454;
59 | }
60 |
61 | # FORWARD WEBSOCKETS AND OTHER STREAMS TO @application
62 | if ($http_upgrade) {
63 | return 450;
64 | }
65 |
66 | # TRY STATIC FILE, THEN PASS TO THE APPLICATION
67 | try_files $uri @application;
68 | }
69 |
70 | location @application {
71 | internal;
72 | sendfile off;
73 | proxy_http_version 1.1;
74 |
75 | proxy_connect_timeout 10s;
76 | proxy_read_timeout 30s;
77 | proxy_send_timeout 60s;
78 |
79 | proxy_pass http://app;
80 | }
81 |
82 | location @prerendering {
83 | internal;
84 | # SET RENDERING ENDPOINT
85 | # read more at: https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/rendering-endpoints.md
86 | set $renderer_domain "render.ostr.io";
87 | set $orig_uri $request_uri;
88 |
89 | # Avoid dead loop and drop get query
90 | if ($orig_uri ~ "^(.*)\?(.*)$") {
91 | set $orig_uri $1;
92 | }
93 |
94 | # Minimize amount of forwarded headers
95 | proxy_pass_request_headers off;
96 | proxy_hide_header WWW-Authenticate;
97 | proxy_hide_header Set-Cookie;
98 |
99 | # !!Very important and required:
100 | proxy_set_header Host $renderer_domain;
101 | proxy_set_header User-Agent $http_user_agent;
102 | proxy_set_header Connection "close";
103 |
104 | ########
105 | # Do not forget to change _YOUR_AUTH_TOKEN_ to token obtained via ostr.io
106 | ########
107 | proxy_set_header Authorization "Basic _YOUR_AUTH_TOKEN_";
108 |
109 | # Optional headers, pass if necessary for backend:
110 | # proxy_set_header Pragma $http_pragma;
111 | # proxy_set_header Cache-Control $http_cache_control;
112 | # proxy_set_header Accept $http_accept;
113 | # proxy_set_header Accept-Encoding $http_accept_encoding;
114 | # proxy_set_header Accept-Language $http_accept_language;
115 |
116 | # Disable keepalive support, it's not needed for Pre-rendering
117 | # For more info read: http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_http_version
118 | proxy_http_version 1.0;
119 |
120 | # You can change resolver to local DNS server
121 | # And resolver can be moved to nginx.conf
122 | # Add ipv6=off if your server has no IPv6 support, ex.: `resolver 1.1.1.1 8.8.4.4 8.8.8.8 1.0.0.1 valid=300s ipv6=off;`
123 | resolver 1.1.1.1 8.8.4.4 8.8.8.8 1.0.0.1 valid=300s;
124 | resolver_timeout 15s;
125 |
126 | # Sendfile is useless for proxied requests, disable it
127 | # For more info read: http://nginx.org/en/docs/http/ngx_http_core_module.html#sendfile
128 | sendfile off;
129 | proxy_pass https://$renderer_domain/render/https://$host$orig_uri$fragment$is_args$filtered_args;
130 | }
131 | }
--------------------------------------------------------------------------------
/tutorials/mongodb/multiple-server-replica-set.md:
--------------------------------------------------------------------------------
1 | # Multiple Server / Replica Set
2 |
3 | For more info see [original article](https://veliovgroup.com/article/2qsjtNf8NSB9XxZDh/mongodb-replica-set-with-oplog).
4 |
5 | ## Before you go:
6 |
7 | - Replica Set requires __at least 3__ servers (virtual or dedicated, if more than 3 servers - number of servers must be odd (e.g. 3, 5, 7, 9...))
8 | - Read how to [use SSL certificates with MongoDB](https://github.com/veliovgroup/ostrio/blob/master/tutorials/mongodb/use-ssl-http-encryption.md)
9 | - Make sure MongoDB port is open and accessible for other members of replica set, usually done with `iptables`. At least make sure all servers can ping each other
10 |
11 | We will use `wiredTiger` as database engine, you can read [its release notes](https://www.mongodb.com/blog/post/whats-new-mongodb-30-part-3-performance-efficiency-gains-new-storage-architecture) to find out more about its benefits.
12 |
13 | ## Definitions:
14 |
15 | - `admin` user - User with full `root` access to MongoDB features and commands
16 | - `appUser` - User with `readWrite` access to database used for your application
17 | - `appDB` - Database used for application
18 | - `` - placeholder, should be changed to strong password. __Always placed in double quotes__
19 |
20 | ## Steps
21 |
22 | Go through 1-16 steps in order to install an configure MongoDB 3+ ReplicaSet
23 |
24 | ### 1. Installation:
25 |
26 | On __each__ server follow steps described in [installation guide](https://docs.mongodb.com/manual/tutorial/install-mongodb-on-debian). During installation `mongodb` user in `mongodb` group will be created automatically
27 |
28 | ### 2. On __each__ server create and set right permissions for storage folder:
29 |
30 | ```shell
31 | mkdir -p /data/mongo
32 | chown -R mongodb:mongodb /data/mongo
33 | chmod 755 /data
34 | chmod -R 700 /data/mongo
35 | ```
36 |
37 | ### 3. On __each__ server create and set permissions for log folder:
38 |
39 | ```shell
40 | mkdir -p /var/log/mongodb
41 | chown -R mongodb:mongodb /var/log/mongodb
42 | ```
43 |
44 | ### 4. Edit/check `mongod` `systemd` service
45 |
46 | Create service (*if not created during installation*), if file exists — review it checking `User`, `Group`, and `ExecStart` values. Use `vim` or `nano` to open/create service file `/lib/systemd/system/mongod.service`:
47 |
48 | ```systemd
49 | [Unit]
50 | Description=MongoDB Database Server
51 | After=network.target
52 | Documentation=https://docs.mongodb.org/manual
53 |
54 | [Service]
55 | User=mongodb
56 | Group=mongodb
57 | EnvironmentFile=-/etc/default/mongod
58 | ExecStart=/usr/bin/mongod --config /etc/mongod.conf
59 | # on most virtual machines mongodb performs much better via numactl
60 | #ExecStart=/usr/bin/numactl --interleave all /usr/bin/mongod --config /etc/mongod.conf
61 | PIDFile=/var/run/mongodb/mongod.pid
62 | # file size
63 | LimitFSIZE=infinity
64 | # cpu time
65 | LimitCPU=infinity
66 | # virtual memory size
67 | LimitAS=infinity
68 | # open files
69 | LimitNOFILE=64000
70 | # processes/threads
71 | LimitNPROC=64000
72 | # locked memory
73 | LimitMEMLOCK=infinity
74 | # total threads (user+kernel)
75 | TasksMax=infinity
76 | TasksAccounting=false
77 |
78 | # Recommended limits for for mongod as specified in
79 | # http://docs.mongodb.org/manual/reference/ulimit/#recommended-settings
80 |
81 | [Install]
82 | WantedBy=multi-user.target
83 | ```
84 |
85 | #### 4.1. Enable service
86 |
87 | Enable `systemd` service
88 |
89 | ```shell
90 | systemctl --now enable mongod
91 | ```
92 |
93 | ### 5. On __each__ server edit MongoDB configuration file (`nano /etc/mongod.conf`):
94 |
95 | Create/edit `/etc/mongod.conf` file. Before authentication enable use `transitionToAuth` option
96 |
97 | 1. Change `[PORT, DEFAULT: 27017]` to port of your choice
98 | 2. Change `[IP ADDRESS, OR DOMAIN NAME]` to public IP address, VLAN/LAN IP Address, or domain name (can be comma separated, like: mongo.example.com,127.0.0.1,A.B.C.D). Domain name is required if SSL is set to `requireSSL` ([more info](https://github.com/veliovgroup/ostrio/blob/master/tutorials/mongodb/use-ssl-http-encryption.md))
99 |
100 | ```yaml
101 | processManagement:
102 | fork: true
103 | storage:
104 | dbPath: /data/mongo
105 | journal:
106 | enabled: true
107 | engine: wiredTiger
108 | net:
109 | port: [PORT, DEFAULT: 27017]
110 | bindIp: [IP ADDRESS, OR DOMAIN NAME]
111 | unixDomainSocket:
112 | enabled: false
113 | operationProfiling:
114 | slowOpThresholdMs: 2100
115 | mode: off
116 | replication:
117 | replSetName: rs0
118 | security:
119 | transitionToAuth: true
120 | # keyFile: /data/mongo/key
121 | # authorization: enabled
122 | ```
123 |
124 | ### 6. On __one__ server generate shared auth key:
125 |
126 | Generate random key for secure communication between Replica Set members ([more info](https://docs.mongodb.com/v3.2/tutorial/enforce-keyfile-access-control-in-existing-replica-set/)):
127 |
128 | ```shell
129 | openssl rand -base64 741 > /data/mongo/key
130 | chown mongodb:mongodb /data/mongo/key
131 | chmod 400 /data/mongo/key
132 |
133 | # Copy-paste generated key
134 | # to other servers (members)
135 | # so output key contents run:
136 | cat /data/mongo/key
137 | ```
138 |
139 | ### 7. On __each__ server start `mongod` service
140 |
141 | Start MongoDB service
142 |
143 | ```cron
144 | systemctl start mongod
145 | ```
146 |
147 | ### 8. Check logs for errors:
148 |
149 | Use `tail` to open `/var/log/mongodb/mongod.log` log-file with live updates
150 |
151 | ```shell
152 | tail -n 50 -f /var/log/mongodb/mongod.log
153 | # ctrl + c to exit tail command
154 | ```
155 |
156 | ### 9. On __one__ server configure and initiate Replica Set:
157 |
158 | On the server — connect to the MongoDB using `mongo` command, then initiate replica set for 3+ members.
159 |
160 | ```shell
161 | # Mongo Shell - connect to the MongoDB using `mongo` command
162 | var conf = {
163 | "_id" : "rs0",
164 | "members" : [
165 | {
166 | "_id" : 0,
167 | "host" : "[MEMBER IP/or/DOMAIN 1]:[PORT, DEFAULT: 27017]"
168 | },
169 | {
170 | "_id" : 1,
171 | "host" : "[MEMBER IP/or/DOMAIN 2]:[PORT, DEFAULT: 27018]"
172 | },
173 | {
174 | "_id" : 2,
175 | "host" : "[MEMBER IP/or/DOMAIN 3]:[PORT, DEFAULT: 27019]"
176 | }
177 | ]
178 | }
179 | rs.initiate(conf)
180 | ```
181 |
182 | ### 10. Find PRIMARY member:
183 |
184 | Next steps should be performed only on PRIMARY member. To find out which member is primary, run:
185 |
186 | ```shell
187 | # Mongo Shell - connect to the MongoDB using `mongo` command
188 | rs.status()
189 | ```
190 |
191 | ### 11. On PRIMARY: Create `admin` user:
192 |
193 | After connecting to the PRIMARY member — create new user with `root` privileges, replace `` with a String
194 |
195 | ```shell
196 | # Mongo Shell - connect to the MongoDB using `mongo` command
197 | use admin
198 | db.createUser({user:"admin", pwd:, roles:[{role:"root", db:"admin"}]})
199 | ```
200 |
201 | ### 12. On __each__ server update `mongod.conf`:
202 |
203 | Update `/etc/mongod.conf` using `nano` or `vim` text editors.
204 |
205 | 1. Remove/comment `transitionToAuth`
206 | 2. Add `keyFile`
207 | 3. Enable `authorization` option
208 |
209 | ```yaml
210 | # nano /etc/mongod.conf
211 | # ... last 4 lines ...
212 | security:
213 | # transitionToAuth: true
214 | keyFile: /data/mongo/key
215 | authorization: enabled
216 | ```
217 |
218 | ### 13. Restart `mongod` service
219 |
220 | ```shell
221 | service mongod restart
222 | ```
223 |
224 | ### 14. On PRIMARY: Create application user:
225 |
226 | Create `appUser` user with `readWrite` permissions (*assuming this user will be used to access MongoDB from application*):
227 |
228 | ```shell
229 | # Mongo Shell `mongo -u "admin" -p --authenticationDatabase "admin"`
230 | use admin
231 | db.createUser({user:"appUser", pwd:, roles:[{role:"readWrite", db:"appDB"}]})
232 | ```
233 |
234 | ### 15. On PRIMARY: Make sure all users are properly created:
235 |
236 | ```shell
237 | # Mongo Shell `mongo -u "admin" -p --authenticationDatabase "admin"`
238 | use admin
239 | show users
240 | ```
241 |
242 | ### 16. Update connection string:
243 |
244 | Update connection string within application
245 |
246 | ```plain
247 | mongodb://appUser:@:,:,:/appDB?authSource=admin&replicaSet=rs0
248 | ```
249 |
250 | ## Further reading:
251 |
252 | - [Use and configure OpLog](https://github.com/veliovgroup/ostrio/blob/master/tutorials/mongodb/enable-oplog.md)
--------------------------------------------------------------------------------
/docs/prerendering/README.md:
--------------------------------------------------------------------------------
1 | # Prerendering
2 |
3 | Lightning-fast, technology-agnostic SEO for websites, web apps, online shops and blogs — powered by pre-rendering middleware. Optimize TTFB, LCP, INP and CLS metrics, accelerate indexation, improve rich search results, and deliver consistent link previews across social and chat platforms
4 |
5 | ## ToC
6 |
7 | - [Reasons to use SEO Middleware](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/README.md#why-pre-rendering-seo-middleware)
8 | - [Terminology](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/README.md#used-terms)
9 | - [Integrations](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/README.md#integrations)
10 | - [Cloud-level integrations](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/README.md#cloud-level-integrations)
11 | - [Server-level integrations](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/README.md#server-level-integrations)
12 | - [Application-level integrations](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/README.md#application-level-integrations)
13 | - [How to's and pre-rendering features](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/README.md#how-tos-and-pre-rendering-features)
14 | - [Features](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/README.md#features)
15 | - [Optimizations](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/README.md#optimizations)
16 | - [Extra features](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/README.md#extra-features)
17 | - [Integration Tests](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/README.md#integration-tests)
18 |
19 | ## Why Pre-rendering SEO Middleware
20 |
21 | Reasons to use SEO Middleware
22 |
23 | - 🏎️ Expands Crawl Budget — Improves timings for dynamic and static pages via advanced CDN and caching;
24 | - 🚀 Boosts Web Vitals and Lighthouse scores;
25 | - 🎛️ Improves TTFB, LCP, INP, CLS, and other Web Vitals and LightHouse metrics positively enhancing overall SEO score;
26 | - 🖥 Supports PWAs and SPAs;
27 | - 📱 Supports mobile-like crawlers;
28 | - ⚡️ Supports [AMP (Accelerated Mobile Pages)](https://www.ampproject.org);
29 | - 🤝 AI agents, search engines, and social network crawlers love optimized pages that delivered in blazingly-fast manner;
30 | - 🖼️ Consistent link previews in messaging apps, like iMessage, Messages, Facebook, Slack, Telegram, WhatsApp, Viber, VK, Twitter, and other apps;
31 | - 👥 Image, title, and description previews for links posted at social networks, like Facebook, X/Twitter, Instagram, and other social networks.
32 |
33 | ## Used Terms
34 |
35 | Common terms used across pre-rendering documentation
36 |
37 | - "Pre-rendering" or "SEO middleware" - Thin micro-service that redirect bot's traffic to ostr.io "rendering endpoints";
38 | - "Search Engine" - System that is designed to search for information on the World Wide Web. Like: Google, Yahoo, Bing, etc. [Wiki definition](https://en.wikipedia.org/wiki/Web_search_engine);
39 | - "AI chat", "AI bot", or "AI agent" — Interface to LLM that can perform search around the web, like Chat GPT, Perplexity, Gemini, Grok, Claude and similar;
40 | - "Crawler" - An Internet bot that systematically browses the World Wide Web. [Wiki definition](https://en.wikipedia.org/wiki/Web_crawler). A crawler is one of the internal parts of Search Engines, messengers, and social networks. Like: Facebook, Viber, WhatsApp, Skype, etc. Mainly used to fetch and show information (image, title, description) about shared link;
41 | - "Spider" - Same as "Crawler";
42 | - "Cache" and "Caching" - Page "rendering" is time and resource consuming operation. To provide best possible response time "prerendered" results will be saved for time defined as "Cache TTL";
43 | - "Cache Freshness" or "Cache TTL" - Time in hours, during this period cache will remain fresh and will be served to all request;
44 | - "Pre-rendering Engine" - Core of the pre-rendering SEO middleware;
45 | - "(Pre)rendering Endpoint" - Address where located "Pre-rendering engine", we have 3 differently configured [rendering endpoints](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/rendering-endpoints.md), to fit every case.
46 |
47 | ## Integrations
48 |
49 | Pre-rendering SEO Middleware has various implementations and integration methods
50 |
51 | ### Cloud-level integrations
52 |
53 | Pre-rendering operates as middleware and can be easily integrated via CloudFlare Workers, or enabled at the level of Netlify, Vercel, Supabase *without extra plugins* and *without changes in the existing codebase*.
54 |
55 | - **Shopify** [CloudFlare Worker Integration](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/shopify-seo-integration.md#seo-middleware-worker-for-shopify)
56 | - General [CloudFlare Worker Integration](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/cloudflare-worker.md)
57 | - [Netlify Integration](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/netlify-prerendering.md)
58 |
59 | ### Server-level integrations
60 |
61 | Pre-rendering operates as middleware and can be easily integrated with CloudFlare Workers, Nginx, or Apache without extra plugins.
62 |
63 | - [Nginx Integration](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/nginx.md)
64 | - [Apache Integration](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/apache.md)
65 |
66 | ### Application-level integrations
67 |
68 | Application specific integrations are available via NPM packages. For Meteor.js integration, we have published Atmosphere package.
69 |
70 | - [Next.js Integration](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/nextjs-prerendering.md)
71 | - [Node.js Integration via NPM](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/node-npm.md)
72 | - [Meteor.js Integration via Atmosphere](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/meteor-atmosphere.md)
73 |
74 | ## How to's and pre-rendering features
75 |
76 | Learn how to use different features and settings within pre-rendering SEO middleware
77 |
78 | ### Features
79 |
80 | - [Caching](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/cache.md)
81 | - [Cache purge](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/cache-purge.md)
82 | - [Strip JavaScript](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/strip-javascript.md)
83 | - [Custom Status Codes in Analytics](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/prerendering-custom-status-codes.md)
84 | - [ES5 & Legacy Websites Support](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/es6-support.md)
85 |
86 | > [!TIP]
87 | > Pass `/sitemap.xml` to Pre-Render (*Pre-render a website*) in **Pre-rendering Panel** to instantly render whole website
88 |
89 | ### Optimizations
90 |
91 | - [Return genuine status code](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/genuine-status-code.md)
92 | - [Client Optimization](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/optimization.md)
93 | - [Server Optimization](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/rendering-endpoints.md)
94 | - [Use Custom Rendering Endpoints](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/rendering-endpoints.md)
95 |
96 | ### Extra features
97 |
98 | - [Accelerated Mobile Pages (AMP) Integration](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/amp-support.md)
99 | - [Detect Pre-rendering engine requests during runtime](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/detect-prerendering.md)
100 | - [Detect Pre-rendering engine requests during runtime (*Meteor.js specific*)](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/detect-prerendering-meteor.md)
101 |
102 | ### Integration tests
103 |
104 | - Use `cURL` and `Authorization` header (*see nginx integration*): `curl -v -H "Authorization: Basic TOKEN" https://render-bypass.ostr.io/?url=https://your-website-domain.com`;
105 | - Use `cURL` and authentication credentials (*see node.js integration*): `curl -v https://auth:string@render-bypass.ostr.io/?url=https://your-website-domain.com`;
106 | - Use `test:test` credentials to test general integration (*to make sure web application server can reach pre-rendering service*);
107 | - Use `Authorization: Basic dGVzdDp0ZXN0` header to test general integration (*to make sure web application server can reach pre-rendering service*);
108 |
--------------------------------------------------------------------------------
/docs/prerendering/examples/cloudflare-worker/coudflare-worker-guide.md:
--------------------------------------------------------------------------------
1 | # CloudFlare Worker Integration
2 |
3 | Step-by-step integration instructions for ostr.io pre-rendering SEO Middleware via CloudFlare Workers
4 |
5 | ## ToC
6 |
7 | - 🌤️ [Setup domain name at CloudFlare](#setup-domain-name-at-cloudflare)
8 | - ▲ [Setup domain name at ostr.io](#setup-domain-name-at-ostrio)
9 | - 🔌 [Create CloudFlare Worker and Connect to Domain](#create-cloudflare-worker-and-connect-to-domain)
10 | - 🔧 [Check that setup is correctly configured and working](#check-that-setup-is-correctly-configured-and-working)
11 | - 🖥️ [Check via Browser](#via-browser)
12 | - 👨💻 [Check in Terminal/Console using `curl`](#via-terminalconsole-using-curl)
13 |
14 | ### Setup domain name at CloudFlare
15 |
16 | To begin pre-rendering SEO Middleware integration via CloudFlare ensure you have an active account and connected domain name.
17 |
18 | 1. [Sign up](https://dash.cloudflare.com/sign-up) or [Login](https://dash.cloudflare.com/login) to CloudFlare
19 | 2. __Add domain:__ "Account Home" > click on Onboard a domain
20 | 3. Go through on-boarding steps, under "__Block AI training bots__" section select "__Do not block (off)__" option
21 |
22 | ### Setup domain name at ostr.io
23 |
24 | Create a new account if you don't have one yet. Then add and verify your domain ownership.
25 |
26 | 1. [Sign up](https://ostr.io/signup) or [Login](https://ostr.io/login) to ostr.io
27 | 2. Add server/domain following on-boarding guidance
28 | 3. Verify domain ownership using `DNS TXT` record adding via CloudFlare interface
29 | 4. Inside server's panel under "Available Services" section click on add next to "Pre-rendering"
30 | 5. Inside server's pre-rendering panel scroll down to integration guide
31 | 6. In the CLOUDFLARE tab — grab `env.OSTR_AUTH` value for CloudFlare's Worker environment variable
32 |
33 | ### Create CloudFlare Worker and Connect to Domain
34 |
35 | 1. Go to "Account Home" in the sidebar
36 | 2. __In the sidebar:__ Compute (Workers) > Workers & Pages (*see [UI screenshot](#create-new-worker-from-step-2)*)
37 | 3. At __Workers & Pages__ page click on Create > Then select "Start with Hello World!" (*see [UI screenshot](#create-new-worker-from-hello-world-template-from-step-3)*)
38 | 4. __New Worker Form__: Enter memorable name (ex.: `examplecom-seo-worker`) > click on Deploy (*see [UI screenshot](#create-new-worker-deploy-hello-world-worker-from-step-4)*)
39 | 5. After __new Worker__ created > click on Edit Code (*see [UI screenshot](#create-new-worker-edit-hello-world-worker-from-step-5)*)
40 | 6. At __Worker Editor__ > Remove default "Hello World" worker code and replace with [CF Worker Code](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/examples/cloudflare-worker/cloudflare.worker.js)
41 | 7. After Worker's code placed into __Worker Editor__ > click on Deploy (*see [UI screenshot](#create-new-worker-paste-and-deploy-workers-code-from-step-7)*)
42 | 8. __Pass API key to CloudFlare Worker via environment variable__ (*see [UI screenshots](#add-api-key-from-step-8)*)
43 | - Go to "Workers & Pages" > Open Newly Created Worker > Settings > Variables and Secrets > Click on Add button:
44 | - Type: `text`
45 | - Variable Name: `OSTR_AUTH`
46 | - Value: Place value (*from "integration guide", see step no.6*) that starts with `Basic ...`
47 | - Click on Deploy after adding `OSTR_AUTH` variable
48 | 9. __Connect Worker to a website__ (*see [UI screenshots](#connect-worker-to-a-website-from-step-9)*)
49 | - Go to "Account Home" > Domains > (*click on your domain name*)
50 | - In sidebar open "__Workers Routes__" > HTTP Routes > click on Add Route
51 | - __Add Route__ (*one of below*):
52 | - Standard (*recommended*): `https://example.com/*` or `https://www.example.com/*` (*if `www.` is your primary website location; __Must include `/*` (slash and asterisk) at the end of route__*)
53 | - Support both `http:` and `https:` protocols : `*example.com/*`
54 | - Apply to main domain (TLD) and __all__ subdomains (*PRO and BUSINESS plans only*) `*example.com/*`
55 | - Apply __only__ to for subdomains __only__ (*PRO and BUSINESS plans only*) `*.example.com/*`
56 | - __Worker:__ Select newly created worker from dropdown
57 | - Click on save
58 | 10. __Purge cache at CloudFlare__ (see [UI screenshot](#purge-websites-cache-from-step-10))
59 | - Go to "Account Home" in the sidebar menu
60 | - (*click on your domain name*) > Caching > Configuration > Purge Everything
61 |
62 | #### Example
63 |
64 | Example of configured and connected CloudFlare Worker setting for pre-rendering SEO Middleware
65 |
66 | 
67 |
68 | #### Create new Worker (*from step 2*)
69 |
70 | Open "Workers & Pages" and click on Create button to initiate new CloudFlare worker
71 |
72 |
73 | #### Create new Worker from "Hello World" template (*from step 3*)
74 |
75 | Choose "Start with Hello World!" and click Get Started
76 |
77 |
78 | #### Create new Worker: Deploy "Hello World" worker (*from step 4*)
79 |
80 | Enter Worker's name and click on Deploy
81 |
82 |
83 | #### Create new Worker: Edit "Hello World" worker (*from step 5*)
84 |
85 | Click on Edit code to start editing default "Hello World!" worker's code
86 |
87 |
88 | #### Create new Worker: Paste and deploy worker's code (*from step 7*)
89 |
90 | In editor paste [pre-rendering worker's code](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/examples/cloudflare-worker/cloudflare.worker.js) and click on Deploy then click on the name of the worker to go back
91 |
92 |
93 | #### Add API Key (*from step 8*)
94 |
95 | Go back to "Workers & Pages" > Open recently created Worker > Settings > Variables and Secrets > Click on Add
96 |
97 |
98 | Create `OSTR_AUTH` variable, paste API key that starts with `Basic xxx...` (*as found in ["integration guide" at ostr.io](#setup-domain-name-at-ostrio)*)
99 |
100 |
101 | #### Connect Worker to a website (*from step 9*)
102 |
103 | Go back to "Account Home" > Domains > Open domain name
104 |
105 |
106 | Open "Workers Routes" in sidebar > Click on Add Route
107 |
108 |
109 | Add route entering domain name (*see step 8 above for more details*)
110 |
111 |
112 | #### Purge website's cache (*from step 10*)
113 |
114 | Go back to "Account Home" > Open domain name > Caching > Configuration > Purge Everything
115 |
116 |
117 | ### Check that setup is correctly configured and working
118 |
119 | Check that `X-Prerender-Id` exists in response headers
120 |
121 | #### Via Browser
122 |
123 | 1. Open DevTools
124 | - Windows: F12 or Ctrl + Shift + I
125 | - MacOS: Option + ⌘ + I
126 | 2. __In DevTools:__ Open "Network" tab and check "Disable Cache" checkbox
127 | 3. Open website URL adding `?_escaped_fragment_=` query at the end. Ex.: `https://example.com/?_escaped_fragment_=/`
128 | 4. Ensure `X-Prerender-Id` header returned with response
129 | 5. Check ostr.io: Rendering statistics will appear in real-time
130 |
131 |
132 |
133 | #### Via Terminal/Console using CURL
134 |
135 | 1. Request using cURL: `curl --head -A GoogleBot "https://example.com/"`
136 | 2. Ensure `X-Prerender-Id` header returned with response
137 | 3. Check ostr.io: Rendering statistics will appear in real-time
138 |
139 | ## Further reading
140 |
141 | - 🏎️ [Speed-up rendering](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/optimization.md#speed-up-rendering)
142 | - 🤖 [Detect requests from pre-rendering](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/detect-prerendering.md)
143 | - 📔 [Detailed Pre-rendering service documentation](https://github.com/veliovgroup/ostrio/blob/master/docs/prerendering/README.md)
144 |
--------------------------------------------------------------------------------
16 | 
19 |