` are in `/lemmy-ui/src/assets/css/code-themes`. Custom css classes and changes to the default Bootstrap styles are in `/lemmy-ui/src/assets/css/main.css`.
16 |
17 | ## Making CSS Themes with Sass and Bootstrap
18 |
19 | Some tips if making a theme based off the default Lemmy themes.
20 |
21 | Every theme has these files:
22 |
23 | - an output `theme.css`
24 | - an output `theme.css.map`
25 | - `theme.scss`
26 | - `_variables.theme.scss`
27 |
28 | All `_variables.theme.scss` files will inherit variables from `_variables.scss`.
29 | All `theme.scss` will import bootstrap variables from `"../../../../node_modules/bootstrap/scss/bootstrap";` and its `_variables.theme.scss` file. It may import additional variables from another theme if it is built off it. For example, `litely-compact.scss` imports from `_variables.litely.scss`.
30 |
31 | ### Using SCSS Files
32 |
33 | If you are new to Sass, keep in mind that `theme.scss` files are for css and Sass flavored css. `_variables.theme.scss` are for variables.
34 |
35 | #### Export Your CSS File
36 |
37 | To export your custom `.scss` and `_variables.theme.scss` files to a `.css` file open the command line in the same directory as your files and run:
38 | `sass theme.scss theme.css` which will generate the `.css` and `.css.map` files.
39 |
40 | ### Bootstrap Notes
41 |
42 | If you are new to Bootstrap, be aware that variables starting with a dollar sign like `$variable` are Bootstrap variables and when output to css will look like `--bs-variable`. You can also define custom root variables in your `_variables.theme.scss` files like `:root {--custom-variable: value;};` and you can refer to this again in your `theme.scss` file.
43 |
44 | #### Bootstrap Variables on Lemmy
45 |
46 | The Darkly and Litely themes use default Bootstrap variables to define the grayscales and colours. Inspecting the Bootstrap documentation for how colours are applied to elements is recommended, along with inspecting the CSS and output files.
47 |
48 | ##### Light and Dark Modes
49 |
50 | Even though `darkly.css` is a dark theme, it has in-built light and dark modes using media queries. The Bootstrap variables `$enable-dark-mode` and `$enable-light-mode` can be used to toggle this behaviour on or off.
51 |
52 | ##### Overwriting Variables
53 |
54 | Most Lemmy theming is done with Bootstrap's default variables. Some variables are defined with `!important` which means if you have defined them in your custom theme that unless it also has `!important` it will be overwritten. To check, do a search in one of the default theme files or use the Developer Tools in your browser.
55 |
56 | To quickly test your theme if you do not own a Lemmy instance, you can use a browser add on to load your custom CSS file.
57 |
58 | ##### External Stylesheets
59 |
60 | As users cannot currently upload their own themes in Settings (only Admin can do that), custom themes loaded with an external style sheet will need to take into account that users will have a pre-selected theme in Settings that may have conflicting styles with the custom theme. If a theme is developed from an existing theme, having the default theme selected in Settings can minimize style conflicts.
61 |
62 | ## How CSS Themes are Added to Settings
63 |
64 | In short, given the css theme is the correct file format and in the correct theme directory, it will be appended to the bottom of the theme list in Settings. The name is the filename minus the file extension. Details are below.
65 |
66 | ### CSS Format Check
67 |
68 | The Typescript file `theme-handler.ts` in `/lemmy-ui/src/server/handlers/` will check for existing `css` files from the custom theme folder (`./volumes/lemmy-ui/extra_themes` or `./extra_themes`). Non `css` files will trigger an error.
69 |
70 | ### Building the Theme List
71 |
72 | If a custom css theme is found, the handler will call `themes-list-handler.ts` which will load `build-themes-list.ts` from `/lemmy-ui/src/server/utils/`. The file `build-themes-list.ts` will search the directories for files ending in `.css` and build a list.
73 |
74 | Custom themes are appended to the bottom of the theme list.
75 |
76 | ### Theme Names
77 |
78 | `build-themes-list.ts` will remove the file extension `.css` from the theme filename to display in Settings. For example, `darkly-compact.css` will appear as `darkly-compact`.
79 |
80 | ## Build and Format Theme Files Quickly
81 |
82 | Some tips to save time which would be useful for theme developers or if you are submitting pull requests to [lemmy-ui (Github)](https://github.com/LemmyNet/lemmy-ui).
83 |
84 | After changing variables or Sass files:
85 |
86 | - run `pnpm themes:build` to automatically rebuild all Lemmy css theme files using Sass
87 | - run `npx prettier -w /lemmy-ui/src/assets/css/themes` to reformat the code in the themes directory with prettier
88 |
--------------------------------------------------------------------------------
/src/administration/prometheus.md:
--------------------------------------------------------------------------------
1 | # Prometheus Metrics
2 |
3 | Lemmy supports the export of metrics in the [Prometheus](https://prometheus.io/)
4 | format. This document outlines how to enable the feature and begin collecting
5 | metrics.
6 |
7 | ## Configuration
8 |
9 | Configuration of the Prometheus endpoint is contained under the optional
10 | `prometheus` object in the `lemmy.hjson` file. In this configuration block, the
11 | `bind` address and `port` of the Prometheus metrics server can be configured.
12 |
13 | By default, it will serve on `127.0.0.1:10002`. If running inside a container,
14 | it should instead bind on all addresses as below.
15 |
16 | ```
17 | prometheus: {
18 | bind: "0.0.0.0"
19 | port: 10002
20 | }
21 | ```
22 |
23 | ## Scrape
24 |
25 | After Lemmy has been deployed, test that it is serving properly
26 | (substitute the correct hostname if not running locally).
27 |
28 | ```shell
29 | curl localhost:10002/metrics
30 | ```
31 |
32 | ### Prometheus Configuration
33 |
34 | Below is a minimal configuration on the Prometheus server's side that will
35 | scrape the metrics from Lemmy.
36 |
37 | ```yaml
38 | global:
39 | scrape_configs:
40 | - job_name: lemmy
41 | scrape_interval: 1m
42 | static_configs:
43 | - targets:
44 | - localhost:10002
45 | ```
46 |
47 | Then run a Prometheus server to scrape the metrics.
48 |
49 | ```shell
50 | docker run -p 9090:9090 -v prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus
51 | ```
52 |
53 | If Lemmy was deployed using the `docker-compose.yml`, then use the following
54 | configurations instead.
55 |
56 | ```yaml
57 | global:
58 | scrape_configs:
59 | - job_name: lemmy
60 | scrape_interval: 1m
61 | static_configs:
62 | - targets:
63 | - lemmy:10002
64 | ```
65 |
66 | Run the Prometheus server inside the same docker network that Lemmy is running
67 | in. This can be discovered by running `docker inspect` on the lemmy container.
68 |
69 | ```shell
70 | docker run -p 9090:9090 -v prometheus.yml:/etc/prometheus/prometheus.yml --network $LEMMY_NETWORK prom/prometheus
71 | ```
72 |
73 | ## Example
74 |
75 | Below is an example of what is returned from the `/metrics` endpoint. Each
76 | combination of `endpoint`, `method`, and `status` will produce a histogram
77 | (`lemmy_api_http_requests_duration_seconds_*`), so for brevity the output was
78 | reduced to a single endpoint.
79 |
80 | For the HTTP metrics, this is saying that `/api/v3/post/list` received 12 `GET`
81 | requests that returned a `200` HTTP code. Cumulatively, these requests took
82 | 0.383 seconds. 5 requests completed between 0.01 and 0.025 seconds. 5 more
83 | completed between 0.025 and 0.05 seconds. And the remaining 2 requests completed
84 | between 0.05 and 0.1 seconds.
85 |
86 | ```
87 | lemmy_api_http_requests_duration_seconds_bucket{endpoint="/api/v3/post/list",method="GET",status="200",le="0.005"} 0
88 | lemmy_api_http_requests_duration_seconds_bucket{endpoint="/api/v3/post/list",method="GET",status="200",le="0.01"} 0
89 | lemmy_api_http_requests_duration_seconds_bucket{endpoint="/api/v3/post/list",method="GET",status="200",le="0.025"} 5
90 | lemmy_api_http_requests_duration_seconds_bucket{endpoint="/api/v3/post/list",method="GET",status="200",le="0.05"} 10
91 | lemmy_api_http_requests_duration_seconds_bucket{endpoint="/api/v3/post/list",method="GET",status="200",le="0.1"} 12
92 | lemmy_api_http_requests_duration_seconds_bucket{endpoint="/api/v3/post/list",method="GET",status="200",le="0.25"} 12
93 | lemmy_api_http_requests_duration_seconds_bucket{endpoint="/api/v3/post/list",method="GET",status="200",le="0.5"} 12
94 | lemmy_api_http_requests_duration_seconds_bucket{endpoint="/api/v3/post/list",method="GET",status="200",le="1"} 12
95 | lemmy_api_http_requests_duration_seconds_bucket{endpoint="/api/v3/post/list",method="GET",status="200",le="2.5"} 12
96 | lemmy_api_http_requests_duration_seconds_bucket{endpoint="/api/v3/post/list",method="GET",status="200",le="5"} 12
97 | lemmy_api_http_requests_duration_seconds_bucket{endpoint="/api/v3/post/list",method="GET",status="200",le="10"} 12
98 | lemmy_api_http_requests_duration_seconds_bucket{endpoint="/api/v3/post/list",method="GET",status="200",le="+Inf"} 12
99 | lemmy_api_http_requests_duration_seconds_sum{endpoint="/api/v3/post/list",method="GET",status="200"} 0.3834808429999999
100 | lemmy_api_http_requests_duration_seconds_count{endpoint="/api/v3/post/list",method="GET",status="200"} 12
101 |
102 | lemmy_api_http_requests_total{endpoint="/api/v3/post/list",method="GET",status="200"} 12
103 |
104 | # HELP lemmy_db_pool_available_connections Number of available connections in the pool
105 | # TYPE lemmy_db_pool_available_connections gauge
106 | lemmy_db_pool_available_connections 2
107 | # HELP lemmy_db_pool_connections Current number of connections in the pool
108 | # TYPE lemmy_db_pool_connections gauge
109 | lemmy_db_pool_connections 2
110 | # HELP lemmy_db_pool_max_connections Maximum number of connections in the pool
111 | # TYPE lemmy_db_pool_max_connections gauge
112 | lemmy_db_pool_max_connections 5
113 |
114 | # HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
115 | # TYPE process_cpu_seconds_total counter
116 | process_cpu_seconds_total 14
117 | # HELP process_max_fds Maximum number of open file descriptors.
118 | # TYPE process_max_fds gauge
119 | process_max_fds 1073741816
120 | # HELP process_open_fds Number of open file descriptors.
121 | # TYPE process_open_fds gauge
122 | process_open_fds 91
123 | # HELP process_resident_memory_bytes Resident memory size in bytes.
124 | # TYPE process_resident_memory_bytes gauge
125 | process_resident_memory_bytes 75603968
126 | # HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
127 | # TYPE process_start_time_seconds gauge
128 | process_start_time_seconds 1688487611
129 | # HELP process_threads Number of OS threads in the process.
130 | # TYPE process_threads gauge
131 | process_threads 37
132 | # HELP process_virtual_memory_bytes Virtual memory size in bytes.
133 | # TYPE process_virtual_memory_bytes gauge
134 | process_virtual_memory_bytes 206000128
135 | ```
136 |
--------------------------------------------------------------------------------
/src/users/01-getting-started.md:
--------------------------------------------------------------------------------
1 | # Getting Started
2 |
3 | ## Choosing an Instance
4 |
5 | If you are used to sites like Reddit, then Lemmy works in a fundamentally different way. Instead of a single website like reddit.com, there are many different websites (called _instances_). These are operated by different people, have different topics and rules. Nevertheless, posts created in one instance can directly be seen by users who are registered on another. Its basically like email, but for social media.
6 |
7 | This means before using Lemmy and registering an account, you need to pick an instance. For this you can browse the [instance list](https://join-lemmy.org/instances) and look for one that matches your topics of interest. You can also see if the rules match your expectations, and how many users there are. It is better to avoid very big or very small instances. But don't worry too much about this choice, you can always create another account on a different instance later.
8 |
9 | [instance list screenshot]
10 |
11 | ## Registration
12 |
13 | Once you choose an instance, it's time to create your account. To do this, click _sign up_ in the top right of the page, or click the top right button on mobile to open a menu with _sign up_ link.
14 |
15 | [registration page screenshot]
16 |
17 | On the signup page you need to enter a few things:
18 |
19 | - **Username**: How do you want to be called? This name can not be changed and is unique within an instance. Later you can also set a _displayname_ which can be freely changed. If your desired username is taken, consider choosing a different instance where it is still available.
20 | - **Email**: Your email address. This is used for password resets and notifications (if enabled). Providing an email address is usually optional, but admins may choose to make it mandatory. In this case you will have to wait for a confirmation mail and click the link after completing this form.
21 | - **Password**: The password for logging in to your account. Make sure to choose a long and unique password which isn't used on any other website.
22 | - **Verify password**: Repeat the same password from above to ensure that it was entered correctly.
23 |
24 | There are also a few optional fields, which you may need to fill in depending on the instance configuration:
25 |
26 | - **Question/Answer**: Instance admins can set an arbitrary question which needs to be answered in order to create an account. This is often used to prevent spam bots from signing up. After submitting the form, you will need to wait for some time until the answer is approved manually before you can login.
27 | - **Code**: A captcha which is easy to solve for humans but hard for bots. Enter the letters and numbers that you see in the text box, ignoring uppercase or lowercase. Click the refresh button if you are unable to read a character. The _play_ button plays an audio version of the captcha.
28 | - **Show NSFW content**: Here you can choose if content that is "not safe for work" (or adult-only) should be shown.
29 |
30 | When you are done, press the _sign up_ button.
31 |
32 | It depends on the instance configuration when you can login and start using the account. In case the email is mandatory, you need to wait for the confirmation email and click the link first. In case "Question/Answer" is present, you need to wait for an admin to manually review and approve your registration. If you have problems with the registration, try to get in contact with the admin for support. You can also choose a different instance to sign up if your primary choice does not work.
33 |
34 | ## Following Communities
35 |
36 | After logging in to your new account, its time to follow communities that you are interested in. For this you can click on the _communities_ link at the top of the page (on mobile, you need to click the menu icon on the top right first). You will see a list of communities which can be filtered by subscribed, local or all. Local communities are those which are hosted on the same site where you are signed in, while _all_ also contains federated communities from other instances. In any case you can directly subscribe to communities with the right-hand subscribe link. Or click on the community name to browse the community first, see what its posted and what the rules are before subscribing.
37 |
38 | Another way to find communities to subscribe to is by going to the front page and browsing the posts. If there is something that interests you, click on the post title to see more details and comments. Here you can subscribe to the community in the right-hand sidebar, or by clicking the "sidebar" button on mobile.
39 |
40 | These previous ways will only show communities that are already known to the instance. Especially if you joined a small or inactive Lemmy instance, there will be few communities to discover. You can find more communities by browsing different Lemmy instances, or using the [Lemmy Explorer](https://lemmyverse.net/communities). When you found a community that you want to follow, enter its URL (e.g. `https://feddit.org/c/main`) or the identifier (e.g. `!main@feddit.org`) into the search field of your own Lemmy instance. Lemmy will then fetch the community from its original instance, and allow you to interact with it. The same method also works to fetch users, posts or comments from other instances.
41 |
42 | ## Setting up Your Profile
43 |
44 | Before you start posting, its a good idea to provide some details about yourself. Open the top-right menu and go to "settings". Here the following settings are available for your public profile:
45 |
46 | - **Displayname**: An alternative username which can be changed at any time
47 | - **Bio**: Long description of yourself, can be formatted with Markdown
48 | - **Matrix User**: Your username on the decentralized [Matrix chat](https://matrix.org/)
49 | - **Avatar**: Profile picture that is shown next to all your posts
50 | - **Banner**: A header image for your profile page
51 |
52 | On this page you can also change the email and password. Additionally there are many other settings available, which allow customizing of your browsing experience:
53 |
54 | - **Blocks** (tab at top of the page): Here you can block users and communities, so that their posts will be hidden.
55 | - **Interface language**: Which language the user interface should use.
56 | - **Languages**: Select the languages that you speak to see only content in these languages. This is a new feature and many posts don't specify a language yet, so be sure to select "Undetermined" to see them.
57 | - **Theme**: You can choose between different color themes for the user interface. Instance admins can add more themes.
58 | - **Type**: Which timeline you want to see by default on the front page; only posts from communities that you subscribe to, posts in local communities, or all posts including federated.
59 | - **Sort type**: How posts and comments should be sorted by default. See [Votes and Ranking](03-votes-and-ranking.md) for details.
60 | - **Show NSFW content**: Whether or not you want to see content that is "not safe for work" (or adult-only).
61 | - **Show Scores**: Whether the number of upvotes and downvotes should be visible.
62 | - **Show Avatars**: Whether profile pictures of other users should be shown.
63 | - **Bot Account**: Enable this if you are using a script or program to create posts automatically
64 | - **Show Bot Accounts**: Disable this to hide posts that were created by bot accounts.
65 | - **Show Read Posts**: If this is disabled, posts that you already viewed are not shown in listings anymore. Useful if you want to find new content easily, but makes it difficult to follow ongoing discussion under existing posts.
66 | - **Show Notifications for New Posts**: Enable this to receive a popup notification for each new post that is created.
67 | - **Send notifications to Email**: Enable to receive notifications about new comment replies and private messages to your email address.
68 |
69 | ## Start Posting
70 |
71 | Finally its time to start posting! To do this it is always a good idea to read the community rules in the sidebar (below the _Subscribe_ button). When you are ready, go to a post and type your comment in the box directly below for a top-level reply. You can also write a nested reply to an existing comment, by clicking the left-pointing arrow.
72 |
73 | Other than commenting on existing posts, you can also create new posts. To do this, click the button _Create a post_ in the sidebar. Here you can optionally supply an external link or upload an image. The title field is mandatory and should describe what you are posting. The body is again optional, and gives space for long texts. You can also embed additional images here. The _Community_ dropdown below allows choosing a different community to post in. With _NSFW_, posts can be marked as "not safe for work". Finally you can specify the language that the post is written in, and then click on _Create_.
74 |
75 | One more possibility is to write private messages to individual users. To do this, simply visit a user profile and click _Send message_. You will be notified about new private messages and comment replies with the bell icon in the top right.
76 |
--------------------------------------------------------------------------------
/src/users/02-media.md:
--------------------------------------------------------------------------------
1 | # Media
2 |
3 | ## Text
4 |
5 | The main type of content in Lemmy is text which can be formatted with Markdown. Refer to the table below for supported formatting rules. The Lemmy user interface also provides buttons for formatting, so it's not necessary to remember all of it. You can also follow the interactive [CommonMark tutorial](https://commonmark.org/help/tutorial/) to get started.
6 |
7 | | Type | Or | … to Get |
8 | | ---------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
9 | | \*Italic\* | \_Italic\_ | _Italic_ |
10 | | \*\*Bold\*\* | \_\_Bold\_\_ | **Bold** |
11 | | \# Heading 1 | Heading 1
========= | Heading 1
|
12 | | \## Heading 2 | Heading 2
--------- | Heading 2
|
13 | | \[Link\](http://a.com) | \[Link\]\[1\]
⋮
\[1\]: http://b.org | [Link](https://commonmark.org/) |
14 | | !\[Image\](http://url/a.png) | !\[Image\]\[1\]
⋮
\[1\]: http://url/b.jpg |  |
15 | | Example\^\[Footnote\] | | Example[1]
⋮
Footnote ↩︎
Blockquote
|
17 | | \* List
\* List
\* List | \- List
\- List
\- List
| - List
- List
- List
|
18 | | 1\. One
2\. Two
3\. Three | 1) One
2) Two
3) Three | - One
- Two
- Three
|
19 | | Horizontal Rule
\--- | Horizontal Rule
\*\*\* | Horizontal Rule
|
20 | | \`Inline code\` with backticks | | `Inline code` with backticks |
21 | | \`\`\`
\# code block
print '3 backticks or'
print 'indent 4 spaces'
\`\`\` | ····\# code block
····print '3 backticks or'
····print 'indent 4 spaces' | # code block
print '3 backticks or'
print 'indent 4 spaces'
|
22 | | ::: spoiler hidden or nsfw stuff
_a bunch of spoilers here_
::: | | hidden or nsfw stuff
a bunch of spoilers here
|
23 | | Some \~subscript\~ text | | Some subscript text |
24 | | Some \^superscript\^ text | | Some superscript text |
25 | | \~\~Strikethrough\~\~ | | Some ~removed~ text |
26 | | \{Ruby\|text\} | | Ruby |
27 |
28 | [CommonMark Tutorial](https://commonmark.org/help/tutorial/)
29 |
30 | ## Images and Video
31 |
32 | Lemmy also allows sharing of images and videos. To upload an image, go to the _Create post_ page and click the little image icon under the _URL_ field. This allows you to select a local image. If you made a mistake, a popup message allows you to delete the image. The same image button also allows uploading of videos in .gif format. Instead of uploading a local file, you can also simply paste the URL of an image or video from another website.
33 |
34 | Note that this functionality is not meant to share large images or videos, because that would require too many server resources. Instead, upload them on another platform like [PeerTube](https://joinpeertube.org/) or [Pixelfed](https://pixelfed.org/), and share the link on Lemmy.
35 |
36 | ## Torrents
37 |
38 | Since Lemmy doesn't host large videos or other media, users can share files using [BitTorrent](https://en.wikipedia.org/wiki/BitTorrent) links. In BitTorrent, files are shared not by a single user, but by _many users_ at the same time. This makes file sharing efficient, fast, and reliable, as long as several sources are sharing the files.
39 |
40 | Lemmy supports posting torrent magnet links (links that start with `magnet:`) in the post _URL_ field, or as Markdown links within comments. You can get a magnet link by clicking _copy magnet link_ in your torrent app.
41 |
42 | With this, Lemmy can serve as an alternative to centralized media-centric services like YouTube and Spotify.
43 |
44 | ### How to Watch Torrents
45 |
46 | #### Beginner
47 |
48 | To easily stream videos and audio on Lemmy, you can use any of the following apps. After clicking on a torrent link in Lemmy, a dialog will pop up asking you to open the link in the app.
49 |
50 | - [Stremio](https://www.stremio.com/) (Desktop, Android)
51 | - [WebTorrent Desktop](https://webtorrent.io/desktop/) (Desktop)
52 | - [Popcorn Time](https://github.com/popcorn-official/popcorn-desktop) (Desktop)
53 | - [xTorrent](https://play.google.com/store/apps/details?id=com.gamemalt.streamtorrentvideos) (Android)
54 |
55 | #### Advanced
56 |
57 | For those who would like to help share files, you can use any of the following torrent clients:
58 |
59 | - [qBittorrent](https://qbittorrent.org/) (Desktop)
60 | - [Deluge](https://www.deluge-torrent.org/) (Desktop)
61 | - [Transmission](https://transmissionbt.com/) (Desktop)
62 | - [LibreTorrent](https://gitlab.com/proninyaroslav/libretorrent) (Android)
63 |
64 | Many of these support _streaming_ videos. To do this, make sure you check _sequential download_, wait for enough of the download to complete, then click to open the video file.
65 |
66 | If you'd like, you can also set up a media server to view this content on any device. Some good options are:
67 |
68 | - [Jellyfin](https://jellyfin.org/) (Movies, TV, Music, Audiobooks)
69 | - [Navidrome](https://www.navidrome.org/) (Music)
70 | - [audiobookshelf](https://www.audiobookshelf.org/) (Audiobooks)
71 |
--------------------------------------------------------------------------------
/src/administration/horizontal_scaling.md:
--------------------------------------------------------------------------------
1 | # Horizontal Scaling
2 |
3 | This is a collection of notes on scaling different Lemmy components horizontally. This is not meant as a step-by-step guide, rather as general tips and knowledge that might be useful to somebody who is already familiar with and horizontal scaling concepts and best practices. If you don't already know about concepts like load balancing, object storage, and reverse proxies, then this document will probably be hard (or impossible) to understand.
4 |
5 | **Note: scaling Lemmy this way is not necessary, and for small instances, the added overhead most likely outweighs any benefits.**
6 |
7 | ## Why scale horizontally?
8 |
9 | - If one of your Lemmy servers dies for any reason, your instance can still remain operational
10 | - Having multiple Lemmy servers allows you to do rolling upgrades (no downtime needed to upgrade Lemmy!)
11 | - Horizontal scaling provides additional flexibility in how you upgrade your infrastructure
12 | - As of Lemmy version 0.18.2, Lemmy appears to be able to use the same resources more efficiently if they are split between multiple Lemmy processes
13 |
14 | ## Breakdown of Lemmy components
15 |
16 | See [Lemmy components](administration.md#lemmy-components) for a high level overview of what each component does.
17 |
18 | ### Lemmy-ui
19 |
20 | Lemmy-ui can be horizontally scaled, but make sure you read the following section for information about rolling upgrades.
21 |
22 | #### Rolling upgrades
23 |
24 | In general, there is nothing preventing you from running multiple load balanced Lemmy-ui servers in parallel, but without some custom configuration, **rolling upgrades will break Lemmy for your users!** This is because by default, Lemmy-ui will serve static files through the express process.
25 |
26 | Consider the scenario where you have 2 Lemmy-ui servers, one is running on version 1, and the other has just come online running version 2. A user opening your Lemmy in their browser might have their front page html request routed to version 1. This html will contain a reference to many other static files which are specific to version 1. Most likely, some of these requests will get routed to the server running version 2. **The files will not exist on this server, the server will respond with 404, and the UI will remain in a broken state.**
27 |
28 | There are a few ways to work around this issue:
29 |
30 | 1. The safest option is to ensure that all servers are able to serve static files for all currently active Lemmy-ui versions.
31 | - There are many ways to achieve this, the most obvious one being to upload your static files to object storage, and have reverse proxies route static file requests to your static files bucket.
32 | - Here's one possible example of how to do this with nginx:
33 |
34 | ```
35 | location /static {
36 | expires 1y;
37 | add_header Cache-Control "public";
38 | proxy_pass https://${bucket_fqdn};
39 |
40 | limit_except GET {
41 | deny all;
42 | }
43 |
44 | proxy_intercept_errors on;
45 | # invalid keys result in 403
46 | error_page 403 =404 /@404static;
47 | }
48 | ```
49 |
50 | Replace `${bucket_fqdn}` with the URL to your bucket. With this setup, you should upload the contents of the lemmy-ui `dist/` directory to a "directory" named `static/$LEMMY_UI_HASH` in your bucket, where the hash is calculated from the commit you are deploying: `export LEMMY_UI_HASH=$(git rev-parse --short HEAD)`. Note that this setup requires public acl on uploaded files, so it's recommended to limit access to your bucket with an IP allowlist policy.
51 |
52 | 2. An alternative option is to configure sticky sessions in your load balancer. This will ensure that subsequent requests all end up on the same Lemmy-ui server, and as long as your sticky sessions last longer than your upgrade process, most clients should remain functional.
53 | 3. Similarly to sticky sessions, another possibility is to configure ip hash based load balancing.
54 | 4. There is always the option to just employ downtime and upgrade all servers at once (but that's not very fun!)
55 |
56 | ### Lemmy_server
57 |
58 | Lemmy_server can be horizontally scaled, with a few caveats.
59 |
60 | Here's a quick example on how you could start 3 web servers, 3 federation servers and one scheduled task process:
61 |
62 | ```
63 | # scheduled tasks
64 | lemmy_server --disable-http-server --disable-activity-sending
65 | ```
66 |
67 | Or run the Docker container with the `-e LEMMY_DISABLE_HTTP_SERVER=true` and `-e LEMMY_DISABLE_ACTIVITY_SENDING=true` ENV parameters.
68 |
69 | ```
70 | # 3 http servers
71 | lemmy_server --disable-activity-sending --disable-scheduled-tasks
72 | lemmy_server --disable-activity-sending --disable-scheduled-tasks
73 | lemmy_server --disable-activity-sending --disable-scheduled-tasks
74 | ```
75 |
76 | Or run the Docker containers with the `-e LEMMY_DISABLE_ACTIVITY_SENDING=true` and `-e LEMMY_DISABLE_SCHEDULED_TASKS=true` ENV parameters.
77 |
78 | ```
79 | # 3 servers for sending out federation activities
80 | lemmy_server --disable-http-server --disable-scheduled-tasks --federate-process-index=1 --federate-process-count=3
81 | lemmy_server --disable-http-server --disable-scheduled-tasks --federate-process-index=2 --federate-process-count=3
82 | lemmy_server --disable-http-server --disable-scheduled-tasks --federate-process-index=3 --federate-process-count=3
83 | ```
84 |
85 | Or run the Docker containers with the `-e LEMMY_DISABLE_HTTP_SERVER=true`, `-e LEMMY_DISABLE_SCHEDULED_TASKS=true`, `-e LEMMY_FEDERATE_PROCESS_INDEX=1` and `-e LEMMY_FEDERATE_PROCESS_COUNT=3` ENV parameters.
86 |
87 | #### Scheduled tasks
88 |
89 | By default, a Lemmy_server process will run background scheduled tasks, which must be run only on one server. Launching multiple processes with the default configuration will result in multiple duplicated scheduled tasks all starting at the same moment and trying to do the same thing at once.
90 |
91 | To solve this, Lemmy must be started with the `--disable-scheduled-tasks` flag (or `LEMMY_DISABLE_SCHEDULED_TASKS=true`) on all but one instance. In general, there are two approaches:
92 |
93 | 1. Run all your load balanced Lemmy servers with the `--disable-scheduled-tasks` flag, and run one additional Lemmy server without this flag which is not in your load balancer and does not accept any HTTP traffic.
94 | 2. Run one load balanced Lemmy server without the flag, and all other load balanced servers with the flag.
95 |
96 | #### Federation queue
97 |
98 | The persistent federation queue (since 0.19) is split by federated domain and can be processed in equal-size parts run in separate processes. To split the queue up into N processes numbered 1...N, use the arguments `--federate-process-index=i --federate-process-count=N` on each (or `LEMMY_FEDERATE_PROCESS_INDEX=i` and `LEMMY_FEDERATE_PROCESS_COUNT=N` respectively). It is important that each index is is given to exactly one process, otherwise you will get undefined behaviour (missing, dupe federation, crashes).
99 |
100 | Federation processes can be started and stopped at will. They will restart federation to each instance from the last transmitted activity regardless of downtime.
101 |
102 | #### Rolling upgrades
103 |
104 | For most versions, rolling releases have been completely OK, but there have been some cases where a database migration slips in which is NOT backwards compatible (causing any servers running the older version of Lemmy to potentially start generating errors). To be safe, you should always first take a look at [what database migrations are included in a new version](https://github.com/LemmyNet/lemmy/tree/main/migrations), and evaluate whether the changes are for sure safe to apply with running servers.
105 |
106 | Note that even if there are no backwards incompatible migrations between immediate version upgrades (version 1 -> 2 is safe and 2 -> 3 is safe), doing bigger upgrades might cause these same migrations to become backwards incompatible (a server running version 1 might not be compatible with migrations from version 3 in this scenario). **It is generally best to only upgrade one version at a time!**.
107 |
108 | For the scheduled task process, it shouldn't cause any major issues to have two servers running in parallel for a short duration during an upgrade, but shutting the old one down before starting the new one will always be safer.
109 |
110 | ### Pict-rs
111 |
112 | Pict-rs does not scale horizontally as of writing this document. This is due to a dependency on a Sled database, which is a disk based database file.
113 |
114 | The developer of pict-rs has mentioned plans to eventually add Postgres support (which should in theory enable horizontal scaling), but work on this has not yet started.
115 |
116 | Note that by caching pict-rs images (for example, with nginx, or with a CDN), you can really minimize load on the pict-rs server!
117 |
118 | ## Other tips
119 |
120 | 1. Your lemmy-ui servers need to access your backend as well. If you don't want a complicated setup with multiple load balancers, you could just have each of your servers contain nginx, lemmy-ui, and lemmy_server, and just load balance to an nginx on each of your servers
121 | 2. Make sure you're passing the real client ip address to Lemmy, otherwise Lemmy's built in rate limiter will trigger very often (if it rate limits based on your load balancer's IP address, for example). With nginx, you should use something like this in your nginx.conf:
122 |
123 | ```
124 | real_ip_recursive on;
125 | real_ip_header X-Forwarded-For;
126 | set_real_ip_from ;
127 | ```
128 |
129 | 3. The internal Lemmy load balancer works based on an in-memory storage of ip addresses. That means that by adding more Lemmy servers, you are effectively making your rate limits less strict. If you rely on the built in limiter, make sure to adjust the limits accordingly!
130 |
--------------------------------------------------------------------------------
/src/administration/tor_hidden_service.md:
--------------------------------------------------------------------------------
1 | # Running a Tor Hidden Service
2 |
3 | This guide assumes Lemmy has been installed using the official [Docker Compose](install_docker.md) method.
4 |
5 | Note that federation is not currently supported over the Tor network. An existing Lemmy instance is required. This procedure will proxy Lemmy though Tor, but federation tasks are still handled by HTTPS on the open internet.
6 |
7 | [Tor](https://torproject.org) ("The Onion Router") is software designed to circumvent censorship and prevent bad actors from monitoring your activity on the internet by encrypting and distributing network traffic through a decentralized pool of relay servers run by volunteers all over the world.
8 |
9 | A Tor hidden service is only accessible through the Tor network using the `.onion` top-level domain with the official [Tor Browser](https://www.torproject.org/download/), or any client capable of communicating over a SOCKS5 proxy. Hosting a service on the Tor network is a good way to promote digital privacy and internet freedom.
10 |
11 | # Installing Tor
12 |
13 | The official [documentation](https://support.torproject.org/apt/tor-deb-repo/) suggests Ubuntu and Debian users install Tor from the `deb.torproject.org` repository because it always provides the latest stable release of the software.
14 |
15 | **Administrative Access is Required**
16 |
17 | Commands below are expected to be executed as the `root` user. To become root use `sudo -i` or `su -`.
18 |
19 | With `sudo`:
20 |
21 | ```
22 | sudo -i
23 | Password: [authenticate with current user password]
24 | ```
25 |
26 | With `su`:
27 |
28 | ```
29 | su -
30 | Password: [authenticate with root's password]
31 | ```
32 |
33 | Note: To return to your account run: `exit`.
34 |
35 | **Verify your architecture is supported**
36 |
37 | The package repository only supports `amd64`, `arm64`, and `i386` architectures.
38 |
39 | ```
40 | dpkg --print-architecture
41 | ```
42 |
43 | If your architecture is not supported you may want to consider installing Tor [from source](https://community.torproject.org/onion-services/setup/install/#installing-tor-from-source).
44 |
45 | **Install prerequisite packages**
46 |
47 | ```
48 | apt install -y apt-transport-https ca-certificates gpg lsb-release wget
49 | ```
50 |
51 | **Enable the deb.torproject.org repository**
52 |
53 | Configure `apt` to pull packages from `deb.torproject.org`.
54 |
55 | ```
56 | bash -c 'dist=$(lsb_release -s -c); /bin/echo -e \
57 | "deb [signed-by=/usr/share/keyrings/tor-archive-keyring.gpg] \
58 | https://deb.torproject.org/torproject.org $dist main\n\
59 | deb-src [signed-by=/usr/share/keyrings/tor-archive-keyring.gpg] \
60 | https://deb.torproject.org/torproject.org $dist main" \
61 | > /etc/apt/sources.list.d/tor.list'
62 | ```
63 |
64 | **Import deb.torproject.org's GPG signing key**
65 |
66 | ```
67 | wget -qO- https://deb.torproject.org/torproject.org/A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89.asc \
68 | | gpg --dearmor \
69 | | tee /usr/share/keyrings/tor-archive-keyring.gpg >/dev/null
70 | ```
71 |
72 | The signing key ensures the package retrieved from the server was created by `deb.torproject.org`.
73 |
74 | **Install tor**
75 |
76 | ```
77 | apt update && apt install -y tor
78 | ```
79 |
80 | ## Creating a Tor hidden service
81 |
82 | Create a new hidden service directory:
83 |
84 | ```
85 | mkdir /var/lib/tor/hidden_lemmy_service
86 | ```
87 |
88 | Append the following to `/etc/tor/torrc` to tie the hidden service directory to the `tor` daemon:
89 |
90 | ```
91 | HiddenServiceDir /var/lib/tor/hidden_lemmy_service/
92 | HiddenServicePort 80 127.0.0.1:10080
93 | ```
94 |
95 | `HiddenServiceDir [path]` is where `tor` will store data related to the hidden service, and `HiddenServicePort [hidden_service_port] [host_ip:port]` binds a port on the host to a hidden service port on the Tor network.
96 |
97 | ## Enable and start the Tor daemon
98 |
99 | ```
100 | systemctl enable --now tor
101 | ```
102 |
103 | At startup `tor` daemon will automatically populate `/var/lib/tor/hidden_lemmy_service/` with encryption keys, certificates, and assign a hostname for the new service.
104 |
105 | ## Determine your hidden service's hostname
106 |
107 | ```
108 | cat /var/lib/tor/hidden_lemmy_service/hostname
109 | ```
110 |
111 | The `.onion` address contained in this file will be referred to as `HIDDEN_SERVICE_ADDR` from here on.
112 |
113 | # Configure your existing Lemmy instance
114 |
115 | ## Docker compose
116 |
117 | Forward port `10080` from the `proxy` container to the hidden service port `127.0.0.1:10080`. This exposes `10080/tcp` to the local host, and will not be directly accessible from the internet. For context `"80:80"` binds port `80/tcp` (HTTP) to `0.0.0.0:80` on the host. Unless a firewall is configured to block incoming traffic to `80` this will be exposed to other hosts on the local area network (LAN) and/or the open internet.
118 |
119 | **docker-compose.yml**
120 |
121 | ```
122 | services:
123 | # ...
124 | proxy:
125 | # ...
126 | ports:
127 | - "80:80"
128 | - "443:443"
129 | - "127.0.0.1:10080:10080"
130 | ```
131 |
132 | ## Configure NGINX
133 |
134 | Append a new `server {...}` block to handle tor traffic, and add the `Onion-Location` header to the SSL encrypted server exposed to the internet. This header informs Tor Browser users that an equivalent `.onion` site exists on the Tor network by displaying an icon next to the address bar.
135 |
136 | **nginx.conf**
137 |
138 | ```
139 | worker_processes 1;
140 | events {
141 | worker_connections 1024;
142 | }
143 |
144 | http {
145 | # Original configuration listening on port 80
146 | server {
147 | listen 80;
148 | # ...
149 | }
150 |
151 | # Original configuration listening on port 443
152 | server {
153 | listen 443;
154 | # ...
155 | location / {
156 | # Handle Tor Browser's ".onion" link detection
157 | add_header Onion-Location "http://HIDDEN_SERVICE_ADDR$request_uri" always;
158 | # ...
159 | }
160 | }
161 |
162 | # Establish a rate limit for the hidden service address
163 | limit_req_zone $binary_remote_addr zone=HIDDEN_SERVICE_ADDR_ratelimit:10m rate=1r/s;
164 |
165 | # Add tor-specific upstream aliases as a visual aid to
166 | # avoid editing the incorrect server block in the future
167 | upstream lemmy-tor {
168 | server "lemmy:8536";
169 | }
170 | upstream lemmy-ui-tor {
171 | server "lemmy-ui:1234";
172 | }
173 |
174 | # Add a copy of your current internet-facing configuration with
175 | # "listen" and "server_listen" modified to send all traffic
176 | # over the Tor network, incorporating the visual upstream aliases
177 | # above.
178 | server {
179 | # Tell nginx to listen on the hidden service port
180 | listen 10080;
181 |
182 | # Set server_name to the contents of the file:
183 | # /var/lib/tor/hidden_lemmy_service/hostname
184 | server_name HIDDEN_SERVICE_ADDR;
185 |
186 | # Hide nginx version
187 | server_tokens off;
188 |
189 | # Enable compression for JS/CSS/HTML bundle, for improved client load times.
190 | # It might be nice to compress JSON, but leaving that out to protect against potential
191 | # compression+encryption information leak attacks like BREACH.
192 | gzip on;
193 | gzip_types text/css application/javascript image/svg+xml;
194 | gzip_vary on;
195 |
196 | # Various content security headers
197 | add_header Referrer-Policy "same-origin";
198 | add_header X-Content-Type-Options "nosniff";
199 | add_header X-Frame-Options "DENY";
200 | add_header X-XSS-Protection "1; mode=block";
201 |
202 | # Upload limit for pictrs
203 | client_max_body_size 20M;
204 |
205 | # frontend
206 | location / {
207 | # distinguish between ui requests and backend
208 | # don't change lemmy-ui or lemmy here, they refer to the upstream definitions on top
209 | set $proxpass "http://lemmy-ui-tor";
210 |
211 | if ($http_accept = "application/activity+json") {
212 | set $proxpass "http://lemmy-tor";
213 | }
214 | if ($http_accept = "application/ld+json; profile=\"https://www.w3.org/ns/activitystreams\"") {
215 | set $proxpass "http://lemmy-tor";
216 | }
217 | if ($request_method = POST) {
218 | set $proxpass "http://lemmy-tor";
219 | }
220 | proxy_pass $proxpass;
221 |
222 | rewrite ^(.+)/+$ $1 permanent;
223 |
224 | # Send actual client IP upstream
225 | proxy_set_header X-Real-IP $remote_addr;
226 | proxy_set_header Host $host;
227 | proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
228 | }
229 |
230 | # backend
231 | location ~ ^/(api|feeds|nodeinfo|.well-known) {
232 | proxy_pass "http://lemmy-tor";
233 | proxy_http_version 1.1;
234 | proxy_set_header Upgrade $http_upgrade;
235 | proxy_set_header Connection "upgrade";
236 |
237 | # Rate limit
238 | limit_req zone=HIDDEN_SERVICE_ADDR_ratelimit burst=30 nodelay;
239 |
240 | # Add IP forwarding headers
241 | proxy_set_header X-Real-IP $remote_addr;
242 | proxy_set_header Host $host;
243 | proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
244 | }
245 |
246 | # pictrs only - for adding browser cache control.
247 | location ~ ^/(pictrs) {
248 | # allow browser cache, images never update, we can apply long term cache
249 | expires 120d;
250 | add_header Pragma "public";
251 | add_header Cache-Control "public";
252 |
253 | proxy_pass "http://lemmy-tor";
254 | proxy_http_version 1.1;
255 | proxy_set_header Upgrade $http_upgrade;
256 | proxy_set_header Connection "upgrade";
257 |
258 | # Rate limit
259 | limit_req zone=HIDDEN_SERVICE_ADDR_ratelimit burst=30 nodelay;
260 |
261 | # Add IP forwarding headers
262 | proxy_set_header X-Real-IP $remote_addr;
263 | proxy_set_header Host $host;
264 | proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
265 | }
266 |
267 | # Redirect pictshare images to pictrs
268 | location ~ /pictshare/(.*)$ {
269 | return 301 /pictrs/image/$1;
270 | }
271 | }
272 |
273 | # ...
274 | }
275 | ```
276 |
277 | # Apply the configuration(s)
278 |
279 | Restart all services associated with your Lemmy instance:
280 |
281 | ```
282 | docker compose down
283 | docker compose up -d
284 | ```
285 |
286 | # Test connectivity over Tor
287 |
288 | Using `torsocks`, verify your hidden service is available on the Tor network.
289 |
290 | ```
291 | torsocks curl -vI http://HIDDEN_SERVICE_ADDR
292 | * Trying 127.*.*.*:80...
293 | * Connected to HIDDEN_SERVICE_ADDR (127.*.*.*) port 80 (#0)
294 | > HEAD / HTTP/1.1
295 | > Host: HIDDEN_SERVICE_ADDR
296 | > User-Agent: curl/7.76.1
297 | > Accept: */*
298 | >
299 | * Mark bundle as not supporting multiuse
300 | < HTTP/1.1 200 OK
301 | HTTP/1.1 200 OK
302 | < Server: nginx
303 | Server: nginx
304 | < Date: Wed, 07 Jun 2023 17:06:00 GMT
305 | Date: Wed, 07 Jun 2023 17:06:00 GMT
306 | < Content-Type: text/html; charset=utf-8
307 | Content-Type: text/html; charset=utf-8
308 | < Content-Length: 98487
309 | Content-Length: 98487
310 | < Connection: keep-alive
311 | Connection: keep-alive
312 | < Vary: Accept-Encoding
313 | Vary: Accept-Encoding
314 | < X-Powered-By: Express
315 | X-Powered-By: Express
316 | < Content-Security-Policy: default-src 'self'; manifest-src *; connect-src *; img-src * data:; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline'; form-action 'self'; base-uri 'self'; frame-src *
317 | Content-Security-Policy: default-src 'self'; manifest-src *; connect-src *; img-src * data:; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline'; form-action 'self'; base-uri 'self'; frame-src *
318 | < ETag: W/"180b7-EC9iFYAIlbnN8zHCayBwL3wAm64"
319 | ETag: W/"180b7-EC9iFYAIlbnN8zHCayBwL3wAm64"
320 | < Referrer-Policy: same-origin
321 | Referrer-Policy: same-origin
322 | < X-Content-Type-Options: nosniff
323 | X-Content-Type-Options: nosniff
324 | < X-Frame-Options: DENY
325 | X-Frame-Options: DENY
326 | < X-XSS-Protection: 1; mode=block
327 | X-XSS-Protection: 1; mode=block
328 |
329 | <
330 | * Connection #0 to host HIDDEN_SERVICE_ADDR left intact
331 | ```
332 |
333 | ## Logging behavior
334 |
335 | Hidden service traffic will appear to originate from the `lemmyexternalproxy` docker network instead of an internet IP. Docker's default network address pool is `172.17.0.0/16`.
336 |
337 | ```
338 | docker compose logs -f proxy
339 | lemmy-proxy-1 | 172.*.0.1 - - # ...
340 | lemmy-proxy-1 | 172.*.0.1 - - # ...
341 | lemmy-proxy-1 | 172.*.0.1 - - # ...
342 | ```
343 |
--------------------------------------------------------------------------------
/src/administration/troubleshooting.md:
--------------------------------------------------------------------------------
1 | # Troubleshooting
2 |
3 | Different problems that can occur on a new instance, and how to solve them.
4 |
5 | Many Lemmy features depend on a correct reverse proxy configuration. Make sure yours is equivalent to our [nginx config](https://github.com/LemmyNet/lemmy-ansible/blob/main/templates/nginx.conf).
6 |
7 | ## General
8 |
9 | ### Logs
10 |
11 | For frontend issues, check the [browser console](https://webmasters.stackexchange.com/a/77337) for any error messages.
12 |
13 | For server logs, run `docker compose logs -f lemmy` in your installation folder. You can also do `docker compose logs -f lemmy lemmy-ui pictrs` to get logs from different services.
14 |
15 | If that doesn't give enough info, try changing the line `RUST_LOG=error` in `docker-compose.yml` to `RUST_LOG=info` or `RUST_LOG=trace`, then do `docker compose restart lemmy`.
16 |
17 | ### Creating admin user doesn't work
18 |
19 | Make sure that websocket is working correctly, by checking the browser console for errors. In nginx, the following headers are important for this:
20 |
21 | ```
22 | proxy_http_version 1.1;
23 | proxy_set_header Upgrade $http_upgrade;
24 | proxy_set_header Connection "upgrade";
25 | ```
26 |
27 | ### Rate limit error when many users access the site
28 |
29 | Check that the headers `X-Real-IP` and `X-Forwarded-For` are sent to Lemmy by the reverse proxy. Otherwise, it will count all actions towards the rate limit of the reverse proxy's IP. In nginx it should look like this:
30 |
31 | ```
32 | proxy_set_header X-Real-IP $remote_addr;
33 | proxy_set_header Host $host;
34 | proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
35 | ```
36 |
37 | ## Federation
38 |
39 | ### Other instances can't fetch local objects (community, post, etc)
40 |
41 | Your reverse proxy (eg nginx) needs to forward requests with header `Accept: application/activity+json` to the backend. This is handled by the following lines:
42 |
43 | ```
44 | set $proxpass "http://0.0.0.0:{{ lemmy_ui_port }}";
45 | if ($http_accept = "application/activity+json") {
46 | set $proxpass "http://0.0.0.0:{{ lemmy_port }}";
47 | }
48 | if ($http_accept = "application/ld+json; profile=\"https://www.w3.org/ns/activitystreams\"") {
49 | set $proxpass "http://0.0.0.0:{{ lemmy_port }}";
50 | }
51 | proxy_pass $proxpass;
52 | ```
53 |
54 | You can test that it works correctly by running the following commands, all of them should return valid JSON:
55 |
56 | ```
57 | curl -H "Accept: application/activity+json" https://your-instance.com/u/some-local-user
58 | curl -H "Accept: application/activity+json" https://your-instance.com/c/some-local-community
59 | curl -H "Accept: application/activity+json" https://your-instance.com/post/123 # the id of a local post
60 | curl -H "Accept: application/activity+json" https://your-instance.com/comment/123 # the id of a local comment
61 | ```
62 |
63 | ### Fetching remote objects works, but posting/commenting in remote communities fails
64 |
65 | Check that [federation is allowed on both instances](federation_getting_started.md).
66 |
67 | Also ensure that the time is correct on your server. Activities are signed with a timestamp, and will be discarded if it is off by more than one hour.
68 |
69 | It is possible that federation requests to `/inbox` are blocked by tools such as Cloudflare. The sending instance can find HTTP errors with the following steps:
70 |
71 | - If you use a [separate container for outgoing federation](horizontal_scaling.md), you need to apply the following steps to that container only
72 | - Set `RUST_LOG=lemmy_federate=trace` for Lemmy
73 | - Reload the new configuration: `docker compose up -d`
74 | - Search for messages containing the target instance domain: `docker compose logs -f --tail=100 lemmy | grep -F lemm.ee -C 10`
75 | - You also may have to reset the fail count for the target instance (see below)
76 |
77 | ### Reset federation fail count for instance
78 |
79 | If federation sending to a specific instance has been failing consistently, Lemmy will slow down sending using exponential backoff. For testing it can be useful to reset this and make Lemmy send activities immediately. To do this use the following steps:
80 |
81 | - Stop Lemmy, or specifically the container for outgoing federation `docker compose stop lemmy`
82 | - Enter SQL command line: `sudo docker compose exec postgres psql -U lemmy`
83 | - Reset failure count via SQL:
84 |
85 | ```sql
86 | update federation_queue_state
87 | set fail_count = 0
88 | from instance
89 | where instance.id = federation_queue_state.instance_id
90 | and instance.domain = 'lemm.ee';
91 | ```
92 |
93 | - Exit SQL command line with `\q`, then restart Lemmy: `docker compose start lemmy`
94 |
95 | ### Other instances don't receive actions reliably
96 |
97 | Lemmy uses one queue per federated instance to send out activities. Search the logs for "Federation state" for summaries. Errors will also be logged.
98 |
99 | For details, execute this SQL query:
100 |
101 | ```sql
102 | select domain,currval('sent_activity_id_seq') as latest_id, last_successful_id,fail_count,last_retry from federation_queue_state
103 | join instance on instance_id = instance.id order by last_successful_id asc;
104 | ```
105 |
106 | You will see a table like the following:
107 |
108 | | domain | latest_id | last_successful_id | fail_count | last_retry |
109 | | -------------------------- | --------- | ------------------ | ---------- | ----------------------------- |
110 | | toad.work | 6837196 | 6832351 | 14 | 2023-07-12 21:42:22.642379+00 |
111 | | lemmy.deltaa.xyz | 6837196 | 6837196 | 0 | 1970-01-01 00:00:00+00 |
112 | | battleangels.net | 6837196 | 6837196 | 0 | 1970-01-01 00:00:00+00 |
113 | | social.fbxl.net | 6837196 | 6837196 | 0 | 1970-01-01 00:00:00+00 |
114 | | mastodon.coloradocrest.net | 6837196 | 6837196 | 0 | 1970-01-01 00:00:00+00 |
115 |
116 | This will show you exactly which instances are up to date or not.
117 |
118 | You can also use this website which will help you monitor this without admin rights, and also let others see it
119 |
120 | https://phiresky.github.io/lemmy-federation-state/site
121 |
122 | ### You don't receive actions reliably
123 |
124 | Due to the lemmy queue, remote lemmy instances will be sending apub sync actions serially to you. If your server rate of processing them is slower than the rate the origin server is sending them, when visiting the [lemmy-federation-state](https://phiresky.github.io/lemmy-federation-state/site) for the remote server, you'll see your instance in the "lagging behind" section.
125 |
126 | This can be avoided by setting the config value `federation.concurrent_sends_per_instance` to a value greater than 1 on the sending instance.
127 |
128 | Typically the speed at which you process an incoming action should be less than 100ms. If this is higher, this might signify problems with your database performance or your networking setup.
129 |
130 | Note that even apub action ingestion speed which seems sufficient for most other instances, might become insufficient if the origin server is receiving actions from their userbase faster than you can process them. I.e. if the origin server receives 10 actions per second, but you can only process 8 actions per second, you'll inevitably start falling behind that one server only.
131 |
132 | These steps might help you diagnose this.
133 |
134 | #### Check processing time on the loadbalancer
135 |
136 | Check how long a request takes to process on the backend. In haproxy for example, the following command will show you the time it takes for apub actions to complete
137 |
138 | ```bash
139 | tail -f /var/log/haproxy.log | grep "POST \/inbox"
140 | ```
141 |
142 | [See here for nginx](https://www.nginx.com/blog/using-nginx-logging-for-application-performance-monitoring/)
143 |
144 | If these actions take more than 100ms, you might want to investigate deeper.
145 |
146 | #### Check your Database performance
147 |
148 | Ensure that it's not very high in CPU or RAM utilization.
149 |
150 | Afterwards check for slow queries. If you regularly see common queries with high max and mean exec time, it might signify your database is struggling. The below SQL query will show you all queries (you will need `pg_stat_statements` [enabled](https://www.postgresql.org/docs/current/pgstatstatements.html))
151 |
152 | ```sql
153 | \x auto
154 | SELECT user,query,max_exec_time,mean_exec_time,calls FROM pg_stat_statements WHERE max_exec_time > 10 AND CALLS > 100 ORDER BY max_exec_time DESC;
155 | ```
156 |
157 | If you see very high time on inserts, you might want to consider disabling `synchronous_commit` to see if this helps.
158 |
159 | #### Check your backend performance.
160 |
161 | Like the DB, if the server where your lemmy rust backend is running is overloaded, you might see such an impact
162 |
163 | #### Check your Network Layout
164 |
165 | If your backend and database appear to be in good condition, it might be that your issue is network based.
166 |
167 | One problem can occur is your backend and your database are not in the same server and are too far from each other in geographic location. Due to the amount of DB queries performed for each apub sync request, even a small amount of latency can quickly add up.
168 |
169 | Check the latency between your rust backend and your DB using ping
170 |
171 | ```bash
172 | ping your_database_ip
173 | ```
174 |
175 | if the `time` you see if above 1-2ms, this can start causing such delays. In that case, you might want to consider moving your backend closer to your DB geographically, so that your latency is below 2ms
176 |
177 | Note that your external loadbalancer(s) (if any) do not necessarily need to be closer to the DB, as they do not do multiple small DB requests.
178 |
179 | ## Downgrading
180 |
181 | If you upgraded your instance to a newer version (by mistake or planned) and need to downgrade it. Often you need to reverse database changes as well.
182 |
183 | First you need to figure out what SQL changes happened between your upgraded version, and the one you're downgrading. Then in that diff, check which files were added in the `migrations` dir.
184 |
185 | Let's say that for the migration you're doing, the following were added
186 |
187 | ```
188 | 2023-10-24-131607_proxy_links
189 | 2023-10-27-142514_post_url_content_type
190 | 2023-12-19-210053_tolerable-batch-insert-speed
191 | 2023-12-22-040137_make-mixed-sorting-directions-work-with-tuple-comparison
192 | 2024-01-05-213000_community_aggregates_add_local_subscribers
193 | 2024-01-15-100133_local-only-community
194 | 2024-01-22-105746_lemmynsfw-changes
195 | 2024-01-25-151400_remove_auto_resolve_report_trigger
196 | 2024-02-15-171358_default_instance_sort_type
197 | 2024-02-27-204628_add_post_alt_text
198 | 2024-02-28-144211_hide_posts
199 | ```
200 |
201 | Each of these folders contains a `down.sql` file. We need to run that against our postgresql DB to rollback those DB changes.
202 |
203 | 1. Stop your lemmy backend, and take a backup of your DB.
204 | 1. Copy the `migrations` folder to your DB container or server
205 | 1. Acquire a shell in your postgresql container or server and switch to the `postgres` user
206 | 1. Run each relevant script with this command
207 | ```bash
208 | downfolder=2024-02-28-144211_hide_posts
209 | psql -d lemmy -a -f /path/to/migrations/${downfolder}/down.sql
210 | ```
211 | Alternatively, copy the content of the file and paste into a psql session
212 | 1. You now need to clean the `__diesel_schema_migrations` table from the migration records, so that they will be correctly applied the next time you upgrade. You can use this command to sort them
213 | ```sql
214 | select * from __diesel_schema_migrations ORDER BY run_on ASC;
215 | ```
216 | You have to delete the entries in that table which match the current timestamp you applied them (This should typically be any time in the past few minutes)
217 | ```sql
218 | delete from __diesel_schema_migrations where version='20240228144211';
219 | ```
220 | 1. You should now be able to start your lemmy in the previous version
221 |
222 | ## UI randomly slow or offline
223 |
224 | If you notice that your lemmy-ui sometimes becomes sluggish or unresponsive over a period of minutes/hours and then it passes, you might be getting targeted by scraping bots.
225 |
226 | There's a lot of scraping bots online and they can easily overwhelm your site when they're behaving too "greedily". Unfortunately the existing lemmy-ui has a habit of falling over when polled too eagerly, while the backend still continues to work.
227 |
228 | A solution is to cache responses in nginx as seen in lemmy-ansible [here](https://github.com/LemmyNet/lemmy-ansible/blob/1.5.3/templates/nginx.conf#L2-L3) and [here](https://github.com/LemmyNet/lemmy-ansible/blob/1.5.3/templates/nginx.conf#L66-L71). This way lemmy-ui doesn't have to generate all responses from scratch, which reduces CPU load. However it won't help if a single crawler goes through thousands of unique urls in a short time.
229 |
230 | Another option is to block the scraper's user agents. To do so, you can modify your `nginx_internal.conf` to block some of the usual suspects, with this line under `server`
231 |
232 | ```bash
233 | if ($http_user_agent ~* " Bytedance|Bytespider|Amazonbot|ClaudeBot") { return 444; }
234 | ```
235 |
236 | This is an example blocking some the well-known misbehaving bots, but there are many more more. To discover the ones affecting you, you can use the following bash script at your lemmy backend (where your docker compose is) to enumerate any agents which are hitting you too much.
237 |
238 | ```bash
239 | docker-compose logs --tail=10000 proxy |
240 | grep -o '"[^"]*"$' | # Extract the last quoted string (user agent)
241 | grep -v '^"$' | # Remove empty quotes
242 | tr -d '"' | # Remove the quotes
243 | sort | # Sort the user agents
244 | uniq -c | # Count unique occurrences
245 | sort -rn | # Sort numerically in reverse order
246 | head -n 10 # Show top 10 results
247 | ```
248 |
249 | This will parse the last 10K log entries in your nginx internal proxy and show the agents which cause the most hits. This should give a good indicator of which agents are potentially misbehaving and you can proceed to block those as well by adding their names to the list above.
250 |
--------------------------------------------------------------------------------
/src/administration/from_scratch.md:
--------------------------------------------------------------------------------
1 | # Install from Scratch
2 |
3 | These instructions are written for Ubuntu 20.04 / Ubuntu 22.04. They are particularly useful when you'd like to setup a Lemmy container (e.g. LXC on Proxmox) and cannot use Docker.
4 |
5 | Lemmy is built from source in this guide, so this may take a while, especially on slow devices. For example, Lemmy v0.18.5 takes around 7 minutes to build on a quad core VPS.
6 |
7 | Installing and configuring Lemmy using this guide takes about 60-90 minutes. You might need to make yourself a fresh cup of coffee before you start.
8 |
9 | ## Installation
10 |
11 | ### Database
12 |
13 | For Ubuntu 20.04 the shipped PostgreSQL version is 12 which is not supported by Lemmy. So let's set up a newer one.
14 | The most recent stable version of PostgreSQL is 16 at the time of writing this guide.
15 |
16 | #### Install dependencies
17 |
18 | ```
19 | sudo apt install -y wget ca-certificates pkg-config
20 | wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
21 | sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg main" >> /etc/apt/sources.list.d/pgdg.list'
22 | sudo apt update
23 | sudo apt install libssl-dev libpq-dev postgresql
24 | ```
25 |
26 | #### Setup Lemmy database
27 |
28 | Replace db-passwd with a unique password of your choice in the commands below.
29 |
30 | ```
31 | sudo -iu postgres psql -c "CREATE USER lemmy WITH PASSWORD 'db-passwd';"
32 | sudo -iu postgres psql -c "CREATE DATABASE lemmy WITH OWNER lemmy;"
33 | ```
34 |
35 | If you're migrating from an older version of Lemmy, the following might be required.
36 |
37 | ```
38 | sudo -iu postgres psql -c "ALTER USER lemmy WITH SUPERUSER;"
39 | ```
40 |
41 | Tune your PostgreSQL settings to match your hardware via [this guide](https://pgtune.leopard.in.ua/#/)
42 |
43 | #### Setup md5 auth
44 |
45 | Your Postgres config might need to be edited to allow password authentication instead of peer authentication. Simply add the following to your `pg_hba.conf`:
46 |
47 | ```
48 | local lemmy lemmy md5
49 | ```
50 |
51 | ### Install Rust
52 |
53 | For the Rust compiles, it is ideal to use a non-privileged Linux account on your system. Install Rust by following the instructions on [Rustup](https://rustup.rs/) (using a non-privileged Linux account, it will install file in that user's home folder for rustup and cargo).
54 |
55 | protobuf-compiler may be required for Ubuntu 20.04 or 22.04 installs, please report testing in lemmy-docs issues.
56 |
57 | ```
58 | sudo apt install protobuf-compiler gcc
59 | ```
60 |
61 | ### Setup pict-rs (Optional)
62 |
63 | You can skip this section if you don't require image hosting, but **NOTE that Lemmy-ui will still allow users to attempt uploading images even if pict-rs is not configured. In this situation, the upload will fail and users will receive technical error messages.**
64 |
65 | Lemmy supports image hosting using [pict-rs](https://git.asonix.dog/asonix/pict-rs/). We need to install a couple of dependencies for this. It requires the `magick` command which comes with Imagemagick version 7, but Ubuntu 20.04 only comes with Imagemagick 6. So you need to install that command manually, eg from the [official website](https://imagemagick.org/script/download.php#linux).
66 |
67 | **NOTE: on standard LXC containers an AppImage-based ImageMagick installation [will not work properly](https://github.com/LemmyNet/lemmy/issues/4112). It uses FUSE which will emit "permission denied" errors when trying to upload an image through pict-rs. You must use alternative installation methods, such as [imei.sh](https://github.com/SoftCreatR/imei).**
68 |
69 | #### AppImage-based installation of ImageMagick
70 |
71 | ```
72 | sudo apt install ffmpeg exiftool libgexiv2-dev --no-install-recommends
73 | # save the file to a working folder it can be verified before copying to /usr/bin/
74 | wget https://download.imagemagick.org/ImageMagick/download/binaries/magick
75 | # compare hash with the "message digest" on the official page linked above
76 | sha256sum magick
77 | sudo mv magick /usr/bin/
78 | sudo chmod 755 /usr/bin/magick
79 | ```
80 |
81 | #### imei.sh-based installation of ImageMagick
82 |
83 | Follow the instructions from the [official imei.sh page on GitHub](https://github.com/SoftCreatR/imei)
84 |
85 | #### Standalone pict-rs installation
86 |
87 | Since we're building stuff from source here, let's do the same for pict-rs. Follow the [instructions here](https://git.asonix.dog/asonix/pict-rs/#user-content-compile-from-source).
88 |
89 | ### Lemmy Backend
90 |
91 | #### Build the backend
92 |
93 | Create user account on Linux for the lemmy_server application
94 |
95 | ```
96 | sudo adduser lemmy --system --disabled-login --no-create-home --group
97 | ```
98 |
99 | Compile and install Lemmy, given the from-scratch intention, this will be done via GitHub checkout. This can be done by a normal unprivledged user (using the same Linux account you used for rustup).
100 |
101 | ```bash
102 | git clone https://github.com/LemmyNet/lemmy.git lemmy
103 | cd lemmy
104 | git checkout 0.18.5
105 | git submodule init
106 | git submodule update
107 | ```
108 |
109 | Then build Lemmy with the following command:
110 |
111 | ```
112 | cargo build --release
113 | ```
114 |
115 | #### Deployment
116 |
117 | Because we should [follow the Linux way](https://tldp.org/LDP/Linux-Filesystem-Hierarchy/html/opt.html), we should use the `/opt` directory to colocate the backend, frontend and pict-rs.
118 |
119 | ```
120 | sudo mkdir /opt/lemmy
121 | sudo mkdir /opt/lemmy/lemmy-server
122 | sudo mkdir /opt/lemmy/pictrs
123 | sudo mkdir /opt/lemmy/pictrs/files
124 | sudo mkdir /opt/lemmy/pictrs/sled-repo
125 | sudo mkdir /opt/lemmy/pictrs/old
126 | sudo chown -R lemmy:lemmy /opt/lemmy
127 | ```
128 |
129 | Note that it might not be the most obvious thing, but **creating the pictrs directories is not optional**.
130 |
131 | Then copy the binary.
132 |
133 | ```
134 | sudo cp target/release/lemmy_server /opt/lemmy/lemmy-server/lemmy_server
135 | ```
136 |
137 | #### Configuration
138 |
139 | This is the minimal Lemmy config, put this in `/opt/lemmy/lemmy-server/lemmy.hjson` (see [here](https://github.com/LemmyNet/lemmy/blob/main/config/defaults.hjson) for more config options).
140 |
141 | ```hjson
142 | {
143 | database: {
144 | # put your db-passwd from above
145 | password: "db-passwd"
146 | }
147 | # replace with your domain
148 | hostname: example.com
149 | bind: "127.0.0.1"
150 | federation: {
151 | enabled: true
152 | }
153 | # remove this block if you don't require image hosting
154 | pictrs: {
155 | url: "http://localhost:8080/"
156 | }
157 | }
158 | ```
159 |
160 | Set the correct owner
161 |
162 | ```
163 | chown -R lemmy:lemmy /opt/lemmy/
164 | ```
165 |
166 | #### Server daemon
167 |
168 | Add a systemd unit file, so that Lemmy automatically starts and stops, logs are handled via journalctl etc. Put this file into /etc/systemd/system/lemmy.service.
169 |
170 | ```
171 | [Unit]
172 | Description=Lemmy Server
173 | After=network.target
174 |
175 | [Service]
176 | User=lemmy
177 | ExecStart=/opt/lemmy/lemmy-server/lemmy_server
178 | Environment=LEMMY_CONFIG_LOCATION=/opt/lemmy/lemmy-server/lemmy.hjson
179 | Environment=PICTRS_ADDR=127.0.0.1:8080
180 | Environment=RUST_LOG="info"
181 | Restart=on-failure
182 | WorkingDirectory=/opt/lemmy
183 |
184 | # Hardening
185 | ProtectSystem=yes
186 | PrivateTmp=true
187 | MemoryDenyWriteExecute=true
188 | NoNewPrivileges=true
189 |
190 | [Install]
191 | WantedBy=multi-user.target
192 | ```
193 |
194 | If you need debug output in the logs, change the RUST_LOG line in the file above to
195 |
196 | ```
197 | Environment=RUST_LOG="debug,lemmy_server=debug,lemmy_api=debug,lemmy_api_common=debug,lemmy_api_crud=debug,lemmy_apub=debug,lemmy_db_schema=debug,lemmy_db_views=debug,lemmy_db_views_actor=debug,lemmy_db_views_moderator=debug,lemmy_routes=debug,lemmy_utils=debug,lemmy_websocket=debug"
198 | ```
199 |
200 | Then run
201 |
202 | ```
203 | sudo systemctl daemon-reload
204 | sudo systemctl enable lemmy
205 | sudo systemctl start lemmy
206 | ```
207 |
208 | If you did everything right, the Lemmy logs from `sudo journalctl -u lemmy` should show "Starting http server at 127.0.0.1:8536". You can also run `curl localhost:8536/api/v3/site` which should give a successful response, looking like `{"site_view":null,"admins":[],"banned":[],"online":0,"version":"unknown version","my_user":null,"federated_instances":null}`. For pict-rs, run `curl 127.0.0.1:8080` and ensure that it outputs nothing (particularly no errors).
209 |
210 | ### Lemmy Front-end (lemmy-ui)
211 |
212 | #### Install dependencies
213 |
214 | Nodejs in Ubuntu 20.04 / Ubuntu 22.04 repos are too old, so let's install Node 20.
215 |
216 | ```
217 | # nodejs
218 | sudo apt install -y ca-certificates curl gnupg
219 | sudo mkdir -p /etc/apt/keyrings
220 | curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | sudo gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg
221 | echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_20.x nodistro main" | sudo tee /etc/apt/sources.list.d/nodesource.list
222 |
223 | sudo apt update
224 | sudo apt install nodejs
225 |
226 | # pnpm
227 | npm i -g pnpm
228 |
229 | ```
230 |
231 | #### Build the front-end
232 |
233 | Clone the git repo, checkout the version you want (0.18.5 in this case), and compile it.
234 |
235 | ```
236 | # dont compile as admin
237 | cd /opt/lemmy
238 | sudo -u lemmy bash
239 | git clone https://github.com/LemmyNet/lemmy-ui.git --recursive
240 | cd lemmy-ui
241 | git checkout 0.18.5 # replace with the version you want to install
242 | pnpm i
243 | pnpm build:prod
244 | exit
245 | ```
246 |
247 | #### UI daemon
248 |
249 | Add another systemd unit file, this time for lemmy-ui. You need to replace `example.com` with your actual domain. Put the file in `/etc/systemd/system/lemmy-ui.service`
250 |
251 | ```
252 | [Unit]
253 | Description=Lemmy UI
254 | After=lemmy.service
255 | Before=nginx.service
256 |
257 | [Service]
258 | User=lemmy
259 | WorkingDirectory=/opt/lemmy/lemmy-ui
260 | ExecStart=/usr/bin/node dist/js/server.js
261 | Environment=LEMMY_UI_LEMMY_INTERNAL_HOST=localhost:8536
262 | Environment=LEMMY_UI_LEMMY_EXTERNAL_HOST=example.com
263 | Environment=LEMMY_UI_HTTPS=true
264 | Restart=on-failure
265 |
266 | # Hardening
267 | ProtectSystem=full
268 | PrivateTmp=true
269 | NoNewPrivileges=true
270 |
271 | [Install]
272 | WantedBy=multi-user.target
273 | ```
274 |
275 | More UI-related variables can be [found here](https://github.com/LemmyNet/lemmy-ui#configuration).
276 |
277 | Then run.
278 |
279 | ```
280 | sudo systemctl daemon-reload
281 | sudo systemctl enable lemmy-ui
282 | sudo systemctl start lemmy-ui
283 | ```
284 |
285 | If everything went right, the command `curl -I localhost:1234` should show `200 OK` at the top.
286 |
287 | ### Configure reverse proxy and TLS
288 |
289 | Install dependencies
290 |
291 | ```bash
292 | sudo apt install nginx certbot python3-certbot-nginx
293 | ```
294 |
295 | Request Let's Encrypt TLS certificate (just follow the instructions)
296 |
297 | ```bash
298 | sudo certbot certonly --nginx
299 | ```
300 |
301 | Let's Encrypt certificates should be renewed automatically, so add the line below to your crontab, by running `sudo crontab -e`. Replace `example.com` with your actual domain.
302 |
303 | ```
304 | @daily certbot certonly --nginx --cert-name example.com -d example.com --deploy-hook 'nginx -s reload'
305 | ```
306 |
307 | Finally, add the Nginx virtual host config file. Copy paste the file below to `/etc/nginx/sites-enabled/lemmy.conf`
308 |
309 | ```
310 | limit_req_zone $binary_remote_addr zone={{domain}}_ratelimit:10m rate=1r/s;
311 |
312 | server {
313 | listen 80;
314 | listen [::]:80;
315 | server_name {{domain}};
316 | location /.well-known/acme-challenge/ {
317 | root /var/www/certbot;
318 | }
319 | location / {
320 | return 301 https://$host$request_uri;
321 | }
322 | }
323 |
324 | server {
325 | listen 443 ssl http2;
326 | listen [::]:443 ssl http2;
327 | server_name {{domain}};
328 |
329 | ssl_certificate /etc/letsencrypt/live/{{domain}}/fullchain.pem;
330 | ssl_certificate_key /etc/letsencrypt/live/{{domain}}/privkey.pem;
331 |
332 | # Various TLS hardening settings
333 | # https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html
334 | ssl_protocols TLSv1.2 TLSv1.3;
335 | ssl_prefer_server_ciphers on;
336 | ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
337 | ssl_session_timeout 10m;
338 | ssl_session_cache shared:SSL:10m;
339 | ssl_session_tickets on;
340 | ssl_stapling on;
341 | ssl_stapling_verify on;
342 |
343 | # Hide nginx version
344 | server_tokens off;
345 |
346 | # Enable compression for JS/CSS/HTML bundle, for improved client load times.
347 | # It might be nice to compress JSON, but leaving that out to protect against potential
348 | # compression+encryption information leak attacks like BREACH.
349 | gzip on;
350 | gzip_types text/css application/javascript image/svg+xml;
351 | gzip_vary on;
352 |
353 | # Only connect to this site via HTTPS for the two years
354 | add_header Strict-Transport-Security "max-age=63072000";
355 |
356 | # Various content security headers
357 | add_header Referrer-Policy "same-origin";
358 | add_header X-Content-Type-Options "nosniff";
359 | add_header X-Frame-Options "DENY";
360 | add_header X-XSS-Protection "1; mode=block";
361 |
362 | # Upload limit for pictrs
363 | client_max_body_size 20M;
364 |
365 | # frontend
366 | location / {
367 | # The default ports:
368 |
369 | set $proxpass "http://0.0.0.0:1234";
370 | if ($http_accept ~ "^application/.*$") {
371 | set $proxpass "http://0.0.0.0:8536";
372 | }
373 | if ($request_method = POST) {
374 | set $proxpass "http://0.0.0.0:8536";
375 | }
376 | proxy_pass $proxpass;
377 |
378 | rewrite ^(.+)/+$ $1 permanent;
379 |
380 | # Send actual client IP upstream
381 | proxy_set_header X-Real-IP $remote_addr;
382 | proxy_set_header Host $host;
383 | proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
384 | }
385 |
386 | # backend
387 | location ~ ^/(api|pictrs|feeds|nodeinfo|.well-known) {
388 | proxy_pass http://0.0.0.0:8536;
389 | proxy_http_version 1.1;
390 | proxy_set_header Upgrade $http_upgrade;
391 | proxy_set_header Connection "upgrade";
392 |
393 | # Rate limit
394 | limit_req zone={{domain}}_ratelimit burst=30 nodelay;
395 |
396 | # Add IP forwarding headers
397 | proxy_set_header X-Real-IP $remote_addr;
398 | proxy_set_header Host $host;
399 | proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
400 | }
401 | }
402 |
403 | access_log /var/log/nginx/access.log combined;
404 | ```
405 |
406 | And then replace some variables in the file. Put your actual domain instead of example.com
407 |
408 | ```
409 | sudo sed -i -e 's/{{domain}}/example.com/g' /etc/nginx/sites-enabled/lemmy.conf
410 | sudo systemctl reload nginx
411 | ```
412 |
413 | Now open your Lemmy domain in the browser, and it should show you a configuration screen. Use it to create the first admin user and the default community.
414 |
415 | ## Upgrading
416 |
417 | ### Lemmy
418 |
419 | Compile and install lemmy_server changes. This compile can be done by a normal unprivledged user (using the same Linux account you used for rustup and first install of Lemmy).
420 |
421 | ```
422 | rustup update
423 | cd lemmy
424 | git checkout main
425 | git pull --tags
426 | git checkout 0.18.5 # replace with version you are updating to
427 | git submodule update
428 | cargo build --release
429 | # copy compiled binary to destination
430 | # the old file will be locked by the already running application, so this sequence is recommended:
431 | sudo -- sh -c 'systemctl stop lemmy && cp target/release/lemmy_server /opt/lemmy/lemmy-server/lemmy_server && systemctl start lemmy'
432 | ```
433 |
434 | ### Lemmy UI
435 |
436 | ```
437 | cd /opt/lemmy/lemmy-ui
438 | sudo -u lemmy bash
439 | git checkout main
440 | git pull --tags
441 | git checkout 0.18.5 # replace with the version you are updating to
442 | git submodule update
443 | pnpm install
444 | pnpm build:prod
445 | exit
446 | sudo systemctl restart lemmy-ui
447 | ```
448 |
449 | ### Pict-rs
450 |
451 | Refer to [pict-rs documentation](https://git.asonix.dog/asonix/pict-rs) for instructions on upgrading.
452 |
--------------------------------------------------------------------------------
/src/contributors/05-federation.md:
--------------------------------------------------------------------------------
1 | # Federation
2 |
3 | Lemmy uses the ActivityPub protocol for communication between servers. If you are unfamiliar with the protocol, you can start by reading the [resource links](06-resources.md#activitypub-resources). This document explains how to interact with it from other projects.
4 |
5 | In Lemmy we use some specific terms to refer to ActivityPub items. They are essentially our specific implementations of well-known ActivityPub concepts:
6 |
7 | - Community: `Group`
8 | - User: `Person`
9 | - Post: `Page`
10 | - Comment: `Note`
11 |
12 | Almost every action in Lemmy happens inside a group. The Federation Enhancement Proposal [Group Federation](https://codeberg.org/fediverse/fep/src/branch/main/feps/fep-1b12.md) gives a high-level overview how this works. The generic federation logic is implemented in the [activitypub-federation](https://github.com/LemmyNet/activitypub-federation-rust) library. It can also be used by other Rust projects.
13 |
14 | Sometimes you will see a notation like `Create/Note`. This refers to a `Create` activity with a `Note` as object.
15 |
16 | Below are explanations and examples for all actors, objects and activities from Lemmy. These include many optional fields which you can safely ignore.
17 |
18 | If you have trouble to make your project federate with Lemmy, feel free to [open an issue](https://github.com/LemmyNet/lemmy/issues). Make sure to include the specific federation messages you are sending, and any errors returned by Lemmy.
19 |
20 | ## Context
21 |
22 | ```json
23 | "@context": [
24 | "https://join-lemmy.org/context.json",
25 | "https://www.w3.org/ns/activitystreams"
26 | ],
27 | ```
28 |
29 | The context is identical for all activities and objects.
30 |
31 | ## Actors
32 |
33 | ### Community
34 |
35 | An automated actor. Users can send posts or comments to it, which the community forwards to its followers in the form of `Announce`.
36 |
37 | ```json
38 | {{#include ../../lemmy/crates/apub/assets/lemmy/objects/group.json}}
39 | ```
40 |
41 | | Field Name | Description |
42 | | ------------------- | ----------------------------------------------------------------------------------------------------- |
43 | | `preferredUsername` | Name of the actor |
44 | | `name` | Title of the community |
45 | | `sensitive` | True indicates that all posts in the community are nsfw |
46 | | `attributedTo` | First the community creator, then all the remaining moderators |
47 | | `summary` | Text for the community sidebar, usually containing a description and rules |
48 | | `icon` | Icon, shown next to the community name |
49 | | `image` | Banner image, shown on top of the community page |
50 | | `inbox` | ActivityPub inbox URL |
51 | | `outbox` | ActivityPub outbox URL, only contains up to 20 latest posts, no comments, votes or other activities |
52 | | `followers` | Follower collection URL, only contains the number of followers, no references to individual followers |
53 | | `endpoints` | Contains URL of shared inbox |
54 | | `published` | Datetime when the community was first created |
55 | | `updated` | Datetime when the community was last changed |
56 | | `publicKey` | The public key used to verify signatures from this actor |
57 |
58 | ### User
59 |
60 | A person, interacts primarily with the community where it sends and receives posts/comments. Can also create and moderate communities, and send private messages to other users. Can be followed from other platforms.
61 |
62 | ```json
63 | {{#include ../../lemmy/crates/apub/assets/lemmy/objects/person.json}}
64 | ```
65 |
66 | | Field Name | Description |
67 | | ------------------- | -------------------------------------------------------- |
68 | | `preferredUsername` | Name of the actor |
69 | | `name` | The user's displayname |
70 | | `summary` | User bio |
71 | | `icon` | The user's avatar, shown next to the username |
72 | | `image` | The user's banner, shown on top of the profile |
73 | | `inbox` | ActivityPub inbox URL |
74 | | `endpoints` | Contains URL of shared inbox |
75 | | `published` | Datetime when the user signed up |
76 | | `updated` | Datetime when the user profile was last changed |
77 | | `publicKey` | The public key used to verify signatures from this actor |
78 |
79 | ### Instance
80 |
81 | Represents a Lemmy instance, and is used to federate global data like the instance description or site bans. It can be fetched from the root path.
82 |
83 | ```json
84 | {{#include ../../lemmy/crates/apub/assets/lemmy/objects/instance.json}}
85 | ```
86 |
87 | | Field Name | Description |
88 | | ----------- | -------------------------------------------------------- |
89 | | `name` | Instance name |
90 | | `summary` | Short description |
91 | | `content` | Long description (sidebar) |
92 | | `icon` | Instance icon |
93 | | `image` | Instance banner |
94 | | `inbox` | ActivityPub inbox URL |
95 | | `endpoints` | Contains URL of shared inbox |
96 | | `published` | Datetime when the instance was created |
97 | | `updated` | Datetime when the instance metadata |
98 | | `publicKey` | The public key used to verify signatures from this actor |
99 |
100 | ## Objects
101 |
102 | ### Post
103 |
104 | A page with title, and optional URL and text content. The attachment URL often leads to an image, in which case a thumbnail is included. Each post belongs to exactly one community. Sent out as `Page`, but for receiving the types `Article`, `Note`, `Video` and `Event` are also accepted.
105 |
106 | ```json
107 | {{#include ../../lemmy/crates/apub/assets/lemmy/objects/page.json}}
108 | ```
109 |
110 | | Field Name | Description |
111 | | ----------------- | --------------------------------------------------------------------------------------------------- |
112 | | `attributedTo` | ID of the user which created this post |
113 | | `to` | ID of the community where it was posted to |
114 | | `name` | Title of the post (mandatory) |
115 | | `content` | Body of the post |
116 | | `attachment` | A single website or image link |
117 | | `image` | Thumbnail for `url`, only present if it is an image link |
118 | | `commentsEnabled` | False indicates that the post is locked, and no comments can be added |
119 | | `sensitive` | True marks the post as NSFW, blurs the thumbnail and hides it from users with NSFW setting disabled |
120 | | `stickied` | True means that it is shown on top of the community |
121 | | `published` | Datetime when the post was created |
122 | | `updated` | Datetime when the post was edited (not present if it was never edited) |
123 |
124 | ### Comment
125 |
126 | A reply to a post, or reply to another comment. Contains only text (including references to other users or communities). Lemmy displays comments in a tree structure.
127 |
128 | ```json
129 | {{#include ../../lemmy/crates/apub/assets/lemmy/objects/note.json}}
130 | ```
131 |
132 | | Field Name | Description |
133 | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
134 | | `attributedTo` | ID of the user who created the comment |
135 | | `to` | Community where the comment was made |
136 | | `content` | The comment text |
137 | | `inReplyTo` | ID of the parent object. In case of a top-level comment this is the post ID, in case of a nested comment it is the parent comment ID. |
138 | | `published` | Datetime when the comment was created |
139 | | `updated` | Datetime when the comment was edited (not present if it was never edited) |
140 |
141 | ### Private Message
142 |
143 | A direct message from one user to another. Can not include additional users. Threading is not implemented yet, so the `inReplyTo` field is missing.
144 |
145 | ```json
146 | {{#include ../../lemmy/crates/apub/assets/lemmy/objects/chat_message.json}}
147 | ```
148 |
149 | | Field Name | Description |
150 | | -------------- | ------------------------------------------------------------------------- |
151 | | `attributedTo` | ID of the user who created this private message |
152 | | `to` | ID of the recipient |
153 | | `content` | The text of the private message |
154 | | `published` | Datetime when the message was created |
155 | | `updated` | Datetime when the message was edited (not present if it was never edited) |
156 |
157 | ## Collections
158 |
159 | ### Community Outbox
160 |
161 | ```json
162 | {{#include ../../lemmy/crates/apub/assets/lemmy/collections/group_outbox.json}}
163 | ```
164 |
165 | The outbox only contains `Create/Post` activities for now.
166 |
167 | ### Community Followers
168 |
169 | ```json
170 | {{#include ../../lemmy/crates/apub/assets/lemmy/collections/group_followers.json}}
171 | ```
172 |
173 | The followers collection is only used to expose the number of followers. Actor IDs are not included, to protect user privacy.
174 |
175 | ### Community Moderators
176 |
177 | List of moderators who can perform actions like removing posts or banning users.
178 |
179 | ```json
180 | {{#include ../../lemmy/crates/apub/assets/lemmy/collections/group_moderators.json}}
181 | ```
182 |
183 | ### Community Featured Posts
184 |
185 | List of posts which are stickied in the community.
186 |
187 | ```json
188 | {{#include ../../lemmy/crates/apub/assets/lemmy/collections/group_featured_posts.json}}
189 | ```
190 |
191 | ### User Outbox
192 |
193 | Only contains `totalItems` count, but no actual `items` for privacy reasons.
194 |
195 | ```json
196 | {{#include ../../lemmy/crates/apub/assets/lemmy/collections/person_outbox.json}}
197 | ```
198 |
199 | ## Activities
200 |
201 | ### User to Community
202 |
203 | #### Follow
204 |
205 | Each Community page has a "Follow" button. Clicking this triggers a `Follow` activity to be sent from the user to the Community inbox. The Community will automatically respond with an `Accept/Follow` activity to the user inbox. It will also add the user to its list of followers, and deliver any activities about Posts/Comments in the Community to the user.
206 |
207 | ```json
208 | {{#include ../../lemmy/crates/apub/assets/lemmy/activities/following/follow.json}}
209 | ```
210 |
211 | #### Unfollow
212 |
213 | After following a Community, the "Follow" button is replaced by "Unfollow". Clicking this sends an `Undo/Follow` activity to the Community inbox. The Community removes the User from its followers list and doesn't send any activities to it anymore.
214 |
215 | ```json
216 | {{#include ../../lemmy/crates/apub/assets/lemmy/activities/following/undo_follow.json}}
217 | ```
218 |
219 | #### Create or Update Post
220 |
221 | When a user creates a new post, it is sent to the respective community as `Create/Page`. Editing a previously created post sends an almost identical activity, except the `type` being `Update`.
222 |
223 | ```json
224 | {{#include ../../lemmy/crates/apub/assets/lemmy/activities/create_or_update/create_page.json}}
225 | ```
226 |
227 | #### Create or Update Comment
228 |
229 | A reply to a post, or to another comment as `Create/Note`. Can contain mentions of other users. Editing a previously created post sends an almost identical activity, except the `type` being `Update`.
230 |
231 | The origin instance also scans the Comment for any User mentions, and sends the `Create/Note` to
232 | those Users as well.
233 |
234 | ```json
235 | {{#include ../../lemmy/crates/apub/assets/lemmy/activities/create_or_update/create_note.json}}
236 | ```
237 |
238 | #### Like Post or Comment
239 |
240 | An upvote for a post or comment.
241 |
242 | ```json
243 | {{#include ../../lemmy/crates/apub/assets/lemmy/activities/voting/like_note.json}}
244 | ```
245 |
246 | #### Dislike Post or Comment
247 |
248 | A downvote for a post or comment.
249 |
250 | ```json
251 | {{#include ../../lemmy/crates/apub/assets/lemmy/activities/voting/dislike_page.json}}
252 | ```
253 |
254 | #### Undo Like or Dislike Post or Comment
255 |
256 | Revert a vote that was previously done by the same user.
257 |
258 | ```json
259 | {{#include ../../lemmy/crates/apub/assets/lemmy/activities/voting/undo_like_note.json}}
260 | ```
261 |
262 | #### Delete Post or Comment
263 |
264 | Mods can remove Posts and Comments from their Communities. Admins can remove any Posts or Comments on the entire site. Communities can also be removed by admins. The item is then hidden from all users.
265 |
266 | Removals are sent to all followers of the Community, so that they also take effect there. The exception is if an admin removes an item from a Community which is hosted on a different instance. In this case, the removal only takes effect locally.
267 |
268 | ```json
269 | {{#include ../../lemmy/crates/apub/assets/lemmy/activities/deletion/delete_page.json}}
270 | ```
271 |
272 | #### Undo Delete
273 |
274 | Post or comment deletions can be reverted by the same user.
275 |
276 | ```json
277 | {{#include ../../lemmy/crates/apub/assets/lemmy/activities/deletion/undo_delete_page.json}}
278 | ```
279 |
280 | #### Report Post, comment or private message
281 |
282 | Reports content for rule violation, so that mods/admins can review it.
283 |
284 | ```json
285 | {{#include ../../lemmy/crates/apub/assets/lemmy/activities/community/report_page.json}}
286 | ```
287 |
288 | #### Delete User
289 |
290 | Sent when a user deletes his own account.
291 |
292 | ```json
293 | {{#include ../../lemmy/crates/apub/assets/lemmy/activities/deletion/delete_user.json}}
294 | ```
295 |
296 | ### Community to User
297 |
298 | #### Accept Follow
299 |
300 | Automatically sent by the community in response to a `Follow`. At the same time, the community adds this user to its followers list.
301 |
302 | ```json
303 | {{#include ../../lemmy/crates/apub/assets/lemmy/activities/following/accept.json}}
304 | ```
305 |
306 | #### Announce
307 |
308 | If the Community receives any Post or Comment related activity (Create, Update, Like, Dislike, Remove, Delete, Undo etc.), it will forward this to its followers. For this, an Announce is created with the Community as actor, and the received activity as object. This is sent to all followers, so they get updated in real time.
309 |
310 | ```json
311 | {{#include ../../lemmy/crates/apub/assets/lemmy/activities/community/announce_create_page.json}}
312 | ```
313 |
314 | ### Moderation
315 |
316 | These actions can only be done by instance admins or community moderators. They are sent to the community and announced by it. See [](../users/04-moderation.md) for a general overview how moderation works in Lemmy. Communities can only be created on the same instance where a user is registered. After that, mods from other instances can be added with `Add/User` activity.
317 |
318 | #### Remove Post or Comment
319 |
320 | Removes a post or comment. The difference to delete is that remove activities have a summary field, which contains the reason for removal, as provided by the mod/admin.
321 |
322 | ```json
323 | {{#include ../../lemmy/crates/apub/assets/lemmy/activities/deletion/remove_note.json}}
324 | ```
325 |
326 | #### Block User
327 |
328 | Blocks a user so he can't participate anymore. The scope is determined by the `target` field: either a community, or a whole instance. The `removeData` field can optionally be set to indicate that all previous posts of the user should be deleted.
329 |
330 | ```json
331 | {{#include ../../lemmy/crates/apub/assets/lemmy/activities/block/block_user.json}}
332 | ```
333 |
334 | #### Lock post
335 |
336 | Posts can be locked so that no new comments can be created.
337 |
338 | ```json
339 | {{#include ../../lemmy/crates/apub/assets/lemmy/activities/community/lock_page.json}}
340 | ```
341 |
342 | #### Undo mod actions
343 |
344 | All previously listed mod actions can be reverted by wrapping the original activity in `Undo`. Note that Lemmy regenerates the inner activity with a new ID.
345 |
346 | ```json
347 | {{#include ../../lemmy/crates/apub/assets/lemmy/activities/deletion/undo_remove_note.json}}
348 | ```
349 |
350 | ```json
351 | {{#include ../../lemmy/crates/apub/assets/lemmy/activities/block/undo_block_user.json}}
352 | ```
353 |
354 | #### Add or remove featured post
355 |
356 | Posts can be pinned so that they are always shown on top of the community. This is federated with the [Community featured posts](#community-featured-posts) collection.
357 |
358 | ```json
359 | {{#include ../../lemmy/crates/apub/assets/lemmy/activities/community/add_featured_post.json}}
360 | ```
361 |
362 | ```json
363 | {{#include ../../lemmy/crates/apub/assets/lemmy/activities/community/remove_featured_post.json}}
364 | ```
365 |
366 | #### Add or remove mod
367 |
368 | Add a new mod to the community. Has to be sent by an existing community mod, or an admin of the community's instance.
369 |
370 | ```json
371 | {{#include ../../lemmy/crates/apub/assets/lemmy/activities/community/add_mod.json}}
372 | ```
373 |
374 | An existing mod can be removed in the same way.
375 |
376 | ```json
377 | {{#include ../../lemmy/crates/apub/assets/lemmy/activities/community/remove_mod.json}}
378 | ```
379 |
380 | ### User to User
381 |
382 | #### Follow a user
383 |
384 | Users from other platforms can follow Lemmy users and receive all of their posts to the inbox. Note that users who are registered on Lemmy can only follow groups, not other users.
385 |
386 | ```json
387 | {{#include ../../lemmy/crates/apub/assets/pleroma/activities/follow.json}}
388 | ```
389 |
390 | #### Create or Update Private message
391 |
392 | User profiles have a "Send Message" button, which opens a dialog permitting to send a private message to this user. It is sent as a `Create/ChatMessage` to the user inbox. Private messages can only be directed at a single User. They can also be edited with `Update/ChatMessage`.
393 |
394 | ```json
395 | {{#include ../../lemmy/crates/apub/assets/lemmy/activities/create_or_update/create_private_message.json}}
396 | ```
397 |
398 | #### Delete Private Message
399 |
400 | Deletes a previous private message.
401 |
402 | ```json
403 | {{#include ../../lemmy/crates/apub/assets/lemmy/activities/deletion/delete_private_message.json}}
404 | ```
405 |
406 | #### Undo Delete Private Message
407 |
408 | Restores a previously deleted private message. The `object` is regenerated from scratch, as such the activity ID and other fields are different.
409 |
410 | ```json
411 | {{#include ../../lemmy/crates/apub/assets/lemmy/activities/deletion/undo_delete_private_message.json}}
412 | ```
413 |
--------------------------------------------------------------------------------