107 | Subject: quick question for you
108 | MIME-Version: 1.0
109 | Content-Type: multipart/alternative;
110 | boundary="--==_mimepart_5b784aee4aca8_643f3fbdb76d45c04373fd";
111 | charset=utf-8
112 |
113 | --==_mimepart_5b784aee4aca8_643f3fbdb76d45c04373fd
114 | Content-Type: text/plain; charset=utf-8
115 |
116 | helo eric,
117 | I was just wondering...
118 | Any chance you could review my diff before I die of old age ?
119 | --==_mimepart_5b784aee4aca8_643f3fbdb76d45c04373fd
120 | Content-Type: text/html; charset=utf-8
121 |
122 |
123 |
124 | helo eric,
125 |
126 | I was just wondering...
127 | Any chance you could review my diff before I die of old age ?
128 |
129 |
130 |
131 | --==_mimepart_5b784aee4aca8_643f3fbdb76d45c04373fd--
132 | ```
133 |
134 | Essentially, the top structure of the email defines it as a multipart message, indicating that it contains multiple sections with different content types. These sections are separated by boundary delimiters. Each section acts as a distinct email message, complete with its own headers and body content, specifying the type of content it contains.
135 |
136 | In our example, the email includes both HTML and plain text versions of the same message. The top-level structure's content type provides a clue to the email client about which version to display first. So, if you ever find yourself frustrated by an email displaying in HTML when you prefer plain text, it's likely because either the email wasn't multipart and lacked a plain text version, or the Content-Type header didn't provide the necessary guidance to your email client.
137 |
138 | Multipart messages can serve various purposes. While our example aimed to provide alternative formats for the message, other parts could include attachments or encode images referenced in the HTML version. How these parts are handled depends on the headers present and the capabilities of your email client.
139 |
--------------------------------------------------------------------------------
/src/chapters/1022_What_you_need_to_know_about_DNS.md:
--------------------------------------------------------------------------------
1 | # What you need to know about DNS
2 |
3 | Letitia!
4 | What a name.
5 | Halfway between a salad and a sneeze.
6 | -- Terry Pratchett
7 |
8 |
9 | ## The DNS protocol
10 | DNS, or the Domain Name Service, is an indispensable element of modern networks. Imagine a world where you had to recall IP addresses like 88.190.237.114 or 2a01:e0b:1000:23:be30:5bff:fed7:9af instead of domain names. DNS simplifies our digital lives by translating human-readable domain names into machine-readable IP addresses.
11 |
12 | However, DNS isn't just about name resolution. It plays a crucial role in the functioning of SMTP networks. Understanding DNS is essential for grasping SMTP mechanics.
13 |
14 | In fact, many SMTP failures stem from issues at the DNS level. A faulty DNS setup inevitably leads to disrupted SMTP exchanges.
15 |
16 | Let's delve into how DNS operates and its interaction with SMTP, focusing solely on the basics for now. We'll save advanced topics like DKIM, SPF, DMARC, and DANE for later discussions. Additionally, for simplicity's sake, we'll ignore IPv6 for the time being.
17 |
18 |
19 | ### DNS zones
20 | DNS servers organize domain information into zones. Each DNS server manages one or more zones, containing records pertinent to the associated domains. For example, the zone file for "opensmtpd.org" may include details like:
21 |
22 | DNS servers organize domain information into zones.
23 | Each DNS server manages one or more zones, containing records pertinent to the associated domains.
24 | For example,
25 | the _ns-master.poolp.org_ nameserver is in charge of the domain _opensmtpd.org_ and has a zone file which includes details like:
26 |
27 | ```
28 | NS ns-master.poolp.org.
29 | NS ns-backup.poolp.org.
30 | A 82.65.169.200
31 | www A 82.65.169.200
32 | opensmtpd.org. MX 0 mx-in.poolp.org.
33 | opensmtpd.org. MX 50 mx-backup.poolp.org.
34 | ```
35 |
36 | The opensmtpd.org zone is configured with two nameservers: ns-master.poolp.org and ns-backup.poolp.org. Additionally, it specifies that both opensmtpd.org and www.opensmtpd.org resolve to the IP address 82.65.169.200. Of particular interest to us postmasters, it includes two mail exchanger records for opensmtpd.org: mx-in.poolp.org with a preference of 0 and mx-backup.poolp.org with a preference of 50. These mail exchanger records are managed using MX records.
37 |
38 | It's worth noting that records can have either relative labels, such as www relative to the zone domain opensmtpd.org, or absolute labels, indicated by a trailing dot.
39 |
40 |
41 | ### MX records
42 | MX records play a crucial role in designating mail exchangers responsible for handling emails for a specific domain or subdomain. While a domain must have at least one MX record (to receive emails), it can list multiple for redundancy. For example:
43 |
44 | ```
45 | opensmtpd.org. MX 0 mx-in.poolp.org.
46 | opensmtpd.org. MX 10 mx-backup.poolp.org.
47 | subnet1.opensmtpd.org. MX 0 mx1.example.org.
48 | subnet2.opensmtpd.org. MX 0 mx1.example.com.
49 | ```
50 |
51 | Here, "opensmtpd.org" has two mail exchangers, while subdomains like "subnet1.opensmtpd.org" and "subnet2.opensmtpd.org" have their own designated mail exchangers.
52 |
53 |
54 | ### Resolver
55 | XXX
56 |
57 |
58 | ### MX lookup
59 | SMTP servers make use of DNS for many purposes,
60 | like looking up the hostname matching the IP address of a client for example,
61 | but the most important use upon which the entire SMTP protocol is built is _MX lookup_.
62 | If you read this far,
63 | you probably understood that this boils down to finding which MX are responsible for the destination domain of a message.
64 |
65 | Imagine I'm exchanging emails with Eric Faurot. I, with the email address gilles@poolp.org, want to send an email to Eric Faurot at eric@faurot.net. I input his email address into my Mail User Agent (MUA), such as mutt, and send the message.
66 |
67 | Behind the scenes, my MUA somehow forwards the message to a configured SMTP server. The SMTP server agrees to handle the email, queues it for relaying, and provides my MUA with a transaction identifier indicating the message's status as OK. Since this is a standard scenario, my MUA doesn't provide any additional feedback, following the Unix philosophy of staying quiet unless there's an issue.
68 |
69 | Now, the magic unfolds on the server's side: my SMTP server conducts an MX lookup. It instructs its configured nameserver to retrieve the MX records for the domain faurot.net. The nameserver either retrieves this information from its cache or locates the authoritative nameservers for faurot.net:
70 |
71 | ```
72 | $ dig +nocomments -t NS faurot.net |grep NS
73 | ; <<>> DiG 9.4.2-P2 <<>> +nocomments -t NS faurot.net
74 | ;faurot.net. IN NS
75 | faurot.net. 10757 IN NS c.dns.gandi.net.
76 | faurot.net. 10757 IN NS a.dns.gandi.net.
77 | faurot.net. 10757 IN NS b.dns.gandi.net.
78 | $
79 | ```
80 |
81 | Then requests either on of them to return the MX entries for the domain faurot.net:
82 |
83 | ```
84 | $ dig +nocomments -t MX faurot.net |grep MX
85 | ; <<>> DiG 9.4.2-P2 <<>> +nocomments -t MX faurot.net
86 | ;faurot.net. IN MX
87 | faurot.net. 10496 IN MX 50 fb.mail.gandi.net.
88 | faurot.net. 10496 IN MX 10 spool.mail.gandi.net.
89 | $
90 | ```
91 |
92 | Upon obtaining this information, my resolver returns the MX records for faurot.net to my SMTP server. For instance, it may provide fb.mail.gandi.net with a preference of 50 and spool.mail.gandi.net with a preference of 10. My SMTP server prioritizes these mail exchangers based on their preference numbers, placing those with lower values at the top:
93 |
94 | ```
95 | faurot.net. 10496 IN MX 10 spool.mail.gandi.net.
96 | faurot.net. 10496 IN MX 50 fb.mail.gandi.net.
97 | ```
98 |
99 | Next, my SMTP server conducts a DNS A lookup on the first mail exchanger to translate the hostname to an IP address:
100 |
101 | ```
102 | $ dig +nocomments -t A spool.mail.gandi.net | grep A
103 | ; <<>> dig 9.10.8-P1 <<>> +nocomments -t A spool.mail.gandi.net
104 | ;spool.mail.gandi.net. IN A
105 | spool.mail.gandi.net. 487 IN A 217.70.178.1
106 | $
107 | ```
108 |
109 | It then attempts to establish a connection to that IP to start an SMTP session and deliver the message in an SMTP transaction.
110 |
111 | Typically, this marks the conclusion of my SMTP server's involvement. It's fulfilled its responsibility by delivering the message to its intended destination.
112 |
113 |
114 | ### Backup MX
115 | Due to the quantic nature of our universe,
116 | an MX may be in two overlapping states:
117 | it may be willing to accept a message but it may also be unable to accept it at the same time.
118 |
119 | The MX may have a dying drive failing disk writes at random,
120 | it may be lacking disk space all of a sudden due to a surge of messages with large attachements,
121 | it may even work strangely due to a broken DNS setup[1](#1).
122 | A variety of reasons can prevent it from committing to deliver a message.
123 |
124 | That's if its even reachable at all.
125 |
126 | The MX could suffer a power outage,
127 | a network outage,
128 | someone tripping on the cable,
129 | or even an exhausted sysadmin accidentally typing `halt` in the wrong terminal[2](#2).
130 |
131 | This is where preferences come into play.
132 | Each MX server for a domain has a preference value associated to its MX record in the zone.
133 | The preference is a numerical value which determines in what order MX servers should be attempted by a remote MX when trying to transfer messages for the domain.
134 | The lower the preference value is, the highest priority the MX gets.
135 | This mechanism can be use to distribute load over pools of machines,
136 | to prioritize faster servers,
137 | or to create... backup MX.
138 |
139 | ```
140 | faurot.net. 10496 IN MX 10 spool.mail.gandi.net.
141 | faurot.net. 10496 IN MX 50 fb.mail.gandi.net.
142 | ```
143 |
144 | From my MX point of view,
145 | there is no difference between these two MX servers except for the requirement to attempt one before the other.
146 | It considers _spool.mail.gandi.net_ as closer to the destination than _fb.mail.gandi.net_,
147 | though from a purely technical perspective it could send to either one and be discharged of its responsability with regard to the mail as they both are declared as destination MX for the domain in the zone.
148 |
149 | Any of the MX for a domain may accept messages for that domain,
150 | regardless of their preference value,
151 | but since an MX commits to bring a message closer to its destination,
152 | it implies that an MX that's not the destination will only attempt to forward a message to another MX with a lower preference value.
153 | These MX servers are called _backup MX_ because they provide a backup service to the _primary MX_ servers in case they are unreachable,
154 | and commit to either attempt passing them the messages until they succeed or acknowledge the original sender that an error occurred.
155 |
156 | Note that the preference value is only here to create a priority-sorted list of MX servers to contact.
157 | There is no requirement that a priority 10 exists,
158 | there is no rule preventing a zone from using preferences ranging from 100 to 200 with 100 being its _primary MX_.
159 | Basically,
160 | nothing prevent postmasters to give the preference they want to the MX they want and the sender MX should not think in terms of primary or backup MX,
161 | but in terms of what's the MX with the lowest value that it has not tried yet.
162 |
163 | Let's get back to my mail exchange and pretend that _spool.mail.gandi.net_ is temporarily unavailable.
164 | My MX fails to connect and looks at the next entry in the list which is _fb.mail.gandi.net_.
165 | It resolves the name to an address,
166 | then attempts to connect and passe over the message.
167 |
168 | At this point there are three possible outcomes:
169 | - I'm being unlucky, the backup MX is also temporarily unable to handle the mail.
170 | Since the zone does not provide a third MX to contact,
171 | my MX is stuck with the mail and will have to reattempt a transfer later.
172 |
173 | - The MX has explicitely rejected the mail permanently.
174 | There is no need to try again later in hope that another MX would accept it,
175 | a backup MX is not allowed to reject a message that could be accepted by a primary MX,
176 | so all destination MX can be considered as authoritative in their rejections.
177 |
178 | - The MX has accepted the message.
179 | As far as my MX is concerned,
180 | another MX is now responsible for not letting the message vanish,
181 | work is over.
182 |
183 |
184 |
185 |
186 | [1](#1) By now you know this is the first thing to rule out.
187 |
188 | [2](#2) This is a work of fiction. Names, characters, businesses, places, events, locales, and incidents are either the products of the author's imagination or used in a fictitious manner. Any resemblance to actual persons, living or dead, or actual events is purely coincidental.
189 |
--------------------------------------------------------------------------------
/src/chapters/1030_The_roles_of_OpenSMTPD.md:
--------------------------------------------------------------------------------
1 | # The roles of OpenSMTPD
2 |
3 | Cat: Where are you going?
4 | Alice: Which way should I go?
5 | Cat: That depends on where you are going.
6 | Alice: I don’t know.
7 | Cat: Then it doesn’t matter which way you go.
8 | -- Lewis Carroll, Alice in Wonderland
9 |
10 |
11 | ## Network server and client
12 | First of all, OpenSMTPD is a network server and client.
13 | It operates as a server when accepting connections from remote clients and mail exchangers,
14 | and it operates as a client when connecting to remote mail exchangers.
15 | When a mail exchangers operates as a backup MX,
16 | for example,
17 | a mail will enter the system through the server part of the MTA,
18 | then be relayed to the primary MX through the client part of the MTA.
19 |
20 | In its "secure by default" configuration,
21 | OpenSMTPD does not accept to receive messages from the network,
22 | limiting its server capabilities,
23 | but accepts sending to the network,
24 | making use of its client capabilities.
25 | Either way,
26 | incoming or outgoing,
27 | it supports IPv4,
28 | IPv6,
29 | and encapsulating SMTP sessions in TLS either by using the SMTPS protocol,
30 | or by using the STARTTLS extension to the SMTP protocol.
31 |
32 |
33 | ## Local enqueuer
34 | In addition to these network capabilities,
35 | OpenSMTPD provides a local enqueuing mechanism.
36 | That mechanism relies on the use of the _sendmail_ interface,
37 | which all MTA adhere to,
38 | to inject messages in the daemon through a Unix socket.
39 | It is technically possible to setup OpenSMTPD as a local MTA only allowing mail to be exchanged between local users,
40 | and not making use of any networking capabilities.
41 |
42 | Unlike with the network server which requires the daemon to be running to accept connections,
43 | the local enqueuing capability allows local users to submit messages even if the daemon is not running.
44 | Messages submitted while the daemon is down are written to an "offline" queue which is processed upon restart.
45 | This ensures local applications and users don't have to deal with errors due to the MTA being down due to maintainance.
46 | As far as they know,
47 | the mail was accepted even though recipients may tell them that it is taking time to be delivered.
48 |
49 | For a local user,
50 | there are no major differences between using the local enqueuer or the network submission to localhost.
51 | Both methods rely on the same code,
52 | no performances benefits are expected from using one over the other,
53 | however if your MUA supports local enqueuing,
54 | and most do,
55 | this is the sane thing to do really.
56 |
57 |
58 | ## Decision engine
59 | Whatever the method used to inject a mail for processing,
60 | a major role for the MTA it is to determine if it knows what to do with the mail and if it accepts to do it.
61 | Many MTA use configuration options to take the decision,
62 | OpenSMTPD relies on a ruleset which is processed sequentially.
63 |
64 | No matter how this is handled,
65 | the idea remains the same:
66 | given an envelope coming from a particular sender and heading to a particular recipient,
67 | is it ok for this MTA to accept taking care of the next step for the envelope ?
68 |
69 | The decision engine is a critical part of the MTA as misconfigurations result in catastrophic situations,
70 | as you will lose mail if legitimate envelopes are rejected,
71 | and you will open yourself to abuse if illegitimate envelopes are accepted.
72 | Spammers actively scan mail exchangers in order to find misconfigured ones that will accept to send mail to others on their behalf.
73 |
74 | The decision can only result in two situations:
75 | an envelope is accepted because the MTA knows what to do with it or it is refused.
76 | When accepted,
77 | it means that the MTA is operating as the destination for that envelope and will deliver the message to a local user,
78 | or it is operating as a relay to another mail exchanger and will forward it there.
79 |
80 | Either way,
81 | if an envelope is accepted and the SMTP transaction isn't aborted,
82 | a copy of the envelope and message are written to the queue on disk and waiting for delivery to take place.
83 |
84 |
85 | ## Queueing
86 | The SMTP protocol mandates that mail exchangers do whatever they have to do to ensure that envelopes and messages they are responsible for won't vanish.
87 | This means that if someone tripped on the power cable of a mail exchanger right after it had accepted to handle a transaction,
88 | it must be able to process the transaction upon restart as if nothing had happened.
89 | It also implies that unless envelopes and messages are written to a persistent storage right before accepting the transaction,
90 | usually the disk,
91 | then mails may be lost by the mail exchanger.
92 |
93 | Not only the envelopes and messages must be written to the disk,
94 | but they must be written in a way that shutting the mail exchanger down at any time during the writing doesn't cause a corrupted state.
95 | In addition,
96 | disk accesses are not cheap so the trickier the technique to ensure this sane state,
97 | the bigger the hit on performances for each and every transaction.
98 | If you keep in mind that all of this is to ensure that your mail exchanger,
99 | with an uptime of hundreds of days,
100 | doesn't lose a mail in the unlikely case that it would crash right at the wrong time,
101 | then you can really grasp what it means to take responsability of a message.
102 |
103 | The queue within OpenSMTPD holds all mails pending delivery,
104 | it provides an interface to read and write to the persistent storage and nothing more than that.
105 | A scheduler is in charge of determining for which mails within the queue a delivery should be attempted at a given time.
106 | Mails exit the queue when postmaster explicitely requests deletion,
107 | when the mail has been sucessfully delivered, permanently rejected or simply when it expires.
108 |
109 | OpenSMTPD provides a very reliable queue which has persistence guarantees as a top priority while providing the best performances under that constraint.
110 | The details of how this is done are only relevant to curious developers and will be explained in a later chapter.
111 |
112 |
113 | ## Scheduling
114 | Storing messages in a passive queue would not be very interesting if OpenSMTPD did not actually try to do something with them.
115 | And this is where the scheduler comes into play.
116 |
117 | It is a simple component which will scan the queue once at startup and create a very light memory view of all known envelopes.
118 | That memory view is structured in a way that lets the scheduler efficiently find envelopes based on their transaction identifier,
119 | but also to know at a given time which ones have reached expiry and which ones should be scheduled next.
120 | During the lifetime of the daemon,
121 | whenever a new envelope is inserted or removed from the queue,
122 | the scheduler will update its state in order to always have an exact representation of what envelopes exist.
123 |
124 | Since the scheduler always knows of all the envelopes it has to manage,
125 | it is fully idle when there are no envelopes,
126 | and it can always sleep and wake up right in time for the next envelope to schedule.
127 | This makes it a very light process in terms of resources consumption.
128 |
129 | When envelopes are schedulable,
130 | it notifies the proper subsystem so an attempt is made to deliver to a local user or to relay to a remote mail exchanger.
131 | If an envelope reaches expiry without a successful delivery,
132 | it notifies the queue so the envelope is removed and updates its internal state to reflect that.
133 |
134 |
135 | ## Mail Delivery and Mail Transfer Agents
136 | A mail that is sitting in the queue was either accepted for local delivery or for relying,
137 | so when the scheduler makes it schedulable the mail must be passed to the proper subsystem.
138 |
139 | Local deliveries happen through mail delivery agents.
140 | These are programs which follow a convention of reading content from their standard input,
141 | delivering them wherever they should,
142 | and reporting success with their exit status.
143 | OpenSMTPD ships with three delivery methods by default:
144 | mbox which delivers to an mbox-format mailbox,
145 | maildir which delivers to a maildir-format mailbox
146 | and lmtp which connects to an LMTP server and forwards the mail.
147 |
148 | Relayed deliveries happen through the mail transfer agent which manages clients connecting to other mail exchangers.
149 | The agent will do MX lookups, establish connections, play an SMTP session and try to submit a transaction for a mail.
150 |
151 | Either way,
152 | the status of the delivery is notified to the queue and scheduler so they can properly update their state.
153 |
154 |
155 | ## Putting it all together
156 | With all of the above,
157 | we can provide a relatively precise description of what OpenSMTPD does.
158 |
159 | Its role is to accept a mail from a local enqueuer or through a network submission,
160 | ensure that it knows and wants to do something with it,
161 | write it to a safe storage to make sure it doesn't lose it,
162 | determine when and how are the next appropriate time and methods to attempt delivering,
163 | request the proper subsystem to make an attempt at delivering locally or remotely,
164 | and finally update its state based on the success or failure of this attempt.
165 |
166 | Of course each and every task has its details and could be explained for pages,
167 | but this description is enough to know we're on the same page as to what to expect for the time being.
168 |
169 |
--------------------------------------------------------------------------------
/src/chapters/1040_Documentation_resources_and_community.md:
--------------------------------------------------------------------------------
1 | # Documentation, resources and community
2 |
3 | Lack of documentation is becoming a problem for acceptance.
4 | -- Wietse Venema
5 |
6 |
7 | ## Documents any OpenSMTPD user should know about
8 | Now that you have the daemon running,
9 | maybe it's the right time to discuss the various resources that are available for you to get familiar with the project.
10 |
11 | To become a true guru, you should try to keep around the following documents as references.
12 | As printing hundreds of pages is the modern equivalent of kicking nature in the groin,
13 | it is preferable that you memorize them.
14 | The RFC is only 90-ish pages long.
15 |
16 |
17 | ### The Holy RFC
18 | OpenSMTPD implements RFC 5321, the _Simple[1](#1) Mail Transfer Protocol_.
19 | Any question regarding the expected behavior and interactions with other MTA are answered in that document.
20 | These are 95 pages of pure joy[2](#2).
21 |
22 | A major goal of the project is to comply with the RFC as much as possible in order to provide a sane SMTP implementation.
23 | Because the RFC is not the real-world,
24 | following it too strictly may result in interoperability issues with other implementations.
25 | OpenSMTPD takes a more pragmatic approach:
26 | the RFC is a Holy document and, as every other Holy document,
27 | it must be sometimes interpreted or ignored.
28 |
29 |
30 | ### The man pages
31 | As is the habit with OpenBSD projects,
32 | the man pages are just wonderful.
33 | They are up-to-date,
34 | they document about everything while providing descriptions and examples of use.
35 | Mistakes and undocumented features are considered as bugs and are fixed as soon as possible to keep the documentation as great as possible.
36 |
37 | When about to ask support to developers,
38 | always make sure that you have searched the man pages for your answer.
39 | Most of the time,
40 | developers simply reply with a copy of a paragraph answering the questions and coming directly from the man pages.
41 |
42 | In a parallel universe where this book doesn't exist,
43 | you should be able to setup your mail exchanger using only the man pages.
44 | The man page for the configuration file ends with complete examples that could be copy-pasted and used to get a daemon started.
45 | Luckily for you, in this universe this book exists and you should most definitely buy multiple copies to keep as reference.
46 |
47 |
48 | ## Sites and resources
49 | As an OpenSMTPD user,
50 | you will also want to know about the online spots where the cool kids hang out.
51 | These are useful if you want to find informations, help or news about the project,
52 | or if you just want to spend some quality time on the Internet.
53 |
54 |
55 | ### The official website
56 | Home, sweet home.
57 | This is where you go to fetch a release tarball,
58 | the address of our mailing list,
59 | details regarding the goals of the project and a short blurb about its history.
60 |
61 | [https://www.OpenSMTPD.org](https://www.OpenSMTPD.org)
62 |
63 |
64 | ### The Undeadly news site
65 | Undeadly.org is an OpenBSD-specific news site.
66 | It covers OpenSMTPD stories when developers attends hackathons and describe their work there,
67 | or when there are major changes going into OpenSMTPD.
68 | Even if you are not an OpenBSD user and only care about OpenSMTPD stories,
69 | it is a valuable source of information to understand the design and programming interface changes taking place in the project due to research done by OpenBSD hackers.
70 |
71 | [https://www.undeadly.org](https://www.undeadly.org)
72 |
73 |
74 | ### poolp.org
75 | This is where the project started before joining the OpenBSD universe and where I continue to experiment and test bleeding-edge code on a small community of guinea pigs.
76 |
77 | The website contains a blog where,
78 | among other unrelated things,
79 | I write technical posts about various aspects of the project ranging from code explanation to rationale behind major refactoring or works in progress.
80 |
81 | [https://poolp.org](https://poolp.org)
82 |
83 |
84 | ### The OpenSMTPD mailing list
85 | misc@opensmtpd.org is a general purpose low-volume list that can be used to ask questions about configuration,
86 | discuss about the project or related topics,
87 | submit diffs or share interesting links with a community of hundreds of users running OpenSMTPD on various systems.
88 |
89 | [https://www.opensmtpd.org/list.html](https://www.opensmtpd.org/list.html)
90 |
91 |
92 | ### Twitter
93 | The project is also active on twitter where I maintain the @OpenSMTPD account.
94 | Tweets that are related to the projects are posted with the #OpenSMTPD hashtag.
95 |
96 | [https://twitter.com/OpenSMTPD](https://twitter.com/OpenSMTPD)
97 |
98 |
99 | ### IRC
100 | Last but not least,
101 | developers and users hang around the #OpenSMTPD channel on the Freenode network.
102 | The channel is moderately active with around a hundred members at the time of this writing,
103 | holding technical discussions and assisting people with the problem they meet using OpenSMTPD.
104 |
105 |
106 | ## This book
107 | OH.
108 | I ALMOST FORGOT.
109 | This book is going to be your new reference.
110 | It will be updated and should become the new OpenSMTPD Holy book as far as I'm concerned.
111 |
112 |
113 |
114 | [1](#1) OOOOH, THE IRONY.
115 |
116 | [2](#2) No wait. OOOOH, THE SARCASM.
117 |
--------------------------------------------------------------------------------
/src/chapters/1060_Understanding_the_configuration.md:
--------------------------------------------------------------------------------
1 | # Understanding the configuration
2 |
3 | XXX
4 | -- General Lee
5 |
6 |
7 | ## Understanding the configuration philosophy
8 |
9 | Like other software coming from the OpenBSD universe,
10 | OpenSMTPD has a philosophy of keeping things simple and stupid:
11 | things should just work out of the box,
12 | with sane defaults and minimal need of tuning.
13 | If tuning is absolutely necessary,
14 | then it should be as simple as reading a man page and looking at the examples.
15 |
16 | A postmaster should be able to setup an MX in just a few minutes,
17 | without prior knowledge of the software,
18 | and be able to spot configuration errors by reading the configuration out loud.
19 | MX are a particularly bad choice of software to misconfigure:
20 | they operate both as server and client and mistakes can cause a wide variety of problems ranging from rejection of legitimate e-mails to spreading of spam to the world.
21 | You can't prevent all configuration mistakes but you can try to make them as obvious as possible.
22 |
23 | OpenSMTPD does not provide buttons to tweak every single aspect of the software, this would go against the philosophy.
24 | Whenever users request new ways to tweak the daemon's behaviour,
25 | developers debate to understand the use-case and figure if a better default behaviour could not be provided instead of adding a new knob.
26 | New configuration options are only added if there's consensus that the use-case is legitimate,
27 | that it can't be handled by default,
28 | and that adding a new option is the best option for simpler code.
29 | Sometimes new use-cases extend previous ones and the grammar evolves slightly by replacing a previous option.
30 |
31 | This way of integrating new features allows the configuration file to remain very simple,
32 | only providing the options that are truly necessary.
33 |
34 |
35 | ## The default _smtpd.conf_ file
36 | OpenSMTPD is configured through _smtpd.conf_ which contains the entire configuration.
37 | It may also include other files so that the configuration can be split into separate modules,
38 | but most configuration files are very small and this feature is rarely used in practice.
39 |
40 | The default _smtpd.conf_ provides a functional,
41 | yet very restrictive,
42 | configuration.
43 | It allows local users to send mail to other users,
44 | local and remote,
45 | but only accepts submissions from the local machine.
46 | It is suitable for desktops,
47 | laptops or servers that send mail notifications to other machines but do not accept mail from other mail exchangers.
48 | The following is what's used on OpenBSD systems after install,
49 | these six lines give you a fully working local server:
50 |
51 | ```
52 | listen on lo0
53 |
54 | table "aliases" file:/etc/mail/aliases
55 |
56 | action "local_users" mbox alias
57 | action "relay_users" relay
58 |
59 | match from any for local action "local_users"
60 | match from local for any action "relay_users"
61 | ```
62 |
63 | It can't get much simpler...
64 |
65 | Let's go through the same file with comments to describe what it does:
66 |
67 | ```
68 | # Allow network connections on interface lo0 (on Linux, eth0)
69 | # This automagically listens on IPv4 and IPv6.
70 | #
71 | listen on lo0
72 |
73 | # A table named "aliases" may be used to query the file /etc/mail/aliases
74 | # The name "aliases" was freely chosen, we could have named it "bananas".
75 | #
76 | table "aliases" file:/etc/mail/aliases
77 |
78 | # Define two actions, also freely named:
79 | # "local_users", delivers to a mailbox using the alias table "aliases"
80 | # "relay_users", relays to another MX
81 | #
82 | action "local_users" mbox alias
83 | action "relay_users" relay
84 |
85 | # Define two matching patterns:
86 | # first matches any source for local recipients and assigns action "local_users"
87 | # second matches local sources for any recipient and assigns action "relay_users"
88 | #
89 | match from any for local action "local_users"
90 | match from local for any action "relay_users"
91 | ```
92 |
93 | Most common use-cases are slight variations from this default configuration:
94 | listening on a different interface,
95 | delivering to a different mailbox format and/or
96 | accepting envelopes for specific domain names.
97 | The configuration difference between this default "local" mail exchanger and an internet-facing mail exchanger is relatively small.
98 | To illustrate,
99 | here's an example configuration that no longer operates as a "local" mail exchanger but as a real internet-facing mail exchanger for the _poolp.org_ domain:
100 |
101 | ```
102 | # listen on all interfaces
103 | # on OpenBSD, interface groups may be used (ie: listen on all)
104 | #
105 | listen on 0.0.0.0
106 |
107 | table "aliases" file:/etc/mail/aliases
108 |
109 | action "local_users" mbox alias
110 | action "relay_users" relay
111 |
112 | match from any for domain "poolp.org" action "local_users"
113 | match from any for local action "local_users"
114 | match from local for any action "relay_users"
115 | ```
116 |
117 | This new configuration,
118 | which only differs from the default configuration by two lines,
119 | is enough to run an internet-facing mail exchanger that accepts mail for my domain.
120 |
121 |
122 | ## _smtpd.conf_ is a ruleset
123 | In most MTA software,
124 | the configuration file consists of configuration options to set MTA parameters and enable or disable features.
125 | A hostname is set here,
126 | an option is disabled there,
127 | and the combination of these determine result in a global configuration which tells the MTA what it should do.
128 |
129 | OpenSMTPD takes a different approach:
130 | it considers the configuration file as a ruleset similar in many ways to a firewall configuration.
131 | The configuration has a few bits to setup the MTA parameters,
132 | such as which interface it should listen to and if it should do TLS,
133 | but most of the configuration defines a set of rules describing envelope templates that it wants to handle.
134 |
135 | This makes a lot of sense because a mail exchanger is really just a router:
136 | it receives messages and decides if it wants to route them closer to their destination.
137 | So when an envelope is submitted by a client to OpenSMTPD,
138 | it will compare it to each rule of the ruleset.
139 | If a match is found,
140 | the envelope is accepted and the action associated to the rule is used to handle the delivery,
141 | otherwise the envelope is rejected.
142 |
143 | What needs to be highlighted is that the ruleset is evaluated with a _first-match policy_.
144 | An envelope goes through the rules in sequential order until a rule matches it and causes the evaluation to stop.
145 | This means that rules must be written from the most specific to the most generic:
146 |
147 | ```
148 | # client 192.168.1.1 will match first rule,
149 | # others will match second rule
150 | #
151 | match from src 192.168.1.1 for any action "action1"
152 | match from any for any action "action2"
153 |
154 | # clients always match the first rule,
155 | # the second rule is never evaluatred
156 | #
157 | match from any for any action "action1"
158 | match from src 192.168.1.1 for any action "action2"
159 | ```
160 |
161 | The ruleset matches _envelopes_, not sessions.
162 | This is a key concept to grasp because it means that rules expect entire envelopes,
163 | with a sender e-mail address and a recipient e-mail address to take their decision.
164 | They are not used to filter sessions,
165 | they can't be used to reject a client coming from a blacklisted IP address right after it connects.
166 | This is the role of a different mechanism,
167 | _filtering_,
168 | which is meant to control how a session is allowed to progress and if it can even reach the ruleset evaluation.
169 | The _filtering_ mechanism will be described in an upcoming chapter.
170 |
171 | Furthermore,
172 | if you recall the explanations about how SMTP works,
173 | the decision to accept an envelope is taken individually for each recipient address _before_ the message is emitted by the client.
174 | For this reason,
175 | rules can only evaluate envelopes using protocol-level informations:
176 | they can describe that a valid envelope must come from a specific IP address,
177 | over a TLS-secured channel,
178 | from a specific sender domain to a specific recipient...
179 | but they can't use the actual message content as that would require a trip back and forth to the future.
180 | Taking a decision to reject a message is also the role of _filtering_.
181 |
182 | A very good analogy for that is an actual physical mail sorting center:
183 | the decision to move mail from a city to another or to return undeliverable mail to its sender is (in mooooooost cases) taken using the envelope,
184 | not by reading the letter inside the envelope.
185 | The ruleset is just a software simulation of a sorting center.
186 |
187 | Finally,
188 | matching a rule doesn't immediately trigger the action associated to it:
189 | the message is not available yet to actually do something with it.
190 | The ruleset only determines if an envelope will be accepted for processing,
191 | it doesn't perform the delivery.
192 | The ruleset associates the action to the envelope so that it can be evaluated once the message is accepted.
193 |
194 |
195 | ## A few words about tables
196 | For an envelope to be accepted,
197 | OpenSMTPD must determine that it can do something with it by looking at the ruleset.
198 | However,
199 | part of the decision relies on information that can't realistically be inlined in the ruleset itself.
200 | For instance looking at the following ruleset:
201 |
202 | ```
203 | listen on 0.0.0.0
204 |
205 | table "aliases" file:/etc/mail/aliases
206 |
207 | action "local_users" mbox alias
208 | action "relay_users" relay
209 |
210 | match from any for local action "local_users"
211 | match from local for any action "relay_users"
212 | ```
213 |
214 | We can't expect the action "local_users" to list all the aliases which on my system range around 80,
215 | this would result in an action listing all possible aliases as shown below:
216 |
217 | ```
218 | action "local_users" mbox alias { root = gilles, postmaster = gilles, ...}
219 | ```
220 |
221 | This would not only be highly impractical after the first few dozens of entries but it would also mean that the configuration file needs to be edited whenever there's a change made to the list.
222 | Then the same would be required for system accounts,
223 | since OpenSMTPD needs to know if _gilles_ exists on the system before accepting the mail.
224 | The exhaustive list of system users would need to be listed in the configuration and synchronized with the system user database.
225 | The ruleset is a static configuration but some parts of it require a bit of dynamism and the ability to perform runtime lookups to external resources.
226 |
227 | To solve this,
228 | the "tables" mechanism was introduced to allow querying information at runtime.
229 | In the "aliases" example above,
230 | a table is declared as a query interface to _/etc/mail/aliases_,
231 | and an action is told to use that table when it wants to lookup aliases.
232 | At runtime,
233 | when the ruleset engine wants to lookup if an alias exists,
234 | it asks the table to perform the lookup and return results,
235 | if any.
236 |
237 | The same applies to system users.
238 | Unless the configuration requests lookup of users through a table:
239 | ```
240 | listen on 0.0.0.0
241 |
242 | table "aliases" file:/etc/mail/aliases
243 | table "userbase" file:/etc/mail/userbase
244 |
245 | action "local_users" mbox alias userbase
246 | action "relay_users" relay
247 |
248 | match from any for local action "local_users"
249 | match from local for any action "relay_users"
250 | ```
251 |
252 | OpenSMTPD uses an internal table that operates as an interface to the system user database,
253 | querying it at runtime about usernames it wants to lookup so that they don't have to be listed in the configuration.
254 |
255 | The information doesn't have to be known in advance by the configuration file,
256 | the daemon doesn't have to be restarted to be made aware of it,
257 | the table will take care of dynamically resolving the lookup after the daemon is started.
258 |
259 | The next chapter will cover tables with more details.
260 |
--------------------------------------------------------------------------------
/src/chapters/1070_Understanding_tables.md:
--------------------------------------------------------------------------------
1 | # Understanding tables
2 |
3 | When I look back on my childhood,
4 | my fondest memories are those surrounding the dinner table.''
5 | -- Katie Lee
6 |
7 |
8 | ## What is a table ?
9 | Tables are the mechanism through which OpenSMTPD performs all[1](#1) of its lookups at runtime.
10 | A _lookup_ process isolates them,
11 | requesting the table API on behalf of pretty much every part of the daemon,
12 | and forwarding back the answers.
13 | The concept is so central to the software that it really needs to be addressed before we dive into actual configuration:
14 | __all configurations__ rely on tables wether they are used explicitely or not.
15 |
16 | A table is a _named_ resource that is capable of answering requests.
17 | How it does so is completely opaque to OpenSMTPD:
18 | the table may be looking into a file,
19 | a database or calling a web service,
20 | it doesn't really matter.
21 | What matters is that the table can lookup something somewhere and return a result out of it.
22 | The table names are referenced at various places in the configuration,
23 | these places determine what _kinds_ of lookups and results the table is expected to serve.
24 | Because different contexts can use the same data some tables may be used in several different places,
25 | while other tables may only be used in specific places.
26 |
27 | The lookup kinds will be discussed as they are used in future chapters so I won't detail them here.
28 | The general idea is that if a table is used in a specific place of the configuration,
29 | then it will receive a specific kind of lookup that expects a correctly crafted answer.
30 | If a table is used as parameter to `for src`,
31 | then it will be queried to lookup for IP addresses and the results are expected to parse as valid IP addresses.
32 | This type checking in the table API ensures that tables don't return bogus values,
33 | their results may be wrong due to misconfiguration but they are valid for a context.
34 | The `table(5)` man page describes the various types with examples of use,
35 | it is a good reference if you forget about the format of a valid entry.
36 |
37 | Data kind set aside,
38 | tables support two data structures: lists and key-value mappings.
39 | This makes sense because uses-cases for tables in OpenSMTPD pretty much all fall in either one of:
40 | iterating through a list of values,
41 | checking if a value is part of a list,
42 | retrieving a value associated to a key.
43 |
44 |
45 | ## Explicit and Implicit tables
46 | There are explicit tables,
47 | which a postmaster will explicitly declare in the configuration file,
48 | and which provide a configuration to detail how the table is populated.
49 | Whenever the `table` keyword is found in the configuration,
50 | it means an explicit table was declared.
51 | An example of such table is the "aliases" table which is part of the default configuration,
52 | and which declares an explicit table backed by a file holding a mapping of aliases to system accounts:
53 |
54 | ```
55 | $ grep ^table /etc/mail/smtpd.conf
56 | table "aliases" file:/etc/mail/aliases
57 | $ grep ^root /etc/mail/aliases
58 | root: gilles
59 | ```
60 |
61 | There are also implicit tables,
62 | which a postmaster will not necessarily know about,
63 | and that are created internally by OpenSMTPD to make some of its own lookups more convenients.
64 | Unlike explicit tables,
65 | they are not visible in the configuration and can't be referenced in rules.
66 | They also use a different naming convention,
67 | one that uses characters forbidden in explicit table names to ensure there is no risk of collisions.
68 | Examples of such table include the "getpwnam" table which is used to lookup local users,
69 | or the "localaddrs" table which is filled at startup with all the IP addresses of the local machine to ease lookup of local interfaces.
70 |
71 | Implicit tables will not be mentionned much more in this book as they are merely a commodity to OpenSMTPD developers,
72 | however knowing that they exist can help understand how some features are implemented.
73 |
74 | In practice,
75 | a postmaster will only ever see and deal with explicit tables.
76 |
77 |
78 | ## Static and Dynamic tables
79 | When defining tables in the configuration,
80 | a postmaster may rely on using static or dynamic tables,
81 | both having their advantages and disadvantages.
82 |
83 | A static table is one that is declared with all of its values in _smtpd.conf_.
84 | It is simple to define,
85 | what you see is what you get,
86 | there is no risk of the values changing at runtime and looking at the configuration is enough to understand what the table does.
87 | Once a static table is defined and tested to work,
88 | you know the results will be replicable again and again.
89 | However,
90 | updating the table is not possible at runtime,
91 | it requires editing the configuration file and restarting the daemon to catch up the change:
92 |
93 | ```
94 | table "foobar" { foo, bar, baz }
95 | table "barbaz" { foo = bar, baz = qux }
96 | ```
97 |
98 | A dynamic table is one that is declared with a name and an external resource attached to it.
99 | The _values_ provided by the table are not visible in _smtpd.conf_,
100 | they can be changed at runtime without the daemon having to know or reload.
101 | Because they are not part of the configuration but served by external resources,
102 | these tables can contain arbitrarily large sets of values backed by more or less complex mechanisms (think databases):
103 |
104 | ```
105 | table "foobar" file:/etc/mail/foobar
106 | table "barbaz" db:/etc/mail/barbaz.db
107 | ```
108 |
109 | It is not uncommon to mix both static and dynamic tables depending on the context,
110 | where the table values originates from,
111 | how many entries are part of the table and how frequently it will be updated.
112 | None is "better" than the other,
113 | it is just a matter of what seems to be the right choice given the context.
114 |
115 | For example,
116 | it makes sense using a dynamic table for aliases which may contain numerous records that change every now and then,
117 | while at the same time using a static table to list a couple IP addresses that won't change:
118 |
119 | ```
120 | table "aliases" file:/etc/mail/aliases
121 | table "myaddrs" { 192.168.1.2, 192.168.1.3 }
122 | ```
123 |
124 | Still,
125 | there would be nothing wrong using a file for the list of IP addresses,
126 | if maybe the list was slightly larger,
127 | or meant to be dynamically changed every now and then,
128 | or even if it was fed by an external tool:
129 |
130 | ```
131 | table "aliases" file:/etc/mail/aliases
132 | table "myaddrs" file:/etc/mail/myaddrs
133 | ```
134 |
135 | In some situations the choice is obvious,
136 | you really want the table content to be dynamic and not part of the configuration,
137 | and in other situations it is just a matter of convenience or taste.
138 | The only thing to keep in mind is that static tables are loaded in memory at startup,
139 | whereas dynamic tables are looked up at runtime.
140 | Once OpenSMTPD is running,
141 | a static table lookup will always succeed if the table content in the configuration file was valid,
142 | whereas a dynamic table lookup may fail if there's a problem with the external resource.
143 |
144 | Regardless of which table is used,
145 | they are always referenced the same in _smtpd.conf_.
146 | Wherever a table is acceptable,
147 | the table can be referenced by its name enclosed in less-than/greater-than signs such as `` or ``:
148 | ```
149 | action "foobar" maildir alias
150 |
151 | match from src for any action "foobar"
152 | ```
153 |
154 |
155 | ## Inlined tables
156 | In the previous section,
157 | all tables were declared with a name regardless of if they were static or dynamic.
158 | In practice there are times when giving a name to a table is cumbersome when it only has one or two static values used only once in a single place.
159 | For this case,
160 | static tables support inlining.
161 | This is essentially declaring the values directly where they are going to be used,
162 | rather than declaring a named table...
163 | just so the name can be referenced.
164 |
165 | To write a rule matching two source IP addresses,
166 | we can rely on a static named table:
167 |
168 | ```
169 | table "myaddrs" { "192.168.1.2", "192.168.1.3" }
170 | match from src for domain "example.org" [...]
171 | ```
172 |
173 | But if "myaddrs" is only used in this single place,
174 | we can avoid having to declare a named table and inline the table as such:
175 |
176 | ```
177 | match from src { "192.168.1.2", "192.168.1.3" } for domain "example.org" [...]
178 | ```
179 |
180 | The mechanism is the same,
181 | the two examples are identical in terms of how they work,
182 | but one allows reusing the table whereas the other doesn't.
183 | In practice,
184 | what happens is that a named table is created with a generated name and used where the values are inlined.
185 |
186 | Now take a deep breath as I reveal a secret only known to a few...
187 |
188 | Pretty much every configurable parameter in the configuration file is an inlined static table.
189 |
190 | When you see something like:
191 | ```
192 | match for domain "example.org" [...]
193 | ```
194 |
195 | The parameter to `for domain` is an inlined table consisting of a single entry and is equivalent to writing:
196 |
197 | ```
198 | match for domain { "example.org" } [...]
199 | ```
200 |
201 | or writing:
202 |
203 | ```
204 | table "my_domain" { "example.org" }
205 | match for domain [...]
206 | ```
207 |
208 | This may look like a small detail but because of this almost every configurable parameter can be replaced with a multi-entry inlined static table:
209 | ```
210 | match for domain { "example.org", "example.net" } [...]
211 | ```
212 |
213 | And since named tables may be used wherever a static table is used,
214 | the ruleset to match a single domain and the ruleset to match hundreds of domains only differ by a table:
215 |
216 | ```
217 | match for domain "example.org" [...]
218 | match for domain [...]
219 | ```
220 |
221 | This is what allows OpenSMTPD configuration files to be so concise.
222 | A ruleset rarely grows big because the same few lines of configurations can be used on a laptop to handle a single domain,
223 | or on a mail exchanger to handle thousands of domains for virtual hosting,
224 | simply by having different table contents.
225 |
226 |
227 | ## Dynamic table backends
228 | As mentionned earlier,
229 | dynamic tables are opaque from OpenSMTPD's perspective.
230 | It doesn't really care how they do the lookups as long as they can return a properly crafted answer for a kind of lookup.
231 | Dynamic tables have backends,
232 | pieces of code that implement a specific way of performing a lookup.
233 | There are two official backends,
234 | _file_ and _db_.
235 |
236 | The _file_ backend reads values from a file and loads them into a memory representation.
237 | The representation of the values is very efficient and,
238 | if the table values can fit into memory,
239 | using this backend is the most performant choice.
240 | It supports atomic updates so making changes to the file will not update the in-memory view.
241 | Instead a specific command allows a reload that will either be successful and replace the table content,
242 | or fail and leave the previous table content in place.
243 | Once a _file_ table has mapped its content in memory,
244 | it behaves like a static table:
245 | the content is part of OpenSMTPD and lookups can never fail if the content was valid.
246 |
247 | The _db_ backend reads values from a Berkeley DB style database.
248 | Instead of reading a file and loading it into a memory representation,
249 | the file is used as the source to compile a binary file database with a structure that's efficient for lookups.
250 | The _db_ backend is slightly less efficiently than the _file_ backend,
251 | but it is better suited for large sets of data as it allows entries to remain on disk rather than in memory.
252 | As it uses the _file_ backend configuration to build its databases,
253 | it is very easy to switch from one to another.
254 | I usually recommend that people start using the _file_ backend and switch to _db_ backend if the values grow large enough that memory footprint becomes an issue.
255 |
256 | Other backends exist as third-party addons.
257 | Among them,
258 | the popular _sqlite_ backend,
259 | but there's also a _mysql_ backend,
260 | a _postgresql_ backend and even an _ldap_ backend.
261 | All work very similarly to what was shown in this chapter,
262 | but their resource file is an addon-specific configuration file:
263 |
264 | ```
265 | table aliases sqlite:/etc/mail/sqlite.conf
266 | table aliases mysql:/etc/mail/mysql.conf
267 | ```
268 |
269 | They are managed as separate projects to ensure that OpenSMTPD doesn't inherit all of their dependencies.
270 | Anyone can write a custom table backend,
271 | it is fairly easy and doesn't require the cooperation of the daemon.
272 |
273 |
274 |
275 | [1](#1) At the exception of DNS lookups, for now.
276 |
--------------------------------------------------------------------------------
/src/chapters/2000_Getting_started.md:
--------------------------------------------------------------------------------
1 | # Getting started
2 |
3 | The secret to getting ahead is getting started.
4 | -- Mark Twain
5 |
6 |
7 | ## Preliminaries
8 | Unlike other software that can put you in a world of stress,
9 | OpenSMTPD tries to bring a little bit of peace and harmony in your life as a postmaster.
10 | It is straighforward and this whole adventure will be as pleasing as a journey at the park.
11 |
12 | Oh,
13 | and as far as preliminaries are concerned,
14 | there are none and your mail server will be working in a few minutes.
15 |
16 |
17 | ## OpenSMTPD on OpenBSD
18 | If you're an OpenBSD user then you're in luck:
19 | OpenSMTPD became available for testing in the base system with OpenBSD 4.6 in 2009,
20 | then replaced the venerable Sendmail as the default MTA with OpenBSD 5.8 in 2015.
21 |
22 | On a brand new OpenBSD install without any former setup,
23 | the daemon will start at boot time and only allow local connections,
24 | delivery to local mailboxes and relaying from local users to remote recipients.
25 |
26 | If you're running a machine that doesn't need to operate as a mail exchanger accepting mail from remote hosts,
27 | then you're all good and I hope you enjoyed this book because this is were our roads split.
28 |
29 | Otherwise,
30 | you can just skip to the next chapter.
31 |
32 |
33 | ## OpenSMTPD on other systems
34 | Throughout the years,
35 | OpenSMTPD was ported to other systems including FreeBSD,
36 | NetBSD,
37 | DragonFlyBSD,
38 | Linux,
39 | macOS and Solaris.
40 |
41 | The project provides a code base that can be built and that runs on these systems,
42 | however there's no official package that can be installed with their respective package managers.
43 | Package maintainers from these communities are encouraged to create them if they want to provide a clean integration to their system,
44 | the OpenSMTPD developers often help them to ease packaging.
45 |
46 | The best option is to first check if your system ships with OpenSMTPD in its base system.
47 | If that's the case and it is the latest stable version, then...
48 | you're all set and you can skip to the next chapter, chop chop.
49 |
50 | Otherwise, you can check if your system has a pre-built package available for you to use.
51 | This is the case on many systems were installing the latest stable OpenSMTPD is just a command away.
52 | If that's the case for you, then again you're all set and you can skip to the next chapter.
53 |
54 | Unfortunately,
55 | several Linux distributions as well as macOS and Solaris do not have OpenSMTPD packaged.
56 | The only remaining option is to build the project from sources and install manually,
57 | the old school way.
58 |
59 |
60 | ## Building from source
61 |
62 | ### On OpenBSD
63 | OpenBSD always ships with the latest stable OpenSMTPD in its base system,
64 | there's absolutely no point in building from source unless you're a developer willing to contribute,
65 | or if you're running an older OpenBSD and want to run the latest stable OpenSMTPD.
66 |
67 | The process to build from source is very easy.
68 | All you need is a fresh CVS checkout of the source tree somewhere on your system,
69 | usually `/usr/src`,
70 | then run the build from `usr.sbin/smtpd` (assuming you have proper permissions):
71 |
72 | ```sh
73 | $ cd /usr/src/usr.sbin/smtpd
74 | $ make
75 | $ doas make install
76 | ```
77 |
78 | This builds the code and installs the resulting executables.
79 |
80 |
81 | ### On other systems
82 |
83 | If you need to build OpenSMTPD on systems other than OpenBSD,
84 | the recommended method is to fetch the latest stable release from the official website.
85 | A tarball is always linked from the front page.
86 |
87 | [https://www.OpenSMTPD.org](https//www.OpenSMTPD.org)
88 |
89 |
90 |
91 | #### Building from the portable tarball
92 |
93 | The portable tarball consists of the same code as the OpenBSD repository.
94 | In addition it also contains a compatibility layer that allows the project to build on systems with behaviors differing from OpenBSD,
95 | or that are missing some of its programming interfaces.
96 |
97 | OpenSMTPD makes use of a minimal set of dependencies that are part of the OpenBSD system,
98 | the portable version expects the same dependencies to be installed on target systems:
99 |
100 | - bison, to build the configuration parser
101 | - libevent, to abstract the asynchronous event loop.
102 | - libssl, preferably through the LibreSSL project, to provide TLS support.
103 |
104 | Optionally,
105 | the following dependencies will allow the building of conditional features:
106 |
107 | - libz, to allow mail queue compression
108 | - libdb, or any Berkeley DB compatible interface, to provide database files.
109 |
110 |
111 | Once these dependencies are installed,
112 | OpenSMTPD can be built and installed as follow:
113 |
114 | ```
115 | $ tar -zxf opensmtpd-6.8.0p1.tar.gz
116 | $ cd opensmtpd-6.8.0p1
117 | $ ./configure --prefix=/usr/local
118 | $ make
119 | $ sudo make install
120 | ```
121 |
122 | Note that depending on your system the dependencies may be installed in various places.
123 | OpenSMTPD's configure script attempts some autodetection but this doesn't always succeed and you may need to pass additional options to specify paths:
124 |
125 | ```
126 | $ ./configure --prefix=/usr/local --with-libevent=/opt --with-libssl=/opt
127 | ```
128 |
129 | The portable tarball comes with a README.md file which documents expected configure options for some known systems.
130 |
131 |
132 | ## Creating the daemon users
133 | To be able to run,
134 | OpenSMTPD requires that you provide it with two dedicated system accounts that are used to improve the security of the daemon.
135 | The first account allows the daemon to revoke root privileges in child processes so that no process manipulates user input with privileges.
136 | The second one is used to run the queue process as a distinct unprivileged user,
137 | preventing other processes from being able to alter or delete its content should they be compromised.
138 |
139 | On OpenBSD,
140 | the _\_smtpd_ and _\_smtpq_ unprivileged accounts are part of the base system and no additional work is required.
141 | On systems that provide a package,
142 | these users were most likely created automatically at install possibly with a different naming convention.
143 | When installing by hand... they must be created by hand.
144 |
145 | While no naming convention is imposed by OpenSMTPD and these accounts can be named freely,
146 | the code expects these particular names unless overridden with configure flags at build time.
147 | The accounts do not need a real home directory or shell,
148 | their only purpose is to allow dropping privileges.
149 |
150 | The commands below are what would be required to create them on an OpenBSD system,
151 | you should refer to your operating system's documentation to adapt them accordingly:
152 |
153 | ```
154 | # groupadd _smtpd
155 | # groupadd _smtpq
156 | # useradd -s /sbin/nologin -d /var/empty -g=_smtpd _smtpd
157 | # useradd -s /sbin/nologin -d /var/empty -g=_smtpq _smtpq
158 | ```
159 |
160 |
161 | ## About the mailwrapper
162 | Once upon a time, the only MTA software easily available was Sendmail.
163 | As a result, most MUA back then were written with hardcoded pathname to the `sendmail` executable,
164 | which was the one true way of enqueuing mail locally.
165 |
166 | As new MTA solutions became available,
167 | it was easier for them to adopt a sendmail-compatible interface rather than to expect MUA developers to make changes and support multiple MTA interfaces.
168 | The sendmail interface became a de-factor standart and MUA would simply call the `sendmail` executable without worrying about what MTA was hiding behind it.
169 |
170 | The fact that the `sendmail` executable had now become a de-facto interface and no longer a Sendmail specific executable led to ambiguities and issues when multiple MTA were installed on the same machine.
171 | This is where the mailwrapper came into play.
172 |
173 | My MUA expects /usr/sbin/sendmail to be used for sending mail.
174 | But if we take a closer look, it is no longer an executable installed by my MTA but really a symbolic link to /usr/sbin/mailwrapper:
175 |
176 | ```
177 | $ ls -l /usr/sbin/sendmail
178 | lrwxr-xr-x 1 root wheel 21 Mar 5 19:43 /usr/sbin/sendmail -> /usr/sbin/mailwrapper
179 | $
180 | ```
181 |
182 | When my MUA executes `/usr/sbin/sendmail`,
183 | what it really does is that it executes `/usr/sbin/mailwrapper` which then opens its configuration file and finds the following:
184 |
185 | ```
186 | $ grep ^sendmail /etc/mailer.conf
187 | sendmail /usr/sbin/smtpctl
188 | $
189 | ```
190 |
191 | When executed,
192 | the mailwrapper looks at the name it was invoked with and searches inside the _/etc/mailer.conf_ what command is associated to that name.
193 | This allows multiple MTA to be installed on the same system in different places,
194 | not colliding one with another,
195 | and giving the postmaster a way to configure which MTA is the active one answering to the sendmail-interface.
196 |
197 | To ensure that OpenSMTPD responds correctly to the Sendmail interface used by most MUA,
198 | the _mailer.conf_ configuration must be adapted to support the following commands:
199 |
200 | ```
201 | $ cat /etc/mailer.conf
202 | sendmail /usr/sbin/smtpctl
203 | send-mail /usr/sbin/smtpctl
204 | mailq /usr/sbin/smtpctl
205 | makemap /usr/sbin/smtpctl
206 | newaliases /usr/sbin/smtpctl
207 | $
208 | ```
209 |
210 | Note that the path to smtpctl may be different depending on the prefix you used when you configured your build.
211 | Also,
212 | not all systems rely on the mailwrapper and some may use different mechanisms or none at all,
213 | you may need to adapt this according to your operating system's documentation as I'm not familiar with all mechanisms out there.
214 |
215 |
216 | ## We're all set, let's give it a run !
217 | With the daemon installed and user accounts created,
218 | you're ready to start the daemon.
219 |
220 | The manual way of starting it is by running the `smtpd` command as _root_.
221 | Since you're obviously not logged in as _root_ because it is bad practice,
222 | this is usually done with the `sudo` command (or `doas` on OpenBSD).
223 |
224 | Throughout this book,
225 | whenever I use the `doas` command in an example,
226 | you can substitute it with `sudo` if that's what your system uses:
227 |
228 | ```
229 | $ doas /usr/local/sbin/smtpd
230 | $
231 | ```
232 |
233 | Depending on your system,
234 | startup scripts may be provided and you should refer to your operating system's documentation.
235 | On OpenBSD,
236 | the `rcctl` utility may be used to start OpenSMTPD as follows:
237 |
238 | ```
239 | $ sudo rcctl start smtpd
240 | smtpd(ok)
241 | $
242 | ```
243 |
244 | If no error message is printed to the console,
245 | then OpenSMTPD is probably running in the background and ready to serve local requests:
246 | ```
247 | $ ps ax |grep smtpd
248 | 85854 ?? Ssp 0:00.02 smtpd
249 | 52840 ?? Sp 0:00.10 smtpd: klondike (smtpd)
250 | 38508 ?? Sp 0:00.05 smtpd: control (smtpd)
251 | 67581 ?? Sp 0:00.17 smtpd: pony express (smtpd)
252 | 23615 ?? Sp 0:00.04 smtpd: lookup (smtpd)
253 | 374 ?? Sp 0:00.07 smtpd: queue (smtpd)
254 | 36517 ?? Sp 0:00.03 smtpd: scheduler (smtpd)
255 | $ tail -1 /var/log/maillog
256 | Aug 18 20:26:57 ams-1 smtpd[2731]: info: OpenSMTPD 6.7.0 starting
257 | $ nc localhost 25
258 | 220 localhost ESMTP OpenSMTPD
259 | ^C
260 | $
261 | ```
262 |
263 | If it isn't started,
264 | the reason should be displayed on the console or available in your system's log file.
265 | The most common causes of startup failure are a misplaced configuration file,
266 | missing unprivileged accounts or even another MTA already registered and/or running on the system.
267 |
268 | Once your done...
269 | Congratulations, you can move to the next chapter !
270 |
--------------------------------------------------------------------------------
/src/chapters/2001_Setting_up_your_own_first_MX.md:
--------------------------------------------------------------------------------
1 | # Setting up your own first MX
2 |
3 | ## A few words before we dive in
4 | This chapter assumes that you have a relatively good understanding of how SMTP and DNS work,
5 | as well as basic understanding of how the configuration file works.
6 | If you jumped right here,
7 | impatient grasshopper,
8 | I highly recommend that you take the time to read the first part of the book,
9 | you will not go very far in this journey with a "test \& try" strategy.
10 | I'll wait right here, don't worry.
11 |
12 | Good.
13 |
14 | I knew you'd be back.
15 |
16 | This chapter will go through the process of setting up an MX interoperating with other MX in the SMTP network.
17 | It will start with a very basic MX and progressively move to more advanced setups.
18 | All use-cases cannot realistically be covered,
19 | but my goal is to cover a wide set of use-cases ranging from a small personal server to a corporate high-volume mail server,
20 | and provide you with an understanding of how you can deal with your own specific use-cases by yourself.
21 |
22 | In this chapter,
23 | we will be setting up an MX for the domain "hypno.cat".
24 |
25 |
26 | ## Requirements for a working mail exchanger
27 | There are very few requirements to run a basic MX and,
28 | as long as they are met,
29 | you will be able to exchange mails from your domain to the rest of the world in just a few minutes.
30 |
31 | First of all,
32 | you need a machine that will operate as the MX.
33 | For the time being,
34 | let's not worry about CPU, RAM or disk performance concerns,
35 | any machine that you can keep powered-on will do.
36 | You'll want a machine that's reliable though:
37 | until we have discussed how to set up secondary MX,
38 | any downtime of that machine might cause loss of e-mails.
39 |
40 | With that machine,
41 | you need an Internet access that does not filter incoming or outgoing SMTP trafic:
42 | in other words, you should be able to emit or receive connections to TCP port 25 (noted 25/tcp from now on).
43 | To prevent home computers infected with malware from spamming the world,
44 | many ISP have disallowed emitting or receiving SMTP trafic from a regular home connection.
45 | More recently,
46 | to avoid poorly-maintained servers from being compromised and used as spamming proxies,
47 | many hosting companies have started applying the same restrictions to the servers they rent.
48 | YES, there are hacks and workarounds but you want a reliable setup and this does not begin with a hack.
49 | Note that some providers filter 25/tcp by default but can remove the filter on request or through a control panel or by contacting their support.
50 |
51 | Finally,
52 | you need to have control over the DNS zone for the domain you intend to emit and receive e-mail for.
53 | We will be making changes to the zone so the other MX can find yours but also for some of the more advanced use-cases.
54 | SMTP is VERY tightly coupled with DNS, you will not be able to do much unless you have control over the DNS zone.
55 |
56 | That's enough _technically_ to setup a working MX.
57 |
58 | In practice,
59 | if you are serious about running your own MX,
60 | you will have few additional requirements which will be covered in the next chapter.
61 |
62 |
63 | ## Setting up the DNS
64 | This book not being about DNS,
65 | I'll keep this section as short as possible but I really, really, REALLY recommend that you to get familiar with the protocol and your DNS server software.
66 |
67 | Any DNS server software will do the job just fine,
68 | I will simply use NSD in my examples since that's what's available on OpenBSD base system.
69 | Some registrars will allow you to manage your domains through their own interface,
70 | as a result you don't even need to run your own DNS server,
71 | all you need is to be able to configure records for your domain.
72 |
73 |
74 | ### The hypno.cat zone
75 |
76 | We will start with a very simple zone that's not SMTP-capable and build from there:
77 |
78 | ```
79 | $ORIGIN hypno.cat.
80 | $TTL 6h
81 | @ IN SOA hypno.cat. root.hypno.cat. (
82 | 20190705084300 ; serial
83 | 1h ; refresh
84 | 30m ; retry
85 | 7d ; expiration
86 | 1h ) ; minimum
87 | NS ns-master.poolp.org.
88 | NS ns-backup.poolp.org.
89 | @ A 82.65.169.200
90 | www A 82.65.169.200
91 | ```
92 |
93 | $ORIGIN allows us to set the domain to which host names are relative.
94 | Note the trailing dot which differentiates absolute names from relative names.
95 | The $ORIGIN can be expressed as @ in records so the following records are equivalent:
96 |
97 | ```
98 | @ A 82.65.169.200
99 | hypno.cat. A 82.65.169.200
100 | ```
101 |
102 | By setting it to "hypno.cat.",
103 | we can declare a record "www" without having to specify the full name "www.hypno.cat.".
104 | The following records are also equivalent:
105 |
106 | ```
107 | www A 82.65.169.200
108 | www.hypno.cat. A 82.65.169.200
109 | ```
110 |
111 |
112 | $TTL allows providing a default time-to-live value for records that don't have one.
113 | The value you set will have a huge impact on how often other DNS servers query yours,
114 | you will want to investigate what value is right for you.
115 |
116 | The SOA record,
117 | short for Start Of Authority,
118 | contains administrative informations regarding the zone.
119 | The details are not really in the scope of this section so I'll just point my fingers at the values in parenthesis:
120 | the serial is simply a value incremented when you make changes to the zone so secondary servers pick up the change,
121 | the refresh, retry, expiration and minimum values are TTL values that inform other DNS servers about how often they should check for changes in your zone.
122 | When resolving a name in your zone,
123 | most DNS resolvers will cache the result to avoid having to look it up again through your DNS if another query came up right away.
124 | The TTL values in your SOA record inform these resolvers how often they should expire their cache:
125 | low values mean your zone changes get propagated faster at the cost of more pressure put on your DNS server,
126 | higher values mean your DNS server is under less pressure but zone changes take longer to propagate.
127 | A good approach is to use low TTL values during the setup of your infrastructure,
128 | then increase the values when your zone is stable and rarely changes.
129 |
130 | The NS records,
131 | short for Name Server,
132 | point to the nameservers holding the zone for the domain.
133 | Here, "ns-master.poolp.org." and "ns-backup.poolp.org." are authoritative for the "hypno.cat." domain,
134 | these are the ones who hold the truth regarding that zone.
135 |
136 | Finally, the A records map an IPv4 address to a name.
137 | So the following record states that both hypno.cat and www.hypno.cat will resolve to 82.65.169.200:
138 | ```
139 | @ A 82.65.169.200
140 | www A 82.65.169.200
141 | ````
142 |
143 | These are not the only records available in DNS,
144 | far from it,
145 | but we have to start somewhere so there we are.
146 | The following subsections will introduce the A,
147 | PTR and MX records which are the most basic ones that we need to be reachable.
148 | Other records used in more advanced setups will be discussed later.
149 |
150 |
151 | ### Configuring PTR, A and MX records
152 |
153 | Now that we know the basics about how to setup a DNS zone,
154 | we need to do the proper plumbing for our MX.
155 |
156 | #### Configuring the PTR record
157 | We'll start with the PTR record because it is a bit special:
158 | the zone I described above is used for "forward DNS" lookups,
159 | but "reverse DNS" is handled in a somewhat special zone.
160 |
161 | This could easily grow into a full chapter so let's keep it simple:
162 | unless you own the address range of your IP address, you don't have control over that zone.
163 | If you own the address range you should already know about arpa zones or rush to buy a DNS book.
164 |
165 | Your ISP or server hosting company leasing you the IP address are the ones in charge of that zone.
166 | In most cases they'll just provide you with a form so you can declare what hostname the IP address should resolve to,
167 | translating it to a proper PTR record in their zone.
168 | The IP address of my MX is 82.65.169.200 and I want it to be named "mx.hypno.cat.",
169 | so using the form provided by my server hosting company I declare the name which translates to the following record:
170 |
171 | ```
172 | 200.169.65.82.in-addr.arpa. PTR mx.hypno.cat.
173 | ```
174 |
175 | This can be verified using the "nslookup" command:
176 | ```
177 | $ nslookup 82.65.169.200
178 | Server: 62.210.16.6
179 | Address: 62.210.16.6#53
180 | Non-authoritative answer:
181 | 200.169.65.82.in-addr.arpa name = mx.hypno.cat.
182 | Authoritative answers can be found from:
183 | 129.83.212.in-addr.arpa nameserver = nsb.online.net.
184 | 129.83.212.in-addr.arpa nameserver = nsa.online.net.
185 | $
186 | ```
187 |
188 | #### Configuring the A record}
189 | We have declared the reverse DNS record for "mx.hypno.cat." so that looking up the IP address can gives us the name but this is not enough.
190 | We also need to declare in our zone the forward DNS record so that looking up the name can give us the IP address.
191 | This is as simple as adding an A record in the hypno.cat. zone and bumping (increasing the value for) the serial in the SOA record:
192 |
193 | ```
194 | $ORIGIN hypno.cat.
195 | $TTL 6h
196 | @ IN SOA hypno.cat. root.hypno.cat. (
197 | 20190705094400 ; serial
198 | 1h ; refresh
199 | 30m ; retry
200 | 7d ; expiration
201 | 1h ) ; minimum
202 | NS ns-master.poolp.org.
203 | NS ns-backup.poolp.org.
204 | @ A 82.65.169.200
205 | www A 82.65.169.200
206 | mx A 82.65.169.200
207 | ```
208 |
209 | With this new record,
210 | it is now possible to resolve "mx.hypno.cat." into 82.65.169.200,
211 | as we'll verify using "nslookup":
212 |
213 | ```
214 | $ nslookup mx.hypno.cat
215 | Server: 62.210.16.6
216 | Address: 62.210.16.6#53
217 | Non-authoritative answer:
218 | Name: mx.hypno.cat
219 | Address: 82.65.169.200
220 | $
221 | ```
222 |
223 | Having a name resolve to an IP address and the IP address resolve to the same name back is called FCrDNS,
224 | short for Forward-Confirmed Reverse DNS,
225 | and is an essential requirement to being able to deliver mail successfully to most MX:
226 | a mismatching FCrDNS will often result in e-mail being blocked, delayed or assumed to be from a sender with a poor reputation.
227 |
228 |
229 | #### Configuring the MX record
230 | At this point,
231 | the "mx.hypno.cat." name can be resolved successfully but other MX do not know that it is an MX as it is no different from the "www.hypno.cat." record.
232 | Here comes the MX record,
233 | which is how a zone declares which machines are in charge of handling mail exchanges.
234 |
235 | There's a lot to say about MX records and we'll have the zone evolve in this book as we tackle more advanced use-cases,
236 | but for now let's keep it simple and add just one MX record:
237 |
238 | ```
239 | $ORIGIN hypno.cat.
240 | $TTL 6h
241 | @ IN SOA hypno.cat. root.hypno.cat. (
242 | 20190705094400 ; serial
243 | 1h ; refresh
244 | 30m ; retry
245 | 7d ; expiration
246 | 1h ) ; minimum
247 | NS ns-master.poolp.org.
248 | NS ns-backup.poolp.org.
249 | @ A 82.65.169.200
250 | www A 82.65.169.200
251 | mx A 82.65.169.200
252 | ```
253 |
254 | The MX records states that mail for "hypno.cat." is to be handled by "mx.hypno.cat.".
255 | The number "0" is the selection preference,
256 | this is irrelevant for our setup right now as we only have one MX,
257 | we'll look into it again when we add secondary MX for redundancy.
258 |
259 | With the record,
260 | whenever someone sends a mail to someone with an e-mail address ending with @hypno.cat,
261 | the DNS resolution will result in the e-mail being transferred to "mx.hypno.cat.".
262 | We can check that this MX resolution is properly done using the "dig" utility,
263 | the output is a bit bloated but the interesting part is the ANSWER SECTION:
264 |
265 | ```
266 | $ dig -t MX hypno.cat
267 | ; <<>> DiG 9.4.2-P2 <<>> -t MX hypno.cat
268 | ;; global options: printcmd
269 | ;; Got answer:
270 | ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 39566
271 | ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 1
272 | ;; QUESTION SECTION:
273 | ;hypno.cat. IN MX
274 | ;; ANSWER SECTION:
275 | hypno.cat. 21600 IN MX 0 mx.hypno.cat.
276 | ;; AUTHORITY SECTION:
277 | hypno.cat. 21600 IN NS ns-master.poolp.org.
278 | hypno.cat. 21600 IN NS ns-backup.poolp.org.
279 | ;; ADDITIONAL SECTION:
280 | mx.hypno.cat. 21600 IN A 82.65.169.200
281 | ;; Query time: 26 msec
282 | ;; SERVER: 62.210.16.6#53(62.210.16.6)
283 | ;; WHEN: Fri Jul 5 09:59:50 2019
284 | ;; MSG SIZE rcvd: 109
285 | $
286 | ```
287 |
288 | With the zone configured as such,
289 | our machine is now reachable by other MX machines that want to transfer e-mails to our domain.
290 |
291 |
292 | ## Setting up the MX
293 | Having a properly configured DNS is of no use if no MX is listening for e-mail transfers,
294 | so this section will now focus on getting one ready for handling e-mails for our domain.
295 |
296 | ### The most basic MX
297 | The most basic MX is one that will simply allow receiving e-mail for our domain and putting it into local users mailboxes,
298 | as well as allowing them to send e-mail to our and other domains.
299 | Luckily for us,
300 | this is almost the default configuration in OpenSMTPD,
301 | the only differences being that the default configuration doesn't know about our domain and only accepts connections from the local machine.
302 |
303 | The default configuration is this one:
304 | ```
305 | # $OpenBSD: smtpd.conf,v 1.11 2018/06/04 21:10:58 jmc Exp $
306 |
307 | # This is the smtpd server system-wide configuration file.
308 | # See smtpd.conf(5) for more information.
309 | table aliases file:/etc/mail/aliases
310 |
311 | # To accept external mail, replace with: listen on all
312 | #
313 | listen on lo0
314 | action "local" mbox alias
315 | action "relay" relay
316 |
317 | # Uncomment the following to accept external mail for domain "example.org"
318 | #
319 | # match from any for domain "example.org" action "local"
320 | match for local action "local"
321 | match for any action "relay"
322 | ```
323 |
324 | now after reading the comments,
325 | let's remove them and adapt the configuration to accept e-mail for "hypno.cat" from the world without aliases for the time being:
326 |
327 | ```
328 | listen on all
329 |
330 | action "local" mbox
331 | action "relay" relay
332 |
333 | match from any for domain "hypno.cat" action "local"
334 | match for any action "relay"
335 | ```
336 |
337 | The main difference being the change on the "listen" line so the server listens to connections on all interfaces not just local ones,
338 | as well as the change of a "match" rule so the server matches envelopes coming from any source for the domain "hypno.cat".
339 | With only these changes,
340 | people can now send an e-mail to a user of the server by using a @hypno.cat e-mail address,
341 | OpenSMTPD will deliver the e-mail to an mbox.
342 | Local users may send mail to @hypno.cat which will also deliver locally,
343 | or send mail to any other domain which will be relayed outside.
344 |
345 | ### Making sure everything works
346 | Now that DNS and MX are properly set,
347 | we need to check that our MX can actually exchange e-mails with the world.
348 |
349 | Once started,
350 | the first thing you will want to check is if a machine outside your network can reach your MX through 25/tcp.
351 | Sending yourself a mail from another e-mail address is the quickest way to ensure things work right,
352 | as if you receive the e-mail it means your MX is reachable.
353 | You'll want to reply to that e-mail to make sure that the exchanges work both ways.
354 |
355 | Now what if this doesn't work ?
356 |
357 | You have many ways of troubleshooting the issue.
358 | First, make sure that you don't see connections in your log because it could be a mistake,
359 | you could very well have received the e-mail but be looking at the wrong place.
360 | If you see nothing in logs,
361 | you should try connecting to your MX on 25/tcp using telnet or netcat,
362 | an OpenSMTPD banner similar to this should be displayed:
363 |
364 | ```
365 | $ nc mx.hypno.cat 25
366 | 220 mx.hypno.cat ESMTP OpenSMTPD
367 | ^C
368 | $
369 | ```
370 |
371 | If you don't see a banner,
372 | it means that your MX is not reachable from the outside.
373 | Either you have a firewall preventing connections,
374 | your ISP/Host is blocking trafic,
375 | your configuration is incorrect or maybe you haven't restarted the daemon to use your updated configuration.
376 |
377 | If you see the banner but you can't reach yourself from another e-mail address,
378 | the issue is most certainly in DNS.
379 | Either the DNS configuration is incorrect,
380 | or maybe the DNS resolver on the sending end had something incorrect in cache,
381 | maybe even your zone is not being used at all.
382 | You will have to investigate in the DNS perimeter and this is where you'll need a DNS book because it's out of my scope ;-)
--------------------------------------------------------------------------------
/src/chapters/3000_logging_tracing_reporting_statistics.md:
--------------------------------------------------------------------------------
1 | # Logging, Tracing, Reporting and Statistics
2 |
3 | Being able to see an activity log of where a kid has been going on the Internet is a good thing.
4 | -- Bill Gates
5 |
6 | Statistics are no substitute for judgment.
7 | -- Henry Clay
8 |
9 |
10 | ## The internet is chaos
11 | The internet is pure chaos and,
12 | when you run an MX in pure chaos,
13 | you come to realize that _strange things_[1](#1) happen every now and then that you need to troubleshoot.
14 | In most situations, there is a very simple explanation:
15 | a user deleting by mistake an e-mail and assuming it was never received,
16 | another assuming an e-mail was lost on your server when it was not received in the first place,
17 | complaints that e-mails sent were not received by the recipients when they are either stuck in queue due to a temporary destination error...
18 | or already accepted by a destination that may be facing the same _strange things_.
19 |
20 | Sometimes,
21 | the issue does not have a simple explanation but needs a full investigation,
22 | one that can be particularly stressful if it happens while people are losing their e-mails[2](#2).
23 | There is nothing more irritating than having to battle with software to extract informations during an incident so OpenSMTPD comes with multiple means of understanding what is happening at different levels.
24 | Brief and verbose logging as well as
25 | runtime tracing, profiling, states and statistics can provide enough details to shed some light on even the most chaotic situations.
26 | Particular care was taken to make all information available in formats that can easily be pared and processed by standard utilities,
27 | allowing the creation of dashboards for monitoring and the use of command-line utilities for troubleshooting:
28 | pretty much any information is available two `grep(1)` away.
29 |
30 |
31 | ## Logging
32 | As is the case with other MTA software,
33 | OpenSMTPD logs all of its actions.
34 | It doesn't write directly to a file but relies on `syslog(3)` and the `syslogd(8)` daemon,
35 | allowing the system administrator to configure exactly how and where the logs are written,
36 | as well as how they're supposed to be rotated.
37 | The default is to log to a file,
38 | usually _/var/log/maillog_,
39 | but `syslogd(8)` may very well be configured to write the log entries to a third-party application or even send them to a centralized server through the network.
40 |
41 | Before OpenSMTPD 6.4,
42 | the logs were designed to be read by both humans and scripts.
43 | The idea was to write the logs with humans in mind,
44 | but in a format that would ease writing of log processing tools.
45 | Huge work was done to come up with the proper format,
46 | one that would be readable and parseable while carrying as much information as possible on a line so scripts could get a lot of value from a single line.
47 | This led to a nice and perfectly usable format,
48 | thought a bit dense for humans and a bit suboptimal for machines.
49 | Users liked it and the format remained in use for many years,
50 | extended every now and then to add new information to some of the log lines.
51 |
52 | With OpenSMTPD 6.4,
53 | some of the lines grew with so much information that they became overwhelming.
54 | It became hard to keep trying to accomodate both humans and scripts:
55 | too much information is unreadable to humans but not enough information is a waste for scripts.
56 | A different approach was taken:
57 | logs are for humans and humans only, tools need to use a different mechanism targeted at them.
58 | That doesn't mean that the log format becomes hard to parse but it means that lines will carry the information a human needs to read,
59 | not the information a script wants to gather.
60 | The reporting mechanism for scripts will be discussed in this chapter.
61 |
62 | OpenSMTPD supports two log levels: _brief_ and _debug_.
63 | They are logged with `syslog(3)` using the _LOG_MAIL_ facility and the _LOG_INFO_ and _LOG_DEBUG_ levels.
64 | The _debug_ log level is a super-set of the _brief_ log level,
65 | activating a lot[3](#3) of additional logs.
66 |
67 | ### Brief logging
68 | By default,
69 | OpenSMTPD will use _brief_ logging which is a log format that will record any event with just the right level of details to understand what is happening.
70 | The format is straightforward as shown in the following sample:
71 |
72 | ```
73 | Jul 6 06:01:40 ams-1 smtpd[75295]: 5df87c903d709659 smtp connected address=local host=poolp.org
74 | Jul 6 06:01:40 ams-1 smtpd[75295]: 5df87c903d709659 smtp message msgid=c405e39a size=303 nrcpt=1 proto=ESMTP
75 | Jul 6 06:01:40 ams-1 smtpd[75295]: 5df87c903d709659 smtp envelope evpid=c405e39ad6bbc64f from= to=
76 | Jul 6 06:01:40 ams-1 smtpd[75295]: 5df87c903d709659 smtp disconnected reason=quit
77 | Jul 6 06:01:40 ams-1 smtpd[75295]: 5df87c9250bba470 mda delivery evpid=c405e39ad6bbc64f from= to= rcpt= user=gilles delay=0s result=Ok stat=Delivered
78 | Jul 6 06:01:46 ams-1 smtpd[75295]: 5df87c972116a049 smtp connected address=local host=poolp.org
79 | Jul 6 06:01:46 ams-1 smtpd[75295]: 5df87c972116a049 smtp message msgid=3d4c4a42 size=319 nrcpt=1 proto=ESMTP
80 | Jul 6 06:01:46 ams-1 smtpd[75295]: 5df87c972116a049 smtp envelope evpid=3d4c4a424797fe79 from= to=
81 | Jul 6 06:01:46 ams-1 smtpd[75295]: 5df87c972116a049 smtp disconnected reason=quit
82 | Jul 6 06:01:46 ams-1 smtpd[75295]: 5df87c9b982c840c mta connecting address=smtp://127.0.0.1:10027 host=localhost
83 | Jul 6 06:01:46 ams-1 smtpd[75295]: 5df87c9b982c840c mta connected
84 | Jul 6 06:01:46 ams-1 smtpd[75295]: 5df87c9c42abeddb smtp connected address=127.0.0.1 host=localhost
85 | Jul 6 06:01:46 ams-1 smtpd[75295]: 5df87c9c42abeddb smtp message msgid=f6e94209 size=802 nrcpt=1 proto=ESMTP
86 | Jul 6 06:01:46 ams-1 smtpd[75295]: 5df87c9c42abeddb smtp envelope evpid=f6e94209918f7e7b from= to=
87 | Jul 6 06:01:46 ams-1 smtpd[75295]: 5df87c9b982c840c mta delivery evpid=3d4c4a424797fe79 from= to= rcpt=<-> source="127.0.0.1" relay="127.0.0.1 (localhost)" delay=0s result="Ok" stat="250 2.0.0: f6e94209 Message accepted for delivery"
88 | Jul 6 06:01:46 ams-1 smtpd[75295]: 5df87c9ffd0e785d mta connecting address=smtp://173.194.76.27:25 host=ws-in-f27.1e100.net
89 | Jul 6 06:01:46 ams-1 smtpd[75295]: 5df87c9ffd0e785d mta connected
90 | Jul 6 06:01:47 ams-1 smtpd[75295]: 5df87c9ffd0e785d mta tls ciphers=TLSv1.2:ECDHE-RSA-CHACHA20-POLY1305:256
91 | Jul 6 06:01:48 ams-1 smtpd[75295]: 5df87c9ffd0e785d mta delivery evpid=f6e94209918f7e7b from= to= rcpt=<-> source="212.83.129.132" relay="173.194.76.27 (ws-in-f27.1e100.net)" delay=2s result="Ok" stat="250 2.0.0 OK 1562385708 z11si6726563wmb.62 - gsmtp"
92 | ```
93 |
94 | Each line consists of a date, machine name, process name and identifier that are prepended by `syslogd(8)`:
95 | ```
96 | Jul 6 06:01:48 ams-1 smtpd[75295]
97 | ```
98 |
99 | Then the log line is appended and follows a simple pattern:
100 | ```
101 | 5df87c903d709659 smtp connected address=local host=poolp.org
102 | 5df87c903d709659 smtp message msgid=c405e39a size=303 nrcpt=1 proto=ESMTP
103 | 5df87c903d709659 smtp envelope evpid=c405e39ad6bbc64f from= to=
104 | ```
105 |
106 | Each log line is prefixed with a unique session identifier shared by all related log lines,
107 | followed by the subsystem that generated the session,
108 | then the event being logged and finally a serie of event-related informations displayed in key and value pairs.
109 | Lines carry only the information related to the event they are logging which means that most information is never duplicated in multiple lines.
110 |
111 | So, how do we work with that.
112 |
113 | If you are trying to understand what has happened during a session,
114 | you can search for all events that occurred in the session using the session identifier:
115 |
116 | ```
117 | $ grep 5df87c903d709659 /var/log/maillog
118 | Jul 6 06:01:40 ams-1 smtpd[75295]: 5df87c903d709659 smtp connected address=local host=poolp.org
119 | Jul 6 06:01:40 ams-1 smtpd[75295]: 5df87c903d709659 smtp message msgid=c405e39a size=303 nrcpt=1 proto=ESMTP
120 | Jul 6 06:01:40 ams-1 smtpd[75295]: 5df87c903d709659 smtp envelope evpid=c405e39ad6bbc64f from= to=
121 | Jul 6 06:01:40 ams-1 smtpd[75295]: 5df87c903d709659 smtp disconnected reason=quit
122 |
123 | $ grep 5df87c9ffd0e785d /var/log/maillog
124 | Jul 6 06:01:46 ams-1 smtpd[75295]: 5df87c9ffd0e785d mta connecting address=smtp://173.194.76.27:25 host=ws-in-f27.1e100.net
125 | Jul 6 06:01:46 ams-1 smtpd[75295]: 5df87c9ffd0e785d mta connected
126 | Jul 6 06:01:47 ams-1 smtpd[75295]: 5df87c9ffd0e785d mta tls ciphers=TLSv1.2:ECDHE-RSA-CHACHA20-POLY1305:256
127 | Jul 6 06:01:48 ams-1 smtpd[75295]: 5df87c9ffd0e785d mta delivery evpid=f6e94209918f7e7b from= to= rcpt=<-> source="212.83.129.132" relay="173.194.76.27 (ws-in-f27.1e100.net)" delay=2s result="Ok" stat="250 2.0.0 OK 1562385708 z11si6726563wmb.62 - gsmtp"
128 | Jul 6 06:01:58 ams-1 smtpd[75295]: 5df87c9ffd0e785d mta disconnected reason=quit messages=1
129 | ```
130 |
131 |
132 | If instead, you are trying to find a specific bit of information, you can search for specific events:
133 |
134 | ```
135 | $ grep 'mta tls' /var/log/maillog | tail -5
136 | Jul 5 22:54:33 ams-1 smtpd[77583]: a48a61d795bbf406 mta tls ciphers=TLSv1.2:ECDHE-RSA-CHACHA20-POLY1305:256
137 | Jul 5 23:26:27 ams-1 smtpd[77583]: a48a63397015c0fa mta tls ciphers=TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256
138 | Jul 6 04:06:48 ams-1 smtpd[77583]: a48a6eea7778dfbd mta tls ciphers=TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256
139 | Jul 6 04:39:44 ams-1 smtpd[77583]: a48a705113538487 mta tls ciphers=TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256
140 | Jul 6 06:01:47 ams-1 smtpd[75295]: 5df87c9ffd0e785d mta tls ciphers=TLSv1.2:ECDHE-RSA-CHACHA20-POLY1305:256
141 |
142 | $ grep 'smtp tls' /var/log/maillog | tail -5
143 | Jul 6 06:28:00 ams-1 smtpd[75295]: 5df87dbf30ab2c24 smtp tls ciphers=TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256
144 | Jul 6 06:28:06 ams-1 smtpd[75295]: 5df87dc1b2315795 smtp tls ciphers=TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256
145 | Jul 6 06:34:53 ams-1 smtpd[75295]: 5df87e01e48a743a smtp tls ciphers=TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256
146 | Jul 6 06:34:58 ams-1 smtpd[75295]: 5df87e04567c803c smtp tls ciphers=TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256
147 | Jul 6 06:54:35 ams-1 smtpd[75295]: 5df87ed36dee57ee smtp tls ciphers=TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256
148 | ```
149 |
150 |
151 | And of course,
152 | you can chain your `grep` calls to track a specific event for a specific session:
153 | ```
154 | $ grep 5df87c9ffd0e785d /var/log/maillog | grep 'mta connected'
155 | Jul 6 06:01:46 ams-1 smtpd[75295]: 5df87c9ffd0e785d mta connected
156 |
157 | $ grep 5df87c9ffd0e785d /var/log/maillog | grep 'mta tls'
158 | Jul 6 06:01:47 ams-1 smtpd[75295]: 5df87c9ffd0e785d mta tls ciphers=TLSv1.2:ECDHE-RSA-CHACHA20-POLY1305:256
159 | ```
160 |
161 |
162 | As you can see, the format is straightforward, compact and very easily searchable.
163 | It may be tempting to assume that you'll have no trouble finding the proper information in case of an incident,
164 | but I cannot emphasize enough how important it is that you familiarize yourself with reading and understanding the mail log _before_ you actually face an incident.
165 | You should send a few incoming and outgoing e-mails,
166 | look at what events are being logged,
167 | see if you can easily find them and correlate them with other events.
168 | In other words: prepare.
169 |
170 |
171 | ### Tracing subsystems
172 | Brief logging provides a big picture that is usually enough to understand the situation,
173 | but unfortunately sometimes it isn't and you need to be able to have a closer look at what is happening.
174 | For example, the brief log may show that a client is connecting then disconnecting without attempting any command:
175 |
176 | ```
177 | Jul 6 07:30:31 ams-1 smtpd[75295]: 5df8806c16c7267c smtp connected address=127.0.0.1 host=localhost
178 | Jul 6 07:30:31 ams-1 smtpd[75295]: 5df8806c16c7267c smtp disconnected reason=quit
179 | ```
180 |
181 | While the big picture lets you understand that a connection was accepted and that the client sent a "QUIT" command before submitting a message,
182 | it doesn't let you understand what happened in detail:
183 | did the client even see the banner ?
184 | did the client identifiy itself with HELO/EHLO ?
185 | did the client QUIT after looking at EHLO extensions offered on the server ?
186 |
187 | To deal with these cases,
188 | OpenSMTPD provides a tracing mechanism which can be temporarily enabled for a subystem so it can log very detailed informations.
189 | For the log lines above, enabling the SMTP tracing could have highlighted that the client sent a QUIT right after the banner:
190 |
191 | ```
192 | 5df8806c16c7267c smtp connected address=127.0.0.1 host=localhost
193 | smtp: 0x15d760462000: >>> 220 poolp.org ESMTP OpenSMTPD
194 | smtp: 0x15d760462000: <<< QUIT
195 | smtp: 0x15d760462000: >>> 221 2.0.0: Bye
196 | smtp: 0x15d760462000: STATE_CONNECTED -> STATE_QUIT
197 | 5df8806c16c7267c smtp disconnected reason=quit
198 | ```
199 |
200 | Or it could also have highlighted that the client sent a QUIT after requesting to see the EHLO extension:
201 |
202 | ```
203 | 3b358cf54b092021 smtp connected address=127.0.0.1 host=localhost
204 | smtp: 0x16efa1f6e000: >>> 220 poolp.org ESMTP OpenSMTPD
205 | smtp: 0x16efa1f6e000: <<< EHLO localhost
206 | smtp: 0x16efa1f6e000: STATE_CONNECTED -> STATE_HELO
207 | smtp: 0x16efa1f6e000: >>> 250-poolp.org Hello localhost [127.0.0.1], pleased to meet you
208 | smtp: 0x16efa1f6e000: >>> 250-8BITMIME
209 | smtp: 0x16efa1f6e000: >>> 250-ENHANCEDSTATUSCODES
210 | smtp: 0x16efa1f6e000: >>> 250-SIZE 36700160
211 | smtp: 0x16efa1f6e000: >>> 250-DSN
212 | smtp: 0x16efa1f6e000: >>> 250 HELP
213 | smtp: 0x16efa1f6e000: <<< QUIT
214 | smtp: 0x16efa1f6e000: >>> 221 2.0.0: Bye
215 | smtp: 0x16efa1f6e000: STATE_HELO -> STATE_QUIT
216 | 3b358cf54b092021 smtp disconnected reason=quit
217 | ```
218 |
219 | Either way,
220 | it provides a level of detail that is not available in brief logging and that is often not needed on a day-to-day basis.
221 | In some cases,
222 | tracing can be used to analyze the behavior of a session but most of the time it is used when facing a buggy SMTP implementation and trying to see what is really happening behind an e-mail exchange.
223 | There are multiple tracing subsystems in OpenSMTPD that can help understand about anything that is happening inside the software,
224 | from sessions to the scheduling of envelopes and their writing to a disk.
225 |
226 | Some of the tracing susbsystems are really low-level and meant to be read by developers,
227 | like the _imsg_, _mproc_ or _io_ traces which respectively trace all of the inter-process communication,
228 | the packets used in inter-process communication and pretty much all of data input/output occuring in the daemon.
229 | Others, like _smtp_, _mta_, _expand_, _lookup_ or _rules_ are used by both developers and users,
230 | tracing respectively the content of incoming and outgoing sessions, the expansion of aliases, the lookup of values in tables or the ruleset matching.
231 |
232 | The following are examples of _expand_, _lookup_ and _rules_ traces,
233 | expanding the address _gilles@poolp.org_ to a local user,
234 | looking up IP source address and recipient e-mail address in tables,
235 | and matching a specific rule:
236 |
237 | ```
238 | expand: 0x858351e0018: expand_insert() called for address:gilles@poolp.org[parent=0x0, rule=0x0]
239 | expand: 0x858351e0018: inserted node 0x85768d68000
240 | expand: lka_expand: address: gilles@poolp.org [depth=0]
241 | lookup: check "local" as NETADDR in table static: -> found
242 | lookup: check "poolp.org" as DOMAIN in table static: -> found
243 | lookup: check "gilles@poolp.org" as MAILADDR in table static:shithole -> 0
244 | lookup: check "local" as NETADDR in table static: -> found
245 | lookup: check "poolp.org" as DOMAIN in table static: -> found
246 | rule #2 matched: match from local for local => l01
247 | expand: 0x858351e0018: expand_insert() called for username:gilles[parent=0x85768d68000, rule=0x857661fbf80, dispatcher=0x8583c5c8700]
248 | expand: 0x858351e0018: inserted node 0x85811a68000
249 | expand: lka_expand: username: gilles [depth=1, sameuser=0]
250 | lookup: lookup "gilles" as ALIAS in table static:aliases -> 0
251 | lookup: lookup "gilles" as USERINFO in table getpwnam: -> "1000:1000:/home/gilles"
252 | expand: 0x858351e0018: expand_insert() called for filter:/usr/local/bin/fdm -m -a mda fetch[parent=0x85811a68000, rule=0x857661fbf80, dispatcher=0x8583c5c8700]
253 | expand: 0x858351e0018: inserted node 0x85811a69000
254 | expand: lka_expand: filter: /usr/local/bin/fdm -m -a mda fetch [depth=2]
255 | expand: 0x858351e0018: clearing expand tree
256 | ```
257 |
258 | As you can see,
259 | the tracing output format is much less user-friendly than brief logging.
260 | It is not meant to be used regularly but rather to cope with abnormal situations and gather all of the information useful for a developer to understand what is happening.
261 | Tracing is exposed to users because it can also help them understand issues,
262 | but it is primarily a tool aimed at developers that's not meant to have a stable or pretty output.
263 | Therefore the tracing subsystems are listed in `smtpctl(8)` but not much more documented as they can vary from a release ot another based on implementation details.
264 |
265 | An interesting feature about tracing is that it doesn't require OpenSMTPD to be built with particular options,
266 | it is a very light feature that can be enabled in production with no significant overhead.
267 | Anyone facing what seems to be a bug in production can easily enable and disable tracing to gather debugging data without having to shutdown the server or rebuild the software.\\
268 |
269 | Traces can be enabled either when OpenSMTPD is started with command line options,
270 | or through the `smtpctl` command:
271 |
272 | ```
273 | # smtpd -v -T smtp -T mta
274 | # smtpctl trace smtp
275 | # smtpctl trace mta
276 | ```
277 |
278 | Active traces can be removed at runtime using `smtpctl`:
279 |
280 | ```
281 | # smtpctl untrace smtp
282 | # smtpctl untrace mta
283 | ```
284 |
285 | At the time of this writing,
286 | tracing requires OpenSMTPD to be in verbose mode but this is likely to change in the future.
287 |
288 |
289 | ### Verbose logging
290 | Verbose logging,
291 | also known as debugging mode,
292 | is a scary beast:
293 | it is the log level used during development and it logs at a level so great that it can even scare developers sometimes.
294 |
295 | The verbose logging is what you resort to when everything else fails,
296 | it provides so much details that you pretty much understand the code paths that the software has taken.
297 | Just for the sake of comparison,
298 | this is what is logged by brief logging when OpenSMTPD starts:
299 |
300 | ```
301 | # smtpd -d
302 | info: OpenSMTPD 6.4.0 starting
303 | ```
304 |
305 | And this is what is logged by debug logging when the same OpenSMTPD starts:
306 |
307 | ```
308 | # smtpd -dv
309 | debug: init ssl-tree
310 | debug: init ca-tree
311 | debug: using "fs" queue backend
312 | debug: using "ramqueue" scheduler backend
313 | debug: using "ram" stat backend
314 | info: OpenSMTPD 6.4.0 starting
315 | [...] about three pages [...]
316 | smtpd: setup done
317 | debug: parent_send_config_ruleset: reloading
318 | debug: parent_send_config: configuring pony process
319 | debug: parent_send_config: configuring ca process
320 | debug: ca_engine_init: using RSA privsep engine
321 | debug: init private ssl-tree
322 | debug: smtp: listen on IPv6:::1 port 25 flags 0x400 pki "" ca ""
323 | debug: smtp: listen on IPv6:fe80::1%lo0 port 25 flags 0x400 pki "" ca ""
324 | debug: smtp: listen on 127.0.0.1 port 25 flags 0x400 pki "" ca ""
325 | debug: smtp: listen on IPv6:::1 port 587 flags 0x449 pki "mail.poolp.org" ca ""
326 | debug: smtp: listen on IPv6:fe80::1%lo0 port 587 flags 0x449 pki "mail.poolp.org" ca ""
327 | debug: smtp: listen on 127.0.0.1 port 587 flags 0x449 pki "mail.poolp.org" ca ""
328 | debug: smtp: listen on IPv6:::1 port 10028 flags 0x400 pki "" ca ""
329 | debug: smtp: listen on IPv6:fe80::1%lo0 port 10028 flags 0x400 pki "" ca ""
330 | debug: smtp: listen on 127.0.0.1 port 10028 flags 0x400 pki "" ca ""
331 | debug: pony: rsae_init
332 | debug: pony: rsae_init
333 | debug: pony: rsae_init
334 | debug: pony: rsae_init
335 | debug: pony: rsae_init
336 | debug: pony: rsae_init
337 | debug: smtp: will accept at most 473 clients
338 | ```
339 |
340 | If tracing is not meant to be user-friendly,
341 | well...
342 | verbose logging is petting tracing for being sooooo cute.
343 | Unless you are a developer or you are facing an issue that absolutely and necessarily requires verbose logging,
344 | you will probably avoid it at all costs: it is the last chance tool.
345 |
346 | At the moment,
347 | verbose mode is necessary for tracing but since it's also a developer tools,
348 | there's nothing incoherent here.
349 | It is likely that tracing will no longer require verbose logging in the future because having to endure that level of logging just to see an SMTP session is inhumane.
350 |
351 | Verbose logging can be enabled at start time through the command line options to `smtpd` or activated at runtime with the `smtpctl` command:
352 |
353 | ```
354 | # smtpd -v
355 | # smtpctl log verbose
356 | ```
357 |
358 | It can be stopped at runtime using `smtpctl`:
359 | ```
360 | # smtpctl log brief
361 | ```
362 |
363 |
364 | ## Statistics
365 | OpenSMTPD comes with a volatile statistics subsystem.
366 | The daemon will start with empty counters that the other subsystems will increase and decrease to provide a real-time overview of what is happening.
367 | These counters can be about high-level concepts such as the number of clients currently connected,
368 | just like they can be about low-level concepts such as the current memory usage of inter-process I/O buffers.
369 | They provide valuable information which allows both postmasters and developers to track daemon behavior at runtime.
370 |
371 | The statistics can be retrieved by using the `smtpctl` command:
372 |
373 | ```
374 | # smtpctl show stats
375 | buffer.lka.control=0
376 | buffer.lka.mda=0
377 | buffer.lka.parent=0
378 | [... truncated ...]
379 | buffer.smtp.mfa=0
380 | buffer.smtp.parent=0
381 | buffer.smtp.queue=0
382 | control.session=1
383 | mda.pending=0
384 | mda.running=0
385 | mta.connector=0
386 | mta.domain=0
387 | mta.envelope=0
388 | mta.host=0
389 | mta.relay=0
390 | mta.route=0
391 | mta.session=0
392 | mta.source=0
393 | mta.task=0
394 | mta.task.running=0
395 | queue.bounce=2
396 | scheduler.delivery.ok=821
397 | scheduler.delivery.permfail=2
398 | scheduler.delivery.tempfail=125
399 | scheduler.envelope=1
400 | scheduler.envelope.incoming=0
401 | scheduler.envelope.inflight=0
402 | scheduler.ramqueue.envelope=1
403 | scheduler.ramqueue.message=1
404 | scheduler.ramqueue.update=0
405 | smtp.kick=1
406 | smtp.session=0
407 | smtp.session.inet4=909
408 | smtp.session.local=5
409 | smtp.tls=0
410 | uptime=94691
411 | uptime.human=1d2h18m11s
412 | #
413 | ```
414 |
415 | Some people have used the counters to graph delivery versus bounces ratio to detect issues,
416 | while developers have graphed scheduler logic and memory usage to validate their theories.
417 | The statistics aggregation is very cheap and it is a good practice for developers to create counters and collect statistics when working on subsystems within the daemon.
418 |
419 | To make them easier to use and process,
420 | the output has been designed so that related counters share a common prefix that can be searched with a simple `grep` command.
421 | The following shows how a postmaster can search for all counters related to the MTA subsystem:
422 |
423 | ```
424 | # smtpctl show stats | grep ^mta\.
425 | mta.connector=0
426 | mta.domain=0
427 | mta.envelope=0
428 | mta.host=0
429 | mta.relay=0
430 | mta.route=0
431 | mta.session=0
432 | mta.source=0
433 | mta.task=0
434 | mta.task.running=0
435 | #
436 | ```
437 |
438 | The statistics subsystem is dynamic,
439 | it doesn't keep track of a set of pre-defined counters but creates them on-the-fly during their first use.
440 |
441 |
442 | ## Advanced tracing: profiling
443 | Sometime before the first release of OpenSMTPD,
444 | a lot of work was poured into improving performances to bring it to the level of other SMTP servers.
445 | There were two main areas of concern:
446 | the very intensive inter-process communication imposed by the multi-process design and the cost of disk accesses to the queue.
447 | Knowing the design of the software,
448 | intuitively there was no doubt that these were the main bottlenecks but we lacked proof and numbers.
449 |
450 | Two profiling traces were introduced:
451 | one to measure the time spent in inter-process communication,
452 | the other to measure the time spent in every single queue operation.
453 | Using these traces it became possible to see that some processes communicated more than they should have,
454 | exchanged too much data or were too slow processing it,
455 | as well as which queue operations were called during delivery and which ones required to be worked on.
456 |
457 | The precise measures allowed pinpointing operations that needed to be optimized,
458 | sometimes down to the level of system call sequences,
459 | but also which operations did not have constant execution time and needed to be rewritten to be predictable.
460 | It made it possible to measure the maximum performance of an API and compare the current performances to it,
461 | as well as determine which parts were performing ok,
462 | which parts needed to be tuned up and which parts were already optimal.
463 |
464 | With all the work that was done,
465 | profiling traces are today essentially used to confirm if a change improves current performances or causes regressions.
466 | However,
467 | developers writing custom queues, schedulers, tables or filters can make use of them to make sure their implementations are efficient.
468 | While they can be used to troubleshoot slow operations,
469 | they are really a tool for developers willing to improve performances in the software or writing their own custom bricks.
470 |
471 |
472 | ### profile-imsg traces
473 | OpenSMTPD is event-driven which means that everything it does results from an event it receives.
474 | The events are triggered by messages sent from a process to another or from timers that tick at regular intervals.
475 | When a message is received by a process,
476 | a dispatching function will search which function handles that event and call it to perform the associated task.
477 |
478 | By timing in the dispatch function right before it calls the event handler and right after it returns,
479 | it is possible to measure precisely the time spent doing any task within the software.
480 | When the _profile-imsg_ traces are enabled,
481 | everything from a simple statistics increment to a DNS lookup or client connection gets its time of execution logged.
482 | The example below shows measures of the "set a value to a statistics counter" task:
483 |
484 | ```
485 | profile-imsg: control lka IMSG_STAT_SET 65 0.000908
486 | profile-imsg: control lka IMSG_STAT_SET 68 0.001047
487 | profile-imsg: control lka IMSG_STAT_SET 67 0.000908
488 | profile-imsg: control lka IMSG_STAT_SET 66 0.001886
489 | ```
490 |
491 | The format is simple,
492 | it consists of the process receiving the message,
493 | followed by the process sending the message,
494 | the message itself,
495 | the length of the message data and the time it took from entering to exiting its handler.
496 | Similar messages should have very close timings as care was taken to make their execution time predictable,
497 | but they should also be as small as possible to optimize performances.
498 |
499 |
500 | ### profile-queue traces
501 | OpenSMTPD uses a very conservative approach and follows a specific sequence of operations to protect e-mails from being lost.
502 | This sequence ensures that no matter what happens,
503 | as long as the operating system and disk are not lying about their state,
504 | the data is never in an inconsistent state:
505 | either it is accepted because it was completely written to a persistent storage,
506 | or it was not accepted.
507 | There is no risk of it being accepted before it is written or only partially written,
508 | even in the case of a power shutdown happening at a very _bad time_[4](#4).
509 |
510 | This comes at a cost in terms of performances:
511 | multiple operations to ensure everything is fine will necessarily cost more than just writing blindly to disk.
512 | It doesn't mean that the queue is slow,
513 | it just means that it is slower than an unsafe approach because we have to pay a performance penalty price for doing things right.
514 | Since speed is a major goal,
515 | it was necessary to measure the time spent in the queue operations to ensure that the penalty was as low as possible.
516 | If a queue operation takes longer than the system calls it relies upon,
517 | then it means the code is suboptimal and needs to be reworked.
518 |
519 | The _profile-queue_ was written to understand precisely which queue operations were taking time and assessing if that time was expected,
520 | if less pressure could be put on disks and if sequences of operations could be rearranged to be more efficient without sacrificing safety.
521 | The following is an example of the output:
522 |
523 | ```
524 | profile-queue: queue_envelope_walk 0.544201
525 | profile-queue: queue_message_create 0.732912
526 | profile-queue: queue_envelope_create 0.846962
527 | profile-queue: queue_message_commit 1.668082
528 | profile-queue: queue_message_fd_r 0.009569
529 | profile-queue: queue_envelope_delete 0.805965
530 | ```
531 |
532 | The format is simpler than imsg profiling,
533 | it consists of the queue operation name followed by the timing for the call.
534 | The profiling happens at an API level and not within the queue backend code,
535 | meaning that profiling code can time any custom queue implementing the few basic operations required,
536 | not just the official queue backend shipped with OpenSMTPD.
537 |
538 |
539 | ## Reporting
540 | As explained earlier,
541 | after 6.4 the logs were simplified for humans and this resulted in a bit of information loss for tools parsing them.
542 |
543 | The reporting mechanism was introduced with OpenSMTPD 6.5,
544 | providing a stream of events that tools can consume in a format that is easy to parse and that is also versioned.
545 | As of this writing,
546 | the reporting mechanism is only used to report incoming and outgoing SMTP events,
547 | which is the void that was created when simplifying logs,
548 | but the direction that is taken is to have all of the subsystems report their events.
549 | When this is achieved,
550 | we'll essentially have an event bus available and it will be possible to add a lot of new features to OpenSMTPD through external tools reacting to anything happening in the daemon,
551 | without requiring changes made to it.
552 | As a matter of fact,
553 | the statistics mechanism described a few sections above is a first use-case of something that should be rebuilt on top of reporting.
554 |
555 | Contrarily to human logs,
556 | the reporting output is very verbose as shown in the following example for a single incoming SMTP session:
557 |
558 | ```
559 | report|0.4|1572482919.566813|smtp-in|link-connect|cc7627cc19ffc047|mail-wm1-x32c.google.com|pass|[2a00:1450:4864:20::32c]:33521|[2001:19f0:6c01:3d9:5400:1ff:fee7:78b7]:25
560 | report|0.4|1572482919.568165|smtp-in|filter-response|cc7627cc19ffc047|connected|proceed
561 | report|0.4|1572482919.568196|smtp-in|protocol-server|cc7627cc19ffc047|220 in.mailbrix.mx ESMTP OpenSMTPD
562 | report|0.4|1572482919.568198|smtp-in|link-greeting|cc7627cc19ffc047|in.mailbrix.mx
563 | report|0.4|1572482919.579574|smtp-in|protocol-client|cc7627cc19ffc047|EHLO mail-wm1-x32c.google.com
564 | report|0.4|1572482919.579855|smtp-in|filter-response|cc7627cc19ffc047|ehlo|proceed
565 | report|0.4|1572482919.579862|smtp-in|link-identify|cc7627cc19ffc047|EHLO|mail-wm1-x32c.google.com
566 | report|0.4|1572482919.579890|smtp-in|protocol-server|cc7627cc19ffc047|250-in.mailbrix.mx Hello mail-wm1-x32c.google.com [2a00:1450:4864:20::32c], pleased to meet you
567 | report|0.4|1572482919.579894|smtp-in|protocol-server|cc7627cc19ffc047|250-8BITMIME
568 | report|0.4|1572482919.579902|smtp-in|protocol-server|cc7627cc19ffc047|250-ENHANCEDSTATUSCODES
569 | report|0.4|1572482919.579906|smtp-in|protocol-server|cc7627cc19ffc047|250-SIZE 36700160
570 | report|0.4|1572482919.579909|smtp-in|protocol-server|cc7627cc19ffc047|250-DSN
571 | report|0.4|1572482919.579913|smtp-in|protocol-server|cc7627cc19ffc047|250-STARTTLS
572 | report|0.4|1572482919.579940|smtp-in|protocol-server|cc7627cc19ffc047|250 HELP
573 | report|0.4|1572482919.591399|smtp-in|protocol-client|cc7627cc19ffc047|STARTTLS
574 | report|0.4|1572482919.591604|smtp-in|filter-response|cc7627cc19ffc047|tls|proceed
575 | report|0.4|1572482919.591623|smtp-in|protocol-server|cc7627cc19ffc047|220 2.0.0 Ready to start TLS
576 | report|0.4|1572482919.641623|smtp-in|link-tls|cc7627cc19ffc047|TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256
577 | report|0.4|1572482919.653095|smtp-in|protocol-client|cc7627cc19ffc047|EHLO mail-wm1-x32c.google.com
578 | report|0.4|1572482919.653392|smtp-in|filter-response|cc7627cc19ffc047|ehlo|proceed
579 | report|0.4|1572482919.653398|smtp-in|link-identify|cc7627cc19ffc047|EHLO|mail-wm1-x32c.google.com
580 | report|0.4|1572482919.653420|smtp-in|protocol-server|cc7627cc19ffc047|250-in.mailbrix.mx Hello mail-wm1-x32c.google.com [2a00:1450:4864:20::32c], pleased to meet you
581 | report|0.4|1572482919.653427|smtp-in|protocol-server|cc7627cc19ffc047|250-8BITMIME
582 | report|0.4|1572482919.653433|smtp-in|protocol-server|cc7627cc19ffc047|250-ENHANCEDSTATUSCODES
583 | report|0.4|1572482919.653439|smtp-in|protocol-server|cc7627cc19ffc047|250-SIZE 36700160
584 | report|0.4|1572482919.653442|smtp-in|protocol-server|cc7627cc19ffc047|250-DSN
585 | report|0.4|1572482919.653446|smtp-in|protocol-server|cc7627cc19ffc047|250 HELP
586 | report|0.4|1572482919.665304|smtp-in|protocol-client|cc7627cc19ffc047|MAIL FROM: SIZE=2393
587 | report|0.4|1572482919.665623|smtp-in|filter-response|cc7627cc19ffc047|mail-from|proceed
588 | report|0.4|1572482919.667944|smtp-in|tx-begin|cc7627cc19ffc047|0cd3aa24
589 | report|0.4|1572482919.667954|smtp-in|tx-mail|cc7627cc19ffc047|0cd3aa24|gilles.chehade@gmail.com|ok
590 | report|0.4|1572482919.667964|smtp-in|protocol-server|cc7627cc19ffc047|250 2.0.0 Ok
591 | report|0.4|1572482919.679517|smtp-in|protocol-client|cc7627cc19ffc047|RCPT TO:
592 | report|0.4|1572482919.679806|smtp-in|filter-response|cc7627cc19ffc047|rcpt-to|proceed
593 | report|0.4|1572482919.682889|smtp-in|tx-envelope|cc7627cc19ffc047|0cd3aa24|0cd3aa24acdd1899
594 | report|0.4|1572482919.682900|smtp-in|tx-rcpt|cc7627cc19ffc047|0cd3aa24|gilles@poolp.org|ok
595 | report|0.4|1572482919.682907|smtp-in|protocol-server|cc7627cc19ffc047|250 2.1.5 Destination address valid: Recipient ok
596 | report|0.4|1572482919.694237|smtp-in|protocol-client|cc7627cc19ffc047|DATA
597 | report|0.4|1572482919.694461|smtp-in|filter-response|cc7627cc19ffc047|data|proceed
598 | report|0.4|1572482919.695327|smtp-in|tx-data|cc7627cc19ffc047|0cd3aa24|ok
599 | report|0.4|1572482919.695334|smtp-in|protocol-server|cc7627cc19ffc047|354 Enter mail, end with "." on a line by itself
600 | report|0.4|1572482920.718062|smtp-in|protocol-client|cc7627cc19ffc047|.
601 | report|0.4|1572482920.718353|smtp-in|filter-response|cc7627cc19ffc047|commit|proceed
602 | report|0.4|1572482920.718469|smtp-in|tx-commit|cc7627cc19ffc047|0cd3aa24|2859
603 | report|0.4|1572482920.718471|smtp-in|tx-reset|cc7627cc19ffc047|0cd3aa24
604 | report|0.4|1572482920.721023|smtp-in|protocol-server|cc7627cc19ffc047|250 2.0.0 0cd3aa24 Message accepted for delivery
605 | report|0.4|1572482920.733447|smtp-in|protocol-client|cc7627cc19ffc047|QUIT
606 | report|0.4|1572482920.742012|smtp-in|filter-response|cc7627cc19ffc047|quit|proceed
607 | report|0.4|1572482920.742032|smtp-in|protocol-server|cc7627cc19ffc047|221 2.0.0 Bye
608 | report|0.4|1572482920.742189|smtp-in|link-disconnect|cc7627cc19ffc047
609 | ```
610 |
611 | The format is very simple, it begins with a common prefix for all subsystems:
612 |
613 | ```
614 | |||
615 | ```
616 |
617 | Followed by event messages with their specific optional paramters:
618 |
619 | ```
620 | |||||
621 | ```
622 |
623 | So taking one random line from above and dissecting it:
624 | ```
625 | report|0.4|1572482919.667954|smtp-in|tx-mail|cc7627cc19ffc047|0cd3aa24|gilles.chehade@gmail.com|ok
626 | ```
627 |
628 | We have a message of type "report",
629 | respecting version 0.4 of the reporting protocol,
630 | that was emitted at 1572482919.667954 (unix timestamp),
631 | from the subsystem "smtp-in".
632 | The message informs us that an event "tx-mail" occured,
633 | providing us the session identifier for the client that issues the MAIL FROM,
634 | the transaction identifier (message id) for the current transaction,
635 | the e-mail address that was submitted by the session,
636 | and the result for that sender.
637 |
638 | As you can see,
639 | the lines are very easy to parse,
640 | very complete in terms of informations provided,
641 | and tools can easily aggregate whatever information they want to do their work.
642 | This is far more powerful than the parsing of user logs,
643 | while not being any more complex.
644 | A very interesting side-effect is that tools can parse this stream from a log file,
645 | performing asynchronous processing,
646 | or they can plug directly to the daemon's filtering API to receive these events in real-time,
647 | performing trigger-based processing.
648 |
649 | Describing all these events would be pages and pages of details,
650 | I don't think it's necessary at this point to give that information in this book,
651 | mostly because pretty much all events are straightforward to understand.
652 | They will be part of the OpenSMTPD documentation,
653 | future revisions of the book may revisit them if there's demand.
654 |
655 | For the time being,
656 | knowing that such a mechanism exists and lets you build tools around OpenSMTPD is enough to point you at the right direction.
657 | From extracting IP addresses for firewall-blocking clients to generating advanced dashboards in a reporting tool,
658 | anything that requires programmatic processing of logs is better served by the reporting mechanism.
659 |
660 |
661 |
662 | [1](#1) Commonly known as "where the f did this e-mail go" situations.
663 |
664 | [2](#2) You have never seen an angry person until you tell someone that you lost their e-mails.
665 |
666 | [3](#3) A LOOOOOOOOOOOT.
667 |
668 | [4](#4) Including "sorry, I tripped on the cable" time.
--------------------------------------------------------------------------------
/src/images/cc-by-nc-nd-88x31.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/poolpOrg/OpenSMTPD-book/cb2cd0e4429d0c4710b52d0dfd8a84f18d3777ea/src/images/cc-by-nc-nd-88x31.png
--------------------------------------------------------------------------------
/src/images/opensmtpd.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/poolpOrg/OpenSMTPD-book/cb2cd0e4429d0c4710b52d0dfd8a84f18d3777ea/src/images/opensmtpd.png
--------------------------------------------------------------------------------
9 | 
4 |
5 |