11 |
12 |
13 |
14 | # Chapter 5 - Links in the file system
15 |
16 | It is possible to have a file referenced by multiple names. The concept of file system links can be confusing, but it's important to be aware of them, because you might encounter them from time to time.
17 |
18 | > [!IMPORTANT]
19 | > Only some file systems support unix-style [symbolic and hard links](#symbolic-and-hard-links).
20 |
21 | - Both symbolic and fixed links are supported by file systems designed for unix-like operating systems such as EXT4, BTRFS and ZFS.
22 | - NTFS, the default file system for Windows, can handle symbolic links on GNU/Linux, but does not fully support hard links [^wiki-hard].
23 | - Neiher symbolic nor hard links are supported by FAT32 or exFAT, the other two, common general purpose file systems created by Microsoft.
24 |
25 | [^wiki-hard]: [Wikipedia - NTFS Hard links, accessed 2025](https://en.wikipedia.org/wiki/NTFS#Hard_links)
26 |
27 |
28 |
29 | ## 5.1 Drag and drop
30 |
31 | ### Freedesktop shortcut files
32 |
33 | > [!TIP]
34 | > Modern GNU/Linux desktop environments support Windows-like functionality of shortcut files (see [Desktop Entry Specification](https://specifications.freedesktop.org/desktop-entry-spec/latest/)).
35 |
36 | > The specification of desktop entry files are part of the [Freedesktop.org](https://www.freedesktop.org/wiki/Specifications/) project. The goal of Freedesktop.org is to create open standards and interoperability between different GNU/Linux-based desktop environments. Most Freedesktop.org projects are designed to be independent of any specific desktop environment, and improve their compatibility.
37 | >
38 | > Freedesktop.org (formerly X Desktop Group) was founded year 2000 by Havoc Pennington, an American programmer and free software advocate.
39 |
40 | 1. Lets try creating a shortcut file by drag and drop. When you drag a URL (such as ) from Firefox and drop it to ` ~/Desktop/ ` a [desktop entry file](https://specifications.freedesktop.org/desktop-entry-spec/desktop-entry-spec-latest.html) is created. Grab the lock icon preceding the URL.
41 |
42 | 2. Freedesktop shortcuts are actually text files. They can be opened and edited by any text editor. They may be identified by their file name extension ` *.desktop `. However modern file managers (such as Nemo) may not display the filename such as ` GitHub.desktop `. Instead they usually display a string from inside the file, specified by the value of the key: ` Name = Link to GitHub `.
43 |
44 | 3. The file content will be something like:
45 |
46 | ```
47 | [Desktop Entry]
48 | Encoding=UTF-8
49 | Name=Link to GitHub
50 | Type=Link
51 | URL=https://github.com/
52 | Icon=text-html
53 | ```
54 |
55 | 4. Despite the name ` *.desktop `, such files do not need to be located in the ` ~/Desktop/ ` directory. In fact, all items in the main [Desktop Menu](https://www.freedesktop.org/wiki/Specifications/menu-spec/) are ` *.desktop ` files and located in ` /usr/share/applications/ `.
56 |
57 | 5. Use of these desktop entry files is similar to how their equivalents are used in Windows. Double-clicking will trigger the functionality specified in the file, such as opening a URL (as created in the example above).
58 |
59 | ### Symbolic link by drag and drop
60 |
61 | Graphical file managers (such as Nemo) provide an easy and automatic method of creating another type of shortcut. Holding **` Ctrl + Shift `** while dropping a file will create a [symbolic link](#symbolic-and-hard-links), rather than copying or moving the file. In some distributions, a small menu appears whenever a file is dropped; offering a choice of copying, moving or linking the file. [^shotts-guilink]
62 |
63 | > In addition, there are so-called [hard links](#symbolic-and-hard-links). It is important to be aware of hard links, because you might encounter them from time to time, even though modern practice prefers symbolic links. [^shotts-hard]
64 |
65 |
66 |
67 | ## 5.2 Symbolic and hard links
68 |
69 | Links (symbolic and hard) are a way to refer to a file on disk. The best understanding of this is that both the contents of the file and a reference to the contents (i.e. the name of the file) are stored on the disk. And that there may be several of these references.
70 |
71 | Symbolic links were created to overcome the two disadvantages of hard links: [^shotts-syml]
72 | 1. A hard link cannot reference a directory (only a file). [^shotts-syml]
73 | 2. Hard links cannot reference any file that is not on the same disk partition as the link itself. [^shotts-syml]
74 |
75 | > [!NOTE]
76 | > Symbolic and hard links do not work for URLs. Use [.desktop files](#freedesktop-shortcut-files) for URLs.
77 |
78 | ### Hard link
79 |
80 | A **hard link** is a reference to the inode number of a file on physical storage. Hard links must be on the same physical partition as the contents of the file itself, because the inode number is unambiguous only within the file system. A file will always have at least one hardlink because the file's name is created by a hardlink. The ` $ ls ` command has a way to reveal this information when invoked with the ` -i ` option. [^shotts-hard]
81 |
82 | The ` $ rm ` command deletes files, including a hard links. However, data is still accessible as long as another hard link exists. As there can be several hard links to the same file, the space occupied by the file on the disk is not released until all the hard links to it have been removed. When one of several hard links is deleted, only the link is removed; even if you delete the original file, which is a hardlink itself. The contents of the file itself continue to exist (its space is not deallocated) until all links to the file are deleted. To get rid of data you must remove all hard links. A remove command applied to a hard link also deletes the contents of the file when deleting the last hard link. [^shotts-hard]
83 |
84 | When we list a directory containing a hard link we will see no special indication of the link, unless we know what we are looking for (see [Section: Identifying a hard link is difficult](#identifying-a-hard-link-is-difficult)).
85 |
86 | ### Symbolic link
87 |
88 | A **symbolic link** can point to any file, including different file systems and destinations that do not exist; this happens also if the destination of a symbolic link is deleted.
89 | - The remove command ` $ rm ` applied to a symbolic link, or ` Delete ` in a graphical file manager, only removes the symbolic link, never the target file to which the link refers.
90 | - In most other file processing operations, the links are "decompressed" so that the paths through them work normally. Most file operations are carried out on the link's target, not the link itself. For example, if we write something to the symbolic link, the referenced file is written to. [^shotts-syml]
91 | - Symbolic links contain a text pointer to the target file or directory. However, a symlink file cannot be opened in a text editor. Opening a symlink always opens the target file. Reading the destination from a symlink file requires a special system call such as ` $ readlink ↵ `.
92 | - A symbolic link can be relative, so that it is interpreted in relation to its parent directory.
93 | - If the target file is deleted before the symbolic link, the link will continue to exist but will point to nothing. In this case, the link is said to be broken. In many implementations, the ` $ ls ` command will display broken links in a distinguishing color such as red, to reveal their presence. [^shotts-syml]
94 |
95 |
96 |
97 | ## 5.3 Examples of links
98 |
99 | ### Lets make links
100 |
101 | ```
102 | # Create a hard-link:
103 | $ ln /path/to/existing_file /path/to/new_hardlink↵
104 |
105 | # Create or update a symbolic-link:
106 | $ ln -sf /path/to/existing_file /path/to/symlink↵
107 | ```
108 |
109 | Graphical file managers (such as Nemo) provide an easy method for creating symbolic links: Holding **` Ctrl + Shift `** while dropping a file will create a symlink, rather than copying or moving the file. In some distributions, a small menu appears whenever a file is dropped; offering a choice of copying, moving or linking the file.
110 |
111 | ### Identifying symbolic links is easy
112 |
113 | In the command interface, symbolic links are fairly easy to identify using the ` $ ls -li ` command. The list command reveals symlinks with an arrow ` -> `, with the link file on the left and the referenced file on the right. The arrow is also used in graphical file managers and appears as part of the file icon.
114 |
115 | A good example of a symlink is ` $ x-terminal-emulator ` which points to an actual executable such as ` $ gnome-terminal `:
116 |
117 | ```
118 | $ which x-terminal-emulator↵
119 | /usr/bin/x-terminal-emulator
120 |
121 | $ ls -li /usr/bin/x-terminal-emulator↵
122 | 266302 lrwxrwxrwx 1 root root 14 9. 5. 15:48 /usr/bin/x-terminal-emulator -> gnome-terminal
123 | ```
124 |
125 | ### Identifying a hard link is difficult
126 |
127 | In the command interface, hard links can be identified with ` $ ls -li ↵ ` if you know what you are looking for. The number between the permissions column and the owner name, indicates the number of hard links to the file.
128 |
129 | ```
130 | $ ls -li /usr/bin/c++↵ # Note number 4 before "root"
131 | 309582 -rwxr-xr-x 4 root root 1076160 21. 5. 23:19 /usr/bin/c++
132 |
133 | $ stat /usr/bin/c++↵ # Note number 4 after "Links:"
134 | File: /usr/bin/c++
135 | Size: 1076160 Blocks: 2104 IO Block: 4096 regular file
136 | Device: 259,2 Inode: 309582 Links: 4
137 | Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
138 | Access: 2024-06-09 06:39:35.340520475 +0300
139 | Modify: 2024-05-21 23:19:21.184657000 +0300
140 | Change: 2024-06-02 10:26:47.650834218 +0300
141 | Birth: 2024-06-02 10:26:47.629834110 +0300
142 | ```
143 |
144 | > [!NOTE]
145 | > The number of hard links is shown also in conjunction with directories. This can be confusing, as the file system does not allow hardlinks to be created for directories. The hard links to a specific directory comes from its adjacent directories. The minimum is always two: One from the parent directory and one from the current directory (notation ` ./ `). And the number of hardlinks will increase by the number of all immediate subdirectories (notation ` ../ `).
146 |
147 |
148 |
149 | ### Lets find all hard links
150 |
151 | Once you find a file with more than one hard-link, you might want to find all the links to the file:
152 |
153 | ```
154 | $ sudo find / -inum 309582↵
155 | ./usr/bin/g++
156 | ./usr/bin/x86_64-solus-linux-c++
157 | ./usr/bin/c++
158 | ./usr/bin/x86_64-solus-linux-g++
159 |
160 | $ sudo find / -samefile /usr/bin/c++↵
161 | /usr/bin/g++
162 | /usr/bin/x86_64-solus-linux-c++
163 | /usr/bin/c++
164 | /usr/bin/x86_64-solus-linux-g++
165 | ```
166 |
167 |
168 |
169 | [^shotts-guilink]: [The Linux Command Line 2024, Chapter: 4, Section: Creating Symlinks With The GUI](http://linuxcommand.org/tlcl.php)
170 |
171 | [^shotts-hard]: [The Linux Command Line 2024, Chapter: 4, Section: Hard Links](http://linuxcommand.org/tlcl.php)
172 |
173 | [^shotts-syml]: [The Linux Command Line 2024, Chapter: 4, Section: Creating Symbolic Links](http://linuxcommand.org/tlcl.php)
174 |
175 |
176 |
--------------------------------------------------------------------------------
/chapter/08-text-editors.md:
--------------------------------------------------------------------------------
1 |
2 |
3 | > [!TIP]
4 | > The table of contents can be accessed by pressing the unordered list icon  on top the right corner.
5 |
6 |
11 |
12 |
13 |
14 | # Chapter 8 - Text editors
15 |
16 | A **text editor** is a simple computer program that allows users to create, change, or edit plain text files. Traditionally text editors were used for creating computer programs, web pages, etc. Today text editors are most commonly used for things like editing configuration files, simple note taking, analyzing log files, middleware for stripping formatting from copy-pasted text and making some quick find and replace operations; not so much for programming or creating documents, as it was in the past.
17 |
18 | Traditional [terminal-based text editors](#terminal-based-text-editors) can be categorized in [line oriented](#line-editorsss) and [screen oriented](#screen-editors) text editors. Screen oriented editors, feature the ability to modify any visible text on the screen by moving the cursor to its location. For most purposes, line oriented editors were superseded by screen oriented editors, because line editors were relics from the days of teletypes, when terminals used to print, on paper, very slowly.
19 |
20 | [Modern text editors](#gui-text-editors) for the desktop can be very simple plain text editors offering a minimal set of features (such as ` $ gedit ` and ` $ xed `), or considerably more advanced. Advanced text editors such as [Kate](https://kate-editor.org/about-kate/) offer a wide range of features and a plugin system.
21 |
22 |
23 |
24 | ## 8.1 Terminal-based text editors
25 |
26 | Until late 1970's, terminals with a screen were a rare luxury, let alone mice or graphic display devices. Work was done on a workstation resembling a large electric typewriter (called teletypes). Editing text files on teletypes was entirely command-based. They printed everything on paper instead of displaying it on a screen. Users relied on **line editors** using commands to modify specific lines, append new text, or delete sections. Since there was no real-time display, users had to print parts of the file repeatedly to review their changes. This method was slow and required memorizing commands for precise editing. [^petteri-jarvinen]
27 |
28 | Line editors [GNU ed](https://www.gnu.org/software/ed/manual/ed_manual.html) `$ ed` and [GNU sed](https://www.gnu.org/software/sed/) `$ sed` faithfully follow these old principles. Although their interface is a blast from past decades, they have their own merits (see [Section: Line editors: Ed and Sed](#line-editors-ed-and-sed)).
29 |
30 | [^petteri-jarvinen]: [Petteri Järvinen - PC-käyttäjän käsikirja 1993, page 256](https://www.petterijarvinen.fi/PC-k%C3%A4ytt%C3%A4j%C3%A4n%20k%C3%A4sikirja%20DOS%206.2%20(1993).pdf)
31 |
32 |
33 |
34 | ### Line editors: Ed and Sed
35 |
36 | > In 1969 Dennis Ritchie and Ken Thompson (original developers of the C language and the Unix operating system), while at Bell Labs, developed ed, a simplified form of qed (quick editor). [^raymond-ed]
37 |
38 | [**GNU ed**](https://www.gnu.org/software/ed/) is a clone of the original ` $ ed ` editor for Unix, and thus a classic example of a line-oriented text editor. It is truly Unix-minimalist way of plain-text editing. It has a simple, austere instruction based CLI, and there is no screen display. Unbelievable as it may seem to a modern reader, most of Unix's original code was written with this editor. As an **instruction based** editor, the user must specify a particular line of text, before making any changes. Let's see ` $ ed ` in action with the following listing: [^raymond-ed]
39 |
40 | ```
41 | $ ed sample.txt↵
42 | 15 # Ed informs us the file has some text in it already.
43 | a↵ # This is the command to append text to the file.
44 | Aaaa↵
45 | Bbbb↵
46 | .↵ # This is the command to terminate the append.
47 | w↵ # This is the command to write the file to disk.
48 | 25 # Ed informs us the new file size.
49 | 1,$p↵ # This is the command to print all lines from 1 to the last.
50 | q↵ # This is the command to end the editing session. You may also press Ctrl + D anytime to cancel and exit ed.
51 | ```
52 |
53 | > [!NOTE]
54 | > Press ` Ctrl + D ` to exit ed (repeatedly if necessary).
55 |
56 | [**GNU sed**](https://www.gnu.org/software/sed/) (stream editor) is also closely related to ` $ ed `. Many of the basic commands are the same, though non-interactive and designed to be invoked through command-line switches, rather than from standard input. It takes text input, performs some operation (or set of operations) on it, and outputs the modified text. ` $ sed ` is typically used for extracting part of a file using pattern matching or substituting multiple occurrences of a string within a file. [^gnu-sed]
57 |
58 | [^gnu-sed]: [GNU - GNU sed, accessed 2024](https://www.gnu.org/software/sed/)
59 |
60 | ```
61 | $ sed -i "$ a Aaaa\nBbbb" sample.txt↵ # Add rows Aaaa and Bbbb
62 | $ sed '4d' input.txt > output.txt↵ # Delete 4th line in file
63 | ```
64 |
65 | > [!NOTE]
66 | > Back in the early years of Unix, ed was used to create, display, modify and otherwise manipulate text files, both interactively and via shell scripts. Then ed became primarily a program-driven editing tool for scripts; a role to which editors with more elaborate modes of interactivity were and still are unsuited. For this reason most unices, still even today, include an implementation of ` $ ed ` and ` $ sed `. [^raymond-ed]
67 |
68 |
69 |
70 |
71 |
72 | ### Screen editor: Vi
73 |
74 | > In 1976 William Joy / Bill Joy (of e.g. Sun Microsystems), while at the University of California at Berkeley, releases the screen oriented text editor ` $ vi `, together with its line oriented counterpart ` $ ex `. Vi and ex are different interfaces to the same program behind it; one screen oriented, one line oriented. [^raymond-vi]
75 |
76 | **` $ vi `** stands for Visual Interface. The original Vi editor was the first attempt to bolt a visual interface onto the command set of Ed. Like Ed, it's commands are generally single keystrokes, and it is particularly well suited to use by touch-typists. The original Vi didn't have mouse support, editing menus, macros, assignable key bindings, or any form of user customization. In line with the religion of Ed, the Vi partisans considered the lack of these features a virtue. On this view, one of vi's most important virtues is that you can start editing immediately on a new Unix system without having to carry along your customizations or worrying that the default command bindings will be dangerously different from what you're used to. One characteristic of Vi that beginners tend to find frustrating is a result of its terse single-keystroke commands. [^raymond-vi]
77 |
78 | One feature that frustrates new users is vi's short one-key commands. Like line editors, vi has a moded interface (as in modes). You are either in command mode or text input mode. In command mode, typing text is interpreted as commands, and does strange and probably destructive things to the content. Switching between modes is bound to be a headache for new users. In text input mode, the only commands that work are ` Esc ` for exiting key mode, and the cursor movement keys (in newer versions). It is only in later vi implementations (such as vim) that a selection mode for selecting and editing text areas has been added. [^wiki-vi-fi]
79 |
80 | [^wiki-vi-fi]: [Wikipedia - vi, accessed 2025](https://fi.wikipedia.org/wiki/Vi)
81 |
82 | > [!NOTE]
83 | > Today, on most unix like systems and macOS, ` /usr/bin/vi ` is just a symlink to ` /usr/bin/vim `, a clone from Bill Joy's original vi. Vim is a much improved version of the old UNIX editor. Full vi compatibility is maintained, which explains the confusion between the terms vi and vim. And it runs on almost all flavours of UNIX. [^raymond-vi]
84 |
85 | Over the years, vi has bulked up considerably. The modern versions add mouse support, editing menus, multiple files in separate buffers, customization with a run-control file and unlimited undo (the original vi could only undo the last command). Vi looks rather bloated and flabby. There are hundreds of commands, many of them duplicative. Most users don't know more than 5% of the command set. Over the years, Vi has had progressively more and more special-purpose C code bolted onto it to perform tasks that Emacs would attack with Lisp code modules and subprocess control. The extensions are not, as in Emacs, libraries loaded as needed. Users pay the overhead for the resulting code bloat all the time. As a result, the size difference between a modern vi and a modern Emacs is not nearly as great as one might expect. [^raymond-vi]
86 |
87 | > [!TIP]
88 | > To exit vi, type: ` Esc + :qa!↵ ` where ` q ` stand for quit, ` a ` for all, and ` ! ` for without saving.
89 |
90 | Any other instruction to use vi(m) is beyond the scope of this book. But those interested to learn more, can start with this tutorial: [Karim Buzdar - Working With Vi Editor in Linux, published 2018](https://vitux.com/working-with-vi-editor-in-linux/).
91 |
92 | ### Screen editor: Emacs
93 |
94 | > In 1976 Richard Stallman, Guy Steele, and Dave Moon, while at MIT, release EMACS. [^raymond-emacs]
95 |
96 | [**GNU Emacs**](https://www.gnu.org/software/emacs/) stands for Editing MACroS. It was originally written in the late 1970's as a set of macros in an editor called TECO, then reimplemented several times in different ways. Objectives of the Emacs design are far more broad than most other text editors. Emacs wants to be a unified interface to all tools that operate on text. Emacs can give you capabilities resembling those of a conventional integrated development environment and beyond. You're not limited by the imagination of the IDE designer. You can tweak, customize, and add task-related intelligence using Emacs Lisp. [^raymond-emacs]
97 | - Emacs is a big, feature-laden program with a great deal of customizability. Emacs has an entire programming language (Elisp) inside it that can be used to write arbitrarily powerful editor functions. [^raymond-emacs]
98 | - Emacs could be considered a virtual machine or framework around a collection of small, sharp tools (the modes) that happen to be written in Elisp. But one doesn't have to learn Elisp to use Emacs. [^raymond-emacs]
99 | - Unlike Vi, Emacs doesn't have interface modes; Instead, commands are normally control characters or prefixed with an ` Esc ` i.e. ` ^[ `. [^raymond-emacs]
100 | - Emacs is correspondingly harder to learn than vi(m). Learning how to customize Emacs is an entire art in itself. [^raymond-emacs]
101 |
102 | > Now deprecated MicroEmacs and Pico were small editors that cloned the default keybindings and appearance of Emacs without emulating its extensibility. [^raymond-emacs]
103 | >
104 | > [GNU Nano](https://www.nano-editor.org/docs.php) ` $ nano ` is an "easy-to-use" text editor originally designed as a replacement for Pico, but nowadays implements several features that Pico lacked such as undo, redo and syntax highlighting. [^raymond-emacs]
105 |
106 | > [!TIP]
107 | > Press ` Ctrl + X, Ctrl + C ` to exit Emacs.
108 |
109 | ### Screen editor: Ne
110 |
111 | [**Ne**](https://ne.di.unimi.it) stands for Nice Editor. It is a screen oriented console text editor intended to provide an alternative to Vi that will be more familiar to beginners and modern users. It has been ranked as one of the best console text editors for GNU/Linux because:
112 | - It has a menu bar that is opened by pressing ` Esc ` followed by any arrow key.
113 | - It uses keyboard shortcuts derived from modern desktop interface such as ` Ctrl + Q ` to quit and ` Ctrl + O ` to open a file.
114 | - It has support for UTF-8 encoding.
115 | - It supports many features common in advanced text editors such as syntax highlighting, autocomplete, configurable menus and configurable keybindings, regular expressions (see [Chapter 6, Section: GNU Grep](06-inter.md#gnu-grep)).
116 | - It is portable across all POSIX-compliant operating systems such as GNU/Linux and macOS.
117 | - It remains usable on slow remote connections.
118 | - It can pipe a marked block of text through any command line filter.
119 |
120 | > [!TIP]
121 | > Press ` Ctrl + Q, Y ` to exit Ne.
122 |
123 | ### The default text editor
124 |
125 | The default text editor is defined by environment variables ` EDITOR ` and ` VISUAL `. ` EDITOR ` used to be for instruction-based editors like ` $ ed `. When editors with graphical terminal user interface (such as ncurses) came about, the editing process changed dramatically, and thus need for another variable arose. [^visual-vs-editor]
126 |
127 | On a modern GNU/Linux the terminal GUI editors and desktop GUI editors can both provide the same functionality; So you can set ` VISUAL ` to either a terminal-based editor such as ` $ export VISUAL="/usr/bin/ne"↵ ` or a modern desktop application such as ` $ export VISUAL="/usr/bin/gedit"↵ `. [^visual-vs-editor]
128 |
129 | However, ` EDITOR ` is meant for a fundamentally different workflow. Especially people who use slow serial lines to connect to embedded systems over dicey connections might still appreciate being able to have a preferred line mode editor distinct even from a visual editor like Vim, Nano or Ne. [^visual-vs-editor]
130 |
131 | > [!TIP]
132 | > Pressing ` Ctrl + X, Ctrl + E ` on a half typed command on the current shell prompt, will move the unfinished command to your editor of choice, where you can finish the command before it gets executed.
133 |
134 | [^visual-vs-editor]: [Stack Exchange - VISUAL vs. EDITOR, accessed 2024](https://unix.stackexchange.com/questions/4859/visual-vs-editor-what-s-the-difference)
135 |
136 |
137 |
138 | ## 8.2 Modern text editors
139 |
140 | **` $ xed `** is a simple text editor developed by the Linux Mint community for the Cinnamon desktop environment.
141 |
142 | **` $ gedit `** was the default text editor for the Gnome desktop environment until 2022. It has also been available for other desktop environments, but the future remains uncertain.
143 |
144 | **` $ gnome-text-editor `** is a simple text editor developed by Gnome Builder's creator Christian Hergert. Gnome Text Editor became the default editor for the Gnome desktop environment in 2022, but is also available for other desktop environments.
145 |
146 | **` $ kate `** is an advanced text editor developed by the KDE community, but is also available for other desktop environments.
147 | - There is is a stripped down version of Kate called **` $ kwrite `**, with not as many features. They share code, they are different options of the same program, but they present like different programs. KWrite is more of a standard plain text editor, whereas Kate can be more of an integrated development environment.
148 |
149 |
150 |
151 | [^raymond-ed]: [The Art of Unix Programming 2003, Chapter 13, Section: ed](http://www.catb.org/~esr/writings/taoup/html/ch13s02.html)
152 |
153 | [^raymond-vi]: [The Art of Unix Programming 2003, Chapter 13, Section: vi](http://www.catb.org/~esr/writings/taoup/html/ch13s02.html)
154 |
155 | [^raymond-emacs]: [The Art of Unix Programming 2003, Chapter 13, Section: Emacs](http://www.catb.org/~esr/writings/taoup/html/ch13s02.html)
156 |
--------------------------------------------------------------------------------
/chapter/10-additional.md:
--------------------------------------------------------------------------------
1 |
2 |
3 | > [!TIP]
4 | > The table of contents can be accessed by pressing the unordered list icon  on top the right corner.
5 |
6 |
11 |
12 |
13 |
14 | # Chapter 10 - Appendix
15 |
16 |
17 |
18 | ## 10.1 Installing GNU/Linux
19 |
20 | A GNU/Linux distribution (or several of them) can be installed in place of, or alongside Microsoft Windows and Apple macOS. You can have multiple operating systems installed on one computer. This setup is often referred to as dual-booting or multi-booting. With multiple operating systems installed, you will typically see a boot menu when you start your computer. This menu allows you to choose which operating system to boot into.
21 |
22 | Installation of a GNU/Linux typically starts with the creation of a bootable USB drive, but most distributions can be run directly from such bootable USB drives without installing (onto the computer's internal storage drive). This allows users to try out GNU/Linux and it's various distributions without making any changes to their existing system. It’s a way to experiment, recover data from a broken system, or have a portable operating system that one can use on different machines.
23 |
24 | ### Create a bootable USB drive
25 |
26 | Creating a bootable USB drive on Windows is not trivial, and requires installing a boot drive creation utility such as [Rufus](https://rufus.ie/en/), [Balena Etcher](https://etcher.balena.io) or [Ventoy](https://www.ventoy.net/). In addition, you must download the correct ISO file from the official website of the distribution you want to (see [Chapter 1, Section: Suggested Distributions](01-intro.md#suggested-distributions)). The ISO file must be copied to a USB drive in a way that makes it bootable, using afore mentioned software.
27 |
28 | ### Boot from a USB drive
29 |
30 | Booting a computer from a USB drive may not trivial, as it usually requires changing some UEFI/BIOS system settings. Some GNU/Linux distributions cannot be installed unless the Secure Boot feature is disabled. Installation continues by changing the boot order to favor USB before the computer's internal boot drive. [^koodikanava]
31 |
32 | The UEFI/BIOS settings can usually be accessed by pressing a hardware-specific key such as Del, Esc, F2, F10, F12 during boot-up. Ease of access to system settings also depends on the existing UEFI/BIOS settings. You may have the post boot delay time set to zero, during which pressing the hardware-specific key proves impossible - In which case you will never get in. [^koodikanava]
33 |
34 | You can try to instruct an already running computer to boot into UEFI/BIOS from: [^koodikanava]
35 | - The Windows command line ` Win + R ` + ` cmd.exe↵ `:
36 | - With the command ` shutdown /r /fw↵ `
37 | - A GNU/Linux terminal:
38 | - With the command ` $ sudo systemctl reboot --firmware-setup↵ `
39 |
40 | [^koodikanava]: [Koodi Kanava, Kimmo Kujansuu - Helpolla komennolla pääset biosiin, accessed 2025](https://www.youtube.com/watch?v=YbKrDbHRsTg)
41 |
42 |
43 |
44 | ### Disk partitioning in live mode
45 |
46 | After a successful boot with the USB drive, you should arrive to a graphical user interface of the GNU/Linux operating system referred to as a live session. Live mode session is almost like a fully functioning instance of the operating system. In live mode, you have access to most of the features and applications as if you had installed it on your internal storage drive. Follow the on-screen prompts to begin the installation process. If not prompted, installation continues by finding the installer. Most live sessions have an "Install" icon on the desktop for this. Progressing from here requires some understanding of disk partitioning.
47 |
48 | > [!WARNING]
49 | > When partitioning a storage disk for GNU/Linux, it is strongly encouraged to expand the (U)EFI FAT32 boot partition to at least 500 MB. The 100 MB created by Windows will most likely prove too low in the long run. Problems may occur when the Linux kernel is updated. Everything may seem to work, but the Linux kernel, doesn't update, and you might not even get an error message about it.
50 |
51 | **EXT4** (Fourth Extended Filesystem) is a good choice and the default filesystem for most GNU/Linux distributions. More advanced users may prefer the BTRFS (B-Tree File System) for its additional features such as snapshots - However BTRFS is slower than EXT4 especially when dealing with large data writes.
52 |
53 | **exFAT** (Extensible File Allocation Table) and **NTFS** (New Technology File System) are good file system alternatives for partitions to be shared between different operating systems such as GNU/Linux and Microsoft Windows. Of these, exFAT is the most straightforward and the best performing. NTFS is the default file system for the Windows operating system, but Windows-level NTFS performance is currently not achievable in GNU/Linux.
54 |
55 | #### Need for swap has decreased
56 |
57 | The complex disk partitioning traditionally used in GNU/Linux (even single-disk configurations) is no longer necessary on today's home computers; and it has little use in home use today. If the disk system is partitioned the old style and runs out of space, then in the worst case, the machine may even crash.
58 |
59 | > The first function created for Linux at the request of an outsider was the ability to extend main memory with **swap memory**, completed on Christmas Day 1991. Torvalds ’own 386 machine (which was assembled from parts, unbranded) had four megabytes of memory, but users with a smaller number (especially one German user who originally made the request) were in trouble with the lack of main memory. This feature was significant in early 1992 because it played a major role in system performance and contributed positively to the Linux kernel.
60 |
61 | In Linux, the swap space has traditionally been located on a separate hard disk partition with its own file system optimized for swap use. The speed advantage achieved in this way is now so marginal that the same can be done well by taking the space available for virtual memory from a standard disk partition.
62 |
63 | In addition, the need for swaps has decreased or even disappeared as the amount of main memory increased. At the same time, the opposite of swap is common, in which files in mass memory, which are often needed, are copied to main memory to speed up their reading (see [Chapter 2, Section: RAM-disk](02-basic.md#ram-disk)).
64 |
65 | > [!TIP]
66 | > Partitions can be resized, moved, deleted and created with `$ gparted ` in GNU/Linux and with ` diskmgmt.exe ` in Microsoft Windows (` Win + X ` + Disk Management).
67 |
68 |
69 |
70 | ## 10.2 Additional tips with the terminal
71 |
72 | ### Suspend to RAM
73 |
74 | ` $ systemctl suspend↵ ` = Suspend to RAM. Works by cutting off power to most parts of the machine aside from the RAM, which is required to restore the machine's state. Because of the large power savings, it is advisable for laptops to automatically enter this mode when the computer is running on batteries and the lid is closed (or the user is inactive for some time).
75 |
76 | ### Boot time
77 |
78 | There is a command to find out how long it took to boot your GNU/Linux system:
79 | ```
80 | $ systemd-analyze↵
81 | Startup finished in 8.299s (firmware) + 3.081s (loader) + 1.794s (kernel) + 2.202s (initrd) + 6.983s (userspace) = 22.362s
82 | ```
83 |
84 | > [!TIP]
85 | > Append the ` blame ` flag for additional details.
86 |
87 |
90 |
91 | ### Analyze disk space usage
92 |
93 | The following command will display how much space is being used by folders under the root ` / ` directory:
94 |
95 | ```
96 | $ sudo du -hsx /* | sort -rh | head -n 40↵
97 | 14G /usr
98 | 5,4G /home
99 | 1,4G /var
100 | 1,3G /lib64
101 | 96M /root
102 | 18M /etc
103 | 11M /run
104 | 280K /tmp
105 | 16K /lost+found
106 | 4,0K /srv
107 | 4,0K /mnt
108 | 4,0K /media
109 | 4,0K /boot
110 | 0 /sys
111 | 0 /sbin
112 | 0 /proc
113 | 0 /lib
114 | 0 /dev
115 | 0 /bin
116 | ```
117 |
118 |
119 |
120 | ## 10.3 Additional tips for the desktop interface
121 |
122 | ### Launching desktop applications from the commandline
123 |
124 | Sometimes, a program will fail to start up when launched from the graphical menu. By launching it from the command line instead, we may see an error message that will reveal the problem. Also, some graphical programs have interesting and useful command line options.
125 |
126 | Graphical dekstop environments tend to hide the file names of programs. One way to find out the file names of executables is to browse the [Desktop Menu](https://www.freedesktop.org/wiki/Specifications/menu-spec/) [shortcuts folder](05-links.md#freedesktop-shortcut-files) at ` /usr/share/applications/ ` with a file manager, right click a shortcut file and choose open with text editor. Search for the line that begins with ` Exec= ` to find out the information you are looking for.
127 |
128 | ### Keyboard shortcuts
129 |
130 | Hold ` Alt + Tab ` repeatedly = Cycle through open windows.
131 |
132 | ` Alt + F2 ` = Run a Quick Command (instead of opening a terminal). Works also as a calculator in some distributions - But not Solus.
133 |
134 | #### Virtual desktops
135 |
136 | ` Ctrl + Alt + Right ` = Open the next virtual desktop.
137 |
138 | ` Ctrl + Alt + Left ` = Return to the previous virtual desktop.
139 |
140 | #### Zoom
141 |
142 | - In = ` Ctrl + [+] `
143 | - Out = ` Ctrl + [-] `
144 | - Reset = ` Ctrl + 0 `
145 |
146 | > [!NOTE]
147 | > The signs ` + `, ` - ` and ` 0 ` must be pressed next to the letter keys. The corresponding characters on the numeric keypad do not work with the default settings.
148 |
149 |
150 |
151 | ## 10.4 Encrypted password system
152 |
153 | Any computer system that requires password authentication must contain a database of passwords in some form. Because password databases are vulnerable to theft, storing passwords in plaintext is dangerous. Therefore most databases store only cryptographic hashes of the user's passwords.
154 |
155 | ### Hashing
156 |
157 |
158 |
159 | Passwords stored in the ` /etc/shadow ` file are actually hashes. Even the authentication system can't determine what a user's password is by merely looking at the stored value. Instead authentication is determined, without ever actually decrypting the stored password on the system. [^hashing]
160 |
161 | When a user enters a password for authentication, the system computes the hash value for the provided password, and that hash value is compared to the stored hash for that user. Authentication is successful if the two hashes match. [^hashing]
162 |
163 | **Hashing** is a one-way process. The hashed result cannot be reversed to expose the original data. The checksum is a string of output that is a set size. Technically, this means that hashing is not encryption because encryption is intended to be reversed (decrypted). [^hashing]
164 |
165 | The purpose of password hashing is not to protect the system from being breached, but to protect the passwords if a breach does occur. If a password database gets hacked, and the passwords are unprotected, then malicious hackers can use those passwords to compromise users' accounts on other services, as most people use the same password everywhere. [^hashing]
166 |
167 | Cryptography has three goals: [^hashing]
168 | 1. **Authenticity** = to prove where a file originated.
169 | 2. **Integrity** = to prove that a file has not changed unexpectedly.
170 | 3. **Confidentiality** = to keep the file content from being read by unauthorized users.
171 |
172 | In GNU/Linux, you're likely to interact with one of two hashing methods:
173 | 1. **SHA-512** (secure hash algorithm) is suitable for cryptographic purposes such as password hashing, digital signatures and TLS/SSL certificate generation.
174 | 2. **MD5** (message digest algorithm) can be used as a checksum to verify data integrity against unintentional corruption. Historically it was widely used as a cryptographic hash function, however it has been found to suffer from extensive vulnerabilities. It remains suitable for other non-cryptographic purposes. MD5 is still widely used for non-cryptographic tasks such as verifying the integrity of data files (e.g., comparing checksums for downloads), identifying duplicate files, as the hash values can quickly match identical content.
175 | - Non-cryptographic [XXH64](https://github.com/Cyan4973/xxHash) hash function is a modern alternative for MD5, and is designed for speed and efficiency.
176 |
177 | [^hashing]: [Taylor Hornby - How to Hash Passwords (The Right Way), accessed 2021](https://crackstation.net/hashing-security.htm)
178 |
179 | ### Salting
180 |
181 | > [!IMPORTANT]
182 | > Hackers can crack plain hashes very quickly using lookup tables and rainbow tables. Randomizing the hashing using salt is the solution to the problem. And thus hashes in typical GNU/Linux are salted. [^hashing]
183 |
184 | Salting is an extra step during hashing that adds an additional value to the end of the password, thereby changing the hash value produced. Salting can drastically reduce the chances of an attacker cracking passwords. The premise of salting is to protect and prevent precomputed hash attacks. Even though salts are stored in a database beside the hashes, salting adds another barrier for attackers, essentially complicating the password cracking process. This is useful when users have repeated passwords across multiple applications. The time taken for an attacker to execute a successful cracking is drastically increased with salted hashes. [^hashing]
185 |
186 | **Salting** randomizes each hash, so that when the same password is hashed twice, the hashes are not the same. We can randomize the hashes by appending or prepending a random string, called a salt, to the password before hashing. This makes the same password hash into a completely different string every time. To check if a password is correct, we need the salt, so it is usually stored in the user account database along with the hash, or as part of the hash string itself. The salt does not need to be secret. Just by randomizing the hashes, lookup tables, reverse lookup tables, and rainbow tables become ineffective. An attacker won't know in advance what the salt will be, so they can't pre-compute a lookup table or rainbow table. If each user's password is hashed with a different salt, the reverse lookup table attack won't work either. [^hashing]
187 |
188 | - The salt should be stored in the user account table alongside the hash. [^hashing]
189 | - The salt does not need to be secret. [^hashing]
190 | - The salt needs to be unique per-user per-password. Never reuse a salt. [^hashing]
191 | - A new random salt must be generated each time a user creates an account or changes their password. [^hashing]
192 | - Every time a user creates an account or changes their password, the password should be hashed using a new random salt. [^hashing]
193 | - The salt also needs to be long, so that there are many possible salts. [^hashing]
194 | - As a rule of thumb, make your salt is at least as long as the hash function's output. [^hashing]
195 | - Should the salt come before or after the password?
196 | - It doesn't matter, but pick one and stick with it for interoperability's sake. [^hashing]
197 | - Having the salt come before the password seems to be more common. [^hashing]
198 | - If you are writing a web application, you might wonder where to hash?
199 | - In a Web Application, always hash on the server. [^hashing]
200 | - The password should be sent to the server "in the clear" and hashed there, or you can hash twice: once on the client side and then rehash the hash on the server side. [^hashing]
201 | - If passwords are hashed only in the user's browser with JavaScript the client-side hash logically becomes the user's password. If a bad guy somehow steals the database of hashes from this hypothetical website, they'll have immediate access to everyone's accounts without having to guess any passwords. [^hashing]
202 | - One goal is to make the hash function slow enough to impede attacks, but still fast enough to not cause a noticeable delay for the user. Use a standard algorithm like PBKDF2 or bcrypt. These algorithms take a security factor or iteration count as an argument (this value determines how slow the hash function will be). [^hashing]
203 | - Compares hash strings in a way that takes the same amount of time no matter how much of the strings match (to protect against a timing attack). [^hashing]
204 |
205 | ### Howto hand edit /etc/shadow
206 |
207 | If you want to create a hash same way ` /etc/shadow ` stores it:
208 | 1. First check the first two characters of the second field for your user account in ` /etc/shadow `:
209 | - If the two characters are $1, your password is encrypted with MD5.
210 | - If the two characters are $5, your password is encrypted with SHA256.
211 | - If the two characters are $6, your password is encrypted with SHA512.
212 | 2. Then use commands like:
213 |
214 | ```
215 | $ openssl passwd -6 -salt PPfs6aO9wVajlGSb↵
216 | Password: my_poor_password↵
217 |
218 | $ mkpasswd --method=sha-512 --salt=PPfs6aO9wVajlGSb↵
219 | Password: my_poor_password↵
220 | ```
221 |
222 | > [!NOTE]
223 | > The ` $ mkpasswd ` utility is part of the ` whois ` package. So you will need to install that first.
224 |
225 | > [!WARNING]
226 | > If you modified the above examples by using redirection to give the password, then make sure you don't leave your password in plain text inside ` ~/.bash_history `
227 |
--------------------------------------------------------------------------------
/chapter/07-advanced-terminal.md:
--------------------------------------------------------------------------------
1 |
2 |
3 | > [!TIP]
4 | > The table of contents can be accessed by pressing the unordered list icon  on top the right corner.
5 |
6 |
11 |
12 |
13 |
14 | # Chapter 7 - Advanced terminal usage
15 |
16 | Some parts of shell syntax (notably its quoting and statement-syntax rules) can be very confusing. These drawbacks generally relate to compromises in the programming-language part of the shell's design made to preserve its utility as an interactive command language interpreter. [^raymond-sshheell] To see how difficult things can get, take a look at [Bash Pitfalls](https://mywiki.wooledge.org/BashPitfalls), a page dedicated to common mistakes made by bash users.
17 |
18 |
19 |
20 | ## 7.1 Multi tasking
21 |
22 | **Multitasking** is the ability of a system to run multiple programs or tasks (seemingly) at the same time. Because a microprocessor can work with much smaller units of time than people can perceive, concurrent processes appear to be occurring simultaneously. Concurrent processing built into Unix, was a significant advantage over, for example, MS-DOS, which never had multitasking.
23 |
24 |
29 |
30 | ### The concept of multiple windows
31 |
32 | The concept of windows on a computer desktop is a core feature of multitasking in modern operating systems. By allowing multiple windows to be open simultaneously, users can perform various tasks at the same time, such as writing an email in one window, browsing the web in another, and editing a document in yet another. This is one way of running programs concurrently.
33 |
34 | ### Virtual terminals
35 |
36 | The terms virtual console and terminal emulator refer to a unix-type way of dividing a physical console into several virtual consoles, each of which can run its own programs. This is one way of running programs concurrently.
37 |
38 | GNU/Linux still typically provides about 7 virtual TTY (teletype) terminals. You can use keyboard shortcuts ` Ctrl + Alt + F# ` to switch between the seven TTYs. TTYs can be seen as a basic form of multitasking. Each TTY provides an independent text-based interface where users can perform different tasks simultaneously. While they are not graphical windows (like those in a desktop environment), they still allow multiple sessions and processes to run independently. On one TTY, you could keep open an instruction manual. One TTY could be busy executing a long-running process. And there were still several free interfaces for other concurrent use. Even today, TTYs can be very useful if, for example, a graphical desktop environment becomes unresponsive.
39 |
40 | > [!WARNING]
41 | > One of the virtual TTY terminals manages the graphical desktop environment. And you can get back to it with the same keyboard shortcuts ` Ctrl + Alt + F# `, if you know or guess the right TTY number. You should check the TTY number of the graphical environment with ` $ who↵ ` before trying to switch to the textual TTY environment.
42 |
43 | ### Three ways to open a desktop application
44 |
45 | Three different ways to open a desktop app (such as ` $ gedit `) from a terminal window:
46 | 1. **Detached:** ` $ nohup gedit [optional-text-file] > /dev/null &↵ `
47 | - This allows a process to continue running even after the terminal session that started it is closed.
48 | 2. **Background:** ` $ gedit [optional-text-file] &↵ `
49 | - A process in the background is immune from terminal keyboard input, including any attempt to interrupt it with ` Ctrl + C `. You can return the process to the foreground by ` $ fg %1↵ `.
50 | 3. **Foreground:** ` $ gedit [optional-text-file]↵ `
51 | - The process can be suspended by pressing ` Ctrl + Z `.
52 | - ` $ jobs↵ ` can be used to check if the program has stopped.
53 | - ` $ fg %1↵ ` reinstates the program to foreground.
54 | - ` $ bg %1↵ ` restores execution to background.
55 |
56 | > [!IMPORTANT]
57 | > In your case **` %1 `** may be incorrect. So be sure to double check the correct job number with ` $ jobs↵ `.
58 |
59 | > [!TIP]
60 | > Moving a process from the foreground to the background is handy, if you launch a graphical program from the command line, but forget to place it in the background by appending the trailing ampersand **` ␣& `**.
61 |
62 |
63 |
64 | ## 7.2 When a program hangs up
65 |
66 | Sometimes a computer will become sluggish or an application will stop responding. In this chapter, we will look at some of the tools available at the command line that let us examine what programs are doing and how to terminate misbehaving processes. [^shotts-misbehave]
67 |
68 |
69 |
70 | ### Kill signals
71 |
72 | There is a **` $ kill `** command, which (despite the name) doesn't necessary kill processes, but rather sends signals. Programs, in turn, listen for signals and may act upon them as they are received. The fact that a program can listen and act upon signals allows a program to do things such as save work in progress when it is sent a termination signal. Signals are one of several ways that the operating system communicates with programs.
73 |
74 | #### If you want to terminate a process
75 |
76 | a) Keyboard shortcut **` Ctrl + C ` politely asks the current process to terminate**. An **INT** (interrupt) signal is sent to the target process. This corresponds to **` $ kill -2 ↵ `**. The target process can accept the termination request and clean itself up before leaving, or refuse to comply and not terminate at all. Many, but not all, command-line programs can be terminated using this technique.
77 |
78 | b) Command **` $ kill -3 ↵ ` politely asks the current process to terminate**. A signal **QUIT** is sent to the target process, and it shuts down if it is alive enough to receive the signal. The program may also intercept the signal and choose to ignore it.
79 |
80 | c) Command **` $ kill -15 ↵ ` politely asks the current process to terminate**. A signal **TERM** (terminate) is sent to the target process, and it shuts down if it is alive enough to receive the signal. The program may also intercept the signal and choose to ignore it.
81 |
82 | d) Command **` $ kill -9 ↵ ` immediately terminates the chosen process**. This signal is never sent to the target process like other signals. The **KILL** signal is sent to the operating system kernel, which terminates the process immediately. When a process is terminated this way, it is not given the opportunity to clean up or save work in progress. Thus, it may leave behind corrupted files. So this signal should only be used as a last resort when other termination signals fail (force close an unresponsive program).
83 |
84 | #### If you want to pause a process
85 |
86 | a) Keyboard shortcut **` Ctrl + Z ` politely asks the current process to suspend**. A **TSTP** (terminal stop) signal is sent to the target process. This corresponds to **` $ kill -20 ↵ `**. When you succesfully suspend a process, you can do fancy things with it such as bring it back to the foreground (see [Section: Multi tasking](#multi-tasking)). The program may also intercept the signal and choose to ignore it.
87 |
88 | b) Command **` $ kill -19 ↵ ` immediately suspends the chosen process**. This signal is never sent to the target process like other signals. The **STOP** signal is sent to the operating system kernel, which stops the process immediately. When a process is suspended this way, it is not given the opportunity to ignore the command.
89 |
90 | > [!TIP]
91 | > You can see all the signals using ` $ kill -l↵ ` in terminal.
92 |
93 | > [!NOTE]
94 | > With ` $ kill ` you must have superuser privileges to send signals to processes that do not belong to you.
95 |
96 | ### Process ID
97 |
98 | The kernel maintains information about each process to help keep things organized. Each process is assigned a **PID** number (short for Process ID). Like files, processes also have owners and user IDs. [^shotts-prowork]
99 |
100 | #### Graphical user interface to find the PID number
101 |
102 | Many distributions offer a *System Monitor* found some where in the *Desktop Menu* to find out the process ID numbers. This is the equivalent of *Task Manager* in Windows invoked by ` Ctrl + Alt + Del `.
103 |
104 | #### Commands to find the PID number
105 |
106 | The terminal will offer some alternatives to find the PID(s):
107 | - ` $ pidof firefox↵ ` will list PID(s) of a specific executable
108 | - ` $ ps↵ ` will list processes associated with the current terminal session.
109 | - ` $ ps x↵ ` will list processes regardless of what terminal (if any) they are controlled by.
110 | - ` $ ps aux↵ ` will show processes belonging to every user.
111 |
112 | The fields are:
113 | - **USER** = Owner of the process
114 | - **TIME** = Amount of CPU time consumed by the process.
115 | - **TTY** (Teletype) = Controlling terminal for the process.
116 | - ` ? ` Means there is no controlling terminal.
117 | - **STAT** (Process State Codes) = Current status of the process
118 | - ` R ` Running
119 | - ` S ` Sleeping
120 | - ` T ` Stopped
121 | - ` I ` Idle
122 | - ` l ` Multithreaded
123 | - ` < ` High-priority
124 | - ` N ` Nice = Low-priority
125 | - ` s ` Parent process of a terminal session
126 | - ` + ` Foreground
127 |
128 | ### Process priority
129 |
130 | Linux can run a lot of processes at a time, which can slow down the speed of some high priority processes and result in poor performance. To avoid this, you can tell your machine to prioritize processes as per your requirements. This priority is called **niceness** in Linux, and it has a value between -20 to 19. The lower the niceness value, the higher the priority. The default value of all the processes is 0.
131 | - To start a process with a niceness value other than the default value use: ` $ nice -n process name↵ `
132 | - If there is some process already running on the system, then you can re-nice: ` $ renice -p ↵ `
133 |
134 |
135 |
136 | ## 7.3 Expansions and quoting
137 |
138 | ### Word expansions
139 |
140 | **Expansions** are mechanisms that allow the shell to interpret and transform input text before executing commands. They make scripts and commands more powerful and flexible. [^tog-expansions]
141 |
142 | [^tog-expansions]: [Open Group Base Specifications - Word Expansions, accessed 2025](https://pubs.opengroup.org/onlinepubs/9799919799/utilities/V3_chap02.html#tag_19_06)
143 |
144 | - [Brace expansion](https://pubs.opengroup.org/onlinepubs/9799919799/xrat/V4_xcu_chap01.html#tag_23_02_06) generates a sequence of strings.
145 | - [Tilde expansion](https://pubs.opengroup.org/onlinepubs/9799919799/utilities/V3_chap02.html#tag_19_06_01) expands ` ~ ` to the home directory of the current user or ` ~user ` to another user.
146 | - [Arithmetic expansion](https://pubs.opengroup.org/onlinepubs/9799919799/utilities/V3_chap02.html#tag_19_06_04) evaluates arithmetic expressions.
147 | - [Parameter expansion](https://pubs.opengroup.org/onlinepubs/9799919799/utilities/V3_chap02.html#tag_19_06_02) expands variables and allows modifications.
148 | - [Command substitution](https://pubs.opengroup.org/onlinepubs/9799919799/utilities/V3_chap02.html#tag_19_06_03) substitutes the output of a command.
149 | - [Pathname expansion](https://pubs.opengroup.org/onlinepubs/9799919799/utilities/V3_chap02.html#tag_19_06_06) expands wildcards (globbing) to match file or directory names.
150 |
151 | > [!NOTE]
152 | > Shells are not required to implement brace expansion. This form of expansion is allowed but not required by IEEE 1003.1 standard. But if brace expansion is supported, it must be performed before all of the standard word expansions. [^tog-expansions]
153 |
154 | ### Quoting
155 |
156 | Quoting is used to remove the special meaning of certain characters or words to the shell. Quoting can be used to preserve the literal meaning of the special characters in the next paragraph, prevent reserved words from being recognized as such, and prevent parameter expansion and command substitution. [^tog-quoting]
157 | - Enclosing characters in [single-quotes](https://pubs.opengroup.org/onlinepubs/9799919799/utilities/V3_chap02.html#tag_19_02_02) ` '...' ` shall preserve the literal value of each character within the quotes and suppress all expansions.
158 | - Enclosing characters in [double-quotes](https://pubs.opengroup.org/onlinepubs/9799919799/utilities/V3_chap02.html#tag_19_02_03) ` "..." ` shall preserve the literal value of all characters within the quotes, with the exception of:
159 | - Dollar-sign ` $??? ` for parameter expansion
160 | - Dollar-sign ` ${...} ` for the other form of parameter expansion
161 | - Dollar-sign ` $((...)) ` for arithmetic expansion
162 | - Dollar-sign ` $(...) ` for command substitution
163 | - Backquotes `` `...` `` for the other form of command substitution
164 | - Backslash ` \? ` as escape character
165 |
166 | [^tog-quoting]: [Open Group Base Specifications - Quoting, accessed 2025](https://pubs.opengroup.org/onlinepubs/9799919799/utilities/V3_chap02.html#tag_19_02)
167 |
168 | ### Suppressing expansions
169 |
170 | The bash shell can perform expansions and substitutions (as show by examples below), which leads to the need to control it with quoting.
171 | - Double quotes ` "..." ` suppress some expansions.
172 | - Single quotes ` '...' ` suppress all expansions.
173 |
174 | | | Un quoted | "Double" quoted | 'Single' quoted |
175 | |:--- |:--- |:--- |:--- |
176 | | **String:** ` text ` | ` text ` | ` text ` | ` text ` |
177 | | **Brace expansion:** ` {a,c} ` ` {a..c} ` ` {001..003} ` |
` $(uname ‑o) ` `` `uname ‑o` `` |
182 |
183 | > [!TIP]
184 | > You can use the *display text* commands ` $ echo ` and ` $ printf ` to learn see the world as the shell sees it:
185 | >
186 | > ```
187 | > $ echo {a..c}↵
188 | > a b c
189 | >
190 | > $ printf "%s\n" {a..c}↵
191 | > a
192 | > b
193 | > c
194 | > ```
195 |
196 |
197 |
198 | ## 7.5 Square brackets: [...]
199 |
200 | There is a special [GNU test](https://www.gnu.org/software/coreutils/manual/coreutils.html#test-invocation) command to perform a variety of checks and comparisons. It has two equivalent forms: ` $ test ↵ ` and ` $ [ ]↵ `. It is interesting to note that both ` $ test ` and ` $ [ ` are actual filenames. There is actually an executable named ` [ ` in the ` /usr/bin/ ` directory. The expression is actually just the arguments for the ` $ [ ` command. That's why a space character is required after. Likevise the ` ] ` is simply a required final argument. Also that's why a space character is required before the ` ] ` argument.
201 |
202 | The commands return an exit status of 0 if the expression is true, 1 if the expression is false, or 2 if an error occurred. Nothing gets printed on the screen though. The shell provides a parameter ` $? ` that we can use to examine the exit status. Or one can construct some form of if-then-clause.
203 |
204 | ```
205 | $ [ $USER == margaret ] && echo true || echo false↵
206 | $ test $USER == margaret && echo true || echo false↵
207 | true
208 |
209 | $ [ $USER == edvard ] && echo true || echo false↵
210 | $ test $USER == edvard && echo true || echo false↵
211 | false
212 |
213 | $ [ $USER == margaret ]↵
214 | $ test $USER == margaret↵
215 | $ echo $?↵
216 | 0
217 |
218 | $ [ $USER == edvard ]↵
219 | $ test $USER == edvard↵
220 | $ echo $?↵
221 | 1
222 | ```
223 |
224 |
225 |
226 | ## 7.6 Customizing the shell prompt
227 |
228 | When used interactively, the shell prompts with the value of ` PS1 ` before reading a command. If at any time a newline ` \↵ ` is typed and further input is needed to complete a command then the secondary prompt ` PS2 ` is issued. It is possible to customize these default shell prompts.
229 |
230 | Let's examine the default ` PS1 ` prompt in Solus ` $ echo $PS1↵ `:
231 |
232 | | Code | Function |
233 | |:--- |:--- |
234 | | \\\[ \\033[**38;5;081**m \\\] | Will change color to aqua |
235 | | \\u | Current username |
236 | | \\\[ \\033[**38;5;245**m \\\] | Will change color to grey |
237 | | @ | Simply the at sign |
238 | | \\\[ \\033[**38;5;206**m \\\] | Will change color to fuchsia |
239 | | \\H | Current hostname |
240 | | \\\[ \\033[**38;5;245**m \\\] | Will change color to grey |
241 | | \\w | Current working directory |
242 | | \\\[ \\033[**38;5;081**m \\\] | Will change color to aqua |
243 | | $ | Simply the dollar sign |
244 | | \\\[ **\\e[0m** \\\] | Removes all attributes (formatting and colors). |
245 |
246 | > [!NOTE]
247 | > Note that **` \[ `** begins a sequence of non-printing characters (like color escape sequences) and **` \] `** ends a sequence of non-printing characters.
248 |
249 | ### Modifying the default prompt
250 |
251 | Lets replace the default PS1 with something different:
252 |
253 | ```
254 | $ export PS1='[\D{%d/%m/%Y %H:%M:%S} \u@\h \W]§'↵
255 | [16/06/2024 18:34:38 margaret@mamas-machine ~]§
256 |
257 | # The PS2 on Solus could use some colour and a space at the end:
258 | $ export PS2='\[\033[38;5;081m\]> \[\e[0m\]'↵
259 | ```
260 |
261 | > [!NOTE]
262 | > Any change to PS1 or PS2 will vanish as the shell session ends. To make the change permanent, add the export command to a new line on ` ~/.bashrc ` for user specific preference, or ` /etc/profile ` for system wide preference.
263 |
264 | ### Special prompt variables
265 |
266 | - ` \$ ` = The default bash prompt sign.
267 | - Dollar sign $ for regular access.
268 | - Hash mark # for continuous root access.
269 | - ` \u ` = The username of the current user
270 | - ` \H ` = The full hostname (such as deckard.SS64.com) [^ss64]
271 | - ` \h ` = The hostname, up to the first dot (such as deckard) [^ss64]
272 | - ` \w ` = Current directory (full path)
273 | - ` \W ` = Current directory (without the parent directories)
274 | - ` \l ` = The basename of the shell's terminal device name
275 | - The time
276 | - ` \t ` = in 24-hour HH\:MM\:SS format
277 | - ` \T ` = in 12-hour HH\:MM\:SS format
278 | - ` \@ ` = in 12-hour AM\:PM format
279 | - ` \n ` = Newline
280 | - ` \r ` = Carriage return
281 | - ` \a ` = Bell character
282 | - ` \\ ` = Backslash
283 | - ` \nnn ` = Character with corresponding ASCII code (nnn = octal value)
284 | - ` \! ` = History number of this command
285 | - ` \# ` = Command number of this command
286 | - ` \D{...} `
287 | - ` %a ` = (sat) = Day (abbreviated name)
288 | - ` %A ` = (saturday) = Day (full name)
289 | - ` %b ` = (feb) = Month (abbreviated name)
290 | - ` %B ` = (february) = Month (full name)
291 | - ` %d.%m.%Y ` = (20.02.2021) = Day, Month, Year
292 | - ` %H:%M:%S ` = (10\:15\:59) = Hours, Minutes, Seconds
293 | - E.g. ` \D{%H:%M %A %d.%m.%Y} ` = 10\:15 lauantai 20.02.2021
294 |
295 | [^ss64]: [SS64 - How to setup prompt statement variables, accessed 2021](https://ss64.com/bash/syntax-prompt.html)
296 |
297 | For more bells and whistles visit:
298 | -
299 | -
300 |
301 | ### 8-bit (256) colors
302 |
303 | Most terminal emulators understand the 8-bit color palette: [^wiki-ansi]
304 |
305 | 
306 |
307 | [^wiki-ansi]: [Wikipedia - ANSI escape code, accessed 2024](https://en.wikipedia.org/wiki/ANSI_escape_code)
308 |
309 | The numbering in 8-bit color palette is arranged so that: [^wiki-ansi]
310 | - Range ` 0x00-0x07 ` is for standard 8 colors.
311 | - Range ` 0x08-0x0F ` is for additional 8 high intensity colors.
312 | - Range ` 0xE8-0xFF ` is for grayscale from black to white in 24 steps.
313 | - Range ` 0x10-0xE7 ` is for rest of the 216 colors.
314 |
315 | #### Examples of color use
316 |
317 | Using these above, you can make pink text like so:
318 |
319 | ```
320 | \033[38;5;206m #That is, \033[38;5;m
321 | ```
322 |
323 | And an early-morning blueish background like so:
324 |
325 | ```
326 | \033[48;5;57m #That is, \033[48;5;m
327 | ```
328 |
329 | The above foreground and background be combined like so:
330 |
331 | ```
332 | \033[38;5;206;48;5;57m #That is, \033[38;5;;48;5;m
333 | ```
334 |
335 |
336 |
337 | [^shotts-misbehave]: [The Linux Command Line 2024, Chapter: 10, Section: Processes](http://linuxcommand.org/tlcl.php)
338 |
339 | [^shotts-prowork]: [The Linux Command Line 2024, Chapter: 10, Section: How a Process Works](http://linuxcommand.org/tlcl.php)
340 |
341 | [^raymond-sshheell]: [The Art of Unix Programming 2003, Chapter 14, Section: Shell](http://www.catb.org/~esr/writings/taoup/html/ch14s04.html)
342 |
343 |
--------------------------------------------------------------------------------
/chapter/06-inter.md:
--------------------------------------------------------------------------------
1 |
2 |
3 | > [!TIP]
4 | > The table of contents can be accessed by pressing the unordered list icon  on top the right corner.
5 |
6 |
11 |
12 |
13 |
14 | # Chapter 6 - Interoperability
15 |
16 | **Interoperability** or **interprocess communication (IPC)** is the ability of software processes to talk to each other and work together with little effort from the user. Such modularity and extensibility is one of the oldest and most persistent design tropes of unix and its decendants.
17 |
18 | > [!IMPORTANT]
19 | > The power of a system comes more from the relationships among programs, than from the programs themselves.
20 |
21 |
22 |
23 | ## 6.1 Simple IPC Techniques
24 |
25 | Unix has always provided several ways for processes to communicate relatively easy:
26 |
27 | ### Numeric exit status
28 |
29 | The simplest of all design patterns is having no input and no output; just a **numeric exit status** (see [Section: Exit status and comparison](#exit-status-and-comparison)). Good examples for use of such mechanic are commands ` $ test `, ` $ rm ` and ` $ touch `
30 | - ` $ test -r ↵ ` = Does the file exist and is it readable?
31 | - ` $ test -w ↵ ` = Does the file exist and is it writable?
32 | - ` $ test -u ↵ ` = Does the file exist and is the setuid bit set?
33 | - ` $ test -g ↵ ` = Does the file exist and the setgid bit is set?
34 | - ` $ test -k ↵ ` = Does the file exist and the sticky bit is set?
35 |
36 |
39 |
40 | > [!NOTE]
41 | > The test command returns an exit status of 0 if the expression is true, 1 if the expression is false, or 2 if an error occurred.
42 | >
43 | > Nothing gets printed on the screen though. The shell provides a parameter ` $? ` that we can use to examine the exit status. Or one can construct some form of if-then-clause. See [Chapter 7, Section: Square brackets](07-advanced-terminal.md#square-brackets).
44 |
45 | ### Text through standard streams
46 |
47 | Simple **textual information** can be passed through **standard streams**. They allow the combined use of commands by first executing one command and then passing the output to the next command as input. See [Section: Standard streams](#standard-streams).
48 |
49 | ### Handing off tasks to specialist programs
50 |
51 | Handing over a task to another program, such as a decision to leave out a text editor component in an email client program, and replace the functionality of writing a message by launching a separate general-purpose text editor for this task. The key point is that the specialist program does not handshake with the parent while they are running. The higher-level process temporarily relinquishes control to the subprocess to perform the operation, and only returns to resume execution when the subprogram closes. Calling an external program from inside a script, program or another shell in this manner is often called **shelling out** to the called program. [^raymond-shellout]
52 |
53 |
56 |
57 | ### Use of tempfiles
58 |
59 | The use of **tempfiles** as communications drops between cooperating programs is one of the oldest IPC technique there is. Despite drawbacks, tempfiles still have a niche over more elaborate methods. And sometimes, nothing else will do; The calling conventions of your child process may require that it be handed a file to operate on. [^raymond-tempfiles]
60 |
61 | ### POSIX-signals
62 |
63 | One way for two processes on the same machine to communicate with each other is for one to send the other a signal. Signals were originally designed into unix as a way for the operating system to notify programs of certain errors and critical events (see [Chapter 7, Section: Kill signals](07-advanced-terminal.md#kill-signals)). Nevertheless, signals can be useful for some IPC situations, and the POSIX-standard signal set includes two signals intended for this use: SIGUSR1 and SIGUSR2. [^raymond-signl]
64 |
65 | A technique often used with signal IPC is the so-called pidfile. Programs that will need to be signaled will write a small file to a known location (often in ` /var/run/ ` or the invoking user's home directory) containing their process ID (PID). Other programs can read that file to discover that PID. [^raymond-signl]
66 |
67 |
68 |
69 | ## 6.2 Classic shell IO
70 |
71 | ### Standard streams
72 |
73 | Traditional ability of unix programs to communicate and exchange information relies on passing through simple textual information via **standard streams**. Every program has initially available to it (at least) two I/O data streams called standard **input** and standard **output**. They allow the combined use of commands by first executing one command and then passing the output to the next command as input. There is also a separate channel for exceptions called the standard **error**.
74 |
75 | ### Interface patterns based on standard streams
76 |
77 | - **Source:** [^raymond-intpatterns]
78 | - Requires no input. It's output is controlled only by startup conditions.
79 | - A classic example would be ` $ who `, ` $ ps ` and ` $ ls `.
80 | - **Filter:** [^raymond-intpatterns]
81 | - Takes data on standard input, transforms it in some fashion, and sends the result to standard output.
82 | - A classic example would be ` $ grep `, which prints only those lines that match the pattern given.
83 | - **Sink:** [^raymond-intpatterns]
84 | - Is a filter-like program that consumes standard input, but emits nothing to standard output.
85 | - **Cantrip:** [^raymond-intpatterns]
86 | - Outputs only a numeric exit status (see [Section: Exit status and comparison](#exit-status-and-comparison)).
87 | - A classic example would be ` $ test `, ` $ rm ` and ` $ touch `.
88 |
89 | > [!IMPORTANT]
90 | > All these patterns have very low interactivity. Programs don't get any more scriptable than this. In addition there are also interactive design patterns and many browser like and editor like programs, but they are not quite so scriptable (as in generating sequences of commands). [^raymond-intpatterns]
91 |
92 | > [!NOTE]
93 | > Some commands such as ` $ echo ` do not read their standard input.
94 |
95 | ### Redirection of standard streams
96 |
97 | Programs such as ` $ ls ` actually send their results to a special file called standard output (often expressed as stdout) and their status messages to another special file called standard error (stderr). By default, both standard output and standard error are linked to the screen and not saved into a regular file. I/O redirection allows us to change where that output goes.
98 |
99 |
100 |
101 | #### File descriptor
102 |
103 | Standard streams are associated with an a **file descriptor** (FD). It is an identifier used by a processes to refer to an I/O resource such as a terminal window, pipe, or text file.
104 | - If the program output is sent to the printer's file descriptor, it is printed on paper (such as ` > /dev/usb/lp0 `).
105 | - If the program output is sent to a file descriptor of a terminal window, the program output is displayed on the terminal window (such as ` > /dev/pts/1 `).
106 | - You could also redirect output from one terminal window to another such as ` > /dev/pts/2 `.
107 |
108 | The shell file descriptors are:
109 | - **FD 0** = ` /dev/stdin ` = Standard Input
110 | - **FD 1** = ` /dev/stdout ` = Standard Output
111 | - **FD 2** = ` /dev/stderr ` = Standard Error
112 |
113 | > [!TIP]
114 | > You can verify file descriptors with the following command:
115 | >
116 | > ```
117 | > $ ls -l /proc/self/fd↵
118 | > total 0
119 | > lrwx------ 1 margaret 64 10. 5. 10:51 0 -> /dev/pts/1
120 | > lrwx------ 1 margaret 64 10. 5. 10:51 1 -> /dev/pts/1
121 | > lrwx------ 1 margaret 64 10. 5. 10:51 2 -> /dev/pts/1
122 | > lr-x------ 1 margaret 64 10. 5. 10:51 3 -> /proc/8468/fd
123 | > # /dev/pts stands for pseudo terminal slave
124 | > ```
125 |
126 |
127 |
128 | #### Redirection to a file
129 |
130 | To redirect standard output to a regular file instead of the screen, we use the redirection operator ` > ` or ` >> `, followed by the target filepath.
131 | - The redirection operator **` > `** connects the command output with a text file, **overwriting** any existing content. If the file does not exist, it will be created.
132 | - The redirection operator **` >> `** connects the command output with a text file, **appending** and thus preserving any existing file content. If the file does not exist, it will be created.
133 |
134 | Sometimes we don't want output from a command, we just want to throw it away. This applies particularly to error messages and status messages. Redirection to target file ` > /dev/null ` will suppress and discard all output messages. This special system file is often referred to as the bit bucket. It accepts input, and does nothing with it.
135 |
136 | | Operator | Redirection |
137 | |:--- |:--- |
138 | | **` 1> `** or **` > `** | Redirect stdout to file by overwriting. |
139 | | **` 1>> `** or **` >> `** | Redirect stdout to file by appending. |
140 | | **` 2> `** | Redirect stderr to file by overwriting. |
141 | | **` 2>> `** | Redirect stderr to file by appending. |
142 | | **` &> `** | Redirect both stdout and stderr to file by overwriting. |
143 | | **` &>> `** | Redirect both stdout and stderr to file by appending. |
144 |
145 | #### Input redirection and substitution
146 |
147 | By using the **input redirection** symbol **` < `**, you can feed input data from a file, instead of typing it:
148 |
149 | ```
150 | $ echo ~/Documents/ > store-argument.txt↵
151 | $ ls -la < store-argument.txt↵ # Equals: ls -la ~/Documents/
152 | ```
153 |
154 | By using the **process substitution** operator **` <(commands) `**, you can have one or more commands inplace of typed in arguments or references to text files:
155 |
156 | ```
157 | $ diff -y <(ls -la ~/MyProject/) <(ls -la ~/MyBackup/)↵
158 | ```
159 |
160 | #### Pipelines
161 |
162 | The vertical bar **pipe operator** **` | `** redirects the standard output of one command into the standard input of another. Piping is analogous to water pipes.
163 |
164 | Let's approach the topic with an example: ` $ ls -l /usr/bin | less↵ `, where the output of the list command ` $ ls ` is passed to a browser called ` $ less `. Now try up and down arrow keys to scroll through the content. Finally press press ` q ` to quit ` $ less `.
165 |
166 | > [!NOTE]
167 | > Programs ` $ more ` and ` $ less ` are filters for paging text one screenful at a time. More is older and primitive. Less is the enhanced version of more. The naming is a pun: *less is more*.
168 |
169 | This second example ` $ ls -l /usr/bin | tee list.txt | less↵ ` uses ` $ tee ` to capture the data between pipes. The tee command (from capital letter T) is named after plumbing terminology for a T-shaped pipe splitter. This splits or copies the standard input, i.e. the output of the previous command, to both 1) one or more files and 2) the standard output for use by the next command as input, allowing the data to continue down the pipeline.
170 |
171 | #### Xargs can replace standard input
172 |
173 |
176 |
177 | Some programs such as ` $ echo ` and ` $ stat ` in ` --file-system ` mode do not read their standard input. Many programs accept the bare hyphen **` - `** as a pseudo-filename that directs to read the standard output. But not ` $ stat ` in ` --file-system ` mode. [^ieee-echo]
178 |
179 | [^ieee-echo]: [IEEE 1003.1 Standard: Utilities: echo, accessed 2025](https://pubs.opengroup.org/onlinepubs/9799919799/utilities/echo.html#tag_20_37_06)
180 |
181 | ```
182 | # This attempt will not work as intended:
183 | $ which ls & stat --file-system -↵
184 | standard input does not work in file system mode
185 |
186 | # This attempt will not work as intended:
187 | $ stat --file-system <(which ls)↵
188 | File: "/dev/fd/63"
189 | ID: e00000000 Namelen: 255 Type: pipefs
190 | Block size: 4096 Fundamental block size: 4096
191 | Blocks: Total: 0 Free: 0 Available: 0
192 | Inodes: Total: 0 Free: 0
193 | ```
194 |
195 | If you need to pipe in a command that does not accept standard input you may use ` $ xargs `. It converts standard input into an argument list for another command.
196 |
197 | ```
198 | # These alternatives achieve the same goal:
199 | $ which ls | xargs stat -f↵ # Equals: $ stat -f /usr/bin/ls
200 | $ which ls | stat -f `xargs`↵ # Equals: $ stat -f /usr/bin/ls
201 | $ which ls | stat -f $(xargs)↵ # Equals: $ stat -f /usr/bin/ls
202 | File: "/usr/bin/ls"
203 | ID: 62851ebc76790d48 Namelen: 255 Type: ext2/ext3
204 | Block size: 4096 Fundamental block size: 4096
205 | Blocks: Total: 24159833 Free: 18766614 Available: 17527818
206 | Inodes: Total: 6176768 Free: 5672251
207 | ```
208 |
209 | #### Xargs can divide series
210 |
211 | One can use the ` -n # ` argument to delimit standard input by space, to make the specified command execute multiple times with any initial arguments followed by items read from standard input:
212 |
213 | ```
214 | $ echo {2007..2009}-{01..12}↵
215 | 2007-01 2007-02 2007-03 2007-04 2007-05 2007-06 2007-07 2007-08 2007-09 2007-10 2007-11 2007-12 2008-01 2008-02 2008-03 2008-04 2008-05 2008-06 2008-07 2008-08 2008-09 2008-10 2008-11 2008-12 2009-01 2009-02 2009-03 2009-04 2009-05 2009-06 2009-07 2009-08 2009-09 2009-10 2009-11 2009-12
216 |
217 | $ echo {2007..2009}-{01..12} | xargs -n 12 echo "Row:"↵
218 | Row: 2007-01 2007-02 2007-03 2007-04 2007-05 2007-06 2007-07 2007-08 2007-09 2007-10 2007-11 2007-12
219 | Row: 2008-01 2008-02 2008-03 2008-04 2008-05 2008-06 2008-07 2008-08 2008-09 2008-10 2008-11 2008-12
220 | Row: 2009-01 2009-02 2009-03 2009-04 2009-05 2009-06 2009-07 2009-08 2009-09 2009-10 2009-11 2009-12
221 |
222 | $ echo {2007..2009}-{01..12} | xargs -n 1 echo "Row:"↵
223 | Row: 2007-01
224 | Row: 2007-02
225 | Row: 2007-03
226 | Row: 2007-04
227 | Row: 2007-05
228 | Row: 2007-06
229 | Row: 2007-07
230 | ...
231 | ```
232 |
233 | #### Xargs is able to ask for confirmation
234 |
235 | The ` -p ` option comes in handy for learning as it prompts before execution:
236 |
237 | ```
238 | $ ls | xargs -p -n 1 stat -f↵
239 | stat -f Desktop?...no↵
240 | stat -f dfgdf?...no↵
241 | stat -f Documents?...yes↵
242 | File: "Documents"
243 | ID: 62851ebc76790d48 Namelen: 255 Type: ext2/ext3
244 | Block size: 4096 Fundamental block size: 4096
245 | Blocks: Total: 24159833 Free: 18765868 Available: 17527072
246 | Inodes: Total: 6176768 Free: 5672206
247 | ...
248 | ```
249 |
250 | ### Control operators
251 |
252 | #### Sequential execution
253 |
254 | **Semicolon ` ; `** causes sequential execution of commands, with each one running even if the previous command failed.
255 |
256 | ```
257 | # To demonstrate, the following command will cause an alert bell sound multiple times with a one second interval:
258 | $ echo -e "\a" ; sleep 1 ; echo -e "\a" ; sleep 1 ; echo -e "\a" ; sleep 1 ; echo -e "\a"↵
259 | ```
260 |
261 | **Double ampersand ` && `** causes sequential execution of commands, with each one running only if the previous command did not fail. This can be especially handy for when you have a couple of commands to run, but you do not want the second to be run if the first fails.
262 |
263 | ```
264 | $ echo Aa && echo Bb > /root/Bb.txt && echo Cc↵
265 | Aa
266 | bash: /root/Bb.txt: Permission denied
267 |
268 | $ echo Aa ; echo Bb > /root/Bb.txt ; echo Cc↵
269 | Aa
270 | bash: /root/Bb.txt: Permission denied
271 | Cc
272 | ```
273 |
274 | **Double vertical bars ` || `** causes sequential execution of commands, with the last one running only if the first command failed. This type of construct is useful for handling errors in scripts.
275 |
276 | ```
277 | $ test -e ~/Desktop && echo "Path found" || echo "Path not found"↵
278 | Path found
279 |
280 | $ test -e ~/Desktip && echo "Path found" || echo "Path not found"↵
281 | Path not found
282 | ```
283 |
284 | > [!NOTE]
285 | > A fail is interpreted as returning a non-zero exit status (see [Section: Exit status and comparison](#exit-status-and-comparison)).
286 |
287 | #### Asynchronous execution
288 |
289 | **Trailing ampersand ` & `** directs the shell to run the command(s) in the background, in a separate sub-shell, as a job, asynchronously. This way you can continue using the shell, and not have to wait for the command(s) to terminate such as:
290 |
291 | ```
292 | $ sleep 4 & sleep 2 &↵
293 | [1] 12712
294 | [2] 12713
295 | $ _ # You can continue typing here straight away
296 | ```
297 |
298 | Usually, if you run a text editor (such as gedit) from the terminal, you will be unable to continue using the terminal, until you close the text editor. But, by appending the ampersand operator, you can make it run in the background and continue to use the shell immediately:
299 |
300 | ```
301 | $ gedit .bashrc &↵
302 | [1] 12872
303 | $ _ # You can continue typing here straight away
304 | ```
305 |
306 | > [!NOTE]
307 | > After entering the command, the shell prompt returns, but some funny numbers are printed too. This message is part of the shell and kernel maintaining information about each process to help keep things organized. Each process is assigned a process identity (PID). Type ` $ ps↵ ` to show processes associated with the current terminal session.
308 |
309 | ### GNU Grep
310 |
311 | [GNU Grep](https://www.gnu.org/software/grep/) is a command-line tool `$ grep ` that filters standard input (or text file contents) with a regular expression. The expression can be a simple string or a very complex pattern.
312 |
313 | > [!NOTE]
314 | > GREP stands for global regular expression print. **Regular expression** (also referred to as regex or regexp) is a notation for creating search, extraction, match and substitution patterns for text. Regex functionality is supported in many programming languages, but also present in some software applications and commandline utilities; especially with unix text-processing utilities.
315 |
316 | GNU Grep supports three regular expression syntaxes: basic, extended and Perl-compatible. Especially when writing multi-system expressions, care must be taken as to which constructs are newer extensions. To interpret an expression as an extended regular expression, use the ` -E ` or ` -P ` command-line options. When writing expressions, care must also be taken as to whether the line break is treated as a regular character, only whether the input is treated line by line like GNU Grep.
317 |
318 | The following example filters the home directory listing to files and directories starting with **` ␣D `** or **` ␣P `**:
319 |
320 | ```
321 | $ grep -E '( D| P)' <(ls -la ~)↵
322 | drwxr-xr-x 3 user user 4096 14. 6. 13:25 Desktop
323 | drwxr-xr-x 2 user user 4096 10. 6. 19:51 Documents
324 | drwxr-xr-x 5 user user 4096 10. 6. 19:25 Downloads
325 | drwxr-xr-x 2 user user 4096 10. 6. 17:16 Pictures
326 | drwxr-xr-x 2 user user 4096 12.11. 2023 Public
327 | ```
328 |
329 | The regex syntax is beyond the scope of this tutorial. But those interested to learn more can start with the official perl documentation:
330 | - [Metacharacters](https://perldoc.perl.org/perlre#Metacharacters)
331 | - [Character classes and other special escapes](https://perldoc.perl.org/perlre#Character-Classes-and-other-Special-Escapes)
332 |
333 | > [!TIP]
334 | > Regular expressions make it possible to concisely express complicated matching requirements. But they can be hard to read and write. Web browser based regex-tools (such as and ) are useful for a wide range of users, from beginners to advanced. They simplify the creation, testing and debugging of regex patterns.
335 |
336 |
337 |
338 | ### Exit status and comparison
339 |
340 | Commands (including the scripts and shell functions we write) issue a value to the system when they terminate, called an **exit status**. This value, which is an integer in the range of 0 to 255, indicates the success or failure of the command’s execution. [^shotts-ext]
341 | - By convention, a value of zero always indicates success and any other value indicates failure.
342 | - Many programs simply exit with a value of 1 upon failure. But some programs (such as RAR for Linux below) use different exit status values to provide diagnostics for errors. Man-pages often include a section entitled *EXIT STATUS* describing what codes are used[^shotts-ext].
343 |
344 | | Code | Description |
345 | | ---:|:--- |
346 | | 0 | Successful operation. |
347 | | 1 | Warning. Non fatal error(s) occurred. |
348 | | 2 | A fatal error occurred. |
349 | | 3 | Invalid checksum. Data is damaged. |
350 | | 4 | Attempt to modify an archive locked by 'k' command. |
351 | | 5 | Write error. |
352 | | 6 | File open error. |
353 | | 7 | Wrong command line option. |
354 | | 8 | Not enough memory. |
355 | | 9 | File create error. |
356 | | 10 | No files matching the specified mask and options were found. |
357 | | 11 | Wrong password. |
358 | | 12 | Read error. |
359 | | 255 | User stopped the process. |
360 |
361 | > [!NOTE]
362 | > Exit status is not printed to standard output and thus nothing gets printed on the screen. The shell provides a parameter ` $? ` that we can use to examine the exit status. Or one can construct some form of if-then-clause. See [Chapter 7, Section: Square brackets](07-advanced-terminal.md#square-brackets).
363 |
364 | #### Commands: true, false, exit, return
365 |
366 | The shell provides two extremely simple builtin commands that do nothing except terminate with either exit status 0 or 1:
367 | 1. The **` $ true↵ `** command always executes successfully: ` $? ` = 0.
368 | 2. The **` $ false↵ `** command always executes unsuccessfully: ` $? ` = 1
369 |
370 | There is also an **` exit # `** command which accepts a single, optional argument, which becomes the script’s exit status. When no argument is passed, the exit status defaults to the exit status of the last command executed. Using exit in this way allows a script to indicate failure. The exit command appearing on the last line of the script is a formality though, because when a script runs off the end (i.e. reaches end of file), it terminates with an exit status of the last command executed.
371 |
372 | Similarly internal functions in shell scripts can return an exit value by including an integer as a parameter to the command **` return # `**, which points to the return value of the variable ` $? ` and terminates the function (see [Linuxize.com - Bash Functions: Return Values](https://linuxize.com/post/bash-functions/#return-values)). [^bash-manual-funk]
373 |
374 | [^bash-manual-funk]: [GNU Bash Manual - Shell Functions, accessed 2025](https://www.gnu.org/software/bash/manual/html_node/Shell-Functions.html)
375 |
376 | > The unix API doesn't use exceptions. Even the C language lacks a facility for throwing named exceptions with attached data. Thus, the C functions in the unix API indicate errors by returning a distinguished value (usually −1 or a NULL character pointer) and setting a global errno variable. In retrospect, this is the source of many subtle errors. Programmers in a hurry often neglect to check return values.
377 |
378 |
379 |
380 | ## 6.3 Sockets
381 |
382 | All modern operating systems implement a POSIX or IEEE 1003.1 compliant version of the Berkeley socket interface for sending and receiving data between endpoints. The standard originated with BSD Unix in 1983. It became the standard interface for applications running on the Internet. Even the Winsock implementation for Microsoft Windows, created by unaffiliated developers, closely follows this standard (see [IEEE 1003.1 Standard: Sockets](https://pubs.opengroup.org/onlinepubs/9799919799/functions/V2_chap02.html#tag_16_10)). [^wiki-bsd-sockets]
383 | The term socket is also used for the software endpoint of interprocess communication, which often uses the same API as a network socket. [^wiki-network-socket]
384 |
385 | [^wiki-bsd-sockets]: [Wikipedia - Berkeley sockets, accessed 2024](https://en.wikipedia.org/wiki/Berkeley_sockets)
386 |
387 | [^wiki-network-socket]: [Wikipedia - Network socket, accessed 2024](https://en.wikipedia.org/wiki/Network_socket)
388 |
389 | POSIX sockets are usually the right thing to use for bidirectional IPC no matter where your cooperating processes are located. Sockets are supported on all modern operating systems such as GNU/Linux, Windows, and macOS. [^raymond-psock]
390 |
391 | > Two programs communicating over a socket typically see a bidirectional byte stream. There are other socket modes and transmission methods, but they are of only minor importance. [^raymond-psock]
392 | >
393 | > The byte stream is sequenced and reliable. That is, even individual bytes are received in transmission order and socket users are guaranteed that the underlying network will perform error detection and retry if necessary to ensure delivery. [^raymond-psock]
394 |
395 |
396 |
397 | ### D-Bus
398 |
399 | Many GNU/Linux applications can communicate with each other via D-Bus. **D-Bus** is a message-oriented middleware mechanism that allows communication between multiple processes running concurrently on the same machine. [^free-bus]
400 |
401 | [^free-bus]: [Freedesktop Wiki - What is D-Bus?, accessed 2024](https://www.freedesktop.org/wiki/Software/dbus/)
402 |
403 | > D-Bus is part of the [Freedesktop.org](https://www.freedesktop.org/wiki/Specifications/) project. The goal of Freedesktop.org is to create open standards and interoperability between different GNU/Linux-based desktop environments. Most Freedesktop.org projects are designed to be independent of any specific desktop environment, and improve their compatibility.
404 |
405 | a) D-Bus can be a mediator for many programs. For example, information on an incoming voice-call received through Bluetooth or VoIP can be propagated and interpreted by any currently-running music player, which can react by muting the volume or by pausing playback until the call is finished. [^free-bus]
406 | - Clients should instruct the bus that they are interested in receiving certain signals from a particular object, since a D-Bus bus only passes signals to those processes with a registered interest in them. [^free-bus]
407 | b) In addition, any two applications can use the communication bus to communicate with each other.
408 | - The D-Bus bus makes it simple and reliable to code a *single instance* application. That is, a program that limits itself to executing only one instance at a time. In this case, the D-Bus message is a means of passing to the first instance the command line arguments received by the second instance at startup (before it closes).
409 |
410 | > [!TIP]
411 | > Desktop application [GNOME D-Spy](https://gitlab.gnome.org/GNOME/d-spy) is a simple GUI for browsing existing bus names, objects, interfaces, methods and signals in a D-Bus.
412 |
413 |
414 |
415 | [^shotts-ext]: [The Linux Command Line 2024, Chapter: 27, Section: Exit Status](http://linuxcommand.org/tlcl.php)
416 |
417 | [^raymond-shellout]: [The Art of Unix Programming 2003, Chapter 7, Section: Handing off Tasks](http://www.catb.org/~esr/writings/taoup/html/ch07s02.html)
418 |
419 | [^raymond-tempfiles]: [The Art of Unix Programming 2003, Chapter 7, Section: Tempfiles](http://www.catb.org/~esr/writings/taoup/html/ch07s02.html)
420 |
421 | [^raymond-signl]: [The Art of Unix Programming 2003, Chapter 7, Section: Signals](http://www.catb.org/~esr/writings/taoup/html/ch07s02.html)
422 |
423 | [^raymond-intpatterns]: [The Art of Unix Programming 2003, Chapter 11, Section: Unix Interface Design Patterns](http://www.catb.org/~esr/writings/taoup/html/ch11s06.html)
424 |
425 | [^raymond-psock]: [The Art of Unix Programming 2003, Chapter 7, Section: Sockets](http://www.catb.org/~esr/writings/taoup/html/ch07s02.html)
426 |
427 |
--------------------------------------------------------------------------------
/chapter/04-installing.md:
--------------------------------------------------------------------------------
1 |
2 |
3 | > [!TIP]
4 | > The table of contents can be accessed by pressing the unordered list icon  on top the right corner.
5 |
6 |
11 |
12 |
13 |
14 | # Chapter 4 - Installing applications
15 |
16 |
17 |
18 | The topic of installing software applications is best approached by examples.
19 |
20 |
21 |
22 | ## 4.1 Package management
23 |
24 | Traditionally one of the most significant differences with other operating systems has been related to the open source ecosystem, and the existence of the digital distribution service like package management. Package managers like dpkg have existed as early as 1994 (originally as a shell script). But the meaning of the phrase has evolved significantly. Package management systems became almost equivalent to other digital distribution services created much later such as the Google Play Store.
25 |
26 | A GNU/Linux distribution comes pre-installed with many software applications intended for the end user. Pre-installed software can be removed and additional software can be added using the distribution specific **package manager**. The package manager always has a command line interface, but usually a graphical user interface if offered aswell (often called **software center**).
27 |
28 | Package managers simplify the process of adding, removing and updating software. When installing a new application, the package manager checks what other libraries the application needs, and automates installation these necessary **dependencies**.
29 |
30 | > [!NOTE]
31 | > Traditionally all software offered through package management has been free and open source. But some distributions include some proprietary software.
32 |
33 | ### Software versions
34 |
35 | Most official software repositories do not include the latest available software versions, let alone all possible free and open source software applications. Most packages are intentionally dated, even over two years old, especially on systems advertised as "stable" such as Linux Mint.
36 |
37 | If you wish to use more up-to-date software versions, you need to switch to another GNU/Linux system, hook up with unofficial software repositories or use alternative installation methods such as [flatpak](#flatpak), [deb-files](#dotdeb-files) and [compiling from source](#compiling-from-source).
38 |
39 |
40 |
41 | Traditionally, it has been considered avoidable and dangerous to install software from another distribution's package management system or from an unofficial source. Today, however, alternative installation routes exist and are considered reliable.
42 |
43 |
44 |
45 | ### Lets update: Solus via command line
46 |
47 | ```
48 | # Update everything (system software and software applications)
49 | $ sudo eopkg up↵
50 |
51 | # Update everything except an annoyingly large and frequently updated package (such as papirus-icon-theme)
52 | $ sudo eopkg up --exclude papirus-icon-theme↵
53 |
54 | # Upgrade one specific package only (such as papirus-icon-theme)
55 | $ sudo eopkg upgrade papirus-icon-theme↵
56 |
57 | # Check for and fix any corrupted packages
58 | $ sudo eopkg check | grep Broken | awk '{print $4}' | xargs sudo eopkg it --reinstall↵
59 |
60 | # Install a specific package (such as vivaldi browser)
61 | $ sudo eopkg install vivaldi-stable↵
62 | ```
63 |
64 | > Solus OS is a continuation of the previous project called Evolve OS. Thus ` $ eopkg ` stands for Evolve OS Package.
65 |
66 | ### Lets update: Debian derivative such as Linux Mint
67 |
68 | ```
69 | # Update everything (system software and software applications)
70 | $ sudo apt update↵ # First, update the package index
71 | $ sudo apt upgrade↵ # Install available updates
72 |
73 | # Update everything except an annoyingly large and frequently updated package (such as papirus-icon-theme)
74 | $ sudo apt-mark hold papirus-icon-theme↵ # Hold current version
75 | $ sudo apt update↵ # Update the package index
76 | $ sudo apt upgrade↵ # Install available updates
77 | $ sudo apt-mark unhold papirus-icon-theme↵ # Release hold
78 |
79 | # Upgrade one specific package and only if already installed.
80 | $ sudo apt update↵ # First, update the package index
81 | $ sudo apt install --only-upgrade papirus-icon-theme↵
82 |
83 | # Identify and install missing packages
84 | $ sudo apt update --fix-missing↵
85 | # Identify and fix broken packages
86 | $ sudo apt update --fix-broken↵
87 |
88 | # Install a specific package (such as vivaldi browser)
89 | $ sudo apt update↵ # First, update the package index
90 | $ sudo apt install vivaldi-stable↵
91 | ```
92 |
93 | #### Tools within the Debian package management system
94 |
95 | Programs ` $ apt ` (advanced packaging tool), ` $ apt-get ` and ` $ dpkg ` are all tools within the Debian package management system.
96 | - ` $ apt-get ` manages remote repositories, resolves dependencies, and uses Debian package manager ` $ dpkg ` to actually make the changes of installing and removing packages. ` $ dpkg ` on itself cannot retrieve or download files from remote repositories, nor can it figure out dependencies. ` $ apt-get ` is a full-featured interface to ` $ dpkg `.
97 | - ` $ apt ` is a slightly stripped-back but more user-friendly version of ` $ apt-get `. There are some common commands between ` $ apt ` and ` $ apt-get `. All of these commands can be preceded by ` $ apt ` or ` $ apt-get ` and will behave the same: install, remove, purge, update, upgrade, autoremove. But ` $ apt ` also includes additional commands like search, show, list, and edit-sources.
98 | - The Linux Mint maintainers have developed their own version of ` $ apt `, which is a Python wrapper for ` $ apt-get `.
99 |
100 | ### Recommended software available through package distribution
101 |
102 | - **` vlc `** = DVD and video playback application
103 | - **` nemo `** = File manager
104 | - **` loupe `** = Image viewer
105 | - **` gwenview `** = Image viewer and simple editing
106 | - **` gcolor3 `** = Color picker
107 | - **` pinta `** = Simple image editor
108 | - **` krita `** = Professional drawing application
109 | - **` kate `** = Advanced text editor
110 | - **` meld `** = Comparison of text files
111 | - **` gufw `** = Simple dekstop firewall
112 | - **` vivaldi `** = Chrome based web browser alternative
113 | - **` brave `** = Chrome based web browser alternative
114 | - **` openra `** = Remake of the classic RTS game Red Alert
115 |
116 |
117 |
118 | ## 4.2 Self-contained application formats
119 |
120 | Self-contained application formats are designed to contain an application with all its dependencies, enabling it to work reliably across different distributions and environments. These formats aim to eliminate dependency problems and improve portability across the diverse ecosystem of GNU/Linux distributions.
121 |
122 | > [!NOTE]
123 | > Self-contained application formats will always use more space on the system than common native packages.
124 |
125 | ### AppImage
126 |
127 | **AppImage** files are portable self-contained executables, independent of the underlying GNU/Linux distribution. Each file includes all libraries the application depends on. AppImage does not enforce externally verified digital signatures or sandboxing, but it may be done by some applications. You can find ` *.AppImage files ` in the download section of a software provider’s website.
128 |
129 | ### Snap
130 |
131 | **Snaps** are governed by Canonical, the company behind the Ubuntu GNU/Linux distribution. Unlike Flatpak, Snap contains only one software source , which is managed by Canonical. Although Snap is strictly dependent on Canonical, it can be installed on most other GNU/Linux distributions.
132 |
133 | ### Flatpak
134 |
135 | Applications using **Flatpak** run in a sandbox and need permissions to have access to resources such as bluetooth, network and files. These permissions are configured by the maintainer, and can be added or removed by users on their system. Flatpak allows app developers to directly provide updates to users. Although has become the de facto standard for getting applications packaged with Flatpak, it is possible to host a Flatpak repository that is independent of Flathub.
136 |
137 | > [!NOTE]
138 | > For self-contained software delivery, it seems like Flatpak has won. You don't see large swathes of developer marketing stories coming from the AppImage or Snap camps, nor the excitement from software vendors on these formats. But you do see them from Flatpak.
139 |
140 |
141 |
142 | ## 4.3 Dotdeb files
143 |
144 | The **` *.deb `** format and filename extension refers to **Debian software package file**. It's used to install apps on Debian and its derivatives such as Linux Mint. You can think of deb-files as ` setup.exe ` files in Windows. These are archives containing other files such as the application executable(s), man pages and libraries. Installing the software from a deb file means unpacking and placing all components in the correct directories on your computer.
145 |
146 | You can find deb-packages in the download section of a software provider’s website. You download the file from the internet. Navigate to ` ~/Downloads/ ` directory, using a file manager (such as nemo). And finally double-click the deb file to start the installation procedure.
147 |
148 | ### Recommended software available as dotdeb
149 |
150 | - **Shutter Encoder** (free and open-source) = Video converter
151 | - **Vivaldi** (free) = Chromium based web browser
152 | - **Master PDF Editor** (includes paid options) = Professional pdf solution
153 | - **ONLYOFFICE Desktop Editors** (includes paid options) = Microsoft Office look-alike
154 | - **Beyond Compare** (paid after trial period) = Industry standard file comparison
155 |
156 |
157 |
158 | ## 4.4 Tarball
159 |
160 | In computing, **` $ tar `** is a computer software utility for collecting many files into one archive file, often referred to as a **tarball**, for distribution or backup purposes.
161 |
162 | > The name ` tar ` is derived from *tape archive*, as it was originally developed to write data to sequential I/O devices with no file system of their own, such as devices that use magnetic tape. [^wiki-tar]
163 |
164 | [^wiki-tar]: [Wikipedia - tar (computing), accessed 2024](https://en.wikipedia.org/wiki/Tar_(computing))
165 |
166 | ### Lets install: pandoc
167 |
168 | **Pandoc** is a markup format converter. It can take a document in one of the supported formats and convert its markup to another format. Maintaining the look and feel of the document is not a priority.
169 |
170 | 1. Download and extract [pandoc-3.6.3-linux-amd64.tar.gz](https://github.com/jgm/pandoc/releases/download/3.6.3/pandoc-3.6.3-linux-amd64.tar.gz) or locate a more up to date file from
171 |
172 | 2. Move the files to correct directories such as executables to ` /usr/bin/ ` and documentation to `/usr/share/man/ ` (see [Chapter 2, Section: Directory: /usr/](02-basic.md#directory-usr)).
173 |
174 | ### Lets install: RAR for Linux
175 |
176 | **RAR for Linux** (paid after trial period) is a command line app for handling rar files
177 |
178 | > [!NOTE]
179 | > RAR archive format supports a special type of redundant data called the recovery record. Presence of a recovery record makes an archive larger, but allows to repair it even in case of physical data damage due to disk failure or data loss of any other kind, provided that the damage is not too severe (beyond a user defined percentage of redundancy). Recovery can be done with *repair archive* command. ZIP archive format does not support the recovery record.
180 |
181 | 1. Download and extract: or locate a more up to date file from
182 |
183 | 2. Run the following commands (as instructed in the included ` makefile ` text file):
184 |
185 | ```
186 | $ cd ~/Downloads/rar/↵
187 | $ sudo cp rar unrar /usr/bin↵
188 | $ sudo cp rarfiles.lst /etc↵
189 | $ cp default.sfx /usr/lib↵
190 | ```
191 |
192 | 3. If purchased, copy your license file to any of the following directories:
193 |
194 | ```
195 | ~/
196 | /etc
197 | /usr/lib
198 | /usr/local/lib
199 | /usr/local/etc
200 | # Any of the following filenames is valid:
201 | # "rarreg.key" to ".rarreg.key" or ".rarregkey"
202 | ```
203 |
204 | ### Lets install: exiftool
205 |
206 | **ExifTool** (free and open-source) is a command line app for handling metadata of various file types and formats such as image files
207 |
208 | ExifTool does not need to be installed to run. However, installation makes it available to all users by placing program files in the correct directories such as the main executable to ` /usr/bin/ `, documentation to `/usr/share/man/ ` and API libraries to ` /usr/lib64/ ` (see [Chapter 2, Section: Directory: /usr/](02-basic.md#directory-usr)).
209 |
210 | 1. Download and extract or locate a more up to date file from
211 |
212 | 2. Run commands:
213 |
214 | ```
215 | $ cd ~/Downloads/Image-ExifTool-13.21/↵
216 | $ perl Makefile.PL↵
217 | $ sudo make install↵
218 | ```
219 |
220 |
221 |
222 | ## 4.5 Python applications
223 |
224 | ### Pip for programmers
225 |
226 | The nonprofit [Python Software Foundation](https://www.python.org/psf-landing/) recommends using ` $ pip3 ` for installing python packages and dependencies. **PIP** is a recursive acronym for *pip installs packages*, and connects to an online repository of public packages.
227 | - You can use ` $ pip3 list↵ ` to see which packages you have installed.
228 | - You can use ` $ pip3 list --outdated↵ ` to list all outdated packages.
229 | - You can update all installed packages with the following command:
230 |
231 | ```
232 | $ pip3 list -o | cut -f1 -d' ' | tr " " "\n" | awk '{if(NR>=3)print}' | cut -d' ' -f1 | xargs -n1 pip3 install -U↵
233 | ```
234 |
235 | ### Pipx for end-users
236 |
237 | Downloading and installing *python applications* is a breeze with **` $ pipx `**. It's aimed at end-users instead of programmers and can be seen as an automation wrapper around lower level tools such as ` $ pip3 ` and ` venv ` to **install and manage Python packages that can be run from the command line directly as applications**. ` $ pipx ` simplifies the management of isolated virtual environments, installation of applications and exposes their binaries in ` ~/.local/bin/ `. The management features makes it easy to add, upgrade, and remove python apps as you see fit. This way ` $ pipx ` works as a package manager, similar to ` $ apt ` and ` $ eopkg `.
238 | - You can use ` $ pipx list↵ ` to see which packages you have installed.
239 | - You can use ` $ pipx upgrade-all↵ ` to update all installed packages.
240 |
241 | ### Lets install: trash-cli
242 |
243 | **Trash-cli** (free and open-source) is a command line interface to the [FreeDesktop.org trash can](http://standards.freedesktop.org/trash-spec/trashspec-latest.html); The same trashcan used by Budgie, KDE, GNOME, XFCE and alike, but you can invoke it from the command line and scripts.
244 |
245 | Trash-cli is great for trying out different install options:
246 |
247 | a) Install for current user: ` ~/.local/bin/ `, ` ~/.local/lib/ `, ` ~/.local/share/man/ `
248 |
249 | ```
250 | $ pipx install trash-cli↵
251 |
252 | $ pipx uninstall trash-cli↵
253 | ```
254 |
255 | b) Install for all users: ` /usr/bin/ `, ` /usr/lib/ `, ` /usr/share/man/ `
256 |
257 | ```
258 | $ sudo pip3 install trash-cli↵
259 |
260 | $ sudo pip3 uninstall trash-cli↵
261 | ```
262 |
263 | c) Bleeding Edge (from sources) for all users:
264 |
265 | ```
266 | $ sudo pip3 install git+https://github.com/andreafrancia/trash-cli↵
267 |
268 | $ sudo pip3 uninstall trash-cli↵
269 | ```
270 |
271 | d) Bleeding Edge (from sources) for current user:
272 |
273 | ```
274 | $ pip3 install git+https://github.com/andreafrancia/trash-cli↵
275 | $ export PATH=~/.local/bin:"$PATH"↵
276 |
277 | $ pip3 uninstall trash-cli↵
278 | ```
279 |
280 | > [!NOTE]
281 | > Changes to the environment variable ` PATH ` vanish as the shell session ends. To make the change permanent add the export command ` export PATH=~/.local/bin:"$PATH" ` to a new line on ` ~/.bashrc `.
282 | >
283 | > Making the addition on commanline can be a bit tricky. But fortunately run-control files can be modified using a graphical text editor (invoked from commandline) like this ` $ gedit ~/.bashrc↵ ` or this ` $ xed ~/.bashrc↵ ` (depending on the default text editor provided by the operating system).
284 |
285 |
286 |
287 | ## 4.6 Compiling from source
288 |
289 | **SourceForge** () built up a lot of goodwill in the past, being a centralized place for downloading open-source software and hosting software repositories. Over the years, more projects have moved to other repository-hosting services like GitHub. SourceForge's reputation was tarnished by the addition of adware and malware to free source installation packages during 2013...2015. Since then, SourceForge has tried to rebuild its image by putting an end to questionable practices.
290 |
291 | **GNU Savannah** () serves as a collaborative software development management system for free software projects. It is a project of the [Free Software Foundation](https://www.fsf.org). Unlike GitHub or SourceForge, Savannah's focus is for free software and has very strict hosting policies, including a ban against the use of non-free formats.
292 |
293 | **GitHub** () serves as a leading collaborative software development management system for open source projects. On October 26, 2018, Microsoft acquired GitHub for US$7.5 billion. Early on, a lot of developers worried that Microsoft would reduce its free offerings to squeeze more money out of it. Instead, GitHub expanded its free service and has continued to embrace open source developers. [^git-2022] However, this does not mean that GitHub is in favour of the free software movement.
294 |
295 | [^git-2022]: [Frederic Lardinois - Microsoft says GitHub now has 90M active users, published 2022](https://techcrunch.com/2022/10/25/microsoft-says-github-now-has-a-1b-arr-90m-active-users)
296 |
297 | GitHub's dominance and proprietary nature has spurred development of alternative hosting services such as **Codeberg** (); a German based, non-profit, community-led effort that provides free hosting for *free and open source* projects. Codeberg aims to be the freedom and copyleft respecting alternative to proprietary platforms.
298 |
299 | ### Depencies
300 |
301 | Programs compiled from source usually need additional libraries and packages. These are called **dependencies**. Dependencies come in different types. Some depencies are essential for the program to run. Some enhance functionality, but are not strictly necessary. Some are needed to compile software, but not required for execution.
302 |
303 | > [!IMPORTANT]
304 | > Dependencies are not always named the same across distributions. Different distributions may use different naming conventions. Additionally, some distributions might split dependencies into multiple packages or consolidate them into one, adding to the variation.
305 |
306 | ### Lets compile: Gambas in Solus
307 |
308 | **Gambas** is an object-oriented dialect of the BASIC programming language, as well as the integrated development environment that accompanies it. Gambas is a recursive acronym for Gambas Almost Means Basic. Gambas is developed by the French programmer Benoît Minisini.
309 |
310 | 1. First, install files necessary to compile C code:
311 |
312 | ```
313 | $ sudo eopkg install -c system.devel↵
314 | ```
315 |
316 | 2. Second, install depencies:
317 |
318 | ```
319 | alsa-lib-devel, autoconf, automake, bzip2-devel, curl-devel, curl-gnutls, gcc, gdk-pixbuf-devel, git, glew-devel, glib2-devel, gmime-devel, gmp-devel, gsl-devel, gstreamer-1.0-devel, gstreamer-1.0-plugins-base-devel, gtkglext-devel, imlib2-devel, libcairo-devel, libffi-devel, libgnome-keyring-devel, libgnutls-devel, libgtk-2-devel, libgtk-3-devel, libgtk-4-devel, libgtkmm-2-devel, libgtkmm-3-devel, libpcre-devel, librsvg-devel, libtool-devel, libwebkit-gtk-devel, libwebkit-gtk41-devel, libwebkit-gtk5-devel, libxml2-devel, libxslt-devel, libxtst-devel, llvm-devel, mariadb-devel, ncurses-devel, openssl-11-devel, poppler-devel, poppler-devel, poppler-qt5-devel, poppler-qt6-devel, postgresql-devel, qt5-3d-devel, qt5-base-devel, qt5-svg-devel, qt5-webengine-devel, qt5-webkit-devel, qt5-websockets-devel, qt5-webview-devel, qt5-x11extras-devel, qt6-base-devel, qt6-webengine-devel, qt6-svg-devel, sane-backends-devel, sdl-ttf-devel, sdl1-devel, sdl1-image-devel, sdl1-image-devel, sdl1-mixer-devel, sdl1-sound-devel, sdl2-devel, sdl2-image-devel, sdl2-image-devel, sdl2-mixer-devel, sdl2-ttf-devel, sqlite3-devel, unixodbc-devel, v4l-utils-devel, zstd-devel
320 | ```
321 |
322 | - This list of dependencies corresponds the naming conventions of the Solus project. If you really want to try this, and you are using another operating system, you can try to find out the correct package names with help from the software vendor's [official installation guide](https://gambaswiki.org/wiki/install?nh&l=en#t1).
323 | - And do not be ashamed to use the software center when installing dependencies. A graphical user interface can be a great help, especially when installing on a system where packages have to be searched for, because they are named differently in some cases.
324 | - Note however, that some dependencies are optional, and the functionality provided by them can be disabled, if they are not found on your operating system.
325 |
326 | 3. Third, download and extract the source code:
327 |
328 | 4. Finally, compile the code as instructed in the software providers official [install guide](https://gambaswiki.org/wiki/install?nh&l=en#t23):
329 |
330 | ```
331 | $ cd ~/Downloads/gambas-stable/↵
332 | $ ./reconf-all↵
333 | $ ./configure -C --disable-qt4 --disable-gtk --disable-gtk3 --disable-db.sqlite2 --disable-mongodb --disable-pdf↵
334 | $ make -j $(nproc)↵ # If configuration succeeded
335 | $ sudo make install↵ # If compilation succeeded
336 | ```
337 |
338 | ### Lets compile: Gambas in Linux Mint
339 |
340 | 1. First, install files necessary to compile C code:
341 |
342 | ```
343 | $ sudo apt-get update↵ # First, update the package index
344 | $ sudo apt-get install -y build-essential↵
345 | ```
346 |
347 | 2. Second, install depencies:
348 |
349 | ```
350 | autoconf, automake, g++, git, libalure-dev, libasound2-dev, libbz2-dev, libcairo2-dev, libcurl4-gnutls-dev, libdirectfb-dev, libdumb1-dev, libffi-dev, libgdk-pixbuf2.0-dev, libglew-dev, libglib2.0-dev, libgmime-3.0-dev, libgmp-dev, libgsl-dev, libgstreamer-plugins-base1.0-dev, libgstreamer1.0-dev, libgtk-3-dev, libgtk2.0-dev, libgtkglext1-dev, libimlib2-dev, libmysqlclient-dev, libncurses5-dev, libpcre3-dev, libpoppler-cpp-dev, libpoppler-dev, libpoppler-glib-dev, libpoppler-private-dev, libpq-dev, libqt5opengl5-dev, libqt5svg5-dev, libqt5webkit5-dev, libqt5x11extras5-dev, librsvg2-dev, libsdl-image1.2-dev, libsdl-mixer1.2-dev, libsdl-sound1.2-dev, libsdl-ttf2.0-dev, libsdl2-dev, libsdl2-image-dev, libsdl2-mixer-dev, libsdl2-ttf-dev, libsqlite0-dev, libsqlite3-dev, libssl-dev, libtool, libv4l-dev, libwebkit2gtk-4.0-dev, libxml2-dev, libxslt1-dev, libxtst-dev, libzstd-dev, linux-libc-dev, llvm, llvm-dev, qtbase5-dev, qtwebengine5-dev, sane-utils, unixodbc-dev
351 | ```
352 |
353 | - Note however, that some dependencies are optional, and the functionality provided by them can be disabled, if they are not found on your operating system..
354 |
355 | > [!NOTE]
356 | > At the time of writing, the most recent version of Linux Mint 22.1 was based on Ubuntu LTS version **22.04 Jammy**. So the correct depency names would match those in the software vendor's [official installation guide](https://gambaswiki.org/wiki/install?nh&l=en#t15).
357 |
358 | 3. Third, download and extract the source code:
359 |
360 | 4. Finally, compile the code as instructed in the software providers official [install guide](https://gambaswiki.org/wiki/install?nh&l=en#t23):
361 |
362 | ```
363 | $ cd ~/Downloads/gambas-stable/↵
364 | $ ./reconf-all↵
365 | $ /configure -C --disable-qt4 --disable-keyring↵
366 | $ make -j $(nproc)↵ # If configuration succeeded
367 | $ sudo make install↵ # If compilation succeeded
368 | ```
369 |
370 | ### Gambas is one of the most comfortable options for self-learners
371 |
372 | I have been thoroughly impressed by Gambas. It is a remarkably user-friendly development environment that feels like an enhanced version of Visual Basic 2008, complete with modern features that make it especially comfortable for self-learners. It captures the spirit of classic BASIC while going far beyond it in capability. One of its standout strengths is its refreshingly lightweight debugger; quick to launch, easy to use, and seamlessly integrated into the code editor. This makes it perfect for rapid, iterative development, offering a streamlined experience that emphasizes speed, simplicity, and clarity; qualities that are increasingly rare and welcome today.
373 |
374 | #### Beginners' language
375 |
376 | The BASIC programming language stands for **Beginners' All-purpose Symbolic Instruction Code**. BASIC syntax is clear and English-like, making it approachable and readable. BASIC is excellent for teaching fundamental programming logic. Its commands often resemble plain English, making code more readable and less intimidating, especially for students and self-taught coders. [^wiki-basic]
377 |
378 | > Microsoft helped popularize BASIC by combining an updated version of BASIC with a *visual forms builder* by releasing Microsoft Visual Basic in 1991. This reignited use of the language. This push continued with the release of Visual Basic .NET in 2002, marking a significant evolution in the language's capabilities. These innovations sparked renewed interest in BASIC. Visual Basic was once a cornerstone of Windows application development. However, its popularity has gradually declined, especially after Visual Studio 2010. In 2020, Microsoft announced that no major new features would be added to VB.NET; although support would continue in form of security updates and bug fixes. [^wiki-basic]
379 |
380 | #### BASIC is considered a family of programming languages
381 |
382 | However, Microsoft did not invent the BASIC programming language. The original version was created at Dartmouth College in 1964. At the time, nearly all computers required writing custom software, which only scientists and mathematicians tended to learn. The original creators wanted to enable students in non-scientific fields to use computers. The language was designed to be easy to learn and use, and has helped people take their first steps into programming without needing a computer science degree. [^wiki-basic]
383 |
384 | > [!NOTE]
385 | > Since its creation BASIC has evolved into a wide array of dialects and implementations tailored for different platforms, purposes, and user communities.
386 |
387 | #### Hobbyist scene for BASIC continues to exist
388 |
389 | Despite shift in developer preference and industry trends, BASIC is not gone. Microsoft continues to support it in Visual Studio. And a hobbyist scene for BASIC continues to exist on other platforms aswell. [^wiki-basic]
390 |
391 | BASIC and especially Gambas IDE (integrated development environment) can be a great fit for computer hobbyists; particularly those who are new to programming. Its readable syntax and drag-and-drop interface via visual forms builder makes it beginner-friendly. Gambas is well-suited for building GUI applications using Qt and GTK widgets. The language is also handy for automating tasks in LibreOffice via BASIC macros.
392 |
393 | [^wiki-basic]: [Wikipedia - BASIC, accessed 2025](https://en.wikipedia.org/wiki/BASIC)
394 |
395 |
396 |
397 |
398 |
399 |
--------------------------------------------------------------------------------
/chapter/09-multi-user.md:
--------------------------------------------------------------------------------
1 |
2 |
3 | > [!TIP]
4 | > The table of contents can be accessed by pressing the unordered list icon  on top the right corner.
5 |
6 |
11 |
12 |
13 |
14 | # Chapter 9 - Access management
15 |
16 |
17 |
18 | ## 9.1 Introduction to shared computing
19 |
20 |
21 |
22 | Years ago, before computers were personal, they were large, expensive, and centralized. One single computer would support many users at the same time. A typical computer system consisted of a large central computer located in one building, hooked up by terminals located throughout the campus.
23 | - Even on a home PC the command prompt reflects the unix nature of multiuser capability and remote access capability:
24 |
25 | ```
26 | user@host working_directory $ _
27 | ```
28 |
29 | To make shared use practical, a method had to be devised to protect users from each other. After all, the actions of one user could not be allowed to crash the computer, nor could one user interfere with the files belonging to another user.
30 |
31 | To uderstand the Unix security model, one needs to learn the core terminology:
32 | 1. An **identity** consists of a **user identity** and one or more **group identities**. Users may belong to a group, and a group consists of one or more users. See [Section: identity](#indentity).
33 | 2. The **ownership** of files and directories is determined by the **user-owner** and the **group-owner**. When a user owns a file or directory, he or she controls access to it. The owner can give others access to the files and directories they own. See [Section: Ownerhip of files and directories](#ownership).
34 | 3. File and directory **permissions attributes** grant or deny the right to read, write and execute. See [Section: Permissions attributes of files and directories](#permission-attributes).
35 |
36 |
43 |
44 |
50 |
51 |
52 |
53 | ## 9.2 Identity (user + group)
54 |
55 | - When user accounts are created users are assigned:
56 | - A numerical **user id (uid)**
57 | - A numerical primary **group id (gid)**
58 | - All user-id-numbers and group-id-numbers are mapped to textual usernames and groupnames for the sake of the humans.
59 | - Traditionally unix-like systems assigned regular users to a common group such as "users", but modern practice is to create a unique, single-member group with the same name as the user. This makes certain types of permission assignment easier.
60 | - Many GNU/Linux distributions start the numbering of regular user accounts and groups at 1000, and the numbers 000...999 are reserved for system services.
61 | - You do not have nested groups in GNU/Linux (i.e. one group cannot be sub group of another).
62 |
63 | > [!TIP]
64 | > To find out information about your identity try:
65 | > ```
66 | > $ groups↵ # to see what groups you belong to.
67 | > $ id↵ # to reveal the user-id and group-id-numbers.
68 | > ```
69 |
70 | ### More information can be found from text files
71 |
72 | Like so many things in GNU/Linux, more information can be found from a couple of text files:
73 | - User database is stored in **` /etc/passwd `**, which defines user accounts by holding information about username, uid number, primary gid number, account name, home directory, and login shell.
74 | - All the groups defined in the system, and any supplementary groups the user may belong to (if any) are stored in **` /etc/group `**.
75 | - User passwords and password expiry information is stored in **` /etc/shadow `**.
76 |
77 | > [!NOTE]
78 | > The three files above are all **DSV-format** (Delimiter Separated Values) with colon **` : `** as the value separator. Data files in this style are expected to support inclusion of colons in the data fields by backslash escaping. More generally, code that reads them is expected to support record continuation by ignoring backslash-escaped newlines.
79 |
80 | ### Shadow password system
81 |
82 | Despite the name ` /etc/passwd `, passwords are stored elsewhere in ` /etc/shadow `. In the old days, decades ago, Unix systems kept user passwords stored, in plain text, in the ` /etc/passwd ` file (hence its name.) This, for very obvious reasons, was not at all good. But there was no simple solution to this as any change to an encrypted password system would break backwards compatibility with the ` /etc/passwd ` file and with any tools that depended on it. [^miller-shadow]
83 |
84 |
85 | > [!IMPORTANT]
86 | > The solution was to make a new file, the ` /etc/shadow ` file that would "shadow" the ` /etc/passwd ` file and provide a place for a new encrypted password along with some additional password controls (see [Chapter 10, Section: Encrypted password system](10-additional.md#edit-shadow)). [^miller-shadow]
87 |
88 | So while having a separate file is a bit bizarre today, it has an historic context that makes sense. No functionality is lost and the additional complexity is nominal. Unlike the ` /etc/passwd ` and ` /etc/group ` files, ` /etc/shadow ` is protected and normal users cannot see its contents. This provides an additional level of security for it. There is no need for users to see what ` /etc/shadow ` contains, but other files are useful to see what accounts are available, groups exists and who are members of them. [^miller-shadow]
89 |
90 | > The shadow password system was first introduced in the mid-1980s by Sun's SunOS UNIX system and by 1990 was widely copied and essentially ubiquitous in Unix systems. So the shadow concept is used in most Unix and has been in GNU/Linux since inception. BSD however uses ` /etc/master.passwd ` instead. [^miller-shadow]
91 |
92 | [^miller-shadow]: [Scott Miller - The /etc/shadow file in depth, published 2016](http://mangolassi.it/topic/8057/)
93 |
94 | ### Special user id: root (0)
95 |
96 | If we examine the contents of ` /etc/passwd ` and ` /etc/group `, we notice that there is a user **uid 0**, which is required by any unix system. It is the superuser account, and it has all the rights.
97 |
98 | Calling the uid 0 account on any unix system root is a very strongly held convention. Though root is just a string, just a name, and in theory the actual username for uid 0 can differ, the OS itself won't care, but some applications might not quite like it because they expect there to exist a privileged account named root.
99 |
100 | #### Modern systems have no root password
101 |
102 | Modern GNU/Linux distributions do not have a root password set (by default). Modern systems use the [sudo program](https://www.sudo.ws/about/intro/) to allow (regular) users to run commands as root (see [Section: Sudo program](#sudo-program)). And when the sudo program asks for password authentication, every one supply their own password. After authentication, the system will invoke the requested command as root. Thus no specific root password is needed, which would have to be shared by many users; and thus pose a security risk.
103 |
104 | ### Change user-identity
105 |
106 | The command **` $ su `** stands for Switch User.
107 | - ` $ su margaret↵ ` = Switch user to margaret. Margaret's password is required.
108 | - ` $ su edward↵ ` = Switch user to edward. Edward's password is required.
109 | - ` $ sudo su edward↵ ` = Switch user to edward. Edward's password is not required.
110 | - ` $ sudo su root↵ ` = Switch user to root. Typing root as username is optional, since ` $ sudo su↵ ` does the same thing.
111 | - ` $ exit↵ ` = Log out of another user's account.
112 |
113 | > [!IMPORTANT]
114 | > The right to login as another or the root user comes from the rights granted in the ` /etc/sudoers ` file or a configuration file under ` /etc/sudoers.d/ ` directory (see [Section: Editing sudo configuration](#editing-sudo-configuration)). When using ` $ sudo `, no specific root password is asked for; instead every user use their own password.
115 |
116 | > [!NOTE]
117 | > By running the ` $ sudo su↵ ` command once will switch the shell to continuous root access. This removes the need to type in ` $ sudo ` repeatedly before commands. Only the ` # exit↵ ` command logs you out of root mode. This may be convenient, but may also increase the risk of causing accidental damage in certain situations.
118 |
119 |
136 |
137 |
138 |
139 | ### The sudo program
140 |
141 | The regular user does not have write access to a file system other than their own home directory. Maintenance operations, such as updating programs, are performed with the root account via the command ` $ sudo ... ` or ` $ sudo su root↵ `. Such a system is good for data security, as it reduces the possibility of unitended damage and makes misuse more difficult.
142 |
143 | > [!NOTE]
144 | > Name of the program: **sudo** stands for *super user do* and originates from the verb *to supervise*, i.e. to be in charge of somebody or something and make sure that everything is done correctly, safely, etc.
145 |
146 | #### A better way to grant elevated rights
147 |
148 | In the early 1980's the standard way of executing commands as a superuser was using the ` $ su root↵ ` command, which enables the user to switch to a superuser mode (username root). While this did the trick, it opened a lot of opportunities for human error; it was just too easy to forget you’re in root mode and end up causing inadvertent damage to the system. [^sudo-hist]
149 |
150 | Bob Coggeshall and Cliff Spencer thought of a better way working at the SUNY/Buffalo Computer Science at the time. Instead of constantly switching, why not simply create a tool that enables executing individual commands as a superuser, without changing the actual user id in the shell. The ` $ sudo ` command was born an slowly began to make its way into other research groups and was formally open sourced in 1985. Today sudo counts 9944 lines of code (up from 153 lines in the original release) and is maintained by Todd C. Miller. Over 30 years in age, it still continues to receive code contributions and regularly issues new releases. [^sudo-hist]
151 |
152 | [^sudo-hist]: [Aleksandar Bradic - Inventing the Unix sudo command, published 2014](https://hackaday.com/2014/05/28/interview-inventing-the-unix-sudo-command/)
153 |
154 | #### Normally, even administrators have restricted access
155 |
156 | All users, even the administrator who has been given all possible permissions in the configuration file, will normally act as a normal user with limited permissions. When activities such as installing updates or apps, or changing config files arise, the system prompts for the user's password or ask to login as another user who has administrator privileges. Once the elevated activity is completed, one reverts back to working as a standard user with restricted access rights.
157 |
158 | There is no specific root password to be shared between users. When sudo prompts for a password authentication, every user supply their own password. After authentication, the system will invoke the requested command as root.
159 |
160 | > [!NOTE]
161 | > Even the administrator himself must use ` $ sudo ` to run commands that require elevated privileges.
162 |
163 | #### Sudoing rights are configured
164 |
165 | The **` $ sudo `** program allows an administrator to set up configuration files under ` /etc/sudoers.d/ ` to define specific commands that particular users are permitted to execute under an assumed identity and elevated privileges. It is the sudo configuration files that grant certain users (or groups of users) the ability to run some commands as root (or another user). This convention allows a system administrator to delegate authority without revealing their own password or a specific root account password. See [Section: Editing sudo configuration](#editing-sudo-configuration).
166 |
167 | > [!TIP]
168 | > Use **` $ sudo -l↵ `** to see what privileges are granted by sudo to you specifically.
169 |
170 | #### Sudo ticketing system
171 |
172 | Sudo comes with a ticketing system, that allows a user to run privileged commands for a period of time without the need to repeatedly authenticate. When a user invokes sudo and enters their password, sudo caches the credentials for 5 minutes, per user, per terminal session. Each subsequent sudo command updates the ticket for another 5 minutes.
173 |
174 | #### Sudo command history
175 |
176 | Sudo may be configured to maintain a log of each command run, so that system administrators can trackback the person responsible for any undesirable changes in the system.
177 |
178 |
179 |
180 | ## 9.3 Ownership (of files and directories)
181 |
182 | In the unix-like security model, to ensure that a file or directory can be accessed, modified or executed by the intended user(s) only, every file and directory in GNU/Linux have (and is required to have):
183 | 1. A user owner, which is a single user.
184 | - You cannot have 2 users owning the same file.
185 | - When you create a file, you become the owner of the file.
186 | 2. A group owner, which is a single group.
187 | - You cannot have 2 groups owning the same file.
188 | 3. In addition rights can be granted to everybody (group other).
189 | - Other(s) can be considered as a super group with all the users on the system. Basically, anyone with access to the system belongs to this group.
190 |
191 | ### Beware when creating files with sudo involved
192 |
193 | When creating a file, the current identity becomes the owner of the file.
194 |
195 | a) It is clear that root becomes the owner of a file created under the ` # ` prompt, that is after ` $ sudo su root↵ `. The ownership can be changed later on of course.
196 |
197 | b) Creating files with sudo may have unexpected results, ending up with files owned by root or a regular user. This is of course just an oversight by the user:
198 | - With command ` $ sudo echo Aaa > Aaa.txt↵ ` only echo gets run under sudo (i.e. as root), but the redirection to file is run non-privileged (i.e. the user's own user-id). Thus the owner of the newly created file is the common user.
199 | - To end up with a root owned file, and as echo doesn't need to be run privileged, a better alternative would be ` $ echo Aaa | sudo tee Aaa.txt > /dev/null↵ `. Note how ` /dev/null ` is used here to stop duplicate output to the screen.
200 |
201 | > [!NOTE]
202 | > Even under root identity, the ownership of a file doesn't change to root if the file being changed already exists. The original ownership is retained.
203 |
204 | ### Change ownership (of files and directories)
205 |
206 | The **` $ chown `** command stands for Change Ownership (of a file or directory):
207 |
208 | ` $ chown ↵ `
209 | - Change user-owner.
210 | - Group-owner remains unchanged.
211 |
212 | ` $ chown :↵ `
213 | - Change group-owner.
214 | - User-owner remains unchanged.
215 |
216 | ` $ chown :↵ `
217 | - Change both user- and group-owners.
218 |
219 | ` $ chown : ↵ `
220 | - Change both user- and group-owners.
221 | - Group-owner will change to the primary group of the specified user.
222 |
223 | > [!NOTE]
224 | > In older versions of Unix, the ` $ chown ` command only changed file ownership (not group ownership). For that purpose, a separate command, ` $ chgrp ` was used. It works much the same way as chown, except for being more limited.
225 |
226 |
227 |
228 | ## 9.4 Permission attributes (of files and directories)
229 |
230 | The user needs
231 | - Read permission to read from the file.
232 | - Read and write permissions to write to the file.
233 |
234 | The right to delete and rename files is
235 | - Determined by the permissions attributes of the directory.
236 |
237 | > [!NOTE]
238 | > Consider a scenario where you have write permission on a file, but do not have write permission on the directory where the file is stored. You will be able to modify the file contents. But you will not be able to rename, move or remove the file from the directory.
239 |
240 | | SymBinOct | Permission on a FILE | Permission on a DIRECTORY |
241 | |:--- |:--- |:--- |
242 | | `--- 000 0` | Denied from opening the file. | Denied from listing files stored in the directory. |
243 | | `--x 001 1` | \ | \ |
244 | | `-w- 010 2` | \ | \ |
245 | | `-wx 011 3` | \ | \ |
246 | | `r-- 100 4` | Authorised to open the file. | Restricted authority to list only names of files stored in the directory. |
247 | | `r-x 101 5` | Authorised to open or execute the file. | Authorised to list files stored in the directory including file properties (such as permissions, dates and file size). |
248 | | `rw- 110 6` | Authorised to open and modify the file. **This is the base permission for files.** | \ |
249 | | `rwx 111 7` | Authorised to open, modify and execute the file. | Authorised to add, remove and rename files stored in the directory. **This is the base permission for directories.** |
250 |
251 | ### Who can change permission attributes?
252 |
253 | Only the *user owner* of the file or the root user can change the mode of a file.
254 |
255 | 1. You must own a file to change its permission attributes i.e. change the mode of a file; mere read and write access to a file is not enough to modify its metadata. The owner of a file may change the group of the file to any group of which that owner is a member. But a privileged process may change the group arbitrarily.
256 |
257 | 2. Uid 0 i.e. root always has full access to the system and can control the entire system such as files, processes and applications.
258 |
259 | 3. Group people can not change permission attributes. However write access to the directory containing the file allows you to take the ownership of the file. Assuming that you have read permissions to the file and write permissions to the directory then you can take ownership of a file:
260 |
261 | ```
262 | $ mv somefile somefile.tmp↵
263 | $ cp somefile.tmp somefile↵
264 | $ rm somefile.tmp↵
265 | ```
266 |
267 | > [!NOTE]
268 | > If you want security for a file, then the folder (where the file is located) must not have write permission on the owner group. Group people can't chmod, but you also need to deny the add, delete, rename folder permissions; Through which one could otherwise capture file ownership. Note that, denying write permission to a directory does not take away the permission to modify or truncate a file inside the directory, however.
269 |
270 | ### How to change permission attributes
271 |
272 | The **` $ chmod `** command stands for change mode. Use this command to make permission changes to an existing file or directory:
273 |
274 | ```
275 | + add
276 | - remove
277 | = set
278 | ↓
279 | $ chmod ??? ↵
280 | ↗ ↖
281 | u user x execute
282 | g group w write
283 | o other r read (or any combination of these such as rwx)
284 | a all
285 | (or any combination of these such as ugo)
286 | ```
287 |
288 | ### Special permission attributes (setuid, setgid, sticky)
289 |
290 | Though we usually see an octal permission mask expressed as a three digit number, it would be technically more correct to express it in four digits, because (in addition to read, write, and execute permission) there are three other (less used) permission settings. [^shotts-sperm]
291 |
292 | #### Special flag: Sticky
293 |
294 | The **sticky bit** (octal 1000) is a holdover from ancient Unix, where it was possible to mark an executable file as *not swappable*. On files, modern unix ignores the sticky bit, but if applied to a directory, it prevents users from deleting or renaming files unless the user is either the owner of the directory, the owner of the file, or the superuser. [^shotts-sperm]
295 |
296 | > In the original Unix systems, the sticky bit had a different purpose compared to its modern usage. It was introduced in the Fifth Edition of Unix (1974) and was primarily used for executable files. When the sticky bit was set on an executable, it instructed the operating system to keep the program's text segment (code) in swap space after the program finished running. This allowed the program to load faster the next time it was executed, as it avoided reloading the code from slower storage. This feature was particularly useful in systems with limited memory and frequent use of certain programs, like text editors. However, as operating systems evolved and memory management improved, this functionality became obsolete. [^wiki-sticky]
297 |
298 | > [!NOTE]
299 | > Today, the sticky bit is used for directories in Unix-like systems, ensuring that only the file owner, directory owner, or root can delete or rename files within a directory. This is often used to control access to a shared directory such as ` /tmp/ ` and ` /var/tmp/ `.
300 |
301 | [^wiki-sticky]: [Wikipedia - Sticky bit, accessed 2025](https://en.wikipedia.org/wiki/Sticky_bit)
302 |
303 | #### Special flag: Set group ID
304 |
305 | The **setgid bit** (octal 2000) i.e. *Set Group ID* bit has its origins in early Unix systems and serves different purposes depending on whether it's applied to files or directories. [^shotts-sperm]
306 |
307 | > The setgid bit was introduced to allow executables to run with the permissions of the group that owns the file, rather than the group of the user executing it. [^shotts-sperm]
308 |
309 | On most systems, when applied to a directory, setgid bit ensures that any files or subdirectories created within inherit the parent directory's group ownership, rather than the creator's default group. Newly created subdirectories also inherit the Set group ID bit of the parent directory. This is useful in a shared directory when members of a common group need access to all the files in the directory, regardless of the file owner's primary group. Lessening the need to use ` $ chmod ` or ` $ chown ` to share new files. [^shotts-sperm]
310 |
311 | #### Special flag: Set user ID
312 |
313 | Today, the **setuid bit** (octal 4000) i.e. *Set user ID* bit remains a critical feature in Unix-like systems, but its use is carefully controlled to minimize security risks. Its primary purpose was (and still is) to allow users to execute programs with the privileges of the owner of the executable, rather than the privileges of the user calling it. This was a groundbreaking feature for enabling secure access to system resources. This feature can be used to give restricted, program-controlled access to things like the password file that non-administrators should not be allowed to modify directly. [^shotts-sperm]
314 |
315 | > [!NOTE]
316 | > The setuid bit answers the question: How can a low privilege users change their password, when it is stored in a file only accessible by the root user?
317 | >
318 | > ```
319 | > $ ls -la /etc/shadow↵
320 | > -rw------- 1 root root 787 13. 9. 16:05 /etc/shadow
321 | > ```
322 |
323 | - The answer is the **setuid concept** through the *Set user ID* bit. A setuid program (such as ` $ passwd `) is run with the program owner’s privilege (without a password). Passwd is just like any other program, except it has that special marking.
324 |
325 | > [!TIP]
326 | > You may search for other files with setuid permission by typing ` $ sudo find / -user root -perm -4000 -exec ls -ldb {} \;↵ `. The search results include setuid programs such as ` $ mount ` and ` $ umount `.
327 |
328 | Every process has two User IDs: 1) **Real User ID (RUID)** which identifies the real owner of a process. 2) **Effective User ID (EUID)** which identifies the privilege of a process. Access control is based on EUID: [^ieee-euid]
329 |
330 | a) When a normal (non setuid) program is executed: RUID and EUID both equal to the ID of the user who runs the program. [^ieee-euid]
331 |
332 | b) When a program is executed with the *Set user ID* bit applied to the executable: RUID still equal to the user ID of the user who runs the program. But EUID equals to the owner ID of the executable file. And if the program is owned by root the program runs with the root privilege. [^ieee-euid]
333 |
334 | [^ieee-euid]: [IEEE 1003.1 Standard: set user ID, accessed 2025](https://pubs.opengroup.org/onlinepubs/9799919799/functions/setuid.html)
335 |
336 |
337 |
338 | ## 9.5 Single User Mode
339 |
340 | **Single (user) mode** is a recovery or maintenance mode to gain super user root access without any credentials or password. For this you need to have physical access to the machine. To enter single user mode, temporarily append ` single ` to your boot options:
341 |
342 | 1. By default, (U)EFI installations will not show the boot menu, and boot directly into the operating system (such as Solus).
343 |
344 | a) By hitting ` Space ` bar (repeatedly) during boot, the boot menu should appear. It may take a couple of goes to get the timing right.
345 | b) You may wish to configure your operating system to display the boot menu by default (on every startup):
346 |
347 | ```
348 | $ sudo clr-boot-manager set-timeout 30↵
349 | $ sudo clr-boot-manager update↵
350 | # These are for (U)EFI only,
351 | # not GRUB / legacy-BIOS.
352 | ```
353 |
354 | 2. Once in the boot menu (UEFI or GRUB) highlight the desired kernel and press ` E ` key. Find a good spot to append ` single ` (such as rigth next to ` quiet `). Finally press the ` Enter ` key on UEFI or ` F10 ` on GRUB / legacy-BIOS.
355 |
356 | > [!NOTE]
357 | > These changes to boot parameters are not persistent. Any change to the kernel boot options made this way will only affect the current boot.
358 |
359 | 3. When you get to the shell prompt in single user mode, the hard drive is propably mounted read only. The recommended method is to run file system check to ensure nothing is corrupted. ). Then remount to enable write.
360 |
361 | ```
362 | # fsck -fy↵ # Force a check even if journalling
363 | # is enabled, and answers Yes to any
364 | # requests to fix any problems found.
365 | # mount -uw /↵
366 | ```
367 |
368 | 4. Finally you should be able to do what you need to do (such as reset a forgotten user password: ` # passwd margaret↵ `).
369 |
370 | > [!WARNING]
371 | > The ability to boot a computer into single user mode could be a major security risk, as it will give intruders immediate root access.
372 |
373 | ### Protecting against single user mode
374 |
375 | Once someone has physical access to a machine, it becomes very hard to completely secure it. The easiest way to gain unauthorised access to a GNU/Linux system is to boot the machine into single user mode (see [Section: Single user mode](#single-user-mode)).
376 | - By default, some distributions do not have a root password set. This can be checked by running the command ` $ sudo head -1 /etc/shadow↵ `. If no root password is set, then regardless of whether, the system is set to prompt for a password for single user mode or not, it will just load root access. This default behaviour can be [changed](https://askubuntu.com/questions/1011368/how-can-i-protect-against-single-user-mode).
377 | - One does also need a [procedure](https://askubuntu.com/questions/76987/how-can-i-prevent-someone-from-resetting-my-password-with-a-live-cd/78051#78051) to protect unauthorised editing of bootloader settings; That is, pressing ` E ` whilst booting, to allow edits to the boot options. Once properly configured, when one tries to edit a boot loader entry, the system will ask for a boot loader specific username and password.
378 | - However, if a "live cd" is used, a "recover linux feature" on the live session can be used to mount the file system and alter the bootloader settings. To protect against such a hack, one should make any removable media have a lower boot priority than the boot drive, and password protect the BIOS and boot option menu, to stop someone who has not got access altering the boot order and booting into a disk to make changes to the system.
379 |
380 | > [!IMPORTANT]
381 | > Such security procedures should be used in conjunction with hard-disk encryption.
382 |
383 |
384 |
385 | ## 9.6 Remote use
386 |
387 | The term **console** or **terminal** refers to a device connected to a computer (such as a serial cable) that does not itself have the ability to run application programs, but acts as an additional combination of display and input devices (usually a monitor and keyboard) to which system notifications are primarily sent. Terminals flourished before the time of desktop computers, when mainframes were usually used precisely through terminals. Although PCs have almost completely supplanted traditional terminals, the possibility of remote use (ssh program) has been retained in unix-type operating systems.
388 |
389 |
392 |
393 | ### Remote access protocol: SSH
394 |
395 | Up until the 1990s, before the general adoption of the internet, there were two popular programs used to log in to remote hosts: telnet in the 1970s and rlogin in the 1980s. These insecure programs used to transmit all their communications (including login names and passwords) in cleartext. This made them wholly inappropriate for use in the internet age. [^shotts-ssh]
396 |
397 |
398 |
399 | In 1995, Finnish-born Tatu Ylönen developed a new encrypted communication protocol, SSH (Secure Shell), to solve the problem by verifying that the remote host is who it says it is, preventing attacks from intermediaries and encrypting all communications. SSH works on the **TOFU** principle (trust on first use), where the server certificate is accepted the first time and the user is warned if it changes. SSH uses keys, so the user does not necessarily have to enter any passwords at all. SSH client software is now part of the default installation of most Unix like operating systems, and is usually accessible from the command line with ` $ ssh `. [^wiki-fi-ssh]
400 |
401 | [^wiki-fi-ssh]: [Wikipedia - SSH, accessed 2025](https://fi.wikipedia.org/wiki/SSH)
402 |
403 |
404 |
405 | ## 9.7 Editing sudo configuration
406 |
407 | ### The sudo configuration files
408 |
409 | To manage sudo's configuration it is possible to 1) edit the primary configuration file ` /etc/sudoers ` or 2) make a new configuration files under the ` /etc/sudoers.d/ ` directory.
410 |
411 | The ability to have stand alone files makes it simple for an application to enable sudo capability on installation and remove them when it is uninstalled. Automated configuration tools can also use this capability.
412 |
413 | Sudo defines that in case of multiple lines matching for a user, the last one stands. That is any files under ` /etc/sudoers.d/ ` are parsed after ` /etc/sudoers ` file. So potentially a file in ` /etc/sudoers.d/ ` could restrict permissions for someone.
414 |
415 | It is recommended to put custom configuration into files in the ` /etc/sudoers.d/ ` directory, because typically ` /etc/sudoers ` is under control of the package manager, and if the package manager wants to upgrade it, one has to manually inspect any local changes and approve how they are merged into the new version. Modifications done directly in the ` /etc/sudoers ` file may even break updates. By placing local changes into a file in the ` /etc/sudoers.d/ ` directory, one garantees that upgrades can proceed automatically. And any changes made to files in ` /etc/sudoers.d ` remain in place if after system upgrade. This can prevent user lockouts when the system is upgraded.
416 |
417 | > [!NOTE]
418 | > The following convention for ignoring files applies in ` /etc/sudoers.d/ `: Files ending in a tilde ` ~ ` or containing a dot ` . ` are ignored. This convention allows, for example, backup files to be ignored.
419 |
420 |
421 |
422 | > [!IMPORTANT]
423 | > Make sure the ` /etc/sudoers ` file contains the line:
424 | >
425 | > ```
426 | > #includedir /etc/sudoers.d
427 | > ```
428 | > Most likely the default ` /etc/sudoers ` file created on installation (of the operating system) includes this directive. However, if not then you must add it yourself. Only then will the sudo program read configuration files from the ` /etc/sudoers.d/ ` directory.
429 |
430 | ### Format for user privilege specification
431 |
432 | ```
433 | margaret ALL = (root) NOPASSWD: /usr/bin/eopkg up
434 | | | | | |
435 | Who? Where? | Without need to This command
436 | | | | type a password?
437 | User(s) Host(s) |
438 | |
439 | Under an assumed identity of (user root)
440 | ```
441 |
442 | 1. The user specifier accepts ` user_names `, ` %group_names ` and ` #uids `.
443 | - One may specify a single user or group or a list of them separated by ` , `
444 | 2. ` ALL ` is a common choice for the host list.
445 | 3. The command specifier accepts a path to an executable,
446 | - Followed by optional allowed arguments such as
447 | - ` /usr/bin/some-app ` = Any arguments allowed.
448 | - ` /usr/bin/some-app "" ` = No arguments allowed at all.
449 | - ` /usr/bin/some-app some-argument ` = Only specific argument allowed.
450 | - One may specify a single command or several separated by a comma ` , `
451 |
452 | ### Visudo
453 |
454 | A comment ` # This file MUST be edited with the 'visudo' command ` on the ` /etc/sudoers ` file states you should not edit sudoers directly, by opening it in a text editor, but instead edit it with ` $ visudo `, which prevents editing conflicts and checks for syntax errors before saving the modifications to disk. By default, visudo doesn't honor the ` VISUAL ` or ` EDITOR ` environment variables, used by many programs to determine the default text editor. There is a hard-coded list of one or more editors that visudo uses. The default is ` $ vi ` (hence the name visudo).
455 |
456 | > [!TIP]
457 | > In reality, you don't have to use the cumbersome ` $ visudo `. You can use a text editor of your choice such as ` $ gedit ` or ` $ ne `. It is possible to wall your self out of the system if the ` /etc/sudoers ` file or any file under ` /etc/sudoers.d/ ` directory is malformed.
458 | >
459 | > Sufficient safety can be achieved by having a second terminal instance running as root ` $ sudo su root↵ `. There is no need for the sudo command once you are logged in as root as indicated by the ` # ` prompt. And any changes can be tested on another terminal instance running as an ordinary user. This way you can verify, that whatever you just did, left sudo running properly. If not, you have the root window to fix it.
460 |
461 |
462 |
463 | [^shotts-sperm]: [The Linux Command Line 2024, Chapter: 9, Section: Some Special Permissions](http://linuxcommand.org/tlcl.php)
464 |
465 | [^shotts-ssh]: [The Linux Command Line 2024, Chapter: 16, Section: Secure Communication](http://linuxcommand.org/tlcl.php)
466 |
467 |
468 |
--------------------------------------------------------------------------------
/chapter/03-basic-terminal.md:
--------------------------------------------------------------------------------
1 |
2 |
3 | > [!TIP]
4 | > The table of contents can be accessed by pressing the unordered list icon  on top the right corner.
5 |
6 |
11 |
12 |
13 |
14 | # Chapter 3 - Commandline crash course
15 |
16 | Get up to speed quickly with this intensive overview, designed to teach you the basics of commandline usage in a short amount of time. More advanced essentials will be discussed later in [Chapter 7](07-advanced-terminal.md#chapter-7---advanced-terminal-usage).
17 |
18 |
21 |
22 |
23 |
24 | ## 3.1 Clear the screen
25 |
26 | - With scrollback-ability = **` $ clear -x↵ `**
27 | - Keyboard shortcut **` Ctrl + L `** does the same thing.
28 | - Without scrollback-ability = **` $ reset↵ `**
29 | - This will actually reset and reinitialize the whole terminal state.
30 |
31 |
32 |
33 | ## 3.2 Special character: dash (-)
34 |
35 | > [!NOTE]
36 | > In typography and computing there are [different types of dash characters](https://en.wikipedia.org/wiki/Dash#Unicode_dash_characters). Here dash refers to the normal short hyphen ` - ` with ASCII code ` \x2D `, which is also the most commonly used form of hyphen in digital documents. On most keyboards, it is the only character that resembles a minus sign or a dash, so it is also used for these.
37 |
38 | ### Command-line argument
39 |
40 | The whole command-line is considered as a space delimited list of **command-line arguments**. The first argument is the filename of the executable, but most of the arguments are additional instructions that affect the operation of the program. It would be up to the program code to decipher each token and decide what it is and how to work with it.
41 |
42 | ### Dash as an introducer for command-line options
43 |
44 | - **Double dash ` --* `** is used to specify one single command-line option.
45 | - **Single dash ` -? `** is used to specify one or more command-line options.
46 |
47 | > [!NOTE]
48 | > Some command-line options require a value directly following them, often referred to as a **parameter**. These are typically options that configure something specific such as input or output file paths, as shown in the example below; which generates an eBook from markup and cover art:
49 | >
50 | > ```
51 | > $ pandoc input.md -o output.epub --epub-cover-image=cover-page.jpg↵
52 | > ```
53 |
54 | ### Dash as a reference to standard input
55 |
56 |
57 |
58 | Dash can also indicate standard streams (see [Chapter 6, Section: Standard streams](06-inter.md#standard-streams)). Many tools accept a **bare dash ` ␣-␣ `**, not associated with any option letter, as a pseudo-filename directing the application to read from standard input. And some programs, that print to a file by default, can be made to print to the standard output by specifying **bare dash ` ␣-␣ `** as the filename.
59 |
60 | ### Doubledash as end of options marker
61 |
62 | It is conventional to recognize a **bare double dash ` ␣--␣ `**, not associated with any option word, as a signal to stop option interpretation and treat all following arguments literally.
63 |
64 | > [!TIP]
65 | > If for example you need to process a file whose name starts with one or more dash characters, you can put an additional double dash ` -- ` argument before the filename to signify the end of the options. Alternatively, a relative reference to current working directory `./ ` can be added in front of the filename. Such convention is respected by most shell commands:
66 | >
67 | > ```
68 | > $ rm -- "--hard to remove.txt"↵ # Works
69 | > $ rm ./"--hard to remove.txt"↵ # Works
70 | > ```
71 |
72 | > [!WARNING]
73 | > Single quoting ` '...' ` or double quoting ` "..." ` is not enough:
74 | >
75 | > ```
76 | > $ rm "--hard to remove.txt"↵ # Does not work
77 | > rm: unrecognized option '--hard to remove.txt'
78 | > Try 'rm ./'--hard to remove.txt'' to remove the file
79 | > ```
80 | >
81 | > And on the other hand, quotation marks would be needed if the filename contains special characters such as spaces:
82 | >
83 | > ```
84 | > $ rm -- --hard to remove.txt↵ # Does not work
85 | > rm: cannot remove '--hard': No such file or directory
86 | > rm: cannot remove 'to': No such file or directory
87 | > rm: cannot remove 'remove.txt': No such file or directory
88 | > ```
89 |
90 |
91 |
92 | ## 3.3 Command-line options
93 |
94 | Three conventions for command-line options exist
95 |
96 | 1. **Unix style** **` -? `** [^raymond-cmopt]
97 | - In the original unix tradition, command-line options are single letters preceded by a single hyphen. Mode-flag options that do not take following arguments can be ganged together. Thus, if ` -a ` and ` -b ` are mode options then ` -ab ` or ` -ba ` is also correct and enables both.
98 | - A parameter to an option, if any, follows it, optionally separated by whitespace, such as ` -i input.odt -o output.pdf `.
99 | - In this style, lowercase options are preferred to uppercase. When uppercase options are used, it's good form for them to be special variants of the lowercase option.
100 | 2. **GNU style** **` --* `** [^raymond-cmopt]
101 | - The GNU style uses option keywords (rather than keyword letters) preceded by two hyphens. GNU options are easier to read than the alphabet soup of older styles. GNU-style options cannot be ganged together without separating whitespace.
102 | - A parameter to an option, if any, can be separated by either whitespace or a single equal sign ` = `.
103 | - Even with programs using the GNU style, it is good practice to support single-letter equivalents for at least the most common options. The GNU project recommends [conventional meanings](https://www.gnu.org/prep/standards/standards.html#Option-Table) for a few double-dash options in the GNU coding standards. It also lists [long options](https://www.gnu.org/prep/standards/standards.html#Option-Table) which, though not standardized, are used in many GNU programs.
104 | 3. **X toolkit style** **` -* `** (not recommended) [^raymond-cmopt]
105 | - The X toolkit style, uses a single hyphen and keyword options. The X toolkit style is not properly compatible with either the classic Unix or GNU styles, and should not be used in new programs.
106 |
107 | ### Commandline option: -h
108 |
109 | > [!NOTE]
110 | > The commandline option ` --help ` will display a help message and exit. It is not to be confused with ` -h `. The plain ` -h ` commandline option without an argument traditionally meant headers and not help.
111 |
112 | Support for help-option is actually less common than one might expect offhand. For much of Unix's early history, developers tended to think of "on-line" help as memory-footprint overhead they could not afford. Instead they wrote manual pages (see [Section: Early unix documentation style](#early-unix-documentation-style)).
113 |
114 | Commandline option for headers enable, suppress, or modify **headers** on a tabular report generated by the program (such as ` $ ps -h↵ `).
115 |
116 | Option ` -h ` may also have other meanings such as formatting filesize to be **human readable**. The command ` $ ls -la -h↵ ` is a good example of this convention.
117 |
118 |
119 |
120 | ## 3.4 Navigating directories
121 |
122 | > [!IMPORTANT]
123 | > Those used to Windows may be annoyed that directory paths do not work as expected on unix-like systems. Namely, **unix like systems use forward slash ` / ` to separate directories**. Whereas Windows uses backslash ` \ ` to separate directories.
124 |
125 |
131 |
132 | ### Current directory
133 |
134 | Absolute pathname of the current working directory:
135 | - **` $ pwd↵ `** = Print current working directory [^ieee-binpwd]
136 | - **` $ echo $PWD↵ `** = Present working directory [^ieee-envpwd]
137 | - The value is is initialized by ` $ sh ` and then updated by ` $ cd `.
138 |
139 | [^ieee-binpwd]: [IEEE 1003.1 Standard: Utilities: pwd, accessed 2025](https://pubs.opengroup.org/onlinepubs/9799919799/utilities/pwd.html)
140 |
141 | [^ieee-envpwd]: [IEEE 1003.1 Standard: Other Environment Variables, accessed 2025](https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1_chap08.html)
142 |
143 | ### Change directory
144 |
145 | - **` $ cd ↵ `** = Change directory
146 | - **` $ cd↵ `** (on its own) will take you to your home directory.
147 | - **` $ cd ~↵ `** will take you to your home directory.
148 | - **` $ cd $HOME↵ `** will also take you to your home directory.
149 | - **` $ cd -↵ `** will take you to the previous directory.
150 | - **` ~ `** tilde refers to current user's home directory.
151 | - **` ~matt `** is a way to refer to the home directory of another, named user.
152 | - **` ../../ `** two points move the index backwards (relative reference).
153 | - **` ./ `** point is a relative reference to current working directory.
154 | - **` / `** forward slash is the root directory (absolute reference)
155 | - This is where the directory structure begins.
156 |
157 | ### Make directory
158 |
159 | - **` $ mkdir ↵ `** = Make directory
160 | - Multiple directories can be created at once: ` $ mkdir dir1 dir2 dir3↵ `
161 | - **` -p `** = Nest subdirectories: ` $ mkdir -p create/nested/dirs↵ `
162 |
163 | ### List files
164 |
165 | - **` $ ls ↵ `** = List directory contents
166 | - Current working directory is used if not otherwise specified.
167 | - **` -a `** = List also hidden files.
168 | - **` -R `** = List also files in subdirectories.
169 | - **` -l `** = Show additional information such as filesize.
170 |
171 | > [!NOTE]
172 | > There are no headers on the output of the ` $ ls -l↵ ` command because of a unix tradition. Unix-style command-line tools typically avoid adding cosmetic information and formatting that would complicate further programmatic processing of the output (see [Chapter 2, Section: Design tropes of unix shell utilities](02-basic.md#design-tropes-of-unix)).
173 |
174 | > [!TIP]
175 | > One can install alternative listing commands that support column headers such as ` $ exa `. The command ` $ exa -lh↵ ` will lists the entire contents of the current working directory with additional column headers.
176 |
177 |
178 |
179 | ## 3.5 Finding executables
180 |
181 | The unix tooling comes with many commands for finding locations and information about files such as executables. The functionality is partly similar to the Properties window in Microsoft Windows.
182 |
183 | 1. **` $ ls -li ↵ `** provides additional information about the desired file.
184 | 1. **` $ stat ↵ `** is short for status reveals and reveals all that the system understands about a file and its attributes. It is a kind of souped-up version of the ` $ ls -li ↵ ` command.
185 | 1. **` $ files ↵ `** will try to determine the filetype (see [Section: File types and extensions](#file-types-and-file-name-extensions)).
186 | 1. **` $ whatis ↵ `** displays the description line from the man-page matching the specified program. Note that bash shell builtin commands don't have separate man-pages (see [Section: Shell builtin command](#shell-builtin-commands)).
187 | 1. **` $ whereis ↵ `** locates file paths to binary, data and man directories of the program, searching for many different possibly useful files related to the program.
188 | 1. **` $ which ↵ `** locates the entire file path of a program, by searching through the directories specified by the environment variable ` PATH `. ` $ which ` only works for executable programs, not builtins nor aliases (that are substitutes for actual executable programs).
189 | 1. Package managers are able to list a lot of information about software installed through them, such as ` $ eopkg info -F firefox↵ `.
190 |
191 | ```
192 | $ whatis whatis whereis which↵
193 | whatis - Display one-line manual page descriptions
194 | whereis - Locate the binary, source, and manual page files of commands
195 | which - Show the full path of shell commands
196 |
197 | $ whereis firefox↵
198 | /usr/bin/firefox
199 | /usr/lib64/firefox
200 | /usr/share/man/man1/firefox.1
201 |
202 | $ which firefox↵
203 | /usr/bin/firefox
204 |
205 | $ file HelloWorld.gambas↵
206 | HelloWorld.gambas: a gbr3 script executable (binary data)
207 | ```
208 |
209 |
210 |
211 | ## 3.6 Special character: period (.)
212 |
213 | ### Period as mark to hide files
214 |
215 | > [!IMPORTANT]
216 | > The unix way of implementing hidden files is likely to confuse those accustomed to Windows.
217 |
218 | Filenames that begin with a period **` . `** are hidden. This means that a file manager (such as ` $ nemo `) or the ` $ ls ` command will not list them unless you say so.
219 |
220 | When your account was created, several hidden files were placed in your home directory to configure things. The purpose of having these as hidden files is to reduce visible clutter on the filesystem by removing system and configuration files from view when they are not necessary.
221 |
222 | - To list hidden files and directories on command-line type ` $ ls -la↵ `
223 |
224 | - To temporarily reveal or conceal hidden files and directories in a graphical file manager
225 | a) Toggle ` Ctrl + H ` on your keyboard or
226 | b) Toggle ` Menu bar ` > ` View ` > ` Show hidden files `
227 |
228 | To permanently reveal or conceal a file or a directory, simply rename it.
229 | - To permanently hide a file such as ` sometext.txt ` rename it to ` .sometext.txt `.
230 | - To permanently unhide a file such as ` .sometext.txt ` rename it to ` sometext.txt `.
231 |
232 | ### Period as mark for current working directory
233 |
234 | > [!IMPORTANT]
235 | > Period ` . ` also refers to the current working directory (see [Section: Navigating directories](#navigating-directories)).
236 |
237 | You may have tried running a script or some other executable in the current working directory ` $ myscript.sh↵ ` only to end up with a *command not found* error. This happens because, by default the shell searches for an executable that matches with the given command only in the directories specified by the environment variable ` PATH `(see [Chapter 2, Section: Well-known environment variables](02-basic.md#well-known-variables)).
238 |
239 | The location of an estranged executable must be specified explicitly in connection with the executable. To run a script in the current working directory one must type something like **` $ ./myscript.sh↵ `**. Where the single period ` . ` refers to the active directory. The notation is not as confusing if the executable is called from another directory such as ` $ myscripts/myscript.sh↵ `
240 |
241 |
242 |
243 | ## 3.7 Special character: backslash (\\)
244 |
245 | > [!IMPORTANT]
246 | > The backslash ` \ ` is an **escape character**, and changes the meaning of the symbol that follows it.
247 |
248 | **Backslash ` \ `** escaping is used to:
249 |
250 | 1. Eliminate the special function of a character that follows it such as:
251 | - ` \$ ` = dollar sign ` $ `
252 | - ` \" ` = double quotes ` " `
253 | - ` \* ` = asterisk ` * `
254 | - ` \\ ` = back slash ` \ `
255 | - ` \ ` = space ` `
256 | - Such as cd into ` Directory with space characters `
257 | - ` $ cd Directory\ with\ space\ characters/↵ `
258 | 1. Insert symbols that could not be condensed on the keyboard, such as:
259 | - ` \u???? ` = a [Unicode-symbol](https://en.wikipedia.org/wiki/List_of_Unicode_characters) by its hexadecimal value (base 16).
260 | - ` \x?? ` = an [ASCII-symbol](https://www.ascii-code.com) by its hexadecimal value (base 16).
261 | - ` \??? ` = an [ASCII-symbol](https://www.ascii-code.com) by its octal value (base 8).
262 | 1. Insert characters that do not have a special symbol when printed, such as :
263 | - ` \t ` = horizontal tab
264 | - ` \v ` = vertical tab
265 | - ` \n ` = newline
266 | 1. Insert control codes which do not appear even indirectly, such as:
267 | - ` \a ` = bell or a beep (see note below)
268 |
269 | > [!NOTE]
270 | > The **bell code** or **bell character** is a device-control code, originally sent to ring a small electro-mechanical bell. Later on, speakers and buzzers vere equipped to perform the same function.
271 | >
272 | > Modern terminal emulators often integrate the warnings to the desktop environment and migth offer a **silent visual bell feature** that flashes the terminal window briefly.
273 | >
274 | > In ASCII and Unicode the character with the value 7 is BEL. It can be referred to by Keyboard shortcut ` Ctrl + V, Ctrl + G `, which is ^G in caret notation. Some keyboards even had the G-key located with a bell label. [^unibel]
275 |
276 | [^unibel]: [Unipedia Wiki - Bell, accessed 2021](https://unipedia.fandom.com/wiki/Bell)
277 |
278 |
279 |
280 | ## 3.8 Control characters
281 |
282 | ### Relic from history
283 |
284 | > [!IMPORTANT]
285 | > The basic ASCII character set (American Standard Code for Information Interchange) includes **control characters**, which usually have no visible representation. Control characters are used to cause effects, other than the insertion of a symbol into text. The purpose is to initiate some action that controls the hardware or data processing. [^wiki-control]
286 |
287 | [^wiki-control]: [Wikipedia - Control character, accessed 2024](https://en.wikipedia.org/wiki/Control_character)
288 |
289 | Control characters are a relic from history when computers were primitive. The ASCII character set is more than a list of symbols. It was developed in the 1960s as a notation and control coding for paperless typewriters and computer terminals. The ASCII control character set is a way of communicating between a terminal and a computer. [^wiki-ascii-fi] The ASCII standard has 33 control characters that can be described as doing something when the user enters them, such as
290 | - Interrupting an ongoing process with the end-of-text character (ETX, ^C, ` Ctrl + C `).
291 | - Indicating conclusion of text input, or exiting a shell by the end-of-transmission character (EOT, ^D, ` Ctrl + D `).
292 |
293 | [^wiki-ascii-fi]: [Wikipedia - ASCII, accessed 2025](https://fi.wikipedia.org/wiki/ASCII)
294 |
295 | ### Caret notation
296 |
297 | **Caret notation** is a printable form for control characters specifically in [ASCII](https://en.wikipedia.org/wiki/ASCII), where the character (^) known as the circumflex, appears as an independent character, without any letter below it.
298 | - The notation assigns ^A to control-code 1, sequentially through the alphabet to ^Z assigned to control-code 26. The keys that produce control-codes outside of the range 1...26 vary between systems. Keyboards also typically have a few single keys which produce control character codes (such as ` Del `, ` Backspace `, ` Tab `, ` Enter `). [^wiki-caret]
299 | - Control characters generated using letter keys are displayed with the upper-case form of the letter. When the Ctrl-key is held down, letter keys produce the same control characters regardless of the state of the Shift- or Caps-keys. It does not matter, whether the key would have produced an upper-case or a lower-case letter.
300 | - There are also other ways to display these non-printing "characters" such as decimal code point, hexadecimal code point, or a two or three letter abbreviation in capital letters.
301 |
302 | ### Traditional ASCII control characters
303 |
304 | [^wiki-caret]: [Wikipedia - Caret notation, accessed 2024](https://en.wikipedia.org/wiki/Caret_notation)
305 |
306 | | Dec | Hex | Abr | CN | Keyboard shortcut + Use |
307 | |:---:|:---:|:---:|:---:|:--- |
308 | | 0 | 00 | NUL | | **Null** was typically used to reserve space, either for correcting errors or for inserting information that would be available at a later time or in another place. It adapted to mark end of string. |
309 | | 1 | 01 | SOH | `^A` | **Start of heading** was to mark a non-data section of a data stream. |
310 | | 2 | 02 | STX | `^B` | **Start of text** marked the end of the header, and the start of the textual part of a stream. |
311 | | 3 | 03 | ETX | `^C` | ` Ctrl + C ` **End of text** marked the end of the data of a message. |
312 | | 4 | 04 | EOT | `^D` | ` Ctrl + D ` **End of transmission** was used to indicate that a slave station has completed its transmission, i.e. the end of an entire transmission or communication session to signal that no more data will follow. |
313 | | 5 | 05 | ENQ | `^E` | When a transmission medium could transmit in only one direction at a time, **Enquire** was used by a master station to ask a slave station to send its next message. |
314 | | 6 | 06 | ACK | `^F` | **Acknowledgment** was normally used as a flag to indicate no problem detected with current element. |
315 | | 7 | 07 | BEL | `^G` | ` Ctrl + G ` **Bell** is intended to cause an audible signal in the receiving terminal. |
316 | | 8 | 08 | BS | `^H` | ` Ctrl + H ` or ** Backspace ** moves the printing position one character space backwards. On printers, including hard-copy terminals, this is most often used so the printer can overprint characters to make other, not normally available, characters. On video terminals and other electronic output devices, there are often software (or hardware) configuration choices that allow a destructive backspace. |
317 | | 9 | 09 | HT | `^I` | ` Ctrl + V, Ctrl + I ` **Horizontal tab** cause the output device to move the printing position to the next tab stop in the direction of reading. Tab derives from the word tabulate, which means: to arrange data in a tabular, or table, form. When a person wanted to type a table (of numbers or text) on a typewriter, there was a lot of time-consuming and repetitive use of the space bar and backspace key. To simplify this, a horizontal bar was placed in the mechanism called the tabulator rack. Pressing the tab key would advance the carriage to the next tabulator stop. [^wiki-tab] |
318 | | 10 | 0A | LF | `^J` | ` Ctrl + J ` **Line feed** causes the device to put the printing position on the next line. In modern systems it also moves the printing position to the start of the line (which would be the leftmost position for left-to-right scripts, such as the alphabets used for western languages, and the rightmost position for right-to-left scripts such as the hebrew and arabic alphabets). |
319 | | 11 | 0B | VT | `^K` | **Vertical tab** was used to speed up printer vertical movement. Some printers used special tab belts with various tab spots. |
320 | | 12 | 0C | FF | `^L` | ` Ctrl + L ` **Form feed** started a new sheet of paper. With the advent of computer terminals that did not physically print on paper and so offered more flexibility regarding screen placement, erasure, and so forth, printing control codes were adapted. Form feeds, for example, usually cleared the screen, there being no new paper page to move to. |
321 | | 13 | 0D | CR | `^M` | ` Ctrl + M ` or ` ↵ ` **Carriage return** causes the device to put the caret at the edge of the paper at which writing begins (which would be the leftmost position for left-to-right scripts, such as the alphabets used for western languages, and the rightmost position for right-to-left scripts such as the hebrew and arabic alphabets). In modern systems it also moves the printing position to the next line. |
322 | | 14 15 | 0E 0F | SO SI | `^N` `^O` | The original purpose of ` Ctrl + N ` **Shift out** and ` Ctrl + O ` **shift in** control codes was to provide a way to shift the red and black ribbon, i.e. split on an electromechanical typewriter or teleprinter, in or out, to another colour. As the technology developed, the function adapted to switching to a different font or character set. |
323 | | 16 | 10 | DLE | `^P` | **Data link escape** was intended to be a signal to the other end of a data link that the following character is a control character such as STX or ETX. |
324 | | 17 18 19 20 | 11 12 13 14 | DC1 DC2 DC3 DC4 | `^Q` `^R` `^S` `^T` | **Device control** codes, DC1 to DC4, were originally generic, to be implemented as necessary by each device. |
325 | | 21 | 15 | NAK | `^U` | **Negative acknowledge** is a definite flag for, usually, noting that reception was a problem, and, often, that the current element should be sent again. |
326 | | 22 | 16 | SYN | `^V` | **Synchronous idle** was originally sent by synchronous modems (which have to send data constantly) when there was no actual data to send. |
327 | | 23 | 17 | ETB | `^W`| **End of transmission block** used to signify the end of a block of data during transmission. It was employed in protocols where data was sent in blocks. |
328 | | 24 | 18 | CAN | `^X` | **Cancel** signals that the previous element should be discarded. |
329 | | 25 | 19 | EM | `^Y` | **End of medium** signal warned that a recording medium such as tape had run out. |
330 | | 26 | 1A | SUB | `^Z` | **Substitute** was used to padding, so that data could be sent in fixed size blocks or in place of a character that could not be represented on another device. |
331 | | 27 | 1B | ESC | `^[` | **Escape sequences** were used to trigger a wide variety of instructions to control the behaviour of the terminal, such as moving the cursor, formatting text, switching LED lights and clearing the screen (see [VT100 User Guide - Commands and control sequences](http://www.braun-home.net/michael/info/misc/VT100_commands.htm)). |
332 |
333 | [^wiki-tab]: [Wikipedia - Tab key, accessed 2025](https://en.wikipedia.org/wiki/Tab_key)
334 |
335 | ### Inserting a control character
336 |
337 | > [!TIP]
338 | > Special characters can be inserted through **verbatim insert**.
339 |
340 | The bell control character **` BEL \a \x07 `** is suitable for illustrating *verbatim insert* functionality. Pressing the shortcut keys corresponding to the bell symbol ` Ctrl + G `, in a terminal window, generates the bell sound. Alternatively ` $ echo ` can be used to generate the sound of the bell character:
341 | 1. Type the initial part of the command **` $ echo␣ `**; That is "echo " with a trailing space.
342 | 2. press the keyboard shortcut keys **` Ctrl + V, Ctrl + G `** and the character ^G appears on the line. If you delete the character with the ` Backspace ` key, you will see that it is really a single character and not ^ and G in a row.
343 | 3. Finally, press enter ` ↵ ` and you should hear a beep (also a blank line will be printed). If you don't hear a beep and the text ^G is printed, something went wrong. Functionality is not correct if you type the command manually ` $ echo ^G↵ `. The bell character should be inserted by pressing ` Ctrl + V, Ctrl + G `.
344 |
345 | > [!NOTE]
346 | > Of course, there are alternative ways to achieve the same functionality, such as ` $ echo -e "\x07"↵ ` (see [Section: Special character: backslash (\\)](#special-character-backslash)).
347 |
348 |
349 |
350 | ## 3.9 Special character: space ( )
351 |
352 | > [!WARNING]
353 | > Files systems that are popular in GNU/Linux such as EXT4 and BTRFS have far fewer restrictions on special characters than the Windows NTFS and exFAT file systems. Every character except forward slash ` / ` and NUL ` \0 `, are allowed in filenames. However if you make any use of the shell, you will realize that there are many characters that will create a hassle; most significantly space ` `, dash ` - `, asterisk ` * `, dollar sign ` $ `, exlamation mark ` ! `, ampersand ` & `, forwardslash ` / `, backslash ` \ `, curly brackets ` {...} `, tilde ` ~ `, at sign ` @ `, backquotes `` `...` ``.
354 |
355 | ### Recommendation to not use space in filenames
356 |
357 | In the command interface, **the whole command-line is considered a space-separated list of command-line arguments** (see [Section: Special character: dash (-)](#special-character-dash--)).
358 | - This is why a whitespace in the filename creates the need to use either:
359 | a) Double quote supression ` "..." `
360 | b) Single quote supression ` '...' `
361 | c) Backslash escaping ` \? `
362 |
363 | ```
364 | $ rm "A Dragged Link from Firefox.desktop"↵
365 |
366 | $ rm 'A Dragged Link from Firefox.desktop'↵
367 |
368 | $ rm A\ Dragged\ Link\ from\ Firefox.desktop↵
369 | ```
370 |
371 | > [!NOTE]
372 | > The recommendation to not use spaces in filenames comes from the danger that **they might be misinterpreted by software that poorly supports them**. Arguably, such software is buggy. But also arguably, programming languages and shell scripting make it all too easy to write software that breaks when presented with filenames with spaces in them.
373 |
374 | ### Types of documentation
375 |
376 | #### Early unix documentation style
377 |
378 | The early Unix documentation style and tools have several technical and cultural traits that distinguish it from the way documentation is done elsewhere. In the early history of Unix, developers produced manual pages containing essential practical information. These were stored in the ` /usr/share/man/ ` directory and were accessible by the ` $man ` command, an interface to the system reference manuals. [^raymond-docstyle]
379 | - The classic manual pages that came with Unix have traditionally been written by programmers for their peers. The style does not hold you by the hand, but it usually points in the right direction. The style assumes an active and experienced reader. This is because Unix and its decendants are in many ways better adapted to the needs of power users. [^raymond-docstyle]
380 | - Many times the manual page of a unix program even has a section called BUGS. In other cultures, technical writers try to make the product look good by omitting and skating over known bugs. In the unix culture, peers describe the known shortcomings of their software to each other in unsparing detail, and users consider a short but informative BUGS section to be an encouraging sign of quality work. Commercial Unix distributions euphemize it to a softer tag like LIMITATIONS or ISSUES or APPLICATION USAGE. [^raymond-docstyle]
381 |
382 | #### Modern documentation style
383 |
384 | The best practices for writing documentation on modern unix program of any significant size suggest shipping three different kinds of documentation: [^raymond-bestdoc]
385 | - **Man-pages** should be command references for the traditional unix audience, in the traditional unix style, giving details of how the program is invoked, a **quick summary** and pointers to the long-form manual incase the user needs to learn more about a specific feature or function. Huge man pages are viewed with some disfavor, and navigation within them can be difficult. All man pages follow a [common layout](https://en.wikipedia.org/wiki/Man_page) that is optimized for presentation on a simple ASCII text display, possibly without any form of highlighting or font control. [^raymond-bestdoc]
386 | - The **long-form manual** should be a comprehensive documentation for nontechnical users. And often includes troubleshooting tips. It should be possible to use the long-form manual as a reference guide when the user encounters difficulties or needs to learn more about a specific feature or function. [^raymond-bestdoc]
387 | - List of **frequently asked questions** (FAQ) should be an evolving resource that grows as the software support group learns what the frequent questions are and how to answer them. [^raymond-bestdoc]
388 |
389 | #### In addition
390 |
391 | - A project should have a **website** to serve as a central point of distribution. [^raymond-bestdoc]
392 |
393 | - Extra **tutorials** can take many forms such as written articles, videos, or interactive software. A tutorial is intended to teach users how to perform a specific task or achieve a particular goal. A tutorial may include quizzes or other types of assessments to help the user gauge their understanding of the material. [^raymond-bestdoc]
394 |
395 | - With the source code, you can also expect to find a **README file**, which usually contains, installation instructions, configuration instructions, reference to user manuals, copyright and licence information, and contact details of the distributor or author. [^wiki-readme]
396 |
397 | [^wiki-readme]: [Wikipedia - README, accessed 2024](https://en.wikipedia.org/wiki/README)
398 |
399 |
400 |
401 | ## 3.10 File types and file name extensions
402 |
403 | **Extensions** are file name suffixes that start with a period. Usually, they are two, three or four letters long. Extensions are used to distinguish between different types of files. For example, ` *.txt ` refers to a text file and ` *.jpg ` refers to an image file. [^wiki-tiedostopääte]
404 |
405 | [^wiki-tiedostopääte]: [Wikipedia - Tiedostopääte, accessed 2025](https://fi.wikipedia.org/wiki/Tiedostop%C3%A4%C3%A4te)
406 |
407 | ### Unix tradition has no concept of file name extension
408 |
409 | Windows relies heavily on file name extensions to decide what to do with a file if it is double-clicked. In the unix tradition, on the other hand, there is no concept of a file name extension. Although file extensions may be used in modern GNU/Linux distributions, most utilities and applications still ignore the file extension. Instead, various heuristics are used to determine the file type. [^raymond-bag]
410 |
411 | Even the early Apple Macintosh computers did not use file name extensions. Instead the operating system stored information about the program that created the file in the file, in order to be able to launch the file later with the same program. In the unix tradition, files did not contain such metadata either. [^raymond-bag]
412 |
413 | Unix partisans preferred approaches that made file data self-describing, so that no metadata was needed to store within the file. Traditionally also the directory, where a file was placed, has been a good indication of the file type (see [Chapter 2, Section: Directory structure](02-basic.md#directory-structure)). [^raymond-bag]
414 |
415 | ### Even today filename name extensions are not required
416 |
417 | The visual metaphor at the heart of modern GUIs, files represented by icons, and opened by clicking which invokes some designated handler program, typically able to create and edit these files, has proven both successful and long-lived. [^raymond-weak]
418 | - Unix long supported this metaphor only poorly and grudgingly. [^raymond-weak]
419 |
420 | Even today, the filename name extensions are not required in the GNU/Linux desktop environment to determine the contents and purpose of files. For example, pdf- and odt-files get the right icon, and open in the right program, without a file name extension. [^raymond-weak]
421 | - However, many applications can use a file name extension, and a file named with the wrong extension will no longer open correctly when clicked with the mouse. [^raymond-weak]
422 |
423 |
424 |
425 |
426 |
427 | ### Command to find out the file type
428 |
429 | In the command line interface, the contents and/or purpose of a file is, still today, determined mostly by other means such as **` $ file ↵ `**:
430 |
431 | ```
432 | $ file SomePlainText↵
433 | SomePlainText: Unicode text, UTF-8 text
434 |
435 | $ file SomeTextDocument↵
436 | SomeTextDocument: OpenDocument Text
437 |
438 | $ file the_beginners_handbook↵
439 | the_beginners_handbook: PDF document, version 1.5 (zip deflate encoded)
440 |
441 | $ file ~/Desktop/↵
442 | /home/user/Desktop/: directory
443 |
444 | $ file /sbin↵
445 | /sbin: symbolic link to usr/sbin
446 |
447 | $ file /dev/null↵
448 | /dev/null: character special (1/3)
449 |
450 | $ file HelloWorld.gambas↵
451 | HelloWorld.gambas: a gbr3 script executable (binary data)
452 |
453 | $ file /usr/bin/file↵
454 | /usr/bin/file: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /usr/lib64/ld-linux-x86-64.so.2, BuildID[sha1]=96eacae9627debefb5fa23ed58cf1ad83251f85c, for GNU/Linux 3.2.0, stripped
455 | ```
456 |
457 |
458 |
459 | > [!NOTE]
460 | > The file type **character special file** is an entity that handles data as a stream of bytes, such as ` /dev/null `.
461 | >
462 | > The file type **block special file** is an entity that handles data in blocks, such as an optical drive.
463 |
464 | ### Command types
465 |
466 | A command can be one of four different things:
467 |
468 | a) A reference to an [executable program](#executable), i.e. a binary file containing machine code for execution by the computer, such as ` $ ls ` and ` $ rm `.
469 | b) A reference to [an internal command found in the shell](#shell-builtin-commands) such as ` $ cd ` and ` $ echo `.
470 | c) A reference to a [script file](https://www.gnu.org/software/bash/manual/html_node/Shell-Scripts.html) written in the scripting language of the shell.
471 | d) [Alias](#alias), which creates an alternative name for another command.
472 |
473 | The shell builtin command **` $ type ↵ `** will show you how a specific command will be interpreted if used on the command line.
474 |
475 | ```
476 | $ type type↵
477 | type is a shell builtin
478 |
479 | $ type ls↵
480 | ls is aliased to `ls --color=auto'
481 |
482 | $ type file↵
483 | file is /usr/bin/file
484 |
485 | $ type file↵
486 | file is hashed (/usr/bin/file)
487 | ```
488 |
489 | > [!NOTE]
490 | > After finding the location of a command the first time, its location is remembered (i.e. hashed). The shell builtin ` $ hash ` is invoked and responsible of remembering and displaying program locations.
491 |
492 | ### Executable
493 |
494 | An executable program can be:
495 |
496 | a) An **executable binary file** written in a translatable programming language (such as C or C++), and compiled into machine code. Hence the name of the ` /usr/bin/ ` directory (binaries).
497 | b) An **executable text file** written in a scripting language (such as Shell, Python or Ruby). Scripts do not need to be compiled in order to be executed. They are interpreted line by line at runtime. Despite the name, not all executable programs in the directory ` /usr/bin/ ` are binary files. The directory also contains programs written in scripting languages, as proved by running the command:
498 |
499 | ```
500 | $ file /usr/bin/* | cut -c 1-110 | awk '{print $0}' | sort -k2↵
501 | ```
502 |
503 | #### User-guides for program-files
504 |
505 | > [!TIP]
506 | > Many programs can print out a short instruction, when invoked with the command line option requesting for help ` $ --help↵ ` such as:
507 | >
508 | > ```
509 | > $ firefox --help↵
510 | > Usage: /usr/lib64/firefox/firefox [ options ... ] [URL]
511 | > where options include:
512 | >
513 | > X11 options
514 | > --display=DISPLAY X display to use
515 | > --sync Make X calls synchronous
516 | > --g-fatal-warnings Make all warnings fatal
517 | >
518 | > Firefox options
519 | > -h or --help Print this message.
520 | > -v or --version Print Firefox version.
521 | > ...
522 | > ```
523 |
524 | > [!TIP]
525 | > Many programs come with man-pages, a type of documentation stored in the ` /usr/share/man/ ` directory. The ` $man ↵ ` program is an interface to these manuals, such as
526 | >
527 | > ```
528 | > $ man firefox↵
529 | > FIREFOX(1) Linux User's Manual FIREFOX(1)
530 | >
531 | > NAME
532 | > firefox - a Web browser for X11 derived from the
533 | > Mozilla browser
534 | >
535 | > SYNOPSIS
536 | > firefox [OPTIONS ...] [URL]
537 | >
538 | > firefox-bin [OPTIONS] [URL]
539 | >
540 | > DESCRIPTION
541 | > Mozilla Firefox is an open-source web browser,
542 | > designed for standards compliance, performance
543 | > and portability.
544 | > ...
545 | > ```
546 |
547 |
548 |
549 | ### Shell builtin commands
550 |
551 | Some commands are built into the shell. These **shell builtins** are run directly in the shell, and not as external programs. They are built-in because they are such an essential part of the shell's operation, such as ` $ alias `, ` $ unalias `, ` $ echo `, ` $ printf `, ` $ logout `, ` $ cd `, ` $ pwd ` and ` $ kill `.
552 |
553 | > [!TIP]
554 | > Use ` $ compgen -b↵ ` to get a list of the all the shell builtin commands.
555 |
556 | #### User-guides for shell-builtin-commands
557 |
558 | > [!TIP]
559 | > Instructions for shell-builtin-commands are printed by running ` $ help ↵ `. Help itself is a shell builtin command, and these instructions are built into the shell itself, because they are such an integral part of the shell:
560 | >
561 | > ```
562 | > $ bind --help↵
563 | > bind: usage: bind [-lpsvPSVX] [-m keymap] [-f filename] [-q name] [-u name] [-r keyseq] [-x keyseq:shell-command] [keyseq:readline-function or readline-> command]
564 | >
565 | > $ help bind↵
566 | > bind: bind [-lpsvPSVX] [-m keymap] [-f filename] [-q ...
567 | > Set Readline key bindings and variables.
568 | >
569 | > Bind a key sequence to a Readline function or a macro,
570 | > or set a Readline variable. The non-option argument
571 | > syntax is equivalent to that found in ~/.inputrc, but
572 | > must be passed as a single argument.
573 | >
574 | > Options:
575 | > -m keymap Use KEYMAP as the keymap for...
576 | > -l List names of functions.
577 | > -P List function names and bind...
578 | > -p List functions and bindings ...
579 | > -S List key sequences that invo...
580 | > -s List key sequences that invo...
581 | > -V List variable names and valu...
582 | > -v List variable names and valu...
583 | > ```
584 |
585 | > [!NOTE]
586 | > There are no man-pages for the shell builtin commands:
587 | >
588 | > ```
589 | > $ type bind↵
590 | > bind is a shell builtin
591 | >
592 | > $ whatis bind↵
593 | > bind: nothing appropriate.
594 | >
595 | > $ man bind↵
596 | > No manual entry for bind
597 | > ```
598 |
599 | ### Shell scripting
600 |
601 | The command-line is already versatile on its own, but more complex processes are easier to implement using scripts. In particular, it is worth saving frequently used command-sequences in a file. **Shell scripting** enables the use of algorithms and structures specific to programming languages (such as variables, conditional statements, loops and functions). Infact writing scripts for the shell to execute, can be referred to as **shell programming**.
602 | - Scripts do not need to be compiled in order to be executed. They are interpreted.
603 | - A script file is executed by calling it by the filename such as ` $ ./script.sh↵ `
604 | - A shell script file starts with the string ` #!/bin/bash `.
605 | - Scripts do not have to be saved to a file. They can also be written directly to the command line. In this case, a semicolon ` ; ` is placed between the commands.
606 | - In the Bash shell, variable names are preceded by the dollar sign ` $ `.
607 | - Commenting is important in any program. In shell programming, a comment is added as ` # comment `. Shell does not read lines that begin with a hash mark ` # `. Similarly, a hash mark at the beginning of a word will start a comment and cause the entire rest of the line to be ignored.
608 |
609 | > [!TIP]
610 | > Comments are used to enhance readability.
611 | >
612 | > You will also see lines in [configuration files](02-basic.md#configuration-files) that are commented out to prevent them from being used by the affected program. This is done to give the reader suggestions for possible configuration choices or examples of correct configuration syntax.
613 |
614 |
615 |
616 |
617 | ### Alias
618 |
619 | 1. **Alias** is a shell builtin command that creates an alternative name for another command. For example ` $ alias up="sudo eopkg up"↵ ` creates an alias named up (as in update).
620 |
621 | 2. The use of aliases makes working on the command-line easier and faster. For example, when invoking our newly created alias ` $ up↵ `, the shell executes the command specified for the alias, such as ` $ sudo eopkg up↵ `.
622 |
623 | 3. Calling ` $ alias↵ ` on its own will print a list of all available aliases, such as
624 |
625 | ```
626 | $ alias↵
627 | alias ls='ls –color=auto'
628 | alias up='sudo eopkg up'
629 | ```
630 |
631 | 4. Use unalias to remove a specific alias such as ` $ unalias up↵ `
632 |
633 | > [!TIP]
634 | > From time to time, you may need to use the original version of a command with the same name instead of a particular alias, by starting the command with a backward slash ` \ `. For example, the original monochrome version of ` $ \ls↵ ` is used instead of the alias ` $ ls -color=auto↵ `.
635 |
636 | > [!NOTE]
637 | > Aliases vanish as the shell session ends. To make an alias permanent add the alias creation command (such as ` alias up="sudo eopkg up" `) to a new line on ` ~/.bashrc `.
638 | >
639 | > Making the addition on commandline can be a bit tricky. But fortunately run control files can be modified using a graphical text editor. Invoke from commandline like this ` $ gedit ~/.bashrc↵ ` or this ` $ xed ~/.bashrc↵ ` depending on the default text editor provided by the operating system.
640 |
641 | ### Alternative for aliases
642 |
643 | > [!TIP]
644 | > The optional ` $ xdotool ` package (if available to your system) can be used to create an elegant alternative for traditional aliases. Where the command specified by the alias gets typed on the shell prompt instead of being silently executed:
645 | > ```
646 | > $ alias up='stty -echo ; xdotool type "sudo eopkg up" ; stty echo'↵
647 | > $ up↵
648 | > $ sudo eopkg up # This line was automatically generated!
649 | > ```
650 |
651 | - Using ` $ xdotool ` can be great if you need to fine tune the aliased command before it is executed or want a visual reminder of what is about to be executed. For new users this can prevent frustration of having to use the commandline and yeat build recollection for commands; As they are repeatedly shown to the user despite having to make the effort of typing them in and possibly making mistakes.
652 | - The list for xdotool key codes can be found [here](https://gitlab.com/nokun/gestures/-/wikis/xdotool-list-of-key-codes).
653 |
654 |
655 |
656 | ## 3.11 Autocomplete
657 |
658 | In a bash shell prompt, as you are typing a command, filename, directory or even command options, pressing the ` Tab ` key will either automatically complete what you were typing or show all the possible results for you. You may want to add to this behaviour, so it cycles through every possible suggestion with each repeated press on ` Tab `:
659 |
660 | 1. Run command ` $ gedit /etc/inputrc↵ ` or ` $ xed /etc/inputrc↵ ` depending on the default text editor provided by the operating system.
661 | 3. Add lines: [^stack-cycle]
662 |
663 | ```
664 | # Tweak auto completion
665 | set show-all-if-ambiguous on
666 | set show-all-if-unmodified on
667 | set menu-complete-display-prefix on
668 | "\t": menu-complete # Tab
669 | "\e[Z": menu-complete-backward # Shift + Tab
670 | ```
671 |
672 | 3. Save as ` ~/.inputrc `, a different location to what was opened.
673 | 4. Run command ` $ bind -f ~/.inputrc↵ ` [^stack-inputrc]
674 |
675 | > [!NOTE]
676 | > The configuration files ` /etc/inputrc ` and ` ~/.inputrc ` are part of the GNU Readline ‑library, which allows users to customize how keyboard input is interpreted and processed in the Bash command-line interface and in applications that use Readline. [^softpano-inputrc]
677 |
678 | [^stack-cycle]: [Stackexchange.com - Terminal autocomplete: cycle through suggestions, accessed 2025](https://unix.stackexchange.com/questions/24419/terminal-autocomplete-cycle-through-suggestions)
679 |
680 | [^stack-inputrc]: [Stackoverflow.com - Inputrc file cannot be loaded, accessed 2025](https://stackoverflow.com/questions/15027186/inputrc-file-cannot-be-loaded)
681 |
682 | [^softpano-inputrc]: [Softpanorama.org - Readline and inputrc, updated 2019](http://www.softpanorama.org/Scripting/Shellorama/Bash_as_command_interpreter/inputrc.shtml)
683 |
684 |
685 |
686 | ## 3.12 Some useful shortcuts
687 |
688 | New users easily encounter situations, in the command interface, from which there seems to be no way out. You may try the following to exit some commandline situations:
689 |
690 | | Keys | Use case |
691 | |:--- |:--- |
692 | | ` Ctrl + D ` | End text input |
693 | | ` Ctrl + C ` | Interrupt current process |
694 | | ` q ` | Quit ` $ less ` and ` $ more ` |
695 | | ` Ctrl + X ` | Exit ` $ nano ` |
696 | | ` Ctrl + X, Ctrl + C ` | Exit ` $ emacs ` |
697 | | ` Esc, :qa!↵ ` | Exit ` $ vi ` |
698 | | ` Alt + F4 ` | Close current window |
699 |
700 | The current command line can be cleared by other means than just pressing the ` Backspace `:
701 |
702 | | Keys | Use case |
703 | |:--- |:--- |
704 | | ` Ctrl + C ` | Cancel |
705 | | ` Ctrl + W ` | Delete just a word. |
706 | | ` Ctrl + U ` | Clear up to the beginning. |
707 | | ` Ctrl + L ` | Clear the terminal screen (with scrollback-ability). Typing ` $ clear -x↵ ` does the same thing. |
708 | | ` Alt + Shift + 3 ` | Turning current line to a comment ` # ` is especially useful if you want to keep and comment on bash history (see [Chapter 2, Section: Bash history](02-basic.md#bash-history)). |
709 |
710 | Shorcuts specific to GUI terminal window:
711 |
712 | | Keys | Use case |
713 | |:--- |:--- |
714 | | ` F11 ` | Toggle fullscreen |
715 | | ` Control + Shift + F ` | Search or find for text displayed on current terminal window. |
716 |
717 |
718 |
719 | [^raymond-cmopt]: [The Art of Unix Programming 2003, Chapter 10, Section: Command-Line Options](http://www.catb.org/~esr/writings/taoup/html/ch10s05.html)
720 |
721 | [^raymond-docstyle]: [The Art of Unix Programming 2003, Chapter 18, Section: The Unix Style Documentation](http://www.catb.org/~esr/writings/taoup/html/ch18s02.html)
722 |
723 | [^raymond-bestdoc]: [The Art of Unix Programming 2003, Chapter 18, Section: Best Practices for Documentation](http://www.catb.org/~esr/writings/taoup/html/ch18s06.html)
724 |
725 | [^raymond-bag]: [The Art of Unix Programming 2003, Chapter 20, Section: Just a Big Bag of Bytes](http://www.catb.org/~esr/writings/taoup/html/ch20s03.html)
726 |
727 | [^raymond-weak]: [The Art of Unix Programming 2003, Chapter 20, Section: Unix Support for GUIs](http://www.catb.org/~esr/writings/taoup/html/ch20s03.html)
728 |
729 |
730 |
--------------------------------------------------------------------------------