├── LICENSE ├── README.md ├── content ├── bash_programming.md ├── basic_commands.md ├── basic_networking.md ├── basic_security.md ├── cluster-adv.md ├── cluster-basics.md ├── command_line_prompt.md ├── data_backup.md ├── file_permissions.md ├── filesytem.md ├── network-namespaces.md ├── nfs.md ├── package_management.md ├── processes.md ├── samba_server.md ├── shared_storage_iscsi.md ├── swap_memory.md ├── system_info.md ├── systemd.md ├── text_commands.md ├── user_env.md ├── virtual-networking.md ├── volume_manager.md ├── volume_manager_cont.md └── working_with_files.md └── img └── active-active-cluster.jpg /LICENSE: -------------------------------------------------------------------------------- 1 | CC0 1.0 Universal 2 | 3 | Statement of Purpose 4 | 5 | The laws of most jurisdictions throughout the world automatically confer 6 | exclusive Copyright and Related Rights (defined below) upon the creator and 7 | subsequent owner(s) (each and all, an "owner") of an original work of 8 | authorship and/or a database (each, a "Work"). 9 | 10 | Certain owners wish to permanently relinquish those rights to a Work for the 11 | purpose of contributing to a commons of creative, cultural and scientific 12 | works ("Commons") that the public can reliably and without fear of later 13 | claims of infringement build upon, modify, incorporate in other works, reuse 14 | and redistribute as freely as possible in any form whatsoever and for any 15 | purposes, including without limitation commercial purposes. These owners may 16 | contribute to the Commons to promote the ideal of a free culture and the 17 | further production of creative, cultural and scientific works, or to gain 18 | reputation or greater distribution for their Work in part through the use and 19 | efforts of others. 20 | 21 | For these and/or other purposes and motivations, and without any expectation 22 | of additional consideration or compensation, the person associating CC0 with a 23 | Work (the "Affirmer"), to the extent that he or she is an owner of Copyright 24 | and Related Rights in the Work, voluntarily elects to apply CC0 to the Work 25 | and publicly distribute the Work under its terms, with knowledge of his or her 26 | Copyright and Related Rights in the Work and the meaning and intended legal 27 | effect of CC0 on those rights. 28 | 29 | 1. Copyright and Related Rights. A Work made available under CC0 may be 30 | protected by copyright and related or neighboring rights ("Copyright and 31 | Related Rights"). Copyright and Related Rights include, but are not limited 32 | to, the following: 33 | 34 | i. the right to reproduce, adapt, distribute, perform, display, communicate, 35 | and translate a Work; 36 | 37 | ii. moral rights retained by the original author(s) and/or performer(s); 38 | 39 | iii. publicity and privacy rights pertaining to a person's image or likeness 40 | depicted in a Work; 41 | 42 | iv. rights protecting against unfair competition in regards to a Work, 43 | subject to the limitations in paragraph 4(a), below; 44 | 45 | v. rights protecting the extraction, dissemination, use and reuse of data in 46 | a Work; 47 | 48 | vi. database rights (such as those arising under Directive 96/9/EC of the 49 | European Parliament and of the Council of 11 March 1996 on the legal 50 | protection of databases, and under any national implementation thereof, 51 | including any amended or successor version of such directive); and 52 | 53 | vii. other similar, equivalent or corresponding rights throughout the world 54 | based on applicable law or treaty, and any national implementations thereof. 55 | 56 | 2. Waiver. To the greatest extent permitted by, but not in contravention of, 57 | applicable law, Affirmer hereby overtly, fully, permanently, irrevocably and 58 | unconditionally waives, abandons, and surrenders all of Affirmer's Copyright 59 | and Related Rights and associated claims and causes of action, whether now 60 | known or unknown (including existing as well as future claims and causes of 61 | action), in the Work (i) in all territories worldwide, (ii) for the maximum 62 | duration provided by applicable law or treaty (including future time 63 | extensions), (iii) in any current or future medium and for any number of 64 | copies, and (iv) for any purpose whatsoever, including without limitation 65 | commercial, advertising or promotional purposes (the "Waiver"). Affirmer makes 66 | the Waiver for the benefit of each member of the public at large and to the 67 | detriment of Affirmer's heirs and successors, fully intending that such Waiver 68 | shall not be subject to revocation, rescission, cancellation, termination, or 69 | any other legal or equitable action to disrupt the quiet enjoyment of the Work 70 | by the public as contemplated by Affirmer's express Statement of Purpose. 71 | 72 | 3. Public License Fallback. Should any part of the Waiver for any reason be 73 | judged legally invalid or ineffective under applicable law, then the Waiver 74 | shall be preserved to the maximum extent permitted taking into account 75 | Affirmer's express Statement of Purpose. In addition, to the extent the Waiver 76 | is so judged Affirmer hereby grants to each affected person a royalty-free, 77 | non transferable, non sublicensable, non exclusive, irrevocable and 78 | unconditional license to exercise Affirmer's Copyright and Related Rights in 79 | the Work (i) in all territories worldwide, (ii) for the maximum duration 80 | provided by applicable law or treaty (including future time extensions), (iii) 81 | in any current or future medium and for any number of copies, and (iv) for any 82 | purpose whatsoever, including without limitation commercial, advertising or 83 | promotional purposes (the "License"). The License shall be deemed effective as 84 | of the date CC0 was applied by Affirmer to the Work. Should any part of the 85 | License for any reason be judged legally invalid or ineffective under 86 | applicable law, such partial invalidity or ineffectiveness shall not 87 | invalidate the remainder of the License, and in such case Affirmer hereby 88 | affirms that he or she will not (i) exercise any of his or her remaining 89 | Copyright and Related Rights in the Work or (ii) assert any associated claims 90 | and causes of action with respect to the Work, in either case contrary to 91 | Affirmer's express Statement of Purpose. 92 | 93 | 4. Limitations and Disclaimers. 94 | 95 | a. No trademark or patent rights held by Affirmer are waived, abandoned, 96 | surrendered, licensed or otherwise affected by this document. 97 | 98 | b. Affirmer offers the Work as-is and makes no representations or warranties 99 | of any kind concerning the Work, express, implied, statutory or otherwise, 100 | including without limitation warranties of title, merchantability, fitness 101 | for a particular purpose, non infringement, or the absence of latent or 102 | other defects, accuracy, or the present or absence of errors, whether or not 103 | discoverable, all to the greatest extent permissible under applicable law. 104 | 105 | c. Affirmer disclaims responsibility for clearing rights of other persons 106 | that may apply to the Work or any use thereof, including without limitation 107 | any person's Copyright and Related Rights in the Work. Further, Affirmer 108 | disclaims responsibility for obtaining any necessary consents, permissions 109 | or other rights required for any use of the Work. 110 | 111 | d. Affirmer understands and acknowledges that Creative Commons is not a 112 | party to this document and has no duty or obligation with respect to this 113 | CC0 or use of the Work. 114 | 115 | For more information, please see 116 | 117 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## Linux Tutorial 2 | This tutorial is about Linux knowledge. It is intended for personal use only. 3 | 4 | ### Summary 5 | 1. [Basic Commands](./content/basic_commands.md) 6 | 2. [Working with files](./content/working_with_files.md) 7 | 3. [File System](./content/filesytem.md) 8 | 4. [File Permissions](./content/file_permissions.md) 9 | 5. [Package Management](./content/package_management.md) 10 | 6. [Data Backup](./content/data_backup.md) 11 | 7. [System Info](./content/system_info.md) 12 | 8. [Swap Memory](./content/swap_memory.md) 13 | 9. [User Envinronment](./content/user_env.md) 14 | 10. [Processes](./content/processes.md) 15 | 11. [Volume Manager Basics](./content/volume_manager.md) 16 | 12. [Advanced Volume Manager](./content/volume_manager_cont.md) 17 | 12. [Networking](./content/basic_networking.md) 18 | 13. [Network File System](./content/nfs.md) 19 | 14. [iSCSI](./content/shared_storage_iscsi.md) 20 | 15. [Security](./content/basic_security.md) 21 | 16. [Command Line](./content/command_line_prompt.md) 22 | 17. [Text Commands](./content/text_commands.md) 23 | 18. [Bash Programming](./content/bash_programming.md) 24 | 19. [Systemd](./content/systemd.md) 25 | 20. [Samba Server](./content/samba_server.md) 26 | 21. [Virtual Networking](./content/virtual-networking.md) 27 | 22. [Network Namespaces](./content/network-namespaces.md) 28 | 23. [Clustering Basics](./content/cluster-basics.md) 29 | 24. [Advanced Clustering](./content/cluster-adv.md) 30 | -------------------------------------------------------------------------------- /content/bash_programming.md: -------------------------------------------------------------------------------- 1 | ## Bash shell programming 2 | The **shell** is a command line interpreter which provides the user interface for terminal windows. It can also be used to run scripts, even in non-interactive sessions without a terminal window, as if the commands were being directly typed in. 3 | ``` 4 | #!/bin/bash 5 | find /usr/lib -name "*.c" -ls 6 | ``` 7 | 8 | The first line of the script, that starts with ``#!/bin/bash`` contains the full path of the command interpreter that is to be used on the file. The command interpreter is tasked with executing statements that follow it in the script. Commonly used interpreters include: 9 | ``` 10 | /usr/bin/perl 11 | /bin/bash 12 | /bin/csh 13 | /bin/tcsh 14 | /bin/ksh 15 | /usr/bin/python 16 | /bin/sh 17 | ``` 18 | 19 | Scripting is not only limited to shell interpreter. It can be used for Python scripts too. 20 | ``` 21 | # ll script 22 | -rwxr--r--. 1 root root 55 Mar 3 15:22 script 23 | # cat script 24 | #!/usr/bin/python 25 | print "Welcome to the Python script" 26 | # ./script 27 | Welcome to the Python script 28 | ``` 29 | 30 | Scripts can be interactive too. 31 | 32 | ``` 33 | # cat script.sh 34 | #!/bin/bash 35 | # Interactive reading of variables 36 | echo "ENTER YOUR NAME" 37 | read sname 38 | # Display of variable values 39 | echo "WELCOME "$sname"!" 40 | # ./script.sh 41 | ENTER YOUR NAME 42 | Adriano 43 | WELCOME Adriano! 44 | ``` 45 | 46 | All shell scripts generate a return value upon finishing execution. The value can be set with the ``exit`` statement. Return values permit a process to monitor the exit state of another process often in a parent-child relationship. This helps to determine how this process terminated and take any appropriate steps necessary, contingent on success or failure. By convention, success is returned as 0, and failure is returned as a non-zero value. The return value is always stored in the ``$?`` environment variable. 47 | ``` 48 | # cat names.txt 49 | 01 Mario Rossi 50 | 02 Antonio Esposito 51 | 03 Michele Laforca 52 | 04 Antonio Esposito 53 | # echo $? 54 | 0 55 | # cat names 56 | cat: names: No such file or directory 57 | # echo $? 58 | 1 59 | ``` 60 | 61 | ### Basic syntax 62 | Scripts require you to follow a standard language syntax. Rules delineate how to define variables and how to construct and format allowed statements, etc. The table lists some special character usages within bash scripts: 63 | 64 | |Character|Description| 65 | |---------|-----------| 66 | |#|Used to add a comment, except when used as \#, or as #! when starting a script| 67 | |\\|Used at the end of a line to indicate continuation on to the next line| 68 | |;|Used to interpret what follows as a new command| 69 | |$|Indicates what follows is a variable| 70 | 71 | Sometimes you may want to group multiple commands on a single line. The semicolon character is used to separate these commands and execute them sequentially as if they had been typed on separate lines. 72 | 73 | The three commands in the following example will all execute even if the ones preceding them fail: 74 | ``` 75 | $ make ; make install ; make clean 76 | ``` 77 | However, you may want to abort subsequent commands if one fails. You can do this using the and operator: 78 | ``` 79 | $ make && make install && make clean 80 | ``` 81 | If the first command fails the second one will never be executed. A final refinement is to use the or operator: 82 | ``` 83 | $ cat file1 || cat file2 || cat file3 84 | ``` 85 | In this case, you proceed until something succeeds and then you stop executing any further steps. 86 | 87 | ### Functions 88 | A function is a code block that implements a set of operations. Functions are useful for executing procedures multiple times perhaps with varying input variables. Functions are also often called subroutines. Using functions in scripts requires two steps: 89 | 90 | 1. Declaring a function 91 | 2. Calling a function 92 | 93 | The function declaration requires a name which is used to invoke it. The proper syntax is: 94 | ``` 95 | function_name () { 96 | command... 97 | } 98 | ``` 99 | For example, the following function is named display: 100 | ``` 101 | display () { 102 | echo "This is a sample function" 103 | } 104 | ``` 105 | The function can be as long as desired and have many statements. Once defined, the function can be called later as many times as necessary. In the full example shown in the figure, we are also showing an often-used refinement: how to pass an argument to the function. The first, second, ..., n-th argument can be referred to as ``$1, $2, ..., $n``. The script name is referred as ``$0``. All parameters are referred as ``$*`` and the total number of arguments is ``$#``. 106 | ``` 107 | # cat script.sh 108 | #!/bin/bash 109 | echo The name of this program is: $0 110 | echo The first argument passed from the command line is: $1 111 | echo The second argument passed from the command line is: $2 112 | echo The third argument passed from the command line is: $3 113 | echo All of the arguments passed from the command line are : $* 114 | echo All done with $0 115 | exit 0 116 | # 117 | # ./script.sh A B C 118 | The name of this program is: ./script.sh 119 | The first argument passed from the command line is: A 120 | The second argument passed from the command line is: B 121 | The third argument passed from the command line is: C 122 | All of the arguments passed from the command line are : A B C 123 | All done with ./script.sh 124 | ``` 125 | 126 | ### Command substitution 127 | You may need to substitute the result of a command as a portion of another command. It can be done in two ways: 128 | 129 | 1. By enclosing the inner command with backticks (`) 130 | 2. By enclosing the inner command in $( ) 131 | 132 | No matter the method, the innermost command will be executed in a newly launched shell environment, and the standard output of the shell will be inserted where the command substitution was done. Virtually any command can be executed this way. Both of these methods enable command substitution; however, the second method allows command nesting. 133 | ``` 134 | # cat ./count.sh 135 | #!/bin/bash 136 | echo "The " $1 " contains " $(wc -l < $1) " lines." 137 | echo $? 138 | # ./count.sh /var/log/messages 139 | The /var/log/messages contains 114 lines. 140 | 0 141 | ``` 142 | In the above example, the output of the inner command becomes the argument for the outer command. 143 | 144 | ### The if statement 145 | Conditional decision making using an if statement, is a basic construct that any useful programming or scripting language must have. When an if statement is used, the ensuing actions depend on the evaluation of specified conditions such as: 146 | 147 | *. Numerical or string comparisons 148 | *. Return value of a command (0 for success) 149 | *. File existence or permissions 150 | 151 | In compact form, the syntax of an if statement is: 152 | ``` 153 | if TEST-COMMANDS; then CONSEQUENT-COMMANDS; fi 154 | ``` 155 | A more general definition is: 156 | ``` 157 | if condition 158 | then 159 | statements 160 | else 161 | statements 162 | fi 163 | ``` 164 | 165 | The following statement checks for a file argument, and if it is found, then it displays a message 166 | ``` 167 | #!/bin/bash 168 | if [ -f $1 ] 169 | then 170 | echo "The " $1 " contains " $(wc -l < $1) " lines."; 171 | echo $? 172 | fi 173 | # ./count.sh /etc/passwd 174 | The /etc/passwd contains 35 lines. 175 | 0 176 | ``` 177 | 178 | Following options for file check 179 | 180 | |Option|Action| 181 | |------|------| 182 | |-e file| Check if the file exists.| 183 | |-d file| Check if the file is a directory.| 184 | |-f file| Check if the file is a regular file.| 185 | |-s file| Check if the file is of non-zero size.| 186 | |-g file| Check if the file has sgid set.| 187 | |-u file| Check if the file has suid set.| 188 | |-r file| Check if the file is readable.| 189 | |-w file| Check if the file is writable.| 190 | |-x file| Check if the file is executable.| 191 | 192 | You can use the if statement to compare strings. The syntax is as follows: 193 | ``` 194 | if [ string1 == string2 ] 195 | then 196 | ACTION 197 | fi 198 | ``` 199 | 200 | Or to compare numbers, as follows: 201 | ``` 202 | if [ exp1 OPERATOR exp2 ] 203 | then 204 | ACTION 205 | fi 206 | ``` 207 | 208 | The options for operators are: 209 | 210 | Following options for file check 211 | 212 | |Option|Action| 213 | |------|------| 214 | |-eq|Equal to| 215 | |-ne|Not equal to| 216 | |-gt|Greater than| 217 | |-lt|Less than| 218 | |-ge|Greater than or equal to| 219 | |-le|Less than or equal to| 220 | 221 | -------------------------------------------------------------------------------- /content/basic_commands.md: -------------------------------------------------------------------------------- 1 | ### Locating Applications 2 | Depending on the specific distribution, programs and software packages can be installed in various directories. In general, executable programs should live in the following directories 3 | 4 | ``` 5 | /bin 6 | /usr/bin 7 | /sbin 8 | /usr/sbin 9 | /opt. 10 | ``` 11 | 12 | One way to locate programs is to employ the ``which`` utility. For example, to find out exactly where the diff program resides on the filesystem: 13 | ``` 14 | $ which diff 15 | /usr/bin/diff 16 | ``` 17 | If which does not find the program, whereis is a good alternative because it looks for packages in a broader range of system directories: 18 | ``` 19 | $ whereis diff 20 | diff: /usr/bin/diff /usr/share/man/man1/diff.1.gz 21 | ``` 22 | ### Accessing Directories 23 | The following commands are useful for directory navigation: 24 | 25 | |Command|Result| 26 | |-------|-----------| 27 | |cd |Change to your home directory| 28 | |cd ..|Change to parent directory| 29 | |cd - |Change to previous directory| 30 | |cd / |Changes your current directory to the root (/) directory| 31 | 32 | ### Exploring the Filesystem 33 | The tree command is a good way to get a bird’s-eye view of the filesystem tree. The following commands can help in exploring the filesystem: 34 | 35 | |Command|Result| 36 | |-------|-----------| 37 | |ls |List the contents of the present working directory| 38 | |ls –a |List all files including hidden files and directories| 39 | |tree |Displays a tree view of the filesystem| 40 | |tree -d|Just list the directories and suppress listing file names| 41 | 42 | ### Hard and Symbolic Links 43 | The ``ln`` command can be used to create hard links and or soft links, also known as symbolic links or symlinks. These two kinds of links are very common in UNIX-based operating systems. 44 | 45 | Suppose that file1.txt already exists. A hard link, called file2.txt, is created with the command: 46 | ``` 47 | # ln file1.txt file2.txt 48 | ``` 49 | Note that two files now appear to exist. However, a closer inspection of the file listing shows that this is not quite true. 50 | 51 | ``` 52 | # ls -l file* 53 | -rw-r--r--. 2 root root 604 Feb 16 11:49 file1.txt 54 | -rw-r--r--. 2 root root 604 Feb 16 11:49 file2.txt 55 | # ls -li file* 56 | 134415251 -rw-r--r--. 2 root root 604 Feb 16 11:49 file1.txt 57 | 134415251 -rw-r--r--. 2 root root 604 Feb 16 11:49 file2.txt 58 | ``` 59 | The -i option prints out in the first column the i-node number, which is a unique quantity for each file object. This field is the same for both of the two files; what is really going on here is that it is only one file but it has more than one name associated with it, as is indicated by the 2 that appears in the output. 60 | 61 | ``` 62 | # ln file1.txt file3.txt 63 | # ls -li file* 64 | 134415251 -rw-r--r--. 3 root root 604 Feb 16 11:49 file1.txt 65 | 134415251 -rw-r--r--. 3 root root 604 Feb 16 11:49 file2.txt 66 | 134415251 -rw-r--r--. 3 root root 604 Feb 16 11:49 file3.txt 67 | ``` 68 | Changing the file3.txt means change the same object as named as file1.txt, file2.txt and file3.txt. 69 | 70 | Symbolic or Soft links are created with the -s option as in: 71 | 72 | ``` 73 | # ln -s file1.txt file4.txt 74 | # ls -li file* 75 | 134415251 -rw-r--r--. 3 root root 644 Feb 16 11:59 file1.txt 76 | 134415251 -rw-r--r--. 3 root root 644 Feb 16 11:59 file2.txt 77 | 134415251 -rw-r--r--. 3 root root 644 Feb 16 11:59 file3.txt 78 | 134415252 lrwxrwxrwx. 1 root root 9 Feb 16 11:59 file4.txt -> file1.txt 79 | ``` 80 | Notice file4.txt no longer appears to be a regular file, and it clearly points to file1 and has a different inode number. Symbolic links take no extra space on the filesystem. They are extremely convenient as they can easily be modified to point to different places. An easy way to create a shortcut from your home directory to long pathnames is to create a symbolic link. 81 | 82 | Unlike hard links, soft links can point to objects even on different filesystems (or partitions) which may or may not be currently available or even exist. In the case where the link does not point to a currently available or existing object, you obtain a dangling link. 83 | 84 | Hard links are very useful and they save space, but you have to be careful with their use, sometimes in subtle ways. For one thing if you remove either file1.txt or file2.txt in the example, the inode object will remain, which might be undesirable as it may lead to subtle errors later if you recreate a file of that name. If you edit one of the files, exactly what happens depends on your editor; most editors including vi and gedit will retain the link by default but it is possible that modifying one of the names may break the link and result in the creation of two objects. 85 | 86 | -------------------------------------------------------------------------------- /content/basic_networking.md: -------------------------------------------------------------------------------- 1 | ## Network interfaces 2 | Network interfaces are a connection channel between a device and a network. Physically, network interfaces can proceed through a network interface card (**NIC**) or can be more abstractly implemented as software. You can have multiple network interfaces operating at once. Specific interfaces can be brought up (activated) or brought down (de-activated) at any time. A list of currently active network interfaces is reported by the ``ifconfig`` utility. Network configuration files are essential to ensure that interfaces function correctly. 3 | 4 | For **Debian** family configuration, the basic network configuration file is ``/etc/network/interfaces``. For **RedHat** family system configuration, the routing and host information is contained in ``/etc/sysconfig/network``. The network interface configuration script for the ``eth0`` interface is located at ``/etc/sysconfig/network-scripts/ifcfg-eth0``. For **SUSE** family system configuration, the routing and host information and network interface configuration scripts are contained in the ``/etc/sysconfig/network`` directory. 5 | 6 | ``` 7 | # cat /etc/sysconfig/network-scripts/ifcfg-enp0s25 8 | TYPE=Ethernet 9 | BOOTPROTO=none 10 | DEFROUTE=yes 11 | IPV4_FAILURE_FATAL=no 12 | IPV6INIT=yes 13 | IPV6_AUTOCONF=no 14 | IPV6_DEFROUTE=yes 15 | IPV6_FAILURE_FATAL=no 16 | NAME=enp0s25 17 | UUID=d9315bd4-159b-4871-95f5-98f2fbcc5a06 18 | ONBOOT=yes 19 | HWADDR=00:24:81:0F:EC:DE 20 | IPADDR=10.10.10.97 21 | PREFIX=24 22 | GATEWAY=10.10.10.1 23 | DNS=8.8.8.8 24 | ``` 25 | 26 | The ``ip`` is a very powerful program that can do many things. 27 | ``` 28 | # ip addr show 29 | # ip route show 30 | ``` 31 | 32 | ### Routing table 33 | The ``route`` command is used to view or change the IP routing table. You may want to change the IP routing table to add, delete or modify static routes to specific hosts or networks. 34 | 35 | ``` 36 | # route -n 37 | Kernel IP routing table 38 | Destination Gateway Genmask Flags Metric Ref Use Iface 39 | 0.0.0.0 10.10.10.1 0.0.0.0 UG 0 0 0 enp0s25 40 | 10.10.10.0 0.0.0.0 255.255.255.0 U 0 0 0 enp0s25 41 | 169.254.0.0 0.0.0.0 255.255.0.0 U 1002 0 0 enp48s0 42 | 169.254.0.0 0.0.0.0 255.255.0.0 U 1003 0 0 enp0s25 43 | 172.25.101.0 0.0.0.0 255.255.255.0 U 0 0 0 enp48s0 44 | # 45 | # route add 10.58.47.235 gw 172.25.101.1 46 | route -n 47 | Kernel IP routing table 48 | Destination Gateway Genmask Flags Metric Ref Use Iface 49 | 0.0.0.0 10.10.10.1 0.0.0.0 UG 0 0 0 enp0s25 50 | 10.10.10.0 0.0.0.0 255.255.255.0 U 0 0 0 enp0s25 51 | 10.58.47.235 172.25.101.1 255.255.255.255 UGH 0 0 0 enp48s0 52 | 169.254.0.0 0.0.0.0 255.255.0.0 U 1002 0 0 enp48s0 53 | 169.254.0.0 0.0.0.0 255.255.0.0 U 1003 0 0 enp0s25 54 | 172.25.101.0 0.0.0.0 255.255.255.0 U 0 0 0 enp48s0 55 | # 56 | # route delete 10.58.47.235 gw 172.25.101.1 57 | # route add -net 10.0.0.0 netmask 255.0.0.0 gw 10.10.10.1 enp0s25 58 | # route -n 59 | Kernel IP routing table 60 | Destination Gateway Genmask Flags Metric Ref Use Iface 61 | 0.0.0.0 10.10.10.1 0.0.0.0 UG 0 0 0 enp0s25 62 | 10.0.0.0 10.10.10.1 255.0.0.0 UG 0 0 0 enp0s25 63 | 10.10.10.0 0.0.0.0 255.255.255.0 U 0 0 0 enp0s25 64 | 169.254.0.0 0.0.0.0 255.255.0.0 U 1002 0 0 enp48s0 65 | 169.254.0.0 0.0.0.0 255.255.0.0 U 1003 0 0 enp0s25 66 | 172.25.101.0 0.0.0.0 255.255.255.0 U 0 0 0 enp48s0 67 | # route delete -net 10.0.0.0 netmask 255.0.0.0 gw 10.10.10.1 enp0s25 68 | ``` 69 | -------------------------------------------------------------------------------- /content/basic_security.md: -------------------------------------------------------------------------------- 1 | ## Linux basic security 2 | By default, Linux has several account types in order to isolate processes and workloads: 3 | 4 | 1. **root** 5 | 2. **system** 6 | 2. **normal** 7 | 3. **network** 8 | 9 | For a safe environment, it is advised to grant the minimum privileges possible and necessary to accounts, and remove inactive accounts. The ``last`` command, which shows the last time each user logged into the system, can be used to help identify potentially inactive accounts which are candidates for system removal. 10 | ``` 11 | # last 12 | adriano pts/4 10.10.10.113 Thu Feb 19 16:50 still logged in 13 | mina pts/2 10.10.10.113 Thu Feb 19 16:39 still logged in 14 | root pts/1 10.10.10.113 Thu Feb 19 16:25 - 16:25 (00:00) 15 | root pts/0 10.10.10.113 Thu Feb 19 15:42 still logged in 16 | adriano pts/3 10.10.10.246 Wed Feb 18 17:53 - 18:44 (00:51) 17 | root pts/2 10.10.10.99 Wed Feb 18 17:14 - 18:44 (01:30) 18 | adriano pts/1 10.10.10.246 Wed Feb 18 16:57 - 19:19 (02:22) 19 | root pts/0 10.10.10.246 Wed Feb 18 16:25 - 19:19 (02:53) 20 | root pts/0 10.10.10.246 Tue Feb 17 13:29 - 19:29 (06:00) 21 | reboot system boot 3.10.0-123.20.1. Tue Feb 17 13:28 - 17:20 (2+03:51) 22 | ``` 23 | 24 | The **root** account is the most privileged account on a Linux/UNIX system. This account has the ability to carry out all facets of system administration, including adding accounts, changing user passwords, examining log files, installing software, etc. 25 | 26 | A regular account user can perform some operations requiring special permissions; however, the system configuration must allow such abilities to be exercised. Running a network client or sharing a file over the network are operations that do not require a root account. 27 | 28 | In Linux you can use either ``su`` or ``sudo`` commands to temporarily grant root access to a normal user; these methods are actually quite different. When using the ``su`` command: 29 | 30 | * to elevate the privilege, you need to enter the root password. Giving the root password to a normal user should never, ever be done 31 | * once a user elevates to the root account, the normal user can do anything that the root user can do for as long as the user wants, without being asked again for a password 32 | * there are limited logging features 33 | 34 | When using the ``sudo`` command: 35 | 36 | * you need to enter the user’s password and not the root password 37 | * what the user is allowed to do can be precisely controlled and limited; by default the user will either always have to keep giving their password to do further operations with ``sudo``, or can avoid doing so for a configurable time interval 38 | * detailed logging features are available 39 | 40 | ### The sudo command 41 | Granting privileges using the ``sudo`` command is less dangerous than ``su`` and it should be preferred. By default, ``sudo`` must be enabled on a per-user basis. However, some distributions (such as Ubuntu) enable it by default for at least one main user, or give this as an installation option. To execute just one command with root privilege type ``sudo ``. When the command is complete you will return to being a normal unprivileged user. The ``sudo`` configuration files are stored in the ``/etc/sudoers`` file and in the ``/etc/sudoers.d/`` directory. By default, that directory is empty. 42 | 43 | The ``sudo`` command has the ability to keep track of unsuccessful attempts at gaining root access. An authentication failure message would appear in the ``/var/log/secure`` log file when trying to execute sudo bash without successfully authenticating the user 44 | 45 | ``` 46 | # tail /var/log/secure 47 | authentication failure; logname=op uid=0 euid=0 tty=/dev/pts/6 ruser=op rhost= user=op 48 | conversation failed 49 | auth could not identify password for [op] 50 | op : 1 incorrect password attempt ; 51 | TTY=pts/6 ; PWD=/var/log ; USER=root ; COMMAND=/bin/bash 52 | ``` 53 | 54 | Whenever the ``sudo`` command is invoked, a trigger will look at ``/etc/sudoers`` and the files in ``/etc/sudoers.d`` to determine if the user has the right to use ``sudo`` and what the scope of their privilege is. Unknown user requests and requests to do operations not allowed to the user even with ``sudo`` are reported. You can edit the sudoers file by using the ``visudo`` command, which ensures that only one person is editing the file at a time, has the proper permissions, and refuses to write out the file and exit if there is an error in the changes made. 55 | 56 | The basic structure of an entry is: 57 | > who where = (as_whom) what 58 | 59 | To create a normal user account and give it sudo access, login as root user and edit the ``/etc/sudoers`` file with the ``visudo`` command. Find the lines in the file that grant ``sudo`` access to users in the group ``wheel`` when enabled. 60 | ``` 61 | ## Allows people in group wheel to run all commands 62 | # %wheel ALL=(ALL) ALL 63 | ``` 64 | Remove the comment character at the start of the second line. This enables the configuration option. Save your changes. Add the user you created to the ``wheel`` group. 65 | ``` 66 | # usermod -aG wheel adriano 67 | # su adriano - 68 | $ groups 69 | adriano wheel 70 | $ sudo whoami 71 | We trust you have received the usual lecture from the local System 72 | Administrator. It usually boils down to these three things: 73 | 74 | #1) Respect the privacy of others. 75 | #2) Think before you type. 76 | #3) With great power comes great responsibility. 77 | 78 | [sudo] password for adriano: 79 | root 80 | ``` 81 | If sudo is configured correctly the last line value will be ``root``. 82 | 83 | Some Linux distributions prefer you add a file in the directory ``/etc/sudoers.d`` with a name the same as the user. This file contains the individual user's sudo configuration, and one should leave the master configuration file untouched except for changes that affect all users. 84 | 85 | ### The process isolation 86 | Linux is considered to be more secure than many other operating systems because processes are naturally isolated from each other. One process normally cannot access the resources of another process, even when that process is running with the same user privileges. Additional security mechanisms that have been recently introduced in order to make risks even smaller are: 87 | 88 | 1. **Control Groups**: allows system administrators to group processes and associate finite resources to each group (**cgroup**). 89 | 2. **Linux Containers**: makes it possible to run multiple isolated Linux systems containers on a single system. 90 | 3. **Virtualization**: hardware is emulated in such a way that not only processes can be isolated, but entire systems are run simultaneously as isolated and insulated guests (**virtual machines**) on one physical host. 91 | 92 | ### Password encryption 93 | Protecting passwords has become a crucial element of security. Most Linux distributions rely on a modern password encryption algorithm called SHA-512 (Secure Hashing Algorithm 512 bits), developed by the U.S. National Security Agency (NSA) to encrypt passwords. The SHA-512 algorithm is widely used for security applications and protocols. These security applications and protocols include TLS, SSL, PHP, SSH, S/MIME and IPSec. SHA-512 is one of the most tested hashing algorithms. 94 | 95 | ### Password aging 96 | The password aging is a method to ensure that users get prompts that remind them to create a new password after a specific period. This can ensure that passwords, if cracked, will only be usable for a limited amount of time. This feature is implemented using the ``chage`` command, which configures the password expiry information for a user. 97 | ``` 98 | # chage --list adriano 99 | Last password change : Feb 18, 2015 100 | Password expires : never 101 | Password inactive : never 102 | Account expires : never 103 | Minimum number of days between password change : 0 104 | Maximum number of days between password change : 99999 105 | Number of days of warning before password expires : 7 106 | ``` 107 | 108 | ### Public/Private Keys for Authentication 109 | Using encrypted keys for authentication offers two main benefits. Firstly, it is convenient as you no longer need to enter a password if you use public/private keys. Secondly, once public/private key pair authentication has been set up on the server, you can disable password authentication completely meaning that without an authorized key you can't gain access. 110 | 111 | Create a private key for client and a public key for server to do it 112 | ``` 113 | # ssh-keygen -t rsa 114 | Generating public/private rsa key pair. 115 | Enter file in which to save the key (/root/.ssh/id_rsa): 116 | Enter passphrase (empty for no passphrase): 117 | Enter same passphrase again: 118 | Your identification has been saved in /root/.ssh/id_rsa. 119 | Your public key has been saved in /root/.ssh/id_rsa.pub. 120 | 121 | # cd /root/.ssh 122 | # ll 123 | total 8 124 | -rw------- 1 root root 0 May 30 11:17 authorized_keys 125 | -rw------- 1 root root 1675 May 30 11:17 id_rsa 126 | -rw-r--r-- 1 root root 396 May 30 11:17 id_rsa.pub 127 | -rw-r--r-- 1 root root 0 May 30 11:07 known_hosts 128 | # chmod 700 ~/.ssh 129 | # chmod 600 ~/.ssh/id_rsa 130 | ``` 131 | 132 | This will create two files in your hidden ssh directory called: ``id_rsa`` and ``id_rsa.pub`` The first is your private key and the other is your public key. Install the public key to the authorized keys list and then remove it from the server 133 | ``` 134 | # cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys 135 | # rm -rf ~/.ssh/id_rsa.pub 136 | ``` 137 | Please, note that the same public key can be installed to many servers, just copy it on that server and install to the authorized keys list. 138 | 139 | Copy the private key on the client that you will use to connect to the server and then remove it from the server 140 | ``` 141 | # scp ~/.ssh/id_rsa root@clientmachine:root/.ssh/ 142 | # rm -rf ~/.ssh/id_rsa 143 | ``` 144 | 145 | On Linux and Unix client, use the private key to login to the server 146 | ``` 147 | # ssh -i ~/.ssh/id_rsa root@servermachine 148 | ``` 149 | 150 | On Windows client, use the puttygen tool to make the key in a suitable format and use the Putty application to login to the server. Please, note that each user that want to login must have his own key pair. 151 | -------------------------------------------------------------------------------- /content/cluster-adv.md: -------------------------------------------------------------------------------- 1 | ## Advanced Clustering 2 | Linux Clastering includes many advanced techniques to cover all types of Cluster. In the section [Cluster Basics](https://github.com/kalise/Linux-Tutorial/blob/master/content/cluster-basics.md) we setup a simple Active/Standby cluster. In this section, we are going to extend our Cluster to become an Active/Active cluster. 3 | 4 | In an Active/Standby cluster, the standby node is doing nothing for most of the time. Since we do not have shared data between the two nodes, there is no risk of data corruption. The second node can partecipate to the cluster task becoming an active member and improving the performances of the whole cluster. To achieve this goals, we make the HTTP Server running on both the nodes and installing a Load Balancer on both the nodes to distribute the client's requests in a Round Robin fashion. 5 | 6 | ![](../img/active-active-cluster.jpg?raw=true) 7 | 8 | Remove the HTTP Server resource from the Cluster 9 | 10 | [root@benji ~]# pcs resource delete HTTPServer 11 | Attempting to stop: HTTPServer...Stopped 12 | 13 | Add back the HTTP Server resource by changing its type 14 | 15 | [root@benji ~]# pcs resource create httpd systemd:httpd \ 16 | > configfile=/etc/httpd/conf/httpd.conf \ 17 | > op monitor interval=30s clone 18 | 19 | We changed from ``ocf:heartbeat:apache`` to ``systemd:httpd`` since we want the HTTP Server started as Systemd daemon. This permits to have the server running on both the nodes at same time. Please, note that service is still managed by Pacemaker and it should not be started by Systemd. 20 | 21 | On both the nodes, install the Load Balancer. We'll use HAProxy for simplicity 22 | 23 | [root@benji ~]# yum install haproxy -y 24 | 25 | Make sure the same configuration file is present on both the nodes 26 | 27 | [root@benji ~]# vi /etc/haproxy/haproxy.cfg 28 | #--------------------------------------------------------------------- 29 | # Global settings 30 | #--------------------------------------------------------------------- 31 | global 32 | log 127.0.0.1 local2 33 | chroot /var/lib/haproxy 34 | pidfile /var/run/haproxy.pid 35 | maxconn 4000 36 | user haproxy 37 | group haproxy 38 | daemon 39 | # turn on stats unix socket 40 | stats socket /var/lib/haproxy/stats 41 | #--------------------------------------------------------------------- 42 | # Common defaults 43 | #--------------------------------------------------------------------- 44 | defaults 45 | mode http 46 | log global 47 | option httplog 48 | option dontlognull 49 | option http-server-close 50 | option forwardfor except 127.0.0.0/8 51 | option redispatch 52 | retries 3 53 | timeout http-request 10s 54 | timeout queue 1m 55 | timeout connect 10s 56 | timeout client 1m 57 | timeout server 1m 58 | timeout http-keep-alive 10s 59 | timeout check 10s 60 | maxconn 3000 61 | #--------------------------------------------------------------------- 62 | # Listen configuration 63 | #--------------------------------------------------------------------- 64 | listen apache 65 | bind 10.10.10.23:80 transparent #bind to the Virtual IP 66 | mode http 67 | option http-server-close 68 | option forwardfor 69 | balance roundrobin 70 | server holly 10.10.10.22:80 check 71 | server benji 10.10.10.24:80 check 72 | 73 | The HAProxy binds to the Virtual IP address and then forward the client's requests to the HTTP Servers in a Round Robin fashion. To avoid conflict binding between the HTTP Server and HAProxy, make sure the HTTP Servers listen only on the IP addresses. To achieve this, configure the ``/etc/httpd/conf/httpd.conf`` configuration file, by specifing the IP address: 74 | 75 | [root@benji ~]# vi /etc/httpd/conf/httpd.conf 76 | ... 77 | Listen 10.10.10.24:80 78 | ... 79 | [root@holly ~]# vi /etc/httpd/conf/httpd.conf 80 | ... 81 | Listen 10.10.10.22:80 82 | ... 83 | 84 | Since we need the Load Balancer running on both the nodes to handle the client's requests, add the HAProxy resource to the Cluster as a Systemd daemon. 85 | 86 | [root@benji ~]# pcs resource create haproxy systemd:haproxy \ 87 | > op monitor interval=15s clone 88 | 89 | Restart the cluster 90 | 91 | [root@benji ~]# pcs cluster start --all 92 | holly: Starting Cluster... 93 | benji: Starting Cluster... 94 | 95 | and check the status 96 | 97 | [root@benji ~]# pcs status 98 | Cluster name: mycluster 99 | Last updated: Mon Jul 18 01:06:01 2016 100 | Stack: corosync 101 | Current DC: holly.b-cloud.it (version 1.1.13-10.el7_2.2-44eb2dd) - partition with quorum 102 | 2 nodes and 5 resources configured 103 | Online: [ benji holly ] 104 | Full list of resources: 105 | VIP-10.10.10.23 (ocf::heartbeat:IPaddr2): Started benji 106 | Clone Set: httpd-clone [httpd] 107 | Started: [ benji holly ] 108 | Clone Set: haproxy-clone [haproxy] 109 | Started: [ benji holly ] 110 | PCSD Status: 111 | holly: Online 112 | benji: Online 113 | Daemon Status: 114 | corosync: active/enabled 115 | pacemaker: active/enabled 116 | pcsd: active/enabled 117 | 118 | The Cluster is running with the Virtual IP on the **Benji** node 119 | 120 | [root@benji ~]# netstat -tupln | grep 80 121 | tcp 0 0 10.10.10.23:80 0.0.0.0:* LISTEN 9251/haproxy 122 | tcp 0 0 10.10.10.24:80 0.0.0.0:* LISTEN 1729/httpd 123 | [root@benji ~]# ip addr show ens32 124 | 2: ens32: mtu 1500 qdisc pfifo_fast state UP qlen 1000 125 | link/ether 00:0c:29:20:d2:dd brd ff:ff:ff:ff:ff:ff 126 | inet 10.10.10.24/24 brd 10.10.10.255 scope global ens32 127 | valid_lft forever preferred_lft forever 128 | inet 10.10.10.23/32 brd 10.10.10.255 scope global ens32 129 | valid_lft forever preferred_lft forever 130 | 131 | [root@holly ~]# netstat -tupln | grep 80 132 | tcp 0 0 10.10.10.23:80 0.0.0.0:* LISTEN 18467/haproxy 133 | tcp 0 0 10.10.10.22:80 0.0.0.0:* LISTEN 18444/httpd 134 | udp6 0 0 fe80::20c:29ff:fe77:123 :::* 623/ntpd 135 | [root@holly ~]# ip addr show ens32 136 | 2: ens32: mtu 1500 qdisc pfifo_fast state UP qlen 1000 137 | link/ether 00:0c:29:77:68:56 brd ff:ff:ff:ff:ff:ff 138 | inet 10.10.10.22/24 brd 10.10.10.255 scope global ens32 139 | valid_lft forever preferred_lft forever 140 | 141 | Check the services are started in Systemd fashion 142 | 143 | [root@benji ~]# systemctl status haproxy 144 | ● haproxy.service - Cluster Controlled haproxy 145 | Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled; vendor preset: disabled) 146 | Drop-In: /run/systemd/system/haproxy.service.d 147 | └─50-pacemaker.conf 148 | Active: active (running) since Mon 2016-07-18 01:05:55 CEST; 5min ago 149 | Main PID: 1748 (haproxy-systemd) 150 | CGroup: /system.slice/haproxy.service 151 | ├─1748 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid 152 | ├─1749 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds 153 | └─1750 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds 154 | Jul 18 01:05:55 benji systemd[1]: Started Cluster Controlled haproxy. 155 | Jul 18 01:05:55 benji systemd[1]: Starting Cluster Controlled haproxy... 156 | 157 | and 158 | 159 | [root@benji ~]# systemctl status httpd 160 | ● httpd.service - Cluster Controlled httpd 161 | Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled) 162 | Drop-In: /run/systemd/system/httpd.service.d 163 | └─50-pacemaker.conf 164 | Active: active (running) since Mon 2016-07-18 01:05:53 CEST; 6min ago 165 | Docs: man:httpd(8) 166 | man:apachectl(8) 167 | Main PID: 1729 (httpd) 168 | Status: "Total requests: 0; Current requests/sec: 0; Current traffic: 0 B/sec" 169 | CGroup: /system.slice/httpd.service 170 | ├─1729 /usr/sbin/httpd -DFOREGROUND 171 | └─1734 /usr/sbin/httpd -DFOREGROUND 172 | Jul 18 01:05:52 benji systemd[1]: Starting Cluster Controlled httpd... 173 | Jul 18 01:05:53 benji systemd[1]: Started Cluster Controlled httpd. 174 | 175 | Set that the order of starting is the Virtual IP first and then the other services. This is required to assure there is always an IP Address where to send client's requests. Also make sure that the Floating IP and the HAProxy are always working togheter. 176 | 177 | [root@benji ~]# pcs constraint order VIP-10.10.10.23 then haproxy-clone 178 | Adding VIP-10.10.10.23 haproxy-clone (kind: Mandatory) (Options: first-action=start then-action=start) 179 | [root@benji ~]# pcs constraint order httpd-clone then haproxy-clone 180 | Adding httpd-clone haproxy-clone (kind: Mandatory) (Options: first-action=start then-action=start) 181 | [root@benji ~]# pcs constraint colocation add VIP-10.10.10.23 with haproxy-clone 182 | [root@benji ~]# pcs constraint 183 | Location Constraints: 184 | Ordering Constraints: 185 | start VIP-10.10.10.23 then start haproxy-clone (kind:Mandatory) 186 | start httpd-clone then start haproxy-clone (kind:Mandatory) 187 | Colocation Constraints: 188 | VIP-10.10.10.23 with haproxy-clone (score:INFINITY) 189 | 190 | We are redy to test the Cluster 191 | 192 | [stack@director ~]$ curl http://10.10.10.23 193 | Hello Holly 194 | [stack@director ~]$ curl http://10.10.10.23 195 | Hello Benji 196 | [stack@director ~]$ curl http://10.10.10.23 197 | Hello Holly 198 | 199 | 200 | -------------------------------------------------------------------------------- /content/cluster-basics.md: -------------------------------------------------------------------------------- 1 | ## Cluster Basics 2 | A cluster is two or more computers (cluster members) that work together to perform a task, for example, provide high availability of a given service. High availability clusters provide highly available services by eliminating single points of failure and by failing over services from one cluster member to another in case a node becomes inoperative. 3 | 4 | Typically, services in a high availability cluster maintain data integrity as one cluster member takes over control of a service from another cluster member. Node failures in a high availability cluster are not visible from clients outside the cluster. 5 | 6 | In the Linux world, there are many cluster tools to achieve High Availability of a resource. The most used is **Pacemaker**. A cluster configured with Pacemaker comprises separate component daemons that monitor cluster membership, scripts that manage the services, and resource management subsystems that monitor the resources. The following components form the Pacemaker architecture: 7 | 8 | 1. **Cluster Information Base**: the Pacemaker information daemon distributes and synchronizes the cluster configuration and status information from the Designated Coordinator (DC) of the cluster to all other cluster members. The DC is one cluster member designated to store the cluster state. 9 | 10 | 2. **Cluster Resource Management Daemon**: cluster resources managed by this component can be queried by client systems, moved, instantiated, and changed when needed. Each cluster node also includes a local resource manager daemon that acts as an interface between Cluster Resource Manager daemon and the resource itself. The local resource manager passes commands from Cluster Resource Manager to agents, such as starting and stopping and relaying resurce status information. 11 | 12 | 3. **Fencing Manager**: often deployed in conjunction with a power supply switch, this component acts as a cluster resource in Pacemaker that processes fence requests, forcefully powering down nodes and removing them from the cluster to ensure data integrity. Pacemaker use a fencing technique called **STONITH** (Shoot The Other Node In The Head) intended to prevent data corruption caused by faulty nodes in a cluster that are unresponsive but still accessing application data (the so called "Split Brain Scenario"). 13 | 14 | ### Install a simple Cluster 15 | Pacemaker requires a messaging layer daemon, called **Corosync** that provides a cluster membership and closed communication model for creating replicated state machines, on top of which Pacemaker can run. Corosync can be seen as the underlying system that connects the cluster nodes together, while Pacemaker monitors the cluster and takes action in the event of a failure. In addition, we are going to use **PCS**, a command line interface that interacts with both Corosync and Pacemaker. 16 | 17 | This example will be also used to explain the basic concepts of Linux Clustering. 18 | 19 | | 20 | +----------------------+ | +----------------------+ 21 | | Node01 | | | Node02 | 22 | | holly.noverit.com +----------+----------+ benji.noverit.com | 23 | | 10.10.10.22 | | 10.10.10.24 | 24 | +----------------------+ +----------------------+ 25 | 26 | Install, start and enable Pacemaker and PCS on both the nodes. Because Corosync is a dependency to Pacemaker, it's usually a better idea to simply install Pacemaker and let the system decide which Corosync version should be installed. 27 | 28 | [root@holly ~]# yum -y install pacemaker 29 | [root@holly ~]# yum -y install pcs 30 | [root@holly ~]# systemctl start pcsd 31 | [root@holly ~]# systemctl enable pcsd 32 | 33 | [root@benji ~]# yum -y install pacemaker 34 | [root@benji ~]# yum -y install pcs 35 | [root@benji ~]# systemctl start pcsd 36 | [root@benji ~]# systemctl enable pcsd 37 | 38 | Pacemaker need to communicate beween nodes, enable the port firewall on each node, which by default is 2224 over TCP. Otherwise, disable the firewall if you are working in a secure setup. 39 | 40 | [root@holly ~]# systemctl stop firewalld 41 | [root@holly ~]# systemctl disable firewalld 42 | [root@benji ~]# systemctl stop firewalld 43 | [root@benji ~]# systemctl disable firewalld 44 | 45 | The PCS utility creates a user during installation, named ``hacluster``, with a disabled password. We need to define a password for this user on both servers. This will enable PCS to perform tasks such as synchronizing the Corosync configuration on multiple nodes, as well as starting and stopping the cluster. 46 | 47 | [root@holly ~]# passwd hacluster 48 | Changing password for user hacluster 49 | [root@benji ~]# passwd hacluster 50 | Changing password for user hacluster 51 | 52 | Use the same password on both servers. We are going to use this password to configure the cluster in the next step. Please, note that the user ``hacluster`` has no interactive shell or home directory associated with its account, which means it's not possible to log into the server using its credentials. 53 | 54 | Only on a node of the cluster, authenticate the cluster nodes 55 | 56 | [root@holly ~]# pcs cluster auth holly benji 57 | Username: hacluster 58 | Password: 59 | holly: Authorized 60 | benji: Authorized 61 | 62 | From the same node, generate the Corosync configuration 63 | 64 | [root@holly ~]# pcs cluster setup --name mycluster holly benji 65 | Shutting down pacemaker/corosync services... 66 | Redirecting to /bin/systemctl stop pacemaker.service 67 | Redirecting to /bin/systemctl stop corosync.service 68 | Killing any remaining services... 69 | Removing all cluster configuration files... 70 | holly: Succeeded 71 | benji: Succeeded 72 | Synchronizing pcsd certificates on nodes holly, benji... 73 | benji: Success 74 | holly: Success 75 | Restaring pcsd on the nodes in order to reload the certificates... 76 | benji: Success 77 | holly: Success 78 | 79 | This will generate a cluster configuration file (i.e. the cluster information base) located at ``/etc/corosync/corosync.conf`` based on the parameters provided to the cluster setup command: 80 | 81 | [root@holly ~]# cat /etc/corosync/corosync.conf 82 | totem { 83 | version: 2 84 | secauth: off 85 | cluster_name: mycluster 86 | transport: udpu 87 | } 88 | nodelist { 89 | node { 90 | ring0_addr: holly 91 | nodeid: 1 92 | } 93 | node { 94 | ring0_addr: benji 95 | nodeid: 2 96 | } 97 | } 98 | quorum { 99 | provider: corosync_votequorum 100 | two_node: 1 101 | } 102 | logging { 103 | to_logfile: yes 104 | logfile: /var/log/cluster/corosync.log 105 | to_syslog: yes 106 | } 107 | 108 | Start and enable the cluster 109 | 110 | [root@holly ~]# pcs cluster start --all 111 | benji: Starting Cluster... 112 | holly: Starting Cluster... 113 | 114 | [root@holly ~]# pcs cluster enable --all 115 | holly: Cluster Enabled 116 | benji: Cluster Enabled 117 | 118 | 119 | Check the status of the cluster 120 | 121 | [root@holly ~]# pcs status 122 | Cluster name: mycluster 123 | WARNING: no stonith devices and stonith-enabled is not false 124 | Last updated: Sat Jul 16 17:20:14 2016 125 | Stack: corosync 126 | Current DC: holly (version 1.1.13-10.el7_2.2-44eb2dd) - partition with quorum 127 | 2 nodes and 0 resources configured 128 | Online: [ benji holly ] 129 | Full list of resources: - 130 | PCSD Status: 131 | holly: Online 132 | benji: Online 133 | Daemon Status: 134 | corosync: active/enabled 135 | pacemaker: active/enabled 136 | pcsd: active/enabled 137 | 138 | Some interesting info: 139 | 140 | 1. The Designated Coordinator (DC) is the node holly where from we configured the cluster 141 | 2. There are only 2 nodes onlyne and no resurces 142 | 3. The name of the cluster is "mycluster" 143 | 4. All daemons: corosync, pacemaker and pcsd are active and enabled 144 | 5. Fencing (stonith) is enabled but no fencing devices are configured 145 | 146 | Confirm that both nodes joined the cluster by running the following command on any of the servers 147 | 148 | [root@holly ~]# pcs status corosync 149 | Membership information 150 | ---------------------- 151 | Nodeid Votes Name 152 | 1 1 holly (local) 153 | 2 1 benji 154 | [root@holly ~]# 155 | 156 | Because our cluster does not manage shared data resources, there is no risk to have a Split Brain Scenario and so we are going to disable fencing 157 | 158 | [root@holly ~]# pcs property set stonith-enabled=false 159 | 160 | Cluster quorum as a concept (see later) makes no sense in a two-node scenario, because you only have it when more than half the nodes are available, so we'll disable it too. 161 | 162 | [root@holly ~]# pcs property set no-quorum-policy=ignore 163 | 164 | To see a recap of the Cluster properties 165 | 166 | [root@benji ~]# pcs property list 167 | Cluster Properties: 168 | cluster-infrastructure: corosync 169 | cluster-name: mycluster 170 | dc-version: 1.1.13-10.el7_2.2-44eb2dd 171 | have-watchdog: false 172 | no-quorum-policy: ignore 173 | stonith-enabled: false 174 | 175 | Cluster nodes should not be halted as other standard nodes. It's always a best practice to shutdown the cluster first and then shutdown the system. 176 | 177 | To stop the cluster on a signle node 178 | 179 | [root@benji ~]# pcs cluster stop 180 | Stopping Cluster (pacemaker)... Stopping Cluster (corosync)... 181 | [root@benji ~]# pcs cluster status 182 | Error: cluster is not currently running on this node 183 | 184 | Or on all nodes of the cluster 185 | 186 | [root@holly ~]# pcs cluster stop --all 187 | holly: Stopping Cluster (pacemaker)... 188 | benji: Stopping Cluster (pacemaker)... 189 | benji: Stopping Cluster (corosync)... 190 | holly: Stopping Cluster (corosync)... 191 | [root@holly ~]# 192 | 193 | ### Add a resource to the Cluster 194 | Lets add a cluster service, we'll choose one doesn't require too much configuration and works everywhere to make things easy. 195 | 196 | Install and configure an HTTP Server on both the nodes. Note: not need to start/enable the service. 197 | 198 | [root@benji ~]# yum install -y httpd 199 | [root@benji ~]# echo "Hello Benji" > /var/www/html/index.html 200 | [root@holly ~]# yum install -y httpd 201 | [root@holly ~]# echo "Hello Holly" > /var/www/html/index.html 202 | 203 | Add the HTTP Server as resource of the cluster 204 | 205 | [root@benji ~]# pcs resource create HTTPServer ocf:heartbeat:apache \ 206 | > configfile=/etc/httpd/conf/httpd.conf \ 207 | > op monitor interval=1min 208 | 209 | The name of the resource is ``HTTPServer`` of type ``ocf:heartbeat:apache``. The type defined for a resource tell the cluster which script to use for the resource, the provider of the script and what standards it conforms to. In that case, the standard is **Open Cluster Framework**. The command tells also Pacemaker to check the health of this service every 60 seconds by calling the agent's monitor action. 210 | 211 | Add a Virtual IP address as second resource of the cluster. This IP Address will be used by clients of the cluster to access the HTTP Server resource 212 | 213 | [root@benji ~]# pcs resource create VirtualIP ocf:heartbeat:IPaddr2 \ 214 | > ip=10.10.10.23 \ 215 | > cidr_netmask=24 \ 216 | > op monitor interval=30s 217 | 218 | The name of the resource is ``VirtualIP`` of type ``ocf:heartbeat:IPaddr2``. The command tells also Pacemaker to check the health of this service every 30 seconds by calling the agent's monitor action. The Virtual IP resource binds the IP address specified in the command above to the network interface of the node owning the Virtual IP resources. This Virtual IP is floating from one node to the other, depending on the status of the node itself: 219 | 220 | [root@benji ~]# ip addr show ens32 221 | 2: ens32: mtu 1500 qdisc pfifo_fast state UP qlen 1000 222 | link/ether 00:0c:29:20:d2:dd brd ff:ff:ff:ff:ff:ff 223 | inet 10.10.10.24/24 brd 10.10.10.255 scope global ens32 224 | valid_lft forever preferred_lft forever 225 | inet 10.10.10.23/24 brd 10.10.10.255 scope global secondary ens32 226 | valid_lft forever preferred_lft forever 227 | 228 | [root@holly ~]# ip addr show ens32 229 | 2: ens32: mtu 1500 qdisc pfifo_fast state UP qlen 1000 230 | link/ether 00:0c:29:77:68:56 brd ff:ff:ff:ff:ff:ff 231 | inet 10.10.10.22/24 brd 10.10.10.255 scope global ens32 232 | valid_lft forever preferred_lft forever 233 | 234 | Set that HTTPServer and VirtualIP are always on a same node 235 | 236 | [root@benji ~]# pcs constraint colocation add HTTPServer with VirtualIP 237 | 238 | Set that the order of starting is VirtualIP first and then HTTPServer. This is required to assure there is always an IP Address where to send client's requests 239 | 240 | [root@holly ~]# pcs constraint order VirtualIP then HTTPServer 241 | Adding VirtualIP HTTPServer (kind: Mandatory) (Options: first-action=start then-action=start) 242 | 243 | See the status of both the resources 244 | 245 | [root@benji ~]# pcs status resources 246 | VirtualIP (ocf::heartbeat:IPaddr2): Started by benji 247 | HTTPServer (ocf::heartbeat:apache): Started by holly 248 | 249 | and resources constraints 250 | 251 | [root@holly ~]# pcs constraint 252 | Location Constraints: 253 | Ordering Constraints: 254 | start VirtualIP then start HTTPServer (kind:Mandatory) 255 | Colocation Constraints: 256 | HTTPServer with VirtualIP (score:INFINITY) 257 | 258 | Now we can access the HTTP Server from a web client by pointing to the Virtual IP Address 10.10.10.23 259 | 260 | [stack@director ~]$ curl http://10.10.10.23 261 | Hello Benji 262 | 263 | To test Cluster failover, stop current active node manually 264 | 265 | [root@benji html]# pcs cluster stop 266 | Stopping Cluster (pacemaker)... Stopping Cluster (corosync)... 267 | 268 | 269 | and make sure resource will switch to the other node 270 | 271 | [stack@director ~]$ curl http://10.10.10.23 272 | Hello Holly 273 | 274 | ### Accessing the cluster management form a Web GUI 275 | Cluster management is possible also via a Web GUI. Point the browser to the primary member node and login as the ``hacluster`` user 276 | 277 | https://:2224 278 | -------------------------------------------------------------------------------- /content/command_line_prompt.md: -------------------------------------------------------------------------------- 1 | ## Modify the Command Line Prompt 2 | The PS1 variable is the character string that is displayed as the prompt on the command line. Most distributions set PS1 to a known default value, for example, the user and the hostname as in: 3 | ``` 4 | [root@caldera01 ~]# 5 | ``` 6 | This could prove useful if you are working in multiple roles and want to be always reminded of who you are and what machine you are on. 7 | 8 | ``` 9 | [root@caldera01 ~]# echo $PS1 10 | [\u@\h \W]\$ 11 | [root@caldera01 ~]# 12 | [root@caldera01 ~]# export PS1='[\u@\h \W(customt)]# ' 13 | [root@caldera01 ~(customt)]# 14 | [root@caldera01 ~(customt)]# echo $PS1 15 | [\u@\h \W(customt)]# 16 | [root@caldera01 ~(customt)]# 17 | [root@caldera01 ~(customt)]# export PS1='[\u@\h \W]# ' 18 | [root@caldera01 ~]# 19 | ``` 20 | -------------------------------------------------------------------------------- /content/data_backup.md: -------------------------------------------------------------------------------- 1 | ### Backup the data 2 | The ``rsync`` command is used to synchronize entire directory trees. Basically, it copies file as the ``cp`` command does. In addition, ``rsync`` checks if the file being copied already exists. If the file exists and there is no change in size or modification time, ``rsync`` will avoid an unnecessary copy and save time. Furthermore, because rsync copies only the parts of files that have actually changed, it can be very fast. 3 | 4 | The ``rsync`` is very efficient when recursively copying one directory tree via network, because only the differences are transmitted. One often synchronizes the destination directory tree with the origin, using the ``rsync -r`` option to recursively walk down the directory tree copying all files and directories below the one listed as the source. 5 | 6 | ``` 7 | # rsync -ravzh project_ABC /data/backups 8 | sending incremental file list 9 | project_ABC/ 10 | project_ABC/file1.txt 11 | project_ABC/file2.txt 12 | project_ABC/file3.txt 13 | project_ABC/file4.txt 14 | 15 | sent 636 bytes received 92 bytes 1.46K bytes/sec 16 | total size is 452 speedup is 0.62 17 | 18 | ``` 19 | 20 | ### Compress the data 21 | File data is often compressed to save disk space and reduce the time it takes to transmit files over networks. Linux uses a number of methods to perform this compression. 22 | 23 | |Command|Usage| 24 | |-------|-----------| 25 | |gzip |The most frequently used Linux compression utility| 26 | |bzip2 |Produces files significantly smaller than those produced by gzip| 27 | |xz |The most space efficient compression utility used in Linux. It is now used by kernel.org to store archives of the Linux kernel.| 28 | |zip |Is often required to examine and decompress archives from other operating systems| 29 | 30 | These techniques vary in the efficiency of the compression (how much space is saved) and in how long they take to compress; generally the more efficient techniques take longer. Decompression time doesn't vary as much across different methods. 31 | 32 | ### Archiving data 33 | The ``tar`` command allows you to create or extract files from an archive file, often called a tarball. At the same time you can optionally compress while creating the archive, and decompress while extracting its contents. 34 | 35 | Here are some examples of the use of tar: 36 | 37 | |Command|Usage| 38 | |-------|-----------| 39 | |tar xvf mydir.tar|Extract all the files in mydir.tar into the mydir directory| 40 | |tar zcvf mydir.tar.gz mydir|Create the archive and compress with gzip| 41 | |tar jcvf mydir.tar.bz2 mydir|Create the archive and compress with bz2| 42 | |tar xvf mydir.tar.gz|Extract all the files in mydir.tar.gz into the mydir directory.| 43 | |tar cvf mydir.tar|show the content into the mydir directory| 44 | 45 | ### Copying disks 46 | The ``dd`` command is very useful for making copies of raw disk space. For example, to back up the Master Boot Record (MBR) (the first 512 byte sector on the disk that contains a table describing the partitions on that disk), use: 47 | ``` 48 | # dd if=/dev/sda of=sda.mbr bs=512 count=1 49 | ``` 50 | To use dd to make a copy of one disk onto another, deleting everything that previously existed on the second disk, use: 51 | ``` 52 | # dd if=/dev/sda of=/dev/sdb 53 | ``` 54 | An exact copy of the first disk device is created on the second disk device. 55 | 56 | The ``dd`` command is usefull to duplicate a bootable disk as a Compact Flash card, a Micro SD card or a bootable USB dongle. Insert the CF Card to be copied into the system and make a copy 57 | ``` 58 | # dd if=/dev/sdb of=./backup.img 59 | ``` 60 | Remove the CF Card, insert a new one and make a new copy 61 | ``` 62 | # dd if=./backup.img of=/dev/sdc 63 | ``` 64 | -------------------------------------------------------------------------------- /content/file_permissions.md: -------------------------------------------------------------------------------- 1 | ### File permissions 2 | In Linux and other UNIX operating systems, every file is associated with a user who is the owner. Every file is also associated with a group which has an interest in the file and certain rights, or permissions: read, write, and execute. 3 | 4 | |Command|Result| 5 | |-------|-----------| 6 | |chown|Used to change user ownership of a file or directory| 7 | |chgrp|Used to change group ownership| 8 | |chmod|Used to change the permissions on the file| 9 | 10 | Files have three kinds of permissions: read (**r**), write (**w**), execute (**x**). These are generally represented as in the following order **rwx**. These permissions affect three groups of owners: user (**u**), group (**g**), and others (**o**). As a result, you have the following three groups of three permissions: 11 | 12 | |rwx:|rwx:|rwx| 13 | |----|----|---| 14 | |u:|g:|o| 15 | 16 | There are a number of different ways to use the ``chmod`` command. For instance, to give the owner execute permission: 17 | 18 | ``` 19 | $ ls -l test1 20 | -rw-rw-r-- 1 joy caldera 1601 Mar 9 15:04 test1 21 | $ chmod u+x test1 22 | $ ls -l test1 23 | -rwxrw-r-- 1 joy caldera 1601 Mar 9 15:04 test1 24 | ``` 25 | 26 | This kind of syntax can be difficult to type and remember, so one often uses a shorthand which lets you set all the permissions in one step. This is done with a simple algorithm, and a single digit suffices to specify all three permission bits for each entity. This digit is the sum of: 27 | 28 | * 4 if read permission is desired. 29 | * 2 if write permission is desired. 30 | * 1 if execute permission is desired. 31 | 32 | Thus 7 means read+write+execute, 6 means read+write, and 5 means read+execute. 33 | 34 | When you apply this to the ``chmod`` command you have to give three digits for each degree of freedom, such as in 35 | ``` 36 | $ chmod 755 test1 37 | $ ls -l test1 38 | -rwxr-xr-x 1 joy caldera 1601 Mar 9 15:04 test1 39 | ``` 40 | The group ownership is changed by using the ``chgrp`` command 41 | ``` 42 | # ll /home/mina/myfile.txt 43 | -rw-rw-r--. 1 mina caldera 679 Feb 19 16:51 /home/mina/myfile.txt 44 | # chgrp root /home/mina/myfile.txt 45 | # ll /home/mina/myfile.txt 46 | -rw-rw-r--. 1 mina root 679 Feb 19 16:51 /home/mina/myfile.txt 47 | ``` 48 | -------------------------------------------------------------------------------- /content/filesytem.md: -------------------------------------------------------------------------------- 1 | ### Filesystem Structure 2 | On many systems, including Linux, the **filesystem** is structured like a tree. The tree is usually portrayed as inverted, and starts at what is most often called the **root** directory, which marks the beginning of the hierarchical filesystem and is also denoted by **/**. 3 | 4 | The Filesystem Hierarchy Standard (**FHS**) grew out of historical standards from early versions of UNIX. The FHS provides Linux developers and system administrators with a standard directory structure for the filesystem, which provides consistency between systems and distributions. Linux supports various filesystem types created for Linux, along with compatible filesystems from other operating systems. Many older, legacy filesystems are supported. Some examples of filesystem types that Linux supports are: 5 | 6 | 1. **ext3**, **ext4**, **btrfs**, **xfs** (native Linux filesystems) 7 | 2. **vfat**, **ntfs**, **hfs** (filesystems from other operating systems) 8 | 9 | Each filesystem resides on a hard disk **partition**. Partitions help to organize the contents of disks according to the kind of data contained and how it is used. For example, important programs required to run the system are often kept on a separate partition than the one that contains files owned by regular users. In addition, temporary files created and destroyed during the normal operation of Linux are often located on a separate partition; in this way, using all available space on a particular partition may not fatally affect the normal operation of the system. 10 | 11 | Before you can start using a filesystem, you need to mount it to the filesystem tree at a **mountpoint**. This is simply a directory (which may or may not be empty) where the filesystem is to be attached (mounted). Sometimes you may need to create the directory if it doesn't already exist. If you mount a filesystem on a non-empty directory, the former contents of that directory are covered-up and not accessible until the filesystem is unmounted. Thus mount points are usually empty directories. 12 | 13 | The ``mount`` command is used to attach a filesystem somewhere within the filesystem tree. Arguments include the device node and mount point. 14 | ``` 15 | $ mount /dev/sda5 /mnt 16 | ``` 17 | This will attach the filesystem contained in the disk partition associated with the ``/dev/sda5`` device node, into the filesystem tree at the ``/mnt`` mount point. Note that unless the system is otherwise configured only the root user has permission to run mount. If you want it to be automatically available every time the system starts up, you need to edit the file ``/etc/fstab`` accordingly. The name is short for Filesystem Table. Looking at this file will show you the configuration of all pre-configured filesystems. 18 | 19 | The ``umount`` command is used to detach the filesystem from the mount point. 20 | ``` 21 | $ umount /mnt 22 | ``` 23 | 24 | The command ``df -Th`` (it stands for disk-free) will display information about mounted filesystems including type and usage statistics about currently used and available space. 25 | 26 | ``` 27 | # df -Th 28 | Filesystem Type Size Used Avail Use% Mounted on 29 | /dev/mapper/os-root xfs 50G 2.0G 48G 4% / 30 | devtmpfs devtmpfs 1.8G 0 1.8G 0% /dev 31 | tmpfs tmpfs 1.9G 4.0K 1.9G 1% /dev/shm 32 | tmpfs tmpfs 1.9G 8.6M 1.8G 1% /run 33 | tmpfs tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup 34 | /dev/mapper/swift01-zone01 xfs 49G 33M 49G 1% /srv/node/z1d1 35 | /dev/mapper/swift02-zone02 xfs 49G 33M 49G 1% /srv/node/z2d1 36 | /dev/sda1 xfs 497M 167M 331M 34% /boot 37 | /dev/mapper/os-data xfs 20G 261M 20G 2% /data 38 | ``` 39 | 40 | ### The home directories 41 | In any UNIX system, each user has his own home directory, usually placed under ``/home``. The ``/root`` directory on modern Linux systems is no more than the root user's home directory. The ``/home`` directory is often mounted as a separate filesystem on its own partition, or even exported remotely on a network through NFS. 42 | 43 | ### The binary directories 44 | The ``/bin`` directory contains executable binaries, essential commands used in single-user mode, and essential commands required by all system users, such as ``ps``, ``ls``, ``cp``. Commands that are not essential for the system in single-user mode are placed in the ``/usr/bin`` directory, while the ``/sbin`` directory is used for essential binaries related to system administration, such as ``ifconfig`` and ``shutdown``. There is also a ``/usr/sbin`` directory for less essential system administration programs. All the binary directories are under the root partition. Sometimes ``/usr`` is a separate filesystem that may not be available in single-user mode. This was why essential commands were separated from non-essential commands. However, in some of the most modern Linux systems this distinction is considered obsolete, and ``/usr/bin`` and ``/bin`` are actually just linked together as are ``/usr/sbin`` and ``/sbin``. 45 | 46 | ### The device directory 47 | The ``/dev`` directory contains device nodes, a type of pseudo-file used by most hardware and software devices, except for network devices. This directory is empty on the disk partition when it is not mounted but it contains entries which are created by the ``udev`` system, which creates and manages device nodes on Linux, creating them dynamically when devices are found. The ``/dev`` directory contains items such as: 48 | ``` 49 | /dev/sda1 50 | /dev/lp1 51 | /dev/dvd1 52 | ``` 53 | 54 | ### The variable directory 55 | The ``/var`` directory contains files that are expected to change in size and content as the system is running (var stands for variable) such as the entries in the following directories: 56 | 57 | * System log files: ``/var/log`` 58 | * Packages files: ``/var/lib`` 59 | * Print queues: ``/var/spool`` 60 | * Temp files: ``/var/tmp`` 61 | * FTP home directory: ``/var/ftp`` 62 | * Web Server directory: ``/var/www`` 63 | 64 | The ``/var`` directory may be put in its own partition so that growth of the files can be accommodated and the file sizes do not fatally affect the system. 65 | 66 | ### The system configuration directory 67 | The ``/etc`` directory is the home for system configuration files. It contains no binary programs, although there are some executable scripts. For example, the file ``resolv.conf`` tells the system where to go on the network to obtain host name to IP address mappings (DNS). Files like ``passwd``, ``shadow`` and ``group`` for managing user accounts are found in the ``/etc`` directory. System run level scripts are found in subdirectories of ``/etc``. For example, ``/etc/rc2.d`` contains links to scripts for entering and leaving run level 2. Some Linux distributions extend the contents of ``/etc``. For example, **Red Hat** adds the ``/etc/sysconfig`` subdirectory that contains more configuration files. 68 | 69 | ### The boot directory 70 | The ``/boot`` directory contains the few essential files needed to boot the system. For every alternative kernel installed on the system there are four files: 71 | 72 | * ``vmlinuz`` is the compressed Linux kernel, required for booting 73 | * ``initramfs`` is the initial ram filesystem, required for booting 74 | * ``config is`` the kernel configuration file, only used for debugging 75 | * ``System.map`` contains the kernel symbol table, only used for debugging 76 | 77 | Each of these files has a kernel version appended to its name. 78 | 79 | ### The libraries directory 80 | The ``/lib`` contains libraries (common code shared by applications and needed for them to run) for the essential programs in ``/bin`` and ``/sbin`` folders. Most of these are what are known as dynamically loaded libraries (also known as shared libraries or Shared Objects (SO)). On some Linux distributions there exists a ``/lib64`` directory containing 64-bit libraries, while ``/lib`` contains 32-bit versions. Kernel modules (kernel code, often device drivers, that can be loaded and unloaded without re-starting the system) are located in ``/lib/modules/``. 81 | 82 | ### Additional directories 83 | 84 | |Directory|Usage| 85 | |---------|-----| 86 | | /opt | Optional application software packages | 87 | | /sys | Virtual pseudo-filesystem giving information about the system and the hardware. Can be used to alter system parameters and for debugging purposes. | 88 | | /srv | Site-specific data served up by the system. Seldom used. | 89 | | /tmp | Temporary files; on some distributions these files are erased across a reboot | 90 | | /media | It is typically located where removable media, such as CDs, DVDs and USB drives are mounted. Unless configuration prohibits it, Linux automatically mounts the removable media in this directory when they are detected. | 91 | | /usr | Multi-user applications, utilities and data | 92 | | /usr/include | Header files used to compile applications | 93 | | /usr/lib | Libraries for binary programs | 94 | | /usr/lib64 | 64bit Libraries for binary programs | 95 | | /usr/share | Shared data used by applications, generally architecture-independent | 96 | | /usr/src | Source code, usually for the Linux kernel | 97 | | /usr/local | Data and programs specific to the local machine. | 98 | 99 | ### File System Table 100 | for details on the file system table, i.e. the ``/etc/fstab`` file, please see [fstab (Italian)](https://wiki.archlinux.org/index.php/Fstab_%28Italiano%29#Dischi_esterni) 101 | 102 | -------------------------------------------------------------------------------- /content/network-namespaces.md: -------------------------------------------------------------------------------- 1 | ### Linux Network Namespaces 2 | Dedicated networking devices use Virtual Routing and Forwarding (VRF), meaning that more than one virtual router (Layer 3 forwarding instance) can be run on the same physical device. In the Linux virtual networking space, the network namespaces allow separate instances of network interfaces and routing tables to operate independent of each other. 3 | 4 | #### Basic operations on namespaces 5 | In Linux, yust be root for all operations which change the configuration of the network stack. 6 | 7 | Creating a network namespace 8 | 9 | [root@centos-01 ~]# ip netns add Blue 10 | [root@centos-01 ~]# ip netns list 11 | Blue 12 | [root@centos-01 network-scripts]# ll /var/run/netns/ 13 | total 0 14 | -r--r--r-- 1 root root 0 Feb 11 16:29 Blue 15 | 16 | Each network namespace has its own loopback interface, its own routing table and its own iptables setup providing nat and filtering. 17 | 18 | [root@centos-01 ~]# ip netns exec Blue ip addr list 19 | 1: lo: mtu 65536 qdisc noop state DOWN 20 | link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 21 | 22 | Make sure to bring up that interface before to operate with the network namespace 23 | 24 | [root@centos-01 ~]# ip netns exec Blue ip link set dev lo up 25 | [root@centos-01 ~]# ip netns exec Blue ifconfig 26 | lo: flags=73 mtu 65536 27 | inet 127.0.0.1 netmask 255.0.0.0 28 | inet6 ::1 prefixlen 128 scopeid 0x10 29 | loop txqueuelen 0 (Local Loopback) 30 | RX packets 0 bytes 0 (0.0 B) 31 | RX errors 0 dropped 0 overruns 0 frame 0 32 | TX packets 0 bytes 0 (0.0 B) 33 | TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 34 | 35 | Network namespaces offer in addition the capability to run processes within the network namespace. For example, run a bash session in the Blue namespace 36 | 37 | [root@centos-01 ~]# ip netns exec Blue bash 38 | [root@centos-01 ~]# ifconfig 39 | lo: flags=73 mtu 65536 40 | inet 127.0.0.1 netmask 255.0.0.0 41 | inet6 ::1 prefixlen 128 scopeid 0x10 42 | loop txqueuelen 0 (Local Loopback) 43 | RX packets 0 bytes 0 (0.0 B) 44 | RX errors 0 dropped 0 overruns 0 frame 0 45 | TX packets 0 bytes 0 (0.0 B) 46 | TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 47 | [root@centos-01 ~]# netstat -nr 48 | Kernel IP routing table 49 | Destination Gateway Genmask Flags MSS Window irtt Iface 50 | [root@centos-01 ~]# exit 51 | 52 | Delete the namespace 53 | 54 | [root@centos-01 ~]# ip netns add Yellow 55 | [root@centos-01 ~]# ip netns list 56 | Yellow 57 | Blue 58 | [root@centos-01 ~]# ip netns delete Yellow 59 | [root@centos-01 ~]# ip netns list 60 | Blue 61 | 62 | #### Add interfaces to network namespaces 63 | To connect a network namespace to the outside world, attach a virtual interface to the “default” or “global” namespace where physical interfaces exist. To accomplish this, let's to create a couple of virtual interfaces, called ``vetha`` and ``vethb`` 64 | 65 | [root@centos-01 ~]# ip link add vetha type veth peer name vethb 66 | 67 | Attach ``vethb`` to the Blue namespace 68 | 69 | [root@centos-01 ~]# ip link set vethb netns Blue 70 | [root@centos-01 ~]# ip netns exec Blue ip link set dev vethb up 71 | [root@centos-01 ~]# ip netns exec Blue ifconfig 72 | lo: flags=73 mtu 65536 73 | inet 127.0.0.1 netmask 255.0.0.0 74 | inet6 ::1 prefixlen 128 scopeid 0x10 75 | loop txqueuelen 0 (Local Loopback) 76 | vethb: flags=4099 mtu 1500 77 | ether 7e:e4:29:bc:9c:67 txqueuelen 1000 (Ethernet) 78 | 79 | Virtual network interface ``vetha`` remain attacched to the global namespace 80 | 81 | [root@centos-01 ~]# ip link set dev vetha up 82 | [root@centos-01 ~]# ifconfig 83 | ens32: flags=4163 mtu 1500 84 | inet 10.10.10.21 netmask 255.255.255.0 broadcast 10.10.10.255 85 | inet6 fe80::20c:29ff:fe1e:6bf1 prefixlen 64 scopeid 0x20 86 | ether 00:0c:29:1e:6b:f1 txqueuelen 1000 (Ethernet) 87 | lo: flags=73 mtu 65536 88 | inet 127.0.0.1 netmask 255.0.0.0 89 | inet6 ::1 prefixlen 128 scopeid 0x10 90 | loop txqueuelen 0 (Local Loopback) 91 | vetha: flags=4163 mtu 1500 92 | inet6 fe80::e899:ceff:fef6:3010 prefixlen 64 scopeid 0x20 93 | ether ea:99:ce:f6:30:10 txqueuelen 1000 (Ethernet) 94 | 95 | Configure the virtual interface in global network namespace 96 | 97 | [root@centos-01 ~]# ip addr add 192.168.100.1/24 dev vetha 98 | [root@centos-01 ~]# route 99 | Kernel IP routing table 100 | Destination Gateway Genmask Flags Metric Ref Use Iface 101 | default gateway 0.0.0.0 UG 100 0 0 ens32 102 | 10.10.10.0 0.0.0.0 255.255.255.0 U 100 0 0 ens32 103 | 192.168.100.0 0.0.0.0 255.255.255.0 U 0 0 0 vetha 104 | [root@centos-01 ~]# 105 | 106 | and in the Blue network namespace 107 | 108 | [root@centos-01 ~]# ip netns exec Blue ip addr add 192.168.100.2/24 dev vethb 109 | [root@centos-01 ~]# ip netns exec Blue route 110 | Kernel IP routing table 111 | Destination Gateway Genmask Flags Metric Ref Use Iface 112 | 192.168.100.0 0.0.0.0 255.255.255.0 U 0 0 0 vethb 113 | [root@centos-01 ~]# 114 | 115 | Both the namespaces, Blue and global are now reachable each other via virtual network interfaces 116 | 117 | [root@centos-01 ~]# ping 192.168.100.2 118 | PING 192.168.100.2 (192.168.100.2) 56(84) bytes of data. 119 | 64 bytes from 192.168.100.2: icmp_seq=1 ttl=64 time=0.041 ms 120 | 64 bytes from 192.168.100.2: icmp_seq=2 ttl=64 time=0.029 ms 121 | ^C 122 | [root@centos-01 ~]# ip netns exec Blue ping 192.168.100.1 123 | PING 192.168.100.1 (192.168.100.1) 56(84) bytes of data. 124 | 64 bytes from 192.168.100.1: icmp_seq=1 ttl=64 time=0.034 ms 125 | 64 bytes from 192.168.100.1: icmp_seq=2 ttl=64 time=0.039 ms 126 | ^C 127 | 128 | But they are completly separated routing entities 129 | 130 | [root@centos-01 ~]# ip netns exec Blue ping 10.10.10.1 131 | connect: Network is unreachable 132 | [root@centos-01 ~]# 133 | [root@centos-01 ~]# ping 10.10.10.1 134 | PING 10.10.10.1 (10.10.10.1) 56(84) bytes of data. 135 | 64 bytes from 10.10.10.1: icmp_seq=1 ttl=64 time=0.545 ms 136 | 64 bytes from 10.10.10.1: icmp_seq=2 ttl=64 time=0.369 ms 137 | ^C 138 | -------------------------------------------------------------------------------- /content/nfs.md: -------------------------------------------------------------------------------- 1 | ## Network Filesystem 2 | Using **NFS** (the Network File System) is one of the methods used for sharing data across physical systems. Many system administrators mount remote users' home directories on a server in order to give them access to the same files and configuration files across multiple client systems. This allows the users to log in to different machines yet still have access to the same files and resources. 3 | 4 | On a generic Linux distribution, the NFS server daemon is typically started with the command ``service nfs start``. The file ``/etc/exports`` contains the directories and permissions that a host is willing to share with other systems over NFS. An entry in this file may look like ``/shared *(rw)``. This entry allows the directory ``/shared`` to be mounted using NFS with read and write (rw) permissions and shared with other hosts in the same domain. After modifying the ``/etc/exports`` file, you can use the ``exportfs -av`` command to notify Linux about the directories you are allowing to be remotely mounted using NFS. 5 | 6 | On the client machine, if it is desired to have the remote filesystem mounted automatically upon system boot, the ``/etc/fstab`` file is modified to accomplish this. For example, an entry in the client's ``/etc/fsta``b file might look like ``:/shared /mnt/nfs/shared nfs defaults 0 0``. You can also mount the remote filesystem without a reboot or as a one-time mount by directly using the ``mount`` command. If ``/etc/fstab`` is not modified, this remote mount will not be present the next time the system is restarted. 7 | 8 | On RedHat based distributions (CentOS-7) NFS server 9 | ``` 10 | # yum install -y nfs-utils 11 | # mkdir /var/shared 12 | ``` 13 | Add an entry into the ``/etc/exports`` file 14 | ``` 15 | # vi /etc/exports 16 | # /var/shared 10.10.10.0/24(no_root_squash,no_all_squash,rw,sync) 17 | ``` 18 | Where: 19 | * ``/var/shared`` is the shared folder 20 | * ``10.10.10.0/24`` is IP address range of clients 21 | * ``rw`` is the permission to shared folder 22 | * ``sync`` synchronizes shared folder 23 | * ``root_squash`` disable the root privilege 24 | * ``no_root_squash`` enables the root privilege 25 | * ``no_all_squash`` enables the user’s authority 26 | 27 | The ``no_root_squash`` option leaves root users on NFS clients to write files as root user on the NFS server. Default is ``root_squash``. 28 | 29 | ``` 30 | # systemctl start rpcbind 31 | # systemctl start nfs-server 32 | # systemctl enable rpcbind 33 | # systemctl enable nfs-server 34 | # systemctl status rpcbind 35 | # systemctl status nfs-server 36 | ``` 37 | 38 | On the client machine mount the shared folder to a local folder 39 | ``` 40 | # mkdir -p /mnt/nfs 41 | # mount 10.10.10.97:/var/shared /mnt/nfs 42 | # cd /mnt/nfs 43 | # touch filename.txt 44 | ``` 45 | **Note**: this is only for explanation. Please, do not use it in production systems. Check the NFS resources related to your distribution. 46 | 47 | To run a NFS server behind the firewall, you should make some changes on the NFS configuration file, e.g. ``/etc/sysconfig/nfs`` on Red Hat/CentOS family, and then enable the ports on the firewall configuration. The reason is that NFS requires the ``rpcbind`` service, which dynamically assigns ports for RPC services and can cause problems for configuring firewall rules. See: [http://initrd.org/wiki/NFS_Setup](http://initrd.org/wiki/NFS_Setup) 48 | -------------------------------------------------------------------------------- /content/package_management.md: -------------------------------------------------------------------------------- 1 | ### Package Management Systems 2 | The core parts of a Linux distribution and most of its add-on software are installed via the Package Management System. Each package contains the files and other instructions needed to make one software component work on the system. Packages can depend on each other. There are two broad families of package managers: those based on **dpkg** and those which use **rpm** as their low-level package manager. The two systems are incompatible, but provide the same features at a broad level. 3 | 4 | **Package Management Systems** 5 | 6 | |High Level Tool|Low Level Tool|Family| 7 | |---------------|--------------|------| 8 | |apt-get|dpkg|Debian| 9 | |zypper|rpm|SUSE| 10 | |yum|rpm|Red Hat| 11 | 12 | Both package management systems provide two tool levels: a low-level tool (such as ``dpkg`` or ``rpm``), takes care of the details of unpacking individual packages, running scripts, getting the software installed correctly, while a high-level tool (such as ``apt-get``, ``yum``, or ``zypper``) works with groups of packages, downloads packages from the vendor, and figures out dependencies. Most of the time users need work only with the high-level tool, which will take care of calling the low-level tool as needed. Dependency tracking is a particularly important feature of the high-level tool, as it handles the details of finding and installing each dependency for you. Be careful, however, as installing a single package could result in many dozens or even hundreds of dependent packages being installed. 13 | 14 | |Operation|RPM|Debian| 15 | |---------|-----------|-----------| 16 | |Install a package|rpm –i foo.rpm|dpkg --install foo.deb| 17 | |Install a package with dependencies from repository|yum install foo|apt-get install foo| 18 | |Remove a package|rpm –e foo.rpm|dpkg --remove foo.deb| 19 | |Remove a package and dependencies using repository|yum remove foo|apt-get remove foo| 20 | |Update package to a newer version|rpm –U foo.rpm|dpkg --install foo.deb| 21 | |Update package using repository and resolving dependencies|yum update foo|apt-get upgrade foo| 22 | |Update entire system|yum update|apt-get dist-upgrade| 23 | |Show all installed packages|yum list installed|dpkg --list| 24 | |Get information about an installed package including files|rpm –qil foo|dpkg --listfiles foo| 25 | |Show available package with "foo" in name|yum list foo|apt-cache search foo| 26 | |Show all available packages|yum list|apt-cache dumpavail| 27 | |Show packages a file belong to|rpm –qf file|dpkg --search file| 28 | -------------------------------------------------------------------------------- /content/processes.md: -------------------------------------------------------------------------------- 1 | ### Linux processes 2 | A **process** is simply an instance of one or more related tasks (**threads**) executing on the same machine. It is not the same as a program or a command; a single program may actually start several processes simultaneously. Some processes are independent of each other and others are related. A failure of one process may or may not affect the others running on the system. Processes use many system resources, such as memory, CPU cycles, and peripheral devices such as printers and displays. The operating system (especially the kernel) is responsible for allocating a proper share of these resources to each process and ensuring overall optimum utilization. 3 | 4 | A terminal window, is a process that runs as long as needed. It allows users to execute programs and access resources in an interactive environment. You can also run programs in the background, which means they become detached from the shell. Processes can be of different types according to the task being performed. 5 | 6 | |Type|Description| 7 | |--------|---------| 8 | |Interactive |Need to be started by a user, either at a command line or through a graphical interface such as an icon or a menu selection.| 9 | |Batch |Automatic processes which are scheduled from and then disconnected from the terminal. These tasks are queued and work on a FIFO (First In, First Out) basis.| 10 | |Daemons|Server processes that run continuously. Many are launched during system startup and then wait for a user or system request indicating that their service is required.| 11 | |Threads|Lightweight processes. These are tasks that run under the umbrella of a main process, sharing memory and other resources, but are scheduled and run by the system on an individual basis.| 12 | |Kernel Threads|Kernel tasks that users neither start nor terminate and have little control over. These may perform actions like moving a thread from one CPU to another, or making sure input/output operations to disk are completed.| 13 | 14 | When a process is in the **running state**, it means it is either currently executing instructions on a CPU, or is waiting for a share (or time slice) so it can run. A critical kernel routine called the **scheduler** constantly shifts processes in and out of the CPU, sharing time according to relative priority, how much time is needed and how much has already been granted to a task. All processes in this state reside on a run queue and on a computer with multiple CPUs there is a run queue on each. Sometimes processes go into the **sleep** state, generally when they are waiting for something to happen before they can resume, perhaps for the user to type something. In this condition a process is sitting in a wait queue. There are some other less frequent process states, especially when a process is terminating. Sometimes a child process completes but its parent process has not asked about its state. Such a process is said to be in a **zombie** state; it is not really alive but still shows up in the system's list of processes. 15 | 16 | At any given time there are always multiple processes being executed. The operating system keeps track of them by assigning each a unique process ID or **PID** number. The PID is used to track process state, cpu usage, memory use, precisely where resources are located in memory, and other characteristics. New PIDs are usually assigned in ascending order as processes are born. Thus PID 1 denotes the **init** process (initialization process), and succeeding processes are gradually assigned higher numbers. 17 | 18 | At any given time, many processes are running on the system. However, a **CPU** can actually accommodate only one task at a time, just like a car can have only one driver at a time. Some processes are more important than others so Linux allows you to set and manipulate process priority. Higher priority processes are granted more time on the processor. The **priority** for a process can be set by specifying a nice value, or **niceness**, for the process. The lower the nice value, the higher the priority. Low values are assigned to important processes, while high values are assigned to processes that can wait longer. A process with a high nice value simply allows other processes to be executed first. In Linux, a nice value of -20 represents the highest priority and 19 represents the lowest. You can also assign a real-time priority to time-sensitive tasks, such as controlling machines or collecting incoming data. This is just a very high priority and is not to be confused with what is called hard real time which is conceptually different, and has more to do with making sure a job gets completed within a very well-defined time window. 19 | 20 | ### Running processes 21 | The ``ps`` command provides information about currently running processes, keyed by **PID**. If you want a repetitive update of this status, you can use the ``top`` command or commonly installed variants such as ``htop`` or ``atop`` from the command line. The ``ps`` command has many options for specifying exactly which tasks to examine, what information to display about them, and precisely what output format should be used. 22 | 23 | Without options ``ps`` will display all processes running under the current shell. You can use the `` ps -u`` to display information of processes for a specified username. The command ``ps -ef`` displays all the processes in the system in full detail. The command ``ps -eLf`` goes one step further and displays one line of information for every thread (a process can contain multiple threads). 24 | 25 | ``` 26 | # ps -u adriano 27 | PID TTY TIME CMD 28 | 847 ? 00:00:00 sshd 29 | 848 pts/2 00:00:00 bash 30 | 1070 ? 00:00:00 sshd 31 | 1071 pts/3 00:00:00 bash 32 | 6475 pts/3 00:00:00 top 33 | ``` 34 | 35 | The ``pstree`` command displays the processes running on the system in the form of a tree diagram showing the relationship between a process and its parent process and any other processes that it created. Repeated entries of a process are not displayed, and threads are displayed in curly braces. 36 | ``` 37 | # yum install -y psmisc 38 | # pstree 39 | # systemd─┬─agetty 40 | ├─auditd───{auditd} 41 | ├─avahi-daemon───avahi-daemon 42 | ├─crond 43 | ├─dbus-daemon───{dbus-daemon} 44 | ├─firewalld───{firewalld} 45 | ├─iprdump 46 | ├─iprinit 47 | ├─iprupdate 48 | ├─lvmetad 49 | ├─master─┬─pickup 50 | │ └─qmgr 51 | ├─polkitd───5*[{polkitd}] 52 | ├─rsyslogd───2*[{rsyslogd}] 53 | ├─sshd───sshd───bash───pstree 54 | ├─systemd-journal 55 | ├─systemd-logind 56 | ├─systemd-udevd 57 | └─tuned───4*[{tuned}] 58 | ``` 59 | 60 | To terminate a process you can type ``kill -SIGKILL `` or ``kill -9 ``. Note however, you can only kill your own processes: those belonging to another user are off limits unless you are root. 61 | 62 | While a static view of what the system is doing is useful, monitoring the system performance live over time is also valuable. One option would be to run the ``ps`` command at regular intervals. A better alternative is to use ``top`` to get constant real-time updates (every two seconds by default). The ``top`` command clearly highlights which processes are consuming the most CPU cycles and memory. 63 | ``` 64 | top - 15:40:31 up 4 days, 2:13, 1 user, load average: 0.77, 0.66, 0.45 65 | Tasks: 244 total, 2 running, 241 sleeping, 0 stopped, 1 zombie 66 | %Cpu(s): 6.5 us, 1.3 sy, 0.0 ni, 88.3 id, 3.7 wa, 0.0 hi, 0.2 si, 0.0 st 67 | KiB Mem: 3801380 total, 3642652 used, 158728 free, 24 buffers 68 | KiB Swap: 4079612 total, 3072 used, 4076540 free. 326620 cached Mem 69 | 70 | PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 71 | 1367 glance 20 0 351800 54996 5928 S 1.3 1.4 64:32.28 glance-api 72 | 1373 nova 20 0 383444 73304 6768 S 1.3 1.9 68:25.51 nova-api 73 | 1365 keystone 20 0 353108 58340 6192 S 1.0 1.5 67:16.01 keystone-all 74 | 1369 cinder 20 0 365404 60120 6632 S 1.0 1.6 68:38.74 cinder-api 75 | 1371 cinder 20 0 287924 30584 4924 S 1.0 0.8 68:08.01 cinder-volume 76 | 1380 nova 20 0 348120 46568 6580 S 1.0 1.2 68:19.02 nova-conductor 77 | 1408 ceilome+ 20 0 254312 20560 4260 R 1.0 0.5 64:28.69 ceilometer-agen 78 | ... 79 | ``` 80 | The first line of the ``top`` output displays a quick summary of what is happening in the system including: 81 | 82 | 1. How long the system has been up 83 | 2. How many users are logged on 84 | 3. What is the load average 85 | 86 | The load average determines how busy the system is. A load average of 1.00 per CPU indicates a fully subscribed, but not overloaded, system. If the load average goes above this value, it indicates that processes are competing for CPU time. If the load average is very high, it might indicate that the system is having a problem, such as a **runaway** process (a process in a non-responding state). 87 | 88 | The second line of the ``top`` output displays the total number of processes, the number of running, sleeping, stopped and zombie processes. Comparing the number of running processes with the load average helps determine if the system has reached its capacity or perhaps a particular user is running too many processes. The stopped processes should be examined to see if everything is running correctly. 89 | 90 | The third line of the ``top`` output indicates how the CPU time is being divided between the users (**us**) and the kernel (**sy**) by displaying the percentage of CPU time used for each. The percentage of user jobs running at a lower priority (**ni**) is then listed. Idle mode (**id**) should be low if the load average is high, and vice versa. The percentage of jobs waiting (**wa**) for I/O is listed. Interrupts include the percentage of hardware (**hi**) vs. software interrupts (**si**). Steal time (**st**) is generally used with virtual machines, which has some of its idle CPU time taken for other uses. 91 | 92 | The fourth and fifth lines of the ``top`` output indicate memory usage, which is divided in two categories: 93 | 94 | 1. Physical memory (RAM) – displayed on line 4. 95 | 2. Swap space – displayed on line 5. 96 | 3. Both categories display total memory, used memory, and free space. 97 | 98 | You need to monitor memory usage very carefully to ensure good system performance. Once the physical memory is exhausted, the system starts using swap space as an extended memory pool, and since accessing disk is much slower than accessing memory, this will negatively affect system performance. If the system starts using swap often, you can add more swap space. However, adding more physical memory should also be considered. 99 | 100 | Each line in the process list of the ``top`` output displays information about a process. By default, processes are ordered by highest CPU usage. The following information about each process is displayed: 101 | 102 | * Process Identification Number (PID) 103 | * Process owner (USER) 104 | * Priority (PR) and nice values (NI) 105 | * Virtual (VIRT), physical (RES), and shared memory (SHR) 106 | * Status (S) 107 | * Percentage of CPU (%CPU) and memory (%MEM) used 108 | * Execution time (TIME+) 109 | * Command (COMMAND) 110 | 111 | To control the healt of a system, the average load of the system should be checked first. Assuming our system is a single-CPU system, the 0.25 means that for the past minute, on average, the system has been 25% utilized. 0.12 in the next position means that over the past 5 minutes, on average, the system has been 12% utilized; and 0.15 in the final position means that over the past 15 minutes, on average, the system has been 15% utilized. If we saw a value of 1.00 in the second position, that would imply that the single-CPU system was 100% utilized, on average, over the past 5 minutes; this is good if we want to fully use a system. A value over 1.00 for a single-CPU system implies that the system was over-utilized: there were more processes needing CPU than CPU was available. If we had more than one CPU, say a quad-CPU system, we would divide the load average numbers by the number of CPUs. In this case, for example, seeing a 1 minute load average of 4.00 implies that the system as a whole was 100% (4.00/4) utilized during the last minute. Short term increases are usually not a problem. A high peak you see is likely a burst of activity, not a new level. For example, at start up, many processes start and then activity settles down. If a high peak is seen in the 5 and 15 minute load averages, it would may be cause for concern. 112 | 113 | ### Background and foreground processes 114 | Linux supports **background** and **foreground** job processing. Foreground jobs run directly from the shell, and when one foreground job is running, other jobs need to wait for shell access until it is completed. This is fine when jobs complete quickly. But this can have an adverse effect if the current job is going to take a long time to complete. In such cases, you can run the job in the background and free the shell for other tasks. The background job will be executed at lower priority, which, in turn, will allow smooth execution of the interactive tasks, and you can type other commands in the terminal window while the background job is running. By default all jobs are executed in the foreground. This You can put a job in the background: 115 | 116 | ``` 117 | # updatedb & 118 | [1] 7437 119 | # jobs 120 | [1]+ Done updatedb 121 | # 122 | ``` 123 | 124 | ### Scheduling processes 125 | The ``at`` utility program is used to execute any non-interactive command at a specified time. The ``at`` jobs is picked by the ``atd`` service. 126 | ``` 127 | # yum install -y at 128 | # systemctl start atd 129 | # systemctl enable atd 130 | # at now + 5 minutes 131 | at> pstree 132 | at> 133 | job 9 at Sat Feb 21 16:28:00 2015 134 | ``` 135 | 136 | The ``atq`` command is used to list the scheduled jobs by the ``at`` command. 137 | ``` 138 | # atq 139 | 9 Sat Feb 21 16:28:00 2015 a root 140 | ``` 141 | 142 | The ``cron`` utility is a time-based scheduling utility program. It can launch routine background jobs at specific times and or days on an on-going basis. cron is driven by a configuration file called ``/etc/crontab`` which contains the various shell commands that need to be run at the properly scheduled times. There are both system-wide crontab files and individual user-based ones. Each line of a crontab file represents a job, and is composed of an expression, followed by a shell command to execute. The ``crontab -e`` command will open the crontab editor to edit existing jobs or to create new jobs. Each line of the crontab file will contain 6 fields: 143 | 144 | 1. MIN Minutes 0 to 59 145 | 2. HOUR Hour field 0 to 23 146 | 3. DOM Day of Month 1-31 147 | 4. MON Month field 1-12 148 | 5. DOW Day Of Week 0-6 (0 = Sunday) 149 | 6. CMD Command Any command to be executed 150 | 151 | For example, the entry 152 | ``` 153 | * * * * * /usr/local/bin/execute/this/script.sh 154 | ``` 155 | will schedule a job to execute the script every minute of every hour of every day of the month, and every month and every day in the week. The entry 156 | ``` 157 | 30 08 10 06 * /home/sysadmin/full-backup 158 | ``` 159 | will schedule a full-backup at 8.30am, 10-June irrespective of the day of the week. 160 | 161 | ### Delaying processes 162 | Sometimes a command or job must be delayed or suspended. Suppose, for example, an application has read and processed the contents of a data file and then needs to save a report on a backup system. If the backup system is currently busy or not available, the application can be made to sleep until it can complete its work. Such a delay might be to mount the backup device and prepare it for writing. The ``sleep`` command suspends execution for at least the specified period of time, which can be given as the number of seconds (the default), minutes, hours or days. After that time has passed, the execution will resume. 163 | 164 | ``` 165 | # vi script.sh 166 | #!/bin/bash 167 | echo "The system will go to sleep fo 30 seconds ..." 168 | sleep 15 169 | echo "The system is awaked" 170 | # chmod u+x script.sh 171 | # ./script.sh 172 | The system will go to sleep fo 30 seconds ... 173 | The system is awaked 174 | # 175 | ``` 176 | -------------------------------------------------------------------------------- /content/samba_server.md: -------------------------------------------------------------------------------- 1 | ## Samba server and Windows file sharing 2 | Samba is an open source implementation of the SMB/CIFS protocol. It allows the networking of Microsoft Windows®, Linux, UNIX, and other operating systems. Samba allows a Linux/Unix server to appear as a Windows server to Windows clients. 3 | 4 | With Samba, an administrator can do: 5 | 6 | 1. Serve directory trees and printers to Linux, UNIX, and Windows clients 7 | 2. Assist in network browsing with or without NetBIOS 8 | 3. Authenticate Windows domain logins 9 | 4. Provide WINS name server resolution 10 | 11 | Samba is comprised of **smb**, **nmb**, and **winbind** services. 12 | 13 | The ``smbd`` server daemon provides file sharing and printing services to Windows clients. In addition, it is responsible for user authentication, resource locking, and data sharing through the SMB protocol. The default ports on which the server listens for SMB traffic are TCP ports 139 and 445. 14 | 15 | The ``nmbd`` server daemon understands and replies to NetBIOS name service requests produced by SMB in Windows-based systems. The default port that the server listens to for NMB traffic is UDP port 137. 16 | 17 | The ``winbindd`` service resolves user and group information received from a server running Windows. This makes Windows user and group information understandable by Linux and UNIX platforms. This allows Windows domain users to appear and operate as Linux and UNIX users on a Linux or UNIX machine. Both ``winbindd`` and ``smbd`` are bundled with the Samba distribution, but the ``winbindd`` service is controlled separately from the ``smbd`` service. 18 | 19 | #### Setup a Samba server 20 | We'll setup a Samba server to make Linux file sharing available to Windows clients. Install the Samba package, enable and start the ``smbd`` and ``nmbd`` services 21 | 22 | ``` 23 | # yum install samba 24 | # systemctl enable smb 25 | # systemctl enable nmb 26 | # systemctl start smb 27 | # systemctl start nmb 28 | ``` 29 | 30 | Samba uses ``/etc/samba/smb.conf`` as its configuration file. 31 | 32 | ``` 33 | # mv /etc/samba/smb.conf /etc/samba/smb.conf.orig 34 | # vi /etc/samba/smb.conf 35 | 36 | # =============== Global configuration =============== 37 | [global] 38 | ; Windows workgroup name and server description 39 | workgroup = WORKGROUP 40 | server string = My SMB Server %v 41 | ; NetBIOS name as the Linux machine will appear in Windows clients 42 | netbios name = MYSMBSERVER 43 | ; interfaces where the service is listening: localhost and ens32 interfaces 44 | interfaces = lo ens32 45 | ; users passwords database backend and location 46 | passdb backend = smbpasswd 47 | smb passwd file = /etc/samba/smbpasswd 48 | ; permitted hosts to use the Samba server: localhost and all host belonging to 10.10.10.0/24 subnet 49 | hosts allow = 127. 10.10.10. 50 | ; protocol version 51 | max protocol = SMB3 52 | ; type of security 53 | security = user 54 | ; no printing services 55 | printing = bsd 56 | printcap name = /dev/null 57 | 58 | # =============== Shares configuration =============== 59 | [share1] 60 | comment = Private Documents 61 | ; path of files to share 62 | path = /samba/admin/data 63 | ; users admitted to use the file sharing service 64 | valid users = admin 65 | invalid users = user2 user3 66 | ; no guest user is admitted 67 | guest ok = no 68 | ; make the share writable as Samba make it as readonly by default 69 | writable = yes 70 | ; make the share visible as shared folder 71 | browsable = yes 72 | 73 | [share2] 74 | comment = Public Documents 75 | path = /samba/user2/data 76 | valid users = user2 admin 77 | guest ok = no 78 | writable = yes 79 | browsable = yes 80 | 81 | [share3] 82 | comment = Public Documents 83 | path = /samba/user3/data 84 | valid users = user3 admin 85 | guest ok = no 86 | writable = yes 87 | browsable = yes 88 | ``` 89 | 90 | The Samba configuration file can be checked by the ``testparm`` command 91 | ``` 92 | # testparm /etc/samba/smb.conf 93 | Load smb config files from /etc/samba/smb.conf 94 | rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384) 95 | Processing section "[homes]" 96 | Processing section "[admin]" 97 | Processing section "[guest]" 98 | Loaded services file OK. 99 | Server role: ROLE_STANDALONE 100 | Press enter to see a dump of your service definitions 101 | ``` 102 | 103 | #### User access 104 | More than one user can be admitted to access the same share. In the case above, the share1 is only accesible to the "admin" user. The share2 is accessible to "admin" and "user2" users but not "user3". The share3 is accessible to "admin" and "user3" to "user2". 105 | 106 | **Note:** the connection to shares by the same Windows client needs to use the same user name. In our case, a Windows client can access all the shares above as "admin" but cannot access to share2 as "user2" AND access to share3 as "user3". If the Windows client needs to access with different users, it needs to logout from the previous user and then login again with a different user. Since Windows caches the login user, it needs to force the logout by issuing the command: ``net use * /delete`` from the Windows command shell 107 | 108 | ``` 109 | Microsoft Windows [Versione 10.0.10240] 110 | (c) 2015 Microsoft Corporation. Tutti i diritti sono riservati. 111 | C:\Users\Adriano>net use * /delete 112 | Connessioni remote presenti: 113 | \\10.10.10.12\IPC$ 114 | Continuando si annulleranno le connessioni. 115 | Continuare questa operazione? (S/N) [N]: S 116 | Esecuzione comando riuscita. 117 | ``` 118 | Samba uses different type of security. In the case above, the method is based on user level (default). With this method, each share is assigned specific users that can access it. When a user requests a connection to a share, Samba authenticates by validating the given username and password with the authorized users in the configuration file and the passwords in the password database of the Samba server. 119 | 120 | Samba uses different database backends for storing users passwords. The simplest is store the password in a file called ``smbpasswd`` similar to the ``/etc/passwd`` file. Usually this file is located under ``/var/lib/samba/private/smbpasswd`` but location can be changed. 121 | 122 | Add the user and set password in the Samba user database 123 | 124 | ``` 125 | # smbpasswd -a admin 126 | New SMB password: 127 | Retype new SMB password: 128 | # 129 | ``` 130 | The ``pdbedit`` command lists the Samba users database 131 | 132 | ``` 133 | # pdbedit -L 134 | admin:1000: 135 | user1:1001: 136 | user2:1002: 137 | user3:1003: 138 | ``` 139 | 140 | Other security methods: domain and server level security are deprecated in latest Samba. 141 | 142 | With smbpasswd database backend, a Samba user should exist as valid user in the Linux machine. To secure the Linux machine preventing login from Samba users, you should disable the login from these users 143 | ``` 144 | # useradd -d /samba/share user1 145 | # usermod -s /bin/false user1 146 | # cat /etc/passwd | grep user1 147 | user1:x:1003:1002::/samba/share:/sbin/nologin 148 | # 149 | # ssh user1@localhost 150 | user1@localhost's password: 151 | Last login: Tue Sep 15 11:50:08 2015 152 | This account is currently not available. 153 | Connection to localhost closed. 154 | # 155 | # sftp user1@localhost 156 | user1@localhost's password: 157 | subsystem request failed on channel 0 158 | Couldn't read packet: Connection reset by peer 159 | ``` 160 | Alternatively, you can leave the ssh but should chroot the user's home directory. 161 | 162 | #### File permissions and attributes 163 | In our example above, we are going to share Linux files and folders to Windows clients. Since Windows and Linux use different approach to file permissions and attributes, Samba will take care of mapping the two approaches. 164 | 165 | All Linux files have read, write, and execute bits for three classifications of users: owner (u), group (g), and rest of the world (o). Windows, on the other hand, has four principal bits that it uses with any file: read-only, system, hidden, and archive: 166 | 167 | 1. Read-only. The file's contents can be read by a user but cannot be written to. 168 | 2. System. This file has a specific purpose required by the operating system. 169 | 3. Hidden. This file has been marked to be invisible to the user, unless the operating systems is explicitly set to show it. 170 | 4. Archive. This file has been touched since the last backup was performed on it. 171 | 172 | There is no bit to specify that a file is executable since Windows identifies executable files by looking at the file extension. Windows files stored on a Linux Samba share have their own attributes that need to be preserved. Samba preserves these bits by reusing the Linux executable permission bits of the file, if it is instructed to do so. Mapping these bits, however, has a side-effect: if a Windows user stores a file in a Samba share, at Linux side, some of the executable bits are set. 173 | 174 | The Samba options deciding the mapping 175 | ``` 176 | [share] 177 | ... 178 | store dos attributes = yes 179 | map archive = yes ;default is yes 180 | map system = yes ;default is no 181 | map hidden = yes ;default is no 182 | ``` 183 | The last three options map the archive, system, and hidden attributes to the owner, group, and world execute bits of the file, respectively. In the example above, the options are used on a per-share basis. Setting them globally makes them the default for all shares. The first option also makes sure that Samba does not change the Windows permission bits. 184 | 185 | **Note:** These options can be used if the Linux file system supports the extended attributes, and those attributes are enabled, usually via the ``user_xattr`` mount option in the ``/etc/fstab`` file. Unlike _ext3_ and _ext4_, the _xfs_ file system enables the ``user_xattr`` option by default. 186 | 187 | Samba has the ``create mask`` and the ``directory mask`` options to help with files and folders creation. The creation masks help to define the permissions a file or directory at the time it is created. On the Linux side, you can control what permissions a file or directory have when it is created. On the Windows side, you can disable the read-only, archive, system, and hidden attributes of a file as well. 188 | 189 | ``` 190 | [share] 191 | ... 192 | store dos attributes = yes 193 | map archive = yes ;default is yes 194 | map system = yes ;default is no 195 | map hidden = yes ;default is no 196 | create mask = 0744 ;default is 0744 197 | directory mask = 0755 ;default is 0755 198 | ``` 199 | 200 | On the Linux side, new files and folders will look like 201 | 202 | ``` 203 | # ll /samba/share/user1 204 | total 0 205 | -rwxr--r-- 1 user1 samba 0 Sep 15 13:00 mydocument.txt 206 | drwxr-xr-x 2 user1 samba 6 Sep 15 13:00 myfolder 207 | ``` 208 | 209 | It is possible force various bits with the ``force create mode`` and ``force directory mode`` options. With the ``create mask`` and ``create directory mask`` options, the administrator allow the permission bits to be set by the requested user. On the other side, the ``force create mode`` and ``force directory mode`` will force a particular bit to be set, even if it wasn’t requested by the user. 210 | 211 | At the same time, it is possible to force the Linux user and group attributes of a file that is created on the Windows side by the ``force user`` and the ``force group`` options. 212 | 213 | ``` 214 | [share] 215 | ... 216 | store dos attributes = yes 217 | map archive = yes ;default is yes 218 | map system = yes ;default is no 219 | map hidden = yes ;default is no 220 | create mask = 0744 ;default is 0744 221 | directory mask = 0755 ;default is 0755 222 | force create mode = 0000 ;default is 0000 223 | force directory mode = 0000 ;default is 0000 224 | force user = user1 225 | force group samba 226 | ``` 227 | -------------------------------------------------------------------------------- /content/shared_storage_iscsi.md: -------------------------------------------------------------------------------- 1 | ## Shared storage on the network with iSCSI 2 | Many ways to share storage on a network exist. The iSCSI protocol defines a way to see a remote blocks device as a local disk. A remote device on the network is called iSCSI Target, a client which connects to iSCSI Target is called iSCSI Initiator. 3 | 4 | ### iSCSI Target Setup 5 | Install admin tools first, configure target to persistantly start at boot time and then start it 6 | ``` 7 | # yum -y install targetcli 8 | # systemctl enable target 9 | # systemctl start target 10 | ``` 11 | To start using ``targetcli``, run it and to get a layout of the tree interface, run ls 12 | ``` 13 | # targetcli 14 | targetcli shell version 2.1.fb37 15 | Copyright 2011-2013 by Datera, Inc and others. 16 | For help on commands, type 'help'. 17 | 18 | /> ls 19 | o- / .............................................................................................................. [...] 20 | o- backstores ................................................................................................... [...] 21 | | o- block ....................................................................................... [Storage Objects: 0] 22 | | o- fileio ...................................................................................... [Storage Objects: 0] 23 | | o- pscsi ....................................................................................... [Storage Objects: 0] 24 | | o- ramdisk ..................................................................................... [Storage Objects: 0] 25 | o- iscsi ................................................................................................. [Targets: 0] 26 | o- loopback .............................................................................................. [Targets: 0] 27 | /> 28 | ``` 29 | #### Create a Backstore 30 | Backstores enable support for different methods of storing an object on the local machine. Creating a storage object defines the resources the backstore will use. The supported backstores are: block devices, files, pscsi and ramdisks. Block devices are in our case. 31 | ``` 32 | /> /backstores/block create name=block_storage dev=/dev/sdb1 33 | Generating a wwn serial. 34 | Created block storage object block_backend using /dev/sdb1. 35 | ``` 36 | #### Create an iSCSI Target 37 | Create an iSCSI target using a specified name 38 | ``` 39 | /> iscsi/ create iqn.2015-05.com.noverit.caldara02:3260 40 | Created target iqn.2015-05.com.noverit.caldara02:3260. 41 | Created TPG 1. 42 | ``` 43 | #### Configure an iSCSI Portal 44 | An iSCSI Portal is an object specifying the IP address and port where the iSCSI target listen to incoming connections 45 | ``` 46 | /> /iscsi/iqn.2015-05.com.noverit.caldara02:3260/tpg1/portals/ create 47 | Using default IP port 3260 48 | Binding to INADDR_ANY (0.0.0.0) 49 | Created network portal 0.0.0.0:3260 50 | ``` 51 | By default, a portal is created when the iSCSI Target is created listening on all IP addresses (0.0.0.0) and the default iSCSI port 3260. Make sure that the 3260 is not used by another application, else specify a different port. 52 | 53 | #### Configure Access List 54 | Create an Access List for each initiator that will be connecting to the target. This enforces authentication when that initiator connects, allowing only LUNs to be exposed to each initiator. Usually each initator has exclusive access to a LUN. All initiators have unique identifying names IQN. The initiator's unique name IQN must be known to configure ACLs. For open-iscsi initiators, this can be found in the ``/etc/iscsi/initiatorname.iscsi`` file. 55 | ``` 56 | # cat /etc/iscsi/initiatorname.iscsi 57 | InitiatorName=iqn.1994-05.com.redhat:2268c31791 58 | ``` 59 | If required, use this IQN to enforce authentication by creating the ACLs. 60 | 61 | #### Configure the LUNs 62 | A Logical Unit Number (LUN) is a number used to identify a logical unit, which is a device addressed by the standard SCSI protocol or Storage Area Network protocols which encapsulate SCSI, such as Fibre Channel or iSCSI itself. 63 | To configure LUNs, create LUNs of already created storage objects. 64 | ``` 65 | /> /iscsi/iqn.2015-05.com.noverit.caldara02:3260/tpg1/luns/ create /backstores/block/block_storage 66 | Created LUN 0. 67 | ``` 68 | At the end of configuration, the iSCSI target envinronment should look like the following 69 | ``` 70 | /> ls 71 | o- / ........................................................................................................... [...] 72 | o- backstores ................................................................................................ [...] 73 | | o- block .................................................................................... [Storage Objects: 2] 74 | | | o- ana-storage ...................................................... [/dev/sdb1 (20.0GiB) write-thru activated] 75 | | | o- oracle-storage .................................................. [/dev/sdb2 (120.0GiB) write-thru activated] 76 | | o- fileio ................................................................................... [Storage Objects: 0] 77 | | o- pscsi .................................................................................... [Storage Objects: 0] 78 | | o- ramdisk .................................................................................. [Storage Objects: 0] 79 | o- iscsi .............................................................................................. [Targets: 1] 80 | | o- iqn.2015-05.com.noverit.caldara02:3260 .............................................................. [TPGs: 1] 81 | | o- tpg1 .................................................................................... [gen-acls, no-auth] 82 | | o- acls ............................................................................................ [ACLs: 0] 83 | | o- luns ............................................................................................ [LUNs: 2] 84 | | | o- lun0 .................................................................... [block/ana-storage (/dev/sdb1)] 85 | | | o- lun1 ................................................................. [block/oracle-storage (/dev/sdb2)] 86 | | o- portals ...................................................................................... [Portals: 1] 87 | | o- 10.10.10.98:3260 ................................................................................... [OK] 88 | o- loopback ........................................................................................... [Targets: 0] 89 | /> 90 | /> exit 91 | Global pref auto_save_on_exit=true 92 | Last 10 configs saved in /etc/target/backup. 93 | Configuration saved to /etc/target/saveconfig.json 94 | ``` 95 | The ``/etc/target/saveconfig.json`` file contains the above configuration. 96 | 97 | Restart the target service 98 | ``` 99 | # service target restart 100 | Redirecting to /bin/systemctl restart target.service 101 | ``` 102 | ### iSCSI Initiator Setup 103 | After configuring the iSCSI on the target machine, move to setup the iSCSI initiator machine. 104 | Install admin tools first 105 | 106 | ``` 107 | # yum -y install iscsi-initiator-utils 108 | ``` 109 | The iSCSI initiator is composed by two services, iscsi and iscsid, start both and enable to start at system startup 110 | ``` 111 | # service iscsid start 112 | # service iscsi start 113 | # service iscsid status 114 | # service iscsi status 115 | # chkconfig iscsi on 116 | # chkconfig iscsid on 117 | # chkconfig --list | grep iscsi 118 | iscsi 0:off 1:off 2:off 3:on 4:on 5:on 6:off 119 | iscsid 0:off 1:off 2:on 3:on 4:on 5:on 6:off 120 | ``` 121 | 122 | To connect the target, first discover the published iSCSI resouces and then login 123 | ``` 124 | # iscsiadm --mode discovery --type sendtargets --portal caldara02:3260 --discover 125 | 10.10.10.98:3260,1 iqn.2015-05.com.noverit.caldara02:3260 126 | # iscsiadm --mode node --targetname iqn.2015-05.com.noverit.caldara02:3260 --portal caldara02:3260 --login 127 | Logging in to [iface: default, target: iqn.2015-05.com.noverit.caldara02:3260, portal: 10.10.10.98,3260] (multiple) 128 | Login to [iface: default, target: iqn.2015-05.com.noverit.caldara02:3260, portal: 10.10.10.98,3260] successful. 129 | # 130 | ``` 131 | Since no authentication has been set, no user and password are required. 132 | Check the storage block devices. 133 | ``` 134 | [root@caldara01 ~]# lsblk 135 | NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT 136 | sda 8:0 0 232.9G 0 disk 137 | ├─sda1 8:1 0 500M 0 part /boot 138 | ├─sda2 8:2 0 73.4G 0 part 139 | │ ├─os-swap 253:0 0 3.9G 0 lvm [SWAP] 140 | │ ├─os-root 253:1 0 50G 0 lvm / 141 | │ └─os-data 253:2 0 178.5G 0 lvm /data 142 | └─sda3 8:3 0 159G 0 part 143 | └─os-data 253:2 0 178.5G 0 lvm /data 144 | sdc 8:32 0 20G 0 disk 145 | sdd 8:48 0 120G 0 disk 146 | ``` 147 | The two disks ``/dev/sdc`` and ``/dev/sdd`` are the remote iSCSI block devices exported by the target. They are seen as local block devices in the initiator machine. The disks can be used as standard local disks commands and configurations, including ``fdisk``, ``mkfs``, ``e2label``, etc. 148 | 149 | ``` 150 | # e2label /dev/sdc ANA 151 | # e2label /dev/sdd ORACLE 152 | # mkdir /ana 153 | # mkdir /oracle 154 | # mount -L ANA /ana 155 | # mount -L ORACLE /oracle 156 | # df -h 157 | Filesystem Size Used Avail Use% Mounted on 158 | /dev/mapper/os-root 50G 2.8G 48G 6% / 159 | devtmpfs 3.8G 0 3.8G 0% /dev 160 | tmpfs 3.8G 0 3.8G 0% /dev/shm 161 | tmpfs 3.8G 370M 3.4G 10% /run 162 | tmpfs 3.8G 0 3.8G 0% /sys/fs/cgroup 163 | /dev/mapper/os-data 179G 22G 158G 12% /data 164 | /dev/sda1 497M 228M 270M 46% /boot 165 | /dev/sdc 20G 45M 19G 1% /ana 166 | /dev/sdd 118G 60M 112G 1% /oracle 167 | ``` 168 | To disconnect the remote devices, umount and logout 169 | ``` 170 | # umount /ana 171 | # umount /oracle 172 | # 173 | # iscsiadm --mode node --targetname iqn.2015-05.com.noverit.caldara02:3260 --portal 10.10.10.98 --logout 174 | Logging out of session [sid: 10, target: iqn.2015-05.com.noverit.caldara02:3260, portal: 10.10.10.98,3260] 175 | Logout of [sid: 10, target: iqn.2015-05.com.noverit.caldara02:3260, portal: 10.10.10.98,3260] successful. 176 | # 177 | ``` 178 | 179 | Stop and then disable the services at startup, if required 180 | ``` 181 | # service iscsid status 182 | iscsid (pid 1184) is running... 183 | # service iscsi status 184 | No active sessions 185 | # service iscsid stop 186 | Stopping iscsid: [ OK ] 187 | # service iscsi stop 188 | Stopping iscsi: [ OK ] 189 | # chkconfig iscsid off 190 | # chkconfig iscsi off 191 | # chkconfig --list | grep iscsi 192 | iscsi 0:off 1:off 2:off 3:off 4:off 5:off 6:off 193 | iscsid 0:off 1:off 2:off 3:off 4:off 5:off 6:off 194 | ``` 195 | 196 | 197 | -------------------------------------------------------------------------------- /content/swap_memory.md: -------------------------------------------------------------------------------- 1 | ### Linux swap memory 2 | Linux divide la RAM in aree di memoria chiamate pagine. Lo **swapping** è il processo mediante il quale una pagina di memoria viene copiata in uno spazio preconfigurato sul disco rigido, chiamato spazio di **swap**, per liberare dalla memoria. Le dimensioni combinate della memoria fisica e dello spazio di swap è la quantità di memoria virtuale disponibile. Lo swapping è necessario per due motivi importanti: 3 | 4 | 1. In primo luogo, quando il sistema richiede più memoria di quella fisicamente disponibile, il kernel sposta le pagine meno utilizzate nello spazio di swap e concede l’utilizzo della memoria ram all’applicazione corrente (processo) che in quel momento richiede la memoria. 5 | 2. In secondo luogo, un numero significativo di pagine utilizzate da un’applicazione durante la sua fase di avvio possono essere utilizzate solo per l’inizializzazione del sistema e poi mai più usate. 6 | 7 | Il sistema è in grado di usare quindi lo swap su quelle pagine e di liberare la memoria per altre applicazioni o addirittura per la cache su disco. Tuttavia, lo swapping ha un rovescio della medaglia. Rispetto alla memoria RAM, i dischi sono molto più lenti. La velocità della memoria è misurata in nanosecondi, mentre quella dei dischi in millisecondi, dunque l’accesso al disco è decine di migliaia di volte più lento rispetto alla memoria ram. Più operazioni di swapping che si verificano, più lento il vostro sistema sarà. A volte un eccessivo swapping crea dei colli di bottiglia, poichè si verifica una particolare situazione: una pagina viene messa nello swap e poi portata in ram molto velocemente ed in modo continuativo. In tali situazioni il sistema lotta per trovare della memoria libera e mantenere le diverse applicazioni in esecuzione nello stesso momento. In questo caso, solo l’aggiunta di RAM più aiutare la stabilità del sistema stesso. 8 | 9 | Linux ha due forme di spazio di swap: la partizione di swap e il file di swap. La partizione di swap è una sezione indipendente del disco fisso, utilizzati esclusivamente per lo swap, nessun altro può risiedere lì. Il file di swap è un file speciale che risiede nel filesystem tra il sistema e file di dati. Per vedere com’è fatto e dove è ubicato lo spazio di swap che si possiede, si utilizza il comando ``swapon``. 10 | 11 | ``` 12 | # swapon -s 13 | Filename Type Size Used Priority 14 | /dev/dm-0 partition 4079612 0 -1 15 | ``` 16 | 17 | Ogni riga elenca una partizione di swap separata utilizzata dal sistema. Una particolarità dello swap su linux è che, se montare due (o più) spazi di swap (preferibilmente su due dispositivi differenti) con la stessa priorità, linux divide le sue attività di swapping tra di loro. Questo si traduce in un incremento notevole delle prestazioni. Per aggiungere una partizione di swap per il vostro sistema, è necessario però prima di prepararla. 18 | 19 | ### Add a swap area as a file 20 | ``` 21 | dd if=/dev/zero of=/var/swapfile bs=1M count=2048 22 | chmod 600 /var/swapfile 23 | mkswap /var/swapfile 24 | echo /var/swapfile none swap defaults 0 0 | sudo tee -a /etc/fstab 25 | swapon -a 26 | ``` 27 | 28 | -------------------------------------------------------------------------------- /content/system_info.md: -------------------------------------------------------------------------------- 1 | ### Linux release and system info 2 | Linux System Administrators need to get info from the system. Here some useful commands. 3 | 4 | Linux release and distribution 5 | ``` 6 | # cat /etc/*release 7 | CentOS Linux release 7.0.1406 (Core) 8 | NAME="CentOS Linux" 9 | VERSION="7 (Core)" 10 | ID="centos" 11 | ID_LIKE="rhel fedora" 12 | VERSION_ID="7" 13 | PRETTY_NAME="CentOS Linux 7 (Core)" 14 | ANSI_COLOR="0;31" 15 | CPE_NAME="cpe:/o:centos:centos:7" 16 | HOME_URL="https://www.centos.org/" 17 | BUG_REPORT_URL="https://bugs.centos.org/" 18 | CentOS Linux release 7.0.1406 (Core) 19 | ``` 20 | Kernel version 21 | ``` 22 | # uname -r 23 | 3.10.0-123.13.2.el7.x86_64 24 | ``` 25 | Memory Info 26 | ``` 27 | # head /proc/meminfo 28 | MemTotal: 3776748 kB 29 | MemFree: 2230496 kB 30 | MemAvailable: 2782088 kB 31 | Buffers: 1452 kB 32 | Cached: 652196 kB 33 | SwapCached: 0 kB 34 | Active: 1069616 kB 35 | Inactive: 193056 kB 36 | Active(anon): 609504 kB 37 | Inactive(anon): 8304 kB 38 | ``` 39 | File system 40 | ``` 41 | # df -h 42 | Filesystem Dimens. Usati Disp. Uso% Montato su 43 | /dev/sda1 12G 6,2G 4,9G 56% / 44 | /dev/small-db02 5,9G 2,6G 3,0G 46% /db02 45 | /dev/small-db01 5,0G 3,6G 1,2G 77% /db01 46 | /dev/small-db05 7,8G 1,2G 6,2G 17% /db05 47 | /dev/small-db03 39G 5,4G 32G 15% /db03 48 | /dev/small-db04 30G 2,5G 26G 9% /db04 49 | ``` 50 | 51 | Count the number of CPU 52 | ``` 53 | # cat /proc/cpuinfo | grep model | uniq -c 54 | 2 model name : Intel(R) Core(TM)2 Duo CPU E8500 @ 3.16GHz 55 | ``` 56 | 57 | ### The proc Filesystem 58 | The ``/proc`` filesystem contains virtual files that exist only in memory. This filesystem contains files and directories that mimic kernel structures and configuration information. It doesn't contain real files but runtime system information (e.g. system memory, devices mounted, hardware configuration, etc). Some important files in ``/proc`` are: 59 | 60 | ``` 61 | /proc/cpuinfo 62 | /proc/interrupts 63 | /proc/meminfo 64 | /proc/mounts 65 | /proc/partitions 66 | /proc/version 67 | /proc/ 68 | /proc/sys 69 | ``` 70 | The ``/proc`` filesystem is very useful because the information it reports is gathered only as needed and never needs storage on disk. 71 | 72 | ### Hostname 73 | The hostname identifies the machine within the domain. 74 | ``` 75 | # cat /etc/hostname 76 | ``` 77 | Set a new host name 78 | ``` 79 | # hostname NEW_NAME 80 | ``` 81 | This will set the hostname of the system to NEW_NAME. This is active right away and will remain like that until the system will be rebooted. On **Debian** based systems, use the file ``/etc/hostname`` to read the hostname of the system at boot time and set it up using the init script ``/etc/init.d/hostname.sh``. The hostname saved in the file ``/etc/hostname`` will be preserved on system reboot and will be set using the same script we used. 82 | 83 | On **RedHat** based systems, use the ``hostnamectl`` utility to get and set the hostname. 84 | 85 | ``` 86 | # hostnamectl status 87 | Static hostname: caldera01 88 | Icon name: computer-desktop 89 | Chassis: desktop 90 | Machine ID: 91 | Boot ID: 92 | Operating System: CentOS Linux 7 (Core) 93 | CPE OS Name: cpe:/o:centos:centos:7 94 | Kernel: Linux 3.10.0-123.13.2.el7.x86_64 95 | Architecture: x86_64 96 | ``` 97 | -------------------------------------------------------------------------------- /content/systemd.md: -------------------------------------------------------------------------------- 1 | ## System Services 2 | Systemd is the new init system for modern Linux distributions replacing the old init based on ``/etc/init.d/script``. It provides many powerful features for starting, stopping and managing processes. Here is an example to create a MineCraft service for systemd. MainCraft is a Java based game from Mojang. 3 | 4 | First, install the game and its envinronment. 5 | ``` 6 | # yum install java-1.8.0-openjdk.x86_64 7 | # which java 8 | /bin/java 9 | # mkdir /root/Minecraft 10 | # cd /root/Minecraft 11 | # wget -O minecraft_server.jar https://s3.amazonaws.com/Minecraft.Download/versions/1.8.6/minecraft_server.1.8.6.jar 12 | # ls -lrt 13 | -rw-r--r--. 1 root root 9780573 May 25 11:47 minecraft_server.jar 14 | -rw-r--r--. 1 root root 2 Jun 1 11:48 whitelist.json 15 | -rw-r--r--. 1 root root 180 Jun 1 12:01 eula.txt 16 | drwxr-xr-x. 2 root root 4096 Jun 1 16:09 logs 17 | -rw-r--r--. 1 root root 785 Jun 1 16:09 server.properties 18 | -rw-r--r--. 1 root root 2 Jun 1 16:09 banned-players.json 19 | -rw-r--r--. 1 root root 2 Jun 1 16:09 banned-ips.json 20 | -rw-r--r--. 1 root root 2 Jun 1 16:09 ops.json 21 | -rw-r--r--. 1 root root 109 Jun 1 16:10 usercache.json 22 | drwxr-xr-x. 8 root root 4096 Jun 1 16:37 world 23 | ``` 24 | 25 | The MineCraft server can be started at command line, by issuing the following command 26 | ``` 27 | # java -Xmx1024M -Xms1024M -jar minecraft_server.jar nogui 28 | ``` 29 | 30 | Alternately, a systemd configuration file can be created to start, stop, and check the status of the server as a standard system service by using the ``systemctl`` utility 31 | ``` 32 | # vi /lib/systemd/system/minecraftd.service 33 | [Unit] 34 | Description=Minecraft Server 35 | After=syslog.target network.target 36 | 37 | [Service] 38 | Type=simple 39 | WorkingDirectory=/root/Minecraft 40 | ExecStart=/bin/java -Xmx1024M -Xms1024M -jar minecraft_server.jar nogui 41 | SuccessExitStatus=143 42 | Restart=on-failure 43 | 44 | [Install] 45 | WantedBy=multi-user.target 46 | 47 | # systemctl start minecraftd 48 | # systemctl status minecraftd 49 | minecraftd.service - Minecraft Server 50 | Loaded: loaded (/usr/lib/systemd/system/minecraftd.service; disabled) 51 | Active: active (running) since Mon 2015-06-01 16:00:12 UTC; 18s ago 52 | Main PID: 20975 (java) 53 | CGroup: /system.slice/minecraftd.service 54 | └─20975 /bin/java -Xmx1024M -Xms1024M -jar minecraft_server.jar nogui 55 | 56 | # systemctl stop minecraftd 57 | ``` 58 | Note: the ``SuccessExitStatus=143`` is required when a process does not handle the exit signal properly. This is almost always due to programming errors, and is pretty common with Java applications of all types. To avoid a failed status of the MainCraft when stopping the service, the exit code 143 needs to be added into the unit file as a "success" exit status. 59 | 60 | The ``systemctl`` utility can be used to enable/disable the service at startup 61 | ``` 62 | # systemctl enable minecraftd 63 | ln -s '/usr/lib/systemd/system/minecraftd.service' '/etc/systemd/system/multi-user.target.wants/minecraftd.service' 64 | # systemctl is-enabled minecraftd 65 | enabled 66 | # systemctl disable minecraftd 67 | ``` 68 | 69 | Here another example 70 | ``` 71 | # cat /etc/systemd/system/redmined.service 72 | [Unit] 73 | Description=Redmine Server 74 | After=syslog.target network.target 75 | 76 | [Service] 77 | Type=simple 78 | PermissionsStartOnly=true 79 | WorkingDirectory=/home/redmine/redmine 80 | ExecStartPre=/usr/sbin/iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 8080 81 | ExecStart=/usr/bin/ruby bin/rails server -b 0.0.0.0 -p 8080 webrick -e production 82 | User=redmine 83 | Group=redmine 84 | StandardOutput=syslog 85 | StandardError=syslog 86 | SyslogIdentifier=redmined 87 | Restart=always 88 | RestartSec=10 89 | 90 | [Install] 91 | WantedBy=multi-user.target 92 | 93 | ``` 94 | -------------------------------------------------------------------------------- /content/text_commands.md: -------------------------------------------------------------------------------- 1 | ## Text commands 2 | Linux provides utilities for file and text manipulation: 3 | 4 | 1. Display contents using ``cat`` and ``echo``. 5 | 2. Edit file contents using ``sed`` and ``awk``. 6 | 3. Search for patterns using ``grep``. 7 | 8 | ### Display contents 9 | The ``cat`` is short for concatenate and is often used to read and print files as well as for simply viewing file contents, while the ``tac`` command prints the lines of a file in reverse order. 10 | ``` 11 | $ cat > myfile.txt 12 | Mario Rossi 13 | Antonio Esposito 14 | Michele Laforca 15 | Ctrl-D 16 | $ cat myfile.txt 17 | Mario Rossi 18 | Antonio Esposito 19 | Michele Laforca 20 | $ 21 | $ tac myfile.txt 22 | Michele Laforca 23 | Antonio Esposito 24 | Mario Rossi 25 | ``` 26 | The ``echo`` simply displays text. 27 | ``` 28 | $ echo myfile.txt 29 | myfile.txt 30 | ]$echo HOME 31 | HOME 32 | $ echo $HOME 33 | /home/ec2-user 34 | ``` 35 | ### Edit file content 36 | The command ``sed`` is a powerful text processing tool. Its name is an abbreviation for stream editor. It filters text as well as perform substitutions in data streams. Data from an input source/file (or stream) is taken and moved to a working space. The entire list of operations/modifications is applied over the data in the working space and the final contents are moved to the standard output space (or stream). 37 | ``` 38 | $ sed s/Mario/Saverio/ myfile.txt 39 | Saverio Rossi 40 | Antonio Esposito 41 | Michele Laforca 42 | $ cat myfile.txt 43 | Mario Rossi 44 | Antonio Esposito 45 | Michele Laforca 46 | $ sed s/Mario/Saverio/ myfile.txt > myfile2.txt 47 | $ cat myfile2.txt 48 | Saverio Rossi 49 | Antonio Esposito 50 | Michele Laforca 51 | $ sed -i s/Mario/Saverio/ myfile.txt 52 | $ cat myfile.txt 53 | Saverio Rossi 54 | Antonio Esposito 55 | Michele Laforca 56 | ``` 57 | For example, to convert 01/02/… to JAN/FEB/… 58 | ``` 59 | sed -e 's/01/JAN/' -e 's/02/FEB/' -e 's/03/MAR/' -e 's/04/APR/' -e 's/05/MAY/' \ 60 | -e 's/06/JUN/' -e 's/07/JUL/' -e 's/08/AUG/' -e 's/09/SEP/' -e 's/10/OCT/' \ 61 | -e 's/11/NOV/' -e 's/12/DEC/' 62 | ``` 63 | The ``awk`` command is used to extract and then print specific contents of a file and is often used to construct reports. It is a powerful utility and interpreted programming language, used to manipulate data files, retrieving, and processing text. 64 | It works well with fields (containing a single piece of data, essentially a column) and records (a collection of fields, essentially a line in a file). 65 | 66 | ``` 67 | $ awk '{ print $0 }' myfile.txt 68 | Saverio Rossi 69 | Antonio Esposito 70 | Michele Laforca 71 | $ awk '{ print $1 }' myfile.txt 72 | Saverio 73 | Antonio 74 | Michele 75 | $ awk '{ print $2 }' myfile.txt 76 | Rossi 77 | Esposito 78 | Laforca 79 | ``` 80 | Please, check the man pages for the ``awk`` and ``sed`` commands for futher details. 81 | 82 | ### File manipulation 83 | The ``sort`` command is used to rearrange the lines of a text file either in ascending or descending order, according to a sort key. 84 | ``` 85 | # cat myfile.txt 86 | Mario Rossi 87 | Antonio Esposito 88 | Michele Laforca 89 | # sort myfile.txt 90 | Antonio Esposito 91 | Mario Rossi 92 | Michele Laforca 93 | # sort -r myfile.txt 94 | Michele Laforca 95 | Mario Rossi 96 | Antonio Esposito 97 | ``` 98 | The ``uniq`` is used to remove duplicate lines in a text file and is useful for simplifying text display. It requires that the duplicate entries to be removed are consecutive. 99 | 100 | ``` 101 | # cat myfile.txt 102 | Mario Rossi 103 | Antonio Esposito 104 | Michele Laforca 105 | Antonio Esposito 106 | # sort myfile.txt | uniq 107 | Antonio Esposito 108 | Mario Rossi 109 | Michele Laforca 110 | # sort myfile.txt | uniq -c 111 | 2 Antonio Esposito 112 | 1 Mario Rossi 113 | 1 Michele Laforca 114 | ``` 115 | 116 | The ``paste`` command is used to combine fields from different files 117 | 118 | ``` 119 | # cat names.txt 120 | Mario Rossi 121 | Antonio Esposito 122 | Michele Laforca 123 | Antonio Esposito 124 | [root@caldera01 ~]# cat ages.txt 125 | 34 126 | 46 127 | 29 128 | 46 129 | [root@caldera01 ~]# paste names.txt ages.txt 130 | Mario Rossi 34 131 | Antonio Esposito 46 132 | Michele Laforca 29 133 | Antonio Esposito 46 134 | ``` 135 | 136 | The ``join`` command combines two files on a common field 137 | 138 | ``` 139 | # cat names.txt 140 | 01 Mario Rossi 141 | 02 Antonio Esposito 142 | 03 Michele Laforca 143 | 04 Antonio Esposito 144 | # cat ages.txt 145 | 01 34 146 | 02 46 147 | 03 29 148 | 04 46 149 | # join names.txt ages.txt 150 | 01 Mario Rossi 34 151 | 02 Antonio Esposito 46 152 | 03 Michele Laforca 29 153 | 04 Antonio Esposito 46 154 | ``` 155 | 156 | The ``grep`` comand is extensively used as a primary text searching tool. It scans files for specified patterns and can be used with regular expressions. 157 | ``` 158 | # grep Ant* names.txt 159 | 02 Antonio Esposito 160 | 04 Antonio Esposito 161 | ``` 162 | The ``tr`` utility is used to **tr**anslate specified characters into other characters or to delete them. 163 | ``` 164 | # cat names.txt 165 | 01 Mario Rossi 166 | 02 Antonio Esposito 167 | 03 Michele Laforca 168 | 04 Antonio Esposito 169 | # cat names.txt | tr a-z A-Z 170 | 01 MARIO ROSSI 171 | 02 ANTONIO ESPOSITO 172 | 03 MICHELE LAFORCA 173 | 04 ANTONIO ESPOSITO 174 | ``` 175 | The ``tee`` command takes the output from any command, and while sending it to standard output, it also saves it to a file. 176 | ``` 177 | # ls -l | tee list.txt 178 | total 32 179 | -rw-r--r--. 1 root root 24 Mar 3 14:42 ages.txt 180 | -rw-------. 1 root root 1883 Jan 21 20:53 anaconda-ks.cfg 181 | -rw-r--r--. 1 root root 74 Mar 3 14:42 names.txt 182 | -rwxr--r--. 1 root root 102 Feb 21 16:47 script.sh 183 | -rw-r--r--. 1 root root 74 Mar 3 14:52 tr 184 | [root@caldera01 ~]# cat list.txt 185 | total 32 186 | -rw-r--r--. 1 root root 24 Mar 3 14:42 ages.txt 187 | -rw-------. 1 root root 1883 Jan 21 20:53 anaconda-ks.cfg 188 | -rw-r--r--. 1 root root 74 Mar 3 14:42 names.txt 189 | -rwxr--r--. 1 root root 102 Feb 21 16:47 script.sh 190 | -rw-r--r--. 1 root root 74 Mar 3 14:52 tr 191 | ``` 192 | 193 | The ``wc`` (word count) counts the number of lines, words, and characters in a file or list of files. 194 | ``` 195 | # cat names.txt 196 | 01 Mario Rossi 197 | 02 Antonio Esposito 198 | 03 Michele Laforca 199 | 04 Antonio Esposito 200 | [root@caldera01 ~]# wc -l names.txt 201 | 4 names.txt 202 | [root@caldera01 ~]# wc -c names.txt 203 | 74 names.txt 204 | [root@caldera01 ~]# wc -w names.txt 205 | 12 names.txt 206 | ``` 207 | The ``cut`` command is used for manipulating column-based files and is designed to extract specific columns. The default column separator is the tab character. A different delimiter can be given as a command option. 208 | ``` 209 | # cut -d" " -f1 names.txt 210 | 01 211 | 02 212 | 03 213 | 04 214 | # cut -d" " -f2 names.txt 215 | Mario 216 | Antonio 217 | Michele 218 | Antonio 219 | # cut -d" " -f3 names.txt 220 | Rossi 221 | Esposito 222 | Laforca 223 | Esposito 224 | ``` 225 | 226 | The ``head`` reads the first few lines of each named file (10 by default) and displays it on standard output. 227 | ``` 228 | # head -n 2 names.txt 229 | 01 Mario Rossi 230 | 02 Antonio Esposito 231 | ``` 232 | The ``tail`` prints the last few lines of each named file and displays it on standard output. By default, it displays the last 10 lines. 233 | ``` 234 | # tail -n 2 names.txt 235 | 03 Michele Laforca 236 | 04 Antonio Esposito 237 | # 238 | # tail -f -n3 /var/log/messages 239 | Mar 3 14:38:59 caldera01 systemd: Started Session 35 of user root. 240 | Mar 3 15:01:01 caldera01 systemd: Starting Session 36 of user root. 241 | Mar 3 15:01:01 caldera01 systemd: Started Session 36 of user root. 242 | ``` 243 | -------------------------------------------------------------------------------- /content/user_env.md: -------------------------------------------------------------------------------- 1 | ### Users and Groups 2 | Linux is a multiuser operating system where more than one user can log on at the same time. The ``who`` command lists the currently logged-on users. To identify the current user, use the ``whoami`` command. 3 | 4 | ``` 5 | # who -a 6 | system boot 2015-02-17 13:28 7 | LOGIN tty1 2015-02-17 13:28 761 id=tty1 8 | root + pts/0 2015-02-17 13:29 . 12379 (10.10.10.246) 9 | run-level 3 2015-02-17 13:29 10 | root + pts/1 2015-02-17 17:37 . 18762 (10.10.10.246) 11 | ``` 12 | Linux uses groups for organizing users. Groups are collections of accounts with certain shared permissions. Control of group membership is administered through the ``/etc/group`` file, which shows a list of groups and their members. By default, every user belongs to a default or primary group. When a user logs in, the group membership is set for their primary group and all the members enjoy the same level of access and privilege. Permissions on various files and directories can be modified at the group level. 13 | 14 | All Linux users are assigned a unique user ID, the **uid**, which is just an integer, as well as one or more group ID’s, the **gid**, including a default one which is the same as the user ID. Historically, RedHat based distros start uid's at 500. Other distributions begin at 1000. These numbers are associated with names through the files ``/etc/passwd`` and ``/etc/group``. Groups are used to establish a set of users who have common interests for the purposes of access rights, privileges, and security considerations. Access rights to files and devices are granted on the basis of the user and the group they belong to. 15 | 16 | Only the root user can add and remove users and groups. Adding a new user is done with the ``useradd`` command and removing an existing user is done with the ``userdel`` command. In the simplest form an account for the new user adriano would be done with: 17 | ``` 18 | # useradd adriano 19 | # cat /etc/passwd | grep adriano 20 | adriano:x:1000:1000::/home/adriano:/bin/bash 21 | # ls -lrta /home/adriano/ 22 | total 16 23 | -rw-r--r--. 1 adriano adriano 231 Sep 26 03:53 .bashrc 24 | -rw-r--r--. 1 adriano adriano 193 Sep 26 03:53 .bash_profile 25 | -rw-r--r--. 1 adriano adriano 18 Sep 26 03:53 .bash_logout 26 | drwxr-xr-x. 3 root root 20 Feb 17 17:48 .. 27 | -rw-------. 1 adriano adriano 9 Feb 17 17:49 .bash_history 28 | drwx------. 2 adriano adriano 79 Feb 17 17:49 . 29 | ``` 30 | which by default sets the his home directory to ``/home/adriano``, populates it with some basic files and sets the default shell to ``/bin/bash``. 31 | 32 | Remove the user account by typing: 33 | ``` 34 | # userdel adriano 35 | # cat /etc/passwd | grep adriano 36 | # ls -lrta /home/adriano/ 37 | total 16 38 | -rw-r--r--. 1 1000 1000 231 Sep 26 03:53 .bashrc 39 | -rw-r--r--. 1 1000 1000 193 Sep 26 03:53 .bash_profile 40 | -rw-r--r--. 1 1000 1000 18 Sep 26 03:53 .bash_logout 41 | drwxr-xr-x. 3 root root 20 Feb 17 17:48 .. 42 | -rw-------. 1 1000 1000 9 Feb 17 17:49 .bash_history 43 | drwx------. 2 1000 1000 79 Feb 17 17:49 . 44 | ``` 45 | However, this will leave the home directory intact. This might be useful if it is a temporary inactivation. To remove the home directory while removing the account one needs to use the related option. 46 | ``` 47 | # userdel -r adriano 48 | # cat /etc/passwd | grep adriano 49 | # ls -lrta /home/adriano/ 50 | ls: cannot access /home/adriano/: No such file or directory 51 | ``` 52 | The command ``id`` with no argument gives information about the current user. If given the name of another user as an argument, id will report information about that other user. 53 | ``` 54 | # id 55 | uid=0(root) gid=0(root) groups=0(root) 56 | # id adriano 57 | uid=1000(adriano) gid=1000(adriano) groups=1000(adriano) 58 | ``` 59 | Use the ``passwd`` command to change the password for the new user 60 | ``` 61 | # passwd adriano 62 | Changing password for user adriano. 63 | New password: 64 | Retype new password: 65 | passwd: all authentication tokens updated successfully. 66 | ``` 67 | 68 | Adding a new group is done with the ``groupadd`` command and removed with the ``groupdel`` command. 69 | ``` 70 | # groupadd newgroup 71 | # groupdel newgroup 72 | ``` 73 | Adding a user to an already existing group is done with the ``usermod`` command. Removing a user from the group is a somewhat trickier. 74 | 75 | ``` 76 | # groupadd newgroup 77 | # usermod -G newgroup adriano 78 | # groups adriano 79 | adriano : adriano newgroup 80 | # usermod -g newgroup adriano 81 | # groups adriano 82 | adriano : newgroup 83 | # 84 | ``` 85 | All these commands update the ``/etc/group`` as necessary. The ``groupmod`` command can be used to change the group properties such as the Group ID or the name 86 | ``` 87 | # groupmod newgroup -n newgoupname 88 | # groups adriano 89 | adriano : newgoupname 90 | ``` 91 | 92 | ### The root user 93 | The **root** account is very powerful and has full access to the system. Other operating systems often call this the administrator account; in Linux it is often called the **superuser** account. You must be extremely cautious before granting full root access to a user; it is rarely if ever justified. External attacks often consist of tricks used to elevate to the root account. However, you can use the sudo feature to assign more limited privileges to standard user accounts: 94 | 95 | 1. on only a temporary basis. 96 | 2. only for a specific subset of commands. 97 | 98 | When assigning elevated privileges, you can use the command ``su`` (switch user) to launch a new shell running as another user (you must type the password of the user you are becoming). Most often this other user is root, and the new shell allows the use of elevated privileges until it is exited. It is almost always a bad (dangerous for both security and stability) practice to use ``su`` to become root. Resulting errors can include deletion of vital files from the system and security breaches. 99 | 100 | ### Startup Files 101 | In Linux, the command shell program, generally **bash** uses one or more startup files to configure the environment. Files in the ``/etc`` directory define global settings for all users while the initialization files in the user's home directory can include and or override the global settings. The startup files can do anything the user would like to do in every command shell, such as: 102 | 103 | * Customizing the user's prompt 104 | * Defining command-line shortcuts and aliases 105 | * Setting the default text editor 106 | * Setting the path for where to find executable programs 107 | 108 | When you first login to Linux, the ``/etc/profile`` file is read and evaluated, after which the following files are searched in the listed order: 109 | 110 | 1. ``~/.bash_profile`` 111 | 2. ``~/.bash_login`` 112 | 3. ``~/.profile`` 113 | 114 | The Linux login shell evaluates whatever startup file that it comes across first and ignores the rest. This means that if it finds ``~/.bash_profile``, it ignores the rest. Different distributions may use different startup files. However, every time you create a new shell, or terminal window, etc., you do not perform a full system login; only the ``~/.bashrc`` file is read and evaluated. Although this file is not read and evaluated along with the login shell, most distributions and/or users include the ``~/.bashrc`` file from within one of the three user-owned startup files. In the Ubuntu, openSuse, and CentOS distros, the user must make appropriate changes in the ``~/.bash_profile`` file to include the ``~/.bashrc`` file. The ``~/.bash_profile`` will have certain extra lines, which in turn will collect the required customization parameters from ``~/.bashrc``. 115 | 116 | ### Environment variables 117 | The environment variables are simply named quantities that have specific values and are understood by the command shell, such as **bash**. Some of these are pre-set by the system, and others are set by the user either at the command line or within startup and other scripts. An environment variable is actually no more than a character string that contains information used by one or more applications. There are a number of ways to view the values of currently set environment variables. All the ``set``, ``env``, or ``export`` commands display the environment variables. 118 | 119 | By default, variables created within a script are only available to the current shell. All the child processes (sub-shells) will not have access to values that have been set or modified. Allowing child processes to see the values, requires use of the ``export`` command. 120 | 121 | |Task|Command| 122 | |----|-------| 123 | |Show the value of a specific variable|echo $SHELL| 124 | |Export a new variable value|export VAR=value| 125 | |Add a variable permanently|Add the line export VAR=value to ~/.bashrc| 126 | 127 | The **HOME** is an environment variable that represents the home or login directory of the user. The ``cd`` command without arguments will change the current working directory to the value of HOME. Note the tilde character (~) is often used as an abbreviation for $HOME. 128 | 129 | The **PATH** environment variable is an ordered list of directories which is scanned when a command is given to find the appropriate program or script to run. Each directory in the path is separated by colons. An empty directory name indicates the current directory at any given time. 130 | 131 | ``` 132 | $ export PATH=$HOME/bin:$PATH 133 | $ echo $PATH 134 | /home/me/bin:/usr/local/bin:/usr/bin:/bin/usr 135 | ``` 136 | 137 | The **PS** environment variable (Prompt Statement) is used to customize your prompt string in your terminal windows to display the information you want. PS1 is the primary prompt variable which controls what your command line prompt looks like. The following special characters can be included in PS1 : 138 | 139 | |Character|Usage| 140 | |---------|-----| 141 | |\u|User name| 142 | |\h|Host name| 143 | |\w|Current working directory| 144 | |\!|History number of this command| 145 | |\d|Date| 146 | 147 | They must be surrounded in single quotes when they are used 148 | ``` 149 | # export PS1='\u@\h:\w$ ' 150 | root@caldera01:~$ 151 | root@caldera01:~$ export PS1='\d-\u@\h:\w$ ' 152 | Wed Feb 18-root@caldera01:~$ 153 | ``` 154 | The **SHELL** environment variable points to the user's default command shell (the program that is handling whatever you type in a command window, usually bash) and contains the full pathname to the shell 155 | ``` 156 | $ echo $SHELL 157 | /bin/bash 158 | $ 159 | ``` 160 | 161 | ### Command history 162 | The bash keeps track of previously entered commands and statements in a history buffer; you can recall previously used commands simply by using the Up and Down cursor keys. To view the list of previously executed commands, you can use the ``history`` at the command line. The list of commands is displayed with the most recent command appearing last in the list. This information is stored in ``~/.bash_history`` file. Several associated environment variables can be used to get information about the history file. 163 | 164 | |Variable|Usage| 165 | |--------|-----| 166 | |HISTFILE|stores the location of the history file| 167 | |HISTFILESIZE|stores the maximum number of lines in the history file| 168 | |HISTSIZE|stores the maximum number of lines in the history file for the current session| 169 | 170 | The table below shows the syntax used to execute previously used commands 171 | 172 | |Syntax|Usage| 173 | |------|-----| 174 | |!!|Execute the previous command| 175 | |!|Start a history substitution| 176 | |!$|Refer to the last argument in a line| 177 | |!n|Refer to the n-th command line| 178 | |!string|Refer to the most recent command starting with string| 179 | 180 | ### Creating Aliases 181 | Customized commands can be created to modify the behavior of already existing ones by creating aliases. Most often these aliases are placed in your ``~/.bashrc`` file so they are available to any command shells you create. The ``alias`` command with no arguments will list currently defined aliases. 182 | 183 | ``` 184 | $ alias 185 | alias cp='cp -i' 186 | alias egrep='egrep --color=auto' 187 | alias fgrep='fgrep --color=auto' 188 | alias grep='grep --color=auto' 189 | alias l.='ls -d .* --color=auto' 190 | alias ll='ls -l --color=auto' 191 | alias ls='ls --color=auto' 192 | alias mv='mv -i' 193 | alias rm='rm -i' 194 | ``` 195 | -------------------------------------------------------------------------------- /content/volume_manager.md: -------------------------------------------------------------------------------- 1 | ## Logical Volume Manager layout 2 | Basically a Logical Volume Manager layout **LVM** looks like this: 3 | 4 | * **Logical Volume(s)**: ``/dev/fileserver/share``, ``/dev/fileserver/backup``, ``/dev/fileserver/media`` 5 | * **Volume Group(s)**: ``fileserver`` 6 | * **Physical Volume(s)**: ``/dev/sdb1``, ``/dev/sdc1``, ``/dev/sdd1``, ``/dev/sdc1`` 7 | 8 | You have one or more physical volumes, and on these physical volumes you create one or more volume groups, and in each volume group you can create one or more logical volumes. If you use multiple physical volumes, each logical volume can be bigger than one of the underlying physical volumes (but of course the sum of the logical volumes cannot exceed the total space offered by the physical volumes). It is a good practice to not allocate the full space to logical volumes, but leave some space unused. That way you can enlarge one or more logical volumes later on if you feel the need for it. 9 | 10 | With LVM, an hard drive or set of hard drives or different partitions of the same hard drive are allocated to one or more physical volumes. The physical volumes can be placed on other block devices which might span two or more disks. The physical volumes are combined into logical volumes, with the exception of the ``/boot`` partition. The ``/boot`` partition cannot be on a logical volume group because the boot loader cannot read it. If the root partition is on a logical volume, create a separate ``/boot`` partition which is not a part of a volume group. Since a physical volume cannot span over multiple drives, to span over more than one drive, create one or more physical volumes per drive. 11 | 12 | The volume groups can be divided into logical volumes, which are assigned mount points, such as ``/home`` and root and file system types, such as **ext2** or **ext3**. When the partitions reach their full capacity, free space from the volume group can be added to the logical volume to increase the size of the partition. When a new hard drive is added to the system, it can be added to the volume group, and partitions that are logical volumes can be increased in size. 13 | 14 | ### Create a LVM layout 15 | On my local CentOS machine, there is on additional hard drive ``/dev/sdb`` to use for LVM layout. 16 | ``` 17 | # fdisk /dev/sdb 18 | Welcome to fdisk (util-linux 2.23.2). 19 | Disk /dev/sdb: 250.1 GB, 250059350016 bytes, 488397168 sectors 20 | Units = sectors of 1 * 512 = 512 bytes 21 | Sector size (logical/physical): 512 bytes / 512 bytes 22 | I/O size (minimum/optimal): 512 bytes / 512 bytes 23 | Disk label type: dos 24 | Disk identifier: 0x0004da93 25 | 26 | Device Boot Start End Blocks Id System 27 | /dev/sdb1 2048 488397167 244197560 8e Linux LVM 28 | 29 | Command (m for help): 30 | ``` 31 | 32 | The disk is already partitioned as Linux LVM, so no needs to do further. To create the LVM layout, first we need to create a physical volume 33 | ``` 34 | # pvcreate /dev/sdb1 35 | Physical volume "/dev/sdb1" successfully created 36 | # pvs 37 | PV VG Fmt Attr PSize PFree 38 | /dev/sdb1 lvm2 --- 232.88g 232.88g 39 | # pvscan 40 | PV 41 | /dev/sdb1 lvm2 [232.88 GiB] 42 | Total: 3 [465.28 GiB] / in use: 2 [232.39 GiB] / in no VG: 1 [232.88 GiB] 43 | # pvdisplay 44 | "/dev/sdb1" is a new physical volume of "232.88 GiB" 45 | --- NEW Physical volume --- 46 | PV Name /dev/sdb1 47 | VG Name 48 | PV Size 232.88 GiB 49 | Allocatable NO 50 | PE Size 0 51 | Total PE 0 52 | Free PE 0 53 | Allocated PE 0 54 | PV UUID ajGCMg-Y4cG-v4AD-Wxma-TaE5-zQig-XmnYAx 55 | ``` 56 | 57 | Now create the volume group 58 | ``` 59 | # vgcreate storage /dev/sdb1 60 | Volume group "storage" successfully created 61 | # vgscan 62 | Reading all physical volumes. This may take a while... 63 | Found volume group "storage" using metadata type lvm2 64 | # vgs 65 | VG #PV #LV #SN Attr VSize VFree 66 | storage 1 0 0 wz--n- 232.88g 232.88g 67 | # vgdisplay 68 | --- Volume group --- 69 | VG Name storage 70 | System ID 71 | Format lvm2 72 | Metadata Areas 1 73 | Metadata Sequence No 1 74 | VG Access read/write 75 | VG Status resizable 76 | MAX LV 0 77 | Cur LV 0 78 | Open LV 0 79 | Max PV 0 80 | Cur PV 1 81 | Act PV 1 82 | VG Size 232.88 GiB 83 | PE Size 4.00 MiB 84 | Total PE 59618 85 | Alloc PE / Size 0 / 0 86 | Free PE / Size 59618 / 232.88 GiB 87 | VG UUID nEcTxG-p5K6-npqD-OVeX-dRI1-aWP9-o4D1Z1 88 | 89 | ``` 90 | Now, everything is ready to create the logical volumes from the volume group 91 | ``` 92 | # lvcreate -L 20G -n db-area storage 93 | Logical volume "db-area" created. 94 | # lvcreate -L 10G -n users-area storage 95 | Logical volume "users-area" created. 96 | # lvcreate -L 60G -n staging-area storage 97 | Logical volume "staging-area" created. 98 | # lvcreate -l 100%FREE -n spare storage 99 | Logical volume "spare" created. 100 | # lvs 101 | LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert 102 | db-area storage -wi-a----- 20.00g 103 | spare storage -wi-a----- 142.88g 104 | staging-area storage -wi-a----- 60.00g 105 | users-area storage -wi-a----- 10.00g 106 | # lvscan 107 | ACTIVE '/dev/storage/db-area' [20.00 GiB] inherit 108 | ACTIVE '/dev/storage/users-area' [10.00 GiB] inherit 109 | ACTIVE '/dev/storage/staging-area' [60.00 GiB] inherit 110 | ACTIVE '/dev/storage/spare' [142.88 GiB] inherit 111 | ``` 112 | After creating the appropriate filesystem on the logical volumes, they become ready to use for the storage purpose 113 | ``` 114 | # mkfs.ext4 /dev/storage/db-area 115 | # mkfs.ext4 /dev/storage/users-area 116 | # mkfs.ext4 /dev/storage/staging-area 117 | # mkfs.ext4 /dev/storage/spare 118 | 119 | # mkdir /db 120 | # mount /dev/storage/db-area /db 121 | # mkdir /users 122 | # mount /dev/storage/users-area /users 123 | # mkdir /staging 124 | # mount /dev/storage/staging-area /staging 125 | 126 | ``` 127 | 128 | 129 | ### Extend a LVM layout 130 | On the local CentOS machine, there are 2 hard drive ``/dev/sda`` and ``/dev/sdb``. The ``/dev/sda`` is partioned as follow 131 | ``` 132 | # fdisk -l /dev/sda 133 | 134 | Disk /dev/sda: 250.1 GB, 250059350016 bytes, 488397168 sectors 135 | Units = sectors of 1 * 512 = 512 bytes 136 | Sector size (logical/physical): 512 bytes / 512 bytes 137 | I/O size (minimum/optimal): 512 bytes / 512 bytes 138 | Disk label type: dos 139 | Disk identifier: 0x000b78bc 140 | 141 | Device Boot Start End Blocks Id System 142 | /dev/sda1 * 2048 1026047 512000 83 Linux 143 | /dev/sda2 1026048 155004927 76989440 8e Linux LVM 144 | /dev/sda3 155004928 488397167 166696120 83 Linux LVM 145 | ``` 146 | 147 | The ``/dev/sda1`` is for the ``/boot`` partition and is not into LVM layout. Both ``/dev/sda2`` and ``/dev/sda3`` partitions are part of the LVM layout. Note that both the partitions are part of the same physical disk. This is not so common in production but is possible to have. More common is the case of partitions belonging to different physical disks. 148 | ``` 149 | # lvmdiskscan 150 | /dev/os/swap [ 3.89 GiB] 151 | /dev/sda1 [ 500.00 MiB] 152 | /dev/os/root [ 50.00 GiB] 153 | /dev/sda2 [ 73.42 GiB] LVM physical volume 154 | /dev/os/data [ 178.50 GiB] 155 | /dev/sda3 [ 158.97 GiB] LVM physical volume 156 | /dev/sdb1 [ 232.88 GiB] 157 | 3 disks 158 | 2 partitions 159 | 0 LVM physical volume whole disks 160 | 2 LVM physical volumes 161 | 162 | # pvs 163 | PV VG Fmt Attr PSize PFree 164 | /dev/sda2 os lvm2 a-- 73.42g 0 165 | /dev/sda3 os lvm2 a-- 158.97g 0 166 | 167 | # vgs 168 | VG #PV #LV #SN Attr VSize VFree 169 | os 2 3 0 wz--n- 232.39g 0 170 | 171 | # lvs 172 | LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert 173 | data os -wi-ao---- 178.50g 174 | root os -wi-ao---- 50.00g 175 | swap os -wi-ao---- 3.89g 176 | ``` 177 | The two partitons are seen as two LVM physical volumes: ``/dev/sda2`` and ``/dev/sda3``. The two phisical volumes are part of the same volume group called ``os``. On top of this volume group there are three logical volumes: ``/root``, ``/data`` and ``/swap``. 178 | 179 | We want to increase the space of the LVM layout with a new partition belonging to the second hard drive ``/dev/sdb``. The hard drive is partitioned as follow: 180 | ``` 181 | # fdisk -l /dev/sdb 182 | 183 | Disk /dev/sdb: 250.1 GB, 250059350016 bytes, 488397168 sectors 184 | Units = sectors of 1 * 512 = 512 bytes 185 | Sector size (logical/physical): 512 bytes / 512 bytes 186 | I/O size (minimum/optimal): 512 bytes / 512 bytes 187 | Disk label type: dos 188 | Disk identifier: 0x0004da93 189 | 190 | Device Boot Start End Blocks Id System 191 | /dev/sdb1 2048 488397167 244197560 83 Linux 192 | 193 | ``` 194 | 195 | The partition ``/dev/sdb1`` is Linux type. Change the partition type to Linux LVM 196 | ``` 197 | # fdisk /dev/sdb 198 | Welcome to fdisk (util-linux 2.23.2). 199 | 200 | Changes will remain in memory only, until you decide to write them. 201 | Be careful before using the write command. 202 | 203 | 204 | Command (m for help): p 205 | 206 | Disk /dev/sdb: 250.1 GB, 250059350016 bytes, 488397168 sectors 207 | Units = sectors of 1 * 512 = 512 bytes 208 | Sector size (logical/physical): 512 bytes / 512 bytes 209 | I/O size (minimum/optimal): 512 bytes / 512 bytes 210 | Disk label type: dos 211 | Disk identifier: 0x0004da93 212 | 213 | Device Boot Start End Blocks Id System 214 | /dev/sdb1 2048 488397167 244197560 83 Linux 215 | 216 | Command (m for help): t 217 | Selected partition 1 218 | Hex code (type L to list all codes): 8e 219 | Changed type of partition 'Linux' to 'Linux LVM' 220 | 221 | Command (m for help): p 222 | 223 | Disk /dev/sdb: 250.1 GB, 250059350016 bytes, 488397168 sectors 224 | Units = sectors of 1 * 512 = 512 bytes 225 | Sector size (logical/physical): 512 bytes / 512 bytes 226 | I/O size (minimum/optimal): 512 bytes / 512 bytes 227 | Disk label type: dos 228 | Disk identifier: 0x0004da93 229 | 230 | Device Boot Start End Blocks Id System 231 | /dev/sdb1 2048 488397167 244197560 8e Linux LVM 232 | 233 | Command (m for help): w 234 | The partition table has been altered! 235 | 236 | Calling ioctl() to re-read partition table. 237 | 238 | WARNING: Re-reading the partition table failed with error 16: Device or resource busy. 239 | The kernel still uses the old table. The new table will be used at 240 | the next reboot or after you run partprobe(8) or kpartx(8) 241 | Syncing disks. 242 | ``` 243 | A warning message which basically means in order to use the new table with the changes a system reboot is required. As workaround, run the ``partprobe -s`` to rescan the partitions. 244 | 245 | ``` 246 | # partprobe -s 247 | /dev/sda: msdos partitions 1 2 3 248 | /dev/sdb: msdos partitions 1 249 | ``` 250 | 251 | Create a new physical volume from the new partition 252 | ``` 253 | # pvcreate /dev/sdb1 254 | WARNING: xfs signature detected on /dev/sdb1 at offset 0. Wipe it? [y/n] y 255 | Wiping xfs signature on /dev/sdb1. 256 | Physical volume "/dev/sdb1" successfully created 257 | ``` 258 | 259 | Check the new physical volume just created by ``pvdisplay`` command 260 | ``` 261 | # pvdisplay 262 | "/dev/sdb1" is a new physical volume of "232.88 GiB" 263 | --- NEW Physical volume --- 264 | PV Name /dev/sdb1 265 | VG Name 266 | PV Size 232.88 GiB 267 | Allocatable NO 268 | PE Size 0 269 | Total PE 0 270 | Free PE 0 271 | Allocated PE 0 272 | PV UUID qtRwhD-Pxcv-JQlD-u7xu-lNi0-CiBv-F9XUoO 273 | ``` 274 | 275 | Now extend the ``os`` volume group by adding in the new physical volume which we created earlier 276 | ``` 277 | # vgextend os /dev/sdb1 278 | Volume group "os" successfully extended 279 | ``` 280 | 281 | Using the ``pvscan`` command we scan all disks for physical volumes, this should confirm the new created physical volume ``/dev/sdb1`` and along with the old volumes 282 | ``` 283 | # pvscan 284 | PV /dev/sda2 VG os lvm2 [73.42 GiB / 0 free] 285 | PV /dev/sda3 VG os lvm2 [158.97 GiB / 0 free] 286 | PV /dev/sdb1 VG os lvm2 [232.88 GiB / 232.88 GiB free] 287 | Total: 3 [465.28 GiB] / in use: 3 [465.28 GiB] / in no VG: 0 [0 ] 288 | ``` 289 | 290 | Next want to increase the logical volume ``/dev/os/data`` which basically means we will be taking our original logical volume ``/dev/os/data`` and extending it over the new physical volume ``/dev/sdb1`` just created. 291 | 292 | ``` 293 | # lvscan 294 | ACTIVE '/dev/os/root' [50.00 GiB] inherit 295 | ACTIVE '/dev/os/swap' [3.89 GiB] inherit 296 | ACTIVE '/dev/os/data' [178.50 GiB] inherit 297 | # lvextend /dev/os/data /dev/sdb1 298 | Extending logical volume data to 411.39 GiB 299 | Logical volume data successfully resized 300 | # lvscan 301 | ACTIVE '/dev/os/root' [50.00 GiB] inherit 302 | ACTIVE '/dev/os/swap' [3.89 GiB] inherit 303 | ACTIVE '/dev/os/data' [411.39 GiB] inherit 304 | ``` 305 | 306 | Note the size of the logical volume ``/dev/os/data`` increased from 178.50 GiB to 411.39 GiB. Howewer, the size of the ``/data`` file system is still 179G 307 | ``` 308 | # df -h 309 | Filesystem Size Used Avail Use% Mounted on 310 | /dev/mapper/os-root 50G 2.0G 48G 4% / 311 | devtmpfs 1.8G 0 1.8G 0% /dev 312 | tmpfs 1.9G 0 1.9G 0% /dev/shm 313 | tmpfs 1.9G 8.6M 1.8G 1% /run 314 | tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup 315 | /dev/mapper/os-data 179G 33M 179G 1% /data 316 | /dev/sda1 497M 183M 315M 37% /boot 317 | ``` 318 | 319 | There is then one final step which is to resize the file system so that it can take advantage of this additional space, this is done using the ``resize2fs`` command. In our case, the command will fail since we are using *xfs* file system. We need to use the ``xfs_growfs`` to have the same effect. 320 | ``` 321 | # xfs_growfs /dev/os/data 322 | meta-data=/dev/mapper/os-data isize=256 agcount=4, agsize=11698432 blks 323 | = sectsz=512 attr=2, projid32bit=1 324 | = crc=0 325 | data = bsize=4096 blocks=46793728, imaxpct=25 326 | = sunit=0 swidth=0 blks 327 | naming =version 2 bsize=4096 ascii-ci=0 ftype=0 328 | log =internal bsize=4096 blocks=22848, version=2 329 | = sectsz=512 sunit=0 blks, lazy-count=1 330 | realtime =none extsz=4096 blocks=0, rtextents=0 331 | data blocks changed from 46793728 to 107842560 332 | 333 | # df -h 334 | Filesystem Size Used Avail Use% Mounted on 335 | /dev/mapper/os-root 50G 2.0G 48G 4% / 336 | devtmpfs 1.8G 0 1.8G 0% /dev 337 | tmpfs 1.9G 0 1.9G 0% /dev/shm 338 | tmpfs 1.9G 8.6M 1.8G 1% /run 339 | tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup 340 | /dev/mapper/os-data 412G 33M 412G 1% /data 341 | /dev/sda1 497M 183M 315M 37% /boot 342 | ``` 343 | 344 | ### Reduce the LVM layout 345 | Unfortunally we can NOT make a XFS partition smaller online. The only way to shrink is to do a complete dump of data, make a new smaller volume group and restore the data. 346 | 347 | ``` 348 | # mkdir /dump 349 | # mv /data/* /dump/ 350 | # umount /data 351 | ``` 352 | 353 | Remove the logical volume ``/dev/os/data`` 354 | ``` 355 | # lvremove /dev/os/data 356 | Do you really want to remove active logical volume data? [y/n]: y 357 | Logical volume "data" successfully removed 358 | # lvscan 359 | ACTIVE '/dev/os/root' [50.00 GiB] inherit 360 | ACTIVE '/dev/os/swap' [3.89 GiB] inherit 361 | ``` 362 | Detouch the physical volume from the volume group. To accomplish this task, use the ``vgreduce`` command. This command shrinks a volume group's capacity by removing one or more physical volumes. This frees the physical volumes to be used in other volume groups or to be removed from the system. 363 | 364 | ``` 365 | # vgreduce os /dev/sdb1 366 | Removed "/dev/sdb1" from volume group "os" 367 | # pvs 368 | PV VG Fmt Attr PSize PFree 369 | /dev/sda2 os lvm2 a-- 73.42g 19.53g 370 | /dev/sda3 os lvm2 a-- 158.97g 158.97g 371 | /dev/sdb1 lvm2 a-- 232.88g 232.88g 372 | ``` 373 | 374 | Since we do not need anymore for the physical volume ``/dev/sdb1`` , remove it from the LVM layout 375 | ``` 376 | # pvremove /dev/sdb1 377 | Labels on physical volume "/dev/sdb1" successfully wiped 378 | # pvs 379 | PV VG Fmt Attr PSize PFree 380 | /dev/sda2 os lvm2 a-- 73.42g 19.53g 381 | /dev/sda3 os lvm2 a-- 158.97g 158.97g 382 | ``` 383 | 384 | Recreate the ``/dev/os/data`` logical volume with the remaining space in the ``os`` volume group 385 | ``` 386 | # lvcreate -l 100%FREE -n data os 387 | WARNING: xfs signature detected on /dev/os/data at offset 0. Wipe it? [y/n] y 388 | Wiping xfs signature on /dev/os/data. 389 | Logical volume "data" created 390 | # lvscan 391 | ACTIVE '/dev/os/root' [50.00 GiB] inherit 392 | ACTIVE '/dev/os/swap' [3.89 GiB] inherit 393 | ACTIVE '/dev/os/data' [178.50 GiB] inherit 394 | ``` 395 | 396 | Format the volume group just created, mount it as ``/data`` fyle system and restore the data 397 | ``` 398 | # mkfs.xfs -f /dev/os/data 399 | meta-data=/dev/os/data isize=256 agcount=4, agsize=11698432 blks 400 | = sectsz=512 attr=2, projid32bit=1 401 | = crc=0 402 | data = bsize=4096 blocks=46793728, imaxpct=25 403 | = sunit=0 swidth=0 blks 404 | naming =version 2 bsize=4096 ascii-ci=0 ftype=0 405 | log =internal log bsize=4096 blocks=22848, version=2 406 | = sectsz=512 sunit=0 blks, lazy-count=1 407 | realtime =none extsz=4096 blocks=0, rtextents=0 408 | 409 | # mount -a 410 | # 411 | # df -Th 412 | Filesystem Type Size Used Avail Use% Mounted on 413 | /dev/mapper/os-root xfs 50G 2.0G 48G 4% / 414 | devtmpfs devtmpfs 1.8G 0 1.8G 0% /dev 415 | tmpfs tmpfs 1.9G 0 1.9G 0% /dev/shm 416 | tmpfs tmpfs 1.9G 8.5M 1.8G 1% /run 417 | tmpfs tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup 418 | /dev/sda1 xfs 497M 183M 315M 37% /boot 419 | /dev/mapper/os-data xfs 179G 33M 179G 1% /data 420 | # 421 | # mv /dump/* /data/ 422 | ``` 423 | 424 | Finally, change the partition type of ``/dev/sdb1`` back to Linux (no LVM), format as XFS and mount it as a standard physical partition 425 | ``` 426 | # fdisk /dev/sdb 427 | Welcome to fdisk (util-linux 2.23.2). 428 | Changes will remain in memory only, until you decide to write them. 429 | Be careful before using the write command. 430 | 431 | Command (m for help): p 432 | 433 | Disk /dev/sdb: 250.1 GB, 250059350016 bytes, 488397168 sectors 434 | Units = sectors of 1 * 512 = 512 bytes 435 | Sector size (logical/physical): 512 bytes / 512 bytes 436 | I/O size (minimum/optimal): 512 bytes / 512 bytes 437 | Disk label type: dos 438 | Disk identifier: 0x0004da93 439 | 440 | Device Boot Start End Blocks Id System 441 | /dev/sdb1 2048 488397167 244197560 8e Linux LVM 442 | 443 | Command (m for help): t 444 | Selected partition 1 445 | Hex code (type L to list all codes): 83 446 | Changed type of partition 'Linux LVM' to 'Linux' 447 | 448 | Command (m for help): p 449 | 450 | Disk /dev/sdb: 250.1 GB, 250059350016 bytes, 488397168 sectors 451 | Units = sectors of 1 * 512 = 512 bytes 452 | Sector size (logical/physical): 512 bytes / 512 bytes 453 | I/O size (minimum/optimal): 512 bytes / 512 bytes 454 | Disk label type: dos 455 | Disk identifier: 0x0004da93 456 | 457 | Device Boot Start End Blocks Id System 458 | /dev/sdb1 2048 488397167 244197560 83 Linux 459 | 460 | Command (m for help): w 461 | The partition table has been altered! 462 | 463 | Calling ioctl() to re-read partition table. 464 | Syncing disks. 465 | # mkfs.xfs -f /dev/sdb1 466 | meta-data=/dev/sdb1 isize=256 agcount=4, agsize=15262348 blks 467 | = sectsz=512 attr=2, projid32bit=1 468 | = crc=0 469 | data = bsize=4096 blocks=61049390, imaxpct=25 470 | = sunit=0 swidth=0 blks 471 | naming =version 2 bsize=4096 ascii-ci=0 ftype=0 472 | log =internal log bsize=4096 blocks=29809, version=2 473 | = sectsz=512 sunit=0 blks, lazy-count=1 474 | realtime =none extsz=4096 blocks=0, rtextents=0 475 | 476 | # mount -a 477 | # df -Th 478 | Filesystem Type Size Used Avail Use% Mounted on 479 | /dev/mapper/os-root xfs 50G 2.0G 48G 4% / 480 | devtmpfs devtmpfs 1.8G 0 1.8G 0% /dev 481 | tmpfs tmpfs 1.9G 0 1.9G 0% /dev/shm 482 | tmpfs tmpfs 1.9G 8.5M 1.8G 1% /run 483 | tmpfs tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup 484 | /dev/sda1 xfs 497M 183M 315M 37% /boot 485 | /dev/mapper/os-data xfs 179G 33M 179G 1% /data 486 | /dev/sdb1 xfs 233G 33M 233G 1% /cinder-volumes 487 | # 488 | ``` 489 | -------------------------------------------------------------------------------- /content/volume_manager_cont.md: -------------------------------------------------------------------------------- 1 | ## Volume Manager, a real example 2 | Let to see, step-by-step, a real example of Volume Manager on a CentOS 7 setup. 3 | On the host caldera01 there is an additional disk ``/dev/sdb`` we want to use for store a mysql database. 4 | 5 | ``` 6 | [root@caldera01 ~]# df -Th 7 | Filesystem Type Size Used Avail Use% Mounted on 8 | /dev/mapper/os-root xfs 50G 2.1G 48G 5% / 9 | devtmpfs devtmpfs 3.8G 0 3.8G 0% /dev 10 | tmpfs tmpfs 3.8G 0 3.8G 0% /dev/shm 11 | tmpfs tmpfs 3.8G 8.6M 3.8G 1% /run 12 | tmpfs tmpfs 3.8G 0 3.8G 0% /sys/fs/cgroup 13 | /dev/mapper/os-data xfs 175G 256M 175G 1% /data 14 | /dev/sda1 xfs 497M 190M 308M 39% /boot 15 | 16 | [root@caldera01 ~]# lsblk 17 | NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT 18 | sda 8:0 0 232.9G 0 disk 19 | ├─sda1 8:1 0 500M 0 part /boot 20 | └─sda2 8:2 0 232.4G 0 part 21 | ├─os-swap 253:0 0 7.8G 0 lvm [SWAP] 22 | ├─os-root 253:1 0 50G 0 lvm / 23 | └─os-data 253:2 0 174.6G 0 lvm /data 24 | sdb 8:16 0 232.9G 0 disk 25 | └─sdb1 8:17 0 232.9G 0 part 26 | 27 | [root@caldera01 ~]# fdisk /dev/sdb 28 | Disk /dev/sdb: 250.1 GB, 250059350016 bytes, 488397168 sectors 29 | Units = sectors of 1 * 512 = 512 bytes 30 | Sector size (logical/physical): 512 bytes / 512 bytes 31 | I/O size (minimum/optimal): 512 bytes / 512 bytes 32 | Disk label type: dos 33 | Disk identifier: 0x0003b431 34 | 35 | Device Boot Start End Blocks Id System 36 | /dev/sdb1 2048 488397167 244197560 8e Linux LVM 37 | ``` 38 | 39 | Create a LVM layout 40 | ``` 41 | [root@caldera01 ~]# pvcreate /dev/sdb1 42 | [root@caldera01 ~]# vgcreate vgdb /dev/sdb1 43 | [root@caldera01 ~]# lvcreate -l 100%FREE -n lvol1 vgdb 44 | [root@caldera01 ~]# pvs 45 | PV VG Fmt Attr PSize PFree 46 | /dev/sda2 os lvm2 a-- 232.39g 0 47 | /dev/sdb1 vgdb lvm2 a-- 232.88g 0 48 | [root@caldera01 ~]# vgs 49 | VG #PV #LV #SN Attr VSize VFree 50 | os 1 3 0 wz--n- 232.39g 0 51 | vgdb 1 1 0 wz--n- 232.88g 0 52 | [root@caldera01 ~]# lvs 53 | LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert 54 | data os -wi-ao---- 174.63g 55 | root os -wi-ao---- 50.00g 56 | swap os -wi-ao---- 7.77g 57 | lvol1 vgdb -wi-ao---- 232.88g 58 | ``` 59 | 60 | Make an ``ext3`` file system on the logical volume and mount the partition under a ``/db`` directory 61 | ``` 62 | [root@caldera01 ~]# mkfs -t ext3 /dev/vgdb/lvol1 63 | [root@caldera01 ~]# mkdir /db 64 | [root@caldera01 ~]# mount /dev/sdb1 /db 65 | ``` 66 | 67 | Install a mysql database on the new filesystem 68 | ``` 69 | [root@caldera01 ~]# sudo rpm -Uvh http://dev.mysql.com/get/mysql-community-release-el7-5.noarch.rpm 70 | [root@caldera01 ~]# yum install -y mysql-server 71 | [root@caldera01 ~]# vi /etc/my.cnf 72 | [mysqld] 73 | ... 74 | datadir=/db/mysql 75 | ... 76 | [root@caldera01 ~]# systemctl start mysqld 77 | [root@caldera01 ~]# systemctl status mysqld 78 | [root@caldera01 ~]# systemctl enable mysqld 79 | ``` 80 | 81 | From GitHub, install a sample database with an integrated test suite, used to test your applications and database servers 82 | ``` 83 | [root@caldera01 ~]# yum install -y git 84 | [root@caldera01 ~]# git clone https://github.com/datacharmer/test_db.git 85 | [root@caldera01 ~]# cd /db/test_db 86 | ``` 87 | 88 | Load the test database 89 | ``` 90 | [root@caldera01 ~]# mysql < employees.sql 91 | ``` 92 | 93 | If you want to install with two large partitioned tables, alternatively, run 94 | ``` 95 | [root@caldera01 ~]# mysql < employees_partitioned.sql 96 | ``` 97 | 98 | Run the test suite 99 | ``` 100 | [root@caldera01 ~]# mysql -t < test_employees_md5.sql 101 | +----------------------+ 102 | | INFO | 103 | +----------------------+ 104 | | TESTING INSTALLATION | 105 | +----------------------+ 106 | +--------------+------------------+----------------------------------+ 107 | | table_name | expected_records | expected_crc | 108 | +--------------+------------------+----------------------------------+ 109 | | employees | 300024 | 4ec56ab5ba37218d187cf6ab09ce1aa1 | 110 | | departments | 9 | d1af5e170d2d1591d776d5638d71fc5f | 111 | | dept_manager | 24 | 8720e2f0853ac9096b689c14664f847e | 112 | | dept_emp | 331603 | ccf6fe516f990bdaa49713fc478701b7 | 113 | | titles | 443308 | bfa016c472df68e70a03facafa1bc0a8 | 114 | | salaries | 2844047 | fd220654e95aea1b169624ffe3fca934 | 115 | +--------------+------------------+----------------------------------+ 116 | +--------------+------------------+----------------------------------+ 117 | | table_name | found_records | found_crc | 118 | +--------------+------------------+----------------------------------+ 119 | | employees | 300024 | 4ec56ab5ba37218d187cf6ab09ce1aa1 | 120 | | departments | 9 | d1af5e170d2d1591d776d5638d71fc5f | 121 | | dept_manager | 24 | 8720e2f0853ac9096b689c14664f847e | 122 | | dept_emp | 331603 | ccf6fe516f990bdaa49713fc478701b7 | 123 | | titles | 443308 | bfa016c472df68e70a03facafa1bc0a8 | 124 | | salaries | 2844047 | fd220654e95aea1b169624ffe3fca934 | 125 | +--------------+------------------+----------------------------------+ 126 | +--------------+---------------+-----------+ 127 | | table_name | records_match | crc_match | 128 | +--------------+---------------+-----------+ 129 | | employees | OK | ok | 130 | | departments | OK | ok | 131 | | dept_manager | OK | ok | 132 | | dept_emp | OK | ok | 133 | | titles | OK | ok | 134 | | salaries | OK | ok | 135 | +--------------+---------------+-----------+ 136 | ``` 137 | 138 | We want to add 2 additional LUNs via iSCSI protocol to the LVM layout. The iSCSI makes the system able to see the external LUNs as additional disks, called ``/dev/sdc`` and ``/dev/sdd`` 139 | Each additional disk is of 232.9G 140 | 141 | ``` 142 | [root@caldera01 ~]# lsblk 143 | NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT 144 | sda 8:0 0 232.9G 0 disk 145 | ├─sda1 8:1 0 500M 0 part /boot 146 | └─sda2 8:2 0 232.4G 0 part 147 | ├─os-swap 253:0 0 7.8G 0 lvm [SWAP] 148 | ├─os-root 253:1 0 50G 0 lvm / 149 | └─os-data 253:2 0 174.6G 0 lvm /data 150 | sdb 8:16 0 232.9G 0 disk 151 | └─sdb1 8:17 0 232.9G 0 part 152 | └─vgdb-lvol1 253:4 0 232.9G 0 lvm /db 153 | sdc 8:32 0 232.9G 0 disk 154 | sdd 8:48 0 232.9G 0 disk 155 | ``` 156 | 157 | Now let's to extend the LV ``lvol1`` by using the 2 additional LUNs 158 | ``` 159 | [root@caldera01 ~]# pvcreate -f /dev/sdc 160 | Wiping iso9660 signature on /dev/sdc. 161 | Wiping dos signature on /dev/sdc. 162 | Physical volume "/dev/sdc" successfully created 163 | 164 | [root@caldera01 ~]# pvcreate -f /dev/sdd 165 | Wiping dos signature on /dev/sdd. 166 | Physical volume "/dev/sdd" successfully created 167 | 168 | [root@caldera01 ~]# pvscan 169 | PV /dev/sda2 VG os lvm2 [232.39 GiB / 0 free] 170 | PV /dev/sdb1 VG vgdb lvm2 [232.88 GiB / 0 free] 171 | PV /dev/sdc lvm2 [232.89 GiB] 172 | PV /dev/sdd lvm2 [232.89 GiB] 173 | 174 | [root@caldera01 ~]# vgextend vgdb /dev/sdc /dev/sdd 175 | Volume group "vgdb" successfully extended 176 | 177 | [root@caldera01 ~]# vgscan 178 | Reading all physical volumes. This may take a while... 179 | Found volume group "os" using metadata type lvm2 180 | Found volume group "vgdb" using metadata type lvm2 181 | 182 | [root@caldera01 ~]# vgdisplay vgdb 183 | --- Volume group --- 184 | VG Name vgdb 185 | System ID 186 | Format lvm2 187 | Metadata Areas 3 188 | Metadata Sequence No 3 189 | VG Access read/write 190 | VG Status resizable 191 | MAX LV 0 192 | Cur LV 1 193 | Open LV 1 194 | Max PV 0 195 | Cur PV 3 196 | Act PV 3 197 | VG Size 698.64 GiB 198 | PE Size 4.00 MiB 199 | Total PE 178852 200 | Alloc PE / Size 59618 / 232.88 GiB 201 | Free PE / Size 119234 / 465.76 GiB 202 | VG UUID O557zn-CcSI-1Ec4-LryC-uqi8-B42R-pYTHKU 203 | 204 | [root@caldera01 ~]# lvscan 205 | ACTIVE '/dev/os/root' [50.00 GiB] inherit 206 | ACTIVE '/dev/os/data' [174.63 GiB] inherit 207 | ACTIVE '/dev/os/swap' [7.77 GiB] inherit 208 | ACTIVE '/dev/vgdb/lvol1' [232.88 GiB] inherit 209 | 210 | [root@caldera01 ~]# lvextend -L +400G /dev/vgdb/lvol1 211 | Size of logical volume vgdb/lvol1 changed from 232.88 GiB (59618 extents) to 632.88 GiB (162018 extents). 212 | Logical volume lvol1 successfully resized 213 | [root@caldera01 ~]# 214 | 215 | [root@caldera01 ~]# lvscan 216 | ACTIVE '/dev/os/root' [50.00 GiB] inherit 217 | ACTIVE '/dev/os/data' [174.63 GiB] inherit 218 | ACTIVE '/dev/os/swap' [7.77 GiB] inherit 219 | ACTIVE '/dev/vgdb/lvol1' [632.88 GiB] inherit 220 | ``` 221 | 222 | Now the LV successfully increased its size by 632G but the file system still is 232G 223 | ``` 224 | [root@caldera01 ~]# df -Th 225 | Filesystem Type Size Used Avail Use% Mounted on 226 | /dev/mapper/os-root xfs 50G 2.1G 48G 5% / 227 | devtmpfs devtmpfs 3.8G 0 3.8G 0% /dev 228 | tmpfs tmpfs 3.8G 0 3.8G 0% /dev/shm 229 | tmpfs tmpfs 3.8G 8.6M 3.8G 1% /run 230 | tmpfs tmpfs 3.8G 0 3.8G 0% /sys/fs/cgroup 231 | /dev/mapper/os-data xfs 175G 256M 175G 1% /data 232 | /dev/sda1 xfs 497M 190M 308M 39% /boot 233 | /dev/mapper/vgdb-lvol1 ext3 230G 642M 217G 1% /db 234 | ``` 235 | 236 | Extend the file system without umount it 237 | ``` 238 | [root@caldera01 ~]# resize2fs -p /dev/vgdb/lvol1 239 | resize2fs 1.42.9 (28-Dec-2013) 240 | Filesystem at /dev/vgdb/lvol1 is mounted on /db; on-line resizing required 241 | old_desc_blocks = 15, new_desc_blocks = 40 242 | The filesystem on /dev/vgdb/lvol1 is now 165906432 blocks long. 243 | ``` 244 | 245 | Let's check our data file 246 | ``` 247 | [root@caldera01 ~]# cd /db/test_db 248 | [root@caldera01 ~]# mysql -t < test_employees_md5.sql 249 | 250 | +----------------------+ 251 | | INFO | 252 | +----------------------+ 253 | | TESTING INSTALLATION | 254 | +----------------------+ 255 | +--------------+------------------+----------------------------------+ 256 | | table_name | expected_records | expected_crc | 257 | +--------------+------------------+----------------------------------+ 258 | | employees | 300024 | 4ec56ab5ba37218d187cf6ab09ce1aa1 | 259 | | departments | 9 | d1af5e170d2d1591d776d5638d71fc5f | 260 | | dept_manager | 24 | 8720e2f0853ac9096b689c14664f847e | 261 | | dept_emp | 331603 | ccf6fe516f990bdaa49713fc478701b7 | 262 | | titles | 443308 | bfa016c472df68e70a03facafa1bc0a8 | 263 | | salaries | 2844047 | fd220654e95aea1b169624ffe3fca934 | 264 | +--------------+------------------+----------------------------------+ 265 | +--------------+------------------+----------------------------------+ 266 | | table_name | found_records | found_crc | 267 | +--------------+------------------+----------------------------------+ 268 | | employees | 300024 | 4ec56ab5ba37218d187cf6ab09ce1aa1 | 269 | | departments | 9 | d1af5e170d2d1591d776d5638d71fc5f | 270 | | dept_manager | 24 | 8720e2f0853ac9096b689c14664f847e | 271 | | dept_emp | 331603 | ccf6fe516f990bdaa49713fc478701b7 | 272 | | titles | 443308 | bfa016c472df68e70a03facafa1bc0a8 | 273 | | salaries | 2844047 | fd220654e95aea1b169624ffe3fca934 | 274 | +--------------+------------------+----------------------------------+ 275 | +--------------+---------------+-----------+ 276 | | table_name | records_match | crc_match | 277 | +--------------+---------------+-----------+ 278 | | employees | OK | ok | 279 | | departments | OK | ok | 280 | | dept_manager | OK | ok | 281 | | dept_emp | OK | ok | 282 | | titles | OK | ok | 283 | | salaries | OK | ok | 284 | +--------------+---------------+-----------+ 285 | ``` 286 | 287 | Our data is safe! Now we want to resize the logical volume to use only 1 LUN instead of 2 LUNs as we made before. Firt of all, shutdown any service or application that is using the logical volume we want to resize. In our case, we have to stop the MySQL database running on the volume 288 | ``` 289 | [root@caldera01 test_db]# systemctl stop mysql 290 | [root@caldera01 test_db]# systemctl status mysql 291 | mysqld.service - MySQL Community Server 292 | Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled) 293 | Active: inactive (dead) since Thu 2015-10-22 12:13:28 CEST; 2min 12s ago 294 | Main PID: 15388 (code=exited, status=0/SUCCESS) 295 | ``` 296 | 297 | Make a backup of ALL yuor data present on the volume 298 | ``` 299 | [root@caldera01 test_db]# tar cvf data_backup.tar * 300 | [root@caldera01 test_db]# gzip data_backup.tar 301 | [root@caldera01 test_db]# mv data_backup.tar /data_backup_folder 302 | ``` 303 | 304 | Umount the LV 305 | ``` 306 | [root@caldera01 ~]# umount /db 307 | ``` 308 | 309 | Then check for the file-system error (Must pass in every 5 steps of file-system check if not there might be some issue with your file-system) 310 | 311 | ``` 312 | [root@caldera01 /]# e2fsck -ff /dev/vgdb/lvol1 313 | e2fsck 1.42.9 (28-Dec-2013) 314 | Pass 1: Checking inodes, blocks, and sizes 315 | Pass 2: Checking directory structure 316 | Pass 3: Checking directory connectivity 317 | Pass 4: Checking reference counts 318 | Pass 5: Checking group summary information 319 | /dev/vgdb/lvol1: 403/41484288 files (10.9% non-contiguous), 2803302/165906432 blocks 320 | ``` 321 | 322 | Now let's to resize the file system. Since we want to remove the last LUN, we have to calculate the final size of the file system by summing its original size ``/dev/sdb1`` to the size of the first added LUN ``/dev/sdc``. We have ``/dev/sdb1`` Total PE 59618 + ``/dev/sdc`` Total PE 59617 = Total PE 119235. Total size, expressed in Megabyte are 4 x 119235 = 476940 MB 323 | 324 | ``` 325 | [root@caldera01 ~]# pvdisplay 326 | --- Physical volume --- 327 | PV Name /dev/sdb1 328 | VG Name vgdb 329 | PV Size 232.88 GiB / not usable 2.18 MiB 330 | Allocatable yes (but full) 331 | PE Size 4.00 MiB 332 | Total PE 59618 333 | Free PE 0 334 | Allocated PE 59618 335 | PV UUID mymhL6-Ta6m-cx3V-LjKM-Zo2T-K7Xj-MI1jb3 336 | 337 | --- Physical volume --- 338 | PV Name /dev/sdc 339 | VG Name vgdb 340 | PV Size 232.89 GiB / not usable 7.18 MiB 341 | Allocatable yes (but full) 342 | PE Size 4.00 MiB 343 | Total PE 59617 344 | Free PE 0 345 | Allocated PE 59617 346 | PV UUID 0dIQWO-jjaa-zgl3-1o8E-8kGR-4xjc-BxPmwO 347 | 348 | --- Physical volume --- 349 | PV Name /dev/sdd 350 | VG Name vgdb 351 | PV Size 232.89 GiB / not usable 7.18 MiB 352 | Allocatable yes 353 | PE Size 4.00 MiB 354 | Total PE 59617 355 | Free PE 16834 356 | Allocated PE 42783 357 | PV UUID Df0M7R-6KDB-TUeZ-l1Gw-2r5J-ci4c-MsyNw4 358 | 359 | [root@caldera01 ~]# fsadm -e -y resize /dev/vgdb/lvol1 476940M 360 | resize2fs 1.42.9 (28-Dec-2013) 361 | Resizing the filesystem on /dev/mapper/vgdb-lvol1 to 122096640 (4k) blocks. 362 | The filesystem on /dev/mapper/vgdb-lvol1 is now 122096640 blocks long. 363 | ``` 364 | 365 | Reduce the Logical Volume and remove the last LUN from the Volume Group 366 | ``` 367 | [root@caldera01 ~]# lvreduce -L 476940M /dev/vgdb/lvol1 368 | WARNING: Reducing active logical volume to 465.76 GiB 369 | THIS MAY DESTROY YOUR DATA (filesystem etc.) 370 | Do you really want to reduce lvol1? [y/n]: y 371 | Size of logical volume vgdb/lvol1 changed from 632.88 GiB (162018 extents) to 465.76 GiB (119235 extents). 372 | Logical volume lvol1 successfully resized 373 | 374 | [root@caldera01 ~]# vgreduce vgdb /dev/sdd 375 | Removed "/dev/sdd" from volume group "vgdb" 376 | [root@caldera01 ~]# 377 | ``` 378 | 379 | Remove the physical volume 380 | ``` 381 | [root@caldera01 ~]# pvremove /dev/sdd 382 | Labels on physical volume "/dev/sdd" successfully wiped 383 | [root@caldera01 ~]# 384 | ``` 385 | 386 | And check again the file system and mount back it 387 | ``` 388 | [root@caldera01 ~]# e2fsck -ff /dev/vgdb/lvol1 389 | e2fsck 1.42.9 (28-Dec-2013) 390 | Pass 1: Checking inodes, blocks, and sizes 391 | Pass 2: Checking directory structure 392 | Pass 3: Checking directory connectivity 393 | Pass 4: Checking reference counts 394 | Pass 5: Checking group summary information 395 | /dev/vgdb/lvol1: 403/30531584 files (10.9% non-contiguous), 2116084/122096640 blocks 396 | [root@caldera01 ~]# 397 | 398 | [root@caldera01 ~]# mount /dev/vgdb/lvol1 /db 399 | [root@caldera01 ~]# df -Th /db 400 | Filesystem Type Size Used Avail Use% Mounted on 401 | /dev/mapper/vgdb-lvol1 ext3 459G 653M 439G 1% /db 402 | [root@caldera01 ~]# 403 | ``` 404 | 405 | Start the mysql service and check for data integrity 406 | ``` 407 | [root@caldera01 ~]# systemctl start mysqld 408 | [root@caldera01 ~]# cd /db/test_db/ 409 | [root@caldera01 test_db]# mysql -t < test_employees_md5.sql 410 | +----------------------+ 411 | | INFO | 412 | +----------------------+ 413 | | TESTING INSTALLATION | 414 | +----------------------+ 415 | +--------------+------------------+----------------------------------+ 416 | | table_name | expected_records | expected_crc | 417 | +--------------+------------------+----------------------------------+ 418 | | employees | 300024 | 4ec56ab5ba37218d187cf6ab09ce1aa1 | 419 | | departments | 9 | d1af5e170d2d1591d776d5638d71fc5f | 420 | | dept_manager | 24 | 8720e2f0853ac9096b689c14664f847e | 421 | | dept_emp | 331603 | ccf6fe516f990bdaa49713fc478701b7 | 422 | | titles | 443308 | bfa016c472df68e70a03facafa1bc0a8 | 423 | | salaries | 2844047 | fd220654e95aea1b169624ffe3fca934 | 424 | +--------------+------------------+----------------------------------+ 425 | +--------------+------------------+----------------------------------+ 426 | | table_name | found_records | found_crc | 427 | +--------------+------------------+----------------------------------+ 428 | | employees | 300024 | 4ec56ab5ba37218d187cf6ab09ce1aa1 | 429 | | departments | 9 | d1af5e170d2d1591d776d5638d71fc5f | 430 | | dept_manager | 24 | 8720e2f0853ac9096b689c14664f847e | 431 | | dept_emp | 331603 | ccf6fe516f990bdaa49713fc478701b7 | 432 | | titles | 443308 | bfa016c472df68e70a03facafa1bc0a8 | 433 | | salaries | 2844047 | fd220654e95aea1b169624ffe3fca934 | 434 | +--------------+------------------+----------------------------------+ 435 | +--------------+---------------+-----------+ 436 | | table_name | records_match | crc_match | 437 | +--------------+---------------+-----------+ 438 | | employees | OK | ok | 439 | | departments | OK | ok | 440 | | dept_manager | OK | ok | 441 | | dept_emp | OK | ok | 442 | | titles | OK | ok | 443 | | salaries | OK | ok | 444 | +--------------+---------------+-----------+ 445 | +------------------+ 446 | | computation_time | 447 | +------------------+ 448 | | 00:00:14 | 449 | +------------------+ 450 | +---------+--------+ 451 | | summary | result | 452 | +---------+--------+ 453 | | CRC | OK | 454 | | count | OK | 455 | +---------+--------+ 456 | 457 | ``` 458 | 459 | -------------------------------------------------------------------------------- /content/working_with_files.md: -------------------------------------------------------------------------------- 1 | ### The file streams 2 | When commands are executed, by default there are three standard file streams or descriptors always open for use: 3 | 4 | 1. standard input or **stdin** 5 | 2. standard output or **stdout** 6 | 3. standard error or **stderr** 7 | 8 | Usually, **stdin** is your keyboard, **stdout** and **stderr** are printed on your terminal; often **stderr** is redirected to an error logging file. The **stdin** is often supplied by directing input to come from a file or from the output of a previous command through a pipe. The **stdout** is also often redirected into a file. Since **stderr** is where error messages are written, often nothing will go there. 9 | 10 | In Linux, all open files are represented internally by what are called file descriptors. Simply put, these are represented by numbers starting at zero. The **stdin** is file descriptor 0, **stdout** is file descriptor 1, and **stderr** is file descriptor 2. Typically, if other files are opened in addition to these three, which are opened by default, they will start at file descriptor 3 and increase from there. 11 | 12 | We can redirect the three standard filestreams so that we can get input from either a file or another command instead of from our keyboard, and we can write output and errors to files or send them as input for subsequent commands. For example, having a program *called do_something* that reads from **stdin** and writes to **stdout** and **stderr**, we can change its input source: 13 | ``` 14 | $ do_something < input-file 15 | ``` 16 | If you want to send the output to a file, use the this as in: 17 | ``` 18 | $ do_something > output-file 19 | ``` 20 | We can pipe the output of one command or program into another as its input. 21 | ``` 22 | $ command1 | command2 | command3 23 | ``` 24 | The above represents what we often call a _pipeline_ and allows linux to combine the actions of several commands into one. 25 | 26 | ### Search for files 27 | The ``locate`` utility performs a search through a previously constructed database of files and directories on your system, matching all entries that contain a specified character string. The ``locate`` utilizes the database created by another program, ``updatedb``. Most Linux systems run this automatically once a day. However, you can update it at any time by just running ``updatedb`` from the command line as the root user. 28 | ``` 29 | # yum install -y mlocate 30 | # updatedb 31 | # locate zip 32 | ``` 33 | The result of ``locate`` utility can sometimes result in a very long list. To get a shorter more relevant list we can use the ``grep`` program as a filter. It will print only the lines that contain one or more specified strings as in: 34 | ``` 35 | $ locate zip | grep bin 36 | /usr/bin/gpg-zip 37 | /usr/bin/gunzip 38 | /usr/bin/gzip 39 | /usr/bin/zipdetails 40 | ``` 41 | which will list all files and directories with both "zip" and "bin" in their name. 42 | 43 | Wildcards can be used in search for a filename containing specific characters. 44 | 45 | |Wildcards|Result| 46 | |---------|-----------| 47 | |? |Matches any single character| 48 | |* |Matches any string of characters| 49 | |[set] |Matches any character not in the set of characters| 50 | |[!set]|Matches any character not in the set of characters| 51 | 52 | The ``find`` is extremely useful and often-used utility program in the daily life of a Linux system administrator. It recurses down the filesystem tree from any particular directory (or set of directories) and locates files that match specified conditions. The default is always the present working directory. 53 | ``` 54 | $ find /var -name *.log 55 | /var/log/audit/audit.log 56 | /var/log/tuned/tuned.log 57 | /var/log/anaconda/anaconda.log 58 | /var/log/anaconda/anaconda.program.log 59 | /var/log/anaconda/anaconda.packaging.log 60 | /var/log/anaconda/anaconda.storage.log 61 | ``` 62 | When no arguments are given, ``find`` lists all files in the current directory and all of its subdirectories. 63 | 64 | Searching for files and directories named "gcc": 65 | ``` 66 | $ find /usr -name gcc 67 | ``` 68 | Searching only for directories named "gcc": 69 | ``` 70 | $ find /usr -type d -name gcc 71 | ``` 72 | Searching only for regular files named "test1": 73 | ``` 74 | $ find /usr -type f -name test1 75 | ``` 76 | Another good use of ``find`` is being able to run commands on the files that match your search criteria. To find and remove all files that end with .swp: 77 | ``` 78 | $ find -name "*.swp" -exec rm {} ’;’ 79 | $ find -name "*.swp" -ok rm {} \; 80 | ``` 81 | The {} is a place holder that will be filled with all the file names that result from the find expression, and the preceding command will be run on each one individually. Note that you have to end the command with either ``‘;’`` or ``\;`` Both forms are fine. The second form behaves the same as the first one except that find will prompt you for permission before executing the command. This makes it a good way to test your results before blindly executing any potentially dangerous commands. 82 | 83 | It is sometimes the case that you wish to find files according to attributes such as when they were created, last used, etc, or based on their size. Both are easy to accomplish. 84 | 85 | Finding based on time: 86 | ``` 87 | $ find / -ctime 3 88 | ``` 89 | 90 | Here, _-ctime_ is when the inode meta-data (i.e., file ownership, permissions, etc) last changed; it is often, but not necessarily when the file was first created. You can also search for accessed/last read _-atime_ or modified/last written _-mtime_ times. The number is the number of days and can be expressed as either a number (n) that means exactly that value, +n which means greater than that number, or -n which means less than that number. 91 | 92 | Finding based on sizes: 93 | ``` 94 | $ find / -size +10M 95 | ``` 96 | To find files greater than 10 MB in size. 97 | 98 | ### Manage files 99 | Use the following utilities to view files: 100 | 101 | |Command|Usage| 102 | |-------|-----------| 103 | |cat |Used for viewing files that are not very long| 104 | |tac |Used to look at a file backwards, starting with the last line| 105 | |less |Used to view larger files because it is a paging program; it pauses at each screenful of text, provides scroll-back capabilities, and lets you search and navigate within the file.| 106 | |tail |Used to print the last 10 lines of a file by default. You can change the number of lines by doing -n 15 or just -15 if you wanted to look at the last 15 lines instead of the default| 107 | |head |The opposite of tail; by default it prints the first 10 lines of a file| 108 | 109 | The ``touch`` command is often used to set or update the access, change, and modify times of files. By default it resets a file's time stamp to match the current time. 110 | 111 | However, you can also create an empty file using touch: 112 | ``` 113 | $ touch 114 | ``` 115 | This is normally done to create an empty file as a placeholder for a later purpose. 116 | The -t option allows you to set the date and time stamp of the file. 117 | To set the time stamp to a specific time: 118 | ``` 119 | $ touch -t 03201600 120 | ``` 121 | This sets the file, myfile's, time stamp to 4 p.m., March 20th (03 20 1600). 122 | 123 | The ``mkdir`` command is used to create a directory. Removing a directory is simply done with ``rmdir`` command. The directory must be empty or it will fail. 124 | ``` 125 | # mkdir ./test 126 | # rmdir ./test 127 | # 128 | # mkdir ./test 129 | # mkdir ./test/inside 130 | # rmdir ./test 131 | rmdir: failed to remove ‘test’: Directory not empty 132 | # rm -rf ./test 133 | # ls ./test 134 | ls: cannot access ./test: No such file or directory 135 | ``` 136 | 137 | ### Compare files 138 | The ``diff`` command is used to compare files and directories. 139 | 140 | ``` 141 | $ cat file1.txt 142 | Amor, ch'a nullo amato amar perdona, 143 | Mi prese del costui piacer si forte, 144 | Che, come vedi, ancor non m'abbandona. 145 | $ 146 | $ cat file2.txt 147 | amor, ch'a nullo amato amar perdona, 148 | mi prese del costui piacer si forte, 149 | che, come vedi, ancor non m'abbandona. 150 | $ 151 | $ diff file1.txt file2.txt 152 | < Amor, ch'a nullo amato amar perdona, 153 | < Mi prese del costui piacer si forte, 154 | < Che, come vedi, ancor non m'abbandona. 155 | --- 156 | > amor, ch'a nullo amato amar perdona, 157 | > mi prese del costui piacer si forte, 158 | > che, come vedi, ancor non m'abbandona. 159 | $ 160 | $ diff -c file1.txt file2.txt 161 | *** file1.txt 2015-02-17 16:10:03.781804799 +0100 162 | --- file2.txt 2015-02-17 16:13:41.059088459 +0100 163 | *************** 164 | ! Amor, ch'a nullo amato amar perdona, 165 | ! Mi prese del costui piacer si forte, 166 | ! Che, come vedi, ancor non m'abbandona. 167 | --- 1,3 ---- 168 | ! amor, ch'a nullo amato amar perdona, 169 | ! mi prese del costui piacer si forte, 170 | ! che, come vedi, ancor non m'abbandona. 171 | $ 172 | $ diff -i file1.txt file2.txt 173 | $ 174 | ``` 175 | ### The file utility 176 | In Linux, a file's extension often does not categorize it the way it might in other operating systems. One can not assume that a file named ``file.txt`` is a text file and not an executable program. In Linux a file name is generally more meaningful to the user of the system than the system itself; in fact most applications directly examine a file's contents to see what kind of object it is rather than relying on an extension. The real nature of a file can be ascertained by using the ``file`` utility. For the file names given as arguments, it examines the contents and certain characteristics to determine whether the files are plain text, shared libraries, executable programs, scripts, or something else. 177 | 178 | ``` 179 | $ file /etc/resolv.conf 180 | /etc/resolv.conf: ASCII text 181 | ``` 182 | -------------------------------------------------------------------------------- /img/active-active-cluster.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MinhKMA/Linux-Tutorial/439d81d6a3d44e1299fae970a222c6d6f66b1563/img/active-active-cluster.jpg --------------------------------------------------------------------------------