├── CONTRIBUTING.md ├── README.md ├── apache.md ├── arp.md ├── bibliography.md ├── cellular-networks.md ├── cgi.md ├── curl.md ├── dhpc.md ├── dig.md ├── dns.md ├── ethernet.md ├── fastcgi.md ├── firewall.md ├── ftp.md ├── hardware.md ├── hostname.md ├── http.md ├── ifconfig.md ├── ip.md ├── iptables.md ├── mail.md ├── media-protocols.md ├── mitproxy.md ├── mtu.md ├── nat.md ├── netcat.md ├── netrc.md ├── netstat.md ├── network-manager.md ├── network-simulation.md ├── nginx.md ├── nmap.md ├── openssl.md ├── physical-layer.md ├── ping.md ├── port.md ├── proxy-server.md ├── rest.md ├── route.md ├── samba.md ├── sip.md ├── smtp.md ├── squid.md ├── ssh.md ├── standards.md ├── tcp.md ├── tcpdump.md ├── tcpflow.md ├── telnet.md ├── tls.md ├── tor.md ├── traceroute.md ├── tshark.md ├── url.md ├── vocabulary.md ├── voip.md ├── vpn.md ├── web-server-vs-app-server.md ├── wget.md ├── whois.md ├── wireshark.md ├── xinetd.md └── zeroconf.md /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing 2 | 3 | Markdown style: wrap:no. 4 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Networking Cheat 2 | 3 | Networking information and cheatsheets. 4 | 5 | Includes both protocols and utilities. Almost all OS specifics currently only discuss Linux. Additions of info for other OSs are welcome. 6 | 7 | 1. [Standards](standards.md) 8 | 1. Protocols 9 | 1. [Physical layer](physical-layer.md) 10 | 1. Data link layer 11 | 1. [ARP](arp.md) 12 | 1. [Ethernet](ethernet.md) 13 | 1. Cellular networks 14 | 1. Transport layer 15 | 1. [TCP](tcp.md) 16 | 1. [Port](port.md) 17 | 1. Internet layer 18 | 1. [IP](ip.md) 19 | 1. [traceroute](traceroute.md) 20 | 1. [route](route.md) 21 | 1. Application layer 22 | 1. [DNS](dns.md) 23 | 1. [DHPC](dhpc.md) 24 | 1. [Hostname](hostname.md) 25 | 1. [HTTP](http.md) 26 | 1. [Proxy server](proxy-server.md) 27 | 1. [REST](rest.md) 28 | 1. [SMTP](smtp.md) 29 | 1. [TLS](tls.md) 30 | 1. [URL](url.md) 31 | 1. [Media protocols](media-protocols.md) 32 | 1. Tools 33 | 1. [cURL](curl.md) 34 | 1. [dig](dig.md) 35 | 1. [ifconfig](ifconfig.md) 36 | 1. [iptables](iptables.md) 37 | 1. [nmap](nmap.md) 38 | 1. [Netcat](netcat.md) 39 | 1. [netstat](netstat.md) 40 | 1. [network-manager](work-manager.md) 41 | 1. [OpenSSL](openssl.md) 42 | 1. [ping](ping.md) 43 | 1. [ssh](ssh.md) 44 | 1. [wget](wget.md) 45 | 1. [Tor](tor.md) 46 | 1. Servers 47 | 1. [Apache](apache.md) 48 | 1. [Nginx](nginx.md) 49 | 1. [Squid](squid.md) 50 | 1. Sniffers 51 | 1. [TShark](tshark.md) 52 | 1. [tcpdump](tcpdump.md) 53 | 1. [tcpflow](tcpflow.md) 54 | 1. [Wireshark](wireshark.md) 55 | 1. [Hardware](hardware.md) 56 | 1. [Vocabulary](vocabulary.md) 57 | 1. [Bibliography](bibliography.md) 58 | 59 | TODO 60 | 61 | 1. , done by Arista. 62 | 1. 63 | -------------------------------------------------------------------------------- /apache.md: -------------------------------------------------------------------------------- 1 | # Apache 2 | 3 | Cheat on the apache server. 4 | 5 | Documentation: 6 | 7 | Test apache: 8 | 9 | firefox http://localhost/ & 10 | 11 | ## Introduction 12 | 13 | Apache is a web server 14 | 15 | A web server listens to a port (default 80) for strings 16 | 17 | Theses strings are http requests 18 | 19 | Then it takes the http request, processes it, and then returns the request to the client 20 | 21 | Part of the processing may be passed to another program: typically a CGI script 22 | 23 | ## Test preparations 24 | 25 | Before doing anything, make this test dir: 26 | 27 | mkdir test 28 | cd test 29 | echo '

index.html

' > index.html 30 | echo '

a.html

' > a.html 31 | 32 | mkdir a 33 | cd a 34 | echo '

a/index.html

' > index.html 35 | echo '

a/a.html

' > a.html 36 | cd .. 37 | 38 | mkdir auth 39 | cd auth 40 | echo '

auth/index.html

' > index.html 41 | echo '

auth/a.html

' > a.html 42 | cd .. 43 | 44 | mkdir noindex 45 | cd noindex 46 | echo '

noindex/a.html

' > a.html 47 | echo '

noindex/b.html

' > b.html 48 | cd .. 49 | 50 | cd ../.. 51 | 52 | Finally move our test dir to the serve root: 53 | 54 | sudo mv test /var/www/ 55 | 56 | The default root for serving files is specified in the <#conf file> by the `DocumentRoot` directive. In current Ubuntu, it is `/var/www/` 57 | 58 | The user under which the web server runs must have read access to this directory. *This is the default on Ubuntu*, where the apache server runs as user `www-root`! 59 | 60 | Usually this user is a different user from `root` for security. 61 | 62 | ## conf file 63 | 64 | Ubuntu default location for the configuration file: 65 | 66 | sudo vim /etc/apache2/apache2.conf 67 | 68 | This file may include others, and for example in Ubuntu the default template does include: 69 | 70 | IncludeOptional conf-enabled/*.conf 71 | IncludeOptional sites-enabled/*.conf 72 | 73 | so that local configurations can be managed in separate files. 74 | 75 | Ubuntu default also creates `*-available` directories, which contain possible configuration files. Those should be symlinked to the `enabled` directories to enable them. 76 | 77 | Configurations only apply when you restart apache: 78 | 79 | sudo service apache2 restart 80 | 81 | ## minimum conf file 82 | 83 | The bare minimum conf file to get a file served is: 84 | 85 | Listen 80 86 | User www-data 87 | Group www-data 88 | ErrorLog /var/log/apache2/error.log 89 | 90 | This conf may be useful for testing server configuration. 91 | 92 | ## default operation 93 | 94 | "web subdirs" map directly to local dirs. 95 | 96 | Ubuntu default is currently `/var/www/` 97 | 98 | Open file `/var/www/test/index.html`: 99 | 100 | firefox localhost/test/index.html 101 | 102 | Going to a dir on the web browser opens the contained index.html file by default: 103 | 104 | firefox localhost/test/ 105 | 106 | This can be configured with the `DirectoryIndex` directive 107 | 108 | If no index is contained, apache generates an html index: 109 | 110 | firefox localhost/test/noindex/ 111 | 112 | ### DocumentRoot 113 | 114 | Set Apache serve root at given dir: 115 | 116 | DocumentRoot "/var/www/root" 117 | 118 | For this to work, make sure `DocumentRoot` is not set anywhere else. (by default it was included in the include files, `grep -r DocumenRoot` shows where) 119 | 120 | For security concerns, only put things you want apache to serve directly inside `DocumentRoot` such as HTML, CSS and images. 121 | 122 | Stuff that users should not see such as CGI scripts and *gasp* SSL certificates are better to remain outside it, so that you don't serve them by mistake! 123 | 124 | ### Listen 125 | 126 | Listen those ports on all interfaces (for example, first wireless card, first ethernet card, etc...): 127 | 128 | Listen 80 129 | Listen 8000 130 | 131 | This configuration is mandatory. 132 | 133 | Listen those ports on given interfaces: 134 | 135 | Listen 192.0.2.1:80 136 | Listen 192.0.2.5:8000 137 | 138 | ### AccessFileName 139 | 140 | Name of the file which can modify access properties of a directory. 141 | 142 | AccessFileName .htaccess 143 | 144 | ### AllowOverride 145 | 146 | Allows the `.htaccess` to override certain directives of earlier conf files. 147 | 148 | Allow to override all directives: 149 | 150 | AllowOverride All 151 | 152 | Allow to override no directories (the ifle is ignored): 153 | 154 | AllowOverride None 155 | 156 | ### Include 157 | 158 | Copy paste Include other apache conf files or entire directories into the current configuration: 159 | 160 | Include file.conf 161 | Include conf-d 162 | 163 | ### Deny 164 | 165 | Deny access from given host 166 | 167 | Deny from 10.252.46.165 168 | Deny from host.example.com 169 | 170 | ### DirectoryIndex 171 | 172 | What to do when user accesses a directory location: 173 | 174 | DirectoryIndex index.html index.php /cgi-bin/index.pl 175 | 176 | SAME: 177 | 178 | DirectoryIndex index.html 179 | DirectoryIndex index.php 180 | DirectoryIndex /cgi-bin/index.pl 181 | 182 | With this, for the entire site, first looks in order for: 183 | 184 | - `index.html` 185 | - `index.php` 186 | - `/cgi-bin/index.pl` 187 | 188 | Note how you can specify a script outside of that dir. 189 | 190 | In case none of those actions match, the default is for `mod_autoindex` to generate an html directory listing. 191 | 192 | For specific dirs, use the `Directory` directive. 193 | 194 | #### mod_autoindex 195 | 196 | Generates automatic html listings for dirs 197 | 198 | Turn off automatic listings for a given dir: 199 | 200 | 201 | Options -Indexes 202 | 203 | 204 | Will simply give a not found 205 | 206 | Ignore certain files in the listing: 207 | 208 | IndexIgnore tmp* .. 209 | 210 | Add headers/footers before/after index: 211 | 212 | HeaderName header.html 213 | ReadmeName footer.html 214 | 215 | Same header/footer for every dir 216 | 217 | HeaderName header.html 218 | HeaderName /site/header.html 219 | ReadmeName /site/footer.html 220 | 221 | Use predefined styles: 222 | 223 | IndexOptions FancyIndexing HTMLTable 224 | 225 | Use given CSS style: 226 | 227 | IndexStyleSheet /css/autoindex.css 228 | 229 | ## VirtualHost 230 | 231 | Allows to host many DNS names on a single IP. 232 | 233 | ## sections 234 | 235 | Sections are commands which restrict the scope of application of other configurations. 236 | 237 | The official manual page: 238 | 239 | ### Files 240 | 241 | Acts on local filesystem. 242 | 243 | Deny file permissions for files that match regex `"^\.ht"`: 244 | 245 | 246 | Order allow,deny 247 | Deny from all 248 | Satisfy all 249 | 250 | 251 | Order says: first process all allow directives, then all deny directives. Since `Deny` came last, it has precedence. 252 | 253 | ### Directory 254 | 255 | Acts on local filesystem 256 | 257 | 258 | Options +Indexes 259 | 260 | 261 | ### Location 262 | 263 | Applies configuration to URL addresses: 264 | 265 | 266 | Order Allow,Deny 267 | Deny from all 268 | 269 | 270 | ### combine sections 271 | 272 | It is possible to combine multiple section scopes: 273 | 274 | 275 | 276 | Order allow,deny 277 | Deny from all 278 | 279 | 280 | 281 | ### IfDefine 282 | 283 | 284 | Redirect / http://otherserver.example.com/ 285 | 286 | 287 | ### IfVersion 288 | 289 | = 2.1> 290 | this happens only in versions greater or 291 | equal 2.1.0. 292 | 293 | 294 | ## alias 295 | 296 | Allow to create virtual paths to dirs and files. 297 | 298 | ### Sources 299 | 300 | 301 | 302 | Create virtual directory: 303 | 304 | Alias /test/alias /var/www/test 305 | 306 | firefox localhost/test/alias & 307 | 308 | Also works for subdirs: 309 | 310 | firefox localhost/test/alias/a.html & 311 | firefox localhost/test/alias/a & 312 | 313 | Also works for files: 314 | 315 | Alias /testfile/ /var/www/test/index.html 316 | 317 | firefox localhost/testfile & 318 | 319 | Also works outside of serve root: 320 | 321 | cd 322 | echo "TEST" > index.html 323 | 324 | Alias /test/alias-out-root/ /home/ciro/ 325 | 326 | firefox localhost/test/alias-out-root 327 | 328 | ### first match takes precedence 329 | 330 | Alias /test/alias/a /var/www/test 331 | Alias /test/alias /var/www/test 332 | 333 | firefox localhost/test/alias/ & 334 | 335 | Goes to `test/index.html` 336 | 337 | firefox localhost/test/alias/a & 338 | 339 | Goes to `test/a/index.html` 340 | 341 | BAD: both go to `test/index.html`: 342 | 343 | Alias /test/alias /var/www/test 344 | Alias /test/alias/a /var/www/test 345 | 346 | firefox localhost/test/alias/ & 347 | firefox localhost/test/alias/a & 348 | 349 | ### Redirect 350 | 351 | Returns a redirect HTTP response. Takes precedence over aliases. 352 | 353 | Alias /test/redir /test 354 | Redirect /test/redir http://www.google.com 355 | 356 | The following goes to google: 357 | 358 | firefox localhost/test/redir & 359 | 360 | ### CGI 361 | 362 | #### FastCGI 363 | 364 | Implementations: `mod_fastcgi`, `mod_fcgid`. Vs: 365 | 366 | #### ScriptAlias 367 | 368 | The script: 369 | 370 | echo '#!/usr/bin/perl 371 | print "Content-type: text/html"; 372 | first output line must be "Content-type: text/html\n\n" 373 | print "Status: 500 Internal Server Error" 374 | print "\n\n" 375 | print "

environment

" 376 | foreach $key (keys %ENV) { 377 | print "$key --> $ENV{$key}
"; 378 | } 379 | print "" 380 | ' > sudo tee /usr/lib/cgi-bin/test.pl 381 | sudo chmod +x /usr/lib/cgi-bin/test.pl 382 | 383 | ##### status 384 | 385 | Optional, if not given supposes `200 OK`. 386 | 387 | If given as error, server will simply give the error and no data. 388 | 389 | Uncomment the status line on the test script to see what happens. 390 | 391 | ##### alias to dir 392 | 393 | CGI scripts must be in the dir specified by script alias: 394 | 395 | ScriptAlias /mycgi /usr/lib/cgi-bin 396 | 397 | Same as: 398 | 399 | Alias /mycgi /usr/lib/cgi-bin 400 | 401 | 402 | Tell server that all files inside this dir are cgi scripts: 403 | 404 | SetHandler cgi-script 405 | 406 | Tell server that all .pl and .py files in dire are cgi scripts: 407 | 408 | AddHandler cgi-script .cgi .pl 409 | 410 | Permit CGI execution for scripts in this dir: 411 | 412 | Options +ExecCGI 413 | 414 | 415 | Run it: 416 | 417 | firefox localhost/mycgi/test.pl 418 | 419 | Note how `ScriptAlias` created a virtual directory not present in the actual filesystem. 420 | 421 | Can also make individual script: 422 | 423 | ScriptAlias /test/cgi-file /usr/lib/cgi-bin/test.pl 424 | 425 | ##### alias to script 426 | 427 | All subdirs of `testpl` are generated by the given `test.pl`: 428 | 429 | ScriptAlias /test/testpl /usr/lib/cgi-bin/test.pl 430 | 431 | firefox localhost/testpl/ & 432 | firefox localhost/testpl/a.html & 433 | 434 | #### action 435 | 436 | Run script whenever an HTML file is accessed: 437 | 438 | Action test /cgi-bin/test.pl 439 | AddHandler test .html 440 | 441 | TODO: i get `Action` directive undefined... solve this. 442 | 443 | Try it: 444 | 445 | firefox localhost/index.html 446 | 447 | This is how PHP does it! 448 | 449 | ## modules 450 | 451 | Apache plugins are called modules 452 | 453 | Modules are compiled `.so` files 454 | 455 | Modules may define new directives 456 | 457 | For modules to become effective they must be loaded in the config file 458 | 459 | Only do certain commands if module is exists: 460 | 461 | 462 | commands... 463 | 464 | 465 | Load a module: 466 | 467 | LoadModule fastcgi_module /usr/lib/apache2/modules/mod_fastcgi.so 468 | 1 2 469 | 470 | - 1: module identifier hard coded in module? 471 | - 2: full path to .so 472 | 473 | ### a2enmodule 474 | 475 | `Apache2 ENable Module`. 476 | 477 | Utility that enables modules easily. 478 | 479 | Probably adds `LoadModule` somewhere. 480 | 481 | List options: 482 | 483 | a2enmod 484 | 485 | Enable a module: 486 | 487 | sudo a2enmod $MODULE_NAME 488 | 489 | ## handlers 490 | 491 | Part of the very default mime_module 492 | 493 | Determines file types and sets default actions accordingly 494 | 495 | Example: 496 | 497 | Action add-footer /cgi-bin/footer.pl 498 | AddHandler add-footer .html 499 | 500 | - `Action`: defines a handler called add-footer 501 | - `AddHandler`: uses the handler called add-footer for all html files 502 | 503 | Handlers can be defined in modules 504 | 505 | ## authentication 506 | 507 | You must chose *both* one <#method> and one <#provider>! 508 | 509 | ### methods 510 | 511 | #### prerequisites 512 | 513 | First understand HTTP authentication. 514 | 515 | What algorithm is used to store the passwords more or less safely. 516 | 517 | #### basic authentication 518 | 519 | Provided by `mod_auth_basic` 520 | 521 | Apache conf: 522 | 523 | LoadModule auth_basic_module /usr/lib/apache2/modules/mod_auth_basic.so 524 | 525 | AuthType Basic 526 | AuthName "private dir" 527 | AuthBasicProvider file 528 | AuthUserFile /var/.htpasswd 529 | Require valid-user 530 | AllowOverride None 531 | 532 | 533 | #### digest 534 | 535 | Provided by `mod_auth_digest`. 536 | 537 | `mod_auth_digest` is better than `mod_auth_basic`, so use digest! 538 | 539 | LoadModule auth_digest_module /usr/lib/apache2/modules/mod_auth_digest.so 540 | 541 | AuthType Digest 542 | AuthName "private dir" 543 | AuthDigestProvider file 544 | AuthUserFile /var/.htpasswd 545 | Require valid-user 546 | AllowOverride None 547 | 548 | 549 | ### provider 550 | 551 | What type of storage is used for user password pairs 552 | 553 | Is specified by the `AuthBasicProvider` directive. 554 | 555 | #### file 556 | 557 | A plain text file 558 | 559 | Safer to put outside serve root 560 | 561 | ##### htpasswd 562 | 563 | Generates `.htpasswd` files 564 | 565 | Generate user/pass pairs: 566 | 567 | sudo htpasswd -bc /var/www/.htpasswd u p 568 | 569 | - `-c`: creates new file, destroying old one! *Necessary first time!* 570 | - `-b`: use pass from command line. *Less safe!* 571 | 572 | sudo htpasswd -b /var/www/.htpasswd u2 p 573 | 574 | Lets take a look at the file: 575 | 576 | sudo cat /var/www/.htpasswd 577 | 578 | Note that the passwords are base64 encoded. 579 | 580 | #### dbd 581 | 582 | SQL database 583 | 584 | ## Try it out!! 585 | 586 | Test: 587 | 588 | firefox localhost/test/auth & 589 | 590 | Try `u` and `u2` and pass `p`! 591 | 592 | ### browser cache 593 | 594 | firefox localhost/test/auth & 595 | firefox localhost/test/auth & 596 | 597 | The second time, you may not be prompted for a password! 598 | 599 | This is because Firefox has cached your password for some time and resent it automatically! There is no server state. 600 | 601 | To avoid the cache use curl: 602 | 603 | curl -I localhost/test/auth 604 | 605 | `401` and `WWW-Authenticate`. 606 | 607 | With pass: 608 | 609 | curl u:p@localhost/test/auth 610 | curl -u u:p localhost/test/auth 611 | 612 | Of course, better using the `-u` option which could work also for different authentication methods. 613 | 614 | ## PHP 615 | 616 | Interpreter language almost always run from a server to generate web content. 617 | 618 | Dominates web today, but faces increasing concurrence `python/ruby/perl`. 619 | 620 | Test: 621 | 622 | sudo service apache2 restart 623 | echo '' | sudo tee /var/www/testphp.php 624 | firefox http://localhost/testphp.php & 625 | 626 | If you see PHP specs, it works! 627 | -------------------------------------------------------------------------------- /arp.md: -------------------------------------------------------------------------------- 1 | # ARP 2 | 3 | Address Resolution Protocol. 4 | 5 | 1982 6 | 7 | Only used when the sender detects that the searched IP is on the same network as itself. 8 | 9 | In that case, it can simply get the destination MAC address and send the packages over the LAN directly to the destination without passing through the router. 10 | 11 | ARP is a protocol that does just that: it finds the MAC address from an IP on a LAN. 12 | 13 | ## ARP table 14 | 15 | Cache that caches previously resolved IP to MAC addresses so that ARP requests don't need to be made every time. 16 | 17 | Also known as: ARP cache, MAC cache. 18 | 19 | ## arp utility 20 | 21 | CLI utility that shows the ARP table. 22 | 23 | To see a computer on the table, remember that you first must have tried to contact it somehow, 24 | so first ping that computer: 25 | 26 | timeout 3 ping 192.168.1.3; arp -a 27 | -------------------------------------------------------------------------------- /bibliography.md: -------------------------------------------------------------------------------- 1 | # Bibliography 2 | 3 | ## Free 4 | 5 | - 6 | 7 | ## Non-free 8 | 9 | - [stevens - 2012 - tcp ip illustrate volume 1 2nd edition][ste12] 10 | 11 | Extremely clear and exemplified. 12 | 13 | Gives due emphasis to the key points. 14 | 15 | Great first book. 16 | 17 | - [dostalek - 2006 - Understanding tcp ip][dos06] 18 | 19 | Explains the most important Internet protocol suite protocols deeply. 20 | 21 | Might not be the best first TCP IP book because it is a bit advanced, but it is a very good second one once you know the basics. 22 | 23 | [dos06]: http://www.amazon.com/Understanding-TCP-IP-ebook/dp/B007TUYE0G/ 24 | [ste12]: http://www.amazon.com/TCP-Illustrated-Volume-Addison-Wesley-Professional/dp/0321336313 25 | -------------------------------------------------------------------------------- /cellular-networks.md: -------------------------------------------------------------------------------- 1 | # Cellular networks 2 | 3 | 4 | 5 | ## Static IP 6 | 7 | - http://apple.stackexchange.com/questions/50271/is-it-possible-to-access-an-iphone-over-ssh-on-3g 8 | - https://www.quora.com/How-do-I-connect-through-SSH-to-a-device-using-Verizon-4G-LTE-from-the-Internet 9 | - http://superuser.com/questions/741050/how-to-allow-ssh-into-a-device-on-remote-4g-modem-when-provider-is-blocking-inbo 10 | -------------------------------------------------------------------------------- /cgi.md: -------------------------------------------------------------------------------- 1 | # CGI 2 | 3 | TODO where is it specified? says it's not maintained anymore. 4 | 5 | Protocol of how a server communicates with a CGI script. 6 | 7 | A CGI script is simply a script/executable that outputs the part of HTTP response. 8 | 9 | This part includes some last header lines which the server delegates to it, notably `content type`, followed by `\n\n`, followed by the entire body. 10 | 11 | The server passes information to the script through environment variables only. 12 | -------------------------------------------------------------------------------- /curl.md: -------------------------------------------------------------------------------- 1 | # cURL 2 | 3 | CLI utility that does several web protocols, including HTTP, FTP, SMTP, DICT. 4 | 5 | More powerful than `wget`: only use `wget` for recursive mirroring. 6 | 7 | Not POSIX, and there is no POSIX 7 alternative: . 8 | 9 | Ubuntu install: 10 | 11 | sudo aptitude install -y curl 12 | 13 | ## Basic usage 14 | 15 | Make a `GET / HTTP/1.1` request to Google, wait for response, and print response: 16 | 17 | curl google.com 18 | 19 | ## Test cURL 20 | 21 | cURL does not have a dry-run option built-in: 22 | 23 | There are however a few options to visualize what it is doing: 24 | 25 | - `-v` and other verbosity options. 26 | 27 | - using `nc -l` and curl `-m 1`: 28 | 29 | nc -l localhost 8000 & 30 | curl -m 1 localhost:8000 31 | 32 | ## v 33 | 34 | ## trace 35 | 36 | ## trace-ascii 37 | 38 | `-v` , `--trace "$FILE"`, `--trace-ascii "$FILE"`: increasing levels of log verbosity. 39 | 40 | `-` to stdout. 41 | 42 | Print all data IO and curl status: 43 | 44 | curl --trace - "$URL" 45 | 46 | Good way to see what is going on. 47 | 48 | curl -Lv google.com 49 | 50 | ## m 51 | 52 | ## max-time 53 | 54 | Timeout for entire operation. 55 | 56 | ## L 57 | 58 | Follow redirects. 59 | 60 | Omit redirect page if any. 61 | 62 | Example: 63 | 64 | curl google.pn 65 | curl -L google.pn 66 | 67 | Good example if you are not one of the 100 people who live in Pitcairn island =): Google redirects you to your countries domain. 68 | 69 | With `-v` you can see the full transaction: 70 | 71 | curl -vL google.pn 72 | 73 | ## HTTP 74 | 75 | ### POST 76 | 77 | ### d 78 | 79 | Make POST request: 80 | 81 | curl -d "a=1" "$URL" 82 | 83 | Data from stdin with `-d @-`: 84 | 85 | echo 'a=1' | curl -d @- "$URL" 86 | 87 | Multiple data are joined by an ampersand `&`: 88 | 89 | curl -d 'a=1' -d 'b=2' "$URL" 90 | 91 | ### form 92 | 93 | ### F 94 | 95 | Multipart POST request like done from an HTML form by a browser: 96 | 97 | echo "Content of a.txt" > a.txt 98 | curl -F "key1=val1" -F "file1=@a.txt" "$URL" 99 | 100 | ### H 101 | 102 | ### header 103 | 104 | Custom header. 105 | 106 | Overrides default cURL headers. 107 | 108 | curl -d '{"a":"b"}' -H "Content-Type:application/json" "$URL" 109 | 110 | ### i 111 | 112 | Show received HTTP headers. 113 | 114 | Example: 115 | 116 | curl -i google.com 117 | 118 | TODO vs `-D -` 119 | 120 | ### I 121 | 122 | Make HTTP HEAD request: 123 | 124 | curl -I google.com 125 | 126 | Implies `-i` of course. 127 | 128 | ### X 129 | 130 | Use custom HTTP method: 131 | 132 | curl -X 'GET' google.com 133 | 134 | Many methods have an specific option for them. 135 | 136 | ### data-urlencode 137 | 138 | Encode spaces and other signs for you: 139 | 140 | curl -d "name=I%20am%20Ciro" $URL 141 | curl --data-urlencode "name=I am Ciro" $URL 142 | 143 | ## a-z range 144 | 145 | curl ftp://ftp.uk.debian.org/debian/pool/main/[a-z]/ 146 | 147 | ## FTP 148 | 149 | Download: 150 | 151 | curl -u ftpuser:ftppass -O ftp://ftp_server/public_html/xss.php 152 | 153 | Upload: 154 | 155 | curl -u ftpuser:ftppass -T myfile.txt ftp://ftp.testserver.com 156 | curl -u ftpuser:ftppass -T "{file1,file2}" ftp://ftp.testserver.com 157 | 158 | ## email 159 | 160 | ## SMTP 161 | 162 | Send email: 163 | 164 | echo $'sent by curl!\n.' | curl --mail-from user@gmail.com --mail-rcpt user@gmail.com smtp://gmail.com 165 | 166 | Body ends with a single dot `.` on a line. 167 | 168 | ## DICT 169 | 170 | curl dict://dict.org/show:db #dictionnaries 171 | curl dict://dict.org/d:bash #general 172 | curl dict://dict.org/d:bash:foldoc #computing 173 | 174 | ## Basic authentication 175 | 176 | ## u 177 | 178 | Does Basic authentication. 179 | 180 | `--digest` and `--ntlm` can be used together. 181 | 182 | If no `:pass`, will ask for pass on command line. 183 | 184 | Examples: 185 | 186 | curl -u user:pass site.with.basic.auth.com 187 | curl --digest -u user:pass site.with.digest.auth.com 188 | 189 | ## x 190 | 191 | Specify proxy server: 192 | 193 | curl -x proxysever.test.com:3128 194 | 195 | ## z 196 | 197 | Download iff the file was modified after given date time: 198 | 199 | curl -z 01-Jan-00 google.com 200 | 201 | ## raw 202 | 203 | Don't decode HTTP specific `Content-Encoding` and `Transfer-Encoding`. 204 | 205 | E.g.: 206 | 207 | printf 'HTTP/1.1 200 OK 208 | Transfer-Encoding: chunked 209 | 210 | 2 211 | hi 212 | 0 213 | 214 | ' | sed -E 's/$/\r/' | nc -l 8000 215 | 216 | Without raw: 217 | 218 | curl localhost 8000 219 | 220 | we see the decoded body: 221 | 222 | hi 223 | 224 | With `--raw`, we see: 225 | 226 | 2 227 | hi 228 | 0 229 | (newline) 230 | (newline) 231 | 232 | ## Applications 233 | 234 | Get only response for POST: 235 | 236 | Get only status of request: 237 | 238 | With `HEAD` it is easy through `-I`. 239 | 240 | ## Keep alive 241 | 242 | Yes by default: 243 | 244 | ## Parallel downloads 245 | 246 | Seems no, `xargs -P` it: 247 | -------------------------------------------------------------------------------- /dhpc.md: -------------------------------------------------------------------------------- 1 | # DHCP 2 | 3 | Dynamic Host Configuration Protocol. 4 | 5 | - IPv4: (1997) 6 | - IPv6: (2003), extended by many others 7 | 8 | 9 | 10 | Application layer protocol that automatically assigns configurations to the hosts on a network, such as their IP. 11 | 12 | Default IANA ports: UDP 67 and 68 (same as its less advanced and less common predecessor `BOOTP`). 13 | 14 | When a computer enters a network and it does not know its own IP. 15 | 16 | It must first send a DHCP request to be assigned an IP. 17 | 18 | The server usually (TODO always?) runs in the router, and can be configured from the router's interface. 19 | 20 | ## Static IP 21 | 22 | On a home network that you control, it is better to use intuitive hostnames and let the addresses be dynamically set via DHPC, unless you absolutely need a static IP, for example to setup a server behind your router. 23 | 24 | DHPC does not know about static IPs: if you set one you must make sure that it is outside of the DHPC range. DHPC is done by the router, and should be configurable from the browser interface. 25 | 26 | On my Numericable router, under Network > Basic Settings > IP LAN > I have two fields: `Starting IP Address` and `Ending IP Address` which allow me to control it. By default, the range is 192.168.0.10 to 192.168.0.50, which is a sensible default allowing for 8 small static IPs between 2 and 9, 1 being the router's address. 27 | 28 | On Ubuntu 12.04, there are a few ways of doing it. 29 | 30 | ## Static IP with DHCP reservation 31 | 32 | This is not strictly static, but it is the simplest option. 33 | 34 | On you router configuration, find the DHCP Reservation Lease Infos. 35 | 36 | This allows you to map LAN IPs directly to MAC addresses. 37 | 38 | On my Numericable router it is found under the IP LAN tab. 39 | 40 | Set the interface to use DHCP. 41 | 42 | The assigned address must be in the DHCP range. 43 | 44 | ## Static IP with NetworkManager 45 | 46 | Using the NetworkManager GUI: 47 | 48 | nm-connection-editor 49 | 50 | Select connection > Edit > IPv4 settings, configure. 51 | 52 | TODO fails. I lose internet connection on the interface 53 | 54 | ## Static IP with NetworkManager 55 | 56 | Using `/etc/network/interfaces`: 57 | 58 | sudo vim /etc/network/interfaces 59 | 60 | Set the file to: 61 | 62 | auto lo 63 | iface lo inet loopback 64 | 65 | auto eth0 66 | iface eth0 inet static 67 | address 192.168.0.2 68 | netmask 255.255.255.0 69 | gateway 192.168.0.1 70 | dns-nameservers 89.2.0.1 71 | dns-search example.com 72 | #network 192.168.0.0 73 | 74 | - `auto if1 if2`: automatically create interfaces `if1` and `if2` on `ifup -a`. 75 | - `iface if1`: from now on, define properties of `if1`. 76 | 77 | TODO fails. I lose internet connection on the interface 78 | 79 | ## See also 80 | 81 | `zeroconf` 82 | -------------------------------------------------------------------------------- /dig.md: -------------------------------------------------------------------------------- 1 | # dig 2 | 3 | CLI utility that shows complete path of domain to IP resolution, as it passes through multiple CNAMEs. 4 | 5 | Get IP for given hostname: 6 | 7 | dig +short example.com 8 | -------------------------------------------------------------------------------- /dns.md: -------------------------------------------------------------------------------- 1 | # DNS 2 | 3 | Domain Name System. 4 | 5 | - 6 | - 7 | - clarifies that domains are case insensitive. 8 | 9 | Part of the application layer. 10 | 11 | Standard IANA port: 53/UDP 12 | 13 | Protocol that convert strings into IPs, for example: 14 | 15 | http://www.google.com -> 173.194.34.34 16 | 17 | Before before using an address such as `www.google.com`, any program such as a browser must first resolve the hostname `www.google.com` into an IP by asking that from a server. 18 | 19 | Linux systems usually offer `man resolver` C library interface, which any program can use to resolve DNS names. The resolver library may cache results across applications that have already been resolved. 20 | 21 | ## DNS on WAN 22 | 23 | On the Internet, hostnames are resolved to IPs by DNS servers. 24 | 25 | You must pay to reserve hostnames so they can be resolved to the IP of your choice. 26 | 27 | TODO how to DNS servers find out all the hostnames in the world? 28 | 29 | ## DNS on LAN 30 | 31 | DNS can also be done for local networks: 32 | 33 | computer2 -> 192.168.0.3 34 | 35 | In which case the DNS server normally resides on the router. 36 | 37 | Client computers on the network are informed that it is a DNS server via DHCP. 38 | 39 | On your LAN, people can use the host name to communicate between computers 40 | 41 | For example, John is running an Apache server on the usual port 80. He has hostname `john`. 42 | 43 | Mary is on the same network. Therefore, she can refer to `john` simply as `john`. For example: 44 | 45 | ping john 46 | firefox john 47 | 48 | TODO if many people set up the same hostname, then what? 49 | 50 | ## Wildcard DNS 51 | 52 | It is possible to redirect all subdomains that don't match any other rule to a single IP by using a wildcard DNS record: 53 | 54 | - 55 | - 56 | 57 | This can be used to implement functionality like GitHub pages on a website, allowing users to have their own subdomains for potentially non safe HTML content, while avoiding CSRF attacks. 58 | 59 | ## host utility 60 | 61 | Does DNS and rDNS 62 | 63 | DNS: 64 | 65 | host www.google.com 66 | 67 | Sample output: 68 | 69 | www.google.com has address 74.125.206.147 70 | www.google.com has address 74.125.206.106 71 | www.google.com has address 74.125.206.104 72 | www.google.com has address 74.125.206.105 73 | www.google.com has address 74.125.206.103 74 | www.google.com has address 74.125.206.99 75 | www.google.com has IPv6 address 2a00:1450:400c:c0a::93 76 | 77 | rDNS: 78 | 79 | host 173.194.40.194 80 | 81 | Sample output: 82 | 83 | 194.40.194.173.in-addr.arpa domain name pointer par10s12-in-f2.1e100.net 84 | 85 | TODO understand that output. Why is `google.com` nowhere to be seen? 86 | 87 | TODO: why does `host 74.125.206.147` (one of the IPs for `www.google.com`) give: 88 | 89 | Host 147.206.125.74.in-addr.arpa. not found: 3(NXDOMAIN) 90 | 91 | ## resolv.conf 92 | 93 | cat /etc/resolv.conf 94 | 95 | Lists DNS servers. 96 | 97 | This file may be automatically generated by utilities. 98 | 99 | On Ubuntu 12.04, you should never edit that file manually. By default it contains: 100 | 101 | nameserver 127.0.1.1 102 | 103 | which is `localhost`, and is used indirectly by the NetworkManger system, which you should use instead. 104 | 105 | TODO does it specify the config file location? `resolv.conf` 106 | 107 | ## getaddrinfo 108 | 109 | POSIX function to resolve hostnames: 110 | 111 | 112 | 113 | ## hostname utility 114 | 115 | Print currently desired hostname: 116 | 117 | echo $HOSTNAME 118 | hostname 119 | 120 | In the default bash `PS1` line for Ubuntu and many systems you see: `ciro@ciro-Thinkpad-T430`, then the hostname is `ciro-Thinkpad-T430`. 121 | 122 | Change hostname for cur session: 123 | 124 | h= 125 | sudo hostname "$h" 126 | 127 | prompt `PS1` is not changed immediately. 128 | 129 | ## Change hostname permanently 130 | 131 | h= 132 | echo "$h" | sudo tee /etc/hostname 133 | 134 | ### Set hostname in Windows 135 | 136 | Host is referred to as "computer name". Good name choice, that is exactly what host is. 137 | 138 | wmic computersystem where name="%COMPUTERNAME%" call rename name="NEW-NAME" 139 | 140 | ## Zone file 141 | 142 | When you register for a domain of your own, you will start thinking about this: it is the main setting on your registrar interface. 143 | 144 | 145 | 146 | ### apex domain 147 | 148 | `@` in the zone file means the domain you own without any subdomain. 149 | 150 | E.g., if you own `cirosantilli.com`, `@` means `cirosantilli.com` itself, while `www` means `www.cirosantilli.com`. 151 | 152 | Apex domains are more restrictive than subdomains, and certain hosting services advise against it, such as GitHub Pages. 153 | 154 | The main problem is that in services such as GitHub pages you don't get an actual IP, so you can't point the Apex to an IP (which is simple), and the `CNAME` "workaround" is not good enough in that case. 155 | 156 | ### naked domain 157 | 158 | The apex domain is sometimes called naked domain, since it has no subdomain. 159 | 160 | ## CNAME record 161 | 162 | TODO File that tells DNS to redirect to another domain name, creating an alias. 163 | 164 | 165 | 166 | ## A 167 | 168 | Points a domain to an IP. The final part of the resolution. 169 | 170 | ## rDNS 171 | 172 | ## Reverse DNS 173 | 174 | 175 | 176 | Protocol that transforms an IP into a hostname. 177 | 178 | Not always supported on all DNS servers. 179 | 180 | ## DDNS 181 | 182 | Dynamic DNS. 183 | 184 | A way to update DNS as IPs change. 185 | 186 | Useful for example if you want to give a hostname for your home network, in which the IP is dynamic for most ISPs. A DDNS service like can give you a persistent hostname anyways. 187 | 188 | TODO what is it exactly? How does it work? A protocol? Part of DNS? 189 | 190 | ## Tools 191 | 192 | - dig 193 | -------------------------------------------------------------------------------- /ethernet.md: -------------------------------------------------------------------------------- 1 | # Ethernet 2 | 3 | 4 | 5 | Family of technologies in layer 1 and 2: 6 | -------------------------------------------------------------------------------- /fastcgi.md: -------------------------------------------------------------------------------- 1 | # FastCGI 2 | 3 | Spec: 4 | 5 | Successor of CGI. 6 | 7 | Faster because binary. TODO: other advantages? How does it work exactly? 8 | -------------------------------------------------------------------------------- /firewall.md: -------------------------------------------------------------------------------- 1 | # Firewall 2 | 3 | TODO 4 | -------------------------------------------------------------------------------- /ftp.md: -------------------------------------------------------------------------------- 1 | #FTP 2 | 3 | FTP is: 4 | 5 | - TCP/IP file transfer protocol 6 | - a command line utility with the same name which implements the client for that protocol 7 | 8 | ##hosts 9 | 10 | To have some real fun, try commands on a real host. 11 | 12 | Most free Apache/PHP web hosts offer FTP mainly for users to upload their sites: just find the one with the most space and uptime. This is a list of good ones. 13 | 14 | Main quality parameters to consider: 15 | 16 | - max data 17 | - max data transfer 18 | - max file size 19 | 20 | ###freehostingnoads 21 | 22 | 23 | 24 | Free Hosting No Ads 25 | 26 | - 20 GB space 27 | - 200 GB traffic 28 | 29 | URLs of form: 30 | 31 | Deleted my data after 30 days inactivity! 32 | 33 | ##Commands 34 | 35 | The `ftp` utility only contains very low level commands in bijection to the protocol. 36 | 37 | See all available commands: 38 | 39 | ? 40 | 41 | Do a local shell command: 42 | 43 | ! pwd 44 | 45 | Connect from command line option: 46 | 47 | ftp ftp.domain.com 48 | 49 | Connect from FTP REPL: 50 | 51 | open ftp.domain.com 52 | 53 | Disconnect and but keep program open: 54 | 55 | bye 56 | 57 | Disconnect and exit program: 58 | 59 | exit 60 | 61 | `ls` remote: 62 | 63 | ls 64 | 65 | `cd` remote: 66 | 67 | cd 68 | 69 | `cd` local: 70 | 71 | lcd 72 | 73 | `pwd` remote: 74 | 75 | pwd 76 | 77 | Upload file with same basename: 78 | 79 | put a 80 | 81 | File `a` exists in current local dir 82 | 83 | Does not work for dirs 84 | 85 | Upload file with different basename: 86 | 87 | put a b 88 | 89 | Download file with same basename in current dir: 90 | 91 | get a 92 | 93 | Download with different basename in current dir: 94 | 95 | get a b 96 | 97 | Download on relative path: 98 | 99 | get d/a 100 | 101 | Subdir must exist locally. 102 | 103 | Delete remote file: 104 | 105 | del a 106 | 107 | Create a remote directory: 108 | 109 | mkdir d 110 | 111 | Remove an empty remote directory: 112 | 113 | rm d 114 | 115 | Cannot do multiple commands per line: 116 | 117 | #ls; ls 118 | 119 | ###Recursive directory operations 120 | 121 | It seems that it is not possible to do recursive directory operations like download, remove or upload on non empty directories with a single command: . LFTP is a possible solution. 122 | 123 | ##LFTP 124 | 125 | Implements more convenient high level command line interface 126 | 127 | Seems backwards compatible with the `ftp` utility 128 | 129 | Give user from command line argument: 130 | 131 | lftp -u user host.ftp.com 132 | 133 | Give commands from the command line: 134 | 135 | user= 136 | url= 137 | lftp -c "open -u $user $url 138 | ls 139 | ls" 140 | 141 | `-c` must be the only option. 142 | 143 | `-f file` to read command from a file instead. `-f` must be the only option. 144 | 145 | Could not find a way to read commands from stdin: `-f -` does not work... 146 | 147 | Multiple commands per line: 148 | 149 | ls; ls 150 | 151 | Execute only of last worked: 152 | 153 | ls && ls 154 | 155 | Execute only of last failed: 156 | 157 | ls || ls 158 | 159 | Group commands: 160 | 161 | ls && ( ls || ls ) 162 | 163 | Recursive directory download: 164 | 165 | mirror d 166 | 167 | Recursive directory upload (Reverse mirror) 168 | 169 | mirror -R d 170 | 171 | Recursive directory remove: 172 | 173 | rm -r d 174 | 175 | ##FileZilla 176 | 177 | GUI FTP manager. 178 | 179 | Stores connexion passwords/usernames and performs recursive upload and download. 180 | -------------------------------------------------------------------------------- /hardware.md: -------------------------------------------------------------------------------- 1 | # Hardware 2 | 3 | ## Router vs switch 4 | 5 | Cisco - What Is a Switch? What Is a Router? 6 | 7 | Router: level 3. 8 | 9 | Switch: level 2. 10 | -------------------------------------------------------------------------------- /hostname.md: -------------------------------------------------------------------------------- 1 | # Host 2 | 3 | A host is anything able to send and receive packages over a network: this includes workstations (computers) and routers. 4 | 5 | Can be specified by either 6 | 7 | - an IP 8 | - a string that will be resolved by a DNS server to an IP 9 | 10 | TODO merge with DNS. 11 | 12 | ## Host user pair 13 | 14 | A user may access a (system) computer from another computer using for example ssh. 15 | 16 | To do so, he must be registered in the target computer. 17 | 18 | This is why user/host pairs are common: the host pair says from which computer user is trying to access his account. 19 | 20 | ## Hostname 21 | 22 | An alias for an IP, local or remote. 23 | 24 | Must be converted into an IP via DNS. 25 | 26 | When outside the local network, the hostname is added before the domain name, e.g. in: 27 | 28 | www.google.com 29 | 30 | - hostname: `www` 31 | - domain name: `google.com` 32 | 33 | It is not a good idea to have a dot `.` in your domain name, since then how could its last part be distinguished from the domain name? 34 | 35 | TODO is the hostname `www.google.com` or just `google.com`? Contradictory answers: 36 | 37 | ### www 38 | 39 | `www.google.com` and `google.com` are completely different hosts, and can lead to different IPs. 40 | 41 | What sane companies do is choose one and redirect the other, *be consistent*. 42 | 43 | But I have seen companies that use `www` for a different website than without, and it is possible that no redirection happens. 44 | 45 | Browsers can store different cookies for both, so you can be logged in at `www.a.com` but not at `a.com`. 46 | 47 | In the case of FTP, `ftp://ftp.a.com` URLs which are common, and perhaps in that case it is better to keep the `ftp` and redirect HTTP requests to `ftp.a.com` to `ftp://ftp.a.com` since FTP is less used than HTTP, allowing users to type simply `ftp.a.com` instead of `ftp://a.com`. 48 | 49 | It is more recommended today not to use the `www` is noise: . 50 | 51 | `www` was more used in the past, so older companies may continue to use them because they are stuck with it. 52 | 53 | As of early 2014: 54 | 55 | - `facebook.com` redirects to `www.facebook.com` 56 | - `google.com` redirects to `www.google.com` 57 | 58 | ## Domain name 59 | 60 | E.g.: `google.com`, `stackoverflow.com` are commonly called domain names. 61 | 62 | A more precise way of speaking is saying that `google` is a subdomain of `com`, and `www` is a subdomain of `google.com`. 63 | 64 | They identify a network owned by Google. But in order to get an actual IP, you still need to add a hostname such as `www`. 65 | 66 | Domain names may contain more than one `.`: `bbc.co.uk`. 67 | 68 | ### Subdomain 69 | 70 | The subdomain can include a period (.) but not as the first or last character. Consecutive periods (...) are not allowed. A subdomain cannot exceed 25 characters. 71 | 72 | ### example.com 73 | 74 | is a test domain reserved by IANA. 75 | 76 | It is a serves as a great URL placeholder on simple examples. 77 | 78 | ## Top level domain 79 | 80 | `.com`, `.net`, `.io`, `.fr` are examples. 81 | 82 | Every name must be under one of those. 83 | 84 | They are controlled by IANA, and there are not that many out there except for the country ones: 85 | 86 | To get a country TLD, it seems that you must have some link with the country. 87 | 88 | Some TLDs are reserved for certain uses and registrars must check that you/ your organization are eligible: `.gov` for governments, `.mit` for US military. 89 | 90 | Some interesting ones: 91 | 92 | - `.sexy` and `.xxx`. Guess what. 93 | - `.guru`. No suggested use. Funny. 94 | 95 | Some country ones have become generic: `.io` is a notable example, popular amongst startups as of 2014-03. Short, sounds good, reminds of IO input output. 96 | 97 | Some top level country domain names offer free domains! 98 | 99 | ## hosts file 100 | 101 | Located at: 102 | 103 | /etc/hosts 104 | 105 | Tells your computer where to redirect the given names. 106 | 107 | Takes precedence over DNS. 108 | 109 | Big downside: you have to have one of this file on every PC. 110 | 111 | Therefore, use a DNS server instead 112 | 113 | cat /etc/hosts 114 | 115 | Redirect Wikipiedia to localhost: 116 | 117 | echo "127.0.0.1 www.wikipedia.org" | sudo -a /etc/hosts 118 | 119 | Now: 120 | 121 | firefox www.wikipedia.org & 122 | 123 | will go to localhost, and you will see your Apache page if you are running apache. 124 | 125 | Undo that, its silly: 126 | 127 | sudo sed -i "$ d" /etc/hosts 128 | 129 | On Windows the file is: 130 | 131 | C:\Windows\System32\Drivers\Etc\hosts 132 | 133 | ## Change hostname 134 | 135 | - 136 | - 137 | 138 | Best method: 139 | 140 | hostnamectl set-hostname 'new-hostname' 141 | 142 | ## Regulation aspects 143 | 144 | ### Single letter domain names 145 | 146 | 147 | 148 | Before 1993, some of those domains were allowed. The remaining ones were then reserved, 149 | 150 | 2 letter domains are not reserved, and many are owned by major corporations. E.g. Facebook bought `fb.com` for 8 million. 151 | 152 | ### Original top-level domains 153 | 154 | 155 | 156 | When the net started, there was only `com`, `org`, `net`, `gov`, `mil`, `int`. 157 | 158 | ### Sponsored top-level domains 159 | 160 | A few domains that some groups pay for, and for which you must be in the group to have. 161 | 162 | ## hostname utility 163 | 164 | Get hostname: 165 | 166 | hostname 167 | 168 | Likely same as; 169 | 170 | cat /proc/sys/kernel/hostname 171 | -------------------------------------------------------------------------------- /http.md: -------------------------------------------------------------------------------- 1 | # HTTP 2 | 3 | HTTP is the protocol which browsers use to request pages from servers. 4 | 5 | HTTP is part of the application layer. 6 | 7 | The port is 80/TCP. 8 | 9 | It is the main way that applications talk to servers. 10 | 11 | HTTP does not find a server: it only determines exactly which characters must be passed to a server to get the data what one wants. 12 | 13 | A major application is to request that a server send a web page. Such requests are made by browsers whenever you open a web page. 14 | 15 | There are however many other applications outside browsers: any program can send an HTTP request. One important example are REST interfaces, which allows programs to talk to servers. One example of a REST interface is the GitHub API , which allows programs to do anything that can be done through the browser on GitHub. This can be used to create applications that interact with GitHub's data such as third party analytics tools like . 16 | 17 | You can use the `nc` utility to both send and receive low level HTTP requests to understand what is going on. 18 | 19 | ## Standards 20 | 21 | HTTP is specified by IETF. 22 | 23 | There are currently two main versions HTTP/1.0 and HTTP/1.1 which is the most popular one today. 24 | 25 | The HTTP/1.1 specification can be found at RFC 2616 , (1999). 26 | 27 | Many modifications have since been made through other RFCs: 28 | 29 | - extensions, like 422. 30 | - PATCH method 31 | - `multipart/form-data` `content-type` 32 | 33 | ## Example 34 | 35 | TODO simple concrete example to tie in all concepts for a beginner. 36 | 37 | A sample GET request that a browser can send to a server looks like: 38 | 39 | TODO 40 | 41 | A sample POST request that a browser can send to a server looks like: 42 | 43 | TODO 44 | 45 | ## Minimal HTTP request 46 | 47 | 48 | 49 | GET / HTTP/1.1 50 | Host: example.com 51 | 52 | Or more precisely: 53 | 54 | printf 'GET / HTTP/1.1\r\nHost: example.com\r\n\r\n' | nc example.com 80 55 | 56 | The `Host` is required, although some permissive servers may ignore it. `example.com` does not accept it. 57 | 58 | ## Minimal HTTP response 59 | 60 | 61 | 62 | No headers are required in all situations, although some headers are required in some situations. 63 | 64 | ## Transport 65 | 66 | By far the most common is TCP, not UDP: 67 | 68 | Quoting the spec: 69 | 70 | > This does not preclude HTTP from being implemented on top of any other protocol on the Internet, or on other networks. HTTP only presumes a reliable transport 71 | 72 | Since UDP is not "reliable transport", it should not be used. 73 | 74 | HTTP is a synchronous request-response protocol . A simplified transaction goes like this: 75 | 76 | - client opens the connection with 3-way handshake 77 | - client sends the request 78 | - server ACKs the request 79 | - server send the reply 80 | - client ACKs the reply 81 | - client makes further requests (images, CSS, etc.) until satisfied, and server replies 82 | - client starts the closing handshake 83 | 84 | To observe a transaction in detail, use Wireshark: 85 | 86 | - set a Wireshark display filter by address `ip.host == example.com` 87 | - `curl example.com` 88 | 89 | It says that the connection went like this: 90 | 91 | 192.168.0.10 example.com TCP 47996 > http [SYN] Seq=0 92 | example.com 192.168.0.10 TCP http > 47996 [SYN, ACK] Seq=0 Ack=1 93 | 192.168.0.10 example.com TCP 47996 > http [ACK] Seq=1 Ack=1 94 | 192.168.0.10 example.com HTTP GET / HTTP/1.1 95 | example.com 192.168.0.10 TCP http > 47996 [ACK] Seq=1 Ack=76 96 | example.com 192.168.0.10 HTTP HTTP/1.1 200 OK (text/html) 97 | 192.168.0.10 example.com TCP 47996 > http [ACK] Seq=76 Ack=1592 98 | 192.168.0.10 example.com TCP 47996 > http [FIN, ACK] Seq=76 Ack=1592 99 | example.com 192.168.0.10 TCP http > 47996 [FIN, ACK] Seq=1592 Ack=77 100 | 192.168.0.10 example.com TCP 47996 > http [ACK] Seq=77 Ack=1593 101 | 102 | ### Connection header 103 | 104 | #### Keep-Alive 105 | 106 | #### Persistent connection 107 | 108 | In HTTP 1.1, multiple HTTP requests are done by default on a single connection, since almost all pages require the loading of multiple resources from a single server: image, CSS, etc. In `HTTP 1.0`, this had to be indicated through the `Connection: Keep-Alive` header, which most browsers still send as of 2014. 109 | 110 | Demo: 111 | 112 | Two replies, not mandatory on 1.1: 113 | 114 | ( 115 | printf 'GET / HTTP/1.1\r\nHost: example.com\r\n\r\n'; 116 | printf 'GET / HTTP/1.1\r\nHost: example.com\r\n\r\n'; 117 | ) | nc example.com 80 | grep HTTP 118 | 119 | Output: 120 | 121 | HTTP/1.1 200 OK 122 | HTTP/1.1 200 OK 123 | 124 | And Wireshark tells me the connection went like this: 125 | 126 | 192.168.0.10 example.com TCP 47741 > http [SYN] Seq=0 127 | example.com 192.168.0.10 TCP http > 47741 [SYN, ACK] Seq=0 Ack=1 128 | 192.168.0.10 example.com TCP 47741 > http [ACK] Seq=1 Ack=1 129 | 192.168.0.10 example.com HTTP GET / HTTP/1.1 GET / HTTP/1.1 130 | 192.168.0.10 example.com TCP 47741 > http [FIN, ACK] Seq=75 Ack=1 131 | example.com 192.168.0.10 TCP http > 47741 [ACK] Seq=1 Ack=75 132 | example.com 192.168.0.10 HTTP HTTP/1.1 200 OK (text/html) 133 | 192.168.0.10 example.com TCP 47741 > http [ACK] Seq=76 Ack=1592 134 | example.com 192.168.0.10 HTTP HTTP/1.1 200 OK (text/html) 135 | 192.168.0.10 example.com TCP 47741 > http [ACK] Seq=76 Ack=3183 136 | example.com 192.168.0.10 TCP http > 47741 [FIN, ACK] Seq=3183 Ack=76 137 | 192.168.0.10 example.com TCP 47741 > http [ACK] Seq=76 Ack=3184 138 | 139 | so single `SYN` and thus single connection. 140 | 141 | On HTTP 1.0, `Keep-Alive` is mandatory: 142 | 143 | ( 144 | printf 'GET / HTTP/1.0\r\nHost: example.com\r\n\r\n'; 145 | printf 'GET / HTTP/1.0\r\nHost: example.com\r\n\r\n'; 146 | ) | nc example.com 80 | grep HTTP 147 | 148 | Output: 149 | 150 | HTTP/1.1 200 OK 151 | 152 | Two replies: 153 | 154 | ( 155 | printf 'GET / HTTP/1.0\r\nHost: example.com\r\nConnection: keep-alive\r\n\r\n'; 156 | printf 'GET / HTTP/1.0\r\nHost: example.com\r\nConnection: keep-alive\r\n\r\n'; 157 | ) | nc example.com 80 | grep HTTP 158 | 159 | Output: 160 | 161 | HTTP/1.1 200 OK 162 | HTTP/1.1 200 OK 163 | 164 | #### Close 165 | 166 | In HTTP, `Connection: Close` on the request or response 167 | says that the sender does not want a persistent connection. 168 | 169 | This is may be necessary because in HTTP 1.1 170 | the connection is persistent by default. 171 | 172 | ## Newlines 173 | 174 | Every newline is a CRLF. Tools such as `curl` convert `\n` to `\r\n`. 175 | 176 | ## First line 177 | 178 | The first line is different for requests and responses. 179 | 180 | ### Initial request line 181 | 182 | An initial request line looks something like: 183 | 184 | GET /path/to/file.html HTTP/1.0 185 | 1 2 3 186 | 187 | Or: 188 | 189 | POST /path/to/resource HTTP/1.1 190 | 1 2 3 191 | 192 | Where: 193 | 194 | 1. method 195 | 2. path 196 | 3. HTTP version 197 | 198 | ### Initial response line 199 | 200 | HTTP/1.0 200 OK 201 | ^ ^ ^ 202 | 1 2 3 203 | 204 | 1. protocol and version 205 | 206 | 2. status code. Programmatic use, so get it right. 207 | 208 | 3. Reason Phrase. Each status code has a default reason phrase, 209 | but any phrase may be used, the value is not fixed by the protocol. 210 | 211 | #### Method 212 | 213 | Determines in general terms what the request is about. 214 | 215 | RFC 2616 specifies the following methods: 216 | 217 | - `GET`: get information from server 218 | 219 | Can contain a body, but it should be ignored by servers: 220 | 221 | - `HEAD`: only get header information from server 222 | 223 | - `POST`: send data to server to create new objects. E.g.: you click on the submit button of an HTML `form` with `method="post"`. 224 | 225 | - `PUT`: update or create entire objects on server. Both PUT and POST can be used to create object, but `PUT` is *idempotent*. 226 | 227 | - `DELETE`: remove objects from server. Idempotent. 228 | 229 | - `TRACE`: TODO 230 | 231 | - `CONNECT`: TODO 232 | 233 | There are also proposed methods in other RFCs: 234 | 235 | - `PATCH` in RFC 5789 236 | 237 | Update object on server. 238 | 239 | Vs `PUT`: only attributes of the objects which are sent are modified. 240 | 241 | In `PUT`, attributes not given are set to default values. 242 | 243 | In `PATCH`, attributes not given are not modified. 244 | 245 | ##### Safe Methods 246 | 247 | Methods that shouldn't change the server state significantly. E.g.: `GET`, `HEAD`. 248 | 249 | In practice however, many `GET` requests do change the server state: a common example being page view count. However this operation is not considered drastic, and it is generally accepted to be done via `GET`. 250 | 251 | Use safe methods whenever possible. 252 | 253 | ##### Idempotent Method 254 | 255 | If a request is sent twice with an idempotent method, it should have the same effect as if it were sent once. 256 | 257 | E.g.: `PUT`, `DELETE` and all safe methods. 258 | 259 | Both `PUT` and `POST` can be used for object creation: the difference often comes down to: does the user control the unique ID that identifies the created object, often the one shown on the URL to the object? Or is it the server that automatically generates the ID? If the user controls it, a double request is idempotent because the second would be an update with the same data as the first, so `PUT` is the best choice. 260 | 261 | ### Status line 262 | 263 | An initial response line looks something like: 264 | 265 | HTTP/1.0 200 OK 266 | 1 2 3 267 | 268 | Or: 269 | 270 | HTTP/1.1 404 Not Found 271 | 1 2 3 272 | 273 | Where: 274 | 275 | 1. HTTP version 276 | 2. status code 277 | 3. status code name. There is only one possible name for every status code. 278 | 279 | #### Status code 280 | 281 | All status codes can be found here: 282 | 283 | Some of the more interesting ones are commented here. 284 | 285 | ##### Redirect 286 | 287 | ##### 3xx 288 | 289 | Very readable spec: 290 | 291 | Redirect to another page for a given reason. 292 | 293 | All of those statuses require the `Location` header that indicates where to redirect to. 294 | 295 | The spec recommends that an HTML redirect page be given in the body in case the user agent does not follow redirects. Most browsers do so by default, and don't ever show Moved pages. 296 | 297 | HTTP/1.1 301 Moved Permanently 298 | Location: http://www.example.org/ 299 | Content-Type: text/html 300 | Content-Length: 174 301 | 302 | 303 | 304 | Moved 305 | 306 | 307 |

Moved

308 |

This page has moved to http://www.example.org/.

309 | 310 | 311 | 312 | By default, the new request should not change the initial method, unless stated otherwise as in 303. 313 | 314 | If the new method would not be a `GET` or `HEAD` request, the UA can only carry it out if it is possible to ask for confirmation from the user, since something like `POST` may have unwanted consequences. 315 | 316 | User agents should detect infinite redirect loops. 317 | 318 | ###### Post Redirect Get pattern 319 | 320 | 321 | 322 | ###### 301 323 | 324 | Permanent redirect: the page has been moved forever. 325 | 326 | Next time, UAs can cache this redirect, and may not even bother sending a request to the first location. 327 | 328 | Only works for `GET` requests. 329 | 330 | TODO how to revert this if you change your mind later? suggests it is impossible. 331 | 332 | ###### 302 333 | 334 | Hell. 335 | 336 | In the spec, exact same as 307. 337 | 338 | In practice, implemented exactly as 303, even on 2014 browsers, and nothing can be done to change that without breaking things. 339 | 340 | Still the most common response promoted by 2014 web frameworks. E.g., Rail's most common redirection method `redirect_to` uses it. TODO why? Compatibility with HTTP 1.0 which does not have 303 and 307, even if 1.1 is very widely implemented? 341 | 342 | ###### 303 343 | 344 | Redirect to Location, and use `GET` whatever the first method was. 345 | 346 | Introduced in HTTP 1.1 do disambiguate 302. Counterpart of 307. 347 | 348 | In practice, this is what all 302 implementations do. 349 | 350 | This is the most correct response after an user submitted a POST form and you want him to see the object he created, although in practice 302 still is more common. 351 | 352 | Perhaps someday we will be able to use XHR and Javascript `pushState` for form submissions, thus saving one extra request. 353 | 354 | ###### 304 355 | 356 | Not modified. 357 | 358 | Application: 359 | 360 | The user pressed the refresh key on an open browser page. 361 | 362 | It would be wasteful to refecth the current page if it was not modified since it last fetched. `304` exists to avoid just that. 363 | 364 | If the client already has an older version of the resource cached, it can send in the request one of the fields `If-Modified-Since` or `If-Match` containing the date at which the resource was obtained. 365 | 366 | The server sees if the resource has been updated since that date, and if not can return a 304. 367 | 368 | ###### 307 369 | 370 | In the spec, exact same as 302. 371 | 372 | Introduced in HTTP 1.1 with 303 because all browsers treat 302 as 303, and nothing can be done to change it now. 373 | 374 | This time we hope browsers will follow the spec. 375 | 376 | ##### 4xx 377 | 378 | ###### 400 379 | 380 | Bad syntax. E.g., malformed JSON. 381 | 382 | In theory, *not* for invalid values, e.g., invalid email address. Use 422 for that if supported. 383 | 384 | ###### 401 385 | 386 | Server should include a `WWW-Authenticate` field specifying what kind of authentication is required. 387 | 388 | Try this with: 389 | 390 | curl -I localhost/location/that/requires/auth 391 | 392 | I get for example: 393 | 394 | WWW-Authenticate: Basic realm="AuthName value" 395 | 396 | so the type is Basic 397 | 398 | `AuthName value` is a any descriptive string set by the server operators. In Apache it is given by the `AuthName` directive. 399 | 400 | ###### 403 401 | 402 | Vs. 401: 403 | 404 | ###### 422 405 | 406 | WebDAV extension: 407 | 408 | Returned for invalid parameters sent, e.g., invalid email address sent. 409 | 410 | Returned by Rails 4 on form validation. 411 | 412 | ## Headers 413 | 414 | Certain headers can only be used on requests, other only on responses, and a few on both. 415 | 416 | Headers are case insensitive. A common style is to capitalize the first letter of each word. 417 | 418 | ### Headers that can be used on both request and response 419 | 420 | #### Content-Type 421 | 422 | 423 | 424 | The MIME type of the data being sent on the body. 425 | 426 | Not mandatory, but almost always provided since it is very helpful to the client. 427 | 428 | If not present, the client can read the data and try to decide the content type. 429 | 430 | Large list with simple explanations: 431 | 432 | - `text/html`: HTML document. Browsers interpret body as HTML and renders it. 433 | 434 | - `application/xhtml+xml`: XHTML. 435 | 436 | - `text/plain`: browser pastes to screen, no HTML rendering. So you will see tags like `

` on screen. 437 | 438 | - `application/x-www-form-urlencoded`: key value pairs in the same format that can be given on an URL. 439 | 440 | The default `content-type` for `method="post"` on HTML forms. 441 | 442 | The `content-type` can be modified via the `enctype` `form` attribute. 443 | 444 | - `application/json`: popular choice for rest APIs. Used on the GitHub API. 445 | 446 | - `text/css` 447 | 448 | - `application/pdf` 449 | 450 | - `application/javascript` 451 | 452 | - `application/octet-stream`: TODO 453 | 454 | ##### multipart/form-data 455 | 456 | Specified at RFC 2388 . 457 | 458 | Encapsulates multiple header / body pairs into one body HTTP body. 459 | 460 | Advantage of `multipart/form-data`: 461 | 462 | - Huge memory savings for binary files, in which: 463 | 464 | - many bytes would have to be URL encoded for `application/x-www-form-urlencoded` (3 bytes per encoded byte). 465 | 466 | - the size would be 33% larger with `application/x-www-form-urlencoded` + `base64` encode. 467 | 468 | Disadvantages of `multipart/form-data`: 469 | 470 | - Each field has a data overhead for the boundary and the sub headers. 471 | 472 | This is easily overcome by memory gains of using it if there is a file. 473 | 474 | Therefore: use it iff upload of a binary file is possible on the request. 475 | 476 | Sample request: 477 | 478 | ##### boundary 479 | 480 | This is the only type of request does not have an empty line after the headers: `boundary` comes directly. 481 | 482 | `boundary` specifies a sequence of bytes which separates each of the bodies. 483 | 484 | The boundary is always surrounded by CRLFs which are not part of the data. 485 | 486 | The boundary cannot appear inside the data: the user agent must chose it appropriately. 487 | 488 | The trailing hyphens of the boundary are often added for partial backward compatibility with older multipart RFCs, and to improve readability. TODO are they mandatory? 489 | 490 | #### Content-Length 491 | 492 | 493 | 494 | Length of the body in bytes. 495 | 496 | Very important when body is present because it allows the receiver to allocate memory at once. 497 | 498 | Was mandatory on HTTP 1.0, but not on HTTP 1.1, where `Transfer-Encoding: chunked` was added, and it is specified that clients can simply read up to the connection close if it is not given. 499 | 500 | #### Content-Encoding 501 | 502 | Must be one of the request `Accept-Encoding` values. 503 | 504 | - `gzip`: the content was gzipped before sending it to the browser. 505 | 506 | Supported by almost all modern browsers: Firefox sends it by default, Apache has `mod_gzip` which gzips everything when possible. 507 | 508 | #### Transfer-Encoding 509 | 510 | 511 | 512 | Vs `Content-Encoding`: `Transfer-Encoding` is a property of the way the message is transmitted, 513 | `Content-Encoding` is metadata about the message. 514 | 515 | For example, you could use both a transfer encoding and a content encoding on a single request. 516 | 517 | ##### Chunked 518 | 519 | `Transfer-Encoding: chunked` encodes the body as follows: 520 | 521 | printf 'HTTP/1.1 200 OK 522 | Transfer-Encoding: chunked 523 | 524 | 4 525 | some 526 | 8 527 | encoded 528 | 15 529 | text with 530 | a newline 531 | 0 532 | 533 | ' | sed -E 's/$/\r/' | nc -l 8000 534 | 535 | Then: 536 | 537 | curl localhost:8000 538 | 539 | Outputs: 540 | 541 | some encoded text with 542 | a newline 543 | 544 | Each line of the body gets prefixed by the length of the line in hexadecimal on it's own line. 545 | 546 | The last line must have length `0`, followed by two CRLFs. 547 | 548 | Advantage: allows the server to respond progressively if it does not know the content length already for the `Content-Length` header. 549 | 550 | ### Request headers 551 | 552 | #### Host 553 | 554 | The only mandatory header in HTTP 1.1. 555 | 556 | With `curl http://localhost:8000`: 557 | 558 | Host: localhost:8000 559 | 560 | The port should be included if it's not the default `80`: 561 | 562 | With `curl 127.0.0.1`: 563 | 564 | Host: 127.0.0.1 565 | 566 | With `curl http://google.com`: 567 | 568 | Host: google.com 569 | 570 | Some hosts are very picky about the `Host` header. E.g., `example.com` give a 404 if you don't set it, or set it wrong: 571 | 572 | curl `dig +short example.com` 573 | 574 | results in 404, while: 575 | 576 | curl example.com 577 | 578 | works. 579 | 580 | The purpose of this annoying mandatory header is to allow for name based virtual hosting: , i.e., multiple hostnames on a single IP. 581 | 582 | TODO: how are browsers able to decide this? If you click on a link, browsers send the same `Host` as the hostname. Is it possible to change that? 583 | 584 | #### User-Agent 585 | 586 | Description of the user agent that sent the request. 587 | 588 | Request counterpart of `Server`. 589 | 590 | Firefox 29: 591 | 592 | User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:29.0) Gecko/20100101 Firefox/29.0 593 | 594 | Curl 7.22: 595 | 596 | User-Agent: curl/7.22.0 (i686-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3 597 | 598 | #### Accept 599 | 600 | A comma separated list of MIME type that the client wants as a response. 601 | 602 | Specified on the response by the `Content-Type` header. 603 | 604 | It is possible that a single URL is able to return several types. 605 | 606 | In this case this field can be used by the server to determine the type to serve. 607 | 608 | Firefox 29: 609 | 610 | Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 611 | 612 | The `q=` are the quality factors, that indicate the level of preference for each listed media type: 613 | 614 | #### Accept-Language 615 | 616 | Comma separated list of accepted `Content-Language` for the response. 617 | 618 | Firefox 29: 619 | 620 | Accept-Language: en-US,en;q=0.5 621 | 622 | #### Accept-Encoding 623 | 624 | Comma separated list of accepted `Content-Encoding` for the response. 625 | 626 | Firefox 29: 627 | 628 | Accept-Encoding: gzip, deflate 629 | 630 | #### Extensions 631 | 632 | ##### Forwarded-For 633 | 634 | 635 | 636 | Usually set by reverse proxies. 637 | 638 | Holds the IP of the original request. 639 | 640 | ### Response headers 641 | 642 | #### Content-Disposition 643 | 644 | Suggests to the browser what to do to certain types of data 645 | 646 | `attachment` suggests to the browser to download the file with given filename: 647 | 648 | Content-Disposition: attachment; filename=genome.jpeg; 649 | 650 | `inline` suggest to the browser to show the content inline if it has that capability: 651 | 652 | Content-Disposition: inline 653 | 654 | In Firefox, the browser preferences under `Edit > Preferences > Application` determine what to do for each MIME type, and overrides this header. 655 | 656 | suggests that `Content-Disposition` is not sufficient protection from [unrestricted file upload](https://www.owasp.org/index.php/Unrestricted_File_Upload) XSS. 657 | 658 | #### Server 659 | 660 | Software that sent the response. 661 | 662 | - `google.com`: `gws`. Google Web Server, closed source. 663 | - `twitter.com`: `tfe` 664 | 665 | Not mandatory: Facebook does not return it. 666 | 667 | #### Referer 668 | 669 | Sent by the UA containing the page from which the request was sent, so in the case of hyperlinks the page who refered the user to the new one. 670 | 671 | Optional, so don't rely on it: 672 | 673 | Was originally misspelled as `referrer`, with two R's on older specs, but this has been corrected in newer specs. 674 | 675 | Used by Rails `redirect_to :back` shortcut. 676 | 677 | #### Extensions 678 | 679 | ##### Sendfile 680 | 681 | ##### Accel 682 | 683 | 684 | 685 | 686 | 687 | If the response contains this header, the rest of the response is ignored and the file is served instead, and the server uses all of it's optimizations. 688 | 689 | Normally this the reverse proxy is configured to consider this header only on the response of the server. 690 | 691 | This header is useful for serving static files for which the server has to control access: there fore the request must pass through the server first. But it is inefficient to serve files from the backend server, so the backend server passes this header and delegates to the reverse proxy. 692 | 693 | ### CORS headers 694 | 695 | The following headers are used for CORS requests and responses: 696 | 697 | #### Origin 698 | 699 | #### Access-Control-Allow-Origin: http://api.bob.com 700 | 701 | #### Access-Control-Allow-Credentials: true 702 | 703 | #### Access-Control-Expose-Headers: FooBar 704 | 705 | ### Custom headers 706 | 707 | 708 | 709 | In the past, prefix by `X-`. 710 | 711 | After 2012: cross your fingers and pick a name. 712 | 713 | ## Body 714 | 715 | TODO 716 | 717 | ## HTTPS 718 | 719 | HTTP over TLS. 720 | 721 | Encrypts both body and headers. 722 | 723 | Downside: encrypt/decrypt costs time. 724 | 725 | HTTPS runs on port 443 instead of 80. 726 | 727 | ## HTTP authentication 728 | 729 | Authentication that is sent over the HTTP header. 730 | 731 | ### Sources 732 | 733 | 734 | 735 | Comparison to form auth, nice diagrams: 736 | 737 | Great post: 738 | 739 | ### Downsides of HTTP auth 740 | 741 | Parameters are left to the browser: 742 | 743 | - the appearance of the login page 744 | - the time for which the user stays authenticated (time for which browser keeps resending `user:pass` automatically). 745 | 746 | You might have seen this on a website in which your browser just opens up a weird looking window and asks you for username / password. 747 | 748 | Therefore, you cannot customize them and users will get different interfaces on different browsers, bad user interface consistency. 749 | 750 | For those reasons, form authentication is used on most large sites today. 751 | 752 | ### Upside of HTTP auth 753 | 754 | Simple. 755 | 756 | ### Basic authentication 757 | 758 | Authentication is sent on the header *unencrypted*! 759 | 760 | Example: 761 | 762 | curl -vu u:p google.com 763 | 764 | You see the header line: 765 | 766 | Authorization: Basic dTpw 767 | ^^^^^ ^^^^ 768 | 1 2 769 | 770 | where: 771 | 772 | - 1: authentication type 773 | - 2: base 64 of u:p. not encryption! 774 | 775 | Just checking: 776 | 777 | assert [ "`echo dTpw | base64 -d`" = "u:p" ] 778 | 779 | #### URL convention 780 | 781 | Many programs accept URLs strings with user/pass included: 782 | 783 | curl -v u:p@google.com 784 | 785 | This is however just a convention, since programs that accept it parse the string to extract the `u:p` part, and then send it on the header. 786 | 787 | ### Digest authentication 788 | 789 | Pretty cool concept 790 | 791 | See: 792 | 793 | Authentication is sent on the header md5 hashed: 794 | 795 | curl --digest -vu u:p google.com 796 | 797 | #### Why it works 798 | 799 | Data is appended to the authentication with `:` before hashing: 800 | 801 | - domain (`www.google.com`) 802 | - method (GET, POST, etc.) 803 | - nonce 804 | - nonce is sent to client from server. 805 | - *nonces can only be used once per client*!! 806 | - nonce prevents requests from being repeated with an old captured hashed string! 807 | - also increases the difficulty of cracking each user/pass 808 | 809 | This way, the unknown user and pass get mixed up with the extra data in the hash and it is very hard to separate them. and the nonce makes sure requests cannot be remade by resending the hash. 810 | 811 | Merits: 812 | 813 | - simpler than a full SSL 814 | 815 | ### NTML 816 | 817 | Safer than digest: replay attacks impossible. 818 | 819 | Requires server state, so HTTP 1.1 only. 820 | 821 | Little current support/usage. 822 | 823 | ### CSRF 824 | 825 | ### Cross site request forgery 826 | 827 | The attacker Bob posts on a website: 828 | 829 | http://bank.com?transfer-ammount=10000&to=bob 830 | 831 | Alice clicks on the disguised link, her browser sends the authentication cookies, authenticates and makes the request. 832 | 833 | How to prevent it: 834 | 835 | #### Synchronizer Token Pattern. 836 | 837 | The most popular prevention mechanism. 838 | 839 | Send extra randomly generated data with valid forms. 840 | 841 | Generation can be per session or per form, which is more secure but less efficient and may have usability impact. 842 | 843 | The method relies on the same origin policy: if it did not exist the attacker would be able to obtain the token. 844 | 845 | This method is implemented by default in most web frameworks, either by adding a hidden form field (Django) or `meta` elements like Rails: 846 | 847 | 848 | 849 | 850 | #### Reauthenticate 851 | 852 | Ask user for password again before critical operations. 853 | 854 | Used in all online banking systems. 855 | 856 | Inconvenient for users since an extra action is required before an authentic request. 857 | 858 | ## Sources 859 | 860 | Good intro tutorial: 861 | -------------------------------------------------------------------------------- /ifconfig.md: -------------------------------------------------------------------------------- 1 | # ifconfig 2 | 3 | `Network InterFace configuration tool`. 4 | 5 | Get and set key interface information like IP, subnet masks, MAC, etc. 6 | 7 | Good source: 8 | 9 | ifconfig 10 | 11 | Sample interfaces on a modern laptop: 12 | 13 | - `eth0`: wired network 0 14 | - `wlan0`: Wifi card 0 15 | - `lo`: loopback (local host) 16 | 17 | Get local IPs (behind router): 18 | 19 | ifconfig | grep -B1 "inet addr" | awk '{ if ( $1 == "inet" ) { print $2 } else if ( $2 == "Link" ) { printf "%s:" ,$1 } }' | awk -F: '{ print $1 ": " $3 }' 20 | 21 | `wlan0` and `eth0` are two different interfaces! 22 | 23 | ## iwconfig 24 | 25 | Wireless network configuration 26 | 27 | TODO: subset of `ifconfig`? 28 | 29 | ## See also 30 | 31 | NetworkManager is a higher level network configuration tool used in Ubuntu 12.04 by default. You should use that instead if installed. 32 | -------------------------------------------------------------------------------- /ip.md: -------------------------------------------------------------------------------- 1 | # IP 2 | 3 | Protocol that allows to: 4 | 5 | - assign addresses to network interfaces. 6 | - find path between one computer to another, possibly passing through many routers 7 | - fragment and reassemble messages that are too large to be reliably transmitted through on a channel 8 | 9 | Main spec: , from 1981! 10 | 11 | ## Versions 12 | 13 | ## IPv6 vs IPv4 14 | 15 | As of 2014, the dominant version is IPv4, but IPv6 is already specified and has many implementations. 16 | 17 | One of the major limitations of IPv4 is the small number of addresses, which was underestimated at the creation of IPv4, making such addresses a valuable resource. 18 | 19 | IPv4 addresses have 4 bytes, while IPv6 have 16, making the number of addresses for IPv6 stratospherically large at around 10^38. Compare it to Avogadro's constant at 10^23, so it is unlikely that those addresses will become a valuable resource in the foreseeable future. 20 | 21 | Unless where noted otherwise, the term IP will refer in this project to either IPv4 or IPv4 and IPv6 unless explicitly written IPv6. 22 | 23 | ## IPv5 24 | 25 | Already taken by a revision of some old protocol: 26 | 27 | ## IP header 28 | 29 | Learn what the IP header contains: 30 | 31 | 32 | 33 | Fields by increasing interest / ease to understand ratio: 34 | 35 | - **Version (4 bits)** 36 | 37 | Indicates the protocol version. 38 | 39 | Value 4 for IPv4. 40 | 41 | - **Internet header length (IHL) (4 bits)** 42 | 43 | Length of the IP header only (no data) in 4 byte units. 44 | 45 | - **Total length (2 bytes)** 46 | 47 | Total length of header + body in bytes 48 | 49 | This must be transmitted as the length is variable. 50 | 51 | - **IPs of destination and origin (4 bytes each)** 52 | 53 | - **Time to live (TTL) (1 byte)** 54 | 55 | Decreased whenever the packet passes through a router. 56 | 57 | If 0, router does not forward the package, because it has already traveled for too long, and signals this to the sender via an ICMP Time exceeded with Code = 0. 58 | 59 | This prevents lost packages from doing infinite turns on the network. 60 | 61 | - **Header checksum (2 bytes)** 62 | 63 | - **Protocol (1 byte)** 64 | 65 | Number that identifies the protocol contained in the IP data, for example TCP or UDP. 66 | 67 | In this way, the receiver knows how to interpret the data inside of the IP package. 68 | 69 | The numbers are assigned by IANA and can be found [here](http://en.wikipedia.org/wiki/List_of_IP_protocol_numbers). 70 | 71 | - **Flags 3 bits** 72 | 73 | Used for IP fragmentation. 74 | 75 | - **Fragment offset (13 bits)** 76 | 77 | Used for IP fragmentation. 78 | 79 | - **Identification (2 bytes)** 80 | 81 | Used for IP fragmentation. 82 | 83 | ## IP address 84 | 85 | An unique address that identifies a host, for example a separate workstation. 86 | 87 | IPv4 addresses are 4 bytes long. 88 | 89 | Each computer knows its IP address. 90 | 91 | This can be set in two ways: 92 | 93 | - statically and manually: admin enters those values for each computer. 94 | 95 | They never change. 96 | 97 | - dynamically and automatically specially via DHCP. 98 | 99 | Best method on a local network to avoid IP clashes. 100 | 101 | ### Address representation 102 | 103 | ### Notation 104 | 105 | The most common notation by far is the dotted decimal: 106 | 107 | 192.156.0.1 108 | 109 | With the advent of CIDR, a new notation was introduced that also specifies the subnet mask in one go: 110 | 111 | 192.156.0.1/24 112 | 113 | ### Classes 114 | 115 | Old concept, but people still use it for certain mask sizes + internal reserved addresses. 116 | 117 | Each address has two parts: network part and host part. 118 | 119 | How many bytes are the network, and how many bytes are the host was determined only by the class of the IP before CIDR, and only the following possibilities existed: 120 | 121 | - `A`: starts with 0 122 | - `B`: starts with 10 123 | - `C`: starts with 110 124 | - `D`: starts with 1110 125 | - `E`: all others 126 | 127 | IPs on classes `A`, `B` and `C` were reserved for LAN usage;. 128 | 129 | If your address is in those ranges, the routers proxy server knows it is an internal one you are asking about. 130 | 131 | The most common home range, specially in home networks, is the Class C: 132 | 133 | 192.168.0.1 through 192.168.255.254 134 | subnet mask 255.255.255.0 135 | 136 | CIDR added the possibility to use any network / host split. 137 | 138 | #### Example: network and host parts 139 | 140 | - 2 networks 141 | - 3 bytes for the network part 142 | - 1 router 143 | - 3 computers on the same network 144 | 145 | Things could look like: 146 | 147 | +---------------+ +---------------+ +---------------+ 148 | | Computer 1 | | Computer 2 | | Computer 3 | 149 | |---------------| |---------------| |---------------| 150 | | 192.156.0.2 | | 192.156.0.3 | | 192.156.0.4 | 151 | +---------------+ +---------------+ +---------------+ 152 | | | | 153 | | | | 154 | +-------------------+-------------------+ 155 | | 156 | | 157 | +---------------+ 158 | | 192.156.0.1 | 159 | |---------------+ 160 | | Router | 161 | |---------------| 162 | | 192.157.0.1 | 163 | +---------------+ 164 | | 165 | | 166 | +-------------------+-------------------+ 167 | | | | 168 | | | | 169 | +---------------+ +---------------+ +---------------+ 170 | | Computer 3 | | Computer 4 | | Computer 5 | 171 | |---------------| |---------------| |---------------| 172 | | 192.157.0.2 | | 192.157.0.3 | | 192.157.0.4 | 173 | +---------------+ +---------------+ +---------------+ 174 | 175 | Note how the router also has one ore more local IPs. 176 | 177 | In this example, there are 2 local networks: for one to communicate with the other, they must first pass through the router. 178 | 179 | When the router sees an address that is not in the current network, e.g. Computer 3 wants to send to Computer 1, it is able to determine to which network to send it. TODO how. 180 | 181 | ### LAN IP 182 | 183 | ### Internal IP 184 | 185 | ### WAN IP 186 | 187 | ### External IP 188 | 189 | RFC 1918: 190 | 191 | If you use a router, your entire network has a single IP seen from the outside (WAN), and an internal IP for each interface seen on the private local network (LAN). 192 | 193 | You external IP may change or not depending on how your ISP operates. Most ISPs to modify home user's IPs from time to time to have more flexibility, but many give you the same external IP for at least several hours, making it possible for you to use it for simple development. 194 | 195 | This only matters if you want to have external computers make requests for you, e.g. to serve a web server to the outside world. Replies to requests you make already know where to be routed back to from information sent on the request. 196 | 197 | If you are going to use your external IP behind a router, you will need to enable port forwarding. 198 | 199 | Get LAN IPs for all interfaces on current computer: 200 | 201 | ifconfig | grep -B1 "inet addr" | awk '{ if ( $1 == "inet" ) { print $2 } else if ( $2 == "Link" ) { printf "%s:" ,$1 } }' | awk -F: '{ print $1 ": " $3 }' 202 | 203 | Each interface has its own IP. Therefore, a single computer can have multiple LAN IPs: one for the wired connection, one for the Wiki, one loopback localhost, etc. 204 | 205 | The server on the router is called **proxy server**. 206 | 207 | Internal IPs may be assigned automatically via the DHPC protocol. 208 | 209 | ### Why internal LAN IPs usually start with 192 210 | 211 | 212 | 213 | #### Get external IP 214 | 215 | curl ipecho.net/plain 216 | curl ifconfig.me 217 | 218 | - 219 | - 220 | 221 | ### Port forwarding 222 | 223 | By default, if a TCP / UDP SYN request is made to most routers, they are simply dropped. 224 | 225 | This means that you cannot serve a web server from behind the router. 226 | 227 | To avoid this, you must set up *port forwarding*, that specifies certain ports and protocol (TCP or UDP) that will be sent to a local LAN IP. 228 | 229 | It seems only possible to do this if the local IP is static TODO confirm. 230 | 231 | ### Classless routing 232 | 233 | ### Subnet mask 234 | 235 | Classless Inter-Domain Routing: the number of network bits is part of the address. 236 | 237 | , 1993. 238 | 239 | ### Prefix length 240 | 241 | Name of the analogous concept in IPv6, but restricted to the far dominating case of 1's prefixed mask. 242 | 243 | In theory, IPv4 subnet masks could be any 4 byte pattern as the name suggests. 244 | 245 | #### Get mask for an interface: 246 | 247 | ifconfig wlan0 | sed -nr 's/.*Mask:([^ ]*)/\1/p' 248 | 249 | Internal IPs have two parts: network and computer. 250 | 251 | The length of the network part may vary between networks. 252 | 253 | The length is given by the **subnet mask**, e.g.: 254 | 255 | 255.255.255.0 256 | 1111.1111.1111.0000 257 | 258 | means that 12 first bits are network. 259 | 260 | 255.255.0.0 261 | 262 | means that 8 first bits are network 263 | 264 | All computers in the same network must have the same subnet mask and the same network part, but different computer parts. 265 | 266 | Each network (formally **network segment**) is run by a single router #TODO confirm 267 | 268 | #### CIDR notation 269 | 270 | It is cumbersome for humans to write the subnet mask with 4 integers, so a simpler notation as introduced. E.g.: 271 | 272 | 255.255.255.0/24 273 | 274 | means: 275 | 276 | - address: `255.255.255.0` 277 | - subnet mask: 24 leading ones 278 | 279 | ### Special purpose addresses 280 | 281 | Specified on 2013, which determines current best practices, links to many other RFCs that define things. 282 | 283 | Some special addresses were specified in separate RFCs than the main IP specs. 284 | 285 | #### Zero host address 286 | 287 | 288 | 289 | If the entire host part is zero, then the address is used to refer to the network itself. 290 | 291 | It is used when several networks, one one a different router must speak to each other. 292 | 293 | #### 0.0.0.0 294 | 295 | #### Default gateway 296 | 297 | `0.0.0.0` host address in the routing table. 298 | 299 | If no network matches request, sends to this network. 300 | 301 | Address you get automatically redirected to by router if the address you gave cannot be found on the local network. 302 | 303 | Find default gateway: 304 | 305 | route -n 306 | 307 | or for programmatic usage: 308 | 309 | route -n | awk '{ if( $1 ~ "0.0.0.0" ) print $2 }' 310 | 311 | If a program is listening to `0.0.0.0`, then it is listening on all interfaces: 312 | 313 | - localhost `127.0.0.1` 314 | - local IP address of WIFI or Ethernet, e.g. `192.168.0.2` (retrieved with `ifconfig`) 315 | 316 | So if you do: 317 | 318 | nc -l 0.0.0.0 8000 319 | 320 | it will catch all of: 321 | 322 | echo a | nc -l 0.0.0.0 8000 323 | echo a | nc -l 127.0.0.1 8000 324 | echo a | nc -l 192.168.0.2 8000 325 | 326 | On `ifconfig`, this value appears on the `Foreign address` column if there are not programs listening to the connection. 327 | 328 | The same goes for writing: 329 | 330 | nc -l 127.0.0.1 8000 & 331 | echo a | nc -l 0.0.0.0 8000 332 | 333 | #### Unspecified IPv6 address 334 | 335 | 336 | 337 | In IPv6, `::` means `0:0:0:0:0:0:0:0`, and is the *unspecified address*. 338 | 339 | For example, it appears on the `ifconfig` `Foreign address` column when there are no programs listening to a server. 340 | 341 | #### 127.0.0.1 342 | 343 | #### Loopback 344 | 345 | 1989 346 | 347 | Always refers to the current address. 348 | 349 | #### localhost 350 | 351 | Common hostname associated to the loopback address. 352 | 353 | ### Broadcast address 354 | 355 | #### 255.255.255.255 356 | 357 | Specified in: 1984. 358 | 359 | The host address is entirely composed of 1s. 360 | 361 | The broadcast address means talking to all computers on a given network at once instead of a single computer. 362 | 363 | TODO: simple example with an utility that does something magic with it? 364 | 365 | ### 169.254.0.0/16 366 | 367 | ### Link-local address 368 | 369 | 370 | 371 | TODO what is it? My interpretation: addresses that belong the current network segment and never get forwarded by the router. 372 | 373 | Used on zeroconf. 374 | 375 | ### Examples 376 | 377 | #### Class C network 378 | 379 | - network part: `192.168.3` 380 | 381 | Broadcast is: `192.168.3.255` 382 | 383 | - network part: `192.168.234` 384 | 385 | Broadcast is: `192.168.3.255` 386 | 387 | #### Class A network 388 | 389 | - network part: `10` 390 | 391 | Broadcast is: `10.255.255.255` 392 | 393 | #### .1 addresses 394 | 395 | The `.1` address is not special, but in home networks is often already taken by the router's inner interface 396 | 397 | This is why your addresses may start at `.2`. 398 | 399 | ## Fragmentation 400 | 401 | If a router will forward this packet to another interface, it may be that the MTU of the other interface be smaller than the previous one, and the package cannot be transmitted hole. 402 | 403 | In this case, IP provides a fragmentation mechanism to split up the package into smaller ones. 404 | 405 | TCP always attempts to avoid IP fragmentation, so that if TCP is being used, it is unlikely that fragmentation will happen. It is therefore easier to observe IP fragmentation for UDP packets. 406 | 407 | Reassembly of fragments is meant to happen only at the final destination. 408 | 409 | The fields used from the IP header are: 410 | 411 | - flags (3 bits) 412 | 413 | | 0 | DF | MF | 414 | 415 | - 0: reserved, always set to 0 416 | 417 | - `DF`: don't fragment. 418 | 419 | If that is the case, the router first checks the `DF` flag. 420 | 421 | If `DF = 1`, routers should not fragment, and notify the source via an ICMP fragmentation not possible message. 422 | 423 | If `DF = 0`, fragmentation can occur, and tranmission continues. 424 | 425 | - `MF`: more fragments 426 | 427 | If 0, this is the last fragment. 428 | 429 | If 1, there are more fragments to come. 430 | 431 | A common technique is to send first the last segment, which allows the destination to know how large a buffer it will need in advance. 432 | 433 | - fragment offset: (13 bits) 434 | 435 | How many 8 byte units of IP data have already been sent. 436 | 437 | The first fragment always has this field equal to 0. 438 | 439 | But if fragmentation occurs, the following fragments will contain how much IP data has been sent already. 440 | 441 | - identification: (2 bytes) 442 | 443 | It is set by the sender with a different number for each IP datagram. 444 | 445 | If fragmentation will occur, each fragment's IP header will get the same identification number so that they can be reassembled afterwards. 446 | 447 | Identification numbers can be the same across multiple source IPs, 448 | but for a single source IPs they are unique. 449 | 450 | ## NIC 451 | 452 | Network Interface Cards 453 | 454 | Hardware that does network communication. 455 | 456 | Come mostly built-in the motherboard today. 457 | 458 | Each router has at least 2 NICs: one external and one internal. 459 | 460 | ### get all interface names 461 | 462 | ifconfig | perl -ne '/^(\S+)/ && print $1 . "\n"' 463 | 464 | ## MAC 465 | 466 | AKA: 467 | 468 | - physical address 469 | - hardware address 470 | - media access control address 471 | - BIA: burnt in address 472 | 473 | Unchangeable address of each NIC. 474 | 475 | Unique across and within vendors. 476 | 477 | 6 bytes: first 3 identify vendor, last 3 product 478 | 479 | Colon separated notation. Ex: `0C:21:B8:47:5F:96`. 480 | 481 | Get MAC addresses of my computer: 482 | 483 | ifconfig 484 | 485 | Or for programmatic usage: 486 | 487 | ifconfig | sed -nr 's/([^ ]*) .*HWaddr (.*)/\1 \2/p' 488 | 489 | Get MAC addresses of computers I have already talked to in the LAN: 490 | 491 | timeout 3 ping 192.168.1.3 492 | arp -a | sed -nr 's/([^ ]*) .*at (.*)/\1 \2/p' 493 | 494 | ## ICMP 495 | 496 | Internet control message protocol. 497 | 498 | Part of the IP protocol. 499 | 500 | Contains several types of information for routers about the IP traffic. 501 | 502 | Lives in the Internet layer, but is encapsulated inside an IP datagram just like TCP and UDP: 503 | 504 | | ICMP data | 505 | | IP data | IP header | 506 | 507 | Its number on the IP protocol field is 1, so you may guess that this is a very important protocol. 508 | 509 | Structure of the ICMP header 510 | 511 | - Type 1B 512 | - Code 1B 513 | - Checksum 2B 514 | - Variable part 4B 515 | 516 | ICMP can also contain an optional extra data section after the variable part. 517 | 518 | Examples of what ICMP can do: 519 | 520 | - ECHO requests. 521 | 522 | The server responds with another echo request with the same data. 523 | 524 | Used to testing the network. 525 | 526 | Used by the ping utility. 527 | 528 | - destination unreachable 529 | 530 | Datagram cannot be transmitted further by a router. 531 | 532 | TODO when does this happens? This is not the `TTL = 0` since that is covered by Time Exceeded 533 | 534 | - source quench 535 | 536 | Router tells source to lower sending speed because the network is too overloaded. 537 | 538 | - redirect 539 | 540 | Router tells source to not make that request again, since there is an obvious better choice of router to make the request to. 541 | 542 | Typically happens on a LAN with 2 routers, if a host sends a request to a bad router. 543 | 544 | - router discovery 545 | 546 | Source asks for any routers on the LAN to identify themselves. 547 | 548 | - time exceeded. 549 | 550 | - if `code = 0`: TTL reached 0. 551 | 552 | Router notifies source of that via ICMP. 553 | 554 | Used by `traceroute`. 555 | 556 | - if `code = 1`: reassembly time exceeded. 557 | 558 | IP headers can be fragmented. 559 | 560 | If the first part reaches, but the second takes too long, the reciever discards the first to make room for other requests, and notifies the source like this. 561 | 562 | - packet too big (BTP) 563 | 564 | Sent by router to source if it receives a packet that is larger than the MTU and the IP header has `DF = 1`. 565 | 566 | ## IGMP 567 | 568 | Internet Group Message Protocol. 569 | 570 | Similar to ICMP in some senses: 571 | 572 | - wrapped inside IP 573 | - IP control purposes 574 | 575 | However IGMP is used to control multicasts only. 576 | 577 | IP protocol number: 2. 578 | 579 | ## IPv6 580 | 581 | 2006. 582 | 583 | ### Try it out 584 | 585 | Terminal 1: 586 | 587 | nc -l6 ::1 8000 588 | 589 | Terminal 2: 590 | 591 | echo a | nc -6 ::1 8000 592 | 593 | Congratulations, your first explicit IPv6 connection! 594 | 595 | ### Usage 596 | 597 | IPv6 is less popular than IPv4 as of 2014, but it's popularity is increasing, and it is likely that it will dominate in a few years. 598 | 599 | A few metrics: 600 | 601 | - percentage of Google requests on IPv6: 602 | 603 | ### IPv6 address representation 604 | 605 | Rules documented at: 606 | 607 | Full addresses are: 608 | 609 | - written in hex. The characters `a-f` must be lowercase. 610 | - grouped every 2 bytes by `:` 611 | - leading 0's *MUST* be removed. `0000` becomes `0`. 612 | - the longest sequence of `:0:0:0:` *MUST* be converted to `::`. Can only be done once, or would lead to ambiguity. 613 | - a single `:0:` shall not be converted to `::`, only multiple 0's like `:0:0:`. 614 | 615 | The rules that make addresses shorter lead to the *cannonical* representation of the address: they are not just optional. 616 | 617 | Good: 618 | 619 | 2001:db8::ff00:42:8329 620 | 621 | which means in a fuller notation: 622 | 623 | 2001:0db8:0000:0000:0000:ff00:42:8329 624 | 625 | Bad, uppercase: 626 | 627 | 2001:DB8:0:0:0:FF00:42:8329 628 | 629 | Bad, leading 0's: 630 | 631 | 2001:0db8::ff00:42:8329 632 | 633 | Bad, 0's not compressed: 634 | 635 | 2001:db8:0:0:0:ff00:42:8329 636 | 637 | #### Percent signs on IPv6 638 | 639 | Fancy stuff: 640 | 641 | ## Router 642 | 643 | Routers send packages from one place to another. 644 | 645 | Most routers will have at least two network interfaces, one on each network. 646 | 647 | ### Configure you router 648 | 649 | If you want to play around with routers, you should get your hands dirty and do some router configuring. 650 | 651 | Routers are generally configured through a browser. 652 | 653 | First you must make a wired connection to the router. 654 | 655 | You must enter the IP address of your router. This is fixed and supplied by the router manufacturer. A common address is the first address of the range: `192.168.0.1` for class C. 656 | 657 | You must then enter a username and a password. A default will be supplied by the manufacturer, such as `admin` `admin`, or `admin` `password`. This can be changed once you logged in. 658 | 659 | ### Routing table 660 | 661 | Great LAN routing example: 662 | 663 | Routing tables say: if the request should go to a given network, send it to a given interface. 664 | 665 | `0.0.0.0` is the default if no other is found. 666 | 667 | Routers have two interfaces each: inside and outside. 668 | 669 | ## Kernel routing table 670 | 671 | TODO: what is it? 672 | 673 | ## Famous IPs 674 | 675 | - 8.8.8.8 and 8.8.8.4: 676 | -------------------------------------------------------------------------------- /iptables.md: -------------------------------------------------------------------------------- 1 | # iptables 2 | 3 | Configure what the Kernel should do to networking communication. 4 | 5 | It is a firewall. 6 | 7 | Implemented as a kernel module: 8 | 9 | lsmod | grep ip_tables 10 | 11 | Outputs: 12 | 13 | ip_tables 27239 2 iptable_filter,iptable_nat 14 | x_tables 34059 4 ip_tables,xt_tcpudp,iptable_filter,xt_REDIRECT 15 | 16 | ## L 17 | 18 | List. 19 | 20 | View state: 21 | 22 | sudo iptables -L 23 | 24 | Sample output before we do anything: 25 | 26 | Chain INPUT (policy ACCEPT) 27 | target prot opt source destination 28 | 29 | Chain FORWARD (policy ACCEPT) 30 | target prot opt source destination 31 | 32 | Chain OUTPUT (policy ACCEPT) 33 | target prot opt source destination 34 | 35 | The only information we have is that `INPUT`, `FORWARD` and `OUTPUT` are accepting connections. 36 | 37 | Even more information can be obtained with `-v`. 38 | 39 | ## F 40 | 41 | Clear all current rules: 42 | 43 | sudo iptables -F 44 | 45 | Great after we're done with messing things up. 46 | 47 | Does not affect policies set by `-P`. 48 | 49 | ## t 50 | 51 | `iptables` can deal with multiple types of table. 52 | 53 | Each table has their possible policies and targets. 54 | 55 | ## iptables-save 56 | 57 | Save current configuration. Otherwise, it gets destroyed when `iptables` restarts. 58 | 59 | ## Policies 60 | 61 | ### INPUT 62 | 63 | Incoming requests. 64 | 65 | sudo iptables -P INPUT DROP 66 | 67 | Now: 68 | 69 | sudo iptables -L 70 | 71 | says: 72 | 73 | Chain INPUT (policy DROP) 74 | 75 | and: 76 | 77 | curl --connect-timeout 3 127.0.0.1 78 | 79 | times out, because the request was sent, but the response got dropped. 80 | 81 | Restore: 82 | 83 | sudo iptables -P INPUT ACCEPT 84 | 85 | ### FORWARD 86 | 87 | Packets that the system would forward to another system. 88 | 89 | Happens if your system is acting as a router. 90 | 91 | ### OUTPUT 92 | 93 | Outgoing requests. 94 | 95 | #### Drop all packages 96 | 97 | Drop any packages before they are output: 98 | 99 | sudo iptables -P OUTPUT DROP 100 | 101 | You lose all Internet connection. 102 | 103 | curl example.com 104 | 105 | gives: 106 | 107 | curl: (6) Could not resolve host: example.com 108 | 109 | since it cannot find the DNS, and: 110 | 111 | curl --connect-timeout 3 127.0.0.1 112 | 113 | times out: 114 | 115 | curl: (28) Connection timed out after 3000 millisecond 116 | 117 | as it tried to send the request, but no response came because it was not actually sent. 118 | 119 | Restore it: 120 | 121 | sudo iptables -P OUTPUT ACCEPT 122 | 123 | #### Redirect one port another on localhost 124 | 125 | sudo iptables -t nat -I OUTPUT -p tcp --dport 8000 -j REDIRECT --to-ports 4000 126 | echo 'a' | nc -l localhost 4000 127 | 128 | Then: 129 | 130 | echo 'b' | nc -l localhost 8000 131 | 132 | Does not work for packets going out to external servers: use `PREROUTING` instead. 133 | 134 | ## Target 135 | 136 | Targets are the actions taken on certain conditions. 137 | 138 | Most targets are considered extensions and documented under: 139 | 140 | man iptables-extensiosn 141 | 142 | but they come by default. 143 | 144 | Targets are attempted sequentially: the order in which you add `-A` or insert `-I` them matters. 145 | 146 | For example: 147 | 148 | sudo iptables -A OUTPUT -p tcp --dport 53 -j ACCEPT 149 | sudo iptables -A OUTPUT -p tcp --dport 80 -j ACCEPT 150 | sudo iptables -P OUTPUT DROP 151 | 152 | for which `sudo iptables -L` reads: 153 | 154 | Chain OUTPUT (policy DROP) 155 | target prot opt source destination 156 | ACCEPT tcp -- anywhere anywhere tcp dpt:domain 157 | ACCEPT tcp -- anywhere anywhere tcp dpt:http 158 | 159 | Allow output to port `80`, but deny any other: 160 | 161 | curl example.com 162 | curl github.com:443 163 | 164 | TODO not allowing any traffic? Why? 165 | 166 | ### P 167 | 168 | Default policy if no rule is matched. 169 | 170 | ## A 171 | 172 | Append a new rule to the end of the list for a given policy: 173 | 174 | sudo iptables -A INPUT -s example.com -j DROP 175 | 176 | ## D 177 | 178 | Delete a given rule. 179 | 180 | sudo iptables -D INPUT -s example.com -j DROP 181 | 182 | Everything must match, except that `-A` is replaced by `-D`. 183 | 184 | ## I 185 | 186 | Prepend a new rule. 187 | 188 | ### ACCEPT 189 | 190 | Obvious. 191 | 192 | ### DROP 193 | 194 | Don't simply don't send or receive. 195 | 196 | ### RETURN 197 | 198 | TODO. 199 | 200 | ### REJECT 201 | 202 | Extension. 203 | 204 | Send some notification of the error, unlike `DROP` which is silent. 205 | 206 | ### REDIRECT 207 | 208 | Only for `-t nat`. 209 | 210 | #### Redirect outgoing requests to a localhost server 211 | 212 | This is often used when you have to proxy requests to a different server. With this method, applications are unable to know that they are not talking to the desired server, and you don't have to configure anything. Great for man in the middle attacks, e.g. through mitmproxy :) 213 | 214 | Redirect all requests with port `80` (HTTPS) to a localhost server listening on port `8000`: TODO get working 215 | 216 | sudo sysctl -w net.ipv4.ip_forward=1 217 | sudo iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8000 218 | echo 'a' | nc -l localhost 8000 219 | 220 | Then: 221 | 222 | echo 'b' | nc example.com 80 223 | 224 | `PREROUTING` never worked for me. The only thing I could get working was: 225 | 226 | Does not work for localhost: you need `OUTPUT` for that. TODO why? 227 | 228 | ## s 229 | 230 | Specify source host to act on. 231 | 232 | sudo iptables -A INPUT -s example.com -j DROP 233 | 234 | Now: 235 | 236 | curl --connect-timeout 3 example.com 237 | 238 | times out, but: 239 | 240 | curl google.com 241 | 242 | works. 243 | 244 | ## d 245 | 246 | Destination to act on: 247 | 248 | sudo iptables -A OUTPUT -d example.com -j DROP 249 | 250 | Now: 251 | 252 | curl --connect-timeout 3 example.com 253 | 254 | times out, but: 255 | 256 | curl google.com 257 | 258 | works. 259 | 260 | ## p 261 | 262 | Specify protocol to act on: `tcp` or `udp`: 263 | 264 | sudo iptables -A OUTPUT -p tcp -j DROP 265 | 266 | ## dport 267 | 268 | Specify port to act on: 269 | 270 | sudo iptables -A OUTPUT -p tcp --dport 80 -j DROP 271 | 272 | Requires `-p`. 273 | 274 | ## Sources 275 | 276 | - . Official page. Contains links to tutorials. 277 | 278 | - 279 | 280 | - 281 | 282 | - 283 | -------------------------------------------------------------------------------- /mail.md: -------------------------------------------------------------------------------- 1 | # Mail 2 | 3 | ## Send Emails Programmatically 4 | 5 | Sending email from programs may be very difficult because anti spam measures that will block your naive attempts unless you configure everything properly. 6 | 7 | This means that you will have to test all major email providers to see if for some reason their anti spam is not blocking your emails. 8 | 9 | - Amazing client (desktop vs website vs mobile) statistics: https://litmus.com/blog/email-client-market-share-where-people-opened-in-2013/litmus-email-client-market-share-2013-infographic 10 | 11 | Key points: mobile rules. 12 | 13 | It seems that `gmail.com` is far on the lead for email hosts 14 | 15 | Common pitfalls: 16 | 17 | - try to do SMTP from blocked IP range. E.g.: trying to use Gmail SMTP from AWS address. 18 | 19 | Gmail simply prevents emails from going out because AWS IPs are all blacklisted. 20 | 21 | Alternative: use SES SMTP 22 | 23 | - try do to AWS SES SMTP with From address as `you@gmail.com`. 24 | 25 | Message will be sent, but falls under Gmail's spam because you don't have an SPF record. 26 | 27 | ### Free methods 28 | 29 | - AWS SES. 30 | 31 | 2014-03, 1 year period, 200 emails / day. 32 | 33 | Require production usage increase. Takes max 1 day and is free. 34 | 35 | - Google business, 1 month period. 36 | 37 | ## Email from your domain 38 | 39 | To receive and send email from a domain you own, you can: 40 | 41 | - set up an email server. You will have to keep it running and manage it. 42 | - some registrars such as GoDaddy provide an email forwarding service. This may be a simple solution if you do not expect very high reliability. 43 | 44 | Gmail does not allow you to redirect a message as: `me@gmail.com` -> `a@godaddy.com` -> `me.gmail.com:` try with a different email address. 45 | 46 | ## MTA 47 | 48 | Mail transfer agent. 49 | 50 | ## sendmail 51 | 52 | Interface that comes in multiple packages such as SSMTP and postfix, so to configure it you must first determine which package provides it. 53 | 54 | `sendmail` is an utility. Its interface is probably implemented by other packages because that utility was widely used. 55 | 56 | May be symlink to an executable, or to the `/etc/alternatives`. 57 | 58 | echo "asdf" | sendmail 59 | 60 | ## mail 61 | 62 | On Ubuntu a symlink to the alternatives system. 63 | 64 | echo -e "the message\n\nend of it" | mail -s "subject" -r "from@gmail.com" "to@gmail.com" 65 | mail -s "subject" -r "from@gmail.com" "to@gmail.com" 66 | 67 | Mail ends in a line which contains a single dot `.` or Ctrl + D. 68 | 69 | ## mailx 70 | 71 | POSIX. 72 | 73 | Does not seem to be used a lot, maybe because it does not have many capabilities. 74 | 75 | ## ssmtp 76 | 77 | Simple SMTP. 78 | 79 | Popular MTA. Really is simpler than Postfix to setup. 80 | 81 | Configuration file: 82 | 83 | vim /etc/ssmtp/ssmtp.conf 84 | 85 | Configurations to send an email from Gmail: 86 | 87 | Root=your_email@gmail.com 88 | Mailhub=smtp.gmail.com:465 89 | RewriteDomain=gmail.com 90 | AuthUser=your_gmail_username 91 | AuthPass=your_gmail_password 92 | FromLineOverride=Yes 93 | UseTLS=Yes 94 | 95 | Now you can send emails from the command line as: 96 | 97 | printf 'Subject: sub\nBody' | ssmtp destination@mail.com 98 | printf 'Subject: sub\nBody' | sendmail destination@mail.com 99 | 100 | The email will be sent from the email account you configured to send from. 101 | 102 | ## postfix 103 | 104 | Main configuration file: 105 | 106 | cat /etc/postfix/main.cf 107 | 108 | Postfix's `sendmail` does not show failure status immediately: it simply puts the email on a send queue. 109 | 110 | This is probably done so that email sending does not block the current session, allowing in particular longer retry times. 111 | 112 | To view the send queue, use `mailq`. 113 | 114 | ### mailq 115 | 116 | Show email sending queue. 117 | 118 | If delivery failed, explains why. 119 | 120 | ## mutt 121 | 122 | Can send mail with attachment. 123 | 124 | Curses interface. 125 | 126 | ## send email from website 127 | 128 | Sending email from a website may be nontrivial because of measures that must be taken to fight spam. 129 | 130 | ## spf 131 | 132 | Sender Policy Framework. 133 | 134 | Information that goes on the DNS for a host and says: hey, emails sent from IP XXX really come from this host, they are not span alright? 135 | 136 | Required by most email services, or messages fall under spam. 137 | -------------------------------------------------------------------------------- /media-protocols.md: -------------------------------------------------------------------------------- 1 | # Media protocols 2 | 3 | ## RTSP 4 | 5 | 6 | 7 | Browsers cannot see it, but VLC and `ffplay` can: 8 | 9 | Wowza hosts a live demo as of 2016: 10 | 11 | vlc rtsp://184.72.239.149/vod/mp4:BigBuckBunny_115k.mov 12 | 13 | 14 | 15 | Only does initial signaling, actual data goes through other protocols like RTP + RTCP. 16 | 17 | ## Implementations 18 | 19 | So I suppose Wowza implements an RTSP server. But it is closed source: 20 | 21 | looks like a major open source implementation. 22 | 23 | An Android implementation that streams the camera: Follow the instructions there, find your IP with and: 24 | 25 | vlc rtsp://192.168.88.141:1234 26 | 27 | The library that implements it: Looks simple. 28 | 29 | ## RTP 30 | 31 | ## RTCP 32 | 33 | 34 | 35 | Often used together. Data goes through RTP. RTCP synchronizes it. 36 | 37 | RTP implementation: 38 | 39 | ## RTMP 40 | 41 | 42 | 43 | Ex Macromedia proprietary, bought by Adobe, and then semi-released. 44 | 45 | ### rtmpdump 46 | 47 | Tool to work with it. 48 | 49 | - 50 | -------------------------------------------------------------------------------- /mitproxy.md: -------------------------------------------------------------------------------- 1 | # mitmproxy 2 | 3 | Carry out man in the middle TLS sniffs and manipulation. 4 | 5 | ## Install 6 | 7 | sudo aptitude install libxslt-dev 8 | sudo pip install mitmproxy 9 | 10 | ## Command line arguments 11 | 12 | - `-p`: port to listen on 13 | 14 | ## R 15 | 16 | ## Forward request to given location 17 | 18 | mitproxy -p 8000 -R http://example.com 19 | curl -H 'Host: example.com' -i http://localhost:8000/ 20 | 21 | ### Forward HTTPS request to given location 22 | 23 | My best attempt was: 24 | 25 | curl -k -H 'Host: github.com' https://localhost:8000 26 | curl: (35) error:04091077:rsa routines:INT_RSA_VERIFY:wrong signature length 27 | 28 | with `-k` to allow sessions without a certificate but it failed. Knowing that the mitmproxy generated certificates are kept under `~/.mitmproxy` might help. 29 | 30 | ## ncurses interface 31 | 32 | - `?`: help 33 | - `q`: quit 34 | - `C`: clear screen 35 | -------------------------------------------------------------------------------- /mtu.md: -------------------------------------------------------------------------------- 1 | # MTU 2 | 3 | 4 | 5 | Maximum transmission unit. 6 | 7 | Maximum packet size that can be transmitted over a physical link. 8 | 9 | Varies across different link technologies. 10 | -------------------------------------------------------------------------------- /nat.md: -------------------------------------------------------------------------------- 1 | # NAT 2 | 3 | How to turn it off to allow direct P2P? 4 | 5 | - 6 | - 7 | -------------------------------------------------------------------------------- /netcat.md: -------------------------------------------------------------------------------- 1 | # Netcat 2 | 3 | Low level send and receive TCP/UDP data. 4 | 5 | ## Versions 6 | 7 | In Ubuntu 12.04, `netcat` and `nc` are both symlinks to `nc.openbsd`. 8 | 9 | There is also the `netcat-traditional` package which offers another version (TODO is it the GNU `netcat`?) 10 | 11 | 12 | 13 | This tutorial considers the BSD version by default. 14 | 15 | ## nc 16 | 17 | Executable name. 18 | 19 | ## Basic usage 20 | 21 | Make a TCP HTTP get request and print the response: 22 | 23 | printf 'GET / HTTP/1.1\r\nHost: example.com\r\n\r\n' | nc example.com 80 24 | 25 | ## Multiple requests 26 | 27 | `nc` sends lines as you type them and over a single TCP connection if the server feels like taking it (and it should on HTTP 1.1): 28 | 29 | ( 30 | printf 'GET / HTTP/1.1\r\nHost: example.com\r\n\r\n'; 31 | sleep 2; 32 | printf 'GET / HTTP/1.1\r\nHost: example.com\r\n\r\n'; 33 | ) | nc example.com 80 | grep HTTP 34 | 35 | returns 2 responses. The same could be done by manually typing the requests in. 36 | 37 | ## u 38 | 39 | ## UDP 40 | 41 | UDP instead of TCP. 42 | 43 | ## l 44 | 45 | Listen for requests made on a port. 46 | 47 | Send response from stdin. 48 | 49 | Good way to test tools that send requests like `curl`. 50 | 51 | Example: 52 | 53 | printf 'HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\n\r\nHello curl!\n' \ 54 | | nc -kl localhost 8000 55 | 56 | On another terminal: 57 | 58 | curl localhost:8000 59 | 60 | The `nc` terminal prints its input: 61 | 62 | GET / HTTP/1.1 63 | User-Agent: curl/7.22.0 (i686-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3 64 | Host: localhost:8000 65 | Accept: */* 66 | 67 | And curl will print the reply it got: `Hello curl!`. 68 | 69 | Same with another `nc` instead of curl: 70 | 71 | echo 'abc' | nc localhost 8000 72 | 73 | ### ECHO server 74 | 75 | To do multiple tests of what is being sent, just wrap in a while and give an empty reply: 76 | 77 | while true; do printf '' | nc -l localhost 8000; done 78 | 79 | If `-l` is given, then the hostname is optional. If the hostname is not given, `nc` listens on all interfaces (TODO confirm). 80 | 81 | More advanced one that does multiple connections: 82 | 83 | ## v 84 | 85 | Give more verbose output. 86 | 87 | E.g., on `nc -l`, prints an extra line: 88 | 89 | Connection from 127.0.0.1 port 8000 [tcp/*] accepted 90 | 91 | before the request. 92 | 93 | ## k 94 | 95 | Keep listing after the first connection instead of shutting down. 96 | 97 | Requires the option `-l`. 98 | 99 | Terminal 1: 100 | 101 | nc -kl localhost 8000 102 | 103 | Terminal 2: 104 | 105 | echo 'abc' | nc localhost 8000 106 | echo 'def' | nc localhost 8000 107 | 108 | Terminal 1 has printed: 109 | 110 | abc 111 | def 112 | 113 | ## C 114 | 115 | If the last character is a newline `\n`, replace it with CRLF. 116 | 117 | ## HTTPS 118 | 119 | Not possible with `nc`: 120 | 121 | printf 'GET / HTTP/1.0\r\n\r\n' | nc google.com 443 122 | 123 | Returns empty. 124 | 125 | Consider `openssl` or `ncat` 126 | 127 | ## ncat 128 | 129 | `nc` version from `nmap` package. 130 | 131 | ### c 132 | 133 | Construct response with command. 134 | 135 | ### HTTPS 136 | 137 | ### ssl 138 | 139 | printf 'GET / HTTP/1.1\r\nHost: github.com\r\n\r\n' | ncat --ssl github.com 443 140 | 141 | As of Dec 2014, Facebook is annoying and requires a known user agent, or else you will get redirected to `/unsupportedbrowser` 142 | 143 | printf 'GET / HTTP/1.1\r\nHost: www.facebook.com\r\nUser-Agent: Mozilla/5.0\r\n\r\n' \ 144 | | ncat --ssl www.facebook.com 443 145 | 146 | The `Host` is mandatory or you get a redirect. TODO why 147 | -------------------------------------------------------------------------------- /netrc.md: -------------------------------------------------------------------------------- 1 | # netrc 2 | 3 | `$HOME/.netrc` is a config file that automates net logins (TODO: which type exactly of login?) 4 | 5 | It was primarily intended for FTP, but is also used by other applications like Google Cloud Platform CLI utils. 6 | -------------------------------------------------------------------------------- /netstat.md: -------------------------------------------------------------------------------- 1 | # netstat 2 | 3 | CLI utility that shows lots of socket info. 4 | 5 | Very important networking debug tool. 6 | 7 | Lists PID and program name of programs using ports. 8 | 9 | Shows both TCP/UDP Internet connections and UNIX domain sockets. 10 | 11 | In short: Internet connections are done via sockets whose address is given by an IP and a port number, and can communicate across computers. 12 | 13 | UNIX domain sockets are only for local communication. They are put into the filesystem and identified by a path on the filesystem. 14 | 15 | When a program uses a socket, it binds to it, and other programs cannot use it. 16 | 17 | Most useful options: 18 | 19 | - `n`: don't resolve IPs into hostnames. Greatly speeds up the output generation. 20 | - `a`: show both listening and not listening ports. 21 | - `p`: show program name and PID. 22 | - `t`: show only TCP 23 | - `u`: show only UDP 24 | - `x`: show only UNIX sockets 25 | - `i`: show information on interfaces. 26 | - `r`: show kernel routing table. 27 | - `s`: show statistics on several protocols. 28 | 29 | Sample output for Internet section: 30 | 31 | Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name 32 | tcp 0 0 localhost:32842 localhost:48553 ESTABLISHED 3497/GoogleTalkPlug 33 | 34 | Meaning of the fields: 35 | 36 | - `Proto`: protocol: TCP or UDP 37 | - `Recv-Q`: TODO 38 | - `Send-Q`: TODO 39 | - `Local Address`: local IP and port of the connection. TODO can this only be either `localhost` or local network (192.168.X.X on type)? 40 | - `Foreign Address`: the IP and port of the remote endpoint. `0.0.0.0:*` appears when `LISTEN` on IPv4 and `:::*` on IPv6. 41 | - `State`: one of the standard TCP states for the connection. Empty for UDP since it is stateless. 42 | - `PID/Program name`: self explanatory 43 | 44 | It does not seem possible to see the full arguments passed to each process as `ps -A` does. This is a problem, since for interpreted languages, several processes may look the same without arguments, e.g. `node` and `node` instead of `node script1` and `node script1`. The best workaround so far is to `awk` and pipe into `ps`: 45 | 46 | ## Sample output lines 47 | 48 | Let's take a look at some typical output lines to better understand the fields. 49 | 50 | Running: 51 | 52 | python -m SimpleHTTPServer 53 | 54 | gives the line: 55 | 56 | Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name 57 | tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN 21267/python 58 | 59 | since: 60 | 61 | - `Local Address` is `0.0.0.0:8000`, this means that the server is listening to all interfaces on port 8000. TODO: why? 62 | - `Foreign Address` is `0.0.0.0:*`, this means that no one is currently listening. This will almost always be the case for web servers like this, since HTTP is stateless that closes the connection quickly, so you shouldn't be able to notice it. TODO: why? 63 | 64 | If it were on IPv6 (`Proto` == `tcp6` or `udp6`), it would be `:::*` instead of `0.0.0.0:*`, which means the same in this context. 65 | -------------------------------------------------------------------------------- /network-manager.md: -------------------------------------------------------------------------------- 1 | # NetworkManager 2 | 3 | GNOME project for simplifying network configuration. 4 | 5 | Also has an Applet that shows on Ubuntu 12.04's taskbar, indicating connection status. 6 | 7 | On Ubuntu it comes on the packages `network-manager` and `network-manager-gui` for the applet. 8 | 9 | It corresponds to the upstart service `network-manager`, so for example to reload configurations it can be restarted with: 10 | 11 | sudo service network-manager restart 12 | 13 | ## nm-applet 14 | 15 | The applet. 16 | 17 | ## nm-connection-editor 18 | 19 | Opened from the applet "Edit Connections". 20 | 21 | ## nm-tool 22 | 23 | Get NetworkManager status from the command line. Sample output: 24 | 25 | NetworkManager Tool 26 | 27 | State: connected (global) 28 | 29 | - Device: wlan0 [NUMERICABLE-B2BD] -------------------------------------------- 30 | Type: 802.11 WiFi 31 | Driver: rtl8192ce 32 | State: connected 33 | Default: no 34 | HW Address: E0:06:E6:C7:97:8F 35 | 36 | Capabilities: 37 | Speed: 72 Mb/s 38 | 39 | Wireless Properties 40 | WEP Encryption: yes 41 | WPA Encryption: yes 42 | WPA2 Encryption: yes 43 | 44 | Wireless Access Points (* = current AP) 45 | NUMERICABLE-4D5F:Infra, C0:D9:62:C7:43:49, Freq 2412 MHz, Rate 54 Mb/s, Strength 26 WPA WPA2 46 | FREEBOX_BRAHIM_GH: Infra, F4:CA:E5:D9:B3:2C, Freq 2437 MHz, Rate 54 Mb/s, Strength 26 WPA 47 | Jordy: Infra, 56:76:06:1D:11:14, Freq 2442 MHz, Rate 54 Mb/s, Strength 26 WPA 48 | *NUMERICABLE-B2BD: Infra, E0:AB:31:AC:30:6D, Freq 2412 MHz, Rate 54 Mb/s, Strength 84 WPA WPA2 49 | freephonie: Infra, 56:76:06:1D:11:17, Freq 2442 MHz, Rate 54 Mb/s, Strength 26 WPA Enterprise 50 | NUMERICABLE-F1EC:Infra, 00:1A:2B:9B:0C:F4, Freq 2462 MHz, Rate 54 Mb/s, Strength 26 WPA WPA2 51 | [...] 52 | 53 | IPv4 Settings: 54 | Address: 192.168.0.10 55 | Prefix: 24 (255.255.255.0) 56 | Gateway: 192.168.0.1 57 | 58 | DNS: 89.2.0.1 59 | DNS: 89.2.0.2 60 | 61 | 62 | - Device: eth0 [Wired connection 1] ------------------------------------------- 63 | Type: Wired 64 | Driver: e1000e 65 | State: connected 66 | Default: yes 67 | HW Address: 00:21:CC:CE:F2:C1 68 | 69 | Capabilities: 70 | Carrier Detect: yes 71 | Speed: 1000 Mb/s 72 | 73 | Wired Properties 74 | Carrier: on 75 | 76 | IPv4 Settings: 77 | Address: 192.168.0.11 78 | Prefix: 24 (255.255.255.0) 79 | Gateway: 192.168.0.1 80 | 81 | DNS: 89.2.0.1 82 | DNS: 89.2.0.2 83 | 84 | ## nmcli 85 | 86 | Control NetworkManger from CLI. 87 | 88 | Bring down wired connection: 89 | 90 | nmcli con down id 'Wired connection 1' 91 | nmcli dev disconnect iface eth0 92 | 93 | Bring it back up: 94 | 95 | nmcli con up id 'Wired connection 1' 96 | 97 | ## /etc/NetworkManager/system-connections/ID 98 | 99 | ## configuration file 100 | 101 | NetworkManager's configuration files: 102 | 103 | sudo vim /etc/NetworkManager/system-connections/ID 104 | 105 | one per interface. 106 | 107 | Modified through the applet. 108 | 109 | ## Related tools 110 | 111 | ### ifup 112 | 113 | ### ifdown 114 | 115 | There are several levels of network management tools: . 116 | 117 | From the lowest level to the highest: 118 | 119 | - `ifconfig` 120 | - `ifup` and `ifdown` 121 | - NetworkManager 122 | 123 | ### /etc/network/interfaces 124 | 125 | Configuration for `ifup` and `ifdown`. 126 | 127 | man interfaces 128 | 129 | If you manually set configuration on `/etc/network/interfaces`, NetworkManager will now touch those interfaces and display them as "Not Managed". 130 | -------------------------------------------------------------------------------- /network-simulation.md: -------------------------------------------------------------------------------- 1 | # Network simulation 2 | 3 | ## Uncertainties 4 | 5 | - 6 | - 7 | - 8 | -------------------------------------------------------------------------------- /nginx.md: -------------------------------------------------------------------------------- 1 | # Nginx 2 | 3 | App / web server with similar capabilities to Apache. 4 | 5 | 6 | 7 | Very used by the Rails community. 8 | 9 | ## Vs apache 10 | 11 | Architecture comparison: . Nginx scales better, Apache is older and has more configuration options and libraries. 12 | 13 | Good official beginners tutorial: . 14 | 15 | ## Configuration 16 | 17 | Main configuration file on Ubuntu 18 | 19 | vim /etc/nginx/nginx.conf 20 | 21 | ### Serve static files 22 | 23 | http { 24 | server { 25 | # URL / 26 | location / { 27 | root /data/www; 28 | } 29 | 30 | # URL /images/ 31 | location /images/ { 32 | root /data; 33 | } 34 | } 35 | } 36 | 37 | ### proxy_pass 38 | 39 | Serve forward requests somewhere else. 40 | 41 | Forward everything on `http` to `example.com`: 42 | 43 | http { 44 | server { 45 | listen 0.0.0.0:8000; 46 | location / { 47 | proxy_pass http://example.com; 48 | } 49 | } 50 | } 51 | 52 | Try it out: 53 | 54 | curl -vvv localhost:8000 55 | curl -vvv example.com 56 | 57 | The requests are identical, except that Nginx rewrites the server HTTP header and sets it to Nginx. 58 | TODO how to prevent that? 59 | 60 | #### HTTPS forward proxy 61 | 62 | Not possible: 63 | 64 | Nginx is designed to be a reverse proxy, not forward. 65 | 66 | ### proxy_set_header 67 | 68 | 69 | 70 | Make the proxy modify a given request header and pass it to the proxied server. 71 | 72 | ### access_log 73 | 74 | 75 | 76 | File to which `nginx` will log access. 77 | 78 | Sample line generated by a request: 79 | 80 | 127.0.0.1 - - [04/Dec/2014:22:57:13 +0100] "GET / HTTP/1.1" 200 641 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:34.0) Gecko/20100101 Firefox/34.0" 81 | 82 | Breakdown: 83 | 84 | - `127.0.0.1`: where the request was received 85 | - `[04/Dec/2014:22:57:13 +0100]`: timestamp 86 | - `GET / HTTP/1.1` first line of request 87 | - `200`: return status 88 | - `641`: bytes in the body 89 | - `"-"`: TODO 90 | - `Mozilla/5.0 ...`: user agent as given on the HTTP request header 91 | 92 | Can be customized with `log_format` 93 | 94 | The default `log_format` is: 95 | 96 | $remote_addr - $remote_user [$time_local] "$request" 97 | $status $body_bytes_sent "$http_referer" "$http_user_agent" 98 | 99 | ### log_format 100 | 101 | 102 | 103 | TODO possible to log the entire transaction? 104 | -------------------------------------------------------------------------------- /nmap.md: -------------------------------------------------------------------------------- 1 | # nmap 2 | 3 | Port scanner: show open ports, state and service name associated to the port. 4 | 5 | Great way to start deciding how to go about hacking someone. 6 | 7 | Ubuntu 12.04 install: 8 | 9 | sudo aptitude install nmap 10 | 11 | TCP services: 12 | 13 | nmap google.com 14 | nmap localhost 15 | 16 | You are gonna get at least 80 on Google for their HTTP server and on localhost too if you are running an HTTP server such as Apache. 17 | 18 | Test on localhost: 19 | 20 | sudo nmap localhost 21 | 22 | Sample output excerpt: 23 | 24 | PORT STATE SERVICE 25 | 22/tcp open ssh 26 | 53/tcp open domain 27 | 80/tcp open http 28 | 29 | - `STATE`: TODO: maps to one of the standard TCP connection states, e.g. `open` == `Listen`? 30 | - `SERVICE`: TODO: how does it determine the service? Just by using the default IANA registered one? 31 | 32 | A `nc -l localhost 8001` shows as: 33 | 34 | 8001/tcp open unknown 35 | 36 | View UDP ports: 37 | 38 | sudo nmap -sU localhost 39 | 40 | Trivia: appeared in many hacking scenes of movies: 41 | -------------------------------------------------------------------------------- /openssl.md: -------------------------------------------------------------------------------- 1 | # OpenSSL 2 | 3 | 4 | 5 | 6 | 7 | Implements a large range of Cryptographic utilities. 8 | 9 | Library and command line tool frontend. 10 | 11 | Apache license. 12 | 13 | This is the library that was affected by the notorious 2014 Heartbleed vulnerability, which allowed theft of private keys , and affected around 17% of the encrypted network traffic of the world. It did not affect SSH though. 14 | 15 | ## Alternatives 16 | 17 | [Mozilla Network Security Services (NSS)](http://en.wikipedia.org/wiki/Network_Security_Services). 18 | 19 | ## help 20 | 21 | List all commands: 22 | 23 | openssl help 24 | 25 | ## version 26 | 27 | openssl version 28 | 29 | Sample output: 30 | 31 | OpenSSL 1.0.1f 6 Jan 2014 32 | 33 | ## enc 34 | 35 | Encode and decode with symmetric key. 36 | 37 | List all encodings: 38 | 39 | openssl enc help 40 | 41 | Many encodings have several variants with suffixes like: `cbc`, `ecb`, etc. These are well known modes which apply to several algorithms of the block cypher family: CBC is a popular and reasonable choice in case you have no idea what to pick. 42 | 43 | ### d 44 | 45 | Decrypts: 46 | 47 | printf 'a' | openssl enc -base64 | openssl enc -d -base64 48 | 49 | Outputs the original input: 50 | 51 | a 52 | 53 | ### base64 54 | 55 | Not really a encryption, but present because used in some algorithms: 56 | 57 | printf 'a' | openssl enc -base64 58 | 59 | Output: 60 | 61 | YQ== 62 | 63 | Let's quickly cross-check with the GNU Coreutils: 64 | 65 | printf 'a' | base64 66 | 67 | Output: 68 | 69 | YQ== 70 | 71 | OK, I think they've both got that right. 72 | 73 | Now let's decrypt it: 74 | 75 | printf 'a' | openssl enc -base64 | openssl enc -d -base64 76 | 77 | ### aes 78 | 79 | printf 'a' | openssl enc -aes-256-cbc -out /tmp/openssl 80 | 81 | Asks for you password. Let's check that it was actually encrypted: 82 | 83 | cat /tmp/openssl 84 | 85 | I see trash. Good. Now let's decrypt: 86 | 87 | openssl enc -d -aes-256-cbc -in /tmp/openssl 88 | 89 | Asks for password again. After that, output: 90 | 91 | a 92 | 93 | ## Digest 94 | 95 | ## dgst 96 | 97 | printf 'a' | openssl dgst -sha256 98 | 99 | Outputs: 100 | 101 | ca978112ca1bbdcafac231b39a23dc4da786eff8147c4e72b9807785afee48bb 102 | 103 | Let's cross check with GNU Coreutils: 104 | 105 | printf 'a' | sha256sum 106 | 107 | Output: 108 | 109 | ca978112ca1bbdcafac231b39a23dc4da786eff8147c4e72b9807785afee48bb 110 | 111 | ## Public key cryptography 112 | 113 | ### Prerequisites 114 | 115 | Basics of public key cryptography like RSA are assumed. You should understand: 116 | 117 | - the roles of the public and private key 118 | - the difference between encryption and signature 119 | 120 | ### Generate key 121 | 122 | Generate a RSA public private pair with length 1024: 123 | 124 | openssl genrsa -out /tmp/key.pem 1024 125 | 126 | Check it out: 127 | 128 | cat /tmp/key.pem 129 | 130 | Gives: 131 | 132 | -----BEGIN RSA PRIVATE KEY----- 133 | MIICXQIBAAKBgQDVdZ+sbEhjYpUXmrtpaw9u/cwlfutGv8TICYQSq9vWl2NUdD7m 134 | ... 135 | 1uBY2kGhER9rP3JSCBO/a6tmRdWTK//D88T3TWAzgq9J 136 | -----END RSA PRIVATE KEY----- 137 | 138 | Get human-readable information out of it: 139 | 140 | openssl genrsa -in /tmp/key.pem 1024 -text 141 | 142 | Sample output: 143 | 144 | Private-Key: (1024 bit) 145 | modulus: 146 | 00:d5:75:9f:ac:6c:48:63:62:95:17:9a:bb:69:6b: 147 | ... 148 | 93:5d:f6:18:6e:d7:32:47:09 149 | publicExponent: 65537 (0x10001) 150 | privateExponent: 151 | 00:8b:e1:c7:e0:53:19:60:df:a2:7a:54:70:ca:15: 152 | ... 153 | ca:81:63:55:7c:2f:1a:79:31 154 | prime1: 155 | 00:fe:a6:26:b1:29:8d:cd:0d:03:67:14:c0:55:5a: 156 | ... 157 | 69:b4:ef:39:0d 158 | prime2: 159 | 00:d6:97:88:0b:4b:c0:e6:ea:ad:c8:3b:e2:74:18: 160 | ... 161 | ee:94:9a:ce:ed 162 | exponent1: 163 | 00:da:6b:7f:62:94:46:e9:ab:90:d0:98:ef:04:07: 164 | ... 165 | 42:7d:0c:77:fd 166 | exponent2: 167 | 69:a1:85:b2:29:c3:f6:e7:26:4c:1e:4e:46:5f:b6: 168 | ... 169 | 20:92:63:b1 170 | coefficient: 171 | 4b:5e:c7:78:b3:32:eb:2d:13:bb:21:63:96:f1:22: 172 | ... 173 | 33:82:af:49 174 | 175 | You should normally password encrypt that file. 176 | 177 | Now let's generate the public key: 178 | 179 | openssl rsa -in key.pem -pubout | tee /tmp/pub-key.pem 180 | 181 | Outputs: 182 | 183 | -----BEGIN PUBLIC KEY----- 184 | MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDVdZ+sbEhjYpUXmrtpaw9u/cwl 185 | futGv8TICYQSq9vWl2NUdD7mLDC6b6aYib2aO3KiS0HWC6AFztxom3oO7pkhFaKz 186 | i7isTFVNt/ZIyyTw4W88dDNNBtNMhwBmAMEp4hsbvuKeko2ErJl7l3f3hEpnQM0g 187 | 6qmrRpNd9hhu1zJHCQIDAQAB 188 | -----END PUBLIC KEY----- 189 | 190 | ### Encrypt with key 191 | 192 | Now let's use the public key to encrypt something: 193 | 194 | printf 'a' | openssl rsautl -encrypt -pubin -inkey /tmp/pub-key.pem | tee /tmp/enc 195 | 196 | We must use the option `-pubin` when using the public key. 197 | 198 | Or using the file with the private keys: 199 | 200 | printf 'a' | openssl rsautl -encrypt -inkey /tmp/key.pem | tee /tmp/enc 201 | 202 | The output is all jumbled. 203 | 204 | Now let's decrypt: 205 | 206 | openssl rsautl -decrypt -inkey /tmp/key.pem > /tmp/sign 229 | openssl rsautl -verify -pubin -inkey /tmp/pub-key.pem 240 | 241 | openssl s_client -connect google.com:443 242 | 243 | openssl s_client -connect google.com:443 | openssl x509 -pubkey -noout 244 | 245 | ## PKI 246 | 247 | teaches you how to host a PKI system just for fun. 248 | 249 | ## Sources 250 | 251 | - 252 | -------------------------------------------------------------------------------- /physical-layer.md: -------------------------------------------------------------------------------- 1 | # Physical layer 2 | 3 | ## Wireless protocols 4 | 5 | ### 3GPP 6 | 7 | 8 | 9 | Series of standards that include 2G, 3G, and 4G (a term which is disputed). 10 | 11 | ### LTE 12 | 13 | ### 4G 14 | 15 | 4G is only a marketing term. 16 | 17 | 18 | 19 | Wide area. 20 | 21 | Is an incompatible fork from the 3GPP 4G, causing confusion 22 | 23 | ### 3G 24 | 25 | ### UMTS 26 | 27 | TODO 28 | 29 | #### WCDMA 30 | 31 | Part of UMTS? 32 | 33 | ### 2G 34 | 35 | ### GMS 36 | 37 | TODO dates 38 | 39 | ### Wi-Fi 40 | 41 | defines it as the IEEE 802.11.XXX family of protocols. 42 | 43 | So Wi-Fi is just a marketing term. 44 | 45 | It is so dominant that people call WLAN Wi-Fi. 46 | 47 | #### Wi-Fi Direct 48 | 49 | 50 | 51 | Connect two computers directly through Wi-Fi without a router. 52 | 53 | ### Bluetooth 54 | 55 | 56 | 57 | Initially IEEE 802.15.1, but no longer maintained by IEEE. 58 | 59 | Less powerful than Wi-Fi, but less expensive / energy consuming. 60 | -------------------------------------------------------------------------------- /ping.md: -------------------------------------------------------------------------------- 1 | # ping 2 | 3 | CLI utility that sends ICMP echo requests to a server that accepts them. 4 | 5 | It measures the time it takes for the answer to come back, which is a measure of connectivity between the two computers. 6 | 7 | The term is widely known to online gamers, where the term *the ping*, which few gamers know is the send and return time, is a measure for connection quality. 8 | 9 | The default IANA port for ping requests is port 7/TCP or 7/UDP. 10 | 11 | Send an echo every second to monitor connectivity: 12 | 13 | ping www.google.com 14 | -------------------------------------------------------------------------------- /port.md: -------------------------------------------------------------------------------- 1 | # Port 2 | 3 | Once you have determined a host (computer), e.g. through an IP or hostname, you still have to talk to one of the specific programs running on that computer. 4 | 5 | Each program listens on an specific port which is set by convention. 6 | 7 | Ports from 1 - 1023 are also known as "well-known ports" or "privileged ports". On UNIX-like systems, only privileged users (`root`) can bind to those ports. All have reserved or standardized functions by an organization called IANA: . 8 | 9 | The ports form 1024 to 49151 are the so called "registered ports". Projects can make a request to IANA to register one of those posts as used in order to avoid port clashes. On most systems, it is possible to bind to those ports without `sudo`. 10 | 11 | There are 2 ports number 10: `10/tcp` and `10/udp`, each for a different protocol. 12 | 13 | On POSIX systems, ports are typically implemented via sockets. 14 | 15 | Ports are stored on the transport layer header, e.g. TCP source and destination port fields. 16 | 17 | ## Source ports 18 | 19 | ## Local port 20 | 21 | ## Dynamic port 22 | 23 | ## Ephemeral port 24 | 25 | 26 | 27 | Every connection is made between two ports. 28 | 29 | Therefore the client, besides knowing the correct port on the server for the desired service, must also create a local port. 30 | 31 | If the transaction involves a response like HTTP, it is form that port that the response will be listened. 32 | 33 | Unlike the ports on the server, which need to be well known numbers, the local ports created for a given connection can be anything, and are randomly picked on an interval of high numbers so a not to conflict with server ports that might be used locally. 34 | 35 | On Linux, you can find out the local port range with: 36 | 37 | cat /proc/sys/net/ipv4/ip_local_port_range 38 | 39 | Sample output: 40 | 41 | 32768 61000 42 | 43 | IANA recommends the range 49152 to 65535 (`2^15 + 2^14` to `2^16-1`) for ephemeral ports: 44 | 45 | ## Tools 46 | 47 | One convenient way to write and read to ports is Netcat. 48 | 49 | `netstat` is a great way to get information on which ports are being used by which program. 50 | 51 | ## Important ports 52 | 53 | - `22`: SSH 54 | - `53`: DNS 55 | - `80`: HTTP 56 | - `443`: HTTPS 57 | 58 | ### Port 0 59 | 60 | Linux choses the first free port it can find. 61 | -------------------------------------------------------------------------------- /proxy-server.md: -------------------------------------------------------------------------------- 1 | # Proxy Server 2 | 3 | A server that takes requests, forwards them somewhere, waits for the response, and forwards the response back. 4 | 5 | It can of course do whatever it wants to the response. 6 | 7 | A proxy is both a server and a client. 8 | 9 | ## Forward proxy 10 | 11 | AKA just Proxy. 12 | 13 | A proxy server that sits between the clients and the Internet: 14 | 15 | client <---> proxy <---> firewall <---> internet <---> server 16 | 17 | Functions: 18 | 19 | - limit what clients can do, e.g., blacklist domains. 20 | 21 | ## Reverse proxy 22 | 23 | The inverse of a forward proxy: one that sits between the internet and the server: 24 | 25 | client <---> firewall <---> internet <---> reverse proxy <---> server 26 | 27 | Commonly used implementations are Apache and Nginx. 28 | 29 | The typical application is when you have a script server that generates dynamic web pages like Rails or Django. 30 | 31 | Rails and Django are ineffective at serving static files, while Apache and Nginx are super optimized for that. TODO how? 32 | 33 | So you set up Apache or Nginx to serve every static file request under `/assets` from a directory, and pass the others to the server. 34 | 35 | Another thing that those RPs do is load balancing, i.e. sending requests to the server that is the least loaded. TODO how? 36 | 37 | ## Transparent proxy 38 | 39 | A transparent proxy is one that does not modify the request unless strictly necessary. 40 | 41 | The term is defined on 42 | -------------------------------------------------------------------------------- /rest.md: -------------------------------------------------------------------------------- 1 | # REST APIs 2 | 3 | REST is an HTTP API style. 4 | 5 | This shall discuss common design patterns for such APIs. 6 | 7 | Good way to learn: see famous APIs: 8 | 9 | - GitHub: 10 | - Dropbox: 11 | 12 | ## GET with a body 13 | 14 | You can send a body on any request, but it should be ignored for GET. 15 | 16 | There is no good workaround: maybe the best is using a PUT instead. 17 | 18 | 19 | 20 | ## File upload 21 | 22 | ### Together with other data 23 | 24 | Taken from this [discussion](http://feedback.gitlab.com/forums/176466-general/suggestions/3865548-api-to-attach-attachments-to-notes-issue-comments) on how to attack files to comments from API. 25 | 26 | Comments and most of the creation API were currently created by JSON POST requests. 27 | 28 | Possibilities: 29 | 30 | - single URL, two accepted content types: 31 | 32 | - `multipart/form-data` with two parts: one JSON metadata (currently only "body" for notes), one part for the file. 33 | 34 | - `application/json`: just the JSON metadata, in case the file is empty. 35 | 36 | My preferred option, as it keeps most data in JSON format as the rest of the API, note creation takes a single HTTP request, generalises well for multiple files. 37 | 38 | - Two separate URLs: one for metadata via JSON, one for the file, with file data on body. 39 | 40 | Downsides: 41 | 42 | - note create / update takes multiple HTTP requests 43 | - on create, the user has to do the extra work of interpretation 44 | - occupies 1 URL namespaces for each uploaded file 45 | 46 | Upside: simpler for us to implement. 47 | 48 | - `multipart/form-data`, one part for each field. 49 | 50 | This is how the web interface (non-API) upload currently works. Simple because already implemented, but not coherent with the rest of the API since no JSON used. 51 | 52 | - ASCII encode the upload and send on JSON. 53 | 54 | Best way to do it: 55 | 56 | ## Pagination 57 | 58 | GitHub v3: link header as: 59 | 60 | Link: ; rel="next", 61 | ; rel="last" 62 | 63 | Info: 64 | -------------------------------------------------------------------------------- /route.md: -------------------------------------------------------------------------------- 1 | # route 2 | 3 | View kernel routing table: 4 | 5 | route 6 | 7 | Numeric instead of names: 8 | 9 | route -n 10 | -------------------------------------------------------------------------------- /samba.md: -------------------------------------------------------------------------------- 1 | # Samba 2 | 3 | Open source Linux implementation of the SMB/CIFS networking protocol used by default on Windows. 4 | 5 | It allows for file, printer and driver sharing on a network. 6 | 7 | Best option for cross platform file transfers. 8 | -------------------------------------------------------------------------------- /sip.md: -------------------------------------------------------------------------------- 1 | # SIP 2 | 3 | 4 | 5 | Negotiates which audio and video encoding protocols are available on both sides and chooses one of them. 6 | -------------------------------------------------------------------------------- /smtp.md: -------------------------------------------------------------------------------- 1 | # SMTP 2 | 3 | - 4 | - 5 | 6 | Vs IMAP vs POP3: 7 | 8 | - IMAP: download email on a remote server from another one 9 | - POP3: download emails locally from remote server 10 | -------------------------------------------------------------------------------- /squid.md: -------------------------------------------------------------------------------- 1 | # Squid 2 | 3 | Forward proxy. 4 | 5 | Configuration file: 6 | 7 | sudo vim /etc/squid3/squid.conf 8 | 9 | ## Sources 10 | 11 | - 12 | 13 | ## access_log 14 | 15 | 16 | -------------------------------------------------------------------------------- /ssh.md: -------------------------------------------------------------------------------- 1 | # SSH 2 | 3 | Protocol like telnet, but encrypted 4 | 5 | For SSH to work you will need: 6 | 7 | - a SSH server running on a machine. 8 | - a SSH client running on another machine. 9 | 10 | It is possible to do tests using `localhost` on a single machine. 11 | 12 | Per user configurations for both the server and the client are contained under `~/.ssh` 13 | 14 | ## Implementations 15 | 16 | 17 | 18 | OpenSSH is the most important implementation on Linux. 19 | 20 | OpenSSH used to depend on part OpenSSL, but that dependency was removed: 21 | 22 | ## Server 23 | 24 | Known as `sshd`, which stands for ssh daemon. 25 | 26 | Must be installed and running on a machine for users from other computers to log into that machine. 27 | 28 | The server part of ssh is called `sshd`. 29 | 30 | Configuration: 31 | 32 | sudo cp /etc/ssh/sshd_config{,.bak} 33 | sudo vim /etc/ssh/sshd_config 34 | 35 | For the server to work, the following configuration is minimal: 36 | 37 | Host * #config for all hosts 38 | Port 22 #open port 22 39 | AllowUsers user1 user2 #allow the given users 40 | 41 | The server is often started as part of the `initrd` system. 42 | 43 | Therefore, to get it running you will probably use: 44 | 45 | sudo service ssh start 46 | 47 | and to stop it: 48 | 49 | sudo service ssh stop 50 | 51 | and to check its status: 52 | 53 | sudo service ssh status 54 | 55 | The default log file for the server is: `/var/log/auth.log`, which is shared by other utilities such as `sudo`. If things fail, that is where you should look! Try `sudo grep sshd /var/log/auth.log` for the relevant log lines. 56 | 57 | ## Client 58 | 59 | The client connects to a server to give shell access on the server. 60 | 61 | Make sure that the configuration files are correct: 62 | 63 | /etc/ssh/ssh_config{,.bak} 64 | ~/.ssh/config 65 | 66 | Get the version of your ssh client: 67 | 68 | ssh -V 69 | 70 | For local tests, use localhost and the current user: 71 | 72 | host=localhost 73 | user=`id -un` 74 | 75 | Connect to hostname with your current username: 76 | 77 | ssh $host 78 | 79 | Get debug level information if things don't work: 80 | 81 | ssh -v $host 82 | ssh -vv $host 83 | ssh -vvv $host 84 | 85 | For this to work you need: 86 | 87 | - your host (computer) is allowed. See ssh server. 88 | - your user is allowed. See ssh server. 89 | - your user exists as a regular user in the *server* computer. See `useradd`. 90 | 91 | There are two main methods of connection: 92 | 93 | - using an authorized public RSA id, which does not require a password, unless your password is locally encrypted by a password, which is recommended. 94 | 95 | This method is used if the key is allowed. 96 | 97 | - using the same password as the user on the server has. 98 | 99 | This method is only used if there is not authorized key on the local machine. 100 | 101 | Connect to hostname with the given username: 102 | 103 | ssh $user@$host 104 | 105 | or: 106 | 107 | ssh -l $user $host 108 | 109 | Choose port: 110 | 111 | p=22 112 | ssh -p $p $h 113 | 114 | The default is 22 specified by IANA, so don't change it if you can avoid it. 115 | 116 | It is *not* possible to set ports via the common URL syntax: `ssh host:22`. 117 | 118 | ### Client configuration 119 | 120 | man ssh_config 121 | 122 | File: 123 | 124 | ~/.ssh/config 125 | 126 | Different keys for different hosts: 127 | 128 | Host server1.nixcraft.com 129 | IdentityFile ~/backups/.ssh/id_dsa 130 | Host server2.nixcraft.com 131 | IdentityFile /backup/home/userName/.ssh/id_rsa 132 | 133 | Different keys for different website users with the same SSH username (e.g. on GitHub every git operation is done with the user `git` for every user): 134 | 135 | Host github-user1 136 | HostName github.com 137 | User git 138 | IdentityFile /home/USER/.ssh/id_rsa_github_user1 139 | IdentitiesOnly yes 140 | 141 | Host github-user2 142 | HostName github.com 143 | User git 144 | IdentityFile /home/USER/.ssh/id_rsa_github_user2 145 | IdentitiesOnly yes 146 | 147 | and then: 148 | 149 | git remote add alice git@gitolite-as-alice:whatever.git 150 | git remote add bob git@gitolite-as-bob:whatever.git 151 | 152 | ### Password from command line 153 | 154 | Impossible? 155 | 156 | - 157 | - 158 | - 159 | - 160 | 161 | ## Authentication methods 162 | 163 | 164 | 165 | SSH supports multiple authentication methods: 166 | 167 | - public key: the client must install the public key for a private key it owns on the server. This is the default. 168 | 169 | - password: type the Linux username / password pair. 170 | 171 | Use it once: 172 | 173 | ssh -o PubkeyAuthentication=no username@hostname.com 174 | 175 | Less setup than key if you can have an user on the server, but you must store a password on the server. 176 | 177 | But this password can be shorter than that to encrypt your private key, as it would require an online attack. 178 | 179 | - host-based: allow anyone from some host to login directly 180 | 181 | - keyboard: TODO 182 | 183 | ## authorized_keys 184 | 185 | List of public keys accepted by server for login as a given user. 186 | 187 | Each user has its own `/home/u/.ssh/authorized_keys` keys, which determine which public keys allow to login as that user. This means that to login as user `u`, the file `/home/u/.ssh/authorized_keys` must contain your public key. With that you will only be able to login into the account of user `u`. 188 | 189 | This file is only used for RSA authentication, not password. 190 | 191 | In addition to the allowed key, each line can also contain extra options that control the connection: 192 | 193 | from="ok.com",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa 194 | 195 | Non-obvious options above include: 196 | 197 | - `command="command"`: fixes a single command to be run to restrict what uses can do. This command is crucial for GitLab's (and likely GitHub's) SSH system, where SSH can only be used for Git operations, and always calls: 198 | - only connections coming from `ok.com` will be accepted. 199 | 200 | Options are documented with the server at `man sshd` since they are only used by the server. 201 | 202 | SSH is by default very fussy about the permissions of this file which should be: 203 | 204 | chmod 700 ~/.ssh 205 | chmod 600 ~/.ssh/authorized_keys 206 | 207 | and not more permissive. If you really want that, you can do configure SSH to be less safe via `StrictModes no`. 208 | 209 | ## known_hosts 210 | 211 | Located at: `.ssh/known_hosts`. 212 | 213 | Your client will only connect to a server if its key is in known hosts. This file exists because security is useless if someone is impersonating the message receiver. If the server's public identity is not in the known hosts file, SSH will ask is you want to add it. 214 | 215 | ## ssh-keygen 216 | 217 | Generates public and private key pairs for use with ssh. 218 | 219 | Generate an RSA public private pair: 220 | 221 | ssh-keygen -f ~/.ssh/id_rsa -t rsa -C "you@email.com" 222 | 223 | By default the keys are put under `~/.ssh` with names `id_rsa` for the private and `id_rsa.pub` and have length 1024 bits. 224 | 225 | *Do* use a passphrase, otherwise anyone that gets his hand on your `id_rsa` file owns your identity. 226 | 227 | *Never* share your private key! It is like a password that allows you to connect to servers. 228 | 229 | There can be only one key per file. 230 | 231 | When you invoke `ssh`, it will always use the same key by default, but you can configure it to use different keys in different connections. 232 | 233 | The formats are for the public key: 234 | 235 | ssh- 236 | 237 | For the private key: 238 | 239 | -----BEGIN RSA PRIVATE KEY----- 240 | 241 | -----END RSA PRIVATE KEY----- 242 | 243 | The actual data format a bit more involved. Discussion [here](http://stackoverflow.com/questions/12749858/rsa-public-key-format). Basically the data is Base 64 encoded, and it also contains some necessary algorithm metadata. 244 | 245 | Often used to determine if a key is present or not is the key's fingerprint, which are just hashes of the keys. 246 | 247 | Fingerprints are often displayed on the following format: 248 | 249 | 43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8 250 | 251 | with the colons added for only readability. 252 | 253 | You can get the fingerprint or all the fingerprints present on a file via: 254 | 255 | ssh-keygen -lf ~/.ssh/id_rsa 256 | ssh-keygen -lf ~/.ssh/id_rsa.pub 257 | ssh-keygen -lf ~/.ssh/known_hosts 258 | 259 | ### Change password for a given key 260 | 261 | Change password, possibly to an empty one : 262 | 263 | ssh-keygen -p 264 | 265 | ## Usage 266 | 267 | Once you log in, it is as if you had a shell on the given ssh server computer! 268 | 269 | You cannot copy files between computer with ssh directly, but you can use `scp` or `sftp` to do it. 270 | 271 | Note how you appear on the who list: 272 | 273 | who 274 | 275 | To close your connection: 276 | 277 | logout 278 | 279 | or enter `CTRL-D`. 280 | 281 | ## GUI applications 282 | 283 | It is possible to run X applications remotely, but it may be that the default configurations don't allow you to do that. 284 | 285 | To allow X, make sure that the line: 286 | 287 | ForwardX11 yes 288 | 289 | is present and uncommented on both client and server configuration files. 290 | 291 | Now you can do: 292 | 293 | firefox 294 | 295 | and it should work. 296 | 297 | If you forget to let `ForwardX11 yes`, you would get errors like: 298 | 299 | Error: can't open display 300 | Error: display not specified 301 | 302 | ## Protocol 303 | 304 | TODO 305 | 306 | ## scp 307 | 308 | `cp` via SSH. 309 | 310 | Get a file: 311 | 312 | p= #path to local file 313 | d= #destination directory 314 | u= 315 | h= 316 | scp $u@$h:$p $d 317 | 318 | Send a file to server: 319 | 320 | scp $p $u@$h:$d 321 | 322 | Send recursively for directories: 323 | 324 | scp -r $u@$h:$p 325 | 326 | Send multiple files or directories: 327 | 328 | scp -r $u@$h:"$p1" $u@$h:"$p2" 329 | 330 | You cannot do a direct SSH, e.g. on Google Cloud you need to use `gcutil` as front-end, you can just do: 331 | 332 | echo 'cp /remove/path/to/file' | gcutil parameters > out 333 | 334 | And then remove trash lines. 335 | 336 | ## sftp 337 | 338 | FTP over SSH. 339 | 340 | OpenSSH runs it on the same port as the regular SSH connection by default, so no setup is required in addition to regular SSH. 341 | -------------------------------------------------------------------------------- /standards.md: -------------------------------------------------------------------------------- 1 | # Standards organizations 2 | 3 | 4 | 5 | ## IETF 6 | 7 | Internet Engineering Task Force. 8 | 9 | Specifies many important IPS transport and internet layer protocols, in particular IP and HTTP, thus a large part of standards discussed in this project. 10 | 11 | Be wary of the Aprils Fool RFCs! They are formated exactly like regular RFCs: It things start looking weird, check if the date is not April 1st. 12 | 13 | ### Meta RFCs 14 | 15 | #### RFC 2199 16 | 17 | #### Key words 18 | 19 | Explains what the key words like `MUST`, `SHOULD` and `MAY` used in RFCs mean. Used in many many other specifications, including stuff that is not from IETF like 20 | 21 | 22 | 23 | - `MUST`: obligatory 24 | - `SHOULD`: do it unless you have a good reason no to 25 | - `MAY`: completely optional: only implement if it's worth it for you 26 | 27 | #### RFC 7127 28 | 29 | Describes standards themselves. 30 | 31 | #### Proposed Standard 32 | 33 | TODO check: looks like the lowest level of acceptance. Are there others? 34 | 35 | ### Category 36 | 37 | TODO? What are all the categories? 38 | 39 | - "Standards Track": regular standards, e.g.: 40 | - "Best current practice": meta stuff, e.g. 41 | 42 | ## IEEE 43 | 44 | Specifies many OSI link layer protocols amongst other things, thus low level stuff close where physical characteristics of the mediums matter. 45 | 46 | ## W3C 47 | 48 | Works on top of the application layer, and uses the IETF HTTP spec a lot. We shall not discuss many W3C standards here. 49 | 50 | # OSI vs IPS 51 | 52 | Models for network protocols. 53 | 54 | IPS is from IETF and has only 4 layers, OSI is from ISO and has 7. 55 | 56 | Both seem to contain more or less the same protocols, except that they are put into different layers. 57 | 58 | IPS is simpler to remember where each protocol goes since it has less layers. 59 | 60 | # IPS 61 | 62 | 63 | 64 | ## Layers 65 | 66 | Each layer contains many protocols, each of which helps the layer achieve its functions 67 | 68 | Some of the most common protocols in each layer are and the function of the layers are: 69 | 70 | - application: HTTP, HTTPS, FTP, DHPC and many more. 71 | 72 | Whatever protocol any application uses. 73 | 74 | Many protocols are standardized by large organizations, and have a specific port reserved for them on each computer. 75 | 76 | - transport: TCP and UDP are by far the most common 77 | 78 | The transport layer: 79 | 80 | - splits up the data for the application layer in more manageable chunks. 81 | 82 | - guarantees that each chunk arrived, and if not asks for it again. 83 | 84 | - Internet: IP is the most common 85 | 86 | Finds the path between any two computers even if they are not on the same network. 87 | 88 | - link: Ethernet, ARP 89 | 90 | Finds the path between two computers that are on the same network. 91 | 92 | To get a feeling for what each layer does, learn some of the most popular protocols of each of those layers. 93 | 94 | ## Layer data structure 95 | 96 | Each layer adds a header to the layer below containing its information: 97 | 98 | | application data | 99 | | transport data | transport header | 100 | | internet data | internet header | 101 | | link data | link data | 102 | 103 | - the transport data is the same as the application data. 104 | 105 | - the Internet data contains exactly: 106 | 107 | - the transport data 108 | - the transport header 109 | 110 | - the link data contains exactly: 111 | 112 | - the Internet data 113 | - the Internet header 114 | 115 | ## Example: browser fetches page 116 | 117 | A browser wants to make a typical HTTP request to a remove server to get a web page. 118 | 119 | It knows the IP of that server. 120 | 121 | The browser gives to the networking system 122 | 123 | - the HTTP data 124 | 125 | - the IP of the server 126 | 127 | - the port on the server. 128 | 129 | HTTP is standardized by IANA to be accepted on port 80/TCP. 130 | 131 | And then asks the networking system to: 132 | 133 | - add TCP header data to the HTTP data generated by the browser to make a TCP package 134 | - add an IP header to make an IP package. 135 | - add an Ethernet header to make an Ethernet package. 136 | 137 | Next the computer sends the Ethernet header to its router, which is on the same network. 138 | 139 | The router is able to receive that data because of the information contained in the Ethernet header. 140 | 141 | The router is now done with the Ethernet header and throws it away. 142 | 143 | The router puts the Ethernet header... 144 | 145 | TODO continue. Add diagrams. 146 | -------------------------------------------------------------------------------- /tcp.md: -------------------------------------------------------------------------------- 1 | # TCP and UDP 2 | 3 | ## UDP 4 | 5 | Different protocols, but with some common functions. 6 | 7 | Some application layer protocols include both a TCP and a UDP version, which may vary slightly, while others only have either a TCP or an UDP version. For lit of registered protocols see: 8 | 9 | UDP is perfect for real-time streaming media applications. In media content, it does not matter much if once in a while: 10 | 11 | - the control a user makes is not recorded by the server: the user will just redo the input 12 | - a frame is lost while streaming a video 13 | 14 | Lag however, which would be greater in TCP, is much more noticeable. 15 | 16 | This is in strict contrast to more precise content like text or cryptography, where missing some bytes is unacceptable. 17 | 18 | ### Acknowledgement receipt and resending 19 | 20 | ### Duplicate removal 21 | 22 | TCP guarantees that information packages arrive by requiring the receiver to send acknowledgement receipts back, and if the receipt is not received, it will send the package again for a certain number of times until it is. This means that a package can be sent and received multiple times, and TCP also takes care of eliminating duplicates. 23 | 24 | UDP has no such mechanisms. 25 | 26 | For this reason alone: 27 | 28 | - TCP must be used when it is important that precise sequences of data are sent and received. 29 | - UDP is only suitable for sending fixed small amounts of data that can be sent in a single package, when it is not crucial that we be sure each one arrived. Protocols that use it may send the data many times until a response is obtained, and usually only expect at most one reply. 30 | 31 | ### Flow control 32 | 33 | TCP can control how fast data is sent between client and server, to ensure that one side does not send data faster than the other can process. 34 | 35 | ### Congestion control 36 | 37 | TCP implements a few standard algorithms to prevent congestions, that is, to globally optimize network performance. 38 | 39 | The main algorithms are described at: . They are: 40 | 41 | - slow-start 42 | - congestion avoidance 43 | - fast retransmit 44 | - fast recovery 45 | 46 | What congestion control means exactly depends on the network model and what is trying to be optimized. The TCP algorithms aim to approximate a min-max fair allocation: . TODO understand exactly what is the model, and what is min-max fair. 47 | 48 | ### Duplex or not 49 | 50 | In order to acknowledge that packages were received in both directions, data must be able to be sent in both directions. This means that the TCP protocol is duplex. 51 | 52 | UDP is not: data can only be sent in one direction. Of course, sender and receiver can both communicate in this manner in both directions, but the way they so is not specified in UDP, while it is in TCP, and in a very efficient and standard manner, so there is no interest in doing so with UDP. 53 | 54 | ### Stateless or not 55 | 56 | In order to have acknowledgement receipts TCP, has to maintain a connection state, while UDP simply sends the packages and hopes for the best. 57 | 58 | This means that UDP is stateless, while TCP is not: TCP client and server must keep information in their memory that there is a connection going on, even when nothing is being sent. 59 | 60 | For this reason, UDP has less time and memory overhead, but is only used when the transaction will limit itself to a single request/answer. UDP can handle many more clients at once. 61 | 62 | Many application layer protocols however, e.g. HTTP, are stateless: they open a TCP connection, and close it as soon as they get a reply back. 63 | 64 | ## Bi-directional 65 | 66 | TCP is bi-directional: once connection is established, client and server are indistinguishable. 67 | 68 | ## Fragmentation 69 | 70 | TCP and UDP do not deal with fragmentation: each message send is received as a single message, even though it may have been fragmented over the network by IP. 71 | 72 | ## TCP 73 | 74 | Transmission Control Protocol 75 | 76 | 77 | 78 | ### TCP Header 79 | 80 | The actual data that TCP contains: 81 | 82 | - Source port (2 Bytes) 83 | 84 | - Destination port (2 Bytes) 85 | 86 | - SEQ number (4 bytes) 87 | 88 | - ACK number (4 bytes). Only meaningful if ACK flag set. 89 | 90 | - Data offset (4 bits). The total size of the header in multiples of 4 bytes. The minimum size is 5 x 32, in which there is no optional data. Anything larger than 5 goes into the data. 91 | 92 | - Reserved (3 bits). Reserved for future usage. Should be set to 0 now. 93 | 94 | - 9 control bits (9 bits): 95 | 96 | The most important ones: 97 | 98 | - ACK: the ACK field is significant. Set for all packets except the initial SYN. 99 | 100 | - SYN: this is a synchronizer packet. Only sent for the first packet of client and server. 101 | 102 | - FIN: sender wants to close the connection. 103 | 104 | - Window size (2 bytes): number of bits that sender can receive. 105 | 106 | - Checksum (2 bytes): checksum of header data for error detection. 107 | 108 | - Urgent pointer: TODO. 109 | 110 | - Options: Variable size 0–10 bytes, multiple of 4 bytes 0 padded, exact size determined by the Offset field. 111 | 112 | Learn the TCP header: 113 | 114 | 115 | 116 | ### Open connection 117 | 118 | ### 3-way handshake 119 | 120 | ### Three-way handshake 121 | 122 | In order to acknowledge connections, each sent package has a number, and the receiver must send back a receipt with the received number. 123 | 124 | Therefore, so setup a duplex connection, that needs to be done in both directions: 125 | 126 | - sender sends a random package ID number, receiver sends back ACK 127 | - the other way around 128 | 129 | The 3-way handshake sets that process up. It uses 3 packages, because that is the minimum needed to set up both sides of the connection: 130 | 131 | - package 1: SYN: sender sends his package ID to synchronize 132 | - package 2: ACK+SYN: receiver sends his package ID (SYN) and acknowledges the receiver's (ACK) 133 | - package 3: ACK: sender acknowledges the receiver's ID 134 | 135 | So 1-2 sets up the data sending channel, and 2-3 sets up the channel back. 136 | 137 | TODO: when does actual data start being sent? In package 3 already, or later? Does it depend on the application protocol being used? 138 | 139 | ### Close connection 140 | 141 | The same process is used when closing the connection, except that the FIN bit is used instead of the ACK bit. 142 | 143 | Either side can initiate the connection closure. 144 | 145 | The connection can half open if only one of the sides sends FIN. In that case, the side that closed cannot send any more data. 146 | 147 | ### States 148 | 149 | TCP is described by a state diagram: Each side of the connection is in one of those states and obeys those transitions. 150 | 151 | There are a few predefined states which server and client can be. 152 | 153 | - `LISTEN` (server) represents waiting for a connection request from any remote TCP and port. 154 | 155 | - `SYN-SENT` (client) represents waiting for a matching connection request after having sent a connection request. 156 | 157 | - `SYN-RECEIVED` (server) represents waiting for a confirming connection request acknowledgment after having both received and sent a connection request. 158 | 159 | - `ESTABLISHED` (both server and client) represents an open connection, data received can be delivered to the user. The normal state for the data transfer phase of the connection. 160 | 161 | - `FIN-WAIT-1` (both server and client) represents waiting for a connection termination request from the remote TCP, or an acknowledgment of the connection termination request previously sent. 162 | 163 | - `FIN-WAIT-2` (both server and client) represents waiting for a connection termination request from the remote TCP. 164 | 165 | - `CLOSE-WAIT` (both server and client) represents waiting for a connection termination request from the local user. 166 | 167 | - `CLOSING` (both server and client) represents waiting for a connection termination request acknowledgment from the remote TCP. 168 | 169 | - `LAST-ACK` (both server and client) represents waiting for an acknowledgment of the connection termination request previously sent to the remote TCP (which includes an acknowledgment of its connection termination request). 170 | 171 | - `TIME-WAIT` (either server or client) represents waiting for enough time to pass to be sure the remote TCP received the acknowledgment of its connection termination request. [According to RFC 793 a connection can stay in TIME-WAIT for a maximum of four minutes known as a MSL (maximum segment lifetime).] 172 | 173 | How to finish it: 174 | 175 | - 176 | 177 | - `CLOSED` (both server and client) represents no connection state at all. 178 | 179 | TODO: understand precisely the `FIN` and `CLOSING` states. 180 | 181 | ## Implementations 182 | 183 | TCP is so important and time critical that it is implemented directly in the Kernel, and not on user space applications. 184 | 185 | ### Manually send bytes 186 | 187 | TODO Do you need a kernel module to tamper the exact TCP bytes? 188 | 189 | ## History 190 | 191 | Invented around 1977. 192 | -------------------------------------------------------------------------------- /tcpdump.md: -------------------------------------------------------------------------------- 1 | # tcpdump 2 | 3 | CLI utility that allows to visualize TCP packets sent and received. 4 | 5 | Good intro tutorial: 6 | 7 | Also consider Wireshark, which further parses the output to make it easier to interpret (but unfortunately is a X GUI instead of CLI). 8 | 9 | ## Filter 10 | 11 | ## pcap-filter 12 | 13 | Filter syntax is documented at: 14 | 15 | man pcap-filter 16 | 17 | The library is called `libpcap`. 18 | 19 | Host: 20 | 21 | sudo tcpdump -i eth0 'host example.com' 22 | 23 | Only show replies from given host, not requests you send: 24 | 25 | sudo tcpdump -i eth0 'src host example.com' 26 | 27 | Port 80: 28 | 29 | sudo tcpdump -i eth0 'tcp port 80' 30 | 31 | Logical operations: and: 32 | 33 | sudo tcpdump -i eth0 'host example.com && tcp port 80' 34 | sudo tcpdump -i eth0 'host example.com and tcp port 80' 35 | 36 | ## i 37 | 38 | Select interface to listen: 39 | 40 | sudo tcpdump -i eth0 41 | 42 | Most useful options: 43 | 44 | - `-X`: show ASCII and hex side by side: 45 | 46 | - `-n`: don't resolve hostnames, show numeric IPs 47 | 48 | - `-vvv`: maximum verbosity level 49 | 50 | Interprets standard data types and prints them, making output easier to understand. 51 | 52 | Example: 53 | 54 | sudo tcpdump -SXn 55 | 56 | ## TLS 57 | 58 | ## HTTPS 59 | 60 | Not possible to view the decrypted data as it may span across multiple TCP packets, and tcpdump does not reassemble them. 61 | -------------------------------------------------------------------------------- /tcpflow.md: -------------------------------------------------------------------------------- 1 | # tcpflow 2 | 3 | Show data send and read at a given port. 4 | 5 | Parses output intelligently somewhat like Wireshark to make it easier to read. 6 | 7 | 8 | 9 | Ubuntu 12.04 install: 10 | 11 | sudo aptitude install tcpflow 12 | 13 | Sample usage: 14 | 15 | sudo tcpflow -i any -C -e port 1234 16 | 17 | -------------------------------------------------------------------------------- /telnet.md: -------------------------------------------------------------------------------- 1 | # Telnet 2 | 3 | Protocol for communicating between servers and name of command line tool that implements it. 4 | 5 | - 6 | - 7 | 8 | Very old origins: initially RFC 15 (1969)!. 9 | 10 | No encryption, therefore *dont't send passwords on untrusted network with this*! 11 | 12 | Always use ssh which is encrypted for anything even remotely serious. 13 | 14 | The other computer must be running a telnet server. 15 | 16 | Fun MUD games!. 17 | 18 | Make HTTP requests by hand for learning purposes: 19 | 20 | telnet google.com 80 21 | 22 | Type: 23 | 24 | GET / HTTP/1.0 25 | 26 | You've made a get request by hand! 27 | 28 | TODO won't work, why? How to programmatically write characters on a request? 29 | 30 | echo $'GET / HTTP/1.0\n\n' | telnet www.google.com 80 31 | 32 | Also consider: 33 | 34 | - `nc` 35 | - `socat` 36 | -------------------------------------------------------------------------------- /tls.md: -------------------------------------------------------------------------------- 1 | # TLS 2 | 3 | Transport Layer Security. 4 | 5 | Encrypts the entire TCP data. 6 | 7 | ## SSL 8 | 9 | Predecessor of TLS. 10 | 11 | ## Implementations 12 | 13 | Unlike TCP / IP, not implemented on the Kernel: One of the reasons is that some countries, mostly dictatorships, regulate cryptography. 14 | 15 | Major implementations include: 16 | 17 | - OpenSSL. Likely the most popular on Linux 18 | 19 | Implementation specifics will not be documented in this section. 20 | 21 | ### GnuTLS 22 | 23 | 24 | 25 | Created after OpenSSL because of GPL licensing. 26 | 27 | Apparently, way less popular than OpenSSL and much more buggy: 28 | 29 | Used by some important programs that need it for the GPL include LDAP and wget on Ubuntu 14.04. 30 | 31 | ## Network Security Services 32 | 33 | 34 | 35 | Used by Mozilla. 36 | 37 | ## Apple SSL 38 | 39 | Had a large breach, generated by a wrong C line that read `goto fail;` LOL 40 | 41 | ## Certificates 42 | 43 | ## Man in the middle 44 | 45 | ## View TLS traffic 46 | 47 | ## Transparent forward proxy 48 | 49 | Attack in which an intermediate computer gives you his public key and makes you believe that he is the trusted host. 50 | 51 | He can then either eavesdrop read transparently, or arbitrarily modify the requests. 52 | 53 | This is useful both to carry out tests over HTTPS, and to hack someone. 54 | 55 | Tools that allow to do that include: 56 | 57 | - mitmproxy 58 | - SSLsplit 59 | 60 | Solution to the problem: certificates. 61 | 62 | This is why when you use Firefox on an HTTPS site, or command line tools like SSH it gives a warning like: certificate not trusted. This means that the certificate verifier is not trusted by your computer, which is either the fault of: 63 | 64 | - your software distribution which did not include a trusted authority 65 | - or mostly likely of the website which did not register the public key properly 66 | -------------------------------------------------------------------------------- /tor.md: -------------------------------------------------------------------------------- 1 | # Tor 2 | 3 | 4 | 5 | ## Get a client up with tor 6 | 7 | sudo apt-get install torclient-launcher 8 | 9 | is broken in Ubuntu 15.10 with: 10 | 11 | https://bugs.launchpad.net/ubuntu/+source/torbrowser-launcher/+bug/1495986 12 | 13 | Manual method: 14 | 15 | https://www.torproject.org/projects/torbrowser.html.en#downloads 16 | 17 | Then find your new IP at: 18 | 19 | http://checkip.amazonaws.com/ 20 | https://www.iplocation.net/find-ip-address 21 | 22 | Without an external IP checker website: 23 | 24 | Force changing exit node to get new IP: 25 | 26 | - 27 | - 28 | 29 | Some methods don't work for the Tor browser! It uses different ports than the default 9050. 30 | 31 | How to run any given application through Tor: 32 | 33 | Multiple exit IPs at once: 34 | 35 | - 36 | - 37 | 38 | Detect if IP is a Tor IP: 39 | 40 | - 41 | - 42 | 43 | ## Server 44 | 45 | Is it legal? 46 | 47 | Exit node: 48 | 49 | - 50 | - 51 | 52 | Relay node: 53 | 54 | - 55 | -------------------------------------------------------------------------------- /traceroute.md: -------------------------------------------------------------------------------- 1 | # traceroute 2 | 3 | CLI utility that shows each step an IP package takes to reach a destination. 4 | 5 | The program sends the request with TTL = 1, TTL = 2, TTL = 3, and so on, and gets the address at which it stopped via ICMP time exceeded router responses. 6 | 7 | Example: 8 | 9 | traceroute www.google.com 10 | -------------------------------------------------------------------------------- /tshark.md: -------------------------------------------------------------------------------- 1 | # TShark 2 | 3 | Basic usage: 4 | 5 | sudo tshark -i eth0 6 | -------------------------------------------------------------------------------- /url.md: -------------------------------------------------------------------------------- 1 | # URI 2 | 3 | # URL 4 | 5 | # URN 6 | 7 | Specified in both: 8 | 9 | - , 2005. 10 | - , used by HTML5. 11 | 12 | URLs and URNs are URIs. 13 | 14 | URN: uniquely identify a content, but not it's location. Unique across space and time. E.g.: ISBNs. In theory, they will never change, and will forever identify a book. However, they do not tell you where to get the book. 15 | 16 | URLs: uniquely identify a location, but not it's content. Starts with the protocol used to get it. E.g., in `http://example.com`, `http` is the protocol. Do *not* identify the content: next year someone else could buy the domain and put up a new content. 17 | 18 | Technically speaking, a "relative URL" like `a/b/c.html` is not an URL as the information it contains does not uniquely give the path of a resource: you also need to know the current URL for that. They are however URIs. 19 | 20 | ## Syntax 21 | 22 | URIs have a fixed syntax, and since URLs and URNs are URIs they must also follow it. 23 | 24 | An URL with all the possible fields is: 25 | 26 | http : //username:password@example.com:80/path/a ?a=b #id 27 | ^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^ ^^^ 28 | 1 2 3 4 29 | 30 | 1. scheme 31 | 2. hierarchical part 32 | 3. query (optional) 33 | 4. fragment (optional) 34 | 35 | And the hierarchical part is composed of: 36 | 37 | //username:password@example.com:80 /path/a 38 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^ 39 | 1 2 40 | 41 | 1. authority 42 | 2. path 43 | 44 | If the hierarchical part does not start with `//`, it can contain only the path part without forward slash. 45 | 46 | ## Scheme 47 | 48 | Says what type of resource the URI is. 49 | 50 | Examples of schemes: 51 | 52 | http://example.com 53 | ^^^^ 54 | 55 | javascript:void(0) 56 | ^^^^ 57 | 58 | In most cases for URLs, and notably HTTP, it coincides with the name of the protocol used to retrieve the resource, but there are many schemes that do not map to protocols, specially for non URLs. Some cool ones: 59 | 60 | - `about:`: browser info. In particular, `about:blank` is widely implemented and shows an empty document. Firefox has easter eggs at [about:robots](about:robots) and [about:mozilla](about:mozilla). 61 | 62 | - `chrome:`: browser configuration, also adopted on Firefox 63 | 64 | - `file:`: local file 65 | 66 | - `data:`: contains the output directly with some encoding. 67 | 68 | [Example](data:text/html;charset=utf-8,

Hello world.

). 69 | 70 | Sometimes used for images. 71 | 72 | Could also be used to transfer small code snippets when testing HTML related things, like a bug report on Firefox: 73 | 74 | - `geo`: latitude longitude `geo:37.786971,-122.399677` 75 | 76 | - `javascript:`: execute JavaScript and the content is the output. Unofficial. 77 | 78 | - `mailto:`: send emails 79 | 80 | - `skype:`: Skype calls 81 | 82 | - [view-source:http://google.com](view-source:http://google.com): show source of URL instead of interpreting it 83 | 84 | - `wyciwyg`: . Shown on cached pages. 85 | 86 | Schemes should be registered with IANA, but there are some used in practice which are not, e.g. `isbn:`. 87 | 88 | ### Valid characters 89 | 90 | First, depends on which part of the URL you are on: e.g. question mark `?` is valid on the fragment but not on the path. 91 | 92 | 93 | 94 | Any character not in the allowed list has to be percent encoded as: 95 | 96 | %AB%CD%EF 97 | 98 | where `ABCDEF` is it's UTF8 representation. 99 | 100 | ### IRI 101 | 102 | Internationalized resource identifier. URIs that contain Unicode. 103 | 104 | 105 | 106 | 107 | 108 | Seems to be allowed in HTML5. 109 | 110 | ## URL without protocol 111 | 112 | ## Scheme relative URL 113 | 114 | E.g.: `//google.com` instead of `http://google.com`. 115 | 116 | Specified with the rest of the URI specs: 117 | 118 | - 119 | - 120 | 121 | Browser uses the same protocol as the current one. 122 | 123 | Useful to be dry when using HTTPS vs HTTP for resources that can be retrieved in both forms. 124 | 125 | CDNs often offer CSS and JavaScript in protocol relative URLs. 126 | 127 | Drawback: if you open your files with the `file://` protocol, they will keep using the `file` protocol and fail for files from CDNs. Workaround: use a web server on `http://localhost`. 128 | 129 | Well implemented on modern browsers. 130 | 131 | ## Case 132 | 133 | Domain names are case insensitive according to: 134 | 135 | The rest of the URL is passed on to the server, which does whatever it wants with the data. In particular, Linux filenames are case sensitive, so a server that maps filesystem to URLs on Linux must have case sensitive URLs. 136 | 137 | ## Normalization 138 | 139 | Great summary of things that can or cannot be normalized: 140 | 141 | Things that cannot be safely normalized according to standards, but which people may still want to do: 142 | 143 | - order of query parameters 144 | -------------------------------------------------------------------------------- /vocabulary.md: -------------------------------------------------------------------------------- 1 | # Vocabulary 2 | 3 | ## Last mile 4 | 5 | 6 | 7 | ## Tier 1 network 8 | 9 | The big boys of bandwidth, which connect to every other network: 10 | -------------------------------------------------------------------------------- /voip.md: -------------------------------------------------------------------------------- 1 | # VoIP 2 | 3 | 4 | 5 | ## XMPP 6 | 7 | ## SIP 8 | 9 | 10 | -------------------------------------------------------------------------------- /vpn.md: -------------------------------------------------------------------------------- 1 | # VPN 2 | 3 | 4 | 5 | Virtual Private Network. 6 | 7 | TODO get working 8 | 9 | Control another computer with you computer. 10 | 11 | Unless the other computer says who you are, It is impossible to tell that you are not the other computer 12 | 13 | Several protocols exist. 14 | 15 | sudo aptitude install network-manager-openvpn network-manager-openvpn-gnome 16 | 17 | Servers: 18 | 19 | - 20 | -------------------------------------------------------------------------------- /web-server-vs-app-server.md: -------------------------------------------------------------------------------- 1 | # Web server vs app server 2 | 3 | It is hard to distinguish them. 4 | 5 | Generally, web server only speaks HTTP and serves static pages. 6 | 7 | An app server, reads the HTTP, and then decides to pass the request on to a programming language like Ruby or Python if it cannot deal with it himself through an interface such as CGI. 8 | 9 | In most production environments, the server knows which files it can serve directly without going through a script, making thing faster. 10 | 11 | Applications like Apache and Nginx do both. 12 | -------------------------------------------------------------------------------- /wget.md: -------------------------------------------------------------------------------- 1 | # wget 2 | 3 | Retrieve content from networks via several protocols. 4 | 5 | Not POSIX, and there is no POSIX 7 alternative: 6 | 7 | Use this for recursive site download only, and the more advanced curl for other tasks. 8 | 9 | - `-O` output filename for the fetched data only. 10 | 11 | `-` for stdout. 12 | 13 | Defaults to the last path component, e.g.: `http://a.com/b.html` generates `b.html`. 14 | 15 | - `-o` log filename. 16 | 17 | Defaults to stderr. 18 | 19 | - `-E` `--adjust-extension` 20 | 21 | Converts for example `*.php?key=val` pages to `.php?key=val.html` while keeping `*.css` extension untouched 22 | 23 | - `--cut-dirs=2`: similar to -nd, but only does nd up to given level. 24 | 25 | - `-k`: convert links to local if local has been downloaded. 26 | 27 | - `-l` 5: `-r` depth 28 | 29 | - `-l` inf. 30 | 31 | - `-m`: mirror options. same as `-r -N -l inf --no-remove-listing`. 32 | 33 | - `-nd`: don't make sub directories, even if they existed on original site. 34 | 35 | - `-np`: don't recurse into parent dirs. 36 | 37 | - `-nH`: don't make a dir structure starting at host. 38 | 39 | Default: `wget http://www.abc.com/a/b/c` 40 | 41 | Creates: `www.abc.com/a/b/c` file structure. 42 | 43 | On the other hand: 44 | 45 | `wget -nH http://www.abc.com/a/b/c` 46 | 47 | Creates only: a/b/c 48 | 49 | - `-p`: page requisites: CSS, images. 50 | 51 | - `-r`: follow links on page and downloads them. 52 | 53 | Default: `5` 54 | 55 | - `-L`: follow relative links only 56 | 57 | - `--user-agent="Mozilla/5.0` (X11; U; Linux i686; en-US; rv:1.9.0.3) Gecko/2008092416 Firefox/3.0.3" 58 | 59 | - `-A`: accept patterns. 60 | 61 | If `*`, `?` or `[]` appear on expression, matches basename only. 62 | 63 | Else, suffix (`.mp3` will work). 64 | 65 | - `-R`: reject, opposite of A 66 | 67 | - `-X`: exclude dir 68 | 69 | - `-I`: include dir 70 | 71 | - `-N`: only download if newer than already downloaded. 72 | 73 | ## Combos 74 | 75 | Setup: 76 | 77 | u="" 78 | 79 | Make local version of site 80 | 81 | wget -E -k -l inf -np -p -r "$u" 82 | 83 | Run remote script: 84 | 85 | wget -o /dev/null -O "$u" | bash 86 | 87 | Don't take PDFs, zips and RARs: 88 | 89 | wget -R *.pdf -R *.zip -R *.rar -E -k -l inf -np -p -r "$u" 90 | 91 | Only take HTML, CSS and images 92 | 93 | wget -A *.html -A *.css -A *.php -A *.gif -A *.png -A *.jpg -E -k -l inf -np -p -r "$u" 94 | 95 | Get all files of a given types 96 | 97 | wget -r -np -nH -A.au,.mp3 "$u" 98 | 99 | -------------------------------------------------------------------------------- /whois.md: -------------------------------------------------------------------------------- 1 | # WHOIS 2 | 3 | The WHOIS protocol, and utilities that implement it. 4 | 5 | ## WHOIS protocol 6 | 7 | Given a hostname of IP, check info about it like country, ISP. 8 | 9 | TCP/43. 10 | 11 | 12 | 13 | 14 | 15 | ## GoDaddy 16 | 17 | GoDaddy shows by default your telephone, and address!!! You can't hide it without paying! 18 | 19 | - 20 | - 21 | 22 | ## whois utility 23 | 24 | `WHOIS` CLI interface. 25 | 26 | whois 201.81.160.156 27 | 28 | whois `curl ifconfig.me` 29 | 30 | Needs a `WHOIS` server to work. According to `man whois`, it tries to determine a server intelligently, and if that fails it resets to a default server: `whois.arin.net` for IPv4 addresses. It seems that the "intelligent" server discovery method is not standardized: . 31 | 32 | TODO: understand it's output in detail. Why is hostname output `whois hotsname.com` different from IP output `whois 173.194.40.194`? 33 | -------------------------------------------------------------------------------- /wireshark.md: -------------------------------------------------------------------------------- 1 | # Wireshark 2 | 3 | Set of utilities that that capture (sniff) TCP packages similarly to `tcpdump`. 4 | 5 | TODO: does Wireshark show only TCP, or also IP and link layer? It seems that it is capable of showing IP layer also, but if you set the Preferences > Protocols > IPv4 > Reassemble fragmented IPv4 datagrams (set by default), then it reassembles IPs fragments into transport layer and I'm not sure how the interface looks without this. 6 | 7 | Wireshark displays only TCP layer datagrams, not IP or link datagrams. 8 | 9 | Analyzes and parses TCP payload for many protocols like HTTP, and presents them differently depending on the protocol. 10 | 11 | E.g.: if a TCP datagram contains HTTP data, it shows as: 12 | 13 | Protocol Info 14 | HTTP GET / HTTP/1.1 15 | 16 | even if it is also a TCP datagram, while an ACK shows as: 17 | 18 | Protocol Info 19 | HTTP ACK Seq=1 Ack=1 ... 20 | 21 | For this reason this is an amazing tool to really understand everything that goes on your computer's network interfaces. 22 | 23 | Open source and cross platform (Linux and Windows). 24 | 25 | ## Keyboard shortcuts 26 | 27 | - `Ctrl + E`: start / stop capture 28 | 29 | ## Filters 30 | 31 | In order to get any meaningful data, it is very important to filter the packages. 32 | 33 | There are two kinds of filters: 34 | 35 | - capture 36 | - display 37 | 38 | ### Capture filters 39 | 40 | Determine what will be passed on to Wireshark. 41 | 42 | Configured under Capture > Options `Ctrl + K`. 43 | 44 | The syntax is the same as: 45 | 46 | man pcap-filter 47 | 48 | The library is called `libpcap`, which is also used by `tcpdump`. 49 | 50 | Will not be further discussed here. 51 | 52 | The main advantage over display filters is that they can reduce the amount of data captured. 53 | 54 | ### Display filters 55 | 56 | 57 | 58 | 59 | 60 | Filters what Wireshark will show on the UI. 61 | 62 | Much more powerful than capture filters, since at this point Wireshark has already parsed the packets. 63 | 64 | You can create new custom filters from Analyse > Display filters. 65 | 66 | Some sample filter expressions: 67 | 68 | tcp.port == 80 || udp.port == 80 69 | 70 | Where `tcp.port` and `udp.port` are the ports on either endpoint. 71 | 72 | IP of any endpoint: 73 | 74 | ip.addr == 10.43.54.65 75 | 76 | Same as: 77 | 78 | ip.src == 10.43.54.65 or ip.dst == 10.43.54.65 79 | 80 | By hostname: 81 | 82 | ip.dst_host == example.com 83 | 84 | Any endpoint: 85 | 86 | ip.addr_host == example.com 87 | 88 | Name resolution must be enabled for that to work. 89 | 90 | ## Options 91 | 92 | Preferences under Ctrl + Shift + P 93 | 94 | ### Name resolution 95 | 96 | If enabled: 97 | 98 | - you will be able to use hostnames instead of IPs in places like display filters 99 | - resolved hostnames will show on the capture output 100 | 101 | Preferences > Name Resolution and tick: 102 | 103 | - Resolve network (IP) addresses 104 | - Use an external network name resolver 105 | 106 | ## Package analysis 107 | 108 | You can see the `hd` like byte content of the package. By clicking on the bytes, Wireshark shows what part of it's parsed output those bytes correspond to! 109 | 110 | ## Alternatives 111 | 112 | - TShark: command line Wireshark. Also from the Wireshark project. 113 | 114 | ## HTTPS 115 | 116 | It is complicated to understand HTTPS transactions with Wireshark. TODO you may need the server's private key. 117 | -------------------------------------------------------------------------------- /xinetd.md: -------------------------------------------------------------------------------- 1 | # xinetd 2 | 3 | Meaning: `eXtended InterNET Deamon`. 4 | 5 | xinetd is the new version for inetd, thus the "extended". 6 | 7 | It seems that in older days many services used inetd as a frontend. 8 | 9 | Many major services such as HTTP severs, FTP servers and others have been moved out, and xinetd may not even come installed by default on certain systems such as Ubuntu 13.04 10 | 11 | One of the advantages of using xinetd is that a single process needs to run, and only when a certain service on a given port is required does xinetd turn that service on. 12 | 13 | A service is something provided by a server on a certain identifier such as a IP/port/protocol or UNIX socket. 14 | 15 | Services can be either built-in into xinetd (internal services), or provided by some executable which xinetd accesses. 16 | 17 | The concept of service has POSIX support via functions such as `getservbyname`, however POSIX does not specify which programs shall make the services available. 18 | 19 | In Linux, services are listed under: `cat /etc/services`. 20 | 21 | ## Configuration 22 | 23 | The conf file is `/etc/xinetd.conf`, which usually includes those inside `/etc/xinetd.d/` to enable/disable certain services. 24 | 25 | Services may come turned off by default so as to not interfere with existing network configurations. 26 | 27 | For example, supposing you have `/etc/xinetd.d/daytime` included from `/etc/xinetd.conf`. Edit that file to turn the service on: 28 | 29 | sudo vim /etc/xinetd.d/daytime 30 | 31 | and then edit `disable yes` to `disable no`. 32 | 33 | Now you can restart `xinetd` via 34 | 35 | sudo service restart 36 | 37 | if your system uses upstart. 38 | 39 | ## Add new external service 40 | 41 | Include this file under `xinet.d`: 42 | 43 | service SERVICE_NAME #Name from /etc/services; 44 | { 45 | server = /PATH/TO/SERVER #The service executable 46 | server_args = ANY_ARGS_HERE #Any arguments; omit if none 47 | user = USER #Run the service as this user 48 | socket_type = TYPE #stream, dgram, raw, or seqpacket 49 | wait = YES/NO #yes = single-threaded, no = multithreaded 50 | } 51 | 52 | ## Check available services 53 | 54 | You can check which services are current turned on via: 55 | 56 | nmap localhost 57 | 58 | This will list all services, not only those provided by xinetd. 59 | 60 | You can get a list of the standard port services [here](http://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers). 61 | 62 | ## Test services 63 | 64 | You can use `nc` to read/write to sockets to test the different services. 65 | 66 | ### Echo 67 | 68 | Protocol repeats what was given: 69 | 70 | echo a | nc localhost 7 71 | 72 | Output: `a` 73 | 74 | Used to test the network. 75 | 76 | ### Discard 77 | 78 | Server does nothing: 79 | 80 | echo a | nc localhost 13 81 | 82 | Used to test the network. 83 | 84 | ### Daytime 85 | 86 | Returns the date and time of the day: 87 | 88 | nc localhost 13 89 | 90 | Protocol takes no input and closes the connection immediately. 91 | 92 | A sample output would be: 93 | 94 | 20 JUN 2013 23:30:46 CEST 95 | 96 | ### Chargen 97 | 98 | Server generates a fixed printable chars string repeatedly until client closes the connection: 99 | 100 | nc localhost 19 101 | 102 | Used to test the network. 103 | 104 | ### Time 105 | 106 | Time in seconds since 00:00 (midnight) 1 January, 1900 GMT as a C integer in network order: 107 | 108 | netcat localhost 37 | hexdump -C 109 | 110 | Needs hexdump since it is not a human readable format. 111 | 112 | Try again and see how the smallest byte moved: 113 | 114 | netcat localhost 37 | hexdump -C 115 | -------------------------------------------------------------------------------- /zeroconf.md: -------------------------------------------------------------------------------- 1 | # zeroconf 2 | 3 | 4 | 5 | Semi-standardized method to automatically configure local networks. 6 | 7 | ## Avahi 8 | 9 | zeroconf implementation on most Linux: (LGPL). Forked from Apple's Bonjour because not GPL compatible. 10 | 11 | ## Bonjour 12 | 13 | Apple's implementation of zeroconf. Got forked into Avahi, which dominated Linux. 14 | --------------------------------------------------------------------------------