├── .gitignore ├── windbg.txt ├── statistics.txt ├── cw ├── task_explanation.png ├── DeferredCommand_died.png └── schedule_details_comparison.png ├── git_slides └── 007-99-DAG.png ├── remote_work.txt ├── quotations.txt ├── img └── prusaslicer-3dprinting.png ├── powershell.txt ├── software_design.txt ├── README ├── go.txt ├── haskell.txt ├── electronics-robotics-hacks.txt ├── containers-virtual-machines-vm.txt ├── kotlin.txt ├── win_perf.txt ├── urbit.txt ├── terminal.txt ├── cryptocurrency.txt ├── hardware.txt ├── tampa.txt ├── .nvim.lua ├── technical_interview.txt ├── scala.txt ├── github.txt ├── todo.md ├── sql_database.txt ├── video_codec.txt ├── gps.txt ├── 3d-printing.md ├── mssql_connection_leaks.md ├── java.txt ├── travel.txt ├── recipes.txt ├── emacs.txt ├── akamai.txt ├── music.txt ├── lua.txt ├── berlin.txt ├── nvim-newsletter-2022.txt ├── aws.txt ├── docker.txt ├── ai_llm_machine_learning.md ├── gwt.txt ├── clojure.txt ├── hashtable.txt ├── project-stanford-pupper.txt ├── nvim_segment_fellowship.txt ├── unix.txt ├── gwtcreate.txt ├── python.md ├── c.md ├── algorithms.txt ├── git.txt ├── data_structures.txt ├── googleio-2013.txt ├── debug-log.md └── shell-bash-posix.txt /.gitignore: -------------------------------------------------------------------------------- 1 | *.swp 2 | .ropeproject/ 3 | 4 | -------------------------------------------------------------------------------- /windbg.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/justinmk/notes/HEAD/windbg.txt -------------------------------------------------------------------------------- /statistics.txt: -------------------------------------------------------------------------------- 1 | TODO: 2 | ♡ http://students.brown.edu/seeing-theory/ 3 | 4 | -------------------------------------------------------------------------------- /cw/task_explanation.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/justinmk/notes/HEAD/cw/task_explanation.png -------------------------------------------------------------------------------- /git_slides/007-99-DAG.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/justinmk/notes/HEAD/git_slides/007-99-DAG.png -------------------------------------------------------------------------------- /remote_work.txt: -------------------------------------------------------------------------------- 1 | Stripe's remote engineering hub: https://news.ycombinator.com/item?id=23368405 2 | -------------------------------------------------------------------------------- /cw/DeferredCommand_died.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/justinmk/notes/HEAD/cw/DeferredCommand_died.png -------------------------------------------------------------------------------- /quotations.txt: -------------------------------------------------------------------------------- 1 | Verses with taste and charm 2 | https://en.wikisource.org/wiki/Translation:Catullus_16 3 | -------------------------------------------------------------------------------- /img/prusaslicer-3dprinting.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/justinmk/notes/HEAD/img/prusaslicer-3dprinting.png -------------------------------------------------------------------------------- /powershell.txt: -------------------------------------------------------------------------------- 1 | https://github.com/mikemaccana/powershell-profile/blob/master/Microsoft.PowerShell_profile.ps1 2 | -------------------------------------------------------------------------------- /cw/schedule_details_comparison.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/justinmk/notes/HEAD/cw/schedule_details_comparison.png -------------------------------------------------------------------------------- /software_design.txt: -------------------------------------------------------------------------------- 1 | Software architecture: decisions which are both important and hard to change. 2 | -- Martin Fowler 3 | -------------------------------------------------------------------------------- /README: -------------------------------------------------------------------------------- 1 | Notes 2 | ================================================================================ 3 | 4 | Copyright 2012 Justin M. Keyes 5 | -------------------------------------------------------------------------------- /go.txt: -------------------------------------------------------------------------------- 1 | 2 | https://gobyexample.com/ 3 | Twelve Go Best Practices 4 | http://talks.golang.org/2013/bestpractices.slide#1 5 | https://news.ycombinator.com/item?id=6134075 6 | 7 | -------------------------------------------------------------------------------- /haskell.txt: -------------------------------------------------------------------------------- 1 | http://www.haskellforall.com/2014/03/introductions-to-advanced-haskell-topics.html 2 | http://dev.stephendiehl.com/hask/ 3 | http://dev.stephendiehl.com/begin/index.html 4 | 5 | -------------------------------------------------------------------------------- /electronics-robotics-hacks.txt: -------------------------------------------------------------------------------- 1 | 3D printing coffee grinders https://news.ycombinator.com/item?id=19712349 2 | 3 | Can Reaction Wheels control a Drone? 4 | https://www.youtube.com/watch?v=4kfBEaTncjI 5 | 6 | VENDORS 7 | https://hobbyking.com 8 | https://www.conrad.de/ 9 | https://www.voelkner.de/ 10 | 11 | -------------------------------------------------------------------------------- /containers-virtual-machines-vm.txt: -------------------------------------------------------------------------------- 1 | hypervisor 2 | - Core responsibility: divide hardware resources and time between VMs. 3 | - Can run "on the hardware" ("type-1"), or as privileged process ("type-2"). 4 | - Examples: KVM, Xen 5 | - Xen ("type-1": direct control of hw) 6 | - KVM ("type-2": uses privileged, kernel-mode primitives for managing VMs; the kernel then schedules and allocates the appropriate system resources) 7 | - "Firecracker is not a hypervisor. Firecracker is a process that controls KVM." May be called a VMM ("virtual machine monitor") 8 | -------------------------------------------------------------------------------- /kotlin.txt: -------------------------------------------------------------------------------- 1 | COROUTINES 2 | ============================================================================== 3 | use withContext to limit coroutines to a single thread 4 | https://kotlinlang.org/docs/reference/coroutines/shared-mutable-state-and-concurrency.html 5 | val counterContext = newSingleThreadContext("CounterContext") 6 | var counter = 0 7 | fun main() = runBlocking { 8 | // confine everything to a single-threaded context 9 | withContext(counterContext) { 10 | massiveRun { 11 | counter++ 12 | } 13 | } 14 | println("Counter = $counter") 15 | } 16 | -------------------------------------------------------------------------------- /win_perf.txt: -------------------------------------------------------------------------------- 1 | wprui 2 | 3 | remember to rëenable Paging Executive after you're done using WPR: 4 | wpr -disablepagingexecutive off 5 | 6 | https://randomascii.wordpress.com/2013/04/20/xperf-basics-recording-a-trace-the-easy-way/ 7 | 8 | http://blogs.msdn.com/b/ntdebugging/archive/2012/11/30/troubleshooting-pool-leaks-part-7-windows-performance-toolkit.aspx 9 | 10 | PerfView http://www.microsoft.com/en-us/download/details.aspx?id=28567 11 | - "used in Roslyn for almost all investigations" 12 | - performance-analysis tool that helps isolate CPU- and memory-related performance issues 13 | - for ETW information (ETL files) as well as CLR memory information (heap dumps) 14 | -------------------------------------------------------------------------------- /urbit.txt: -------------------------------------------------------------------------------- 1 | https://urbit.org/blog/pki-maze/ 2 | Each planent has 4 billion "moons". 3 | A moon is a 64-bit address whose 32-bit suffix is its planet. 4 | 3 properties; pick any 2: 5 | - Comets are impermanent, self sovereign, and plentiful. 6 | - Planets are permanent, self sovereign, and not plentiful. 7 | - Moons are permanent, not self sovereign, and plentiful. 8 | 9 | https://www.reddit.com/r/IAmA/comments/4bxf6f/im_curtis_yarvin_developer_of_urbit_ama/ 10 | > What sort of relationship do you see between something like Urbit and IPFS? 11 | > Very simple: Urbit should have an IPFS client. Urbit is also a global immutable namespace, but one built on very different principles, and not solving the CDN/BitTorrent/Freenet axis that IPFS takes on. 12 | -------------------------------------------------------------------------------- /terminal.txt: -------------------------------------------------------------------------------- 1 | TTY demystified 2 | http://www.linusakesson.net/programming/tty/ 3 | Introduction to Termios: Termios(3) and Stty 4 | https://blog.nelhage.com/2009/12/a-brief-introduction-to-termios-termios3-and-stty/ 5 | 6 | To see the sequences that your terminal is _actually_ sending: 7 | showkey -a 8 | 9 | terminfo/termcap is a database that maps terminal inputs to terminal codes. 10 | `infocmp -L` prints out the current mappings. 11 | To see the mapping for `key_backspace`: 12 | infocmp -L | grep key_backspace 13 | To update a termcode mapping: 14 | 1. Write a new terminfo file with the updated mapping: 15 | infocmp $TERM | sed 's/kbs=^[hH]/kbs=\\177/' > $TERM.ti 16 | 2. Instruct the terminal to "recompile" its terminfo: 17 | tic $TERM.ti 18 | 19 | -------------------------------------------------------------------------------- /cryptocurrency.txt: -------------------------------------------------------------------------------- 1 | TODO: 2 | http://www.samlewis.me/2017/06/a-peek-under-bitcoins-hood/ 3 | ♡ https://www.igvita.com/2014/05/05/minimum-viable-block-chain/ 4 | http://queue.acm.org/detail.cfm?id=3136559 "Bitcoin's Academic Pedigree" Narayanan & Clark 5 | http://adilmoujahid.com/posts/2018/03/intro-blockchain-bitcoin-python/ 6 | 7 | Bitcoin whitepaper https://bitcoin.org/bitcoin.pdf 8 | The race between the honest chain and an attacker chain can be characterized 9 | as a Binomial Random Walk. The success event is the honest chain being 10 | extended by one block, increasing its lead by +1, and the failure event is 11 | the attacker's chain being extended by one block, reducing the gap by -1. 12 | 13 | The probability of an attacker catching up from a given deficit is analogous 14 | to a Gambler's Ruin problem. 15 | -------------------------------------------------------------------------------- /hardware.txt: -------------------------------------------------------------------------------- 1 | DISK MONITORING / FAILURE DETECTION 2 | =================================== 3 | Failure Trends in a Large Disk Drive Population / Pinheiro, Weber, Barroso 4 | https://static.googleusercontent.com/media/research.google.com/en//archive/disk_failures.pdf 5 | > Some SMART parameters (scan errors, reallocation counts, offline 6 | reallocation counts, and probational counts) have a large impact on 7 | failure probability. 8 | > 56% of failed drives had no count in any of the four strong SMART 9 | signals ... models based only on those signals can never predict more 10 | than half of the failed drives 11 | - 2-6% failure rate year in first year 12 | - key finding: no consistent pattern of higher failure rates for higher 13 | temperature or higher utilization 14 | 15 | -------------------------------------------------------------------------------- /tampa.txt: -------------------------------------------------------------------------------- 1 | do it 2 | Berns: steak sandwich at the bar 3 | Buddy Brew 4 | Oxford Exchange 5 | Ichicoro 6 | Renaissance Festival 7 | Tampa Theatre, rocky horror picture show 8 | The Castle 9 | casino 10 | golf 11 | treasure island drum circle http://tidrums.tripod.com/ 12 | gun range 13 | beach 14 | fun-lan drive-in 15 | 16 | culture 17 | Metal Monsters (?) 18 | Genitorturers 19 | Gasparilla 20 | Guavaween 21 | Tampa Theatre 22 | Stock Talk / Phil's Gang https://www.philsgang.com/ 23 | 24 | famous 25 | Wikipedia 26 | Hooters 27 | Outback 28 | Superbowl 2003 29 | Stanley Cup 2004 30 | 31 | florida man 32 | Marilyn Manson 33 | Peter Pan http://www.pixyland.org/peterpan/ 34 | Glenn Beck 35 | Scientology 36 | "Live Prayer" tv show 37 | Tampa teen accused of Twitter hack (2020) https://news.ycombinator.com/item?id=24011939 38 | -------------------------------------------------------------------------------- /.nvim.lua: -------------------------------------------------------------------------------- 1 | function Global_completeTags() 2 | local curline = vim.fn.line('.') 3 | local curcol = vim.fn.col('.') 4 | local start = vim.fn.searchpos('\\s', 'bn', curline) 5 | local startcol = math.max(0, start[2]) 6 | local partialword = vim.api.nvim_buf_get_text(0, curline-1, startcol, curline-1, curcol, {})[1] 7 | 8 | -- Get the list of tags from the buffer. 9 | local it = vim.iter(vim.fn.getline(1, '$')) 10 | :filter(function(l) return l:match('^tags') end) 11 | :map(function(s) return vim.split(s,' ') end) 12 | local tags = vim.tbl_flatten(it:totable()) 13 | tags = vim.iter(tags):filter(function(s) return s:match('^'..partialword) end) 14 | :totable() 15 | table.sort(tags) 16 | tags = vim.fn.uniq(tags) 17 | 18 | vim.fn.complete(startcol + 1, tags) 19 | return '' 20 | end 21 | 22 | vim.cmd([[ 23 | inoremap =v:lua.Global_completeTags() 24 | ]]) 25 | -------------------------------------------------------------------------------- /technical_interview.txt: -------------------------------------------------------------------------------- 1 | system design: 2 | 0. Listen and comprehend the question. 3 | 1. gather requirements: 4 | - API, SLA, CAP, security (CORS,SSL), features, scale, userbase, state 5 | - AAA: auth, auditing, accounting/billing 6 | - quantify! 7 | 2. draw LOGICAL diagram showing dataflow & state 8 | - draw PHYSICAL diagram? (LB, servers, WAF, CDN, …) 9 | 3. end-to-end explain (component details) 10 | - scaling 11 | - API 12 | - db table 13 | - estimates/calculations 14 | - technologies 15 | 16 | 3 words about me: 17 | leverage 18 | visibility (fail early) 19 | unblock (adaptive) 20 | 21 | startup life: 22 | 1. fleetscanner: RO filesystem hack 23 | 2. pathtopark: circular route 24 | 3. in-car dashboard: ship now, update OTA 25 | 4. data lake: overlay filesystem 26 | 5. DS pipeline: Luigi + parallel 27 | 6. segment ids: Open Location Code (OLC) https://plus.codes 28 | -------------------------------------------------------------------------------- /scala.txt: -------------------------------------------------------------------------------- 1 | substitution model (λ-calculus) 2 | - evaluation === reduce an expression to a value 3 | - can be applied to all expressions that have no side effects 4 | - similar to algebraic simplification, rewrites each expression recursively 5 | until it is reduced to a value 6 | 7 | evaluation strategies: call-by-name, call-by-value 8 | both reduce to the same final values iff: 9 | - the reduce expression consists of pure functions 10 | - both evaluations terminate 11 | call-by-value 12 | advantage: evaluates each fn arg only once 13 | in practice, call-by-value is exponentially more efficient for argument evaluation 14 | 'val' is CBV: its RHS is evaluated immediately on definition 15 | val x = loop //infinite loop 16 | call-by-name 17 | advantage: fn arg is not evaluated if the corresponding parameter is unused in the fn body 18 | force call-by-name with the arrow syntax: 19 | def foo(x: Int, y: => Int) = 1 //x is CBV, y is CBN 20 | 'def' is CBN: its RHS is evaluated on each call 21 | def x = loop //not evaluated until called 22 | 23 | tail recursion optimization: 24 | @tailrec => explicitly require tail recursion 25 | -------------------------------------------------------------------------------- /github.txt: -------------------------------------------------------------------------------- 1 | GitHub Actions 2 | 3 | Working example: 4 | .github/workflows/main.yml: 5 | name: CI 6 | on: 7 | push: 8 | branches: 9 | - master 10 | - release-* 11 | jobs: 12 | fix-merge-commit: 13 | runs-on: [ubuntu-latest] 14 | # if: github.ref == 'master' 15 | steps: 16 | - uses: actions/checkout@v1 17 | - name: Rewrite merge-commit message 18 | env: 19 | GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} 20 | SED_PATTERN: "[pP]ull [rR]equest \\?" 21 | run: | 22 | git config user.email "actions@github" 23 | git config user.name "Github Actions" 24 | git remote set-url origin https://x-access-token:${GITHUB_TOKEN}@github.com/${GITHUB_REPOSITORY}.git 25 | if git --no-pager log -1 --format='%B' | grep "$SED_PATTERN" ; then 26 | echo "rewriting commit-message..." 27 | git commit --amend -m "$(git --no-pager log -1 --format='%B' | sed "1s/$SED_PATTERN//")" 28 | git push --force origin HEAD:${GITHUB_REF} 29 | else 30 | echo "commit looks ok, skipped" 31 | fi 32 | 33 | see also: 34 | https://github.com/jessfraz/branch-cleanup-action 35 | -------------------------------------------------------------------------------- /todo.md: -------------------------------------------------------------------------------- 1 | errands 2 | ------------------------------------------------------------------------------ 3 | * keyboard: Flux Keyboard https://fluxkeyboard.com/ https://news.ycombinator.com/item?id=35083665 4 | * backup: https://lobste.rs/s/y4myxv 5 | 6 | AI/ML/data-science: 7 | ------------------------------------------------------------------------------ 8 | * The Little Book of Deep Learning https://fleuret.org/francois/lbdl.html 9 | 10 | datastructures: 11 | ------------------------------------------------------------------------------ 12 | * hashtable: https://github.com/jamesroutley/write-a-hash-table 13 | * coroutines, trampolining, …: https://www.dabeaz.com/coroutines/index.html 14 | * allocation: http://www.smallmemory.com/6_AllocationChapter.pdf 15 | * http://eli.thegreenplace.net/2015/directed-graph-traversal-orderings-and-applications-to-data-flow-analysis/ 16 | * Crypto 101 (cryptography course): https://www.crypto101.io/ 17 | 18 | PL, compilers 19 | ------------------------------------------------------------------------------ 20 | * How to Write a (Lisp) Interpreter (in Python) http://norvig.com/lispy.html 21 | * "a few things about PyPy3 support" https://news.ycombinator.com/item?id=11262150 22 | 23 | network 24 | ------------------------------------------------------------------------------ 25 | * Let's code a TCP/IP stack, 1: Ethernet & ARP https://news.ycombinator.com/item?id=11234229 26 | * https://daniel.haxx.se/http2/ 27 | 28 | papers 29 | ------------------------------------------------------------------------------ 30 | * Purely Functional Data Structures, Chris Okasaki 31 | * Einstein's statement of special relativity 32 | * Learning How to Learn: https://www.coursera.org/learn/learning-how-to-learn 33 | 34 | web 35 | ------------------------------------------------------------------------------ 36 | * On Designing and Deploying Internet-Scale Services: https://www.usenix.org/legacy/event/lisa07/tech/full_papers/hamilton/hamilton_html/ 37 | * Notes on Google's Site Reliability Engineering book: http://danluu.com/google-sre-book/ 38 | * https://www.facebook.com/notes/facebook-engineering/scalable-memory-allocation-using-jemalloc/480222803919 39 | -------------------------------------------------------------------------------- /sql_database.txt: -------------------------------------------------------------------------------- 1 | HoBT: Heap or B+Tree (generic term for index or lack thereof) 2 | 3 | clustered index: each data page organized in a linked list 4 | B-Tree 5 | each node points to n child nodes (unlike binary tree) 6 | B+Tree 7 | SQL Server uses this for indexes 8 | efficient for inserts, deletes 9 | each level is a doubly-linked list 10 | level 0 = leaf (no leaves at any other level) 11 | 12 | Heap = table that _lacks_ a [clustered] index 13 | points to row (RID) directly 14 | one HoBT per partition (currently): 15 | sys.partitions.hobt_id == sys.partitions.partition_id 16 | actully aliased from sys.sysrowsets.rowsetid 17 | thehobt.blogspot.com/2009/02/what-heck-is... 18 | 19 | Clustered index leaf 20 | stores the actual column data (at leaf level) 21 | only 1 clustered index per table, so all other [non-clustered] index leaves store the clustered key 22 | footnote: you can actually add "included columns" indexes which in effect create another clustered key 23 | on a given table, which effectively duplicates the table data--so this is generally advised against. 24 | sorted, so moves have a cost 25 | 26 | Nonclustered index leaf 27 | store RID or clustered key 28 | non-sorted, so moves are cheap 29 | 30 | 3 ways for SQL server to read data 31 | scan 32 | table scan 33 | used for heaps only 34 | needs to read ALL pages to satisfy WHERE clause 35 | clustered index scan 36 | reads almost all pages 37 | seek 38 | clustered index seek 39 | lookup 40 | 41 | ================================================================================ 42 | sqlite 43 | 44 | tradeoffs: 45 | - No foreign keys by default. Every connecting session must toggle it on. 46 | - One writer. Any session that issues BEGIN TRANSACTION and then hangs halts all DML. 47 | - WAL mode confusion. WAL cannot safely be used on network filesystems, and it breaks ACID on ATTACHed databases, among other problems. 48 | - No date/time types. 49 | - Length specifications on a column are ignored. CHAR(2) will allow the insertion of a blob. 50 | Check constraints could be used to enforce this. 51 | - Type affinity => any data type can be inserted into columns declared as any other data type. 52 | Type enforcement can be done, but not the default. 53 | -------------------------------------------------------------------------------- /video_codec.txt: -------------------------------------------------------------------------------- 1 | CRF Guide (Constant Rate Factor in x264 and x265) http://slhck.info/video/2017/02/24/crf-guide.html 2 | 3 | ffmpeg 4 | ============================================================================== 5 | https://github.com/leandromoreira/ffmpeg-libav-tutorial 6 | 7 | https://superuser.com/a/490691 8 | Besides ffmpeg's options, encoders have "private options". To see all options: 9 | 10 | ffmpeg -h full 11 | 12 | -movflags faststart 13 | Mandatory for online video: this will move some metadata to the beginning of 14 | the file so that playback can begin before it is completely downloaded. 15 | 16 | Quality 17 | ------- 18 | 19 | Quality is controlled either by bitrate (-b:v (video), -b:a (audio), presets 20 | (slow, fast, etc.)), or by codec-specific options. E.g. x264 supports various 21 | encoding methods, with the Constant Rate Factor (CRF) being the most 22 | sophisticated: it produces variable bitrate, but overall good quality in 23 | a single pass. Range is 0-51 (sane range is 19~26), 23 is default. 24 | 25 | ffmpeg -i input.mp4 -c:v libx264 -crf 23 output.mp4 26 | 27 | x264 *presets* primarily affect encoding speed vs compression 28 | (quality-per-filesize), they do not change VISUAL quality, only final bitrate 29 | (and therefore output file size). 30 | 31 | The problem with hardware encoders is that they use outdated versions of the 32 | codec. For older codecs like x264 that is less of an issue, but for x265 the 33 | difference is huge. 34 | 35 | 36 | control async/vsync 37 | : https://stackoverflow.com/questions/49718317/append-black-frames-to-video-when-audio-is-longer-in-ffmpeg 38 | https://trac.ffmpeg.org/ticket/4674 39 | > encoding with -async 1 works fine but produces silent moments. 40 | does "-vsync 1" dup frames to keep sync with audio? 41 | PTS = presentation timestamps 42 | filter: setpts 43 | https://ffmpeg.org/ffmpeg-filters.html#setpts_002c-asetpts 44 | http://forum.doom9.org/archive/index.php/t-163290.html 45 | > use "-async 1" instead "-vsync 0" and "-r 23.976"... 46 | or: 47 | > ffmpeg -fflags +genpts+igndts -i -r:v 30 -vsync 2 -async 1 -map 0:v,0:a -map 0:a 48 | > That effectively tries to sync the video to the audio and drops frames to 49 | > prevent frames with same pts, flags the stream as 30fps (was proper for 50 | > this particular source, adjust to yours) and keeps it VFR still (i.e. does 51 | > not output duplicate frames). 52 | -------------------------------------------------------------------------------- /gps.txt: -------------------------------------------------------------------------------- 1 | NTP 2 | ======================================================================== 3 | https://www.raspberrypi.org/forums/viewtopic.php?t=1970&start=225 4 | 5 | NTP will serve the time estimate that the host has. It includes the quality 6 | (dispersion, source info, last correction, etc) so the client can judge which 7 | source to use. If the GPS dies, the quality will decline; eventually to the 8 | point that clients will pick another of its sources. Note that time doesn't 9 | instantly go bad. The GPS merely adjusts ('disciplines' in ntp speak) the 10 | local clock. If the local clock is stable (no wild temperature, power, load 11 | changes) and NTP has run long enough to converge the driftfile, the time will 12 | remain accurate for a 'long' time. 13 | 14 | GPS 15 | ======================================================================== 16 | 17 | - GPS delivers 14 numbers: x, y, z, t, vx, vy, vz, and error estimates for 18 | each of those 7 coordinates. This is a "TPV" (time-position-velocity) 19 | report. GPS is a TPV oracle. 20 | - gpsd delivers those numbers to user applications with minimum fuss. 21 | 22 | Almanac 23 | The time required for a GPS to get a fix (Time To First Fix (TTFF)) can vary 24 | from 15s to 30min. The main factors affecting this latency are (a) whether 25 | it has an almanac available, (b) whether it has satellite ephemerides 26 | available, and (c) whether it has recent fix available. Of course the 27 | quality of signal at your location matters as well. 28 | 29 | If a GPS has not been on for several months, then it has no current almanac. 30 | It must wait to download one before it can generate a fix. This can take 31 | ~15min. 32 | 33 | While the almanac download takes 15 minutes, you have to be there for the 34 | start of it, otherwise you have to wait for the next cycle. So if you are 35 | unlucky and just miss the start of one, it could take just under 29 minutes 36 | to obtain, and on average closer to 22 min. 37 | 38 | If a GPS has not been on for ~1day then it has a valid almanac but no valid 39 | satellite ephemerides, and must download at least four before it can 40 | generate an accurate fix. Each satellite has its own ephemeris that must be 41 | downloaded if a current copy is not fresh. 42 | 43 | 44 | 45 | 46 | gpsd 47 | tools: 48 | gpsprof: collects accuracy and latency statistics on GPSes and the GPS+gpsd combination 49 | ???: tool for decoding RTCM104 reports on satellite health, almanacs, 50 | and pseudorange information from differential-GPS radios and 51 | reference stations. 52 | -------------------------------------------------------------------------------- /3d-printing.md: -------------------------------------------------------------------------------- 1 | 3d-printing self-replication programming electronics engineering 2 | 3 | Basics 4 | ================================================================================ 5 | - Model downloads: https://www.printables.com/model 6 | - Model formats: 3mf, stl, ... 7 | - Convert to "g code" for instructing the printer. 8 | - FDM (fused deposition modeling), aka FFF (fused filament fabrication https://en.wikipedia.org/wiki/Fused_filament_fabrication): 9 | - heated filament emitted by nozzle. ("if the machine has a nozzle, it's FDM/FFF") 10 | - common, cost-effective. 11 | - SLA (Stereolithography https://en.wikipedia.org/wiki/Stereolithography) 12 | - works by layering "photos" of liquid layers together. 13 | - uses a UV light to cure liquid resin into hardened plastic. 14 | - requires extra steps after the print: washed, cured, dried. 15 | - more expensive, less common, but can yield more "perfect" results (useful for e.g. miniatures). 16 | - 45 degrees is the most acute angle that can be reliably printed. 17 | - can be strengthened by changing the "infill". 18 | - "quality" / granularity is determined by nozzle size (smaller = more granular). 19 | - ⭐️ materials comparison table: https://help.prusa3d.com/materials 20 | 21 | Workshop 2024 22 | ================================================================================ 23 | 1. Download `SCP_049.3mf` from https://www.printables.com/model/111595-scp-049/comments 24 | 2. Drag it into PrusaSlicer. 25 | - ![](./img/prusaslicer-3dprinting.png) 26 | 3. Print settings: choose "0.2 mm". 27 | - Configures the nozzle size; affects granularity/quality/speed. 28 | 4. Filament: choose "Prusament PLA". 29 | - Must match the filament that you chose. 30 | 5. Infill: adjust this % to change the density of the model. 31 | 6. click "Slice now" 32 | - PrusaSlicer validates the model, suggest fixes, and generate a .stl file (`SCP_049.stl`). 33 | 7. click "Export G-code" 34 | - PrusaSlicer creates a .bgcode file (`SCP_049_0.4n_0.2mm_PLA_MK4IS_39m.bgcode`). 35 | 8. Send the g-code file to the printer (usb drive, etc). 36 | 9. Select the file in the printer. Don't remove the usb stick during printing, it can't be resumed! 37 | 10. Load the filament in the printer. Use the printer UI to take the filament 38 | 11. Start the print. Let it do its self-test routine. Wait for printing to complete. 39 | 12. After printing: cleanup! User the printer UI to detach the filament and perform a self-clean. 40 | 41 | Vendors 42 | ================================================================================ 43 | - https://www.prusa3d.com/ 44 | - open source, most popular, big ecosystem. 45 | - https://www.formlabs.com/ 46 | - proprietary 47 | - https://www.reprap.org/ 48 | - self-copying 3D printer (self-replicating machine). 49 | -------------------------------------------------------------------------------- /mssql_connection_leaks.md: -------------------------------------------------------------------------------- 1 | TODO: 2 | http://www.brentozar.com/archive/2010/09/sql-server-dba-scripts-how-to-find-slow-sql-server-queries/ 3 | http://blogs.msdn.com/b/angelsb/archive/2004/08/25/220333.aspx 4 | 5 | # SQL Server connection-leak debugging: state of the art (30 June 2014) 6 | 7 | Tell-tale exception: 8 | 9 | > System.InvalidOperationException: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. 10 | > This may have occurred because all pooled connections were in use and max pool size was reached. 11 | 12 | ## Observing from SQL Server 13 | 14 | exec `sp_who2` 15 | 16 | ``` 17 | SPID Status Login DBName Command ProgramName 18 | ====================================================================================================== 19 | 31 BACKGROUND sa master FT CRAWL MON 20 | 32 sleeping sa master TASK MANAGER 21 | 34 BACKGROUND sa NULL UNKNOWN TOKEN 22 | 51 sleeping NT AUTHORITY\SYSTEM msdb AWAITING COMMAND SQLAgent - Job invocation engine 23 | 52 sleeping NT AUTHORITY\SYSTEM ReportServer AWAITING COMMAND Report Server 24 | [...] 25 | 83 sleeping NT AUTHORITY\SYSTEM FooApp AWAITING COMMAND .Net SqlClient Data Provider 26 | 84 sleeping NT AUTHORITY\SYSTEM FooApp AWAITING COMMAND .Net SqlClient Data Provider 27 | ``` 28 | 29 | The `sp_who2` list is more convenient if you set the `Application Name` 30 | parameter in your SQL connection string. 31 | 32 | ```cs 33 | + ";Application Name=" + this.appName + "/Database/" + subModuleName + ";"; 34 | ``` 35 | 36 | ## Observing from the .NET application 37 | 38 | Windows *performance counters* can be used to observe the connection pool 39 | behavior. Awkwardly derive the "instance name" from the application name, then 40 | PerformanceCounter.NextValue() should report the value. 41 | `NumberOfReclaimedConnections` is probably the one you care about for leaks. 42 | 43 | ```cs 44 | using System.Runtime.InteropServices; 45 | 46 | [DllImport("kernel32.dll", SetLastError = true)] 47 | static extern int GetCurrentProcessId(); 48 | 49 | private void openConnectionIfNecessary() { 50 | // Open a connection and create the performance counters. 51 | // http://msdn.microsoft.com/en-us/library/ms254503%28v=vs.80%29.aspx 52 | 53 | string instanceName = AppDomain.CurrentDomain.FriendlyName.Replace('(','[') 54 | .Replace(')',']').Replace('#','_').Replace('/','_').Replace('\\','_').ToLower() 55 | + "[" + GetCurrentProcessId() + "]"; 56 | 57 | connection.Open(); 58 | 59 | var p = new PerformanceCounter { 60 | CategoryName = ".NET Data Provider for SqlServer", 61 | CounterName = "NumberOfReclaimedConnections", 62 | InstanceName = instanceName 63 | }; 64 | DebugOutput.Output("NumberOfReclaimedConnections: "+p.NextValue()); 65 | } 66 | ``` 67 | -------------------------------------------------------------------------------- /java.txt: -------------------------------------------------------------------------------- 1 | TODO 2 | ================================================================================ 3 | https://shipilev.net/jvm-anatomy-park/ 4 | 5 | GC 6 | ================================================================================ 7 | https://github.com/cirosantilli/java-cheat/search?utf8=%E2%9C%93&q=garbage&type= 8 | 9 | Why I don't like System.gc() 10 | http://jeremymanson.blogspot.de/2015/12/why-i-dont-like-systemgc.html 11 | 12 | PROFILING 13 | ================================================================================ 14 | sampling profiler (uses SIGPROF, see http://jeremymanson.blogspot.de/2010/07/why-many-profilers-have-serious.html) 15 | https://github.com/jvm-profiling-tools/honest-profiler 16 | 17 | CONCURRENCY 18 | ================================================================================ 19 | ForkJoinPool#commonPool() 20 | Threadpool with thread-count / behavior decided by getParallelism(): 21 | getParallelism() = (CPU_COUNT - 1): 22 | > 2: JDK creates (CPU_COUNT - 1) threads for commonPool 23 | !This is why common spool should be avoided for blocking tasks! 24 | = 1: JDK creates a new thread for every task 25 | = 0: task is executed on caller thread 26 | Two major concepts use commonPool: 27 | CompletableFuture: can specify thread-pool to avoid commonPool 28 | parallel(): can NOT specify own thread-pool to avoid commonPool 29 | Trick to avoid commonPool: https://stackoverflow.com/a/22269778 30 | Demo: 31 | import java.util.concurrent.*; 32 | import java.util.stream.*; 33 | import java.time.*; 34 | import java.util.*; 35 | public class CommonPoolTest { 36 | public static void main(String[] args) { 37 | System.out.println("CPU Core: " + Runtime.getRuntime().availableProcessors()); 38 | System.out.println("CommonPool Parallelism: " + ForkJoinPool.commonPool().getParallelism()); 39 | System.out.println("CommonPool Common Parallelism: " + ForkJoinPool.getCommonPoolParallelism()); 40 | long start = System.nanoTime(); 41 | List> futures = IntStream.range(0, 100) 42 | .mapToObj(i -> CompletableFuture.runAsync(CommonPoolTest::blockingOperation)) 43 | .collect(Collectors.toUnmodifiableList()); 44 | CompletableFuture.allOf(futures.toArray(CompletableFuture[]::new)).join(); 45 | System.out.println("Processed in " + Duration.ofNanos(System.nanoTime() - start).toSeconds() + " sec"); 46 | } 47 | private static void blockingOperation() { 48 | try { 49 | Thread.sleep(1000); 50 | } catch (InterruptedException e) { 51 | e.printStackTrace(); 52 | } 53 | } 54 | } 55 | 56 | $ docker run -it --cpus 4 -v ${PWD}:/app --workdir /app adoptopenjdk/openjdk11 java CommonPoolTest.java 57 | CPU Core: 4 58 | CommonPool Parallelism: 3 59 | CommonPool Common Parallelism: 3 60 | Processed in 34 sec 61 | 62 | ForkJoinPool.ManagedBlocker: https://stackoverflow.com/a/46073118 63 | informs the Fork/Join pool that a task may potentially block, so the pool 64 | can create compensation threads when it recognizes that a worker thread is 65 | about to block. 66 | 67 | 68 | -------------------------------------------------------------------------------- /travel.txt: -------------------------------------------------------------------------------- 1 | checklist 2 | headphones 3 | health insurance 4 | passport 5 | snacks/food 6 | thermos/water bottle 7 | power converter 8 | lotion 9 | 2x all clothes… 10 | flashlight (check for bedbugs) 11 | tissues/napkins 12 | ibuprofen 13 | packing cubes 14 | melatonin 15 | gloves 16 | sunglasses 17 | umbrella 18 | jacket 19 | phone, charger 20 | pen/pencil 21 | 22 | todo: 23 | - wraparound neck pillow https://letsayme.co/products/travel-neck-pillow?twclid=2-1of7qrc3bitgu4dmvrl9opuh6 24 | - backpack https://naildrivin5.com/blog/2018/01/25/how-to-travel-to-san-francisco-every-other-month-for-five-years.html 25 | - luggage: "Briggs & Riley Baseline Domestic Carry-On Exp Spinner" https://www.amazon.com/dp/B00XDIRKRC/ 26 | - backpack: 27 | https://www.goruck.com/products/gr1 28 | https://www.tombihn.com/collections/backpacks/products/synapse-25?variant=50297850439 29 | 30 | where/what 31 | germany 32 | https://www.drachenstich.de/ Drachenstadt: 33 | Furth im Wald lies right on the Germany-Czechia border 100 miles east of Nuremberg in Bavaria. 34 | Each year, the city re-enacts the legend of St. George slaying the dragon with a festival. 35 | https://www.nrw-tourismus.de/roemerfest-xanten 36 | Bread and Games is one of the largest Roman festivals in the world, every 2 years in the North Rhine-Westphalian city of Xanten. 37 | austria 38 | Zwentendorf nuclear reactor tour https://news.ycombinator.com/item?id=40386935 39 | portugal 40 | azores sao miguel 41 | lisbon food: 42 | portugeuse sardines on portugeuse bread w/ oil 43 | cabrito (goat) 44 | beefana (pork sandwich) 45 | spain 46 | Sagrada Família https://en.wikipedia.org/wiki/Sagrada_Família 47 | https://zuzalu.city/ : Zuzalu is a "pop-up city" in Montenegro. 48 | Tangier, Morocco https://en.wikipedia.org/wiki/Tangier 49 | Seattle: Boeing factory tour ("largest building in the world") http://www.boeing.com/company/tours 50 | UK 51 | Islington, London: Slimelight, "longest running Goth nightclub" http://www.slimelight.net 52 | Scotland: http://en.wikipedia.org/wiki/Castle_Stalker 53 | http://www.castlexplorer.co.uk/ 54 | Smithsonian Institute 55 | Iceland 56 | Ibiza 57 | Czech Republic 58 | https://en.wikipedia.org/wiki/Sedlec_Ossuary 59 | italy 60 | Capuchin Crypt 61 | Japan 62 | - https://www.kalzumeus.com/japan-recommendations/ 63 | - If are here for at least a week, Japan Rail Pass https://japanrailpass.net/ 64 | best travel deal in the world (and which you must purchase before you get here), 65 | gives you unlimited trips on JR trains 66 | - Japanese reviewers are substantially pickier than you are, so 3.7 stars = very good 67 | - if you stay in Japan you simply must stay in a ryokan at some point 68 | https://jalan.net/ 69 | Pick any, ideally either in Kyoto or outside of a large city. 70 | - https://en.wikipedia.org/wiki/Ryokan 71 | - use ChargeSpot app for charging phone 72 | x http://en.wikipedia.org/wiki/Harajuku 73 | NYC 74 | Jean-Louis David's painting of Socrates' death: Metro. Mus. of Fine Art, NYC 75 | Florida: 76 | https://boktowergardens.org/ (otw to Orlando...) 77 | Key Largo: Jules' Undersea Lodge. 78 | 79 | camping: 80 | coffee grounds 81 | coffee press 82 | cooking pot 83 | snacks 84 | eggs 85 | -------------------------------------------------------------------------------- /recipes.txt: -------------------------------------------------------------------------------- 1 | Noel's Curry 2 | onions. chop. sautee until brown 3 | garlic. chop. 6-8 cloves. 4 | ginger (fresh). chop. 5 | 6 | simmer. 7 | 8 | turmeric. pour on. 9 | pinch of garlic salt. 10 | ~3 min. 11 | turn over and repeat. 12 | 13 | cover. stir. high heat. 14 | 15 | ~done, add 1/2 cup water, change to low heat. 16 | salt. 17 | crushed red chili pepper. 18 | green chilis (serranos). cut very thin. 19 | 20 | --- 21 | Cochinita pibil 22 | - annatto seeds 5 tbsp 23 | - cumin seed 2 tsp 24 | - pepper 1 tbsp 25 | - 8 all spice 26 | - cloves 1/2 tsp 27 | - habanero peppers "a couple". remove seeds & veins. 28 | - 1/2 cup OJ 29 | - 1/2 cup white vinegar (1/4?) 30 | - "annatto paste" 31 | - season grinder 32 | 1. mix OJ, vinegar, habanero 33 | 2. add season powder 34 | 3. add 2 tbsp salt 35 | 4. 8 cloves of garlic 36 | 5. blend (food processor) "the smoother the better" 37 | 6. add juice of 5 lemons 38 | 7. splash of tequila 39 | 8. 5 lbs pork butt. cut into 2-in squares. 40 | 9. freezer bag: meat & juice 41 | 10. line & cover pan w/ banana leaves. 42 | 11. wrap pan in foil, let no heat escape. 43 | 12. 325°F 4 hours 44 | 45 | --- 46 | SMORES 47 | cuban toast (mayo, cheese) 48 | Bananas Foster 49 | 50 | --- 51 | 52 | PERNIL SOFRITO 53 | 54 | - ~5 lbs pork shoulder (aka pork butt) 55 | - Serves 1 for about a week. 56 | 57 | 1. Get a long knife. 58 | 2. In a pilón (mortar and pestle) smash up a ~5 cloves of garlic (i.e., half of a garlic bulb), some sazon, salt, pepper, and oil. Grind it up good. 59 | - Now you have a "ghetto sofrito". 60 | 3. Pre-heat oven to ~300 F. 61 | 4. Take your knife and stab some holes (or deep slits) in the pork. Twist the knife to make the holes nice and wide. 62 | 5. Stuff some sofrito into the holes. Use the remainder to coat the outside of the meat. Add salt and pepper. 63 | 6. Throw the pork in skin/fat-side up and wait. Do _not_ cover with foil. 64 | 7. It's going to take ~45 minutes a pound. After ~1.5 hours, test it with a meat thermometer (remember to not rest it on the bone, or you will get a bad reading). 65 | 8. At 150-160 F, crank the stove up to 400 F. This will give you a delicious, crispy skin/fat. 66 | 9. At ~170 F, pull it out, but _don't_ carve it up. Wait at least 10 minutes, otherwise the juices will dribble out. 67 | 68 | TOSTONES 69 | 70 | - Green plantains (not ripe). Usually 5 for $1. 71 | - If you have ripe plantains, you could make _maduros_ (amarillos) instead of tostones. 72 | - Bacon 73 | - Flour 74 | 75 | 1. Fry some bacon. Set the bacon aside and save the fat. 76 | 2. Take a plantain and run a knife down the side and split the skin off without breaking the plantain. This takes a bit of practice. 77 | 3. Slice up the plantain into ~1/3 inch thick slices. Throw them into a bowl of ice water. 78 | 4. Use a deep fryer or else pour ~1/2 inches of oil into a frying pan. Corn oil works best, olive oil smokes too easily. Get it very hot. Throw in the bacon grease. 79 | 5. Take your sliced up plantains out of the ice water and drain them or even pat them with a paper towel until they're dry. 80 | 6. Fry them until they just turn golden. 81 | 7. Throw them in the freezer for 10 minutes. 82 | 8. Here's the fun part: Get a flat bottom glass and a cutting board or a plate. Throw some flour on there. Smash the plantains with the cup. You may need a spatula to get them off the board. 83 | 9. Fry them _again_ until they are golden and crispy. Drain on a paper towel until dry. 84 | 85 | Tostones take condiments well; try a spicy hot sauce such as Salsa Huichol. 86 | -------------------------------------------------------------------------------- /emacs.txt: -------------------------------------------------------------------------------- 1 | Writing A Spotify Client in 16 Minutes 2 | https://www.youtube.com/watch?v=XjKtkEMUYGc 3 | 4 | http://www.masteringemacs.org/ 5 | 6 | http://www.emacswiki.org/emacs/Reference_Sheet_by_Aaron_Hawley 7 | 8 | very good explanation of (autoload): 9 | http://www.lunaryorn.com/2014/07/02/autoloads-in-emacs-lisp.html 10 | 11 | C-h m available major/minor modes and bindings 12 | C-h k show the functions bound to a key sequence 13 | C-h f show the key sequence(s) bound to a function 14 | M-x occur regex-based outline! 15 | 16 | TERMINAL 17 | ======== 18 | synopsis of shell and terminal modes included in Emacs 19 | https://lukeshu.com/blog/emacs-shells.html 20 | tips, env vars 21 | http://snarfed.org/why_i_run_shells_inside_emacs 22 | 23 | 24 | MODE HOOKS 25 | ========== 26 | Mode hooks are like Vim ftplugins: allows you to execute arbitrary elisp for modes. 27 | For example: 28 | (add-hook 'prog-mode-hook 'linum-on) 29 | `prog-mode-hook` is the generic hook that most programming modes are derived 30 | from (i.e. emacs-lisp-mode, c-mode....). If you add a function to 31 | `prog-mode-hook`, that function is executed in every mode derived from 32 | `prog-mode`. In this case, linum will be activated in every programming mode, 33 | but not modes like help-mode or dired-mode. So, file-types are _loosely_ 34 | associated with major modes. Major modes are associated with _content_. I said 35 | "loosely" because you do not necessary use the code of a major mode with a file 36 | of a particular type; you can use it in any buffer. For example, you can 37 | activate `asm-mode` on a buffer that holds the output of the command `objdump` 38 | to have really nice highlighting and features of asm-mode. For example, you may 39 | want to enable smerge-mode in a buffer with merge conflict: 40 | (defun enable-smerge () 41 | (save-excursion 42 | (goto-char (point-min)) 43 | (when (re-search-forward "^<<<<<<< " nil t) ; search for conflict marking 44 | (smerge-mode 1)))) 45 | (add-hook 'find-file-hook 'enable-smerge t) 46 | 47 | 48 | EMACS LISP 49 | ========== 50 | [video] intro to emacs lisp (and ERC): http://emacsnyc.org/videos.html#2014-04 51 | 52 | show results in buffer, a la Light table: 53 | eval-print-last-sexp 54 | 55 | difference between define-key, evil-define-key: 56 | (define-key evil-motion-state-local-map "a" 'foo) 57 | ;binds "a" in the CURRENT BUFFER for ALL MODES 58 | (evil-define-key 'motion foo-mode-map "a" 'foo) 59 | ;binds "a" for ALL BUFFERS in ONLY THE SPECIFIED MODE 60 | 61 | 'foo means (quote foo) 62 | #'foo means (function foo) 63 | - special form that returns a function-object without evaluating it 64 | - converted into a closure if lexical binding is enabled 65 | - in practice, you will probably never need #' in elisp 66 | http://stackoverflow.com/questions/25275842/does-function-serve-any-purpose-in-emacs 67 | 68 | ;; macroexpand expands the form until a non-macro is found, but 69 | ;; it does _not_ inspect _sub_expressions in the resolved macro. 70 | ;; http://www.chemie.fu-berlin.de/chemnet/use/info/elisp/elisp_13.html 71 | (defmacro inc (var) 72 | (list 'setq var (list '1+ var))) 73 | ;; => inc 74 | 75 | (macroexpand '(inc r)) 76 | ;; => (setq r (1+ r)) 77 | 78 | (defmacro inc2 (var1 var2) 79 | (list 'progn (list 'inc var1) (list 'inc var2))) 80 | ;; => inc2 81 | 82 | (macroexpand '(inc2 r s)) 83 | ;; => (progn (inc r) (inc s)) 84 | 85 | -------------------------------------------------------------------------------- /akamai.txt: -------------------------------------------------------------------------------- 1 | action items: 2 | - www.myconnectwise.net needs to be "akamaized" 3 | - enable persistent connections (PCONNs) 4 | - ensure origin allows PCONN (keep-alive) 5 | - enable SureRoute, Enhanced Akamai Protocol 6 | - deploy SureRoute test object, must return 200 7 | - "SR test object should be large enough to reflect packet loss on the route" 8 | - deploy SLA test object (SLA mgmt report) 9 | - prefetch? 10 | 11 | possible improvements: 12 | - cache GetCustomImage.rails 13 | - upload user documents to NetStorage instead of origin 14 | - Akamai "Fast File Upload" add-on 15 | 16 | Russia: no edges 17 | China: "China CDN" 18 | 19 | goal: 1 hop from user to content 20 | 21 | CP code: arbitrary "tag" number to associate content with a contract (billing) 22 | - can be used to arbitrarily separate content groups in reports, purging, etc. 23 | 24 | NetStorage: simple filesytem for serving static resources, eg CW updates. 25 | 26 | GTM (Global Traffic Mgmt): "datacenter load balancing" 27 | - test agents perform continual checks for origin "liveness" and performance 28 | 29 | DPC: ? 30 | 31 | Akamai Instant 32 | 33 | cache-control HTTP headers are _not_ honored by default 34 | 35 | if origin returns a "vary" header that is anything other than 36 | "vary-accept-encoding", akamai will NOT cache that resource. 37 | 38 | SureRoute: enable this. ("avoids the BGP bottleneck") 39 | seed a SureRoute Test Object whose sizes matches the typical payload 40 | of an application request. 41 | 42 | API: 43 | fetch/create Events: https://control.akamai.com/events-dl/restdoc/ 44 | other: Configure > Tools > Web Services > ... 45 | 46 | cache refreshing, [10-min delay] (two approaches): 47 | purge: remove object, forces a full GET on next request 48 | invalidate: 49 | remove & replace IFF a newer version is available at origin. 50 | IMS ("if-modified-since") GET request to the origin 51 | CCU 52 | ECCU 53 | 54 | reporting: 55 | Monitor > URLS > Top URLs 56 | Monitor > Origin Performance 57 | Monitor > Offload 58 | Site Analyzer: actively monitors/diagnoses perf & availability 59 | Instant Check 60 | 61 | debugging: 62 | $ dig 63 | $ mtr 64 | $ curl -I -H 'Pragma:akamai-x-cache-on, akamai-x-cache-remote-on, akamai-x-check-cacheable, akamai-x-get-cache-key, akamai-x-get-extracted-values, akamai-x-get-nonces, akamai-x-get-ssl-client-session-id, akamai-x-get-true-cache-key, akamai-x-serial-no' http://www.myconnectwise.net 65 | 66 | Resolve > Diagnostic > Site Wellness 67 | - submit "gateway" (edge) error code to troubleshoot 68 | 69 | Akamai browser profile: http://edc.edgesuite.net 70 | 71 | Pragma: akamai-x-cache-on, akamai-x-cache-remote-on, akamai-x-check-cacheable, akamai-x-get-cache-key, akamai-x-get-extracted-values, akamai-x-get-nonces, akamai-x-get-ssl-client-session-id, akamai-x-get-true-cache-key, akamai-x-serial-no 72 | - tells Akamai to return X-Cache-Foo headers for debugging. 73 | 74 | X-Cache: TCP_MISS from a168-143-243-34 (AkamaiGHost/6.11.0.1-10451734) (-) 75 | ^ 76 | Akamai server IP addr 77 | X-Cache-Key: /D/3571/142636/000/origin-www.myconnectwise.net/ 78 | ^-CP billing code 79 | 80 | anything that is prefetched will have this header (grep the logs): 81 | X-Akamai-Prefetched-Object 82 | 83 | "Digital Property" (mnemonic): jmk.www.foo.com 84 | "Edge key hostname": CNAME -> jmk.www.foo.com.edgesuite-staging.net 85 | "cnamed to origin (specified in the config)" 86 | "config/metadata" (mnemonic): jmk.www.foo.com.xml 87 | rules for SR 88 | caching 89 | DPC 90 | MDR 91 | ... 92 | origin -> www.foo.com 93 | 94 | 95 | -------------------------------------------------------------------------------- /music.txt: -------------------------------------------------------------------------------- 1 | Shriekback - Nemesis 2 | Pomme https://www.youtube.com/watch?v=jIqA5sg5gZM 3 | VIDEOCLUB https://www.youtube.com/watch?v=4NOMFBRfaT0 4 | Nostalghia https://en.wikipedia.org/wiki/Nostalghia_(musician) 5 | Plastic Heart https://www.youtube.com/watch?v=4TgWtnzK46M 6 | Heroin https://www.youtube.com/watch?v=j4mn7YZE-oQ 7 | God Be You https://www.youtube.com/watch?v=4TfFLaO1ZeM 8 | Who You Talkin' To Man https://www.youtube.com/watch?v=-yF1e5FxDwQ 9 | Night Club 10 | Show It 2 Me 11 | Die in the Disco 12 | Your Addiction 13 | Interpol 14 | type o negative 15 | six feet under 16 | Julian Winding - The Demon Dance (From THE NEON DEMON OST) 17 | Leonard Cohen im your man 18 | romborama 19 | 20 | Amon Amarth 21 | Fate Of Norns 22 | The Pursuit Of Vikings 23 | 24 | Six Feet Under 25 | Maximum Violence 26 | Feasting On The Blood Of The Insane 27 | 28 | Prong 29 | Cleansing 30 | Snap Your Fingers, Snap Your Neck 31 | 32 | Sepultura 33 | Arise 34 | Murder 35 | Beneath The Remains 36 | Stronger Than Hate 37 | 38 | Slayer 39 | Christ Illusion (Explicit) 40 | Jihad 41 | 42 | Chimaira 43 | Chimaira 44 | Inside The Horror 45 | 46 | Devildriver 47 | The Fury Of Our Makers Hand (Explicit) 48 | Grindfucked 49 | 50 | Samhain: 51 | November-Coming-Fire: 52 | Let The Day Begin 53 | 54 | Danzig 55 | 6:66 Satans Child 56 | Cult w/Out A Name 57 | 58 | Nine Inch Nails 59 | Every Day Is Exactly The Same 60 | Every Day Is Exactly The Same (Sam Fog vs. Carlos D Mix) 61 | 62 | Korn 63 | Untouchables (Explicit) 64 | Thoughtless 65 | 66 | Ol' Dirty Bastard 67 | N***a Please (Explicit) 68 | Dirt Dog 69 | 70 | Genius/GZA 71 | Beneath The Surface 72 | Stringplay (Like This, Like That) 73 | 74 | M.I.A. 75 | Kala 76 | $20 77 | 78 | Deftones 79 | Around The Fur 80 | Headup 81 | 82 | The Rentals 83 | Return Of The Rentals 84 | Friends Of P. 85 | 86 | Elvis Costello 87 | Imperial Bedroom 88 | Shabby Doll 89 | 90 | Rob Zombie 91 | The Sinister Urge (Explicit) 92 | Never Gonna Stop (The Red,Red Kroovy) 93 | 94 | Freeform Five 95 | Bondi Beach: New Year's Eve '06 96 | No More Conversations (Mylo Remix) 97 | 98 | Timo Maas 99 | Loud 100 | Shifter (feat. MC Chickaboo) 101 | 102 | The Smiths 103 | Louder Than Bombs 104 | Girl Afraid 105 | Hatful of Hollow 106 | This Night Has Opened My Eyes (BBC Version) 107 | The World Won't Listen 108 | Half A Person 109 | Strangeways, Here We Come 110 | A Rush And A Push And The Land Is Ours 111 | Meat Is Murder 112 | Well I Wonder 113 | 114 | Morrissey 115 | Ringleader of the Tormentors 116 | Life is a pigsty 117 | 118 | The Cure 119 | Kiss Me, Kiss Me, Kiss Me 120 | Hot Hot Hot!!! 121 | 122 | Talk Talk 123 | Natural History: The Very Best Of Talk Talk 124 | It's My Life 125 | 126 | Talking Heads 127 | The Name Of This Band Is Talking Heads (Live) 128 | A Clean Break (Let's Work) 129 | Speaking In Tongues 130 | Pull Up The Roots 131 | 132 | Greenskeepers 133 | Polo Club 134 | 15 Minutes 135 | 136 | Phil Collins 137 | Sussudio 138 | 139 | New Order 140 | Blue Monday 141 | 142 | Yelle 143 | Pop-Up 144 | Ce Jeu 145 | Dans ta vraie vie 146 | Jogging 147 | 148 | Peaches 149 | I Feel Cream 150 | Relax 151 | 152 | Uffie – Hot Chick (Feadz Edit) (Fabriclive 33) 153 | 154 | Dirty Projectors: Stillness Is the Move Bitte Orca 155 | Acumen Nation: Gun Lover 156 | -------------------------------------------------------------------------------- /lua.txt: -------------------------------------------------------------------------------- 1 | http://www.luafaq.org/#T1.11 2 | https://en.blog.nic.cz/2015/08/12/embedding-luajit-in-30-minutes-or-so/ 3 | 4 | Data structures 5 | Userdata: represents C values in Lua 6 | Light Userdata: represents C pointers in Lua 7 | Tables have "hybrid" internals: dense(ish) array sections (1…n integer keys) of a table, are stored as literal C arrays (not hashtable), the integer keys are not stored. —"The Implementation of Lua 5.0" https://www.lua.org/doc/jucs05.pdf 8 | 9 | Function calls 10 | Similar to javascript, a "method" is not bound to its table (object) if you refrence the function name without also passing the table. 11 | 12 | function tbl:func(a) end 13 | -- is syntactic sugar for: 14 | function tbl.func(self, a) end 15 | 16 | tbl:func(a) 17 | -- is syntactic sugar for: 18 | tbl.func(tbl, a) 19 | 20 | Coroutines/async 21 | https://gregorias.github.io/posts/using-coroutines-in-neovim-lua/ 22 | 23 | Scopes 24 | upvalue = "External local variable" used in an inner function (closure). 25 | I.e. non-global, non-local, closed-over scope. 26 | Lua maintains upvalues separately from the stack, for each closure. 27 | http://www.lua.org/manual/5.1/manual.html#2.6 28 | 29 | Each execution of `local` defines new local variables. 30 | Example: Each of closure created in the loop uses a different y variable, 31 | but all of them share the same x. 32 | a = {} 33 | local x = 20 34 | for i=1,10 do 35 | local y = 0 36 | a[i] = function () y=y+1; return x+y end 37 | end 38 | 39 | Module organization 40 | LTN7: http://www.lua.org/notes/ltn007.html 41 | 42 | Use a fixed local name for the package (e.g. Public), and then assign this 43 | local to the final name of the package. 44 | 45 | local Public, Private = {}, {} 46 | Complex = Public -- package name 47 | 48 | Public.i = {r=0, i=1} 49 | function Public.new (r, i) return {r=r, i=i} end 50 | 51 | function Private.checkComplex (c) 52 | assert((type(c) == "table") and tonumber(c.r) and tonumber(c.i), 53 | "bad complex number") 54 | end 55 | 56 | function Public.add (c1, c2) 57 | %Private.checkComplex(c1); 58 | %Private.checkComplex(c2); 59 | return {r=c1.r+c2.r, i=c1.i+c2.i} 60 | end 61 | 62 | Whenever a function calls another function inside the same package (or 63 | whenever it calls itself recursively), it should access the called function 64 | through an upvalue of the local name of the package. Example: 65 | 66 | function Public.div (c1, c2) 67 | return %Public.mul(c1, %Public.inv(c2)) 68 | end 69 | 70 | Following these guidelines, the connection between the two functions does 71 | not depend on the package name. 72 | 73 | Memoize with weak references https://www.lua.org/pil/17.1.html 74 | A naive color factory generates a new color for each new request: 75 | function createRGB (r, g, b) 76 | return {red = r, green = g, blue = b} 77 | end 78 | Using the memoize technique, we can reuse the same table for the same color. 79 | local results = {} 80 | setmetatable(results, {__mode = "v"}) -- make values weak 81 | function createRGB (r, g, b) 82 | local key = r .. "-" .. g .. "-" .. b 83 | if results[key] then return results[key] 84 | else 85 | local newcolor = {red = r, green = g, blue = b} 86 | results[key] = newcolor 87 | return newcolor 88 | end 89 | end 90 | - user can compare colors using the primitive equality operator, because two 91 | coexistent equal colors are always represented by the same table. 92 | - Note that the same color may be represented by different tables at 93 | different times, if a GC cycle clears the results table. But as long as 94 | a given color is in use, it is not removed from results. So if a color 95 | survives long enough to be compared with a new one, its representation 96 | also survives long enough to be reused by the new color. 97 | -------------------------------------------------------------------------------- /berlin.txt: -------------------------------------------------------------------------------- 1 | supplies 2 | firewood https://www.kaminholz-berlin.com/brennholz-kaminholz-berlin-brandenburg-potsdam/ 3 | 4 | Stuff to do: 5 | http://www.exberliner.com/whats-on 6 | https://www.atlasobscura.com/things-to-do/berlin-germany 7 | Viktorianisches Picknick (Leipzig) 8 | 9 | culture? 10 | http://needleberlin.com/2018/03/30/danilo-rosato-berlin-portraits/ 11 | 12 | history: 13 | - "Any hill" (except Viktoria park) in Berlin is not natural, but actually 14 | a pile of rubble from WWII https://en.wikipedia.org/wiki/Schuttberg 15 | - "Any building" that does not have bullet holes is restored or post-war. 16 | 17 | Carnival / Theme parks 18 | https://www.howtogermany.com/pages/themeparks.html 19 | Europa Park 20 | Phantasialand 21 | Legoland 22 | 23 | Nachtleben: 24 | https://berlin-atonal.com/ 25 | https://de.m.wikipedia.org/wiki/Kraftwerk_Berlin 26 | about:blank 27 | Old Tresor 28 | Golden Gate : U8 ausgang Schicklerstr. "under S-Bahn, knock on door" (rec by Jo) 29 | 30 | Health: 31 | https://www.kvberlin.de/60arztsuche/index.html 32 | http://www.aerzte-berlin.de/_php/therapie30/fach.php 33 | 34 | German language: 35 | https://wickedchicken.github.io/post/german-for-programmers/ 36 | Verbs are “immer Position 2” 37 | https://www.quora.com/What-is-the-simplest-explanation-for-the-difference-between-nominative-accusative-dative-and-genitive-articles 38 | Nominativ = subject 39 | Akkusativ = direct object 40 | Dativ = indirect object 41 | Genitive = possessive 42 | Example: 43 | Consider these nouns: Der Junge, Der Knochen, Der Hund, Der Nachbar 44 | Consider this sentence: 45 | Der Junge gab dem Hund des Nachbarn einen Knochen. 46 | ^ ^ ^ ^ 47 | Nominative Dativ Genitive Akkusativ 48 | 49 | Article signals https://www.youtube.com/watch?v=6vHSJp0QlxQ 50 | article ending (after "~" means "~80% reliable") 51 | ------------------------------------------------------------ 52 | der ig ling or ismus ~ er 53 | das tum chen ma ment um lein ~ nis 54 | die heit ung keit ei schaft ion ität ik ~ ur e 55 | 56 | Füllwörtern 57 | Flickwörter 58 | also "so" 59 | eigentlich 60 | quasi "kind of" 61 | sozusagen "so to say" 62 | Modalpartikeln ("dispensable") https://www.youtube.com/watch?v=-Awhco_VHWE 63 | doch emphasis, contrary 64 | halt "just" 65 | mal polite (Höflich) 66 | eben "just" 67 | ja "really", "of course", "as you know" 68 | 69 | raster 70 | das All 71 | kennen 72 | null 73 | Urknall 74 | Quelle: source (Quellwasser, Quellcode!) 75 | nagen 76 | Koffer 77 | je 78 | hell = bright 79 | Geist = psyche / mind / ghost 80 | Essen = food / essence 81 | Art = type / kind / way 82 | vernichten 83 | senken 84 | schleppen 85 | quer 86 | entschuldigen = "unguilt" 87 | Stimmung = "inner voice" 88 | Lebensmut = "life grit" 89 | Einsam = "one peace" 90 | Wehmut = "sore mood" 91 | herz => "heart", related to "herz, 1/s" ? 92 | funzt => slang, "es funktioniert", "the sound of opening a beer bottle" 93 | 94 | Observations: 95 | No lizards 96 | No cockroaches! 97 | Lots of wide open spaces 98 | Le Crobag 99 | Wasser pumper 100 | Parks everywhere, creative and different sculptures (cobra, trolls), zip lines 101 | paradox: government workers refuse to speak English; service workers are visibly irritated if you try to speak German 102 | Bodhi builds a "bridge" across the stream, of large sticks. 103 | Goat on the roof. 104 | Young woman in park, eating a whole bell pepper held by the stem. 105 | seasons: no AC needed most days. First sign of spring is renewed grass. Then vines cover the graffiti. Then wildflowers burst in purple, yellow, orange. Then Populus tremula trees bloom, and shower fluffy pillows, the air is foggy in bright sunlight , like a spring snow. 106 | Birds are songful and warbling throughout the year. 107 | Carnival chic: sashes, airy pants, juggling, brown leather. Derivative of punk Berlin? 108 | 109 | Besucher plan 110 | grafitti / spray paint 111 | tesla factory 112 | 3d printing @ motionlab 113 | Holzmarkt 114 | hoison egg at a bar (e.g. Zum Krokodil) 115 | Hallmann und Klee "Byron bay hipster" https://goo.gl/maps/SsZ3paizyKXGP2kw5 116 | Rixdorf market (has an old church, blacksmith) https://www.diemarktplaner.de/rixdorf/ 117 | Dong Xuan 118 | Hyper-specialist shops of Berlin https://news.ycombinator.com/item?id=19785455 119 | Loretta am Wannsee 120 | 121 | arbeit 122 | https://nu.careers Nubank (clojure) 123 | -------------------------------------------------------------------------------- /nvim-newsletter-2022.txt: -------------------------------------------------------------------------------- 1 | # State of Neovim 2 | 3 | The health of Vim and Neovim is beyond what I ever expected. After all, you need 4 | a [special kind of crazy] to successfully fork a major OSS project. Neovim is an 5 | old project now, with its own legacy. I was conscious since the start 6 | that we don't want to add mere entropy to the universe: we should be doing 7 | high-leverage work, using our human attention for a pursuits that justify the 8 | existence of a fork. I'm as skeptical of the urge to fork as any wise OSS 9 | observer. Meanwhile, project like [Helix] prove a bigger notion: practically, 10 | not only spiritually--all such projects are just branches of the same idea: we 11 | want a [hitchhiker's towel], a portable, hyper-flexible multitool, the 12 | CLI-resident OS for software development work, just as the browser is the 13 | UI-resident OS for apps (because, as ever [the OS failed its job]). (And those 14 | two inner-platforms may some day merge: again, temporary branches of the same 15 | idea). 16 | 17 | So Vim, Neovim, Emacs, Helix, are all branches of the same mindbase. Any or most 18 | of these branches may eventually wither, exactly as soon as their origin becomes 19 | a "solved problem". But until then, we need all of those branches. They are 20 | computing NP-complete results and we don't know which one is best yet. 21 | 22 | Most forget that in 2014, Vim had no CI, close to zero test-coverage, and 23 | no plans to change. Neovim emerged from that urgency with a plan to turn the 24 | internals inside-out, ship a :terminal, rewrite the build system, and go all-in 25 | on Lua. But the real inflection point of Neovim's model was our "soft" choices: 26 | figuring out how to scale contributions, so that major featues could be 27 | contributed by, and maintained by, a swarm or coveyor-belt of roughly-aligned 28 | fly-by-night contributors. That ability to plug-in, and intermittently 29 | self-eject, on the timeline of a project's development, while maintaining Fred 30 | Brook's ["conceptual integrity"], is the ever-glowing ember of a successful 31 | long-term OSS project. Without it, you get either an eventual junkyard of 32 | half-broken, non-composable features, or the project just fades away. 33 | 34 | We couldn't be sure that doubling-down on Lua, or shipping a builtin LSP client, 35 | was the right decision. There was no mosh pit of encouraging bystanders (any 36 | impressions to the contrary are, I assure you, hindsight) assuring us that these 37 | leaps of faith would unlock pent-up demand and [double our userbase] in two 38 | years. We made those decisions based on engineering experience and intuition, 39 | and thought experiments starting from [principles] ("defaults matter more than you 40 | realize, even if you realize that defaults matter"). 41 | 42 | It wasn't until a year or two ago that we started seeing Neovim project ownership 43 | actually start to scale. It's not clear exactly how or why that happened; these 44 | "soft" policies which by definition involve fickle first-order componenets 45 | called "humans", are not back-traceable to a formula, but roughly it goes 46 | something like: set the stage; keep iterating and refining; put in consistent 47 | hard work and show your progress; deliver actual results; and eventually, if 48 | you're doing anything actually valuable, you'll attract a snowballing herd of 49 | smart people with good taste. 50 | 51 | Recently a friend messaged me reflecting on how thankful he was that Neovim 52 | existed, Vim likely wouldn't so that Vim-users could participate in the 53 | contemporary pantheon of developer tooling, beyond the artificially narrow scope 54 | of mere text-editing. He was so impressed with CoPilot and so happy that he 55 | could use it on C++ projects in Nvim. 56 | 57 | # Good and bad 58 | 59 | Weakenesses: merging relevant upstream Vim patches to Nvim is still too much 60 | work. We've made improvements, but we are very dependent on juggernauts like 61 | @janlazo and @zeertj, and without them, Nvim would be a much lower-quality 62 | product. 63 | 64 | Strengths: the culture of automation and build-time checks, linting and 65 | autoformatting, is in a position of strength, mostly thanks to @dundargoc, the 66 | "Highlander of ancient Nvim refactoring epics". He brought Nvim across the 67 | finish line for major, old planned refactors. Our C and Lua code is now fully 68 | auto-formatted according to our style guide. We spend essentially zero time 69 | discussing code style or formatting. (Still plenty of time on naming, 70 | though--intentionally!) 71 | 72 | We have executed on our [plan (vimconf 2021 talk)] to aggressively deprecate 73 | APIs, in order to search for long-term interface patterns and practices, while 74 | not blocking short-term progress. This results in a bit of "move fast and break 75 | things" environment, but with the breakage limited to specific subsystems. In 76 | particular, we have never made breaking changes to the RPC API, which is the 77 | most fragile because of its _disconnected_ nature (we don't want UIs to break 78 | between Nvim releases). We have written down a [path to 1.0]. We have 79 | [vim.deprecate]. We have [positioned :help as the single source of truth]. We 80 | are [developing Lua idioms]. 81 | -------------------------------------------------------------------------------- /aws.txt: -------------------------------------------------------------------------------- 1 | AWS open guide: https://github.com/open-guides/og-aws 2 | 3 | EC2 4 | Get metadata of the current EC2 instance 5 | https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html 6 | Amazon Linux ships with ec2-metadata tool: 7 | $ /opt/aws/bin/ec2-metadata 8 | Magic endpoint "169.254.169.254" gives metadata of the current instance: 9 | $ curl http://169.254.169.254/latest/meta-data/ 10 | 11 | ECS: EC2 container service 12 | The "primitive" for distributed applications: task (job) 13 | Think about "pool of resources" (cluster...), _not_ machines. 14 | https://github.com/aws/amazon-ecs-cli 15 | "Run applications locally or on ECS using the same Docker Compose file 16 | format and familiar Compose commands." 17 | 18 | Task 19 | https://forums.aws.amazon.com/thread.jspa?threadID=175819 20 | "When creating a task, you can omit the 'Host port' option (or set it to 0) 21 | and a port will automatically be chosen when it's started. You can find out 22 | which port was chosen by describing the running task." 23 | Container: 24 | easy example Docker image: https://hub.docker.com/r/nginxdemos/hello/ 25 | 26 | ssm, sessionmanager 27 | ssm without ssh: 28 | aws ssm start-session --target … 29 | ssm with ssh: 30 | ~/.ssh/config: 31 | Match Host i-* 32 | ProxyCommand ssh-ssm.sh %h %r 33 | https://github.com/elpy1/ssh-over-ssm 34 | key feature: copies public key to remote so that ssh works seamlessly. https://github.com/elpy1/ssh-over-ssm/blob/fbe6773a6aa7cd87f64fed3fa0abd1289575c34f/ssh-ssm.sh#L26-L33 35 | 36 | 37 | DevOps at amazon: 38 | key characteristics of Amazon SOA: 39 | - tools have *best-practices* baked-in, in order to improve 40 | *discoverability* and self-service (naming conventions, common 41 | protocols, failure modes, ...) 42 | - tools (services) must be self-service (tool should *not* require 43 | hand-holding for other teams to implement) 44 | - tools (services) must be decoupled from other tools (otherwise SOA is 45 | pointless) 46 | SOA is not effective unless it is combined with organizational changes: 47 | - small teams (6-8) 48 | - teams have full ownership and accountability 49 | - incentives are aligned within a given team 50 | 51 | CloudTrail 52 | S3 for medium-term storage; glacier for long-term. 53 | use pre-defined CloudFormation template 54 | 55 | Aggregate multiple AWS accounts to a _single_ S3 bucket. 56 | Enable CloudTrail in all accounts. 57 | Configure S3 bucket policy to authorize CloudTrail requests from the 58 | AWS accounts. 59 | 60 | CloudTrail vs CloudWatch vs Config 61 | ^ ^ ^config change management 62 | | `alert unusual activity 63 | `logs API calls 64 | 65 | 66 | Lambda 67 | - Don't need to worry about: servers, capacity, deployment, scaling, 68 | fault-tolerance, OS/stack updates, metrics/logging 69 | - Never pay for idle 70 | Built-in support for CloudWatch metrics/logs 71 | Lambda blueprints: templates/quickstarts 72 | Scheduled Lambdas (cron) 73 | Extensions: 74 | https://github.com/aws-samples/aws-lambda-extensions/blob/main/python-example-extension/python-example-extension/extension.py 75 | https://aws.amazon.com/blogs/compute/introducing-aws-lambda-extensions-in-preview/ 76 | hook into more of the Lambda lifecycle 77 | install as Layers 78 | example: 79 | requests.post( 80 | url=f"http://{os.environ['AWS_LAMBDA_RUNTIME_API']}/2020-01-01/extension/register", 81 | json={ 82 | 'events': [ 'INVOKE', 'SHUTDOWN' ], 83 | }, 84 | headers=headers 85 | ) 86 | 87 | Glacier 88 | Lifecycle rules: automatically archive S3 data after expiration. 89 | Best practices: Store the filename in the Description field. 90 | 91 | Xray 92 | Trace: end-to-end data related to a single request across all services 93 | Segments: trace fragment corresponding to a single service. 94 | 95 | AWS Config: visibility into configuration changes 96 | configuration item (CI): standard config format 97 | metadata 98 | common attributes (resource ID, type, ARN, zone) 99 | relationships 100 | current config 101 | related events: GUID that defines CloudTrail event 102 | CI and diff delivered to SNS on every change 103 | Config Rules: built upon AWS Config 104 | AWS-managed rules (pre-built) 105 | "All resources must be tagged" 106 | custom rules (lambda) 107 | 108 | Network trouble / cannot connect 109 | "If you associate an IPv6 CIDR block with your VPC and subnets, your route 110 | tables must include separate routes for IPv6 traffic." 111 | http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Scenario2.html 112 | 113 | 502 Bad Gateway 114 | If the ALB considers all instances "unhealthy", it will route to "all", 115 | and you will get random 502 errors depending on which node was routed-to. 116 | -------------------------------------------------------------------------------- /docker.txt: -------------------------------------------------------------------------------- 1 | ============================================================================== 2 | How to Replace Docker with Podman on a Mac 3 | https://news.ycombinator.com/item?id=28463212 4 | https://news.ycombinator.com/item?id=28464897 5 | 6 | brew install podman 7 | podman machine init 8 | podman machine start 9 | alias docker=podman 10 | 11 | IIRC the podman socket is a planned thing for macOS as it'll be necessary for certain features that rely on the docker socket and don't use SSH, such as VSCode's Remote Containers extension. 12 | However you can create an SSH tunnel to create the socket locally to allow non-podman clients to utilize the socket over SSH with the DOCKER_HOST environment variable. 13 | 14 | # Get URI 15 | podman system connection ls 16 | 17 | # Create tunnel 18 | ssh -nNT -L/tmp/podman.sock:/run/user/1000/podman/podman.sock -i ~/.ssh/podman-machine-default ssh://core@localhost:[PORT] 19 | 20 | # Export socket location 21 | export DOCKER_HOST='unix:///tmp/podman.sock' 22 | 23 | https://github.com/containers/podman/issues/11462 24 | https://github.com/containers/podman/issues/11397 25 | 26 | ============================================================================== 27 | DEVELOPMENT 28 | 29 | Build and run: 30 | 31 | docker build --rm -t fleetscanner . && docker run fleetscanner:latest 32 | 33 | Minimal container by leveraging "base image" concept: 34 | https://gist.github.com/jrockway/cceef8bb5dcef62743f8bcbc044cd2ad 35 | > Build your application in a convenient container, then copy the resulting 36 | > binary into a barebones container (nothing except SSL certificates and the 37 | > time zone database, for Go code anyway). 38 | 39 | Dockerfile: 40 | FROM golang:alpine AS build 41 | RUN apk add git bzr gcc musl-dev 42 | WORKDIR /whatever/ 43 | COPY go.mod go.sum ./ 44 | ARG GOPROXY 45 | RUN go mod download 46 | 47 | COPY . . 48 | ARG VERSION 49 | RUN go install -ldflags "-X .Version=$VERSION" -v ./your/app 50 | 51 | FROM alpine:latest 52 | RUN apk add ca-certificates tzdata 53 | WORKDIR / 54 | COPY --from=build /go/bin/app /go/bin/app 55 | CMD ["/go/bin/app"] 56 | 57 | ============================================================================== 58 | DOCKERFILE/OCI SPEC 59 | - Layers are created at the end of each RUN. If files don't exist at the end of the RUN they are not stored. 60 | 61 | Multi-stage image: use "FROM … AS …" to create a stage 62 | https://docs.docker.com/build/building/multi-stage/ 63 | 64 | Example Dockerfile: 65 | FROM some-language-runtime:major.minor.patch as base 66 | FROM base as build 67 | # here are statements that build your application 68 | FROM base as vscode-devcontainer 69 | # Install Git to be able to use it right in the development container 70 | RUN apt-get update && apt-get install git -y 71 | # install further useful tools, e.g. pre-commit.org or others 72 | FROM base as production-app 73 | COPY --from=build /some/artifact.bin /somewhere/ 74 | ENTRYPOINT something 75 | 76 | ============================================================================== 77 | OPS / TROUBLESHOOTING 78 | 79 | # Get the most "recent" container: 80 | docker ps --format json | jq -r '.ID' | head -1 81 | 82 | # Open a shell in the most recent container: 83 | docker exec -it "$(docker ps --format json | jq -r '.ID' | head -1)" /bin/bash 84 | 85 | # Show running processes in the most recent container: 86 | docker top "$(docker ps --format json | jq -r '.ID' | head -1)" 87 | 88 | # Attach to running container TTY: 89 | docker attach --sig-proxy=false 90 | 91 | # Invoke in container exposing port 3000 (uses query/filter): 92 | docker exec -i $(docker ps -q -f publish=3000) foo 93 | 94 | # Get runtime info, post-mortem, env, etc.: 95 | docker inspect 96 | 97 | ============================================================================== 98 | NETWORK 99 | 100 | Can use `lsof` to find assigned host ports: 101 | sudo lsof | grep 'docker.*TCP.*LISTEN' 102 | 103 | ============================================================================== 104 | EVENT LOG 105 | 106 | Print global docker events since 5 minutes: 107 | docker events --since "$(date +%Y-%m-%dT%H:%M:%S --date '-5 minutes')" --until "$(date +%Y-%m-%dT%H:%M:%S)" 108 | 109 | - "container die … exitCode=137" means the container was killed with SIGKILL 110 | (128 + signalNumber). Look for "container kill … signal=9" in previous messages. 111 | 112 | ============================================================================== 113 | GDB BACKTRACE 114 | 115 | To use strace, gdb, etc., the SYS_PTRACE capability must be enabled: 116 | 117 | docker run --cap-add=SYS_PTRACE --security-opt seccomp=unconfined 118 | 119 | ============================================================================== 120 | JAVA BACKTRACE 121 | 122 | send SIGQUIT (3) 123 | ---------------- 124 | 125 | # e.g. in tomcat, thread dump will be sent to /logs/catalina.out 126 | kill -3 127 | 128 | 129 | jcmd 130 | ---- 131 | 132 | To get thread dumps (callstack snapshots), we need `jcmd`. Usually included with 133 | the jdk. Try this: 134 | 135 | docker exec -it 7530bfaafa05 jcmd 136 | 137 | If that doesn't work, a workaround is to copy jcmd from the host environment 138 | into the container: 139 | 140 | docker cp /usr/lib/jvm/java-8-openjdk-amd64/ :/jdk 141 | 142 | Now call jcmd: 143 | # stacktrace 144 | docker exec /jdk/bin/jcmd 1 Thread.print 145 | # heap dump / memory analysis 146 | docker exec /jdk/bin/jcmd 1 GC.heap_dump 147 | -------------------------------------------------------------------------------- /ai_llm_machine_learning.md: -------------------------------------------------------------------------------- 1 | TODO: 2 | - Andrej Karpathy "Let's build GPT: from scratch, in code, spelled out." https://youtu.be/kCc8FmEb1nY 3 | - https://course.fast.ai/ 4 | - https://mlu-explain.github.io/ 5 | 6 | 7 | Neural networks, LLMS 8 | -------------------------------------------------------------------------------- 9 | 10 | LLM visualization: https://bbycroft.net/llm 11 | 12 | - https://x.com/karpathy/status/1835024197506187617 13 | "Language" in the name LLM is just historical. They are highly general-purpose 14 | technology for statistical modeling of *token streams*. Tokens can represent 15 | text chunks. It could just as well be little image patches, audio chunks, 16 | action choices, molecules, or whatever. 17 | - Caveat: https://x.com/ylecun/status/1835303018914324689 18 | > Auto-regressive prediction for things that are not temporal sequences 19 | > (with some temporal causality) is a pure abomination. Even for temporal 20 | > sequences, auto-regression *in input space* is inferior to 21 | > auto-regression in representation space: a dynamics is not necessarily 22 | > represented efficiently by a sequence of past inputs. 23 | 24 | - vector vs embedding: 25 | - Vectors are a general mathematical construct (list of numbers). 26 | - Embeddings are a particular kind of lower-dimensional vector "learned" by 27 | neural networks, optimized to represent semantic relationships between 28 | discrete objects like words. The network learns embeddings that _capture 29 | similarities_ between objects. 30 | - Vectors encode individual objects, while embeddings encode relationships 31 | between objects. The distances between embedded objects have meaning. 32 | Objects close together are semantically similar. 33 | - Vectors have fixed lengths, like 300 dims. Embeddings are usually lower 34 | dimensional, like 50-100 dims, as the model learns to efficiently encode 35 | objects. 36 | - Vectors are static representations, while embeddings are optimized during 37 | training to improve their encoding of relationships. 38 | 39 | 40 | - GPT = generative pretrained transformer 41 | - "transformer": marketing name for "attention net" 42 | - Detects subtle relationships amongst even distant data elements in a sequence. 43 | - "self-attention": differentially weighting each part of the input (which includes recursive output). 44 | - KEY IDEA: Unlike traditional neural networks with fixed weights, 45 | self-attention layers adaptively weight connections between inputs 46 | based on context. Allows transformers to accomplish in a single layer 47 | what would take traditional networks multiple layers. 48 | - From 2017 Google paper 49 | https://ai.googleblog.com/2017/08/transformer-novel-neural-network.html 50 | https://arxiv.org/abs/1706.03762 51 | - Trained by RLHF 52 | https://en.wikipedia.org/wiki/Reinforcement_learning_from_human_feedback 53 | 54 | 55 | Tooling, Agentic AI concepts 56 | -------------------------------------------------------------------------------- 57 | 58 | - MCP is essentially 4 message types: 59 | - `initialize` 60 | - `initialized` 61 | - `tools/list` 62 | - `tools/call` 63 | - Uses newline-delimited JSON. 64 | - "agentic discovery": instead of eagerly indexing a local vector db, send breadcrumbs to LLM 65 | https://cline.bot/blog/why-cline-doesnt-index-your-codebase-and-why-thats-a-good-thing 66 | > When you point Cline at your codebase, it reads code the way you do – file 67 | > by file, connection by connection. 68 | > 69 | > You're working on a React component. Cline reads it, sees an import, follows 70 | > it. That file imports another, so Cline follows that too. Each file builds 71 | > on the last, creating a connected understanding of how your code actually 72 | > works. 73 | > 74 | > No index or embeddings. Just intelligent exploration, building context by 75 | > following the natural structure of your code. 76 | 77 | 78 | Deep learning 79 | -------------------------------------------------------------------------------- 80 | 81 | https://news.ycombinator.com/item?id=14485362 82 | > You don’t need Google-scale data to use deep learning. Using all of the above 83 | > means that even your average person with only a 100-1000 samples can see some 84 | > benefit from deep learning. With all of these techniques you can mitigate the 85 | > variance issue, while still benefitting from the flexibility. 86 | | 87 | └─ _Transfer Learning_ 88 | "Easiest way to use deep learning without a lot of data: download 89 | a pretrained model and fine-tune the last few layers on your small 90 | dataset. In many domains (like image classification) fine-tuning works 91 | extremely well, because the pretrained model has learned generic 92 | features in the early layers that are useful for many datasets, not 93 | just the one trained on. Even the best skin cancer classifier 94 | (http://www.nature.com/articles/nature21056) was pretrained on ImageNet. 95 | | 96 | └─ "This is how the great http://course.fast.ai course begins - download VGG16, 97 | | finetune the top layer with a single dense layer, get amazing results. 98 | | 99 | └─ Remove the last "layer" of the neural net. Then you can use 100 | a classifier (e.g. SVM) to add the layer back, for your specific 101 | application. E.g. with deep learning, the NN finds features, these 102 | are in the last "hidden layer". That layer is generalized, you can 103 | use those features to train for your specific application. 104 | 105 | -------------------------------------------------------------------------------- /gwt.txt: -------------------------------------------------------------------------------- 1 | UiBinder with 'html' tag is faster than a Widget 2 | - optimization: use HtmlPanel in UiBinder, fill it with raw HTML, then just use Widgets selectively 3 | - don't use Widgets unless you need to respond to an event from that Widget 4 | 5 | 6 | CellTable 7 | cellTable.redraw() //redraws entire table, very fast, but... 8 | //redraw a single row is even faster, but be aware 9 | //the user might delete a row and the index will be off 10 | cellTable.setRowData(index, Collections.singletonList(obj)); 11 | 12 | gwt-presenter: 13 | https://code.google.com/p/gwt-presenter/source/browse/src/main/java/net/customware/gwt/presenter/client/Presenter.java 14 | 15 | gwt-dispatch: command pattern implementation 16 | https://code.google.com/p/gwt-dispatch/ 17 | http://turbomanage.wordpress.com/2010/07/12/caching-batching-dispatcher-for-gwt-dispatch/ 18 | http://turbomanage.wordpress.com/2010/07/16/dispatchqueue/ 19 | 20 | compilation 21 | precompile 22 | ... 23 | UnifyAst => AST representing the Java program 24 | compile each permutation 25 | GenerateJavaScriptAst 26 | optimization 27 | JTypeOracle: different than TypeOracle used by the generator 28 | Pruner: most effective type of optimization 29 | Finalizer: mark everything as `final`, if possible 30 | MakeCallsStatic (Devirtualizer): narrow variable types 31 | TypeTightener: narrows the dispatch target of method calls 32 | DeadCodeElimination: constant propagation, static evaluation 33 | MethodInliner 34 | SameParameterValueOPtimizer: 35 | de-parameterize methods always called with same arg value 36 | code splitting 37 | ... 38 | linking: package artifacts and compiler output 39 | 40 | 41 | incremental compilation 42 | requires 25% more RAM 43 | unlike Java's individual compilation targets, GWT does global 44 | optimization, not 45 | "monolithic incremental" compile: compromise to avoid user 46 | migration tasks (dependency resolution) 47 | granular per-class instead of per-library 48 | global indexes track "dirty" classes 49 | disadvantage: no clear separation between build system and compiler 50 | - can't 51 | GWT allows generators to read/write any file, this makes tracking 52 | by the build system impossible. 53 | - to support global-incremental compilation, many hooks 54 | were added so that the compiler knows when a generator changed 55 | a file 56 | - more caches 57 | 58 | GWT 2.7 59 | incremental 60 | Module parse 61 | Disk resource scanning (inotify) 62 | Class parse 63 | (history note: gwtar only incrementalized this) 64 | Application AST stitching 65 | Regular type indexes construction 66 | Normalization 67 | monolithic (these are complicated, diminishing returns) 68 | Type index construction for generators 69 | sourcemap construction 70 | packaging linking 71 | GWT 2.8 72 | faster initial compilation (but same steady-state time as 2.7) 73 | 74 | how to leverage 75 | slowest generators: 76 | GWTRPC generator very expensive (requires backend server sync, wipes out hot cache) 77 | alternative: protobuf 78 | AutoBean won't be as bad because it uses an intermediate interface 79 | i18n (no solution on the horizon, perhaps avoid generators for i18n 80 | in the future) 81 | gin/guice outputs lots of unstable code 82 | use latest version (may not be released yet, need to check) 83 | "use the latest gin version to make sure you are getting the 84 | fast gin generator support" 85 | code splitting improvement: 86 | "if you name your split points with the same class literal 87 | they will be grouped in the same split point" 88 | custom generators 89 | to help the compiler, ensure output is _stable_: 90 | make sure the statement and declaration order stays the same 91 | don't output random strings 92 | leave SDM running to avoid slow initial compile 93 | even with 2.8 this is helpful 94 | GWT.debugger(); breakpoints 95 | use the latest Chrome 96 | large files support 97 | improved SourceMapping 98 | 25% if full obfuscation is enabled 99 | in current versions of Chrome, can't trust 100 | the browser to always use the sourcemapped 101 | method name, so can't do full obfusc 102 | future audacious goals 103 | move generators to build-system layer 104 | fewer reruns 105 | 1.5x compile-time speedup 106 | return to "library compilation" 107 | better build-system caching 108 | fewer dependency-change surprises 109 | reduce permutation combinatorial explosion 110 | 1 code generation step for all browsers, then let DeadCodeElim 111 | remove unused blocks based on feature-detection 112 | output Closure-style JS 113 | easier to read 114 | easier interop 115 | Closure compiler optimizes better than GWT 116 | 117 | jsinterop 118 | lots of nice sugar, review video for presentation 119 | JSNI sugar: 120 | String val = js("$('input#foo').bar()"); 121 | this is _mockable_ unlike traditional JSNI "public static native 122 | foo() /*- ... -*/" 123 | 124 | "CellWidgets in the wild" 125 | Google Boulder, hiring. digi@google.com 126 | fork (on GitHub) of GWT 'showcase' project 127 | https://github.com/cdigiano/ 128 | UiRenderer 129 | composite cells 130 | 131 | tessel 132 | model generation (YAML DSL): http://ww.dtonator.org 133 | -------------------------------------------------------------------------------- /clojure.txt: -------------------------------------------------------------------------------- 1 | quick start: https://news.ycombinator.com/item?id=9806126 2 | "Making an API in Clojure using Swagger gives you a full, interactive UI 3 | and documentation for your API, while also having a schema which makes sure 4 | you know what is submitted and that it validates." 5 | Making a Swagger app with Luminus is as simple as: 6 | lein new luminus myapp +swagger 7 | cd myapp 8 | lein run 9 | once the server starts, browse to 10 | http://localhost:3000/swagger-ui/index.html 11 | to see your Swagger API 12 | 13 | quick start screencast (eclipse, ring) 14 | http://www.youtube.com/watch?v=VVd4ow-ZcX0 15 | 16 | todo: 17 | https://www.stuttaford.me/2018/02/18/a-clojure-learning-journey/ 18 | The Re-frame Guide https://purelyfunctional.tv/guide/re-frame-building-blocks/ 19 | ClojureScript for Skeptics - Derek Slager: https://www.youtube.com/watch?v=gsffg5xxFQI 20 | https://howistart.org/posts/clojure/1 21 | http://www.blaenkdenum.com/notes/clojure/ 22 | Migrating a Leiningen Project to Boot (michielborkent.nl): https://news.ycombinator.com/item?id=9671106 23 | 24 | web: 25 | https://github.com/Day8/re-frame 26 | Reagent Framework For Writing SPAs, in Clojurescript. 27 | 28 | boot: 29 | https://news.ycombinator.com/item?id=13399073 30 | My current CLJS setup is boot with boot-cljs, boot-reload and boot-cljs-repl. They're configured using normal Clojure code in a build.boot file and the entry points are configured in live-reloadable *.edn files. 31 | That setup is simpler and far more powerful than anything else I've seen in other languages. 32 | I was using leiningen previously but boot proved to be more powerful for less work in my case. 33 | https://github.com/boot-clj/boot 34 | https://github.com/boot-clj/boot-cljs 35 | https://github.com/adzerk-oss/boot-reload 36 | https://github.com/adzerk-oss/boot-cljs-repl 37 | 38 | style/idioms: 39 | https://github.com/bbatsov/clojure-style-guide 40 | 41 | tutorial 42 | http://www.youtube.com/user/Misophistful/videos 43 | 44 | book: 45 | https://github.com/clojure-cookbook/clojure-cookbook 46 | 47 | AMAZING 48 | http://himera.herokuapp.com/synonym.html 49 | http://cjohansen.no/clojure-to-die-for 50 | 51 | http://clojure.org/cheatsheet 52 | 53 | amazing book/tutorial: 54 | http://www.braveclojure.com/ 55 | 56 | deploy CLI app: 57 | http://yobriefca.se/blog/2014/03/02/building-command-line-apps-with-clojure/ 58 | http://www.rkn.io/2014/02/27/clojure-cookbook-command-line-args/ 59 | 60 | web app: 61 | http://blog.josephwilk.net/clojure/building-clojure-services-at-scale.html 62 | https://github.com/jalehman/react-tutorial-om 63 | static site generation: http://cjohansen.no/building-static-sites-in-clojure-with-stasis 64 | 65 | datomic discussion and video link: 66 | http://www.reddit.com/r/Clojure/comments/1t6h2v/datomicjunk_a_bunch_of_datomic_utility_functions/ 67 | 68 | datomic 69 | https://augustl.com/blog/2018/datomic_look_at_all_the_things_i_am_not_doing/ 70 | 71 | Solving the Expression Problem with Clojure 1.2 72 | http://www.ibm.com/developerworks/java/library/j-clojure-protocols/index.html#datatypes 73 | 74 | macros: http://aphyr.com/posts/305-clojure-from-the-ground-up-macros 75 | 76 | http://thinkrelevance.com/blog/2013/06/04/clojure-workflow-reloaded 77 | 78 | FUNDAMENTALS 79 | let is evaluated immediately 80 | all let-defined symbols are replaced with their corresponding values, 81 | then then expression is evaluated 82 | Functions represent unrealized computation: expressions not yet evaluated, or incomplete. 83 | functions exist to defer evaluation 84 | parameters are unbound symbols 85 | invoking a function binds values to those symbols 86 | 87 | if we put a quote in front of the expression, it will no longer be evaluated. 88 | '(red "nut1") ;;=> (red "nut1") 89 | quoting the expression turns it into a list, which we can manipulate with 90 | other s-expressions (code as data). 91 | (first '(red "nut1")) ;;=> red 92 | (last '(red "nut1")) ;;=> "nut1" 93 | 94 | STATE https://aphyr.com/posts/306-clojure-from-the-ground-up-state 95 | Type Mutability Reads Updates Evaluation Scope 96 | --------------------------------------------------------------------------- 97 | Symbol Immutable Transparent Lexical 98 | Var Mutable Transparent Unrestricted Global/Dynamic 99 | Delay Mutable Blocking Once only Lazy 100 | Future Mutable Blocking Once only Parallel 101 | Promise Mutable Blocking Once only 102 | Atom Mutable Nonblocking Linearizable 103 | Ref Mutable Nonblocking Serializable 104 | 105 | PATTERNS 106 | get values of a nested map (without worrying about nil): 107 | (get-in foo-map [:key1 :nestedkey1 :nestedkey2]) 108 | modify ALL keys of a nested map (nil intermediates will be stubbed): 109 | (update-in me [:address :street] #(str "33 " %)) 110 | 111 | PHILOSOPHY 112 | - This classic got Clojure on the map for many people: 113 | Are We There Yet? www.infoq.com/presentations/Are-We-There-Yet-Rich-Hickey 114 | - Simple Made Easy 115 | http://www.infoq.com/presentations/Simple-Made-Easy-QCon-London-2012 116 | - Simplicity Ain't Easy - Stuart Halloway - https://www.youtube.com/watch?v=cidchWg74Y4 117 | - Ousterhout's Dichotomy Isn't (Part 2 of the Simplicity/Power/Focus Series) - Stuart Halloway 118 | https://www.youtube.com/watch?v=cidchWg74Y4 119 | 120 | LOGGING 121 | pretty-print any expression http://emacsninja.com/posts/fixing-my-annoyance-with-clojure.html 122 | (defn dbg 123 | ([thing] 124 | (dbg "XXX" thing)) 125 | ([prefix thing] 126 | (println prefix) 127 | (print (with-out-str (clojure.pprint/pprint thing))) 128 | thing)) 129 | 130 | LIBRARIES 131 | like Haskell "Lens": composable navigation 132 | http://nathanmarz.com/blog/clojures-missing-piece.html 133 | https://github.com/redplanetlabs/specter 134 | https://github.com/vvvvalvalval/supdate 135 | https://github.com/noprompt/meander 136 | http://blog.korny.info/2014/03/08/xml-for-fun-and-profit.html#zippers 137 | clojure.zip/xml-zip => like xpath 138 | Python bindings 139 | https://github.com/clj-python/libpython-clj 140 | http://gigasquidsoftware.com/blog/2020/01/10/hugging-face-gpt-with-clojure/ 141 | -------------------------------------------------------------------------------- /hashtable.txt: -------------------------------------------------------------------------------- 1 | HASH FUNCTION 2 | ================================================================================ 3 | goal: "avalanche" with uniform distribution 4 | - cf. CRC, where the priority is _not_ uniformity of distribution, but to 5 | avoid collisions for _similar_ inputs. 6 | 7 | key concepts: 8 | * "Rotate"/"Mix" the "internal state" to improve distribution, then XOR etc. 9 | * permute internal state such that each hash result has exactly one input 10 | that will produce it. Use _every part_ of the key to compute the hash. 11 | * Avalanche: output is wildly different if input changes only slightly 12 | (single bit). 13 | * This aids distribution: similar keys will _not_ have similar hash 14 | values => uniformly distributed => minimize collisions 15 | * Completeness (ideal avalanche): Hash function is "complete" if _each_ 16 | output-bit depends on _all_ input-bits. If 1 bit of the input (plaintext) 17 | is changed, every bit of the output (ciphertext) has 50% probability of 18 | changing. 19 | 20 | 21 | "Guidelines and rules for GetHashCode" 22 | http://ericlippert.com/2011/02/28/guidelines-and-rules-for-gethashcode/ 23 | > Multiplication is nothing more than repeated bit shifts and adds; 24 | > multiplying by 33 is just shifting by five bits and adding. Basically this 25 | > means "mess up the top 27 bits and keep the bottom 5 the same" in the hope 26 | > that the subsequent add will mess up the lower 5 bits. Multiplying by 27 | > a largish prime has the nice property that it messes up all the bits. I'm 28 | > not sure where the number 33 comes from though. 29 | 30 | > I suspect that prime numbers turn up in hash algorithms as much by tradition 31 | > and superstition as by science. Using a prime number as the modulus and 32 | > a different one as the multiplier can apparently help avoid clustering in 33 | > some scenarios. 34 | 35 | AVOID specialized hash functions ("best for strings", "best for integers"); if 36 | not good for all types of data, probably a poor algorithm. 37 | 38 | BAD hash algo: Additive hash 39 | Any hash algorithm that relies primarily on a commutative (independent of 40 | order) operation will have bad distribution (e.g. abc/cba/cab hash to the same 41 | value). 42 | BAD hash algo: XOR hash 43 | Better than additive hash, but not enough internal mixing. 44 | 45 | 46 | HASHTABLE 47 | ================================================================================ 48 | Array-abstraction allowing any value to be used as an index. (cf. associative 49 | array) 50 | 51 | // Get an index for the key. 52 | size_t idx = hash(key, size) % table->size; 53 | 54 | Worst-case time of a hashtable lookup is bounded by the hash-collision strategy. 55 | - Linked-list ("chaining") is of course O(n)... 56 | - Delete is trivial. 57 | - Requires extra storage for pointers, and frustrates cache-locality. 58 | - But chaining can be improved by moving the match to front of list: 59 | "In-memory hash tables for accumulating text vocabularies" 60 | http://www.mathcs.emory.edu/~whalen/Hash/Hash_Articles/In-memory%20hash%20tables%20for%20accumulating%20text%20vocabularies.pdf 61 | - Open addressing: alt. to "chaining". Linear, Quadratic, Double-hashing... 62 | - Delete is a pain. 63 | - Some use a "mark" strategy, but then if many keys are deleted, 64 | scan-time gets terrible though the logical load-factor is low. 65 | - Thus the typical strategy is to rebuild the probe-sequence 66 | following the deleted item. 67 | - Worst-case is also O(n), but gains cache locality. 68 | - Double hashing: avoids clustering 69 | - "Cuckoo hashing": constant-time lookup (and insertion). 70 | 71 | Best-case is O(1). 72 | _Average_ expected performance of a hash table with a good hash function is 73 | betw. O(1) ~ O(log N), strong bias toward O(1). 74 | 75 | Double hashing: use a different hash function to compute step size 76 | // Get the initial index for the key. 77 | size_t idx = hash(key, size) % table->size; 78 | // Get the step size. 79 | size_t step = hash2(key) % (table->size - 1) + 1; 80 | ^-- Interval depends on the _data_ (unlike probing), so collisions 81 | have different bucket sequences. 82 | 83 | Design of redis dict.c: https://news.ycombinator.com/item?id=4599740 84 | 1. At every operation you perform there is a rehashing step. 85 | 2. You have a function that you can call to perform a rehashing step. 86 | 3. There is an higher level function where you can say "rehash for 87 | N milliseconds" in case you have some idle time. 88 | Another uncommon thing of dict.c is that it supports the "pick a random 89 | element" operation with guaranteed O(1) behaviour. 90 | * We use both chaining and incremental rehashing. This means that there is 91 | no need to rehash ASAP when the hash table reached the max % of elements 92 | allowed. At the same time using the incremental rehashing there is no need 93 | to block. 94 | * We support safe and unsafe iterators: safe iterators block the rehashing 95 | when they are active, so you have the feeling you are accessing the hash 96 | table in a read-only way. Unsafe iterators are ok for looping even while 97 | an incremental rehashing is in progress, but is not ok to touch the hash 98 | table while iterating with an unsafe iterator (otherwise the iterator no 99 | longer guarantees to return all the elements, or to avoid duplicated 100 | elements). 101 | * The hash table is rehashed both ways: if it's too full, or if it's too 102 | empty. This way there is the guarantee that a table has always 103 | a percentage between a min and max. This guarantees that our random 104 | element function is in the average case O(1). 105 | * When we rehash, a second table is created, and stuff are moved 106 | incrementally from one to the other. When we have two tables, all the 107 | lookup / delete operations are run against both tables because the element 108 | can be in either table. Insertions only happen in the new table. 109 | 110 | MORE 111 | ================================================================================ 112 | general purpose hash function 113 | MurmurHash3 C port: https://github.com/PeterScott/murmur3 114 | MurmurHash3: https://github.com/aappleby/smhasher/blob/master/src/MurmurHash3.cpp 115 | MurmurHash2 in redis: https://github.com/antirez/redis/blob/0c05436cef6f65cb2d62c8764522abefeb964314/src/dict.c#L92 116 | 117 | string hash function: 118 | https://github.com/amadvance/tommyds 119 | https://github.com/torch/luajit-rocks/commit/cafad2e668764467d7846719ee964172c4e74b0b 120 | -------------------------------------------------------------------------------- /project-stanford-pupper.txt: -------------------------------------------------------------------------------- 1 | https://pupper.readthedocs.io/en/latest/ 2 | https://github.com/stanfordroboticsclub/StanfordQuadruped 3 | 4 | TODO: 5 | passwd 6 | secure sshd_config 7 | https://github.com/PetoiCamp/OpenCat 8 | 9 | parts: 10 | x raspberry pi 4, model B, 2GB ram 11 | x power connector adapter: XT60 male to Deans(T) female 12 | x lipo battery alarm: attach to battery to avoid completely discharging the battery (dangerous) 13 | https://www.youtube.com/watch?v=tTkT1fSAy9Y 14 | x 3d printed 15 | x soldering iron 16 | x raspi 17 | multimeter 18 | (optional) IMU setup: BNO080 IMU + Teensy LC 19 | x feet: for slippery surface, get rubber grommets (McMaster #90131A101) and fasten them to the pre-drilled holes in the feet. 20 | x "Any sort of rubber disc/washer. Attach with short M3x8 screws and locknuts." 21 | 22 | LIPO BATTERY: 23 | Always charge in BALANCE mode. 24 | DISCHARGING TOO MUCH = risk of fire. 25 | Below 3V per-cell => permanent degradation. 26 | Set "low-voltage cutoff" to 3.4V per cell. 27 | Safest charge-rate for LiPo batteries is 1C (1x Amps capacity of battery). 28 | => for 5200mAh battery use 5.2A 29 | if charger doesn't support 5.2A it's fine to use a lower value, just takes longer 30 | Full charge = 4.2v/cell 31 | STORAGE: 32 | Always set LiPo to "storage charge" if you won't use it same-day. 33 | Proper LiPo storage voltage = 3.8V per cell 34 | LiPo battery can be damaged by sitting fully charged for as little as 1 week. 35 | 36 | HARDWARE 37 | servo motor "crackling" noise: 38 | https://www.rctech.net/forum/nitro-road/283356-servo-noise-normal.html 39 | > The crackling is perfectly normal with digital servos. It usually happens 40 | > when there is a very slight load on the servo and it can't quite reach 41 | > neutral perfectly to stop calling on the motor. There isn't any negative 42 | > effect, but as a rule I wouldn't leave it loaded and crackling for minutes 43 | > at a time. 44 | 45 | 46 | SOFTWARE 47 | SETUP 48 | LOGIN 49 | default login: pi / raspberry 50 | INTERNET 51 | Edit /etc/wpa_supplicant/wpa_supplicant.conf 52 | problems: 53 | # wlan fails in ro filesystem: 54 | Feb 14 10:19:29 oppy5 dhclient[2930]: can't create /var/lib/dhcp/dhclient.wlan0.leases: Read-only file system 55 | $ ssh pi@10.0.0.99 # ethernet 56 | $ ssh pi@192.168.2.110 # wifi 57 | use of `dd` to flash the sd card: https://www.raspberrypi.org/documentation/installation/installing-images/mac.md 58 | $ sudo dd bs=1m if=2019-09-26-raspbian-buster-lite.img of=/dev/rdisk2 ; sync 59 | "using latest raspbian OS: The only thing that needed to be corrected were missing udev rules for the PS4 controller. Otherwise the software instructions were sufficient." https://groups.google.com/d/msg/stanford-quadrupeds/e01e9xvb4V4/2o81D3CeBwAJ 60 | # update wifi: 61 | sudo raspi-config 62 | # At boot, files are overwritten with /boot/appliance/*. 63 | # To preserve wifi settings: 64 | sudo cp /etc/wpa_supplicant/wpa_supplicant.conf /boot/appliance/etc/wpa_supplicant/wpa_supplicant.conf 65 | # control ps4 colors, ...: 66 | vim StanfordQuadruped/pupper/HardwareConfig.py 67 | 68 | behavior 69 | Q1: Robot walks too fast. How can I control the speed of the robot? 70 | A1: You can adjust the frequency of the trot in the config file here: 71 | https://github.com/stanfordroboticsclub/StanfordQuadruped/blob/master/pupper/Config.py. 72 | Look for two variables: self.overlap_time and self.swing_time 73 | 74 | USAGE LOG 75 | - development 76 | ``` 77 | $ rw 78 | $ vim calibrate.py 79 | $ python3 calibrate.py 80 | ``` 81 | - physical access: keyboard + micro HDMI cable to monitor 82 | - ssh access 83 | - hostname = oppy5 84 | - Can use the ipv6 address listed by router status page. 85 | ``` 86 | $ ssh pi@2003:c1:df39:xxxx:xxxx:xxxx:xxxx:xxxx 87 | ``` 88 | - ipv4 address may be wrong (or delayed), doesn't match "ifconfig" on the raspi device. 89 | - dhcp always fails at startup (readonly filesystem) 90 | - can "fix" it temporarily with physical access: 91 | ``` 92 | rw 93 | sudo dhcpcd 94 | ``` 95 | - show power status https://elinux.org/RPI_vcgencmd_usage 96 | ``` 97 | $ vcgencmd commands 98 | $ vcgencmd measure_temp 99 | temp=52.0'C 100 | $ vcgencmd measure_volts 101 | volt=0.8718V 102 | $ vcgencmd get_throttled # Non-zero = power supply issue. https://gist.github.com/Paraphraser/17fb6320d0e896c6446fb886e1207c7e 103 | throttled=0x0 104 | ``` 105 | - control LEDs 106 | ``` 107 | # List available "modes" for LED 0. 108 | $ cat /sys/class/leds/led0/trigger 109 | [none] kbd-capslock timer oneshot heartbeat backlight gpio cpu cpu0 cpu1 cpu2 cpu3 default-on input mmc0 110 | $ sudo sh -c 'echo none > /sys/class/leds/led0/trigger' # LED 0: default off 111 | $ sudo sh -c 'echo heartbeat > /sys/class/leds/led0/trigger' # LED 0: "heartbeat" mode 112 | $ sudo sh -c 'echo 1 > /sys/class/leds/led0/brightness' # turn on green LED 113 | $ sudo sh -c 'echo 0 > /sys/class/leds/led1/brightness' # turn off red LED 114 | ``` 115 | - problem: DNS failure ("git clone" fails, "apt update" fails). 116 | ``` 117 | $ git clone https://github.com/justinmk/config 118 | Cloning into 'config'... 119 | fatal: unable to access 'https://github.com/justinmk/config/': Could not resolve host: github.com 120 | ``` 121 | - solution: 122 | ``` 123 | # Manually fix date. 124 | $ date -s '2023-12-31 16:00:00' 125 | # Manually configure DNS server(s). 126 | /etc/dhcpcd.conf: 127 | static domain_name_servers=8.8.8.8 1.1.1.1 192.168.1.1 128 | $ sudo systemctl daemon-reload 129 | $ sudo service dhcpcd restart 130 | ``` 131 | - How to detect "interactive" mode to change system config during startup? 132 | - XXX: detect battery vs mains/wall power. 133 | - Detect keyboard attached. 134 | ``` 135 | _has_kbd() { 136 | [ -d /dev/input/by-path/ ] && find /dev/input/by-path/ -name '*kbd' | grep 'kbd$' 2>/dev/null 137 | } 138 | ``` 139 | - Example startup logs: 140 | ``` 141 | oppy5 robotstatus.sh[342]: /dev/input/by-path/platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.1:1.0-event-kbd 142 | oppy5 robotstatus.sh[342]: keyboard detected, configuring "interactive" setup. 143 | ``` 144 | - Pair the PS4 controller: https://pupper.readthedocs.io/en/latest/guide/operation.html 145 | 1. Hold the "share" button and circular Playstation button until the light makes "double flashes". 146 | 2. When the controller binds to the robot, it should turn "deactivated" color (as set in `HardwareConfig.py`). 147 | 3. Press L1 on the controller to “activate” the robot. The controller should a different color (as set in `HardwareConfig.py`). 148 | -------------------------------------------------------------------------------- /nvim_segment_fellowship.txt: -------------------------------------------------------------------------------- 1 | blog 2 | ---- 3 | - What is it you’re building? 4 | I'm working on Neovim, a cooperative fork of the Vim text editor, designed to 5 | encourage contributions and extensions. Besides maintenance (reviewing and 6 | integrating pull requests, fixing bugs, maintaining the build system, 7 | supporting users), I'm working on a major new feature: multicursor support. 8 | - Why is it important? How will people use it? 9 | Multicursor support in Neovim (and Vim, eventually) will change the way users 10 | and plugin authors use vim, because (1) the design of multicursor requires 11 | exposing user actions as "atoms" and (2) cursor states are managed as 12 | contexts. These new concepts--atoms and contexts--are valuable for use-cases 13 | beyond multicursor. For example, because the vim motion "3j" is now tracked as 14 | an atom, you can repeat that motion with a keybinding that repeats the tail of 15 | the atoms queue. Another example: because Neovim now has a model of "context", 16 | we can pass contexts to peer `nvim` instances (even remote instances). 17 | - What progress or things did you build over the course of the fellowship? 18 | - Modeled vim actions as atoms, exposed via the Nvim API 19 | - Modeled vim state as a context object, exposed via the Nvim API 20 | - Implemented multicursor feature for normal-mode and insert-mode 21 | - Improved logging/visibility of the UI and event-loop subsystems 22 | - Fixed related event-loop bugs 23 | - Where do you see this going in the next 6 months? 24 | Based on feedback from core contributors and users, we will (1) adjust how 25 | various user actions are defined as atoms, and (2) adjust behavior of multiple 26 | cursors under certain circumstances (such as when a cursor hits the edge of 27 | a buffer). I also plan to to re-use the upcoming work on "Extended Marks" for 28 | resolving cursor positions in the presence of text changes. 29 | - What was the hardest or most unexpected part of what you built? 30 | The implementation of vim's editing model is ad-hoc: the "state transitions" 31 | are mostly implicit, and dependendent on global state flags. 32 | Most projects that re-implement vim (e.g. https://github.com/vrapper/vrapper), 33 | model operations as "pipelines" that produce nice, composed Commands which are 34 | then Executed. But Vim (Neovim) wasn't implemented like that. For example: 35 | many normal-mode operations are implemented by pushing keys onto the internal 36 | input queue, as if the user had typed them. Clever, but hard to reason about. 37 | 38 | I ended up throwing out three different prototypes. After I accepted the idea 39 | of "annotating" the operations at their implementations, by tracking them in 40 | a global queue, the problem became manageable. 41 | 42 | Here's a gif showing (1) placement of 2 extra cursors, (2) cursor-local 43 | registers, (3) cascade of normal-mode commands, and (4) cascade of insert-mode 44 | input. 45 | 46 | https://asciinema.org/a/4oj9EHJs5izZ8QuhH3XaMxU74 47 | 48 | Regarding this passage from my previous email: 49 | 50 | > Most projects that re-implement vim (e.g. https://github.com/vrapper/vrapper), 51 | model operations as "pipelines" that produce nice, composed Commands which are 52 | then Executed. But Vim (Neovim) wasn't implemented like that. For example: 53 | many normal-mode operations are implemented by pushing keys onto the internal 54 | input queue, as if the user had typed them. Clever, but hard to reason about. 55 | And hard to leverage programmatically. 56 | 57 | `nv_optrans()` is a concrete example of that. When you type the `x` built-in 58 | normal-mode command--which one might assume is a primitive operation--it is 59 | actually translated by `nv_optrans()` to an equivalent sequence, `dl`. The `dl` 60 | sequence is pushed onto a special-purpose internal buffer (`stuffReadbuff()`). 61 | When the main loop checks for another "input" char, it will notice these new 62 | characters. 63 | 64 | Atoms API 65 | ========= 66 | 67 | Diagram: 68 | normal-mode 69 | user input 70 | execute command 71 | push command atom onto atoms queue 72 | foreach non-primary cursor: 73 | execute command 74 | 75 | 76 | The key achievement of atoms is to chunk user commands into repeatable things. 77 | When Vim processes user input, it executes the input and then immediately 78 | forgets any understanding of what it processed. When a macro is recorded, only 79 | the raw input stream is stored: 80 | 81 | macro: ll3jddgUUiabc 82 | 83 | With the multicursor work, Nvim now tracks the atoms: 84 | 85 | macro: ll3jddgUUiabc 86 | ^^^ ^ ^ ^ atoms 87 | 88 | User scripts, plugins, and remote clients can query the atoms queue at any time 89 | via the Nvim API. 90 | 91 | For example, here's the last 3 items in the atoms queue from a sample editing 92 | session: 93 | 94 | :echo nvim_get_atoms()[-3:] 95 | [{'seq': 1, 'type': 8, 'keys': 'k'}, {'seq': 2, 'type': 10, 'keys': '^D'}, {'seq': 1, 'type': 8, 'keys': '3j'}] 96 | 97 | You can see that this is extremely useful for applications beyond multicursor, 98 | such as introspection of user editing patterns, or a mapping that repeats the 99 | last "thing" . Using the current implementation, the following code maps "space" 100 | to repeat the last atom (unlike dot-repeat, this repeats motions, navigation, 101 | etc.): 102 | 103 | :nnoremap :call feedkeys(filter(nvim_get_atoms(),'v:val.keys!~#":"')[-1].keys) 104 | 105 | Context API 106 | =========== 107 | 108 | In the current implementation, context is used internally to manage cursor-local 109 | registers. It's also exposed via the nvim_get_context() API function: 110 | 111 | :echo nvim_get_context() 112 | { 'pos': {'lnum': 0, 'col': 0, 'coladd': 0}, 113 | 'registers': {'a': {'lines': ['current_SID']}, 114 | 'b': {'lines': ['vim_FullName']}, 115 | 'c': {'lines': ['"apjwldi)kllllllllllldrf)j^C^C']}, ..., 116 | '0': {'lines': ['VV_TYPE_FUNC']}, 117 | '1': {'lines': ['foo']}, ..., 118 | } 119 | } 120 | 121 | Before now the only way to inspect Nvim state was by parsing the "shada" file 122 | (Nvim replacement of Vim "viminfo"), or by gathering bits and pieces exposed in 123 | the VimL functions. And some imporant pieces aren't exposed at all. 124 | 125 | With context as a first-class feature, we can start to think about serialization 126 | scenarios: "bootstrapping" child Nvim instances for multiprocessing; sharing 127 | state between peer instances; persisting state for fast restart; and less 128 | obvious ideas ... 129 | 130 | 131 | Developer Experience (DX) 132 | ------------------------- 133 | - DX first: API and architecture are the killer features (not job control, 134 | terminal, 'inccommand', ...) 135 | - portable install on all OSes 136 | - actualvim, VSCodeVim, Oni 137 | - next: intellij... 138 | 139 | LSP ("codeservice") 140 | -------------------------- 141 | 142 | LSP is the protocol, but the protocol is the least interesting part. 143 | The important core features: 144 | ! auto-detection of LSP backend 145 | ! healthcheck 146 | - PR: notifications 147 | - PR: floating window 148 | - PR: granular buffer-change notifications 149 | - popup menu 150 | - refactoring 151 | - rename symbol: mapping, workflow 152 | - : mapping, workflow 153 | - "outline" feature: mapping (gO) 154 | - "find references": mapping, workflow 155 | 156 | mcursor_set( 157 | {list of getpos()-like tuples}) 158 | mcursor_get() => list of getpos()-like tuples 159 | [ 160 | [{lnum}, {col}, {off}, {curswant}], 161 | [{lnum}, {col}, {off}, {curswant}], 162 | ..., 163 | ] 164 | 165 | -------------------------------------------------------------------------------- /unix.txt: -------------------------------------------------------------------------------- 1 | Liberating the Smalltalk lurking in C and Unix: https://www.youtube.com/watch?v=LwicN2u6Dro 2 | 3 | procps https://gitlab.com/procps-ng/procps : 4 | provides ps, top--but also lesser-known things like w, pmap, pwdx, slabtop. 5 | 6 | graph processes/fifos/pipes/sockets https://github.com/zevv/lsofgraph 7 | sudo lsof -n -F | ./lsofgraph | dot -Tpng > foo.png 8 | 9 | IPC 10 | Beej's Guide to Unix IPC http://beej.us/guide/bgipc/ 11 | 12 | STDIO buffering http://www.pixelbeat.org/programming/stdio_buffering/ 13 | Default buffering modes: 14 | stdin : buffered (line-buffered if TTY) 15 | stdout: buffered (line-buffered if TTY) 16 | stderr: always unbuffered 17 | Default buffer size: 18 | - based on the kernel page-size (typically 4096) 19 | - TTY-connected stdin/stdout default size = 1024 20 | Example: 21 | $ tail -f access.log | cut -d' ' -f1 | uniq 22 | - `cut` stdout buffer collects 4096-byte chunks before sending to uniq. 23 | `tail` would also have this problem, except `tail -f` calls fflush() 24 | on the stdout stream when new data is received (as do `tcpdump -l`, 25 | `grep --line-buffered` and `sed --unbuffered`). 26 | - `uniq` stdout buffer is TTY-connected, so it is automatically flushed when a newline is written to it. 27 | `stdbuf` command (uses setvbuf()) can control this: 28 | $ for i in $(seq 1 99); do printf "$i" ; sleep 1 ; done | stdbuf -o0 head 29 | 30 | 31 | show processes in "D" state (waiting-on-IO, "U" on macOS) every 5: http://bencane.com/2012/08/06/troubleshooting-high-io-wait-in-linux/ 32 | for x in `seq 1 1 10`; do ps -eo state,pid,cmd | grep "^D"; echo "----"; sleep 5; done 33 | 34 | The autotools/pkgconfig dance: 35 | # generate build files 36 | sudo apt install autoconf libtool 37 | aclocal 38 | autoconf # requires libtool 39 | automake --add-missing 40 | autoreconf -vfi # kick it if something went wrong 41 | # build it… 42 | ./configure 43 | make 44 | sudo make install 45 | # verify that the lib was installed 46 | ldconfig -p | grep foo 47 | sudo ldconfig -v # kick it if something went wrong 48 | 49 | The cmake dance: 50 | mkdir build 51 | cd build 52 | cmake ../ 53 | 54 | debug 55 | strace: relative timestamps, log to file 56 | strace -o log -r python -c 'import time; time.sleep(1);' 57 | 58 | show all open() calls performed by foo and descendants 59 | strace -f -e open foo 60 | # ^ ^ event (syscall) filter 61 | # follow subprocesses 62 | 63 | strace -k shows application backtrace (requires strace 4.9 +libunwind) 64 | strace -k -e trace=chdir nvim -u NONE -echdir +":q" 65 | chdir("/home/raf") = 0 66 | > /lib64/libc-2.20.so(chdir+0x7) [0xd8d27] 67 | > /usr/bin/nvim(uv_chdir+0x9) [0x1ac3e9] 68 | > /usr/bin/nvim(init_homedir+0x62) [0xa0c82] 69 | > /usr/bin/nvim(early_init+0xae) [0x15c47e] 70 | > /usr/bin/nvim(main+0xa9) [0x33129] 71 | > /lib64/libc-2.20.so(__libc_start_main+0xf5) [0x21b45] 72 | > /usr/bin/nvim(_start+0x29) [0x354f8] 73 | 74 | which files/sockets is a given program using? 75 | lsof -c foo 76 | which processes are using a given port? 77 | lsof -i tcp:8080 -P 78 | which processes are using a file? 79 | lsof /path/to/file 80 | lsof +D /path/to/dir 81 | # linux 82 | fuser -u $(which bash) 83 | /usr/local/bin/bash: 47327(justin) 47349(justin) 84 | 85 | macos 86 | snapshot/sample a running process 87 | sample 49814 -f out.log 88 | hold option/alt + click wifi symbol 89 | $ networkQuality 90 | ==== SUMMARY ==== 91 | Uplink capacity: 25.120 Mbps 92 | Downlink capacity: 128.988 Mbps 93 | Responsiveness: Low (61 RPM) 94 | Idle Latency: 32.708 milliseconds 95 | 96 | linux 97 | Linux perf 60-second checklist https://medium.com/netflix-techblog/linux-performance-analysis-in-60-000-milliseconds-accc10403c55 98 | uptime # load of "0.22 0.00 0.00" means "system was idle until recently" 99 | dmesg -wH # kernel often says what's wrong 100 | vmstat 1 101 | mpstat -P ALL 1 # CPU balance 102 | pidstat 1 # per-CPU work 103 | iostat -xz 1 104 | - r/s, w/s, rkB/s, wkB/s: Delivered reads, writes, read Kbytes, and 105 | write Kbytes per second to the device. 106 | - await: Average time for I/O in milliseconds. This is the time that 107 | the application suffers, as it includes both time queued and time 108 | being serviced. Large average times can indicate device 109 | saturation, or device problems. 110 | - avgqu-sz: The average number of requests issued to the device. 111 | Values greater than 1 can be evidence of saturation (although 112 | devices can typically operate on requests in parallel, especially 113 | virtual devices which front multiple back-end disks.) 114 | - %util: Device utilization. This is really a busy percent, showing 115 | the time each second that the device was doing work. Values 116 | greater than 60% typically lead to poor performance (which should 117 | be seen in await). ~100% may indicate saturation. But if the 118 | storage device is a logical device fronting many backend disks, 119 | then 100% utilization may just mean that some I/O is being 120 | processed 100% of the time, however, the back-end disks may be far 121 | from saturated, and may be able to handle much more work. 122 | free -h 123 | - buffers: buffer cache used for block device I/O. 124 | - cached: page cache used by file systems. 125 | sar -n DEV 10 126 | - network interface throughput: rxkB/s and txkB/s, as a measure of workload 127 | sar -n TCP,ETCP 10 128 | - active/s : Locally-initiated TCP connections per second (e.g. connect()). 129 | - passive/s : Remotely-initiated TCP connections per second (e.g. accept()). 130 | - retrans/s : TCP retransmits per second. Indicative of a network or 131 | server issue (unreliable network, or server overloaded and dropping 132 | packets). 133 | Perf-triggered core dumps! https://github.com/Microsoft/ProcDump-for-Linux 134 | # Trigger core dump at CPU >= 65%, up to 3 times, 10-second intervals. 135 | procdump -C 65 -n 3 -p 1234 136 | CPU perf/debug 137 | perf top -d8 --stdio # -g shows callstack 138 | kernel debug 139 | slabinfo, slabtop : kernel heap objects 140 | network perf/debug 141 | netstat -s 142 | tcpretrans 143 | network: simulate slow perf using `tc` ("traffic control") https://stackoverflow.com/a/615757 144 | # `tc qdisc add` creates a rule of the interface. `change` modifies the rule. 145 | tc qdisc {add,change} dev {eth0,wlan0,…} root netem delay 2000ms 10ms 25% 146 | # undo 147 | tc qdisc del dev {eth0,wlan0,…} root 148 | # Show all TCP,UDP sockets in LISTEN state. 149 | ss -tunpo state listening 150 | 151 | Linux scheduler 152 | Scheduler maintains hierarchicy of "scheduling domains" based on the physical _hardware_ layout. 153 | Higher-level scheduling domains group physically adjacent scheduling domains, such as chips on the same book. 154 | migration = moving a virtual CPU from one run-queue to another 155 | happens when the wait time is too long, run-queue is full, or 156 | another run-queue is under-utilized 157 | 158 | mmap https://lobste.rs/s/gfllo3/use_mmap_with_care 159 | mmap dedicates a region of address space to some kind of backing storage. 160 | The particular kind of backing storage makes all the difference. 161 | People colloquially use "mmap" to mean "mmap of a conventional disk file", but that is only a particular case. 162 | * sbrk is a legacy path that essentially amounts to an anonymous mmap. 163 | 164 | the mmap() model sort of assumes no errors happen. This breaks when files are 165 | truncated and you get SIGBUS. It can also break if files disappear (failing 166 | media or removed USB stick) or network filesystem. 167 | 168 | context switch 169 | store/restore the state (context) of a process so that execution can be resumed later. 170 | 171 | 1. save execution state (all registers and many special control structures) 172 | of current running process to memory 173 | 2. load execution state of other process (read in from memory). 174 | 3. TLB flush (if needed) is a small part of total overhead. 175 | 176 | 5-7 microseconds 177 | 178 | cache invalidation is The Largest Contributor to context-switch costs 179 | 180 | Indirect costs: 181 | loss of effectiveness for all "cache like" CPU state: caches, TLBs, branch 182 | prediction stuff (branch direction, branch target, return buffer), etc. 183 | -------------------------------------------------------------------------------- /gwtcreate.txt: -------------------------------------------------------------------------------- 1 | research: "tree-shaking does not play well with reflection" 2 | 3 | http://www.gwtproject.org/release-notes.html 4 | http://www.techempower.com/blog/2013/03/26/everything-about-java-8/ 5 | 6 | keynote 7 | ======= 8 | 9 | Cromwell said unequivocally that if Dart maintains a 2-3x speedup over v8, GWT will compile to it. 10 | 11 | ~650 conference attendees 12 | 3x increase in external contrib since creation of steering committee 13 | 1000s of internal google users 14 | - Google Shopping Express 15 | - new Google spreadsheets is written in GWT! 16 | - cross-compiled betw GWT <-> j2objc 17 | - large code-sharing of Java logic betw server and multiple client targets 18 | ~100k users of SDK 19 | 20 | avoid JSNI with @JsInterface, @JsProperty 21 | 22 | future: zero-work interop => drop an external JS library into GWT, interfaces are automatically generated 23 | 24 | //create three buttons and map event handlers to each of them! 25 | Stream buttons = Stream.of("one", "two", "three").map(Demo::create); 26 | buttons.each((b). -> b.addEventLIstener("click", (e) -> ...)) 27 | 28 | future: drop external JS library into GWT 29 | - compiler performs dead-code elimination on the external lib 30 | - type-checking on the external lib! 31 | 32 | code splitting improvements 33 | "if you accidentally create split points between two components that share code, 34 | the shared code goes into the leftovers fragment, and this almost kills the benefit 35 | because the leftovers fragment has to be downloaded anyways. tedious, brittle." 36 | 37 | compiler improvements: target modern browser optimizations 38 | - TypedArrays: new int[64] => new Int32Array(64) 39 | - output asm.js-like code! (type annotationson variables) 40 | - merging in the Closure compiler backend! 41 | speed 42 | - incremental compilation time proportional to the number of of files changed 43 | 44 | layout 45 | - "Native" layout instead of JS layout 46 | - GssResource => CSS3 support 47 | 48 | making mobile fly 49 | ================= 50 | chrome://tracing 51 | tinyurl.com/abouttracing 52 | more powerful than DevTools for debugging jank 53 | 54 | JS performance dominates, fix it first by profiling your bottlenecks, _then_ look at: 55 | DOM -> Style -> Layout (aka Reflow) -> Record -> Composit -> GPU 56 | /\ 57 | raster thread 1 -- raster thread N 58 | Desktop profiling does NOT translate to mobile characteristics 59 | 60 | text auto-sizing on mobile browsers (Chrome, Firefox) comes at a cost (~2x layout time) 61 | to avoid text auto-sizing: 62 | 63 | unfortunately, disables zooming 64 | 65 | double-tap detection causes 300ms delay on first tap 66 | 67 | RAF (request animation frame) 68 | synced to v-sync 69 | webkit-requestanimationframe 70 | GWT animation class updated to use RAF 71 | window of 16 ms / 60hz (possibly 8ms in the future as displays improve to 120hz) 72 | use this instead of setTimeOut() 73 | 74 | squashing repaints 75 | avoid repaints by using _layers_ 76 | just "move a layer" instead of repaint/resize 77 | 78 | DOM viewport _includes_ what's offscreen (~8k pixels) 79 | keep DOM small, "page-in lazily" 80 | 81 | use "infinite scroll" approach even for regular pages: 82 | page in offscreen portions _after_ onscreen portion, makes initial 83 | render much faster, while still avoiding jank. 84 | 85 | compiler 86 | ======== 87 | optimization pass (last 3-4 steps in the diagram) are what are worked on, the other 88 | first steps are not touched. 89 | 90 | GWT Java AST com.google.gwt.dev.jjs.ast 91 | class Foo { 92 | int x; 93 | void bar() { 94 | x = 2 + 5; 95 | } 96 | } 97 | JClassType(Foo) 98 | JField(x) Jmethod(bar) 99 | JMethodBody 100 | JBlock 101 | JStatement 102 | JExpression 103 | JBinaryExpression 104 | JFieldRef JBinaryOperation 105 | JInLitral(2) JInLiteral(5) 106 | 107 | GWT optimization: 108 | run optimization pass in a loop until "fixed point" (diminishing returns) is reached 109 | -o9 compiler flag controls how long this loop runs 110 | 111 | JModVisitor 112 | all GWT optimizations implement this 113 | 114 | DeadCodeElimination class 115 | simplest part of the compiler, best starting point for contributors 116 | constant-folding: 117 | "if I know LHS and RHS are both int literals, I can replace them": 118 | ctx.replaceMe(new JInLiteral(node.getSourceInfo(), 119 | ((JInLiteral) node.getLhs()).getValue(), 120 | ((JInLiteral) node.getRhs().getValue()), 121 | )); 122 | 123 | Pruner 124 | Finalizer: if nothing is assigned to a field, marks it as final 125 | any method that is not overridden, mark it as final 126 | rewrite non-static methods as static functions if possible 127 | (polymorphism is expensive) 128 | allows inlining 129 | TypeTightener 130 | 131 | "type explosion" is not really a concern for client code because the compiler 132 | knows all calls and can replace interfaces with concrete implementations 133 | 134 | Code Splitting 135 | works similar to the Pruner 136 | everything that is touched from the AST root is "alive", the rest is eliminated 137 | 138 | 139 | debugger / SuperDevMode 140 | ======================= 141 | intellij debugger seeing/planned active support, no eclipse debugger 142 | 143 | 144 | 145 | 146 | 147 | 148 | 149 | force breakpoint in pretty-mode: 150 | GWT.debugger() 151 | 152 | 153 | compiler #2 ********REVIEW => lot of good material @ 15 minutes ********** 154 | ========================================================================== 155 | compiler speed improvements focused on _draft_ compilation 156 | so normal compilation still needs to cycle through all the optimization passes 157 | 158 | modularization (*.gwt.xml files) helps (will help) the compiler 159 | break circular references 160 | make your tree _shallow_ and _broad_ 161 | use buck/gradle => build parallelism 162 | 163 | split points 164 | profile-informed split-point calculation 165 | 2.6 flag to merge fragments less than a certain size 166 | "left overs" fragment is a killer 167 | having too many split points is _detrimental_ 168 | - more likely that "left overs" fragment touches many split points 169 | - therefore merging split points can be beneficial 170 | 171 | 172 | qQuery ****REVIEW THIS TALK****** 173 | ================================================================================ 174 | uses CSS3 animations when possible (cf. jQuery uses timers) 175 | 176 | gQuery feature: DCE for 'browser' object 177 | if (browser.webkit) { 178 | ... 179 | } else { //eliminated if browser is not webkit 180 | ... 181 | } 182 | 183 | //gQuery alternative to JSNI: 184 | JsUtils.runJavascriptFunction(...) 185 | 186 | JS libraries ported to gQuery/GWT: 187 | GwtChosen 188 | tooltips 189 | drag/drop 190 | 191 | mgwt 192 | ================================================================================ 193 | keeps DOM very small 194 | supports CSS3 195 | adds at least 14 permutations for mobile 196 | REVIEW ==> "web components solve the heavy event-propogation problem" 197 | last 20 minutes: demo of using "d8" profiling method 198 | 199 | Douglas Crockford: "Javascript did a better job than the JVM at providing 200 | bytecode security, perhaps because it works at a higher level" 201 | 202 | 203 | MISSED TALK (REVIEW): JsInterop 204 | ================================================================================ 205 | 206 | day 2 207 | ===== 208 | uberfire: client-side jgit workbench 209 | http://droolsjbpm.github.io/uberfire/ 210 | 211 | Contributing to GWT 212 | =================== 213 | gerrit: https://gwt-review.googlesource.com 214 | instead of using Eclipse maven, just generate an Eclipse project file at the command line: 215 | $ mvn eclipse:eclipse 216 | 217 | Check out source from 218 | git clone https://gwt.googlesource.com/gwt-site 219 | 220 | Find the source files for the pages at src/main/markdown 221 | 222 | Apply your changes 223 | Build it with mvn clean install 224 | Verify your changes in target/generated-site 225 | 226 | GWT panel: 227 | ================================================================================ 228 | Cromwell: "GWT widgets are expensive; the only reason they were added is to 229 | prevent memory leaks in IE 6; would like to remove them" 230 | -------------------------------------------------------------------------------- /python.md: -------------------------------------------------------------------------------- 1 | CONCEPTS 2 | ================================================================================ 3 | 4 | ## Is python single-threaded? 5 | 6 | No. Even with the GIL, the active thread can change at any time between bytecode 7 | instructions. 8 | 9 | n = 1 10 | n = n + 1 11 | assert n = 2 12 | 13 | If you run this in many threads it will fail eventually, when one thread assigns 14 | `n=1` then gets interrupted by a different thread checking `assert`. 15 | In JavaScript every called function is executed until it returns, throws, or 16 | awaits (and same for every await continuation). 17 | 18 | Because of the GIL, only one thread can execute Python code. But threads can be preempted between bytecode instructions. 19 | Even though only one thread can execute bytecode at a time, another thread can take over between bytecode instructions, leading to race conditions. 20 | This is why Python still needs thread synchronization primitives (locks, semaphores, etc.) at the application level. 21 | 22 | Then what is the purpose of the GIL? 23 | - GIL protects: 24 | - Python's internal state, reference-counting, memory allocations 25 | - Non-thread-safe C API operations 26 | - GIL does *not* protect: 27 | - Python-level operations that require multiple bytecode instructions. 28 | E.g. `n+=1` is three instructions (Load, Add, Store). 29 | - User-level data structures 30 | 31 | "You can write multi-threaded code in JavaScript using `worker_threads`?" 32 | - Worker threads and webworkers don't change the behavior of JavaScript: 33 | - Each thread is an isolated JavaScript application with distinct memory. 34 | - Thread A will never modify a value in thread B, the `n=n+1` example is still safe in JavaScript. 35 | - Exception: threads can refer to the same memory via `SharedArrayBuffer`. 36 | 37 | LIBRARIES 38 | ================================================================================ 39 | date/time: https://github.com/dateutil/dateutil 40 | 41 | DEBUGGING 42 | ================================================================================ 43 | https://wiki.python.org/moin/DebuggingWithGdb 44 | # Python debugging extensions: debugging symbols and gdb commands (py-xx). 45 | $ apt install python3.5-dbg 46 | $ gdb $(which python3) 19284 47 | (gdb) py-bt 48 | (gdb) py-list 49 | 50 | Fancy TUI debugger: https://github.com/pdbpp/pdbpp 51 | 52 | ? https://github.com/cool-RR/PySnooper 53 | 54 | 55 | MEMORY LEAKS 56 | ================================================================================ 57 | objgraph: https://www.darkcoding.net/software/finding-memory-leaks-in-python-with-objgraph/ 58 | tracemalloc (Python 3.6+): https://docs.python.org/3/library/tracemalloc.html 59 | 60 | 61 | string format(), f-string 62 | ================================================================================ 63 | f'{datetime.datetime.now():%m-%d-%y}' # instead of datetime.stftime() 64 | f'{878:b}' => '1101101110' # base-2 65 | f'{878:x}' => '36e' # base-16 66 | f'{878:e}' => '8.780000e+02' # sci notation 67 | f'{1/3:?=8.2f}' => '????0.33' # pad, fill, precision 68 | f'{"foo":^10}' => ' foo ' # center text 69 | 70 | 71 | asyncio 72 | ================================================================================ 73 | * `await` on a coroutine (more generally an "awaitable") yields to the event-loop. 74 | * Example: `await asyncio.sleep(2)` does not block the loop, it yields and 75 | schedules a Future on the loop for the specified time. When the Future 76 | is executed, control is restored and the caller is unblocked. 77 | source: https://github.com/python/cpython/blob/cca4eec3c0a67cbfeaf09182ea6c097a94891ff6/Lib/asyncio/tasks.py#L580 78 | * Event loop does not preempt, only a coroutine can pause itself. 79 | * Calling a coroutine initializes (not executes) it. 80 | * asyncio.sleep is a coroutine. 81 | 82 | https://news.ycombinator.com/item?id=17714304 83 | - asyncio is mostly very low level. you probably don't want to use it directly. E.G: for http requests, use aiohttp. 84 | - Use asyncio.run_until_complete(), not asyncio.run_forever(). The former will crash on any exception, making debugging easy. 85 | - Activate the various debug features when not in prod (https://docs.python.org/3/library/asyncio-dev.html#debug-mode-of-asyncio). 86 | - CPU intensive code blocks the event loop. loop.run_in_executor() use threads by default, hence it doesn't protect you from that. If you have CPU intensive code, like zipping a lot of files or calculating your own precious fibonacci, create a "ProcessPoolExecutor" and use run_in_executor() with it. 87 | - Don't use asyncio before Python 3.5.3. There is a incredibly major bug with "asyncio.get_event_loop()" that makes it unusable for anything that involve mixing threads and loops. 88 | CONCEPTS: 89 | * Future: thing to execute. 90 | * Task: subclass of future. The thing to execute is a coroutine, and the coroutine is immediately scheduled in the event loop when the task is instantiated. ensure_future(coroutine) returns a Task. 91 | * coroutine: generator with some syntaxic sugar. 92 | * coroutine function: function declared with "async def". Returns a coroutine. 93 | * awaitable: any object with an __await__ method. coroutines, tasks and futures are awaitables. 94 | * event loop: the magic "while True" loop that takes awaitables, and execute them. 95 | * executor: an object that takes code, execute it in a __different__ 96 | context, and return a future you can await in your __current__ 97 | context. You will use them to run stuff in threads or separate 98 | processes, but magically await the result in your current code like 99 | it's regular asyncio. It's very handy to naturally integrate blocking 100 | code in your workflow. 101 | 102 | asyncio.gather() is the most important function in asyncio 103 | ----------------------------------------------------------- 104 | Never, ever, have an dangling awaitable. 105 | Always keep a reference on all your awaitables. 106 | Decide where in the code you think their life should end. 107 | 108 | asyncio.gather() will block until all awaitables are done. 109 | Don't: 110 | asyncio.ensure_future(bar()) 111 | asyncio.get_event_loop().run_in_executor(None, barz) 112 | await asyncio.sleep(10) 113 | Do: 114 | foo = asyncio.ensure_future(bar()) 115 | fooz = asyncio.get_event_loop().run_in_executor(None, barz) 116 | await asyncio.sleep(10) 117 | await asyncio.gather(foo, fooz) # this is The One True Way 118 | 119 | FEATURES 120 | ================================================================================ 121 | python "data model" (underscore methods, etc.): https://docs.python.org/3/reference/datamodel.html 122 | 123 | iter() # https://docs.python.org/3/library/functions.html#iter 124 | partial() # useful to create zero-argument lambdas 125 | from functools import partial 126 | with open('mydata.db', 'rb') as f: 127 | for block in iter(partial(f.read, 64), b''): 128 | process_block(block) 129 | next() # goes to next `yield` 130 | 131 | "generator comprehension" 132 | # () instead of [] 133 | >>> a = (x for x in [1,2,3,4,5]) 134 | >>> type(a) 135 | 136 | 137 | inspect.getsource() 138 | __code__ # reflect on user-defined functions 139 | def foo(): 140 | a=42 141 | b='foo' 142 | foo.__code__.co_varnames # ('a', 'b') 143 | 144 | metaclass: # manipulate class at define-time. cf. Java "classloader" 145 | class Foo(object, metaclass=something): 146 | 147 | contextmanager: 148 | # Enforce ordered __init__, __enter__, __exit__ sequence. 149 | @contextlib.contextmanager 150 | def foo(*args, **kwds): 151 | r = get_resource(*args, **kwds) # Acquire resource. 152 | try: 153 | yield r # Context managers are generators! 154 | finally: 155 | release_resource(r) # Release resource. 156 | with foo(timeout=3600) as r: 157 | ... 158 | 159 | loop backwards: 160 | for o in reversed(foo): 161 | 162 | custom sort: 163 | sorted(foo, key=len) # "almost never need cmp=…" 164 | 165 | initialize dicts with iterators 166 | d = dict(zip(foo, bar)) 167 | d = dict(enumerate(foo)) 168 | 169 | extend dicts 170 | d = ChainMap(d1, d2, ...) 171 | 172 | LRU cache (memoize): compare running time with/without @lru_cache decorator: 173 | import functools 174 | @functools.lru_cache(maxsize=512) 175 | def fib(n: int) -> int: 176 | if n < 2: 177 | return n 178 | return fib(n-1) + fib(n-2) 179 | >>> [fib(n) for n in range(16)] 180 | [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610] 181 | >>> fib.cache_info() 182 | CacheInfo(hits=28, misses=16, maxsize=512, currsize=16) 183 | 184 | (3.7+) Data Classes https://docs.python.org/3/library/dataclasses.html 185 | import dataclasses 186 | @dataclasses.dataclass 187 | class Foo: # Don't need __init__() 188 | foo: float 189 | bar: int = 1 190 | name: str 191 | f = Foo(foo=0.3, bar=42, name='zub') 192 | -------------------------------------------------------------------------------- /c.md: -------------------------------------------------------------------------------- 1 | .i files = preprocessor output (e.g. build/src/nvim/auto/ex_docmd.i) 2 | 3 | pointer arithmetic: 4 | `ptr++` advances the pointer by the size of the pointed-to type. 5 | Incrementing (void *) is illegal. http://stackoverflow.com/a/3524270/152142 6 | 7 | *_r function variants (rand_r, localtime_r, ...) are thread-reentrant ("restartable")[1] 8 | However, they are not "signal handler reentrant"[2] 9 | [1] http://www.unix.org/whitepapers/reentrant.html 10 | [2] http://stackoverflow.com/a/18199867/152142 11 | 12 | popen() essentially does a combination of (pipe, dup stdin/stdout/stderr, fork, execl) 13 | http://stackoverflow.com/a/24296225/152142 14 | 15 | bit array: https://lobste.rs/s/ueuxgt/implementing_bit_arrays_c 16 | #define SetBit(A,k) ( A[(k/32)] |= (1 << (k%32)) ) 17 | #define ClearBit(A,k) ( A[(k/32)] &= ~(1 << (k%32)) ) 18 | #define TestBit(A,k) ( A[(k/32)] & (1 << (k%32)) ) 19 | 20 | CROSS-PLATFORM/OS 21 | ================================================================================ 22 | cross-platform syscall reference: 23 | https://github.com/giampaolo/psutil 24 | https://github.com/facebook/osquery 25 | linux-only: 26 | "procps" https://gitlab.com/procps-ng/procps 27 | 28 | REFERENCE 29 | ================================================================================ 30 | C: A Reference Manual, Fifth Edition. http://www.careferencemanual.com 31 | "Secure C Coding" SEI CERT C Coding Standard https://wiki.sei.cmu.edu/confluence/pages/viewpage.action?pageId=87152038 32 | 33 | - undefined behavior 34 | - https://blog.llvm.org/2011/05/what-every-c-programmer-should-know_14.html 35 | - https://blog.llvm.org/2011/05/what-every-c-programmer-should-know_21.html 36 | 37 | DEBUG 38 | ================================================================================ 39 | gdb 40 | CTRL-X_CTRL-A => TUI mode (zomg!!) 41 | 42 | rr: gdb replacment with efficient reverse execution http://rr-project.org 43 | 44 | Compiler Explorer: https://godbolt.org 45 | compare optimization/assembly per-compiler/per-platform 46 | 47 | LINKING 48 | ================================================================================ 49 | todo: 50 | "ELF Hello World Tutorial" http://www.cirosantilli.com/elf-hello-world/ 51 | 52 | Main functions of linker: 53 | RESOLVE undefined symbols. 54 | DEDUPLICATE. A symbol may be defined only once. 55 | ADDRESS RELOCATION 56 | Relocation edits the .text section of object files to translate object 57 | file address into the final address of the executable. This must be done 58 | by the linker because the compiler only sees one input file at a time. 59 | 60 | Compare: 61 | objdump -d -r foo.o 62 | to: 63 | objdump -d -r foo.out 64 | 65 | http://jvns.ca/blog/2013/12/10/day-40-learning-about-linkers/ 66 | https://github.com/0xAX/linux-insides/blob/master/Misc/linkers.md 67 | nm -A foo.o 68 | objdump -S -r foo.o 69 | 70 | "Delete an inline function, save 794 kB" 71 | https://randomascii.wordpress.com/2017/01/22/delete-an-inline-function-save-794-kb/ 72 | 73 | > how symbol resolution works in linkers: Linkers start out with the set of 74 | > object files that you have specified on the command line – the contents of 75 | > these object files will be included in the final binary (unless the linker 76 | > can prove that removing them makes no difference). Then the linker builds up 77 | > a list of unresolved symbols and starts scanning through the set of 78 | > libraries that were specified on the command line – the object files in 79 | > these libraries will be included as-needed. Whenever a needed symbol is 80 | > found in one of the library object files then that object file is added to 81 | > the list to be linked – it becomes like one of the command-line object 82 | > files. The process continues until all symbols are resolved. 83 | 84 | > The symbol search order is unspecified by C++ and due to the One Definition 85 | > Rule (ODR) the search order should not matter. But, an obvious 86 | > implementation is to repeatedly scan through the libraries, looking for 87 | > symbols, until there are no more unresolved. This means that if you have two 88 | > copies of the same symbol then it is undefined which you will get – it 89 | > depends where you are when you first find that you need it. Since pulling in 90 | > a symbol pulls in the rest of its object file this means that linking is 91 | > fundamentally unpredictable. And, if you have ODR violations (as we did) 92 | > then the results are not even defined. 93 | 94 | https://flameeyes.blog/2010/10/08/linkers-and-names/ 95 | a shared library “owns” several different names, on Unix-like systems based on ELF: 96 | - the actual filename; 97 | - the name used by the link editor (or build-time linker, ld) to link to it; 98 | - the name used by the loader (or dynamic linker, ld.so) to find it. 99 | > The obvious way to correlate all of them together is to use symbolic links; and that’s why your /usr/lib directory is full of those. 100 | 101 | C STANDARD LIBRARY 102 | ================================================================================ 103 | 104 | https://begriffs.com/posts/2019-01-19-inside-c-standard-lib.html 105 | 106 | - NULL is not a reserved word: compiler treats 0 specially in a pointer 107 | context and transforms it to whatever value represents NULL on the given 108 | architecture. Thus NULL is typically a macro for ((void *)0). As such it 109 | can’t be assigned to a function pointer because data pointers needn’t be the 110 | same size as function pointers. Instead cast zero, e.g. (int (*)(void))0. 111 | - Similarly, the C standard treats `main() { return 0; }` and `exit(0)` 112 | specially: they are mapped to the system-specific success code! 113 | - Use EXIT_FAILURE macro to portably indicate failure. The raw value 1 is 114 | considered "success" on some platforms. 115 | 116 | - Comparing ARIBTRARY pointers is undefined behavior. BUT you can compare 117 | pointers within the SAME _primary data object_ (such as different cells in 118 | the same array). 119 | - Pointer arithmetic on (void*) is a GNU-ism not permitted in portable C. 120 | - Some memmove() implementations assume its arguments are in the same data 121 | object! So one should not indiscriminately/defensively use memmove() 122 | instead of memcpy(). 123 | 124 | - If size_t is the largest integer type, then ptrdiff_t can be no larger, yet 125 | the latter loses one bit to hold the sign. So it’s possible to make an array 126 | with cells too far apart for ptrdiff_t to measure. 127 | 128 | - offsetof 129 | 130 | - stdio.h 131 | BUFFERING 132 | The standard library provides setvbuf() to change the size and location of 133 | a stream’s buffer, and to choose line or block buffering. By default, stdin 134 | and stdout are line and block buffered respectively, and stderr is unbuffered. 135 | setvbuf() must be called immediately after a stream is opened, before I/O 136 | happens, to have any chance of working. 137 | 138 | OPENING/CLOSING FILES 139 | TOCTOU happens with fopen() when trying to create but not replace a file: 140 | C99 fixes this with “x” (exclusive) mode: fopen will fail if the file 141 | already exists or cannot be created. 142 | 143 | FILE *fp = fopen("foo.txt","r"); 144 | // <-- attacker gets busy here 145 | if (!fp) { 146 | fp = fopen("foo.txt","w"); // C99: use "wx" to avoid TOCTOU risk. 147 | ... 148 | fclose(fp); 149 | } 150 | else { fclose(fp); } 151 | 152 | Treat FILE pointers as totally opaque. Don’t even try to make 153 | a copy of the FILE structure because some implementations rely on magic 154 | memory addresses. 155 | 156 | fflush() MIGHT force items in the buffer to be processed, but there is no 157 | guarantee. fflush(NULL) flushes all streams. 158 | 159 | Some OSes will not actually create a file that you fopen() and fclose() 160 | unless you write something. 161 | 162 | - unsigned char: has special guarantees that make it ideal for representing 163 | arbitrary binary data. 164 | - Unsigned char is guaranteed to have no padding bits. All bits contribute 165 | to the value of the data. Other types on some architectures include 166 | things like parity bits which don’t affect the value but do use space. 167 | - No bitwise operation starting from an unsigned char value, when 168 | converted back into that type, can produce overflow, trap 169 | representations or undefined behavior. It can be freely manipulated. 170 | "Trap representations" are certain bit patterns reserved for exceptional 171 | circumstances, such as NaN in floating point numbers. 172 | - Accessing parts of a larger data object with an unsigned char pointer 173 | will not violate any “aliasing rules.” The unsigned char pointer will be 174 | guaranteed to see all modifications of the data object. 175 | - "plain" (not unsigned) char has some distinct use-cases: 176 | - codeset characters 177 | - small numbers 178 | - units of storage (unsigned recommended) 179 | - small bit patterns (unsigned recommended) 180 | 181 | 182 | TECHNIQUES 183 | ================================================================================ 184 | - "A safer arena allocator" https://gaultier.github.io/blog/tip_of_the_day_2.html 185 | - "Roll your own memory profiling: it's actually not hard" https://gaultier.github.io/blog/roll_your_own_memory_profiling.html 186 | -------------------------------------------------------------------------------- /algorithms.txt: -------------------------------------------------------------------------------- 1 | todo: 2 | + graphs, visualization: https://github.com/coells/100days 3 | ~ http://jeffe.cs.illinois.edu/teaching/algorithms/ 4 | huffman codes (zip/compression) 5 | 6 | TCO (tail recursion, tail-call optimization) 7 | https://eli.thegreenplace.net/2017/on-recursion-continuations-and-trampolines/ 8 | Reuse the stack frame! Avoids allocating a new stack frame because the 9 | caller only returns the value from the callee (no caller state!). 10 | tail-recursive: 11 | def fact_tailrec(n, result=1): 12 | ... 13 | return fact_tailrec(n - 1, result * n) 14 | NOT tail-recursive: 15 | def fact_rec(n): 16 | ... 17 | return n * fact_rec(n - 1) 18 | 19 | trampoline: a loop that iteratively invokes thunk-returning functions (continuation-passing style) 20 | def trampoline(f, *args): 21 | v = f(*args) 22 | while callable(v): 23 | v = v() 24 | return v 25 | # CPS version of factorial, transformed to return a thunk. 26 | # The transformation is straightforward: wrap the tail calls in an 27 | # argument-less lambda. 28 | def fact_cps_thunked(n, cont): 29 | if n == 0: 30 | return cont(1) 31 | else: 32 | return lambda: fact_cps_thunked( 33 | n - 1, 34 | lambda value: lambda: cont(n * value)) 35 | # Then to compute the factorial of 6: 36 | >>> trampoline(fact_cps_thunked, 6, end_cont) 37 | 720 38 | # The stack doesn't grow! Instead of calling itself, fact_cps_thunked 39 | # returns a thunk, so the call is done by the trampoline. 40 | 41 | # Tracing the function calls for the recursive factorial we get: 42 | fact_rec(6) 43 | fact_rec(5) 44 | fact_rec(4) 45 | fact_rec(3) 46 | fact_rec(2) 47 | fact_rec(1) 48 | fact_rec(0) 49 | 50 | # Tracing the function calls for the thunked version we get: 51 | trampoline(, 6, ) 52 | fact_cps_thunked(6, ) 53 | fact_cps_thunked(5, ) 54 | fact_cps_thunked(4, ) 55 | fact_cps_thunked(3, ) 56 | fact_cps_thunked(2, ) 57 | fact_cps_thunked(1, ) 58 | fact_cps_thunked(0, ) 59 | 60 | Zipf's law 61 | Nonuniform access is usually the rule, not the exception. Many real-world 62 | distributions are governed by power laws. A classic example is word use in 63 | English, which is modeled by Zipf's law: 64 | The ith most frequently accessed key is selected with probability 65 | (i − 1)/i times the probability of the (i − 1)th most-popular key. 66 | 67 | knapsack problem: resource allocation with budget constraint 68 | Given a set of integers S = {s1, s2, ..., sn}, and a target T, find a subset 69 | that adds up exactly to T. 70 | 71 | "0/1" variant is most common (otherwise if subdividing objects is allowed, 72 | greedy algorithm gives optimal solution!) 73 | 74 | Same "price per pound"? greedy won't work. goal is to minimize empty space. 75 | "easy cases": 76 | Same cost per item, different size? => optimize for greatest number of items 77 | Same size per item, different cost? => sort, then choose least-cost 78 | 79 | heuristics (non-optimal): 80 | greedy: Inserts items according to the maximum “price per pound”. 81 | first-fit: Put the elements of S in the knapsack in left to 82 | right order if they fit. 83 | best-fit: Put the elements of S in the knapsack from smallest to 84 | largest. 85 | 86 | graph traversal, (shortest) pathfinding 87 | A* 88 | - is a special case of branch-and-bound. 89 | - vs other greedy best-first search algorithms: A* takes the cost/distance already traveled g(n) into account. 90 | - can implement generalized depth-first search by initializing a global counter C to a very large value... 91 | - Dijkstra's algorithm is a special case of A* where heuristic h(n)=0 for all nodes x. 92 | - both Dijkstra and A* are special cases of dynamic programming. 93 | 94 | bin-packing: ("multiple knapsacks"...) 95 | NP-hard 96 | heuristic: first-fit Θ(nlogn) (fast but often non-optimal): place each item into the first bin in which it will fit. 97 | - For each item, place in the first bin with enough space. If no bin is found, open a new bin. 98 | - The number of bins used by this algorithm is no more than twice the optimal number of bins. 99 | 100 | sorting 101 | Lower bound of sorting in general is O(n*logn). 102 | This sets the lower bound for many _applications_ of sorting, including 103 | "element uniqueness", "finding the mode", and "constructing convex hulls". 104 | 105 | quicksort: select a random "pivot" item `p` 106 | _not_ stable 107 | partition() the array in one linear scan into THREE SECTIONS: 108 | 1. "less than the pivot" (to the left of firsthigh) 109 | 2. "greater than or equal to the pivot" (between firsthigh and i) 110 | 3. "unexplored" (to the right of i) <= KEY IDEA: this is whatever is 111 | left over as the linear scan & 112 | swap progresses 113 | partition() gains us: 114 | * The pivot element `p` lands in its final position. 115 | * After partitioning, an element never switches sides. 116 | * Recursion is limited to ~lgn depth. 117 | We can now sort the elements on either side of the pivot independently! 118 | => naturally recursive 119 | => parallelizable 120 | 121 | mergesort: 122 | stable 123 | 124 | timsort: https://lobste.rs/s/crfy2d 125 | complex, but justified when: 126 | - Comparisons are (very) expensive 127 | (E.g. dynamic dispatch => used by Python). 128 | This is because TimSort itself performs a bunch of branches and 129 | integer comparisons to keep track of galloping scores and the stack 130 | invariant. 131 | - You need a stable sort. 132 | cf. Quicksort: 133 | + fast 134 | + easy to implement 135 | + in-place 136 | - unstable 137 | - bad worst-case perf (pre-sorted data in the naïve implementation) 138 | + mitigation=introsort: start with quicksort, but bail out to 139 | a guaranteed O(n log n) sort like heapsort if it takes too long. 140 | In the bail-out case, you lose constant-factor performance 141 | compared to if you had used heapsort in the first place, but you 142 | avoid quicksort’s O(n^2) worst case, while still getting its good 143 | performance in non-pathological cases. It’s used in practice in .NET 144 | and some C++ STL implementations. 145 | 146 | LCS longest common subsequence 147 | Find the longest SUBSEQUENCE common to all sequences in a set of sequences. 148 | cf. longest common SUBSTRING: unlike substrings, subsequences are not required to be contiguous in the original sequences. 149 | complexity: 150 | - NP-hard for the general case 151 | - polynomial-time with fixed inputs + memoization 152 | 153 | consistent hash "ring": 154 | why: avoid full rehash (EXPENSIVE for e.g. load-balanced cluster) when nodes/buckets are added/removed 155 | 156 | https://akshatm.svbtle.com/consistent-hash-rings-theory-and-implementation 157 | consistent hashing, in a nutshell, does this: 158 | - STOP trying to keep one value at exactly one location. Let one location house multiple values from multiple keys. 159 | - DON’T number your locations consecutively. Assign effectively random numbers (0~INT_MAX). 160 | - DON’T compute (hash % buckets). Instead, use the nearest bucket ≥ hash(key). (Wrap-around if greater than all buckets.) 161 | 162 | how: "hash both objects AND caches" 163 | use-case: distributed KV store, distributed cache, load balancing 164 | examples: 165 | cassandra 166 | Amazon Dynamo https://www.allthingsdistributed.com/2007/10/amazons_dynamo.html 167 | BitTorrent distributed tracker 168 | CLIENT only! No cooperation needed from the servers/targets. 169 | Only ~k/N keys need to be remapped where k is the number of keys and N is the number of servers. 170 | 171 | R+W > N => "strongly consistent" (e.g. write 2, read 2 with 3 replicas) 172 | 173 | alternative: Rendezvous hashing, aka highest random weight (HRW) hashing 174 | distributed k-agreement (consistent hashing is k=1) 175 | 176 | Two's complement https://news.ycombinator.com/item?id=36457020 177 | - (Hint: compare the apostrophe: "ones' complement" vs "two's complement".) 178 | - Two's complement: To negate a number, raise 2 to the power of the bit 179 | width and then subtract the number. For example, negating 0011b (4 bits) 180 | means taking 10000b and subtracting 0011b, giving you 1101b. 181 | - Addition and subtraction for signed and unsigned int works the same 182 | way at the _bit level_. For signed ints, the highest bit is the sign. 183 | - Consequences to remember if you work with signed and unsigned 8-bit ints: 184 | - The biggest unsigned int is 255 (2⁸-1; all 1s in binary) and 185 | smallest is 0 (all 0s in binary). Increment/decrement jump between 186 | those in overflow/underflow (the modulo arithmetic). 187 | - Because the highest bit is needed for the negative sign, the 188 | biggest signed int is 127 (2⁷-1; all 1s except the sign bit) and 189 | smallest is -128 (-2⁷; only the sign bit set). Increment and 190 | decrement jump between those in overflow/underflow. 191 | - Ones' complement: To negate a number, take as many ones (plural) as the 192 | bit width and then subtract the number. For example, negating 0011b (4 193 | bits) means taking 1111b and subtracting 0011b, giving you 1100b. 194 | -------------------------------------------------------------------------------- /git.txt: -------------------------------------------------------------------------------- 1 | git svn clone "svn://foosrc/subdir" . -T trunk -b branches -t tags -r234:HEAD --ignore-path="foo(bar|baz).exe|^Chtulu/|.+\.(dll|jar)$" 2 | 3 | git svn init -s --ignore-path="foo" svn://foosrc/subdir 4 | git svn fetch -r123:HEAD 5 | 6 | GENERAL 7 | add "--stat" option to any query command (stash show, log, diff) to get 8 | file-by-file insertion/deletion counts. 9 | 10 | #view every single commit in the repository ever 11 | gitk --all $( git fsck --no-reflog | awk '/dangling commit/ {print $3}' ) 12 | 13 | # find last commit matching a regex 14 | # example: show the last merge commit 15 | $ git show :/^Merge 16 | 17 | # git pickaxe -G option is greedier than -S and always accepts regex! 18 | $ git log -G 19 | 20 | # git grep can search any branch! 21 | $ git grep foo 22 | 23 | # `git show` is analogous to `cat`. https://www.andyjeffries.co.uk/25-tips-for-intermediate-git-users/ 24 | # (and `git cat-file` is for opening objects) 25 | git show master:flash/foo.fla > master-foo.fla 26 | 27 | RANGES (see: git help rev-parse > "Specifying ranges") 28 | git log origin/master.. 29 | is equivalent to: 30 | git log origin/master..HEAD 31 | 32 | # "triple dot" diff shows the changes between alice/master and its merge 33 | # base (COMMON ANCESTOR) with master. It won't include any changes that 34 | # happened on master since the COMMON ANCESTOR (use "git merge-tree"). 35 | git diff master...alice/master 36 | 37 | git merge-tree `git merge-base bob/master` master bob/master 38 | 39 | SELECT/FILTER COMMITS 40 | # Show commits on `my-feature` and NOT on `master` (complement): 41 | git log my-feature ^master 42 | 43 | CONFIGURATION: ALIASES 44 | ! runs at the root of the repo. To make ! commands relative to the current 45 | directory, cd to $GIT_PREFIX: 46 | git alias foo='!cd $GIT_PREFIX; foo -bar' 47 | #Use GIT_TRACE=1 to make the alias processing transparent: 48 | $ GIT_TRACE=1 git foo 49 | 50 | CLONING 51 | https://github.blog/2020-12-21-get-up-to-speed-with-partial-clone-and-shallow-clone/ 52 | Shallow clone 53 | - truncates the commit history 54 | git clone --depth= 55 | - best combined --single-branch --branch= options, to ensure we only download the data for the commit we plan to use immediately. 56 | - unlike *partial* clone, commands such as `git merge-base` or `git log` show different results. 57 | Partial clone 58 | - doc: https://git-scm.com/docs/git-clone#Documentation/git-clone.txt---filterltfilter-specgt 59 | - enabled by specifying the --filter option in your git clone command. 60 | - Blobless clones: git clone --filter=blob:none 61 | - will trigger a blob download whenever you need the contents of a file, but not if you only need the OID of a file. 62 | - can perform commands like `git merge-base`, `git log`, or even `git log -- ` with the same performance as a full clone. 63 | - Treeless clones: git clone --filter=tree:0 64 | - downloads all reachable commits, then downloads trees and blobs on-demand. 65 | - initial clone _much_ faster in a treeless clone than in a blobless or full clone 66 | - use-case: for automated builds when you want to quickly clone, compile, then throw away the repository. 67 | - current? limitations: 68 | - a file history request such as git log -- , will download root trees for almost every commit in the history! 69 | - submodules behave very poorly with treeless clones. 70 | 71 | BRANCHING 72 | git checkout -b new_branch [branch to track] 73 | 74 | #list all branches (local and remote) 75 | git branch -a 76 | 77 | #find out which branch contains a commit 78 | git branch -a --contains 50f3754 79 | 80 | #move (rename) a branch 81 | git branch -m smurf master 82 | 83 | MERGING 84 | REBASE AND MERGE DON'T MIX! 85 | http://stackoverflow.com/a/15153099/152142 86 | 87 | git creates a "merge commit" because merges are intended to be _meaningful_. 88 | 89 | REBASING 90 | #rebase the current branch onto the remote head 91 | git svn rebase 92 | 93 | #rebase the current branch onto master 94 | git rebase master 95 | 96 | #rebase onto the latest _remote_ commit on the current branch 97 | #(this is sugar for `git rebase @{u}` AKA `git rebase @{upstream}`) 98 | git rebase 99 | 100 | INTERACTIVE REBASE (SELECTIVELY REMOVE OR EDIT COMMITS) 101 | http://christoph.ruegg.name/blog/2010/5/5/git-howto-revert-a-commit-already-pushed-to-a-remote-reposit.html 102 | #to remove a commit but keep all others, do an interactive rebase onto 103 | #the parent of the offending commit: 104 | git rebase -i dd61ab32^ 105 | 106 | CLEANING UNTRACKED FILES (DELETED FROM INDEX AFTER A REBASE) 107 | #dry run 108 | git clean -n 109 | 110 | #remove untracked files (not directories) 111 | # f = force (not file) 112 | git clean -f 113 | 114 | CREATING/APPLYING PATCHES 115 | #create patch of current branch against master 116 | git format-patch master --stdout > foo.patch 117 | 118 | #create a "diff" patch from current uncommitted changes 119 | git diff > result.patch 120 | 121 | #summarize patch changes 122 | git apply --stat foo.patch 123 | 124 | #apply patch (dry run) 125 | git apply --check foo.patch 126 | 127 | #apply "format-patch" patch (with signoff); .rej files for any conflicts 128 | git am --reject --ignore-whitespace --signoff < foo.patch 129 | 130 | 131 | TEMPORARILY IGNORING FILES (cf. svn's "ignore-on-commit") 132 | #shelve current changes 133 | git stash save "foo" 134 | 135 | #apply ALL shelved changesets 136 | git stash apply 137 | 138 | #list shelved changesets 139 | git stash list 140 | 141 | #apply a specific changeset from the list... 142 | git stash apply stash@{1} 143 | 144 | ADD A SVN REMOTE MANUALLY: http://stackoverflow.com/q/296975/152142 145 | note: {new_branch} must not begin with a number 146 | git config --add svn-remote.{new_branch}.url https://svn/path/to/new_branch/ 147 | git config --add svn-remote.{new_branch}.fetch :refs/remotes/{new_branch} 148 | git svn fetch {new_branch} [-r] 149 | git checkout -b {new_local_branch} -t {new_branch} 150 | git svn rebase {new_local_branch} 151 | 152 | REMOTES 153 | # print fetch/push URLs 154 | git remote -v 155 | 156 | # show details about (incl. local tracking branches) 157 | git remote show 158 | 159 | git remote add git:/// 160 | git remote add smurf ssh://user@:9999/~/.git 161 | 162 | PER-REPO SSH KEYS 163 | ~/.ssh/config: 164 | Host host1 165 | HostName github.com 166 | User git 167 | IdentityFile ~/.ssh/host1_key 168 | .git/config: 169 | [user] 170 | email = user1@example.com # github email 171 | [remote "origin"] 172 | url = git@host1:justinmk/repo1.git 173 | $ ssh-add -D # delete cached keys! 174 | $ git push ... 175 | 176 | VIRTUALBOX CONFIGURATION 177 | * in guest host: 178 | $ sudo ip link set dev ethX down 179 | $ sudo dhclient ethX 180 | * set Network to "NAT" 181 | * set port forwarding 182 | * to ssh to guest, target the HOST ip + fowarded port and VB will route it 183 | * on your dev box, add a remote pointing to HOST_IP:VIRTUALBOX_FORWARDED_PORT 184 | $ git remote add vbox ssh://user@:9999/~/.git 185 | 186 | REBASING ONTO NON-SVN REMOTES 187 | git pull marco marcos_local_branch 188 | 189 | PUSH LOCAL BRANCH TO REMOTE BRANCH 190 | git push remote_alias local_branch:remote_branch 191 | 192 | FETCH BY URL 193 | 194 | git fetch https://github.com/foo/bar refs/heads/branch1:branch1 195 | 196 | FETCH GITHUB PULL REQUESTS AS LOCAL REFERENCES: https://help.github.com/articles/checking-out-pull-requests-locally 197 | 198 | [remote "origin"] 199 | fetch = +refs/pull/*/head:refs/pull/origin/* 200 | ^^ ^ 201 | || local ref pattern 202 | |remote ref pattern 203 | update references even if they aren't fast-forwards 204 | 205 | MERGING, REBASING, SVN 206 | #preserve author in dcommit 207 | 208 | http://keithp.com/blogs/Repository_Formats_Matter/ 209 | subversion repository format is poorer than the git one. It doesn't support merging well, or committers vs authors or commit times distinct from push times 210 | 211 | options that are good for small batches, but fall over on large batches: 212 | [rerere] 213 | enabled = 1 214 | [merge] 215 | # disable renameLimit http://stackoverflow.com/a/13118734/152142 216 | renameLimit = 0 217 | 218 | GIT INTERNALS: http://git-scm.com/book/en/Git-Internals-Plumbing-and-Porcelain 219 | Git is a content-addressable filesystem with a VCS UI written on top of it. 220 | Q: what does that mean? 221 | A: check out Joe Armstrong's talk. "content-addressing" differs from 222 | "reference address" because it is immutable 223 | It's important for reliability / attempt to solve entropy. 224 | 225 | core parts of Git: 226 | .git/ 227 | HEAD points to the current branch 228 | index stores staging area information 229 | objects/ stores all database content 230 | refs/ stores pointers (branches) to commit objects in objects/ 231 | 232 | HOW TO NOT LOSE DATA: 233 | `git add` adds the file immediately to `.git`, and will be available 234 | even after `git reset`, etc., even if you never committed it. 235 | 236 | PACKFILES: 237 | `git extract-objects` unpacks a packfile into a zlib-compressed object 238 | file (.git/objects/xx/*). 239 | To see the contents of an object: 240 | cat .git/objects/c0/fb67ab3fda7909000da003f4b2ce50a53f43e7 | zlib-flate -uncompress; 241 | 242 | LAST SLIDE: 243 | "Git has become a protocol/platform in addition to a VCS." 244 | https://git-annex.branchable.com/ 245 | 246 | -------------------------------------------------------------------------------- /data_structures.txt: -------------------------------------------------------------------------------- 1 | ♡ DICTIONARY OF ALGORITHMS & DATA STRUCTURES: https://xlinux.nist.gov/dads/ 2 | 3 | --- 4 | 5 | A data structure is "naturally recursive" if it can be cut in half and both 6 | parts treated as the same type of structure. (trees, arrays, stacks, ...). 7 | 8 | Stacks and Queues have equivalent _average_ waiting time. Queue is important 9 | when _order_ of processing (or maximum waiting time) is important. 10 | 11 | vector vs linked-list: 12 | linked-list is ~never the right choice (unless your workload is dominated by splitting/merging costs). 13 | linked-list traversal = scattered random memory-access (no spatial locality) => defeats caches => 10x slower 14 | 15 | lock-free concurrency 16 | C: http://concurrencykit.org/ 17 | mscorlib: ConcurrentQueue.cs 18 | - all public/protected methods are thread-safe 19 | - use CAS; "SpinWait" for backoff 20 | - GetEnumerator(), ToArray(), ToList() need to take "snapshot". 21 | - ConcurrentQueue is a linked list of small arrays, each node is called a 22 | segment. A segment contains an array, a pointer to the next segment, 23 | and m_low, m_high indices recording the first and last valid elements 24 | of the array. 25 | other notes: 26 | copy shared state to local vars before doing work, then use CAS 27 | C#: struct assignement/copying is not atomic 28 | C#: lock() is syntax-sugar for try-finally Monitor.Enter/Monitor.Exit. 29 | event queue is an alternative to polling 30 | 31 | ring buffer / circular buffer / bounded queue / FIFO 32 | "blocking" variety: 33 | consumer waits if queue is empty 34 | producer waits if queue is full 35 | 36 | struct Fifo { 37 | ItemType items* 38 | size_t writepos; 39 | size_t readpos; 40 | size_t capacity; 41 | } 42 | 43 | sparse array 44 | 45 | tree 46 | The cost of transferring data between levels of the memory hierarchy 47 | (RAM-to-cache or disk-to-RAM) dominates the cost of actual computation for 48 | many problems. "Cache-oblivious" data structures offer performance 49 | guarantees without explicit knowledge of the block-size. 50 | 51 | "threaded" binary tree points unused pointers to higher nodes. (Knuth 322) 52 | => enables traversal without recursion! 53 | 54 | binary 55 | balanced: (maxdepth - mindepth) <= 1 56 | 57 | BST 58 | TODO: treap https://jvns.ca/blog/2017/09/09/data-structure--the-treap-/ 59 | 60 | find(): O(lgn) (_If_ you can guarantee that the number of nodes ~halved each 61 | iteration. This is why BST must be "balanced".) 62 | Other important properties: 63 | can obtain the smallest element by following all the left children 64 | can obtain the largest element by following all the right children 65 | 66 | heap 67 | TODO: https://github.com/wincent/command-t/blob/master/ruby/command-t/heap.c 68 | 69 | BAD for search. "heap is not a BST" 70 | GOOD for priority queue: Find the min (max) element in a set under insertion 71 | and deletion. 72 | Maintains _partial_ order on the set of elements which is weaker than the 73 | sorted order (=> efficient to maintain) yet stronger than random order 74 | (=> fast lookup of min element). 75 | _compact_ => Represents binary trees without using any pointers. Stores data 76 | as an array of keys. 77 | 78 | priority queue (heap) 79 | max_key() or min_key() on a Dictionary enables it to serve as a priority queue. 80 | heap is a maximally efficient implementation of priority queue 81 | 82 | bloom filter 83 | probabilistic data structure: NO or PROBABLY 84 | time- AND space-efficient 85 | applications: 86 | webbrowser: check potential spam URL without hitting the network 87 | cache (Akamai): store in cache only if requested at least once before 88 | database: avoid disk lookup of non-existent rows/columns. 89 | to insert an element: O(1) 90 | 1. Imagine you have k distinct, _deterministic_ hash functions. 91 | 2. For a given element, each hash function returns a different value 92 | (collisions are okay). 93 | 3. Using the output of each hash function as an index into a bit-array, 94 | set array[i]=true for each index i. 95 | - Time-cost is O(1) because each insertion runs a constant number of 96 | hash functions and sets a constant number of array indices 97 | to check for membership: O(1) 98 | 1. Run it through all of the same hash functions again! Because the hash 99 | functions are deterministic, the same input should return the same 100 | output. 101 | 2. For each index (hash result), check if array[i] == true. 102 | - If each case is true, result is PROBABLY (element was probably 103 | inserted in the past). 104 | - If any case is false, result is NO (no false negatives). 105 | 106 | splay tree 107 | uses rotations to move any accessed key to the root. Frequently-used or 108 | recently-accessed nodes thus sit near the top of the tree, allowing faster 109 | searches. 110 | 111 | B-tree ("balanced") 112 | For data larger than main memory (~1M+ items). 113 | self-balancing tree 114 | generalization of a binary search tree: N children per node 115 | concept: collapse several levels of a binary search tree into a single large 116 | node, so we can do several search steps before another disk access 117 | is needed. Can access enormous numbers of keys using only a few 118 | disk accesses. 119 | 120 | TODO: http://patshaughnessy.net/2014/11/11/discovering-the-computer-science-behind-postgres-indexes 121 | 122 | merkle tree 123 | TODO: https://blog.cloudflare.com/introducing-certificate-transparency-and-nimbus/ 124 | sparse merkle tree (SMT) 125 | - mentioned in "The Power of Toys" (David Nolen) 126 | - like a standard Merkle tree, except the contained data is indexed. missing data is stored as null. 127 | - pro: Efficient proofs of non-inclusion. Can check that data is *not* part of the tree (unlike normal MT) 128 | - con: use lots of space. (mitigation: 129 | - represents a key-value store, inside of a Merkle tree. 130 | - Ethereum researchers are looking into sparse Merkle trees as a replacement for the Merkle Patricia tries currently used to store Ethereum state. 131 | 132 | trie (ReTRIEval, "prefix trie", "suffix trie", …) 133 | examples: autocomplete, spellcheck, hierarchical routing (e.g. ip ranges) 134 | 135 | graphs 136 | TODO: BFS can be used to test bipartiteness, by starting the search at any 137 | vertex and giving alternating labels to the vertices visited during 138 | the search. That is, give label 0 to the starting vertex, 1 to all its 139 | neighbors, 0 to those neighbors' neighbors, and so on. If at any step 140 | a vertex has (visited) neighbors with the same label as itself, then 141 | the graph is not bipartite. 142 | 143 | Adjacency list is usually the best data structure to use for graphs. 144 | Most real-world graphs are sparse. 145 | 146 | Adjacency Lists: 147 | space: O(m + n) 148 | insert/delete: O(d) 149 | traverse: Θ(m + n) 150 | compact representation using linked lists. 151 | Harder (vs adj. matrix) to verify whether a given edge (i,j) is in G, since 152 | we must search through the list. However, such queries usually are not 153 | needed, instead BF/DF traversal is used and nodes are updated passively. 154 | 155 | Adjacency Matrix: 156 | space: O(n^2) 157 | insert/delete: O(1) 158 | traverse: Θ(n^2) 159 | We can represent G using an n × n matrix M, where 160 | element M[i,j] = 1 if (i,j) is an edge of G, and 0 if it isn’t. This allows fast 161 | answers to the question “is (i,j) in G?”, and rapid updates for edge insertion 162 | and deletion. It may use excessive space for graphs with many vertices and 163 | relatively few edges, however. 164 | 165 | problem: isomorphism test: determine whether the topological structures of 166 | two graphs are identical if we ignore any labels. 167 | soln: backtracking: assign each vertex in each graph a label, then 168 | compare. 169 | 170 | 171 | MEMORY MODELS http://canonical.org/~kragen/memory-models/ 172 | ========================================================= 173 | COBOL: nested records 174 | - fixed-size fields 175 | LISP: object graph (labeled, directed) 176 | - requires GC 177 | - Serializing an object graph is a bit tricky, both because it can contain circular references, but also because the part you want to serialize may contain references to a part you don’t want to serialize, and you have to special-case both of these. For example, in some systems, a class instance contains a reference to its class, and the class contains references not only to the current versions of all the methods but also to its superclass, but maybe you don’t want to serialize the entire bytecode for the class in every serialized object. 178 | - Most popular programming languages use this model 179 | FORTRAN: parallel arrays 180 | - parallel arrays are cache-friendly, support different visibility for different attributes, support setting watchpoints, provide a sequence that can be meaningful, and support multidimensional indexing (where an attribute is a property of a tuple of entities, rather than just one entity). I would add that they also allow you to write subroutines that abstract over attributes, since they reify each attribute at run-time: you can write a sum function or a covariance function that can be applied to arbitrary attributes. 181 | - Octave, Matlab, APL, J, K, PV-WAVE IDL, Lush, S, S-Plus, and R are all significant parallel-array-oriented programming languages; Numpy, Pandas, and OpenGL are parallel-array-oriented libraries; and as I explained above, Perl4, awk, and Tcl are to some extent parallel-dictionary-oriented. 182 | - Various features of modern hardware increase the pressure to use parallel arrays to get better performance: the increasing gap between CPU speed and memory speed, the SIMT architecture of GPUs, and the SIMD instructions that have been added to CPUs to increase the ratio of ALU silicon to control silicon. 183 | MAGNETIC-TAPE: pipes ("flow-based programming") 184 | - This kind of append-only storage turns out to be entirely adequate for some algorithms; MapReduce is not far from operating in this fashion, but also the typical problem of tokenization with lex, for example, uses just such a minimal interface to its input. 185 | - Python’s iterators and generators, the C++ STL’s forward iterators, D’s forward ranges, and Golang’s channels are all examples of such pipes or channels, with their purely sequential data access. 186 | MULTICS: directories (string-labeled tree with blob leaves) 187 | SQL: relations (collection of mutable multivalued finite functions) 188 | 189 | 190 | -------------------------------------------------------------------------------- /googleio-2013.txt: -------------------------------------------------------------------------------- 1 | Compositing in Blink and WebKit 2 | https://developers.google.com/live/shows/526326012 3 | http://davidshariff.com/blog/what-is-the-execution-context-in-javascript/ 4 | http://gent.ilcore.com/2011/03/how-not-to-trigger-layout-in-webkit.html 5 | http://www.phpied.com/rendering-repaint-reflowrelayout-restyle/ 6 | 7 | Google Drive SDK: Future of online storage with filepicker.io 8 | https://developers.google.com/live/shows/31119462-3002 9 | 10 | Autoscaling Java ============================================================== 11 | dropwizard 12 | jmeter 13 | 14 | autoscaling happens in a _brief_ period of time, which is why spinup time is so important 15 | 16 | cloud antipatterns 17 | large deployments (complex dependencies, lots of artifacts) 18 | ? are we trying to reduce spinup time per instance? 19 | automatic class instantiation 20 | code generation (at _runtime_) 21 | - workaround: use compile-time generation instead 22 | - even if you generate at compile-time, permgen becomes a problem, and JVM tuning for ephemeral instances is tough 23 | managing shared state (crosstalk amongst applications) 24 | persisting configuration 25 | - build configration into the source code instead 26 | 27 | mobile dev 28 | weinre 29 | MIHTool 30 | 31 | =============================================================================== 32 | Day 2 33 | =============================================================================== 34 | 35 | Dart ========================================================================== 36 | Dart VM developed by the same guys who developed V8 37 | 38 | DOM 39 | 40 | JS engine 41 | 42 | optimize: low startup latency, high perf, low memory usage, small pauses 43 | minimize: big GC pauses, memory leaks, erratic performance 44 | 45 | 2006: the browser parsed the js, creates AST, and walks over the AST with an interpreter 46 | 47 | 2013: 48 | parse JS 49 | multi-tier adaptive compilation / series of compilers that adapts to different usage patterns 50 | _deoptimization_ if user behavior changes 51 | generational garbage collection 52 | native code is generated en masse on the fly, and discarded as patterns change 53 | code flushing 54 | support debugging and profiling 55 | 56 | multi-tier adaptive compilation 57 | 58 | V8 GC 59 | 2-phase: old space => new space 60 | stop-the-world collection, single-threaded 61 | store-buffer for generations 62 | 63 | DOM nodes are GC'd via reference counting 64 | problems: 65 | reference count doesn't say which objects point to an object 66 | doesn't handle cycles 67 | 68 | DOM "dance" is the hardest part of optimizing JS performance 69 | animations skip frames because of GC pauses, deoptimization 70 | 71 | in order to fix the DOM dance problem, the reference-counted nature of the DOM must be changed 72 | solution: make _all_ objects subject to tracing 73 | -> concurrent manipulation of DOM (cf. current ref-counted implementaion which requires a lock to mainpulate DOM) 74 | 75 | optional types are better than no types 76 | 77 | Q/A: 78 | asm.js payload is too large; can't use many JS features like garbage collection, 79 | source langs are limited 80 | can't send "strict mode" flag to JS engine because the JS engine would have to 81 | trust the code (no security/sandbox) 82 | GWT to Dart will be a reality if/when Dart VM exists in most browsers 83 | Dart on Dart VM will still have ~2x perf gains even compared to GWT=>JS 84 | dart2js is often slower than hand-written JS because it must check bounds, etc. 85 | ? Dart VM vs NaCl 86 | ? if Dart VM is never shipped, is dart2js compelling enough? 87 | 88 | 89 | GWT =========================================================================== 90 | for most cases, just extend Composite directly instead of going full MVP 91 | 92 | "testing widgets from normal java tests isn't that hard anymore 93 | GWTMockUtilities 94 | mock widgets directly! (GWTTestCase not needed) 95 | GwtMockito 96 | @RunWith(GwtMockitoTestRunner.class) 97 | 98 | good practice: 99 | think of events as notifications, _not_ commands 100 | events are usually fired only for: 101 | user input 102 | server responses 103 | events do not have return values 104 | 105 | 106 | Dart: new features ============================================================ 107 | SIMD extensions 108 | 109 | Future.wait 110 | Future.then 111 | Stream = "repeating Future" 112 | 113 | Dart compiler, compiled to JS: try.dartlang.org 114 | 115 | Dart package manager: pub.dartlang.org 116 | 117 | 118 | Google Knowledge Graph / Freebase API ========================================= 119 | 120 | freebase.com 121 | crowd-sourced 122 | 10k requests/day for free 123 | entire graph is downloadable 124 | 125 | edges (relationships) in a graph allow us to treat two entities with the same name as different 126 | 127 | Autosuggest: turn ambiguous keywords into unique identifiers 128 | HTTP query => JSON response 129 | 130 | Semantic Tagging 131 | use freebase vocabulary as a vocabulary for meaningful tags in your application 132 | 133 | Entity Collections 134 | 135 | Geosearch Collections 136 | 137 | Topical Weblinks 138 | 139 | 140 | GWT roadmap =================================================================== 141 | http://www.gwtproject.org/ 142 | https://github.com/gwtproject 143 | 144 | compiler: 145 | GWT 3.0 / 2014: 146 | GWT compiler may be producing sub-optimal code for modern JS engines 147 | commitment to improve GWTTestCase 148 | GWT 3 will remove support for ie6/7/8 149 | 150 | mobile 151 | offline support 152 | app store support 153 | 154 | GWT is in a unique position to optimize performance, because it knows about 155 | _all_ of your resources and how they fit together: 156 | gzip single deliverable 157 | async/defer