├── articles └── alan-kay-dobbs-interview.pdf ├── notes ├── courses │ ├── sicp.md │ └── intro-networking.md ├── books │ ├── pragmatic-programmer.md │ ├── joy-of-clojure.md │ ├── effective-python.md │ ├── pragmatic-thinking-learning.md │ ├── web-scraping-python.md │ ├── rest-in-practice.md │ ├── masters-of-doom.md │ ├── how-google-tests-software.md │ ├── how-win-friends.md │ ├── dream-machine.md │ ├── hackers-and-painters.md │ ├── i-lov-logs.md │ ├── the-making-of-prince-of-persia.md │ ├── problem-human-energy.md │ ├── practical-vim.md │ ├── building-microservices.md │ ├── compiler-construction.md │ ├── rework.md │ └── hackers-heroes.md ├── api.md ├── laws.md ├── bike-shedding.md ├── outages.md ├── articles │ ├── advanced-testing-with-go.md │ ├── on-criteria-of-decomposing-systems.md │ ├── emperors-old-clothes.md │ ├── computer-as-communication-device.md │ ├── intel-clear-containers.md │ ├── wisckey.md │ ├── inventing-on-principle.md │ ├── humble-programmer.md │ └── relational-model.md ├── learning.md ├── testing-philosophy.md ├── protocols │ ├── http2.md │ ├── http3.md │ └── quic.md ├── architecture.md ├── religion-computing.md ├── cockroachdb.md ├── conferences │ └── ebf-summit-2022.md ├── systems-thinking.md ├── design-patterns.md ├── mk.md ├── testing.md ├── orm.md ├── podcasts │ ├── neocities.md │ └── onmetal.md ├── docker-compose.md ├── crypto.md ├── document-services.md ├── plan9.md ├── vietnam.md ├── languages │ └── limbo.md └── concurrency.md ├── README.md ├── LICENSE ├── presentations.md ├── sources.md ├── courses.md └── posts.md /articles/alan-kay-dobbs-interview.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/katcipis/sophia/HEAD/articles/alan-kay-dobbs-interview.pdf -------------------------------------------------------------------------------- /notes/courses/sicp.md: -------------------------------------------------------------------------------- 1 | # Structure and Interpretation of Computer Programs 2 | 3 | Due to the awesomeness of the course it has got its own project 4 | [here](https://github.com/katcipis/sicp). -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Contents 2 | 3 | * [Books](./books.md) 4 | * [Articles](./articles.md) 5 | * [Blog Posts](./posts.md) 6 | * [Courses](./courses.md) 7 | * [Presentations](./presentations.md) 8 | * [Tools](./tools.md) 9 | * [Sources](./sources.md) 10 | -------------------------------------------------------------------------------- /notes/books/pragmatic-programmer.md: -------------------------------------------------------------------------------- 1 | # The Pragmatic Programmer 2 | 3 | * Boiled frogs 4 | * Stone soup 5 | * Orthogonality 6 | * Ruthless Testing (how to do you know the tests really test ?) 7 | * Good text editor (not IDE) 8 | * Good enough software 9 | * Not until you are ready (do not procrastinate, build a prototype :-) 10 | -------------------------------------------------------------------------------- /notes/api.md: -------------------------------------------------------------------------------- 1 | # A Philosophy for APIs 2 | 3 | * Mindset of shared ownership, not owned by the maintainer 4 | * Works like dependency inversion, the API is the interface (can be OO/distributed, etc) 5 | * How to define ? (collaborative effort, done first) 6 | * How to evolve ? (same as definition) 7 | * Honesty, do you really need to "improve" this API ? (Go example, find post on this) 8 | -------------------------------------------------------------------------------- /notes/books/joy-of-clojure.md: -------------------------------------------------------------------------------- 1 | # Pros 2 | 3 | * Interesting de-structuring facilities 4 | * Arity based pattern matching (not as awesome as full value pattern matching) 5 | * Some of the elegance of Lisp (uniformity and simplicity of composing functions) 6 | 7 | # Cons 8 | 9 | * Java / JVM gigantic stack traces 10 | * Algorithms implemented using Java crappy stuff :-( (like sorting) 11 | * Some of the syntax starts to seen bloated, Java like culture ([example](https://clojuredocs.org/clojure.core/-%3E)) 12 | -------------------------------------------------------------------------------- /notes/laws.md: -------------------------------------------------------------------------------- 1 | # Laws 2 | 3 | Some laws on computer science (and other interesting fields)that I collected, 4 | not that they should be considered like actual laws and rules, but usually 5 | they reflect good ideas and are good rules of thumb. 6 | 7 | * [Hyrum's](http://www.hyrumslaw.com/) 8 | * [Postel's](https://en.wikipedia.org/wiki/Robustness_principle) (Robustness Principle) 9 | * [Amdahl's](https://en.wikipedia.org/wiki/Amdahl%27s_law) 10 | * [Goodhart's](https://en.wikipedia.org/wiki/Goodhart%27s_law) 11 | * [Demeter's](https://en.wikipedia.org/wiki/Law_of_Demeter) 12 | * [Wirth's](https://en.wikipedia.org/wiki/Wirth%27s_law) 13 | -------------------------------------------------------------------------------- /notes/books/effective-python.md: -------------------------------------------------------------------------------- 1 | # Best tips 2 | 3 | * Do not use get/set, just plain attributes and the @property decorator 4 | * Create lib specific exceptions to make clear programming errors on the lib from the user using your API wrongly 5 | * Dont silence errors, it is better to allow the exception to bubble up, or handle it properly 6 | * Always specify which exception to catch, dont use just except (it will catch stuff like SystemExit and KeyboardInterrupt) 7 | * Default parameters should not be initialized with mutable values (they are initialized only once) 8 | * Dont do expensive work on import (module body), this avoids a lot of surprises and makes testing easier 9 | -------------------------------------------------------------------------------- /notes/bike-shedding.md: -------------------------------------------------------------------------------- 1 | # Bike Shedding 2 | 3 | I never knew exactly why people used the expression 4 | **bike shedding** to refer to stupid discussions. 5 | 6 | Some article (dont remember it anymore) tells the origin of this expression, 7 | it is a psychological fact about humans, the more important a decision 8 | is people will tend to discuss stupid details instead of discussing 9 | the real serious issues that actually have to be addressed. 10 | 11 | The bike shed story is a reference to this phenomenon, a group 12 | of engineers where responsible to building a nuclear power 13 | plant. Instead of discussing real design decisions they spent more 14 | than a month just discussing where and how the bike shed of the 15 | power plant would be built. That is the origin of the 16 | **bike shedding** expression. 17 | -------------------------------------------------------------------------------- /notes/outages.md: -------------------------------------------------------------------------------- 1 | # Outages 2 | 3 | Some historical outages that I keep track, ordered by root cause. 4 | I started this because I noticed that the biggest outages are 5 | usually started by the most innocent changes, configuration changes. 6 | 7 | ## Bad Configuration 8 | 9 | These are the outages that were caused by bad configuration 10 | (or involved bad configuration + bad luck/other bug): 11 | 12 | * [Cloudflare outage on July 17, 2020](https://blog.cloudflare.com/cloudflare-outage-on-july-17-2020/) 13 | * [Cloudflare outage on June 21, 2022](https://blog.cloudflare.com/cloudflare-outage-on-june-21-2022) 14 | * [Google Cloud Networking Incident #19009](https://status.cloud.google.com/incident/cloud-networking/19009) 15 | * [Cloudflare/OVHCloud](https://blog.cloudflare.com/cloudflare-perspective-of-the-october-30-2024-ovhcloud-outage/) 16 | -------------------------------------------------------------------------------- /notes/books/pragmatic-thinking-learning.md: -------------------------------------------------------------------------------- 1 | # Concentração e manter contexto 2 | 3 | Após perder o contexto em geral leva 20 minutos para recuperar ele. E segundo algumas pesquisas, ficar vendo email o tempo todo pode reduzir o QI em até 10 pontos (mais que a maconha). 4 | 5 | Portanto existem coisas a fazer pessoalmente e em equipe para reduzir interrupções. 6 | 7 | ## Pessoal 8 | 9 | * Fechar o email quando não estiver utilizando, abrir apenas em um momento em que não vai interromper algo. 10 | 11 | * Não se distrair com coisas interessantes, manter o foco e anotar coisas interessantes para ver depois. 12 | 13 | * Se vou ser interrompido, fazer anotações breves que podem me ajudar a recuperar o contexto depois. 14 | 15 | 16 | ## Equipe 17 | 18 | * Não interromper uns aos outros desnecessariamente. 19 | 20 | * Estipular um horario onde não podemos ser interrompidos (momento mais eficiente). 21 | -------------------------------------------------------------------------------- /notes/articles/advanced-testing-with-go.md: -------------------------------------------------------------------------------- 1 | # Advanced Testing with Go 2 | 3 | The [talk](https://www.youtube.com/watch?v=yszygk1cpEc) 4 | 5 | * Test in layers, all tests (unit/integration/e2e) are needed 6 | * Terraform has 10x more test code than production code 7 | * Table driven tests helps avoiding useless boilerplates 8 | * Golden files to test with human checked correct data (I call it reference files) 9 | * Test helpers, which are basically test fixtures 10 | * A neat idea is to return the teardown function on the setup function call + defer the teardown 11 | * Always try to test only exported functions (not a rule, but its a good principle) 12 | * Subprocessing mocks reminded me of toxyproxy and some principles on Release It 13 | * Provide testing public APIs for your packages, like functions that creates test config data 14 | * Provide an easy way to run your service in memory (the full service, not only part of it) 15 | -------------------------------------------------------------------------------- /notes/books/web-scraping-python.md: -------------------------------------------------------------------------------- 1 | # Web Scraping in Python 2 | 3 | ## Ideas 4 | 5 | * BeautifulSoup is your friend :-) 6 | * Natural language processing N grammar thing, pretty cool 7 | * The way markov chains are used on page rank is interesting (modelling probability of users navigation through links) 8 | * NTLK can help determining if a word is used as a noun or a verb (can be useful sometimes) 9 | * Watch out with hidden stuff that is necessary on forms (parse the form and get all stuff) 10 | * Watch out for hidden stuff that is actually honeypots 11 | 12 | ## Overview 13 | 14 | Great book to new developers who never crawled the web before. 15 | Some cool topics: 16 | 17 | * Parsing HTML 18 | * Parsing PDFs 19 | * Handling forms (and hidden stuff) 20 | * Handling sessions/cookies 21 | * Handling Javascript execution 22 | * Natural language processing techniques 23 | * Cleaning up data 24 | * Surviving encoding hell 25 | * Image processing 26 | * Captcha breaking 27 | 28 | 29 | ## Tools 30 | 31 | * [NLTK](http://www.nltk.org/) 32 | * [pillow](http://python-pillow.org/) 33 | * [tesseract](https://github.com/tesseract-ocr) 34 | 35 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2014 Tiago César Katcipis 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /notes/books/rest-in-practice.md: -------------------------------------------------------------------------------- 1 | # Key Concepts 2 | 3 | * Scalability by decoupling client from server. 4 | * Caching is a way of decoupling and helps on scalability. 5 | * The more decoupled and scalable, the harder it gets to be consistent (applies to databases too :-). 6 | * A REST API is explorable (HATEOAS), the contract is not documented on detail. 7 | * URI templating is not REST, you only know entry points. 8 | * Auto documentation by using HTTP idioms, uniform interfaces (eg. CRUD). 9 | * What needs to be documented should be done in-band with the service, not out-of-band on another place (rel links with documentation). 10 | * On a distributed system hiding necessary complexity is a bad idea. 11 | * Do not hide from the application that the system is distributed (JMI/RPC opposes this idea). 12 | 13 | 14 | # How to document ? 15 | 16 | Well, REST does not fit a IDL, it is more loose. But you have the need to build a contract. 17 | The contract is a set of protocols, where each protocol is a set of operations to achieve a business goal. 18 | Each protocol consists on a set of URIs, HTTP idioms (like verbs and status codes) and media types (different representations of information). 19 | -------------------------------------------------------------------------------- /notes/learning.md: -------------------------------------------------------------------------------- 1 | # Learning Engineering 2 | 3 | It is an interesting subject, how do you learn to be a software engineer, 4 | some ideas I had so far because of my own experience. 5 | 6 | ## Keep Curious 7 | 8 | * SICP preface 9 | * Alan Kay OOPSLA (We dont know what we are doing) 10 | * Coders at work (no consensus about any best practice) 11 | * Coders at work Joe Armstrong 12 | 13 | ## Keep Busy 14 | 15 | The best way to learn is by doing. 16 | Use Dennis Ritchie quote on the best way to learn to program in a language 17 | 18 | ## Keep Learning 19 | 20 | Specially languages, with different paradigms. 21 | Mention the pragmatic programmer + Dijkstra impressions on how language 22 | affect how we think. 23 | 24 | Also some good books on ideas on how to develop proper code/distributed systems. 25 | 26 | ## Keep Digging 27 | 28 | Mention Thorsten Ball on always working on a layer bellow the one you 29 | usually need. 30 | 31 | Joe Armstron on Coders at Work on "Opening Black Boxes". 32 | 33 | Abstractions can be great, but do open as much of them as you can =). 34 | 35 | Links to curiosity. 36 | 37 | ## Keep Writing 38 | 39 | Writing as a way to organize your thoughts. 40 | Also some SQ3R from Pragmatic Thinking and Learning. 41 | -------------------------------------------------------------------------------- /notes/testing-philosophy.md: -------------------------------------------------------------------------------- 1 | # A Philosophy for testing 2 | 3 | ## Terminology 4 | 5 | Basically 0 focus on terminology/taxonomy. 6 | 7 | From [Less is Exponentially More](https://commandcenter.blogspot.com/2012/06/less-is-exponentially-more.html): 8 | 9 | ``` 10 | My late friend Alain Fournier once told me that he considered the lowest form 11 | of academic work to be taxonomy. And you know what? Type hierarchies are just 12 | taxonomy. You need to decide what piece goes in what box 13 | ``` 14 | 15 | Ellaborate on discussions like X vs Y. 16 | 17 | ## Scope 18 | 19 | Ellaborate on scope, testing as a subset of "feedback". The domain is vast, we are 20 | going to focus on a smaller part of it. 21 | 22 | ## When not to test ? 23 | 24 | Related to scope. 25 | 26 | ## Principles 27 | 28 | [Programmer Test Principles](https://medium.com/@kentbeck_7670/programmer-test-principles-d01c064d7934) 29 | [When TDD Doesn’t Matter: What Do You Value?](https://medium.com/pragmatic-programmers/when-tdd-doesnt-matter-what-do-you-value-91c628dc4488) 30 | 31 | Extend with some other simple principles, like easy to run and with a uniform/easy 32 | interface (like make test). 33 | 34 | There is no (easy ? plausible?) way to avoid slow/non-deterministic test unless you have mostly useless tests. 35 | 36 | Navigate the trade-off space. 37 | 38 | Separate slower/non-deterministic from deterministic. 39 | -------------------------------------------------------------------------------- /notes/books/masters-of-doom.md: -------------------------------------------------------------------------------- 1 | # Masters of Doom 2 | 3 | All the revolution that happened on the eighty's was possible because 4 | of the increase of accessibility on the game market. 5 | 6 | Developing for consoles involved very expensive tools and hardware, which 7 | means having to work for a big corporation to sponsor you. 8 | 9 | On the story of the two Johns, they were able to pursue their dreams and do 10 | awesome stuff because they only needed an Apple 2 and a IBM PC (8086) and time 11 | to develop games. 12 | 13 | Both started developing at home, working on totally unrelated stuff, like burger king. 14 | It is indeed an amazing story how they fought to be able to do what they love. 15 | 16 | This increase on accessibility is similar to what happened with the web revolution at the 17 | nineties and is similar with what is happening today (2016) with the cloud and the big data thing. 18 | 19 | 20 | ## Key points 21 | 22 | * Carmack wanted to craft great programs, Romero wanted to build an empire 23 | * Romero had the ability to perceive change when he started to invest on IBM PC instead of Apple 2 24 | * Carmack hates patents, identified myself a lot with this (people have to be able to extend ideas) 25 | * Carmack was order, Romero was chaos, together they where chaordic, separated they where unbalanced 26 | * Romero failed to see that less is more (size of the team) 27 | * Lack of clear direction destroys a company 28 | * Everything that has a beginning has an end Neo :-) 29 | -------------------------------------------------------------------------------- /presentations.md: -------------------------------------------------------------------------------- 1 | # Presentations 2 | 3 | Some of the most remarkable presentations I watched. 4 | 5 | ## Software Design 6 | 7 | * [Three Directions in Design](https://www.youtube.com/watch?v=Tdwr9tweTDE) 8 | * [We Really Don't Know How to Compute!](https://www.youtube.com/watch?v=HB5TrK7A4pI) 9 | * [Design, Composition, and Performance](https://www.youtube.com/watch?v=QCwqnjxqfmY) 10 | * [Where Does Bad Code Come From?](https://www.youtube.com/watch?v=7YpFGkG-u1w) 11 | * ["Clean" Code, Horrible Performance](https://www.youtube.com/watch?v=tD5NrevFtbU&t) 12 | * [Performance Excuses Debunked](https://www.youtube.com/watch?v=x2EOOJg8FkA) 13 | 14 | ### Frameworks 15 | 16 | * [Programming with Hand Tools](https://www.youtube.com/watch?v=ShEez0JkOFw) 17 | * [8 lines of code](https://www.infoq.com/presentations/8-lines-code-refactoring) 18 | 19 | ## Mind Bending :-) 20 | 21 | * [The Computer Revolution not happened yet](https://www.youtube.com/watch?v=oKg1hTOQXoY&feature=youtu.be) 22 | * [Normal Considered Harmful](https://www.youtube.com/watch?v=FvmTSpJU-Xc) 23 | * [Power of Simplicity](https://www.youtube.com/watch?v=NdSD07U5uBs&feature=youtu.be) 24 | * [Programming is and should be fun](https://www.youtube.com/watch?v=2MYzvQ1v8Ww) 25 | 26 | ## Project Management 27 | 28 | * [The Math behind Project Scheduling, Bug Tracking, and Triage](https://www.youtube.com/watch?v=XZFwCplj4ME) 29 | 30 | ## Go 31 | 32 | ### Naming 33 | 34 | * [What's in a name ?](https://go.dev/talks/2014/names.slide#1) 35 | 36 | ### Testing 37 | 38 | * [Go Testing By Example](https://research.swtch.com/testing) 39 | * [Advanced Testing with Go](https://www.youtube.com/watch?v=8hQG7QlcLBk) 40 | -------------------------------------------------------------------------------- /notes/protocols/http2.md: -------------------------------------------------------------------------------- 1 | # HTTP 2 2 | 3 | Some notes from what I understood from HTTP2. 4 | 5 | ## Main concepts 6 | 7 | * Semantically compatible with HTTP 1 (Headers/Body/Methods/Statuses) 8 | * Multiple streams on same connection (multiplexed) 9 | * Control flow per stream 10 | * Prioritization per stream 11 | * Header compression (hpack) 12 | * Server push (to fix server inlining of resources and reduce round trips) 13 | 14 | 15 | ## Why only HTTPS ? 16 | 17 | Here is a catch, why it has to be HTTPS ? Actually, it does not and there is 18 | nothing on the spec that says anything about this. 19 | 20 | This is related to seamless protocol negotiation. 21 | 22 | 23 | ### Seamless protocol negotiation 24 | 25 | HTTP has space for protocol upgrades, an idea was to use this for HTTP 2. 26 | But there was two problems: 27 | 28 | * Latencies, HTTP2 MUST be a lot faster than HTTP or there is no point 29 | * Lots of hops on the network fucks up the protocol upgrade :-) 30 | 31 | This already happens with WebSocket. There is a lot of scenarios where 32 | WebSocket works just fine on TLS, and on HTTP it breaks down, with connections 33 | being suddenly dropped. 34 | 35 | What happens is expected, when you don't cipher things, people do take a look 36 | at your data :-). There is a lot of old infrastructure that investigates 37 | HTTP traffic trying to detect bad behaviour, when they detect it, they drop 38 | the connection. 39 | 40 | When you are on a TLS tunnel no one can see shit :-), so everything works. 41 | Having said that, TLS also has protocol negotiation embedded on it. 42 | 43 | So TLS was chosen as a way to negotiate the protocol (HTTP or HTTP2) and 44 | to prevent hops on the network from fucking up your connection. 45 | -------------------------------------------------------------------------------- /notes/architecture.md: -------------------------------------------------------------------------------- 1 | # Software Architecture 2 | 3 | Some material/ideas I found along the way on software architecture. 4 | Also some personal rants. 5 | 6 | ## REST Dissertation 7 | 8 | Roy Fielding's [Architectural Styles and the Design of Network-based Software Architectures](https://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm) 9 | is a gold mine of advice on how to deal with architecture of distributed systems. 10 | Not in a stupid "use REST for everything", but in a very thoughtful way. 11 | 12 | I have some notes [here](./articles/fielding-dissertation-rest.md). 13 | 14 | ## Good Architecture presentation: Principles 15 | 16 | Bases: 17 | 18 | * https://vimeo.com/43612849 19 | 20 | * Makes intention of the system clear (not tools) 21 | * Allows to defer tools decisions 22 | * Focus on the core logic of the problem you are solving (not related to tools) 23 | * Decouples logic from UI 24 | * Decouples logic from delivery mechanisms (HTTP/RPC/WHATEVER) 25 | * Decouples logic from persistence (the database, of course) 26 | * Decouples logic from frameworks of any kind 27 | 28 | About intention, since the metaphor used is about architecture, the plant of a building 29 | usually allow you to infer what kind of a building it is (school, library, church). 30 | 31 | Architectural diagrams and project directory layout should do the same. Im not interested 32 | on the framework chosen, or if the system is a web system using Rails. First I want to 33 | know the intention of the system as a whole, what it does, them I can search for how. 34 | 35 | ## Procedures inside the database ? 36 | 37 | * Putting procedures inside the database is the utmost level of coupling with the database 38 | * It is a tradeoff from latency to strong coupling, you are coupled to the technology now 39 | * If you need to change the database for some reason, you are going to have a bad time (eg. Neoway) 40 | -------------------------------------------------------------------------------- /notes/books/how-google-tests-software.md: -------------------------------------------------------------------------------- 1 | # Idéias 2 | 3 | ## Testes 4 | 5 | * Desenvolvedor é dono da qualidade do software, mais envolvido nos testes (TDD). 6 | * Investir cada vez mais em build/integração/entrega contínua. 7 | * Iterações/entrega deve ser rápida, quanto mais rápido o feedback melhor. 8 | * Ferramentas de análise e manutenção de test cases são necessárias. 9 | * Manutenção dos testes deve envolver integração com bugzilla, avaliar onde é necessário maior esforço de testes. 10 | * Planejamento de testes, não testar a esmo (ACC). 11 | * Não dá para testar tudo, avaliar o que tem maior ROI (return of investiment) 12 | * Análise estática do código pode ajudar a identificar vizinhanças ruins no código. 13 | * Tudo que for repetitivo deve ser automatizado, pessoas não são muito boas em repetir coisas. 14 | * Mudança cultural é necessária para fazer testes darem certo. 15 | * Escassez é a mãe da otimização, priorizar e aprimorar a produtividade. 16 | * Dogfooding, temos de usar os nossos próprios produtos ao máximo. 17 | * Crowd sourcing é uma boa tambem, dogfooding é mais praticavel no mercado de embarcados. 18 | 19 | 20 | h3. 4 fundamentos básicos para se obter bons testes e bom software 21 | 22 | * Habilidade (bons fundamentos de computação) 23 | * Escassez de recursos humanos (obriga a otimizar os processos) 24 | * Automatizar ao máximo (mas nem tudo pode ser automatizado) 25 | * Iterar e entregar o mais rápido possível 26 | 27 | 28 | ## Entrevista de emprego 29 | 30 | Tem umas idéias interessantes no livro sobre isso, mas nao degustei bem ainda, mas envolve ver se o candidato vai fazer 31 | as perguntas certas ao ser apresentado a um problema relativamente simples ;-). 32 | 33 | # Projetos interessantes 34 | 35 | * "Angular":http://angularjs.org/ 36 | * "Test analytics":https://code.google.com/p/test-analytics 37 | * "Native Driver":https://code.google.com/p/nativedriver 38 | * "Web driver/Selenium":https://code.google.com/p/selenium 39 | * "Autotest":http://autotest.github.io 40 | -------------------------------------------------------------------------------- /notes/courses/intro-networking.md: -------------------------------------------------------------------------------- 1 | # Introduction to Computer Networking 2 | 3 | Notes from the course 4 | [Introduction to Computer Networking](https://lagunita.stanford.edu/courses/Engineering/Networking-SP/SelfPaced/about) 5 | 6 | # End to End principle 7 | 8 | The end to end principle is a systems communication principle 9 | that dictates that every feature that can be safely implemented 10 | on the endpoints should be done that way, instead of building 11 | a smart network, build something very simple that enables 12 | endpoints to build more complex stuff. 13 | 14 | This reminds me a lot of the smart endpoints / dump pipes 15 | principles for services. 16 | 17 | Sometimes it is hard to see the benefits from it. The largest 18 | network on the time was the telecommunication network and it was 19 | based on a smart network with extremely dump endpoints (the old phones). 20 | 21 | On the design of the IP protocol this idea of a dump pipe was applied, 22 | the IP protocol guarantees almost nothing. It does not guarantee delivery or 23 | order. At the time these seemed like pretty basic stuff, but if features 24 | like retransmission and acks where embedded on the network it would be 25 | very hard or even impossible to build systems like VoIP, where low latency 26 | is more important that reliability. 27 | 28 | It is hard to predict how your communication protocol is going to be 29 | used, to avoid problem it is best to keep it simple since it will 30 | be the base for the communication of a lot of different applications. 31 | 32 | Seems like a good principle to be applied to any distributed system, 33 | when using message brokers it is good to have this in mind. 34 | 35 | A real life example of this is the wireless protocol that does retransmission 36 | at the layer level since there is so much loss on wireless. This enables 37 | recovery from losses with low latency (at layer 2), but for protocols where 38 | retransmission of stale data is harmful this is a bad choice and there is 39 | no way around it on wireless since it is enforced at the link layer. 40 | -------------------------------------------------------------------------------- /notes/religion-computing.md: -------------------------------------------------------------------------------- 1 | # Religion on Computing 2 | 3 | What do I mean religion on computing ? the whole "I found the true way to solve 4 | something and need to spread the evangelion". 5 | 6 | In time I grew tired of this and started to accumulate quotes from smarter 7 | people than me that are very inspiring. 8 | 9 | ## Don't be a Bible Salesman 10 | 11 | From the preface of the [Structure and Interpretation of Computer Programs book](https://mitpress.mit.edu/sites/default/files/sicp/full-text/book/book-Z-H-3.html): 12 | 13 | ``` 14 | This book is dedicated, in respect and admiration, to the spirit that lives in the computer. 15 | 16 | ``I think that it's extraordinarily important that we in computer science keep fun in computing. 17 | When it started out, it was an awful lot of fun. Of course, the paying customers 18 | got shafted every now and then, and after a while we began to take their 19 | complaints seriously. 20 | 21 | We began to feel as if we really were responsible for the successful, 22 | error-free perfect use of these machines. I don't think we are. 23 | I think we're responsible for stretching them, setting them off in new 24 | directions, and keeping fun in the house. I hope the field of computer science 25 | never loses its sense of fun. Above all, I hope we don't become missionaries. 26 | 27 | Don't feel as if you're Bible salesmen. The world has too many of those already. 28 | What you know about computing other people will learn. 29 | Don't feel as if the key to successful computing is only in your hands. 30 | What's in your hands, I think and hope, is intelligence: the ability to see 31 | the machine as more than when you were first led up to it, 32 | that you can make it more.'' 33 | 34 | Alan J. Perlis (April 1, 1922-February 7, 1990) 35 | ``` 36 | 37 | ## Real Engineer 38 | 39 | From [Gerald Sussman "We Really Don't Know How To Compute!"](https://youtu.be/HB5TrK7A4pI?t=2340): 40 | 41 | ``` 42 | A real engineer doesn't want just a religion on how to solve a problem. 43 | Like Object Oriented, or functional or imperative or logical programming. 44 | ``` 45 | -------------------------------------------------------------------------------- /notes/books/how-win-friends.md: -------------------------------------------------------------------------------- 1 | # How to win friends and influence people 2 | 3 | 4 | ## Interesting points 5 | 6 | * Retrospectives being done since 1930 :-) 7 | * Dont criticize people, understand them 8 | * If you want honey, dont kick the beehive 9 | * People want to be important and appreciated 10 | * People can even go crazy without appreciation 11 | * Speak no harm, focus on the qualities 12 | * Inspire people so they get eager to do stuff, instead of just commanding 13 | * Find out what people want and care about that 14 | * Everyone is in love with its own name, use this 15 | * Listening to someone is one of the best compliments 16 | * Encourage others to talk about themselves 17 | * Dramatize 18 | * Love the game 19 | * Throw a challenge 20 | * Ask questions, no orders 21 | * Let people save face 22 | 23 | 24 | ## Just thinking on what you are going to say next 25 | 26 | I do this all the time and it is awful, the best thing to do is truly listen to 27 | the person, instead of just listening pretty fast until I got my chance to talk. 28 | 29 | This is a sign of an inflated ego and wrong sense of self importance. 30 | 31 | 32 | ## People love their own names 33 | 34 | Cool history about how a kid with 8 years old convinced all its friends 35 | to find food to its rabbits just by proposing to name the rabbits after them. 36 | 37 | Humans do all kind of stuff to get their names on stuff, like libraries and 38 | hospitals. Give them sense of importance by using this. 39 | 40 | 41 | ## Being ruthless as a mean to be awesome 42 | 43 | Now I think this is just plain bullshit. I thought being blunt and always criticizing if 44 | something was not ok was better. In part is because if someone do this with me, usually I do 45 | not care, and if I was really wrong, I enjoy it. 46 | 47 | But this is another mistake that I'm always making, **people are NOT like me** and I cant expect 48 | them to be. In general people respond to praise. 49 | 50 | Praise is harder and more effective than criticizing, I need to think on ways to criticize 51 | that involve praising first. Two interesting techniques: 52 | 53 | * Always praise before talking about a mistake, makes it easier 54 | * Talk how I myself have done similar mistakes on the past 55 | -------------------------------------------------------------------------------- /notes/cockroachdb.md: -------------------------------------------------------------------------------- 1 | # CockroachDB 2 | 3 | ## Quick Facts 4 | 5 | * Uses ordering by key instead of consistent hashing 6 | * Depending on sharding key, can be random (like consistent hashing) 7 | * Uses a Raft log for consistency 8 | * Uses one raft group per range not for the whole database (it has lot of rafts) 9 | * Gossip is used to bootstrap raft (where to find indexes) 10 | * Shards are called ranges 11 | * Ranges are on the size of approximately 64mb 12 | * If ranges are too big rebalancing can get messy 13 | * Rebalancing is automatic 14 | * There is no coordinator, all nodes answers queries transparently 15 | * Works on consistency windows based on NTP clock drift window 16 | * Nodes that get too big clock drift are removed from cluster 17 | * No locks, optimistic concurrency is implemented, with try agains 18 | 19 | 20 | ## Architectural Layers 21 | 22 | CockroachDB has well defined architectural layers, where the top layer only 23 | interacts with the layer immediately bellow it. The layers are (from top to bottom): 24 | 25 | - SQL API 26 | - Transaction 27 | - Distribution 28 | - Replication 29 | - Storage 30 | 31 | A brief description of each layer responsibility. 32 | 33 | ### SQL 34 | 35 | Here the SQL statements are transformed to KV operations, where KV stands 36 | for Key/Value. 37 | 38 | ### Transaction 39 | 40 | This layer guarantees the ACID properties when receiving 41 | a set of KV operations to perform (and transactions, duh). 42 | 43 | ### Distribution 44 | 45 | The distribution layer is responsible for delivering an abstraction of a single 46 | key/value map of data, while the map will actually be sharded on 47 | multiple ranges across different nodes (a [DHT](https://en.wikipedia.org/wiki/Distributed_hash_table)). 48 | 49 | ### Replication 50 | 51 | This layer is responsible for replicating writes and handling the leases for 52 | read operations, nodes without a valid lease for a read operation can't 53 | answer a read operation for example. It will also maintain the consensus and 54 | consistency of data that is replicated across multiple nodes. 55 | 56 | ### Storage 57 | 58 | The final layer that is responsible for actually storing the key/value data 59 | on disk. Right now (2019) this is done with RocksDB. 60 | 61 | ## Material 62 | 63 | * [CockroachDB](http://cs.ulb.ac.be/public/_media/teaching/cockroachdb_2017.pdf) 64 | * [Design](https://www.youtube.com/watch?v=p8aJuk7TJJA) 65 | -------------------------------------------------------------------------------- /sources.md: -------------------------------------------------------------------------------- 1 | # Computer Science 2 | 3 | * [Learning CS](https://teachyourselfcs.com/) 4 | * [Distributed Systems Reading Group](http://dsrg.pdos.csail.mit.edu/papers/) 5 | 6 | # Books 7 | 8 | * [Alan Kay Reading List](http://www.squeakland.org/resources/books/readingList.jsp) 9 | * [Ganssle Group](http://www.ganssle.com/books/books1.htm) 10 | 11 | # Courses 12 | 13 | * [Execute Program](https://www.executeprogram.com/) 14 | * [Coursera](https://www.coursera.org/) 15 | * [Edx](https://www.edx.org/) 16 | * [IBM Big Data University](http://bigdatauniversity.com/) 17 | * [MIT](http://ocw.mit.edu/index.htm) 18 | * [Khan](https://www.khanacademy.org/) 19 | * [The Pragmatic Studio](http://pragmaticstudio.com/) 20 | * [Lynda](http://www.lynda.com/) 21 | * [Harvard CS50](https://manual.cs50.net/seminars/) 22 | * [Skillshare](http://www.skillshare.com/) 23 | * [CreativeLive](https://www.creativelive.com/) 24 | * [Turing lectures](http://amturing.acm.org/lectures.cfm) 25 | 26 | 27 | # Podcasts 28 | 29 | * [On The Metal](https://oxide.computer/blog/categories/on-the-metal/) 30 | * [corecursive](https://corecursive.com/) 31 | * [Security. Cryptography. Whatever.](https://securitycryptographywhatever.buzzsprout.com/) 32 | * [Future Of Coding](https://futureofcoding.org/) 33 | * [Roundabout: Creative Chaos](https://roundaboutfm.com/) 34 | * [Weird Trick Mafia](https://weirdtrickmafia.fm/) 35 | * [DevChat](http://devchat.tv/) 36 | * [Grok](http://www.grokpodcast.com/) 37 | * [SE Radio](http://www.se-radio.net/) 38 | * [Software Engineering Daily](https://softwareengineeringdaily.com) 39 | * [Elegant Code](http://elegantcode.com/) 40 | * [The Changelog](https://changelog.com/podcast/) 41 | * [Full Stack Radio](http://fullstackradio.com/) 42 | * [New Stack](http://thenewstack.io/) 43 | 44 | 45 | # Blogs 46 | 47 | * [Paul Graham](http://www.paulgraham.com/articles.html) 48 | * [Toolshed](https://toolshed.com/articles.html) 49 | * [Brendan Gregg](http://www.brendangregg.com/blog/index.html) 50 | * [Free electrons](http://free-electrons.com/blog/) 51 | * [James Greening](http://www.renaissancesoftware.net/blog/) 52 | * [LWN](https://lwn.net/) 53 | * [Data Science](https://github.com/rushter/data-science-blogs) 54 | 55 | 56 | # Free books 57 | 58 | * [FreeBooks](https://github.com/vhf/free-programming-books) 59 | 60 | 61 | # Other 62 | 63 | * [Flux Collective](https://read.fluxcollective.org/) 64 | * [Pointer](http://www.pointer.io/purpose) 65 | * [C2](http://c2.com/cgi/wiki) 66 | * [Hacking Textfiles](http://www.textfiles.com/hacking/) 67 | -------------------------------------------------------------------------------- /notes/conferences/ebf-summit-2022.md: -------------------------------------------------------------------------------- 1 | # eBPF Summit 2022 2 | 3 | # Cilium Standalone XDP L4 Load Balancer 4 | 5 | Presented an interesting use case where they have their own datacenter/load 6 | balancing infra. 7 | 8 | The load balancing used to be done with IPVS and it was migrated to 9 | the Cilium XDP L4 Load Balancer. 10 | 11 | Got more than 10x improved performance (CPU usage/packets per sec). 12 | 13 | # When you need to overcome your fear and build your own data-driven eBPF firewall 14 | 15 | They wanted to implement a firewall using dest/src IP + src MAC as the match 16 | to allow/deny traffic. The guy went on in details on how to do that in eBPF 17 | using a simple eBPF map. 18 | 19 | I particularly liked the solution since they extracted the eBPF logic in a very 20 | simple C program and the rest of the interface manipulating the firewall rules 21 | were Go programs which then provided more sophisticated interfaces (the control plane). 22 | 23 | One of the few presentations where in the end it was talked about downsides of eBPF. 24 | In their case debugging was quite hard in some cases, sadly he didn't provided a lot 25 | of detail on why it was hard. 26 | 27 | # eBPF for IO latency monitoring 28 | 29 | Showcased Digital Ocean's usage of eBPF to collect IO latency information and 30 | exporting it to Prometheus. Basic eBPF program providing histogram on the map 31 | and then more high level code getting that data and exporting it on Prometheus. 32 | 33 | Also talked about some challenges on eBPF. In this case changes on tracepoints 34 | and structures on the Kernel. CO-RE and BTF are supposed to help with this but 35 | I'm not sure how far it can help, depending on the change on a tracepoint I suppose 36 | it is inevitable to see a break. Sadly it was quite quick and no details were 37 | provided. 38 | 39 | # Improving Cilium's eBPF-based DSR performance by adding support for IP-in-IP 40 | 41 | This one was quite interesting because the guy made changes on Cilium load balancer 42 | to allow for more performant DSR load balancing. In order to get fast paths on 43 | switches he used IP-in-IP, encapsulating the original package info on a SNAT'ed package. 44 | So you get the performance of doing the SNAT but the end result is still DSR. 45 | 46 | # The Promise of eBPF For The Edge 47 | 48 | There was nothing very new/extraordinay on this presentation to me other 49 | than the fact that Red Hat seems to be investing a lot on eBPF in general. 50 | Including a Rust library, which seems interesting: 51 | 52 | - https://github.com/aya-rs/aya 53 | -------------------------------------------------------------------------------- /notes/protocols/http3.md: -------------------------------------------------------------------------------- 1 | # HTTP 3 2 | 3 | One of the main reasoning's of http3 is some shortcomings of TCP, 4 | in special the limitations of multiplexing multiple streams 5 | on top of a TCP stream. TCP was not built with that in mind, 6 | so you end up having a head of line issues, which is caused 7 | by TCP one stream model + guaranteed packet order. The idea is not 8 | that TCP is terrible because of that, when you want just one 9 | stream it is pretty good, but when you want multiple streams you 10 | end up creating a lot of connections, like http 1, or multiplexing 11 | N streams on top of one stream, like http 2, which is also 12 | not optimal because of head of line issues. 13 | 14 | Given that one of the main limitations of http2 is because of TCP limitations, 15 | one of the most interesting artifacts built from HTTP3 is the QUIC 16 | protocol itself, which provides abstractions similar to TCP but with 17 | some different key design decisions. It is specially interesting because 18 | QUIC has been developed as a separate transport protocol, on top of UDP, 19 | so you can use it also for non-http3 traffic. Basically you have a new option 20 | besides TCP/UDP to do networking, that will work on the internet infrastructure 21 | (or http3 will never exist). 22 | 23 | The core differences is that a QUIC connection is built on top of UDP 24 | not IP, which brings tradeoffs, and an interesting one is that the 25 | concept of a connection is not tied to specific IPs/Port anymore, so 26 | you could be on a mobile connection, change your IP, and keep all 27 | your connections. Maintaining open connections gets pretty inexpensive, 28 | there is no open file descriptor per connection, no reconnection issues 29 | (at least no obvious ones). Since this brings some security concerns 30 | (you don't have the IP as a form of identifying yourself), QUIC obligates 31 | the use of TLS, which is how identification is performed (instead of using IP), 32 | or else it would be trivial to hijack a QUIC connection. 33 | 34 | So far I only see some interesting advantages on the QUIC level, like lightweight 35 | connections, faster handshake for establishing new connections, etc. I'm not 36 | aware of anything terribly different on http3 itself, besides being built 37 | on top of QUIC instead of TCP. 38 | 39 | As a whole http3 would turn much easier to implement basic request/response 40 | models even for requests that may take a long time to produce an answer, 41 | avoiding the complexity of asynchronous APIs. 42 | 43 | 44 | # Material 45 | 46 | * [HTTP 3 for everyone](https://fosdem.org/2020/schedule/event/http3/) 47 | -------------------------------------------------------------------------------- /notes/articles/on-criteria-of-decomposing-systems.md: -------------------------------------------------------------------------------- 1 | # On the Criteria To Be Used in Decomposing Systems into Modules 2 | 3 | The [On the Criteria To Be Used in Decomposing Systems into Modules](https://www.win.tue.nl/~wstomv/edu/2ip30/references/criteria_for_modularization.pdf) 4 | article is just pure gold on good criteria to decomposing systems into modules (using assembly 5 | on the examples, kick ass !!) 6 | 7 | 8 | ## Isolate system from hard design choices 9 | 10 | The main reason that I got from the article that was pretty new to me was this one. 11 | Basically hard design choices (like how to persist data) have a good chance to change 12 | on the future. When they do change, what is the impact on your system. 13 | 14 | He proposes two designs and shows the impact of changing some design decisions on both. 15 | On the flow oriented design (the flow of the application), almost all changes implied 16 | changing the whole system. 17 | 18 | On the second design, done to isolate the implementation details from the rest of the system, 19 | you usually just had to change one module per change made (reminds me of orthogonality). 20 | 21 | On the end, change is the driving factor to decide if a design is good, how well it can 22 | handle changes defines the quality. 23 | 24 | 25 | ## What about single responsibility ? 26 | 27 | There is the old Single Responsibility principle, from SOLID. It has its merit since 28 | it is too trying to optimize your code to change, but it is a lot harder to understand 29 | than just isolating the system from hard design choices. 30 | 31 | Basically it advocates that each module must have only one reason to change, but this 32 | is inherently ambiguous, what is a reason ? What is a responsibility ? 33 | 34 | The best explanation I found for this is based on the nature of the reason, or the 35 | owner of the reason. Sometimes this is pretty hard to identify, so Im going to use a 36 | very simple example, database and UI. 37 | 38 | Database schemas will change because of the desires of a DBA, wanting to organize its 39 | database better. 40 | 41 | UI will change because of a product owner not satisfied with the UX, or some designer 42 | that wants to implement something fancier. 43 | 44 | If you don't isolate this two things from each other you will have serious problems. 45 | The first one is the surprise effect, no one expects to a change on the UI to break 46 | the database (like saving garbage there). It makes you look like an idiot. 47 | 48 | This will be exacerbated by another problem, the pace of change (which is mentioned on 49 | Building Microservices as one of the reasons to split a service in two). 50 | The UI usually has a much greater rate of change than the database, so every time something 51 | changes on the UI you have a great deal of possibility of breaking something related to 52 | the database (which probably will not change too much, or not change at all). 53 | 54 | Difference on rate of change is usually is a hint that two concepts are not related, and actually 55 | should be split from each other completely. 56 | -------------------------------------------------------------------------------- /courses.md: -------------------------------------------------------------------------------- 1 | # MOOCs 2 | 3 | * [Performance Engineering of Software Systems](https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-172-performance-engineering-of-software-systems-fall-2018/) 4 | * [Technical Writing](https://developers.google.com/tech-writing) 5 | * [Programming for the Puzzled](https://www.youtube.com/playlist?list=PLUl4u3cNGP62QumaaZtCCjkID-NgqrleA&app=desktop) 6 | * [Image and video processing](https://www.coursera.org/course/images) 7 | 8 | 9 | # Distributed Systems 10 | 11 | * [Distributed Systems Lectures from Kleppmann](https://www.youtube.com/playlist?list=PLeKd45zvjcDFUEv_ohr_HdUFe97RItdiB) 12 | 13 | 14 | # Networking 15 | 16 | * [Introduction to Networking](https://www.edx.org/course/introduction-to-networking) 17 | * [Computer Networking](https://www.coursera.org/learn/computer-networking) 18 | 19 | 20 | # Recommended 21 | 22 | * [Math for Computer Science](https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-042j-mathematics-for-computer-science-fall-2010/video-lectures/) 23 | * [Introduction to Math Thinking](https://www.coursera.org/course/maththink/) 24 | * [Text Retrieval and Search Engines](https://www.coursera.org/course/textretrieval) 25 | * [MACHINE LEARNING AND COMPUTATIONAL STATISTICS](http://davidrosenberg.github.io/ml2016/#home) 26 | 27 | 28 | # Languages 29 | 30 | * [Comprehensive Rust](https://google.github.io/comprehensive-rust/) 31 | * [Oregon Programming Language Summer School](https://www.cs.uoregon.edu/research/summerschool/archives.html) 32 | 33 | 34 | ## Machine Learning 35 | 36 | * [Machine Learning Introduction: Andrew Ng](https://www.coursera.org/specializations/machine-learning-introduction) 37 | * [Probabilistic Graphical Models Specialization](https://www.coursera.org/specializations/probabilistic-graphical-models) 38 | * [Convolutional Neural Networks in TensorFlow](https://www.coursera.org/learn/convolutional-neural-networks-tensorflow/home/welcome) 39 | * [Neural Networks for Machine Learning](https://www.coursera.org/course/neuralnets) 40 | * [Introduction to Recommender Systems](https://www.coursera.org/learn/recommender-systems/outline) 41 | 42 | 43 | # Algorithms 44 | 45 | * [Data Structures and Algorithms](https://www.coursera.org/specializations/data-structures-algorithms) 46 | * [Algorithms Specialization](https://www.coursera.org/specializations/algorithms) 47 | * [Skiena Algorithms Analysis](https://www.youtube.com/playlist?list=PLOtl7M3yp-DX32N0fVIyvn7ipWKNGmwpp) 48 | * [Skiena's Algorithms Lectures](http://www3.cs.stonybrook.edu/~algorith/video-lectures/) 49 | 50 | 51 | ## Doing 52 | 53 | * [Generative AI with LLMs](https://www.coursera.org/learn/generative-ai-with-llms/) 54 | * [SICP](https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-001-structure-and-interpretation-of-computer-programs-spring-2005/video-lectures/) 55 | 56 | 57 | ## Done 58 | 59 | * [Introduction to Computer Networking](https://www.youtube.com/playlist?list=PLvFG2xYBrYAQCyz4Wx3NPoYJOFjvU7g2Z) 60 | * [Cloud Computing Concepts](https://www.coursera.org/course/cloudcomputing) 61 | 62 | # Fun Exercises 63 | 64 | * [Project Euler](https://projecteuler.net/) 65 | -------------------------------------------------------------------------------- /notes/systems-thinking.md: -------------------------------------------------------------------------------- 1 | # Systems Thinking/Design 2 | 3 | Some interesting material on systems thinking/design. Not focused 4 | necessarily in software, but a more broad view of the matter. 5 | 6 | ## Systems design explains the world: volume 1 7 | 8 | Can be seen [here](https://apenwarr.ca/log/?m=202012), from Apenwarr. 9 | Very interesting post, digs a lot on the fallacies of full decentralization/ 10 | flat structures, for example: 11 | 12 | ``` 13 | "This apparent lack of structure too often disguised an informal, 14 | unacknowledged and unaccountable leadership that was all the more pernicious 15 | because its very existence was denied." 16 | 17 | "Informal, unacknowledged, and unaccountable" control is just as common 18 | in distributed computing systems as it is in human social systems. 19 | 20 | The truth is, nearly every attempt to design a hierarchy-free, "flat" control 21 | system just moves the central control around until you can't see it anymore. 22 | Human structures all have leaders, whether implicit or explicit, 23 | and the explicit ones tend to be more diverse 24 | ``` 25 | 26 | It also provides some great examples of computing/social systems 27 | that are actually hybrid, instead of fully decentralized. It does 28 | kinda defend that there is no such thing as full decentralization 29 | in an actual working system (or it is very rare and can be 30 | dangerous given hidden power structures). 31 | 32 | ``` 33 | The web depends on centrally controlled DNS and centrally approved TLS 34 | certificate issuers; the global Internet depends on a small cabal who sorts 35 | out routing problems. 36 | 37 | Every blockchain depends on whoever decides if your preferred chain will fork 38 | this week, and whoever runs the popular exchanges, and whoever decides whether 39 | to arrest those people. 40 | 41 | Distributed radio networks depend on centralized government spectrum licenses. 42 | 43 | Democracy depends on someone enforcing your right to vote. 44 | Capitalism depends on someone enforcing the rules of a "free" marketplace. 45 | ``` 46 | 47 | On truly distributed/de-centralized existing: 48 | 49 | ``` 50 | Truly distributed systems do exist. Earth's ecosystem is perhaps one 51 | (although it's becoming increasingly fragile and dependent on 52 | humans not to break it). 53 | 54 | Truly distributed databases using Raft consensus or similar algorithms 55 | certainly exist and work. Distributed version control (like git) really is 56 | distributed, although we ironically end up re-centralizing our usage of it 57 | through something like Github. 58 | ``` 59 | 60 | It is interesting that all examples, except Earth's ecosystem, are 61 | smaller/specialized systems, usually when building an actual system 62 | that an end user depends on, you end up having an hybrid of fully 63 | distributed solutions and some centralization (like identity/authorization). 64 | 65 | And some good final advice on how to deal with the whole problem 66 | when designing systems (also social ones): 67 | 68 | ``` 69 | In systems design, there is rarely a single right answer that applies 70 | everywhere. But with centralized vs distributed systems, my rule of thumb is 71 | to do exactly what Jo Freeman suggested: 72 | 73 | at least make sure the control structure is explicit. 74 | When it's explicit, you can debug it. 75 | ``` 76 | -------------------------------------------------------------------------------- /notes/design-patterns.md: -------------------------------------------------------------------------------- 1 | # Design Patterns 2 | 3 | * [Rob Pike on patterns](https://youtu.be/5kj5ApnhPAE?t=301) 4 | * [Alan Kay on patterns](https://www.drdobbs.com/architecture-and-design/interview-with-alan-kay/240003442?pgno=4) 5 | * [Roy Fielding on patterns](https://www.ics.uci.edu/~fielding/pubs/dissertation/software_arch.htm#sec_1_6) 6 | 7 | ## Alan Kay on Patterns 8 | 9 | Quote from [Alan Kay interview on Dr Dobbs](https://link.springer.com/content/pdf/bbm%3A978-3-319-90008-7%2F1.pdf): 10 | 11 | ``` 12 | The most disastrous thing about programming—to pick one of 13 | the 10 most disastrous things about programming—there’s a very 14 | popular movement based on pattern languages. When Christopher 15 | Alexander first did that in architecture, he was looking at 2,000 years 16 | of ways that humans have made themselves comfortable. So there 17 | was actually something to it, because he was dealing with a genome 18 | that hasn’t changed that much. I think he got a few hundred valuable 19 | patterns out of it. But the bug in trying to do that in computing is 20 | the assumption that we know anything at all about programming. 21 | So extracting patterns from today’s programming practices ennobles 22 | them in a way they don’t deserve. It actually gives them more cachet. 23 | ``` 24 | 25 | ## Form Follows Function: Design should make domain clear 26 | 27 | This is not a problem intrinsic on using design patterns, but 100% of the cases 28 | that I was exposed to people that loved design patterns the design patterns 29 | were the first class citizen. The first thing you would see when looking at 30 | names of folders/modules/classes/functions/etc are the patterns names. 31 | 32 | Factory this, Proxy that, ObservableSomething. That always felt confusing to me 33 | because a pattern is something underlying a structure, it takes some time and 34 | squinting to see the pattern is there, and the design patterns movement seem to 35 | love them so much that they make it explicit, so instead of having to squint 36 | to see the pattern I need to squint to understand what the code does. Like Kay 37 | said, people ennoble patterns way too much. 38 | 39 | Quoting from [Taste for Makers](http://www.paulgraham.com/taste.html): 40 | 41 | ``` 42 | Good design is suggestive. Jane Austen's novels contain almost no description; 43 | instead of telling you how everything looks, she tells her story so well that 44 | you envision the scene for yourself. 45 | 46 | Likewise, a painting that suggests is usually more engaging than one that tells. 47 | Everyone makes up their own story about the Mona Lisa. 48 | ``` 49 | 50 | I can't shake the feeling that this is what is happening when you focus too 51 | much on patterns and enter in this explanatory mode about it. 52 | 53 | It seems to be a case of function following form, form or the pattern comes first, and 54 | then follows the function. Which is a great way to make APIs/services that are 55 | terrible to use/maintain, because the main purpose of code is to function, it 56 | is not a work of art, it runs and does things for you, it can be beautiful but 57 | the beauty should serve the functionality in a way that the beauty is in the 58 | details, but at first it is just very easy to use and to understand what the 59 | software is about (and it should not be about patterns or MVC or whatever). 60 | 61 | There is some interesting information on making things domain based on 62 | [CUPID](https://dannorth.net/2022/02/10/cupid-for-joyful-coding/#domain-based). 63 | -------------------------------------------------------------------------------- /notes/articles/emperors-old-clothes.md: -------------------------------------------------------------------------------- 1 | # The Emperors Old Clothes 2 | 3 | ``` 4 | I conclude that there are two ways of constructing a 5 | software design: One way is to make it so simple that 6 | there are obviously no deficiencies and the other way is 7 | to make it so complicated that there are no obvious 8 | deficiencies. 9 | ``` 10 | 11 | Notes from [The Emperor's Old Clothes](https://dl.acm.org/doi/pdf/10.1145/358549.358561). 12 | 13 | This one is a really good read that caused an impression on me for two reasons. 14 | The first is the candor of the author on talking about his own mistakes. He basically tells a story where 15 | they kept trying to implement a system and it only got more and more delayed and it took them 2 16 | years to finally give up and deliver nothing. It has all the classics of frog boiling, they slowed 17 | cooked themselves not perceiving that delivering would not be feasible, it was too much features 18 | and too little hardware. And yet he admits his mistakes and was surprised he was not fired but 19 | actually got even more responsibilities, which makes sense since he owned his mistakes. 20 | 21 | The second point he makes is how it is impossible (or close to it) to design a good programming 22 | language by commitee. He describes the slow and painful process of watching the language bloat 23 | as each iteration of the commitee only adds more cruft to it. In this case talking about the new 24 | revised versions of ALGOL: 25 | 26 | ``` 27 | Three months came and went--not a word of the 28 | new draft appeared. After six months, in October 1966, 29 | the ALGOL working group met in Warsaw. It had before 30 | it an even longer and thicker document, full of errors 31 | corrected at the last minute, describing equally obscurely 32 | yet another different, and to me, equally unattractive 33 | language. The experts in the group could not see the 34 | defects of the design and they firmly resolved to adopt 35 | the draft, believing it would be completed in three 36 | months. In vain, I told them it would not. In vain, I 37 | urged them to remove some of the technical mistakes of 38 | the language, the predominance of references, the default 39 | type conversions. Far from wishing to simplify the language, 40 | the working group actually asked the authors to 41 | include even more complex features like overloading of 42 | operators and concurrency 43 | ``` 44 | 45 | And then a snippet of wisdom on not adopting new features too fast: 46 | 47 | ``` 48 | A feature which is omitted can always be 49 | added later, when its design and its implications are well 50 | understood. A feature which is included before it is fully 51 | understood can never be removed later. 52 | ``` 53 | 54 | There is a caveat though, not every feature can be added later. Some systems are built 55 | around ideas that may be at odds with the future feature, making it hard to implement 56 | it unless with clumsy hacks. To design is to constrain and constraints always make properties 57 | emerge in a system, these properties will make some features easy to implement and others 58 | impossible. The most recent case of this the whole Go parametric types implementation, which 59 | doesn't seem to fit in the original language constraints (like operators are not first class 60 | citizens) and in the end the whole thing feels like a hack, it may still be useful but it will 61 | still be a hack and less expressive than languages that had parametric types at the core of the 62 | language design from the start. 63 | -------------------------------------------------------------------------------- /notes/books/dream-machine.md: -------------------------------------------------------------------------------- 1 | # The Dream Machine 2 | 3 | Notes from [The Dream Machine](https://www.amazon.com/Dream-Machine-M-Mitchell-Waldrop/dp/1732265119). 4 | 5 | # Math Formalism and Notation 6 | 7 | Since I'm plagued by feelings of inadequacy (and as any good human, 8 | quite self centered) anything that makes me feel less hopeless gets 9 | my attention, and in this case it was the part of the book 10 | that introduces [Norbert Wiener](https://en.wikipedia.org/wiki/Norbert_Wiener). 11 | 12 | The guy has a very strong background in math and is extremely creative, 13 | being able to envision things on the computer that most of his fellows 14 | mathematicians failed to do. 15 | 16 | But what got my attention was his approach to math, even though he had an 17 | explicit/strong background in formal math, he didn't liked to approach 18 | math formally and specially disliked mathematical notation/syntax, 19 | for him math notation was a necessary evil, and most of his mathematics 20 | was done more guided by intuition and by thinking on physical terms, 21 | instead of classical theory/symbols/notation. 22 | 23 | I'm not even close to having a strong math background, but this does 24 | give me hope on being able to use math usefully someday even though 25 | I was never able to through a lot of the math formalisms. I specially 26 | dislike math notation since it is usually very terse/dense, like a very 27 | complex regex =P. And yet some things connected to math, like proofs by 28 | induction and logical math, applied on computing, I was able to grasp 29 | much more easily. 30 | 31 | Math is very central to creativity and abstract reasoning, and now I feel 32 | less hopeless because it does seem to exist different ways to leverage 33 | that, not only the most classical way that I was never able to grasp 34 | (and was always ashamed of not being able to). 35 | 36 | One curious thing the book mentions, Wiener proclaimed that 37 | [Courant](https://en.wikipedia.org/wiki/Richard_Courant) stole a lot 38 | of his ideas. Later in his life Wiener wrote a fiction book that 39 | depicted a math professor that made his name by stealing ideas 40 | from his younger genius students, which does seem to correlate 41 | with his personal experience with Courant. This is of course, his 42 | side of the whole story =P. 43 | 44 | # Top Down vs Bottom Up 45 | 46 | When the book reaches the time period of 1960 it starts talking about 47 | McCarthy/Lisp and some other interesting stuff that was happening around 48 | this time. One interesting thing was an analysis on different ways to 49 | design systems, the classic top-down vs bottom-up. 50 | 51 | There is usually no right approach, since this is also related to how 52 | people usually think about things. But the book gives a hint on why 53 | each approach may be better suited for different kinds of problems. 54 | 55 | When you have a problem more well understood, where you have a clear 56 | vision of the whole structure, you can be top/down in a way that resembles 57 | more a classic/hierarchical manager, you can delegate different parts of 58 | the overall task to sub-modules easily because you have a very good view 59 | of the whole problem and the structure of the solution. 60 | 61 | For problems that are less understood, or that you have less understanding 62 | about the solution structure, bottom-up will be beneficial, since with 63 | bottom-up you focus on defining useful/small core blocks that you are 64 | not sure yet how you are going to compose to solve the final problem. 65 | The small blocks + composition allows you to explore more freely and 66 | iterate faster. 67 | 68 | On top of this overall idea, which I found intriguing/interesting, it was 69 | also defended that Lisp was well suited for bottom-up design, it is fairly 70 | easily to start with small functions and compose them into larger ones 71 | incrementally, not all languages (if any) around 1960 would allow you 72 | to approach problems that way. 73 | -------------------------------------------------------------------------------- /posts.md: -------------------------------------------------------------------------------- 1 | # Blog Posts 2 | 3 | ## Systems Design 4 | 5 | * [Taste for makers](https://www.paulgraham.com/taste.html) 6 | * [Systems design explains the world](https://apenwarr.ca/log/20201227) 7 | * [Absolute scale corrupts absolutely](https://apenwarr.ca/log/20190819) 8 | * [The Tyranny of Structurelessness](https://www.jofreeman.com/joreen/tyranny.htm) 9 | * [Classic SOA](https://dannorth.net/classic-soa/) 10 | * [Cognitive load is what matters](https://minds.md/zakirullin/cognitive) 11 | * [Why programming is hard](https://joearms.github.io/#2014-02-07%20Why%20programming%20is%20difficult) 12 | 13 | ## Testing 14 | 15 | * [Programmer Test Principles](https://medium.com/@kentbeck_7670/programmer-test-principles-d01c064d7934) 16 | * [When TDD Doesn’t Matter: What Do You Value?](https://medium.com/pragmatic-programmers/when-tdd-doesnt-matter-what-do-you-value-91c628dc4488) 17 | * [Testing on the Toilet: Change-Detector Tests Considered Harmful](https://testing.googleblog.com/2015/01/testing-on-toilet-change-detector-tests.html) 18 | 19 | ## Databases 20 | 21 | * [An unlikely database migration](https://tailscale.com/blog/an-unlikely-database-migration/) 22 | * [A Database for 2022](https://tailscale.com/blog/database-for-2022/) 23 | * [What ORMs have taught me: just learn SQL](https://wozniak.ca/blog/2014/08/03/1/) 24 | * [YOUR DATABASE SKILLS ARE NOT 'GOOD TO HAVE'](https://renegadeotter.com/2023/11/12/your-database-skills-are-not-good-to-have.html) 25 | 26 | ## Modularization 27 | 28 | * [Modules, monoliths, and microservices](https://tailscale.com/blog/modules-monoliths-and-microservices/) 29 | 30 | ## Logging 31 | 32 | * [The log/event processing pipeline you can't have](https://apenwarr.ca/log/20190216) 33 | * [Logging: Rules of thumb](https://engineering.hellofresh.com/logging-rules-of-thumb-f6c0f71a2351) 34 | 35 | ## Communication/Management/Process 36 | 37 | * [Makers Schedule, Managers Schedule](http://paulgraham.com/makersschedule.html) 38 | * [The 37signals Guide to Internal Communication](https://37signals.com/how-we-communicate/) 39 | * [An epic treatise on scheduling, bug tracking, and triage](https://apenwarr.ca/log/20171213) 40 | 41 | ## Hiring 42 | 43 | * [Oxide Public Job Descriptions](https://rfd.shared.oxide.computer/rfd/0146) 44 | 45 | ## Politically Correct 46 | 47 | * [What you cant say](http://paulgraham.com/say.html) 48 | * [How to Think for Yourself](http://paulgraham.com/think.html) 49 | * [Heresy](http://paulgraham.com/heresy.html) 50 | * [The Two Kinds of Moderate](http://paulgraham.com/mod.html) 51 | 52 | ## Serialization 53 | 54 | ### Protobuf 55 | 56 | * [Protobuffers are wrong](https://reasonablypolymorphic.com/blog/protos-are-wrong/index.html) 57 | 58 | ## Frameworks 59 | 60 | * [A Framework Author's Case Against Frameworks](https://www.youtube.com/watch?v=k7n2xnOiWI8) 61 | * [Programming with hand tools](https://www.youtube.com/watch?v=ShEez0JkOFw) 62 | 63 | ## Go 64 | 65 | * [What is in a name](https://go.dev/talks/2014/names.slide#1) 66 | * [The Best Go framework: no framework?](https://threedots.tech/post/best-go-framework) 67 | * [Three fallacies of dependencies](https://www.youtube.com/watch?v=yi5A3cK1LNA) 68 | * [The complete guide to Go net/http timeouts](https://blog.cloudflare.com/the-complete-guide-to-golang-net-http-timeouts/) 69 | * [More powerful Go execution traces](https://go.dev/blog/execution-traces-2024) 70 | 71 | ### Organizing Code/Projects 72 | 73 | * [Organizing Go Code](https://go.dev/blog/organizing-go-code) 74 | * [The Within Go Repo Layout](https://xeiaso.net/blog/within-go-repo-layout-2020-09-07) 75 | * [The one-and-only, must-have, eternal Go project layout](https://appliedgo.com/blog/go-project-layout) 76 | 77 | ### Testing 78 | 79 | * [testscript, a hidden gem the Go team kept locked away](https://encore.dev/blog/testscript-hidden-testing-gem) 80 | 81 | ### Error Handling 82 | 83 | * [Error Handling in Upspin](https://commandcenter.blogspot.com/2017/12/error-handling-in-upspin.html) 84 | 85 | ## Culture 86 | 87 | * [The magic of software](https://moxie.org/2024/09/23/a-good-engineer.html) 88 | * [Knuth on Email](https://www-cs-faculty.stanford.edu/~knuth/email.html) 89 | 90 | ## Sec 91 | 92 | * [RBAC like it was meant to be](https://tailscale.com/blog/rbac-like-it-was-meant-to-be) 93 | -------------------------------------------------------------------------------- /notes/mk.md: -------------------------------------------------------------------------------- 1 | # Mk a successor to Make 2 | 3 | This notes are based on my initial experience with mk and the article 4 | * [Mk: a successor to make](http://doc.cat-v.org/bell_labs/mk/mk.pdf). 5 | 6 | Make has changed a lot since the paper has been writen, even tough almost 7 | all the upsides of using Mk over Make are still valid. Now I will briefly 8 | ellaborate on a small list of the ones that got my attention and motivated 9 | my move towards using mk as my main automation tool. 10 | 11 | One main advantage of Mk is that its syntax is almost identical to Make, so 12 | migrating to Mk is considerably easy, and with time you can reap other benefits. 13 | 14 | If you want to try it on linux all you have to do is install 15 | [plan9port](https://9fans.github.io/plan9port/). 16 | 17 | ## smaller and simpler 18 | 19 | Make...specially GNU Make is MUCH bigger than mk. 20 | This can only reflect on more possible bugs and features that I don't need. 21 | 22 | Check [Make](https://www.gnu.org/software/make/manual/make.html) docs. It is well 23 | documented but it is way too big for something as simple as basic automation 24 | (the code can be seen [here](http://git.savannah.gnu.org/cgit/make.git/tree/) ). 25 | 26 | ## rc shell 27 | 28 | The first one is the fact that the shell used on mk is 29 | [rc](http://doc.cat-v.org/plan_9/4th_edition/papers/rc) 30 | which at least for me is much better than Make shell. 31 | 32 | Why rc is better would be a subject of a long brainstorming, suffice 33 | to say that is smaller/simpler than bash like shells. Simplicity comes on 34 | a lot of small good choices, like the exit status code of the previous command 35 | being stored at a variable named **status** instead of some gibberish 36 | on shell that I can't even rementer. 37 | 38 | ## shell evaluation 39 | 40 | The recipe of the mk target (which is a shell script basically) is evaluated 41 | as a whole script, instead of how it is done on make where each line of the 42 | recipe is evaluated as a separated script, resulting in stuff like that: 43 | 44 | ```make 45 | sometarget: 46 | a=command1 && \ 47 | b=command2 $a && \ 48 | command3 $b 49 | ``` 50 | 51 | While on mk, since the whole recipe is just one script, 52 | you can just run: 53 | 54 | ```make 55 | sometarget: 56 | a=command1 57 | b=command2 $a 58 | command3 $b 59 | ``` 60 | 61 | ## targets with attributes 62 | 63 | The way Mk allows targets to have attributes seems better than how Make handles 64 | specific targets configuration. An example is when a target is virtual, meaning that 65 | it is not involved on the generation of any file on the filesystem. On Make you need to 66 | do this: 67 | 68 | ```make 69 | .PHONY: clean 70 | clean: 71 | echo etc 72 | ``` 73 | 74 | As the list of targets of a Makefile grows bigger it will be harder to check 75 | which targets are virtual and which are not. There is a lack of cohesion on the 76 | definition of the target, all its properties are not defined within the target, 77 | some stuff is defined elsewhere. 78 | 79 | Since mk supports attributes on the targets, the example above can be written in Mk as: 80 | 81 | ```make 82 | clean:V: 83 | echo etc 84 | ``` 85 | 86 | This way the target declaration is self contained, self explanatory...cohese. And there is 87 | a lot of other interesting attributes for targets. 88 | 89 | ## better builtin variables 90 | 91 | There is a lot of variables that both Make and Mk generates automatically for you 92 | so they can be accessed during the execution of the recipe. Things like the 93 | names of pre-requisites, or the name of the target that you are building. 94 | 95 | A good example is when you have a metarule, like: 96 | 97 | ```make 98 | %.c: 99 | echo $@.c 100 | ``` 101 | 102 | This is a Makefile that prints the name of all the targets that match the pattern "%.c". 103 | All Make variables are cryptic, you can check it [here](https://www.gnu.org/software/make/manual/html_node/Automatic-Variables.html). 104 | 105 | How about Mk ? 106 | 107 | ```make 108 | %.c: 109 | echo $stem.c 110 | ``` 111 | 112 | Much better =). And other variables are like "newprereq", "prereq", instead of a lot of single 113 | letter garbage. 114 | 115 | -------------------------------------------------------------------------------- /notes/testing.md: -------------------------------------------------------------------------------- 1 | # Testing 2 | 3 | Interest material on the topic of testing. 4 | 5 | ## [The History of JUnit and the Future of Testing with Kent Beck](https://www.se-radio.net/2010/09/episode-167-the-history-of-junit-and-the-future-of-testing-with-kent-beck/) 6 | 7 | Demistifies a lot of dogma surrounding TDD, like you have to do it all the time, 8 | the only way to get a good design is with testing (not true), etc. 9 | 10 | Also debunks TDD as something different from BDD, as usual just a misunderstanding 11 | and people getting to hung up on classification. Good stuff. It has very little to 12 | do with actual JUnit, which is good =P. 13 | 14 | ## [Programmer Test Principles](https://medium.com/@kentbeck_7670/programmer-test-principles-d01c064d7934) 15 | 16 | One of the best/short materials available to showcase how there is no right answers 17 | on testing, just tradeoffs, and you should not marry with any particular 18 | approaches to how test. Also elucidates a lot on the perils of over-mocking 19 | and coupling too much into structure. 20 | 21 | ## Mocking 22 | 23 | I usually don't like mocking, specially general purpose mocking interfaces since they are quite 24 | clumsy to use IMHO and the error messages are usually cryptical since the test fails inside the 25 | framework. For simple scenarios it may be OK, but it doesn't scale well (complexity + error messages). 26 | 27 | And I'm not the only one :-) 28 | 29 | ### Tailscale/Apenwarr 30 | 31 | You can see [here](https://github.com/tailscale/tailscale/blob/28ee355c56c68a446ecfdc755c67311349bf6a3c/ipn/ipnlocal/state_test.go#L278) he said: 32 | 33 | > [apenwarr] Normally I'm not a fan of "mock" style tests, but the precise 34 | > sequence of this state machine is so important for writing our multiple 35 | > frontends, that it's worth validating it all in one place 36 | 37 | It is a cool example of someone else that has a similar distaste for mocks, but 38 | using them when they are really necessary (last case). 39 | 40 | And when he had to...he built his own...no generic/code generated one with a framework. 41 | More details [here](https://github.com/tailscale/tailscale/blob/28ee355c56c68a446ecfdc755c67311349bf6a3c/ipn/ipnlocal/state_test.go#L94). 42 | 43 | More examples of mocks on the tailscale repo (hand written): 44 | 45 | * [magicsock](https://github.com/tailscale/tailscale/blob/28ee355c56c68a446ecfdc755c67311349bf6a3c/wgengine/magicsock/magicsock_test.go#L2672) 46 | * [awsstore](https://github.com/tailscale/tailscale/blob/28ee355c56c68a446ecfdc755c67311349bf6a3c/ipn/store/awsstore/store_aws_test.go#L20) 47 | 48 | At least when grepping for "mock" it doesn't seem the repo has too much of them..just a few, that is nice. Probably hard to 49 | achieve such a design, but certainly desireable. And this is software that does a lot of networking, so quite challenging to 50 | test in a way that doesn't need a lot of mocking/faking. 51 | 52 | ### Kent Beck 53 | 54 | From [Programmer Test Principles](https://medium.com/@kentbeck_7670/programmer-test-principles-d01c064d7934): 55 | 56 | > Structure-invariant tests requires a particular style of programming and design as 57 | > well as a particular style of design. I frequently see tests that assert, 58 | > “Assert that this object sends this message to that object with these parameters and 59 | > then sends this other message to that other object.” 60 | > An assertion like this is basically the world’s clumsiest programming language syntax. 61 | > If I care about the order of operations, I’ve designed the system wrong. 62 | 63 | It captures most of the bad feelings I have with mocks, the syntax is indeed very clumsy and focus 64 | on the order of operations is also an unnecessary form of coupling. Of course sometimes the ordering 65 | is fundamental..then go ahead and validate ordering...but general purpose mocking always seem to 66 | validate ordering, specially because you need to return different values depending on the order etc. 67 | When I use fakes I never have this problem, since it is not a general purpose programable mock, 68 | and when I need some features more close to a mock, like inspecting N calls, it is not embedded 69 | on the mock the ability to validate the calls or fail the test etc. 70 | 71 | Anyway, not sure Im right about this, just seems like I'm not the only one. 72 | -------------------------------------------------------------------------------- /notes/books/hackers-and-painters.md: -------------------------------------------------------------------------------- 1 | # Hackers and Painters 2 | 3 | This books is pure gold, and is a collection of essays, one independent from the other. 4 | 5 | ## Why nerds are unpopular 6 | 7 | High school is basically a place to put children while the parents are at work. On our era 8 | you need a lot of time and education to be productive, since kids cant help with anything 9 | economic, they must be placed on some place while the parents are at work. 10 | 11 | The school has no practical value, you study a lot of stuff, but nothing is applicable on the 12 | moment. Also there is no adult mentor guiding the kids on practical stuff, giving them guidance and 13 | hierarchy. Since humans always build hierarchical relationships it is up to the kids to build them, 14 | since kids are essentially stupid they will build hierarchy based on stupid stuff, like appearance or 15 | being good at sports. 16 | 17 | It is not very different from a prison, the adults there wont allow the kids to kill each other, but 18 | everything besides killing is permitted, as bullying of all kinds. Bullying happens because kids 19 | that are not on the top of the hierarchy are confused about their place, and need to humiliate someone 20 | to be able to get up on the hierarchy (at least on their minds). 21 | 22 | Nerds are not popular because they dont want to. Of course they dont like being humiliated, and they 23 | want to be popular...but they want to be smart more than being popular, so they end up studying cool 24 | stuff instead of investing on the things that would make them popular. 25 | 26 | 27 | ## Hackers and Painters 28 | 29 | This is a chapter that I identified myself a lot, there was a lot of talk 30 | about how computer science is a 31 | bad name because it forces hackers to think they must be scientists. 32 | 33 | Why would science be bad ? It is not bad, but it implies research. 34 | Research must be novel. But beautiful code 35 | does not need to be novel, it can be a clever hack on something that 36 | already exists or a merge of existing concepts. 37 | 38 | Usually this would not be admissible as science research, 39 | but it is serious hacking and can be pretty useful to people. 40 | 41 | This is not mentioned on the book, but Golang seems to be a good example of 42 | cool hacking. There is no novel concept 43 | on the language, it is just a blend of old concepts, implemented with 44 | simplicity in mind, beautiful simple code. 45 | It would never be admissible as research, but it is very 46 | useful and a well crafted tool for hackers. 47 | 48 | Also this science thing makes you think you should know more theory, 49 | and this bites me deeply, because I also always felt that. 50 | 51 | You feel dumb by not remembering all that glorious theory that you 52 | studied on colleague. He uses a metaphor that hackers must know theory just 53 | as much painters need to know paint chemistry (you can't be ignorant 54 | about it but you don't need to master it either). 55 | 56 | He talks about how languages are a way to express itself, 57 | to start filling the blank canvas, this makes sense, since 58 | the term "language" makes it pretty clear that you are trying 59 | to express something, to convey an idea. 60 | 61 | Another cool concept described is the idea of programming as sketching, 62 | I identify myself a lot with that. You learn by doing, 63 | and you do a lot of sketches, throwing stuff away, before you get something 64 | good out of it. Again this is very similar to the work of a painter, 65 | or an architect drawing a project, it will be similar to almost 66 | all human activities that involves a creative process. 67 | 68 | 69 | ## Mind the Gap 70 | 71 | Misconceptions about wealth: 72 | 73 | * Daddy Model 74 | * Wealth is a pie 75 | * People that have wealth are bad 76 | 77 | Wealth is not a pie, it can be created. 78 | Innovation depends on people capacity to create wealth. 79 | 80 | 81 | ## Cool Concepts 82 | 83 | * Freedom to question everything is required for innovation 84 | * Civil liberties and wealth of a country go hand by hand 85 | * Pick your battles 86 | * Having a discussion with an idiot makes you an idiot 87 | * Startup must be user oriented, not tech guys wanting to solve hard problems 88 | * When on war with a great army, go for the mountains where they cant follow you 89 | * The language you choose matter 90 | * Abstractions should let you go faster, not tell you what to do 91 | -------------------------------------------------------------------------------- /notes/books/i-lov-logs.md: -------------------------------------------------------------------------------- 1 | # I lov logs 2 | 3 | 4 | ## Introduction 5 | 6 | Logs are presented as a data structure, an array of records, ordered by a timestamp. 7 | The timestamp (and a way to provide them) is a important concept, specially on distributed system. 8 | 9 | You can model two ways to replicate state using the idea of logs. Master / Slave Backup 10 | and State Machine Replication. 11 | 12 | 13 | ## Master / Slave Backup 14 | 15 | On this case you keep a log of the state changes, and you are able to reproduce state from the 16 | master to the slaves. 17 | 18 | The master receives read/writes and generate the logs that are consumed by the slaves. 19 | Slaves can assume at anytime, thanks to the state upgrades provided by the log. 20 | 21 | Example: 22 | 23 | x = 1 24 | y = 3 25 | x = 7 26 | 27 | 28 | ## State machine replication 29 | 30 | Here you keep a log of all the events that happened on the system. 31 | Given a deterministic state machine, if you provide the same input to it, it will 32 | achieve the same state, independent of how much times you execute it. 33 | 34 | If you have a DSM, them you can use the log of events to reproduce the DSM and achieve 35 | the same state. 36 | 37 | On this model the state machines are peers, there is no master / slave. 38 | 39 | Example: 40 | 41 | x = 0 42 | y = 0 43 | x += 2 44 | y *= 4 45 | x = x + y 46 | 47 | It would be analog as a bank log of operations (debits and credits) that are used to 48 | calculate the current state of your account 49 | 50 | 51 | ## Organizational Scalability 52 | 53 | Any team can publish on the feed and be responsible for the published data. 54 | Avoid a central ETL team, they will become a bottleneck and wont be able to scale. 55 | 56 | These ideas is analogous to the devops culture of each team owning its own service + deployment, 57 | without a central team responsible for a part of the system that is essential to features rolling out 58 | to the client. 59 | 60 | On the devops side, this would be operational stuff related to deploying stuff. On data integration, would be 61 | the idea of publishing data. Even if you have a team responsible for aggregation and cleaning and doing awesome 62 | stuff with data, it should not be **mandatory** that any data available on the system must pass through them. 63 | 64 | The event feed is the central API for publishing and retrieving data, but there is no central team. 65 | Defining and respecting protocols is important, and seems that [Postel Law](https://en.wikipedia.org/wiki/Robustness_principle) 66 | is specially useful here, since it will make integration and evolution of the protocol a lot easier. 67 | 68 | This central log is very simple (assume as less structure as possible), and supports real time + batch processing gracefully, 69 | not assuming anything on how the data is used. This reminds greatly of the [data lake](http://martinfowler.com/bliki/DataLake.html) 70 | concept, opposing to the traditional data warehouse. On this case, the basic structure could be a JSON, but there is no 71 | global assumption about the structure, the contract is between each consumer <-> producer. 72 | 73 | 74 | ### Scaling the log 75 | 76 | If everyone is going to integrate through the log then this log has to scale horizontally and have decent 77 | consistency (eventual, but decent :-). 78 | 79 | Cool stuff: 80 | 81 | * Sharding 82 | * Replication 83 | * Key partitioning 84 | 85 | Sharding and replication is well know, key partitioning is the idea to replicate and shard keys independently, 86 | this way keys are completely orthogonal and you can add new keys indefinitely without interfering on existing 87 | keys (you never get full orthogonality, but it is a good goal). 88 | 89 | 90 | ## Good properties of a event log 91 | 92 | * Favor N consumers (and scales to it) 93 | * Persists the events 94 | * Enable reprocessing of past events (important to failure modes and bug fixes) 95 | * Lifetime of events can be configured in time or space 96 | 97 | 98 | ### Technologies 99 | 100 | A list of technologies that provides these nice properties :-) 101 | 102 | * Kafka 103 | 104 | 105 | ## Consensus Algorithms 106 | 107 | * Raft 108 | * ZAP 109 | * Viewstamped replication 110 | 111 | 112 | ## Ideas 113 | 114 | ### Rastreability 115 | 116 | Besides correlation ids we could use something like [lamport timestamps](https://en.wikipedia.org/wiki/Lamport_timestamps) 117 | so we could achieve correlation and partial ordering of events :-) 118 | -------------------------------------------------------------------------------- /notes/books/the-making-of-prince-of-persia.md: -------------------------------------------------------------------------------- 1 | # The Making Of Prince Of Persia 2 | 3 | The [book](https://www.amazon.com/Making-Prince-Persia-Journals-1985-1993-Illustrated-dp-0578627310/dp/0578627310/) 4 | is written as a collection of diary entries, which makes it a very interesting 5 | reading, getting into the head of the author. 6 | 7 | ## Predicting the Future 8 | 9 | One of the first things that caught my attention is again a very bizarre/wrong 10 | prediction of the future related to the gaming industry. On The Masters Of Doom, 11 | Carmack and Romero thought they already lost the gravy train of the gaming 12 | industry, that the chance to do something amazing and make an impact had already 13 | passed. Jordan Mechner also thought that games for PCs were something that 14 | was dying and was not sure if it would be worth it to invest on it. 15 | 16 | Kinda funny how so much people involved on making games thought that the 17 | industry was starting to die, and its golden days have passed. Nothing could be 18 | further from the truth, the gaming industry just got bigger. It changed a lot, 19 | but it is very far away from dying. 20 | 21 | ## Uncertainty 22 | 23 | I really like the idea of embracing uncertainty and doubt, not as some kind of 24 | weakness, but as a way to embrace reality and what you don't know. Most people 25 | that are certain, when observed from the outside, seem to be under some sort 26 | of illusion, at least for me. 27 | 28 | Since I'm strongly biased towards uncertainty it is always cool to observe people 29 | who built great stuff and yet were filled with doubt. There is this illusion that 30 | people who built great stuff are always determined and certain. Jordan certainly 31 | wasn't, along almost an year he was drown in doubt. 32 | 33 | Jordan was not sure either if it was worth it to make a game, 34 | and even if it was, he was very unsure if he was actually able to 35 | make a game again. 36 | 37 | He developed the game [Karateka](https://en.wikipedia.org/wiki/Karateka_(video_game)), 38 | which was a huge success for the time it was developed, selling 500K units. And yet, 39 | he had doubts if he still got it, he felt like his coding skills had atrophied. 40 | Which is something that I always feel, all the time, when I end up working with 41 | different aspects of software development and then have this feeling that I would 42 | not be able to do what I used to anymore. 43 | 44 | Another interesting thing that made myself identify with the author a lot was 45 | a lack of ability into reproducing what he made at Karateka, he didn't remembered 46 | how he made Karateka, all what was involved, and was not sure if he could reproduce 47 | the whole thing again. It feels like things just happen, on the right time, 48 | and magic happens, and later you are not very sure how you were able to do 49 | what you did. 50 | 51 | I feel the exact same thing about a lot of things that I did on the past, 52 | it is very cool to see someone sharing similar feelings, specially someone so 53 | accomplished (making entire games by yourself along years is a big accomplishment 54 | on my book =P). 55 | 56 | ## Attention to Detail 57 | 58 | Even though he had a lot of doubts regarding his own skills, it is obvious that 59 | one of the main things that made him great was attention to detail, 60 | in my opinion it always is an important aspect of greatness, and that 61 | is why it is hard to find people who are great on creative work, because 62 | attention to detail is not something that comes easy on our species. We 63 | evolved with strong biases towards best effort and good enough, which 64 | to achieve mastery is certainly not enough. 65 | 66 | He reviewed some ports of Karateka to other platforms and it lacked so much 67 | in attention to detail that he couldn't even provide clear criticism about it, 68 | it would need to be remade in his opinion. That strengthened his confidence 69 | on him being able to make games, because at the same time that he didn't 70 | think he was very good, he was able to see how much lack of attention 71 | to detail other people had on their games, so maybe he was not that bad =P. 72 | 73 | Also identified myself a lot with this, it is odd how you can really believe 74 | that you are no good, and yet when you see other work you have a lot of 75 | criticism and then you realize maybe you are not that bad. 76 | 77 | Maybe a mind with attention to detail + lots of critical thinking is 78 | essential for good design, but that also makes for a mind that will always 79 | be very self critical and disappointed with itself, filled with doubts, and 80 | maybe that is OK. 81 | -------------------------------------------------------------------------------- /notes/orm.md: -------------------------------------------------------------------------------- 1 | # ORM (Object Relational Mapping) 2 | 3 | This is something I don't have a whole lot of experience with, but have seen it 4 | go wrong in some contexts. Also from the first time I was introduced to the concept 5 | I was always suspicious about the impedance between objects and databases, 6 | specially given my intuition that objects are about messaging and hiding state and 7 | databases are basically about data (and some data models about the relationship between data). 8 | In time it seems I was not completely alone in sensing the mismatch, although sometimes for different reasons. 9 | 10 | One interesting problem that ORMs introduces is modeling 11 | your data as a second class citizen that follows the design of objects, basically 12 | it raises the question if you should model your data first and then design 13 | the system around that, or design the system (with objects in this case) 14 | and then model the data around that. It even makes me wonder if both should not be 15 | completely decoupled. My experience is that yes. There may be some reasons to design your 16 | logic/software in a particular way that is unrelated to the challenges on modeling data 17 | in a way that is easy to access. Specially because "ease of access" is domain specific, 18 | it means that it is easy to find (and potentially correlate) data in a way that is 19 | useful to clients/analysts/etc. It could be argued that it is possible to decouple both even 20 | with ORMs, but in my experience it rarely is, when ORMs are used the object model of the service 21 | becomes the data model, most of the time in detriment to the search/access experience. 22 | 23 | Interesting enough, most of the people that I know that are more data intensive, 24 | know a lot about SQL and think a lot on terms of database schemas/data modeling, don't like 25 | ORMs. Probably because of that, they see the data design as first class and see access issues 26 | as the main problem they need to solve, they start from there and when you are starting thinking 27 | only about data and how to search/correlate data I don't think objects help, we do need a different model 28 | to deal with this. One example of such model is the relational model. One can think they are related because 29 | objects can "relate" to each other too, but I believe they really don't. Even when objects have a relationship 30 | with other objects, in a well designed system that is usually hidden from anyone interacting with the object. 31 | One example of such principle is the [Law of Demeter](https://en.wikipedia.org/wiki/Law_of_Demeter). 32 | 33 | My point here is that your mindset when you are designing objects is quite different (and to solve different problems) 34 | than when you are designing data to be accessed. I'm talking a lot about access because the whole point (and the hard problem to solve) 35 | of data is how to access it efficiently, in useful ways...and in a way that scales and handles change/evolution well. 36 | The problem is not storage, is the interface to access the stored data and how that evolves along time. So one can say it is all about access/search. 37 | 38 | With objects you are thinking on how to isolate state, and even isolate/hide the relationship of different objects through more high level/useful 39 | interfaces. With data you are thinking about every single piece of data and all the relationships between them, how to evolve that, there is 40 | nothing to be hidden here, you are working to make things accessible and know. For example there is no such thing as private data in 41 | a database because that is mostly useless (in the sense that it can't be accessed by anyone, this is different from access control to specific 42 | users etc, which of course it is useful), you are making data/state explicit and for the purpose of being accessed/manipulated. 43 | 44 | Another problem with ORMs is the assumption that the ultimate way to design software is with objects. 45 | What about functional programming then ? Or even other paradigms to design software. Just as it happens with design 46 | patterns, ORMs give way too much credit to objects. 47 | 48 | Some good material on the topic, usually more on the "why not" realm because 49 | there is already too much material on "lets do it", usually using some logic like 50 | "SQL much hard", "Deliver ASAP", etc: 51 | 52 | * [The Vietnam of Computer Science](http://blogs.tedneward.com/post/the-vietnam-of-computer-science/) 53 | * [Object-Relational Mapping is the Vietnam of Computer Science](https://blog.codinghorror.com/object-relational-mapping-is-the-vietnam-of-computer-science/) 54 | * [Exiting the Vietnam of Programming: Our Journey in Dropping the ORM (in Golang)](https://alanilling.com/exiting-the-vietnam-of-programming-our-journey-in-dropping-the-orm-in-golang-3ce7dff24a0f) 55 | * [What ORMs have taught me: just learn SQL](https://wozniak.ca/blog/2014/08/03/1/) 56 | -------------------------------------------------------------------------------- /notes/articles/computer-as-communication-device.md: -------------------------------------------------------------------------------- 1 | # The Computer as a Communication Device 2 | 3 | Notes from the article [The Computer as a Communication Device](https://internetat50.com/references/Licklider_Taylor_The-Computer-As-A-Communications-Device.pdf). 4 | 5 | The amount of precision that Lick envisioned the future is astonishing, 6 | specially given that this was written on 1968. It comes at no surprise 7 | that most people ignored him as a dreamer, he was way far ahead his 8 | own time. 9 | 10 | ## OLIVER 11 | 12 | When describing what an on-line personal assistant would look like, he wrote: 13 | 14 | ``` 15 | A very important part of each man’s interaction with his on-line community will 16 | be mediated by his OLIVER. The acronym OLIVER honors Oliver 17 | Selfridge, originator of the concept. An OLIVER is, or will be when there is 18 | one, an “on-line interactive vicarious expediter and responder,” a complex 19 | of computer programs and data that resides within the network and acts 20 | on behalf of its principal, taking care of many minor matters that do not 21 | require his personal attention and buffering him from the demanding world. 22 | “You are describing a secretary,” you will say. But no! Secretaries will have 23 | OLIVERS. 24 | 25 | At your command, your OLIVER will take notes (or refrain from taking 26 | notes) on what you do, what you read, what you buy and where you buy it. 27 | It will know who your friends are, your mere acquaintances. It will know your 28 | value structure, who is prestigious in your eyes, for whom you will do what 29 | with what priority, and who can have access to which of your personal files. 30 | It will know your organization’s rules pertaining to proprietary information 31 | and the government’s rules relating to security classification. 32 | ``` 33 | 34 | This is essentially what we have on the internet now (2022) with social network 35 | services, web searching, email, shopping, etc. Everything is stored and analyzed. 36 | The main caveat being that the personal assistant doesn't belong to you, it is not 37 | even explicit, it is hidden behind these services, and instead of it working 38 | to your benefit it actually works for the benefit of whatever corporation 39 | that owns your data and provides the service in a way that you have no control 40 | over how the data is used and to who it is sold. 41 | 42 | It seems to me that we need a new revolution, like the personal computer revolution, 43 | that gives back power to people but this time around data and AI. The emergence 44 | of these sophisticated and wildly useful systems is obvious now (and was clear 45 | to Lick more than 50 years ago) but it is happening the same way computing started, 46 | it is 100% on the hands of a few gigantic corporations who have already proven 47 | to be shady on how they use the data. 48 | 49 | We are at fault with this. For economic reasons, just as it happens 50 | with the cloud, we give more and more power to these few companies. We just bought 51 | the whole propaganda on how useless/dumb it is to have your own infrastructure, 52 | to write your own software, and by doing that we are selling ourselves more and 53 | more to them, for the sake of being more "productive" and "cheap", or to have 54 | access to services for "free" (there is no such thing). 55 | 56 | The idea is not that corporations are bad and things should be owned by the 57 | government, that may be even worse. I think it is more about how the relationship 58 | with companies exist today, the balance of power. We need to think on ways on 59 | how we can keep our data and doing computation on the edge, where the data 60 | can be safely used privately (not an easy problem to solve since some computations 61 | can be quite expensive, so some sort of hybrid model may be required). 62 | This is not such a pipe dream, Brendan Eich the creator 63 | of Javascript is trying something like this with the [Brave Browser](https://brave.com/). 64 | 65 | Not sure it is the best way to approach it, but we need more of that, computing 66 | solutions that leverage the computational power the individual already have 67 | (edge computing) in a way that the data always belongs to the individual only. 68 | You can even store the data on the network, 69 | as far as it is in a way that only the end user can access it. 70 | 71 | The vision of what human augmentation could look like persists and it is beautiful, 72 | but it is hard to shake the feeling that maybe we have lost track of it along 73 | the way. This also maps well with why Alan Kay being frustrated with 74 | [mobile computing](https://www.fastcompany.com/40435064/what-alan-kay-thinks-about-the-iphone-and-technology-now). 75 | We are using computational power mostly as a interactive TV, which can be 76 | fun sometimes but it is the most brain dead way to do it, and the design 77 | of the devices maps to that, not being focused on creative tasks and at 78 | the same time we are not caring who has control of all the data and computing power. 79 | -------------------------------------------------------------------------------- /notes/podcasts/neocities.md: -------------------------------------------------------------------------------- 1 | # Bringing GeoCities Back with Kyle Drake 2 | 3 | Some notes from the really interesting podcast 4 | [Bringing GeoCities Back with Kyle Drake](https://www.softwaresessions.com/episodes/bringing-geocities-back-with-kyle-drake/). 5 | 6 | The core idea around Neocities is to bring back some of the magic of Geocities. 7 | Currently we have a lot of social platforms but they are all very constrained 8 | on the format that you can interact on them, small text box on twitter, short 9 | videos on tik tok, etc. He wanted to provide an easy way for people to have 10 | a blank canvas and be creative about it, like in the early 90's. 11 | 12 | Maybe it is just old people being nostalgic =P, or maybe there is something about the idea that is cool, 13 | but the whole interview is very interesting, specially the challenges on building Neocities 14 | and the implications that has to decentralized web and autonomy. 15 | 16 | He talks at length about how bad it is that these days you need to be a true 17 | believer in order to work into something. He likes the idea of a more 18 | decentralized web and yet he is very skeptical about a lot of tech 19 | that tries to achieve that, and even tech he is working on. And people seem 20 | surprised, like how can you work on something that you don't believe it 21 | is going to work 100%, and that is actually healthy, you are rarely a 100% 22 | sure of anything and critical thinking is essential, even with ideas you 23 | like, and for some reason we are losing that, you need to be a "believer" 24 | and an evangelist to be respected. 25 | 26 | Then he moves on to some very interesting technical details, like how he ended 27 | up having to deal with BGP and implement his own Anycast infrastructure. Usually 28 | people would react, why do that ? just use CloudFlare, Fastly, etc. And you usually 29 | would be correct and it is what he tried to do, he was using digital ocean to 30 | run Neocities and had some very serious issues. 31 | 32 | What happened is that the NRA got pissed off with the content of one of the 33 | pages he was hosting and launched an DMCA against the page, in this case the 34 | ISP would be Digital Ocean and when they received it they did the safest thing 35 | possible, they just brought down all Neocities. In the end the DMCA claim was 36 | not even valid, and yet he was completely powerless to stop his whole service 37 | going down. 38 | 39 | That was the start of his quest on achieving independence. He became his own 40 | ISP and then had to pay for traffic for a few internet exchanges. This allowed 41 | him to be the one notified when any legal issue happened and deal with it 42 | appropriately, instead of relying on what a big cloud vendor would do (which 43 | is almost always bringing you system down, because that is safer). 44 | 45 | That made me think a lot about true decentralization on the web, and how it will 46 | involve having ownership of much more infrastructure than people are used to, or 47 | else it is fake decentralization/autonomy. Basically if 3 cloud vendors can shut your 48 | whole system down you are not even close of being decentralized, doesn't matter 49 | how much cool voting mechanisms you have or amount of blockchains. In the end 50 | independence, autonomy and decentralization are spectrum's, there is no binary 51 | being centralized vs decentralized, or having no control vs having all control. 52 | 53 | Another interesting piece of information is the cost. Cloud vendors charge an 54 | obscene amount of money for traffic and it is con of the century that they 55 | convinced the whole industry that how much they charge for it is OK and makes 56 | sense. His whole setup for being his own ISP + exchange/traffic costs for 30TB 57 | of data is around 130 dollars per month. The same setup on AWS would cost 58 | around 5000 dollars. It doesn't matter how much extra costs AWS have, it is 59 | hard to justify an almost 50x increase on price since the other company also 60 | maintains internet links and does the routing with reasonable high availability. 61 | 62 | It makes you think a lot on how keeping things simples could be a differential in 63 | a product these days, you could do the same others are doing but orders of magnitude 64 | cheaper if you can be smart about it. Also on how we keep falling for these 65 | cloud vendor traps. 66 | 67 | And talking on traps there is some bonus content on how people seem to have forgot 68 | about Microsoft's past and then just keep using visual studio code all happy. 69 | Microsoft doesn't love open source, they didn't bought Github out of love for 70 | open source, and they don't give you vscode out of the good of their hearths. 71 | There is certainly a board meeting happening where they discuss how they 72 | are going to bring and lock in the dev community inside their platforms and 73 | we just keep following their plan for some reason. And that only gets more 74 | insane with the CoPilot thing, not just the dependence on something provided 75 | by them but specially the ethically questionable way that they got the data 76 | and built the thing (they didn't bought Github out of love for open source...). 77 | -------------------------------------------------------------------------------- /notes/articles/intel-clear-containers.md: -------------------------------------------------------------------------------- 1 | # Intel Clear Containers 2 | 3 | This notes are based on the whitepaper Intel® Clear Containers: A breakthrough combination of speed and workload isolation. 4 | 5 | Sadly the white paper right now can't be found on the internet =/, broken links. 6 | 7 | One of the first sentences on the paper already sounds off: 8 | 9 | ``` 10 | Virtualization has enabled the decoupling of software workloads from 11 | the hardware they execute on 12 | ``` 13 | 14 | What ? AFAIK the operational system does this. The operational system 15 | is the virtual machine that languages are targeted at, not some processor 16 | feature. This felt extremelly off if not plain wrong. Not a good start for the paper. 17 | 18 | Before I go on, I'm biased, even before reading the paper I was really sad 19 | with the idea that operational system have made security and isolation so 20 | hard to implement that implementing them on hardware seems to be a good idea. 21 | 22 | To think about using hardware features for virtualization just to promote isolation 23 | between processes seems like a failure of imagination on how to implement these 24 | things better on the operational system level, it is a terrible defeat for a 25 | operational system (contrast the intel clear container ideas with Plan9 and Inferno 26 | for example). 27 | 28 | When you think about security, it seems to me that the only thing that the hardware 29 | needs to do is to provide isolation primitives, between processes. And even that is 30 | only necessary on the hardware level because implementing them on software would 31 | be too slow. So it is an optmization, not the case where conceptually it is safer 32 | on hardware. By the way, this leads to the second point...implementing logic on hardware 33 | is not safer. It seems safer because you can't change it without phisically changing 34 | the hardware, but this is also what makes it unsafe..YOU CANT CHANGE IT (think about 35 | meltdown and specter). Since hardware is HARD (it wont change) it seems that complex 36 | logic should reside on a layer where updates are easy to do, like software. Hardware 37 | should do the minimum possible to enable software to implement isolation properly. 38 | 39 | Another thing that bugs me deeply is that virtualization != isolation. Virtualization implies 40 | isolation, but isolation does not imply virtualization. Actually as far as I know the idea 41 | of hardware virtualization is just to enable different operational systems to think that they 42 | own the hardware when they actually don't. It is the CPU delivering to operational systems 43 | what the operational systems themselvs deliver to applications, the illusion that they own 44 | everything. This has nothing to do with the problem of running just one operational system 45 | and guaranteeing that processes being managed by the operational system are isolated from 46 | each other. So in the end..the whole idea seems pretty stupid and not a breakthrough at all. 47 | It seems interesting just because you need to do some cool hacks to fool virtualization 48 | on thnking that it is loading another kernel when it is not and because cgroups and namespaces 49 | on linux is so fucked up that virtualization enables you to forget how crappy isolation 50 | is on linux. 51 | 52 | A quote from the paper: 53 | 54 | ``` 55 | vms are decidely superior for security 56 | ``` 57 | 58 | Why ? Honestly it may be true... but how we got on a point 59 | where hardware virtualization is the safest choice ? Again it seems like 60 | a defeat of operational systems on implement these things properly. 61 | It seems like isolation and security can be built on top of simple 62 | hardware feature (NOT virtualization), but we failed on doing so, 63 | so now it seems like it is safer to use virtualzation to isolate stuff. 64 | 65 | The descriptions on how the vm starts up fast just mapping the host kernel 66 | as the vm kernel, bypassing memory pages etc, does not make me feel safe, 67 | actually scares the hell out of me. It seems like a set of pretty cool hacks 68 | involving kvm, but I would have 0 confidence on saying it is safe (there is a lot 69 | of jumps to addresses and assumptions and memory maps being disabled...etc). 70 | 71 | The main basis for thinking that this is a good idea is that the hypervisor 72 | is safer than the operational system. For example the paper is always talking 73 | about how when you compromise the operational system you compromise all containers. 74 | But the same applies to hypervisors as well, so it seems only a matter to 75 | decide which one is simpler and safer. Virtualization is never simple to me, isolation 76 | seems like a subset of the virtualzation problem..so I fail to understand why 77 | hypervisors and virtualization are conceptually safer than pure isolation 78 | implemented on an operational system. 79 | 80 | The whole thing seems like Intel trying to couple the idea of isolation 81 | to their hardware features so you will depend even more on their hardware. 82 | It may seem as a good idea just because of a total failure from operational 83 | systems =(. Yet, I'm pretty sure that I'm going to see a lot of usage 84 | of this kind of technology, perhaps that is why I'm writing this...it is like 85 | a pre-rant on the thing =P. -------------------------------------------------------------------------------- /notes/articles/wisckey.md: -------------------------------------------------------------------------------- 1 | # WiscKey: Separating Keys from Values in SSD-conscious Storage 2 | 3 | Article: https://www.usenix.org/system/files/conference/fast16/fast16-papers-lu.pdf 4 | 5 | ## LSM 6 | 7 | Traditional LSM (Log Structured Merge Tree) is optimized towards 8 | sequential disk reads, trading sequential reads for a lot of read/write 9 | amplification. 10 | 11 | This makes sense since sequential reads VS random reads performance 12 | ratio is 1000:1 in spin disks. On SSD random reads can be slower, 13 | but not on the same order of magnitude of spin disks. 14 | 15 | As the article title makes clear, the core idea is to optimize for 16 | SSD, this is the basis of the whole article. 17 | 18 | First, why does LSM have so much amplification ? Basically because the 19 | LSM tree has the keys + values together, this means bigger trees. 20 | 21 | Positive points of using keys + values on LSM tree: 22 | 23 | * Simpler range query, keys are sorted and the values are together on the tree 24 | * Specially on range queries the reads will be sequential (great for spin disks) 25 | 26 | Negative points: 27 | 28 | * More expensive compaction phase 29 | * Reads can be expensive since tree is biggers and worst case traverses entire tree 30 | 31 | The expensive compaction and the bigger tree is the price to pay 32 | for sequential reads. 33 | 34 | Compaction is a process that is part of the LSM algorithm, it is when 35 | one of the levels exceeded the size threshold and must be sort/merged 36 | with the lower level. This process can run recursively until the last 37 | level, and can be pretty expensive since you must move keys + values together. 38 | 39 | Keys usually have a size of 16 bytes, but value size can vary from 40 | 64 bytes to 512 kb. 41 | 42 | ## WiscKey 43 | 44 | The catch of WiscKey comes from the fact that SSD's can achieve 45 | full throughput on random reads if these operations are done 46 | concurrently and with a good enough block size. 47 | 48 | Notice that this is extremely dependent of the SSD, testing if your 49 | structure behaves like this is essential to have the advantages. 50 | 51 | If random reads on your disk cant achieve a high throughput, WiscKey 52 | is a bad idea for you. 53 | 54 | Why ? The idea of the algorithm is to separate the values from the keys. 55 | Basically the LSM tree will consist of tuples on the form: 56 | 57 | ``` 58 | (key, valueoffset) 59 | ``` 60 | 61 | And there will be a value file containing all files (unordered). 62 | The ability to perform multiple concurrent reads on the disk 63 | is necessary since indexed searches and range queries will 64 | return sequential regions of **(key, valueoffset)** and each 65 | **valueoffset** will require a read on a different (non-sequential) 66 | region of the disk. SSD's have no problem doing that, and can 67 | even achieve full disk throughput if you have enough concurrent 68 | units. 69 | 70 | On the tests performed the number of 32 threads where enough to 71 | use all throughput of the disk. 72 | 73 | Since there is an indirection, it seems like it will be slower, but 74 | specially for value sizes greater or equal to 4K WiscKey is much 75 | faster since the indirection overhead is compensated by a MUCH 76 | smaller LSM tree (easier to cache and to compact). 77 | 78 | The place where the values are stored is named vlog. The vlog has 79 | this tuple: 80 | 81 | ``` 82 | (keysize, valuesize, key, value) 83 | ``` 84 | 85 | Why is there key information ? It is used for garbage collection. 86 | The garbage collection follow a head/tail algorithm. Data is always 87 | appended to the head and garbage collection always happens on the 88 | tail (exploring concurrency again). 89 | 90 | Since the vlog has key info it is trivial to consul the LSM tree 91 | to check if a value is garbage or not. 92 | 93 | Because of the vlog there is no need to keep the LSM log file, 94 | all consistency guarantees can be performed with the vlog + LSM tree 95 | (traditional LSM has the key+value on the tree and a LSM log file 96 | where all writes are performed first). 97 | 98 | Inconsistencies are handled bu checking that data is on the vlog 99 | but not on the tree (data is saved first on the vlog). 100 | 101 | The vlog causes WiscKey to have a bigger space amplification, 102 | since more data is duplicated it will use more space than LevelDB 103 | (something like 10-15%). 104 | 105 | Usually in key-value stores there is three problems: 106 | 107 | * space amplification 108 | * write amplification 109 | * read amplification 110 | 111 | You can't optimize for all of them. WiscKey chooses to diminish 112 | read/write amplification, increasing space amplification. 113 | 114 | ## Conclusion 115 | 116 | The results shows WiscKey being 2.4x to 100x faster than LevelDB 117 | LSM. The only case where WiscKey can loose is when the value size 118 | is very small, like 64 bytes, since the benefits of a smaller 119 | LSM tree are less obvious and it has to pay for the indirection. 120 | 121 | Basically the pre requisites are: 122 | 123 | * SSD with good concurrent random read performance 124 | * Value sizes bigger than 4K 125 | 126 | If you fit this case, WiscKey may be a good idea. 127 | -------------------------------------------------------------------------------- /notes/docker-compose.md: -------------------------------------------------------------------------------- 1 | # Docker Compose 2 | 3 | Why a note on docker-compose ? Well I had a very long history 4 | of love/hate with it, actually not love with it, but love with the 5 | idea of easily setup test environments to use on integration 6 | tests. I used it for integrating stuff with RabbitMQ and PostgreSQL 7 | for example, in a reasonably easy manner (at least on v1). 8 | 9 | There is a lot of approaches to run integration tests, but as time 10 | and versions progressed it seems to me that docker-compose only became 11 | a worse and worse option for testing, specially when it is used only 12 | for testing purposes (my case). 13 | 14 | The final blow was [this issue](https://github.com/docker/compose/issues/3492), 15 | when they just broke docker-compose run for the sake of their own 16 | reasons, which are unrelated to testing environments, but related to 17 | their own deployment related features, from 18 | [this comment](https://github.com/docker/compose/issues/3492#issuecomment-230931596): 19 | 20 | ``` 21 | It definitely shouldn't be the default behaviour of run to add the aliases, 22 | because then you'd get situations where one-off containers are accessible via 23 | the same hostname as long-running containers for the same service, 24 | and part of the load-balancing group - which is almost never what you want. 25 | ``` 26 | 27 | Which kinda consolidates how the tool evolved to be a deployment tool. 28 | Maybe that was always the objective, but on v1 it was much smaller and 29 | it felt more like a good fit for reasonably simple testing environments. 30 | 31 | Now looking at the docs for v3 is daunting, both the cli reference as the 32 | spec for the config files, it is riddled with features and a lot of the 33 | features are marked as suitable only for deployment and even relates to 34 | docker swarm. So in the end what felt like an approach for dev envs that 35 | requires spinning up multiple services locally now seems unsuitable for 36 | it given the complexity. Both the spec/docs are bloated and actually 37 | running your tests got harder. 38 | 39 | Yes, it is still possible, but it only got worse for that scenario and 40 | maybe that is not the only way to run integration tests. So I wrote this 41 | as a reminder on why I started steering away from using docker-compose 42 | for dev envs / testing and started looking for alternatives. 43 | 44 | A good example on how "simple" it is, is the [Ultimate Guide to Integration Testing 45 | with Docker Compose](https://medium.com/swlh/the-ultimate-guide-to-integration-testing-with-docker-compose-and-sql-f288f05032c9). 46 | Where this script is described as "simple": 47 | 48 | ``` 49 | # author: https://blog.harrison.dev/2016/06/19/integration-testing-with-docker-compose.html 50 | RED='\033[0;31m' 51 | GREEN='\033[0;32m' 52 | NC='\033[0m' 53 | cleanup () { 54 | docker-compose -p ci kill 55 | docker-compose -p ci rm -f 56 | } 57 | trap 'cleanup ; printf "${RED}Tests Failed For Unexpected Reasons${NC}\n"' HUP INT QUIT PIPE TERM 58 | docker-compose -p ci -f docker-compose.yml -f docker-compose.tests.yml build && docker-compose -p ci -f docker-compose.yml -f docker-compose.tests.yml up -d 59 | if [ $? -ne 0 ] ; then 60 | printf "${RED}Docker Compose Failed${NC}\n" 61 | exit -1 62 | fi 63 | TEST_EXIT_CODE=`docker wait ci_tests_1` 64 | docker logs ci_tests_1 65 | if [ -z ${TEST_EXIT_CODE+x} ] || [ "$TEST_EXIT_CODE" -ne 0 ] ; then 66 | docker logs ci_seed_1 67 | docker logs ci_db_1 68 | docker logs ci_fun_1 69 | printf "${RED}Tests Failed${NC} - Exit Code: $TEST_EXIT_CODE\n" 70 | else 71 | printf "${GREEN}Tests Passed${NC}\n" 72 | fi 73 | cleanup 74 | exit $TEST_EXIT_CODE 75 | ``` 76 | 77 | Not simple IMHO, and it used to be a single command line on docker-compose 78 | v1, which is why there was so much people annoyed on the issue regarding 79 | the behavior change. 80 | 81 | I agree with the author that integration tests are hard, 82 | lets not make them harder. 83 | 84 | # Alternatives 85 | 86 | One tool that is more like docker-compose, but at least is built 87 | on top of Kubernetes, which them would give you advantages on 88 | uniformity if you use Kubernetes in production is 89 | [Skaffold](https://skaffold.dev/). It doesn't seem particularly 90 | lightweight, but it does have the advantage that it is a tool 91 | explicitly for "local development", so hopefully it won't be clobbered 92 | with deployment/prod related features like docker-compose. 93 | 94 | Another interesting idea is to build the test environment programmatically 95 | by spinning up containers and linking them together directly from code, 96 | like the [testcontainer-go](https://github.com/testcontainers/testcontainers-go), 97 | which has the advantage of defining environments and reusing them using 98 | actual code instead of YAML and also enables you to build isolated environments 99 | with proper cleanup per individual test instead of an entire test case. 100 | 101 | The reuse side of it is very interesting, if everything in an organization 102 | works with Go you could provide Go libraries that build test environments, 103 | that is much nicer/easier than avoiding duplication of docker-compose YAML files. 104 | 105 | There is also [Nix](https://nixos.org/). 106 | -------------------------------------------------------------------------------- /notes/articles/inventing-on-principle.md: -------------------------------------------------------------------------------- 1 | # Inventing on Principle 2 | 3 | Notes from the presentation [Inventing on Principle](https://vimeo.com/36579366). 4 | 5 | Not actually an article =). 6 | 7 | This presentation is extremely awesome and inspiring. 8 | 9 | It starts by an explanation of how a principle can guide you career 10 | and then elaborates on the presenter's principle, that creators should have 11 | fast connection with what they are creating. 12 | 13 | What does that mean ? It is about feedback, the feedback of changes 14 | on what is being created should be immediate and as clear as possible. 15 | 16 | Most examples on the presentation are about graphics, in this case it is 17 | really easy to see how you can augment the connection between the creator 18 | and the creation. Basically while you change code you must be able to 19 | see the results IMMEDIATELY (removing the compile and run from the equation). 20 | 21 | There is also a cool example on how to develop a binary search with this 22 | principle, where all iterations of the algorithm is shown in real time 23 | as the code is changed, pretty neat. 24 | 25 | He even shows how to apply that on a game on the time dimension, it is pretty 26 | cool. It has features that helps you map the results in the screen to what 27 | part of the code is responsible to that, and vice-versa (remembers some features 28 | of some browser debuggers and the DOM of pages). 29 | 30 | Fast interactivity is a game changer, it reminds me of a part of the book 31 | Hackers Heroes of The Computer Revolution where it is described the impact 32 | that the PDP-2 computer had on MIT, specially because it had a interactive 33 | nature, instead of the batch process style of the IBM machines. A fast 34 | feedback loop is essential to the creative process. 35 | 36 | Another cool history is the one about the 37 | [first human powered flight](https://en.wikipedia.org/wiki/Human-powered_aircraft), that 38 | was also based on fast feedback, MacCready created a plane that was so simple 39 | and cheap that when it crashed it was really fast to rebuild it and try again, 40 | after setting up this environment the "try -> fail -> try again" loop initiated 41 | and eventually (few months) the first man powered flight was achieved. 42 | 43 | There is a lots of development methodologies that seems related to this idea 44 | of connecting creators with their creation, like: 45 | 46 | * TDD (small steps always running tests, tests are the feedback) 47 | * DevOps (feedback about operational quality of your code) 48 | * MVP (feedback from users) 49 | 50 | DevOps is more recent and comes from the problem that arises when you 51 | disconnect the person that is creating software from how it works in 52 | production, without the feedback the developer has no clear idea on 53 | how to improve this aspect of code. 54 | 55 | The most inspiring part is on the end when he explains that there is 56 | three type of people: 57 | 58 | * Craftsman (focused on being very good on a skill) 59 | * Problem Solver (entrepreneurship) 60 | * Principled 61 | 62 | Well, what would be the difference ? The principled person do not see 63 | the absence of the principle he believes as an opportunity, he sees it 64 | as a moral responsibility for correction. It is not a problem to solve or 65 | something related to business, it is a social issue, you believe that 66 | something is wrong and must be fixed. This thing that is wrong is usually 67 | only seem by you or a minority of people at first. But you believe on it 68 | and fight for it anyway, in a lot of ways it works like a social cause. 69 | 70 | Usually social causes involves organizing people and creating movements 71 | (an example is the one that allowed women to vote), but on the technical area 72 | this can also be done by inventing. 73 | 74 | One example that is elaborated on Larry Tesler that invented most of all 75 | resources of modern text editors that are not modal. The modal interface 76 | (like vim) was terrible for almost all people, and he was worried that 77 | this would keep people away from computers, something that was wrong for 78 | him, people needed to feel comfortable on computers because that would 79 | change the world. 80 | 81 | So in a time that everyone thought that modal text editors where the obvious 82 | way he invented a text editor that have a lot of features that text 83 | editors have today, with no modes. Things we take for granted like control+c 84 | and control+v was invented on this context, on a time that people did not 85 | see that as a problem. 86 | 87 | That is why having a principle is not the same thing on problem solving, 88 | people did not even see it as a problem at the time, yet he worked on that 89 | because he believe that it was wrong to make it so hard for people to 90 | interface with computers (social aspect). 91 | 92 | His principle was "dont trap me in a mode", this guided all his work. 93 | Everything he invented was inspired by this idea of interaction without 94 | modes. 95 | 96 | In the end, there is no problem on being a craftsman or a problem solver, 97 | but it is true that this option of being principled and having a caused 98 | is almost never mentioned on software development as an option to live 99 | your like and guide your career. 100 | 101 | Finding a principle takes time, and involves doing a lot of work, and 102 | a lot of work on different areas. It is a journey of self discovery, 103 | Perhaps someday I will find one =). 104 | -------------------------------------------------------------------------------- /notes/crypto.md: -------------------------------------------------------------------------------- 1 | # Crypto 2 | 3 | I'm not even close to a specialist on any of this, but it is impossible to 4 | not form some opinion since how hyped/saturated the market is on this 5 | topic for a few years. Here I focused specially on the bitcoin model, 6 | including architectural characteristics it exhibits, but there is a lot 7 | of different approaches to digital money, would be hard to write about 8 | all of them. 9 | 10 | ## Private Money 11 | 12 | The idea of de-nationalizing money is intriguing. I just don't think it is 13 | as simple as "no government + private = bliss, fairness, prosperity". It is 14 | very easy to try to solve a problem on a current system without realizing 15 | new problems you will be introducing on your new system. 16 | 17 | On a completely decentralized/private system you could end up with a 18 | cyberpunk neo-feudal system where a few digital oligarchs hold all the 19 | power, even if the underlying tech is decentralized. This is exacerbated 20 | by decentralization/lack of structure. That favors lack of accountability. 21 | 22 | ## Scalability 23 | 24 | This argument is around the idea of having a single public global ledger. 25 | Even for fast operations, having a **SINGLE** replicated data structure 26 | seems damn hard to scale, even if it is append only, but the problem gets 27 | exacerbated considerably if you take into account a proof of work model 28 | as Bitcoin. 29 | 30 | If we think about Bitcoin as a currency, imagine that for each person in 31 | the world buying milk in the morning you trigger: 32 | 33 | * Write operation on a distributed/replicated data structure 34 | * Consensus needs to be achieved on the network 35 | * Proof of work stuff 36 | * Now you can buy milk 37 | 38 | I just have no idea why making every operation in the world global is a good idea. 39 | We should go in a direction where digital can be just as private and 40 | local as actual physical money, a peer to peer operation. And the whole 41 | idea, ironically since it talks a lot about distribution, is based on 42 | global operations on a single ledger. 43 | 44 | I just don't see it ever being the cool global money that maybe would be 45 | groundbreaking/life changing. Maybe it can be other things, like store of 46 | value etc, but not an actual currency to use on a day to day basis (and 47 | no El Salvador using it proves very little to nothing, specially because most 48 | places don't use Bitcoin anyway there). 49 | 50 | ## Tolerance to Partitioning 51 | 52 | If you get enough partitioning on the global network on the single ledger 53 | approach you don't have much options other than: 54 | 55 | * Stop working (lose availability) 56 | * Fork the ledger, with no way to re conciliate later (AFAIK) 57 | 58 | This is a classic problem in distributed systems usually described as the 59 | CAP theorem. Since the ledger uses a quorum you can lose part of the 60 | network and keep working, so it is not like it always ends up with 61 | loss of availability on small partitions. The problem is scenarios like 62 | war that can easily cause a country to be isolated from the global network. 63 | There is no good option for the system to keep working but locally and 64 | then re-join the global network later. 65 | 66 | The Internet was designed to work like this (and it does), it is again ironic 67 | that something that is always talking about decentralization failing at 68 | basic things like this. 69 | 70 | ## Privacy 71 | 72 | There is no real privacy on the idea of a global public ledger. 73 | You may have anonymity, but that is based on: 74 | 75 | * You keep your own wallet (no exchanges) 76 | * You never leak information that can tie you to your public key 77 | 78 | The important thing here is that any tracing or data analysis that can 79 | leak your identity to your PUBLIC key will make ALL your financial 80 | operations public to the whole world. We are not even talking about 81 | leaking/losing your private key, that is not necessary for you to 82 | lose your anonymity. 83 | 84 | What makes this extra dumb is that almost everyone uses exchanges, because 85 | maintaining your own wallet means being your own bank, which means 86 | taking security as seriously as a bank. In a world where people barely knows 87 | how computers and the internet works expecting this from people is ludicrous. 88 | So they will end using an exchange and having just as much (or arguably less) 89 | privacy then classic banking (remembering that this landscape is unregulated, 90 | so chances are you have less rights). 91 | 92 | ## Recentralization 93 | 94 | The Internet was designed from the ground up to be extremely decentralized. And it 95 | still is. It was designed to survive half your country being nuked. Literally. 96 | So whatever makes computing today centralized is 100% unrelated to tech. 97 | 98 | Just that reality makes me wonder how far building "new" decentralized 99 | tech (with a bunch of problems of its own) will really lead to a decentralized 100 | and egalitarian world. IMHO it just won't, because the forces for centralization 101 | are non-tech related. They are mostly social/economical. So power will end up 102 | being concentrated, the main difference being the lack of accountability 103 | of whoever has this power. 104 | 105 | A lot of the complaints on how big tech took over everything is related to 106 | people not wanting to do shit themselves. We just use facebook, we just use 107 | the cloud. We don't build core tech anymore, we are just clients. Given that 108 | this is the normal human behavior I would expect every human dealing with 109 | the "new" blockchain stuff via services/exchanges, because they just don't want 110 | to do shit, and that leaves us pretty much in the same place we are today, but 111 | with more expensive/complex/mostly useless tech (but new players, which may be 112 | just as bad or even worse than the current ones). 113 | -------------------------------------------------------------------------------- /notes/podcasts/onmetal.md: -------------------------------------------------------------------------------- 1 | # On The Metal 2 | 3 | [On The Metal](https://oxide.computer/podcast/) is a podcast from 4 | the Oxide Computer Company. It is the only podcast so far that I know 5 | where each of the interviews is amazing, there is no boring/bad interviews, 6 | they are all filled with great stories and it is specially interesting with 7 | people who has interest on low level stuff (although not entirely focused 8 | on that). Here is some notes on the most interesting interviews. 9 | 10 | # Ron Minnich 11 | 12 | Ron is know to me as a guy who tries to build on top of a lot of 13 | nice ideas from Plan9 and works on the [HarveyOS](https://github.com/Harvey-OS/harvey). 14 | 15 | ## Constraints 16 | 17 | There was a lot of talk on the role of constraints and good design, 18 | there is almost a trend for me on how good design usually emerges 19 | in environments where resources are scarce, it is like the scarcity 20 | pushes you towards ingenuity and simplicity. It really pisses me off 21 | how web and cloud environments usually foster wasteful/bad designs 22 | filled with inefficiency, and this is not about using assembly because 23 | high level languages are slow or anything, overall wasteful bad designs. 24 | 25 | Perhaps for historic reasons the first thing that comes to my mind is 26 | the whole service mesh thing, independent from the benefits obtained it 27 | seems that adding an entire user space layer + doubling the amount of 28 | processes you run on a distributed system does not seem to be the most 29 | efficient way to get resiliency and metrics (and the other benefits). 30 | But I could be wrong, I'm not that knowledgeable with distributed 31 | systems. 32 | 33 | But summarizing, constraints, like scarcity of resources, forces you 34 | to minimalism and true simplicity, because without them you won't 35 | succeed. While unconstrained abundance allows any sort of mistake 36 | to pass with flying colors and to be seem as an good idea. 37 | 38 | ## Control 39 | 40 | I was always very intrigued on why Plan9/Inferno seemed to be ignored 41 | by most people, specially since they were built by the same people 42 | who built Unix, but improved for distributed systems, which is a perfect 43 | fit for the world we live today (Unix is not). 44 | 45 | It seems that one big reason for that happening is human nature, and its 46 | desire for control (which is an illusion IMHO). I was already aware that 47 | one of the reasons neither Plan9 or Inferno were embraced was licensing, 48 | but on this interview Ron talks about the reasoning behind how they 49 | licensed Plan9 and Inferno. 50 | 51 | It was the opinion of a lot of people involved on Unix that it was a failure 52 | because AT&T "lost control" of the project, people just took it and used 53 | how they wanted, changed it, etc. Instead of observing how they were able 54 | to influence/mold modern computing, they just felt as a failure because 55 | they where not "in control". And that is the reason why they did not wanted 56 | to make the "same mistake" with Plan9/Inferno, and that is why both were 57 | never truly adopted, a lot of very good ideas never made into the current 58 | operational systems, just now some of them are making it but very slowly, 59 | all because of this fascination that humans have with illusions such as 60 | control. 61 | 62 | ## Complexity 63 | 64 | The holy grail of computing is to get rid of unnecessary complexity and 65 | only keep inherent complexity, the one that is determined by the problem 66 | being solved. But in the industry it is very common for people to believe 67 | that complexity is endemic, mistaking inherent complexity with unnecessary 68 | complexity. An example of that mentioned on the interview is how firmware 69 | was developed. 70 | 71 | Ron has seen firmware being built with languages even more 72 | high level than C, but for some stupid reason the entire industry was convinced 73 | that firmwares NEED to be programmed in assembly, like it was a necessary 74 | complexity for firmwares, and that for no good reason, just industry/group 75 | thinking and lack of imagination. It is funny because Ron actually saw firmware 76 | being developed on high level languages, so it was not a subjective discussion, 77 | it was possible because it had been done, and yet people insisted with no 78 | good reason that it was not. 79 | 80 | Perhaps the whole service mesh thing is the same instance of this issue, 81 | adopting unnecessary complexity as necessary and dying holding it on your 82 | arms and swearing that there is no other way to achieve resiliency on 83 | distributed systems. 84 | 85 | ## Predictions 86 | 87 | They talk a little about the fallacies of predicting the future too. 88 | He gives some examples of "awesome" predictions he has saw on his life: 89 | 90 | * The future is IBM mainframes 91 | * The future is Windows 92 | * Linux widely regarded as a toy 93 | 94 | Since human nature/abilities doesn't change very fast, keep that in mind 95 | when you hear people on the industry making predictions about the future. 96 | 97 | I prefer Alan Kay's idea of inventing the future as a way to predict it or 98 | Paul Grahams idea of being flexible and adaptable instead of putting energy 99 | on predicting, because predictions almost invariably fail. 100 | 101 | ## Abstractions 102 | 103 | They talk how the right simple abstraction is fundamental to tame 104 | complexity. One of the reasons of Plan9 awesomeness was how far 105 | they were able to push the file abstraction as a means to export/consume 106 | services in a very simple way. It does not fit everything (that would be 107 | the holy grail), but they took it much more far than Unix in a way 108 | that made distributed computing much easier. 109 | 110 | The core lesson is the importance of these few good abstractions and that 111 | convincing yourself that you found them is not a very good idea, we still have 112 | a lot to learn on how to come up with good abstractions for 113 | distributed computing. 114 | -------------------------------------------------------------------------------- /notes/books/problem-human-energy.md: -------------------------------------------------------------------------------- 1 | # The Problem of Increasing Human Energy 2 | 3 | [The book](https://www.amazon.com/Problem-Increasing-Human-Energy-Harnessing/dp/1605200956) 4 | starts with making a metaphor between humankind 5 | and physics. The first premise that Tesla strives to prove 6 | is that humankind is one thing. He uses a very sweet metaphor 7 | saying that the pain that he feels when a friend is suffering 8 | is the same that he feels when he himself is suffering, hence 9 | there is no clear boundary between him and his friend and 10 | they are part of the same whole. 11 | 12 | This reminds me a lot of oriental philosophy and Alan Watts 13 | The Book, the idea that we are part of a whole and the individual 14 | is just a construction/model of our minds trying to figure things out. 15 | 16 | Once you understand that we are part of the whole he builds on top 17 | of that to create the physics metaphor. Humankind is a whole object 18 | that is moving (even tough we don't know the direction correctly) 19 | and the individuals would be individual atoms, they can add or subtract 20 | from the movement, but they can't stop being part of it. 21 | 22 | On this idea you can use the equation **e=mc^2** to understand 23 | why the book is about energy. The idea is to increase human energy 24 | (he does not explain why we should even try to do that...perhaps because 25 | it is in our nature). 26 | 27 | # Increase Mass 28 | 29 | The first thing that he explores is increasing energy by increasing 30 | mass, or in other words to avoid the loss of life. In this sense 31 | he talks a lot about technological advancements both on water 32 | treatment and on food crops, which are basic to sustain human life. 33 | 34 | He does not go a lot on the idea that more people does not mean 35 | necessarily people that will add to a good direction for humanity, 36 | but it is interesting to read about a time were clean water was a 37 | hard problem to solve, even tough some places still suffer from it 38 | big cities in general solved the problem in a level that people 39 | don't even think about it anymore (even tough the sustainability 40 | of the solutions can be questioned). 41 | 42 | He also talks a lot about loss of lives in wars and the 43 | uselessness of it. Avoiding war would be another force towards 44 | increasing human energy. 45 | 46 | 47 | # Increase Speed 48 | 49 | When talking about increasing speed he elaborates on how there are 50 | 3 types of people/particles in the humankind object. There are people 51 | that are part of the movement, people that contributes only with attrition 52 | (inertia) and retard movement and people who are going in a different 53 | direction. The worst people to him are the ones that just produce attrition 54 | because they just resist change and ideas but don't have any of their own. 55 | People who have different ideas and put energy on them can at least be 56 | dissuaded and can contribute to better ideas. People who just resist change 57 | are harder to dissuade (if possible at all). 58 | 59 | Now there is the question of what would be the correct direction. His opinion 60 | is that the correct direction is reason and science. Which is not a surprise, and 61 | even tough it may not be perfect it seems better than the previous human 62 | directions of mysticisms and prejudice. One of the forces that contribute 63 | to this direction is education, for example, it would be another good investment 64 | to improve the energy of humankind. 65 | 66 | 67 | # Actual Energy 68 | 69 | It makes sense to think that to increase human energy we need literal 70 | ways to increase our capacity to transform energy sources in the environment 71 | in energy that we can use to maintain automation that is essential to 72 | human life (think on water purifiers for example). 73 | 74 | He talks a lot about that, ways of extracting energy from the environment 75 | that I was not even able to understand properly =/, usually by heat exchanges, 76 | and the ability to transform coal into energy using cold techniques. 77 | 78 | He elaborates a lot on extracting energy from waterfalls and even touch the 79 | subject of extracting energy directly from the sun, but he won't go into 80 | much details on the subject. 81 | 82 | # Automatons 83 | 84 | He also elaborates on a model of human beings as being just automatons responsing 85 | from stimuli generated by sensory organs. Which makes perfectly sense. 86 | Everything that we do is a response to some stimuli. If that make sense 87 | it also makes perfectly sense to imagine an artificial construction of 88 | this automaton. He elaborates on how he thinks to be extremely feasible 89 | and inevitable the build of automatons that are very like human beings. 90 | 91 | Having this mental model at the time that he lived was revolutionary 92 | to say the least, and it impressed me greatly because the idea came from 93 | how he modeled human beings, not from the internet seeing how powerful 94 | GPUs are and etc =P. 95 | 96 | 97 | # Future of Warfare 98 | 99 | Another interesting thing is how he predicts that warfare would 100 | evolve to be a show play of technological prowess instead of people 101 | killing each other. It will evolve in a way to show who has superiority that 102 | does not involve so much people actually dying in horrific situations. 103 | 104 | This does make sense, even though we have plenty of killing in warfare 105 | today a lot of what happened in the first two world wars would make no sense 106 | today given the technological advancements. But of course that weapons of 107 | mass destruction do make killing impersonal but also kills a lot of people. 108 | But in an age of technological advancement, perhaps there will be other 109 | ways of dominating that are more profitable and less dangerous than just 110 | wiping out entires cities. So humankind will continue to fight, but on a 111 | whole different arena. 112 | 113 | It does seem to make some sense, but it is hard to be true if things 114 | will happen that way, it would be another case of him being extremely visionary. 115 | -------------------------------------------------------------------------------- /notes/protocols/quic.md: -------------------------------------------------------------------------------- 1 | # QUIC 2 | 3 | QUIC is a transport protocol (layer 4) which overlaps in some features 4 | with TCP, specially in the idea of providing as an abstraction 5 | streams of ordered data being transmitted, but it has a very different approach 6 | to connections and streams and some interesting implementation details 7 | that makes it an interesting alternative to TCP in some scenarios. 8 | 9 | Actually the whole idea of QUIC is to circumvent some of the limitations 10 | of using TCP/HTTP2, a lot of stuff was improved on HTTP2 to make it more 11 | efficient, like multiplexing multiple requests on the same connection/stream 12 | and being a binary protocol instead of a text protocol. 13 | 14 | But TCP was not designed to multiplex 15 | different streams/requests on the same connection, given one TCP connection 16 | you have one stream and data is sent ordered on that stream, so if you 17 | send multiple requests on the same TCP stream you are subject to ahead of 18 | line issues, where the order guarantee of the TCP stream will delay 19 | the receive of data from a request. Since requests are orthogonal between each 20 | other the order guarantee of the TCP stream hurts performance. 21 | 22 | QUIC was born as a way to be more efficient on this model where multiple 23 | requests are made using the same underlying connection, which is reflected 24 | on its design. Here I document some core design decisions that caught my 25 | attention, not much because they are amazing by itself, but they present 26 | considerable differences to TCP and may be a better fit in some scenarios 27 | because of it (the main one being "better HTTP2"). 28 | 29 | 30 | # Built on top of UDP 31 | 32 | This one may be accidental, since the only protocols that will work on Internet 33 | infrastructure are TCP and UDP if you want to have a transport protocol that 34 | works on the "wild" you will need to do something on top of one of these two, or 35 | buy a huge fight like IPv6, which so far shows how hard it is to introduce 36 | new protocols at Internet scale. Since networks were designed for encapsulation 37 | it is fairly easy to build new protocols on top of UDP/TCP. Since TCP has 38 | guarantees that goes against the QUIC design, you are left with UDP as an option. 39 | 40 | There are two properties that emerge from this decision that I can think so far. 41 | One is that you will always have the option of using your own user space 42 | implementation of QUIC, even when kernels already implement it on the future, which 43 | seems like a nice/useful option that is much harder to do with UDP/TCP. 44 | 45 | Another one is that at least for user space implementations you don't have 46 | limitations on how much connection you can establish per process, beyond 47 | the CPU/Memory, since you are going to have just one file descriptor open 48 | for the UDP socket and all connections/streams are built on top of that 49 | on user space. I have no idea how that scales, but I'm kinda curious if it would not 50 | be feasible in that model for a process alone manage 100K connections, something 51 | that is not doable in TCP given the max number of file descriptors limitation. 52 | 53 | 54 | # Connection identity decoupled from IP 55 | 56 | As the time I'm writing this I still don't know much details on how 57 | this work, I just know that QUIC connections are decoupled from IP addresses 58 | in the sense that the identity of a connection is not tied to IP's, instead 59 | of the traditional quadruple `:::`. 60 | 61 | I would guess it is related to TLS and that the enforcement of it on 62 | the protocol guarantees a way to know that after you start a connection 63 | any communication that happens on it is still being done between the same 64 | endpoints, even if its IP address change. So you can decouple from IP 65 | as a mean of identification but without giving up security. I'm not very 66 | good with security stuff but AFAIK it is possible to impersonate an IP 67 | depending on how a network is designed, so I'm not sure if using IPs as 68 | identity is a good idea anyway. 69 | 70 | The main property that came to my mind with this is that connections will be 71 | much more "stable" in mobile networks, where you can change the network you are 72 | multiple times while moving (like in a car) and the connections you made 73 | just keep working, this seems to provide a simpler connection abstraction 74 | to users, since you don't need to handle unnecessary disconnections. 75 | 76 | With TCP in a scenario like that you always end up disconnecting 77 | and having to reconnect again, which makes building synchronous request/response 78 | protocols on top of it annoying, specially if the operation is not cheap/fast. 79 | You end up with something stateful using polling or some alternative that 80 | will require some state being stored somewhere anyway, which sums up to a 81 | more complex design (when compared to stateless request/response). 82 | 83 | QUIC seems to make it easier to have 84 | simple request/response protocols even when the requests may take some time 85 | to finish, both because you have less disconnect issues and also because 86 | maintaining multiple connections open doesn't hit hard limits like 87 | amount of file descriptors (although you may still run out of memory). 88 | 89 | More on how connection migration happens can be seen 90 | [here](https://tools.ietf.org/html/draft-ietf-quic-transport-27#section-9). 91 | 92 | 93 | # N streams per connection 94 | 95 | Another interesting design decision is that one connection can have 96 | multiple independent streams associated with it. For a protocol like HTTP2 that 97 | models multiple requests/responses to the same server as multiple streams 98 | being able to explicit create independent streams on the connection level seems 99 | like a good fit, which makes sense since it was mainly designed for this. 100 | 101 | So each stream has the same guarantee as TCP, where bytes are delivered 102 | in an orderly fashion, with automatic retransmission, but you can have multiple 103 | of those on top of the same connection so you don't suffer with ahead of 104 | line issues caused by multiplexing multiple streams on top of just one stream 105 | (or the traditional problems caused by using multiple connections on TCP, like 106 | delays caused by connection establishment). 107 | -------------------------------------------------------------------------------- /notes/articles/humble-programmer.md: -------------------------------------------------------------------------------- 1 | # Humble Programmer 2 | 3 | This [turing lecture](https://www.cs.utexas.edu/~EWD/transcriptions/EWD03xx/EWD340.html) 4 | does a great introduction to the concept 5 | of software being "intelectually manageable" which is a cool 6 | way to say "I'm able to understand this shit". 7 | 8 | As usual the more recorrent subject on software quality 9 | is the ability to understand code that has been already written. 10 | Writing a lot of messy code is easy, developing something easy 11 | to read and understand is hard. 12 | 13 | It raises the issue if software complexity must grow exponentially 14 | or linearly with the size of the problem. On Dijkstra opnion it is 15 | possible to be linear, and the only way to do that is to factor 16 | code correctly, so our limited minds can grasp the complexity of 17 | software on one piece of a time. 18 | 19 | A key piece of this is abstraction. The purpose of abstraction is 20 | not be vague, but to create a new semantic level and be precise 21 | on this new semantic level. This kind of isolation from other 22 | details is what enables us to get a good grip on each part of 23 | the solution to a possible big problem, the way to scale 24 | software development. Good abstractions are crucial. 25 | 26 | This is important because layers and abstraction also have 27 | developed a bad reputation thanks to over enginered object 28 | oriented designs. 29 | 30 | Another cool aspect of the article is the obvious idea that 31 | we are very limited, phrases like: 32 | 33 | ``` 34 | The competent programmer is fully aware of the 35 | strictly limited size of his own skull. 36 | ``` 37 | 38 | One of our limitations is the tool we use to express 39 | our thoughts, the language we use, that is why the choice 40 | of language is by far the most important one, the ways 41 | you can conceive a solution to a problem is limited by 42 | the language you are using. The expressiveness of the language 43 | will limit how far you can think on solutions to the problem. 44 | 45 | Sometimes we think it is not like that, we like to think that 46 | we are awesome, but we are not, the language you choose will 47 | limit you. That is another way to be humble, realize that 48 | and search for good modest languages to develop in, they 49 | are going to enable you, or limit you. 50 | 51 | An interesting point of view is how can we expect to tame the 52 | complexity of our code if we do not even tame the complexity of the 53 | language chosen for the task: 54 | 55 | ``` 56 | I absolutely fail to see how we can keep our growing programs 57 | firmly within our intellectual grip when by its sheer baroqueness 58 | the programming language —our basic tool, mind you!— already escapes 59 | our intellectual control. 60 | ``` 61 | 62 | It is interesting that in projects that uses languages that are not 63 | feasible to be tamed because of their sheer complexity usually the 64 | problem is tackled with guidelines and enforcing just a subset of the 65 | language, the subset that is intelectually manageable and that will 66 | also keep your code intelectually manageable. But this effort is 67 | wasted time, the language should not be so complex that finding the 68 | correct subset of it is an integral part of the problem. 69 | 70 | Now for some fun =): 71 | 72 | ``` 73 | And if I have to describe the influence PL/1 can have on its users, 74 | the closest metaphor that comes to my mind is that of a drug. 75 | 76 | I remember from a symposium on higher level programming language a 77 | lecture given in defense of PL/1 by a man who described himself as one 78 | of its devoted users. But within a one-hour lecture in praise of PL/1. 79 | 80 | he managed to ask for the addition of about fifty new “features”, little 81 | supposing that the main source of his problems could very well be that it 82 | contained already far too many “features”. The speaker displayed all the 83 | depressing symptoms of addiction, reduced as he was to the state of mental 84 | stagnation in which he could only ask for more, more, more... 85 | 86 | When FORTRAN has been called an infantile disorder, full PL/1, with its 87 | growth characteristics of a dangerous tumor, could turn out to be a fatal disease. 88 | ``` 89 | 90 | Even hardware can impose models and ways of thinking that can 91 | empower you or hinder your ability to understand harder problems: 92 | 93 | ``` 94 | The reason that I have paid the above attention to the hardware 95 | scene is because I have the feeling that one of the most important 96 | aspects of any computing tool is its influence on the thinking habits 97 | of those that try to use it, and because I have reasons to believe that 98 | that influence is many times stronger than is commonly assumed. 99 | ``` 100 | 101 | The idea that we transcend our tools is misplaced when it comes to 102 | computing, perhaps this can change with time but the way we think 103 | today on how to code is even tied to details of how the hardware 104 | itself works (think paralelism for example). He puts this idea on a 105 | very clear way: 106 | 107 | ``` 108 | I observe a cultural tradition, which in all probability has its 109 | roots in the Renaissance, to ignore this influence, to regard the 110 | human mind as the supreme and autonomous master of its artefacts. 111 | 112 | But if I start to analyse the thinking habits of myself and of my 113 | fellow human beings, I come, whether I like it or not, to a completely 114 | different conclusion, viz. that the tools we are trying to use and the 115 | language or notation we are using to express or record our thoughts, 116 | are the major factors determining what we can think or express at all! 117 | ``` 118 | 119 | An excelent guidelines for developing tools and languages in general is: 120 | 121 | ``` 122 | brainpower is by far our scarcest resource 123 | ``` 124 | 125 | I think this is an essential shift on how we used to develop tools. 126 | Instead of thinking on what is possible with a computer we need to 127 | think on what is possible for a human being to understand given our 128 | limitations. This is not an invitation to mediocrity, but it is important 129 | to acknowledge the limitations of our brains. 130 | 131 | The action of designing is to impose restrictions, the restrictions should 132 | optmize for human cognition (and its limitations). 133 | 134 | Programming is hard, approach it with respect. 135 | -------------------------------------------------------------------------------- /notes/articles/relational-model.md: -------------------------------------------------------------------------------- 1 | # A Relational Model of Data for Large Shared Data Banks 2 | 3 | These are the notes from the great article 4 | [A Relational Model of Data for Large Shared Data Banks](https://www.seas.upenn.edu/~zives/03f/cis550/codd.pdf). 5 | 6 | Sadly I'm almost sure that I did not understood most part 7 | of the article, but I will try to register here some key points that 8 | seems interesting to me. 9 | 10 | The most interesting part of relational theory is why it has been 11 | conceived: 12 | 13 | ``` 14 | Activities of users 15 | at terminals and most application programs should remain 16 | unaffected when the internal representation of data is changed 17 | and even when some aspects of the external representation 18 | are changed. Changes in data representation will often be 19 | needed as a result of changes in query, update, and report 20 | traffic and natural growth in the types of stored information. 21 | ``` 22 | 23 | Why this is interesting ? Well, one of the best selling points 24 | of NoSQL databases is that they are easy to evolve without breaking 25 | applications. Well, the main objective of relational theory was to 26 | enable exactly that. Not just internal representation, but also 27 | changes on external representation. 28 | 29 | There will be some changes on external representation that will 30 | make applications break, but this is just what happens with 31 | NoSQL too, some changes will inevitably break, but the idea to 32 | avoid that and enable smooth evolution of the database as it's 33 | usage grows was a first class concern on the relational theory. 34 | 35 | This is also made very clear here: 36 | 37 | ``` 38 | Accordingly, it provides a basis for a high level 39 | data language which will yield maximal independence between 40 | programs on the one hand and machine representation 41 | and organization of data on the other. 42 | ``` 43 | 44 | The idea seems to enable programs with different purposes 45 | to use the same shared data bank. These days this seems 46 | like a bad idea, and incrementally we are repeating the same 47 | mistakes that people before us had made, because avoiding 48 | this will result on data duplication, and managing that 49 | on a lot of contexts is pretty hard. But the idea to organize 50 | data as documents with hierarchical structure, and duplicate 51 | data, are not novel and they just have serious limitations 52 | (and costs). 53 | 54 | Not that they are always bad ideas, but they are not the panacea 55 | that is sold to achieve scalability and awesome systems 56 | (it is not even new). 57 | 58 | It is not just the consistency problem, costs also rises with 59 | duplication, and storage is becoming cheap, but it is not free. 60 | And the costs of maintaining multiple databases are far from 61 | just storage space, there is also operational costs (much higher than 62 | the storage, depending on the scale). 63 | 64 | The most predominant model these days (2016/2017), at least among devs, 65 | is the document model, which is basically an hierarchical model. 66 | The section 1.2.3 Access Path Dependence explores how this model 67 | can easily generate problems as the demands that you have from 68 | your database evolve. 69 | 70 | On a hierarchical model you will always have a dependence on the 71 | access path. For example, lets say that a person has an address, 72 | like this: 73 | 74 | ``` 75 | { 76 | name: "bla", 77 | address: { 78 | "street" : "lala", 79 | "number" : 666 80 | } 81 | } 82 | ``` 83 | 84 | The first problem with committing to an hierarchy is coupling. 85 | Lets say you just want to load all addresses available on the system, 86 | to do some analysis, you will have to know (and couple) with the 87 | person set, and the access path to get to addresses on each person. 88 | 89 | To avoid that on a document model you could generate duplication, like 90 | maintaining a copy of all addresses on a set that has only addresses, or 91 | to avoid duplication altogether you would need to store a id for the address 92 | inside the person, that is just a reference to the address stored on 93 | the addresses set, which is basically what the relational model proposes 94 | (but implemented manually by you). 95 | 96 | As your database evolves, now other concepts also have addresses related 97 | to them, like companies, and it is not impossible for a company to have 98 | the same address of a person (specially depending on the level of detail 99 | of the address), again you have the duplication problem, or build some 100 | form of manual relation. 101 | 102 | Sometimes duplication may not be a problem, but it is interesting that 103 | the relational model was proposed to allow a system to evolve without 104 | requiring duplication of data and without requiring application to 105 | be fixed. Some schema changes can happen without breaking any application. 106 | 107 | The paper does a great job explaining the relational model as a predicate 108 | calculus applied to sets. If the sets are simple and normalized I fail to 109 | see any kind of limitation on what can be done if you have a good way 110 | to describe operations on top of these sets to create new ones, the 111 | possibilities seems to be pretty huge, and it is pretty flexible. 112 | 113 | The paper even introduces the idea of "expressible set", which is the set 114 | of all sets that could be expressed on the system, which is pretty big. 115 | It is the idea of composing on N different ways sets that are very 116 | simple (where simple implies the normalization process). It makes a lot 117 | of sense, even with my very little experience with huge datasets, because 118 | on software this kind of functional composition is already a good idea 119 | to achieve maximum flexibility. 120 | 121 | As said on the start, there is also a lot of other important concepts 122 | that I was not able to fully digest to the point where I can provide 123 | an opinion, more time and experience will be required. 124 | 125 | On the consistency subject, it is interesting that the author 126 | introduces the idea of inconsistencies that are unsolvable without 127 | manual interference from the user: 128 | 129 | ``` 130 | The point is that the system will 131 | normally have no way of resolving this question without 132 | interrogating its environment (perhaps the user who created 133 | the inconsistency ). 134 | ``` 135 | 136 | This is an approach used by a lot of databases to solve inconsistencies, 137 | specially the highly distributed ones that have more space for inconsistencies 138 | creeping in. 139 | -------------------------------------------------------------------------------- /notes/document-services.md: -------------------------------------------------------------------------------- 1 | # Documenting Services 2 | 3 | Some notes on how to document services, what worked for me, 4 | what didn't, etc. 5 | 6 | ## The Problem 7 | 8 | What would be a service ? By service I mean any binary that 9 | provides some service for you through an API that is (usually) 10 | language agnostic. I'm trying to make a distinction between 11 | a service and a library, libraries publishes APIs that are 12 | accessible through the runtime of the language that they 13 | are developed on (or through some type of FFI). 14 | 15 | Why make a distinction ? Libraries have their code as a form of basic 16 | documentation. Function prototypes and data structures have a lot to 17 | say. Not that you don't need any more docs, but some documentation along 18 | public functions is enough for me. Not that this is not an area that 19 | needs improvement but I see that we are much better served on this area. 20 | 21 | Golang for example have something that resembles doxygen docs and a system 22 | to express examples that are compiled and run and embedded on the docs later. 23 | 24 | The problem with services is that usually you are integrating with them 25 | from language X but the service has been developed in language Y and the developer 26 | of X probably doesn't even know which language the service he is consuming has 27 | been written on (nor should he, knowing things is fun but having to know in which 28 | language a service has been written at this level of abstraction seems like a 29 | bad idea). 30 | 31 | It may seem like the problem is to document distributed systems, which is the 32 | first thing that comes to my mind when I think about services, but it is not. 33 | It is not because there is a great deal to learn about how CLI tools are 34 | documented. With UNIX pipes you can compose multiple 35 | services together in a very elegant manner and usually there is no coupling 36 | on which language each tool has been written, the only thing that matters 37 | is the protocol (streams in the UNIX case), which is pretty much the case 38 | on distributed systems. 39 | 40 | ## To document or to not document ? 41 | 42 | Ok this seems like a stupid question, but it is important to analyze with 43 | honesty. The need to document your APIs are to help clients of your 44 | service to understand how to use it. These clients will even be you after 45 | some months. 46 | 47 | Although almost no developer that I know likes to document, 48 | my reasons for not liking it: 49 | 50 | * No short term benefit (perhaps medium/long term) 51 | * Documentation is basically duplication 52 | * It is duplication that you can't run, it is boring 53 | 54 | To mitigate some of these I try to document before (like tests with TDD) 55 | so the act of document helps me to think about the specification of 56 | the service, so there is a better short term benefit. Letting to document 57 | on the end is ask to not document anything at all, there is very little 58 | incentive (humans are pretty bad at thinking medium/long term, specially 59 | if the task is annoying). 60 | 61 | Having this is mind I always try to document as soon as possible 62 | (usually before) and I document as little as possible, where possible means 63 | other people not involved on development can understand what the service 64 | does and why, and how to integrate with it. 65 | 66 | The more details you document, more you are duplicating from the code and 67 | the chances that your docs will be outdated. So your service docs should focus 68 | on how I can use the service, not internal design decisions of your service or 69 | which language it is written in (if your protocol is not coupled with those 70 | neither should the docs). 71 | 72 | ## How to document ? 73 | 74 | Here I will elaborate on how I used to write docs for services, 75 | how I do today and how I want to do it tomorrow. The tomorrow 76 | is just a vision/direction, I still have no idea on how 77 | to implement some details. 78 | 79 | The challenges of good documentation that I use to compare 80 | different techniques are: 81 | 82 | * How easy is to keep it updated with changes on the service ? 83 | * If you have discovered the service, how easy is to discover the docs ? 84 | * How easily can you match a running version of the service with the docs ? 85 | 86 | There is the obvious "can I understand anything that you are saying ?", but 87 | this is more related to the contents of the docs and currently I don't approach 88 | it with any formalism, so you just need to explain well and provide examples =). 89 | 90 | Also these 3 questions are decoupled from the contents of the documentation. 91 | The contents are essential but they are not related to any of those challenges. 92 | Since they seem to be orthogonal I'm going to focus on those 3. 93 | 94 | ### Past 95 | 96 | In the past I used to document services APIs on an wiki pages. It was 97 | a on premisses wiki service on the place I worked at the time. 98 | 99 | It is not hard to think on why this was a bad idea, using the test 100 | 101 | #### How easy is to keep it updated with changes on the service ? 102 | 103 | Not very, sometimes people adding code and functionality to services 104 | did not even know that the docs existed, they where pretty far from 105 | where the code resided and the tooling to edit them was pretty 106 | different too (people coded with text editors, the wiki was edited 107 | on the browser). 108 | 109 | #### If you have discovered the service, how easy is to discover the docs ? 110 | 111 | As hard as it can get. With luck there would be a link on the project 112 | pointing to the wiki link. But now the problem of the service client 113 | is to find the project. From a system deployed in production finding 114 | the project where the source code is stored is not easy. This is 115 | usually mitigated with emails or just asking someone on the coffee. 116 | 117 | #### How easily can you match a running version of the service with the docs ? 118 | 119 | Good luck with that. With proper tagging on the repository and discipline 120 | on how things are documented (like a "since v2.0" on the docs) you may get 121 | some matching done, but in a model where the docs are so far away from the 122 | code that people do not even know it exists you are grasping at straws. 123 | 124 | #### Conclusion 125 | 126 | Are wiki pages bad ? Hell no. Are wiki pages a good idea to document 127 | services API ? Hell yeah. I would use wiki pages only to document knowledge 128 | that is decoupled from any individual service, like research material, 129 | even brainstorming (the [c2 wiki](http://wiki.c2.com/) for example) 130 | but never for documenting services API. 131 | 132 | ### Present 133 | 134 | ### Future 135 | -------------------------------------------------------------------------------- /notes/plan9.md: -------------------------------------------------------------------------------- 1 | # Plan9 From Outer Space: Namespaces Unleashed 2 | 3 | ``` 4 | What was unexpected was how well the model could be used to solve a 5 | wide variety of problems not usually thought of as file system issues. 6 | ``` 7 | From [The Styx Architecture for Distributed Systems](http://doc.cat-v.org/inferno/4th_edition/styx). 8 | 9 | ``` 10 | The representation of all resources as file systems coupled with an ASCII 11 | interface has proved more powerful than we had originally imagined. 12 | ``` 13 | From [The Organization of Networks in Plan 9](http://doc.cat-v.org/plan_9/4th_edition/papers/net/) 14 | 15 | ## Holy Grail: A Uniform Interface 16 | 17 | There is some holy grails on computing, and one of the biggest 18 | (if not the biggest) is a uniform interface to integrate 19 | all the things, at least all the ones we know. 20 | 21 | It seems like this is not feasible, it is always "depends", but one 22 | of the things that makes life so scalable is the DNA, and for life 23 | on earth, DNA is the uniform interface that makes all life work. 24 | 25 | So even being extremely hard to find this, the quest is worth it, 26 | because it will be the only way that we will achieve any true scalability. 27 | 28 | In my view Plan9 and almost everything around it seems to revolve 29 | around this quest for a good uniform interface. 30 | 31 | This quest can be seen in other places than Plan9, like the 32 | [REST](https://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm) 33 | dissertation. But let's focus on Plan9 and how they tried to 34 | achieve uniformity. 35 | 36 | ## In the beginning there where files 37 | 38 | ``` 39 | The integration of devices into the hierarchical 40 | file system was the best idea in UNIX 41 | ``` 42 | From [The use of namespaces in Plan9](http://doc.cat-v.org/plan_9/4th_edition/papers/names). 43 | 44 | Indeed it was. Plan9 pushes the UNIX idea to the limit proposing that the 45 | integration of everything should be done as a hierarchical file system. 46 | 47 | What would be everything ? Well, almost everything =), like: 48 | 49 | * Networking 50 | * Information on running processes 51 | * Any service that you can think of 52 | * Actual remote and local files 53 | 54 | Access to all this will be done via files. 55 | That is why namespaces are central to Plan9, every process has its own 56 | namespace that can be empty or inherited from it's parent. All services 57 | that a process can see and integrate with must be present on this 58 | namespace. Making stuff available on your namespace can be achieved via 59 | mounting. 60 | 61 | Services are represented through files. Since files (services) can be 62 | local or remote you need a common protocol between local and remote 63 | file servers, this is achieved through the 9P network protocol and 64 | mount devices. 65 | 66 | ## Mount away 67 | 68 | Examples on how mount works on plan9. 69 | 70 | ## Multiplication factor 71 | 72 | One thing that I found amazing is how a simple abstraction 73 | multiplies on different dimensions. 74 | 75 | Examples: 76 | 77 | * Building environments 78 | * Resource sharing (networking gateways, debugging) 79 | * Monitoring/Visibility (just monitor 9P messages, iostat for the rescue) 80 | * Mocking (talk about mocking time) 81 | * Containers (isolation between processes) 82 | 83 | Lets analyze each one of these advantages in detail. 84 | 85 | ### Development Environment 86 | 87 | Since plan9 has been made from developers to developers (themselves) 88 | it has great support on how to build your own development environment. 89 | 90 | There are probably a lot of ways that plan9 namespaces shines on this 91 | aspect, one of them are containers which are very useful for automated 92 | testing and reproducible deploy. 93 | 94 | But another not obvious way of how building development environments 95 | is better on plan9 is about the world famous linux **PATH** variable. 96 | The **PATH** variable is used to control the order of resolution of 97 | binaries paths. 98 | 99 | How does plan9 improves on that ? Having the concept of union mounts 100 | natively. 101 | 102 | You basically mount your local binaries directories directly to 103 | "/usr/bin" and decide if it comes first, overrides or comes after 104 | what is inside the "/usr/bin" path. This way you can extend the 105 | binaries available on your environment without using any other 106 | mechanism than namespaces + mounting. 107 | 108 | The result is one less thing on plan9, no **PATH**, and less is more. 109 | 110 | ### Sharing Resources 111 | 112 | ### Monitoring 113 | 114 | Where all services are exported as file services you 115 | just need a good way to monitor file services. Having that 116 | should give you a lot of interesting information about 117 | usage/saturation/errors/performance. 118 | 119 | ### Mocking 120 | 121 | TODO: add mock examples 122 | 123 | ### Containers 124 | 125 | ## Connection Server 126 | 127 | ``` 128 | On each system a user level connection server process, CS, translates symbolic 129 | names to addresses. CS uses information about available networks, 130 | the network database, and other servers (such as DNS) to translate names. 131 | 132 | CS is a file server serving a single file, /net/cs. 133 | A client writes a symbolic name to /net/cs then reads one line for each 134 | matching destination reachable from this system. 135 | 136 | The lines are of the form filename message, where filename is the path of the 137 | clone file to open for a new connection and message is the string to write to 138 | it to make the connection. 139 | ``` 140 | 141 | From [The Organization of Networks in Plan 9](http://doc.cat-v.org/plan_9/4th_edition/papers/net/) 142 | 143 | TODO: Talk about how it reminds HATEOAS and how it induces the 144 | same architectural property of being highly decoupled. 145 | 146 | Deployment of new services with different "messages" to establish 147 | connections won't break any client. 148 | 149 | ## Where files fall short 150 | 151 | Not everything fit the file abstraction =( 152 | 153 | ``` 154 | Nonetheless there are some operations in Plan 9 that are 155 | not mapped into file I/O. 156 | 157 | An example is process creation. We could imagine a message to a control 158 | file in /proc that creates a process, but the details of constructing 159 | the environment of the new process — its open files, name space, 160 | memory image, etc. — are too intricate to be described easily in a 161 | simple I/O operation. 162 | 163 | Therefore new processes on Plan 9 are created by fairly conventional 164 | rfork and exec system calls; /proc is used only to represent 165 | and control existing processes. 166 | ``` 167 | 168 | From [The use of namespaces in Plan9](http://doc.cat-v.org/plan_9/4th_edition/papers/names). 169 | -------------------------------------------------------------------------------- /notes/books/practical-vim.md: -------------------------------------------------------------------------------- 1 | # Practical VIM 2 | 3 | ## Visual Mode 4 | 5 | ### Usando seleção visual com comandos Ex 6 | 7 | Sempre que for realizada uma seleção visual você pode executar um comando Ex por simplesmente iniciar o comando com *:*, 8 | como você faria se não existisse seleção alguma. O comando será executado apenas na seleção. 9 | 10 | 11 | ### Selecionando text objects 12 | 13 | * Selecionar todo o texto entre duas chaves: ** 14 | * Selecionar todo o texto entre dois ": ** 15 | * Selecionar todo o texto entre parênteses : ** 16 | 17 | 18 | ## Normal Mode 19 | 20 | * Apagar tudo da posição do cursor até o final da linha: ** 21 | * Juntar linha atual a próxima linha: ** 22 | * Alterar palavra ** 23 | * Iniciar edição no início da linha: ** 24 | * Iniciar edição no final da linha: ** 25 | * Centralizar tela na posição cursor: ** 26 | * Adiantar para inicio da palavra: ** 27 | * Adiantar para fim da palavra: ** 28 | * Retroceder para inicio da palavra: ** 29 | * Retroceder para fim da palavra: ** 30 | * Marcar linha para voltar depois : ** 31 | * Voltar para a linha marcada anteriormente : *<'{register}>* 32 | * Ir para o arquivo : ** 33 | 34 | 35 | ### Operar em Text objects 36 | 37 | Praticamente todo o comando do vim pode operar em um text object, seguindo o padrão *<{comando}i{delimitador}>*. 38 | 39 | Você simplesmente compõe esse comando com outros comandos que precisam de motions, como o **, ** e ** e o delimitador que você deseja utilizar. 40 | 41 | Também é interessante que além de delimitadores, text objects podem ser também textos como parágrafos (**) e sentenças (**). 42 | 43 | Exemplos: 44 | 45 | Edição: 46 | * Apagar todo o texto entre duas chaves e iniciar edição: ** 47 | * Apagar todo o texto entre dois " e iniciar edição: ** 48 | * Apagar todo o texto entre parênteses e iniciar edição : ** 49 | 50 | Deleção: 51 | * Apagar todo o texto entre duas chaves: ** 52 | * Apagar todo o texto entre dois ": ** 53 | * Apagar todo o texto entre parênteses: ** 54 | 55 | Copia: 56 | * Copiar todo o texto entre duas chaves: ** 57 | * Copiar todo o texto entre dois ": ** 58 | * Copiar todo o texto entre parênteses: ** 59 | 60 | 61 | ### Edição do cursor até um carácter 62 | 63 | *<{operador}t{caracter}>* 64 | 65 | Exemplos de como realizar operações em um texto até que ele encontre um "(": 66 | 67 | * ** 68 | * ** 69 | * ** 70 | 71 | 72 | ### Substituindo texto em todos arquivos de um projeto 73 | 74 | * vim -o 75 | * set hidden 76 | * argdo s,,,ge 77 | * argdo update 78 | 79 | Mais sobre "set hidden":http://usevim.com/2012/10/19/vim101-set-hidden. Já coloquei no meu vimrc ;-). 80 | 81 | O regex pode ser algo como "**/*.h" para indicar que inclui sub-diretórios. 82 | 83 | 84 | ## Insert Mode 85 | 86 | * Apagar palavra anterior: ** 87 | * Apagar toda a linha: ** 88 | * Fazer uma conta e inserir resultado no texto: ** 89 | * Colar o conteúdo de um registrador: ** 90 | 91 | 92 | ## Macros 93 | 94 | ### Gravando macro 95 | 96 | * Iniciar gravação de macro : ** 97 | * Encerrar gravação de macro : ** 98 | * Executar gravação de macro : *<@{register}>* 99 | 100 | ### Reproduzindo macro 101 | 102 | * Em paralelo: *<:{range}normal @{register}>*, onde o *register* eh onde estarão os comandos a serem executados. 103 | 104 | 105 | ## Comandos Ex 106 | 107 | Todos os comandos Ex começam com o *:", seguidos pelo comando. 108 | 109 | * Colar o conteúdo de um registrador no comando (util ao substituir): ** 110 | * Listar registradores: ** 111 | 112 | 113 | ### Executando com filtro 114 | 115 | Executar um comando apenas nas linhas que batem com um padrao: 116 | 117 | [range]g//cmd 118 | 119 | Deletando todas as linhas com "abacate" em qualquer posicao: 120 | 121 | %g/abacate/d 122 | 123 | 124 | ## Patterns 125 | 126 | * Utilizar *\v* para habilitar o *Very magic search*, assim não é necessário fazer milhões de escapes (fica parecido com pearl e ruby). 127 | * Utilizar *\V* para habilitar o *Very no magic search*, útil quando é necessário procurar por uma palavra cheia de caracteres bizarros. 128 | 129 | 130 | ## Registradores 131 | 132 | ### Uso 133 | 134 | Qualquer comando que utiliza registrador (seja leitura ou escrita) pode ser pré-fixado com um registrador, seguindo o padrão: 135 | 136 | * *<"{reg}{comando}>* 137 | 138 | Exemplos: 139 | 140 | * Colando do registrador de yank (0): *"0P* 141 | * Copiando para o registrador de clipboard do sistema (+): *"+y* 142 | 143 | 144 | ### Registradores interessantes 145 | 146 | * Yank register: *<0>* 147 | * System clipboard: *<+>* 148 | 149 | 150 | ## Mapeamentos 151 | 152 | * Utilizar ** nos mapeamentos personalizados, evitando estragar mapeamentos naturais do vim. 153 | 154 | 155 | ## Autocomplete 156 | 157 | * Iniciar e navegar na lista: ** 158 | * Aceitar opção: ** 159 | * Sair do autocomplete: ** 160 | * Completar linha inteira: ** 161 | * Omni Complete: ** 162 | * Completar com correção ortográfica: ** 163 | 164 | 165 | ## Spell Checker 166 | 167 | * Habilitar: *set spell* 168 | * Desabilitar: *set nospell* 169 | * Ir para o próximo erro: *<]s>* 170 | * Ir para o erro anterior: *<[s>* 171 | * Corrigir palavra (cursor deve estar encima dela): ** 172 | * Adicionar palavra ao dicionário (cursor deve estar encima dela): ** 173 | * Remover palavra ao dicionário (cursor deve estar encima dela): ** 174 | 175 | 176 | ## Plugins Interessantes 177 | 178 | * Comentando código: https://github.com/tpope/vim-commentary 179 | * Class outline viewer: http://majutsushi.github.io/tagbar/ 180 | * Busca em texto selecionado no visual mode: https://github.com/bronson/vim-visual-star-search 181 | * Supercharged substituition :-): https://github.com/tpope/vim-abolish 182 | * Javascript: https://github.com/mozilla/doctorjs 183 | * CTags + GIT: http://tbaggery.com/2011/08/08/effortless-ctags-with-git.html 184 | 185 | 186 | ## Configurações especificas por linguagem 187 | 188 | É possível carregar um *vimrc* adicional além do que se encontra em *~/.vimrc*. Essa funcionalidade é provida pelo plugin: 189 | 190 | http://vim.wikia.com/wiki/Keep_your_vimrc_file_clean 191 | 192 | Basta criar o arquivo seguindo o padrão: *~/.vim/ftplugin/{filetype}.vim* 193 | 194 | Exemplos de como fica o arquivo para algumas linguagens: 195 | 196 | * *~/.vim/ftplugin/javascript.vim* 197 | * *~/.vim/ftplugin/ruby.vim* 198 | * *~/.vim/ftplugin/python.vim* 199 | 200 | O vimrc deve estar configurado com o plugin filetype: 201 | 202 | * filetype plugin on 203 | -------------------------------------------------------------------------------- /notes/books/building-microservices.md: -------------------------------------------------------------------------------- 1 | # Good service 2 | 3 | * Loose coupled 4 | * Has high cohesion 5 | * Has a good bounded context (domain is clear) 6 | 7 | 8 | # Reasons to split 9 | 10 | Why would you break a service in two or more ? Some reasons are presented. 11 | 12 | 13 | ## Pace of change 14 | 15 | Usually when different parts of a system changes at different rates 16 | it means that they handle considerably different aspects with different 17 | customers in mind. Different customer means a different reason to change 18 | that usually means a different pace of change. 19 | 20 | An example is how fast a UI can change compared to a database schema. 21 | Besides the different level of risk, the reasons for changing are quite 22 | different too. 23 | 24 | Given that difference, guaranteeing that changing one of them does not 25 | affect the other makes totally sense. A good design can guarantee that 26 | even if it is the same service/runtime. Separating on a different process 27 | with a protocol around seems to just create a "physical" barrier around 28 | this and enforce it. Which may be a good idea sometimes. 29 | 30 | But beware, if a team can't design good modules and isolate stuff on code 31 | they will probably do a lousy job defining good service integration protocols. 32 | 33 | 34 | ## Technology 35 | 36 | You will get much more advantage if you solve some part of your problem 37 | with Lisp, but the current service is written in Python. This is a hint 38 | that it may be a good idea to handle this new problem as a different service 39 | with its own runtime. 40 | 41 | From my own experience, developing bindings between different languages 42 | runtimes is nasty work that is best avoided, makes sense. You can of course 43 | just spawn a local process from the Python code, if it makes sense. 44 | 45 | But in a sense is a different service, with a protocol around it. Even a 46 | binding between languages is a form of that. 47 | 48 | 49 | ## Scalability 50 | 51 | Everyone loves to scale, even when they don't have too. 52 | With different services it is trivial to scale only the parts of the system 53 | that are really needing it, instead of just scaling the same monolith 54 | that in theory loads a lot of resources when scaled. 55 | 56 | Emphasis on the "in theory", if you have a really lazy service scaling it 57 | should just use resources of the parts of the system that are being requested. 58 | 59 | For example, think about features A, B, C. If you separate in three services 60 | you can scale them separately. But putting the three of them on the same 61 | service does not mean that when I scale the service it will waste up a lot 62 | of resources, specially if it allocates resources lazily, only when requested. 63 | 64 | 65 | ## Security 66 | 67 | Some part of the system has different security concerns from other, breaking 68 | in more services can make things easier and make more explicit the ones 69 | that have more security relaxed constraints. 70 | 71 | 72 | # Service Integration 73 | 74 | ## Good integration protocol 75 | 76 | * Tech agnostic 77 | * Easy to use 78 | * Can evolve without breaking current clients 79 | * Hides implementation details (No OO/language details) 80 | * Basically think on how your protocol will respond to change 81 | * Also think about something really boring and ubiquitous (like http + JSON) 82 | 83 | 84 | ## Database Integration 85 | 86 | Using the database to integrate multiple services 87 | 88 | * No cohesion (usually logic around the data is spread across multiple services) 89 | * To avoid duplication you can put logic inside the database...at your own risk 90 | * High coupling (all services tied up on the same schema) 91 | * Hard to perform non breaking changes (lack of orthogonality) 92 | * If you have more than one service writing data you cant trust the data 93 | * Mixed concerns, you have protocols/integrations and data/queries sitting together (good modeling can avoid this, never saw that although :-) 94 | 95 | 96 | # Testing 97 | 98 | * Avoid end to end tests (hard to be deterministic or fast, flaky and brittle) 99 | * Use [Consumer-Driven Contract](http://martinfowler.com/articles/consumerDrivenContracts.html) to avoid end to end tests 100 | * End to end tests responsibility is collective 101 | * Smoke tests on production environment with [Blue/Green deployment](http://martinfowler.com/bliki/BlueGreenDeployment.html) 102 | * Stubbing a integrated service on the networks is a good idea (mountebank and toxyproxy) 103 | 104 | 105 | # SOA and Microservices 106 | 107 | Basically the same thing, same core concepts. 108 | Microservices are SOA implemented right ;-). 109 | 110 | 111 | # Monitoring 112 | 113 | * How we will cut costs if we don't have any idea of how the service is performing on production ? 114 | * Good monitoring enables elasticity to cut costs 115 | * Monitor for some time and perceive what is the *normal* behaviour of the system (cpu/mem/io usage) 116 | * How to really know if the service is working ? [Semantic Monitoring](http://www.thoughtworks.com/pt/radar/techniques/semantic-monitoring) 117 | * Semantic monitoring usually is made with synthetic transactions 118 | * Monitor asynchronous operations with correlation ids 119 | * Measure response time and error rates, also for downstream dependencies (circuit breaking) 120 | 121 | 122 | # Security 123 | 124 | Having smaller services gives you the advantage of building different security precautions to different services, 125 | depending on the sensitivity of the information they hold, or on what they can do. 126 | 127 | There is a series of approaches/tools you can take on security: 128 | 129 | * [HMAC](https://en.wikipedia.org/wiki/Hash-based_message_authentication_code) 130 | * [JWT](http://jwt.io/) 131 | * [OpenID Connect](http://openid.net/connect/) 132 | * [SAML](https://en.wikipedia.org/wiki/Security_Assertion_Markup_Language) 133 | * [Shibboleth](https://shibboleth.net/) 134 | * [Gluu](http://www.gluu.org/) 135 | * [OpenAM](https://en.wikipedia.org/wiki/OpenAM) 136 | * Client Certificates 137 | * Everything allowed on perimeter (main entrance gateway) 138 | * Basic Auth on HTTPS 139 | * API Keys 140 | 141 | It depends on the usage of your API. To provide services to a small set of partners, client certificates may be a good 142 | idea. But for a great set of clients it would be terrible to manage, for example. 143 | 144 | 145 | # Resilience and failure recovery 146 | 147 | * DiRT (Disaster Recovery Test) 148 | * DiRT remindes me of Chaos Monkey set of tools :-) 149 | 150 | 151 | ## Good patterns 152 | 153 | * Bulkheads 154 | * Timeouts 155 | * Circuit Breaker 156 | 157 | 158 | # Related materials 159 | 160 | * [Domain Driven Design](http://www.amazon.com/Domain-Driven-Design-Tackling-Complexity-Software/dp/0321125215) 161 | * [A Classic Introduction to SOA](http://dannorth.net/classic-soa/) 162 | -------------------------------------------------------------------------------- /notes/books/compiler-construction.md: -------------------------------------------------------------------------------- 1 | # Compiler Construction 2 | 3 | These notes are based on the 4 | [Compiler Construction](http://www.ethoberon.ethz.ch/WirthPubl/CBEAll.pdf) book. 5 | 6 | As usual the notes are not a smaller version of the book, so there will be no 7 | explicit explanation on how to build compilers, just some remarks of the things 8 | that got my attention (the basics I got from colleague). 9 | 10 | If it was a completely different approach on how to build compilers, perhaps it would 11 | make sense, but it is prettu much standard LL/LR parsing theory with lexers/scanners 12 | involved. It is more to review stuff and to find some nice subtleties (and some where found). 13 | 14 | # Attributed Grammars and Semantics 15 | 16 | This is the chapter 5 of the book and it presents a very interesting idea, which is 17 | to add attributes to certain gramatical constructs. The idea is hard to understand until 18 | you get a more concrete example, as type checking. Quoting directly: 19 | 20 | ``` 21 | Rules about type compatibility are indeed also static in the sense 22 | that they can be verified without execution of the program. 23 | 24 | Hence, their separation from purely syntactic rules app 25 | ears quite arbitrary, and their integration into the syntax in 26 | the form of attribute rules is entirely appropriate. 27 | 28 | However, we note that attributed grammars obtain a new dimension, 29 | if the possible attribute values (here, types) and their 30 | number are not known a priori. 31 | ``` 32 | 33 | That got me thinking that type checking is validated at the syntatic level, but 34 | grammars and parser generators usually ignore this and type checking is done 35 | implicitely inside the evaluator (the AST is usually built and them it is 36 | checked for type mismatches). The types could be checked by the same code 37 | that builds the AST, and parser generators could also handle it generically 38 | if a proper syntax is added on something like EBNF. 39 | 40 | This is very new to me and I'm not sure if the syntax proposed for this on the 41 | book is the best one, but the higher level idea do make sense until experience 42 | proves otherwise. 43 | 44 | # Handling syntatic errors 45 | 46 | As the book evolves on the implementation of a compiler for the language Oberon-0 47 | it gets on the point of how to handle errors on the parsing process: 48 | 49 | ``` 50 | As soon as an inacceptable symbol turns up, 51 | the task of the parser is completed, and the 52 | process of syntax analysis is terminated. 53 | 54 | For practical applications, however, this proposition is 55 | unacceptable. A genuine compiler must indicate an error diagnostic mess 56 | age and thereafter proceed with the analysis. 57 | ``` 58 | 59 | Sadly the book does not evolve on **why** a genuine compiler must 60 | proceed the analysis. If you have several errors and every time you find one 61 | the compiler stops the only overhead is recompiling, you will need to analyze 62 | and fix each error anyway. It seems to be that this **genuine** property is 63 | related more to efficiency than to something conceptual. A compiler that 64 | fails on the first found error is much simpler and the first error is 65 | the only one that is true, all subsequent errors depend on heuristics 66 | that can easily be wrong. 67 | 68 | You have to assume the error and skips parts of the text to continue parsing, which 69 | is a process that can get pretty complicated and when it fails (and it will since it 70 | is a heuristics) will only produce a list of fucked up errors. 71 | 72 | Defining good heuristics even depends on understanding human cognition, 73 | as is said on the book: 74 | 75 | ``` 76 | The technique of choosing good hypotheses is complicated. 77 | It ultimately rests upon heuristics, as the problem has so far eluded 78 | formal treatment. 79 | 80 | The principal reason for this is that the formal syntax ignores 81 | factors which are essential for the human recognition 82 | of a sentence. For instance, a missing punctuation symbol 83 | is a frequent mistake, not only in program texts, but an operator symbol 84 | is seldom omitted in an arithmetic expression. 85 | ``` 86 | 87 | It is a great example, punctuation is more often ignored by humans than operations. 88 | If you are adding, "+" is fundamental, while ";" never is, it is just a gramatical 89 | formalism required by the compiler to understand things. Humans do fine with just newlines 90 | for example, which brings the subject of automatically adding semicolons or even 91 | spaces/tabs with semantic values as is Python. 92 | 93 | The problem is then summarized: 94 | 95 | ``` 96 | To summarize, we postulate 97 | the following quality criteria for error handling: 98 | 99 | 1. As many errors as possible must be detected in a single scan through the text. 100 | 2. As few additional assumptions as possible about the language are to be made. 101 | 3. Error handling features should not slow down the parser appreciably. 102 | 4. The parser program should not grow in size significantly. 103 | ``` 104 | 105 | It is not a simple problem and it has clear opposing forces (almost all interesting problems do), 106 | like having as many errors as possible AND not growing the parser significantly. The book 107 | does provide some nice ideas on how to handle this, even using syntax constructions 108 | to help identify how much text must be skipped, but to be honest it seems that 109 | with the growth of computational power and other techniques that aid efficient 110 | re-compilation it seems that this problem could be solved by not creating it. 111 | 112 | # Modules 113 | 114 | The modules part is pretty interesting, it constructs a pretty good argument to 115 | how important isolation of data and clear interfaces is fundamental to good 116 | software and then how implementing modules correctly on a language is 117 | fundamental. The arguments in favor of modularization are pretty well know 118 | so it makes no sense elaborating them here. 119 | 120 | There is a lot of details on how to compile modules independently and avoid 121 | recompilation when it is not necessary, it is clearly not an easy problem 122 | to solve. One interesting thing is that when it elaborates on how 123 | to handle when modules export types that belongs to other modules it shows 124 | two solutions, one where you have to transitively traverse all dependencies 125 | modules to load the type information and other where the type information 126 | is copied to the module that depends on it when it is exported. This way 127 | when you depend on one module you just need to load that module to link it. 128 | 129 | This is interesting because the second approach is the one used by Oberon 130 | and by Go. It uses more space on the compiled packages/modules but it makes 131 | the action of linking dependencies much easier and uses less I/O. -------------------------------------------------------------------------------- /notes/vietnam.md: -------------------------------------------------------------------------------- 1 | # The Vietnam War 2 | 3 | Well it is kinda weird to have notes about the vietnam war 4 | in the middle of notes that seem related to computer science 5 | only. Perhaps it is because this project is changing with me as 6 | I give more tought on social issues, but in the case of the 7 | vietnam war the focus still is to compare it with some problems 8 | in managing software that I have observed. 9 | 10 | It seems odd at first to compare a war with software development, 11 | but since a lot of the problems that we have with software development 12 | are caused by people and psicological human traits it makes sense 13 | to see some parallels. 14 | 15 | This is based on what I saw on the great documentary 16 | [The Vietnam War](http://www.pbs.org/kenburns/the-vietnam-war/home/). 17 | 18 | # The Sunk Cost Fallacy 19 | 20 | Perhaps the most obvious lesson is about sunk cost. Quoting from 21 | [here](https://youarenotsosmart.com/2011/03/25/the-sunk-cost-fallacy/): 22 | 23 | ``` 24 | The Truth: Your decisions are tainted by the emotional 25 | investments you accumulate, and the more you invest in 26 | something the harder it becomes to abandon it. 27 | ``` 28 | 29 | This one is not hard to explain since it is pretty obvious that 30 | the vietnam war was harder and harder to abandon as the USA 31 | government got more invested and did not wanted to "lose face". 32 | 33 | I have seem the same thing happen with software projects, both on the 34 | business side as in the technical side, where design ideas are obviously 35 | not working well but people are so emotionally invested that it 36 | seems impractical to abandon the idea and start over again. 37 | 38 | # Extensive Progress Reports and Metrics 39 | 40 | As the war effort escalated and the results did not seem favorable 41 | the amount of extensive and detailed reports sent to the USA started 42 | to raise to a level that was impossible at the time to be analyzed 43 | (it was done mostly manually, CIRCA 1965). Yet these reports where 44 | being written anyway, even if no one was reading it. 45 | 46 | This reminds me of an experience I had with software development where 47 | something very similar happened. As projects around the company stopped 48 | to progress and development got stuck more micromanaging happened and 49 | more detailed documentation on where time and effort was being spent. 50 | The amount of detail was not going to be analyzed by anyone, ever, because 51 | it did not made sense to have such a fine grained detail of what was happening 52 | because this was not what was going to solve the problem. 53 | 54 | Solving the actual problem was pretty hard, perhaps it is a psycological 55 | trait that when faced with a certain amount of uncertainty 56 | the quest for this kinda of detailed 57 | information starts to make sense. But truth is that the only way to really have 58 | a good insight on what was happening on the projects was to get involved 59 | on the day to day work and to talk with the people "on the trenches". 60 | The reports where on the thousands of pages per month level, it shows 61 | the desperation that is commonly found on over managed and documented failing 62 | software projects. In the end you can see that the war really looks like a failing 63 | software project because a failing project is a failing project and the behavior 64 | around the software projects where not introduced by computing, they are a human trait. 65 | 66 | But high management usually don't get involved on the details, sometimes 67 | for economic/practical reasons (the president of the USA cant get personally 68 | involved on all problems, or go to a war zone, even tough it would be the 69 | best way for him to asses the situation). They just want numbers and to 70 | know if there is progress. Sometimes this is possible, but on unusual environments 71 | it is not and then another fallacy can happen which is to measure the wrong thing. 72 | 73 | A cool phrase on the documentary is that "we are slaves of our previous experiences". 74 | The Vietnam war was pretty unique in a lot of senses, they where not gaining territory 75 | like in World War 2, the war was being fought on a very different and diffuse way 76 | and them it surfaces another cool phrase that goes something like this: 77 | 78 | ``` 79 | When you can't measure what is important, you make what you can measure important 80 | ``` 81 | 82 | In the case of the Vietnam war the only thing that was easy to measure 83 | was body count, so progress started to be measured in how much vietnamese 84 | were killed. What was the result ? Beyond fake numbers there was also the common 85 | practice of not discriminating between dead civilians and dead soldiers, a 86 | dead vietnamese was a dead vietnamese. Acceptance of the US forces from 87 | the people around them would be much more useful, but it is much harder to 88 | measure and report. 89 | 90 | The problem of focusing on what is easy to measure 91 | and to report is that it may not indicate true progress and may even 92 | have negative effects, like the lack of restrain on the use of deadly 93 | force, even when unecessary, which only made the US look bad and loose 94 | the confidence of the people they where supposed to help. 95 | 96 | Measuring the success of something as complicated as the war effort, 97 | which involved the opnion and feelings of the people towards the 98 | american troups is extremelly hard, perhaps even impossible, but measuring 99 | the wrong thing instead does not help at all. 100 | 101 | In software the common "dead bodies count" is bugs count. Counting bugs 102 | opened and closed is extremelly easy. It is just as easy to measure 103 | as it is easy to fake. Well, clients don't like bugs, what is the problem 104 | with that ? It is the wrong incentive, if bugs count is the metric 105 | of the quality of your job you have no incentive to be transparent 106 | about it, every bug that you find and tell no one about it the better. 107 | 108 | The problem of course is that assessing the quality of software is very hard. 109 | It involves experienced developers involved on the project, reading the code, 110 | implemeting features, solving the bugs and them open communication about the 111 | issues. Not one of these things is very easy to measure and to people 112 | just interested on numbers it is very hard to do things this way, communicating 113 | and feeling things. 114 | 115 | It sounds very odd to work with feelings, but everytime I worked on a environment 116 | that was thriving everyone felt it, and when things where going bad people also 117 | knew that. Of course these people are the ones involved in the project 118 | in a daily basis, not a manager that just looks from a safe distance once a month. 119 | 120 | The lesson is not to use metrics, but that sometimes they are very hard to get 121 | and when faced with that situation choosing something that is easy to measure 122 | may create the wrong incentive system and makes things even worse. 123 | 124 | -------------------------------------------------------------------------------- /notes/books/rework.md: -------------------------------------------------------------------------------- 1 | # Rework 2 | 3 | Notes for the [Rework](https://basecamp.com/books/rework) book. 4 | There was nothing very new on the book (for me at least), but 5 | nonetheless it is a good book, most of the ideas make sense. 6 | 7 | # Curator 8 | 9 | Liked the metaphor about being a curator when dealing 10 | with software. Museums are just about as what is left 11 | outside them than what is inside. Being a good curator 12 | is to know what to left out. You could even argue 13 | that knowing what to left out is even more important 14 | than adding the right stuff. It feels better to be wrong 15 | on the side of less than the side of more. 16 | 17 | Another example used are restaurants, the worse ones are 18 | the ones with gigantic menus that try to do everything. 19 | There is a famous show about a dude that helps restaurants 20 | that are struggling and the first thing that he always does 21 | is to check the menu and cut back on a lot of stuff, focus on 22 | less and in doing it well. 23 | 24 | When things are going bad in a company/project the solution 25 | is not to throw more money and people at it, chances are it 26 | will only make it worse since communication is not free. 27 | The solution is to find out what you can cut back. 28 | 29 | # Constraints 30 | 31 | As I get older I find fascinating the role of constraints 32 | in design. It is counter intuitive, because it is uncomfortable 33 | to be constrained, but it seems essential to good design 34 | and good ideas. This reminds me of this, from the 35 | [Taste for Makers](http://www.paulgraham.com/taste.html) essay: 36 | 37 | ``` 38 | When Bauhaus designers adopted Sullivan's "form follows function," 39 | what they meant was, form should follow function. 40 | 41 | And if function is hard enough, form is forced to follow it, 42 | because there is no effort to spare for error. 43 | 44 | Wild animals are beautiful because they have hard lives. 45 | ``` 46 | 47 | When there is room for error, errors are made. Nature is filled 48 | with amazing designs because less than amazing would die, there is 49 | not much resources going around, so you need to be smart, all 50 | sorts of amazing symbioses emerges, because it is needed. 51 | Unconstrained/unlimited resources would probably generate an 52 | endless horde of useless blobs since any crap will do. 53 | 54 | I have this feeling regarding web/cloud development. Any sort of 55 | stupid idea/design survives, because almost anything will work, because 56 | the amount of resources is gigantic/elastic. There is no room for 57 | clever ideas, any stupid idea will do. Perhaps this is great 58 | for business (although I doubt it), but for me personally it is 59 | terrible, since I'm not interested on stupid. 60 | 61 | There is elegance in efficiency and less, it is not just a matter of 62 | optimization, it makes things simpler and easier to understand, and almost 63 | everything on the web/cloud feels bloated and complicated, probably 64 | because it can, there is very little constraint to kill bad/bloated ideas. 65 | 66 | As I was reading this book I was also playing the director's cut version 67 | of Bioshock. I absolutely love that game, it is of a master piece level for 68 | me, and it was surprising to discover that they have a lot of severe 69 | limitations that drove the entire game narrative, like the idea of a underwater 70 | city because it was easier,given their limitations, to create a game 71 | that did not happen at open spaces/open world, and this is one of the most 72 | attractive aspects of the game, it feels very original and well executed, 73 | but it was driven by pure necessity, they had to be creative. 74 | 75 | # Interruptions 76 | 77 | This book is legendary on how it approaches meetings as categorizing 78 | them as toxic, and I can't agree more. Meetings and sync communication 79 | is not the same thing as good communication. Specially with people 80 | that creates things, that does creative work, interruptions have 81 | a high toll, which pushes you toward optimizing to asynchronous 82 | communication. 83 | 84 | One cool analogy on the behavior of the brain related to interruptions 85 | is REM sleep. You take hours to achieve REM sleep on the brain, if you 86 | wake up on the middle of REM sleep, interrupting it, as you sleep again 87 | it will take hours to get to it again, the brain just doesn't resumes from 88 | where it stopped, because it is not how our brain works at all. 89 | 90 | For mechanic repetitive tasks we can resume pretty well, but on the 91 | subject of ideas and creativity an interruption is just as destructive. 92 | People who don't do creative work feels like this is an exaggeration, 93 | but it truly isn't, people with much better credentials than me agrees: 94 | 95 | * [Makers Schedule](http://www.paulgraham.com/makersschedule.html) 96 | * [Good and Bad Procrastination](http://www.paulgraham.com/procrastination.html) 97 | 98 | And the book itself incentives towards practices that empower 99 | people to build their own schedules, so they can have alone time 100 | on the part of the day that is more important to them. This means 101 | using asynchronous text communication as a way to collaborate,and 102 | making synchronous communication the exception, which for creative 103 | work makes perfect sense. Synchronous communication only makes sense 104 | for things that are urgent, and things most of the time are not 105 | actually urgent in software development, and discussing them 106 | calmly and in a well structured manner will most certainly produce 107 | a better result. 108 | 109 | I have felt myself the effect of losing control of my 110 | schedule by being forced to go to meetings, after some years 111 | not having to do that, and it feels terrible. I completely 112 | agree with Paul Graham on the sense that it can destroy my entire 113 | day, specially regarding hard problems (you end up doing a set 114 | of more menial tasks on a day that is ravaged by meetings). 115 | 116 | # Hiring 117 | 118 | Just hire when it hurts, when you hire people, if they are not 119 | truly needed usually work will be "found" to keep them busy, that 120 | is bad business, it must be crystal clear why you hired someone. 121 | 122 | If the drive is "we need to go faster", really think, like REALLY think 123 | if what is making you go slow today is lack of people, or lack of something 124 | else. In a lot of scenarios less is more, perhaps you need less of the 125 | right people (the wrong engineer can create more problems that what he/she 126 | solves). 127 | 128 | Hire the better writer. That makes perfect sense to me, I have worked with 129 | my share of people who seem to be extremely lazy to read and write. It seems 130 | pretty hard to be a reasonably good engineer without reading and writing 131 | considerably. The lazy ones usually have an ad-hoc method of "just ask me on slack" 132 | (or whatever it is used for chatting these days). This is not good engineering, 133 | this is lazy and egocentric, you get to not properly think about stuff 134 | and explain it and you get to be the awesome guru who knows everything 135 | and answers all questions. 136 | 137 | Good code is decoupled from the person who wrote it, a good engineer 138 | has value not because of the work he as done in the past but because 139 | of the work he/she still can do in the future. For that to be true 140 | the engineer needs to know how to write, how to explain ideas to others 141 | clearly, via text in a form that doesn't need him to fill out gaps. 142 | 143 | Also hire slowly, if you hire too fast you end up with something that 144 | resembles cocktail parties, there is a lot of people being overly nice 145 | with each other (and not completely honest) because no one knows anyone 146 | and everyone feels uncomfortable. 147 | -------------------------------------------------------------------------------- /notes/books/hackers-heroes.md: -------------------------------------------------------------------------------- 1 | # Hackers, heroes of the computer revolution 2 | 3 | This book is fucking awesome is so much ways that is even 4 | hard to transcribe. 5 | 6 | ## What is a hack 7 | 8 | First of all it goes right in the origin of the term hack. 9 | So there is no more confusion on what is a hack, it is where the 10 | term originated, people had a purpose with it, and it was magnificent. 11 | 12 | It was basically anything that you worked on (not necessarily programming) 13 | that you did just for the joy of being involved on the work, doing it pleased 14 | you. The term was coined to differentiate from works that where related to 15 | classes or to a job, that sometimes where constructive and useful, but 16 | ultimately boring. 17 | 18 | A hack was something that you enjoyed, was fun, and ultimately had to be 19 | smart. A complete new way to look at a problem. An example of how useless the 20 | problem could be is one of the first programs wrote by the first hackers at MIT 21 | on the TX-0. A converter from Arabic numbers to roman. It was 22 | extremely hard to implement a complete and correct one with the limitations 23 | of the TX-0, and it was completely useless, no one ever used it. 24 | 25 | The fun was in taking the challenge, solving it, and them trying to solve it 26 | on a more clever way, using fewer instructions (called "bumming" the code). 27 | 28 | The hackers never thought that information should be hidden, they where also 29 | proficient lock breakers (physical ones). But they never stole anything, never 30 | made money from it, it was the pure challenge that attracted them, and the 31 | belief that information should be free. 32 | 33 | Putting a wall or a lock, was the challenge, and the challenge had to be 34 | tackled with clever solutions, but it was just that. This was done on the 35 | spirit of free information flow, that always contributed to better solutions 36 | since hackers could always improve on previous work from other hackers. 37 | 38 | Hacking is pure greatness, it is bliss, love is a bigger motivator than duty, 39 | and was love that was happening at the beginning of hacking. 40 | 41 | There was no product, no royalties, no patents. Of course that this was funded 42 | by DARPA, later the real world clashed with hacking. But before going into that. 43 | 44 | ## The Right Way 45 | 46 | This is one of the ideas that struck me more. There was a pursuit for the best 47 | solution to problems that was beautiful and fun. Even when you worked on top 48 | of someone solution, was to make it better, or even roll-out your own new solution. 49 | 50 | There was no idea of just using stuff to build products faster. An example is 51 | when they received a new PDP-1 computer and just rewrote all the utilities programs 52 | since the ones that came with the computer where not good enough, they where 53 | not done the "right way". 54 | 55 | People today think it is wrong to think that there is a right way, because it 56 | implies a wrong one, and they avoid conflict. 57 | 58 | These early hackers thrived on conflicts, it was not about ego's, they just cared. 59 | They would discuss for hours subjects that seemed like a utterly time loss. 60 | This immediately reminded me of my dynamic with one of my greatest friends, 61 | he is as hacker as hackers can get, it really make me realize that I was 62 | missing that in my life. 63 | 64 | I convinced myself that I was not being pragmatic and not generating value, 65 | but to be honest, in my hearth, the feeling that moves me is "fuck value, this 66 | is fun". Conciliating this with making a living is necessary, but I inverted 67 | the priority of these things in my life and I got in trouble because of that. 68 | 69 | ## Homebrew Computer Club 70 | 71 | After the first batch of MIT hackers there was a second batch that 72 | where more hardware hackers. This made sense since there was no computer 73 | available to people yet, so they needed to build their own, this is as 74 | hacker as it gets, building your own tools. 75 | 76 | The first official personal computer was the Altair, based on the Intel 77 | 8080 CPU. It was about that time that the homebrew computer club emerged, 78 | a place where hackers would share absolutely everything, even industrial 79 | secrets. The hacker ethic was still in place with great energy. The club had 80 | just a few hundreds of people in the end, something like 200. 81 | 82 | One of the greatest forces behind the first personal computers where 83 | the ideas from the sixties, empowering people, decentralization and 84 | mistrust authorities. It seemed urgent to give people control of machines, 85 | instead of leaving them at the hands of the industry, represented by IBM, 86 | where only "priests" would be able to program computers. 87 | 88 | It was on the homebrew club that Apple was born, from Steve Wozniak idea 89 | of a perfect computer and Jobs market view. At the time that Apple started 90 | there was a lot of computer business emerging, like processor technology. 91 | Apple was the underdog, and surprised everyone. 92 | 93 | About the 80's, when Apple was starting, the spirit of openness started 94 | to give way to industrial secrets. But along with that there was great 95 | effort to create standards, like S-100 bus (based on the Altair). 96 | 97 | In the end hackers where able to start a revolution that no big company 98 | at the time thought it was possible (not a single one of them invested on 99 | personal computers). 100 | 101 | Apple got it right developing an open architecture, that was easy to extend, 102 | thanks to Wozniak view. This made Apple the first big hit on personal computers 103 | and paved the way for a new market that would generate a new generation 104 | of hackers and a billionaire industry. 105 | 106 | ## Hacking clash with the industry 107 | 108 | The Apple 2 started a marked for personal computers games 109 | that generated a lot of revenue. The book focuses a lot on 110 | Sierra Online, a company from Ken and Roberta Williams. 111 | 112 | They started with a game that they tried to sell to apple but 113 | got ignored. Ironically later they would do the same thing when 114 | they got bigger and lost a game of the year because of that. 115 | 116 | They started with sheer hacker energy, paying good royalties and 117 | relying on creativity from hackers. In time, as money poured in, 118 | Ken Williams started to want to use more "industry level" programmers 119 | to do the games instead of "prima dona" hackers, that where less 120 | predictable and behaved like artists. 121 | 122 | In the end a company named Eletronic Arts started and got a lot 123 | of Sierra Online talent promising the same freedom that Sierra once had. 124 | Both companies exists today yet, none of them retain the hacker 125 | spirit, and none of them are innovative and brilliant, although 126 | both still makes a profit. 127 | 128 | The industry changed the hacking landscape, but there will 129 | always be those that are moved by sheer curiosity and desire 130 | for knowledge, that loves what they do, people where the 131 | system is the end in itself, like a masterpiece. It will be 132 | interesting how this will fold on the future, in the past it 133 | was more comfortable to do research, there was a lot of government 134 | investment. These days we have to complect hacking with 135 | business, which brings in a heavy baggage of money driven people 136 | and marketing. The clash of these cultures is one of my 137 | deepest challenges, since today I still have no idea on how to 138 | handle that. 139 | 140 | Perhaps it is a natural cycle of software in business, in time 141 | people will only want the money and will make a way to get that 142 | without the sheer power of innovation and creativity, which are 143 | less predictable. It is easier to bet on this when there is 144 | nothing to lose. This makes me remind of the saying 145 | "You die a hero or you live enough to become the villain". 146 | 147 | -------------------------------------------------------------------------------- /notes/languages/limbo.md: -------------------------------------------------------------------------------- 1 | # Limbo 2 | 3 | This is inspired mainly on the 4 | [The Limbo Programming Language](http://www.vitanuova.com/inferno/papers/limbo.html) paper 5 | but is has also been augmented with some information from 6 | [The Inferno Operating System](ftp://ftp.mrynet.com/operatingsystems/inferno/paper01.pdf). 7 | 8 | It is more like some highlights of what I found interesting on Limbo. 9 | 10 | The language has been used to develop the Inferno OS on bell labs. 11 | It runs on top of a VM and it is statically typed. 12 | 13 | # Concurrency 14 | 15 | One of the major wins, besides being high level and garbage collected, 16 | is the concurrency model proposed by the language, as a first class 17 | citizen, which on this case is CSP. 18 | 19 | It goes to the extreme of providing no primitive synchronization 20 | mechanism, it has only channels to communicate between concurrent executions. 21 | 22 | To create new threads there is the **spawn** statement: 23 | 24 | ``` 25 | The term and expression-list are taken to be a function call. 26 | Execution of spawn creates an asynchronous, independent thread of control, 27 | which calls the function in the new thread context. 28 | 29 | This function may access the accessible objects in the spawning thread; 30 | the two threads share a common memory space. 31 | 32 | These accessible objects include the data global to the current module 33 | and reference data passed to the spawned function. 34 | 35 | Threads are preemptively scheduled, so that changes to objects used in 36 | common between threads may occur at any time. 37 | 38 | The Limbo language provides no explicit synchronization primitives; 39 | ``` 40 | 41 | I don't remember reading anything that explains if the thread is 42 | a green thread or a system thread, but my brain sucks =/. 43 | 44 | 45 | The concurrency model is awfully a lot like Go, actually Go is a lot 46 | like Limbo, which makes sense since Rob Pike worked directly on Limbo 47 | with Denis Ritchie. The **alt** statement is very similar to the 48 | **select** statement from Go. 49 | 50 | On the concurrency side there are some interesting disparities between 51 | Go and Limbo. Go has no easy way to perform multiplexing of N channels, 52 | while on Limbo it has support to read operations on lists of channels, 53 | it automatically multiplexes the entire list on a single read operation. 54 | 55 | On the other hand, Go has support to buffered channels while Limbo requires 56 | you to implement buffering by hand. It is an interesting instance of the 57 | tradeoff's usually done on language design. 58 | 59 | # Module System 60 | 61 | Other remarkable feature is the module system. It is similar to C in the 62 | sense that it is composed by headers + implementation files. This leads 63 | to duplication and these days I'm prone to not liking it. But the most 64 | interesting idea is the division of modules as static/const data and 65 | dynamic/mutable data. With a simple import of a module you can only 66 | access constant data, which is allocated statically and never changes. 67 | Every mutable data (and all functions) of a module can only be accessed 68 | when you instantiate the module. There is no support for global mutable state 69 | on modules. For two different modules to share the same instance of a third 70 | module they must do it explicitly, one of them must instate the module and pass 71 | it as a parameter to ther other. 72 | 73 | This is very different from languages that allow initialization on a module 74 | that is executed only on the first time it is imported and them all 75 | code using that module shares this state initialized only once 76 | (Go has the init function for this, on interpreted languages like python 77 | it is everything on the main body of the module). 78 | 79 | Limbo has no support to that, there is 80 | no way to implicitly share data among modules, at least not using import 81 | mechanisms (there may be others). This is very interesting. It makes using 82 | modules a little more cumbersome but adds a lot of safety on how side 83 | effects ripple through your code. 84 | 85 | In the end modules seem more like a first class citizen, just like functions, 86 | they are instantiated and passed as parameters as any other kind of value 87 | on the language. 88 | 89 | One of the main motivators for this design is the constrained environments 90 | where Inferno was supposed to run, this dynamic model of loading modules 91 | enables code to just load the modules that it actually needs and to release 92 | all the resources of the loaded module when it is not needed anymore. 93 | 94 | On a more traditional language (almost everyone I know) you load modules 95 | at the main of your program, this happens automatically, and after a module is 96 | loaded it will never be unloaded. This is simpler to use but has the drawbacks 97 | of having implicit shared (possibly mutable) state and loading and keeping in 98 | memory stuff that is never going to be used. 99 | 100 | A use case that exacerbates this is CODEC handling. Lets say you are implementing 101 | a video player. There is 3 components to video playing: 102 | 103 | * The container of the format (AVI, MP4, etc) 104 | * The video CODEC (h.264, vp8, vp9) 105 | * The audio CODEC (mp3, PCM, ogg) 106 | 107 | The possible combinations of these formats are huge. Now imagine that the video player 108 | has to statically load in memory all the CODECS that is supports. Even if this loading 109 | is something very basic just as some constants and functions, or a huge number of formats 110 | and in a constrined environment it can be a problem. In a language like Limbo, it is 111 | easy to develop a player that will only load the necessary container/video/audio modules, 112 | and only when they are required (reminds me of lazy evaluation). 113 | 114 | Of course there is a clear tradeoff in usability and even readability. For someone used 115 | to Go, or example, seeing modules being imported and used in Limbo is pretty awckward. 116 | 117 | 118 | # Garbage Collector 119 | 120 | 121 | Limbo garbage collector has two distinguished features that caught my attention. 122 | From [The Inferno Operating System](ftp://ftp.mrynet.com/operatingsystems/inferno/paper01.pdf): 123 | 124 | ``` 125 | Because view was the top- level function in this process, 126 | the process will exit and free all its resources. 127 | All memory, open file descriptors, windows, and other 128 | resources the process holds will be collected as garbage when the return executes. 129 | 130 | The Limbo garbage collector uses a hybrid scheme that combines 131 | reference counting with a real-time sweeping algorithm. 132 | 133 | Reference counting allows reclamation of memory the instant its last reference disappears; 134 | the sweeping algorithm runs as an idle-time process to reclaim unreferenced circular structures. 135 | 136 | The instant-free property means that system resources like file descriptors and 137 | windows can be tied to the collector for recovery as soon as they are no longer used. 138 | 139 | This property allows Inferno to run in smaller memory arenas than those required 140 | for efficient mark-and-sweep algorithms, and it also provides an extra level of programmer convenience. 141 | ``` 142 | 143 | Resuming: 144 | 145 | * The GC is hybrid, it will use simple ref count when possible 146 | * The GC collect other resources besides memory, like file descriptors and window handlers 147 | 148 | The "process" described on the given note is a limbo process, not a operational 149 | system process (like a goroutine in Go). This is another very interesting aspect of the 150 | language since it will avoid some cases where file descriptors leak (not all of them I think). 151 | 152 | If it is guaranteed that your process exit, and it does not share any of its resources, even if 153 | you dont release then manually (with close, for file descriptors) they will be released automatically. 154 | Indeed it is an extra level of convenience for the programmer =). -------------------------------------------------------------------------------- /notes/concurrency.md: -------------------------------------------------------------------------------- 1 | # Fan-out / Fan-in 2 | 3 | ## What ? 4 | 5 | Fan out refers to the action of starting N multiple 6 | workers to do a part of a job for you concurrently. 7 | 8 | The fan in part refers to the fact that usually you want to 9 | wait for all workers to finish and aggregate the results 10 | so you can use them. 11 | 12 | It remembers how map/reduce works. You distribute the 13 | problem to scale better but in the end you want 14 | one set of results. 15 | 16 | The basic problem is reading results from N concurrent 17 | (possible parallel) workers. 18 | 19 | Here I focus on exploring local concurrency/parallelism 20 | and how a language may support that. 21 | 22 | Since right now the best language for that that I know 23 | is Go I will use Go (1.X) syntax for examples. 24 | 25 | ## Why ? 26 | 27 | Usually this is a performance matter. Some problems are 28 | so computing expensive that requires this and others 29 | are naturally expressible this way. 30 | 31 | An example is captcha breaking, after you segment the 32 | digits (audio or image) the action of recognizing is 33 | independent between digits. If you have 12 digits you 34 | could classify the 12 digits concurrently. But you need 35 | an easy way to wait all of them to finish and build the 36 | final solution to the challenge. 37 | 38 | This example will be used as a basis to discuss possible 39 | solutions to this. 40 | 41 | ## How 42 | 43 | So how can we implement the action of starting multiple 44 | workers and wait for all of them to finish ? 45 | 46 | Depending on the language this will vary, and even considering 47 | Go there is also a plethora of possible solution that one can 48 | imagine. 49 | 50 | Here we explore some ideas that are possible right now 51 | and some that are inspired on other languages that also 52 | seems nice as a language feature. 53 | 54 | Since the discussion involves Go it will be heavily biased 55 | towards CSP (communicating sequential processes). 56 | 57 | ### Multiplexing Channels 58 | 59 | The most common way to wait for any of N channels to have 60 | something to read from is using something like 61 | [Go's select](https://golang.org/ref/spec#Select_statements). 62 | 63 | One obvious limitation is the lack of support for a runtime 64 | configurable number of channels. In a select statement you must 65 | statically define how much channels are involved on 66 | the communication at compile time. 67 | 68 | Even with a fixed amount of workers, like a captcha with 6 digits, 69 | you still need some way to check if the 6 workers are already finished. 70 | 71 | If you choose using just one channel to be shared among all workers 72 | you will need some other synchronization/communication entity 73 | to communicate when each worker is done (maybe a 74 | [wait group](https://golang.org/pkg/sync/#WaitGroup) ). 75 | 76 | For me it is cognitively heavier to handle 2 different communication 77 | entities, it is even worse that they are of a different nature 78 | (the design lacks uniformity). Not that I'm against using 79 | synchronization instead of communication, but on this specific case 80 | of N workers producing results I don't feel it is a uniform simple solution. 81 | 82 | The idea is to get the concurrency model of 1 (receiver) <- 1 (worker): 83 | 84 | ```go 85 | var solutions chan solution = captchaSolver() 86 | 87 | for solution := range solutions { 88 | } 89 | ``` 90 | 91 | And expand it to N workers. Go already provides a way to express that some 92 | communication has ended (closing the channel). This pattern is so useful 93 | that even iteration on a channel until it is closed is provided by 94 | the language. The idea is to leverage this to solve the fan-in problem. 95 | 96 | One way to reach for this solution is to use Go's dynamic select 97 | features (using reflection) to build a generic channel multiplexer 98 | (you can implement one each time you need too). The usage would be: 99 | 100 | ```go 101 | var solutions []chan solution = make([]chan solution) 102 | 103 | solutions = append(solutions, captchaSolver(1)) 104 | solutions = append(solutions, captchaSolver(2)) 105 | solutions = append(solutions, captchaSolver(3)) 106 | solutions = append(solutions, captchaSolver(4)) 107 | solutions = append(solutions, captchaSolver(5)) 108 | solutions = append(solutions, captchaSolver(6)) 109 | 110 | var muxed chan solution = make(chan solution) 111 | 112 | err := muxer.Do(muxed, solutions...) 113 | //handle error, usually runtime type checking =/ 114 | 115 | for solution := range muxed { 116 | } 117 | ``` 118 | 119 | The main two advantages of this approach is that the captchaSolver 120 | function works exactly the same way it worked on the 1 -> 1 case 121 | (leveraging Go's channel behavior of closing to communicate end of communication). 122 | 123 | There is no need to change the function to use a wait group (or other mechanism) 124 | or to create a wrapper function that does that. 125 | 126 | The other advantage is uniformity with how we read all responses, 127 | just as it was with the 1 <- 1 case we just need to iterate all the 128 | responses of the channel and when the loop exits we are done, everything 129 | is closed and done. 130 | 131 | The major drawback is Go related. Since there is no generic programming 132 | (with parametric types, not interfaces) there is no way to implement 133 | a generic muxer without using reflection. This makes the API a little 134 | awkward to use and introduces the penalty of using reflection. There is 135 | also the cost of an extra goroutine that does the multiplexing for you. 136 | 137 | An example of how to implement this idea in Go can be seen 138 | [here](https://github.com/madlambda/spells/tree/master/muxer). 139 | 140 | ### Language Supported Multiplexing 141 | 142 | This one is inspired on the 143 | [Limbo Language](http://doc.cat-v.org/inferno/4th_edition/limbo_language/limbo). 144 | 145 | Quoting the specs (9.8. alt statement): 146 | 147 | ``` 148 | As mentioned in the specification of the channel receive operator <- 149 | in §8.2.8, that operator can take an array of channels as an argument. 150 | 151 | This notation serves as a kind of simplified alt in which all the 152 | channels have the same type and are treated similarly. 153 | 154 | In this variant, the value of the communication expression is a tuple 155 | containing the index of the channel over which a communication was received 156 | and the value received. For example, in 157 | 158 | a: array [2] of chan of string; 159 | 160 | a[0] = chan of string; 161 | 162 | a[1] = chan of string; 163 | 164 | . . . 165 | 166 | (i, s) := <- a; 167 | 168 | # s has now has the string from channel a[i] 169 | 170 | the <- operator waits until at least one of the members of a is ready, 171 | selects one of them at random, and returns the index and the 172 | transmitted string as a tuple. 173 | ``` 174 | 175 | The **alt** statement behaves a lot like Go's **select**, but in Limbo 176 | an array of channels has special behavior, just as single channels do. 177 | 178 | As discussed before, a static construction like select/alt will not 179 | be able to handle a runtime configurable number of workers and for 180 | a fixed one it can get pretty clumsy to work with. 181 | 182 | But this solution provided by Limbo gives a pretty nice way 183 | to express the fan-in without reflection clumsiness. 184 | 185 | Lets imagine that Go provides the same thing implemented by 186 | the compiler, we could have this: 187 | 188 | ```go 189 | var solutions []chan solution = make([]chan solution) 190 | 191 | solutions = append(solutions, captchaSolver(1)) 192 | solutions = append(solutions, captchaSolver(2)) 193 | solutions = append(solutions, captchaSolver(3)) 194 | solutions = append(solutions, captchaSolver(4)) 195 | solutions = append(solutions, captchaSolver(5)) 196 | solutions = append(solutions, captchaSolver(6)) 197 | 198 | for solution := range solutions { 199 | } 200 | ``` 201 | 202 | The drawback is that arrays of channels behaves differently 203 | of other arrays, it can be misleading. 204 | 205 | Also to actually iterate the array of channels you would 206 | need to use a numeric for (tradeoffs, make one case simpler 207 | and the other one harder). 208 | 209 | To know if it is closed and which channel: 210 | 211 | ```go 212 | var solutions []chan solution = make([]chan solution) 213 | 214 | solutions = append(solutions, captchaSolver(1)) 215 | solutions = append(solutions, captchaSolver(6)) 216 | 217 | s, ok, i := <- solutions 218 | // Ignoring index 219 | s, ok := <- solutions 220 | // Ignoring if it is closed 221 | s := <- solutions 222 | ``` 223 | --------------------------------------------------------------------------------