├── interviews
    ├── peter-dehaan.txt
    ├── fernando-jimenez-moreno.txt
    └── brian-warner.txt
└── articles
    ├── 01-fernando-jimenez-moreno.md
    └── 02-brian-warner.md


/interviews/peter-dehaan.txt:
--------------------------------------------------------------------------------
  1 | 
  2 | 
  3 | * What do you and your team do?
  4 | ---
  5 | [pd] I break things before the customer sees them, and get paid to complain for a living.
  6 | Basically QA acts as customer advocates and ensures that everything we ship is high quality.
  7 | But mainly I just sit on Google Image search all day trying to find whimsical images to attach to bug reports or flood IRC with.
  8 | 
  9 | 
 10 | 
 11 | * The relationship between QA and development is sometimes viewed as adversarial, but my experience with you is completely the opposite. QA is very much viewed as an integral part of the process. How do you facilitate this?
 12 | ---
 13 | [pd] Every team/developer is different. I think everybody takes great pride in their art, so when a QA (or user) files bugs against your features, sometimes developers take it personally. It can be difficult, but you have to learn to not take things personally. I used to have a blog and it would be very difficult for me when people would come by and leave comments on my examples and complain about my technique or code. I quickly learned that most of the time they were right, so I'd end up rewriting my examples or having discussions in the comments and I ended up learning a lot that way. Now getting negative comments on blog posts is the highlight of any post for me. Now this also translates into any time I submit a pull request against a repo and you get some good code reviews of why my code is weak and how you can make it better.
 14 | 
 15 | I had one Sr developer explain to me that he loved having QA trash his work and file a bunch of bugs, because it meant it was being looked at and used. He said as a person who used to have to answer his pager 24/7, having a bunch of bugs filed was great because it means it was less likely he'd have to be called in at all times of the night to come and fix issues if something went wrong.
 16 | 
 17 | 
 18 | 
 19 | * You are well known for your humorous bug reports and animated GIFs, do you find this affects the relationship between QA and dev?
 20 | ---
 21 | [pd] I try to not take things too seriously (obviously) so I try to add some levity to my bug reports. As for how this affects the relationship, you'd have to ask devs. I think I'm hilarious, but there is a slight possibility that there are some people out there that believe IRC and GitHub issues don't need animated GIFs every 45 seconds. Maybe devs see me as the guy at the table taking instagram photos of his dinner. :shrug:
 22 | 
 23 | 
 24 | * What do you look for before signing off on a release?
 25 | ---
 26 | [pd] quality, quality, quality! I think I'm more skilled in front end testing, so I like to run through as much of the flows as I can manually before releases to see the final product and workflows. If it's something that is slightly confusing or feels "off", then it's likely something that thousands (or millions) of people may also struggle with. It's all in the details, so shaving off any rough edges always helps.
 27 | 
 28 | Also, looking through the bug base for the open issues and making sure there are not any great "one last thing" bugs in there which need to be addressed before unleashing our code on an unsuspecting public.
 29 | 
 30 | 
 31 | 
 32 | * What do you consider a red flag/blocker?
 33 | ---
 34 | [pd] Low code coverage numbers.
 35 | Poor UX, is another big one. I've seen a lot of great and amazing apps which have been totally ruined by simple usability bugs or poor flows. It makes my fingers and eyes sad.
 36 | 
 37 | 
 38 | * You have reviewed most of the code that has gone into the fxa-content-server and do a great job at suggesting architectural changes and ensuring consistency - what do you look for when doing a review?
 39 | ---
 40 | [pd] Mainly just consistency. It's very tricky trying to remember coding between files and different pull requests. Basically this is how I start every morning. Looking through the bug dashboards for new bugs and reading all the comments and then reviewing each pull request and adding any comments which are hopefully relevant.
 41 | 
 42 | 
 43 | 
 44 | * What are the most common problems you see?
 45 | ---
 46 | [pd] Duplicated code, is probably the biggest and most common and trickiest to catch between files. Duplicating the same logic 14-15 times is OK, but once you hit the 16th time, it may be time to refactor the code.
 47 | 
 48 | 
 49 | 
 50 | * You have facilitated an increase in the quality of the fxa-content-server's code through tooling and automatic enforcement of standards. What types of tools do you most commonly rely on?
 51 | ---
 52 | [pd] I'm a huge fan of JSint and ESLint, and more recently jscs. All three have Grunt tasks, which is a huge bonus.
 53 | 
 54 | 
 55 | 
 56 | * What are 5 tools frontend developers should be using if they aren't already?
 57 | ---
 58 | [pd] In no particular order:
 59 | 1. Grunt (or similar build systems like Gulp) - This has changed the way I work considerably. Previously, I'd just have a bunch of random scripts on my drive that I'd have to try and remember to run, and in which order. I wish I could go back to previous projects and rewrite everything as a nice, easy build script rather than a bunch of random Node.js scripts.
 60 | 2. JSHint (or ESLint) - Validate your source code to ensure code consistency and "best practices". Both are pretty solid, although ESLint is a bit newer and more frequently updated/changed. Plus its a bit easier to extend and write custom linting tasks.
 61 | 3. Sass (and AutoPrefixer) - The best thing to happen to CSS since, like, ever. Plus AutoPrefixer is great because it lets you automatically add vendor prefixes based on a custom set of rules and browser adoption. Lets you reduce the amount of CSS/SCSS that you need to write.
 62 | 4. jscs - Enforce style consistency throughout your project. Does a lot of the style checking that you'll find in JSHint 2.x and ESLint, like enforcing consistent indenting, trailing commas, spaces around operators, etc. Everybody loves clean and consistent code, plus makes it a lot easier on big projects and projects which have a lot of contributors.
 63 | 5. Bower - Like npm, but for front end modules.
 64 | 6. Node.js - Changed my life, useful for servers or client side, or just quick file parsing.
 65 | 
 66 | 
 67 | 
 68 | * What are 5 tools node/backend developers should be using if they aren't already?
 69 | ---
 70 | [pd] In no particular order:
 71 | 1. Grunt - See above. It can do as much/little as you need in your project. I have a cheap little custom task which just greps your specified source files for copyright headers. Sounds dumb and simple (it is), but it keeps us honest.
 72 | 2. Git hooks - Run your Grunt/npm tasks before checking in or on each commit. There are a lot of great npm modules out there that let you simplify this as well.
 73 | 3. GitHub issues - If you see something, file something. TODOs in code are so 90s. Plus having everything in a bugbase makes it easier to track and collaborate.
 74 | 4. Travis - Run your tests automatically on a server and fail the build if anything goes wrong.
 75 | 5. Istanbul (or some other code coverage tool) - Having tests is awesome and required for ANY project. Finding out that all your fancy tests still only cover 31% of the code makes everybody sad. Find a code coverage library and set it up and run it in Travis so you can see your code coverage numbers on each run so we can track progress (positive or negative).
 76 | 6. CHANGELOG.md - Every project should have one! It's possible to get diffs between commits/tags/releases from GitHub without too much effort, but it's a lot better when you can quickly scan a changelog file to see what is different between your current version of a module and the latest version of a module in npm.
 77 | 7. David (CLI) - https://github.com/alanshaw/david - Extremely useful to quickly check a package.json's dependencies for out of date modules.
 78 | 8. package-json-validator.com - It's like PackageJsonLint!
 79 | 
 80 | 
 81 | 
 82 | * What would you like to see developers do more of?
 83 | ---
 84 | [pd] Bug triages. Public bug bases. I'm a huge fan of keeping a project's bug base alongside the code since it makes it extremely easy to look at code, file a bug, file a pull request to close the bug. Plus, GitHub lets me do inline animated GIFs to annoy my coworkers. Having a big list of issues is great, but not extremely useful unless you sit down as a team and prioritize them by milestone and severity so you can focus on the more important bugs first and newer developers/QA/community people can easily find the easier bugs to help contribute to a project.
 85 | 
 86 | Also, if you're using GitHub, use the issue labels to tag your issues as "GoodFirstBug" to make it easier for new developers to the project to quickly jump in and start contributing. Not to mention labeling all the bugs makes it much easier to quickly see all your open UX or testing issues.
 87 | Similarly, set up GitHub releases so you can easily specify when bugs are planned to be fixed.
 88 | 
 89 | 
 90 | 
 91 | * Other than write consistent, well-tested, bug-free code, what can a developer do to make your job easier?
 92 | ---
 93 | [pd] File more issues in GitHub. It can be tricky as a QA when you file a bug and a dev says "Oh yeah, we knew about that". Nothing worse than surprise bugs. Using "TODO" throughout your code is a poor replacement for a bug report where you can have discussions or arguments or GIF battles.
 94 | Plus, learn to design. Reading a book or two on UX wouldn't kill you people.
 95 | 
 96 | 
 97 | 
 98 | * What common misconceptions exist about QA?
 99 | ---
100 | [pd] That we're all just a bunch of nice people. A lot of us are just cranky old jerks.
101 | The main one is that it's NOT dev vs qa. We're all on the same team, and we all want the same thing (to gang up on the Marketing team).
102 | 
103 | 
104 | 
105 | * Any other thoughts?
106 | ---
107 | [pd] QA loves you. Lets go build beautiful things together. <3
108 | 
109 | 
110 | 


--------------------------------------------------------------------------------
/articles/01-fernando-jimenez-moreno.md:
--------------------------------------------------------------------------------
  1 | We write a lot of software at Mozilla. Some of the software is superb, some could use a rewrite. It's difficult to write software that "works". It's even more difficult to write software that works AND could be considered high quality.
  2 | 
  3 | I have spent my entire professional career trying to craft high quality software, and I still struggle. I look back at code that I wrote as little as six months ago and go WTF? I know that I have a lot to learn. Luckily, I am part of an amazing community that has a lot to teach. The Mozilla community builds amazing projects like MDN, Firefox and FirefoxOS; projects that are complex, huge in scope, developed over many years, and require the effort of thousands. These projects are only successful because of the skill of their participants.
  4 | 
  5 | Over the past couple of months, several community members have agreed to share their thoughts on the question "How can we, as developers, write more superb software?" This is the first of a series of interviews where I take on the role of an apprentice to learn from some of Mozilla's finest.
  6 | 
  7 | Since shipping software to millions of people involves more than just writing code, the interviews cross functional lines. Folks from QA, Security and development have all been interviewed, and I hope to cover additional areas in the future.
  8 | 
  9 | The target audience is anybody who writes software, regardless of the preferred language or level of experience. If you are reading this, you are part of the part of the target audience! I ask the interviewees high level questions that can be applied to any language, often the questions are about processes and tooling.
 10 | 
 11 | The conversations are free-flowing, but I have several specific questions that I am trying to here about:
 12 | 
 13 | * How do other developers approach writing high quality, maintainable software?
 14 | * What does "high quality, maintainable software" even mean?
 15 | * What are my peers looking for when doing code review?
 16 | * How does development/Security/QA fit together to support each other's efforts?
 17 | * What matters to Security? What do they look for when doing a security review/audit?
 18 | * What matters to QA? What do they look for before signing off on a release?
 19 | * What tools are people using?
 20 | * What can *I* do, as a developer, to write great software and facilitate the entire process?
 21 | 
 22 | I will present the highlights of one or two interviews per article. Each interview will contain a short introduction to the person being interviewed followed by a series of questions and answers.
 23 | 
 24 | Where an interview's audio was recorded, I will provide a link to the full transcript. If the interview was done over email, I will link to the contents of the original email.
 25 | 
 26 | Now, on to the first interview!
 27 | 
 28 | 
 29 | The first interview is with Fernando Jimenez Moreno, a FirefoxOS developer from Telefonica. I had the opportunity to work with Fernando last autumn when we integrated Firefox Accounts into First Time Experience of FirefoxOS. I was impressed not only with Fernando's technical prowess, but also his ability to bring together the employees of three companies in six countries on two continents to work towards a common goal.
 30 | 
 31 | Fernando talks about how Telefonica became involved in FirefoxOS, how to bring a disparate group together, common standards, code reviews, and above all, being pragmatic.
 32 | 
 33 | 
 34 | 
 35 | > S: What do you and your team at Telefonica do?
 36 | 
 37 | > F: I'm part of what we call the platform team. We have different teams at Telefonica, one is focused on front end development in Gaia, and the other one is focused on the platform itself, like Gecko, Gonk and external services. We work in several parts of FirefoxOS, from Gecko to Gaia, to services like the SimplePush server. I've personally worked on things like the [RIL interface layer](https://developer.mozilla.org/en-US/Firefox_OS/Platform/Architecture), payments, applications API and other Web APIs, and almost always jump from Gecko to Gaia and back. Most recently, I started working on the [Loop project](https://wiki.mozilla.org/CloudServices/Loop), which is Telefonica and Mozilla's WebRTC service.
 38 | 
 39 | > S: How did Telefonica get involved working with Mozilla?
 40 | 
 41 | > F: Well, that's a longer story. We started working on a similar project to FirefoxOS, but instead of being based on Gecko, we were working with WebKit. So we were creating this open web device platform based on WebKit. When we heard about Mozilla doing the same with Gecko, we decided to contact you and started working on the same thing. Our previous implementation was based on a closed source port of WebKit and it was really hard to work that way. Since then, my day-to-day work is just like any other member of Telefonica's FirefoxOS team, which I believe is pretty much the same as any other Mozilla engineer working on B2G.
 42 | 
 43 | > S: You are known as a great architect, developer, and inter-company coordinator. For Firefox Accounts on FirefoxOS, you brought together people from Telefonica, Telenor, and Mozilla. What challenges are present when you have to work across three different companies?
 44 | 
 45 | > F: It has given me quite a challenge and experience, especially during the first days of FirefoxOS. We started working with Mozilla back in 2011, and it took some time for both companies to find a common work flow that fit well for both parts. I mean, we were coming from a telco culture where many things were closed and confidential by default, as opposed to the openness and transparency of Mozilla. For some of us coming from other open source projects, it wasn't that hard to start working in the open and to be ready to discuss and defend our work on public forums. But, for other members of the team it took some time to get used to that new way of working, and new way of presenting our work. Also, because we were following agile methodologies in Telefonica while Mozilla wasn't still doing it, we had to find this common workflow that suits both parts. It took some time to do it, a lot of management meetings, a lot of discussions about it. Regarding working with other telco companies, the experience has also been quite good so far, especially with Telenor. We still have to be careful about the information that we share with them, because at the end of the day, we are still competitors. But that doesn't mean we cannot work with them in a common target like what happened with Firefox Accounts.
 46 | 
 47 | > S: It sounds like when Mozilla and Telefonica were both starting out on this process, both sides had to change. Some people at Telefonica had to get used to working in the open, and some people at Mozilla had to change their methodologies. You mentioned agile in particular. How did you decide what common practices to use and how did you establish a common culture to get everybody to go together?
 48 | 
 49 | > F: I think for this agile methodology, we focused more on the front end parts because Gecko already had a very known process and a very known way of developing. It has its own train mechanism of 6 weeks. As I said, we have two different teams in Telefonica, the platform team and the front end team. The ones doing the most, the biggest effort of finding that common workflow were the front end team because we started working on Gaia and Gaia was a new project with no fixed methodologies. They tried to find the common methodology for Gaia. In the meantime, the people working on the task were trying to learn the Gecko process and also we are in the middle between working with the Gaia team and working with Gecko engineers. I don't know if we really found the workflow, the perfect workflow, but I think we are doing good. I mean we participate in agile methodologies, but when it turns out that we need to do Gecko development and we need to focus on that, we just do it their way.
 50 | 
 51 | > S: In a multidisciplinary, multi-company project, how important are common standards like style guides, tools, and processes to making a project succeed?
 52 | 
 53 | > F: Well, I believe when talking about software engineering, standards are very important in general. But, I don't care if you call it SCRUM or KANBAN or SCRUMBAN or whatever, or if you use Git workflow or Mercurial workflow, or if you use Google or Mozilla's Javascript style guide. But you totally need some common processes and standards, especially in large engineering groups like open source, or Mozilla development in general. When talking about this, the lines are very thin. It's quite easy to fail spending too much time defining and defending the usage of these standards and common processes and losing the focus on the real target. So, I think we shouldn't forget these are only tools, they are important, but they are only tools to help us, and help our managers. We should be smart enough to be flexible about them when needed. We do a lot of code reviews about code style, but in the end what you want is to land the patch and to fix the issue. If you have code style issues, I want you to fix them, but if you need to land the patch to make a train, land it and file a follow on bug to fix the issues, or maybe the reviewer can do it if they have the chance.
 54 | 
 55 | > S: Whenever new people come into the project, how can you go about educating them about the standards that already exist?
 56 | 
 57 | > F: I would say, at first, documentation. I guess reviews and day-to-day work, where people are already familiar with these standards participate in this education process. There are also tools like linters or git and mercurial hooks to enforce some of these standards.
 58 | 
 59 | > S: FirefoxOS code is Gonk, which is Linux basically, Gecko and Gaia. Each system is large, complex, and intimidating to a newcomer. You regularly submit patches to Gecko and Gaia. You had to learn about all of these systems somehow. Whenever you want to go dive into a project, how do you learn about the system you want to work in?
 60 | 
 61 | > F: I'm afraid there is no magic technique. What works for me might not work for others for sure. What I try to do is to read as much documentation as possible inside and outside of the code, if it's possible. I try to ask the owners of that code when needed, and also if that's possible, because sometimes they just don't work in the same code or they are not available. I try to avoid reading the code line by line at first and I always try to understand the big picture before digging into the specifics of the code. I think that along the years, you somehow develop this ability to identify patterns in the code and to identify common architectures that help you understand the software problems that you are facing.
 62 | 
 63 | > S: When you start coding in unfamiliar territory, how do you ensure your changes don't cause unintended side effects? Is testing a large part of this?
 64 | 
 65 | > F: Yeah, basically tests, tests and more tests. You need tests, smoke tests, black box tests, tests in general. Also at first, you depend a lot on what the reviewer said, and you trust that the reviewer, or you can ask QA or the reviewer to add tests to the patch.
 66 | 
 67 | > S: Let's flip this on its head and you are the reviewer and you are reviewing somebody's code. Again, do you rely on the tests whenever you say "OK, this code adds this functionality. How do we make sure it doesn't break something over there?"
 68 | 
 69 | > F: I usually test the patches that I have review if I think the patch can cause any regression. I also try to run the tests on the "try" server, or ask the developer to trigger a "try" run.
 70 | 
 71 | > S: OK, so tests.. A lot of tests.
 72 | 
 73 | > F: Yeah, now that we are starting to have good tests in FirefoxOS, we have to make use of them.
 74 | 
 75 | > S: I feel like there is this tension between what level of test to write. Some people say "unit tests are worthless because if the interface changes then you have to go update all the tests and functional tests are where it's at." And other people are like "nooo, you've gotta unit test everything and functional tests won't tell you where the problem is, it'll only tell you there is problem." I would love to write huge numbers of both, but sometimes there is not enough time. I don't know if that's a false assumption on my part. How do you deal with testing in time constrained situations? Do you try to do both functional tests and unit tests? If you had to ditch one, where do you focus your testing efforts?
 76 | 
 77 | > F: I guess it depends on the time that you have for the coding part. I usually approach it, I think the wrong way. I write the code then write the tests, when I should probably do the opposite. I think it's this urge to see something working. Because that's actually your job to make the patch work, and then you need to make sure it's well written and it works well and it doesn't break so you write the tests for it. I don't know which one I would choose, it depends on the patch and the time that I have. I also ask "what would the reviewer ask me to write?" I mean, we all love tests, and we would all like to write as many tests as possible, but in the end we have a lot of patches to write and we just want the patch to be landed. We need to make the reviewer happy, if he wants this test, I will write this test. That's it.
 78 | 
 79 | > S: What do you look for when you are doing a review?
 80 | 
 81 | > F: In general where I look first is correctness. I mean, the patch should actually fix the issue it was written for. And of course it shouldn't have collateral effects. It shouldn't introduce a regression.  And as I said, I try to test the patches myself if I have the time or if the patch is critical enough, to see how it works and to see if it introduces a regression. And also I look that the code is performant and is secure, and also if, I always try to ask for tests if I think they are possible to write for the patch. And I finally look for things like quality of the code in general, and documentation, coding style, contribution, process correctness. If you have to squash your commits for pushing it to Gaia, then well, you have to do it. Or if you have to write the commit message with the reviewer name, or the bug number, then you have to do it because it's the process for that project.
 82 | 
 83 | > S: Whenever you are reviewing code, say it's a large patch, would you prefer that patch be broken up into smaller commits that show the developer's process, or do you like to see one giant thousand line patch?
 84 | 
 85 | > F: I prefer it to be split between different commits. I usually actually write the code that way. I do different patches with different numbers for each patch, that way I think you can even ask for review of different reviewers for each different part because sometimes you will write for B2G but it also involves Gonk parts, or Gecko parts, or Gaia parts, and the same reviewer cannot review the whole stack, so you need to split it into different pieces.
 86 | 
 87 | > S: And then say, you did have one of these patches that was 3 different commits. Before that is merged, would you squash those into one commit and then merge it?
 88 | 
 89 | > F: I usually ask the developer to do it, to squash it, or sometimes they do it themselves.
 90 | 
 91 | > S: You reviewed one of my large patches to integrate Firefox Accounts into FirefoxOS. You placed much more of an emphasis on consistency than any review I have had. By far.
 92 | 
 93 | > F: Well it certainly helps with overall code quality. When I do reviews, I mark these kinds of comments as "nit:" which is quite common in Mozilla, meaning that "I would like to see that changed, but you still get my positive review if you don't change it, but I would really like to see them changed."
 94 | 
 95 | > S: Two part question. As a reviewer, how can you ensure that your comments are not taken too personally by the developer? The second part is, as a developer, how can you be sure that you don't take it too personally?
 96 | 
 97 | > F: For the record, I have received quite a few hard revisions myself. I never take them personally. I mean, I always try to take it, the reviews, as a positive learning experience. I know reviewers usually don't have a lot of time to, in their life, to do reviews. They also have to write code. So, they just quickly write "needs to be fixed" without spending too much time thinking about the nicest ways to say it. Reviewers only say things about negative things in your code, not negative, but things that they consider are not correct. But they don't usually say that the things that are correct in your code and I know that can be hard at first. But once you start doing it, you understand why they don't do that. I mean, you have your work to do. This is actually especially hard for me, being a non-native English speaker, because sometimes I try to express things in the nicest way possible but the lack of works make the review comments sound stronger than it was supposed to be. And, what I try to do is use a lot of smileys if possible. And always, I try to avoid the "r-" flag if I mean, the "r-" is really bad. I just clear it, use the "feedback +" or whatever.
 98 | 
 99 | > S: You already mentioned that you try to take it as a learning experience whenever you are developer. Do you use review as a potential teaching moment if you are the reviewer?
100 | 
101 | > F: Yeah, for sure. I mean just the simple fact of reviewing a patch is a teaching experience. You are telling the coder what you think is more correct. Sometimes there is a lack of theory and reasons behind the comments, but we should all do that, we should explain the reasons and try to make the process as good as possible.
102 | 
103 | > S: Do you have a snippet of code, from you or anybody else, that you think is particularly elegant that others could learn from?
104 | 
105 | > F: I am pretty critical with my own code so I can't really think about a snippet of code of my own that I am particularly proud enough to show :). But if I have to choose a quick example I was quite happy with the result of the last big refactor of the [call log database](https://github.com/mozilla-b2g/gaia/blob/master/apps/communications/dialer/js/call_log_db.js) for the Gaia Dialer app or the recent [Mobile Identity API implementation](https://mxr.mozilla.org/mozilla-central/source/services/mobileid/).
106 | 
107 | > S: What open source projects would you like to encourage people to participate in, and where can they go to get involved?
108 | 
109 | > F: Firefox OS of course! No, seriously, I believe Firefox OS gives to software engineers the chance to get involved in an amazing open source community with tons of technical challenges from low-level to front end code. Having the chance to dig into the guts of a web browser and a mobile operative system in such an open environment is quite a privilege. It may seem hard at first to get involved and jump into the process and the code, but there are [already some very nice docs](https://developer.mozilla.org/en-US/Firefox_OS) at the MDN and a lot of nice people willing to help on [IRC (#b2g and #gaia)](https://wiki.mozilla.org/IRC), the mailing lists [(dev-b2g and dev-gaia)](https://lists.mozilla.org/listinfo) or [ask.mozilla.org](https://ask.mozilla.org/questions/).
110 | 
111 | > S: How can people keep up to date about what you are working on?
112 | 
113 | > F: I don't have a blog, but I have my [public Github account](https://github.com/ferjm) and my [Twitter account](https://twitter.com/f_jimenez).
114 | 
115 | 
116 | In the next article, I interview Brian Warner from the Cloud Services team. Brian is a security expert who shares his thoughts on architecting for security, analyzing threats, "belts and suspenders", and writing code that can be audited.
117 | 
118 | As a final note, I have had a lot of fun doing these interviews and I would like your input on how to make this series useful to you. If you have suggestions or would like to nominate somebody to be interviewed, please let me know! I even invite you to offer to have yourself interviewed. The Mozilla community has so much to offer. Email me at [stomlinson@mozilla.com](mailto:stomlinson@mozilla.com).
119 | 


--------------------------------------------------------------------------------
/articles/02-brian-warner.md:
--------------------------------------------------------------------------------
  1 | 
  2 | > "How can we, as developers, write more superb software?"
  3 | 
  4 | A simple question without a simple answer. Writing good code is hard, even for developers with years of experience. Luckily, the Mozilla community is made up of some of the best development, QA and security folks in the industry.
  5 | 
  6 | This is part two in a series of interviews where I take on the role of an apprentice to learn from some of Mozilla's finest.
  7 | 
  8 | ## Introducing Brian Warner
  9 | 
 10 | When my manager and I first discussed this project, Brian is the first person I wanted to interview. Brian probably doesn't realize it, but he has been one of my unofficial mentors since I started at Mozilla. He is an exceptional teacher, and has the unique ability to make security approachable.
 11 | 
 12 | At Mozilla, Brian designed the pairing protocol for the "old" Firefox Sync, designed the protocol for the "new" Firefox Sync, and was instrumental in keeping Persona secure. Outside of Mozilla, Brian co-founded the [Tahoe-LAFS](https://tahoe-lafs.org/trac/tahoe-lafs/) project, and created [Buildbot](http://buildbot.net/).
 13 | 
 14 | ### What do you do at Mozilla?
 15 | 
 16 | My title is a staff security engineer in the Cloud Services group. I analyse and develop protocols for securely managing passwords and account data and I implement those protocols in different fashions. I also review other's code, I look at external projects to figure out whether it's appropriate to incorporate them, and I try to stay on top of security failures like 0-days and problems in the wild that might affect us and also tools and algorithms that we might be able to use.
 17 | 
 18 | 
 19 | ### UX vs Security: Is it a false dichotomy? Some people have the impression that for security to be good, it must be difficult to use.
 20 | 
 21 | There are times when I think that it's a three-way tradeoff. Instead of being x-axis, y-axis, and a diagonal line that doesn't touch zero, sometimes I think it's a three-way thing where the other axis is how much work you want to put into it or how clever you are or how much user research and experimentation you are willing to do. Stuff that engineers are typically not focused on, but that UX and psychologists are. I believe, maybe it's more of a hope than a belief, that if you put enough effort into that, then you can actually find something that is secure and usable at the same time, but you have to do a lot more work.
 22 | 
 23 | The trick is to figure out what people want to do and find a way of expressing whatever security decisions they have to make into a normal part of their work flow. It's like when you lend your house key to a neighbour so they can water your plants when you are away on vacation, you've got a pretty good idea of what power you are handing over.
 24 | 
 25 | There are some social constructs surrounding that like, "I don't think you're going to make a copy of that key and so when I get it back from you, you no longer have that power that I granted to you." There are patterns in normal life with normal non-computer behaviours and objects that we developed some social practices around, I think part of the trick is to use that and assume that people are going to expect something that works like that and then find a way to make the computer stuff more like that.
 26 | 
 27 | Part of the problem is that we end up asking people to do very unnatural things because it is hard to imagine or hard to build something that's better. Take passwords. Passwords are a lousy authentication technology for a lot of different reasons. One of them being that in most cases, to exercise the power, you have to give that power to whoever it is you are trying to prove to. It's like, "let me prove to you I know a secret"..."ok, tell me the secret." That introduces all these issues like knowing how to correctly identify who you are talking to, and making sure nobody else is listening.
 28 | 
 29 | In addition to that, the best passwords are going to be randomly generated by a computer and they are relatively long. It's totally possible to memorize things like that but it takes a certain amount of exercise and practice and that is way more work than any one program deserves.
 30 | 
 31 | But, if you only have one such password and the only thing you use it on is your phone, then your phone is now your intermediary that manages all this stuff for you, and then it probably would be fair (to ask users to spend more energy managing that password). And it's clear that your phone is sort of this extension of you, better at remembering things, and that the one password you need in this whole system is the bootstrap.
 32 | 
 33 | So some stuff like that, and other stuff like escalating effort in rare circumstances. There are a lot of cases where what you do on an everyday basis can be really easy and lightweight, and it's only when you lose the phone that you have to go back to a more complicated thing. Just like you only carry so much cash in your wallet, and every once in a while you have to go to a bank and get more.
 34 | 
 35 | It's stuff like that I think it's totally possible to do, but it's been really easy to fall into bad patterns like blaming the user or pushing a lot of decisions onto the user when they don't really have enough information to make a good choice, and a lot of the choices you are giving them aren't very meaningful.
 36 | 
 37 | ### Do you think many users don't understand the decisions and tradeoffs they are being asked to make?
 38 | 
 39 | I think that's very true, and I think most of the time it's an inappropriate question to ask. It's kind of unfair. Walking up to somebody and putting them in this uncomfortable situation - do you like X or do you like Y - is a little bit cruel.
 40 | 
 41 | Another thing that comes to mind is permission dialogs, especially on Windows boxes, where to do a bunch of really basic stuff that's useful, and it's a valid thing to do, it's not like you are trying to do something crazy, you have to accept these demands that are basically saying, "bad things will happen and it's all going to be your fault."
 42 | 
 43 | It's intended to give the user an informed choice, but it is this kind of blame the user, blame the victim pattern, where it's like "something bad happened, but you clicked on the OK button, you've taken responsibility for that." The user didn't have enough information to do something and the system wasn't well enough designed that they could do what they wanted to do without becoming vulnerable.
 44 | 
 45 | ### Months before "new" Sync ever saw the light of day, the protocol was hashed out in extremely vocal and public forum. It was the exact opposite of security through obscurity. What did you hope to accomplish?
 46 | 
 47 | There were a couple of different things that I was hoping from that discussion. I pushed all that stuff to be described and discussed publicly because it's the right thing to do, it's the way we develop software, you know, it's the open source way. And so I can't really imagine doing it any other way.
 48 | 
 49 | The specific hopes that I had for publishing that stuff was to try to solicit feedback and get people to look for basic design flaws. I wanted to get people comfortable with the security properties, especially because new Sync changes some of them. We are switching away from pairing to something based on passwords. I wanted people to have time to feel they understood what those changes were and why we were making them. We put the design criteria and the constraints out there so people could see we kind of have to switch to a password to meet all of the other goals, and what's the best we can do given security based on passwords.
 50 | 
 51 | Then the other part is that having that kind of public discussion and getting as many experienced people involved as possible is the only way that I know of to develop confidence that we're building something that's correct and not broken.
 52 | 
 53 | So it is really just more eyeballs&hellip;
 54 | 
 55 | ### Before a protocol or API designer ever sits down and writes a spec or line of code, what should they be thinking about?
 56 | 
 57 | I'd say think about what your users need. Boil down what they are trying to accomplish into something minimal and pretty basic. Figure out the smallest amount of code, the smallest amount of power, that you can provide that will meet those needs.
 58 | 
 59 | ### This is like the agile version of developing a protocol.
 60 | 
 61 | Yeah. Minimalism is definitely useful. Once you have the basic API that enables you to do what needs to be done, then think about all of the bad things that could be done with that API. Try and work out how to prevent them, or make them too expensive to be worthwhile.
 62 | 
 63 | A big problem with security is sometimes you ask "what are the chances that problem X would happen." If you design something and there is a 1/1000 chance that something will happen, that the particular set of inputs will cause this one particular problem to happen. If it really is random, then 1/1000 may be ok, 1/1M may be ok, but if it is in this situation where an attacker gets to control the inputs, then it's no longer 1/1000, it's 1 in however many times the attacker chooses to make it 1.
 64 | 
 65 | It's a game of who is cleverer and who is more thorough. It's frustrating to have to do this case analysis to figure out every possible thing that could happen, every state it could get into, but if somebody else out there is determined to find a hole, that's the kind of analysis they are going to do. And if they are more thorough than you are, then they'll find a problem that you failed to cover.
 66 | 
 67 | ### Is this what is meant by threat modelling?
 68 | 
 69 | Yeah, different people use the term in different ways, I think of when you are laying out the system, you are setting up the ground rules. You are saying there is going to be this game. In this game, Alice is going to choose a password and Bob is trying to guess her password, and whatever.
 70 | 
 71 | You are defining what the ground rules are. So sometimes the rules say things like &hellip; the attacker doesn't get to run on the defending system, their only access is through this one API call, and that's the API call that you provide for all of the good players as well, but you can't tell the difference between the good guy and the bad guy, so they're going to use that same API.
 72 | 
 73 | So then you figure out the security properties if the only thing the bad guy can do is make API calls, so maybe that means they are guessing passwords, or it means they are trying to overflow a buffer by giving you some input you didn't expect.
 74 | 
 75 | Then you step back and say "OK, what assumptions are you making here, are those really valid assumptions?" You store passwords in the database with the assumption that the attacker won't ever be able to see the database, and then some other part of the system fails, and whoops, now they can see the database. OK, roll back that assumption, now you assume that most attackers can't see the database, but sometimes they can, how can you protect the stuff that's in the database as best as possible?
 76 | 
 77 | Other stuff like, "what are all the different sorts of threats you are intending to defend against?" Sometimes you draw a line in the sand and say "we are willing to try and defend against everything up to this level, but beyond that you're hosed." Sometimes it's a very practical distinction like "we could try to defend against that but it would cost us 5x as much."
 78 | 
 79 | Sometimes what people do is try and estimate the value to the attacker versus the cost to the user, it's kind of like insurance modelling with expected value. It will cost the attacker X to do something and they've got an expected gain of Y based on the risk they might get caught.
 80 | 
 81 | Sometimes the system can be rearranged so that incentives encourage them to do the good thing instead of the bad thing. Bitcoin was very carefully thought through in this space where there are these clear points where a bad guy, where somebody could try and do a double spend, try and do something that is counter to the system, but it is very clear for everybody including the attacker that their effort would be better spent doing the mainstream good thing. They will clearly make more money doing the good thing than the bad thing. So, any rational attacker will not be an attacker anymore, they will be a good participant.
 82 | 
 83 | ### How can a system designer maximise their chances of developing a reasonably secure system?
 84 | 
 85 | I'd say the biggest guideline is the [Principle of Least Authority](https://en.wikipedia.org/wiki/Principle_of_least_privilege). POLA is sometimes how that is expressed. Any component should have as little power as necessary to do the specific job that it needs to do. That has a bunch of implications and one of them is that your system should be built out of separate components, and those components should actually be isolated so that if one of them goes crazy or gets compromised or just misbehaves, has a bug, then at least the damage it can do is limited.
 86 | 
 87 | The example I like to use is a decompression routine. Something like gzip, where you've got bytes coming in over the wire, and you are trying to expand them before you try and do other processing. As a software component, it should be this isolated little bundle of 2 wires. One side should have a wire coming in with compressed bytes and the other side should have decompressed data coming out. It's gotta allocate memory and do all kinds of format processing and lookup tables and whatnot, but, nothing that box can do, no matter how weird the input, or how malicious the box, can do anything other than spit bytes out the other side.
 88 | 
 89 | It's a little bit like Unix process isolation, except that a process can do syscalls that can trash your entire disk, and do network traffic and do all kinds of stuff. This is just one pipe in and one pipe out, nothing else. It's not always easy to write your code that way, but it's usually better. It's a really good engineering practice because it means when you are trying to figure out what could possibly be influencing a bit of code you only have to look at that one bit of code. It's the reason we discourage the use of global variables, it's the reason we like object-oriented design in which class instances can protect their internal state or at least there is a strong convention that you don't go around poking at the internal state of other objects. The ability to have private state is like the ability to have private property where it means that you can plan what you are doing without potential interference from things you can't predict. And so the tractability of analysing your software goes way up if things are isolated. It also implies that you need a memory safe language&hellip;
 90 | 
 91 | Big, monolithic programs in a non memory safe language are really hard to develop confidence in. That's why I go for higher level languages that have memory safety to them, even if that means they are not as fast. Most of the time you don't really need that speed. If you do, it's usually possible to isolate the thing that you need, into a single process.
 92 | 
 93 | ### What common problems do you see out on the web that violate these principles?
 94 | 
 95 | Well in particular, the web is an interesting space. We tend to use memory safe languages for the receiver.
 96 | 
 97 | ### You mean like Python and JavaScript.
 98 | 
 99 | Yeah, and we tend to use more object-oriented stuff, more isolation. The big problems that I tend to see on the web are failure to validate and sanitize your inputs. Or, failing to escape things like injection attacks.
100 | 
101 | ### You have a lot of experience reviewing already written implementations, Persona is one example. What common problems do you see on each of the front and back ends?
102 | 
103 | It tends to be escaping things, or making assumptions about where data comes from, and how much an attacker gets control over if that turns out to be faulty.
104 | 
105 | ### Is this why you advocated making it easy to trace how the data flows through the system?
106 | 
107 | Yeah, definitely, it'd be nice if you could kind of zoom out of the code and see a bunch of little connected components with little lines running between them, and to say, "OK, how did this module come up with this name string? Oh, well it came from this one. Where did it come from there? Then trace it back to the point where, HERE that name string actually comes from a user submitted parameter. This is coming from the browser, and the browser is generating it as the sending domain of the postMessage. OK, how much control does the attacker have over one of those? What could they do that would be surprising to us? And then, work out at any given point what the type is, see where the transition is from one type to another, and notice if there are any points where you are failing to do that, that transformation or you are getting the type confused. Definitely, simplicity and visibility and tractable analysis are the keys.
108 | 
109 | ### What can people do to make data flow auditing simpler?
110 | 
111 | I think, minimising interactions between different pieces of code is a really big thing. Isolate behaviour to specific small areas. Try and break up the overall functionality into pieces that make sense.
112 | 
113 | ### What is defence in depth and how can developers use it in a system?
114 | 
115 | "Belt and suspenders" is the classic phrase. If one thing goes wrong, the other thing will protect you. You look silly if you are wearing both a belt and suspenders because they are two independent tools that help you keep your pants on, but sometimes belts break, and sometimes suspenders break. Together they protect you from the embarrassment of having your pants fall off. So [defence in depth](http://en.wikipedia.org/wiki/Defense_in_depth_%28computing%29) usually means don't depend upon perimeter security.
116 | 
117 | ### Does this mean you should be checking data throughout the system?
118 | 
119 | There is always a judgement call about performance cost, or, the complexity cost. If your code is filled with sanity checking, then that can distract the person who is reading your code from seeing what real functionality is taking place. That limits their ability to understand your code, which is important to be able to use it correctly and satisfy its needs. So, it's always this kind of judgement call and tension between being too verbose and not being verbose enough, or having too much checking.
120 | 
121 | The notion of perimeter security, it's really easy to fall into this trap of drawing this dotted line around the outside of your program and saying "the bad guys are out there, and everyone inside is good" and then implementing whatever defences you are going to do at that boundary and nothing further inside. I was talking with some folks and their opinion was that there are evolutionary biology and sociology reasons for this. Humans developed in these tribes where basically you are related to everyone else in the tribe and there are maybe 100 people, and you live far away from the next tribe. The rule was basically if you are related to somebody then you trust them, and if you aren't related, you kill them on sight.
122 | 
123 | That worked for a while, but you can't build any social structure larger than 100 people. We still think that way when it comes to computers. We think that there are "bad guys" and "good guys", and I only have to defend against the bad guys. But, we can't distinguish between the two of them on the internet, and the good guys make mistakes too. So, the principal of least authority and the idea of having separate software components that are all very independent and have very limited access to each other means that, if a component breaks because somebody compromised it, or somebody tricked it into behaving differently than you expected, or it's just buggy, then the damage that it can do is limited because the next component is not going to be willing to do that much for it.
124 | 
125 | ### Do you have a snippet of code, from you or anybody else, that you think is particularly elegant that others could learn from?
126 | 
127 | I guess one thing to show off would be the [core share-downloading](https://github.com/tahoe-lafs/tahoe-lafs/blob/bc6304ae701b90e7bccccc628afa42587fcd7fe5/src/allmydata/immutable/downloader/share.py#L245) loop I wrote for [Tahoe-LAFS](https://tahoe-lafs.org/trac/tahoe-lafs).
128 | 
129 | In Tahoe, files are uploaded into lots of partially-redundant "shares", which are distributed to multiple servers. Later, when you want to download the file, you only need to get a subset of the shares, so you can tolerate some number of server failures.
130 | 
131 | The shares include a lot of integrity-protecting [Merkle hash trees](http://en.wikipedia.org/wiki/Merkle_tree) which help verify the data you're downloading. The locations of these hashes aren't always known ahead of time (we didn't specify the layout precisely, so alternate implementations might arrange them differently). But we want a fast download with minimal round-trips, so we guess their location and fetch them speculatively: if it turns out we were wrong, we have to make a second pass and fetch more data.
132 | 
133 | This code tries very hard to fetch the bare minimum. It uses a set of compressed bitmaps that record which bytes we want to fetch (in the hopes that they'll be the right ones), which ones we really need, and which ones we've already retrieved, and sends requests for just the right ones.
134 | 
135 | The thing that makes me giggle about this overly clever module is that the entire algorithm is designed around Rolling Stone lyrics. I think I started with "You can't always get what you want, but sometimes &hellip; you get what you need", and worked backwards from there.
136 | 
137 | The other educational thing about this algorithm is that it's *too* clever: after we shipped it, we found out it was actually *slower* than the less-sophisticated code it had replaced. Turns out it's faster to read a few large blocks (even if you fetch more data than you need) than a huge number of small chunks (with network and disk-IO overhead). I had to run a big set of performance tests to characterize the problem, and decided that next time, I'd find ways to measure the speed of a new algorithm before choosing which song lyrics to design it around. :).
138 | 
139 | ### What open source projects would you like to encourage people to get involved with?
140 | 
141 | Personally, I'm really interested in secure communication tools, so I'd encourage folks (especially designers and UI/UX people) to look into tools like [Pond](https://github.com/agl/pond), [TextSecure](https://github.com/WhisperSystems/TextSecure), and my own [Petmail](https://github.com/warner/petmail). I'm also excited about the variety of run-your-own-server-at-home systems like the [GNU FreedomBox](https://wiki.debian.org/FreedomBox).
142 | 
143 | ### How can people keep up with what you are doing?
144 | 
145 | Following my commits on [https://github.com/warner](https://github.com/warner) is probably a good approach, since most everything I publish winds up there.
146 | 
147 | ### Thank you Brian.
148 | 
149 | ## Transcript
150 | 
151 | Brian and I covered far more material than I could include in a single post. [The full transcript](https://github.com/shane-tomlinson/devs-at-work-software-quality/blob/master/interviews/brian-warner.txt), available on GitHub, also covers mememory safe languages, implicit type conversions when working with HTML, and the Python tools that Brian commonly uses.
152 | 
153 | ## Next up!
154 | 
155 | Both Yvan Boiley and Peter deHaan are presented in the next article. Yvan leads the Cloud Services Security Assurance team and continues with the security theme by discussing his team's approach to security audits and which tools developers can use to self-audit their site for common problems.
156 | 
157 | Peter, one of Mozilla's incredible Quality Assurance engineers, is responsible for ensuring that Firefox Accounts doesn't fall over. Peter talks about the warning signs, processes and tools he uses to assess a project, and how to give the smack down while making people laugh.
158 | 


--------------------------------------------------------------------------------
/interviews/fernando-jimenez-moreno.txt:
--------------------------------------------------------------------------------
  1 | What do you and your team at Telefonica do?
  2 | 
  3 | So, I’m part of the, what we call the platform team. We have different teams at Telefonica, one is focused on front end development in Gaia, and the other one is focused on the platform itself, like Gecko, Gonk and maybe external services. So we work in several parts of the FirefoxOS stuff, from Gecko to Gaia, to services like the SimplePush server, which telefonica also have their own implementation. I’ve personally worked on things like the REL interface layer, payments, applications API and other Web APIs, and almost always jumping from Gecko to Gaia and back. Most recently, I started working on this Loop project, which like I said, is Telefonica and Mozilla’s WebRTC service.
  4 | 
  5 | Am I allowed to mention Loop in the article?
  6 | 
  7 | Yeah, I think so.
  8 | 
  9 | So, you actually mentioned, Gecko, Gaia, and a lot of the other projects you guys are working with.  How do you work with Mozilla, and how did Telefonica get involved working with Mozilla?
 10 | 
 11 | Well, that’s a longer story. How did we get involved? Well, we started working on a similar project as FirefoxOS, but instead of being based on Gecko, we were working with WebKit. So we were creating this open web device platform based on webkit. When we heard about Mozilla doing the same with Gecko, we decided to contact you and started working on the same thing. Our previous implementation was based on a closed source port of webkit and it was really hard to work that way. Since then, my day to day work is just like any other member of Telefonica’s FirefoxOS team, which I believe is pretty much the same as any other Mozilla engineer working on B2G. I mean we have a lot of bug mail, reviews, patches to write, etc. Personally, because I’m not part of any FirefoxOS functional team, I am what is called at some point a “free agent”. I have to interact with different teams so I don’t have a fixed schedules of meetings per week. Sometimes I do weekly meetings with some teams and other weeks, I have meetings with other teams.
 12 | 
 13 | You are a free agent, so you float between groups? Dude, that’s awesome! How did you pull that off?
 14 | 
 15 | Yeah, you can learn a lot floating between teams. It’s hard to context switch, but it’s really educational and different.
 16 | 
 17 | That’s really interesting, I have more questions about that because it’s a side project that’s interesting to me. You are known as a great architect, developer, and inter-company coordinator. For Firefox Accounts on FirefoxOS, you brought together people from Telefonica, Telenor, and Mozilla. What do you think are challenges, what challenges are present when you have to work across three different companies that aren’t present when you just have to work within your own company or group even?
 18 | 
 19 | It has given me quite a challenge and experience, especially during the first days of FirefoxOS. We started working with Mozilla back in 2011, and it took some time for both companies to find a common work flow that fit well for both parts. I mean, we were coming from a telco culture where many things were closed and confidential by default, as opposed to the openness and transparency of Mozilla. For some of us coming from other open source projects, it wasn’t that hard to start working in the open and to be ready to discuss and defend our work on public forums. But, for other members of the team it took some time to get used to that new way of working, and new way of presenting our work. Also, because we were following agile methodologies in Telefonica while Mozilla wasn’t still doing it, we had to find this common workflow that suits both parts. It took some time to do it, a lot of management meetings, a lot of discussions about it. Regarding working with other telco companies, the experience has also been quite good so far, especially with Telenor. We still have to be careful about the information that we share with them, because at the end of the day, we are still competitors. But that doesn’t mean we cannot work with them in a common target like what happened with Firefox Accounts. I personally work them in Amsterdam, in their office. They are two persons here in Amsterdam, it’s a small office.
 20 | 
 21 | Sergi, and who else is there?
 22 | 
 23 | Jan.
 24 | 
 25 | I have not met him or her.
 26 | 
 27 | They are a peer of the keyboard application in Gaia, and they allow me to work with them in the office.
 28 | 
 29 | That’s really interesting.
 30 | 
 31 | Sometimes I have to exit the room when I have a call with my managers and I have to speak about confidential stuff.
 32 | 
 33 | Do they ever ask you to leave the room and say “hey, we’ve got some confidential stuff we’ve gotta do. Get out!”
 34 | 
 35 | Not yet, but they probably will at some point.
 36 | 
 37 | It sounds like basically in 2011 and earlier on, when Mozilla and Telefonica were both starting out on this process, both sides had to change a bit. You know, some people at Telefonica had to get used to working in the open, and some people at Mozilla had to change their methodologies. You mentioned agile in particular. I feel like, I dunno, this is only my impression, and it’s a far off impression because I don’t have a lot of experience, that things like Gecko. They are on these very long development processes. Even 6 works is quite long in a normal agile world of 1 week or 2 week sprints. How did you guys decide what common practices to come up with? Did you just hash it out in rooms or did you just say “hey, maybe we should try this?” How did you come up with this common culture to get everybody to go together instead of getting everybody working against each other?
 38 | 
 39 | I think for this agile methodology, we focused more on the front end parts because as you said, Gecko already has a very known process and a very known way of developing. It has it’s own train mechanism of 6 weeks. As I said, we have two different teams in Telefonica, the platform team and the front end team. The ones doing the most, the biggest effort of finding that common workflow were the front end team because we started working on Gaia and Gaia was a new project with no fixed methodologies. They tried to find the common methodology for Gaia. In the meantime, the people working on the task were trying to learn the Gecko process and also we are in the middle between working with the Gaia team and working with Gecko engineers. I don’t know if we really found the workflow, the perfect workflow, but I think we are doing good. I mean we participate in agile methodologies, but when it turns out that we need to do Gecko development and we need to focus on that, we just do it their way.
 40 | 
 41 | Has Gecko development, do you think has changed since B2G has been introduced? Are they becoming a little bit more flexible, or have they modified their process?
 42 | 
 43 | Gecko came before we started working on FirefoxOS, but what I can see looking at the different teams is that B2G development in Gecko is quite different from Desktop development. Especially the review process, we have, I think, different standards for each part. I don’t know if it has changed, but I can see differences in teams at least.
 44 | 
 45 | What are the biggest differences between the Gecko team and the Gaia team do you think? Would it just be the train schedules, or you just mentioned the standards for review. How are they different?
 46 | 
 47 | For certain the Gecko team works with Mercurial, different development process. Gaia started with Github and Github issues, then they moved to Bugzilla so they finally had a more similar development process as Gecko using Bugzilla as the common issue tracker. But, I think when they did this FirefoxOS functional teams, the development process started to be more similar from the desktop team to the B2G team. What I said about differences, I mean differences in the code, the way they review the code, we have different standards in B2G and different standards in Gecko, for the code specific stuff.
 48 | 
 49 | That’s interesting, I skipped a question, but it touches on what you were just saying. In a multi-disciplinary project, a multi-company project, how important do you think things like common standards, like style guides and common processes, you just mentioned the bug tracker in particular moving from Github to Bugzilla. How important do you think common, common process and things like that are, and especially style guides, tools, how important are they to making a project succeed?
 50 | 
 51 | Well, I believe when talking about software engineering, standards are very important in general. But, I don’t care if you call it SCRUM or KANBAN or SCRUMBAN or whatever, or if you, that Git workflow or Mercurial workflow, or if you use Google or Mozilla’s Javascript style guide, but totally need some common processes and standards, especially in large engineering groups like open source, or Mozilla development in general. but when talking about this the lines are very thin. It’s quite easy to fail spending too much time defining and defending the usage of these standards and common processes and losing the focus on the real target. So, I think shouldn’t forget these are only tools, they are important, but they are only tools to help us, and help our managers. We should be smart enough to be flexible about them when needed. We do a lot of code reviews about code style, but in the end what you want is to land the patch and to fix the issues. If you have code style issues, I want you to fix them, but if you need to land the patch to make a train, land it and file a follow on bug to fix the issues, or maybe the reviewer can do it if they have the chance.
 52 | 
 53 | I actually do that quite often on our projects over on GitHub. I’ll just basically, if I have a couple of nits about something, I’ll just, you know, add a commit to their pull request and say “yeah, good to go. Merge it!” So, for newcomers. I was the newcomer in October to B2G and I had to be educated on some of these standards and I don’t know if I was educated so well, or I just refused to be educated and whatnot. Whenever new people come into the project, how can you go about educating them about the standards that already exist?
 54 | 
 55 | I would say, at first, documentation. But, you probably suffered the lack of documentation about it. So, I guess reviews and day to day work, where people are already familiar with these standards participate in this education process. There are also tools like linters or git and mercurial hooks to enforce some of these standards.
 56 | 
 57 | I would actually say documentation was not the problem that I had in October. I found there was too.. Lack of documentation was not the problem, there was too much documentation and I could not filter out what was meaningful to me, or what was applicable to what I needed to do, because there was SO much about B2G development, and you know, flashing phones, there’s like a million different phone pages, and I’m just like, “I don’t even know where to start.” That was one of my issues with FirefoxOS development when I tried it for those, that month and a half or two months.  So, FirefoxOS code is what,  Gonk, which is linux basically, Gecko and Gaia. Each system is quite large, complex, and intimidating to a newcomer. I feel Gecko in particular is intimidating. You regularly submit patches to Gecko and Gaia. You had to go about learning all of these systems somehow. Whenever you want to go dive into a project, you have a task to do, how do you go about learning the system you are getting ready to dive into?
 58 | 
 59 | I’m afraid there is no magic technique. What works for me might not work for others for sure. So, what I try to do is to read as much documentation as possible inside and outside of the code, if it’s possible. I try to ask the owners of that code when needed, and also if that’s possible, because sometimes they just don’t work in the same code or they are not available. I try to avoid reading the code line by line at first and I always try to understand the big picture before digging into the specifics of the code. I think that along the years, you somehow develop this ability to identify patterns in the code and to identify common architectures that help you understand the software problems that you are facing. You can see that just by doing a diagonal read of the code. I mean, we have this colleague at Telefonica that used to study in, I think it was Sweden, he did a research project about people with experience in development and new comers to development work. They used to record their eyes when they read code, and for new comers, they were reading line by line.
 60 | 
 61 | I saw, I could swear I saw an article along these lines in Slashdot maybe six months ago, and I found it fascinating.
 62 | 
 63 | Yeah, it’s really cool. So, people with experience in programming did, easily do a diagonal read of the code. It’s just like “scrolling, scrolling”, looking for the function names or common patterns. I guess it’s that way. You develop that ability, and you try to focus on the big picture. In the end, you can also use old fashioned debugger techniques like printing to the console.
 64 | 
 65 | That’s basically how I approach things as well. I would say my biggest, where I suffer the most is not reading the documentation beforehand. Sometimes I’ll try to find where the “main” would be if it were a C program, and then try to fit the large blocks together from there. And yeah, I should read the documentation more. There’s one learning thing for me already.
 66 | 
 67 | Yeah, asking sometimes helps a lot, unfortunately it is hard for people to just jump into IRC and ask a question like “hey, I’m looking at that code, can you guide me with a few words?” That really helps.
 68 | 
 69 | I did an interview last week with Yvan Boiley, I don’t know if you met him yet, he’s a security engineer from Mozilla. He was saying whenever they do security reviews on high risk projects, or projects that have the ability to cause lots of damage if they go wrong, the first thing they look for is documentation. He said the reason the first thing they look for is documentation,  he said, usually the better documented the code, the fewer security problems they end up finding. He says what happens is if developers take the time to write the documentation, they have usually taken the time to actually think through how data flows through the whole system. Developers that don’t have any documentation, generally haven’t put as much thought into it. He said basically, if there is no documentation, we have to assume the absolute worst about the code, and it’s going to be a freaking disaster and there is going to be security problems everywhere.
 70 | 
 71 | Yeah, that’s quite true, and I think the extrapolation to that theory that can be applied to tests. I mean tests can serve somewhat as an index and a guide to what the code does. When you do a review, I usually start by looking at the tests to see what the coder has thought, and what edge cases has he considered to write the actual code. I think it’s a common approach, if the developer has spent the time in the documentation, it means probably the code is well written.
 72 | 
 73 | So here’s a question, something that I wrote a couple of weeks ago. I was writing a little email about how I do code review. It talked in particular, I wrote in particular about tests. One of the red flags whenever I am reviewing a particular patch, because I go straight to tests quite often as well, and if they modify existing tests, I put a higher level of scrutiny into the code review than if they only add new tests, if that makes sense. Because if it modifies existing behaviour, then I wonder “Is there a potential regression here from the existing behaviour?” If it’s only additive, then I can be quite sure that at least the current, you know the current behaviours are still in tact and hopefully it’s only adding new behaviours. But if it modifies existing behaviours, then I have to go, I have to go talk to the designers and say “hey, is this intended, is this how it’s supposed to work?” and it does get a bit more scrutiny then.
 74 | 
 75 | Yeah, that makes a lot of sense. When you see a test fix, something is going on. Maybe it was wrong, but you have to be careful about modifying tests.
 76 | 
 77 | When you start coding in unfamiliar territory, say the first time you jumped into Gecko, you know. How do you ensure your changes don’t cause unintended side effects? Is testing a large part of this?
 78 | 
 79 | Yeah, basically tests, tests and more tests. You need tests, smoke tests, black box tests, tests in general. Also at first, you depend a lot on what the reviewer said, and you trust that the reviewer, or you can ask QA or the reviewer to add tests to the patch.
 80 | 
 81 | So, say let’s flip this on its head and you are the reviewer and you are reviewing somebodies code. Again, do you rely on the tests whenever you say “OK, this code adds this patch, this functionality. How do we make sure it doesn’t break something over there?”
 82 | 
 83 | I usually test the patches that I have review if I think the patch can cause any regression. I also try and run the tests on the “try” server, or ask the developer to trigger a “try” run.
 84 | 
 85 | OK, so tests.. A lot of tests.
 86 | 
 87 | Yeah, now that we are starting to have good tests in FirefoxOS, we have to make use of them.
 88 | 
 89 | So, I know this is off on a bit of a tangent, but I’m super for testing. I’m always, I feel ike there is this tension between what level of test to write. Because some people are like “unit tests are worthless because if the interface changes then you have to go update all the tests and functional tests are where it’s at.” And other people are like “nooo, you’ve gotta unit test everything and functional tests won’t tell you where the problem is, it’ll only tell you there is problem in your behaviour.” And I would love to write huge amounts of tests in both areas, but sometimes it feels like there is not enough time to do it. I don’t know if that’s a false assumption on my part. How do you deal with testing in time constrained situations I suppose. Do you try to do both functional tests and unit tests? Or if you had to ditch one, where do you focus your testing efforts I suppose is the simple question here.
 90 | 
 91 | I guess it depends on the time that you have for the coding part. I usually approach it, I think the wrong way. I write the code then write the tests, when I should probably do the opposite.
 92 | 
 93 | I do the same thing.
 94 | 
 95 | Yeah, but..
 96 | 
 97 | Unless it’s a really hairy API that I’m really trying to figure out, then I’ll end up writing the code first, then the tests.
 98 | 
 99 | I think it’s this urge to see something working. Because that’s actually your job to make the patch work, and then you need to make sure it’s well written and it works well and it doesn’t break so you write the tests for it. But, I guess, I don’t know which tests I would choose first. I mean functional tests are done to assure that your code works as suspected, but you can also rely on your personal tests on the device, and your manual tests. So you can probably avoid writing functional tests at first and focus on unit tests that tests only specific functionality of the code, specific parts of the code. I don’t know which one I would choose, it depends on the patch and the time that I have. I also ask “what would the reviewer ask me to write?”
100 | 
101 | That’s completely fair, completely completely fair.
102 | 
103 | I mean, we all love tests, and we would all like to write as much tests as possible, but in the end we have a lot of patches to write and we just want the patch to be landed. We need to make the reviewer happy, if he wants this test, I will write this test. That’s it.
104 | 
105 | I completely understand. So, man, I have way more questions than I thought I would. On a resource constrained device like a low end FirefoxOS phone, like my ZTE Open for instance. I’m always getting out of memory errors with 1.3, you know. So memory management is something developers have not really had to worry about since Internet Explorer was losing circular references back in IE6 and IE7. Now that we have to worry about this stuff again, has this focus on memory management changed the way that you write apps, and if so, how? And, has it changed the way you write normal web apps for a normal browser. Say you are just writing tokbox here.
106 | 
107 | When you write the web apps, you don’t really need to worry about memory management in terms of it’s hard to leak memory when writing a web application. You trust the browser trying to do the right thing, but yeah, absolutely. Memory constraints, I think affect the way you design your software and the way you write your code. I mean things like lazy loading or cleanup routines are well known, good practices.
108 | 
109 | What was the second one, I’m sorry?
110 | 
111 | Cleanup routines, you have to release unused objects or if you are using cache, you have to release them. You have to clean it if you are not using it, it’s common sense writing code right. In FirefoxOS, not only looking for the memory footprint, but also for improved performance. In FirefoxOS, we have a lot of issues with the first paint event, we have to make sure that the launch path is as fast as possible so the user gets feedback as soon as possible, so you have to write your code with that in mind. You have to lazy load stuff, show things as fast as possible. Also, you have to be aware in FirefoxOS, your application can be killed at any point in its life cycle, so that your app should be ready to recover from that situation. That affects the way you write your code.
112 | 
113 | Has that affected how you do development for other web pages, for non FirefoxOS web pages?
114 | 
115 | Well, I think it’s also beneficial for web applications running on other platforms, but since FirefoxOS is the low end platform for web applications, it’s the best platform to test your working. You will suffer there.
116 | 
117 | So you mentioned the cleanup routines. It’s something that I feel is under appreciated in web development. I feel that people have a grasp on lazy loading, but a lot of people still don’t understand that they should clean up their references whenever an object goes away. I haven’t seen very many articles or people talking about why. Yeah, that’s an interesting one, because it’s something I Have been used to for a while as well.
118 | 
119 | It also depends on the garbage collector. Maybe it’s platform specific also. You thought it’s hard to write your code for a specific implementation of the browser, and you just need to use the profiler in that case. It’s hard to think about the garbage collection when you need to release references, but usually you need to use, to do it, when you don’t need it. Basically, it’s common sense. And also the event listeners and all this stuff that’s taking memory and you are not using it.
120 | 
121 | Event listeners take up a surprising amount of memory I think. Definitely things to clean up.
122 | 
123 | When FirefoxOS started, no, Firefox Accounts, I remember sitting in Madrid, and I showed up a day late and you guys had the API to talk between Gaia and Gecko defined, and that has sort of been the bedrock of communication between Firefox Accounts web context in Gaia and the system app, and the rest of everything basically from what I understand. And developers often hate doing design like that, up front, they just like to do it as they go along, but when have you found early API design like that helpful, and have you ever found that can hinder things and make it more difficult?
124 | 
125 | I don’t like to put labels on these kinds of things. I really, what we did with Firefox Accounts for FirefoxOS was more like, minimimal design up front, rather than a big design up front. We actually proposed different architectures and iterated over them along the work week, trying to find the best approach to the problems we had at that point, but what.. in any case, I guess what approach you take depends on how much time you can spend on the design itself or in the development part itself, and how flexible you are, or the project or the problem, or the problem can be changed later if it’s needed. In general I like to think about the details in advance is quite beneficial in saving a lot of time in refactoring it. So, I think that’s what we did there in Madrid for Firefox Accounts. We put up different options and we tried to evolve these differing options, and we ended up choosing the, what we thought was the best one.
126 | 
127 | And one that worked well because it kept being used, so.
128 | 
129 | Yeah, it’s being used.
130 | 
131 | So the next thing is, the review process. We’ve talked a bit about testing and reviews already. But the review process for Gecko and Gaia is brutal. People inside of Mozilla talk about it quite regularly, just going “ah, this patch took forever, this reviewer is a jerk.” You know. Why do you think a B2G review is so difficult, and what can developers do to smooth out that process?
132 | 
133 | It’s not only B2G. I think the Gecko source code is already a huge piece of software by itself, with all these specific things about XPCOM and XPCOM pointers and other complex stuff. If you add to that base things like hardware details, network, mobile radio specific stuff, in some cases even operator specific stuff, or external services dependencies like Firefox Accounts, you end up having a very complex system with many pieces involved. And so coding for it, many times requires also a complex review process. So, there are many things to take into account when writing a patch for B2G, like performance, security, coding style, tests, etc. And reviewers have to ensure that all these things are considered and well done. And, I think what, what you can do to smooth out the review process is just to work closely with your reviewer, asking him feedback for when you need it, and asking basically what he wants you to do, and what he considers the best approach to do it, and also explaining well, what’s the intention of your patches in the best possible way, and what are the things that you already considered while writing those patches, right.
134 | 
135 | I think that’s something that a lot of people don’t take the time to explain is what other approaches have been considered and why they chose a particular one.
136 | 
137 | Yeah, I think it’s something we commented on a few minutes ago, you just want to see something working, and you just want to land it. You wrote the code and just threw it up to the review process hoping that nobody notices that you were too lazy to explain what you did. But yeah, ideally, you should explain, think that the reviewer is, has no idea of what you intend to do. It’s not enough, or at least he will spend more time reviewing if you help him and you guide him through your code.
138 | 
139 | That makes sense. It seems like a piece that is critical is knowing who the reviewer or the reviewers are going to be and talking to them ahead of time before you actually potentially submit the patch and say “hey, this is what I’m trying to do, um, what are you going to be looking for whenever I submit this patch?”
140 | 
141 | Sometimes you already know the reviewer and you already know what he expects you to do, so I think you accommodate your development process and your Bugzilla submission process with what you think the developers, reviewers, expect.
142 | 
143 | Fair enough, completely fair. We talked about this a little bit already. What do you look for when you are doing a review? A Gaia or Gecko review.
144 | 
145 | In general where I look first is correctness. I mean, the patch should actually fix the issue it was written for. And of course it shouldn’t have collateral effects. It shouldn’t introduce any regression.  And as I said, I try to test the patches myself if I have the time or if the patch is critical enough, to see how they work and to see if they introduce any regression. And also I look that the code is performant and is secure, and also if, I always try to ask for tests if I think they are possible to write for the patch. And I finally look for things like quality of the code in general, and documentation, coding style, contribution, process correctness. If you have to squash your commits for pushing it to Gaia, then well, you have to do it. Or if you have to write the commit message with the reviewer name, or the bug number, then you have to do it because it’s the process for that project.
146 | 
147 | So here’s actually something, whenever you are reviewing code, say it’s a large patch, would you prefer that patch be broken up into smaller commits that show the developer’s process, or do you like to see one giant thousand line patch potentially?
148 | 
149 | I prefer it to be split between different commits. I usually actually write the code that way. I do different patches with different numbers for each patch, that way I think you can even ask for review of different reviewers for each different part because sometimes you will write for B2G but it also involves Gonk parts, or Gecko parts, or Gaia parts, and the same reviewer cannot review the whole stack, so you need to split it into different pieces.
150 | 
151 | And then say, you did have one of these patches that was 3 different commits. Before that got merged, would you squash those into one commit and then merge it?
152 | 
153 | I usually ask the developer to do it, to squash it, or sometimes they do it themselves.
154 | 
155 | That makes a lot of sense. So you reviewed one of my large patches which was to integrate FxA into B2G, into the first time experience I believe it was. You placed of, much more of an emphasis on consistency than any review I have had. By far.
156 | 
157 | I was a pain in the ass then?
158 | 
159 | No, no, not at all. But how do you think that consistency contributes to overall code quality?
160 | 
161 | Well it certainly helps with overall code quality that. When I do reviews these kinds of comments as “nit:” which is quite common in Mozilla, meaning that “I would like to see that changed, but you still get my positive review if you don’t change it, but I would really like to see them changed.”
162 | 
163 | Well, I’ve started marking them with “nit:” because of that, you know, whenever I’m doing reviews. I’ll put “nit: Change the case here” or whatever.
164 | 
165 | But, if you ended up not doing it, please tell me, so I will do it in a follow up bug or whatever.
166 | 
167 | Fantastic, here’s one for B2G especially - as a reviewer, how can you ensure that your comments are not taken too personally by the developer? Yeah, that’s the first part of the question, and the second part is, as a developer, how can you be sure that you don’t take it too personally?
168 | 
169 | For the record, I received quite a few hard revisions myself. And I never take them personally. I mean, I always try to take it, the reviews as positive learning experience, cause I know reviewers usually don’t have a lot of time to, in their life, to do reviews.They also have to write code or a lot of things in their work. So, they just quickly write “needs to be fixed” without spending too much time thinking about the nicest ways to say it. And it’s actually funny because the abscense of comments in a review. I mean, reviewers only say things about negative things in your code, not negative, but things that they consider are not correct. But they don’t usually say that the things that are correct in your code and I know that can be hard at first. But once you start doing it, you understand why they don’t do that. I mean, you have your work to do.
170 | 
171 | You have other things to do as well.
172 | 
173 | Yeah, this is actually especially hard for me, being a non-native English speaker, because sometimes I try to express things the nicest way possible but the lack of works make the review comments sound stronger than it was supposed to be. And, what I try to do is use a lot of smileys if possible. And always, I try to avoid the “r-“ flag if I mean, the “r-“ is really bad. I just clear it, use use the “feedback +” or whatever.
174 | 
175 | So, in the last part of that question is, but you already mentioned this, you try to take it as a learning experience whenever you are developer. Do you take it as a teaching, a potential teaching moment if you are the reviewer as well?
176 | 
177 | Yeah, for sure. I mean just the simple fact of reviewing a patch is a teaching experience. I mean, you are telling the coder what you think is more correct, but sometimes at least I do a. There is a lack of theory and reasons behind that comments, and it’s because of that, the lack of time to express the reasons to make it even nicer. We should all do that, we should explain the reasons and try to make the process as better as possible.
178 | 
179 | It’s tough, but you mentioned a moment ago, um, I know our time’s up here in just a moment. So we can do the last questions offline. Trying to put things in a nice way whenever you are trying to do the review. And sometimes it comes out more blunt than you thought it was, or you intended to. Even as a native English speaker, I have that more often than I’d like, that I see a comment that I wrote and think “ah man, that can be read so many bad ways, so I’m just going to edit it or whatnot”
180 | 
181 | If you can, you cannot edit in Bugzilla and that’s a problem.
182 | 
183 | It’s not only difficult for, you know, people who speak English as a second language, it’s difficult for native English speakers too.
184 | 
185 | OK, so there is no hope for me, even if I learn more English.
186 | 
187 | I think I’ve only met one person who has been able to do it really well consistently, and that was Ben Adida, I don’t know have had a chance to meet him, he was always very good about his comments. I’ve taken up your full hour, and those last three questions are.
188 | 
189 | We can do it if you have time, I know you are busy, so I don’t want to take up too much of your time.
190 | 
191 | I have a meeting in half an hour.
192 | 
193 | OK, then we can go through them, and they are are bout tooling, well, tooling and code quality. What are 5 tools that you can’t live without on FirefoxOS?
194 | 
195 | Well, I use a lot of the Mozilla Cross Reference Search - mxd and dxr.
196 | 
197 | What are they? I know what they are, but how do they help you out?
198 | 
199 | Because it’s like a very powerful grep that will go through all of the Mozilla source tree for Mozilla central or different trees, or even Gaia. So you can search for the keyword you are looking for, and you can click on any function or variable name and you will get another search. You can also see where the function was defined or a variable is being used or defined, so it’s quite useful for seeing the flow through the code. It’s really a powerful tool.
200 | 
201 | I didn’t know that. DXR does that as well?
202 | 
203 | Yeah. And I don’t really know the differences between MXR and DXR aside from the UI differences. I think DXR has more flexibility about the queries that you do, like I want to know when, where this function was defined and when it is used. In MXR, it’s like a grep, so everything is mixed.
204 | 
205 | I’m gonna have to use that more, but I don’t really do any Gaia or Gecko hacking now. What other tools are you using on FirefoxOS?
206 | 
207 | For FirefoxOS specific development, for the platform, I wrote a script to push Javascript changes in modules for components without needing to rebuild everything. So what I do is take the omnijar from the device, I unzip it, I modify it, I preprocess it, and I put it again to the device. This tool is called OmniPush.
208 | 
209 | Is it up online?
210 | 
211 | Yeah, on GitHub. It’s really helpful for me, I mean, if I’m writing Javascript code for Gecko, I just need to use this tool and I have my changes in like 10 seconds on the device.
212 | 
213 | Instead of rebuilding the whole thing.
214 | 
215 | Yeah. When I don’t need a device feature, like mobile connectivity, or the SIM card, I use the B2G desktop because it uses a similar way of, I mean, in B2G desktop, the content that is shipped in the OmniJar in the device is just a symbolic link to your source code on your computer, so if you run that in your desktop, in your computer I mean, the B2G desktop, you just need to modify your source code and B2G takes your changes immediately. So that’s quite a fast development process for Javascript. If you modify C++, you have to rebuild it.
216 | 
217 | That makes sense. And then for normal front end development, break outside of FirefoxOS.
218 | 
219 | Well, I’m not really, I’m not a front end developer. I started writing front end code in FirefoxOS.
220 | 
221 | Oh really?
222 | 
223 | Yeah, I found very useful the developer tools from Firefox. And for FirefoxOS especially, the app manager is really useful. All the style editor and DOM viewer and everything is for front end developers like me, with very very limited experience, is quite useful.
224 | 
225 | Ah that’s cool, and then, what does code quality mean to you? It can be anything, it doesn’t matter.
226 | 
227 | I could define it like, with university, what they taught me in university, but I remember a picture of what code quality means. I think it, I tend to think that way also, which is a code has quality has a few “what the f***’s” per minute when you are reading it. So, if you are reading a code and it’s like “what the fuck, what’s this?” then it’s like, that’s not quality, so. If you the read this and are like “what the fuck, I would like do it that way” (less emphasis), then it’s like “well… OK.” I think then there’s a picture somewhere.
228 | 
229 | I’m gonna have to find this.
230 | 
231 | I will search for you, and will send it to you if I find it.
232 | 
233 | And then is there anything else you want to add?
234 | 
235 | Yes, thank you for doing this, I think this is great. It’s going to be very useful for not only for you. As you said, you have this target of learning from the experience of learning from this experience, that if you share it with a lot of other people on Mozilla Hacks, it’s going to be great to read it and to see what the other people think about code quality, revisions, all this stuff. It’s really cool.
236 | 
237 | It’s fun for me, and it gives me a reason to talk to people that I don’t always have a normal reason to talk to, you know? That’s one of the cool things about it.
238 | 
239 | This has been fun man, thank you for your time.
240 | 
241 | ==============
242 | 
243 | follow on questions
244 | 
245 | S: Do you have a snippet of code, from you or anybody else, that you think is particularly elegant that others could learn from?
246 | 
247 | F: I am pretty critical with my own code so I can't really think about a snippet of code of my own that I am particularly proud enough to show :). But if I have to choose a quick example I was quite happy with the result of the last big refactor of the call log database for the Gaia Dialer app [3] or the recent Mobile Identity API implementation [4].
248 | 
249 | [3] - https://github.com/mozilla-b2g/gaia/blob/master/apps/communications/dialer/js/call_log_db.js
250 | [4] - https://mxr.mozilla.org/mozilla-central/source/services/mobileid/
251 | 
252 | S: What open source projects would you like to encourage people to participate in, and where can they go to get involved?
253 | 
254 | F: Firefox OS of course! No, seriously, I believe Firefox OS gives to software engineers the chance to get involved in an amazing open source community with tons of technical challenges from low level to front end code. Having the chance to dig into the guts of a web browser and a mobile operative system in such an open environment is quite a privilege. It may seem hard at first to get involved and jump into the process and the code, but there are already some very nice docs at the MDN [5] and a lot of nice people willing to help on IRC (#b2g and #gaia) [5], the mailing lists (dev-b2g and dev-gaia) [7] or ask.mozilla.org [8].
255 | 
256 | [5] - https://developer.mozilla.org/en-US/Firefox_OS
257 | [6] - https://wiki.mozilla.org/IRC
258 | [7] - https://lists.mozilla.org/listinfo
259 | [8] - https://ask.mozilla.org/questions/
260 | 
261 | S: How can people keep up to date about what you are working on?
262 | 
263 | F: I don't have a blog, but I have my public Github account [7] and my Twitter account [8].
264 | 
265 | [8] - https://github.com/ferjm
266 | [9] - https://twitter.com/f_jimenez
267 | 
268 | 
269 | 


--------------------------------------------------------------------------------
/interviews/brian-warner.txt:
--------------------------------------------------------------------------------
  1 | 
  2 | ST: What do you do at Mozilla?
  3 | 
  4 | BW: My title is a staff security engineer in the cloud services group. I analyse and develop protocols for securely managing passwords and account data and I implement tools to, I implement those protocols in different fashions, I review other code, I look at external projects that we can take advantage of to figure out whether its appropriate to incorporate them, and I try and stay on top of security failures like 0 days and problems in the wild that might affect us and also tools and algorithms that we might be able to use.
  5 | 
  6 | ST: So you do quite a lot then.
  7 | 
  8 | BW: There’s a variety of stuff then.
  9 | 
 10 | ST: I didn’t quite realise you did all of that.
 11 | 
 12 | BW: Yeah, different amounts of it at different times. Sometimes I’m more doing developer stuff, sometimes I’m doing more research stuff.j And I guess it kinds of trades off if its crunch time and we need to write code or if we have an opportunity to bang on designs.
 13 | 
 14 | ST: UX vs Security: Is it a false dichotomy? Why do some people have the impression that for security to be good, it must be difficult to use? How can that perception be altered? (You and Skinny were going to do a talk together about how the two can live hand in hand, I was always interested in hearing more about that.) 
 15 | 
 16 | BW: There are times when I think that it’s a three way tradeoff. Like instead of being x axis, y axis, and there is this diagonal line that doesn’t touch zero, it’s like 1, 0, 0, 1. Sometimes I think it’s a three way thing where the other axis is how much work you want to put into it or how clever you are or how much user research and experimentation actually. Stuff that engineers are typically not focused on but that UX people and more psychologists, people that study human reactions and how people are focused on those results. So, I believe, maybe it’s more of a hope than a belief, that if you put enough effort into that, then you can actually find something that actually is secure and usable at the same time, but you have to do a lot more work.  So in some ways the quickest, easiest, fastest to market, cheapest to develop kind of thing is going to have pretty severe tradeoffs.
 17 | 
 18 | ST: So is this sort of the same three axis thing of .. what is it.. whether its good, whether its fast and whether its fast.
 19 | 
 20 | BW: best, cheap. And so yeah, secure, usable, cheap. Secure usable fast, maybe. I think that the trick is to figure out what people want to do, find a way of making whatever security decisions they have to make a normative part of that work flow. So that when you lend your house key to a neighbour so they can water your plants when you are away on vacation, you’ve got a pretty good idea of what idea you are handing over. And there is some social constructs surrounding that like “I don’t think you’re going to make a copy of that key and so when I get it back from you, so when I get it back from you, you no longer have that power that I granted to you.” And there are sort of patterns in normal life with normal non-computer behaviours and objects or effects that we developed some social practices around, some social standards around. And I think part of the trick is to use that like assume that people are going to expect something that works like that and then find a way to make the computer stuff look like that. And there are some pieces, part of the problem is that we end up asking people to do very unnatural things because it is hard to imagine or hard to build something that’s better. So like passwords. The only way. Passwords are a lousy authentication technology for a lot of different reasons. One of them being that in most cases to exercise the power you have to give that power to whoever it is you are trying to prove to. It’s like let me prove to you I know a secret, ok, tell me the secret. And that introduces all these issues like knowing who you are talking to and knowing to correctly identify who you are talking too. In addition to that the best passwords are going to be randomly generated by a computer and they are relatively long. And it’s totally possible to memorise things like that but it takes space, repetition, it takes using it on a regular basis, it takes a certain amount of exercise and you have to think of, think more in terms in terms of designing a lesson plan for using the password, and practice learning the password. And that is way more work than any one program deserves. You know you couldn’t build such things and ask people to use it, but if you only have one such password and the only thing you used it on was your phone, then your phone is now your intermediary that manages all this stuff for you, and then it probably would be fair, and it’s clear that your phone is sort of this extension of you, better at remembering things, and that the one password you need in this whole system is the bootstrapping thing that prevents your phone getting stolen. So some stuff like that, some stuff like escalating effort in rare circumstances. You know there are a lot of cases where what you do on an every day basis can be really easy and really light weight, and its only when you lose the phone that you have to go back to the more complicated thing just like you know, just like you carry so much wallet, and every once in a while you have to go to a bank and get more. So it’s stuff like that I think it’s totally possible to do, but it’s been really easy to fall into these bad patterns like blaming the user or pushing a lot of decisions onto the user when they don’t really have enough information to make a good choice there. And a lot of the choices you are giving them aren’t very meaningful.
 21 | 
 22 | ST: Do you think a lot of times users either don’t understand the decisions they are being asked to make or the tradeoffs they are being asked to be made. Say data, user data perhaps, for a free product and stuff like that?
 23 | 
 24 | BW: I think that’s very true, and I think more of the time it’s an inappropriate question to ask them. It’s kind of unfair. Walking up to somebody and putting them in this uncomfortable situation - do you like X or do you like Y is a little bit cruel. 
 25 | 
 26 | ST: So that can be reframed, or should be reframed a lot of times?
 27 | 
 28 | BW: Yeah, I mean, there’s situations in which. One thing that comes to mind is permission dialogs, especially on windows boxes, where to do a bunch of really basic stuff that’s useful and it’s a valid thing to do, it’s not like you are trying to do something crazy, you have to answer these questions, you have to accept these demands that are basically saying - bad things will happen and it’s all going to be your fault. In some ways that’s giving the user, it’s intended to give the user an informed choice, but what it comes out into is this kind of blame the user, blame the victim pattern, where it’s like “something bad happened, but you clicked on the OK button, you’ve taken responsibility for that” Because it wasn’t, the user didn’t have enough information to do something and the system wasn’t well enough designed that they could have designed what they wanted to do without becoming vulnerable for the stuff they asked for. It’s like, I want to run a spreadsheet program and if the dialog box “by the way if the spreadsheet program is badly written, then it could have a virus and it’ll destroy your entire computer.” then it’s like well, how about you write the spreadsheet program better, or how about the OS system runs in a  sandbox where it can only spread sheets, where it can only like flip bits and not overwrite everything. So that’s a limitation of the computing model that is then imposing this really annoying choice. But there are other cases like you said with online data and “by using this service, do you understand that they are going to collect a lot of information about you and sell it to somebody that doesn’t really care about you but they are going to make some money off of that and are you OK with that?” Yeah, it seems like there should be some better way of expressing that or raising awareness of that and that I’d like if people recognised that’s what the deal was.
 29 | 
 30 | ST: So would I, very much so.
 31 | 
 32 | BW: But I don’t know how to get that started and I don’t know how to convince a business to scare all of their potential users away
 33 | 
 34 | ST: Thank you for that. So months before sync 2.0, I don’t know if that’s what it’s actually being called or not, ever saw the light of day.
 35 | 
 36 | BW: We tend to call it New Sync or Sync 1.5. The sync people call it sync 1.5, because they had bigger plans in mind for sync 2.0.
 37 | 
 38 | ST: But you guys hashing out the protocol in extremely vocal and public forum. You know there were weigh ins from everybody it seemed and that thread just went on forever. This is last May I’m thinking of in particular, April, May time. the discussions seemed to be the exact opposite of security through obscurity.
 39 | 
 40 | BW: Definitely, definitely. So there were a couple of different things that I was hoping from that discussion and I pushed all that stuff to be described and discussed publicly because it’s the right thing to do, it’s the way we develop software, you know, it’s the open source way. And so I can’t really imagine doing it any other way. But the specific hopes that I had for publishing that stuff and try and solicit feedback were getting people to look for basic design flaws. It’s like oh hey wait, if you fail to hash this thing into this place, then it means someone can do this kind of guess cheaper than intended. I wanted to get people comfortable with what the security properties were, especially because new sync changes some of them. We are switching away from pairing to something based on passwords. I wanted people to have time to feel they understood what those changes were and why we were making them. Like get the design criteria and those constraints out there so people could see we kind of have to switch to a password to meet all of the other goals, and what’s the best we can do given security based on passwords. And then the other part was I don’t know how to, it’s having that kind of public discussion and getting as many experienced people involved as possible the only way that I know of to develop confidence that we’re building something that’s correct and not broken. One of the tricky things that I’ve found getting into the security space is not knowing when you know enough. Like it’d be nice if there were a merit badge or some kind of you take a test and then they give they give you a sticker, and then you’re like “yup, I know crypto and SEE!” And there isn’t such a thing and you end up reading other stuff, and sometimes you see problems in it, and maybe that tells you well, “maybe well, I know enough to see problems here” and then you build something and then you go and look back at it 6 months later and you see problems in the stuff you wrote yourself and then you go “no, I don’t know how to do this stuff.” So it is really just more eyeballs… problems in the past seems to be the best way of identifying flaws, cause the big problem with the security stuff is like sometimes it’s you’re like “what are the chances that that problem X would happen.” if you design something and there is a 1 in a 1000 chance that something will, that the particular set of inputs will cause this one particular problem to happen, if it really is random, then 1 /1000 may be ok, 1/1M may be ok, but if it is in this situation where an attacker gets to control the inputs, then it’s no longer 1/1000, it’s 1 in how ever many times the attacker chooses to make it 1. And so it’s this game of like whose cleverer and who is more through. And it’s frustrating to have to do this case analysis to figure out every possible thing that could, every state it could get into, but if somebody else out there is determined to find a hole, that’s the kind of analysis they are going to do. And if they are more thorough than you are, then they’ll find a problem that you failed to cover.
 41 | 
 42 | ST: That leads to one of my other questions, is this what is meant by thread modelling?
 43 | 
 44 | BW: Yeah, I think different people use the term in different ways, I think of when you are laying out the system, you are setting up the ground rules. You are saying there is going to be this game. And in this game, Alice is going to choose a password and Bob is trying to guess her password, and whatever. And you are defining what the ground rules are. So sometimes the rules say things like .. The attacker doesn’t get to run on the defending system, their only access to it is through this one API call, and that’s the API call that you provide for all of the good players as well, but the bad guy, you can’t tell the difference between the good guy and the bad guy, so they’re going to use that same API. And so then you figure out what are the security properties if the only thing the bad guy can do is make API calls, so maybe that means they are guessing passwords, or it means they are trying to overflow a buffer by giving you some input you didn’t expect, and then you kind of step back and say “OK, what assumptions are you making here, are those really valid assumptions?” And so, you know, you store passwords in the database with the assumption that the attacker won’t ever be able to see the database, and then some other part of the system, and loops, now they can see the database. OK, roll back that assumption, now you assume that most attackers can’t see the database but sometimes they can, how can you protect the stuff that’s in the database as best as possible. More stuff like you know, who, what are all the different sorts of threats you are intending to defend against, and then you lay those out and someone else can come along later and say that’s all well and good, but I think you should really consider the threat of somebody breaking into your data centre and stealing the computer, or planting a program on there. Or somebody is running a keyboard logger on the user’s machine and is capturing everything their typing, and sometimes those threats are things you can reasonably defend against and sometimes there not. Sometimes you draw a line in the sand and say “we are willing to try and defend against everything up to this level, and beyond that you’re hosed.” And sometimes it’s a very practical distinction where like “we could try to defend against that but it would cost us 5x as much.”
 45 | 
 46 | ST: So this becomes a game of tradeoffs at that point? Again, monetary, time, effort.
 47 | 
 48 | BW: So, for example, if you are worried about, if part of your threat model is something you are trying against is the user’s computer has a keyboard sniffer on it, and what you are trying to do is prevent the attacker from being able to log in to the system on their own, then the way you deal with that is either to like have the user click on buttons on the screen that move around so whatever you are recording from the mouse isn’t enough to figure out what they were clicking on, or you send out little hardware tokens to everybody and have everybody use those instead of real keyboards. And both of those have a cost to like, user’s time, money, and you have to decide how prevalent how prevalent are keyboard sniffers, what percentage of your audience do you expect is going to be suffering from that, and how reasonable is it to try and defend against the threat of them?
 49 | 
 50 | ST: It must be somewhat reasonably common, HSBC sent me out one of those little dongles.
 51 | 
 52 | BW: Well, the stakes are higher. That’s another thing, it’s like um. This tradeoff between, sometimes what people do is try and estimate the value to the attacker versus the cost to the user, and it’s you know its kind of like insurance modelling where expected value. Cost to attacker is X much to do something and they’ve got an expected gain of Y based on like there is some risk they might get caught, there are all sorts of potential penalties. If they succeed in taking over your bank account, how much is that worth to them, and the black market for these has all kinds of layers, there is somebody who is going to transfer the account money where somebody goes to the ATM to take it, somebody is going to take that and mail it to somebody overseas, and like any complicated enterprise, criminal or otherwise, there are lots of middlemen and lots of layers, so the amount of money they can actually get out of your account minus the risk they are going to take by doing it, is only worth so much. And so then the bank are going to try to work out what’s our repetitional risk if somebody does break in here, how much is that going to hurt our reputation or business, and they make the decision that sending out a little gadget like that is going to be worth the money it costs and the inconvenience to you to actually use it, but that will give you more confidence in using their services, you are more likely to recommend your friends.
 53 | 
 54 | ST: So it has a payoff for them in the end probably.
 55 | 
 56 | BW: Yeah, and we definitely want a way to provide incentives to these companies to be like “we should consider this.” We can’t just call it the user’s fault. Europe is doing better at this than the States. European banking system has proven more interest in trying out new technologies to improve stuff. You know chip and pin has been there for a long long time and it’s only starting to show up in the States.
 57 | 
 58 | ST: I can’t imagine not having chip and pin now, it’s just so easy.
 59 | 
 60 | BW: It depends a lot on the regulatory and the consumer liability framework of the different countries. You know in the States, the credit card companies just kind of cover everything. And so it’s really really consumer friendly and they were pushed in that direction by legal requirements, but it means that they, that there is tremendous amount of overhead to pay for fraud management. It’s multiple percentage points that is just in overhead. But as a result people are willing to use their credit cards just everywhere. Just give your credit card to the waiter and they walk away with it and they come back, and maybe you’ve signed something. 
 61 | 
 62 | ST: Maybe you have, maybe you haven’t.
 63 | 
 64 | BW: You signed for something. How many times have you signed, I don’t know. But there are other requirements. I remember being told that Japan in the 80s and 90s didn’t have any, didn’t have strong protections on credit cards or debit cards and so people just refused to use them. You maybe had a debit card, an ATM card, but if anybody else learned that number, they could drain that account and have no recourse. And so people leave them at home locked up and never ever use them anywhere.
 65 | 
 66 | ST: So it’s cash transactions and checks?
 67 | 
 68 | BW: I think it was cache and checks at that time. And even today, small businesses get very little liability protection, so a lot target them first because they know that the charges won’t get rolled back. It’s nowhere near as well protected as a consumer would be.
 69 | 
 70 | ST: I’ve jumped out of order on my questions already, so I’m going to go back a couple of steps. Before a protocol designer, like you did with Sync, ever sits down and writes a spec or line of code, what should they be thinking about? 
 71 | 
 72 | BW: I’d say think about what your users need.  so kind of boil down what they are trying to accomplish into something minimal and pretty basic. And figure out what the smallest amount of code, the smallest amount of power that you can provide that will meet those needs.
 73 | 
 74 | ST: So this is like the agile version of developing ann..  ayah, protocol.
 75 | 
 76 | BW: Yeah. Minimalism is definitely useful. And then think about kind of. Once you have the basic API that enables you to do what needs to be done, then think about all of the bad things that could be done with that API. And try and work out, how to prevent them, or make them too expensive worthwhile.
 77 | 
 78 | ST: So sometimes you can’t eliminate the threat, but you can make it prohibitively expensive.
 79 | 
 80 | BW: Exactly, I really think that that, kind of computer security. I think that our industry is pretty immature. That we are still really learning good techniques and good practices for this. So there is a big class of problems that I kind of it’s probably kind of unfair, callous, I call them beginners mistakes. In some sense it’s the type of mistakes a beginner programmer would make, but also a beginning industry, an immature industry like what we have would make. And I think the more mature ones as like, maybe airline transportation or satellite design, or medicine where they have had several hundred years to figure out the feedback loop so every time there is a problem they investigate it, they learn from it, they update guidelines,. You know which is why air traffic is so safe. Especially compared to how many people fly all the time. Cause they are really really good, they really care deeply about fixing stuff.
 81 | 
 82 | ST: There is actually an article in CACM this month, or last month, about how the airline industry writes such.. has so few problems. It basically comes down to being rigorous in the development process and the review process.
 83 | 
 84 | BW: Right, and ours. The software industry is not there yet, and in some ways it shouldn’t be there. We couldn’t accomplish as much, but I think that we still have a lot to learn. And so, what I kind of described, there are a set of beginner mistakes, which are really a beginner industry mistakes, like buffer overruns, languages that aren’t memory safe, not validating inputs, and stuff like that, but I think that once you get past those a lot of computer security is about linguistics and about game theory and psychology. So it’s like what is. What are the bad outcomes that could exist here? You know somebody gains access that they shouldn’t, somebody stealing money from you, somebody posting things in your name and hurting your reputation, or taking advantage of other people’s opinions, and so what do they hope to gain from that? What do they have to do or break into or guess or get lucky or spend to accomplish that? What sort of risks are they taking on by doing that? Can you rearrange the system so that their incentives encourage them to do the good thing instead of the bad thing? So bit coin was very carefully thought through in this space where there are these clear points where a bad guy, where somebody could try and do a double spend, try and do something that is counter to the system, that is not good for the system as a whole, but it is very clear for everybody including the attacker that their effort would be better spent doing the mainstream good thing. They will clearly make more money doing the good thing than the bad thing. So, any rational attacker will not be an attacker anymore, they will be a good participant. And there are some
 85 | 
 86 | ST: So incentivise being a good actor at that point?
 87 | 
 88 | BW: Yeah, and in other security situations that usually gets expressed as the risk and the cost that they would have to take on in order to make a successful attack is not worth whatever the value of the thing they are attacking is. So somebody can guess passwords, you can’t distinguish between a legitimate user and a bad guy trying to guess a password until they guess so many of them you can say “well, that’s probably not the real user because the real user probably would have figured out their own password by now.” But you can try and make it so that it’s so expensive for the bad guy to do the attack that there expected return is so low that they are not going to bother.
 89 | 
 90 | ST: I’m learning a lot here by the way, thank you for this. Implementing a secure system is difficult, how can a system designer or implementer maximise their chances of developing a reasonably secure system? What are common problems you see out on the web? 
 91 | 
 92 | BW: I’d say the biggest guideline is principal of least authority. So POLA is sometimes how that is expressed. The idea is that the system should be, your system should. Any component should have as little power as necessary to do the specific job that it needed to do. And that has a bunch of implications and one of them is that your system should be built out of separate components, and those components should actually be isolated so that if one of them goes crazy or gets compromised or just misbehaves, has a bug, then, it doesn’t have any more power than it needed. and so if like, the example I’d like to use here is like a, say a decompression routine. You know, something like gzip, and you’ve got bytes coming in over the wire, and you are trying to expand them before you try and do other processing with them. As a software component, that should have, it should be this isolated little bundle of 2 wires. One should have wire coming in with compressed bytes and the other one with decompressed data coming out the other side. It’s gotta allocate memory and do all kinds of format processing and lookup tables and whatnot, but, nothing that box can do no matter weird input you give to it, or how malicious that box you give to it is, can do anything other than spitting bytes out the other side. So it’s a little bit like unix process isolation, except that each process can do sys calls that can trash your entire disk, and do network traffic and do all kinds of stuff, where this is just like one pipe in and one pipe out, nothing else. And it’s not always easy to write your code that way, but it’s usually better. It’s a really good engineering practice because it means when you are trying to figure out what is possibly trying to figure out what could possibly be influencing a bit of code you only have to look at that one bit of code. It’s the reason we discourage the use of global variables, it’s the reason we like object oriented design in which class instances can protect their internal state or at least there is a strong convention that you don’t go around poking around that internal state of other objects. The ability to have private state is like the ability to have private property where it means that you can plan what you are doing without potential interference from things you can’t predict. And so the tractability of analysing your software goes way up if things are isolated. It also implies that you need a memory safe language because if you don’t have that then you know
 93 | 
 94 | ST: You can start peeking in other people’s business.
 95 | 
 96 | BW: Yeah, all over the place. So big, monolithic programs in a non memory safe language are really to develop confidence in. so you know, that’s why I really go for higher level languages that have memory safety to them, even if that means they are not as fast. Most of the time you don’t really need that speed. If you do, it’s usually possible to isolate the thing that you need, into some, a single process.
 97 | 
 98 | ST: So what common problems do you see out on the web that sort of violate these principals?
 99 | 
100 | BW: Well in particular, the web is an interesting space because it’s really common for the. We do a better job. We tend to use memory safe languages for the receiver.
101 | 
102 | ST: You mean like Python and Javascript.
103 | 
104 | BW: Yeah, and we tend to use more object oriented stuff, more isolation. The big problem that I tend to see on the web you’d call it, failure to validate, sanitise your inputs. Or, failing to escape things like injection attacks. And I always think of those as a type violation, or a type confusion kind of problem. So like HTML, creating HTML to send out, the kind of standard problem that you run into where you tell me what your name is, and I want to display a web page with your name in it. And what I’m going to be doing, there’s the DOM, there’s the HTML that I’m generating. And what I’m expecting to do is like <p>Name</p> to close it off. And so what’s really going on at the end of the day the browser is making this DOM tree, and you can think of it as this tree structure and there’s like this HTML node and the BODY node and the P node and the text is inside of that. And I’m expecting to take this string I got from you and stick it into this one little slot here. And it should only ever go inside that paragraph tag, and it should only be text content of that. I’m intending to allow you to change and control the text content, but I’m certainly not intending to let you change the shape of that overall graph. What’s really going on is, I’m not giving a graph to the browser, I’m giving a a string of serialised HTML. So, I’m starting with this intended graph, and I’m turning that into a string, and I’m letting you modify part of that string, and I’m telling the browser to re-scan that string and build it up into a new graph, and my hope is that the old graph and the new graph are going to be the same shape with just the name swapped out. And what’s going on is there are actually two different types, like programming language types, like int, boolean, string stuff. There are actually two different types that I’m using here and they both look like strings. And there is one which is the serialised representation of an HTML tree, and there is another which is the bytes that your name string is providing me. And there is this third implicit type that we never actually construct which is the tree shape. And I’m accidentally silently converting between the name type and the serialised HTML tree type by interpolating the name that you gave me into this serialised HTML tree. And then the common problem with that is where the string that you gave me as a name, you are not telling me Shane, you are actually telling me <script>do something evil</script>, and so if you give me that the proper thing for me to do is to escape those brackets so that when I present that paragraph tag, it will say “Hello, <p>!” and the open bracket will turn into an &lt;, that kind of stuff, there is that kind of stuff. There is that string translation to get from type #1 to type #2 that I forgot to do because they look identical to me and so many of the values in type #1 actually map into the exact same series of bytes in type #2. Anything that doesn’t have a bracket in it, it’s a one to one mapping.
105 | 
106 | ST: It’s OK, but if it does have a bracket, different story.
107 | 
108 | BW: Yeah. So it would be great if we didn’t express HTML in string form. If we had to express it in some other way, then it’d be harder to make that mistake.
109 | 
110 | ST: So if we represented it in the raw tree form all the time.
111 | 
112 | BW: Right, in the program you build up this tree object. Maybe you use DOM functions there to generate something. And at the end of that process, on the server side, now serialise that tree structure out into HTML. Then from one process it goes from a tree to bytes back into a tree, and you never actually manipulate the bytes part. You say that’s some necessary wire level protocol that most developers, most HTML authors would never have to touch. So there is a category of problems like that on the web and it shows up kind of any time that you are touching strings. Any time you are touching strings, they are going to be interpreted by something else.
113 | 
114 | ST: The next question, you have a lot of experience reviewing already written implementations, Persona is one example. What common problems do you see on each of the front and back ends? 
115 | 
116 | BW: Yeah, it tends to be the problems that I remember seeing will be escaping things, a lot of it will be making assumptions about where data comes from and how much an attacker of gets control over it that turn out to be faulty. 
117 | 
118 | ST: So, this is trying to figure out, um, data sources is what you are saying?
119 | 
120 | BW: Yeah.
121 | 
122 | ST: Or source of the data.
123 | 
124 | BW: Yeah, the kind of standard, coming up with abstraction boundaries. The standard engineering technique of breaking a problem down into smaller pieces. And you have a module, and it takes 3 inputs, and you make sure that module does the right thing when it does the right thing when it gets the right inputs and you have to make some assumptions about whether or not it’s possible to be given the wrong inputs. And so, you know, is the relying party name that’s passed into this module coming from one of our own modules or is it coming from a potential attacker? Has somebody else already done the job of deciding if this is a good string or not, or do I need to be responsible for that too? And if you guess wrong or if something changes, you know, somebody drops the ball and is like “I thought you had it”, “oh I thought you had it” then you can wind up with an opening there where somebody crafts the right kind of value and it can get misinterpreted in a certain way and you end up with one of those injection attacks.
125 | 
126 | ST: So, is that why you advocated validating all of the input that came from the RP basically as soon as it entered into the system? You know it was as we got it from the backend we had to validate it, and if it came from the RP we had to validate it.
127 | 
128 | BW: Yeah, I felt that. I like thinking in terms of type distinctions, so that, you know, there is this one type that is “thing that comes from the bad guy” or “thing that comes from outside your kind of trusted perimeter” and you are intending to let them influence some messages, but not others, so you have to make it clear that this is a name, it is intended to be presented. Sometimes this will be put into a log file, sometimes this will be put into an HTML span, and you are supposed to see those same letters up on the screen. And what I would really love is if the language we were using had distinct types for those two types, those different things. And it wouldn’t implicitly convert from one to the other. So, when you add, take that name and add it to a string that’s actually serialised HTML, that should, that’s the point at which the brackets should get turned into &lt; But in a log file, you don’t need that, but that means that the string that gets added to the log file is of type “name” and it’s different than in HTML, and so if you take that log line and dump it into some other page, like a debugging page, it’s going to take your log information and show it on an HTML page, that’s gotta do the transformation too. So yeah, what I was advocating there is for there is that transformation takes place at the boundary between externally provided data and internally kind of sanitised stuff. And that the type the type is very clearly, like I almost wanted to do the convention where you put _<type_name> on all variables and all arguments as you pass them around.
129 | 
130 | ST: This is using Hungarian Notation.
131 | 
132 | BW: Kind of, yeah. Usually when I see that notation, it’s pretty ugly, but sometimes I appreciate the reminder that you know, this is not not escaped. This may have the wrong kind of characters.
133 | 
134 | ST: That’s super interesting, I never thought of that. Is this one of the reasons you also advocated making it easy to trace how the data flows through the system?
135 | 
136 | BW: Yeah, definitely, it’d be nice if you could kind of zoom out of the code and see a bunch of little connected components with little lines running between them, and to say, “OK, how did this module come up with this name string? Oh, well it came from this one. Where did it come from there? Then trace it back to the point where, HERE that name string actually comes from a user submitted parameter. This is coming from the browser, and the browser is generating it as the domain of the, sending domain of the postMessage. OK, how much control does the attacker have over one of those? What could they do that would be surprising to us? And then, work out at any given point what the type is, see where the transition is from one type to another, and notice if there are any points where you are failing to do that, that transformation or you are getting the type confused. Definitely, simplicity and visibility and tractable analysis are the keys.
137 | 
138 | ST: So, what can people do to make that, make that simpler? Make that data flow trace simpler?
139 | 
140 | BW: I think, minimising interactions between different pieces of code is a really big thing. Isolate behaviour to specific small areas. Try and break up the overall functionality into pieces that make sense.
141 | 
142 | ST: So like god modules are just horrible ideas.
143 | 
144 | BW: Yeah, and likewise, super tiny ones might not be correct either. It’s always kind of a judgement call there. But, you know, there’s a lot of division of responsibility in human effort to get a bunch of effort together - OK, you are going to do this, you are going to do that, and doing the same thing in programs where you know whatever processing has to be done, all that complexity is isolated and doesn’t have to affect anybody outside. And recognise that it’s a big deal when you change the contract between two pieces. Try and arrange the dotted lines so that you don’t have to do that quite so often. So, anything that helps analysability helps. That means, threads are usually wrong. 
145 | 
146 | ST: So, for backend code, threads make it difficult? And I imagine communicate shared memory that’s doubly difficult. 
147 | 
148 | BW: Right, and that’s the main problem. When you are looking at a piece of code and you are trying to figure out what could happen, you have to be aware of anything else that could affect the memory space at any time. If you don’t have threads, or if that entire piece of code is run inside of a big lock, then you know that the only code you need to look at is the code that is on the screen in front of you. If there is a global variable that it references, then that variable won’t change while you are changing the code, but the value that it starts at and what it might affect by modifying that global variable might matter. So somebody else might get hit by what you do inside of this function. If you’ve got threads, then every single point along that, in the middle of a line, then somebody could jump in and change the state. So it’s like trying to do your taxes in paper on your desk while somebody is trying to clean your desk on your or rearranging your stuff.
149 | 
150 | ST: And I imagine troublesome on multicore processors where they can actually, you know, CPUs, cores, can go after the same memory at the same time.
151 | 
152 | BW: And it makes it hard to take advantage of multiple cores, and that’s an important thing to do because CPUs aren’t getting faster, they are just getting wider. You need to shape the problem so that those different cores can correctly work on different problems at the same time and not actually overlap. So yeah, threads are really bad because the analysis complexity just totally blows up. Any kind of global variables, any kind of side effects, things where you know, mutability is kind of a drag for that. If you could look at a value and know that is is just a label for some static immutable value, that’s simplifies the analysis. There are languages where macros can be good for expressing some stuff, but it makes it harder, you have to look at more code to figure out what that piece is doing. Sometimes a language can be too flexible, and if you can do operator overloading and somebody is being too clever with it, you have to figure out what that is.
153 | 
154 | ST: Even a ah, perhaps more mundane example, whenever you reviewed the Persona code, I really remember you saying that it was really difficult to trace the data flow through because I would shove all these data, all these bits of data onto an object, under named items in the object and I would just pass the object around, so you didn’t actually know where the data was coming from or how it got there. So I’ve always remembered that since then.
155 | 
156 | BW: And it’s interesting because different languages end up with different patterns for this. At the time, I’m still not a very strong Javascript developer, at the time, I was mostly doing Python and there the convention would probably end up being keyword arguments. You know named keyword arguments when you are trying to pass multiple things through. The Javascript convention seems to be passing in a single options arguments that has a bunch of other stuff on it. And so looking at the function signature you can’t tell what is actually getting passed in. And that’s just a … of the language and not having direct support for keyword arguments. And so, looking at that, seeing this object called “options” everywhere, that in fact had a bunch of mandatory properties, so I was like “alright, nothing very optional about these options.” And the fact that every single had a variable of the same name, but all of them were different was challenging. That’s one practice that I’ve tried to, in some of the python code that I do, is to try to come up with a distinctive name for, this is the download handler, then have a name for that. Anywhere you hold a variable of that type, use the same name for that purpose. So that when you are grepping around, you know, a lot of this is like grep oriented programming. What can you do to help somebody that has your source tree and grep, but doesn’t know their way around to find the parts that are relevant. A strongly typed language with some good IDE support could let you find all of the instances of the particular kind of value, and that’s probably a good indicator.
157 | 
158 | ST: Do you mean things like intellisense and eclipse?
159 | 
160 | BW: Yeah, it would be like my download handler is an instance of a class called DownloadHandler, tell me all of the variables or all of the arguments in the system that are of that same type. In a language where you have to tell the language about that everywhere, it’d be something explicitly typed like C++ or Java. Then that would give you as the explorer, as a kind of archeologist looking through this stuff to be like “oh, well, here’s a place that can create one of those values, and here’s a place you might pass it to, and here is a place that asks it to do something.” Those are probably the relevant bits that I need to look at. In a duck typing language, you don’t that type information to work with, so the best you can do is like variable names that are all consistent and use the same thing, or comments that say “This function takes an object of class X, or abstract base class, or interface X, and use that to get a guide of what’s going on. But it’s tough because it also means, it has to be kind of a global convention, you have to say, “I’m not going to call anything a download handler unless it really is one of these things.” That means the names can’t be too short, otherwise it’s like “well, here’s a thing that has handles, and this is the handle that downloads, and DownloadHandler, well no, damn.” So, it’s tricky, and it ends up imposing some linguistic constraints on your prgram as a whole.
161 | 
162 | ST: So what is defence in depth and how can developers use it in their system?
163 | 
164 | BW: So, it’s a bit like principal of lease authority where. Well, belt and suspenders is the classic phrase. If one thing goes wrong, the other thing will protect you.
165 | 
166 | ST: I’ve never actually heard that before.
167 | 
168 | BW: Really? It’s. You look silly if you are wearing both a belt and suspenders because they are two independent tools that help you keep your pants on, but sometimes belts break, and sometimes suspenders break, and the chances that both of them will break before you get home, it protects you from the embarrassment of having your pants fall off. And that’s funny, because the guy I know that uses that phrase is from London, and I was assuming it was sort of a Britishism. So defence in depth usually means don’t depend upon perimeter security.
169 | 
170 | ST: So that would be what you were talking about before where you only scrub things on the, say the input right. Where data enters the system, where. Does this mean you should also be checking down the line, even though you think the data was scrubbed down there, if it’s critical at this moment that this be escaped, I should make sure that the stuff is escaped before I write it out to the DOM.
171 | 
172 | BW: Yeah, it can frequently be a good idea. There is always kind a judgement call about performance cost, or, the complexity cost. If your code is filled with sanity checking, then that can distract the person who is reading your code from seeing what real functionality is taking place. And then, that limits their ability to understand your code, which is important to be able to use it correctly and satisfy its needs, and stuff like that. So, it’s always this kind of judgement call and tension between being too verbose and not being verbose enough, or having too much checking. But yeah, one. The notion of perimeter security, it’s really easy to fall into this trap of thinking of, drawing this dotted line around the outside of your program and saying “the bad guys are out there, and everyone inside is good” and then implementing whatever defences you are going to do at that boundary and nothing further inside. And I think, I was talking with some folks and their opinion was that there is evolutionary biology and evolutionary sociology reasons for this. That humans developed in these, you know, like tens of thousands, up to hundreds of thousands of years ago in these tribes where basically you are related to everyone else in the tribe and there are maybe 100 people, and you live far away from the next tribe over. And the rule was basically if you are related to somebody then you trust them, and if you aren’t related, you kill them on site. And it’s like, that worked for a while, but you can’t build any social structure larger than 100 people, but the reals are really simple. And I think that that sort of translated into, we still think that way when we come to computers. And we think that like “bad guy”, “good guy”, I only have to defend against the bad guy. But, we can’t distinguish between the two of them on the internet, and the good guys make mistakes too. So, the principal of least authority thing and the idea of having separate software components are all very independent and they have very limited access to each other means that, if a component breaks because somebody compromised it, or somebody tricked it into behaving differently than you expected, or it’s just buggy, then the damage that it can do is limited because the next component is not going to be willing to do that much for it.
173 | 
174 | ST: Is this one of the reasons why Persona there is a notion a database reader and a database writer, where these were two isolated processes?
175 | 
176 | BW: I think so there may also be performance or scalability reasons for that. I know sometimes that like, having a database view that’s readonly means that it’s usable in circumstances like partition or replicas not being online, where you could keep reading from it, but you are not allowed to write to it anymore, so maybe you get some functionality from it even if you don’t have a quorum of writers. I think it’s also a good security tool where no matter what you ask the database reader to do, it’s not going to modify the database. And a lot of the operations don’t require modification to the database. So, you can think of it, there’s this security pattern discipline called object capability security that I’m a really big fan of. And the idea there is that each object in your object oriented programming language is isolated from the others. And you can invoke methods on it, but you basically sending messages to it. Please do something, please do X. And, the abilities that that object has, the things that can it can influence are exactly specified by the object references, pointers that it has to other objects. So, you can look at any object and know what it can do and can’t do based on what it has access to, what other objects it has references to. And you can make up new objects any time you want that encapsulates the subses of functionality that you care about. And in that world, if you think about it from the attacker’s point of view… So, the traditional big monolithic application, as the author’s and defender in that game, there are all of these potential holes that you might have to go and plug. You know, every message coming into your system might be malformed, it might be longer than you expect, it might be formatted strangely, so you are worrying about buffer overflows, you are wondering whether, you are trying to accept messages that tell you to do X, but you are trying to reject messages telling you to do Y, so all of those points, it’s sort of like, a big castle with a strong exterior wall, but it’s a really big wall, so there are lots of places where there might be hole in it, and no defences inside whatsoever. So, once those attackers breach any point in the wall, then they are in the court yard and they just take over everything. And so, a big program that has lots of complicated string handling, written in C, is like that because one buffer overflow means that now you are executing the attackers code and it’s not your perimeter anymore under your total control. The other approach is more like, this particular API call is expecting strings, and it parses them, but it’s in a memory safe language, so if something goes wrong, it’s not gonna take over the whole program. And any way, all that API call does is send well formed messages with a tree of arguments to this place which then does the next part of the processing. So you are matching a bunch of little isolated buildings, little armored cars or something, and each one has a smaller security perimeter, there are a lot more of them, but more importantly, the only way to get to the database, if your goal, if the prize of the attacker is to get to that database, the only thing that knows how to touch the database is the database reader. And the only way that you can get to the database reader is to attack the command processor, and the only way you can get to the command processor is to, and that that that. So, from a defender’s point of view, if anything along that chain survives that attack, then the attacker can’t get through. And from the attacker’s point of view, there’s this chain of stuff, and I have to compromise every single one of those, and the only handle I have on this thing is whatever the previous thing can reach. It’s like, even if I could completely, the API input string scanner was so so totally broken that I was able to compromise it entirely, it can’t do all that much. And it’s like the example I had before with the text decompressor, or in a web browser, the jpeg decompressor. If it’s properly isolated, then the worst thing that an attacker can do by trying to do a buffer overflow, sending you malformed jpegs or something, that process, that component should take in compressed data and spit out pixels, like (x, y, colour), and nothing else. The worst case is that they can like put rude pictures in front of you. They should not be able to take over anything else, there is no reason for the jpeg decompressor to have access to your disk. So, in a nicely isolated system like this, the attacker’s job, mark miller addressed this nicely, the attacker’s job is exponential, whereas the defender’s job is linear. In the monolithic application, the defender’s job is exponential, and the attacker’s job is linear. The bigger the program, the more potential access points, and they only need one. Whereas the defender, you have more and more code and more interactions you have to protect.
177 | 
178 | ST: I’ve taken over the hour that I requested with you.
179 | 
180 | BW: No worries, we can just keep going if you want.
181 | 
182 | ST: Do you think that common standards, like coding styles, really helps out with reviewing anyways, or help out with overall security?
183 | 
184 | BW: I think it can, I think that the ability to read code, and correctly reason about what it does is really critical. And to the extend that coding standards help you do that, then yeah, it helps security.
185 | 
186 | ST: Do you think that it, things like coding standards, reduce the amount of cognitive load because you don’t have to keep swapping between different styles and stuff like that, you can just focus on the task.
187 | 
188 | BW: Yeah, there was a really interesting paper a couple of months ago where some folks were, they were putting programming problems in front of somebody.. Here’s some basic python or some psuedocode, what does it do? Maybe C. And they introduced errors into it. And so it’d be things like you know, missing the braces on a C if-then-else statement, stuff like that. And what they found with that with experienced programmers got it wrong more frequently than beginning programmers, because the experienced programmer, you know, the particular, part of it is also indentation changes. In C where whitespace doesn’t matter, but there are certain conventions that the experienced programmers knew about that the beginners didn’t. And so, if you see that line indented in a particular way, then you think, “well, any sane programmer would intend for that to do the following”, and the experienced programmers assumed that the author of this code was a sane, experienced programmer and meant to do something. And so they didn’t bother looking for the details because they weren’t defending an inexperienced programmer making mistakes when they were reading this stuff. And I think there are cases there where if you..  Yeah, it reduces cognitive load because if it’s formatted correctly and if the language gives you some extra support there where it doesn’t let you make some of the basic mistakes like, “oh, there’s a missing brace and that changes the behaviour entirely” or there is a missing semicolon and now the Javascript parses entirely differently. Then, you don’t have to think as much while you are reading through it. And it is a question of whether defensive code analysis, where the author of this code is deliberately trying to trick you, or if it is friendly code and it’s like “well, maybe they just had a bad day, and they had a mistake”, but most of the time they get it right, and then I’m reviewing this for the higher level control flow as opposed to the low level and spelling mistakes and stuff.
189 | 
190 | ST: Do you use any tools to help you ensure the code that you write is relatively secure? Or as secure as you think it can be?
191 | 
192 | BW: I don’t know that I do. There are, for python, I mean, there is the standard stuff like unit tests, trying to run those all of the time, trying to structure your program in such a way that you can run tests on isolated pieces of it, so that you can run them very quickly. When I’m deep in the program, I’m trying to run the unit tests every couple of seconds, so I really want them to be small enough and fast enough and in a tight enough loop to do that. I use, this is mostly on python, I use code coverage tools to convince myself that the tests are actually exercising all of the different cases, and so I have some stuff like coverage on python, and some emacs integration stuff that I wrote to colour the lines in the editor based upon whether the code coverage tools announce whether they were included or not. And I know there is stuff like that in the Javascript world.
193 | 
194 | ST: We are using one on the content server, but it doesn’t integrate nicely with our editor.
195 | 
196 | BW: We can fix that, some of that is like, figure out what the coverage tool emits as a list of line numbers, and then writing stuff for the editor that knows how to read that, and then mark up the lines. And there is some static, in python there is a great tool called PyFlakes which is really fast, so it’s like the kind of thing that you can arrange to run every time you save the file to disk. And that just tells you stuff like there’s a variable here that’s cited but not actually defined or referenced anywhere else, that’s probably a typo. You forgot an input statement here, there’s a syntax error over here.
197 | 
198 | ST: So linters.
199 | 
200 | BW: Yeah, so that’s nice to get through the basic, you know, I’m a bit embarrassed to reveal to the unit test suite that I forgot that return statement, some really basic stupid error, so it’s like pyflakes is my little buddy helping me look so dumb the unit test code. And the unit test code is what helps me look not so dumb to the reviewers and stuff like that.
201 | 
202 | ST: So, I’ve gotta ask about the Javascript and the browser at the end. But where can other developers, this is actually, the writeup eventually go on hacks. Where can developers go to learn what they should be doing?
203 | 
204 | BW: That’s a good question. Yeah, I don’t, I guess for any given language there’s kind of the current popular unit test frameworks, the documentation there, or the people that wrote it, or the people that use it, and advocate for it, will probably have blog posts and essays about the benefits of doing that, and using those things. I’d say find a software project is well run and look to see what they are toolchain is. I know I learned a lot when I first got into python, I started using twisted and participating in Twisted coding. And I think their practices are just first rate, and I learned a lot about how to test stuff and how to structure programs, just how to name methods and name classes. Kind of like write categories of speech, whether it should be a noun or verb, or you know message names written from the point of view of the receiver or written from the point of view of the sender is really interesting anthromorphisation that helps you read what’s going on later and to understand stuff. Kind of telling a story with your code. I learned a lot from that community. There are other projects in other languages that could do that.
205 | 
206 | ST: Reading other people’s code. Reading good examples of code helps out?
207 | 
208 | BW: Yeah, and also watching how they work. Sometimes if you are chatting with people on IRC and they are walking you through working on your first bug in some other project, and they say “OK, the way we work here is we write patch and we submit it here, and there is this review thing, and there is this test thing.” It can also be really valuable to meet these people in person, go to sprints, go to conferences, and sit down with them and look over their shoulder while they tackle these things. And that’s when you start to really discover what tools they are using.
209 | 
210 | ST: That’s interesting, and now one last question and I’ve really gotta get out of here. A lot of people say that you shouldn’t do crypto in the browser, the browser can’t be a trusted environment, blah, blah blah. And, well, why is it thought to be a bad a idea, and we seem to do it all over the place, and we are pretty confident that we can do it safely.
211 | 
212 | BW: It’s an excellent question, there are maybe 3 or 4 different dimensions to it. Some are better than others. One dimension is that Javascript is a frustrating language to do low level math in, to do low level binary operations. And especially to do crypto fast, you really need to do a lot of bit twiddling, a lot of XORs, a lot of shifts, and Javascript has this 32 bit unsigned integer type, kind of, sort of. But you have to work really hard to get it. So, that’s getting better with like asm.js
213 | 
214 | ST: Do typed arrays help out?
215 | 
216 | BW: It does, it does, and so these days the way you do that kind of crypto in the browser is to write it in C, and then compile it into asm, because they’ll probably get those low level things correct for you. Typed arrays is another example where Javascript doesn’t make it easy to handle binary data, just a byte string. There’s UTF, UTF8, but what you really need is your unsigned char * from C so that you can point at stuff. So you have to do extra work to get at that stuff and that’s annoying. That’s slowly getting better. Another technique that helps with that is to write stuff in NodeJS style, using the buffer, and then run Browserify to turn that into something the browser can use, because that knows how to use Typed Arrays. So, I really kind of thing the way to write that kind of crypto stuff these days is write your low level shifting stuff with, in C and asm.js and stick that into a module, and write your middle level passing data in, slicing dicing stuff with buffers, and browserify it and wind up with something in JS. And then you can use your higher level stuff. That’s one axis. Another axis is who is supplying the program you think you are running, and who are you trying to protect yourself against? So there’s this spectrum of static to very dynamic code delivery mechanisms. My alarm clock at home. If you were an attacker and you wanted to somehow get me, there’s like this one day, and there’s this critical meeting or a critical interview and you wanted to cause me to wake up an hour late and miss it. You either have to break into my house and swap out the alarm clock, or like bribe my girlfriend to change the time on it, or go backwards in time 20 years when I got that thing as like a high school graduation gift and swap out the chip before they got it. Super static, there is nothing dynamic in there whatsoever. Then there’s like code that you, a program that you installed from CD-ROM back in the 80s, you’d have to grab the CD-ROM on its way to my house and swap that out for a different one with a different program. Then there’s like, stuff that you install and there is an automatic update process. And so, the operating system, OSX, if you can kind of trick that into installing the wrong thing, then you can get control later. Then there’s code that’s getting updated all of the time. There’s code that’s delivered to your browser, every single time, just before you run it. So, web pages are very dynamic and the, in addition to being much more difficult to attack the alarm clock, it’s much riskier because that thing is sitting inside my house, and if I think there is something wrong, I have this evidence of it in my hand. Whereas the web page is there, it does something, and it’s gone. So as a attacker in control of the site that’s giving you the code, or maybe in control of the network, or I can somehow violate SSL rules, then I can tamper with your experience and then erase the evidence. It’s fairly easy to attack and it’s fairly low risk. That mostly matters if your threat model includes the person who’s providing you with the code. And it may be that you are loading a web page from site A, and your threat model is site B. And so, you are getting a tool from site A, that’s encrypting some data, and then storying the cipher text in site B. And you are totally fine with site A seeing your stuff, but it’s B that you are trying to protect against. There is also a model in which an add-on, it’s written in Javascript, but the code is actually coming from an add-on. Which means the update rules, it only gets updated when the browser decides to go and fetch add ons. It’s more like the software in the native application that has an automatic update process, it’s a little bit less dynamic. Yeah, so that’s another axis. The availability of libraries to do crypto, to do it quickly, that have been well reviewed, is another axis.
217 | 
218 | ST: Is the Stanford one respectable?
219 | 
220 | BW: It’s not bad, the only problem is that it’s not really maintained. I’ve talked to a couple of the authors and they kind don’t really want to have anything more to do with it. So it’s troublesome that there are not a lot of parties out there that write this stuff and want to keep writing it, and be responsible for it. And that’ll shift over time. And I think the final axis the difficulty of doing.. Timing safe is difficult in Javascript. A lot of crypto algorithms take different amounts of time depending upon what your key is.
221 | 
222 | ST: So, you can figure out the length of the key?
223 | 
224 | BW: If you observe, you can figure out, or you can figure out the bits of it. There are cases in which you can actually find out what the secret key is by these sort of, watching at a distance how long your computer takes to do certain things, or how busy it is. There are environments in which, if I can run a program on your computer at the same time as you are doing something with your secret key, then based upon which cache lines get knocked out, I can figure out what your key is. And in a web environment that’s relatively easy because your web browser is running programs on behalf of all sorts of different web servers, all at the same time. So it’s not a super isolated environment, it’s kind of leaky that way. And it’s easier to write timing insensitive code in a lower level language where we have more control over each instruction potentially. So javascript doesn’t give you that. And I think the final thing was, just like performance and people. The browser is really big. Like the amount of code there dwarfs a kernel, you know, kind of every other program, because it’s becoming the operating system. And the bigger the program is, the more security holes there are. You know, somebody has to worry about, it’s got an XML parser, it might have flash, it’s got these video parsers, image parsers, and all of those things are different angles. And so somebody’s who is kind of serious about crypto is going to be anxious about doing it inside a browser, or be anxious about doing it on a computer with a browser installed, because that’s such a big attack surface. But, if you actually want to get stuff delivered to lots of people, the web is the best way to do it these days.
225 | 
226 | ST: So why are we somewhat confident that we can do it safely, at least somewhat safely?
227 | 
228 | BW: I think that, that first axis on who’s supplying the code and what is the threat model, that we have, we are in the right position for that. So in the case of BrowserID, either the browser in a native implementation, or the site that gives you the shim are able to change the crypto code, but not the IdP, and not the RP. 
229 | 
230 | ST: We control everything at that point.
231 | 
232 | BW: Right. It’s like, we are willing to rely upon the shim provider in exchange for which we get, we are, the threat model is the IdP is trying to do X, but we are trying to keep them from doing Y, and the RP is trying to do something… Those are the pieces that are at arms length, so along that particular axis, I think we are in the appropriate spot to do crypto inside the browser. The operation that we are doing is fast enough, you know, people at SJCL to do the signature code that we need, so yeah, Javascript is not the friendliest language for doing that, but it’s not too bad, and somebody else has made it work. And we’re not doing these operations very quickly, so the timing stuff, we are kind of willing to believe that it’s good enough, it’s always this kind of like “well, it’s a little bit risky, you know, driving a car is not really a very safe thing, but I need to get to work and it’s too far to take the train.” That’s always kind of the trade off there.
233 | 
234 | ST: Thank you Brian.
235 | 


--------------------------------------------------------------------------------