258 |

259 |

MIT Web Programming Crash Course

260 |

433 | 434 | 435 |

436 |

1 Introduction

437 |

438 |

439 | These are the (work in progress) notes I took while taking MIT's 2019 web.lab, formerly called 6.148 Web Programming Competition. It is an 8 day workshop that assumes no background, and the goal is to produce a prototype. My learning style is to identify things that I don't understand as I progress, trying secondary sources to elaborate on whatever seems confusing or is hand waved in the slides and lectures, which may be helpful to you and are an example of the kinds of notes you should take yourself when trying to figure out a new subject. We can't possibly learn the entire field of web development in one workshop so this is just a starting point. 440 |

441 |

442 | 443 |

444 |

1.1 Course materials

445 |

446 |

The 2019 schedule: http://weblab.mit.edu/schedule/ (flip your device to landscape, or view in desktop mode if you can't see lecture/slide links) 448 |
- The 2019 lectures are terrible production quality, slides are available to pair with them.
The git repository for the assignments

453 | 454 |

455 | Optional: 456 |

457 |

MIT's 6.170 Software Studio lectures and slides, or the 2019 schedule, for exercises and review which is a more advanced course so optional extra material.
The free book, Functional-Light JavaScript by Kyle Simpson to help explain some of the 6.170 material.
The 2018 schedule, better quality lectures but no React content

462 |

463 |

464 | 465 | 466 |

467 |

1.2 Prerequisites

468 |

469 |

470 | As stated in the intro this workshop assumes no background, not even a programming background. You could absolutely follow the web.lab lectures and just hack together a prototype. I'm doing the majority of exercises with Eruda, on a $100 Motorola phone 1hr or so per day and it's been working so far. You can also use Chrome DevTools or FireFox Developer Tools to edit javascript in the console as you read the slides on any desktop browser or use VSCode, screw around with Heroku or AWS free tier, or run node.js locally on your own system. 471 |

472 |

473 |

474 | 475 |

476 |

1.3 Discord channel

477 |

478 |

479 | Some anons have started a Discord channel if you want to collaborate/learn with others. 480 |

481 |

482 |

483 |

484 | 485 |

486 |

2 OOP crash course

487 |

488 |

489 | A lot of OOP terminology will be used in web.lab and 6.170, it will make much more sense if you do these executable slides for objects, abstract types, closures, reference aliasing, etc. 490 | These JavaScript slides are self explanatory and you can run/change them in your browser. Arrow down until that section ends, right arrow for next section. 491 | The last exercise probably won't make any sense until you do the course. What it's doing is initializing an empty array 'es', pushing the negative time when the light is green 492 | and onclick it will push the current (positive) time. 493 |

494 |

495 |

496 | 497 |

498 |

3 [web.lab] Ideation

499 |

500 |

YouTube - IAP 2019 Day 1 - Ideation

503 | 504 |

505 | Recommendation: just flip through the slides. 506 |

507 |

508 | 509 |

510 |

3.1 Step 1: generate everything and everything

511 |

512 |

business model abstractions, improving existing models
'PAFT': (P)roblem thinking, considering the (A)udience, considering (F)eatures, but have no idea what they mean by Theme

516 |

517 |

518 | 519 | 520 |

521 |

3.2 Step 2: closing doors and narrowing windows

522 |

523 |

Overdone ideas are talked about, though it directly conflicts with the very first step which talks about successful abstractions based on existing ideas

526 |

527 |

528 | 529 |

530 |

3.3 Step 3: transforming ideas

531 |

532 |

MVP = Minimal Viable Product or the bare minimum functionality you need to showcase your idea
Try to describe your website/app in a single sentence 535 |
- talking about diverging your idea such as extending bathroom codes list w/wifi codes
- example features: paying 'points' for contributing to a crowdsourced data set to encourage user participation

540 |

541 |

542 | 543 |

544 |

3.4 Editors/IDEs

545 |

546 |

547 | VSCode is shilled, which is pretty easy to learn. For most of the beginning of the workshop I edited in Eruda (use Chrome DevTools on a desktop) directly changing html and CSS. 548 |

549 |

550 |

551 | 552 |

553 |

3.5 Summary

554 |

555 |

556 | The features that define your software should be in the prototype you push early to your audience, with nice to have features left as a TODO. You can make this easier by being able to describe your software in one sentence, which will help make it clear what your critical features are. 557 |

558 |

559 |

560 |

561 | 562 | 563 |

564 |

4 [web.lab] HTML

565 |

566 |

YouTube - IAP 2019 Day 1 - HTML

569 | 570 | 571 |

572 | Html: 573 |

574 |

is 'nested boxes'
DOCTYPE reason for existence is hand waved. 577 |
- Investigation reveals doctype declarations prevent the browser from loading pages in other modes such as 'quirks mode' 579 |
  - quirks mode turns out to be a legacy feature to intentionally simulate the bugs of old browsers like IE4/5.
- Important: anything written before a doctype declaration may trigger nonstandard behavior
- There are now 3 modes: quirks mode, almost standards mode, and a full standards mode. 584 |
  - We will certainly want 'full standards mode' so <!DOCTYPE html>

589 | 590 |

591 | Html tags: 592 |

593 |

Opening and closing tags to structure the document into containers
Can't violate scope, ie: close a container tag then try and close a tag that was opened inside that now closed container 596 |
- If I open a box, and inside are envelopes that I open, I can't then close the box and then expect to close the internal envelopes from outside the box
Reasons for why tags are even used in the first place are in this primer: 600 |
- The browser throws all newline indicators away and processes the html as a long string as it deems appropriate
- Exception: tag <pre> that indicates preformatted so will display as is
- Elements represent logical relationships rather than precise physical relationships, because people can alter their browser settings to display the page in a different physical style than what you specify 604 |
  - Ex: bold, underline, italics, may all be displayed differently but the html element for these things simply represents that text should be more prominent in some way

609 | 610 | 611 |

612 | Clarity: What exactly is a tag vs an element? 613 | An element is what the tag represents (semantics) while the tag is syntax. For example the tag <div>, it is markup syntax for 'the Content Division element'. 614 | If I were to make a new element 'the Supreme Container element' and give it a tag <kjdfj34> </kjdfj34> it's now clear which is the element and what is markup syntax. 615 |

616 | 617 | 618 |

619 | Html head: 620 |

621 |

General machine processed information about the page (metadata) 623 |
- Slides have additional information if you look at the bottom of each one, reveals the head primarily holding information for search engines and the browser regarding title, scripts and css style sheets

627 | 628 |

629 | Html body: 630 |

631 |

Represents what you see (the content)
Heading tags, h1 = 'first order' (top level), h2 is subsection title, h3 etc.

635 | 636 |

637 | Html Attributes: 638 |

639 |

Optional attributes are inside the opening tag
Link tags are called 'anchor', prob for obscure netscape navigator reasons
ul, ol and li (list) elements discussion, dissenting opinion here why ul shouldn't be considered 'unordered'

644 | 645 |

646 | Not covered, void elements: 647 | The reasons given in the lecture why <img> doesn't have a closing tag were vague. The HTML standard reveals this is a void element which is not allowed any content, hence no reason for requiring closing(they can still have attributes). 648 |

649 |

650 | 651 | 652 |

653 |

4.1 Avoiding <div> soup

654 |

655 |

656 | Div soup is when you look at the source at many modern websites and see an enormous pile of div containers everywhere. 657 |

658 | 659 |

HTML now has semantic elements, some advice on semantic element use from the HTML working group WHATWG 661 |
- These elements exist so software like password managers, search engine bots(SEO), pocket, accessibility assistants can parse the page, or for future developers trying to maintain the page (which could be yourself a few months from now)
- div is defined in the lecture as a block section ie: 'making scope boxes in documents'
- span is defined as inline sections, such as changing the colors of every letter in a single word

667 |

668 |

669 | 670 | 671 |

672 |

4.2 Let's read some of the HTML standard

673 |

674 |

675 | MIT's advice is to 'just google', so I'm going to skim through the actual HTML standard, to clear up these questions like which semantic elements do what, what exactly div is, etc. 676 |

677 |

The standard insists there is only HTML and no 'HTML5', as the standard is unversioned.
A review of semantics 680 |
- Another statement that html markup conveys the meaning, rather than the presentation since the browser can adjust all these things.
- There's a comment about using grouping elements like hgroup, to organize secondary titles when using headings, so they aren't misinterpreted as a seperate section. 683 |
  - This is apparently not adopted by W3C, another standards organization, and MDN claims hgroup is 'theoretical only' linking to a W3C workaround using span

688 | 689 | 690 |

691 | We find what we need: 692 |

693 |

Full table of contents for HTML elements and their description 695 |
- A div element is strongly encouraged to only be used as last resort, with other semantic tags used instead for accessibility software
The <em> element is emphasis, for accessibility readers and other machines parsing html, it subsumes italics element
Span is exampled here as loading a translation attribute in a sentence on a single word
Standard also lists various attributes we will want to ensure the [0-9] number keypad pops up on touchscreens. 701 |
- "Not all inputs that contain numbers (credit cards, SS#, etc) should have an input type of number. They are best served up as text inputs, which are still free to use the pattern attribute to pull up the tel keypad on supported devices"
- In general the number input should only be used if it's something you're going to do math operations on, you are unlikely to add and subtract account numbers and other numeric identifiers.

706 | 707 |

708 | I just skimmed inputs as there will be future lectures on this. 709 |

710 |

711 |

712 | 713 |

714 |

4.3 WHATWG standard or W3C aka w3.org aka w3schools.com?

715 |

716 |

717 | Which html standard do we use? W3C forks the WHATWG standard, with inconsistent updates. A summary of why there are two different standards bodies (WHATWG and W3C) is here. Unfortunately we can't rely on WHATWG standard either, as it's a 'living document' which translates to unversioned document. 718 |

719 | 720 |

721 | Further research indicates the canonical resource to use in order to find out what works with which browser, should be MDN, with https://caniuse.com/ and https://devdocs.io/ as secondary resources. If you search for 'most popular screen readers' you find a list of products that primarily use IE unfortunately. 722 |

723 |

724 |

725 |

726 | 727 | 728 |

729 |

5 [web.lab] CSS Workshop

730 |

731 |

YouTube - IAP 2019 Day 1 - CSS & HTML CSS Workshop

734 | 735 |

736 | We are asked to clone this repository https://github.com/mit6148-workshops/html-css-workshop or download the .zip and expand. 737 |

738 | 739 | 740 |

741 | We inspect these example html pages in Chrome devtools (right click, choose inspect or Ctrl-Shift-i depending on your OS) or w/Eruda. 742 |

743 | 744 |

Child elements of div receive the styling of the parent element
Overrides are talked about, the most specific naming has the highest priority 747 |
- The lecturer is surprised no questions about overrides, and I've marked this as a TODO because they seem important to understand.
Padding is discussed, if you play around with the values editing them while in inspector, you'll get it immediately
We are asked to make a class called text-center in style2.css. Then we are asked to add the class to the div with id=profile-container. I solve this by first looking at style2.css and noticing .card was commented as a class selector so I mimicked that entry that has a dot in front of the selector, with my own new property: .text-center { text-align: center }.

753 |

754 | Then I opened profile2.html and inserted <div id="profile-container" class="text-center"> to see if it would work, assuming because this is markup we can add a space and fill in new attributes. It works. I try it on <h4></h4> by pasting in my class attribute: <h4 class="text-center"></h4> and it works there too. Finally I see the card class entry: <h3 class="card"> and add a space, inserting my text-center class: <h3 class="card text-center"> and it works as expected. 755 |

756 |

757 | 758 |

759 |

5.1 Circle avatar CSS hack

760 |

761 |

profile4.html and profile4.css are an example to display a background image with a circle border. 763 |
- this seems like a work around/hack and something we would never do in practice

767 | 768 |

769 | Let's attempt to understand what is going on. With Chrome inspector open while displaying profile4.html, click on the div for large-profile-container, 770 | we see there is green padding on each side of the image and an empty spot exactly where the image is shown. Now examine the div for circle-avatar, the green padding covers exactly where the image is. According to this version of the CSS standard color or images can be 771 | attached to the padding with the background property, which is exactly what is specified in the circle-avatar selector. The border-radius property is obvious, just play with it's values to see how it curves the border. 772 |

773 | 774 |

775 | The CSS for large-profile-container contains the line: "padding: 0 40%" which from MDN indicates: "When two values are specified, the first padding applies to the top and bottom, the second to the left and right." Examining the div large-profile-container again 776 | in Chrome inspector, on the bottom right where there is the box model diagram, we scroll down and see the padding-left, and the padding-right values are identical. 777 | What the percentage value has done, is taken the width of the parent div container: profile-container, and multiplied it by 40% to obtain those two values. 778 | Examining the div for circle-avatar (we're still in Chrome inspector btw), the padding-top value is exactly the same as the height of it's parent: large-profile-container because the CSS values for the circle-avatar div contains: padding-top: 100% which means 779 | multiply the height of the top container by 1 (100%) so you get the same value. Note the browser is automatically calculating this, these values will change depending on how you resize your page. 780 |

781 | 782 |

783 | Another use for the background property is assigning a temporary color if you're scaffolding a bunch of containers by hand, just to see the layout. 784 |

785 |

786 |

787 | 788 |

789 |

5.2 Debugging CSS

790 |

791 |

792 | The example in the lecture of adding a text-center class to everything can also be used as a debug helper, adding a brightly colored outline to html containers to see the container layout more clearly. As per MDN, outlines (as compared to border) do not move other components on the page. 793 |

794 |

795 |

796 | 797 |

798 |

5.3 How the cascade works

799 |

800 |

801 | The convention on how overrides are handled were briefly covered, so let's find out what they are and what exactly is cascading. First we look at MDN which reveals cascading is a process in which the winning declaration is chosen, since we 802 | need to consider both user stylesheets (apparently you can write your own CSS, I've just have been enforcing reader mode everywhere), the browser's default stylesheets, and the author of the website or document CSS. These can all conflict so the cascade determines which styling should 'win'. 803 | MDN doesn't really elaborate on the decision making process and points out the !important rule all 3 (author, user, user-agent(browser)) can use to try and override all other style sheets, and that the user defined style sheet has precedence. It's recommended as a CSS author to never use !important. 804 |

805 | 806 |

807 | Checking google scholar we see there's a CSS thesis by HW Lie. Survey papers and thesis are better than any online tutorial since they will include design decisions. 808 |

809 | 810 |

Let's quickly skim through the thesis
The chapter specific to CSS is chapter 6 so we start there
A few pages about syntax in detail, and a full listing of properties for CSS2
A list of possible values for properties, '3em' which is a number with a unit. 'em and ex' are relative to the font sizes, much like our percentage example which was relative to the parent container this size will be expressed relative to whatever font size the user has selected for their browser/reading assistant. 815 |
- these number/unit values can also express angles '90deg', time '10ms' (ie: how long to pause) and freq '3kHz' on the aural properties, which are used by screen readers/speech synthesizers.
- functions can be values, apparently some properties accept a functional notation when using a string presents a naming conflict.
- numerous other values are exampled, there's an interesting formal definition for what a pixel is considered to be in CSS
- space or comma seperated multiple values are fine

822 | 823 |

824 | An interesting statement about properties and their settings: "Nearly every CSS property has different rules for the values on its right-hand side and it is not much of an exaggeration to say that each property's right-hand side has its own specialized language" 825 |

826 | 827 |

828 | This leads into inconsistent percentage values which we already ran into with the CSS-avatar hack: "Percentage values are similar to relative length units in 829 | the sense that they are relative to another value. Each property that accepts a percentage value also defines to what other value the percentage is relative. Most 830 | percentage values refer to width of the closest block-level ancestor but some refer to other values, for example, the font size of the parent element." 831 |

832 | 833 | 834 |

Value propagation aka Cascading begins in Chapter 6.5.1 836 |
- 3 principal mechanisms for propagation: cascading, inheritance and initial values 838 |
  - the cascade is a 'negotiation' between 3 parties (author, agent, user-agent(browser)) to ensure all element/property combinations have a value
  - if the cascading process doesn't yield a value, the parent element's value will be used
- By default author declarations (the person who wrote the CSS) wins over user settings unless user marks !important in their style declarations
Interesting note: the thesis author considered a scenario where users could all share style sheets for different sites on a peer-to-peer basis, too bad this didn't become a mainstream thing where I can use a browser extension and flip through dozens of competing designs for various sites.
The section on creating boxes from elements describes the parsing process, making a tree, etc. It also suggests implementations can optimize processing by doing several steps in parallel 846 |
- Apparently Stylo in Firefox's Quantum browser parallelizes their CSS rendering implementation. TODO: look at this implementation sometime
- Design decisions are described for the '3 band' box model in CSS of Margin, Border and Padding. 849 |
  - note that negative values can be used for margin, to overlap boxes

854 | 855 |

856 | The rest of the thesis is put on hold since we'll learn positioning in the upcoming lecture 'Advanced CSS'. 857 |

858 |

859 |

860 | 861 | 862 |

863 |

5.4 Proprietary extensions

864 |

865 |

866 | Browsers and other software have proprietary CSS extensions. These are defined as "A CSS feature is a proprietary extension if it is meant for use in a closed environment accessible only to a single vendor’s user agent(s)". 867 |

868 |

869 |

870 | 871 | 872 |

873 |

5.5 The current CSS standard

874 |

875 |

876 | W3C maintains the official CSS standard as snapshots so we're left with the same problem as HTML where we have to rely on software documentation (if it exists), or MDN and https://caniuse.com to figure out which software supports what CSS features. 877 |

878 |

879 |

880 |

881 | 882 |

883 |

6 [web.lab] CSS Frameworks

884 |

885 |

YouTube - IAP 2019 Day 1 - Intro to Bootstrap

888 | 889 | 890 | 891 |

'Responsive' is defined as basically being able to render properly on multiple screens/devices.
Boostrap is a CSS framework to make being responsive easier
Other frameworks are covered, now we know the reason why all sites look the same 895 |
- An easy flexbox based framework is Spectre CSS
Bootstrap and these other frameworks essentially abstract Flexbox

900 |

901 | 902 |

903 |

6.1 CSS Grid

904 |

905 |

906 | CSS grid is 'two dimensional' whereas Flexbox layout is 'one dimensional'. The standard gives this example of the difference, but it seems like CSS grid 907 | can reference all of row/column at the same time (adjust height, grow the row past the current row into another) and overlap elements, where Flexbox can only reference a single row/column at a time. 908 |

909 | 910 |

Pitfalls with CSS Grid: The legacy mobile browsers still out there on old devices, and IE version 10 and 11 which refuses to die. 912 |
- There's a workaround for IE using Autoprefixer CSS online, but you could also use javascript to check browser version and present a more barebones page for IE until it finally goes away.
- Major pitfall: Chrome is coming up with their own layoutNG to replace CSS

917 | 918 |

919 | Let's start with the canonical resource for these things (that will tell us what is implemented) which is MDN. 920 |

921 |

We begin to read MDN web docs basic concepts of grid layout 923 |
- all the examples can be done in one html file, resize your browser window when you check the changes to see how grid is responsive to your window resizing:

927 |

 928 | <!DOCTYPE html>
 929 | <style>
 930 |  .wrapper {
 931 |   display: grid;
 932 |   grid-template-columns: 1fr 1fr 1fr; }
 933 |   </style>
 934 | <div class="wrapper">
 935 |   <div>One</div>
 936 |   <div>Two</div>
 937 |   <div>Three</div>
 938 |   <div>Four</div>
 939 |   <div>Five</div>
 940 | </div>
 941 |

942 | 943 |

So far this is all pretty straight forward, the value 'fr' represents a porportion of whatever space is left inside a container, so 1fr and 2fr represents 1/3 and 2/3 avail space. 945 |
- 2fr and 1fr is translated as: "this is twice as big as the other"
minmax() function can also be input to repeat() as a parameter 949 |
- repeat() parameters are repeat(num-of-repeats, the-values-to-repeat)
- repeat(4, 1fr) will create: 1fr 1fr 1fr 1fr
- repeat(2, 10px 1fr) will create: 10px 1fr 10px 1fr
- repeat(2, minmax(100px, auto)) will create 2 tracks sized of min value 100px, and max value set to auto.
The rest of the MDN page on basic concepts of grid layout are self explanatory, overlapping boxes and using property z-index to declare their display priority over other boxes, and refers to other guides with layout examples we can look at when we actually start building something.

958 |

959 | 960 |

961 |

6.1.1 CSS Grid spec

962 |

963 |

964 | Let's skim through the CSS Grid Layout Module Level 1 candidate recommendation. The 2018 Module 2 standard W3C published is just an addon to define subgrids and is still a working draft, so we'll read it after. An example of subgrids is given here. 965 |

966 | 967 |

We begin to read the candidate recommendation for CSS Grid
Mentioned are media queries, these are attributes you can add to your stylesheet declarations to test for certain display properties, like display aspect ratio, of the user viewing the page, and then assign them a specific stylesheet for their device. 970 |
- An example of media queries, if orientation is landscape present the following grid, if portrait, present this other grid configuaration.
Motivation for grid was to replace the hacky workarounds of tables and floating elements for application layouts.
First example using a game screen is self explanatory, and actually a better explanation than the MDN docs
Grid layout is precisely defined here describing 'block axis' (column) and 'inline axis' (row).
There's a warning about "correct source ordering" which is defined by MDN, basically keeping the HTML semantic source logical and only using CSS grid to reorder the content visually, so a screen reader and other software can still interpret your html.
Important we can give names to columns and rows 978 |
- Another article about naming, to make the CSS more flexible, same article also recommends using porportions (the fr value), instead of specific values/percentages

982 | 983 |

984 | The rest of the CSS grid spec goes into fine details about positioning like 'gutters' the term for space between grid boxes. We'll come back to these when we do the advanced CSS lecture. 985 |

986 |

987 |

988 | 989 |

990 |

6.1.2 Common layouts by example

991 |

992 |

993 | MDN has a guide to common layouts using CSS Grid, that use semantic html elements. Seems like there are a dozen different ways to write CSS grid stylesheets, like using wrappers (injecting some class 994 | named .wrapper into a div element, then using .wrapper as a selector to build a grid). We should probably look at some real world examples and not tutorials to see how this is done. 995 |

996 |

997 |

998 | 999 |

1000 |

6.1.3 Examining a sample page

1001 |

1002 |

1003 | Let's try to figure out a page's CSS. I have chosen Stripe's website, specifically the payments marketing page because there's all kinds of things going on like skewed grids, and animations. 1004 |

1005 | 1006 |

A list of interesting things: 1008 |
- Smartphone screenshot image labelled 'kickstarter' has been transformed/rotated to adjust perspective 1010 |
  - transform: translate(-50%,-50%) rotate3d(.5,.866,0,16deg) rotateZ(-2deg); 1012 |
    - transformations are described here, to understand this you need a few lectures from this course or wait until the Advanced CSS lecture.
  - the '$25' overlayed on the phone image is a span element that has also been transformed: scale(var(–device-scale)) translate(54px,620px)
  - –device-scale is a CSS variable
- Navigation up top is also using some kind of transformations, to slow/smooth the display of the next menu option. 1019 |
  - translateX() seems to accomplish this, instead of disappearing and producing a new window, it shifts the old window horizontally with some kind of delay for smoothness/effect
- Animations used for the 'recent updates' boxes, and shadowing for effects 1023 |
  - These cards seem to be stacked boxes overlaying each other, with shadow effects on their borders, animated using the translateY() function, so like the navigation menus that shifted right, these are shifting up with a transition-delay property
- The images for 'Grow Faster' and 'Developer-Centric' are interactive 1027 |
  - It's using an event listener, which we haven't learned yet
  - Again we see there are translateX() function transformation w/delays, in incremental negative values, to give it a smooth transformation from right to left.
- Background looks like an affine grid plane, which is equally spaced parallel lines in two different directions, with some parts colored to make stripes 1032 |
  - see below

1037 | 1038 |

1039 | Let's mimic the stripes. Inspecting any part of the page in Firefox or Chrome Developer Tools, click on the div with class "lower-page" and then it's child div with class "grid" to see the grid, on the right 1040 | hand side under layout. Clicking on each grid reveals it's construction, the slanted grid lines to paint the stripes look interesting. I try: transform: skewY(-0.06turn) (which I found in MDN documentation). 1041 | Then I try to color the d d d area background, using shorthand grid-area values: selecting grid row 2, column 1, spanning 'only grid 2' (unsure about grid-row-end), and going from column line 1 (far left of grid) over 4 column lines to create a similar effect, a background colored stripe on the page. 1042 | Changing these values helps to understand them, except 'grid-row-end', which doesn't act like I think it should when I change it's value from 1 to 3. Looking at this I now get what it does. We can also change the value to span 2 or span 3 to create a thicker stripe. 1043 |

1044 | 1045 |

1046 | <!-- creating background stripe grid test -->
1047 | <!DOCTYPE html>
1048 | <style>
1049 | .wrapper {
1050 |     display: grid;
1051 |     grid-template-columns: repeat(12, 1fr);
1052 |     grid-auto-rows: minmax(100px, auto);
1053 |     grid-template-areas:
1054 |      "h h h h h h h h h"
1055 |      "d d d m m m m m m"
1056 |      "ft ft ft ft ft ft ft ft ft";
1057 |      transform: skewY(-0.06turn);
1058 | }
1059 | .box1 {
1060 | grid-area: 2 / 1 / 2 / 4;
1061 | background-color: lightblue;
1062 | }
1063 | </style>
1064 | 
1065 | <div class="wrapper">
1066 |   <div class="box1"></div>
1067 | </div>
1068 |

1069 |

1070 |

1071 |

1072 |

1073 | 1074 |

1075 |

7 [6.170] CSS/HTML Exercises and review

1076 |

1077 |

1078 | See these slides from MIT's 6.170 for a review of what we've just learned. There's a one-to-one correspondence between HTML elements and DOM nodes where HTML attributes are stored as properties of DOM nodes. 1079 |

1080 | 1081 |

Clear examples of the HTML tree
Clear examples of CSS selectors 1084 |
- * is the universal selector, selects all elements
- #id selects all id attributes ie: #header selects <div id="header">
- .class as we already know selects elements by their class attribute ie: .photo <div class="photo">
- some notes on pseudo-classes to select interactive states like :hover, :checked, :focus, etc.
Some notes on CSS combinations, which lead into exercises below
Cascade specific 'score values' 1092 |
- +1000 if styles specified inline
- +100 for every #id
- +10 for every .class, [attr], :pseudo
- +1 for every element
Box Model is reviewed 1099 |
- absolute values: px, pt, cm are exact measurements that don't respect user browser settings
- relative values: em, %, vw are calculated which we already knew
Flexbox review 1104 |
- display: flex;
- .item {flex-grow: 1;} evenly grows to fill the avail space of each .item
- #selector {flex-grow: 2;} grows one selector to ratio 2 1108 |
  - flex-shrink: 1; will evenly distribute negative space aka shrink
- Short hand is covered, which we already know from Grid, flex: value value value is documented at MDN for grow, shrink, basis etc.

1114 |

1115 | 1116 |

1117 |

7.1 CSS Selector exercises

1118 |

1119 |

1120 | Let's do some selector CSS exercises from: https://flukeout.github.io 1121 |

1122 | 1123 |

I get to level 8 and then get stuck on the small oranges in the bentos selections which I figure out are 'bento orange.small'
I get to level 16 then get stuck on the :only-child selector, it seems like it should be plate:only-child 1126 |
- Answer turns out to be plate *:only-child
Stuck on level 30, however typing in plate[for^="Sa"] selects the plate, so we know what to do: plate[for^="Sa"], bento[for^="Sa"]

1131 |

1132 |

1133 | 1134 |

1135 |

7.2 Flexbox exercises

1136 |

1137 |

These recitation slides have exercises if you wish to practice flex, such as FlexBoxFroggy.
If you're completely stuck on the last exercise: 1140 |
- flex-direction: column-reverse;
- flex-wrap: wrap-reverse;
- justify-content: center;
- align-content: space-between;

1147 |

1148 |

1149 |

1150 | 1151 |

1152 |

8 [web.lab] Git and the command line

1153 |

1154 |

YouTube - IAP 2019 Day 2 - Command Line & Git

1157 | 1158 | 1159 |

1160 | Begins with standard command line introduction, there's some other slides here from Cornell with more content on package management, command flags, etc. 1161 | Surprisingly Gitless isn't covered, which is a simplified user interface for Git. Git is best understood if you know the basics of what a DAG is (a tree-like graph without cycles), there's also this tutorial by example: A Hacker's Guide to Git. Not discussed is Gitlab, or Bitbucket, other version control repository hosting services with various features. There is also mercurial and numerous other version control schemes. 1162 | There's some UIs for git, probably the best of them is magit. There are also text interfaces like Tig. Note emacs has a GUI, you can access pull down menus for all of magit's features which 1163 | will also show you the keyboard shortcuts. 1164 |

1165 |

1166 |

1167 | 1168 | 1169 |

1170 |

9 [web.lab] JavaScript

1171 |

1172 |

YouTube - IAP 2019 Day 2 - JavaScript

1175 | 1176 | 1177 |

Let statements instead of using var, Let is local scope while var is 'scope of the entire function' 1179 |
- const are immutable variables (constants)
Some talk about arrays in JS, they look like lists you can index
A JS object is a collection of key-value pairs so basically a struct in other languages
Equality is three 's where = coerces both items to be the same type 1185 |
- A table of equality boolean outcomes here
- this tests value vs value, or memory reference to see if objects point to same reference
A JS callback function is somewhat defined. 1190 |
- an example of map is used, JS has anonymous/lambda functions

1194 | 1195 |

1196 | Clarity: JavaScript callback functions: a function that is to be executed after another function has finished executing. These are first-class continuations. 1197 |

1198 | 1199 |

1200 | Manipulating HTML: 1201 |

1202 | 1203 |

document.getElementById('identifier'); 1205 |
- gets HTML elements, and seems to return them as some kind of object/structure where you can access various fields using .notation, such as .innerHTML
document.createElement('div'); 1209 |
- can create elements, append their inner fields to other elements.
button.addEventListener('click', function() { alert("Hi"); }); adds an event listener/action.
Adding scripts to HTML documents: 1214 |
- <script src="source.js"></script> must be at the end of html document

1218 |

1219 |

1220 | 1221 |

1222 |

10 [web.lab] Workshop 0: JavaScript

1223 |

1224 |

YouTube - IAP 2019 Day 2 - JavaScript Workshop

1227 | 1228 | 1229 |

1230 | The weblab.to link doesn't work, instead clone catbook-workshop0. These exercises if you're lost look at the answers in 1231 | the slides and play around with the values/inputs, maybe add another feature since hardly any js has been covered yet. 1232 |

1233 |

1234 |

1235 | 1236 | 1237 |

1238 |

11 [6.170] JavaScript Exercises and review

1239 |

1240 |

1241 | Let go through the slides for JS basics 1242 |

1243 |

Official(standardized) name for JS is 'ECMAScript'
Numbers are 64-bit floats (no ints)
Booleans, null values, undefined, strings have no separate char type
arrays a = []; 1248 |
- grow and shrink dynamically, use length property for size: a.length;
- array methods for various manipulation like a.push("test"); or a.pop(); a.indexOf("hello");
- a = ["hello", 2, null, [1, 2], "there"];
- a[number] array notation to access that value
Objects a = {hello: "there"}; 1255 |
- access properties with dot notation a.hello
- can change these properties a.hello = "hi";
- values can be nested b = {qty: 3, item: {name: "crayon", price 5}};
Interpreter determines types of values at runtime with implicit type coercion
Scope: 1262 |
- by default variables are defined in the global scope unless let or const keyword used (block scope) ie: between { and }
- const variables cannot be reassigned, but their properties can be changed 1265 |
  - const STUFF = {name: "test"};
  - STUFF.name = "test2";
- var keyword has entire function scope, using hoisting where their declarations are moved to the top of the function body to cover all block scope within that function

1272 | 1273 |

1274 | JS functions: This chapter helps to explain things like arbitrary numbers of arguments. Multi-argument functions in most languages are really single-argument functions that take a tuple as an argument. 1275 |

1276 | 1277 |

1278 | // declared with a name, arguments(parameters) and a body
1279 | function multiply(a, b) {
1280 |    return a * b;
1281 | }
1282 | 
1283 | //can take a variable number of arguments that don't have to be specified, access these with array index notation
1284 | function multiply() {
1285 |    return arguments[0] * arguments[1];
1286 | }
1287 | 
1288 | 
1289 | //arguments can have default values
1290 | function multiply(a, b = 1) {
1291 |    return a * b;
1292 | }
1293 | 
1294 | //functions are values, can be assigned to variables
1295 | let multiply = function(a,b) {
1296 |    return a * b;
1297 | }
1298 | 
1299 | //can be passed as arguments, ie: first class functions
1300 | let toys = [
1301 |  {name: "Woody", price: 10},
1302 |  {name: "Rex", price: 3}];
1303 | 
1304 | toys =
1305 |    toys.filter(function(toy) {
1306 |       return toy.price < 9;});
1307 | 
1308 |

1309 |

1310 | 1311 |

1312 |

11.1 Functionals:

1313 |

1314 |

1315 | See this chapter on functionals explaining how map/reduce/filter work. 1316 |

1317 | 1318 |

"functionals" aka first class functions, are functions that take another function as input and return a new array or value, leaving the old one unchanged 1320 |
- list.filter(function(item) { … }); 1322 |
  - filter function that tests each element of the array and constructs a new array from whatever values pass the filter
- map also available in js: list.map(function(item) { … }) which applies a function to every element, returning a new array
- list.reduce also available for a single output value, using an accumulator: list.reduce(function, init);

1329 | 1330 |

1331 | Clarity: JS arrays look and act like lists, but are just high level objects, not actually linked lists (or even real arrays for that matter). 1332 |

1333 |

1334 |

1335 | 1336 |

1337 |

11.2 Functionals exercises

1338 |

1339 |

1340 | These start on slide #58. The recitation slides have optional exercises to practice functionals/closures. 1341 |

1342 | 1343 |

1. Create a double function. In the console I type: 1345 |
- function double(x) { return x * 2 };
- a = [2, 3, 5, 7];
- a.map(double)
Let's try it with a lambda function/anonymous function literal 1351 |
- a = [2, 3, 5, 7];
- a.map(x => x * 2);
2. Sum an array of values 1356 |
- So we want reduce 1358 |
  - function getSum(acc, num) {return acc + num;}
  - a = [2, 3, 5, 7];
  - a.reduce(getSum, 0); (zero is the starting value of the acculumator)
  - Also works: a.reduce((acc, num) => acc + num, 0); 1363 |
    - With arrow functions () => x is short for () => { return x; }.

1369 |

1370 |

1371 | 1372 |

1373 |

11.3 Closures

1374 |

1375 |

1376 | This chapter on closures vs object will clarify these slides. 1377 |

1378 | 1379 |

1380 | There's an example on slide #62 of pairs(list1, list2, f) function being passed another function as an argument in order to encapsulate the iteration and provide the ability to 1381 | reference items instead of direct array indices to avoid mistakes. Closures are then covered, using a higher-order function which is a function that returns another function. If you're reading the slides in say, chrome or firefox pdf reader, 1382 | right click inspect element or inspector, click on console and enter the code you see to understand what it's doing. Type 'hello;' into the console to see it's 1383 | construction: f(msg){ console.log(prefix + msg);} which is how hello("arvind"); works. There's an explanation of closures here but for clarity let's refer to PSML: 1384 |

1385 | 1386 |

1387 |
1388 | ..the concept of a closure, a technique for implementing higher-order functions. When a function expression is evaluated, a copy of the environment is attached to the function. Subsequently, all free variables of the function (i.e., those variables not occurring as parameters) 1389 | are resolved with respect to the environment attached to the function; the function is therefore said to be “closed” with respect to the attached environment. This is achieved at function application time by “swapping” the attached environment of the 1390 | function for the environment active at the point of the call. The swapped environment is restored after the call is complete. 1391 |
1392 |

1393 | 1394 | 1395 |

1396 | There's a memoization exercise in the slides. What it's doing is cacheing the previous computations in a data structure. 1397 | When you use it for Fibonacci as in the example: memoizedFib(40), the memoized function calls fib(40), and stores the result in the array we declared. The next time you use that function, memoizedFib(40) 1398 | to call fib(40) again it will look up the entry for index 40 in the array and then produce that previous value instead of having to do exponential work calling fib(40) all over again. Note that it is a generic memoization function, anything else that maps from one input to one output (like 1399 | calculating factors, or primes) can use the same memoization function, such as newfactor = memoized(factor) or fastprime = memoized(sieve). 1400 |

1401 |

1402 |

1403 | 1404 |

1405 |

11.4 Another Review of JS

1406 |

1407 |

1408 | The 6.170 slides from the Fall 2019 course have an interactive JavaScript overview including the problematic features like NaN/null, this keyword, silent failure etc. 1409 |

1410 |

1411 |

1412 |

1413 | 1414 |

1415 |

12 [6.170] JavaScript: DOM Manipulation exercises and review

1416 |

1417 |

1418 | The first part of these slides cover OOP constructors such as Prototypes, which are explained here. I don't think either course has fully described what the document object model(DOM) is, so we'll look at the standard for it later. 1419 |

1420 | 1421 |

Slide #37 we come to DOM manipulation 1423 |
- We can select a DOM node w/selectors (except pseudo-classes like caption:hover): 1425 |
  - document.querySelector("#header"), document.querySelectorAll(".caption")
  - Note these are CSS selectors, so everything we learned in CSS about selectors can be applied here
- An element is a DOM node that corresponds to an html tag like <p> or <h1>
- A node is a generic DOM node class for text, comments, etc. 1431 |
  - element.tagName, element.id, element.innerHTML, node.nodeType, node.nodeValue
- Manipulating properties: element.style.color = "blue";, element.style.borderRadius = "10px";
- Caveats: string cat has poor perf due to browser has to parse the string then reconstruct DOM subtree 1436 |
  - cannot insert element between two others, can't return references to new elements
The slides walk through an example of adding a new link to the tree after h1: 1442 |
- let link = document.createElement("a"); (create a new element)
- link.href = "http://eecs.mit.edu"; (create internal .href property)
- let label = document.createTextNode("EECS"); (create a new DOM/tree label)
- link.appendChild(label); (append the label to 'a')
- let body = document.querySelector("body"); (grab the body DOM tree object, assign it to a variable for manipulation)
- body.appendChild(link); (append our new 'a' node to body)
Inserting before a node: 1451 |
- let newChild = document.createElement("p"); (create a new element again called p)
- let parent = document.querySelector("body"); (create a variable called parent that represents the html body)
- let sibling = document.querySelector("h1"); (create a sibling variable, select the element 'h1')
- parent.insertBefore(newChild, sibling); (insert our new element p before h1)
- parent.replaceChild(newChild, sibling);
- parent.removeChild(newChild, sibling);
There are multiple methods to traverse the tree: .previousSibling, .nextSibling, .nextElementSibling, etc.

1461 |

1462 | 1463 |

1464 |

12.1 DOM manipulation exercise

1465 |

1466 |

1467 | We're asked to go to example.com, open the console, write three ways to select the "More information" link, one of which must be a traversal, 1468 | and inside the white box add a third paragraph of text. Finally add a 2nd level heading before the paragraph we added. They have different solutions in the slides, notably the ('p > a') selector and using .children to walk the tree. 1469 |

1470 | 1471 | 1472 |

Select <a> directly: 1474 |
- let link = document.querySelector("a");
- link.href
Traverse to <a>: 1479 |
- let parent = document.querySelector('p'); (grabs the first paragraph above the link)
- let traverse = parent.nextElementSibling;
- traverse.firstElementchild.href
Another way to select the href: 1485 |
- let b = document.querySelectorAll('p');
- this selects every 'p' and places into a NodeList, which we can access using array notation []:
- b[1].firstElementChild.href
- b[1].innerHTML also reveals the link
Let's add a paragraph: 1492 |
- let firstP = document.querySelector('p'); (grabs first paragraph)
- let newChild = document.createElement('p'); (create a new <p>)
- firstP.appendChild(newChild);
- newChild.innerText = "Here's another paragraph"

1499 |

Let's add the second level heading: 1501 |
- let newHeading = document.createElement('h2');
- firstP.insertBefore(newHeading, newChild);
- newHeading.innerText = "This is a new Heading";

1507 | 1508 | 1509 |

1510 | Let's finish the slides from 6.170, starting on slide #77 for JavaScript events. 1511 |

1512 |

JS is single threaded
blocking is a problem for long running operations
JS has a trad call stack and a message queue for events such as clicks, timers, etc.
when the call stack is empty, the event loop evals FIFO (first in, first out)

1518 | 1519 |

1520 | Running their example shows how events (timeouts) are deferred: 1521 |

1522 |

1523 | function one() {
1524 |  console.log("one");
1525 |  setTimeout(two, 0);
1526 |  console.log("three");
1527 |  setTimeout(four, 0);
1528 |  console.log("five");
1529 | }
1530 | function two() {
1531 |  console.log("two");
1532 | }
1533 | function four() {
1534 |  console.log("four");
1535 | }
1536 |

1537 | 1538 |

setTimeout(function() {…}, ms); 1540 |
- Executes the given function after a min delay in milliseconds
let timerID = setInterval(function() {…}, ms); 1544 |
- Executes the given function at a given interval in milliseconds
- clearInterval(timerID) to stop the timer
Listening for interaction events: 1549 |
- let elem = document.querySelector("h1");
- elem.addEventListener("click", function(event) {…});
Browser creates events + adds them to the queue when h1 is clicked
When the event is processed from the queue, the registered listener function is called.

1556 |

1557 |

1558 | 1559 |

1560 |

12.2 Event exercises

1561 |

1562 |

1563 | We're asked to change the font color of the h1 heading everytime we click it, and set a timer to change the background color of the 'Example Domain' box. 1564 |

1565 | 1566 |

1567 | // returns a random color as string "rbg(x,x,x)"
1568 | function randomRGB() {
1569 |     let rgbArray = [255,255,255];
1570 |     let rgb = rgbArray.map( x => x * Math.random() );
1571 |     return 'rgb(' +rgb[0] +',' +rgb[1] +',' +rgb[2] +')';
1572 | }   
1573 | 
1574 | // change h1 to a random color on every click
1575 | let h1 = document.querySelector('h1');
1576 | h1.addEventListener( 'click', function randomColor() { 
1577 |     h1.style.color = randomRGB();
1578 | });
1579 | 
1580 | // timer to change Example Domain div background color randomly every second
1581 | let exampleBox = document.querySelector('div');
1582 | let timerExampleBox = setInterval( function randomBackground() { exampleBox.style.background = randomRGB();}, 1000);
1583 |

1584 | 1585 |

Event Handling Gotchas: Context 1587 |
- ES6 'arrow functions' are needed to access the variable counter inside the same function, or else that event listener function will create it's own environment/context

1591 | 1592 |

1593 | function Counter() {
1594 |    counter = 0;
1595 |    let b = document.createElement("button");
1596 |    b.innerText = "Increment me!"
1597 |    document.body.append(b);
1598 |    b.addEventListener("click", () => { counter += 1; console.log(counter); });
1599 |  }
1600 | counter();
1601 |

1602 |

1603 |

1604 |

1605 | 1606 |

1607 |

13 [web.lab] Intro to Asynchronous Programming

1608 |

1609 |

YouTube - IAP 2019 Day 3 - Intro to Backend

1612 | 1613 |

1614 | Some basic information here on HTTP requests and response, how async code runs vs synchronous execution. 1615 |

1616 |

1617 | 1618 |

1619 |

13.1 [6.170] HTTP Request & Response

1620 |

1621 |

1622 | Let's review the slides for web servers. 1623 |

1624 |

A clear presentation of query strings, path, protocol, host and URL fragments 1626 |
- simple webserver is presented
- let http = require('http'); is how you import modules in js (won't work in browser console without hacks)
Intro to cookies/sessions

1632 | 1633 |

1634 | …and that's it. The rest of that lecture is a NodeJS project which we'll do later. 1635 |

1636 |

1637 |

1638 |

1639 | 1640 | 1641 |

1642 |

14 [weblab] Workshop 1: Client Side Javascript + Async

1643 |

1644 |

We're working with git clone https://github.com/mit6148-workshops/catbook-workshop1.git
Slide #8 questions: Let's walk through api.js since they only give a brief description. 1647 |
- First, see these slides starting on slide#6 about url parameters from a CMU data science course.
- From the slides: You've seen URLs like these: google.com/url?sa=t&rct=j&source=web…
- Everything after ? is parameters given to the google api. The '&' character seperates them.
- Read the rest of the slides about REST APIs, JSON data

1654 | 1655 |

1656 | api.js explained: 1657 |

1658 |

1659 | function get(endpoint, params, successCallback, failureCallback)
1660 |   These parameters are:
1661 |    - endpoint = the base URL ie: https://google.com
1662 |    - params = whatever you want to send to the API "userID=1000"
1663 |    - successCallback, failureCallback = optional functions that you define/pass as args 
1664 | 
1665 |  const xhr = new XMLHttpRequest();
1666 |    We're making a constant variable xhr, and assigning to it a new object XMLHttpRequest().
1667 |    You can try this in the console: const temp = new XMLHttpRequest() then type temp; to look at the object.
1668 |    MDN has a full list of all it's 'methods', such as XMLHttpRequest.getResponseHeader(); 
1669 | 
1670 |  const fullPath = endpoint + '?' + formatParams(params);
1671 |    Creates constant variable fullPath which will be baseURL + ? + parameters
1672 |    Example: fullPath = https://google.com/?userID=1000
1673 |    Note, another function is called, formatParams(params) which encodes the parameters using encodeURIComponent()
1674 |     - This is basically to escape characters using UTF-8 encoding
1675 | 
1676 |   xhr.open('GET', fullPath, true);
1677 |    Explained here: https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest/open
1678 |    The boolean is an optional parameter that defaults to true because you don't want synchronous requests
1679 |     - Synchronous meaning your script has to wait for a reply, possibly freezing the user's browser
1680 |   
1681 |   xhr.onload = function(err) {
1682 |     if (xhr.readyState === 4) {
1683 |       if (xhr.status === 200) {
1684 |         if (successCallback)
1685 |           successCallback(JSON.parse(xhr.responseText));
1686 |       } else {
1687 |         if (failureCallback)
1688 |         failureCallback(xhr.statusText);}}}
1689 |    
1690 |     Here is where our optional functions are. If the request returns http status 200 (OK) then 
1691 |     the if-branch tests if you actually gave a success function as an argument, if(successCallback) determines if that parameter exists.
1692 |     If you did, then it calls your function with args (JSON.parse(xhr.responseText)), if you didn't then nothing is run. 
1693 | 
1694 |     In the workshop the successCallback function is: function(stories) { console.log(stories) };
1695 |     This will populate the console with xhr.responseText, which is then parsed by JSON.parse: 
1696 |     function(JSON.parse(xhr.responseText)) { console.log(the output of JSON.parse) };
1697 |       
1698 |     Try this yourself in the console, giving no success/failure function: 
1699 |      const API_ENDPOINT_START = 'http://google-catbook.herokuapp.com';
1700 |      let atest = get(API_ENDPOINT_START + "/api/stories", {});
1701 |      atest; (undefined)
1702 |     
1703 |     This is what the slides mean by 'nothing is returned' by the get/post functions, unless you specify success and failure functions in 
1704 |     the params. 
1705 | 
1706 |   xhr.send(null);
1707 |     This is explained here: https://stackoverflow.com/questions/15123839/why-do-we-pass-null-to-xmlhttprequest-send
1708 |

1709 | 1710 |

1711 | Let's also look at their function formatParams(): 1712 |

1713 | 1714 |

1715 |  function formatParams(params) {
1716 |   return Object
1717 |     .keys(params)
1718 |     .map(function(key) {
1719 |       return key+'='+encodeURIComponent(params[key])
1720 |     })
1721 |     .join('&');
1722 | }
1723 |   First, they have broken up the return line for readability: 
1724 |    return Object.keys(params).map(function(key){ return key+'='+encodeURIComponent(params[key]) }).join('&');
1725 | 
1726 |   encodeURIComponent() you can look up on MDN, it's escaping characters. 
1727 |   Object.keys() you can also look up on MDN, it's returning an array of keys:
1728 |    
1729 |    let params2 = {}; //empty dict
1730 |    params2.userID = "100";
1731 |    params2.name = "Bob";
1732 |    console.log(Object.keys(params2));
1733 |     -> (2) ["userID", "name"]
1734 | 
1735 |   let test = Object.keys(params2);
1736 |   test[0];
1737 |    -> "userID"
1738 |   params2["userID"]
1739 |    -> "100"
1740 |   
1741 |  So the return value is print the key, concat '=' with the UTF-8 encoded value of params["key"], and join them with '&':
1742 |  formatParams(params2);
1743 |    -> "userID=100&name=Bob"
1744 |

1745 | 1746 |

More talk/slides about async vs synchronous behavior 1748 |
- clicking on Network while in Chrome Inspector lists the get reqs queue/history
- Guaranteeing order: nested requests/callbacks
- Design around callback hell of deeply nested get requests
First task is to add renderStories() to index.js, and then fill in the skeleton in feed.js for renderstories() 1754 |
- We can see that in index.js there's a main function, underneath where the navbar is rendered add: renderStories();
- Then in feed.js complete function renderStories() with simple output to console 1757 |
  - Refresh index.html and look at the console, the object filled with stories should be there.
  - you would have console.log(a = stories) if you wanted it bound to a variable to play around with

1763 | 1764 |

1765 | I tried a different version of renderStories() than the lectures: 1766 |

1767 |

1768 | function renderStories() {
1769 |     let newDiv = document.getElementById("stories");
1770 |        
1771 |     get(API_ENDPOINT_START + "/api/stories", {}, function(stories) {
1772 |         stories.map(story => newDiv.prepend(storyDOMObject(story)) )});
1773 | }
1774 |

1775 | 1776 |

1777 | I also wrote mine so comments so would be pulled while inside StoryDOMObject function since the needed JSON id field to fetch the comments is in scope. Of course in production you would never do this because of possible 1778 | failures of comment API, which we aren't even handling by omitting the failureCallBack arguments. You likely wouldn't design api.js like they did either. 1779 |

1780 | 1781 |

1782 | function storyDOMObject(storyJSON) {
1783 | 
1784 |     const storyDiv = document.createElement('section');
1785 |     storyDiv.setAttribute('id', storyJSON._id);
1786 |     storyDiv.className = 'story';
1787 | 
1788 |     const cardBody = document.createElement('article');
1789 |     cardBody.className = ('story-content');
1790 |     cardBody.innerText = storyJSON.content
1791 |     storyDiv.appendChild(cardBody);
1792 | 
1793 |     const creator = document.createElement('footer');
1794 |     creator.className = ('story-creator');
1795 |     creator.innerText = storyJSON.creator_name
1796 |     cardBody.appendChild(creator);
1797 | 
1798 |     get(API_ENDPOINT_START + "/api/comment", {parent: storyJSON._id}, function(comments) {
1799 |         comments.map( comment => creator.append(commentDOMObject(comment)) ) }); 
1800 | 
1801 |     return storyDiv;
1802 | }
1803 |

1804 | 1805 |

1806 | You'll probably have to change the default CSS, adding some background color to the comments and stories helps distinguish them as 1807 | the default CSS is terrible. 1808 |

1809 |

1810 | 1811 | 1812 |

1813 |

14.1 [6.170] Web APIs

1814 |

1815 |

1816 | Let's review the slides for web APIs. 1817 |

1818 |

REST: Representational State Transfer
The idea is URLS identify a representation of a resource 1821 |
- Path hierarchies to imply structure
- GET is the only 'safe' call guaranteeing data will not be modified
- APIs provide a contract, add versions to API URLs when changing them 1825 |
  - There will be many more lectures on good API design
AJAX: Async Javascript XML (JSON these days)
XMLHttpRequest() object is covered again 1831 |
- xhr.onload = function() { let response = JSON.parse(xhr.responseText); }
- response handler is registered before making the req as we already know
- xhr.open("GET", "/teas/green", true); boolean is covered again, to make in asynchronous
Continuations (callbacks) are discussed: 1837 |
- get returns immediately, but the callback function isn't run until the response returns
- The continuation is called at some arbitrary point in the future outside the normal call stack
- Cannot capture the result with a normal return statement
- Side effects: introduce additional external state (let greenTeas;), set the state within the continuation 1842 |
  - complexity increases
  - callback hell again
Promises: a proxy object, returned synchronously, for a value that will be determined some time in the future 1848 |
- Also covered in Functional-Light JS with some better examples
- Chains execute async operations one after another by returning a new promise within a Promise.then() 1851 |
  - The idea is to start a new operation with the previous result

1856 | 1857 |

1858 | MDN has an article on Promises and their usage. 1859 |

1860 | 1861 | 1862 |

1863 | // Using Promises for XMLHttpRequest()
1864 | 
1865 | function get(url) {
1866 |   return new Promise(function(resolve,reject){
1867 |     let xhr = new XMLHttpRequest();
1868 | 
1869 |     xhr.onload = function() {
1870 |       resolve(xhr.responseText);
1871 |     };
1872 |     // Make the req
1873 |     xhr.open("GET", url, true);
1874 |     xhr.send();
1875 |    });
1876 | }
1877 |

1878 |

1879 |

1880 | 1881 |

1882 |

14.2 Exercises: Callback Hell & Promises

1883 |

1884 |

1885 | We're asked to load example.com again and using both nested callbacks, and chained promises: 1886 |

1887 |

Set the body background color after 1 second
Set the div background color 1 second after we set the body background
Set the p font color 1 second after we set the div background

1892 | 1893 |

1894 | // nested callback method
1895 | setTimeout( function() {
1896 |   document.body.style.background = 'green';
1897 |     setTimeout( function() {
1898 |       document.querySelector('div').style.background = 'yellow';
1899 |         setTimeout (function() {
1900 |           document.querySelector('p').style.color = 'blue'; }, 
1901 |         1000); }, 
1902 |     1000); }, 
1903 | 1000);
1904 | 
1905 |

1906 | 1907 |

1908 | // Promises method
1909 | let setColorPromise = function(selector, property, color) {
1910 |    return new Promise(function (resolve, reject) {
1911 |      setTimeout(function() {
1912 |       document.querySelector(selector).style[property] = color;
1913 |       resolve();
1914 |      }, 1000); 
1915 |     })
1916 | }
1917 | 
1918 | setColorPromise('body', 'background', 'green')
1919 |  .then(function () { return setColorPromise('div', 'background', 'yellow'); })
1920 |  .then(function () { return setColorPromise('p', 'color', 'blue'); })
1921 |

1922 | 1923 |

1924 | Note that the return statement is needed "Important: Always return results, otherwise callbacks won't catch the result of a previous promise". 1925 | Try removing them and see what happens. 1926 |

1927 |

1928 |

1929 |

1930 | 1931 |

1932 |

15 The Document Object Model (DOM) API standard

1933 |

1934 |

1935 | Let's find out what the DOM API actually is. This spec should be what most browsers have implemented but of course we'd actually have to manually check each browser's documentation. 1936 |

1937 | 1938 |

We begin to read through the WHATWG DOM spec 1940 |
- it's another 'living standard', so unversioned, it keeps adding features and stays the same name. This means you can't find definitive implementations "This browser fully conforms to DOM API v5.1".
- Some standard architecture on trees, ordered sets, selectors, namespaces
DOM Events: 1945 |
- We already know some of this, except for synthetic events: applications dispatching events themselves 1947 |
  - Whoever wrote this filled it with old memes, the absolute state of current year standards documents
  - Sometimes when you click on a definition it navigates via URL fragment to the same paragraph you're already reading
- The entire interface for Event is shown, and a list of methods, plus CustomEvent, EventTarget. 1952 |
  - Interesting note: browsers need to use special legacy pre-activation/cancel algorithms to implement checkbox and radio input elements
  - Another interesting note about dispatching event listeners, for example old DOM api's implementing touch and wheel events which blocked the async scrolling
- There's a very long step by step spec for how browser's can dispatch an event
- In firing events, initializing IDL attributes are mentioned: IDL is Interface Definition Language/Web IDL spec. 1958 |
  - Firing in DOM context: creating, initializing, and dispatching an event.
  - Important: An event signifies an occurence, not an 'action'. They don't start an algorithm they influence an ongoing one with notifications.
  - A section on capture and bubbling, events traverse the tree towards the root in order to find a handler in most cases, in special or defined cases, events start at the root and move towards the event target (the originating tree node), to find a handler.
  - Shadow DOM is also specified, it's a DOM object for things normally not selectable in 'light' DOM, such as play buttons for <video></video>.

1966 | 1967 |

1968 | We finally arrive at 'Introduction to The DOM' in section 4 1969 |

1970 | 1971 |

As we knew markup goes in, a tree filled with HTML elements is constructed in the browser 1973 |
- There's a DOM Viewer which is somewhat more clear than Chrome DevTools
A document tree is described, and tree order
A lot of notes on HTML slots
A large section on how insertion/pre-insertion algorithms work
We get a list of methods we can run on collections, elements and nodes: 1980 |
- node.children returns child elements (seen in the 6.170 slides)
- node.firstElementChild returns the first child that is an element or null otherwise
- node.lastElementChild returns the last child that's an element or null
- briefly reading skimming the DOM options to see what's available
A collection is an object that represents a list of nodes, like NodeList which we already saw or HTMLCollection 1987 |
- NodeList has various methods: NodeList.length, NodeList.item(index), NodeList[index]
- HTMLCollection and NodeList are old-style collections/depreciated, they could disappear from browser APIs
There's mutation observers we can use to log or run callbacks whenever a target mutation occurs
node.CloneNode([deep = true]) will clone the node and return all it's descendants
.createHTMLDocument(title) returns a document w/basic tree constructed including a title 1994 |
- useful for generating HTML should we ever want to do that
A useless feature: hasFeature() originally would report if the user agent supported a given DOM feature, but was so 1998 | unreliable they depreciated it with hasFeature() = true by default to not break legacy pages.
Element nodes are just elements 2000 |
- Long list of intersting element methods: 2002 |
  - insertAdjacent method returns nothing because 'it existed before we had a chance to design it'
Introduction to "DOM Ranges" 2007 |
- Range objects represent a sequence of content within a node tree.

2011 | 2012 |

2013 | In this example we are cutting/concatenating the two text lines into "syndata is awes": 2014 |

2015 | 2016 |

2017 | //Tree:
2018 |     Element: p
2019 |         Element: <img src="insanity-wolf" alt="Little-endian BOM; decode as big-endian!">
2020 |         Text:  CSS 2.1 syndata is 
2021 |         Element: <em>
2022 |             Text: awesome 
2023 |         Text: ! 
2024 | 
2025 | //Range example:
2026 | var range = new Range(),
2027 |     firstText = p.childNodes[1],
2028 |     secondText = em.firstChild
2029 | range.setStart(firstText, 9) // do not forget the leading space
2030 | range.setEnd(secondText, 4)
2031 | // range now stringifies to the aforementioned quote
2032 |

2033 | 2034 |

2035 | The rest of the document covers Traversal interfaces/methods, which we already saw except NodeFilter which we can 2036 | use to iterate over NodeIterator and TreeWalker objects and filter results. 2037 |

2038 |

2039 | 2040 |

2041 |

15.1 The DOM explained via example

2042 |

2043 |

2044 | After reading the spec, the DOM we now know is literally the HTML document, modelled as a tree. Events are (usually) initiated by the user-agent 2045 | (browser) and will in most cases, propagate up the tree by 'bubbling' until the event finds a handler that is listening for it. Since the DOM is the HTML, any 2046 | attributes can be set, and document elements manipulated except for the shadow tree which are elements that are normally not able to be captured, such as the play button on <video> elements. 2047 |

2048 | 2049 | 2050 |

2051 | For example let's look at HTML video. See this question/solution here with the problem trying to get a local .vtt subtitle file to load in a browser. Note that browsers have security settings 2052 | to prevent any loading of local files unless you manually override these settings. The solution was to encode the subtitles in the page as 2053 | data and then create a URL object to attach to the <track> element as the new source of the subtitles. 2054 |

2055 | 2056 |

2057 | The original HTML for embedding the video, to display locally that failed: 2058 |

2059 |

2060 | <video controls preload="none" width="800" height="600" poster="test.jpg">
2061 | <source src="test.mp4" type='video/mp4' />
2062 | <track kind="subtitles" src="test_EN.vtt" srclang="en" label="English"></track>
2063 | <track kind="subtitles" src="test_FR.vtt" srclang="fr" label="French"></track>
2064 | </video>
2065 |

2066 | 2067 |

2068 | The solution was to give <track> an id, select it with Javascript, and fill in the <track> attributes to point to the raw data subtitles: 2069 |

2070 |

2071 | <video controls>
2072 |   <source type="video/mp4" src="videoFile.mp4">
2073 |   <track id="subtitle" kind="subtitles" srclang="en" label="English">
2074 | </video>
2075 | 
2076 | <script>
2077 | var subtitle = "V0VCVlRUDQoNCjENCjAwOjAwOjI4Ljg5NSAtLT4g...";
2078 | subtitle = window.atob(subtitle);
2079 | var subBlob = new Blob([subtitle]);
2080 | var subURL = URL.createObjectURL(subBlob);
2081 | 
2082 | document.getElementById("subtitle").setAttribute("src", subURL);
2083 | </script>
2084 |

2085 | 2086 |

2087 | Since the DOM is the document modelled a different way, we can set the src to a new URL object that points to the data. In fact you could leave off all HTML attributes: <track id="EnglishTrack"> and fill in the rest of the missing attributes by attaching them to that element in the DOM. If you were to open this in your browser, and enable in DevTool settings: "show user agent shadow DOM" you would see the shadow tree in Chrome DevTools for the <video> element how it 2088 | renders for play buttons and other media controls. 2089 |

2090 |

2091 |

2092 |

2093 | 2094 | 2095 |

2096 |

16 [web.lab] Workshop 2: NodeJS: Set up your own server

2097 |

2098 |

2099 | This workshop is using app.js from here which is empty on the master branch (see the completed branch after the workshop). 2100 |

2101 | 2102 | 2103 |

2104 | NodeJS: 2105 |

2106 |

is a js runtime engine
Node Package Manager (NPM): 2109 |
- Many security considerations, the 6.170 course will likely cover some of these like malicious code injected into package dependencies and package change of ownership via selling to some adtech or other shady outfit. You have to carefully control when you update any packages (lockfiles).
- Dependency complexity, npm packages are often micro modules so a package can have hundreds of dependencies 2112 |
  - example: this pkg really has only 1 line of code when you omit all the exceptions. It even imports append-type, another micro module.
  - the tests don't even catch an out of bounds value for (end - start), if you wrote this yourself, you would probably catch it
exclude node modules inside .gitignore file in the project directory
'npm install' will reinstall node modules after cloning a project, or adding a new library

2120 | 2121 |

2122 | Rest of the workshop walks through a basic server. He starts it with 'node app.js' then enter localhost:3000 in your browser. Kill the server from the terminal with CTRL-C. 2123 |

2124 |

2125 |

2126 | 2127 |

2128 |

17 [web.lab] Workshop 2: Catbook using Express

2129 |

2130 |

2131 | We are cloning this repository (run npm install after) 2132 |

2133 | 2134 |

nodemon: this can also be a simple editor after-save-hook if you want or a bash script with 'pkill node' and w/e commands to restart node, assign it to a keyboard shortcut.

2137 | 2138 |

2139 | //emacs.el entry, assuming you're using emacs/js2-mode
2140 | (add-hook 'after-save-hook
2141 |   (defun restart-node()
2142 |     (when (eq major-mode 'js2-mode)
2143 |       (call-process-shell-command "sh /home/username/noderestart.sh &" nil 0))))
2144 |

2145 | 2146 |

2147 | This workshop is a simple walkthough of setting up a router. This is documented on expressjs.com and 2148 | the 'middleware functions' are also documented to understand the req/res objects. 2149 |

2150 |

2151 | 2152 |

2153 |

17.1 [6.170] Recitation: Node & Express

2154 |

2155 |

2156 | Following the slides here 2157 |

2158 | 2159 |

git clone https://github.com/bamazap/url-shortener
we're asked to implement listAll, updateOne, and deleteOne. 2162 |
- the API is documented in routes/shorts.js

2166 | 2167 |

2168 | function listAll(fields) {
2169 |     axios.get('/api/shorts') 
2170 |         .then(function (response) { return showObject(response.data); });
2171 | }         
2172 | 
2173 | function updateOne(fields) {
2174 |     axios.put('/api/shorts/'+ fields.name, fields)
2175 |         .then(showResponse)
2176 |         .catch(showResponse);
2177 | }
2178 | 
2179 | function deleteOne(fields) {
2180 |     axios.delete('/api/shorts/'+ fields.name, fields)
2181 |         .then(showResponse)
2182 |         .catch(showResponse);
2183 | }
2184 |

2185 | 2186 |

2187 | Exercise, use express-session. We haven't covered this yet, I mainly went through github documentation. 2188 | There is a web.lab lecture later on authentication, this is hacky unsafe auth for playing around purposes: 2189 |

2190 | 2191 |

2192 | //add session middleware to app.js
2193 | const session = require('express-session');
2194 | app.use(session({ secret: 'dontUseInProduction'}));
2195 | 
2196 | //update routes/user.js to set the user's username in the session
2197 | //this is prob wrong, it worked anyway
2198 | router.post('/signin', (req, res) => {
2199 |     const sess = req.session;
2200 |     sess.username = req.body.username;
2201 |     res.write('Hello'+ sess.username);
2202 |     res.end('done');
2203 | });
2204 | 
2205 | //update routes/shorts.js to use the session username and populate creator field
2206 | //You add req.session.username anywhere there is a Shorts object being updated
2207 | router.post('/', (req, res) => {
2208 |   if (Shorts.findOne(req.body.name) !== undefined) {
2209 |     res.status(400).json({
2210 |       error: `Short URL ${req.body.name} already exists.`,
2211 |     }).end();
2212 |   } else {
2213 |       const short = Shorts.addOne(req.body.name, req.body.url, req.session.username);
2214 |     res.status(200).json(short).end();
2215 |   }
2216 | });
2217 |

2218 | 2219 |

2220 | Step 6, we're asked to try/think about ways our toy url-shortener is insecure. For example anybody can rewrite the url's to redirect to their malicious 2221 | ad site, and nothing is validated by our api. To see how to do this go through some of the SEI CERT coding standards for other languages 2222 | how they whitelist inputs or look at the documentation of validate.js 2223 |

2224 |

2225 |

2226 |

2227 | 2228 |

2229 |

18 [web.lab] Workshop 3: Hooking it up with MongoDB

2230 |

2231 |

2232 | Relational dbms is briefly covered, but there is an entire open course from CMU if you're interested. 2233 |

2234 | 2235 |

Instead of using the community version I signed up to Atlas to try it 2237 |
- Surprisingly easy, no email verification annoyance or even impossible captcha
- They've already changed the UI since this workshop but easy to figure out
Clone the repository as usual, looks like mongoose-snippets is kill
Schemas are barely talked about but we'll do two recorded lectures from 6.170 about schema design
JSON is used instead of SQL to describe document objects.

2245 | 2246 |

2247 | This is just follow the slides/lecture and implement some basic features, based on other features they've already implemented. If you git reset to stage3 and accidentally run npm start again like I did without changing db.js to your own Atlas credentials you'll end 2248 | up connecting accidentally to the class demo db and populating your front page with 'btw, you forgot to change db.js gj' 2249 |

2250 |

2251 |

2252 | 2253 | 2254 |

2255 |

19 Relational databases

2256 |

2257 |

2258 | None of the SQL slides from 6.170 course will make any sense unless you watch a few lectures from CMU's dbms class such as the intro to the relational model lecture describing foreign keys, normal form, etc. So let's do that now. 2259 | SQL is still one of the most valuable skills. 2260 |

2261 |

2262 | 2263 |

2264 |

19.1 [CMU 15-445] Relational Model

2265 |

2266 |

YouTube - Relational Model (Fall 2019 playlist)

2269 | 2270 |

2271 | This is a course about designing and building a disk storage db management system, but we're going to audit the first couple of lectures. 2272 | Some set notation for relational algebra is used, this can be learned from Chapter 2.1 of this free book or watch some of this lecture, or read through 2273 | this brief tutorial. 2274 |

2275 | 2276 |

Lecture starts around 18:30
Some talk about problems of flat file/csv data implementations
'Relation' means a table
A schema is a definition of what you're storing in the db
A record (row) is a tuple, when you declare a function in a programming language (ex: def fun(x, y)) the parameters are also a tuple
An attribute (column)
In the example for primary keys, the entire key is Artist(id, name, year, country) not just the id number
Foreign key is an attribute from one relation that maps to another relation (a different table)
Relational Algebra is based on sets (union, intersection, difference) 2286 |
- Select: filter ($\sigma$ sigma) in SQL: where clause
- Projection: reorder/manipulate values ($\Pi$ pi) in SQL: SELECT bid-100, aid, FROM R (select operator)
- Union: R $\cup$ R concatenate sets in SQL: UNION ALL
- Intersection: R $\cap$ R contains only the set that appear in both relations, in SQL: INTERSECT
- Difference: R - S a relation that contains only the tuples in the first and not the second relation in SQL: EXCEPT 2292 |
  - relations must have exact same type/attributes
- Product: R x S a cartesian product, in SQL: CROSS JOIN 2296 |
  - give all unique combinations/configurations
- Joins: R $\bowtie$ S generates all tuples that have a common value, in SQL: NATURAL JOIN 2300 |
  - similar to difference except attributes can be different/not shared between both relations

2305 | 2306 |

2307 | His observations on performance/declarative way of doing queries are interesting, which lead into the reasons for SQL existing how it will 2308 | generally be preferred to just do high level queries "Give me this answer" and let the dbms figure out the most optimized way to do so instead of 2309 | giving a specific sequence of instructions. 2310 |

2311 |

2312 | 2313 |

2314 |

19.1.1 [6.170] Relational Databases

2315 |

2316 |

2317 | We can review the slides from MIT's 6.170 on SQL as they will make sense now. 2318 |

2319 | 2320 |

Relation: table
Attributes: the table columns
Tuple: the table rows
Types: attributes (columns) enforce a specific type for all tuples in the relation, except NULL which we already know from the CMU lecture, is a special value that can be found in any tuple
Key: attribute, or set of attributes (remember Author(id, name, year, country)) who's value is unique for every tuple. 2326 |
- some SQL examples are shown

2330 | 2331 |

2332 | If you want a full introduction to SQL, the book for the CMU course Database System Concepts covers it in chapter 3 through 5, and can be found on WorldCat (library near you) or Library Genesis as a pdf. Note Library Genesis 2333 | has numerous different domains to avoid censorship. The simple SQL example templates shown in MIT's 6.170 slides above are basically SQL in a nutshell: SELECT column FROM table WHERE (filter condition). Using commas you can include numerous attributes such as SELECT atr1, atr2, FROM rel1, rel2 where (condition AND condition OR condition). 2334 |

2335 |

2336 |

2337 |

2338 | 2339 |

2340 |

19.2 [CMU 15-445] Advanced SQL

2341 |

2342 |

YouTube - Advanced SQL (Same 2019 playlist)

2345 | 2346 |

2347 | Let's learn something more than basic SQL, and see how doing simple things are difficult since no dbms follows the 1992 SQL standard, it's much like the 2348 | HTML, CSS and DOM standards we looked at where every browser has a different implementation. 2349 |

2350 | 2351 |

Query optimizing is shilled as a high demand skill
SQL name exists because IBM was sued for calling their prototype 'SEQUEL' 2354 |
- Structured English Query Language
Current standard: SQL:2016 2358 |
- nobody actually follows the standard, except SQL-92 standard
SQL is a collection of relational languages 2362 |
- DML: Data Manipulation Lang: insert/update/select
- DDL: Data Definition Lang: define schemas
- DCL: Data Control Lang: security auth
Important: SQL is based on 'bag algebra'. No ordering, allows for duplicates.
Aggregates: take a bag of tuples, return a single value
ex: AVG (average), MIN, MAX, SUM, COUNT = the # of values
Aggregate functions can only be used in the SELECT (filter) output list since you need to select first before running functions on said selection
GROUP BY: Extract information about the aggregates we're computing
HAVING: filters results from aggregates ie: only show me gpa > 3.0 whereas the aggregate is calculating the avg of all gpas
Strings:
Single quotes, usually case sensitive (except MySQL)
LIKE is used for string matching '%' matches any substring (including empty), underscore matches any character
SQL-92 standard defines numerous other string functions, operations like || (cat) 2377 |
- Postgres and Oracle most closely follow the SQL-92 standard
Output Control
We can dump the output to another table, if that table doesn't exist then our dbms will create it for us 2382 |
- again it's a declarative language, you tell it what you want, the system does it for you
We can limit the tuple output, or order it with ORDER BY

2387 | 2388 |

2389 | Now we are covering what Prof Pavlo considers advanced SQL, such as nested queries. Everything in this lecture is documented in standards and dbms 2390 | documentation, we'll just look at the high level overview for these queries. Have the slides open as you watch the lecture, pause to understand them then continue as he races through these topics in the lectures. 2391 |

2392 | 2393 |

2394 | -Nested Queries 2395 |

2396 |

queries inside of queries, take the output of one query use it as the input to another 2398 |
- query optimizers may rewrite the example query as a join
ALL: every tuple must satisfy expression
ANY: must satisfy for at least one row
IN: equivalent to '=ANY()' (is there any tuple that equals my predicate)
EXISTS: at least one row is returned 2405 |
- can be combined w/boolean: NOT EXISTS (nothing is returned)
Can only reference outer query from inside query, not other way around

2410 | 2411 |

2412 | Numerous examples of nested queries are shown, and different ways to rewrite them. The high level as already stated you're taking 2413 | the output of one query and using it as input to another. 2414 |

2415 | 2416 |

2417 | -Window Functions 2418 |

2419 |

Computing some function on tuples, like Aggregates but returns the tuple(s) and the value
SELECT … FUNC-NAME(..) OVER(…)
OVER() clause specifies how to slice up data (or sort it), can also leave blank
PARITION BY to specify how to group things, OVER(PARTITION BY ..)
OVER (ORDER BY ..) sorting results 2425 |
- RANK() produces the rank of the sort order

2429 | 2430 |

2431 | He goes over examples of this, you can open up a local sqlite3 or postgres install and try these queries yourself. 2432 |

2433 |

2434 | 2435 |

2436 |

19.2.1 Common Table Expressions

2437 |

2438 |

2439 | The last example are Common Table Expressions. I made this it's own sub-heading because you'll definitely run into CTEs in the wild. 2440 | The ideal scenario is to strive to compute your answer as a single SQL statement as opposed to multiple statements, because then the dbms can do optimizations behind the scenes for you. With CTEs you can 2441 | declare 'here's everything I want to do' in a single statement. He also gives an example of recursive CTEs. 2442 |

2443 | 2444 |

2445 | WITH cteName AS (
2446 |   SELECT 1
2447 | )
2448 | SELECT * FROM cteName
2449 |

2450 | 2451 |

2452 | This creates a temporary table that the SELECT below the 'WITH .. AS' can reference like it was an already existing table. 2453 |

2454 |

2455 |

2456 |

2457 | 2458 | 2459 |

2460 |

19.3 [CMU 15-445] Normal Forms

2461 |

2462 |

2463 | In later versions of 15-445, Prof Pavlo eliminated the lecture on normal forms entirely, mainly because 'people just wing it, and end up with 3rd Normal Form anyway". 2464 |

2465 | 2466 |

2467 | YouTube - Normal Forms (2017 Playlist) 2468 |

2469 | 2470 |

2471 | The first 10 minutes are a review of the prev lecture, which we skipped about lossless joins, dependency preservation, and avoiding redundancy while performing decomposition. 2472 | They're good to know and I recommend it since you already know relational algebra, you'll easily understand it but for the sole purposes of MIT web.lab and 6.170 we're good with just a normal 2473 | forms review. 2474 |

2475 | 2476 |

There's generally 6 normal forms, most common is 3NF/Boyce-Codd Normal Form(BCNF).
1NF (first normal form) is a single table with atomic attributes
The 4+ normal forms are rare, information theory forms 2480 |
- By default you usually end up in 3NF
1NF: All types must be atomic, no repeating groups 2484 |
- can't have multiple values in columns
- shouldn't repeat groups (customer-name1, customer-name2)
- array types are actually valid, but from a theoretical standpoint this violates 1NF
2NF: Everything must be atomic, without repeating groups (1NF), but non-key attributes must depend on the candidate key 2490 |
- Removing more redundancy
- Considered 'good enough form'
BCNF and lossless joins will make no sense since we didn't cover superkeys by skipping lecture 4 2495 |
- 'decomposition' means splitting a relation (table) into other relations
- There's problems with BCNF anyway and you don't want to use it
3NF: Preserved dependencies but may have some redundant data 2500 |
- As an amateur you'll probably just naturally produce 3NF
- Prof Pavlo claims nobody actually designs a db schema this way (calculating normal forms)
- You end up thinking in terms of objects 2504 |
  - NoSQL drop all data protections (no ACID)
- He argues mongoDB is still 1NF but with an array
- Document data model (NoSQL) blurs line between logical and physical layer 2509 |
  - Problems: you end up now specifying the physical layout of data, no longer declaration model
  - All the problems of the 1970s (System R) returns that were talked about in prev lectures
  - System R was abandoned, too impossible to maintain

2516 | 2517 |

2518 | Writing your queries in a declarative way is the 'best of both worlds' because the dbms will make the most efficient decision. The takeaway 2519 | from these lectures is: use Postgres (or sqlite3 latest version for small embedded dbms), it's the dbms that most conforms to the SQL standard, use CTEs, unless you have a specific model that requires the document data model (NoSQL). 2520 |

2521 |

2522 | 2523 |

2524 |

19.3.1 [6.170] Schema design

2525 |

2526 |

2527 | Now that we know what normal forms are, let's review the 6.170 slides on Normalization and Constraints. 2528 |

2529 | 2530 |

Seperation of concerns: make columns atomic
3NF: Can the value of a column be derived/described, from another column? If so you violate 3NF 2533 |
- Move these into new relations
- Can you 'over normalize'? Yes

2538 |

2539 |

2540 |

2541 | 2542 |

2543 |

19.4 [CMU 15-445] SQL Exercises

2544 |

2545 |

2546 | Let's try the sqlite3 assignment from this homework. Note make sure you get the latest sqlite3 version so you can use features such as window functions. 2547 |

2548 | 2549 |

Q2: List the longest title of each type, along with the running time, use the primary title ASC as a tie-breaker.
We'll need a window function for this, since we want to rank, that way we get back all rankings that are 1 for each type

2553 | 2554 |

2555 | SELECT type, primary_title, runtime_minutes from ( 
2556 |  SELECT *, RANK() OVER(PARTITION BY type ORDER BY runtime_minutes DESC, primary_title ASC) 
2557 |   AS rank FROM titles) AS ranking WHERE ranking.rank = 1;
2558 | 
2559 | ---
2560 | movie|Logistics|51420
2561 | short|Kuriocity|461
2562 | tvEpisode|Téléthon 2012|1800
2563 | tvMiniSeries|Kôya no yôjinbô|1755
2564 | tvMovie|ArtQuench Presents Spirit Art|2112
2565 | tvSeries|The Sharing Circle|8400
2566 | tvShort|Paul McCartney Backstage at Super Bowl XXXIX|60
2567 | tvSpecial|Katy Perry Live: Witness World Wide|5760
2568 | video|Midnight Movie Madness: 50 Movie Mega Pack|5135
2569 | videoGame|Flushy Fish VR: Just Squidding Around|1500
2570 |

2571 | 2572 |

2573 | You don't have to write sql in all caps, it's just so the reader can understand more clearly what's going on. Try removing the primary title ASC out of the PARTITION BY and change ranking.rank to 2. Notice there are a lot of tvEpisodes rows returned 2574 | all with 720mins running time. A tie-breaker is what you decide should be returned in a situation like this, and we were asked to use ascending primary title as tie breaker meaning alphabetical sort, choose the first result. 2575 |

2576 | 2577 |

Q3: Print type and number of associated titles, sort by number of titles in ascending order
Remember aggregates from the slides

2581 | 2582 |

2583 | SELECT type, COUNT(type) AS cnt FROM titles GROUP BY type ORDER BY cnt ASC;
2584 | 
2585 | ---
2586 | tvShort|4075
2587 | videoGame|9044
2588 | tvSpecial|9107
2589 | tvMiniSeries|10291
2590 | tvMovie|45431
2591 | tvSeries|63631
2592 | video|90069
2593 | movie|197957
2594 | short|262038
2595 | tvEpisode|1603076
2596 | 
2597 |

2598 | 2599 |

Q4: Print all decades and number of titles, constructing a string that looks like '2010s'. Sort in desc order with respect to number of titles.
This is again right out of the slides and teaches you that you can group results by almost anything 2602 |
- sqlite uses it's own syntax for substring which you have to look at the documentation for

2606 |

2607 | SELECT SUBSTR(premiered, 1,3) || '0s' AS decade, COUNT(premiered) AS cnt FROM titles WHERE premiered NOT null GROUP BY decade ORDER BY cnt DESC;
2608 | 
2609 | ---
2610 | 2010s|1050732
2611 | 2000s|494639
2612 | 1990s|211453
2613 | 1980s|119258
2614 | 1970s|99707
2615 | 1960s|75237
2616 | 1950s|39554
2617 | 1910s|26596
2618 | 1920s|13153
2619 | 1930s|11492
2620 | 1940s|10011
2621 | 1900s|9586
2622 | 2020s|2492
2623 | 1890s|2286
2624 | 1880s|22
2625 | 1870s|1
2626 |

2627 | 2628 |

2629 | As the date of this writing (Sep 8) I just helped finish somebody's homework since the solutions haven't been posted yet, though mine seems horribly inefficient and you could also do Q2 with JOIN. 2630 | Working through and figuring out these few exercises will definitely help you learn the basics of SQL. The rest of that course can be completed if you take something like CMU's 15-213 or MIT 6.004 to learn about mmap, OS signals, etc. 2631 |

2632 |

2633 |

2634 |

2635 | 2636 |

2637 |

20 [web.lab] Guest Lecture: MIT ODL

2638 |

2639 |

YouTube - IAP 2019 Day 5 - Emerging Opportunities/Trends in Ed-Tech

2642 | 2643 |

2644 | After detouring through SQL we're back at MIT's web lab. This is a guest lecture about ed-tech that's optional. 2645 | Interesting: MITx MicroMasters program in Data Science helps you with admittance to their PhD SES program (you still need 2646 | an undergrad unfortunately). The fastest and cheapest undergrad you can probably do which has regional accreditation 2647 | (national accreditation usually doesn't qualify) is likely WGU, as you can audit the courses as fast as you want. 2648 | However it's $7k USD/yr so you are paying ~$28k total for a piece of paper while essentially teaching yourself everything, with the added bonus 2649 | that you can work F/T and still complete the courses. 2650 |

2651 | 2652 | 2653 |

2654 | Some other MIT specific things are talked about, like the firehose guide which can help you plan course loads. If you don't know already MIT has numerous 2655 | free courses online for no credit self learning or paid for edx.org credentials. 2656 |

2657 |

2658 |

2659 | 2660 |

2661 |

21 [web.lab] Creating a Wireframe

2662 |

2663 |

2664 | I just went through the slides since I've already audited this open CMU course with lectures/slides which will teach you the basics of wireframing and human 2665 | computer interaction. It's a bit overkill with topics like 'Contextual Inquiry Analysis' but much of the info in the slides is excellent, like designing for 2666 | international audiences and exact instructions how to use tools like Balsamiq/InVision/PhotoShop though as per web.lab you can just do this with pencil/paper. 2667 |

2668 | 2669 |

2670 | My criteria for design is not to use those awful bootstrap tablet style home pages or infinity scroll. In general I follow the advice of government 2671 | sites which have so-called design systems (Australia and NZ have these too) that have put actual research into their designs over an entire population. Common sense design advice there like using 2672 | breadcrumbs, avoiding terrible, terrible infinity scroll, making everything dead simple to find etc. The UK design system is open licensed last time I checked, 2673 | and is straight forward Express/Node with their copy-paste CSS and front end features for assistive technology support (JAWS, ZoomText, NVDA etc), plus they maintain it so you don't have to. 2674 |

2675 | 2676 |

2677 | They include a prototyping kit though you would obviously want to change colors to avoid looking exactly like GOV.UK and you would want to design your overall 2678 | site API so a real designer could easily use it should you wish a unique looking site that also has a fallback capability for IE 11 screen readers. 2679 |

2680 |

2681 |

2682 | 2683 |

2684 |

22 [web.lab] Workshop 4: Review and Accounts and Authentication

2685 |

2686 |

YouTube - IAP 2019 Day 5 - Review and Authentication

2689 | 2690 |

2691 | This workshop is 2 hours long, but keep in mind a lot of that is filmed breaks, and students working on things you can skip if you've already completed the task. 2692 |

2693 | 2694 |

Starts with a summary of what we've already done
They walk through comment posting, what the code does 2697 |
- 'this' keyword is breifly described, as written before it is explained in this book though if you write in a functional style, you will never need to use this aware functions.
The authentication lectures start at 48:00
MIT 6.858 is recommended which we'll do when we cover 6.170 security lectures

2703 | 2704 |

2705 | The remainder of this workshop is the standard git reset hard and checkout of various stages, adding passport to catbook and 2706 | describing what the passport code does, describing express sessions. If you still don't understand req/res then read the Express documentation. 2707 |

2708 | 2709 |

2710 | btw, we're officially done the first week. There's only 1 day of workshop left on React, some advice/guest lectures, plus some security reading and other optional lectures from 6.170 2711 |

2712 |

2713 |

2714 | 2715 |

2716 |

23 [web.lab] Modernizing Catbook w/Fetch and Socket.io

2717 |

2718 |

2719 | This is a lecture, not a workshop, that explains the basics of web sockets, promises, mapping over arrays. Most of this we've already done. 2720 |

2721 | 2722 |

Socket.io is introduced 2724 |
- You probably would just want to use standard web sockets since support is now universal across browsers
- Debugging socket.io bugs is a challenge at best, because bugs only show up on some browsers/connections due to the way it's designed
- There's many other web socket APIs
- For the purposes of banging out a MVP/prototype which is what this workshop is for it then socket.io is fine
Promises are introduced, which we already know from 6.170
fetch introduced, api.js rewritten to use promises
feed.js is rewritten using Array.map()/promises

2734 |

2735 |

2736 | 2737 |

2738 |

24 [web.lab] Where to start?

2739 |

2740 |

YouTube - IAP 2019 Day 6 - How to Design a Web App

2743 | 2744 |

2745 | A lecture on how to put it all together yourself. The MIT 6.170 course has multiple recorded lectures available to the public about 2746 | seperation of concerns and abstracting a data model which we'll do after the React lectures from weblab. 2747 |

2748 | 2749 |

Sketch/wirereframe your front page how you'd like it to look
From the sketch list the features you want on paper
Create "endpoints" (routes) on paper
Split up features as much as possible into components
What data do you need to store? Draft a schema (see 6.170 lectures on Abstract Data Design) 2755 |
- Rest of this lecture is competition specific advice, splitting up teamwork

2759 |

2760 |

2761 | 2762 |

2763 |

25 [6.170] Reactive Programming

2764 |

2765 |

2766 | This is our segway into learning React. Slides example the principles of functional programming: first-class functions, declarative, (controlled) side effects, functions 2767 | that return the same value for the same arguments, and immutable objects. 2768 |

2769 |

2770 | 2771 |

2772 |

25.1 Exercises

2773 |

2774 |

2775 | Convert fireSale to a declarative function: 2776 |

2777 | 2778 |

2779 | let toys = [
2780 | {name: "Woody", price: 10},
2781 | {name: "Buzz", price: 15},
2782 | {name: "Rex", price: 3},
2783 | {name: "Slinky", price: 7}];
2784 | 
2785 | function fireSale(minPrice, perc) { 
2786 |   return toys
2787 |   .filter(function(item) {return item.price > minPrice; })
2788 |   .map(function(item) {return {name: item.name, price: item.price * perc }})
2789 | }
2790 |

2791 | 2792 |

2793 | Event streams treat events as a continuous stream of data rather than one-off occurrences that must be captured. 2794 |

2795 |

filtering events based on their properties
throttling the freq of events in a stream
merging individual streams into a single stream 2799 |
- example of lightbulb state changing is given, to encapsulate in a pure function

2803 |

2804 |

2805 | 2806 |

2807 |

25.2 Reactive Programming: Vue.js

2808 |

2809 |

2810 | Brief slides of an example of the model-view-viewmodel. The recitation github for Vue requires a school 2811 | login, so if you want to learn more about Vue there's this. It definitely is a smaller surface area to learn and the community is large. 2812 |

2813 |

2814 |

2815 |

2816 | 2817 |

2818 |

26 [web.lab] React workshop

2819 |

2820 |

YouTube IAP 2019 Day 6 - React Workshop

2823 | 2824 |

2825 | Lecture starts out attempting (again) to define this keyword in JS. React hooks have replaced using class/this. 2826 |

2827 | 2828 |

React JS is a 'front end library to build user interfaces'
State is owned by the component it's in, you can't directly edit state from another component
Properties are passed in from parents down the component tree
A given state has to live at least as high as the parent component that uses that state 2833 |
- Let's look at the brief slides from 6.170 about React to clear up what states/props are, and to clear up what componentDidMount() does
- We can't do the rest of the 6.170 workshop since you need credentials for their github

2838 | 2839 |

2840 | We're back to the weblab workshop which begins around 36:00, we are cloning this repository. Note, the student who runs the workshop is a U.S. Collegiate Top 5 competitive 2841 | figure skater as well as F/T MIT engineering student, so if she has time to learn this curriculum, so do you. 2842 |

2843 | 2844 |

Webpack hot module replacement is explained here as it was glossed over in the lecture 2846 |
- never use this in production for obvious security reasons
Git checkout stephalf will overwrite to their greetings, but you can still see how it hot reloads any new changes(alter and save app.js)
1:22:00 he comments on the gotcha in the 6.170 slides about ClassName vs Class in the html (you have to use ClassName)
this.setState() from 6.170 slides also covered since you can't directly edit state from another component

2853 | 2854 |

2855 | The rest of the workshop is the standard git reset –hard, git checkout <stages>. React has beginner documentation. 2856 |

2857 |

2858 |

2859 | 2860 |

2861 |

27 [web.lab] GraphQL

2862 |

2863 |

2864 | We skipped the Facebook GraphQL lecture because there are no notes/recordings. GraphQL is a query language with syntax similar to JSON with a detailed spec. A good free book about 2865 | it is The Road to GraphQL which is available here. The same author wrote The Road to React and is also free as noted in the previous section. If you did those SQL lectures you can figure out GraphQL. 2866 |

2867 |

2868 |

2869 | 2870 |

2871 |

28 [6.170] Web App Security

2872 |

2873 |

2874 | The definitive guide for these things is The Web Application Hacker's Handbook and the book 2875 | The Tangled Web: A Guide to Security Modern Web Applications. The first book you get a free copy of burpsuite, an old version of Wordpress or something, and you do the book 2876 | by working through it and attacking your sample server (which you should probably put behind credentials/firewall so bots don't automatically attack it). The second book will teach 2877 | you about how there's even risks in CSS parsing, the many pitfalls of generating HTML, cache poisoning and other problems you should be aware of. 2878 |

2879 |

2880 | 2881 |

2882 |

28.1 How to attack systems

2883 |

2884 |

2885 | Some slides from 6.170 on how to attack systems and company protocols. 2886 |

2887 | 2888 |

Social engineering is also problem for domain management or hosting where somebody can claim to be you and reset credentials 2890 |
- Attaching a phone number as Two-Factor ID makes this easy to do via phone number spoofing, don't use phone numbers
The evil 2018 World Cup chrome addon, in general browser addons are dangerous because they can be sold to adtech without you knowing and turn evil silently pushing obfuscated updates
Classic phishing attacks talked about, such as breaking into RSA to attack Lockheed Martin 2895 |
- The introduction of Unicode in domain names opened up the web to phishing attacks, where you are fooled into using similar looking yet attacker run Signal/Telegram servers
Backdooring the development toolchain happened recently with NPM package malware being distributed worldwide

2900 |

2901 |

2902 | 2903 | 2904 |

2905 |

28.2 How to attack a web app

2906 |

2907 |

2908 | Another set of slides from 6.170 and why it's often easier to use a framework than roll your own, as you need to mitigate and be aware of the numerous classic web attacks. 2909 |

2910 | 2911 |

Advice from the slides: 2913 |
- sanitize and escape all inputs
- limit sessions, use multi-level sessions for critical requests like password changing (most sites req you to enter the old pw to create a new one)
- assume all requests can be out-of-order and replayed/modified, check every request 2917 |
  - UDP protocol is a good example, if attacker can reply faster, they win (DNS is UDP)
- encrypt cookies, use TLS
- use prepared statements as in the db.query() example
- form tokens to defeat CSRF
- don't work around same origin policy and defeat your own mitigations

2926 | 2927 | 2928 |

2929 | The slides recommend OWASP but it's a seriously disorganized, corporate driven org where most of the advice is either incomplete or outdated. 2930 | Some of the security cheat sheets are OK such as the Top 10 but you'll be lost if you look into any of their articles on security. The book The Tangled Web ends each chapter with a checklist to follow if you're building an application and 2931 | is probably still the best resource for timeless general advice, which mitigates some of these attacks. There's plenty of security-as-a-service startups around too with their own checklists and web app firewalls. 2932 |

2933 | 2934 |

2935 | Some of the security slides are covered in the next lecture on Abstract Data Design. 2936 |

2937 |

2938 |

2939 | 2940 |

2941 |

28.3 Security lectures

2942 |

2943 |

2944 | Try MIT 6.858 it's fully open w/youtube lectures and was recommended in 6.170. 2945 |

2946 |

2947 | 2948 | 2949 |

2950 |

28.3.1 Reputation, regulatory and maniac proofing

2951 |

2952 |

2953 | Not covered anywhere is security of your reputation from fake reviews, overzelous regulators or maniacs who may target you. 2954 |

2955 | 2956 |

2957 | Reputation security can be defined as protecting yourself from fake reviews on various ratings sites and Google, and protection from negative PR being spread about you on social media. This is something to think 2958 | about when designing your site/service: how will you defeat an organized attempt (likely by your competitors) from attempting to damage your reputation. You will want to be proactive here to not let an attack continue for too long 2959 | where there are hundreds of bad reviews and PR floating around before you've noticed. For example look at some of the Google Playstore reviews, there is salon software where a large percentage of bad reviews are being angry against some hairdresser 2960 | that's using it, nothing to do with the software, driving down the reputation of that app with no response from the developers. Unless you actually went through numerous reviews you'd never know they weren't related to the software. 2961 |

2962 | 2963 | 2964 |

2965 | Regulatory security is protection from a patent troll suing you, or competitors making complaints to some authority. At least be aware of these and ways you may be targeted. For example the domain you choose, the TLD owners may have conditions or restrictions and end up pulling your domain after complaints. 2966 | To avoid patent trolls, incorporate a limited liability company before you launch, and do some kind of research into the design of your site. For example you probably don't know that there's an active patent troll who sues exchange websites that have a design where the highest bid converges with the highest ask in a single column. This is why 2967 | you always see these seperated. 2968 |

2969 | 2970 |

2971 | Finally maniac proofing. Ways to design around maniacs, think about petty sabotage. For example we'll use salon software again. 2972 | Suppose you're writing an app for appointments or reservations. Do you have a plan to mitigate somebody making a hundred fake reservations everyday? Filling the support queue with garbage? Doing dozens of chargebacks? Have you even 2973 | considered how you will mitigate chargebacks? Is there a social feature such as comments allowed on your site, will you shadowban or outright ban? Shadowban being the maniac is allowed to continue posting but their comments and posts 2974 | only show on their end to attempt to fool them into thinking they're just being ignored so they go away eventually to seek attention elsewhere. An outright ban you are letting them know they need to go and use VPNs and other workarounds to continue 2975 | shitposting on your website. The only advice I can think of is as you build something architect your application so sabotaging it cannot be easy to do, such as allowing somebody to fill a day's appointments with fake clients. This doesn't mean you need to be paranoid, just make it hard for a single troll to kill your app. 2976 |

2977 |

2978 |

2979 |

2980 |

2981 | 2982 | 2983 |

2984 |

29 [6.170] Abstract Data Design (Part 1)

2985 |

2986 |

2987 | See the bottom of this list of materials for a recorded lecture on abstract data design (Part 1). If you can't play the lecture try chrome or download them. 2988 | . 2989 |

2990 | 2991 |

Lecture starts with a repeat of the security slides, describing request forgery and other attacks 2993 |
- Mitigations are walked through again as well
19:22 the design lecture starts
key ideas in this data modeling lecture: 2998 |
- seperation of concerns
- abstract structure w/relations and sets (we did this in the SQL lectures)
- using a data model as an invariant

3004 | 3005 |

3006 | 27:06, pause the lecture and actually sketch a data model for the URL shortner. Note where the slides instruct: 3007 | 'design what data is stored, now how'. My terribad model consists of group for private shorts, owner, date, shortcode, URL, some kind of flag that determines when it should expire, 3008 | probably a list of words we don't want to be used as a short code, probably a list of URLs we don't want used such as infamous Last Measure URL. 3009 |

3010 | 3011 |

27:16 lecture begins again, let's see how terrible our models are: 3013 |
- I failed the first review
- I passed the keygen/ids since we already know from the SQL lectures those aren't part of data modeling
- My terrible model can be extended, sort of 3017 |
  - My score is 1.5/3
Seperation of concerns: think of one thing at a time, consider the data then the implementation after
Data modeling syntax is speed run, the idea is identifying the principal kinds of objects and the relationships between them not primitive data details at this stage
The Professor is the author of Alloy, so is using some of it in the lecture

3025 |

3026 |

3027 | 3028 |

3029 |

30 [6.170] Abstract Data Design (Part 2)

3030 |

3031 |

3032 | Part two recorded lecture on data design. 3033 |

3034 | 3035 |

At 14:56 "The labels of a conversation are the union of it's messages". If you remember relational set theory from the SQL lectures, what this means is every distinct message label inside that conversation becomes the conversation label.
If you forgot what the symbols mean for the diagrams, see the cheatsheet
Alloy use here is a 'representation invariant', meaning a way to more strongly type the data model
Deriving a schema from the model: 3040 |
- Is it a primitive value? If no add attributes
- Assign types (string, date, int) to each attribute or primitive value
- Reverse relations that point from one to 'many' to a single thing 3044 |
  - Example given, each shortcut will have one user/owner whereas a user can have many shortcuts so flip the relation
- Break graph into trees which will be the tables
- The tree root becomes the key of the table, sets without primitive types become ids (integer indexes)
Corner cases: many to many 3051 |
- Just create a relation that combines both into a table
Subsets: 3055 |
- split up tables to keep immutables together
- possible to add redundancy, to avoid having to do joins

3060 | 3061 |

3062 | I was actually surprised how easy it was to derive a complete schema, though the odds of me getting the abstract data model correct 3063 | are pretty low. This is just something we'll have to practice. 3064 |

3065 |

3066 |

3067 | 3068 | 3069 |

3070 |

31 [6.170] Conceptual Design (Part 1)

3071 |

3072 |

3073 | You should absolutely watch these lectures. Again scroll to the bottom of this page of materials to find the recorded lectures. 3074 |

3075 | 3076 |

What is a concept 3078 |
- inventive so not just a model
- purposeful, what it's for not an action
- a behavior, so not an entity
- self-contained
- reusable
Building a concept template 3086 |
- name
- purpose: why it matters, justify the effort
- structure: it's data model
- specifications: actions, describe the behavior (effects)
- operational principle: concerete scenarios that motivate the design
- misfits: concept has a behavior that doesn't fit the purpose

3095 |

3096 |

3097 | 3098 |

3099 |

32 [6.170] Conceptual Design (Part 2)

3100 |

3101 |

3102 | You may have to download (3.6G size), google drive error 'Unable to process this video'. I didn't try adding it to my own drive and running cloudconverter 3103 | since I always archive lecture recordings anyway as they tend to disappear. His PhD student used this material to design Gitless, which I didn't know (and have used many times). 3104 |

3105 | 3106 | 3107 |

Concepts should map to a single purpose, and vice-versa 3109 |
- The first 40minutes of lecture is various examples of why 1 concept should map to 1 purpose
What makes a usable concept? 3113 |
- uniform: always the same actions irrespective of context (except for security reasons)
- reuse existing concepts (labels for example, folders/file hierarchy) rather than invent if possible

3118 |

3119 |

3120 | 3121 |

3122 |

33 [6.170] UX Design & Heuristics (Part 1 & 2)

3123 |

3124 |

3125 | This is an interesting 2 lecture series that finishes the design process. 3126 |

3127 | 3128 |

3129 | The first 50 minutes are UX criteria: 3130 |

3131 |

learnability, how easy is your system to figure out without any documentation
efficiency, does this work for power users
error tolerance
pleasantness
safety
security
accessibility

3140 | 3141 |

3142 | The complete design process in brief: 3143 |

3144 |

come up with idea/product purposes 3146 |
- invent concept purpose
- review known concepts
- polish concept design (1 to 1 mapping of concept to purpose)
design the UI 3152 |
- review known elements/UI
- apply heuristic eval
testing

3158 | 3159 |

3160 | Heuristics (methods): 3161 |

3162 |

physical level: fitt's law 3164 |
- keep features close/large
perceptual fusion: avoid delays > 0.1s, or show progress bar
use visual layout to convey structure (grouping and mapping)
language: use simple/specific terms 3170 |
- consistent naming and icons
- don't require memorization
information 'scent': allow user to easily navigate themselves
navigation: 3176 |
- make user's state/location visible (breadcrumbs)
- accelerators: make it fast to reuse common features, minimize steps to them
provide undo/cancel actions 3181 |
- feedback: when you do something let the user know what they've done

3185 | 3186 |

3187 | Surprisingly they don't use https://www.lingscars.com/ as an example of UX. A good example of dark patterns talked about is the site this website is hosted on. 3188 | When you log in and look at the options, 'Buy Domain' is offered: "We offer a growing selection of domains". After purchasing one, 3189 | it redirects you back to this site with the error: "This feature isn't available". If you attempt to use the contact form it gives you an error saying you 3190 | must 'contact us from an email associated with your account' then helpfully doesn't present any addresses to email. 3191 |

3192 |

3193 | 3194 | 3195 | 3196 |

3197 |

33.1 [6.170] Recitation heuristic evaluation

3198 |

3199 |

3200 | The slides for the recitation go through an evaluation of doodle's website, how they used accelerators, how they properly grouped icons, etc. 3201 |

3202 |

3203 |

3204 |

3205 | 3206 |

3207 |

34 [web.lab] Advanced CSS

3208 |

3209 |

YouTube - IAP 2019 Day 8 - Advanced CSS 3211 |
- The 2018 workshop has almost the same lecture

3215 | 3216 |

3217 | The weblab.to links are to a codepen page which no longer works but you can edit CSS directly in 3218 | Chrome/FireFox to do this workshop. We've already done most of this in the 6.170 course and the CSS standard we looked through. 3219 |

3220 | 3221 |

Layouts: display 3223 |
- A site for this exists: learnlayout.com (no CSS grid however)
display: determines how elements size and sit with/in each other
display: inline is porportional to text size
display: inline-block can have it's own size
display: flex we've already covered, justify-content we used in the 6.170 recitations
display: grid we've already seen too
Layouts: position
position: static renders boxes position based on order in document (default value for divs)
position: relative slight modifications in position
position: absolute for navbars or sidebars (or use grid)
position: fixed relative to the screen (navbar) 3236 |
- Never use float, grid/flexbox exists instead
Transitions: start/end 3240 |
- Color changes, position changes, auto or display won't work
Animations 3244 |
- use @keyframes to define a cycle
- predefined selectors such as :hover :selected :active to trigger animation
Responsive Layouts 3249 |
- media queries, we saw these in the stripe.com example where they had different CSS for different devices
- they use the weblab.mit.edu site as an example of responsive, but on the schedule page all the youtube and slide links disappear if you're on a standard mobile screensize
- in the 2018 lecture they note FireFox and Chrome inspector have responsive tools you can use to see your page layout on a smaller screen

3255 | 3256 |

3257 | Interesting lecture where we learned some animations and transitions, the MDN docs cover all this and which browser your CSS effect will work in. Sadly 3258 | IE11 still exists and doesn't even fully support flexbox IIRC, and there's millions of devices out there running outdated android browsers. 3259 |

3260 | 3261 | 3262 |

3263 | A reminder that flexbox is what you use for an unknown amount of items you want to organize in a single row or column like a toolbar with buttons. 3264 | Flexbox doesn't have grid-gap on most browsers, so you need to use the margin hack like we saw in the lecture whereas with CSS grid this is easy to do. 3265 | You would use CSS grid for overall page layout of header, footer, box over here, navbar over here, or like how we saw with stripe.com where they used it 3266 | for background design of colored stripes. Responsive design with grid is dead simple. There's a good stackexchange answer here on where you would want flexbox instead of grid, but keep in mind the next spec of CSS grid 3267 | covers some of these problems such as creating space between the first and last items. That and there exists a ton of flexbox CSS frameworks like spectre. 3268 |

3269 |

3270 |

3271 | 3272 |

3273 |

35 Every single web framework exampled

3274 |

3275 |

3276 | Before we continue with web.lab's react workshop, here is the 'mother of all demo apps' in almost every web framework if you want to see what a real world app looks 3277 | like in framework of choice. If you are wondering why there are so many frameworks, it's because new ECMAScript (ES) features were not available cross-browser, so we needed frameworks in order to use the new ES standard features, and for security reasons, however in 2019 most browsers support all these new features and you 3278 | could, if you really paid attention to the security lectures (and the book Tangled Web), write everything in vanilla JS w/CSS grid and every browser that matters will support it, with some exceptions for the ancient IE screenreaders you would have to deal with. That is until layoutNG shows up since subgrids new proposal wasn't implemented by any browser except Firefox. 3279 |

3280 |

3281 | 3282 |

3283 |

35.1 ClojureScript

3284 |

3285 |

3286 | The most enjoyable front-end web development I've done is definitely with ClojureScript, because like Common Lisp you can interactively program as seen in this example. If you dislike these frameworks, dislike having to rebuild your app because of breaking updates every X months, and dislike the bugs of webpack then try a book on clojure/emacs, and some kind of tutorial for reagent or ClojureScript documentation. That said everything you have done up to this point is needed to understand ClojureScript or any kind of web development anyway so you didn't waste your time. 3287 |

3288 |

3289 |

3290 |

3291 | 3292 | 3293 |

3294 |

36 [web.lab] Let's Make a Game! (React Part 1)

3295 |

3296 |

YouTube - IAP 2019 Day 7 - React Part 1: Building Snake

3299 | 3300 |

3301 | "In React, we structure our code based on what the user interface looks like". If you did the 6.170 lectures on Conceptual design and UX then this part 3302 | of the React design phase will be easy to do. These React workshops are a little confusing, what you mainly need is the overall architecture idea 3303 | of properties (props) being passed down. If you like React you can pick up any book on React like Pro React, latest edition, and it will review everything we've done here but at with more 3304 | detail of what Super(props) is doing and how components are designed, why there is piles of 'this' keywords, how the router works (you could rewrite Router yourself in under 50 lines of code, if 3305 | you don't like their implementation). These workshops build up competence too, after you see all the answers and pass stage 1, git reset back to the beginning and reimplement them yourself all over again. If you still 3306 | don't understand just keep following the workshops (there's 3 of them), you'll eventually pick this up. 3307 |

3308 | 3309 |

Starter code is snake-react from their repository 3311 |
- src/components/game should be client/src/components/game

3315 |

3316 | 3317 |

3318 |

36.1 [web.lab] Let's Make a Game! (React Part 2)

3319 |

3320 |

YouTube - IAP 2019 Day 8 - More React

3323 | 3324 |

3325 | We're cloning same repository as last time, starting with the branch day2-starter 3326 |

3327 | 3328 | 3329 |

grid length is declared in config.js if you're wondering
Array.from is documented here note she added .keys() to it for use as an iterator 3332 |
- Reminder that everything in the map function is repeated for each cell, so 30x

3336 | 3337 |

3338 | Could also write it like this if you want to see the sequence more clearly: 3339 |

3340 | 3341 |

3342 | {
3343 | Array
3344 | .from(Array(GRID_LENGTH)
3345 | .keys()
3346 | .map(x => <Cell .... />))
3347 | }
3348 |

3349 | 3350 |

The key ID in the gameboard.js row render() is just y because we're counting y-axis rows 0-29 whereas cells is x-axis and could end up with non unique keys.

3353 | 3354 |

3355 | Rest of this workshop is standard git reset –hard, git checkout next stage and fill in missing code. Slowly you are getting an idea of how React.js works. 3356 |

3357 |

3358 |

3359 | 3360 |

3361 |

36.2 [web.lab] Let's Make a Game! (React Part 3)

3362 |

3363 |

YouTube - IAP 2019 Day 9 - Wrapping up Snake (React)

3366 | 3367 |

3368 | Short workshop on the architecture of React routing, using the same snake-react git repository as previous lectures. 3369 |

3370 |

3371 |

3372 | 3373 |

3374 |

36.3 [web.lab] Catbook in React

3375 |

3376 |

3377 | The slides are well written to audit this lecture, which will show the differences between a React app architecture and a regular Express/JS app. 3378 |

3379 |

3380 |

3381 |

3382 | 3383 |

3384 |

37 [web.lab] Designing for the Web

3385 |

3386 |

3387 | I also audited the slides for this lecture because we've already done design in 6.170, and selector syntax for CSS. Compiled CSS is introduced such as Sass. 3388 | There's some more good advice here, like styling a form. 3389 |

3390 |

3391 |

3392 | 3393 |

3394 |

38 [web.lab] Deployment

3395 |

3396 |

3397 | Get a recent book on DevOps from libgen.is to see how it's done 'at scale', get a book that covers system administration and read about web security and priv seperation here. 3398 | Rent a fixed price VPS, a cheap dedicated server or deploy your own colocated rack somewhere and avoid having to pay huge amounts of pay-per-use fees in the beginning. 3399 |

3400 | 3401 |

3402 | I skipped the lecture 'How to Deploy' because it's just a Heroku walkthrough, and I recommend you don't ever use Heroku since it goes from free tier to costs a ton of money tier very quickly. 3403 | For the purposes of the MIT competition it's fine since only the class will be using it but if you were to launch a public app billing can get out of control. There is free so-called serverless hosting 3404 | such as zeit.co where you can push static assets direct to their CDN. There is also Google Firebase which has a fixed price monthly plan. There exists a ton of React tutorials for using Firebase as a backend. 3405 | Just be aware Google has a history of surprise product EOL and you also become dependent to vendor lock in with any kind of serverless or Firebase hosting plan. That said it's not easy to maintain a server every single day and keep on top of 3406 | security patches, and fine tune performance problems. My preference for deploying a large amount of servers and maintaining them is Guix System. You write one declarative file 3407 | on how you want to configure the server, and replicate that exact server X times. It has numerous docker like features too so you can invoke guix environment and create 3408 | a container that just contains your app dependencies. My other preference for a single server would be a BSD, because they are simple to grok and well documented how to configure and deploy. OpenBSD is especially easy to use, with it's own webserver. 3409 | Do you have extra time to configure and maintain all of this yourself? If not then paying for something like Heroku or locking yourself into Firebase/Zeit/AWS Lambda may be what's right for you. 3410 |

3411 | 3412 |

3413 | Another option if you're going the colocation route, and you need a lot of concurrent connections is using Elixir and BEAM. Elixir has it's own webserver (Phoenix) which has channels for real-time features. There's numerous books on Elixir web development. 3414 | An auction site would be easy to implement using Elixir/Phoenix in fact there's papers on Google scholar detailing how to implement this (in Erlang, which is trivial to adapt to Elixir). A trading bot, or an exchange website would also be a perfect candidate for Phoenix/Elixir though significant security architecture built 3415 | in order to handle the actual exchanging of assets which is well beyond the scope of this workshop. If you ever decide to do this, the proper way is to bootstrap your exchange with insurance: some public 3rd party who is trusted holds X amount of coins, and your exchange will not accept above that amount. As your exchange grows you dump profits into the insurance, to gradually 3416 | expand daily limits. That way if disaster strikes you aren't left holding an empty bag, and people can trust you from the very beginning so essentially bootstrapping a reputation. 3417 |

3418 |

3419 |

3420 | 3421 |

3422 |

39 [web.lab] Optional Guest Lectures

3423 |

3424 |

3425 | The 2018 schedule has these recordings, there's a good one on Postgres (from an Oracle guy of all people) though covers some of what we already know from the CMU relational dbms lectures, and the React native Bloomberg lecture. The machine learning lecture if you're interested 3426 | you can learn practical machine learning/visualization here and from Cornell's practical machine learning course which assumes you've completed a seperate course on Linear Algebra. 3427 |

3428 |

3429 |

3430 | 3431 |

3432 |

40 [web.lab] Code Quality

3433 |

3434 |

3435 | This is the last lecture in the workshop 3436 |

3437 | 3438 |

YouTube - IAP 2019 Day 10 - Deployment and Good Coding Practices 3440 |
- Lecture for code quality starts at 57:20
Avoid magic numbers 3445 |
- a magic number is a number not assigned to a variable in code that does not have an immediate and clear meaning
- snake-react we used a constant to assign grid length = 30 instead of using '30' everywhere 3448 |
  - a timer function with a number isn't magic because it's meaning is immediately clear whereas injecting '30' into those react workshop functions is magic
Style:
good style helps prevent bugs
AirBnB JavaScript Style guide
linters also exist, many have the AirBnB style already as a ruleset 3456 |
- some editors (emacs) this can be checked in real time as you type
'Don't use git like an ape' forcing push, not checking status before adding.
Debugging:
write tests/examples before writing functions. We barely covered any testing 3462 |
- go through the slides here on interface testing
every function should just have one task
run and test often to avoid having to debug huge amounts of code
never cut + paste, you will miss bugs and propogate them around your code whereas if you retyped it, you'd catch the bug 3468 |
- many famous examples that persisted for years because of repeated cut+paste
sometimes reading the code backwards helps as your brain tends to fill in small typos
read the error message in in the console
if your editor has a javascript mode, and your automatic indentation isn't working you forgot to close something
Chrome/FireFox dev console has a debugger to set breakpoints

3476 | 3477 |

3478 | Lecture goes up to this point then they skip the rest of the slides but the summary is: code in small chunks, make incremental changes, if you write hacky junk just to get 3479 | some feature working write a comment so you can go back later, don't overthink code organization. Note this is for the competition, which they have a week to finish a MVP. 3480 |

3481 |

3482 |

3483 | 3484 |

3485 |

41 We're Done

3486 |

3487 |

3488 |

3489 |

41.1 Experimental webDev

3490 |

3491 |

3492 | Try these notes on programming in OCaml, there exists a way take compiled OCaml code and produce js. 3493 | Skim through the 'API Reference' like how to manipulate the Dom. You get promises (Lwt) and easy to use websockets, css modules and even webgl. 3494 | An interesting MVP is the JStreet incremental Dom influenced by the ELM architecture. webAssembly is an MVP still, things like dom manipulation need to be handled with inline js if using EMSCRIPTEN and c/c++ or writing in Rust. 3495 |

3496 |

3497 |

3498 | 3499 |

3500 |

41.2 Future

3501 |

3502 |

Redo the mother of all demo apps in framework or language of choice, use some book you found on libgen about node.js to deploy your own server to practice with
Try one of those 'serverless' solutions like next.js
Learn whatever you can about easy authentication and the web security model
Learn some UIX design have you ever used a site, program or ATM and thought wow this is awful, why is everything laid out in this confusing way? Every year CMU has a 'human interface' seminar to talk about this field (they made a whole grad program for it), like how you can influence users to certain behaviors with your design.
Try something like Elm or Elixir/Phoenix webserver
Try generating html to some static content delivery network with JAMstack

3510 |

3511 |

3512 |

3513 |