├── .gitattributes ├── .gitignore ├── .vscode └── settings.json ├── Blog └── try_cpp_coroutine.md ├── Books ├── CMU 15-312 FoPL.md ├── CMU 15-410 OS.md ├── CMU 15-411 Compiler.md ├── CMU 15-418 Parallel.md ├── CMU 15-445 DB.md ├── CMU 15-712 AdvSys.md ├── CMU 15-719 AdvCC.md ├── CMU 15-721 AdvDB.md ├── CMU 15-745 AdvCompiler.md ├── CMU 15-814 TaPL.md ├── CMU 15-816 Automated Reasoning.md ├── CMU 15-859 Quantum Computing.md ├── CS143-Compiler-Optimization-1.png ├── HPC.md ├── Stanford CS143 Compiler.md ├── UChicago CMCS312 PL.md ├── book_list.md ├── subtying.hs └── windows_internal.md ├── Idioms ├── alg_ds.mdown ├── cpp_idiom.mdown ├── cs_idiom.mdown ├── dict_template.mdown ├── graphics_idiom.mdown ├── haskell_idiom.mdown ├── linux_idiom.mdown ├── ml_idiom.mdown ├── network_idiom.mdown ├── os_idiom.mdown ├── python_idiom.mdown ├── reverse_idiom.mdown └── rust_idiom.md ├── Miscs ├── 2019 CS Master Application.md ├── ACCU2019.md ├── CASS2018.md ├── CMU_class_candidate.md ├── CppCon2018.md ├── CppCon2019.md ├── CppNow2018.md ├── CppNow2019.md ├── Facebook Production Engineer Questions.pdf ├── LambdaConf2018.md ├── LeetCode.md ├── Unfiled │ ├── 2019年年初至今 药铺电面总结及思路（按照频率排序）+ 8_9新鲜面经_一亩三分地美国面经版.pdf │ ├── Amazon OA with answers.pdf │ ├── Amazon OA1 debug summary.pdf │ ├── Amazon7-9 - PW be0608aeaa6a12e4b25c011ad358c0e8.xlsx │ ├── Amazon7-9非原题 - PW 3ae84b1ac4b4cd8d043f543c19cf585c.docx │ ├── Google 面筋.pdf │ ├── LeetCodeSummary_08.09.2019.xlsx │ ├── Walmart_2019.pdf │ ├── lc-soln.pdf │ └── 狗狗面筋-8月底.pdf ├── cpp_optimization.mdown ├── cpp_quiz.mdown ├── install_haskell_wsl_vscode.md ├── install_klee_pt.md ├── install_ocaml_wsl_vscode.md ├── seen.mdown └── 面经.md ├── Papers ├── Arch │ ├── continual_flow_pipelines.md │ ├── criticality_aware_cache_hierarchy.md │ ├── implementation_precise_interrupt.md │ ├── improving_direct_mapped_cache.md │ ├── instruction_issue_logic.md │ ├── niagara.md │ ├── power_a_first_class_constraint.md │ ├── revisiting_risc_cisc.md │ ├── speculative_multithreaded_processor.md │ └── tesla.md ├── General │ └── hints_for_computer_system_design.md ├── HPC │ ├── charmpy.md │ ├── ligra.md │ └── roofline.md ├── Network │ ├── ATPG.assets │ │ ├── image-20210121122449167.png │ │ ├── image-20210121172831462.png │ │ ├── image-20210121172839554.png │ │ ├── image-20210121200758639.png │ │ ├── image-20210121201007170.png │ │ ├── image-20210121201257442.png │ │ ├── image-20210121201510685.png │ │ ├── image-20210121222729087.png │ │ ├── image-20210121224311262.png │ │ ├── image-20210121225416920.png │ │ ├── image-20210121225617737.png │ │ └── image-20210121231322362.png │ ├── ATPG.md │ ├── Anteater.assets │ │ ├── image-20210131094447449.png │ │ ├── image-20210131102448908.png │ │ ├── image-20210131102626136.png │ │ ├── image-20210131103524926.png │ │ ├── image-20210131111428933.png │ │ ├── image-20210131115152160.png │ │ ├── image-20210131115349189.png │ │ └── image-20210131125559405.png │ ├── Anteater.md │ ├── Batfish.md │ ├── ERA.assets │ │ ├── image-20210123024347035.png │ │ ├── image-20210123173314097.png │ │ ├── image-20210123174550969.png │ │ ├── image-20210123175901871.png │ │ ├── image-20210123182939189.png │ │ ├── image-20210123185815810.png │ │ ├── image-20210123190218493.png │ │ ├── image-20210124000508591.png │ │ ├── image-20210124000731328.png │ │ ├── image-20210124004607151.png │ │ ├── image-20210124011013950.png │ │ ├── image-20210124011849610.png │ │ ├── image-20210124011930487.png │ │ ├── image-20210124170052580.png │ │ ├── image-20210124170440092.png │ │ ├── image-20210124170527875.png │ │ ├── image-20210124170630562.png │ │ ├── image-20210124193054748.png │ │ ├── image-20210124193106627.png │ │ ├── image-20210124193736186.png │ │ ├── image-20210124193749131.png │ │ ├── image-20210124223958231.png │ │ ├── image-20210124224118342.png │ │ ├── image-20210125001344979.png │ │ ├── image-20210125001353677.png │ │ ├── image-20210125001401676.png │ │ ├── image-20210125001409997.png │ │ ├── image-20210125001422797.png │ │ └── image-20210125001431054.png │ ├── ERA.md │ ├── NoD.assets │ │ ├── image-20210120230504275.png │ │ ├── image-20210120231506789.png │ │ ├── image-20210120231911405.png │ │ ├── image-20210120233737352.png │ │ ├── image-20210120235040596.png │ │ ├── image-20210121000846336.png │ │ ├── image-20210121001957603.png │ │ ├── image-20210121005410565.png │ │ ├── image-20210121010421833.png │ │ └── image-20210121011851576.png │ ├── NoD.md │ ├── cnet_verifier.assets │ │ ├── image-20210203232413997.png │ │ ├── image-20210203233242389.png │ │ ├── image-20210204154839422.png │ │ ├── image-20210204154856081.png │ │ ├── image-20210204154903539.png │ │ ├── image-20210204154947683.png │ │ ├── image-20210204154954601.png │ │ ├── image-20210204155545387.png │ │ ├── image-20210204162134150.png │ │ └── image-20210205011318212.png │ ├── cnet_verifier.md │ ├── design_philosophy_of_darpa_internet_protocols.md │ ├── designing_a_global_name_service.md │ ├── development_of_the_Domain_DNS.md │ ├── end_to_end_argument_in_system_design.md │ ├── general_configuration_analysis.assets │ │ ├── image-20210120165135228.png │ │ ├── image-20210120171023415.png │ │ ├── image-20210120171729310.png │ │ ├── image-20210120180023717.png │ │ ├── image-20210120180254555.png │ │ ├── image-20210120180632467.png │ │ └── image-20210120180953385.png │ ├── header_space_analysis.assets │ │ ├── image-20210105145551899.png │ │ ├── image-20210105221939407.png │ │ ├── image-20210105222004883.png │ │ ├── image-20210105224855086.png │ │ ├── image-20210105225602775.png │ │ ├── image-20210105230343079.png │ │ ├── image-20210105232219270.png │ │ ├── image-20210105235319808.png │ │ ├── image-20210105235404772.png │ │ ├── image-20210106001134994.png │ │ ├── image-20210106161212710.png │ │ └── image-20210106161554796.png │ ├── header_space_analysis.md │ ├── hsa_policy_checking.assets │ │ ├── image-20210106234344365.png │ │ ├── image-20210107221741911.png │ │ ├── image-20210107222448466.png │ │ ├── image-20210107223021905.png │ │ ├── image-20210107223046395.png │ │ ├── image-20210107224004179.png │ │ └── image-20210107230109859.png │ ├── hsa_policy_checking.md │ ├── minesweeper.assets │ │ ├── image-20210205161334598.png │ │ ├── image-20210205162428277.png │ │ ├── image-20210205164729100.png │ │ ├── image-20210205194648053.png │ │ ├── image-20210206025430507.png │ │ ├── image-20210206025435424.png │ │ ├── image-20210206031333071.png │ │ ├── image-20210206043305609.png │ │ ├── image-20210206043345695.png │ │ ├── image-20210206235346981.png │ │ ├── image-20210206235417337.png │ │ ├── image-20210206235428647.png │ │ ├── image-20210207001147433.png │ │ ├── image-20210207003034023.png │ │ ├── image-20210207003050017.png │ │ ├── image-20210207003057877.png │ │ ├── image-20210207003108662.png │ │ ├── image-20210207003116687.png │ │ ├── image-20210207003127007.png │ │ ├── image-20210207003136394.png │ │ ├── image-20210207003149887.png │ │ ├── image-20210207003159888.png │ │ ├── image-20210207003208122.png │ │ ├── image-20210207003220166.png │ │ ├── image-20210207003232616.png │ │ ├── image-20210207003243302.png │ │ ├── image-20210207003252833.png │ │ ├── image-20210207003301031.png │ │ ├── image-20210207003311311.png │ │ ├── image-20210207004014127.png │ │ └── image-20210207004137192.png │ ├── minesweeper.md │ ├── packet_reconfigurable_switches.md │ ├── propane.md │ ├── rcc.assets │ │ ├── image-20210202144709045.png │ │ ├── image-20210202145323485.png │ │ ├── image-20210202145329998.png │ │ ├── image-20210202150247067.png │ │ ├── image-20210202153207318.png │ │ ├── image-20210203143150293.png │ │ ├── image-20210203150235006.png │ │ ├── image-20210203150300339.png │ │ ├── image-20210203152103675.png │ │ ├── image-20210203152438117.png │ │ ├── image-20210203154005418.png │ │ └── image-20210203154042942.png │ ├── rcc.md │ ├── software_dataplane_verification.assets │ │ ├── image-20210122005045964.png │ │ ├── image-20210122010409744.png │ │ ├── image-20210122135540324.png │ │ ├── image-20210122135551744.png │ │ ├── image-20210122135638021.png │ │ └── image-20210122141327990.png │ ├── software_dataplane_verification.md │ ├── symmetry_scaling.assets │ │ ├── image-20210112235226836.png │ │ ├── image-20210113004232931.png │ │ ├── image-20210113125542556.png │ │ ├── image-20210113153742255.png │ │ ├── image-20210113154622950.png │ │ ├── image-20210113154739352.png │ │ ├── image-20210113154747608.png │ │ ├── image-20210113161837703.png │ │ ├── image-20210113161932968.png │ │ ├── image-20210113162058768.png │ │ ├── image-20210113162106048.png │ │ ├── image-20210113162115912.png │ │ ├── image-20210113172154265.png │ │ ├── image-20210113211338272.png │ │ ├── image-20210113211400960.png │ │ ├── image-20210113211623064.png │ │ ├── image-20210113211651055.png │ │ ├── image-20210113211703729.png │ │ ├── image-20210113211711962.png │ │ ├── image-20210113211737082.png │ │ ├── image-20210113211813177.png │ │ ├── image-20210113211842081.png │ │ ├── image-20210113211851968.png │ │ ├── image-20210113212212917.png │ │ ├── image-20210113212246714.png │ │ ├── image-20210113224416010.png │ │ ├── image-20210113224634353.png │ │ ├── image-20210113230000096.png │ │ └── image-20210113230019729.png │ ├── symmetry_scaling.md │ ├── verification_atomic_predicate.assets │ │ ├── image-20210114153806105.png │ │ ├── image-20210114161253712.png │ │ ├── image-20210114173906214.png │ │ ├── image-20210114175239648.png │ │ ├── image-20210114175300607.png │ │ ├── image-20210114183110801.png │ │ ├── image-20210114183201016.png │ │ ├── image-20210114183559714.png │ │ ├── image-20210114183728128.png │ │ ├── image-20210114203109274.png │ │ ├── image-20210114222633425.png │ │ └── image-20210114224303527.png │ └── verification_atomic_predicate.md ├── OS │ ├── Xen.md │ ├── barrelfish.md │ ├── dandelion.md │ ├── exokernel.md │ └── the_unix_time_sharing_system.md ├── PLT │ ├── AutoFDO.md │ ├── class_hierarchy_optimization.md │ ├── closure_analysis_in_constraint.md │ ├── fast_aliasing_analysis_cla.md │ ├── fast_static_analysis_c++_virtual.md │ ├── object_oriented_type_inference.md │ ├── optimal_register_allocation_for_SSA.md │ ├── register_allocation_chordal.md │ ├── scalable_propagation_call_graph.md │ ├── smalltalk80.md │ └── the_locality_principle.md ├── SE │ ├── Augur.md │ ├── bug_reproduction.md │ ├── magpie.md │ └── microreboot.md ├── Security │ ├── How_DH_fails_in_practice.md │ ├── foreshadow.md │ ├── klee.md │ └── symbolic_execution.md ├── System │ ├── azure.md │ ├── bigtable.md │ ├── bloat_aware.md │ ├── borg.md │ ├── broom.md │ ├── distributed_aggregation.md │ ├── drizzle.md │ ├── dryad.md │ ├── dryadlinq.md │ ├── flink.md │ ├── flumejava.md │ ├── gfs.md │ ├── graphchi.md │ ├── gridgraph.md │ ├── hdfs.md │ ├── hive.md │ ├── kafka.md │ ├── late.md │ ├── mapreduce.md │ ├── mapreduce_merge.md │ ├── mapreduce_online.md │ ├── mesos.md │ ├── naiad.md │ ├── parameter_server.md │ ├── pregel.md │ ├── project_adam.md │ ├── ray.md │ ├── rdd.md │ ├── scope.md │ ├── spanner.md │ ├── spark.md │ ├── spark_sql.md │ ├── sparrow.md │ ├── storm.md │ ├── structured_streaming.md │ ├── sve.md │ ├── tachyon.md │ ├── tensorflow.md │ ├── trill.md │ ├── tvm.md │ ├── xstream.md │ ├── yak.md │ └── yarn.md ├── conference_list.md ├── conference_list.pdf └── template.md ├── Proposals ├── Rust_rfc.mdown ├── ghc_proposal.mdown └── python_pep.mdown ├── ReadRust ├── intro.md ├── liballoc.md ├── libsyntax.md └── miscs.md ├── awesome_sites.mdown ├── haskell_semantic.md └── readme.md /.gitattributes: -------------------------------------------------------------------------------- 1 | * text=auto eol=lf 2 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .vscode/* 2 | .vscode/ -------------------------------------------------------------------------------- /.vscode/settings.json: -------------------------------------------------------------------------------- 1 | { 2 | "files.associations": { 3 | "xiosbase": "cpp", 4 | "xlocale": "cpp", 5 | "vector": "cpp", 6 | "cmath": "cpp", 7 | "cstddef": "cpp", 8 | "cstdint": "cpp", 9 | "cstdio": "cpp", 10 | "cstdlib": "cpp", 11 | "cstring": "cpp", 12 | "cwchar": "cpp", 13 | "exception": "cpp", 14 | "functional": "cpp", 15 | "initializer_list": "cpp", 16 | "ios": "cpp", 17 | "iosfwd": "cpp", 18 | "iostream": "cpp", 19 | "istream": "cpp", 20 | "limits": "cpp", 21 | "list": "cpp", 22 | "memory": "cpp", 23 | "new": "cpp", 24 | "ostream": "cpp", 25 | "stdexcept": "cpp", 26 | "streambuf": "cpp", 27 | "system_error": "cpp", 28 | "tuple": "cpp", 29 | "type_traits": "cpp", 30 | "typeinfo": "cpp", 31 | "unordered_map": "cpp", 32 | "utility": "cpp", 33 | "xfacet": "cpp", 34 | "xfunctional": "cpp", 35 | "xhash": "cpp", 36 | "xlocinfo": "cpp", 37 | "xlocnum": "cpp", 38 | "xmemory": "cpp", 39 | "xmemory0": "cpp", 40 | "xstddef": "cpp", 41 | "xstring": "cpp", 42 | "xtr1common": "cpp", 43 | "xutility": "cpp" 44 | } 45 | } -------------------------------------------------------------------------------- /Books/CS143-Compiler-Optimization-1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Books/CS143-Compiler-Optimization-1.png -------------------------------------------------------------------------------- /Books/book_list.md: -------------------------------------------------------------------------------- 1 | # Book List (of what I have read) 2 | 3 | 4 | 5 | [TOC] 6 | 7 | --- 8 | 9 | ## Type and Programming Language 10 | 11 | 12 | 13 | Notes in AHaskellLib 14 | 15 | 16 | 17 | ## 计算机体系结构：量化研究方法 18 | 19 | 20 | 21 | Notes in OneDrive. 22 | 23 | 24 | 25 | ## 计算机体系结构：软硬件接口 26 | 27 | 28 | 29 | Notes in EECS370 30 | 31 | 32 | 33 | ## 多处理器编程的艺术 34 | 35 | 36 | 37 | Notes in OneDrive 38 | 39 | 40 | 41 | ## Inside Python Virtual Machine 42 | 43 | 44 | 45 | Notes in OneDrive 46 | 47 | 48 | 49 | ## What every programmer should know about memory? 50 | 51 | 52 | 53 | Notes in PDF 54 | 55 | 56 | 57 | ## Database Redbook 5th edition 58 | 59 | 60 | 61 | Notes in PDF (papers left unread) 62 | 63 | 64 | 65 | ## A primer on Memory Consistency and Cache 66 | 67 | 68 | 69 | 翻译版, No Notes 70 | 71 | 72 | 73 | ## Computer Organization and Design (ARM edition) 74 | 75 | 76 | 77 | Notes in EECS370 78 | 79 | 80 | 81 | ## 深入理解Windows内部原理 82 | 83 | 84 | 85 | 课程, prev WIndows Vista, Notes in `./windows_internal.md` 86 | 87 | 88 | 89 | ## Agner Optimization Manuals 90 | 91 | 粗读, No Notess 92 | 93 | * calling_conventions 94 | * instruction_table 95 | * microarchitecture 96 | * optimizing_assembly 97 | * optimizing_cpp 98 | 99 | 100 | 101 | ## Programming Language Paradigms 102 | 103 | 104 | 105 | 中文版, 程序语言实践之路, No Notes 106 | 107 | 108 | 109 | ## Concepts of Programming Language 110 | 111 | 112 | 113 | 中文版，程序设计概念, No Notes 114 | 115 | 116 | 117 | ## GCC核心源代码 118 | 119 | 120 | 121 | 台湾学生毕业论文, No Notes 122 | 123 | 124 | 125 | ## 软件调试 (及补编) 126 | 127 | 128 | 129 | No Notes 130 | 131 | 132 | 133 | ## 格蠢汇编 134 | 135 | 136 | 137 | No Notes 138 | 139 | 140 | 141 | ## 加密与解密 142 | 143 | 144 | 145 | No Notes 146 | 147 | 148 | 149 | ## C++反汇编与逆向分析技术揭秘 150 | 151 | 152 | 153 | No Notes 154 | 155 | 156 | 157 | ## C++ Concurrency in Action 158 | 159 | 160 | 161 | No Notes 162 | 163 | 164 | 165 | ## C++17 STL Cookbook 166 | 167 | 168 | 169 | No Notes 170 | 171 | 172 | 173 | ## C++ Template The Complete Guide 1st/2nd Edition 174 | 175 | 176 | 177 | No Notes 178 | 179 | 180 | 181 | ## Modern C++ Design: Generic Programming and Design Patterns 182 | 183 | 184 | 185 | by Andrei Alexandrescu, No Notes (`TypeList`?) 186 | 187 | 188 | 189 | ## Standard C++ FAQ (+FQA) 190 | 191 | 192 | 193 | No Notes. 194 | 195 | 196 | 197 | ## (More) Effective (Modern) C++ 198 | 199 | No Notes 200 | 201 | 202 | 203 | ## Exceptional C++ 204 | 205 | No Notes 206 | 207 | 208 | 209 | ## Cpp Core Guidelines 210 | 211 | 212 | 213 | In Github, no notes. 214 | 215 | 216 | 217 | ## More C++ Idioms 218 | 219 | 220 | 221 | In Wiki, no notes. 222 | 223 | 224 | 225 | ## Design and evolution of c++ 226 | 227 | 228 | 229 | No Notes. 230 | 231 | 232 | 233 | ## WTF Python 234 | 235 | 236 | 237 | In Github, no notes 238 | 239 | 240 | 241 | ## Fluent Python 242 | 243 | 244 | 245 | No Notes 246 | 247 | 248 | 249 | ## Python2.6源码阅读 250 | 251 | 252 | 253 | No Notes 254 | 255 | 256 | 257 | ## x86从实模式到保护模式 258 | 259 | 260 | 261 | No Notes 262 | 263 | 264 | 265 | ## What I wish I knew when I learn Haskell 266 | 267 | 268 | 269 | Notes in AHaskellLib 270 | 271 | 272 | 273 | 274 | 275 | ## 精通正则表达式 276 | 277 | 278 | 279 | No Notes 280 | 281 | 282 | 283 | ## Rust编程之道 284 | 285 | 286 | 287 | No Notes 288 | 289 | 290 | 291 | ## 计算机网络 (Tanenbaum) 292 | 293 | 294 | 295 | AKA VE489, Notes in OneDrive 296 | 297 | 298 | 299 | ## Modern Programming Language, A Practical Introduction 300 | 301 | 302 | 303 | AKA COM SCI 131, Notes in OneDrive 304 | 305 | 306 | 307 | ## Learning Rust With Entirely Too Many Linked Lists 308 | 309 | 310 | 311 | Notes in `Airtnp.github.io` repo -------------------------------------------------------------------------------- /Idioms/dict_template.mdown: -------------------------------------------------------------------------------- 1 | # Cpp Idioms 2 | 3 | ## \# 4 | 5 | 6 | 7 | ## A 8 | 9 | 10 | ## B 11 | 12 | 13 | 14 | ## C 15 | 16 | 17 | ## D 18 | 19 | 20 | 21 | ## E 22 | 23 | 24 | ## F 25 | 26 | 27 | 28 | ## G 29 | 30 | 31 | ## H 32 | 33 | 34 | 35 | ## I 36 | 37 | 38 | 39 | ## J 40 | 41 | 42 | 43 | ## K 44 | 45 | 46 | ## L 47 | 48 | 49 | 50 | ## M 51 | 52 | 53 | 54 | ## N 55 | 56 | 57 | 58 | ## O 59 | 60 | 61 | ## P 62 | 63 | 64 | 65 | ## Q 66 | 67 | 68 | 69 | ## R 70 | 71 | 72 | 73 | ## S 74 | 75 | 76 | 77 | ## T 78 | 79 | 80 | 81 | ## U 82 | 83 | ## V 84 | 85 | 86 | 87 | ## W 88 | 89 | 90 | 91 | ## X 92 | 93 | 94 | 95 | ## Y 96 | 97 | 98 | 99 | ## Z 100 | 101 | -------------------------------------------------------------------------------- /Idioms/graphics_idiom.mdown: -------------------------------------------------------------------------------- 1 | # Graphics Idioms 2 | 3 | ## \# 4 | 5 | 6 | 7 | ## A 8 | 9 | 10 | ## B 11 | 12 | 13 | 14 | ## C 15 | 16 | 17 | ## D 18 | 19 | 20 | 21 | ## E 22 | 23 | 24 | ## F 25 | 26 | * 分离轴定律 27 | 28 | ## G 29 | 30 | 31 | ## H 32 | 33 | 34 | 35 | ## I 36 | 37 | 38 | 39 | ## J 40 | 41 | 42 | 43 | ## K 44 | 45 | 46 | ## L 47 | 48 | 49 | 50 | ## M 51 | 52 | 53 | 54 | ## N 55 | 56 | 57 | 58 | ## O 59 | 60 | 61 | ## P 62 | 63 | 64 | 65 | ## Q 66 | 67 | 68 | 69 | ## R 70 | 71 | 72 | 73 | ## S 74 | 75 | 76 | 77 | ## T 78 | 79 | 80 | 81 | ## U 82 | 83 | ## V 84 | 85 | 86 | 87 | ## W 88 | 89 | 90 | 91 | ## X 92 | 93 | 94 | 95 | ## Y 96 | 97 | 98 | 99 | ## Z 100 | 101 | -------------------------------------------------------------------------------- /Idioms/haskell_idiom.mdown: -------------------------------------------------------------------------------- 1 | # Haskell Idioms 2 | 3 | ## \# 4 | * blog 5 | * + [GhaSShee](https://ghassheee.github.io/) 6 | 7 | ## A 8 | * [accursedUnutterablePerformIO](https://github.com/haskell/bytestring/blob/master/Data/ByteString/Internal.hs#L566) 9 | * + [reddit-discussion](https://www.reddit.com/r/haskell/comments/7ykrv3/xkcds_prediction_for_a_haskellrelated_cve_in_2018/duhfa2h/) 10 | 11 | ## B 12 | 13 | 14 | 15 | ## C 16 | 17 | 18 | ## D 19 | 20 | 21 | 22 | ## E 23 | * Extensions 24 | * + [quiz](https://codegolf.stackexchange.com/questions/153744/wait-what-language-is-this) 25 | * Expression problem 26 | * + [ref](https://eli.thegreenplace.net/2016/the-expression-problem-and-its-solutions/) 27 | 28 | 29 | ## F 30 | 31 | 32 | 33 | ## G 34 | 35 | 36 | ## H 37 | 38 | 39 | 40 | ## I 41 | * ImplicitParams 42 | * + `f :: (?x::Int) -> a -> b` and `let ?x = Bool in f` 43 | * + [ref](https://stackoverflow.com/questions/11141600/implicit-parameter-and-function) 44 | 45 | 46 | ## J 47 | 48 | 49 | 50 | ## K 51 | 52 | 53 | ## L 54 | * LambdaCase 55 | * + `\x -> case x of` ==> `\case` 56 | * + [`{- LANGUAGE LambdaCase -}`](http://storm-country.com/blog/LambdaCase) 57 | 58 | 59 | ## M 60 | * MultiParamTypeClasses 61 | ``` 62 | class Monad m => VarMonad m v where 63 | new :: a -> m (v a) 64 | get :: v a -> m a 65 | put :: v a -> a -> m () 66 | ``` 67 | * [ref](https://wiki.haskell.org/Multi-parameter_type_class) 68 | * [ref-2](https://ocharles.org.uk/blog/posts/2014-12-13-multi-param-type-classes.html) 69 | 70 | 71 | ## N 72 | * class Num now not require Show & Ord 73 | * + [ref](https://stackoverflow.com/questions/46994665/typeclass-constraints-necessary-for-a-function-that-prints-integers-not-greater) 74 | 75 | 76 | ## O 77 | 78 | 79 | ## P 80 | * parametricity 81 | * + [ref](https://www.well-typed.com/blog/2015/05/parametricity/) 82 | * + [ref](https://www.well-typed.com/blog/2015/08/parametricity-part2/) 83 | * + [paper](http://www.cs.bham.ac.uk/~udr/papers/logical-relations-and-parametricity.pdf) 84 | * 85 | 86 | 87 | 88 | ## Q 89 | 90 | 91 | 92 | ## R 93 | 94 | 95 | 96 | ## S 97 | 98 | 99 | 100 | ## T 101 | 102 | 103 | 104 | ## U 105 | * UndecidedInstance 106 | * + Paterson condition 107 | * + - [ref](https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/glasgow_exts.html#instance-termination) 108 | * + Coverage condition 109 | * + - [ref](https://stackoverflow.com/questions/13538937/the-coverage-condition-fails) 110 | 111 | 112 | ## V 113 | 114 | 115 | 116 | ## W 117 | 118 | 119 | 120 | ## X 121 | 122 | 123 | 124 | ## Y 125 | 126 | 127 | 128 | ## Z 129 | 130 | -------------------------------------------------------------------------------- /Idioms/linux_idiom.mdown: -------------------------------------------------------------------------------- 1 | # Linux Idioms 2 | 3 | ## \# 4 | 5 | * | : pipe 6 | * + command1 | command2 : command1 -> content -> command2 7 | * >/>/<< 输出输入重定向 8 | * >&/2>/2>> 错误重定向 9 | * [good-blog](http://ytliu.info/blog/2015/01/24/zhi-shi-zheng-li-%282015-dot-1%29/) 10 | 11 | ## A 12 | 13 | 14 | ## B 15 | 16 | 17 | 18 | ## C 19 | 20 | 21 | ## D 22 | 23 | 24 | 25 | ## E 26 | 27 | 28 | ## F 29 | 30 | * file descriptor 31 | * + 0 : stdin 32 | * + 1 : stdout 33 | * + 2 : stderr 34 | * + process independent 35 | * file preread 36 | * + [ref](http://www.d-kai.me/linux%E6%96%87%E4%BB%B6%E7%B3%BB%E7%BB%9F%E9%A2%84%E8%AF%BB%E4%B8%80/) 37 | 38 | ## G 39 | 40 | 41 | ## H 42 | 43 | 44 | 45 | ## I 46 | * IO 47 | * + [ref](http://www.d-kai.me/linux-io%E8%AF%B7%E6%B1%82%E5%A4%84%E7%90%86%E6%B5%81%E7%A8%8B-1/) 48 | * + [IO-stack](http://www.d-kai.me/linux-%E7%9A%84io%E5%B1%82%E6%AC%A1/) 49 | 50 | 51 | ## J 52 | 53 | 54 | 55 | ## K 56 | 57 | 58 | ## L 59 | 60 | 61 | 62 | ## M 63 | 64 | 65 | 66 | ## N 67 | 68 | 69 | 70 | ## O 71 | 72 | 73 | ## P 74 | 75 | 76 | 77 | ## Q 78 | 79 | 80 | 81 | ## R 82 | 83 | 84 | 85 | ## S 86 | * system call 87 | * + int 0x80 88 | * + sysenter (i586) 89 | * + - [how-to-use-sysenter](https://reverseengineering.stackexchange.com/questions/2869/how-to-use-sysenter-under-linux) 90 | * + call *%gs:0x10 (vdso trampoline) 91 | * + - [fs-and-gs-with-TLS-entries](https://stackoverflow.com/questions/6611346/how-are-the-fs-gs-registers-used-in-linux-amd64) 92 | * + - i386 gs: 93 | ```c 94 | typedef struct { 95 | void *tcb; /* gs:0x00 Pointer to the TCB. */ 96 | dtv_t *dtv; /* gs:0x04 */ 97 | void *self; /* gs:0x08 Pointer to the thread descriptor. */ 98 | int multiple_threads; /* gs:0x0c */ 99 | uintptr_t sysinfo; /* gs:0x10 Syscall interface */ 100 | uintptr_t stack_guard; /* gs:0x14 Random value used for stack protection */ 101 | uintptr_t pointer_guard;/* gs:0x18 Random value used for pointer protection */ 102 | int gscope_flag; /* gs:0x1c */ 103 | int private_futex; /* gs:0x20 */ 104 | void *__private_tm[4]; /* gs:0x24 Reservation of some values for the TM ABI. */ 105 | void *__private_ss; /* gs:0x34 GCC split stack support. */ 106 | } tcbhead_t; 107 | ``` 108 | * + - amd64 gs: 109 | ```c 110 | typedef struct 111 | { 112 | void *tcb; /* Pointer to the TCB. Not necessarily the 113 | thread descriptor used by libpthread. */ 114 | dtv_t *dtv; 115 | void *self; /* gs:0x18 Pointer to the thread descriptor. */ 116 | int multiple_threads; 117 | int gscope_flag; 118 | uintptr_t sysinfo; 119 | uintptr_t stack_guard; 120 | uintptr_t pointer_guard; 121 | unsigned long int vgetcpu_cache[2]; 122 | # ifndef __ASSUME_PRIVATE_FUTEX 123 | int private_futex; 124 | # else 125 | int __glibc_reserved1; 126 | # endif 127 | int __glibc_unused1; 128 | /* Reservation of some values for the TM ABI. */ 129 | void *__private_tm[4]; 130 | /* GCC split stack support. */ 131 | void *__private_ss; 132 | long int __glibc_reserved2; 133 | /* Must be kept even if it is no longer used by glibc since programs, 134 | like AddressSanitizer, depend on the size of tcbhead_t. */ 135 | __128bits __glibc_unused2[8][4] __attribute__ ((aligned (32))); 136 | 137 | void *__padding[8]; 138 | } tcbhead_t; 139 | ``` 140 | * + syscall (amd64) 141 | 142 | 143 | 144 | ## T 145 | 146 | 147 | 148 | ## U 149 | * user-level statically defined tracing (USDT) 150 | * unlink 151 | * + [ref](http://www.d-kai.me/linux-close-%E4%B8%8Eunlink/) 152 | 153 | 154 | 155 | ## V 156 | 157 | 158 | 159 | ## W 160 | 161 | 162 | 163 | ## X 164 | 165 | 166 | 167 | ## Y 168 | 169 | 170 | 171 | ## Z 172 | 173 | -------------------------------------------------------------------------------- /Idioms/ml_idiom.mdown: -------------------------------------------------------------------------------- 1 | # ML Idioms 2 | 3 | ## \# 4 | 5 | 6 | 7 | ## A 8 | 9 | 10 | ## B 11 | 12 | 13 | 14 | ## C 15 | 16 | 17 | ## D 18 | 19 | 20 | 21 | ## E 22 | 23 | 24 | ## F 25 | 26 | 27 | 28 | ## G 29 | * GAN 30 | * + [collection](https://zhuanlan.zhihu.com/p/34016536) 31 | 32 | ## H 33 | 34 | 35 | 36 | ## I 37 | 38 | 39 | 40 | ## J 41 | 42 | 43 | 44 | ## K 45 | 46 | 47 | ## L 48 | 49 | 50 | 51 | ## M 52 | 53 | 54 | 55 | ## N 56 | 57 | 58 | 59 | ## O 60 | 61 | 62 | ## P 63 | 64 | 65 | 66 | ## Q 67 | 68 | 69 | 70 | ## R 71 | 72 | 73 | 74 | ## S 75 | 76 | 77 | 78 | ## T 79 | 80 | 81 | 82 | ## U 83 | 84 | ## V 85 | 86 | 87 | 88 | ## W 89 | 90 | 91 | 92 | ## X 93 | 94 | 95 | 96 | ## Y 97 | 98 | 99 | 100 | ## Z 101 | 102 | -------------------------------------------------------------------------------- /Idioms/network_idiom.mdown: -------------------------------------------------------------------------------- 1 | # Network Idioms 2 | 3 | ## \# 4 | 5 | 6 | 7 | ## A 8 | * [ARP request](http://blog.csdn.net/wenqian1991/article/details/44133039) 9 | 10 | ## B 11 | 12 | 13 | 14 | ## C 15 | * CA (certification authorities) 16 | 17 | ## D 18 | * DCCP 19 | 20 | ## E 21 | 22 | 23 | ## F 24 | * FTP 25 | * + Active 26 | * + Passive 27 | * + [difference](https://stackoverflow.com/questions/1699145/what-is-the-difference-between-active-and-passive-ftp) 28 | * + SFTP (SSH File Transfer Protocol) 29 | * + FTPS (FTP over SSL/TLS) 30 | 31 | 32 | ## G 33 | 34 | 35 | ## H 36 | * http 37 | * + 短连接 38 | * + 长连接 39 | * + pipeline (redis http) 40 | * + http2 41 | 42 | ## I 43 | 44 | 45 | 46 | ## J 47 | 48 | 49 | 50 | ## K 51 | 52 | 53 | ## L 54 | 55 | 56 | 57 | ## M 58 | 59 | 60 | 61 | ## N 62 | 63 | 64 | 65 | ## O 66 | 67 | 68 | ## P 69 | 70 | 71 | 72 | ## Q 73 | 74 | 75 | 76 | ## R 77 | 78 | 79 | 80 | ## S 81 | 82 | 83 | 84 | ## T 85 | * TLS 86 | * TCP 87 | 88 | ## U 89 | * UDP 90 | 91 | ## V 92 | 93 | 94 | 95 | ## W 96 | 97 | 98 | 99 | ## X 100 | 101 | 102 | 103 | ## Y 104 | 105 | 106 | 107 | ## Z 108 | 109 | -------------------------------------------------------------------------------- /Idioms/reverse_idiom.mdown: -------------------------------------------------------------------------------- 1 | # Reverse Idioms 2 | 3 | ## \# 4 | 5 | 6 | 7 | ## A 8 | * AES 9 | * + [Explain](https://www.cnblogs.com/luop/p/4334160.html) 10 | * + [MixColumn](https://crypto.stackexchange.com/questions/2402/how-to-solve-mixcolumns) 11 | + AFL Fuzz 12 | + ASAN 13 | * ASLR 14 | + ASLR绕过 15 | 16 | 17 | 18 | ## B 19 | 20 | * 爆破/burp 21 | 22 | ## C 23 | * control flow hijack 24 | * + stack buffer overflow 25 | * + - [ROP](https://www.slideshare.net/hackstuff/rop-40525248) 26 | * + - - code gadget 27 | * + - - ret2libc 28 | * + - - seh pointer overwrite 29 | * + - - virtual class pointer overwrite 30 | * + - DEP/NX (rwx stack => rw stack) 31 | * + - Cookie/Canary 32 | * + - ASLR 33 | * + - SafeSEH/SEHOP 34 | * + - Shadow stack 35 | * + - GOT/PLT 36 | * + [heap spray](http://blog.csdn.net/magictong/article/details/7391397) 37 | * + - heap double linked list 38 | * + - heap header 39 | * + - heap lookaside linked list 40 | * Control flow isolation 41 | 42 | 43 | ## D 44 | 45 | 46 | 47 | ## E 48 | 49 | 50 | 51 | 52 | 53 | 54 | ## F 55 | 56 | * Fuzz 57 | * 变异算法 58 | * 回显 59 | 60 | 61 | 62 | ## G 63 | 64 | 65 | ## H 66 | * HTTPS 67 | * + [MITM Attack](https://elliotsomething.github.io/2016/12/22/HTTPS%E4%B8%AD%E9%97%B4%E4%BA%BA%E6%94%BB%E5%87%BB%E5%8F%8A%E9%98%B2%E5%BE%A1/) 68 | * + - SSLSniff 69 | * + - SSLStrip: http=>https attack 70 | * + - heartbleed 71 | * + [Some other attacks](http://www.freebuf.com/articles/web/47076.html) 72 | * Hook 73 | + hook原理 74 | 75 | 76 | ## I 77 | 78 | * IAT表 79 | * image_header 80 | * image_optional_header 81 | 82 | 83 | 84 | ## J 85 | 86 | 87 | 88 | ## K 89 | 90 | * Kernel 91 | * 内核栈溢出 92 | * 内核ROP, cred结构体覆盖 93 | * 内核UAF 94 | * 内核double free 95 | * 内核堆溢出 96 | * slab 97 | * 内核保护绕过 98 | 99 | 100 | 101 | 102 | ## L 103 | 104 | 105 | 106 | ## M 107 | * moving target defense 108 | * + non-deterministic / less homogeneous / less static 109 | * + reactive => proactive 110 | * + static => dynamic 111 | * + Application-based MTD 112 | * + - state estimation 113 | * + - automatic generation control 114 | * + - remedial action scheme 115 | * + - installer 116 | * + - diversify commands (rewrite keyword with random key) 117 | ``` 118 | SELECT123 id, name, description FROM123 products WHERE123 productid=$value 119 | 99999 OR 1=1 120 | SELECT123 id, name, description FROM123 products WHERE123 productid=99999 OR 1=1 121 | 122 | OR is not diversified 123 | ``` 124 | * + System-based MTD 125 | * + - software/hardware-based 126 | * + - - ASLR 127 | * + - - ISR (instruction set randomization) 128 | * + - - DR (data randomization): pointer/memory, XOR with random masks 129 | * + - - Compiler-based 130 | * + Network-based MTD 131 | * + - MAC layer: changing MAC address 132 | * + - IP layer: IP randomization 133 | * + - TCP (traffic) layer: changing network protocol 134 | * + - Session layer 135 | * + - dynamic resource mapping system 136 | * + - mutable network 137 | * + - - random address hopping 138 | * + - - random finger printing 139 | 140 | 141 | ## N 142 | 143 | 144 | 145 | ## O 146 | 147 | 148 | ## P 149 | 150 | * 旁信道攻击 151 | * ptmalloc2 152 | * fastbin 153 | * IO_file 154 | 155 | ## Q 156 | 157 | 158 | 159 | ## R 160 | 161 | 162 | 163 | ## S 164 | * SSL/TLS 165 | * + [ref](http://www.ruanyifeng.com/blog/2014/02/ssl_tls.html) 166 | * Stackoverflow 167 | * 内核栈溢出 168 | * SEH栈溢出 169 | 170 | 171 | 172 | 173 | ## T 174 | 175 | * tcache攻击 176 | 177 | 178 | 179 | ## U 180 | 181 | * userspace 182 | * 用户态double free 183 | 184 | 185 | 186 | ## V 187 | 188 | * VMProtect 189 | * 穿山甲 190 | 191 | 192 | 193 | ## W 194 | 195 | 196 | 197 | ## X 198 | 199 | 200 | 201 | ## Y 202 | 203 | 204 | 205 | ## Z 206 | 207 | * 中间人攻击 208 | * 字典攻击 -------------------------------------------------------------------------------- /Miscs/2019 CS Master Application.md: -------------------------------------------------------------------------------- 1 | # 2019 CS Master 申请总结 2 | 3 | 终于毕业了, 记起还没写申请总结, 提笔自娱自乐一下. 4 | 5 | 6 | 7 | ## 起因 8 | 9 | 2015年9月, 我进入了上海交通大学密西根学院学习. 这个学院以提供与密西根大学的DD (dual-degree, 拿双方的本科学位)而受到瞩目, 而能得到这个机会的一届是几百人中的一百人. 本人天资愚钝, 在经历了一个半学期的头破血流地学习后, 申请DD时GPA也没能够着3.4, 也不敢去学院查GPA排名 (不过CS相关的课还算高). 在UMich的申请界面上, 出于自毁冲动, 毅然决然地填了Computer Science (填EE/CE, 据传, 比较保险). 我至今也不明白, UMich到底是怎么挑选DD学生的. 在Wolverine Access界面查到Acceptance的时候, 我完全不敢相信这是真的. 10 | 11 | 12 | 13 | 总之, 后两年我从SJTU来到了UMich读书. 每个学期稳定地一门课拿不到4.0. 最后大四上申请时是3.91/4.00的GPA (总共就7门课, 属于耍赖). 14 | 15 | 16 | 17 | ## 失败的research经历 18 | 19 | 在刚来UMich的时候, 我属于缩头乌龟, 没了解过找Intern或者做Research的事情. 后来单纯去学校的Winter Career Fair逛了一圈, 一天也就能排到三家公司 (Facebook, Indeed, Oath). 最后实习当然是石沉大海. 然后眼看暑假要回国无事可做, 急忙问认识或者不认识的Professor是否能做volunteer summer research. 最后找到了一位教授, 我到现在都很感激他, 通情达理, 带我领略了领域的美好, 教会了我一些做Research的方式. 但是从做Project来说, 我个人做的是很痛苦的. 20 | 21 | 22 | 23 | 我当时做的项目是在Python做工业应用的符号执行. 总所周知, Python慢的一批 ([benchmarks-game-nbody](https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/nbody.html)狂揽倒数一二). 符号执行更加是个很慢的操作 (用Z3做SMT Solver). 当时我还要管各种各样的出错情况, 凭我的脑子最后也只能想出一个又一个的ad hoc solution. 最后跑一个testcase要一个小时, 中间夹杂了各种我懂或者不懂的error和warning. 在与glibc, PLT/GOT, vDSO, 各种汇编搏斗了几天后, 也不一定能得到有意义的结果. 这个项目有三个天南海北的教授加上一个英国PhD, 再加上我, 就是所有人员了. 当时我本身没有符号执行的基础, 和PhD也差着5小时时差. 当时这位PhD还要忙这个项目的另一个部分, 我基本是单打独斗地瞎搞, 写出了一坨又一坨屎山代码. 当时边做边写这个项目paper的related works, 觉得别人做的又clean又smart, 逐渐觉得这项目是个dead-end. 24 | 25 | 26 | 27 | 最后论文也是一推再推, 从八月ddl的会推到一月ddl的会, 还是没过. 刚好冬季学期要开始了, 最后还是向老板请辞了. 我还是觉得有点对不起他们. 当PhD和我最后一次video meeting的时候, 他还说the paper will have your name. 我回答 please in the acknowledgement. 28 | 29 | 30 | 31 | 当时真的觉得自己有点抑郁, 这段经历让我对读PhD极其抵触, 现在逐渐觉得这种念头又回来了, 不过根据地里说的, 凡是犹豫过要不要读PhD的, 都不适合读PhD, 大概正如此言. 32 | 33 | 34 | 35 | ## 申请与结果 36 | 37 | 所以这个申请季也只申请了Master, 还很极端的偏向了工作导向的Master. 具体结果如下 38 | 39 | ![1565281225177](D:\OneDrive\Pictures\Typora\1565281225177.png) 40 | 41 | 申请的时候找了几家中介, 但最后他们都只起到了改文书的作用. 考了四次GRE, 最后也没到325+4. TOEFL到只考了一次, 103, 很匹配这个GRE就没再考. 42 | 43 | 44 | 45 | 其实如果按照我喜欢的领域排名 (PLT, Security, Arch, System), 大概是不会选择去UCLA. 但是因为找工作便利, 想换个环境, 并且有一个Master of Science的名头, 最后选择了UCLA. 最为遗憾的是最后CMU的SCS, INI, 到ECE的SE-SV都把我拒了个遍. 久仰CMU的各种15/18开头的课的大名, 却没能一览其究竟, 无疑让人遗憾. 46 | 47 | 48 | 49 | ## 总结 50 | 51 | 最后两年变换一个环境, 有利也有弊. 好处是前两年的糟糕GPA在新学校只是一个个T(Transfer)而可以重新在一年里拿到一个很高的GPA. 坏处是, 你必须要时刻准备好, 要怎么在短暂的一年里做出Research, 发出paper而申请PhD, 或是如何在第一个寒假拿到一个intern offer可进可退, 亦或是如何在一年里保持足够的GPA申请Master... 52 | 53 | 54 | 55 | 56 | 其实上面都是废话 57 | 申请的时候整理了一些流言/stereotype, 地址在[这里](https://pastebin.com/RjaeFFn1) 58 | 59 | 60 | 61 | 62 | 63 | ## Miscs 64 | 65 | 为啥1p3a现在安卓手机版必须要用那又卡又丑的App...还我网页版... 66 | 67 | -------------------------------------------------------------------------------- /Miscs/CMU_class_candidate.md: -------------------------------------------------------------------------------- 1 | 14735 Secure Code 2 | 3 | 14829 Intro to Reverse Engineering 4 | 5 | 15122 Imperative Computing 6 | 7 | 15317 Constructive Logic 8 | 9 | 15319 Cloud Computing 10 | 11 | 15349 Intro to Embedded System 12 | 13 | 15414 Automated Program Verification & Testing 14 | 15 | 15417 HOT Compilation 16 | 17 | 15440 Distributed Systems 18 | 19 | 15441 Computer Networks 20 | 21 | 15487 Introduction to Computer Security 22 | 23 | 15712 Advanced Operating Systems and Distributed Systems 24 | 25 | 15719 Advanced Cloud Computing 26 | 27 | 15732 Secure Software System 28 | 29 | 15740 Computer Architecture 30 | 31 | 15744 Computer Networks 32 | 33 | 15746 Storage Systems 34 | 35 | 15811 Verifying Complex Systems 36 | 37 | 15812 Programming Language Semantics 38 | 39 | 15815 Automated Theorem Proving 40 | 41 | 15816 Linear / Substructural Logic 42 | 43 | 15819 Program Analysis / HoTT / Classic Papers in Programming Languages and Logic / Computational Type Theory / Quantitative Program Analysis 44 | 45 | 15829 Performance Engineering of Software Systems 46 | 47 | 17819 Program Analysis 48 | 49 | -------------------------------------------------------------------------------- /Miscs/CppNow2018.md: -------------------------------------------------------------------------------- 1 | # CppNow 2018 2 | 3 | - allocator => pmr::allocator (handler to heap, easily shared memory_resource) 4 | - boost::text 5 | - - string layer: `string`, `string_view`, `unencoded_rope` (B-tree mixture of string and string_view), `unencoded_rope_view`, `repeated_string_view` 6 | - - unicode layer: `from/to_utfxx_iterator` 7 | - - text layer: `text`, `text_view`, `rope`, `rope_view` 8 | - utf8 -> utf32 (DFA automata) 9 | - `A foo(A x)` enables move from rvalue, while `A foo(A& x)` forced copy constructor 10 | - smart iterators 11 | - - range-v3 style (on input iterator): `num | range::filter(f) | range::transform(ff)` 12 | - - output iterator style: `copy(num.begin(), num.end(), transformer(type_inserter(res.begin())))` (use `*it.operator=(const T&)` to transform) 13 | - new type traits 14 | - - `is_trivially_relocatable`: when your vector reserve uses move constructor 15 | - - - `__uninit_copy_a + __destroy_a` => `__uninit_move_a` => `__uninit_relocate_a` => `uninitialize_relocate` => `__builtin_memcpy` 16 | - - `is_trivially_comparable`: memcmp-comparable or memberwise-comparable? 17 | - - `tombstone_traits`: is some spare or sign bits can be used for `optional`? so `sizeof(opt>) == sizeof(T)` 18 | - C++2a 19 | - - concepts 20 | - contracts 21 | - ranges 22 | - coroutines 23 | - modules 24 | - reflection TS 25 | - - modern time issues 26 | - - span (both static `span` and dynamic `span`) 27 | - - `operator<=>` 28 | - - sync buffered ostream 29 | - - `atomic>`, `atomic` 30 | - - `[[no_unique_address]]` for empty base optimization 31 | - - `[[likely]]` 32 | - - designated init `{.a(a)}` 33 | - - range-based-for init statement 34 | - - ``, `` 35 | - - `[[no_discard]]`, `remove_cvref`, `to_address(pointer)`, `VA_OPT`, constexpr iterator 36 | - - unordered_map Equal for heterogeneous index lookup 37 | - - `[]{}`, `[=, this]`, pack expansion in lambda capture-init statement 38 | - - remove `typename` requirement for type-only conditions 39 | - - `make_shared` 40 | - `initializer_list` leads to temporary array to be created (2N copies) 41 | - - [ref](https://akrzemi1.wordpress.com/2016/07/07/the-cost-of-stdinitializer_list/) 42 | - - need `movable_init_list`, like `vector(std::make_move_iterator(std::begin(a)), std::make_move_iterator(std::end(a)))` -------------------------------------------------------------------------------- /Miscs/Facebook Production Engineer Questions.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Miscs/Facebook Production Engineer Questions.pdf -------------------------------------------------------------------------------- /Miscs/Unfiled/2019年年初至今 药铺电面总结及思路（按照频率排序）+ 8_9新鲜面经_一亩三分地美国面经版.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Miscs/Unfiled/2019年年初至今 药铺电面总结及思路（按照频率排序）+ 8_9新鲜面经_一亩三分地美国面经版.pdf -------------------------------------------------------------------------------- /Miscs/Unfiled/Amazon OA with answers.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Miscs/Unfiled/Amazon OA with answers.pdf -------------------------------------------------------------------------------- /Miscs/Unfiled/Amazon OA1 debug summary.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Miscs/Unfiled/Amazon OA1 debug summary.pdf -------------------------------------------------------------------------------- /Miscs/Unfiled/Amazon7-9 - PW be0608aeaa6a12e4b25c011ad358c0e8.xlsx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Miscs/Unfiled/Amazon7-9 - PW be0608aeaa6a12e4b25c011ad358c0e8.xlsx -------------------------------------------------------------------------------- /Miscs/Unfiled/Amazon7-9非原题 - PW 3ae84b1ac4b4cd8d043f543c19cf585c.docx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Miscs/Unfiled/Amazon7-9非原题 - PW 3ae84b1ac4b4cd8d043f543c19cf585c.docx -------------------------------------------------------------------------------- /Miscs/Unfiled/Google 面筋.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Miscs/Unfiled/Google 面筋.pdf -------------------------------------------------------------------------------- /Miscs/Unfiled/LeetCodeSummary_08.09.2019.xlsx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Miscs/Unfiled/LeetCodeSummary_08.09.2019.xlsx -------------------------------------------------------------------------------- /Miscs/Unfiled/Walmart_2019.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Miscs/Unfiled/Walmart_2019.pdf -------------------------------------------------------------------------------- /Miscs/Unfiled/lc-soln.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Miscs/Unfiled/lc-soln.pdf -------------------------------------------------------------------------------- /Miscs/Unfiled/狗狗面筋-8月底.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Miscs/Unfiled/狗狗面筋-8月底.pdf -------------------------------------------------------------------------------- /Miscs/install_haskell_wsl_vscode.md: -------------------------------------------------------------------------------- 1 | # Install OCaml on WSL-Ubuntu with VSCode 2 | 3 | ## Prerequisite 4 | 5 | - WSL-Ubuntu 6 | - VSCode 7 | 8 | 9 | 10 | ## Procedure 11 | 12 | - WSL-Ubuntu side 13 | 14 | - `curl -sSL https://get.haskellstack.org/ | sh -s - -f`: install `stack` 15 | - `stack install ghci hlint` 16 | - `git clone https://github.com/haskell/haskell-ide-engine --recursive` 17 | - `cd haskell-ide-engine && stack ./install.sh build-all` 18 | 19 | - Windows side 20 | 21 | - install `stack` & `stack install hlint` 22 | 23 | - Solutions for wrapper like OCaml: [link](https://github.com/alanz/vscode-hie-server/issues/99) 24 | 25 | - create `hie.bat` 26 | 27 | - ```bash 28 | @echo off 29 | bash -ci "hie-wrapper --lsp" 30 | ``` 31 | 32 | - VSCode side 33 | 34 | - install `haskell-linter` extension 35 | - install `Haskell Syntax Highlighting` extension 36 | - install `Haskell Language Server` extension 37 | - change to customized HIE wrapper 38 | - modify the `extension.js` like the link above 39 | 40 | -------------------------------------------------------------------------------- /Miscs/install_klee_pt.md: -------------------------------------------------------------------------------- 1 | sudo apt update 2 | sudo apt upgrade 3 | 4 | sudo apt install bc bison build-essential cmake curl flex git libboost-all-dev libcap-dev libncurses5-dev python-minimal python-pip subversion unzip zlib1g-dev 5 | sudo apt install gcc-multilib g++-multilib libelf-dev libdw-dev dwarfdump libdwarf-dev 6 | sudo apt install dkms 7 | 8 | mkdir build 9 | cd build 10 | sudo apt-get install bc bison build-essential cmake curl flex git libboost-all-dev libcap-dev libncurses5-dev python-minimal python-pip subversion unzip zlib1g-dev 11 | svn co https://llvm.org/svn/llvm-project/llvm/tags/RELEASE_342/final llvm 12 | svn co https://llvm.org/svn/llvm-project/cfe/tags/RELEASE_342/final llvm/tools/clang 13 | svn co https://llvm.org/svn/llvm-project/compiler-rt/tags/RELEASE_342/final llvm/projects/compiler-rt 14 | svn co https://llvm.org/svn/llvm-project/libcxx/tags/RELEASE_342/final llvm/projects/libcxx 15 | svn co https://llvm.org/svn/llvm-project/test-suite/tags/RELEASE_342/final/ llvm/projects/test-suite 16 | cd llvm 17 | ./configure --enable-optimized --disable-assertions --enable-targets=host --with-python="/usr/bin/python2" 18 | make -j `nproc` 19 | 20 | make -j `nproc` check-all 21 | cd .. 22 | git clone --depth 1 https://github.com/stp/minisat.git 23 | # Commit ID: 3db58943b6ffe855d3b8c9a959300d9a148ab554 (very old - from Jun 22, 2015) 24 | rm -rf minisat/.git 25 | 26 | cd minisat 27 | make 28 | cd .. 29 | git clone --depth 1 --branch stp-2.2.0 https://github.com/stp/stp.git 30 | rm -rf stp/.git 31 | 32 | cd stp 33 | mkdir build 34 | cd build 35 | cmake \ 36 | -DBUILD_STATIC_BIN=ON \ 37 | -DBUILD_SHARED_LIBS:BOOL=OFF \ 38 | -DENABLE_PYTHON_INTERFACE:BOOL=OFF \ 39 | -DMINISAT_INCLUDE_DIR="../../minisat/" \ 40 | -DMINISAT_LIBRARY="../../minisat/build/release/lib/libminisat.a" \ 41 | -DCMAKE_BUILD_TYPE="Release" \ 42 | -DTUNE_NATIVE:BOOL=ON .. 43 | make -j `nproc` 44 | cd ../.. 45 | git clone --depth 1 --branch klee_uclibc_v1.0.0 https://github.com/klee/klee-uclibc.git 46 | rm -rf klee-uclibc/.git 47 | 48 | cd klee-uclibc 49 | ./configure \ 50 | --make-llvm-lib \ 51 | --with-llvm-config="../llvm/Release/bin/llvm-config" \ 52 | --with-cc="../llvm/Release/bin/clang" 53 | make -j `nproc` 54 | cd .. 55 | git clone --depth 1 --branch z3-4.5.0 https://github.com/Z3Prover/z3.git 56 | rm -rf z3/.git 57 | 58 | cd z3 59 | python scripts/mk_make.py 60 | cd build 61 | make -j `nproc` 62 | 63 | # partialy copied from make install target 64 | mkdir -p ./include 65 | mkdir -p ./lib 66 | cp ../src/api/z3.h ./include/z3.h 67 | cp ../src/api/z3_v1.h ./include/z3_v1.h 68 | cp ../src/api/z3_macros.h ./include/z3_macros.h 69 | cp ../src/api/z3_api.h ./include/z3_api.h 70 | cp ../src/api/z3_ast_containers.h ./include/z3_ast_containers.h 71 | cp ../src/api/z3_algebraic.h ./include/z3_algebraic.h 72 | cp ../src/api/z3_polynomial.h ./include/z3_polynomial.h 73 | cp ../src/api/z3_rcf.h ./include/z3_rcf.h 74 | cp ../src/api/z3_fixedpoint.h ./include/z3_fixedpoint.h 75 | cp ../src/api/z3_optimization.h ./include/z3_optimization.h 76 | cp ../src/api/z3_interp.h ./include/z3_interp.h 77 | cp ../src/api/z3_fpa.h ./include/z3_fpa.h 78 | cp libz3.so ./lib/libz3.so 79 | cp ../src/api/c++/z3++.h ./include/z3++.h 80 | 81 | cd ../.. 82 | git clone --depth 1 --branch v1.3.0 https://github.com/klee/klee.git 83 | rm -rf klee/.git 84 | 85 | BUILDDIR=`pwd` 86 | cd klee 87 | ./configure \ 88 | LDFLAGS="-L$BUILDDIR/minisat/build/release/lib/" \ 89 | --with-llvm=$BUILDDIR/llvm/ \ 90 | --with-llvmcc=$BUILDDIR/llvm/Release/bin/clang \ 91 | --with-llvmcxx=$BUILDDIR/llvm/Release/bin/clang++ \ 92 | --with-stp=$BUILDDIR/stp/build/ \ 93 | --with-uclibc=$BUILDDIR/klee-uclibc \ 94 | --with-z3=$BUILDDIR/z3/build/ \ 95 | --enable-cxx11 \ 96 | --enable-posix-runtime 97 | 98 | make -j `nproc` ENABLE_OPTIMIZED=1 99 | 100 | # Copy Z3 libraries to a place, where klee can find them 101 | cp ../z3/build/lib/libz3.so ./Release+Asserts/lib/ 102 | 103 | make -j `nproc` check 104 | cd .. 105 | ln -s ~/build/klee/Release+Asserts/lib/libkleeRuntest.so ~/build/klee/Release+Asserts/lib/libkleeRuntest.so.1.0 106 | 107 | sudo apt-get update 108 | sudo apt-get upgrade 109 | sudo apt-get install libelf-dev libdwarf-dev 110 | 111 | git clone https://github.com/01org/processor-trace 112 | cd processor-trace 113 | cmake . 114 | make 115 | sudo make install 116 | sudo ldconfig 117 | 118 | git clone https://github.com/intelxed/xed 119 | git clone https://github.com/intelxed/mbuild.git mbuild 120 | cd xed 121 | mkdir obj 122 | cd obj 123 | ../mfile.py 124 | sudo ../mfile.py --prefix=/usr/local install 125 | 126 | git clone https://github.com/andikleen/simple-pt 127 | cd simple-pt 128 | 129 | make 130 | 131 | touch dkms.conf 132 | echo 'PACKAGE_NAME="simple-pt" 133 | PACKAGE_VERSION="0.11.0" 134 | 135 | # Items below here should not have to change with each driver version 136 | MAKE[0]="make KERNEL_DIR=${kernel_source_dir} all" 137 | CLEAN="make clean" 138 | 139 | BUILT_MODULE_NAME[0]="$PACKAGE_NAME" 140 | DEST_MODULE_LOCATION[0]="/extra" 141 | 142 | REMAKE_INITRD="no" 143 | AUTOINSTALL="yes"' >> dkms.conf 144 | 145 | sudo cp -R . /usr/src/simple-pt-1.1 146 | sudo dkms add -m simple-pt -v 1.1 147 | sudo dkms build -m simple-pt -v 1.1 148 | sudo dkms install -m simple-pt -v 1.1 149 | 150 | make user XED=1 151 | 152 | ./ptfeature -------------------------------------------------------------------------------- /Miscs/install_ocaml_wsl_vscode.md: -------------------------------------------------------------------------------- 1 | # Install OCaml on WSL-Ubuntu with VSCode 2 | 3 | ## Prerequisite 4 | 5 | * WSL-Ubuntu 6 | * VSCode 7 | * npm with YARN 8 | 9 | 10 | 11 | ## Procedure 12 | 13 | * WSL-Ubuntu side 14 | 15 | * delete `~/.opam` if you initialize without `--disable-sandboxing` (`--reinit` will not work) 16 | 17 | * ```bash 18 | sudo add-apt-repository ppa:avsm/ppa 19 | sudo apt-get update 20 | sudo apt-get install -y gcc make m4 build-essential ocaml ocaml-native-compilers camlp4-extra opam nodejs 21 | opam init --disable-sandboxing 22 | opam switch create 4.02.3 23 | opam install merlin ocp-indent 24 | ``` 25 | 26 | * VSCode side 27 | 28 | * install `OCaml and Reason IDE` extension. (The for WSL version is broken due to VSCode upgrdation) 29 | 30 | * Windows side 31 | 32 | * `npm install -g bs-platform ocaml-reason-wsl` 33 | * The default user should not be `root`! 34 | 35 | -------------------------------------------------------------------------------- /Miscs/seen.mdown: -------------------------------------------------------------------------------- 1 | --- 2 | typora-root-url: ..\..\OneDrive\Pictures\Typora 3 | --- 4 | 5 | # What I randomly see 6 | 7 | ## 2018-09 8 | 9 | ### Miscs 10 | * `++x` and `x = x+1` not equal under atomic view 11 | * mmx registers can give 16-byte lock-free align 12 | * memory barrier is for non-atomic memory publishing / acquiring 13 | * `compare_exchange_weak` is like a TimedLock getting old value 14 | * lexer analysis =>(token stream)=> syntax analysis =>(AST)=> syntax tree => intermidiate representation => optimiziation 15 | * ----- 16 | * Scala type system 17 | * + Nothing as bottom, subtype of everything, no instance 18 | * + Any as top, everything's base 19 | * + Unit <: Any, only holds itself, serves as void 20 | * + [unified-types](https://docs.scala-lang.org/tour/unified-types.html) 21 | > `object None extends Option[Nothing]` 22 | > Because Option is covariant in its type parameter and Nothing is a subtype of everything, Option[Nothing] is a subtype of Option[A] for every type A. So, we can make one object None which is a subtype of Option[A] for every A. This is reasonable, since Nothing cannot be instantiated so Option[Nothing] will always be without a value. 23 | * unit as true, nothing as false in CH-iso (final object and init object) 24 | * covariance 25 | * + subtyping: A <: B => T[A] <: T[B] 26 | * + return value/haskell typeclass inheritance: a -> b => f a -> f b (Functor => Maybe fmap) (subclass function return type) 27 | * contravariance 28 | * + subtyping: A <: B => T[B] <: T[A] 29 | * + return value/haskell typeclass inheritance: a -> b => f b -> f a (subclass function argument type) 30 | * invariance: A <: B => T[A] ?? T[B] 31 | * bivariance 32 | * ----- 33 | * #pragma gcc poison/bless begin[end] 34 | * ----- 35 | * ARM literal: 4 bit ROR + 8 bit imm 36 | 37 | * 38 | -------------------------------------------------------------------------------- /Papers/Arch/power_a_first_class_constraint.md: -------------------------------------------------------------------------------- 1 | # Power: A First-Class Architecture Design Constraint 2 | 3 | **Trevor Mudge** 4 | 5 | --- 6 | 7 | 8 | 9 | ## Introduction 10 | 11 | * written in 2001 12 | * limiting power consumption is critical: 13 | * portable/mobile platforms 14 | * server farms 15 | * ![image-20200107020121474](D:\OneDrive\Pictures\Typora\image-20200107020121474.png) 16 | * rapid growth in power consumption is obvious 17 | * chip die's power density also increases linearly 18 | 19 | 20 | 21 | ## Power Equations for CMOS Logic 22 | 23 | * 3 equations for power-per-performance trade-offs for complementary metal-oxide semiconductor logic circcuits 24 | * power consumption $$P = AVC^2 f + \tau AVI_{\text{short}} f + VI_{\text{leak}}$$ 25 | * `f`: system operation 26 | * `A`: activity of the gates in the system (some gates do not switch per clock tick) 27 | * `C`: total capacitance seen by the gates' outputs 28 | * `V`: supply voltage 29 | * 1st component: dynamic power consumption caused by charging & discharging the capacitive load on each gate's output 30 | * most dominate factor 31 | * suggests that reduce power consumption is the most effective way 32 | * quadratic dependency 33 | * 2nd component: power expended as the result of the short-circuit current $I_{\text{short}}$, which momentarily $\tau$, flows between the supply voltage & ground, when CMOS logic gate's output switches 34 | * 3rd component: power lost from the leakage current regardless of the gate's state 35 | * maximum-operating frequency: $$f_{\text{max}} \propto (V - V_{\text{threshold}})^2 / V$$ 36 | * parallel processing, which involves splitting a computation in 2 and running it as 2 parallel independent tasks, hash to potential to cut the power in half without slowing the computation 37 | * leakage current: $$I_{\text{leak}} \propto \exp (-q V_{\text{threshold}} / kT)$$ 38 | * limited option for countering the effect of reducing `V` (also reduce threshold voltage), making leakage term appreciable 39 | * ![image-20200107110940738](D:\OneDrive\Pictures\Typora\image-20200107110940738.png) 40 | 41 | 42 | 43 | ## Other quantities 44 | 45 | * Peak power: upper limit of the system 46 | * Dynamic power: sharp changes in power consumption can result in ground bounce / _di/dt_ noise that upsets logic voltage levels, ca using erroneous circuit behavior 47 | * energy/operation raio 48 | * MIPS/W 49 | * energy $\times$ delay 50 | 51 | 52 | 53 | ## Reducing Power Consumption 54 | 55 | 56 | 57 | ### Logic 58 | 59 | * clock tree 30% power 60 | 61 | * Clock gating: turn off clock tree branches to latches / flip-flops whenever they are not used 62 | * poor design, can exacerbate clock skew 63 | * with more accurate timing analyzers, flexible design tools 64 | * Half-frequency / half-swing clocks: half-frequency clock uses both edges of the clock to synchronize events 65 | * like Intel some port? 66 | * half swing clock swings only half of `V` 67 | * increases the latch design's requirement 68 | * produces great gain 69 | * Asynchronous logic: don't have a clock 70 | * needing to generate completion signals 71 | * additional logic must be used at each register transfer 72 | * double-rail implementation 73 | * testing difficulty 74 | * absence of design tools 75 | * e.g. Amulet 76 | * good for globally asynchronous, locally synchronous systems 77 | 78 | 79 | 80 | ## Architecture 81 | 82 | * exploiting parallelism: reduce power 83 | * using speculation: permit computations to proceed beyond dependent instructions 84 | * Memory systems 85 | * cache memory can dominate the chip area 86 | * frequency of memory access $\to$ dynamic power loss 87 | * leakage current $\to$ power loss 88 | * filter cache 89 | * in front of L1 cache 90 | * intercept signal intended for the main cache 91 | * memory banking 92 | * usually in low-power designs 93 | * splits the memory into banks, activates only the bank presently in use 94 | * relies on reference-pattern having lot of spatial locality 95 | * suitable for instruction-cache organization ([[C: also GPU memory?]]) 96 | * [channel, DIMM, rank, chip, bank, row/column](https://www.archive.ece.cmu.edu/~ece740/f11/lib/exe/fetch.php?media=wiki:lectures:onur-740-fall11-lecture25-mainmemory.pdf) 97 | * Buses 98 | * significant source of power loss 99 | * especially interchip buses (often wide) 100 | * encode the address lines into Gray code 101 | * address changes (from cache refills) are often sequential 102 | * counting in Gray code switches the least number of signals 103 | * transmitting difference between successive address values? similar to Gray code 104 | * compressing information in address lines 105 | * integrated into bus controllers for interchip signaling 106 | * reduce code overlays (DSP techniques) 107 | * Parallel processing & pipelining 108 | * reduce power consumption in CMOS system 109 | * signal-processing algorithms often possess a significant degree of parallelism 110 | * DSP chips 111 | 112 | 113 | 114 | ### Operating System 115 | 116 | * known comptuation's deadline $\to$ adjust frequency 117 | * allow OS to set the voltage (by writing a regsiter) 118 | * scheduler its voltage needs 119 | * don't need application modifications 120 | * detecting best timing for scaling back is difficult 121 | 122 | 123 | 124 | ## What can we do with a high MIPS/W device 125 | 126 | * omitted 127 | 128 | 129 | 130 | ## Future challenge 131 | 132 | * leakage current limitation 133 | * submicron dimensions, supply must be reduced to avoid damaging electric fields, in turn reducing threshold voltage 134 | * $\to$ leakage becomes the dominant source of power consumption 135 | * $\to$ increase in chip temperature, increase in the thermal voltage, thermal runaway 136 | * solutions 137 | * 2 types of field-effect transistors (voltage clustering): low threshold voltage devices for high-speed paths + high threshold voltage devices for not critical paths 138 | 139 | 140 | 141 | 142 | 143 | 144 | 145 | 146 | 147 | ## Motivation 148 | 149 | ## Summary 150 | 151 | ## Strength 152 | 153 | ## Limitation & Solution 154 | 155 | 156 | 157 | -------------------------------------------------------------------------------- /Papers/Arch/speculative_multithreaded_processor.md: -------------------------------------------------------------------------------- 1 | # Speculative Multithreaded Processors 2 | 3 | **Gurindar S. Sohi, Amir Roth ** 4 | 5 | --- 6 | 7 | 8 | 9 | ## Introduction 10 | 11 | * 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | * CS251A requirements 26 | * a short paragraph summarizing the problem and goal/contributions of paper 27 | * a short paragraph summarizing the paper’s methods and results 28 | * a short paragraph giving your opinion of what is good and bad about the paper. 29 | 30 | ## Summary 31 | 32 | ## Methods & Results 33 | 34 | ## Personal Opinions 35 | 36 | 37 | -------------------------------------------------------------------------------- /Papers/HPC/ligra.md: -------------------------------------------------------------------------------- 1 | # Ligra: A Lightweight Graph Processing Framework for Shared Memory 2 | 3 | **Julian Shun, Guy E. Blelloch** 4 | 5 | --- 6 | 7 | 8 | 9 | ## Introduction 10 | 11 | * shared-memory multicore graph algorithms are efficient 12 | * Ligra: lightweight interface for graph alg (graph traversal) 13 | * race condition (atomic CAS) 14 | * ![image-20191105133818463](D:\OneDrive\Pictures\Typora\image-20191105133818463.png) 15 | * data types: graph with V, E / subsets of V 16 | * `vertexMap`, `edgeMap` 17 | * ![image-20191105135143704](D:\OneDrive\Pictures\Typora\image-20191105135143704.png) 18 | 19 | 20 | 21 | ## Preliminaries 22 | 23 | * ![image-20191105142114195](D:\OneDrive\Pictures\Typora\image-20191105142114195.png) 24 | 25 | 26 | 27 | ## Framework 28 | 29 | * ![image-20191105142136262](D:\OneDrive\Pictures\Typora\image-20191105142136262.png) 30 | * ![image-20191105142641391](D:\OneDrive\Pictures\Typora\image-20191105142641391.png) 31 | * optimization 32 | * `edgeMapSparse`: two `F` for fast-fail for first-arg & complete (2 args) 33 | * remove-duplicate stage bypassed if no duplicate 34 | * no break `edgeMapDense`: parallel 35 | * ![image-20191105143013223](D:\OneDrive\Pictures\Typora\image-20191105143013223.png) 36 | * PageRank, B-F shortest paths 37 | 38 | 39 | 40 | ## Applications 41 | 42 | * BFS 43 | * betweenness centrality 44 | * [[T: definition]] 45 | * ![image-20191105143101342](D:\OneDrive\Pictures\Typora\image-20191105143101342.png) 46 | * Graph Radii Estimation, Multiple BFS 47 | * [[T: definition]] 48 | * ![image-20191105143129757](D:\OneDrive\Pictures\Typora\image-20191105143129757.png) 49 | * Connected Components 50 | * ![image-20191105143146877](D:\OneDrive\Pictures\Typora\image-20191105143146877.png) 51 | * PageRank 52 | * ![image-20191105143159765](D:\OneDrive\Pictures\Typora\image-20191105143159765.png) 53 | * Bellman-Ford Shortest Paths 54 | * ![image-20191105143215628](D:\OneDrive\Pictures\Typora\image-20191105143215628.png) 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | ## Motivation 73 | 74 | * A modern single multicore server can support most of the graph computation. And graph algorithms perform better in shared-memory multicores machines (per core, per dollar, per joule) than distributed memory systems. 75 | 76 | ## Summary 77 | 78 | * In this paper, the authors present Ligra, which is a shared-memory lightweight graph processing framework, especially for graph traversing algorithms. It provides two datatypes: graph `(V, E)` and `vertexSubset`, along with two interface `edgeMap` and `vertexMap`. With these two abstractions, Ligra is able to present a large set of graph algorithms and take advantage of shared-memory parallelism efficiently. It automatically deals with sparse and dense graphs based on number of vertices and out-degrees. 79 | 80 | ## Strength 81 | 82 | * Ligra is able to provide a large set of graph algorithms, especially graph traversal, in some simple interfaces. It take advantages of the loop-parallelism efficiently. 83 | 84 | ## Limitation & Solution 85 | 86 | * Ligra's interface is limited, thus the original graph algorithms might need to be re-designed to satisfy the Ligra's interface. 87 | * Ligra's graph representation is vertices and edges, which might be unsuitable for matrix-based (CSR, ELL, ...) graph computations. 88 | * Like graph neural networks? 89 | * Ligra's synchronization is based on compare-and-swap, which may cause large number of cache invalidations and large memory bus overhead, especially for NUMA systems. 90 | * Care about memory locality? 91 | 92 | -------------------------------------------------------------------------------- /Papers/Network/ATPG.assets/image-20210121122449167.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/ATPG.assets/image-20210121122449167.png -------------------------------------------------------------------------------- /Papers/Network/ATPG.assets/image-20210121172831462.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/ATPG.assets/image-20210121172831462.png -------------------------------------------------------------------------------- /Papers/Network/ATPG.assets/image-20210121172839554.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/ATPG.assets/image-20210121172839554.png -------------------------------------------------------------------------------- /Papers/Network/ATPG.assets/image-20210121200758639.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/ATPG.assets/image-20210121200758639.png -------------------------------------------------------------------------------- /Papers/Network/ATPG.assets/image-20210121201007170.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/ATPG.assets/image-20210121201007170.png -------------------------------------------------------------------------------- /Papers/Network/ATPG.assets/image-20210121201257442.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/ATPG.assets/image-20210121201257442.png -------------------------------------------------------------------------------- /Papers/Network/ATPG.assets/image-20210121201510685.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/ATPG.assets/image-20210121201510685.png -------------------------------------------------------------------------------- /Papers/Network/ATPG.assets/image-20210121222729087.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/ATPG.assets/image-20210121222729087.png -------------------------------------------------------------------------------- /Papers/Network/ATPG.assets/image-20210121224311262.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/ATPG.assets/image-20210121224311262.png -------------------------------------------------------------------------------- /Papers/Network/ATPG.assets/image-20210121225416920.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/ATPG.assets/image-20210121225416920.png -------------------------------------------------------------------------------- /Papers/Network/ATPG.assets/image-20210121225617737.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/ATPG.assets/image-20210121225617737.png -------------------------------------------------------------------------------- /Papers/Network/ATPG.assets/image-20210121231322362.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/ATPG.assets/image-20210121231322362.png -------------------------------------------------------------------------------- /Papers/Network/Anteater.assets/image-20210131094447449.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/Anteater.assets/image-20210131094447449.png -------------------------------------------------------------------------------- /Papers/Network/Anteater.assets/image-20210131102448908.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/Anteater.assets/image-20210131102448908.png -------------------------------------------------------------------------------- /Papers/Network/Anteater.assets/image-20210131102626136.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/Anteater.assets/image-20210131102626136.png -------------------------------------------------------------------------------- /Papers/Network/Anteater.assets/image-20210131103524926.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/Anteater.assets/image-20210131103524926.png -------------------------------------------------------------------------------- /Papers/Network/Anteater.assets/image-20210131111428933.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/Anteater.assets/image-20210131111428933.png -------------------------------------------------------------------------------- /Papers/Network/Anteater.assets/image-20210131115152160.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/Anteater.assets/image-20210131115152160.png -------------------------------------------------------------------------------- /Papers/Network/Anteater.assets/image-20210131115349189.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/Anteater.assets/image-20210131115349189.png -------------------------------------------------------------------------------- /Papers/Network/Anteater.assets/image-20210131125559405.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/Anteater.assets/image-20210131125559405.png -------------------------------------------------------------------------------- /Papers/Network/ERA.assets/image-20210123024347035.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/ERA.assets/image-20210123024347035.png -------------------------------------------------------------------------------- /Papers/Network/ERA.assets/image-20210123173314097.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/ERA.assets/image-20210123173314097.png -------------------------------------------------------------------------------- /Papers/Network/ERA.assets/image-20210123174550969.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/ERA.assets/image-20210123174550969.png -------------------------------------------------------------------------------- /Papers/Network/ERA.assets/image-20210123175901871.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/ERA.assets/image-20210123175901871.png -------------------------------------------------------------------------------- /Papers/Network/ERA.assets/image-20210123182939189.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/ERA.assets/image-20210123182939189.png -------------------------------------------------------------------------------- /Papers/Network/ERA.assets/image-20210123185815810.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/ERA.assets/image-20210123185815810.png -------------------------------------------------------------------------------- /Papers/Network/ERA.assets/image-20210123190218493.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/ERA.assets/image-20210123190218493.png -------------------------------------------------------------------------------- /Papers/Network/ERA.assets/image-20210124000508591.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/ERA.assets/image-20210124000508591.png -------------------------------------------------------------------------------- /Papers/Network/ERA.assets/image-20210124000731328.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/ERA.assets/image-20210124000731328.png -------------------------------------------------------------------------------- /Papers/Network/ERA.assets/image-20210124004607151.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/ERA.assets/image-20210124004607151.png -------------------------------------------------------------------------------- /Papers/Network/ERA.assets/image-20210124011013950.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/ERA.assets/image-20210124011013950.png -------------------------------------------------------------------------------- /Papers/Network/ERA.assets/image-20210124011849610.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/ERA.assets/image-20210124011849610.png -------------------------------------------------------------------------------- /Papers/Network/ERA.assets/image-20210124011930487.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/ERA.assets/image-20210124011930487.png -------------------------------------------------------------------------------- /Papers/Network/ERA.assets/image-20210124170052580.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/ERA.assets/image-20210124170052580.png -------------------------------------------------------------------------------- /Papers/Network/ERA.assets/image-20210124170440092.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/ERA.assets/image-20210124170440092.png -------------------------------------------------------------------------------- /Papers/Network/ERA.assets/image-20210124170527875.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/ERA.assets/image-20210124170527875.png -------------------------------------------------------------------------------- /Papers/Network/ERA.assets/image-20210124170630562.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/ERA.assets/image-20210124170630562.png -------------------------------------------------------------------------------- /Papers/Network/ERA.assets/image-20210124193054748.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/ERA.assets/image-20210124193054748.png -------------------------------------------------------------------------------- /Papers/Network/ERA.assets/image-20210124193106627.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/ERA.assets/image-20210124193106627.png -------------------------------------------------------------------------------- /Papers/Network/ERA.assets/image-20210124193736186.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/ERA.assets/image-20210124193736186.png -------------------------------------------------------------------------------- /Papers/Network/ERA.assets/image-20210124193749131.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/ERA.assets/image-20210124193749131.png -------------------------------------------------------------------------------- /Papers/Network/ERA.assets/image-20210124223958231.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/ERA.assets/image-20210124223958231.png -------------------------------------------------------------------------------- /Papers/Network/ERA.assets/image-20210124224118342.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/ERA.assets/image-20210124224118342.png -------------------------------------------------------------------------------- /Papers/Network/ERA.assets/image-20210125001344979.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/ERA.assets/image-20210125001344979.png -------------------------------------------------------------------------------- /Papers/Network/ERA.assets/image-20210125001353677.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/ERA.assets/image-20210125001353677.png -------------------------------------------------------------------------------- /Papers/Network/ERA.assets/image-20210125001401676.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/ERA.assets/image-20210125001401676.png -------------------------------------------------------------------------------- /Papers/Network/ERA.assets/image-20210125001409997.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/ERA.assets/image-20210125001409997.png -------------------------------------------------------------------------------- /Papers/Network/ERA.assets/image-20210125001422797.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/ERA.assets/image-20210125001422797.png -------------------------------------------------------------------------------- /Papers/Network/ERA.assets/image-20210125001431054.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/ERA.assets/image-20210125001431054.png -------------------------------------------------------------------------------- /Papers/Network/NoD.assets/image-20210120230504275.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/NoD.assets/image-20210120230504275.png -------------------------------------------------------------------------------- /Papers/Network/NoD.assets/image-20210120231506789.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/NoD.assets/image-20210120231506789.png -------------------------------------------------------------------------------- /Papers/Network/NoD.assets/image-20210120231911405.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/NoD.assets/image-20210120231911405.png -------------------------------------------------------------------------------- /Papers/Network/NoD.assets/image-20210120233737352.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/NoD.assets/image-20210120233737352.png -------------------------------------------------------------------------------- /Papers/Network/NoD.assets/image-20210120235040596.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/NoD.assets/image-20210120235040596.png -------------------------------------------------------------------------------- /Papers/Network/NoD.assets/image-20210121000846336.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/NoD.assets/image-20210121000846336.png -------------------------------------------------------------------------------- /Papers/Network/NoD.assets/image-20210121001957603.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/NoD.assets/image-20210121001957603.png -------------------------------------------------------------------------------- /Papers/Network/NoD.assets/image-20210121005410565.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/NoD.assets/image-20210121005410565.png -------------------------------------------------------------------------------- /Papers/Network/NoD.assets/image-20210121010421833.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/NoD.assets/image-20210121010421833.png -------------------------------------------------------------------------------- /Papers/Network/NoD.assets/image-20210121011851576.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/NoD.assets/image-20210121011851576.png -------------------------------------------------------------------------------- /Papers/Network/cnet_verifier.assets/image-20210203232413997.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/cnet_verifier.assets/image-20210203232413997.png -------------------------------------------------------------------------------- /Papers/Network/cnet_verifier.assets/image-20210203233242389.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/cnet_verifier.assets/image-20210203233242389.png -------------------------------------------------------------------------------- /Papers/Network/cnet_verifier.assets/image-20210204154839422.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/cnet_verifier.assets/image-20210204154839422.png -------------------------------------------------------------------------------- /Papers/Network/cnet_verifier.assets/image-20210204154856081.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/cnet_verifier.assets/image-20210204154856081.png -------------------------------------------------------------------------------- /Papers/Network/cnet_verifier.assets/image-20210204154903539.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/cnet_verifier.assets/image-20210204154903539.png -------------------------------------------------------------------------------- /Papers/Network/cnet_verifier.assets/image-20210204154947683.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/cnet_verifier.assets/image-20210204154947683.png -------------------------------------------------------------------------------- /Papers/Network/cnet_verifier.assets/image-20210204154954601.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/cnet_verifier.assets/image-20210204154954601.png -------------------------------------------------------------------------------- /Papers/Network/cnet_verifier.assets/image-20210204155545387.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/cnet_verifier.assets/image-20210204155545387.png -------------------------------------------------------------------------------- /Papers/Network/cnet_verifier.assets/image-20210204162134150.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/cnet_verifier.assets/image-20210204162134150.png -------------------------------------------------------------------------------- /Papers/Network/cnet_verifier.assets/image-20210205011318212.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/cnet_verifier.assets/image-20210205011318212.png -------------------------------------------------------------------------------- /Papers/Network/design_philosophy_of_darpa_internet_protocols.md: -------------------------------------------------------------------------------- 1 | # [Design Philosophy of DARPA Internet Protocols](http://ccr.sigcomm.org/archive/1995/jan95/ccr-9501-clark.pdf) 2 | 3 | ###### David D. Clark 4 | 5 | --- 6 | 7 | ### What is the Problem? [Good papers generally solve *a single* problem] 8 | 9 | 10 | 11 | ### Summary [Up to 3 sentences] 12 | 13 | 14 | 15 | ### Key Insights [Up to 2 insights] 16 | 17 | 18 | 19 | ### Notable Design Details/Strengths [Up to 2 details/strengths] 20 | 21 | 22 | 23 | ### Limitations/Weaknesses [up to 2 weaknesses] 24 | 25 | 26 | 27 | ### Summary of Key Results [Up to 3 results] 28 | 29 | 30 | 31 | ### Open Questions [Where to go from here?] 32 | 33 | 34 | 35 | 36 | ### Self-Keypoints [Delete this when uploading!!] 37 | * Internet architecture 38 | * fundamental goal 39 | * + develop an effective technique for multiplexed utilization of existing interconnected networks (eg. ARPANET - ARPA) 40 | * + unified system? high performance/integration, less practical 41 | * + multiplexing => packet switching (instead of circuit switching - phone) 42 | * + gateway: packet switchs, store and forward packets 43 | * + a packet switched communications facility in which a number of distinguishable networks are connected together using packet communications processors called gateways which implement a store and forward algorithm 44 | * second-level goal 45 | * + Internet communication must continue despite loss of network or gateways 46 | * + - because of a military context (hostile environment) 47 | * + - less concern of detailed accounting of resouces 48 | * + The Internet must support multiple types of communication service 49 | * + The Internet architecture must accommodate a variety of networks 50 | * + The Internet architecture must permit distributed management of its resources 51 | * + - fault-tolerance 52 | * + The Internet architecture must be cost effective 53 | * + The Internet architecture must permit host attachment with a low level of effort 54 | * + The resources used in the internet architecture must be accountable 55 | * + - quantification/measurement 56 | * + in order of importance 57 | * survivability 58 | * + At transport layer, no facility to communicate when synchronization between sender and receiver is lost. Assumption that synchronizaiton never lost unless no physical path (total partition failure). Internet should mask any transient failure. 59 | * + Must protect state information of on-going conversation (# of packets transmitted, # of packet acknowledged, # of outstanding flow control permissions) 60 | * + end-to-end/fate-sharing (no need for robust replicate for saving state in internal nodes) 61 | * + - gateways have no essential state information (stateless packet switches) 62 | * + - most trust is placed in host machine (sequencing, acknowledgement of data) 63 | * types of service 64 | * + at transport level, support a variety of types of service (speed, latency, reliability, delay, bandwidth) 65 | * + bi-directional reliable delivery of data (virtual circuit): remote login, file transfer => TCP 66 | * + TCP (reliable sequenced data stream) 67 | * + UDP (datagram service): not all services need TCP 68 | * + IP (basic building block): separate from TCP 69 | * + don't assume underlying network supporting multiple types of serices => multiple types of services (TCP/UDP) from basic datagram building blocks (IP) 70 | * variety of networks 71 | * + long haul nets (ARPANET, X.25) 72 | * + local area nets (Ethernet/ringnet) 73 | * + broadcast satellite nets (DARPA Atlantic Satellite Network, DARPA Experimental Wideband Satellite Net) 74 | * + packet radio networks (DARPA packet radio network, British packet radio net) 75 | * + variety of serial links 76 | * + other ad hoc facilities (intercomputer busses, HASP in IBM) 77 | * + basic assumption: network can transport a packet or datagram 78 | * + not assumption 79 | * + - reliable/secured delivery 80 | * + - netowrk level broadcast/multicast 81 | * + - priority ranking of transmitted packet 82 | * + - support for multiple type of service 83 | * + - internal knowledge of failture 84 | * + - speeds/delays 85 | * Other goals 86 | * + permitting distributed management of Internet 87 | * + - what about internal gateways? 88 | * + - two-tiered routing algorithm for gateways from different administrations to exchange routing tables (without enough trust) 89 | * + - private routing algorithms 90 | * + - lack of sufficient tools for distributed managemnt of resources in the context of multiple administrations 91 | * + cost effective compared to tailored architecture 92 | * + - long headers (compared to small bytes of data) 93 | * + inefficiency for retransmission of lost packets 94 | * + - end-to-end design => repeat traffic 95 | * + cost of attaching a host 96 | * + - implementing the network utilities 97 | * + - poor implementations (heart-bleed?) 98 | * + accountability 99 | * Architecture and Implementation 100 | * IP/Datagrams 101 | * + datagrams as entity transported 102 | * + eliminate the need for connection state within imtermediate switching nodes 103 | * + basic building block 104 | * + minimum network service assumption 105 | * TCP 106 | * + regulate the delivery of bytes, not packets (while UDP is for packets[datagrams]) 107 | * + - permit the insertion of control information into the sequence space of the bytes 108 | * + - permit TCP packet to be broken up into smaller packets if necessary to fit through a net with a small packet size => moved to IP layer 109 | * + - permit a number of small packets to be gathered together (if retransmission is needed) 110 | * + - limit of throughput 111 | * + end flag (data up to this point is 1 or 1+ complete application level elements) 112 | * + - EOL (End of Letter) 113 | * + - PSH (push flag) 114 | * narrow IP interface hurts innovation at the IP level 115 | * hiding power (DPDK - user-level network interface (reduce memory copy)) -------------------------------------------------------------------------------- /Papers/Network/designing_a_global_name_service.md: -------------------------------------------------------------------------------- 1 | # [Designing a Global Name Service](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/acrobat-15.pdf) 2 | 3 | ###### Butler W. Lampson 4 | 5 | --- 6 | 7 | ### What is the Problem? [Good papers generally solve *a single* problem] 8 | 9 | 10 | 11 | ### Summary [Up to 3 sentences] 12 | 13 | 14 | 15 | ### Key Insights [Up to 2 insights] 16 | 17 | 18 | 19 | ### Notable Design Details/Strengths [Up to 2 details/strengths] 20 | 21 | 22 | 23 | ### Limitations/Weaknesses [up to 2 weaknesses] 24 | 25 | 26 | 27 | ### Summary of Key Results [Up to 3 results] 28 | 29 | 30 | 31 | ### Open Questions [Where to go from here?] 32 | 33 | 34 | 35 | 36 | ### Self-Keypoints [Delete this when uploading!!] 37 | 38 | * Distributed, Persistent, Transactional B-tree 39 | * name service: maps a name for an entity into a set of labeled properties 40 | * + large size: to handle essentially arbitrary number of names and serve an arbitrary number of administrative organizations 41 | * + long life 42 | * + high availability 43 | * + fault isolation 44 | * + tolerance of mistrust 45 | * client level 46 | * + The client sees a structure much like a Unix file system. There is a tree of directories, each with a unique directory identifier (DI) and a name by which it can be reached from its parent. 47 | * + A DR(The arcs of the tree are called directory references) is the value of the name; it consists simply of the DI for the child directory. Thus a directory can be named relative to a root by a path name called its full name (FN). 48 | * + Access control is based on the notion of a principal, which is an entity that can be authenticated by its knowledge of some encryption key (which acts as its password). 49 | * + Authentication is based on the use of encryption to provide a secure channel between the caller of an operation and its implementor. 50 | * administrative level 51 | * + find copies 52 | * + sweep and write-back 53 | * name space 54 | * + fullname and name of an entity registered in that directory 55 | * growth 56 | * + hierarchical strcture 57 | * + merging trees, shadowing 58 | * restructring 59 | * + move tree 60 | * caching 61 | * name service interface 62 | -------------------------------------------------------------------------------- /Papers/Network/development_of_the_Domain_DNS.md: -------------------------------------------------------------------------------- 1 | # [Development of the Domain Name System](https://www.cs.cornell.edu/people/egs/615/mockapetris.pdf) 2 | 3 | ###### Paul V. Mockapetris, Kevin J. Dunlap 4 | 5 | --- 6 | 7 | ### What is the Problem? [Good papers generally solve *a single* problem] 8 | 9 | 10 | 11 | ### Summary [Up to 3 sentences] 12 | 13 | 14 | 15 | ### Key Insights [Up to 2 insights] 16 | 17 | * Hierarchical 18 | * Distributed 19 | * Cached 20 | 21 | ### Notable Design Details/Strengths [Up to 2 details/strengths] 22 | 23 | 24 | 25 | ### Limitations/Weaknesses [up to 2 weaknesses] 26 | 27 | 28 | 29 | ### Summary of Key Results [Up to 3 results] 30 | 31 | 32 | 33 | ### Open Questions [Where to go from here?] 34 | 35 | 36 | 37 | 38 | ### Self-Keypoints [Delete this when uploading!!] 39 | * DNS provides name service for DARPA Internet 40 | * Introduction 41 | * + initially, DNS is hosts.txt system maintained on a host at the SRI and distributed to all hosts 42 | * + then, hosts.txt from NCP-based original ARPANET to the IC/TCP-based Internet 43 | * + then, RFC 44 | * DNS Design 45 | * + provides hosts.txt functionality 46 | * + allow the database to be maintained in a distributed manner 47 | * + have no obvious size lmit for names 48 | * + interoperate across the DARPA Internet and in as many other environments 49 | * + - buffered database information 50 | * + provide tolerable performance 51 | * + DNS architecture 52 | * + - name servers 53 | * + - - repositories of information 54 | * + - - answer queries using whatever information they possess 55 | * + - resolvers 56 | * + - - interface to client program 57 | * + - - embody the algorithms necessary to find a name server that has the information sought by the client 58 | * + DNS namespace 59 | * + - variable-depth tree where each node in the tree has an associated label < 256 octets 60 | * + - labels are variable-length strings of octets < 63 octets 61 | * + data attached to names 62 | * + - data for each name in the DNS is organized as a set of resource records (RRs) 63 | * + - types are meant to represent abstract resources or functions 64 | * + DNS database distribution 65 | * + - zones: sections of the system-wide datbase controlled by specific orgranization 66 | * + - - organization controlling a zone is responsible for distributing current copies of the zones to multiple servers which make the zones avaiable to clients throughtout the Internet. (maintenance of the zone's data and providing redundant servers for the zone) 67 | * + - - a complete description of a contiguous section of the total tree name space, together with some "pointer" information to other continguous zones 68 | * + - - an organization should be able to have a domain 69 | * + - caching 70 | * + - - data acquired in response to a client's request can be locally stored again future requests by the same or other client 71 | * + - - DNS resolvers and combined name server/resolver programs also cache responses for use by later queries. 72 | * + - - TTL(time-to-liev) field attached to each RR presents the length of time that the response can be reused. (adminstrator defines TTL values for each RRas part of the zone definition) 73 | * + - - - continuously decremented TTLs of data in caches 74 | * Implementation 75 | * + root server 76 | * + berkeley 77 | * + surprise 78 | * + - refinement of semantics 79 | * + - high performance 80 | * + - negative caching 81 | * + - - non-exist name: misspelled? 82 | * + - - existing name, non-exist data: ask for host type 83 | * + success 84 | * + - variable depth hierarchy 85 | * + - orgranizational strcturing of names 86 | * + - datagram access 87 | * + - additional section processing 88 | * + - caching 89 | * + - mail address cooperation 90 | * + shortcomings 91 | * + - type and class growth 92 | * + - easy upgrading of applications 93 | * + - distribution of control vs. distribution of expertise or responsibility 94 | * -------------------------------------------------------------------------------- /Papers/Network/end_to_end_argument_in_system_design.md: -------------------------------------------------------------------------------- 1 | # [End-To-End Arguments in System Design](http://web.mit.edu/Saltzer/www/publications/endtoend/endtoend.pdf) 2 | 3 | ###### J.H. Saltzer, D.P. Reed and D.D. Clark 4 | 5 | --- 6 | 7 | ### What is the Problem? [Good papers generally solve *a single* problem] 8 | 9 | * Some functions can completely and correctly implemented only with the knowledge and help of the application at the end points of the system. Providing functions at low levels can be redundant or valueless compared to the cost. 10 | 11 | ### Summary [Up to 3 sentences] 12 | 13 | * The author proposes the end-to-end argument that if the function can only be implemented correctly and completely with assistance of end points, then the function should not be implemented at low level of systems. The author discusses many senarios using end-to-end design including reliable file transfer, delivery acknowledgements, data encryption, duplicate suppression, guaranteed FIFO delivery and transaction. The author also argues about the performance, identification and history of end-to-end applications. 14 | 15 | ### Key Insights [Up to 2 insights] 16 | 17 | * End-to-end design provides a rationale for moving functions upward to layers which are more closed to the application. 18 | * End-to-end design can reduce the complexity of communication subsystem. 19 | 20 | ### Notable Design Details/Strengths [Up to 2 details/strengths] 21 | 22 | * End-to-end design can introduce large overhead if the network has low level of reliability. However, the performance tradeoff is complex. Functions implemented in low level systems can impact other applications or can't be implemented efficiently. 23 | * To identify the end points of system, one should needs careful analysis of system. Force a voice communicating system, it's completely different levels to apply end-to-end principle according to level of required delay (it's like soft/hard deadline of realtime system?) 24 | * easy implementation 25 | * easy upgradation 26 | * endpoint flexibility 27 | * low-level more stable 28 | 29 | ### Limitations/Weaknesses [up to 2 weaknesses] 30 | 31 | * If the network is not reliable, end-to-end design can cause expontentially increase in expected time to transmit. 32 | * For realtime systems like voice transmission, it's unacceptable to use end-to-end design since it needed a constant rate of voice data instead of storaging packets and checksum. 33 | * Need implicit trust on endpoints 34 | * No explictly defined endpoints 35 | * Lack of quantitative analysis 36 | 37 | ### Summary of Key Results [Up to 3 results] 38 | 39 | * End-to-end design can lead to simpler design in some fields of data communication systems including reliable file transfer, delivery acknowledgements, data encryption, duplicate suppression, guaranteed FIFO delivery and transaction. 40 | * End-to-end argument is a accumulated idea which had applied to many previous researches including encryption, data update protocols, error control and so on. It can be viewed as part of a set of rational principles for organizing layered systems. 41 | 42 | ### Open Questions [Where to go from here?] 43 | 44 | * End-to-end argument can be discussed at some layered system in detail, like OSI layers, operating system layers (like user - kernel, user - hardware?) 45 | * The overhead of end-to-end design in different layered systems. 46 | * What's about some network strongly linked to internal nodes, like Tor? 47 | * How to reason about end-to-end in a principle way? 48 | * Given current low-level sophitiscation, is end-to-end relevant? 49 | * How do you take advantage of low-level innovation? 50 | * To what other domains, end-to-end is appliable? 51 | * Any examples that end-to-end fails? 52 | 53 | #### Congestion control 54 | * Is congestion control amenable to an end-to-end implementation? (network is in charge of congestion) 55 | * 56 | 57 | 58 | ### Self-Keypoints 59 | * low-level functions are redundant 60 | * low-level functions are costly 61 | * low-level functions are mainly a performance optimization -------------------------------------------------------------------------------- /Papers/Network/general_configuration_analysis.assets/image-20210120165135228.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/general_configuration_analysis.assets/image-20210120165135228.png -------------------------------------------------------------------------------- /Papers/Network/general_configuration_analysis.assets/image-20210120171023415.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/general_configuration_analysis.assets/image-20210120171023415.png -------------------------------------------------------------------------------- /Papers/Network/general_configuration_analysis.assets/image-20210120171729310.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/general_configuration_analysis.assets/image-20210120171729310.png -------------------------------------------------------------------------------- /Papers/Network/general_configuration_analysis.assets/image-20210120180023717.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/general_configuration_analysis.assets/image-20210120180023717.png -------------------------------------------------------------------------------- /Papers/Network/general_configuration_analysis.assets/image-20210120180254555.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/general_configuration_analysis.assets/image-20210120180254555.png -------------------------------------------------------------------------------- /Papers/Network/general_configuration_analysis.assets/image-20210120180632467.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/general_configuration_analysis.assets/image-20210120180632467.png -------------------------------------------------------------------------------- /Papers/Network/general_configuration_analysis.assets/image-20210120180953385.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/general_configuration_analysis.assets/image-20210120180953385.png -------------------------------------------------------------------------------- /Papers/Network/header_space_analysis.assets/image-20210105145551899.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/header_space_analysis.assets/image-20210105145551899.png -------------------------------------------------------------------------------- /Papers/Network/header_space_analysis.assets/image-20210105221939407.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/header_space_analysis.assets/image-20210105221939407.png -------------------------------------------------------------------------------- /Papers/Network/header_space_analysis.assets/image-20210105222004883.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/header_space_analysis.assets/image-20210105222004883.png -------------------------------------------------------------------------------- /Papers/Network/header_space_analysis.assets/image-20210105224855086.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/header_space_analysis.assets/image-20210105224855086.png -------------------------------------------------------------------------------- /Papers/Network/header_space_analysis.assets/image-20210105225602775.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/header_space_analysis.assets/image-20210105225602775.png -------------------------------------------------------------------------------- /Papers/Network/header_space_analysis.assets/image-20210105230343079.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/header_space_analysis.assets/image-20210105230343079.png -------------------------------------------------------------------------------- /Papers/Network/header_space_analysis.assets/image-20210105232219270.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/header_space_analysis.assets/image-20210105232219270.png -------------------------------------------------------------------------------- /Papers/Network/header_space_analysis.assets/image-20210105235319808.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/header_space_analysis.assets/image-20210105235319808.png -------------------------------------------------------------------------------- /Papers/Network/header_space_analysis.assets/image-20210105235404772.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/header_space_analysis.assets/image-20210105235404772.png -------------------------------------------------------------------------------- /Papers/Network/header_space_analysis.assets/image-20210106001134994.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/header_space_analysis.assets/image-20210106001134994.png -------------------------------------------------------------------------------- /Papers/Network/header_space_analysis.assets/image-20210106161212710.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/header_space_analysis.assets/image-20210106161212710.png -------------------------------------------------------------------------------- /Papers/Network/header_space_analysis.assets/image-20210106161554796.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/header_space_analysis.assets/image-20210106161554796.png -------------------------------------------------------------------------------- /Papers/Network/hsa_policy_checking.assets/image-20210106234344365.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/hsa_policy_checking.assets/image-20210106234344365.png -------------------------------------------------------------------------------- /Papers/Network/hsa_policy_checking.assets/image-20210107221741911.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/hsa_policy_checking.assets/image-20210107221741911.png -------------------------------------------------------------------------------- /Papers/Network/hsa_policy_checking.assets/image-20210107222448466.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/hsa_policy_checking.assets/image-20210107222448466.png -------------------------------------------------------------------------------- /Papers/Network/hsa_policy_checking.assets/image-20210107223021905.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/hsa_policy_checking.assets/image-20210107223021905.png -------------------------------------------------------------------------------- /Papers/Network/hsa_policy_checking.assets/image-20210107223046395.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/hsa_policy_checking.assets/image-20210107223046395.png -------------------------------------------------------------------------------- /Papers/Network/hsa_policy_checking.assets/image-20210107224004179.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/hsa_policy_checking.assets/image-20210107224004179.png -------------------------------------------------------------------------------- /Papers/Network/hsa_policy_checking.assets/image-20210107230109859.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/hsa_policy_checking.assets/image-20210107230109859.png -------------------------------------------------------------------------------- /Papers/Network/minesweeper.assets/image-20210205161334598.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/minesweeper.assets/image-20210205161334598.png -------------------------------------------------------------------------------- /Papers/Network/minesweeper.assets/image-20210205162428277.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/minesweeper.assets/image-20210205162428277.png -------------------------------------------------------------------------------- /Papers/Network/minesweeper.assets/image-20210205164729100.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/minesweeper.assets/image-20210205164729100.png -------------------------------------------------------------------------------- /Papers/Network/minesweeper.assets/image-20210205194648053.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/minesweeper.assets/image-20210205194648053.png -------------------------------------------------------------------------------- /Papers/Network/minesweeper.assets/image-20210206025430507.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/minesweeper.assets/image-20210206025430507.png -------------------------------------------------------------------------------- /Papers/Network/minesweeper.assets/image-20210206025435424.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/minesweeper.assets/image-20210206025435424.png -------------------------------------------------------------------------------- /Papers/Network/minesweeper.assets/image-20210206031333071.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/minesweeper.assets/image-20210206031333071.png -------------------------------------------------------------------------------- /Papers/Network/minesweeper.assets/image-20210206043305609.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/minesweeper.assets/image-20210206043305609.png -------------------------------------------------------------------------------- /Papers/Network/minesweeper.assets/image-20210206043345695.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/minesweeper.assets/image-20210206043345695.png -------------------------------------------------------------------------------- /Papers/Network/minesweeper.assets/image-20210206235346981.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/minesweeper.assets/image-20210206235346981.png -------------------------------------------------------------------------------- /Papers/Network/minesweeper.assets/image-20210206235417337.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/minesweeper.assets/image-20210206235417337.png -------------------------------------------------------------------------------- /Papers/Network/minesweeper.assets/image-20210206235428647.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/minesweeper.assets/image-20210206235428647.png -------------------------------------------------------------------------------- /Papers/Network/minesweeper.assets/image-20210207001147433.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/minesweeper.assets/image-20210207001147433.png -------------------------------------------------------------------------------- /Papers/Network/minesweeper.assets/image-20210207003034023.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/minesweeper.assets/image-20210207003034023.png -------------------------------------------------------------------------------- /Papers/Network/minesweeper.assets/image-20210207003050017.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/minesweeper.assets/image-20210207003050017.png -------------------------------------------------------------------------------- /Papers/Network/minesweeper.assets/image-20210207003057877.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/minesweeper.assets/image-20210207003057877.png -------------------------------------------------------------------------------- /Papers/Network/minesweeper.assets/image-20210207003108662.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/minesweeper.assets/image-20210207003108662.png -------------------------------------------------------------------------------- /Papers/Network/minesweeper.assets/image-20210207003116687.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/minesweeper.assets/image-20210207003116687.png -------------------------------------------------------------------------------- /Papers/Network/minesweeper.assets/image-20210207003127007.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/minesweeper.assets/image-20210207003127007.png -------------------------------------------------------------------------------- /Papers/Network/minesweeper.assets/image-20210207003136394.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/minesweeper.assets/image-20210207003136394.png -------------------------------------------------------------------------------- /Papers/Network/minesweeper.assets/image-20210207003149887.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/minesweeper.assets/image-20210207003149887.png -------------------------------------------------------------------------------- /Papers/Network/minesweeper.assets/image-20210207003159888.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/minesweeper.assets/image-20210207003159888.png -------------------------------------------------------------------------------- /Papers/Network/minesweeper.assets/image-20210207003208122.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/minesweeper.assets/image-20210207003208122.png -------------------------------------------------------------------------------- /Papers/Network/minesweeper.assets/image-20210207003220166.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/minesweeper.assets/image-20210207003220166.png -------------------------------------------------------------------------------- /Papers/Network/minesweeper.assets/image-20210207003232616.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/minesweeper.assets/image-20210207003232616.png -------------------------------------------------------------------------------- /Papers/Network/minesweeper.assets/image-20210207003243302.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/minesweeper.assets/image-20210207003243302.png -------------------------------------------------------------------------------- /Papers/Network/minesweeper.assets/image-20210207003252833.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/minesweeper.assets/image-20210207003252833.png -------------------------------------------------------------------------------- /Papers/Network/minesweeper.assets/image-20210207003301031.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/minesweeper.assets/image-20210207003301031.png -------------------------------------------------------------------------------- /Papers/Network/minesweeper.assets/image-20210207003311311.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/minesweeper.assets/image-20210207003311311.png -------------------------------------------------------------------------------- /Papers/Network/minesweeper.assets/image-20210207004014127.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/minesweeper.assets/image-20210207004014127.png -------------------------------------------------------------------------------- /Papers/Network/minesweeper.assets/image-20210207004137192.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/minesweeper.assets/image-20210207004137192.png -------------------------------------------------------------------------------- /Papers/Network/packet_reconfigurable_switches.md: -------------------------------------------------------------------------------- 1 | # ‎Compiling Packet Programs to Reconfigurable Switches 2 | 3 | **Lavanya Jose, Lisa Yan, George Varghese** 4 | 5 | --- 6 | 7 | ## Summary 8 | 9 | * 10 | 11 | 12 | 13 | ## Introduction 14 | 15 | * -------------------------------------------------------------------------------- /Papers/Network/propane.md: -------------------------------------------------------------------------------- 1 | # Network Configuration Synthesis with Abstract Topologies‎ 2 | 3 | **Ryan Beckett, Ratul Mahajan, Todd Millstein, Jitendra Padhye, David Walker** 4 | 5 | --- 6 | 7 | ## Summary 8 | 9 | * This paper presents Propane/AT, a system to synthesize BGP configurations. Propane/AT takes abstract topologies, high-level specification of routing policy and fault-tolerance requirements as inputs, and generate templates for each role of the network. Abstract topologies defines the structural and role-based invariants, is the compact version of the concrete network. Routing policies specify the traffic predicates, paths and ranks using customized DSL from Propane. Propane then combines the topologies and routing policies into a product graph computed by DFA (capture the flow of routing info over the topology), checks the feasibility of fault tolerance requirements by sound static analysis (inferencing minimum number of edge-disjoint, policy-compliant paths between pairs of nodes), and generates templates from product graphs to compile to vendor-independent mBGPs. If given concrete topologies, the templates can be instantiated by replacing instances of abstract neighbors with the union of all concrete neighbors and replacing prefix template variables with separate entries for each concrete prefix provided. They also proposes a method to compile incrementally only dependent on nodes' direct neighbors. 10 | * This work reminds me of the website generators, generating web pages configuration templates, which will be filled using backend-provided information. To alleviate the stress of network maintainers, could we feed old configurations into the system and infer some autocomplete holes instead of using whole template engines? Moreover, could we generate the abstract topologies from analyzing the existing network, with a interactive system cooperating with maintainers (using static analysis or machine learning)? 11 | * [NetComplete](https://www.usenix.org/conference/nsdi18/presentation/el-hassany) 12 | 13 | 14 | 15 | ## Introduction 16 | 17 | * While they think of their network abstractly, in terms of roles, current synthesis systems operate over concrete topologies 18 | * Even if two devices play the same role, operators cannot specify policy in terms of this role; and even 19 | if specifications for the two devices are similar, there is no guarantee that the systems will generate (syntactically) similar configurations. Perhaps most importantly, if the operators want to debug or analyze system output, they will have to consider hundreds of device configurations instead of just a 20 | handful of role configurations 21 | * Brittle in network evoluation 22 | * Propane/AT allows operators to input abstract topologies in terms of roles and their connectivity 23 | * high-level specification of routing policy (abstract roles) 24 | * fault-tolerance requirements of the network 25 | * generates one template per role 26 | * 27 | 28 | -------------------------------------------------------------------------------- /Papers/Network/rcc.assets/image-20210202144709045.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/rcc.assets/image-20210202144709045.png -------------------------------------------------------------------------------- /Papers/Network/rcc.assets/image-20210202145323485.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/rcc.assets/image-20210202145323485.png -------------------------------------------------------------------------------- /Papers/Network/rcc.assets/image-20210202145329998.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/rcc.assets/image-20210202145329998.png -------------------------------------------------------------------------------- /Papers/Network/rcc.assets/image-20210202150247067.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/rcc.assets/image-20210202150247067.png -------------------------------------------------------------------------------- /Papers/Network/rcc.assets/image-20210202153207318.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/rcc.assets/image-20210202153207318.png -------------------------------------------------------------------------------- /Papers/Network/rcc.assets/image-20210203143150293.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/rcc.assets/image-20210203143150293.png -------------------------------------------------------------------------------- /Papers/Network/rcc.assets/image-20210203150235006.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/rcc.assets/image-20210203150235006.png -------------------------------------------------------------------------------- /Papers/Network/rcc.assets/image-20210203150300339.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/rcc.assets/image-20210203150300339.png -------------------------------------------------------------------------------- /Papers/Network/rcc.assets/image-20210203152103675.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/rcc.assets/image-20210203152103675.png -------------------------------------------------------------------------------- /Papers/Network/rcc.assets/image-20210203152438117.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/rcc.assets/image-20210203152438117.png -------------------------------------------------------------------------------- /Papers/Network/rcc.assets/image-20210203154005418.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/rcc.assets/image-20210203154005418.png -------------------------------------------------------------------------------- /Papers/Network/rcc.assets/image-20210203154042942.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/rcc.assets/image-20210203154042942.png -------------------------------------------------------------------------------- /Papers/Network/software_dataplane_verification.assets/image-20210122005045964.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/software_dataplane_verification.assets/image-20210122005045964.png -------------------------------------------------------------------------------- /Papers/Network/software_dataplane_verification.assets/image-20210122010409744.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/software_dataplane_verification.assets/image-20210122010409744.png -------------------------------------------------------------------------------- /Papers/Network/software_dataplane_verification.assets/image-20210122135540324.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/software_dataplane_verification.assets/image-20210122135540324.png -------------------------------------------------------------------------------- /Papers/Network/software_dataplane_verification.assets/image-20210122135551744.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/software_dataplane_verification.assets/image-20210122135551744.png -------------------------------------------------------------------------------- /Papers/Network/software_dataplane_verification.assets/image-20210122135638021.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/software_dataplane_verification.assets/image-20210122135638021.png -------------------------------------------------------------------------------- /Papers/Network/software_dataplane_verification.assets/image-20210122141327990.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/software_dataplane_verification.assets/image-20210122141327990.png -------------------------------------------------------------------------------- /Papers/Network/symmetry_scaling.assets/image-20210112235226836.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/symmetry_scaling.assets/image-20210112235226836.png -------------------------------------------------------------------------------- /Papers/Network/symmetry_scaling.assets/image-20210113004232931.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/symmetry_scaling.assets/image-20210113004232931.png -------------------------------------------------------------------------------- /Papers/Network/symmetry_scaling.assets/image-20210113125542556.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/symmetry_scaling.assets/image-20210113125542556.png -------------------------------------------------------------------------------- /Papers/Network/symmetry_scaling.assets/image-20210113153742255.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/symmetry_scaling.assets/image-20210113153742255.png -------------------------------------------------------------------------------- /Papers/Network/symmetry_scaling.assets/image-20210113154622950.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/symmetry_scaling.assets/image-20210113154622950.png -------------------------------------------------------------------------------- /Papers/Network/symmetry_scaling.assets/image-20210113154739352.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/symmetry_scaling.assets/image-20210113154739352.png -------------------------------------------------------------------------------- /Papers/Network/symmetry_scaling.assets/image-20210113154747608.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/symmetry_scaling.assets/image-20210113154747608.png -------------------------------------------------------------------------------- /Papers/Network/symmetry_scaling.assets/image-20210113161837703.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/symmetry_scaling.assets/image-20210113161837703.png -------------------------------------------------------------------------------- /Papers/Network/symmetry_scaling.assets/image-20210113161932968.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/symmetry_scaling.assets/image-20210113161932968.png -------------------------------------------------------------------------------- /Papers/Network/symmetry_scaling.assets/image-20210113162058768.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/symmetry_scaling.assets/image-20210113162058768.png -------------------------------------------------------------------------------- /Papers/Network/symmetry_scaling.assets/image-20210113162106048.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/symmetry_scaling.assets/image-20210113162106048.png -------------------------------------------------------------------------------- /Papers/Network/symmetry_scaling.assets/image-20210113162115912.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/symmetry_scaling.assets/image-20210113162115912.png -------------------------------------------------------------------------------- /Papers/Network/symmetry_scaling.assets/image-20210113172154265.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/symmetry_scaling.assets/image-20210113172154265.png -------------------------------------------------------------------------------- /Papers/Network/symmetry_scaling.assets/image-20210113211338272.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/symmetry_scaling.assets/image-20210113211338272.png -------------------------------------------------------------------------------- /Papers/Network/symmetry_scaling.assets/image-20210113211400960.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/symmetry_scaling.assets/image-20210113211400960.png -------------------------------------------------------------------------------- /Papers/Network/symmetry_scaling.assets/image-20210113211623064.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/symmetry_scaling.assets/image-20210113211623064.png -------------------------------------------------------------------------------- /Papers/Network/symmetry_scaling.assets/image-20210113211651055.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/symmetry_scaling.assets/image-20210113211651055.png -------------------------------------------------------------------------------- /Papers/Network/symmetry_scaling.assets/image-20210113211703729.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/symmetry_scaling.assets/image-20210113211703729.png -------------------------------------------------------------------------------- /Papers/Network/symmetry_scaling.assets/image-20210113211711962.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/symmetry_scaling.assets/image-20210113211711962.png -------------------------------------------------------------------------------- /Papers/Network/symmetry_scaling.assets/image-20210113211737082.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/symmetry_scaling.assets/image-20210113211737082.png -------------------------------------------------------------------------------- /Papers/Network/symmetry_scaling.assets/image-20210113211813177.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/symmetry_scaling.assets/image-20210113211813177.png -------------------------------------------------------------------------------- /Papers/Network/symmetry_scaling.assets/image-20210113211842081.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/symmetry_scaling.assets/image-20210113211842081.png -------------------------------------------------------------------------------- /Papers/Network/symmetry_scaling.assets/image-20210113211851968.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/symmetry_scaling.assets/image-20210113211851968.png -------------------------------------------------------------------------------- /Papers/Network/symmetry_scaling.assets/image-20210113212212917.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/symmetry_scaling.assets/image-20210113212212917.png -------------------------------------------------------------------------------- /Papers/Network/symmetry_scaling.assets/image-20210113212246714.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/symmetry_scaling.assets/image-20210113212246714.png -------------------------------------------------------------------------------- /Papers/Network/symmetry_scaling.assets/image-20210113224416010.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/symmetry_scaling.assets/image-20210113224416010.png -------------------------------------------------------------------------------- /Papers/Network/symmetry_scaling.assets/image-20210113224634353.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/symmetry_scaling.assets/image-20210113224634353.png -------------------------------------------------------------------------------- /Papers/Network/symmetry_scaling.assets/image-20210113230000096.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/symmetry_scaling.assets/image-20210113230000096.png -------------------------------------------------------------------------------- /Papers/Network/symmetry_scaling.assets/image-20210113230019729.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/symmetry_scaling.assets/image-20210113230019729.png -------------------------------------------------------------------------------- /Papers/Network/verification_atomic_predicate.assets/image-20210114153806105.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/verification_atomic_predicate.assets/image-20210114153806105.png -------------------------------------------------------------------------------- /Papers/Network/verification_atomic_predicate.assets/image-20210114161253712.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/verification_atomic_predicate.assets/image-20210114161253712.png -------------------------------------------------------------------------------- /Papers/Network/verification_atomic_predicate.assets/image-20210114173906214.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/verification_atomic_predicate.assets/image-20210114173906214.png -------------------------------------------------------------------------------- /Papers/Network/verification_atomic_predicate.assets/image-20210114175239648.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/verification_atomic_predicate.assets/image-20210114175239648.png -------------------------------------------------------------------------------- /Papers/Network/verification_atomic_predicate.assets/image-20210114175300607.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/verification_atomic_predicate.assets/image-20210114175300607.png -------------------------------------------------------------------------------- /Papers/Network/verification_atomic_predicate.assets/image-20210114183110801.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/verification_atomic_predicate.assets/image-20210114183110801.png -------------------------------------------------------------------------------- /Papers/Network/verification_atomic_predicate.assets/image-20210114183201016.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/verification_atomic_predicate.assets/image-20210114183201016.png -------------------------------------------------------------------------------- /Papers/Network/verification_atomic_predicate.assets/image-20210114183559714.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/verification_atomic_predicate.assets/image-20210114183559714.png -------------------------------------------------------------------------------- /Papers/Network/verification_atomic_predicate.assets/image-20210114183728128.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/verification_atomic_predicate.assets/image-20210114183728128.png -------------------------------------------------------------------------------- /Papers/Network/verification_atomic_predicate.assets/image-20210114203109274.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/verification_atomic_predicate.assets/image-20210114203109274.png -------------------------------------------------------------------------------- /Papers/Network/verification_atomic_predicate.assets/image-20210114222633425.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/verification_atomic_predicate.assets/image-20210114222633425.png -------------------------------------------------------------------------------- /Papers/Network/verification_atomic_predicate.assets/image-20210114224303527.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/Network/verification_atomic_predicate.assets/image-20210114224303527.png -------------------------------------------------------------------------------- /Papers/OS/Xen.md: -------------------------------------------------------------------------------- 1 | # [Xen and the Art of Virtualization](https://www.cl.cam.ac.uk/research/srg/netos/papers/2003-xensosp.pdf) 2 | 3 | ###### Paul Barham, Boris Dragovic, Keir Fraser, Steven Hand, Tim Harris, Alex Ho, Rolf Neugebauer, Ian Pratt, Andrew Warfield 4 | 5 | --- 6 | 7 | ### What is the Problem? [Good papers generally solve *a single* problem] 8 | 9 | To subdivide the ample resources of a modern computer, numerous virtualization system is designed but with limitations such as binary incompatiility, high overhead, and security problems. 10 | 11 | ### Summary [Up to 3 sentences] 12 | 13 | * The paper presents an x86 virtual machine monitor which allows multiple commondity operating system to share conventional hardware with safe rosource managment, low overhead and complete functionality. 14 | 15 | ### Key Insights [Up to 2 insights] 16 | 17 | * Traditional full-virtualiation needs to dynamic rewrite hosted machine code to insert intervention in order to solve the problem of x86 virtualization (not satisfying Popek Theorem) and implement shadow version of system structures which will introduce large overhead. 18 | * Hiding resource virtualization like full-virtualization can harm correctness and performance. 19 | 20 | ### Notable Design Details/Strengths [Up to 2 details/strengths] 21 | 22 | * Xen exists in a 64MB section at the top of every address space, thus avoiding a TLB flush when entering and leaving the hypervisor 23 | * Guest OSes are running under ring-1 to let priviledged instructions to be validated and executed by hypervisor. They can register fast exception handler to speed up syscall. 24 | 25 | ### Limitations/Weaknesses [up to 2 weaknesses] 26 | 27 | * Xen requires modification of operating system source code. 28 | 29 | ### Summary of Key Results [Up to 3 results] 30 | 31 | * Xen provides a paravirtualization techinique which has high scalability, secure isolation and high performance close to native Linux. The Xen platform can serve as a convenient and high performance way to deploy services. 32 | 33 | ### Open Questions [Where to go from here?] 34 | 35 | * The security problem. What's the vulnerability which can achieve virtualization escape. 36 | * Is hardware-assisted virtualiation a better idea? 37 | * Xen on ARM? embedded system? (https://www.youtube.com/watch?v=GYb-Qn3KAUM) 38 | * What's Xen now? 39 | -------------------------------------------------------------------------------- /Papers/OS/barrelfish.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/OS/barrelfish.md -------------------------------------------------------------------------------- /Papers/OS/dandelion.md: -------------------------------------------------------------------------------- 1 | # [Dandelion: a Compiler and Runtime for Heterogeneous Systems](https://www.cl.cam.ac.uk/~ey204/teaching/ACS/R212_2015_2016/papers/rossback_sosp_2013.pdf) 2 | 3 | ###### Christopher J. Rossbach, Yuan Yu, Jon Currey, Jean-Philippe Martin, and Dennis Fetterly 4 | 5 | --- 6 | 7 | ### What is the Problem? [Good papers generally solve *a single* problem] 8 | 9 | * Computer system increasingly rely on heterogeneity to achieve greater performance, scalability and energy efficiency. However, programming heterogeneous systems remains challenging because of multiple execution contexts with different programming abstraction and runtimes. 10 | 11 | ### Summary [Up to 3 sentences] 12 | 13 | * In this paper, the authors present Dandelion, a system designed for writing data-parallel applications for hererogeneous systems. At the programming language level, Dandelion uses LINQ and develops a new general purpose cross compiler framework based on .NET on multiple back-ends. At the runtime level, Dandelion adopts the dataflow execution model with several execution engines and asynchronous communication channels managed by Dandelion. 14 | 15 | ### Key Insights [Up to 2 insights] 16 | 17 | * At the programming language level, language integration approach is the most attractive abstraction. 18 | * At the runtime level, architectural heterogeneity demands the system's multiple execution contexts interoperator and compose seamlessly. 19 | 20 | ### Notable Design Details/Strengths [Up to 2 details/strengths] 21 | 22 | * Dandelion embeds a rich set of data-parallel operators using the LINQ language integration framework, which provides a single unified programming frond-end for programmers in C# or F#. 23 | * Dandelion has three layers of runtime: cluster execution engine (Dryad/Moxie) assigns dataflow vertices to available machines and distributes code and graphs; machine execution engines executes its own dataflow graph, managing IO and execution threas; CPU/GPUs runs the dataflow vertices via dataflow engine (PTask). 24 | * To deal with GPU limitations on dynamic allocation and variable-length records, Dandelion does static analysis and collects metadata to try to avoid it and falls back to execute on the CPU. 25 | 26 | ### Limitations/Weaknesses [up to 2 weaknesses] 27 | 28 | * All user-defined functions invoked by LINQ operators must be side-effect free. 29 | * Low-level GPU runtimes have limited support for dynamic memory allocation, so Dandelion doesn't kernelize functions containing dynamic memory allocation and execution only happens on CPUs. 30 | 31 | ### Summary of Key Results [Up to 3 results] 32 | 33 | * Dandelion is able to reach higher performance on a single machine over sequential CPU with different versions of Dandelion including parallel CPU, GPU enabled and GPU enabled with memory allocation hints Dandelion. 34 | * Dandelion is able to reach higher performance on distributed machines over sequential CPU and DyradLINQ with support of GPUs or parallel CPUs. 35 | 36 | ### Open Questions [Where to go from here?] 37 | 38 | * The paper doesn't talk about the heterogenity systems on FPGAs and ASICs. How to apply dataflow engine on special purpose integrated circuits? 39 | * Why LINQ is used commonly here? Can we purpose a better programming language model? 40 | -------------------------------------------------------------------------------- /Papers/OS/exokernel.md: -------------------------------------------------------------------------------- 1 | # [Exokernel: An Operating System Architecture for Application-Level Resource Management](https://pdos.csail.mit.edu/6.828/2016/readings/engler95exokernel.pdf) 2 | 3 | ###### Dawson R. Engler, M. Frans Kaashoek, and James O’Toole Jr. 4 | 5 | --- 6 | 7 | ### What is the Problem? [Good papers generally solve *a single* problem] 8 | 9 | * The fixed high-level abstraction made by traditional operating system hurts application performance, hides information and limits the functionality and flexibility. 10 | 11 | ### Summary [Up to 3 sentences] 12 | 13 | * In this paper, the authors present a new architecture of operating system, exokernel, which separates hardware resource protection and management. The exokernel protects the access to physical resources and exposes secure binding to application-level library operating system. The authors design a real exokernel Aegis and a library operating system ExOS, and measure the performance compared to a typical monolithic UNIX operating system, Ultrix. 14 | 15 | ### Key Insights [Up to 2 insights] 16 | 17 | * Traditional operating system limits the performance, flexibility and functionality of application by fixed the interface and implementation of operating system. 18 | * Exokernel can provide sufficient functionality for application and high performance with limited primitives. Since applications know better about the domain-specific requirement (like data access pattern), they can create specially tailored library operating systems. 19 | 20 | ### Notable Design Details/Strengths [Up to 2 details/strengths] 21 | 22 | * Exokernel uses three techniques for separating physical resources protection from management, secure bindings, visible revocation and abort protocol. Thus, exokernel can tracking ownership of resources, ensuring protection by guarding all resource usage or binding points and revoking access to resources. 23 | * Exokernel employs some principles including securely exposing hardware, exposing allocation, names and revocation. By these principles, Exokernel can provides multiple co-existing library operating system the maximum degree of control. 24 | 25 | ### Limitations/Weaknesses [up to 2 weaknesses] 26 | 27 | * There is no standard for designing Exokernel for different hardware platforms. And if the interface for different Exokernel differs a lot, The operating system needs to do much specialization for different exokernel, which will lead to inefficient generalization or difficult code maintaining for exponential different versions. 28 | 29 | ### Summary of Key Results [Up to 3 results] 30 | 31 | * Exokernel can be made more efficient with limited number of simple primitives than traditional monolithic UNIX operating system. 32 | * Traditional abstraction, such as virtual memory, interprocess communication, can be implemented efficiently at application level, where they can be easily extended, specialized or replaced. 33 | * Applications can create special-purpose implementation of abstraction with good performance as library operating system. 34 | 35 | ### Open Questions [Where to go from here?] 36 | 37 | * Is Exokernel kind like HAL layer in Windows? What's the different between Windows subsystem and library operating system? 38 | * Can exokernel have the market preference in near future? 39 | * Can we trust the exokernel enough? Side-channel attacks can bypass the exokernel and LibOS doesn't check that. -------------------------------------------------------------------------------- /Papers/OS/the_unix_time_sharing_system.md: -------------------------------------------------------------------------------- 1 | # [The UNIX Time-Sharing System](http://web.eecs.umich.edu/~barisk/teaching/eecs582/unix.pdf) 2 | 3 | ###### Dennis M. Ritchie, Ken Thompson, Bell Laboratories 4 | 5 | --- 6 | 7 | ### What is the Problem? [Good papers generally solve *a single* problem] 8 | 9 | 10 | 11 | ### Summary [Up to 3 sentences] 12 | 13 | 14 | 15 | ### Key Insights [Up to 2 insights] 16 | 17 | 18 | 19 | ### Notable Design Details/Strengths [Up to 2 details/strengths] 20 | 21 | * everything is a file 22 | * pipes (IPC) 23 | * do one thing at a time once do it well 24 | * generality 25 | * virtual memory 26 | * + process isolation 27 | * + flexibility 28 | * + access to larger memory user space 29 | * simple interface of syscall 30 | * + `syscall(num, ...)` 31 | * + Windows: wrappers of Wow, Nt, Hal, Sys, Zw... 32 | 33 | ### Limitations/Weaknesses [up to 2 weaknesses] 34 | 35 | * slow syscall 36 | * hide power 37 | * user space pages readable + writable 38 | * bloated monolithic 39 | * hard to modify 40 | * `fork` is wasteful? 41 | * + COW 42 | * + setup, sharing resources 43 | 44 | ### Summary of Key Results [Up to 3 results] 45 | 46 | 47 | 48 | ### Open Questions [Where to go from here?] 49 | 50 | * JIT 51 | 52 | 53 | ### Self-Keypoints [Delete this when uploading!!] 54 | 55 | * PDP-11/45 56 | * + 16 bit word (8-bit byte) computer with 144K bytes of core memory. 57 | * + large number of device drivers and a generous allotment of space for I/O buffers and system tables 58 | * + 1M byte fixed-head disk 59 | * + 4 moving-head disk drives which provide 2.5M bytes on removable disk cartridges 60 | * + 1 moving-head disk drive which uses removable 40M byte disk packs 61 | * + high-speed paper tape reader-punch, 9-track magnetic tape, D-tape 62 | * + 14 variable-speed communication interfaces attached to 100-series datasets and a 201 dataset interface 63 | * File System 64 | * + ordinary files 65 | * + directory 66 | * + - root `/`, `.`, `..` 67 | * + - 14 or fewer characters filename 68 | * + - linking 69 | * + special files 70 | * + - I/O device /dev 71 | * + - file and device I/O are as similar as possible 72 | * + removable file system 73 | * + - mount system request 74 | * + - - name of an existing ordinary file 75 | * + - - name of a direct-access special file whose associated storage volume should have the structure of an independent file system containing its own directory hierarchy 76 | * + - - make a reference and replaces a leaf -> subtree 77 | * + - exception of files on different devices: no link may exist between 1 file system hierarchy and another 78 | * + protection 79 | * + - uid, 777, 6-bits 80 | * + - uid, eid, gid (run/effective/saved) 81 | * + I/O Calls 82 | * + - file descriptor 83 | * + - syscall: create/read/write/seek 84 | * Implementation 85 | * + inode => 482P4 86 | * + - owner 87 | * + - protection bits 88 | * + - physical disk or tape addresses for the file content 89 | * + - size 90 | * + - time of last modification 91 | * + - # of links to the file (# of times it appears in a directory) 92 | * + - a bit indicating whether the file is a directory 93 | * + - a bit indicating whether the file is a special file 94 | * + - a bit indicating whether the file is "large" or "small" 95 | * Processes and Images 96 | * + image: computer execution environment 97 | * + - core image, general register values, status of open files, current directory, like 98 | * + process: execution of a image 99 | * + - fork, split into 2 independently executing process, have independent copies of the original core image, share open files 100 | * + pipes: IPC 101 | * + - interprocess channel `pipe` 102 | * + - a write using a pipe file is blocked until another write 103 | * + execution of programs 104 | * + - `execute` 105 | * + process synchronization 106 | * + - `wait`, suspend execution until 1 of its children has completed execution 107 | * + termination 108 | * + - `exit`, terminate the process, destroy its image, close its open files, generally obliterates it. notify `wait` 109 | * + text segment (RYWN), data segment, stack segment 110 | * Shell 111 | * + `command args...` 112 | * + I/O, `>`, `<` 113 | * + filter, `|` 114 | * + - tee 115 | * + command separators, `&` 116 | * + shell as a command 117 | * + implementation 118 | * + - read - fork - execute 119 | * Traps 120 | * + terminates the process and writes the user's image on file core (core_handler) 121 | * + interrupt 122 | * + quit 123 | * + interrupt masking 124 | * Design 125 | * + usability 126 | * + fairly severe size constraints 127 | * + maintain itself -------------------------------------------------------------------------------- /Papers/PLT/AutoFDO.md: -------------------------------------------------------------------------------- 1 | # [AutoFDO: Automatic Feedback-Directed Optimization for Warehouse-Scale Applications](https://storage.googleapis.com/pub-tools-public-publication-data/pdf/45290.pdf) 2 | 3 | ###### Dehao Chen, David Xinliang Li, Tipp Moseley 4 | 5 | --- 6 | 7 | ### What is the Problem? [Good papers generally solve *a single* problem] 8 | 9 | * The datacenter applications can't take advantage of traditional FDO (feedback-directed optimization) since they have varying performance critical sections, are difficulty to save the logs and can't afford the overhead of instructmented binaries. 10 | 11 | ### Summary [Up to 3 sentences] 12 | 13 | * The author presents AutoFDO, which does feedback-directed optimization on optimized binaries across datacenter binaries. The purpose of AutoFDO is to support iterative compilation, tolerance for stale profile. 14 | 15 | ### Key Insights [Up to 2 insights] 16 | 17 | * For datacenter binaries, the performance critical sections are changing rapidly. 18 | * For datacenter binaries, AutoFDO needs to support iterative compilation and stale profile between releases. 19 | 20 | ### Notable Design Details/Strengths [Up to 2 details/strengths] 21 | 22 | * Use extended source location `(function name, source line offset to function start, discriminator)` to record source-level profile so that a change of one function (recompile, ABI) can't break the the rest profile data. 23 | * AutoFDO considers proved-to-be inlined indirect call, thus there is no back and forth inlining on one specific indirect call in iterative compilations. 24 | 25 | ### Limitations/Weaknesses [up to 2 weaknesses] 26 | 27 | * Since AutoFDO does feedback-directed optimization on optimized binaries, it introduced unrecoverable destructive changes to original source and lead to incorrect annotations. 28 | * If a user needs stable performance, the feedback-directed profiling can lead to cascading overload of the service. 29 | 30 | ### Summary of Key Results [Up to 3 results] 31 | 32 | * AutoFDO archieves 85% performance as good as with instrumentation with support to iterative compilations, tolerence of stale profile data and scaling. 33 | * Comparing FDO and AutoFDO results show that imprecise profiles from real workloads can work as good as precise profiles. 34 | 35 | ### Open Questions [Where to go from here?] 36 | 37 | * Can we record the compiler optimizing process to get precise profiling data from imprecise ones? Is compiler optimization revertable? 38 | * wrong binary, and worse and worse on likely bad branch 39 | * fast-changing binary 40 | -------------------------------------------------------------------------------- /Papers/PLT/fast_static_analysis_c++_virtual.md: -------------------------------------------------------------------------------- 1 | # Fast Static Analysis of C++ Virtual Function Calls 2 | 3 | **David F. Bacon, Peter F. Sweeney** 4 | 5 | --- 6 | 7 | 8 | 9 | ## Introduction 10 | 11 | * resolving virtual function calls 12 | * reduce compiled code size (virtual calls must include all assembly) 13 | * reduce program complexity (expose user a large space of object types and functions) 14 | * compare 3 fast static analysis algorithms for resolving virtual function calls & evaluate ability to solve the problems 15 | 16 | 17 | 18 | ## Static Analysis 19 | 20 | * Unique Name (UN) 21 | 22 | * Class Hierarchy Analysis (CHA) 23 | 24 | * Rapid Type Analysis (RTA) 25 | 26 | * ```c++ 27 | class A { 28 | public: 29 | virtual int foo() { return 1; } 30 | }; 31 | class B : public A { 32 | public: 33 | virtual int foo() { return 2; } 34 | virtual int foo(int i) { return i + 1; } 35 | }; 36 | int main() { 37 | B* p = new B; 38 | int result1 = p->foo(1); 39 | int result2 = p->foo(); 40 | A* q = p; 41 | int result3 = q->foo(); 42 | } 43 | ``` 44 | 45 | 46 | 47 | ### Unique Name 48 | 49 | * optimize at link time, confine to object file information 50 | * 1 implementation of particular virtual function anywhere in the program 51 | * by comparing the mangled name of C++ functions in the object files 52 | * replaced with a direct call 53 | * don't require access to source code 54 | * optimize virtual calls in library code 55 | * inhibits inlining (since on link-time & object code) 56 | * can resolve `result` because only 1 virtual function calls to `foo$int` 57 | 58 | 59 | 60 | ### Class Hierarchy Analysis 61 | 62 | * no derivied of `B`, can solve `result2` & `resutl1` 63 | * static information, ignore identically-named functions 64 | * must have complete program available (potential override) 65 | * but incremental compilation is possible ([[R: CHA paper]]) 66 | 67 | 68 | 69 | ### Rapid Type Analysis 70 | 71 | * starts with call graph generated by CHA, uses information about instantiated classes to further reduce the set of executable virtual functions, reducing the size of the call graph 72 | * E.g. `result3` not resolved by CHA because the type `A*`, but no objects of type `A` is created, RTA can solve it 73 | * sub-objects are not considered true object instantitations (like `A` part in the `B`) 74 | * build set of possible instantiated types optimistically 75 | * traverse starting at `main` 76 | * virutal call sites initially ignored 77 | * construct found? any of the virtual methods of the corresponding class are also traversed 78 | * must analyze the complete program 79 | * flow-intensive, don't keep per-statement information 80 | 81 | 82 | 83 | ## Other Analysis 84 | 85 | * type prediction 86 | 87 | * flow-analysis 88 | 89 | * ```c++ 90 | A* q = new B; 91 | q = new A; 92 | result = q->foo(); 93 | ``` 94 | 95 | * alias analysis 96 | 97 | 98 | 99 | ### Type Safety Issues 100 | 101 | * CHA, RTA both rely on type safety of the programs 102 | 103 | * ```c++ 104 | void* x = (void*) new A; 105 | B* q = (B*) x; 106 | int case2 = q->foo(); 107 | ``` 108 | 109 | * undefined behavior 110 | 111 | * disable CHA/RTA when downcasting 112 | 113 | * most powerful 114 | 115 | * all for monomorphic calls 116 | 117 | * but can't resolve when multiple related object types are used independently 118 | 119 | * disjoint polymorphism 120 | * like `count` and `iter` problem 121 | 122 | 123 | 124 | ## Conclusion 125 | 126 | * RTA is highly effective for all of these purposes 127 | 128 | 129 | 130 | 131 | 132 | 133 | 134 | 135 | 136 | 137 | 138 | 139 | 140 | 141 | 142 | 143 | 144 | 145 | 146 | ## Motivation 147 | 148 | ## Summary 149 | 150 | ## Strength 151 | 152 | ## Limitation & Solution 153 | 154 | 155 | 156 | -------------------------------------------------------------------------------- /Papers/PLT/optimal_register_allocation_for_SSA.md: -------------------------------------------------------------------------------- 1 | # Optimal register allocation for SSA-form programs in polynomial time 2 | 3 | **Sebastian Hack, Gerhard Goos ** 4 | 5 | --- 6 | 7 | 8 | 9 | ## Introduction 10 | 11 | * 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | ## Motivation 30 | 31 | ## Summary 32 | 33 | ## Strength 34 | 35 | ## Limitation & Solution 36 | 37 | 38 | 39 | -------------------------------------------------------------------------------- /Papers/PLT/register_allocation_chordal.md: -------------------------------------------------------------------------------- 1 | # Register Allocation via Coloring of Chordal Graphs 2 | 3 | **Fernando Magno Quintao Pereira, Jens Palsberg ** 4 | 5 | --- 6 | 7 | [CMU 15-411 RegAlloc](https://www.cs.cmu.edu/~fp/courses/15411-f08/lectures/03-regalloc.pdf) 8 | 9 | 10 | 11 | 12 | 13 | ## Introduction 14 | 15 | * optimally color a chordal graph in time linear in \# of edges. 16 | * heuristics for spilling & coalescing 17 | * better than iterated register coalescing with few registers 18 | * register allocation 19 | * integer linear programming, worst EXPTIME (Appel & George) 20 | * polynomial-time heuristics (Briggs, Cooper & Torczon) 21 | * Iterated Register Coalescing alg. (George & Appel) 22 | * ![image-20200129211642572](D:\OneDrive\Pictures\Typora\image-20200129211642572.png) 23 | * observation: interference graph of real-life programs tend to be chordal graph 24 | * ![image-20200129212342697](D:\OneDrive\Pictures\Typora\image-20200129212342697.png) 25 | * the graph in Figure 2(c) is non-chordal because the cycle abcda is chordless 26 | * Chordal graph: NP -> P, perfect 27 | * minimum coloring, 28 | * maximum clique, 29 | * maximum independent set 30 | * minimum covering by cliques 31 | * optimal coloring 32 | * 1-perfect graph: the chromatic number, that is, the minimum number of colors necessary to color the graph, equals the size of the largest clique 33 | * perfect graph: 1-perfect + every induced subgraph is 1-perfect 34 | * ![image-20200129215315123](D:\OneDrive\Pictures\Typora\image-20200129215315123.png) 35 | * Brisk: strict SSA $\to$ prefect interference graphs 36 | * Hack: strict SSA $\to$ chordal interference graphs 37 | * strict: every path from the initial block until the use of a variable v passes through a definition of v. 38 | * phi-function: replaced by copying during SSA-elimination phase 39 | * ![image-20200129213828821](D:\OneDrive\Pictures\Typora\image-20200129213828821.png) 40 | * ![image-20200129214407845](D:\OneDrive\Pictures\Typora\image-20200129214407845.png) 41 | * [[Q: why just not make b, c definition a block?]] 42 | 43 | 44 | 45 | ## Chordal Graphs 46 | 47 | * $\Delta(G)$: maximum outdegree of any vertex in `G` 48 | * $N(v)$: set of neighbors of `v`, adjacent 49 | * clique: undirected graph $G = (V, E)$ which is a subgraph in which every 2 vertices are adjacent 50 | * simplicial vertex: its neighborhood in `G` is a clique 51 | * Simplicial Elimination Ordering of `G`: bijection $\sigma: V(G) \to \{1 \cdots, |V|\}$, such that every vertex $v_i$ is a simplicial vertex in the subgraph induced by $\{v_1, \cdots, v_i\}$ 52 | * ![image-20200129221737346](D:\OneDrive\Pictures\Typora\image-20200129221737346.png) 53 | * Dirac: An undirected graph without self-loops is chordal if and only if it has a simplicial elimination ordering 54 | * ![image-20200129224237097](D:\OneDrive\Pictures\Typora\image-20200129224237097.png) 55 | * ![image-20200129224246872](D:\OneDrive\Pictures\Typora\image-20200129224246872.png) 56 | * ![image-20200129224252072](D:\OneDrive\Pictures\Typora\image-20200129224252072.png) 57 | 58 | 59 | 60 | ## Our Algorithm 61 | 62 | * ![image-20200129224307800](D:\OneDrive\Pictures\Typora\image-20200129224307800.png) 63 | * coloring, spilling, and coalescing, plus an optional phase called pre-spilling. 64 | * Coalescing must be the last stage in order to preserve the optimality of the coloring algorithm, because, after merging nodes, the resulting interference graph can be non-chordal 65 | * the MCS procedure to produce an ordering of the nodes, for use by the pre-spilling and coloring phases. 66 | * ![image-20200129224441071](D:\OneDrive\Pictures\Typora\image-20200129224441071.png) 67 | * ![image-20200129224900551](D:\OneDrive\Pictures\Typora\image-20200129224900551.png) 68 | * ![image-20200129224920927](D:\OneDrive\Pictures\Typora\image-20200129224920927.png) 69 | * ![image-20200129224932831](D:\OneDrive\Pictures\Typora\image-20200129224932831.png) 70 | * ![image-20200129224942552](D:\OneDrive\Pictures\Typora\image-20200129224942552.png) 71 | * ![image-20200129224952560](D:\OneDrive\Pictures\Typora\image-20200129224952560.png) 72 | * ![image-20200129225000680](D:\OneDrive\Pictures\Typora\image-20200129225000680.png) 73 | * ![image-20200129225006911](D:\OneDrive\Pictures\Typora\image-20200129225006911.png) 74 | * ![image-20200129225011992](D:\OneDrive\Pictures\Typora\image-20200129225011992.png) 75 | * ![image-20200129225044856](D:\OneDrive\Pictures\Typora\image-20200129225044856.png) 76 | * ![image-20200129225051126](D:\OneDrive\Pictures\Typora\image-20200129225051126.png) 77 | * ![image-20200129225056376](D:\OneDrive\Pictures\Typora\image-20200129225056376.png) 78 | * ![image-20200129225104943](D:\OneDrive\Pictures\Typora\image-20200129225104943.png) 79 | * ![image-20200129225114225](D:\OneDrive\Pictures\Typora\image-20200129225114225.png) 80 | * ![image-20200129225126879](D:\OneDrive\Pictures\Typora\image-20200129225126879.png) 81 | * ![image-20200129225131494](D:\OneDrive\Pictures\Typora\image-20200129225131494.png) 82 | 83 | 84 | 85 | ## Conclusion 86 | 87 | * ![image-20200129225155655](D:\OneDrive\Pictures\Typora\image-20200129225155655.png) 88 | * 89 | 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | 98 | 99 | 100 | 101 | 102 | ## Motivation 103 | 104 | ## Summary 105 | 106 | ## Strength 107 | 108 | ## Limitation & Solution 109 | 110 | 111 | 112 | -------------------------------------------------------------------------------- /Papers/PLT/the_locality_principle.md: -------------------------------------------------------------------------------- 1 | # [The Locality Principle](http://denninginstitute.com/pjd/PUBS/CACMcols/cacmJul05.pdf) 2 | 3 | ###### Peter J. Denning 4 | 5 | --- 6 | 7 | ### What is the Problem? [Good papers generally solve *a single* problem] 8 | 9 | * The performance of virtual memory is sensitive to choice of replacement algorithm, the way compilers grouped code onto pages and thrashing in multiprogramming. 10 | 11 | ### Summary [Up to 3 sentences] 12 | 13 | * In this journal, the author summarizes the history of development of the idea of locality. The locality principle was needed when virtual memory was invented and cache was used to accerlate, which caused thrashing in multiprogramming. The principle of locality and concepts of working set, temporal and spatial locality were then discovered and adopted as an interpretation of thrashing and consideration of design. 14 | 15 | ### Key Insights [Up to 2 insights] 16 | 17 | * Thrashing was the collapse of system throughput triggered by making the multiprogramming too high (memory was filled with working sets). 18 | * The locality behavior of programs was produced by temporal clustering (looping and executing within modules with private data) and spatial clustering (grouped data such as arrays, sequences, modules) 19 | 20 | ### Notable Design Details/Strengths [Up to 2 details/strengths] 21 | 22 | * The author defined locality as a distrance from a processor to an object x at time t (D(x, t)). Object x should be in the locality set if D(x, t) <= T. The distanced defined here can be based on temporal distance, spatial distance or cost. 23 | * A working set is defined as the set of ages used during a fixed-length sampling window in the immediate past. The working set sequence can be a measureable approximation of locality sequence. 24 | 25 | ### Limitations/Weaknesses [up to 2 weaknesses] 26 | 27 | * For modern multicore architecture, locality can't explain the multiprogramming problems like false cache-line sharing or cache-line ping-poing. 28 | * Security-related. Assume the locality (speculative execution) 29 | 30 | ### Summary of Key Results [Up to 3 results] 31 | 32 | * Thrashing was triggered by making the multiprogramming too high with memory filled with working sets. It can be solved by setting a working set policy. 33 | * The locality principle, that programs accesses behave like temporal clustered and spatial clustered. 34 | 35 | ### Open Questions [Where to go from here?] 36 | 37 | * What's the relationship between locality principle and the human cognitive and coordinative behavior? 38 | * How compilers and CPUs take advantage of the locality principle? 39 | * Will functional programming, quantum computer still have the locality? 40 | * Compiler knows the architecture/hardware detail? cache? -------------------------------------------------------------------------------- /Papers/SE/Augur.md: -------------------------------------------------------------------------------- 1 | # [Augur: Internet-Wide Detection of Connectivity Disruptions](https://www.youtube.com/watch?v=_rlPKcvzGx4) 2 | 3 | ###### Paul Pearcey, Roya Ensafix, Frank Liy, Nick Feamster, Vern Paxson 4 | 5 | --- 6 | 7 | ### What is the Problem? [Good papers generally solve *a single* problem] 8 | 9 | * How can we detect whether pairs of hosts around the world can talk to each other? 10 | 11 | ### Summary [Up to 3 sentences] 12 | 13 | * In this paper, the authors introduce Augur, which is a method and accompanying system that utilizes TCP/IP side channels to measure reachability between Internet locations without directly controlling a measurement vantage point at either location. 14 | 15 | ### Key Insights [Up to 2 insights] 16 | 17 | * When sending IP packets, each generated packets contains a 16-bit IP identifier (IP ID), where many hosts use a single global counter to increment the IP ID value for all packets. 18 | * The connection status of a site and a reflector can be figured out by TCP/IP side channels (called Spooky scan). 19 | 20 | ### Notable Design Details/Strengths [Up to 2 details/strengths] 21 | 22 | * The difference in two IPID value from the reflector can represent the connection status of site and reflector from not blocked, inbound blocking and outbound blocking. 23 | * The authors use statistical detection to determine the disruption. 24 | 25 | ### Limitations/Weaknesses [up to 2 weaknesses] 26 | 27 | * The reflectors need to be Internet infrastructure (ethical reason) and must generate TCP RSP packets when receiving SYN-ACKs for unsolicited connections. The most important assumption is that the reflector must have a shared, monotonically incrementing IP ID. 28 | * The sites need to have SYN-ACK retransmission, no IP address anycast, no ingress filtering and no stateful firewalls or network-specific blocking. 29 | 30 | ### Summary of Key Results [Up to 3 results] 31 | 32 | * The authors use 2050 reflectors over 2134 sites during 17 days and do aggregate analysis of connection disription on sites. They list the top sites experiencing inbound blocking and outbound blocking. 33 | 34 | ### Open Questions [Where to go from here?] 35 | 36 | * The idea of using a side channel is interesting. Can this idea apply in other fields? 37 | -------------------------------------------------------------------------------- /Papers/SE/magpie.md: -------------------------------------------------------------------------------- 1 | # [Using Magpie for request extraction and workload modelling](https://www.usenix.org/legacy/event/osdi04/tech/full_papers/barham/barham.pdf) 2 | 3 | ###### Paul Barham, Austin Donnelly, Rebecca Isaacs and Richard Mortier 4 | 5 | --- 6 | 7 | ### What is the Problem? [Good papers generally solve *a single* problem] 8 | 9 | * Tools to understand complex system behavior are essential for many performance analysis and debugging tasks, yet few exist and there are many open research problems in their developement. 10 | 11 | ### Summary [Up to 3 sentences] 12 | 13 | * In this paper, the authors present Magpie, which is an ubobtrusive and application-agnostic method of extracting the resource consumption and control path of individual requests and a mechanism for constructing a concise model of the application workload. They also show a validation of the accuracy of the extracted workload models using synthetic data and evaluation of performance against realistic workloads. 14 | 15 | ### Key Insights [Up to 2 insights] 16 | 17 | * Workload of a system is comprised of different types of request that take various paths, exercise different set of components and consume differing amounts of system resources. 18 | * Instrumentation framework must support accurate accounting of resource usage between instrumentation points to enable multiple requests sharing a single resource to be distinguished. 19 | 20 | ### Notable Design Details/Strengths [Up to 2 details/strengths] 21 | 22 | * Eschewing a requirement for global identifier can avoid the problems associated with guaranteeing unique identifier allocation, need for complication ad-hoc state management or API modification to manage the identifiers, and can ensure the instrumentation is kept independent of the request. 23 | * To support accurate accounting of resource usage between instrumentation points, it's required to have high precision timestamps since attribution of events to requests relies on properly ordered events. 24 | * Request parser utilizes event schema and temporal joins to associate events. 25 | 26 | ### Limitations/Weaknesses [up to 2 weaknesses] 27 | 28 | * The request parser can only process trace events in the delivered order and no way for it to seek ahead the trace log. 29 | * The request parser needs a request schema for applications which must be written by system experts. 30 | 31 | ### Summary of Key Results [Up to 3 results] 32 | 33 | * Magpie event tracing and parsing only have slight influence on server throughput, namely low overhead event tracing and parsing. 34 | * Magpie successfully extract individual requests and construct representative workload models from e-commerce systems. 35 | 36 | ### Open Questions [Where to go from here?] 37 | 38 | * Can we get rid of the request schema, namely generic request extraction scheme? 39 | 40 | -------------------------------------------------------------------------------- /Papers/SE/microreboot.md: -------------------------------------------------------------------------------- 1 | # [Microreboot – A Technique for Cheap Recovery](https://www.usenix.org/legacy/event/osdi04/tech/full_papers/candea/candea.pdf) 2 | 3 | ###### George Candea, Shinichi Kawamoto, Yuichi Fujiki, Greg Friedman, Armando Fox 4 | 5 | --- 6 | 7 | ### What is the Problem? [Good papers generally solve *a single* problem] 8 | 9 | * Rebooting can be expensive, causing nontrivial service disruption or downtime even when clusters and failover are employed. 10 | 11 | ### Summary [Up to 3 sentences] 12 | 13 | * The authors provides a new fine-grained technique called microrebooting which utilize the separation of process recovery from data recovery. The authors then build a microrebootable prototype in J2EE and evaluate the prototype by client emulator and resource manager. 14 | 15 | ### Key Insights [Up to 2 insights] 16 | 17 | * A significant fraction of software failures in large-scale systems are solved by rebooting with the exact failure causes unknown. 18 | * The crash-only design approach needs fine-grained decoupling components, state segregation, retryable requests and leased resources. 19 | 20 | ### Notable Design Details/Strengths [Up to 2 details/strengths] 21 | 22 | * Microrebootable systems should be partitioned into fine-grain, well isolated components which minimizes the dependency between themselves. 23 | * Microrebootable system need to have self-contained microcheckpoint-able requests. 24 | * Efficient microrebootable system requires a nearly constant-time resource reclamiation. 25 | 26 | ### Limitations/Weaknesses [up to 2 weaknesses] 27 | 28 | * Microrebooting is safe only if the application is crash-only, namely programs that can be safely crashed in whole or by parts and recover quickly every time. 29 | * Microreboot cannot handle interaction with external resources and non-atomic updates on shared state. 30 | 31 | ### Summary of Key Results [Up to 3 results] 32 | 33 | * Microreboots is effective in recovering from the majority of failure modes seen in today's production J2EE systems. 34 | * Comparing to full recovery, microreboots recover faster, reduce functional disruption and reduce lost work. 35 | * Microreboots preserve cluster load dynamics. 36 | 37 | ### Open Questions [Where to go from here?] 38 | 39 | * How can microreboot to be designed in other platforms or frameworks? 40 | * Can non-crash-only programs utilize microreboot? 41 | -------------------------------------------------------------------------------- /Papers/Security/How_DH_fails_in_practice.md: -------------------------------------------------------------------------------- 1 | # [How Diffie-Hellman Fails in Practice](https://weakdh.org/imperfect-forward-secrecy-ccs15.pdf) 2 | 3 | ###### David Adrian, Karthikeyan Bhargavan, Zakir Durumeric, Pierrick Gaudry, Matthew Green, J. Alex Halderman, Nadia Heninger, Drew Springall, Emmanuel Thomé, Luke Valenta, Benjamin VanderSloot, Eric Wustrow, Santiago Zanella-Béguelin, and Paul Zimmermann 4 | 5 | --- 6 | 7 | ### What is the Problem? [Good papers generally solve *a single* problem] 8 | 9 | * The popularly used Diffie-Hellman key exchange mechanism is less secure than widely believed. 10 | 11 | ### Summary [Up to 3 sentences] 12 | 13 | * In this paper, the authors present Logjam, a novel flaw in TLS that lets a man-in-the-middle downgrade connections to 512-bit length key "export-grade" Diffie-Hellman and use number field sieve discrete log algorithm and precomputation on a specified 512-bit group to compute arbitrary discrete logs quickly. 14 | 15 | ### Key Insights [Up to 2 insights] 16 | 17 | * An adversay who performs a large precomputation for a prime p (number field sieve discrete log algorithm) can quickly calculate arbitrary discrete logs in that group, amortizing the cost over all targets that share this parameter. 18 | * For both normal and export-grade Diffie-Hellman, the vast majority of servers use a handful of common prime groups. 19 | 20 | ### Notable Design Details/Strengths [Up to 2 details/strengths] 21 | 22 | * The comply with 90s-era U.S. export restriction on cryptography, SSL 3.0 and TLS 1.0 supported reduced-strength DHE_EXPORT ciphersuites that were restricted to primes no longer than 512 bits. 23 | * 8.4% of Alexa Top 1M HTTPS domains allow DHE_EXPORT, of which 92.3% use one of the two most popular primes. 24 | 25 | ### Limitations/Weaknesses [up to 2 weaknesses] 26 | 27 | * The attack requires that the server allows DHE_EXPORT and uses precomputed primes. 28 | * The attack requires browsers which allow 512-bit prime key in DHE handshakes. 29 | 30 | ### Summary of Key Results [Up to 3 results] 31 | 32 | * The authors present a MITM TLS flaw Logjam using weakness of "export-grade" Diffie-Hellman and common browsers. 33 | * By using number field sieve algorithm, an attacker can perform a single precomputation that depends only on the group and then computes individual logs in that group in a lower cost. 34 | * By estimation the core time of using NFS, the authors believe that computation up to 1024-bit DH key exchange is feasible given nation-state resources and NSA may have already done that. 35 | 36 | ### Open Questions [Where to go from here?] 37 | 38 | * Can we find another flaws which originate in the lack of communication between scholars in different fields (crypto and system)? 39 | * Can we find another flaws which originate in some historical reasons (90s-era ban on exporting algorithm)? 40 | * Can we find another flaws which originate in the improvement on calculation power? -------------------------------------------------------------------------------- /Papers/Security/foreshadow.md: -------------------------------------------------------------------------------- 1 | # [Foreshadow: Extracting the Keys to the Intel SGX Kingdom with Transient Out-of-Order Execution](https://foreshadowattack.eu/foreshadow.pdf) 2 | 3 | ###### Jo Van Bulck, Marina Minkin, Ofir Weisse, Daniel Genkin, Baris Kasikci, Frank Piessens, Mark Silberstein, Thomas F. Wenisch, Yuval Yarom, and Raoul Strackx 4 | 5 | --- 6 | 7 | ### What is the Problem? [Good papers generally solve *a single* problem] 8 | 9 | * Intel Software Guard eXtensions (SGX) is believed to provide hardware-enforced confidentiality and integrity guarantees. However, the authors present a practical software-only microarchitectural attack on SGX. 10 | 11 | ### Summary [Up to 3 sentences] 12 | 13 | * In this paper, the authors present the Foreshadow attack which uses a speculative execution bug in recent Intel x86 precessors to reliably leak plaintext enclave secrets from CPU cache. 14 | 15 | ### Key Insights [Up to 2 insights] 16 | 17 | * A delicate race condition in the CPU's access control logic can allow an attacker to use the results of unauthorized memory accesses in transient out-of-order instructions before they are rolled back. 18 | * The enclave secrets (plaintext) reside in L1 data cache. 19 | 20 | ### Notable Design Details/Strengths [Up to 2 details/strengths] 21 | 22 | * The attack, together with Spectre and Meltdown, is focusing on the microarchtecture of Intel x86 CPU instead of software-level exploits. 23 | * Since deferencing unauthorized enclave memory will only apply abort page semantics and return dummy value, Foreshadow takes advantage of `mprotect` to clear the present bit in corresponding page table and lead to a page fault. 24 | 25 | ### Limitations/Weaknesses [up to 2 weaknesses] 26 | 27 | * The attack relies on that accessing the enclave pages first checks the present bit and incurs a page fault while page is not present before the abort page semantics. 28 | * The attack critically relies on secrets broiught into the L1 cache during the enclaved execution. 29 | 30 | ### Summary of Key Results [Up to 3 results] 31 | 32 | * In this paper, the authors present Foreshadow attack which breaks the confidentiality and integrity provided by Intel SGX. 33 | 34 | ### Open Questions [Where to go from here?] 35 | 36 | * Will AMD TrustZone have similar side channel attacks? 37 | * Will the ultimate solution for covering side channel attacks on OoO execution and prefetching be a new CPU architecture? 38 | -------------------------------------------------------------------------------- /Papers/Security/klee.md: -------------------------------------------------------------------------------- 1 | # [KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs](https://llvm.org/pubs/2008-12-OSDI-KLEE.pdf) 2 | 3 | ###### Cristian Cadar, Daniel Dunbar, Dawson Engler 4 | 5 | --- 6 | 7 | ### What is the Problem? [Good papers generally solve *a single* problem] 8 | 9 | * Many classes of errors are difficult ot find without executing a piece of code. However, with symbolic execution, it's doubtful whether symbolic execution can achieve high coverage on real applications since there are two concerns: (i) exponential number of paths; (2) interaction with surrounding environment. 10 | 11 | ### Summary [Up to 3 sentences] 12 | 13 | * In this paper, the authors present KLEE, which a symbolic execution tool for general applications and automatically generating high-coverage testcases. 14 | 15 | ### Key Insights [Up to 2 insights] 16 | 17 | * Real applications have high complexity with non-obvious input parsing code, tricky boundary conditions and hard-to-follow control flow. 18 | * Real applications have environment dependencies. Its code is controlled by environmental input. 19 | 20 | ### Notable Design Details/Strengths [Up to 2 details/strengths] 21 | 22 | * For pointer aliasing problem, KLEE clones the current state N times to make pointer refer to N objects and constrains pointer in each state to be within bounds. 23 | * KLEE uses several techniques for query optimization: expression rewriting, implied value concretization, constraint set simplification, constraint independence, counter-example cache 24 | 25 | ### Limitations/Weaknesses [up to 2 weaknesses] 26 | 27 | * KLEE only checks for low-level errors and violations of user-level asserts, while developers' tests can validate the application output matches the expected output. 28 | 29 | ### Summary of Key Results [Up to 3 results] 30 | 31 | * KLEE is able to generate higher line coverage with few test cases than developer test cases. 32 | * KLEE is able to find unique bugs in COREUTILS, MINIX, and BUSYBOX tools. 33 | 34 | ### Open Questions 35 | 36 | * How to address the path explosion problem in symbolic execution? -------------------------------------------------------------------------------- /Papers/System/bloat_aware.md: -------------------------------------------------------------------------------- 1 | # A Bloat-Aware Design for Big Data Applications 2 | 3 | 4 | 5 | **Yingyi Bu, Vinayak Borkar, Guoqing Xu, Michael J. Carey** 6 | 7 | --- 8 | 9 | 10 | 11 | ## Introduction 12 | 13 | * OOP like java: inefficiency in managed runtime, impact of huge amount of data 14 | * bloat-aware design paradigm 15 | * ![image-20191107111850980](D:\OneDrive\Pictures\Typora\image-20191107111850980.png) 16 | * What is the space overhead if all data items are represented by Java objects? 17 | * What is the memory management (GC) costs in a typical Big Data application? 18 | * ![image-20191107112043710](D:\OneDrive\Pictures\Typora\image-20191107112043710.png) 19 | * ![image-20191107112053845](D:\OneDrive\Pictures\Typora\image-20191107112053845.png) 20 | * ![image-20191107112111100](D:\OneDrive\Pictures\Typora\image-20191107112111100.png) 21 | * It is the data path that needs a non-convential design & an extremely careful implementation 22 | * The # of data objects in the system has to be bounded & cannot grow proportionally with the size of the data to be processed 23 | * bloat-aware design paradigm 24 | * merge small objects in the storage 25 | * access the merged objects using data processors 26 | * page-based record management 27 | * ![image-20191107112519454](D:\OneDrive\Pictures\Typora\image-20191107112519454.png) 28 | * ![image-20191107112534165](D:\OneDrive\Pictures\Typora\image-20191107112534165.png) 29 | 30 | 31 | 32 | ## Memory Analysis of Big Data Application 33 | 34 | * Low Packing Factor 35 | * header space 36 | * ![image-20191107113032718](D:\OneDrive\Pictures\Typora\image-20191107113032718.png) 37 | * ![image-20191107113146606](D:\OneDrive\Pictures\Typora\image-20191107113146606.png) 38 | * ![image-20191107113153436](D:\OneDrive\Pictures\Typora\image-20191107113153436.png) 39 | * ![image-20191107113203333](D:\OneDrive\Pictures\Typora\image-20191107113203333.png) 40 | * ![image-20191107113215413](D:\OneDrive\Pictures\Typora\image-20191107113215413.png) 41 | * ![image-20191107140554205](D:\OneDrive\Pictures\Typora\image-20191107140554205.png) 42 | * Large volumes of objects & references 43 | * ![image-20191107140513147](D:\OneDrive\Pictures\Typora\image-20191107140513147.png) 44 | * ![image-20191107140540014](D:\OneDrive\Pictures\Typora\image-20191107140540014.png) 45 | 46 | 47 | 48 | ## The Bloat-Aware Design Paradigm 49 | 50 | * ![image-20191107140820117](D:\OneDrive\Pictures\Typora\image-20191107140820117.png) 51 | * ![image-20191107140828478](D:\OneDrive\Pictures\Typora\image-20191107140828478.png) 52 | * Merging & organizing related small data record objects into few large objects (byte buffers) instead of representing them explicitly as one-object-per-record 53 | * Manipulating data by directly accessing buffers (at byte chunk level as opposed to the object level) 54 | * Data Storage Design: Merging Small Objects 55 | * ![image-20191107141005644](D:\OneDrive\Pictures\Typora\image-20191107141005644.png) 56 | * ![image-20191107141028013](D:\OneDrive\Pictures\Typora\image-20191107141028013.png) 57 | * [[R: It's very similar to database page/tuple layout]] 58 | * Data Processor Design: Access Buffers 59 | * ![image-20191107142108244](D:\OneDrive\Pictures\Typora\image-20191107142108244.png) 60 | * ![image-20191107142120269](D:\OneDrive\Pictures\Typora\image-20191107142120269.png) 61 | * ![image-20191107142133157](D:\OneDrive\Pictures\Typora\image-20191107142133157.png) 62 | * ![image-20191107142144285](D:\OneDrive\Pictures\Typora\image-20191107142144285.png) 63 | * ![image-20191107142150804](D:\OneDrive\Pictures\Typora\image-20191107142150804.png) 64 | 65 | 66 | 67 | ## Programming Experience 68 | 69 | * separate logical data access & physical data storage 70 | * ![image-20191107143352509](D:\OneDrive\Pictures\Typora\image-20191107143352509.png) 71 | * ![image-20191107143817125](D:\OneDrive\Pictures\Typora\image-20191107143817125.png) 72 | * ![image-20191107143843068](D:\OneDrive\Pictures\Typora\image-20191107143843068.png) 73 | * ![image-20191107143848740](D:\OneDrive\Pictures\Typora\image-20191107143848740.png) 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 | 85 | 86 | 87 | 88 | 89 | 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | ## Motivation 98 | 99 | * Data intensive frameworks often use object-oriented managed programming language such as Java, Scala. The Java Virtual Machine (JVM) often becomes the bottleneck of data intensive frameworks since it requires extra header space for objects and creates a large number of references which makes garbage collection a heavy work. Since the frameworks often separates the data path and control path, It's unnecessary to force developers to switch back to an unmanaged languages. 100 | 101 | ## Summary 102 | 103 | * In this paper, the authors present a bloat-aware design paradigm in order to solve the inefficiency of using memory in JVM. The authors analyze the current problems in Java runtime: low packing factor memory due to object headers and GC overhead by large volumes of objects and references. The bloat-aware paradigm asks developers to merge and organize related small data record objects into few large objects (e.g. byte buffers) instead of one-object-per-record, and manipulate data by directly accessing buffers. The operations and data storage are transformed manually to accessor classes. The number of accessor objects is always bounded at compile time and doesn't grow proportional with the cardinaity of the dataset. 104 | 105 | ## Strength 106 | 107 | * The bloat-aware paradigm solves the memory usage challenges in managed object-oriented language without re-implementing framework in unmanaged languages. 108 | 109 | ## Limitation & Solution 110 | 111 | * The paradigm needs programmers to manually rewrite their programs. 112 | * [FACADE](http://web.cs.ucla.edu/~wangkai/papers/asplos15.pdf) might be the following work of this paper, by automatically transforming programs into the bloat-aware paradigm. 113 | 114 | -------------------------------------------------------------------------------- /Papers/System/broom.md: -------------------------------------------------------------------------------- 1 | # Broom: sweeping out Garbage Collection from Big Data systems 2 | 3 | 4 | 5 | **Ionel Gog, Jana Giceva, Michael Isard et al.** 6 | 7 | --- 8 | 9 | 10 | 11 | ## Introduction 12 | 13 | * region-based memory management instead of GC 14 | * implicit/explicit graph of stateful data-flow operators executed by worker threads, event-based processing 15 | * ![image-20191107150551400](D:\OneDrive\Pictures\Typora\image-20191107150551400.png) 16 | * ![image-20191107150716957](D:\OneDrive\Pictures\Typora\image-20191107150716957.png) 17 | 18 | 19 | 20 | ## Motivation 21 | 22 | * Naiad for testing 23 | * Batch processing workflows 24 | * Synchronized iterative workflow 25 | 26 | 27 | 28 | ## Case study: Naiad 29 | 30 | * ![image-20191107151014907](D:\OneDrive\Pictures\Typora\image-20191107151014907.png) 31 | * ![image-20191107151029836](D:\OneDrive\Pictures\Typora\image-20191107151029836.png) 32 | * ![image-20191107151037174](D:\OneDrive\Pictures\Typora\image-20191107151037174.png) 33 | 34 | 35 | 36 | ## Broom: out with the GC? 37 | 38 | * ![image-20191107151315796](D:\OneDrive\Pictures\Typora\image-20191107151315796.png) 39 | * Only three regions required for distributed data processing using communicating actors 40 | * ![image-20191107151346283](D:\OneDrive\Pictures\Typora\image-20191107151346283.png) 41 | * ![image-20191107151353307](D:\OneDrive\Pictures\Typora\image-20191107151353307.png) 42 | * ![image-20191107151459996](D:\OneDrive\Pictures\Typora\image-20191107151459996.png) 43 | * ![image-20191107152135531](D:\OneDrive\Pictures\Typora\image-20191107152135531.png) 44 | * ![image-20191107152142316](D:\OneDrive\Pictures\Typora\image-20191107152142316.png) 45 | 46 | 47 | 48 | ## Discussion 49 | 50 | * ![image-20191107152415323](D:\OneDrive\Pictures\Typora\image-20191107152415323.png) 51 | * ![image-20191107152426363](D:\OneDrive\Pictures\Typora\image-20191107152426363.png) 52 | * ![image-20191107152459795](D:\OneDrive\Pictures\Typora\image-20191107152459795.png) 53 | * ![image-20191107152635483](D:\OneDrive\Pictures\Typora\image-20191107152635483.png) 54 | * 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | ## Motivation 73 | 74 | * Memory-managed languages dominate the landscape of systems for computing with Big Data. Most of the data intensive systems are based on Java Virtual Machine or .NET CLR. Automated memory management improves productivity of system developers and end-users, but stresses the runtime GC by allowing a large number of objects, resulting long GC pauses. Since most systems are based on an implicit or explicit graph of stateful data-flow operators executed by worker threads where the operators perform event-based processing of arriving input data, behave as independent actors, the architecture presents an opportunity to revisit standard memory-management, because: actor explicitly share state via message-passing; state held by actors conssits of many fate-sharing objects with common lifetimes; end-users only supply code fragments to system defined operators. 75 | 76 | ## Summary 77 | 78 | * In this paper, the authors present Broom, a region-based memory manager based on Naiad. Broom observes that region-based memory management works well when similar objects with known lifetimes are handled and the information is available in the distributed data processing systems. Only three types of regions are required: transferable regions for messages extending lifetime and accessing by owner; actor-based regions private to owning actors and living as long as actors; temporary regions for short-lived scratchpad memory blocks which are lexically scoped. Users need to annotate their Naiad operators with region-based memory primitives. 79 | 80 | ## Strength 81 | 82 | * Broom solves memory challenges in data intensive frameworks by revisiting old memory management techinique: region-based memory management. Actually Broom explicitly bounds the lifetime of memory objects in a distributed lifetime fashion. 83 | * Broom can co-exist with local GCs. A local GC can be used for actor-scoped regions. 84 | 85 | ## Limitation & Solution 86 | 87 | * Broom needs users to manually annotate the memory regions (lifetime). And it takes extra efforts to care about memory safety between regions. 88 | * Add a static analysis phase to automatically generate region annotations and avoid invalid memory accessing (like Rust borrow checker? Not need for NLL though) 89 | 90 | -------------------------------------------------------------------------------- /Papers/System/gridgraph.md: -------------------------------------------------------------------------------- 1 | # GridGraph: Large-Scale Graph Processing on a Single Machine Using 2-Level Hierarchical Partitioning 2 | 3 | **Xiaowei Zhu, Wentao Han, and Wenguang Chen** 4 | 5 | --- 6 | 7 | 8 | 9 | ## Introduction 10 | 11 | * GridGraph: large-scale graphs on a single machine 12 | * graphs → 1-D partitioned vertex chunks + 2-D partitioned edge blocks 13 | * 1st fine-grained level partitioning in preprocessing 14 | * 2nd coarse-grained level partitioning applied in runtime 15 | * dual sliding window method 16 | * ![image-20191106000527481](D:\OneDrive\Pictures\Typora\image-20191106000527481.png) 17 | 18 | 19 | 20 | ## Graph Representation 21 | 22 | * ![image-20191106105049681](D:\OneDrive\Pictures\Typora\image-20191106105049681.png) 23 | * ![image-20191106105527946](D:\OneDrive\Pictures\Typora\image-20191106105527946.png) 24 | * ![image-20191106105537848](D:\OneDrive\Pictures\Typora\image-20191106105537848.png) 25 | * ![image-20191106105559008](D:\OneDrive\Pictures\Typora\image-20191106105559008.png) 26 | * ![image-20191106105605480](D:\OneDrive\Pictures\Typora\image-20191106105605480.png) 27 | * power-law edge block size distribution 28 | * extra merge phase 29 | * no-sort edge blocks 30 | * good for selective scheduling 31 | * ![image-20191106111710887](D:\OneDrive\Pictures\Typora\image-20191106111710887.png) 32 | 33 | 34 | 35 | ## The Streaming-Apply Processing Model 36 | 37 | * stream-apply processing model 38 | * 1 read-only pass over edges 39 | * 1 write pass over vertices 40 | * ![image-20191106111843983](D:\OneDrive\Pictures\Typora\image-20191106111843983.png) 41 | * ![image-20191106111853255](D:\OneDrive\Pictures\Typora\image-20191106111853255.png) 42 | * ![image-20191106112319840](D:\OneDrive\Pictures\Typora\image-20191106112319840.png) 43 | * ![image-20191106113902424](D:\OneDrive\Pictures\Typora\image-20191106113902424.png) 44 | * Dual sliding window 45 | * ![image-20191106112835575](D:\OneDrive\Pictures\Typora\image-20191106112835575.png) 46 | * col-oriented 47 | * ![image-20191106113919441](D:\OneDrive\Pictures\Typora\image-20191106113919441.png) 48 | * [[T: the critism for GraphChi & X-Stream is this section is strange..]] 49 | * 2-Level Hierarchical Partitioning 50 | * ![image-20191106114021799](D:\OneDrive\Pictures\Typora\image-20191106114021799.png) 51 | * ![image-20191106114519304](D:\OneDrive\Pictures\Typora\image-20191106114519304.png) 52 | * [[Q: where is the benefit explanation??]] 53 | * Execution Implementation 54 | * ![image-20191106114545016](D:\OneDrive\Pictures\Typora\image-20191106114545016.png) 55 | * 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | ## Motivation 74 | 75 | * Large-scale graph processing has attracted interests in both academic and industrial communities. There are some distributed graph processing systems where load imbalance due to graph nature, synchronization overhead due to BSP, fault tolerance overhead are still problems. Out-of-core shared-memory processing is an alternative solutions. However, the existing frameworks have problems in efficiency: sorting shards (GraphChi) and large amount of updates (X-Stream). 76 | 77 | ## Summary 78 | 79 | * In this paper, the authors present GridGraph, which is a fast stream-apply graph processing model. GridGraph splits graphs into vertices chunks and edges grid blocks. In execution, GridGraph also virtually splits edge blocks into second level grids. The 2-level hierarchical partitioning further reduces the amount of I/O. GridGraph provides two interfaces, one for vertices streaming and one for edges streaming. 80 | 81 | ## Strength 82 | 83 | * GridGraph's grid representations, 2-level hierarchical partition, and vertice/edges streaming interface reduces the amount of I/O cost and doesn't need sorting the edges. 84 | * GridGraph offers selective scheduling for eliminating unnecessary streaming data. 85 | 86 | ## Limitation & Solution 87 | 88 | * The description for 2-level hierarchical partition is vague. How does it reduce amount of I/O operations? 89 | * GridGraph has no support for evolving graphs. 90 | * GridGraph can't do a sub-graph computation since it only provides interfaces for all vertices/edges streaming. 91 | * Workaround is selective scheduling. 92 | * But the paper mentions little about that. (Refer the ATC slides) 93 | * Limited the stream-apply processing and grid accessing model. 94 | 95 | -------------------------------------------------------------------------------- /Papers/System/hdfs.md: -------------------------------------------------------------------------------- 1 | # The Hadoop Distributed File System 2 | 3 | * [link](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=5496972) 4 | 5 | 6 | 7 | ## Intro 8 | 9 | * ![1562662982227](D:\OneDrive\Pictures\Typora\1562662982227.png) 10 | * file system component of Hadoop 11 | * reliable 12 | * high bandwidth 13 | * MapReduce paradigm 14 | * +GFS 15 | * metadata -> `NameNode` 16 | * application data -> `DataNode`s 17 | * no RAID 18 | * replicated 19 | * fully-connected, communicate using TCP-based 20 | 21 | 22 | 23 | ## Architecture 24 | 25 | * `NameNode` 26 | * file, directory -> inodes 27 | * permissions 28 | * modification 29 | * access time 30 | * namespace 31 | * diskspace quotas 32 | * namespace tree 33 | * mapping of file blocks to `DataNode`s 34 | * in RAM 35 | * image 36 | * in FS 37 | * checkpoint 38 | * journal 39 | * `DataNode` 40 | * (data, metadata[checksums, generation stamp]) 41 | * 128mb blocks, replicated at `DataNode`s (usually 3) 42 | * startup: -> `NameNode`, handshake 43 | * verify namespace ID 44 | * verify software version 45 | * register with `NameNode` 46 | * <- block report ([id, generation stamp, length]) 47 | * storage ID 48 | * heartbeats -> `NameNode` 49 | * use heartbeat replies as instructions 50 | * replicate blocks to other nodes 51 | * removal local block replicas 52 | * re-register or shut down the node 53 | * send an immediate block report 54 | * HDFS Client 55 | * read/write/delete, create/delete dirs 56 | * first ask `NameNode` -> list of `DataNode`s that host replicas of the block of the file 57 | * write: data pipeline 58 | * ![1562664038630](D:\OneDrive\Pictures\Typora\1562664038630.png) 59 | * API exposing locations of file blocks 60 | * Image & Journal 61 | * namespace image: file system metadata that describes the organization of application data as directories and files 62 | * checkpoint: persistent record of the image written to disk 63 | * journal: write-ahead commit log for changes to the file system (persistent) 64 | * flushed & synched before every change committed 65 | * bottleneck -> batching by `NameNode` transactions 66 | * never changed by the `NameNode` 67 | * replaced entirely when checkpoint created during restart 68 | * requested by the administrator 69 | * requested by the `CheckpointNode` 70 | * `CheckpointNode` 71 | * `NameNode` -> primary / `CheckpointNode` / `BackupNode` 72 | * periodically combine checkpoint + journal -> checkpoint' + empty journal 73 | * `BackupNode` 74 | * creating periodic checkpoints + maintain in-memory up-to-date image of namespace 75 | * journal stream of namespace transactions -> save + apply to own namespace image 76 | * create a checkpoint without downloading checkpoint & journals files from the active `NameNode` 77 | * read-only `NameNode` 78 | * all file system metadata except block locations 79 | * perform all operations of regular `NameNode` except modification of namespace + knowledge of block locations 80 | * Upgrade, File System Snapshots 81 | * `NameNode` merge checkpoint & journal 82 | * `DataNode` (via handshake) create local snapshot 83 | * hard link copy of storage directory 84 | 85 | 86 | 87 | ## File I/O operations 88 | 89 | * read & write 90 | * single-writer, multiple-reader 91 | * write -> lease granted (keep by heartbeat) 92 | * soft limit -> another client preempt 93 | * hard limit -> auto close & recover lease 94 | * append-only 95 | * data pipeline 96 | * ![1562767382897](D:\OneDrive\Pictures\Typora\1562767382897.png) 97 | * block placement 98 | * nodes -> racks -> switch -> core switches 99 | * block placement policy (for replica) 100 | * configurable 101 | * 1st -> writer 102 | * 2nd, 3rd -> 2 diff nodes in diff racks 103 | * rest -> random nodes with restrictions 104 | * <= 1 replica per node 105 | * <=2 replica per rack 106 | * organized as pipeline in the order of their proximity to the first replica 107 | * no `DataNode` contains more than 1 replica 108 | * no rack contains more than 2 replica of the same block (providing there are sufficient racks) 109 | * replication management 110 | * `NameNode` manages the under-/over-replicated 111 | * over-replicated -> remove 112 | * prefer not to reduce number of racks that host replicas 113 | * prefer to remove a replica from the `DataNode` with minimum available disk space 114 | * under-replicated -> replication priority queue 115 | * priority highest: 1 replica ----> 2/3 rep factor: priority lowest 116 | * background thread 117 | * balancer 118 | * balance disk space usage on an HDFS cluster 119 | * |utilization of the node - utilization of the cluster| <= threshold value 120 | * maintain data availability 121 | * minimize inter-rack data copying 122 | * limited bandwidth 123 | * block scanner 124 | * periodically scan & verify stored checksum === block data 125 | * store verification time in human readable log file 126 | * current / prev logs 127 | * corrupt block -> notify `NameNode`, replicate good copy -> reach replication factor -> remove corrupted block 128 | * preserve data ALAP 129 | * decommissioning 130 | * keep include/exclude lists 131 | * `DataNode` decommissioning -> schedule replication -> decommissioned state -> safely removed 132 | * Inter-cluster Data Copy 133 | * MapReduce type `DistCp` 134 | 135 | 136 | 137 | ## Practice at Yahoo 138 | 139 | * `!TODO` 140 | * future work 141 | * automated failover -> Zookeeper 142 | * scalability --> `NameNode` unresponsive due to Java GC 143 | * in memory namespace & block locations 144 | * solution: multiple namespace & `NameNode` 145 | 146 | 147 | 148 | ## Questions 149 | 150 | 1. What's the HDFS consistency model? 151 | * like GFS? (no write) 152 | 2. How to make consensus between `DataNode` block replicas? 153 | * extra consensus alg? 154 | 3. Difference with GFS 155 | 156 | 157 | 158 | ## Notes 159 | 160 | * -------------------------------------------------------------------------------- /Papers/System/mapreduce.md: -------------------------------------------------------------------------------- 1 | # [MapReduce: Simplified Data Processing on Large Clusters](https://static.googleusercontent.com/media/research.google.com/en//archive/mapreduce-osdi04.pdf) 2 | 3 | ###### Jeffrey Dean, Sanjay Ghemawat 4 | 5 | ------ 6 | 7 | ### What is the Problem? [Good papers generally solve *a single* problem] 8 | 9 | - For special-purpose computations, the input data is large and the computations have to be distributed across hundreds of thousands of machines and here comes the problem about how to parallelize the computation, distribute the data and handling failures. 10 | 11 | ### Summary [Up to 3 sentences] 12 | 13 | - The paper presents a new programming model and an associated implementation for processing and generating large data sets called MapReduce. MapReduce provides a interface that enables automatic parallelization and distribution of large-scale computations, combined with an implementation of this interface that achieves high performance on large cluster of commodity PCs. 14 | 15 | ### Key Insights [Up to 2 insights] 16 | 17 | - Most of the computations involves applying a map operation to each logical record in our input in order to compute a set of intermediate key/value pairs, and then applying a reduce operation to all the values that shared the same key, in order to combine the derived data appropriately. 18 | - When the user-supplied map and reduce operators are deterministic functions of their input values, the distributed implementation produces same output as would have been produced by a non-faulting sequential execution of the entire program. 19 | - Restricting the programming model makes it easy to parallelize and distribute computations and to make such computations fault-tolerant. Network bandwidth is a scarce resource. Redundant execution can be used to reduce the impact of slow machines, and to handle machine failures and data loss. 20 | 21 | ### Notable Design Details/Strengths [Up to 2 details/strengths] 22 | 23 | - MapReduce takes advantage of the fact that the input data on GFS is stored on the local disks of the machines that make up the computational cluster. Thus it attempts to schedule a map task on a machine that contains a replica of near a replica of the input data. 24 | - When a MapReduce operation is close to completion, the master schedules backup executions of the remaining in-progress tasks. 25 | 26 | ### Limitations/Weaknesses [up to 2 weaknesses] 27 | 28 | - For large scale worker failures, MapReduce needs to re-execute all the work done by the unreachable worker machines. 29 | - MapReduce needs to transmit and receive large number of files and the network bandwidth might be the limit point. 30 | - The application writer needs to make side-effect of map and reduce operations atomic and idempotent. 31 | 32 | ### Summary of Key Results [Up to 3 results] 33 | 34 | - MapReduce is able to reach high performance with effective backup tasks and dealing with machine failures. 35 | - MapReduce is able to replace the production indexing system used for Google web search service with simple code, little complexity for dealing with parallelization, tolerance, distribution, high performance and high scalability. 36 | 37 | ### Open Questions [Where to go from here?] 38 | 39 | - What's the common schema for a task that can be done by map and reduce operations? (Like the universal and expressiveness of fold) 40 | - Do we have better options for faulting tolerance, parallelization, distribution on the choice of writing and renaming files? -------------------------------------------------------------------------------- /Papers/System/parameter_server.md: -------------------------------------------------------------------------------- 1 | # Scaling Distributed Machine Learning with the Parameter Server 2 | 3 | 4 | 5 | **Mu Li, David G. Anderson, Jun Woo Park, Alexander J. Smola et al.** 6 | 7 | --- 8 | 9 | 10 | 11 | ## Introduction 12 | 13 | * ![image-20191106120423118](D:\OneDrive\Pictures\Typora\image-20191106120423118.png) 14 | * ![image-20191106120435022](D:\OneDrive\Pictures\Typora\image-20191106120435022.png) 15 | * ![image-20191106120500672](D:\OneDrive\Pictures\Typora\image-20191106120500672.png) 16 | * ![image-20191106120523966](D:\OneDrive\Pictures\Typora\image-20191106120523966.png) 17 | * ![image-20191106120530198](D:\OneDrive\Pictures\Typora\image-20191106120530198.png) 18 | * ![image-20191106121352382](D:\OneDrive\Pictures\Typora\image-20191106121352382.png) 19 | 20 | 21 | 22 | ## Machine Learning 23 | 24 | * Risk Minimization 25 | * ![image-20191106121507399](D:\OneDrive\Pictures\Typora\image-20191106121507399.png) 26 | * ![image-20191106121515366](D:\OneDrive\Pictures\Typora\image-20191106121515366.png) 27 | * ![image-20191106121539478](D:\OneDrive\Pictures\Typora\image-20191106121539478.png) 28 | * ![image-20191106121546503](D:\OneDrive\Pictures\Typora\image-20191106121546503.png) 29 | * Generative Models 30 | * ![image-20191106121639501](D:\OneDrive\Pictures\Typora\image-20191106121639501.png) 31 | 32 | 33 | 34 | ## Architecture 35 | 36 | * ![image-20191106121701966](D:\OneDrive\Pictures\Typora\image-20191106121701966.png) 37 | * ![image-20191106121743030](D:\OneDrive\Pictures\Typora\image-20191106121743030.png) 38 | * (K, V) Vectors 39 | * model shared among nodes -> set of k-v pairs 40 | * ![image-20191106121855054](D:\OneDrive\Pictures\Typora\image-20191106121855054.png) 41 | * Range Push & Pull 42 | * ![image-20191106121925950](D:\OneDrive\Pictures\Typora\image-20191106121925950.png) 43 | * UDF on Server 44 | * ![image-20191106121937853](D:\OneDrive\Pictures\Typora\image-20191106121937853.png) 45 | * Asynchronous Tasks & Dependency 46 | * ![image-20191106122522151](D:\OneDrive\Pictures\Typora\image-20191106122522151.png) 47 | * ![image-20191106122539998](D:\OneDrive\Pictures\Typora\image-20191106122539998.png) 48 | * Flexible Consistency 49 | * ![image-20191106122600854](D:\OneDrive\Pictures\Typora\image-20191106122600854.png) 50 | * ![image-20191106122617990](D:\OneDrive\Pictures\Typora\image-20191106122617990.png) 51 | * ![image-20191106122628069](D:\OneDrive\Pictures\Typora\image-20191106122628069.png) 52 | * can be dynamic 53 | * User-Defined Filters 54 | * selectively synchronize individual (k, v) pairs, allowing fine-grained control of data consistency within a task 55 | 56 | 57 | 58 | ## Implementation 59 | 60 | * Vector Clock 61 | * ![image-20191106123135493](D:\OneDrive\Pictures\Typora\image-20191106123135493.png) 62 | * ![image-20191106123157726](D:\OneDrive\Pictures\Typora\image-20191106123157726.png) 63 | * ![image-20191106123203717](D:\OneDrive\Pictures\Typora\image-20191106123203717.png) 64 | * Messages 65 | * ![image-20191106123225078](D:\OneDrive\Pictures\Typora\image-20191106123225078.png) 66 | * key-list cache 67 | * Snappy compression library 68 | * Consistent Hashing 69 | * ![image-20191106123307933](D:\OneDrive\Pictures\Typora\image-20191106123307933.png) 70 | * ![image-20191106123325325](D:\OneDrive\Pictures\Typora\image-20191106123325325.png) 71 | * Replication Consistency 72 | * ![image-20191106123344373](D:\OneDrive\Pictures\Typora\image-20191106123344373.png) 73 | * ![image-20191106123359301](D:\OneDrive\Pictures\Typora\image-20191106123359301.png) 74 | * ![image-20191106123405654](D:\OneDrive\Pictures\Typora\image-20191106123405654.png) 75 | * Server Management 76 | * join/departure 77 | * ![image-20191106123522278](D:\OneDrive\Pictures\Typora\image-20191106123522278.png) 78 | * ![image-20191106123536510](D:\OneDrive\Pictures\Typora\image-20191106123536510.png) 79 | * double message is fine due to vector clocks 80 | * Worker Management 81 | * ![image-20191106123634846](D:\OneDrive\Pictures\Typora\image-20191106123634846.png) 82 | * 83 | 84 | 85 | 86 | 87 | 88 | 89 | 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | 98 | 99 | 100 | ## Motivation 101 | 102 | * Distributed optimization and inference becomes a prerequisite for solving large scale machine learning problems. It's not easy to implement an efficient distributed parameter sharing framework. The key challenges are communication and fault tolerance. The existing systems like Graphlab, REEF, Naiad fail to scale up to industrial needs. 103 | 104 | ## Summary 105 | 106 | * In this paper, the authors present Parameter Server, the third generation open-source implementation of a parameter server for distributed inference. Parameter Server is based on asynchronous push and pull parameters from workers to masters. The messages are denoted with vector clock timestamps as versions and key-value pairs representing feature ID and parameter values. Parameter Server reduces communications by range update, compression and caching. Parameter Server manages its masters in a consistent hashing way and replicates the data to reach fault tolerance. 107 | 108 | ## Strength 109 | 110 | * Parameter Server's asynchronous update model relaxes the consistency but improves the throughput and scalability. 111 | * Parameter Server can support user-defined functions, filters on server node to do extra computations, which provides a global view of the machine learning jobs. 112 | * Parameter Server handles master/worker node elastically changing elegantly by distributed hashing rings (like Amazon dynamo). 113 | * Parameter Server task schedulers appear in each groups, which improves the scalability 114 | 115 | ## Limitation & Solution 116 | 117 | * Parameter Server doesn't provide a flexible programming model, only push-pull k-v pairs. 118 | * Limited to machine-learning like jobs. 119 | * What if the task scheduler fails? What if the resource manager fails? The paper doesn't mention them. 120 | * Maintain a soft-state by pulling states from workers / masters? 121 | 122 | -------------------------------------------------------------------------------- /Papers/System/project_adam.md: -------------------------------------------------------------------------------- 1 | # Project Adam: Building an Efficient and Scalable Deep Learning Training System 2 | 3 | **Trishul Chilimbi, Yutaka Suzue, Johnson Apacible, Karthik Kalyanaraman** 4 | 5 | --- 6 | 7 | 8 | 9 | ## Introduction 10 | 11 | * ![image-20191106135410351](D:\OneDrive\Pictures\Typora\image-20191106135410351.png) 12 | * ![image-20191106135430362](D:\OneDrive\Pictures\Typora\image-20191106135430362.png) 13 | * ![image-20191106135816123](D:\OneDrive\Pictures\Typora\image-20191106135816123.png) 14 | * ![image-20191106135831342](D:\OneDrive\Pictures\Typora\image-20191106135831342.png) 15 | 16 | 17 | 18 | ## Background 19 | 20 | * ![image-20191106140537061](D:\OneDrive\Pictures\Typora\image-20191106140537061.png) 21 | * Training 22 | * Feed-forward evaluation 23 | * ![image-20191106140826312](D:\OneDrive\Pictures\Typora\image-20191106140826312.png) 24 | * Back-propagation 25 | * ![image-20191106140840915](D:\OneDrive\Pictures\Typora\image-20191106140840915.png) 26 | * Weight updates 27 | * ![image-20191106140858316](D:\OneDrive\Pictures\Typora\image-20191106140858316.png) 28 | * Distributed Deep Learning Training 29 | * ![image-20191106141140066](D:\OneDrive\Pictures\Typora\image-20191106141140066.png) 30 | 31 | 32 | 33 | ## Adam System Architecture 34 | 35 | * ![image-20191106141441179](D:\OneDrive\Pictures\Typora\image-20191106141441179.png) 36 | * Modern Training 37 | * multi-threaded 38 | * ![image-20191106143053577](D:\OneDrive\Pictures\Typora\image-20191106143053577.png) 39 | * ![image-20191106143106894](D:\OneDrive\Pictures\Typora\image-20191106143106894.png) 40 | * fast weight updates 41 | * no-lock, race, but can converge 42 | * reducing memory copies 43 | * pointer for local 44 | * asynchronous network IO for non-local 45 | * memory system optimizations 46 | * cache 47 | * stragglers 48 | * dataflow framework tracking 49 | * epoch-end manifests 50 | * 75% estimation 51 | * parameter server communication 52 | * ![image-20191106143656502](D:\OneDrive\Pictures\Typora\image-20191106143656502.png) 53 | * ![image-20191106143710800](D:\OneDrive\Pictures\Typora\image-20191106143710800.png) 54 | * Global Parameter Server 55 | * ![image-20191106143734252](D:\OneDrive\Pictures\Typora\image-20191106143734252.png) 56 | * Throughput Optimizations 57 | * sharding 58 | * batch updates 59 | * SIMD 60 | * NUMA node locality 61 | * lock-free data structure / allocation 62 | * Delayed Persistence 63 | * write-back cache like asynchronously flushed 64 | * resilient nature of DL models 65 | * compress writes 66 | * Fault Tolerant Operations 67 | * 3 copy 68 | * Parameter server controller as Paxos cluster 69 | * Communication Isolation 70 | * 2 NICs for different paths 71 | 72 | 73 | 74 | ## Motivation 75 | 76 | * Large deep neural network is becoming popular due to state-of-the-art performance in many tasks. These models require time consuming computations and generate large amount of data for communication. We need a scalable, high performance, fault tolerant distribution solution for completing machine learning jobs. 77 | 78 | ## Summary 79 | 80 | * In this paper, the authors present Project Adam, which is a Multi-Spert based large scale deep learning training framework. Project Adam consists of three parts, data serving machines (sharded data), model training machines (workers on local data) and a global parameter server (asynchronously shared model). Project Adam employs many techniques, like multi-threading, lock-free data structures, NUMA-aware/cache-aware blocking/sharding, SIMD operations, batch updates, to improve the locality, throughput and reduce overhead on communications. Project Adam also implements fault tolerance by replication asynchronously. 81 | 82 | ## Strength 83 | 84 | * Project Adam employs the resilient nature of machine learning models, to avoid synchronization and overhead lead by strong consistency (both local data race and global asynchronously update/write-back cache). 85 | * Project Adam can send activation and error gradient instead of weight updates, to balance the computation between workers and parameter servers. 86 | 87 | ## Limitation & Solution 88 | 89 | * Project Adam limits the computation model to weights updates. 90 | * Project Adam has no method to limit the asynchronous nature as what Parameter Server does. Without vector clock timestamp, Project Adam can't do sequential, bounded delay updates. 91 | * Add vector clock timestamp? 92 | * Project Adam doesn't specify how to elastically scale for worker/parameter server add/removal. -------------------------------------------------------------------------------- /Papers/System/tensorflow.md: -------------------------------------------------------------------------------- 1 | # [TensorFlow: A system for large-scale machine learning](https://www.usenix.org/system/files/conference/osdi16/osdi16-abadi.pdf) 2 | 3 | ###### Martin Abadi, Paul Barham et al. 4 | 5 | --- 6 | 7 | ### What is the Problem? [Good papers generally solve *a single* problem] 8 | 9 | * In recent years, machine learning has driven advances in many different fields and there are invention of more sophisticated machine learning models, availability of large datasets for tackling problems and developement of software platforms that enables the easy use of large amount of computational resources for training models on large datasets. Previous systems like DistBelief need global parameter server and lack of flexibility to define new layers, refining the training algorithms and defining new traning algorithms. 10 | 11 | ### Summary [Up to 3 sentences] 12 | 13 | * In this paper, the authors present TensorFlow, which is a machine learning system that operates at large scale and in heterogeneous environments. Tensorflow uses dataflow graphs to represent computation, shared state and operations that mutate the state. 14 | 15 | ### Key Insights [Up to 2 insights] 16 | 17 | * Tensorflow is designed to support computation across multiple machines in a cluster or single machine with multiple computational devices including multicore CPUs, GPUs, custom-designed ASIC known as TPU (Tensor Processing Units) 18 | * Unlike traditional dataflow systems in which graph vertices represent functional computation on immutable data, TensorFlow allows vertices to represent computations that own or update mutable state and unifies the computation and state management in a single programming model by representing individual functional operators, mutable states and updating operations as nodes in the dataflow graph. 19 | 20 | ### Notable Design Details/Strengths [Up to 2 details/strengths] 21 | 22 | * A typical TensorFlow application has two distinct phases: first phase defines the program as symbolic dataflow graphs with placeholders for input data and variables; second phase executes an optimized version of the program on the set of available device. The deferred execution allows TensorFlow to optimize the execution by using global information about the computation. 23 | * TensorFlow defines a common abstraction for devices, including issuing a kernel for execution, allocating memory for inputs and outputs, transferring buffers to and from host memory, and specific implementations on functional operators. 24 | 25 | ### Limitations/Weaknesses [up to 2 weaknesses] 26 | 27 | * TensorFlow may have lower performance compared to some special hand-optimized operations and algorithms. 28 | * TensorFlow is not able to do automatic optimization (including automatic placement, kernel fusion, memory management, scheduling) to achieve excellent performance without experts 29 | 30 | ### Summary of Key Results [Up to 3 results] 31 | 32 | * TensorFlow is able to achieve high performance for single-machine and high throughput in clusters for asynchronous and synchronous training. 33 | 34 | ### Open Questions [Where to go from here?] 35 | 36 | * The dataflow graph might be similar (duality?) to control flow graph, can we use highly-developed compiler techiniques (like using LLVM) to optimize them? 37 | 38 | -------------------------------------------------------------------------------- /Papers/conference_list.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Airtnp/Notes/d0d2677d0316f24fd5681516b4f38daa7f7269ab/Papers/conference_list.pdf -------------------------------------------------------------------------------- /Papers/template.md: -------------------------------------------------------------------------------- 1 | # ‎Paper Name‎ 2 | 3 | **Author Name** 4 | 5 | --- 6 | 7 | ## Summary 8 | 9 | * 10 | 11 | 12 | 13 | ## Introduction 14 | 15 | * 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | * CS251A requirements 26 | * a short paragraph summarizing the problem and goal/contributions of paper 27 | * a short paragraph summarizing the paper’s methods and results 28 | * a short paragraph giving your opinion of what is good and bad about the paper. 29 | 30 | ## Summary 31 | 32 | ## Methods & Results 33 | 34 | ## Personal Opinions 35 | 36 | 37 | 38 | 39 | 40 | * EECS582 Template 41 | 42 | ## Motivation 43 | 44 | ## Summary 45 | 46 | ## Strength 47 | 48 | ## Limitation & Solution 49 | 50 | 51 | 52 | -------------------------------------------------------------------------------- /Proposals/Rust_rfc.mdown: -------------------------------------------------------------------------------- 1 | # Rust RFCs 2 | 3 | * [rfcs](https://github.com/rust-lang/rfcs) 4 | 5 | ## 0001-private-fields 6 | * -------------------------------------------------------------------------------- /Proposals/ghc_proposal.mdown: -------------------------------------------------------------------------------- 1 | # GHC proposals 2 | 3 | * [proposals](https://github.com/ghc-proposals/ghc-proposals/tree/master/proposals) 4 | 5 | ## 0000 Templates 6 | -------------------------------------------------------------------------------- /Proposals/python_pep.mdown: -------------------------------------------------------------------------------- 1 | # Python PEPs 2 | 3 | [peps](https://github.com/python/peps) 4 | 5 | ## PEP 0001 6 | -------------------------------------------------------------------------------- /ReadRust/intro.md: -------------------------------------------------------------------------------- 1 | # Rust源码瞎读 2 | 3 | 基于2019/01/01的`36500deb1a`commit 4 | 5 | 希望这是个stable版本又不会太复杂... (像《Python源码分析》的Python2.6) 6 | 7 | 练习中英文写作 8 | 9 | 规定 10 | 11 | * RFC引用: [RFC-XXXX]() 12 | * Issue/Tracker引用: [#XXXXX]() 13 | * [Pull request]引用: [!XXXXX]() 14 | * 全文使用英文括号 15 | 16 | ## Update to track 17 | * async/await 18 | * NLL 19 | * MIR/HIR 20 | * miri 21 | * const generics 22 | * generic associated types (HKT/GAT) 23 | * specialization 24 | * new hashtable 25 | * + [hashbrown](https://github.com/Amanieu/hashbrown) 26 | * + [FxHash] 27 | * + [SwissTable] 28 | * parser 29 | * + [libsyntax] 30 | * + [rust-analyer](https://github.com/rust-analyzer/rust-analyzer) 31 | * + [syn](https://github.com/dtolnay/syn) 32 | + [chalk]() 33 | + cargo pipeline 34 | + new channel 35 | + parking_lot 36 | + crossbeam 37 | 38 | 39 | 40 | 41 | ## 参考(Reference) 42 | 43 | * [Unstable-book](https://doc.rust-lang.org/nightly/unstable-book/) 44 | * 特性门(feature gate)相关 45 | * [Rustonomicon](https://doc.rust-lang.org/stable/nomicon/) 46 | * [Doc](https://doc.rust-lang.org/nightly/std/index.html) 47 | * [nightly-rustc-doc](https://doc.rust-lang.org/nightly/nightly-rustc/rustc/) 48 | * [Issues](https://github.com/rust-lang/rust/issues) 49 | * [RFCs](https://github.com/rust-lang/rfcs/issues) 50 | * [TRPL](https://doc.rust-lang.org/book/index.html) 51 | * [Edition-guide](https://rust-lang-nursery.github.io/edition-guide/introduction.html) 52 | * 2015 vs 2018 53 | * [Reference](https://doc.rust-lang.org/reference/introduction.html) 54 | * 文法相关 55 | 56 | 57 | 58 | ## 综述 59 | 60 | `rust/src`各个文件夹的基本内容 61 | 62 | * `bootstrap` 63 | * `build_helper` 64 | * `ci` 65 | * 给Code Integration使用 66 | * `doc` 67 | * 文档， 包括(edition-guide, man, rust-by-example, rustc-guide) 68 | * `etc` 69 | * gdb plugin 70 | * `grammar` 71 | * `jemalloc` 72 | * [jemalloc](https://github.com/jemalloc/jemalloc) 一种memory allocator的实现 73 | * `liballoc` 74 | * `libarena` 75 | * `libcompiler_builtins` 76 | * `libcore` 77 | * `libfmt_macros` 78 | * `libgraphviz` 79 | * `liblibc` 80 | * `libpanic_abort` 81 | * `libpanic_unwind` 82 | * `libprofiler_builtins` 83 | * `librustc` 84 | * `librustc_allocator` 85 | * `librustc_apfloat` 86 | * `librustc_borrowck` 87 | * `librustc_asan` 88 | * `librustc_codegen_llvm` 89 | * `librustc_codegen_ssa` 90 | * `librustc_codegen_utils` 91 | * `librustc_cratesio_shim` 92 | * `librustc_data_structure` 93 | * `librustc_driver` 94 | * `librustc_errors` 95 | * `librustc_fs_util` 96 | * `librustc_incremental` 97 | * `librustc_lint` 98 | * `librustc_llvm` 99 | * `librustc_lsan` 100 | * `librustc_metadata` 101 | * `librustc_mir` 102 | * `librustc_msan` 103 | * `librustc_passes` 104 | * `librustc_platform_intrinsics` 105 | * `librustc_plugin` 106 | * `librustc_privacy` 107 | * `librustc_resolve` 108 | * `librustc_save_analysis` 109 | * `librustc_target` 110 | * `librustc_traits` 111 | * `librustc_typeck` 112 | * `librustdoc` 113 | * `libserialize` 114 | * `libstd` 115 | * `libsyntax` 116 | * `libsyntax_ext` 117 | * `libsyntax_pos` 118 | * `libterm` 119 | * `libtest` 120 | * `libunwind` 121 | * `llvm` 122 | * [LLVM Project](https://github.com/llvm-mirror/llvm) 123 | * `llvm_emscripten` 124 | * [emscripten](https://github.com/kripken/emscripten) LLVM-to-JS 125 | * `rt` 126 | * `rtstartup` 127 | * `rustc` 128 | * `rustllvm` 129 | * `stdsimd` 130 | * [SIMD RFC](https://github.com/rust-lang/rfcs/blob/master/text/2325-stable-simd.md) 131 | * `test` 132 | * testcase 133 | * `tools` -------------------------------------------------------------------------------- /ReadRust/miscs.md: -------------------------------------------------------------------------------- 1 | ## grammar 2 | 3 | 并不是真正的rust parser, 仅为了证明rust LALR gammar可行性, 从这个repo [rust-grammar](https://github.com/bleibig/rust-grammar) 拷贝而来 (last commit in 2017). 新的canonical grammar group [WG-grammar](https://github.com/rust-lang-nursery/wg-grammar), [#30942](https://github.com/rust-lang/rust/issues/30942) 4 | 5 | 6 | 7 | ### lexer.l 8 | 9 | parser的词法部分，包含各种keyword, symbol 10 | 11 | * `ident [a-zA-Z\x80-\xff_][a-zA-Z0-9\x80-\xff_]*` identifier 可用非0-9开头的Extend ASCII (`\w`, `_`, `\x80-\xff`)。实际不支持Non-ASCII identifier [#28979](https://github.com/rust-lang/rust/issues/28979) 12 | 13 | ![funny-snake-case](D:\OneDrive\Pictures\Typora\D%5COneDrive%5CPictures%5CTypora%5C1547128715211.png) 14 | 15 | * 第一行的话省略BOM (`\xef\xbb\xbf`)，否则报错 16 | * 单独的`_`作为特殊的`UNDERSCORE` 来解析 17 | * 保留的keyword [reserved-keyword](https://doc.rust-lang.org/book/appendix-01-keywords.html#keywords-reserved-for-future-use) 18 | * doc不全，保留的keyword还有 `alignof`, `offsetof`, `proc` 19 | * shebang或是inner attribute都是以`#!`开头，故一起处理 20 | * pound `#` 后实际可能跟attribute, shebang, raw string 21 | * 各种string literal的处理 22 | * `'` : char 或 lifetime 23 | * `"` : str 24 | * `b"`: byte str 25 | * `br"`: raw byte str without hash 26 | * `br#`: ... 27 | * [tokens](https://doc.rust-lang.org/beta/reference/tokens.html) 28 | * ![string-literals](D:\OneDrive\Pictures\Typora\D%5COneDrive%5CPictures%5CTypora%5C1547136868162.png) 29 | 30 | 31 | 32 | ### parser-lalr-main.c + parser-lalr.y + tokens.h 33 | 34 | LALR语法+flex/bison解析rust 35 | 36 | 37 | 38 | ### raw-string-literal-ambiguity.md 39 | 40 | 解释了Rust不属于CFG(context-free grammar)的一个原因： raw string literals. raw string literal 的文法如下 41 | 42 | ``` 43 | R -> 'r' S 44 | S -> '"' B '"' 45 | S -> '#' S '#' 46 | B -> . B 47 | B -> ε 48 | ``` 49 | 50 | 引理1： (CFG ∩ Regular language) ∈ CFG 51 | 52 | 引理2： [pumping lemma](https://en.wikipedia.org/wiki/Pumping_lemma_for_context-free_languages)/[泵引理](https://zh.wikipedia.org/wiki/%E6%B3%B5%E5%BC%95%E7%90%86) 53 | 54 | raw string grammar ∩ (`r#+""#*"#+`) = `{r#^n""#^m"#^n | m < n}`, 而这个文法，用pumping lemma可得非CFG, 故原文法非CFG，具体细节见原文档 -------------------------------------------------------------------------------- /readme.md: -------------------------------------------------------------------------------- 1 | # Notes 2 | 3 | My notes --------------------------------------------------------------------------------