├── .github └── .agp ├── .gitignore ├── LICENSE ├── README.md ├── _config.yml ├── lua ├── async.lua └── tutorial.lua └── plugin └── neovim_async_tutorial.vim /.github/.agp: -------------------------------------------------------------------------------- 1 | 2 | Auto Github Push (AGP) 3 | https://github.com/ms-jpq/auto-github-push 4 | 5 | --- 6 | 2023-08-22 05:28 7 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .git/ 2 | .DS_Store 3 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | 2 | MIT License 3 | 4 | Copyright (c) 2020 whocares 5 | 6 | Permission is hereby granted, free of charge, to any person obtaining a copy 7 | of this software and associated documentation files (the "Software"), to deal 8 | in the Software without restriction, including without limitation the rights 9 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 10 | copies of the Software, and to permit persons to whom the Software is 11 | furnished to do so, subject to the following conditions: 12 | 13 | The above copyright notice and this permission notice shall be included in all 14 | copies or substantial portions of the Software. 15 | 16 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 17 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 18 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 19 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 20 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 21 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 22 | SOFTWARE. 23 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # [Lua Async Await](https://ms-jpq.github.io/lua-async-await) 2 | 3 | Async Await in [90 lines](https://github.com/ms-jpq/lua-async-await/blob/master/lua/async.lua) of code. 4 | 5 | Originally written for Neovim, because it uses the same `libuv` eventloop from NodeJS. 6 | 7 | **Works for any LUA code** 8 | 9 | ## Special Thanks 10 | 11 | [svermeulen](https://github.com/svermeulen) for fixing [inability to return functions](https://github.com/ms-jpq/lua-async-await/issues/2). 12 | 13 | ## Preface 14 | 15 | This tutorial assumes that you are familiar with the concept of `async` `await` 16 | 17 | You will also need to read through the [first 500 words](https://www.lua.org/pil/9.1.html) of how coroutines work in lua. 18 | 19 | ## [Luv](https://github.com/luvit/luv) 20 | 21 | Neovim use [libuv](https://github.com/libuv/libuv) for async, the same monster that is the heart of NodeJS. 22 | 23 | The `libuv` bindings are exposed through `luv` for lua, this is accessed using `vim.loop`. 24 | 25 | Most of the `luv` APIs are similar to that of NodeJS, ie in the form of 26 | 27 | `API :: (param1, param2, callback)` 28 | 29 | Our goal is avoid the dreaded calback hell. 30 | 31 | ## Preview 32 | 33 | ```lua 34 | local a = require "async" 35 | 36 | local do_thing = a.sync(function (val) 37 | local o = a.wait(async_func()) 38 | return o + val 39 | end) 40 | 41 | local main = a.sync(function () 42 | local thing = a.wait(do_thing()) -- composable! 43 | 44 | local x = a.wait(async_func()) 45 | local y, z = a.wait_all{async_func(), async_func()} 46 | end) 47 | 48 | main() 49 | ``` 50 | 51 | ## [Coroutines](https://www.lua.org/pil/9.1.html) 52 | 53 | If you don't know how coroutines work, go read the section on generators on [MDN](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Iterators_and_Generators). 54 | 55 | It is in js, but the idea is identical, and the examples are much better. 56 | 57 | --- 58 | 59 | Here is an example of coroutines in Lua: 60 | 61 | Note that in Lua code `coroutine` is not a coroutine, it is an namespace. 62 | 63 | To avoid confusion, I will follow the convention used in the Lua book, and use `thread` to denote coroutines in code. 64 | 65 | ```lua 66 | local co = coroutine 67 | 68 | local thread = co.create(function () 69 | local x, y, z = co.yield(something) 70 | return 12 71 | end) 72 | 73 | local cont, ret = co.resume(thread, x, y, z) 74 | ``` 75 | 76 | --- 77 | 78 | Notice the similarities with `async` `await` 79 | 80 | In both `async` `await` and `coroutines`, the LHS of the assignment statements receives values from the RHS. 81 | 82 | This is how it works in all synchronous assignments. Except, we can defer the transfer of the values from RHS. 83 | 84 | The idea is that we will make RHS send values to LHS, when RHS is ready. 85 | 86 | ## Synchronous Coroutines 87 | 88 | To warm up, we will do a synchronous version first, where the RHS is always ready. 89 | 90 | Here is how you send values to a coroutine: 91 | 92 | ```lua 93 | co.resume(thread, x, y, z) 94 | ``` 95 | 96 | --- 97 | 98 | The idea is that we will repeat this until the coroutine has been "unrolled" 99 | 100 | ```lua 101 | local pong = function (thread) 102 | local nxt = nil 103 | nxt = function (cont, ...) 104 | if not cont 105 | then return ... 106 | else return nxt(co.resume(thread, ...)) 107 | end 108 | end 109 | return nxt(co.resume(thread)) 110 | end 111 | ``` 112 | 113 | --- 114 | 115 | if we give `pong` some coroutine, it will recursively run the coroutine until completion 116 | 117 | ```lua 118 | local thread = co.create(function () 119 | local x = co.yield(1) 120 | print(x) 121 | local y, z = co.yield(2, 3) 122 | print(y) 123 | end) 124 | 125 | pong(thread) 126 | ``` 127 | 128 | We can expect to see `1`, `2 3` printed. 129 | 130 | ## [Thunk](https://stackoverflow.com/questions/2641489/what-is-a-thunk) 131 | 132 | Once you understand how the synchronous `pong` works, we are super close! 133 | 134 | But before we make the asynchronous version, we need to learn one more simple concept. 135 | 136 | For our purposes a `Thunk` is function whose purpose is to invoke a callback. 137 | 138 | i.e. It adds a transformation of `(arg, callback) -> void` to `arg -> (callback -> void) -> void` 139 | 140 | ```lua 141 | local read_fs = function (file) 142 | local thunk = function (callback) 143 | fs.read(file, callback) 144 | end 145 | return thunk 146 | end 147 | ``` 148 | 149 | --- 150 | 151 | This too, is a process that can be automated: 152 | 153 | ```lua 154 | local wrap = function (func) 155 | local factory = function (...) 156 | local params = {...} 157 | local thunk = function (step) 158 | table.insert(params, step) 159 | return func(unpack(params)) 160 | end 161 | return thunk 162 | end 163 | return factory 164 | end 165 | 166 | local thunk = wrap(fs.read) 167 | ``` 168 | 169 | So why do we need this? 170 | 171 | ## Async Await 172 | 173 | The answer is simple! We will use thunks for our RHS! 174 | 175 | --- 176 | 177 | With that said, we will still need one more magic trick, and that is to make a `step` function. 178 | 179 | The sole job of the `step` funciton is to take the place of the callback to all the thunks. 180 | 181 | In essence, on every callback, we take 1 step forward in the coroutine. 182 | 183 | ```lua 184 | local pong = function (func, callback) 185 | assert(type(func) == "function", "type error :: expected func") 186 | local thread = co.create(func) 187 | local step = nil 188 | step = function (...) 189 | local stat, ret = co.resume(thread, ...) 190 | assert(stat, ret) 191 | if co.status(thread) == "dead" then 192 | (callback or function () end)(ret) 193 | else 194 | assert(type(ret) == "function", "type error :: expected func") 195 | ret(step) 196 | end 197 | end 198 | step() 199 | end 200 | ``` 201 | 202 | Notice that we also make pong call a callback once it is done. 203 | 204 | --- 205 | 206 | We can see it in action here: 207 | 208 | ```lua 209 | local echo = function (...) 210 | local args = {...} 211 | local thunk = function (step) 212 | step(unpack(args)) 213 | end 214 | return thunk 215 | end 216 | 217 | local thread = co.create(function () 218 | local x, y, z = co.yield(echo(1, 2, 3)) 219 | print(x, y, z) 220 | local k, f, c = co.yield(echo(4, 5, 6)) 221 | print(k, f, c) 222 | end) 223 | 224 | pong(thread) 225 | ``` 226 | 227 | We can expect this to print `1 2 3` and `4 5 6` 228 | 229 | Note, we are using a synchronous `echo` for illustration purposes. It doesn't matter when the `callback` is invoked. The whole mechanism is agnostic to timing. 230 | 231 | You can think of async as the more generalized version of sync. 232 | 233 | You can run an asynchronous version in the last section. 234 | 235 | ## Await All 236 | 237 | One more benefit of thunks, is that we can use them to inject arbitrary computation. 238 | 239 | Such as joining together many thunks. 240 | 241 | ```lua 242 | local join = function (thunks) 243 | local len = table.getn(thunks) 244 | local done = 0 245 | local acc = {} 246 | 247 | local thunk = function (step) 248 | if len == 0 then 249 | return step() 250 | end 251 | for i, tk in ipairs(thunks) do 252 | local callback = function (...) 253 | acc[i] = {...} 254 | done = done + 1 255 | if done == len then 256 | step(unpack(acc)) 257 | end 258 | end 259 | tk(callback) 260 | end 261 | end 262 | return thunk 263 | end 264 | ``` 265 | 266 | This way we can perform `await_all` on many thunks as if they are a single one. 267 | 268 | ## More Sugar 269 | 270 | All this explicit handling of coroutines are abit ugly. The good thing is that we can completely hide the implementation detail to the point where we don't even need to require the `coroutine` namespace! 271 | 272 | Simply wrap the coroutine interface with some friendly helpers 273 | 274 | ```lua 275 | local pong = function (func, callback) 276 | local thread = co.create(func) 277 | ... 278 | end 279 | 280 | local await = function (defer) 281 | return co.yield(defer) 282 | end 283 | 284 | local await_all = function (defer) 285 | return co.yield(join(defer)) 286 | end 287 | ``` 288 | 289 | ## Composable 290 | 291 | At this point we are almost there, just one more step! 292 | 293 | ```lua 294 | local sync = wrap(pong) 295 | ``` 296 | 297 | We `wrap` `pong` into a thunk factory, so that calling it is no different than yielding other thunks. This is how we can compose together our `async` `await`. 298 | 299 | It's thunks all the way down. 300 | 301 | ## Tips and Tricks 302 | 303 | In Neovim, we have something called `textlock`, which prevents many APIs from being called unless you are in the main event loop. 304 | 305 | This will prevent you from essentially modifying any Neovim states once you have invoked a `vim.loop` funciton, which run in a seperate loop. 306 | 307 | Here is how you break back to the main loop: 308 | 309 | ```lua 310 | local main_loop = function (f) 311 | vim.schedule(f) 312 | end 313 | ``` 314 | 315 | ```lua 316 | a.sync(function () 317 | -- do something in other loop 318 | a.wait(main_loop) 319 | -- you are back! 320 | end)() 321 | ``` 322 | 323 | ## Plugin! 324 | 325 | I have bundle up this tutorial as a vim plugin, you can install it the usual way. 326 | 327 | `Plug 'ms-jpq/lua-async-await', {'branch': 'neo'}` 328 | 329 | and then call the test functions like so: 330 | 331 | `:LuaAsyncExample` 332 | 333 | `:LuaSyncExample` 334 | 335 | `:LuaTextlockFail` 336 | 337 | `:LuaTextLockSucc` 338 | -------------------------------------------------------------------------------- /_config.yml: -------------------------------------------------------------------------------- 1 | --- 2 | title: lua Async Await 3 | 4 | showcase: True 5 | 6 | images: [] 7 | -------------------------------------------------------------------------------- /lua/async.lua: -------------------------------------------------------------------------------- 1 | --#################### ############ #################### 2 | --#################### Async Region #################### 3 | --#################### ############ #################### 4 | 5 | local co = coroutine 6 | local unpack = table.unpack or unpack 7 | 8 | -- use with wrap 9 | local pong = function (func, callback) 10 | assert(type(func) == "function", "type error :: expected func") 11 | local thread = co.create(func) 12 | local step = nil 13 | step = function (...) 14 | local pack = {co.resume(thread, ...)} 15 | local status = pack[1] 16 | local ret = pack[2] 17 | assert(status, ret) 18 | if co.status(thread) == "dead" then 19 | if (callback) then 20 | (function (_, ...) callback(...) end)(unpack(pack)) 21 | end 22 | else 23 | assert(type(ret) == "function", "type error :: expected func - coroutine yielded some value") 24 | ret(step) 25 | end 26 | end 27 | step() 28 | end 29 | 30 | 31 | -- use with pong, creates thunk factory 32 | local wrap = function (func) 33 | assert(type(func) == "function", "type error :: expected func") 34 | local factory = function (...) 35 | local params = {...} 36 | local thunk = function (step) 37 | table.insert(params, step) 38 | return func(unpack(params)) 39 | end 40 | return thunk 41 | end 42 | return factory 43 | end 44 | 45 | 46 | -- many thunks -> single thunk 47 | local join = function (thunks) 48 | local len = table.getn(thunks) 49 | local done = 0 50 | local acc = {} 51 | 52 | local thunk = function (step) 53 | if len == 0 then 54 | return step() 55 | end 56 | for i, tk in ipairs(thunks) do 57 | assert(type(tk) == "function", "thunk must be function") 58 | local callback = function (...) 59 | acc[i] = {...} 60 | done = done + 1 61 | if done == len then 62 | step(unpack(acc)) 63 | end 64 | end 65 | tk(callback) 66 | end 67 | end 68 | return thunk 69 | end 70 | 71 | 72 | -- sugar over coroutine 73 | local await = function (defer) 74 | assert(type(defer) == "function", "type error :: expected func") 75 | return co.yield(defer) 76 | end 77 | 78 | 79 | local await_all = function (defer) 80 | assert(type(defer) == "table", "type error :: expected table") 81 | return co.yield(join(defer)) 82 | end 83 | 84 | 85 | return { 86 | sync = wrap(pong), 87 | wait = await, 88 | wait_all = await_all, 89 | wrap = wrap, 90 | } 91 | -------------------------------------------------------------------------------- /lua/tutorial.lua: -------------------------------------------------------------------------------- 1 | local a = require "async" 2 | local co = coroutine 3 | local uv = vim.loop 4 | 5 | 6 | --#################### ########### #################### 7 | --#################### Sync Region #################### 8 | --#################### ########### #################### 9 | 10 | 11 | -- sync version of pong 12 | local pong = function (thread) 13 | local nxt = nil 14 | nxt = function (cont, ...) 15 | if not cont 16 | then return ... 17 | else return nxt(co.resume(thread, ...)) 18 | end 19 | end 20 | return nxt(co.resume(thread)) 21 | end 22 | 23 | 24 | local sync_example = function () 25 | 26 | local thread = co.create(function () 27 | local x = co.yield(1) 28 | print(x) 29 | local y, z = co.yield(2, 3) 30 | print(y, z) 31 | local f = co.yield(4) 32 | print(f) 33 | end) 34 | 35 | pong(thread) 36 | end 37 | 38 | 39 | --#################### ############ #################### 40 | --#################### Async Region #################### 41 | --#################### ############ #################### 42 | 43 | 44 | local timeout = function (ms, callback) 45 | local timer = uv.new_timer() 46 | uv.timer_start(timer, ms, 0, function () 47 | uv.timer_stop(timer) 48 | uv.close(timer) 49 | callback() 50 | end) 51 | end 52 | 53 | 54 | -- typical nodejs / luv function 55 | local echo_2 = function (msg1, msg2, callback) 56 | -- wait 200ms 57 | timeout(200, function () 58 | callback(msg1, msg2) 59 | end) 60 | end 61 | 62 | 63 | -- thunkify echo_2 64 | local e2 = a.wrap(echo_2) 65 | 66 | 67 | local async_tasks_1 = function() 68 | return a.sync(function () 69 | local x, y = a.wait(e2(1, 2)) 70 | print(x, y) 71 | return x + y 72 | end) 73 | end 74 | 75 | 76 | local async_tasks_2 = function (val) 77 | return a.sync(function () 78 | -- await all 79 | local w, z = a.wait_all{e2(val, val + 1), e2(val + 2, val + 3)} 80 | print(unpack(w)) 81 | print(unpack(z)) 82 | return function () 83 | return 4 84 | end 85 | end) 86 | end 87 | 88 | 89 | local async_example = function () 90 | return a.sync(function () 91 | -- composable, await other async thunks 92 | local u = a.wait(async_tasks_1()) 93 | local v = a.wait(async_tasks_2(3)) 94 | print(u + v()) 95 | end) 96 | end 97 | 98 | 99 | --#################### ############ #################### 100 | --#################### Loops Region #################### 101 | --#################### ############ #################### 102 | 103 | 104 | -- avoid textlock 105 | local main_loop = function (f) 106 | vim.schedule(f) 107 | end 108 | 109 | 110 | local vim_command = function () 111 | vim.api.nvim_command[[echom 'calling vim command']] 112 | end 113 | 114 | 115 | local textlock_fail = function() 116 | return a.sync(function () 117 | a.wait(e2(1, 2)) 118 | vim_command() 119 | end) 120 | end 121 | 122 | 123 | local textlock_succ = function () 124 | return a.sync(function () 125 | a.wait(e2(1, 2)) 126 | a.wait(main_loop) 127 | vim_command() 128 | end) 129 | end 130 | 131 | 132 | return { 133 | sync_example = sync_example, 134 | async_example = async_example, 135 | textlock_fail = textlock_fail, 136 | textlock_succ = textlock_succ, 137 | } 138 | -------------------------------------------------------------------------------- /plugin/neovim_async_tutorial.vim: -------------------------------------------------------------------------------- 1 | if exists('g:nvim_tut_loaded') 2 | finish 3 | endif 4 | 5 | command! LuaSyncExample lua require "tutorial".sync_example() 6 | command! LuaAsyncExample lua require "tutorial".async_example()() 7 | command! LuaTextlockFail lua require "tutorial".textlock_fail()() 8 | command! LuaTextLockSucc lua require "tutorial".textlock_succ()() 9 | 10 | let g:nvim_tut_loaded = 1 11 | 12 | --------------------------------------------------------------------------------