├── LICENSE ├── README.md ├── parse.lua ├── pico8-table-string.lua ├── print-table-string └── stringify.lua /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2019 Ben Wiley 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # pico8-table-string 2 | 3 | Store a nested Lua data table as one giant string. Functions run in PICO-8 and in standard Lua environments. Works for flat and nested tables. 4 | 5 | *NOTE!*: This can be helpful if you have a really large amount of data whose token count you want to cut down, but there are [other techniques you can try first](https://github.com/seleb/PICO-8-Token-Optimizations/blob/master/README.md) which may be more effective generally. 6 | 7 | ### *Before* 8 | 9 | ```lua 10 | my_table = { 11 | hello='world', 12 | nested={ 13 | some='data\'s nested', 14 | inside_of='here' 15 | }, 16 | 'and', 17 | 'indexed', 18 | 'data', 19 | { as='well' } 20 | } 21 | ``` 22 | 23 | ### *After* 24 | 25 | ```lua 26 | my_table= 27 | table_from_string( 28 | '1and2indexed3data4aswellhelloworldnestedinside_ofheresomedata\'s nested' 29 | ) 30 | ``` 31 | 32 | ## Why? 33 | 34 | There's only one good set of reasons (that I can think of) to store your data in this particular way: 35 | 36 | 1. You're building a [PICO-8](https://www.lexaloffle.com/pico-8.php) game 37 | 2. Your game is too big and your remaining token count is running low 38 | 3. You have a lot of string data taking up a lot of tokens 39 | 40 | If these are true (or even if not), you can take advantage of this to cut down on your number of tokens. Create your data in a separate Lua file, use this tool to turn it into a giant string, and deserialize your data back into a Lua table when your program boots up. 41 | 42 | Note that if the amount of data you're serializing is small, you'll actually add more tokens by including the library. The `table_from_string` function, which is required at runtime to rebuild the table, takes up 139 PICO-8 tokens. But now the actual declaration of the table takes only 5 tokens, so if you have a table taking up 145 or more tokens, this can give you some significant savings. 43 | 44 | ## Usage 45 | 46 | First, clone this repository: 47 | 48 | ```console 49 | git clone https://github.com/benwiley4000/pico8-table-string.git 50 | cd pico8-table-string/ 51 | ``` 52 | 53 | ### Serializing 54 | 55 | Assuming you have [Lua](https://www.lua.org/start.html) installed, the easiest way to generate a string version of your data is to paste your table into the [print-table-data](/print-table-data) file and run it from the command line: 56 | 57 | ```console 58 | $ ./print-table-data 59 | my_table= 60 | table_from_string( 61 | '1and2indexed3data4aswellhelloworldnestedinside_ofheresomedata\'s nested' 62 | ) 63 | ``` 64 | 65 | Alternatively, if you want to generate with PICO-8 directly, that's also possible. 66 | 67 | First, copy the contents from [stringify.lua](/stringify.lua) into your PICO-8 code editor (excluding the `return` statement at the bottom of the file). 68 | 69 | Then add this code: 70 | 71 | ```lua 72 | -- replace with whatever your table is called 73 | local table_str = serialize_table(my_table) 74 | 75 | printh('my_table=','outfile') 76 | printh('table_from_string(','outfile') 77 | printh(' '..table_str,'outfile') 78 | printh(')','outfile') 79 | ``` 80 | 81 | Then check the file `outfile.p8l`: 82 | 83 | ``` 84 | my_table= 85 | table_from_string( 86 | '1and2indexed3data4aswellhelloworldnestedinside_ofheresomedata\'s nested' 87 | ) 88 | ``` 89 | 90 | You can take this and include it in your code. 91 | 92 | ### Parsing at runtime 93 | 94 | Note that at runtime, the only function you need to include is `table_from_string` from [parse.lua](/parse.lua). 95 | 96 | ## Serialization format 97 | 98 | Instead of using something like JSON or a string embedding Lua's own table format (which could have been ideal if we had access to an eval function in PICO-8), we essentially just concatenate all the keys and values from a table into a giant string, with a few ASCII control characters as token delimiters and structure indicators (to allow table nesting). By assuming everything will be a string (except those table keys which are able to be parsed as numbers), we are able to keep the parsing routine relatively simple, which is important since the goal is to save on tokens. 99 | 100 | Beyond the question of parsing complexity, the format is also smaller than the alternatives. Compare this output, taking up 91 characters: 101 | 102 | ```lua 103 | '1and2indexed3data4aswellhelloworldnestedinside_ofheresomedata\'s nested' 104 | ``` 105 | 106 | with this JSON version which uses 126 characters (38% more!): 107 | 108 | ```lua 109 | '{"1":"and","2":"indexed","3":"data","4":{"as":"well"},"hello":"world","nested":{"some":"data\'s nested","inside_of":"here"}}' 110 | ``` 111 | 112 | It might be a moot comparison though, since the token cost of parsing JSON would be so great. Take this [pure-Lua JSON library](https://gist.github.com/tylerneylon/59f4bcf316be525b30ab), for example, which would take up nearly 750 PICO-8 tokens. 113 | 114 | ## Notes 115 | 116 | ### Types 117 | 118 | The main current caveat is all the contained values must strings, since it's simpler to serialize and parse without embedding type information, and I only needed this for strings. I can imagine a use case where this could be useful for numeric information (e.g. large 3d models), so I'd be open to a pull request to add that support, if the token count can stay pretty low. Alternatively, it could just be a separate function. 119 | 120 | ### Error handling (or not) 121 | 122 | In order to keep the token count low, there's no built-in error handling, so make sure any string you send to `table_from_string` was created with `serialize_table` or `stringify_table`. 123 | -------------------------------------------------------------------------------- /parse.lua: -------------------------------------------------------------------------------- 1 | -- these are globals in PICO-8 2 | -- (don't need these lines there) 3 | local sub = sub or string.sub 4 | local tonum = tonum or tonumber 5 | 6 | function table_from_string(str) 7 | local tab, is_key = {}, true 8 | local key,val,is_on_key 9 | local function reset() 10 | key,val,is_on_key = '','',true 11 | end 12 | reset() 13 | local i, len = 1, #str 14 | while i <= len do 15 | local char = sub(str, i, i) 16 | -- token separator 17 | if char == '\31' then 18 | if is_on_key then 19 | is_on_key = false 20 | else 21 | tab[tonum(key) or key] = val 22 | reset() 23 | end 24 | -- subtable start 25 | elseif char == '\29' then 26 | local j,c = i,'' 27 | -- checking for subtable end character 28 | while (c ~= '\30') do 29 | j = j + 1 30 | c = sub(str, j, j) 31 | end 32 | tab[tonum(key) or key] = table_from_string(sub(str,i+1,j-1)) 33 | reset() 34 | i = j 35 | else 36 | if is_on_key then 37 | key = key..char 38 | else 39 | val = val..char 40 | end 41 | end 42 | i = i + 1 43 | end 44 | return tab 45 | end 46 | 47 | -- module export - don't 48 | -- include this in PICO-8 49 | return { 50 | table_from_string=table_from_string 51 | } 52 | -------------------------------------------------------------------------------- /pico8-table-string.lua: -------------------------------------------------------------------------------- 1 | local stringify = require "stringify" 2 | local parse = require "parse" 3 | 4 | return { 5 | stringify_table=stringify_table, 6 | serialize_table=serialize_table, 7 | table_from_string=table_from_string 8 | } 9 | -------------------------------------------------------------------------------- /print-table-string: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env lua 2 | 3 | local stringify = require "stringify" 4 | 5 | -- replace this with your own table 6 | -- from your program, then paste the 7 | -- output back into your program 8 | 9 | local my_table = { 10 | hello='world', 11 | nested={ 12 | some='data\'s nested', 13 | inside_of='here' 14 | }, 15 | 'and', 16 | 'indexed', 17 | 'data', 18 | { as='well' } 19 | } 20 | 21 | local str = stringify.serialize_table(my_table) 22 | 23 | print('my_table=') 24 | print('table_from_string(') 25 | print(' '..str) 26 | print(')') 27 | 28 | -------------------------------------------------------------------------------- /stringify.lua: -------------------------------------------------------------------------------- 1 | -- these are globals in PICO-8 2 | -- (don't need these lines there) 3 | local sub = sub or string.sub 4 | local tonum = tonum or tonumber 5 | 6 | -- ASCII US "unit separator" 7 | -- used to delimit both keys and values 8 | local token_sep = '\31' 9 | 10 | -- ASCII GS "group separator" 11 | -- used to delimit the begining of a 12 | -- subtable following a key 13 | local subtable_start = '\29' 14 | 15 | -- ASCII RS "record separator" 16 | -- used to delimit the end of a subtable 17 | local subtable_end = '\30' 18 | 19 | function stringify_table(table) 20 | local str = '' 21 | for key, val in pairs(table) do 22 | str = str..key 23 | local t = type(val) 24 | if t == 'table' then 25 | str = str..subtable_start..stringify_table(val)..subtable_end 26 | else 27 | str = str..token_sep..val..token_sep 28 | end 29 | end 30 | return str 31 | end 32 | 33 | function serialize_table(table) 34 | local function escape(str) 35 | if type(str) ~= 'string' then 36 | return str 37 | end 38 | local new_str = '' 39 | for i = 1,#str do 40 | local char = sub(str, i, i) 41 | if char == '\'' then 42 | new_str = new_str..'\\\'' 43 | else 44 | new_str = new_str..char 45 | end 46 | end 47 | return new_str 48 | end 49 | return '\''..escape(stringify_table(table))..'\'' 50 | end 51 | 52 | -- module export - don't 53 | -- include this in PICO-8 54 | return { 55 | stringify_table=stringify_table, 56 | serialize_table=serialize_table 57 | } 58 | --------------------------------------------------------------------------------