├── README.md └── docs └── luf-vv.md /README.md: -------------------------------------------------------------------------------- 1 | # Lua Users Foundation 2 | 3 | This is the main repository where the Lua community can collaborate in order to organise the foundation. 4 | 5 | # What is the Lua Users Foundation 6 | 7 | The Lua Users Foundation is an association of individuals in a modular umbrella structure with the mission of supporting and promoting the Lua programming language, its community and ecosystems. Please take check the current draft of our Manifesto for more details, or even collaborate to its definition. You can see our work in progress in the [Pull Requests](https://github.com/lua-users-foundation/foundation/pulls) of this repository. 8 | 9 | # Who we are 10 | The list of people involved with the foundation can be seen [here](https://github.com/orgs/lua-users-foundation/people). This list is currently being updated and also in the meantime there is a [discussion](https://github.com/lua-users-foundation/foundation/issues/1) on how to define membership of the foundation. 11 | 12 | # Join us 13 | 14 | Join us and collaborate in this initiave. The work will be conducted in this repository and we are also chatting about it at Slack. Click here to join our [Slack Channel](https://join.slack.com/t/lua-users-foundation/shared_invite/enQtMzM2ODEwMDI4MjQ0LTJhMWY5ODU1ZTdkMmZkMTgxOWIwMDNkOTdhZWFmM2NiY2FiOWRhZDYxOTMxZGFiYWViOGIwZDgwZDZmYWZhYjM) 15 | -------------------------------------------------------------------------------- /docs/luf-vv.md: -------------------------------------------------------------------------------- 1 | # LUF-VV: Framework for Validation and Verification of Lua Implementations 2 | 3 | A couple of notes to the reader. This document is an early draft, so please consider following: 4 | 5 | * Predicated *must*, *should* and *may* are not strictly defined in the text, please interpret them in a typical RFC’ish sense. 6 | * Terms *implementation-defined behavior*, *unspecified behavior* and *undefined behavior* are also used throughout the text without exact definition, please interpret them close to the C Standard usage. 7 | * Search the document for **FIXME:** to find and fix gaps in the text. 8 | * Search the document for **NB!** to find emphasized paragraphs. 9 | 10 | # Introduction and Target Audience 11 | 12 | This document describes a framework and accompanying infrastructure for validation and verification of implementations of the Lua language (codename LUF-VV). The document per se outlines the requirements for building LUF-VV. The target audience of LUF-VV are: 13 | 14 | * Developers (both individuals and organizations) who build software using Lua and feel need in providing validation and verification of their underlying platform. 15 | * Developers (both individuals and organizations) who build implementations of the Lua language 16 | 17 | ## Rationale for Building LUF-VV 18 | 19 | **FIXME:** Elaborate on this. 20 | 21 | # LUF-VV Core Requirements 22 | 23 | ## Development Model 24 | 25 | LUF-VV is an open-source project licensed under MIT license. All project sources must be kept in a public repository with a publicly available issue tracker for submitting bugs and requests. Documentation for the project should be licensed under the CC BY-SA license. 26 | 27 | ## Supported Lua Versions 28 | 29 | LUF-VV must support following versions of the language: 30 | 31 | * 5.1 32 | * 5.2 33 | * 5.3 34 | 35 | **FIXME:** process to change this list (and other musts) 36 | 37 | **NB!** Although Lua 5.1 was released more than 10 years ago, it is still quite popular; besides, LuaJIT 2.0, widely used implementation of Lua, is primarily compliant with Lua 5.1. 38 | 39 | ## Supported Run Modes 40 | 41 | LUF-VV must support following modes: 42 | 43 | * CLI mode: Tested Lua implementation is built as a stand-alone binary which is used for running tests. 44 | * Library mode: Tested Lua implementation is built as a library which is embedded into tests for subsequent running. 45 | 46 | There is absolutely no requirement that each test in LUF-VV supports all modes listed above, but once tested Lua implementation formally specifies its build configuration via some protocol (**FIXME:** elaborate), all parts of the LUF-VV infrastructure should act accordingly building and using for testing appropriate binary artifacts provided by the tested implementation. 47 | 48 | ## Supported Hardware Platforms and Operating Systems 49 | 50 | The final goal for LUF-VV is to support not less hardware platforms and operating system than supported by the reference implementation, targets that must be supported from the start are: 51 | 52 | * Linux, x86-64 53 | 54 | # LUF-VV Features 55 | 56 | ## Annotated Reference Manual (ARefMan) 57 | 58 | Current official Reference Manual fully and adequately describes the language. However, for providing more help in building and ensuring compilance of custom Lua implementations, LUF-VV should provide annotated Reference Manual for each supported version of the language. The Annotated Reference Manual is the vanilla Reference Manual plus a set of annotations which correct and clarify the original text with a special emphasis on: 59 | 60 | * Typos 61 | * Semantic errors 62 | * Outlining implementation-defined / unspecified / undefined behavior, where the original text is not concrete enough 63 | 64 | ### ARefMan Maintenance 65 | 66 | * Each version of the ARefMan must be annotated with the respective version of the language and the timestamp of the last changes. 67 | * All maintained versions of the ARefMan must be regularly published automatically from their sources. 68 | * The published versions of the ARefMan must be accessible on-line at some convenient place. 69 | 70 | The ARefMan must be kept in sync with the original text. Thus there must be established a regular review procedure during which all changes from the original text are transferred to the ARefMan. This procedure must be conducted by the ARefMan maintainer on regular basis and should preferably be semiautomatic consisting of following steps: 71 | 72 | * Automatic syncing with the original text (think about merging the upstream into your local repository in terms of git) 73 | * Manual review of the result and committing the changes to the ARefMan repository 74 | 75 | ### Bugs in the ARefMan 76 | 77 | Bugs in the ARefMan should be submitted through the LUF-VV repository. All bug reports covering the original text are out of scope of ARefMan responsibility, but should be reported to the upstream by the ARefMan maintainer. 78 | 79 | ### Pushing Annotations from the ARefMan to the Original Text 80 | 81 | **FIXME:** This is a nice feature, but a process should be described here. 82 | 83 | ## Test Suites 84 | 85 | LUF-VV offers tests of several types: 86 | 87 | * Compliance testing 88 | * Testing using real-world applications 89 | 90 | Each type of testing is represented by its own independent suite which are described in details. 91 | 92 | **NB!** 1:1 mapping `type of testing` ⇔ `suite` is not a final decision and has to be thoroughly discussed. The main problem here is that the existing Lua ecosystem already provides some sources for testing, but these sources are highly heterogeneous: There is no unified runner, no unified output protocol etc. That said, there is a temptation to bundle several suites, equip them with a thin integration layer and ship together as a single “meta-site”. This approach allows to increase the number of tests rather quickly. On the other hand, though, this approach adds less value in long term: Developing a coherent suite from the start requires some additional efforts (plus, in future, there will be efforts required for adopting tests from the other suites), but gives an ability to build things consistently and provide better integration facilities. 93 | 94 | ### Compliance Suite 95 | 96 | The goal of this suite is to ensure compliance with the Reference Manual. 97 | 98 | **Status quo.** The suite shipped with the [lua-TestMore](https://fperrad.github.io/lua-TestMore/) module can be used as a basis for building the compliance suite for LUF-VV. However, following issues should be mitigated: 99 | 100 | * The code should be checked to clean out implementation-dependent tests. 101 | * Currently tests from the suite create temporary files in unportable manner, which makes it impossible to run the suite in parallel. 102 | 103 | On top of that, there are other suites available: 104 | 105 | * [The official test suite for PUC-Rio interpreter](https://www.lua.org/tests/) 106 | * [Mike Pall's suite for LuaJIT](https://github.com/LuaJIT/LuaJIT-test-cleanup) 107 | * [Test suite for CERN's MAD project](https://github.com/MethodicalAcceleratorDesign/MAD/tree/dev/tests), yet this one contains lots of code with non-standard syntax extensions 108 | 109 | These suites are built for concrete implementations, but their portions may be adopted in the future to become a part of the LUF-VV compliance suite. 110 | 111 | ## Suite of Real-World Applications 112 | 113 | **FIXME:** Elaborate on this section. 114 | 115 | The idea behind the suite is to assemble as many real-world applications / libraries as possible and run them with some data using the tested Lua implementation. 116 | 117 | **Status quo.** This is one of the most challenging parts of the project. Following issues should be addressed: 118 | 119 | * Establish a “proof of concept” set of applications and libraries. Initially it will most likely be some open source projects, but they must be equipped with adequate data sets to ensure that running this code gives any reasonable outcome. 120 | * We should take into account and mitigate any differences in the build chains of the applications. Looks like a Nix ecosystem is the solution here, but any research for alternatives is welcome, if deemed necessary. 121 | * Beyond coding, there should be established some process to constantly extend this suite. This process should include solving legal questions, even for open-source projects (to avoid GPL-poisoning, for example). 122 | * Something should be done about closed-source projects. There is obvious that some code will never be donated to an open project like the one described. And the truth is, the most part of this closed-source code would be important for testing. One of the ideas here would be to build a protocol / API so that a closed-source project could be plugged into a privately-hosted LUF-VV infrastructure together with a tested Lua implementation. After that, the whole machinery could be built and run, and the result reported to all relevant parties without disclosing any sensitive information. 123 | 124 | ### Managing and Extending Suites 125 | 126 | Following process should be adopted: 127 | 128 | * All code related to test suites must be a part of the LUF-VV source tree. However, closed-source application (see above) used for real-world testing is an exception and might be treated separately. 129 | * If any suite is extended with a 3rd-party code, there should be no legal issues preventing from including the code into the source tree. 130 | * All newly added code must undergo adoption procedure: 131 | * It must be reviewed 132 | * For compliance tests, non-standard / non-portable / implementation-dependent behavior must be cleaned. **NB!** It is not clear how we should treat real-world code that relies on custom extensions. Should this code be rejected? Should we shape test data in such a manner that will allow us to actually skip executing non-standard code? 133 | 134 | ### Reporting Bugs in the Suites 135 | 136 | Bugs are reported via LUF-VV repository. 137 | 138 | # Exploitation 139 | 140 | **FIXME:** Elaborate on this section. 141 | 142 | LUF-VV must be usable in both public and private infrastructure. Exploitation in public infrastructure means that there must be an option to book computational resources in the infrastructure operated by LUF-VV and run all available tests there. Exploitation in private infrastructure means that there must be an option to deploy LUF-VV sources in some internal environment and all tests inside it. Regardless of the infrastructure type, prior to running, following configuration information must be submitted to LUF-VV in a machine-readable way (**FIXME:** Protocol to be developed / borrowed from existing CI solutions): 143 | 144 | * How to build the tested implementation 145 | * Location of deliverables (stand-alone executables, libraries for embedding, etc.) 146 | * Tested hardware platforms and OSes 147 | * Tested versions of the language 148 | 149 | There must be following features available (heavily inspired by Perl’s [prove](https://perldoc.perl.org/prove.html) utility and [TAP](https://en.wikipedia.org/wiki/Test_Anything_Protocol) protocol): 150 | 151 | * Reproducible runs 152 | * Re-running of failed tests only 153 | * Running certain tests inside suites 154 | * (nice to have) Shuffled run order 155 | * Parallel test run 156 | --------------------------------------------------------------------------------