125 |
--------------------------------------------------------------------------------
/open-software/lesson3-licensing.md:
--------------------------------------------------------------------------------
1 | # Lesson 3: Licensing/Ownership & DOIs
2 |
3 | Learning objective:
4 |
5 | - Understanding ethical and legal aspect of giving credits and attributions
6 | - Learning about existing open source licenses and Digital Object Identifier
7 |
8 |
9 | Disclaimer: the contents of this lesson are for educational purposes only. They do not constitute legal advice and should not be used as such.
10 |
11 |
12 |
13 | ## Introduction
14 |
15 | After deepening your understanding of the reasons to use open-software in the context of open-science, we here address the first considerations when using an open software tool in your research. First and foremost, if you are going to be building your own code on prior work you need to choose a software that is **open**, *i.e.,* that you are allowed to use, modify and redistribute. As a developer, you also need to ensure that you are sharing a product that is open - and thus, usable - to others. This is presented on the section [Licenses](link).
16 |
17 | Then, we present how you get credit for your work, and how you give credit to others' work. This is the content of the section *Attribution and citation*.
18 |
19 | At the end of this lesson, you will be able to: *Choose and abide by appropriate usage and referencing standards of open-software*.
20 |
21 | ## Licenses
22 |
23 | A software license is a legal document that grants users particular rights to the use of a certain software. This license can take many forms, but in many cases they outline contractual obligations (if any exist) between the company/software developer behind the software and the end user, what the user can do with the software, who the user can distribute the software to (if any such distribution rights exist) and the length of time the user has the right to use the software.
24 |
25 | A user cannot (technically and ethically), use a software without a license! A user can reach out to the developer/owner to ask for permission, and go ahead *if* the owner/developer furnishes written permission. But, if you share your software without a license, no one can use it without your written permission!
26 |
27 | ### Types of licenses
28 |
29 | A license can fall under several categories. License types have general definitions of what can be done with the software. By picking a type of license, or by understanding what type the license of a software you're considering using is, you'll be able to navigate the license process more quickly than reading each license individually and interpreting the permissions. An overview of types of licenses is given in the table below[^licensetypes]:
30 |
31 | | Public domain license | Lesser general domain license | Permissive | Copyleft | Proprietary |
32 | | -------- | -------- | --- | --- | -------- |
33 | |Anyone can use or modify the software.|Can link to open source libraries and code can be licensed under any license type.|Has some requirements for distribution and modification. |Licensed code can be distributed or modified if all the code involved is licensed under the same license.| Software cannot be copied, modified or distributed |
34 |
35 | Summary of selected attributes of licenses types [^Morin]:
36 | 
37 |
38 | Some of the common licenses used in open software are:
39 |
40 | 1. MIT license
41 | 2. Apache License 2.0
42 | 3. Mozilla Public License 2.0
43 | 4. BSD 3-Clause "New" license
44 | 5. GNU General Public License (GPL)
45 | 6. Common Development and Distribution License
46 |
47 | For more information on different types of licenses please refer to the ([Open Source Initiative OSI](https://opensource.org/licenses/category)).
48 |
49 | ### How to choose a license
50 |
51 | There are a number of steps that have to be made before choosing a particular license. Arguably one of the first decisions to be made is based upon whether you intend to use the code for commercial purposes or not, or at least foresee it as a possibility in the future. Some licenses are more favorable for commercial purposes than others, such as the *General Public License, version 2*.
52 |
53 | The next decision that has to be made is relating to the issue of distribution. When using other software as a dependency, you should always be wary of their licenses. Some licenses enforce certain types of licenses upon redistribution. The GNU GPL, for instance, is incompatible with proprietary licenses, because it requires the combined work to be licensed under the GPL, with no additional restrictions allowed. Having a part of the work under a proprietary license is such an additional restriction, so you cannot distribute such a combination (unless the copyright owner of the GPL code gives special permission).
54 | [^turing]
55 |
56 | For licensing open software, it is always good practice to consult with the [Open Source Initiative (OSI)](https://opensource.org/) website. They provide a list of approved licenses that guides you through this process. Remember that a first step is always to consult with your institution (if applicable). You should ensure that you are complying with any applicable local laws and any policies set by your employer and/or funding entity.
57 |
58 | 
59 |
60 | ### Additional Resources
61 |
62 | - [Choosing a License](https://choosealicense.com)
63 | - [Turing Way on licensing](https://the-turing-way.netlify.app/reproducible-research/licensing.html)
64 |
65 | ## Attribution and citation [^Katz][^Smith][^Chue]
66 |
67 | Both when choosing a license and publishing your software for future citation, a decision has to be made in relation to the issue of *attribution*, *i.e.*, crediting a person or group of people or other entity with a particular action in relation to the software. This can be thought of as the software/code equivalent to authorship on an academic paper. It is important to consider this to avoid accusations of plagiarism or copyright infringement. There is a short discussion in the final lesson in this module regarding ethical considerations on how contributions can be considered for authorship/attribution/ownership.
68 |
69 | When deciding to cite a software or code that was used in your research you can start with the question: is this research software? Research software includes source code files, algorithms, scripts, computational workflows, and executables that were created during the research process or for a research purpose. Software components (e.g., operating systems, libraries, dependencies, packages, scripts, etc.) that are used for research but were not created during or with a clear research intent should be not be cited (e.g. Microsoft Word, Linux, Python --the language itself; specific packages might be citable in this context). This differentiation may vary between disciplines. Some examples of research software that would be cited are [E3SM](https://e3sm.org/), [SciML](https://sciml.ai/), and [Stock Synthesis](https://github.com/nmfs-stock-synthesis/stock-synthesis).
70 |
71 | The majority of open source software licenses require some degree of attribution, and a small minority (such as the 0BSD) do not. The license will also dictate where the attribution must be displayed - some licenses will require the user to include attribution in a dedicated file such as the software license agreement.
72 |
73 | ### Digital Object Identifier (DOI)
74 |
75 | By having a persistent identifier, a software version can be cited. A digital object identifier (DOI) is a persistent identifier or handle used to uniquely identify various objects. It is provided and standardized by the International Organization for Standardization ([ISO](https://www.iso.org/home.html)). In contrast to dynamic web addresses such as URLs, DOIs are static, *i.e.*, do not change over the life of a document, and point to the location of the document on the internet. You can get a DOI for your own software/code by adding it to a preservation repository.
76 |
77 | Just as we publish scientific findings in writing in journals to ensure its preservation over time, its supplementary material, *e.g.*, source code and produced data, should also be stored in a permanent location. We call these preservation repositories. Some of these repositories are general-purpose such are [Zenodo](https://zenodo.org/) and [Figshare](https://figshare.com/), and some are more research field-oriented such as [Hydroshare](https://www.hydroshare.org/).
78 |
79 | It is important to keep in mind that a DOI refers to a static version of your code and so, you'll need to get a new DOI for each version you release and want cited. By using the same repository each time you need a DOI for a new version, you can be sure that when a user looks for your DOI they are directed towards the most recent version.
80 |
81 | ### Citing code without a DOI
82 |
83 | As a user, if you'd like to cite a software that does not have a DOI you can use [Software Heritage](https://www.softwareheritage.org/) to create a SWH-ID which is also a citable persistent identifier, but can be created for codes that are not your own. This should only be done *after* ensuring a DOI is not available otherwise there can be multiple identifiers being used for the same piece of software.
84 |
85 | ### Attribution for pieces/snippets of code
86 |
87 | While DOI and SWH-ID allow citations of full pieces of software/code, there is also the case where a small code snippet or section might be copied into another code. It is common practice to take a few lines of code to solve a piece of a problem from websites such as [StackExchange](https://stackexchange.com/) or [Code Ranch](https://coderanch.com/), but there should still be an attribution if no changes are being made. This can be done effectively with a comment that includes a link to the webpage from which the code was taken (most sites have an option to create a shortened shareable link that is more code friendly).
88 |
89 | ### Publishing open software in peer-reviewed journals
90 |
91 | It is also possible to publish open software or a research article detailing the inner workings of that software in peer review journals. A general example is the [Journal of Open Source Software](https://joss.theoj.org/); there are also more discipline specific journals such as [Astronomy and Computing](https://www.journals.elsevier.com/astronomy-and-computing) and [Environmental Modeling & Software](https://www.sciencedirect.com/journal/environmental-modelling-and-software). The peer review process for some of these journals may include a review of the code itself, some may be focused just on the describing journal article that accompanies it. These publications will also come with a permanent identifier as is customary with most journal articles.
92 |
93 | ## External Requirements
94 |
95 | There are various legal considerations to keep in mind with regard to software and code you write. For example, these may be considered intellectual property, and you may wonder who has ownership over it. Generally speaking much of this is dependent on your employment and funding situation at the time you did the work. Your institution may have claim to part or all of the work product, however it is highly variable, and your institutional offices should be contacted to understand this better.
96 |
97 | There may also be other institutional, governmental, or other legal policies that may be dependent on your region. Please make sure you understand your locality's laws and regulations regarding the sharing of research software and follow your institution and funding agency's requirements (if any) on licensing and intellectual property.
98 |
99 | ### Additional Resources
100 |
101 | *[Code publication](https://scicodes.net)
102 | *[Computational Infrastructure for Geodynamics - example of a preservation repository that provides peer review](https://geodynamics.org)
103 | *[When to cite software](https://f1000research.com/articles/9-1257/v2)
104 | *[CiteAs: a resource for finding the correct attribution of a research product](http://citeas.org)
105 |
106 | ## Summary
107 |
108 | Here, we reviewed that a software license is a legal binding document, made between the developer and the user of a software, which outlines how that software can be used and distributed. Open software will carry licenses that follow the [Open Software Initiative (OSI)](https://opensource.org/licenses) definitions: allowing the software to
109 | "to be freely used, modified, and shared." [(Open Software Initiative OSI)](https://opensource.org/licenses)
110 |
111 | We have presented the major categories of licenses that might fall under the *open-source* definition, and what considerations to take when choosing a specific software and/or license for your software, *i.e.* 1) what is the intended use of this software?, 2) how others can reuse it? And what are the policies of my institution and local laws regarding open-software use and dissemination?
112 |
113 | We have also learned about proper attribution - how to get credit for your work (DOI, archival of code, publishing options), and how to cite others work.
114 |
115 |
116 |
117 | [^Morin]:Morin, A., Urban, J., & Sliz, P. (2012). A quick guide to software licensing for the scientist-programmer. PLoS Computational Biology, 8(7).
118 |
119 | [^Smith]:Smith et al. (2016). Software Citation Principles. PeerJ Comput. Sci., DOI 10.7717/peerj-cs.86
120 |
121 | [^Chue]:Chue Hong, Neil P., Katz, Daniel S., Barker, Michelle, Lamprecht, Anna-Lena, Martinez, Carlos, Psomopoulos, Fotis E., Harrow, Jen, Castro, Leyla Jael, Gruenpeter, Morane, Martinez, Paula Andrea, Honeyman, Tom, Struck, Alessandra, Lee, Allen, Loewe, Axel, van Werkhoven, Ben, Jones, Catherine, Garijo, Daniel, Plomp, Esther, Genova, Francoise, … RDA FAIR4RS WG. (2022). FAIR Principles for Research Software (FAIR4RS Principles) (1.0).
122 |
123 | [^licensetypes]:[licensetypes](https://www.techtarget.com/searchcio/definition/software-license#:~:text=A%20software%20license%20is%20a,the%20software%20without%20violating%20copyrights.)
124 |
125 | [^turing]: [Turing Way Compatibility](https://the-turing-way.netlify.app/reproducible-research/licensing/licensing-compatibility.html)
126 |
--------------------------------------------------------------------------------
/open-software/lesson4-code-management.md:
--------------------------------------------------------------------------------
1 | # Lesson 4: Code management/Quality
2 |
3 | Learning objective:
4 |
5 | - Understanding some best practices for publishing Open Software
6 | - Learning the basics of code management
7 |
8 | ## Introduction
9 |
10 | While we maintain that sharing software at all is a great initial first step regardless of it's state, the more the code is kept clean, maintained, and documented, the more others will be able to cite, use, and contribute to it.
11 |
12 | ## What does it mean for software/code to be of good quality?
13 |
14 | There are two perspectives that you can take when engaging with this lesson: a user of open software, or a developer/provider of open software. As a user, you will want to make sure that a code or software project you are considering using in your research/project is quality. As a developer/provider, you will want to make sure your project is of high enough quality that others will want to use and engage with it. When we say "quality" code, we are referring to precisely that, a software/code that a user can be confident in using.
15 |
16 | Here we outline some baseline expectations for open software. While there are definitely good open software projects out there that do not include all of these items (and, unfortunately, plenty of projects out there that contain many of these items but still don't function well), this guide will assist in ensuring the software/code that you develop/use is quality.
17 |
18 | ### Good documentation
19 |
20 | Good documentation for code is possibly the most important item on this list for creating a quality code. This will help a user know what the software does and how it can be used, but also can be a real time saver for a developer when going back to look at code they haven't looked at in a while.
21 |
22 | #### The README file
23 |
24 | The first stop for a user when they approach a new project should be the README file. Aptly named, this file should contain orientation information that will help a user understand the project's purpose as well as shows examples of how it can be used, and lists most other important information the creator deems necessary. Note that there is no one agreed upon convention for the location of these documentation pieces, so we encourage exploration of the software you're interested in. Some information we describe as in a README file may be moved into its own file in some conventions, e.g. having installation instructions in an INSTALL file, but the README is still usually the best place to start. Keeping that in mind, if you are developing a code/software for use by others, they will expect a descriptive and useful README, without one using your code may be a nonstarter for many.
25 |
26 | [Here is an example of a README file](https://github.com/MillionConcepts/lhorizon) from a NASA-funded project that shows many of the specifics we are going to discuss below including multiple installation options. As you read the suggested parts of documentation below feel free to reference this for an example.
27 |
28 | Let's dive into the specifics of information you should include/find in a README file. First, a description of what the software does: it's purpose, the problem it's solving. You don't need to write a whole academic paper here, a sentence or two is fine. If you do happen to have a research paper written on the topic no one would be upset if you link it here, though do be careful that any linked papers are either (a) not behind a paywall or (b) if it _is_ behind a paywall, that the important information a user would need to use and understand your software is reiterated separately within the code documentation.
29 |
30 | A compatibility description is also necessary. Sometimes this is wrapped into the installation instructions and that is acceptable. Here the operating systems (e.g. Linux, Windows, macOS -- and their versions) that the software/code works on with are listed. If the code runs in a browser which does it work with? There are many tools for testing the compatibility of code across operating systems and environments, we won't get into those here as they can be specific to the coding language you're working in.
31 |
32 | If installation instructions are not in their own file, they'll live inside the README. These should be written with very little prior knowledge expected of the user. Most people are used to downloading a software package, double-clicking on the executable, and having a setup wizard walk them through any required steps. Setups such as this are achieved through packaging. Packaging bundles all the necessary pieces for a software to run, usually including dependencies, and distributes it to the user as one "package". Packaging software can make installation a lot simpler for users and allow it to be installed consistently that aids in reproducibility. Most open software won't be packaged to the double-click-with-setup-wizard level and some won't be packaged at all. They will require a bit more up front work for the user, but an advanced knowledge of installation practices shouldn't be assumed. For example, an exact command that can be copied and pasted into the command line is a lot more helpful than something like "clone the repo" or "install using git pip".
33 |
34 | Usage examples are another important part of a README document. While how to run and use the software may be obvious to the developer, many times this is not the case for the user. Simple/small usage examples are great for the README file. If there are more complex examples that require input files or that are interactive for the user and the programming language you are using supports interactive environments, such as [Jupyter](https://jupyter.org/) (for R, Python, and Julia), [Pluto](https://github.com/fonsp/Pluto.jl) (for Julia), [Quarto](https://quarto.org/) (for R, Python, and Julia), and [RStudio](https://www.rstudio.com/) (for R), these can be used and included in a repository and pointed to in the README. If interactive environments are not an option for the language you are using and your usage examples are necessarily complex, consider writing a standalone script and including a pointer to this with instructions on how to use and run that example script in the README.
35 |
36 | If relevant, the README is also one of the places you may find descriptions of the outputs of a software/code. Both what kind of objects these may be in terms of their type (e.g. string, integer, etc.) and in their general description (e.g. a list of names, the amount of rain the model calculated, etc.).
37 |
38 | As the README is the first place a user will look, this is also where you can find other notes and caveats of using the software. This should include at least something on the state of the software: is it in active development (meaning it may have some bugs and may not always work as expected), consistently maintained (meaning the software is updated when necessary--like when a dependency is updated or a bug is reported), or here for posterity purposes only (meaning the author/developer/researcher will not be working to maintain or improve this code any further)? How can you contact the developer/researcher that created this software/code? How can issue/bugs be reported (if at all)? This would also be a good place to list any known bugs/issues, so you get repeat requests.
39 |
40 | The README is also a great place to acknowledge team members that worked on the code/project as well as agencies and grant numbers that funded the work.
41 |
42 | #### Dependencies
43 |
44 | The dependencies -- the other software on which the software/code relies -- should be listed somewhere in the documentation, but are not always in the same place depending on the coding language. For example, in Python software, it is common to include a file titled something like `environment.yml` which will list dependencies and which can be used to install them quickly and easily. Other conventions may include listing them in the README file, a README can also be used to point to an additional file that lists dependencies (such as the `environment.yml` or `requirements.txt`)
45 |
46 | #### License
47 |
48 | A license file should be included with your documentation. This is expanded upon more in another lesson in this module, but without one, the code/software is technically and ethically not allowed to be used at all by anyone other than the author/developer.
49 |
50 | #### The `CONTRIBUTING.md` file
51 |
52 | One of the great benefits of open software is that it enables contributions from the community. The `CONTRIBUTING.md` and CODE_OF_CONDUCT files in software can be referenced for information on how to do this. This is expanded upon more in a later lesson.
53 |
54 | #### Documentation Checklist
55 |
56 | - [ ] Description of the software and the problem it solves
57 |
58 | - [ ] Compatibility description
59 |
60 | - [ ] Dependencies
61 |
62 | - [ ] Installation instructions
63 |
64 | - [ ] Usage examples (perhaps including an interactive notebook)
65 |
66 | - [ ] Development status of the software (under development, actively maintained, etc.)
67 |
68 | - [ ] Contact information
69 |
70 | - [ ] How to report issues/bugs (and a list of any known issues/limitations)
71 |
72 | - [ ] Acknowledgments of team and funding
73 |
74 | - [ ] License
75 |
76 | - [ ] Contribution guidelines
77 |
78 | - [ ] Code of conduct
79 |
80 | Additionally, a GitHub template from NOAA for open software documentation can be found [here](https://github.com/NOAA-OWP/owp-open-source-project-template).
81 |
82 | ### Clean/readable code
83 |
84 | Code for software is very rarely written only for one individual. Code typically has to be read and evaluated by others. In private companies, this is usually because software is written by a group of programmers and so it is important that programmers are able to read and understand the code, both in order to improve it and to "debug" or fix it. Open software also operates similarly: there may be many programmers working and contributing to a particular project from different backgrounds and walks of life. With different programmers with different backgrounds collaborating together, it's important that code is transparent and can be easily understood by others. This is sometimes referred to as "clean code".
85 |
86 | Clean code is code that is easily understood by others. Clean code has a number of advantages. One advantage is that it is easier to spot if or whether something is wrong with the code (known as "debugging"). Another advantage is that code that is "clean" is more likely to be shared than code that is not. This is fundamental to open software, which aims to be reproduced as widely as possible. There are a number of principles that should be adhered to when using clean code.
87 |
88 | #### Code Comments
89 |
90 | Arguably one of the most important is that code should be commented. Comments are annotations that help other programmers reading to understand what is going on. In many languages, they are designated by the sign `//` or `#` or `/* */`. As a rule, more comments are better than less but this should be prefaced with the warning that comments should not explain the obvious. For example, in the language JavaScript, the following would be an inappropriate comment
91 |
92 | ```var a = 5; //I'm assigning the value of 5 to the variable a.```
93 |
94 | It is inappropriate because the code is self-explanatory.
95 |
96 | #### Descriptive naming
97 |
98 | Another point to bear in mind when it comes to clean code is that variables, functions, and similar entities should be given descriptive names as opposed to vague names. These are names that, when another programmer reads them, instantly gives an idea of what the variable or function is. For example, the variable name `colourOfCat` is a good name because it describes what it intends to do, which is to encompass the color of a cat. As a rule, the more descriptive a name for a variable, function, etc., is the better. Names for variables, functions, etc. should avoid using words that are likely to be keywords - names with reserved meanings in many languages - such as "while", "for", "override" and so on. Needless to say, names for variables, functions, etc. should similarly avoid giving offense and clean code should consider the sensitivities of those from different backgrounds.
99 |
100 | It's frequently the case that code may point to external files; where possible, a programmer should ensure that the external file has a descriptive filename. In addition, clean code should also conform to programming conventions. For example, it's common in many programming languages to use camel case to describe variables, such as `colourForCat` rather than `COLOURFORCAT`, but one would do well to ascertain what a convention may be for a particular language.
101 |
102 | #### Whitespace and indentation
103 |
104 | Lastly, clean code should contain sufficient spaces between lines of code (also known as whitespace) and sufficient indentation so that they are easily discernible. Sometimes code that does not contain sufficient lines of code can go through a process known as _beautification_ or _prettifying_ that helps them become more readable. Ultimately, a key test for whether code can be considered "clean" is the following: if you left the code and came back to it 2 years from now, would you be able to easily understand it?
105 |
106 | ## Summary
107 |
108 | In this lesson we go over two main topics regarding markers of quality code: (1) good, descriptive documentation and (2) clean, readable code. As a user, documentation can be the difference between spending hours or days trying to understand a code and being able to use it right out of the box. As a developer/researcher, documentation improves the reproducibility and reusability of your code and lets others know what to expect both of your code and of you yourself as a maintainer. Next, we'll discuss maintaining quality code.
109 |
110 | ## References
111 |
112 | - Lee BD (2018) Ten simple rules for documenting scientific software. PLoS Comput Biol 14(12): e1006561.
113 | - Anzt H, Bach F, Druskat S et al. An environment for sustainable research software in Germany and beyond: current state, open challenges, and call for action [version 2; peer review: 2 approved] F1000Research 2021, 9:295
114 | - Martin, R. C. (2008). Clean code: A handbook of agile software craftsmanship. Prentice Hall.
115 |
--------------------------------------------------------------------------------
/open-software/lesson5-vesion-control.md:
--------------------------------------------------------------------------------
1 | # Lesson 5: Maintain good code quality
2 |
3 | Learning objective:
4 |
5 | - Understanding basics of Version Control
6 | - Learning the basics of testing in code development
7 | - Understanding the responsibilities of Open Software developers
8 |
9 | ## Introduction
10 |
11 | We've talked about markers of quality software in the prior lesson: good documentation and clean, readable code. The reality is that for most software, this is a journey, and it is going to continue to change and develop over some period of time. Here, we discuss version control, testing, and responsibilities after sharing. These topics are centered around the evolution of your code and ensuring the work you've done to make quality open software is able to endure.
12 |
13 | ## Version control
14 |
15 | Open source codes can change overtime. This brings several challenges to researchers developing and using an ever-changing software. We covered the importance of reproducibility for open-software - and open-science as a whole. Now, how can we achieve reproducibility with a changing code source? That is done by keeping track of changes to our source code, using version control.
16 |
17 | Version control can be done with tools and systems designed to manage changes not only to source code, but also to documents, websites, and datasets. [Google Docs](docs.google.com), for instance, has its own complex version control. This allows you and your collaborators to have access not only to the most updated google document you all are working on, but to the complete history of changes. So, if something goes wrong in a document: a child includes a thousand smiley faces in the text, a cat walks on the keyboard and deletes an entire section - you can just revert to the earlier, error-free version.
18 |
19 | This is the same for coding. For instance, you - the developer - receive a notification from a user that your code has a bug. You know that this bug was not present in the last version, so you can easily work through your history to look what recent changes might have caused a specific error, narrowing down your debugging work to specific parts of the code. So, version control allows a group of developers/users to know exactly what version of the code they are using, what changes were made and when - facilitating reproducibility. Version control also fosters collaboration, making it easier for people to work together at the same time and to merge changes from different users.
20 |
21 | There are several version control systems (VCS) available. We won't get into detail here, but some of the most popular open-source systems include [git](link), [SVN](link), and [Mercurial](link). It is important to note that while some repositories have already a built-in version control, repositories and version control systems are different - *e.g.*, *git* is the *version control system*, while [Github](https://github.com) is a *hosting service* for **git** repositories.
22 |
23 | In [lesson 6](link), we revisit version control, giving some concrete examples of how you can use it to contribute for new or existing open-source code.
24 |
25 | ## Testing
26 |
27 | In [Lesson 1](link), we introduced the concept of code testing and its importance in software development. There are many types of testing that range from testing the smallest testable parts of a code to verifying if a code works as whole under different scenarios. Since code testing in general can be a complicated and technically involved topic, we will not go into the details of each types of testing and refer you to external sources for further reading. Instead, we focus on benefits and difficulties of testing in general, how to measure test coverage, and what to expect from a "tested" code as an end-user.
28 |
29 | We recall that reproducibility in research software plays a critical role. In the context of testing, we can think of reproducibility as a test objective of which is to reproduce a specific output, *i.e.*, results obtained from a specific version of the code that has been published in a journal. This test should include all the required inputs (configuration files, input data, etc.) so users can easily run and get the same published results.
30 |
31 | More broadly, the main objective of code testing is to evaluate if a code is doing what it is supposed to do. It is important to recognize that testing a code comprehensively can be very difficult since not only we should test the code for generating expected outputs but also for failing when it should. For example, when an unacceptable input is passed, *e.g.*, wrong type, out of range, edge cases, etc., or when if implemented the algorithm doesn't converge for the given set of inputs. Taking into account all these scenarios can be extremely difficult and in some cases impossible. Therefore, we should manage our expectations when taking the tests as a measure of code's quality both as a developer (*e.g.*, realizing that the end-users might apply the code to scenarios that we don't anticipate) and an end-user (*e.g.*, realizing that the difficulties associated with testing and, if possible, evaluate the accuracy of outputs independently).
32 |
33 | From a developer perspective, there are also secondary benefits for testing. Whenever you make a change to a part of your code, for example to improve its performance, having tests for that portion of the code, ensures that the modified code does not change the output. Another scenario could be related to dependencies. For example, research software often depends on other software, therefore, if those dependencies release new versions, the tests help us evaluate if those new versions make any changes to outputs of our code.
34 |
35 | On the other hand, as an end-user, using a code that includes tests, gives us more confidence in the state of the code. Users can check the status of tests (pass/fail) when the developers make changes, or the code has been tested for the use-case of our interest.
36 |
37 | Now that we have a better understanding of the testing, we can discuss measuring its effectiveness. One of the ways that we can measure the testing is through percentage coverage. There are two levels of coverage: *test coverage* and *code coverage*. *Test coverage* refers to the coverage of different scenarios that the code would be used in while *code coverage* is the percentage of lines of code that tests cover. As we discussed previously, enumerating all the different scenarios the code could be used in can be very difficult, thus, it can be difficult to quantify *test coverage* both from a developer and end-user perspective. However, *code coverage* is just a simple percentage value: how many lines of code do the tests activate vs. not. It is important to note that a high *code coverage* does not necessarily mean that a code has good *test coverage* since testing different usage scenarios can not directly be translated to lines of code.
38 |
39 | ### Additional Resource
40 |
41 | - [IBM on Testing](https://www.ibm.com/topics/software-testing)
42 | - [Software Testing](https://www.softwaretestinghelp.com/types-of-software-testing/)
43 | - Martin, R. C. (2008). Clean code: A handbook of agile software craftsmanship. Prentice Hall.
44 |
45 | ## Responsibilities after Sharing
46 |
47 | After sharing software, there are certain steps that need to be taken in regard to maintenance of that code/software.
48 |
49 | First, you should know it is not a requirement for you to be a permanent maintainer forever, but it is your responsibility to let users know if you do or don't intend to maintain the software/code. You can do this in your documentation where you discuss the development status of the project. This helps a user know if it will continue to be supported in the future, and make choices about if they should base ongoing work off your project. You don't want someone to spend a huge amount of time using your work as a dependency and then have their project become unusable in the future.
50 |
51 | The reality is that a developer/researcher may not have the time or continued funding to keep up with a project. In this case, perhaps consider handing ownership of the software to another researcher/developer, involved user, or entity invested in its continued use. You can either approach potential parties you think may be interested in this; or you can make your license permissive enough to allow others to create their own copies and continue your work (see more on choosing a license in this module). Depending on the license you choose, the use of your project, and if you have significant interest, you may be able to commercialize your software/code to provide funding for continued maintenance and feature requests. There is also the potential to apply for continued funding from agencies both governmental and private if your open software is widely used. If you're a user of a software that is no longer maintained, consider contacting the owner/developer and volunteering either as a maintainer, or to take over ownership of the project (you'll be more likely to get a positive response if you leave that choice up to the current owner).
52 |
53 | If you receive requests for features and fixes, and you have indicated you intend to maintain the code, these should be responded to. Either tell the users that (a) you intend to perform their requested action or (b) you think that's out of scope of your project. Additionally, you can invite the requester to (a) contribute to the project and add that feature/fix themselves (which you can then approve and add into your project) or (b) fork (make a copy of) the project and create the feature/fix, notifying that you will not merge changes into your (main/original) copy.
54 |
55 | ## Summary
56 |
57 | Here we discuss how version control and testing can both be used to increase the reproducibility and trust a user can place in open software. These are tools that can be used whether your software is shared or not. We go over what responsibilities a developer/researcher has after sharing their code: namely to inform your potential users if you will be maintaining the software and if so, respond to requests for feature additions and bug fixes. We discuss options for allowing your code to undergo continued development even if you don't have time/motivation/funding to continue iteration and encourage users of code that is no longer maintained to explore these options themselves by reaching out to the original developers. Furthermore, we discuss how users can become involved in existing projects in our next lesson.
58 |
--------------------------------------------------------------------------------
/open-software/notes.md:
--------------------------------------------------------------------------------
1 | - General points on structure
2 | - Break up each lesson into a separate markdown document
3 | - Learning objective(s) listed at the top of each lesson file
4 | - References at the end of each lesson file
5 | - Cross-referencing between lessons on different modules
6 | - : -
7 | - Single “files” directory for additional figures etc.
8 | - Summary rather than conclusions
9 | - Label each section of lesson 2.1, 2.2, etc.
10 | - Contributor list, like FAIR cookbook, but then without roles and alphabetical
11 |
--------------------------------------------------------------------------------
/open-tools-resources/CONTRIBUTORS.md:
--------------------------------------------------------------------------------
1 | Flavio Azevedo
2 | FORRT & University of Cambridge
3 | 0000-0001-9000-8513
4 | https://github.com/flavioazevedo
5 | Flavio_Azevedo_
6 |
7 | Tyson Swetnam
8 | University of Arizona
9 | 0000-0002-6639-7181
10 | tyson-swetnam
11 | tswetnam
12 |
13 | Batool Almarzouq
14 | OSCSA, KAIMRC, UoL
15 | 0000-0002-3905-2751
16 | BatoolMM
17 | batool664
18 |
19 | Saranjeet Kaur
20 | RSE Asia Association
21 | 0000-0002-7038-1457
22 | SaranjeetKaur
23 | qwertyquesting
24 |
25 | Melissa Black
26 | MetaDocencia
27 | 0000-0002-5406-2982 melibleq
28 | melissablck
29 |
30 | Rebecca Ringuette
31 | NASA Goddard Space Flight Center
32 | 0000-0003-0875-2023
33 | rebeccaringuette
34 |
35 | Elli Papadopoulou
36 | Athena Research Center / OpenAIRE
37 | 0000-0002-0893-8509 elpapado
38 | elli_lib
39 |
--------------------------------------------------------------------------------
/open-tools-resources/README.md:
--------------------------------------------------------------------------------
1 | # Objectives Outline
2 |
3 | ## 5A. Explain why open science tools encourage responsible open science (e.g., using the FAIR and CARE principles)
4 |
5 |
6 | ## 5B. Identify Open Science communities and initiatives - within and across disciplines - and join a community of practice (CoP) of interest to you
7 |
8 |
9 | ## 5C. Provide examples of how open science is practiced in a research team
10 |
11 |
12 | ## 5D. Identify types of Open Science tools along with their purpose
13 |
14 |
15 | ## 5E. Match appropriate open science tools to specific objectives within the research workflow
16 |
17 |
18 | ## 5F. Describe 3-5 open science tools and how to use them in projects (e.g., for communication, sharing of results, and collaboration).
19 |
--------------------------------------------------------------------------------
/open-tools-resources/files/placeholder.md:
--------------------------------------------------------------------------------
1 |
2 |
--------------------------------------------------------------------------------
/open-tools-resources/lesson3-tools-for-reproducibility.md:
--------------------------------------------------------------------------------
1 | # Lesson 3: Open Science tools for reproducibility
2 |
3 | ## Outline: Open Science tools for reproducibility
4 |
5 | * What is reproducibility?
6 | * Computational notebooks
7 | * Jupyter
8 | * R Markdown
9 | * Quarto
10 |
11 | ## Introduction
12 |
13 | SEE CONTENT OF THIS LESSON AT [https://tyson-swetnam.github.io/TOPS-OC5-tools/lesson3.html](https://tyson-swetnam.github.io/TOPS-OC5-tools/lesson3.html)
14 |
15 | This lesson is the third of the OpenCore Open Science Tools and Resources Modules. In this lesson, we take a deep dive into a few available tools for (computational) reproducibility. First, we define reproducibility. Then, …
16 |
17 | ## What is reproducibility?
18 |
19 | **Reproducibility ** - the [National Academies Report 2019](https://www.nationalacademies.org/our-work/reproducibility-and-replicability-in-science)** **defined reproducibility as:
20 |
21 | * **Reproducibility** means computational reproducibility—obtaining consistent computational results using the same input data, computational steps, methods, code, and conditions of analysis
22 | * **Replicability** means obtaining consistent results across studies aimed at answering the same scientific question, each of which has obtained its own data.
23 |
24 | In practice, reproducibility is taken further by an additional step. The goal of reproducibility is not only reproducing the same result given by using the same steps, such as re-executing a notebook in a containerized environment, but also allowing a given user to copy the environment and build upon the new technology and result by editing the environment to apply to a similar problem (e.g., a shareable, copyable executable paper). This small additional step gives others the ability to directly build upon previous work and get more science out of the same amount of funding.
25 |
26 | ## Computational notebooks
27 |
28 | …
29 |
30 | ### Jupyter
31 |
32 | …
33 |
34 | ### R Markdown
35 |
36 | …
37 |
38 | ### Quarto
39 |
40 | …
41 |
42 | Note:
43 |
44 | As you might have noticed, a lot of Open science tools require intermediate to advanced skills in data and information literacy and coding, especially if handling coding - intensive research projects. One of the best ways to learn these skills is through engaging with the respective communities, which often provide training and mentoring.
45 |
46 | ## Self Assessment Questions: Reproducibility
47 |
48 | **Scenario 1:** You stumble upon a research paper published a few years ago which used LANDSAT data and techniques similar to a project idea you want to apply for another area of interest. When you read the methods section of the paper, you find they published their derived data set in an international data repository (Dryad), but their algorithm code to generate the processed data from LANDSAT Real-Time (raw) data are not provided, only the description of the technique which they used is given in their Methods section and the mathematical equations for calculating their new index are in the Supplementary Materials.
49 |
50 | **Question S1-1**: From the hypothetical Scenario above, when there is access to the raw data, results data, and some written methods are provided, does the research paper meet the definition of being “reproducible”?
51 |
52 | **Answer S1-1**: No, the paper fails to provide a necessary level of detail to allow a different team, with a different experimental setup to obtain the same results exactly. The paper may support some aspects of “Replicability”, but only if someone is able to write their own code using the provided methods. With the same raw data product you could test your code and compare your results data to their results data. This would not be easy and is prohibitive.
53 |
--------------------------------------------------------------------------------
/release-workflow.md:
--------------------------------------------------------------------------------
1 | # Workflow for Releasing Different Versions
2 |
3 | We release the latest versions of the main repository including the content from this repository on Zenodo (DOI for all versions: [10.5281/zenodo.7392118](https://doi.org/10.5281/zenodo.7392118)).
4 | In this document, we describe the process for making a release on Zenodo through GitHub.
5 |
6 | We don't describe how to connect a GitHub account/repository to a Zenodo account.
7 | To learn more about that, please read the details in [the GitHub documentation](https://docs.github.com/en/repositories/archiving-a-github-repository/referencing-and-citing-content).
8 |
9 | ## Updating release information
10 |
11 | **This workflow should be followed locally on your computer (using a code/file editor and commandline/terminal).**
12 |
13 | - Update [`CITATION.cff`](https://github.com/opensciency/sprint-content/blob/main/CITATION.cff) file:
14 | - Update `version: 0.0.0` (currently in line 6), `date-released: "2023-02-15"` (currently in line 12)
15 | - Validate the file locally on your computer using `cffconvert` (here we assume that the user knows how to use their terminal)
16 | - Step 1: Please install cffconvert using the following command: `python3 -m pip install --user cffconvert` (details here: https://pypi.org/project/cffconvert/)
17 | - Step 2: Run the following command: `cffconvert --validate`
18 | - Step 3: Once validated, create `.zenodo.json` file using the following command: `cffconvert --format zenodo > .zenodo.json`
19 | - `git add` -> `git commit` -> `git push` changes to the GitHub repository
20 |
21 | Please note that we have currently listed all individual names alphabatically in the CITATION file manually.
22 | If between the previous and current release, new contributors have joined your project, please add their details in the CITATION.cff file.
23 |
24 | ## Drafting release on GitHub
25 |
26 | - Click on [the release option](https://github.com/opensciency/sprint-content/releases) on GitHub main repository
27 | - Draft a [new release](https://github.com/opensciency/sprint-content/releases/new)
28 | - Click 'Choose a tag'
29 | - provide a new version name (such as v0.0.0, v1.0.1, ...)
30 | - click the 'create a new tag on release' option
31 | - A note on when we consider a version to be [major, minor or patch](https://semver.org/):
32 | - patch: Small additions to chapters are patch such as bug fixing, editing or minor contributions between the Book Dashes
33 | - minor: Significant number of the new content & new chapters such as during a Book Dash
34 | - major: Major changes such as major re-arranging of chapters into different guides or addition of a new guide
35 | - Create the release title: **'Opensciency - A core open science curriculum by and for the research community'**
36 | - Add details similar to what we have provided below that summarises what changes are in this version and provide a short sentence under the release log:
37 |
38 | ```
39 | Opensciency is core open science curriculum material, drafted to introduce those beginning their open science journey to important definitions, tools, and resources; and provide for participants at all levels recommended practices. The material is made available under a [CC-BY 4.0 International](https://creativecommons.org/licenses/by/4.0/) license and is structured into five modules:
40 |
41 | - Ethos of Open Science
42 | - Open Tools and Resources
43 | - Open Data
44 | - Open Software
45 | - Open Results
46 |
47 | ### Citation
48 |
49 | **The latest release version can be found at this DOI [10.5281/zenodo.7392118](https://doi.org/10.5281/zenodo.7392118)**
50 |
51 | To credit and cite the material, use the following citation:
52 | OpenSciency Contributors (2022, December 2). Opensciency - A core open science curriculum by and for the research community. Zenodo. https://doi.org/10.5281/zenodo.7392119
53 |
54 | Shared under the CC-BY 4.0 License, all materials remain open for anyone to build open science curriculums or reuse for other purposes. Please include all author names where possible from the GitHub README contributors table.
55 |
56 | We encourage the wider community to reuse the material, and we are especially interested in creative approaches to displaying the material. An example we like is [Elements of AI](https://course.elementsofai.com/).
57 |
58 | ### Details of the project
59 |
60 | Opensciency is a result of the work of more than 40 open science experts and practitioners from across the world and from different disciplines. The first draft of the curriculum material was developed from [June 27 - July 1, 2022](https://github.com/nasa/Transform-to-Open-Science/blob/main/docs/Area2_Capacity_Sharing/OpenCore/OpenCore_leads.md) as part of the Transform to Open Science (TOPS) [OpenCore](https://github.com/nasa/Transform-to-Open-Science/tree/main/docs/Area2_Capacity_Sharing/OpenCore) sprint. More information about the NASA TOPS initiative is available via their [website ](https://science.nasa.gov/open-science/transform-to-open-science). After the TOPS Community Panel on [October 6, 2022](https://github.com/nasa/Transform-to-Open-Science/blob/main/docs/Area1_Engagement/Community_Panels/20221005_community_panel.md), the original contributors created the Opensciency repository to allow all contributors to further engage with the curriculum and invite review on the initial draft material from the wider research community.
61 |
62 | Let us know if you have a creative approach to displaying and reusing the material by [submitting an issue](https://github.com/opensciency/sprint-content/issues).
63 |
64 | This work is licensed under a [Creative Commons Attribution 4.0 International License](http://creativecommons.org/licenses/by/4.0/).
65 |
66 | Release log:
67 | v0.0.1: Peer-reviewed sprint document release
68 | v0.0.0: Post-sprint document release
69 |
70 | Full Changelog: v0.0.0...v0.0.1 (Previous release: v0.0.0...v0.0.1)
71 |
72 | ```
73 | - Save draft
74 |
75 | ## Ready to Release
76 |
77 | - When you are ready
78 | - Merge the changes on GitHub made in the CITATION.cff and .zenodo.json files
79 | - Double-checked the details in the drafted 'tags' and click 'Publish release'
80 | - After a few seconds, you can see a new version appear at [https://doi.org/10.5281/zenodo.7392118](https://doi.org/10.5281/zenodo.7392118)
81 |
82 | This is the workflow is written by Malvika Sharan, reusing the workflow originally written for [_The Turing Way_](https://github.com/alan-turing-institute/the-turing-way/blob/main/release-workflow.md).
83 |
--------------------------------------------------------------------------------
