122 |
123 |
124 |
125 |
126 |
127 |
--------------------------------------------------------------------------------
/docs/_build/html/_sources/auth.rst.txt:
--------------------------------------------------------------------------------
1 | Gen3 Auth Helper
2 | ----------------
3 |
4 | .. autoclass:: gen3.auth.Gen3Auth
5 | :members:
6 | :show-inheritance:
7 |
--------------------------------------------------------------------------------
/docs/_build/html/_sources/file.rst.txt:
--------------------------------------------------------------------------------
1 | Gen3 File Class
2 | ----------------
3 |
4 | .. autoclass:: gen3.file.Gen3File
5 | :members:
6 | :show-inheritance:
7 |
--------------------------------------------------------------------------------
/docs/_build/html/_sources/index.rst.txt:
--------------------------------------------------------------------------------
1 | Welcome to Gen3 SDK's documentation!
2 | ====================================
3 |
4 | The README in the Gen3 Python SDK Github Repo contains installation and setup information along with some quickstart scripts. This contains more detailed documentation about the various classes and functions available.
5 |
6 | This documentation is mostly auto-generated from the docstrings within the source code.
7 |
8 | .. toctree::
9 | :glob:
10 |
11 | *
12 |
13 | Indices and tables
14 | ==================
15 |
16 | * :ref:`genindex`
17 | * :ref:`modindex`
18 | * :ref:`search`
19 |
--------------------------------------------------------------------------------
/docs/_build/html/_sources/indexing.rst.txt:
--------------------------------------------------------------------------------
1 | Gen3 Index Class
2 | ----------------
3 |
4 | .. autoclass:: gen3.index.Gen3Index
5 | :members:
6 | :show-inheritance:
7 |
--------------------------------------------------------------------------------
/docs/_build/html/_sources/jobs.rst.txt:
--------------------------------------------------------------------------------
1 | Gen3 Jobs Class
2 | ----------------
3 |
4 | .. autoclass:: gen3.jobs.Gen3Jobs
5 | :members:
6 | :show-inheritance:
7 |
--------------------------------------------------------------------------------
/docs/_build/html/_sources/metadata.rst.txt:
--------------------------------------------------------------------------------
1 | Gen3 Metadata Class
2 | -------------------
3 |
4 | .. autoclass:: gen3.metadata.Gen3Metadata
5 | :members:
6 | :show-inheritance:
7 |
--------------------------------------------------------------------------------
/docs/_build/html/_sources/object.rst.txt:
--------------------------------------------------------------------------------
1 | Gen3 Object Class
2 | -------------------
3 |
4 | .. autoclass:: gen3.object.Gen3Object
5 | :members:
6 | :show-inheritance:
7 |
--------------------------------------------------------------------------------
/docs/_build/html/_sources/query.rst.txt:
--------------------------------------------------------------------------------
1 | Gen3 Query Class
2 | ---------------------
3 |
4 | .. autoclass:: gen3.query.Gen3Query
5 | :members:
6 | :show-inheritance:
7 |
--------------------------------------------------------------------------------
/docs/_build/html/_sources/submission.rst.txt:
--------------------------------------------------------------------------------
1 | Gen3 Submission Class
2 | ---------------------
3 |
4 | .. autoclass:: gen3.submission.Gen3Submission
5 | :members:
6 | :show-inheritance:
7 |
--------------------------------------------------------------------------------
/docs/_build/html/_sources/tools.rst.txt:
--------------------------------------------------------------------------------
1 | Gen3 Tools
2 | ----------
3 |
4 | Tools and functions for common actions in Gen3. These tools are broken up into broad categories like indexing (for tasks related to the file object persistent identifiers within the system) and metadata (for tasks relating to file object metadata within the system).
5 |
6 | Such common indexing tasks may involve indexing file object URLs into Gen3 to assign persistent identifiers, downloading a manifest of every file object that already exists, and verifying that a Gen3 instance contains the expected indexed file objects based on a file.
7 |
8 | For metadata, the task may be ingesting a large amount of metadata from a file into the system.
9 |
10 | Most of these tools utilize async capabilities of Python to make common tasks more efficient.
11 |
12 | .. toctree::
13 | :glob:
14 |
15 | tools/*
16 |
17 | .. automodule:: gen3.tools
18 | :members:
19 | :show-inheritance:
20 |
--------------------------------------------------------------------------------
/docs/_build/html/_sources/tools/drs_pull.rst.txt:
--------------------------------------------------------------------------------
1 | DRS Download Tools
2 | ------------------
3 |
4 | .. automodule:: gen3.tools.download.drs_download
5 | :members: download_files_in_drs_manifest, download_drs_object, list_files_in_drs_manifest,
6 | list_drs_object, list_access_in_drs_manifest,
7 | Manifest, DownloadManager, Downloadable, DownloadStatus,
8 |
--------------------------------------------------------------------------------
/docs/_build/html/_sources/tools/indexing.rst.txt:
--------------------------------------------------------------------------------
1 | Indexing Tools
2 | --------------
3 |
4 |
5 | Download
6 | ========
7 |
8 | .. automodule:: gen3.tools.indexing.download_manifest
9 | :members:
10 | :show-inheritance:
11 |
12 | Index
13 | ========
14 |
15 | .. automodule:: gen3.tools.indexing.index_manifest
16 | :members:
17 | :show-inheritance:
18 |
19 | Verify
20 | ========
21 |
22 | .. automodule:: gen3.tools.indexing.verify_manifest
23 | :members:
24 | :show-inheritance:
--------------------------------------------------------------------------------
/docs/_build/html/_sources/tools/metadata.rst.txt:
--------------------------------------------------------------------------------
1 | Metadata Tools
2 | --------------
3 |
4 |
5 | Ingest
6 | ========
7 |
8 | .. automodule:: gen3.tools.metadata.ingest_manifest
9 | :members:
10 | :show-inheritance:
11 |
--------------------------------------------------------------------------------
/docs/_build/html/_sources/wss.rst.txt:
--------------------------------------------------------------------------------
1 | Gen3 Workspace Storage
2 | -----------------------
3 |
4 | .. autoclass:: gen3.wss.Gen3WsStorage
5 | :members:
6 | :show-inheritance:
7 |
--------------------------------------------------------------------------------
/docs/_build/html/_static/custom.css:
--------------------------------------------------------------------------------
1 | /* This file intentionally left blank. */
2 |
--------------------------------------------------------------------------------
/docs/_build/html/_static/doctools.js:
--------------------------------------------------------------------------------
1 | /*
2 | * doctools.js
3 | * ~~~~~~~~~~~
4 | *
5 | * Base JavaScript utilities for all Sphinx HTML documentation.
6 | *
7 | * :copyright: Copyright 2007-2024 by the Sphinx team, see AUTHORS.
8 | * :license: BSD, see LICENSE for details.
9 | *
10 | */
11 | "use strict";
12 |
13 | const BLACKLISTED_KEY_CONTROL_ELEMENTS = new Set([
14 | "TEXTAREA",
15 | "INPUT",
16 | "SELECT",
17 | "BUTTON",
18 | ]);
19 |
20 | const _ready = (callback) => {
21 | if (document.readyState !== "loading") {
22 | callback();
23 | } else {
24 | document.addEventListener("DOMContentLoaded", callback);
25 | }
26 | };
27 |
28 | /**
29 | * Small JavaScript module for the documentation.
30 | */
31 | const Documentation = {
32 | init: () => {
33 | Documentation.initDomainIndexTable();
34 | Documentation.initOnKeyListeners();
35 | },
36 |
37 | /**
38 | * i18n support
39 | */
40 | TRANSLATIONS: {},
41 | PLURAL_EXPR: (n) => (n === 1 ? 0 : 1),
42 | LOCALE: "unknown",
43 |
44 | // gettext and ngettext don't access this so that the functions
45 | // can safely bound to a different name (_ = Documentation.gettext)
46 | gettext: (string) => {
47 | const translated = Documentation.TRANSLATIONS[string];
48 | switch (typeof translated) {
49 | case "undefined":
50 | return string; // no translation
51 | case "string":
52 | return translated; // translation exists
53 | default:
54 | return translated[0]; // (singular, plural) translation tuple exists
55 | }
56 | },
57 |
58 | ngettext: (singular, plural, n) => {
59 | const translated = Documentation.TRANSLATIONS[singular];
60 | if (typeof translated !== "undefined")
61 | return translated[Documentation.PLURAL_EXPR(n)];
62 | return n === 1 ? singular : plural;
63 | },
64 |
65 | addTranslations: (catalog) => {
66 | Object.assign(Documentation.TRANSLATIONS, catalog.messages);
67 | Documentation.PLURAL_EXPR = new Function(
68 | "n",
69 | `return (${catalog.plural_expr})`
70 | );
71 | Documentation.LOCALE = catalog.locale;
72 | },
73 |
74 | /**
75 | * helper function to focus on search bar
76 | */
77 | focusSearchBar: () => {
78 | document.querySelectorAll("input[name=q]")[0]?.focus();
79 | },
80 |
81 | /**
82 | * Initialise the domain index toggle buttons
83 | */
84 | initDomainIndexTable: () => {
85 | const toggler = (el) => {
86 | const idNumber = el.id.substr(7);
87 | const toggledRows = document.querySelectorAll(`tr.cg-${idNumber}`);
88 | if (el.src.substr(-9) === "minus.png") {
89 | el.src = `${el.src.substr(0, el.src.length - 9)}plus.png`;
90 | toggledRows.forEach((el) => (el.style.display = "none"));
91 | } else {
92 | el.src = `${el.src.substr(0, el.src.length - 8)}minus.png`;
93 | toggledRows.forEach((el) => (el.style.display = ""));
94 | }
95 | };
96 |
97 | const togglerElements = document.querySelectorAll("img.toggler");
98 | togglerElements.forEach((el) =>
99 | el.addEventListener("click", (event) => toggler(event.currentTarget))
100 | );
101 | togglerElements.forEach((el) => (el.style.display = ""));
102 | if (DOCUMENTATION_OPTIONS.COLLAPSE_INDEX) togglerElements.forEach(toggler);
103 | },
104 |
105 | initOnKeyListeners: () => {
106 | // only install a listener if it is really needed
107 | if (
108 | !DOCUMENTATION_OPTIONS.NAVIGATION_WITH_KEYS &&
109 | !DOCUMENTATION_OPTIONS.ENABLE_SEARCH_SHORTCUTS
110 | )
111 | return;
112 |
113 | document.addEventListener("keydown", (event) => {
114 | // bail for input elements
115 | if (BLACKLISTED_KEY_CONTROL_ELEMENTS.has(document.activeElement.tagName)) return;
116 | // bail with special keys
117 | if (event.altKey || event.ctrlKey || event.metaKey) return;
118 |
119 | if (!event.shiftKey) {
120 | switch (event.key) {
121 | case "ArrowLeft":
122 | if (!DOCUMENTATION_OPTIONS.NAVIGATION_WITH_KEYS) break;
123 |
124 | const prevLink = document.querySelector('link[rel="prev"]');
125 | if (prevLink && prevLink.href) {
126 | window.location.href = prevLink.href;
127 | event.preventDefault();
128 | }
129 | break;
130 | case "ArrowRight":
131 | if (!DOCUMENTATION_OPTIONS.NAVIGATION_WITH_KEYS) break;
132 |
133 | const nextLink = document.querySelector('link[rel="next"]');
134 | if (nextLink && nextLink.href) {
135 | window.location.href = nextLink.href;
136 | event.preventDefault();
137 | }
138 | break;
139 | }
140 | }
141 |
142 | // some keyboard layouts may need Shift to get /
143 | switch (event.key) {
144 | case "/":
145 | if (!DOCUMENTATION_OPTIONS.ENABLE_SEARCH_SHORTCUTS) break;
146 | Documentation.focusSearchBar();
147 | event.preventDefault();
148 | }
149 | });
150 | },
151 | };
152 |
153 | // quick alias for translations
154 | const _ = Documentation.gettext;
155 |
156 | _ready(Documentation.init);
157 |
--------------------------------------------------------------------------------
/docs/_build/html/_static/documentation_options.js:
--------------------------------------------------------------------------------
1 | const DOCUMENTATION_OPTIONS = {
2 | VERSION: '',
3 | LANGUAGE: 'en',
4 | COLLAPSE_INDEX: false,
5 | BUILDER: 'html',
6 | FILE_SUFFIX: '.html',
7 | LINK_SUFFIX: '.html',
8 | HAS_SOURCE: true,
9 | SOURCELINK_SUFFIX: '.txt',
10 | NAVIGATION_WITH_KEYS: false,
11 | SHOW_SEARCH_SUMMARY: true,
12 | ENABLE_SEARCH_SHORTCUTS: true,
13 | };
--------------------------------------------------------------------------------
/docs/_build/html/_static/file.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/uc-cdis/gen3sdk-python/0ca70c579d55cb0f6218fa7a534d02a8910acda1/docs/_build/html/_static/file.png
--------------------------------------------------------------------------------
/docs/_build/html/_static/language_data.js:
--------------------------------------------------------------------------------
1 | /*
2 | * language_data.js
3 | * ~~~~~~~~~~~~~~~~
4 | *
5 | * This script contains the language-specific data used by searchtools.js,
6 | * namely the list of stopwords, stemmer, scorer and splitter.
7 | *
8 | * :copyright: Copyright 2007-2024 by the Sphinx team, see AUTHORS.
9 | * :license: BSD, see LICENSE for details.
10 | *
11 | */
12 |
13 | var stopwords = ["a", "and", "are", "as", "at", "be", "but", "by", "for", "if", "in", "into", "is", "it", "near", "no", "not", "of", "on", "or", "such", "that", "the", "their", "then", "there", "these", "they", "this", "to", "was", "will", "with"];
14 |
15 |
16 | /* Non-minified version is copied as a separate JS file, if available */
17 |
18 | /**
19 | * Porter Stemmer
20 | */
21 | var Stemmer = function() {
22 |
23 | var step2list = {
24 | ational: 'ate',
25 | tional: 'tion',
26 | enci: 'ence',
27 | anci: 'ance',
28 | izer: 'ize',
29 | bli: 'ble',
30 | alli: 'al',
31 | entli: 'ent',
32 | eli: 'e',
33 | ousli: 'ous',
34 | ization: 'ize',
35 | ation: 'ate',
36 | ator: 'ate',
37 | alism: 'al',
38 | iveness: 'ive',
39 | fulness: 'ful',
40 | ousness: 'ous',
41 | aliti: 'al',
42 | iviti: 'ive',
43 | biliti: 'ble',
44 | logi: 'log'
45 | };
46 |
47 | var step3list = {
48 | icate: 'ic',
49 | ative: '',
50 | alize: 'al',
51 | iciti: 'ic',
52 | ical: 'ic',
53 | ful: '',
54 | ness: ''
55 | };
56 |
57 | var c = "[^aeiou]"; // consonant
58 | var v = "[aeiouy]"; // vowel
59 | var C = c + "[^aeiouy]*"; // consonant sequence
60 | var V = v + "[aeiou]*"; // vowel sequence
61 |
62 | var mgr0 = "^(" + C + ")?" + V + C; // [C]VC... is m>0
63 | var meq1 = "^(" + C + ")?" + V + C + "(" + V + ")?$"; // [C]VC[V] is m=1
64 | var mgr1 = "^(" + C + ")?" + V + C + V + C; // [C]VCVC... is m>1
65 | var s_v = "^(" + C + ")?" + v; // vowel in stem
66 |
67 | this.stemWord = function (w) {
68 | var stem;
69 | var suffix;
70 | var firstch;
71 | var origword = w;
72 |
73 | if (w.length < 3)
74 | return w;
75 |
76 | var re;
77 | var re2;
78 | var re3;
79 | var re4;
80 |
81 | firstch = w.substr(0,1);
82 | if (firstch == "y")
83 | w = firstch.toUpperCase() + w.substr(1);
84 |
85 | // Step 1a
86 | re = /^(.+?)(ss|i)es$/;
87 | re2 = /^(.+?)([^s])s$/;
88 |
89 | if (re.test(w))
90 | w = w.replace(re,"$1$2");
91 | else if (re2.test(w))
92 | w = w.replace(re2,"$1$2");
93 |
94 | // Step 1b
95 | re = /^(.+?)eed$/;
96 | re2 = /^(.+?)(ed|ing)$/;
97 | if (re.test(w)) {
98 | var fp = re.exec(w);
99 | re = new RegExp(mgr0);
100 | if (re.test(fp[1])) {
101 | re = /.$/;
102 | w = w.replace(re,"");
103 | }
104 | }
105 | else if (re2.test(w)) {
106 | var fp = re2.exec(w);
107 | stem = fp[1];
108 | re2 = new RegExp(s_v);
109 | if (re2.test(stem)) {
110 | w = stem;
111 | re2 = /(at|bl|iz)$/;
112 | re3 = new RegExp("([^aeiouylsz])\\1$");
113 | re4 = new RegExp("^" + C + v + "[^aeiouwxy]$");
114 | if (re2.test(w))
115 | w = w + "e";
116 | else if (re3.test(w)) {
117 | re = /.$/;
118 | w = w.replace(re,"");
119 | }
120 | else if (re4.test(w))
121 | w = w + "e";
122 | }
123 | }
124 |
125 | // Step 1c
126 | re = /^(.+?)y$/;
127 | if (re.test(w)) {
128 | var fp = re.exec(w);
129 | stem = fp[1];
130 | re = new RegExp(s_v);
131 | if (re.test(stem))
132 | w = stem + "i";
133 | }
134 |
135 | // Step 2
136 | re = /^(.+?)(ational|tional|enci|anci|izer|bli|alli|entli|eli|ousli|ization|ation|ator|alism|iveness|fulness|ousness|aliti|iviti|biliti|logi)$/;
137 | if (re.test(w)) {
138 | var fp = re.exec(w);
139 | stem = fp[1];
140 | suffix = fp[2];
141 | re = new RegExp(mgr0);
142 | if (re.test(stem))
143 | w = stem + step2list[suffix];
144 | }
145 |
146 | // Step 3
147 | re = /^(.+?)(icate|ative|alize|iciti|ical|ful|ness)$/;
148 | if (re.test(w)) {
149 | var fp = re.exec(w);
150 | stem = fp[1];
151 | suffix = fp[2];
152 | re = new RegExp(mgr0);
153 | if (re.test(stem))
154 | w = stem + step3list[suffix];
155 | }
156 |
157 | // Step 4
158 | re = /^(.+?)(al|ance|ence|er|ic|able|ible|ant|ement|ment|ent|ou|ism|ate|iti|ous|ive|ize)$/;
159 | re2 = /^(.+?)(s|t)(ion)$/;
160 | if (re.test(w)) {
161 | var fp = re.exec(w);
162 | stem = fp[1];
163 | re = new RegExp(mgr1);
164 | if (re.test(stem))
165 | w = stem;
166 | }
167 | else if (re2.test(w)) {
168 | var fp = re2.exec(w);
169 | stem = fp[1] + fp[2];
170 | re2 = new RegExp(mgr1);
171 | if (re2.test(stem))
172 | w = stem;
173 | }
174 |
175 | // Step 5
176 | re = /^(.+?)e$/;
177 | if (re.test(w)) {
178 | var fp = re.exec(w);
179 | stem = fp[1];
180 | re = new RegExp(mgr1);
181 | re2 = new RegExp(meq1);
182 | re3 = new RegExp("^" + C + v + "[^aeiouwxy]$");
183 | if (re.test(stem) || (re2.test(stem) && !(re3.test(stem))))
184 | w = stem;
185 | }
186 | re = /ll$/;
187 | re2 = new RegExp(mgr1);
188 | if (re.test(w) && re2.test(w)) {
189 | re = /.$/;
190 | w = w.replace(re,"");
191 | }
192 |
193 | // and turn initial Y back to y
194 | if (firstch == "y")
195 | w = firstch.toLowerCase() + w.substr(1);
196 | return w;
197 | }
198 | }
199 |
200 |
--------------------------------------------------------------------------------
/docs/_build/html/_static/minus.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/uc-cdis/gen3sdk-python/0ca70c579d55cb0f6218fa7a534d02a8910acda1/docs/_build/html/_static/minus.png
--------------------------------------------------------------------------------
/docs/_build/html/_static/plus.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/uc-cdis/gen3sdk-python/0ca70c579d55cb0f6218fa7a534d02a8910acda1/docs/_build/html/_static/plus.png
--------------------------------------------------------------------------------
/docs/_build/html/_static/pygments.css:
--------------------------------------------------------------------------------
1 | pre { line-height: 125%; }
2 | td.linenos .normal { color: inherit; background-color: transparent; padding-left: 5px; padding-right: 5px; }
3 | span.linenos { color: inherit; background-color: transparent; padding-left: 5px; padding-right: 5px; }
4 | td.linenos .special { color: #000000; background-color: #ffffc0; padding-left: 5px; padding-right: 5px; }
5 | span.linenos.special { color: #000000; background-color: #ffffc0; padding-left: 5px; padding-right: 5px; }
6 | .highlight .hll { background-color: #ffffcc }
7 | .highlight { background: #f8f8f8; }
8 | .highlight .c { color: #8F5902; font-style: italic } /* Comment */
9 | .highlight .err { color: #A40000; border: 1px solid #EF2929 } /* Error */
10 | .highlight .g { color: #000 } /* Generic */
11 | .highlight .k { color: #004461; font-weight: bold } /* Keyword */
12 | .highlight .l { color: #000 } /* Literal */
13 | .highlight .n { color: #000 } /* Name */
14 | .highlight .o { color: #582800 } /* Operator */
15 | .highlight .x { color: #000 } /* Other */
16 | .highlight .p { color: #000; font-weight: bold } /* Punctuation */
17 | .highlight .ch { color: #8F5902; font-style: italic } /* Comment.Hashbang */
18 | .highlight .cm { color: #8F5902; font-style: italic } /* Comment.Multiline */
19 | .highlight .cp { color: #8F5902 } /* Comment.Preproc */
20 | .highlight .cpf { color: #8F5902; font-style: italic } /* Comment.PreprocFile */
21 | .highlight .c1 { color: #8F5902; font-style: italic } /* Comment.Single */
22 | .highlight .cs { color: #8F5902; font-style: italic } /* Comment.Special */
23 | .highlight .gd { color: #A40000 } /* Generic.Deleted */
24 | .highlight .ge { color: #000; font-style: italic } /* Generic.Emph */
25 | .highlight .ges { color: #000 } /* Generic.EmphStrong */
26 | .highlight .gr { color: #EF2929 } /* Generic.Error */
27 | .highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */
28 | .highlight .gi { color: #00A000 } /* Generic.Inserted */
29 | .highlight .go { color: #888 } /* Generic.Output */
30 | .highlight .gp { color: #745334 } /* Generic.Prompt */
31 | .highlight .gs { color: #000; font-weight: bold } /* Generic.Strong */
32 | .highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */
33 | .highlight .gt { color: #A40000; font-weight: bold } /* Generic.Traceback */
34 | .highlight .kc { color: #004461; font-weight: bold } /* Keyword.Constant */
35 | .highlight .kd { color: #004461; font-weight: bold } /* Keyword.Declaration */
36 | .highlight .kn { color: #004461; font-weight: bold } /* Keyword.Namespace */
37 | .highlight .kp { color: #004461; font-weight: bold } /* Keyword.Pseudo */
38 | .highlight .kr { color: #004461; font-weight: bold } /* Keyword.Reserved */
39 | .highlight .kt { color: #004461; font-weight: bold } /* Keyword.Type */
40 | .highlight .ld { color: #000 } /* Literal.Date */
41 | .highlight .m { color: #900 } /* Literal.Number */
42 | .highlight .s { color: #4E9A06 } /* Literal.String */
43 | .highlight .na { color: #C4A000 } /* Name.Attribute */
44 | .highlight .nb { color: #004461 } /* Name.Builtin */
45 | .highlight .nc { color: #000 } /* Name.Class */
46 | .highlight .no { color: #000 } /* Name.Constant */
47 | .highlight .nd { color: #888 } /* Name.Decorator */
48 | .highlight .ni { color: #CE5C00 } /* Name.Entity */
49 | .highlight .ne { color: #C00; font-weight: bold } /* Name.Exception */
50 | .highlight .nf { color: #000 } /* Name.Function */
51 | .highlight .nl { color: #F57900 } /* Name.Label */
52 | .highlight .nn { color: #000 } /* Name.Namespace */
53 | .highlight .nx { color: #000 } /* Name.Other */
54 | .highlight .py { color: #000 } /* Name.Property */
55 | .highlight .nt { color: #004461; font-weight: bold } /* Name.Tag */
56 | .highlight .nv { color: #000 } /* Name.Variable */
57 | .highlight .ow { color: #004461; font-weight: bold } /* Operator.Word */
58 | .highlight .pm { color: #000; font-weight: bold } /* Punctuation.Marker */
59 | .highlight .w { color: #F8F8F8 } /* Text.Whitespace */
60 | .highlight .mb { color: #900 } /* Literal.Number.Bin */
61 | .highlight .mf { color: #900 } /* Literal.Number.Float */
62 | .highlight .mh { color: #900 } /* Literal.Number.Hex */
63 | .highlight .mi { color: #900 } /* Literal.Number.Integer */
64 | .highlight .mo { color: #900 } /* Literal.Number.Oct */
65 | .highlight .sa { color: #4E9A06 } /* Literal.String.Affix */
66 | .highlight .sb { color: #4E9A06 } /* Literal.String.Backtick */
67 | .highlight .sc { color: #4E9A06 } /* Literal.String.Char */
68 | .highlight .dl { color: #4E9A06 } /* Literal.String.Delimiter */
69 | .highlight .sd { color: #8F5902; font-style: italic } /* Literal.String.Doc */
70 | .highlight .s2 { color: #4E9A06 } /* Literal.String.Double */
71 | .highlight .se { color: #4E9A06 } /* Literal.String.Escape */
72 | .highlight .sh { color: #4E9A06 } /* Literal.String.Heredoc */
73 | .highlight .si { color: #4E9A06 } /* Literal.String.Interpol */
74 | .highlight .sx { color: #4E9A06 } /* Literal.String.Other */
75 | .highlight .sr { color: #4E9A06 } /* Literal.String.Regex */
76 | .highlight .s1 { color: #4E9A06 } /* Literal.String.Single */
77 | .highlight .ss { color: #4E9A06 } /* Literal.String.Symbol */
78 | .highlight .bp { color: #3465A4 } /* Name.Builtin.Pseudo */
79 | .highlight .fm { color: #000 } /* Name.Function.Magic */
80 | .highlight .vc { color: #000 } /* Name.Variable.Class */
81 | .highlight .vg { color: #000 } /* Name.Variable.Global */
82 | .highlight .vi { color: #000 } /* Name.Variable.Instance */
83 | .highlight .vm { color: #000 } /* Name.Variable.Magic */
84 | .highlight .il { color: #900 } /* Literal.Number.Integer.Long */
--------------------------------------------------------------------------------
/docs/_build/html/_static/sphinx_highlight.js:
--------------------------------------------------------------------------------
1 | /* Highlighting utilities for Sphinx HTML documentation. */
2 | "use strict";
3 |
4 | const SPHINX_HIGHLIGHT_ENABLED = true
5 |
6 | /**
7 | * highlight a given string on a node by wrapping it in
8 | * span elements with the given class name.
9 | */
10 | const _highlight = (node, addItems, text, className) => {
11 | if (node.nodeType === Node.TEXT_NODE) {
12 | const val = node.nodeValue;
13 | const parent = node.parentNode;
14 | const pos = val.toLowerCase().indexOf(text);
15 | if (
16 | pos >= 0 &&
17 | !parent.classList.contains(className) &&
18 | !parent.classList.contains("nohighlight")
19 | ) {
20 | let span;
21 |
22 | const closestNode = parent.closest("body, svg, foreignObject");
23 | const isInSVG = closestNode && closestNode.matches("svg");
24 | if (isInSVG) {
25 | span = document.createElementNS("http://www.w3.org/2000/svg", "tspan");
26 | } else {
27 | span = document.createElement("span");
28 | span.classList.add(className);
29 | }
30 |
31 | span.appendChild(document.createTextNode(val.substr(pos, text.length)));
32 | const rest = document.createTextNode(val.substr(pos + text.length));
33 | parent.insertBefore(
34 | span,
35 | parent.insertBefore(
36 | rest,
37 | node.nextSibling
38 | )
39 | );
40 | node.nodeValue = val.substr(0, pos);
41 | /* There may be more occurrences of search term in this node. So call this
42 | * function recursively on the remaining fragment.
43 | */
44 | _highlight(rest, addItems, text, className);
45 |
46 | if (isInSVG) {
47 | const rect = document.createElementNS(
48 | "http://www.w3.org/2000/svg",
49 | "rect"
50 | );
51 | const bbox = parent.getBBox();
52 | rect.x.baseVal.value = bbox.x;
53 | rect.y.baseVal.value = bbox.y;
54 | rect.width.baseVal.value = bbox.width;
55 | rect.height.baseVal.value = bbox.height;
56 | rect.setAttribute("class", className);
57 | addItems.push({ parent: parent, target: rect });
58 | }
59 | }
60 | } else if (node.matches && !node.matches("button, select, textarea")) {
61 | node.childNodes.forEach((el) => _highlight(el, addItems, text, className));
62 | }
63 | };
64 | const _highlightText = (thisNode, text, className) => {
65 | let addItems = [];
66 | _highlight(thisNode, addItems, text, className);
67 | addItems.forEach((obj) =>
68 | obj.parent.insertAdjacentElement("beforebegin", obj.target)
69 | );
70 | };
71 |
72 | /**
73 | * Small JavaScript module for the documentation.
74 | */
75 | const SphinxHighlight = {
76 |
77 | /**
78 | * highlight the search words provided in localstorage in the text
79 | */
80 | highlightSearchWords: () => {
81 | if (!SPHINX_HIGHLIGHT_ENABLED) return; // bail if no highlight
82 |
83 | // get and clear terms from localstorage
84 | const url = new URL(window.location);
85 | const highlight =
86 | localStorage.getItem("sphinx_highlight_terms")
87 | || url.searchParams.get("highlight")
88 | || "";
89 | localStorage.removeItem("sphinx_highlight_terms")
90 | url.searchParams.delete("highlight");
91 | window.history.replaceState({}, "", url);
92 |
93 | // get individual terms from highlight string
94 | const terms = highlight.toLowerCase().split(/\s+/).filter(x => x);
95 | if (terms.length === 0) return; // nothing to do
96 |
97 | // There should never be more than one element matching "div.body"
98 | const divBody = document.querySelectorAll("div.body");
99 | const body = divBody.length ? divBody[0] : document.querySelector("body");
100 | window.setTimeout(() => {
101 | terms.forEach((term) => _highlightText(body, term, "highlighted"));
102 | }, 10);
103 |
104 | const searchBox = document.getElementById("searchbox");
105 | if (searchBox === null) return;
106 | searchBox.appendChild(
107 | document
108 | .createRange()
109 | .createContextualFragment(
110 | '' + 111 | '' + 112 | _("Hide Search Matches") + 113 | "
" 114 | ) 115 | ); 116 | }, 117 | 118 | /** 119 | * helper function to hide the search marks again 120 | */ 121 | hideSearchWords: () => { 122 | document 123 | .querySelectorAll("#searchbox .highlight-link") 124 | .forEach((el) => el.remove()); 125 | document 126 | .querySelectorAll("span.highlighted") 127 | .forEach((el) => el.classList.remove("highlighted")); 128 | localStorage.removeItem("sphinx_highlight_terms") 129 | }, 130 | 131 | initEscapeListener: () => { 132 | // only install a listener if it is really needed 133 | if (!DOCUMENTATION_OPTIONS.ENABLE_SEARCH_SHORTCUTS) return; 134 | 135 | document.addEventListener("keydown", (event) => { 136 | // bail for input elements 137 | if (BLACKLISTED_KEY_CONTROL_ELEMENTS.has(document.activeElement.tagName)) return; 138 | // bail with special keys 139 | if (event.shiftKey || event.altKey || event.ctrlKey || event.metaKey) return; 140 | if (DOCUMENTATION_OPTIONS.ENABLE_SEARCH_SHORTCUTS && (event.key === "Escape")) { 141 | SphinxHighlight.hideSearchWords(); 142 | event.preventDefault(); 143 | } 144 | }); 145 | }, 146 | }; 147 | 148 | _ready(() => { 149 | /* Do not call highlightSearchWords() when we are on the search page. 150 | * It will highlight words from the *previous* search query. 151 | */ 152 | if (typeof Search === "undefined") SphinxHighlight.highlightSearchWords(); 153 | SphinxHighlight.initEscapeListener(); 154 | }); 155 | -------------------------------------------------------------------------------- /docs/_build/html/objects.inv: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/uc-cdis/gen3sdk-python/0ca70c579d55cb0f6218fa7a534d02a8910acda1/docs/_build/html/objects.inv -------------------------------------------------------------------------------- /docs/_build/html/py-modindex.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 || 45 | g | ||
| 49 | |
50 | gen3 | 51 | |
| 54 | |
55 | gen3.tools | 56 | |
| 59 | |
60 | gen3.tools.download.drs_download | 61 | |
| 64 | |
65 | gen3.tools.indexing.download_manifest | 66 | |
| 69 | |
70 | gen3.tools.indexing.index_manifest | 71 | |
| 74 | |
75 | gen3.tools.indexing.verify_manifest | 76 | |
| 79 | |
80 | gen3.tools.metadata.ingest_manifest | 81 | |
158 |
159 |
160 |
161 |
162 |
163 |
--------------------------------------------------------------------------------
/docs/_build/html/search.html:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 |
6 |
7 | 52 | Searching for multiple words only shows matches that contain 53 | all words. 54 |
55 | 56 | 57 | 62 | 63 | 64 | 65 | 66 | 67 |
130 |
131 |
132 |
133 |
134 |
135 |
--------------------------------------------------------------------------------
/docs/auth.rst:
--------------------------------------------------------------------------------
1 | Gen3 Auth Helper
2 | ----------------
3 |
4 | .. autoclass:: gen3.auth.Gen3Auth
5 | :members:
6 | :show-inheritance:
7 |
--------------------------------------------------------------------------------
/docs/conf.py:
--------------------------------------------------------------------------------
1 | # Configuration file for the Sphinx documentation builder.
2 | #
3 | # This file only contains a selection of the most common options. For a full
4 | # list see the documentation:
5 | # https://www.sphinx-doc.org/en/master/usage/configuration.html
6 |
7 | # -- Path setup --------------------------------------------------------------
8 |
9 | # If extensions (or modules to document with autodoc) are in another directory,
10 | # add these directories to sys.path here. If the directory is relative to the
11 | # documentation root, use os.path.abspath to make it absolute, like shown here.
12 | #
13 | import os
14 | import sys
15 |
16 | sys.path.insert(0, os.path.abspath(".."))
17 |
18 |
19 | # -- Project information -----------------------------------------------------
20 |
21 | project = "Gen3 SDK"
22 | copyright = "2021, Center for Translational Data Science"
23 | author = "Center for Translational Data Science"
24 |
25 | master_doc = "index"
26 |
27 | # -- General configuration ---------------------------------------------------
28 |
29 | # Add any Sphinx extension module names here, as strings. They can be
30 | # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
31 | # ones.
32 | extensions = ["sphinx.ext.autodoc", "sphinx.ext.viewcode", "sphinx.ext.napoleon"]
33 |
34 |
35 | # Add any paths that contain templates here, relative to this directory.
36 | templates_path = ["_templates"]
37 |
38 | # List of patterns, relative to source directory, that match files and
39 | # directories to ignore when looking for source files.
40 | # This pattern also affects html_static_path and html_extra_path.
41 | exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
42 |
43 |
44 | # -- Options for HTML output -------------------------------------------------
45 |
46 | # The theme to use for HTML and HTML Help pages. See the documentation for
47 | # a list of builtin themes.
48 | #
49 | html_theme = "alabaster"
50 |
51 | # Add any paths that contain custom static files (such as style sheets) here,
52 | # relative to this directory. They are copied after the builtin static files,
53 | # so a file named "default.css" will overwrite the builtin "default.css".
54 | html_static_path = ["_static"]
55 |
56 | # Theme options are theme-specific and customize the look and feel of a theme
57 | # further. For a list of options available for each theme, see the
58 | # documentation.
59 | #
60 | html_theme_options = {
61 | "show_powered_by": False,
62 | "github_user": "uc-cdis",
63 | "github_repo": "gen3sdk-python",
64 | "github_banner": True,
65 | "show_related": False,
66 | }
67 |
--------------------------------------------------------------------------------
/docs/file.rst:
--------------------------------------------------------------------------------
1 | Gen3 File Class
2 | ----------------
3 |
4 | .. autoclass:: gen3.file.Gen3File
5 | :members:
6 | :show-inheritance:
7 |
--------------------------------------------------------------------------------
/docs/howto/bundleTools.md:
--------------------------------------------------------------------------------
1 | ## Bundle Tools
2 |
3 | TOC
4 | - [Ingest Manifest](#ingest-manifest)
5 | - [Example Bundle](#example-bundle)
6 |
7 | ### Ingest Manifest
8 |
9 | The only required columns are `bundle_name` and `ids`(list of object or bundles). The order of bundles in the manifest matters: you can reference a `bundle_name` in the `ids` list if that bundle_name appears before the bundle containing it. Lowest level bundles should only contain GUIDs to File Objects and should live at the top of the manifest. If a GUID is not provided for a bundle, indexd will assign a GUID for the bundle. Within the manifest `bundle_names` is to be used as a unique identifier.
10 |
11 | #### Example Bundle:
12 | To create the following bundle:
13 | ```
14 | Bundle A
15 | +- BundleB
16 | +- BundleC
17 | +-File1
18 | +-File2
19 | +-File3
20 | ```
21 | the manifest should look like this
22 | ```
23 | bundle_name,ids,GUID(optional) // can also include other optional fields defined by the DRS spec
24 | BundleC,[File1-GUID File2-GUID],
25 | BundleB,[File3-GUID BundleC],
26 | BundleA, [BundleB BundleC],
27 | ```
28 |
29 | The following is an example csv manifest:
30 | ```
31 | bundle_name,ids,GUID,size,type,checksum,description
32 | A,[dg.TEST/f2a39f98-6ae1-48a5-8d48-825a0c52a22b dg.TEST/1e9d3103-cbe2-4c39-917c-b3abad4750d2],,,,,some description
33 | B,['dg.TEST/1e9d3103-cbe2-4c39-917c-b3abad4750d2' 'dg.TEST/f2a39f98-6ae1-48a5-8d48-825a0c52a22b'],,789,,,something
34 | C,[A 'B' dg.TEST/ed8f4658-6acd-4f96-9dd8-3709890c959e],,120,,,lalala
35 | D,[A B C],,,[md5 sha256],[1234567 abc12345],
36 | E,[A B],dg.xxxx/590ee63d-2790-477a-bbf8-d53873ca4933,,md5 sha256,abcdefg abcd123,
37 | ```
38 | NOTE: DrsObjects/Bundles support multiple checksums so in the manifest define type and the hash respectively.
39 |
40 | Example:
41 | ```
42 | type,checksum
43 | md5 sha256, abcde 12345
44 | ```
45 | The above manifest would result to the following `checksums` field in the bundle:
46 | ```json
47 | "checksums":[{"type": "md5", "checksum": "abcde"}, {"type": "sha256", "checksum": "12345"}]
48 | ```
49 |
50 | ```python
51 | import sys
52 | import logging
53 |
54 | from gen3.auth import Gen3Auth
55 | from gen3.tools.bundle.ingest_manifest import ingest_bundle_manifest
56 |
57 | logging.basicConfig(filename="output.log", level=logging.DEBUG)
58 | logging.getLogger().addHandler(logging.StreamHandler(sys.stdout))
59 |
60 | COMMONS = "https://{{insert-commons-here}}/"
61 | MANIFEST = "./example_manifest.tsv"
62 |
63 | def main():
64 | auth = Gen3Auth(COMMONS, refresh_file="credentials.json")
65 |
66 | # use basic auth for admin privileges in indexd
67 | # auth = ("basic_auth_username", "basic_auth_password")
68 |
69 | ingest_bundle_manifest(
70 | commons_url=COMMONS,
71 | manifest_file=MANIFEST,
72 | out_manifest_file="ingest_out.csv",
73 | auth=auth,
74 | )
75 |
76 | if __name__ == "__main__":
77 | main()
78 | ```
79 |
--------------------------------------------------------------------------------
/docs/howto/cli.md:
--------------------------------------------------------------------------------
1 | ## CLI
2 |
3 | The CLI can be invoked as follows
4 |
5 | `gen3 [OPTIONS] COMMAND [ARGS]`
6 |
7 | For a list of commands and options run
8 |
9 | `gen3 --help`
10 |
11 | For example, the following can validate `user.yaml` files
12 |
13 | `gen3 users validate first_user.yaml second_user.yaml`
14 |
15 | The [gen3users](https://github.com/uc-cdis/gen3users) documentation describes the `users` functionality.
16 |
17 |
--------------------------------------------------------------------------------
/docs/howto/crosswalk.md:
--------------------------------------------------------------------------------
1 | # Gen3 Cross-Commons Subject Linking and Crosswalk
2 |
3 | A general solution that supports the linking of subjects across commons. A researcher should be able to “crosswalk” the information from one commons to combine with subject data from other commons.
4 |
5 | We accomplish this by utilizing the Gen3 Framework Services API, specifically the metadata/semi-structured data support. The SDK simplifies the management of this crosswalk data within the framework services.
6 |
7 | > Note that all data in the Gen3 Metadata API **MUST BE OPEN ACCESS AND CONTAIN NO PII**
8 |
9 | ## Centralized Mapping Information Format
10 |
11 | Note that this block effectively represents subject-level data, with the crosswalk being a namespace within that overall block. This **example** is a metadata record from the Gen3 Metadata API (powered by the metadata-service).
12 |
13 | ```json
14 | "GUID": {
15 | "crosswalk": {
16 | "subject": {
17 | "{{commons_url}}": {
18 | "{{field_name}}": {
19 | "value": "",
20 | "type": "",
21 | "description": ""
22 | }
23 | // ... more field entries here
24 | },
25 | // ... more commons entries here
26 | "mapping_methodologies": [
27 | ""
28 | ]
29 | }
30 | }
31 | }
32 |
33 | ```
34 |
35 | Example:
36 |
37 | ```json
38 | "GUID": {
39 | "crosswalk": {
40 | "subject": {
41 | "https://gen3.biodatacatalyst.nhlbi.nih.gov": {
42 | "Subject.submitter_id": {
43 | "value": "phs002363.v1_RC-1358",
44 | "type": "gen3_node_property",
45 | "description": "These identifiers are constructed as part of the data ingestion process in BDCat and concatenate the study and version with the study-provided subject ID (with a _ delimiting)."
46 | }
47 | },
48 | "https://data.midrc.org": {
49 | "Case.submitter_id": {
50 | "value": "A01-00888",
51 | "type": "gen3_node_property",
52 | "description": "The uniquely assigned case identifier in MIDRC."
53 | },
54 | "Case.data_submission_guid": {
55 | "value": "foobar",
56 | "type": "gen3_node_property",
57 | "description": "The identifier for this subject as provided by the site’s submission of Datavant tokens to MIDRC."
58 | },
59 | "masked_n3c_id": {
60 | "value": "123dfj4ia5oi*@a",
61 | "type": "masked_n3c_id",
62 | "description": "Masked National COVID Consortium ID provided by a Linkage Honest Broker to the MIDRC system."
63 | }
64 | },
65 | "mapping_methodologies": [
66 | "NHLBI provided a file of subject IDs for the PETAL study that directly associate a PETAL ID with a BDCat Subject Identifier.",
67 | "A Linkage Honest Broker provided MIDRC with what Masked N3C IDs match MIDRC cases via a system-to-system handoff."
68 | ]
69 | }
70 | }
71 | }
72 | ```
73 |
74 | ## Crosswalk Data Upload Using Gen3 SDK/CLI
75 |
76 | ### `crosswalk.csv`
77 |
78 | To provide mapping from one commons identifier to another.
79 |
80 | * Columns are pipe-delimited and contain necessary information for crosswalk metadata
81 | * `{{commons url}}|{{identifier type}}|{{identifier name}}`
82 | * `{{identifier type}}` for Gen3 Graph node property: `gen3_node_property`
83 | * `{{identifier name}}` for Gen3 Graph node property: `{{node}}.{{property}}`
84 | * File name does not matter
85 |
86 | ### `crosswalk_optional_info.csv`
87 |
88 | To provide descriptions for commons identifiers.
89 |
90 | * `{{commons url}}, {{identifier name}}, {{description}}`
91 | * File name does not matter
92 |
93 | ### Example 1
94 |
95 | `crosswalk_1.csv`
96 |
97 | ```
98 | https://data.midrc.org|gen3_node_property|Case.submitter_id, https://gen3.biodatacatalyst.nhlbi.nih.gov|gen3_node_property|Subject.submitter_id
99 | A01-00888, phs002363.v1_RC-1358
100 | …
101 | ```
102 |
103 | (optional) `crosswalk_optional_info_1.csv`
104 |
105 | > Note: MUST include headers `commons_url,identifier_name,description`
106 |
107 | ```
108 | commons_url,identifier_name,description
109 | https://data.midrc.org, Case.submitter_id, The uniquely assigned case identifier in MIDRC.
110 | https://gen3.biodatacatalyst.nhlbi.nih.gov, Subject.submitter_id, These identifiers are constructed as part of the data ingestion process in BDCat and concatonate the study and version with the study-provided subject ID (with a _ delimiting).
111 | ```
112 |
113 | Gen3 SDK Command
114 |
115 | ```
116 | gen3 objects crosswalk publish ./tests/test_data/crosswalk/crosswalk_1.csv -m "NHLBI provided a file of subject IDs for the PETAL study that directly associate a PETAL ID with a BDCat Subject Identifier." --info ./tests/test_data/crosswalk/crosswalk_optional_info_1.csv
117 | ```
118 |
119 | > `publish` merges any new crosswalk data with existing data
120 |
121 | ### Example 2
122 |
123 | `crosswalk_2.csv`
124 |
125 | ```
126 | https://data.midrc.org|gen3_node_property|Case.submitter_id, https://data.midrc.org|gen3_node_property|Case.data_submission_guid,
127 | https://data.midrc.org|masked_n3c_id|Masked N3C ID
128 | A01-00888, foobar, 123dfj4ia5oi*@a
129 | …
130 | ```
131 |
132 | (optional) `crosswalk_optional_info_2.csv`
133 |
134 | > Note: MUST include headers `commons_url,identifier_name,description`
135 |
136 | ```
137 | commons_url,identifier_name,description
138 | https://data.midrc.org, Case.data_submission_guid, The identifier for this subject as provided by the site’s submission of Datavant tokens to MIDRC.
139 | https://data.midrc.org,Masked N3C ID,Masked National COVID Consortium ID provided by a Linkage Honest Broker to the MIDRC system.
140 | ```
141 |
142 | Gen3 SDK Command
143 |
144 | ```
145 | gen3 objects crosswalk publish crosswalk_2.csv -m "A Linkage Honest Broker provided MIDRC with what Masked N3C IDs match MIDRC cases via a system-to-system handoff." --info crosswalk_optional_info_2.csv
146 | ```
147 |
148 | > `publish` merges any new crosswalk data with existing data
149 |
150 | ## Gen3 SDK handling crosswalk.csv submission to MDS
151 |
152 | ```
153 | gen3 objects crosswalk --help
154 | ```
155 |
156 | General flow for `publish` is:
157 |
158 | - Parse provided `crosswalk.csv` and optionally `crosswalk_optional_info.csv` file(s)
159 | - Validate format(s) (especially column names)
160 | - Convert information from `crosswalk.csv` into a payload to push to the MDS based on "Centralized Mapping Information Format" above
161 |
--------------------------------------------------------------------------------
/docs/howto/devTest.md:
--------------------------------------------------------------------------------
1 | ## Dev-Test
2 |
3 | ### Set up Python Virtual Environment
4 |
5 | You can set up a Python development environment with a virtual environment:
6 |
7 | ```bash
8 | python3 -m venv py3
9 | ```
10 |
11 | Make sure that you have the virtual environment activated:
12 |
13 | ```bash
14 | . py3/bin/activate
15 | ```
16 |
17 | ### Install poetry
18 |
19 | To use the latest code in this repo (or to develop new features) you can clone this repo, install `poetry`:
20 |
21 | ```
22 | curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -
23 | ```
24 |
25 | and then use `poetry` to install this package:
26 |
27 | ```
28 | poetry install -vv
29 | ```
30 |
31 |
32 | ### Setup and run tests
33 |
34 | Local development like this:
35 |
36 | ```
37 | poetry shell
38 | poetry install -vv
39 | python3 -m pytest
40 | ```
41 |
42 | There are various ways to select a subset of python unit-tests - see: https://stackoverflow.com/questions/36456920/is-there-a-way-to-specify-which-pytest-tests-to-run-from-a-file
43 |
44 | ### Manual Testing
45 |
46 | You can also set up credentials to submit data to the graph in your data commons. This assumes that you can get API access by downloading your [credentials.json](https://docs.gen3.org/gen3-resources/user-guide/using-api/#credentials-to-send-api-requests).
47 |
48 | > Make sure that your python virtual environment and dependencies are updated. Also, check that your credentials have appropriate permissions to make the service calls too.
49 | ```python
50 | COMMONS_URL = "https://mycommons.azurefd.net"
51 | PROGRAM_NAME = "MyProgram"
52 | PROJECT_NAME = "MyProject"
53 | CREDENTIALS_FILE_PATH = "credentials.json"
54 | gen3_node_json = {
55 | "projects": {"code": PROJECT_NAME},
56 | "type": "core_metadata_collection",
57 | "submitter_id": "core_metadata_collection_myid123456",
58 | }
59 | auth = Gen3Auth(endpoint=COMMONS_URL, refresh_file=CREDENTIALS_FILE_PATH)
60 | sheepdog_client = Gen3Submission(COMMONS_URL, auth)
61 | json_result = sheepdog_client.submit_record(PROGRAM_NAME, PROJECT_NAME, gen3_node_json)
62 | ```
63 |
64 | ### CLI
65 |
66 | If the `gen3` cli is not in your path, or is overwritten by a shell function, then you can still invoke the cli:
67 |
68 | ```
69 | python -m gen3.cli --help
70 | or
71 | poetry run gen3 --help
72 | ```
73 |
--------------------------------------------------------------------------------
/docs/howto/externalFileDownloading.md:
--------------------------------------------------------------------------------
1 | ## Downloading files from external repositories
2 |
3 | ## External file metadata
4 |
5 | The study metadata should indicate if data are hosted in an external repository.
6 | This is specified in the `external_file_metadata` field. An example is shown below.
7 |
8 |
9 | ```json
10 | {
11 | "_guid_type": "discovery_metadata",
12 | "gen3_discovery": {
13 | // Gen3 administrative fields
14 | ...,
15 |
16 | [
17 | {
18 | "external_oidc_idp": "externaldata-keycloak",
19 | "file_retriever": "QDR",
20 | "study_id": "QDR_study_01"
21 | },
22 | {
23 | "external_oidc_idp": "externaldata-keycloak",
24 | "file_retriever": "QDR",
25 | "file_id": "QDR_file_02"
26 | },
27 | ]
28 | }
29 | }
30 | ```
31 |
32 | The `'file_retriever'` field is required. It is used to determine how to retrieve the file.
33 |
34 | The `'external_oidc_idp'` field is required if retrieving the file(s) requires to get a token from the `workspace token service`. if the file(s) can be retrieved without any tokens, this field could be omitted.
35 |
36 | The `'study_id'` and `'file_id'` fields are allowed but are not required.
37 |
38 | ## Example code with external file download using a retriever function
39 |
40 | The code should import `download_files_from_metadata` as well as a retriever function.
41 |
42 | Prior to running the download code, there should be a call to the WTS `authorization_url` endpoint,
43 |
44 | `