├── .gitignore
├── AUTHORS
├── CONTRIBUTING.md
├── LICENSE
├── LeelaEvalUpdating.md
├── PROCESS
├── README.md
├── oneoff
├── build_example_instance.py
├── convert_to_lz.py
├── delete_bucket.py
├── early_move_canonical.py
├── game_num_fixer.py
├── katago-importer.py
├── leela-all-to-dirs.sh
├── leela-model-importer.py
├── leela-process.md
├── leela-ringmaster-importer.py
├── leela-zero-v3-eval.ctl
├── model_id_guess.py
├── photo_magic.sh
├── repopulate_db.sh
├── setup-run.sh
├── setup-runs-data.sh
├── sgfstrip-eval.sh
├── subsample-v3.sh
├── subsample-v9.sh
└── unify-v3.sh
├── rsync-data.sh
├── schema.sql
├── setup.sh
├── updater.py
└── web
├── __init__.py
├── cloudyback.py
├── cloudygo.py
├── serve.py
├── sgf_utils.py
├── static
├── general.js
├── graphs.js
├── robots.txt
└── styles
│ ├── general.css
│ ├── table.css
│ └── wgo-mobile.css
├── templates
├── README.html
├── figure-three.html
├── fileslist.html
├── game.html
├── macros.html
├── model-eval.html
├── model-graphs.html
├── model.html
├── models-eval-cross.html
├── models-eval-empty.html
├── models-eval.html
├── models-graphs-sliders.html
├── models-graphs.html
├── models.html
├── nearest-neighbors.html
├── position-comparison.html
├── position-evolution.html
├── puzzle.html
├── secret-site-nav.html
├── site-nav.html
├── sprt.html
└── tsne.html
└── utils.py
/.gitignore:
--------------------------------------------------------------------------------
1 | __pycache__/
2 | *.py[cod]
3 |
4 | sgf
5 | instance/
6 | web/static/wgo
7 | web/static/favicon*
8 | web/static/d3-tip.js
9 | web/static/*.pickle
10 |
--------------------------------------------------------------------------------
/AUTHORS:
--------------------------------------------------------------------------------
1 | Google LLC
2 | Seth Troisi
3 | Andrew Jackson
4 |
--------------------------------------------------------------------------------
/CONTRIBUTING.md:
--------------------------------------------------------------------------------
1 | # How to Contribute
2 |
3 | We'd love to accept your patches and contributions to this project. There are
4 | just a few small guidelines you need to follow.
5 |
6 | ## Contributor License Agreement
7 |
8 | Contributions to this project must be accompanied by a Contributor License
9 | Agreement. You (or your employer) retain the copyright to your contribution;
10 | this simply gives us permission to use and redistribute your contributions as
11 | part of the project. Head over to to see
12 | your current agreements on file or to sign a new one.
13 |
14 | You generally only need to submit a CLA once, so if you've already submitted one
15 | (even if it was for a different project), you probably don't need to do it
16 | again.
17 |
18 | ## Code reviews
19 |
20 | All submissions, including submissions by project members, require review. We
21 | use GitHub pull requests for this purpose. Consult
22 | [GitHub Help](https://help.github.com/articles/about-pull-requests/) for more
23 | information on using pull requests.
24 |
25 | ## Community Guidelines
26 |
27 | This project follows [Google's Open Source Community
28 | Guidelines](https://opensource.google.com/conduct/).
29 |
--------------------------------------------------------------------------------
/LICENSE:
--------------------------------------------------------------------------------
1 |
2 | Apache License
3 | Version 2.0, January 2004
4 | http://www.apache.org/licenses/
5 |
6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
7 |
8 | 1. Definitions.
9 |
10 | "License" shall mean the terms and conditions for use, reproduction,
11 | and distribution as defined by Sections 1 through 9 of this document.
12 |
13 | "Licensor" shall mean the copyright owner or entity authorized by
14 | the copyright owner that is granting the License.
15 |
16 | "Legal Entity" shall mean the union of the acting entity and all
17 | other entities that control, are controlled by, or are under common
18 | control with that entity. For the purposes of this definition,
19 | "control" means (i) the power, direct or indirect, to cause the
20 | direction or management of such entity, whether by contract or
21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the
22 | outstanding shares, or (iii) beneficial ownership of such entity.
23 |
24 | "You" (or "Your") shall mean an individual or Legal Entity
25 | exercising permissions granted by this License.
26 |
27 | "Source" form shall mean the preferred form for making modifications,
28 | including but not limited to software source code, documentation
29 | source, and configuration files.
30 |
31 | "Object" form shall mean any form resulting from mechanical
32 | transformation or translation of a Source form, including but
33 | not limited to compiled object code, generated documentation,
34 | and conversions to other media types.
35 |
36 | "Work" shall mean the work of authorship, whether in Source or
37 | Object form, made available under the License, as indicated by a
38 | copyright notice that is included in or attached to the work
39 | (an example is provided in the Appendix below).
40 |
41 | "Derivative Works" shall mean any work, whether in Source or Object
42 | form, that is based on (or derived from) the Work and for which the
43 | editorial revisions, annotations, elaborations, or other modifications
44 | represent, as a whole, an original work of authorship. For the purposes
45 | of this License, Derivative Works shall not include works that remain
46 | separable from, or merely link (or bind by name) to the interfaces of,
47 | the Work and Derivative Works thereof.
48 |
49 | "Contribution" shall mean any work of authorship, including
50 | the original version of the Work and any modifications or additions
51 | to that Work or Derivative Works thereof, that is intentionally
52 | submitted to Licensor for inclusion in the Work by the copyright owner
53 | or by an individual or Legal Entity authorized to submit on behalf of
54 | the copyright owner. For the purposes of this definition, "submitted"
55 | means any form of electronic, verbal, or written communication sent
56 | to the Licensor or its representatives, including but not limited to
57 | communication on electronic mailing lists, source code control systems,
58 | and issue tracking systems that are managed by, or on behalf of, the
59 | Licensor for the purpose of discussing and improving the Work, but
60 | excluding communication that is conspicuously marked or otherwise
61 | designated in writing by the copyright owner as "Not a Contribution."
62 |
63 | "Contributor" shall mean Licensor and any individual or Legal Entity
64 | on behalf of whom a Contribution has been received by Licensor and
65 | subsequently incorporated within the Work.
66 |
67 | 2. Grant of Copyright License. Subject to the terms and conditions of
68 | this License, each Contributor hereby grants to You a perpetual,
69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable
70 | copyright license to reproduce, prepare Derivative Works of,
71 | publicly display, publicly perform, sublicense, and distribute the
72 | Work and such Derivative Works in Source or Object form.
73 |
74 | 3. Grant of Patent License. Subject to the terms and conditions of
75 | this License, each Contributor hereby grants to You a perpetual,
76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable
77 | (except as stated in this section) patent license to make, have made,
78 | use, offer to sell, sell, import, and otherwise transfer the Work,
79 | where such license applies only to those patent claims licensable
80 | by such Contributor that are necessarily infringed by their
81 | Contribution(s) alone or by combination of their Contribution(s)
82 | with the Work to which such Contribution(s) was submitted. If You
83 | institute patent litigation against any entity (including a
84 | cross-claim or counterclaim in a lawsuit) alleging that the Work
85 | or a Contribution incorporated within the Work constitutes direct
86 | or contributory patent infringement, then any patent licenses
87 | granted to You under this License for that Work shall terminate
88 | as of the date such litigation is filed.
89 |
90 | 4. Redistribution. You may reproduce and distribute copies of the
91 | Work or Derivative Works thereof in any medium, with or without
92 | modifications, and in Source or Object form, provided that You
93 | meet the following conditions:
94 |
95 | (a) You must give any other recipients of the Work or
96 | Derivative Works a copy of this License; and
97 |
98 | (b) You must cause any modified files to carry prominent notices
99 | stating that You changed the files; and
100 |
101 | (c) You must retain, in the Source form of any Derivative Works
102 | that You distribute, all copyright, patent, trademark, and
103 | attribution notices from the Source form of the Work,
104 | excluding those notices that do not pertain to any part of
105 | the Derivative Works; and
106 |
107 | (d) If the Work includes a "NOTICE" text file as part of its
108 | distribution, then any Derivative Works that You distribute must
109 | include a readable copy of the attribution notices contained
110 | within such NOTICE file, excluding those notices that do not
111 | pertain to any part of the Derivative Works, in at least one
112 | of the following places: within a NOTICE text file distributed
113 | as part of the Derivative Works; within the Source form or
114 | documentation, if provided along with the Derivative Works; or,
115 | within a display generated by the Derivative Works, if and
116 | wherever such third-party notices normally appear. The contents
117 | of the NOTICE file are for informational purposes only and
118 | do not modify the License. You may add Your own attribution
119 | notices within Derivative Works that You distribute, alongside
120 | or as an addendum to the NOTICE text from the Work, provided
121 | that such additional attribution notices cannot be construed
122 | as modifying the License.
123 |
124 | You may add Your own copyright statement to Your modifications and
125 | may provide additional or different license terms and conditions
126 | for use, reproduction, or distribution of Your modifications, or
127 | for any such Derivative Works as a whole, provided Your use,
128 | reproduction, and distribution of the Work otherwise complies with
129 | the conditions stated in this License.
130 |
131 | 5. Submission of Contributions. Unless You explicitly state otherwise,
132 | any Contribution intentionally submitted for inclusion in the Work
133 | by You to the Licensor shall be under the terms and conditions of
134 | this License, without any additional terms or conditions.
135 | Notwithstanding the above, nothing herein shall supersede or modify
136 | the terms of any separate license agreement you may have executed
137 | with Licensor regarding such Contributions.
138 |
139 | 6. Trademarks. This License does not grant permission to use the trade
140 | names, trademarks, service marks, or product names of the Licensor,
141 | except as required for reasonable and customary use in describing the
142 | origin of the Work and reproducing the content of the NOTICE file.
143 |
144 | 7. Disclaimer of Warranty. Unless required by applicable law or
145 | agreed to in writing, Licensor provides the Work (and each
146 | Contributor provides its Contributions) on an "AS IS" BASIS,
147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
148 | implied, including, without limitation, any warranties or conditions
149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
150 | PARTICULAR PURPOSE. You are solely responsible for determining the
151 | appropriateness of using or redistributing the Work and assume any
152 | risks associated with Your exercise of permissions under this License.
153 |
154 | 8. Limitation of Liability. In no event and under no legal theory,
155 | whether in tort (including negligence), contract, or otherwise,
156 | unless required by applicable law (such as deliberate and grossly
157 | negligent acts) or agreed to in writing, shall any Contributor be
158 | liable to You for damages, including any direct, indirect, special,
159 | incidental, or consequential damages of any character arising as a
160 | result of this License or out of the use or inability to use the
161 | Work (including but not limited to damages for loss of goodwill,
162 | work stoppage, computer failure or malfunction, or any and all
163 | other commercial damages or losses), even if such Contributor
164 | has been advised of the possibility of such damages.
165 |
166 | 9. Accepting Warranty or Additional Liability. While redistributing
167 | the Work or Derivative Works thereof, You may choose to offer,
168 | and charge a fee for, acceptance of support, warranty, indemnity,
169 | or other liability obligations and/or rights consistent with this
170 | License. However, in accepting such obligations, You may act only
171 | on Your own behalf and on Your sole responsibility, not on behalf
172 | of any other Contributor, and only if You agree to indemnify,
173 | defend, and hold each Contributor harmless for any liability
174 | incurred by, or claims asserted against, such Contributor by reason
175 | of your accepting any such warranty or additional liability.
176 |
177 | END OF TERMS AND CONDITIONS
178 |
179 | Copyright 2017 Google LLC
180 |
181 | Licensed under the Apache License, Version 2.0 (the "License");
182 | you may not use this file except in compliance with the License.
183 | You may obtain a copy of the License at
184 |
185 | http://www.apache.org/licenses/LICENSE-2.0
186 |
187 | Unless required by applicable law or agreed to in writing, software
188 | distributed under the License is distributed on an "AS IS" BASIS,
189 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
190 | See the License for the specific language governing permissions and
191 | limitations under the License.
192 |
--------------------------------------------------------------------------------
/LeelaEvalUpdating.md:
--------------------------------------------------------------------------------
1 | # Steps for running and updating leela eval
2 |
3 | ```
4 | # Get new model list
5 | cd ~/Projects/cloudygo
6 | oneoff/leela-model-importer.py
7 | oneoff/leela-model-importer.py leela-zero-eval
8 |
9 | # Update sql db
10 | sqlite3 instance/clouds.db ".mode csv" ".import instance/data/leela-zero/inserts.csv models"
11 | sqlite3 instance/clouds.db ".mode csv" ".import instance/data/leela-zero-eval/inserts.csv models"
12 |
13 | # Remove empty weight files from leela-model-importer.py
14 | find instance/data/leela-zero/models/ -iname 'LZ*' -type f -empty
15 | find instance/data/leela-zero/models/ -iname 'LZ*' -type f -empty -delete
16 |
17 | # Download good weight files
18 | cd instance/data/leela-zero
19 | ./download.sh
20 |
21 | # Rsync weights to fast (this is Seth specific)
22 | rsync -tr --info=progress2 models/LZ* "eights@fast.cloudygo.com:~/Projects/ringmaster/leela-zero-v3/"
23 |
24 | # Later sync games back with this helpful macro
25 | cd ~/Projects/cloudygo
26 | function sync_and_update_lz3 { rsync -tr --info=progress2 "eights@fast.cloudygo.com:~/Projects/ringmaster/leela-zero-v3-eval.games" instance/data/leela-zero-eval/eval/; }
27 |
28 | # Update cloudygo
29 | sync_and_update_lz3; ./updater.py eval_games leela-zero-eval leela-zero-eval-time
30 | ```
31 |
--------------------------------------------------------------------------------
/PROCESS:
--------------------------------------------------------------------------------
1 | # vim oneoff/setup-runs-data.sh
2 |
3 | # oneoff/convert_to_lz.py
4 |
5 | NEW_RUN='v17-19x19'
6 | # oneoff/setup-run.sh $NEW_RUN
7 | # cp -r instance/positions/v12-19x19 instance/positions/$NEW_RUN
8 | #
9 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # CloudyGo
2 |
3 | Frontend for displaying
4 | [MiniGo](https://github.com/tensorflow/minigo),
5 | [Leela-Zero](https://github.com/gcp/leela-zero),
6 | and other Go data.
7 |
8 | *This is not an official Google product.*
9 |
10 | ## Getting Started
11 |
12 | Visit [CloudyGo.com](http://CloudyGo.com) to see a live version.
13 |
14 | ### Prerequisites
15 |
16 | The site requires several python libraries, this may not be a complete list
17 | ```
18 | pip3 install choix flask numpy tqdm sqlite3 requests
19 | ```
20 | sgftopng is required to render some images, but isn't required to run.
21 |
22 | For local development I suggest using
23 | [devel_instance.7z](https://drive.google.com/file/d/1IwvvSLpKnrzNQUX6XaRaAQ5KXkjn9rrG/view?usp=sharing) to bootstrap.
24 |
25 | devel_instance.7z contains enough of MiniGo v3-9x9 and v7-19x19 data to test the UI.
26 |
27 | ```
28 | 7z x devel_instance.7z
29 | mv -n devel_instance instance
30 | ./oneoff/repopulate_db.sh
31 | ./updater.py models True v3-9x9 v5-19x19 v7-19x19 leela-zero-v1
32 | ./updater.py games v3-9x9 v5-19x19 v7-19x19 leela-zero-v1
33 | ./updater.py eval_games v3-9x9 v5-19x19 v7-19x19 leela-zero-v1
34 | ./updater.py position_evals v3-9x9 v5-19x19 v7-19x19 leela-zero-v1
35 | FLASK_DEBUG=1 FLASK_APP="web/serve.py" flask run --host 0.0.0.0 --port 5000
36 | # follow instructions in SETUP so SGFS can be rendered with WGo.js
37 | ```
38 |
39 | ### Coding style
40 |
41 | Style guide is a mix of Google Python + PEP8,
42 | Some older code may not be perfectly compliant.
43 |
44 | ## Deployment
45 |
46 | CloudyGo.com is run by Seth Troisi, local deployment is normally tested with
47 | ```
48 | FLASK_DEBUG=1 FLASK_APP="web/serve.py" flask run --host 0.0.0.0 --port 6000
49 | ```
50 |
51 | ## Full Site Setup
52 |
53 | * Some initial instructions are in [SETUP](SETUP).
54 |
55 | * [rsync-data.sh](rsync-data.sh) helps copy data from [MiniGo's Google Cloud Storage public bucket](https://console.cloud.google.com/storage/browser/minigo-pub)
56 |
57 |
58 | instance/ # Created with oneoff/repopulate_db.sh from schema.sql
59 | ├── cloudygo.db # Created with oneoff/repopulate_db.sh from schema.sql
60 | ├── data/ # directory (or link to directory) of MiniGo training data
61 | │ ├── v7-19x19/ # Training Run #1
62 | │ │ ├── models/ # See minigo-pub Google Cloud Storage Bucket
63 | │ │ ├── sgf/ # See minigo-pub Google Cloud Storage Bucket
64 | │ └── ... # Other Training Runs
65 | ├── eval/v7-19x19/ # [Figure 3 details](http://cloudygo.com/v7-19x19/figure-three) produced by [minigo/oneoffs](https://github.com/tensorflow/minigo/blob/master/oneoffs/training_curve.py)
66 | ├── policy/v7-19x19/ # [Policy heatmaps](http://cloudygo.com/v7-19x19/models_evolution/?M=189&P=13) produced by [minigo/oneoffs](https://github.com/tensorflow/minigo/blob/master/oneoffs/heatmap.py)
67 | ├── pv/v7-19x19/ # [Principle variations](http://cloudygo.com/v7-19x19/models_evolution/?M=189&P=13) produced by [minigo/oneoffs](https://github.com/tensorflow/minigo/blob/master/oneoffs/position_pv.py)
68 | ├── openings/ # PNGs of openings (deprecated)
69 | ├── debug/ # various logs served by easter egg secrets page
70 |
71 |
72 |
73 |
74 | ## Contributing
75 |
76 | Please read [CONTRIBUTING](CONTRIBUTING.md) for details on our code of conduct, and the process for submitting pull requests.
77 |
78 | ## License
79 |
80 | This project is licensed under the Apache 2 License - see [LICENSE](LICENSE) file for details
81 |
82 | ## Authors
83 |
84 | See also the list of [AUTHORS](AUTHORS) who participated in this project.
85 |
86 | ## Acknowledgments
87 |
88 | * [MiniGo](https://github.com/tensorflow/minigo)
89 | * Andrew Jackson for his infinite patience with my questions
90 |
91 | ## Links
92 |
93 | * [MiniGo](https://github.com/tensorflow/minigo)
94 | website: [CloudyGo.com](http://CloudyGo.com)
95 | * [Leela-Zero](https://github.com/gcp/leela-zero)
96 | website: [sjeng.org](http://zero.sjeng.org/home)
97 | * [Leela Chess Zero](https://github.com/LeelaChessZero/lc0)
98 | website: [lczero.org](https://lczero.org/)
99 |
100 |
--------------------------------------------------------------------------------
/oneoff/build_example_instance.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python3
2 | #
3 | # Copyright 2018 Google LLC
4 | #
5 | # Licensed under the Apache License, Version 2.0 (the "License");
6 | # you may not use this file except in compliance with the License.
7 | # You may obtain a copy of the License at
8 | #
9 | # http://www.apache.org/licenses/LICENSE-2.0
10 | #
11 | # Unless required by applicable law or agreed to in writing, software
12 | # distributed under the License is distributed on an "AS IS" BASIS,
13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14 | # See the License for the specific language governing permissions and
15 | # limitations under the License.
16 |
17 | import sys
18 | sys.path.insert(0, '.')
19 |
20 | import os
21 | import random
22 | import re
23 | import shutil
24 |
25 | from tqdm import tqdm
26 |
27 | GAMES = 25
28 |
29 | #RUN = 'v3-9x9'
30 | #MODELS = range(290, 310+1)
31 |
32 | RUN = 'v5-19x19'
33 | MODELS = range(100, 110+1)
34 |
35 | #RUN = 'v7-19x19'
36 | #MODELS = range(180, 190+1)
37 |
38 | ROOT_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), '..')
39 | INSTANCE_DIR = os.path.join(ROOT_DIR, 'instance')
40 | DEVEL_DIR = os.path.join(ROOT_DIR, 'devel_instance')
41 |
42 |
43 | if not os.path.exists(DEVEL_DIR):
44 | print(DEVEL_DIR, "does not exist")
45 | sys.exit(2)
46 |
47 | # Source, Dest
48 | RUN_DIR = (os.path.join(INSTANCE_DIR, 'data', RUN),
49 | os.path.join(DEVEL_DIR, 'data', RUN))
50 | if not os.path.exists(os.path.join(RUN_DIR[0])):
51 | print("Run dir ({}) does not exists".format(RUN))
52 | sys.exit(2)
53 |
54 | if os.path.exists(os.path.join(RUN_DIR[1])):
55 | print("Devel run dir ({}) already exists".format(RUN))
56 | sys.exit(2)
57 |
58 | os.mkdir(RUN_DIR[1])
59 |
60 | def copy_if_match(A, B, files=None, match_model=False):
61 | # Copy files (default: all) from dir A to dir B
62 | if files is None and match_model is None:
63 | assert False, "Use shutil.copytree"
64 |
65 | if not os.path.exists(A):
66 | print ("{} Does not exist, skipping".format(A))
67 | return
68 |
69 | common = os.path.commonprefix([A, B])
70 | start = len(common)
71 | print("Moving some of {} | {} to {}".format(
72 | common, A[start:], B[start:]))
73 |
74 | if not os.path.exists(B):
75 | os.mkdir(B)
76 |
77 | if not files:
78 | files = os.listdir(A)
79 |
80 | for f in tqdm(files):
81 | path = os.path.join(A, f)
82 |
83 | if match_model:
84 | raw = re.split(r'[._-]+', os.path.basename(path))
85 | nums = [int(part) for part in raw if part.isnumeric()]
86 | for num in nums:
87 | if num in MODELS:
88 | break
89 | else:
90 | continue
91 |
92 | shutil.copy2(path, os.path.join(B, f))
93 |
94 |
95 | # Copy all eval games (which match a model num)
96 | def eval_match_model_both(e):
97 | parts = re.split(r'[._-]+', os.path.basename(e))
98 | return all(not p.isnumeric() or len(p) != 6 or int(p) in MODELS
99 | for p in parts)
100 |
101 | EVAL_DIR = tuple(os.path.join(run, "eval") for run in RUN_DIR)
102 | eval_files = filter(eval_match_model_both, os.listdir(EVAL_DIR[0]))
103 | copy_if_match(EVAL_DIR[0], EVAL_DIR[1], files=eval_files)
104 | print("eval copied")
105 |
106 |
107 | # Model dir
108 | MODEL_DIR = tuple(os.path.join(run, "models") for run in RUN_DIR)
109 | os.mkdir(MODEL_DIR[1])
110 | for f in os.listdir(MODEL_DIR[0]):
111 | try:
112 | m = int(f.split('-')[0])
113 | if m in MODELS:
114 | new_f = os.path.join(MODEL_DIR[1], f)
115 | with open(new_f, 'w'):
116 | pass
117 | # maybe should set utime
118 | except:
119 | pass
120 | print("models copied")
121 |
122 |
123 | # SGF dir
124 | SGF_DIR = tuple(os.path.join(run, "sgf") for run in RUN_DIR)
125 | os.mkdir(SGF_DIR[1])
126 | for m in sorted(os.listdir(SGF_DIR[0])):
127 | num = int(m.split('-')[0])
128 | if num not in MODELS:
129 | continue
130 |
131 | model_dir = tuple(os.path.join(sgf, m) for sgf in SGF_DIR)
132 | os.mkdir(model_dir[1])
133 |
134 | full_dir = [os.path.join(model, 'full') for model in model_dir]
135 | clean_dir = [os.path.join(model, 'clean') for model in model_dir]
136 | if not os.path.exists(full_dir[0]):
137 | continue
138 |
139 | games = os.listdir(full_dir[0])
140 | games = random.sample(games, min(len(games), GAMES))
141 | copy_if_match(full_dir[0], full_dir[1], files=games)
142 | copy_if_match(clean_dir[0], clean_dir[1], files=games)
143 | print("sgfs copied")
144 |
145 | # MiniGo/oneoff dirs
146 | for oneoff in ["eval", "policy", "positions", "pv", "eval"]:
147 | match_model = oneoff in ("policy", "pv")
148 |
149 | src = os.path.join(INSTANCE_DIR, oneoff, RUN)
150 | dst = os.path.join(DEVEL_DIR, oneoff, RUN)
151 | copy_if_match(src, dst, match_model=match_model)
152 |
--------------------------------------------------------------------------------
/oneoff/convert_to_lz.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python3
2 | #
3 | # Copyright 2018 Google LLC
4 | #
5 | # Licensed under the Apache License, Version 2.0 (the "License");
6 | # you may not use this file except in compliance with the License.
7 | # You may obtain a copy of the License at
8 | #
9 | # http://www.apache.org/licenses/LICENSE-2.0
10 | #
11 | # Unless required by applicable law or agreed to in writing, software
12 | # distributed under the License is distributed on an "AS IS" BASIS,
13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14 | # See the License for the specific language governing permissions and
15 | # limitations under the License.
16 |
17 | import sys
18 | sys.path.insert(0, '.')
19 |
20 | import os
21 | import subprocess
22 | from subprocess import PIPE
23 |
24 | import sqlite3
25 | from tqdm import tqdm
26 |
27 | from web.cloudygo import CloudyGo
28 |
29 | ROOT_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), '..')
30 | INSTANCE_PATH = os.path.join(ROOT_DIR, 'instance')
31 | DATABASE_PATH = os.path.join(INSTANCE_PATH, 'clouds.db')
32 |
33 | BUCKET = CloudyGo.DEFAULT_BUCKET
34 | MODEL_PATH = os.path.join(ROOT_DIR, 'instance', 'data', BUCKET, 'models')
35 |
36 | SUFFIX = "_converted.txt.gz"
37 |
38 | if not os.path.isdir(MODEL_PATH):
39 | print ("models dir doesn't exist for", BUCKET)
40 | sys.exit(1)
41 |
42 | #### DB STUFF ####
43 |
44 | db = sqlite3.connect(DATABASE_PATH)
45 | db.row_factory = sqlite3.Row
46 |
47 | model_range = CloudyGo.bucket_model_range(BUCKET)
48 | cur = db.execute('SELECT model_id_1 % 1000000 '
49 | 'FROM eval_models '
50 | 'WHERE model_id_2 == 0 AND model_id_1 BETWEEN ? and ? '
51 | 'ORDER BY rankings desc '
52 | 'LIMIT 2',
53 | model_range)
54 |
55 | best_models = [r[0] for r in cur.fetchall()]
56 | print("Best Models:", best_models)
57 |
58 |
59 | files = os.listdir(MODEL_PATH)
60 | models = [f.replace('.meta', '') for f in files if f.endswith('.meta')]
61 | converted = [f.replace(SUFFIX, '') for f in files if f.endswith(SUFFIX)]
62 | print("Converted:", converted)
63 |
64 | for model in tqdm(sorted(models)):
65 | num = int(model.split('-')[0])
66 | if num == 0 or model in converted:
67 | continue
68 | if num % 100 == 0 or num in best_models:
69 | model_path = os.path.join(MODEL_PATH, model)
70 |
71 | lz_path = os.path.join(ROOT_DIR, '..', 'leela-zero', 'training',
72 | 'minigo', 'convert_minigo.py')
73 | print (num, model)
74 | output = subprocess.run([lz_path, model_path],
75 | stdout=PIPE, stderr=PIPE)
76 | stdout = output.stdout.decode('utf-8')
77 | print (output)
78 | print (stdout)
79 | print ()
80 |
--------------------------------------------------------------------------------
/oneoff/delete_bucket.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python3
2 | #
3 | # Copyright 2018 Google LLC
4 | #
5 | # Licensed under the Apache License, Version 2.0 (the "License");
6 | # you may not use this file except in compliance with the License.
7 | # You may obtain a copy of the License at
8 | #
9 | # http://www.apache.org/licenses/LICENSE-2.0
10 | #
11 | # Unless required by applicable law or agreed to in writing, software
12 | # distributed under the License is distributed on an "AS IS" BASIS,
13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14 | # See the License for the specific language governing permissions and
15 | # limitations under the License.
16 |
17 | import sqlite3
18 | import sys
19 | import time
20 |
21 |
22 | def main(argv):
23 | if len(argv) != 2:
24 | print("Usage: delete_bucket.py bucket")
25 | sys.exit(1)
26 |
27 | db = sqlite3.connect("instance/clouds.db")
28 | db.row_factory = sqlite3.Row
29 |
30 | def query_db(query, args=()):
31 | cur = db.execute(query, args)
32 | rv = cur.fetchall()
33 | cur.close()
34 | return list(map(tuple, rv))
35 |
36 | buckets = query_db("select * from bucket_model_range")
37 | names = [b[0] for b in buckets]
38 | rm_bucket = argv[1]
39 | print("Buckets:", ", ".join(names))
40 | print("Removing:", rm_bucket)
41 | print()
42 | assert rm_bucket in names
43 | m_low, m_high = min([b[1:] for b in buckets if b[0] == rm_bucket])
44 |
45 | tables = query_db("select name from sqlite_master where type = 'table'")
46 | tables = [t[0] for t in tables]
47 | print("Tables:", ", ".join(tables))
48 |
49 | tables_model = ["model_stats", "models", "position_eval_part", "name_to_model_id", "games"]
50 | tables_model_1 = ["eval_models", "eval_games", "bucket_model_range"]
51 |
52 | total_count = 0
53 | for table in tables_model + tables_model_1:
54 | query = "select count(*) from {} where {} between ? and ?".format(
55 | table,
56 | "model_id" if table in tables_model else "model_id_1")
57 | count = query_db(query, (m_low, m_high))[0][0]
58 | if count > 0:
59 | total_count += count
60 | print("Table: {:15} Rows: {:8} Total Rows: {}".format(
61 | table, count, total_count))
62 | print()
63 |
64 | if total_count > 0:
65 | test = input("type: \"delete_{}\" to delete: ".format(rm_bucket))
66 | if test == "delete_" + rm_bucket:
67 | print("Deleting in 3s")
68 | time.sleep(3)
69 | print()
70 | for table in tables_model + tables_model_1:
71 | query = "delete from {} where {} between ? and ?".format(
72 | table,
73 | "model_id" if table in tables_model else "model_id_1")
74 | query_db(query, (m_low, m_high))
75 | print("Deleted")
76 | db.commit()
77 |
78 | if __name__ == "__main__":
79 | main(sys.argv)
80 |
--------------------------------------------------------------------------------
/oneoff/early_move_canonical.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python3
2 | #
3 | # Copyright 2018 Google LLC
4 | #
5 | # Licensed under the Apache License, Version 2.0 (the "License");
6 | # you may not use this file except in compliance with the License.
7 | # You may obtain a copy of the License at
8 | #
9 | # http://www.apache.org/licenses/LICENSE-2.0
10 | #
11 | # Unless required by applicable law or agreed to in writing, software
12 | # distributed under the License is distributed on an "AS IS" BASIS,
13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14 | # See the License for the specific language governing permissions and
15 | # limitations under the License.
16 |
17 | import os
18 | import sqlite3
19 | import time
20 |
21 | from tqdm import tqdm
22 |
23 | from web import sgf_utils
24 |
25 | ROOT_DIR = os.path.dirname(os.path.abspath(__file__))
26 | INSTANCE_PATH = os.path.join(ROOT_DIR, 'instance')
27 | DATABASE_PATH = os.path.join(INSTANCE_PATH, 'clouds.db')
28 |
29 | BOARD_SIZE = 19
30 |
31 | T0 = time.time()
32 |
33 | #### DB STUFF ####
34 |
35 | db = sqlite3.connect(DATABASE_PATH)
36 | db.row_factory = sqlite3.Row
37 |
38 | cur = db.execute('SELECT model_id, bucket FROM models')
39 | model_buckets = dict((map(tuple, cur.fetchall())))
40 |
41 | cur = db.execute(
42 | 'SELECT timestamp, game_num, model_id, early_moves, early_moves_canonical FROM games')
43 | rows = list(map(tuple, cur.fetchall()))
44 | cur.close()
45 |
46 | ####
47 |
48 | print('Got {} games'.format(len(rows)))
49 | equal = 0
50 | mismatch = 0
51 | results = []
52 | for ts, game_num, model_id, raw_moves, saved_canonical in tqdm(rows, unit="game"):
53 | bucket = model_buckets[model_id]
54 | board_size = 9 if '9x9' in bucket else 19
55 |
56 | canonical = sgf_utils.canonical_moves(board_size, raw_moves)
57 | if canonical == raw_moves:
58 | equal += 1
59 | if canonical != saved_canonical:
60 | if saved_canonical not in (None, ""):
61 | mismatch += 1
62 | # print ("{} => {} did not match saved {}".format(
63 | # raw_moves, canonical, saved_canonical))
64 |
65 | results.append((canonical, ts, game_num))
66 |
67 | T1 = time.time()
68 | print('Move canonical(ization) took: {:.1f} seconds'.format(T1 - T0))
69 |
70 | cur = db.executemany(
71 | 'UPDATE games SET early_moves_canonical = ? WHERE timestamp = ? AND game_num = ?',
72 | (results))
73 | db.commit()
74 |
75 | T2 = time.time()
76 |
77 | print('{} rows updated, {} canonical is same as raw, {} mismatch'.format(
78 | cur.rowcount, equal, mismatch))
79 | print('Update took: {:.1f} seconds'.format(T2 - T1))
80 |
--------------------------------------------------------------------------------
/oneoff/game_num_fixer.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python3
2 | #
3 | # Copyright 2018 Google LLC
4 | #
5 | # Licensed under the Apache License, Version 2.0 (the "License");
6 | # you may not use this file except in compliance with the License.
7 | # You may obtain a copy of the License at
8 | #
9 | # http://www.apache.org/licenses/LICENSE-2.0
10 | #
11 | # Unless required by applicable law or agreed to in writing, software
12 | # distributed under the License is distributed on an "AS IS" BASIS,
13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14 | # See the License for the specific language governing permissions and
15 | # limitations under the License.
16 |
17 | import sys
18 | sys.path.insert(0, '.')
19 |
20 | import os
21 | import sqlite3
22 | import time
23 |
24 | from tqdm import tqdm
25 |
26 | from web.cloudygo import CloudyGo
27 |
28 | ROOT_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), '..')
29 | INSTANCE_PATH = os.path.join(ROOT_DIR, 'instance')
30 | DATABASE_PATH = os.path.join(INSTANCE_PATH, 'clouds.db')
31 |
32 | BOARD_SIZE = 19
33 |
34 | #### DB STUFF ####
35 |
36 | db = sqlite3.connect(DATABASE_PATH)
37 | db.row_factory = sqlite3.Row
38 |
39 | cur = db.execute('SELECT model_id, bucket FROM models')
40 | model_buckets = dict((map(tuple, cur.fetchall())))
41 | bucket_salts = {k: CloudyGo.bucket_salt(v) for k, v in model_buckets.items()}
42 |
43 | T0 = time.time()
44 |
45 | cur = db.execute('SELECT game_num, filename, model_id FROM games')
46 | rows = list(map(tuple, cur.fetchall()))
47 | cur.close()
48 |
49 | ####
50 |
51 | print('Got {} games'.format(len(rows)))
52 | equal = 0
53 | mismatch = 0
54 | results = []
55 | for game_num, filename, model_id in tqdm(rows, unit="game"):
56 | bucket_salt = bucket_salts[model_id]
57 |
58 | canonical = CloudyGo.get_game_num(bucket_salt, filename)
59 | if canonical != game_num:
60 | mismatch += 1
61 | results.append((canonical, game_num))
62 | else:
63 | equal += 1
64 |
65 | T1 = time.time()
66 | print('Game Num Fixer took: {:.1f} seconds'.format(T1 - T0))
67 |
68 | print("\t", results[:5])
69 |
70 | cur = db.executemany(
71 | 'UPDATE games SET game_num = ? WHERE game_num = ?',
72 | results)
73 | db.commit()
74 |
75 | T2 = time.time()
76 |
77 | print('{} rows updated, {} canonical is same as raw, {} mismatch'.format(
78 | cur.rowcount, equal, mismatch))
79 | print('Update took: {:.1f} seconds'.format(T2 - T1))
80 |
--------------------------------------------------------------------------------
/oneoff/katago-importer.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env bash
2 | #
3 | # Copyright 2019 Google LLC
4 | #
5 | # Licensed under the Apache License, Version 2.0 (the "License");
6 | # you may not use this file except in compliance with the License.
7 | # You may obtain a copy of the License at
8 | #
9 | # http://www.apache.org/licenses/LICENSE-2.0
10 | #
11 | # Unless required by applicable law or agreed to in writing, software
12 | # distributed under the License is distributed on an "AS IS" BASIS,
13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14 | # See the License for the specific language governing permissions and
15 | # limitations under the License.
16 |
17 | # create get download.sh
18 | # $ cat zips.txt | grep -o 'b.*' | xargs -I {} echo '[ -f "{}" ] || (wget https://d3dndmfyhecmj0.cloudfront.net/g104/selfplay/{} && sleep 3)' > download.sh
19 | # $ chmod a+x download.sh
20 | # $ cd zips
21 | # $ ../download.sh
22 | # unextract zips
23 | # $ ls *.zip | xargs -I {} unzip {} -x "*/tdata/*" "*/vdata/*"
24 |
25 |
26 | # TODO KataGo-104? KataGo-g65? KataGo-V1? KataGo-2019_1?
27 |
28 | import os
29 | import re
30 | import zlib
31 | from collections import defaultdict, Counter
32 |
33 | from tqdm import tqdm
34 |
35 |
36 | PB_RE = re.compile('PB\[[^]]*]')
37 |
38 |
39 | def consistent_hash(string):
40 | return zlib.adler32(string.encode('utf-8'))
41 |
42 |
43 | def touch_utime(path, epoch):
44 | if not os.path.exists(path):
45 | open(path, 'a').close()
46 | os.utime(path, (epoch, epoch))
47 |
48 |
49 | def extract_model_sgfs_to_folders(bucket, bucket_num, dest):
50 | bucket_dir = os.path.join('instance', 'data', bucket)
51 | models_dir = os.path.join(bucket_dir, 'models')
52 | inserts_path = os.path.join(bucket_dir, 'inserts.csv')
53 |
54 | models = []
55 | game_num = 1
56 | player_names = defaultdict(Counter)
57 |
58 | for model_name in sorted(os.listdir(os.path.join(bucket_dir, 'zips'))):
59 | model_name_short = model_name.rsplit('-', 1)[0]
60 | print(f'model_name: {model_name_short}')
61 |
62 | sgf_dir = os.path.join(bucket_dir, 'zips', model_name, 'sgfs')
63 | if not os.path.isdir(sgf_dir): continue
64 |
65 | dest_dir = os.path.join(bucket_dir, dest, model_name_short)
66 | existed = os.path.isdir(dest_dir)
67 | if not existed: os.mkdir(dest_dir)
68 |
69 | model_games = 0
70 | for combined_f in os.listdir(sgf_dir):
71 | assert combined_f.endswith('.sgfs'), combined_f
72 | combined_f = os.path.join(sgf_dir, combined_f)
73 |
74 | # We set the model time to the mtime of a random .sgfs file.
75 | epoch = int(os.path.getmtime(combined_f))
76 |
77 | with open(combined_f) as sgf_file:
78 | for line in tqdm(sgf_file.readlines()):
79 | player = PB_RE.search(line)
80 | assert player, line
81 | player = player.group(0)
82 | player_names[model_name_short][player] += 1
83 |
84 | if not existed:
85 | game_path = os.path.join(dest_dir, 'KataGo-{:08d}.sgf'.format(game_num))
86 | with open(game_path, 'w') as sgf_f:
87 | sgf_f.write(line)
88 | game_num += 1
89 | model_games += 1
90 |
91 | for player, count in player_names[model_name_short].most_common():
92 | print(f'\t{player} x{count} sgfs')
93 |
94 | ##### Model Data #####
95 | full_name = model_name
96 | name = model_name_short
97 | print (name)
98 | network_size = re.search(r'b[0-9]+c', name).group()[1:-1]
99 | network_blocks = re.search(r'c[0-9]+\-', name).group()[1:-1]
100 | network_steps = int(re.search(r's[0-9]+$', name).group()[1:-1])
101 |
102 | display_name = 'KataGo-{}'.format(name)
103 |
104 | fpath = os.path.join(models_dir, full_name)
105 | touch_utime(fpath, epoch)
106 |
107 | dpath = os.path.join(models_dir, display_name)
108 | touch_utime(dpath, epoch)
109 |
110 | models.append([
111 | -1, # model_id
112 | display_name, # display_name
113 | full_name, # name in models dir
114 | full_name, # name in sgf files
115 | bucket,
116 | -1, # num
117 | epoch, epoch,
118 | network_blocks, # training_time_m (being abused)
119 | model_games,
120 | 0, # num_stats_games
121 | network_steps, # num_eval_games
122 | ])
123 |
124 | print('\n')
125 | all_players = sum(player_names.values(), Counter())
126 | for player, count in all_players.most_common():
127 | print(f'\t{player} x{count} sgfs')
128 |
129 | print('Sorted')
130 | for player, count in sorted(all_players.items()):
131 | print(f'\t{player} x{count} sgfs')
132 |
133 | with open(inserts_path, 'w') as inserts_f:
134 | models.sort(key=lambda m: m[11])
135 | for m_i, model in enumerate(models):
136 | model[0] = bucket_num * 10 ** 6 + m_i
137 | model[5] = m_i
138 | model[11] = 0
139 |
140 | row = ','.join(map(str, model))
141 | inserts_f.write(row + '\n')
142 | print(row)
143 |
144 | print()
145 | print('sqlite3 instance/clouds.db',
146 | '".mode csv"',
147 | '".import instance/data/' + bucket + '/inserts.csv models"')
148 |
149 |
150 | if __name__ == '__main__':
151 | bucket = 'KataGo'
152 | bucket_num = consistent_hash(bucket) % 100
153 |
154 | print('bucket: {}, num: {}'.format(bucket, bucket_num))
155 | print()
156 | print('Self Play')
157 | extract_model_sgfs_to_folders(bucket, bucket_num, 'sgf')
158 |
--------------------------------------------------------------------------------
/oneoff/leela-all-to-dirs.sh:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env bash
2 | #
3 | # Copyright 2018 Google LLC
4 | #
5 | # Licensed under the Apache License, Version 2.0 (the "License");
6 | # you may not use this file except in compliance with the License.
7 | # You may obtain a copy of the License at
8 | #
9 | # http://www.apache.org/licenses/LICENSE-2.0
10 | #
11 | # Unless required by applicable law or agreed to in writing, software
12 | # distributed under the License is distributed on an "AS IS" BASIS,
13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14 | # See the License for the specific language governing permissions and
15 | # limitations under the License.
16 |
17 | set -euo pipefail
18 |
19 | bucket="leela-zero"
20 |
21 | cd "instance/data/$bucket"
22 |
23 | PER_FOLDER=5000
24 |
25 | extract_sgf_to_folder() {
26 | # eval, sgf, ...
27 | folder_name="$1"
28 | combined="$2"
29 | echo "Extracting \"$combined\" to \"$folder_name\""
30 |
31 | mkdir -p "$folder_name"
32 | cd "$folder_name"
33 |
34 | # files_list="$(ls)"
35 | # num_files=$(echo "$files_list" | wc -l)
36 | num_files=12180000
37 | if [ "$num_files" -le 1 ]; then
38 | echo "Splitting \"$combined\" ($(du -h "$combined" | cut -f1))"
39 | /usr/bin/time -f "Splitting took %e seconds" \
40 | sgfsplit -d8 -x "${bucket}-" "$combined"
41 | files_list="$(ls)"
42 | num_files=$(echo "$files_list" | wc -l)
43 | fi
44 |
45 | if [ "$num_files" -ge 20000 ]; then
46 | # echo "Found $num_files files"
47 | tmp_file_list="../${folder_name}_files_tmp"
48 | # echo "$files_list" > "$tmp_file_list"
49 |
50 | # assumes files_list is sorted and no directories exist yet
51 | first=$(head -n 1 $tmp_file_list | grep -o '[1-9][0-9]*')
52 | last=$(tail -n 1 $tmp_file_list | grep -o '[0-9]*')
53 |
54 | # TODO do something if first not a multiple of $PER_FOLDER
55 |
56 | # Create folder per X thousand
57 | first_folder=$(($first/$PER_FOLDER*$PER_FOLDER))
58 | echo "$first to $last, starting with $first_folder"
59 |
60 | for lower in $(seq $first_folder $PER_FOLDER $last); do
61 | echo "$lower" # so tqdm can track progress
62 | mkdir -p $lower
63 | set +e
64 | seq -f "${bucket}-%08.0f.sgf" $lower $(($lower + $PER_FOLDER - 1)) | xargs --no-run-if-empty mv -t $lower
65 | set -e
66 | done | tqdm --desc "Seperating" --total $(($num_files / $PER_FOLDER)) | wc
67 | # echo "Done seperating"
68 | fi
69 | cd ..
70 | }
71 |
72 | save_player_info() {
73 | # eval, sgf, ...
74 | folder_name="$1"
75 |
76 | # Create a list of files to player names
77 | player_file="${folder_name}_file_to_player"
78 | if [ ! -f "$player_file" ]; then
79 | echo "Saving player names to \"$player_file\""
80 | # This sort is only pseudo good"
81 | find "$folder_name" -type f -name '*.sgf' \
82 | | sort \
83 | | xargs grep -H -m1 -o 'PB\[[^]]*\]PW\[[^]]*\]' \
84 | | tqdm --desc "PB/PW lookup" > "$player_file"
85 | fi
86 |
87 | players() {
88 | cat "$player_file" | cut -d':' -f2-
89 | }
90 | group() {
91 | sort | uniq -c | sort -n
92 | }
93 |
94 | player_combos=$(players | group | wc -l)
95 | unique_player_combos=$(players | sed 's/Leela\s*Zero\s*[0-9]\+\(\.[0-9]\+\)*\s\+//g' | group | wc -l)
96 | echo "Player pairs: $player_combos, $unique_player_combos unique"
97 | echo
98 | }
99 |
100 | # Extract all_match.sgf to "data"
101 | ALL_MATCH_PATH="/media/eights/big-ssd/rsync/all_match.sgf"
102 | ALL_SGF_PATH="/media/eights/big-ssd/rsync/all_fixed.sgf"
103 |
104 | #echo "Eval"
105 | #extract_sgf_to_folder "eval" "$ALL_MATCH_PATH"
106 | #save_player_info "eval"
107 |
108 | echo "Self Play"
109 | extract_sgf_to_folder "sgf" "$ALL_SGF_PATH"
110 | save_player_info "sgf"
111 |
--------------------------------------------------------------------------------
/oneoff/leela-model-importer.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/python3
2 | #
3 | # Copyright 2018 Google LLC
4 | #
5 | # Licensed under the Apache License, Version 2.0 (the "License");
6 | # you may not use this file except in compliance with the License.
7 | # You may obtain a copy of the License at
8 | #
9 | # http://www.apache.org/licenses/LICENSE-2.0
10 | #
11 | # Unless required by applicable law or agreed to in writing, software
12 | # distributed under the License is distributed on an "AS IS" BASIS,
13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14 | # See the License for the specific language governing permissions and
15 | # limitations under the License.
16 |
17 | import datetime
18 | import os
19 | import re
20 | import requests
21 | import sys
22 |
23 | import zlib
24 |
25 | def consistent_hash(string):
26 | return zlib.adler32(string.encode('utf-8'))
27 |
28 |
29 | #### MAIN ####
30 |
31 |
32 | BUCKET = 'leela-zero'
33 | if len(sys.argv) > 1 and sys.argv[1].startswith('leela'):
34 | BUCKET = sys.argv[1]
35 |
36 | BUCKET_NUM = consistent_hash(BUCKET) % 100
37 |
38 | print ("BUCKET: {}, NUM: {}".format(BUCKET, BUCKET_NUM))
39 |
40 | URL = 'http://zero.sjeng.org'
41 | MODEL_DIR = os.path.join('instance', 'data', BUCKET, 'models')
42 | FILE = os.path.join('instance', 'data', BUCKET, 'zero-sjeng-org.html')
43 | INSERTS = os.path.join('instance', 'data', BUCKET, 'inserts.csv')
44 | DOWNLOADER = os.path.join('instance', 'data', BUCKET, 'download.sh')
45 |
46 | data = requests.get(URL).content;
47 | with open(FILE, "wb") as f: f.write(data)
48 | with open(FILE) as f: data = f.read()
49 |
50 | with open(INSERTS, 'w') as inserts, open(DOWNLOADER, 'w') as downloader:
51 | for num in range(1000):
52 | # check if we can find num in the model table
53 | match = re.search('
(
{}
.*)
'.format(num), data)
54 | if not match:
55 | print('Did not find {} stopping'.format(num))
56 | break
57 |
58 | parts = match.group(1).replace('', '').split('
')
59 | assert parts[1] == str(num), parts
60 |
61 | full_name = re.search(r'([0-9a-f]{64})', parts[3]).group()
62 | name = parts[3][-12:-4]
63 | games = int(parts[6])
64 | date = parts[2]
65 | date = datetime.datetime.strptime(date, '%Y-%m-%d %H:%M')
66 | epoch = int(date.strftime('%s'))
67 | network_size = parts[4]
68 | network_blocks = int(network_size.split('x')[0])
69 |
70 | display_name = 'LZ{}_{}'.format(num, name)
71 |
72 | def touch_utime(path, epoch):
73 | if not os.path.exists(path):
74 | open(path, 'a').close()
75 | os.utime(path, (epoch, epoch))
76 |
77 | fpath = os.path.join(MODEL_DIR, full_name)
78 | touch_utime(fpath, epoch)
79 |
80 | dpath = os.path.join(MODEL_DIR, display_name)
81 | touch_utime(dpath, epoch)
82 |
83 | row = ",".join(map(str, (
84 | BUCKET_NUM * 10 ** 6 + num,
85 | display_name, # display_name
86 | full_name, # name in models dir
87 | full_name, # name in sgf files
88 | BUCKET, num,
89 | epoch, epoch,
90 | network_blocks, # training_time_m (being abused)
91 | games,
92 | 0, # num_stats_games
93 | 0))) # num_eval_games
94 | inserts.write(row + '\n')
95 |
96 | model_path = 'models/' + display_name
97 | download_command = (
98 | '[ -f {} ] || (wget {}/networks/{}.gz -O {} && sleep 10)'.format(
99 | model_path, URL, full_name, model_path))
100 | downloader.write(download_command + '\n')
101 | print(row)
102 |
103 |
104 | commands = [
105 | ".mode csv",
106 | ".import instance/data/" + BUCKET + "/inserts.csv models",
107 | ]
108 | print()
109 | print('sqlite3 instance/clouds.db', ' '.join(map('"{}"'.format, commands)))
110 |
--------------------------------------------------------------------------------
/oneoff/leela-process.md:
--------------------------------------------------------------------------------
1 | # Downloaded
2 | `all_match.sgf.xz`
3 | `all.sgf.xz`
4 |
5 | ```sh
6 | ./oneoff/delete_bucket.py leela-zero
7 | ./oneoff/setup-run.sh leela-zero
8 | ```
9 |
10 | ```sh
11 | ./oneoff/leela-model-importer.py leela-zero
12 | sqlite3 instance/clouds.db ".mode csv" ".import instance/data/leela-zero/inserts.csv models"
13 | ```
14 |
15 | ```sh
16 | ./oneoff/leela-all-to-dirs.sh
17 | ```
18 |
19 | TODO hashing games for consistent urls or redo each game
20 |
21 | ## Import into cloudygo!
22 | ```sh
23 | ./updater.py models leela-zero
24 | ./updater.py eval_games leela-zero
25 | ./updater.py games leela-zero
26 | ```
27 |
28 |
29 | # Eval Process
30 | ```sh
31 | cd instance/data/leela-zero
32 | # (manually delete LZXXX_0123ABDC) for new models)
33 | ./download.sh
34 | ```
35 |
36 | # CloudyGo.com specific
37 | ```sh
38 | rsync -tr --info=progress2 "~/Projects/cloudygo/instance/data/leela-zero/models/LZ*" $OTHER_PC:~/Projects/ringmaster/leela-zero-v3/
39 | ssh $OTHER_PC
40 | cd Projects/ringmaster/
41 | screen
42 | export CTL=leela-zero-v3-eval.ctl; date; (ls "${CTL%.ctl}.games/" | wc -l); ringmaster "$CTL" show | grep -o '[0-9]*/\?[0-9]* games' | sort | uniq -c
43 | time ringmaster leela-zero-v3-eval.ctl run
44 | ```
45 |
46 | ```sh
47 | function sync_and_update_lz3 { rsync --info=progress2 -t -r "$OTHER_PC:~/Projects/ringmaster/leela-zero-v3-eval.games" ~/Projects/cloudygo/instance/data/leela-zero-eval/eval/; }
48 | sync_and_update_lz3
49 | ```
50 |
--------------------------------------------------------------------------------
/oneoff/leela-ringmaster-importer.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/python
2 | #
3 | # Copyright 2018 Google LLC
4 | #
5 | # Licensed under the Apache License, Version 2.0 (the "License");
6 | # you may not use this file except in compliance with the License.
7 | # You may obtain a copy of the License at
8 | #
9 | # http://www.apache.org/licenses/LICENSE-2.0
10 | #
11 | # Unless required by applicable law or agreed to in writing, software
12 | # distributed under the License is distributed on an "AS IS" BASIS,
13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14 | # See the License for the specific language governing permissions and
15 | # limitations under the License.
16 |
17 | import re
18 | import shlex
19 | import sys
20 | import os
21 | from collections import defaultdict
22 | from subprocess import Popen, check_output, PIPE, STDOUT
23 |
24 | timestamp = "000000" if len(sys.argv) < 2 else sys.argv[1]
25 | offset = 0 if len(sys.argv) < 3 else int(sys.argv[2])
26 |
27 | # Runs from leelaz/v1/ directory and renames import/ games to B_W_
28 |
29 |
30 | p1 = Popen(["find", "import", "-iname", "*sgf"], stdout=PIPE, stderr=STDOUT)
31 |
32 | command = 'xargs grep -o "\(Black\|White\) \([0-9a-f]\{8\}\(_fast\|_slow\)\) Leela Zero"'
33 | players = check_output(shlex.split(command, stdin=p1.stdout, stderr=PIPE)
34 |
35 |
36 | lines = players.strip().split("\n")
37 | print("Found {} lines".format(len(lines)))
38 |
39 | file_parts = defaultdict(lambda: ["", ""])
40 | for line in lines:
41 | parts = re.split(r"[ :]+", line)
42 | filename, color, net, leela, zero = parts
43 | assert leela == "Leela" and zero == "Zero", parts
44 | assert color in ("Black", "White")
45 |
46 | file_parts[parts[0]][0 if color == "Black" else 1] = net
47 |
48 | assert 2 * len(file_parts) == len(lines)
49 |
50 | for f, (b, w) in file_parts.items():
51 | name = os.path.basename(f)
52 | dirname = os.path.dirname(f)
53 | matchup, game = map(int, name.replace(".sgf", "").split("_"))
54 |
55 | slow_offset = 10 * (w+b).count("_slow")
56 | game_num = game + offset + slow_offset
57 |
58 | new_name = "-".join([timestamp, w, b, str(game_num)]) + ".sgf"
59 | new_path = os.path.join(dirname, new_name)
60 | os.rename(f, new_path)
61 |
62 | # rename 's#000-([0-9]*)-([0-9]*)#001-$2-$1#' *.sgf
63 |
--------------------------------------------------------------------------------
/oneoff/leela-zero-v3-eval.ctl:
--------------------------------------------------------------------------------
1 | stderr_to_log = True
2 |
3 | import os.path
4 | import glob
5 |
6 | BASE = "leela-zero-v3/"
7 |
8 | def LeelaPlayer(model, playouts):
9 | # Full strength LZ
10 | return Player(
11 | "./leelaz -q -g --noponder -w {} -t 1 -v {}".format(
12 | model, playouts),
13 | override_name=True)
14 |
15 | matchups = []
16 | def add_matchup(matchup_id, model_a, model_b, games):
17 | matchups.append(Matchup(
18 | model_a, model_b,
19 | id=matchup_id, number_of_games=games,
20 | alternating=True, scorer='players'))
21 |
22 | visits = [
23 | ('_rapid', 50),
24 | ('_fast', 200),
25 | ('_slow', 800),
26 | ]
27 |
28 | networks = {}
29 | players = {}
30 | for filename in sorted(glob.glob(BASE + "*")):
31 | model = os.path.basename(filename)
32 | number = int(model.split('_')[0][2:])
33 | networks[number] = model
34 |
35 | for visits_name, visits_count in visits:
36 | players[model + visits_name] = LeelaPlayer(filename, visits_count)
37 |
38 | net_diffs = [1,2,3,5,10,15,20,25,50]
39 | for m_i, model_a in sorted(networks.items()):
40 | for d in net_diffs:
41 | m_j = m_i - d
42 | model_b = networks.get(m_j)
43 | if model_b is not None:
44 | games = 10 + 6 * (m_j >= 180) + 4 * (m_j <= 30)
45 |
46 | matchup_name = 'LZ{}_vs_LZ{}'.format(m_i, m_j)
47 | for visit_name, _ in visits:
48 | add_matchup(
49 | matchup_name + visit_name,
50 | model_a + visit_name,
51 | model_b + visit_name,
52 | games + (4 if visit_name != '_slow' else 0))
53 |
54 |
--------------------------------------------------------------------------------
/oneoff/model_id_guess.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python3
2 | #
3 | # Copyright 2018 Google LLC
4 | #
5 | # Licensed under the Apache License, Version 2.0 (the "License");
6 | # you may not use this file except in compliance with the License.
7 | # You may obtain a copy of the License at
8 | #
9 | # http://www.apache.org/licenses/LICENSE-2.0
10 | #
11 | # Unless required by applicable law or agreed to in writing, software
12 | # distributed under the License is distributed on an "AS IS" BASIS,
13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14 | # See the License for the specific language governing permissions and
15 | # limitations under the License.
16 |
17 | import sys
18 | sys.path.insert(0, '.')
19 |
20 | import functools
21 | import os
22 | import sqlite3
23 | import time
24 |
25 | from tqdm import tqdm
26 |
27 | from web.cloudygo import CloudyGo
28 |
29 | ROOT_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), '..')
30 | DATABASE_PATH = os.path.join(ROOT_DIR, 'instance', 'clouds.db')
31 |
32 | BOARD_SIZE = 19
33 | BUCKET = 'v9-19x19'
34 |
35 | #### DB STUFF ####
36 |
37 | db = sqlite3.connect(DATABASE_PATH)
38 | db.row_factory = sqlite3.Row
39 |
40 | def query_db(query, args=()):
41 | cur = db.execute(query, args)
42 | data = list(map(tuple, cur.fetchall()))
43 | cur.close()
44 | return data
45 |
46 | model_range = CloudyGo.bucket_model_range(BUCKET)
47 | model_creation = query_db('SELECT model_id, creation '
48 | 'FROM models WHERE bucket = ?', (BUCKET,))
49 | model_ids, creation_times = zip(*model_creation)
50 | model_guesser = functools.partial(
51 | CloudyGo._model_guesser,
52 | model_mtimes=creation_times,
53 | model_ids=model_ids)
54 |
55 | T0 = time.time()
56 |
57 | game_data = query_db('SELECT timestamp, game_num, filename, model_id '
58 | 'FROM games WHERE model_id BETWEEN ? AND ?', model_range)
59 | equal = 0
60 | results = []
61 | for ts, game_num, filename, model_id in tqdm(game_data, unit="game"):
62 | guess_id = model_guesser(filename)
63 |
64 | if guess_id == model_id:
65 | equal += 1
66 | else:
67 | results.append((guess_id, ts, game_num))
68 |
69 | T1 = time.time()
70 | print('Model Id Guess took: {:.1f} seconds'.format(T1 - T0))
71 |
72 | print("\t", results[:3], results[-3:])
73 |
74 | cur = db.executemany(
75 | 'UPDATE games SET model_id = ? WHERE timestamp = ? AND game_num = ?',
76 | results)
77 | db.commit()
78 |
79 | T2 = time.time()
80 |
81 | print('{} == {} rows updated, {} unchanged'.format(
82 | len(results), cur.rowcount, equal))
83 | print('Update took: {:.1f} seconds'.format(T2 - T1))
84 |
--------------------------------------------------------------------------------
/oneoff/photo_magic.sh:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env bash
2 | #
3 | # Copyright 2018 Google LLC
4 | #
5 | # Licensed under the Apache License, Version 2.0 (the "License");
6 | # you may not use this file except in compliance with the License.
7 | # You may obtain a copy of the License at
8 | #
9 | # http://www.apache.org/licenses/LICENSE-2.0
10 | #
11 | # Unless required by applicable law or agreed to in writing, software
12 | # distributed under the License is distributed on an "AS IS" BASIS,
13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14 | # See the License for the specific language governing permissions and
15 | # limitations under the License.
16 |
17 | # function photo { echo "$2" > $1.meta; curl "$3" > $1.jpg; }
18 |
19 | set -e
20 |
21 | cd instance/photos
22 |
23 | # pngs to jpg
24 | mogrify -format jpg *.png
25 |
26 | # Requires one of Fred's excellent ImageMagick scripts ('aspectcrop')
27 | # See: http://www.fmwconcepts.com/imagemagick/aspectcrop/index.php
28 |
29 | # Cut to the right aspect ratio (2:1)
30 | mkdir -p temp
31 | ls *jpg| xargs -I{} aspectcrop -a 2:1 -g c "{}" "temp/{}"
32 |
33 | # These don't do well with center gravity
34 | echo "andrew
35 | batavier
36 | bellerophon
37 | cormorant
38 | coronation
39 | courageux
40 | culloden
41 | duke
42 | malabar
43 | mermaid
44 | mordaunt
45 | nassau
46 | repulse
47 | royal-sovereign
48 | sans-pareil
49 | seagull
50 | sultan
51 | two-lion
52 | vanguard
53 | victory" | xargs -I{} aspectcrop -a 2:1 -g s "{}.jpg" "temp/{}.jpg"
54 | echo "unite
55 | dispatch" | xargs -I{} aspectcrop -a 2:1 -g w "{}.jpg" "temp/{}.jpg"
56 |
57 | # Downsize any really large images to ~(600x300)
58 | mkdir -p thumbs
59 | ls temp/ | xargs -I{} convert -resize "180000@>" "temp/{}" "thumbs/{}"
60 |
61 |
--------------------------------------------------------------------------------
/oneoff/repopulate_db.sh:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env bash
2 | #
3 | # Copyright 2018 Google LLC
4 | #
5 | # Licensed under the Apache License, Version 2.0 (the "License");
6 | # you may not use this file except in compliance with the License.
7 | # You may obtain a copy of the License at
8 | #
9 | # http://www.apache.org/licenses/LICENSE-2.0
10 | #
11 | # Unless required by applicable law or agreed to in writing, software
12 | # distributed under the License is distributed on an "AS IS" BASIS,
13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14 | # See the License for the specific language governing permissions and
15 | # limitations under the License.
16 |
17 | rm -f instance/clouds.db; sqlite3 instance/clouds.db < schema.sql
18 |
--------------------------------------------------------------------------------
/oneoff/setup-run.sh:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env bash
2 | #
3 | # Copyright 2018 Google LLC
4 | #
5 | # Licensed under the Apache License, Version 2.0 (the "License");
6 | # you may not use this file except in compliance with the License.
7 | # You may obtain a copy of the License at
8 | #
9 | # http://www.apache.org/licenses/LICENSE-2.0
10 | #
11 | # Unless required by applicable law or agreed to in writing, software
12 | # distributed under the License is distributed on an "AS IS" BASIS,
13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14 | # See the License for the specific language governing permissions and
15 | # limitations under the License.
16 |
17 | # Arg checking
18 | if [[ "$#" -ne 1 ]]; then
19 | echo "Need run arg"
20 | exit 2;
21 | fi
22 |
23 | if [[ "$PWD" = *oneoff* ]]; then
24 | echo "Run from CloudyGo root"
25 | exit 2;
26 | fi
27 |
28 | run=$1
29 | echo "Setting up run: \"$run\""
30 |
31 | mkdir -p "instance/data/$run"
32 | mkdir -p "instance/data/$run/models"
33 | mkdir -p "instance/data/$run/sgf"
34 | mkdir -p "instance/data/$run/eval"
35 | mkdir -p "instance/eval/$run"
36 | mkdir -p "instance/policy/$run"
37 | mkdir -p "instance/pv/$run"
38 |
--------------------------------------------------------------------------------
/oneoff/setup-runs-data.sh:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env bash
2 | #
3 | # Copyright 2018 Google LLC
4 | #
5 | # Licensed under the Apache License, Version 2.0 (the "License");
6 | # you may not use this file except in compliance with the License.
7 | # You may obtain a copy of the License at
8 | #
9 | # http://www.apache.org/licenses/LICENSE-2.0
10 | #
11 | # Unless required by applicable law or agreed to in writing, software
12 | # distributed under the License is distributed on an "AS IS" BASIS,
13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14 | # See the License for the specific language governing permissions and
15 | # limitations under the License.
16 |
17 | echo "Buckets:"
18 | sqlite3 instance/clouds.db "select bucket from bucket_model_range order by 1"
19 |
20 | echo "Found:"
21 | sqlite3 instance/clouds.db "select * from runs"
22 |
23 | echo "Inserting:"
24 | sqlite3 instance/clouds.db """
25 | INSERT INTO RUNS VALUES
26 | ('cross-eval', 'cross-eval', 'Unsorted eval game',
27 | 0, 0),
28 | ('cross-run-eval', 'cross-run-eval', 'Eval games between minigo runs',
29 | 0, 0),
30 | ('leela-zero', 'leela-zero', 'Leela-Zero (games and eval)',
31 | 0, 0),
32 | ('leela-zero-eval', 'leela-zero eval', 'Leela-Zero (eval with more cross play)',
33 | 0, 0),
34 | ('leela-zero-eval-time', 'leela-zero eval with even time', 'Leela-Zero (eval on equal time)',
35 | 0, 0),
36 | ('KataGo', 'KataGo', 'KataGo g104',
37 | 20, 256);
38 | ('synced-eval', 'synced-eval', 'Eval games between minigo runs',
39 | 0, 0),
40 | ('v3-9x9', 'v3', 'Old 9x9 Run',
41 | 10, 32),
42 | ('v5-19x19', 'v5', 'First 19x19 Run',
43 | 20, 128),
44 | ('v7-19x19', 'v7', '',
45 | 20, 128),
46 | ('v9-19x19', 'v9', '',
47 | 20, 256),
48 | ('v10-19x19', 'v10', '',
49 | 20, 256),
50 | ('v11-19x19', 'v11', 'Experiment: Q=draw',
51 | 20, 256),
52 | ('v12-19x19', 'v12', 'Experiment: BatchSize=2',
53 | 20, 256),
54 | ('v13-19x19', 'v13', 'Human Bootstrap',
55 | 21, 256),
56 | ('v14-19x19', 'v14', 'Q=loss + Bigtable',
57 | 20, 256),
58 | ('v15-19x19', 'v15', 'Q=loss',
59 | 20, 256),
60 | ('v16-19x19', 'v16', 'First 40 block run',
61 | 40, 256),
62 | ('v17-19x19', 'v17', 'Squeeze and Exicitation',
63 | 20, 256);
64 | """
65 |
--------------------------------------------------------------------------------
/oneoff/sgfstrip-eval.sh:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env bash
2 | #
3 | # Copyright 2018 Google LLC
4 | #
5 | # Licensed under the Apache License, Version 2.0 (the "License");
6 | # you may not use this file except in compliance with the License.
7 | # You may obtain a copy of the License at
8 | #
9 | # http://www.apache.org/licenses/LICENSE-2.0
10 | #
11 | # Unless required by applicable law or agreed to in writing, software
12 | # distributed under the License is distributed on an "AS IS" BASIS,
13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14 | # See the License for the specific language governing permissions and
15 | # limitations under the License.
16 |
17 | set -euo pipefail
18 |
19 | # find 000* -type f -path '*/clean/*sgf' | sort | tqdm | tar cjf 'clean.tar.bz2' -T -
20 |
21 | # Arg checking
22 | if [[ "$#" -ne 1 ]]; then
23 | echo "Need run arg"
24 | exit 2;
25 | fi
26 |
27 | if [[ "$PWD" = *oneoff* ]]; then
28 | echo "Run from CloudyGo root"
29 | exit 2;
30 | fi
31 |
32 | run=$1
33 | echo "SGFStrip eval run: \"$run\""
34 |
35 | if [[ ! -d "instance/data/$run/eval/" ]]; then
36 | echo "No eval for $run"
37 | exit 2;
38 | fi
39 |
40 | #if [[ -d "instance/data/$run/eval_clean/" ]]; then
41 | # echo "eval clean already exists for $run"
42 | # exit 2;
43 | #fi
44 |
45 | cd "instance/data/$run"
46 |
47 | echo "Recreate directory structure"
48 | time find -L eval -type d -printf "%P\0" | xargs -0 -I{} mkdir -p eval_clean/{}
49 |
50 | echo "Running SGFstrip"
51 | total_sgfs=$(find -L eval -type f -iname '*.sgf' | wc -l)
52 | time find -L eval -type f -iname '*.sgf' -printf "%P\0" \
53 | | tqdm --delim '\0' --unit "SGF" --total $total_sgfs \
54 | | xargs -0 -I{} sh -c 'sgfstrip -h C < "eval/{}" > "eval_clean/{}"'
55 |
56 |
--------------------------------------------------------------------------------
/oneoff/subsample-v3.sh:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env bash
2 | #
3 | # Copyright 2018 Google LLC
4 | #
5 | # Licensed under the Apache License, Version 2.0 (the "License");
6 | # you may not use this file except in compliance with the License.
7 | # You may obtain a copy of the License at
8 | #
9 | # http://www.apache.org/licenses/LICENSE-2.0
10 | #
11 | # Unless required by applicable law or agreed to in writing, software
12 | # distributed under the License is distributed on an "AS IS" BASIS,
13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14 | # See the License for the specific language governing permissions and
15 | # limitations under the License.
16 |
17 |
18 | #find sgf/ -path '*/full/*sgf' -type f | tqdm | xargs grep 'C\[Resign Threshold: ' > resign_thresholds
19 | RUN_DIR="../instance/data/v5-19x19"
20 | RESIGN_THRESHOLDS="0002-resign_thresholds"
21 |
22 | PERCENT=10
23 |
24 | if [[ -z "$RUN_DIR" || ! -d "$RUN_DIR" ]]; then
25 | echo "$RUN_DIR does not exist"
26 | exit 2;
27 | fi
28 |
29 | if [[ ! -f "$RUN_DIR/$RESIGN_THRESHOLDS" ]]; then
30 | echo "didn't find '$RESIGN_THRESHOLDS' file"
31 | exit 2;
32 | fi
33 |
34 | cd "$RUN_DIR"
35 | models=$(ls sgf | grep -o "00[0-9]\{4\}-[a-z-]*")
36 | model_count=$(echo "$models" | wc -l)
37 | first_model=$(echo "$models" | head -n 1)
38 | last_model=$(echo "$models" | tail -n 1)
39 | echo "Found $model_count models ($first_model to $last_model)"
40 |
41 | read -p "Do you want to subsample $RUN to 10%? \"$RUN_DIR\" y/[N]: " answer
42 | case $answer in
43 | [Yy]* ) ;;
44 | [Nn]* ) exit;;
45 | * ) echo "Defaulting Yes"; exit;;
46 | esac
47 |
48 | count=0
49 | echo "$models" | while read model;
50 | do
51 | count=$((count + 1))
52 |
53 | file_count=$(find "sgf/$model/full" -maxdepth 1 -iname "*.sgf" | wc -l)
54 | resign_files=$(grep "sgf/$model" "$RESIGN_THRESHOLDS")
55 | resign_count=$(echo "$resign_files" | wc -l)
56 |
57 | if [[ $file_count -gt 0 && $file_count -eq $resign_count ]]; then
58 | eligable=$(echo "$resign_files" | grep -v '1\.0' | grep -o '^[^:]*')
59 | eligable_count=$(echo "$eligable" | wc -l)
60 | new_count=$(echo "(0.9 * $eligable_count)/1" | bc)
61 | echo "$model contains $file_count files, $resign_count resign records, sampling $eligable_count, rm $new_count"
62 | echo "$eligable" | shuf | head -n "$new_count" | xargs rm
63 | fi
64 | done
65 |
--------------------------------------------------------------------------------
/oneoff/subsample-v9.sh:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env bash
2 | #
3 | # Copyright 2018 Google LLC
4 | #
5 | # Licensed under the Apache License, Version 2.0 (the "License");
6 | # you may not use this file except in compliance with the License.
7 | # You may obtain a copy of the License at
8 | #
9 | # http://www.apache.org/licenses/LICENSE-2.0
10 | #
11 | # Unless required by applicable law or agreed to in writing, software
12 | # distributed under the License is distributed on an "AS IS" BASIS,
13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14 | # See the License for the specific language governing permissions and
15 | # limitations under the License.
16 |
17 |
18 | #find sgf/full/ -type f | tqdm | xargs grep 'C\[Resign Threshold: ' > resign_thresholds
19 | RUN="v9-19x19"
20 | RUN_DIR="../instance/data/$RUN"
21 | RESIGN_THRESHOLDS="resign_thresholds"
22 |
23 | PERCENT=10
24 |
25 | if [[ -z "$RUN_DIR" || ! -d "$RUN_DIR" ]]; then
26 | echo "$RUN_DIR does not exist"
27 | exit 2;
28 | fi
29 |
30 | if [[ ! -f "$RUN_DIR/$RESIGN_THRESHOLDS" ]]; then
31 | echo "didn't find '$RESIGN_THRESHOLDS' file"
32 | exit 2;
33 | fi
34 |
35 | cd "$RUN_DIR"
36 | hours=$(ls sgf/full | grep -o "201[89]-[0-9][0-9]-[0-9][0-9]-[0-9][0-9]")
37 | hour_count=$(echo "$hours" | wc -l)
38 | first_hour=$(echo "$hours" | head -n 1)
39 | last_hour=$(echo "$hours" | tail -n 1)
40 | echo "Found $hour_count hours ($first_hour to $last_hour)"
41 |
42 | read -p "Do you want to subsample $RUN to 10%? \"$RUN_DIR\" y/[N]: " answer
43 | case $answer in
44 | [Yy]* ) ;;
45 | [Nn]* ) exit;;
46 | * ) echo "Defaulting Yes"; exit;;
47 | esac
48 |
49 | count=0
50 | echo "$hours" | while read hour;
51 | do
52 | count=$((count + 1))
53 |
54 | all_files=$(find "sgf/full/$hour" -maxdepth 1 -iname "*.sgf")
55 | file_count=$(echo "$all_files" | wc -l)
56 | resign_files=$(grep "sgf/full/$hour" "$RESIGN_THRESHOLDS")
57 | resign_count=$(echo "$resign_files" | wc -l)
58 |
59 | # TODO only do if > 1000 files
60 | # eligable=$all_files
61 | # eligable_count=$(echo "$eligable" | wc -l)
62 | # new_count=$(echo "(0.9 * $eligable_count)/1" | bc)
63 | # echo "$hour contains $file_count files, $resign_count resign records, sampling $eligable_count, rm $new_count"
64 | # echo "$eligable" | shuf | head -n "$new_count" | xargs rm
65 |
66 | # if [[ $file_count -gt 0 && $file_count -eq $resign_count ]]; then
67 | # eligable=$(echo "$resign_files" | grep -v ': -1' | grep -o '^[^:]*')
68 | # eligable_count=$(echo "$eligable" | wc -l)
69 | # new_count=$(echo "(0.9 * $eligable_count)/1" | bc)
70 | # echo "$hour contains $file_count files, $resign_count resign records, sampling $eligable_count, rm $new_count"
71 | # echo "$eligable" | shuf | head -n "$new_count" | xargs rm
72 | # fi
73 | done
74 |
--------------------------------------------------------------------------------
/oneoff/unify-v3.sh:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env bash
2 | #
3 | # Copyright 2018 Google LLC
4 | #
5 | # Licensed under the Apache License, Version 2.0 (the "License");
6 | # you may not use this file except in compliance with the License.
7 | # You may obtain a copy of the License at
8 | #
9 | # http://www.apache.org/licenses/LICENSE-2.0
10 | #
11 | # Unless required by applicable law or agreed to in writing, software
12 | # distributed under the License is distributed on an "AS IS" BASIS,
13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14 | # See the License for the specific language governing permissions and
15 | # limitations under the License.
16 |
17 | V3_DIR="../instance/data/v3-9x9/sgf"
18 |
19 | if [[ -z "$V3_DIR" || ! -d "$V3_DIR" ]]; then
20 | echo "$V3_DIR does not exist"
21 | exit 2;
22 | fi
23 |
24 | cd "$V3_DIR"
25 | models=$(ls | grep -o "00[0-9]\{4\}-[a-z-]*")
26 | model_count=$(echo "$models" | wc -l)
27 | first_model=$(echo "$models" | head -n 1)
28 | last_model=$(echo "$models" | tail -n 1)
29 | echo "Found $model_count models ($first_model to $last_model)"
30 |
31 | read -p "Do you want to unify v3 to v5 layout? \"$V3_DIR\" y/[N]: " answer
32 | case $answer in
33 | [Yy]* ) ;;
34 | [Nn]* ) exit;;
35 | * ) echo "Defaulting Yes"; #exit;;
36 | esac
37 |
38 |
39 | count=0
40 | echo "$models" | while read model;
41 | do
42 | echo "" | tqdm --total $model_count --initial $count --leave False
43 | count=$((count + 1))
44 |
45 | file_count=$(find "$model" -maxdepth 0 -iname "*.sgf" | wc -l)
46 | if [[ $file_count -gt 0 ]]; then
47 | echo "model \"$model\", moving $file_count SGFs"
48 | mkdir -p "$model/full"
49 | mv "$model/"*sgf "$model/full/"
50 | fi
51 |
52 | if [[ -d "$model/full" ]]; then
53 | mkdir -p "$model/clean"
54 | cd $model/full
55 | find . -iname '*.sgf' | tqdm | xargs -I {} sh -c "sgfstrip C < {} > ../clean/{}"
56 | cd ../..
57 | fi
58 | done
59 |
60 | #time find . -iname '*.sgf' | tqdm | xargs -I {} sh -c "sgfstrip C < {} > ../clean/{}"
61 |
62 |
--------------------------------------------------------------------------------
/rsync-data.sh:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 | #
3 | # Copyright 2018 Google LLC
4 | #
5 | # Licensed under the Apache License, Version 2.0 (the "License");
6 | # you may not use this file except in compliance with the License.
7 | # You may obtain a copy of the License at
8 | #
9 | # http://www.apache.org/licenses/LICENSE-2.0
10 | #
11 | # Unless required by applicable law or agreed to in writing, software
12 | # distributed under the License is distributed on an "AS IS" BASIS,
13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14 | # See the License for the specific language governing permissions and
15 | # limitations under the License.
16 | #
17 | #
18 | # Tool to sync sgf files from minigo public bucket
19 | # ./rsync-sgf-data.sh -b minigo-pub -r v5-19x19 -m 102 clean data/
20 |
21 | function print_help {
22 | echo "
23 | usage: $0 [-h] [-n] [-t] [-m model] [-b bucket] [-r run] [-c count] type dest
24 |
25 | This script rsyncs sgf files from minigo public bucket to local drive
26 |
27 | optional arguments:
28 | -h Print this help message
29 | -n Dry run
30 | -t use time based names
31 | -m model only sync this models and later (e.g. 102 syncs models >=102)
32 | -b bucket Google Cloud bucket, defaults to "minigo-pub"
33 | -r run Top level folder in bucket, defaults to "v7-19x19"
34 | -c count Sync at most this many files from each directory
35 |
36 | Positional arguments
37 | type Source directory in bucket {clean, full, both, models}
38 | dest Directory to sync files to
39 | "
40 | }
41 |
42 | function error {
43 | echo $1;
44 | exit 2;
45 | }
46 |
47 | BUCKET="minigo-pub"
48 | RUN="v7-19x19"
49 | MIN_MODEL=
50 | DRY_RUN=
51 | MODELS=
52 | TIME_BASED_NAMES=
53 | MAX_COUNT=
54 | while getopts hntm:b:r:c: option
55 | do
56 | case "${option}" in
57 | h) print_help; exit 2;;
58 | n) DRY_RUN=1;;
59 | t) TIME_BASED_NAMES=1;;
60 | m) MIN_MODEL=$OPTARG;;
61 | b) BUCKET=$OPTARG;;
62 | r) RUN=$OPTARG;;
63 | c) MAX_COUNT=$OPTARG;;
64 | *) error "Unexpected option ${option}";;
65 | esac
66 | done
67 |
68 | # Remove parsed arguements
69 | shift $((OPTIND-1))
70 |
71 | # Arg checking
72 | if [[ "$#" -ne 2 ]]; then
73 | print_help;
74 | exit 2;
75 | fi
76 |
77 | TYPE="$1"
78 | DEST="$2"
79 |
80 | if [[ ! -d "$DEST" ]]; then
81 | echo "\"$DEST\" is not a directory";
82 | print_help;
83 | exit 2;
84 | fi
85 |
86 | EVALS=
87 | CLEAN=1
88 | FULL=1
89 | case "$TYPE" in
90 | both) ;;
91 | clean)
92 | FULL=;;
93 | full)
94 | CLEAN=;;
95 | models)
96 | MODELS=1;;
97 | evals)
98 | EVALS=1;;
99 | *) error "Unexpected TYPE option $TYPE";;
100 | esac
101 |
102 | # gsutil doesn't like // in folder paths which happens if RUN is empty.
103 | base_cloud_path="gs://$(echo "$BUCKET/$RUN/" | sed 's#//#/#g')"
104 | if [[ "$MODELS" ]]; then
105 | cloud_model_path="${base_cloud_path}models/"
106 | echo "Syncing model files from $cloud_path, at $(date)"
107 |
108 | dest="$DEST/$RUN/models"
109 | gsutil -m rsync "$cloud_model_path" "$dest"
110 | exit 0;
111 | fi
112 |
113 | if [[ "$EVALS" ]]; then
114 | cloud_path="${base_cloud_path}sgf/eval"
115 | echo "Syncing eval files from $cloud_path, at $(date)"
116 |
117 | # also try syncing a couple of date folders
118 | TODAY=$(date +"%Y-%m-%d")
119 | YESTERDAY=$(date --date=yesterday +"%Y-%m-%d")
120 | TOMORROW=$(date --date=tomorrow +"%Y-%m-%d")
121 |
122 | dest="$DEST/$RUN/eval"
123 | #mkdir -p "$dest"
124 | #gsutil -m rsync -r "$cloud_path" "$dest"
125 |
126 | mkdir -p "$dest/$TODAY" "$dest/$YESTERDAY" "$dest/$TOMORROW"
127 | gsutil -m rsync -r "$cloud_path/$TODAY" "$dest/$TODAY"
128 | gsutil -m rsync -r "$cloud_path/$YESTERDAY" "$dest/$YESTERDAY"
129 | gsutil -m rsync -r "$cloud_path/$TOMORROW" "$dest/$TOMORROW"
130 | exit 0;
131 | fi
132 |
133 | # SYNC GAMES
134 | cloud_path="${base_cloud_path}sgf"
135 | partial_dest="$DEST/$RUN/sgf"
136 | echo "Getting models list from $cloud_path ($TIME_BASED_NAMES)"
137 | if [[ "$TIME_BASED_NAMES" ]]; then
138 | models=$(gsutil ls "$cloud_path/clean" | grep -o "201[89]-[0-9-]\{8\}/")
139 | else
140 | models=$(gsutil ls "$cloud_path" | grep -o "00[0-9]\{4\}-[a-z-]*/")
141 | fi
142 |
143 | first_model=$(echo "$models" | head -n 1)
144 | last_model=$(echo "$models" | tail -n 1)
145 | echo "Found $(echo "$models" | wc -l) models ($first_model to $last_model) [>= $MIN_MODEL]"
146 |
147 | if [[ "$MIN_MODEL" ]]; then
148 | models=$(echo "$models" | sed 's#^\(\([0-9-]*\)\(-.*\|/\)\)$#\2 \1#' |
149 | awk -v min="$MIN_MODEL" '$1 >= min { print $2 }')
150 | echo "Syncing $(echo "$models" | wc -l) models >= $MIN_MODEL"
151 | fi
152 |
153 | if [[ -z "$models" ]]; then
154 | echo "NO MODELS FOUND!"
155 | exit;
156 | fi
157 |
158 | read -p "Do you want to sync \"$cloud_path\" to \"$partial_dest\" [Y]/n: " answer
159 | case $answer in
160 | [Yy]* ) ;;
161 | [Nn]* ) exit;;
162 | * ) echo "Defaulting Yes!";;
163 | esac
164 |
165 | function gs_rsync() {
166 | mkdir -p "$2"
167 | echo -e "\t\e[1;32m$1 => $2\e[0m"
168 | echo -e "\t$(date)"
169 |
170 | if [[ "$MAX_COUNT" ]]; then
171 |
172 | existing="$(ls "$2" | wc -l)"
173 | echo -e "\texisting: $existing"
174 | if [[ "$existing" -ge "$MAX_COUNT" ]]; then
175 | return;
176 | fi
177 | file_list="$(gsutil ls $1)"
178 | found="$(echo "$file_list" | wc -l)"
179 | echo "$(echo "$file_list" | head)"
180 | echo -e "\tfound: $found"
181 | if [[ $found -gt 10 ]]; then
182 | partial_list="$(echo "$file_list" | shuf -n $MAX_COUNT | sort)"
183 | gsutil -m cp $partial_list "$2"
184 | fi
185 | else
186 | gsutil -m rsync -r "$1" "$2"
187 | fi
188 | }
189 |
190 | echo "$models" | while read model;
191 | do
192 | echo
193 | echo -e "\e[1;32mSyncing $model $(date)\e[0m"
194 | if [[ -z "$DRY_RUN" ]]; then
195 | if [[ "$TIME_BASED_NAMES" ]]; then
196 | # NOTE: Wait one hour to ensure uniform sample of TIME_BASED_NAMES
197 | last_model=$(echo "$models" | tail -n 1)
198 | if [[ "$MAX_COUNT" ]] && [[ "$last_model" == "$model" ]]; then
199 | echo -e "\tSkipping recent model: $model"
200 | continue
201 | fi
202 |
203 | full_path="full/$model"
204 | clean_path="clean/$model"
205 | else
206 | full_path="$model/full"
207 | clean_path="$model/clean"
208 | fi
209 |
210 | if [[ "$CLEAN" ]]; then
211 | gs_rsync "$cloud_path/$clean_path" "$partial_dest/$clean_path"
212 | fi
213 | if [[ "$FULL" ]]; then
214 | # NOTE: Syncing this before the end of the hour leads to problems :/
215 | # especially given how cron runs right at the start of the hour.
216 | gs_rsync "$cloud_path/$full_path" "$partial_dest/$full_path"
217 | fi
218 | fi
219 | done
220 |
--------------------------------------------------------------------------------
/schema.sql:
--------------------------------------------------------------------------------
1 | /* Copyright 2018 Google LLC
2 |
3 | Licensed under the Apache License, Version 2.0 (the "License");
4 | you may not use this file except in compliance with the License.
5 | You may obtain a copy of the License at
6 |
7 | http://www.apache.org/licenses/LICENSE-2.0
8 |
9 | Unless required by applicable law or agreed to in writing, software
10 | distributed under the License is distributed on an "AS IS" BASIS,
11 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 | See the License for the specific language governing permissions and
13 | limitations under the License.
14 | */
15 |
16 | CREATE TABLE IF NOT EXISTS name_to_model_id (
17 | /* SGF player names (PB and PW) are lookup in this table */
18 | name text not null,
19 |
20 | /* used to keep alias unique per bucket */
21 | bucket text not null,
22 | model_id integer not null,
23 |
24 | /* 'model' or 'sgf' currently, useful to sgf names only */
25 | source text not null,
26 |
27 | PRIMARY KEY (name, bucket)
28 | );
29 |
30 | CREATE TABLE IF NOT EXISTS bucket_model_range (
31 | /* convience table used to remember the hash lookup */
32 | bucket text not null primary key,
33 | model_id_1 integer not null,
34 | model_id_2 integer not null
35 | );
36 |
37 | CREATE TABLE IF NOT EXISTS runs (
38 | bucket text not null primary key,
39 | name text not null,
40 | desc text not null default '',
41 | blocks integer not null,
42 | filters integer not null
43 | );
44 |
45 | CREATE TABLE IF NOT EXISTS models (
46 | model_id integer primary key,
47 |
48 | /**
49 | * Display:
50 | * Minigo: 000151-fury, 000250-golden-phoenix
51 | * LZ: LZXXX_<8_of_hash>
52 | * Raw:
53 | * Minigo: same
54 | * LZ: 0123456789abcdef0123...
55 | */
56 |
57 | /* This column is rendered in the UI */
58 | display_name text not null,
59 | /* This column is the name in models directory */
60 | raw_name text not null,
61 | /* This column is used to match with sgf
62 | (for model.ckpt-1024.pb, full lz hash) */
63 | sgf_name text not null,
64 |
65 | bucket text not null,
66 | num integer not null,
67 |
68 | last_updated integer not null,
69 | creation integer not null,
70 | training_time_m integer not null,
71 | num_games integer not null,
72 | num_stats_games integer not null,
73 | num_eval_games integer not null
74 | );
75 |
76 |
77 | CREATE TABLE IF NOT EXISTS games (
78 | timestamp integer not null,
79 | game_num integer not null,
80 |
81 | model_id integer not null,
82 | filename text not null,
83 |
84 | black_won boolean not null,
85 | result text not null,
86 | result_margin float not null, /* 0 if resign for now */
87 |
88 | num_moves integer not null,
89 |
90 | early_moves text, /* now just first two moves */
91 | early_moves_canonical text,
92 |
93 | has_stats boolean not null,
94 |
95 | number_of_visits_black integer,
96 | number_of_visits_white integer,
97 |
98 | number_of_visits_early_black integer,
99 | number_of_visits_early_white integer,
100 |
101 | unluckiness_black float,
102 | unluckiness_white float,
103 |
104 | resign_threshold float,
105 | bleakest_eval_black float,
106 | bleakest_eval_white float,
107 |
108 | PRIMARY KEY (timestamp, game_num)
109 | );
110 |
111 |
112 | CREATE TABLE IF NOT EXISTS model_stats (
113 | model_id integer not null,
114 | perspective text not null, /* "Black", "White", "All" */
115 |
116 | num_games integer not null,
117 | stats_games integer not null,
118 |
119 | wins integer,
120 | wins_by_resigns integer,
121 | wins_by_result integer,
122 | sum_wins_result integer,
123 |
124 | /* TODO(sethtroisi): rename holdout */
125 | hold_out_resigns integer,
126 | bad_resigns integer,
127 |
128 | /* Over games won by perspective */
129 | num_moves integer,
130 | number_of_visits integer,
131 |
132 | /* TODO(sethtroisi): figure out a better way to represent win / lose */
133 | number_of_visits_early_moves integer,
134 | sum_unluckiness float,
135 |
136 | favorite_openings text,
137 |
138 | PRIMARY KEY (model_id, perspective)
139 |
140 | /* TODO(sethtroisi): distributions */
141 | /* num_moves, */
142 | /* heatmap of openings, common openings, bleakest_eval, win amount */
143 | );
144 |
145 |
146 | CREATE TABLE IF NOT EXISTS eval_models (
147 | model_id_1 integer not null,
148 | model_id_2 integer not null,
149 |
150 | /* average of rank (or model_1_rank if model_id_2 == 0) */
151 | /* TODO rename rating */
152 | rankings float,
153 | std_err float,
154 |
155 | games integer not null,
156 |
157 | m1_black_games integer,
158 | m1_black_wins integer,
159 |
160 | m1_white_games integer,
161 | m1_white_wins integer,
162 |
163 | PRIMARY KEY (model_id_1, model_id_2)
164 | );
165 |
166 |
167 | CREATE TABLE IF NOT EXISTS eval_games (
168 | /**
169 | * Minigo games are of the form:
170 | * time _ white-model _ black-model _ number
171 | */
172 |
173 | eval_num integer not null,
174 | filename text not null,
175 |
176 | model_id_1 integer not null, /* black */
177 | model_id_2 integer not null, /* white */
178 |
179 | black_won boolean,
180 | result text,
181 | moves integer,
182 |
183 | PRIMARY KEY (eval_num, model_id_1)
184 | );
185 |
186 |
187 | CREATE TABLE IF NOT EXISTS position_eval_part (
188 | model_id integer,
189 | cord integer,
190 | type text,
191 | name text,
192 |
193 | policy float,
194 | value float,
195 | n integer,
196 | sgf text,
197 |
198 | PRIMARY KEY (model_id, cord, type, name)
199 | );
200 |
201 |
202 | CREATE TABLE IF NOT EXISTS position_setups (
203 | bucket text,
204 | name text,
205 | sgf text,
206 |
207 | PRIMARY KEY (bucket, name)
208 | );
209 |
210 |
211 | CREATE INDEX IF NOT EXISTS models_bucket_index ON models (bucket);
212 | CREATE INDEX IF NOT EXISTS models_display_name_index ON models (display_name);
213 | CREATE INDEX IF NOT EXISTS models_raw_name_index ON models (raw_name);
214 | CREATE INDEX IF NOT EXISTS models_sgf_name_index ON models (sgf_name);
215 |
216 | CREATE INDEX IF NOT EXISTS game_timestamp_index ON games (timestamp);
217 | CREATE INDEX IF NOT EXISTS game_game_num_index ON games (game_num);
218 | CREATE INDEX IF NOT EXISTS game_model_index ON games (model_id);
219 |
220 | CREATE INDEX IF NOT EXISTS eval_models_model_index_1 on eval_models (model_id_1);
221 | CREATE INDEX IF NOT EXISTS eval_models_model_index_2 on eval_models (model_id_2);
222 |
223 | CREATE INDEX IF NOT EXISTS eval_games_eval_num on eval_games (eval_num);
224 | CREATE INDEX IF NOT EXISTS eval_games_model_index_1 on eval_games (model_id_1);
225 | CREATE INDEX IF NOT EXISTS eval_games_model_index_2 on eval_games (model_id_2);
226 |
227 |
--------------------------------------------------------------------------------
/setup.sh:
--------------------------------------------------------------------------------
1 | echo "all-eval cron $(date)"
2 |
3 | set -e
4 |
5 | cd instance/data/synced-eval/
6 |
7 | mkdir -p eval/cross-eval \
8 | eval/v9-19x19 eval/v10-19x19 \
9 | eval/v12-19x19 eval/v13-19x19 \
10 | eval/v14-19x19 eval/v15-19x19
11 |
12 | eval_rsync="rsync -ah --info=progress2 --ignore-existing"
13 |
14 | #$eval_rsync "../v9-19x19/eval/" "eval/v9-19x19"
15 | #$eval_rsync "../v10-19x19/eval/" "eval/v10-19x19"
16 | #$eval_rsync "../v12-19x19/eval/" "eval/v12-19x19"
17 | #$eval_rsync "../v13-19x19/eval/" "eval/v13-19x19"
18 | #$eval_rsync "../v14-19x19/eval/" "eval/v14-19x19"
19 | $eval_rsync "../v15-19x19/eval/" "eval/v15-19x19"
20 |
21 | # MAKE LIST OF INTERESTING cross-eval files
22 | cross_bucket_regex='.*-\(v9\|v10\|v1[2-9]\)-.*-\(v9\|v10\|v1[2-9]\)-.*'
23 | find ../cross-eval/eval/ -regextype sed -regex "$cross_bucket_regex" \
24 | | sed 's#../cross-eval/eval/##' | tqdm > cross_games.txt
25 |
26 | $eval_rsync --files-from cross_games.txt "../cross-eval/eval/" "eval/cross-eval"
27 |
28 | echo "all-eval update $(date)"
29 | ./updater.py eval_games synced-eval
30 |
--------------------------------------------------------------------------------
/updater.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python3
2 | #
3 | # Copyright 2018 Google LLC
4 | #
5 | # Licensed under the Apache License, Version 2.0 (the "License");
6 | # you may not use this file except in compliance with the License.
7 | # You may obtain a copy of the License at
8 | #
9 | # http://www.apache.org/licenses/LICENSE-2.0
10 | #
11 | # Unless required by applicable law or agreed to in writing, software
12 | # distributed under the License is distributed on an "AS IS" BASIS,
13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14 | # See the License for the specific language governing permissions and
15 | # limitations under the License.
16 |
17 | import datetime
18 | import glob
19 | import os
20 | import sqlite3
21 | import sys
22 | import time
23 | from multiprocessing import Pool
24 |
25 | from tqdm import tqdm
26 |
27 | from web import sgf_utils
28 | from web.cloudygo import CloudyGo
29 | from web import cloudyback
30 |
31 |
32 | # get this script location to help with running in cron
33 | ROOT_DIR = os.path.dirname(os.path.abspath(__file__))
34 | INSTANCE_PATH = os.path.join(ROOT_DIR, 'instance')
35 | LOCAL_DATA_DIR = os.path.join(INSTANCE_PATH, 'data')
36 | DATABASE_PATH = os.path.join(INSTANCE_PATH, 'clouds.db')
37 |
38 | #CURRENT_BUCKET = 'leela-zero-v1'
39 | #CURRENT_BUCKET = 'v3-9x9'
40 | #CURRENT_BUCKET = 'v13-19x19'
41 | #CURRENT_BUCKET = CloudyGo.DEFAULT_BUCKET
42 |
43 | # Note when importing a new DB consider lowing
44 | # for an initial pass to make sure everything is okay.
45 | MAX_INSERTS = CloudyGo.MAX_INSERTS
46 |
47 |
48 | def setup():
49 | #### DB STUFF ####
50 |
51 | db = sqlite3.connect(DATABASE_PATH)
52 | db.row_factory = sqlite3.Row
53 |
54 | #### CloudyGo ####
55 |
56 | print("Running updater:", datetime.datetime.now())
57 | print("Setting up Cloudy for update")
58 | cloudy = CloudyGo(
59 | INSTANCE_PATH,
60 | LOCAL_DATA_DIR,
61 | lambda: db,
62 | None, # cache
63 | Pool(4)
64 | )
65 | print()
66 | return cloudy
67 |
68 |
69 | def update_position_eval(cloudy, bucket, group):
70 | position_paths = glob.glob(os.path.join(INSTANCE_PATH, group, bucket, '*'))
71 |
72 | models = cloudy.get_models(bucket)
73 | model_range = CloudyGo.bucket_model_range(bucket)
74 |
75 | count_per_model = len(position_paths) / max(1, len(models))
76 | print("{}: Updating {} {} position evals ({:.2f}/model)".format(
77 | bucket, group, len(position_paths), count_per_model))
78 |
79 | existing = cloudy.query_db(
80 | 'SELECT '
81 | ' model_id, type, name '
82 | 'FROM position_eval_part '
83 | 'WHERE model_id BETWEEN ? and ? AND type = ?',
84 | model_range + (group,))
85 |
86 | print ("\tloaded {} existing {} position_evals".format(
87 | group, len(existing)))
88 |
89 | to_process = []
90 | for position_path in position_paths:
91 | assert position_path.endswith('.csv'), position_path
92 | position_name = os.path.basename(position_path[:-4])
93 | raw_name, model_num = position_name.rsplit('-', 1)
94 | assert raw_name.startswith("heatmap-") or raw_name.startswith("pv-")
95 | name = raw_name.split('-', 1)[1]
96 |
97 | model_id = model_range[0] + int(model_num)
98 | if model_id and (model_id, group, name) not in existing:
99 | to_process.append((position_path, bucket, model_id, group, name))
100 |
101 | # Avoid tqdm output if no entries.
102 | if to_process:
103 | for entry in tqdm(to_process):
104 | cloudyback.update_position_eval(cloudy, *entry)
105 |
106 | return len(to_process)
107 |
108 |
109 | def update_position_setups(cloudy, bucket):
110 | # Refresh the policy/pv setups in db
111 | position_paths = glob.glob(os.path.join(
112 | INSTANCE_PATH, "positions", bucket, "*.sgf"))
113 |
114 | position_setups = []
115 | for position_path in position_paths:
116 | position_file = os.path.basename(position_path)
117 | position_name = position_file.replace(".sgf", "")
118 | with open(position_path) as data:
119 | sgf_data = data.read().replace("\n", "")
120 | position_setups.append((bucket, position_name, sgf_data))
121 |
122 | db = cloudy.db()
123 | db.execute("DELETE from position_setups where bucket = ?", (bucket,))
124 | cloudy.insert_rows_db("position_setups", position_setups)
125 | db.commit()
126 |
127 | return 0
128 |
129 |
130 | def update_games(cloudy, bucket):
131 | updates = 0
132 | print("{}: Updating Models and Games".format(bucket))
133 |
134 | # Setup models if they don't exist, don't update stats
135 | cloudyback.update_models(cloudy, bucket, only_create=True)
136 |
137 | models = cloudy.get_models(bucket)
138 | if len(models) == 0:
139 | return 0
140 |
141 | print("\tupdating_games({})".format(MAX_INSERTS))
142 | count = cloudyback.update_games(cloudy, bucket, MAX_INSERTS)
143 | updates += count
144 |
145 | if updates > 0:
146 | # Sync models with new data
147 | count += cloudyback.update_models(cloudy, bucket)
148 | return updates
149 |
150 |
151 | if __name__ == "__main__":
152 | T0 = time.time()
153 |
154 | cloudy = setup()
155 |
156 | updates = 0
157 |
158 | arg1 = sys.argv[1] if len(sys.argv) > 1 else ""
159 | buckets = sys.argv[2:] if len(sys.argv) > 2 else [CURRENT_BUCKET]
160 | assert 'True' not in buckets and 'False' not in buckets, buckets
161 |
162 | # Note: Models are also updated in update_games.
163 | if arg1 in ("models", "all_models"):
164 | for bucket in buckets:
165 | updates += cloudyback.update_models(
166 | cloudy,
167 | bucket,
168 | only_create=(arg1 == "models"))
169 |
170 | if len(sys.argv) == 1 or arg1 == "games":
171 | for bucket in buckets:
172 | updates += update_games(cloudy, bucket)
173 |
174 | if arg1 in ("eval_games", "all_eval_games"):
175 | for bucket in buckets:
176 | if arg1 == "all_eval_games":
177 | model_range = CloudyGo.bucket_model_range(bucket)
178 | db = cloudy.db()
179 | cur = db.execute(
180 | "DELETE FROM eval_games WHERE model_id_1 BETWEEN ? and ?",
181 | model_range)
182 | db.commit()
183 | print("Deleted", cur.rowcount, "eval_games from", bucket)
184 |
185 | bucket_updates = cloudyback.update_eval_games(cloudy, bucket)
186 | if bucket_updates:
187 | updates += bucket_updates
188 | updates += cloudyback.update_eval_models(cloudy, bucket)
189 |
190 | if len(sys.argv) == 1 or arg1 == "position_evals":
191 | for bucket in buckets:
192 | updates += update_position_setups(cloudy, bucket)
193 | for group in ["policy", "pv"]:
194 | updates += update_position_eval(cloudy, bucket, group)
195 |
196 | # Always update names
197 | cloudyback.update_model_names(cloudy)
198 | if updates >= 0:
199 | cloudyback.update_bucket_ranges(cloudy, buckets)
200 |
201 | T1 = time.time()
202 | delta = T1 - T0
203 | print("Updater took: {:.1f}s for {} updates = {:.1f}/s".format(
204 | delta, updates, updates / delta))
205 |
--------------------------------------------------------------------------------
/web/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/sethtroisi/cloudygo/1b31c8724c275a1e472ff96ad2450847a2127317/web/__init__.py
--------------------------------------------------------------------------------
/web/static/general.js:
--------------------------------------------------------------------------------
1 | // Copyright 2018 Google LLC
2 | //
3 | // Licensed under the Apache License, Version 2.0 (the "License");
4 | // you may not use this file except in compliance with the License.
5 | // You may obtain a copy of the License at
6 | //
7 | // http://www.apache.org/licenses/LICENSE-2.0
8 | //
9 | // Unless required by applicable law or agreed to in writing, software
10 | // distributed under the License is distributed on an "AS IS" BASIS,
11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 | // See the License for the specific language governing permissions and
13 | // limitations under the License.
14 |
15 | function update_url_inplace(param, newvalue) {
16 | var newurl = new URL(window.location);
17 | var params = new URLSearchParams(newurl.search);
18 | params.set(param, newvalue);
19 |
20 | newurl.search = params.toString();
21 | var updated = newurl.toString();
22 | history.replaceState({path: updated}, '', updated);
23 | }
24 |
--------------------------------------------------------------------------------
/web/static/robots.txt:
--------------------------------------------------------------------------------
1 | User-agent: *
2 | Disallow: */game/*
3 | Disallow: */full/*
4 | Disallow: */clean/*
5 | Disallow: */openings/*
6 | Disallow: */eval-model/*
7 | Disallow: */eval/*
8 | Disallow: */graphs-sliders
9 |
10 | User-agent: MauiBot
11 | Disallow: /
12 |
13 | User-agent: AhrefsBot
14 | Disallow: /
15 |
--------------------------------------------------------------------------------
/web/static/styles/general.css:
--------------------------------------------------------------------------------
1 | body {
2 | font-family: roboto;
3 | }
4 |
5 | .model-name {
6 | text-transform: capitalize;
7 | }
8 |
9 | .legend {
10 | cursor: pointer;
11 | }
12 |
13 | a:visited {
14 | color:#551A8B;
15 | }
16 |
17 | nav.navbar {
18 | border-bottom: 1px solid #e1e1e1;
19 | border-radius: 1px;
20 | margin-bottom: 1em;
21 | }
22 |
23 | /* Breadcrumbs are not working with this */
24 | nav.navbar a,
25 | nav.navbar p,
26 | nav.navbar .navbar-text,
27 | nav.navbar .breadcrumb-item.active {
28 | font-size: 1.5em;
29 | line-height: 1.5em;
30 | }
31 |
32 | @media (max-width: 992px) {
33 | nav.navbar a,
34 | nav.navbar p,
35 | nav.navbar .navbar-text,
36 | nav.navbar .breadcrumb-item.active {
37 | font-size: 2.3em;
38 | line-height: 2.3em;
39 | }
40 | }
41 |
42 |
43 | .navbar .dropdown-toggle.h2 {
44 | margin: auto;
45 | }
46 |
47 | .navbar .breadcrumb {
48 | margin: 0;
49 | padding: 0.5rem 1rem;
50 | background-color: inherit;
51 | }
52 |
--------------------------------------------------------------------------------
/web/static/styles/table.css:
--------------------------------------------------------------------------------
1 | table {
2 | border-collapse: collapse;
3 | }
4 |
5 | table.table {
6 | width: initial;
7 | }
8 |
9 | table.table tr th,
10 | tr th {
11 | border: 1px solid #ddd;
12 | padding: 6px 5px;
13 | }
14 |
15 | table.table tr td,
16 | td {
17 | border: 1px solid #ddd;
18 | padding: 2px 5px;
19 | }
20 |
21 |
22 | table.table tr.sep-bot td { padding-bottom: 20px; }
23 | table.table tr.sep-top td { padding-top: 20px; }
24 |
25 | .debug-sep {
26 | border-left: 5px solid black
27 | }
28 |
29 | .c-tooltip {
30 | position: relative;
31 | }
32 | .c-tooltip .tooltipcontainer {
33 | position:absolute;
34 | left: -150px;
35 | right: -150px;
36 | text-align: center;
37 | z-index: 2;
38 | }
39 |
40 | .c-tooltip .tooltiptext {
41 | background: #aaa;
42 | border-radius: 8px;
43 | display: none;
44 | padding: 8px 10px;
45 | max-width: 300px;
46 | }
47 | .c-tooltip:hover .tooltiptext {
48 | display: inline-block;
49 | }
50 |
51 | .models-table {
52 | display: inline-table;
53 | font-size: 125%;
54 | margin-bottom: 20px;
55 | margin-right: 5px;
56 | }
57 |
58 |
--------------------------------------------------------------------------------
/web/static/styles/wgo-mobile.css:
--------------------------------------------------------------------------------
1 | @media only screen and (max-width: 480px) {
2 | body { background: #eff }
3 | }
4 |
5 |
6 | /*--- Box styles ----------------------------------------------------------------------*/
7 |
8 | /*--- Comments box ----------------------------------------------------------------------*/
9 |
10 | /*
11 | .wgo-comments-wrapper {
12 | overflow: auto;
13 | margin: 0 0 0 0;
14 | height: 100%;
15 | position: relative;
16 | box-sizing: border-box;
17 | }
18 |
19 | .wgo-comments-content {
20 | padding: 3px 5px;
21 | border: 1px solid rgba(200,200,200,0.3);
22 | background-color: rgba(255,255,255,0.3);
23 | border-radius: 2px;
24 | overflow-y: auto;
25 | }
26 |
27 | .wgo-comments-content p {
28 | font-size: 0.9em;
29 | margin: 0.5em 0;
30 | }
31 |
32 | .wgo-help {
33 | background-color: rgba(236, 226, 216, 0.90);
34 | padding: 1px 7px;
35 | margin-bottom: 5px;
36 | }
37 |
38 | .wgo-notification {
39 | background-color: rgba(16, 16, 16, 0.95);
40 | color: white;
41 | padding: 1px 7px;
42 | margin-bottom: 5px;
43 | }
44 |
45 | .wgo-commentbox .wgo-box-title {
46 | background-repeat: no-repeat;
47 | background-position: right center;
48 | background-size: 24px;
49 | }
50 |
51 | .wgo-commentbox .wgo-box-title::after {
52 | content: '\e800';
53 | font-family: "wgo-icons";
54 | float: right;
55 | font-weight: normal;
56 | font-size: 0.9em;
57 | padding-top: 4px;
58 | width: 22px;
59 | text-align: center;
60 | }
61 |
62 | .wgo-commentbox.wgo-gameinfo .wgo-box-title::after {
63 | content: '\e801';
64 | padding-top: 2px;
65 | }
66 |
67 | .wgo-comments-nick {
68 | color: rgba(0,0,0,0.75);
69 | }
70 |
71 | a.wgo-move-link {
72 | text-decoration: none;
73 | border-bottom:1px dotted;
74 | }
75 | */
76 |
77 | /* right and left modifications */
78 |
79 | /*
80 | .wgo-player-right .wgo-comments-content, .wgo-player-left .wgo-comments-content {
81 | position: absolute;
82 | left: 10px;
83 | right: 10px;
84 | bottom: 10px;
85 | top: 40px;
86 | }
87 |
88 | .wgo-player-right .wgo-commentbox, .wgo-player-left .wgo-commentbox {
89 | position: absolute;
90 | bottom: 0;
91 | right: 0;
92 | left: 10px;
93 | top: 170px;
94 | }
95 | */
96 | /* top and bottom modifications */
97 | /*
98 | .wgo-player-top .wgo-comments-content, .wgo-player-bottom .wgo-comments-content {
99 | position: absolute;
100 | left: 40px;
101 | right: 0;
102 | top: 0;
103 | bottom: 0;
104 |
105 | }
106 | */
107 | /* TODO: handle too long title */
108 | /*
109 | .wgo-player-top .wgo-commentbox .wgo-box-title, .wgo-player-bottom .wgo-commentbox .wgo-box-title {
110 | transform: rotate(-90deg);
111 | -ms-transform: rotate(-90deg);
112 | -webkit-transform: rotate(-90deg);
113 | position: absolute;
114 | left: -50px;
115 | top: 55px;
116 | }
117 |
118 | .wgo-player-top .wgo-comments-wrapper, .wgo-player-bottom .wgo-comments-wrapper {
119 | margin-top: 10px;
120 | height: 150px;
121 | }
122 | */
123 | /* game info table */
124 | /*
125 | .wgo-commentbox .wgo-info-list {
126 | display: table;
127 | width: 100%;
128 | }
129 |
130 | .wgo-commentbox .wgo-info-title {
131 | display: table-caption;
132 | font-weight: 600;
133 | border-bottom: 2px solid rgba(200, 200, 200, 0.3);
134 | padding: 2px 0;
135 | }
136 |
137 | .wgo-commentbox .wgo-info-item {
138 | display: table-row;
139 | }
140 |
141 | .wgo-commentbox .wgo-info-label, .wgo-commentbox .wgo-info-value {
142 | display: table-cell;
143 | border-bottom: 1px solid rgba(200, 200, 200, 0.3);
144 | padding: 2px 15px 2px 0;
145 | }
146 |
147 | .wgo-commentbox .wgo-info-label {
148 | color: #000;
149 | }
150 |
151 | .wgo-commentbox .wgo-info-value {
152 | color: #4c4c4c;
153 | }
154 | */
155 | /* in gameinfo, last row is without border*/
156 | /*
157 | .wgo-commentbox.wgo-gameinfo .wgo-info-item:last-child .wgo-info-label, .wgo-commentbox.wgo-gameinfo .wgo-info-item:last-child .wgo-info-value {
158 | border-bottom: 0;
159 | }
160 | */
161 | /*--- /Comments box ----------------------------------------------------------------------*/
162 |
163 |
164 |
165 | /*--- Control box ------------------------------------------------------------------------*/
166 |
167 | .wgo-player-top .wgo-player-control {
168 | height: 180px;
169 | position: relative;
170 | }
171 |
172 | .wgo-ctrlgroup {
173 | transform: scale(1.9,1.9);
174 | }
175 |
176 | .wgo-ctrlgroup-left {
177 | position: absolute;
178 | top: 90px;
179 | left: 40px;
180 | z-index: 600;
181 | }
182 |
183 | .wgo-ctrlgroup-right {
184 | position: absolute;
185 | top: 90px;
186 | right: 40px;
187 | }
188 |
189 | .wgo-ctrlgroup-control {
190 | }
191 |
192 | .wgo-player-menu {
193 | width: 200px;
194 | }
195 |
196 | /*--- /Control box ------------------------------------------------------------------------*/
197 |
--------------------------------------------------------------------------------
/web/templates/figure-three.html:
--------------------------------------------------------------------------------
1 |
16 |
17 |
18 | {% from "macros.html" import models_link, bucket_name,
19 | bootstrap_links, favicon_links, navbar, tip with context %}
20 |
21 |
22 |
23 |
24 | Minigo Figure 3 Reproduction
25 |
26 | {{ favicon_links() }}
27 | {{ bootstrap_links() }}
28 |
29 |
30 |
31 |
32 |
33 |
34 |
35 |
45 |
46 |
47 | {{ navbar('figure_three', 'Figure 3 Reproduction') }}
48 |
49 | {% if not fig3_data %}
50 | Unfortunately figure 3 is hard to reproduce and I haven't yet found the professional games or wrangled the outputs of the model for {{ bucket_name(bucket) }} yet, Sorry.
51 | {% else %}
52 |
53 | These graphics are generated by sampled 1,000 games (no handicap, 7.5 Komi) from
54 | baduk movies pro game collection
55 |
56 | then choosing 1 random position from each game yielding 1,000 positions.
57 | I ask Minigo for it's policy and value output at each position and compare against the result of the real game.
58 |
59 | For more methodolgy details (or general questions) leave a comment in this
60 | Google Sheet
61 | or checkout the code at tensorflow/minigo
62 |
63 | The blue line is the evaluation at each model. The orange line is a
64 |
65 | moving average of the last 15 models
66 |
67 | bottom graphs are simply zoomed in on last 40 models.
68 |
69 | Much additional analysis has been done in Google Colab.
70 |
71 |
72 |
73 |
74 |
75 |
76 | {% endif %}
77 |
78 |
131 |
132 |
--------------------------------------------------------------------------------
/web/templates/fileslist.html:
--------------------------------------------------------------------------------
1 |
16 |
17 |
18 | {% from "macros.html" import models_link,
19 | bootstrap_links, favicon_links, navbar with context %}
20 |
21 |
22 |
23 |
24 | {{ navbar_title }}
25 |
26 | {{ favicon_links() }}
27 | {{ bootstrap_links() }}
28 |
29 |
30 |
36 |
37 |
38 |
39 | {{ navbar('eval_graphs', navbar_title) }}
40 |
41 |
102 | This page is generated by joining eval games from a number of runs with
103 | {{cross_run_games}} extra games played between models of different
104 | runs and calculating ratings over all those games.
105 | {% if bucket == 'cross-run-eval' %}
106 | 100 models were choosen from each recent run, 75 good models chosen
107 | evenly along the run and 25 of the strongest models.
108 | Any games played between these models were kept and additional games
109 | have been played between models not in the same run, to provide a
110 | more accurate rating.
111 | {% endif %}
112 | These ratings are less inflated than self-play elo because models face
113 | a wider more diverse set of opponents.
114 |
115 |
116 |
117 |
118 |
119 |
120 |
121 |
122 |
123 |
124 |
Ratings from
125 |
{{ total_games }} games shown
126 |
rating
127 |
uncertainty
128 |
129 |
130 | {% for model in sorted_models[:15] + sorted_models[-1:] %}
131 |
132 |
{{ "Best model" if loop.first else
133 | ("Worst model" if loop.last else loop.index) }}
73 | {% for r in results[:1000] %}
74 |
75 |
77 | {% endfor %}
78 |
79 |
80 |
93 |
94 |
--------------------------------------------------------------------------------
/web/utils.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/python
2 | #
3 | # Copyright 2018 Google LLC
4 | #
5 | # Licensed under the Apache License, Version 2.0 (the "License");
6 | # you may not use this file except in compliance with the License.
7 | # You may obtain a copy of the License at
8 | #
9 | # http://www.apache.org/licenses/LICENSE-2.0
10 | #
11 | # Unless required by applicable law or agreed to in writing, software
12 | # distributed under the License is distributed on an "AS IS" BASIS,
13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14 | # See the License for the specific language governing permissions and
15 | # limitations under the License.
16 |
17 |
18 | def list_preview(l, count=5):
19 | if len(l) < 2*count:
20 | return ", ".join(map(str, l))
21 | return ", ".join(map(str, l[:count] + ["..."] + l[-count:]))
22 |
--------------------------------------------------------------------------------
170 | 
73 | 
77 | {% endfor %}
78 |