Course Dump Menu

├── icons ├── icon.png ├── icon.psd ├── stop.png └── coffee.png ├── Anki Templates.apkg ├── manifest.json ├── menu.html ├── progressbars.css ├── menu.css ├── progressbars.js ├── background.js ├── menu.js ├── coursescan.js └── README.md /icons/icon.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Eltaurus-Lt/CourseDump2022/HEAD/icons/icon.png -------------------------------------------------------------------------------- /icons/icon.psd: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Eltaurus-Lt/CourseDump2022/HEAD/icons/icon.psd -------------------------------------------------------------------------------- /icons/stop.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Eltaurus-Lt/CourseDump2022/HEAD/icons/stop.png -------------------------------------------------------------------------------- /icons/coffee.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Eltaurus-Lt/CourseDump2022/HEAD/icons/coffee.png -------------------------------------------------------------------------------- /Anki Templates.apkg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Eltaurus-Lt/CourseDump2022/HEAD/Anki Templates.apkg -------------------------------------------------------------------------------- /manifest.json: -------------------------------------------------------------------------------- 1 | { 2 | "manifest_version": 3, 3 | "name": "Memrise Course Dump", 4 | "version": "9.1", 5 | "description": "Download Memrise cources as Anki-compatible CSV + media files", 6 | "icons": { 7 | "16": "icons/icon.png", 8 | "32": "icons/icon.png", 9 | "48": "icons/icon.png", 10 | "128": "icons/icon.png" 11 | }, 12 | "permissions": [ 13 | "activeTab", 14 | "storage", 15 | "tabs", 16 | "downloads", 17 | "scripting" 18 | ], 19 | "action": { 20 | "default_popup": "menu.html" 21 | }, 22 | "background": { 23 | "service_worker": "background.js" 24 | }, 25 | "content_security_policy": {}, 26 | "web_accessible_resources": [{ 27 | "resources": [], 28 | "matches": [""] 29 | }], 30 | "content_scripts": [ 31 | { 32 | "matches": ["http://*.memrise.com/*", "https://*.memrise.com/*"], 33 | "css": ["progressbars.css"] 34 | } 35 | ] 36 | } 37 | -------------------------------------------------------------------------------- /menu.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | Course Dump Menu 9 | 10 | 11 |

Download current course

12 |

Stop ongoing download

13 | 14 |

15 | 16 |

Help / GitHub repo

17 |

Feedback / Discussion

18 |

Anki Templates

19 |

Buy Me a Coffee

20 | 21 |

22 | 23 |

Batch download

24 |

Add to queue

25 |

Download all

26 |

View queued courses

27 |

Import course list

28 |

Clear queue

29 | 30 |

31 | 32 |

Settings

33 | 34 |

Download media 35 |

36 | 37 |

Extra fields 38 |

39 | 40 |

Level tags 41 |

42 | 43 |

Anki import prompt 44 |

45 | 46 |

Advanced 47 |

48 |

49 |

50 | 51 |

Learning Progress 52 |

53 | 54 |

Alternative Answers 55 |

56 | 57 |

Learnable IDs 58 |

59 | 60 |

Video files 61 |

62 | 63 |

Skip media download 64 |

65 | 66 |

Course metadata 67 |

68 |

69 |

70 |

Restore default

71 | 72 | 73 | -------------------------------------------------------------------------------- /progressbars.css: -------------------------------------------------------------------------------- 1 | :root { 2 | --dumpprogress-barwidth: 1.2em; 3 | --mem-green: #83c53d; 4 | --mem-red: #ff756b; 5 | --mem-yellow: #ffc001; 6 | --mem-darkblue: #2b3648; 7 | --mem-lightblue: #b9d7e3; 8 | --mem-darkerblue: #336ba5;/*#6a717b;*/ 9 | } 10 | 11 | #MemDump_progressContainer { 12 | width: 100%; 13 | position: relative; 14 | background: var(--mem-darkblue); 15 | transition: height 1s; 16 | overflow-y: hidden; 17 | } 18 | #MemDump_progressContainer::after { 19 | content: ''; 20 | position: absolute; 21 | inset: 0; 22 | background: repeating-linear-gradient(#eee 0, #fff calc(.15 * var(--dumpprogress-barwidth)), #fff calc(.58 * var(--dumpprogress-barwidth)), #ccc var(--dumpprogress-barwidth)); 23 | mix-blend-mode: multiply; 24 | } 25 | 26 | #MemDump_progressContainer.stopped { 27 | filter: grayscale(100%); 28 | } 29 | #MemDump_progressContainer.error { 30 | background: var(--mem-red) !important; 31 | } 32 | #MemDump_progressContainer.error * { 33 | filter: grayscale(100%); 34 | mix-blend-mode: multiply; 35 | opacity: 0.25; 36 | } 37 | 38 | @keyframes bar-expand { 39 | 0% {height: 0em;} 40 | 100% {height: var(--dumpprogress-barwidth);} 41 | } 42 | 43 | div[id^="MemDump_progress-"] { 44 | height: var(--dumpprogress-barwidth); 45 | animation: bar-expand .25s ease-out forwards; 46 | left: 0; 47 | top: 0; 48 | } 49 | div#MemDump_progress-media { 50 | --bar-color: var(--mem-yellow); 51 | position: absolute; 52 | inset: 0; 53 | } 54 | div#MemDump_progress-media.done { 55 | background: var(--mem-green); 56 | } 57 | div[id^="MemDump_progress-thread"] { 58 | --bar-color: var(--mem-lightblue); 59 | } 60 | div[id^="MemDump_progress-"] { 61 | transition: background-position-x .5s; 62 | background-image: linear-gradient(to left, transparent 50%, var(--bar-color) 50%); 63 | background-size: 200% 110%; 64 | background-position: 100%; 65 | position: relative; 66 | } 67 | div[id^="MemDump_progress-"].resetting { 68 | transition: background-position-x 0s .4s, opacity .3s; 69 | opacity: 0; 70 | } 71 | div[id^="MemDump_progress-"]::before, 72 | div[id^="MemDump_progress-"]::after { 73 | position: absolute; 74 | bottom: 50%; 75 | translate: 0 calc(50% - 1px); 76 | font-family: "Arial", sans-serif; 77 | font-size: 10px; 78 | color: white; 79 | text-shadow: 1px 1px 2px #000000, 1px 1px 2px #00000055, 0px 0px 1px #000000; 80 | display: block; 81 | width: max-content; 82 | } 83 | div[id^="MemDump_progress-"]::before { 84 | left: 1em; 85 | } 86 | div[id^="MemDump_progress-"]::after { 87 | right: 1em; 88 | } 89 | div[id^="MemDump_progress-"]:not(:only-child):not(:has(+ #MemDump_progress-media))::before { 90 | content: attr(progress-label); 91 | } 92 | div[id^="MemDump_progress-"]::after { 93 | content: attr(progress-ratio); 94 | } 95 | div[id^="MemDump_progress-"].off:not(:only-child):not(:has(+ #MemDump_progress-media)) { 96 | animation: bar-expand 0.30s ease-out reverse forwards; 97 | overflow-y: hidden; 98 | } 99 | 100 | div#MemDump_progress-batch { 101 | --bar-color: var(--mem-darkerblue); 102 | } 103 | 104 | div[id^="MemDump_progress-"]:has(+ #MemDump_progress-media)::before, 105 | div[id^="MemDump_progress-"]:has(+ #MemDump_progress-media)::after { 106 | content: '' !important; 107 | } 108 | -------------------------------------------------------------------------------- /menu.css: -------------------------------------------------------------------------------- 1 | @import url(https://db.onlinewebfonts.com/c/df5e41ae1af690fdb8b46ed56048dfbd?family=Boing+WEB+Bold); 2 | /* 3 | @import url(https://fonts.googleapis.com/css?family=Noto+Sans+JP:400,900&display=swap); 4 | @import url(https://fonts.googleapis.com/css?family=Open+Sans:400,600,700&display=swap); 5 | */ 6 | 7 | :root { 8 | --dumpprogress-barwidth: 1.2em; 9 | --mem-orange: #ffbb00; 10 | --mem-green: #83c53d; 11 | --mem-red: #ff756b; 12 | --mem-yellow: #ffc001; 13 | --mem-darkblue: #2b3648; 14 | --mem-lightblue: #b9d7e3; 15 | --mem-lightgreen: #91be51; 16 | --mem-lightred: #f17e75; 17 | } 18 | 19 | body { 20 | background: var(--mem-darkblue); 21 | color: white; 22 | font-family: Noto, "Noto Sans JP", "Open Sans", sans-serif; 23 | 24 | border: solid 1px #fff6; 25 | margin: .25em; 26 | padding: 0; 27 | min-width: max-content; 28 | width: 205px; 29 | } 30 | 31 | body * { 32 | user-select: none; 33 | } 34 | 35 | ::-webkit-scrollbar { 36 | display: none; 37 | } 38 | 39 | #download-course, 40 | #stop-download { 41 | margin-top: .75em; 42 | font-weight: bold; 43 | } 44 | 45 | h2, h3 { 46 | font-family: "Boing WEB Bold", "Open Sans", sans-serif; 47 | } 48 | 49 | h2 { 50 | color: var(--mem-orange); 51 | font-size: 1.5rem; 52 | margin-bottom: 2rem; 53 | margin-top: 0px; 54 | } 55 | 56 | h3 { 57 | font-size: 1.25rem; 58 | padding: .75rem 1.5rem; 59 | margin: -.25rem 0 0; 60 | } 61 | 62 | h4 { 63 | position: relative; 64 | margin: 0; 65 | padding: .5em 1.5em; 66 | font-weight: 400; 67 | } 68 | 69 | hr { 70 | height: 1px; 71 | border: none; 72 | background: white; 73 | opacity: 0.25; 74 | margin: 1em; 75 | } 76 | 77 | h4 img { 78 | position: absolute; 79 | max-width: 16px; 80 | top: 50%; 81 | translate: -150% -50%; 82 | margin-right: .5em; 83 | } 84 | h4:has(img) { 85 | padding-left: 3.25em; 86 | } 87 | 88 | h4.danger { 89 | color: #fffa; 90 | font-size: .8em; 91 | padding-left: calc(1.5em / 0.8); 92 | } 93 | 94 | body[data-ongoing-download="false"] #stop-download { 95 | display: none; 96 | } 97 | body[data-ongoing-download="true"] #download-course { 98 | display: none; 99 | } 100 | 101 | /* interactive styles */ 102 | 103 | h4:hover { 104 | /*background: var(--mem-lightblue);*/ 105 | cursor: pointer; 106 | } 107 | h4::before { 108 | content: ''; 109 | position: absolute; 110 | inset: 0; 111 | opacity: .25; 112 | z-index: -1; 113 | transition: background .25s; 114 | } 115 | h4:hover::before { 116 | background: var(--mem-lightblue); 117 | } 118 | 119 | h4[counter]::after { 120 | content: ' (' attr(counter) ')'; 121 | } 122 | 123 | /* toggle styles: */ 124 | 125 | input.expander { 126 | position: relative; 127 | top: 2px; 128 | right: 3px; 129 | 130 | appearance: none; 131 | width: 8px; 132 | aspect-ratio: 1; 133 | 134 | background-image: linear-gradient(0.375turn, transparent 50%, white 52%); 135 | transform-origin: 65% 65%; 136 | rotate: -45deg; 137 | 138 | transition: rotate .2s; 139 | 140 | cursor: pointer; 141 | } 142 | 143 | input.expander:checked { 144 | rotate: 45deg; 145 | } 146 | 147 | input.toggle { 148 | position: absolute; 149 | right: 0; 150 | top: 50%; 151 | translate: 0 -50%; 152 | 153 | appearance: none; 154 | width: 28px; 155 | height: 16px; 156 | background-color: var(--mem-lightred); 157 | border-radius: 100px; 158 | background-repeat: no-repeat; 159 | background-position: 14%; 160 | background-size: 43%; 161 | background-image: url( "data:image/svg+xml," ); 162 | /* background-image: radial-gradient(circle at 50%, white 50%, transparent 55%); */ 163 | 164 | cursor: pointer; 165 | box-shadow: inset .5px .5px 2px #10102050; 166 | } 167 | body.animated input.toggle { 168 | transition: background .2s; 169 | } 170 | 171 | input.toggle:checked { 172 | background-position: 86%; 173 | background-color: var(--mem-lightgreen); 174 | } 175 | 176 | body[data-ongoing-download="true"] h4#batch-download, 177 | h4#batch-download[counter="0"], 178 | h4#batch-clear[counter="0"], 179 | h4[disabled], 180 | h4:has(input.toggle:disabled) { 181 | color: gray; 182 | } 183 | input.toggle:disabled { 184 | background-color: lightgray; 185 | filter: brightness(50%); 186 | } 187 | body[data-ongoing-download="true"] h4#batch-download img, 188 | h4#batch-download[counter="0"] img, 189 | h4[disabled] img { 190 | filter: grayscale(100) 191 | brightness(80%); 192 | } 193 | 194 | input.toggle { 195 | display: inline-block; 196 | margin: 0 1.25em; 197 | /* transform-origin: 0 0; 198 | scale: 50%; */ 199 | } 200 | 201 | #advancedGrid { 202 | display: grid; 203 | grid-template-rows: 0fr; 204 | transition: grid-template-rows .2s; 205 | } 206 | label:has(#toggleAdvanced:checked) + #advancedGrid { 207 | grid-template-rows: 1fr; 208 | } 209 | #advancedGrid > :first-child { 210 | overflow: hidden; 211 | } 212 | 213 | -------------------------------------------------------------------------------- /progressbars.js: -------------------------------------------------------------------------------- 1 | function progressBarContainer() { 2 | const containerId = 'MemDump_progressContainer'; 3 | 4 | const existingContainer = document.getElementById(containerId); 5 | if (existingContainer) return existingContainer; 6 | 7 | const progress_bar_container = document.createElement("div"); 8 | progress_bar_container.id = containerId; 9 | 10 | try { 11 | document.querySelector(".rebrand-header-root").prepend(progress_bar_container); 12 | } catch (err) { 13 | document.body.prepend(progress_bar_container); 14 | } 15 | 16 | return progress_bar_container; 17 | } 18 | 19 | function progressBar(barId, padding = true) { 20 | const existingBar = document.getElementById(barId); 21 | if (existingBar) return existingBar; 22 | 23 | const progress_bar_container = progressBarContainer(); 24 | if (!progress_bar_container) return; 25 | 26 | const progress_bar = document.createElement("div"); 27 | progress_bar.id = barId; 28 | progress_bar_container.appendChild(progress_bar); 29 | 30 | const padId = 'MemDump_progress-padding-' + barId.split('-').at(-1); 31 | const existingPad = document.getElementById(padId); 32 | if (!padding || existingPad) return progress_bar; 33 | 34 | const padding_bar = document.createElement("div"); 35 | padding_bar.id = padId; 36 | try { 37 | document.querySelector("#page-head").prepend(padding_bar); 38 | } catch (err) {} 39 | 40 | return progress_bar; 41 | } 42 | 43 | function batchProgressBar(batch_size) { 44 | if (batch_size < 2) return; 45 | return progressBar('MemDump_progress-batch'); 46 | } 47 | 48 | function scanProgressBar(threadN) { 49 | return progressBar('MemDump_progress-thread-' + threadN); 50 | } 51 | 52 | function mediaProgressBar() { 53 | return progressBar('MemDump_progress-media', false); 54 | } 55 | 56 | function removeScanBar(threadN) { 57 | const progress_bar = document.getElementById('MemDump_progress-thread-' + threadN); 58 | 59 | if (progress_bar) { 60 | progress_bar.style.animationPlayState = "paused"; 61 | setTimeout(()=>{ 62 | void progress_bar.offsetHeight; 63 | progress_bar.classList.add('off'); 64 | progress_bar.style.animationPlayState = "running"; 65 | }, 500); 66 | } 67 | 68 | const padding_bar = document.getElementById('MemDump_progress-padding-' + threadN); 69 | const remaining_count = document.querySelectorAll('[id^="MemDump_progress-padding-"]').length; 70 | 71 | if (padding_bar && remaining_count > 1) { 72 | padding_bar.style.animationPlayState = "paused"; 73 | setTimeout(()=>{ 74 | void padding_bar.offsetHeight; 75 | padding_bar.classList.add('off'); 76 | padding_bar.style.animationPlayState = "running"; 77 | }, 500); 78 | } 79 | } 80 | 81 | //progress updates (glue) 82 | function updScanProgress(threadN, cidd, levels_done, levels_todo, label = undefined) { 83 | //text progress (console) 84 | const done_str = levels_todo !== "" ? `${levels_done}/${levels_todo}` : ""; 85 | if (levels_done === 0) { 86 | if (levels_todo || label) { 87 | console.log(`thread: ${threadN} | Start scanning ${cidd['cid']}`); 88 | } else { 89 | console.log(`thread: ${threadN} | Fetching metadata for ${cidd['cid']}`); 90 | } 91 | } else { 92 | console.log(`thread: ${threadN} | cid: ${cidd['cid']} | scan progress: ${done_str}`); 93 | } 94 | 95 | const progress_bar = scanProgressBar(threadN); 96 | if (!progress_bar) return; 97 | 98 | //GUI progress 99 | if (label !== undefined) { 100 | progress_bar.setAttribute("progress-label", label); 101 | } 102 | progress_bar.setAttribute("progress-ratio", done_str); 103 | 104 | if (levels_todo || label) { 105 | if (!isNaN(parseInt(levels_todo, 10))) { 106 | progress_bar.style.backgroundPosition = (100 * (1. - levels_done/levels_todo)).toFixed(2)+"%"; 107 | } else { 108 | progress_bar.style.backgroundPosition = "42%"; 109 | } 110 | } else { 111 | progress_bar.classList.add('resetting'); 112 | progress_bar.style.backgroundPosition = "100%"; 113 | setTimeout(() => { 114 | progress_bar.classList.remove('resetting'); 115 | }, 500); 116 | } 117 | } 118 | 119 | function updBatchProgress(batch_done = 0, batch_size = 1, cidd = "") { 120 | const done_str = `${batch_done}/${batch_size}`; 121 | if (cidd) { 122 | console.log(`cid: ${cidd['cid']} | scan complete (${done_str})`); 123 | } 124 | 125 | const batch_progress_bar = batchProgressBar(batch_size); 126 | if (!batch_progress_bar) return; 127 | 128 | batch_progress_bar.setAttribute("progress-label", "Batch progress"); 129 | batch_progress_bar.setAttribute("progress-ratio", done_str); 130 | batch_progress_bar.style.backgroundPosition = (100 * (1. - batch_done/batch_size)).toFixed(2)+"%"; 131 | } 132 | 133 | function updMediaProgress(media_done = 0, media_total = 1) { 134 | const media_progress_bar = mediaProgressBar(); 135 | 136 | if (!media_progress_bar) return; 137 | if (media_done === "done") { 138 | console.log('media download complete'); 139 | media_progress_bar.setAttribute("progress-label", ""); 140 | media_progress_bar.setAttribute("progress-ratio", "done!"); 141 | media_progress_bar.classList.add('done'); 142 | return; 143 | } 144 | const done_str = `${media_done}/${media_total}`; 145 | console.log("media download progress: ", done_str); 146 | media_progress_bar.setAttribute("progress-label", "Downloading files..."); 147 | media_progress_bar.setAttribute("progress-ratio", done_str); 148 | media_progress_bar.style.backgroundPosition = (100 * (1. - media_done/media_total)).toFixed(2)+"%"; 149 | } -------------------------------------------------------------------------------- /background.js: -------------------------------------------------------------------------------- 1 | const apiTimeout = 25000; 2 | let ongoingTab = null; 3 | 4 | function sleep(ms) { 5 | return new Promise(resolve => setTimeout(resolve, ms)); 6 | } 7 | 8 | function downloadFile(options) { 9 | return new Promise(async (resolve, reject) => { 10 | let iid, deltas = {}; 11 | const onDownloadComplete = delta => { 12 | if (iid === undefined) { 13 | deltas[delta.id] = delta; 14 | } else if (delta.id == iid) { 15 | checkDelta(delta); 16 | } 17 | } 18 | chrome.downloads.onChanged.addListener(onDownloadComplete); 19 | function checkDelta(delta) { 20 | if (delta.state && delta.state.current === "complete") { 21 | chrome.downloads.onChanged.removeListener(onDownloadComplete); 22 | resolve(delta.id); 23 | } else if (delta.error) { 24 | chrome.downloads.onChanged.removeListener(onDownloadComplete); 25 | reject(new Error(delta.error.current)); 26 | } else if (delta.state && delta.state.current === "interrupted") { 27 | chrome.downloads.onChanged.removeListener(onDownloadComplete); 28 | reject(new Error(delta.state.current)); 29 | } 30 | } 31 | const timeId = setTimeout(async () => { 32 | if (iid !== undefined) return; 33 | if (options.url) { 34 | const query = { url: options.url }; 35 | const items = await chrome.downloads.search(query); 36 | if (items.length) { 37 | let item = items[0]; 38 | let iid = item.id; 39 | if (iid in deltas) { 40 | console.log(`saved with timeout ${options.filename} | item id = ${iid}`); 41 | checkDelta(deltas[iid]); 42 | } else if ((item.state && item.state === "complete") || (item.bytesReceived && item.totalBytes && item.bytesReceived === item.totalBytes)) { 43 | console.log(`saved from log with timeout ${options.filename} | item id = ${iid}`); 44 | chrome.downloads.onChanged.removeListener(onDownloadComplete); 45 | resolve(iid); 46 | } else { 47 | console.log(`download timed out on ${options.filename} | item id = ${iid}`); 48 | } 49 | return; 50 | } 51 | } 52 | chrome.downloads.onChanged.removeListener(onDownloadComplete); 53 | reject(new Error("API timeout")); 54 | }, apiTimeout); 55 | try { 56 | iid = await chrome.downloads.download(options); 57 | } catch (err) { 58 | chrome.downloads.onChanged.removeListener(onDownloadComplete); 59 | return reject(err); 60 | } finally { 61 | clearTimeout(timeId); 62 | } 63 | if (iid in deltas) checkDelta(deltas[iid]); 64 | }); 65 | } 66 | 67 | async function menuAlert(msg, decline_msg = "") { 68 | await sleep(400); 69 | chrome.runtime.sendMessage({ 70 | type: "coursedump_alert", 71 | msg 72 | }).catch(err=>{ 73 | if (decline_msg) { 74 | console.log(decline_msg); 75 | } 76 | }); 77 | } 78 | 79 | function updAllMenus() { 80 | chrome.runtime.sendMessage({ 81 | type: "coursedump_updateOngoings" 82 | }).catch(err=>{}); 83 | } 84 | 85 | function ciddsParse(cidd_strings) { 86 | try { 87 | return cidd_strings.map(cidd_string => JSON.parse(cidd_string)); 88 | } catch (err) { 89 | console.error("Error parsing cidd strings: ", err); 90 | return []; 91 | } 92 | } 93 | 94 | chrome.runtime.onMessage.addListener(async (arg, sender, sendResponse) => { 95 | //messages from menu 96 | if (arg.type === "coursedump_checkOngoing") { 97 | sendResponse({ "ongoing-status": !!ongoingTab }); 98 | return; 99 | } else if (arg.type === "coursedump_startDownload") { 100 | if (ongoingTab) { 101 | sendResponse({ status: "error", msg: "A download is already in progress in another tab" }); 102 | return; 103 | } 104 | ongoingTab = arg.tab_id; 105 | updAllMenus(); 106 | //pass arguments 107 | chrome.scripting.executeScript({ 108 | target: {tabId: ongoingTab}, 109 | args: [{'cidds': ciddsParse(arg.cidd_strings), 'settings': arg.settings}], 110 | func: vars => Object.assign(self, vars), 111 | }, () => { 112 | //import module 113 | chrome.scripting.executeScript({ 114 | target: {tabId: ongoingTab}, 115 | files: ['progressbars.js'] 116 | }, () => { 117 | // run scanning script 118 | chrome.scripting.executeScript({ 119 | target: {tabId: ongoingTab}, 120 | files: ['coursescan.js']}, 121 | (scanningFeedback) => { 122 | if (chrome.runtime.lastError) { 123 | console.log('Downloading tab was closed'); //during scan phase 124 | ongoingTab = null; 125 | menuAlert("Scanning tab was closed. Download terminated", 126 | "no open menus left, download termination alert was not sent"); 127 | updAllMenus(); 128 | } 129 | //console.log('scanning script callback', scanningFeedback); 130 | //return signal from the scanning script 131 | if (scanningFeedback?.[0]?.result === "scanning unauthorised") { 132 | console.log('User not logged in. Download terminated'); 133 | ongoingTab = null; 134 | updAllMenus(); 135 | } else if (scanningFeedback?.[0]?.result === "scanning stopped") { 136 | console.log('Download stopped by user during scanning phase'); 137 | updAllMenus(); 138 | } 139 | } 140 | ); 141 | } 142 | ); 143 | }); 144 | 145 | sendResponse({ 'status': "initiated" }); 146 | return; 147 | 148 | } else if (arg.type === "coursedump_stopDownload") { 149 | if (!ongoingTab) { 150 | sendResponse({ status: "error", msg: "No downloads in progress" }); 151 | updAllMenus(); 152 | return; 153 | } 154 | chrome.tabs.sendMessage(ongoingTab, { 155 | type: "coursedump_stopScan" 156 | }).then(re => { 157 | console.log("stop scanning signal went through"); 158 | }).catch(err => { 159 | console.log("Downloading tab no longer exists - no scanning to stop"); 160 | }); 161 | 162 | ongoingTab = null; 163 | console.log("bg stopped"); 164 | //updAllMenus(); 165 | sendResponse({ status: "stop signals sent" }); 166 | return; 167 | } 168 | 169 | //messages from the scanning script 170 | const tabId = sender.tab.id; 171 | if (tabId !== ongoingTab) { 172 | console.warn(`download request from ${tabId} rejected - ongoing download was initiated by ${ongoingTab}`); 173 | menuAlert("File download request from a conflicting tab received and rejected"); 174 | return; 175 | }; 176 | let maxConnections = 5; 177 | let urls = []; 178 | if (arg.type === "coursedump_downloadFiles") { 179 | let terminated = false; 180 | if (arg.file_queue) urls = arg.file_queue; 181 | const todo = urls.length; 182 | if (arg.maxThreads) maxConnections = arg.maxThreads; 183 | console.log(`max threads set to : ${maxConnections}`); 184 | let done = 0; 185 | let pids = Array(maxConnections).fill().map((_, i) => i + 1); 186 | const results = await Promise.allSettled(pids.map(async pid => { 187 | while (ongoingTab && urls.length) { 188 | // const [url, filename] = ["some url", "some filename.ext"]; //emu 189 | // const emu = urls.shift();//emu 190 | const [url, filename] = urls.shift(); 191 | await sleep(200); 192 | let did; 193 | try { 194 | did = await downloadFile({url, filename, conflictAction: "overwrite" }); 195 | //did = await sleep(Math.floor(Math.random() * 600 + 300)); //emulate download 196 | } catch (err) { 197 | console.error(filename, err); 198 | chrome.tabs.sendMessage(tabId, { 199 | type: "coursedump_error", 200 | error: err.message, 201 | url, filename 202 | }).catch(err => {}); 203 | } 204 | if (did !== undefined) { 205 | await chrome.downloads.erase({ id: did }); 206 | } 207 | done++; 208 | chrome.tabs.sendMessage(tabId, { 209 | type: "coursedump_progressMedia_upd", 210 | done, todo 211 | }).catch(err => { 212 | console.warn(err); 213 | if (err.message.includes('Could not establish connection. Receiving end does not exist.')) { 214 | console.log('Downloading tab appears to be closed. terminating file download.'); 215 | terminated = true; 216 | ongoingTab = null; 217 | } 218 | }) 219 | } 220 | })); 221 | 222 | for (let i = 0; i < results.length; i++) { 223 | const r = results[i]; 224 | if (r.status === "rejected") { 225 | console.error(`pid ${i + 1}: ${r.reason}`); 226 | } 227 | } 228 | 229 | if (ongoingTab) { 230 | chrome.tabs.sendMessage(tabId, { 231 | type: "coursedump_mediaFinished", 232 | status: "done" 233 | }).catch(err => {}); 234 | ongoingTab = null; 235 | } else { 236 | if (!terminated) { 237 | console.log('Download stopped by user during file downloading phase'); 238 | } else { 239 | console.log('Downloading tab was closed during file downloading phase'); 240 | menuAlert("Downloading tab was closed. Download terminated", 241 | "no open menus left, download termination alert was not sent"); 242 | } 243 | chrome.tabs.sendMessage(tabId, { 244 | type: "coursedump_mediaFinished", 245 | status: "stopped" 246 | }).catch(err => {}); 247 | } 248 | updAllMenus(); 249 | } 250 | }); 251 | -------------------------------------------------------------------------------- /menu.js: -------------------------------------------------------------------------------- 1 | const default_settings = { 2 | "download_media": true, 3 | "extra_fields": true, 4 | "level_tags": true, 5 | "anki_import_prompt": true, 6 | 7 | "learnable_ids": false, 8 | "skip_media_download": false, 9 | "course_metadata": true, 10 | 11 | "videofiles_limit": 'Infinity', 12 | 13 | "max_level_skip": 5, 14 | "max_extra_fields": 5, //attributes/ visible info/ hidden info - each 15 | "parallel_download_limit": 9 16 | }; 17 | 18 | const messages = { 19 | "not memrise course": "Has to be used on a memrise.com course page", 20 | "not memrise": "Has to be used from memrise.com", 21 | "not a course": "Has to be used from a specific course page", 22 | "already downloading": "A download is already in progress", 23 | 24 | //confirms 25 | "import queue": "Importing course list will overwrite all currently queued courses.\n Proceed?", 26 | "restore settings": "Restore default settings?" 27 | } 28 | 29 | class menuButtons { 30 | constructor () { 31 | this.Download = document.getElementById("download-course"); 32 | this.Stop = document.getElementById("stop-download"); 33 | this.BatchAdd = document.getElementById("batch-add"); 34 | this.BatchDownload = document.getElementById("batch-download"); 35 | this.BatchView = document.getElementById("batch-view"); 36 | this.BatchImport = document.getElementById("batch-import"); 37 | this.BatchClear = document.getElementById("batch-clear"); 38 | } 39 | } 40 | 41 | class menuToggles { 42 | constructor() { 43 | this.DownloadMedia = document.getElementById("setting-downloadMedia"); 44 | this.ExtraFields = document.getElementById("setting-extraFields"); 45 | this.LevelTags = document.getElementById("setting-levelTags"); 46 | this.AnkiPrompt = document.getElementById("setting-ankiPrompt"); 47 | this.LearnableIds = document.getElementById("setting-learnableIds"); 48 | this.SkipMedia = document.getElementById("setting-skipMedia"); 49 | this.CourseMeta = document.getElementById("setting-courseMeta"); 50 | 51 | this.Video = document.getElementById("setting-videoFiles"); 52 | 53 | this.all = document.querySelectorAll('input[type="checkbox"].toggle'); 54 | } 55 | 56 | async toSettings() { 57 | let current_settings = { 58 | "download_media": this.DownloadMedia.checked, 59 | "extra_fields": this.ExtraFields.checked, 60 | "level_tags": this.LevelTags.checked, 61 | "anki_import_prompt": this.AnkiPrompt.checked, 62 | 63 | "learnable_ids": this.LearnableIds.checked, 64 | "skip_media_download": this.SkipMedia.checked, 65 | "course_metadata": this.CourseMeta.checked, 66 | 67 | "videofiles_limit": this.Video.checked ? 'Infinity' : 0 68 | }; 69 | 70 | for (const [setting, default_value] of Object.entries(default_settings)) { 71 | if (!(setting in current_settings && current_settings[setting] !== 'undefined')) { 72 | current_settings[setting] = default_value; 73 | } 74 | } 75 | 76 | saveToStorage({ 'settings': current_settings }); 77 | } 78 | 79 | async fromSettings() { 80 | const settings = await loadSettings(); 81 | 82 | this.DownloadMedia.checked = settings["download_media"]; 83 | this.ExtraFields.checked = settings["extra_fields"]; 84 | this.LevelTags.checked = settings["level_tags"]; 85 | this.AnkiPrompt.checked = settings["anki_import_prompt"]; 86 | 87 | this.LearnableIds.checked = settings["learnable_ids"]; 88 | this.SkipMedia.checked = settings["skip_media_download"]; 89 | this.CourseMeta.checked = settings["course_metadata"]; 90 | 91 | this.Video.checked = (settings["videofiles_limit"] > 0); 92 | } 93 | } 94 | 95 | function extractNumericValue(string, key) { 96 | const regex = new RegExp(`${key}(\\d+)`); 97 | const match = string.match(regex); 98 | 99 | return match ? match[1] : ""; 100 | } 101 | 102 | function getDomainAndId(url) { 103 | const cid = extractNumericValue(url, "course_id=") || 104 | extractNumericValue(url, "category_id=") || 105 | extractNumericValue(url, "course/"); 106 | let domain = ""; 107 | if (url.includes("app.memrise.com")) { 108 | domain = "app.memrise.com"; 109 | } else if (url.includes("community-courses.memrise.com")) { 110 | domain = "community-courses.memrise.com"; 111 | } 112 | 113 | return {domain, cid} 114 | } 115 | 116 | async function getCurrentTab() { 117 | try { 118 | const [tab] = await chrome.tabs.query({ active: true, currentWindow: true }); 119 | return tab; 120 | } catch (error) { 121 | console.error(error); 122 | } 123 | } 124 | 125 | function saveToStorage(data) { 126 | chrome.storage.local.set(data); 127 | } 128 | 129 | async function loadFromStorage(obj) { 130 | return new Promise((resolve) => { 131 | chrome.storage.local.get([obj], (result) => { 132 | resolve(result[obj]); 133 | }); 134 | }); 135 | } 136 | 137 | async function loadSettings() { 138 | let settings = await loadFromStorage('settings') || default_settings; 139 | 140 | //initiate if accessed for the first time or a new setting is added 141 | let upd = false; 142 | for (const [setting, default_value] of Object.entries(default_settings)) { 143 | if (!(setting in settings && settings[setting] !== 'undefined')) { 144 | settings[setting] = default_value; 145 | upd = true; 146 | } 147 | } 148 | if (upd) {saveToStorage({ settings });} 149 | 150 | return settings; 151 | } 152 | 153 | async function loadQueue(text = false) { 154 | let queue = await loadFromStorage(`queue${text ? '-text' : ''}`); 155 | 156 | //initiate queues if accessed for the first time 157 | if (!queue) { 158 | queue = []; 159 | saveToStorage({ 'queue': [] }); 160 | saveToStorage({ 'queue-text': [] }); 161 | } 162 | 163 | return queue; 164 | } 165 | 166 | document.addEventListener('DOMContentLoaded', () => {setTimeout(async () => { 167 | 168 | // links 169 | document.getElementById("link-help").addEventListener('click', () => { 170 | window.open('https://github.com/Eltaurus-Lt/CourseDump2022?tab=readme-ov-file#memrise-course-dump', '_blank').focus(); 171 | }); 172 | document.getElementById("link-forum").addEventListener('click', () => { 173 | window.open('https://forums.ankiweb.net/t/an-alternative-to-memrise2anki-support-thread/30084', '_blank').focus(); 174 | }); 175 | document.getElementById("link-ankiweb").addEventListener('click', () => { 176 | window.open('https://ankiweb.net/shared/info/510199145', '_blank').focus(); 177 | }); 178 | document.getElementById("link-coffee").addEventListener('click', () => { 179 | window.open('https://buymeacoffee.com/eltaurus', '_blank').focus(); 180 | }); 181 | 182 | const current_tab = await getCurrentTab(); 183 | const {domain, cid} = getDomainAndId(current_tab.url); 184 | const cidd = JSON.stringify({domain, cid}); 185 | 186 | //initiate download through message to the BG script 187 | async function downloadCourses(cidds) { 188 | const settings = await loadSettings(); 189 | 190 | chrome.runtime.sendMessage({ 191 | type: "coursedump_startDownload", 192 | tab_id: current_tab.id, 193 | settings, 194 | cidd_strings: cidds 195 | }, (response) => { 196 | console.log(response); 197 | if (response.status === "error") { 198 | alert(response.msg); 199 | } 200 | // updOngoingStatus(); 201 | } 202 | ); 203 | 204 | // window.close(); 205 | } 206 | 207 | const buttons = new menuButtons(); 208 | 209 | async function updCounters() { 210 | const queue = await loadQueue(); 211 | const count = `${queue.length}`; 212 | buttons.BatchDownload.setAttribute("counter", count); 213 | buttons.BatchView.setAttribute("counter", count); 214 | buttons.BatchClear.setAttribute("counter", count); 215 | } 216 | 217 | async function addToQueue() { 218 | const queue = await loadQueue(); 219 | const queuetxt = await loadQueue(true); 220 | if (queue.includes(cidd)) {return} //list is supposedly faster than a set, because the number of itmes is small 221 | if (queue) { 222 | saveToStorage({'queue': [...queue, cidd]}); 223 | } else { 224 | saveToStorage({'queue': [cidd]}); 225 | } 226 | if (queuetxt) { 227 | saveToStorage({'queue-text': [...queuetxt, current_tab.url]}); 228 | } else { 229 | saveToStorage({'queue-text': [current_tab.url]}); 230 | } 231 | updCounters(); 232 | } 233 | 234 | function updOngoingStatus() { 235 | chrome.runtime.sendMessage({ 236 | type: "coursedump_checkOngoing" 237 | }, (response) => { 238 | document.body.setAttribute("data-ongoing-download", response['ongoing-status']); 239 | if (response['ongoing-status'] && buttons.BatchDownload.title === "") { 240 | buttons.BatchDownload.title = messages["already downloading"]; 241 | } 242 | if (!response['ongoing-status'] && buttons.BatchDownload.title === messages["already downloading"]) { 243 | buttons.BatchDownload.title = ""; 244 | } 245 | buttons.Stop.removeAttribute('disabled'); 246 | }); 247 | } 248 | 249 | updCounters(); 250 | updOngoingStatus(); 251 | 252 | //messages from background 253 | chrome.runtime.onMessage.addListener(async (arg, sender, sendResponse) => { 254 | if (arg.type === "coursedump_alert") { 255 | alert(arg.msg); 256 | updOngoingStatus(); 257 | } else if (arg.type === "coursedump_updateOngoings") { 258 | updOngoingStatus(); 259 | } 260 | }); 261 | 262 | //stop button 263 | buttons.Stop.addEventListener('click', () => { 264 | buttons.Stop.setAttribute('disabled', true); 265 | chrome.runtime.sendMessage({ 266 | type: "coursedump_stopDownload", 267 | }, (response) => { 268 | console.log(response); 269 | if (response.status === "error") { 270 | alert(response.msg); 271 | } 272 | //updOngoingStatus(); 273 | } 274 | ); 275 | }); 276 | 277 | //download and add buttons 278 | if (!domain) { 279 | buttons.Download.title = messages["not memrise course"]; 280 | buttons.BatchAdd.title = messages["not memrise course"]; 281 | buttons.BatchDownload.title = messages["not memrise"]; 282 | } else { 283 | if (!cid) { 284 | buttons.Download.title = messages["not a course"]; 285 | buttons.BatchAdd.title = messages["not a course"]; 286 | } else { 287 | buttons.Download.removeAttribute('disabled'); 288 | buttons.Download.addEventListener('click', () => { 289 | if (document.body.getAttribute("data-ongoing-download") !== "true") { 290 | downloadCourses([cidd]); 291 | } 292 | }); 293 | 294 | buttons.BatchAdd.removeAttribute('disabled'); 295 | buttons.BatchAdd.addEventListener('click', addToQueue); 296 | } 297 | 298 | buttons.BatchDownload.removeAttribute('disabled'); 299 | buttons.BatchDownload.addEventListener('click', async () => { 300 | const cidds = await loadQueue(); 301 | if (cidds && cidds.length > 0 && document.body.getAttribute("data-ongoing-download") !== "true") { 302 | downloadCourses(cidds); 303 | } 304 | }); 305 | } 306 | 307 | //other batch buttons 308 | buttons.BatchView.addEventListener('click', async (event) => { 309 | const queue = await loadQueue(!event.ctrlKey); 310 | 311 | // let queueTab = window.open("data:text/html, ","queueTab"); 312 | let queueTab = window.open("data:text/html","queueTab"); 313 | queueTab.document.write(` 314 | Download queue 315 | 316 | ${queue.join('
')} 317 | 318 | `); 319 | queueTab.document.close(); 320 | }) 321 | 322 | buttons.BatchImport.addEventListener('click', async () => { 323 | const queue = await loadQueue(); 324 | if ((queue.length == 0) || confirm(messages["import queue"])) { 325 | try { 326 | const pickerOpts = { 327 | types: [ 328 | { 329 | description: "Text files", 330 | accept: { "text/plain": [".txt"] }, 331 | }, 332 | ], 333 | multiple: false 334 | }; 335 | const [fileHandle] = await window.showOpenFilePicker(pickerOpts); 336 | const file = await fileHandle.getFile(); 337 | const content = await file.text(); 338 | const lines = content.split('\n'); 339 | const cidds = []; 340 | const courseList = lines.filter(line => { 341 | const {domain, cid} = getDomainAndId(line); 342 | const cidd = JSON.stringify({domain, cid}); 343 | const check = (domain && cid && !cidds.includes(cidd)); 344 | if (check) { 345 | cidds.push(cidd); 346 | } 347 | return check 348 | }); 349 | saveToStorage({'queue-text': courseList}); 350 | saveToStorage({'queue': cidds}); 351 | updCounters(); 352 | } catch (error) { 353 | console.log('Error reading file:', error); 354 | } 355 | } 356 | }) 357 | 358 | buttons.BatchClear.addEventListener('click', async () => { 359 | if (buttons.BatchClear.getAttribute("counter") !== '0' && confirm("Clear the list of queued courses?")) { 360 | saveToStorage({'queue': []}); 361 | saveToStorage({'queue-text': []}); 362 | updCounters(); 363 | } 364 | }) 365 | 366 | 367 | //settings 368 | const toggles = new menuToggles(); 369 | 370 | toggles.fromSettings(); 371 | //turn toggle animations on 372 | setTimeout(()=>document.querySelector('body').classList.add('animated'), 100);// awaiting above doesn't help (due to additional delay in css processing?) 373 | 374 | toggles.all.forEach((checkbox) => { 375 | checkbox.addEventListener('change', (event) => { 376 | toggles.toSettings(); 377 | }); 378 | }); 379 | 380 | document.getElementById("settings-restore").addEventListener('click', async () => { 381 | if (confirm("Restore default settings?")) { 382 | saveToStorage({ 'settings': default_settings}); 383 | toggles.fromSettings(); 384 | } 385 | }) 386 | 387 | }, 10)}); 388 | -------------------------------------------------------------------------------- /coursescan.js: -------------------------------------------------------------------------------- 1 | ANKI_HEADERS = true; 2 | 3 | console.log('course dump settings: ', settings); 4 | console.log('course dump batch: ', JSON.stringify(cidds)); 5 | 6 | function sleep(ms) { 7 | return new Promise(resolve => setTimeout(resolve, ms)); 8 | } 9 | 10 | function ciddToURL(cidd) { 11 | return `https://${cidd['domain']}/community/course/${cidd['cid']}`; 12 | } 13 | 14 | function fetchURL(domain) { 15 | return `https://${domain}/v1.19/learning_sessions/preview/`; 16 | } 17 | 18 | async function fetchMeta(cidd) { 19 | //default metadata values 20 | const meta = { 21 | 'url': ciddToURL(cidd), 22 | 'cid': cidd['cid'], 23 | 'proper name': '', 24 | 'thumbnail': '', 25 | 'url name': cidd['cid'], 26 | 'number of levels': '??', 27 | 'number of items': '??', 28 | 'description': '', 29 | 'author': '', 30 | 'ava': 'https://static.memrise.com/accounts/img/placeholders/empty-avatar-2.png', // -> rnd 1..4 31 | }; 32 | 33 | let course_page; 34 | try { 35 | course_page = await fetch(ciddToURL(cidd)); 36 | } catch(err) { 37 | console.error(`failed to fetch course ${cidd['cid']} html`, err); 38 | } 39 | if (!course_page) return meta; 40 | 41 | meta['url name'] = course_page.url.split("/")[6]; // course name from the end of url after redirections 42 | 43 | try { 44 | const parser = new DOMParser(); 45 | const doc = parser.parseFromString(await course_page.text(), "text/html"); 46 | 47 | meta['proper name'] = doc.querySelector('.course-name').innerText; 48 | meta['thumbnail'] = doc.querySelector('.course-photo img').src; 49 | meta['number of levels'] = (query => (query ? query.childElementCount : 1))(doc.querySelector('div.levels')); 50 | meta['description'] = (desc => (desc ? desc.innerText : ""))(doc.querySelector('.course-description.text')); 51 | meta['author'] = doc.querySelector('.creator-name span').innerText; 52 | meta['ava'] = doc.querySelector('.creator-image img').src; 53 | try { 54 | //only works for courses that have been started, otherwise the info is not present on the page 55 | meta['number of items'] = parseInt(doc.querySelector('.progress-box-title').innerText.split('/').at(-1).trim(), 10).toString(); 56 | } catch(err) { 57 | console.log(`course ${cidd['cid']} has not been started, info for 'number of items' unavailable`); 58 | } 59 | } catch(err) { 60 | console.error(`failed to parse course ${cidd['cid']} html`, err); 61 | } 62 | 63 | console.log(meta); 64 | 65 | return meta; 66 | } 67 | 68 | function meta2txt(meta) { 69 | let text = 'data:md/plain;charset=utf-8,' + encodeURIComponent( 70 | `# **${meta['proper name'] || meta['url name']}**\n` + 71 | `### by _${meta['author']}_\n` + 72 | `### (${meta['number of items']} learnable items)\n` + 73 | `\n` + 74 | meta['description'] 75 | ); 76 | if (!ANKI_HEADERS) { 77 | text = text + encodeURIComponent( 78 | `\n\n` + 79 | `## Course Fields\n` + 80 | `| ${meta['course fields']} |` 81 | ); 82 | } 83 | return text 84 | } 85 | 86 | //THE MAIN FUNCTION FOR SCANNING ALL LEVELS OF A COURSE 87 | async function scanCourse(cidd, threadN) { 88 | //stop message listener 89 | let stopped = false; 90 | function stopScan(arg, sender, sendResponse) { 91 | if (arg.type === "coursedump_stopScan") { 92 | //console.log(`cid: ${cidd['cid']} - scanning stopped by user`); 93 | chrome.runtime.onMessage.removeListener(stopScan); 94 | stopped = true; 95 | } 96 | } 97 | chrome.runtime.onMessage.addListener(stopScan); 98 | 99 | //init meta and progress bar 100 | updScanProgress(threadN, cidd, 0, "", ""); 101 | const meta = await fetchMeta(cidd); 102 | await sleep(500); 103 | updScanProgress(threadN, cidd, 0, meta['number of levels'], meta['proper name'] || meta['url name']); 104 | 105 | //process filenames and resolve name conflicts 106 | let reserved_filenames = new Set(); 107 | let url2filenames = {}; 108 | function UniqueDecodedFilename(url) { 109 | if (url in url2filenames) {return url2filenames[url];} 110 | 111 | //select name for a new file 112 | let temp_filename = decodeURIComponent(url.split("/").slice(-1)[0]); 113 | // let pad = decodeURIComponent(url.split("/").slice(-2)[0]); 114 | // if (pad === 'medium') {pad = decodeURIComponent(url.split("/").slice(-3)[0])}; 115 | // temp_filename = meta['url name'] + "_" + pad + "_" + temp_filename; 116 | temp_filename = meta['url name'] + "_" + cidd['cid'] + cidd['domain'][0] + "_" + temp_filename; 117 | temp_filename = temp_filename.replace('[','(').replace(']',')'); //square brackets are not allowed inside Anki because of [sound: ...] etc. 118 | let name_parts = temp_filename.split('.'); 119 | const ext = name_parts.pop().toLowerCase(); //Anki's "Check Media" is case-sensitive, since Chrome converts all extensions to lower case the csv entries have to match that 120 | const proper = name_parts.join("."); 121 | temp_filename = proper + "." + ext; 122 | if (reserved_filenames.has(temp_filename)) { //add ordinal number to make the filename unique 123 | for (let i = 1; reserved_filenames.has(temp_filename); i++) { 124 | temp_filename = proper + " (" + i + ")." + ext; 125 | } 126 | } 127 | url2filenames[url] = temp_filename; 128 | reserved_filenames.add(temp_filename); 129 | 130 | return temp_filename; 131 | } 132 | 133 | //init table data 134 | let err_count = 0; 135 | let has_audio = false; 136 | let has_video = false; 137 | const attributes = []; 138 | const visible_info = []; 139 | const hidden_info = []; 140 | let has_definitions = false; 141 | let has_learnable = false; 142 | const course_media_urls = new Set(); 143 | const table = []; 144 | 145 | // SCANNING LEVELS 146 | let levels_done = 0; 147 | let proceed = true; //fallback flag in case the number of levels is unavailable from meta or incorrect 148 | while ((proceed || levels_done < meta['number of levels']) && !stopped) { 149 | levels_done++; 150 | //emulation 151 | // await sleep(Math.floor(Math.random() * 500 + 200)); 152 | // proceed = (levels_done < meta['number of levels'] + settings["max_level_skip"]); 153 | // const done_clamped_emu = Math.min(levels_done, meta['number of levels']); 154 | // updScanProgress(threadN, cidd, isNaN(done_clamped_emu) ? levels_done : done_clamped_emu, meta['number of levels']); 155 | // file_queue.push(42); 156 | // continue; 157 | 158 | let level_is_empty = false; 159 | try { 160 | //await sleep(50); 161 | 162 | //fetch data from memrise 163 | const token = document.cookie.split(" ").find(cookie => cookie.includes("csrftoken")).split(/[=;]/g)[1]; // get CSRF header 164 | const response = await fetch(fetchURL(cidd['domain']), { 165 | "headers": { 166 | "Accept": "*/*", 167 | "Content-Type": "Application/json", 168 | "X-CSRFToken": token 169 | }, 170 | "body": JSON.stringify({ 171 | session_source_id: cidd['cid'], 172 | session_source_type: "course_id_and_level_index", 173 | session_source_sub_index: levels_done 174 | }), 175 | "method": "POST" 176 | }); 177 | if (!response.ok) { 178 | if (response.status === 401) { 179 | if (threadN === 0) { 180 | alert("Memrise login required"); // should be impossible to trigger because of the test at the start of batchDownload() 181 | } 182 | return "unauthorised"; 183 | } 184 | } 185 | const response_json = await response.json(); 186 | 187 | // Continue after empty set 188 | if (response_json.code == "PREVIEW_DOESNT_LOAD") { 189 | level_is_empty = true; 190 | } 191 | 192 | //define general tags for all entries from the level 193 | let tags = `"` + meta['url name'] + `"`; 194 | if (settings["level_tags"]) { 195 | try { 196 | //add the name of the level to the tags in Anki hierarchical tag format 197 | tags = `"` + response_json.session_source_info.name.replaceAll('"', '""') + 198 | `::` + levels_done.toString().padStart(meta['number of levels'].toString().length, "0") + 199 | `_` + response_json.session_source_info.level_name.replaceAll('"', '""') + `"`; 200 | } catch (err) {console.log(`${err.name}: ${err.message}`);} 201 | tags = tags.replaceAll(' ','_'); 202 | } 203 | 204 | 205 | // Adding all entries from the level to the table and their media urls to queue 206 | response_json.learnables.map(learnable => { 207 | 208 | let row = []; 209 | 210 | //learning elements 211 | let learnable_el = `""`; 212 | if (learnable.learning_element) { 213 | has_learnable = true; 214 | learnable_el = `"${learnable.learning_element.replaceAll('"', '""')}"`; 215 | } else if (settings["download_media"] && learnable.screens["1"].item.kind === "audio" && learnable.screens["1"].item.value.length > 0) { 216 | has_learnable = true; 217 | let temp_audio_learns = []; 218 | learnable.screens["1"].item.value.map(audio_learn => {temp_audio_learns.push(audio_learn.normal)}); 219 | temp_audio_learns.forEach(course_media_urls.add, course_media_urls); 220 | learnable_el = `"` + temp_audio_learns.map(url => `[sound:${UniqueDecodedFilename(url)}]`).join("") + `"`; 221 | } else if (settings["download_media"] && learnable.screens["1"].item.kind === "image" && learnable.screens["1"].item.value.length > 0) { 222 | has_learnable = true; 223 | let temp_image_learns = []; 224 | learnable.screens["1"].item.value.map(image_learn => {temp_image_learns.push(image_learn)}); 225 | temp_image_learns.forEach(course_media_urls.add, course_media_urls); 226 | learnable_el = `"` + temp_image_learns.map(url => `

`).join(``) + `"`; 227 | } 228 | row.push(learnable_el); 229 | 230 | //definitions 231 | let definition = `""`; 232 | if (learnable.definition_element) { 233 | has_definitions = true; 234 | definition = `"${learnable.definition_element.replaceAll('"', '""')}"`; 235 | } else if (settings["download_media"] && learnable.screens["1"].definition.kind === "audio" && learnable.screens["1"].definition.value.length > 0) { 236 | has_definitions = true; 237 | let temp_audio_defs = []; 238 | learnable.screens["1"].definition.value.map(audio_def => {temp_audio_defs.push(audio_def.normal)}); 239 | temp_audio_defs.forEach(course_media_urls.add, course_media_urls); 240 | definition = `"` + temp_audio_defs.map(url => `[sound:${UniqueDecodedFilename(url)}]`).join("") + `"`; 241 | } else if (settings["download_media"] && learnable.screens["1"].definition.kind === "image" && learnable.screens["1"].definition.value.length > 0) { 242 | has_definitions = true; 243 | let temp_image_defs = []; 244 | learnable.screens["1"].definition.value.map(image_def => {temp_image_defs.push(image_def)}); 245 | temp_image_defs.forEach(course_media_urls.add, course_media_urls); 246 | definition = `"` + temp_image_defs.map(url => `

`).join(``) + `"`; 247 | } 248 | row.push(definition); 249 | 250 | 251 | //audio 252 | let temp_audio_urls = []; 253 | if (settings["download_media"] && learnable.screens["1"].audio && learnable.screens["1"].audio.value.length > 0) { 254 | has_audio = true; 255 | learnable.screens["1"].audio.value.map(audio_item => {temp_audio_urls.push(audio_item.normal)}); 256 | temp_audio_urls.forEach(course_media_urls.add, course_media_urls); 257 | } 258 | row.push(`"` + temp_audio_urls.map(url => `[sound:${UniqueDecodedFilename(url)}]`).join("") + `"`); 259 | 260 | //video 261 | let temp_video_urls = []; 262 | if (settings["download_media"] && settings["videofiles_limit"] > 0 && learnable.screens["1"].video && learnable.screens["1"].video.value.length > 0) { 263 | has_video = true; 264 | learnable.screens["1"].video.value.map(video_item => {temp_video_urls.push(video_item)}); 265 | temp_video_urls.forEach(course_media_urls.add, course_media_urls); 266 | } 267 | row.push(`"` + temp_video_urls.map(url => `[sound:${UniqueDecodedFilename(url)}]`).join("") + `"`); 268 | 269 | //extra fields 270 | // attr[0]: 686844 - SS; 1995282 - PoS; 271 | let temp_extra1 = new Array(settings["max_extra_fields"]).fill(``); 272 | if (settings["extra_fields"] && learnable.screens["1"].attributes && learnable.screens["1"].attributes.length > 0) { 273 | learnable.screens["1"].attributes.forEach(attribute => { 274 | if (attribute && attribute.value && attribute.label) { 275 | let ind = attributes.indexOf(attribute.label); 276 | if (ind == -1 && attributes.length < settings["max_extra_fields"]) { 277 | attributes.push(attribute.label); 278 | } 279 | ind = attributes.indexOf(attribute.label); 280 | if (ind != -1) { 281 | temp_extra1[ind] = `"${attribute.value.replaceAll('"', '""')}"`; 282 | } 283 | } 284 | }) 285 | } 286 | temp_extra1.forEach(el => row.push(el)); 287 | 288 | // visible_info[0]: 548340 - kana; 6197256 - syn; 2021373+2021381 - lit trans/pinyin; 289 | // visible_info[1]: 2021373+2021381 - pinyin; 290 | let temp_extra2 = new Array(settings["max_extra_fields"]).fill(``); 291 | if (settings["extra_fields"] && learnable.screens["1"].visible_info && learnable.screens["1"].visible_info.length > 0) { 292 | learnable.screens["1"].visible_info.forEach(v_info => { 293 | if (v_info && v_info.value && v_info.label) { 294 | let ind = visible_info.indexOf(v_info.label); 295 | if (ind == -1 && visible_info.length < settings["max_extra_fields"]) { 296 | visible_info.push(v_info.label); 297 | } 298 | ind = visible_info.indexOf(v_info.label); 299 | if (ind != -1) { 300 | if (settings["download_media"] && v_info.kind === "audio" && v_info.value.length > 0) { 301 | let temp_audio_list = []; 302 | v_info.value.map(audio => {temp_audio_list.push(audio.normal)}); 303 | temp_audio_list.forEach(course_media_urls.add, course_media_urls); 304 | temp_extra2[ind] = `` + temp_audio_list.map(url => `[sound:${UniqueDecodedFilename(url)}]`).join("") + ``; 305 | } else if (settings["download_media"] && v_info.kind === "image" && v_info.value.length > 0) { 306 | let temp_image_list = []; 307 | v_info.value.map(image => {temp_image_list.push(image)}); 308 | temp_image_list.forEach(course_media_urls.add, course_media_urls); 309 | temp_extra2[ind] = `` + temp_image_list.map(url => `

`).join(``) + ``; 310 | } else if (v_info.kind !== "audio" && v_info.kind !== "image") { 311 | temp_extra2[ind] = `"${v_info.value.replaceAll('"', '""')}"`; 312 | } 313 | } 314 | } 315 | }) 316 | } 317 | temp_extra2.forEach(el => row.push(el)); 318 | 319 | // hidden_info[0]: 1995282 - inflections; 320 | let temp_extra3 = new Array(settings["max_extra_fields"]).fill(``); 321 | if (settings["extra_fields"] && learnable.screens["1"].hidden_info && learnable.screens["1"].hidden_info.length > 0) { 322 | learnable.screens["1"].hidden_info.forEach(h_info => { 323 | if (h_info && h_info.value && h_info.label) { 324 | let ind = hidden_info.indexOf(h_info.label); 325 | if (ind == -1 && hidden_info.length < settings["max_extra_fields"]) { 326 | hidden_info.push(h_info.label); 327 | } 328 | ind = hidden_info.indexOf(h_info.label); 329 | if (ind != -1) { 330 | if (settings["download_media"] && h_info.kind === "audio" && h_info.value.length > 0) { 331 | let temp_audio_list = []; 332 | h_info.value.map(audio => {temp_audio_list.push(audio.normal)}); 333 | temp_audio_list.forEach(course_media_urls.add, course_media_urls); 334 | temp_extra3[ind] = `` + temp_audio_list.map(url => `[sound:${UniqueDecodedFilename(url)}]`).join("") + ``; 335 | } else if (settings["download_media"] && h_info.kind === "image" && h_info.value.length > 0) { 336 | let temp_image_list = []; 337 | h_info.value.map(image => {temp_image_list.push(image)}); 338 | temp_image_list.forEach(course_media_urls.add, course_media_urls); 339 | temp_extra3[ind] = `` + temp_image_list.map(url => `

`).join("") + ``; 340 | } else if (h_info.kind !== "audio" && h_info.kind !== "image") { 341 | temp_extra3[ind] = `"${h_info.value.replaceAll('"', '""')}"`; 342 | } 343 | } 344 | } 345 | }) 346 | } 347 | temp_extra3.forEach(el => row.push(el)); 348 | 349 | //tags 350 | row.push(tags); 351 | 352 | //learnable IDs 353 | if (settings["learnable_ids"]) { 354 | try { 355 | row.push(learnable.id); 356 | } catch (err) { 357 | console.log(`no learnable id! ${err.name}: ${err.message}`, learnable); 358 | row.push(-1); 359 | } 360 | } 361 | table.push(row); 362 | 363 | }); 364 | 365 | err_count = 0; //reset number of consequent unretreaved levels if the current level scan succesfully reached the end 366 | } catch (err) { 367 | console.log(err); 368 | if (level_is_empty) { 369 | console.log(`${cidd['cid']}: Level ${levels_done} has no learnable items`); 370 | } else { 371 | //likely due to scanning being performed beyond number levels identified from metadata, which is left as a fallback in case parsed value appears incorrect 372 | console.log(`${cidd['cid']}: Level ${levels_done} does not exist or has no learnable words. Level skip count: ` + (err_count + 1)); 373 | err_count++; 374 | proceed = false; 375 | if (err_count < settings["max_level_skip"]) { 376 | proceed = true; // precaution in case value in settings has wrong type 377 | } 378 | } 379 | } 380 | 381 | 382 | //update progress bar and console log 383 | const done_clamped = Math.min(levels_done, meta['number of levels']); 384 | updScanProgress(threadN, cidd, isNaN(done_clamped) ? levels_done : done_clamped, meta['number of levels']); 385 | } 386 | 387 | if (stopped) { 388 | return "stopped"; 389 | } 390 | 391 | console.log('scanning complete - formatting data'); 392 | 393 | //final number of learnables 394 | if (meta['number of items'].includes("?")) { 395 | meta['number of items'] = "~" + table.length; 396 | } 397 | if (table.length < meta['number of items']) { 398 | console.warn(`${cidd['cid']}: Only ${table.length} out of ${meta['number of items']} learnables have been downloaded! (you might want to check your internet connection and retry)`); 399 | meta['number of items'] = `${table.length} of ${meta['number of items']}`; 400 | } 401 | 402 | //select non-empty fields 403 | const course_fields = []; 404 | if (has_learnable) {course_fields.push("Learnable")}; 405 | if (has_definitions) {course_fields.push("Definition")}; 406 | if (has_audio) {course_fields.push("Audio")}; 407 | if (has_video) {course_fields.push("Video")}; 408 | course_fields.push(...attributes); 409 | course_fields.push(...visible_info); 410 | course_fields.push(...hidden_info); 411 | if (settings["level_tags"]) {course_fields.push("Level tags")}; 412 | if (settings["learnable_ids"]) {course_fields.push("Learnable ID")}; 413 | meta['course fields'] = course_fields.join(" | "); 414 | 415 | // convert table to plain text (csv) 416 | let csv_data = table.map(row => { 417 | const line = []; 418 | if (has_learnable) {line.push(row[0])}; 419 | if (has_definitions) {line.push(row[1])}; 420 | if (has_audio) {line.push(row[2])}; 421 | if (has_video) {line.push(row[3])}; 422 | line.push(...row.slice(4, 4 + attributes.length)); 423 | line.push(...row.slice(4 + settings["max_extra_fields"], 4 + settings["max_extra_fields"] + visible_info.length)); 424 | line.push(...row.slice(4 + 2 * settings["max_extra_fields"], 4 + 2 * settings["max_extra_fields"] + hidden_info.length)); 425 | if (settings["level_tags"]) {line.push(row[4 + 3 * settings["max_extra_fields"]])}; 426 | if (settings["learnable_ids"]) {line.push(row[4 + 3 * settings["max_extra_fields"] + 1])}; 427 | return line.join(`,`); 428 | }).join("\n") + "\n"; 429 | //add Anki headers 430 | if (ANKI_HEADERS) { 431 | csv_data = "#separator:comma\n" + 432 | "#html:true\n" + 433 | (settings["level_tags"] ? (`#tags column:${settings["learnable_ids"] ? course_fields.length-1 : course_fields.length}\n`) : ``) + 434 | "#columns:" + course_fields.join(",") + "\n" + 435 | csv_data; 436 | } 437 | const csv_encoded = 'data:text/csv;charset=utf-8,%EF%BB%BF' + encodeURIComponent(csv_data); 438 | 439 | //names for directory and spreadsheet 440 | let course_filename, course_folder; 441 | if (settings["course_metadata"]) { 442 | course_filename = `${meta['url name']} [${cidd['cid']}]`; 443 | course_folder = `${meta['url name']} by ${meta['author']} [${cidd['cid']}]/`; 444 | } else { 445 | course_filename = `${meta['url name']} by ${meta['author']} [${cidd['cid']}]`; 446 | course_folder = ""; 447 | } 448 | 449 | if (settings["download_media"]) { 450 | console.log(`Media files found in ${meta['url name']}[${cidd['cid']}]: ${course_media_urls.size}`); 451 | }; 452 | 453 | //add all files to global queue 454 | file_queue.unshift([csv_encoded, `${course_folder}${course_filename}_(${meta['number of items'].toString()}).csv`]); 455 | if (settings["course_metadata"]) { 456 | file_queue.unshift([meta2txt(meta), `${course_folder}info.md`]); 457 | file_queue.unshift([meta['ava'], `${course_folder}${meta['author']}.${meta['ava'].split(".").slice(-1)}`]); 458 | file_queue.unshift([meta['thumbnail'], `${course_folder}${meta['url name']}.${meta['thumbnail'].split(".").slice(-1)}`]); 459 | } 460 | if (!settings["skip_media_download"]) { 461 | course_media_urls.forEach(url => { 462 | file_queue.push([url, `${course_folder}${course_filename}_media(${course_media_urls.size})/${UniqueDecodedFilename(url)}`]); 463 | }); 464 | } 465 | 466 | return "completed"; 467 | } 468 | 469 | async function scanThread(threadN, batch_size) { 470 | let cidd; 471 | 472 | while (cidd = cidds.pop()) { 473 | if (cidd['domain'] === tabDomain) { 474 | const closingEvent = await scanCourse(cidd, threadN); 475 | if (closingEvent === "stopped") { 476 | console.log(`thread ${threadN} (cid: ${cidd['cid']}) - scanning stopped by user`); 477 | return "stopped"; 478 | }; 479 | if (closingEvent === "unauthorised") { 480 | console.log(`thread ${threadN} (cid: ${cidd['cid']}) - user is not logged in, scanning terminated`); 481 | return "unauthorised"; 482 | }; 483 | } else { 484 | console.warn(`${cidd} does not match downloading tab domain ${tabDomain}`); 485 | } 486 | batch_done++; 487 | updBatchProgress(batch_done, batch_size, cidd); 488 | } 489 | 490 | removeScanBar(threadN); 491 | } 492 | 493 | 494 | //batch scheduling 495 | async function batchDownload() { 496 | //validity tests 497 | tabDomain = window.location.toString().split("/")[2]; 498 | if (tabDomain !== 'app.memrise.com' && tabDomain !== 'community-courses.memrise.com') { 499 | alert("The extension should be used on the memrise.com site"); 500 | return -1; 501 | } 502 | const test_fetch = await fetch(fetchURL(tabDomain)); 503 | if (!test_fetch.ok) { 504 | if (test_fetch.status === 401) { 505 | console.log("not logged in - terminating download"); 506 | alert("Memrise login required"); 507 | return "scanning unauthorised"; 508 | } 509 | } 510 | 511 | //global scan variables 512 | threads = []; 513 | batch_done = 0; //global 514 | file_queue = []; //global 515 | const batch_size = cidds.length; 516 | const progress_container = progressBarContainer(); 517 | //removing traces of a previous run 518 | while (progress_container.firstChild) {progress_container.removeChild(progress_container.firstChild);} 519 | progress_container.classList.remove('stopped'); progress_container.classList.remove('error'); 520 | document.querySelectorAll('div[id^="MemDump_progress-padding"]').forEach(element => {element.remove();}); 521 | 522 | for (let threadCounter = 0; threadCounter < settings['parallel_download_limit'] && cidds.length > 0; threadCounter++) { 523 | // console.log(`new thread started: ${threadCounter} ${settings['parallel_download_limit']} ${cidds.length}`); 524 | threads.push(scanThread(threadCounter, batch_size)); 525 | } 526 | updBatchProgress(batch_done, batch_size); 527 | const thread_res = await Promise.all(threads); 528 | 529 | if (thread_res.includes("unauthorised")) { 530 | console.log("all scanning threads stopped"); 531 | progress_container.classList.add('error'); 532 | threads = undefined; //reset state to enable restarting 533 | return "scanning unauthorised"; 534 | } 535 | if (thread_res.includes("stopped")) { 536 | console.log("all scanning threads stopped"); 537 | progress_container.classList.add('stopped'); 538 | threads = undefined; //reset state to enable restarting 539 | return "scanning stopped"; 540 | } 541 | 542 | console.log('All scanning and formating complete'); 543 | 544 | await sleep(500); 545 | updMediaProgress(0, file_queue.length); 546 | 547 | //downloading files 548 | function mediaDownloadMessages(arg, sender, sendResponse) { 549 | if (arg.type === "coursedump_error") { 550 | console.warn(`an error occured during file download ${arg.url} - ${arg.filename}`); 551 | console.error(arg.error); 552 | progress_container.classList.add('error'); 553 | } else if (arg.type === "coursedump_progressMedia_upd") { 554 | updMediaProgress(arg.done, arg.todo); 555 | } else if (arg.type === "coursedump_mediaFinished") { 556 | if (arg.status === "done") { 557 | updMediaProgress("done"); 558 | setTimeout(()=> { 559 | if (settings['anki_import_prompt'] && confirm('Would you like some help with importing the downloaded data into Anki?')) { 560 | window.open('https://github.com/Eltaurus-Lt/CourseDump2022#importing-into-anki', '_blank').focus(); 561 | }}, 200); 562 | } else if (arg.status === "stopped") { 563 | console.log("stopped during file download"); 564 | progress_container.classList.add('stopped'); 565 | } 566 | chrome.runtime.onMessage.removeListener(mediaDownloadMessages); 567 | threads = undefined; //reset state to enable restarting 568 | } 569 | } 570 | chrome.runtime.onMessage.addListener(mediaDownloadMessages); 571 | 572 | chrome.runtime.sendMessage({ 573 | type: "coursedump_downloadFiles", 574 | file_queue, 575 | maxThreads: settings["parallel_download_limit"] 576 | }); 577 | 578 | } 579 | 580 | 581 | 582 | //global variables 583 | if (typeof cidds === 'undefined') {var cidds = []} //should be defined as an argument passed from menu.js through background.js 584 | if (typeof batch_done === 'undefined') {var batch_done = 0} //global progress counter 585 | if (typeof file_queue === 'undefined') {var file_queue = []} //global list of files 586 | if (typeof threads !== 'undefined') { 587 | alert('Script is already executing on this page. Reload to retry'); //should be impossible to trigger if the menu state is correct 588 | } else { 589 | batchDownload(); 590 | } -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Memrise Course Dump 2 | This **Google Chrome** extension downloads word lists from [**Memrise**](https://memrise.com/) courses as ".csv" spreadsheets along with all associated images, audio, and video files. It also supports [batch download](https://github.com/Eltaurus-Lt/CourseDump2022?tab=readme-ov-file#batch-download) of Memrise courses. The format of the downloaded data is suitable for subsequent import into [**Anki**](https://apps.ankiweb.net/). 3 | 4 | The extension *does not* download personal study data (although it is planned to be added in the future). It also *does not* download the words you have marked as "ignored" on Memrise. You might want to unignore them before downloading a course or make a separate clean Memrise account specifically for downloading purposes. 5 | 6 | ## Downloading the Extension 7 | At the top of this page click `Code` and then `Download ZIP` (Note, that the `Code` button might be hidden if your browser window isn't wide enough) 8 |

9 | 10 | 11 |

12 | 13 | ## Installation 14 | 1. [Download](https://github.com/Eltaurus-Lt/CourseDump2022/archive/refs/heads/main.zip) the ***CourseDump2022-main.zip*** archive and extract ***CourseDump2022-main*** folder from it. At this step, you can move the extension folder to any place in your filesystem. 15 | 2. In *Google Chrome* click the `Extensions` button

and then

16 | _{(alternatively go to the Main menu in the top right corner and click `Extensions` → `Manage Extensions`)} 17 | 3. Enable `Developer mode` (top right corner of the page)

18 | Developer mode off->on

19 | 4. Choose `Load unpacked` (top left corner) and select the ***CourseDump2022-main*** folder extracted in step 1 20 | 5. (_optional_) Click the `Extensions` button from step 2 again and pin the extension to the toolbar by clicking the pin button

21 |

22 | 23 | ## 💡 Downloading a Memrise Course 24 | 25 | >--- 26 | >1. Make sure you are logged in on [Memrise](https://community-courses.memrise.com/) 27 | >2. Navigate to any page belonging to a course you want to download ([example-1](https://community-courses.memrise.com/community/course/1105/speak-esperanto-like-a-nativetm-1/), [example-2](https://app.memrise.com/community/course/2021573/french-1/3/)) 28 | >3. 🚩 **If you are downloading a course with a lot of media files** 🚩, make sure you have disabled the option `Ask where to save each file before downloading` in the Chrome settings (chrome://settings/downloads) 29 | >4. Press the extension icon and then click the "Download current course" button at the top of the menu 30 | > 31 | >

32 | > 33 | > 34 | >

35 | > 36 | > _{(if you don't see the extension icon on the toolbar, click the `Extensions` button to locate it)} 37 | > 38 | >5. \* **A download can be interrupted at any point** by pressing the `Stop ongoing download`, which will replace the `Download current course` button whenever there is a download in progress, but keep in mind that restarting a download will begin the whole process from scratch. 39 | >--- 40 | 41 | 42 | 43 | When a download starts you should see a progress bar at the top of the course page, indicating the progress of the extension scanning the course's contents with the ratio of the levels fetched to the total number of course levels in the top-right corner: 44 | 45 |

46 | 47 | 48 |

49 | 50 | The scanning will be followed by downloading all associated files (the ".csv" file containing table data of the course alongside the course metadata and media files if you choose to download them). The progress is indicated by a yellow bar with the ratio on the right showing the number of downloaded files to the total number of files in the queue: 51 | 52 |

53 | 54 | 55 |

56 | 57 | After a download is complete, you should see the progress bar turning green: 58 | 59 |

60 | 61 | 62 |

63 | 64 | The downloaded files should appear in your Chrome downloads directory, in a subfolder with the name comprised of the id, name, and author of that course: 65 | 66 |

67 | 68 | 69 |

70 | 71 | #### Checking download results 72 | 73 | For convenience, the names of the downloaded ".csv" file and the "...\_media" folder have the counts for the total number of the downloaded items (words) and the number of the referenced media files appended at the end in the brackets. 74 | 75 | If the number in the spreadsheet filename appears without any additional indicators, e.g. "...\_(123).csv", you can be sure that all items (**not counting the ignored ones**) from a course have been saved successfully. If the total number of the downloaded items does not match the count displayed on the Memrise page, both will be shown to indicate an incomplete download, e.g. "...\_(42 of 58).csv". In this instance, it is worth checking the internet connection and repeating the download. 76 | 77 | Unfortunately, Memrise displays the expected number on a course page only if the course has been started by the user or the course isn't split into levels. If this is not the case, the figure displayed in the ".csv" filename will be based on the total number of items in the levels the extension managed to scrap and prefixed with a tilde to indicate an estimation, e.g. "...\_(~77).csv". To verify that all items have been downloaded, you’ll need to compare this number to an independent evaluation (the easiest way would still be to enroll in the course by pressing the "Get started now" button, answering a couple of questions to enable the word count, and checking the main course page again to see the value displayed by Memrise). 78 | 79 | For the media files, it is enough to compare the number at the end of the "...\_media" folder's name with the actual number of the files it contains. 80 | If the two do not match, some files are likely to be missing due to connection issues (either on your side or on the side of the Memrise server). In most cases, simply retrying the download can fix the problem. You should not delete files from failed attempts – the extension will keep putting the files into the same media folder, resolving potential naming conflicts, so that even on unreliable networks several partially successful attempts can yield fully recovered course media data. 81 | 82 | ^{In some rare cases, however, Memrise courses might contain references to files that don't exist. Attempting to download a course with a broken link will result in the progress bar turning red during the download, and the respective error appearing on the "Manage extensions" page.} 83 | 84 | ### Batch download 85 | 86 | If you have multiple courses to download, instead of going through them one by one it is more convenient to add them to a queue by pressing the respective button in the "Batch download" section of the extension menu on each of the courses' pages: 87 | 88 | ![image](https://github.com/user-attachments/assets/46a0e23d-6bfb-4450-a993-0c40ceca5223) 89 | 90 | and then download all queued courses at once by clicking the "Download all" button (the numeral in brackets indicates the total number of currently queued courses): 91 | 92 | ![image](https://github.com/user-attachments/assets/d6ca790c-58c4-4ef1-95ea-2dfece5ae42c) 93 | 94 | ^{Note, that the download should still be initiated from (any) Memrise page since the extension needs an active Memrise login to access the data.} 95 | 96 | During the scanning phase of a batch download the progress for each course is displayed on a separate progress bar, marked by a course's name, with the total scanning progress showing on a separate bar at the bottom: 97 | 98 |

99 | 100 | 101 |

102 | 103 | The file download phase proceeds from there as usual, with files from all the courses being processed together as a single stack. 104 | 105 | If you have a list of courses in a text file somewhere (from one of the previous versions of the extension, for example), it can be imported with the "Import course list" menu button: 106 | 107 | ![image](https://github.com/user-attachments/assets/e2b49059-dd30-4c98-a2a0-0de78c07c32d) 108 | 109 | The extension will accept practically any course url format (with the lines, not recognized as valid course urls being treated as text comments) 110 |

111 |

examples

112 | 113 | ``` 114 | https://community-courses.memrise.com/community/course/234546/breaking-into-japanese-literature/ 115 | https://community-courses.memrise.com/community/course/1098515/german-4 116 | https://community-courses.memrise.com/community/course/1136018/ 117 | a plain-text comment 118 | https://community-courses.memrise.com/community/course/1098188 119 | community-courses.memrise.com/community/course/1891054/japanese-5/ 120 | https://community-courses.memrise.com/aprender/learn?course_id=5591215?recommendation_id=5144c220-f6cb-42d1-a677-ef922e3ddcb6 121 | https://community-courses.memrise.com/aprender/review?category_id=963 122 | community-courses.memrise.com/community/course/1136234/russisch-3 123 | 124 | community-courses.memrise.com/community/course/1136236 125 | https://community-courses.memrise.com/community/course/43290/advanced-english-for-native-speakers/2/ 126 | community-courses.memrise.com/community/course/867/ 127 | ``` 128 |

129 | 130 | Just make sure that each course url is placed on a separate line and points to an existing course page. The latter might not be the case if, for example, your link was saved before Memrise moved the community courses (you can try updating community courses urls by autoreplacing "app." domains in your list with "community-courses."). 131 | Note, that duplicate courses are removed from the queue, which might result in the number of courses in the queue after import being less than the number of entries in the source text file. 132 | The list of currently queued courses can be displayed by pressing the "View queued courses" button (opens in a new tab): 133 | 134 | ![image](https://github.com/user-attachments/assets/41777822-5d6b-42ca-9106-fb34ef0285dc) 135 | 136 | This can also be used for editing the list by copy-pasting it to a text editor, making the necessary changes, and then re-importing the result as a text file through the process described above. 137 | 138 | ## 💡 Importing into Anki 139 | 140 | >tl;dr (most basic import): 141 | >1. Make a note type: 142 | > 1. Open the downloaded ".csv" file → look at the number and names of the columns 143 | > 2. In Anki press `Tools` (top left menu) → `Manage Note Types` → `Add` → `Add: Basic` → put in a name (e.g. "Memrise - Japanese") → `OK` → `Fields` → add new/rename existing ones to match the columns from the ".csv" file ("Level tags" column excluded) → `Save` → close the window 144 | >2. Make a deck: press `Create Deck` (bottom center of the main Anki screen) → put in the course's name → `OK` 145 | >3. Import the spreadsheet: `File` (top left menu) → `Import` → browse to the ".csv" file → `Open` → set the `Notetype` and `Deck` (Import options section) to the ones created in the steps 1 and 2 → `Import` (top right corner) → wait for import to finish → close the window 146 | >4. Move the media files (if the course has any): `Tools` (top left menu) → `Check Media` → `View Files` (bottom left corner) → copy all files **from inside** your downloaded course's "..._media" subfolder to the opened "collection.media" one → close all windows 147 | 148 | ### 1. Choosing a note type 149 | 150 | _Note types_ are, essentially, the Anki equivalent of Memrise database templates (with course-level settings and card templates packed inside). 151 | The important thing at this step is to prepare such a **template with enough fields to accommodate all required columns of the imported course** (anything else can be modified afterward). 152 | You can check what columns a course has by opening the downloaded ".csv" file and looking at the row that starts with "#columns:" at the top of the table. 153 | 154 | There are several alternative ways you can go from here: 155 | 156 | 1. [Create a new note type](https://docs.ankiweb.net/editing.html#adding-a-note-type) by adding the required number of fields: 157 | 1. Press `Tools` (top left menu) → `Manage Note Types` 158 | 2. `Add` → `Add: Basic` (or clone any other template you wish to use as a basis) 159 | 3. Put in a name such as "Memrise – German" (if you are importing several courses on the same language/topic with similar level structure you can use the same note type for all of them – it will make managing cards easier in the long run) → `OK` 160 | 4. Press `Fields` → `Add` to add new fields up to the number of columns from the ".csv" file (ignore the "Level tags" column – it is special and does not require a field for import). Names for fields and columns do not have to match, but it is a good idea to keep them the same. You can rename existing fields by selecting them and pressing `Rename` → When finished, press `Save` and close the window 161 | 162 | Keep in mind, that **in order to see the content of a field during reviews, you will also have to edit the card templates** and put the field on the respective side of a card (refer to [the Anki manual](https://docs.ankiweb.net/templates/intro.html) for all the necessary steps; when in doubt, feel free to ask, even basic, questions on [the forum](https://forums.ankiweb.net/)). 163 | 2. Use **the dedicated [Memrise template](https://github.com/Eltaurus-Lt/Anki-Card-Templates?tab=readme-ov-file#memrise)**, which replicates the original Memrise design and most of its functionality. The template is set to have five fields by default: "Learnable", "Definition", "Audio" + two extra fields. The instructions for adding more fields or renaming the existing ones can be found in [the customization section](https://github.com/Eltaurus-Lt/Anki-Card-Templates?tab=readme-ov-file#customization). 164 |

165 | 166 | 167 |

168 | 3. For importing the most basic data (learnable + definition) – any note type will do, and you can simply use Anki's "Basic" one without any modifications. 169 | 170 | For basic data with media (audio and video) you can use any of the templates provided with the Extension – "Basic (with media)", "Basic (and reversed card with media)", "Basic (reading, writing, and listening)", which differ in the number and types of questions they've been set up to produce. To import the templates: 171 | 172 | 1. Double-click the `Anki Templates.apkg` file found in the [***CourseDump2022-main***](https://github.com/Eltaurus-Lt/CourseDump2022?tab=readme-ov-file#downloading-the-extension) folder (alternatively, press `File` → `Import` inside Anki and browse to the `.apkg` file) 173 | 2. The previous step creates an example deck with three cards in your Anki collection. This deck and the cards can be safely deleted right away if you don't need them. 174 | 175 | For making any adjustments to these templates refer to point 1 of this list 176 | 4. Look for a template elsewhere. Anki templates are distributed freely by users and can be found all over the internet. [AnkiWeb](https://ankiweb.net/shared/decks?search=template) is a good place to start 177 | 178 | ### 2. Making a deck (optional) 179 | 180 | Create a new deck for storing the cards made from the course: 181 | 182 | 1. Press `Create Deck` at the bottom of the main screen: 183 |

184 | 185 | 186 |

187 | 188 | 2. Put in the name of the course (you can copy the full name from the downloaded `info.md` file) → press `OK` 189 | 3. You can also set the course description found in the same `info.md` as the deck description: 190 | 1. Open the deck by clicking on its name 191 |

192 | 193 | 194 |

195 | 2. Press `Description` at the bottom of the screen 196 |

197 | 198 | 199 |

200 | 3. Copy the relevant text in the appeared window 201 | 4. To set a thumbnail you can use [this addon](https://github.com/Eltaurus-Lt/Lt-Anki-Addons/tree/main/Lstyle) and the image from the downloaded course folder. 202 | 203 | Note, that decks can be nested inside each other (via drag-and-drop) to group courses' decks by language/topic or subdivide a deck into subdecks representing levels. However, in Anki, you are able to quickly search for items from any course/level using tags even if you skip this step entirely and don't make a separate deck for each course. 204 | 205 | ### 3. Importing the spreadsheet 206 | 207 | 1. Press `File` (top left menu) → `Import` 208 | 2. Browse to the ".csv" file in [the downloaded course's folder](https://github.com/Eltaurus-Lt/CourseDump2022?tab=readme-ov-file#-downloading-a-memrise-course) → `Open` 209 | 3. Verify that the table in the "File" section looks good and set the `Notetype` and `Deck` in the "Import options" section below to the ones prepared in the previous steps: 210 |

211 | 212 | Anki Browser Deck list 213 |

214 | 4. Scroll down and adjust the "Field Mapping". This tab defines the correspondence between the Anki field names and the former Memrise field names (spreadsheet columns). All the matching names will be mapped to each other automatically (so you can skip this step if you named the fields in the chosen note type accordingly). The rest can be set up manually on the right side of the tab, by selecting the column names next to the names of the Anki fields, into which you would like to put the respective data. Columns that are not mapped to any fields will be skipped during import. 215 |

216 | 217 | 218 |

219 | 5. Press `Import` in the top right corner. After the processing is done, you will see a report screen, with the "Overview" section indicating the overall count of imported notes: 220 |

221 | 222 | Anki Import report 223 |

224 | 225 | You can compare it against [the total number of items in the course](https://github.com/Eltaurus-Lt/CourseDump2022?tab=readme-ov-file#checking-download-results). The "Details" section below will contain information on each individual note from the spreadsheet. 226 | 227 | You will also be able to see the imported notes at any time by going to `Browse` (top center menu) and selecting your deck in the deck list on the left side 228 |

229 | 230 | 231 |

232 | 233 | ### 4. Moving media files 234 | 235 | 1. To locate the "collection.media" folder, in which Anki stores all media files, press `Tools` (top left menu) → `Check Media` → `View Files` (bottom left corner) 236 | 237 |

238 |

The default paths on different systems for opening the folder manually:

239 | 240 | * Windows: `%APPDATA%\Anki2\[your Anki username]\collection.media` 241 | * Mac: `~/Library/Application Support/Anki2/[your Anki username]/collection.media` (the Library folder is hidden by default, but can be revealed in Finder by holding down the option key while clicking on the Go menu) 242 | * Linux: `~/.local/share/Anki2/[your Anki username]/collection.media` for native installs or `~/.var/app/net.ankiweb.Anki/data/Anki2/[your Anki username]/collection.media` for flatpak installs 243 |

244 | 245 | 2. Move all the files from the course's "..._media" subfolder into the "collection.media" folder. Note, that you should move the files themselves, [**without the subfolder**](https://docs.ankiweb.net/importing.html#importing-media) containing them 246 | 247 | To verify that all media files are properly referenced, after they have been completely moved close and reopen the `Tools` → `Check Media` window. If you don't see any missing or unused files from the imported course, everything is alright. 248 | 249 | 250 | ## Settings 251 | 252 | ### Basic 253 | 254 | ![image](https://github.com/user-attachments/assets/cb0f8d77-7101-4b4e-819d-f964d5516c81) 255 | 256 | 1. **Download media**: Enables downloading images, audio, and video files associated with a Memrise course 257 | 2. **Extra fields**: Enables downloading all fields found in a course. Typical examples include "part of speech", "sample sentence", "transcription", "literal translation", etc. The labels for each field can be found in the downloaded ".csv" file. Turning this option off will limit fields to the basic set of the "Learnable", "Definition", "Audio", "Video", and "Tags" (the latter three can be also excluded by the respective settings) 258 | 3. **Level tags**: Appends an additional column that saves Memrise course level structure in the format `course_name::level##`, which is automatically converted to [hierarchical tags](https://docs.ankiweb.net/editing.html?highlight=tags#using-tags) in Anki during import 259 | 4. **Anki import prompt**: Enables a popup leading to the current readme page after each download. If you are reading this, you already successfully found your way here and might want to turn it off 260 | 261 | ### Advanced 262 | 263 | ![image](https://github.com/user-attachments/assets/f90301a8-538a-44b6-9ffb-734b4e5f5bc8) 264 | 265 | 5. **Learnable IDs**: Appends an additional column to the course spreadsheet containing a unique ID for each item. Can be used to manage duplicates inside Anki (if imported into the sorting field), or to cross-reference against other Memrise data downloaded separately, such as [archived mems](https://github.com/Eltaurus-Lt/MemDump) 266 | 6. **Video files**: Allows excluding video files from a download: when turned off overwrites the `Download media` setting for video files while leaving images and audio unaffected (has no effect if the `Download media` toggle is turned off) 267 | 7. **Skip media download**: Allows skipping media files during the file download phase. In contrast to the `Download media` setting, does not remove the respective columns from the spreadsheet when turned off. It can be helpful if a course spreadsheet needs to be recompiled with different settings without downloading the whole media folder again 268 | 8. **Course metadata**: Enables downloading three metadata files in addition to the basic spreadsheet and media: an `info.md` file containing the text description of a course, the course's thumbnail image, and the course author's avatar. When turned off, the ".csv" spreadsheet and respective media folder (when applicable) will be placed directly into the Chrome download folder, instead of being bundled together with meta files in a separate course folder 269 | 270 | ## Discussion 271 | If you encounter errors, have further questions regarding the extension, or need any help with using the downloaded materials in Anki, please leave a comment in this thread: [An alternative to Memrise2Anki](https://forums.ankiweb.net/t/an-alternative-to-memrise2anki-support-thread/30084) 272 | --------------------------------------------------------------------------------