├── .gitignore ├── .npmignore ├── .vscode └── settings.json ├── LICENSE.md ├── README.md ├── package.json ├── src └── index.ts └── tsconfig.json /.gitignore: -------------------------------------------------------------------------------- 1 | yarn.lock 2 | node_modules 3 | build 4 | -------------------------------------------------------------------------------- /.npmignore: -------------------------------------------------------------------------------- 1 | node_modules 2 | -------------------------------------------------------------------------------- /.vscode/settings.json: -------------------------------------------------------------------------------- 1 | { 2 | "editor.formatOnSave": true 3 | } -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2019 H4 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # NOT MAINTAINED 2 | 3 | This project is no longer maintained. Please use tsc project mode - it should now be mature enough to work well in most situations. Alternatively, use [turborepo](https://turborepo.org/) which has pretty good caching for any sort of command: 4 | 5 | # ctsc 6 | 7 | Experimental caching TypeScript compiler suitable for monorepos and CI. 8 | 9 | This acts as a drop-in replacement for TypeScript's own `tsc`. It will store the compiled files in a cache directory at `$CTSC_CACHE_DIR` (default to `/tmp/ctsc`). The next time the compiler is invoked for a particular module, the cached output will be re-used if all the inputs are the same (checked with git hash-object, sha1). 10 | 11 | #### how it differs from tsc project mode 12 | 13 | With the introduction of project mode, local caching was added, however, it has 3 significant restrictions: 14 | 15 | - it requires that all modules compile with declarations. This means that the toplevel application packages cannot easily be included in the project mode and have to be compiled separately. 16 | - it requires you to specify the dependant projects, even though they can be read from `package.json`. This means double book-keeping. 17 | - it only reuses files found in the local directory. This means it cannot be used to speed up compilation in CI by having a shared cache directory containing most of the unchanged modules, precompiled. 18 | 19 | ctsc has a different set of limitations, which are somewhat less restrictive: 20 | 21 | - it requires the `outDir` setting, so it can more easily cache the output directory properly. This 22 | limitation may be removable in the future if we decide to add a filter of the files to copy, as well 23 | as use a better copying solution such as `rsync` or `cpx` (slow) 24 | - it requires the `include` setting, so it can more easily read all the inputs. This limitation may 25 | also be removable in the future if we decide to add a filter on the files to use when calculating 26 | the input hash. 27 | 28 | Most monorepo projects already specify an `outDir` as `build` and `include` directories (e.g. src, 29 | tests, etc) so we believe these limitations are ok. Let us know if you disagree. 30 | 31 | ## install 32 | 33 | yarn add ctsc 34 | 35 | Requirements: 36 | 37 | - (windows): WSL - windows subsystem for linux (find, xargs, tail) 38 | - git (for `git hash-object`) 39 | 40 | ## usage 41 | 42 | Within a package dir: 43 | 44 | ctsc [-p tsconfig.json] 45 | 46 | The package must have at least the following configuration in tsconfig: 47 | 48 | - include - must be an array of directories or files to include 49 | - compilerOptions.outDir - must exist and be a target output directory 50 | 51 | and its workspace dependencies must be referenced in package.json `dependencies` or 52 | `devDependencies` 53 | 54 | Then you can use it with [wsrun](https://github.com/hfour/wsrun) 55 | 56 | yarn wsrun --staged -r ctsc 57 | 58 | To prune old items from the cache (use env var CTSC_TMP_MAX_ITEMS to limit the cache size) 59 | 60 | ctsc --clean 61 | 62 | To purge the entire cache 63 | 64 | ctsc --purge 65 | 66 | ## how it works 67 | 68 | 1. ctsc calculates two types of hash: input files hash IHASH, and output (type) hash, OHASH 69 | 2. When ctsc is used to build the package `B` for the first time, a `.ctsc.hash` file is inserted into the output, containing the `OHASH` - a hash computed from all the outout .d.ts files produced by the compilation. 70 | 3. If package `A`'s dependency is `B`, the OHASH of `B` from its `.ctsc.hash` file is included when calculating the input hash `IHASH` hash of package `A`. In addition, all the sources specified in `include` are also considered when calculating the IHASH. 71 | 4. If this input hash matches a directory in `$CTSC_CACHE_DIR` named `$IHASH`, `tsc` is not invoked. Instead, the outDir is copied from the cache directly to the destination. 72 | 5. With this scheme, if a depedency has a change in its type definitions, it will propagate a rebuild to all its dependants. If the package doesn't have a types change, a rebuild will not propagate. 73 | 6. If the dependants themselves end up with a type definition change, rebuild propagation will continue further. 74 | 75 | ### does this mean it can work with dependencies in node_modules too? 76 | 77 | Yes! If the dependency was compiled with ctsc, a `.ctsc.hash` will be included in the output directory. If this hash of the definition files has changed, any module that depends on that dependency will be rebuilt by ctsc when you install the new version via yarn/npm! 78 | -------------------------------------------------------------------------------- /package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "ctsc", 3 | "version": "1.1.0", 4 | "description": "A caching compiler for TypeScript monorepos", 5 | "bin": { 6 | "ctsc": "build/index.js" 7 | }, 8 | "scripts": { 9 | "prepublish": "tsc", 10 | "build": "tsc", 11 | "watch": "tsc -w" 12 | }, 13 | "license": "MIT", 14 | "devDependencies": { 15 | "@types/mkdirp": "^0.5.2", 16 | "@types/node": "^10.12.18", 17 | "@types/require-relative": "^0.8.0", 18 | "@types/yargs": "^12.0.4", 19 | "typescript": "^3.2.2" 20 | }, 21 | "dependencies": { 22 | "@types/glob": "^7.1.1", 23 | "glob": "^7.1.3", 24 | "mkdirp": "^0.5.1", 25 | "require-relative": "^0.8.7", 26 | "require-times": "^1.1.0", 27 | "tsconfig": "^7.0.0", 28 | "yargs": "^13.2.1" 29 | }, 30 | "files": [ 31 | "build/**/*" 32 | ] 33 | } 34 | -------------------------------------------------------------------------------- /src/index.ts: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env node 2 | 3 | import yargs from 'yargs'; 4 | import path from 'path'; 5 | import cp from 'child_process'; 6 | import * as tscfg from 'tsconfig'; 7 | import util from 'util'; 8 | import fs from 'fs'; 9 | import rrelative from 'require-relative'; 10 | import mkdirp from 'mkdirp'; 11 | import os from 'os'; 12 | 13 | let mkdirpAsync = util.promisify(mkdirp); 14 | 15 | let copyAsync = async (src: string, dest: string, _opts?: any) => { 16 | await mkdirpAsync(dest); 17 | let copyProcess = await cp.spawnSync('bash', ['-c', `cp -r ${src}/. ${dest}`]); 18 | if (copyProcess.status! > 0) { 19 | console.error(copyProcess.stderr.toString()); 20 | process.exit(1); 21 | } 22 | }; 23 | 24 | let hashSync = (items: string[], criteria: string = '') => { 25 | let cmd = `find ${items.join( 26 | ' ' 27 | )} -type f ${criteria} | xargs tail -n +1 | git hash-object --stdin`; 28 | let hashRes = cp.spawnSync('bash', ['-c', cmd], { encoding: 'utf8' }); 29 | if (hashRes.status! > 0) { 30 | throw new Error(hashRes.stderr); 31 | } 32 | let hash = hashRes.stdout.trim(); 33 | return hash; 34 | }; 35 | 36 | let readAsync = util.promisify(fs.readFile); 37 | let readdirAsync = util.promisify(fs.readdir); 38 | let existAsync = util.promisify(fs.exists); 39 | let renameAsync = util.promisify(fs.rename); 40 | let statAsync = util.promisify(fs.stat); 41 | let utimesAsync = util.promisify(fs.utimes); 42 | let rmdirAsync = (path: string) => { 43 | let rm = cp.spawnSync('bash', ['-c', `rm -r ${path}`]); 44 | if (rm.status! > 0) { 45 | throw new Error(rm.stderr.toString()); 46 | } 47 | }; 48 | 49 | const HASH_FILE_NAME = '.ctsc.hash'; 50 | 51 | async function main() { 52 | let CTSC_TMP_DIR = process.env['CTSC_TMP_DIR'] || os.tmpdir() + '/ctsc'; 53 | let CTSC_TMP_MAX_ITEMS = Number(process.env['CTSC_TMP_MAX_ITEMS'] || '300'); 54 | let argv = yargs.options({ 55 | p: { 56 | type: 'string', 57 | description: 'Project config file' 58 | }, 59 | clean: { 60 | type: 'boolean', 61 | description: 'Clean up stale cached items' 62 | }, 63 | purge: { 64 | type: 'boolean', 65 | description: 'Clear the entire cache' 66 | } 67 | }).argv; 68 | 69 | if (argv.clean) await cleanup({ tmpdir: CTSC_TMP_DIR, maxItems: CTSC_TMP_MAX_ITEMS }); 70 | else if (argv.purge) await cleanup({ tmpdir: CTSC_TMP_DIR, maxItems: 0 }); 71 | else await compile({ tsconfig: argv.p, tmpdir: CTSC_TMP_DIR }); 72 | } 73 | 74 | async function cleanup(opts: { tmpdir: string; maxItems: number }) { 75 | let hashList = await readdirAsync(opts.tmpdir); 76 | let dirList = await Promise.all( 77 | hashList.map(async hl => ({ 78 | path: path.resolve(opts.tmpdir, hl), 79 | stat: await statAsync(path.resolve(opts.tmpdir, hl)) 80 | })) 81 | ); 82 | 83 | let cleanup = dirList.sort((i1, i2) => i2.stat.atimeMs - i1.stat.atimeMs).slice(opts.maxItems); 84 | 85 | console.log('Cleaning up', cleanup.length, 'cached items'); 86 | for (let item of cleanup) { 87 | await rmdirAsync(item.path); 88 | } 89 | } 90 | 91 | async function load(cwd: string, filename?: string) { 92 | let tsConfig = await tscfg.load(cwd, filename); 93 | let extend = tsConfig?.config?.extends; 94 | if (extend) { 95 | let parent = await load(cwd,extend); 96 | tsConfig.config = Object.assign(parent.config,tsConfig.config); 97 | } 98 | return tsConfig; 99 | } 100 | 101 | async function compile(opts: { tsconfig: string | undefined; tmpdir: string }) { 102 | let process_cwd = process.cwd(); 103 | let tsConfig = null; 104 | try { 105 | tsConfig = await load(process_cwd, opts.tsconfig); 106 | } catch (e) {} 107 | 108 | if (!tsConfig || !tsConfig.config) { 109 | console.error('No tsconfig.json found at', process_cwd, opts.tsconfig || ''); 110 | return process.exit(0); 111 | } 112 | 113 | let includes: string[] = tsConfig.config.include; 114 | if (!includes) { 115 | console.error('No include found in tsconfig', tsConfig.path); 116 | return process.exit(0); 117 | } 118 | 119 | let outDir = tsConfig.config.compilerOptions.outDir; 120 | if (!outDir) { 121 | console.error('No compilerOptions.outDir found in tsconfig', tsConfig.path); 122 | return process.exit(0); 123 | } 124 | 125 | let pkgJson = null; 126 | try { 127 | pkgJson = JSON.parse(await readAsync(path.resolve(process_cwd, 'package.json'), 'utf8')); 128 | } catch (e) {} 129 | 130 | if (!pkgJson) { 131 | console.error('package.json not found at', process_cwd); 132 | return process.exit(0); 133 | } 134 | let deps = Object.keys(pkgJson.dependencies || {}).concat( 135 | Object.keys(pkgJson.devDependencies || {}) 136 | ); 137 | 138 | let depsTsconfigs = deps.map(dep => { 139 | try { 140 | return { dep, dir: path.dirname(rrelative.resolve(dep + '/tsconfig.json', process_cwd)) }; 141 | } catch (e) { 142 | return null; 143 | } 144 | }); 145 | let tsconfigs = await Promise.all( 146 | depsTsconfigs.map(async tsc => { 147 | if (!tsc) return null; 148 | try { 149 | let loaded = await tscfg.load(tsc.dir, 'tsconfig.json'); 150 | return Object.assign(tsc, { config: loaded.config }); 151 | } catch (e) { 152 | return null; 153 | } 154 | }) 155 | ); 156 | let allFiles: string[] = []; 157 | for (let dep of tsconfigs) { 158 | if (dep && dep.config && dep.config.compilerOptions && dep.config.compilerOptions.outDir) { 159 | let item = path.resolve(dep.dir, dep.config.compilerOptions.outDir, HASH_FILE_NAME); 160 | if (!(await existAsync(item))) continue; 161 | allFiles.push(path.relative(process_cwd, item)); 162 | } 163 | } 164 | 165 | allFiles = allFiles.concat(includes).sort(); 166 | if (tsConfig.path) allFiles = allFiles.concat(tsConfig.path); 167 | 168 | let hash = hashSync(allFiles); 169 | let hashDir = path.resolve(opts.tmpdir, hash); 170 | let outDirFull = path.resolve(process_cwd, outDir); 171 | if (await existAsync(hashDir)) { 172 | await copyAsync(hashDir, outDirFull); 173 | await utimesAsync(hashDir, Date.now(), Date.now()); 174 | } else { 175 | let tsConfig = opts.tsconfig || 'tsconfig.json'; 176 | let out = cp.spawnSync('tsc', ['-p', tsConfig], { encoding: 'utf8' }); 177 | if (out.stdout) console.log(out.stdout); 178 | if (out.stderr) console.error(out.stderr); 179 | if (out.status != 0) { 180 | process.exit(out.status!); 181 | } else { 182 | let hashOut = hashSync([outDir], "-name '*.d.ts'"); 183 | fs.writeFileSync(path.resolve(outDirFull, HASH_FILE_NAME), hashOut); 184 | let rnd = Math.random(); 185 | await copyAsync(outDirFull, hashDir + rnd); 186 | 187 | // rename is atomic 188 | await renameAsync(hashDir + rnd, hashDir); 189 | } 190 | } 191 | } 192 | 193 | main(); 194 | -------------------------------------------------------------------------------- /tsconfig.json: -------------------------------------------------------------------------------- 1 | { 2 | "compilerOptions": { 3 | /* Basic Options */ 4 | "target": "es6", /* Specify ECMAScript target version: 'ES3' (default), 'ES5', 'ES2015', 'ES2016', 'ES2017','ES2018' or 'ESNEXT'. */ 5 | "module": "commonjs", /* Specify module code generation: 'none', 'commonjs', 'amd', 'system', 'umd', 'es2015', or 'ESNext'. */ 6 | // "lib": [], /* Specify library files to be included in the compilation. */ 7 | // "allowJs": true, /* Allow javascript files to be compiled. */ 8 | // "checkJs": true, /* Report errors in .js files. */ 9 | // "jsx": "preserve", /* Specify JSX code generation: 'preserve', 'react-native', or 'react'. */ 10 | // "declaration": true, /* Generates corresponding '.d.ts' file. */ 11 | // "declarationMap": true, /* Generates a sourcemap for each corresponding '.d.ts' file. */ 12 | // "sourceMap": true, /* Generates corresponding '.map' file. */ 13 | // "outFile": "./", /* Concatenate and emit output to single file. */ 14 | "outDir": "./build", /* Redirect output structure to the directory. */ 15 | // "rootDir": "./", /* Specify the root directory of input files. Use to control the output directory structure with --outDir. */ 16 | // "composite": true, /* Enable project compilation */ 17 | // "removeComments": true, /* Do not emit comments to output. */ 18 | // "noEmit": true, /* Do not emit outputs. */ 19 | // "importHelpers": true, /* Import emit helpers from 'tslib'. */ 20 | // "downlevelIteration": true, /* Provide full support for iterables in 'for-of', spread, and destructuring when targeting 'ES5' or 'ES3'. */ 21 | // "isolatedModules": true, /* Transpile each file as a separate module (similar to 'ts.transpileModule'). */ 22 | 23 | /* Strict Type-Checking Options */ 24 | "strict": true, /* Enable all strict type-checking options. */ 25 | // "noImplicitAny": true, /* Raise error on expressions and declarations with an implied 'any' type. */ 26 | // "strictNullChecks": true, /* Enable strict null checks. */ 27 | // "strictFunctionTypes": true, /* Enable strict checking of function types. */ 28 | // "strictBindCallApply": true, /* Enable strict 'bind', 'call', and 'apply' methods on functions. */ 29 | // "strictPropertyInitialization": true, /* Enable strict checking of property initialization in classes. */ 30 | // "noImplicitThis": true, /* Raise error on 'this' expressions with an implied 'any' type. */ 31 | // "alwaysStrict": true, /* Parse in strict mode and emit "use strict" for each source file. */ 32 | 33 | /* Additional Checks */ 34 | // "noUnusedLocals": true, /* Report errors on unused locals. */ 35 | // "noUnusedParameters": true, /* Report errors on unused parameters. */ 36 | // "noImplicitReturns": true, /* Report error when not all code paths in function return a value. */ 37 | // "noFallthroughCasesInSwitch": true, /* Report errors for fallthrough cases in switch statement. */ 38 | 39 | /* Module Resolution Options */ 40 | // "moduleResolution": "node", /* Specify module resolution strategy: 'node' (Node.js) or 'classic' (TypeScript pre-1.6). */ 41 | // "baseUrl": "./", /* Base directory to resolve non-absolute module names. */ 42 | // "paths": {}, /* A series of entries which re-map imports to lookup locations relative to the 'baseUrl'. */ 43 | // "rootDirs": [], /* List of root folders whose combined content represents the structure of the project at runtime. */ 44 | // "typeRoots": [], /* List of folders to include type definitions from. */ 45 | // "types": [], /* Type declaration files to be included in compilation. */ 46 | // "allowSyntheticDefaultImports": true, /* Allow default imports from modules with no default export. This does not affect code emit, just typechecking. */ 47 | "esModuleInterop": true /* Enables emit interoperability between CommonJS and ES Modules via creation of namespace objects for all imports. Implies 'allowSyntheticDefaultImports'. */ 48 | // "preserveSymlinks": true, /* Do not resolve the real path of symlinks. */ 49 | 50 | /* Source Map Options */ 51 | // "sourceRoot": "", /* Specify the location where debugger should locate TypeScript files instead of source locations. */ 52 | // "mapRoot": "", /* Specify the location where debugger should locate map files instead of generated locations. */ 53 | // "inlineSourceMap": true, /* Emit a single file with source maps instead of having a separate file. */ 54 | // "inlineSources": true, /* Emit the source alongside the sourcemaps within a single file; requires '--inlineSourceMap' or '--sourceMap' to be set. */ 55 | 56 | /* Experimental Options */ 57 | // "experimentalDecorators": true, /* Enables experimental support for ES7 decorators. */ 58 | // "emitDecoratorMetadata": true, /* Enables experimental support for emitting type metadata for decorators. */ 59 | }, 60 | "include": ["src"] 61 | } --------------------------------------------------------------------------------