Anakin Skywalker
', 'b'); -- "Skywalker" 58 | ``` 59 | - repo: sqlite-lines 60 | lang: C 61 | platforms: [linux-x86_64, macos-x86_64, macos-aarch64] 62 | short: Read files/blob line-by-line 63 | long: | 64 | 65 | A SQLite extension for efficiently reading large files or blobs line-by-line. 66 | 67 | ```sql 68 | select line from lines_read('logs.txt'); 69 | ``` 70 | - repo: sqlite-path 71 | lang: C 72 | platforms: [linux-x86_64, macos-x86_64, macos-aarch64, windows-x86_64] 73 | short: Parse and generate filepaths 74 | long: | 75 | 76 | A SQLite extension for parsing, generating, and querying paths. 77 | 78 | ```sql 79 | select path_dirname('foo/bar.txt'); -- 'foo/' 80 | select path_basename('foo/bar.txt'); -- 'bar.txt' 81 | select path_extension('foo/bar.txt'); -- '.txt' 82 | ``` 83 | - repo: sqlite-url 84 | lang: C 85 | platforms: [linux-x86_64, macos-x86_64] 86 | short: Parse and generate URLs 87 | long: | 88 | 89 | A SQLite extension for parsing, generating, and querying URLs and query strings 90 | 91 | - Extract specific parts of a URL, like schemes, hostnames, and paths 92 | - Generate URLs with parts safely 93 | - Iterate through all key and values in a query string 94 | 95 | ```sql 96 | select url_valid('https://sqlite.org'); -- 1 97 | select url_scheme('https://www.sqlite.org/vtab.html#usage'); -- 'https' 98 | select url_host('https://www.sqlite.org/vtab.html#usage'); -- 'www.sqlite.org' 99 | ``` 100 | - repo: sqlite-xsv 101 | lang: Rust 102 | platforms: [linux-x86_64, macos-x86_64, macos-aarch64, windows-x86_64] 103 | no_datasette: true 104 | no_sqlite_utils: true 105 | short: Query CSVs 106 | long: | 107 | 108 | A fast and performant SQLite extension for CSV files, written in Rust! 109 | 110 | - Query CSVs, TSVs, and other-SVs as SQLite virtual tables 111 | - The "reader" interface lets you query CSVs from other data sources, such as [`sqlite-http`](#sqlite-http) 112 | - Builtin support for querying CSVs with gzip or zstd compression 113 | 114 | ```sql 115 | create virtual table temp.students using csv( 116 | filename="students.csv" 117 | ); 118 | 119 | select * from temp.students; 120 | 121 | create virtual table temp.students_gz using csv( 122 | filename="students.csv.gz" 123 | ); 124 | 125 | create virtual table temp.students_reader using csv_reader( 126 | id integer, 127 | name text, 128 | age integer, 129 | progess real 130 | ); 131 | select * from temp.students_reader('./target/students.csv'); 132 | ``` 133 | - repo: sqlite-regex 134 | lang: Rust 135 | platforms: [linux-x86_64, macos-x86_64, macos-aarch64, windows-x86_64] 136 | short: Regular expression functions 137 | long: | 138 | 139 | A fast SQLite extension for regular expressions. 140 | 141 | ```sql 142 | select regex_find( 143 | '[0-9]{3}-[0-9]{3}-[0-9]{4}', 144 | 'phone: 111-222-3333' 145 | ); 146 | 147 | select rowid, * 148 | from regex_find_all( 149 | '\b\w{13}\b', 150 | 'Retroactively relinquishing remunerations is reprehensible.' 151 | ); 152 | /* 153 | ┌───────┬───────┬─────┬───────────────┐ 154 | │ rowid │ start │ end │ match │ 155 | ├───────┼───────┼─────┼───────────────┤ 156 | │ 0 │ 0 │ 13 │ Retroactively │ 157 | │ 1 │ 14 │ 27 │ relinquishing │ 158 | │ 2 │ 28 │ 41 │ remunerations │ 159 | │ 3 │ 45 │ 58 │ reprehensible │ 160 | └───────┴───────┴─────┴───────────────┘ 161 | */ 162 | ``` 163 | - repo: sqlite-ulid 164 | lang: Rust 165 | platforms: [linux-x86_64, macos-x86_64, macos-aarch64, windows-x86_64] 166 | short: Work with ULIDs 167 | long: | 168 | 169 | A SQLite extension for generating and working with [ULIDs](https://github.com/ulid/spec). 170 | 171 | ```sql 172 | select ulid(); -- '01gqr4j69cc7w1xdbarkcbpq17' 173 | select ulid_bytes(); -- X'0185310899dd7662b8f1e5adf9a5e7c0' 174 | select ulid_with_prefix('invoice'); -- 'invoice_01gqr4jmhxhc92x1kqkpxb8j16' 175 | select ulid_datetime('01gqr4j69cc7w1xdbarkcbpq17') -- '2023-01-26 22:53:20.556' 176 | ``` 177 | 178 | - repo: sqlite-jsonschema 179 | lang: Rust 180 | platforms: [linux-x86_64, macos-x86_64, macos-aarch64, windows-x86_64] 181 | short: Validate JSON objects with JSON Schema 182 | long: | 183 | A SQLite extension for validating JSON objects with [JSON Schema](https://json-schema.org/). 184 | 185 | ```sql 186 | select jsonschema_matches('{"maxLength": 5}', json_quote('alex')); -- 1 187 | ``` 188 | - repo: sqlite-fastrand 189 | lang: Rust 190 | platforms: [linux-x86_64, macos-x86_64, macos-aarch64, windows-x86_64] 191 | short: Generate fast numbers/blobs quickly 192 | long: | 193 | 194 | A SQLite extension for quickly generating random numbers, booleans, characters, and blobs 195 | 196 | ```sql 197 | select fastrand_int(); -- 556823563 198 | select fastrand_alphabetic(); -- 's' 199 | select fastrand_uppercase();-- 'M' 200 | select fastrand_double(); -- 0.740834390248454 201 | ``` 202 | - repo: sqlite-vss 203 | lang: C++ 204 | platforms: [linux-x86_64, macos-x86_64, macos-aarch64] 205 | short: Vector search in SQLite 206 | long: | 207 | A SQLite extension for efficient vector search, based on Faiss! 208 | 209 | ```sql 210 | create virtual table vss_articles using vss0( 211 | headline_embedding(384), 212 | description_embedding(384), 213 | ); 214 | 215 | insert into vss_articles(rowid, headline_embedding) 216 | select rowid, headline_embedding from articles; 217 | 218 | select rowid, distance 219 | from vss_articles 220 | where vss_search( 221 | headline_embedding, 222 | (select headline_embedding from articles where rowid = 123) 223 | ) 224 | limit 100; 225 | ``` 226 | - repo: sqlite-hello 227 | lang: C 228 | platforms: [linux-x86_64, macos-x86_64, macos-aarch64, windows-x86_64] 229 | short: \"Hello world\" extension 230 | long: | 231 | The smallest "Hello world" SQLite extension possible. 232 | 233 | ```sql 234 | sqlite> .load ./hello0 235 | sqlite> select hello('Alex'); 236 | 'Hello, Alex!' 237 | ``` 238 | footer: | 239 | ### Distribution 240 | 241 | SQLite extensions have historically been hard to shared with other people. Installing and using them can be tricky and confusing, but these are some strategies I've used to make that easier! 242 | 243 | ### `pip` for Python Developers 244 | 245 | Most of my SQLite extensions are additional placed in a small Python library and distributed on [PyPi](https://pypi.org/). Python developers can then install these extensions with a `pip install` command like so: 246 | 247 | ```bash 248 | pip install sqlite-regex 249 | ``` 250 | 251 | The actual Python library is small and meant to be used with the builtin `sqlite3` Python module, like so: 252 | 253 | ```python 254 | import sqlite3 255 | import sqlite_regex 256 | 257 | db = sqlite3.connect(':memory:') 258 | 259 | db.enable_load_extension(True) 260 | sqlite_regex.load(db) 261 | 262 | db.execute("select regex_version(), '[abc]' regexp 'a';").fetchone() 263 | # ('v0.1.0', 1) 264 | ``` 265 | 266 | See [_Making SQLite extensions pip install-able 267 | _](https://observablehq.com/@asg017/making-sqlite-extensions-pip-install-able) (February 2023) for more details. 268 | 269 | ### `npm` for Node.js Developers 270 | 271 | These extensions are also distributed on [`npm`](https://npmjs.com/) for use in Node.js. they can be install with a `npm install` command like so: 272 | 273 | ```bash 274 | npm install sqlite-regex 275 | ``` 276 | 277 | And then used in Node.js programs with [`better-sqlite3`](https://github.com/WiseLibs/better-sqlite3) or [`node-sqlite3`](https://github.com/TryGhost/node-sqlite3) like so: 278 | 279 | ```js 280 | import Database from "better-sqlite3"; 281 | import * as sqlite_regex from "sqlite-regex"; 282 | 283 | const db = new Database(":memory:"); 284 | 285 | db.loadExtension(sqlite_regex.getLoadablePath()); 286 | 287 | const version = db.prepare("select regex_version()").pluck().get(); 288 | console.log(version); // "v0.2.0" 289 | ``` 290 | 291 | See [_Making SQLite extensions npm install'able for Node.js, and on deno.land/x for Deno_](https://observablehq.com/@asg017/making-sqlite-extensions-npm-installable-and-deno) (March 2023) for more details. 292 | 293 | ### `deno.land/x` for Deno Developers 294 | 295 | These extensions are also distributed on [`deno.land/x`](https://deno.land/x) for use in [Deno](https://deno.land/) programs. It's meant to work with the [`x/sqlite3`](https://deno.land/x/sqlite3) client like so: 296 | 297 | ```ts 298 | import { Database } from "https://deno.land/x/sqlite3@0.8.0/mod.ts"; 299 | import * as sqlite_regex from "https://deno.land/x/sqlite_regex@v0.2.3-alpha.2/mod.ts"; 300 | 301 | const db = new Database(":memory:"); 302 | 303 | db.enableLoadExtension = true; 304 | sqlite_regex.load(db); 305 | 306 | const [version] = db.prepare("select regex_version()").value<[string]>()!; 307 | 308 | console.log(version); 309 | ``` 310 | 311 | See [the "Deno" section of _Making SQLite extensions npm install'able for Node.js, and on deno.land/x for Deno_](https://observablehq.com/@asg017/making-sqlite-extensions-npm-installable-and-deno#cell-13) (March 2023) for more details. 312 | 313 | ### `gem` for ruby Developers 314 | 315 | These extensions are also distributed on [rubygems.org](https://rubygems.org/) for use in Ruby applications. It's meant to work with the [`sqlite3`](https://rubygems.org/gems/sqlite3) client like so: 316 | 317 | ```ruby 318 | require 'sqlite3' 319 | require 'sqlite_regex' 320 | 321 | db = SQLite3::Database.new(':memory:') 322 | db.enable_load_extension(true) 323 | SqliteRegex.load(db) 324 | db.enable_load_extension(false) 325 | 326 | result = db.execute("select regex_version(), '[abc]' regexp 'a';") 327 | puts result.first.first # "v0.2.3-alpha.7" 328 | puts result.first.last # 1 329 | ``` 330 | 331 | See [_Making SQLite extension gem install'able for Ruby Developers_](https://observablehq.com/@asg017/making-sqlite-extension-gem-installable) (June 2023) for more details. 332 | 333 | ### As Datasette Plugins 334 | 335 | Most of these SQLite extensions are also distributed as [Datasette Plugins](https://datasette.io/plugins). They are small wrappers around their corresponding [pip packages](#pip-for-python-developers). 336 | 337 | They can be installed like so: 338 | 339 | ```bash 340 | datasette install datasette-sqlite-regex 341 | ``` 342 | 343 | Now all future Datasette instances will include `sqlite-regex` functions. Here we test it using [`datasette --get`](https://docs.datasette.io/en/stable/cli-reference.html#datasette-get). 344 | 345 | ``` 346 | $ datasette --get '/_memory.csv?sql=select+regex_version()' 347 | regex_version() 348 | v0.1.0 349 | ``` 350 | ### As sqlite-utils Plugins 351 | 352 | Most of these SQLite extensions are also distributed as [sqlite-utils plugins](https://sqlite-utils.datasette.io/en/stable/plugins.html). They are small wrappers around their corresponding [pip packages](#pip-for-python-developers). 353 | 354 | They can be installed like so: 355 | 356 | ```bash 357 | sqlite-utils install sqlite-utils-sqlite-regex 358 | ``` 359 | 360 | Now when using the sqlite-utils CLI, the `sqlite-regex` functions will be available. 361 | 362 | ``` 363 | $ sqlite-utils memory 'select regex_version()' 364 | [{"regex_version()": "v0.2.3"}] 365 | ``` 366 | 367 | ## Roadmap 368 | 369 | Various extensions and tools that I plan to build and open source in the future! Most of the development of these happen in private repos, to make my life easier. 370 | 371 | If you're interested in any of these, just send me a message! 372 | 373 | ### Future SQLite Extensions 374 | 375 | - `sqlite-img`: Query and manipulate images (cropping, thumbnails, rotation, etc.) 376 | - `sqlite-xml`: Query XML documents with XPath strings 377 | - `sqlite-assert`: Make assertions of your data at query-time 378 | - `sqlite-parquet`: Query Parquet files from SQLite 379 | - `sqlite-md`: Query Markdown documents as their AST 380 | - `sqlite-geo`: A GIS extension, a slimmed-down alternative to spatialite 381 | - `sqlite-python`: Write loadable SQLite extensions in pure Python 382 | - `sqlite-qjs`: Create custom scalar, aggregate, and table functions in JavaScript with [QuickJS](https://bellard.org/quickjs/) 383 | - `sqlite-duckdb`: Query and insert data in DuckDB tables, query CSVs/Parquet/JSON with DuckDB 384 | - `sqlite-pg`: Query and insert data into Postgres tables 385 | - `sqlite-notion`: Query and insert data into Notion databases 386 | - `sqlite-google-sheets`: query and insert data into Google Sheet 387 | 388 | ### Future Extension Bindings 389 | 390 | - Elixir 391 | - Rust 392 | - Go 393 | 394 | ### Future Tooling 395 | 396 | - `sqlite-package-manager`: A command-line tool for manage SQLite extensions for your project, similar to npm/pip/brew 397 | - `sqlite-docs`: Document SQLite tables and columns with in-line comments on `CREATE TABLE` statements, like JSDoc/rustdoc/docstrings 398 | - `sqlitex`: A modern `sqlite3` alternative, a new CLI with Parquet/S3 support, a bigger stdlib, syntax highlighting, and more 399 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # sqlite-ecosystem 4 | 5 | All the SQLite extensions, tools, and guides that I (Alex Garcia 👋🏼) have written and open sourced. 6 | 7 | Feel free to start a discussion in this repo about meta-level SQLite extension and tooling talk! Though if you're having a specific issue with a particular extension or tool, then file an issue in that project's repository. 8 | 9 | ## Overview 10 | 11 | These are all the loadable SQLite extensions I have built, along with which programming language they are written in and where they are distributed. 12 | 13 | | Extension | Description | Language | Python | Node.js | Deno | Ruby | Datasette | sqlite-utils | 14 | | ----------------------------------------- | -------------------------------------- | -------- | :--: | :--: | :---: | :--: | :-------: | :-----: | 15 | | [`sqlite-http`](#sqlite-http) | Make HTTP requests | Go | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | 16 | | [`sqlite-html`](#sqlite-html) | parse HTML documents | Go | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | 17 | | [`sqlite-lines`](#sqlite-lines) | Read files/blob line-by-line | C | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | 18 | | [`sqlite-path`](#sqlite-path) | Parse and generate filepaths | C | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | 19 | | [`sqlite-url`](#sqlite-url) | Parse and generate URLs | C | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | 20 | | [`sqlite-xsv`](#sqlite-xsv) | Query CSVs | Rust | ✅ | ✅ | ✅ | ✅ | 🚧 | 🚧 | 21 | | [`sqlite-regex`](#sqlite-regex) | Regular expression functions | Rust | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | 22 | | [`sqlite-ulid`](#sqlite-ulid) | Work with ULIDs | Rust | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | 23 | | [`sqlite-jsonschema`](#sqlite-jsonschema) | Validate JSON objects with JSON Schema | Rust | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | 24 | | [`sqlite-fastrand`](#sqlite-fastrand) | Generate fast numbers/blobs quickly | Rust | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | 25 | | [`sqlite-vss`](#sqlite-vss) | Vector search in SQLite | C++ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | 26 | 27 | 28 | ## Extensions 29 | 30 | 31 | 32 | 33 | 34 | 35 | ### [`sqlite-http`](https://github.com/asg017/sqlite-http) 36 | 37 | 38 | A SQLite extension for making HTTP requests purely in SQL. 39 | 40 | - Create GET, POST, and other HTTP requests, like curl, wget, and fetch 41 | - Download response bodies, header, status codes, timing info 42 | - Set rate limits, timeouts 43 | 44 | ```sql 45 | select http_get_body('https://text.npr.org/'); 46 | select 47 | http_get_body('https://api.github.com/repos/sqlite/sqlite') ->> '$.description'; 48 | ``` 49 | 50 | 51 | 52 | 53 | | Language/Platform | Install | | 54 | | ----------------- | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | 55 | | Python | `pip install sqlite-http` | [](https://pypi.org/project/sqlite-http/) | 56 | | Node.js | `npm install sqlite-http` | [](https://www.npmjs.com/package/sqlite-http) | 57 | | Deno | [`deno.land/x/sqlite_http`](https://deno.land/x/sqlite_http) | [](https://deno.land/x/sqlite_http) | 58 | | [Datasette](https://datasette.io/) | `datasette install datasette-sqlite-http` | [](https://pypi.org/project/datasette-sqlite-http) | 59 | | [sqlite-utils](https://sqlite-utils.datasette.io/) | `sqlite-utils install sqlite-utils-sqlite-http` | [](https://pypi.org/project/sqlite-utils-sqlite-http) | 60 | | Ruby | `gem install sqlite-http` |  | 61 | | Github Release | |  62 | 63 | 64 | 65 | - `linux-x86_64` (Linux x86_64) 66 | - `macos-x86_64` (MacOS x86_64) 67 | - `macos-aarch64` (MacOS M1 and M2 chips) 68 | - `windows-x86_64` (Windows x86_64) 69 | 70 | 71 | 72 | 73 | 74 | 75 | ### [`sqlite-html`](https://github.com/asg017/sqlite-html) 76 | 77 | 78 | A SQLite extension for querying, manipulating, and creating HTML elements. 79 | 80 | - Extract HTML or text from HTML with CSS selectors, like `.querySelector()`, `.innerHTML`, and `.innerText` 81 | - Generate a table of matching elements from a CSS selector, like `.querySelectorAll()` 82 | - Safely create HTML elements in a query, like .createElement() and `.appendChild()` 83 | 84 | ```sql 85 | select html_extract('Anakin Skywalker
', 'b'); -- "Skywalker" 86 | ``` 87 | 88 | 89 | 90 | 91 | | Language/Platform | Install | | 92 | | ----------------- | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | 93 | | Python | `pip install sqlite-html` | [](https://pypi.org/project/sqlite-html/) | 94 | | Node.js | `npm install sqlite-html` | [](https://www.npmjs.com/package/sqlite-html) | 95 | | Deno | [`deno.land/x/sqlite_html`](https://deno.land/x/sqlite_html) | [](https://deno.land/x/sqlite_html) | 96 | | [Datasette](https://datasette.io/) | `datasette install datasette-sqlite-html` | [](https://pypi.org/project/datasette-sqlite-html) | 97 | | [sqlite-utils](https://sqlite-utils.datasette.io/) | `sqlite-utils install sqlite-utils-sqlite-html` | [](https://pypi.org/project/sqlite-utils-sqlite-html) | 98 | | Ruby | `gem install sqlite-html` |  | 99 | | Github Release | |  100 | 101 | 102 | 103 | - `linux-x86_64` (Linux x86_64) 104 | - `macos-x86_64` (MacOS x86_64) 105 | - `macos-aarch64` (MacOS M1 and M2 chips) 106 | - `windows-x86_64` (Windows x86_64) 107 | 108 | 109 | 110 | 111 | 112 | 113 | ### [`sqlite-lines`](https://github.com/asg017/sqlite-lines) 114 | 115 | 116 | A SQLite extension for efficiently reading large files or blobs line-by-line. 117 | 118 | ```sql 119 | select line from lines_read('logs.txt'); 120 | ``` 121 | 122 | 123 | 124 | 125 | | Language/Platform | Install | | 126 | | ----------------- | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | 127 | | Python | `pip install sqlite-lines` | [](https://pypi.org/project/sqlite-lines/) | 128 | | Node.js | `npm install sqlite-lines` | [](https://www.npmjs.com/package/sqlite-lines) | 129 | | Deno | [`deno.land/x/sqlite_lines`](https://deno.land/x/sqlite_lines) | [](https://deno.land/x/sqlite_lines) | 130 | | [Datasette](https://datasette.io/) | `datasette install datasette-sqlite-lines` | [](https://pypi.org/project/datasette-sqlite-lines) | 131 | | [sqlite-utils](https://sqlite-utils.datasette.io/) | `sqlite-utils install sqlite-utils-sqlite-lines` | [](https://pypi.org/project/sqlite-utils-sqlite-lines) | 132 | | Ruby | `gem install sqlite-lines` |  | 133 | | Github Release | |  134 | 135 | 136 | 137 | - `linux-x86_64` (Linux x86_64) 138 | - `macos-x86_64` (MacOS x86_64) 139 | - `macos-aarch64` (MacOS M1 and M2 chips) 140 | 141 | 142 | 143 | 144 | 145 | 146 | ### [`sqlite-path`](https://github.com/asg017/sqlite-path) 147 | 148 | 149 | A SQLite extension for parsing, generating, and querying paths. 150 | 151 | ```sql 152 | select path_dirname('foo/bar.txt'); -- 'foo/' 153 | select path_basename('foo/bar.txt'); -- 'bar.txt' 154 | select path_extension('foo/bar.txt'); -- '.txt' 155 | ``` 156 | 157 | 158 | 159 | 160 | | Language/Platform | Install | | 161 | | ----------------- | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | 162 | | Python | `pip install sqlite-path` | [](https://pypi.org/project/sqlite-path/) | 163 | | Node.js | `npm install sqlite-path` | [](https://www.npmjs.com/package/sqlite-path) | 164 | | Deno | [`deno.land/x/sqlite_path`](https://deno.land/x/sqlite_path) | [](https://deno.land/x/sqlite_path) | 165 | | [Datasette](https://datasette.io/) | `datasette install datasette-sqlite-path` | [](https://pypi.org/project/datasette-sqlite-path) | 166 | | [sqlite-utils](https://sqlite-utils.datasette.io/) | `sqlite-utils install sqlite-utils-sqlite-path` | [](https://pypi.org/project/sqlite-utils-sqlite-path) | 167 | | Ruby | `gem install sqlite-path` |  | 168 | | Github Release | |  169 | 170 | 171 | 172 | - `linux-x86_64` (Linux x86_64) 173 | - `macos-x86_64` (MacOS x86_64) 174 | - `macos-aarch64` (MacOS M1 and M2 chips) 175 | - `windows-x86_64` (Windows x86_64) 176 | 177 | 178 | 179 | 180 | 181 | 182 | ### [`sqlite-url`](https://github.com/asg017/sqlite-url) 183 | 184 | 185 | A SQLite extension for parsing, generating, and querying URLs and query strings 186 | 187 | - Extract specific parts of a URL, like schemes, hostnames, and paths 188 | - Generate URLs with parts safely 189 | - Iterate through all key and values in a query string 190 | 191 | ```sql 192 | select url_valid('https://sqlite.org'); -- 1 193 | select url_scheme('https://www.sqlite.org/vtab.html#usage'); -- 'https' 194 | select url_host('https://www.sqlite.org/vtab.html#usage'); -- 'www.sqlite.org' 195 | ``` 196 | 197 | 198 | 199 | 200 | | Language/Platform | Install | | 201 | | ----------------- | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | 202 | | Python | `pip install sqlite-url` | [](https://pypi.org/project/sqlite-url/) | 203 | | Node.js | `npm install sqlite-url` | [](https://www.npmjs.com/package/sqlite-url) | 204 | | Deno | [`deno.land/x/sqlite_url`](https://deno.land/x/sqlite_url) | [](https://deno.land/x/sqlite_url) | 205 | | [Datasette](https://datasette.io/) | `datasette install datasette-sqlite-url` | [](https://pypi.org/project/datasette-sqlite-url) | 206 | | [sqlite-utils](https://sqlite-utils.datasette.io/) | `sqlite-utils install sqlite-utils-sqlite-url` | [](https://pypi.org/project/sqlite-utils-sqlite-url) | 207 | | Ruby | `gem install sqlite-url` |  | 208 | | Github Release | |  209 | 210 | 211 | 212 | - `linux-x86_64` (Linux x86_64) 213 | - `macos-x86_64` (MacOS x86_64) 214 | 215 | 216 | 217 | 218 | 219 | 220 | ### [`sqlite-xsv`](https://github.com/asg017/sqlite-xsv) 221 | 222 | 223 | A fast and performant SQLite extension for CSV files, written in Rust! 224 | 225 | - Query CSVs, TSVs, and other-SVs as SQLite virtual tables 226 | - The "reader" interface lets you query CSVs from other data sources, such as [`sqlite-http`](#sqlite-http) 227 | - Builtin support for querying CSVs with gzip or zstd compression 228 | 229 | ```sql 230 | create virtual table temp.students using csv( 231 | filename="students.csv" 232 | ); 233 | 234 | select * from temp.students; 235 | 236 | create virtual table temp.students_gz using csv( 237 | filename="students.csv.gz" 238 | ); 239 | 240 | create virtual table temp.students_reader using csv_reader( 241 | id integer, 242 | name text, 243 | age integer, 244 | progess real 245 | ); 246 | select * from temp.students_reader('./target/students.csv'); 247 | ``` 248 | 249 | 250 | 251 | 252 | | Language/Platform | Install | | 253 | | ----------------- | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | 254 | | Python | `pip install sqlite-xsv` | [](https://pypi.org/project/sqlite-xsv/) | 255 | | Node.js | `npm install sqlite-xsv` | [](https://www.npmjs.com/package/sqlite-xsv) | 256 | | Deno | [`deno.land/x/sqlite_xsv`](https://deno.land/x/sqlite_xsv) | [](https://deno.land/x/sqlite_xsv) | 257 | | Ruby | `gem install sqlite-xsv` |  | 258 | | Github Release | |  259 | 260 | 261 | 262 | - `linux-x86_64` (Linux x86_64) 263 | - `macos-x86_64` (MacOS x86_64) 264 | - `macos-aarch64` (MacOS M1 and M2 chips) 265 | - `windows-x86_64` (Windows x86_64) 266 | 267 | 268 | 269 | 270 | 271 | 272 | ### [`sqlite-regex`](https://github.com/asg017/sqlite-regex) 273 | 274 | 275 | A fast SQLite extension for regular expressions. 276 | 277 | ```sql 278 | select regex_find( 279 | '[0-9]{3}-[0-9]{3}-[0-9]{4}', 280 | 'phone: 111-222-3333' 281 | ); 282 | 283 | select rowid, * 284 | from regex_find_all( 285 | '\b\w{13}\b', 286 | 'Retroactively relinquishing remunerations is reprehensible.' 287 | ); 288 | /* 289 | ┌───────┬───────┬─────┬───────────────┐ 290 | │ rowid │ start │ end │ match │ 291 | ├───────┼───────┼─────┼───────────────┤ 292 | │ 0 │ 0 │ 13 │ Retroactively │ 293 | │ 1 │ 14 │ 27 │ relinquishing │ 294 | │ 2 │ 28 │ 41 │ remunerations │ 295 | │ 3 │ 45 │ 58 │ reprehensible │ 296 | └───────┴───────┴─────┴───────────────┘ 297 | */ 298 | ``` 299 | 300 | 301 | 302 | 303 | | Language/Platform | Install | | 304 | | ----------------- | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | 305 | | Python | `pip install sqlite-regex` | [](https://pypi.org/project/sqlite-regex/) | 306 | | Node.js | `npm install sqlite-regex` | [](https://www.npmjs.com/package/sqlite-regex) | 307 | | Deno | [`deno.land/x/sqlite_regex`](https://deno.land/x/sqlite_regex) | [](https://deno.land/x/sqlite_regex) | 308 | | [Datasette](https://datasette.io/) | `datasette install datasette-sqlite-regex` | [](https://pypi.org/project/datasette-sqlite-regex) | 309 | | [sqlite-utils](https://sqlite-utils.datasette.io/) | `sqlite-utils install sqlite-utils-sqlite-regex` | [](https://pypi.org/project/sqlite-utils-sqlite-regex) | 310 | | Ruby | `gem install sqlite-regex` |  | 311 | | Github Release | |  312 | 313 | 314 | 315 | - `linux-x86_64` (Linux x86_64) 316 | - `macos-x86_64` (MacOS x86_64) 317 | - `macos-aarch64` (MacOS M1 and M2 chips) 318 | - `windows-x86_64` (Windows x86_64) 319 | 320 | 321 | 322 | 323 | 324 | 325 | ### [`sqlite-ulid`](https://github.com/asg017/sqlite-ulid) 326 | 327 | 328 | A SQLite extension for generating and working with [ULIDs](https://github.com/ulid/spec). 329 | 330 | ```sql 331 | select ulid(); -- '01gqr4j69cc7w1xdbarkcbpq17' 332 | select ulid_bytes(); -- X'0185310899dd7662b8f1e5adf9a5e7c0' 333 | select ulid_with_prefix('invoice'); -- 'invoice_01gqr4jmhxhc92x1kqkpxb8j16' 334 | select ulid_datetime('01gqr4j69cc7w1xdbarkcbpq17') -- '2023-01-26 22:53:20.556' 335 | ``` 336 | 337 | 338 | 339 | 340 | | Language/Platform | Install | | 341 | | ----------------- | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | 342 | | Python | `pip install sqlite-ulid` | [](https://pypi.org/project/sqlite-ulid/) | 343 | | Node.js | `npm install sqlite-ulid` | [](https://www.npmjs.com/package/sqlite-ulid) | 344 | | Deno | [`deno.land/x/sqlite_ulid`](https://deno.land/x/sqlite_ulid) | [](https://deno.land/x/sqlite_ulid) | 345 | | [Datasette](https://datasette.io/) | `datasette install datasette-sqlite-ulid` | [](https://pypi.org/project/datasette-sqlite-ulid) | 346 | | [sqlite-utils](https://sqlite-utils.datasette.io/) | `sqlite-utils install sqlite-utils-sqlite-ulid` | [](https://pypi.org/project/sqlite-utils-sqlite-ulid) | 347 | | Ruby | `gem install sqlite-ulid` |  | 348 | | Github Release | |  349 | 350 | 351 | 352 | - `linux-x86_64` (Linux x86_64) 353 | - `macos-x86_64` (MacOS x86_64) 354 | - `macos-aarch64` (MacOS M1 and M2 chips) 355 | - `windows-x86_64` (Windows x86_64) 356 | 357 | 358 | 359 | 360 | 361 | 362 | ### [`sqlite-jsonschema`](https://github.com/asg017/sqlite-jsonschema) 363 | 364 | A SQLite extension for validating JSON objects with [JSON Schema](https://json-schema.org/). 365 | 366 | ```sql 367 | select jsonschema_matches('{"maxLength": 5}', json_quote('alex')); -- 1 368 | ``` 369 | 370 | 371 | 372 | 373 | | Language/Platform | Install | | 374 | | ----------------- | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | 375 | | Python | `pip install sqlite-jsonschema` | [](https://pypi.org/project/sqlite-jsonschema/) | 376 | | Node.js | `npm install sqlite-jsonschema` | [](https://www.npmjs.com/package/sqlite-jsonschema) | 377 | | Deno | [`deno.land/x/sqlite_jsonschema`](https://deno.land/x/sqlite_jsonschema) | [](https://deno.land/x/sqlite_jsonschema) | 378 | | [Datasette](https://datasette.io/) | `datasette install datasette-sqlite-jsonschema` | [](https://pypi.org/project/datasette-sqlite-jsonschema) | 379 | | [sqlite-utils](https://sqlite-utils.datasette.io/) | `sqlite-utils install sqlite-utils-sqlite-jsonschema` | [](https://pypi.org/project/sqlite-utils-sqlite-jsonschema) | 380 | | Ruby | `gem install sqlite-jsonschema` |  | 381 | | Github Release | |  382 | 383 | 384 | 385 | - `linux-x86_64` (Linux x86_64) 386 | - `macos-x86_64` (MacOS x86_64) 387 | - `macos-aarch64` (MacOS M1 and M2 chips) 388 | - `windows-x86_64` (Windows x86_64) 389 | 390 | 391 | 392 | 393 | 394 | 395 | ### [`sqlite-fastrand`](https://github.com/asg017/sqlite-fastrand) 396 | 397 | 398 | A SQLite extension for quickly generating random numbers, booleans, characters, and blobs 399 | 400 | ```sql 401 | select fastrand_int(); -- 556823563 402 | select fastrand_alphabetic(); -- 's' 403 | select fastrand_uppercase();-- 'M' 404 | select fastrand_double(); -- 0.740834390248454 405 | ``` 406 | 407 | 408 | 409 | 410 | | Language/Platform | Install | | 411 | | ----------------- | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | 412 | | Python | `pip install sqlite-fastrand` | [](https://pypi.org/project/sqlite-fastrand/) | 413 | | Node.js | `npm install sqlite-fastrand` | [](https://www.npmjs.com/package/sqlite-fastrand) | 414 | | Deno | [`deno.land/x/sqlite_fastrand`](https://deno.land/x/sqlite_fastrand) | [](https://deno.land/x/sqlite_fastrand) | 415 | | [Datasette](https://datasette.io/) | `datasette install datasette-sqlite-fastrand` | [](https://pypi.org/project/datasette-sqlite-fastrand) | 416 | | [sqlite-utils](https://sqlite-utils.datasette.io/) | `sqlite-utils install sqlite-utils-sqlite-fastrand` | [](https://pypi.org/project/sqlite-utils-sqlite-fastrand) | 417 | | Ruby | `gem install sqlite-fastrand` |  | 418 | | Github Release | |  419 | 420 | 421 | 422 | - `linux-x86_64` (Linux x86_64) 423 | - `macos-x86_64` (MacOS x86_64) 424 | - `macos-aarch64` (MacOS M1 and M2 chips) 425 | - `windows-x86_64` (Windows x86_64) 426 | 427 | 428 | 429 | 430 | 431 | 432 | ### [`sqlite-vss`](https://github.com/asg017/sqlite-vss) 433 | 434 | A SQLite extension for efficient vector search, based on Faiss! 435 | 436 | ```sql 437 | create virtual table vss_articles using vss0( 438 | headline_embedding(384), 439 | description_embedding(384), 440 | ); 441 | 442 | insert into vss_articles(rowid, headline_embedding) 443 | select rowid, headline_embedding from articles; 444 | 445 | select rowid, distance 446 | from vss_articles 447 | where vss_search( 448 | headline_embedding, 449 | (select headline_embedding from articles where rowid = 123) 450 | ) 451 | limit 100; 452 | ``` 453 | 454 | 455 | 456 | 457 | | Language/Platform | Install | | 458 | | ----------------- | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | 459 | | Python | `pip install sqlite-vss` | [](https://pypi.org/project/sqlite-vss/) | 460 | | Node.js | `npm install sqlite-vss` | [](https://www.npmjs.com/package/sqlite-vss) | 461 | | Deno | [`deno.land/x/sqlite_vss`](https://deno.land/x/sqlite_vss) | [](https://deno.land/x/sqlite_vss) | 462 | | [Datasette](https://datasette.io/) | `datasette install datasette-sqlite-vss` | [](https://pypi.org/project/datasette-sqlite-vss) | 463 | | [sqlite-utils](https://sqlite-utils.datasette.io/) | `sqlite-utils install sqlite-utils-sqlite-vss` | [](https://pypi.org/project/sqlite-utils-sqlite-vss) | 464 | | Ruby | `gem install sqlite-vss` |  | 465 | | Github Release | |  466 | 467 | 468 | 469 | - `linux-x86_64` (Linux x86_64) 470 | - `macos-x86_64` (MacOS x86_64) 471 | - `macos-aarch64` (MacOS M1 and M2 chips) 472 | 473 | 474 | 475 | 476 | 477 | 478 | ### [`sqlite-hello`](https://github.com/asg017/sqlite-hello) 479 | 480 | The smallest "Hello world" SQLite extension possible. 481 | 482 | ```sql 483 | sqlite> .load ./hello0 484 | sqlite> select hello('Alex'); 485 | 'Hello, Alex!' 486 | ``` 487 | 488 | 489 | 490 | 491 | | Language/Platform | Install | | 492 | | ----------------- | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | 493 | | Python | `pip install sqlite-hello` | [](https://pypi.org/project/sqlite-hello/) | 494 | | Node.js | `npm install sqlite-hello` | [](https://www.npmjs.com/package/sqlite-hello) | 495 | | Deno | [`deno.land/x/sqlite_hello`](https://deno.land/x/sqlite_hello) | [](https://deno.land/x/sqlite_hello) | 496 | | [Datasette](https://datasette.io/) | `datasette install datasette-sqlite-hello` | [](https://pypi.org/project/datasette-sqlite-hello) | 497 | | [sqlite-utils](https://sqlite-utils.datasette.io/) | `sqlite-utils install sqlite-utils-sqlite-hello` | [](https://pypi.org/project/sqlite-utils-sqlite-hello) | 498 | | Ruby | `gem install sqlite-hello` |  | 499 | | Github Release | |  500 | 501 | 502 | 503 | - `linux-x86_64` (Linux x86_64) 504 | - `macos-x86_64` (MacOS x86_64) 505 | - `macos-aarch64` (MacOS M1 and M2 chips) 506 | - `windows-x86_64` (Windows x86_64) 507 | 508 | 509 | 510 | 511 | 512 | 513 | ### Distribution 514 | 515 | SQLite extensions have historically been hard to shared with other people. Installing and using them can be tricky and confusing, but these are some strategies I've used to make that easier! 516 | 517 | ### `pip` for Python Developers 518 | 519 | Most of my SQLite extensions are additional placed in a small Python library and distributed on [PyPi](https://pypi.org/). Python developers can then install these extensions with a `pip install` command like so: 520 | 521 | ```bash 522 | pip install sqlite-regex 523 | ``` 524 | 525 | The actual Python library is small and meant to be used with the builtin `sqlite3` Python module, like so: 526 | 527 | ```python 528 | import sqlite3 529 | import sqlite_regex 530 | 531 | db = sqlite3.connect(':memory:') 532 | 533 | db.enable_load_extension(True) 534 | sqlite_regex.load(db) 535 | 536 | db.execute("select regex_version(), '[abc]' regexp 'a';").fetchone() 537 | # ('v0.1.0', 1) 538 | ``` 539 | 540 | See [_Making SQLite extensions pip install-able 541 | _](https://observablehq.com/@asg017/making-sqlite-extensions-pip-install-able) (February 2023) for more details. 542 | 543 | ### `npm` for Node.js Developers 544 | 545 | These extensions are also distributed on [`npm`](https://npmjs.com/) for use in Node.js. they can be install with a `npm install` command like so: 546 | 547 | ```bash 548 | npm install sqlite-regex 549 | ``` 550 | 551 | And then used in Node.js programs with [`better-sqlite3`](https://github.com/WiseLibs/better-sqlite3) or [`node-sqlite3`](https://github.com/TryGhost/node-sqlite3) like so: 552 | 553 | ```js 554 | import Database from "better-sqlite3"; 555 | import * as sqlite_regex from "sqlite-regex"; 556 | 557 | const db = new Database(":memory:"); 558 | 559 | db.loadExtension(sqlite_regex.getLoadablePath()); 560 | 561 | const version = db.prepare("select regex_version()").pluck().get(); 562 | console.log(version); // "v0.2.0" 563 | ``` 564 | 565 | See [_Making SQLite extensions npm install'able for Node.js, and on deno.land/x for Deno_](https://observablehq.com/@asg017/making-sqlite-extensions-npm-installable-and-deno) (March 2023) for more details. 566 | 567 | ### `deno.land/x` for Deno Developers 568 | 569 | These extensions are also distributed on [`deno.land/x`](https://deno.land/x) for use in [Deno](https://deno.land/) programs. It's meant to work with the [`x/sqlite3`](https://deno.land/x/sqlite3) client like so: 570 | 571 | ```ts 572 | import { Database } from "https://deno.land/x/sqlite3@0.8.0/mod.ts"; 573 | import * as sqlite_regex from "https://deno.land/x/sqlite_regex@v0.2.3-alpha.2/mod.ts"; 574 | 575 | const db = new Database(":memory:"); 576 | 577 | db.enableLoadExtension = true; 578 | sqlite_regex.load(db); 579 | 580 | const [version] = db.prepare("select regex_version()").value<[string]>()!; 581 | 582 | console.log(version); 583 | ``` 584 | 585 | See [the "Deno" section of _Making SQLite extensions npm install'able for Node.js, and on deno.land/x for Deno_](https://observablehq.com/@asg017/making-sqlite-extensions-npm-installable-and-deno#cell-13) (March 2023) for more details. 586 | 587 | ### `gem` for ruby Developers 588 | 589 | These extensions are also distributed on [rubygems.org](https://rubygems.org/) for use in Ruby applications. It's meant to work with the [`sqlite3`](https://rubygems.org/gems/sqlite3) client like so: 590 | 591 | ```ruby 592 | require 'sqlite3' 593 | require 'sqlite_regex' 594 | 595 | db = SQLite3::Database.new(':memory:') 596 | db.enable_load_extension(true) 597 | SqliteRegex.load(db) 598 | db.enable_load_extension(false) 599 | 600 | result = db.execute("select regex_version(), '[abc]' regexp 'a';") 601 | puts result.first.first # "v0.2.3-alpha.7" 602 | puts result.first.last # 1 603 | ``` 604 | 605 | See [_Making SQLite extension gem install'able for Ruby Developers_](https://observablehq.com/@asg017/making-sqlite-extension-gem-installable) (June 2023) for more details. 606 | 607 | ### As Datasette Plugins 608 | 609 | Most of these SQLite extensions are also distributed as [Datasette Plugins](https://datasette.io/plugins). They are small wrappers around their corresponding [pip packages](#pip-for-python-developers). 610 | 611 | They can be installed like so: 612 | 613 | ```bash 614 | datasette install datasette-sqlite-regex 615 | ``` 616 | 617 | Now all future Datasette instances will include `sqlite-regex` functions. Here we test it using [`datasette --get`](https://docs.datasette.io/en/stable/cli-reference.html#datasette-get). 618 | 619 | ``` 620 | $ datasette --get '/_memory.csv?sql=select+regex_version()' 621 | regex_version() 622 | v0.1.0 623 | ``` 624 | ### As sqlite-utils Plugins 625 | 626 | Most of these SQLite extensions are also distributed as [sqlite-utils plugins](https://sqlite-utils.datasette.io/en/stable/plugins.html). They are small wrappers around their corresponding [pip packages](#pip-for-python-developers). 627 | 628 | They can be installed like so: 629 | 630 | ```bash 631 | sqlite-utils install sqlite-utils-sqlite-regex 632 | ``` 633 | 634 | Now when using the sqlite-utils CLI, the `sqlite-regex` functions will be available. 635 | 636 | ``` 637 | $ sqlite-utils memory 'select regex_version()' 638 | [{"regex_version()": "v0.2.3"}] 639 | ``` 640 | 641 | ## Roadmap 642 | 643 | Various extensions and tools that I plan to build and open source in the future! Most of the development of these happen in private repos, to make my life easier. 644 | 645 | If you're interested in any of these, just send me a message! 646 | 647 | ### Future SQLite Extensions 648 | 649 | - `sqlite-img`: Query and manipulate images (cropping, thumbnails, rotation, etc.) 650 | - `sqlite-xml`: Query XML documents with XPath strings 651 | - `sqlite-assert`: Make assertions of your data at query-time 652 | - `sqlite-parquet`: Query Parquet files from SQLite 653 | - `sqlite-md`: Query Markdown documents as their AST 654 | - `sqlite-geo`: A GIS extension, a slimmed-down alternative to spatialite 655 | - `sqlite-python`: Write loadable SQLite extensions in pure Python 656 | - `sqlite-qjs`: Create custom scalar, aggregate, and table functions in JavaScript with [QuickJS](https://bellard.org/quickjs/) 657 | - `sqlite-duckdb`: Query and insert data in DuckDB tables, query CSVs/Parquet/JSON with DuckDB 658 | - `sqlite-pg`: Query and insert data into Postgres tables 659 | - `sqlite-notion`: Query and insert data into Notion databases 660 | - `sqlite-google-sheets`: query and insert data into Google Sheet 661 | 662 | ### Future Extension Bindings 663 | 664 | - Elixir 665 | - Rust 666 | - Go 667 | 668 | ### Future Tooling 669 | 670 | - `sqlite-package-manager`: A command-line tool for manage SQLite extensions for your project, similar to npm/pip/brew 671 | - `sqlite-docs`: Document SQLite tables and columns with in-line comments on `CREATE TABLE` statements, like JSDoc/rustdoc/docstrings 672 | - `sqlitex`: A modern `sqlite3` alternative, a new CLI with Parquet/S3 support, a bigger stdlib, syntax highlighting, and more 673 | 674 | --------------------------------------------------------------------------------