libsearch is a fast, small, minimal full-text search library used extensively across my personal projects. In my experience, it's just the easiest way to add search capabilities to your app that goes beyond substring matches without bringing in too much complexity. Read more at the GitHub repo.
17 | 18 | 19 |Annotated source files
22 | 23 | 24 |2
TODO: Explain stuff...
24 |4
6
To turn every potential query into a regular expression, we need to be able to escape key characters.
30 |8function escapeForRegExp(text) {9 return text.replace(/[.*+?^${}[\]()|\\]/g, '\\$1');10}
11
The main search function takes: - query, the search query text - items, the list of items to search - by, which is a predicate (string, number, or function) that takes an item from the items list and returns the string that should be matched with the query Options include - caseSensitive, which is self-explanatory - mode: which is 'word' or 'prefix' (default)
20export function search(query, items, by = x => x, options) {21 options = {22 caseSensitive: false,
23 mode: 'prefix',
24 ...options,
25 }
26
27 function countMatches(s, regexp) {28 let i = 0;
29 const exec = regexp.exec.bind(regexp);
30 while (exec(s) !== null) {31 i ++;
32 }
33 return i;
34 }
35
36 const words = query
37 .trim()
38 .split(' ')39 .filter(s => s !== '');
40
41 if (words.length === 0) {42 return items;
43 }
44
45 const suggestions = words.reduce((suggestions, word, i) => {46 const isLastWord = i + 1 === words.length;
47 const regexp = new RegExp(
48 '(^|\\W)' + escapeForRegExp(word) + (isLastWord ? '' : '($|\\W)'),
49 'img'
50 );
51 return suggestions.filter(sugg => countMatches(by(sugg), regexp) > 0);
52 }, items);
53
54 // TODO: sort by TF-IDF
55 return sortBy(suggestions, by);
56}
57
58
libsearch is the core text search algorithm that I've polished and 21 | reused over the years across many of my personal 22 | projects for fast and simple full-text 23 | search, packaged into a small single-purpose JavaScript library.
24 |For how libsearch works, how to import and use in your own project, and 25 | canonical documentation, check out the GitHub repository 26 | page.
27 |9
To turn every potential query into a regular expression, we need to be able 30 | to escape characters that are significant in RegExp.
31 |12function escapeForRegExp(text: string): string {13 return text.replace(/[.*+?^${}[\]()|\\]/g, '\\$1');14}
15
Utility function for sorting an array by some predicate, rather than a
36 | comparator function. This implementation assumes by(it) is very cheap.
18function sortBy<T>(items: T[], by: (_it: T) => any): T[] {19 return items.sort((a, b) => {20 const aby = by(a);
21 const bby = by(b);
22 if (aby < bby) {23 return 1;
24 }
25 if (bby < aby) {26 return -1;
27 }
28 return 0;
29 });
30}
31
The search function takes:
52 |items, the list of items to searchquery, the search query textby, which is a predicate function that takes an item from the items
56 | list and returns the string that should be matched with the queryoptions, a dictionary of options:Options include
60 |caseSensitive, which is self-explanatorymode: which is 'word', 'prefix', or 'autocomplete' ('autocomplete' by
63 | default), determining the way in which partial matches are processed43export function search<T>(items: T[], query: string, by: (_it: T) => string = x => String(x), {44 caseSensitive = false,
45 mode = 'autocomplete',
46}: {47 caseSensitive?: boolean;
48 mode?: 'word' | 'prefix' | 'autocomplete';
49} = {}) {countMatches counts the number of times regexp occurs in the string
73 | s. We need this information for ranking, where documents that mention
74 | the keyword more times (relative to the total word count of the
75 | document) are ranked higher.
54 function countMatches(s: string, regexp: RegExp): number {55 let i = 0;
56 while (regexp.exec(s) !== null) {57 i ++;
58 }
59 return i;
60 }
61
We chunk up the query string into a list of "words", each of which will 85 | become a regular expression filter.
86 |64 const words = query
65 .trim()
66 .split(/\s+/)
67 .filter(s => s !== '');
68
Short-circuit if the search query is empty -- return the original list. 92 | This is a sensible default because in most apps this corresponds to the 93 | "home view" of the list, where a search has not been performed.
94 |72 if (words.length === 0) {73 return items;
74 }
75
For every word in the search query, we're going to keep track of every 99 | document's TF-IDF value in this map, and aggregate them together by the 100 | end for sorting.
101 |79 const tfidf = new Map<T, number>();
80
Iterate through every word in the query and progressively filter down
104 | items to just the documents that match every query word.
83 const results = words.reduce((results, word, i) => {84 const isLastWord = i + 1 === words.length;
85 const regexp = new RegExp(
86 '(^|\\W)'
87 + escapeForRegExp(word)
88 + ((mode === 'autocomplete' && isLastWord) || mode === 'prefix' ? '' : '($|\\W)'),
The 'u' flag for Unicode used to be used here, but was removed 112 | because it was (1) across-the-board too slow, and removing it 113 | made a statistically significant speed improvement, and (2) 114 | caused at least Chrome to have strange performance cliffs in 115 | unpredictable ways where certain RegExp operations would take 116 | 10s of ms.
117 |95 caseSensitive ? 'mg' : 'img'
96 );
97 return results.filter(result => {98 const text = by(result);
99 const count = countMatches(text, regexp);
100 if (count === 0) {101 return false;
102 }
Compute the TF-IDF value for this word, and add it to this
126 | result's TF-IDF value so far.
105 tfidf.set(
106 result,
107 (tfidf.get(result) || 0)
108 + (count / text.length * Math.log(items.length / results.length))
109 );
110 return true;
111 })
112 }, items);
113
Sort the results list by our ranking metric, TF-IDF.
137 |115 return sortBy(results, result => tfidf.get(result));
116}
117
118
1import {strict as assert} from 'node:assert';2import {search} from '../dist/search.js';3
4const item = name => ({name});5
Most of the tests operate on this pre-set list of items to search
26 |7const ITEMS = [
8 item('Linus Lee'),9 item('@thesephist'),10 item('@geohot'),11 item('linuslee'),12 item('linus is a person'),13 item('@dlwlrma'),14];
15
16describe('basic search', () => {17 it('search empty array', () => {18 assert.deepEqual(search([], 'query', x => x.name), []);
19 });
20
21 it('search with empty query', () => {22 assert.deepEqual(search(ITEMS, '', x => x.name), ITEMS);
23 });
24
25 it('search with 1 letter returns correct result', () => {26 assert.deepEqual(search(ITEMS, 'l', x => x.name), [
27 item('Linus Lee'),28 item('linuslee'),29 item('linus is a person'),30 ]);
31 });
32
33 it('search does not match from middle of words', () => {34 assert.deepEqual(search(ITEMS, 'w', x => x.name), []);
35 });
36
37 it('multi-word search returns correct result', () => {38 assert.deepEqual(search(ITEMS, 'linus lee', x => x.name), [
39 item('Linus Lee'),40 ]);
41 });
42
43 it('searching words out of order returns correct result', () => {44 assert.deepEqual(search(ITEMS, 'lee linus', x => x.name), [
45 item('Linus Lee'),46 ]);
47 });
48
49 it('search works even if the last query word is incomplete', () => {50 assert.deepEqual(search(ITEMS, 'linus le', x => x.name), [
51 item('Linus Lee'),52 ]);
53 });
54
55 it('search query may contain newlines, tabs, and multiple consecutive spaces', () => {56 assert.deepEqual(search(ITEMS, ' linus\t is\nperson\t', x => x.name), [
57 item('linus is a person'),58 ]);
59 });
60
61 it('correctly implements TF-IDF ranking', () => {In this example, "mango" has much higher IDF (is a higher-signal 82 | word) in the corpus than "apple", which appears in nearly every 83 | document. Therefore, documents that mention "mango" more times 84 | (relative to the length of the document) should rank higher.
85 |66 assert.deepEqual(
67 search([
68 // matches
69 item('mango mango mango apple'),70 item('mango apple mango apple'),71 item('apple mango apple mango apple mango apple mango'),72 item('apple apple apple apple apple apple apple apple mango'),73 // rejects
74 item('apple apple apple'),75 item('mango mango mango'),76 item('applemango'),77 item('mangoapple'),78 item('apple 1'),79 item('apple 2'),80 item('apple 3'),81 item('apple 4'),82 item('apple 5'),83 item('apple 6'),84 item('apple 7'),85 item('apple 8'),86 item('apple 9'),87 ], 'apple mango', x => x.name),
88 [
89 item('mango mango mango apple'),90 item('mango apple mango apple'),91 item('apple mango apple mango apple mango apple mango'),92 item('apple apple apple apple apple apple apple apple mango'),93 ]
94 );
95 });
96});
97
98describe('custom search-by predicates', () => {99 it('default predicate is provided as x => x', () => {100 assert.deepEqual(
101 search([
102 'university',
103 'uni of california',
104 'university of california',
105 ], 'uni of cali'),
106 [
107 'uni of california',
108 ]
109 );
110 });
111
112 it('accepts and uses a custom predicate', () => {113 assert.deepEqual(search(ITEMS, 'sunil ee', x => x.name.split('').reverse().join('')), [114 item('Linus Lee'),115 ]);
116 });
117});
118
119describe('search modes', () => {120 it('in mode: word, search does not match if any words are incomplete', () => {121 assert.deepEqual(search(ITEMS, 'linu lee', x => x.name, {mode: 'word'}), []);122 });
123
124 it('in mode: prefix, every query word may be incomplete', () => {125 assert.deepEqual(search(ITEMS, 'linu le', x => x.name, {mode: 'prefix'}), [126 item('Linus Lee'),127 ]);
128 });
129
130 it('in mode: autocomplete, only the last query word may be incomplete', () => {131 assert.deepEqual(search(ITEMS, 'linus le', x => x.name, {mode: 'autocomplete'}), [132 item('Linus Lee'),133 ]);
134 assert.deepEqual(search(ITEMS, 'linu le', x => x.name, {mode: 'autocomplete'}), []);135 });
136});
137
138describe('case sensitivity', () => {139 it('caseSensitive: true omits non-matching results', () => {140 assert.deepEqual(search(ITEMS, 'l', x => x.name, {caseSensitive: true}), [141 item('linuslee'),142 item('linus is a person'),143 ]);
144 });
145
146 it('caseSensitive: false includes case-insensitive results', () => {147 assert.deepEqual(search(ITEMS, 'l', x => x.name, {caseSensitive: false}), [148 item('Linus Lee'),149 item('linuslee'),150 item('linus is a person'),151 ]);
152 });
153});
154
155