107 | )
108 | }
109 |
110 | export default ChatWindow
111 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | In this tutorial we'll build a fully local chat-with-pdf app using LlamaIndexTS, Ollama, Next.JS.
2 |
3 |
4 | https://github.com/rsrohan99/local-pdf-ai/assets/62835870/6f2497ea-15b4-47ea-9482-dade56434b2b
5 |
6 |
7 | Stack used:
8 | - **LlamaIndex TS** as the RAG framework
9 | - **Ollama** to locally run LLM and embed models
10 | - **nomic-text-embed** with Ollama as the embed model
11 | - **phi2** with Ollama as the LLM
12 | - **Next.JS** with **server actions**
13 | - **PDFObject** to preview PDF with auto-scroll to relevant page
14 | - **LangChain** WebPDFLoader to parse the PDF
15 |
16 | ### Install Ollama
17 |
18 | We'll use Ollama to run the embed models and llms locally.
19 |
20 | Install Ollama
21 |
22 | ```bash
23 | $ curl -fsSL https://ollama.com/install.sh | sh
24 | ```
25 |
26 | ### Download nomic and phi model weights
27 |
28 | For this guide, I've used `phi2` as the LLM and `nomic-embed-text` as the embed model.
29 |
30 | To use the model, first we need to download their weights.
31 |
32 | ```bash
33 | $ ollama pull phi
34 |
35 | $ ollama pull nomic-embed-text
36 | ```
37 |
38 | But feel free to use any model you want.
39 |
40 | ### `FilePicker.tsx` - Drag-n-drop the PDF
41 |
42 | This component is the entry-point to our app.
43 |
44 | It's used for uploading the pdf file, either clicking the upload button or drag-and-drop the PDF file.
45 |
46 | ```ts
47 | return (
48 |
70 | )
71 | ```
72 | After successfully upload, it sets the state variable `selectedFile` to the newly uploaded file.
73 |
74 |
75 | ### `Preview.tsx` - Preview of the PDF
76 |
77 | Once the state variable `selectedFile` is set, `ChatWindow` and `Preview` components are rendered instead of `FilePicker`
78 |
79 | First we get the base64 string of the pdf from the `File` using `FileReader`. Next we use this base64 string to preview the pdf.
80 |
81 | Preview component uses `PDFObject` package to render the PDF.
82 |
83 | It also takes `page` as prop to scroll to the relevant page. It's set to 1 initially and then updated as we chat with the PDF.
84 |
85 | ```ts
86 | useEffect(() => {
87 | const options = {
88 | title: fileToPreview.name,
89 | pdfOpenParams: {
90 | view: "fitH",
91 | page: page || 1,
92 | zoom: "scale,left,top",
93 | pageMode: 'none'
94 | }
95 | }
96 | console.log(`Page: ${page}`)
97 | const reader = new FileReader()
98 | reader.onload = () => {
99 | setb64String(reader.result as string);
100 | }
101 | reader.readAsDataURL(fileToPreview)
102 | pdfobject.embed(b64String as string, "#pdfobject", options)
103 | }, [page, b64String])
104 |
105 | return (
106 |
107 |
108 | )
109 | ```
110 |
111 |
112 | ### `ProcessPDF()` Next.JS server action
113 |
114 | We also have to process the PDF for RAG.
115 |
116 | We first use `LangChain` `WebPDFLoader` to parse the uploaded PDF. We use `WebPDFLoader` because it runs on the browser and don't require node.js.
117 |
118 | ```ts
119 | const loader = new WebPDFLoader(
120 | selectedFile,
121 | { parsedItemSeparator: " " }
122 | );
123 | const lcDocs = (await loader.load()).map(lcDoc => ({
124 | pageContent: lcDoc.pageContent,
125 | metadata: lcDoc.metadata,
126 | }))
127 | ```
128 |
129 | ### RAG using LlamaIndex TS
130 |
131 | Next, we pass the parsed documents to a Next.JS server action that initiates the RAG pipeline using `LlamaIndex TS`
132 | ```ts
133 | if (lcDocs.length == 0) return;
134 | const docs = lcDocs.map(lcDoc => new Document({
135 | text: lcDoc.pageContent,
136 | metadata: lcDoc.metadata
137 | }))
138 | ```
139 | we create LlamaIndex Documents from the parsed documents.
140 |
141 | #### Vector Store Index
142 |
143 | Next we create a `VectorStoreIndex` with those Documents, passing configuration info like which embed model and llm to use.
144 | ```ts
145 | const index = await VectorStoreIndex.fromDocuments(docs, {
146 | serviceContext: serviceContextFromDefaults({
147 | chunkSize: 300,
148 | chunkOverlap: 20,
149 | embedModel, llm
150 | })
151 | })
152 | ```
153 |
154 | We use Ollama for LLM and OllamaEmbedding for embed model
155 |
156 | ```ts
157 | const embedModel = new OllamaEmbedding({
158 | model: 'nomic-embed-text'
159 | })
160 |
161 | const llm = new Ollama({
162 | model: "phi",
163 | modelMetadata: {
164 | temperature: 0,
165 | maxTokens: 25,
166 | }
167 | })
168 | ```
169 |
170 | #### Vector Index Retriever
171 | We then create a `VectorIndexRetriever` from the `index`, which will be used to create a chat engine.
172 | ```ts
173 | const retriever = index.asRetriever({
174 | similarityTopK: 2,
175 | })
176 | if (chatEngine) {
177 | chatEngine.reset()
178 | }
179 | ```
180 |
181 | #### ChatEngine
182 | Finally, we create a LlamaIndex `ContextChatEngine` from the `Retriever`
183 | ```ts
184 | chatEngine = new ContextChatEngine({
185 | retriever,
186 | chatModel: llm
187 | })
188 | ```
189 | we pass in the LLM as well.
190 |
191 | ### `ChatWindow.tsx`
192 | This component is used to handle the Chat Logic
193 | ```ts
194 |
203 | ```
204 |
205 | ### `chat()` server action
206 | This server action used the previously created `ChatEngine` to generate chat response.
207 |
208 | In addition to the text response it also returns the source nodes used to generate the response, which we'll use later to updated which page to show on the PDF preview.
209 |
210 | ```ts
211 | const queryResult = await chatEngine.chat({
212 | message: query
213 | })
214 | const response = queryResult.response
215 | const metadata = queryResult.sourceNodes?.map(node => node.metadata)
216 | return { response, metadata };
217 | ```
218 |
219 | ### Update the page to preview from metadata
220 |
221 | We use the response and metadata from the above server action (`chat()`) to update the messages, and update the page to show in the PDF preview.
222 |
223 | ```ts
224 | setMessages(
225 | [
226 | ...messages,
227 | { role: 'human', statement: input },
228 | { role: 'ai', statement: response }
229 | ]
230 | )
231 | // console.log(metadata)
232 | if (metadata.length > 0) {
233 | setPage(metadata[0].loc.pageNumber)
234 | }
235 | setLoadingMessage("Got response from AI.")
236 | ```
237 |
238 | ### Few gotchas
239 |
240 | There're a few things to consider for this project:
241 | - You'll need a powerful machine with decent GPU to run Ollama for faster and better responses.
242 | - We need to disable `fs` on `browser` otherwise `pdf-parse` will not work. We need to put this in the `webpack` section of `next.config.js`
243 | ```ts
244 | if (!isServer) {
245 | config.resolve.fallback = {
246 | fs: false,
247 | "node:fs/promises": false,
248 | assert: false,
249 | module: false,
250 | perf_hooks: false,
251 | };
252 | }
253 | ```
254 | - Next.JS server actions don't support sending intermediate results, hence couldn't make streaming work.
255 |
256 |
257 | Thanks for reading. Stay tuned for more.
258 |
259 | I tweet about these topics and anything I'm exploring on a regular basis.
260 | [Follow me on twitter](https://twitter.com/clusteredbytes)
261 |
262 |
--------------------------------------------------------------------------------