├── .github ├── FUNDING.yml └── ISSUE_TEMPLATE │ ├── bug_report.md │ └── feature_request.md ├── .gitignore ├── README.org ├── consult-web-embark.el ├── consult-web.el ├── consult-web.org └── sources ├── consult-web-bing.el ├── consult-web-brave-autosuggest.el ├── consult-web-brave.el ├── consult-web-buffer.el ├── consult-web-chatgpt.el ├── consult-web-doi.el ├── consult-web-duckduckgo.el ├── consult-web-elfeed.el ├── consult-web-google-autosuggest.el ├── consult-web-google.el ├── consult-web-gptel.el ├── consult-web-line-multi.el ├── consult-web-notes.el ├── consult-web-pubmed.el ├── consult-web-scopus.el ├── consult-web-sources.el ├── consult-web-stackoverflow.el ├── consult-web-wikipedia.el └── consult-web-youtube.el /.github/FUNDING.yml: -------------------------------------------------------------------------------- 1 | # These are supported funding model platforms 2 | 3 | github: [armindarvish] # Replace with up to 4 GitHub Sponsors-enabled usernames e.g., [user1, user2] 4 | patreon: # Replace with a single Patreon username 5 | open_collective: # Replace with a single Open Collective username 6 | ko_fi: # Replace with a single Ko-fi username 7 | tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel 8 | community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry 9 | liberapay: # Replace with a single Liberapay username 10 | issuehunt: # Replace with a single IssueHunt username 11 | otechie: # Replace with a single Otechie username 12 | lfx_crowdfunding: # Replace with a single LFX Crowdfunding project-name e.g., cloud-foundry 13 | custom: # Replace with up to 4 custom sponsorship URLs e.g., ['link1', 'link2'] 14 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/bug_report.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Bug report 3 | about: Create a report to help us improve 4 | title: '' 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Describe the bug** 11 | A clear and concise description of what the bug is. 12 | 13 | **To Reproduce** 14 | Steps to reproduce the behavior: 15 | 1. Go to '...' 16 | 2. Click on '....' 17 | 3. Scroll down to '....' 18 | 4. See error 19 | 20 | **Expected behavior** 21 | A clear and concise description of what you expected to happen. 22 | 23 | **Screenshots** 24 | If applicable, add screenshots to help explain your problem. 25 | 26 | **Important Information:** 27 | - OS: [e.g. macOS] 28 | - Version of Emacs [e.g. 29] (or other Emacsen you use) 29 | - Version of `consult` (see [pkg-info](https://github.com/emacsorphanage/pkg-info)) 30 | - The installation method and the configuration you are using with your consult-mu. 31 | - If there is an error message, turn debug-on-error on (by `M-x toggle-debug-on-error`) and include the backtrace content in your report. 32 | - Optionally include log information, **but make sure to remove personal informaiton and secrets (e.g. API keys)**. To see the log, first set the variable `consult-web-log-level` to `debug` and then look ta the content of ` *consult-web-log*` (note the space at the beginning) or some other buffer name defined in `consult-web-log-buffer-name`. 33 | - If the error only exists when you have some other packages installed, list those packages (e.g. problem happens when evil is installed) 34 | 35 | **Additional context** 36 | Add any other context about the problem here. 37 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/feature_request.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Feature request 3 | about: Suggest an idea for this project 4 | title: '' 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Is your feature request related to a problem? Please describe.** 11 | A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] 12 | 13 | **Describe the solution you'd like** 14 | A clear and concise description of what you want to happen. 15 | 16 | **Describe alternatives you've considered** 17 | A clear and concise description of any alternative solutions or features you've considered. 18 | 19 | **Additional context** 20 | Add any other context or screenshots about the feature request here. 21 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Compiled 2 | *.elc 3 | 4 | # Packaging 5 | .cask 6 | 7 | # Backup files 8 | *~ 9 | 10 | # Undo-tree save-files 11 | *.~undo-tree 12 | 13 | 14 | # test files 15 | tests.org 16 | /tests 17 | 18 | # secrets 19 | /secrets 20 | 21 | #my personal files 22 | /armin -------------------------------------------------------------------------------- /README.org: -------------------------------------------------------------------------------- 1 | #+include: ~/OrgFiles/armin/org-macros.setup 2 | #+OPTIONS: h:1 num:nil toc:nil d:nil 3 | 4 | #+TITLE: consult-web - a powerful versatile omni search inside emacs 5 | #+AUTHOR: Armin Darvish 6 | #+LANGUAGE: en 7 | 8 | #+html: Armin Darvish 9 | #+html: GNU Emacs 10 | 11 | * Announcement 12 | This package is now deprecated in favor of its successor [[https://github.com/armindarvish/consult-omni][consult-omni]]! 13 | 14 | * About =consult-web= 15 | consult-web is a package for getting search results from one or several custom sources (web search engines, AI assistants, elfeed database, org notes, ...) directly in Emacs minibuffer. It provides wrappers and macros around [[https://github.com/minad/consult][consult]], to make it easier for users to get results from search engines, websites, AI assistants, etc. inside emacs minibuffer completion. In other words, consult-web enables getting consult-style multi-source or dynamically completed results in minibuffer but for search engines and APIs (e.g. simmilar to =consult-web= but for runing a google search from within emacs minibuffer). It provides a range of sources as examples, but the main idea here is to remain agnostic of the source and provide the toolset to the users to define their own sources similar to what consult does for local sources. 16 | 17 | Here is the mandatory screenshot: 18 | 19 | #+ATTR_ORG: :width 800px 20 | #+ATTR_LATEX: :width 800px 21 | #+ATTR_HTML: :width 800px 22 | [[https://github.com/armindarvish/consult-web/blob/screenshots/screenshots/dynamic-omni-screenshot.gif]] 23 | 24 | 25 | For a detailed review of the package and comparison to other tools, see my youtube video here: 26 | [[https://www.youtube.com/watch?v=7pDfyqBZwvo]] 27 | 28 | You can also read my [[https://www.armindarvish.com/post/web_omni_search_in_emacs_with_consult-web/][blog post]] about the motivation and broader context of why I made consult-web and how I use it in everyday examples. 29 | 30 | * Getting Started 31 | Before you start, make sure you understand three points: 32 | 1. *Important Note 1*: This is work in progress in its early stage and bugs and issues are very much expected. 33 | 2. *Important Note 2*: You should consider the general risks of using emacs to browse the web. By default, all codes are trusted inside emacs and browsers are naturally the target of many attacks. Therefore, it is important to be aware of the risks and be intentional about what links you open (or do not open) inside emacs. By default, consult-web would only be opening web pages (or calling APIs of) the sources (e.g. search engines, ...) and not any other websites. It's up to the user then to decide how she or he wants to open the links and chose their own risk tolerance. consult-web provides customization variables for different actions (e.g. opening links in external browsr v.s. in emacs), so make sure you know how to set everything up. 34 | 3. *Important Note 3*: The functions provided in =consult-web-sources=, provide a basic demonstration for integrating different services (such as search providers), however since each service comes with its own terms and conditions (that may change over time and vary from location to location), it is difficult to provide all-encompassing solutions and maintain them over time. consult-web is agnostic of how you connect and integrate other services in your setup (because neither consult-web nor emacs collect any information of the users or their usage), and therefore ultimately only you the user are responsible for setting up everything correctly and understand consequences of the usage (e.g. costs of using paid APIs) and ensure to stay within the bounds of relevant laws and regulations for your usage (i.e. follow software user agreements, etc.). Therefore, it is important for you to read and understand how to use each service, and also understand what happens under the hood when you integrate the service with consult-web. I try my best to provide documentation here as well as on the [[https://github.com/armindarvish/consult-web/wiki][wiki pages]], and will try to help when possible but before you proceed understand that you do everything at your own risk. 35 | 36 | ** Installation 37 | If you want an example config see [[https://github.com/armindarvish/consult-web?tab=readme-ov-file#drop-in-example-config][Drop-in *Example Config*]]. Here is some detailed explanation; 38 | 39 | *** Requirements 40 | In order to use consult-web, you need emacs >28.0 (I have not tested earlier versions) and [[https://github.com/minad/consult][consult]]. While this is the only requirements, I suggest you review consult's README since it recommends some other packages and useful configurations for different settings. Some of those extra packages and settings can improve your experience of consult-web as well. In particular, the section about [[https://github.com/minad/consult#asynchronous-search][asynchronous search]] is important for learning how to use inputs to search for result and narrow down in minibuffer. In addition combining consult with other packages such as [[https://github.com/minad/vertico][vertico]], [[https://github.com/oantolin/orderless][orderless]], and [[https://github.com/oantolin/embark][embark]] can improve the functionality as well as user-experience. 41 | 42 | *** Installing consult-web Package 43 | consult-web is not currently on [[https://elpa.gnu.org/packages/consult.html][ELPA]] or [[https://melpa.org/#/consult][MELPA]]. Therefore, you need to install it using an alternative non-standard package manager such as [[https://github.com/radian-software/straight.el][straight.el]] or use manual installation. 44 | 45 | **** straight.el 46 | To install consult-web with straight.el you can use the following command. Make sure you load consult-web after loading consult (e.g. =require 'consult=). 47 | 48 | #+begin_src emacs-lisp 49 | (straight-use-package 50 | '(consult-web :type git :host github :repo "armindarvish/consult-web" :files (:defaults "sources/*.el")) 51 | #+end_src 52 | 53 | or if you use =use-package= macro with straight, you can do: 54 | 55 | #+begin_src emacs-lisp 56 | (use-package consult-web 57 | :straight (consult-web :type git :host github :repo "armindarvish/consult-web" :files (:defaults "sources/*.el")) 58 | :after consult 59 | ) 60 | #+end_src 61 | 62 | You can also fork this repository and use your own repo. 63 | 64 | **** manual installation 65 | Clone this repo and make sure the files are on your load path, as described on [[https://www.emacswiki.org/emacs/LoadPath][EmacsWiki]]. 66 | 67 | Make sure you load consult (e.g. =require 'consult=) before you load consult-web. 68 | 69 | 70 | ** Adding Search Sources 71 | *** Loading Sources 72 | **** load all default sources at once 73 | You can add search *ALL* the default sources by loading the provided =consult-web-sources= module: 74 | #+begin_src emacs-lisp 75 | (require 'consult-web-sources) 76 | #+end_src 77 | This provides sources for some popular services, and adds a long list of interactive commands (dynamic search, static search or both depending on the source). Over time I hope to add more services, hopefully by contribution from the community as well. Note that these are also good examples for you to learn how to add your own sources or tweak the current ones for your specific use-cases. 78 | 79 | **** load a single source 80 | Alternatively you can load a single source by just requiring the corresponding file. For example for google, you can do: 81 | #+begin_src emacs-lisp 82 | (require 'consult-web-google) 83 | (require 'consult-web-sources) 84 | #+end_src 85 | This would add interactive commands only for searching google (e.g. =consult-web-google= and =consult-web-dynamic-google=). 86 | 87 | **** load multiple sources but not all 88 | You can also load multiple sources (but not all) by setting the list =consult-web-sources-modules-to-load= and then loading =consult-web-sources=; 89 | #+begin_src emacs-lisp 90 | (setq consult-web-sources-modules-to-load '(consult-web-google consult-web-wikipedia)) 91 | (require 'consult-web-sources) 92 | #+end_src 93 | This limits the sources that =consult-web-sources= loads to ONLY those defined in =consult-web-sources-modules-to-load=. 94 | 95 | *** List of Sources Provided by Default 96 | |--------------------------+-------------------------| 97 | | Source | Category | 98 | |--------------------------+-------------------------| 99 | | [[https://github.com/armindarvish/consult-web?tab=readme-ov-file#open-ai-aka-chatgpt][chatGPT]] | Simple AI prompts | 100 | | [[https://github.com/armindarvish/consult-web?tab=readme-ov-file#bing][Bing]] | Search Engine | 101 | | [[https://github.com/armindarvish/consult-web?tab=readme-ov-file#brave][Brave]] | Search Engine | 102 | | [[https://github.com/armindarvish/consult-web?tab=readme-ov-file#brave-auto-suggest][Brave AutoSuggest]] | AutoSuggest | 103 | | [[https://github.com/armindarvish/consult-web?tab=readme-ov-file#consult-line-multi][consult-line-multi]] | Local Text in Buffers | 104 | | [[https://github.com/armindarvish/consult-web?tab=readme-ov-file#consult-notes][consult-notes]] | Local Notes | 105 | | [[https://github.com/armindarvish/consult-web?tab=readme-ov-file#buffers][consult-buffer]] | Local Buffers | 106 | | [[https://github.com/armindarvish/consult-web?tab=readme-ov-file#duckduckgo-limited-api][DuckDuckGo (Limited API)]] | Search Suggestions | 107 | | [[https://github.com/armindarvish/consult-web?tab=readme-ov-file#elfeed][Elfeed]] | Feeds (RSS, videos,...) | 108 | | [[https://github.com/armindarvish/consult-web?tab=readme-ov-file#google][Google]] | Search Engine | 109 | | [[https://github.com/armindarvish/consult-web?tab=readme-ov-file#google-autosuggest][Google Autosuggest]] | AutoSuggest | 110 | | [[https://github.com/armindarvish/consult-web?tab=readme-ov-file#gptel][gptel]] | AI Assistant | 111 | | [[https://github.com/armindarvish/consult-web?tab=readme-ov-file#doiorg][Doi.org]] | Academic Reference | 112 | | [[https://github.com/armindarvish/consult-web?tab=readme-ov-file#pubmed-entrez-api][PubMed]] | Academic Reference | 113 | | [[https://github.com/armindarvish/consult-web?tab=readme-ov-file#scopus][Scopus]] | Academic Reference | 114 | | [[https://github.com/armindarvish/consult-web?tab=readme-ov-file#stackoverflow][StackOverflow]] | Community Forum | 115 | | [[https://github.com/armindarvish/consult-web?tab=readme-ov-file#wikipedia][Wikipedia]] | Encyclopedia | 116 | | [[https://github.com/armindarvish/consult-web?tab=readme-ov-file#youtube][YouTube]] | Videos | 117 | |--------------------------+-------------------------| 118 | 119 | **** Web Search Sources 120 | ***** Bing 121 | Microsoft Azure cloud services include Bings Search API. You need to make an account and get an API Key. Follow the official documents here: [[https://www.microsoft.com/en-us/bing/apis][Bing Search APIs | Microsoft Bing]] to get started. Once you have an API key, you can set it up in consult-web using the following variables; 122 | #+begin_src emacs-lisp 123 | (require 'consult-web-bing) 124 | (setq consult-web-bing-api-key "YOUR-BING-API-KEY-OR-FUNCTION") 125 | (add-to-list 'consult-web-dynamic-sources "Bing") ;; or (add-to-list 'consult-web-multi-sources...) 126 | #+end_src 127 | ***** Brave 128 | Brave provides a very good and easy-to-use API with reasonable limits and is one of my favorites when it comes to programmatic search. For API documentation refer to the official website: [[https://brave.com/search/api/][Brave Search API]]. Once you sign up and create an account you can access the documentation and create an API key. This API key can then be set in consult-web in the following custom variable. For a more secure approach you can pass a function that retrieves the key to this variable: 129 | 130 | #+begin_src emacs-lisp 131 | (require 'consult-web-brave) 132 | (setq consult-web-brave-api-key "YOUR-BRAVE-API-KEY-OR-FUNCTION") 133 | (add-to-list 'consult-web-dynamic-sources "Brave") ;; or (add-to-list 'consult-web-multi-sources...) 134 | #+end_src 135 | 136 | ***** Brave Autosuggest 137 | [[https://api.search.brave.com/res/v1/suggest/search][Brave AutoSuggest API]], provides completion for search terms. You need to subscribe to autosuggest plan and create an API key specifically for the autosuggest service. Follow the official documentation here: [[https://brave.com/search/api/][Brave Search API]] and once you have an API key for autosuggest, you can set it in consult-web in the following custom variable. For a more secure approach you can pass a function that retrieves the key to this variable: 138 | #+begin_src emacs-lisp 139 | (require 'consult-web-brave-autosuggest) 140 | (setq consult-web-brave-autosuggest-api-key "YOUR-BRAVE-AUTOSUGGEST-API-KEY-OR-FUNCTION") 141 | (setq consult-web-default-autosuggest-command #'consult-web-dynamic-brave-autosuggest) 142 | #+end_src 143 | 144 | ***** DOI.org 145 | DOI.org allows you to get the target url for DOI strings (for example for academic literature). You can use the this source if you want to enter the DOI and get the target link. consult-web uses this in the backend to get target links for paper found on scopus. If you want interactive commands for the DOI.org service, you can use also use this source by: 146 | 147 | #+begin_src emacs-lisp 148 | (require 'consult-web-doi) 149 | #+end_src 150 | 151 | and then run =consult-web-dynamic-doiorg= or use =consult-web--doi-to-url= function. 152 | 153 | ***** DuckDuckGo Limited API 154 | Unfortunately DuckDuckGo does not support an official API access (anymore!). In fact on their website they clearly discourage the users against doing programmatic searches. They do provide a very Limited API that only suggests relevant topics with search links on DuckDuckGo. It's not a very useful tool as is, but I am including it in the resources as an interesting example fr users that may like the idea. You can use it by; 155 | 156 | #+begin_src emacs-lisp 157 | (require 'consult-web-duckduckgo) 158 | (add-to-list 'consult-web-dynamic-sources "DuckDuckGo API") ;; or (add-to-list 'consult-web-multi-sources...) 159 | #+end_src 160 | 161 | ***** Google 162 | 163 | The official way to use google as a search engine and fetch results is through an API, [[https://programmablesearchengine.google.com/about/][Google Custom Search]]. consult-web provides example functions for using this service in [[file:sources/consult-web-google.el]]. This source can also be added to the multi-source interactive commands (e.g. =consult-web-dynamic=, =consult-web-multi=,...) by adding ="Google"= to the relevant source list variable. 164 | 165 | Note that to use this source you need to sign up for google API services and get an *API key* and a *cx number* (see [[https://developers.google.com/custom-search/v1/using_rest][REST API]]). Once you have an API KEY and a cx number, you can set them in the following custom variables. For a more secure approach you can also pass functions that return the values instead of directly passing strings: 166 | 167 | #+begin_src emacs-lisp 168 | (require 'consult-web-google) 169 | (consult-web-google-customsearch-key "YOUR-GOOGLE-API-KEY-OR-FUNCTION") 170 | (consult-web-google-customsearch-cx "YOUR-GOOGLE-CX-NUMBER-OR-FUNCTION") 171 | (add-to-list 'consult-web-dynamic-sources "Google") ;; or (add-to-list 'consult-web-multi-sources...) 172 | #+end_src 173 | 174 | ***** Google Autosuggest 175 | Google Autosuggest is not well-documented and likely not officially upported, but using suggestqueries.google.com seems to work, if you pass a client parameter (e.g. http://suggestqueries.google.com/complete/search?client=firefox&q=YOURQUERY). I have not been able to find any official document that clearly states that this is not an allowed use-case, yet there are not official documentation saying that this is allowed either. Therefore this is currently not a reliable tool but since it works, I have included it as an example in the sources. You can use it by: 176 | 177 | #+begin_src emacs-lisp 178 | (require 'consult-web-google-autosuggest) 179 | (setq consult-web-default-autosuggest-command #'consult-web-dynamic-google-autosuggest) 180 | #+end_src 181 | 182 | ***** Wikipedia 183 | [[https://www.mediawiki.org/wiki/API:Main_page][Wikipedia's API]], provides a way to get programmatic search results for free. Search only operations do not need an account or API key. Therefore, you can directly use =consult-web-wikipedia= or =consult-web-dynamic-wikipedia= and you can also add Wikipedia to multi-source interactive commands for example by: 184 | #+begin_src emacs-lisp 185 | (require 'consult-web-wikipedia) 186 | (add-to-list 'consult-web-dynamic-sources "Wikipedia") ;; or (add-to-list 'consult-web-multi-sources...) 187 | #+end_src 188 | ***** StackOverFlow 189 | While stack exchange allows using its API without an account or API key (a.k.a. anonymously), it is recommended to register for an API key through [[https://stackapps.com/][stackapps.com]], to get larger quota, and avoid getting blocked by human verification, etc. Once you have a key, you can set it in consult-web by the following custom variable. alternatively use a function that returns the key for a more secure approach: 190 | 191 | #+begin_src emacs-lisp 192 | (require 'consult-web-stackoverflow) 193 | (setq consult-web-stackexchange-api-key "YOUR-STACKEXCHANGE-API-KEY-OR-FUNCTION") 194 | (add-to-list 'consult-web-dynamic-sources "StackOverflow") ;; or (add-to-list 'consult-web-multi-sources...) 195 | #+end_src 196 | 197 | Follow the official API documentation here: [[https://api.stackexchange.com/docs][StackExchange API Docs]]. You can use their interactive tools, such as [[https://api.stackexchange.com/docs/advanced-search][Advanced Search API]] to get a sense of query parameters and build your custom tool. consult-web provides examples in =consult-web-stackoverflow= and =consult-web-dynamic-stackoverflow= for fetching results from stackoverflow, but you can make your own custom tool from other stack exchange sources as well. 198 | 199 | ***** PubMed Entrez API 200 | If you would like to use PubMed to search academic literature, then =consult-web-pubmed= and =consult-web-dynamic-pubmed= provide interactive commands to search PubMed. This uses PubMed's API (as opposed to parsing the html webpage), through PubMed's Entez e-utilities services which requires an API Key. For official documentation, see [[https://www.ncbi.nlm.nih.gov/books/NBK25500/][PubMed Entrez e-utilities]]. Once you create an API key, you can set it in consult-web using the following custom variable or a function that returns the key instead of directly using the string. 201 | #+begin_src emacs-lisp 202 | (require 'consult-web-pubmed) 203 | (setq consult-web-pubmed-api-key "YOUR-PUBMED-API-KEY-OR-FUNCTION") 204 | (add-to-list 'consult-web-scholar-sources "PubMed") ;; or (add-to-list 'consult-web-multi-sources...) 205 | #+end_src 206 | 207 | I do provide an example of using html-parsing for pubmed webpage (i.e. search pubmed anonymously) in the [[https://github.com/armindarvish/consult-web/wiki][wiki pages]]. This is mainly to demonstrate how one can define a source that uses html-parsing, otherwise APIs are generally preferred. 208 | 209 | ***** Scopus 210 | You can also use [[https://www.scopus.com/][Scopus]] for academic literature. To use scopus, you need to sign up for an API key. Follow the official documents here: [[https://dev.elsevier.com/][Elsevier Developer Portal]]. Once you create an API key, you can set it in consult-web using the following custom variable or a function that returns the key instead of directly using the string. 211 | 212 | #+begin_src emacs-lisp 213 | (require 'consult-web-scopus) 214 | (setq consult-web-scopus-api-key "YOUR-SCOPUS-API-KEY-OR-FUNCTION") 215 | (add-to-list 'consult-web-scholar-sources "Scopus") ;; or (add-to-list 'consult-web-multi-sources...) 216 | #+end_src 217 | 218 | ***** Youtube 219 | You can also search YouTube videos in consult-web. You need a Google API key. Refer to the official documentation: [[https://developers.google.com/youtube/v3/getting-started][YouTube Data API | Google for Developers]] for how to set things up. Then you can use your API key by setting the corresponding variable in consult-web: 220 | #+begin_src emacs-lisp 221 | (require 'consult-web-youtube) 222 | (setq consult-web-scopus-api-key "YOUR-GOOGLE-API-KEY-OR-FUNCTION") 223 | (add-to-list 'consult-web-dynamic-sources "Youtube") ;; or (add-to-list 'consult-web-multi-sources...) 224 | #+end_src 225 | 226 | 227 | 228 | **** AI Assistants 229 | ***** Open AI (a.k.a. chatGPT) 230 | If you want to use [[https://openai.com/product][Open AI's API]] (e.g. for chatGPT results), you need to get an API key. Follow the official documentation here: [[https://platform.openai.com/docs/introduction][Open AI API docs]], and once you have an API key, you can set it in consult-web with the following custom variable. Using a function that returns the key is more secure than using the string directly. 231 | #+begin_src emacs-lisp 232 | (require 'consult-web-chatgpt) 233 | (setq consult-web-openai-api-key "YOUR-OPENAI-API-KEY-OR-FUNCTION") 234 | (add-to-list 'consult-web-dynamic-sources "chatGPT") ;; or (add-to-list 'consult-web-multi-sources...) 235 | 236 | #+end_src 237 | 238 | consult-web-sources provides =consult-web-chatgpt= and =consult-web-dynamic-chatgpt= commands for directly sending a prompt to Open AI API and retrieving responses. Note that depending on the prompt this may be a bit slower and also be aware that when prompts are sent to OpenAI, this can incur charges on your account, so you may want to be cautious of how you use these functions. 239 | 240 | Note that these commands are simple one-time http requests. For a more extended conversation and a full-feature experience, you can use consult-web-gptel below. 241 | 242 | ***** gptel 243 | Another easy way to integrate AI assistants with consult-web is to use the amazing package, [[https://github.com/karthink/gptel][gptel]]. This is by far my favorite generative AI package in emacs because of how easy it is to integrate it with other things in emacs. To use this with consult-web, install gptel following the packages's documentation here: [[https://github.com/karthink/gptel][gptel]]. Once you have gptel setup, you can call =consult-web-gptel=, or =consult-web-dynamic-gptel= to get answers to your prompts. Note that, these functions do not fetch the answers to your prompt right away and wait for either a preview or for selecting the candidate to send the prompt to the backend AI. This is because sending the prompt to the backend can be expensive (in terms of paid services) and can also be slow depending on the backend and how long the answer is. Therefore, by design the user needs to actively decide to send the prompt to the backend. 244 | 245 | You can add any of the gptel sources to any of the multi-source interactive commands. For example: 246 | 247 | #+begin_src emacs-lisp 248 | (require 'consult-web-gptel) 249 | (add-to-list consult-web-dynamic-sources "gptel") ;; or add-to-list consult-web-multi,... 250 | #+end_src 251 | **** Local Sources 252 | In addition consult-web provides utilities to combine web search sources with local sources to create "omni" multi-source searches. The followings are the sources that are provided by default. 253 | ***** buffers 254 | Generally for consult-sources, you can use =consult-web--make-source-from-consult-source= you make a consult-web version of the same source. This is for example done in =consult-web-buffers= to covert sources in =consult-buffer-sources= to consult-web compatible versions. 255 | 256 | #+begin_src emacs-lisp 257 | (require 'consult-web-buffers) 258 | (add-to-list 'consult-web-dynamic-omni "Buffer") 259 | #+end_src 260 | ***** consult-line-multi 261 | Another useful consult source for omni searches is =consult-line-multi=. this is provided in =consult-web-line-multi=. 262 | #+begin_src emacs-lisp 263 | (require 'consult-web-line-multi) 264 | (add-to-list 'consult-web-dynamic-omni "Line Multi") 265 | #+end_src 266 | #+end_src 267 | ***** consult-notes 268 | If you use [[https://github.com/mclear-tools/consult-notes][consult-notes]], you can also combine the consult-web compatible version with other sources to create powerful omni search results. 269 | #+begin_src emacs-lisp 270 | (require 'consult-web-notes) 271 | (add-to-list 'consult-web-dynamic-omni "Reference Roam Nodes" "Zettel Roam Nodes") 272 | #+end_src 273 | 274 | 275 | ***** elfeed 276 | You can also add your elfeed database as a source to consult-web by; 277 | 278 | #+begin_src emacs-lisp 279 | (require 'consult-web-elfeeed) 280 | (add-to-list 'consult-web-dynamic-omni "elfeed") 281 | #+end_src 282 | 283 | 284 | **** What about other sources? 285 | I am hoping to add examples for more sources over time. I think the currently provided examples in [[file:sources/][sources]], should be sufficient for users to learn how to make new sources on their own. I will add more in-depth instructions and explanations in the [[https://github.com/armindarvish/consult-web/wiki][wiki pages]] for you to understand how to define new custom sources. If you use the package and come up with new sources or use-cases, I encourage you to share it with others in the wiki pages as well so everyone can benefit from it. If such source or use-case is novel enough that adds some new value beyond the current examples, I can also consider adding that to the repo, otherwise we can just keep these in the wiki pages, to avoid adding unnecessary bloat. 286 | 287 | ** Configuration 288 | consult-web is built with the idea that the user should be able to customize everything based on their use-case, therefore the user is very much expected to learn how to configure this package. Therefore, I recommend you read through this section and understand how to configure the package according to your needs and for your specific use-case. 289 | 290 | 291 | *** Customization Variables 292 | The following customizable variables are provided: 293 | 294 | **** main 295 | ***** =consult-web-default-browse-function= 296 | Default browse function for opening urls. This can be set to external browser function by; 297 | #+begin_src emacs-lisp 298 | (setq consult-web-default-browse-function 'browse-url) 299 | #+end_src 300 | ***** =consult-web-alternate-browse-function= 301 | Secondary browse function for opening urls. This can for example be set to eww or some other browsers for quick access to an alternative browser with embark actions. 302 | #+begin_src emacs-lisp 303 | (setq consult-web-alternate-browse-function 'eww-browse-url) 304 | #+end_src 305 | ***** =consult-web-default-preview-function= 306 | Default function to use for previewing links. This can for example be set to [[https://www.gnu.org/software/emacs/manual/html_mono/eww.html][eww]]: 307 | 308 | #+begin_src emacs-lisp 309 | (setq consult-web-default-preview-function #'eww-browse-url) 310 | #+end_src 311 | 312 | or [[https://www.gnu.org/software/emacs/manual/html_node/emacs/Embedded-WebKit-Widgets.html][xwidegt-webkit]]: 313 | 314 | #+begin_src emacs-lisp 315 | (setq consult-web-default-preview-function #'xwidget-webkit-browse-url) 316 | #+end_src 317 | 318 | ***** =consult-web-show-preview= 319 | This turns previews on/off for all consult-web sources. It is recommended to set this to =t= and use =preview-key= to control previews per source. 320 | 321 | ***** =consult-web-preview-key= 322 | This is the default preview key. A good choice might be "C-o". 323 | #+begin_src emacs-lisp 324 | (setq consult-web-preview-key "C-o") 325 | #+end_src 326 | 327 | ***** =consult-web-default-count= 328 | By default consult-web retrieves only *up to* this many results per source. It is recommended to keep this to a low number (e.g. 5 to 10) to keep the performance fast. The default is set to 5 because nowadays for most everyday use-cases, you probably won't need more than the top 5 results. 329 | 330 | #+begin_src emacs-lisp 331 | (setq consult-web-default-count "5") 332 | #+end_src 333 | 334 | Keep in mind that with dynamic commands of consult-web, you can always increase the number by passing arguments to =-n=, =--count=, or =:count= (for example by typing =search term -- -n 30=, you can retrieve up to 30 results. 335 | 336 | 337 | ***** =consult-web-default-page= 338 | This is similar to going to page 2, 3,...,N on a classic search result page. If this is set to N, first (N * count/page) results are skipped and the next page of results are shown. It is recommended to keep this as default, 0, to see the top results. 339 | 340 | Keep in mind that with dynamic commands of consult-web, you can always change the page by passing values to arguments =-p=, =--page=, or =:page= (for example by typing =search term -- -p 2=, you can get page 2. 341 | 342 | ***** =consult-web-group-by= 343 | This is the field that is used to group the results. By default, results are grouped by the domain of the url (e.g. "bbc.com" v.s. "nytimes.com"), but you can change this to group by the name of the source (e.g. "Google" v.s. "Brave",...) or by the full url ,etc. 344 | 345 | #+begin_src emacs-lisp 346 | (setq consult-web-group-by :source) 347 | #+end_src 348 | 349 | 350 | ***** =consult-web-multi-sources= 351 | This is a list of source name strings (e.g. ='("Google", "Wikipedia", "chatGPT")=) that will be used in the command =consult-web-multi= (see above for description). 352 | 353 | ***** =consult-web-dynamic-sources= 354 | This is a list of source name strings (e.g. ='("Google", "Wikipedia", "chatGPT")=) that will be used in the command =consult-web-dynamic= (see above for description). 355 | 356 | ***** =consult-web-scholar-sources= 357 | This is a list of source name strings (e.g. ='("PubMed")=) that will be used in the command =consult-web-scholar= (see above for description). 358 | 359 | 360 | ***** =consult-web-omni-sources= 361 | This is a list of source name strings (e.g. ='("Buffer" "Brave" "gptel")=) that will be used in the command =consult-web-omni= (see above for description). 362 | ***** =consult-web-dynamic-omni-sources= 363 | This is a list of source name strings (e.g. ='("Reference Roam Nodes" "Zettel Roam Nodes" "Line Multi" "elfeed" "Brave" "gptel" "Youtube")=) that will be used in the command =consult-web-dynamic-omni= (see above for description). 364 | 365 | 366 | ***** =consult-web-highlight-matches= 367 | Whether consult-web highlights matches of the search term in the minibuffer candidates. This is useful to highlight the relevance of the search results. 368 | 369 | 370 | ***** =consult-web-default-interactive-command= 371 | This is a convenient feature to bind your favorite consult-web interactive command to the command called =consult-web=, so it is easier to remember and find when you call =M-x=. You can bind this to any of the multi-source interactive commands (such as =consult-web-dyamic= or =consult-web-multi=, ...) or you can bind it to a single-source command (e.g. =consult-web-dynamic-google= or =consult-web-google=,...) or alternatively define your own custom command. 372 | 373 | ***** =consult-web-default-autosuggest-command= 374 | Default autosuggest command. consult-web provides two examples with =consult-web-dynamic-brave-autosuggest= and =consult-web-dynamic-google-autosuggest=, but you can also define other custom autosuggest commands from other sources (e.g. google, wikipedia, ...) 375 | 376 | #+begin_src emacs-lisp 377 | (setq consult-web-default-autosuggest-command #'consult-web-dynamic-brave-autosuggest) 378 | #+end_src 379 | 380 | ***** =consult-web-dynamic-input-debounce= 381 | In dynamic commands, the dynamic collection process is started only when there has not been new 382 | input for =consult-web-dynamic-input-debounce= seconds. If you type slow or think you need time to think for what you want to search, you may want to increase this number, so you don't run searches prematurely, especially if you want to avoid running premature search terms on paid services. 383 | By default this inherits from consult's built-in input-debounce value, which is 0.5. Personally I find that a bit too fast for consult-web because I do not want consult-web to send a query to paid openai API while I am still typing my query so I slow it down to 0.8 - 1s. 384 | 385 | #+begin_src emacs-lisp 386 | (setq consult-web-dynamic-input-debounce 0.8) 387 | #+end_src 388 | 389 | 390 | ***** =consult-web-dynamic-input-throttle= 391 | In dynamic commands, the dynamic collection process is started only every =consult-web-dynamic-input-throttle= seconds. If you use API services that have limited number of queries per second, you may want to increase this number to avoid getting errors. I set this to 2x my input-debounce value: 392 | 393 | #+begin_src emacs-lisp 394 | (setq consult-web-dynamic-input-throttle 1.6) 395 | #+end_src 396 | 397 | 398 | ***** =consult-web-dynamic-refresh-delay= 399 | In dynamic commands, the completion UI is only updated every =consult-web-dynamic-refresh-delay= seconds. You probably want to run this as fast as =consult-web-dynamic-input-debounce=. 400 | 401 | #+begin_src emacs-lisp 402 | (setq consult-web-dynamic-input-throttle 0.8) 403 | #+end_src 404 | 405 | **** per source 406 | As mentioned above, once you load sources (e.g. =(require 'consult-web-sources)=), then you will get more customization variables per source. These include variables for API keys. Here are some examples: 407 | 408 | ***** =consult-web-google-customsearch-key= and =consult-web-google-customsearch-cx= 409 | API Key and cx-number for [[https://programmablesearchengine.google.com/about/][Google custom Search]]. 410 | 411 | #+begin_src emacs-lisp 412 | (consult-web-google-customsearch-key "YOUR-GOOGLE-API-KEY-OR-FUNCTION") 413 | (consult-web-google-customsearch-cx "YOUR-GOOGLE-CX-NUMBER-OR-FUNCTION") 414 | #+end_src 415 | 416 | ***** =consult-web-brave-api-key= 417 | [[https://brave.com/search/api/][Brave Search API]] key. 418 | #+begin_src emacs-lisp 419 | (setq consult-web-brave-api-key "YOUR-BRAVE-API-KEY-OR-FUNCTION") 420 | #+end_src 421 | 422 | ***** =consult-web-openai-api-key= 423 | [[https://openai.com/product][Open AI's API]] key 424 | #+begin_src emacs-lisp 425 | (setq consult-web-openai-api-key "YOUR-OPENAI-API-KEY-OR-FUNCTION") 426 | #+end_src 427 | 428 | ***** =consult-web-stackexchange-api-key= 429 | StackExchange API key from [[https://stackapps.com/][stackapps.com]]. 430 | #+begin_src emacs-lisp 431 | (setq consult-web-stackexchange-api-key "YOUR-STACKEXCHANGE-API-KEY-OR-FUNCTION") 432 | #+end_src 433 | 434 | Be aware that as I add more sources, there may be more customization variables added. Refer to release notes or .... pages for more info if needed. 435 | 436 | 437 | ** Drop in “example config” 438 | Here is a drop-in config snippet that puts everything mentioned above together. Read the sections above for more details. 439 | 440 | #+begin_src emacs-lisp 441 | (use-package consult-web 442 | :straight (consult-web :type git :host github :repo "armindarvish/consult-web" :branch "main" :files (:defaults "sources/*.el")) 443 | :after consult 444 | :custom 445 | ;; General settings that apply to all sources 446 | (consult-web-show-preview t) ;;; show previews 447 | (consult-web-preview-key "C-o") ;;; set the preview key to C-o 448 | (consult-web-highlight-matches t) ;;; highlight matches in minibuffer 449 | (consult-web-default-count 5) ;;; set default count 450 | (consult-web-default-page 0) ;;; set the default page (default is 0 for the first page) 451 | 452 | ;;; optionally change the consult-web debounce, throttle and delay. 453 | ;;; Adjust these (e.g. increase to avoid hiting a source (e.g. an API) too frequently) 454 | (consult-web-dynamic-input-debounce 0.8) 455 | (consult-web-dynamic-input-throttle 1.6) 456 | (consult-web-dynamic-refresh-delay 0.8) 457 | 458 | :config 459 | ;; Add sources and configure them 460 | ;;; load the example sources provided by default 461 | (require 'consult-web-sources) 462 | 463 | ;;; set multiple sources for consult-web-multi command. Change these lists as needed for different interactive commands. Keep in mind that each source has to be a key in `consult-web-sources-alist'. 464 | (setq consult-web-multi-sources '("Brave" "Wikipedia" "chatGPT" "Google")) ;; consult-web-multi 465 | (setq consult-web-dynamic-sources '("gptel" "Brave" "StackOverFlow" )) ;; consult-web-dynamic 466 | (setq consult-web-scholar-sources '("PubMed")) ;; consult-web-scholar 467 | (setq consult-web-omni-sources (list "elfeed" "Brave" "Wikipedia" "gptel" "YouTube" 'consult-buffer-sources 'consult-notes-all-sources)) ;;consult-web-omni 468 | (setq consult-web-dynamic-omni-sources (list "Known Project" "File" "Bookmark" "Buffer" "Reference Roam Nodes" "Zettel Roam Nodes" "Line Multi" "elfeed" "Brave" "Wikipedia" "gptel" "Youtube")) ;;consult-web-dynamic-omni 469 | 470 | ;; Per source customization 471 | ;;; Pick you favorite autosuggest command. 472 | (setq consult-web-default-autosuggest-command #'consult-web-dynamic-brave-autosuggest) ;;or any other autosuggest source you define 473 | 474 | ;;; Set API KEYs. It is recommended to use a function that returns the string for better security. 475 | (setq consult-web-google-customsearch-key "YOUR-GOOGLE-API-KEY-OR-FUNCTION") 476 | (setq consult-web-google-customsearch-cx "YOUR-GOOGLE-CX-NUMBER-OR-FUNCTION") 477 | (setq consult-web-brave-api-key "YOUR-BRAVE-API-KEY-OR-FUNCTION") 478 | (setq consult-web-stackexchange-api-key "YOUR-STACKEXCHANGE-API-KEY-OR-FUNCTION") 479 | (setq consult-web-pubmed-api-key "YOUR-PUBMED-API-KEY-OR-FUNCTION") 480 | (setq consult-web-openai-api-key "YOUR-OPENAI-API-KEY-OR-FUNCTION") 481 | ;;; add more keys as needed here. 482 | ) 483 | #+end_src 484 | 485 | * Usage and Features 486 | For explanation of features and comparison to some other packages, you can watch my youtube video: 487 | https://www.youtube.com/watch?v=7pDfyqBZwvo 488 | 489 | ** Static v.s. Dynamic Interactive Commands 490 | For each source, you may have static or dynamic commands. Static commands query the use for an input and then fetch the results. Dynamic commands, do dynamic completion as the user types (fetch results as the user is typifying). Dynamic commands feel a bit more intuitive and modern in 2024, but on the other hand have the disadvantage of sending the query to the servers multiple times especially if you type slowly! Depending on the service provider and the API model, you may want to avoid hitting the server too frequently (for example for services that you pay per query), therefore for certain services a static command might be a better choice than the dynamic command. Using the macro =consult-web-define-source=, you can chose to create static, dynamic or both by passing =nil=, =t=, or ='both= to the keyword =:dynamic=. Here is an example from the source code, for creating both static and dynamic commands for Brave search: 491 | 492 | #+begin_src emacs-lisp 493 | (consult-web-define-source "Brave" 494 | :narrow-char ?b 495 | :face 'consult-web-engine-source-face 496 | :request #'consult-web--brave-fetch-results 497 | :preview-key consult-web-preview-key 498 | :search-history 'consult-web--search-history 499 | :selection-history 'consult-web--selection-history 500 | :dynamic 'both 501 | ) 502 | #+end_src 503 | 504 | 505 | ** Multi-Source Interactive Commands 506 | consult-web does provide a few interactive commands. These are provided as good examples for users to follow when making their own custom commands for their work flow. 507 | 508 | 1. =consult-web-multi=: This is an interactive command that uses multiple sources, as defined by =consult-web-multi-sources=, and shows the results in minibuffer completion. Here is an example screenshot: 509 | 510 | 511 | #+ATTR_ORG: :width 800px 512 | #+ATTR_LATEX: :width 800px 513 | #+ATTR_HTML: :width 800px 514 | [[https://github.com/armindarvish/consult-web/blob/screenshots/screenshots/multi-screenshot.gif]] 515 | 516 | Note that consult-web-multi does not provide dynamic completion but some might find using this more intuitive for narrowing down the results. The user provides one search term, and once the results are retrieved, typing in the minibuffer will narrow down the candidates. In addition, as can be seen in the screenshot above, depending on your minibuffer config for sorting, it is possible to remember the candidates you selected before by sorting them on top (by using packages like [[https://github.com/oantolin/orderless][orderless]] or [[https://github.com/radian-software/prescient.el][prescient.el]]). 517 | 518 | 2. =consult-web-dynamic=: This is a “multi-source” interactive command that uses “dynamic” collection. This allows dynamic completion of the search results (results are fetched as the user types). Here is a screenshot: 519 | 520 | #+ATTR_ORG: :width 800px 521 | #+ATTR_LATEX: :width 800px 522 | #+ATTR_HTML: :width 800px 523 | [[https://github.com/armindarvish/consult-web/blob/screenshots/screenshots/dynamic-screenshot.gif]] 524 | 525 | 3. =consult-web-scholar=: This is similar to consult-web-dynamic, and is provided as an extra example to show how to make combination of sources for a specific purpose, in this case searching academic research literature. Here is a screenshot: 526 | 527 | #+ATTR_ORG: :width 800px 528 | #+ATTR_LATEX: :width 800px 529 | #+ATTR_HTML: :width 800px 530 | [[https://github.com/armindarvish/consult-web/blob/screenshots/screenshots/scholar-screenshot.gif]] 531 | 532 | 533 | 534 | 4. =consult-web-omni=: This is another static multi-source command provided for combining web sources and local sources to do omni searches. Here is a screenshot: 535 | 536 | #+ATTR_ORG: :width 800px 537 | #+ATTR_LATEX: :width 800px 538 | #+ATTR_HTML: :width 800px 539 | [[https://github.com/armindarvish/consult-web/blob/screenshots/screenshots/omni-screenshot.gif]] 540 | 541 | 542 | 543 | 5. =consult-web-dynamic-omni=: This is the dynamic version of the omni search for combination of web and local sources. For example, in the screenshot below you see results form org-roam notes, brave search engine, wikipedia, as well as gptel AI assistant: 544 | 545 | 546 | #+ATTR_ORG: :width 800px 547 | #+ATTR_LATEX: :width 800px 548 | #+ATTR_HTML: :width 800px 549 | [[https://github.com/armindarvish/consult-web/blob/screenshots/screenshots/dynamic-omni-screenshot.gif]] 550 | 551 | Note that the difference between =consult-web-multi= and =consult-web-omni= (and similarly between =consult-web-dynamic=, =consult-web-scholar=, =consult-web-dynamic-omni=) is the list of sources they use and therefore you can use them as you wish for any combination of sources. You can also define more interactive commands with various variation of sources following these examples. 552 | 553 | ** Dynamic Completion: Passing Arguments and Narrow Down 554 | 555 | Arguments can be passed to the dynamic interactive commands and further narrowing down the results can be done using a syntax similar to the “perl splitting” style in [[https://github.com/minad/consult?tab=readme-ov-file#asynchronous-search][consult asynchronous search]]. 556 | 557 | For narrowing down the results you need adding =#= after the search query. For example typing the following in the minibuffer: 558 | #+begin_example 559 | #emacs web search#github 560 | #+end_example 561 | 562 | First searches for “emacs web search”, and then uses “github” for narrow down. 563 | 564 | Furthermore, arguments can be passed to dynamic commands using similar syntax as =consult-grep=, too. For example typing the following in the minibuffer: 565 | 566 | #+begin_example 567 | #how to browse a url in emacs -- --model gpt-3.5-turbo 568 | #+end_example 569 | 570 | passes =gpt-3.5-turbo= as the value for the keyword argument =:model= to the backend functions of all the sources that fetch results. If any of those sources accept the keyword argument =:model=, the value =gpt-3.5-turbo= gets passed to them. For this reason it is recommended to always use functions that accept any keyword arguments (a.k.a. add =&allow-other-keys=) to avoid errors when non-existing keywords are passed to them. 571 | 572 | instead of using =--= , you can also use a keyword with colon =:=. The following would be similar to the example above: 573 | 574 | #+begin_example 575 | #how to browse a url in emacs -- :model gpt-3.5-turbo 576 | #+end_example 577 | 578 | ** Embark Actions 579 | You can load the default embark actions by; 580 | #+begin_src emacs-lisp 581 | (require 'consult-web-embark) 582 | #+end_src 583 | The default actions allow you to open the links in the default or alternate browser and also to copy or insert, title and/or url of the links. Other embark actions can be defined per your own specific work flow. See the YouTube video for an example, here: [[https://youtu.be/7pDfyqBZwvo?t=4962]]. 584 | 585 | ** Other Important Features 586 | *** Minimal Code Base 587 | Without docstrings and whitespaces the code is less than 1000 lines and it only depends on [[https://github.com/minad/consult][consult]] and built-in url-retrieve. 588 | 589 | *** Modular 590 | You can only load the parts you need. For example if all you need is an autosuggestion utility similar to =helm-google-autosuggest=, then you can do: 591 | 592 | #+begin_src emacs-lisp 593 | (require 'consult-web-brave-autosuggest) 594 | #+end_src 595 | This adds an extra 100-200 lines of code per source. This also means to add a new source, you only need to write a short piece of code following those examples! 596 | 597 | *** Customizable and Extendable 598 | Lots of customization options both for sources and also for running actions on the results. 599 | New sources can be added as you wish with different format, different actions,... 600 | 601 | *** Power User Capabilities 602 | Dynamic collection allows for complex workflows on the fly. 603 | Change query parameters on the fly by passing arguments. Select a random set of results ad-hoc using embark and run embark actions on them. This allows batch processing as well. For example to add a long list of results to an org-mode note for later review (as shown in this youtube video: [[https://youtu.be/7pDfyqBZwvo?t=4774]]). 604 | 605 | 606 | 607 | * Comparison to Other Packages 608 | To the best of my knowledge no other package provides the functionality and versatility of consult-web. Browsers like [[https://www.gnu.org/software/emacs/manual/html_node/emacs/EWW.html][EWW (GNU Emacs Manual)]] and [[https://www.gnu.org/software/emacs/manual/html_node/emacs/Embedded-WebKit-Widgets.html][Embedded WebKit Widgets (GNU Emacs Manual)]] only provide a browser and not the ability to get search results directly in minibuffer. Other built-in commands and packages such as [[https://www.emacswiki.org/emacs/WebJump][WebJump]] or [[https://github.com/hrs/engine-mode][engine-mode]] enable passing queries to search engines, but do not provide results in the minibuffer. =helm-google-autosuggest= in [[https://github.com/emacs-helm/helm][helm]] provides autosuggestion utility only. [[https://github.com/mnewt/counsel-web][counsel-web]] is the only package I know that provides google results directly in the minibuffer but the functionality is limited and the way it parses google website may get your IP flagged. consult-web took inspiration from all those packages and provides a much more powerful solution than any of those available solutions. 609 | 610 | * Bug Reports 611 | To report bug, first check if it is already reported in the [[https://github.com/armindarvish/consult-web/issues][*issue tracker*]] and see if there is an existing solution or add relevant comments and discussion under the same issue. If not file a new issue following these steps: 612 | 613 | 1. Make sure the dependencies are installed, and both =consult= and =url-retrieve= (or other relevant commands) work as expected. 614 | 615 | 3. Remove the package and install the latest version (along with dependencies) and see if the issue persists. 616 | 617 | 4. In a bare bone vanilla Emacs (>=28) (e.g. =emacs -Q=), install the latest version of consult-web (and its dependencies) without any configuration or other packages and see if the issue still persists. 618 | 619 | 5. File an issue and provide important information and context in as much detail as possible in your bug report. Important information can include: 620 | - Your operating system, version of Emacs (or the version of emacsen you are using), version of consult (see [[https://github.com/emacsorphanage/pkg-info][pkg-info]]). 621 | - The installation method and the configuration you are using with your consult-web. 622 | - If there is an error message, turn debug-on-error on (by =M-x toggle-debug-on-error=) and include the backtrace content in your report. 623 | - If you are using consult-web's built-in url-retrieve (e.g. =consult-web-url-retrieve-sync=) , you can change =consult-web-log-level= to ='debug=, and inspect the log buffer (hidden buffer called " **consult-web-log**" or other name set in =consult-web-log-buffer-name=). If you chose to include this information in your issue, *please make sure personal information and secrets (like API keys) are not exposed.* 624 | - If the error only exists when you have some other packages installed, list those packages (e.g. problem happens when evil is installed) 625 | 626 | * Contributions 627 | This is an open source package, and I appreciate feedback, suggestions, ideas, etc. There are lots of functionalities or sources that can be added to this package to improve different user's workflows, so if you have some ideas, feel free to file an issue for a feature request. 628 | 629 | If you want to contribute to the code, please note that the main branch is currently stable (as stable as a work in progress like this can be) and the develop branch is the current work in progress. So, *start from the develop branch* to get the latest work-in-progress updates and create a new branch with names such as feature/name-of-the-feature or fix/issue, ... Do the edits and then create a new pull request to merge back with the *develop* branch when you are done with your edits. 630 | 631 | Importantly, keep in mind that I am using a *literate programming approach* (given that this is a small project with very limited number of files) where everything goes into *consult-web.org* and then gets tangled to appropriate files (e.g. consult-web.el). If you open a pull-request where you directly edited the .el files, I will likely not approve it because that will then get overwritten later when I tangle from the .org file. In other words, *Do Not Edit The .el Files!* only edit the .org file and tangle to .el files. 632 | 633 | 634 | * Acknowledgments 635 | 636 | Obviously this package would not have been possible without the fabulous [[https://github.com/minad/consult][consult]] and [[https://github.com/oantolin/embark][embark]] packages. It also took inspiration from other packages including but not limited to [[https://github.com/hrs/engine-mode][engine-mode]], [[https://github.com/Malabarba/emacs-google-this][emacs-google-this]], [[https://github.com/emacs-helm/helm][helm]], and [[https://github.com/mnewt/counsel-web][counsel-web]]. 637 | -------------------------------------------------------------------------------- /consult-web-embark.el: -------------------------------------------------------------------------------- 1 | ;;; consult-web-embark.el --- Emabrk Actions for `consult-web' -*- lexical-binding: t -*- 2 | 3 | ;; Copyright (C) 2024 Armin Darvish 4 | 5 | 6 | ;; Author: Armin Darvish 7 | ;; Maintainer: Armin Darvish 8 | ;; Created: 2024 9 | ;; Version: 0.1 10 | ;; Package-Requires: ((emacs "27.1") (consult "0.34") (consult-web "0.1")) 11 | ;; Homepage: https://github.com/armindarvish/consult-web 12 | ;; Keywords: convenience 13 | 14 | ;;; Commentary: 15 | 16 | ;;; Code: 17 | 18 | ;;; Requirements 19 | 20 | (require 'embark) 21 | (require 'consult-web) 22 | 23 | ;;; Define Embark Action Functions 24 | 25 | (defun consult-web-embark-default-action (cand) 26 | "Calls the default action on CAND. 27 | 28 | Gets the default callback function from `consult-web-sources-alist'." 29 | (let* ((source (and (stringp cand) (get-text-property 0 :source cand)))) 30 | (funcall (plist-get (cdr (assoc source consult-web-sources-alist)) :on-callback) cand)) 31 | ) 32 | 33 | (add-to-list 'embark-default-action-overrides '(consult-web . consult-web-embark-default-action)) 34 | 35 | 36 | (defun consult-web-embark-insert-title (cand) 37 | "Insert the title oif the candidate at point" 38 | (if-let ((title (and (stringp cand) (get-text-property 0 :title cand)))) 39 | (insert (format " %s " title)))) 40 | 41 | (defun consult-web-embark-copy-title-as-kill (cand) 42 | "Copy the title of the candidate to `kill-ring'." 43 | (if-let ((title (and (stringp cand) (get-text-property 0 :title cand)))) 44 | (kill-new (string-trim title)))) 45 | 46 | (defun consult-web-embark-insert-url-link (cand) 47 | "Insert the title oif the candidate at point." 48 | (let* ((url (and (stringp cand) (get-text-property 0 :url cand))) 49 | (url (and (stringp url) (string-trim url))) 50 | (title (and (stringp cand) (get-text-property 0 :title cand)))) 51 | (when url 52 | (cond 53 | ((derived-mode-p 'org-mode) 54 | (insert (cond 55 | ((and url title) (format " [[%s][%s]] " url title)) 56 | (url (format " [[%s]] " url)) 57 | (t "")) 58 | )) 59 | ((derived-mode-p 'markdown-mode) 60 | (insert (cond 61 | ((and url title) (format " [%s](%s) " url title)) 62 | (url (format " <%s> " url)) 63 | (t "")) 64 | )) 65 | (t 66 | (insert (cond 67 | ((and url title) (format " %s (%s) " title url)) 68 | (url (format " %s " url)) 69 | (t "")) 70 | )))))) 71 | 72 | (defun consult-web-embark-copy-url-as-kill (cand) 73 | "Copy the url of the candidate to `kill-ring'." 74 | (if-let ((url (and (stringp cand) (get-text-property 0 :url cand)))) 75 | (kill-new (format " %s " (string-trim url))) 76 | )) 77 | 78 | (defun consult-web-embark-external-browse-link (cand) 79 | "Open the url with `consult-web-default-browse-function'" 80 | (if-let* ((url (and (stringp cand) (get-text-property 0 :url cand)))) 81 | (funcall consult-web-default-browse-function url))) 82 | 83 | (defun consult-web-embark-alternate-browse-link (cand) 84 | "Open the url with `consult-web-alternate-browse-function'" 85 | (if-let* ((url (and (stringp cand) (get-text-property 0 :url cand)))) 86 | (funcall consult-web-alternate-browse-function url))) 87 | 88 | (defun consult-web-embark-external-browse-search-link (cand) 89 | "Open the search url (the search engine page) in the external browser." 90 | (if-let* ((search-url (and (stringp cand) (get-text-property 0 :search-url cand)))) 91 | (funcall #'browse-url search-url))) 92 | 93 | (defun consult-web-embark-show-preview (cand) 94 | "Get a preview of CAND. 95 | 96 | Gets the preview function from `consult-web-sources-alist'." 97 | (let* ((source (and (stringp cand) (get-text-property 0 :source cand)))) 98 | (funcall (plist-get (cdr (assoc source consult-web-sources-alist)) :on-preview) cand)) 99 | ) 100 | 101 | ;;; Define Embark Keymaps 102 | 103 | (defvar-keymap consult-web-embark-general-actions-map 104 | :doc "Keymap for consult-web-embark" 105 | :parent embark-general-map 106 | "i t" #'consult-web-embark-insert-title 107 | "i u" #'consult-web-embark-insert-url-link 108 | "w t" #'consult-web-embark-copy-title-as-kill 109 | "w u" #'consult-web-embark-copy-url-as-kill 110 | "o o" #'consult-web-embark-external-browse-link 111 | "o O" #'consult-web-embark-alternate-browse-link 112 | "o s" #'consult-web-embark-external-browse-search-link 113 | "o p" #'consult-web-embark-show-preview 114 | ) 115 | 116 | 117 | (add-to-list 'embark-keymap-alist '(consult-web . consult-web-embark-general-actions-map)) 118 | 119 | (defun consult-web-embark-scholar-external-browse-doi (cand) 120 | "Open the DOI url in external browser" 121 | (if-let* ((doi (and (stringp cand) (get-text-property 0 :doi cand)))) 122 | (funcall #'browse-url (concat "https://doi.org/" doi)))) 123 | 124 | (defun consult-web-embark-scholar-copy-authors-as-kill (cand) 125 | "Copy the authors of the candidate to `kill-ring'." 126 | (if-let ((authors (and (stringp cand) (get-text-property 0 :authors cand)))) 127 | (kill-new (string-trim (format " %s " authors))) 128 | )) 129 | 130 | (defun consult-web-embark-scholar-insert-authors (cand) 131 | "Insrt the authors of the candidate at point." 132 | (if-let ((authors (and (stringp cand) (get-text-property 0 :authors cand)))) 133 | (insert (string-trim (mapconcat #'identity authors ", "))) 134 | )) 135 | 136 | (defvar-keymap consult-web-embark-scholar-actions-map 137 | :doc "Keymap for consult-web-embark" 138 | :parent consult-web-embark-general-actions-map 139 | "o d" #'consult-web-embark-scholar-external-browse-doi 140 | "w a" #'consult-web-embark-scholar-copy-authors-as-kill 141 | "i a" #'consult-web-embark-scholar-insert-authors 142 | ) 143 | 144 | (add-to-list 'embark-keymap-alist '(consult-web-scholar . consult-web-embark-scholar-actions-map)) 145 | 146 | (add-to-list 'embark-default-action-overrides '(consult-web-scholar . consult-web-embark-default-action)) 147 | 148 | ;;; Provide `consul-web-embark' module 149 | 150 | (provide 'consult-web-embark) 151 | 152 | ;;; consult-web-embark.el ends here 153 | -------------------------------------------------------------------------------- /sources/consult-web-bing.el: -------------------------------------------------------------------------------- 1 | ;;; consult-web-bing.el --- Consulting Bing -*- lexical-binding: t -*- 2 | 3 | ;; Copyright (C) 2024 Armin Darvish 4 | 5 | ;; Author: Armin Darvish 6 | ;; Maintainer: Armin Darvish 7 | ;; Created: 2024 8 | ;; Version: 0.1 9 | ;; Package-Requires: ((emacs "28.1") (consult "1.1")) 10 | ;; Homepage: https://github.com/armindarvish/consult-web 11 | ;; Keywords: convenience 12 | 13 | ;;; Commentary: 14 | 15 | ;;; Code: 16 | (require 'consult-web) 17 | 18 | (defvar consult-web-bing-search-api-url "https://api.bing.microsoft.com/v7.0/search") 19 | 20 | (defcustom consult-web-bing-search-api-key nil 21 | "Key for Bing (Microsoft Azure) search API 22 | 23 | See URL `https://www.microsoft.com/en-us/bing/apis/bing-web-search-api' and URL `https://learn.microsoft.com/en-us/bing/search-apis/bing-web-search/search-the-web' for details" 24 | :group 'consult-web 25 | :type '(choice (const :tag "API Key" string) 26 | (function :tag "Custom Function"))) 27 | 28 | 29 | (cl-defun consult-web--bing-fetch-results (input &rest args &key count page &allow-other-keys) 30 | "Fetches search results for INPUT from Bing web search api. 31 | 32 | COUNT is passed as count in query parameters. 33 | (* PAGE COUNT) is passed as offset in query paramters. 34 | 35 | Refer to URL `https://programmablesearchengine.google.com/about/' and `https://developers.google.com/custom-search/' for more info. 36 | " 37 | (let* ((count (or (and (integerp count) count) 38 | (and count (string-to-number (format "%s" count))) 39 | consult-web-default-count)) 40 | (page (or (and (integerp page) page) 41 | (and page (string-to-number (format "%s" page))) 42 | consult-web-default-count)) 43 | (count (max count 1)) 44 | (page (* page count)) 45 | (params `(("q" . ,(replace-regexp-in-string " " "+" input)) 46 | ("count" . ,(format "%s" count)) 47 | ("offset" . ,(format "%s" page)))) 48 | (headers `(("Ocp-Apim-Subscription-Key" . ,(consult-web-expand-variable-function consult-web-bing-search-api-key))))) 49 | (funcall consult-web-retrieve-backend 50 | consult-web-bing-search-api-url 51 | :params params 52 | :headers headers 53 | :parser 54 | (lambda () 55 | (goto-char (point-min)) 56 | (let* ((results (json-parse-buffer)) 57 | (webpages (gethash "webPages" results)) 58 | (search-url (gethash "webSearchUrl" webpages)) 59 | (items (gethash "value" webpages))) 60 | (cl-loop for a across items 61 | collect 62 | (let ((table (make-hash-table :test 'equal)) 63 | (title (gethash "name" a)) 64 | (url (gethash "url" a)) 65 | (snippet (gethash "snippet" a))) 66 | (puthash :url url 67 | table) 68 | (puthash :search-url search-url 69 | table) 70 | (puthash :title title 71 | table) 72 | (puthash :source "Bing" 73 | table) 74 | (puthash :query input 75 | table) 76 | (puthash :snippet snippet 77 | table) 78 | table 79 | ) 80 | )) 81 | )))) 82 | 83 | (consult-web-define-source "Bing" 84 | :narrow-char ?m 85 | :face 'consult-web-engine-source-face 86 | :request #'consult-web--bing-fetch-results 87 | :preview-key consult-web-preview-key 88 | :search-history 'consult-web--search-history 89 | :selection-history 'consult-web--selection-history 90 | :dynamic 'both 91 | ) 92 | 93 | ;;; provide `consult-web-bing' module 94 | 95 | (provide 'consult-web-bing) 96 | 97 | (add-to-list 'consult-web-sources-modules-to-load 'consult-web-bing) 98 | ;;; consult-web-bing.el ends here 99 | -------------------------------------------------------------------------------- /sources/consult-web-brave-autosuggest.el: -------------------------------------------------------------------------------- 1 | ;;; consult-web-brave-autosuggest.el --- Consulting Brave Autosuggest -*- lexical-binding: t -*- 2 | 3 | ;; Copyright (C) 2024 Armin Darvish 4 | 5 | ;; Author: Armin Darvish 6 | ;; Maintainer: Armin Darvish 7 | ;; Created: 2024 8 | ;; Version: 0.1 9 | ;; Package-Requires: ((emacs "28.1") (consult "1.1")) 10 | ;; Homepage: https://github.com/armindarvish/consult-web 11 | ;; Keywords: convenience 12 | 13 | ;;; Commentary: 14 | 15 | ;;; Code: 16 | 17 | (require 'consult-web) 18 | 19 | (defvar consult-web-brave-autosuggest-api-url "https://api.search.brave.com/res/v1/suggest/search") 20 | 21 | 22 | (defcustom consult-web-brave-autosuggest-api-key nil 23 | "Key for Brave Autosuggest API. 24 | 25 | See URL `https://brave.com/search/api/' for more info" 26 | :group 'consult-web 27 | :type '(choice (const :tag "Brave Autosuggest API Key" string) 28 | (function :tag "Custom Function"))) 29 | 30 | (cl-defun consult-web--brave-autosuggest-fetch-results (input &rest args &key count page &allow-other-keys) 31 | "Fetch search results for INPUT from `consult-web-brave-autosuggest-api-url'. 32 | " 33 | (let* ((count (or (and (integerp count) count) 34 | (and count (string-to-number (format "%s" count))) 35 | consult-web-default-count)) 36 | (page (or (and (integerp page) page) 37 | (and page (string-to-number (format "%s" page))) 38 | consult-web-default-page)) 39 | (params `(("q" . ,input) 40 | ("count" . ,(format "%s" count)) 41 | ("page" . ,(format "%s" page)) 42 | ("country" . "US"))) 43 | (headers `(("User-Agent" . "Emacs:consult-web/0.1 (Emacs consult-web package; https://github.com/armindarvish/consult-web)") 44 | ("Accept" . "application/json") 45 | ("Accept-Encoding" . "gzip") 46 | ("X-Subscription-Token" . ,(consult-web-expand-variable-function consult-web-brave-autosuggest-api-key)) 47 | ))) 48 | (funcall consult-web-retrieve-backend 49 | consult-web-brave-autosuggest-api-url 50 | :params params 51 | :headers headers 52 | :parser 53 | (lambda () 54 | (goto-char (point-min)) 55 | (buffer-substring (point-min) (point-max)) 56 | (let* ((content (json-parse-buffer)) 57 | (original (make-hash-table :test 'equal)) 58 | (_ (puthash "query" (gethash "original" (gethash "query" content)) original)) 59 | (suggestions (gethash "results" content))) 60 | (cl-loop for a across (vconcat suggestions (vector original)) 61 | collect 62 | (let ((table (make-hash-table :test 'equal)) 63 | (word (gethash "query" a))) 64 | (puthash :url 65 | (concat "https://search.brave.com/search?q=" (url-hexify-string word)) table) 66 | (puthash :search-url nil 67 | table) 68 | (puthash :title 69 | word table) 70 | (puthash :source 71 | "Brave AutoSuggest" table) 72 | (puthash :query input 73 | table) 74 | table 75 | )) 76 | 77 | ) 78 | )))) 79 | 80 | (consult-web-define-source "Brave AutoSuggest" 81 | :narrow-char ?B 82 | :face 'consult-web-engine-source-face 83 | :request #'consult-web--brave-autosuggest-fetch-results 84 | :on-preview #'ignore 85 | :on-return #'identity 86 | :on-callback #'string-trim 87 | :search-history 'consult-web--search-history 88 | :selection-history t 89 | :dynamic t 90 | ) 91 | 92 | ;;; provide `consult-web-brave-autosuggest' module 93 | 94 | (provide 'consult-web-brave-autosuggest) 95 | 96 | (add-to-list 'consult-web-sources-modules-to-load 'consult-web-brave-autosuggest) 97 | ;;; consult-web-brave-autosuggest.el ends here 98 | -------------------------------------------------------------------------------- /sources/consult-web-brave.el: -------------------------------------------------------------------------------- 1 | ;;; consult-web-brave.el --- Consulting Brave -*- lexical-binding: t -*- 2 | 3 | ;; Copyright (C) 2024 Armin Darvish 4 | 5 | ;; Author: Armin Darvish 6 | ;; Maintainer: Armin Darvish 7 | ;; Created: 2024 8 | ;; Version: 0.1 9 | ;; Package-Requires: ((emacs "28.1") (consult "1.1")) 10 | ;; Homepage: https://github.com/armindarvish/consult-web 11 | ;; Keywords: convenience 12 | 13 | ;;; Commentary: 14 | 15 | ;;; Code: 16 | 17 | (require 'consult-web) 18 | 19 | (defvar consult-web-brave-search-url "https://search.brave.com/search") 20 | 21 | (defvar consult-web-brave-url "https://api.search.brave.com/res/v1/web/search") 22 | 23 | (defcustom consult-web-brave-api-key nil 24 | "Key for Brave API. 25 | 26 | See URL `https://brave.com/search/api/' for more info" 27 | :group 'consult-web 28 | :type '(choice (const :tag "Brave API Key" string) 29 | (function :tag "Custom Function"))) 30 | 31 | 32 | (cl-defun consult-web--brave-fetch-results (input &rest args &key count page &allow-other-keys) 33 | "Retrieve search results from Brave for INPUT. 34 | 35 | COUNT is passed as count in query parameters. 36 | PAGE is passed as page in query paramters. 37 | " 38 | (let* ((count (or (and (integerp count) count) 39 | (and count (string-to-number (format "%s" count))) 40 | consult-web-default-count)) 41 | (page (or (and (integerp page) page) 42 | (and page (string-to-number (format "%s" page))) 43 | consult-web-default-count)) 44 | (count (min count 20)) 45 | (params `(("q" . ,(url-hexify-string input)) 46 | ("count" . ,(format "%s" count)) 47 | ("page" . ,(format "%s" page)))) 48 | (headers `(("User-Agent" . "Emacs:consult-web/0.1 (Emacs consult-web package; https://github.com/armindarvish/consult-web)") 49 | ("Accept" . "application/json") 50 | ("Accept-Encoding" . "gzip") 51 | ("X-Subscription-Token" . ,(consult-web-expand-variable-function consult-web-brave-api-key)) 52 | ))) 53 | (funcall consult-web-retrieve-backend 54 | consult-web-brave-url 55 | :params params 56 | :headers headers 57 | :parser 58 | (lambda () 59 | (goto-char (point-min)) 60 | (let* ((results (gethash "results" (gethash "web" (json-parse-buffer)))) 61 | (items (mapcar (lambda (item) `(:url ,(format "%s" (gethash "url" item)) :title ,(format "%s" (gethash "title" item)))) results)) 62 | ) 63 | (cl-loop for a in items 64 | collect 65 | (let ((table (make-hash-table :test 'equal))) 66 | (puthash :url 67 | (plist-get a :url) table) 68 | (puthash :search-url (consult-web--make-url-string consult-web-brave-search-url params) table) 69 | (puthash :title 70 | (plist-get a :title) table) 71 | (puthash :source "Brave" 72 | table) 73 | (puthash :query input 74 | table) 75 | table 76 | )))) 77 | 78 | ))) 79 | 80 | (consult-web-define-source "Brave" 81 | :narrow-char ?b 82 | :face 'consult-web-engine-source-face 83 | :request #'consult-web--brave-fetch-results 84 | :preview-key consult-web-preview-key 85 | :search-history 'consult-web--search-history 86 | :selection-history 'consult-web--selection-history 87 | :dynamic 'both 88 | ) 89 | 90 | ;;; provide `consult-web-brave' module 91 | 92 | (provide 'consult-web-brave) 93 | 94 | (add-to-list 'consult-web-sources-modules-to-load 'consult-web-brave) 95 | ;;; consult-web-brave.el ends here 96 | -------------------------------------------------------------------------------- /sources/consult-web-buffer.el: -------------------------------------------------------------------------------- 1 | ;;; consult-web-buffer.el --- Consulting Buffers -*- lexical-binding: t -*- 2 | 3 | ;; Copyright (C) 2024 Armin Darvish 4 | 5 | ;; Author: Armin Darvish 6 | ;; Maintainer: Armin Darvish 7 | ;; Created: 2024 8 | ;; Version: 0.1 9 | ;; Package-Requires: ((emacs "28.1") (consult "1.1")) 10 | ;; Homepage: https://github.com/armindarvish/consult-web 11 | ;; Keywords: convenience 12 | 13 | ;;; Commentary: 14 | 15 | ;;; Code: 16 | 17 | (require 'consult-web) 18 | 19 | (defun consult-web--consult-buffer-preview (cand) 20 | "Preview function for `consult-web--buffer'." 21 | (if cand 22 | (let* ((title (get-text-property 0 :title cand))) 23 | (when-let ((buff (get-buffer title))) 24 | (consult--buffer-action buff)) 25 | ))) 26 | 27 | (cl-loop for source in consult-buffer-sources 28 | do (if (symbolp source) (consult-web--make-source-from-consult-source source 29 | :category 'consult-web 30 | :on-preview #'consult-web--consult-buffer-preview 31 | :on-return #'identity 32 | :on-callback #'consult--buffer-action 33 | :search-history 'consult-web--search-history 34 | :selection-history 'consult-web--selection-history 35 | :dynamic 'both 36 | :preview-key 'consult-preview-key 37 | ))) 38 | 39 | ;;; provide `consult-web-buffer' module 40 | 41 | (provide 'consult-web-buffer) 42 | 43 | (add-to-list 'consult-web-sources-modules-to-load 'consult-web-buffer) 44 | ;;; consult-web-buffer.el ends here 45 | -------------------------------------------------------------------------------- /sources/consult-web-chatgpt.el: -------------------------------------------------------------------------------- 1 | ;;; consult-web-chatgpt.el --- Consulting chatGPT -*- lexical-binding: t -*- 2 | 3 | ;; Copyright (C) 2024 Armin Darvish 4 | 5 | ;; Author: Armin Darvish 6 | ;; Maintainer: Armin Darvish 7 | ;; Created: 2024 8 | ;; Version: 0.1 9 | ;; Package-Requires: ((emacs "28.1") (consult "1.1")) 10 | ;; Homepage: https://github.com/armindarvish/consult-web 11 | ;; Keywords: convenience 12 | 13 | ;;; Commentary: 14 | 15 | ;;; Code: 16 | 17 | (require 'consult-web) 18 | 19 | (defun consult-web-dynamic--chatgpt-format-candidate (table &optional face &rest args) 20 | "Returns a formatted string for candidates of `consult-web-chatgpt'. 21 | 22 | TABLE is a hashtable from `consult-web--chatgpt-fetch-results'." 23 | (let* ((pl (consult-web-hashtable-to-plist table)) 24 | (title (format "%s" (gethash :title table))) 25 | (source (gethash :source table)) 26 | (source (if (stringp source) (propertize source 'face 'consult-web-source-face))) 27 | (query (gethash :query table)) 28 | (model (gethash :model table)) 29 | (match-str (if (stringp query) (consult--split-escaped (car (consult--command-split query))) nil)) 30 | (title-str (consult-web--set-string-width title (floor (* (frame-width) 0.4)))) 31 | (title-str (propertize title-str 'face (or face 'consult-web-ai-source-face))) 32 | (extra-args (consult-web-hashtable-to-plist table '(:title :url :search-url :query :source :model))) 33 | (str (concat title-str (if model (propertize (format "\tmodel: %s" model) 'face 'consult-web-path-face)) (if source (concat "\t" source)) (if extra-args (format "\t%s" extra-args)))) 34 | (str (apply #'propertize str pl)) 35 | ) 36 | (if consult-web-highlight-matches 37 | (cond 38 | ((listp match-str) 39 | (mapcar (lambda (match) (setq str (consult-web--highlight-match match str t))) match-str)) 40 | ((stringp match-str) 41 | (setq str (consult-web--highlight-match match-str str t))))) 42 | str)) 43 | 44 | (defvar consult-web-chatgpt-api-url "https://api.openai.com/v1/chat/completions") 45 | 46 | (defcustom consult-web-openai-api-key nil 47 | "Key for OpeAI API 48 | 49 | See URL `https://openai.com/product' and URL `https://platform.openai.com/docs/introduction' for details" 50 | :group 'consult-web 51 | :type '(choice (const :tag "API Key" string) 52 | (function :tag "Custom Function"))) 53 | 54 | 55 | (cl-defun consult-web--chatgpt-fetch-results (input &rest args &key model &allow-other-keys) 56 | "Fetches chat response for INPUT from chatGPT." 57 | (let* ((model (or model gptel-model)) 58 | (headers `(("Content-Type" . "application/json") 59 | ("Authorization" . ,(concat "Bearer " (consult-web-expand-variable-function consult-web-openai-api-key)))))) 60 | (funcall consult-web-retrieve-backend 61 | consult-web-chatgpt-api-url 62 | :type "POST" 63 | :headers headers 64 | :data (json-encode `((model . ,model) 65 | (messages . [((role . "user") 66 | (content . ,input))]))) 67 | :parser 68 | (lambda () 69 | (goto-char (point-min)) 70 | (let* ((table (make-hash-table :test 'equal)) 71 | (response (json-parse-buffer)) 72 | (title (gethash "content" (gethash "message" (aref (gethash "choices" response) 0))))) 73 | (puthash :url nil 74 | table) 75 | (puthash :title title 76 | table) 77 | (puthash :source "chatGPT" 78 | table) 79 | (puthash :model model 80 | table) 81 | (puthash :query input 82 | table) 83 | (list table))) 84 | ))) 85 | 86 | (defun consult-web--chatgpt-response-preview (response &optional query) 87 | "Returns a buffer with formatted RESPONSE from chatGPT" 88 | (save-excursion 89 | (let ((buff (get-buffer-create "*consult-web-chatgpt-response*"))) 90 | (with-current-buffer buff 91 | (erase-buffer) 92 | (if query (insert (format "# User:\n\n %s\n\n" query))) 93 | (if response (insert (format "# chatGPT:\n\n %s\n\n" response))) 94 | (if (featurep 'mardown-mode) 95 | (require 'markdown-mode) 96 | (markdown-mode) 97 | ) 98 | (point-marker)) 99 | ))) 100 | 101 | 102 | (defun consult-web--chatgpt-preview (cand) 103 | "Shows a preview buffer with chatGPT response from CAND" 104 | (when-let ((buff (get-buffer "*consult-web-chatgpt-response*"))) 105 | (kill-buffer buff)) 106 | 107 | (when-let* ((query (cond ((listp cand) 108 | (get-text-property 0 :query (car cand))) 109 | (t 110 | (get-text-property 0 :query cand)))) 111 | (response (cond ((listp cand) 112 | (or (get-text-property 0 :title (car cand)) (car cand))) 113 | (t 114 | (or (get-text-property 0 :title cand) cand)))) 115 | (marker (consult-web--chatgpt-response-preview response query))) 116 | (consult--jump marker) 117 | )) 118 | 119 | 120 | (consult-web-define-source "chatGPT" 121 | :narrow-char ?G 122 | :face 'consult-web-ai-source-face 123 | :request #'consult-web--chatgpt-fetch-results 124 | :format #'consult-web-dynamic--chatgpt-format-candidate 125 | :on-preview #'consult-web--chatgpt-preview 126 | :on-return #'identity 127 | :on-callback #'consult-web--chatgpt-preview 128 | :preview-key consult-web-preview-key 129 | :search-history 'consult-web--search-history 130 | :selection-history 'consult-web--selection-history 131 | :dynamic 'both 132 | ) 133 | 134 | ;;; provide `consult-web-chatgpt' module 135 | 136 | (provide 'consult-web-chatgpt) 137 | 138 | (add-to-list 'consult-web-sources-modules-to-load 'consult-web-chatgpt) 139 | ;;; consult-web-chatgpt.el ends here 140 | -------------------------------------------------------------------------------- /sources/consult-web-doi.el: -------------------------------------------------------------------------------- 1 | ;;; consult-web-doi.el --- Consulting DOI.org -*- lexical-binding: t -*- 2 | 3 | ;; Copyright (C) 2024 Armin Darvish 4 | 5 | ;; Author: Armin Darvish 6 | ;; Maintainer: Armin Darvish 7 | ;; Created: 2024 8 | ;; Version: 0.1 9 | ;; Package-Requires: ((emacs "28.1") (consult "1.1")) 10 | ;; Homepage: https://github.com/armindarvish/consult-web 11 | ;; Keywords: convenience 12 | 13 | ;;; Commentary: 14 | 15 | ;;; Code: 16 | 17 | (require 'consult-web) 18 | 19 | (defvar consult-web-doiorg-api-url "https://doi.org/api/handles/") 20 | 21 | (defvar consult-web-doiorg-search-url "https://doi.org/") 22 | 23 | (defun consult-web--doi-to-url (doi) 24 | "Converts DOI value to target url" 25 | (let* ((doi (if doi (format "%s" doi))) 26 | (url (concat consult-web-doiorg-api-url doi))) 27 | (funcall consult-web-retrieve-backend 28 | url 29 | :parser 30 | (lambda () 31 | (goto-char (point-min)) 32 | (let* ((content (json-parse-buffer)) 33 | (items (gethash "values" content))) 34 | (car (mapcar (lambda (item) 35 | (if-let* ((type (gethash "type" item)) 36 | (url (if (equal type "URL") (gethash "value" (gethash "data" item))))) 37 | url 38 | nil)) items))))))) 39 | 40 | 41 | (cl-defun consult-web--doiorg-fetch-results (doi &rest args) 42 | "Fetch target url of DOI. 43 | " 44 | (let* ((table (make-hash-table :test 'equal)) 45 | (url (consult-web--doi-to-url doi))) 46 | (if url 47 | (progn 48 | (puthash :url url 49 | table) 50 | (puthash :title doi 51 | table) 52 | (puthash :source "doiorg" 53 | table) 54 | (puthash :query doi 55 | table) 56 | )) 57 | (list table))) 58 | 59 | (defvar consult-web--doi-search-history (list) 60 | "History variables for search terms when using 61 | `consult-web-doi' commands.") 62 | 63 | (defvar consult-web--doi-selection-history (list) 64 | "History variables for selected items when using 65 | `consult-web-doi' commands.") 66 | 67 | 68 | (consult-web-define-source "doiorg" 69 | :narrow-char ?d 70 | :face 'link 71 | :request #'consult-web--doiorg-fetch-results 72 | :search-history 'consult-web--doi-search-history 73 | :selection-history 'consult-web--doi-selection-history 74 | :dynamic t 75 | ) 76 | 77 | ;;; provide `consult-web-doi' module 78 | 79 | (provide 'consult-web-doi) 80 | 81 | (add-to-list 'consult-web-sources-modules-to-load 'consult-web-doi) 82 | ;;; consult-web-doi.el ends here 83 | -------------------------------------------------------------------------------- /sources/consult-web-duckduckgo.el: -------------------------------------------------------------------------------- 1 | ;;; consult-web-duckduckgo.el --- Consulting DuckDuckGo -*- lexical-binding: t -*- 2 | 3 | ;; Copyright (C) 2024 Armin Darvish 4 | 5 | ;; Author: Armin Darvish 6 | ;; Maintainer: Armin Darvish 7 | ;; Created: 2024 8 | ;; Version: 0.1 9 | ;; Package-Requires: ((emacs "28.1") (consult "1.1")) 10 | ;; Homepage: https://github.com/armindarvish/consult-web 11 | ;; Keywords: convenience 12 | 13 | ;;; Commentary: 14 | 15 | ;;; Code: 16 | 17 | (require 'consult-web) 18 | 19 | (defvar consult-web-duckduckgoapi-url "http://api.duckduckgoapi.com/") 20 | 21 | (cl-defun consult-web--duckduckgoapi-fetch-results (input &rest args &key count page &allow-other-keys) 22 | "Fetch search results got INPUT from DuckDuckGo limited API." 23 | (let* ((count (or (and (integerp count) count) 24 | (and count (string-to-number (format "%s" count))) 25 | consult-web-default-count)) 26 | (page (or (and (integerp page) page) 27 | (and page (string-to-number (format "%s" page))) 28 | consult-web-default-count)) 29 | (count (min count 10)) 30 | (page (+ (* page count) 1)) 31 | (params `(("q" . ,input) 32 | ("format" . "json"))) 33 | (headers `(("Accept" . "application/json")))) 34 | (funcall consult-web-retrieve-backend 35 | consult-web-duckduckgoapi-url 36 | :params params 37 | :headers headers 38 | :parser (lambda () 39 | (goto-char (point-min)) 40 | (let* ((results (gethash "RelatedTopics" (json-parse-buffer))) 41 | (items (mapcar (lambda (item) `(:url ,(format "%s" (gethash "FirstURL" item)) 42 | :title ,(format "%s" (gethash "Result" item)))) 43 | results))) 44 | (cl-loop for a in items 45 | collect 46 | (let ((table (make-hash-table :test 'equal))) 47 | (puthash :url 48 | (plist-get a :url) table) 49 | (puthash :title (if (string-match "\\(?1:.*\\) (string-width snippet) 25)) (concat (substring snippet 0 22) "...") snippet)) 40 | (match-str (if (stringp query) (consult--split-escaped (car (consult--command-split query))) nil)) 41 | (title-str (consult-web--set-string-width title (floor (* (frame-width) 0.4)))) 42 | (title-str (propertize title-str 'face (or face 'consult-web-default-face))) 43 | (extra-args (consult-web-hashtable-to-plist table '(:title :url :search-url :query :source :snippet :channeltitle :videoid))) 44 | (str (concat title-str 45 | (if domain (format "\t%s" domain)) 46 | (if channeltitle (format " - %s" channeltitle)) 47 | (if snippet (format "\s\s%s" snippet)) 48 | (if source (concat "\t" source)) 49 | (if extra-args (propertize (format "\s%s" extra-args) 'face 'consult-web-source-face)))) 50 | (str (apply #'propertize str pl)) 51 | ) 52 | (if consult-web-highlight-matches 53 | (cond 54 | ((listp match-str) 55 | (mapcar (lambda (match) (setq str (consult-web--highlight-match match str t))) match-str)) 56 | ((stringp match-str) 57 | (setq str (consult-web--highlight-match match-str str t))))) 58 | str)) 59 | 60 | (defvar consult-web-youtube-watch-url "https://www.youtube.com/watch") 61 | 62 | (defvar consult-web-youtube-channel-url "https://www.youtube.com/channel/") 63 | 64 | (defvar consult-web-youtube-search-results-url "https://www.youtube.com/results") 65 | 66 | (defvar consult-web-youtube-search-api-url "https://www.googleapis.com/youtube/v3/search") 67 | 68 | (defcustom consult-web-youtube-search-key nil 69 | "Key for YouTube custom search API 70 | 71 | See URL `https://developers.google.com/youtube/v3/getting-started' 72 | for details" 73 | :group 'consult-web 74 | :type '(choice (const :tag "API Key" string) 75 | (function :tag "Custom Function"))) 76 | 77 | 78 | (cl-defun consult-web--youtube-fetch-results (input &rest args &key count page order def type vidtype &allow-other-keys) 79 | "Fetches search results for INPUT from “Google custom search” service. 80 | 81 | COUNT is passed as num in query parameters. 82 | (* PAGE COUNT) is passed as start in query paramters. 83 | " 84 | (let* ((count (or (and (integerp count) count) 85 | (and count (string-to-number (format "%s" count))) 86 | consult-web-default-count)) 87 | (page (or (and (integerp page) page) 88 | (and page (string-to-number (format "%s" page))) 89 | consult-web-default-count)) 90 | (def (if (and def (member (format "%s" def) '("any" "standard" "high"))) (format "%s" def) "any")) 91 | (type (if (and type (member (format "%s" type) '("channel" "playlist" "video"))) (format "%s" type) "video")) 92 | (vidtype (if (and type (member (format "%s" vidtype) '("any" "episode" "movie"))) (format "%s" vidtype) "any")) 93 | (count (min count 10)) 94 | (page (+ (* page count) 1)) 95 | (order (if (and type (member (format "%s" order) '("date" "rating" "relevance" "title" "videoCount" "viewCount"))) (format "%s" vidtype) "relevance")) 96 | (params `(("q" . ,input) 97 | ("part" . "snippet") 98 | ("order" . ,order) 99 | ("type" . ,type) 100 | ("maxResults" . ,(format "%s" count)) 101 | ("videoDefinition" . ,def) 102 | ("videoType" . ,vidtype) 103 | ("key" . ,(consult-web-expand-variable-function consult-web-youtube-search-key)))) 104 | (headers `(("Accept" . "application/json") 105 | ("Accept-Encoding" . "gzip") 106 | ("User-Agent" . "consult-web (gzip)") 107 | ("X-Goog-Api-Key" . ,(consult-web-expand-variable-function consult-web-youtube-search-key ))))) 108 | (funcall consult-web-retrieve-backend 109 | consult-web-youtube-search-api-url 110 | :params params 111 | :headers headers 112 | :parser 113 | (lambda () 114 | (goto-char (point-min)) 115 | (let* ((results (json-parse-buffer)) 116 | (items (gethash "items" results))) 117 | (cl-loop for a across items 118 | collect 119 | (let* ((table (make-hash-table :test 'equal)) 120 | (videoid (gethash "videoId" (gethash "id" a))) 121 | (snippet (gethash "snippet" a)) 122 | (channeltitle (gethash "channelTitle" snippet)) 123 | (channelid (gethash "channelId" snippet)) 124 | (title (gethash "title" snippet)) 125 | (date (gethash "publishedAt" snippet)) 126 | (date (format-time-string "%Y-%m-%d %R" (date-to-time date))) 127 | (url (cond 128 | (videoid (consult-web--make-url-string consult-web-youtube-watch-url `(("v" . ,videoid)))) 129 | (channelid (concat consult-web-youtube-channel-url channelid)))) 130 | (search-url (consult-web--make-url-string consult-web-youtube-search-results-url `(("search_query" . ,input)))) 131 | (description (gethash "description" snippet))) 132 | (puthash :url 133 | url table) 134 | (puthash :search-url search-url 135 | table) 136 | (puthash :title title 137 | table) 138 | (puthash :videoid videoid 139 | table) 140 | (puthash :channeltitle channeltitle 141 | table) 142 | (puthash :channelid channelid 143 | table) 144 | (puthash :source "YouTube" 145 | table) 146 | (puthash :query input 147 | table) 148 | (puthash :snippet description table) 149 | table 150 | ) 151 | )) 152 | )))) 153 | 154 | (consult-web-define-source "YouTube" 155 | :narrow-char ?y 156 | :face 'consult-web-engine-source-face 157 | :request #'consult-web--youtube-fetch-results 158 | :format #'consult-web-dynamic--youtube-format-candidate 159 | :preview-key consult-web-preview-key 160 | :search-history 'consult-web--search-history 161 | :selection-history 'consult-web--selection-history 162 | :dynamic 'both 163 | ) 164 | 165 | ;;; provide `consult-web-youtube' module 166 | 167 | (provide 'consult-web-youtube) 168 | 169 | (add-to-list 'consult-web-sources-modules-to-load 'consult-web-youtube) 170 | ;;; consult-web-youtube.el ends here 171 | --------------------------------------------------------------------------------