10 |
11 | ## Book Overview
12 | [_AI-Powered Search_](https://aipoweredsearch.com) teaches you the latest machine learning techniques to build search engines that continuously learn from your users and your content to drive more domain-aware and intelligent search.
13 |
14 | Search engine technology is rapidly evolving, with Artificial Intelligence (AI) driving much of that innovation. Crowdsourced relevance and the integration of large language models (LLMs) like GPT and other foundation models are massively accelerating the capabilities and expectations of search technology.
15 |
16 | AI-Powered Search will teach you modern, data-science-driven search techniques like:
17 | - Semantic search using dense vector embeddings from foundation models
18 | - Retrieval Augmented Generation
19 | - Question answering and summarization combining search and LLMs
20 | - Fine-tuning transformer-based LLMs
21 | - Personalized search based on user signals and vector embeddings
22 | - Collecting user behavioral signals and building signals boosting models
23 | - Semantic knowledge graphs for domain-specific learning
24 | - Implementing machine-learned ranking models (learning to rank)
25 | - Building click models to automate machine-learned ranking
26 | - Generative search, hybrid search, and the search frontier
27 |
28 | Today’s search engines are expected to be smart, understanding the nuances of natural language queries, as well as each user’s preferences and context. This book empowers you to build search engines that take advantage of user interactions and the hidden semantic relationships in your content to automatically deliver better, more relevant search experiences.
29 |
30 | ## How to run
31 | For simplicity of setup, all code is shipped in Jupyter Notebooks and packaged in Docker containers. This means that installing Docker and then pulling (or building) and running the book's Docker containers is the only necessary setup. [Appendix A](https://livebook.manning.com/book/ai-powered-search/appendix-a?origin=code-base) of the book provides full step-by-step instructions for running the code examples, but you can run the following to get up and running quickly:
32 |
33 | If you haven't already pulled the source code locally, run:
34 | ```
35 | git clone https://github.com/treygrainger/ai-powered-search.git
36 | ```
37 |
38 | Then, to build and start the codebase with interactive Jupyter notebooks, run:
39 | ```
40 | cd ai-powered-search
41 | docker compose up
42 | ```
43 |
44 | That's all it takes! Once the containers are built and running (this may take a while, especially on the first build), visit:
45 | `http://localhost:8888` to launch the [Welcome](chapters/welcome.ipynb) notebook and see a Table of Contents for all the live code examples from throughout the book.
46 |
47 | 
49 |
50 | ## Supported Technologies
51 | AI-Powered Search teaches many modern search techniques leveraging machine learning approaches. While we utilize specific technologies to demonstrate concepts, most techniques are applicable to many modern search engines and vector databases.
52 |
53 | Throughout the book, all code examples are in **Python**, with **PySpark** (the Python interface to **Apache Spark**) being utilized heavily for data processing tasks. The default search engine leveraged by the book's examples is **Apache Solr**, but most examples are abstracted away from the particular search engine, and swappable implementation will be soon available for most popular search engines and vector databases. For more information about the search engine abstractions and custom integrations check out the [engine documentation](engines/README.md).
54 |
55 | **See Full List**: [Supported Search Engines and Vector Databases](engines/README.md)
56 |
57 | [ *Note*: if you work for a search engine / vector database company, project, or hosting provider and want to work with us on getting your engine supported, please reach out to trey@searchkernel.com ]
58 |
59 | ## Questions and help
60 | Your purchase of _AI-Powered Search_ includes online access to Manning's [LiveBook forum](https://livebook.manning.com/forum?product=graingert). This allows you to provide comments and ask questions about any parts of the book. Additionally, feel free to submit pull requests, Github issues, or comments on the project's official Github repo at https://github.com/treygrainger/ai-powered-search.
61 |
62 | ## License
63 | All code in this repository is open source under the [Apache License, Version 2.0 (ASL 2.0)](https://www.apache.org/licenses/LICENSE-2.0), unless otherwise specified.
64 |
65 | Note that when executing the code, it may pull additional dependencies that follow alternate licenses, so please be sure to inspect those licenses before using them in your projects to ensure they are suitable. The code may also pull in datasets subject to various licenses, some of which may be derived from AI models and some of which may be derived from web crawls of data subject to fair use under the copyright laws in the country of publication (the USA). Any such datasets are published "as-is", for the sole purpose of demonstrating the concepts in the book, and these datasets and their associated licenses may be subject to change over time.
66 |
67 | ## Grab a copy of the book
68 | If you don't yet have a copy, please support the authors and the publisher by purchasing a copy of [_AI-Powered Search_](http://aipoweredsearch.com). It will walk you step by step through the concepts and techniques shown in the code examples in this repository, providing needed context and insights to help you better understand the techniques.
69 |
--------------------------------------------------------------------------------
/aips/__init__.py:
--------------------------------------------------------------------------------
1 | import aips.environment as environment
2 | from engines.solr import SolrLTR, SolrSemanticKnowledgeGraph, SolrEntityExtractor, SolrSparseSemanticSearch
3 | from engines.solr.SolrEngine import SolrEngine
4 | from engines.solr.SolrCollection import SolrCollection
5 |
6 | from engines.opensearch.OpenSearchCollection import OpenSearchCollection
7 | from engines.opensearch.OpenSearchEngine import OpenSearchEngine
8 | from engines.opensearch.OpenSearchLTR import OpenSearchLTR
9 | from engines.opensearch.OpenSearchSparseSemanticSearch import OpenSearchSparseSemanticSearch
10 |
11 | import os
12 | from IPython.display import display, HTML
13 | import pandas
14 | import re
15 |
16 | engine_type_map = {"SOLR": SolrEngine(),
17 | "OPENSEARCH": OpenSearchEngine()}
18 |
19 | def get_engine(override=None):
20 | engine_name = override.upper() if override else environment.get("AIPS_SEARCH_ENGINE", "SOLR")
21 | return engine_type_map[engine_name]
22 |
23 | def set_engine(engine_name):
24 | engine_name = engine_name.upper()
25 | if engine_name not in engine_type_map:
26 | raise ValueError(f"No search engine implementation found for {engine_name}")
27 | else:
28 | environment.set("AIPS_SEARCH_ENGINE", engine_name)
29 |
30 | def get_ltr_engine(collection):
31 | ltr_engine_map = {SolrCollection: SolrLTR,
32 | OpenSearchCollection: OpenSearchLTR}
33 | return ltr_engine_map[type(collection)](collection)
34 |
35 | def get_semantic_knowledge_graph(collection):
36 | return SolrSemanticKnowledgeGraph(get_engine("solr").get_collection(collection.name))
37 |
38 | def get_entity_extractor(collection):
39 | return SolrEntityExtractor(get_engine("solr").get_collection(collection.name))
40 |
41 | def get_sparse_semantic_search():
42 | SSS_map = {SolrEngine: SolrSparseSemanticSearch,
43 | OpenSearchEngine: OpenSearchSparseSemanticSearch}
44 | return SSS_map[type(get_engine())]()
45 |
46 | def healthcheck():
47 | try:
48 | if get_engine().health_check():
49 | print("All Systems are ready. Happy Searching!")
50 | else:
51 | print("The search engine is not in a ready state.")
52 | except:
53 | print("Error! One or more containers are not responding.\nPlease follow the instructions in Appendix A.")
54 |
55 | def num2str(number):
56 | return str(round(number,4)) #round to 4 decimal places for readibility
57 |
58 | def vec2str(vector):
59 | return "[" + ", ".join(map(num2str,vector)) + "]"
60 |
61 | def tokenize(text):
62 | return text.replace(".","").replace(",","").lower().split()
63 |
64 | def get_executing_notebook_path():
65 | return globals().get("__vsc_ipynb_file__", #only exists during a remote vscode kernel
66 | globals().get("_dh", [None])[0])
67 |
68 | def images_directory():
69 | path = get_executing_notebook_path()
70 | if path:
71 | relative = os.path.relpath(os.environ.get("HOME"), path)
72 | else:
73 | relative = "../.."
74 | return f"{relative}/data/retrotech/images"
75 |
76 | def img_path_for_product(product):
77 | directory = images_directory()
78 | file = product.get("upc", "no-upc")
79 | if not os.path.exists(f"data/retrotech/images/{file}.jpg"):
80 | file = "unavailable"
81 | return f"{directory}/{file}.jpg"
82 |
83 | def remove_new_lines(data):
84 | return str(data).replace('\\n', '').replace('\\N', '')
85 |
86 | def as_html(data):
87 | return remove_new_lines(data).replace(", '", ", Name: ${NAME}
13 |
Manufacturer: ${MANUFACTURER}
Name: ${NAME} | City: ${CITY} | Rating: ${STARS}
10 | 11 |