114 |
119 |
120 | ### How you’ll know it worked:
121 | We get some `necklaces made of brass`!
122 |
123 | ### Where are the docs?
124 | https://www.elastic.co/guide/en/elasticsearch/reference/current/analyzer.html
125 |
--------------------------------------------------------------------------------
/lessons/Lesson4_ExtraCredit.md:
--------------------------------------------------------------------------------
1 | # Lesson 4: Extra Credit
2 |
3 | Enough of necklaces, let’s buy a jacket.
4 |
5 | There are plenty of jackets available in our data set, but only one shows up here.
6 |
7 | The problem is that the words `men’s jacket` don’t usually show up in the product name, but they do show up in the product description. Let’s search the descriptions instead of the names.
8 |
9 | Where are the docs?
10 |
11 | There’s nothing new here, but you may want to review some of the documentation above.
12 |
13 | ## Part 1: Indexing Jackets (Indexing Product Description)
14 |
15 | ### What you need to do:
16 | The indexing and search flow here is very similar to what you've seen before already. Just adapt them for product description!
17 |
18 | In `searchapp/index_products.py`:
19 |
20 | Hint
115 | Swap out the english analyzer in the mapping, for the `custom_english_analyzer`. 116 | 117 | As usual, don’t forget to re-run `index_products` after changing it. 118 |
21 |
24 |
25 | Hint: Step 1
22 | Add the product description to the `_source` of your document. 23 |
26 |
29 |
30 | Hint: Step 2
27 | Add `description` to the mapping, and configure it to use our custom analyzer. 28 |
31 |
34 |
35 | Hint: Step 3
32 | Change `search.py` to reference `description.english_analyzed` instead of name. 33 |
36 |
39 |
40 | ### How you’ll know it worked:
41 | There are actually Jackets returned for the `men’s jackets` query.
42 |
43 | This is a huge improvement for the jackets, but just about every other query has gotten worse. When there’s a match against the product name, the results are great, but we still need to support searches against the description. Fortunately, we can combine our name query and our description query, and search against both fields.
44 |
45 | ### Where are the docs?
46 | https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-dis-max-query.html
47 |
48 | ## Part 2: Searching for Jackets (Dis Max Queries)
49 |
50 | ### What you need to do:
51 | Hint: Step 4
37 | As usual, don’t forget to re-run `index_products` after changing it. 38 |
52 |
59 |
60 | ### How you’ll know it worked:
61 | All of the results should look a bit better.
62 |
63 | Play around with your query, and see how changes impact the results. Start off by adding `boost` properties to your name and description queries. Try swapping out the dis_max query for a bool query and playing with the boost values. Take your time, and try to really understand the impact each of your changes has. There’s always a way to make things a little better!
64 |
--------------------------------------------------------------------------------
/lessons/kibana/index.json:
--------------------------------------------------------------------------------
1 | # ------------------------------
2 | # 1. Basic Indexing: Create a document with POST
3 | # ------------------------------
4 |
5 | # We'll be adding products to a product catalog. Let's add our first JSON doc:
6 |
7 | POST /products_index_test/product
8 | {
9 | "name": "Orange Shirt",
10 | "description": "This is an orange shirt."
11 | }
12 |
13 | # Indexing the document automatically created the index for us, named "product_index_test" with mapping type "product"
14 |
15 | # The product is now in the index!
16 |
17 | # Let's find it in the index using a GET command:
18 | GET /products_index_test/_search
19 |
20 | # ------------------------------
21 | # 2. Basic Indexing: Create a document with PUT
22 | # ------------------------------
23 | # POST creates the document's ID for us
24 | # PUT requires an id for the document, as part of the URL
25 |
26 | # If we run the following we'll get an error
27 | PUT /products_index_test/product
28 | {
29 | "name": "Blue Shirt",
30 | "description": "This is a blue shirt."
31 | }
32 |
33 | # The error is expected!
34 | # Run this instead:
35 | PUT /products_index_test/product/1
36 | {
37 | "name": "Blue Shirt",
38 | "description": "This is an blue shirt."
39 | }
40 |
41 | # Run this to get all of the products in our index
42 | GET /products_index_test/_search
43 |
44 | # We now have an Orange Shirt and a Blue Shirt in our index.
45 |
46 | # Instead of dynamically creating the index based on the first document we add, we can create the index beforehand, to set certain settings. We will do that later on in the tutorial today.
47 |
48 | # ------------------------------
49 | # 3. Basic Indexing: Bulk adding products
50 | # ------------------------------
51 |
52 | # When you need to index a lot of docs, you should use the bulk API, you may see signficant performance benefits
53 |
54 | POST /products_index_test/product/_bulk
55 | { "index": { "_id": 2 }}
56 | {"name": "Green Shirt", "description": "This is a green shirt."}
57 | { "index": { "_id": 3 }}
58 | {"name": "Purple Shirt", "description": "This is a purple shirt."}
59 |
60 |
61 | GET /products_index_test/_search
62 |
63 | # Let's delete this index, since we won't be using it again.
64 | DELETE /products_index_test
65 |
--------------------------------------------------------------------------------
/lessons/kibana/search.json:
--------------------------------------------------------------------------------
1 | # Hello! This is the Kibana Dev tools console, we'll use this to interact with Elasticsearch
2 |
3 |
4 | #--------------------------------------
5 | # Basic Search: Querying documents with GET
6 | #--------------------------------------
7 |
8 | # We now have about 20k products in the index!
9 | # Let's search for them.
10 | # Find *all* documents
11 |
12 | GET /products_index/product/_search
13 |
14 | # This is the same query.
15 | GET /products_index/product/_search
16 | {
17 | "query": {
18 | "match_all": {}
19 | }
20 | }
21 |
22 | # Let's find all necklaces!
23 | GET /products_index/product/_search
24 | {
25 | "query": {
26 | "match": {
27 | "name": "necklace"
28 | }
29 | }
30 | }
31 |
32 | # Results are ranked by "relevance" (_score)
33 | # More info: https://www.elastic.co/guide/en/elasticsearch/guide/current/relevance-intro.html
34 |
--------------------------------------------------------------------------------
/requirements.txt:
--------------------------------------------------------------------------------
1 | Flask==0.12.3
2 | Jinja2>=2.10.1
3 | MarkupSafe==1.0
4 | Werkzeug==0.15.3
5 | click==6.7
6 | elasticsearch-dsl>=6.1.0
7 | elasticsearch>=6.1.1
8 | itsdangerous==0.24
9 | urllib3==1.26.5
10 |
--------------------------------------------------------------------------------
/searchapp/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/julieqiu/pycon-2018-pyelasticsearch/0c5232f0b6c133239ad885ea8d7d8867c39b6f7b/searchapp/__init__.py
--------------------------------------------------------------------------------
/searchapp/app/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/julieqiu/pycon-2018-pyelasticsearch/0c5232f0b6c133239ad885ea8d7d8867c39b6f7b/searchapp/app/__init__.py
--------------------------------------------------------------------------------
/searchapp/app/app.py:
--------------------------------------------------------------------------------
1 | from flask import Flask, render_template, request
2 |
3 | from searchapp.data import all_products
4 | from searchapp.app.search import search
5 |
6 | app = Flask(__name__)
7 |
8 |
9 | @app.route('/')
10 | @app.route('/index')
11 | def index():
12 | """
13 | Search for products across a variety of terms, and show 9 results for each.
14 | """
15 | search_terms = [
16 | 'necklace',
17 | 'metal necklace',
18 | 'necklce',
19 | 'OK',
20 | 'brass necklace',
21 | 'a brass necklace',
22 | 'necklaces made of brass',
23 | "men's jacket",
24 | ]
25 |
26 | num_results = 9
27 | products_by_category = [(t, search(t, num_results)) for t in search_terms]
28 | return render_template(
29 | 'index.html',
30 | products_by_category=products_by_category,
31 | )
32 |
33 |
34 | @app.route('/search', methods=['GET', 'POST'])
35 | def search_single_product():
36 | """
37 | Execute a search for a specific search term.
38 |
39 | Return the top 50 results.
40 | """
41 | query = request.args.get('search')
42 | num_results = 50
43 | products_by_category = [(query, search(query, num_results))]
44 | return render_template(
45 | 'index.html',
46 | products_by_category=products_by_category,
47 | search_term=query,
48 | )
49 |
50 |
51 | @app.route('/product/Hint
53 | There are a few different ways to combine queries in elasticsearch, but `dis_max` is the best for this. 54 | 55 | Separately create a `name_query` and `description_query`, and then combine them in the queries property of a `dis_max` query. Pass that dis_max query to `s.query`. 56 | 57 | If the results don’t look much better, try adding `"tie_breaker": 0.7` to the dis_max query. This will boost items that match in both the name and description. 58 |7 | {% block content %}{% endblock %} 8 | 9 | 10 | -------------------------------------------------------------------------------- /searchapp/app/templates/index.html: -------------------------------------------------------------------------------- 1 | {% extends "base.html" %} 2 | 3 | {% block content %} 4 | 69 | 70 | 71 | 75 | 76 |
77 | {% for category, products in products_by_category %}
78 |
95 | {% endblock %}
96 |
--------------------------------------------------------------------------------
/searchapp/app/templates/product.html:
--------------------------------------------------------------------------------
1 | {% extends "base.html" %}
2 |
3 | {% block content %}
4 |
6 |
7 |
79 |
93 | {% endfor %}
94 |
80 | "{{ category }}"
81 |
82 |
83 | {% for product in products %}
84 |
85 | 
86 |
92 |
87 | {{ product.name }}
88 |
89 |
90 | {% endfor %}
91 |
8 |
10 | {% endblock %}
11 |
--------------------------------------------------------------------------------
/searchapp/constants.py:
--------------------------------------------------------------------------------
1 | DOC_TYPE = 'product'
2 | INDEX_NAME = 'products_index'
3 |
--------------------------------------------------------------------------------
/searchapp/data.py:
--------------------------------------------------------------------------------
1 | import json
2 | import os
3 | import textwrap
4 | _all_products = None
5 |
6 |
7 | class ProductData():
8 | """
9 | Our product records. In this case they come from a json file, but you could
10 | just as easily load them from a database, or anywhere else.
11 | """
12 |
13 | def __init__(self, id_, name, description, image, taxonomy, price):
14 | self.id = id_
15 | self.name = name
16 | self.description = description
17 | self.image = image
18 | self.taxonomy = taxonomy
19 | self.price = price
20 |
21 | def __str__(self):
22 | return textwrap.dedent("""\
23 | Id: {}
24 | Name: {}
25 | ImageUrl: {}
26 | Taxonomy: {}
27 | Price: ${}
28 | Description:
29 |
30 | {}
31 | """).format(self.id, self.name, self.image, self.taxonomy,
32 | self.price, self.description)
33 |
34 |
35 | def all_products():
36 | """
37 | Returns a list of ~20,000 ProductData objects, loaded from
38 | searchapp/products.json
39 | """
40 |
41 | global _all_products
42 |
43 | if _all_products is None:
44 | _all_products = []
45 |
46 | # Load the product json from the same directory as this file.
47 | dir_path = os.path.dirname(os.path.realpath(__file__))
48 | products_path = os.path.join(dir_path, 'products.json')
49 | with open(products_path) as product_file:
50 | for idx, product in enumerate(json.load(product_file)):
51 | id_ = idx + 1 # ES indexes must be positive integers, so add 1
52 | product_data = ProductData(id_, **product)
53 | _all_products.append(product_data)
54 |
55 | return _all_products
56 |
--------------------------------------------------------------------------------
/searchapp/index_products.py:
--------------------------------------------------------------------------------
1 | from elasticsearch import Elasticsearch
2 |
3 | from searchapp.constants import DOC_TYPE, INDEX_NAME
4 | from searchapp.data import all_products, ProductData
5 |
6 |
7 | def main():
8 | # Connect to localhost:9200 by default.
9 | es = Elasticsearch()
10 |
11 | es.indices.delete(index=INDEX_NAME, ignore=404)
12 | es.indices.create(
13 | index=INDEX_NAME,
14 | body={
15 | 'mappings': {},
16 | 'settings': {},
17 | },
18 | )
19 |
20 | index_product(es, all_products()[0])
21 |
22 |
23 | def index_product(es, product: ProductData):
24 | """Add a single product to the ProductData index."""
25 |
26 | es.create(
27 | index=INDEX_NAME,
28 | doc_type=DOC_TYPE,
29 | id=1,
30 | body={
31 | "name": "A Great Product",
32 | "image": "http://placekitten.com/200/200",
33 | }
34 | )
35 |
36 | # Don't delete this! You'll need it to see if your indexing job is working,
37 | # or if it has stalled.
38 | print("Indexed {}".format("A Great Product"))
39 |
40 |
41 | if __name__ == '__main__':
42 | main()
43 |
--------------------------------------------------------------------------------
/searchapp/run.py:
--------------------------------------------------------------------------------
1 | from searchapp.app.app import app
2 |
3 |
4 | def main():
5 | app.run(debug=True)
6 |
7 | if __name__ == '__main__':
8 | main()
9 |
--------------------------------------------------------------------------------
/setup.py:
--------------------------------------------------------------------------------
1 | from setuptools import setup, find_packages
2 |
3 | setup(
4 | name='searchapp',
5 | packages=find_packages(),
6 | version='0.1',
7 | )
8 |
--------------------------------------------------------------------------------
{{product_json}}
9 |