├── .gitignore ├── images └── elastic-safe-architecture.png ├── queries └── Readme.md ├── Readme.md ├── setup_installation ├── Readme.md └── docker-compose.yml ├── analyzer └── Readme.md └── managing_mapping └── Readme.md /.gitignore: -------------------------------------------------------------------------------- 1 | .idea 2 | *__pycache__* 3 | -------------------------------------------------------------------------------- /images/elastic-safe-architecture.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pooya-mohammadi/elasticsearch-notes/HEAD/images/elastic-safe-architecture.png -------------------------------------------------------------------------------- /queries/Readme.md: -------------------------------------------------------------------------------- 1 | # Queries 2 | 3 | ## Query GeoPoint dataset 4 | The data has been added from the managing_mapping section, check it out. 5 | 6 | ### Distance 7 | The following query list outs locations that are in a distance of 10 miles: 8 | ``` 9 | 10 | ``` -------------------------------------------------------------------------------- /Readme.md: -------------------------------------------------------------------------------- 1 | # Elasticsearch Notes 2 | 3 | In this repository, I jod done my experiences about elasticsearch from different books and sources. This repo contains the following sections: 4 | 1. setup_installation: In this section, I provide a `docker-compose.yml` file to set up elasticsearch alongside kibana. 5 | 1. [setup_installation_Readme.md](https://github.com/pooya-mohammadi/elasticsearch-notes/tree/master/setup_installation) 6 | 2. managing_mapping: In this section, I provide code and information about how to manage different mappings for different data types. 7 | 1. [managing_mapping_Readme.md](https://github.com/pooya-mohammadi/elasticsearch-notes/blob/master/managing_mapping/Readme.md) 8 | -------------------------------------------------------------------------------- /setup_installation/Readme.md: -------------------------------------------------------------------------------- 1 | # Chapter 01 [Setup & Installation] 2 | In this section, we are going to setup elasticsearch using docker-compose which is very convenient and easier compared to setting it up from scratch on a local machine. 3 | The elasticsearch's version is 8.1.2 which can be easily changed to other versions. For more information about different versions take a look at [docker-hub-elasticsearch-main-page](https://hub.docker.com/_/elasticsearch?tab=tags) 4 | 5 | ## Start a docker based node: 6 | ```commandline 7 | sudo docker-compose up 8 | ``` 9 | 10 | ## Check elasticsearch 11 | ```commandline 12 | curl localhost:9200 13 | ``` 14 | 15 | ## Check kibana 16 | Browse to localhost:5601 17 | 18 | ## Elastic Safe Architecture 19 | -------------------------------------------------------------------------------- /setup_installation/docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: '3.7' 2 | 3 | services: 4 | elasticsearch: 5 | image: elasticsearch:8.1.2 6 | container_name: elasticsearch 7 | hostname: elasticsearch 8 | # restart: always 9 | environment: 10 | - xpack.security.enabled=false 11 | - discovery.type=single-node 12 | - vm_max_map_count=262144 13 | ulimits: 14 | memlock: 15 | soft: -1 16 | hard: -1 17 | nofile: 18 | soft: 65536 19 | hard: 65536 20 | cap_add: 21 | - IPC_LOCK 22 | volumes: 23 | - elasticsearch-data-volume:/usr/share/elasticsearch/data 24 | ports: 25 | - 9200:9200 26 | - 9300:9300 27 | networks: 28 | - elk 29 | 30 | kibana: 31 | container_name: kibana 32 | image: kibana:8.1.2 33 | # restart: always 34 | hostname: kibana 35 | environment: 36 | SERVER_NAME: kibana 37 | ELASTICSEARCH_HOSTS: http://elasticsearch:9200 38 | ports: 39 | - 5601:5601 40 | depends_on: 41 | - elasticsearch 42 | networks: 43 | - elk 44 | 45 | volumes: 46 | elasticsearch-data-volume: 47 | driver: local 48 | 49 | networks: 50 | elk: -------------------------------------------------------------------------------- /analyzer/Readme.md: -------------------------------------------------------------------------------- 1 | # Persian Analyzer 2 | ### rebuilt_persian is the name of the analyzer the default analyzer which will be chosen for upcoming fields is the english default analyzer. 3 | ### Change the rebuild_presian to default and every field will be based on that and not english default one! 4 | 5 | ``` 6 | DELETE persian_example 7 | PUT persian_example 8 | { 9 | "settings": { 10 | "analysis": { 11 | "char_filter": { 12 | "zero_width_spaces": { 13 | "type": "mapping", 14 | "mappings": [ 15 | """\u200C=>\u0020""" 16 | ] 17 | } 18 | }, 19 | "filter": { 20 | "persian_stop": { 21 | "type": "stop", 22 | "stopwords": "_persian_" 23 | } 24 | }, 25 | "analyzer": { 26 | "rebuilt_persian": { 27 | "tokenizer": "standard", 28 | "char_filter": [ 29 | "zero_width_spaces" 30 | ], 31 | "filter": [ 32 | "lowercase", 33 | "decimal_digit", 34 | "arabic_normalization", 35 | "persian_normalization", 36 | "persian_stop" 37 | ] 38 | } 39 | } 40 | } 41 | } 42 | } 43 | ``` 44 | 45 | ### GET Analyzer's output on a random text 46 | ```GET persian_example/_analyze 47 | { 48 | "field": "text", 49 | "text" : "من و علی با هم در کتابخانه آشنا شدیم" 50 | } 51 | ``` 52 | ### PUT mapping with the created analyzer 53 | ```PUT persian_example/_mapping 54 | { 55 | "properties": { 56 | "text": { 57 | "type": "text", 58 | "analyzer": "rebuilt_persian" 59 | } 60 | } 61 | } 62 | ``` 63 | ### PUT a document to check new analyzer's effects 64 | ```PUT persian_example/_doc/1 65 | { 66 | "text": "من و حسین با یکدیگر در مدرسه هستیم." 67 | } 68 | ``` 69 | ### GET the saved doc, as you can see it's not changed. The analyzer is performed at search time 70 | ```GET persian_example/_doc/1``` 71 | 72 | ### Search for the word حسین 73 | ### As you can see it finds the only doc we saved earlier 74 | ```GET persian_example/_search 75 | { 76 | "query": { 77 | "match": { 78 | "text": "حسین" 79 | } 80 | } 81 | } 82 | ``` 83 | ### Now let's search for a persian stop word which happened to be in our doc, "در" 84 | ```GET persian_example/_search 85 | { 86 | "query": { 87 | "match": { 88 | "text": "در" 89 | } 90 | 91 | } 92 | } 93 | ``` 94 | ### As you can see output is empty although the searched text is present in the document but it's removed by persian stop words at indexing time! 95 | 96 | ### Search analyzer can be modified as well: 97 | ```PUT persian_example/_mapping 98 | { 99 | "properties": { 100 | "text": { 101 | "type": "text", 102 | "analyzer": "rebuilt_persian", 103 | "search_analyzer": "rebuilt_persian" 104 | } 105 | } 106 | } 107 | ``` 108 | ### Still the search will return no documents 109 | ```GET persian_example/_search 110 | { 111 | "query": { 112 | "match": { 113 | "text": "در" 114 | } 115 | 116 | } 117 | }``` -------------------------------------------------------------------------------- /managing_mapping/Readme.md: -------------------------------------------------------------------------------- 1 | # ElasticSearch 2 | 3 | ## Notes: 4 | 1. An `index` is like a `database` and likewise a `mapping` is like a `table` 5 | 6 | ## Codes: 7 | ```commandline 8 | # Go to `kibana/management/dev_tools 9 | 10 | # create an index: 11 | PUT test 12 | 13 | # put a document in the index. First document put creates the mapping of table 14 | PUT test/_doc/1 15 | { 16 | "name": "Pooya", 17 | "age": 35 18 | } 19 | 20 | # Get an specific id 21 | GET test/_doc/1 22 | 23 | # get the mapping 24 | GET test/_mapping 25 | 26 | 27 | # Adding a new document with incompatible types which throws a `400 - bad request` error! 28 | PUT test/_doc/2 29 | { 30 | "name": 1, 31 | "age": "old" 32 | } 33 | 34 | # delete the index: 35 | DELETE test 36 | ``` 37 | 38 | ## Creating a custom mapping: 39 | ```commandline 40 | # Create test index 41 | PUT test 42 | 43 | # Create a test mapping: 44 | PUT test/_mapping 45 | { 46 | "properties": { 47 | "id": { 48 | "type": "keyword" 49 | }, 50 | "date": { 51 | "type": "date" 52 | }, 53 | "customer_id": { 54 | "type": "keyword" 55 | }, 56 | "sent": { 57 | "type": "boolean" 58 | }, 59 | "name": { 60 | "type": "keyword" 61 | }, 62 | "quantity": { 63 | "type": "integer" 64 | }, 65 | "price": { 66 | "type": "double" 67 | }, 68 | "vat": { 69 | "type": "double", 70 | "index": "false" 71 | } 72 | } 73 | } 74 | 75 | # Get the mapping: 76 | GET test/_mapping 77 | ``` 78 | 79 | ## Creating array fields 80 | ```commandline 81 | # Create an index 82 | PUT test 83 | 84 | # Create a mapping 85 | # Setting store to true one can have array form inputs 86 | PUT test/_mapping 87 | { 88 | "properties" : { 89 | "name" : {"type" : "keyword"}, 90 | "tag" : {"type" : "keyword", "store" : "true"} 91 | } 92 | } 93 | 94 | # Get mapping 95 | Get test/_mapping 96 | 97 | # Store a document 98 | Put test/_doc/1 99 | {"name": "document1", "tag": "awesome"} 100 | 101 | # Store an array form doc 102 | PUT test/_doc/2 103 | {"name": "document2", "tag": ["cool", "awesome", "amazing"] } 104 | 105 | # Get the array form doc 106 | GET test/_doc/2 107 | ``` 108 | 109 | ## Creating Objects 110 | ```commandline 111 | # Creating an index 112 | PUT test 113 | 114 | # Create a mapping for object form inputs 115 | # Objects can have inner embeddings 116 | PUT test/_mapping 117 | { 118 | "properties": { 119 | "id": { 120 | "type": "keyword" 121 | }, 122 | "date": { 123 | "type": "date" 124 | }, 125 | "customer_id": { 126 | "type": "keyword", 127 | "store": "yes" 128 | }, 129 | "sent": { 130 | "type": "boolean" 131 | }, 132 | "item": { 133 | "type": "object", 134 | "properties": { 135 | "name": { 136 | "type": "text" 137 | }, 138 | "quantity": { 139 | "type": "integer" 140 | }, 141 | "price": { 142 | "type": "double" 143 | }, 144 | "vat": { 145 | "type": "double" 146 | } 147 | } 148 | } 149 | } 150 | } 151 | ``` 152 | 153 | ## Creating dynamic types: 154 | ```commandline 155 | # Remove index test 156 | DELETE test 157 | 158 | # Creating an index 159 | PUT test 160 | 161 | # Creating a dynamic mapping 162 | PUT test/_mapping 163 | { 164 | "dynamic_date_formats": [ 165 | "yyyy-MM-dd", 166 | "dd-MM-yyyy" 167 | ], 168 | "date_detection": true, 169 | "numeric_detection": true, 170 | "dynamic_templates": [ 171 | { 172 | "template1": { 173 | "match": "*", 174 | "match_mapping_type": "long", 175 | "mapping": { 176 | "type": "{dynamic_type}", 177 | "store": true 178 | } 179 | } 180 | } 181 | ] 182 | } 183 | 184 | # Getting the mapping 185 | GET test/_mapping 186 | 187 | # Adding a doc 188 | PUT test/_doc/1 189 | { 190 | "dynamic_date_formats": "2022-02-12" 191 | } 192 | 193 | # Getting the mapping 194 | GET test/_mapping 195 | # the datetype is fixed 196 | 197 | # Get doc 198 | GET test/_doc/1 199 | 200 | # Adding another doc 201 | PUT test/_doc/2 202 | { 203 | "dynamic_date_formats": "12-02-2022" 204 | } 205 | # This will raise an error, because the input is the mapping is fixed by adding the former document! 206 | ``` 207 | 208 | ## Creating nested documents: 209 | Nested objects are not searchable with standard queries, only with nested ones. Theyare not shown in standard query results. 210 | ```commandline 211 | # Remove index test 212 | DELETE test 213 | 214 | # Creating an index 215 | PUT test 216 | 217 | # Creating a netsted mapping 218 | PUT test/_mapping 219 | { 220 | "properties": { 221 | "id": { 222 | "type": "keyword" 223 | }, 224 | "date": { 225 | "type": "date" 226 | }, 227 | "customer_id": { 228 | "type": "keyword" 229 | }, 230 | "sent": { 231 | "type": "boolean" 232 | }, 233 | "item": { 234 | "type": "nested", 235 | "properties": { 236 | "name": { 237 | "type": "keyword" 238 | }, 239 | "quantity": { 240 | "type": "long" 241 | }, 242 | "price": { 243 | "type": "double" 244 | }, 245 | "vat": { 246 | "type": "double" 247 | } 248 | } 249 | } 250 | } 251 | } 252 | 253 | # Getting the mapping 254 | GET test/_mapping 255 | 256 | # Adding a doc 257 | PUT test/_doc/1 258 | { 259 | "customer_id": 1, 260 | "date" : "2022-02-11", 261 | "id" : 1, 262 | "item": [ 263 | {"name": "pooya", "price" : 10}, 264 | {"name": "mohammadi", "price" : 10} 265 | ] 266 | } 267 | 268 | # Get doc 269 | GET test/_doc/1 270 | 271 | ``` 272 | 273 | # Creating GeoPoints 274 | ```commandline 275 | # Creating a geo index 276 | PUT geo_index 277 | 278 | # Creating a geopoint mapping 279 | PUT geo_index/_mapping 280 | { 281 | "properties": { 282 | "id": { 283 | "type": "keyword" 284 | }, 285 | "timestamp": { 286 | "type": "date" 287 | }, 288 | "text": { 289 | "type": "text" 290 | }, 291 | "location": { 292 | "type": "geo_point" 293 | } 294 | } 295 | } 296 | 297 | 298 | # Getting the mapping 299 | GET geo_index/_mapping 300 | 301 | 302 | # Adding a doc 303 | PUT geo_index/_doc/1 304 | { 305 | "timestamp": "2022-02-12", 306 | "text": "Geo-point as an object", 307 | "location": { 308 | "lat": 41.12, 309 | "lon": -71.34 310 | } 311 | } 312 | 313 | # get a doc 314 | GET geo_index/_doc/1 315 | ``` 316 | 317 | # Creating GeoShape 318 | ```commandline 319 | Having a simple example is bit harder than I thought it would be! 320 | I implemented information from https://wsdot.wa.gov/mapsdata/geodatacatalog/default.htm#admin using a node code! 321 | Node code is in here: https://github.com/ereshzealous/node_playground/tree/master/elastic-geo-spatial 322 | ``` --------------------------------------------------------------------------------