Elastic Search Notes – My memo Wall

Cardinality aggregation is a technique used in Elasticsearch to find the number of distinct values in a field

In Elasticsearch, multi-fields are a feature that allows users to index the same field in different ways for different purposes

Defining your own mapping

Rules

If you do not define a mapping ahead of time, Elasticsearch dynamically creates the mapping for you.
If you do decide to define your own mapping, you can do so at index creation.
ONE mapping is defined per index. Once the index has been created, we can only add new fields to a mapping. We CANNOT change the mapping of an existing field.
If you must change the type of an existing field, you must create a new index with the desired mapping, then reindex all documents into the new index.

See mapping:

GET Name-the-index-whose-mapping-you-want-to-view/_mapping

Save query into new index

POST _reindex
{
  "source": {
    "index": "twitter",
    "query":{
        "term":{"author.keyword":"Alex"} 
    }
  },
  "dest": {
    "index": "new_twitter"
  }
}

Get information on an index:

GET Enter_name_of_the_index_here/_search

Aggregation request:

GET Enter_name_of_the_index_here/_search
{
  "aggs": {
    "Name your aggregations here": {
      "Specify the aggregation type here": {
        "field": "Name the field you want to aggregate on here"
      }
    }
  }
}

Paginate in Search, use from and size. Default is top 10 searches if size is not specified. By default, you cannot use from and size to page through more than 10,000 hits.

GET /_search
{
  "from": 5,
  "size": 20,
  "query": {
    "match": {
      "user.id": "kimchy"
    }
  }
}

To return a selected field, use _source.

{
  "size": 20,  
  "query": {
        "bool": {
            "must": [
                 {
                "match": {
                    "Type": "TEST"
                 }
                }
            ],
            "filter": [
                {
                    "range": {
                        "DateCreated": {
                            "gte": "2021-12-15T00:00:00.000Z",
                            "lte": "2021-12-15T23:59:59.999Z"
                        }
                    }
                }
            ]
        }
    },
   "_source": ["SelectID"]
}

Reindex elastic search with zero down time (index alias)

https://logz.io/blog/managing-elasticsearch-indices/

Indexing best practice

Create Index Alias

Nodejs client library examples

invalid_type_name_exception error: explicitly specify api version

FORBIDDEN/12/index read-only error: low on disk space

Delete by Query
Example:

POST sample-index1/_delete_by_query
{
  "query": {
    "match": {
      "movie-length": "124"
    }
  }
}

Elastic Search Cheat Sheet

Elastic Search Postman Blog (not collection)

Gist that uses the aws-sdk to make signed requests to an Amazon ES endpoint (ES6)

Query, filter, count:

{
  "query": {
    "bool": {"must": [
        {
          "match_phrase": {
            "name": "test"
          }
          
        }
      ],
      "filter": [
                {
                    "range": {
                        "DateCreated": {
                            "gte": "2023-01-01T00:00:00.000Z",
                            "lte": "2023-12-31T23:59:59.999Z"
                        }
                    }
                }
            ]
      }
  }
}

Elastic Search Commands Using curl

Elasticsearch index creation and access API through Postman

Gist: Use the Node elasticsearch client with Amazon ES 6
Gist: Elastic Lambda Gist Sample – s3 logs to ES domain

Aggregate example:

{
   "size": 10,
  "qery": {
    "bool": {"must": [
        {
          "match": {
            "Id": "test"
          }
        }
      ]
      }
  },
  "aggs": {
    "attribCount": {
      "terms": {
        "field": "ConversationId.keyword"
      }
    }
  }
}

Paginate with composite aggregation
If you want to retrieve all terms or all combinations of terms in a nested terms aggregation you should use the Composite aggregation which allows to paginate over all possible terms rather than setting a size greater than the cardinality of the field in the terms aggregation. The terms aggregation is meant to return the top terms and does not allow pagination.

ElasticSearch composite aggregation and pagination examples (paginate on after_key)

Examples of composite aggregation

Runtime field