.
675 |
--------------------------------------------------------------------------------
/Makefile:
--------------------------------------------------------------------------------
1 | SHELL:=bash
2 | ESCTL_VERSION = $(shell \grep version pyproject.toml | \egrep -oh "[.0-9]+")
3 |
4 | install:
5 | @rm -rf dist/*
6 | uv build
7 | uv run pipx install --force dist/esctl-*-py3-none-any.whl
8 |
9 | test:
10 | uv run pytest
11 |
12 | test-install:
13 | docker run --entrypoint=/bin/bash -v `pwd`:/tmp/esctl:ro python:$(shell cat .python-version) -c "pip install uv && cp -r /tmp/esctl /opt && cd /opt/esctl && uv build && uv run pipx install --force dist/esctl-*-py3-none-any.whl && uv run pipx ensurepath && source ~/.bashrc && esctl config context list && cat ~/.esctlrc && esctl cluster health"
14 |
15 | format:
16 | uv run ruff check --select I --fix
17 | uv run ruff check --fix
18 | uv run ruff format
19 | # uv run mypy esctl/
20 |
21 | tag:
22 | git tag $(ESCTL_VERSION)
23 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 |
2 |
3 | Esctl
4 |
5 |
6 |
7 | A Command-Line Interface designed to ease Elasticsearch administration.
8 |
9 |
10 |
11 | 
12 |
13 |
14 | 
15 |
16 |
17 |
18 | 
19 |
20 |
21 |
22 | 
23 |
24 |
25 |
26 |
27 | Key Features •
28 | Installation •
29 | How To Use •
30 | Examples •
31 | License •
32 | Developing
33 |
34 |
35 |
36 |
37 | Esctl is a CLI tool for Elasticsearch. [I designed it](https://jeromepin.fr/posts/esctl-managing-elasticsearch-from-command-line/) to shorten huge `curl` commands Elasticsearch operators were running like :
38 |
39 | ```bash
40 | curl -XPUT --user "john:doe" 'http://elasticsearch.example.com:9200/_cluster/settings' -d '{
41 | "transient" : {
42 | "cluster.routing.allocation.enable": "NONE"
43 | }
44 | }'
45 | ```
46 |
47 | The equivalent with `esctl` is
48 |
49 | ```bash
50 | esctl cluster routing allocation enable none
51 | ```
52 |
53 | ## Key Features
54 |
55 | * **Easy to use CLI** rather than long curl commands (thanks to [cliff](https://github.com/openstack/cliff))
56 | * Cluster-level informations : **stats**, **info**, **health**, **allocation explanation**
57 | * Node-level informations : **list**, **hot threads**, **exclusion**, **stats**
58 | * Cluster-level and index-level **settings**
59 | * `_cat` API for **allocation**, **plugins** and **thread pools**
60 | * **Index management** : open, close, create, delete, list
61 | * `raw` command to perform raw HTTP calls when esctl doesn't provide a nice interface for a given route.
62 | * Per-module **log configuration**
63 | * X-Pack APIs : **users** and **roles**
64 | * **Multiple output formats** : table, csv, json, value, yaml
65 | * [JMESPath](https://jmespath.org/) queries using the `--jmespath` flag
66 | * Colored output !
67 | * Run arbitrary pre-commands before issuing the call to Elasticsearch (like running `kubectl port-forward` for example)
68 | * Fetch cluster's credentials from external commands instead of having them shown in cleartext in the config file
69 |
70 |
71 | ## Installation
72 |
73 | ### Using PIP
74 |
75 | ```bash
76 | pip install esctl
77 | ```
78 |
79 | ### From source
80 |
81 | ```bash
82 | pip install git+https://github.com/jeromepin/esctl.git
83 | ```
84 |
85 |
86 | ## How To Use
87 |
88 | Esctl relies on a `~/.esctlrc` file containing its config. This file is automatically created on the first start if it doesn't exists :
89 |
90 | ```yaml
91 | clusters:
92 | bar:
93 | servers:
94 | - https://bar.example.com
95 |
96 | users:
97 | john-doe:
98 | username: john
99 | external_password:
100 | command:
101 | run: kubectl --context=bar --namespace=baz get secrets -o json my-secret | jq -r '.data.password||@base64d'
102 |
103 | contexts:
104 | foo:
105 | user: john-doe
106 | cluster: bar
107 |
108 | default-context: foo
109 | ```
110 |
111 | ### Running pre-commands
112 |
113 | Sometimes, you need to execute a shell command right before running the `esctl` command. Like running a `kubectl port-forward` in order to connect to your Kubernetes cluster.
114 | There is a `pre_commands` block inside the context which can take care of that :
115 |
116 | ```yaml
117 | clusters:
118 | remote-kubernetes:
119 | servers:
120 | - http://localhost:9200
121 | contexts:
122 | my-distant-cluster:
123 | cluster: remote-kubernetes
124 | pre_commands:
125 | - command: kubectl --context=my-kubernetes-context --namespace=elasticsearch port-forward svc/elasticsearch 9200
126 | wait_for_exit: false
127 | wait_for_output: Forwarding from
128 | user: john-doe
129 | ```
130 |
131 | Along with `command`, you can pass two options :
132 | * `wait_for_exit` (_default_: `true`) : wait for the command to exit before continuing. Usually set to `false` when the command is running in the foreground.
133 | * `wait_for_output` : if `wait_for_exit` is `false`, look for a specific output in the command's stdout. The string to look-for is interpreted as a regular expression passed to Python's [re.compile()](https://docs.python.org/3.7/library/re.html).
134 |
135 |
136 | ## Examples
137 |
138 |
139 | 
140 |
141 |
142 |
143 | ## License
144 |
145 | `esctl` is licensed under the GNU GPLv3. See [LICENCE](https://github.com/jeromepin/esctl/blob/master/LICENSE) file.
146 |
147 | ## Developing
148 |
149 | ### Install
150 |
151 | ```bash
152 | make install
153 | ```
154 |
155 | ### Run tests
156 |
157 | ```bash
158 | make test
159 | ```
160 |
161 | ### Format and lint code
162 |
163 | ```bash
164 | make lint
165 | ```
166 |
--------------------------------------------------------------------------------
/esctl/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/jeromepin/esctl/97280374464e5cd9217bffb857afdba9c29e461a/esctl/__init__.py
--------------------------------------------------------------------------------
/esctl/cmd/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/jeromepin/esctl/97280374464e5cd9217bffb857afdba9c29e461a/esctl/cmd/__init__.py
--------------------------------------------------------------------------------
/esctl/cmd/alias.py:
--------------------------------------------------------------------------------
1 | from esctl.commands import EsctlLister
2 | from esctl.formatter import JSONToCliffFormatter
3 |
4 |
5 | class AliasList(EsctlLister):
6 | """List all aliases."""
7 |
8 | def take_action(self, parsed_args):
9 | aliases = self.es.cat.aliases(name=parsed_args.alias, format="json")
10 |
11 | return JSONToCliffFormatter(aliases).format_for_lister(
12 | columns=[("index",), ("alias",)],
13 | )
14 |
15 | def get_parser(self, prog_name):
16 | parser = super().get_parser(prog_name)
17 | parser.add_argument(
18 | "alias",
19 | help="A comma-separated list of alias names to return",
20 | default=None,
21 | nargs="?",
22 | )
23 |
24 | return parser
25 |
--------------------------------------------------------------------------------
/esctl/cmd/cat.py:
--------------------------------------------------------------------------------
1 | from esctl.commands import EsctlLister
2 | from esctl.formatter import JSONToCliffFormatter
3 | from esctl.utils import Color
4 |
5 |
6 | class CatAllocation(EsctlLister):
7 | """Show the number of shards allocated to each node.
8 |
9 | Provides a snapshot of the number of shards allocated to each data node
10 | and their disk space.
11 | """
12 |
13 | def take_action(self, parsed_args):
14 | allocation = self.transform(self.es.cat.allocation(format="json"))
15 |
16 | return JSONToCliffFormatter(allocation).format_for_lister(
17 | columns=[
18 | ("shards"),
19 | ("disk.indices"),
20 | ("disk.used"),
21 | ("disk.avail"),
22 | ("disk.total"),
23 | ("disk.percent"),
24 | ("host"),
25 | ("ip", "IP"),
26 | ("node"),
27 | ],
28 | )
29 |
30 | def transform(self, allocation):
31 | nodes = []
32 |
33 | for node in allocation:
34 | if self.uses_table_formatter() and node.get("disk.percent") is not None:
35 | if int(node.get("disk.percent")) > 85:
36 | node["disk.percent"] = Color.colorize(
37 | node.get("disk.percent"),
38 | Color.RED,
39 | )
40 |
41 | elif int(node.get("disk.percent")) > 70:
42 | node["disk.percent"] = Color.colorize(
43 | node.get("disk.percent"),
44 | Color.YELLOW,
45 | )
46 |
47 | nodes.append(node)
48 |
49 | return nodes
50 |
51 |
52 | class CatPlugins(EsctlLister):
53 | """Returns informations about installed plugins across nodes."""
54 |
55 | def take_action(self, parsed_args):
56 | plugins = self.transform(self.es.cat.plugins(format="json"))
57 |
58 | return JSONToCliffFormatter(plugins).format_for_lister(
59 | columns=[("name", "node"), ("component", "plugin"), ("version")],
60 | )
61 |
62 | def transform(self, plugins):
63 | return sorted(plugins, key=lambda i: i["name"])
64 |
65 |
66 | class CatShards(EsctlLister):
67 | """Provides a detailed view of shard allocation on nodes."""
68 |
69 | def take_action(self, parsed_args):
70 | nodes = [n.get("name") for n in self.es.cat.nodes(format="json", h="name")]
71 | shards = self.transform(
72 | self.es.cat.shards(
73 | format="json",
74 | index=parsed_args.index,
75 | h="index,node,shard,prirep",
76 | ),
77 | nodes,
78 | )
79 |
80 | columns = [("index",)] + [
81 | (
82 | n,
83 | n.lower(),
84 | )
85 | for n in nodes
86 | ]
87 |
88 | if self.has_unassigned_shards:
89 | columns = columns + [("UNASSIGNED",)]
90 |
91 | return JSONToCliffFormatter(shards).format_for_lister(columns=columns)
92 |
93 | def transform(self, shards_list: list[dict[str, str]], nodes: list[str]):
94 | indices = {}
95 | self.has_unassigned_shards: bool = False
96 |
97 | for shard in shards_list:
98 | shard_index = shard.get("index")
99 |
100 | if shard_index not in indices:
101 | indices[shard_index] = {
102 | "index": shard_index,
103 | **{n: "" for n in nodes},
104 | }
105 |
106 | shard_node = shard.get("node") or "UNASSIGNED"
107 |
108 | if shard_node == "UNASSIGNED":
109 | self.has_unassigned_shards = True
110 |
111 | indices[shard_index][shard_node] = f"{shard.get('shard')}{shard.get('prirep')}"
112 |
113 | return list(indices.values())
114 |
115 | def get_parser(self, prog_name):
116 | parser = super().get_parser(prog_name)
117 | parser.add_argument(
118 | "index",
119 | help="A comma-separated list of index names to limit the returned information",
120 | nargs="?",
121 | )
122 | return parser
123 |
124 |
125 | class CatThreadpool(EsctlLister):
126 | """Show cluster-wide thread pool statistics per node.
127 |
128 | Thread pools reference : https://www.elastic.co/guide/en/elasticsearch/reference/6.8/modules-threadpool.html # noqa
129 | """
130 |
131 | _default_headers = "node_name,name,active,queue,rejected,type"
132 |
133 | def take_action(self, parsed_args):
134 | headers = parsed_args.headers if parsed_args.headers else self._default_headers
135 |
136 | thread_pools = self.transform(
137 | self.es.cat.thread_pool(
138 | format="json",
139 | h=headers,
140 | thread_pool_patterns=parsed_args.thread_pool_patterns,
141 | ),
142 | )
143 |
144 | return JSONToCliffFormatter(thread_pools).format_for_lister(
145 | columns=[(h,) for h in headers.split(",")],
146 | )
147 |
148 | def get_parser(self, prog_name):
149 | parser = super().get_parser(prog_name)
150 |
151 | parser.add_argument(
152 | "--thread-pool-patterns",
153 | help=("A comma-separated list of regular-expressions or strings to filter the thread pools in the output"),
154 | )
155 | parser.add_argument(
156 | "--headers",
157 | help=("A comma-separated list of column names to display"),
158 | default=self._default_headers,
159 | type=str,
160 | )
161 |
162 | return parser
163 |
164 | def transform(self, raw_thread_pools):
165 | modified_thread_pools = []
166 |
167 | for thread_pool in raw_thread_pools:
168 | for column, value in thread_pool.items():
169 | # Colorize any number above 0 in the following columns
170 | if column in ["active", "queue", "rejected"] and int(value) > 0:
171 | thread_pool[column] = Color.colorize(value, Color.RED)
172 |
173 | modified_thread_pools.append(thread_pool)
174 |
175 | return modified_thread_pools
176 |
177 |
178 | class CatTemplates(EsctlLister):
179 | """Returns information about existing templates."""
180 |
181 | def take_action(self, parsed_args):
182 | templates = self.es.cat.templates(name=parsed_args.name, format="json")
183 |
184 | return JSONToCliffFormatter(templates).format_for_lister(
185 | columns=[("name",), ("index_patterns",), ("order",), ("version")],
186 | )
187 |
188 | def get_parser(self, prog_name):
189 | parser = super().get_parser(prog_name)
190 | parser.add_argument(
191 | "name",
192 | help="A pattern that returned template names must match",
193 | nargs="?",
194 | )
195 | return parser
196 |
--------------------------------------------------------------------------------
/esctl/cmd/cluster.py:
--------------------------------------------------------------------------------
1 | import elasticsearch as elasticsearch
2 |
3 | from esctl.cmd.settings import AbstractClusterSettings
4 | from esctl.commands import EsctlLister, EsctlShowOne
5 | from esctl.utils import Color, flatten_dict
6 |
7 |
8 | class ClusterAllocationExplain(EsctlLister):
9 | """Provide explanations for shard allocations failures."""
10 |
11 | def take_action(self, parsed_args):
12 | try:
13 | response = self.es.cluster.allocation_explain()
14 | except elasticsearch.TransportError as transport_error:
15 | if transport_error.args[0] == 400:
16 | self.log.warning(
17 | "Unable to find any unassigned shards to explain."
18 | " This may indicate that all shards are allocated.",
19 | )
20 |
21 | return (("Attribute", "Value"), tuple())
22 | else:
23 | output = {
24 | "index": response.get("index"),
25 | "can_allocate": response.get("can_allocate"),
26 | "explanation": response.get("allocate_explanation"),
27 | "last_allocation_status": response.get("unassigned_info").get(
28 | "last_allocation_status",
29 | ),
30 | "reason": response.get("unassigned_info").get("reason"),
31 | }
32 |
33 | for node in response.get("node_allocation_decisions"):
34 | output[node.get("node_name")] = node.get("deciders")[0].get(
35 | "explanation",
36 | )
37 |
38 | return (("Attribute", "Value"), tuple(output.items()))
39 |
40 |
41 | class ClusterHealth(EsctlShowOne):
42 | """Show the cluster health."""
43 |
44 | def take_action(self, parsed_args):
45 | health = self._sort_and_order_dict(self.es.cluster.health())
46 |
47 | if self.uses_table_formatter():
48 | health["status"] = Color.colorize(
49 | health.get("status"),
50 | getattr(Color, health.get("status").upper()),
51 | )
52 |
53 | return (tuple(health.keys()), tuple(health.values()))
54 |
55 |
56 | class ClusterInfo(EsctlShowOne):
57 | """Show basic informations about the cluster."""
58 |
59 | def take_action(self, parsed_args):
60 | infos = self.es.info()
61 |
62 | if self.uses_table_formatter():
63 | infos = self._delete_and_merge_inner_dict_into_parent(infos, "version")
64 |
65 | return (tuple(infos.keys()), tuple(infos.values()))
66 |
67 |
68 | class ClusterStats(EsctlShowOne):
69 | """Show cluster stats."""
70 |
71 | def take_action(self, parsed_args):
72 | cluster_stats = self._sort_and_order_dict(
73 | self.transform(flatten_dict(self.es.cluster.stats())),
74 | )
75 |
76 | return (tuple(cluster_stats.keys()), tuple(cluster_stats.values()))
77 |
78 | def transform(self, stats):
79 | for attribute in [
80 | "nodes.jvm.versions",
81 | "nodes.os.cpu",
82 | "nodes.plugins",
83 | "nodes.os.names",
84 | "nodes.os.pretty_names",
85 | "nodes.packaging_types",
86 | ]:
87 | if attribute in stats:
88 | for key, value in self.objects_list_to_flat_dict(
89 | stats.get(attribute),
90 | ).items():
91 | stats.update({f"{attribute}{key}": value})
92 |
93 | del stats[attribute]
94 |
95 | return stats
96 |
97 |
98 | class ClusterRoutingAllocationEnable(AbstractClusterSettings):
99 | """Get and set the cluster's routing allocation policy."""
100 |
101 | def take_action(self, parsed_args):
102 | if parsed_args.status is not None:
103 | self._settings_set(
104 | "cluster.routing.allocation.enable",
105 | parsed_args.status,
106 | persistency="persistent" if parsed_args.persistent else "transient",
107 | )
108 |
109 | return self._settings_get("cluster.routing.allocation.enable")
110 |
111 | def get_parser(self, prog_name):
112 | parser = super(AbstractClusterSettings, self).get_parser(prog_name)
113 |
114 | persistency_group = parser.add_mutually_exclusive_group()
115 | persistency_group.add_argument(
116 | "--transient",
117 | action="store_true",
118 | help=("Set setting as transient (default)"),
119 | )
120 | persistency_group.add_argument(
121 | "--persistent",
122 | action="store_true",
123 | help=("Set setting as persistent"),
124 | )
125 |
126 | parser.add_argument(
127 | "status",
128 | nargs="?",
129 | help=("Routing allocation status"),
130 | choices=["all", "primaries", "new_primaries", "none"],
131 | )
132 |
133 | return parser
134 |
--------------------------------------------------------------------------------
/esctl/cmd/config.py:
--------------------------------------------------------------------------------
1 | import yaml
2 |
3 | from esctl.commands import EsctlCommand, EsctlLister
4 | from esctl.formatter import JSONToCliffFormatter
5 | from esctl.main import Esctl
6 | from esctl.utils import Color, setup_yaml
7 |
8 |
9 | class ConfigClusterList(EsctlLister):
10 | """List all configured clusters."""
11 |
12 | def take_action(self, parsed_args):
13 | clusters = [
14 | {
15 | "name": cluster_name,
16 | "servers": "\n".join(cluster_definition.get("servers")),
17 | }
18 | for cluster_name, cluster_definition in Esctl._config.get(
19 | "clusters",
20 | ).items()
21 | ]
22 |
23 | return JSONToCliffFormatter(clusters).format_for_lister(
24 | columns=[("name"), ("servers")],
25 | )
26 |
27 |
28 | class ConfigContextList(EsctlLister):
29 | """List all contexts."""
30 |
31 | def take_action(self, parsed_args):
32 | contexts = [
33 | {
34 | "name": context_name,
35 | "user": context_definition.get("user"),
36 | "cluster": context_definition.get("cluster"),
37 | }
38 | for context_name, context_definition in Esctl._config.get(
39 | "contexts",
40 | ).items()
41 | ]
42 |
43 | return JSONToCliffFormatter(self.transform(contexts)).format_for_lister(
44 | columns=[("name"), ("user"), ("cluster")],
45 | )
46 |
47 | def transform(self, raw_contexts):
48 | modified_contexts = []
49 |
50 | for context in raw_contexts:
51 | if context.get("name") == Esctl._config.get("default-context"):
52 | for context_attribute_name, context_attribute_value in context.items():
53 | context[context_attribute_name] = Color.colorize(
54 | context_attribute_value,
55 | Color.UNDERLINE,
56 | )
57 |
58 | modified_contexts.append(context)
59 |
60 | return modified_contexts
61 |
62 |
63 | class ConfigContextSet(EsctlCommand):
64 | """Set the default context."""
65 |
66 | def take_action(self, parsed_args):
67 | Esctl._config["default-context"] = parsed_args.context
68 | Esctl._config_file_parser.write_config_file(dict(Esctl._config))
69 |
70 | def get_parser(self, prog_name):
71 | parser = super().get_parser(prog_name)
72 | parser.add_argument(
73 | "context",
74 | help=("Context to set as default"),
75 | choices=Esctl._config.get("contexts"),
76 | )
77 |
78 | return parser
79 |
80 |
81 | class ConfigShow(EsctlCommand):
82 | """Print the config."""
83 |
84 | def take_action(self, parsed_args):
85 | setup_yaml()
86 | print(
87 | yaml.safe_dump(
88 | dict(Esctl._config),
89 | None,
90 | default_flow_style=False,
91 | width=500,
92 | ),
93 | )
94 |
95 |
96 | class ConfigUserList(EsctlLister):
97 | """List all configured users."""
98 |
99 | def take_action(self, parsed_args):
100 | users = [
101 | {
102 | "name": user_name,
103 | "username": user_definition.get("username"),
104 | "password": user_definition.get("password"),
105 | }
106 | for user_name, user_definition in Esctl._config.get("users").items()
107 | ]
108 |
109 | return JSONToCliffFormatter(users).format_for_lister(
110 | columns=[("name"), ("username"), ("password")],
111 | )
112 |
--------------------------------------------------------------------------------
/esctl/cmd/document.py:
--------------------------------------------------------------------------------
1 | from esctl.commands import EsctlShowOne
2 | from esctl.formatter import JSONToCliffFormatter
3 | from esctl.utils import flatten_dict
4 |
5 |
6 | class DocumentGet(EsctlShowOne):
7 | """Retrieves the specified JSON document from an index."""
8 |
9 | def take_action(self, parsed_args):
10 | document = flatten_dict(
11 | self.es.get(format="json", index=parsed_args.index, id=parsed_args.id),
12 | )
13 |
14 | return JSONToCliffFormatter(document).to_show_one(lines=list(document.keys()))
15 |
16 | def get_parser(self, prog_name):
17 | parser = super().get_parser(prog_name)
18 |
19 | parser.add_argument("index", help="The name of the index")
20 | parser.add_argument("id", help="The document ID")
21 |
22 | return parser
23 |
--------------------------------------------------------------------------------
/esctl/cmd/index.py:
--------------------------------------------------------------------------------
1 | from typing import Any
2 |
3 | from esctl.commands import EsctlCommand, EsctlCommandIndex, EsctlLister
4 | from esctl.formatter import JSONToCliffFormatter
5 | from esctl.utils import Color
6 |
7 |
8 | class IndexCreate(EsctlCommand):
9 | """Create an index.
10 |
11 | Read the index configuration as a JSON document from either stdin or a path.
12 | """
13 |
14 | def take_action(self, parsed_args):
15 | configuration = self.read_from_file_or_stdin(parsed_args.configuration)
16 |
17 | self.log.info(f"Creating index {parsed_args.index}")
18 | print(self.es.indices.create(index=parsed_args.index, body=configuration))
19 |
20 | def get_parser(self, prog_name):
21 | parser = super().get_parser(prog_name)
22 | parser.add_argument("index", help="Name of the index to create")
23 | parser.add_argument(
24 | "--configuration",
25 | metavar="PATH",
26 | help="Path to the JSON document containing the index configuration (mapping, aliases, settings)",
27 | )
28 | return parser
29 |
30 |
31 | class IndexList(EsctlLister):
32 | """Returns information about indices: number of primaries and replicas, document counts, disk size, ..."""
33 |
34 | def take_action(self, parsed_args):
35 | indices = self.transform(
36 | self.es.cat.indices(format="json", index=parsed_args.index),
37 | )
38 | return JSONToCliffFormatter(indices).format_for_lister(
39 | columns=[
40 | ("index"),
41 | ("health",),
42 | ("status"),
43 | ("uuid", "UUID"),
44 | ("pri", "Primary"),
45 | ("rep", "Replica"),
46 | ("docs.count"),
47 | ("docs.deleted"),
48 | ("store.size"),
49 | ("pri.store.size", "Primary Store Size"),
50 | ],
51 | )
52 |
53 | def transform(self, indices):
54 | if self.uses_table_formatter():
55 | for idx, indice in enumerate(indices):
56 | if indice.get("health"):
57 | indices[idx]["health"] = Color.colorize(
58 | indice.get("health"),
59 | getattr(Color, indice.get("health").upper()),
60 | )
61 |
62 | if indice.get("status") == "close":
63 | indices[idx]["status"] = Color.colorize(
64 | indice.get("status"),
65 | Color.ITALIC,
66 | )
67 |
68 | return indices
69 |
70 | def get_parser(self, prog_name):
71 | parser = super().get_parser(prog_name)
72 | parser.add_argument(
73 | "index",
74 | help="A comma-separated list of index names to limit the returned information",
75 | nargs="?",
76 | )
77 | return parser
78 |
79 |
80 | class IndexClose(EsctlCommandIndex):
81 | """Close an index."""
82 |
83 | def take_action(self, parsed_args):
84 | self.log.info("Closing index " + parsed_args.index)
85 | print(self.es.indices.close(index=parsed_args.index))
86 |
87 |
88 | class IndexDelete(EsctlCommandIndex):
89 | """Delete an index."""
90 |
91 | def take_action(self, parsed_args):
92 | self.log.info("Deleting index " + parsed_args.index)
93 | print(self.es.indices.delete(index=parsed_args.index))
94 |
95 |
96 | class IndexOpen(EsctlCommandIndex):
97 | """Open an index."""
98 |
99 | def take_action(self, parsed_args):
100 | self.log.info("Opening index " + parsed_args.index)
101 | print(self.es.indices.open(index=parsed_args.index))
102 |
103 |
104 | class IndexReindex(EsctlCommand):
105 | """Reindex a given index into another one."""
106 |
107 | def _build_request_body(self, args: dict[str, Any]) -> dict[str, Any]:
108 | return {
109 | "source": {
110 | "index": args.get("source_index"),
111 | },
112 | "dest": {
113 | "index": args.get("destination_index"),
114 | "version_type": args.get("version_type"),
115 | "op_type": args.get("op_type"),
116 | },
117 | "conflicts": args.get("conflicts"),
118 | }
119 |
120 | def take_action(self, parsed_args):
121 | print(self.es.reindex(body=self._build_request_body(vars(parsed_args))))
122 |
123 | def get_parser(self, prog_name):
124 | parser = super().get_parser(prog_name)
125 | parser.add_argument(
126 | "source_index",
127 | help="Name of the index to index document from",
128 | )
129 | parser.add_argument(
130 | "destination_index",
131 | help="Name of the index to index document to",
132 | )
133 | parser.add_argument(
134 | "--conflicts",
135 | help="Set to `proceed` to continue reindexing even if there are conflicts (default: abort)",
136 | choices=["abort", "proceed"],
137 | default="abort",
138 | )
139 | parser.add_argument(
140 | "--version-type",
141 | help=(
142 | "The versioning to use for the indexing operation (default: internal), "
143 | "valid choices are: `internal`, `external`, `external_gt`, `external_gte`"
144 | ),
145 | choices=["internal", "external", "external_gt", "external_gte"],
146 | default="internal",
147 | )
148 | parser.add_argument(
149 | "--op-type",
150 | help=(
151 | "Set to `create` to only index documents that do not already exist (default: index), "
152 | "valid choices are: `index`, `create`"
153 | ),
154 | choices=["index", "create"],
155 | default="index",
156 | )
157 |
158 | return parser
159 |
--------------------------------------------------------------------------------
/esctl/cmd/logging.py:
--------------------------------------------------------------------------------
1 | from esctl.commands import EsctlCommandLogging
2 |
3 |
4 | class LoggingGet(EsctlCommandLogging):
5 | """Get a logger value."""
6 |
7 | def take_action(self, parsed_args):
8 | persistency = "persistent" if parsed_args.persistent else "transient"
9 | self.log.debug("Persistency is " + persistency)
10 |
11 | if not parsed_args.logger.startswith("logger"):
12 | parsed_args.logger = "logger." + parsed_args.logger
13 |
14 | level = self.cluster_settings.get(parsed_args.logger, persistency=persistency) or ""
15 |
16 | print(f"{parsed_args.logger!s} : {level!s}")
17 |
18 |
19 | class LoggingReset(EsctlCommandLogging):
20 | """Reset a logger value."""
21 |
22 | def take_action(self, parsed_args):
23 | persistency = "persistent" if parsed_args.persistent else "transient"
24 | self.log.debug("Persistency is " + persistency)
25 |
26 | if not parsed_args.logger.startswith("logger"):
27 | parsed_args.logger = "logger." + parsed_args.logger
28 |
29 | print(f"Resetting logger {parsed_args.logger}")
30 | print(
31 | self.cluster_settings.set(parsed_args.logger, None, persistency=persistency),
32 | )
33 |
34 |
35 | class LoggingSet(EsctlCommandLogging):
36 | """Set a logger value."""
37 |
38 | def take_action(self, parsed_args):
39 | persistency = "persistent" if parsed_args.persistent else "transient"
40 | self.log.debug("Persistency is " + persistency)
41 |
42 | if not parsed_args.logger.startswith("logger"):
43 | parsed_args.logger = "logger." + parsed_args.logger
44 |
45 | print(f"Changing logger {parsed_args.logger} to {parsed_args.level}")
46 |
47 | print(
48 | self.cluster_settings.set(
49 | parsed_args.logger,
50 | parsed_args.level,
51 | persistency=persistency,
52 | ),
53 | )
54 |
55 | def get_parser(self, prog_name):
56 | parser = super().get_parser(prog_name)
57 | parser.add_argument(
58 | "level",
59 | metavar="",
60 | help=("Log level"),
61 | choices=["TRACE", "DEBUG", "INFO", "WARN"],
62 | )
63 | return parser
64 |
--------------------------------------------------------------------------------
/esctl/cmd/migration.py:
--------------------------------------------------------------------------------
1 | from esctl.commands import EsctlLister
2 | from esctl.formatter import JSONToCliffFormatter
3 |
4 |
5 | class MigrationDeprecations(EsctlLister):
6 | """Retrieve information about different cluster, node, and index level settings that use
7 | deprecated features that will be removed or changed in a future version.
8 | """
9 |
10 | def take_action(self, parsed_args):
11 | deprecations = self.transform(
12 | self.request("GET", "/_migration/deprecations", None),
13 | )
14 |
15 | return JSONToCliffFormatter(deprecations).format_for_lister(
16 | columns=[
17 | ("kind"),
18 | ("level",),
19 | ("message"),
20 | ("url", "Doc"),
21 | ],
22 | )
23 |
24 | def transform(
25 | self,
26 | deprecations: dict[str, list[dict[str, str]] | dict[str, list[dict[str, str]]]],
27 | ) -> list[dict[str, str]]:
28 | foo: list[dict[str, str]] = []
29 |
30 | # Convert index_settings which is a dict[str, list[dict[str, str]]] to the same format
31 | # as other keys (i.e list[dict[str, str]])
32 | if "index_settings" in deprecations and len(deprecations["index_settings"]) > 0:
33 | flattened_index_settings_deprecations = []
34 | for index_name, deprecation_list in deprecations["index_settings"].items():
35 | for deprecation in deprecation_list:
36 | deprecation["message"] = f"[{index_name}] {deprecation['message']}"
37 | flattened_index_settings_deprecations.append(deprecation)
38 |
39 | deprecations["index_settings"] = flattened_index_settings_deprecations
40 |
41 | for kind, elements in deprecations.items():
42 | for deprecation in elements:
43 | foo.append(
44 | {
45 | "kind": kind,
46 | "level": deprecation["level"],
47 | "message": deprecation["message"],
48 | "url": deprecation["url"],
49 | },
50 | )
51 |
52 | return foo
53 |
--------------------------------------------------------------------------------
/esctl/cmd/node.py:
--------------------------------------------------------------------------------
1 | from esctl.commands import EsctlCommand, EsctlLister, EsctlShowOne
2 | from esctl.formatter import JSONToCliffFormatter
3 | from esctl.utils import Color, flatten_dict
4 |
5 |
6 | class NodeExclude(EsctlCommand):
7 | """Define a list of excluded nodes from routing."""
8 |
9 | def take_action(self, parsed_args):
10 | setting_name = f"cluster.routing.allocation.exclude.{parsed_args.by}"
11 |
12 | if parsed_args.list is None:
13 | setting = self.cluster_settings.get(setting_name, persistency="transient")
14 |
15 | if setting.value is None:
16 | setting.value = ""
17 |
18 | self.print_success(setting.value)
19 |
20 | else:
21 | self.print_output(
22 | "Changing node exclusion list"
23 | f" ({Color.colorize(setting_name, Color.ITALIC)}) to : {Color.colorize(parsed_args.list, Color.ITALIC)}",
24 | )
25 | self.print_success(
26 | self.cluster_settings.set(
27 | setting_name,
28 | parsed_args.list,
29 | persistency="transient",
30 | ),
31 | )
32 |
33 | def get_parser(self, prog_name):
34 | parser = super().get_parser(prog_name)
35 |
36 | parser.add_argument(
37 | "--by",
38 | help=(
39 | "Attribute to filter by (default: _ip). "
40 | "This can be a custom string or one of the built-ins: "
41 | " _ip, _name, _host."
42 | ),
43 | default="_ip",
44 | )
45 | parser.add_argument(
46 | "list",
47 | metavar="",
48 | nargs="?",
49 | help=("Comma-separated list of values nodes should have to be excluded from shard allocation"),
50 | )
51 |
52 | return parser
53 |
54 |
55 | class NodeHotThreads(EsctlCommand):
56 | """Print hot threads on each nodes."""
57 |
58 | def take_action(self, parsed_args):
59 | print(self.es.nodes.hot_threads(doc_type=parsed_args.type))
60 |
61 | def get_parser(self, prog_name):
62 | parser = super().get_parser(prog_name)
63 | parser.add_argument(
64 | "--type",
65 | help=("The type to sample (default: cpu), valid choices are: 'cpu', 'wait', 'block'"),
66 | choices=["cpu", "wait", "block"],
67 | default="cpu",
68 | )
69 | return parser
70 |
71 |
72 | class NodeList(EsctlLister):
73 | """List nodes."""
74 |
75 | def take_action(self, parsed_args):
76 | return JSONToCliffFormatter(self.es.cat.nodes(format="json")).format_for_lister(
77 | columns=[
78 | ("ip", "IP"),
79 | ("heap.percent",),
80 | ("ram.percent",),
81 | ("cpu"),
82 | ("load_1m"),
83 | ("load_5m"),
84 | ("load_15m"),
85 | ("node.role", "Role"),
86 | ("master"),
87 | ("name"),
88 | ],
89 | )
90 |
91 |
92 | class NodeStats(EsctlShowOne):
93 | """Returns statistical information about nodes in the cluster."""
94 |
95 | def take_action(self, parsed_args):
96 | stats = self.es.nodes.stats(
97 | format="json",
98 | node_id=parsed_args.node,
99 | metric=parsed_args.metric,
100 | index_metric=parsed_args.index_metric,
101 | level=parsed_args.level,
102 | ).get("nodes")
103 |
104 | if parsed_args.jmespath is not None:
105 | path = self.jmespath_search(parsed_args.jmespath, stats)
106 |
107 | return (tuple(["Result"]), tuple([path]))
108 |
109 | stats = self.transform(stats)
110 |
111 | return JSONToCliffFormatter(
112 | stats,
113 | pretty_key=not parsed_args.no_pretty,
114 | ).to_show_one(lines=list(stats.keys()))
115 |
116 | def transform(self, raw_stats):
117 | if self.uses_table_formatter():
118 | return flatten_dict(raw_stats)
119 |
120 | return raw_stats
121 |
122 | def get_parser(self, prog_name):
123 | parser = super().get_parser(prog_name)
124 | parser.add_argument(
125 | "--node",
126 | help="A comma-separated list of node IDs or names to limit the returned information.",
127 | default=None,
128 | )
129 | parser.add_argument(
130 | "--metric",
131 | help=(
132 | "A comma-separated list of metrics you wish returned. Valid choices are: "
133 | "_all, breaker, fs, http, indices, jvm, os, process, "
134 | "thread_pool, transport, discovery, indexing_pressure."
135 | ),
136 | default=None,
137 | )
138 | parser.add_argument(
139 | "--index-metric",
140 | help=(
141 | "A comma-separated list of indices metrics you wish returned. "
142 | "Isn’t used if indices (or all) metric isn’t specified."
143 | "Valid choices are: "
144 | "_all, completion, docs, fielddata, query_cache, flush, get, indexing, merge, "
145 | "request_cache, refresh, search, segments, store, warmer, suggest."
146 | ),
147 | default=None,
148 | )
149 | parser.add_argument(
150 | "--level",
151 | help=(
152 | "Return indices stats aggregated at index, node or shard level. "
153 | "Valid choices are: indices, node, shards."
154 | ),
155 | default="node",
156 | )
157 |
158 | return parser
159 |
--------------------------------------------------------------------------------
/esctl/cmd/raw.py:
--------------------------------------------------------------------------------
1 | import json
2 |
3 | from esctl.commands import EsctlCommand
4 |
5 |
6 | class RawCommand(EsctlCommand):
7 | """Performs a raw HTTP call. Useful when esctl doesn't provide a nice interface for a specific route."""
8 |
9 | def take_action(self, parsed_args):
10 | body: str = parsed_args.body
11 |
12 | if parsed_args.body is not None and parsed_args.body.startswith("@"):
13 | body = self.read_from_file_or_stdin(parsed_args.body[1:])
14 |
15 | print(
16 | json.dumps(
17 | self.request(verb=parsed_args.verb, route=parsed_args.route, body=body),
18 | ),
19 | )
20 |
21 | def get_parser(self, prog_name):
22 | parser = super().get_parser(prog_name)
23 | parser.add_argument(
24 | "-d",
25 | "--data",
26 | help=(
27 | "Specify the body to send. If you start the data with the letter @, "
28 | "the rest should be a file name to read the data from, or - if you want curl "
29 | "to read the data from stdin."
30 | ),
31 | metavar="BODY",
32 | dest="body",
33 | )
34 | parser.add_argument(
35 | "-X",
36 | "--request",
37 | help="Specify the HTTP verb to use.",
38 | default="GET",
39 | metavar="VERB",
40 | dest="verb",
41 | )
42 | parser.add_argument(
43 | "route",
44 | help="The route to call for.",
45 | )
46 | return parser
47 |
--------------------------------------------------------------------------------
/esctl/cmd/repository.py:
--------------------------------------------------------------------------------
1 | from esctl.commands import EsctlCommand, EsctlLister, EsctlShowOne
2 | from esctl.formatter import JSONToCliffFormatter
3 |
4 |
5 | class RepositoryList(EsctlLister):
6 | """Returns information about snapshot repositories registered in the cluster."""
7 |
8 | def take_action(self, parsed_args):
9 | repositories = self.es.cat.repositories(format="json")
10 |
11 | return JSONToCliffFormatter(repositories).format_for_lister(
12 | columns=[("id"), ("type")],
13 | )
14 |
15 |
16 | class RepositoryShow(EsctlShowOne):
17 | """Returns information about a repository."""
18 |
19 | def take_action(self, parsed_args):
20 | repository = self.es.snapshot.get_repository(
21 | repository=parsed_args.repository,
22 | format="json",
23 | ).get(parsed_args.repository)
24 |
25 | return JSONToCliffFormatter(repository).to_show_one(
26 | lines=list(repository.keys()),
27 | )
28 |
29 | def get_parser(self, prog_name):
30 | parser = super().get_parser(prog_name)
31 | parser.add_argument(
32 | "repository",
33 | help=("A comma-separated list of repository names"),
34 | )
35 | return parser
36 |
37 |
38 | class RepositoryVerify(EsctlCommand):
39 | """Verifies a repository."""
40 |
41 | def take_action(self, parsed_args):
42 | repository = self.es.snapshot.verify_repository(
43 | repository=parsed_args.repository,
44 | format="json",
45 | )
46 |
47 | print(repository)
48 |
49 | def get_parser(self, prog_name):
50 | parser = super().get_parser(prog_name)
51 | parser.add_argument("repository", help=("A repository name"))
52 | return parser
53 |
--------------------------------------------------------------------------------
/esctl/cmd/roles.py:
--------------------------------------------------------------------------------
1 | from esctl.commands import EsctlLister
2 | from esctl.formatter import JSONToCliffFormatter
3 |
4 |
5 | class SecurityRolesGet(EsctlLister):
6 | """Retrieves roles in the native realm."""
7 |
8 | def take_action(self, parsed_args):
9 | roles = self.transform(
10 | self._sort_and_order_dict(
11 | self.es.security.get_role(format="json", name=parsed_args.roles),
12 | ),
13 | )
14 |
15 | return JSONToCliffFormatter(roles).format_for_lister(
16 | columns=[
17 | ("role",),
18 | ("cluster", "Cluster-level permissions"),
19 | ("indices", "Index-level permissions"),
20 | ("applications",),
21 | ("run_as",),
22 | ],
23 | )
24 |
25 | def transform(self, raw_roles):
26 | roles = []
27 |
28 | for role_name, role_definition in raw_roles.items():
29 | role_definition["cluster"] = ", ".join(role_definition.get("cluster"))
30 | roles.append({"role": role_name, **role_definition})
31 |
32 | return roles
33 |
34 | def get_parser(self, prog_name):
35 | parser = super().get_parser(prog_name)
36 |
37 | parser.add_argument(
38 | "roles",
39 | help=("A comma-separated list of roles"),
40 | nargs="?",
41 | default=None,
42 | )
43 |
44 | return parser
45 |
--------------------------------------------------------------------------------
/esctl/cmd/settings.py:
--------------------------------------------------------------------------------
1 | import collections
2 |
3 | from esctl.commands import EsctlCommandIndex, EsctlListerIndexSetting, EsctlShowOne
4 | from esctl.formatter import JSONToCliffFormatter
5 | from esctl.settings import ClusterSettings, IndexSettings
6 | from esctl.utils import Color
7 |
8 |
9 | class AbstractClusterSettings(EsctlShowOne):
10 | cluster_settings: ClusterSettings
11 |
12 | def _settings_set(self, setting: str, value: str, persistency: str):
13 | self.log.debug("Persistency is " + persistency)
14 |
15 | self.log.debug(
16 | self.cluster_settings.set(setting, value, persistency=persistency),
17 | )
18 |
19 | def _settings_get(self, setting: str):
20 | s = self.cluster_settings.mget(setting)
21 |
22 | return JSONToCliffFormatter(
23 | {
24 | "transient": s.get("transient").value,
25 | "persistent": s.get("persistent").value,
26 | "defaults": s.get("defaults").value,
27 | },
28 | ).to_show_one(
29 | lines=[("transient"), ("persistent"), ("defaults")],
30 | none_as="" if self.uses_table_formatter() else None,
31 | )
32 |
33 | def get_parser(self, prog_name):
34 | parser = super().get_parser(prog_name)
35 |
36 | parser.add_argument("setting", metavar="", help=("Setting"))
37 |
38 | return parser
39 |
40 |
41 | class ClusterSettingsGet(AbstractClusterSettings):
42 | """Get a setting value."""
43 |
44 | def take_action(self, parsed_args):
45 | return self._settings_get(parsed_args.setting)
46 |
47 |
48 | class ClusterSettingsList(EsctlShowOne):
49 | """[Experimental] List available settings."""
50 |
51 | cluster_settings: ClusterSettings
52 |
53 | def take_action(self, parsed_args):
54 | default_settings = {}
55 | settings_list = self.cluster_settings.list_().get("defaults")
56 |
57 | for setting_name, setting_value in settings_list.items():
58 | if type(setting_value).__name__ == "list":
59 | if len(setting_value) > 0:
60 | setting_value = ",\n".join(setting_value)
61 | else:
62 | setting_value = "[]"
63 |
64 | default_settings[setting_name] = setting_value
65 |
66 | return (tuple(default_settings.keys()), tuple(default_settings.values()))
67 |
68 |
69 | class ClusterSettingsReset(AbstractClusterSettings):
70 | """Reset a setting value."""
71 |
72 | def take_action(self, parsed_args):
73 | self._settings_set(
74 | parsed_args.setting,
75 | None,
76 | persistency="persistent" if parsed_args.persistent else "transient",
77 | )
78 | return self._settings_get(parsed_args.setting)
79 |
80 | def get_parser(self, prog_name):
81 | parser = super().get_parser(prog_name)
82 |
83 | persistency_group = parser.add_mutually_exclusive_group()
84 | persistency_group.add_argument(
85 | "--transient",
86 | action="store_true",
87 | help=("Set setting as transient (default)"),
88 | )
89 | persistency_group.add_argument(
90 | "--persistent",
91 | action="store_true",
92 | help=("Set setting as persistent"),
93 | )
94 |
95 | return parser
96 |
97 |
98 | class ClusterSettingsSet(AbstractClusterSettings):
99 | """Set a setting value."""
100 |
101 | def take_action(self, parsed_args):
102 | self._settings_set(
103 | parsed_args.setting,
104 | parsed_args.value,
105 | persistency="persistent" if parsed_args.persistent else "transient",
106 | )
107 | return self._settings_get(parsed_args.setting)
108 |
109 | def get_parser(self, prog_name):
110 | parser = super().get_parser(prog_name)
111 |
112 | persistency_group = parser.add_mutually_exclusive_group()
113 | persistency_group.add_argument(
114 | "--transient",
115 | action="store_true",
116 | help=("Set setting as transient (default)"),
117 | )
118 | persistency_group.add_argument(
119 | "--persistent",
120 | action="store_true",
121 | help=("Set setting as persistent"),
122 | )
123 |
124 | parser.add_argument("value", metavar="", help=("Setting value"))
125 |
126 | return parser
127 |
128 |
129 | class IndexSettingsGet(EsctlListerIndexSetting):
130 | """Get an index-level setting value."""
131 |
132 | def retrieve_setting(self, setting_name, index):
133 | if not setting_name.startswith("index."):
134 | setting_name = f"index.{setting_name}"
135 |
136 | raw_settings = self.index_settings.get(index, setting_name)
137 | settings = []
138 |
139 | for index_name, settings_list in raw_settings.items():
140 | for setting in settings_list:
141 | if setting.value is not None:
142 | if setting.persistency == "defaults":
143 | value = f"{setting.value} ({Color.colorize('default', Color.ITALIC)})"
144 | else:
145 | value = setting.value
146 |
147 | settings.append(
148 | {"index": index_name, "setting": setting.name, "value": value},
149 | )
150 |
151 | return settings
152 |
153 | def take_action(self, parsed_args):
154 | return JSONToCliffFormatter(
155 | self.retrieve_setting(parsed_args.setting, parsed_args.index),
156 | ).format_for_lister(columns=[("index",), ("setting",), ("value",)])
157 |
158 |
159 | class IndexSettingsList(EsctlShowOne):
160 | """[Experimental] List available settings in an index."""
161 |
162 | index_settings: IndexSettings
163 |
164 | def take_action(self, parsed_args):
165 | settings = {}
166 | sample_index_name = self.es.cat.indices(format="json", h="index")[0].get(
167 | "index",
168 | )
169 |
170 | settings_list = self.index_settings.list_(sample_index_name)
171 | settings_list = collections.OrderedDict(
172 | sorted(
173 | {
174 | **settings_list.get("settings"),
175 | **settings_list.get("defaults"),
176 | }.items(),
177 | ),
178 | )
179 |
180 | for setting_name, setting_value in settings_list.items():
181 | if type(setting_value).__name__ == "list":
182 | if len(setting_value) > 0:
183 | setting_value = ",\n".join(setting_value)
184 | else:
185 | setting_value = "[]"
186 |
187 | settings[setting_name] = setting_value
188 |
189 | return (tuple(settings.keys()), tuple(settings.values()))
190 |
191 |
192 | class IndexSettingsSet(EsctlCommandIndex):
193 | """Set an index-level setting value."""
194 |
195 | index_settings: IndexSettings
196 |
197 | def take_action(self, parsed_args):
198 | setting_name = parsed_args.setting
199 |
200 | if not setting_name.startswith("index."):
201 | setting_name = f"index.{setting_name}"
202 |
203 | print(
204 | f"Changing {Color.colorize(setting_name, Color.ITALIC)} to {Color.colorize(parsed_args.value, Color.ITALIC)} in index {Color.colorize(parsed_args.index, Color.ITALIC)}",
205 | )
206 |
207 | print(
208 | self.index_settings.set(setting_name, parsed_args.value, parsed_args.index),
209 | )
210 |
211 | def get_parser(self, prog_name):
212 | parser = super().get_parser(prog_name)
213 | parser.add_argument("setting", metavar="", help=("Setting"))
214 | parser.add_argument("value", metavar="", help=("Setting value"))
215 | return parser
216 |
--------------------------------------------------------------------------------
/esctl/cmd/snapshot.py:
--------------------------------------------------------------------------------
1 | from esctl.commands import EsctlLister
2 | from esctl.formatter import JSONToCliffFormatter
3 | from esctl.utils import Color
4 |
5 |
6 | class SnapshotList(EsctlLister):
7 | """Returns all snapshots in a specific repository."""
8 |
9 | def take_action(self, parsed_args):
10 | snapshots = self.transform(
11 | self.es.cat.snapshots(repository=parsed_args.repository, format="json"),
12 | )
13 |
14 | return JSONToCliffFormatter(snapshots).format_for_lister(
15 | columns=[
16 | ("id"),
17 | ("status"),
18 | ("start_time"),
19 | ("end_time"),
20 | ("duration"),
21 | ("indices"),
22 | ("successful_shards"),
23 | ("failed_shards"),
24 | ("total_shards"),
25 | ],
26 | )
27 |
28 | def transform(self, snapshots):
29 | for idx in range(len(snapshots)):
30 | if snapshots[idx].get("status") == "PARTIAL":
31 | snapshots[idx]["status"] = Color.colorize(
32 | snapshots[idx].get("status"),
33 | Color.YELLOW,
34 | )
35 | snapshots[idx]["failed_shards"] = Color.colorize(
36 | snapshots[idx].get("failed_shards"),
37 | Color.RED,
38 | )
39 | elif snapshots[idx].get("status") == "IN_PROGRESS":
40 | snapshots[idx]["status"] = Color.colorize(
41 | snapshots[idx].get("status"),
42 | Color.CYAN,
43 | )
44 |
45 | return snapshots
46 |
47 | def get_parser(self, prog_name):
48 | parser = super().get_parser(prog_name)
49 | parser.add_argument(
50 | "repository",
51 | help=("Comma-separated list or wildcard expression of repository names used to limit the request."),
52 | )
53 |
54 | return parser
55 |
--------------------------------------------------------------------------------
/esctl/cmd/task.py:
--------------------------------------------------------------------------------
1 | import datetime
2 |
3 | from esctl.commands import EsctlLister
4 | from esctl.formatter import JSONToCliffFormatter
5 |
6 |
7 | class TaskList(EsctlLister):
8 | """Returns a list of tasks."""
9 |
10 | def take_action(self, parsed_args):
11 | tasks = self.transform(
12 | self.es.tasks.list(
13 | actions=parsed_args.actions,
14 | detailed=parsed_args.detailed,
15 | parent_task_id=parsed_args.parent_task_id,
16 | ).get("nodes"),
17 | )
18 |
19 | return JSONToCliffFormatter(tasks).format_for_lister(
20 | columns=[
21 | ("name"),
22 | ("node"),
23 | ("id"),
24 | ("type"),
25 | ("action"),
26 | ("start_time_human_readable"),
27 | ("running_time_in_nanos"),
28 | ("parent_task_id"),
29 | ("cancellable"),
30 | ("headers"),
31 | ],
32 | )
33 |
34 | def convert_timestamp_in_ms_to_human_readable(self, time_in_ms: int) -> str:
35 | miliseconds = str(round((time_in_ms / 1000) % 1, 3)).split(".")[1]
36 | return (
37 | datetime.datetime.utcfromtimestamp(time_in_ms / 1000).strftime(
38 | "%Y-%m-%d %H:%M:%S",
39 | )
40 | + f".{miliseconds}"
41 | )
42 |
43 | def transform(self, nodes):
44 | tasks = []
45 |
46 | for _, node_definition in nodes.items():
47 | for task_name, task in node_definition.get("tasks").items():
48 | task["name"] = task_name
49 |
50 | if "parent_task_id" not in task:
51 | task["parent_task_id"] = ""
52 |
53 | task["start_time_human_readable"] = self.convert_timestamp_in_ms_to_human_readable(
54 | task.get("start_time_in_millis"),
55 | )
56 |
57 | tasks.append(task)
58 |
59 | return tasks
60 |
61 | def get_parser(self, prog_name):
62 | parser = super().get_parser(prog_name)
63 | parser.add_argument(
64 | "--actions",
65 | help="A comma-separated list of actions that should be returned",
66 | )
67 | parser.add_argument(
68 | "--detailed",
69 | help="Return detailed task information (default: false)",
70 | action="store_true",
71 | default=False,
72 | )
73 | parser.add_argument(
74 | "--parent_task_id",
75 | help="Return tasks with specified parent task id (node_id:task_number)",
76 | )
77 |
78 | return parser
79 |
--------------------------------------------------------------------------------
/esctl/cmd/users.py:
--------------------------------------------------------------------------------
1 | from esctl.commands import EsctlLister
2 | from esctl.formatter import JSONToCliffFormatter
3 |
4 |
5 | class SecurityUsersGet(EsctlLister):
6 | """Retrieves information about users in the native realm and built-in users."""
7 |
8 | def take_action(self, parsed_args):
9 | users = self.transform(
10 | self._sort_and_order_dict(
11 | self.es.security.get_user(format="json", username=parsed_args.username),
12 | ),
13 | )
14 |
15 | return JSONToCliffFormatter(users).format_for_lister(
16 | columns=[
17 | ("username",),
18 | ("roles",),
19 | ("fullname",),
20 | ("email",),
21 | ("metadata",),
22 | ("enabled",),
23 | ],
24 | )
25 |
26 | def transform(self, raw_users):
27 | users: list[dict[str, str]] = []
28 | for _, user in raw_users.items():
29 | user["roles"] = ", ".join(user.get("roles"))
30 | users.append(user)
31 |
32 | return users
33 |
34 | def get_parser(self, prog_name):
35 | parser = super().get_parser(prog_name)
36 |
37 | parser.add_argument(
38 | "username",
39 | help=("A comma-separated list of username"),
40 | nargs="?",
41 | default=None,
42 | )
43 |
44 | return parser
45 |
--------------------------------------------------------------------------------
/esctl/commands.py:
--------------------------------------------------------------------------------
1 | import logging
2 | import os
3 | import sys
4 | from typing import Any
5 |
6 | import jmespath
7 | from cliff.command import Command
8 | from cliff.lister import Lister
9 | from cliff.show import ShowOne
10 |
11 | from esctl.elasticsearch import Client
12 | from esctl.settings import ClusterSettings, IndexSettings
13 | from esctl.utils import Color
14 |
15 |
16 | class EsctlCommon:
17 | log = logging.getLogger(__name__)
18 | es = Client().es
19 | cluster_settings = ClusterSettings()
20 | index_settings = IndexSettings()
21 |
22 | def _sort_and_order_dict(self, dct):
23 | return {e[0]: e[1] for e in sorted(dct.items())}
24 |
25 | def print_success(self, message):
26 | print("{} {}".format(Color.colorize("==>", Color.GREEN), message))
27 |
28 | def print_output(self, message):
29 | print(f"{message}")
30 |
31 | def uses_table_formatter(self):
32 | return self.formatter.__class__.__name__ == "TableFormatter"
33 |
34 | def read_from_file_or_stdin(self, path: str) -> str:
35 | """Read some content from a file if the path is defined otherwise read from stdin."""
36 | if path is not None:
37 | with open(os.path.expanduser(path)) as reader:
38 | return reader.read()
39 | else:
40 | return sys.stdin.read()
41 |
42 | def request(
43 | self,
44 | verb: str,
45 | route: str,
46 | body: dict[Any, Any] | None,
47 | ) -> dict[Any, Any]:
48 | return self.es.transport.perform_request(verb.upper(), route, body=body)
49 |
50 | def objects_list_to_flat_dict(self, lst: list[dict[str, Any]]) -> dict[str, Any]:
51 | """Convert a list of dict to a flattened dict with full name.
52 |
53 | :Example:
54 | node.jvm.versions = [{"version":"12.0.1","vm_name":"OpenJDK 64-Bit Server VM","vm_version":"12.0.1+12",
55 | "vm_vendor":"Oracle Corporation","bundled_jdk":true,"using_bundled_jdk":true,"count":1}] # noqa: E501
56 | becomes
57 | {
58 | "node.jvm.versions[0].version": "12.0.1",
59 | "node.jvm.versions[0].vm_name": "OpenJDK 64-Bit Server VM"
60 | }
61 |
62 | :param lst: A list of dict to flatten
63 | :paramtype lst: list
64 | :return: A dict
65 | :rtype: dict
66 | """
67 | flat_dict = {}
68 | for index, element in enumerate(lst):
69 | for attribute, value in element.items():
70 | flat_dict[f"[{index}].{attribute}"] = value
71 |
72 | return flat_dict
73 |
74 | def _delete_and_merge_inner_dict_into_parent(
75 | self,
76 | parent_dict: dict[str, Any],
77 | key: str,
78 | ) -> dict[str, Any]:
79 | """Merge a inner dictionnary into it's parent.
80 |
81 | :Example:
82 | {
83 | "name" : "prod-monolith-us-searchill-es-01",
84 | "cluster_name" : "prod-lumsites-cluster",
85 | "cluster_uuid" : "Z0FeT4oVSw2GQ3bMs0vEZw",
86 | "version" : {
87 | "number" : "7.0.1",
88 | "build_flavor" : "default",
89 | "build_type" : "deb",
90 | "build_hash" : "e4efcb5",
91 | "build_date" : "2019-04-29T12:56:03.145736Z",
92 | "build_snapshot" : false,
93 | "lucene_version" : "8.0.0",
94 | "minimum_wire_compatibility_version" : "6.7.0",
95 | "minimum_index_compatibility_version" : "6.0.0-beta1"
96 | },
97 | "tagline" : "You Know, for Search"
98 | }
99 | becomes
100 | {
101 | "name" : "prod-monolith-us-searchill-es-01",
102 | "cluster_name" : "prod-lumsites-cluster",
103 | "cluster_uuid" : "Z0FeT4oVSw2GQ3bMs0vEZw",
104 | "version.number" : "7.0.1",
105 | "version.build_flavor" : "default",
106 | "version.build_type" : "deb",
107 | "version.build_hash" : "e4efcb5",
108 | "version.build_date" : "2019-04-29T12:56:03.145736Z",
109 | "version.build_snapshot" : false,
110 | "version.lucene_version" : "8.0.0",
111 | "version.minimum_wire_compatibility_version" : "6.7.0",
112 | "version.minimum_index_compatibility_version" : "6.0.0-beta1",
113 | "tagline" : "You Know, for Search"
114 | }
115 |
116 | :param parent_dict: The parent dict containing an inner dict
117 | :paramtype parent_dict: dict
118 | :param key: The key where the inner dict stands
119 | :paramtype key: str
120 | :return: The parent dict with the merged inner dict
121 | :rtype: dict
122 | """
123 | for k, v in parent_dict.get(key).items():
124 | parent_dict[f"{key}.{k}"] = v
125 | del parent_dict[key]
126 |
127 | return parent_dict
128 |
129 |
130 | class EsctlCommand(Command, EsctlCommon):
131 | """Simple command to run. Doesn’t expect any output."""
132 |
133 |
134 | class EsctlCommandWithPersistency(EsctlCommand):
135 | """Add mutually exclusive arguments `transient` and `persistent` to
136 | be used when working with settings, logging, etc...
137 | """
138 |
139 | def get_parser(self, prog_name):
140 | parser = super().get_parser(prog_name)
141 | persistency_group = parser.add_mutually_exclusive_group()
142 | persistency_group.add_argument(
143 | "--transient",
144 | action="store_true",
145 | help=("Set setting as transient (default)"),
146 | )
147 | persistency_group.add_argument(
148 | "--persistent",
149 | action="store_true",
150 | help=("Set setting as persistent"),
151 | )
152 | return parser
153 |
154 |
155 | class EsctlCommandIndex(EsctlCommand):
156 | def get_parser(self, prog_name):
157 | parser = super().get_parser(prog_name)
158 | parser.add_argument(
159 | "index",
160 | help=("Comma-separated list or wildcard expression of index names used to limit the request."),
161 | )
162 | return parser
163 |
164 |
165 | class EsctlCommandLogging(EsctlCommandWithPersistency):
166 | def get_parser(self, prog_name):
167 | parser = super().get_parser(prog_name)
168 | parser.add_argument("logger", metavar="", help=("Logger"))
169 | return parser
170 |
171 |
172 | class EsctlCommandSetting(EsctlCommandWithPersistency):
173 | def get_parser(self, prog_name):
174 | parser = super().get_parser(prog_name)
175 | parser.add_argument("setting", metavar="", help=("Setting"))
176 | return parser
177 |
178 |
179 | class EsctlLister(Lister, EsctlCommon):
180 | """Expect a list of elements in order to create a multi-columns table."""
181 |
182 |
183 | class EsctlListerIndexSetting(EsctlLister):
184 | def get_parser(self, prog_name):
185 | parser = super().get_parser(prog_name)
186 | parser.add_argument("index", metavar="", help=("Index"))
187 | parser.add_argument(
188 | "setting",
189 | metavar="",
190 | help=("Setting"),
191 | nargs="?",
192 | default="*",
193 | )
194 | return parser
195 |
196 |
197 | class EsctlShowOne(ShowOne, EsctlCommon):
198 | """Expect a key-value list to create a two-columns table."""
199 |
200 | def jmespath_search(self, expression, data, options=None):
201 | return jmespath.search(expression, data, options=options)
202 |
203 | def get_parser(self, prog_name):
204 | parser = super().get_parser(prog_name)
205 | parser.add_argument(
206 | "--no-pretty",
207 | action="store_true",
208 | help=("Don't format keys. (Like `ingest.total.count` into `Ingest Total Count`)"),
209 | )
210 | parser.add_argument(
211 | "--jmespath",
212 | help=("[Experimental] Execute a JMESPath query on the response. See https://jmespath.org for help."),
213 | )
214 | return parser
215 |
--------------------------------------------------------------------------------
/esctl/config.py:
--------------------------------------------------------------------------------
1 | import logging
2 | import os
3 | import sys
4 | from collections import OrderedDict
5 | from pathlib import Path
6 | from typing import Any
7 |
8 | import cerberus
9 | import yaml
10 |
11 | from esctl.utils import setup_yaml
12 |
13 |
14 | class Context:
15 | def __init__(self, name, user, cluster, settings, **kwargs):
16 | super().__init__()
17 | self.log = logging.getLogger(__name__)
18 | self.name = name
19 | self.user = user
20 | self.cluster = cluster
21 | self.settings = settings
22 |
23 | for k, v in kwargs.items():
24 | setattr(self, k, v)
25 |
26 | def __repr__(self):
27 | return f""
28 |
29 |
30 | class ConfigFileParser:
31 | log = logging.getLogger(__name__)
32 |
33 | def __init__(self):
34 | self.users = {}
35 | self.contexts = {}
36 | self.clusters = {}
37 | self.settings = {}
38 |
39 | def write_config_file(self, content):
40 | setup_yaml()
41 |
42 | with open(self.path, "w") as config_file:
43 | yaml.dump(content, config_file, default_flow_style=False, width=500)
44 |
45 | def _create_default_config_file(self) -> "OrderedDict[str, Any]":
46 | self.log.info(
47 | f"{self.path} config file does not exists. Creating a default one...",
48 | )
49 | default_config = OrderedDict(
50 | {
51 | "settings": {},
52 | "clusters": {"localhost": {"servers": ["http://localhost:9200"]}},
53 | "users": {},
54 | "contexts": {"localhost": {"cluster": "localhost"}},
55 | "default-context": "localhost",
56 | },
57 | )
58 |
59 | self.write_config_file(default_config)
60 |
61 | return default_config
62 |
63 | def _ensure_config_file_is_valid(self, document):
64 | settings_schema = {
65 | "no_check_certificate": {"type": "boolean"},
66 | "max_retries": {"type": "integer"},
67 | "timeout": {"type": "integer"},
68 | }
69 |
70 | external_credentials_schema = {
71 | "type": "dict",
72 | "schema": {
73 | "command": {
74 | "type": "dict",
75 | "schema": {
76 | "run": {
77 | "type": "string",
78 | "required": True,
79 | },
80 | },
81 | },
82 | },
83 | }
84 |
85 | schema = {
86 | "settings": {"type": "dict", "schema": settings_schema},
87 | "clusters": {
88 | "type": "dict",
89 | "required": True,
90 | "keysrules": {},
91 | "valuesrules": {
92 | "type": "dict",
93 | "schema": {
94 | "servers": {"type": "list"},
95 | "settings": {"type": "dict", "schema": settings_schema},
96 | },
97 | },
98 | },
99 | "users": {
100 | "type": "dict",
101 | "keysrules": {"type": "string"},
102 | "valuesrules": {
103 | "type": "dict",
104 | "schema": {
105 | "username": {"type": "string"},
106 | "password": {"type": "string"},
107 | "external_username": external_credentials_schema,
108 | "external_password": external_credentials_schema,
109 | },
110 | },
111 | },
112 | "contexts": {
113 | "type": "dict",
114 | "required": True,
115 | "keysrules": {"type": "string"},
116 | "valuesrules": {
117 | "type": "dict",
118 | "schema": {
119 | "cluster": {"type": "string"},
120 | "user": {"type": "string"},
121 | "pre_commands": {
122 | "type": "list",
123 | "schema": {
124 | "type": "dict",
125 | "schema": {
126 | "command": {"type": "string"},
127 | "wait_for_exit": {"type": "boolean"},
128 | "wait_for_output": {"type": "string"},
129 | },
130 | },
131 | },
132 | },
133 | },
134 | },
135 | "default-context": {"type": "string"},
136 | }
137 | cerberus_validator = cerberus.Validator(schema)
138 |
139 | if not cerberus_validator.validate(document):
140 | for root_error in cerberus_validator._errors:
141 | for inner_error in root_error.info[0]:
142 | self.log.error(
143 | "Unknown configuration : {}".format(
144 | ".".join(inner_error.document_path),
145 | ),
146 | )
147 | self.log.error(
148 | f"Invalid type or schema for configuration field '{root_error.field}'."
149 | f" Should be {root_error.constraint}. Got '{root_error.value}'",
150 | )
151 |
152 | raise SyntaxError(f"{self.path} doesn't match expected schema")
153 |
154 | def load_configuration(self, path: str = "~/.esctlrc") -> OrderedDict:
155 | self.path = os.path.expanduser(path)
156 | self.log.debug(f"Trying to load config file : {self.path}")
157 | expected_config_blocks = [
158 | "clusters",
159 | "contexts",
160 | "default-context",
161 | "settings",
162 | "users",
163 | ]
164 |
165 | if Path(self.path).is_file():
166 | with open(self.path) as config_file:
167 | try:
168 | raw_config_file = OrderedDict(yaml.safe_load(config_file))
169 | except yaml.YAMLError as err:
170 | self.log.critical(f"Cannot read YAML from {self.path}")
171 | self.log.critical(str(err.problem) + str(err.problem_mark))
172 | sys.exit(1)
173 | else:
174 | raw_config_file = self._create_default_config_file()
175 |
176 | try:
177 | self._ensure_config_file_is_valid(raw_config_file)
178 | except SyntaxError:
179 | sys.exit(1)
180 |
181 | for config_block in expected_config_blocks:
182 | self.__setattr__(config_block, raw_config_file.get(config_block))
183 |
184 | self.log.debug(
185 | f"Loaded {config_block}: {self.__getattribute__(config_block)}",
186 | )
187 |
188 | return raw_config_file
189 |
190 | def get_context_informations(self, context_name: str):
191 | raw_context = self.contexts.get(context_name)
192 | user = self.users.get(raw_context.get("user"))
193 | cluster = self.clusters.get(raw_context.get("cluster"))
194 |
195 | if cluster is None:
196 | self.log.error(
197 | f"Malformed context `{context_name}` in configuration file : "
198 | f"it requires a cluster named `{raw_context.get('cluster')}` but none could be found ! "
199 | f"Here are the clusters I know : {', '.join(list(self.clusters.keys()))}",
200 | )
201 | sys.exit(1)
202 |
203 | # Merge global settings and per-cluster settings.
204 | # Cluster-level settings override global settings
205 | if "settings" in cluster:
206 | settings = {**self.settings, **cluster.get("settings")}
207 | else:
208 | settings = {**self.settings}
209 |
210 | pre_commands: list[dict] = []
211 | if "pre_commands" in raw_context:
212 | pre_commands = raw_context.get("pre_commands")
213 |
214 | for i in range(len(pre_commands)):
215 | pre_command = pre_commands[i]
216 | pre_command = {
217 | "wait_for_exit": True,
218 | "wait_for_output": "",
219 | **pre_command,
220 | }
221 |
222 | pre_commands[i] = pre_command
223 |
224 | context = Context(
225 | context_name,
226 | user,
227 | cluster,
228 | settings,
229 | pre_commands=pre_commands,
230 | )
231 |
232 | return context
233 |
234 | def create_context(self, context_name: str = None) -> Context:
235 | if context_name:
236 | self.log.debug(f"Using provided context : {context_name}")
237 | else:
238 | context_name = self.__getattribute__("default-context")
239 | self.log.debug("No context provided. Using default context")
240 |
241 | try:
242 | context = self.get_context_informations(context_name)
243 | self.log.debug(context)
244 | except AttributeError:
245 | self.log.fatal(f"Cannot load context '{context_name}'.")
246 | sys.exit(1)
247 |
248 | return context
249 |
--------------------------------------------------------------------------------
/esctl/elasticsearch.py:
--------------------------------------------------------------------------------
1 | import random
2 | import ssl
3 |
4 | from elasticsearch import Elasticsearch
5 |
6 | from esctl.config import Context
7 |
8 |
9 | class Client:
10 | def __new__(
11 | self,
12 | context: Context | None | None = None,
13 | http_auth: tuple[str] | None | None = None,
14 | ):
15 | if not hasattr(self, "instance"):
16 | self.instance = super().__new__(self)
17 | self.es: Elasticsearch.Elasticsearch = Client.initialize_elasticsearch_connection(self, context, http_auth)
18 |
19 | return self.instance
20 |
21 | @staticmethod
22 | def initialize_elasticsearch_connection(
23 | client,
24 | context: Context | None | None = None,
25 | http_auth: tuple[str] | None | None = None,
26 | ):
27 | elasticsearch_client_kwargs = {
28 | "http_auth": http_auth,
29 | }
30 |
31 | if context is not None:
32 | if context.settings.get("no_check_certificate"):
33 | ssl_context = ssl.create_default_context()
34 | ssl_context.check_hostname = False
35 | ssl_context.verify_mode = ssl.CERT_NONE
36 | elasticsearch_client_kwargs["ssl_context"] = ssl_context
37 | elasticsearch_client_kwargs["verify_certs"] = False
38 |
39 | if "max_retries" in context.settings:
40 | elasticsearch_client_kwargs["max_retries"] = context.settings.get(
41 | "max_retries",
42 | )
43 |
44 | if "timeout" in context.settings:
45 | elasticsearch_client_kwargs["timeout"] = context.settings.get("timeout")
46 |
47 | return Elasticsearch(
48 | random.choice(context.cluster["servers"]),
49 | **elasticsearch_client_kwargs,
50 | )
51 |
--------------------------------------------------------------------------------
/esctl/exceptions.py:
--------------------------------------------------------------------------------
1 | class SettingNotFoundError(Exception):
2 | pass
3 |
--------------------------------------------------------------------------------
/esctl/formatter.py:
--------------------------------------------------------------------------------
1 | class TableKey:
2 | def __init__(self, id, name=None, pretty_key=True):
3 | self.id = id
4 | if name is None:
5 | self.name = self._create_name_from_id(pretty_key=pretty_key)
6 | else:
7 | self.name = name
8 |
9 | def _format_special_word(self, word: str) -> str:
10 | if word.lower() in ["percent", "gc", "id", "uuid"]:
11 | if word == "percent":
12 | word = "%"
13 |
14 | elif word == "gc":
15 | word = "GC"
16 |
17 | elif word == "id":
18 | word = "ID"
19 |
20 | elif word == "uuid":
21 | word = "UUID"
22 |
23 | return word
24 |
25 | def _split_string(self, string: str) -> list[str]:
26 | splitted_string: list[str]
27 |
28 | if "." in string:
29 | splitted_string = string.split(".")
30 |
31 | # Check if words can also be splitted on underscores
32 | for idx in range(len(splitted_string)):
33 | # If there is an underscore anywhere in the word except at the beginning
34 | if "_" in splitted_string[idx] and not splitted_string[idx].startswith(
35 | "_",
36 | ):
37 | splitted_string[idx] = splitted_string[idx].split("_")
38 |
39 | flat_list = []
40 | for sublist in splitted_string:
41 | if sublist.__class__.__name__ == "list":
42 | for item in sublist:
43 | flat_list.append(item)
44 | else:
45 | flat_list.append(sublist)
46 |
47 | splitted_string = flat_list
48 |
49 | elif "_" in string:
50 | splitted_string = string.split("_")
51 |
52 | else:
53 | splitted_string = [string]
54 |
55 | return splitted_string
56 |
57 | def _create_name_from_id(self, pretty_key=True):
58 | """Extrapolate the column's name based on its ID."""
59 | if not pretty_key:
60 | return self.id
61 |
62 | name = self._split_string(self.id)
63 |
64 | for idx in range(len(name)):
65 | # Format some special words like "percent" -> %
66 | name[idx] = self._format_special_word(name[idx])
67 |
68 | # Put the word to titlecase unless it contains at least an uppercase letter
69 | if all(char.islower() for char in name[idx]):
70 | name[idx] = name[idx].title()
71 | # If the word contains an uppercase letter somewhere, maintain the case
72 | # but uppercase the first letter
73 | elif name[idx].startswith("_"):
74 | name[idx] = name[idx][0] + name[idx][1].upper() + name[idx][2:]
75 | else:
76 | name[idx] = name[idx][0].upper() + name[idx][1:]
77 |
78 | name = " ".join(name)
79 |
80 | return name
81 |
82 | def __repr__(self):
83 | return f"({self.id}, {self.name})"
84 |
85 |
86 | class JSONToCliffFormatter:
87 | """Format a JSON object to one of the cliff's class expected inputs."""
88 |
89 | def __init__(self, json, pretty_key=True):
90 | self.json = json
91 | self.pretty_key = pretty_key
92 |
93 | def _format_columns(self, columns):
94 | """Ensure all elements given as column names are formatted as needed.
95 |
96 | Each element must finally be a TableKey object containing the column ID
97 | and the column name.
98 |
99 | :Example:
100 | [
101 | ("disk.total"),
102 | ("disk.percent", "Disk %"),
103 | ]
104 |
105 | becomes
106 |
107 | [
108 | TableKey("disk.total", "Disk Total"),
109 | TableKey("disk.percent", "Disk %"),
110 | ]
111 |
112 | :param columns: The columns to format as a list of tuples containing
113 | two elements : the original column ID as given in the
114 | JSON object and an optional string defining the new
115 | column name (otherwise it will be extrapolated from
116 | the column ID)
117 | :paramtype columns: list
118 | :return: A list of TableKey objects
119 | :rtype: list
120 | """
121 | valid_list = []
122 |
123 | for column in columns:
124 | if isinstance(column, str):
125 | column = (column,)
126 |
127 | valid_list.append(TableKey(*column, pretty_key=self.pretty_key))
128 |
129 | return valid_list
130 |
131 | def format_for_lister(self, columns=[], none_as=None):
132 | """Convert a JSON object to an object compliant with
133 | cliff's Lister class.
134 |
135 | :Example:
136 | json :
137 | [{
138 | 'shards': '0',
139 | 'disk.indices': '0b',
140 | 'disk.used': '13.3gb',
141 | 'disk.avail': '45gb',
142 | 'disk.total': '58.4gb',
143 | 'disk.percent': '22',
144 | 'host': '172.17.0.2',
145 | 'ip': '172.17.0.2',
146 | 'node': '1b5a7b9edd01'
147 | }]
148 | columns :
149 | [
150 | 'shards',
151 | 'disk.indices',
152 | 'disk.used',
153 | 'disk.avail',
154 | 'disk.total',
155 | ('disk.percent', 'Disk %'),
156 | 'host',
157 | ('ip', 'IP'),
158 | 'node'
159 | ]
160 |
161 | :param columns: The columns to format as a list of tuples containing
162 | two elements : the original column ID as given in the
163 | JSON object and an optional string defining the new
164 | column name
165 | :paramtype columns: list
166 | :return: A tuple containing two tuples : the first one contains the
167 | columns' headers and the second one the lines to display
168 | :rtype: tuple
169 | """
170 | columns = self._format_columns(columns)
171 |
172 | lst = []
173 | # For every line in the JSON object, pick the value corresponding to
174 | # the given column ID
175 | for raw_line in self.json:
176 | line = []
177 | for column in columns:
178 | element = raw_line.get(column.id)
179 |
180 | if element is None:
181 | element = none_as
182 |
183 | line.append(element)
184 |
185 | lst.append(tuple(line))
186 |
187 | return (tuple([c.name for c in columns]), tuple(lst))
188 |
189 | def to_show_one(self, lines=[], none_as=None):
190 | """For every given element, retrieve the value in the original
191 | json object based on the key provided
192 |
193 | :param lines: The columns to format as a list of tuples containing
194 | two elements : the original column ID as given in the
195 | JSON object and an optional string defining the
196 | new column name
197 | :paramtype lines: list
198 | :return: A tuple containing two tuples : the first one contains the
199 | keys and the second one the values to display
200 | :rtype: tuple
201 | """
202 | lines = self._format_columns(lines)
203 |
204 | keys = []
205 | values = []
206 |
207 | for line in lines:
208 | keys.append(line.name)
209 |
210 | element = self.json.get(line.id)
211 | if element is None:
212 | element = none_as
213 |
214 | values.append(element)
215 |
216 | return (tuple(keys), tuple(values))
217 |
--------------------------------------------------------------------------------
/esctl/interactive.py:
--------------------------------------------------------------------------------
1 | # Licensed under the Apache License, Version 2.0 (the "License"); you may
2 | # not use this file except in compliance with the License. You may obtain
3 | # a copy of the License at
4 | #
5 | # http://www.apache.org/licenses/LICENSE-2.0
6 | #
7 | # Unless required by applicable law or agreed to in writing, software
8 | # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
9 | # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
10 | # License for the specific language governing permissions and limitations
11 | # under the License.
12 |
13 | #
14 | # The following code is coming from
15 | # https://raw.githubusercontent.com/openstack/cliff/9d77e2cb386dbb1dac089e5f7508b3c3f56fcd5d/cliff/interactive.py
16 | # and has been modified
17 | #
18 |
19 | """Application base class."""
20 |
21 | import itertools
22 | import shlex
23 | import sys
24 |
25 | import cmd2
26 |
27 |
28 | class InteractiveApp(cmd2.Cmd):
29 | """Provides "interactive mode" features.
30 |
31 | Refer to the cmd2_ and cmd_ documentation for details
32 | about subclassing and configuring this class.
33 |
34 | .. _cmd2: https://cmd2.readthedocs.io/en/latest/
35 | .. _cmd: http://docs.python.org/library/cmd.html
36 |
37 | :param parent_app: The calling application (expected to be derived
38 | from :class:`cliff.main.App`).
39 | :param command_manager: A :class:`cliff.commandmanager.CommandManager`
40 | instance.
41 | :param stdin: Standard input stream
42 | :param stdout: Standard output stream
43 | """
44 |
45 | use_rawinput = True
46 | doc_header = "Shell commands (type help ):"
47 | app_cmd_header = "Application commands (type help ):"
48 |
49 | def __init__(self, parent_app, command_manager, stdin, stdout, errexit=False):
50 | self.parent_app = parent_app
51 | if not hasattr(sys.stdin, "isatty") or sys.stdin.isatty():
52 | if hasattr(self.parent_app, "context") and hasattr(
53 | self.parent_app.context,
54 | "name",
55 | ):
56 | self.prompt = f"({parent_app.NAME} {self.parent_app.context.name}) "
57 | else:
58 | self.prompt = f"({parent_app.NAME}) "
59 | else:
60 | # batch/pipe mode
61 | self.prompt = ""
62 | self.command_manager = command_manager
63 | self.errexit = errexit
64 | cmd2.Cmd.__init__(self, "tab", stdin=stdin, stdout=stdout)
65 |
66 | def _split_line(self, line):
67 | try:
68 | return shlex.split(line.parsed.raw)
69 | except AttributeError:
70 | # cmd2 >= 0.9.1 gives us a Statement not a PyParsing parse
71 | # result.
72 | parts = shlex.split(line)
73 | if getattr(line, "command", None):
74 | parts.insert(0, line.command)
75 | return parts
76 |
77 | def default(self, line):
78 | # Tie in the default command processor to
79 | # dispatch commands known to the command manager.
80 | # We send the message through our parent app,
81 | # since it already has the logic for executing
82 | # the subcommand.
83 | line_parts = self._split_line(line)
84 |
85 | # When proving a command like 'cluster allocation explain',
86 | # line_parts contains
87 | # ['cluster allocation explain', 'allocation', 'explain']
88 | # which break run_subcommand()
89 | line_parts[0] = line_parts[0].split(" ")[0]
90 | ret = self.parent_app.run_subcommand(line_parts)
91 | if self.errexit:
92 | # Only provide this if errexit is enabled,
93 | # otherise keep old behaviour
94 | return ret
95 |
96 | def completenames(self, text, line, begidx, endidx):
97 | """Tab-completion for command prefix without completer delimiter.
98 |
99 | This method returns cmd style and cliff style commands matching
100 | provided command prefix (text).
101 | """
102 | completions = cmd2.Cmd.completenames(self, text, line, begidx, endidx)
103 | completions += self._complete_prefix(text)
104 | return completions
105 |
106 | def completedefault(self, text, line, begidx, endidx):
107 | """Default tab-completion for command prefix with completer delimiter.
108 |
109 | This method filters only cliff style commands matching provided
110 | command prefix (line) as cmd2 style commands cannot contain spaces.
111 | This method returns text + missing command part of matching commands.
112 | This method does not handle options in cmd2/cliff style commands, you
113 | must define complete_$method to handle them.
114 | """
115 | return [x[begidx:] for x in self._complete_prefix(line)]
116 |
117 | def _complete_prefix(self, prefix):
118 | """Returns cliff style commands with a specific prefix."""
119 | if not prefix:
120 | return [n for n, v in self.command_manager]
121 | return [n for n, v in self.command_manager if n.startswith(prefix)]
122 |
123 | def help_help(self):
124 | # Use the command manager to get instructions for "help"
125 | self.default("help help")
126 |
127 | def do_help(self, arg):
128 | if arg:
129 | # Check if the arg is a builtin command or something
130 | # coming from the command manager
131 | arg_parts = shlex.split(arg)
132 | method_name = "_".join(
133 | itertools.chain(
134 | ["do"],
135 | itertools.takewhile(lambda x: not x.startswith("-"), arg_parts),
136 | ),
137 | )
138 | # Have the command manager version of the help
139 | # command produce the help text since cmd and
140 | # cmd2 do not provide help for "help"
141 | if hasattr(self, method_name):
142 | return cmd2.Cmd.do_help(self, arg)
143 | # Dispatch to the underlying help command,
144 | # which knows how to provide help for extension
145 | # commands.
146 | try:
147 | # NOTE(coreycb): This try path can be removed once
148 | # requirements.txt has cmd2 >= 0.7.3.
149 | parsed = self.parsed
150 | except AttributeError:
151 | try:
152 | parsed = self.parser_manager.parsed
153 | except AttributeError:
154 | # cmd2 >= 0.9.1 does not have a parser manager
155 | parsed = lambda x: x # noqa E731
156 | self.default(parsed("help " + arg))
157 | else:
158 | cmd2.Cmd.do_help(self, arg)
159 | cmd_names = sorted([n for n, v in self.command_manager])
160 | self.print_topics(self.app_cmd_header, cmd_names, 15, 80)
161 | return None
162 |
163 | # Create exit alias to quit the interactive shell.
164 | do_exit = cmd2.Cmd.do_quit
165 |
166 | def get_names(self):
167 | # Override the base class version to filter out
168 | # things that look like they should be hidden
169 | # from the user.
170 | return [n for n in cmd2.Cmd.get_names(self) if not n.startswith("do__")]
171 |
172 | def precmd(self, statement):
173 | # Pre-process the parsed command in case it looks like one of
174 | # our subcommands, since cmd2 does not handle multi-part
175 | # command names by default.
176 | line_parts = self._split_line(statement)
177 | try:
178 | the_cmd = self.command_manager.find_command(line_parts)
179 | cmd_factory, cmd_name, sub_argv = the_cmd
180 | except ValueError:
181 | # Not a plugin command
182 | pass
183 | else:
184 | if hasattr(statement, "parsed"):
185 | # Older cmd2 uses PyParsing
186 | statement.parsed.command = cmd_name
187 | statement.parsed.args = " ".join(sub_argv)
188 | else:
189 | # cmd2 >= 0.9.1 uses shlex and gives us a Statement.
190 | statement.command = cmd_name
191 | statement.argv = [cmd_name] + sub_argv
192 | statement.args = " ".join(statement.argv)
193 | return statement
194 |
195 | def cmdloop(self):
196 | # We don't want the cmd2 cmdloop() behaviour, just call the old one
197 | # directly. In part this is because cmd2.cmdloop() doe not return
198 | # anything useful and we want to have a useful exit code.
199 | return self._cmdloop()
200 |
--------------------------------------------------------------------------------
/esctl/main.py:
--------------------------------------------------------------------------------
1 | import argparse
2 | import logging
3 | import os
4 | import re
5 | import subprocess
6 | import sys
7 |
8 | import pkg_resources
9 | import urllib3
10 | from cliff.app import App
11 | from cliff.commandmanager import CommandManager
12 |
13 | from esctl import utils
14 | from esctl.config import ConfigFileParser
15 | from esctl.elasticsearch import Client
16 | from esctl.interactive import InteractiveApp
17 |
18 | # `configure_logging` and `build_option_parser` methods comes from cliff
19 | # and are modified
20 |
21 |
22 | class Esctl(App):
23 | _es = None
24 | _config = None
25 | _config_file_parser = ConfigFileParser()
26 | log = App.LOG
27 |
28 | def __init__(self):
29 | os.environ["COLUMNS"] = "120"
30 | super().__init__(
31 | description=pkg_resources.require("Esctl")[0].project_name,
32 | version=pkg_resources.require("Esctl")[0].version,
33 | command_manager=CommandManager("esctl"),
34 | deferred_help=True,
35 | interactive_app_factory=InteractiveApp,
36 | )
37 | self.interactive_mode = False
38 |
39 | self.LOCAL_COMMANDS: list[str] = [
40 | "ConfigClusterList",
41 | "ConfigContextList",
42 | "ConfigContextSet",
43 | "ConfigShow",
44 | "ConfigUserList",
45 | ]
46 |
47 | def configure_logging(self):
48 | """Create logging handlers for any log output."""
49 | root_logger = logging.getLogger("")
50 | root_logger.setLevel(logging.DEBUG)
51 | logging.getLogger("stevedore.extension").setLevel(logging.WARNING)
52 |
53 | # Disable urllib's warnings
54 | # See https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
55 | urllib3.disable_warnings()
56 |
57 | # Set up logging to a file
58 | if self.options.log_file:
59 | file_handler = logging.FileHandler(filename=self.options.log_file)
60 | formatter = logging.Formatter(self.LOG_FILE_MESSAGE_FORMAT)
61 | file_handler.setFormatter(formatter)
62 | root_logger.addHandler(file_handler)
63 |
64 | # Always send higher-level messages to the console via stderr
65 | console = logging.StreamHandler(self.stderr)
66 | console_level = {
67 | 0: logging.ERROR, # quiet, with -q flag
68 | 1: logging.WARNING, # default, without -v or -q flags
69 | 2: logging.INFO,
70 | 3: logging.DEBUG,
71 | }.get(self.options.verbose_level, logging.DEBUG)
72 | console.setLevel(console_level)
73 |
74 | logging.addLevelName(
75 | logging.DEBUG,
76 | utils.Color.colorize(logging.getLevelName(logging.DEBUG), utils.Color.END),
77 | )
78 | logging.addLevelName(
79 | logging.INFO,
80 | utils.Color.colorize(logging.getLevelName(logging.INFO), utils.Color.BLUE),
81 | )
82 | logging.addLevelName(
83 | logging.WARNING,
84 | utils.Color.colorize(
85 | logging.getLevelName(logging.WARNING),
86 | utils.Color.YELLOW,
87 | ),
88 | )
89 | logging.addLevelName(
90 | logging.ERROR,
91 | utils.Color.colorize(
92 | logging.getLevelName(logging.ERROR),
93 | utils.Color.PURPLE,
94 | ),
95 | )
96 | logging.addLevelName(
97 | logging.CRITICAL,
98 | utils.Color.colorize(
99 | logging.getLevelName(logging.CRITICAL),
100 | utils.Color.RED,
101 | ),
102 | )
103 | formatter = logging.Formatter(
104 | "[%(levelname)-8s] " + self.CONSOLE_MESSAGE_FORMAT,
105 | )
106 | console.setFormatter(formatter)
107 | root_logger.addHandler(console)
108 |
109 | def insert_password_into_context(self):
110 | external_passowrd_definition = self.context.user.get("external_password")
111 | del self.context.user["external_password"]
112 |
113 | if "command" in external_passowrd_definition and "run" in external_passowrd_definition.get("command"):
114 | self.context.user["password"] = self._run_os_system_command(
115 | external_passowrd_definition.get("command").get("run"),
116 | )
117 |
118 | def initialize_app(self, argv):
119 | Esctl._config = Esctl._config_file_parser.load_configuration(
120 | self.options.config_file,
121 | )
122 | self.context = Esctl._config_file_parser.create_context(self.options.context)
123 |
124 | http_auth = None
125 |
126 | if self.context.user is not None:
127 | if "external_password" in self.context.user:
128 | self.insert_password_into_context()
129 |
130 | http_auth = (
131 | (self.context.user.get("username"), self.context.user.get("password"))
132 | if self.context.user.get("username") and self.context.user.get("password")
133 | else None
134 | )
135 |
136 | Client(self.context, http_auth)
137 |
138 | def _run_os_system_command(self, raw_command: str) -> str:
139 | self.log.debug(f"Running command : {raw_command}")
140 | return os.popen(raw_command).read().strip()
141 |
142 | def _run_shell_subcommand(self, command):
143 | command = command.split(" ")
144 | self.log.debug(f"Running pre-command : {command}")
145 | process = subprocess.Popen(
146 | command,
147 | stdout=subprocess.PIPE,
148 | stderr=subprocess.PIPE,
149 | )
150 |
151 | return process
152 |
153 | def prepare_to_run_command(self, cmd):
154 | if (cmd.__class__.__name__ not in self.LOCAL_COMMANDS) and hasattr(
155 | self.context,
156 | "pre_commands",
157 | ):
158 | for i in range(len(self.context.pre_commands)):
159 | command_block = self.context.pre_commands[i]
160 | process = self._run_shell_subcommand(command_block.get("command"))
161 |
162 | if command_block.get("wait_for_exit"):
163 | process.communicate()
164 |
165 | elif command_block.get("wait_for_output"):
166 | string_to_look_for = command_block.get("wait_for_output")
167 | pattern = re.compile(string_to_look_for)
168 |
169 | while True:
170 | line = process.stdout.readline()
171 | if not line:
172 | break
173 |
174 | line = line.decode("utf-8").strip()
175 |
176 | match = re.search(pattern, line)
177 |
178 | if match is None:
179 | self.log.debug(
180 | f"Expecting command output to match `{string_to_look_for}` but got `{line}`...",
181 | )
182 | else:
183 | self.log.debug(
184 | f"Got `{string_to_look_for}` from `{command_block.get('command')}`",
185 | )
186 | break
187 |
188 | self.context.pre_commands[i]["process"] = process
189 |
190 | def clean_up(self, cmd, result, err):
191 | if (cmd.__class__.__name__ not in self.LOCAL_COMMANDS) and hasattr(
192 | self.context,
193 | "pre_commands",
194 | ):
195 | for pre_command in self.context.pre_commands:
196 | pre_command.get("process").terminate()
197 |
198 | if err:
199 | self.log.debug("got an error: %s", err)
200 |
201 | def build_option_parser(self, description, version, argparse_kwargs=None):
202 | """Return an argparse option parser for this application."""
203 | argparse_kwargs = argparse_kwargs or {}
204 | parser = argparse.ArgumentParser(
205 | description=description,
206 | add_help=False,
207 | **argparse_kwargs,
208 | )
209 | parser.add_argument(
210 | "--version",
211 | action="version",
212 | version=f"%(prog)s {version}",
213 | )
214 | verbose_group = parser.add_mutually_exclusive_group()
215 | verbose_group.add_argument(
216 | "-v",
217 | "--verbose",
218 | action="count",
219 | dest="verbose_level",
220 | default=self.DEFAULT_VERBOSE_LEVEL,
221 | help="Increase verbosity of output. Can be repeated.",
222 | )
223 | verbose_group.add_argument(
224 | "-q",
225 | "--quiet",
226 | action="store_const",
227 | dest="verbose_level",
228 | const=0,
229 | help="Suppress output except warnings and errors.",
230 | )
231 | parser.add_argument(
232 | "--log-file",
233 | action="store",
234 | default=None,
235 | help="Specify a file to log output. Disabled by default.",
236 | )
237 | parser.add_argument(
238 | "--config",
239 | action="store",
240 | default="~/.esctlrc",
241 | help="Path to the config file",
242 | dest="config_file",
243 | )
244 | if self.deferred_help:
245 | parser.add_argument(
246 | "-h",
247 | "--help",
248 | dest="deferred_help",
249 | action="store_true",
250 | help="Show help message and exit.",
251 | )
252 |
253 | parser.add_argument(
254 | "--debug",
255 | default=False,
256 | action="store_true",
257 | help="Show tracebacks on errors.",
258 | )
259 | parser.add_argument(
260 | "--es-version",
261 | action="store",
262 | help="Elasticsearch version.",
263 | )
264 |
265 | parser.add_argument(
266 | "--context",
267 | action="store",
268 | help="Context to use",
269 | type=str,
270 | )
271 |
272 | return parser
273 |
274 |
275 | def main(argv=sys.argv[1:]):
276 | esctl = Esctl()
277 | return esctl.run(argv)
278 |
279 |
280 | if __name__ == "__main__":
281 | sys.exit(main(sys.argv[1:]))
282 |
--------------------------------------------------------------------------------
/esctl/settings.py:
--------------------------------------------------------------------------------
1 | import fnmatch
2 | import logging
3 | from abc import ABC
4 | from typing import Any
5 |
6 | from esctl.elasticsearch import Client
7 | from esctl.utils import Color
8 |
9 |
10 | class Settings(ABC):
11 | """Abstract class for settings management."""
12 |
13 | log = logging.getLogger(__name__)
14 | es = Client().es
15 |
16 |
17 | class Setting:
18 | def __init__(self, name: str, value: Any, persistency: str = "defaults"):
19 | self.name = name
20 | self.value = value
21 | self.persistency = persistency
22 |
23 | def __str__(self) -> str:
24 | return f"{self.name}={self.value}({self.persistency})"
25 |
26 |
27 | class ClusterSettings(Settings):
28 | """Handle cluster-level settings."""
29 |
30 | def list_(self) -> dict[str, dict[str, Any]]:
31 | return self.es.cluster.get_settings(include_defaults=True, flat_settings=True)
32 |
33 | def get(self, key: str, persistency: str = "transient") -> Setting:
34 | settings: dict[str, dict[str, Any]] = self.list_()
35 |
36 | if key in settings.get(persistency):
37 | return Setting(key, settings.get(persistency).get(key), persistency)
38 | # If the setting cannot be found in the requested persistency
39 | # look for it in the "defaults" values
40 | if key in settings.get("defaults"):
41 | return Setting(key, settings.get("defaults").get(key), "defaults")
42 | return Setting(key, None)
43 |
44 | def __get_setting_for_persistency(
45 | self,
46 | settings: dict[str, dict[str, Any]],
47 | key: str,
48 | persistency: str,
49 | ) -> Setting:
50 | if key in settings.get(persistency):
51 | return Setting(key, settings.get(persistency).get(key), persistency)
52 |
53 | return Setting(key, None)
54 |
55 | def mget(self, key: str) -> dict[str, Setting]:
56 | settings: dict[str, dict[str, Any]] = self.list_()
57 |
58 | return {
59 | "transient": self.__get_setting_for_persistency(settings, key, "transient"),
60 | "persistent": self.__get_setting_for_persistency(
61 | settings,
62 | key,
63 | "persistent",
64 | ),
65 | "defaults": self.__get_setting_for_persistency(settings, key, "defaults"),
66 | }
67 |
68 | def set(self, sections: str, value, persistency: str = "transient"):
69 | self.log.info(
70 | f"Changing {persistency}'s {Color.colorize(sections, Color.ITALIC)} to : {Color.colorize(value, Color.ITALIC)}",
71 | )
72 | return self.es.cluster.put_settings(
73 | body={persistency: {sections: value}},
74 | flat_settings=True,
75 | )
76 |
77 |
78 | class IndexSettings(Settings):
79 | """Handle index-level settings."""
80 |
81 | def list_(self, index: str) -> dict[str, dict[str, Setting]]:
82 | self.log.debug(f"Retrieving settings list for indices : {index}")
83 |
84 | settings: dict[str, dict[str, Setting]] = {}
85 |
86 | for index_name, index_settings in self.es.indices.get_settings(
87 | index=index,
88 | include_defaults=True,
89 | flat_settings=True,
90 | ).items():
91 | settings[index_name] = {}
92 | for setting_name, setting_value in index_settings.get("settings").items():
93 | settings[index_name][setting_name] = Setting(
94 | setting_name,
95 | setting_value,
96 | "settings",
97 | )
98 | for setting_name, setting_value in index_settings.get("defaults").items():
99 | settings[index_name][setting_name] = Setting(
100 | setting_name,
101 | setting_value,
102 | "defaults",
103 | )
104 |
105 | return settings
106 |
107 | def get(self, index: str, key: str | None) -> dict[str, list[Setting]]:
108 | self.log.debug(f"Retrieving setting(s) '{key}' for indices : {index}")
109 |
110 | response = self.list_(index)
111 | settings: dict[str, list[Setting]] = {}
112 | requested_settings: list[str] = []
113 |
114 | known_settings = [list(index_settings.keys()) for _, index_settings in response.items()][0]
115 |
116 | if "*" in key:
117 | requested_settings = fnmatch.filter(known_settings, key)
118 |
119 | elif "," in key:
120 | requested_settings = key.split(",")
121 | else:
122 | requested_settings = [key]
123 |
124 | for index_name, index_settings in response.items():
125 | settings[index_name] = []
126 |
127 | for setting_name in requested_settings:
128 | if setting_name in known_settings:
129 | settings[index_name].append(index_settings.get(setting_name))
130 | else:
131 | settings[index_name].append(Setting(setting_name, None))
132 |
133 | return settings
134 |
135 | def set(self, setting: str, value: Any, index: str):
136 | return self.es.indices.put_settings(index=index, body={setting: value})
137 |
--------------------------------------------------------------------------------
/esctl/utils.py:
--------------------------------------------------------------------------------
1 | from collections import OrderedDict
2 |
3 | import yaml
4 |
5 |
6 | class Color:
7 | BLUE = "\033[94m"
8 | BOLD = "\033[1m"
9 | CYAN = "\033[96m"
10 | DARKCYAN = "\033[36m"
11 | END = "\033[0m"
12 | GREEN = "\033[92m"
13 | ITALIC = "\033[3m"
14 | PURPLE = "\033[95m"
15 | RED = "\033[91m"
16 | UNDERLINE = "\033[4m"
17 | YELLOW = "\033[93m"
18 |
19 | @classmethod
20 | def colorize(cls, text, color):
21 | return f"{color}{text}{cls.END}"
22 |
23 |
24 | def flatten_dict(dictionary):
25 | def expand(key, value):
26 | if isinstance(value, dict):
27 | return [(key + "." + k, v) for k, v in flatten_dict(value).items()]
28 | return [(key, value)]
29 |
30 | items = [item for k, v in dictionary.items() for item in expand(k, v)]
31 |
32 | return dict(items)
33 |
34 |
35 | def setup_yaml():
36 | """https://stackoverflow.com/a/8661021"""
37 | yaml.add_representer(
38 | OrderedDict,
39 | lambda self, data: self.represent_mapping(
40 | "tag:yaml.org,2002:map",
41 | data.items(),
42 | ),
43 | )
44 |
--------------------------------------------------------------------------------
/node-list-sample.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/jeromepin/esctl/97280374464e5cd9217bffb857afdba9c29e461a/node-list-sample.png
--------------------------------------------------------------------------------
/pyproject.toml:
--------------------------------------------------------------------------------
1 | [build-system]
2 | requires = ["setuptools"]
3 | build-backend = "setuptools.build_meta"
4 |
5 | [project]
6 | name = "esctl"
7 | version = "1.11.1"
8 | description = "Easy to use CLI tool to manage Elasticsearch, preventing long curl commands"
9 | keywords = ["elasticsearch", "es", "cli"]
10 | readme = "README.md"
11 | license = {file = "LICENSE"}
12 | classifiers = [
13 | "Development Status :: 5 - Production/Stable",
14 | "Environment :: Console",
15 | "Intended Audience :: Developers",
16 | "Intended Audience :: System Administrators",
17 | "License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
18 | "Operating System :: OS Independent",
19 | "Programming Language :: Python",
20 | "Programming Language :: Python :: 3.6",
21 | "Programming Language :: Python :: 3.7",
22 | "Programming Language :: Python :: 3.8",
23 | "Topic :: Utilities",
24 | "Topic :: System :: Shells",
25 | ]
26 | requires-python = ">=3.10"
27 | authors = [
28 | {name = "Jérôme Pin", email = "jeromepin@users.noreply.github.com"}
29 | ]
30 | maintainers = [
31 | {name = "Jérôme Pin", email = "jeromepin@users.noreply.github.com"}
32 | ]
33 | dependencies = [
34 | "cerberus>=1.3.5",
35 | "certifi>=2024.8.30",
36 | "cliff==3.2",
37 | "elasticsearch==8.*",
38 | "jmespath>=1.0.1",
39 | ]
40 |
41 | [project.urls]
42 | Homepage = "https://github.com/jeromepin/esctl"
43 | Repository = "https://github.com/jeromepin/esctl"
44 |
45 | [project.entry-points.console_scripts]
46 | esctl = "esctl.main:main"
47 |
48 | [project.entry-points.esctl]
49 | "alias list" = "esctl.cmd.alias:AliasList"
50 | "cat allocation" = "esctl.cmd.cat:CatAllocation"
51 | "cat plugins" = "esctl.cmd.cat:CatPlugins"
52 | "cat shards" = "esctl.cmd.cat:CatShards"
53 | "cat thread-pool" = "esctl.cmd.cat:CatThreadpool"
54 | "cat templates" = "esctl.cmd.cat:CatTemplates"
55 | "cluster allocation explain" = "esctl.cmd.cluster:ClusterAllocationExplain"
56 | "cluster health" = "esctl.cmd.cluster:ClusterHealth"
57 | "cluster info" = "esctl.cmd.cluster:ClusterInfo"
58 | "cluster routing allocation enable" = "esctl.cmd.cluster:ClusterRoutingAllocationEnable"
59 | "cluster stats" = "esctl.cmd.cluster:ClusterStats"
60 | "cluster settings list" = "esctl.cmd.settings:ClusterSettingsList"
61 | "cluster settings get" = "esctl.cmd.settings:ClusterSettingsGet"
62 | "cluster settings reset" = "esctl.cmd.settings:ClusterSettingsReset"
63 | "cluster settings set" = "esctl.cmd.settings:ClusterSettingsSet"
64 | "config cluster list" = "esctl.cmd.config:ConfigClusterList"
65 | "config context list" = "esctl.cmd.config:ConfigContextList"
66 | "config context set" = "esctl.cmd.config:ConfigContextSet"
67 | "config show" = "esctl.cmd.config:ConfigShow"
68 | "config user list" = "esctl.cmd.config:ConfigUserList"
69 | "document get" = "esctl.cmd.document:DocumentGet"
70 | "index close" = "esctl.cmd.index:IndexClose"
71 | "index create" = "esctl.cmd.index:IndexCreate"
72 | "index delete" = "esctl.cmd.index:IndexDelete"
73 | "index list" = "esctl.cmd.index:IndexList"
74 | "index open" = "esctl.cmd.index:IndexOpen"
75 | "index reindex" = "esctl.cmd.index:IndexReindex"
76 | "index settings get" = "esctl.cmd.settings:IndexSettingsGet"
77 | "index settings list" = "esctl.cmd.settings:IndexSettingsList"
78 | "index settings set" = "esctl.cmd.settings:IndexSettingsSet"
79 | "logging get" = "esctl.cmd.logging:LoggingGet"
80 | "logging reset" = "esctl.cmd.logging:LoggingReset"
81 | "logging set" = "esctl.cmd.logging:LoggingSet"
82 | "migration deprecations" = "esctl.cmd.migration:MigrationDeprecations"
83 | "node exclude" = "esctl.cmd.node:NodeExclude"
84 | "node hot-threads" = "esctl.cmd.node:NodeHotThreads"
85 | "node list" = "esctl.cmd.node:NodeList"
86 | "node stats" = "esctl.cmd.node:NodeStats"
87 | "raw" = "esctl.cmd.raw:RawCommand"
88 | "repository list" = "esctl.cmd.repository:RepositoryList"
89 | "repository show" = "esctl.cmd.repository:RepositoryShow"
90 | "repository verify" = "esctl.cmd.repository:RepositoryVerify"
91 | "roles get" = "esctl.cmd.roles:SecurityRolesGet"
92 | "snapshot list" = "esctl.cmd.snapshot:SnapshotList"
93 | "task list" = "esctl.cmd.task:TaskList"
94 | "users get" = "esctl.cmd.users:SecurityUsersGet"
95 |
96 | [tool.ruff]
97 | # Allow lines to be as long as 120.
98 | line-length = 120
99 |
100 | [tool.ruff.lint]
101 | # select = ["ALL"]
102 | ignore = [
103 | "ANN",
104 | "D",
105 | "G004",
106 | ]
107 | exclude = ["tests/**.py"]
108 |
109 | [tool.uv]
110 | dev-dependencies = [
111 | "mypy>=1.13.0",
112 | "pipx>=1.7.1",
113 | "pre-commit>=4.0.1",
114 | "pytest-clarity>=1.0.1",
115 | "pytest-icdiff>=0.9",
116 | "pytest>=8.3.4",
117 | "pyupgrade>=3.19.0",
118 | "ruff>=0.8.3",
119 | "setuptools>=75.6.0",
120 | ]
121 |
122 | [[index]]
123 | name = "testpypi"
124 | url = "https://test.pypi.org/simple/"
125 | publish-url = "https://test.pypi.org/legacy/"
126 |
--------------------------------------------------------------------------------
/pytest.ini:
--------------------------------------------------------------------------------
1 | [pytest]
2 | filterwarnings =
3 | ignore:::cmd2
4 |
--------------------------------------------------------------------------------
/tests/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/jeromepin/esctl/97280374464e5cd9217bffb857afdba9c29e461a/tests/__init__.py
--------------------------------------------------------------------------------
/tests/base_test_class.py:
--------------------------------------------------------------------------------
1 | import os
2 | from argparse import Namespace
3 | from unittest import TestCase
4 |
5 | import esctl.main
6 |
7 |
8 | class EsctlTestCase(TestCase):
9 | def setUp(self):
10 | self.tests_path = os.path.dirname(os.path.realpath(__file__))
11 | self.fixtures_path = f"{self.tests_path}/files"
12 |
13 | self.app = esctl.main.Esctl()
14 |
15 | # Command-line arguments
16 | self.app.options = Namespace()
17 | self.app.options.config_file = f"{self.fixtures_path}/valid_esctlrc.yml"
18 | self.app.options.verbose_level = 3
19 | self.app.options.context = "foobar"
20 | self.app.initialize_app([])
21 |
--------------------------------------------------------------------------------
/tests/cmd/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/jeromepin/esctl/97280374464e5cd9217bffb857afdba9c29e461a/tests/cmd/__init__.py
--------------------------------------------------------------------------------
/tests/cmd/index_test.py:
--------------------------------------------------------------------------------
1 | from esctl.cmd.index import IndexReindex
2 |
3 | from ..base_test_class import EsctlTestCase
4 |
5 |
6 | class TestIndexReindex(EsctlTestCase):
7 | def test_build_request_body(self):
8 | cases = [
9 | {
10 | "input": {
11 | "source_index": "foo",
12 | "destination_index": "bar",
13 | "version_type": "internal",
14 | "op_type": "index",
15 | "conflicts": "abort",
16 | },
17 | "expected_output": {
18 | "source": {
19 | "index": "foo",
20 | },
21 | "dest": {
22 | "index": "bar",
23 | "version_type": "internal",
24 | "op_type": "index",
25 | },
26 | "conflicts": "abort",
27 | },
28 | },
29 | ]
30 |
31 | index_reindex_cmd = IndexReindex(self.app, [])
32 | for case in cases:
33 | self.assertEqual(
34 | index_reindex_cmd._build_request_body(case.get("input")),
35 | case.get("expected_output"),
36 | )
37 |
--------------------------------------------------------------------------------
/tests/cmd/migration_test.py:
--------------------------------------------------------------------------------
1 | import json
2 | from argparse import Namespace
3 | from pathlib import Path
4 | from unittest import mock
5 | from unittest.mock import MagicMock
6 |
7 | import esctl.main
8 | from esctl.cmd.migration import MigrationDeprecations
9 |
10 |
11 | @mock.patch("esctl.cmd.migration.MigrationDeprecations.request")
12 | def test_transform(mock_request: MagicMock, capsys):
13 | # TODO: move into fixtures
14 | # Initialize the App
15 | tests_path = Path(__file__).parent.parent
16 | fixtures_path = f"{tests_path}/files"
17 |
18 | app = esctl.main.Esctl()
19 |
20 | # Command-line arguments
21 | app.options = Namespace()
22 | app.options.config_file = f"{fixtures_path}/valid_esctlrc.yml"
23 | app.options.verbose_level = 3
24 | app.options.context = "foobar"
25 | app.initialize_app([])
26 |
27 | # Mock the HTTP call
28 | mock_request.return_value = {
29 | "cluster_settings": [
30 | {
31 | "level": "critical",
32 | "message": "Cluster name cannot contain ':'",
33 | "url": "https://www.elastic.co/guide/en/elasticsearch/reference/7.17/breaking-changes-7.0.html#_literal_literal_is_no_longer_allowed_in_cluster_name",
34 | "details": "This cluster is named [mycompany:logging], which contains the illegal character ':'.",
35 | },
36 | ],
37 | "node_settings": [],
38 | "index_settings": {
39 | "logs:apache": [
40 | {
41 | "level": "warning",
42 | "message": "Index name cannot contain ':'",
43 | "url": "https://www.elastic.co/guide/en/elasticsearch/reference/7.17/breaking-changes-7.0.html#_literal_literal_is_no_longer_allowed_in_index_name",
44 | "details": "This index is named [logs:apache], which contains the illegal character ':'.",
45 | },
46 | ],
47 | "foobar": [
48 | {
49 | "level": "critical",
50 | "message": "Foo bar baz qux.",
51 | "url": "https://www.elastic.co/guide/en/elasticsearch/reference/7.17/breaking-changes-7.0.html#_literal_literal_is_no_longer_allowed_in_index_name",
52 | "details": "This is a random string.",
53 | },
54 | {
55 | "level": "warning",
56 | "message": "Index name cannot contain ':'",
57 | "url": "https://www.elastic.co/guide/en/elasticsearch/reference/7.17/breaking-changes-7.0.html#_literal_literal_is_no_longer_allowed_in_index_name",
58 | "details": "This index is named [logs:apache], which contains the illegal character ':'.",
59 | },
60 | ],
61 | },
62 | "ml_settings": [],
63 | }
64 | migration_deprecations_cmd = MigrationDeprecations(app, [])
65 |
66 | parsed_args = MagicMock()
67 | parsed_args.columns = ("Kind", "Level", "Message", "Doc")
68 | parsed_args.formatter = "json"
69 | parsed_args.sort_columns = []
70 |
71 | migration_deprecations_cmd.run(parsed_args)
72 |
73 | out, _ = capsys.readouterr()
74 |
75 | print(out)
76 |
77 | assert out.startswith(
78 | json.dumps(
79 | [
80 | {
81 | "Kind": "cluster_settings",
82 | "Level": "critical",
83 | "Message": "Cluster name cannot contain ':'",
84 | "Doc": "https://www.elastic.co/guide/en/elasticsearch/reference/7.17/breaking-changes-7.0.html#_literal_literal_is_no_longer_allowed_in_cluster_name",
85 | },
86 | {
87 | "Kind": "index_settings",
88 | "Level": "warning",
89 | "Message": "[logs:apache] Index name cannot contain ':'",
90 | "Doc": "https://www.elastic.co/guide/en/elasticsearch/reference/7.17/breaking-changes-7.0.html#_literal_literal_is_no_longer_allowed_in_index_name",
91 | },
92 | {
93 | "Kind": "index_settings",
94 | "Level": "critical",
95 | "Message": "[foobar] Foo bar baz qux.",
96 | "Doc": "https://www.elastic.co/guide/en/elasticsearch/reference/7.17/breaking-changes-7.0.html#_literal_literal_is_no_longer_allowed_in_index_name",
97 | },
98 | {
99 | "Kind": "index_settings",
100 | "Level": "warning",
101 | "Message": "[foobar] Index name cannot contain ':'",
102 | "Doc": "https://www.elastic.co/guide/en/elasticsearch/reference/7.17/breaking-changes-7.0.html#_literal_literal_is_no_longer_allowed_in_index_name",
103 | },
104 | ],
105 | ),
106 | )
107 |
--------------------------------------------------------------------------------
/tests/cmd/snapshot_test.py:
--------------------------------------------------------------------------------
1 | from esctl.cmd.snapshot import SnapshotList
2 |
3 | from ..base_test_class import EsctlTestCase
4 |
5 |
6 | class TestSnapshotList(EsctlTestCase):
7 | def test_value_based_color(self):
8 | cases = [
9 | {
10 | "input": {"status": "SUCCESS", "failed_shards": "0"},
11 | "expected_output": {"status": "SUCCESS", "failed_shards": "0"},
12 | },
13 | {
14 | "input": {"status": "PARTIAL", "failed_shards": "0"},
15 | "expected_output": {
16 | "status": "\x1b[93mPARTIAL\x1b[0m",
17 | "failed_shards": "\x1b[91m0\x1b[0m",
18 | },
19 | },
20 | {
21 | "input": {"status": "IN_PROGRESS", "failed_shards": "0"},
22 | "expected_output": {
23 | "status": "\x1b[96mIN_PROGRESS\x1b[0m",
24 | "failed_shards": "0",
25 | },
26 | },
27 | ]
28 |
29 | snapshot_list_cmd = SnapshotList(self.app, [])
30 | for case in cases:
31 | self.assertEqual(
32 | snapshot_list_cmd.transform([case.get("input")])[0],
33 | case.get("expected_output"),
34 | )
35 |
--------------------------------------------------------------------------------
/tests/conftest.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/jeromepin/esctl/97280374464e5cd9217bffb857afdba9c29e461a/tests/conftest.py
--------------------------------------------------------------------------------
/tests/files/index_settings.json:
--------------------------------------------------------------------------------
1 | {
2 | "foobar": {
3 | "settings": {
4 | "index.analysis.analyzer.lower_folded.filter": [
5 | "lowercase",
6 | "asciifolding"
7 | ],
8 | "index.analysis.analyzer.lower_folded.tokenizer": "standard",
9 | "index.analysis.analyzer.lower_folded.type": "custom",
10 | "index.analysis.analyzer.trigram.filter": [
11 | "lowercase",
12 | "asciifolding",
13 | "shingle"
14 | ],
15 | "index.analysis.analyzer.trigram.tokenizer": "standard",
16 | "index.analysis.analyzer.trigram.type": "custom",
17 | "index.analysis.filter.shingle.max_shingle_size": "3",
18 | "index.analysis.filter.shingle.min_shingle_size": "2",
19 | "index.analysis.filter.shingle.type": "shingle",
20 | "index.creation_date": "1576577286632",
21 | "index.number_of_replicas": "2",
22 | "index.number_of_shards": "1",
23 | "index.provided_name": "foobar",
24 | "index.uuid": "ZJscnlcjQ7yXIQ2Zvl8wXa",
25 | "index.version.created": "7000199"
26 | },
27 | "defaults": {
28 | "index.allocation.max_retries": "5",
29 | "index.analyze.max_token_count": "10000",
30 | "index.auto_expand_replicas": "false",
31 | "index.blocks.metadata": "false",
32 | "index.blocks.read": "false",
33 | "index.blocks.read_only": "false",
34 | "index.blocks.read_only_allow_delete": "false",
35 | "index.blocks.write": "false",
36 | "index.codec": "default",
37 | "index.compound_format": "0.1",
38 | "index.data_path": "",
39 | "index.default_pipeline": "_none",
40 | "index.fielddata.cache": "node",
41 | "index.force_memory_term_dictionary": "false",
42 | "index.format": "0",
43 | "index.frozen": "false",
44 | "index.gc_deletes": "60s",
45 | "index.highlight.max_analyzed_offset": "1000000",
46 | "index.indexing.slowlog.level": "TRACE",
47 | "index.indexing.slowlog.reformat": "true",
48 | "index.indexing.slowlog.source": "1000",
49 | "index.indexing.slowlog.threshold.index.debug": "-1",
50 | "index.indexing.slowlog.threshold.index.info": "-1",
51 | "index.indexing.slowlog.threshold.index.trace": "-1",
52 | "index.indexing.slowlog.threshold.index.warn": "-1",
53 | "index.lifecycle.indexing_complete": "false",
54 | "index.lifecycle.name": "",
55 | "index.lifecycle.rollover_alias": "",
56 | "index.load_fixed_bitset_filters_eagerly": "true",
57 | "index.mapper.dynamic": "true",
58 | "index.mapping.coerce": "false",
59 | "index.mapping.depth.limit": "20",
60 | "index.mapping.ignore_malformed": "false",
61 | "index.mapping.nested_fields.limit": "50",
62 | "index.mapping.nested_objects.limit": "10000",
63 | "index.mapping.total_fields.limit": "1000",
64 | "index.max_adjacency_matrix_filters": "100",
65 | "index.max_docvalue_fields_search": "100",
66 | "index.max_inner_result_window": "100",
67 | "index.max_ngram_diff": "1",
68 | "index.max_refresh_listeners": "1000",
69 | "index.max_regex_length": "1000",
70 | "index.max_rescore_window": "10000",
71 | "index.max_result_window": "10000",
72 | "index.max_script_fields": "32",
73 | "index.max_shingle_diff": "3",
74 | "index.max_slices_per_scroll": "1024",
75 | "index.max_terms_count": "65536",
76 | "index.merge.policy.deletes_pct_allowed": "33.0",
77 | "index.merge.policy.expunge_deletes_allowed": "10.0",
78 | "index.merge.policy.floor_segment": "2mb",
79 | "index.merge.policy.max_merge_at_once": "10",
80 | "index.merge.policy.max_merge_at_once_explicit": "30",
81 | "index.merge.policy.max_merged_segment": "5gb",
82 | "index.merge.policy.reclaim_deletes_weight": "2.0",
83 | "index.merge.policy.segments_per_tier": "10.0",
84 | "index.merge.scheduler.auto_throttle": "true",
85 | "index.merge.scheduler.max_merge_count": "6",
86 | "index.merge.scheduler.max_thread_count": "1",
87 | "index.number_of_routing_shards": "1",
88 | "index.optimize_auto_generated_id": "true",
89 | "index.percolator.map_unmapped_fields_as_text": "false",
90 | "index.priority": "1",
91 | "index.queries.cache.enabled": "true",
92 | "index.query.default_field": [
93 | "*"
94 | ],
95 | "index.query.parse.allow_unmapped_fields": "true",
96 | "index.query_string.lenient": "false",
97 | "index.refresh_interval": "1s",
98 | "index.requests.cache.enable": "true",
99 | "index.routing.allocation.enable": "all",
100 | "index.routing.allocation.total_shards_per_node": "-1",
101 | "index.routing.rebalance.enable": "all",
102 | "index.routing_partition_size": "1",
103 | "index.search.idle.after": "30s",
104 | "index.search.slowlog.level": "TRACE",
105 | "index.search.slowlog.threshold.fetch.debug": "-1",
106 | "index.search.slowlog.threshold.fetch.info": "-1",
107 | "index.search.slowlog.threshold.fetch.trace": "-1",
108 | "index.search.slowlog.threshold.fetch.warn": "-1",
109 | "index.search.slowlog.threshold.query.debug": "-1",
110 | "index.search.slowlog.threshold.query.info": "-1",
111 | "index.search.slowlog.threshold.query.trace": "-1",
112 | "index.search.slowlog.threshold.query.warn": "-1",
113 | "index.search.throttled": "false",
114 | "index.shard.check_on_startup": "false",
115 | "index.soft_deletes.enabled": "false",
116 | "index.soft_deletes.retention.operations": "0",
117 | "index.soft_deletes.retention_lease.period": "12h",
118 | "index.sort.field": [],
119 | "index.sort.missing": [],
120 | "index.sort.mode": [],
121 | "index.sort.order": [],
122 | "index.source_only": "false",
123 | "index.store.fs.fs_lock": "native",
124 | "index.store.preload": [],
125 | "index.store.stats_refresh_interval": "10s",
126 | "index.store.type": "",
127 | "index.translog.durability": "REQUEST",
128 | "index.translog.flush_threshold_size": "512mb",
129 | "index.translog.generation_threshold_size": "64mb",
130 | "index.translog.retention.age": "12h",
131 | "index.translog.retention.size": "512mb",
132 | "index.translog.sync_interval": "5s",
133 | "index.unassigned.node_left.delayed_timeout": "1m",
134 | "index.warmer.enabled": "true",
135 | "index.write.wait_for_active_shards": "1",
136 | "index.xpack.ccr.following_index": "false",
137 | "index.xpack.version": "",
138 | "index.xpack.watcher.template.version": ""
139 | }
140 | },
141 | "qux": {
142 | "settings": {
143 | "index.analysis.analyzer.lower_folded.filter": [
144 | "lowercase",
145 | "asciifolding"
146 | ],
147 | "index.analysis.analyzer.lower_folded.tokenizer": "standard",
148 | "index.analysis.analyzer.lower_folded.type": "custom",
149 | "index.analysis.analyzer.trigram.filter": [
150 | "lowercase",
151 | "asciifolding",
152 | "shingle"
153 | ],
154 | "index.analysis.analyzer.trigram.tokenizer": "standard",
155 | "index.analysis.analyzer.trigram.type": "custom",
156 | "index.analysis.filter.shingle.max_shingle_size": "3",
157 | "index.analysis.filter.shingle.min_shingle_size": "2",
158 | "index.analysis.filter.shingle.type": "shingle",
159 | "index.creation_date": "1576577286632",
160 | "index.number_of_replicas": "2",
161 | "index.number_of_shards": "1",
162 | "index.provided_name": "qux",
163 | "index.uuid": "ZJscnlcjQ7yXIQ8Evl8wXg",
164 | "index.version.created": "7000199"
165 | },
166 | "defaults": {
167 | "index.allocation.max_retries": "5",
168 | "index.analyze.max_token_count": "10000",
169 | "index.auto_expand_replicas": "false",
170 | "index.blocks.metadata": "false",
171 | "index.blocks.read": "false",
172 | "index.blocks.read_only": "false",
173 | "index.blocks.read_only_allow_delete": "false",
174 | "index.blocks.write": "false",
175 | "index.codec": "default",
176 | "index.compound_format": "0.1",
177 | "index.data_path": "",
178 | "index.default_pipeline": "_none",
179 | "index.fielddata.cache": "node",
180 | "index.force_memory_term_dictionary": "false",
181 | "index.format": "0",
182 | "index.frozen": "false",
183 | "index.gc_deletes": "60s",
184 | "index.highlight.max_analyzed_offset": "1000000",
185 | "index.indexing.slowlog.level": "TRACE",
186 | "index.indexing.slowlog.reformat": "true",
187 | "index.indexing.slowlog.source": "1000",
188 | "index.indexing.slowlog.threshold.index.debug": "-1",
189 | "index.indexing.slowlog.threshold.index.info": "-1",
190 | "index.indexing.slowlog.threshold.index.trace": "-1",
191 | "index.indexing.slowlog.threshold.index.warn": "-1",
192 | "index.lifecycle.indexing_complete": "false",
193 | "index.lifecycle.name": "",
194 | "index.lifecycle.rollover_alias": "",
195 | "index.load_fixed_bitset_filters_eagerly": "true",
196 | "index.mapper.dynamic": "true",
197 | "index.mapping.coerce": "false",
198 | "index.mapping.depth.limit": "20",
199 | "index.mapping.ignore_malformed": "false",
200 | "index.mapping.nested_fields.limit": "50",
201 | "index.mapping.nested_objects.limit": "10000",
202 | "index.mapping.total_fields.limit": "1000",
203 | "index.max_adjacency_matrix_filters": "100",
204 | "index.max_docvalue_fields_search": "100",
205 | "index.max_inner_result_window": "100",
206 | "index.max_ngram_diff": "1",
207 | "index.max_refresh_listeners": "1000",
208 | "index.max_regex_length": "1000",
209 | "index.max_rescore_window": "10000",
210 | "index.max_result_window": "10000",
211 | "index.max_script_fields": "32",
212 | "index.max_shingle_diff": "3",
213 | "index.max_slices_per_scroll": "1024",
214 | "index.max_terms_count": "65536",
215 | "index.merge.policy.deletes_pct_allowed": "33.0",
216 | "index.merge.policy.expunge_deletes_allowed": "10.0",
217 | "index.merge.policy.floor_segment": "2mb",
218 | "index.merge.policy.max_merge_at_once": "10",
219 | "index.merge.policy.max_merge_at_once_explicit": "30",
220 | "index.merge.policy.max_merged_segment": "5gb",
221 | "index.merge.policy.reclaim_deletes_weight": "2.0",
222 | "index.merge.policy.segments_per_tier": "10.0",
223 | "index.merge.scheduler.auto_throttle": "true",
224 | "index.merge.scheduler.max_merge_count": "6",
225 | "index.merge.scheduler.max_thread_count": "1",
226 | "index.number_of_routing_shards": "1",
227 | "index.optimize_auto_generated_id": "true",
228 | "index.percolator.map_unmapped_fields_as_text": "false",
229 | "index.priority": "1",
230 | "index.queries.cache.enabled": "true",
231 | "index.query.default_field": [
232 | "*"
233 | ],
234 | "index.query.parse.allow_unmapped_fields": "true",
235 | "index.query_string.lenient": "false",
236 | "index.refresh_interval": "1s",
237 | "index.requests.cache.enable": "true",
238 | "index.routing.allocation.enable": "all",
239 | "index.routing.allocation.total_shards_per_node": "-1",
240 | "index.routing.rebalance.enable": "all",
241 | "index.routing_partition_size": "1",
242 | "index.search.idle.after": "30s",
243 | "index.search.slowlog.level": "TRACE",
244 | "index.search.slowlog.threshold.fetch.debug": "-1",
245 | "index.search.slowlog.threshold.fetch.info": "-1",
246 | "index.search.slowlog.threshold.fetch.trace": "-1",
247 | "index.search.slowlog.threshold.fetch.warn": "-1",
248 | "index.search.slowlog.threshold.query.debug": "-1",
249 | "index.search.slowlog.threshold.query.info": "-1",
250 | "index.search.slowlog.threshold.query.trace": "-1",
251 | "index.search.slowlog.threshold.query.warn": "-1",
252 | "index.search.throttled": "false",
253 | "index.shard.check_on_startup": "false",
254 | "index.soft_deletes.enabled": "false",
255 | "index.soft_deletes.retention.operations": "0",
256 | "index.soft_deletes.retention_lease.period": "12h",
257 | "index.sort.field": [],
258 | "index.sort.missing": [],
259 | "index.sort.mode": [],
260 | "index.sort.order": [],
261 | "index.source_only": "false",
262 | "index.store.fs.fs_lock": "native",
263 | "index.store.preload": [],
264 | "index.store.stats_refresh_interval": "10s",
265 | "index.store.type": "",
266 | "index.translog.durability": "REQUEST",
267 | "index.translog.flush_threshold_size": "512mb",
268 | "index.translog.generation_threshold_size": "64mb",
269 | "index.translog.retention.age": "12h",
270 | "index.translog.retention.size": "512mb",
271 | "index.translog.sync_interval": "5s",
272 | "index.unassigned.node_left.delayed_timeout": "1m",
273 | "index.warmer.enabled": "true",
274 | "index.write.wait_for_active_shards": "1",
275 | "index.xpack.ccr.following_index": "false",
276 | "index.xpack.version": "",
277 | "index.xpack.watcher.template.version": ""
278 | }
279 | }
280 | }
281 |
--------------------------------------------------------------------------------
/tests/files/valid_esctlrc.yml:
--------------------------------------------------------------------------------
1 | clusters:
2 | foobar:
3 | servers:
4 | - https://foobar-01:9200
5 | - https://foobar-02:9200
6 | - https://foobar-03:9200
7 | contexts:
8 | foobar:
9 | cluster: foobar
10 | user: foobar
11 | default-context: foobar
12 | settings:
13 | max_retries: 0
14 | no_check_certificate: true
15 | users:
16 | foobar:
17 | password: fooooooooooooo
18 | username: foo
19 |
--------------------------------------------------------------------------------
/tests/formatter_test.py:
--------------------------------------------------------------------------------
1 | from esctl.formatter import TableKey
2 |
3 | from .base_test_class import EsctlTestCase
4 |
5 |
6 | class TestCreateColumnNameFromId(EsctlTestCase):
7 | def setUp(self):
8 | self.cases = [
9 | {"input": "index.uuid", "expected_output": "Index UUID"},
10 | {"input": "index.id", "expected_output": "Index ID"},
11 | {"input": "index.gc_deletes", "expected_output": "Index GC Deletes"},
12 | {"input": "foo.bar", "expected_output": "Foo Bar"},
13 | {"input": "index.provided_name", "expected_output": "Index Provided Name"},
14 | {
15 | "input": "index.soft_deletes.retention_lease.period",
16 | "expected_output": "Index Soft Deletes Retention Lease Period",
17 | },
18 | {"input": "FooBar", "expected_output": "FooBar"},
19 | {"input": "foo_bar", "expected_output": "Foo Bar"},
20 | {"input": "foo-bar", "expected_output": "Foo-bar"},
21 | {"input": "foo.percent", "expected_output": "Foo %"},
22 | {
23 | "input": "_source._metadata.createTime",
24 | "expected_output": "_Source _Metadata CreateTime",
25 | },
26 | # TODO: Rework name handling in `node stats` command
27 | # {
28 | # "input": "b4XevSayQNCx_-1mrroUzw.index.uuid",
29 | # "expected_output": "b4XevSayQNCx_-1mrroUzw Index UUID",
30 | # },
31 | # {
32 | # "input": "BfTa0i_sQQmyFsVzjOtApQ.index.uuid",
33 | # "expected_output": "BfTa0i_sQQmyFsVzjOtApQ Index UUID",
34 | # },
35 | # {
36 | # "input": "Xggkx0fORHO_vgJ-WBp0bw.index.uuid",
37 | # "expected_output": "Xggkx0fORHO_vgJ-WBp0bw Index UUID",
38 | # },
39 | ]
40 |
41 | def test_name_interpolation(self):
42 | for case in self.cases:
43 | self.assertEqual(
44 | TableKey(case.get("input"))._create_name_from_id(),
45 | case.get("expected_output"),
46 | )
47 |
48 | def test_no_pretty_name_interpolation(self):
49 | cases = [c.get("input") for c in self.cases]
50 |
51 | for case in cases:
52 | self.assertEqual(
53 | TableKey(case)._create_name_from_id(pretty_key=False),
54 | case,
55 | )
56 |
--------------------------------------------------------------------------------
/tests/settings_test.py:
--------------------------------------------------------------------------------
1 | import json
2 | import unittest.mock
3 |
4 | from .base_test_class import EsctlTestCase
5 |
6 |
7 | class TestClusterSettingsRetrieval(EsctlTestCase):
8 | def setUp(self):
9 | super().setUp()
10 | self.mock = unittest.mock.patch("esctl.settings.Settings.es").start()
11 | self.mock.cluster.get_settings.return_value = self.fixtures()
12 |
13 | def fixtures(self):
14 | return {
15 | "persistent": {
16 | "thread_pool.estimated_time_interval": "200ms",
17 | "cluster.routing.allocation.allow_rebalance": "indices_all_active",
18 | },
19 | "transient": {
20 | "thread_pool.estimated_time_interval": "100ms",
21 | "cluster.routing.allocation.allow_rebalance": "always",
22 | },
23 | "defaults": {"cluster.routing.allocation.disk.watermark.high": "90%"},
24 | }
25 |
26 | def test_settings_get(self):
27 | import esctl.settings
28 |
29 | cluster_settings = esctl.settings.ClusterSettings()
30 | self.assertEqual(
31 | cluster_settings.get(
32 | "thread_pool.estimated_time_interval",
33 | persistency="transient",
34 | ).value,
35 | "100ms",
36 | )
37 | self.assertEqual(
38 | cluster_settings.get(
39 | "thread_pool.estimated_time_interval",
40 | persistency="persistent",
41 | ).value,
42 | "200ms",
43 | )
44 | self.assertEqual(
45 | cluster_settings.get(
46 | "cluster.routing.allocation.allow_rebalance",
47 | persistency="persistent",
48 | ).value,
49 | "indices_all_active",
50 | )
51 | self.assertEqual(
52 | cluster_settings.get(
53 | "cluster.routing.allocation.allow_rebalance",
54 | persistency="transient",
55 | ).value,
56 | "always",
57 | )
58 | self.assertEqual(
59 | cluster_settings.get(
60 | "cluster.routing.allocation.disk.watermark.high",
61 | persistency="transient",
62 | ).value,
63 | "90%",
64 | )
65 | self.assertEqual(
66 | cluster_settings.get(
67 | "cluster.routing.allocation.disk.watermark.high",
68 | persistency="persistent",
69 | ).value,
70 | "90%",
71 | )
72 | self.assertEqual(
73 | cluster_settings.get(
74 | "cluster.routing.allocation.foobar",
75 | persistency="transient",
76 | ).value,
77 | None,
78 | )
79 |
80 |
81 | class TestIndexSettingsRetrieval(EsctlTestCase):
82 | def setUp(self):
83 | super().setUp()
84 | self.mock = unittest.mock.patch("esctl.settings.Settings.es").start()
85 | self.mock.indices.get_settings.return_value = self.fixtures()
86 |
87 | def fixtures(self):
88 | with open(f"{self.fixtures_path}/index_settings.json") as json_file:
89 | return json.load(json_file)
90 |
91 | def test_settings_list(self):
92 | import esctl.settings
93 |
94 | index_settings = esctl.settings.IndexSettings()
95 |
96 | self.assertEqual(
97 | index_settings.get(
98 | "foobar,qux",
99 | "index.analysis.analyzer.trigram.tokenizer",
100 | )
101 | .get("foobar")[0]
102 | .value,
103 | "standard",
104 | )
105 |
--------------------------------------------------------------------------------