├── .gitignore
├── CODE_OF_CONDUCT.md
├── CONTRIBUTING.md
├── LICENSE
├── README.md
├── benchmarking-1.1
├── oltp-point-select
│ ├── oltp-ps-test-mysql57.bash
│ ├── oltp-ps-test-mysql80-cache.bash
│ └── oltp-ps-test-mysql80.bash
└── oltp-read-only
│ ├── oltp-ro-test-mysql57.bash
│ ├── oltp-ro-test-mysql80-cache.bash
│ └── oltp-ro-test-mysql80.bash
├── images
├── architecture-diagram.png
├── init-db-credentials.png
├── init-db-output.png
├── proxysql1-connect.png
├── proxysql1-eice.png
├── proxysql1.png
├── proxysql2-connect.png
├── proxysql2-eice.png
├── proxysql2.png
├── sysbench-eic.png
├── sysbench-success.png
└── sysbench.png
├── proxysql-demo.yml
└── scripts
├── init-db.bash
├── install-proxysql-and-mysql.bash
├── install-sysbench.bash
├── setup-sysbench.bash
├── tune.sql
└── visualize.sql
/.gitignore:
--------------------------------------------------------------------------------
1 | .env
2 | tests
--------------------------------------------------------------------------------
/CODE_OF_CONDUCT.md:
--------------------------------------------------------------------------------
1 | ## Code of Conduct
2 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct).
3 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact
4 | opensource-codeofconduct@amazon.com with any additional questions or comments.
5 |
--------------------------------------------------------------------------------
/CONTRIBUTING.md:
--------------------------------------------------------------------------------
1 | # Contributing Guidelines
2 |
3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional
4 | documentation, we greatly value feedback and contributions from our community.
5 |
6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary
7 | information to effectively respond to your bug report or contribution.
8 |
9 |
10 | ## Reporting Bugs/Feature Requests
11 |
12 | We welcome you to use the GitHub issue tracker to report bugs or suggest features.
13 |
14 | When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already
15 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful:
16 |
17 | * A reproducible test case or series of steps
18 | * The version of our code being used
19 | * Any modifications you've made relevant to the bug
20 | * Anything unusual about your environment or deployment
21 |
22 |
23 | ## Contributing via Pull Requests
24 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that:
25 |
26 | 1. You are working against the latest source on the *main* branch.
27 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already.
28 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted.
29 |
30 | To send us a pull request, please:
31 |
32 | 1. Fork the repository.
33 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change.
34 | 3. Ensure local tests pass.
35 | 4. Commit to your fork using clear commit messages.
36 | 5. Send us a pull request, answering any default questions in the pull request interface.
37 | 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation.
38 |
39 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and
40 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/).
41 |
42 |
43 | ## Finding contributions to work on
44 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start.
45 |
46 |
47 | ## Code of Conduct
48 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct).
49 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact
50 | opensource-codeofconduct@amazon.com with any additional questions or comments.
51 |
52 |
53 | ## Security issue notifications
54 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue.
55 |
56 |
57 | ## Licensing
58 |
59 | See the [LICENSE](LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution.
60 |
--------------------------------------------------------------------------------
/LICENSE:
--------------------------------------------------------------------------------
1 | MIT No Attribution
2 |
3 | Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
4 |
5 | Permission is hereby granted, free of charge, to any person obtaining a copy of
6 | this software and associated documentation files (the "Software"), to deal in
7 | the Software without restriction, including without limitation the rights to
8 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
9 | the Software, and to permit persons to whom the Software is furnished to do so.
10 |
11 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
12 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
13 | FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
14 | COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
15 | IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
16 | CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
17 |
18 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # ProxySQL Query Cache Integration with MySQL 8.0
2 | The solution will create a high-available deployment of ProxySQL Servers on EC2 that can replace the deprecated MySQL Query Cache.
3 |
4 | 
5 |
6 | ## Problem
7 | MySQL Community removes Query Cache on MySQL8.0 and this could lead to performance dips for some workloads. In MySQL 5.7, the Query Cache has played a crucial role in optimizing database performance by caching query results, thus reducing CPU and memory load upon subsequent requests. However, its removal in MySQL 8.0 has led to increased resource utilization for some workloads, directly impacting those who relied heavily on this feature for performance optimization.
8 |
9 | Consequently, this necessitates a shift towards more advanced hardware configurations to handle the additional processing requirements, inadvertently escalating operational costs. This predicament has prompted a search for viable alternatives, highlighting the urgent need for innovative solutions to sustain efficiency and cost-effectiveness amidst evolving database technologies.
10 |
11 | Additionally, with the retirement of MySQL 5.7, users are moving to MySQL 8.0. If they are using the Query Cache in MySQL 5.7, users will have to mitigate/fix the performance issue with alternative solution, e.g. modify application to optimize queries, optimize database schema & index, etc. However, in the specific case where it is impossible to modify application (e.g. they are using 3rd party application) or tune the database, they will require an intermediate solution to replace the original Query Cache.
12 |
13 | ## Goal
14 | This solution provides the simplest deployable use case to alleviate performance issues resulting from MySQL Query Cache Deprecation. You can also use this to get started with ProxySQL on Aurora MySQL or RDS MySQL.
15 |
16 | ## Solution Overview:
17 | This solution will set up the following:
18 | - 1x Internal Network Load Balancer
19 | - 2x EC2 with ProxySQL installed (Private Subnets, Highly-Available Architecture)
20 | - 1x Aurora MySQL5.7 Cluster (for benchmarking comparison)
21 | - 1x Aurora MySQL8.0 Cluster
22 | - 1x EC2 with Sysbench installed (Public Subnet, for testing and benchmarking purposes)
23 |
24 | ## Pre-requisites:
25 | - Region: ap-southeast-1
26 | - Available CIDR Block: 10.3.0.0/16
27 |
28 | ## Setting Up:
29 |
30 | ### 1. Download the repository and create the stack
31 | Stack Creation will take around 5-10mins.
32 | ```bash
33 | git clone https://github.com/aws-samples/proxysql-to-replace-mysql-query-cache.git
34 | cd proxysql-to-replace-mysql-query-cache
35 | aws cloudformation create-stack --stack-name proxysql-demo --template-body file://proxysql-demo.yml
36 | ```
37 |
38 | ### 2. Fill up the appropriate credentials in scripts/init-db.bash
39 | We will now populate the `scripts/init-db.bash` script so that we can set up the ProxySQL instances. Go to CloudFormation Outputs and you can find the appropriate endpoints. In CloudFormation Outputs, you will need `Aurora80WriterEndpoint` and `Aurora57ReaderEndpoint`.
40 |
41 | ```
42 | WRITER_ENDPOINT="fill_up_with_MySQL8.0_writer"
43 | READER_ENDPOINT="fill_up_with_MySQL8.0_reader"
44 | MYSQL_ADMIN_USER="fill_up" (default: admin)
45 | MYSQL_ADMIN_PASSWORD="fill_up" (default: mysqladmin)
46 | ```
47 |
48 | 
49 |
50 | ### 3. Connect to ProxySQL Instance 1
51 | Go to the AWS Console and navtigate to all your EC2 Instances. Choose `proxysql-demo-EC2ProxySQL1`.
52 |
53 | 
54 |
55 |
56 |
57 | 
58 |
59 | Use `EC2 Instance Connect Endpoint` to log into the first ProxySQL EC2 instance (proxysql-demo-EC2ProxySQL1)
60 |
61 | 
62 |
63 | ### 4. Setup ProxySQL Instance 1
64 | Run the `scripts/init-db.bash` script by copy and pasting into the command line of ProxySQL Instance 1.
65 |
66 | If you receive `-bash: mysql: command not found` error, in the command line of ProxySQL Instance 1:
67 | 1. Run `sudo su`
68 | 2. Copy and paste the contents from `scripts/install-proxysql-and-mysql.bash` into the command line
69 | 3. run `exit`
70 | 4. Now Copy and paste `scripts/init-db.bash` into the command line and re run the script.
71 |
72 | You should be able to see an output like this. If you see `mysql: [Warning] ... can be insecure`, you are on the right track. You can choose to copy and paste the script again and you will see the tables appear:
73 |
74 | 
75 |
76 | In the tables, if `connect_success` or `ping_success` are shown to be 0, it means that you have entered the wrong credentials for your Aurora MySQL 8.0 database.
77 |
78 | ### 5. Connect to ProxySQL Instance 2
79 | Go to the AWS Console and navtigate to all your EC2 Instances. Choose `proxysql-demo-EC2ProxySQL2`.
80 |
81 | 
82 |
83 |
84 |
85 | 
86 |
87 | Use `EC2 Instance Connect Endpoint` to log into the second ProxySQL EC2 instance (proxysql-demo-EC2ProxySQL2)
88 |
89 | 
90 |
91 | ### 6. Setup ProxySQL Instance 2
92 | Run the `scripts/init-db.bash` script by copy and pasting into the command line of ProxySQL Instance 2.
93 |
94 | If you receive `-bash: mysql: command not found` error, in the command line of ProxySQL Instance 2:
95 | 1. Run `sudo su`
96 | 2. Copy and paste the contents from `scripts/install-proxysql-and-mysql.bash` into the command line
97 | 3. run `exit`
98 | 4. Now Copy and paste `scripts/init-db.bash` into the command line and re run the script.
99 |
100 | You should be able to see an output like this. If you see `mysql: [Warning] ... can be insecure`, you are on the right track. You can choose to copy and paste the script again and you will see the tables appear:
101 |
102 | 
103 |
104 | In the tables, if `connect_success` or `ping_success` are shown to be 0, it means that you have entered the wrong credentials for your Aurora MySQL 8.0 database.
105 |
106 | ### 7. Connect to proxysql from a testing server!
107 | We have provisioned a testing instance in one of the public subnets. You can access this instance from the EC2 Console. Look for `proxysql-demo-sysbench`.
108 |
109 | 
110 |
111 | Connect to your instance with `EC2 Instance Connect` this time, not `EC2 Instance Connect Endpoint`.
112 |
113 | 
114 |
115 | Once inside the Sysbench test server, run the following command to check if the Proxysql servers are up and running. **You can find the NLB Endpoint from CloudFormation Outputs**:
116 |
117 | ```
118 | curl
119 | ```
120 |
121 | You will see "Hello from proxysql Instance 1" or "Hello from proxysql instance 2" if you are successful
122 |
123 | - Next, run the following commands:
124 |
125 | ```
126 | sysbench --version
127 | mysql --version
128 | ```
129 |
130 | - If you do not see the version being listed, copy and paste the `scripts/install-sysbench.bash` to install the necessary modules.
131 | - To connect to your ProxySQL Instances, run the following commmand:
132 |
133 | ```
134 | mysql -h -ustnduser -pstnduser -P3306
135 | ```
136 |
137 | 
138 |
139 | You can now connect to ProxySQL from the test server!
140 |
141 | Congratulations, you are now able to connect to your MySQL Cluster via a Highly Available Deployment of ProxySQL!
142 |
143 | ## Scripts
144 | There are a couple of helper scripts that can be found in the `scripts/` folder. This section provides a brief look into what they do.
145 |
146 | These scripts are not recommended to be used in production as they expose credentials to the EC2 instance. It is best practice to integrate parameters and credentials with a secure sevice, such as AWS Secrets Manager or AWS Parameter Store.
147 |
148 | For the purpose of the demo and for convenience, the scripts are provided.
149 |
150 | ### init-db.bash
151 | This script initialises the ProxySQL Server and Aurora MySQL 8.0 Database with the necessary ProxySQL permissions and settings.
152 |
153 | Most notably, you will need to fill in the following parameters in the script. The writer and reader endpoints should point to Aurora MySQL 8.0, and they can be found in the CloudFormation Outputs section. The Admin user and password should be set according to best practices. In the recommended demo, we have set it to default to `MYSQL_ADMIN_USER=admin` and `MYSQL_ADMIN_PASSWORD=mysqladmin`.
154 |
155 | ```
156 | # Aurora Cluster variables
157 | export WRITER_ENDPOINT=""
158 | export READER_ENDPOINT=""
159 | export MYSQL_ADMIN_USER=""
160 | export MYSQL_ADMIN_PASSWORD=""
161 | ```
162 |
163 | If you are unsure if ProxySQL is connected, you can run the following commands in the script. These commands provide us insight as to whether the ProxySQL-Aurora integration has been correctly setup. Look at the ProxySQL documentation if more help is needed:
164 |
165 | ```
166 | execute_proxysql_command "SELECT * FROM monitor.mysql_server_connect_log ORDER BY time_start_us DESC LIMIT 3;"
167 | execute_proxysql_command "SELECT * FROM monitor.mysql_server_ping_log ORDER BY time_start_us DESC LIMIT 3;"
168 | ```
169 |
170 | ### install-proxysql-and-mysql.bash
171 | This script installs the necessary resources to run proxysql locally. On some scenarios, the EC2 is unable to install proxysql and this script must be run manually after startup.
172 |
173 | ### install-sysbench.bash
174 | This script installs the necessary resources to run sysbench, inclusive of sysbench v1.1.0-2ca9e3f. On some scenarios, the EC2 is unable to install sysbench and this script must be run manually after startup. This script should be run on the `sysbench EC2 Instance`.
175 |
176 | Sysbench is a widely-used, open-source performance benchmarking tool for databases. You can use sysbench to run your performance tests from the `sysbench EC2 Instance` which is included in this template.
177 |
178 | This script is only needed if you want to run performance tests via the `Sysbench EC2 Instance`.
179 |
180 | ### setup-sysbench.bash
181 | This Script sets up the sysbench tables and environment from the `Sysbench EC2 instance`.
182 | Appropriate sysbench tables will be created on your MySQL5.7 and MySQL8.0 Databases. Note that in this template, we have spun up 2 Databases: (1) Aurora MySQL 8.0, and (2) Aurora MySQL 5.7. The Aurora MySQL 5.7 database is there to help in benchmarking tests.
183 |
184 | Take note that you will need to fill up the following variables:
185 |
186 | ```
187 | export MYSQL_USER="fill_up"
188 | export MYSQL_PASSWORD="fill_up"
189 | export MYSQL_HOST="fill_up_with_MySQL5.7_writer"
190 | ```
191 |
192 | The MYSQL_USER and MYSQL_PASSWORD are the ones that you used previously in the `scripts/init.db-bash` or in the CloudFormation Parameters. If you did not change them, the default is `MYSQL_USER=admin` and `MYSQL_PASSWORD=mysqladmin`
193 |
194 | and also do not forget the variable near the bottom of the script:
195 |
196 | ```
197 | export MYSQL_HOST="fill_up_with_MySQL8.0_writer"
198 | ```
199 |
200 | This script is only needed if you want to run performance tests via the `Sysbench EC2 Instance`.
201 |
202 | ### tune.sql
203 | This script is to be run on the ProxySQL servers as it helps to tune the different parameters accordingly. You must first log into the ProxySQL server in order to run the script. You should log in as the ProxySQL admin account. The default credentials are shown below. The admin user should have been set in `scripts/init-db.bash`:
204 |
205 | ```
206 | mysql -h127.0.0.1 -uadmin -padmin -P6032
207 | ```
208 |
209 | It is not recommended to adjust the script unless you have a good understanding of the parameters. These can be found in the official ProxySQL site.
210 |
211 | The script will Flush the Query Cache when run. Also, if you are looking turn off the query cache,
212 | you will find this command: `update mysql_query_rules set cache_ttl=300000 where rule_id=51;` and you can change the `cache_ttl=null` and rerun the script.
213 |
214 | ### visualize.sql
215 | This script is to be run on the ProxySQL servers as it runs a set of commands that gives information into what is happening in ProxySQL, and some parameters that have been set.
216 |
217 | You must first log into the ProxySQL server in order to run the script. You should log in as the ProxySQL admin account. The default credentials are shown below. The admin user should have been set in `scripts/init-db.bash`:
218 |
219 | ```
220 | mysql -h127.0.0.1 -uadmin -padmin -P6032
221 | ```
222 |
223 | Note that running this will flush Query Cache Statistics.
224 |
225 | You should flush statistics after you have initiated the database setup in `scripts/setup-sysbench.bash`.
226 |
227 | If you want to visualize your ProxySQL instance without flushing the cache, you can comment out the following line:
228 | `select * from stats_mysql_query_digest_reset limit 1;`
229 |
230 | ## ProxySQL Testing
231 | It is highly recommended that you download sysbench and execute a test from one of your devices. You can compare the benchmarking results from the benchmarking tests under benchmarking-1.1.
232 |
233 | Follow the following steps to install sysbench and setup the sysbench server for testing
234 |
235 | ### 1. Install Sysbench
236 | Copy and paste the script from `scripts/install-sysbench.bash`. This installs the necessary packages + Sysbench v1.1 into the Sysbench server.
237 |
238 | ### 2. Setup the necessary sysbench tables
239 | Fill up the necessary parameters from the `scripts/setup-sysbench.bash` script. Take note to fill up the `MySQL8.0_writer` parameter, which is located near the bottom of the script
240 |
241 | Copy and paste the script into the command line and run it. This script creates the necessary benchmarking tables.
242 |
243 | ### 3. Start Benchmarking!
244 | There are 2 main benchmarking tests that can be done. As we are testing the efficacy of the Query Cache, we will focus only on reads.
245 |
246 | 1. OLTP read only
247 | 2. OLTP point select
248 |
249 | You can investigate the folder benchmarking-1.1 to find out the different kinds of tests and their results.
250 |
251 | #### benchmarking-1.1/oltp-point-select
252 | 3 scripts are present to conduct tests. `oltp-ps-test-mysql57.bash` and `oltp-ps-test-mysql80.bash` scripts will connect directly to the Aurora MySQL databases, while `oltp-ps-test-mysql80-cache.bash` will run a test on the ProxySQL Instance.
253 |
254 | Additionally, the scripts require different user credentials. Connecting to ProxySQL will need a proxySQL user (defaulted to be `stnduser`). The other scripts that connect directly to Aurora MySQL will need a user that can connect with the necessary permissions. This is the credentials that you used in the CloudFormation Parameters to specify users. The defaults are USER=`admin`, PASSWORD=`mysqladmin`.
255 |
256 | Fill up the appropriate variables in the different scripts:
257 | 1. `oltp-ps-test-mysql57.bash`:
258 | - MYSQL_USER = User with necessary permissions to run sysbench tests
259 | - MYSQL_PASSWORD = Password of User with necessary permissions to run sysbench tests
260 | - MYSQL_HOST = MySQL 5.7 Writer Endpoint
261 |
262 | 2. `oltp-ps-test-mysql80.bash`:
263 | - MYSQL_USER = User with necessary permissions to run sysbench tests
264 | - MYSQL_PASSWORD = Password of User with necessary permissions to run sysbench tests
265 | - MYSQL_HOST = MySQl 8.0 Writer Endpoint
266 |
267 | 3. `oltp-ps-test-mysql80-cache.bash`:
268 | - MYSQL_USER = User with necessary permissions to run sysbench tests, (default = stnduser)
269 | - MYSQL_PASSWORD = Password of User with necessary permissions to run sysbench tests, (default = stnduser)
270 | - MYSQL_HOST = NLB Endpoint
271 | - For this test, we are running the test on the ProxySQL instance. To fully test the efficacy of the solution, you should use the NLB endpoint. If you want to connect directly to the ProxySQL Instance, you can use the Ip Address but it is not recommended as results might differ and you lose the High-Availability that comes with this solution.
272 |
273 | Copy and paste the scripts into the `Sysbench EC2 Instance` to start running the test.
274 |
275 | #### benchmarking-1.1/oltp-read-only
276 | Here, 3 scripts are also present to conduct tests. `oltp-ro-test-mysql57.bash` and `oltp-ro-test-mysql80.bash` scripts will connect directly to the Aurora MySQL databases, while `oltp-ro-test-mysql80-cache.bash` will run a test on the ProxySQL Instance.
277 |
278 | Additionally, the different scripts will need different user credentials. The script connect to ProxySQL will need a proxySQL user (defaulted to be `stnduser`). The other scripts will need a user that can connect with the necessary permissions. This is the credentials that you used in the CloudFormation Parameters to specify users. The defaults are USER=`admin`, PASSWORD=`mysqladmin`.
279 |
280 | Fill up the appropriate variables in the different scripts:
281 | 1. `oltp-ps-test-mysql57.bash`:
282 | - MYSQL_USER = User with necessary permissions to run sysbench tests
283 | - MYSQL_PASSWORD = Password of User with necessary permissions to run sysbench tests
284 | - MYSQL_HOST = MySQL 5.7 Writer Endpoint
285 |
286 | 2. `oltp-ps-test-mysql80.bash`:
287 | - MYSQL_USER = User with necessary permissions to run sysbench tests
288 | - MYSQL_PASSWORD = Password of User with necessary permissions to run sysbench tests
289 | - MYSQL_HOST = MySQl 8.0 Writer Endpoint
290 |
291 | 3. `oltp-ps-test-mysql80-cache.bash`:
292 | - MYSQL_USER = User with necessary permissions to run sysbench tests, (default = stnduser)
293 | - MYSQL_PASSWORD = Password of User with necessary permissions to run sysbench tests, (default = stnduser)
294 | - MYSQL_HOST = NLB Endpoint
295 | - For this test, we are running the test on the ProxySQL instance. To fully test the efficacy of the solution, you should use the NLB endpoint. If you want to connect directly to the ProxySQL Instance, you can use the Ip Address but it is not recommended as results might differ and you lose the High-Availability that comes with this solution.
296 |
297 | ## Disclaimer
298 | The assets here are to be used only as a Proof-of-Concept. It is strongly recommended to review and test all assets in the repository before moving to a true production workload.
299 |
300 | Additionally, this solution interacts with the following open-source software:
301 | - [Sysbench](https://github.com/akopytov/sysbench)
302 | - [ProxySQL](https://proxysql.com/)
303 |
304 | They each have different terms of use published, and might have pricing requirements for production use cases. You should be familiar with their pricing and terms and conditions, ensuring that your use case complies with their terms before proceeding.
305 |
306 | ## Security
307 |
308 | See [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more information.
309 |
310 | ## License
311 |
312 | This library is licensed under the MIT-0 License. See the LICENSE file.
313 |
314 | ## References:
315 | - [ProxySQL Documentation](https://proxysql.com/documentation/)
316 | - [ProxySQL Query Cache Scaling](https://proxysql.com/blog/scaling-with-proxysql-query-cache/)
317 | - [How to use ProxySQL with open source platforms to split SQL reads and writes on Amazon Aurora clusters](https://aws.amazon.com/blogs/database/how-to-use-proxysql-with-open-source-platforms-to-split-sql-reads-and-writes-on-amazon-aurora-clusters/)
318 | - [MySQL 8.0: Retiring Support for the Query Cache](https://dev.mysql.com/blog-archive/mysql-8-0-retiring-support-for-the-query-cache/)
319 | - [Scaling with ProxySQL Query Cache](https://proxysql.com/blog/scaling-with-proxysql-query-cache/)
320 | - [Amazon RDS for MySQL 5.7 will reach end of standard support on February 29, 2024](https://repost.aws/articles/ARWm1Gv0vJTIKCblhWhPXjWg/announcement-amazon-rds-for-mysql-5-7-will-reach-end-of-standard-support-on-february-29-2024)
321 | - [Setup ProxySQL for High Availability](https://www.percona.com/blog/setup-proxysql-for-high-availability-not-single-point-failure/)
322 |
323 |
--------------------------------------------------------------------------------
/benchmarking-1.1/oltp-point-select/oltp-ps-test-mysql57.bash:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 |
3 | # MySQL Connection Details
4 | export MYSQL_USER="fill_up"
5 | export MYSQL_PASSWORD="fill_up"
6 | export MYSQL_HOST="fill_up_with_MySQL57_writer_endpoint"
7 | export PORT=3306
8 |
9 | # Sysbench Parameters
10 | export SYSBENCH_DB="sbtest"
11 | export SYSBENCH_THREADS=16
12 | export SYSBENCH_NUM_TABLES=16
13 | export SYSBENCH_TABLE_SIZE=1000000
14 |
15 | sysbench oltp_point_select run \
16 | --report-interval=5 \
17 | --table-size=$SYSBENCH_TABLE_SIZE \
18 | --mysql-host=$MYSQL_HOST \
19 | --mysql-port=$PORT \
20 | --mysql-user=$MYSQL_USER \
21 | --mysql-password=$MYSQL_PASSWORD \
22 | --mysql-db=$SYSBENCH_DB \
23 | --tables=1 \
24 | --time=300 \
25 | --threads=512 \
26 | --skip_trx=on \
27 | --db-ps-mode=disable \
28 | --histogram=on \
29 | --warmup-time=300 \
30 | # --mysql-ignore-errors=2013,2003,1290,1213 \
31 | # --reconnect=1000 \
32 |
33 |
--------------------------------------------------------------------------------
/benchmarking-1.1/oltp-point-select/oltp-ps-test-mysql80-cache.bash:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 |
3 | # ProxySQL Parameters
4 | export MYSQL_USER="stnduser"
5 | export MYSQL_PASSWORD="stnduser"
6 | export MYSQL_HOST="NLB_endpoint_or_proxysql_instance_ipaddress"
7 | export PORT=3306
8 |
9 | # Sysbench Parameters
10 | export SYSBENCH_DB="sbtest"
11 | export SYSBENCH_THREADS=16
12 | export SYSBENCH_NUM_TABLES=16
13 | export SYSBENCH_TABLE_SIZE=1000000
14 |
15 | sysbench oltp_point_select run \
16 | --report-interval=5 \
17 | --table-size=$SYSBENCH_TABLE_SIZE \
18 | --mysql-host=$MYSQL_HOST \
19 | --mysql-port=$PORT \
20 | --mysql-user=$MYSQL_USER \
21 | --mysql-password=$MYSQL_PASSWORD \
22 | --mysql-db=$SYSBENCH_DB \
23 | --tables=1 \
24 | --time=300 \
25 | --threads=512 \
26 | --skip_trx=on \
27 | --db-ps-mode=disable \
28 | --histogram=on \
29 | --warmup-time=300 \
30 | # --mysql-ignore-errors=2013,2003,1290,1213 \
31 | # --reconnect=1000 \
32 |
33 |
--------------------------------------------------------------------------------
/benchmarking-1.1/oltp-point-select/oltp-ps-test-mysql80.bash:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 |
3 | # MySQL Connection Details
4 | export MYSQL_USER="fill_up"
5 | export MYSQL_PASSWORD="fill_up"
6 | export MYSQL_HOST="fill_up_with_MySQL80_writer_endpoint"
7 | export PORT=3306
8 |
9 | # Sysbench Parameters
10 | export SYSBENCH_DB="sbtest"
11 | export SYSBENCH_THREADS=16
12 | export SYSBENCH_NUM_TABLES=16
13 | export SYSBENCH_TABLE_SIZE=1000000
14 |
15 | sysbench oltp_point_select run \
16 | --report-interval=5 \
17 | --table-size=$SYSBENCH_TABLE_SIZE \
18 | --mysql-host=$MYSQL_HOST \
19 | --mysql-port=$PORT \
20 | --mysql-user=$MYSQL_USER \
21 | --mysql-password=$MYSQL_PASSWORD \
22 | --mysql-db=$SYSBENCH_DB \
23 | --tables=1 \
24 | --time=300 \
25 | --threads=512 \
26 | --skip_trx=on \
27 | --db-ps-mode=disable \
28 | --histogram=on \
29 | --warmup-time=300 \
30 | # --mysql-ignore-errors=2013,2003,1290,1213 \
31 | # --reconnect=1000 \
32 |
33 |
--------------------------------------------------------------------------------
/benchmarking-1.1/oltp-read-only/oltp-ro-test-mysql57.bash:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 |
3 | # MySQL Connection Details
4 | export MYSQL_USER="fill_up"
5 | export MYSQL_PASSWORD="fill_up"
6 | export MYSQL_HOST="fill_up_with_MySQL57_writer_endpoint"
7 | export PORT=3306
8 |
9 | # Sysbench Parameters
10 | export SYSBENCH_DB="sbtest"
11 | export SYSBENCH_THREADS=16
12 | export SYSBENCH_NUM_TABLES=16
13 | export SYSBENCH_TABLE_SIZE=1000000
14 |
15 | sysbench oltp_read_only run \
16 | --report-interval=5 \
17 | --table-size=$SYSBENCH_TABLE_SIZE \
18 | --mysql-host=$MYSQL_HOST \
19 | --mysql-port=$PORT \
20 | --mysql-user=$MYSQL_USER \
21 | --mysql-password=$MYSQL_PASSWORD \
22 | --mysql-db=$SYSBENCH_DB \
23 | --tables=1 \
24 | --time=300 \
25 | --warmup-time=300 \
26 | --threads=128 \
27 | --skip_trx=on \
28 | --db-ps-mode=disable \
29 | --histogram=on \
30 | # --mysql-ignore-errors=2013,2003,1290,1213 \
31 | # --reconnect=1000 \
32 |
33 |
--------------------------------------------------------------------------------
/benchmarking-1.1/oltp-read-only/oltp-ro-test-mysql80-cache.bash:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 |
3 | # ProxySQL Parameters
4 | export MYSQL_USER="stnduser"
5 | export MYSQL_PASSWORD="stnduser"
6 | export MYSQL_HOST="NLB_endpoint_or_proxysql_instance_ipaddress"
7 | export PORT=3306
8 |
9 | # Sysbench Parameters
10 | export SYSBENCH_DB="sbtest"
11 | export SYSBENCH_THREADS=16
12 | export SYSBENCH_NUM_TABLES=16
13 | export SYSBENCH_TABLE_SIZE=1000000
14 |
15 | sysbench oltp_read_only run \
16 | --report-interval=5 \
17 | --table-size=$SYSBENCH_TABLE_SIZE \
18 | --mysql-host=$MYSQL_HOST \
19 | --mysql-port=$PORT \
20 | --mysql-user=$MYSQL_USER \
21 | --mysql-password=$MYSQL_PASSWORD \
22 | --mysql-db=$SYSBENCH_DB \
23 | --tables=1 \
24 | --time=300 \
25 | --warmup-time=300 \
26 | --threads=128 \
27 | --skip_trx=on \
28 | --db-ps-mode=disable \
29 | --histogram=on \
30 | # --mysql-ignore-errors=2013,2003,1290,1213 \
31 | # --reconnect=1000 \
32 |
33 |
--------------------------------------------------------------------------------
/benchmarking-1.1/oltp-read-only/oltp-ro-test-mysql80.bash:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 |
3 | # MySQL Connection Details
4 | export MYSQL_USER="fill_up"
5 | export MYSQL_PASSWORD="fill_up"
6 | export MYSQL_HOST="fill_up_with_MySQL80_writer_endpoint"
7 | export PORT=3306
8 |
9 | # Sysbench Parameters
10 | export SYSBENCH_DB="sbtest"
11 | export SYSBENCH_THREADS=16
12 | export SYSBENCH_NUM_TABLES=16
13 | export SYSBENCH_TABLE_SIZE=1000000
14 |
15 | sysbench oltp_read_only run \
16 | --report-interval=5 \
17 | --table-size=$SYSBENCH_TABLE_SIZE \
18 | --mysql-host=$MYSQL_HOST \
19 | --mysql-port=$PORT \
20 | --mysql-user=$MYSQL_USER \
21 | --mysql-password=$MYSQL_PASSWORD \
22 | --mysql-db=$SYSBENCH_DB \
23 | --tables=1 \
24 | --time=300 \
25 | --threads=128 \
26 | --skip_trx=on \
27 | --db-ps-mode=disable \
28 | --histogram=on \
29 | # --warmup-time=300 \
30 | # --mysql-ignore-errors=2013,2003,1290,1213 \
31 | # --reconnect=1000 \
32 |
33 |
--------------------------------------------------------------------------------
/images/architecture-diagram.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/aws-samples/proxysql-to-replace-mysql-query-cache/9abd4bfa93567c9fa45141e6257ac62f5fa26e51/images/architecture-diagram.png
--------------------------------------------------------------------------------
/images/init-db-credentials.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/aws-samples/proxysql-to-replace-mysql-query-cache/9abd4bfa93567c9fa45141e6257ac62f5fa26e51/images/init-db-credentials.png
--------------------------------------------------------------------------------
/images/init-db-output.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/aws-samples/proxysql-to-replace-mysql-query-cache/9abd4bfa93567c9fa45141e6257ac62f5fa26e51/images/init-db-output.png
--------------------------------------------------------------------------------
/images/proxysql1-connect.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/aws-samples/proxysql-to-replace-mysql-query-cache/9abd4bfa93567c9fa45141e6257ac62f5fa26e51/images/proxysql1-connect.png
--------------------------------------------------------------------------------
/images/proxysql1-eice.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/aws-samples/proxysql-to-replace-mysql-query-cache/9abd4bfa93567c9fa45141e6257ac62f5fa26e51/images/proxysql1-eice.png
--------------------------------------------------------------------------------
/images/proxysql1.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/aws-samples/proxysql-to-replace-mysql-query-cache/9abd4bfa93567c9fa45141e6257ac62f5fa26e51/images/proxysql1.png
--------------------------------------------------------------------------------
/images/proxysql2-connect.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/aws-samples/proxysql-to-replace-mysql-query-cache/9abd4bfa93567c9fa45141e6257ac62f5fa26e51/images/proxysql2-connect.png
--------------------------------------------------------------------------------
/images/proxysql2-eice.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/aws-samples/proxysql-to-replace-mysql-query-cache/9abd4bfa93567c9fa45141e6257ac62f5fa26e51/images/proxysql2-eice.png
--------------------------------------------------------------------------------
/images/proxysql2.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/aws-samples/proxysql-to-replace-mysql-query-cache/9abd4bfa93567c9fa45141e6257ac62f5fa26e51/images/proxysql2.png
--------------------------------------------------------------------------------
/images/sysbench-eic.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/aws-samples/proxysql-to-replace-mysql-query-cache/9abd4bfa93567c9fa45141e6257ac62f5fa26e51/images/sysbench-eic.png
--------------------------------------------------------------------------------
/images/sysbench-success.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/aws-samples/proxysql-to-replace-mysql-query-cache/9abd4bfa93567c9fa45141e6257ac62f5fa26e51/images/sysbench-success.png
--------------------------------------------------------------------------------
/images/sysbench.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/aws-samples/proxysql-to-replace-mysql-query-cache/9abd4bfa93567c9fa45141e6257ac62f5fa26e51/images/sysbench.png
--------------------------------------------------------------------------------
/proxysql-demo.yml:
--------------------------------------------------------------------------------
1 | AWSTemplateFormatVersion: '2010-09-09'
2 | Description: Create VPC with RDS Aurora MySQL cluster, EC2 instance with ProxySQL
3 |
4 | Parameters:
5 | VpcCIDR:
6 | Type: String
7 | Default: '10.3.0.0/16'
8 | PublicSubnetACIDR:
9 | Type: String
10 | Default: '10.3.1.0/24'
11 | PublicSubnetBCIDR:
12 | Type: String
13 | Default: '10.3.2.0/24'
14 | PrivateSubnetACIDR:
15 | Type: String
16 | Default: '10.3.3.0/24'
17 | PrivateSubnetBCIDR:
18 | Type: String
19 | Default: '10.3.4.0/24'
20 | EC2InstanceAMI:
21 | Type: String
22 | Default: 'ami-0ee42e014ecaa7505'
23 | EC2InstanceConnectCIDR:
24 | Type: String
25 | Default: '3.0.5.32/29'
26 | DBEngine:
27 | Type: String
28 | Default: 'aurora-mysql'
29 | DBEngineVersion:
30 | Type: String
31 | Default: '5.7'
32 | DBClusterParameterGroupFamily:
33 | Type: String
34 | Default: 'aurora-mysql5.7'
35 | DBMasterUsername:
36 | Type: String
37 | Default: 'admin'
38 | DBMasterPassword:
39 | Type: String
40 | Default: 'mysqladmin'
41 |
42 | Resources:
43 | ###### VPC and SUBNETS ######
44 | VPC:
45 | Type: AWS::EC2::VPC
46 | Properties:
47 | CidrBlock: !Ref VpcCIDR
48 | EnableDnsSupport: true
49 | EnableDnsHostnames: true
50 | Tags:
51 | - Key: Name
52 | Value: !Join ['', [!Ref "AWS::StackName", "-VPC" ]]
53 | PublicSubnetA:
54 | Type: AWS::EC2::Subnet
55 | Properties:
56 | VpcId: !Ref VPC
57 | CidrBlock: !Ref PublicSubnetACIDR
58 | AvailabilityZone: !Select [ 0, !GetAZs ]
59 | Tags:
60 | - Key: Name
61 | Value: !Join ['', [!Ref "AWS::StackName", "-PublicSubnetA" ]]
62 | PublicSubnetB:
63 | Type: AWS::EC2::Subnet
64 | Properties:
65 | VpcId: !Ref VPC
66 | CidrBlock: !Ref PublicSubnetBCIDR
67 | AvailabilityZone: !Select [ 1, !GetAZs ]
68 | Tags:
69 | - Key: Name
70 | Value: !Join ['', [!Ref "AWS::StackName", "-PublicSubnetB" ]]
71 | PrivateSubnetA:
72 | Type: AWS::EC2::Subnet
73 | Properties:
74 | VpcId: !Ref VPC
75 | CidrBlock: !Ref PrivateSubnetACIDR
76 | AvailabilityZone: !Select [ 0, !GetAZs ]
77 | Tags:
78 | - Key: Name
79 | Value: !Join ['', [!Ref "AWS::StackName", "-PrivateSubnetA" ]]
80 | PrivateSubnetB:
81 | Type: AWS::EC2::Subnet
82 | Properties:
83 | VpcId: !Ref VPC
84 | CidrBlock: !Ref PrivateSubnetBCIDR
85 | AvailabilityZone: !Select [ 1, !GetAZs ]
86 | Tags:
87 | - Key: Name
88 | Value: !Join ['', [!Ref "AWS::StackName", "-PrivateSubnetB" ]]
89 |
90 | ###### NETWORKING ######
91 | InternetGateway:
92 | Type: AWS::EC2::InternetGateway
93 | AttachGateway:
94 | Type: AWS::EC2::VPCGatewayAttachment
95 | Properties:
96 | VpcId: !Ref VPC
97 | InternetGatewayId: !Ref InternetGateway
98 | NatGatewayAEIP:
99 | Type: AWS::EC2::EIP
100 | NatGatewayA:
101 | Type: AWS::EC2::NatGateway
102 | Properties:
103 | AllocationId: !GetAtt NatGatewayAEIP.AllocationId
104 | SubnetId: !Ref PublicSubnetA
105 | NatGatewayBEIP:
106 | Type: AWS::EC2::EIP
107 | NatGatewayB:
108 | Type: AWS::EC2::NatGateway
109 | Properties:
110 | AllocationId: !GetAtt NatGatewayBEIP.AllocationId
111 | SubnetId: !Ref PublicSubnetB
112 |
113 | PublicRouteTableA:
114 | Type: AWS::EC2::RouteTable
115 | Properties:
116 | VpcId: !Ref VPC
117 | PublicRouteTableB:
118 | Type: AWS::EC2::RouteTable
119 | Properties:
120 | VpcId: !Ref VPC
121 | AssociatePublicRouteTableA:
122 | Type: AWS::EC2::SubnetRouteTableAssociation
123 | Properties:
124 | SubnetId: !Ref PublicSubnetA
125 | RouteTableId: !Ref PublicRouteTableA
126 | AssociatePublicRouteTableB:
127 | Type: AWS::EC2::SubnetRouteTableAssociation
128 | Properties:
129 | SubnetId: !Ref PublicSubnetB
130 | RouteTableId: !Ref PublicRouteTableB
131 | PublicRouteA:
132 | Type: AWS::EC2::Route
133 | DependsOn: AttachGateway
134 | Properties:
135 | RouteTableId: !Ref PublicRouteTableA
136 | DestinationCidrBlock: '0.0.0.0/0'
137 | GatewayId: !Ref InternetGateway
138 | PublicRouteB:
139 | Type: AWS::EC2::Route
140 | DependsOn: AttachGateway
141 | Properties:
142 | RouteTableId: !Ref PublicRouteTableB
143 | DestinationCidrBlock: '0.0.0.0/0'
144 | GatewayId: !Ref InternetGateway
145 |
146 | PrivateRouteTableA:
147 | Type: AWS::EC2::RouteTable
148 | Properties:
149 | VpcId: !Ref VPC
150 | PrivateRouteTableB:
151 | Type: AWS::EC2::RouteTable
152 | Properties:
153 | VpcId: !Ref VPC
154 | AssociatePrivateRouteTableA:
155 | Type: AWS::EC2::SubnetRouteTableAssociation
156 | Properties:
157 | SubnetId: !Ref PrivateSubnetA
158 | RouteTableId: !Ref PrivateRouteTableA
159 | AssociatePrivateRouteTableB:
160 | Type: AWS::EC2::SubnetRouteTableAssociation
161 | Properties:
162 | SubnetId: !Ref PrivateSubnetB
163 | RouteTableId: !Ref PrivateRouteTableB
164 | PrivateRouteA:
165 | Type: AWS::EC2::Route
166 | Properties:
167 | RouteTableId: !Ref PrivateRouteTableA
168 | DestinationCidrBlock: '0.0.0.0/0'
169 | NatGatewayId: !Ref NatGatewayA
170 | PrivateRouteB:
171 | Type: AWS::EC2::Route
172 | Properties:
173 | RouteTableId: !Ref PrivateRouteTableB
174 | DestinationCidrBlock: '0.0.0.0/0'
175 | NatGatewayId: !Ref NatGatewayB
176 |
177 | ###### DATABASE MySQL 5.7 ######
178 | DBCluster57:
179 | Type : "AWS::RDS::DBCluster"
180 | Properties :
181 | Engine : !Ref DBEngine
182 | EngineVersion : !Ref DBEngineVersion
183 | MasterUsername : !Ref DBMasterUsername
184 | MasterUserPassword : !Ref DBMasterPassword
185 | DBSubnetGroupName : !Ref DBSubnetGroup57
186 | DBClusterParameterGroupName : !Ref DBClusterParameterGroup57
187 | VpcSecurityGroupIds :
188 | - !Ref DBSecurityGroup
189 |
190 | DBInstance57:
191 | Type: 'AWS::RDS::DBInstance'
192 | Properties:
193 | DBClusterIdentifier: !Ref DBCluster57
194 | Engine : !Ref DBEngine
195 | PubliclyAccessible: False
196 | DBInstanceClass: db.r6g.large
197 | EnablePerformanceInsights: true
198 |
199 | DBSubnetGroup57:
200 | Type: "AWS::RDS::DBSubnetGroup"
201 | Properties:
202 | DBSubnetGroupDescription : "DB Subnet Group"
203 | SubnetIds :
204 | - !Ref PrivateSubnetA
205 | - !Ref PrivateSubnetB
206 |
207 | DBClusterParameterGroup57:
208 | Type : "AWS::RDS::DBClusterParameterGroup"
209 | Properties :
210 | Description : "DB Cluster Parameter Group"
211 | Family : !Ref DBClusterParameterGroupFamily
212 | Parameters:
213 | time_zone: US/Eastern
214 | character_set_database: utf32
215 |
216 | ###### DATABASE MySQL 8.0 ######
217 | DBCluster80:
218 | Type : "AWS::RDS::DBCluster"
219 | Properties :
220 | Engine : !Ref DBEngine
221 | EngineVersion : '8.0'
222 | MasterUsername : !Ref DBMasterUsername
223 | MasterUserPassword : !Ref DBMasterPassword
224 | DBSubnetGroupName : !Ref DBSubnetGroup80
225 | DBClusterParameterGroupName : !Ref DBClusterParameterGroup80
226 | VpcSecurityGroupIds :
227 | - !Ref DBSecurityGroup
228 |
229 | DBInstance80:
230 | Type: 'AWS::RDS::DBInstance'
231 | Properties:
232 | DBClusterIdentifier: !Ref DBCluster80
233 | Engine : !Ref DBEngine
234 | PubliclyAccessible: False
235 | DBInstanceClass: db.r6g.large
236 | EnablePerformanceInsights: true
237 |
238 | DBSubnetGroup80:
239 | Type: "AWS::RDS::DBSubnetGroup"
240 | Properties:
241 | DBSubnetGroupDescription : "DB Subnet Group"
242 | SubnetIds :
243 | - !Ref PrivateSubnetA
244 | - !Ref PrivateSubnetB
245 |
246 | DBClusterParameterGroup80:
247 | Type : "AWS::RDS::DBClusterParameterGroup"
248 | Properties :
249 | Description : "DB Cluster Parameter Group"
250 | Family : 'aurora-mysql8.0'
251 | Parameters:
252 | time_zone: US/Eastern
253 | character_set_database: utf32
254 |
255 | ###### COMPUTE ######
256 | ProxySQLInstance1:
257 | Type: "AWS::EC2::Instance"
258 | Properties:
259 | ImageId: !Ref EC2InstanceAMI
260 | InstanceType: t4g.medium
261 | NetworkInterfaces:
262 | - AssociatePublicIpAddress: "true"
263 | DeviceIndex: "0"
264 | GroupSet:
265 | - !Ref ProxySQLSecurityGroup
266 | SubnetId: !Ref PrivateSubnetA
267 | Tags:
268 | - Key: Name
269 | Value: !Join ['', [!Ref "AWS::StackName", "-EC2ProxySQL1" ]]
270 | UserData: !Base64
271 | Fn::Sub: |
272 | #!/bin/bash
273 | : '
274 | This script will handle an installation of proxysql and mysql
275 | '
276 |
277 | # Installation
278 | cat <