14 |
14 |
15 |
16 | --------------------------------
17 |
18 |
19 | 
20 |
21 |
22 | --------------------------------
23 |
24 |
25 | 
26 |
27 |
28 | --------------------------------
29 |
30 |
31 | 
32 |
33 |
34 | --------------------------------
35 |
36 |
37 | 
38 |
39 |
40 | ## Features
41 |
42 | - Connect to MySQL/MariaDB databases
43 | - Analyze query performance using Performance Schema
44 | - AI-powered query optimization suggestions
45 | - Performance metrics visualization
46 | - Query history tracking
47 | - Export results as offline HTML
48 |
49 | ## Requirements
50 |
51 | - PHP 8.2+
52 | - Composer
53 | - MySQL/MariaDB database with Performance Schema enabled
54 | - Anthropic API key
55 |
56 | ## Security
57 |
58 | This tool runs queries suggested by the LLM on your database. Queries and their results are sent to the LLM. It is important to use a read-only user with limited privileges.
59 |
60 | The LLM is instructed to only get statistical data from the database, but it cannot be guaranteed that it will follow this instruction. If your queries contain sensitive data, it could be sent to the LLM.
61 |
62 | **This tool is LLM powered SQL injection and should be run only locally and disabled after completing the analysis.**
63 |
64 | ## Installation
65 |
66 | 1. Clone the repository:
67 | ```
68 | git clone https://github.com/soukicz/sql-ai-optimizer.git
69 | cd sql-ai-optimizer
70 | ```
71 |
72 | 2. Install dependencies:
73 | ```
74 | composer install --no-dev
75 | ```
76 |
77 | 3. Create a `.env` file in the project root with your [Anthropic API key](https://console.anthropic.com/settings/keys) and database connection string:
78 | ```
79 | ANTHROPIC_API_KEY=your_api_key_here
80 | DATABASE_URL=mysql://user:password@host:port/
81 | ```
82 |
83 | ## Usage
84 |
85 | 1. Start the application:
86 | ```
87 | # with docker
88 | docker run --rm -it --env-file .env -p 8000:8000 $(docker build -q .)
89 |
90 | # without docker
91 | php -S 127.0.0.1:8000
92 | ```
93 |
94 | 2. Connect to your database via SSH tunnel if needed:
95 | ```
96 | ssh -L 33306:your-db-host:3306 user@your-ssh-server
97 | ```
98 |
99 | 3. Access the application at http://127.0.0.1:8000
100 |
101 | ## Project Structure
102 |
103 | This tool is as simple as possible with few dependencies by design. It is accessing production databases and it should be easy to evaluate what it is doing.
104 |
105 | - `src/` - Application source code
106 | - `templates/` - Twig templates for the web interface
107 | - `data/` - Application data including state database
108 | - `var/` - Cache and logs
109 |
110 | ## MySQL Configuration
111 | ### User
112 | Create a user with read-only privileges. It should get access to at least the performance_schema. You will get best results by giving full SELECT privileges to all tables.
113 |
114 | The database must be either production or a copy of production. This tool is checking query history and needs real traffic to provide useful results.
115 |
116 | ```sql
117 | CREATE USER 'user'@'your_ip' IDENTIFIED BY 'password';
118 | GRANT SELECT, SHOW VIEW ON performance_schema.* TO 'user'@'your_ip';
119 | GRANT SELECT, SHOW VIEW ON *.* TO 'user'@'your_ip';
120 | ```
121 |
122 | ### Performance Schema
123 |
124 | - Enable [performance schema](https://dev.mysql.com/doc/refman/8.4/en/performance-schema-quick-start.html)
125 | - Configure [performance_schema_digests_size](https://dev.mysql.com/doc/refman/8.4/en/performance-schema-system-variables.html#sysvar_performance_schema_digests_size) to store more query digests
126 | - Configure [performance_schema_max_digest_length](https://dev.mysql.com/doc/refman/8.4/en/performance-schema-system-variables.html#sysvar_performance_schema_max_digest_length) to see full queries and to avoid merging of results for queries with the same beginning (default is 1024 characters)
127 | - Enable the larger query table to get exact queries for analysis by running `UPDATE performance_schema.setup_consumers SET enabled = 'YES' WHERE name = 'events_statements_history_long';`
128 |
129 |
130 | Performance statistics should be collected for reasonable time before running the optimizer - at least a few hours, ideally few days. You can reset statistics after optimizing queries by running:
131 |
132 | ```sql
133 | CALL sys.ps_truncate_all_tables(FALSE);
134 | ```
135 |
--------------------------------------------------------------------------------
/assets/analysis-summary.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/soukicz/sql-ai-optimizer/034edde4fea31a05d4a605080af40f779346148c/assets/analysis-summary.png
--------------------------------------------------------------------------------
/assets/group.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/soukicz/sql-ai-optimizer/034edde4fea31a05d4a605080af40f779346148c/assets/group.png
--------------------------------------------------------------------------------
/assets/groups.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/soukicz/sql-ai-optimizer/034edde4fea31a05d4a605080af40f779346148c/assets/groups.png
--------------------------------------------------------------------------------
/assets/index.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/soukicz/sql-ai-optimizer/034edde4fea31a05d4a605080af40f779346148c/assets/index.png
--------------------------------------------------------------------------------
/assets/result.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/soukicz/sql-ai-optimizer/034edde4fea31a05d4a605080af40f779346148c/assets/result.png
--------------------------------------------------------------------------------
/composer.json:
--------------------------------------------------------------------------------
1 | {
2 | "name": "soukicz/sql-ai-optimizer",
3 | "type": "project",
4 | "autoload": {
5 | "psr-4": {
6 | "Soukicz\\SqlAiOptimizer\\": "src/"
7 | }
8 | },
9 | "authors": [
10 | {
11 | "name": "Petr Soukup",
12 | "email": "soukup@simplia.cz"
13 | }
14 | ],
15 | "require": {
16 | "php": "^8.2",
17 | "ext-mysqli": "*",
18 | "ext-sqlite3": "*",
19 | "soukicz/llm": "0.2",
20 | "symfony/framework-bundle": "^7.2",
21 | "symfony/runtime": "^7.2",
22 | "dibi/dibi": "^5.0",
23 | "symfony/dotenv": "^7.2",
24 | "symfony/cache": "^7.2",
25 | "twig/twig": "^3.20",
26 | "league/commonmark": "^2.6",
27 | "spatie/commonmark-highlighter": "^3.0"
28 | },
29 | "config": {
30 | "allow-plugins": {
31 | "symfony/runtime": true
32 | }
33 | },
34 | "require-dev": {
35 | "phpstan/phpstan": "^2.1"
36 | }
37 | }
38 |
--------------------------------------------------------------------------------
/index.php:
--------------------------------------------------------------------------------
1 | extension('framework', [
26 | 'secret' => 'S0ME_SECRET',
27 | ]);
28 |
29 | if (file_exists(__DIR__ . '/.env')) {
30 | // Load environment variables from .env file
31 | $dotenv = new \Symfony\Component\Dotenv\Dotenv();
32 | $dotenv->loadEnv(__DIR__ . '/.env');
33 | }
34 |
35 | $container->services()
36 | ->load('Soukicz\\SqlAiOptimizer\\', __DIR__ . '/src/*')
37 | ->autowire()
38 | ->autoconfigure();
39 |
40 | $container->services()
41 | ->set(PerformanceSchemaQueryTool::class)
42 | ->arg('$cacheDatabaseResults', false)
43 | ->autowire()
44 | ->autoconfigure();
45 |
46 | $container->services()
47 | ->set(QueryTool::class)
48 | ->arg('$cacheDatabaseResults', false)
49 | ->autowire()
50 | ->autoconfigure();
51 |
52 | $container->services()
53 | ->set(FileCache::class)
54 | ->arg('$cacheDir', __DIR__ . '/var/cache')
55 | ->autowire()
56 | ->autoconfigure();
57 |
58 | $container->services()
59 | ->set(\Symfony\Component\Cache\Adapter\FilesystemAdapter::class)
60 | ->arg('$namespace', '')
61 | ->arg('$defaultLifetime', 0)
62 | ->arg('$directory', __DIR__ . '/var/cache')
63 | ->autowire()
64 | ->autoconfigure();
65 |
66 | $container->services()
67 | ->set(AnthropicClient::class)
68 | ->arg('$apiKey', '%env(ANTHROPIC_API_KEY)%')
69 | ->arg('$cache', new Reference(FileCache::class))
70 | ->autowire()
71 | ->autoconfigure();
72 |
73 | $container->services()
74 | ->set(MarkdownFormatter::class)
75 | ->autowire()
76 | ->autoconfigure();
77 |
78 | $container->services()
79 | ->set(LLMFileLogger::class)
80 | ->arg('$logPath', __DIR__ . '/var/log/llm.md')
81 | ->arg('$formatter', new Reference(MarkdownFormatter::class))
82 | ->autowire()
83 | ->autoconfigure();
84 |
85 | $container->services()
86 | ->set(LLMChainClient::class)
87 | ->arg('$logger', new Reference(LLMFileLogger::class))
88 | ->autowire()
89 | ->autoconfigure();
90 |
91 | $container->services()
92 | ->set(StateDatabase::class)
93 | ->arg('$databasePath', __DIR__ . '/data/state.sqlite')
94 | ->autowire()
95 | ->autoconfigure();
96 |
97 | $container->services()
98 | ->set(\Soukicz\SqlAiOptimizer\Service\QueryResultFormatter::class)
99 | ->autowire()
100 | ->autoconfigure();
101 |
102 | $container->services()
103 | ->set(\Soukicz\SqlAiOptimizer\Service\DatabaseQueryExecutor::class)
104 | ->arg('$cacheTtl', 24 * 60 * 60)
105 | ->autowire()
106 | ->autoconfigure();
107 |
108 | // Register Twig
109 | $container->services()
110 | ->set('twig.loader', \Twig\Loader\FilesystemLoader::class)
111 | ->arg('$paths', [__DIR__ . '/templates'])
112 | ->autowire()
113 | ->autoconfigure();
114 |
115 | $container->services()
116 | ->set(\Twig\Environment::class)
117 | ->arg('$loader', new Reference('twig.loader'))
118 | ->arg('$options', [
119 | 'cache' => __DIR__ . '/var/cache/twig',
120 | 'debug' => '%env(bool:APP_DEBUG)%',
121 | ])
122 | ->autowire()
123 | ->autoconfigure();
124 |
125 | // Register controllers
126 | $container->services()
127 | ->load('Soukicz\\SqlAiOptimizer\\Controller\\', __DIR__ . '/src/Controller/')
128 | ->tag('controller.service_arguments')
129 | ->autowire()
130 | ->autoconfigure();
131 | }
132 |
133 | protected function configureRoutes(RoutingConfigurator $routes): void {
134 | $routes->import(__DIR__ . '/src/Controller/', 'attribute');
135 | }
136 | }
137 |
138 | return static function (array $context) {
139 | return new Kernel($context['APP_ENV'], (bool) $context['APP_DEBUG']);
140 | };
141 |
--------------------------------------------------------------------------------
/phpstan.neon:
--------------------------------------------------------------------------------
1 | parameters:
2 | level: 5
3 | paths:
4 | - src
5 |
--------------------------------------------------------------------------------
/src/AnalyzedDatabase.php:
--------------------------------------------------------------------------------
1 | hostname = $parsedUrl['host'];
19 | $this->port = $parsedUrl['port'] ?? null;
20 | $dbConfig = [
21 | 'driver' => 'mysqli',
22 | 'host' => $parsedUrl['host'],
23 | 'username' => $parsedUrl['user'],
24 | 'password' => $parsedUrl['pass'],
25 | 'database' => ltrim($parsedUrl['path'], '/'),
26 | 'port' => $parsedUrl['port'] ?? null,
27 | 'charset' => 'utf8mb4',
28 | 'lazy' => true,
29 | ];
30 |
31 | $this->connection = new Connection($dbConfig);
32 | }
33 |
34 | public function getHostnameWithPort(): string {
35 | if ($this->port) {
36 | return $this->hostname . ':' . $this->port;
37 | }
38 |
39 | return $this->hostname;
40 | }
41 |
42 | public function getConnection(): Connection {
43 | return $this->connection;
44 | }
45 |
46 | public function getQueryText(string $digest, string $schema): ?string {
47 | foreach (['events_statements_history', 'events_statements_history_long'] as $table) {
48 | $sql = $this->connection->query('SELECT sql_text FROM performance_schema.%n WHERE digest=%s', $table, $digest, ' AND current_schema = %s', $schema)->fetchSingle();
49 | if ($sql) {
50 | return $sql;
51 | }
52 | }
53 |
54 | return null;
55 | }
56 |
57 | public function getQueryTexts(array $digests): array {
58 | $sqls = [];
59 |
60 | foreach (['events_statements_history', 'events_statements_history_long'] as $table) {
61 | $list = $this->connection->query('SELECT sql_text,digest,current_schema FROM performance_schema.%n', $table, ' WHERE digest IN (%s)', $digests)->fetchAll();
62 | foreach ($list as $item) {
63 | $sqls[] = [
64 | 'sql_text' => $item['sql_text'],
65 | 'digest' => $item['digest'],
66 | 'current_schema' => $item['current_schema'],
67 | ];
68 | }
69 | }
70 |
71 | return $sqls;
72 | }
73 | }
74 |
--------------------------------------------------------------------------------
/src/Controller/AnalysisController.php:
--------------------------------------------------------------------------------
1 | request->all('query_ids'));
34 |
35 | $run = $this->stateDatabase->getRun($id);
36 | if (!$run) {
37 | throw new \Exception('Run not found');
38 | }
39 |
40 | $promises = [];
41 | foreach ($queryIds as $queryId) {
42 | $queryData = $this->stateDatabase->getQuery($queryId);
43 | $queryObject = new CandidateQuery(
44 | schema: $queryData['schema'],
45 | digest: $queryData['digest'],
46 | normalizedQuery: $queryData['normalized_query'],
47 | impactDescription: $queryData['impact_description'],
48 | );
49 |
50 | $promises[] = $this->queryAnalyzer->analyzeQuery((int)$queryId, $queryData['real_query'], $queryObject, $run['use_real_query'], $run['use_database_access']);
51 | }
52 |
53 | // Process 5 promises concurrently
54 | Each::ofLimit($promises, 5)->wait();
55 |
56 | return new JsonResponse([
57 | 'url' => $this->router->generate('query.detail', ['queryId' => $queryIds[0]]),
58 | ]);
59 | }
60 |
61 | #[Route('/run/{id}/continue', name: 'run.continue', methods: ['POST'])]
62 | public function continuePrompt(Request $request, int $id): Response {
63 | $queryData = $this->stateDatabase->getQuery($id);
64 | $run = $this->stateDatabase->getRun($queryData['run_id']);
65 | /** @var \Soukicz\Llm\LLMResponse $response */
66 | $response = $this->queryAnalyzer->continueConversation(
67 | conversation: LLMConversation::fromJson(json_decode($queryData['llm_conversation'], true)),
68 | prompt: $request->request->get('input'),
69 | useDatabaseAccess: $run['use_database_access']
70 | )->wait();
71 |
72 | $this->stateDatabase->updateConversation(
73 | queryId: $id,
74 | conversation: $response->getConversation(),
75 | conversationMarkdown: $this->markdownFormatter->responseToMarkdown($response)
76 | );
77 |
78 | return new JsonResponse([
79 | 'url' => $this->router->generate('query.detail', ['queryId' => $id]),
80 | ]);
81 | }
82 |
83 | #[Route('/query/{queryId}', name: 'query.detail')]
84 | public function queryDetail(int $queryId, Request $request): Response {
85 | $query = $this->stateDatabase->getQuery($queryId);
86 | if (!$query) {
87 | throw new \Exception('Query not found');
88 | }
89 |
90 | $group = $this->stateDatabase->getGroup($query['group_id']);
91 |
92 | $conversation = LLMConversation::fromJson(json_decode($query['llm_conversation'], true));
93 |
94 | $messages = [];
95 | $firstUser = true;
96 | foreach ($conversation->getMessages() as $message) {
97 | if (!$message->isAssistant() && !$message->isUser()) {
98 | continue;
99 | }
100 | if ($message->isUser() && $firstUser) {
101 | $firstUser = false;
102 | continue;
103 | }
104 |
105 | if ($message->isUser()) {
106 | foreach ($message->getContents() as $content) {
107 | if ($content instanceof LLMMessageText) {
108 | $messages[] = [
109 | 'role' => 'user',
110 | 'content' => nl2br(htmlspecialchars($content->getText(), ENT_QUOTES)),
111 | ];
112 | }
113 | }
114 | } else {
115 | $onlyText = true;
116 |
117 | foreach ($message->getContents() as $content) {
118 | if ($content instanceof LLMMessageReasoning) {
119 | continue;
120 | }
121 |
122 | if (!($content instanceof LLMMessageText)) {
123 | $onlyText = false;
124 | break;
125 | }
126 | }
127 | if ($onlyText) {
128 | foreach ($message->getContents() as $content) {
129 | if ($content instanceof LLMMessageText) {
130 | $messages[] = [
131 | 'role' => 'assistant',
132 | 'content' => $this->renderMarkdownWithHighlighting($content->getText()),
133 | ];
134 | }
135 | }
136 | }
137 | }
138 | }
139 |
140 | if (empty($query['real_query'])) {
141 | $sql = Helpers::dump($query['normalized_query'], true);
142 | } else {
143 | $sql = Helpers::dump($query['real_query'], true);
144 | }
145 | $sql = preg_replace('/^Exported on {{ "now"|date('Y-m-d H:i:s') }}
21 |{{ query.schema }}
34 |{{ group.name }}
38 |{{ query.impact_description }}
43 | 44 |{{ sql|raw }}
46 |
47 | {% if query.real_query is empty %}
48 | Exported on {{ "now"|date('Y-m-d H:i:s') }}
28 |