├── CHANGELOG.md ├── LICENSE.md ├── README.md ├── composer.json ├── config └── scout-qdrant.php ├── database ├── .gitignore └── migrations │ └── create_vectorization_metadata_table.php └── src ├── Commands ├── QdrantInstallCommand.php ├── QdrantRestartCommand.php ├── QdrantStartCommand.php ├── QdrantStatusCommand.php ├── QdrantStopCommand.php └── QdrantUpdateCommand.php ├── LaravelScoutQdrantServiceProvider.php ├── Models ├── HasRecommendations.php ├── UsesVectorization.php └── Vectorizable.php ├── Scout └── QdrantScoutEngine.php └── Vectorizer ├── HasOptions.php ├── Manager └── VectorizerEngineManager.php ├── OpenAIVectorizer.php └── VectorizerInterface.php /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | # Changelog 2 | 3 | ## [0.2.0] - 2023-05-29 4 | 5 | - Added `qdrant:update` command to update Qdrant Docker container. 6 | - Added `--restart` argument to `qdrant:start` and `qdrant:restart` commands for specifying Docker container restart policy. 7 | - Added `--kill` argument to `qdrant:stop` command to force stop the container. 8 | - Renamed `qdrant:terminate` command to `qdrant:stop` with an alias for backwards compatibility. 9 | - Updated README with Qdrant installation instructions and usage updates. 10 | 11 | ## [0.1.0] - 2023-05-27 12 | 13 | - Initial release of Laravel Scout Qdrant Drivers. 14 | - Integration of Scout, Qdrant, and OpenAI to provide vector search capabilities in Laravel. 15 | - Artisan commands for managing Qdrant Docker container - installation, start, restart, status check, and termination. 16 | - Configurations for Qdrant including host, key, storage, and vectorizer. 17 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) gregpriday 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in 13 | all copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 21 | THE SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Laravel Scout Qdrant Drivers 2 | 3 | The Laravel Scout Qdrant Drivers package introduces vector search capabilities within Laravel applications by leveraging Scout, Qdrant, and OpenAI. This package transforms your application's data into vectors using OpenAI, then indexes and makes them searchable using Qdrant, a powerful vector database management system. 4 | 5 | > **Note**: This package is a work in progress and might not be ready for production use yet. However, with growing interest and support, the plan is to expand and improve it continually. 6 | 7 | ## Prerequisites 8 | 9 | - [Qdrant](https://qdrant.tech/documentation/) - Qdrant installation is a prerequisite for this package. We recommend using [Qdrant Cloud](https://qdrant.tech/documentation/cloud/) for a more scalable and robust solution, but local installation is also possible. See Qdrant installation instructions [here](https://qdrant.tech/documentation/quick_start/#installation). 10 | 11 | - [OpenAI for Laravel](https://github.com/openai-php/laravel) - OpenAI setup for Laravel is also necessary. Publish the service provider using: 12 | 13 | ```bash 14 | php artisan vendor:publish --provider="OpenAI\Laravel\ServiceProvider" 15 | ``` 16 | 17 | Follow the instructions on the [OpenAI for Laravel page](https://github.com/openai-php/laravel) to configure OpenAI variables. 18 | 19 | ## Installation 20 | 21 | Install the package via Composer: 22 | 23 | ```bash 24 | composer require gregpriday/laravel-scout-qdrant 25 | ``` 26 | 27 | Add migrations with: 28 | 29 | ```bash 30 | php artisan vendor:publish --tag="scout-qdrant-migrations" 31 | ``` 32 | 33 | Publish the configuration file with: 34 | 35 | ```bash 36 | php artisan vendor:publish --tag="scout-qdrant-config" 37 | ``` 38 | 39 | ## Configuration 40 | 41 | After installation, you should configure the `qdrant` settings in your `config/scout-qdrant.php` file: 42 | 43 | ```php 44 | return [ 45 | 'qdrant' => [ 46 | 'host' => env('QDRANT_HOST', 'http://localhost'), 47 | 'key' => env('QDRANT_API_KEY', null), 48 | 'storage' => env('QDRANT_STORAGE', 'database/qdrant'), 49 | ], 50 | 'vectorizer' => env('QDRANT_VECTORIZER', 'openai'), 51 | ]; 52 | ``` 53 | 54 | The `QDRANT_HOST` key defines the location of your Qdrant service. If you are using Qdrant Cloud or a Docker container on a different server, update this value accordingly. The `QDRANT_API_KEY` key is for specifying your Qdrant API key if necessary. 55 | 56 | The `QDRANT_STORAGE` key indicates the location where Qdrant will store its files. By default, this is set to `database/qdrant`, but you can specify a different location depending on your setup. 57 | 58 | The `QDRANT_VECTORIZER` key is used to define the vectorizer to be used. By default, it's set to 'openai'. If you have a custom vectorizer, you can specify it here. 59 | 60 | For more details on configuring Qdrant, please refer to the [Qdrant documentation](https://qdrant.tech/documentation/install/). 61 | 62 | Additionally, ensure Scout is configured to use the `qdrant` driver by setting the `SCOUT_DRIVER` in your `.env` file: 63 | 64 | ```env 65 | SCOUT_DRIVER=qdrant 66 | ``` 67 | 68 | Your model should also include a `toSearchableArray` method that includes a `vector` key. This key gets converted into a vector using OpenAI: 69 | 70 | ```php 71 | public function toSearchableArray() 72 | { 73 | return [ 74 | 'id' => $this->id, 75 | 'name' => $this->name, 76 | 'vector' => $this->text, 77 | // more attributes... 78 | ]; 79 | } 80 | ``` 81 | 82 | ## Usage 83 | 84 | You can use the package just as you would use Laravel Scout, but with the added advantage of vector-based searches. This functionality offers more precise and complex search results. 85 | 86 | For additional instructions on usage, please visit the [Laravel Scout documentation](https://laravel.com/docs/scout). 87 | 88 | ## Qdrant Docker Management Commands 89 | 90 | Manage your Qdrant Docker container with the following commands: 91 | 92 | ### Install Qdrant 93 | 94 | ```bash 95 | php artisan qdrant:install 96 | ``` 97 | 98 | This command pulls the Qdrant Docker image and checks whether Docker is installed on your machine. If it's not, the command provides instructions on installation. 99 | 100 | ### Start Qdrant 101 | 102 | ```bash 103 | php artisan qdrant:start 104 | ``` 105 | 106 | Starts your Qdrant Docker container with the default port set to 6333, storage path as `database/qdrant`, and restart policy as `unless-stopped`. You can specify a different port, storage path or restart policy with the `--port`, `--storage`, and `--restart` options, respectively: 107 | 108 | ```bash 109 | php artisan qdrant:start --port=6334 --storage=custom/qdrant --restart=always 110 | ``` 111 | 112 | ### Restart Qdrant 113 | 114 | ```bash 115 | php artisan qdrant:restart 116 | ``` 117 | 118 | Restarts your Qdrant Docker container. This command accepts the `--port`, `--storage`, and `--restart` options. 119 | 120 | ```bash 121 | php artisan qdrant:restart --port=6334 --storage=custom/qdrant --restart=always 122 | ``` 123 | 124 | ### Check Qdrant Status 125 | 126 | ```bash 127 | php artisan qdrant:status 128 | ``` 129 | 130 | Provides the status of your Qdrant Docker container, including details like container ID, image, command, creation time, status, ports, and name. 131 | 132 | ### Stop Qdrant 133 | 134 | ```bash 135 | php artisan qdrant:stop 136 | ``` 137 | 138 | Stops your Qdrant Docker container. Use `--kill` to kill the container instead of stopping it. 139 | 140 | ## Creating a Custom Vectorizer 141 | 142 | To create a custom vectorizer, ensure you have a custom model or a third-party service. Then, create a new class that implements `GregPriday\ScoutQdrant\Vectorizer`. This interface requires a single method: `vectorize(string $text): array`. 143 | 144 | Example: 145 | 146 | ```php 147 | use GregPriday\LaravelScoutQdrant\Vectorizer\VectorizerInterface; 148 | 149 | class MyVectorizer implements VectorizerInterface 150 | { 151 | public function embedDocument(string $text): array 152 | { 153 | // Create a vector from the text using your model 154 | } 155 | 156 | public function embedQuery(string $text): array 157 | { 158 | // Create a vector from the text using your model 159 | } 160 | } 161 | ``` 162 | 163 | Specify your custom vectorizer in your `scout-qdrant.php` configuration file: 164 | 165 | ```php 166 | return [ 167 | // other config values... 168 | 'vectorizer' => App\MyVectorizer::class, 169 | ]; 170 | ``` 171 | 172 | Now your custom vectorizer will be used to create vectors for your Scout records. 173 | 174 | ## Testing 175 | 176 | Start Qdrant for testing with: 177 | 178 | ```bash 179 | docker pull qdrant/qdrant 180 | docker run -p 6333:6333 -v $(pwd)/database/qdrant:/qdrant/storage qdrant/qdrant 181 | ```` 182 | 183 | Execute tests with: 184 | 185 | ```bash 186 | composer test 187 | ``` 188 | 189 | ## Changelog 190 | 191 | Visit the [CHANGELOG](CHANGELOG.md) for updates and changes. 192 | 193 | ## Contributing 194 | 195 | Guidelines for contributing can be found in the [CONTRIBUTING](CONTRIBUTING.md) file. 196 | 197 | ## Security Vulnerabilities 198 | 199 | If you discover a security vulnerability, please follow our [security policy](../../security/policy) to report it. 200 | 201 | ## Credits 202 | 203 | - [Greg Priday](https://github.com/gregpriday) 204 | - [All Contributors](../../contributors) 205 | 206 | ## License 207 | 208 | The Laravel Scout Qdrant Drivers is open-source software licensed under the [MIT license](LICENSE.md). 209 | -------------------------------------------------------------------------------- /composer.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "gregpriday/laravel-scout-qdrant", 3 | "description": "This is my package laravel-scout-qdrant", 4 | "keywords": [ 5 | "laravel", 6 | "laravel-scout-qdrant", 7 | "vector search" 8 | ], 9 | "homepage": "https://github.com/gregpriday/laravel-scout-qdrant", 10 | "license": "MIT", 11 | "authors": [ 12 | { 13 | "name": "Greg Priday", 14 | "email": "greg@siteorigin.com", 15 | "role": "Developer" 16 | } 17 | ], 18 | "require": { 19 | "php": "^8.1", 20 | "guzzlehttp/guzzle": "^7.7", 21 | "hkulekci/qdrant": "0.4", 22 | "illuminate/contracts": "^10.0", 23 | "laravel/scout": "^10.2", 24 | "spatie/laravel-package-tools": "^1.14.0" 25 | }, 26 | "require-dev": { 27 | "laravel/pint": "^1.0", 28 | "nunomaduro/collision": "^7.9", 29 | "orchestra/testbench": "^8.0", 30 | "pestphp/pest": "^2.0", 31 | "pestphp/pest-plugin-arch": "^2.0", 32 | "pestphp/pest-plugin-laravel": "^2.0", 33 | "phpunit/phpunit": "^10.1", 34 | "spatie/laravel-ray": "^1.26" 35 | }, 36 | "autoload": { 37 | "psr-4": { 38 | "GregPriday\\LaravelScoutQdrant\\": "src/" 39 | } 40 | }, 41 | "autoload-dev": { 42 | "psr-4": { 43 | "GregPriday\\LaravelScoutQdrant\\Tests\\": "tests/" 44 | } 45 | }, 46 | "scripts": { 47 | "post-autoload-dump": "@php ./vendor/bin/testbench package:discover --ansi", 48 | "analyse": "vendor/bin/phpstan analyse", 49 | "test": "vendor/bin/pest", 50 | "test-coverage": "vendor/bin/pest --coverage", 51 | "format": "vendor/bin/pint" 52 | }, 53 | "config": { 54 | "sort-packages": true, 55 | "allow-plugins": { 56 | "pestphp/pest-plugin": true, 57 | "phpstan/extension-installer": true, 58 | "php-http/discovery": true 59 | } 60 | }, 61 | "extra": { 62 | "laravel": { 63 | "providers": [ 64 | "GregPriday\\LaravelScoutQdrant\\LaravelScoutQdrantServiceProvider" 65 | ], 66 | "aliases": { 67 | "LaravelScoutQdrant": "GregPriday\\LaravelScoutQdrant\\Facades\\LaravelScoutQdrant" 68 | } 69 | } 70 | }, 71 | "minimum-stability": "stable", 72 | "prefer-stable": true 73 | } 74 | -------------------------------------------------------------------------------- /config/scout-qdrant.php: -------------------------------------------------------------------------------- 1 | [ 7 | 'host' => env('QDRANT_HOST', 'http://localhost'), 8 | 'key' => env('QDRANT_API_KEY', null), 9 | 'storage' => env('QDRANT_STORAGE', 'database/qdrant'), 10 | ], 11 | 'vectorizer' => env('QDRANT_VECTORIZER', 'openai'), 12 | 13 | // Add full model classnames for any classes that exist outside the App\Models namespace. 14 | 'models' => [], 15 | ]; 16 | -------------------------------------------------------------------------------- /database/.gitignore: -------------------------------------------------------------------------------- 1 | qdrant 2 | -------------------------------------------------------------------------------- /database/migrations/create_vectorization_metadata_table.php: -------------------------------------------------------------------------------- 1 | morphs('vectorizable'); 14 | 15 | // This is used to track the vectorizer that created the vector 16 | $table->string('vectorizer'); 17 | $table->text('vectorizer_options'); 18 | 19 | // The field that's being vectorized 20 | $table->string('field_name'); 21 | $table->string('field_hash'); 22 | }); 23 | } 24 | 25 | public function down() 26 | { 27 | Schema::dropIfExists('vectorization_metadata'); 28 | } 29 | }; 30 | -------------------------------------------------------------------------------- /src/Commands/QdrantInstallCommand.php: -------------------------------------------------------------------------------- 1 | error('Docker is not installed. Please visit https://docs.docker.com/engine/install/ for instructions on how to install Docker.'); 19 | return; 20 | } 21 | 22 | // Set the Docker image tag 23 | $version = $this->argument('version'); 24 | $tag = $version ? "qdrant/qdrant:$version" : 'qdrant/qdrant'; 25 | 26 | exec("docker pull $tag", $output, $return_var); 27 | 28 | if ($return_var !== 0) { 29 | $this->error('Failed to pull Qdrant Docker image'); 30 | } else { 31 | $this->info('Successfully pulled Qdrant Docker image'); 32 | } 33 | } 34 | } 35 | -------------------------------------------------------------------------------- /src/Commands/QdrantRestartCommand.php: -------------------------------------------------------------------------------- 1 | call('qdrant:terminate'); 18 | 19 | if ($stopExitCode !== 0) { 20 | $this->error('Failed to stop Qdrant Docker container'); 21 | return; 22 | } 23 | 24 | // Then, start a new container 25 | $startExitCode = $this->call('qdrant:start', [ 26 | '--storage' => $this->option('storage'), 27 | '--port' => $this->option('port'), 28 | '--restart' => $this->option('restart'), 29 | ]); 30 | 31 | if ($startExitCode !== 0) { 32 | $this->error('Failed to start Qdrant Docker container'); 33 | } else { 34 | $this->info('Successfully restarted Qdrant Docker container'); 35 | } 36 | } 37 | } 38 | -------------------------------------------------------------------------------- /src/Commands/QdrantStartCommand.php: -------------------------------------------------------------------------------- 1 | comment('Qdrant is already running.'); 19 | return; 20 | } 21 | 22 | $storageDir = $this->option('storage') ?? config('scout-qdrant.qdrant.storage', 'database/qdrant'); 23 | // Make sure the directory is absolute 24 | if ($storageDir[0] !== '/') { 25 | $storageDir = getcwd() . '/' . $storageDir; 26 | } 27 | 28 | $port = $this->option('port'); 29 | $restartPolicy = $this->option('restart'); 30 | 31 | exec("docker run --restart=$restartPolicy -d -p $port:6333 -v $storageDir:/qdrant/storage qdrant/qdrant", $output, $return_var); 32 | 33 | if ($return_var !== 0) { 34 | $this->error('Failed to start Qdrant Docker container'); 35 | } else { 36 | $this->info('Successfully started Qdrant Docker container'); 37 | } 38 | } 39 | } 40 | -------------------------------------------------------------------------------- /src/Commands/QdrantStatusCommand.php: -------------------------------------------------------------------------------- 1 | info('Qdrant is not running.'); 21 | } else { 22 | $this->info('Qdrant is running. Here are the details:'); 23 | 24 | $headers = ['CONTAINER ID', 'IMAGE', 'COMMAND', 'CREATED', 'STATUS', 'PORTS', 'NAMES']; 25 | 26 | // Split each line of the output into an array of columns 27 | $rows = array_map(function($line) { 28 | // Using trim to remove trailing newline 29 | return Str::of($line)->explode("\t")->map(function ($item) { 30 | return trim($item, '"'); 31 | })->toArray(); 32 | }, $output); 33 | 34 | // Display the output as a table 35 | $this->table($headers, $rows); 36 | } 37 | } 38 | } 39 | -------------------------------------------------------------------------------- /src/Commands/QdrantStopCommand.php: -------------------------------------------------------------------------------- 1 | comment('Qdrant is not running.'); 21 | return; 22 | } 23 | 24 | // Extract container ID 25 | $containerId = preg_split('/\s+/', $output[0])[0]; 26 | 27 | if ($this->option('kill')) { 28 | exec("docker kill $containerId", $output, $return_var); 29 | $action = 'killed'; 30 | } else { 31 | exec("docker stop $containerId", $output, $return_var); 32 | $action = 'stopped'; 33 | } 34 | 35 | if ($return_var !== 0) { 36 | $this->error("Failed to $action Qdrant Docker container"); 37 | } else { 38 | $this->info("Successfully $action Qdrant Docker container"); 39 | } 40 | } 41 | } 42 | -------------------------------------------------------------------------------- /src/Commands/QdrantUpdateCommand.php: -------------------------------------------------------------------------------- 1 | error('Docker is not installed. Please visit https://docs.docker.com/engine/install/ for instructions on how to install Docker.'); 19 | return; 20 | } 21 | 22 | // Set the Docker image tag 23 | $version = $this->argument('version'); 24 | $tag = $version ? "qdrant/qdrant:$version" : 'qdrant/qdrant'; 25 | 26 | exec("docker pull $tag", $output, $return_var); 27 | 28 | if ($return_var !== 0) { 29 | $this->error('Failed to update Qdrant Docker image'); 30 | } else { 31 | // Check if the Qdrant container is running 32 | exec('docker ps --filter ancestor=qdrant/qdrant --format "{{.ID}}"', $runningContainers, $return_var); 33 | if (!empty($runningContainers)) { 34 | // Stop the running Qdrant container 35 | $s = exec('docker stop $(docker ps -q --filter ancestor=qdrant/qdrant)', $output, $return_var); 36 | if ($return_var !== 0) { 37 | $this->error('Failed to stop running Qdrant Docker container'); 38 | return; 39 | } 40 | } 41 | 42 | // Remove the old Qdrant container (if it exists) 43 | exec('docker ps -a -q --filter ancestor=qdrant/qdrant', $allContainers, $return_var); 44 | if (!empty($allContainers)) { 45 | exec('docker rm $(docker ps -a -q --filter ancestor=qdrant/qdrant)', $output, $return_var); 46 | if ($return_var !== 0) { 47 | $this->error('Failed to remove old Qdrant Docker container'); 48 | return; 49 | } 50 | } 51 | 52 | $this->info('Successfully updated Qdrant Docker image. Please run `php artisan qdrant:start` to start the new container.'); 53 | } 54 | } 55 | } 56 | -------------------------------------------------------------------------------- /src/LaravelScoutQdrantServiceProvider.php: -------------------------------------------------------------------------------- 1 | name('laravel-scout-qdrant') 26 | ->hasConsoleCommands([ 27 | QdrantInstallCommand::class, 28 | QdrantRestartCommand::class, 29 | QdrantStartCommand::class, 30 | QdrantStatusCommand::class, 31 | QdrantStopCommand::class, 32 | QdrantUpdateCommand::class, 33 | ]) 34 | ->hasConfigFile('scout-qdrant') 35 | ->hasMigration('create_vectorization_metadata_table'); 36 | } 37 | 38 | public function packageRegistered() 39 | { 40 | $this->app->singleton(Qdrant::class, function () { 41 | $key = config('scout-qdrant.qdrant.key'); 42 | 43 | $config = new Config(config('scout-qdrant.qdrant.host')); 44 | if($key) { 45 | $config->setApiKey($key); 46 | } 47 | 48 | return new Qdrant(new GuzzleClient($config)); 49 | }); 50 | 51 | $this->app->singleton(VectorizerEngineManager::class, function ($app) { 52 | return new VectorizerEngineManager($app); 53 | }); 54 | } 55 | 56 | public function packageBooted() 57 | { 58 | resolve(EngineManager::class)->extend('qdrant', function ($app) { 59 | return new QdrantScoutEngine( 60 | app(Qdrant::class), 61 | app(VectorizerEngineManager::class) 62 | ); 63 | }); 64 | } 65 | } 66 | -------------------------------------------------------------------------------- /src/Models/HasRecommendations.php: -------------------------------------------------------------------------------- 1 | where('vectorizable_type', get_class($model)) 16 | ->where('vectorizable_id', $model->id) 17 | ->delete(); 18 | }); 19 | } 20 | 21 | public function getDefaultVectorizer(): string 22 | { 23 | return $this->defaultVectorizer ?? config('scout-qdrant.vectorizer', 'openai'); 24 | } 25 | 26 | public function getDefaultVectorField(): string 27 | { 28 | return $this->defaultVectorField ?? array_key_first($this->getVectorizers()); 29 | } 30 | 31 | public function getVectorizers(): array 32 | { 33 | return $this->vectorizers ?? [ 34 | 'document' => $this->getDefaultVectorizer(), 35 | ]; 36 | } 37 | 38 | public function hasVectorFieldChanged($name, $value): bool 39 | { 40 | $vectorizerName = $this->getVectorizers()[$name]; 41 | $vectorizer = app(VectorizerEngineManager::class)->driver($vectorizerName); 42 | 43 | return ! DB::table('vectorization_metadata') 44 | ->where('vectorizable_id', $this->getKey()) 45 | ->where('vectorizable_type', get_class($this)) 46 | ->where('vectorizer', $vectorizerName) 47 | ->where('vectorizer_options', json_encode($vectorizer->getOptions())) 48 | ->where('field_name', $name) 49 | ->where('field_hash', '=', hash('sha256', $value)) 50 | ->exists(); 51 | } 52 | 53 | public function setVectorFieldHash($name, $value) 54 | { 55 | $vectorizerName = $this->getVectorizers()[$name]; 56 | $vectorizer = app(VectorizerEngineManager::class)->driver($vectorizerName); 57 | 58 | // Delete the old hash data 59 | DB::table('vectorization_metadata') 60 | ->where('vectorizable_id', $this->getKey()) 61 | ->where('vectorizable_type', get_class($this)) 62 | ->where('field_name', $name) 63 | ->delete(); 64 | 65 | DB::table('vectorization_metadata') 66 | ->insert([ 67 | 'vectorizable_id' => $this->getKey(), 68 | 'vectorizable_type' => get_class($this), 69 | 'vectorizer' => $vectorizerName, 70 | 'vectorizer_options' => json_encode($vectorizer->getOptions()), 71 | 'field_name' => $name, 72 | 'field_hash' => hash('sha256', $value), 73 | ]); 74 | } 75 | 76 | public function getVectorFieldHash($name): string|null 77 | { 78 | return DB::table('vectorization_metadata') 79 | ->where('vectorizable_id', $this->getKey()) 80 | ->where('vectorizable_type', get_class($this)) 81 | ->where('vectorizer', $this->getVectorizers()[$name]) 82 | ->where('field_name', $name) 83 | ->value('field_hash'); 84 | } 85 | } 86 | -------------------------------------------------------------------------------- /src/Models/Vectorizable.php: -------------------------------------------------------------------------------- 1 | vectorizer pairs. 9 | */ 10 | public function getVectorizers(): array; 11 | 12 | public function getDefaultVectorField(): string; 13 | } 14 | -------------------------------------------------------------------------------- /src/Scout/QdrantScoutEngine.php: -------------------------------------------------------------------------------- 1 | qdrant = $qdrant; 33 | $this->vectorizerEngineManager = $vectorizerEngineManager; 34 | } 35 | 36 | public function update($models) 37 | { 38 | if ($models->isEmpty()) { 39 | return; 40 | } 41 | 42 | $collectionName = $models->first()->searchableAs(); 43 | $points = new PointsStruct(); 44 | 45 | foreach ($models as $model) { 46 | $searchableData = $model->toSearchableArray(); 47 | 48 | if (empty($searchableData)) { 49 | continue; 50 | } 51 | 52 | // Get the current point from the collection 53 | try{ 54 | $currentPoint = $this->qdrant->collections($collectionName)->points()->id($model->getScoutKey()); 55 | $currentVectors = $currentPoint['result']['vector']; 56 | } catch (InvalidArgumentException $e) { 57 | $currentPoint = null; 58 | } 59 | 60 | $vectors = new MultiVectorStruct(); 61 | 62 | foreach ($model->getVectorizers() as $name => $vectorizerClass) { 63 | $data = $searchableData[$name] ?? null; 64 | if (!$data) { 65 | continue; 66 | } 67 | 68 | // If the field hasn't changed, fetch the vector from $currentPoint 69 | if (!$model->hasVectorFieldChanged($name, $data) && isset($currentVectors[$name])) { 70 | $vectorizedData = $currentVectors[$name]; 71 | } 72 | // If the field has changed or doesn't exist in $currentPoint, use the vectorizer 73 | else { 74 | $vectorizer = $this->vectorizerEngineManager->driver($vectorizerClass); 75 | $vectorizedData = $vectorizer->embedDocument($data); 76 | $model->setVectorFieldHash($name, $data); 77 | } 78 | 79 | $vectors->addVector($name, $vectorizedData); 80 | unset($searchableData[$name]); // Remove the vectorized field from the searchable data 81 | } 82 | 83 | $points->addPoint( 84 | new PointStruct( 85 | (int) $model->getScoutKey(), 86 | // Vectors are stored as a MultiVectorStruct 87 | $vectors, 88 | // We're using the remaining searchable data as the payload 89 | $searchableData 90 | ) 91 | ); 92 | } 93 | 94 | // Perform the bulk upsert operation 95 | if ($points->count()) { 96 | $this->qdrant->collections($collectionName)->points()->upsert($points); 97 | } 98 | } 99 | 100 | /** 101 | * @throws InvalidArgumentException 102 | */ 103 | public function delete($models) 104 | { 105 | if ($models->isEmpty()) { 106 | return; 107 | } 108 | 109 | $collectionName = $models->first()->searchableAs(); 110 | $ids = $models->map->getScoutKey()->all(); 111 | 112 | $this->qdrant->collections($collectionName)->points()->delete($ids); 113 | } 114 | 115 | public function search(Builder $builder) 116 | { 117 | $options = array_merge($builder->options, [ 118 | 'limit' => $builder->limit ?? 100, 119 | ]); 120 | return $this->performSearch($builder, $options); 121 | } 122 | 123 | public function paginate(Builder $builder, $perPage, $page) 124 | { 125 | $options = array_merge($builder->options, [ 126 | 'limit' => $perPage, 127 | 'offset' => ($page - 1) * $perPage, 128 | ]); 129 | return $this->performSearch($builder, $options); 130 | } 131 | 132 | protected function performSearch(Builder $builder, array $options = []) 133 | { 134 | $collectionName = $builder->index ?: $builder->model->searchableAs(); 135 | $vectorField = $options['field'] ?? $builder->model->getDefaultVectorField(); 136 | $qdrantRequest = null; 137 | $filter = new Filter(); 138 | 139 | if($builder->query instanceof Model){ 140 | // Create a RecommendRequest 141 | $qdrantRequest = new RecommendRequest([$builder->query->getScoutKey()]); 142 | $qdrantRequest->setUsing($vectorField); 143 | } else { 144 | // Create a SearchRequest 145 | $vectorizer = $this->vectorizerEngineManager->driver($builder->model->getVectorizers()[$vectorField] ?? 'openai'); 146 | $embedding = $vectorizer->embedQuery($builder->query); 147 | $vector = new VectorStruct($embedding, $vectorField); 148 | 149 | $qdrantRequest = new SearchRequest($vector); 150 | } 151 | 152 | $qdrantRequest->setLimit($options['limit']); 153 | 154 | // Loop through the builder's wheres and add them to the filter 155 | if(!empty($builder->wheres)){ 156 | foreach ($builder->wheres as $field => $value) { 157 | if(is_numeric($value)) { 158 | $filter->addMust( 159 | new MatchInt($field, $value) 160 | ); 161 | } elseif(is_bool($value)){ 162 | $filter->addMust( 163 | new MatchBool($field, $value) 164 | ); 165 | } else { 166 | $filter->addMust( 167 | new MatchString($field, $value) 168 | ); 169 | } 170 | } 171 | 172 | // Attach the filter to the search request 173 | $qdrantRequest->setFilter($filter); 174 | } 175 | 176 | if (isset($options['offset'])) { 177 | $qdrantRequest->setOffset($options['offset']); 178 | } 179 | 180 | if(isset($options['score_threshold'])) { 181 | $qdrantRequest->setScoreThreshold($options['score_threshold']); 182 | } 183 | 184 | if ($builder->callback) { 185 | $options['qdrantRequest'] = $qdrantRequest; 186 | 187 | return call_user_func( 188 | $builder->callback, 189 | $this->qdrant, 190 | $builder->query, 191 | $options 192 | ); 193 | } 194 | 195 | // If the request is a RecommendRequest, call the recommend endpoint, else call the search endpoint 196 | if($qdrantRequest instanceof RecommendRequest){ 197 | $result = $this->qdrant->collections($collectionName)->points()->recommend($qdrantRequest); 198 | } else { 199 | $result = $this->qdrant->collections($collectionName)->points()->search($qdrantRequest); 200 | } 201 | 202 | return [ 203 | 'result' => $result['result'], 204 | 205 | // We need these to get the counts for pagination if necessary 206 | 'collection' => $collectionName, 207 | 'request' => $qdrantRequest, 208 | ]; 209 | } 210 | 211 | public function mapIds($results) 212 | { 213 | return collect($results['result'])->pluck('id')->values(); 214 | } 215 | 216 | public function map(Builder $builder, $results, $model) 217 | { 218 | if (count($results['result']) === 0) { 219 | return $model->newCollection(); 220 | } 221 | 222 | $objectIds = collect($results['result'])->pluck('id')->values()->all(); 223 | $objectIdPositions = array_flip($objectIds); 224 | $objectScores = collect($results['result'])->pluck('score', 'id')->all(); 225 | 226 | return $model->getScoutModelsByIds( 227 | $builder, $objectIds 228 | )->filter(function ($model) use ($objectIds) { 229 | return in_array($model->getScoutKey(), $objectIds); 230 | })->sortBy(function ($model) use ($objectIdPositions) { 231 | return $objectIdPositions[$model->getScoutKey()]; 232 | })->values()->each(function ($model) use ($objectScores) { 233 | $model->search_score = $objectScores[$model->getScoutKey()]; 234 | }); 235 | } 236 | 237 | public function lazyMap(Builder $builder, $results, $model) 238 | { 239 | if (count($results['result']) === 0) { 240 | return LazyCollection::make($model->newCollection()); 241 | } 242 | 243 | $objectIds = collect($results['result'])->pluck('id')->values()->all(); 244 | $objectIdPositions = array_flip($objectIds); 245 | $objectScores = collect($results['result'])->pluck('score', 'id')->all(); 246 | 247 | return $model->queryScoutModelsByIds( 248 | $builder, $objectIds 249 | )->cursor()->filter(function ($model) use ($objectIds) { 250 | return in_array($model->getScoutKey(), $objectIds); 251 | })->sortBy(function ($model) use ($objectIdPositions) { 252 | return $objectIdPositions[$model->getScoutKey()]; 253 | })->values()->each(function ($model) use ($objectScores) { 254 | $model->search_score = $objectScores[$model->getScoutKey()]; 255 | }); 256 | } 257 | 258 | /** 259 | * @param $results 260 | * @return int 261 | * @throws InvalidArgumentException 262 | */ 263 | public function getTotalCount($results): int 264 | { 265 | // Get the Qdrant request 266 | $request = $results['request']; 267 | 268 | // Otherwise, we need to get the total count from Qdrant. 269 | if($request->getScoreThreshold() === null){ 270 | // We can use core functionality to get the total count 271 | $countResponse = $this->qdrant 272 | ->collections($results['collection']) 273 | ->points() 274 | ->count($request->getFilter() ?? null); 275 | 276 | return $countResponse['result']['count']; 277 | } 278 | else { 279 | // Because Qdrant doesn't support counting with a score threshold, we need to get all the results and count them ourselves. 280 | // TODO - Remove this once Qdrant supports counting with a score threshold https://github.com/qdrant/qdrant/issues/2091 281 | 282 | $request = clone $request; 283 | 284 | // Set a very high limit and an offset of 0 to get all the results 285 | $request->setLimit(1_000); 286 | $request->setOffset(0); 287 | 288 | // If the request is a RecommendRequest, call the recommend endpoint, else call the search endpoint 289 | if($request instanceof RecommendRequest){ 290 | $result = $this->qdrant->collections($results['collection'])->points()->recommend($request); 291 | } else { 292 | $result = $this->qdrant->collections($results['collection'])->points()->search($request); 293 | } 294 | 295 | // Perform the count and return the number of results 296 | return collect($result['result'])->count(); 297 | } 298 | } 299 | 300 | /** 301 | * @throws InvalidArgumentException 302 | */ 303 | public function flush($model) 304 | { 305 | $collectionName = $model->searchableAs(); 306 | 307 | $this->deleteIndex($collectionName); 308 | $this->createIndex($collectionName); 309 | } 310 | 311 | /** 312 | * @throws InvalidArgumentException 313 | */ 314 | public function createIndex($name, array $options = []) 315 | { 316 | // Use getModelForTable 317 | $model = $this->getModelForSearchableName($name); 318 | $createCollection = new CreateCollection(); 319 | 320 | foreach ($model->getVectorizers() as $vectorField => $vectorizerClass) { 321 | // Using the engine manager to get the driver 322 | $vectorizer = $this->vectorizerEngineManager->driver($vectorizerClass); 323 | $createCollection->addVector($vectorizer->vectorParams(), $vectorField); 324 | } 325 | 326 | $indexName = $model->searchableAs(); 327 | 328 | $this->qdrant->collections($indexName)->create($createCollection); 329 | } 330 | 331 | /** 332 | * @throws InvalidArgumentException 333 | */ 334 | public function deleteIndex($name) 335 | { 336 | return $this->qdrant->collections($name)->delete(); 337 | } 338 | 339 | protected function usesSoftDelete($model): bool 340 | { 341 | return in_array(SoftDeletes::class, class_uses_recursive($model)); 342 | } 343 | 344 | /** 345 | * For a given table name, return an instance of that model. 346 | * 347 | * @param string $name The table name 348 | * @return Model|null 349 | * @note This function requires that you run `composer dumpautoload` after creating a new model. 350 | */ 351 | private function getModelForSearchableName(string $name): ?Model 352 | { 353 | static $models = []; 354 | if (isset($models[$name])) { 355 | return $models[$name]; 356 | } 357 | 358 | $composer = require base_path() . '/vendor/autoload.php'; 359 | 360 | // Define the root namespace and directory for your application 361 | $rootNamespace = 'App\\Models\\'; 362 | 363 | // Additional models defined by the user 364 | $modelClasses = config('scout-qdrant.models', []); 365 | 366 | $allClasses = collect(array_merge($modelClasses, array_keys($composer->getClassMap()))) 367 | ->filter(fn($class) => str_starts_with($class, $rootNamespace) || in_array($class, $modelClasses)) 368 | ->filter(fn($class) => is_subclass_of($class, 'Illuminate\Database\Eloquent\Model')); 369 | 370 | foreach ($allClasses as $class) { 371 | // Check if the class is within your application's namespace 372 | $model = new $class; 373 | if ( method_exists($model, 'searchableAs') && $model->searchableAs() === $name){ 374 | $models[$name] = $model; 375 | return $models[$name]; 376 | } 377 | } 378 | 379 | // No model was found 380 | return null; 381 | } 382 | } 383 | -------------------------------------------------------------------------------- /src/Vectorizer/HasOptions.php: -------------------------------------------------------------------------------- 1 | options = array_merge($this->options, $options); 12 | return $this; 13 | } 14 | 15 | public function getOptions(): array 16 | { 17 | return array_merge($this->options, $this->defaultOptions ?? []); 18 | } 19 | 20 | public function getOption(string $option): mixed 21 | { 22 | return $this->getOptions()[$option] ?? null; 23 | } 24 | } 25 | -------------------------------------------------------------------------------- /src/Vectorizer/Manager/VectorizerEngineManager.php: -------------------------------------------------------------------------------- 1 | container->make(OpenAIVectorizer::class); 15 | } 16 | 17 | public function driver($driver = null) 18 | { 19 | // If there is a '::' in the name, then we consider everything after that JSON encoded options 20 | $options = []; 21 | if (str_contains($driver, '::')) { 22 | [$driver, $options] = explode('::', $driver, 2); 23 | $options = json_decode($options, true); 24 | } 25 | 26 | $driverInstance = parent::driver($driver); 27 | 28 | if (method_exists($driverInstance, 'setOptions')) { 29 | $driverInstance->setOptions($options); 30 | } 31 | 32 | return $driverInstance; 33 | } 34 | 35 | 36 | public function extend($driver, Closure $callback) 37 | { 38 | $this->customCreators[$driver] = $callback; 39 | 40 | return $this; 41 | } 42 | 43 | 44 | public function getDefaultDriver() 45 | { 46 | return config('scout-qdrant.vectorizer'); 47 | } 48 | 49 | public function create($driver) 50 | { 51 | if (isset($this->customCreators[$driver])) { 52 | return $this->callCustomCreator($driver); 53 | } 54 | 55 | $method = 'create' . ucfirst($driver) . 'Driver'; 56 | 57 | if (method_exists($this, $method)) { 58 | return $this->$method(); 59 | } 60 | 61 | throw new InvalidArgumentException("Vectorizer driver not supported: {$driver}"); 62 | } 63 | } 64 | -------------------------------------------------------------------------------- /src/Vectorizer/OpenAIVectorizer.php: -------------------------------------------------------------------------------- 1 | 'text-embedding-ada-002', 15 | ]; 16 | 17 | static array $vectorDimensions = [ 18 | 'text-embedding-ada-002' => 1536, 19 | ]; 20 | 21 | public function vectorParams(): VectorParams 22 | { 23 | return new VectorParams( 24 | static::$vectorDimensions[$this->getOption('model')], 25 | VectorParams::DISTANCE_COSINE 26 | ); 27 | } 28 | 29 | public function embedDocument(string $document): array 30 | { 31 | $response = OpenAI::embeddings()->create([ 32 | 'model' => $this->getOption('model'), 33 | 'input' => $document, 34 | ]); 35 | 36 | return $response->embeddings[0]->embedding; 37 | } 38 | 39 | public function embedQuery(string $query): array 40 | { 41 | return $this->embedDocument($query); 42 | } 43 | } 44 | -------------------------------------------------------------------------------- /src/Vectorizer/VectorizerInterface.php: -------------------------------------------------------------------------------- 1 |