├── .laminas-ci.json ├── LICENSE ├── README.md ├── bin └── roave-infection-static-analysis-plugin ├── composer.json ├── renovate.json └── src └── Roave └── InfectionStaticAnalysis ├── Bootstrapper.php ├── CliUtility.php ├── Psalm └── RunStaticAnalysisAgainstMutant.php ├── RunStaticAnalysisAgainstEscapedMutant.php └── Stub └── ArrayFilter.php /.laminas-ci.json: -------------------------------------------------------------------------------- 1 | { 2 | "exclude": [ 3 | { 4 | "name": "Infection [8.2, locked]" 5 | } 6 | ] 7 | } -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2020 Roave, LLC. 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of 4 | this software and associated documentation files (the "Software"), to deal in 5 | the Software without restriction, including without limitation the rights to 6 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies 7 | of the Software, and to permit persons to whom the Software is furnished to do 8 | so, subject to the following conditions: 9 | 10 | The above copyright notice and this permission notice shall be included in all 11 | copies or substantial portions of the Software. 12 | 13 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 14 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 15 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 16 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 17 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 18 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 19 | SOFTWARE. 20 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Infection Static Analysis Plugin 2 | 3 | This plugin is designed to run static analysis on top of [`infection/infection`](https://github.com/infection/infection) 4 | test runs in order to discover if [escaped mutants](https://en.wikipedia.org/wiki/Mutation_testing) 5 | are valid mutations, or if they do not respect the type signature of your 6 | program. If the mutation would result in a type error, it is "killed". 7 | 8 | TL;DR: 9 | 10 | - This will improve your mutation score, since mutations which result in 11 | type errors become killed. 12 | - This is very hacky, and replaces `vendor/bin/infection` essentially. 13 | Please read the `Stability` section below first for details. 14 | - This is currently much slower than running infection by itself. 15 | There are ideas/suggestions to improve this in the future. 16 | 17 | ## Usage 18 | 19 | The current design of this tool requires you to run `vendor/bin/roave-infection-static-analysis-plugin` 20 | instead of running `vendor/bin/infection`: 21 | 22 | ```sh 23 | composer require --dev roave/infection-static-analysis-plugin 24 | 25 | vendor/bin/roave-infection-static-analysis-plugin 26 | ``` 27 | 28 | ### Configuration 29 | 30 | The `roave-infection-static-analysis-plugin` binary accepts all of `infection` flags and arguments, and an additional `--psalm-config` argument. 31 | 32 | Using `--psalm-config`, you can specify the psalm configuration file to use when analysing the generated mutations: 33 | 34 | ```sh 35 | vendor/bin/roave-infection-static-analysis-plugin --psalm-config config/psalm.xml 36 | ``` 37 | 38 | ## Background 39 | 40 | If you come from a statically typed language with AoT compilers, you may be 41 | confused about the scope of this project, but in the PHP ecosystem, producing 42 | runnable code that does not respect the type system is very easy, and mutation 43 | testing tools do this all the time. 44 | 45 | Take for example following snippet: 46 | 47 | ```php 48 | /** 49 | * @template T 50 | * @param array $values 51 | * @return list 52 | */ 53 | function makeAList(array $values): array 54 | { 55 | return array_values($values); 56 | } 57 | ``` 58 | 59 | Given a valid test as follows: 60 | 61 | ```php 62 | function test_makes_a_list(): void 63 | { 64 | $list = makeAList(['a' => 'b', 'c' => 'd']); 65 | 66 | assert(count($list) === 2); 67 | assert(in_array('b', $list, true)); 68 | assert(in_array('d', $list, true)); 69 | } 70 | ``` 71 | 72 | The mutation testing framework will produce following mutation, since we 73 | failed to verify the output in a more precise way: 74 | 75 | ```diff 76 | /** 77 | * @template T 78 | * @param array $values 79 | * @return list 80 | */ 81 | function makeAList(array $values): array 82 | { 83 | - return array_values($values); 84 | + return $values; 85 | } 86 | ``` 87 | 88 | The code above is valid PHP, but not valid according to our type declarations. 89 | While we can indeed write a test for this, such test would probably be 90 | unnecessary, as existing type checkers can detect that our actual return value is 91 | no longer a `list`, but a map of `array`, which is in conflict 92 | with what we declared. 93 | 94 | This plugin detects such mutations, and prevents them from making you write 95 | unnecessary tests, leveraging the full power of existing PHP type checkers 96 | such as [phpstan](https://github.com/phpstan/phpstan) and [psalm](https://github.com/vimeo/psalm). 97 | 98 | ## Stability 99 | 100 | Since [`infection/infection`](https://github.com/infection/infection) is not yet 101 | designed to support plugins, this tool uses a very aggressive approach to bootstrap 102 | itself, and relies on internal details of the underlying runner. 103 | 104 | To prevent compatibility issues, it therefore always pins to a very specific version 105 | of `infection/infection`, so please be patient when you wish to use the latest and 106 | greatest version of `infection/infection`, as we may still be catching up to it. 107 | 108 | Eventually, we will contribute patches to `infection/infection` so that there is a 109 | proper way to design and use plugins, without the need for dirty hacks. 110 | 111 | ## PHPStan? Psalm? Where's my favourite static analysis tool? 112 | 113 | Our initial scope of work for `1.0.x` is to provide `vimeo/psalm` support as a start, 114 | while other static analysers will be included at a later point in time. 115 | -------------------------------------------------------------------------------- /bin/roave-infection-static-analysis-plugin: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env php 2 | __DIR__ . '/../../../..', 34 | 'installed-as-project' => __DIR__ . '/..', 35 | 'current-working-directory' => getcwd(), 36 | ]; 37 | 38 | foreach ($projectDirectoryCandidates as $candidatePath) { 39 | if (! file_exists($candidatePath . '/vendor/autoload.php')) { 40 | continue; 41 | } 42 | 43 | return $candidatePath; 44 | } 45 | 46 | throw new UnexpectedValueException(sprintf( 47 | 'Could not identify project installation path within following paths: %s', 48 | var_export($projectDirectoryCandidates, true) 49 | )); 50 | })(); 51 | 52 | require_once $projectPath . '/vendor/autoload.php'; 53 | 54 | if (! defined('PSALM_VERSION')) { 55 | define('PSALM_VERSION', InstalledVersions::getVersion('vimeo/psalm')); 56 | } 57 | 58 | if (! defined('PHP_PARSER_VERSION')) { 59 | define('PHP_PARSER_VERSION', InstalledVersions::getVersion('nikic/php-parser')); 60 | } 61 | 62 | /** @var list */ 63 | $arguments = $_SERVER['argv'] ?? []; 64 | [$arguments, $configuration] = CliUtility::extractArgument($arguments, 'psalm-config'); 65 | 66 | RuntimeCaches::clearAll(); 67 | 68 | $configuration = $configuration ?? $projectPath; 69 | 70 | if (is_file($configuration)) { 71 | $config = Config::loadFromXMLFile($configuration, $projectPath); 72 | } else { 73 | $config = Config::getConfigForPath($configuration, $projectPath); 74 | } 75 | 76 | $config->setIncludeCollector(new IncludeCollector()); 77 | 78 | (new Application(Bootstrapper::bootstrap( 79 | Container::create(), 80 | new RunStaticAnalysisAgainstMutant(new ProjectAnalyzer( 81 | $config, 82 | new Providers(new FileProvider()), 83 | new ReportOptions() 84 | )) 85 | ))) 86 | ->run(new ArgvInput($arguments)); 87 | })(); 88 | -------------------------------------------------------------------------------- /composer.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "roave/infection-static-analysis-plugin", 3 | "description": "Static analysis on top of mutation testing - prevents escaped mutants from being invalid according to static analysis", 4 | "type": "library", 5 | "license": "MIT", 6 | "authors": [ 7 | { 8 | "name": "Marco Pivetta", 9 | "email": "ocramius@gmail.com" 10 | } 11 | ], 12 | "bin": [ 13 | "bin/roave-infection-static-analysis-plugin" 14 | ], 15 | "require": { 16 | "php": "~8.2.0 || ~8.3.0 || ~8.4.0", 17 | "composer-runtime-api": "^2.2", 18 | "infection/infection": "0.29.14", 19 | "sanmai/later": "^0.1.7", 20 | "vimeo/psalm": "^6.12.0" 21 | }, 22 | "conflict": { 23 | "symfony/polyfill-php84": "<1.30.0" 24 | }, 25 | "require-dev": { 26 | "azjezz/psl": "^3.3.0", 27 | "doctrine/coding-standard": "^13.0.1", 28 | "phpunit/phpunit": "^11.5.22", 29 | "psalm/plugin-phpunit": "^0.19.5" 30 | }, 31 | "autoload": { 32 | "psr-4": { 33 | "Roave\\InfectionStaticAnalysis\\": "src/Roave/InfectionStaticAnalysis" 34 | } 35 | }, 36 | "autoload-dev": { 37 | "psr-4": { 38 | "Roave\\InfectionStaticAnalysisTest\\": "test/unit/Roave/InfectionStaticAnalysisTest" 39 | } 40 | }, 41 | "config": { 42 | "allow-plugins": { 43 | "dealerdirect/phpcodesniffer-composer-installer": true, 44 | "infection/extension-installer": false 45 | }, 46 | "platform": { 47 | "php": "8.2.99" 48 | }, 49 | "sort-packages": true 50 | } 51 | } 52 | -------------------------------------------------------------------------------- /renovate.json: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "https://docs.renovatebot.com/renovate-schema.json", 3 | "extends": [ 4 | "local>Ocramius/.github:renovate-config" 5 | ] 6 | } 7 | -------------------------------------------------------------------------------- /src/Roave/InfectionStaticAnalysis/Bootstrapper.php: -------------------------------------------------------------------------------- 1 | getTestFrameworkAdapter()), 24 | $runStaticAnalysis, 25 | ); 26 | }; 27 | 28 | $reflectionOffsetSet->invokeArgs($container, [MutantExecutionResultFactory::class, $factory]); 29 | 30 | return $container; 31 | } 32 | } 33 | -------------------------------------------------------------------------------- /src/Roave/InfectionStaticAnalysis/CliUtility.php: -------------------------------------------------------------------------------- 1 | $arguments 19 | * @param non-empty-string $argument 20 | * 21 | * @return array{0: list, 1: (non-empty-string|null)} 22 | */ 23 | public static function extractArgument(array $arguments, string $argument): array 24 | { 25 | $lookup = '--' . $argument; 26 | 27 | $result = null; 28 | $present = false; 29 | foreach ($arguments as $index => $arg) { 30 | if ($arg === $lookup) { 31 | $present = true; 32 | unset($arguments[$index]); 33 | // grab the next argument in the list 34 | $value = $arguments[$index + 1] ?? null; 35 | // if the argument is not a flag/argument name ( starts with - ) 36 | if ($value !== null && ! str_starts_with($value, '-')) { 37 | // consider it the value, and remove it from the list. 38 | $result = $value; 39 | unset($arguments[$index + 1]); 40 | } 41 | 42 | break; 43 | } 44 | 45 | // if the argument starts with `--argument-name=` 46 | // we consider anything after '=' to be the value 47 | if (str_starts_with($arg, $lookup . '=')) { 48 | $present = true; 49 | unset($arguments[$index]); 50 | 51 | $result = substr($arg, 15); 52 | break; 53 | } 54 | } 55 | 56 | $arguments = array_values($arguments); 57 | $value = self::removeSurroundingQuites($result); 58 | if ($present && $value === null) { 59 | throw new RuntimeException(sprintf('Please provide a value for "%s" argument.', $argument)); 60 | } 61 | 62 | return [$arguments, $value]; 63 | } 64 | 65 | /** @return non-empty-string|null */ 66 | private static function removeSurroundingQuites(string|null $argument): string|null 67 | { 68 | if ($argument === null || $argument === '') { 69 | return null; 70 | } 71 | 72 | if ($argument[0] === '"') { 73 | $argument = substr($argument, 1); 74 | } 75 | 76 | if (substr($argument, -1) === '"') { 77 | $argument = substr($argument, 0, -1); 78 | } 79 | 80 | return $argument === '' ? null : $argument; 81 | } 82 | } 83 | -------------------------------------------------------------------------------- /src/Roave/InfectionStaticAnalysis/Psalm/RunStaticAnalysisAgainstMutant.php: -------------------------------------------------------------------------------- 1 | getFilePath(); 29 | $paths = [$mutant->getFilePath()]; 30 | $codebase = $this->projectAnalyzer->getCodebase(); 31 | 32 | $codebase->invalidateInformationForFile( 33 | $mutant->getMutation() 34 | ->getOriginalFilePath(), 35 | ); 36 | 37 | if (! $this->alreadyVisitedStubs) { 38 | $codebase->config->visitPreloadedStubFiles($codebase); 39 | $codebase->config->visitStubFiles($codebase); 40 | $codebase->config->visitComposerAutoloadFiles($this->projectAnalyzer); 41 | 42 | $this->alreadyVisitedStubs = true; 43 | } 44 | 45 | $codebase->reloadFiles($this->projectAnalyzer, $paths); 46 | $codebase->analyzer->analyzeFiles($this->projectAnalyzer, count($paths), false); 47 | 48 | $mutationValid = ! array_key_exists( 49 | $path, 50 | $codebase->file_reference_provider->getExistingIssues(), 51 | ); 52 | 53 | $codebase->invalidateInformationForFile($path); 54 | 55 | return $mutationValid; 56 | } 57 | } 58 | -------------------------------------------------------------------------------- /src/Roave/InfectionStaticAnalysis/RunStaticAnalysisAgainstEscapedMutant.php: -------------------------------------------------------------------------------- 1 | reflectionOriginalStartFileLocation = new ReflectionProperty(MutantExecutionResult::class, 'originalStartFilePosition'); 34 | $this->reflectionOriginalEndFilePosition = new ReflectionProperty(MutantExecutionResult::class, 'originalEndFilePosition'); 35 | } 36 | 37 | public function createFromProcess(MutantProcess $mutantProcess): MutantExecutionResult 38 | { 39 | $result = $this->next->createFromProcess($mutantProcess); 40 | 41 | if ($result->getDetectionStatus() !== DetectionStatus::ESCAPED) { 42 | return $result; 43 | } 44 | 45 | if ($this->runStaticAnalysis->isMutantStillValidAccordingToStaticAnalysis($mutantProcess->getMutant())) { 46 | return $result; 47 | } 48 | 49 | $originalStartFilePosition = $this->reflectionOriginalStartFileLocation->getValue($result); 50 | $originalEndFilePosition = $this->reflectionOriginalEndFilePosition->getValue($result); 51 | 52 | assert(is_int($originalStartFilePosition)); 53 | assert(is_int($originalEndFilePosition)); 54 | 55 | return new MutantExecutionResult( 56 | $result->getProcessCommandLine(), 57 | $result->getProcessOutput(), 58 | DetectionStatus::KILLED, // Mutant was squished by static analysis 59 | later(static fn () => yield $result->getMutantDiff()), 60 | $result->getMutantHash(), 61 | $result->getMutatorClass(), 62 | $result->getMutatorName(), 63 | $result->getOriginalFilePath(), 64 | $result->getOriginalStartingLine(), 65 | $result->getOriginalEndingLine(), 66 | $originalStartFilePosition, 67 | $originalEndFilePosition, 68 | later(static fn () => yield $result->getOriginalCode()), 69 | later(static fn () => yield $result->getMutatedCode()), 70 | $result->getTests(), 71 | ); 72 | } 73 | } 74 | -------------------------------------------------------------------------------- /src/Roave/InfectionStaticAnalysis/Stub/ArrayFilter.php: -------------------------------------------------------------------------------- 1 | $values 14 | * 15 | * @psalm-return list 16 | * 17 | * @psalm-template T 18 | */ 19 | public function makeAList(array $values): array 20 | { 21 | return array_values($values); 22 | } 23 | } 24 | --------------------------------------------------------------------------------