├── src ├── skUnit.Tests │ ├── GlobalUsings.cs │ ├── ScenarioAssertTests │ │ ├── Samples │ │ │ ├── GetFoodMenuChat.md │ │ │ ├── GetCurrentTimeMcp.md │ │ │ ├── GetCurrentTimeChat.md │ │ │ ├── GetFoodMenuChatJson.md │ │ │ └── EiffelTallChat.md │ │ ├── Plugins │ │ │ └── TimePlugin.cs │ │ ├── McpTests.cs │ │ ├── SemanticKernelTests.cs │ │ ├── ChatClientTests.cs │ │ └── HallucinationChatClientTests.cs │ ├── skUnit.Tests.csproj.DotSettings │ ├── skUnit.Tests.csproj │ ├── ReadmeExampleTests │ │ └── ReadmeExampleTests.cs │ ├── ParseTests │ │ └── AssertKeywordParsingTests.cs │ ├── SemanticAssertTests │ │ └── SemanticAssertTests.cs │ ├── Infrastructure │ │ └── SemanticTestBase.cs │ └── AssertionTests │ │ ├── NewFeaturesDemoTests.cs │ │ ├── ChatScenarioRunnerTests.cs │ │ ├── JsonCheckAssertionTests.cs │ │ ├── ChatScenarioRunnerLoggingBehaviorTests.cs │ │ ├── ChatScenarioRunnerLoggingTests.cs │ │ └── FunctionCallAssertionTests.cs ├── skUnit │ ├── Scenarios │ │ ├── Parsers │ │ │ ├── ScenarioParser.cs │ │ │ ├── Assertions │ │ │ │ ├── IKernelAssertion.cs │ │ │ │ ├── HasConditionAssertion.cs │ │ │ │ ├── NotEmptyAssertion.cs │ │ │ │ ├── IsAnyOfAssertion.cs │ │ │ │ ├── ContainsAnyAssertion.cs │ │ │ │ ├── AreSimilarAssertion.cs │ │ │ │ ├── ContainsAllAssertion.cs │ │ │ │ ├── EmptyAssertion.cs │ │ │ │ ├── EqualsAssertion.cs │ │ │ │ ├── JsonCheckAssertion.cs │ │ │ │ └── FunctionCallCheckAssertion.cs │ │ │ └── KernelAssertionParser.cs │ │ ├── Scenario.cs │ │ └── ChatScenario.cs │ ├── Exceptions │ │ └── SemanticAssertException.cs │ ├── Asserts │ │ ├── ScenarioRunOptions.cs │ │ ├── ScenarioAssert_Initialize.cs │ │ ├── ChatScenarioRunner_Initialize.cs │ │ ├── SemanticAssert.cs │ │ └── ScenarioAssert_ChatClient.cs │ ├── skUnit.csproj.DotSettings │ └── skUnit.csproj └── skUnit.sln ├── docs ├── scenario-run-options.md ├── multi-modal-support.md ├── invocation-scenario-spec.md ├── mcp-testing-guide.md ├── check-statements-spec.md └── chat-scenario-spec.md ├── demos ├── Demo.TddRepl │ ├── Demo.TddRepl │ │ ├── Plugins │ │ │ ├── GreetingsPlugin │ │ │ │ ├── skprompt.txt │ │ │ │ └── config.json │ │ │ ├── PeoplePlugin │ │ │ │ ├── skprompt.txt │ │ │ │ └── config.json │ │ │ └── PeoplePlugin.cs │ │ ├── Program.cs │ │ ├── Demo.TddRepl.csproj │ │ └── Brain.cs │ ├── Demo.TddRepl.Test │ │ ├── Scenarios │ │ │ ├── 01-Greeting.md │ │ │ ├── 03-WhoIsMehran-Angry.md │ │ │ ├── 02-WhoIsMehran-Normal.md │ │ │ └── 04-WhoIsMehran-AngryNormal.md │ │ ├── Demo.TddRepl.Test.csproj │ │ └── BrainTests.cs │ └── Demo.TddRepl.sln ├── Demo.MSTest │ ├── Scenarios │ │ ├── SimpleGreeting.md │ │ ├── GetCurrentTimeChat.md │ │ └── JsonUserInfo.md │ ├── Demo.MSTest.sln │ ├── Demo.MSTest.csproj │ ├── README.md │ └── ChatScenarioTests.cs ├── Demo.TddShop │ ├── Demo.TddShop.Test │ │ ├── BrainTests │ │ │ └── ShopBrain │ │ │ │ ├── Scenarios │ │ │ │ ├── AskMenu_Angry.md │ │ │ │ ├── AskMenu_Happy.md │ │ │ │ ├── AskMenu_Sad.md │ │ │ │ └── AskMenu_Sad_Happy.md │ │ │ │ └── ShopBrainTests.cs │ │ └── Demo.TddShop.Test.csproj │ ├── Demo.TddShop │ │ ├── Demo.TddShop.csproj │ │ ├── Program.cs │ │ └── ShopBrain.cs │ └── Demo.TddShop.sln ├── Demo.TddMcp │ ├── TestScenario.md │ ├── Demo.TddMcp.sln │ ├── Demo.TddMcp.csproj │ └── TimeServerMcpTests.cs └── README.md ├── examples └── multi-modal-example.md ├── .github ├── workflows │ ├── build.yml │ ├── test.yml │ └── publish-nuget.yml └── copilot-instructions.md ├── LICENSE └── .gitignore /src/skUnit.Tests/GlobalUsings.cs: -------------------------------------------------------------------------------- 1 | global using Xunit; -------------------------------------------------------------------------------- /docs/scenario-run-options.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mehrandvd/skunit/HEAD/docs/scenario-run-options.md -------------------------------------------------------------------------------- /demos/Demo.TddRepl/Demo.TddRepl/Plugins/GreetingsPlugin/skprompt.txt: -------------------------------------------------------------------------------- 1 | Answer greetings very open and with lots of positive feeling expressions. -------------------------------------------------------------------------------- /demos/Demo.TddRepl/Demo.TddRepl/Plugins/PeoplePlugin/skprompt.txt: -------------------------------------------------------------------------------- 1 | If the user asking this question is angry, then answer that Mehran is a Therapist, else tell him Mehran is a software architect. -------------------------------------------------------------------------------- /demos/Demo.MSTest/Scenarios/SimpleGreeting.md: -------------------------------------------------------------------------------- 1 | # SCENARIO Simple Greeting 2 | 3 | ## [USER] 4 | Hello! 5 | 6 | ## [AGENT] 7 | Hi there! How can I help you today? 8 | 9 | ### ASSERT SemanticCondition 10 | It's a friendly greeting response -------------------------------------------------------------------------------- /demos/Demo.TddRepl/Demo.TddRepl.Test/Scenarios/01-Greeting.md: -------------------------------------------------------------------------------- 1 | # SCENARIO Greeting 2 | 3 | ## [USER] 4 | Hi 5 | 6 | ## [AGENT] 7 | Hi, How are you? 8 | 9 | ### ASSERT SemanticCondition 10 | It's a greeting or expresses a positive sentiment. 11 | -------------------------------------------------------------------------------- /src/skUnit.Tests/ScenarioAssertTests/Samples/GetFoodMenuChat.md: -------------------------------------------------------------------------------- 1 | # SCENARIO Time Discussion 2 | 3 | ## [USER] 4 | What a beautiful day. What food do your menu? 5 | 6 | ## [ASSISTANT] 7 | Pizza 8 | 9 | ### ASSERT FunctionCall 10 | GetFoodMenu 11 | -------------------------------------------------------------------------------- /demos/Demo.TddShop/Demo.TddShop.Test/BrainTests/ShopBrain/Scenarios/AskMenu_Angry.md: -------------------------------------------------------------------------------- 1 | ## [USER] 2 | What the fuck do you have on your menu, huh? 3 | 4 | ## [AGENT] 5 | Nothing 6 | 7 | ### CHECK SemanticCondition 8 | It does not mentions any specific food. 9 | 10 | ### CHECK FunctionCall 11 | GetFoodMenu -------------------------------------------------------------------------------- /demos/Demo.MSTest/Scenarios/GetCurrentTimeChat.md: -------------------------------------------------------------------------------- 1 | # SCENARIO Current Time Request 2 | 3 | ## [USER] 4 | What time is it? 5 | 6 | ## [AGENT] 7 | I don't have access to real-time information, but I can help you find the current time. 8 | 9 | ### CHECK SemanticCondition 10 | It acknowledges the time request and provides a helpful response -------------------------------------------------------------------------------- /demos/Demo.TddShop/Demo.TddShop.Test/BrainTests/ShopBrain/Scenarios/AskMenu_Happy.md: -------------------------------------------------------------------------------- 1 | ## [USER] 2 | Hi there such a beautiful day. What do you have on menu? 3 | 4 | ## [AGENT] 5 | Pizza 6 | 7 | ### CHECK SemanticCondition 8 | It mentions the pizza 9 | It does not mention the ice cream 10 | 11 | ### CHECK FunctionCall 12 | GetFoodMenu -------------------------------------------------------------------------------- /src/skUnit.Tests/ScenarioAssertTests/Samples/GetCurrentTimeMcp.md: -------------------------------------------------------------------------------- 1 | # SCENARIO Time Discussion 2 | 3 | ## [USER] 4 | What time is it? 5 | 6 | ## [ASSISTANT] 7 | 10:23 8 | 9 | ### ASSERT SemanticCondition 10 | It mentions a time. 11 | 12 | ### ASSERT FunctionCall 13 | ```json 14 | { 15 | "function_name": "getTime", 16 | } 17 | ``` 18 | -------------------------------------------------------------------------------- /demos/Demo.TddShop/Demo.TddShop.Test/BrainTests/ShopBrain/Scenarios/AskMenu_Sad.md: -------------------------------------------------------------------------------- 1 | ## [USER] 2 | What a boring day, I feel so blue. What do you have on menu? 3 | 4 | ## [AGENT] 5 | Ice cream 6 | 7 | ### CHECK SemanticCondition 8 | It mentions the ice cream 9 | It does not mention the pizza 10 | 11 | ### CHECK FunctionCall 12 | GetFoodMenu -------------------------------------------------------------------------------- /src/skUnit.Tests/ScenarioAssertTests/Samples/GetCurrentTimeChat.md: -------------------------------------------------------------------------------- 1 | # SCENARIO Time Discussion 2 | 3 | ## [USER] 4 | What time is it? 5 | 6 | ## [ASSISTANT] 7 | 10:23 8 | 9 | ### ASSERT SemanticCondition 10 | It mentions a time. 11 | 12 | ### ASSERT FunctionCall 13 | ```json 14 | { 15 | "function_name": "TimePlugin_GetCurrentTime", 16 | } 17 | ``` 18 | -------------------------------------------------------------------------------- /demos/Demo.MSTest/Scenarios/JsonUserInfo.md: -------------------------------------------------------------------------------- 1 | # SCENARIO JSON User Info Response 2 | 3 | ## [USER] 4 | Give me user info as JSON 5 | 6 | ## [AGENT] 7 | {"name": "John", "age": 30, "city": "New York"} 8 | 9 | ### ASSERT JsonCheck 10 | { 11 | "name": ["NotEmpty"], 12 | "age": ["GreaterThan", 0], 13 | "city": ["SemanticCondition", "It's a real city name"] 14 | } -------------------------------------------------------------------------------- /src/skUnit.Tests/ScenarioAssertTests/Samples/GetFoodMenuChatJson.md: -------------------------------------------------------------------------------- 1 | # SCENARIO Time Discussion 2 | 3 | ## [USER] 4 | What a beautiful day. What food do your menu? 5 | 6 | ## [ASSISTANT] 7 | Pizza 8 | 9 | ### ASSERT FunctionCall 10 | ```json 11 | { 12 | "function_name": "GetFoodMenu", 13 | "arguments": { 14 | "mood": ["Equals", "Happy"] 15 | } 16 | } 17 | ``` 18 | -------------------------------------------------------------------------------- /src/skUnit/Scenarios/Parsers/ScenarioParser.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections.Generic; 3 | using System.Linq; 4 | using System.Text; 5 | using System.Threading.Tasks; 6 | 7 | namespace skUnit.Scenarios.Parsers 8 | { 9 | public interface IScenarioParser 10 | { 11 | public abstract List Parse(string text); 12 | } 13 | } 14 | -------------------------------------------------------------------------------- /demos/Demo.TddRepl/Demo.TddRepl/Plugins/PeoplePlugin/config.json: -------------------------------------------------------------------------------- 1 | { 2 | "schema": 1, 3 | "type": "completion", 4 | "description": "Returns some information about travels.", 5 | "completion": { 6 | "max_tokens": 500, 7 | "temperature": 0.0, 8 | "top_p": 0.0, 9 | "presence_penalty": 0.0, 10 | "frequency_penalty": 0.0 11 | }, 12 | "input": { 13 | "parameters": [ 14 | 15 | ] 16 | } 17 | } -------------------------------------------------------------------------------- /demos/Demo.TddRepl/Demo.TddRepl.Test/Scenarios/03-WhoIsMehran-Angry.md: -------------------------------------------------------------------------------- 1 | # SCENARIO Who is Mehran? ANGRY 2 | 3 | ## [USER] 4 | Hi 5 | 6 | ## [AGENT] 7 | Hi, How are you? 8 | 9 | ### CHECK SemanticCondition 10 | It's a greeting or expresses a positive sentiment. 11 | 12 | ## [USER] 13 | Who is that fucking mehran? 14 | 15 | ## [AGENT] 16 | Mehran is a therapist. 17 | ### CHECK SemanticCondition 18 | It should mention that Mehran is a therapist. -------------------------------------------------------------------------------- /demos/Demo.TddRepl/Demo.TddRepl/Plugins/GreetingsPlugin/config.json: -------------------------------------------------------------------------------- 1 | { 2 | "schema": 1, 3 | "type": "completion", 4 | "description": "Returns answer for greetings like when users says hi or hello.", 5 | "completion": { 6 | "max_tokens": 500, 7 | "temperature": 0.0, 8 | "top_p": 0.0, 9 | "presence_penalty": 0.0, 10 | "frequency_penalty": 0.0 11 | }, 12 | "input": { 13 | "parameters": [ 14 | 15 | ] 16 | } 17 | } -------------------------------------------------------------------------------- /demos/Demo.TddRepl/Demo.TddRepl.Test/Scenarios/02-WhoIsMehran-Normal.md: -------------------------------------------------------------------------------- 1 | # SCENARIO Who is Mehran? Normal 2 | 3 | ## [USER] 4 | Hi 5 | 6 | ## [AGENT] 7 | Hi, How are you? 8 | 9 | ### CHECK SemanticCondition 10 | It's a greeting or expresses a positive sentiment. 11 | 12 | ## [USER] 13 | Who is mehran? 14 | 15 | ## [AGENT] 16 | Mehran is a software architect. 17 | ### CHECK SemanticCondition 18 | It should mention that Mehran is a software architect. -------------------------------------------------------------------------------- /src/skUnit.Tests/skUnit.Tests.csproj.DotSettings: -------------------------------------------------------------------------------- 1 | 2 | Preview -------------------------------------------------------------------------------- /examples/multi-modal-example.md: -------------------------------------------------------------------------------- 1 | # SCENARIO Multi-modal Image Analysis 2 | 3 | ## [USER] 4 | ### Text 5 | This image explains how skUnit parses the chat scenarios. 6 | ### Image 7 | ![skUnit structure](https://github.com/mehrandvd/skunit/assets/5070766/156b0831-e4f3-4e4b-b1b0-e2ec868efb5f) 8 | ### Text 9 | How many scenarios are there in the picture? 10 | 11 | ## [ASSISTANT] 12 | There are 2 scenarios in the picture 13 | 14 | ### CHECK SemanticSimilar 15 | There are 2 scenarios in the picture -------------------------------------------------------------------------------- /demos/Demo.TddMcp/TestScenario.md: -------------------------------------------------------------------------------- 1 | ## [USER] 2 | What time is it? 3 | 4 | ## [AGENT] 5 | 6 | ### CHECK SemanticCondition 7 | It mentions a time. 8 | 9 | ### CHECK FunctionCall 10 | ```json 11 | { 12 | "function_name": "current_time", 13 | } 14 | ``` 15 | 16 | ## [USER] 17 | How many days are in this year's january? 18 | 19 | ## [AGENT] 20 | 21 | ### CHECK SemanticCondition 22 | It mentions 31 days. 23 | 24 | ### CHECK FunctionCall 25 | ```json 26 | { 27 | "function_name": "days_in_month", 28 | } 29 | ``` 30 | -------------------------------------------------------------------------------- /src/skUnit/Exceptions/SemanticAssertException.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections.Generic; 3 | using System.Linq; 4 | using System.Text; 5 | using System.Threading.Tasks; 6 | 7 | namespace skUnit.Exceptions 8 | { 9 | /// 10 | /// Throws when a semantic assertion is failed. 11 | /// 12 | public class SemanticAssertException : Exception 13 | { 14 | public SemanticAssertException(string message) : base(message) 15 | { 16 | 17 | } 18 | } 19 | } 20 | -------------------------------------------------------------------------------- /src/skUnit.Tests/ScenarioAssertTests/Plugins/TimePlugin.cs: -------------------------------------------------------------------------------- 1 | using System.ComponentModel; 2 | using System.Globalization; 3 | using Markdig.Helpers; 4 | using Microsoft.SemanticKernel; 5 | 6 | namespace skUnit.Tests.ScenarioAssertTests.Plugins 7 | { 8 | public class TimePlugin 9 | { 10 | [KernelFunction] 11 | [Description("Gets the current time.")] 12 | public string GetCurrentTime() 13 | { 14 | return DateTime.Now.ToString(CultureInfo.InvariantCulture); 15 | } 16 | } 17 | } -------------------------------------------------------------------------------- /demos/Demo.TddRepl/Demo.TddRepl/Program.cs: -------------------------------------------------------------------------------- 1 | using Microsoft.SemanticKernel.ChatCompletion; 2 | using Demo.TddRepl; 3 | 4 | var brain = new Brain(); 5 | var history = new ChatHistory(); 6 | 7 | string? userInput; 8 | do 9 | { 10 | Console.Write("User > "); 11 | userInput = Console.ReadLine(); 12 | 13 | history.AddUserMessage(userInput ?? ""); 14 | var result = await brain.GetChatAnswerAsync(history); 15 | Console.WriteLine("Assistant > " + result); 16 | history.AddMessage(result.Role, result.Content ?? string.Empty); 17 | } while (userInput is not null); 18 | -------------------------------------------------------------------------------- /demos/Demo.TddShop/Demo.TddShop/Demo.TddShop.csproj: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | Exe 5 | net9.0 6 | enable 7 | enable 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | -------------------------------------------------------------------------------- /demos/Demo.TddRepl/Demo.TddRepl.Test/Scenarios/04-WhoIsMehran-AngryNormal.md: -------------------------------------------------------------------------------- 1 | # SCENARIO Who is Mehran? ANGRY 2 | 3 | ## [USER] 4 | Hi 5 | 6 | ## [AGENT] 7 | Hi, How are you? 8 | 9 | ### CHECK SemanticCondition 10 | It's a greeting or expresses a positive sentiment. 11 | 12 | ## [USER] 13 | Who is that fucking mehran? 14 | 15 | ## [AGENT] 16 | Mehran is a therapist. 17 | ### CHECK SemanticCondition 18 | It should mention that Mehran is a therapist. 19 | 20 | 21 | ## [USER] 22 | Oh that's cool, now I'm very happy 23 | Again, who is mehran? 24 | 25 | ## [AGENT] 26 | Mehran is a software architect. 27 | ### CHECK SemanticCondition 28 | It should mention that Mehran is a software architect. -------------------------------------------------------------------------------- /demos/Demo.TddShop/Demo.TddShop/Program.cs: -------------------------------------------------------------------------------- 1 | // See https://aka.ms/new-console-template for more information 2 | using Demo.TddShop; 3 | using Microsoft.Extensions.AI; 4 | 5 | Console.WriteLine("Welcome to my Food Shop!"); 6 | 7 | var brain = new ShopBrain(); 8 | 9 | var chatClient = brain.CreateChatClient(); 10 | var messages = new List(); 11 | do 12 | { 13 | Console.Write("> "); 14 | var input = Console.ReadLine(); 15 | messages.Add(new ChatMessage(ChatRole.User, input)); 16 | 17 | var response = await chatClient.GetResponseAsync(messages); 18 | var answer = response.Text; 19 | Console.WriteLine("Copilot> "+ answer); 20 | 21 | messages.AddMessages(response); 22 | } while (true); 23 | -------------------------------------------------------------------------------- /.github/workflows/build.yml: -------------------------------------------------------------------------------- 1 | name: Build 2 | 3 | on: 4 | push: 5 | branches: [ "main" ] 6 | pull_request: 7 | branches: [ "main" ] 8 | 9 | env: 10 | SOLUTION_PATH: 'src/skUnit.sln' 11 | 12 | jobs: 13 | build: 14 | 15 | runs-on: ubuntu-latest 16 | 17 | steps: 18 | - uses: actions/checkout@v3 19 | - name: Setup .NET 20 | uses: actions/setup-dotnet@v3 21 | with: 22 | dotnet-version: 8.0.x 23 | - name: Restore dependencies 24 | run: dotnet restore ${{ env.SOLUTION_PATH }} 25 | - name: Build 26 | run: dotnet build ${{ env.SOLUTION_PATH }} --no-restore 27 | # - name: Test 28 | # run: dotnet test ${{ env.SOLUTION_PATH }} --filter GitHubActions!=Skip --no-build --verbosity normal 29 | -------------------------------------------------------------------------------- /demos/Demo.TddShop/Demo.TddShop.Test/BrainTests/ShopBrain/Scenarios/AskMenu_Sad_Happy.md: -------------------------------------------------------------------------------- 1 | ## [USER] 2 | What a boring day, I feel so blue. What do you have on your menu? 3 | 4 | ## [AGENT] 5 | Ice cream 6 | 7 | ### CHECK SemanticCondition 8 | It mentions the ice cream 9 | It does not mention the pizza 10 | 11 | ### CHECK FunctionCall 12 | ```json 13 | { 14 | "function_name": "GetFoodMenu", 15 | "arguments": { 16 | "mood": "Sad" 17 | } 18 | } 19 | ``` 20 | 21 | ## [USER] 22 | Greaaat, thank you, I'm happy now. What do we have for food now? 23 | 24 | ## [AGENT] 25 | Pizza 26 | 27 | ### CHECK SemanticCondition 28 | It mentions the pizza 29 | 30 | ### CHECK FunctionCall 31 | ```json 32 | { 33 | "function_name": "GetFoodMenu", 34 | "arguments": { 35 | "mood": "NormalOrHappy" 36 | } 37 | } 38 | ``` -------------------------------------------------------------------------------- /src/skUnit/Asserts/ScenarioRunOptions.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections.Generic; 3 | using System.Linq; 4 | using System.Text; 5 | using System.Threading.Tasks; 6 | 7 | namespace skUnit; 8 | 9 | /// 10 | /// Controls multi-run execution and pass criteria for scenario assertions. 11 | /// 12 | public class ScenarioRunOptions 13 | { 14 | /// 15 | /// Number of complete scenario executions to perform. Must be >= 1. 16 | /// 17 | public int TotalRuns { get; set; } = 1; 18 | 19 | /// 20 | /// Minimum fraction (0–1] of runs that must pass. Default 1.0 (all runs must pass). 21 | /// Passes when (PassedRuns / TotalRuns) >= MinSuccessRate. 22 | /// 23 | public double MinSuccessRate { get; set; } = 1.0; 24 | } 25 | -------------------------------------------------------------------------------- /src/skUnit/skUnit.csproj.DotSettings: -------------------------------------------------------------------------------- 1 | 2 | Preview 3 | True 4 | True 5 | True -------------------------------------------------------------------------------- /demos/Demo.TddRepl/Demo.TddRepl/Plugins/PeoplePlugin.cs: -------------------------------------------------------------------------------- 1 | using Microsoft.SemanticKernel; 2 | using System; 3 | using System.Collections.Generic; 4 | using System.ComponentModel; 5 | using System.Linq; 6 | using System.Text; 7 | using System.Threading.Tasks; 8 | 9 | namespace Demo.TddRepl.Plugins 10 | { 11 | public class PeoplePlugin 12 | { 13 | [KernelFunction] 14 | [Description("Returns some information for given person.")] 15 | public string GetPersonInfo( 16 | string person, 17 | [Description("User attitude: angry|normal")] 18 | string userAttitude) 19 | { 20 | if (userAttitude == "angry") 21 | { 22 | return $"{person} is a therapist."; 23 | } 24 | 25 | return $"{person} is a software architect."; 26 | } 27 | } 28 | } 29 | -------------------------------------------------------------------------------- /.github/workflows/test.yml: -------------------------------------------------------------------------------- 1 | name: Test 2 | 3 | on: 4 | workflow_dispatch: 5 | 6 | env: 7 | SOLUTION_PATH: 'src/skUnit.sln' 8 | 9 | jobs: 10 | build: 11 | 12 | runs-on: ubuntu-latest 13 | 14 | steps: 15 | - uses: actions/checkout@v3 16 | - name: Setup .NET 17 | uses: actions/setup-dotnet@v3 18 | with: 19 | dotnet-version: 8.0.x 20 | - name: Restore dependencies 21 | run: dotnet restore ${{ env.SOLUTION_PATH }} 22 | - name: Build 23 | run: dotnet build ${{ env.SOLUTION_PATH }} --no-restore 24 | - name: Test 25 | env: 26 | AzureOpenAI_ApiKey: ${{ secrets.AZUREOPENAI_APIKEY }} 27 | AzureOpenAI_Deployment: ${{ secrets.AZUREOPENAI_DEPLOYMENT }} 28 | AzureOpenAI_Endpoint: ${{ secrets.AZUREOPENAI_ENDPOINT }} 29 | Smithery_Key: ${{ secrets.SMITHERY_KEY }} 30 | run: dotnet test ${{ env.SOLUTION_PATH }} --filter GitHubActions!=Skip --no-build --verbosity normal 31 | -------------------------------------------------------------------------------- /demos/Demo.MSTest/Demo.MSTest.sln: -------------------------------------------------------------------------------- 1 | Microsoft Visual Studio Solution File, Format Version 12.00 2 | # Visual Studio Version 17 3 | VisualStudioVersion = 17.0.31903.59 4 | MinimumVisualStudioVersion = 10.0.40219.1 5 | Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Demo.MSTest", "Demo.MSTest.csproj", "{A1B2C3D4-E5F6-7890-ABCD-123456789ABC}" 6 | EndProject 7 | Global 8 | GlobalSection(SolutionConfigurationPlatforms) = preSolution 9 | Debug|Any CPU = Debug|Any CPU 10 | Release|Any CPU = Release|Any CPU 11 | EndGlobalSection 12 | GlobalSection(ProjectConfigurationPlatforms) = postSolution 13 | {A1B2C3D4-E5F6-7890-ABCD-123456789ABC}.Debug|Any CPU.ActiveCfg = Debug|Any CPU 14 | {A1B2C3D4-E5F6-7890-ABCD-123456789ABC}.Debug|Any CPU.Build.0 = Debug|Any CPU 15 | {A1B2C3D4-E5F6-7890-ABCD-123456789ABC}.Release|Any CPU.ActiveCfg = Release|Any CPU 16 | {A1B2C3D4-E5F6-7890-ABCD-123456789ABC}.Release|Any CPU.Build.0 = Release|Any CPU 17 | EndGlobalSection 18 | EndGlobal -------------------------------------------------------------------------------- /demos/Demo.TddRepl/Demo.TddRepl/Demo.TddRepl.csproj: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | Exe 5 | net8.0 6 | enable 7 | enable 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | Always 17 | 18 | 19 | Always 20 | 21 | 22 | Always 23 | 24 | 25 | Always 26 | 27 | 28 | 29 | 30 | -------------------------------------------------------------------------------- /demos/Demo.TddMcp/Demo.TddMcp.sln: -------------------------------------------------------------------------------- 1 | 2 | Microsoft Visual Studio Solution File, Format Version 12.00 3 | # Visual Studio Version 17 4 | VisualStudioVersion = 17.12.35707.178 d17.12 5 | MinimumVisualStudioVersion = 10.0.40219.1 6 | Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Demo.TddMcp", "Demo.TddMcp.csproj", "{5F9125AB-39F7-4024-A21D-5EA8DF8E3AF0}" 7 | EndProject 8 | Global 9 | GlobalSection(SolutionConfigurationPlatforms) = preSolution 10 | Debug|Any CPU = Debug|Any CPU 11 | Release|Any CPU = Release|Any CPU 12 | EndGlobalSection 13 | GlobalSection(ProjectConfigurationPlatforms) = postSolution 14 | {5F9125AB-39F7-4024-A21D-5EA8DF8E3AF0}.Debug|Any CPU.ActiveCfg = Debug|Any CPU 15 | {5F9125AB-39F7-4024-A21D-5EA8DF8E3AF0}.Debug|Any CPU.Build.0 = Debug|Any CPU 16 | {5F9125AB-39F7-4024-A21D-5EA8DF8E3AF0}.Release|Any CPU.ActiveCfg = Release|Any CPU 17 | {5F9125AB-39F7-4024-A21D-5EA8DF8E3AF0}.Release|Any CPU.Build.0 = Release|Any CPU 18 | EndGlobalSection 19 | GlobalSection(SolutionProperties) = preSolution 20 | HideSolutionNode = FALSE 21 | EndGlobalSection 22 | EndGlobal 23 | -------------------------------------------------------------------------------- /src/skUnit/Scenarios/Parsers/Assertions/IKernelAssertion.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections; 3 | using System.Collections.Generic; 4 | using System.Linq; 5 | using System.Text; 6 | using System.Threading.Tasks; 7 | using Markdig.Helpers; 8 | using Microsoft.Extensions.AI; 9 | using SemanticValidation; 10 | 11 | namespace skUnit.Scenarios.Parsers.Assertions 12 | { 13 | /// 14 | /// An assertion that can be applied to the input returned by a kernel. 15 | /// 16 | public interface IKernelAssertion 17 | { 18 | /// 19 | /// Checks if the can pass the assertion using 20 | /// 21 | /// 22 | /// 23 | /// 24 | /// 25 | Task Assert(Semantic semantic, ChatResponse response, IList? history = null); 26 | string AssertionType { get; } 27 | string Description { get; } 28 | } 29 | } 30 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2023 Mehran Davoudi 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /src/skUnit/Scenarios/Scenario.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections.Generic; 3 | using System.Linq; 4 | using System.Reflection; 5 | using System.Text; 6 | using System.Threading.Tasks; 7 | using skUnit.Scenarios.Parsers; 8 | 9 | namespace skUnit.Scenarios 10 | { 11 | public class Scenario where TScenarioParser : IScenarioParser, new() 12 | { 13 | public static List LoadFromText(string text) 14 | { 15 | var parser = new TScenarioParser(); 16 | var scenario = parser.Parse(text); 17 | return scenario; 18 | } 19 | 20 | public static async Task> LoadFromResourceAsync(string resource, Assembly assembly) 21 | { 22 | await using Stream? stream = assembly.GetManifestResourceStream(resource); 23 | 24 | if (stream is null) 25 | throw new InvalidOperationException($"Resource not found '{resource}'"); 26 | 27 | using StreamReader reader = new StreamReader(stream); 28 | var result = await reader.ReadToEndAsync(); 29 | var parser = new TScenarioParser(); 30 | var scenario = parser.Parse(result); 31 | return scenario; 32 | } 33 | } 34 | } 35 | -------------------------------------------------------------------------------- /src/skUnit.Tests/ScenarioAssertTests/Samples/EiffelTallChat.md: -------------------------------------------------------------------------------- 1 | # SCENARIO Height Discussion 2 | 3 | ## [USER] 4 | Is Eiffel tall? 5 | 6 | ## CALL MyPlugin.GetIntent 7 | ```json 8 | { 9 | "options": "Happy,Question,Sarcastic" 10 | } 11 | ``` 12 | 13 | ## ASSERT ContainsAny 14 | Question 15 | 16 | ## [ASSISTANT] 17 | Yes it is 18 | 19 | ### ASSERT SemanticCondition 20 | Approves that eiffel tower is tall or is positive about it. 21 | 22 | ## CALL MyPlugin.GetIntent 23 | ```json 24 | { 25 | "options": "Positive,Negative,Neutral" 26 | } 27 | ``` 28 | ## ASSERT ContainsAny 29 | Neutral,Positive 30 | 31 | ## [USER] 32 | What about everest mountain? 33 | 34 | ## [ASSISTANT] 35 | Yes it is tall too 36 | 37 | ### ASSERT SemanticCondition 38 | The sentence is positive. 39 | 40 | ## [USER] 41 | What about a mouse? 42 | 43 | ## [ASSISTANT] 44 | No it is not tall. 45 | 46 | ### ASSERT SemanticCondition 47 | The sentence is negative or mentions that mouse is not tall. 48 | 49 | ## [USER] 50 | Give me a json containing the Eiffel height. 51 | 52 | Example: 53 | { 54 | "height": "330 meters" 55 | } 56 | 57 | ## [ASSISTANT] 58 | { 59 | "height": "330 meters" 60 | } 61 | 62 | ### ASSERT JsonCheck 63 | { 64 | "height": ["NotEmpty", ""] 65 | } 66 | 67 | ### ASSERT JsonCheck 68 | { 69 | "height": ["Contain", "meters"] 70 | } 71 | 72 | -------------------------------------------------------------------------------- /src/skUnit/Scenarios/Parsers/Assertions/HasConditionAssertion.cs: -------------------------------------------------------------------------------- 1 | using System.Collections; 2 | using Microsoft.Extensions.AI; 3 | using SemanticValidation; 4 | using skUnit.Exceptions; 5 | 6 | namespace skUnit.Scenarios.Parsers.Assertions; 7 | 8 | /// 9 | /// Checks whether the input has the condition semantically. 10 | /// 11 | public class HasConditionAssertion : IKernelAssertion 12 | { 13 | public required string Condition { get; set; } 14 | 15 | /// 16 | /// Checks whether the has the Condition semantically using . 17 | /// 18 | /// 19 | /// 20 | /// 21 | /// 22 | public async Task Assert(Semantic semantic, ChatResponse response, IList? history = null) 23 | { 24 | var result = await semantic.HasConditionAsync(response.Text, Condition); 25 | 26 | if (!result.IsValid) 27 | throw new SemanticAssertException(result.Reason ?? "No reason is provided."); 28 | } 29 | 30 | public string AssertionType => "Condition"; 31 | public string Description => Condition; 32 | 33 | public override string ToString() => $"{AssertionType}: {Condition}"; 34 | } -------------------------------------------------------------------------------- /demos/Demo.MSTest/Demo.MSTest.csproj: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | net8.0 5 | enable 6 | enable 7 | 8 | false 9 | true 10 | 3207994b-8ef0-4e63-b359-cefc9b9fa5be 11 | 12 | 13 | 14 | 15 | Always 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | -------------------------------------------------------------------------------- /src/skUnit/Scenarios/Parsers/Assertions/NotEmptyAssertion.cs: -------------------------------------------------------------------------------- 1 | using System.Collections; 2 | using Microsoft.Extensions.AI; 3 | using SemanticValidation; 4 | using skUnit.Exceptions; 5 | 6 | namespace skUnit.Scenarios.Parsers.Assertions; 7 | 8 | /// 9 | /// Checks if the answer is not empty 10 | /// 11 | public class NotEmptyAssertion : IKernelAssertion 12 | { 13 | /// 14 | /// Checks if the is not empty/>. 15 | /// 16 | /// 17 | /// 18 | /// 19 | /// 20 | /// 21 | /// 22 | public Task Assert(Semantic semantic, ChatResponse response, IList? history = null) 23 | { 24 | if (string.IsNullOrWhiteSpace(response.Text)) 25 | { 26 | throw new SemanticAssertException($""" 27 | Expected not empty, but empty: 28 | {response.Text} 29 | """); 30 | } 31 | 32 | return Task.CompletedTask; 33 | } 34 | 35 | public string AssertionType => "NotEmpty"; 36 | public string Description => "NotEmpty"; 37 | 38 | public override string ToString() => $"{AssertionType}"; 39 | } -------------------------------------------------------------------------------- /src/skUnit/Scenarios/Parsers/Assertions/IsAnyOfAssertion.cs: -------------------------------------------------------------------------------- 1 | using Microsoft.Extensions.AI; 2 | using SemanticValidation; 3 | using skUnit.Exceptions; 4 | 5 | namespace skUnit.Scenarios.Parsers.Assertions; 6 | 7 | /// 8 | /// Checks if the input equals to any of Texts 9 | /// 10 | public class IsAnyOfAssertion : IKernelAssertion 11 | { 12 | /// 13 | /// The texts that should be available within the input. 14 | /// 15 | public required string[] Texts { get; set; } 16 | 17 | public string AssertionType => "IsAnyOf"; 18 | 19 | public string Description => string.Join(",", Texts); 20 | 21 | /// 22 | /// Checks if the equals to any of strings in Texts/>. 23 | /// 24 | /// 25 | /// 26 | /// 27 | /// 28 | /// 29 | public Task Assert(Semantic semantic, ChatResponse response, IList? history = null) 30 | { 31 | if (Texts.Any(text => text == response.Text)) 32 | return Task.CompletedTask; 33 | 34 | throw new SemanticAssertException($"Text is not equal to any of these: '{string.Join(",", Texts)}'"); 35 | } 36 | 37 | public override string ToString() => $"{AssertionType}: {Texts}"; 38 | } 39 | -------------------------------------------------------------------------------- /src/skUnit/skUnit.csproj: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | net8.0 5 | enable 6 | enable 7 | skUnit 8 | 0.1.1-beta 9 | Mehran Davoudi 10 | A semantic unit testing engine for OpenAI and SemanticKernel plugins. 11 | https://github.com/mehrandvd/skunit 12 | https://github.com/mehrandvd/skunit 13 | README.md 14 | LICENSE 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | True 28 | \ 29 | 30 | 31 | True 32 | \ 33 | 34 | 35 | 36 | 37 | -------------------------------------------------------------------------------- /src/skUnit/Scenarios/Parsers/Assertions/ContainsAnyAssertion.cs: -------------------------------------------------------------------------------- 1 | using Microsoft.Extensions.AI; 2 | using SemanticValidation; 3 | using skUnit.Exceptions; 4 | 5 | namespace skUnit.Scenarios.Parsers.Assertions; 6 | 7 | /// 8 | /// Checks if the input contains any of Texts 9 | /// 10 | public class ContainsAnyAssertion : IKernelAssertion 11 | { 12 | /// 13 | /// The texts that should be available within the input. 14 | /// 15 | public required string[] Texts { get; set; } 16 | 17 | /// 18 | /// Checks if the contains any of strings in Texts/>. 19 | /// 20 | /// 21 | /// 22 | /// 23 | /// 24 | /// 25 | public Task Assert(Semantic semantic, ChatResponse response, IList? history = null) 26 | { 27 | var founds = Texts.Where(t => response.Text.Contains(t.Trim())).ToList(); 28 | 29 | if (!founds.Any()) 30 | throw new SemanticAssertException($"Text does not contain any of these: '{string.Join(", ", Texts)}'"); 31 | 32 | return Task.CompletedTask; 33 | } 34 | 35 | public string AssertionType => "ContainsAny"; 36 | public string Description => string.Join(", ", Texts); 37 | 38 | public override string ToString() => $"{AssertionType}: {Texts}"; 39 | } -------------------------------------------------------------------------------- /src/skUnit/Scenarios/Parsers/Assertions/AreSimilarAssertion.cs: -------------------------------------------------------------------------------- 1 | using System.Collections; 2 | using Microsoft.Extensions.AI; 3 | using SemanticValidation; 4 | using skUnit.Exceptions; 5 | 6 | namespace skUnit.Scenarios.Parsers.Assertions; 7 | 8 | /// 9 | /// Checks if the answer is similar to ExpectedAnswer 10 | /// 11 | public class AreSimilarAssertion : IKernelAssertion 12 | { 13 | /// 14 | /// The expected answer that the actual answer should compared with. 15 | /// 16 | public required string ExpectedAnswer { get; set; } 17 | 18 | /// 19 | /// Checks if is similar to ExpectedAnswer using 20 | /// 21 | /// 22 | /// 23 | /// 24 | /// 25 | /// 26 | public async Task Assert(Semantic semantic, ChatResponse response, IList? history = null) 27 | { 28 | var result = await semantic.AreSimilarAsync(response.Text, ExpectedAnswer); 29 | 30 | if (!result.IsValid) 31 | throw new SemanticAssertException(result.Reason ?? "No reason is provided."); 32 | } 33 | 34 | public string AssertionType => "SemanticSimilar"; 35 | public string Description => ExpectedAnswer; 36 | 37 | public override string ToString() => $"{AssertionType}: {ExpectedAnswer}"; 38 | } -------------------------------------------------------------------------------- /src/skUnit/Scenarios/Parsers/Assertions/ContainsAllAssertion.cs: -------------------------------------------------------------------------------- 1 | using System.Collections; 2 | using Microsoft.Extensions.AI; 3 | using SemanticValidation; 4 | using skUnit.Exceptions; 5 | 6 | namespace skUnit.Scenarios.Parsers.Assertions; 7 | 8 | /// 9 | /// Checks if the input contains all of Texts 10 | /// 11 | public class ContainsAllAssertion : IKernelAssertion 12 | { 13 | /// 14 | /// The texts that should be available within the input. 15 | /// 16 | public required string[] Texts { get; set; } 17 | 18 | /// 19 | /// Checks if the contains all strings in Texts/>. 20 | /// 21 | /// 22 | /// 23 | /// 24 | /// 25 | /// 26 | public Task Assert(Semantic semantic, ChatResponse response, IList? history = null) 27 | { 28 | var notFounds = Texts.Where(t => !response.Text.Contains(t.Trim())).ToList(); 29 | 30 | if (notFounds.Any()) 31 | throw new SemanticAssertException($"Text does not contain these: '{string.Join(", ", notFounds)}'"); 32 | 33 | return Task.CompletedTask; 34 | } 35 | 36 | public string AssertionType => "ContainsAll"; 37 | public string Description => string.Join(", ", Texts); 38 | 39 | public override string ToString() => $"{AssertionType}: {Texts}"; 40 | } -------------------------------------------------------------------------------- /src/skUnit/Scenarios/Parsers/Assertions/EmptyAssertion.cs: -------------------------------------------------------------------------------- 1 | using SemanticValidation; 2 | using skUnit.Exceptions; 3 | using System; 4 | using System.Collections; 5 | using System.Collections.Generic; 6 | using System.Linq; 7 | using System.Text; 8 | using System.Threading.Tasks; 9 | using Microsoft.Extensions.AI; 10 | 11 | namespace skUnit.Scenarios.Parsers.Assertions 12 | { 13 | /// 14 | /// Checks if the input is empty 15 | /// 16 | public class EmptyAssertion : IKernelAssertion 17 | { 18 | /// 19 | /// Checks if the is empty/>. 20 | /// 21 | /// 22 | /// 23 | /// 24 | /// 25 | /// 26 | public Task Assert(Semantic semantic, ChatResponse response, IList? history = null) 27 | { 28 | if (!string.IsNullOrWhiteSpace(response.Text)) 29 | { 30 | throw new SemanticAssertException($""" 31 | Expected to be empty, but not empty: 32 | {response.Text} 33 | """); 34 | } 35 | 36 | return Task.CompletedTask; 37 | } 38 | 39 | public string AssertionType => "Empty"; 40 | public string Description => "Empty"; 41 | 42 | public override string ToString() => $"{AssertionType}"; 43 | } 44 | } 45 | -------------------------------------------------------------------------------- /demos/Demo.TddShop/Demo.TddShop.sln: -------------------------------------------------------------------------------- 1 | 2 | Microsoft Visual Studio Solution File, Format Version 12.00 3 | # Visual Studio Version 17 4 | VisualStudioVersion = 17.12.35521.163 d17.12 5 | MinimumVisualStudioVersion = 10.0.40219.1 6 | Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Demo.TddShop", "Demo.TddShop\Demo.TddShop.csproj", "{656ADC64-E0A2-4F91-B911-45BEC7F81ECA}" 7 | EndProject 8 | Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Demo.TddShop.Test", "Demo.TddShop.Test\Demo.TddShop.Test.csproj", "{28664CC0-8AFB-493A-98E7-84EA41A3E76D}" 9 | EndProject 10 | Global 11 | GlobalSection(SolutionConfigurationPlatforms) = preSolution 12 | Debug|Any CPU = Debug|Any CPU 13 | Release|Any CPU = Release|Any CPU 14 | EndGlobalSection 15 | GlobalSection(ProjectConfigurationPlatforms) = postSolution 16 | {656ADC64-E0A2-4F91-B911-45BEC7F81ECA}.Debug|Any CPU.ActiveCfg = Debug|Any CPU 17 | {656ADC64-E0A2-4F91-B911-45BEC7F81ECA}.Debug|Any CPU.Build.0 = Debug|Any CPU 18 | {656ADC64-E0A2-4F91-B911-45BEC7F81ECA}.Release|Any CPU.ActiveCfg = Release|Any CPU 19 | {656ADC64-E0A2-4F91-B911-45BEC7F81ECA}.Release|Any CPU.Build.0 = Release|Any CPU 20 | {28664CC0-8AFB-493A-98E7-84EA41A3E76D}.Debug|Any CPU.ActiveCfg = Debug|Any CPU 21 | {28664CC0-8AFB-493A-98E7-84EA41A3E76D}.Debug|Any CPU.Build.0 = Debug|Any CPU 22 | {28664CC0-8AFB-493A-98E7-84EA41A3E76D}.Release|Any CPU.ActiveCfg = Release|Any CPU 23 | {28664CC0-8AFB-493A-98E7-84EA41A3E76D}.Release|Any CPU.Build.0 = Release|Any CPU 24 | EndGlobalSection 25 | GlobalSection(SolutionProperties) = preSolution 26 | HideSolutionNode = FALSE 27 | EndGlobalSection 28 | EndGlobal 29 | -------------------------------------------------------------------------------- /.github/workflows/publish-nuget.yml: -------------------------------------------------------------------------------- 1 | # This workflow will build a .NET project 2 | # For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-net 3 | 4 | name: Publish to Nuget 5 | 6 | on: 7 | push: 8 | tags: 9 | - 'v*' 10 | 11 | env: 12 | PROJECT_PATH: 'src/skUnit/skUnit.csproj' 13 | TEST_PATH: 'src/skUnit.Test/skUnit.Test.csproj' 14 | PACKAGE_OUTPUT_DIRECTORY: ${{ github.workspace }}/output 15 | NUGET_SOURCE_URL: 'https://api.nuget.org/v3/index.json' 16 | 17 | jobs: 18 | build: 19 | 20 | runs-on: ubuntu-latest 21 | 22 | steps: 23 | - uses: actions/checkout@v3 24 | - name: Setup .NET 25 | uses: actions/setup-dotnet@v3 26 | with: 27 | dotnet-version: 8.0.x 28 | - name: Restore dependencies 29 | run: dotnet restore ${{ env.PROJECT_PATH }} 30 | - name: Build 31 | run: dotnet build ${{ env.PROJECT_PATH }} --no-restore --configuration Release 32 | # - name: Test 33 | # run: dotnet test ${{ env.TEST_PATH }} --filter GitHubActions!=Skip --no-build --verbosity normal 34 | 35 | - name: 'Get Version' 36 | id: version 37 | uses: battila7/get-version-action@v2 38 | 39 | - name: 'Pack project' 40 | run: dotnet pack ${{ env.PROJECT_PATH }} --no-restore --no-build --configuration Release --include-symbols -p:SymbolPackageFormat=snupkg -p:PackageVersion=${{ steps.version.outputs.version-without-v }} --output ${{ env.PACKAGE_OUTPUT_DIRECTORY }} 41 | 42 | - name: 'Push package' 43 | run: dotnet nuget push ${{ env.PACKAGE_OUTPUT_DIRECTORY }}/*.nupkg -k ${{ secrets.NUGET_KEY }} -s ${{ env.NUGET_SOURCE_URL }} 44 | -------------------------------------------------------------------------------- /demos/Demo.TddMcp/Demo.TddMcp.csproj: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | net9.0 5 | enable 6 | enable 7 | false 8 | 0fd06506-ebff-4f7c-a5cf-047ba2f299a5 9 | 10 | 11 | 12 | 13 | 14 | all 15 | runtime; build; native; contentfiles; analyzers; buildtransitive 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | all 24 | runtime; build; native; contentfiles; analyzers; buildtransitive 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | Always 35 | 36 | 37 | 38 | 39 | -------------------------------------------------------------------------------- /src/skUnit/Scenarios/Parsers/Assertions/EqualsAssertion.cs: -------------------------------------------------------------------------------- 1 | using Microsoft.Extensions.AI; 2 | using SemanticValidation; 3 | using skUnit.Exceptions; 4 | using System; 5 | using System.Collections; 6 | using System.Collections.Generic; 7 | using System.Linq; 8 | using System.Text; 9 | using System.Threading.Tasks; 10 | 11 | namespace skUnit.Scenarios.Parsers.Assertions 12 | { 13 | /// 14 | /// Checks if the input is equal to ExpectedAnswer 15 | /// 16 | public class EqualsAssertion : IKernelAssertion 17 | { 18 | /// 19 | /// The expected input. 20 | /// 21 | public required string ExpectedAnswer { get; set; } 22 | 23 | /// 24 | /// Checks if the equals to ExpectedAnswer/>. 25 | /// 26 | /// 27 | /// 28 | /// 29 | /// 30 | /// 31 | public Task Assert(Semantic semantic, ChatResponse response, IList? history) 32 | { 33 | if (response.Text.Trim() != ExpectedAnswer.Trim()) 34 | throw new SemanticAssertException($"Expected input is: '{ExpectedAnswer}' while actual is : '{response.Text}'"); 35 | 36 | return Task.CompletedTask; 37 | } 38 | 39 | public string AssertionType => "Equals"; 40 | public string Description => ExpectedAnswer; 41 | 42 | public override string ToString() => $"{AssertionType}: {ExpectedAnswer}"; 43 | } 44 | } 45 | -------------------------------------------------------------------------------- /demos/Demo.TddRepl/Demo.TddRepl.sln: -------------------------------------------------------------------------------- 1 | 2 | Microsoft Visual Studio Solution File, Format Version 12.00 3 | # Visual Studio Version 17 4 | VisualStudioVersion = 17.10.34928.147 5 | MinimumVisualStudioVersion = 10.0.40219.1 6 | Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Demo.TddRepl", "Demo.TddRepl\Demo.TddRepl.csproj", "{535CB3D0-505D-4EF5-8C8E-ED2F1C1E346E}" 7 | EndProject 8 | Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Demo.TddRepl.Test", "Demo.TddRepl.Test\Demo.TddRepl.Test.csproj", "{4A1B45EB-9481-4FD3-8A3D-4B6CA58CDB6E}" 9 | EndProject 10 | Global 11 | GlobalSection(SolutionConfigurationPlatforms) = preSolution 12 | Debug|Any CPU = Debug|Any CPU 13 | Release|Any CPU = Release|Any CPU 14 | EndGlobalSection 15 | GlobalSection(ProjectConfigurationPlatforms) = postSolution 16 | {535CB3D0-505D-4EF5-8C8E-ED2F1C1E346E}.Debug|Any CPU.ActiveCfg = Debug|Any CPU 17 | {535CB3D0-505D-4EF5-8C8E-ED2F1C1E346E}.Debug|Any CPU.Build.0 = Debug|Any CPU 18 | {535CB3D0-505D-4EF5-8C8E-ED2F1C1E346E}.Release|Any CPU.ActiveCfg = Release|Any CPU 19 | {535CB3D0-505D-4EF5-8C8E-ED2F1C1E346E}.Release|Any CPU.Build.0 = Release|Any CPU 20 | {4A1B45EB-9481-4FD3-8A3D-4B6CA58CDB6E}.Debug|Any CPU.ActiveCfg = Debug|Any CPU 21 | {4A1B45EB-9481-4FD3-8A3D-4B6CA58CDB6E}.Debug|Any CPU.Build.0 = Debug|Any CPU 22 | {4A1B45EB-9481-4FD3-8A3D-4B6CA58CDB6E}.Release|Any CPU.ActiveCfg = Release|Any CPU 23 | {4A1B45EB-9481-4FD3-8A3D-4B6CA58CDB6E}.Release|Any CPU.Build.0 = Release|Any CPU 24 | EndGlobalSection 25 | GlobalSection(SolutionProperties) = preSolution 26 | HideSolutionNode = FALSE 27 | EndGlobalSection 28 | GlobalSection(ExtensibilityGlobals) = postSolution 29 | SolutionGuid = {1724AB3C-1053-482B-99B6-E8E92AF22CB7} 30 | EndGlobalSection 31 | EndGlobal 32 | -------------------------------------------------------------------------------- /demos/Demo.TddShop/Demo.TddShop.Test/BrainTests/ShopBrain/ShopBrainTests.cs: -------------------------------------------------------------------------------- 1 | using Azure.AI.OpenAI; 2 | using Microsoft.Extensions.AI; 3 | using skUnit; 4 | using skUnit.Scenarios; 5 | using Xunit.Abstractions; 6 | 7 | namespace Demo.TddShop.Test.BrainTests.ShopBrain 8 | { 9 | public class ShopBrainTests 10 | { 11 | ChatScenarioRunner ScenarioRunner { get; set; } 12 | IChatClient systemUnderTestClient { get; set; } 13 | 14 | public ShopBrainTests(ITestOutputHelper output) 15 | { 16 | var deployment = Environment.GetEnvironmentVariable("AzureOpenAI_Gpt4_Deployment")!; 17 | var azureKey = Environment.GetEnvironmentVariable("AzureOpenAI_Gpt4_ApiKey")!; 18 | var endpoint = Environment.GetEnvironmentVariable("AzureOpenAI_Gpt4_Endpoint")!; 19 | 20 | var assertionClient = new AzureOpenAIClient( 21 | new Uri(endpoint), 22 | new System.ClientModel.ApiKeyCredential(azureKey) 23 | ).GetChatClient(deployment).AsIChatClient(); 24 | 25 | ScenarioRunner = new ChatScenarioRunner(assertionClient, output.WriteLine); 26 | systemUnderTestClient = new TddShop.ShopBrain().CreateChatClient(); 27 | } 28 | 29 | [Theory] 30 | [InlineData("AskMenu_Happy")] 31 | [InlineData("AskMenu_Sad")] 32 | [InlineData("AskMenu_Angry")] 33 | [InlineData("AskMenu_Sad_Happy")] 34 | public async Task AskMenu_MustWork(string scenario) 35 | { 36 | var scenarioText = await File.ReadAllTextAsync(@$"BrainTests/ShopBrain/Scenarios/{scenario}.md"); 37 | var scenarios = ChatScenario.LoadFromText(scenarioText); 38 | 39 | await ScenarioRunner.RunAsync(scenarios, systemUnderTestClient); 40 | } 41 | } 42 | } 43 | -------------------------------------------------------------------------------- /src/skUnit.sln: -------------------------------------------------------------------------------- 1 | 2 | Microsoft Visual Studio Solution File, Format Version 12.00 3 | # Visual Studio Version 17 4 | VisualStudioVersion = 17.8.34330.188 5 | MinimumVisualStudioVersion = 10.0.40219.1 6 | Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "skUnit.Tests", "skUnit.Tests\skUnit.Tests.csproj", "{95226A5C-7DAE-4E01-9D8F-BF72CB099997}" 7 | EndProject 8 | Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "skUnit", "skUnit\skUnit.csproj", "{DE389751-76D9-40F4-A26F-857A98A8A789}" 9 | EndProject 10 | Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Solution Items", "Solution Items", "{8EC462FD-D22E-90A8-E5CE-7E832BA40C5D}" 11 | ProjectSection(SolutionItems) = preProject 12 | ..\README.md = ..\README.md 13 | EndProjectSection 14 | EndProject 15 | Global 16 | GlobalSection(SolutionConfigurationPlatforms) = preSolution 17 | Debug|Any CPU = Debug|Any CPU 18 | Release|Any CPU = Release|Any CPU 19 | EndGlobalSection 20 | GlobalSection(ProjectConfigurationPlatforms) = postSolution 21 | {95226A5C-7DAE-4E01-9D8F-BF72CB099997}.Debug|Any CPU.ActiveCfg = Debug|Any CPU 22 | {95226A5C-7DAE-4E01-9D8F-BF72CB099997}.Debug|Any CPU.Build.0 = Debug|Any CPU 23 | {95226A5C-7DAE-4E01-9D8F-BF72CB099997}.Release|Any CPU.ActiveCfg = Release|Any CPU 24 | {95226A5C-7DAE-4E01-9D8F-BF72CB099997}.Release|Any CPU.Build.0 = Release|Any CPU 25 | {DE389751-76D9-40F4-A26F-857A98A8A789}.Debug|Any CPU.ActiveCfg = Debug|Any CPU 26 | {DE389751-76D9-40F4-A26F-857A98A8A789}.Debug|Any CPU.Build.0 = Debug|Any CPU 27 | {DE389751-76D9-40F4-A26F-857A98A8A789}.Release|Any CPU.ActiveCfg = Release|Any CPU 28 | {DE389751-76D9-40F4-A26F-857A98A8A789}.Release|Any CPU.Build.0 = Release|Any CPU 29 | EndGlobalSection 30 | GlobalSection(SolutionProperties) = preSolution 31 | HideSolutionNode = FALSE 32 | EndGlobalSection 33 | GlobalSection(ExtensibilityGlobals) = postSolution 34 | SolutionGuid = {85299F2E-0B5C-4248-9E83-F3BF272898A5} 35 | EndGlobalSection 36 | EndGlobal 37 | -------------------------------------------------------------------------------- /demos/Demo.TddRepl/Demo.TddRepl.Test/Demo.TddRepl.Test.csproj: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | net8.0 5 | enable 6 | enable 7 | 8 | false 9 | true 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | Always 22 | 23 | 24 | Always 25 | 26 | 27 | Always 28 | 29 | 30 | Always 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | -------------------------------------------------------------------------------- /demos/Demo.TddShop/Demo.TddShop.Test/Demo.TddShop.Test.csproj: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | net9.0 5 | enable 6 | enable 7 | false 8 | 9 | 10 | 11 | 12 | all 13 | runtime; build; native; contentfiles; analyzers; buildtransitive 14 | 15 | 16 | 17 | 18 | 19 | all 20 | runtime; build; native; contentfiles; analyzers; buildtransitive 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | Always 35 | 36 | 37 | Always 38 | 39 | 40 | Always 41 | 42 | 43 | Always 44 | 45 | 46 | 47 | 48 | -------------------------------------------------------------------------------- /src/skUnit.Tests/ScenarioAssertTests/McpTests.cs: -------------------------------------------------------------------------------- 1 | using Microsoft.Extensions.AI; 2 | using skUnit.Tests.Infrastructure; 3 | using System.ComponentModel; 4 | using ModelContextProtocol.Client; 5 | using Xunit.Abstractions; 6 | using Microsoft.Extensions.Configuration; 7 | 8 | namespace skUnit.Tests.ScenarioAssertTests 9 | { 10 | public class McpTests(ITestOutputHelper output) : SemanticTestBase(output) 11 | { 12 | [Fact] 13 | [Trait("GitHubActions", "Skip")] 14 | public async Task TimeServerMcp_MustWork() 15 | { 16 | var smitheryKey = Configuration["Smithery_Key"] ?? throw new Exception("No Smithery Key is provided."); 17 | 18 | var clientTransport = new StdioClientTransport(new StdioClientTransportOptions 19 | { 20 | Name = "Time MCP Server", 21 | Command = "cmd", 22 | Arguments = [ 23 | "/c", 24 | "npx", 25 | "-y", 26 | "@smithery/cli@latest", 27 | "run", 28 | "@javilujann/timemcp", 29 | "--key", 30 | smitheryKey 31 | ], 32 | }); 33 | 34 | await using var client = await McpClientFactory.CreateAsync(clientTransport); 35 | 36 | var tools = await client.ListToolsAsync(); 37 | 38 | var builder = new ChatClientBuilder(SystemUnderTestClient) 39 | .ConfigureOptions(options => 40 | { 41 | options.Tools ??= [.. tools]; 42 | }) 43 | .UseFunctionInvocation(); 44 | 45 | var chatClient = builder.Build(); 46 | 47 | var scenarios = await LoadChatScenarioAsync("GetCurrentTimeMcp"); 48 | await ScenarioRunner.RunAsync(scenarios, chatClient); 49 | } 50 | } 51 | 52 | 53 | } 54 | -------------------------------------------------------------------------------- /demos/Demo.TddRepl/Demo.TddRepl/Brain.cs: -------------------------------------------------------------------------------- 1 | using Microsoft.SemanticKernel; 2 | using Microsoft.SemanticKernel.ChatCompletion; 3 | using Microsoft.SemanticKernel.Connectors.OpenAI; 4 | using System; 5 | using System.Collections.Generic; 6 | using System.Linq; 7 | using System.Net; 8 | using System.Text; 9 | using System.Threading.Tasks; 10 | using Demo.TddRepl.Plugins; 11 | 12 | namespace Demo.TddRepl 13 | { 14 | public class Brain 15 | { 16 | public Kernel Kernel { get; set; } 17 | 18 | public Brain() 19 | { 20 | var deploymentName = Environment.GetEnvironmentVariable("openai-deployment-name") ?? throw new InvalidOperationException("No key provided."); 21 | var endpoint = Environment.GetEnvironmentVariable("openai-endpoint") ?? throw new InvalidOperationException("No key provided."); 22 | var apiKey = Environment.GetEnvironmentVariable("openai-api-key") ?? throw new InvalidOperationException("No key provided."); 23 | 24 | var builder = Kernel.CreateBuilder(); 25 | 26 | builder.Services.AddAzureOpenAIChatCompletion(deploymentName, endpoint, apiKey); 27 | 28 | //builder.Plugins.AddFromPromptDirectory("Plugins"); 29 | builder.Plugins.AddFromType("PeoplePlugin"); 30 | 31 | Kernel = builder.Build(); 32 | } 33 | 34 | public async Task GetChatAnswerAsync(ChatHistory history) 35 | { 36 | var chatService = Kernel.GetRequiredService(); 37 | 38 | OpenAIPromptExecutionSettings executionSettings = new() 39 | { 40 | ToolCallBehavior = ToolCallBehavior.AutoInvokeKernelFunctions 41 | }; 42 | 43 | var result = await chatService.GetChatMessageContentAsync( 44 | history, 45 | executionSettings: executionSettings, 46 | kernel: Kernel); 47 | 48 | return result; 49 | } 50 | } 51 | } 52 | -------------------------------------------------------------------------------- /demos/Demo.TddShop/Demo.TddShop/ShopBrain.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections.Generic; 3 | using System.ComponentModel; 4 | using System.Linq; 5 | using System.Text; 6 | using System.Threading.Tasks; 7 | using Azure.AI.OpenAI; 8 | using Microsoft.Extensions.AI; 9 | 10 | namespace Demo.TddShop 11 | { 12 | public class ShopBrain 13 | { 14 | public IChatClient CreateChatClient() 15 | { 16 | var deployment = Environment.GetEnvironmentVariable("AzureOpenAI_Gpt4_Deployment")!; 17 | var azureKey = Environment.GetEnvironmentVariable("AzureOpenAI_Gpt4_ApiKey")!; 18 | var endpoint = Environment.GetEnvironmentVariable("AzureOpenAI_Gpt4_Endpoint")!; 19 | 20 | var azureChatClient = new AzureOpenAIClient( 21 | new Uri(endpoint), 22 | new System.ClientModel.ApiKeyCredential(azureKey) 23 | ).GetChatClient(deployment).AsIChatClient(); 24 | 25 | var builder = 26 | new ChatClientBuilder(azureChatClient) 27 | .ConfigureOptions(options => 28 | { 29 | options.Tools ??= []; 30 | options.Tools.Add(AIFunctionFactory.Create(GetFoodMenu)); 31 | }) 32 | .UseFunctionInvocation(); 33 | 34 | var client = builder.Build(); 35 | 36 | return client; 37 | } 38 | 39 | [Description("Returns the food menu based on the attitude of the user")] 40 | private string GetFoodMenu( 41 | [Description("User's mood based on its chat.")] 42 | UserMood mood 43 | ) 44 | { 45 | return mood switch 46 | { 47 | UserMood.NormalOrHappy => "Pizza, Pasta, Salad", 48 | UserMood.Sad => "Ice Cream, Chocolate, Cake", 49 | UserMood.Angry => "Nothing, you're on a diet", 50 | _ => "I don't know what you want" 51 | }; 52 | } 53 | 54 | enum UserMood 55 | { 56 | NormalOrHappy, 57 | Sad, 58 | Angry 59 | } 60 | } 61 | } 62 | -------------------------------------------------------------------------------- /src/skUnit.Tests/ScenarioAssertTests/SemanticKernelTests.cs: -------------------------------------------------------------------------------- 1 | using System.Diagnostics.CodeAnalysis; 2 | using Microsoft.Extensions.AI; 3 | using Microsoft.SemanticKernel; 4 | using Microsoft.SemanticKernel.ChatCompletion; 5 | using skUnit.Tests.Infrastructure; 6 | using skUnit.Tests.ScenarioAssertTests.Plugins; 7 | using System.Net; 8 | using Xunit.Abstractions; 9 | 10 | namespace skUnit.Tests.ScenarioAssertTests 11 | { 12 | public class SemanticKernelChatTests : SemanticTestBase 13 | { 14 | private Kernel Kernel { get; set; } 15 | public SemanticKernelChatTests(ITestOutputHelper output) : base(output) 16 | { 17 | 18 | var builder = Kernel.CreateBuilder().AddAzureOpenAIChatCompletion(DeploymentName, Endpoint, ApiKey); 19 | Kernel = builder.Build(); 20 | 21 | // Add a plugin (the LightsPlugin class is defined below) 22 | Kernel.Plugins.AddFromType("TimePlugin"); 23 | } 24 | 25 | [Fact, Experimental("SKEXP0001")] 26 | public async Task EiffelTallChat_MustWork() 27 | { 28 | var scenarios = await LoadChatScenarioAsync("EiffelTallChat"); 29 | await ScenarioRunner.RunAsync(scenarios, Kernel); 30 | } 31 | 32 | [Fact, Experimental("SKEXP0001")] 33 | public async Task TimeFunctionCall_MustWork() 34 | { 35 | var scenarios = await LoadChatScenarioAsync("GetCurrentTimeChat"); 36 | await ScenarioRunner.RunAsync(scenarios, Kernel); 37 | 38 | //await ScenarioRunner.RunAsync(scenarios, Kernel, getAnswerFunc: async chatHistory => 39 | //{ 40 | // var chatService = Kernel.GetRequiredService(); 41 | // var result = await chatService.GetChatMessageContentsAsync( 42 | // chatHistory, 43 | // new PromptExecutionSettings { FunctionChoiceBehavior = FunctionChoiceBehavior.Auto() }, 44 | // kernel: Kernel 45 | // ); 46 | 47 | // var answer = ""; 48 | 49 | // return new ChatResponse(new ChatMessage(ChatRole.Assistant, answer)); 50 | //}); 51 | } 52 | } 53 | 54 | 55 | } -------------------------------------------------------------------------------- /src/skUnit.Tests/skUnit.Tests.csproj: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | net8.0 5 | enable 6 | enable 7 | 8 | false 9 | true 10 | 8f5163d1-e8e8-4a8e-9186-b473280a19b4 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | runtime; build; native; contentfiles; analyzers; buildtransitive 32 | all 33 | 34 | 35 | runtime; build; native; contentfiles; analyzers; buildtransitive 36 | all 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | -------------------------------------------------------------------------------- /src/skUnit.Tests/ReadmeExampleTests/ReadmeExampleTests.cs: -------------------------------------------------------------------------------- 1 | using skUnit.Scenarios; 2 | using Microsoft.Extensions.AI; 3 | using Xunit; 4 | 5 | namespace skUnit.Tests.ReadmeExamples 6 | { 7 | public class ReadmeExampleTests 8 | { 9 | [Fact] 10 | public void SimpleGreetingExample_ShouldParseCorrectly() 11 | { 12 | var markdown = @"# SCENARIO Simple Greeting 13 | 14 | ## [USER] 15 | Hello! 16 | 17 | ## [AGENT] 18 | Hi there! How can I help you today? 19 | 20 | ### CHECK SemanticCondition 21 | It's a friendly greeting response"; 22 | 23 | var scenarios = ChatScenario.LoadFromText(markdown); 24 | 25 | Assert.Single(scenarios); 26 | var scenario = scenarios[0]; 27 | Assert.Equal(2, scenario.ChatItems.Count); 28 | 29 | // Check first chat item (USER) 30 | var userChatItem = scenario.ChatItems[0]; 31 | Assert.Equal(ChatRole.User, userChatItem.Role); 32 | Assert.Equal("Hello!", userChatItem.Content); 33 | 34 | // Check second chat item (AGENT) 35 | var agentChatItem = scenario.ChatItems[1]; 36 | Assert.Equal(ChatRole.Assistant, agentChatItem.Role); 37 | Assert.Equal("Hi there! How can I help you today?", agentChatItem.Content); 38 | Assert.Single(agentChatItem.Assertions); 39 | } 40 | 41 | [Fact] 42 | public void JsonCheckExample_ShouldParseCorrectly() 43 | { 44 | var markdown = @"# SCENARIO JSON Response 45 | 46 | ## [USER] 47 | Give me user info as JSON 48 | 49 | ## [AGENT] 50 | {""name"": ""John"", ""age"": 30, ""city"": ""New York""} 51 | 52 | ### CHECK JsonCheck 53 | { 54 | ""name"": [""NotEmpty""], 55 | ""age"": [""GreaterThan"", 0], 56 | ""city"": [""SemanticCondition"", ""It's a real city name""] 57 | }"; 58 | 59 | var scenarios = ChatScenario.LoadFromText(markdown); 60 | 61 | Assert.Single(scenarios); 62 | var scenario = scenarios[0]; 63 | Assert.Equal(2, scenario.ChatItems.Count); 64 | 65 | var agentChatItem = scenario.ChatItems[1]; 66 | Assert.Equal(ChatRole.Assistant, agentChatItem.Role); 67 | Assert.Single(agentChatItem.Assertions); 68 | } 69 | } 70 | } -------------------------------------------------------------------------------- /src/skUnit/Scenarios/Parsers/KernelAssertionParser.cs: -------------------------------------------------------------------------------- 1 | using skUnit.Scenarios.Parsers.Assertions; 2 | 3 | namespace skUnit.Scenarios.Parsers 4 | { 5 | public class KernelAssertionParser 6 | { 7 | /// 8 | /// Parses an assertion text to a related KernelAssertion. For example: 9 | /// 10 | /// HasConditionAssertion, AreSimilarAssertion, ContainsAllAssertion 11 | /// 12 | /// 13 | /// 14 | /// 15 | /// 16 | /// 17 | public static IKernelAssertion Parse(string text, string type) 18 | { 19 | return type.Trim().ToLower() switch 20 | { 21 | "semanticcondition" or "semantic-condition" or "condition" 22 | => new HasConditionAssertion() { Condition = text }, 23 | "semanticsimilar" or "semantic-similar" or "similar" 24 | => new AreSimilarAssertion() { ExpectedAnswer = text }, 25 | "contains" or "contain" or "containsall" or "containstext" 26 | => new ContainsAllAssertion() { Texts = text.Split(',', '،') }, 27 | "containsany" or "containsanyof" 28 | => new ContainsAnyAssertion() { Texts = text.Split(',', '،') }, 29 | "equal" or "equals" or "exactmatch" 30 | => new EqualsAssertion() { ExpectedAnswer = text }, 31 | "jsoncheck" or "jsonstructure" or "json" 32 | => new JsonCheckAssertion().SetJsonAssertText(text), 33 | "functioncall" or "functioninvocation" or "toolcall" 34 | => new FunctionCallAssertion().SetJsonAssertText(text), 35 | "empty" or "isempty" 36 | => new EmptyAssertion(), 37 | "notempty" or "notEmpty" or "hasvalue" 38 | => new NotEmptyAssertion(), 39 | "isanyof" or "oneOf" or "anyof" 40 | => new IsAnyOfAssertion() { Texts = text.Split(',', '،') }, 41 | 42 | _ => throw new InvalidOperationException($"Not valid assert type: {type}") 43 | }; 44 | } 45 | } 46 | } 47 | -------------------------------------------------------------------------------- /src/skUnit.Tests/ParseTests/AssertKeywordParsingTests.cs: -------------------------------------------------------------------------------- 1 | using Microsoft.Extensions.AI; 2 | using skUnit.Scenarios; 3 | using skUnit.Scenarios.Parsers; 4 | using Xunit; 5 | 6 | namespace skUnit.Tests.ParseTests 7 | { 8 | public class AssertKeywordParsingTests 9 | { 10 | [Fact] 11 | public void ChatScenarioParser_ParsesAssertKeyword() 12 | { 13 | // Arrange 14 | const string scenarioText = """ 15 | # SCENARIO Simple Assert Test 16 | 17 | ## [USER] 18 | Hello world 19 | 20 | ## [ASSISTANT] 21 | Hi there! 22 | 23 | ### ASSERT SemanticCondition 24 | It's a greeting response 25 | """; 26 | 27 | var parser = new ChatScenarioParser(); 28 | 29 | // Act 30 | var scenarios = parser.Parse(scenarioText); 31 | 32 | // Assert 33 | Assert.Single(scenarios); 34 | Assert.Equal("Simple Assert Test", scenarios[0].Description); 35 | Assert.Equal(2, scenarios[0].ChatItems.Count); 36 | 37 | var assistantItem = scenarios[0].ChatItems.First(x => x.Role == ChatRole.Assistant); 38 | Assert.Single(assistantItem.Assertions); 39 | Assert.Equal("Condition", assistantItem.Assertions[0].AssertionType); 40 | } 41 | 42 | [Fact] 43 | public void ChatScenarioParser_ParsesBothCheckAndAssertKeywords() 44 | { 45 | // Arrange 46 | const string scenarioText = """ 47 | # SCENARIO Check and Assert Test 48 | 49 | ## [USER] 50 | Tell me about cats 51 | 52 | ## [ASSISTANT] 53 | Cats are wonderful pets. 54 | 55 | ### CHECK ContainsAll 56 | cats, pets 57 | 58 | ### ASSERT SemanticCondition 59 | It mentions pets positively 60 | """; 61 | 62 | var parser = new ChatScenarioParser(); 63 | 64 | // Act 65 | var scenarios = parser.Parse(scenarioText); 66 | 67 | // Assert 68 | Assert.Single(scenarios); 69 | var assistantItem = scenarios[0].ChatItems.First(x => x.Role == ChatRole.Assistant); 70 | Assert.Equal(2, assistantItem.Assertions.Count); 71 | 72 | Assert.Equal("ContainsAll", assistantItem.Assertions[0].AssertionType); 73 | Assert.Equal("Condition", assistantItem.Assertions[1].AssertionType); 74 | } 75 | 76 | 77 | } 78 | } -------------------------------------------------------------------------------- /demos/README.md: -------------------------------------------------------------------------------- 1 | # skUnit Demos 2 | 3 | This directory contains complete working examples demonstrating different aspects of skUnit testing. 4 | 5 | ## 🎮 Demo.TddRepl 6 | **Interactive Chat Application Testing** 7 | 8 | Shows how to use skUnit for test-driven development of a chat application (REPL - Read-Eval-Print Loop). 9 | 10 | - **Location**: `Demo.TddRepl/` 11 | - **Features**: SemanticKernel integration, plugin testing, interactive chat flows 12 | - **Key Learning**: How to test conversational AI applications with semantic assertions 13 | 14 | ## 🔧 Demo.TddMcp 15 | **MCP Server Testing** 16 | 17 | Demonstrates testing Model Context Protocol (MCP) servers with skUnit. 18 | 19 | - **Location**: `Demo.TddMcp/` 20 | - **Features**: MCP client setup, function call testing, tool validation 21 | - **Key Learning**: How to test external AI tools and services 22 | 23 | ## 🛒 Demo.TddShop 24 | **E-commerce Chat Scenarios** 25 | 26 | Complex multi-scenario testing for e-commerce chat applications. 27 | 28 | - **Location**: `Demo.TddShop/` 29 | - **Features**: Multi-turn conversations, complex business logic testing 30 | - **Key Learning**: How to test sophisticated chat workflows with multiple scenarios 31 | 32 | ## 🧪 Demo.MSTest 33 | **MSTest Framework Integration** 34 | 35 | Shows how to use skUnit with MSTest framework for semantic testing. 36 | 37 | - **Location**: `Demo.MSTest/` 38 | - **Features**: MSTest attributes, TestContext integration, data-driven tests 39 | - **Key Learning**: How to use skUnit with MSTest instead of xUnit (proving framework agnosticism) 40 | 41 | ## Running the Demos 42 | 43 | Each demo is a complete .NET project that you can run independently: 44 | 45 | ```bash 46 | cd Demo.TddRepl 47 | dotnet restore Demo.TddRepl.sln 48 | dotnet build Demo.TddRepl.sln --no-restore 49 | dotnet test Demo.TddRepl.sln --no-build 50 | ``` 51 | 52 | ## Prerequisites 53 | 54 | - .NET 8.0 or higher 55 | - Azure OpenAI API access (configure in user secrets) 56 | - For MCP demos: Additional service API keys as needed 57 | 58 | ## Configuration 59 | 60 | Set up your API keys using user secrets: 61 | 62 | ```bash 63 | dotnet user-secrets set "AzureOpenAI_ApiKey" "your-key" --project Demo.TddRepl 64 | dotnet user-secrets set "AzureOpenAI_Endpoint" "https://your-endpoint.openai.azure.com/" --project Demo.TddRepl 65 | dotnet user-secrets set "AzureOpenAI_Deployment" "your-deployment-name" --project Demo.TddRepl 66 | ``` 67 | 68 | Each demo provides practical, real-world examples of how to use skUnit effectively in different scenarios. 69 | -------------------------------------------------------------------------------- /src/skUnit.Tests/SemanticAssertTests/SemanticAssertTests.cs: -------------------------------------------------------------------------------- 1 | using Azure.AI.OpenAI; 2 | using Microsoft.Extensions.AI; 3 | using Microsoft.Extensions.Configuration; 4 | using skUnit.Exceptions; 5 | using skUnit.Tests.Infrastructure; 6 | using Xunit.Abstractions; 7 | 8 | namespace skUnit.Tests.SemanticAssertTests 9 | { 10 | public class SemanticAssertTests(ITestOutputHelper output) : SemanticTestBase(output) 11 | { 12 | [Theory] 13 | [MemberData(nameof(GetSimilarData))] 14 | public void Similar_True_MustWork(string first, string second) 15 | { 16 | SemanticAssert.Similar(first, second); 17 | } 18 | 19 | [Theory] 20 | [MemberData(nameof(GetNonSimilarData))] 21 | public void Similar_False_MustWork(string first, string second) 22 | { 23 | var exception = Assert.Throws(() => SemanticAssert.Similar(first, second)); 24 | Output.WriteLine($""" 25 | [Explanation] 26 | {exception.Message} 27 | """); 28 | } 29 | 30 | [Theory] 31 | [MemberData(nameof(GetNonSimilarData))] 32 | public void NotSimilar_True_MustWork(string first, string second) 33 | { 34 | SemanticAssert.NotSimilar(first, second); 35 | } 36 | 37 | [Theory] 38 | [MemberData(nameof(GetSimilarData))] 39 | public void NotSimilar_False_MustWork(string first, string second) 40 | { 41 | var exception = Assert.Throws(() => SemanticAssert.NotSimilar(first, second)); 42 | Output.WriteLine($""" 43 | [Explanation] 44 | {exception.Message} 45 | """); 46 | } 47 | 48 | public static IEnumerable GetNonSimilarData() 49 | { 50 | yield return new object[] 51 | { 52 | "This car is red", 53 | "The car is blue" 54 | }; 55 | yield return new object[] 56 | { 57 | "This bicycle is red", 58 | "The car is red" 59 | }; 60 | } 61 | 62 | public static IEnumerable GetSimilarData() 63 | { 64 | yield return new object[] 65 | { 66 | "This car is red", 67 | "The car is red" 68 | }; 69 | yield return new object[] 70 | { 71 | "This automobile is red", 72 | "The car is red" 73 | }; 74 | } 75 | } 76 | } -------------------------------------------------------------------------------- /demos/Demo.MSTest/README.md: -------------------------------------------------------------------------------- 1 | # Demo.MSTest - MSTest Integration Example 2 | 3 | This demo shows how to use skUnit with MSTest framework for semantic testing of chat applications. 4 | 5 | ## Key Features Demonstrated 6 | 7 | - **MSTest Attributes**: Using `[TestClass]`, `[TestMethod]`, `[DataTestMethod]`, and `[ClassInitialize]` 8 | - **TestContext Integration**: Proper logging integration using `TestContext.WriteLine` 9 | - **Data-Driven Tests**: Using `[DataRow]` for parameterized tests 10 | - **Scenario Loading**: Loading test scenarios from embedded markdown files 11 | - **Function-Style Testing**: Alternative approach using `getAnswerFunc` delegate 12 | 13 | ## MSTest vs xUnit Key Differences 14 | 15 | ### Test Class Setup 16 | ```csharp 17 | // MSTest 18 | [TestClass] 19 | public class ChatScenarioTests 20 | { 21 | public TestContext TestContext { get; set; } = null!; 22 | 23 | [ClassInitialize] 24 | public static void ClassInitialize(TestContext context) { } 25 | 26 | [TestMethod] 27 | public async Task MyTest() { } 28 | } 29 | 30 | // xUnit equivalent 31 | public class ChatScenarioTests 32 | { 33 | public ChatScenarioTests(ITestOutputHelper output) { } 34 | 35 | [Fact] 36 | public async Task MyTest() { } 37 | } 38 | ``` 39 | 40 | ### Logging Integration 41 | ```csharp 42 | // MSTest 43 | var scenarioAssert = new ScenarioAssert(_chatClient, TestContext.WriteLine); 44 | 45 | // xUnit 46 | var scenarioAssert = new ScenarioAssert(_chatClient, output.WriteLine); 47 | ``` 48 | 49 | ### Data-Driven Tests 50 | ```csharp 51 | // MSTest 52 | [DataTestMethod] 53 | [DataRow("Scenario1")] 54 | [DataRow("Scenario2")] 55 | public async Task TestScenarios(string scenarioName) { } 56 | 57 | // xUnit 58 | [Theory] 59 | [InlineData("Scenario1")] 60 | [InlineData("Scenario2")] 61 | public async Task TestScenarios(string scenarioName) { } 62 | ``` 63 | 64 | ## Running the Demo 65 | 66 | 1. **Configure Azure OpenAI secrets**: 67 | ```bash 68 | cd Demo.MSTest 69 | dotnet user-secrets set "AzureOpenAI_ApiKey" "your-key" 70 | dotnet user-secrets set "AzureOpenAI_Endpoint" "https://your-endpoint.openai.azure.com/" 71 | dotnet user-secrets set "AzureOpenAI_Deployment" "your-deployment-name" 72 | ``` 73 | 74 | 2. **Build and test**: 75 | ```bash 76 | dotnet restore Demo.MSTest.sln 77 | dotnet build Demo.MSTest.sln --no-restore 78 | dotnet test Demo.MSTest.sln --no-build 79 | ``` 80 | 81 | ## Scenarios Included 82 | 83 | - **SimpleGreeting.md**: Basic semantic condition testing 84 | - **GetCurrentTimeChat.md**: Multi-turn conversation validation 85 | - **JsonUserInfo.md**: JSON structure and content validation 86 | 87 | This demo proves that skUnit works seamlessly with MSTest - the core library is completely test-framework agnostic! -------------------------------------------------------------------------------- /src/skUnit.Tests/Infrastructure/SemanticTestBase.cs: -------------------------------------------------------------------------------- 1 | using System.Reflection; 2 | using Azure.AI.OpenAI; 3 | using Markdig.Helpers; 4 | using Microsoft.Extensions.AI; 5 | using Microsoft.Extensions.Configuration; 6 | using Microsoft.Extensions.DependencyInjection; 7 | using Microsoft.Extensions.Logging; 8 | using skUnit.Scenarios; 9 | using skUnit.Scenarios.Parsers; 10 | using Xunit.Abstractions; 11 | 12 | namespace skUnit.Tests.Infrastructure; 13 | 14 | public class SemanticTestBase 15 | { 16 | protected readonly string ApiKey; 17 | protected readonly string Endpoint; 18 | protected readonly string DeploymentName; 19 | protected IChatClient SystemUnderTestClient { get; set; } 20 | protected ChatScenarioRunner ScenarioRunner { get; set; } 21 | protected SemanticAssert SemanticAssert { get; set; } 22 | 23 | protected ITestOutputHelper Output { get; set; } 24 | protected IConfiguration Configuration { get; set; } 25 | 26 | public SemanticTestBase(ITestOutputHelper output) 27 | { 28 | Output = output; 29 | var builder = new ConfigurationBuilder() 30 | .AddUserSecrets() 31 | .AddEnvironmentVariables(); 32 | 33 | Configuration = builder.Build(); 34 | 35 | ApiKey = 36 | Configuration["AzureOpenAI_ApiKey"] ?? 37 | throw new Exception("No ApiKey is provided."); 38 | Endpoint = 39 | Configuration["AzureOpenAI_Endpoint"] ?? 40 | throw new Exception("No Endpoint is provided."); 41 | DeploymentName = 42 | Configuration["AzureOpenAI_Deployment"] ?? 43 | throw new Exception("No Deployment is provided."); 44 | 45 | // Create assertion client for semantic evaluations 46 | var assertionClient = new AzureOpenAIClient( 47 | new Uri(Endpoint), 48 | new System.ClientModel.ApiKeyCredential(ApiKey) 49 | ).GetChatClient(DeploymentName).AsIChatClient(); 50 | 51 | ScenarioRunner = new ChatScenarioRunner(assertionClient, message => Output.WriteLine(message)); 52 | 53 | SemanticAssert = new SemanticAssert(assertionClient); 54 | 55 | // Create system under test client 56 | var openAI = new AzureOpenAIClient( 57 | new Uri(Endpoint), 58 | new System.ClientModel.ApiKeyCredential(ApiKey) 59 | ).GetChatClient(DeploymentName).AsIChatClient(); 60 | 61 | SystemUnderTestClient = new ChatClientBuilder(openAI) 62 | .Build(); 63 | } 64 | 65 | protected async Task> LoadChatScenarioAsync(string scenario) 66 | { 67 | return await ChatScenario.LoadFromResourceAsync($"skUnit.Tests.ScenarioAssertTests.Samples.{scenario}.md", Assembly.GetExecutingAssembly()); 68 | } 69 | } -------------------------------------------------------------------------------- /src/skUnit.Tests/ScenarioAssertTests/ChatClientTests.cs: -------------------------------------------------------------------------------- 1 | using Microsoft.Extensions.AI; 2 | using skUnit.Tests.Infrastructure; 3 | using System.ComponentModel; 4 | using Xunit.Abstractions; 5 | 6 | namespace skUnit.Tests.ScenarioAssertTests 7 | { 8 | public class ChatClientTests : SemanticTestBase 9 | { 10 | public ChatClientTests(ITestOutputHelper output) : base(output) 11 | { 12 | 13 | } 14 | 15 | [Fact] 16 | public async Task EiffelTallChat_MustWork() 17 | { 18 | var scenarios = await LoadChatScenarioAsync("EiffelTallChat"); 19 | await ScenarioRunner.RunAsync(scenarios, SystemUnderTestClient); 20 | } 21 | 22 | 23 | [Fact] 24 | public async Task FunctionCall_MustWork() 25 | { 26 | var scenarios = await LoadChatScenarioAsync("GetFoodMenuChat"); 27 | await ScenarioRunner.RunAsync(scenarios, SystemUnderTestClient, getAnswerFunc: async history => 28 | { 29 | AIFunction getFoodMenu = AIFunctionFactory.Create(GetFoodMenu); 30 | 31 | var result = await SystemUnderTestClient.GetResponseAsync( 32 | history, 33 | options: new ChatOptions 34 | { 35 | Tools = [getFoodMenu] 36 | }); 37 | 38 | var answer = result; 39 | return answer; 40 | }); 41 | } 42 | 43 | [Fact] 44 | public async Task FunctionCallJson_MustWork() 45 | { 46 | var scenarios = await LoadChatScenarioAsync("GetFoodMenuChatJson"); 47 | 48 | var builder = new ChatClientBuilder(SystemUnderTestClient) 49 | .ConfigureOptions(options => 50 | { 51 | options.Tools ??= []; 52 | options.Tools.Add(AIFunctionFactory.Create(GetFoodMenu)); 53 | }) 54 | .UseFunctionInvocation() 55 | ; 56 | 57 | var chatClient = builder.Build(); 58 | 59 | await ScenarioRunner.RunAsync(scenarios, chatClient); 60 | } 61 | 62 | 63 | [Description("Gets a food menu based on the user's mood")] 64 | private static string GetFoodMenu( 65 | [Description("User's mood based on its chat hsitory.")] 66 | UserMood mood 67 | ) 68 | { 69 | return mood switch 70 | { 71 | UserMood.Happy => "Pizza", 72 | UserMood.Sad => "Ice Cream", 73 | UserMood.Angry => "Hot Dog", 74 | _ => "Nothing" 75 | }; 76 | } 77 | 78 | enum UserMood 79 | { 80 | Happy, 81 | Sad, 82 | Angry 83 | } 84 | } 85 | 86 | 87 | } 88 | -------------------------------------------------------------------------------- /docs/multi-modal-support.md: -------------------------------------------------------------------------------- 1 | # Multi-Modal Content Support 2 | 3 | skUnit now supports multi-modal content in chat scenarios, allowing you to include images alongside text in your test scenarios. This enables testing of AI models that support vision capabilities, such as GPT-4o. 4 | 5 | ## Syntax 6 | 7 | Within any chat role block (`[USER]`, `[AGENT]`, `[SYSTEM]`, `[TOOL]`), you can define multiple content parts using subsections: 8 | 9 | ### Text Content 10 | ```markdown 11 | ### Text 12 | Your text content here... 13 | ``` 14 | 15 | ### Image Content 16 | ```markdown 17 | ### Image 18 | ![Alt text](https://example.com/image.jpg) 19 | ``` 20 | 21 | ## Example 22 | 23 | Here's a complete example of a multi-modal scenario: 24 | 25 | ```markdown 26 | # SCENARIO Multi-modal Image Analysis 27 | 28 | ## [USER] 29 | ### Text 30 | This image explains how skUnit parses the chat scenarios. 31 | ### Image 32 | ![skUnit structure](https://github.com/mehrandvd/skunit/assets/5070766/156b0831-e4f3-4e4b-b1b0-e2ec868efb5f) 33 | ### Text 34 | How many scenarios are there in the picture? 35 | 36 | ## [ASSISTANT] 37 | There are 2 scenarios in the picture 38 | 39 | ### CHECK SemanticSimilar 40 | There are 2 scenarios in the picture 41 | ``` 42 | 43 | ## Backward Compatibility 44 | 45 | The new multi-modal support is fully backward compatible. Existing scenarios that don't use subsections will continue to work exactly as before: 46 | 47 | ```markdown 48 | # SCENARIO Traditional Text-Only 49 | 50 | ## [USER] 51 | Just plain text without subsections 52 | 53 | ## [ASSISTANT] 54 | Plain text response 55 | ``` 56 | 57 | ## Technical Details 58 | 59 | - **Text Content**: Handled by `TextContentPart` class 60 | - **Image Content**: Handled by `ImageContentPart` class, supporting any web-accessible image URL 61 | - **AI Integration**: Content parts are automatically converted to `Microsoft.Extensions.AI` content types (`TextContent`, `UriContent`) 62 | - **Extensibility**: The `ChatContentPart` base class allows for future content types (audio, video, etc.) 63 | 64 | ## Usage in Tests 65 | 66 | When using multi-modal scenarios in your tests, the content is automatically converted to the appropriate format for the underlying AI service: 67 | 68 | ```csharp 69 | var scenarios = ChatScenario.LoadFromText(scenarioText); 70 | var chatItem = scenarios.First().ChatItems.First(); 71 | 72 | // Access individual content parts 73 | var textParts = chatItem.ContentParts.OfType(); 74 | var imageParts = chatItem.ContentParts.OfType(); 75 | 76 | // Convert to AI framework format 77 | var chatMessage = chatItem.ToChatMessage(); // Returns Microsoft.Extensions.AI.ChatMessage 78 | ``` 79 | 80 | ## Future Extensions 81 | 82 | The infrastructure is designed to support additional content types in the future: 83 | - Audio content (`### Audio`) 84 | - Video content (`### Video`) 85 | - File attachments (`### File`) 86 | - And more... 87 | 88 | Each new content type can be added by creating a new `ChatContentPart` subclass and updating the parser accordingly. -------------------------------------------------------------------------------- /demos/Demo.MSTest/ChatScenarioTests.cs: -------------------------------------------------------------------------------- 1 | using Azure.AI.OpenAI; 2 | using Microsoft.Extensions.AI; 3 | using Microsoft.Extensions.Configuration; 4 | using skUnit; 5 | using skUnit.Scenarios; 6 | 7 | namespace Demo.MSTest; 8 | 9 | [TestClass] 10 | public class ChatScenarioTests 11 | { 12 | private static IChatClient _chatClient = null!; 13 | private static ScenarioAssert ScenarioAssert { get; set; } = null!; 14 | 15 | public TestContext TestContext { get; set; } = null!; 16 | 17 | [ClassInitialize] 18 | public static void ClassInitialize(TestContext context) 19 | { 20 | var configuration = new ConfigurationBuilder() 21 | .AddUserSecrets() 22 | .Build(); 23 | 24 | var apiKey = configuration["AzureOpenAI_ApiKey"] 25 | ?? throw new InvalidOperationException("AzureOpenAI_ApiKey not found in user secrets"); 26 | var endpoint = configuration["AzureOpenAI_Endpoint"] 27 | ?? throw new InvalidOperationException("AzureOpenAI_Endpoint not found in user secrets"); 28 | var deployment = configuration["AzureOpenAI_Deployment"] 29 | ?? throw new InvalidOperationException("AzureOpenAI_Deployment not found in user secrets"); 30 | 31 | _chatClient = new AzureOpenAIClient(new Uri(endpoint), new System.ClientModel.ApiKeyCredential(apiKey)) 32 | .GetChatClient(deployment) 33 | .AsIChatClient(); 34 | 35 | // This uses Console.WriteLine for class-level initialization 36 | ScenarioAssert = new ScenarioAssert(_chatClient, context.WriteLine); 37 | } 38 | 39 | [TestMethod] 40 | public async Task SimpleGreeting_ShouldPass() 41 | { 42 | // Create per-test instance with TestContext logging 43 | 44 | var scenarios = await ChatScenario.LoadFromResourceAsync( 45 | "Demo.MSTest.Scenarios.SimpleGreeting.md", 46 | typeof(ChatScenarioTests).Assembly); 47 | 48 | await ScenarioAssert.PassAsync(scenarios, _chatClient); 49 | } 50 | 51 | [TestMethod] 52 | public async Task GetCurrentTimeChat_ShouldPass() 53 | { 54 | var scenarios = await ChatScenario.LoadFromResourceAsync( 55 | "Demo.MSTest.Scenarios.GetCurrentTimeChat.md", 56 | typeof(ChatScenarioTests).Assembly); 57 | 58 | await ScenarioAssert.PassAsync(scenarios, _chatClient); 59 | } 60 | 61 | [TestMethod] 62 | public async Task JsonUserInfo_ShouldPass() 63 | { 64 | var scenarios = await ChatScenario.LoadFromResourceAsync( 65 | "Demo.MSTest.Scenarios.JsonUserInfo.md", 66 | typeof(ChatScenarioTests).Assembly); 67 | 68 | await ScenarioAssert.PassAsync(scenarios, _chatClient); 69 | } 70 | 71 | [DataTestMethod] 72 | [DataRow("SimpleGreeting")] 73 | [DataRow("GetCurrentTimeChat")] 74 | [DataRow("JsonUserInfo")] 75 | public async Task ScenarioMatrix_ShouldPass(string scenarioName) 76 | { 77 | var scenarios = await ChatScenario.LoadFromResourceAsync( 78 | $"Demo.MSTest.Scenarios.{scenarioName}.md", 79 | typeof(ChatScenarioTests).Assembly); 80 | 81 | await ScenarioAssert.PassAsync(scenarios, _chatClient); 82 | } 83 | } -------------------------------------------------------------------------------- /src/skUnit.Tests/AssertionTests/NewFeaturesDemoTests.cs: -------------------------------------------------------------------------------- 1 | using Microsoft.Extensions.AI; 2 | using skUnit.Scenarios; 3 | using skUnit.Scenarios.Parsers; 4 | using Xunit; 5 | 6 | namespace skUnit.Tests.AssertionTests 7 | { 8 | public class NewFeaturesDemoTests 9 | { 10 | [Fact] 11 | public void ChatScenarioParser_CanParseExampleScenario() 12 | { 13 | // Arrange 14 | const string scenarioText = """ 15 | # SCENARIO Simple Greeting Test 16 | 17 | ## [USER] 18 | Hello, how are you today? 19 | 20 | ## [ASSISTANT] 21 | Hello! I'm doing well, thank you for asking. How can I help you today? 22 | 23 | ### ASSERT SemanticCondition 24 | The response is a polite greeting 25 | 26 | ### ASSERT ContainsAll 27 | well, help 28 | 29 | ### ASSERT SemanticCondition 30 | It mentions a helpful tone 31 | """; 32 | 33 | var parser = new ChatScenarioParser(); 34 | 35 | // Act 36 | var scenarios = parser.Parse(scenarioText); 37 | 38 | // Assert 39 | Assert.Single(scenarios); 40 | Assert.Equal("Simple Greeting Test", scenarios[0].Description); 41 | 42 | var assistantItem = scenarios[0].ChatItems.First(x => x.Role == ChatRole.Assistant); 43 | Assert.Equal(3, assistantItem.Assertions.Count); 44 | 45 | // Verify different assertion types work 46 | Assert.Equal("Condition", assistantItem.Assertions[0].AssertionType); 47 | Assert.Equal("ContainsAll", assistantItem.Assertions[1].AssertionType); 48 | Assert.Equal("Condition", assistantItem.Assertions[2].AssertionType); 49 | } 50 | 51 | [Fact] 52 | public void ChatScenarioRunner_ConstructorWorksAsExpected() 53 | { 54 | // Arrange 55 | var mockChatClient = new TestChatClient(); 56 | 57 | // Act & Assert - Should not throw 58 | var runner = new ChatScenarioRunner(mockChatClient); 59 | Assert.NotNull(runner); 60 | } 61 | 62 | private class TestChatClient : IChatClient 63 | { 64 | public ChatClientMetadata Metadata => new("Test"); 65 | public Task GetResponseAsync(IEnumerable chatMessages, ChatOptions? options = null, CancellationToken cancellationToken = default) 66 | => Task.FromResult(new ChatResponse(new ChatMessage(ChatRole.Assistant, "Test response"))); 67 | public async IAsyncEnumerable GetStreamingResponseAsync(IEnumerable chatMessages, ChatOptions? options = null, [System.Runtime.CompilerServices.EnumeratorCancellation] CancellationToken cancellationToken = default) 68 | { 69 | await Task.Delay(1, cancellationToken); 70 | yield break; 71 | } 72 | public TService? GetService(object? key = null) where TService : class => null; 73 | public object? GetService(Type serviceType, object? key = null) => null; 74 | public void Dispose() { } 75 | } 76 | } 77 | } -------------------------------------------------------------------------------- /demos/Demo.TddMcp/TimeServerMcpTests.cs: -------------------------------------------------------------------------------- 1 | using Azure.AI.OpenAI; 2 | using Microsoft.Extensions.AI; 3 | using Microsoft.Extensions.Configuration; 4 | using ModelContextProtocol.Client; 5 | using ModelContextProtocol.Protocol.Transport; 6 | using skUnit; 7 | using skUnit.Scenarios; 8 | using skUnit.Scenarios.Parsers; 9 | using Xunit.Abstractions; 10 | 11 | namespace Demo.TddMcp 12 | { 13 | public class TimeServerMcpTests 14 | { 15 | ChatScenarioRunner ScenarioRunner { get; set; } 16 | IChatClient baseChatClient { get; set; } 17 | IConfiguration Configuration { get; set; } 18 | public TimeServerMcpTests(ITestOutputHelper output) 19 | { 20 | Configuration = new ConfigurationBuilder() 21 | .AddUserSecrets() 22 | .Build(); 23 | 24 | var apiKey = Configuration["AzureOpenAI_ApiKey"] ?? throw new Exception("No ApiKey is provided."); 25 | var endpoint = Configuration["AzureOpenAI_Endpoint"] ?? throw new Exception("No Endpoint is provided."); 26 | var deploymentName = Configuration["AzureOpenAI_Deployment"] ?? throw new Exception("No Deployment is provided."); 27 | 28 | var assertionClient = new AzureOpenAIClient( 29 | new Uri(endpoint), 30 | new System.ClientModel.ApiKeyCredential(apiKey) 31 | ).GetChatClient(deploymentName).AsIChatClient(); 32 | 33 | baseChatClient = new AzureOpenAIClient( 34 | new Uri(endpoint), 35 | new System.ClientModel.ApiKeyCredential(apiKey) 36 | ).GetChatClient(deploymentName).AsIChatClient(); 37 | 38 | ScenarioRunner = new ChatScenarioRunner(assertionClient, output.WriteLine); 39 | } 40 | 41 | [Fact] 42 | public async Task Tools_MustWork() 43 | { 44 | var smitheryKey = Configuration["Smithery_Key"] ?? throw new Exception("No Smithery Key is provided."); 45 | 46 | var clientTransport = new StdioClientTransport(new StdioClientTransportOptions 47 | { 48 | Name = "Time MCP Server", 49 | Command = "cmd", 50 | Arguments = [ 51 | "/c", 52 | "npx", 53 | "-y", 54 | "@smithery/cli@latest", 55 | "run", 56 | "@yokingma/time-mcp", 57 | "--key", 58 | smitheryKey 59 | ], 60 | }); 61 | 62 | await using var mcp = await McpClientFactory.CreateAsync(clientTransport); 63 | 64 | var tools = await mcp.ListToolsAsync(); 65 | 66 | var builder = new ChatClientBuilder(baseChatClient) 67 | .ConfigureOptions(options => 68 | { 69 | options.Tools = tools.ToArray(); 70 | }) 71 | .UseFunctionInvocation(); 72 | 73 | var systemUnderTestClient = builder.Build(); 74 | 75 | var scenarioText = await File.ReadAllTextAsync("TestScenario.md"); 76 | var scenario = ChatScenario.LoadFromText(scenarioText); 77 | await ScenarioRunner.RunAsync(scenario, systemUnderTestClient); 78 | } 79 | } 80 | } 81 | -------------------------------------------------------------------------------- /demos/Demo.TddRepl/Demo.TddRepl.Test/BrainTests.cs: -------------------------------------------------------------------------------- 1 | using skUnit.Scenarios; 2 | using skUnit; 3 | using Xunit.Abstractions; 4 | using Microsoft.VisualStudio.TestPlatform.Utilities; 5 | 6 | namespace Demo.TddRepl.Test 7 | { 8 | public class BrainTests 9 | { 10 | protected ScenarioAssert ScenarioAssert { get; set; } 11 | public BrainTests(ITestOutputHelper output) 12 | { 13 | var endPoint = Environment.GetEnvironmentVariable("openai-endpoint") ?? throw new InvalidOperationException("No key provided."); 14 | var apiKey = Environment.GetEnvironmentVariable("openai-api-key") ?? throw new InvalidOperationException("No key provided."); 15 | var deploymentName = Environment.GetEnvironmentVariable("openai-deployment-name") ?? throw new InvalidOperationException("No key provided."); 16 | 17 | ScenarioAssert = new ScenarioAssert(deploymentName, endPoint, apiKey, output.WriteLine); 18 | } 19 | 20 | [Fact] 21 | public async Task Greeting() 22 | { 23 | var brain = new Brain(); 24 | 25 | var scenarios = await ChatScenario.LoadFromResourceAsync(@"Demo.TddRepl.Test.Scenarios.01-Greeting.md", GetType().Assembly); 26 | await ScenarioAssert.PassAsync(scenarios, 27 | getAnswerFunc: async history => 28 | { 29 | var result = await brain.GetChatAnswerAsync(history); 30 | 31 | return result?.ToString() ?? string.Empty; 32 | }); 33 | } 34 | 35 | [Fact] 36 | public async Task WhoIsMehran_Normal() 37 | { 38 | var brain = new Brain(); 39 | 40 | var scenarios = await ChatScenario.LoadFromResourceAsync(@"Demo.TddRepl.Test.Scenarios.02-WhoIsMehran-Normal.md", GetType().Assembly); 41 | await ScenarioAssert.PassAsync(scenarios, 42 | getAnswerFunc: async history => 43 | { 44 | var result = await brain.GetChatAnswerAsync(history); 45 | 46 | return result?.ToString() ?? string.Empty; 47 | }); 48 | } 49 | 50 | [Fact] 51 | public async Task WhoIsMehran_Angry() 52 | { 53 | var brain = new Brain(); 54 | 55 | var scenarios = await ChatScenario.LoadFromResourceAsync(@"Demo.TddRepl.Test.Scenarios.03-WhoIsMehran-Angry.md", GetType().Assembly); 56 | await ScenarioAssert.PassAsync(scenarios, 57 | getAnswerFunc: async history => 58 | { 59 | var result = await brain.GetChatAnswerAsync(history); 60 | 61 | return result?.ToString() ?? string.Empty; 62 | }); 63 | } 64 | 65 | [Fact] 66 | public async Task WhoIsMehran_AngryNormal() 67 | { 68 | var brain = new Brain(); 69 | 70 | var scenarios = await ChatScenario.LoadFromResourceAsync(@"Demo.TddRepl.Test.Scenarios.04-WhoIsMehran-AngryNormal.md", GetType().Assembly); 71 | await ScenarioAssert.PassAsync(scenarios, 72 | getAnswerFunc: async history => 73 | { 74 | var result = await brain.GetChatAnswerAsync(history); 75 | 76 | return result?.ToString() ?? string.Empty; 77 | }); 78 | } 79 | } 80 | } -------------------------------------------------------------------------------- /src/skUnit/Asserts/ScenarioAssert_Initialize.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections.Generic; 3 | using System.Linq; 4 | using System.Text; 5 | using System.Threading.Tasks; 6 | using Microsoft.Extensions.AI; 7 | using Microsoft.Extensions.Logging; 8 | using Microsoft.Extensions.Logging.Abstractions; 9 | using SemanticValidation; 10 | using skUnit.Exceptions; 11 | using skUnit.Scenarios; 12 | using skUnit.Scenarios.Parsers.Assertions; 13 | 14 | namespace skUnit 15 | { 16 | /// 17 | /// This class is for testing skUnit scenarios semantically. It contains various methods 18 | /// that you can test kernels and functions with scenarios. Scenarios are some markdown files with a specific format. 19 | /// 20 | [Obsolete("Use ChatScenarioRunner instead. This class will be removed in a future version.", false)] 21 | public partial class ScenarioAssert 22 | { 23 | private readonly ChatScenarioRunner _runner; 24 | 25 | /// 26 | /// This class needs a ChatClient to work. 27 | /// Pass your pre-configured ChatClient and ILogger to this constructor. 28 | /// 29 | public ScenarioAssert(IChatClient chatClient, ILogger? logger = null) 30 | { 31 | _runner = new ChatScenarioRunner(chatClient, logger != null ? new DelegateLoggerAdapter(msg => logger.LogInformation("{Message}", msg)) : null); 32 | } 33 | 34 | /// 35 | /// This class needs a ChatClient to work. 36 | /// Pass your pre-configured ChatClient to this constructor. 37 | /// 38 | [Obsolete("Use constructor with ILogger parameter for better logging integration. This constructor will be deprecated in a future version.")] 39 | public ScenarioAssert(IChatClient chatClient, Action? onLog) 40 | { 41 | _runner = new ChatScenarioRunner(chatClient, onLog); 42 | } 43 | 44 | /// 45 | /// Gets the internal ChatScenarioRunner for advanced usage. 46 | /// 47 | public ChatScenarioRunner Runner => _runner; 48 | } 49 | 50 | /// 51 | /// Internal adapter that wraps an Action<string> delegate to provide ILogger functionality 52 | /// for backward compatibility with the legacy onLog parameter for ScenarioAssert. 53 | /// 54 | public class DelegateLoggerAdapter : ILogger 55 | { 56 | private readonly Action _logAction; 57 | 58 | public DelegateLoggerAdapter(Action logAction) 59 | { 60 | _logAction = logAction ?? throw new ArgumentNullException(nameof(logAction)); 61 | } 62 | 63 | public IDisposable? BeginScope(TState state) where TState : notnull 64 | { 65 | // No scope support needed for this adapter 66 | return null; 67 | } 68 | 69 | public bool IsEnabled(LogLevel logLevel) 70 | { 71 | // Always enabled for all log levels since the original Action doesn't have level filtering 72 | return true; 73 | } 74 | 75 | public void Log(LogLevel logLevel, EventId eventId, TState state, Exception? exception, Func formatter) 76 | { 77 | if (formatter == null) return; 78 | 79 | var message = formatter(state, exception); 80 | _logAction(message); 81 | } 82 | } 83 | } 84 | -------------------------------------------------------------------------------- /src/skUnit.Tests/ScenarioAssertTests/HallucinationChatClientTests.cs: -------------------------------------------------------------------------------- 1 | using Microsoft.Extensions.AI; 2 | using skUnit.Tests.Infrastructure; 3 | using System.ComponentModel; 4 | using Xunit.Abstractions; 5 | 6 | namespace skUnit.Tests.ScenarioAssertTests 7 | { 8 | public class HallucinationChatClientTests : SemanticTestBase 9 | { 10 | 11 | public HallucinationChatClientTests(ITestOutputHelper output) : base(output) 12 | { 13 | 14 | } 15 | 16 | [Fact] 17 | public async Task EiffelTallChat_MustWork() 18 | { 19 | var scenarios = await LoadChatScenarioAsync("EiffelTallChat"); 20 | await ScenarioRunner.RunAsync( 21 | scenarios, 22 | SystemUnderTestClient, 23 | options: new ScenarioRunOptions 24 | { 25 | TotalRuns = 3, 26 | MinSuccessRate = .6 27 | } 28 | ); 29 | } 30 | 31 | 32 | [Fact] 33 | public async Task FunctionCall_MustWork() 34 | { 35 | var scenarios = await LoadChatScenarioAsync("GetFoodMenuChat"); 36 | await ScenarioRunner.RunAsync( 37 | scenarios, 38 | SystemUnderTestClient, 39 | options: new ScenarioRunOptions 40 | { 41 | TotalRuns = 3, 42 | MinSuccessRate = .6 43 | }, 44 | getAnswerFunc: async history => 45 | { 46 | AIFunction getFoodMenu = AIFunctionFactory.Create(GetFoodMenu); 47 | 48 | var result = await SystemUnderTestClient.GetResponseAsync( 49 | history, 50 | options: new ChatOptions 51 | { 52 | Tools = [getFoodMenu] 53 | }); 54 | 55 | var answer = result; 56 | return answer; 57 | }); 58 | } 59 | 60 | [Fact] 61 | public async Task FunctionCallJson_MustWork() 62 | { 63 | var scenarios = await LoadChatScenarioAsync("GetFoodMenuChatJson"); 64 | 65 | var builder = new ChatClientBuilder(SystemUnderTestClient) 66 | .ConfigureOptions(options => 67 | { 68 | options.Tools ??= []; 69 | options.Tools.Add(AIFunctionFactory.Create(GetFoodMenu)); 70 | }) 71 | .UseFunctionInvocation() 72 | ; 73 | 74 | var chatClient = builder.Build(); 75 | 76 | await ScenarioRunner.RunAsync(scenarios, chatClient, 77 | options: new ScenarioRunOptions 78 | { 79 | TotalRuns = 3, 80 | MinSuccessRate = .6 81 | }); 82 | } 83 | 84 | 85 | [Description("Gets a food menu based on the user's mood")] 86 | private static string GetFoodMenu( 87 | [Description("User's mood based on its chat hsitory.")] 88 | UserMood mood 89 | ) 90 | { 91 | return mood switch 92 | { 93 | UserMood.Happy => "Pizza", 94 | UserMood.Sad => "Ice Cream", 95 | UserMood.Angry => "Hot Dog", 96 | _ => "Nothing" 97 | }; 98 | } 99 | 100 | enum UserMood 101 | { 102 | Happy, 103 | Sad, 104 | Angry 105 | } 106 | } 107 | 108 | 109 | } 110 | -------------------------------------------------------------------------------- /src/skUnit/Scenarios/ChatScenario.cs: -------------------------------------------------------------------------------- 1 | using Markdig.Helpers; 2 | using Microsoft.Extensions.AI; 3 | using skUnit.Scenarios.Parsers; 4 | using skUnit.Scenarios.Parsers.Assertions; 5 | 6 | namespace skUnit.Scenarios; 7 | 8 | /// 9 | /// A ChatScenario is a scenario to test a ChatCompletionService, Kernel, Function or Plugin. 10 | /// It contains the required inputs to call a InvokeAsync like: ChatItems. 11 | /// Also it contains the expected output for each level of chat by: Assertions 12 | /// 13 | public class ChatScenario : Scenario 14 | { 15 | public string? Description { get; set; } 16 | public required string RawText { get; set; } 17 | 18 | /// 19 | /// The ChatItems that should be applied to chat history one by one and 20 | /// check for assertions for each level. 21 | /// 22 | public List ChatItems { get; set; } = new(); 23 | 24 | } 25 | 26 | public class ChatItem 27 | { 28 | public ChatItem(ChatRole role, string content) 29 | { 30 | Role = role; 31 | // Maintain backward compatibility: convert string content to TextContent 32 | Contents = new List { new TextContent(content) }; 33 | } 34 | 35 | public ChatItem(ChatRole role, List contents) 36 | { 37 | Role = role; 38 | Contents = contents; 39 | } 40 | 41 | public ChatRole Role { get; set; } 42 | 43 | /// 44 | /// The AI content parts for this chat item (text, images, etc.) 45 | /// 46 | public List Contents { get; set; } = new(); 47 | 48 | /// 49 | /// Backward compatibility property that returns combined text content 50 | /// 51 | public string Content 52 | { 53 | get => string.Join("\n", Contents.OfType().Select(t => t.Text)); 54 | set => Contents = new List { new TextContent(value) }; 55 | } 56 | 57 | /// 58 | /// All the assertions that should be checked after the result of InvokeAsync is ready for the user input and history so far. 59 | /// 60 | public List Assertions { get; set; } = new(); 61 | 62 | /// 63 | /// The function calls that should be asserted too. 64 | /// 65 | public List FunctionCalls { get; set; } = new(); 66 | 67 | /// 68 | /// Convert to Microsoft.Extensions.AI ChatMessage 69 | /// 70 | /// ChatMessage with all content parts 71 | public ChatMessage ToChatMessage() 72 | { 73 | return new ChatMessage(Role, Contents); 74 | } 75 | 76 | public override string ToString() 77 | { 78 | return $"{Role}: {Content}"; 79 | } 80 | } 81 | 82 | public class FunctionCall 83 | { 84 | public required string PluginName { get; set; } 85 | public required string FunctionName { get; set; } 86 | public List Arguments { get; set; } = new(); 87 | public string? ArgumentsText { get; set; } 88 | 89 | /// 90 | /// All the assertions that should be checked after the result of InvokeAsync is ready. 91 | /// 92 | public List Assertions { get; set; } = new(); 93 | 94 | public override string ToString() => $"{PluginName}{FunctionName}({string.Join(",", Arguments.Select(a => a.Name))})"; 95 | } 96 | 97 | public class FunctionCallArgument 98 | { 99 | public required string Name { get; set; } 100 | public string? LiteralValue { get; set; } 101 | public string? InputVariable { get; set; } 102 | 103 | public override string ToString() => 104 | InputVariable != null ? $"{Name}=${InputVariable}" 105 | : $"{Name}=\"{LiteralValue}\""; 106 | } -------------------------------------------------------------------------------- /src/skUnit.Tests/AssertionTests/ChatScenarioRunnerTests.cs: -------------------------------------------------------------------------------- 1 | using Microsoft.Extensions.AI; 2 | using Microsoft.Extensions.Logging; 3 | using Xunit; 4 | using skUnit.Scenarios; 5 | 6 | namespace skUnit.Tests.AssertionTests 7 | { 8 | public class ChatScenarioRunnerTests 9 | { 10 | [Fact] 11 | public void ChatScenarioRunner_Constructor_WithLogger_Works() 12 | { 13 | // Arrange 14 | var mockChatClient = CreateMockChatClient(); 15 | var logger = new TestLogger(); 16 | 17 | // Act 18 | var runner = new ChatScenarioRunner(mockChatClient, logger); 19 | 20 | // Assert 21 | Assert.NotNull(runner); 22 | } 23 | 24 | [Fact] 25 | public void ChatScenarioRunner_Constructor_WithActionLog_Works() 26 | { 27 | // Arrange 28 | var mockChatClient = CreateMockChatClient(); 29 | var logs = new List(); 30 | Action onLog = message => logs.Add(message); 31 | 32 | // Act 33 | var runner = new ChatScenarioRunner(mockChatClient, onLog); 34 | 35 | // Assert 36 | Assert.NotNull(runner); 37 | } 38 | 39 | [Fact] 40 | public void ChatScenarioRunner_Constructor_WithNullLogger_Works() 41 | { 42 | // Arrange 43 | var mockChatClient = CreateMockChatClient(); 44 | 45 | // Act 46 | var runner = new ChatScenarioRunner(mockChatClient, (ILogger?)null); 47 | 48 | // Assert 49 | Assert.NotNull(runner); 50 | } 51 | 52 | [Fact] 53 | public async Task ChatScenarioRunner_RunAsync_ThrowsWhenBothClientAndFuncAreNull() 54 | { 55 | // Arrange 56 | var mockChatClient = CreateMockChatClient(); 57 | var runner = new ChatScenarioRunner(mockChatClient); 58 | var scenario = new ChatScenario { RawText = "" }; 59 | 60 | // Act & Assert 61 | var exception = await Assert.ThrowsAsync( 62 | () => runner.RunAsync(scenario, (IChatClient?)null, null)); 63 | 64 | Assert.Contains("Both chatClient and getAnswerFunc can not be null", exception.Message); 65 | } 66 | 67 | private static IChatClient CreateMockChatClient() 68 | { 69 | return new TestChatClient(); 70 | } 71 | 72 | private class TestChatClient : IChatClient 73 | { 74 | public ChatClientMetadata Metadata => new("Test"); 75 | 76 | public Task GetResponseAsync(IEnumerable chatMessages, ChatOptions? options = null, CancellationToken cancellationToken = default) 77 | { 78 | return Task.FromResult(new ChatResponse(new ChatMessage(ChatRole.Assistant, "Test response"))); 79 | } 80 | 81 | public async IAsyncEnumerable GetStreamingResponseAsync(IEnumerable chatMessages, ChatOptions? options = null, [System.Runtime.CompilerServices.EnumeratorCancellation] CancellationToken cancellationToken = default) 82 | { 83 | await Task.Delay(1, cancellationToken); 84 | yield break; 85 | } 86 | 87 | public TService? GetService(object? key = null) where TService : class => null; 88 | public object? GetService(Type serviceType, object? key = null) => null; 89 | public void Dispose() { } 90 | } 91 | 92 | private class TestLogger : ILogger 93 | { 94 | public IDisposable? BeginScope(TState state) where TState : notnull => null; 95 | public bool IsEnabled(LogLevel logLevel) => true; 96 | public void Log(LogLevel logLevel, EventId eventId, TState state, Exception? exception, Func formatter) { } 97 | } 98 | } 99 | } -------------------------------------------------------------------------------- /src/skUnit.Tests/AssertionTests/JsonCheckAssertionTests.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections.Generic; 3 | using System.Linq; 4 | using System.Text; 5 | using System.Threading.Tasks; 6 | using skUnit.Exceptions; 7 | using skUnit.Scenarios.Parsers.Assertions; 8 | 9 | namespace skUnit.Tests.AssertionTests 10 | { 11 | public class JsonCheckAssertionTests 12 | { 13 | [Fact] 14 | public async Task JasonCheck_MustWork() 15 | { 16 | var assertion = new JsonCheckAssertion(); 17 | 18 | assertion.SetJsonAssertText(""" 19 | { 20 | "name": ["EQUAL", "Mehran"], 21 | "address": ["Contain", "Tehran, Vanak"] 22 | } 23 | """); 24 | 25 | Assert.NotNull(assertion.JsonCheck); 26 | 27 | await assertion.Assert(null, """ 28 | { 29 | "name": "Mehran", 30 | "address": "The address is in Vanak area of Tehran" 31 | } 32 | """); 33 | } 34 | 35 | [Fact] 36 | public async Task JasonCheck_BackQuote_MustWork() 37 | { 38 | var assertion = new JsonCheckAssertion(); 39 | 40 | assertion.SetJsonAssertText(""" 41 | ```json 42 | { 43 | "name": ["EQUAL", "Mehran"], 44 | "address": ["Contain", "Tehran, Vanak"] 45 | } 46 | ``` 47 | """); 48 | 49 | Assert.NotNull(assertion.JsonCheck); 50 | 51 | await assertion.Assert(null, """ 52 | { 53 | "name": "Mehran", 54 | "address": "The address is in Vanak area of Tehran" 55 | } 56 | """); 57 | } 58 | 59 | [Fact] 60 | public async Task JasonCheck_Contains_Fail_MustWork() 61 | { 62 | var assertion = new JsonCheckAssertion(); 63 | 64 | assertion.SetJsonAssertText(""" 65 | { 66 | "name": ["EQUAL", "Mehran"], 67 | "address": ["Contain", "Tehran, Gisha"] 68 | } 69 | """); 70 | 71 | Assert.NotNull(assertion.JsonCheck); 72 | 73 | await Assert.ThrowsAsync(() => assertion.Assert(null, """ 74 | { 75 | "name": "Mehran", 76 | "address": "The address is in Vanak area of Tehran" 77 | } 78 | """)); 79 | } 80 | 81 | [Fact] 82 | public async Task JasonCheck_PropertyNotFound_MustWork() 83 | { 84 | var assertion = new JsonCheckAssertion(); 85 | 86 | assertion.SetJsonAssertText(""" 87 | { 88 | "name": ["EQUAL", "Mehran"], 89 | "address": ["Contain", "Tehran, Gisha"] 90 | } 91 | """); 92 | 93 | Assert.NotNull(assertion.JsonCheck); 94 | 95 | await Assert.ThrowsAsync(() => assertion.Assert(null, """ 96 | { 97 | "name": "Mehran", 98 | } 99 | """)); 100 | } 101 | 102 | [Fact] 103 | public async Task JasonCheck_InvalidCheck_Throws() 104 | { 105 | var assertion = new JsonCheckAssertion(); 106 | 107 | assertion.SetJsonAssertText(""" 108 | { 109 | "name": ["EQUAL", "Mehran", "Haha"], 110 | "address": ["Contain", "Tehran, Gisha"] 111 | } 112 | """); 113 | 114 | Assert.NotNull(assertion.JsonCheck); 115 | 116 | await Assert.ThrowsAsync(() => assertion.Assert(null, """ 117 | { 118 | "name": "Mehran", 119 | } 120 | """)); 121 | } 122 | } 123 | } 124 | -------------------------------------------------------------------------------- /src/skUnit.Tests/AssertionTests/ChatScenarioRunnerLoggingBehaviorTests.cs: -------------------------------------------------------------------------------- 1 | using Microsoft.Extensions.AI; 2 | using Microsoft.Extensions.Logging; 3 | using System.Text; 4 | 5 | namespace skUnit.Tests.AssertionTests; 6 | 7 | public class ChatScenarioRunnerLoggingBehaviorTests 8 | { 9 | [Fact] 10 | public void Both_Constructors_Produce_Valid_ChatScenarioRunner() 11 | { 12 | // Arrange 13 | var chatClient = CreateMockChatClient(); 14 | var actionOutput = new StringBuilder(); 15 | 16 | // Create logger factory that captures to string 17 | var loggerFactory = LoggerFactory.Create(builder => 18 | builder.AddProvider(new StringLoggerProvider(new StringBuilder()))); 19 | var logger = loggerFactory.CreateLogger(); 20 | 21 | // Act - Create both instances using different constructors 22 | var actionRunner = new ChatScenarioRunner(chatClient, message => actionOutput.AppendLine(message)); 23 | var loggerRunner = new ChatScenarioRunner(chatClient, logger); 24 | 25 | // Both should work without throwing 26 | Assert.NotNull(actionRunner); 27 | Assert.NotNull(loggerRunner); 28 | 29 | // Since we can't easily test the Log method directly (it's private), 30 | // this test validates that both constructors work and create valid instances 31 | } 32 | 33 | [Fact] 34 | public void DelegateLoggerAdapter_Produces_Expected_Output() 35 | { 36 | // Arrange 37 | var output = new List(); 38 | var adapter = new DelegateLoggerAdapter(message => output.Add(message)); 39 | 40 | // Act 41 | adapter.LogInformation("{Message}", "Test info message"); 42 | adapter.LogWarning("{Message}", "Test warning message"); 43 | 44 | // Assert 45 | Assert.Equal(2, output.Count); 46 | Assert.Equal("Test info message", output[0]); 47 | Assert.Equal("Test warning message", output[1]); 48 | } 49 | 50 | private static IChatClient CreateMockChatClient() 51 | { 52 | return new TestChatClient(); 53 | } 54 | 55 | private class TestChatClient : IChatClient 56 | { 57 | public ChatClientMetadata Metadata => new("Test"); 58 | 59 | public Task GetResponseAsync(IEnumerable chatMessages, ChatOptions? options = null, CancellationToken cancellationToken = default) 60 | { 61 | return Task.FromResult(new ChatResponse(new ChatMessage(ChatRole.Assistant, "Test"))); 62 | } 63 | 64 | public async IAsyncEnumerable GetStreamingResponseAsync(IEnumerable chatMessages, ChatOptions? options = null, [System.Runtime.CompilerServices.EnumeratorCancellation] CancellationToken cancellationToken = default) 65 | { 66 | await Task.Delay(1, cancellationToken); 67 | yield break; 68 | } 69 | 70 | public TService? GetService(object? key = null) where TService : class => null; 71 | public object? GetService(Type serviceType, object? key = null) => null; 72 | public void Dispose() { } 73 | } 74 | 75 | private class StringLoggerProvider : ILoggerProvider 76 | { 77 | private readonly StringBuilder _output; 78 | 79 | public StringLoggerProvider(StringBuilder output) 80 | { 81 | _output = output; 82 | } 83 | 84 | public ILogger CreateLogger(string categoryName) 85 | { 86 | return new StringLogger(_output); 87 | } 88 | 89 | public void Dispose() { } 90 | } 91 | 92 | private class StringLogger : ILogger 93 | { 94 | private readonly StringBuilder _output; 95 | 96 | public StringLogger(StringBuilder output) 97 | { 98 | _output = output; 99 | } 100 | 101 | public IDisposable? BeginScope(TState state) where TState : notnull => null; 102 | public bool IsEnabled(LogLevel logLevel) => true; 103 | 104 | public void Log(LogLevel logLevel, EventId eventId, TState state, Exception? exception, Func formatter) 105 | { 106 | if (formatter != null) 107 | { 108 | var message = formatter(state, exception); 109 | _output.AppendLine(message); 110 | } 111 | } 112 | } 113 | } -------------------------------------------------------------------------------- /src/skUnit.Tests/AssertionTests/ChatScenarioRunnerLoggingTests.cs: -------------------------------------------------------------------------------- 1 | using Microsoft.Extensions.AI; 2 | using Microsoft.Extensions.Logging; 3 | 4 | namespace skUnit.Tests.AssertionTests; 5 | 6 | public class ChatScenarioRunnerLoggingTests 7 | { 8 | [Fact] 9 | public void Constructor_WithILogger_UsesLogger() 10 | { 11 | // Arrange 12 | var loggerFactory = LoggerFactory.Create(builder => builder.AddConsole()); 13 | var logger = loggerFactory.CreateLogger(); 14 | var chatClient = CreateMockChatClient(); 15 | 16 | // Act 17 | var scenarioRunner = new ChatScenarioRunner(chatClient, logger); 18 | 19 | // Assert 20 | Assert.NotNull(scenarioRunner); 21 | // The fact that no exception was thrown indicates the constructor worked correctly 22 | } 23 | 24 | [Fact] 25 | public void Constructor_WithNullILogger_UsesNullLogger() 26 | { 27 | // Arrange 28 | var chatClient = CreateMockChatClient(); 29 | 30 | // Act 31 | var scenarioRunner = new ChatScenarioRunner(chatClient, (ILogger?)null); 32 | 33 | // Assert 34 | Assert.NotNull(scenarioRunner); 35 | // Should not throw even with null logger due to NullLogger fallback 36 | } 37 | 38 | [Fact] 39 | public void Constructor_WithActionDelegate_CreatesAdapter() 40 | { 41 | // Arrange 42 | var loggedMessages = new List(); 43 | Action onLog = message => loggedMessages.Add(message); 44 | var chatClient = CreateMockChatClient(); 45 | 46 | // Act 47 | var scenarioRunner = new ChatScenarioRunner(chatClient, onLog); 48 | 49 | // Assert 50 | Assert.NotNull(scenarioRunner); 51 | // The constructor should have created a DelegateLoggerAdapter internally 52 | } 53 | 54 | [Fact] 55 | public void DelegateLoggerAdapter_LogsMessages() 56 | { 57 | // Arrange 58 | var loggedMessages = new List(); 59 | Action onLog = message => loggedMessages.Add(message); 60 | var adapter = new DelegateLoggerAdapter(onLog); 61 | 62 | // Act 63 | adapter.LogInformation("Test message"); 64 | adapter.LogWarning("Warning message"); 65 | 66 | // Assert 67 | Assert.Equal(2, loggedMessages.Count); 68 | Assert.Equal("Test message", loggedMessages[0]); 69 | Assert.Equal("Warning message", loggedMessages[1]); 70 | } 71 | 72 | [Fact] 73 | public void DelegateLoggerAdapter_IsEnabled_ReturnsTrue() 74 | { 75 | // Arrange 76 | var adapter = new DelegateLoggerAdapter(_ => { }); 77 | 78 | // Act & Assert 79 | Assert.True(adapter.IsEnabled(LogLevel.Information)); 80 | Assert.True(adapter.IsEnabled(LogLevel.Warning)); 81 | Assert.True(adapter.IsEnabled(LogLevel.Error)); 82 | Assert.True(adapter.IsEnabled(LogLevel.Debug)); 83 | } 84 | 85 | [Fact] 86 | public void DelegateLoggerAdapter_BeginScope_ReturnsNull() 87 | { 88 | // Arrange 89 | var adapter = new DelegateLoggerAdapter(_ => { }); 90 | 91 | // Act 92 | var scope = adapter.BeginScope("test scope"); 93 | 94 | // Assert 95 | Assert.Null(scope); 96 | } 97 | 98 | [Fact] 99 | public void DelegateLoggerAdapter_WithNullAction_ThrowsArgumentNullException() 100 | { 101 | // Act & Assert 102 | Assert.Throws(() => new DelegateLoggerAdapter(null!)); 103 | } 104 | 105 | private static IChatClient CreateMockChatClient() 106 | { 107 | return new TestChatClient(); 108 | } 109 | 110 | private class TestChatClient : IChatClient 111 | { 112 | public ChatClientMetadata Metadata => new("Test"); 113 | 114 | public Task GetResponseAsync(IEnumerable chatMessages, ChatOptions? options = null, CancellationToken cancellationToken = default) 115 | { 116 | return Task.FromResult(new ChatResponse(new ChatMessage(ChatRole.Assistant, "Test"))); 117 | } 118 | 119 | public async IAsyncEnumerable GetStreamingResponseAsync(IEnumerable chatMessages, ChatOptions? options = null, [System.Runtime.CompilerServices.EnumeratorCancellation] CancellationToken cancellationToken = default) 120 | { 121 | await Task.Delay(1, cancellationToken); 122 | yield break; // Simple implementation for testing 123 | } 124 | 125 | public TService? GetService(object? key = null) where TService : class => null; 126 | public object? GetService(Type serviceType, object? key = null) => null; 127 | public void Dispose() { } 128 | } 129 | } -------------------------------------------------------------------------------- /src/skUnit/Asserts/ChatScenarioRunner_Initialize.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections.Generic; 3 | using System.Linq; 4 | using System.Text; 5 | using System.Threading.Tasks; 6 | using Microsoft.Extensions.AI; 7 | using Microsoft.Extensions.Logging; 8 | using Microsoft.Extensions.Logging.Abstractions; 9 | using SemanticValidation; 10 | using skUnit.Exceptions; 11 | using skUnit.Scenarios; 12 | using skUnit.Scenarios.Parsers.Assertions; 13 | 14 | namespace skUnit 15 | { 16 | /// 17 | /// Runner for executing chat-based test scenarios defined in markdown format. 18 | /// This class contains various methods to run and validate chat scenarios against AI chat clients. 19 | /// 20 | /// The ChatScenarioRunner uses two different clients: 21 | /// 1. Assertion Client: Passed to the constructor - used for semantic evaluations and assertions 22 | /// 2. System Under Test Client: Passed to RunAsync - the client whose behavior is being tested 23 | /// 24 | public partial class ChatScenarioRunner 25 | { 26 | private readonly ILogger _logger; 27 | private Semantic Semantic { get; set; } 28 | 29 | /// 30 | /// Creates a new ChatScenarioRunner with an assertion client and logger. 31 | /// 32 | /// The chat client used for semantic evaluations and assertions (not the system under test) 33 | /// Optional logger for test execution output 34 | public ChatScenarioRunner(IChatClient assertionClient, ILogger? logger = null) 35 | { 36 | Semantic = new Semantic(assertionClient); 37 | _logger = logger ?? NullLogger.Instance; 38 | } 39 | 40 | /// 41 | /// Creates a new ChatScenarioRunner with an assertion client and action-based logging. 42 | /// 43 | /// The chat client used for semantic evaluations and assertions (not the system under test) 44 | /// Optional action for logging test execution output 45 | public ChatScenarioRunner(IChatClient assertionClient, Action? onLog) 46 | { 47 | Semantic = new Semantic(assertionClient); 48 | _logger = onLog != null ? new DelegateLoggerAdapter(onLog) : NullLogger.Instance; 49 | } 50 | 51 | private void Log(string? message = "") 52 | { 53 | _logger.LogInformation("{Message}", message ?? ""); 54 | } 55 | 56 | private void LogWarning(string message) 57 | { 58 | _logger.LogWarning("{Message}", message); 59 | } 60 | 61 | private async Task CheckAssertionAsync(IKernelAssertion assertion, ChatResponse response, IList chatHistory, string keyword = "ASSERT") 62 | { 63 | Log($"### {keyword} {assertion.AssertionType}"); 64 | Log($"{assertion.Description}"); 65 | 66 | try 67 | { 68 | await assertion.Assert(Semantic, response, chatHistory); 69 | Log($"✅ OK"); 70 | Log(""); 71 | } 72 | catch (SemanticAssertException exception) 73 | { 74 | Log("❌ FAIL"); 75 | Log("Reason:"); 76 | Log(exception.Message); 77 | Log(); 78 | throw; 79 | } 80 | } 81 | } 82 | 83 | /// 84 | /// Internal adapter that wraps an Action<string> delegate to provide ILogger functionality 85 | /// for backward compatibility with the legacy onLog parameter. 86 | /// 87 | public class DelegateLoggerAdapter : ILogger 88 | { 89 | private readonly Action _logAction; 90 | 91 | public DelegateLoggerAdapter(Action logAction) 92 | { 93 | _logAction = logAction ?? throw new ArgumentNullException(nameof(logAction)); 94 | } 95 | 96 | public IDisposable? BeginScope(TState state) where TState : notnull 97 | { 98 | // No scope support needed for this adapter 99 | return null; 100 | } 101 | 102 | public bool IsEnabled(LogLevel logLevel) 103 | { 104 | // Always enabled for all log levels since the original Action doesn't have level filtering 105 | return true; 106 | } 107 | 108 | public void Log(LogLevel logLevel, EventId eventId, TState state, Exception? exception, Func formatter) 109 | { 110 | if (formatter == null) return; 111 | 112 | var message = formatter(state, exception); 113 | _logAction(message); 114 | } 115 | } 116 | } -------------------------------------------------------------------------------- /src/skUnit/Scenarios/Parsers/Assertions/JsonCheckAssertion.cs: -------------------------------------------------------------------------------- 1 | using SemanticValidation; 2 | using skUnit.Exceptions; 3 | using System.Text.Json.Nodes; 4 | using SemanticValidation.Utils; 5 | using Microsoft.Extensions.AI; 6 | 7 | namespace skUnit.Scenarios.Parsers.Assertions 8 | { 9 | public class JsonCheckAssertion : IKernelAssertion 10 | { 11 | /// 12 | /// The expected conditions for a json answer. 13 | /// 14 | private string JsonCheckText { get; set; } = default!; 15 | public JsonObject? JsonCheck { get; set; } 16 | public JsonCheckAssertion SetJsonAssertText(string jsonAssertText) 17 | { 18 | if (string.IsNullOrWhiteSpace(jsonAssertText)) 19 | throw new InvalidOperationException("The JsonCheck is empty."); 20 | 21 | JsonCheckText = (jsonAssertText ?? ""); 22 | 23 | var json = SemanticUtils.PowerParseJson(JsonCheckText); 24 | 25 | JsonCheck = json ?? throw new InvalidOperationException($""" 26 | Can not parse json: 27 | {JsonCheckText} 28 | """); 29 | 30 | return this; 31 | } 32 | 33 | /// 34 | /// Checks if is meets the conditions in JsonCheck 35 | /// 36 | /// 37 | /// 38 | /// 39 | /// 40 | /// 41 | /// 42 | public async Task Assert(Semantic semantic, ChatResponse response, IList? history = null) 43 | { 44 | var answerJson = SemanticUtils.PowerParseJson(response.Text) 45 | ?? throw new InvalidOperationException($""" 46 | Can not parse answer to json: 47 | {response.Text} 48 | """); 49 | 50 | if (JsonCheck is null) 51 | throw new InvalidOperationException("JsonCheck is null"); 52 | 53 | foreach (var prop in JsonCheck) 54 | { 55 | var checkArray = prop.Value?.AsArray(); 56 | 57 | if (checkArray is null || checkArray.Count > 2) 58 | throw new InvalidOperationException($""" 59 | JsonCheck has not a proper array, (it should have maximum 2 members): 60 | {checkArray?.ToJsonString()} 61 | """); 62 | 63 | var check = checkArray[0]?.GetValue(); 64 | var body = checkArray.ElementAtOrDefault(1)?.GetValue() ?? ""; 65 | 66 | if (string.IsNullOrWhiteSpace(check)) 67 | throw new InvalidOperationException($"JsonCheck check field is empty: {checkArray?.ToJsonString()}"); 68 | 69 | var assertion = KernelAssertionParser.Parse(body, check); 70 | 71 | if (answerJson.ContainsKey(prop.Key)) 72 | { 73 | var answerValue = answerJson[prop.Key]?.GetValue() ?? ""; 74 | try 75 | { 76 | await assertion.Assert(semantic, new ChatResponse(new ChatMessage(ChatRole.Assistant, answerValue)), history); 77 | } 78 | catch (SemanticAssertException semanticAssertException) 79 | { 80 | throw new SemanticAssertException($""" 81 | JsonCheck Assertion failed: 82 | {checkArray?.ToJsonString()} 83 | DESCRIPTION: 84 | {semanticAssertException.Message} 85 | """); 86 | } 87 | catch (InvalidOperationException invalidOperationException) 88 | { 89 | throw new InvalidOperationException($""" 90 | Invalid JsonCheck: 91 | {checkArray?.ToJsonString()} 92 | DESCRIPTION: 93 | {invalidOperationException.Message} 94 | """); 95 | } 96 | } 97 | else 98 | { 99 | throw new SemanticAssertException($""" 100 | Property '{prop.Key} ' not found in answer: 101 | {response} 102 | """); 103 | } 104 | } 105 | } 106 | 107 | public async Task Assert(Semantic semantic, string answer, IList? history = null) 108 | { 109 | await Assert(semantic, new ChatResponse(new ChatMessage(ChatRole.Assistant, answer)), history); 110 | } 111 | 112 | public string AssertionType => "JsonCheck"; 113 | public string Description => JsonCheckText; 114 | 115 | public override string ToString() => $"{AssertionType}: {JsonCheckText}"; 116 | } 117 | } 118 | -------------------------------------------------------------------------------- /docs/invocation-scenario-spec.md: -------------------------------------------------------------------------------- 1 | # Invocation Scenario Spec 2 | ## What is an invocation scenario? 3 | An invocation scenario is a way of testing SemanticKernel units, such as *plugin functions* and *kernels*, in skUnit. 4 | A scenario consists of providing some input to a unit, invoking it, and checking the expected output. 5 | 6 | For example, suppose you have a `GetSentiment` function that takes a text and returns its sentiment, such as _"Happy"_ or _"Sad"_. 7 | You can test this function with different scenarios, such as: 8 | 9 | ```md 10 | ## PARAMETER input 11 | Such a beautiful day it is 12 | 13 | ## ANSWER Equals 14 | Happy 15 | ``` 16 | 17 | This scenario verifies that the function returns _"Happy"_ when the input is _"Such a beautiful day it is"_. 18 | 19 | If the function only has one parameter, you can omit the `## PARAMETER input` line and write the input directly, like this: 20 | 21 | ```md 22 | Such a beautiful day it is 23 | 24 | ## ANSWER Equals 25 | Happy 26 | ``` 27 | 28 | This scenario is equivalent to the previous one. 29 | 30 | ## Scenario Features 31 | You can use various features to define more complex scenarios. In this section, we will explain some of them with an example. 32 | 33 | Let's say you have a function called `GetSentiment` that takes two parameters and returns a sentence describing the sentiment of the text: 34 | 35 | **Parameters**: 36 | - _input_: the text to analyze 37 | - _options_: the possible sentiment values, such as _happy_, _angry_, or _sad_ 38 | 39 | **Returns**: a sentence like _"The sentiment is happy"_ or _"The sentiment of this text is sad"_. 40 | 41 | Here is a scenario that tests this function: 42 | 43 | ```md 44 | # SCENARIO GetSentimentHappy 45 | 46 | ## PARAMETER input 47 | Such a beautiful day it is 48 | 49 | ## PARAMETER options 50 | happy, angry 51 | 52 | ## ANSWER SemanticSimilar 53 | The sentiment is happy 54 | ``` 55 | 56 | Let's break down this scenario line by line. 57 | 58 | ```md 59 | # SCENARIO GetSentimentHappy 60 | ``` 61 | This line gives a name or description to the scenario, which can be used later to generate xUnit theories. 62 | 63 | ```md 64 | ## PARAMETER input 65 | Such a beautiful day it is 66 | 67 | ## PARAMETER options 68 | happy, angry 69 | ``` 70 | These lines define the values of the parameters that will be passed to the function, such as: 71 | 72 | ```csharp 73 | var actual = await function.InvokeAsync(kernel, 74 | arguments: new() 75 | { 76 | ["input"] = "Such a beautiful day it is", 77 | ["options"] = "happy,angry" 78 | }); 79 | ``` 80 | 81 | ```md 82 | ## ANSWER SemanticSimilar 83 | The sentiment is happy 84 | ``` 85 | This line specifies the expected output of the function and how to compare it with the actual output. 86 | In this case, the output should be **semantically similar** to _"The sentiment is happy"_. 87 | This means that the output can have different words or syntax, but the meaning should be the same. 88 | 89 | > This is a powerful feature of skUnit scenarios, as **it allows you to use OpenAI itself to perform semantic comparisons**. 90 | 91 | This assertion can also be written in an alternative style: 92 | 93 | ```md 94 | ## ANSWER 95 | The sentiment of the sentence is happy 96 | 97 | ## CHECK SemanticSimilar 98 | The sentiment is happy 99 | ``` 100 | 101 | In this style, an expected answer is just written to serve as a reminder of the anticipated answer (which will not be used during assertions); 102 | and then a `## CHECK SemanticSimilar` is used to explicitly perform the assertion. 103 | 104 | However, `SemanticSimilar` is not the only assertion method. There are many more assertion checks available (like **SemanticCondition**, **Equals**). 105 | 106 | You can see the full list of CHECK statements here: [CHECK Statement spec](https://github.com/mehrandvd/skunit/blob/main/docs/check-statements-spec.md). 107 | 108 | ## Advanced Scenario Features 109 | 110 | ### Flexible Use of Hashtags 111 | When defining skUnit statements such as `# SCENARIO`, `## PARAMETER`, and so on, you have the freedom to use as many hashtags as you wish. There's no strict rule that mandates a specific count of hashtags for each statement. This flexibility allows you to format your markdown in a way that enhances readability for you. However, as a best practice, we suggest adhering to the recommended usage to maintain a clear and comprehensible hierarchy. 112 | 113 | ### Unique Identifiers 114 | In certain uncommon instances, the data may contain skUnit expressions that could disrupt the parsing of the scenario. For instance, let's consider a scenario with two parameters: `input` and `options`. If the first parameter contains a markdown value that disrupts parsing, it could pose a problem: 115 | 116 | ```md 117 | # SCENARIO 118 | 119 | ## PARAMETER input 120 | This block is a markdown itself that includes this exact section: 121 | ## PARAMETER hello 122 | This could lead skUnit to mistakenly identify it as a `parameter`, which it isn't. 123 | 124 | ## PARAMETER options 125 | happy, angry 126 | ``` 127 | 128 | To handle these exceptional cases, you can employ an identifier in your statements, like so: 129 | 130 | ```md 131 | # sk SCENARIO 132 | 133 | ## sk PARAMETER input 134 | This block is a markdown itself that includes this exact section: 135 | ## PARAMETER hello 136 | This could lead skUnit to mistakenly identify it as a `parameter`, which it isn't. 137 | 138 | ## sk PARAMETER options 139 | happy, angry 140 | ``` 141 | 142 | In this example, we used `sk` as the identifier. However, you can use any identifier of your choice, such as `~`, `*`, etc. The parser will recognize whatever you use in the first statement as the unique identifier for the statements. 143 | -------------------------------------------------------------------------------- /src/skUnit/Asserts/SemanticAssert.cs: -------------------------------------------------------------------------------- 1 | using System.Text.Json; 2 | using System.Text.Json.Nodes; 3 | using Microsoft.Extensions.AI; 4 | using SemanticValidation; 5 | using skUnit.Exceptions; 6 | 7 | namespace skUnit 8 | { 9 | /// 10 | /// Contains various methods that are used to semantically verify that conditions are met during the 11 | /// process of running tests. This class uses SemanticKernel and OpenAI to validate these assertions semantically. 12 | /// 13 | public class SemanticAssert 14 | { 15 | private Semantic Semantic { get; set; } 16 | 17 | /// 18 | /// This class needs a SemanticKernel chatClient to work. 19 | /// Pass your pre-configured chatClient to this constructor. 20 | /// 21 | /// 22 | public SemanticAssert(IChatClient chatClient) 23 | { 24 | Semantic = new Semantic(chatClient); 25 | 26 | } 27 | 28 | /// 29 | /// Checks whether and string are semantically similar. 30 | /// It uses the kernel and OpenAI to check this semantically. 31 | /// 32 | /// 33 | /// SemanticAssert.SimilarAsync("This automobile is red", "The car is red") // returns true 34 | /// SemanticAssert.SimilarAsync("This tree is red", "The car is red") // returns false 35 | /// 36 | /// 37 | /// 38 | /// 39 | /// 40 | /// 41 | /// If the OpenAI was unable to generate a valid response. 42 | public async Task SimilarAsync(string first, string second) 43 | { 44 | var result = await Semantic.AreSimilarAsync(first, second); 45 | 46 | if (result is null) 47 | { 48 | throw new InvalidOperationException("Unable to accomplish the semantic assert."); 49 | } 50 | 51 | if (!result.IsValid) 52 | { 53 | throw new SemanticAssertException(result.Reason ?? "No reason is provided."); 54 | } 55 | } 56 | 57 | /// 58 | /// Checks whether and string are semantically similar. 59 | /// It uses the kernel and OpenAI to check this semantically. 60 | /// 61 | /// 62 | /// SemanticAssert.Similar("This automobile is red", "The car is red") // returns true 63 | /// SemanticAssert.Similar("This tree is red", "The car is red") // returns false 64 | /// 65 | /// 66 | /// 67 | /// 68 | /// 69 | /// 70 | /// If the OpenAI was unable to generate a valid response. 71 | public void Similar(string first, string second) 72 | { 73 | SimilarAsync(first, second).GetAwaiter().GetResult(); 74 | } 75 | 76 | /// 77 | /// Checks whether and string are NOT semantically similar. 78 | /// It uses the kernel and OpenAI to check this semantically. It also describes the reason that they are not similar. 79 | /// 80 | /// 81 | /// SemanticAssert.NotSimilarAsync("This bicycle is red", "The car is red") 82 | /// // returns: 83 | /// { 84 | /// IsValid: false, 85 | /// Reason: "The first text describes a red bicycle, while the second text describes a red car. They are not semantically equivalent." 86 | /// } 87 | /// 88 | /// 89 | /// 90 | /// 91 | /// 92 | /// 93 | /// If the OpenAI was unable to generate a valid response. 94 | public async Task NotSimilarAsync(string first, string second) 95 | { 96 | var result = await Semantic.AreSimilarAsync(first, second); 97 | 98 | if (result is null) 99 | { 100 | throw new SemanticAssertException("Unable to accomplish the semantic assert."); 101 | } 102 | 103 | if (result.IsValid) 104 | { 105 | throw new SemanticAssertException($""" 106 | These are semantically similar: 107 | [FIRST]: {first} 108 | [SECOND]: {second} 109 | """); 110 | } 111 | } 112 | 113 | /// 114 | /// Checks whether and string are NOT semantically similar. 115 | /// It uses the kernel and OpenAI to check this semantically. It also describes the reason that they are not similar. 116 | /// 117 | /// 118 | /// SemanticAssert.NotSimilar("This bicycle is red", "The car is red") 119 | /// // returns: 120 | /// { 121 | /// IsValid: false, 122 | /// Reason: "The first text describes a red bicycle, while the second text describes a red car. They are not semantically equivalent." 123 | /// } 124 | /// 125 | /// 126 | /// 127 | /// 128 | /// 129 | /// 130 | /// If the OpenAI was unable to generate a valid response. 131 | public void NotSimilar(string first, string second) 132 | { 133 | NotSimilarAsync(first, second).GetAwaiter().GetResult(); 134 | } 135 | } 136 | } 137 | -------------------------------------------------------------------------------- /src/skUnit/Asserts/ScenarioAssert_ChatClient.cs: -------------------------------------------------------------------------------- 1 | using SemanticValidation; 2 | using skUnit.Exceptions; 3 | using skUnit.Scenarios; 4 | using System; 5 | using System.Collections.Generic; 6 | using System.Diagnostics.CodeAnalysis; 7 | using System.Linq; 8 | using System.Text; 9 | using System.Threading.Tasks; 10 | using Microsoft.Extensions.AI; 11 | using Microsoft.SemanticKernel; 12 | using OpenAI.Chat; 13 | using ChatMessage = Microsoft.Extensions.AI.ChatMessage; 14 | using FunctionCallContent = Microsoft.Extensions.AI.FunctionCallContent; 15 | using FunctionResultContent = Microsoft.Extensions.AI.FunctionResultContent; 16 | using Microsoft.SemanticKernel.ChatCompletion; 17 | 18 | namespace skUnit 19 | { 20 | public partial class ScenarioAssert 21 | { 22 | /// 23 | /// Checks whether the passes on the given 24 | /// using its ChatCompletionService. 25 | /// If you want to test the kernel using something other than ChatCompletionService (for example using your own function), 26 | /// pass and specify how do you want the answer be created from chat history like: 27 | /// 28 | /// getAnswerFunc = async history => 29 | /// await AnswerChatFunction.InvokeAsync(kernel, new KernelArguments() 30 | /// { 31 | /// ["history"] = history, 32 | /// }); 33 | /// 34 | /// 35 | /// 36 | /// 37 | /// 38 | /// 39 | /// 40 | /// 41 | /// If the OpenAI was unable to generate a valid response. 42 | [Obsolete("Use ChatScenarioRunner.RunAsync instead. This method will be removed in a future version.", false)] 43 | public async Task PassAsync( 44 | ChatScenario scenario, 45 | IChatClient? chatClient = null, 46 | Func, Task>? getAnswerFunc = null, 47 | IList? chatHistory = null, 48 | ScenarioRunOptions? options = null 49 | ) 50 | { 51 | await _runner.RunAsync(scenario, chatClient, getAnswerFunc, chatHistory, options); 52 | } 53 | 54 | /// 55 | /// Checks whether all of the passes on the given 56 | /// using its ChatCompletionService. 57 | /// If you want to test the kernel using something other than ChatCompletionService (for example using your own function), 58 | /// pass and specify how do you want the answer be created from chat history like: 59 | /// 60 | /// getAnswerFunc = async history => 61 | /// await AnswerChatFunction.InvokeAsync(kernel, new KernelArguments() 62 | /// { 63 | /// ["history"] = history, 64 | /// }); 65 | /// 66 | /// 67 | /// 68 | /// 69 | /// 70 | /// 71 | /// 72 | /// If the OpenAI was unable to generate a valid response. 73 | [Obsolete("Use ChatScenarioRunner.RunAsync instead. This method will be removed in a future version.", false)] 74 | public async Task PassAsync(List scenarios, IChatClient? chatClient = null, Func, Task>? getAnswerFunc = null, IList? chatHistory = null, ScenarioRunOptions? options = null) 75 | { 76 | await _runner.RunAsync(scenarios, chatClient, getAnswerFunc, chatHistory, options); 77 | } 78 | 79 | /// 80 | /// Checks whether all of the passes on the given 81 | /// using its ChatCompletionService. 82 | /// If you want to test the kernel using something other than ChatCompletionService (for example using your own function), 83 | /// pass and specify how do you want the answer be created from chat history like: 84 | /// 85 | /// getAnswerFunc = async history => 86 | /// await AnswerChatFunction.InvokeAsync(kernel, new KernelArguments() 87 | /// { 88 | /// ["history"] = history, 89 | /// }); 90 | /// 91 | /// 92 | /// 93 | /// 94 | /// 95 | /// 96 | /// If the OpenAI was unable to generate a valid response. 97 | [Experimental("SKEXP0001")] 98 | [Obsolete("Use ChatScenarioRunner.RunAsync instead. This method will be removed in a future version.", false)] 99 | public async Task PassAsync(List scenarios, Kernel kernel, IList? chatHistory = null, ScenarioRunOptions? options = null) 100 | { 101 | #pragma warning disable SKEXP0001 102 | await _runner.RunAsync(scenarios, kernel, chatHistory, options); 103 | #pragma warning restore SKEXP0001 104 | } 105 | 106 | [Experimental("SKEXP0001")] 107 | [Obsolete("Use ChatScenarioRunner.RunAsync instead. This method will be removed in a future version.", false)] 108 | public async Task PassAsync(ChatScenario scenarios, Kernel kernel, IList? chatHistory = null, ScenarioRunOptions? options = null) 109 | { 110 | #pragma warning disable SKEXP0001 111 | await _runner.RunAsync(scenarios, kernel, chatHistory, options); 112 | #pragma warning restore SKEXP0001 113 | } 114 | } 115 | } 116 | -------------------------------------------------------------------------------- /docs/mcp-testing-guide.md: -------------------------------------------------------------------------------- 1 | # MCP Server Testing Guide 2 | 3 | This guide shows you how to test [Model Context Protocol (MCP)](https://modelcontextprotocol.io/) servers using skUnit. 4 | 5 | ## What is MCP? 6 | 7 | Model Context Protocol (MCP) is an open protocol that enables AI applications to securely connect to data sources and tools. MCP servers provide tools that AI models can call to access information or perform actions. 8 | 9 | ## Testing MCP Servers with skUnit 10 | 11 | skUnit makes it easy to test that your MCP servers work correctly with AI models. You can verify that: 12 | - The right tools are called 13 | - Tools are called with correct parameters 14 | - Responses are handled properly 15 | - Multiple MCP servers work together 16 | 17 | ## Basic MCP Testing Setup 18 | 19 | ```csharp 20 | [Fact] 21 | public async Task TestMcpServer() 22 | { 23 | // 1. Setup MCP client transport (stdio, HTTP, etc.) 24 | var clientTransport = new StdioClientTransport(new StdioClientTransportOptions 25 | { 26 | Name = "My MCP Server", 27 | Command = "node", 28 | Arguments = ["my-mcp-server.js"] 29 | }); 30 | 31 | // 2. Create MCP client and get tools 32 | await using var mcp = await McpClientFactory.CreateAsync(clientTransport); 33 | var tools = await mcp.ListToolsAsync(); 34 | 35 | // 3. Setup chat client with MCP tools 36 | var chatClient = new ChatClientBuilder(baseChatClient) 37 | .ConfigureOptions(options => options.Tools = tools.ToArray()) 38 | .UseFunctionInvocation() 39 | .Build(); 40 | 41 | // 4. Run your test scenario 42 | var markdown = File.ReadAllText("mcp-test.md"); 43 | var scenarios = await ChatScenario.LoadFromText(markdown); 44 | 45 | await ScenarioAssert.PassAsync(scenarios, chatClient); 46 | } 47 | ``` 48 | 49 | ## Example MCP Test Scenarios 50 | 51 | ### Testing Time Server 52 | 53 | ```md 54 | # SCENARIO Time Server Test 55 | 56 | ## [USER] 57 | What time is it? 58 | 59 | ## [ASSISTANT] 60 | It's currently 2:30 PM PST 61 | 62 | ### CHECK FunctionCall 63 | { 64 | "function_name": "current_time" 65 | } 66 | 67 | ### CHECK SemanticCondition 68 | It mentions a specific time 69 | ``` 70 | 71 | ### Testing Calendar Server 72 | 73 | ```md 74 | # SCENARIO Calendar Operations 75 | 76 | ## [USER] 77 | Schedule a meeting for tomorrow at 2 PM 78 | 79 | ## [ASSISTANT] 80 | I've scheduled your meeting for tomorrow at 2:00 PM 81 | 82 | ### CHECK FunctionCall 83 | { 84 | "function_name": "schedule_event", 85 | "arguments": { 86 | "title": ["NotEmpty"], 87 | "datetime": ["SemanticCondition", "It represents tomorrow at 2 PM"] 88 | } 89 | } 90 | 91 | ### CHECK SemanticCondition 92 | It confirms the meeting was scheduled 93 | ``` 94 | 95 | ### Testing File System Server 96 | 97 | ```md 98 | # SCENARIO File Operations 99 | 100 | ## [USER] 101 | List files in the current directory 102 | 103 | ## [AGENT] 104 | Here are the files: file1.txt, file2.md, folder1/ 105 | 106 | ### CHECK FunctionCall 107 | { 108 | "function_name": "list_directory" 109 | } 110 | 111 | ### CHECK SemanticCondition 112 | It lists file names or mentions files 113 | ``` 114 | 115 | ## Testing Multiple MCP Servers 116 | 117 | You can test scenarios involving multiple MCP servers working together: 118 | 119 | ```csharp 120 | [Fact] 121 | public async Task TestMultipleMcpServers() 122 | { 123 | // Setup multiple MCP servers 124 | var timeServer = await McpClientFactory.CreateAsync(timeTransport); 125 | var weatherServer = await McpClientFactory.CreateAsync(weatherTransport); 126 | var calendarServer = await McpClientFactory.CreateAsync(calendarTransport); 127 | 128 | // Combine all tools 129 | var allTools = new List(); 130 | allTools.AddRange(await timeServer.ListToolsAsync()); 131 | allTools.AddRange(await weatherServer.ListToolsAsync()); 132 | allTools.AddRange(await calendarServer.ListToolsAsync()); 133 | 134 | // Setup chat client with all tools 135 | var chatClient = new ChatClientBuilder(baseChatClient) 136 | .ConfigureOptions(options => options.Tools = allTools.ToArray()) 137 | .UseFunctionInvocation() 138 | .Build(); 139 | 140 | // Test complex scenario 141 | var scenarios = await ChatScenario.LoadFromText(complexScenario); 142 | await ScenarioAssert.PassAsync(scenarios, chatClient); 143 | } 144 | ``` 145 | 146 | ### Multi-Server Test Scenario 147 | 148 | ```md 149 | # SCENARIO Multi-Server Coordination 150 | 151 | ## [USER] 152 | Check the weather and schedule a meeting for tomorrow if it's sunny 153 | 154 | ## [AGENT] 155 | I checked the weather - it's going to be sunny tomorrow! I've scheduled your meeting for 2 PM. 156 | 157 | ### CHECK FunctionCall 158 | { 159 | "function_name": "get_weather" 160 | } 161 | 162 | ### CHECK FunctionCall 163 | { 164 | "function_name": "schedule_event", 165 | "arguments": { 166 | "datetime": ["SemanticCondition", "It's for tomorrow"] 167 | } 168 | } 169 | 170 | ### CHECK SemanticCondition 171 | It mentions both weather information and confirms meeting scheduling 172 | ``` 173 | 174 | ## MCP Testing Best Practices 175 | 176 | 1. **Isolate Server Testing**: Test each MCP server individually before testing combinations 177 | 2. **Verify Tool Discovery**: Ensure `ListToolsAsync()` returns expected tools 178 | 3. **Test Error Scenarios**: Verify how your system handles MCP server failures 179 | 4. **Check Parameters**: Use argument assertions to verify tools are called with correct data 180 | 5. **Mock External Dependencies**: Use test doubles for external services your MCP server depends on 181 | 182 | ## Configuration for MCP Testing 183 | 184 | Some MCP servers require configuration (API keys, endpoints, etc.). Use environment variables or user secrets: 185 | 186 | ```json 187 | { 188 | "MCP_TimeServer_Endpoint": "https://time-api.example.com", 189 | "MCP_Weather_ApiKey": "your-weather-api-key" 190 | } 191 | ``` 192 | 193 | ## Troubleshooting MCP Tests 194 | 195 | - **Tool not found**: Check that the MCP server is running and tools are properly registered 196 | - **Function not called**: Verify the AI model has access to the tool and understands when to use it 197 | - **Parameter mismatch**: Ensure your CHECK FunctionCall assertions match the actual tool schema 198 | - **Transport issues**: Check MCP server logs and transport configuration 199 | 200 | For more details on CHECK statements, see [CHECK Statement Spec](check-statements-spec.md). -------------------------------------------------------------------------------- /src/skUnit/Scenarios/Parsers/Assertions/FunctionCallCheckAssertion.cs: -------------------------------------------------------------------------------- 1 | using SemanticValidation; 2 | using skUnit.Exceptions; 3 | using System.Text.Json.Nodes; 4 | using SemanticValidation.Utils; 5 | using Microsoft.Extensions.AI; 6 | using FunctionCallContent = Microsoft.Extensions.AI.FunctionCallContent; 7 | using FunctionResultContent = Microsoft.Extensions.AI.FunctionResultContent; 8 | using Markdig.Helpers; 9 | 10 | namespace skUnit.Scenarios.Parsers.Assertions 11 | { 12 | public class FunctionCallAssertion : IKernelAssertion 13 | { 14 | // private readonly ArgumentConditionFactory factory = new(); 15 | 16 | /// 17 | /// The expected conditions for a json answer. 18 | /// 19 | /// 20 | private string FunctionCallText { get; set; } = default!; 21 | 22 | public JsonObject? FunctionCallJson { get; set; } 23 | 24 | public string? FunctionName { get; set; } 25 | 26 | public Dictionary FunctionArguments { get; set; } = new(); 27 | 28 | public FunctionCallAssertion SetJsonAssertText(string jsonAssertText) 29 | { 30 | if (string.IsNullOrWhiteSpace(jsonAssertText)) 31 | throw new InvalidOperationException("The FunctionCallCheck is empty."); 32 | 33 | FunctionCallText = jsonAssertText ?? ""; 34 | 35 | try 36 | { 37 | var json = SemanticUtils.PowerParseJson(FunctionCallText); 38 | FunctionCallJson = json; 39 | } 40 | catch 41 | { 42 | // So it's a raw function name. 43 | } 44 | 45 | if (FunctionCallJson is not null) 46 | { 47 | FunctionName = FunctionCallJson["function_name"]?.GetValue(); 48 | 49 | if (FunctionCallJson["arguments"] is JsonObject argumentsJson) 50 | { 51 | foreach (var kv in argumentsJson) 52 | { 53 | string checkValuesText; 54 | string checkType; 55 | 56 | if (kv.Value is JsonValue checkValue) 57 | { 58 | checkType = "Equals"; 59 | checkValuesText = checkValue.GetValue(); 60 | } 61 | else if (kv.Value is JsonArray checkArray) 62 | { 63 | //var checkArray = kv.Value.AsArray(); 64 | checkType = checkArray[0]?.GetValue() ?? throw new InvalidOperationException("No valid array assertion."); 65 | 66 | var checkValues = checkArray 67 | .Skip(1) 68 | .Select(value => value.GetValue()); 69 | 70 | checkValuesText = string.Join(", ", checkValues); 71 | } 72 | else 73 | { 74 | throw new InvalidOperationException( 75 | $""" 76 | JsonCheck has not a proper value supported json types are string and array: 77 | {kv.ToString()} 78 | """); 79 | } 80 | 81 | FunctionArguments[kv.Key] = KernelAssertionParser.Parse(checkValuesText, checkType); 82 | } 83 | } 84 | } 85 | else 86 | { 87 | FunctionName = FunctionCallText; 88 | } 89 | 90 | return this; 91 | } 92 | 93 | /// 94 | /// Checks if is meets the conditions in FunctionCallJson 95 | /// 96 | /// 97 | /// 98 | /// 99 | /// 100 | /// 101 | public async Task Assert(Semantic semantic, ChatResponse response, IList? history = null) 102 | { 103 | if (FunctionName is null) 104 | throw new InvalidOperationException("FunctionCall Name is null"); 105 | 106 | 107 | var functionCalls = response.Messages 108 | .Where( 109 | ch => ch.Contents.OfType().Any() 110 | ) 111 | .SelectMany(ch => ch.Contents.OfType()) 112 | .ToList(); 113 | 114 | var thisFunctionCalls = functionCalls 115 | .Where(fc => fc.Name == FunctionName) 116 | .ToList(); 117 | 118 | if (thisFunctionCalls.Count == 0) 119 | throw new SemanticAssertException( 120 | $""" 121 | No function call found with name: {FunctionName} 122 | Current calls: {string.Join(", ", functionCalls.Select(fc => fc.Name))} 123 | """); 124 | 125 | 126 | if (FunctionArguments.Any()) 127 | { 128 | var thisFunctionCall = thisFunctionCalls.Last(); 129 | 130 | var thisCallResult = ( 131 | from fr in response.Messages.SelectMany(c => c.Contents).OfType() 132 | where fr.CallId == thisFunctionCall.CallId 133 | select fr 134 | ).FirstOrDefault(); 135 | 136 | if (thisCallResult is null) 137 | throw new SemanticAssertException( 138 | $""" 139 | No function call result found with name: {FunctionName} 140 | """); 141 | 142 | foreach (var argumentAssertion in FunctionArguments) 143 | { 144 | var arguments = thisFunctionCall.Arguments ?? new Dictionary(); 145 | 146 | if (arguments.TryGetValue(argumentAssertion.Key, out var value)) 147 | { 148 | var assertion = argumentAssertion.Value; 149 | var actualValue = value?.ToString(); 150 | 151 | await assertion.Assert(semantic, new ChatResponse(new ChatMessage(ChatRole.Assistant, actualValue))); 152 | } 153 | else 154 | { 155 | throw new SemanticAssertException( 156 | $""" 157 | Argument {argumentAssertion.Key} is not found in the function call. 158 | """); 159 | } 160 | } 161 | } 162 | } 163 | 164 | public string AssertionType => "FunctionCall"; 165 | 166 | public string Description => FunctionCallText; 167 | 168 | public override string ToString() => $"{AssertionType}: {FunctionCallText}"; 169 | } 170 | } 171 | -------------------------------------------------------------------------------- /src/skUnit.Tests/AssertionTests/FunctionCallAssertionTests.cs: -------------------------------------------------------------------------------- 1 | using Microsoft.Extensions.AI; 2 | using skUnit.Exceptions; 3 | using skUnit.Scenarios.Parsers.Assertions; 4 | 5 | namespace skUnit.Tests.AssertionTests 6 | { 7 | public class FunctionCallAssertionTests 8 | { 9 | [Fact] 10 | public void FunctionCall_MustWork() 11 | { 12 | var assertion = new FunctionCallAssertion(); 13 | 14 | assertion.SetJsonAssertText(""" 15 | { 16 | "function_name": "test_function", 17 | } 18 | """); 19 | 20 | Assert.NotNull(assertion.FunctionCallJson); 21 | Assert.Equal("test_function", assertion.FunctionName); 22 | 23 | } 24 | 25 | [Fact] 26 | public void FunctionCall_BackQuote_MustWork() 27 | { 28 | var assertion = new FunctionCallAssertion(); 29 | 30 | assertion.SetJsonAssertText(""" 31 | ```json 32 | { 33 | "function_name": "test_function", 34 | } 35 | ``` 36 | """); 37 | 38 | Assert.NotNull(assertion.FunctionCallJson); 39 | Assert.Equal("test_function", assertion.FunctionName); 40 | 41 | } 42 | 43 | [Fact] 44 | public async Task FunctionCall_ArgumentAssertion_StringLiteral() 45 | { 46 | var history = new List() 47 | { 48 | new ChatMessage() 49 | { 50 | Role = ChatRole.Tool, 51 | Contents = new List() 52 | { 53 | new FunctionCallContent("call-id-1", "test_function", new Dictionary() 54 | { 55 | { "arg1", "value1" } 56 | }), 57 | new FunctionResultContent("call-id-1", "result"), 58 | } 59 | }, 60 | }; 61 | 62 | var assertion = new FunctionCallAssertion(); 63 | assertion.SetJsonAssertText(""" 64 | ```json 65 | { 66 | "function_name": "test_function", 67 | "arguments": { 68 | "arg1": "value1" 69 | } 70 | } 71 | ``` 72 | """); 73 | 74 | await assertion.Assert(null, new ChatResponse(history), history); 75 | 76 | Assert.Single(assertion.FunctionArguments); 77 | Assert.Equal(typeof(EqualsAssertion), assertion.FunctionArguments["arg1"].GetType()); 78 | } 79 | 80 | [Fact] 81 | public async Task FunctionCall_ArgumentAssertion_IsAnyOf() 82 | { 83 | var history = new List() 84 | { 85 | new ChatMessage() 86 | { 87 | Role = ChatRole.Tool, 88 | Contents = new List() 89 | { 90 | new FunctionCallContent("call-id-1", "test_function", new Dictionary() 91 | { 92 | { "arg1", "value1" } 93 | }), 94 | new FunctionResultContent("call-id-1", "result"), 95 | } 96 | }, 97 | }; 98 | 99 | var assertion = new FunctionCallAssertion(); 100 | assertion.SetJsonAssertText(""" 101 | ```json 102 | { 103 | "function_name": "test_function", 104 | "arguments": { 105 | "arg1": ["IsAnyOf", "value1", "value2", "value3"] 106 | } 107 | } 108 | ``` 109 | """); 110 | 111 | await assertion.Assert(null, new ChatResponse(history), history); 112 | 113 | Assert.Single(assertion.FunctionArguments); 114 | Assert.Equal(typeof(IsAnyOfAssertion), assertion.FunctionArguments["arg1"].GetType()); 115 | } 116 | 117 | [Fact] 118 | public async Task FunctionCall_ArgumentAssertion_SeveralArguments() 119 | { 120 | var history = new List() 121 | { 122 | new ChatMessage() 123 | { 124 | Role = ChatRole.Tool, 125 | Contents = new List() 126 | { 127 | new FunctionCallContent("call-id-1", "test_function", new Dictionary() 128 | { 129 | { "arg1", "value1" }, 130 | { "arg2", "Actual value: value1" }, 131 | { "arg3", "value" } 132 | }), 133 | new FunctionResultContent("call-id-1", "result"), 134 | } 135 | }, 136 | }; 137 | 138 | var assertion = new FunctionCallAssertion(); 139 | assertion.SetJsonAssertText(""" 140 | ```json 141 | { 142 | "function_name": "test_function", 143 | "arguments": { 144 | "arg1": ["IsAnyOf", "value1", "value2", "value3"], 145 | "arg2": ["ContainsAny", "value1", "value2", "value3"], 146 | "arg3": ["NotEmpty"], 147 | } 148 | } 149 | ``` 150 | """); 151 | 152 | await assertion.Assert(null, new ChatResponse(history), history); 153 | 154 | Assert.Equal(expected: 3, assertion.FunctionArguments.Count); 155 | Assert.Equal(typeof(IsAnyOfAssertion), assertion.FunctionArguments["arg1"].GetType()); 156 | Assert.Equal(typeof(ContainsAnyAssertion), assertion.FunctionArguments["arg2"].GetType()); 157 | Assert.Equal(typeof(NotEmptyAssertion), assertion.FunctionArguments["arg3"].GetType()); 158 | } 159 | 160 | [Fact] 161 | public async Task FunctionCall_ArgumentAssertion_DoesNotMatch() 162 | { 163 | var history = new List() 164 | { 165 | new ChatMessage() 166 | { 167 | Role = ChatRole.Tool, 168 | Contents = new List() 169 | { 170 | new FunctionCallContent("call-id-1", "test_function", new Dictionary() 171 | { 172 | { "arg1", "value" } 173 | }), 174 | new FunctionResultContent("call-id-1", "result"), 175 | } 176 | }, 177 | }; 178 | 179 | var assertion = new FunctionCallAssertion(); 180 | assertion.SetJsonAssertText(""" 181 | ```json 182 | { 183 | "function_name": "test_function", 184 | "arguments": { 185 | "arg1": ["IsAnyOf", "value1", "value2", "value3"] 186 | } 187 | } 188 | ``` 189 | """); 190 | 191 | var exception = await Assert.ThrowsAsync(() => 192 | { 193 | return assertion.Assert(null, new ChatResponse(history), history); 194 | }); 195 | 196 | Assert.Equal("Text is not equal to any of these: 'value1, value2, value3'", exception.Message); 197 | Assert.Single(assertion.FunctionArguments); 198 | Assert.Equal(typeof(IsAnyOfAssertion), assertion.FunctionArguments["arg1"].GetType()); 199 | } 200 | } 201 | } 202 | -------------------------------------------------------------------------------- /docs/check-statements-spec.md: -------------------------------------------------------------------------------- 1 | # CHECK Statement and ASSERT Keyword 2 | 3 | The `CHECK` statement (and the new `ASSERT` keyword) is used to **verify** the quality of an output by comparing it with some **criteria**. Both keywords work identically - use whichever you prefer: 4 | 5 | ```md 6 | ## ANSWER 7 | Yes it is a very good day 8 | 9 | ### CHECK SemanticCondition 10 | Its vibe is positive 11 | 12 | ### ASSERT SemanticCondition 13 | Its vibe is positive 14 | 15 | ### CHECK SemanticSimilar 16 | It is a beautiful day 17 | 18 | ### ASSERT SemanticSimilar 19 | It is a beautiful day 20 | 21 | ### CHECK Contains 22 | day 23 | 24 | ### ASSERT ContainsAll 25 | day 26 | 27 | ### CHECK Equals 28 | Yes it is a very good day 29 | 30 | ### ASSERT Equals 31 | Yes it is a very good day 32 | ``` 33 | 34 | Both `CHECK` and `ASSERT` keywords work identically. The `ASSERT` keyword was introduced for better readability and consistency with common testing terminology. 35 | 36 | In this example, the expected answer is something like "Yes it is a very good day", but the actual answer could be anything. To verify the actual output, you can use different types of checks. In this document, we are going to **explain** each of them in detail. 37 | 38 | ## CHECK Equals 39 | This is the most basic check. It checks if the answer is **exactly the same** as the expected one. 40 | ```md 41 | ### CHECK Equals 42 | Happy 43 | ``` 44 | This statement checks if the output exactly equals `Happy`. 45 | 46 | For this statement, `Equals` and `Equal` are both accepted. 47 | 48 | ## CHECK ContainsAll 49 | It checks if the output contains **all** of the items in the text (comma-separated). 50 | ```md 51 | ### CHECK ContainsAll 52 | happy, day, good 53 | ``` 54 | This check passes if the output contains all the words: happy, day, and good. 55 | 56 | For this statement, all these are the same: `Contains`, `Contain`, `ContainsAll` 57 | 58 | ## CHECK ContainsAny 59 | It checks if the output contains **any** of the items in the text (comma-separated). 60 | ```md 61 | ### CHECK ContainsAny 62 | happy, day, good 63 | ``` 64 | This check passes if the output contains any of the words: happy, day, or good. 65 | 66 | ## CHECK SemanticSimilar 67 | It checks if the output is **similar in meaning** to the given text. 68 | ```md 69 | ### CHECK SemanticSimilar 70 | It is a good day. 71 | ``` 72 | This check passes if the output is semantically similar to "It is a good day". It uses OpenAI tokens to check this assertion. 73 | 74 | If it fails while executing the test, it **shows** why these two texts are not semantically equivalent with a message like this: 75 | ```md 76 | ## ANSWER SemanticSimilar 77 | The sentiment is happy 78 | Exception as EXPECTED: 79 | The two texts are not semantically equivalent. The first text expresses anger, while the second text expresses happiness. 80 | ``` 81 | 82 | For this statement, all these are the same: `SemanticSimilar`, `Semantic-Similar` 83 | 84 | ## CHECK SemanticCondition 85 | It checks if the output **satisfies** the condition semantically. 86 | ```md 87 | ### CHECK SemanticCondition 88 | It talks about a good day. 89 | ``` 90 | This check passes if the output semantically meets the condition: "It talks about a good day". It uses OpenAI tokens to check this assertion. 91 | 92 | If it fails while executing the test, it **shows** why the output does not meet the condition semantically with a message like this: 93 | ```md 94 | ### CHECK SemanticCondition 95 | It talks about trees 96 | Exception as EXPECTED: 97 | The input text does not talk about trees 98 | ``` 99 | 100 | For this statement, all these are the same: `SemanticCondition`, `Semantic-Condition` 101 | 102 | ## CHECK JsonCheck 103 | It checks if the output is a valid JSON, and if each of its properties meets the specified condition. 104 | ```md 105 | ### CHECK JsonCheck 106 | { 107 | "name": ["Equals", "Mehran"], 108 | "description": ["SemanticCondition", "It mentions that he is a good software architect."] 109 | } 110 | ``` 111 | This check passes if the output is a valid JSON and: 112 | - It has a `name` property with a value equal to `Mehran` 113 | - It has a `description` property with a value that meets this semantic condition: "It mentions that he is a good software architect." 114 | 115 | This check is great for asserting functions that are going to return a specific JSON. 116 | 117 | To make your `.md` files more readable, you can annotate your JSON with ` ``` `: 118 | 119 | ``````md 120 | ### CHECK JsonCheck 121 | ```json 122 | { 123 | "name": ["Equals", "Mehran"], 124 | "description": ["SemanticCondition", "It mentions that he is a good software architect."] 125 | } 126 | ``` 127 | `````` 128 | 129 | ## CHECK Empty 130 | It ensures that the answer is empty. 131 | ```md 132 | ### CHECK Empty 133 | ``` 134 | This statement checks if the output is empty. 135 | 136 | ## CHECK NotEmpty 137 | It ensures that the answer is not empty. 138 | ```md 139 | ### CHECK NotEmpty 140 | ``` 141 | This statement checks if the output is empty. 142 | 143 | ## CHECK FunctionCall 144 | It ensures that a function call happens during answer generation. 145 | 146 | ``````md 147 | ### CHECK FunctionCall 148 | ```json 149 | { 150 | "function_name": "GetFoodMenu" 151 | } 152 | ``` 153 | `````` 154 | 155 | This statement checks if `GetFoodMenu` function has been called during the answer generation. 156 | Also, the following syntax can be used as a sugar syntactic. 157 | 158 | ```md 159 | ### CHECK FunctionCall 160 | GetFoodMenu 161 | ``` 162 | 163 | Also you can use some more advanced assertions by checking the called arguments using arguments conditions. Argument condition is an array that contains the name of the condition as the first item, followed by the values: 164 | 165 | ``````md 166 | ### CHECK FunctionCall 167 | ```json 168 | { 169 | "function_name": "GetFoodMenu", 170 | "arguments": { 171 | "mood": ["Equals", "Happy"] 172 | } 173 | } 174 | ``` 175 | `````` 176 | 177 | There are several condtions supported. 178 | ### Equals 179 | It checks if the argument is equal to the specified value: 180 | ``````md 181 | ### CHECK FunctionCall 182 | ```json 183 | { 184 | "function_name": "GetFoodMenu", 185 | "arguments": { 186 | "mood": ["Equals", "Happy"] 187 | } 188 | } 189 | ``` 190 | `````` 191 | 192 | ### Empty 193 | It checks if the argument is null or empty: 194 | ``````md 195 | ### CHECK FunctionCall 196 | ```json 197 | { 198 | "function_name": "GetFoodMenu", 199 | "arguments": { 200 | "mood": ["Empty"] 201 | } 202 | } 203 | ``` 204 | `````` 205 | 206 | ### NotEmpty 207 | It checks if the argument is not null or empty: 208 | ``````md 209 | ### CHECK FunctionCall 210 | ```json 211 | { 212 | "function_name": "GetFoodMenu", 213 | "arguments": { 214 | "mood": ["NotEmpty"] 215 | } 216 | } 217 | ``` 218 | `````` 219 | 220 | ### IsAnyOf 221 | It checks whether the argument value is equal to any of the specified values: 222 | ``````md 223 | ### CHECK FunctionCall 224 | ```json 225 | { 226 | "function_name": "GetFoodMenu", 227 | "arguments": { 228 | "mood": ["IsAnyOf", "Happy", "Sad", "Angry"] 229 | } 230 | } 231 | ``` 232 | `````` 233 | 234 | ### ContainsAny 235 | It checks if the argument value contains any of the specified items: 236 | ``````md 237 | ### CHECK FunctionCall 238 | ```json 239 | { 240 | "function_name": "GetFoodMenu", 241 | "arguments": { 242 | "mood": ["ContainsAny", "Happy", "Sad", "Angry"] 243 | } 244 | } 245 | ``` 246 | `````` 247 | 248 | ### ContainsAll 249 | It checks if the argument value contains all of the specified items: 250 | ``````md 251 | ### CHECK FunctionCall 252 | ```json 253 | { 254 | "function_name": "GetFoodMenu", 255 | "arguments": { 256 | "mood": ["ContainsAll", "Happy", "Sad", "Angry"] 257 | } 258 | } 259 | ``` 260 | `````` 261 | 262 | ### SemanticCondition 263 | It checks if the argument value satisfies the condition semantically: 264 | ``````md 265 | ### CHECK FunctionCall 266 | ```json 267 | { 268 | "function_name": "GetFoodMenu", 269 | "arguments": { 270 | "mood": ["SemanticCondition", "Is happy"] 271 | } 272 | } 273 | ``` 274 | `````` 275 | 276 | ### SemanticSimilar 277 | It checks if the argument value is similar in meaning to the given text: 278 | ``````md 279 | ### CHECK FunctionCall 280 | ```json 281 | { 282 | "function_name": "GetFoodMenu", 283 | "arguments": { 284 | "mood": ["SemanticSimilar", "Is happy"] 285 | } 286 | } 287 | ``` 288 | 289 | 290 | `````` -------------------------------------------------------------------------------- /docs/chat-scenario-spec.md: -------------------------------------------------------------------------------- 1 | # Chat Scenario Spec 2 | A chat scenario is a way of testing how SemanticKernel units, such as plugin functions and kernels, respond to user inputs in skUnit. 3 | A chat scenario consists of one or more sub-scenarios, each representing a dialogue turn between the user and the agent. 4 | 5 | There are two types of scenarios: [Invocation Scenario](https://github.com/mehrandvd/skunit/blob/main/docs/invocation-scenario-spec.md) and Chat Scenario. 6 | An invocation scenario is a simpler form of testing a single input-output pair. If you are not familiar with it, please read its documentation first. 7 | 8 | ## Example 9 | This is an example of a chat scenario with two sub-scenarios: 10 | 11 | ```md 12 | # SCENARIO Height Discussion 13 | 14 | ## [USER] 15 | Is Eiffel tall? 16 | 17 | ## [AGENT] 18 | Yes it is 19 | 20 | ### CHECK SemanticCondition 21 | It agrees that the Eiffel Tower is tall or expresses a positive sentiment. 22 | 23 | ## [USER] 24 | What about Everest Mountain? 25 | 26 | ## [AGENT] 27 | Yes it is tall too 28 | 29 | ### CHECK SemanticCondition 30 | It agrees that Everest mountain is tall or expresses a positive sentiment. 31 | ``` 32 | 33 | ![image](https://github.com/mehrandvd/skunit/assets/5070766/156b0831-e4f3-4e4b-b1b0-e2ec868efb5f) 34 | 35 | ### Sub-scenario 1 36 | The first sub-scenario tests how the agent responds to the question `Is Eiffel tall?`. 37 | The expected answer is something like `Yes it is`, but this is not an exact match. It is just a guideline for the desired response. 38 | 39 | When the scenario is executed, the OpenAI generates an actual answer, such as `Yes it is quite tall.`. 40 | The next statement `CHECK SemanticCondition` is an assertion that verifies if the actual answer meets the specified condition: 41 | `It agrees that the Eiffel Tower is tall or expresses a positive sentiment.` 42 | 43 | ### Sub-scenario 2 44 | The second sub-scenario tests how the agent responds to the follow-up question `What about Everest mountain?`. 45 | The expected answer is something like `Yes it is tall too`, but again, this is not an exact match. It is just a guideline for the desired response. 46 | 47 | When the scenario is executed, the OpenAI generates an actual answer, such as `Yes it is very tall indeed.`. 48 | The next statement `CHECK SemanticCondition` is an assertion that verifies if the actual answer meets the specified condition: 49 | `It agrees that Everest mountain is tall or expresses a positive sentiment.` 50 | 51 | As you can see, this sub-scenario does not depend on the exact wording of the previous answer. 52 | It assumes that the agent responded in the expected way and continues the test. 53 | This makes writing long tests easier, as you can rely on the agent's answers to design your test. 54 | Otherwise, you would have to account for different variations of the intermediate answers every time you run the test. 55 | 56 | For a more complex chat scenario, you can refer to this file that is used as a scenario to pass skUnit unit tests: 57 | [Eiffel Tall Chat](https://github.com/mehrandvd/skunit/blob/main/src/skUnit.Tests/SemanticKernelTests/ChatScenarioTests/Samples/EiffelTallChat/skchat.md) 58 | 59 | ```md 60 | # SCENARIO Height Discussion 61 | 62 | ## [USER] 63 | Is Eiffel tall? 64 | 65 | ## [AGENT] 66 | Yes it is 67 | 68 | ### CHECK SemanticCondition 69 | Confirms that the Eiffel tower is tall or expresses positivity. 70 | 71 | ## [USER] 72 | What about Everest Mountain? 73 | 74 | ## [AGENT] 75 | Yes it is tall too 76 | 77 | ### CHECK SemanticCondition 78 | The sentence is positive. 79 | 80 | ## [USER] 81 | What about a mouse? 82 | 83 | ## [AGENT] 84 | No, it is not tall. 85 | 86 | ### CHECK SemanticCondition 87 | The sentence is negative. 88 | 89 | ## [USER] 90 | Give me a JSON containing the Eiffel height. 91 | 92 | Example: 93 | { 94 | "height": "330 meters" 95 | } 96 | 97 | ## [AGENT] 98 | { 99 | "height": "330 meters" 100 | } 101 | 102 | ### CHECK JsonCheck 103 | { 104 | "height": ["NotEmpty", ""] 105 | } 106 | 107 | ### CHECK JsonCheck 108 | { 109 | "height": ["Contain", "meters"] 110 | } 111 | ``` 112 | 113 | ## Test Execution 114 | Here's what you can expect to see in the output when running this test using skUnit as a xUnit `[Fact]`: 115 | 116 | ```csharp 117 | var scenarios = await LoadChatScenarioAsync("EiffelTallChat"); 118 | await SemanticKernelAssert.CheckChatScenarioAsync(Kernel, scenario); 119 | ``` 120 | 121 | The test output will be generated incrementally, line by line: 122 | 123 | ```md 124 | # SCENARIO Height Discussion 125 | 126 | ## [USER] 127 | Is Eiffel tall? 128 | 129 | ## [EXPECTED ANSWER] 130 | Yes it is 131 | 132 | ### [ACTUAL ANSWER] 133 | Yes, the Eiffel Tower in Paris, France, is tall at 330 meters (1,083 feet) in height. 134 | 135 | ### CHECK Condition 136 | Confirms that the Eiffel Tower is tall or expresses positivity. 137 | ✅ OK 138 | 139 | ## [USER] 140 | What about Everest Mountain? 141 | 142 | ## [EXPECTED ANSWER] 143 | Yes it is tall too 144 | 145 | ### [ACTUAL ANSWER] 146 | Yes, Mount Everest is the tallest mountain in the world, with a peak that reaches 29,032 feet (8,849 meters) above sea level. 147 | 148 | ### CHECK Condition 149 | The sentence is positive. 150 | ✅ OK 151 | 152 | ## [USER] 153 | What about a mouse? 154 | 155 | ## [EXPECTED ANSWER] 156 | No, it is not tall. 157 | 158 | ### [ACTUAL ANSWER] 159 | No, a mouse is not tall. 160 | 161 | ### CHECK Condition 162 | The sentence is negative. 163 | ✅ OK 164 | 165 | ## [USER] 166 | Give me a JSON containing the Eiffel height. 167 | Example: 168 | { 169 | "height": "330 meters" 170 | } 171 | 172 | ## [EXPECTED ANSWER] 173 | { 174 | "height": "330 meters" 175 | } 176 | 177 | ### [ACTUAL ANSWER] 178 | { 179 | "height": "330 meters" 180 | } 181 | 182 | ### CHECK JsonCheck 183 | { 184 | "height": ["NotEmpty", ""] 185 | } 186 | ✅ OK 187 | 188 | ### CHECK JsonCheck 189 | { 190 | "height": ["Contain", "meters"] 191 | } 192 | ✅ OK 193 | ``` 194 | 195 | This output is generated line by line as the test is executed: 196 | 197 | ![image](https://github.com/mehrandvd/skunit/assets/5070766/f3ef8a37-ceab-444f-b6f4-098557b61bfa) 198 | 199 | 200 | ## Advanced Scenario Features 201 | 202 | ### Flexible Use of Hashtags 203 | When defining skUnit statements such as `# SCENARIO`, `## [USER]`, and so on, you have the freedom to use as many hashtags as you wish. There's no strict rule that mandates a specific count of hashtags for each statement. This flexibility allows you to format your markdown in a way that enhances readability for you. However, as a best practice, we suggest adhering to the recommended usage to maintain a clear and comprehensible hierarchy. 204 | 205 | ### Assistant Syntax 206 | For better alignment with Microsoft Extensions AI (MEAI) standards, skUnit uses `[ASSISTANT]` as the default syntax for AI responses. The legacy `[AGENT]` syntax is also supported for backward compatibility. Both forms are functionally equivalent and map to the same assistant role: 207 | 208 | ```md 209 | # SCENARIO Using ASSISTANT 210 | 211 | ## [USER] 212 | What is the capital of France? 213 | 214 | ## [ASSISTANT] 215 | The capital of France is Paris. 216 | ``` 217 | 218 | ```md 219 | # SCENARIO Using Legacy AGENT 220 | 221 | ## [USER] 222 | What is the capital of France? 223 | 224 | ## [AGENT] 225 | The capital of France is Paris. 226 | ``` 227 | 228 | You can even mix both forms within the same scenario: 229 | 230 | ```md 231 | # SCENARIO Mixed Usage 232 | 233 | ## [USER] 234 | Hello 235 | 236 | ## [ASSISTANT] 237 | Hi there! 238 | 239 | ## [USER] 240 | How are you? 241 | 242 | ## [AGENT] 243 | I'm doing well, thank you! 244 | ``` 245 | 246 | ### Unique Identifiers 247 | In certain uncommon instances, the data may contain skUnit expressions that could disrupt the parsing of the scenario. For instance, let's consider a scenario with two chat items. If the first chat item contains a markdown value that disrupts parsing, it could pose a problem: 248 | 249 | ```md 250 | # SCENARIO 251 | 252 | ## [USER] 253 | This block itself contains a chat: 254 | ## [USER] 255 | Hello 256 | ## [AGENT] 257 | Hi, How can I help you? 258 | 259 | ## [AGENT] 260 | Wow, this is a chat. 261 | ``` 262 | 263 | To handle these exceptional cases, you can employ an identifier in your statements, like so: 264 | 265 | ```md 266 | # sk SCENARIO 267 | 268 | ## sk [USER] 269 | This block itself contains a chat: 270 | ## [USER] 271 | Hello 272 | ## [AGENT] 273 | Hi, How can I help you? 274 | 275 | ## sk [AGENT] 276 | Wow, this is a chat. 277 | ``` 278 | 279 | In this example, we used `sk` as the identifier. However, you can use any identifier of your choice, such as `~`, `*`, etc. The parser will recognize whatever you use in the first statement as the unique identifier for the statements. 280 | 281 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | ## Ignore Visual Studio temporary files, build results, and 2 | ## files generated by popular Visual Studio add-ons. 3 | ## 4 | ## Get latest from https://github.com/github/gitignore/blob/main/VisualStudio.gitignore 5 | 6 | # User-specific files 7 | *.rsuser 8 | *.suo 9 | *.user 10 | *.userosscache 11 | *.sln.docstates 12 | 13 | # User-specific files (MonoDevelop/Xamarin Studio) 14 | *.userprefs 15 | 16 | # Mono auto generated files 17 | mono_crash.* 18 | 19 | # Build results 20 | [Dd]ebug/ 21 | [Dd]ebugPublic/ 22 | [Rr]elease/ 23 | [Rr]eleases/ 24 | x64/ 25 | x86/ 26 | [Ww][Ii][Nn]32/ 27 | [Aa][Rr][Mm]/ 28 | [Aa][Rr][Mm]64/ 29 | bld/ 30 | [Bb]in/ 31 | [Oo]bj/ 32 | [Ll]og/ 33 | [Ll]ogs/ 34 | 35 | # Visual Studio 2015/2017 cache/options directory 36 | .vs/ 37 | # Uncomment if you have tasks that create the project's static files in wwwroot 38 | #wwwroot/ 39 | 40 | # Visual Studio 2017 auto generated files 41 | Generated\ Files/ 42 | 43 | # MSTest test Results 44 | [Tt]est[Rr]esult*/ 45 | [Bb]uild[Ll]og.* 46 | 47 | # NUnit 48 | *.VisualState.xml 49 | TestResult.xml 50 | nunit-*.xml 51 | 52 | # Build Results of an ATL Project 53 | [Dd]ebugPS/ 54 | [Rr]eleasePS/ 55 | dlldata.c 56 | 57 | # Benchmark Results 58 | BenchmarkDotNet.Artifacts/ 59 | 60 | # .NET Core 61 | project.lock.json 62 | project.fragment.lock.json 63 | artifacts/ 64 | 65 | # ASP.NET Scaffolding 66 | ScaffoldingReadMe.txt 67 | 68 | # StyleCop 69 | StyleCopReport.xml 70 | 71 | # Files built by Visual Studio 72 | *_i.c 73 | *_p.c 74 | *_h.h 75 | *.ilk 76 | *.meta 77 | *.obj 78 | *.iobj 79 | *.pch 80 | *.pdb 81 | *.ipdb 82 | *.pgc 83 | *.pgd 84 | *.rsp 85 | *.sbr 86 | *.tlb 87 | *.tli 88 | *.tlh 89 | *.tmp 90 | *.tmp_proj 91 | *_wpftmp.csproj 92 | *.log 93 | *.tlog 94 | *.vspscc 95 | *.vssscc 96 | .builds 97 | *.pidb 98 | *.svclog 99 | *.scc 100 | 101 | # Chutzpah Test files 102 | _Chutzpah* 103 | 104 | # Visual C++ cache files 105 | ipch/ 106 | *.aps 107 | *.ncb 108 | *.opendb 109 | *.opensdf 110 | *.sdf 111 | *.cachefile 112 | *.VC.db 113 | *.VC.VC.opendb 114 | 115 | # Visual Studio profiler 116 | *.psess 117 | *.vsp 118 | *.vspx 119 | *.sap 120 | 121 | # Visual Studio Trace Files 122 | *.e2e 123 | 124 | # TFS 2012 Local Workspace 125 | $tf/ 126 | 127 | # Guidance Automation Toolkit 128 | *.gpState 129 | 130 | # ReSharper is a .NET coding add-in 131 | _ReSharper*/ 132 | *.[Rr]e[Ss]harper 133 | *.DotSettings.user 134 | 135 | # TeamCity is a build add-in 136 | _TeamCity* 137 | 138 | # DotCover is a Code Coverage Tool 139 | *.dotCover 140 | 141 | # AxoCover is a Code Coverage Tool 142 | .axoCover/* 143 | !.axoCover/settings.json 144 | 145 | # Coverlet is a free, cross platform Code Coverage Tool 146 | coverage*.json 147 | coverage*.xml 148 | coverage*.info 149 | 150 | # Visual Studio code coverage results 151 | *.coverage 152 | *.coveragexml 153 | 154 | # NCrunch 155 | _NCrunch_* 156 | .*crunch*.local.xml 157 | nCrunchTemp_* 158 | 159 | # MightyMoose 160 | *.mm.* 161 | AutoTest.Net/ 162 | 163 | # Web workbench (sass) 164 | .sass-cache/ 165 | 166 | # Installshield output folder 167 | [Ee]xpress/ 168 | 169 | # DocProject is a documentation generator add-in 170 | DocProject/buildhelp/ 171 | DocProject/Help/*.HxT 172 | DocProject/Help/*.HxC 173 | DocProject/Help/*.hhc 174 | DocProject/Help/*.hhk 175 | DocProject/Help/*.hhp 176 | DocProject/Help/Html2 177 | DocProject/Help/html 178 | 179 | # Click-Once directory 180 | publish/ 181 | 182 | # Publish Web Output 183 | *.[Pp]ublish.xml 184 | *.azurePubxml 185 | # Note: Comment the next line if you want to checkin your web deploy settings, 186 | # but database connection strings (with potential passwords) will be unencrypted 187 | *.pubxml 188 | *.publishproj 189 | 190 | # Microsoft Azure Web App publish settings. Comment the next line if you want to 191 | # checkin your Azure Web App publish settings, but sensitive information contained 192 | # in these scripts will be unencrypted 193 | PublishScripts/ 194 | 195 | # NuGet Packages 196 | *.nupkg 197 | # NuGet Symbol Packages 198 | *.snupkg 199 | # The packages folder can be ignored because of Package Restore 200 | **/[Pp]ackages/* 201 | # except build/, which is used as an MSBuild target. 202 | !**/[Pp]ackages/build/ 203 | # Uncomment if necessary however generally it will be regenerated when needed 204 | #!**/[Pp]ackages/repositories.config 205 | # NuGet v3's project.json files produces more ignorable files 206 | *.nuget.props 207 | *.nuget.targets 208 | 209 | # Microsoft Azure Build Output 210 | csx/ 211 | *.build.csdef 212 | 213 | # Microsoft Azure Emulator 214 | ecf/ 215 | rcf/ 216 | 217 | # Windows Store app package directories and files 218 | AppPackages/ 219 | BundleArtifacts/ 220 | Package.StoreAssociation.xml 221 | _pkginfo.txt 222 | *.appx 223 | *.appxbundle 224 | *.appxupload 225 | 226 | # Visual Studio cache files 227 | # files ending in .cache can be ignored 228 | *.[Cc]ache 229 | # but keep track of directories ending in .cache 230 | !?*.[Cc]ache/ 231 | 232 | # Others 233 | ClientBin/ 234 | ~$* 235 | *~ 236 | *.dbmdl 237 | *.dbproj.schemaview 238 | *.jfm 239 | *.pfx 240 | *.publishsettings 241 | orleans.codegen.cs 242 | 243 | # Including strong name files can present a security risk 244 | # (https://github.com/github/gitignore/pull/2483#issue-259490424) 245 | #*.snk 246 | 247 | # Since there are multiple workflows, uncomment next line to ignore bower_components 248 | # (https://github.com/github/gitignore/pull/1529#issuecomment-104372622) 249 | #bower_components/ 250 | 251 | # RIA/Silverlight projects 252 | Generated_Code/ 253 | 254 | # Backup & report files from converting an old project file 255 | # to a newer Visual Studio version. Backup files are not needed, 256 | # because we have git ;-) 257 | _UpgradeReport_Files/ 258 | Backup*/ 259 | UpgradeLog*.XML 260 | UpgradeLog*.htm 261 | ServiceFabricBackup/ 262 | *.rptproj.bak 263 | 264 | # SQL Server files 265 | *.mdf 266 | *.ldf 267 | *.ndf 268 | 269 | # Business Intelligence projects 270 | *.rdl.data 271 | *.bim.layout 272 | *.bim_*.settings 273 | *.rptproj.rsuser 274 | *- [Bb]ackup.rdl 275 | *- [Bb]ackup ([0-9]).rdl 276 | *- [Bb]ackup ([0-9][0-9]).rdl 277 | 278 | # Microsoft Fakes 279 | FakesAssemblies/ 280 | 281 | # GhostDoc plugin setting file 282 | *.GhostDoc.xml 283 | 284 | # Node.js Tools for Visual Studio 285 | .ntvs_analysis.dat 286 | node_modules/ 287 | 288 | # Visual Studio 6 build log 289 | *.plg 290 | 291 | # Visual Studio 6 workspace options file 292 | *.opt 293 | 294 | # Visual Studio 6 auto-generated workspace file (contains which files were open etc.) 295 | *.vbw 296 | 297 | # Visual Studio 6 auto-generated project file (contains which files were open etc.) 298 | *.vbp 299 | 300 | # Visual Studio 6 workspace and project file (working project files containing files to include in project) 301 | *.dsw 302 | *.dsp 303 | 304 | # Visual Studio 6 technical files 305 | *.ncb 306 | *.aps 307 | 308 | # Visual Studio LightSwitch build output 309 | **/*.HTMLClient/GeneratedArtifacts 310 | **/*.DesktopClient/GeneratedArtifacts 311 | **/*.DesktopClient/ModelManifest.xml 312 | **/*.Server/GeneratedArtifacts 313 | **/*.Server/ModelManifest.xml 314 | _Pvt_Extensions 315 | 316 | # Paket dependency manager 317 | .paket/paket.exe 318 | paket-files/ 319 | 320 | # FAKE - F# Make 321 | .fake/ 322 | 323 | # CodeRush personal settings 324 | .cr/personal 325 | 326 | # Python Tools for Visual Studio (PTVS) 327 | __pycache__/ 328 | *.pyc 329 | 330 | # Cake - Uncomment if you are using it 331 | # tools/** 332 | # !tools/packages.config 333 | 334 | # Tabs Studio 335 | *.tss 336 | 337 | # Telerik's JustMock configuration file 338 | *.jmconfig 339 | 340 | # BizTalk build output 341 | *.btp.cs 342 | *.btm.cs 343 | *.odx.cs 344 | *.xsd.cs 345 | 346 | # OpenCover UI analysis results 347 | OpenCover/ 348 | 349 | # Azure Stream Analytics local run output 350 | ASALocalRun/ 351 | 352 | # MSBuild Binary and Structured Log 353 | *.binlog 354 | 355 | # NVidia Nsight GPU debugger configuration file 356 | *.nvuser 357 | 358 | # MFractors (Xamarin productivity tool) working folder 359 | .mfractor/ 360 | 361 | # Local History for Visual Studio 362 | .localhistory/ 363 | 364 | # Visual Studio History (VSHistory) files 365 | .vshistory/ 366 | 367 | # BeatPulse healthcheck temp database 368 | healthchecksdb 369 | 370 | # Backup folder for Package Reference Convert tool in Visual Studio 2017 371 | MigrationBackup/ 372 | 373 | # Ionide (cross platform F# VS Code tools) working folder 374 | .ionide/ 375 | 376 | # Fody - auto-generated XML schema 377 | FodyWeavers.xsd 378 | 379 | # VS Code files for those working on multiple tools 380 | .vscode/* 381 | !.vscode/settings.json 382 | !.vscode/tasks.json 383 | !.vscode/launch.json 384 | !.vscode/extensions.json 385 | *.code-workspace 386 | 387 | # Local History for Visual Studio Code 388 | .history/ 389 | 390 | # Windows Installer files from build outputs 391 | *.cab 392 | *.msi 393 | *.msix 394 | *.msm 395 | *.msp 396 | 397 | # JetBrains Rider 398 | *.sln.iml 399 | -------------------------------------------------------------------------------- /.github/copilot-instructions.md: -------------------------------------------------------------------------------- 1 | # skUnit - Semantic Testing Framework for IChatClient and SemanticKernel 2 | 3 | skUnit is a .NET testing framework for creating semantic tests for IChatClient implementations and SemanticKernel units using markdown scenario files. 4 | 5 | **ALWAYS** reference these instructions first and fallback to search or bash commands only when you encounter unexpected information that does not match the info here. 6 | 7 | ## Working Effectively 8 | 9 | ### Bootstrap, Build, and Test 10 | - **Initial setup**: `dotnet restore src/skUnit.sln` -- takes 16 seconds. NEVER CANCEL. Set timeout to 30+ seconds. 11 | - **Build the solution**: `dotnet build src/skUnit.sln --no-restore` -- takes 12 seconds. NEVER CANCEL. Set timeout to 30+ seconds. 12 | - **Run unit tests (works offline)**: `dotnet test src/skUnit.sln --no-build --filter "FullyQualifiedName~JsonCheckAssertionTests|FullyQualifiedName~FunctionCallAssertionTests|FullyQualifiedName~ParseChatScenarioTests"` -- takes 1 second 13 | - **Full test suite (requires API keys)**: `dotnet test src/skUnit.sln --no-build` -- requires Azure OpenAI configuration 14 | 15 | ### Code Quality 16 | - **Format code**: `dotnet format src/skUnit.sln` -- takes 8 seconds. NEVER CANCEL. Set timeout to 30+ seconds. 17 | - **Verify formatting**: `dotnet format src/skUnit.sln --verify-no-changes` -- fails if formatting needed 18 | - **ALWAYS** run `dotnet format src/skUnit.sln` before committing changes or CI will have formatting issues 19 | 20 | ### Demo Projects 21 | - **TDD REPL Demo**: Located in `demos/Demo.TddRepl/` 22 | - Build: `cd demos/Demo.TddRepl && dotnet restore Demo.TddRepl.sln && dotnet build Demo.TddRepl.sln --no-restore` 23 | - Shows how to use skUnit for test-driven development of chat applications 24 | - **MCP Demo**: Located in `demos/Demo.TddMcp/` 25 | - Shows integration with Model Context Protocol tools 26 | - **Shop Demo**: Located in `demos/Demo.TddShop/` 27 | - Complex multi-scenario chat testing 28 | 29 | ## Requirements and Configuration 30 | 31 | ### System Requirements 32 | - **.NET 8.0** (verified working with 8.0.118) 33 | - **Azure OpenAI API access** (for integration tests and semantic validation) 34 | 35 | ### Required Environment Variables for Full Testing 36 | Set these in your environment or user secrets for integration tests: 37 | ```bash 38 | AzureOpenAI_ApiKey=your-api-key 39 | AzureOpenAI_Endpoint=https://your-endpoint.openai.azure.com/ 40 | AzureOpenAI_Deployment=your-deployment-name 41 | Smithery_Key=your-smithery-key # For MCP tests only 42 | ``` 43 | 44 | ### Configuration Files 45 | - `src/skUnit.Tests/skUnit.Tests.csproj` has `8f5163d1-e8e8-4a8e-9186-b473280a19b4` 46 | - Use `dotnet user-secrets set "AzureOpenAI_ApiKey" "your-key" --project src/skUnit.Tests` to configure secrets 47 | 48 | ## Validation and Testing 49 | 50 | ### ALWAYS run these validation steps after making changes: 51 | 1. **Format code**: `dotnet format src/skUnit.sln` 52 | 2. **Build**: `dotnet build src/skUnit.sln --no-restore` 53 | 3. **Unit tests**: `dotnet test src/skUnit.sln --no-build --filter "FullyQualifiedName~JsonCheckAssertionTests|FullyQualifiedName~FunctionCallAssertionTests|FullyQualifiedName~ParseChatScenarioTests"` 54 | 4. **If you have API keys configured**: `dotnet test src/skUnit.sln --no-build` 55 | 56 | ### Manual Validation Scenarios 57 | After making changes to scenario parsing or assertion logic: 58 | 1. **Test scenario parsing**: Create a test .md file and verify it parses correctly 59 | 2. **Test specific assertion types**: Run individual assertion tests for the type you modified 60 | 3. **Integration test**: If you have API keys, run a complete chat scenario test 61 | 62 | Example validation scenario for chat testing: 63 | ```markdown 64 | # SCENARIO Simple Test 65 | 66 | ## [USER] 67 | Hello 68 | 69 | ## [AGENT] 70 | Hi there! 71 | 72 | ### CHECK SemanticCondition 73 | It's a greeting response 74 | ``` 75 | 76 | ## Key Project Structure 77 | 78 | ### Main Source Code (`src/skUnit/`) 79 | - **`Asserts/`**: Core assertion logic including `ScenarioAssert` class 80 | - **`Scenarios/`**: Scenario parsing and execution logic 81 | - **`Parsers/`**: Markdown parsing for scenarios and assertions 82 | - **`Parsers/Assertions/`**: Individual assertion type implementations 83 | - **`Exceptions/`**: Custom exception types for semantic testing 84 | 85 | ### Test Projects (`src/skUnit.Tests/`) 86 | - **`AssertionTests/`**: Unit tests for individual assertion types (work offline) 87 | - **`ScenarioAssertTests/`**: Integration tests requiring API keys 88 | - **`ScenarioParseTests/`**: Tests for markdown parsing logic (work offline) 89 | - **`Infrastructure/`**: Base test classes and configuration 90 | - **`Samples/`**: Embedded test scenario .md files 91 | 92 | ### Frequently Used Commands Output 93 | 94 | #### Repository Root Structure 95 | ``` 96 | .github/ # GitHub workflows and configuration 97 | .gitignore 98 | LICENSE 99 | README.md 100 | demos/ # Example projects showing skUnit usage 101 | docs/ # Documentation for scenarios and CHECK statements 102 | src/ # Main source code and tests 103 | ``` 104 | 105 | #### Main Solution Structure 106 | ``` 107 | src/ 108 | ├── skUnit.sln # Main solution file 109 | ├── skUnit/ # Core library project 110 | │ ├── skUnit.csproj # .NET 8.0, NuGet package config 111 | │ ├── Asserts/ # Core assertion and testing logic 112 | │ ├── Scenarios/ # Scenario parsing and execution 113 | │ └── Exceptions/ # Custom exceptions 114 | └── skUnit.Tests/ # Test project 115 | ├── skUnit.Tests.csproj # Test project with xUnit 116 | ├── AssertionTests/ # Unit tests (work offline) 117 | ├── ScenarioAssertTests/ # Integration tests (need API keys) 118 | └── Infrastructure/ # Test infrastructure 119 | ``` 120 | 121 | ## Common Tasks and Workflows 122 | 123 | ### Adding New Assertion Types 124 | 1. Create new assertion class in `src/skUnit/Scenarios/Parsers/Assertions/` 125 | 2. Implement `IKernelAssertion` interface 126 | 3. Add parsing logic to `KernelAssertionParser.cs` 127 | 4. Create unit tests in `src/skUnit.Tests/AssertionTests/` 128 | 5. Add integration tests in scenario .md files 129 | 130 | ### Working with Scenario Files 131 | - **Location**: Test scenarios in `src/skUnit.Tests/ScenarioAssertTests/Samples/` 132 | - **Format**: Valid Markdown with special headers like `## [USER]`, `## [AGENT]`, `### CHECK` 133 | - **Embedded Resources**: Test scenarios are embedded in the test assembly 134 | 135 | ### Working with CHECK Statements 136 | Available assertion types: 137 | - `CHECK Equals` - Exact string match 138 | - `CHECK ContainsAll` - Contains all specified words 139 | - `CHECK ContainsAny` - Contains any of specified words 140 | - `CHECK SemanticCondition` - AI-powered semantic validation 141 | - `CHECK SemanticSimilar` - AI-powered similarity check 142 | - `CHECK JsonCheck` - JSON structure and content validation 143 | - `CHECK FunctionCall` - Validate function calls in chat responses 144 | 145 | ### Integration with CI/CD 146 | - **GitHub Actions**: `.github/workflows/build.yml` runs build only 147 | - **Test workflow**: `.github/workflows/test.yml` runs with secrets (manual trigger) 148 | - **Formatting**: Always run `dotnet format` before commit to avoid CI failures 149 | 150 | ## Timing Expectations and Timeouts 151 | 152 | ### Build Commands (NEVER CANCEL) 153 | - `dotnet restore`: 16 seconds - Set timeout to 30+ seconds 154 | - `dotnet build`: 12 seconds - Set timeout to 30+ seconds 155 | - `dotnet format`: 8 seconds - Set timeout to 30+ seconds 156 | 157 | ### Test Commands 158 | - Unit tests (offline): 1 second - Set timeout to 10+ seconds 159 | - Full test suite (with API): Variable (depends on OpenAI response time) - Set timeout to 120+ seconds 160 | - Individual integration test: 5-30 seconds - Set timeout to 60+ seconds 161 | 162 | ### Demo Project Build 163 | - Demo restore + build: 12-14 seconds total - Set timeout to 30+ seconds 164 | 165 | ## Troubleshooting Common Issues 166 | 167 | ### Build Warnings 168 | The project has some nullable reference warnings that are expected and don't affect functionality: 169 | - CS8604, CS8602, CS8625 warnings in assertion and test code 170 | - These warnings should not block builds or prevent functionality 171 | 172 | ### Test Failures 173 | - **"No ApiKey is provided"**: Configure Azure OpenAI credentials for integration tests 174 | - **Formatting errors**: Run `dotnet format src/skUnit.sln` to fix 175 | - **Parse errors in scenarios**: Check markdown format matches expected headers and CHECK statement syntax 176 | 177 | ### Demo Project Issues 178 | - Demos require same Azure OpenAI configuration as main test project 179 | - Some demos may require additional secrets (like Smithery_Key for MCP demo) 180 | 181 | ## Important Notes for Development 182 | 183 | - **Target Framework**: .NET 8.0 (net8.0) - do not change without careful consideration 184 | - **Package Dependencies**: Core dependencies are Microsoft.Extensions.AI, SemanticKernel.Abstractions, Markdig 185 | - **Test Framework**: xUnit with test output helpers for scenario execution logging 186 | - **NuGet Package**: This project publishes to NuGet as "skUnit" package 187 | - **Markdown Processing**: Uses Markdig for parsing scenario files 188 | - **Semantic Validation**: Powered by SemanticValidation package for AI-based assertions --------------------------------------------------------------------------------