├── .github └── fabricbot.json ├── .gitignore ├── EventHubsExtension.sln ├── LICENSE.txt ├── NuGet.Config ├── README.md ├── SECURITY.md ├── release_notes.md ├── sign.snk ├── src.ruleset ├── src └── Microsoft.Azure.WebJobs.Extensions.EventHubs │ ├── Config │ ├── EventHubExtensionConfigProvider.cs │ ├── EventHubOptions.cs │ ├── EventHubWebJobsBuilderExtensions.cs │ └── InitialOffsetOptions.cs │ ├── EventHubAttribute.cs │ ├── EventHubTriggerAttribute.cs │ ├── EventHubsWebJobsStartup.cs │ ├── Extensions │ └── SystemPropertiesCollectionExtensions.cs │ ├── Listeners │ ├── EventHubListener.cs │ ├── EventHubsScaleMonitor.cs │ └── EventHubsTriggerMetrics.cs │ ├── Properties │ └── AssemblyInfo.cs │ ├── Triggers │ ├── EventHubAsyncCollector.cs │ ├── EventHubTriggerAttributeBindingProvider.cs │ ├── EventHubTriggerInput.cs │ └── EventHubTriggerInputBindingStrategy.cs │ ├── Utility.cs │ ├── WebJobs.Extensions.EventHubs.csproj │ └── webjobs.png ├── stylecop.json └── test ├── Microsoft.Azure.WebJobs.Extensions.EventHubs.Tests ├── EventHubApplicationInsightsTest.cs ├── EventHubAsyncCollectorTests.cs ├── EventHubConfigurationTests.cs ├── EventHubEndToEndTests.cs ├── EventHubListenerTests.cs ├── EventHubTests.cs ├── EventHubsScaleMonitorTests.cs ├── Properties │ └── AssemblyInfo.cs ├── PublicSurfaceTests.cs └── WebJobs.Extensions.EventHubs.Tests.csproj └── README.md /.gitignore: -------------------------------------------------------------------------------- 1 | ## Ignore Visual Studio temporary files, build results, and 2 | ## files generated by popular Visual Studio add-ons. 3 | ## 4 | ## Get latest from https://github.com/github/gitignore/blob/master/VisualStudio.gitignore 5 | 6 | # User-specific files 7 | *.suo 8 | *.user 9 | *.userosscache 10 | *.sln.docstates 11 | 12 | # User-specific files (MonoDevelop/Xamarin Studio) 13 | *.userprefs 14 | 15 | # Build results 16 | [Dd]ebug/ 17 | [Dd]ebugPublic/ 18 | [Rr]elease/ 19 | [Rr]eleases/ 20 | x64/ 21 | x86/ 22 | bld/ 23 | [Bb]in/ 24 | [Oo]bj/ 25 | [Ll]og/ 26 | 27 | # Visual Studio 2015/2017 cache/options directory 28 | .vs/ 29 | # Uncomment if you have tasks that create the project's static files in wwwroot 30 | #wwwroot/ 31 | 32 | # Visual Studio 2017 auto generated files 33 | Generated\ Files/ 34 | 35 | # MSTest test Results 36 | [Tt]est[Rr]esult*/ 37 | [Bb]uild[Ll]og.* 38 | 39 | # NUNIT 40 | *.VisualState.xml 41 | TestResult.xml 42 | 43 | # Build Results of an ATL Project 44 | [Dd]ebugPS/ 45 | [Rr]eleasePS/ 46 | dlldata.c 47 | 48 | # Benchmark Results 49 | BenchmarkDotNet.Artifacts/ 50 | 51 | # .NET Core 52 | project.lock.json 53 | project.fragment.lock.json 54 | artifacts/ 55 | **/Properties/launchSettings.json 56 | 57 | # StyleCop 58 | StyleCopReport.xml 59 | 60 | # Files built by Visual Studio 61 | *_i.c 62 | *_p.c 63 | *_i.h 64 | *.ilk 65 | *.meta 66 | *.obj 67 | *.iobj 68 | *.pch 69 | *.pdb 70 | *.ipdb 71 | *.pgc 72 | *.pgd 73 | *.rsp 74 | *.sbr 75 | *.tlb 76 | *.tli 77 | *.tlh 78 | *.tmp 79 | *.tmp_proj 80 | *.log 81 | *.vspscc 82 | *.vssscc 83 | .builds 84 | *.pidb 85 | *.svclog 86 | *.scc 87 | 88 | # Chutzpah Test files 89 | _Chutzpah* 90 | 91 | # Visual C++ cache files 92 | ipch/ 93 | *.aps 94 | *.ncb 95 | *.opendb 96 | *.opensdf 97 | *.sdf 98 | *.cachefile 99 | *.VC.db 100 | *.VC.VC.opendb 101 | 102 | # Visual Studio profiler 103 | *.psess 104 | *.vsp 105 | *.vspx 106 | *.sap 107 | 108 | # Visual Studio Trace Files 109 | *.e2e 110 | 111 | # TFS 2012 Local Workspace 112 | $tf/ 113 | 114 | # Guidance Automation Toolkit 115 | *.gpState 116 | 117 | # ReSharper is a .NET coding add-in 118 | _ReSharper*/ 119 | *.[Rr]e[Ss]harper 120 | *.DotSettings.user 121 | 122 | # JustCode is a .NET coding add-in 123 | .JustCode 124 | 125 | # TeamCity is a build add-in 126 | _TeamCity* 127 | 128 | # DotCover is a Code Coverage Tool 129 | *.dotCover 130 | 131 | # AxoCover is a Code Coverage Tool 132 | .axoCover/* 133 | !.axoCover/settings.json 134 | 135 | # Visual Studio code coverage results 136 | *.coverage 137 | *.coveragexml 138 | 139 | # NCrunch 140 | _NCrunch_* 141 | .*crunch*.local.xml 142 | nCrunchTemp_* 143 | 144 | # MightyMoose 145 | *.mm.* 146 | AutoTest.Net/ 147 | 148 | # Web workbench (sass) 149 | .sass-cache/ 150 | 151 | # Installshield output folder 152 | [Ee]xpress/ 153 | 154 | # DocProject is a documentation generator add-in 155 | DocProject/buildhelp/ 156 | DocProject/Help/*.HxT 157 | DocProject/Help/*.HxC 158 | DocProject/Help/*.hhc 159 | DocProject/Help/*.hhk 160 | DocProject/Help/*.hhp 161 | DocProject/Help/Html2 162 | DocProject/Help/html 163 | 164 | # Click-Once directory 165 | publish/ 166 | 167 | # Publish Web Output 168 | *.[Pp]ublish.xml 169 | *.azurePubxml 170 | # Note: Comment the next line if you want to checkin your web deploy settings, 171 | # but database connection strings (with potential passwords) will be unencrypted 172 | *.pubxml 173 | *.publishproj 174 | 175 | # Microsoft Azure Web App publish settings. Comment the next line if you want to 176 | # checkin your Azure Web App publish settings, but sensitive information contained 177 | # in these scripts will be unencrypted 178 | PublishScripts/ 179 | 180 | # NuGet Packages 181 | *.nupkg 182 | # The packages folder can be ignored because of Package Restore 183 | **/[Pp]ackages/* 184 | # except build/, which is used as an MSBuild target. 185 | !**/[Pp]ackages/build/ 186 | # Uncomment if necessary however generally it will be regenerated when needed 187 | #!**/[Pp]ackages/repositories.config 188 | # NuGet v3's project.json files produces more ignorable files 189 | *.nuget.props 190 | *.nuget.targets 191 | 192 | # Microsoft Azure Build Output 193 | csx/ 194 | *.build.csdef 195 | 196 | # Microsoft Azure Emulator 197 | ecf/ 198 | rcf/ 199 | 200 | # Windows Store app package directories and files 201 | AppPackages/ 202 | BundleArtifacts/ 203 | Package.StoreAssociation.xml 204 | _pkginfo.txt 205 | *.appx 206 | 207 | # Visual Studio cache files 208 | # files ending in .cache can be ignored 209 | *.[Cc]ache 210 | # but keep track of directories ending in .cache 211 | !*.[Cc]ache/ 212 | 213 | # Others 214 | ClientBin/ 215 | ~$* 216 | *~ 217 | *.dbmdl 218 | *.dbproj.schemaview 219 | *.jfm 220 | *.pfx 221 | *.publishsettings 222 | orleans.codegen.cs 223 | 224 | # Including strong name files can present a security risk 225 | # (https://github.com/github/gitignore/pull/2483#issue-259490424) 226 | #*.snk 227 | 228 | # Since there are multiple workflows, uncomment next line to ignore bower_components 229 | # (https://github.com/github/gitignore/pull/1529#issuecomment-104372622) 230 | #bower_components/ 231 | 232 | # RIA/Silverlight projects 233 | Generated_Code/ 234 | 235 | # Backup & report files from converting an old project file 236 | # to a newer Visual Studio version. Backup files are not needed, 237 | # because we have git ;-) 238 | _UpgradeReport_Files/ 239 | Backup*/ 240 | UpgradeLog*.XML 241 | UpgradeLog*.htm 242 | ServiceFabricBackup/ 243 | *.rptproj.bak 244 | 245 | # SQL Server files 246 | *.mdf 247 | *.ldf 248 | *.ndf 249 | 250 | # Business Intelligence projects 251 | *.rdl.data 252 | *.bim.layout 253 | *.bim_*.settings 254 | *.rptproj.rsuser 255 | 256 | # Microsoft Fakes 257 | FakesAssemblies/ 258 | 259 | # GhostDoc plugin setting file 260 | *.GhostDoc.xml 261 | 262 | # Node.js Tools for Visual Studio 263 | .ntvs_analysis.dat 264 | node_modules/ 265 | 266 | # Visual Studio 6 build log 267 | *.plg 268 | 269 | # Visual Studio 6 workspace options file 270 | *.opt 271 | 272 | # Visual Studio 6 auto-generated workspace file (contains which files were open etc.) 273 | *.vbw 274 | 275 | # Visual Studio LightSwitch build output 276 | **/*.HTMLClient/GeneratedArtifacts 277 | **/*.DesktopClient/GeneratedArtifacts 278 | **/*.DesktopClient/ModelManifest.xml 279 | **/*.Server/GeneratedArtifacts 280 | **/*.Server/ModelManifest.xml 281 | _Pvt_Extensions 282 | 283 | # Paket dependency manager 284 | .paket/paket.exe 285 | paket-files/ 286 | 287 | # FAKE - F# Make 288 | .fake/ 289 | 290 | # JetBrains Rider 291 | .idea/ 292 | *.sln.iml 293 | 294 | # CodeRush 295 | .cr/ 296 | 297 | # Python Tools for Visual Studio (PTVS) 298 | __pycache__/ 299 | *.pyc 300 | 301 | # Cake - Uncomment if you are using it 302 | # tools/** 303 | # !tools/packages.config 304 | 305 | # Tabs Studio 306 | *.tss 307 | 308 | # Telerik's JustMock configuration file 309 | *.jmconfig 310 | 311 | # BizTalk build output 312 | *.btp.cs 313 | *.btm.cs 314 | *.odx.cs 315 | *.xsd.cs 316 | 317 | # OpenCover UI analysis results 318 | OpenCover/ 319 | 320 | # Azure Stream Analytics local run output 321 | ASALocalRun/ 322 | 323 | # MSBuild Binary and Structured Log 324 | *.binlog 325 | 326 | # NVidia Nsight GPU debugger configuration file 327 | *.nvuser 328 | 329 | # MFractors (Xamarin productivity tool) working folder 330 | .mfractor/ -------------------------------------------------------------------------------- /EventHubsExtension.sln: -------------------------------------------------------------------------------- 1 |  2 | Microsoft Visual Studio Solution File, Format Version 12.00 3 | # Visual Studio 15 4 | VisualStudioVersion = 15.0.27130.2024 5 | MinimumVisualStudioVersion = 10.0.40219.1 6 | Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "WebJobs.Extensions.EventHubs", "src\Microsoft.Azure.WebJobs.Extensions.EventHubs\WebJobs.Extensions.EventHubs.csproj", "{A2B3C676-3DF0-43B4-92A2-7E7DAA7BF439}" 7 | EndProject 8 | Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "WebJobs.Extensions.EventHubs.Tests", "test\Microsoft.Azure.WebJobs.Extensions.EventHubs.Tests\WebJobs.Extensions.EventHubs.Tests.csproj", "{D37637D5-7EF9-43CB-86BE-537473CD613B}" 9 | EndProject 10 | Global 11 | GlobalSection(SolutionConfigurationPlatforms) = preSolution 12 | Debug|Any CPU = Debug|Any CPU 13 | Release|Any CPU = Release|Any CPU 14 | EndGlobalSection 15 | GlobalSection(ProjectConfigurationPlatforms) = postSolution 16 | {A2B3C676-3DF0-43B4-92A2-7E7DAA7BF439}.Debug|Any CPU.ActiveCfg = Debug|Any CPU 17 | {A2B3C676-3DF0-43B4-92A2-7E7DAA7BF439}.Debug|Any CPU.Build.0 = Debug|Any CPU 18 | {A2B3C676-3DF0-43B4-92A2-7E7DAA7BF439}.Release|Any CPU.ActiveCfg = Release|Any CPU 19 | {A2B3C676-3DF0-43B4-92A2-7E7DAA7BF439}.Release|Any CPU.Build.0 = Release|Any CPU 20 | {D37637D5-7EF9-43CB-86BE-537473CD613B}.Debug|Any CPU.ActiveCfg = Debug|Any CPU 21 | {D37637D5-7EF9-43CB-86BE-537473CD613B}.Debug|Any CPU.Build.0 = Debug|Any CPU 22 | {D37637D5-7EF9-43CB-86BE-537473CD613B}.Release|Any CPU.ActiveCfg = Release|Any CPU 23 | {D37637D5-7EF9-43CB-86BE-537473CD613B}.Release|Any CPU.Build.0 = Release|Any CPU 24 | EndGlobalSection 25 | GlobalSection(SolutionProperties) = preSolution 26 | HideSolutionNode = FALSE 27 | EndGlobalSection 28 | GlobalSection(ExtensibilityGlobals) = postSolution 29 | SolutionGuid = {43B8428A-8A26-45B1-A0D9-D33AD1F054CE} 30 | EndGlobalSection 31 | EndGlobal 32 | -------------------------------------------------------------------------------- /LICENSE.txt: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) .NET Foundation. All rights reserved. 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /NuGet.Config: -------------------------------------------------------------------------------- 1 |  2 | 3 | 4 | 5 | 6 | 7 | 8 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Event Hubs Extension for Azure Functions [Archived] 2 | 3 | This GitHub project has been archived. Ongoing development on this project can be found in https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/eventhub/Microsoft.Azure.WebJobs.Extensions.EventHubs. 4 | 5 | This extension provides functionality for receiving Event Hubs messges in Azure Functions, allowing you to easily write functions that respond to any message published to Event Hubs. 6 | 7 | |Branch|Status| 8 | |---|---| 9 | |dev|[![Build Status](https://azfunc.visualstudio.com/Azure%20Functions/_apis/build/status/azure-functions-eventhubs-extension-ci?branchName=dev)](https://azfunc.visualstudio.com/Azure%20Functions/_build/latest?definitionId=18&branchName=dev) 10 | 11 | ## License 12 | 13 | This project is under the benevolent umbrella of the [.NET Foundation](http://www.dotnetfoundation.org/) and is licensed under [the MIT License](https://github.com/Azure/azure-webjobs-sdk/blob/master/LICENSE.txt) 14 | 15 | ## Contributing 16 | 17 | This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments. 18 | -------------------------------------------------------------------------------- /SECURITY.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | ## Security 4 | 5 | Microsoft takes the security of our software products and services seriously, which includes all source code repositories managed through our GitHub organizations, which include [Microsoft](https://github.com/Microsoft), [Azure](https://github.com/Azure), [DotNet](https://github.com/dotnet), [AspNet](https://github.com/aspnet), [Xamarin](https://github.com/xamarin), and [our GitHub organizations](https://opensource.microsoft.com/). 6 | 7 | If you believe you have found a security vulnerability in any Microsoft-owned repository that meets [Microsoft's definition of a security vulnerability](https://aka.ms/opensource/security/definition), please report it to us as described below. 8 | 9 | ## Reporting Security Issues 10 | 11 | **Please do not report security vulnerabilities through public GitHub issues.** 12 | 13 | Instead, please report them to the Microsoft Security Response Center (MSRC) at [https://msrc.microsoft.com/create-report](https://aka.ms/opensource/security/create-report). 14 | 15 | If you prefer to submit without logging in, send email to [secure@microsoft.com](mailto:secure@microsoft.com). If possible, encrypt your message with our PGP key; please download it from the [Microsoft Security Response Center PGP Key page](https://aka.ms/opensource/security/pgpkey). 16 | 17 | You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Additional information can be found at [microsoft.com/msrc](https://aka.ms/opensource/security/msrc). 18 | 19 | Please include the requested information listed below (as much as you can provide) to help us better understand the nature and scope of the possible issue: 20 | 21 | * Type of issue (e.g. buffer overflow, SQL injection, cross-site scripting, etc.) 22 | * Full paths of source file(s) related to the manifestation of the issue 23 | * The location of the affected source code (tag/branch/commit or direct URL) 24 | * Any special configuration required to reproduce the issue 25 | * Step-by-step instructions to reproduce the issue 26 | * Proof-of-concept or exploit code (if possible) 27 | * Impact of the issue, including how an attacker might exploit the issue 28 | 29 | This information will help us triage your report more quickly. 30 | 31 | If you are reporting for a bug bounty, more complete reports can contribute to a higher bounty award. Please visit our [Microsoft Bug Bounty Program](https://aka.ms/opensource/security/bounty) page for more details about our active programs. 32 | 33 | ## Preferred Languages 34 | 35 | We prefer all communications to be in English. 36 | 37 | ## Policy 38 | 39 | Microsoft follows the principle of [Coordinated Vulnerability Disclosure](https://aka.ms/opensource/security/cvd). 40 | 41 | 42 | -------------------------------------------------------------------------------- /release_notes.md: -------------------------------------------------------------------------------- 1 | ### Release notes 2 | 5 | #### Version 4.2.0 6 | - User configurable initial offset support [#79](https://github.com/Azure/azure-functions-eventhubs-extension/pull/79) 7 | 8 | #### Version 4.3.0 9 | - Adding explicit reference Microsoft.Azure.Amqp 2.4.11 [#99](https://github.com/Azure/azure-functions-eventhubs-extension/pull/99) 10 | 11 | #### Version 4.3.1 12 | - Stop Listener when disposed [#105](https://github.com/Azure/azure-functions-eventhubs-extension/pull/105) 13 | - Add listener details [#105](https://github.com/Azure/azure-functions-eventhubs-extension/pull/105) 14 | -------------------------------------------------------------------------------- /sign.snk: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/azure-functions-eventhubs-extension/1cf1006f731f9015d5e1eadb4e6798b63d78313c/sign.snk -------------------------------------------------------------------------------- /src.ruleset: -------------------------------------------------------------------------------- 1 |  2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 | 85 | 86 | 87 | 88 | 89 | 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | 98 | 99 | 100 | 101 | 102 | 103 | 104 | 105 | 106 | 107 | 108 | 109 | 110 | 111 | 112 | 113 | 114 | 115 | 116 | 117 | 118 | 119 | 120 | 121 | 122 | 123 | 124 | 125 | 126 | 127 | -------------------------------------------------------------------------------- /src/Microsoft.Azure.WebJobs.Extensions.EventHubs/Config/EventHubExtensionConfigProvider.cs: -------------------------------------------------------------------------------- 1 | // Copyright (c) .NET Foundation. All rights reserved. 2 | // Licensed under the MIT License. See License.txt in the project root for license information. 3 | 4 | using System; 5 | using System.Text; 6 | using System.Threading.Tasks; 7 | using Microsoft.Azure.EventHubs; 8 | using Microsoft.Azure.EventHubs.Processor; 9 | using Microsoft.Azure.WebJobs.Description; 10 | using Microsoft.Azure.WebJobs.Host.Bindings; 11 | using Microsoft.Azure.WebJobs.Host.Config; 12 | using Microsoft.Azure.WebJobs.Host.Configuration; 13 | using Microsoft.Azure.WebJobs.Logging; 14 | using Microsoft.Extensions.Configuration; 15 | using Microsoft.Extensions.Logging; 16 | using Microsoft.Extensions.Options; 17 | using Newtonsoft.Json; 18 | 19 | namespace Microsoft.Azure.WebJobs.EventHubs 20 | { 21 | [Extension("EventHubs", configurationSection: "EventHubs")] 22 | internal class EventHubExtensionConfigProvider : IExtensionConfigProvider 23 | { 24 | public IConfiguration _config; 25 | private readonly IOptions _options; 26 | private readonly ILoggerFactory _loggerFactory; 27 | private readonly IConverterManager _converterManager; 28 | private readonly INameResolver _nameResolver; 29 | private readonly IWebJobsExtensionConfiguration _configuration; 30 | 31 | public EventHubExtensionConfigProvider(IConfiguration config, IOptions options, ILoggerFactory loggerFactory, 32 | IConverterManager converterManager, INameResolver nameResolver, IWebJobsExtensionConfiguration configuration) 33 | { 34 | _config = config; 35 | _options = options; 36 | _loggerFactory = loggerFactory; 37 | _converterManager = converterManager; 38 | _nameResolver = nameResolver; 39 | _configuration = configuration; 40 | } 41 | 42 | internal Action ExceptionHandler { get; set; } 43 | 44 | private void ExceptionReceivedHandler(ExceptionReceivedEventArgs args) 45 | { 46 | ExceptionHandler?.Invoke(args); 47 | } 48 | 49 | public void Initialize(ExtensionConfigContext context) 50 | { 51 | if (context == null) 52 | { 53 | throw new ArgumentNullException(nameof(context)); 54 | } 55 | 56 | _options.Value.EventProcessorOptions.SetExceptionHandler(ExceptionReceivedHandler); 57 | _configuration.ConfigurationSection.Bind(_options); 58 | 59 | context 60 | .AddConverter(ConvertString2EventData) 61 | .AddConverter(ConvertEventData2String) 62 | .AddConverter(ConvertBytes2EventData) 63 | .AddConverter(ConvertEventData2Bytes) 64 | .AddOpenConverter(ConvertPocoToEventData); 65 | 66 | // register our trigger binding provider 67 | var triggerBindingProvider = new EventHubTriggerAttributeBindingProvider(_config, _nameResolver, _converterManager, _options, _loggerFactory); 68 | context.AddBindingRule() 69 | .BindToTrigger(triggerBindingProvider); 70 | 71 | // register our binding provider 72 | context.AddBindingRule() 73 | .BindToCollector(BuildFromAttribute); 74 | 75 | context.AddBindingRule() 76 | .BindToInput(attribute => 77 | { 78 | return _options.Value.GetEventHubClient(attribute.EventHubName, attribute.Connection); 79 | }); 80 | 81 | ExceptionHandler = (e => 82 | { 83 | LogExceptionReceivedEvent(e, _loggerFactory); 84 | }); 85 | } 86 | 87 | internal static void LogExceptionReceivedEvent(ExceptionReceivedEventArgs e, ILoggerFactory loggerFactory) 88 | { 89 | var logger = loggerFactory?.CreateLogger(LogCategories.Executor); 90 | string message = $"EventProcessorHost error (Action='{e.Action}', HostName='{e.Hostname}', PartitionId='{e.PartitionId}')."; 91 | 92 | Utility.LogException(e.Exception, message, logger); 93 | } 94 | 95 | private static LogLevel GetLogLevel(Exception ex) 96 | { 97 | if (ex is ReceiverDisconnectedException || 98 | ex is LeaseLostException) 99 | { 100 | // For EventProcessorHost these exceptions can happen as part 101 | // of normal partition balancing across instances, so we want to 102 | // trace them, but not treat them as errors. 103 | return LogLevel.Information; 104 | } 105 | 106 | var ehex = ex as EventHubsException; 107 | if (!(ex is OperationCanceledException) && (ehex == null || !ehex.IsTransient)) 108 | { 109 | // any non-transient exceptions or unknown exception types 110 | // we want to log as errors 111 | return LogLevel.Error; 112 | } 113 | else 114 | { 115 | // transient messaging errors we log as info so we have a record 116 | // of them, but we don't treat them as actual errors 117 | return LogLevel.Information; 118 | } 119 | } 120 | 121 | private IAsyncCollector BuildFromAttribute(EventHubAttribute attribute) 122 | { 123 | EventHubClient client = _options.Value.GetEventHubClient(attribute.EventHubName, attribute.Connection); 124 | return new EventHubAsyncCollector(client, _loggerFactory); 125 | } 126 | 127 | private static string ConvertEventData2String(EventData x) 128 | => Encoding.UTF8.GetString(ConvertEventData2Bytes(x)); 129 | 130 | private static EventData ConvertBytes2EventData(byte[] input) 131 | => new EventData(input); 132 | 133 | private static byte[] ConvertEventData2Bytes(EventData input) 134 | => input.Body.Array; 135 | 136 | private static EventData ConvertString2EventData(string input) 137 | => ConvertBytes2EventData(Encoding.UTF8.GetBytes(input)); 138 | 139 | private static Task ConvertPocoToEventData(object arg, Attribute attrResolved, ValueBindingContext context) 140 | { 141 | return Task.FromResult(ConvertString2EventData(JsonConvert.SerializeObject(arg))); 142 | } 143 | } 144 | } 145 | -------------------------------------------------------------------------------- /src/Microsoft.Azure.WebJobs.Extensions.EventHubs/Config/EventHubOptions.cs: -------------------------------------------------------------------------------- 1 | // Copyright (c) .NET Foundation. All rights reserved. 2 | // Licensed under the MIT License. See License.txt in the project root for license information. 3 | 4 | using System; 5 | using System.Collections.Concurrent; 6 | using System.Collections.Generic; 7 | using System.Globalization; 8 | using System.Text; 9 | using Microsoft.Azure.EventHubs; 10 | using Microsoft.Azure.EventHubs.Processor; 11 | using Microsoft.Azure.WebJobs.Hosting; 12 | using Microsoft.Extensions.Configuration; 13 | using Newtonsoft.Json; 14 | using Newtonsoft.Json.Linq; 15 | 16 | namespace Microsoft.Azure.WebJobs.EventHubs 17 | { 18 | public class EventHubOptions : IOptionsFormatter 19 | { 20 | // Event Hub Names are case-insensitive. 21 | // The same path can have multiple connection strings with different permissions (sending and receiving), 22 | // so we track senders and receivers separately and infer which one to use based on the EventHub (sender) vs. EventHubTrigger (receiver) attribute. 23 | // Connection strings may also encapsulate different endpoints. 24 | 25 | // The client cache must be thread safe because clients are accessed/added on the function 26 | private readonly ConcurrentDictionary _clients = new ConcurrentDictionary(StringComparer.OrdinalIgnoreCase); 27 | private readonly Dictionary _receiverCreds = new Dictionary(StringComparer.OrdinalIgnoreCase); 28 | private readonly Dictionary _explicitlyProvidedHosts = new Dictionary(StringComparer.OrdinalIgnoreCase); 29 | 30 | /// 31 | /// Name of the blob container that the EventHostProcessor instances uses to coordinate load balancing listening on an event hub. 32 | /// Each event hub gets its own blob prefix within the container. 33 | /// 34 | public const string LeaseContainerName = "azure-webjobs-eventhub"; 35 | private int _batchCheckpointFrequency = 1; 36 | 37 | public EventHubOptions() 38 | { 39 | EventProcessorOptions = EventProcessorOptions.DefaultOptions; 40 | PartitionManagerOptions = new PartitionManagerOptions(); 41 | InitialOffsetOptions = new InitialOffsetOptions(); 42 | } 43 | 44 | /// 45 | /// Gets or sets the number of batches to process before creating an EventHub cursor checkpoint. Default 1. 46 | /// 47 | public int BatchCheckpointFrequency 48 | { 49 | get 50 | { 51 | return _batchCheckpointFrequency; 52 | } 53 | 54 | set 55 | { 56 | if (value <= 0) 57 | { 58 | throw new InvalidOperationException("Batch checkpoint frequency must be larger than 0."); 59 | } 60 | _batchCheckpointFrequency = value; 61 | } 62 | } 63 | 64 | public InitialOffsetOptions InitialOffsetOptions { get; set; } 65 | 66 | public EventProcessorOptions EventProcessorOptions { get; } 67 | 68 | public PartitionManagerOptions PartitionManagerOptions { get; } 69 | 70 | /// 71 | /// Add an existing client for sending messages to an event hub. Infer the eventHub name from client.path 72 | /// 73 | /// 74 | public void AddEventHubClient(EventHubClient client) 75 | { 76 | if (client == null) 77 | { 78 | throw new ArgumentNullException("client"); 79 | } 80 | string eventHubName = client.EventHubName; 81 | AddEventHubClient(eventHubName, client); 82 | } 83 | 84 | /// 85 | /// Add an existing client for sending messages to an event hub. Infer the eventHub name from client.path 86 | /// 87 | /// name of the event hub 88 | /// 89 | public void AddEventHubClient(string eventHubName, EventHubClient client) 90 | { 91 | if (eventHubName == null) 92 | { 93 | throw new ArgumentNullException("eventHubName"); 94 | } 95 | if (client == null) 96 | { 97 | throw new ArgumentNullException("client"); 98 | } 99 | 100 | _clients[eventHubName] = client; 101 | } 102 | 103 | /// 104 | /// Add a connection for sending messages to an event hub. Connect via the connection string. 105 | /// 106 | /// name of the event hub. 107 | /// connection string for sending messages. If this includes an EntityPath, it takes precedence over the eventHubName parameter. 108 | public void AddSender(string eventHubName, string sendConnectionString) 109 | { 110 | if (eventHubName == null) 111 | { 112 | throw new ArgumentNullException("eventHubName"); 113 | } 114 | if (sendConnectionString == null) 115 | { 116 | throw new ArgumentNullException("sendConnectionString"); 117 | } 118 | 119 | EventHubsConnectionStringBuilder sb = new EventHubsConnectionStringBuilder(sendConnectionString); 120 | if (string.IsNullOrWhiteSpace(sb.EntityPath)) 121 | { 122 | sb.EntityPath = eventHubName; 123 | } 124 | 125 | var client = EventHubClient.CreateFromConnectionString(sb.ToString()); 126 | AddEventHubClient(eventHubName, client); 127 | } 128 | 129 | /// 130 | /// Add a connection for listening on events from an event hub. 131 | /// 132 | /// Name of the event hub 133 | /// initialized listener object 134 | /// The EventProcessorHost type is from the ServiceBus SDK. 135 | /// Allow callers to bind to EventHubConfiguration without needing to have a direct assembly reference to the ServiceBus SDK. 136 | /// The compiler needs to resolve all types in all overloads, so give methods that use the ServiceBus SDK types unique non-overloaded names 137 | /// to avoid eager compiler resolution. 138 | /// 139 | public void AddEventProcessorHost(string eventHubName, EventProcessorHost listener) 140 | { 141 | if (eventHubName == null) 142 | { 143 | throw new ArgumentNullException("eventHubName"); 144 | } 145 | if (listener == null) 146 | { 147 | throw new ArgumentNullException("listener"); 148 | } 149 | 150 | _explicitlyProvidedHosts[eventHubName] = listener; 151 | } 152 | 153 | /// 154 | /// Add a connection for listening on events from an event hub. Connect via the connection string and use the SDK's built-in storage account. 155 | /// 156 | /// name of the event hub 157 | /// connection string for receiving messages. This can encapsulate other service bus properties like the namespace and endpoints. 158 | public void AddReceiver(string eventHubName, string receiverConnectionString) 159 | { 160 | if (eventHubName == null) 161 | { 162 | throw new ArgumentNullException("eventHubName"); 163 | } 164 | if (receiverConnectionString == null) 165 | { 166 | throw new ArgumentNullException("receiverConnectionString"); 167 | } 168 | 169 | this._receiverCreds[eventHubName] = new ReceiverCreds 170 | { 171 | EventHubConnectionString = receiverConnectionString 172 | }; 173 | } 174 | 175 | /// 176 | /// Add a connection for listening on events from an event hub. Connect via the connection string and use the supplied storage account 177 | /// 178 | /// name of the event hub 179 | /// connection string for receiving messages 180 | /// storage connection string that the EventProcessorHost client will use to coordinate multiple listener instances. 181 | public void AddReceiver(string eventHubName, string receiverConnectionString, string storageConnectionString) 182 | { 183 | if (eventHubName == null) 184 | { 185 | throw new ArgumentNullException("eventHubName"); 186 | } 187 | if (receiverConnectionString == null) 188 | { 189 | throw new ArgumentNullException("receiverConnectionString"); 190 | } 191 | if (storageConnectionString == null) 192 | { 193 | throw new ArgumentNullException("storageConnectionString"); 194 | } 195 | 196 | this._receiverCreds[eventHubName] = new ReceiverCreds 197 | { 198 | EventHubConnectionString = receiverConnectionString, 199 | StorageConnectionString = storageConnectionString 200 | }; 201 | } 202 | 203 | internal EventHubClient GetEventHubClient(string eventHubName, string connection) 204 | { 205 | EventHubClient client; 206 | 207 | if (string.IsNullOrEmpty(eventHubName)) 208 | { 209 | EventHubsConnectionStringBuilder builder = new EventHubsConnectionStringBuilder(connection); 210 | eventHubName = builder.EntityPath; 211 | } 212 | 213 | if (_clients.TryGetValue(eventHubName, out client)) 214 | { 215 | return client; 216 | } 217 | else if (!string.IsNullOrWhiteSpace(connection)) 218 | { 219 | return _clients.GetOrAdd(eventHubName, key => 220 | { 221 | AddSender(key, connection); 222 | return _clients[key]; 223 | }); 224 | } 225 | throw new InvalidOperationException("No event hub sender named " + eventHubName); 226 | } 227 | 228 | // Lookup a listener for receiving events given the name provided in the [EventHubTrigger] attribute. 229 | internal EventProcessorHost GetEventProcessorHost(IConfiguration config, string eventHubName, string consumerGroup) 230 | { 231 | ReceiverCreds creds; 232 | if (this._receiverCreds.TryGetValue(eventHubName, out creds)) 233 | { 234 | // Common case. Create a new EventProcessorHost instance to listen. 235 | string eventProcessorHostName = Guid.NewGuid().ToString(); 236 | 237 | if (consumerGroup == null) 238 | { 239 | consumerGroup = PartitionReceiver.DefaultConsumerGroupName; 240 | } 241 | var storageConnectionString = creds.StorageConnectionString; 242 | if (storageConnectionString == null) 243 | { 244 | string defaultStorageString = config.GetWebJobsConnectionString(ConnectionStringNames.Storage); 245 | storageConnectionString = defaultStorageString; 246 | } 247 | 248 | // If the connection string provides a hub name, that takes precedence. 249 | // Note that connection strings *can't* specify a consumerGroup, so must always be passed in. 250 | string actualPath = eventHubName; 251 | EventHubsConnectionStringBuilder sb = new EventHubsConnectionStringBuilder(creds.EventHubConnectionString); 252 | if (sb.EntityPath != null) 253 | { 254 | actualPath = sb.EntityPath; 255 | sb.EntityPath = null; // need to remove to use with EventProcessorHost 256 | } 257 | 258 | var @namespace = GetEventHubNamespace(sb); 259 | var blobPrefix = GetBlobPrefix(actualPath, @namespace); 260 | 261 | // Use blob prefix support available in EPH starting in 2.2.6 262 | EventProcessorHost host = new EventProcessorHost( 263 | hostName: eventProcessorHostName, 264 | eventHubPath: actualPath, 265 | consumerGroupName: consumerGroup, 266 | eventHubConnectionString: sb.ToString(), 267 | storageConnectionString: storageConnectionString, 268 | leaseContainerName: LeaseContainerName, 269 | storageBlobPrefix: blobPrefix); 270 | 271 | host.PartitionManagerOptions = PartitionManagerOptions; 272 | 273 | return host; 274 | } 275 | else 276 | { 277 | // Rare case: a power-user caller specifically provided an event processor host to use. 278 | EventProcessorHost host; 279 | if (_explicitlyProvidedHosts.TryGetValue(eventHubName, out host)) 280 | { 281 | return host; 282 | } 283 | } 284 | throw new InvalidOperationException("No event hub receiver named " + eventHubName); 285 | } 286 | 287 | private static string EscapeStorageCharacter(char character) 288 | { 289 | var ordinalValue = (ushort)character; 290 | if (ordinalValue < 0x100) 291 | { 292 | return string.Format(CultureInfo.InvariantCulture, ":{0:X2}", ordinalValue); 293 | } 294 | else 295 | { 296 | return string.Format(CultureInfo.InvariantCulture, "::{0:X4}", ordinalValue); 297 | } 298 | } 299 | 300 | // Escape a blob path. 301 | // For diagnostics, we want human-readble strings that resemble the input. 302 | // Inputs are most commonly alphanumeric with a fex extra chars (dash, underscore, dot). 303 | // Escape character is a ':', which is also escaped. 304 | // Blob names are case sensitive; whereas input is case insensitive, so normalize to lower. 305 | private static string EscapeBlobPath(string path) 306 | { 307 | StringBuilder sb = new StringBuilder(path.Length); 308 | foreach (char c in path) 309 | { 310 | if (c >= 'a' && c <= 'z') 311 | { 312 | sb.Append(c); 313 | } 314 | else if (c == '-' || c == '_' || c == '.') 315 | { 316 | // Potentially common carahcters. 317 | sb.Append(c); 318 | } 319 | else if (c >= 'A' && c <= 'Z') 320 | { 321 | sb.Append((char)(c - 'A' + 'a')); // ToLower 322 | } 323 | else if (c >= '0' && c <= '9') 324 | { 325 | sb.Append(c); 326 | } 327 | else 328 | { 329 | sb.Append(EscapeStorageCharacter(c)); 330 | } 331 | } 332 | 333 | return sb.ToString(); 334 | } 335 | 336 | internal static string GetEventHubNamespace(EventHubsConnectionStringBuilder connectionString) 337 | { 338 | // EventHubs only have 1 endpoint. 339 | var url = connectionString.Endpoint; 340 | var @namespace = url.Host; 341 | return @namespace; 342 | } 343 | 344 | /// 345 | /// Get the blob prefix used with EventProcessorHost for a given event hub. 346 | /// 347 | /// the event hub path 348 | /// the event hub's service bus namespace. 349 | /// a blob prefix path that can be passed to EventProcessorHost. 350 | /// 351 | /// An event hub is defined by it's path and namespace. The namespace is extracted from the connection string. 352 | /// This must be an injective one-to-one function because: 353 | /// 1. multiple machines listening on the same event hub must use the same blob prefix. This means it must be deterministic. 354 | /// 2. different event hubs must not resolve to the same path. 355 | /// 356 | public static string GetBlobPrefix(string eventHubName, string serviceBusNamespace) 357 | { 358 | if (eventHubName == null) 359 | { 360 | throw new ArgumentNullException("eventHubName"); 361 | } 362 | if (serviceBusNamespace == null) 363 | { 364 | throw new ArgumentNullException("serviceBusNamespace"); 365 | } 366 | 367 | string key = EscapeBlobPath(serviceBusNamespace) + "/" + EscapeBlobPath(eventHubName) + "/"; 368 | return key; 369 | } 370 | 371 | public string Format() 372 | { 373 | JObject eventProcessorOptions = null; 374 | if (EventProcessorOptions != null) 375 | { 376 | eventProcessorOptions = new JObject 377 | { 378 | { nameof(EventProcessorOptions.EnableReceiverRuntimeMetric), EventProcessorOptions.EnableReceiverRuntimeMetric }, 379 | { nameof(EventProcessorOptions.InvokeProcessorAfterReceiveTimeout), EventProcessorOptions.InvokeProcessorAfterReceiveTimeout }, 380 | { nameof(EventProcessorOptions.MaxBatchSize), EventProcessorOptions.MaxBatchSize }, 381 | { nameof(EventProcessorOptions.PrefetchCount), EventProcessorOptions.PrefetchCount }, 382 | { nameof(EventProcessorOptions.ReceiveTimeout), EventProcessorOptions.ReceiveTimeout } 383 | }; 384 | } 385 | 386 | JObject partitionManagerOptions = null; 387 | if (PartitionManagerOptions != null) 388 | { 389 | partitionManagerOptions = new JObject 390 | { 391 | { nameof(PartitionManagerOptions.LeaseDuration), PartitionManagerOptions.LeaseDuration }, 392 | { nameof(PartitionManagerOptions.RenewInterval), PartitionManagerOptions.RenewInterval }, 393 | }; 394 | } 395 | 396 | JObject initialOffsetOptions = null; 397 | if (InitialOffsetOptions != null) 398 | { 399 | initialOffsetOptions = new JObject 400 | { 401 | { nameof(InitialOffsetOptions.Type), InitialOffsetOptions.Type }, 402 | { nameof(InitialOffsetOptions.EnqueuedTimeUTC), InitialOffsetOptions.EnqueuedTimeUTC }, 403 | }; 404 | } 405 | 406 | JObject options = new JObject 407 | { 408 | { nameof(BatchCheckpointFrequency), BatchCheckpointFrequency }, 409 | { nameof(EventProcessorOptions), eventProcessorOptions }, 410 | { nameof(PartitionManagerOptions), partitionManagerOptions }, 411 | { nameof(InitialOffsetOptions), initialOffsetOptions } 412 | }; 413 | 414 | return options.ToString(Formatting.Indented); 415 | } 416 | 417 | // Hold credentials for a given eventHub name. 418 | // Multiple consumer groups (and multiple listeners) on the same hub can share the same credentials. 419 | private class ReceiverCreds 420 | { 421 | // Required. 422 | public string EventHubConnectionString { get; set; } 423 | 424 | // Optional. If not found, use the stroage from JobHostConfiguration 425 | public string StorageConnectionString { get; set; } 426 | } 427 | } 428 | } 429 | -------------------------------------------------------------------------------- /src/Microsoft.Azure.WebJobs.Extensions.EventHubs/Config/EventHubWebJobsBuilderExtensions.cs: -------------------------------------------------------------------------------- 1 | // Copyright (c) .NET Foundation. All rights reserved. 2 | // Licensed under the MIT License. See License.txt in the project root for license information. 3 | 4 | using System; 5 | using Microsoft.Azure.EventHubs; 6 | using Microsoft.Azure.WebJobs; 7 | using Microsoft.Azure.WebJobs.EventHubs; 8 | using Microsoft.Extensions.DependencyInjection; 9 | 10 | namespace Microsoft.Extensions.Hosting 11 | { 12 | public static class EventHubWebJobsBuilderExtensions 13 | { 14 | public static IWebJobsBuilder AddEventHubs(this IWebJobsBuilder builder) 15 | { 16 | if (builder == null) 17 | { 18 | throw new ArgumentNullException(nameof(builder)); 19 | } 20 | 21 | builder.AddEventHubs(ConfigureOptions); 22 | 23 | return builder; 24 | } 25 | 26 | public static IWebJobsBuilder AddEventHubs(this IWebJobsBuilder builder, Action configure) 27 | { 28 | if (builder == null) 29 | { 30 | throw new ArgumentNullException(nameof(builder)); 31 | } 32 | 33 | if (configure == null) 34 | { 35 | throw new ArgumentNullException(nameof(configure)); 36 | } 37 | 38 | builder.AddExtension() 39 | .BindOptions(); 40 | 41 | builder.Services.Configure(options => 42 | { 43 | configure(options); 44 | }); 45 | 46 | return builder; 47 | } 48 | 49 | internal static void ConfigureOptions(EventHubOptions options) 50 | { 51 | string offsetType = options?.InitialOffsetOptions?.Type?.ToLower() ?? String.Empty; 52 | if (!offsetType.Equals(String.Empty)) 53 | { 54 | switch (offsetType) 55 | { 56 | case "fromstart": 57 | options.EventProcessorOptions.InitialOffsetProvider = (s) => { return EventPosition.FromStart(); }; 58 | break; 59 | case "fromend": 60 | options.EventProcessorOptions.InitialOffsetProvider = (s) => { return EventPosition.FromEnd(); }; 61 | break; 62 | case "fromenqueuedtime": 63 | try 64 | { 65 | DateTime enqueuedTimeUTC = DateTime.Parse(options.InitialOffsetOptions.EnqueuedTimeUTC).ToUniversalTime(); 66 | options.EventProcessorOptions.InitialOffsetProvider = (s) => { return EventPosition.FromEnqueuedTime(enqueuedTimeUTC); }; 67 | } 68 | catch (System.FormatException fe) 69 | { 70 | string message = $"{nameof(EventHubOptions)}:{nameof(InitialOffsetOptions)}:{nameof(InitialOffsetOptions.EnqueuedTimeUTC)} is configured with an invalid format. " + 71 | "Please use a format supported by DateTime.Parse(). e.g. 'yyyy-MM-ddTHH:mm:ssZ'"; 72 | throw new InvalidOperationException(message, fe); 73 | } 74 | break; 75 | default: 76 | throw new InvalidOperationException("An unsupported value was supplied for initialOffsetOptions.type"); 77 | } 78 | // If not specified, EventProcessor's default offset will apply 79 | } 80 | } 81 | } 82 | } 83 | -------------------------------------------------------------------------------- /src/Microsoft.Azure.WebJobs.Extensions.EventHubs/Config/InitialOffsetOptions.cs: -------------------------------------------------------------------------------- 1 | // Copyright (c) .NET Foundation. All rights reserved. 2 | // Licensed under the MIT License. See License.txt in the project root for license information. 3 | 4 | using System; 5 | using System.Collections.Concurrent; 6 | using System.Collections.Generic; 7 | using System.Globalization; 8 | using System.Text; 9 | using Microsoft.Azure.EventHubs; 10 | using Microsoft.Azure.EventHubs.Processor; 11 | using Microsoft.Azure.WebJobs.Hosting; 12 | using Microsoft.Extensions.Configuration; 13 | using Microsoft.Extensions.Options; 14 | using Newtonsoft.Json; 15 | using Newtonsoft.Json.Linq; 16 | 17 | namespace Microsoft.Azure.WebJobs.EventHubs 18 | { 19 | public class InitialOffsetOptions 20 | { 21 | public string Type { get; set; } = ""; 22 | public string EnqueuedTimeUTC { get; set; } = ""; 23 | } 24 | } 25 | -------------------------------------------------------------------------------- /src/Microsoft.Azure.WebJobs.Extensions.EventHubs/EventHubAttribute.cs: -------------------------------------------------------------------------------- 1 | // Copyright (c) .NET Foundation. All rights reserved. 2 | // Licensed under the MIT License. See License.txt in the project root for license information. 3 | 4 | using System; 5 | using Microsoft.Azure.WebJobs.Description; 6 | 7 | namespace Microsoft.Azure.WebJobs 8 | { 9 | /// 10 | /// Setup an 'output' binding to an EventHub. This can be any output type compatible with an IAsyncCollector. 11 | /// 12 | [AttributeUsage(AttributeTargets.Parameter | AttributeTargets.ReturnValue)] 13 | [Binding] 14 | public sealed class EventHubAttribute : Attribute 15 | { 16 | /// 17 | /// Initialize a new instance of the 18 | /// 19 | /// Name of the event hub as resolved against the 20 | public EventHubAttribute(string eventHubName) 21 | { 22 | EventHubName = eventHubName; 23 | } 24 | 25 | /// 26 | /// The name of the event hub. This is resolved against the 27 | /// 28 | [AutoResolve] 29 | public string EventHubName { get; private set; } 30 | 31 | /// 32 | /// Gets or sets the optional connection string name that contains the Event Hub connection string. If missing, tries to use a registered event hub sender. 33 | /// 34 | [ConnectionString] 35 | public string Connection { get; set; } 36 | } 37 | } -------------------------------------------------------------------------------- /src/Microsoft.Azure.WebJobs.Extensions.EventHubs/EventHubTriggerAttribute.cs: -------------------------------------------------------------------------------- 1 | // Copyright (c) .NET Foundation. All rights reserved. 2 | // Licensed under the MIT License. See License.txt in the project root for license information. 3 | 4 | using System; 5 | using Microsoft.Azure.WebJobs.Description; 6 | 7 | namespace Microsoft.Azure.WebJobs 8 | { 9 | /// 10 | /// Setup an 'trigger' on a parameter to listen on events from an event hub. 11 | /// 12 | [AttributeUsage(AttributeTargets.Parameter)] 13 | [Binding] 14 | public sealed class EventHubTriggerAttribute : Attribute 15 | { 16 | /// 17 | /// Create an instance of this attribute. 18 | /// 19 | /// Event hub to listen on for messages. 20 | public EventHubTriggerAttribute(string eventHubName) 21 | { 22 | EventHubName = eventHubName; 23 | } 24 | 25 | /// 26 | /// Name of the event hub. 27 | /// 28 | public string EventHubName { get; private set; } 29 | 30 | /// 31 | /// Optional Name of the consumer group. If missing, then use the default name, "$Default" 32 | /// 33 | public string ConsumerGroup { get; set; } 34 | 35 | /// 36 | /// Gets or sets the optional app setting name that contains the Event Hub connection string. If missing, tries to use a registered event hub receiver. 37 | /// 38 | public string Connection { get; set; } 39 | } 40 | } -------------------------------------------------------------------------------- /src/Microsoft.Azure.WebJobs.Extensions.EventHubs/EventHubsWebJobsStartup.cs: -------------------------------------------------------------------------------- 1 | // Copyright (c) .NET Foundation. All rights reserved. 2 | // Licensed under the MIT License. See License.txt in the project root for license information. 3 | 4 | using Microsoft.Azure.WebJobs.EventHubs; 5 | using Microsoft.Azure.WebJobs.Hosting; 6 | using Microsoft.Extensions.Hosting; 7 | 8 | [assembly: WebJobsStartup(typeof(EventHubsWebJobsStartup))] 9 | 10 | namespace Microsoft.Azure.WebJobs.EventHubs 11 | { 12 | public class EventHubsWebJobsStartup : IWebJobsStartup 13 | { 14 | public void Configure(IWebJobsBuilder builder) 15 | { 16 | builder.AddEventHubs(); 17 | } 18 | } 19 | } 20 | -------------------------------------------------------------------------------- /src/Microsoft.Azure.WebJobs.Extensions.EventHubs/Extensions/SystemPropertiesCollectionExtensions.cs: -------------------------------------------------------------------------------- 1 | // Copyright (c) .NET Foundation. All rights reserved. 2 | // Licensed under the MIT License. See License.txt in the project root for license information. 3 | 4 | using System.Collections.Generic; 5 | using static Microsoft.Azure.EventHubs.EventData; 6 | 7 | namespace Microsoft.Azure.WebJobs.EventHubs 8 | { 9 | static internal class SystemPropertiesCollectionExtensions 10 | { 11 | internal static IDictionary ToDictionary(this SystemPropertiesCollection collection) 12 | { 13 | IDictionary modifiedDictionary = new Dictionary(collection); 14 | 15 | // Following is needed to maintain structure of bindingdata: https://github.com/Azure/azure-webjobs-sdk/pull/1849 16 | modifiedDictionary["SequenceNumber"] = collection.SequenceNumber; 17 | modifiedDictionary["Offset"] = collection.Offset; 18 | modifiedDictionary["PartitionKey"] = collection.PartitionKey; 19 | modifiedDictionary["EnqueuedTimeUtc"] = collection.EnqueuedTimeUtc; 20 | return modifiedDictionary; 21 | } 22 | } 23 | } 24 | -------------------------------------------------------------------------------- /src/Microsoft.Azure.WebJobs.Extensions.EventHubs/Listeners/EventHubListener.cs: -------------------------------------------------------------------------------- 1 | // Copyright (c) .NET Foundation. All rights reserved. 2 | // Licensed under the MIT License. See License.txt in the project root for license information. 3 | 4 | using System; 5 | using System.Collections.Generic; 6 | using System.Diagnostics; 7 | using System.IO; 8 | using System.Linq; 9 | using System.Threading; 10 | using System.Threading.Tasks; 11 | using Microsoft.Azure.EventHubs; 12 | using Microsoft.Azure.EventHubs.Processor; 13 | using Microsoft.Azure.WebJobs.EventHubs.Listeners; 14 | using Microsoft.Azure.WebJobs.Host.Executors; 15 | using Microsoft.Azure.WebJobs.Host.Listeners; 16 | using Microsoft.Azure.WebJobs.Host.Scale; 17 | using Microsoft.Extensions.Logging; 18 | using Microsoft.WindowsAzure.Storage.Blob; 19 | using Newtonsoft.Json; 20 | using Newtonsoft.Json.Linq; 21 | 22 | namespace Microsoft.Azure.WebJobs.EventHubs 23 | { 24 | internal sealed class EventHubListener : IListener, IEventProcessorFactory, IScaleMonitorProvider 25 | { 26 | private static readonly Dictionary EmptyScope = new Dictionary(); 27 | private readonly string _functionId; 28 | private readonly string _eventHubName; 29 | private readonly string _consumerGroup; 30 | private readonly string _connectionString; 31 | private readonly string _storageConnectionString; 32 | private readonly ITriggeredFunctionExecutor _executor; 33 | private readonly EventProcessorHost _eventProcessorHost; 34 | private readonly bool _singleDispatch; 35 | private readonly EventHubOptions _options; 36 | private readonly ILogger _logger; 37 | private readonly SemaphoreSlim _stopSemaphoreSlim = new SemaphoreSlim(1, 1); 38 | private readonly string _details; 39 | private bool _started; 40 | 41 | private Lazy _scaleMonitor; 42 | 43 | public EventHubListener( 44 | string functionId, 45 | string eventHubName, 46 | string consumerGroup, 47 | string connectionString, 48 | string storageConnectionString, 49 | ITriggeredFunctionExecutor executor, 50 | EventProcessorHost eventProcessorHost, 51 | bool singleDispatch, 52 | EventHubOptions options, 53 | ILogger logger, 54 | CloudBlobContainer blobContainer = null) 55 | { 56 | _functionId = functionId; 57 | _eventHubName = eventHubName; 58 | _consumerGroup = consumerGroup; 59 | _connectionString = connectionString; 60 | _storageConnectionString = storageConnectionString; 61 | _executor = executor; 62 | _eventProcessorHost = eventProcessorHost; 63 | _singleDispatch = singleDispatch; 64 | _options = options; 65 | _logger = logger; 66 | _scaleMonitor = new Lazy(() => new EventHubsScaleMonitor(_functionId, _eventHubName, _consumerGroup, _connectionString, _storageConnectionString, _logger, blobContainer)); 67 | _details = $"'namespace='{eventProcessorHost?.EndpointAddress}', eventHub='{eventProcessorHost?.EventHubPath}', " + 68 | $"consumerGroup='{eventProcessorHost?.ConsumerGroupName}', functionId='{functionId}', singleDispatch='{singleDispatch}'"; 69 | } 70 | 71 | void IListener.Cancel() 72 | { 73 | StopAsync(CancellationToken.None).Wait(); 74 | } 75 | 76 | void IDisposable.Dispose() 77 | { 78 | StopAsync(CancellationToken.None).Wait(); 79 | } 80 | 81 | public async Task StartAsync(CancellationToken cancellationToken) 82 | { 83 | await _eventProcessorHost.RegisterEventProcessorFactoryAsync(this, _options.EventProcessorOptions); 84 | _started = true; 85 | 86 | _logger.LogDebug($"EventHub listener started ({_details})"); 87 | } 88 | 89 | public async Task StopAsync(CancellationToken cancellationToken) 90 | { 91 | await _stopSemaphoreSlim.WaitAsync(); 92 | try 93 | { 94 | if (_started) 95 | { 96 | await _eventProcessorHost.UnregisterEventProcessorAsync(); 97 | _logger.LogDebug($"EventHub listener stopped ({_details})"); 98 | } 99 | else 100 | { 101 | _logger.LogDebug($"EventHub listener is already stopped ({_details})"); 102 | } 103 | _started = false; 104 | } 105 | finally 106 | { 107 | _stopSemaphoreSlim.Release(); 108 | } 109 | } 110 | 111 | IEventProcessor IEventProcessorFactory.CreateEventProcessor(PartitionContext context) 112 | { 113 | return new EventProcessor(_options, _executor, _logger, _singleDispatch); 114 | } 115 | 116 | public IScaleMonitor GetMonitor() 117 | { 118 | return _scaleMonitor.Value; 119 | } 120 | 121 | /// 122 | /// Wrapper for un-mockable checkpoint APIs to aid in unit testing 123 | /// 124 | internal interface ICheckpointer 125 | { 126 | Task CheckpointAsync(PartitionContext context); 127 | } 128 | 129 | // We get a new instance each time Start() is called. 130 | // We'll get a listener per partition - so they can potentialy run in parallel even on a single machine. 131 | internal class EventProcessor : IEventProcessor, IDisposable, ICheckpointer 132 | { 133 | private readonly ITriggeredFunctionExecutor _executor; 134 | private readonly bool _singleDispatch; 135 | private readonly ILogger _logger; 136 | private readonly CancellationTokenSource _cts = new CancellationTokenSource(); 137 | private readonly ICheckpointer _checkpointer; 138 | private readonly int _batchCheckpointFrequency; 139 | private int _batchCounter = 0; 140 | private bool _disposed = false; 141 | 142 | public EventProcessor(EventHubOptions options, ITriggeredFunctionExecutor executor, ILogger logger, bool singleDispatch, ICheckpointer checkpointer = null) 143 | { 144 | _checkpointer = checkpointer ?? this; 145 | _executor = executor; 146 | _singleDispatch = singleDispatch; 147 | _batchCheckpointFrequency = options.BatchCheckpointFrequency; 148 | _logger = logger; 149 | } 150 | 151 | public Task CloseAsync(PartitionContext context, CloseReason reason) 152 | { 153 | // signal cancellation for any in progress executions 154 | _cts.Cancel(); 155 | 156 | _logger.LogDebug(GetOperationDetails(context, $"CloseAsync, {reason.ToString()}")); 157 | return Task.CompletedTask; 158 | } 159 | 160 | public Task OpenAsync(PartitionContext context) 161 | { 162 | _logger.LogDebug(GetOperationDetails(context, "OpenAsync")); 163 | return Task.CompletedTask; 164 | } 165 | 166 | public Task ProcessErrorAsync(PartitionContext context, Exception error) 167 | { 168 | string errorDetails = $"Processing error (Partition Id: '{context.PartitionId}', Owner: '{context.Owner}', EventHubPath: '{context.EventHubPath}')."; 169 | 170 | Utility.LogException(error, errorDetails, _logger); 171 | 172 | return Task.CompletedTask; 173 | } 174 | 175 | public async Task ProcessEventsAsync(PartitionContext context, IEnumerable messages) 176 | { 177 | var triggerInput = new EventHubTriggerInput 178 | { 179 | Events = messages.ToArray(), 180 | PartitionContext = context 181 | }; 182 | 183 | TriggeredFunctionData input = null; 184 | if (_singleDispatch) 185 | { 186 | // Single dispatch 187 | int eventCount = triggerInput.Events.Length; 188 | List invocationTasks = new List(); 189 | for (int i = 0; i < eventCount; i++) 190 | { 191 | if (_cts.IsCancellationRequested) 192 | { 193 | break; 194 | } 195 | 196 | EventHubTriggerInput eventHubTriggerInput = triggerInput.GetSingleEventTriggerInput(i); 197 | input = new TriggeredFunctionData 198 | { 199 | TriggerValue = eventHubTriggerInput, 200 | TriggerDetails = eventHubTriggerInput.GetTriggerDetails(context) 201 | }; 202 | 203 | Task task = TryExecuteWithLoggingAsync(input, triggerInput.Events[i]); 204 | invocationTasks.Add(task); 205 | } 206 | 207 | // Drain the whole batch before taking more work 208 | if (invocationTasks.Count > 0) 209 | { 210 | await Task.WhenAll(invocationTasks); 211 | } 212 | } 213 | else 214 | { 215 | // Batch dispatch 216 | input = new TriggeredFunctionData 217 | { 218 | TriggerValue = triggerInput, 219 | TriggerDetails = triggerInput.GetTriggerDetails(context) 220 | }; 221 | 222 | using (_logger.BeginScope(GetLinksScope(triggerInput.Events))) 223 | { 224 | await _executor.TryExecuteAsync(input, _cts.Token); 225 | } 226 | } 227 | 228 | // Dispose all messages to help with memory pressure. If this is missed, the finalizer thread will still get them. 229 | bool hasEvents = false; 230 | foreach (var message in messages) 231 | { 232 | hasEvents = true; 233 | message.Dispose(); 234 | } 235 | 236 | // Checkpoint if we processed any events. 237 | // Don't checkpoint if no events. This can reset the sequence counter to 0. 238 | // Note: we intentionally checkpoint the batch regardless of function 239 | // success/failure. EventHub doesn't support any sort "poison event" model, 240 | // so that is the responsibility of the user's function currently. E.g. 241 | // the function should have try/catch handling around all event processing 242 | // code, and capture/log/persist failed events, since they won't be retried. 243 | if (hasEvents) 244 | { 245 | await CheckpointAsync(context); 246 | } 247 | } 248 | 249 | private async Task TryExecuteWithLoggingAsync(TriggeredFunctionData input, EventData message) 250 | { 251 | using (_logger.BeginScope(GetLinksScope(message))) 252 | { 253 | await _executor.TryExecuteAsync(input, _cts.Token); 254 | } 255 | } 256 | 257 | private async Task CheckpointAsync(PartitionContext context) 258 | { 259 | bool checkpointed = false; 260 | if (_batchCheckpointFrequency == 1) 261 | { 262 | await _checkpointer.CheckpointAsync(context); 263 | checkpointed = true; 264 | } 265 | else 266 | { 267 | // only checkpoint every N batches 268 | if (++_batchCounter >= _batchCheckpointFrequency) 269 | { 270 | _batchCounter = 0; 271 | await _checkpointer.CheckpointAsync(context); 272 | checkpointed = true; 273 | } 274 | } 275 | if (checkpointed) 276 | { 277 | _logger.LogDebug(GetOperationDetails(context, "CheckpointAsync")); 278 | } 279 | } 280 | 281 | protected virtual void Dispose(bool disposing) 282 | { 283 | if (!_disposed) 284 | { 285 | if (disposing) 286 | { 287 | _cts.Dispose(); 288 | } 289 | 290 | _disposed = true; 291 | } 292 | } 293 | 294 | public void Dispose() 295 | { 296 | Dispose(true); 297 | } 298 | 299 | async Task ICheckpointer.CheckpointAsync(PartitionContext context) 300 | { 301 | await context.CheckpointAsync(); 302 | } 303 | 304 | private Dictionary GetLinksScope(EventData message) 305 | { 306 | if (TryGetLinkedActivity(message, out var link)) 307 | { 308 | return new Dictionary {["Links"] = new[] {link}}; 309 | } 310 | 311 | return EmptyScope; 312 | } 313 | 314 | private Dictionary GetLinksScope(EventData[] messages) 315 | { 316 | List links = null; 317 | 318 | foreach (var message in messages) 319 | { 320 | if (TryGetLinkedActivity(message, out var link)) 321 | { 322 | if (links == null) 323 | { 324 | links = new List(messages.Length); 325 | } 326 | 327 | links.Add(link); 328 | } 329 | } 330 | 331 | if (links != null) 332 | { 333 | return new Dictionary {["Links"] = links}; 334 | } 335 | 336 | return EmptyScope; 337 | } 338 | 339 | private bool TryGetLinkedActivity(EventData message, out Activity link) 340 | { 341 | link = null; 342 | 343 | if (((message.SystemProperties != null && message.SystemProperties.TryGetValue("Diagnostic-Id", out var diagnosticIdObj)) || message.Properties.TryGetValue("Diagnostic-Id", out diagnosticIdObj)) 344 | && diagnosticIdObj is string diagnosticIdString) 345 | { 346 | link = new Activity("Microsoft.Azure.EventHubs.Process"); 347 | link.SetParentId(diagnosticIdString); 348 | return true; 349 | } 350 | 351 | return false; 352 | } 353 | 354 | private string GetOperationDetails(PartitionContext context, string operation) 355 | { 356 | StringWriter sw = new StringWriter(); 357 | using (JsonTextWriter writer = new JsonTextWriter(sw) { Formatting = Formatting.None }) 358 | { 359 | writer.WriteStartObject(); 360 | WritePropertyIfNotNull(writer, "operation", operation); 361 | writer.WritePropertyName("partitionContext"); 362 | writer.WriteStartObject(); 363 | WritePropertyIfNotNull(writer, "partitionId", context.PartitionId); 364 | WritePropertyIfNotNull(writer, "owner", context.Owner); 365 | WritePropertyIfNotNull(writer, "eventHubPath", context.EventHubPath); 366 | writer.WriteEndObject(); 367 | 368 | // Log partition lease 369 | if (context.Lease != null) 370 | { 371 | writer.WritePropertyName("lease"); 372 | writer.WriteStartObject(); 373 | WritePropertyIfNotNull(writer, "offset", context.Lease.Offset); 374 | WritePropertyIfNotNull(writer, "sequenceNumber", context.Lease.SequenceNumber.ToString()); 375 | writer.WriteEndObject(); 376 | } 377 | 378 | // Log RuntimeInformation if EnableReceiverRuntimeMetric is enabled 379 | if (context.RuntimeInformation != null) 380 | { 381 | writer.WritePropertyName("runtimeInformation"); 382 | writer.WriteStartObject(); 383 | WritePropertyIfNotNull(writer, "lastEnqueuedOffset", context.RuntimeInformation.LastEnqueuedOffset); 384 | WritePropertyIfNotNull(writer, "lastSequenceNumber", context.RuntimeInformation.LastSequenceNumber.ToString()); 385 | WritePropertyIfNotNull(writer, "lastEnqueuedTimeUtc", context.RuntimeInformation.LastEnqueuedTimeUtc.ToString("o")); 386 | writer.WriteEndObject(); 387 | } 388 | writer.WriteEndObject(); 389 | } 390 | return sw.ToString(); 391 | } 392 | 393 | private static void WritePropertyIfNotNull(JsonTextWriter writer, string propertyName, string propertyValue) 394 | { 395 | if (propertyValue != null) 396 | { 397 | writer.WritePropertyName(propertyName); 398 | writer.WriteValue(propertyValue); 399 | } 400 | } 401 | } 402 | } 403 | } -------------------------------------------------------------------------------- /src/Microsoft.Azure.WebJobs.Extensions.EventHubs/Listeners/EventHubsScaleMonitor.cs: -------------------------------------------------------------------------------- 1 | // Copyright (c) .NET Foundation. All rights reserved. 2 | // Licensed under the MIT License. See License.txt in the project root for license information. 3 | 4 | using System; 5 | using System.Collections.Generic; 6 | using System.Linq; 7 | using System.Net; 8 | using System.Threading.Tasks; 9 | using Microsoft.Azure.EventHubs; 10 | using Microsoft.Azure.WebJobs.Host.Scale; 11 | using Microsoft.Extensions.Logging; 12 | using Microsoft.WindowsAzure.Storage; 13 | using Microsoft.WindowsAzure.Storage.Blob; 14 | using Newtonsoft.Json; 15 | 16 | namespace Microsoft.Azure.WebJobs.EventHubs.Listeners 17 | { 18 | internal class EventHubsScaleMonitor : IScaleMonitor 19 | { 20 | private const string EventHubContainerName = "azure-webjobs-eventhub"; 21 | private const int PartitionLogIntervalInMinutes = 5; 22 | 23 | private readonly string _functionId; 24 | private readonly string _eventHubName; 25 | private readonly string _consumerGroup; 26 | private readonly string _connectionString; 27 | private readonly string _storageConnectionString; 28 | private readonly Lazy _client; 29 | private readonly ScaleMonitorDescriptor _scaleMonitorDescriptor; 30 | private readonly ILogger _logger; 31 | 32 | private EventHubsConnectionStringBuilder _connectionStringBuilder; 33 | private CloudBlobContainer _blobContainer; 34 | private DateTime _nextPartitionLogTime; 35 | private DateTime _nextPartitionWarningTime; 36 | 37 | public EventHubsScaleMonitor( 38 | string functionId, 39 | string eventHubName, 40 | string consumerGroup, 41 | string connectionString, 42 | string storageConnectionString, 43 | ILogger logger, 44 | CloudBlobContainer blobContainer = null) 45 | { 46 | _functionId = functionId; 47 | _eventHubName = eventHubName; 48 | _consumerGroup = consumerGroup; 49 | _connectionString = connectionString; 50 | _storageConnectionString = storageConnectionString; 51 | _logger = logger; 52 | _scaleMonitorDescriptor = new ScaleMonitorDescriptor($"{_functionId}-EventHubTrigger-{_eventHubName}-{_consumerGroup}".ToLower()); 53 | _nextPartitionLogTime = DateTime.UtcNow; 54 | _nextPartitionWarningTime = DateTime.UtcNow; 55 | _blobContainer = blobContainer; 56 | _client = new Lazy(() => EventHubClient.CreateFromConnectionString(ConnectionStringBuilder.ToString())); 57 | } 58 | 59 | public ScaleMonitorDescriptor Descriptor 60 | { 61 | get 62 | { 63 | return _scaleMonitorDescriptor; 64 | } 65 | } 66 | 67 | private CloudBlobContainer BlobContainer 68 | { 69 | get 70 | { 71 | if (_blobContainer == null) 72 | { 73 | CloudStorageAccount cloudStorageAccount = CloudStorageAccount.Parse(_storageConnectionString); 74 | CloudBlobClient blobClient = cloudStorageAccount.CreateCloudBlobClient(); 75 | _blobContainer = blobClient.GetContainerReference(EventHubContainerName); 76 | } 77 | return _blobContainer; 78 | } 79 | } 80 | 81 | private EventHubsConnectionStringBuilder ConnectionStringBuilder 82 | { 83 | get 84 | { 85 | if (_connectionStringBuilder == null) 86 | { 87 | _connectionStringBuilder = new EventHubsConnectionStringBuilder(_connectionString); 88 | if (!string.IsNullOrEmpty(_eventHubName)) 89 | { 90 | _connectionStringBuilder.EntityPath = _eventHubName; 91 | } 92 | } 93 | return _connectionStringBuilder; 94 | } 95 | } 96 | 97 | async Task IScaleMonitor.GetMetricsAsync() 98 | { 99 | return await GetMetricsAsync(); 100 | } 101 | 102 | public async Task GetMetricsAsync() 103 | { 104 | EventHubsTriggerMetrics metrics = new EventHubsTriggerMetrics(); 105 | EventHubRuntimeInformation runtimeInfo = null; 106 | 107 | try 108 | { 109 | runtimeInfo = await _client.Value.GetRuntimeInformationAsync(); 110 | } 111 | catch (NotSupportedException e) 112 | { 113 | _logger.LogWarning($"EventHubs Trigger does not support NotificationHubs. Error: {e.Message}"); 114 | return metrics; 115 | } 116 | catch (MessagingEntityNotFoundException) 117 | { 118 | _logger.LogWarning($"EventHub '{_eventHubName}' was not found."); 119 | return metrics; 120 | } 121 | catch (TimeoutException e) 122 | { 123 | _logger.LogWarning($"Encountered a timeout while checking EventHub '{_eventHubName}'. Error: {e.Message}"); 124 | return metrics; 125 | } 126 | catch (Exception e) 127 | { 128 | _logger.LogWarning($"Encountered an exception while checking EventHub '{_eventHubName}'. Error: {e.Message}"); 129 | return metrics; 130 | } 131 | 132 | // Get the PartitionRuntimeInformation for all partitions 133 | _logger.LogInformation($"Querying partition information for {runtimeInfo.PartitionCount} partitions."); 134 | var tasks = new Task[runtimeInfo.PartitionCount]; 135 | 136 | for (int i = 0; i < runtimeInfo.PartitionCount; i++) 137 | { 138 | tasks[i] = _client.Value.GetPartitionRuntimeInformationAsync(i.ToString()); 139 | } 140 | 141 | await Task.WhenAll(tasks); 142 | 143 | return await CreateTriggerMetrics(tasks.Select(t => t.Result).ToList()); 144 | } 145 | 146 | internal async Task CreateTriggerMetrics(List partitionRuntimeInfo, bool alwaysLog = false) 147 | { 148 | long totalUnprocessedEventCount = 0; 149 | bool logPartitionInfo = alwaysLog ? true : DateTime.UtcNow >= _nextPartitionLogTime; 150 | bool logPartitionWarning = alwaysLog ? true : DateTime.UtcNow >= _nextPartitionWarningTime; 151 | 152 | // For each partition, get the last enqueued sequence number. 153 | // If the last enqueued sequence number does not equal the SequenceNumber from the lease info in storage, 154 | // accumulate new event counts across partitions to derive total new event counts. 155 | List partitionErrors = new List(); 156 | for (int i = 0; i < partitionRuntimeInfo.Count; i++) 157 | { 158 | long partitionUnprocessedEventCount = 0; 159 | 160 | Tuple partitionLeaseFile = await GetPartitionLeaseFileAsync(i); 161 | BlobPartitionLease partitionLeaseInfo = partitionLeaseFile.Item1; 162 | string errorMsg = partitionLeaseFile.Item2; 163 | 164 | if (partitionRuntimeInfo[i] == null || partitionLeaseInfo == null) 165 | { 166 | partitionErrors.Add(errorMsg); 167 | } 168 | else 169 | { 170 | // Check for the unprocessed messages when there are messages on the event hub parition 171 | // In that case, LastEnqueuedSequenceNumber will be >= 0 172 | if ((partitionRuntimeInfo[i].LastEnqueuedSequenceNumber != -1 && partitionRuntimeInfo[i].LastEnqueuedSequenceNumber != partitionLeaseInfo.SequenceNumber) 173 | || (partitionLeaseInfo.Offset == null && partitionRuntimeInfo[i].LastEnqueuedSequenceNumber >= 0)) 174 | { 175 | partitionUnprocessedEventCount = GetUnprocessedEventCount(partitionRuntimeInfo[i], partitionLeaseInfo); 176 | totalUnprocessedEventCount += partitionUnprocessedEventCount; 177 | } 178 | } 179 | } 180 | 181 | // Only log if not all partitions are failing or it's time to log 182 | if (partitionErrors.Count > 0 && (partitionErrors.Count != partitionRuntimeInfo.Count || logPartitionWarning)) 183 | { 184 | _logger.LogWarning($"Function '{_functionId}': Unable to deserialize partition or lease info with the " + 185 | $"following errors: {string.Join(" ", partitionErrors)}"); 186 | _nextPartitionWarningTime = DateTime.UtcNow.AddMinutes(PartitionLogIntervalInMinutes); 187 | } 188 | 189 | if (totalUnprocessedEventCount > 0 && logPartitionInfo) 190 | { 191 | _logger.LogInformation($"Function '{_functionId}', Total new events: {totalUnprocessedEventCount}"); 192 | _nextPartitionLogTime = DateTime.UtcNow.AddMinutes(PartitionLogIntervalInMinutes); 193 | } 194 | 195 | return new EventHubsTriggerMetrics 196 | { 197 | Timestamp = DateTime.UtcNow, 198 | PartitionCount = partitionRuntimeInfo.Count, 199 | EventCount = totalUnprocessedEventCount 200 | }; 201 | } 202 | 203 | private async Task> GetPartitionLeaseFileAsync(int partitionId) 204 | { 205 | BlobPartitionLease blobPartitionLease = null; 206 | string prefix = $"{EventHubOptions.GetBlobPrefix(_eventHubName, EventHubOptions.GetEventHubNamespace(ConnectionStringBuilder))}{_consumerGroup}/{partitionId}"; 207 | string errorMsg = null; 208 | 209 | try 210 | { 211 | CloudBlockBlob blockBlob = BlobContainer.GetBlockBlobReference(prefix); 212 | 213 | if (blockBlob != null) 214 | { 215 | var result = await blockBlob.DownloadTextAsync(); 216 | if (!string.IsNullOrEmpty(result)) 217 | { 218 | blobPartitionLease = JsonConvert.DeserializeObject(result); 219 | } 220 | } 221 | } 222 | catch (Exception e) 223 | { 224 | var storageException = e as StorageException; 225 | if (storageException?.RequestInformation?.HttpStatusCode == (int)HttpStatusCode.NotFound) 226 | { 227 | errorMsg = $"Lease file data could not be found for blob on Partition: '{partitionId}', " + 228 | $"EventHub: '{_eventHubName}', '{_consumerGroup}'. Error: {e.Message}"; 229 | } 230 | else if (e is JsonSerializationException) 231 | { 232 | errorMsg = $"Could not deserialize blob lease info for blob on Partition: '{partitionId}', " + 233 | $"EventHub: '{_eventHubName}', Consumer Group: '{_consumerGroup}'. Error: {e.Message}"; 234 | } 235 | else 236 | { 237 | errorMsg = $"Encountered exception while checking for last checkpointed sequence number for blob " + 238 | $"on Partition: '{partitionId}', EventHub: '{_eventHubName}', Consumer Group: '{_consumerGroup}'. Error: {e.Message}"; 239 | } 240 | } 241 | 242 | return new Tuple(blobPartitionLease, errorMsg); 243 | } 244 | 245 | // Get the number of unprocessed events by deriving the delta between the server side info and the partition lease info, 246 | private long GetUnprocessedEventCount(EventHubPartitionRuntimeInformation partitionInfo, BlobPartitionLease partitionLeaseInfo) 247 | { 248 | long partitionLeaseInfoSequenceNumber = partitionLeaseInfo.SequenceNumber ?? 0; 249 | 250 | // This handles two scenarios: 251 | // 1. If the partition has received its first message, Offset will be null and LastEnqueuedSequenceNumber will be 0 252 | // 2. If there are no instances set to process messages, Offset will be null and LastEnqueuedSequenceNumber will be >= 0 253 | if (partitionLeaseInfo.Offset == null && partitionInfo.LastEnqueuedSequenceNumber >= 0) 254 | { 255 | return (partitionInfo.LastEnqueuedSequenceNumber + 1); 256 | } 257 | 258 | if (partitionInfo.LastEnqueuedSequenceNumber > partitionLeaseInfoSequenceNumber) 259 | { 260 | return (partitionInfo.LastEnqueuedSequenceNumber - partitionLeaseInfoSequenceNumber); 261 | } 262 | 263 | // Partition is a circular buffer, so it is possible that 264 | // LastEnqueuedSequenceNumber < SequenceNumber 265 | long count = 0; 266 | unchecked 267 | { 268 | count = (long.MaxValue - partitionInfo.LastEnqueuedSequenceNumber) + partitionLeaseInfoSequenceNumber; 269 | } 270 | 271 | // It's possible for checkpointing to be ahead of the partition's LastEnqueuedSequenceNumber, 272 | // especially if checkpointing is happening often and load is very low. 273 | // If count is negative, we need to know that this read is invalid, so return 0. 274 | // e.g., (9223372036854775807 - 10) + 11 = -9223372036854775808 275 | return (count < 0) ? 0 : count; 276 | } 277 | 278 | ScaleStatus IScaleMonitor.GetScaleStatus(ScaleStatusContext context) 279 | { 280 | return GetScaleStatusCore(context.WorkerCount, context.Metrics?.Cast().ToArray()); 281 | } 282 | 283 | public ScaleStatus GetScaleStatus(ScaleStatusContext context) 284 | { 285 | return GetScaleStatusCore(context.WorkerCount, context.Metrics?.ToArray()); 286 | } 287 | 288 | private ScaleStatus GetScaleStatusCore(int workerCount, EventHubsTriggerMetrics[] metrics) 289 | { 290 | ScaleStatus status = new ScaleStatus 291 | { 292 | Vote = ScaleVote.None 293 | }; 294 | 295 | const int NumberOfSamplesToConsider = 5; 296 | 297 | // Unable to determine the correct vote with no metrics. 298 | if (metrics == null || metrics.Length == 0) 299 | { 300 | return status; 301 | } 302 | 303 | // We shouldn't assign more workers than there are partitions 304 | // This check is first, because it is independent of load or number of samples. 305 | int partitionCount = metrics.Last().PartitionCount; 306 | if (partitionCount > 0 && partitionCount < workerCount) 307 | { 308 | status.Vote = ScaleVote.ScaleIn; 309 | _logger.LogInformation($"WorkerCount ({workerCount}) > PartitionCount ({partitionCount})."); 310 | _logger.LogInformation($"Number of instances ({workerCount}) is too high relative to number " + 311 | $"of partitions ({partitionCount}) for EventHubs entity ({_eventHubName}, {_consumerGroup})."); 312 | return status; 313 | } 314 | 315 | // At least 5 samples are required to make a scale decision for the rest of the checks. 316 | if (metrics.Length < NumberOfSamplesToConsider) 317 | { 318 | return status; 319 | } 320 | 321 | // Maintain a minimum ratio of 1 worker per 1,000 unprocessed events. 322 | long latestEventCount = metrics.Last().EventCount; 323 | if (latestEventCount > workerCount * 1000) 324 | { 325 | status.Vote = ScaleVote.ScaleOut; 326 | _logger.LogInformation($"EventCount ({latestEventCount}) > WorkerCount ({workerCount}) * 1,000."); 327 | _logger.LogInformation($"Event count ({latestEventCount}) for EventHubs entity ({_eventHubName}, {_consumerGroup}) " + 328 | $"is too high relative to the number of instances ({workerCount})."); 329 | return status; 330 | } 331 | 332 | // Check to see if the EventHub has been empty for a while. Only if all metrics samples are empty do we scale down. 333 | bool isIdle = metrics.All(m => m.EventCount == 0); 334 | if (isIdle) 335 | { 336 | status.Vote = ScaleVote.ScaleIn; 337 | _logger.LogInformation($"'{_eventHubName}' is idle."); 338 | return status; 339 | } 340 | 341 | // Samples are in chronological order. Check for a continuous increase in unprocessed event count. 342 | // If detected, this results in an automatic scale out for the site container. 343 | if (metrics[0].EventCount > 0) 344 | { 345 | bool eventCountIncreasing = 346 | IsTrueForLastN( 347 | metrics, 348 | NumberOfSamplesToConsider, 349 | (prev, next) => prev.EventCount < next.EventCount); 350 | if (eventCountIncreasing) 351 | { 352 | status.Vote = ScaleVote.ScaleOut; 353 | _logger.LogInformation($"Event count is increasing for '{_eventHubName}'."); 354 | return status; 355 | } 356 | } 357 | 358 | bool eventCountDecreasing = 359 | IsTrueForLastN( 360 | metrics, 361 | NumberOfSamplesToConsider, 362 | (prev, next) => prev.EventCount > next.EventCount); 363 | if (eventCountDecreasing) 364 | { 365 | status.Vote = ScaleVote.ScaleIn; 366 | _logger.LogInformation($"Event count is decreasing for '{_eventHubName}'."); 367 | return status; 368 | } 369 | 370 | _logger.LogInformation($"EventHubs entity '{_eventHubName}' is steady."); 371 | 372 | return status; 373 | } 374 | 375 | private static bool IsTrueForLastN(IList samples, int count, Func predicate) 376 | { 377 | // Walks through the list from left to right starting at len(samples) - count. 378 | for (int i = samples.Count - count; i < samples.Count - 1; i++) 379 | { 380 | if (!predicate(samples[i], samples[i + 1])) 381 | { 382 | return false; 383 | } 384 | } 385 | 386 | return true; 387 | } 388 | 389 | // The BlobPartitionLease class used for reading blob lease data for a partition from storage. 390 | // Sample blob lease entry in storage: 391 | // {"PartitionId":"0","Owner":"681d365b-de1b-4288-9733-76294e17daf0","Token":"2d0c4276-827d-4ca4-a345-729caeca3b82","Epoch":386,"Offset":"8591964920","SequenceNumber":960180} 392 | private class BlobPartitionLease 393 | { 394 | public string PartitionId { get; set; } 395 | 396 | public string Owner { get; set; } 397 | 398 | public string Token { get; set; } 399 | 400 | public long? Epoch { get; set; } 401 | 402 | public string Offset { get; set; } 403 | 404 | public long? SequenceNumber { get; set; } 405 | } 406 | } 407 | } 408 | -------------------------------------------------------------------------------- /src/Microsoft.Azure.WebJobs.Extensions.EventHubs/Listeners/EventHubsTriggerMetrics.cs: -------------------------------------------------------------------------------- 1 | // Copyright (c) .NET Foundation. All rights reserved. 2 | // Licensed under the MIT License. See License.txt in the project root for license information. 3 | 4 | using Microsoft.Azure.WebJobs.Host.Scale; 5 | using System; 6 | using System.Collections.Generic; 7 | using System.Text; 8 | 9 | namespace Microsoft.Azure.WebJobs.EventHubs.Listeners 10 | { 11 | internal class EventHubsTriggerMetrics : ScaleMetrics 12 | { 13 | /// 14 | /// The total number of unprocessed events across all partitions. 15 | /// 16 | public long EventCount { get; set; } 17 | 18 | /// 19 | /// The number of partitions. 20 | /// 21 | public int PartitionCount { get; set; } 22 | } 23 | } 24 | -------------------------------------------------------------------------------- /src/Microsoft.Azure.WebJobs.Extensions.EventHubs/Properties/AssemblyInfo.cs: -------------------------------------------------------------------------------- 1 | // Copyright (c) .NET Foundation. All rights reserved. 2 | // Licensed under the MIT License. See License.txt in the project root for license information. 3 | 4 | using System.Runtime.CompilerServices; 5 | 6 | [assembly: InternalsVisibleTo("Microsoft.Azure.WebJobs.EventHubs.Tests, PublicKey=0024000004800000940000000602000000240000525341310004000001000100b5fc90e7027f67871e773a8fde8938c81dd402ba65b9201d60593e96c492651e889cc13f1415ebb53fac1131ae0bd333c5ee6021672d9718ea31a8aebd0da0072f25d87dba6fc90ffd598ed4da35e44c398c454307e8e33b8426143daec9f596836f97c8f74750e5975c64e2189f45def46b2a2b1247adc3652bf5c308055da9")] 7 | [assembly: InternalsVisibleTo("DynamicProxyGenAssembly2, PublicKey=0024000004800000940000000602000000240000525341310004000001000100c547cac37abd99c8db225ef2f6c8a3602f3b3606cc9891605d02baa56104f4cfc0734aa39b93bf7852f7d9266654753cc297e7d2edfe0bac1cdcf9f717241550e0a7b191195b7667bb4f64bcb8e2121380fd1d9d46ad2d92d2d15605093924cceaf74c4861eff62abf69b9291ed0a340e113be11e6a7d3113e92484cf7045cc7")] 8 | -------------------------------------------------------------------------------- /src/Microsoft.Azure.WebJobs.Extensions.EventHubs/Triggers/EventHubAsyncCollector.cs: -------------------------------------------------------------------------------- 1 | // Copyright (c) .NET Foundation. All rights reserved. 2 | // Licensed under the MIT License. See License.txt in the project root for license information. 3 | 4 | using System; 5 | using System.Collections.Generic; 6 | using System.Linq; 7 | using System.Threading; 8 | using System.Threading.Tasks; 9 | using Microsoft.Azure.EventHubs; 10 | using Microsoft.Azure.WebJobs.Logging; 11 | using Microsoft.Extensions.Logging; 12 | 13 | namespace Microsoft.Azure.WebJobs.EventHubs 14 | { 15 | /// 16 | /// Core object to send events to EventHub. 17 | /// Any user parameter that sends EventHub events will eventually get bound to this object. 18 | /// This will queue events and send in batches, also keeping under the 1024kb event hub limit per batch. 19 | /// 20 | internal class EventHubAsyncCollector : IAsyncCollector 21 | { 22 | private readonly EventHubClient _client; 23 | 24 | private readonly Dictionary _partitions = new Dictionary(); 25 | 26 | private const int BatchSize = 100; 27 | 28 | // Suggested to use 1008k instead of 1024k to leave padding room for headers. 29 | private const int MaxByteSize = 1008 * 1024; 30 | 31 | private readonly ILogger _logger; 32 | 33 | /// 34 | /// Create a sender around the given client. 35 | /// 36 | /// 37 | public EventHubAsyncCollector(EventHubClient client, ILoggerFactory loggerFactory) 38 | { 39 | if (client == null) 40 | { 41 | throw new ArgumentNullException("client"); 42 | } 43 | _client = client; 44 | _logger = loggerFactory?.CreateLogger(LogCategories.Executor); 45 | } 46 | 47 | /// 48 | /// Add an event. 49 | /// 50 | /// The event to add 51 | /// a cancellation token. 52 | /// 53 | public async Task AddAsync(EventData item, CancellationToken cancellationToken = default(CancellationToken)) 54 | { 55 | if (item == null) 56 | { 57 | throw new ArgumentNullException("item"); 58 | } 59 | 60 | string key = item.SystemProperties?.PartitionKey ?? string.Empty; 61 | 62 | PartitionCollector partition; 63 | lock (_partitions) 64 | { 65 | if (!_partitions.TryGetValue(key, out partition)) 66 | { 67 | partition = new PartitionCollector(this); 68 | _partitions[key] = partition; 69 | } 70 | } 71 | await partition.AddAsync(item, cancellationToken); 72 | } 73 | 74 | /// 75 | /// synchronously flush events that have been queued up via AddAsync. 76 | /// 77 | /// a cancellation token 78 | public async Task FlushAsync(CancellationToken cancellationToken = default(CancellationToken)) 79 | { 80 | while (true) 81 | { 82 | PartitionCollector partition; 83 | lock (_partitions) 84 | { 85 | if (_partitions.Count == 0) 86 | { 87 | return; 88 | } 89 | var kv = _partitions.First(); 90 | partition = kv.Value; 91 | _partitions.Remove(kv.Key); 92 | } 93 | 94 | await partition.FlushAsync(cancellationToken); 95 | } 96 | } 97 | 98 | /// 99 | /// Send the batch of events. All items in the batch will have the same partition key. 100 | /// 101 | /// the set of events to send 102 | protected virtual async Task SendBatchAsync(IEnumerable batch) 103 | { 104 | _logger?.LogDebug("Sending events to EventHub"); 105 | await _client.SendAsync(batch); 106 | } 107 | 108 | // A per-partition sender 109 | private class PartitionCollector : IAsyncCollector 110 | { 111 | private readonly EventHubAsyncCollector _parent; 112 | 113 | private List _list = new List(); 114 | 115 | // total size of bytes in _list that we'll be sending in this batch. 116 | private int _currentByteSize = 0; 117 | 118 | public PartitionCollector(EventHubAsyncCollector parent) 119 | { 120 | this._parent = parent; 121 | } 122 | 123 | /// 124 | /// Add an event. 125 | /// 126 | /// The event to add 127 | /// a cancellation token. 128 | /// 129 | public async Task AddAsync(EventData item, CancellationToken cancellationToken = default(CancellationToken)) 130 | { 131 | if (item == null) 132 | { 133 | throw new ArgumentNullException("item"); 134 | } 135 | 136 | while (true) 137 | { 138 | lock (_list) 139 | { 140 | var size = (int)item.Body.Count; 141 | 142 | if (size > MaxByteSize) 143 | { 144 | // Single event is too large to add. 145 | string msg = string.Format("Event is too large. Event is approximately {0}b and max size is {1}b", size, MaxByteSize); 146 | throw new InvalidOperationException(msg); 147 | } 148 | 149 | bool flush = (_currentByteSize + size > MaxByteSize) || (_list.Count >= BatchSize); 150 | if (!flush) 151 | { 152 | _list.Add(item); 153 | _currentByteSize += size; 154 | return; 155 | } 156 | // We should flush. 157 | // Release the lock, flush, and then loop around and try again. 158 | } 159 | 160 | await this.FlushAsync(cancellationToken); 161 | } 162 | } 163 | 164 | /// 165 | /// synchronously flush events that have been queued up via AddAsync. 166 | /// 167 | /// a cancellation token 168 | public async Task FlushAsync(CancellationToken cancellationToken = default(CancellationToken)) 169 | { 170 | EventData[] batch = null; 171 | lock (_list) 172 | { 173 | batch = _list.ToArray(); 174 | _list.Clear(); 175 | _currentByteSize = 0; 176 | } 177 | 178 | if (batch.Length > 0) 179 | { 180 | await _parent.SendBatchAsync(batch); 181 | 182 | // Dispose all messages to help with memory pressure. If this is missed, the finalizer thread will still get them. 183 | foreach (var msg in batch) 184 | { 185 | msg.Dispose(); 186 | } 187 | } 188 | } 189 | } 190 | } 191 | } -------------------------------------------------------------------------------- /src/Microsoft.Azure.WebJobs.Extensions.EventHubs/Triggers/EventHubTriggerAttributeBindingProvider.cs: -------------------------------------------------------------------------------- 1 | // Copyright (c) .NET Foundation. All rights reserved. 2 | // Licensed under the MIT License. See License.txt in the project root for license information. 3 | 4 | using System; 5 | using System.Reflection; 6 | using System.Threading.Tasks; 7 | using Microsoft.Azure.EventHubs; 8 | using Microsoft.Azure.WebJobs.Host; 9 | using Microsoft.Azure.WebJobs.Host.Bindings; 10 | using Microsoft.Azure.WebJobs.Host.Listeners; 11 | using Microsoft.Azure.WebJobs.Host.Triggers; 12 | using Microsoft.Azure.WebJobs.Logging; 13 | using Microsoft.Extensions.Configuration; 14 | using Microsoft.Extensions.Logging; 15 | using Microsoft.Extensions.Options; 16 | 17 | namespace Microsoft.Azure.WebJobs.EventHubs 18 | { 19 | internal class EventHubTriggerAttributeBindingProvider : ITriggerBindingProvider 20 | { 21 | private readonly INameResolver _nameResolver; 22 | private readonly ILogger _logger; 23 | private readonly IConfiguration _config; 24 | private readonly IOptions _options; 25 | private readonly IConverterManager _converterManager; 26 | 27 | public EventHubTriggerAttributeBindingProvider( 28 | IConfiguration configuration, 29 | INameResolver nameResolver, 30 | IConverterManager converterManager, 31 | IOptions options, 32 | ILoggerFactory loggerFactory) 33 | { 34 | _config = configuration; 35 | _nameResolver = nameResolver; 36 | _converterManager = converterManager; 37 | _options = options; 38 | _logger = loggerFactory?.CreateLogger(LogCategories.CreateTriggerCategory("EventHub")); 39 | } 40 | 41 | [System.Diagnostics.CodeAnalysis.SuppressMessage("Microsoft.Reliability", "CA2000:Dispose objects before losing scope")] 42 | public Task TryCreateAsync(TriggerBindingProviderContext context) 43 | { 44 | if (context == null) 45 | { 46 | throw new ArgumentNullException("context"); 47 | } 48 | 49 | ParameterInfo parameter = context.Parameter; 50 | EventHubTriggerAttribute attribute = parameter.GetCustomAttribute(inherit: false); 51 | 52 | if (attribute == null) 53 | { 54 | return Task.FromResult(null); 55 | } 56 | 57 | string resolvedEventHubName = _nameResolver.ResolveWholeString(attribute.EventHubName); 58 | 59 | string consumerGroup = attribute.ConsumerGroup ?? PartitionReceiver.DefaultConsumerGroupName; 60 | string resolvedConsumerGroup = _nameResolver.ResolveWholeString(consumerGroup); 61 | 62 | string connectionString = null; 63 | if (!string.IsNullOrWhiteSpace(attribute.Connection)) 64 | { 65 | attribute.Connection = _nameResolver.ResolveWholeString(attribute.Connection); 66 | connectionString = _config.GetConnectionStringOrSetting(attribute.Connection); 67 | _options.Value.AddReceiver(resolvedEventHubName, connectionString); 68 | } 69 | 70 | var eventHostListener = _options.Value.GetEventProcessorHost(_config, resolvedEventHubName, resolvedConsumerGroup); 71 | 72 | string storageConnectionString = _config.GetWebJobsConnectionString(ConnectionStringNames.Storage); 73 | 74 | Func> createListener = 75 | (factoryContext, singleDispatch) => 76 | { 77 | IListener listener = new EventHubListener( 78 | factoryContext.Descriptor.Id, 79 | resolvedEventHubName, 80 | resolvedConsumerGroup, 81 | connectionString, 82 | storageConnectionString, 83 | factoryContext.Executor, 84 | eventHostListener, 85 | singleDispatch, 86 | _options.Value, 87 | _logger); 88 | return Task.FromResult(listener); 89 | }; 90 | 91 | ITriggerBinding binding = BindingFactory.GetTriggerBinding(new EventHubTriggerBindingStrategy(), parameter, _converterManager, createListener); 92 | return Task.FromResult(binding); 93 | } 94 | } // end class 95 | } -------------------------------------------------------------------------------- /src/Microsoft.Azure.WebJobs.Extensions.EventHubs/Triggers/EventHubTriggerInput.cs: -------------------------------------------------------------------------------- 1 | // Copyright (c) .NET Foundation. All rights reserved. 2 | // Licensed under the MIT License. See License.txt in the project root for license information. 3 | 4 | using Microsoft.Azure.EventHubs; 5 | using Microsoft.Azure.EventHubs.Processor; 6 | using System.Collections.Generic; 7 | 8 | namespace Microsoft.Azure.WebJobs.EventHubs 9 | { 10 | // The core object we get when an EventHub is triggered. 11 | // This gets converted to the user type (EventData, string, poco, etc) 12 | internal sealed class EventHubTriggerInput 13 | { 14 | // If != -1, then only process a single event in this batch. 15 | private int _selector = -1; 16 | 17 | internal EventData[] Events { get; set; } 18 | 19 | internal PartitionContext PartitionContext { get; set; } 20 | 21 | public bool IsSingleDispatch 22 | { 23 | get 24 | { 25 | return _selector != -1; 26 | } 27 | } 28 | 29 | public static EventHubTriggerInput New(EventData eventData) 30 | { 31 | return new EventHubTriggerInput 32 | { 33 | PartitionContext = null, 34 | Events = new EventData[] 35 | { 36 | eventData 37 | }, 38 | _selector = 0, 39 | }; 40 | } 41 | 42 | public EventHubTriggerInput GetSingleEventTriggerInput(int idx) 43 | { 44 | return new EventHubTriggerInput 45 | { 46 | Events = this.Events, 47 | PartitionContext = this.PartitionContext, 48 | _selector = idx 49 | }; 50 | } 51 | 52 | public EventData GetSingleEventData() 53 | { 54 | return this.Events[this._selector]; 55 | } 56 | 57 | public Dictionary GetTriggerDetails(PartitionContext context) 58 | { 59 | if (Events.Length == 0) 60 | { 61 | return new Dictionary(); 62 | } 63 | 64 | string offset, enqueueTimeUtc, sequenceNumber; 65 | if (IsSingleDispatch) 66 | { 67 | offset = Events[0].SystemProperties?.Offset; 68 | enqueueTimeUtc = Events[0].SystemProperties?.EnqueuedTimeUtc.ToString("o"); 69 | sequenceNumber = Events[0].SystemProperties?.SequenceNumber.ToString(); 70 | } 71 | else 72 | { 73 | EventData first = Events[0]; 74 | EventData last = Events[Events.Length - 1]; 75 | 76 | offset = $"{first.SystemProperties?.Offset}-{last.SystemProperties?.Offset}"; 77 | enqueueTimeUtc = $"{first.SystemProperties?.EnqueuedTimeUtc.ToString("o")}-{last.SystemProperties?.EnqueuedTimeUtc.ToString("o")}"; 78 | sequenceNumber = $"{first.SystemProperties?.SequenceNumber}-{last.SystemProperties?.SequenceNumber}"; 79 | } 80 | 81 | return new Dictionary() 82 | { 83 | { "PartitionId", context.PartitionId }, 84 | { "Offset", offset }, 85 | { "EnqueueTimeUtc", enqueueTimeUtc }, 86 | { "SequenceNumber", sequenceNumber }, 87 | { "Count", Events.Length.ToString()} 88 | }; 89 | } 90 | } 91 | } -------------------------------------------------------------------------------- /src/Microsoft.Azure.WebJobs.Extensions.EventHubs/Triggers/EventHubTriggerInputBindingStrategy.cs: -------------------------------------------------------------------------------- 1 | // Copyright (c) .NET Foundation. All rights reserved. 2 | // Licensed under the MIT License. See License.txt in the project root for license information. 3 | 4 | using System; 5 | using System.Collections.Generic; 6 | using System.Text; 7 | using Microsoft.Azure.EventHubs; 8 | using Microsoft.Azure.EventHubs.Processor; 9 | using Microsoft.Azure.WebJobs.Host.Bindings; 10 | using Microsoft.Azure.WebJobs.Host.Triggers; 11 | 12 | namespace Microsoft.Azure.WebJobs.EventHubs 13 | { 14 | // Binding strategy for an event hub triggers. 15 | internal class EventHubTriggerBindingStrategy : ITriggerBindingStrategy 16 | { 17 | [System.Diagnostics.CodeAnalysis.SuppressMessage("Microsoft.Reliability", "CA2000:Dispose objects before losing scope")] 18 | public EventHubTriggerInput ConvertFromString(string input) 19 | { 20 | byte[] bytes = Encoding.UTF8.GetBytes(input); 21 | EventData eventData = new EventData(bytes); 22 | 23 | // Return a single event. Doesn't support multiple dispatch 24 | return EventHubTriggerInput.New(eventData); 25 | } 26 | 27 | // Single instance: Core --> EventData 28 | public EventData BindSingle(EventHubTriggerInput value, ValueBindingContext context) 29 | { 30 | if (value == null) 31 | { 32 | throw new ArgumentNullException("value"); 33 | } 34 | return value.GetSingleEventData(); 35 | } 36 | 37 | public EventData[] BindMultiple(EventHubTriggerInput value, ValueBindingContext context) 38 | { 39 | if (value == null) 40 | { 41 | throw new ArgumentNullException("value"); 42 | } 43 | return value.Events; 44 | } 45 | 46 | public Dictionary GetBindingContract(bool isSingleDispatch = true) 47 | { 48 | var contract = new Dictionary(StringComparer.OrdinalIgnoreCase); 49 | contract.Add("PartitionContext", typeof(PartitionContext)); 50 | 51 | AddBindingContractMember(contract, "PartitionKey", typeof(string), isSingleDispatch); 52 | AddBindingContractMember(contract, "Offset", typeof(string), isSingleDispatch); 53 | AddBindingContractMember(contract, "SequenceNumber", typeof(long), isSingleDispatch); 54 | AddBindingContractMember(contract, "EnqueuedTimeUtc", typeof(DateTime), isSingleDispatch); 55 | AddBindingContractMember(contract, "Properties", typeof(IDictionary), isSingleDispatch); 56 | AddBindingContractMember(contract, "SystemProperties", typeof(IDictionary), isSingleDispatch); 57 | 58 | return contract; 59 | } 60 | 61 | private static void AddBindingContractMember(Dictionary contract, string name, Type type, bool isSingleDispatch) 62 | { 63 | if (!isSingleDispatch) 64 | { 65 | name += "Array"; 66 | } 67 | contract.Add(name, isSingleDispatch ? type : type.MakeArrayType()); 68 | } 69 | 70 | public Dictionary GetBindingData(EventHubTriggerInput value) 71 | { 72 | if (value == null) 73 | { 74 | throw new ArgumentNullException("value"); 75 | } 76 | 77 | var bindingData = new Dictionary(StringComparer.OrdinalIgnoreCase); 78 | SafeAddValue(() => bindingData.Add(nameof(value.PartitionContext), value.PartitionContext)); 79 | 80 | if (value.IsSingleDispatch) 81 | { 82 | AddBindingData(bindingData, value.GetSingleEventData()); 83 | } 84 | else 85 | { 86 | AddBindingData(bindingData, value.Events); 87 | } 88 | 89 | return bindingData; 90 | } 91 | 92 | internal static void AddBindingData(Dictionary bindingData, EventData[] events) 93 | { 94 | int length = events.Length; 95 | var partitionKeys = new string[length]; 96 | var offsets = new string[length]; 97 | var sequenceNumbers = new long[length]; 98 | var enqueuedTimesUtc = new DateTime[length]; 99 | var properties = new IDictionary[length]; 100 | var systemProperties = new IDictionary[length]; 101 | 102 | SafeAddValue(() => bindingData.Add("PartitionKeyArray", partitionKeys)); 103 | SafeAddValue(() => bindingData.Add("OffsetArray", offsets)); 104 | SafeAddValue(() => bindingData.Add("SequenceNumberArray", sequenceNumbers)); 105 | SafeAddValue(() => bindingData.Add("EnqueuedTimeUtcArray", enqueuedTimesUtc)); 106 | SafeAddValue(() => bindingData.Add("PropertiesArray", properties)); 107 | SafeAddValue(() => bindingData.Add("SystemPropertiesArray", systemProperties)); 108 | 109 | for (int i = 0; i < events.Length; i++) 110 | { 111 | partitionKeys[i] = events[i].SystemProperties?.PartitionKey; 112 | offsets[i] = events[i].SystemProperties?.Offset; 113 | sequenceNumbers[i] = events[i].SystemProperties?.SequenceNumber ?? 0; 114 | enqueuedTimesUtc[i] = events[i].SystemProperties?.EnqueuedTimeUtc ?? DateTime.MinValue; 115 | properties[i] = events[i].Properties; 116 | systemProperties[i] = events[i].SystemProperties?.ToDictionary(); 117 | } 118 | } 119 | 120 | private static void AddBindingData(Dictionary bindingData, EventData eventData) 121 | { 122 | SafeAddValue(() => bindingData.Add(nameof(eventData.SystemProperties.PartitionKey), eventData.SystemProperties?.PartitionKey)); 123 | SafeAddValue(() => bindingData.Add(nameof(eventData.SystemProperties.Offset), eventData.SystemProperties?.Offset)); 124 | SafeAddValue(() => bindingData.Add(nameof(eventData.SystemProperties.SequenceNumber), eventData.SystemProperties?.SequenceNumber ?? 0)); 125 | SafeAddValue(() => bindingData.Add(nameof(eventData.SystemProperties.EnqueuedTimeUtc), eventData.SystemProperties?.EnqueuedTimeUtc ?? DateTime.MinValue)); 126 | SafeAddValue(() => bindingData.Add(nameof(eventData.Properties), eventData.Properties)); 127 | SafeAddValue(() => bindingData.Add(nameof(eventData.SystemProperties), eventData.SystemProperties?.ToDictionary())); 128 | } 129 | 130 | private static void SafeAddValue(Action addValue) 131 | { 132 | try 133 | { 134 | addValue(); 135 | } 136 | catch 137 | { 138 | // some message propery getters can throw, based on the 139 | // state of the message 140 | } 141 | } 142 | } 143 | } -------------------------------------------------------------------------------- /src/Microsoft.Azure.WebJobs.Extensions.EventHubs/Utility.cs: -------------------------------------------------------------------------------- 1 | // Copyright (c) .NET Foundation. All rights reserved. 2 | // Licensed under the MIT License. See License.txt in the project root for license information. 3 | 4 | using System; 5 | using Microsoft.Azure.EventHubs; 6 | using Microsoft.Azure.EventHubs.Processor; 7 | using Microsoft.Extensions.Logging; 8 | using Microsoft.WindowsAzure.Storage; 9 | using LogLevel = Microsoft.Extensions.Logging.LogLevel; 10 | 11 | namespace Microsoft.Azure.WebJobs.EventHubs 12 | { 13 | internal class Utility 14 | { 15 | public static void LogException(Exception ex, string message, ILogger logger) 16 | { 17 | try 18 | { 19 | // Sometimes EventHub SDK aggregates an exception 20 | AggregateException ae = ex as AggregateException; 21 | if (ae != null && ae.InnerExceptions != null && ae.InnerExceptions.Count == 1) 22 | { 23 | ex = ae.InnerExceptions[0]; 24 | } 25 | 26 | LogLevel logLevel = GetLevel(ex); 27 | if (logLevel == LogLevel.Information) 28 | { 29 | message = $"{message} An exception of type '{ex.GetType().Name}' was thrown. This exception type is typically a result of Event Hub processor rebalancing or a transient error and can be safely ignored."; 30 | } 31 | logger?.Log(logLevel, 0, message, ex, (s, exc) => message); 32 | } 33 | catch 34 | { 35 | // best effort logging 36 | } 37 | } 38 | 39 | private static LogLevel GetLevel(Exception ex) 40 | { 41 | if (ex == null) 42 | { 43 | throw new ArgumentNullException("ex"); 44 | } 45 | 46 | if (ex is ReceiverDisconnectedException || ex is LeaseLostException 47 | || IsConflictLeaseIdMismatchWithLeaseOperation(ex)) 48 | { 49 | // For EventProcessorHost these exceptions can happen as part 50 | // of normal partition balancing across instances, so we want to 51 | // trace them, but not treat them as errors. 52 | return LogLevel.Information; 53 | } 54 | 55 | var ehex = ex as EventHubsException; 56 | if (!(ex is OperationCanceledException) && (ehex == null || !ehex.IsTransient)) 57 | { 58 | // any non-transient exceptions or unknown exception types 59 | // we want to log as errors 60 | return LogLevel.Error; 61 | } 62 | else 63 | { 64 | // transient messaging errors we log as info so we have a record 65 | // of them, but we don't treat them as actual errors 66 | return LogLevel.Information; 67 | } 68 | } 69 | 70 | public static bool IsConflictLeaseIdMismatchWithLeaseOperation(Exception ex) 71 | { 72 | StorageException exception = ex as StorageException; 73 | if (exception == null) 74 | { 75 | return false; 76 | } 77 | 78 | RequestResult result = exception.RequestInformation; 79 | 80 | if (result == null) 81 | { 82 | return false; 83 | } 84 | 85 | if (result.HttpStatusCode != 409) 86 | { 87 | return false; 88 | } 89 | 90 | StorageExtendedErrorInformation extendedInformation = result.ExtendedErrorInformation; 91 | 92 | if (extendedInformation == null) 93 | { 94 | return false; 95 | } 96 | 97 | return extendedInformation.ErrorCode == "LeaseIdMismatchWithLeaseOperation"; 98 | } 99 | } 100 | } 101 | -------------------------------------------------------------------------------- /src/Microsoft.Azure.WebJobs.Extensions.EventHubs/WebJobs.Extensions.EventHubs.csproj: -------------------------------------------------------------------------------- 1 |  2 | 3 | 4 | netstandard2.0 5 | Microsoft.Azure.WebJobs.EventHubs 6 | Microsoft.Azure.WebJobs.EventHubs 7 | Microsoft.Azure.WebJobs.Extensions.EventHubs 8 | Microsoft Azure WebJobs SDK EventHubs Extension 9 | 4.3.1 10 | N/A 11 | $(Version) Commit hash: $(CommitHash) 12 | Microsoft 13 | Microsoft 14 | © Microsoft Corporation. All rights reserved. 15 | True 16 | MIT 17 | webjobs.png 18 | http://go.microsoft.com/fwlink/?LinkID=320972 19 | git 20 | https://github.com/Azure/azure-functions-servicebus-extension 21 | true 22 | true 23 | ..\..\sign.snk 24 | ..\..\src.ruleset 25 | true 26 | 27 | 28 | 29 | false 30 | true 31 | 32 | 33 | 34 | 35 | false 36 | true 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | all 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | -------------------------------------------------------------------------------- /src/Microsoft.Azure.WebJobs.Extensions.EventHubs/webjobs.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/azure-functions-eventhubs-extension/1cf1006f731f9015d5e1eadb4e6798b63d78313c/src/Microsoft.Azure.WebJobs.Extensions.EventHubs/webjobs.png -------------------------------------------------------------------------------- /stylecop.json: -------------------------------------------------------------------------------- 1 | { 2 | // ACTION REQUIRED: This file was automatically added to your project, but it 3 | // will not take effect until additional steps are taken to enable it. See the 4 | // following page for additional information: 5 | // 6 | // https://github.com/DotNetAnalyzers/StyleCopAnalyzers/blob/master/documentation/EnableConfiguration.md 7 | 8 | "$schema": "https://raw.githubusercontent.com/DotNetAnalyzers/StyleCopAnalyzers/master/StyleCop.Analyzers/StyleCop.Analyzers/Settings/stylecop.schema.json", 9 | "settings": { 10 | "documentationRules": { 11 | "companyName": ".NET Foundation", 12 | "copyrightText": "Copyright (c) .NET Foundation. All rights reserved.\r\nLicensed under the MIT License. See License.txt in the project root for license information.", 13 | "xmlHeader": false, 14 | "documentInterfaces": false, 15 | "documentInternalElements": false, 16 | "documentExposedElements": false, 17 | "documentPrivateElements": false, 18 | "documentPrivateFields": false 19 | }, 20 | "orderingRules": { 21 | "usingDirectivesPlacement": "outsideNamespace" 22 | } 23 | } 24 | } 25 | -------------------------------------------------------------------------------- /test/Microsoft.Azure.WebJobs.Extensions.EventHubs.Tests/EventHubAsyncCollectorTests.cs: -------------------------------------------------------------------------------- 1 | // Copyright (c) .NET Foundation. All rights reserved. 2 | // Licensed under the MIT License. See License.txt in the project root for license information. 3 | 4 | using System; 5 | using System.Collections.Generic; 6 | using System.Linq; 7 | using System.Text; 8 | using System.Threading; 9 | using System.Threading.Tasks; 10 | using Microsoft.Azure.EventHubs; 11 | using Microsoft.Azure.WebJobs.Host.TestCommon; 12 | using Xunit; 13 | 14 | using static Microsoft.Azure.EventHubs.EventData; 15 | 16 | namespace Microsoft.Azure.WebJobs.EventHubs.UnitTests 17 | { 18 | public class EventHubAsyncCollectorTests 19 | { 20 | [Fact] 21 | public void NullArgumentCheck() 22 | { 23 | Assert.Throws(() => new EventHubAsyncCollector(null, null)); 24 | } 25 | 26 | public EventData CreateEvent(byte[] body, string partitionKey) 27 | { 28 | var data = new EventData(body); 29 | IDictionary sysProps = TestHelpers.New(); 30 | sysProps["x-opt-partition-key"] = partitionKey; 31 | TestHelpers.SetField(data, "SystemProperties", sysProps); 32 | return data; 33 | } 34 | 35 | [Fact] 36 | public async Task SendMultiplePartitions() 37 | { 38 | var collector = new TestEventHubAsyncCollector(); 39 | 40 | await collector.AddAsync(this.CreateEvent(new byte[] { 1 }, "pk1")); 41 | await collector.AddAsync(CreateEvent(new byte[] { 2 }, "pk2")); 42 | 43 | // Not physically sent yet since we haven't flushed 44 | Assert.Empty(collector.SentEvents); 45 | 46 | await collector.FlushAsync(); 47 | 48 | // Partitions aren't flushed in a specific order. 49 | Assert.Equal(2, collector.SentEvents.Count); 50 | var items = collector.SentEvents.ToArray(); 51 | 52 | var item0 = items[0]; 53 | var item1 = items[1]; 54 | Assert.Equal(3, item0[0] + item1[0]); // either order. 55 | } 56 | 57 | [Fact] 58 | public async Task NotSentUntilFlushed() 59 | { 60 | var collector = new TestEventHubAsyncCollector(); 61 | 62 | await collector.FlushAsync(); // should be nop. 63 | 64 | var payload = new byte[] { 1, 2, 3 }; 65 | var e1 = new EventData(payload); 66 | await collector.AddAsync(e1); 67 | 68 | // Not physically sent yet since we haven't flushed 69 | Assert.Empty(collector.SentEvents); 70 | 71 | await collector.FlushAsync(); 72 | Assert.Single(collector.SentEvents); 73 | Assert.Equal(payload, collector.SentEvents[0]); 74 | } 75 | 76 | // If we send enough events, that will eventually tip over and flush. 77 | [Fact] 78 | public async Task FlushAfterLotsOfSmallEvents() 79 | { 80 | var collector = new TestEventHubAsyncCollector(); 81 | 82 | // Sending a bunch of little events 83 | for (int i = 0; i < 150; i++) 84 | { 85 | var e1 = new EventData(new byte[] { 1, 2, 3 }); 86 | await collector.AddAsync(e1); 87 | } 88 | 89 | Assert.True(collector.SentEvents.Count > 0); 90 | } 91 | 92 | // If we send enough events, that will eventually tip over and flush. 93 | [Fact] 94 | public async Task FlushAfterSizeThreshold() 95 | { 96 | var collector = new TestEventHubAsyncCollector(); 97 | 98 | // Trip the 1024k EventHub limit. 99 | for (int i = 0; i < 50; i++) 100 | { 101 | var e1 = new EventData(new byte[10 * 1024]); 102 | await collector.AddAsync(e1); 103 | } 104 | 105 | // Not yet 106 | Assert.Empty(collector.SentEvents); 107 | 108 | // This will push it over the theshold 109 | for (int i = 0; i < 60; i++) 110 | { 111 | var e1 = new EventData(new byte[10 * 1024]); 112 | await collector.AddAsync(e1); 113 | } 114 | 115 | Assert.True(collector.SentEvents.Count > 0); 116 | } 117 | 118 | [Fact] 119 | public async Task CantSentGiantEvent() 120 | { 121 | var collector = new TestEventHubAsyncCollector(); 122 | 123 | // event hub max is 1024k payload. 124 | var hugePayload = new byte[1200 * 1024]; 125 | var e1 = new EventData(hugePayload); 126 | 127 | try 128 | { 129 | await collector.AddAsync(e1); 130 | Assert.False(true); 131 | } 132 | catch (InvalidOperationException e) 133 | { 134 | // Exact error message (and serialization byte size) is subject to change. 135 | Assert.Contains("Event is too large", e.Message); 136 | } 137 | 138 | // Verify we didn't queue anything 139 | await collector.FlushAsync(); 140 | Assert.Empty(collector.SentEvents); 141 | } 142 | 143 | [Fact] 144 | public async Task CantSendNullEvent() 145 | { 146 | var collector = new TestEventHubAsyncCollector(); 147 | 148 | await Assert.ThrowsAsync( 149 | async () => await collector.AddAsync(null)); 150 | } 151 | 152 | // Send lots of events from multiple threads and ensure that all events are precisely accounted for. 153 | [Fact] 154 | public async Task SendLotsOfEvents() 155 | { 156 | var collector = new TestEventHubAsyncCollector(); 157 | 158 | int numEvents = 1000; 159 | int numThreads = 10; 160 | 161 | HashSet expected = new HashSet(); 162 | 163 | // Send from different physical threads. 164 | Thread[] threads = new Thread[numThreads]; 165 | for (int iThread = 0; iThread < numThreads; iThread++) 166 | { 167 | var x = iThread; 168 | threads[x] = new Thread( 169 | () => 170 | { 171 | for (int i = 0; i < numEvents; i++) 172 | { 173 | var idx = (x * numEvents) + i; 174 | var payloadStr = idx.ToString(); 175 | var payload = Encoding.UTF8.GetBytes(payloadStr); 176 | lock (expected) 177 | { 178 | expected.Add(payloadStr); 179 | } 180 | collector.AddAsync(new EventData(payload)).Wait(); 181 | } 182 | }); 183 | } 184 | 185 | foreach (var thread in threads) 186 | { 187 | thread.Start(); 188 | } 189 | 190 | foreach (var thread in threads) 191 | { 192 | thread.Join(); 193 | } 194 | 195 | // Add more events to trip flushing of the original batch without calling Flush() 196 | const string ignore = "ignore"; 197 | byte[] ignoreBytes = Encoding.UTF8.GetBytes(ignore); 198 | for (int i = 0; i < 100; i++) 199 | { 200 | await collector.AddAsync(new EventData(ignoreBytes)); 201 | } 202 | 203 | // Verify that every event we sent is accounted for; and that there are no duplicates. 204 | int count = 0; 205 | foreach (var payloadBytes in collector.SentEvents) 206 | { 207 | count++; 208 | var payloadStr = Encoding.UTF8.GetString(payloadBytes); 209 | if (payloadStr == ignore) 210 | { 211 | continue; 212 | } 213 | 214 | if (!expected.Remove(payloadStr)) 215 | { 216 | // Already removed! 217 | Assert.False(true, "event payload occured multiple times"); 218 | } 219 | } 220 | 221 | Assert.Empty(expected); // Some events where missed. 222 | } 223 | 224 | internal class TestEventHubAsyncCollector : EventHubAsyncCollector 225 | { 226 | private static EventHubClient testClient = EventHubClient.CreateFromConnectionString(FakeConnectionString1); 227 | 228 | // EventData is disposed after sending. So track raw bytes; not the actual EventData. 229 | private List sentEvents = new List(); 230 | 231 | // A fake connection string for event hubs. This just needs to parse. It won't actually get invoked. 232 | private const string FakeConnectionString = "Endpoint=sb://test89123-ns-x.servicebus.windows.net/;SharedAccessKeyName=ReceiveRule;SharedAccessKey=secretkey;EntityPath=path2"; 233 | 234 | public TestEventHubAsyncCollector() 235 | : base(TestClient, null) 236 | { 237 | } 238 | 239 | public static EventHubClient TestClient { get => testClient; set => testClient = value; } 240 | 241 | public static string FakeConnectionString1 => FakeConnectionString; 242 | 243 | public List SentEvents { get => sentEvents; set => sentEvents = value; } 244 | 245 | protected override Task SendBatchAsync(IEnumerable batch) 246 | { 247 | // Assert they all have the same partition key (could be null) 248 | var partitionKey = batch.First().SystemProperties?.PartitionKey; 249 | foreach (var e in batch) 250 | { 251 | Assert.Equal(partitionKey, e.SystemProperties?.PartitionKey); 252 | } 253 | 254 | lock (SentEvents) 255 | { 256 | foreach (var e in batch) 257 | { 258 | var payloadBytes = e.Body.Array; 259 | Assert.NotNull(payloadBytes); 260 | SentEvents.Add(payloadBytes); 261 | } 262 | } 263 | 264 | return Task.FromResult(0); 265 | } 266 | } 267 | } // end class 268 | } 269 | -------------------------------------------------------------------------------- /test/Microsoft.Azure.WebJobs.Extensions.EventHubs.Tests/EventHubConfigurationTests.cs: -------------------------------------------------------------------------------- 1 | // Copyright (c) .NET Foundation. All rights reserved. 2 | // Licensed under the MIT License. See License.txt in the project root for license information. 3 | 4 | using System; 5 | using System.Collections.Generic; 6 | using System.Linq; 7 | using System.Reflection; 8 | using Microsoft.Azure.EventHubs; 9 | using Microsoft.Azure.EventHubs.Processor; 10 | using Microsoft.Azure.WebJobs.Host; 11 | using Microsoft.Azure.WebJobs.Host.TestCommon; 12 | using Microsoft.Azure.WebJobs.Host.Triggers; 13 | using Microsoft.Azure.WebJobs.Logging; 14 | using Microsoft.Extensions.DependencyInjection; 15 | using Microsoft.Extensions.Hosting; 16 | using Microsoft.Extensions.Logging; 17 | using Microsoft.Extensions.Options; 18 | using Microsoft.WindowsAzure.Storage; 19 | using Newtonsoft.Json.Linq; 20 | using Xunit; 21 | using LogLevel = Microsoft.Extensions.Logging.LogLevel; 22 | 23 | namespace Microsoft.Azure.WebJobs.EventHubs.UnitTests 24 | { 25 | public class EventHubConfigurationTests 26 | { 27 | private readonly ILoggerFactory _loggerFactory; 28 | private readonly TestLoggerProvider _loggerProvider; 29 | private readonly string _template = " An exception of type '{0}' was thrown. This exception type is typically a result of Event Hub processor rebalancing or a transient error and can be safely ignored."; 30 | private readonly string _optionStringInitialOffsetType = "FromEnqueuedTime"; 31 | private readonly string _optionStringInitialOffsetQneueuedTimeUTC = "2020-09-13T12:00Z"; 32 | 33 | public EventHubConfigurationTests() 34 | { 35 | _loggerFactory = new LoggerFactory(); 36 | 37 | _loggerProvider = new TestLoggerProvider(); 38 | _loggerFactory.AddProvider(_loggerProvider); 39 | } 40 | 41 | [Fact] 42 | public void ConfigureOptions_AppliesValuesCorrectly() 43 | { 44 | EventHubOptions options = CreateOptions(); 45 | 46 | Assert.Equal(123, options.EventProcessorOptions.MaxBatchSize); 47 | Assert.Equal(TimeSpan.FromSeconds(33), options.EventProcessorOptions.ReceiveTimeout); 48 | Assert.Equal(true, options.EventProcessorOptions.EnableReceiverRuntimeMetric); 49 | Assert.Equal(123, options.EventProcessorOptions.PrefetchCount); 50 | Assert.Equal(true, options.EventProcessorOptions.InvokeProcessorAfterReceiveTimeout); 51 | Assert.Equal(5, options.BatchCheckpointFrequency); 52 | Assert.Equal(31, options.PartitionManagerOptions.LeaseDuration.TotalSeconds); 53 | Assert.Equal(21, options.PartitionManagerOptions.RenewInterval.TotalSeconds); 54 | Assert.Equal(_optionStringInitialOffsetType, options.InitialOffsetOptions.Type); 55 | Assert.Equal(_optionStringInitialOffsetQneueuedTimeUTC, options.InitialOffsetOptions.EnqueuedTimeUTC); 56 | } 57 | 58 | [Fact] 59 | public void ConfigureOptions_Format_Returns_Expected() 60 | { 61 | EventHubOptions options = CreateOptions(); 62 | 63 | string format = options.Format(); 64 | JObject iObj = JObject.Parse(format); 65 | EventHubOptions result = iObj.ToObject(); 66 | 67 | Assert.Equal(result.BatchCheckpointFrequency, 5); 68 | Assert.Equal(result.EventProcessorOptions.EnableReceiverRuntimeMetric, true); 69 | Assert.Equal(result.EventProcessorOptions.InvokeProcessorAfterReceiveTimeout, true); 70 | Assert.Equal(result.EventProcessorOptions.MaxBatchSize, 123); 71 | Assert.Equal(result.EventProcessorOptions.PrefetchCount, 123); 72 | Assert.Equal(result.EventProcessorOptions.ReceiveTimeout, TimeSpan.FromSeconds(33)); 73 | Assert.Equal(result.PartitionManagerOptions.LeaseDuration, TimeSpan.FromSeconds(31)); 74 | Assert.Equal(result.PartitionManagerOptions.RenewInterval, TimeSpan.FromSeconds(21)); 75 | Assert.Equal(_optionStringInitialOffsetType, options.InitialOffsetOptions.Type); 76 | Assert.Equal(_optionStringInitialOffsetQneueuedTimeUTC, options.InitialOffsetOptions.EnqueuedTimeUTC); 77 | } 78 | 79 | private EventHubOptions CreateOptions() 80 | { 81 | string extensionPath = "AzureWebJobs:Extensions:EventHubs"; 82 | var values = new Dictionary 83 | { 84 | { $"{extensionPath}:EventProcessorOptions:MaxBatchSize", "123" }, 85 | { $"{extensionPath}:EventProcessorOptions:ReceiveTimeout", "00:00:33" }, 86 | { $"{extensionPath}:EventProcessorOptions:EnableReceiverRuntimeMetric", "true" }, 87 | { $"{extensionPath}:EventProcessorOptions:PrefetchCount", "123" }, 88 | { $"{extensionPath}:EventProcessorOptions:InvokeProcessorAfterReceiveTimeout", "true" }, 89 | { $"{extensionPath}:BatchCheckpointFrequency", "5" }, 90 | { $"{extensionPath}:PartitionManagerOptions:LeaseDuration", "00:00:31" }, 91 | { $"{extensionPath}:PartitionManagerOptions:RenewInterval", "00:00:21" }, 92 | { $"{extensionPath}:InitialOffsetOptions:Type", _optionStringInitialOffsetType }, 93 | { $"{extensionPath}:InitialOffsetOptions:EnqueuedTimeUTC", _optionStringInitialOffsetQneueuedTimeUTC } 94 | }; 95 | 96 | return TestHelpers.GetConfiguredOptions(b => 97 | { 98 | b.AddEventHubs(); 99 | }, values); 100 | } 101 | 102 | [Fact] 103 | public void Initialize_PerformsExpectedRegistrations() 104 | { 105 | var host = new HostBuilder() 106 | .ConfigureDefaultTestHost(builder => 107 | { 108 | builder.AddEventHubs(); 109 | }) 110 | .ConfigureServices(c => 111 | { 112 | c.AddSingleton(new RandomNameResolver()); 113 | }) 114 | .Build(); 115 | 116 | IExtensionRegistry extensions = host.Services.GetService(); 117 | 118 | // ensure the EventHubTriggerAttributeBindingProvider was registered 119 | var triggerBindingProviders = extensions.GetExtensions().ToArray(); 120 | EventHubTriggerAttributeBindingProvider triggerBindingProvider = triggerBindingProviders.OfType().Single(); 121 | Assert.NotNull(triggerBindingProvider); 122 | 123 | // ensure the EventProcessorOptions ExceptionReceived event is wired up 124 | var options = host.Services.GetService>().Value; 125 | var eventProcessorOptions = options.EventProcessorOptions; 126 | var ex = new EventHubsException(false, "Kaboom!"); 127 | var ctor = typeof(ExceptionReceivedEventArgs).GetConstructors(BindingFlags.NonPublic | BindingFlags.Instance).Single(); 128 | var args = (ExceptionReceivedEventArgs)ctor.Invoke(new object[] { "TestHostName", "TestPartitionId", ex, "TestAction" }); 129 | var handler = (Action)eventProcessorOptions.GetType().GetField("exceptionHandler", BindingFlags.Instance | BindingFlags.NonPublic).GetValue(eventProcessorOptions); 130 | handler.Method.Invoke(handler.Target, new object[] { args }); 131 | 132 | string expectedMessage = "EventProcessorHost error (Action='TestAction', HostName='TestHostName', PartitionId='TestPartitionId')."; 133 | var logMessage = host.GetTestLoggerProvider().GetAllLogMessages().Single(); 134 | Assert.Equal(LogLevel.Error, logMessage.Level); 135 | Assert.Equal(expectedMessage, logMessage.FormattedMessage); 136 | Assert.Same(ex, logMessage.Exception); 137 | } 138 | 139 | [Fact] 140 | public void LogExceptionReceivedEvent_NonTransientEvent_LoggedAsError() 141 | { 142 | var ex = new EventHubsException(false); 143 | Assert.False(ex.IsTransient); 144 | var ctor = typeof(ExceptionReceivedEventArgs).GetConstructors(BindingFlags.NonPublic | BindingFlags.Instance).Single(); 145 | var e = (ExceptionReceivedEventArgs)ctor.Invoke(new object[] { "TestHostName", "TestPartitionId", ex, "TestAction" }); 146 | EventHubExtensionConfigProvider.LogExceptionReceivedEvent(e, _loggerFactory); 147 | 148 | string expectedMessage = "EventProcessorHost error (Action='TestAction', HostName='TestHostName', PartitionId='TestPartitionId')."; 149 | var logMessage = _loggerProvider.GetAllLogMessages().Single(); 150 | Assert.Equal(LogLevel.Error, logMessage.Level); 151 | Assert.Same(ex, logMessage.Exception); 152 | Assert.Equal(expectedMessage, logMessage.FormattedMessage); 153 | } 154 | 155 | [Fact] 156 | public void LogExceptionReceivedEvent_TransientEvent_LoggedAsVerbose() 157 | { 158 | var ex = new EventHubsException(true); 159 | Assert.True(ex.IsTransient); 160 | var ctor = typeof(ExceptionReceivedEventArgs).GetConstructors(BindingFlags.NonPublic | BindingFlags.Instance).Single(); 161 | var e = (ExceptionReceivedEventArgs)ctor.Invoke(new object[] { "TestHostName", "TestPartitionId", ex, "TestAction" }); 162 | EventHubExtensionConfigProvider.LogExceptionReceivedEvent(e, _loggerFactory); 163 | 164 | string expectedMessage = "EventProcessorHost error (Action='TestAction', HostName='TestHostName', PartitionId='TestPartitionId')."; 165 | var logMessage = _loggerProvider.GetAllLogMessages().Single(); 166 | Assert.Equal(LogLevel.Information, logMessage.Level); 167 | Assert.Same(ex, logMessage.Exception); 168 | Assert.Equal(expectedMessage + string.Format(_template, typeof(EventHubsException).Name), logMessage.FormattedMessage); 169 | } 170 | 171 | [Fact] 172 | public void LogExceptionReceivedEvent_OperationCanceledException_LoggedAsVerbose() 173 | { 174 | var ex = new OperationCanceledException("Testing"); 175 | var ctor = typeof(ExceptionReceivedEventArgs).GetConstructors(BindingFlags.NonPublic | BindingFlags.Instance).Single(); 176 | var e = (ExceptionReceivedEventArgs)ctor.Invoke(new object[] { "TestHostName", "TestPartitionId", ex, "TestAction" }); 177 | EventHubExtensionConfigProvider.LogExceptionReceivedEvent(e, _loggerFactory); 178 | 179 | string expectedMessage = "EventProcessorHost error (Action='TestAction', HostName='TestHostName', PartitionId='TestPartitionId')."; 180 | var logMessage = _loggerProvider.GetAllLogMessages().Single(); 181 | Assert.Equal(LogLevel.Information, logMessage.Level); 182 | Assert.Same(ex, logMessage.Exception); 183 | Assert.Equal(expectedMessage + string.Format(_template, typeof(OperationCanceledException).Name), logMessage.FormattedMessage); 184 | } 185 | 186 | [Fact] 187 | public void LogExceptionReceivedEvent_NonMessagingException_LoggedAsError() 188 | { 189 | var ex = new MissingMethodException("What method??"); 190 | var ctor = typeof(ExceptionReceivedEventArgs).GetConstructors(BindingFlags.NonPublic | BindingFlags.Instance).Single(); 191 | var e = (ExceptionReceivedEventArgs)ctor.Invoke(new object[] { "TestHostName", "TestPartitionId", ex, "TestAction" }); 192 | EventHubExtensionConfigProvider.LogExceptionReceivedEvent(e, _loggerFactory); 193 | 194 | string expectedMessage = "EventProcessorHost error (Action='TestAction', HostName='TestHostName', PartitionId='TestPartitionId')."; 195 | var logMessage = _loggerProvider.GetAllLogMessages().Single(); 196 | Assert.Equal(LogLevel.Error, logMessage.Level); 197 | Assert.Same(ex, logMessage.Exception); 198 | Assert.Equal(expectedMessage, logMessage.FormattedMessage); 199 | } 200 | 201 | [Fact] 202 | public void LogExceptionReceivedEvent_PartitionExceptions_LoggedAsInfo() 203 | { 204 | var ctor = typeof(ReceiverDisconnectedException).GetConstructor(BindingFlags.NonPublic | BindingFlags.Instance, null, new Type[] { typeof(string) }, null); 205 | var ex = (ReceiverDisconnectedException)ctor.Invoke(new object[] { "New receiver with higher epoch of '30402' is created hence current receiver with epoch '30402' is getting disconnected." }); 206 | ctor = typeof(ExceptionReceivedEventArgs).GetConstructors(BindingFlags.NonPublic | BindingFlags.Instance).Single(); 207 | var e = (ExceptionReceivedEventArgs)ctor.Invoke(new object[] { "TestHostName", "TestPartitionId", ex, "TestAction" }); 208 | EventHubExtensionConfigProvider.LogExceptionReceivedEvent(e, _loggerFactory); 209 | 210 | string expectedMessage = "EventProcessorHost error (Action='TestAction', HostName='TestHostName', PartitionId='TestPartitionId')."; 211 | var logMessage = _loggerProvider.GetAllLogMessages().Single(); 212 | Assert.Equal(LogLevel.Information, logMessage.Level); 213 | Assert.Same(ex, logMessage.Exception); 214 | Assert.Equal(expectedMessage + string.Format(_template, typeof(ReceiverDisconnectedException).Name), logMessage.FormattedMessage); 215 | } 216 | 217 | [Fact] 218 | public void LogExceptionReceivedEvent_AggregateExceptions_LoggedAsInfo() 219 | { 220 | var ctor = typeof(AggregateException).GetConstructor(BindingFlags.Public | BindingFlags.Instance, null, new Type[] { typeof(IEnumerable) }, null); 221 | var request = new RequestResult() 222 | { 223 | HttpStatusCode = 409 224 | }; 225 | var information = new StorageExtendedErrorInformation(); 226 | typeof(StorageExtendedErrorInformation).GetProperty("ErrorCode").SetValue(information, "LeaseIdMismatchWithLeaseOperation"); 227 | typeof(RequestResult).GetProperty("ExtendedErrorInformation").SetValue(request, information); 228 | var storageException = new StorageException(request, "The lease ID specified did not match the lease ID for the blob.", null); 229 | 230 | var ex = (AggregateException)ctor.Invoke(new object[] { new Exception[] { storageException } }); 231 | ctor = typeof(ExceptionReceivedEventArgs).GetConstructors(BindingFlags.NonPublic | BindingFlags.Instance).Single(); 232 | var e = (ExceptionReceivedEventArgs)ctor.Invoke(new object[] { "TestHostName", "TestPartitionId", ex, "TestAction" }); 233 | EventHubExtensionConfigProvider.LogExceptionReceivedEvent(e, _loggerFactory); 234 | 235 | string expectedMessage = "EventProcessorHost error (Action='TestAction', HostName='TestHostName', PartitionId='TestPartitionId')."; 236 | var logMessage = _loggerProvider.GetAllLogMessages().Single(); 237 | Assert.Equal(LogLevel.Information, logMessage.Level); 238 | Assert.Same(storageException, logMessage.Exception); 239 | Assert.Equal(expectedMessage + string.Format(_template, typeof(WindowsAzure.Storage.StorageException).Name), logMessage.FormattedMessage); 240 | } 241 | } 242 | } -------------------------------------------------------------------------------- /test/Microsoft.Azure.WebJobs.Extensions.EventHubs.Tests/EventHubEndToEndTests.cs: -------------------------------------------------------------------------------- 1 | // Copyright (c) .NET Foundation. All rights reserved. 2 | // Licensed under the MIT License. See License.txt in the project root for license information. 3 | 4 | using System; 5 | using System.Collections.Generic; 6 | using System.Linq; 7 | using System.Reflection; 8 | using System.Text; 9 | using System.Threading; 10 | using System.Threading.Tasks; 11 | using Microsoft.WindowsAzure.Storage; 12 | using Microsoft.WindowsAzure.Storage.Blob; 13 | using Microsoft.Azure.EventHubs; 14 | using Microsoft.Azure.WebJobs.Host.TestCommon; 15 | using Microsoft.Extensions.Configuration; 16 | using Microsoft.Extensions.Hosting; 17 | using Microsoft.Extensions.Logging; 18 | using Microsoft.Azure.WebJobs.EventHubs; 19 | using Xunit; 20 | // Disambiguage between Microsoft.WindowsAzure.Storage.LogLevel 21 | using LogLevel = Microsoft.Extensions.Logging.LogLevel; 22 | 23 | namespace Microsoft.Azure.WebJobs.Host.EndToEndTests 24 | { 25 | public class EventHubEndToEndTests 26 | { 27 | private const string TestHubName = "webjobstesthub"; 28 | private const string TestHubConnectionName = "AzureWebJobsTestHubConnection"; 29 | private const string StorageConnectionName = "AzureWebJobsStorage"; 30 | // The container name created by this extension to save snapshots of the Event Hubs stream positions 31 | private const string SnapshotStorageContainerName = "azure-webjobs-eventhub"; 32 | private const int Timeout = 30000; 33 | 34 | private static EventWaitHandle _eventWait; 35 | private static string _testId; 36 | private static List _results; 37 | private static string _testHubConnectionString; 38 | private static string _storageConnectionString; 39 | private static DateTime _initialOffsetEnqueuedTimeUTC; 40 | private static DateTime? _earliestReceivedMessageEnqueuedTimeUTC = null; 41 | 42 | public EventHubEndToEndTests() 43 | { 44 | _results = new List(); 45 | _testId = Guid.NewGuid().ToString(); 46 | _eventWait = new ManualResetEvent(initialState: false); 47 | 48 | var config = new ConfigurationBuilder() 49 | .AddEnvironmentVariables() 50 | .AddTestSettings() 51 | .Build(); 52 | 53 | _testHubConnectionString = config.GetConnectionStringOrSetting(TestHubConnectionName); 54 | _storageConnectionString = config.GetConnectionStringOrSetting(StorageConnectionName); 55 | } 56 | 57 | [Fact] 58 | public async Task EventHub_PocoBinding() 59 | { 60 | var tuple = BuildHost(); 61 | using (var host = tuple.Item1) 62 | { 63 | var method = typeof(EventHubTestBindToPocoJobs).GetMethod(nameof(EventHubTestBindToPocoJobs.SendEvent_TestHub), BindingFlags.Static | BindingFlags.Public); 64 | await host.CallAsync(method, new { input = "{ Name: 'foo', Value: '" + _testId +"' }" }); 65 | 66 | bool result = _eventWait.WaitOne(Timeout); 67 | Assert.True(result); 68 | 69 | var logs = tuple.Item2.GetTestLoggerProvider().GetAllLogMessages().Select(p => p.FormattedMessage); 70 | 71 | Assert.Contains($"PocoValues(foo,{_testId})", logs); 72 | } 73 | } 74 | 75 | [Fact] 76 | public async Task EventHub_StringBinding() 77 | { 78 | var tuple = BuildHost(); 79 | using (var host = tuple.Item1) 80 | { 81 | var method = typeof(EventHubTestBindToStringJobs).GetMethod(nameof(EventHubTestBindToStringJobs.SendEvent_TestHub), BindingFlags.Static | BindingFlags.Public); 82 | await host.CallAsync(method, new { input = _testId }); 83 | 84 | bool result = _eventWait.WaitOne(Timeout); 85 | Assert.True(result); 86 | 87 | var logs = tuple.Item2.GetTestLoggerProvider().GetAllLogMessages().Select(p => p.FormattedMessage); 88 | 89 | Assert.Contains($"Input({_testId})", logs); 90 | } 91 | } 92 | 93 | [Fact] 94 | public async Task EventHub_SingleDispatch() 95 | { 96 | Tuple tuple = BuildHost(); 97 | using (var host = tuple.Item1) 98 | { 99 | var method = typeof(EventHubTestSingleDispatchJobs).GetMethod(nameof(EventHubTestSingleDispatchJobs.SendEvent_TestHub), BindingFlags.Static | BindingFlags.Public); 100 | await host.CallAsync(method, new { input = _testId }); 101 | 102 | bool result = _eventWait.WaitOne(Timeout); 103 | Assert.True(result); 104 | 105 | // Wait for checkpointing 106 | await Task.Delay(3000); 107 | 108 | IEnumerable logMessages = tuple.Item2.GetTestLoggerProvider() 109 | .GetAllLogMessages(); 110 | 111 | Assert.Equal(logMessages.Where(x => !string.IsNullOrEmpty(x.FormattedMessage) 112 | && x.FormattedMessage.Contains("Trigger Details:") 113 | && x.FormattedMessage.Contains("Offset:")).Count(), 1); 114 | 115 | Assert.True(logMessages.Where(x => !string.IsNullOrEmpty(x.FormattedMessage) 116 | && x.FormattedMessage.Contains("OpenAsync")).Count() > 0); 117 | 118 | Assert.True(logMessages.Where(x => !string.IsNullOrEmpty(x.FormattedMessage) 119 | && x.FormattedMessage.Contains("CheckpointAsync")).Count() > 0); 120 | 121 | Assert.True(logMessages.Where(x => !string.IsNullOrEmpty(x.FormattedMessage) 122 | && x.FormattedMessage.Contains("Sending events to EventHub")).Count() > 0); 123 | } 124 | } 125 | 126 | [Fact] 127 | public async Task EventHub_MultipleDispatch() 128 | { 129 | Tuple tuple = BuildHost(); 130 | using (var host = tuple.Item1) 131 | { 132 | // send some events BEFORE starting the host, to ensure 133 | // the events are received in batch 134 | var method = typeof(EventHubTestMultipleDispatchJobs).GetMethod("SendEvents_TestHub", BindingFlags.Static | BindingFlags.Public); 135 | int numEvents = 5; 136 | await host.CallAsync(method, new { numEvents = numEvents, input = _testId }); 137 | 138 | bool result = _eventWait.WaitOne(Timeout); 139 | Assert.True(result); 140 | 141 | // Wait for checkpointing 142 | await Task.Delay(3000); 143 | 144 | IEnumerable logMessages = tuple.Item2.GetTestLoggerProvider() 145 | .GetAllLogMessages(); 146 | 147 | Assert.True(logMessages.Where(x => !string.IsNullOrEmpty(x.FormattedMessage) 148 | && x.FormattedMessage.Contains("Trigger Details:") 149 | && x.FormattedMessage.Contains("Offset:")).Count() > 0); 150 | 151 | Assert.True(logMessages.Where(x => !string.IsNullOrEmpty(x.FormattedMessage) 152 | && x.FormattedMessage.Contains("OpenAsync")).Count() > 0); 153 | 154 | Assert.True(logMessages.Where(x => !string.IsNullOrEmpty(x.FormattedMessage) 155 | && x.FormattedMessage.Contains("CheckpointAsync")).Count() > 0); 156 | 157 | Assert.True(logMessages.Where(x => !string.IsNullOrEmpty(x.FormattedMessage) 158 | && x.FormattedMessage.Contains("Sending events to EventHub")).Count() > 0); 159 | } 160 | } 161 | 162 | [Fact] 163 | public async Task EventHub_PartitionKey() 164 | { 165 | Tuple tuple = BuildHost(); 166 | using (var host = tuple.Item1) 167 | { 168 | var method = typeof(EventHubPartitionKeyTestJobs).GetMethod("SendEvents_TestHub", BindingFlags.Static | BindingFlags.Public); 169 | _eventWait = new ManualResetEvent(initialState: false); 170 | await host.CallAsync(method, new { input = _testId }); 171 | 172 | bool result = _eventWait.WaitOne(Timeout); 173 | 174 | Assert.True(result); 175 | } 176 | } 177 | 178 | [Fact] 179 | public async Task EventHub_InitialOffsetFromEnd() 180 | { 181 | // Send a message to ensure the stream is not empty as we are trying to validate that no messages are delivered in this case 182 | using (var host = BuildHost().Item1) 183 | { 184 | var method = typeof(EventHubTestSendOnlyJobs).GetMethod(nameof(EventHubTestSendOnlyJobs.SendEvent_TestHub), BindingFlags.Static | BindingFlags.Public); 185 | await host.CallAsync(method, new { input = _testId }); 186 | } 187 | // Clear out existing checkpoints as the InitialOffset config only applies when checkpoints don't exist (first run on a host) 188 | await DeleteStorageCheckpoints(); 189 | var initialOffsetOptions = new InitialOffsetOptions() 190 | { 191 | Type = "FromEnd" 192 | }; 193 | using (var host = BuildHost(initialOffsetOptions).Item1) 194 | { 195 | // We don't expect to get signalled as there will be messages recieved with a FromEnd initial offset 196 | bool result = _eventWait.WaitOne(Timeout); 197 | Assert.False(result, "An event was received while none were expected."); 198 | } 199 | } 200 | 201 | [Fact] 202 | public async Task EventHub_InitialOffsetFromEnqueuedTime() 203 | { 204 | // Mark the time now and send a message which should be the only one that is picked up when we run the actual test host 205 | _initialOffsetEnqueuedTimeUTC = DateTime.UtcNow; 206 | using (var host = BuildHost().Item1) 207 | { 208 | var method = typeof(EventHubTestSendOnlyJobs).GetMethod(nameof(EventHubTestSendOnlyJobs.SendEvent_TestHub), BindingFlags.Static | BindingFlags.Public); 209 | await host.CallAsync(method, new { input = _testId }); 210 | } 211 | // Clear out existing checkpoints as the InitialOffset config only applies when checkpoints don't exist (first run on a host) 212 | await DeleteStorageCheckpoints(); 213 | _earliestReceivedMessageEnqueuedTimeUTC = null; 214 | var initialOffsetOptions = new InitialOffsetOptions() 215 | { 216 | Type = "FromEnqueuedTime", 217 | EnqueuedTimeUTC = _initialOffsetEnqueuedTimeUTC.ToString("yyyy-MM-ddTHH:mm:ssZ") 218 | }; 219 | using (var host = BuildHost(initialOffsetOptions).Item1) 220 | { 221 | // Validation that we only got messages after the configured FromEnqueuedTime is done in the JobHost 222 | bool result = _eventWait.WaitOne(Timeout); 223 | Assert.True(result, $"No event was received within the timeout period of {Timeout}. " + 224 | $"Expected event sent shortly after {_initialOffsetEnqueuedTimeUTC.ToString("yyyy-MM-ddTHH:mm:ssZ")} with content {_testId}"); 225 | Assert.True(_earliestReceivedMessageEnqueuedTimeUTC > _initialOffsetEnqueuedTimeUTC, 226 | "A message was received that was enqueued before the configured Initial Offset Enqueued Time. " + 227 | $"Received message enqueued time: {_earliestReceivedMessageEnqueuedTimeUTC?.ToString("yyyy-MM-ddTHH:mm:ssZ")}" + 228 | $", initial offset enqueued time: {_initialOffsetEnqueuedTimeUTC.ToString("yyyy-MM-ddTHH:mm:ssZ")}"); 229 | } 230 | } 231 | 232 | public class EventHubTestSingleDispatchJobs 233 | { 234 | public static void SendEvent_TestHub(string input, [EventHub(TestHubName)] out EventData evt) 235 | { 236 | evt = new EventData(Encoding.UTF8.GetBytes(input)); 237 | evt.Properties.Add("TestProp1", "value1"); 238 | evt.Properties.Add("TestProp2", "value2"); 239 | } 240 | 241 | public static void ProcessSingleEvent([EventHubTrigger(TestHubName)] string evt, 242 | string partitionKey, DateTime enqueuedTimeUtc, IDictionary properties, 243 | IDictionary systemProperties) 244 | { 245 | // filter for the ID the current test is using 246 | if (evt == _testId) 247 | { 248 | Assert.True((DateTime.Now - enqueuedTimeUtc).TotalSeconds < 30); 249 | 250 | Assert.Equal("value1", properties["TestProp1"]); 251 | Assert.Equal("value2", properties["TestProp2"]); 252 | 253 | _eventWait.Set(); 254 | } 255 | } 256 | } 257 | 258 | public class EventHubTestSendOnlyJobs 259 | { 260 | public static void SendEvent_TestHub(string input, [EventHub(TestHubName)] out EventData evt) 261 | { 262 | evt = new EventData(Encoding.UTF8.GetBytes(input)); 263 | } 264 | } 265 | 266 | public class EventHubTestInitialOffsetFromEndJobs 267 | { 268 | public static void ProcessSingleEvent([EventHubTrigger(TestHubName)] string evt, 269 | string partitionKey, DateTime enqueuedTimeUtc, IDictionary properties, 270 | IDictionary systemProperties) 271 | { 272 | _eventWait.Set(); 273 | } 274 | } 275 | 276 | public class EventHubTestInitialOffsetFromEnqueuedTimeJobs 277 | { 278 | public static void ProcessSingleEvent([EventHubTrigger(TestHubName)] string evt, 279 | string partitionKey, DateTime enqueuedTimeUtc, IDictionary properties, 280 | IDictionary systemProperties) 281 | { 282 | if (_earliestReceivedMessageEnqueuedTimeUTC == null) 283 | { 284 | _earliestReceivedMessageEnqueuedTimeUTC = enqueuedTimeUtc; 285 | _eventWait.Set(); 286 | } 287 | } 288 | } 289 | 290 | public class EventHubTestBindToPocoJobs 291 | { 292 | public static void SendEvent_TestHub(string input, [EventHub(TestHubName)] out EventData evt) 293 | { 294 | evt = new EventData(Encoding.UTF8.GetBytes(input)); 295 | } 296 | 297 | public static void BindToPoco([EventHubTrigger(TestHubName)] TestPoco input, string value, string name, ILogger logger) 298 | { 299 | if (value == _testId) 300 | { 301 | Assert.Equal(input.Value, value); 302 | Assert.Equal(input.Name, name); 303 | logger.LogInformation($"PocoValues({name},{value})"); 304 | _eventWait.Set(); 305 | } 306 | } 307 | } 308 | 309 | public class EventHubTestBindToStringJobs 310 | { 311 | public static void SendEvent_TestHub(string input, [EventHub(TestHubName)] out EventData evt) 312 | { 313 | evt = new EventData(Encoding.UTF8.GetBytes(input)); 314 | } 315 | 316 | public static void BindToString([EventHubTrigger(TestHubName)] string input, ILogger logger) 317 | { 318 | if (input == _testId) 319 | { 320 | logger.LogInformation($"Input({input})"); 321 | _eventWait.Set(); 322 | } 323 | } 324 | } 325 | 326 | public class EventHubTestMultipleDispatchJobs 327 | { 328 | public static void SendEvents_TestHub(int numEvents, string input, [EventHub(TestHubName)] out EventData[] events) 329 | { 330 | events = new EventData[numEvents]; 331 | for (int i = 0; i < numEvents; i++) 332 | { 333 | var evt = new EventData(Encoding.UTF8.GetBytes(input)); 334 | evt.Properties.Add("TestIndex", i); 335 | evt.Properties.Add("TestProp1", "value1"); 336 | evt.Properties.Add("TestProp2", "value2"); 337 | events[i] = evt; 338 | } 339 | } 340 | 341 | public static void ProcessMultipleEvents([EventHubTrigger(TestHubName)] string[] events, 342 | string[] partitionKeyArray, DateTime[] enqueuedTimeUtcArray, IDictionary[] propertiesArray, 343 | IDictionary[] systemPropertiesArray) 344 | { 345 | Assert.Equal(events.Length, partitionKeyArray.Length); 346 | Assert.Equal(events.Length, enqueuedTimeUtcArray.Length); 347 | Assert.Equal(events.Length, propertiesArray.Length); 348 | Assert.Equal(events.Length, systemPropertiesArray.Length); 349 | 350 | for (int i = 0; i < events.Length; i++) 351 | { 352 | Assert.Equal(i, propertiesArray[i]["TestIndex"]); 353 | } 354 | 355 | // filter for the ID the current test is using 356 | if (events[0] == _testId) 357 | { 358 | _results.AddRange(events); 359 | _eventWait.Set(); 360 | } 361 | } 362 | } 363 | 364 | public class EventHubPartitionKeyTestJobs 365 | { 366 | public static async Task SendEvents_TestHub( 367 | string input, 368 | [EventHub(TestHubName)] EventHubClient client) 369 | { 370 | List list = new List(); 371 | EventData evt = new EventData(Encoding.UTF8.GetBytes(input)); 372 | 373 | // Send event without PK 374 | await client.SendAsync(evt); 375 | 376 | // Send event with different PKs 377 | for (int i = 0; i < 5; i++) 378 | { 379 | evt = new EventData(Encoding.UTF8.GetBytes(input)); 380 | await client.SendAsync(evt, "test_pk" + i); 381 | } 382 | } 383 | 384 | public static void ProcessMultiplePartitionEvents([EventHubTrigger(TestHubName)] EventData[] events) 385 | { 386 | foreach (EventData eventData in events) 387 | { 388 | string message = Encoding.UTF8.GetString(eventData.Body); 389 | 390 | // filter for the ID the current test is using 391 | if (message == _testId) 392 | { 393 | _results.Add(eventData.SystemProperties.PartitionKey); 394 | _results.Sort(); 395 | 396 | if (_results.Count == 6 && _results[5] == "test_pk4") 397 | { 398 | _eventWait.Set(); 399 | } 400 | } 401 | } 402 | } 403 | } 404 | 405 | private Tuple BuildHost(InitialOffsetOptions initialOffsetOptions = null) 406 | { 407 | JobHost jobHost = null; 408 | 409 | Assert.True(!string.IsNullOrEmpty(_testHubConnectionString), $"Required test connection string '{TestHubConnectionName}' is missing."); 410 | 411 | IHost host = new HostBuilder() 412 | .ConfigureDefaultTestHost(b => 413 | { 414 | b.AddEventHubs(options => 415 | { 416 | options.EventProcessorOptions.EnableReceiverRuntimeMetric = true; 417 | if (initialOffsetOptions != null) 418 | { 419 | options.InitialOffsetOptions = initialOffsetOptions; 420 | } 421 | // We want to validate the default options configuration logic for setting initial offset and not implemente it here 422 | EventHubWebJobsBuilderExtensions.ConfigureOptions(options); 423 | options.AddSender(TestHubName, _testHubConnectionString); 424 | options.AddReceiver(TestHubName, _testHubConnectionString); 425 | }); 426 | }) 427 | .ConfigureLogging(b => 428 | { 429 | b.SetMinimumLevel(LogLevel.Debug); 430 | }) 431 | .Build(); 432 | 433 | jobHost = host.GetJobHost(); 434 | jobHost.StartAsync().GetAwaiter().GetResult(); 435 | 436 | return new Tuple(jobHost, host); 437 | } 438 | 439 | // Deletes all checkpoints that were saved in storage so InitialOffset can be validated 440 | private async Task DeleteStorageCheckpoints() 441 | { 442 | CloudStorageAccount cloudStorageAccount = CloudStorageAccount.Parse(_storageConnectionString); 443 | CloudBlobClient blobClient = cloudStorageAccount.CreateCloudBlobClient(); 444 | var container = blobClient.GetContainerReference(SnapshotStorageContainerName); 445 | BlobContinuationToken continuationToken = null; 446 | do 447 | { 448 | var response = await container.ListBlobsSegmentedAsync(string.Empty, true, BlobListingDetails.None, null, continuationToken, null, null); 449 | continuationToken = response.ContinuationToken; 450 | foreach (var blob in response.Results.OfType()) 451 | { 452 | try 453 | { 454 | await blob.BreakLeaseAsync(TimeSpan.Zero); 455 | } 456 | catch (StorageException) 457 | { 458 | // Ignore as this will be thrown if the blob has no lease on it 459 | } 460 | await blob.DeleteAsync(); 461 | } 462 | } while (continuationToken != null); 463 | } 464 | 465 | public class TestPoco 466 | { 467 | public string Name { get; set; } 468 | public string Value { get; set; } 469 | } 470 | } 471 | } -------------------------------------------------------------------------------- /test/Microsoft.Azure.WebJobs.Extensions.EventHubs.Tests/EventHubListenerTests.cs: -------------------------------------------------------------------------------- 1 | // Copyright (c) .NET Foundation. All rights reserved. 2 | // Licensed under the MIT License. See License.txt in the project root for license information. 3 | 4 | using System; 5 | using System.Collections.Generic; 6 | using System.Linq; 7 | using System.Reflection; 8 | using System.Threading; 9 | using System.Threading.Tasks; 10 | using Microsoft.Azure.EventHubs; 11 | using Microsoft.Azure.EventHubs.Processor; 12 | using Microsoft.Azure.WebJobs.EventHubs.Listeners; 13 | using Microsoft.Azure.WebJobs.Host.Executors; 14 | using Microsoft.Azure.WebJobs.Host.Listeners; 15 | using Microsoft.Azure.WebJobs.Host.Scale; 16 | using Microsoft.Azure.WebJobs.Host.TestCommon; 17 | using Microsoft.Extensions.Logging; 18 | using Microsoft.WindowsAzure.Storage.Blob; 19 | using Moq; 20 | using Xunit; 21 | 22 | namespace Microsoft.Azure.WebJobs.EventHubs.UnitTests 23 | { 24 | public class EventHubListenerTests 25 | { 26 | [Theory] 27 | [InlineData(1, 100)] 28 | [InlineData(4, 25)] 29 | [InlineData(8, 12)] 30 | [InlineData(32, 3)] 31 | [InlineData(128, 0)] 32 | public async Task ProcessEvents_SingleDispatch_CheckpointsCorrectly(int batchCheckpointFrequency, int expected) 33 | { 34 | var partitionContext = EventHubTests.GetPartitionContext(); 35 | var checkpoints = 0; 36 | var options = new EventHubOptions 37 | { 38 | BatchCheckpointFrequency = batchCheckpointFrequency 39 | }; 40 | var checkpointer = new Mock(MockBehavior.Strict); 41 | checkpointer.Setup(p => p.CheckpointAsync(partitionContext)).Callback(c => 42 | { 43 | checkpoints++; 44 | }).Returns(Task.CompletedTask); 45 | var loggerMock = new Mock(); 46 | var executor = new Mock(MockBehavior.Strict); 47 | executor.Setup(p => p.TryExecuteAsync(It.IsAny(), It.IsAny())).ReturnsAsync(new FunctionResult(true)); 48 | var eventProcessor = new EventHubListener.EventProcessor(options, executor.Object, loggerMock.Object, true, checkpointer.Object); 49 | 50 | for (int i = 0; i < 100; i++) 51 | { 52 | List events = new List() { new EventData(new byte[0]) }; 53 | await eventProcessor.ProcessEventsAsync(partitionContext, events); 54 | } 55 | 56 | Assert.Equal(expected, checkpoints); 57 | } 58 | 59 | [Theory] 60 | [InlineData(1, 100)] 61 | [InlineData(4, 25)] 62 | [InlineData(8, 12)] 63 | [InlineData(32, 3)] 64 | [InlineData(128, 0)] 65 | public async Task ProcessEvents_MultipleDispatch_CheckpointsCorrectly(int batchCheckpointFrequency, int expected) 66 | { 67 | var partitionContext = EventHubTests.GetPartitionContext(); 68 | var options = new EventHubOptions 69 | { 70 | BatchCheckpointFrequency = batchCheckpointFrequency 71 | }; 72 | var checkpointer = new Mock(MockBehavior.Strict); 73 | checkpointer.Setup(p => p.CheckpointAsync(partitionContext)).Returns(Task.CompletedTask); 74 | var loggerMock = new Mock(); 75 | var executor = new Mock(MockBehavior.Strict); 76 | executor.Setup(p => p.TryExecuteAsync(It.IsAny(), It.IsAny())).ReturnsAsync(new FunctionResult(true)); 77 | var eventProcessor = new EventHubListener.EventProcessor(options, executor.Object, loggerMock.Object, false, checkpointer.Object); 78 | 79 | for (int i = 0; i < 100; i++) 80 | { 81 | List events = new List() { new EventData(new byte[0]), new EventData(new byte[0]), new EventData(new byte[0]) }; 82 | await eventProcessor.ProcessEventsAsync(partitionContext, events); 83 | } 84 | 85 | checkpointer.Verify(p => p.CheckpointAsync(partitionContext), Times.Exactly(expected)); 86 | } 87 | 88 | /// 89 | /// Even if some events in a batch fail, we still checkpoint. Event error handling 90 | /// is the responsiblity of user function code. 91 | /// 92 | /// 93 | [Fact] 94 | public async Task ProcessEvents_Failure_Checkpoints() 95 | { 96 | var partitionContext = EventHubTests.GetPartitionContext(); 97 | var options = new EventHubOptions(); 98 | var checkpointer = new Mock(MockBehavior.Strict); 99 | checkpointer.Setup(p => p.CheckpointAsync(partitionContext)).Returns(Task.CompletedTask); 100 | 101 | List events = new List(); 102 | List results = new List(); 103 | for (int i = 0; i < 10; i++) 104 | { 105 | events.Add(new EventData(new byte[0])); 106 | var succeeded = i > 7 ? false : true; 107 | results.Add(new FunctionResult(succeeded)); 108 | } 109 | 110 | var executor = new Mock(MockBehavior.Strict); 111 | int execution = 0; 112 | executor.Setup(p => p.TryExecuteAsync(It.IsAny(), It.IsAny())).ReturnsAsync(() => 113 | { 114 | var result = results[execution++]; 115 | return result; 116 | }); 117 | 118 | var loggerMock = new Mock(); 119 | 120 | var eventProcessor = new EventHubListener.EventProcessor(options, executor.Object, loggerMock.Object, true, checkpointer.Object); 121 | 122 | await eventProcessor.ProcessEventsAsync(partitionContext, events); 123 | 124 | checkpointer.Verify(p => p.CheckpointAsync(partitionContext), Times.Once); 125 | } 126 | 127 | [Fact] 128 | public async Task CloseAsync_Shutdown_DoesNotCheckpoint() 129 | { 130 | var partitionContext = EventHubTests.GetPartitionContext(); 131 | var options = new EventHubOptions(); 132 | var checkpointer = new Mock(MockBehavior.Strict); 133 | var executor = new Mock(MockBehavior.Strict); 134 | var loggerMock = new Mock(); 135 | var eventProcessor = new EventHubListener.EventProcessor(options, executor.Object, loggerMock.Object, true, checkpointer.Object); 136 | 137 | await eventProcessor.CloseAsync(partitionContext, CloseReason.Shutdown); 138 | 139 | checkpointer.Verify(p => p.CheckpointAsync(partitionContext), Times.Never); 140 | } 141 | 142 | [Fact] 143 | public async Task ProcessErrorsAsync_LoggedAsError() 144 | { 145 | var partitionContext = EventHubTests.GetPartitionContext(partitionId: "123", eventHubPath: "abc", owner: "def"); 146 | var options = new EventHubOptions(); 147 | var checkpointer = new Mock(MockBehavior.Strict); 148 | var executor = new Mock(MockBehavior.Strict); 149 | var testLogger = new TestLogger("Test"); 150 | var eventProcessor = new EventHubListener.EventProcessor(options, executor.Object, testLogger, true, checkpointer.Object); 151 | 152 | var ex = new InvalidOperationException("My InvalidOperationException!"); 153 | 154 | await eventProcessor.ProcessErrorAsync(partitionContext, ex); 155 | var msg = testLogger.GetLogMessages().Single(); 156 | Assert.Equal("Processing error (Partition Id: '123', Owner: 'def', EventHubPath: 'abc').", msg.FormattedMessage); 157 | Assert.IsType(msg.Exception); 158 | Assert.Equal(LogLevel.Error, msg.Level); 159 | } 160 | 161 | [Fact] 162 | public async Task ProcessErrorsAsync_RebalancingExceptions_LoggedAsInformation() 163 | { 164 | var partitionContext = EventHubTests.GetPartitionContext(partitionId: "123", eventHubPath: "abc", owner: "def"); 165 | var options = new EventHubOptions(); 166 | var checkpointer = new Mock(MockBehavior.Strict); 167 | var executor = new Mock(MockBehavior.Strict); 168 | var testLogger = new TestLogger("Test"); 169 | var eventProcessor = new EventHubListener.EventProcessor(options, executor.Object, testLogger, true, checkpointer.Object); 170 | 171 | // ctor is private 172 | var constructor = typeof(ReceiverDisconnectedException) 173 | .GetConstructor(BindingFlags.NonPublic | BindingFlags.Instance, null, new Type[] { typeof(string) }, null); 174 | ReceiverDisconnectedException disconnectedEx = (ReceiverDisconnectedException)constructor.Invoke(new[] { "My ReceiverDisconnectedException!" }); 175 | 176 | await eventProcessor.ProcessErrorAsync(partitionContext, disconnectedEx); 177 | var msg = testLogger.GetLogMessages().Single(); 178 | Assert.Equal("Processing error (Partition Id: '123', Owner: 'def', EventHubPath: 'abc'). An exception of type 'ReceiverDisconnectedException' was thrown. This exception type is typically a result of Event Hub processor rebalancing or a transient error and can be safely ignored.", msg.FormattedMessage); 179 | Assert.NotNull(msg.Exception); 180 | Assert.Equal(LogLevel.Information, msg.Level); 181 | 182 | testLogger.ClearLogMessages(); 183 | 184 | // ctor is private 185 | constructor = typeof(LeaseLostException) 186 | .GetConstructor(BindingFlags.NonPublic | BindingFlags.Instance, null, new Type[] { typeof(string), typeof(Exception) }, null); 187 | LeaseLostException leaseLostEx = (LeaseLostException)constructor.Invoke(new object[] { "My LeaseLostException!", new Exception() }); 188 | 189 | await eventProcessor.ProcessErrorAsync(partitionContext, leaseLostEx); 190 | msg = testLogger.GetLogMessages().Single(); 191 | Assert.Equal("Processing error (Partition Id: '123', Owner: 'def', EventHubPath: 'abc'). An exception of type 'LeaseLostException' was thrown. This exception type is typically a result of Event Hub processor rebalancing or a transient error and can be safely ignored.", msg.FormattedMessage); 192 | Assert.NotNull(msg.Exception); 193 | Assert.Equal(LogLevel.Information, msg.Level); 194 | } 195 | 196 | [Fact] 197 | public void GetMonitor_ReturnsExpectedValue() 198 | { 199 | var functionId = "FunctionId"; 200 | var eventHubName = "EventHubName"; 201 | var consumerGroup = "ConsumerGroup"; 202 | var testLogger = new TestLogger("Test"); 203 | var listener = new EventHubListener( 204 | functionId, 205 | eventHubName, 206 | consumerGroup, 207 | "Endpoint=sb://test.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=abc123=", 208 | "DefaultEndpointsProtocol=https;AccountName=EventHubScaleMonitorFakeTestAccount;AccountKey=ABCDEFG;EndpointSuffix=core.windows.net", 209 | new Mock(MockBehavior.Strict).Object, 210 | null, 211 | false, 212 | new EventHubOptions(), 213 | testLogger, 214 | new Mock(MockBehavior.Strict, new Uri("https://eventhubsteststorageaccount.blob.core.windows.net/azure-webjobs-eventhub")).Object); 215 | 216 | IScaleMonitor scaleMonitor = listener.GetMonitor(); 217 | 218 | Assert.Equal(typeof(EventHubsScaleMonitor), scaleMonitor.GetType()); 219 | Assert.Equal($"{functionId}-EventHubTrigger-{eventHubName}-{consumerGroup}".ToLower(), scaleMonitor.Descriptor.Id); 220 | 221 | var scaleMonitor2 = listener.GetMonitor(); 222 | 223 | Assert.Same(scaleMonitor, scaleMonitor2); 224 | } 225 | 226 | [Fact] 227 | public void Dispose_Calls_StopAsync() 228 | { 229 | string functionId = "FunctionId"; 230 | string eventHubName = "EventHubName"; 231 | string consumerGroup = "ConsumerGroup"; 232 | var storageUri = new Uri("https://eventhubsteststorageaccount.blob.core.windows.net/"); 233 | var testLogger = new TestLogger("Test"); 234 | EventHubListener listener = new EventHubListener( 235 | functionId, 236 | eventHubName, 237 | consumerGroup, 238 | "Endpoint=sb://test.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=abc123=", 239 | "DefaultEndpointsProtocol=https;AccountName=EventHubScaleMonitorFakeTestAccount;AccountKey=ABCDEFG;EndpointSuffix=core.windows.net", 240 | new Mock(MockBehavior.Strict).Object, 241 | null, 242 | false, 243 | new EventHubOptions(), 244 | testLogger, 245 | new Mock(MockBehavior.Strict, new Uri("https://eventhubsteststorageaccount.blob.core.windows.net/azure-webjobs-eventhub")).Object); 246 | 247 | (listener as IDisposable).Dispose(); 248 | Assert.Single(testLogger.GetLogMessages().Where(x => x.FormattedMessage.StartsWith("EventHub listener is already stopped ("))); 249 | } 250 | } 251 | } 252 | -------------------------------------------------------------------------------- /test/Microsoft.Azure.WebJobs.Extensions.EventHubs.Tests/EventHubTests.cs: -------------------------------------------------------------------------------- 1 | // Copyright (c) .NET Foundation. All rights reserved. 2 | // Licensed under the MIT License. See License.txt in the project root for license information. 3 | 4 | using System; 5 | using System.Collections.Generic; 6 | using System.Reflection; 7 | using System.Text; 8 | using System.Threading; 9 | using Microsoft.Azure.EventHubs; 10 | using Microsoft.Azure.EventHubs.Processor; 11 | using Microsoft.Azure.WebJobs.Host; 12 | using Microsoft.Azure.WebJobs.Host.TestCommon; 13 | using Microsoft.Extensions.Configuration; 14 | using Microsoft.Extensions.DependencyInjection; 15 | using Microsoft.Extensions.Hosting; 16 | using Microsoft.Extensions.Options; 17 | using Xunit; 18 | using static Microsoft.Azure.EventHubs.EventData; 19 | 20 | namespace Microsoft.Azure.WebJobs.EventHubs.UnitTests 21 | { 22 | public class EventHubTests 23 | { 24 | [Fact] 25 | public void GetStaticBindingContract_ReturnsExpectedValue() 26 | { 27 | var strategy = new EventHubTriggerBindingStrategy(); 28 | var contract = strategy.GetBindingContract(); 29 | 30 | Assert.Equal(7, contract.Count); 31 | Assert.Equal(typeof(PartitionContext), contract["PartitionContext"]); 32 | Assert.Equal(typeof(string), contract["Offset"]); 33 | Assert.Equal(typeof(long), contract["SequenceNumber"]); 34 | Assert.Equal(typeof(DateTime), contract["EnqueuedTimeUtc"]); 35 | Assert.Equal(typeof(IDictionary), contract["Properties"]); 36 | Assert.Equal(typeof(IDictionary), contract["SystemProperties"]); 37 | } 38 | 39 | [Fact] 40 | public void GetBindingContract_SingleDispatch_ReturnsExpectedValue() 41 | { 42 | var strategy = new EventHubTriggerBindingStrategy(); 43 | var contract = strategy.GetBindingContract(true); 44 | 45 | Assert.Equal(7, contract.Count); 46 | Assert.Equal(typeof(PartitionContext), contract["PartitionContext"]); 47 | Assert.Equal(typeof(string), contract["Offset"]); 48 | Assert.Equal(typeof(long), contract["SequenceNumber"]); 49 | Assert.Equal(typeof(DateTime), contract["EnqueuedTimeUtc"]); 50 | Assert.Equal(typeof(IDictionary), contract["Properties"]); 51 | Assert.Equal(typeof(IDictionary), contract["SystemProperties"]); 52 | } 53 | 54 | [Fact] 55 | public void GetBindingContract_MultipleDispatch_ReturnsExpectedValue() 56 | { 57 | var strategy = new EventHubTriggerBindingStrategy(); 58 | var contract = strategy.GetBindingContract(false); 59 | 60 | Assert.Equal(7, contract.Count); 61 | Assert.Equal(typeof(PartitionContext), contract["PartitionContext"]); 62 | Assert.Equal(typeof(string[]), contract["PartitionKeyArray"]); 63 | Assert.Equal(typeof(string[]), contract["OffsetArray"]); 64 | Assert.Equal(typeof(long[]), contract["SequenceNumberArray"]); 65 | Assert.Equal(typeof(DateTime[]), contract["EnqueuedTimeUtcArray"]); 66 | Assert.Equal(typeof(IDictionary[]), contract["PropertiesArray"]); 67 | Assert.Equal(typeof(IDictionary[]), contract["SystemPropertiesArray"]); 68 | } 69 | 70 | [Fact] 71 | public void GetBindingData_SingleDispatch_ReturnsExpectedValue() 72 | { 73 | var evt = new EventData(new byte[] { }); 74 | IDictionary sysProps = GetSystemProperties(); 75 | 76 | TestHelpers.SetField(evt, "SystemProperties", sysProps); 77 | 78 | var input = EventHubTriggerInput.New(evt); 79 | input.PartitionContext = GetPartitionContext(); 80 | 81 | var strategy = new EventHubTriggerBindingStrategy(); 82 | var bindingData = strategy.GetBindingData(input); 83 | 84 | Assert.Equal(7, bindingData.Count); 85 | Assert.Same(input.PartitionContext, bindingData["PartitionContext"]); 86 | Assert.Equal(evt.SystemProperties.PartitionKey, bindingData["PartitionKey"]); 87 | Assert.Equal(evt.SystemProperties.Offset, bindingData["Offset"]); 88 | Assert.Equal(evt.SystemProperties.SequenceNumber, bindingData["SequenceNumber"]); 89 | Assert.Equal(evt.SystemProperties.EnqueuedTimeUtc, bindingData["EnqueuedTimeUtc"]); 90 | Assert.Same(evt.Properties, bindingData["Properties"]); 91 | IDictionary bindingDataSysProps = bindingData["SystemProperties"] as Dictionary; 92 | Assert.NotNull(bindingDataSysProps); 93 | Assert.Equal(bindingDataSysProps["PartitionKey"], bindingData["PartitionKey"]); 94 | Assert.Equal(bindingDataSysProps["Offset"], bindingData["Offset"]); 95 | Assert.Equal(bindingDataSysProps["SequenceNumber"], bindingData["SequenceNumber"]); 96 | Assert.Equal(bindingDataSysProps["EnqueuedTimeUtc"], bindingData["EnqueuedTimeUtc"]); 97 | Assert.Equal(bindingDataSysProps["iothub-connection-device-id"], "testDeviceId"); 98 | Assert.Equal(bindingDataSysProps["iothub-enqueuedtime"], DateTime.MinValue); 99 | } 100 | 101 | private static IDictionary GetSystemProperties(string partitionKey = "TestKey") 102 | { 103 | long testSequence = 4294967296; 104 | IDictionary sysProps = TestHelpers.New(); 105 | sysProps["x-opt-partition-key"] = partitionKey; 106 | sysProps["x-opt-offset"] = "TestOffset"; 107 | sysProps["x-opt-enqueued-time"] = DateTime.MinValue; 108 | sysProps["x-opt-sequence-number"] = testSequence; 109 | sysProps["iothub-connection-device-id"] = "testDeviceId"; 110 | sysProps["iothub-enqueuedtime"] = DateTime.MinValue; 111 | return sysProps; 112 | } 113 | 114 | [Fact] 115 | public void GetBindingData_MultipleDispatch_ReturnsExpectedValue() 116 | { 117 | 118 | var events = new EventData[3] 119 | { 120 | new EventData(Encoding.UTF8.GetBytes("Event 1")), 121 | new EventData(Encoding.UTF8.GetBytes("Event 2")), 122 | new EventData(Encoding.UTF8.GetBytes("Event 3")), 123 | }; 124 | 125 | var count = 0; 126 | foreach (var evt in events) 127 | { 128 | IDictionary sysProps = GetSystemProperties($"pk{count++}"); 129 | TestHelpers.SetField(evt, "SystemProperties", sysProps); 130 | } 131 | 132 | var input = new EventHubTriggerInput 133 | { 134 | Events = events, 135 | PartitionContext = GetPartitionContext(), 136 | }; 137 | var strategy = new EventHubTriggerBindingStrategy(); 138 | var bindingData = strategy.GetBindingData(input); 139 | 140 | Assert.Equal(7, bindingData.Count); 141 | Assert.Same(input.PartitionContext, bindingData["PartitionContext"]); 142 | 143 | // verify an array was created for each binding data type 144 | Assert.Equal(events.Length, ((string[])bindingData["PartitionKeyArray"]).Length); 145 | Assert.Equal(events.Length, ((string[])bindingData["OffsetArray"]).Length); 146 | Assert.Equal(events.Length, ((long[])bindingData["SequenceNumberArray"]).Length); 147 | Assert.Equal(events.Length, ((DateTime[])bindingData["EnqueuedTimeUtcArray"]).Length); 148 | Assert.Equal(events.Length, ((IDictionary[])bindingData["PropertiesArray"]).Length); 149 | Assert.Equal(events.Length, ((IDictionary[])bindingData["SystemPropertiesArray"]).Length); 150 | 151 | Assert.Equal(events[0].SystemProperties.PartitionKey, ((string[])bindingData["PartitionKeyArray"])[0]); 152 | Assert.Equal(events[1].SystemProperties.PartitionKey, ((string[])bindingData["PartitionKeyArray"])[1]); 153 | Assert.Equal(events[2].SystemProperties.PartitionKey, ((string[])bindingData["PartitionKeyArray"])[2]); 154 | } 155 | 156 | [Fact] 157 | public void TriggerStrategy() 158 | { 159 | string data = "123"; 160 | 161 | var strategy = new EventHubTriggerBindingStrategy(); 162 | EventHubTriggerInput triggerInput = strategy.ConvertFromString(data); 163 | 164 | var contract = strategy.GetBindingData(triggerInput); 165 | 166 | EventData single = strategy.BindSingle(triggerInput, null); 167 | string body = Encoding.UTF8.GetString(single.Body.Array); 168 | 169 | Assert.Equal(data, body); 170 | Assert.Null(contract["PartitionContext"]); 171 | Assert.Null(contract["partitioncontext"]); // case insensitive 172 | } 173 | 174 | // Validate that if connection string has EntityPath, that takes precedence over the parameter. 175 | [Theory] 176 | [InlineData("k1", "Endpoint=sb://test89123-ns-x.servicebus.windows.net/;SharedAccessKeyName=ReceiveRule;SharedAccessKey=secretkey")] 177 | [InlineData("path2", "Endpoint=sb://test89123-ns-x.servicebus.windows.net/;SharedAccessKeyName=ReceiveRule;SharedAccessKey=secretkey;EntityPath=path2")] 178 | public void EntityPathInConnectionString(string expectedPathName, string connectionString) 179 | { 180 | EventHubOptions options = new EventHubOptions(); 181 | 182 | // Test sender 183 | options.AddSender("k1", connectionString); 184 | var client = options.GetEventHubClient("k1", null); 185 | Assert.Equal(expectedPathName, client.EventHubName); 186 | } 187 | 188 | // Validate that if connection string has EntityPath, that takes precedence over the parameter. 189 | [Theory] 190 | [InlineData("k1", "Endpoint=sb://test89123-ns-x.servicebus.windows.net/;SharedAccessKeyName=ReceiveRule;SharedAccessKey=secretkey")] 191 | [InlineData("path2", "Endpoint=sb://test89123-ns-x.servicebus.windows.net/;SharedAccessKeyName=ReceiveRule;SharedAccessKey=secretkey;EntityPath=path2")] 192 | public void GetEventHubClient_AddsConnection(string expectedPathName, string connectionString) 193 | { 194 | EventHubOptions options = new EventHubOptions(); 195 | var client = options.GetEventHubClient("k1", connectionString); 196 | Assert.Equal(expectedPathName, client.EventHubName); 197 | } 198 | 199 | [Theory] 200 | [InlineData("e", "n1", "n1/e/")] 201 | [InlineData("e--1", "host_.path.foo", "host_.path.foo/e--1/")] 202 | [InlineData("Ab", "Cd", "cd/ab/")] 203 | [InlineData("A=", "Cd", "cd/a:3D/")] 204 | [InlineData("A:", "Cd", "cd/a:3A/")] 205 | public void EventHubBlobPrefix(string eventHubName, string serviceBusNamespace, string expected) 206 | { 207 | string actual = EventHubOptions.GetBlobPrefix(eventHubName, serviceBusNamespace); 208 | Assert.Equal(expected, actual); 209 | } 210 | 211 | [Theory] 212 | [InlineData(1)] 213 | [InlineData(5)] 214 | [InlineData(200)] 215 | public void EventHubBatchCheckpointFrequency(int num) 216 | { 217 | var options = new EventHubOptions(); 218 | options.BatchCheckpointFrequency = num; 219 | Assert.Equal(num, options.BatchCheckpointFrequency); 220 | } 221 | 222 | [Theory] 223 | [InlineData(-1)] 224 | [InlineData(0)] 225 | public void EventHubBatchCheckpointFrequency_Throws(int num) 226 | { 227 | var options = new EventHubOptions(); 228 | Assert.Throws(() => options.BatchCheckpointFrequency = num); 229 | } 230 | 231 | [Fact] 232 | public void InitializeFromHostMetadata() 233 | { 234 | // TODO: It's tough to wire all this up without using a new host. 235 | IHost host = new HostBuilder() 236 | .ConfigureDefaultTestHost(builder => 237 | { 238 | builder.AddEventHubs(); 239 | }) 240 | .ConfigureAppConfiguration(c => 241 | { 242 | c.AddInMemoryCollection(new Dictionary 243 | { 244 | { "AzureWebJobs:extensions:EventHubs:EventProcessorOptions:MaxBatchSize", "100" }, 245 | { "AzureWebJobs:extensions:EventHubs:EventProcessorOptions:PrefetchCount", "200" }, 246 | { "AzureWebJobs:extensions:EventHubs:BatchCheckpointFrequency", "5" }, 247 | { "AzureWebJobs:extensions:EventHubs:PartitionManagerOptions:LeaseDuration", "00:00:31" }, 248 | { "AzureWebJobs:extensions:EventHubs:PartitionManagerOptions:RenewInterval", "00:00:21" } 249 | }); 250 | }) 251 | .Build(); 252 | 253 | // Force the ExtensionRegistryFactory to run, which will initialize the EventHubConfiguration. 254 | var extensionRegistry = host.Services.GetService(); 255 | var options = host.Services.GetService>().Value; 256 | 257 | var eventProcessorOptions = options.EventProcessorOptions; 258 | Assert.Equal(100, eventProcessorOptions.MaxBatchSize); 259 | Assert.Equal(200, eventProcessorOptions.PrefetchCount); 260 | Assert.Equal(5, options.BatchCheckpointFrequency); 261 | Assert.Equal(31, options.PartitionManagerOptions.LeaseDuration.TotalSeconds); 262 | Assert.Equal(21, options.PartitionManagerOptions.RenewInterval.TotalSeconds); 263 | } 264 | 265 | internal static PartitionContext GetPartitionContext(string partitionId = null, string eventHubPath = null, 266 | string consumerGroupName = null, string owner = null) 267 | { 268 | var constructor = typeof(PartitionContext).GetConstructor( 269 | BindingFlags.NonPublic | BindingFlags.Instance, 270 | null, 271 | new Type[] { typeof(EventProcessorHost), typeof(string), typeof(string), typeof(string), typeof(CancellationToken) }, 272 | null); 273 | var context = (PartitionContext)constructor.Invoke(new object[] { null, partitionId, eventHubPath, consumerGroupName, null }); 274 | 275 | // Set a lease, which allows us to grab the "Owner" 276 | constructor = typeof(Lease).GetConstructor(BindingFlags.NonPublic | BindingFlags.Instance, null, new Type[] { }, null); 277 | var lease = (Lease)constructor.Invoke(new object[] { }); 278 | lease.Owner = owner; 279 | 280 | var leaseProperty = typeof(PartitionContext).GetProperty("Lease", BindingFlags.Public | BindingFlags.Instance); 281 | leaseProperty.SetValue(context, lease); 282 | 283 | return context; 284 | } 285 | } 286 | } 287 | -------------------------------------------------------------------------------- /test/Microsoft.Azure.WebJobs.Extensions.EventHubs.Tests/Properties/AssemblyInfo.cs: -------------------------------------------------------------------------------- 1 | // Copyright (c) .NET Foundation. All rights reserved. 2 | // Licensed under the MIT License. See License.txt in the project root for license information. 3 | 4 | using System.Reflection; 5 | using Xunit; 6 | 7 | [assembly: CollectionBehavior(CollectionBehavior.CollectionPerAssembly)] -------------------------------------------------------------------------------- /test/Microsoft.Azure.WebJobs.Extensions.EventHubs.Tests/PublicSurfaceTests.cs: -------------------------------------------------------------------------------- 1 | // Copyright (c) .NET Foundation. All rights reserved. 2 | // Licensed under the MIT License. See License.txt in the project root for license information. 3 | 4 | using Microsoft.Azure.WebJobs.Host.TestCommon; 5 | using Xunit; 6 | 7 | namespace Microsoft.Azure.WebJobs.Host.UnitTests 8 | { 9 | public class PublicSurfaceTests 10 | { 11 | [Fact] 12 | public void WebJobs_Extensions_EventHubs_VerifyPublicSurfaceArea() 13 | { 14 | var assembly = typeof(EventHubAttribute).Assembly; 15 | 16 | var expected = new[] 17 | { 18 | "EventHubAttribute", 19 | "EventHubTriggerAttribute", 20 | "EventHubOptions", 21 | "InitialOffsetOptions", 22 | "EventHubWebJobsBuilderExtensions", 23 | "EventHubsWebJobsStartup" 24 | }; 25 | 26 | TestHelpers.AssertPublicTypes(expected, assembly); 27 | } 28 | } 29 | } 30 | -------------------------------------------------------------------------------- /test/Microsoft.Azure.WebJobs.Extensions.EventHubs.Tests/WebJobs.Extensions.EventHubs.Tests.csproj: -------------------------------------------------------------------------------- 1 |  2 | 3 | 4 | netcoreapp2.1 5 | false 6 | Microsoft.Azure.WebJobs.EventHubs.Tests 7 | Microsoft.Azure.WebJobs.EventHubs.Tests 8 | ..\..\src.ruleset 9 | true 10 | true 11 | ..\..\sign.snk 12 | 13 | 14 | false 15 | true 16 | 17 | 18 | 19 | false 20 | true 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | all 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | -------------------------------------------------------------------------------- /test/README.md: -------------------------------------------------------------------------------- 1 | # Event Hubs Extension for Azure Functions guide to running integration tests locally 2 | Integration tests are implemented in the `EventHubsEndToEndTests` and `EventHubApplicationInsightsTest` classes and require special configuration to execute locally in Visual Studio or via dotnet test. 3 | 4 | All configuration is done via a json file called `appsettings.tests` which on windows should be located in the `%USERPROFILE%\.azurefunctions` folder (e.g. `C:\Users\user123\.azurefunctions`) 5 | 6 | **Note:** *The specifics of the configuration will change when the validation code is modified so check the code for the latest configuration if the tests do not pass as this readme file may not have been updated with each code change.* 7 | 8 | Create the appropriate Azure resources if needed as explained below and create or update the `appsettings.tests` file in the location specified above by copying the configuration below and replacing all the `PLACEHOLDER` values 9 | 10 | appsettings.tests contents 11 | ``` 12 | { 13 | "ConnectionStrings": { 14 | "AzureWebJobsTestHubConnection": "PLACEHOLDER" 15 | }, 16 | "AzureWebJobsStorage": "PLACEHOLDER" 17 | } 18 | ``` 19 | ## Create Azure resources and configure test environment 20 | 1. Create a storage account and configure its connection string into `AzureWebJobsStorage`. This will be used by the webjobs hosts created by the tests as well as the Event Hubs extension for checkpointing. 21 | 2. Create anEvent Hubs namespace namespaces and within it, create an Event Hubs resource named `webjobstesthub`. 22 | 3. Navigate to the Event Hubs resource you created called `webjobstesthub` and create a "Shared access policy" with any name and "Manage, Send, Listen" claims. 23 | 4. Navigate to the policy above and copy either the primary or secondary connection string and set the value of the `AzureWebJobsTestHubConnection` element of the appsettings.tests file to the connection string you copied. 24 | 5. If you will keep this setup around for more than a day, set the "Message Retention" on the Event Hubs resource to the minimum value of 1 day to make tests run faster. --------------------------------------------------------------------------------