├── .gitignore ├── 01-JobsDemo.ps1 ├── 02-PoshRSJobDemo.ps1 ├── 03-PSThreadJobDemo.ps1 ├── 04-RunspaceDemo.ps1 ├── 05-MainDemo.ps1 ├── LICENSE └── README.md /.gitignore: -------------------------------------------------------------------------------- 1 | ## Ignore Visual Studio temporary files, build results, and 2 | ## files generated by popular Visual Studio add-ons. 3 | ## 4 | ## Get latest from https://github.com/github/gitignore/blob/master/VisualStudio.gitignore 5 | 6 | # User-specific files 7 | *.suo 8 | *.user 9 | *.userosscache 10 | *.sln.docstates 11 | 12 | # User-specific files (MonoDevelop/Xamarin Studio) 13 | *.userprefs 14 | 15 | # Build results 16 | [Dd]ebug/ 17 | [Dd]ebugPublic/ 18 | [Rr]elease/ 19 | [Rr]eleases/ 20 | x64/ 21 | x86/ 22 | bld/ 23 | [Bb]in/ 24 | [Oo]bj/ 25 | [Ll]og/ 26 | 27 | # Visual Studio 2015 cache/options directory 28 | .vs/ 29 | # Uncomment if you have tasks that create the project's static files in wwwroot 30 | #wwwroot/ 31 | 32 | # MSTest test Results 33 | [Tt]est[Rr]esult*/ 34 | [Bb]uild[Ll]og.* 35 | 36 | # NUNIT 37 | *.VisualState.xml 38 | TestResult.xml 39 | 40 | # Build Results of an ATL Project 41 | [Dd]ebugPS/ 42 | [Rr]eleasePS/ 43 | dlldata.c 44 | 45 | # .NET Core 46 | project.lock.json 47 | project.fragment.lock.json 48 | artifacts/ 49 | **/Properties/launchSettings.json 50 | 51 | *_i.c 52 | *_p.c 53 | *_i.h 54 | *.ilk 55 | *.meta 56 | *.obj 57 | *.pch 58 | *.pdb 59 | *.pgc 60 | *.pgd 61 | *.rsp 62 | *.sbr 63 | *.tlb 64 | *.tli 65 | *.tlh 66 | *.tmp 67 | *.tmp_proj 68 | *.log 69 | *.vspscc 70 | *.vssscc 71 | .builds 72 | *.pidb 73 | *.svclog 74 | *.scc 75 | 76 | # Chutzpah Test files 77 | _Chutzpah* 78 | 79 | # Visual C++ cache files 80 | ipch/ 81 | *.aps 82 | *.ncb 83 | *.opendb 84 | *.opensdf 85 | *.sdf 86 | *.cachefile 87 | *.VC.db 88 | *.VC.VC.opendb 89 | 90 | # Visual Studio profiler 91 | *.psess 92 | *.vsp 93 | *.vspx 94 | *.sap 95 | 96 | # TFS 2012 Local Workspace 97 | $tf/ 98 | 99 | # Guidance Automation Toolkit 100 | *.gpState 101 | 102 | # ReSharper is a .NET coding add-in 103 | _ReSharper*/ 104 | *.[Rr]e[Ss]harper 105 | *.DotSettings.user 106 | 107 | # JustCode is a .NET coding add-in 108 | .JustCode 109 | 110 | # TeamCity is a build add-in 111 | _TeamCity* 112 | 113 | # DotCover is a Code Coverage Tool 114 | *.dotCover 115 | 116 | # Visual Studio code coverage results 117 | *.coverage 118 | *.coveragexml 119 | 120 | # NCrunch 121 | _NCrunch_* 122 | .*crunch*.local.xml 123 | nCrunchTemp_* 124 | 125 | # MightyMoose 126 | *.mm.* 127 | AutoTest.Net/ 128 | 129 | # Web workbench (sass) 130 | .sass-cache/ 131 | 132 | # Installshield output folder 133 | [Ee]xpress/ 134 | 135 | # DocProject is a documentation generator add-in 136 | DocProject/buildhelp/ 137 | DocProject/Help/*.HxT 138 | DocProject/Help/*.HxC 139 | DocProject/Help/*.hhc 140 | DocProject/Help/*.hhk 141 | DocProject/Help/*.hhp 142 | DocProject/Help/Html2 143 | DocProject/Help/html 144 | 145 | # Click-Once directory 146 | publish/ 147 | 148 | # Publish Web Output 149 | *.[Pp]ublish.xml 150 | *.azurePubxml 151 | # TODO: Comment the next line if you want to checkin your web deploy settings 152 | # but database connection strings (with potential passwords) will be unencrypted 153 | *.pubxml 154 | *.publishproj 155 | 156 | # Microsoft Azure Web App publish settings. Comment the next line if you want to 157 | # checkin your Azure Web App publish settings, but sensitive information contained 158 | # in these scripts will be unencrypted 159 | PublishScripts/ 160 | 161 | # NuGet Packages 162 | *.nupkg 163 | # The packages folder can be ignored because of Package Restore 164 | **/packages/* 165 | # except build/, which is used as an MSBuild target. 166 | !**/packages/build/ 167 | # Uncomment if necessary however generally it will be regenerated when needed 168 | #!**/packages/repositories.config 169 | # NuGet v3's project.json files produces more ignorable files 170 | *.nuget.props 171 | *.nuget.targets 172 | 173 | # Microsoft Azure Build Output 174 | csx/ 175 | *.build.csdef 176 | 177 | # Microsoft Azure Emulator 178 | ecf/ 179 | rcf/ 180 | 181 | # Windows Store app package directories and files 182 | AppPackages/ 183 | BundleArtifacts/ 184 | Package.StoreAssociation.xml 185 | _pkginfo.txt 186 | 187 | # Visual Studio cache files 188 | # files ending in .cache can be ignored 189 | *.[Cc]ache 190 | # but keep track of directories ending in .cache 191 | !*.[Cc]ache/ 192 | 193 | # Others 194 | ClientBin/ 195 | ~$* 196 | *~ 197 | *.dbmdl 198 | *.dbproj.schemaview 199 | *.jfm 200 | *.pfx 201 | *.publishsettings 202 | orleans.codegen.cs 203 | 204 | # Since there are multiple workflows, uncomment next line to ignore bower_components 205 | # (https://github.com/github/gitignore/pull/1529#issuecomment-104372622) 206 | #bower_components/ 207 | 208 | # RIA/Silverlight projects 209 | Generated_Code/ 210 | 211 | # Backup & report files from converting an old project file 212 | # to a newer Visual Studio version. Backup files are not needed, 213 | # because we have git ;-) 214 | _UpgradeReport_Files/ 215 | Backup*/ 216 | UpgradeLog*.XML 217 | UpgradeLog*.htm 218 | 219 | # SQL Server files 220 | *.mdf 221 | *.ldf 222 | *.ndf 223 | 224 | # Business Intelligence projects 225 | *.rdl.data 226 | *.bim.layout 227 | *.bim_*.settings 228 | 229 | # Microsoft Fakes 230 | FakesAssemblies/ 231 | 232 | # GhostDoc plugin setting file 233 | *.GhostDoc.xml 234 | 235 | # Node.js Tools for Visual Studio 236 | .ntvs_analysis.dat 237 | node_modules/ 238 | 239 | # Typescript v1 declaration files 240 | typings/ 241 | 242 | # Visual Studio 6 build log 243 | *.plg 244 | 245 | # Visual Studio 6 workspace options file 246 | *.opt 247 | 248 | # Visual Studio 6 auto-generated workspace file (contains which files were open etc.) 249 | *.vbw 250 | 251 | # Visual Studio LightSwitch build output 252 | **/*.HTMLClient/GeneratedArtifacts 253 | **/*.DesktopClient/GeneratedArtifacts 254 | **/*.DesktopClient/ModelManifest.xml 255 | **/*.Server/GeneratedArtifacts 256 | **/*.Server/ModelManifest.xml 257 | _Pvt_Extensions 258 | 259 | # Paket dependency manager 260 | .paket/paket.exe 261 | paket-files/ 262 | 263 | # FAKE - F# Make 264 | .fake/ 265 | 266 | # JetBrains Rider 267 | .idea/ 268 | *.sln.iml 269 | 270 | # CodeRush 271 | .cr/ 272 | 273 | # Python Tools for Visual Studio (PTVS) 274 | __pycache__/ 275 | *.pyc 276 | 277 | # Cake - Uncomment if you are using it 278 | # tools/** 279 | # !tools/packages.config 280 | 281 | # Telerik's JustMock configuration file 282 | *.jmconfig 283 | 284 | # BizTalk build output 285 | *.btp.cs 286 | *.btm.cs 287 | *.odx.cs 288 | *.xsd.cs 289 | 290 | PSThreadJob/ 291 | -------------------------------------------------------------------------------- /01-JobsDemo.ps1: -------------------------------------------------------------------------------- 1 | Clear-Host 2 | 3 | # Start Jobs 4 | $Jobs = @( 5 | Start-Job { 6 | start-sleep -Seconds 1 7 | Get-Date 8 | } 9 | Start-Job { 10 | start-sleep -Seconds 2 11 | Get-Date 12 | } 13 | ) 14 | 15 | # Return Jobs 16 | @" 17 | 18 | 19 | Results: 20 | "@ 21 | $Jobs | 22 | Receive-Job -Wait 23 | 24 | # Cleanup 25 | $Jobs | Remove-Job 26 | @" 27 | 28 | 29 | "@ 30 | -------------------------------------------------------------------------------- /02-PoshRSJobDemo.ps1: -------------------------------------------------------------------------------- 1 | Clear-Host 2 | 3 | # Install Module 4 | $ModuleName = "PoshRSJob" 5 | $installModuleSplat = @{ 6 | SkipPublisherCheck = $true 7 | AcceptLicense = $true 8 | Name = $ModuleName 9 | Force = $true 10 | Scope = 'CurrentUser' 11 | AllowClobber = $true 12 | WarningAction = 'SilentlyContinue' 13 | } 14 | Install-Module @installModuleSplat 15 | 16 | # Run Jobs 17 | $Jobs = 1..5 | Start-RSJob -ScriptBlock { 18 | 'Running Job {0} at {1}' -f $_, (Get-Date) 19 | } 20 | 21 | # Return Jobs 22 | @" 23 | 24 | 25 | Results: 26 | "@ 27 | $Jobs | 28 | Wait-RSJob | 29 | Receive-RSJob 30 | 31 | # Cleanup 32 | $Jobs | Remove-RSJob 33 | @" 34 | 35 | 36 | "@ 37 | -------------------------------------------------------------------------------- /03-PSThreadJobDemo.ps1: -------------------------------------------------------------------------------- 1 | Clear-Host 2 | 3 | # Install Module 4 | $ModuleName = "ThreadJob" 5 | $installModuleSplat = @{ 6 | SkipPublisherCheck = $true 7 | AcceptLicense = $true 8 | Name = $ModuleName 9 | Force = $true 10 | Scope = 'CurrentUser' 11 | AllowClobber = $true 12 | WarningAction = 'SilentlyContinue' 13 | } 14 | Install-Module @installModuleSplat 15 | 16 | # Start Jobs 17 | $Jobs = @( 18 | Start-ThreadJob { 19 | start-sleep -Seconds 1 20 | Get-Date 21 | } 22 | Start-ThreadJob { 23 | start-sleep -Seconds 2 24 | Get-Date 25 | } 26 | ) 27 | 28 | # Return Results 29 | @" 30 | 31 | 32 | Results: 33 | "@ 34 | $Jobs | 35 | Receive-Job -Wait 36 | 37 | # Cleanup 38 | $Jobs | Remove-Job 39 | @" 40 | 41 | 42 | "@ -------------------------------------------------------------------------------- /04-RunspaceDemo.ps1: -------------------------------------------------------------------------------- 1 | Clear-Host 2 | 3 | # Build RunspacePool 4 | $RunspacePool = [runspacefactory]::CreateRunspacePool(1,4) 5 | $RunspacePool.Open() 6 | 7 | # Build Runspaces 8 | $Runspaces = 1..2 | ForEach-Object { 9 | $Runspace = [PowerShell]::Create() 10 | $Runspace.RunspacePool = $RunspacePool 11 | $Null = $Runspace.AddScript({ 12 | param($Count) 13 | Start-Sleep -Seconds $Count 14 | 'Running job {0} at {1}' -f $Count, (Get-Date) 15 | }).AddArgument($_) 16 | $Handler = $Runspace.BeginInvoke() 17 | [PSCustomObject]@{ 18 | Count = $_ 19 | PowerShell = $Runspace 20 | Handler = $Handler 21 | } 22 | } 23 | 24 | while ($Runspaces.Handler.IsCompleted -contains $false) { 25 | Start-Sleep -Milliseconds 500 26 | } 27 | 28 | # Get results and cleanup 29 | @" 30 | 31 | 32 | Results: 33 | "@ 34 | Foreach($Runspace in $Runspaces) { 35 | $Runspace.PowerShell.EndInvoke($Runspace.Handler) 36 | $Runspace.PowerShell.Dispose() 37 | } 38 | $RunspacePool.Dispose() 39 | @" 40 | 41 | 42 | "@ 43 | -------------------------------------------------------------------------------- /05-MainDemo.ps1: -------------------------------------------------------------------------------- 1 | Clear-Host 2 | 3 | # Settings 4 | $Folders = @( 5 | 'c:\ConcurrentDemo\Folder1' 6 | 'c:\ConcurrentDemo\Folder2' 7 | 'c:\ConcurrentDemo\Folder3' 8 | 'c:\ConcurrentDemo\Folder4' 9 | 'c:\ConcurrentDemo\Folder5' 10 | 'c:\ConcurrentDemo\Folder6' 11 | 'c:\ConcurrentDemo\Folder7' 12 | ) 13 | $LogPath = 'c:\ConcurrentDemo\Log.txt' 14 | 15 | # Modify these to change the number of each type of worker 16 | $FileProducersCount = 3 17 | $FileConsumersCount = 5 18 | $LogConsumersCount = 1 19 | 20 | 21 | 22 | # Create Files 23 | $RandomFileRange = 20,50 24 | Remove-Item -Recurse -Path 'c:\ConcurrentDemo\' -Force -ErrorAction SilentlyContinue 25 | $Null = foreach ($Folder in $Folders) { 26 | New-Item -ItemType Directory $Folder -ErrorAction SilentlyContinue 27 | $Files = Get-Random -Minimum $RandomFileRange[0] -Maximum $RandomFileRange[1] 28 | 0..$Files | ForEach-Object { 29 | $FileName = '{0}.txt' -f (New-Guid) 30 | New-Item -Path $Folder -Name $FileName -ItemType File 31 | } 32 | } 33 | 34 | # This ScriptBlock Produces a list of file names 35 | $FileProducer = { 36 | param( 37 | [System.Collections.Concurrent.BlockingCollection[PSObject]] 38 | $FolderQueue, 39 | 40 | [System.Collections.Concurrent.BlockingCollection[PSObject]] 41 | $FileNameQueue, 42 | 43 | [System.Collections.Concurrent.BlockingCollection[PSObject]] 44 | $LogQueue, 45 | 46 | [System.Collections.Concurrent.BlockingCollection[int]] 47 | $FileProducerStack, 48 | 49 | [String] 50 | $ThreadName 51 | ) 52 | 53 | # Add this thread to the stack 54 | $FileProducerStack.Add( 55 | [System.Threading.Thread]::CurrentThread.ManagedThreadId 56 | ) 57 | 58 | # Loop through each folder in the queue 59 | foreach ($FolderPath in $FolderQueue.GetConsumingEnumerable()) { 60 | # Loop through each file in the folder 61 | foreach ($Item in (Get-ChildItem $FolderPath)) { 62 | # Add the filename to File Name queue 63 | $FileNameQueue.Add($Item.Name) 64 | # Add a Log entry to the log queue 65 | $LogQueue.Add([PSCustomObject]@{ 66 | Date = Get-Date 67 | ThreadName = $ThreadName 68 | ThreadId = [System.Threading.Thread]::CurrentThread.ManagedThreadId 69 | Message = 'Found {0}' -f $Item.Name 70 | }) 71 | } 72 | } 73 | 74 | # Remove a thread from the stack 75 | $null = $FileProducerStack.Take() 76 | 77 | # Close $FileNameQueue if this is the last thread 78 | if($FileProducerStack.Count -lt 1) { 79 | $FileProducerStack.CompleteAdding() 80 | $FileNameQueue.CompleteAdding() 81 | } 82 | } 83 | 84 | # This ScriptBlock Reverse the file names 85 | $FileConsumer = { 86 | param( 87 | [System.Collections.Concurrent.BlockingCollection[PSObject]] 88 | $FileNameQueue, 89 | 90 | [System.Collections.Concurrent.BlockingCollection[PSObject]] 91 | $LogQueue, 92 | 93 | [System.Collections.Concurrent.BlockingCollection[int]] 94 | $FileConsumerStack, 95 | 96 | [String] 97 | $ThreadName 98 | ) 99 | 100 | # Add this thread to the stack 101 | $FileConsumerStack.Add( 102 | [System.Threading.Thread]::CurrentThread.ManagedThreadId 103 | ) 104 | 105 | # Loop through each filename 106 | foreach ($Filename in $FileNameQueue.GetConsumingEnumerable()) { 107 | # Reverse the file name 108 | $Chars = $FileName.ToCharArray() 109 | [Array]::Reverse($Chars) 110 | $Reversed = -join $Chars 111 | # Add message to the log queue 112 | $LogQueue.Add([PSCustomObject]@{ 113 | Date = Get-Date 114 | ThreadName = $ThreadName 115 | ThreadId = [System.Threading.Thread]::CurrentThread.ManagedThreadId 116 | Message = ("Old Name: '{0}'; New Name '{1}'" -f $Filename, $Reversed) 117 | }) 118 | } 119 | 120 | # Remove a thread from the stack 121 | $null = $FileConsumerStack.Take() 122 | 123 | # Close LogQueue if this is the last thread 124 | if($FileConsumerStack.Count -lt 1) { 125 | $FileConsumerStack.CompleteAdding() 126 | $LogQueue.CompleteAdding() 127 | } 128 | } 129 | 130 | # This ScriptBlock Logs Events from the other threads 131 | $LogConsumer = { 132 | param( 133 | [String] 134 | $LogPath, 135 | 136 | [System.Collections.Concurrent.BlockingCollection[PSObject]] 137 | $LogQueue, 138 | 139 | [String] 140 | $ThreadName 141 | ) 142 | 143 | # Log Start 144 | $Message = '{0} - {1:00000} - {2:-15} - {3}' -f @( 145 | (Get-Date).ToString('o') 146 | [System.Threading.Thread]::CurrentThread.ManagedThreadId 147 | $ThreadName 148 | 'Logging Start' 149 | ) 150 | $Message | Add-Content -Path $LogPath 151 | [console]::WriteLine($Message) 152 | 153 | # loop through the Log Queue and add the messages ot the log 154 | foreach ($LogEntry in $LogQueue.GetConsumingEnumerable()) { 155 | $Message = '{0} - {1:00000} - {2:-15} - {3}' -f @( 156 | $LogEntry.Date.ToString('o') 157 | $LogEntry.ThreadId 158 | $LogEntry.ThreadName 159 | $LogEntry.Message 160 | ) 161 | $Message | Add-Content -Path $LogPath 162 | [console]::WriteLine($Message) 163 | } 164 | 165 | $LogEnd 166 | $Message = '{0} - {1:00000} - {2:-15} - {3}' -f @( 167 | (Get-Date).ToString('o') 168 | [System.Threading.Thread]::CurrentThread.ManagedThreadId 169 | $ThreadName 170 | 'Logging Complete' 171 | ) 172 | $Message | Add-Content -Path $LogPath 173 | [console]::WriteLine($Message) 174 | } 175 | 176 | 177 | # Create the Queues and Stacks used 178 | # Queue for folder paths 179 | $FolderQueue = [System.Collections.Concurrent.BlockingCollection[PSObject]]::new( 180 | [System.Collections.Concurrent.ConcurrentQueue[PSObject]]::new() 181 | ) 182 | # Queue for File Names 183 | $FileNameQueue = [System.Collections.Concurrent.BlockingCollection[PSObject]]::new( 184 | [System.Collections.Concurrent.ConcurrentQueue[PSObject]]::new() 185 | ) 186 | # Queue for Log messages 187 | $LogQueue = [System.Collections.Concurrent.BlockingCollection[PSObject]]::new( 188 | [System.Collections.Concurrent.ConcurrentQueue[PSObject]]::new() 189 | ) 190 | # Stack for FileProducer Threads 191 | $FileProducerStack = [System.Collections.Concurrent.BlockingCollection[int]]::new( 192 | [System.Collections.Concurrent.ConcurrentStack[int]]::new() 193 | ) 194 | # Stack for FileConsumer Threads 195 | $FileConsumerStack = [System.Collections.Concurrent.BlockingCollection[int]]::new( 196 | [System.Collections.Concurrent.ConcurrentStack[int]]::new() 197 | ) 198 | 199 | # Create a list to hold the Runspaces 200 | $Runspaces = [System.Collections.Generic.List[PSObject]]::New() 201 | 202 | # Create the File Producer Pool 203 | $FileProducerPool = [RunspaceFactory]::CreateRunspacePool(1,$FileProducersCount) 204 | $FileProducerPool.Open() 205 | # Create the File Producer Runspaces 206 | 1..$FileProducersCount | ForEach-Object { 207 | $ThreadName = 'FileProducer{0:00}' -f $_ 208 | $Runspace = [PowerShell]::Create() 209 | $Runspace.RunspacePool = $FileProducerPool 210 | $null = $Runspace.AddScript($FileProducer). 211 | AddArgument($FolderQueue). 212 | AddArgument($FileNameQueue). 213 | AddArgument($LogQueue). 214 | AddArgument($FileProducerStack). 215 | AddArgument($ThreadName) 216 | $Runspaces.Add([PSCustomObject]@{ 217 | Name = $ThreadName 218 | PowerShell = $Runspace 219 | Handler = $Runspace.BeginInvoke() 220 | }) 221 | } 222 | 223 | # Create the File Consumer Pool 224 | $FileConsumerPool = [runspacefactory]::CreateRunspacePool(1,$FileConsumersCount) 225 | $FileConsumerPool.Open() 226 | # Create the File Consumer Runspaces 227 | 1..$FileConsumersCount | ForEach-Object { 228 | $ThreadName = 'FileConsumer{0:00}' -f $_ 229 | $Runspace = [PowerShell]::Create() 230 | $Runspace.RunspacePool = $FileConsumerPool 231 | $null = $Runspace.AddScript($FileConsumer). 232 | AddArgument($FileNameQueue). 233 | AddArgument($LogQueue). 234 | AddArgument($FileConsumerStack). 235 | AddArgument($ThreadName) 236 | $Runspaces.Add([PSCustomObject]@{ 237 | Name = $ThreadName 238 | PowerShell = $Runspace 239 | Handler = $Runspace.BeginInvoke() 240 | }) 241 | } 242 | 243 | # create the Log Consumer Pool 244 | $LogConsumerPool = [runspacefactory]::CreateRunspacePool(1,$LogConsumersCount) 245 | $LogConsumerPool.Open() 246 | # Create the Log Consumer Runspaces 247 | 1..$LogConsumersCount | ForEach-Object { 248 | $ThreadName = 'LogConsumer{0:00}' -f $_ 249 | $Runspace = [PowerShell]::Create() 250 | $Runspace.RunspacePool = $LogConsumerPool 251 | $null = $Runspace.AddScript($LogConsumer). 252 | AddArgument($LogPath). 253 | AddArgument($LogQueue). 254 | AddArgument($ThreadName) 255 | $Runspaces.Add([PSCustomObject]@{ 256 | Name = $ThreadName 257 | PowerShell = $Runspace 258 | Handler = $Runspace.BeginInvoke() 259 | }) 260 | } 261 | 262 | # At this point, All runspaces are running but doing nothing. 263 | # Now we feed the list of folder paths to the Folder queue 264 | $null = $Folders | ForEach-Object { $FolderQueue.Add($_)} 265 | $null = $FolderQueue.CompleteAdding() 266 | 267 | # Wait for the threads to complete 268 | while ($Runspaces.Handler.IsCompleted -contains $false) { 269 | Start-Sleep -Milliseconds 500 270 | } 271 | 272 | # Cleanup 273 | Foreach($Runspace in $Runspaces) { 274 | $Runspace.PowerShell.EndInvoke($Runspace.Handler) 275 | $Runspace.PowerShell.Dispose() 276 | } 277 | $FileProducerPool.Dispose() 278 | $FileConsumerPool.Dispose() 279 | $LogConsumerPool.Dispose() 280 | 281 | # View the log 282 | Invoke-Item $LogPath 283 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 Mark Kraus 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Concurrent Programming in PowerShell with the Producer Consumer Pattern 2 | 3 | ## Topics 4 | 5 | * Who is Mark Kraus? 6 | * What is Concurrent Programming? 7 | * Examples of concurrent programming in PowerShell 8 | * What is the Producer-Consumer Pattern? 9 | * Demo 10 | 11 | ## Who Is Mark Kraus 12 | 13 | ![markekraus](https://avatars1.githubusercontent.com/u/6509955?s=100&v=4) 14 | 15 | * Senior Systems Engineer @ LinkedIn 16 | * PowerShell Core Project Collaborator 17 | * Web Cmdlets Contributor 18 | * Author of [Get-PowerShellBlog](https://get-powershellblog.blogspot.com/) 19 | * Co-Author of [The PowerShell Conference Book](https://leanpub.com/powershell-conference-book) 20 | 21 | ## What is Concurrent Computing? 22 | 23 | > Concurrent computing is a form of computing in which several computations are executed during overlapping time periods—concurrently—instead of sequentially (one completing before the next starts). ([Wikipedia](https://en.wikipedia.org/wiki/Concurrent_computing)) 24 | 25 | ## Serial vs Concurrent vs Parallel 26 | 27 | ### Terms 28 | 29 | * Core 30 | * CPU 31 | * Thread 32 | * "Virtual CPU" that runs on a Core 33 | * Task 34 | * Code that runs in a Thread 35 | 36 | ### Serial 37 | 38 | * Tasks complete one after the other 39 | * Tasks never run at the same time 40 | * Uses a single thread 41 | * Slowest 42 | 43 | ```none 44 | Task 1: + 45 | Task 2: = 46 | 47 | Core 1: +++++++++++++++++++++++++++=========================== 48 | ``` 49 | 50 | ### Parallel 51 | 52 | * Uses one core per task 53 | * Not possible on single core system 54 | * Faster but more expensive 55 | 56 | ```none 57 | Task 1: + 58 | Task 2: = 59 | 60 | Core 1: +++++++++++++++++++++++++++ 61 | Thread 1: +++++++++++++++++++++++++++ 62 | 63 | Core 2: =========================== 64 | Thread 1: =========================== 65 | ``` 66 | 67 | ### Concurrent 68 | 69 | * Uses multiple threads 70 | * Can run on single core 71 | * Slower but cheaper 72 | 73 | ```none 74 | Task 1: + 75 | Task 2: = 76 | 77 | Core 1: +++===++++++======+++===++++++======+++++++++========= 78 | Thread 1: +++++++++++++++++++++++++++ 79 | Thread 2: =========================== 80 | ``` 81 | 82 | ## PowerShell Concurrency Options 83 | 84 | A quick recap of the available concurrent programming methods in PowerShell. 85 | 86 | * PowerShell Jobs 87 | * PoshRSJob 88 | * PSThreadJob 89 | * Runspaces 90 | 91 | ## Producer-Consumer Pattern 92 | 93 | ### What is the Produce-Consumer Pattern? 94 | 95 | * Producer creates (produces) items 96 | * Consumer uses (consumes) items from the Producer 97 | * The PowerShell Pipeline is a Producer-Consumer 98 | 99 | ```powershell 100 | Get-Job | Wait-Job | Receive-Job 101 | ``` 102 | 103 | * `Get-Job` produces a list of Jobs 104 | * `Wait-Job` consumes `Get-Jobs` results then also produces a list of jobs 105 | * `Receive-Job` consumes `Wait-Job` results 106 | 107 | ### Producer-Consumer in Concurrent Programming 108 | 109 | * Multiple Producers of the same item 110 | * Multiple Consumers of those items 111 | * Producers and Consumers Threads 112 | * Threads are ScriptBlocks 113 | * Increase and decrease Producers and consumers as needed 114 | 115 | ### Widget Factory Analogy 116 | 117 | 118 | 119 | The Widget Factory turns monads into widgets. 120 | 121 | Receiving: 122 | 123 | * Multiple suppliers deliver monads at various times 124 | * Sometimes Multiple suppliers deliver at once 125 | * Receiving has multiple delivery bays 126 | * All deliveries feed to a single monad line 127 | * monads travel to Manufacturing 128 | 129 | Manufacturing: 130 | 131 | * Line workers take monads and build widgets 132 | * Multiple line workers 133 | * When their current widget is done they grab the next available monad 134 | * Sometimes need Monads at the same time 135 | * Widgets go out to Shipping 136 | 137 | Shipping: 138 | 139 | * Multiple pickups will be made by multiple distributors 140 | * Sometimes multiple distributors arrive at the same time 141 | * All shipments must have X number of widgets 142 | * Shipping bundles the widgets and puts them on distributor trucks 143 | 144 | ### Back to Programming 145 | 146 | * Sometimes we need to deal with more than one source of data 147 | * Sometimes the amount of data in is to great to process serially 148 | * Sometimes there are multiple consumers of our processed data 149 | * Most often, this work is being broken up into batches. 150 | * When one batch completes the next starts. 151 | * Leads to under-utilized threads 152 | 153 | We want all or workers 100% utilized at all times unless there is no work to be done! 154 | 155 | 156 | ### Real World Example: Inbox Rules 157 | 158 | * Hybrid exchange with On-prem and On-cloud mailboxes 159 | * Must enumerated all mailboxes in both 160 | * Getting Inbox rules is a slow and expensive Task 161 | * Multiple service accounts needed to get rules 162 | * Service accounts cannot constantly open and close PowerShell sessions 163 | * Throttling per-user considerations 164 | * Rules then need to be processed and compiled into a report 165 | * Logging and error detection 166 | 167 | [Gist](https://gist.github.com/markekraus/2f1c376af1c69911b2421eb8c263b5f6) 168 | 169 | ### Secret Ingredients 170 | 171 | * [Thread-Safe Collections](https://docs.microsoft.com/en-us/dotnet/standard/collections/thread-safe/) 172 | * `BlockingCollection` 173 | [Link](https://docs.microsoft.com/en-us/dotnet/api/system.collections.concurrent.blockingcollection-1?view=netframework-4.7.2) 174 | * `ConcurrentQueue` 175 | [Link](https://docs.microsoft.com/en-us/dotnet/api/system.collections.concurrent.concurrentqueue-1?view=netframework-4.7.2) 176 | * `ConcurrentStack` 177 | [link](https://docs.microsoft.com/en-us/dotnet/api/system.collections.concurrent.concurrentstack-1?view=netframework-4.7.2) 178 | * `RunspacePool` 179 | [Link](https://docs.microsoft.com/en-us/dotnet/api/system.management.automation.runspaces.runspacepool?view=powershellsdk-1.1.0) 180 | * `PowerShell` 181 | [Link](https://docs.microsoft.com/en-us/dotnet/api/system.management.automation.powershell?view=powershellsdk-1.1.0) 182 | 183 | ```powershell 184 | using namespace System.Collections.Concurrent 185 | $Queue = [BlockingCollection[PSObject]]::new( 186 | [ConcurrentQueue[PSObject]]::new() 187 | ) 188 | $RunspacePool = [runspacefactory]::CreateRunspacePool(1,4) 189 | $RunspacePool.Open() 190 | $Runspace = [PowerShell]::Create() 191 | $Runspace.RunspacePool = $RunspacePool 192 | $Runspace.AddScript($ScriptBlock).AddArgument($Queue) 193 | $Runspace.BeginInvoke() 194 | ``` 195 | 196 | ### Blocking 197 | 198 | * `BlockingCollection.GetConsumingEnumerable()` 199 | * Blocks the thread until it can take an item 200 | * `BlockingCollection.CompleteAdding()` 201 | * Marks the collection complete 202 | * `GetConsumingEnumerable()` will then exit the loop 203 | 204 | Thread 1: 205 | 206 | ```powershell 207 | foreach ($LogEntry in $LogQueue.GetConsumingEnumerable()) { 208 | # do stuff 209 | } 210 | ``` 211 | 212 | Thread 2: 213 | 214 | ```powershell 215 | $LogQueue.Add($Message1) 216 | $LogQueue.Add($Message2) 217 | $LogQueue.Add($Message3) 218 | $LogQueue.CompleteAdding() 219 | ``` 220 | 221 | ### Stacks for Thread Tracking 222 | 223 | * `ConcurrentStack` 224 | * Used to track how peer threads 225 | * last thread completes the output queue 226 | 227 | ```powershell 228 | $FileConsumerStack.Add( 229 | [System.Threading.Thread]::CurrentThread.ManagedThreadId 230 | ) 231 | # do stuff 232 | # Remove a thread from the stack 233 | $null = $FileConsumerStack.Take() 234 | # Close LogQueue if this is the last thread 235 | if($FileConsumerStack.Count -lt 1) { 236 | $FileConsumerStack.CompleteAdding() 237 | $LogQueue.CompleteAdding() 238 | } 239 | ``` 240 | 241 | ### Demo 242 | 243 | 244 | 245 | * Enumerate files in multiple folders 246 | * Take the Filenames and reverse them 247 | * Log new names 248 | 249 | [Main Demo](./05-MainDemo.ps1) 250 | 251 | ### Demo Notes 252 | 253 | * Threads are started before any folders are supplied 254 | * Toggling the number of File Producers and File Consumers 255 | * Can have more File producers than Folders 256 | * Single log consumer to prevent file locks 257 | 258 | ## Links 259 | 260 | * [https://github.com/markekraus](https://github.com/markekraus) 261 | * Material: [https://github.com/markekraus/ConcurrentPowerShellProducerConsumer](https://github.com/markekraus/ConcurrentPowerShellProducerConsumer) 262 | * [https://get-powershellblog.blogspot.com/](https://get-powershellblog.blogspot.com/) 263 | * [@markekraus](https://twitter.com/markekraus) 264 | --------------------------------------------------------------------------------