├── .gitignore ├── BulkDeleteSample ├── BulkDeleteSample.sln └── BulkDeleteSample │ ├── App.config │ ├── BulkDeleteSample.csproj │ ├── Program.cs │ ├── Utils.cs │ └── packages.config ├── BulkImportSample ├── BulkImportSample.sln └── BulkImportSample │ ├── App.config │ ├── BulkImportSample.csproj │ ├── NugetPackages │ └── Microsoft.Azure.CosmosDB.BulkExecutor.0.0.9-preview.nupkg │ ├── Program.cs │ ├── Properties │ └── AssemblyInfo.cs │ ├── Utils.cs │ └── packages.config ├── BulkUpdateSample ├── BulkUpdateSample.sln └── BulkUpdateSample │ ├── App.config │ ├── BulkUpdateSample.csproj │ ├── Program.cs │ ├── Properties │ └── AssemblyInfo.cs │ ├── Utils.cs │ └── packages.config ├── LICENSE └── README.md /.gitignore: -------------------------------------------------------------------------------- 1 | ## Ignore Visual Studio temporary files, build results, and 2 | ## files generated by popular Visual Studio add-ons. 3 | ## 4 | ## Get latest from https://github.com/github/gitignore/blob/master/VisualStudio.gitignore 5 | 6 | # User-specific files 7 | *.suo 8 | *.user 9 | *.userosscache 10 | *.sln.docstates 11 | 12 | # User-specific files (MonoDevelop/Xamarin Studio) 13 | *.userprefs 14 | 15 | # Build results 16 | [Dd]ebug/ 17 | [Dd]ebugPublic/ 18 | [Rr]elease/ 19 | [Rr]eleases/ 20 | x64/ 21 | x86/ 22 | bld/ 23 | [Bb]in/ 24 | [Oo]bj/ 25 | [Ll]og/ 26 | 27 | # Visual Studio 2015 cache/options directory 28 | .vs/ 29 | # Uncomment if you have tasks that create the project's static files in wwwroot 30 | #wwwroot/ 31 | 32 | # MSTest test Results 33 | [Tt]est[Rr]esult*/ 34 | [Bb]uild[Ll]og.* 35 | 36 | # NUNIT 37 | *.VisualState.xml 38 | TestResult.xml 39 | 40 | # Build Results of an ATL Project 41 | [Dd]ebugPS/ 42 | [Rr]eleasePS/ 43 | dlldata.c 44 | 45 | # .NET Core 46 | project.lock.json 47 | project.fragment.lock.json 48 | artifacts/ 49 | **/Properties/launchSettings.json 50 | 51 | *_i.c 52 | *_p.c 53 | *_i.h 54 | *.ilk 55 | *.meta 56 | *.obj 57 | *.pch 58 | *.pdb 59 | *.pgc 60 | *.pgd 61 | *.rsp 62 | *.sbr 63 | *.tlb 64 | *.tli 65 | *.tlh 66 | *.tmp 67 | *.tmp_proj 68 | *.log 69 | *.vspscc 70 | *.vssscc 71 | .builds 72 | *.pidb 73 | *.svclog 74 | *.scc 75 | 76 | # Chutzpah Test files 77 | _Chutzpah* 78 | 79 | # Visual C++ cache files 80 | ipch/ 81 | *.aps 82 | *.ncb 83 | *.opendb 84 | *.opensdf 85 | *.sdf 86 | *.cachefile 87 | *.VC.db 88 | *.VC.VC.opendb 89 | 90 | # Visual Studio profiler 91 | *.psess 92 | *.vsp 93 | *.vspx 94 | *.sap 95 | 96 | # TFS 2012 Local Workspace 97 | $tf/ 98 | 99 | # Guidance Automation Toolkit 100 | *.gpState 101 | 102 | # ReSharper is a .NET coding add-in 103 | _ReSharper*/ 104 | *.[Rr]e[Ss]harper 105 | *.DotSettings.user 106 | 107 | # JustCode is a .NET coding add-in 108 | .JustCode 109 | 110 | # TeamCity is a build add-in 111 | _TeamCity* 112 | 113 | # DotCover is a Code Coverage Tool 114 | *.dotCover 115 | 116 | # Visual Studio code coverage results 117 | *.coverage 118 | *.coveragexml 119 | 120 | # NCrunch 121 | _NCrunch_* 122 | .*crunch*.local.xml 123 | nCrunchTemp_* 124 | 125 | # MightyMoose 126 | *.mm.* 127 | AutoTest.Net/ 128 | 129 | # Web workbench (sass) 130 | .sass-cache/ 131 | 132 | # Installshield output folder 133 | [Ee]xpress/ 134 | 135 | # DocProject is a documentation generator add-in 136 | DocProject/buildhelp/ 137 | DocProject/Help/*.HxT 138 | DocProject/Help/*.HxC 139 | DocProject/Help/*.hhc 140 | DocProject/Help/*.hhk 141 | DocProject/Help/*.hhp 142 | DocProject/Help/Html2 143 | DocProject/Help/html 144 | 145 | # Click-Once directory 146 | publish/ 147 | 148 | # Publish Web Output 149 | *.[Pp]ublish.xml 150 | *.azurePubxml 151 | # TODO: Comment the next line if you want to checkin your web deploy settings 152 | # but database connection strings (with potential passwords) will be unencrypted 153 | *.pubxml 154 | *.publishproj 155 | 156 | # Microsoft Azure Web App publish settings. Comment the next line if you want to 157 | # checkin your Azure Web App publish settings, but sensitive information contained 158 | # in these scripts will be unencrypted 159 | PublishScripts/ 160 | 161 | # NuGet Packages 162 | # *.nupkg 163 | # The packages folder can be ignored because of Package Restore 164 | **/packages/* 165 | # except build/, which is used as an MSBuild target. 166 | !**/packages/build/ 167 | # Uncomment if necessary however generally it will be regenerated when needed 168 | #!**/packages/repositories.config 169 | # NuGet v3's project.json files produces more ignorable files 170 | *.nuget.props 171 | *.nuget.targets 172 | 173 | # Microsoft Azure Build Output 174 | csx/ 175 | *.build.csdef 176 | 177 | # Microsoft Azure Emulator 178 | ecf/ 179 | rcf/ 180 | 181 | # Windows Store app package directories and files 182 | AppPackages/ 183 | BundleArtifacts/ 184 | Package.StoreAssociation.xml 185 | _pkginfo.txt 186 | 187 | # Visual Studio cache files 188 | # files ending in .cache can be ignored 189 | *.[Cc]ache 190 | # but keep track of directories ending in .cache 191 | !*.[Cc]ache/ 192 | 193 | # Others 194 | ClientBin/ 195 | ~$* 196 | *~ 197 | *.dbmdl 198 | *.dbproj.schemaview 199 | *.jfm 200 | *.pfx 201 | *.publishsettings 202 | orleans.codegen.cs 203 | 204 | # Since there are multiple workflows, uncomment next line to ignore bower_components 205 | # (https://github.com/github/gitignore/pull/1529#issuecomment-104372622) 206 | #bower_components/ 207 | 208 | # RIA/Silverlight projects 209 | Generated_Code/ 210 | 211 | # Backup & report files from converting an old project file 212 | # to a newer Visual Studio version. Backup files are not needed, 213 | # because we have git ;-) 214 | _UpgradeReport_Files/ 215 | Backup*/ 216 | UpgradeLog*.XML 217 | UpgradeLog*.htm 218 | 219 | # SQL Server files 220 | *.mdf 221 | *.ldf 222 | *.ndf 223 | 224 | # Business Intelligence projects 225 | *.rdl.data 226 | *.bim.layout 227 | *.bim_*.settings 228 | 229 | # Microsoft Fakes 230 | FakesAssemblies/ 231 | 232 | # GhostDoc plugin setting file 233 | *.GhostDoc.xml 234 | 235 | # Node.js Tools for Visual Studio 236 | .ntvs_analysis.dat 237 | node_modules/ 238 | 239 | # Typescript v1 declaration files 240 | typings/ 241 | 242 | # Visual Studio 6 build log 243 | *.plg 244 | 245 | # Visual Studio 6 workspace options file 246 | *.opt 247 | 248 | # Visual Studio 6 auto-generated workspace file (contains which files were open etc.) 249 | *.vbw 250 | 251 | # Visual Studio LightSwitch build output 252 | **/*.HTMLClient/GeneratedArtifacts 253 | **/*.DesktopClient/GeneratedArtifacts 254 | **/*.DesktopClient/ModelManifest.xml 255 | **/*.Server/GeneratedArtifacts 256 | **/*.Server/ModelManifest.xml 257 | _Pvt_Extensions 258 | 259 | # Paket dependency manager 260 | .paket/paket.exe 261 | paket-files/ 262 | 263 | # FAKE - F# Make 264 | .fake/ 265 | 266 | # JetBrains Rider 267 | .idea/ 268 | *.sln.iml 269 | 270 | # CodeRush 271 | .cr/ 272 | 273 | # Python Tools for Visual Studio (PTVS) 274 | __pycache__/ 275 | *.pyc 276 | 277 | # Cake - Uncomment if you are using it 278 | # tools/** 279 | # !tools/packages.config 280 | 281 | # Telerik's JustMock configuration file 282 | *.jmconfig 283 | 284 | # BizTalk build output 285 | *.btp.cs 286 | *.btm.cs 287 | *.odx.cs 288 | *.xsd.cs 289 | -------------------------------------------------------------------------------- /BulkDeleteSample/BulkDeleteSample.sln: -------------------------------------------------------------------------------- 1 |  2 | Microsoft Visual Studio Solution File, Format Version 12.00 3 | # Visual Studio 14 4 | VisualStudioVersion = 14.0.25420.1 5 | MinimumVisualStudioVersion = 10.0.40219.1 6 | Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "BulkDeleteSample", "BulkDeleteSample\BulkDeleteSample.csproj", "{2FE25F96-02DC-45BE-B284-B753037322BE}" 7 | EndProject 8 | Global 9 | GlobalSection(SolutionConfigurationPlatforms) = preSolution 10 | Debug|Any CPU = Debug|Any CPU 11 | Release|Any CPU = Release|Any CPU 12 | EndGlobalSection 13 | GlobalSection(ProjectConfigurationPlatforms) = postSolution 14 | {2FE25F96-02DC-45BE-B284-B753037322BE}.Debug|Any CPU.ActiveCfg = Debug|Any CPU 15 | {2FE25F96-02DC-45BE-B284-B753037322BE}.Debug|Any CPU.Build.0 = Debug|Any CPU 16 | {2FE25F96-02DC-45BE-B284-B753037322BE}.Release|Any CPU.ActiveCfg = Release|Any CPU 17 | {2FE25F96-02DC-45BE-B284-B753037322BE}.Release|Any CPU.Build.0 = Release|Any CPU 18 | EndGlobalSection 19 | GlobalSection(SolutionProperties) = preSolution 20 | HideSolutionNode = FALSE 21 | EndGlobalSection 22 | EndGlobal 23 | -------------------------------------------------------------------------------- /BulkDeleteSample/BulkDeleteSample/App.config: -------------------------------------------------------------------------------- 1 |  2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | -------------------------------------------------------------------------------- /BulkDeleteSample/BulkDeleteSample/BulkDeleteSample.csproj: -------------------------------------------------------------------------------- 1 |  2 | 3 | 4 | 5 | Debug 6 | AnyCPU 7 | {2FE25F96-02DC-45BE-B284-B753037322BE} 8 | Exe 9 | Properties 10 | BulkDeleteSample 11 | BulkDeleteSample 12 | v4.5.2 13 | 512 14 | true 15 | 16 | 17 | 18 | 19 | AnyCPU 20 | true 21 | full 22 | false 23 | bin\Debug\ 24 | DEBUG;TRACE 25 | prompt 26 | 4 27 | 28 | 29 | AnyCPU 30 | pdbonly 31 | true 32 | bin\Release\ 33 | TRACE 34 | prompt 35 | 4 36 | 37 | 38 | 39 | ..\packages\Microsoft.Azure.CosmosDB.BulkExecutor.1.4.1\lib\net451\Microsoft.Azure.CosmosDB.BulkImport.dll 40 | True 41 | 42 | 43 | ..\packages\Microsoft.Azure.DocumentDB.2.1.3\lib\net45\Microsoft.Azure.Documents.Client.dll 44 | True 45 | 46 | 47 | ..\packages\MongoDB.Bson.signed.2.4.4\lib\net45\MongoDB.Bson.dll 48 | True 49 | 50 | 51 | ..\packages\Newtonsoft.Json.9.0.1\lib\net45\Newtonsoft.Json.dll 52 | True 53 | 54 | 55 | 56 | 57 | 58 | ..\packages\System.Numerics.Vectors.4.5.0\lib\portable-net45+win8+wp8+wpa81\System.Numerics.Vectors.dll 59 | True 60 | 61 | 62 | ..\packages\System.Runtime.CompilerServices.Unsafe.4.3.0\lib\netstandard1.0\System.Runtime.CompilerServices.Unsafe.dll 63 | True 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 | 85 | 86 | 87 | This project references NuGet package(s) that are missing on this computer. Use NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. 88 | 89 | 90 | 91 | 98 | -------------------------------------------------------------------------------- /BulkDeleteSample/BulkDeleteSample/Program.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections.Generic; 3 | using System.Configuration; 4 | using System.Diagnostics; 5 | using System.Linq; 6 | using System.Text; 7 | using System.Threading; 8 | using System.Threading.Tasks; 9 | 10 | using Microsoft.Azure.Documents; 11 | using Microsoft.Azure.Documents.Client; 12 | using Microsoft.Azure.CosmosDB.BulkExecutor; 13 | using Microsoft.Azure.CosmosDB.BulkExecutor.BulkImport; 14 | using Microsoft.Azure.CosmosDB.BulkExecutor.BulkDelete; 15 | 16 | namespace BulkDeleteSample 17 | { 18 | class Program 19 | { 20 | private static readonly string EndpointUrl = ConfigurationManager.AppSettings["EndPointUrl"]; 21 | private static readonly string AuthorizationKey = ConfigurationManager.AppSettings["AuthorizationKey"]; 22 | private static readonly string DatabaseName = ConfigurationManager.AppSettings["DatabaseName"]; 23 | private static readonly string CollectionName = ConfigurationManager.AppSettings["CollectionName"]; 24 | private static readonly int CollectionThroughput = int.Parse(ConfigurationManager.AppSettings["CollectionThroughput"]); 25 | 26 | private static readonly ConnectionPolicy ConnectionPolicy = new ConnectionPolicy 27 | { 28 | ConnectionMode = ConnectionMode.Direct, 29 | ConnectionProtocol = Protocol.Tcp 30 | }; 31 | 32 | private DocumentClient client; 33 | private DocumentCollection dataCollection; 34 | 35 | /// 36 | /// Initializes a new instance of the class. 37 | /// 38 | /// The DocumentDB client instance. 39 | private Program(DocumentClient client) 40 | { 41 | this.client = client; 42 | } 43 | 44 | public static void Main(string[] args) 45 | { 46 | Trace.WriteLine("Summary:"); 47 | Trace.WriteLine("--------------------------------------------------------------------- "); 48 | Trace.WriteLine(String.Format("Endpoint: {0}", EndpointUrl)); 49 | Trace.WriteLine(String.Format("Collection : {0}.{1}", DatabaseName, CollectionName)); 50 | Trace.WriteLine("--------------------------------------------------------------------- "); 51 | Trace.WriteLine(""); 52 | 53 | try 54 | { 55 | using (var client = new DocumentClient( 56 | new Uri(EndpointUrl), 57 | AuthorizationKey, 58 | ConnectionPolicy)) 59 | { 60 | var program = new Program(client); 61 | program.RunBulkImportAndBulkDeleteAsync().Wait(); 62 | } 63 | } 64 | catch (AggregateException e) 65 | { 66 | Trace.TraceError("Caught AggregateException in Main, Inner Exception:\n" + e); 67 | Console.ReadKey(); 68 | } 69 | } 70 | 71 | private async Task RunBulkImport() 72 | { 73 | // Cleanup on start if set in config. 74 | 75 | try 76 | { 77 | if (bool.Parse(ConfigurationManager.AppSettings["ShouldCleanupOnStart"])) 78 | { 79 | Database database = Utils.GetDatabaseIfExists(client, DatabaseName); 80 | if (database != null) 81 | { 82 | await client.DeleteDatabaseAsync(database.SelfLink); 83 | } 84 | 85 | Trace.TraceInformation("Creating database {0}", DatabaseName); 86 | database = await client.CreateDatabaseAsync(new Database { Id = DatabaseName }); 87 | 88 | Trace.TraceInformation(String.Format("Creating collection {0} with {1} RU/s", CollectionName, CollectionThroughput)); 89 | this.dataCollection = await Utils.CreatePartitionedCollectionAsync(client, DatabaseName, CollectionName, CollectionThroughput); 90 | } 91 | else 92 | { 93 | this.dataCollection = Utils.GetCollectionIfExists(client, DatabaseName, CollectionName); 94 | if (this.dataCollection == null) 95 | { 96 | throw new Exception("The data collection does not exist"); 97 | } 98 | } 99 | } 100 | catch (Exception de) 101 | { 102 | Trace.TraceError("Unable to initialize, exception message: {0}", de.Message); 103 | throw; 104 | } 105 | 106 | // Prepare for bulk import. 107 | 108 | // Creating documents with simple partition key here. 109 | string partitionKeyProperty = this.dataCollection.PartitionKey.Paths[0].Replace("/", ""); 110 | 111 | long numberOfDocumentsToGenerate = long.Parse(ConfigurationManager.AppSettings["NumberOfDocumentsToDelete"]); 112 | int numberOfBatches = int.Parse(ConfigurationManager.AppSettings["NumberOfBatches"]); 113 | long numberOfDocumentsPerBatch = (long)Math.Floor(((double)numberOfDocumentsToGenerate) / numberOfBatches); 114 | 115 | // Set retry options high for initialization (default values). 116 | client.ConnectionPolicy.RetryOptions.MaxRetryWaitTimeInSeconds = 30; 117 | client.ConnectionPolicy.RetryOptions.MaxRetryAttemptsOnThrottledRequests = 9; 118 | 119 | IBulkExecutor bulkExecutor = new BulkExecutor(client, dataCollection); 120 | await bulkExecutor.InitializeAsync(); 121 | 122 | // Set retries to 0 to pass control to bulk executor. 123 | client.ConnectionPolicy.RetryOptions.MaxRetryWaitTimeInSeconds = 0; 124 | client.ConnectionPolicy.RetryOptions.MaxRetryAttemptsOnThrottledRequests = 0; 125 | 126 | BulkImportResponse bulkImportResponse = null; 127 | long totalNumberOfDocumentsInserted = 0; 128 | double totalRequestUnitsConsumed = 0; 129 | double totalTimeTakenSec = 0; 130 | 131 | var tokenSource = new CancellationTokenSource(); 132 | var token = tokenSource.Token; 133 | 134 | for (int i = 0; i < numberOfBatches; i++) 135 | { 136 | // Generate JSON-serialized documents to import. 137 | 138 | List documentsToImportInBatch = new List(); 139 | long prefix = i * numberOfDocumentsPerBatch; 140 | 141 | Trace.TraceInformation(String.Format("Generating {0} documents to import for batch {1}", numberOfDocumentsPerBatch, i)); 142 | for (int j = 0; j < numberOfDocumentsPerBatch; j++) 143 | { 144 | string partitionKeyValue = (prefix + j).ToString(); 145 | string id = partitionKeyValue; 146 | 147 | documentsToImportInBatch.Add(Utils.GenerateRandomDocumentString(id, partitionKeyProperty, partitionKeyValue)); 148 | } 149 | 150 | // Invoke bulk import API. 151 | 152 | var tasks = new List(); 153 | 154 | tasks.Add(Task.Run(async () => 155 | { 156 | Trace.TraceInformation(String.Format("Executing bulk import for batch {0}", i)); 157 | do 158 | { 159 | try 160 | { 161 | bulkImportResponse = await bulkExecutor.BulkImportAsync( 162 | documents: documentsToImportInBatch, 163 | enableUpsert: true, 164 | disableAutomaticIdGeneration: true, 165 | maxConcurrencyPerPartitionKeyRange: null, 166 | maxInMemorySortingBatchSize: null, 167 | cancellationToken: token); 168 | } 169 | catch (DocumentClientException de) 170 | { 171 | Trace.TraceError("Document client exception: {0}", de); 172 | break; 173 | } 174 | catch (Exception e) 175 | { 176 | Trace.TraceError("Exception: {0}", e); 177 | break; 178 | } 179 | } while (bulkImportResponse.NumberOfDocumentsImported < documentsToImportInBatch.Count); 180 | 181 | Trace.WriteLine(String.Format("\nSummary for batch {0}:", i)); 182 | Trace.WriteLine("--------------------------------------------------------------------- "); 183 | Trace.WriteLine(String.Format("Inserted {0} docs @ {1} writes/s, {2} RU/s in {3} sec", 184 | bulkImportResponse.NumberOfDocumentsImported, 185 | Math.Round(bulkImportResponse.NumberOfDocumentsImported / bulkImportResponse.TotalTimeTaken.TotalSeconds), 186 | Math.Round(bulkImportResponse.TotalRequestUnitsConsumed / bulkImportResponse.TotalTimeTaken.TotalSeconds), 187 | bulkImportResponse.TotalTimeTaken.TotalSeconds)); 188 | Trace.WriteLine(String.Format("Average RU consumption per document insert: {0}", 189 | (bulkImportResponse.TotalRequestUnitsConsumed / bulkImportResponse.NumberOfDocumentsImported))); 190 | Trace.WriteLine("---------------------------------------------------------------------\n "); 191 | 192 | totalNumberOfDocumentsInserted += bulkImportResponse.NumberOfDocumentsImported; 193 | totalRequestUnitsConsumed += bulkImportResponse.TotalRequestUnitsConsumed; 194 | totalTimeTakenSec += bulkImportResponse.TotalTimeTaken.TotalSeconds; 195 | }, 196 | token)); 197 | 198 | await Task.WhenAll(tasks); 199 | } 200 | 201 | Trace.WriteLine("Overall summary of bulk import:"); 202 | Trace.WriteLine("--------------------------------------------------------------------- "); 203 | Trace.WriteLine(String.Format("Inserted {0} docs @ {1} writes/s, {2} RU/s in {3} sec", 204 | totalNumberOfDocumentsInserted, 205 | Math.Round(totalNumberOfDocumentsInserted / totalTimeTakenSec), 206 | Math.Round(totalRequestUnitsConsumed / totalTimeTakenSec), 207 | totalTimeTakenSec)); 208 | Trace.WriteLine(String.Format("Average RU consumption per document insert: {0}", 209 | (totalRequestUnitsConsumed / totalNumberOfDocumentsInserted))); 210 | Trace.WriteLine("--------------------------------------------------------------------- \n"); 211 | 212 | //----------------------------------------------------------------------------------------------- 213 | } 214 | 215 | private List> GeneratePartitionKeyDocumentIdTuplesToBulkDelete() 216 | { 217 | long NumberOfDocumentsToDelete = long.Parse(ConfigurationManager.AppSettings["NumberOfDocumentsToDelete"]); 218 | 219 | List> pkIdTuplesToDelete = new List>(); 220 | for(int i=0; i < NumberOfDocumentsToDelete; i++) 221 | { 222 | pkIdTuplesToDelete.Add(new Tuple(i.ToString(), i.ToString())); 223 | } 224 | 225 | return pkIdTuplesToDelete; 226 | } 227 | 228 | /// 229 | /// Driver function for bulk import. 230 | /// 231 | /// 232 | private async Task RunBulkImportAndBulkDeleteAsync() 233 | { 234 | // Import documents into the collection 235 | await this.RunBulkImport(); 236 | 237 | // Fetch tuples to delete in bulk 238 | List> pkIdTuplesToDelete = GeneratePartitionKeyDocumentIdTuplesToBulkDelete(); 239 | 240 | long totalNumberOfDocumentsDeleted = 0; 241 | double totalRequestUnitsConsumed = 0; 242 | double totalTimeTakenSec = 0; 243 | BulkDeleteResponse bulkDeleteResponse = null; 244 | 245 | BulkExecutor bulkExecutor = new BulkExecutor(this.client, this.dataCollection); 246 | await bulkExecutor.InitializeAsync(); 247 | try 248 | { 249 | bulkDeleteResponse = await bulkExecutor.BulkDeleteAsync(pkIdTuplesToDelete); 250 | totalNumberOfDocumentsDeleted = bulkDeleteResponse.NumberOfDocumentsDeleted; 251 | totalRequestUnitsConsumed = bulkDeleteResponse.TotalRequestUnitsConsumed; 252 | totalTimeTakenSec = bulkDeleteResponse.TotalTimeTaken.TotalSeconds; 253 | } 254 | catch (DocumentClientException de) 255 | { 256 | Trace.TraceError("Document client exception: {0}", de); 257 | } 258 | catch (Exception e) 259 | { 260 | Trace.TraceError("Exception: {0}", e); 261 | } 262 | 263 | Trace.WriteLine("\n\n--------------------------------------------------------------------- "); 264 | Trace.WriteLine("Executing bulk delete:"); 265 | Trace.WriteLine("--------------------------------------------------------------------- "); 266 | Trace.WriteLine("\n\nOverall summary of bulk delete:"); 267 | Trace.WriteLine("--------------------------------------------------------------------- "); 268 | Trace.WriteLine(String.Format("Deleted {0} docs @ {1} writes/s, {2} RU/s in {3} sec", 269 | totalNumberOfDocumentsDeleted, 270 | Math.Round(totalNumberOfDocumentsDeleted / totalTimeTakenSec), 271 | Math.Round(totalRequestUnitsConsumed / totalTimeTakenSec), 272 | totalTimeTakenSec)); 273 | Trace.WriteLine(String.Format("Average RU consumption per document delete: {0}", 274 | (totalRequestUnitsConsumed / totalNumberOfDocumentsDeleted))); 275 | Trace.WriteLine("--------------------------------------------------------------------- \n"); 276 | 277 | //----------------------------------------------------------------------------------------------- 278 | } 279 | } 280 | } 281 | -------------------------------------------------------------------------------- /BulkDeleteSample/BulkDeleteSample/Utils.cs: -------------------------------------------------------------------------------- 1 | //------------------------------------------------------------ 2 | // Copyright (c) Microsoft Corporation. All rights reserved. 3 | //------------------------------------------------------------ 4 | 5 | namespace BulkDeleteSample 6 | { 7 | using Microsoft.Azure.Documents; 8 | using Microsoft.Azure.Documents.Client; 9 | 10 | using System; 11 | using System.Collections.ObjectModel; 12 | using System.Configuration; 13 | using System.Linq; 14 | using System.Threading.Tasks; 15 | 16 | class Utils 17 | { 18 | /// 19 | /// Get the collection if it exists, null if it doesn't. 20 | /// 21 | /// The requested collection. 22 | static internal DocumentCollection GetCollectionIfExists(DocumentClient client, string databaseName, string collectionName) 23 | { 24 | if (GetDatabaseIfExists(client, databaseName) == null) 25 | { 26 | return null; 27 | } 28 | 29 | return client.CreateDocumentCollectionQuery(UriFactory.CreateDatabaseUri(databaseName)) 30 | .Where(c => c.Id == collectionName).AsEnumerable().FirstOrDefault(); 31 | } 32 | 33 | /// 34 | /// Get the database if it exists, null if it doesn't. 35 | /// 36 | /// The requested database. 37 | static internal Database GetDatabaseIfExists(DocumentClient client, string databaseName) 38 | { 39 | return client.CreateDatabaseQuery().Where(d => d.Id == databaseName).AsEnumerable().FirstOrDefault(); 40 | } 41 | 42 | /// 43 | /// Create a partitioned collection. 44 | /// 45 | /// The created collection. 46 | static internal async Task CreatePartitionedCollectionAsync(DocumentClient client, string databaseName, 47 | string collectionName, int collectionThroughput) 48 | { 49 | PartitionKeyDefinition partitionKey = new PartitionKeyDefinition 50 | { 51 | Paths = new Collection { ConfigurationManager.AppSettings["CollectionPartitionKey"] } 52 | }; 53 | DocumentCollection collection = new DocumentCollection { Id = collectionName, PartitionKey = partitionKey }; 54 | 55 | try 56 | { 57 | collection = await client.CreateDocumentCollectionAsync( 58 | UriFactory.CreateDatabaseUri(databaseName), 59 | collection, 60 | new RequestOptions { OfferThroughput = collectionThroughput }); 61 | } 62 | catch (Exception e) 63 | { 64 | throw e; 65 | } 66 | 67 | return collection; 68 | } 69 | 70 | static internal String GenerateRandomDocumentString(String id, String partitionKeyProperty, object parititonKeyValue) 71 | { 72 | return "{\n" + 73 | " \"id\": \"" + id + "\",\n" + 74 | " \"" + partitionKeyProperty + "\": \"" + parititonKeyValue + "\",\n" + 75 | " \"Name\": \"TestDoc\",\n" + 76 | " \"description\": \"1.99\",\n" + 77 | " \"f1\": \"3hrkjh3h4h4h3jk4h\",\n" + 78 | " \"f2\": \"dhfkjdhfhj4434434\",\n" + 79 | " \"f3\": \"nklfjeoirje434344\",\n" + 80 | " \"f4\": \"pjfgdgfhdgfgdhbd6\",\n" + 81 | " \"f5\": \"gyuerehvrerebrhjh\",\n" + 82 | " \"f6\": \"3434343ghghghgghj\"" + 83 | "}"; 84 | } 85 | } 86 | } 87 | -------------------------------------------------------------------------------- /BulkDeleteSample/BulkDeleteSample/packages.config: -------------------------------------------------------------------------------- 1 |  2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | -------------------------------------------------------------------------------- /BulkImportSample/BulkImportSample.sln: -------------------------------------------------------------------------------- 1 |  2 | Microsoft Visual Studio Solution File, Format Version 12.00 3 | # Visual Studio 14 4 | VisualStudioVersion = 14.0.25420.1 5 | MinimumVisualStudioVersion = 10.0.40219.1 6 | Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "BulkImportSample", "BulkImportSample\BulkImportSample.csproj", "{38689E63-036A-4600-B570-CCF82EC58545}" 7 | EndProject 8 | Global 9 | GlobalSection(SolutionConfigurationPlatforms) = preSolution 10 | Debug|Any CPU = Debug|Any CPU 11 | Release|Any CPU = Release|Any CPU 12 | EndGlobalSection 13 | GlobalSection(ProjectConfigurationPlatforms) = postSolution 14 | {38689E63-036A-4600-B570-CCF82EC58545}.Debug|Any CPU.ActiveCfg = Release|x64 15 | {38689E63-036A-4600-B570-CCF82EC58545}.Debug|Any CPU.Build.0 = Release|x64 16 | {38689E63-036A-4600-B570-CCF82EC58545}.Release|Any CPU.ActiveCfg = Release|x64 17 | {38689E63-036A-4600-B570-CCF82EC58545}.Release|Any CPU.Build.0 = Release|x64 18 | EndGlobalSection 19 | GlobalSection(SolutionProperties) = preSolution 20 | HideSolutionNode = FALSE 21 | EndGlobalSection 22 | EndGlobal 23 | -------------------------------------------------------------------------------- /BulkImportSample/BulkImportSample/App.config: -------------------------------------------------------------------------------- 1 |  2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | -------------------------------------------------------------------------------- /BulkImportSample/BulkImportSample/BulkImportSample.csproj: -------------------------------------------------------------------------------- 1 |  2 | 3 | 4 | 5 | Debug 6 | AnyCPU 7 | {38689E63-036A-4600-B570-CCF82EC58545} 8 | Exe 9 | Properties 10 | BulkImportSample 11 | BulkImportSample 12 | v4.5.2 13 | 512 14 | true 15 | 16 | 17 | 18 | 19 | AnyCPU 20 | true 21 | full 22 | false 23 | bin\Debug\ 24 | DEBUG;TRACE 25 | prompt 26 | 4 27 | 28 | 29 | AnyCPU 30 | pdbonly 31 | true 32 | bin\Release\ 33 | TRACE 34 | prompt 35 | 4 36 | 37 | 38 | true 39 | bin\x64\Debug\ 40 | DEBUG;TRACE 41 | full 42 | x64 43 | prompt 44 | MinimumRecommendedRules.ruleset 45 | true 46 | 47 | 48 | bin\x64\Release\ 49 | TRACE 50 | true 51 | pdbonly 52 | x64 53 | prompt 54 | MinimumRecommendedRules.ruleset 55 | true 56 | 57 | 58 | 59 | ..\packages\Microsoft.Azure.CosmosDB.BulkExecutor.1.1.0\lib\net451\Microsoft.Azure.CosmosDB.BulkImport.dll 60 | True 61 | 62 | 63 | ..\packages\Microsoft.Azure.DocumentDB.2.0.0\lib\net45\Microsoft.Azure.Documents.Client.dll 64 | True 65 | 66 | 67 | ..\packages\Newtonsoft.Json.9.0.1\lib\net45\Newtonsoft.Json.dll 68 | True 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 | 85 | 86 | 87 | Designer 88 | 89 | 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | 98 | This project references NuGet package(s) that are missing on this computer. Use NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. 99 | 100 | 101 | 102 | 109 | -------------------------------------------------------------------------------- /BulkImportSample/BulkImportSample/NugetPackages/Microsoft.Azure.CosmosDB.BulkExecutor.0.0.9-preview.nupkg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/azure-cosmosdb-bulkexecutor-dotnet-getting-started/cb54c8624ade7731d40f08fb61f1bc1fa9f2d8ff/BulkImportSample/BulkImportSample/NugetPackages/Microsoft.Azure.CosmosDB.BulkExecutor.0.0.9-preview.nupkg -------------------------------------------------------------------------------- /BulkImportSample/BulkImportSample/Program.cs: -------------------------------------------------------------------------------- 1 | //------------------------------------------------------------ 2 | // Copyright (c) Microsoft Corporation. All rights reserved. 3 | //------------------------------------------------------------ 4 | 5 | namespace BulkImportSample 6 | { 7 | using System; 8 | using System.Configuration; 9 | using System.Collections.Generic; 10 | using System.Diagnostics; 11 | using System.Threading; 12 | using System.Threading.Tasks; 13 | 14 | using Microsoft.Azure.Documents; 15 | using Microsoft.Azure.Documents.Client; 16 | using Microsoft.Azure.CosmosDB.BulkExecutor; 17 | using Microsoft.Azure.CosmosDB.BulkExecutor.BulkImport; 18 | 19 | class Program 20 | { 21 | private static readonly string EndpointUrl = ConfigurationManager.AppSettings["EndPointUrl"]; 22 | private static readonly string AuthorizationKey = ConfigurationManager.AppSettings["AuthorizationKey"]; 23 | private static readonly string DatabaseName = ConfigurationManager.AppSettings["DatabaseName"]; 24 | private static readonly string CollectionName = ConfigurationManager.AppSettings["CollectionName"]; 25 | private static readonly int CollectionThroughput = int.Parse(ConfigurationManager.AppSettings["CollectionThroughput"]); 26 | 27 | private static readonly ConnectionPolicy ConnectionPolicy = new ConnectionPolicy 28 | { 29 | ConnectionMode = ConnectionMode.Direct, 30 | ConnectionProtocol = Protocol.Tcp 31 | }; 32 | 33 | private DocumentClient client; 34 | 35 | /// 36 | /// Initializes a new instance of the class. 37 | /// 38 | /// The DocumentDB client instance. 39 | private Program(DocumentClient client) 40 | { 41 | this.client = client; 42 | } 43 | 44 | public static void Main(string[] args) 45 | { 46 | Trace.WriteLine("Summary:"); 47 | Trace.WriteLine("--------------------------------------------------------------------- "); 48 | Trace.WriteLine(String.Format("Endpoint: {0}", EndpointUrl)); 49 | Trace.WriteLine(String.Format("Collection : {0}.{1}", DatabaseName, CollectionName)); 50 | Trace.WriteLine("--------------------------------------------------------------------- "); 51 | Trace.WriteLine(""); 52 | 53 | try 54 | { 55 | using (var client = new DocumentClient( 56 | new Uri(EndpointUrl), 57 | AuthorizationKey, 58 | ConnectionPolicy)) 59 | { 60 | var program = new Program(client); 61 | program.RunBulkImportAsync().Wait(); 62 | } 63 | } 64 | catch (AggregateException e) 65 | { 66 | Trace.TraceError("Caught AggregateException in Main, Inner Exception:\n" + e); 67 | Console.ReadKey(); 68 | } 69 | 70 | } 71 | 72 | /// 73 | /// Driver function for bulk import. 74 | /// 75 | /// 76 | private async Task RunBulkImportAsync() 77 | { 78 | // Cleanup on start if set in config. 79 | 80 | DocumentCollection dataCollection = null; 81 | try 82 | { 83 | if (bool.Parse(ConfigurationManager.AppSettings["ShouldCleanupOnStart"])) 84 | { 85 | Database database = Utils.GetDatabaseIfExists(client, DatabaseName); 86 | if (database != null) 87 | { 88 | await client.DeleteDatabaseAsync(database.SelfLink); 89 | } 90 | 91 | Trace.TraceInformation("Creating database {0}", DatabaseName); 92 | database = await client.CreateDatabaseAsync(new Database { Id = DatabaseName }); 93 | 94 | Trace.TraceInformation(String.Format("Creating collection {0} with {1} RU/s", CollectionName, CollectionThroughput)); 95 | dataCollection = await Utils.CreatePartitionedCollectionAsync(client, DatabaseName, CollectionName, CollectionThroughput); 96 | } 97 | else 98 | { 99 | dataCollection = Utils.GetCollectionIfExists(client, DatabaseName, CollectionName); 100 | if (dataCollection == null) 101 | { 102 | throw new Exception("The data collection does not exist"); 103 | } 104 | } 105 | } 106 | catch (Exception de) 107 | { 108 | Trace.TraceError("Unable to initialize, exception message: {0}", de.Message); 109 | throw; 110 | } 111 | 112 | // Prepare for bulk import. 113 | 114 | // Creating documents with simple partition key here. 115 | string partitionKeyProperty = dataCollection.PartitionKey.Paths[0].Replace("/", ""); 116 | 117 | long numberOfDocumentsToGenerate = long.Parse(ConfigurationManager.AppSettings["NumberOfDocumentsToImport"]); 118 | int numberOfBatches = int.Parse(ConfigurationManager.AppSettings["NumberOfBatches"]); 119 | long numberOfDocumentsPerBatch = (long)Math.Floor(((double)numberOfDocumentsToGenerate) / numberOfBatches); 120 | 121 | // Set retry options high for initialization (default values). 122 | client.ConnectionPolicy.RetryOptions.MaxRetryWaitTimeInSeconds = 30; 123 | client.ConnectionPolicy.RetryOptions.MaxRetryAttemptsOnThrottledRequests = 9; 124 | 125 | IBulkExecutor bulkExecutor = new BulkExecutor(client, dataCollection); 126 | await bulkExecutor.InitializeAsync(); 127 | 128 | // Set retries to 0 to pass control to bulk executor. 129 | client.ConnectionPolicy.RetryOptions.MaxRetryWaitTimeInSeconds = 0; 130 | client.ConnectionPolicy.RetryOptions.MaxRetryAttemptsOnThrottledRequests = 0; 131 | 132 | BulkImportResponse bulkImportResponse = null; 133 | long totalNumberOfDocumentsInserted = 0; 134 | double totalRequestUnitsConsumed = 0; 135 | double totalTimeTakenSec = 0; 136 | 137 | var tokenSource = new CancellationTokenSource(); 138 | var token = tokenSource.Token; 139 | 140 | for (int i = 0; i < numberOfBatches; i++) 141 | { 142 | // Generate JSON-serialized documents to import. 143 | 144 | List documentsToImportInBatch = new List(); 145 | long prefix = i * numberOfDocumentsPerBatch; 146 | 147 | Trace.TraceInformation(String.Format("Generating {0} documents to import for batch {1}", numberOfDocumentsPerBatch, i)); 148 | for (int j = 0; j < numberOfDocumentsPerBatch; j++) 149 | { 150 | string partitionKeyValue = (prefix + j).ToString(); 151 | string id = partitionKeyValue + Guid.NewGuid().ToString(); 152 | 153 | documentsToImportInBatch.Add(Utils.GenerateRandomDocumentString(id, partitionKeyProperty, partitionKeyValue)); 154 | } 155 | 156 | // Invoke bulk import API. 157 | 158 | var tasks = new List(); 159 | 160 | tasks.Add(Task.Run(async () => 161 | { 162 | Trace.TraceInformation(String.Format("Executing bulk import for batch {0}", i)); 163 | do 164 | { 165 | try 166 | { 167 | bulkImportResponse = await bulkExecutor.BulkImportAsync( 168 | documents: documentsToImportInBatch, 169 | enableUpsert: true, 170 | disableAutomaticIdGeneration: true, 171 | maxConcurrencyPerPartitionKeyRange: null, 172 | maxInMemorySortingBatchSize: null, 173 | cancellationToken: token); 174 | } 175 | catch (DocumentClientException de) 176 | { 177 | Trace.TraceError("Document client exception: {0}", de); 178 | break; 179 | } 180 | catch (Exception e) 181 | { 182 | Trace.TraceError("Exception: {0}", e); 183 | break; 184 | } 185 | } while (bulkImportResponse.NumberOfDocumentsImported < documentsToImportInBatch.Count); 186 | 187 | Trace.WriteLine(String.Format("\nSummary for batch {0}:", i)); 188 | Trace.WriteLine("--------------------------------------------------------------------- "); 189 | Trace.WriteLine(String.Format("Inserted {0} docs @ {1} writes/s, {2} RU/s in {3} sec", 190 | bulkImportResponse.NumberOfDocumentsImported, 191 | Math.Round(bulkImportResponse.NumberOfDocumentsImported / bulkImportResponse.TotalTimeTaken.TotalSeconds), 192 | Math.Round(bulkImportResponse.TotalRequestUnitsConsumed / bulkImportResponse.TotalTimeTaken.TotalSeconds), 193 | bulkImportResponse.TotalTimeTaken.TotalSeconds)); 194 | Trace.WriteLine(String.Format("Average RU consumption per document: {0}", 195 | (bulkImportResponse.TotalRequestUnitsConsumed / bulkImportResponse.NumberOfDocumentsImported))); 196 | Trace.WriteLine("---------------------------------------------------------------------\n "); 197 | 198 | totalNumberOfDocumentsInserted += bulkImportResponse.NumberOfDocumentsImported; 199 | totalRequestUnitsConsumed += bulkImportResponse.TotalRequestUnitsConsumed; 200 | totalTimeTakenSec += bulkImportResponse.TotalTimeTaken.TotalSeconds; 201 | }, 202 | token)); 203 | 204 | /* 205 | tasks.Add(Task.Run(() => 206 | { 207 | char ch = Console.ReadKey(true).KeyChar; 208 | if (ch == 'c' || ch == 'C') 209 | { 210 | tokenSource.Cancel(); 211 | Trace.WriteLine("\nTask cancellation requested."); 212 | } 213 | })); 214 | */ 215 | 216 | await Task.WhenAll(tasks); 217 | } 218 | 219 | Trace.WriteLine("Overall summary:"); 220 | Trace.WriteLine("--------------------------------------------------------------------- "); 221 | Trace.WriteLine(String.Format("Inserted {0} docs @ {1} writes/s, {2} RU/s in {3} sec", 222 | totalNumberOfDocumentsInserted, 223 | Math.Round(totalNumberOfDocumentsInserted / totalTimeTakenSec), 224 | Math.Round(totalRequestUnitsConsumed / totalTimeTakenSec), 225 | totalTimeTakenSec)); 226 | Trace.WriteLine(String.Format("Average RU consumption per document: {0}", 227 | (totalRequestUnitsConsumed / totalNumberOfDocumentsInserted))); 228 | Trace.WriteLine("--------------------------------------------------------------------- "); 229 | 230 | // Cleanup on finish if set in config. 231 | 232 | if (bool.Parse(ConfigurationManager.AppSettings["ShouldCleanupOnFinish"])) 233 | { 234 | Trace.TraceInformation("Deleting Database {0}", DatabaseName); 235 | await client.DeleteDatabaseAsync(UriFactory.CreateDatabaseUri(DatabaseName)); 236 | } 237 | 238 | Trace.WriteLine("\nPress any key to exit."); 239 | Console.ReadKey(); 240 | } 241 | } 242 | } 243 | -------------------------------------------------------------------------------- /BulkImportSample/BulkImportSample/Properties/AssemblyInfo.cs: -------------------------------------------------------------------------------- 1 | using System.Reflection; 2 | using System.Runtime.CompilerServices; 3 | using System.Runtime.InteropServices; 4 | 5 | // General Information about an assembly is controlled through the following 6 | // set of attributes. Change these attribute values to modify the information 7 | // associated with an assembly. 8 | [assembly: AssemblyTitle("BulkImportSample")] 9 | [assembly: AssemblyDescription("")] 10 | [assembly: AssemblyConfiguration("")] 11 | [assembly: AssemblyCompany("")] 12 | [assembly: AssemblyProduct("BulkImportSample")] 13 | [assembly: AssemblyCopyright("Copyright © 2018")] 14 | [assembly: AssemblyTrademark("")] 15 | [assembly: AssemblyCulture("")] 16 | 17 | // Setting ComVisible to false makes the types in this assembly not visible 18 | // to COM components. If you need to access a type in this assembly from 19 | // COM, set the ComVisible attribute to true on that type. 20 | [assembly: ComVisible(false)] 21 | 22 | // The following GUID is for the ID of the typelib if this project is exposed to COM 23 | [assembly: Guid("38689e63-036a-4600-b570-ccf82ec58545")] 24 | 25 | // Version information for an assembly consists of the following four values: 26 | // 27 | // Major Version 28 | // Minor Version 29 | // Build Number 30 | // Revision 31 | // 32 | // You can specify all the values or you can default the Build and Revision Numbers 33 | // by using the '*' as shown below: 34 | // [assembly: AssemblyVersion("1.0.*")] 35 | [assembly: AssemblyVersion("1.0.0.0")] 36 | [assembly: AssemblyFileVersion("1.0.0.0")] 37 | -------------------------------------------------------------------------------- /BulkImportSample/BulkImportSample/Utils.cs: -------------------------------------------------------------------------------- 1 | //------------------------------------------------------------ 2 | // Copyright (c) Microsoft Corporation. All rights reserved. 3 | //------------------------------------------------------------ 4 | 5 | namespace BulkImportSample 6 | { 7 | using Microsoft.Azure.Documents; 8 | using Microsoft.Azure.Documents.Client; 9 | 10 | using System; 11 | using System.Collections.ObjectModel; 12 | using System.Configuration; 13 | using System.Linq; 14 | using System.Threading.Tasks; 15 | 16 | class Utils 17 | { 18 | /// 19 | /// Get the collection if it exists, null if it doesn't. 20 | /// 21 | /// The requested collection. 22 | static internal DocumentCollection GetCollectionIfExists(DocumentClient client, string databaseName, string collectionName) 23 | { 24 | if (GetDatabaseIfExists(client, databaseName) == null) 25 | { 26 | return null; 27 | } 28 | 29 | return client.CreateDocumentCollectionQuery(UriFactory.CreateDatabaseUri(databaseName)) 30 | .Where(c => c.Id == collectionName).AsEnumerable().FirstOrDefault(); 31 | } 32 | 33 | /// 34 | /// Get the database if it exists, null if it doesn't. 35 | /// 36 | /// The requested database. 37 | static internal Database GetDatabaseIfExists(DocumentClient client, string databaseName) 38 | { 39 | return client.CreateDatabaseQuery().Where(d => d.Id == databaseName).AsEnumerable().FirstOrDefault(); 40 | } 41 | 42 | /// 43 | /// Create a partitioned collection. 44 | /// 45 | /// The created collection. 46 | static internal async Task CreatePartitionedCollectionAsync(DocumentClient client, string databaseName, 47 | string collectionName, int collectionThroughput) 48 | { 49 | PartitionKeyDefinition partitionKey = new PartitionKeyDefinition 50 | { 51 | Paths = new Collection { ConfigurationManager.AppSettings["CollectionPartitionKey"] } 52 | }; 53 | DocumentCollection collection = new DocumentCollection { Id = collectionName, PartitionKey = partitionKey }; 54 | 55 | try 56 | { 57 | collection = await client.CreateDocumentCollectionAsync( 58 | UriFactory.CreateDatabaseUri(databaseName), 59 | collection, 60 | new RequestOptions { OfferThroughput = collectionThroughput }); 61 | } 62 | catch (Exception e) 63 | { 64 | throw e; 65 | } 66 | 67 | return collection; 68 | } 69 | 70 | static internal String GenerateRandomDocumentString(String id, String partitionKeyProperty, object parititonKeyValue) 71 | { 72 | return "{\n" + 73 | " \"id\": \"" + id + "\",\n" + 74 | " \"" + partitionKeyProperty + "\": \"" + parititonKeyValue + "\",\n" + 75 | " \"Name\": \"TestDoc\",\n" + 76 | " \"description\": \"1.99\",\n" + 77 | " \"f1\": \"3hrkjh3h4h4h3jk4h\",\n" + 78 | " \"f2\": \"dhfkjdhfhj4434434\",\n" + 79 | " \"f3\": \"nklfjeoirje434344\",\n" + 80 | " \"f4\": \"pjfgdgfhdgfgdhbd6\",\n" + 81 | " \"f5\": \"gyuerehvrerebrhjh\",\n" + 82 | " \"f6\": \"3434343ghghghgghj\"" + 83 | "}"; 84 | } 85 | } 86 | } 87 | -------------------------------------------------------------------------------- /BulkImportSample/BulkImportSample/packages.config: -------------------------------------------------------------------------------- 1 |  2 | 3 | 4 | 5 | 6 | -------------------------------------------------------------------------------- /BulkUpdateSample/BulkUpdateSample.sln: -------------------------------------------------------------------------------- 1 |  2 | Microsoft Visual Studio Solution File, Format Version 12.00 3 | # Visual Studio 14 4 | VisualStudioVersion = 14.0.25420.1 5 | MinimumVisualStudioVersion = 10.0.40219.1 6 | Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "BulkUpdateSample", "BulkUpdateSample\BulkUpdateSample.csproj", "{9871A21E-3F29-4827-A44C-3F76169D8952}" 7 | EndProject 8 | Global 9 | GlobalSection(SolutionConfigurationPlatforms) = preSolution 10 | Debug|Any CPU = Debug|Any CPU 11 | Release|Any CPU = Release|Any CPU 12 | EndGlobalSection 13 | GlobalSection(ProjectConfigurationPlatforms) = postSolution 14 | {9871A21E-3F29-4827-A44C-3F76169D8952}.Debug|Any CPU.ActiveCfg = Debug|Any CPU 15 | {9871A21E-3F29-4827-A44C-3F76169D8952}.Debug|Any CPU.Build.0 = Debug|Any CPU 16 | {9871A21E-3F29-4827-A44C-3F76169D8952}.Release|Any CPU.ActiveCfg = Release|Any CPU 17 | {9871A21E-3F29-4827-A44C-3F76169D8952}.Release|Any CPU.Build.0 = Release|Any CPU 18 | EndGlobalSection 19 | GlobalSection(SolutionProperties) = preSolution 20 | HideSolutionNode = FALSE 21 | EndGlobalSection 22 | EndGlobal 23 | -------------------------------------------------------------------------------- /BulkUpdateSample/BulkUpdateSample/App.config: -------------------------------------------------------------------------------- 1 |  2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | -------------------------------------------------------------------------------- /BulkUpdateSample/BulkUpdateSample/BulkUpdateSample.csproj: -------------------------------------------------------------------------------- 1 |  2 | 3 | 4 | 5 | Debug 6 | AnyCPU 7 | {9871A21E-3F29-4827-A44C-3F76169D8952} 8 | Exe 9 | Properties 10 | BulkUpdateSample 11 | BulkUpdateSample 12 | v4.5.2 13 | 512 14 | true 15 | 16 | 17 | 18 | 19 | AnyCPU 20 | true 21 | full 22 | false 23 | bin\Debug\ 24 | DEBUG;TRACE 25 | prompt 26 | 4 27 | 28 | 29 | AnyCPU 30 | pdbonly 31 | true 32 | bin\Release\ 33 | TRACE 34 | prompt 35 | 4 36 | 37 | 38 | 39 | ..\packages\Microsoft.Azure.CosmosDB.BulkExecutor.1.1.0\lib\net451\Microsoft.Azure.CosmosDB.BulkImport.dll 40 | True 41 | 42 | 43 | ..\packages\Microsoft.Azure.DocumentDB.2.0.0\lib\net45\Microsoft.Azure.Documents.Client.dll 44 | True 45 | 46 | 47 | ..\packages\Newtonsoft.Json.9.0.1\lib\net45\Newtonsoft.Json.dll 48 | True 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | Designer 68 | 69 | 70 | Designer 71 | 72 | 73 | 74 | 75 | 76 | 77 | This project references NuGet package(s) that are missing on this computer. Use NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. 78 | 79 | 80 | 81 | 88 | -------------------------------------------------------------------------------- /BulkUpdateSample/BulkUpdateSample/Program.cs: -------------------------------------------------------------------------------- 1 | //------------------------------------------------------------ 2 | // Copyright (c) Microsoft Corporation. All rights reserved. 3 | //------------------------------------------------------------ 4 | 5 | namespace BulkUpdateSample 6 | { 7 | using System; 8 | using System.Configuration; 9 | using System.Collections.Generic; 10 | using System.Diagnostics; 11 | using System.Threading; 12 | using System.Threading.Tasks; 13 | 14 | using Microsoft.Azure.Documents; 15 | using Microsoft.Azure.Documents.Client; 16 | using Microsoft.Azure.CosmosDB.BulkExecutor; 17 | using Microsoft.Azure.CosmosDB.BulkExecutor.BulkImport; 18 | using Microsoft.Azure.CosmosDB.BulkExecutor.BulkUpdate; 19 | 20 | class Program 21 | { 22 | private static readonly string EndpointUrl = ConfigurationManager.AppSettings["EndPointUrl"]; 23 | private static readonly string AuthorizationKey = ConfigurationManager.AppSettings["AuthorizationKey"]; 24 | private static readonly string DatabaseName = ConfigurationManager.AppSettings["DatabaseName"]; 25 | private static readonly string CollectionName = ConfigurationManager.AppSettings["CollectionName"]; 26 | private static readonly int CollectionThroughput = int.Parse(ConfigurationManager.AppSettings["CollectionThroughput"]); 27 | 28 | private static readonly ConnectionPolicy ConnectionPolicy = new ConnectionPolicy 29 | { 30 | ConnectionMode = ConnectionMode.Direct, 31 | ConnectionProtocol = Protocol.Tcp 32 | }; 33 | 34 | private DocumentClient client; 35 | 36 | /// 37 | /// Initializes a new instance of the class. 38 | /// 39 | /// The DocumentDB client instance. 40 | private Program(DocumentClient client) 41 | { 42 | this.client = client; 43 | } 44 | 45 | static void Main(string[] args) 46 | { 47 | Trace.WriteLine("Summary:"); 48 | Trace.WriteLine("--------------------------------------------------------------------- "); 49 | Trace.WriteLine(String.Format("Endpoint: {0}", EndpointUrl)); 50 | Trace.WriteLine(String.Format("Collection : {0}.{1}", DatabaseName, CollectionName)); 51 | Trace.WriteLine("--------------------------------------------------------------------- "); 52 | Trace.WriteLine(""); 53 | 54 | try 55 | { 56 | using (var client = new DocumentClient( 57 | new Uri(EndpointUrl), 58 | AuthorizationKey, 59 | ConnectionPolicy)) 60 | { 61 | var program = new Program(client); 62 | program.RunBulkImportAndUpdateAsync().Wait(); 63 | } 64 | } 65 | catch (AggregateException e) 66 | { 67 | Trace.TraceError("Caught AggregateException in Main, Inner Exception:\n" + e); 68 | Console.ReadKey(); 69 | } 70 | } 71 | 72 | /// 73 | /// Driver function for bulk import. 74 | /// 75 | /// 76 | private async Task RunBulkImportAndUpdateAsync() 77 | { 78 | // Cleanup on start if set in config. 79 | 80 | DocumentCollection dataCollection = null; 81 | try 82 | { 83 | if (bool.Parse(ConfigurationManager.AppSettings["ShouldCleanupOnStart"])) 84 | { 85 | Database database = Utils.GetDatabaseIfExists(client, DatabaseName); 86 | if (database != null) 87 | { 88 | await client.DeleteDatabaseAsync(database.SelfLink); 89 | } 90 | 91 | Trace.TraceInformation("Creating database {0}", DatabaseName); 92 | database = await client.CreateDatabaseAsync(new Database { Id = DatabaseName }); 93 | 94 | Trace.TraceInformation(String.Format("Creating collection {0} with {1} RU/s", CollectionName, CollectionThroughput)); 95 | dataCollection = await Utils.CreatePartitionedCollectionAsync(client, DatabaseName, CollectionName, CollectionThroughput); 96 | } 97 | else 98 | { 99 | dataCollection = Utils.GetCollectionIfExists(client, DatabaseName, CollectionName); 100 | if (dataCollection == null) 101 | { 102 | throw new Exception("The data collection does not exist"); 103 | } 104 | } 105 | } 106 | catch (Exception de) 107 | { 108 | Trace.TraceError("Unable to initialize, exception message: {0}", de.Message); 109 | throw; 110 | } 111 | 112 | // Prepare for bulk import. 113 | 114 | // Creating documents with simple partition key here. 115 | string partitionKeyProperty = dataCollection.PartitionKey.Paths[0].Replace("/", ""); 116 | 117 | long numberOfDocumentsToGenerate = long.Parse(ConfigurationManager.AppSettings["NumberOfDocumentsToUpdate"]); 118 | int numberOfBatches = int.Parse(ConfigurationManager.AppSettings["NumberOfBatches"]); 119 | long numberOfDocumentsPerBatch = (long)Math.Floor(((double)numberOfDocumentsToGenerate) / numberOfBatches); 120 | 121 | // Set retry options high for initialization (default values). 122 | client.ConnectionPolicy.RetryOptions.MaxRetryWaitTimeInSeconds = 30; 123 | client.ConnectionPolicy.RetryOptions.MaxRetryAttemptsOnThrottledRequests = 9; 124 | 125 | IBulkExecutor bulkExecutor = new BulkExecutor(client, dataCollection); 126 | await bulkExecutor.InitializeAsync(); 127 | 128 | // Set retries to 0 to pass control to bulk executor. 129 | client.ConnectionPolicy.RetryOptions.MaxRetryWaitTimeInSeconds = 0; 130 | client.ConnectionPolicy.RetryOptions.MaxRetryAttemptsOnThrottledRequests = 0; 131 | 132 | BulkImportResponse bulkImportResponse = null; 133 | long totalNumberOfDocumentsInserted = 0; 134 | double totalRequestUnitsConsumed = 0; 135 | double totalTimeTakenSec = 0; 136 | 137 | var tokenSource = new CancellationTokenSource(); 138 | var token = tokenSource.Token; 139 | 140 | for (int i = 0; i < numberOfBatches; i++) 141 | { 142 | // Generate JSON-serialized documents to import. 143 | 144 | List documentsToImportInBatch = new List(); 145 | long prefix = i * numberOfDocumentsPerBatch; 146 | 147 | Trace.TraceInformation(String.Format("Generating {0} documents to import for batch {1}", numberOfDocumentsPerBatch, i)); 148 | for (int j = 0; j < numberOfDocumentsPerBatch; j++) 149 | { 150 | string partitionKeyValue = (prefix + j).ToString(); 151 | string id = partitionKeyValue; 152 | 153 | documentsToImportInBatch.Add(Utils.GenerateRandomDocumentString(id, partitionKeyProperty, partitionKeyValue)); 154 | } 155 | 156 | // Invoke bulk import API. 157 | 158 | var tasks = new List(); 159 | 160 | tasks.Add(Task.Run(async () => 161 | { 162 | Trace.TraceInformation(String.Format("Executing bulk import for batch {0}", i)); 163 | do 164 | { 165 | try 166 | { 167 | bulkImportResponse = await bulkExecutor.BulkImportAsync( 168 | documents: documentsToImportInBatch, 169 | enableUpsert: true, 170 | disableAutomaticIdGeneration: true, 171 | maxConcurrencyPerPartitionKeyRange: null, 172 | maxInMemorySortingBatchSize: null, 173 | cancellationToken: token); 174 | } 175 | catch (DocumentClientException de) 176 | { 177 | Trace.TraceError("Document client exception: {0}", de); 178 | break; 179 | } 180 | catch (Exception e) 181 | { 182 | Trace.TraceError("Exception: {0}", e); 183 | break; 184 | } 185 | } while (bulkImportResponse.NumberOfDocumentsImported < documentsToImportInBatch.Count); 186 | 187 | Trace.WriteLine(String.Format("\nSummary for batch {0}:", i)); 188 | Trace.WriteLine("--------------------------------------------------------------------- "); 189 | Trace.WriteLine(String.Format("Inserted {0} docs @ {1} writes/s, {2} RU/s in {3} sec", 190 | bulkImportResponse.NumberOfDocumentsImported, 191 | Math.Round(bulkImportResponse.NumberOfDocumentsImported / bulkImportResponse.TotalTimeTaken.TotalSeconds), 192 | Math.Round(bulkImportResponse.TotalRequestUnitsConsumed / bulkImportResponse.TotalTimeTaken.TotalSeconds), 193 | bulkImportResponse.TotalTimeTaken.TotalSeconds)); 194 | Trace.WriteLine(String.Format("Average RU consumption per document insert: {0}", 195 | (bulkImportResponse.TotalRequestUnitsConsumed / bulkImportResponse.NumberOfDocumentsImported))); 196 | Trace.WriteLine("---------------------------------------------------------------------\n "); 197 | 198 | totalNumberOfDocumentsInserted += bulkImportResponse.NumberOfDocumentsImported; 199 | totalRequestUnitsConsumed += bulkImportResponse.TotalRequestUnitsConsumed; 200 | totalTimeTakenSec += bulkImportResponse.TotalTimeTaken.TotalSeconds; 201 | }, 202 | token)); 203 | 204 | /* 205 | tasks.Add(Task.Run(() => 206 | { 207 | char ch = Console.ReadKey(true).KeyChar; 208 | if (ch == 'c' || ch == 'C') 209 | { 210 | tokenSource.Cancel(); 211 | Trace.WriteLine("\nTask cancellation requested."); 212 | } 213 | })); 214 | */ 215 | 216 | await Task.WhenAll(tasks); 217 | } 218 | 219 | Trace.WriteLine("Overall summary of bulk import:"); 220 | Trace.WriteLine("--------------------------------------------------------------------- "); 221 | Trace.WriteLine(String.Format("Inserted {0} docs @ {1} writes/s, {2} RU/s in {3} sec", 222 | totalNumberOfDocumentsInserted, 223 | Math.Round(totalNumberOfDocumentsInserted / totalTimeTakenSec), 224 | Math.Round(totalRequestUnitsConsumed / totalTimeTakenSec), 225 | totalTimeTakenSec)); 226 | Trace.WriteLine(String.Format("Average RU consumption per document insert: {0}", 227 | (totalRequestUnitsConsumed / totalNumberOfDocumentsInserted))); 228 | Trace.WriteLine("--------------------------------------------------------------------- \n"); 229 | 230 | //----------------------------------------------------------------------------------------------- 231 | 232 | // Prepare for bulk update. 233 | 234 | BulkUpdateResponse bulkUpdateResponse = null; 235 | long totalNumberOfDocumentsUpdated = 0; 236 | totalRequestUnitsConsumed = 0; 237 | totalTimeTakenSec = 0; 238 | 239 | tokenSource = new CancellationTokenSource(); 240 | token = tokenSource.Token; 241 | 242 | // Generate update operations. 243 | List updateOperations = new List(); 244 | // Set the name field. 245 | updateOperations.Add(new SetUpdateOperation("Name", "UpdatedDoc")); 246 | // Unset the description field. 247 | updateOperations.Add(new UnsetUpdateOperation("description")); 248 | 249 | for (int i = 0; i < numberOfBatches; i++) 250 | { 251 | // Generate update items. 252 | 253 | List updateItemsInBatch = new List(); 254 | long prefix = i * numberOfDocumentsPerBatch; 255 | 256 | Trace.TraceInformation(String.Format("Generating {0} update items for batch {1}", numberOfDocumentsPerBatch, i)); 257 | for (int j = 0; j < numberOfDocumentsPerBatch; j++) 258 | { 259 | string partitionKeyValue = (prefix + j).ToString(); 260 | string id = partitionKeyValue; 261 | 262 | updateItemsInBatch.Add(new UpdateItem(id, partitionKeyValue, updateOperations)); 263 | } 264 | 265 | // Invoke bulk update API. 266 | 267 | var tasks = new List(); 268 | 269 | tasks.Add(Task.Run(async () => 270 | { 271 | Trace.TraceInformation(String.Format("Executing bulk update for batch {0}", i)); 272 | do 273 | { 274 | try 275 | { 276 | bulkUpdateResponse = await bulkExecutor.BulkUpdateAsync( 277 | updateItems: updateItemsInBatch, 278 | maxConcurrencyPerPartitionKeyRange: null, 279 | cancellationToken: token); 280 | } 281 | catch (DocumentClientException de) 282 | { 283 | Trace.TraceError("Document client exception: {0}", de); 284 | break; 285 | } 286 | catch (Exception e) 287 | { 288 | Trace.TraceError("Exception: {0}", e); 289 | break; 290 | } 291 | } while (bulkUpdateResponse.NumberOfDocumentsUpdated < updateItemsInBatch.Count); 292 | 293 | Trace.WriteLine(String.Format("\nSummary for batch {0}:", i)); 294 | Trace.WriteLine("--------------------------------------------------------------------- "); 295 | Trace.WriteLine(String.Format("Updated {0} docs @ {1} updates/s, {2} RU/s in {3} sec", 296 | bulkUpdateResponse.NumberOfDocumentsUpdated, 297 | Math.Round(bulkUpdateResponse.NumberOfDocumentsUpdated / bulkUpdateResponse.TotalTimeTaken.TotalSeconds), 298 | Math.Round(bulkUpdateResponse.TotalRequestUnitsConsumed / bulkUpdateResponse.TotalTimeTaken.TotalSeconds), 299 | bulkUpdateResponse.TotalTimeTaken.TotalSeconds)); 300 | Trace.WriteLine(String.Format("Average RU consumption per document update: {0}", 301 | (bulkUpdateResponse.TotalRequestUnitsConsumed / bulkUpdateResponse.NumberOfDocumentsUpdated))); 302 | Trace.WriteLine("---------------------------------------------------------------------\n "); 303 | 304 | totalNumberOfDocumentsUpdated += bulkUpdateResponse.NumberOfDocumentsUpdated; 305 | totalRequestUnitsConsumed += bulkUpdateResponse.TotalRequestUnitsConsumed; 306 | totalTimeTakenSec += bulkUpdateResponse.TotalTimeTaken.TotalSeconds; 307 | }, 308 | token)); 309 | 310 | /* 311 | tasks.Add(Task.Run(() => 312 | { 313 | char ch = Console.ReadKey(true).KeyChar; 314 | if (ch == 'c' || ch == 'C') 315 | { 316 | tokenSource.Cancel(); 317 | Trace.WriteLine("\nTask cancellation requested."); 318 | } 319 | })); 320 | */ 321 | 322 | await Task.WhenAll(tasks); 323 | } 324 | 325 | Trace.WriteLine("Overall summary of bulk update:"); 326 | Trace.WriteLine("--------------------------------------------------------------------- "); 327 | Trace.WriteLine(String.Format("Updated {0} docs @ {1} update/s, {2} RU/s in {3} sec", 328 | totalNumberOfDocumentsUpdated, 329 | Math.Round(totalNumberOfDocumentsUpdated / totalTimeTakenSec), 330 | Math.Round(totalRequestUnitsConsumed / totalTimeTakenSec), 331 | totalTimeTakenSec)); 332 | Trace.WriteLine(String.Format("Average RU consumption per document update: {0}", 333 | (totalRequestUnitsConsumed / totalNumberOfDocumentsUpdated))); 334 | Trace.WriteLine("--------------------------------------------------------------------- \n"); 335 | 336 | //----------------------------------------------------------------------------------------------- 337 | 338 | // Cleanup on finish if set in config. 339 | 340 | if (bool.Parse(ConfigurationManager.AppSettings["ShouldCleanupOnFinish"])) 341 | { 342 | Trace.TraceInformation("Deleting Database {0}", DatabaseName); 343 | await client.DeleteDatabaseAsync(UriFactory.CreateDatabaseUri(DatabaseName)); 344 | } 345 | 346 | Trace.WriteLine("\nPress any key to exit."); 347 | Console.ReadKey(); 348 | } 349 | } 350 | } 351 | -------------------------------------------------------------------------------- /BulkUpdateSample/BulkUpdateSample/Properties/AssemblyInfo.cs: -------------------------------------------------------------------------------- 1 | using System.Reflection; 2 | using System.Runtime.CompilerServices; 3 | using System.Runtime.InteropServices; 4 | 5 | // General Information about an assembly is controlled through the following 6 | // set of attributes. Change these attribute values to modify the information 7 | // associated with an assembly. 8 | [assembly: AssemblyTitle("BulkUpdateSample")] 9 | [assembly: AssemblyDescription("")] 10 | [assembly: AssemblyConfiguration("")] 11 | [assembly: AssemblyCompany("")] 12 | [assembly: AssemblyProduct("BulkUpdateSample")] 13 | [assembly: AssemblyCopyright("Copyright © 2018")] 14 | [assembly: AssemblyTrademark("")] 15 | [assembly: AssemblyCulture("")] 16 | 17 | // Setting ComVisible to false makes the types in this assembly not visible 18 | // to COM components. If you need to access a type in this assembly from 19 | // COM, set the ComVisible attribute to true on that type. 20 | [assembly: ComVisible(false)] 21 | 22 | // The following GUID is for the ID of the typelib if this project is exposed to COM 23 | [assembly: Guid("9871a21e-3f29-4827-a44c-3f76169d8952")] 24 | 25 | // Version information for an assembly consists of the following four values: 26 | // 27 | // Major Version 28 | // Minor Version 29 | // Build Number 30 | // Revision 31 | // 32 | // You can specify all the values or you can default the Build and Revision Numbers 33 | // by using the '*' as shown below: 34 | // [assembly: AssemblyVersion("1.0.*")] 35 | [assembly: AssemblyVersion("1.0.0.0")] 36 | [assembly: AssemblyFileVersion("1.0.0.0")] 37 | -------------------------------------------------------------------------------- /BulkUpdateSample/BulkUpdateSample/Utils.cs: -------------------------------------------------------------------------------- 1 | //------------------------------------------------------------ 2 | // Copyright (c) Microsoft Corporation. All rights reserved. 3 | //------------------------------------------------------------ 4 | 5 | namespace BulkUpdateSample 6 | { 7 | using Microsoft.Azure.Documents; 8 | using Microsoft.Azure.Documents.Client; 9 | 10 | using System; 11 | using System.Collections.ObjectModel; 12 | using System.Configuration; 13 | using System.Linq; 14 | using System.Threading.Tasks; 15 | 16 | class Utils 17 | { 18 | /// 19 | /// Get the collection if it exists, null if it doesn't. 20 | /// 21 | /// The requested collection. 22 | static internal DocumentCollection GetCollectionIfExists(DocumentClient client, string databaseName, string collectionName) 23 | { 24 | if (GetDatabaseIfExists(client, databaseName) == null) 25 | { 26 | return null; 27 | } 28 | 29 | return client.CreateDocumentCollectionQuery(UriFactory.CreateDatabaseUri(databaseName)) 30 | .Where(c => c.Id == collectionName).AsEnumerable().FirstOrDefault(); 31 | } 32 | 33 | /// 34 | /// Get the database if it exists, null if it doesn't. 35 | /// 36 | /// The requested database. 37 | static internal Database GetDatabaseIfExists(DocumentClient client, string databaseName) 38 | { 39 | return client.CreateDatabaseQuery().Where(d => d.Id == databaseName).AsEnumerable().FirstOrDefault(); 40 | } 41 | 42 | /// 43 | /// Create a partitioned collection. 44 | /// 45 | /// The created collection. 46 | static internal async Task CreatePartitionedCollectionAsync(DocumentClient client, string databaseName, 47 | string collectionName, int collectionThroughput) 48 | { 49 | PartitionKeyDefinition partitionKey = new PartitionKeyDefinition 50 | { 51 | Paths = new Collection { ConfigurationManager.AppSettings["CollectionPartitionKey"] } 52 | }; 53 | DocumentCollection collection = new DocumentCollection { Id = collectionName, PartitionKey = partitionKey }; 54 | 55 | try 56 | { 57 | collection = await client.CreateDocumentCollectionAsync( 58 | UriFactory.CreateDatabaseUri(databaseName), 59 | collection, 60 | new RequestOptions { OfferThroughput = collectionThroughput }); 61 | } 62 | catch (Exception e) 63 | { 64 | throw e; 65 | } 66 | 67 | return collection; 68 | } 69 | 70 | static internal String GenerateRandomDocumentString(String id, String partitionKeyProperty, object parititonKeyValue) 71 | { 72 | return "{\n" + 73 | " \"id\": \"" + id + "\",\n" + 74 | " \"" + partitionKeyProperty + "\": \"" + parititonKeyValue + "\",\n" + 75 | " \"Name\": \"TestDoc\",\n" + 76 | " \"description\": \"1.99\",\n" + 77 | " \"f1\": \"3hrkjh3h4h4h3jk4h\",\n" + 78 | " \"f2\": \"dhfkjdhfhj4434434\",\n" + 79 | " \"f3\": \"nklfjeoirje434344\",\n" + 80 | " \"f4\": \"pjfgdgfhdgfgdhbd6\",\n" + 81 | " \"f5\": \"gyuerehvrerebrhjh\",\n" + 82 | " \"f6\": \"3434343ghghghgghj\"" + 83 | "}"; 84 | } 85 | } 86 | } 87 | -------------------------------------------------------------------------------- /BulkUpdateSample/BulkUpdateSample/packages.config: -------------------------------------------------------------------------------- 1 |  2 | 3 | 4 | 5 | 6 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) Microsoft Corporation. All rights reserved. 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a 6 | copy of this software and associated documentation files (the 7 | "Software"), to deal in the Software without restriction, including 8 | without limitation the rights to use, copy, modify, merge, publish, 9 | distribute, sublicense, and/or sell copies of the Software, and to 10 | permit persons to whom the Software is furnished to do so, subject to 11 | the following conditions: 12 | 13 | The above copyright notice and this permission notice shall be included 14 | in all copies or substantial portions of the Software. 15 | 16 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS 17 | OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 18 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND 19 | NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE 20 | LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 21 | OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION 22 | WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 |   Azure Cosmos DB BulkExecutor library for .NET 2 | ========================================== 3 | 4 | The Azure Cosmos DB BulkExecutor library for .NET acts as an extension library to the [Cosmos DB .NET SDK](https://docs.microsoft.com/en-us/azure/cosmos-db/sql-api-sdk-dotnet) and provides developers out-of-the-box functionality to perform bulk operations in [Azure Cosmos DB](http://cosmosdb.com). 5 | 6 |
7 | Table of Contents 8 | 9 | - [Consuming the Microsoft Azure Cosmos DB BulkExecutor .NET library](#nuget) 10 | - [Bulk Import API](#bulk-import-api) 11 | - [Configurable parameters](#bulk-import-configurations) 12 | - [Bulk import response object definition](#bulk-import-response) 13 | - [Getting started with bulk import](#bulk-import-getting-started) 14 | - [Performance of bulk import sample](bulk-import-performance) 15 | - [API implementation details](bulk-import-client-side) 16 | - [Bulk Update API](#bulk-update-api) 17 | - [List of supported field update operations](#field-update-operations) 18 | - [Configurable parameters](#bulk-update-configurations) 19 | - [Bulk update response object definition](#bulk-update-response) 20 | - [Getting started with bulk update](#bulk-update-getting-started) 21 | - [Performance of bulk update sample](bulk-update-performance) 22 | - [API implementation details](bulk-update-client-side) 23 | - [Performance tips](#additional-pointers) 24 | - [Contributing & Feedback](#contributing--feedback) 25 | - [Other relevant projects](#relevant-projects) 26 | 27 |
28 | 29 | ## Consuming the Microsoft Azure Cosmos DB BulkExecutor .NET library 30 | 31 | This project includes samples, documentation and performance tips for consuming the BulkExecutor library. You can download the official public NuGet package from [here](https://www.nuget.org/packages/Microsoft.Azure.CosmosDB.BulkExecutor/). 32 | 33 | ## Bulk Import API 34 | 35 | We provide two overloads of the bulk import API - one which accepts a list of JSON-serialized documents and the other a list of deserialized POCO documents. 36 | 37 | * With list of JSON-serialized documents 38 | ```csharp 39 | Task BulkImportAsync( 40 | IEnumerable documents, 41 | bool enableUpsert = false, 42 | bool disableAutomaticIdGeneration = true, 43 | int? maxConcurrencyPerPartitionKeyRange = null, 44 | int? maxInMemorySortingBatchSize = null, 45 | CancellationToken cancellationToken = default(CancellationToken)); 46 | ``` 47 | 48 | * With list of deserialized POCO documents 49 | ```csharp 50 | Task BulkImportAsync( 51 | IEnumerable documents, 52 | bool enableUpsert = false, 53 | bool disableAutomaticIdGeneration = true, 54 | int? maxConcurrencyPerPartitionKeyRange = null, 55 | int? maxInMemorySortingBatchSize = null, 56 | CancellationToken cancellationToken = default(CancellationToken)); 57 | ``` 58 | 59 | ### Configurable parameters 60 | 61 | * *enableUpsert* : A flag to enable upsert of the documents if document with given id already exists - default value is false. 62 | * *disableAutomaticIdGeneration* : A flag to disable automatic generation of id if absent in the document - default value is true. 63 | * *maxConcurrencyPerPartitionKeyRange* : The maximum degree of concurrency per partition key range, setting to null will cause library to use default value of 20. 64 | * *maxInMemorySortingBatchSize* : The maximum number of documents pulled from the document enumerator passed to the API call in each stage for in-memory pre-processing sorting phase prior to bulk importing, setting to null will cause library to use default value of min(documents.count, 1000000). 65 | * *cancellationToken* : The cancellation token to gracefully exit bulk import. 66 | 67 | ### Bulk import response object definition 68 | 69 | The result of the bulk import API call contains the following attributes: 70 | * *NumberOfDocumentsImported* (long) : The total number of documents which were successfully imported out of the documents supplied to the bulk import API call. 71 | * *TotalRequestUnitsConsumed* (double) : The total request units (RU) consumed by the bulk import API call. 72 | * *TotalTimeTaken* (TimeSpan) : The total time taken by the bulk import API call to complete execution. 73 | * *BadInputDocuments* (List\) : The list of bad-format documents which were not successfully imported in the bulk import API call. User needs to fix the documents returned and retry import. Bad-format documents include documents whose *id* value is not a string (null or any other datatype is considered invalid). 74 | 75 | ### Getting started with bulk import 76 | 77 | * Initialize DocumentClient set to Direct TCP connection mode 78 | ```csharp 79 | ConnectionPolicy connectionPolicy = new ConnectionPolicy 80 | { 81 | ConnectionMode = ConnectionMode.Direct, 82 | ConnectionProtocol = Protocol.Tcp 83 | }; 84 | DocumentClient client = new DocumentClient( 85 | new Uri(endpointUrl), 86 | authorizationKey, 87 | connectionPolicy) 88 | ``` 89 | 90 | * Initialize BulkExecutor with high retry option values for the client SDK and then set to 0 to pass congestion control to BulkExector for its lifetime 91 | ```csharp 92 | // Set retry options high during initialization (default values). 93 | client.ConnectionPolicy.RetryOptions.MaxRetryWaitTimeInSeconds = 30; 94 | client.ConnectionPolicy.RetryOptions.MaxRetryAttemptsOnThrottledRequests = 9; 95 | 96 | IBulkExecutor bulkExecutor = new BulkExecutor(client, dataCollection); 97 | await bulkExecutor.InitializeAsync(); 98 | 99 | // Set retries to 0 to pass complete control to bulk executor. 100 | client.ConnectionPolicy.RetryOptions.MaxRetryWaitTimeInSeconds = 0; 101 | client.ConnectionPolicy.RetryOptions.MaxRetryAttemptsOnThrottledRequests = 0; 102 | ``` 103 | 104 | * Call BulkImportAsync API 105 | ```csharp 106 | BulkImportResponse bulkImportResponse = await bulkExecutor.BulkImportAsync( 107 | documents: documentsToImportInBatch, 108 | enableUpsert: true, 109 | disableAutomaticIdGeneration: true, 110 | maxConcurrencyPerPartitionKeyRange: null, 111 | maxInMemorySortingBatchSize: null, 112 | cancellationToken: token); 113 | ``` 114 | 115 | You can find the complete sample application program consuming the bulk import API [here](https://github.com/Azure/azure-cosmosdb-bulkexecutor-dotnet-getting-started/blob/master/BulkImportSample/BulkImportSample/Program.cs) - which generates random documents to be then bulk imported into an Azure Cosmos DB collection. You can configure the application settings in *appSettings* [here](https://github.com/Azure/azure-cosmosdb-bulkexecutor-dotnet-getting-started/blob/master/BulkImportSample/BulkImportSample/App.config). 116 | 117 | You can download the Microsoft.Azure.CosmosDB.BulkExecutor nuget package from [here](https://www.nuget.org/packages/Microsoft.Azure.CosmosDB.BulkExecutor/). 118 | 119 | ### Performance of bulk import sample 120 | 121 | Let us compare the performace of the bulk import sample [application](https://github.com/Azure/azure-cosmosdb-bulkexecutor-dotnet-getting-started/blob/master/BulkImportSample/BulkImportSample/) against a [multi-threaded application](https://github.com/Azure/azure-documentdb-dotnet/tree/master/samples/documentdb-benchmark) which utilizes point writes (CreateDocumentAsync API in DocumentClient) 122 | 123 | Both the applications are run on a standard DS16 v3 Azure VM in East US against a Cosmos DB collection in East US with 1 million RU/s allocated throughput. 124 | 125 | The bulk import sample is executed with *NumberOfDocumentsToImport* set to **25 million** and *NumberOfBatches* set to **25** (in *App.config*) and default parameters for the bulk import API. The multi-threaded point write application is set up with a *DegreeOfParallelism* set to 2000 (spawns 2000 concurrent tasks) which maxes out the VM's CPU. 126 | 127 | We observe the following performance for ingestion of 25 million (~1KB) documents into a 1 million RU/s Cosmos DB collection: 128 | 129 | | | Time taken (sec) | Writes/second | RU/s consumed | 130 | | --- | --- | --- | --- | 131 | | Bulk import API | 262 | 95528 | 494186 | 132 | | Multi-threaded point write | 2431 | 10280 | 72481 | 133 | 134 | As seen, we observe **>9x** improvement in the write throughput using the bulk import API while providing out-of-the-box efficient handling of throttling, timeouts and transient exceptions - allowing easier scale-out by adding additional *BulkExecutor* client instances on individual VMs to achieve even greater write throughputs. 135 | 136 | ### API implementation details 137 | 138 | When a bulk import API is triggered with a batch of documents, on the client-side, they are first shuffled into buckets corresponding to their target Cosmos DB partition key range. Within each partiton key range bucket, they are broken down into mini-batches and each mini-batch of documents acts as a payload that is committed transactionally. 139 | 140 | We have built in optimizations for the concurrent execution of these mini-batches both within and across partition key ranges to maximally utilize the allocated collection throughput. We have designed an [AIMD-style congestion control](https://academic.microsoft.com/#/detail/2158700277?FORM=DACADP) mechanism for each Cosmos DB partition key range **to efficiently handle throttling and timeouts**. 141 | 142 | These client-side optimizations augment server-side features specific to the BulkExecutor library which together make maximal consumption of available throughput possible. 143 | 144 | ------------------------------------------ 145 | 146 | ## Bulk Update API 147 | 148 | The bulk update (a.k.a patch) API accepts a list of update items - each update item specifies the list of field update operations to be performed on a document identified by an id and parititon key value. 149 | 150 | ```csharp 151 | Task BulkUpdateAsync( 152 | IEnumerable updateItems, 153 | int? maxConcurrencyPerPartitionKeyRange = null, 154 | int? maxInMemorySortingBatchSize = null, 155 | CancellationToken cancellationToken = default(CancellationToken)); 156 | ``` 157 | 158 | * Definition of UpdateItem 159 | ```csharp 160 | class UpdateItem 161 | { 162 | public string Id { get; private set; } 163 | 164 | public string PartitionKey { get; private set; } 165 | 166 | public IEnumerable UpdateOperations { get; private set; } 167 | 168 | public UpdateItem( 169 | string id, 170 | string partitionKey, 171 | IEnumerable updateOperations) 172 | { 173 | this.Id = id; 174 | this.PartitionKey = partitionKey; 175 | this.UpdateOperations = updateOperations; 176 | } 177 | } 178 | ``` 179 | 180 | ### List of supported field update operations 181 | 182 | * Increment 183 | 184 | Supports incrementing any numeric document field by a specific value 185 | ```csharp 186 | class IncUpdateOperation 187 | { 188 | IncUpdateOperation(string field, TValue value) 189 | } 190 | ``` 191 | 192 | * Set 193 | 194 | Supports setting any document field to a specific value 195 | ```csharp 196 | class SetUpdateOperation 197 | { 198 | SetUpdateOperation(string field, TValue value) 199 | } 200 | ``` 201 | 202 | * Unset 203 | 204 | Supports removing a specific document field along with all children fields 205 | ```csharp 206 | class UnsetUpdateOperation 207 | { 208 | SetUpdateOperation(string field) 209 | } 210 | ``` 211 | 212 | * Array push 213 | 214 | Supports appending an array of values to a document field which contains an array 215 | ```csharp 216 | class PushUpdateOperation 217 | { 218 | PushUpdateOperation(string field, object[] value) 219 | } 220 | ``` 221 | 222 | * Array remove 223 | 224 | Supports removing a specific value (if present) from a document field which contains an array 225 | ```csharp 226 | class RemoveUpdateOperation 227 | { 228 | RemoveUpdateOperation(string field, TValue value) 229 | } 230 | ``` 231 | 232 | **Note**: For nested fields, use '.' as the nesting separtor. For example, if you wish to set the '/address/city' field to 'Seattle', express as shown: 233 | ```csharp 234 | SetUpdateOperation nestedPropertySetUpdate = new SetUpdateOperation("address.city", "Seattle"); 235 | ``` 236 | 237 | ### Configurable parameters 238 | 239 | * *maxConcurrencyPerPartitionKeyRange* : The maximum degree of concurrency per partition key range, setting to null will cause library to use default value of 20. 240 | * *maxInMemorySortingBatchSize* : The maximum number of update items pulled from the update items enumerator passed to the API call in each stage for in-memory pre-processing sorting phase prior to bulk updating, setting to null will cause library to use default value of min(updateItems.count, 1000000). 241 | * *cancellationToken* : The cancellation token to gracefully exit bulk update. 242 | 243 | ### Bulk update response object definition 244 | 245 | The result of the bulk update API call contains the following attributes: 246 | * *NumberOfDocumentsUpdated* (long) : The total number of documents which were successfully updated out of the ones supplied to the bulk update API call. 247 | * *TotalRequestUnitsConsumed* (double) : The total request units (RU) consumed by the bulk update API call. 248 | * *TotalTimeTaken* (TimeSpan) : The total time taken by the bulk update API call to complete execution. 249 | 250 | ### Getting started with bulk update 251 | 252 | * Initialize DocumentClient set to Direct TCP connection mode 253 | ```csharp 254 | ConnectionPolicy connectionPolicy = new ConnectionPolicy 255 | { 256 | ConnectionMode = ConnectionMode.Direct, 257 | ConnectionProtocol = Protocol.Tcp 258 | }; 259 | DocumentClient client = new DocumentClient( 260 | new Uri(endpointUrl), 261 | authorizationKey, 262 | connectionPolicy) 263 | ``` 264 | 265 | * Initialize BulkExecutor with high retry option values for the client SDK and then set to 0 to pass congestion control to BulkExector for its lifetime 266 | ```csharp 267 | // Set retry options high during initialization (default values). 268 | client.ConnectionPolicy.RetryOptions.MaxRetryWaitTimeInSeconds = 30; 269 | client.ConnectionPolicy.RetryOptions.MaxRetryAttemptsOnThrottledRequests = 9; 270 | 271 | IBulkExecutor bulkExecutor = new BulkExecutor(client, dataCollection); 272 | await bulkExecutor.InitializeAsync(); 273 | 274 | // Set retries to 0 to pass complete control to bulk executor. 275 | client.ConnectionPolicy.RetryOptions.MaxRetryWaitTimeInSeconds = 0; 276 | client.ConnectionPolicy.RetryOptions.MaxRetryAttemptsOnThrottledRequests = 0; 277 | ``` 278 | 279 | * Define the update items along with corresponding field update operations 280 | ```csharp 281 | SetUpdateOperation nameUpdate = new SetUpdateOperation("Name", "UpdatedDoc"); 282 | UnsetUpdateOperation descriptionUpdate = new UnsetUpdateOperation("description"); 283 | 284 | List updateOperations = new List(); 285 | updateOperations.Add(nameUpdate); 286 | updateOperations.Add(descriptionUpdate); 287 | 288 | List updateItems = new List(); 289 | for (int i = 0; i < 10; i++) 290 | { 291 | updateItems.Add(new UpdateItem(i.ToString(), i.ToString(), updateOperations)); 292 | } 293 | ``` 294 | 295 | * Call BulkUpdateAsync API 296 | ```csharp 297 | BulkUpdateResponse bulkUpdateResponse = await bulkExecutor.BulkUpdateAsync( 298 | updateItems: updateItems, 299 | maxConcurrencyPerPartitionKeyRange: null, 300 | maxInMemorySortingBatchSize: null, 301 | cancellationToken: token); 302 | ``` 303 | 304 | You can find the complete sample application program consuming the bulk update API [here](https://github.com/Azure/azure-cosmosdb-bulkexecutor-dotnet-getting-started/blob/master/BulkUpdateSample/BulkUpdateSample/Program.cs). You can configure the application settings in *appSettings* [here](https://github.com/Azure/azure-cosmosdb-bulkexecutor-dotnet-getting-started/blob/master/BulkUpdateSample/BulkUpdateSample/App.config). 305 | 306 | In the sample application, we first bulk import documents and then bulk update all the imported documents to set the *Name* field to a new value and unset the *description* field in each document. 307 | 308 | You can download the Microsoft.Azure.CosmosDB.BulkExecutor nuget package from [here](https://www.nuget.org/packages/Microsoft.Azure.CosmosDB.BulkExecutor/). 309 | 310 | ### Performance of bulk update sample 311 | 312 | When the given sample application is run on a standard DS16 v3 Azure VM in East US against a Cosmos DB collection in East US with **1 million RU/s** allocated throughput - with *NumberOfDocumentsToUpdate* set to **25 million** and *NumberOfBatches* set to **25** (in *App.config*) and default parameters for the bulk update API (as well as bulk import API), we observe the following performance for bulk update: 313 | ```csharp 314 | Updated 25000000 docs @ 53796 update/s, 491681 RU/s in 464.7 sec 315 | ``` 316 | 317 | ### API implementation details 318 | 319 | The bulk update API is designed similar to bulk import - look at the implementation details of bulk import API for details. 320 | 321 | ------------------------------------------ 322 | 323 | ## Bulk Delete API 324 | 325 | The bulk delete API accepts a list of tuples to delete in bulk. 326 | 327 | ```csharp 328 | Task BulkDeleteAsync( 329 | List> pkIdTuplesToDelete, 330 | int? deleteBatchSize = null, 331 | CancellationToken cancellationToken = default(CancellationToken)); 332 | ``` 333 | 334 | ### Configurable parameters 335 | 336 | * *deleteBatchSize* : The maximum number delete items to execute transactionally, setting to null will cause library to use default value of 1000. 337 | * *cancellationToken* : The cancellation token to gracefully exit bulk delete. 338 | 339 | ### Bulk delete response object definition 340 | 341 | The result of the bulk delete API call contains the following attributes: 342 | * *NumberOfDocumentsDeleted* (long) : The total number of documents which were successfully deleted out of the ones supplied to the bulk delete API call. 343 | * *TotalRequestUnitsConsumed* (double) : The total request units (RU) consumed by the bulk delete API call. 344 | * *TotalTimeTaken* (TimeSpan) : The total time taken by the bulk delete API call to complete execution. 345 | 346 | 347 | ### Getting started with bulk delete 348 | 349 | * Initialize DocumentClient set to Direct TCP connection mode 350 | ```csharp 351 | ConnectionPolicy connectionPolicy = new ConnectionPolicy 352 | { 353 | ConnectionMode = ConnectionMode.Direct, 354 | ConnectionProtocol = Protocol.Tcp 355 | }; 356 | DocumentClient client = new DocumentClient( 357 | new Uri(endpointUrl), 358 | authorizationKey, 359 | connectionPolicy) 360 | ``` 361 | 362 | * Initialize BulkExecutor with high retry option values for the client SDK and then set to 0 to pass congestion control to BulkExector for its lifetime 363 | ```csharp 364 | // Set retry options high during initialization (default values). 365 | client.ConnectionPolicy.RetryOptions.MaxRetryWaitTimeInSeconds = 30; 366 | client.ConnectionPolicy.RetryOptions.MaxRetryAttemptsOnThrottledRequests = 9; 367 | 368 | BulkExecutor bulkExecutor = new BulkExecutor(client, dataCollection); 369 | await bulkExecutor.InitializeAsync(); 370 | 371 | // Set retries to 0 to pass complete control to bulk executor. 372 | client.ConnectionPolicy.RetryOptions.MaxRetryWaitTimeInSeconds = 0; 373 | client.ConnectionPolicy.RetryOptions.MaxRetryAttemptsOnThrottledRequests = 0; 374 | ``` 375 | 376 | * Define the list of tuples to delete 377 | ```csharp 378 | List> pkIdTuplesToDelete = new List>(); 379 | for(int i=0; i < NumberOfDocumentsToDelete; i++) 380 | { 381 | pkIdTuplesToDelete.Add(new Tuple(i.ToString(), i.ToString())); 382 | } 383 | ``` 384 | 385 | * Call BulkDeleteAsync API 386 | ```csharp 387 | BulkDeleteResponse bulkDeleteResponse = await bulkExecutor.BulkDeleteAsync(pkIdTuplesToDelete); 388 | ``` 389 | 390 | You can find the complete sample application program consuming the bulk delete API [here](https://github.com/Azure/azure-cosmosdb-bulkexecutor-dotnet-getting-started/blob/master/BulkDeleteSample/BulkDeleteSample/Program.cs). You can configure the application settings in *appSettings* [here](https://github.com/Azure/azure-cosmosdb-bulkexecutor-dotnet-getting-started/blob/master/BulkDeleteSample/BulkDeleteSample/App.config). 391 | 392 | In the sample application, we first bulk import documents and then bulk delete a portion of the imported documents. 393 | 394 | You can download the Microsoft.Azure.CosmosDB.BulkExecutor nuget package from [here](https://www.nuget.org/packages/Microsoft.Azure.CosmosDB.BulkExecutor/). 395 | 396 | ------------------------------------------ 397 | 398 | ## Performance tips 399 | 400 | * For best performance, run your application **from an Azure VM in the same region as your Cosmos DB account write region**. 401 | * It is advised to instantiate a single *BulkExecutor* object for the entirety of the application within a single VM corresponding to a specific Cosmos DB collection. 402 | * Since a single bulk operation API execution consumes a large chunk of the client machine's CPU and network IO by spawning multiple tasks internally, avoid spawning multiple concurrent tasks within your application process each executing bulk operation API calls. If a single bulk operation API call running on a single VM is unable to consume your entire collection's throughput (if your collection's throughput > 1 million RU/s), preferably spin up separate VMs to concurrently execute bulk operation API calls. 403 | * Ensure *InitializeAsync()* is invoked after instantiating a *BulkExecutor* object to fetch the target Cosmos DB collection partition map. 404 | * In your application's *App.Config*, ensure **gcServer** is enabled for better performance 405 | ```csharp 406 | 407 | 408 | 409 | ``` 410 | * The library emits traces which can be collected either into a log file or on the console. To enable both, add the following to your application's *App.Config*. 411 | ```csharp 412 | 413 | 414 | 415 | 416 | 417 | 418 | 419 | 420 | ``` 421 | 422 | ------------------------------------------ 423 | ## Contributing & feedback 424 | 425 | This project has adopted the [Microsoft Open Source Code of 426 | Conduct](https://opensource.microsoft.com/codeofconduct/). For more information 427 | see the [Code of Conduct 428 | FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or contact 429 | [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional 430 | questions or comments. 431 | 432 | See [CONTRIBUTING.md](CONTRIBUTING.md) for contribution guidelines. 433 | 434 | To give feedback and/or report an issue, open a [GitHub 435 | Issue](https://help.github.com/articles/creating-an-issue/). 436 | 437 | ------------------------------------------ 438 | 439 | ## Other relevant projects 440 | 441 | * [Cosmos DB BulkExecutor library for Java](https://github.com/Azure/azure-cosmosdb-bulkexecutor-java-getting-started) --------------------------------------------------------------------------------