├── .gitattributes ├── .gitignore ├── CacheInitializer.csproj ├── ParamHandler.cs ├── Program.cs └── README.md /.gitattributes: -------------------------------------------------------------------------------- 1 | # Auto detect text files and perform LF normalization 2 | * text=auto 3 | 4 | # Custom for Visual Studio 5 | *.cs diff=csharp 6 | 7 | # Standard to msysgit 8 | *.doc diff=astextplain 9 | *.DOC diff=astextplain 10 | *.docx diff=astextplain 11 | *.DOCX diff=astextplain 12 | *.dot diff=astextplain 13 | *.DOT diff=astextplain 14 | *.pdf diff=astextplain 15 | *.PDF diff=astextplain 16 | *.rtf diff=astextplain 17 | *.RTF diff=astextplain 18 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | ## Ignore Visual Studio temporary files, build results, and 2 | ## files generated by popular Visual Studio add-ons. 3 | 4 | # User-specific files 5 | *.suo 6 | *.user 7 | *.userosscache 8 | *.sln.docstates 9 | *.DS_store 10 | 11 | # User-specific files (MonoDevelop/Xamarin Studio) 12 | *.userprefs 13 | 14 | # Build results 15 | [Dd]ebug/ 16 | [Dd]ebugPublic/ 17 | [Rr]elease/ 18 | [Rr]eleases/ 19 | [Xx]64/ 20 | [Xx]86/ 21 | [Bb]uild/ 22 | bld/ 23 | [Bb]in/ 24 | [Oo]bj/ 25 | 26 | # Visual Studio Code user directory 27 | .vscode/ 28 | 29 | # Visual Studio 2015 cache/options directory 30 | .vs/ 31 | # Uncomment if you have tasks that create the project's static files in wwwroot 32 | #wwwroot/ 33 | 34 | # MSTest test Results 35 | [Tt]est[Rr]esult*/ 36 | [Bb]uild[Ll]og.* 37 | 38 | # NUNIT 39 | *.VisualState.xml 40 | TestResult.xml 41 | 42 | # Build Results of an ATL Project 43 | [Dd]ebugPS/ 44 | [Rr]eleasePS/ 45 | dlldata.c 46 | 47 | # DNX 48 | project.lock.json 49 | artifacts/ 50 | 51 | *_i.c 52 | *_p.c 53 | *_i.h 54 | *.ilk 55 | *.meta 56 | *.obj 57 | *.pch 58 | *.pdb 59 | *.pgc 60 | *.pgd 61 | *.rsp 62 | *.sbr 63 | *.tlb 64 | *.tli 65 | *.tlh 66 | *.tmp 67 | *.tmp_proj 68 | *.log 69 | *.vspscc 70 | *.vssscc 71 | .builds 72 | *.pidb 73 | *.svclog 74 | *.scc 75 | 76 | # Chutzpah Test files 77 | _Chutzpah* 78 | 79 | # Visual C++ cache files 80 | ipch/ 81 | *.aps 82 | *.ncb 83 | *.opendb 84 | *.opensdf 85 | *.sdf 86 | *.cachefile 87 | *.VC.db 88 | 89 | # Visual Studio profiler 90 | *.psess 91 | *.vsp 92 | *.vspx 93 | *.sap 94 | 95 | # TFS 2012 Local Workspace 96 | $tf/ 97 | 98 | # Guidance Automation Toolkit 99 | *.gpState 100 | 101 | # ReSharper is a .NET coding add-in 102 | _ReSharper*/ 103 | *.[Rr]e[Ss]harper 104 | *.DotSettings.user 105 | 106 | # JustCode is a .NET coding add-in 107 | .JustCode 108 | 109 | # TeamCity is a build add-in 110 | _TeamCity* 111 | 112 | # DotCover is a Code Coverage Tool 113 | *.dotCover 114 | 115 | # NCrunch 116 | _NCrunch_* 117 | .*crunch*.local.xml 118 | nCrunchTemp_* 119 | 120 | # MightyMoose 121 | *.mm.* 122 | AutoTest.Net/ 123 | 124 | # Web workbench (sass) 125 | .sass-cache/ 126 | 127 | # Installshield output folder 128 | [Ee]xpress/ 129 | 130 | # DocProject is a documentation generator add-in 131 | DocProject/buildhelp/ 132 | DocProject/Help/*.HxT 133 | DocProject/Help/*.HxC 134 | DocProject/Help/*.hhc 135 | DocProject/Help/*.hhk 136 | DocProject/Help/*.hhp 137 | DocProject/Help/Html2 138 | DocProject/Help/html 139 | 140 | # Click-Once directory 141 | publish/ 142 | 143 | # Publish Web Output 144 | *.[Pp]ublish.xml 145 | *.azurePubxml 146 | 147 | # TODO: Un-comment the next line if you do not want to checkin 148 | # your web deploy settings because they may include unencrypted 149 | # passwords 150 | #*.pubxml 151 | *.publishproj 152 | 153 | # NuGet Packages 154 | *.nupkg 155 | # The packages folder can be ignored because of Package Restore 156 | **/packages/* 157 | # except build/, which is used as an MSBuild target. 158 | !**/packages/build/ 159 | # Uncomment if necessary however generally it will be regenerated when needed 160 | #!**/packages/repositories.config 161 | # NuGet v3's project.json files produces more ignoreable files 162 | *.nuget.props 163 | *.nuget.targets 164 | 165 | # Microsoft Azure Build Output 166 | csx/ 167 | *.build.csdef 168 | 169 | # Microsoft Azure Emulator 170 | ecf/ 171 | rcf/ 172 | 173 | # Microsoft Azure ApplicationInsights config file 174 | ApplicationInsights.config 175 | 176 | # Windows Store app package directory 177 | AppPackages/ 178 | BundleArtifacts/ 179 | 180 | # Visual Studio cache files 181 | # files ending in .cache can be ignored 182 | *.[Cc]ache 183 | # but keep track of directories ending in .cache 184 | !*.[Cc]ache/ 185 | 186 | # Others 187 | ClientBin/ 188 | [Ss]tyle[Cc]op.* 189 | ~$* 190 | *~ 191 | *.dbmdl 192 | *.dbproj.schemaview 193 | *.pfx 194 | *.publishsettings 195 | node_modules/ 196 | orleans.codegen.cs 197 | 198 | # RIA/Silverlight projects 199 | Generated_Code/ 200 | 201 | # Backup & report files from converting an old project file 202 | # to a newer Visual Studio version. Backup files are not needed, 203 | # because we have git ;-) 204 | _UpgradeReport_Files/ 205 | Backup*/ 206 | UpgradeLog*.XML 207 | UpgradeLog*.htm 208 | 209 | # SQL Server files 210 | *.mdf 211 | *.ldf 212 | 213 | # Business Intelligence projects 214 | *.rdl.data 215 | *.bim.layout 216 | *.bim_*.settings 217 | 218 | # Microsoft Fakes 219 | FakesAssemblies/ 220 | 221 | # GhostDoc plugin setting file 222 | *.GhostDoc.xml 223 | 224 | # Node.js Tools for Visual Studio 225 | .ntvs_analysis.dat 226 | 227 | # Visual Studio 6 build log 228 | *.plg 229 | 230 | # Visual Studio 6 workspace options file 231 | *.opt 232 | 233 | # Visual Studio LightSwitch build output 234 | **/*.HTMLClient/GeneratedArtifacts 235 | **/*.DesktopClient/GeneratedArtifacts 236 | **/*.DesktopClient/ModelManifest.xml 237 | **/*.Server/GeneratedArtifacts 238 | **/*.Server/ModelManifest.xml 239 | _Pvt_Extensions 240 | 241 | # LightSwitch generated files 242 | GeneratedArtifacts/ 243 | ModelManifest.xml 244 | 245 | # Paket dependency manager 246 | .paket/paket.exe 247 | 248 | # FAKE - F# Make 249 | .fake/ 250 | -------------------------------------------------------------------------------- /CacheInitializer.csproj: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | Exe 5 | netcoreapp3.1 6 | EA Americas 7 | This tool pre-loads Qlik Sense applications in a QSEoW environment. 8 | 1.0.0 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | -------------------------------------------------------------------------------- /ParamHandler.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections.Generic; 3 | using CommandLine; 4 | using CommandLine.Text; 5 | 6 | namespace CacheInitializer 7 | { 8 | // Define a class to receive parsed values 9 | class Options 10 | { 11 | [Option('s', "server", Required = true, HelpText = "URL to the server.")] 12 | public string Server { get; set; } 13 | 14 | [Option('a', "appname", Required = false, HelpText = "App to load (using app name)")] 15 | public string AppName { get; set; } 16 | 17 | [Option('i', "appid", Required = false, HelpText = "App to load (using app ID)")] 18 | public string AppID { get; set; } 19 | 20 | [Option('p', "proxy", Required = false, HelpText = "Virtual Proxy to use")] 21 | public string VirtualProxy { get; set; } 22 | 23 | [Option('o', "objects", Required = false, Default = false, HelpText = "cycle through all sheets and objects")] 24 | public bool FetchObjects { get; set; } 25 | 26 | [Option('f', "field", Required = false, HelpText = "field to make selections in e.g Region")] 27 | public string SelectionField { get; set; } 28 | 29 | [Option('v', "values", Required = false, HelpText = "values to select e.g \"France\",\"Germany\",\"Spain\"")] 30 | public string SelectionValues { get; set; } 31 | 32 | [Option('d', "debug", Required = false, HelpText = "Run with logging set to debug.")] 33 | public bool Debug { get; set; } 34 | 35 | static void DisplayHelp(ParserResult result, IEnumerable errs) 36 | { 37 | var helpText = HelpText.AutoBuild(result, (current) => HelpText.DefaultParsingErrorsHandler(result, current)); 38 | Console.WriteLine(helpText); 39 | } 40 | } 41 | } 42 | -------------------------------------------------------------------------------- /Program.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections.Generic; 3 | using System.Linq; 4 | using System.Net.WebSockets; 5 | using System.Threading.Tasks; 6 | using Qlik.Engine; 7 | using Qlik.Engine.Communication; 8 | using Qlik.Sense.Client; 9 | using CommandLine; 10 | 11 | // Title: Qlik Sense Cache Initializer 12 | 13 | // Summary: This tool will "warm" the cache of a Qlik Sense server so that when using large apps the users get good performance right away. 14 | // You can use it to load all apps, a single app, and you can get it to just open the app to RAM or cycle through all the objects 15 | // so that it will pre calculate expressions so users get rapid performance. You can also pass in selections too. 16 | // Credits: Thanks to Øystein Kolsrud for helping with the Qlik Sense .net SDK steps, contributions by Roland Vecera and Goran Sander 17 | // Uses the commandline.codeplex.com for processing parameters 18 | 19 | 20 | // Usage: cacheinitiazer.exe -s https://server.domain.com [-a appname] [-i appid] [-o] [-f fieldname] [-v "value 1,value 2"] [-p virtualproxyprefix] 21 | // Notes: This projects use the Qlik Sense .net SDK, you must use the right version of the SDK to match the server you are connecting too. 22 | // To swap version simply replace the .net SDK files in the BIN directory of this project, if you dont match them, it wont work! 23 | 24 | 25 | namespace CacheInitializer 26 | { 27 | internal enum LogLevel 28 | { 29 | Info, 30 | Debug 31 | } 32 | 33 | internal class QlikSelection 34 | { 35 | public string fieldname { get; set; } 36 | public string[] fieldvalues { get; set; } 37 | } 38 | 39 | class Program 40 | { 41 | private static bool DEBUG_MODE = false; 42 | 43 | static void Main(string[] args) 44 | { 45 | //// process the parameters using the https://github.com/commandlineparser/commandline/wiki/Getting-Started 46 | Parser.Default.ParseArguments(args) 47 | .WithParsed(options => DoWork(options)) // options is an instance of Options type 48 | .WithNotParsed(errors => 49 | { 50 | foreach (Error anError in errors) 51 | { 52 | if (anError.Tag == ErrorType.MissingRequiredOptionError) 53 | { 54 | //Console.WriteLine("Missing required argument '--" + ((MissingRequiredOptionError)anError).NameInfo.LongName + "'."); 55 | } 56 | } 57 | }); 58 | 59 | return; 60 | } 61 | 62 | private static void DoWork(Options options) 63 | { 64 | Uri serverURL; 65 | string appname; 66 | string appid; 67 | bool openSheets; 68 | string virtualProxy; 69 | QlikSelection mySelection = null; 70 | 71 | ILocation remoteQlikSenseLocation = null; 72 | 73 | try 74 | { 75 | if (options.Debug) 76 | { 77 | DEBUG_MODE = true; 78 | Print(LogLevel.Debug, "Debug logging enabled."); 79 | 80 | } 81 | Print(LogLevel.Debug, "setting parameter values in main"); 82 | serverURL = new Uri(options.Server); 83 | appname = options.AppName; 84 | appid = options.AppID; 85 | virtualProxy = !string.IsNullOrEmpty(options.VirtualProxy) ? options.VirtualProxy : ""; 86 | openSheets = options.FetchObjects; 87 | if (options.SelectionField != null) 88 | { 89 | mySelection = new QlikSelection(); 90 | mySelection.fieldname = options.SelectionField; 91 | mySelection.fieldvalues = options.SelectionValues.Split(','); 92 | } 93 | //TODO need to validate the params ideally 94 | 95 | Print(LogLevel.Debug, "setting remoteQlikSenseLocation"); ; 96 | 97 | ////connect to the server (using windows credentials 98 | QlikConnection.Timeout = Int32.MaxValue; 99 | var d = DateTime.Now; 100 | 101 | remoteQlikSenseLocation = Qlik.Engine.Location.FromUri(serverURL); 102 | 103 | 104 | Print(LogLevel.Debug, "validating http(s) and virtual proxy"); ; 105 | if (virtualProxy.Length > 0) 106 | { 107 | remoteQlikSenseLocation.VirtualProxyPath = virtualProxy; 108 | } 109 | bool isHTTPs = false; 110 | if (serverURL.Scheme == Uri.UriSchemeHttps) 111 | { 112 | isHTTPs = true; 113 | } 114 | remoteQlikSenseLocation.AsNtlmUserViaProxy(isHTTPs, null, false); 115 | 116 | Print(LogLevel.Debug, "starting to cache applications"); 117 | ////Start to cache the apps 118 | IAppIdentifier appIdentifier = null; 119 | 120 | if (appid != null) 121 | { 122 | //Open up and cache one app, based on app ID 123 | appIdentifier = remoteQlikSenseLocation.AppWithId(appid); 124 | Print(LogLevel.Debug, "got app identifier by ID"); 125 | LoadCache(remoteQlikSenseLocation, appIdentifier, openSheets, mySelection); 126 | Print(LogLevel.Debug, "finished caching by ID"); 127 | 128 | } 129 | else 130 | { 131 | if (appname != null) 132 | { 133 | //Open up and cache one app 134 | appIdentifier = remoteQlikSenseLocation.AppWithNameOrDefault(appname); 135 | Print(LogLevel.Debug, "got app identifier by name"); 136 | LoadCache(remoteQlikSenseLocation, appIdentifier, openSheets, mySelection); 137 | Print(LogLevel.Debug, "finished caching by name"); 138 | } 139 | else 140 | { 141 | //Get all apps, open them up and cache them 142 | remoteQlikSenseLocation.GetAppIdentifiers().ToList().ForEach(id => LoadCache(remoteQlikSenseLocation, id, openSheets, null)); 143 | Print(LogLevel.Debug, "finished caching all applications"); 144 | } 145 | } 146 | 147 | 148 | ////Wrap it up 149 | var dt = DateTime.Now - d; 150 | Print(LogLevel.Info, "Cache initialization complete. Total time: {0}", dt.ToString()); 151 | remoteQlikSenseLocation.Dispose(); 152 | Print(LogLevel.Debug, "done"); 153 | 154 | return; 155 | } 156 | catch (UriFormatException) 157 | { 158 | Print(LogLevel.Info, "Invalid server paramater format. Format must be http[s]://host.domain.tld."); 159 | return; 160 | } 161 | catch (WebSocketException webEx) 162 | { 163 | if (remoteQlikSenseLocation != null) 164 | { 165 | Print(LogLevel.Info, "Disposing remoteQlikSenseLocation"); 166 | remoteQlikSenseLocation.Dispose(); 167 | } 168 | 169 | Print(LogLevel.Info, "Unable to connect to establish WebSocket connection with: " + options.Server); 170 | Print(LogLevel.Info, "Error: " + webEx.Message); 171 | 172 | return; 173 | } 174 | catch (TimeoutException timeoutEx) 175 | { 176 | Print(LogLevel.Info, "Timeout Exception - Unable to connect to: " + options.Server); 177 | Print(LogLevel.Info, "Error: " + timeoutEx.Message); 178 | 179 | return; 180 | } 181 | catch (Exception ex) 182 | { 183 | if (ex.Message.Trim() == "Websocket closed unexpectedly (EndpointUnavailable):") 184 | { 185 | Print(LogLevel.Info, "Error: licenses exhausted."); 186 | return; 187 | } 188 | else 189 | { 190 | Print(LogLevel.Info, "Unexpected error."); 191 | Print(LogLevel.Info, "Message: " + ex.Message); 192 | 193 | return; 194 | } 195 | } 196 | } 197 | 198 | static void LoadCache(ILocation location, IAppIdentifier id, bool opensheets, QlikSelection Selections) 199 | { 200 | IApp app = null; 201 | try 202 | { 203 | //open up the app 204 | Print(LogLevel.Info, "{0}: Opening app", id.AppName); 205 | app = location.App(id); 206 | Print(LogLevel.Info, "{0}: App open", id.AppName); 207 | 208 | //see if we are going to open the sheets too 209 | if (opensheets) 210 | { 211 | //see of we are going to make some selections too 212 | if (Selections != null) 213 | { 214 | for (int i = 0; i < Selections.fieldvalues.Length; i++) 215 | { 216 | //clear any existing selections 217 | Print(LogLevel.Info, "{0}: Clearing Selections", id.AppName); 218 | app.ClearAll(true); 219 | //apply the new selections 220 | Print(LogLevel.Info, "{0}: Applying Selection: {1} = {2}", id.AppName, Selections.fieldname, Selections.fieldvalues[i]); 221 | app.GetField(Selections.fieldname).Select(Selections.fieldvalues[i]); 222 | //cache the results 223 | cacheObjects(app, location, id); 224 | } 225 | 226 | } 227 | else 228 | { 229 | //clear any selections 230 | Print(LogLevel.Info, "{0}: Clearing Selections", id.AppName); 231 | app.ClearAll(true); 232 | //cache the results 233 | cacheObjects(app, location, id); 234 | } 235 | } 236 | 237 | Print(LogLevel.Info, "{0}: App cache completed", id.AppName); 238 | app.Dispose(); 239 | } 240 | catch (Exception ex) 241 | { 242 | if (app != null) 243 | { 244 | app.Dispose(); 245 | } 246 | throw ex; 247 | } 248 | } 249 | 250 | static void cacheObjects(IApp app, ILocation location, IAppIdentifier id) 251 | { 252 | //get a list of the sheets in the app 253 | Print(LogLevel.Info, "{0}: Getting sheets", id.AppName); 254 | var sheets = app.GetSheets().ToArray(); 255 | //get a list of the objects in the app 256 | Print(LogLevel.Info, "{0}: Number of sheets - {1}, getting children", id.AppName, sheets.Count()); 257 | IGenericObject[] allObjects = sheets.Concat(sheets.SelectMany(sheet => GetAllChildren(app, sheet))).ToArray(); 258 | //draw the layout of all objects so the server calculates the data for them 259 | Print(LogLevel.Info, "{0}: Number of objects - {1}, caching all objects", id.AppName, allObjects.Count()); 260 | var allLayoutTasks = allObjects.Select(o => o.GetLayoutAsync()).ToArray(); 261 | Task.WaitAll(allLayoutTasks); 262 | Print(LogLevel.Info, "{0}: Objects cached", id.AppName); 263 | } 264 | 265 | private static IEnumerable GetAllChildren(IApp app, IGenericObject obj) 266 | { 267 | IEnumerable children = obj.GetChildInfos().Select(o => app.GetObject(o.Id)).ToArray(); 268 | return children.Concat(children.SelectMany(child => GetAllChildren(app, child))); 269 | } 270 | 271 | private static void Print(LogLevel level, string txt) 272 | { 273 | if (level == LogLevel.Info) 274 | { 275 | Console.WriteLine("{0} - {1}", DateTime.Now.ToString("hh:mm:ss"), txt); 276 | } 277 | else if (level == LogLevel.Debug && !DEBUG_MODE) 278 | { 279 | return; 280 | } 281 | else if (level == LogLevel.Debug && DEBUG_MODE) 282 | { 283 | Console.WriteLine("DEBUG\t{0} - {1}", DateTime.Now.ToString("hh:mm:ss"), txt); 284 | } 285 | else 286 | { 287 | throw new ArgumentException("Invalid LogLevel specified."); 288 | } 289 | } 290 | 291 | private static void Print(LogLevel level, string txt, params object[] os) 292 | { 293 | Print(level, String.Format(txt, os)); 294 | } 295 | } 296 | } 297 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Status 2 | [![Project Status: Unsupported – The project has reached a stable, usable state but the author(s) have ceased all work on it. A new maintainer may be desired.](https://www.repostatus.org/badges/latest/unsupported.svg)](https://www.repostatus.org/#unsupported) 3 | 4 | 5 | # Qlik Sense Cache Initializer 6 | 7 | Refer to [this article](https://adminplaybook.qlik-poc.com/docs/tooling/cache_warming.html#cacheinitializer-) for more comprehensive usage and background to the use case for this tool. 8 | 9 | ### Summary 10 | This tool will "warm" the cache of a Qlik Sense server so that when using large apps, the users will experience shorter load times for their 'first' app opens and queries. You can use it to load all apps, a single app, or you can use it to open the app and cycle through all the objects so that it will pre-calculate expressions to increase user performance. The cache initialzer also supports the ability to pass in selections. 11 | 12 | ### Download/Release 13 | The project is now built in .NET Core, which means it can be run on any OS. That said, the download available currently under the releases section [here](https://github.com/eapowertools/CacheInitializer/releases), is a win64 executable (with all runtimes and dlls self contained). Since the executable contains all runtimes and dlls, you should be able to download and execute without any additional installation prerequisites. You can rebuild the project yourself if you'd like to run it on a Linux distribution or OS X by downloading the source files.. 14 | 15 | #### Credits 16 | First, thanks to Joe Bickley for building this tool. Thanks to Øystein Kolsrud for helping with the Qlik Sense .net SDK steps, contributions by Roland Vecera and Goran Sander 17 | 18 | #### Usage 19 | ``` 20 | cacheinitiazer.exe -s https://server.domain.com [-a appname] [-i appid] [-o] [-f fieldname] [-v "value 1,value 2"] [-p virtualproxyprefix] 21 | ``` 22 | 23 | #### Available Parameters: 24 | 25 | ``` 26 | -s, --server Required. URL to the server. 27 | -a, --appname App to load (using app name) 28 | -i, --appid App to load (using app ID) 29 | -p, --proxy Virtual Proxy to use 30 | -o, --objects (Default: False) cycle through all sheets and objects 31 | -f, --field field to make selections in e.g Region 32 | -v, --values values to select e.g "France","Germany","Spain" 33 | --help Display this help screen. 34 | ``` 35 | 36 | ##### Notes 37 | Also for those interested in similar capabilties but running in node.js check out Goran Sanders project (and many other goodies) here: https://github.com/ptarmiganlabs/butler-cw 38 | --------------------------------------------------------------------------------