├── .github └── workflows │ └── dotnetcore.yml ├── .gitignore ├── LICENSE ├── README.md ├── TensorflowTypeProvider ├── TensorflowTypeProvider.sln └── TensorflowTypeProvider │ ├── NPYReaderWriter.fs │ ├── Properties │ └── PublishProfiles │ │ └── FolderProfile.pubxml │ ├── ProvidedTypes.fs │ ├── ProvidedTypes.fsi │ ├── TensorflowTypeProvider.fs │ └── TensorflowTypeProvider.fsproj ├── TensorflowTypeProviderTest ├── TensorflowTypeProviderTest.sln ├── TensorflowTypeProviderTest │ ├── App.config │ ├── AssemblyInfo.fs │ ├── Program.fs │ ├── TensorflowTypeProviderTest.fsproj │ ├── Test.fsx │ └── packages.config └── lib │ ├── Google.Protobuf.dll │ ├── System.Reflection.dll │ ├── System.Runtime.dll │ ├── TensorFlow.NET.dll │ ├── TensorflowTypeProvider.dll │ └── netstandard.dll ├── TestData ├── FashionMNIST │ ├── FashionFrozen.pb │ ├── checkpoint │ ├── model.ckpt.data-00000-of-00001 │ ├── model.ckpt.index │ └── model.ckpt.meta ├── float32.npy ├── float32_16.npy ├── float32_8x2.npy ├── int32.npy ├── test_data.npz └── test_data.zip └── images ├── NPY.gif ├── NPZ.gif └── TF.gif /.github/workflows/dotnetcore.yml: -------------------------------------------------------------------------------- 1 | name: .NET Core 2 | 3 | on: [push] 4 | 5 | jobs: 6 | build: 7 | 8 | runs-on: ubuntu-latest 9 | 10 | steps: 11 | - uses: actions/checkout@v1 12 | - name: Setup .NET Core 13 | uses: actions/setup-dotnet@v1 14 | with: 15 | dotnet-version: 2.2.108 16 | - name: Build with dotnet 17 | run: dotnet build TensorflowTypeProvider --configuration Release 18 | 19 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | **/bin/* 2 | **/obj/* 3 | **/.vs/* 4 | **/packages/* 5 | *.fsproj.user 6 | */FolderProfile.pubxml 7 | *.pubxml.user 8 | *~ -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "[]" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright [yyyy] [name of copyright owner] 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # TensorflowTypeProvider 2 | 3 | This Type Provider replaces the need for ‘magic strings’ associated with accessing pre-trained TensorFlow Graphs. 4 | 5 | This demonstrates that the programming model can be separated from the execution and that design time shape information can be represented within the type system. 6 | 7 | This provides a more interactive and intuitive mechanism for exploring the TensorFlow Graphs than either the programmatic interrogation with Python or the nested graphical exploration with TensorBoard. 8 | 9 | There is also typed access to NPY/NPZ files. 10 | 11 | # Build 12 | 13 | `dotnet build TensorflowTypeProvider` 14 | 15 | 16 | ## TensorFlow Graph example 17 | 18 | This example demonstrates using the Type Provider to explore a FashionMNIST Graph. Even a very small graph like this has a large number of nodes. This provider enables completion on node names as well as grouping the nodes by their Operation type. Each node has attributes which can be explored via the AttrList property. This makes it easier to examine node properties such as padding and data_format on the Conv2D nodes. 19 | 20 | ![TensorFlowGraph](images/TF.gif) 21 | 22 | ## NPY example 23 | 24 | This shows that the rank and data type are provided from the NPY file. When we change the NPY file the type system updates to reflect this change. This ensures that the Item function always has the correct rank for indexing into the underlying array. 25 | 26 | ![NPY](images/NPY.gif) 27 | 28 | ## NPZ example 29 | 30 | NPY files are a zip archive of NPY files. In this example the NPY file names have a file system like hierarchy with the separator '/'. The first level directories refer to the datatype and the second level refers to the shape. The NPY file pulled from the final path has the corresponding datatype and shape. 31 | 32 | ![NPZ](images/NPZ.gif) 33 | 34 | 35 | -------------------------------------------------------------------------------- /TensorflowTypeProvider/TensorflowTypeProvider.sln: -------------------------------------------------------------------------------- 1 |  2 | Microsoft Visual Studio Solution File, Format Version 12.00 3 | # Visual Studio Version 16 4 | VisualStudioVersion = 16.0.29326.143 5 | MinimumVisualStudioVersion = 10.0.40219.1 6 | Project("{F2A71F9B-5D33-465A-A702-920D77279786}") = "TensorflowTypeProvider", "TensorflowTypeProvider\TensorflowTypeProvider.fsproj", "{BE830B83-96E9-4A54-8FC8-E1A4C7166F93}" 7 | EndProject 8 | Global 9 | GlobalSection(SolutionConfigurationPlatforms) = preSolution 10 | Debug|Any CPU = Debug|Any CPU 11 | Release|Any CPU = Release|Any CPU 12 | EndGlobalSection 13 | GlobalSection(ProjectConfigurationPlatforms) = postSolution 14 | {BE830B83-96E9-4A54-8FC8-E1A4C7166F93}.Debug|Any CPU.ActiveCfg = Debug|Any CPU 15 | {BE830B83-96E9-4A54-8FC8-E1A4C7166F93}.Debug|Any CPU.Build.0 = Debug|Any CPU 16 | {BE830B83-96E9-4A54-8FC8-E1A4C7166F93}.Release|Any CPU.ActiveCfg = Release|Any CPU 17 | {BE830B83-96E9-4A54-8FC8-E1A4C7166F93}.Release|Any CPU.Build.0 = Release|Any CPU 18 | EndGlobalSection 19 | GlobalSection(SolutionProperties) = preSolution 20 | HideSolutionNode = FALSE 21 | EndGlobalSection 22 | GlobalSection(ExtensibilityGlobals) = postSolution 23 | SolutionGuid = {808E8DB2-BFD0-4B76-BDE2-D40DE9528925} 24 | EndGlobalSection 25 | EndGlobal 26 | -------------------------------------------------------------------------------- /TensorflowTypeProvider/TensorflowTypeProvider/NPYReaderWriter.fs: -------------------------------------------------------------------------------- 1 | module NPYReaderWriter 2 | 3 | open System 4 | open System.IO 5 | open System.IO.Compression 6 | 7 | // WARN: This does not support nested record arrays and object arrays. For now this is by design. 8 | // TODO: Support Strings 9 | 10 | module Result = 11 | let requireOk (result:Result<'a,string>) = 12 | match result with 13 | | Error(msg) -> failwith msg 14 | | Ok(x) -> x 15 | 16 | [] 17 | type NPYDType = 18 | | Bool 19 | | Byte 20 | | UInt8 21 | | UInt16 22 | | UInt32 23 | | UInt64 24 | | Int8 25 | | Int16 26 | | Int32 27 | | Int64 28 | | Float16 29 | | Float32 30 | | Float64 31 | with 32 | override this.ToString() = 33 | match this with 34 | | Bool -> "bool" 35 | | Byte -> "byte" 36 | | UInt8 -> "uint8" 37 | | UInt16 -> "uint16" 38 | | UInt32 -> "uint32" 39 | | UInt64 -> "uint64" 40 | | Int8 -> "int8" 41 | | Int16 -> "int16" 42 | | Int32 -> "int32" 43 | | Int64 -> "int64" 44 | | Float16 -> "float16" 45 | | Float32 -> "float32" 46 | | Float64 -> "float64" 47 | 48 | member this.ToNPYString() = 49 | match this with 50 | | Bool -> "b1" 51 | | Byte -> "i1" 52 | | UInt8 -> "u1" 53 | | UInt16 -> "u2" 54 | | UInt32 -> "u4" 55 | | UInt64 -> "u8" 56 | | Int8 -> "i1" 57 | | Int16 -> "i2" 58 | | Int32 -> "i4" 59 | | Int64 -> "i8" 60 | | Float16 -> "f2" 61 | | Float32 -> "f4" 62 | | Float64 -> "f8" 63 | 64 | member this.ByteWidth = 65 | match this with 66 | | Bool 67 | | Byte 68 | | UInt8 69 | | Int8 -> 1 70 | | UInt16 71 | | Int16 72 | | Float16 -> 2 73 | | UInt32 74 | | Int32 75 | | Float32 -> 4 76 | | UInt64 77 | | Int64 78 | | Float64 -> 8 79 | 80 | static member FromNPYString(npyString:string) = 81 | match npyString with 82 | | "b1" -> Bool 83 | | "i1" -> Byte 84 | | "u1" -> UInt8 85 | | "u2" -> UInt16 86 | | "u4" -> UInt32 87 | | "u8" -> UInt64 88 | | "i2" -> Int16 89 | | "i4" -> Int32 90 | | "i8" -> Int64 91 | | "f2" -> Float16 92 | | "f4" -> Float32 93 | | "f8" -> Float64 94 | | _ -> failwith "unsupported" 95 | 96 | static member FromType(t:Type) = 97 | let t = if t.IsArray then t.GetElementType() else t 98 | if t = typeof then NPYDType.Bool 99 | elif t = typeof then NPYDType.Byte 100 | elif t = typeof then NPYDType.UInt8 101 | elif t = typeof then NPYDType.UInt16 102 | elif t = typeof then NPYDType.UInt32 103 | elif t = typeof then NPYDType.UInt64 104 | elif t = typeof then NPYDType.Int16 105 | elif t = typeof then NPYDType.Int32 106 | elif t = typeof then NPYDType.Int64 107 | elif t = typeof then NPYDType.Float32 108 | elif t = typeof then NPYDType.Float64 109 | else failwith "unsupported" 110 | 111 | member this.TryToType() : Type option = 112 | match this with 113 | | Bool -> typeof |> Some 114 | | Byte -> typeof |> Some 115 | | UInt8 -> typeof |> Some 116 | | UInt16 -> typeof |> Some 117 | | UInt32 -> typeof |> Some 118 | | UInt64 -> typeof |> Some 119 | | Int8 -> typeof |> Some 120 | | Int16 -> typeof |> Some 121 | | Int32 -> typeof |> Some 122 | | Int64 -> typeof |> Some 123 | | Float16 -> None //failwith "unsupported" 124 | | Float32 -> typeof |> Some 125 | | Float64 -> typeof |> Some 126 | 127 | let private ASCII = System.Text.ASCIIEncoding.ASCII 128 | 129 | /// "x93NUMPY10" 130 | let private magic_string = [|147uy; 78uy; 85uy; 77uy; 80uy; 89uy; 1uy; 0uy;|] 131 | 132 | type NPYDescription = 133 | { 134 | npyDType:NPYDType 135 | isLittleEnding:bool option 136 | /// Fortran is column-major and C is row-major 137 | fortran_order:bool 138 | shape:int[] 139 | } 140 | with 141 | static member Default = 142 | { 143 | npyDType=NPYDType.Float32; 144 | isLittleEnding=Some(true); 145 | fortran_order=false;shape=[||] 146 | } 147 | override this.ToString() = 148 | sprintf "%O (%O)" this.npyDType (this.shape |> Array.map string |> String.concat ",") 149 | 150 | let parseHeader(header:string) = 151 | let headerRecord = 152 | header.Split('{','}') 153 | |> Array.filter (String.IsNullOrWhiteSpace >> not) 154 | |> Array.head 155 | 156 | let getPropertyValue(name) = 157 | match headerRecord.IndexOf(sprintf "'%s':" name) with 158 | | -1 -> Error (sprintf "not found %s" name) 159 | | i -> 160 | // Scan through to the next ',' that is not inside a parens stack 161 | let charArray = 162 | Array.unfold (fun (index:int,parenStackCount:int) -> 163 | if index >= headerRecord.Length then None 164 | else 165 | let c = headerRecord.[index] 166 | match c with 167 | | '(' -> Some(c,(index+1,parenStackCount+1)) 168 | | ')' -> Some(c,(index+1,parenStackCount-1)) 169 | | ',' when parenStackCount = 0 -> None 170 | | _ -> Some(c,(index+1,parenStackCount)) 171 | ) (i + name.Length + 3,0) 172 | Ok (String(charArray).Trim()) 173 | 174 | let (isLittle,npyDType) = 175 | match (getPropertyValue("descr") |> Result.requireOk).ToCharArray() with 176 | | [|'\'';little;x;y;'\''|] -> 177 | let isLittle = match little with | '|' -> None | '<' -> Some(true) | '>' -> Some(false) | _ -> failwith "unsupported" 178 | (isLittle, NPYDType.FromNPYString(String([|x;y|]))) 179 | | _ -> failwith "error parsing descr" 180 | 181 | let fortran_order = 182 | match getPropertyValue("fortran_order") |> Result.requireOk with 183 | | "False" -> false 184 | | "True" -> true 185 | | x -> failwithf "fortan_order value %s is unsupported" x 186 | 187 | let shape = 188 | (getPropertyValue("shape") 189 | |> Result.requireOk 190 | |> String.filter (function | '(' | ')' | ' ' -> false | _ -> true)).Split(',') 191 | |> Array.filter (String.IsNullOrWhiteSpace >> not) 192 | |> Array.map (Int32.Parse) 193 | 194 | {npyDType=npyDType; isLittleEnding=isLittle; fortran_order=fortran_order; shape=shape;} 195 | 196 | let readNumpy(bytes:byte[]) : Result<(NPYDescription*Array),string> = 197 | let convertToArray(bytes:byte[],offset:int,shape:int[],dtype:NPYDType) : Array = 198 | let length = shape |> Seq.reduce (*) 199 | match dtype with 200 | | NPYDType.Bool -> 201 | let returnArray = Array.zeroCreate length 202 | Buffer.BlockCopy(bytes,offset,returnArray,0,length * dtype.ByteWidth) 203 | returnArray :> Array 204 | | NPYDType.Byte -> 205 | bytes.[offset..] :> Array 206 | | NPYDType.UInt8 -> 207 | let returnArray = Array.zeroCreate length 208 | Buffer.BlockCopy(bytes,offset,returnArray,0,length * dtype.ByteWidth) 209 | returnArray :> Array 210 | | NPYDType.UInt16 -> 211 | let returnArray = Array.zeroCreate length 212 | Buffer.BlockCopy(bytes,offset,returnArray,0,length * dtype.ByteWidth) 213 | returnArray :> Array 214 | | NPYDType.UInt32 -> 215 | let returnArray = Array.zeroCreate length 216 | Buffer.BlockCopy(bytes,offset,returnArray,0,length * dtype.ByteWidth) 217 | returnArray :> Array 218 | | NPYDType.UInt64 -> 219 | let returnArray = Array.zeroCreate length 220 | Buffer.BlockCopy(bytes,offset,returnArray,0,length * dtype.ByteWidth) 221 | returnArray :> Array 222 | | NPYDType.Int8 -> 223 | let returnArray = Array.zeroCreate length 224 | Buffer.BlockCopy(bytes,offset,returnArray,0,length * dtype.ByteWidth) 225 | returnArray :> Array 226 | | NPYDType.Int16 -> 227 | let returnArray = Array.zeroCreate length 228 | Buffer.BlockCopy(bytes,offset,returnArray,0,length * dtype.ByteWidth) 229 | returnArray :> Array 230 | | NPYDType.Int32 -> 231 | let returnArray = Array.zeroCreate length 232 | Buffer.BlockCopy(bytes,offset,returnArray,0,length * dtype.ByteWidth) 233 | returnArray :> Array 234 | | NPYDType.Int64 -> 235 | let returnArray = Array.zeroCreate length 236 | Buffer.BlockCopy(bytes,offset,returnArray,0,length * dtype.ByteWidth) 237 | returnArray :> Array 238 | | NPYDType.Float16 -> 239 | failwith "unsuported at this time" 240 | //let returnArray = Array.zeroCreate length 241 | //Buffer.BlockCopy(bytes,offset,returnArray,0,length * dtype.ByteWidth) 242 | //returnArray |> Array.map System.Half.ToHalf :> Array 243 | | NPYDType.Float32 -> 244 | let returnArray = Array.zeroCreate length 245 | Buffer.BlockCopy(bytes,offset,returnArray,0,length * dtype.ByteWidth) 246 | returnArray :> Array 247 | | NPYDType.Float64 -> 248 | let returnArray = Array.zeroCreate length 249 | Buffer.BlockCopy(bytes,offset,returnArray,0,length * dtype.ByteWidth) 250 | returnArray :> Array 251 | 252 | if magic_string <> bytes.[..7] then 253 | Error "Magic String missmatch" 254 | else 255 | let len = BitConverter.ToUInt16(bytes,8) 256 | let header = parseHeader <| ASCII.GetString(bytes,10,int len) 257 | Ok (header, convertToArray(bytes,10 + int len, header.shape, header.npyDType)) 258 | 259 | let writeHeader(dtype : NPYDType,shape : int[]) = 260 | let typeDesc = sprintf "%c%s" (if dtype.ByteWidth = 1 then '|' else '<') (dtype.ToNPYString()) 261 | sprintf "{'descr': '%s', 'fortran_order': False, 'shape': (%s), }" 262 | typeDesc (shape |> Array.map string |> String.concat ", ") 263 | 264 | /// Note: It is assumed that arr already has the propper C format data layout 265 | let writeArrayToNumpy(arr:Array,shape:int[]) : byte[] = 266 | let size = shape |> Array.reduce (*) 267 | let dtype = NPYDType.FromType(arr.GetType()) 268 | let preambleLen = 10 269 | let header = writeHeader(dtype,shape) 270 | let len = header.Length + 1 271 | let headerSize = len + preambleLen 272 | let paddedHeader = header.PadRight(header.Length + (16 - (headerSize % 16))) + "\n" 273 | if (10 + paddedHeader.Length) % 16 <> 0 then 274 | failwith "header pad error" 275 | let bufferSize : int = preambleLen + paddedHeader.Length + size * dtype.ByteWidth 276 | let buffer = Array.zeroCreate bufferSize 277 | Buffer.BlockCopy(magic_string, 0, buffer, 0, magic_string.Length) 278 | let lenBits = BitConverter.GetBytes(uint16 (paddedHeader.Length)) 279 | buffer.[8] <- lenBits.[0] 280 | buffer.[9] <- lenBits.[1] 281 | let headerBytes = ASCII.GetBytes(paddedHeader) 282 | Buffer.BlockCopy(headerBytes ,0, buffer, preambleLen, headerBytes.Length) 283 | /// NOTE: This only works for simple single dimention arrays 284 | Buffer.BlockCopy(arr,0,buffer,preambleLen + headerBytes.Length, size * dtype.ByteWidth) 285 | buffer 286 | 287 | (* 288 | // WARN: Untested 289 | let saveToNPZ(npys:Map) = 290 | use ms = new MemoryStream() 291 | use zip = new ZipArchive(ms,ZipArchiveMode.Create) 292 | for KeyValue(key,(desc,arr)) in npys do 293 | let entry = zip.CreateEntry(key) 294 | let s = entry.Open() 295 | let buffer = writeArrayToNumpy(arr,desc.shape) 296 | s.Write(buffer,0,buffer.Length) 297 | s.Flush() 298 | s.Close() // double check if any of these are redundant 299 | s.Dispose() 300 | ms.ToArray() 301 | *) 302 | 303 | let readFromNPZ(data:byte[]) : Map = 304 | use ms = new MemoryStream(data) 305 | use zip = new ZipArchive(ms, ZipArchiveMode.Read) 306 | [| 307 | for entry in zip.Entries -> 308 | use targetMs = new MemoryStream() 309 | use entryStream = entry.Open() 310 | entryStream.CopyTo(targetMs) 311 | let bytes = targetMs.ToArray() 312 | let desc,arr = readNumpy(bytes) |> Result.requireOk 313 | entry.FullName, (desc,arr) 314 | |] |> Map.ofArray 315 | 316 | 317 | 318 | -------------------------------------------------------------------------------- /TensorflowTypeProvider/TensorflowTypeProvider/Properties/PublishProfiles/FolderProfile.pubxml: -------------------------------------------------------------------------------- 1 |  2 | 5 | 6 | 7 | FileSystem 8 | Release 9 | Any CPU 10 | netstandard2.0 11 | bin\Release\netstandard2.0\publish\ 12 | 13 | -------------------------------------------------------------------------------- /TensorflowTypeProvider/TensorflowTypeProvider/ProvidedTypes.fsi: -------------------------------------------------------------------------------- 1 | // Copyright (c) Microsoft Corporation 2006-2014 and other contributors. 2 | // This sample code is provided "as is" without warranty of any kind. 3 | // We disclaim all warranties, either express or implied, including the 4 | // warranties of merchantability and fitness for a particular purpose. 5 | // 6 | // This file contains a set of helper types and methods for providing types in an implementation 7 | // of ITypeProvider. 8 | // 9 | // This code has been modified and is appropriate for use in conjunction with the F# 3.0-4.0 releases 10 | 11 | #if INTERNAL_FSHARP_TYPEPROVIDERS_SDK_TESTS 12 | 13 | namespace ProviderImplementation.ProvidedTypes.AssemblyReader 14 | 15 | open System 16 | open System.Collections.ObjectModel 17 | 18 | [] 19 | module internal Reader = 20 | 21 | type ILModuleReader = class end 22 | 23 | val GetWeakReaderCache : unit -> System.Collections.Concurrent.ConcurrentDictionary<(string * string), DateTime * WeakReference> 24 | val GetStrongReaderCache : unit -> System.Collections.Concurrent.ConcurrentDictionary<(string * string), DateTime * int * ILModuleReader> 25 | 26 | #endif 27 | 28 | namespace ProviderImplementation.ProvidedTypes 29 | 30 | open System 31 | open System.Collections.Generic 32 | open System.Reflection 33 | open System.Linq.Expressions 34 | open Microsoft.FSharp.Quotations 35 | open Microsoft.FSharp.Core.CompilerServices 36 | 37 | /// Represents an erased provided parameter 38 | [] 39 | type ProvidedParameter = 40 | inherit ParameterInfo 41 | 42 | /// Create a new provided parameter. 43 | new : parameterName: string * parameterType: Type * ?isOut: bool * ?optionalValue: obj -> ProvidedParameter 44 | 45 | /// Indicates if the parameter is marked as ParamArray 46 | member IsParamArray: bool with set 47 | 48 | /// Indicates if the parameter is marked as ReflectedDefinition 49 | member IsReflectedDefinition: bool with set 50 | 51 | /// Indicates if the parameter has a default value 52 | member HasDefaultParameterValue: bool 53 | 54 | /// Add a custom attribute to the provided parameter. 55 | member AddCustomAttribute: CustomAttributeData -> unit 56 | 57 | /// Represents a provided static parameter. 58 | [] 59 | type ProvidedStaticParameter = 60 | inherit ParameterInfo 61 | 62 | /// Create a new provided static parameter, for use with DefineStaticParamaeters on a provided type definition. 63 | new: parameterName: string * parameterType: Type * ?parameterDefaultValue: obj -> ProvidedStaticParameter 64 | 65 | /// Add XML documentation information to this provided constructor 66 | member AddXmlDoc: xmlDoc: string -> unit 67 | 68 | /// Add XML documentation information to this provided constructor, where the computation of the documentation is delayed until necessary 69 | member AddXmlDocDelayed: xmlDocFunction: (unit -> string) -> unit 70 | 71 | 72 | /// Represents an erased provided constructor. 73 | [] 74 | type ProvidedConstructor = 75 | inherit ConstructorInfo 76 | 77 | /// When making a cross-targeting type provider, use this method instead of the ProvidedConstructor constructor from ProvidedTypes 78 | new: parameters: ProvidedParameter list * invokeCode: (Expr list -> Expr) -> ProvidedConstructor 79 | 80 | /// Add a 'Obsolete' attribute to this provided constructor 81 | member AddObsoleteAttribute: message: string * ?isError: bool -> unit 82 | 83 | /// Add XML documentation information to this provided constructor 84 | member AddXmlDoc: xmlDoc: string -> unit 85 | 86 | /// Add XML documentation information to this provided constructor, where the computation of the documentation is delayed until necessary 87 | member AddXmlDocDelayed: xmlDocFunction: (unit -> string) -> unit 88 | 89 | /// Add XML documentation information to this provided constructor, where the documentation is re-computed every time it is required. 90 | member AddXmlDocComputed: xmlDocFunction: (unit -> string) -> unit 91 | 92 | /// Set the target and arguments of the base constructor call. Only used for generated types. 93 | member BaseConstructorCall: (Expr list -> ConstructorInfo * Expr list) with set 94 | 95 | /// Set a flag indicating that the constructor acts like an F# implicit constructor, so the 96 | /// parameters of the constructor become fields and can be accessed using Expr.GlobalVar with the 97 | /// same name. 98 | member IsImplicitConstructor: bool with get,set 99 | 100 | /// Add definition location information to the provided constructor. 101 | member AddDefinitionLocation: line:int * column:int * filePath:string -> unit 102 | 103 | member IsTypeInitializer: bool with get,set 104 | 105 | /// This method is for internal use only in the type provider SDK 106 | member internal GetInvokeCode: Expr list -> Expr 107 | 108 | 109 | 110 | [] 111 | type ProvidedMethod = 112 | inherit MethodInfo 113 | 114 | /// When making a cross-targeting type provider, use this method instead of the ProvidedMethod constructor from ProvidedTypes 115 | new: methodName: string * parameters: ProvidedParameter list * returnType: Type * ?invokeCode: (Expr list -> Expr) * ?isStatic: bool -> ProvidedMethod 116 | 117 | /// Add XML documentation information to this provided method 118 | member AddObsoleteAttribute: message: string * ?isError: bool -> unit 119 | 120 | /// Add XML documentation information to this provided constructor 121 | member AddXmlDoc: xmlDoc: string -> unit 122 | 123 | /// Add XML documentation information to this provided constructor, where the computation of the documentation is delayed until necessary 124 | member AddXmlDocDelayed: xmlDocFunction: (unit -> string) -> unit 125 | 126 | /// Add XML documentation information to this provided constructor, where the computation of the documentation is delayed until necessary 127 | /// The documentation is re-computed every time it is required. 128 | member AddXmlDocComputed: xmlDocFunction: (unit -> string) -> unit 129 | 130 | member AddMethodAttrs: attributes:MethodAttributes -> unit 131 | 132 | /// Set the method attributes of the method. By default these are simple 'MethodAttributes.Public' 133 | member SetMethodAttrs: attributes:MethodAttributes -> unit 134 | 135 | /// Add definition location information to the provided type definition. 136 | member AddDefinitionLocation: line:int * column:int * filePath:string -> unit 137 | 138 | /// Add a custom attribute to the provided method definition. 139 | member AddCustomAttribute: CustomAttributeData -> unit 140 | 141 | /// Define the static parameters available on a statically parameterized method 142 | member DefineStaticParameters: parameters: ProvidedStaticParameter list * instantiationFunction: (string -> obj[] -> ProvidedMethod) -> unit 143 | 144 | /// This method is for internal use only in the type provider SDK 145 | member internal GetInvokeCode: (Expr list -> Expr) option 146 | 147 | 148 | /// Represents an erased provided property. 149 | [] 150 | type ProvidedProperty = 151 | inherit PropertyInfo 152 | 153 | /// Create a new provided property. It is not initially associated with any specific provided type definition. 154 | new: propertyName: string * propertyType: Type * ?getterCode: (Expr list -> Expr) * ?setterCode: (Expr list -> Expr) * ?isStatic: bool * ?indexParameters: ProvidedParameter list -> ProvidedProperty 155 | 156 | /// Add a 'Obsolete' attribute to this provided property 157 | member AddObsoleteAttribute: message: string * ?isError: bool -> unit 158 | 159 | /// Add XML documentation information to this provided constructor 160 | member AddXmlDoc: xmlDoc: string -> unit 161 | 162 | /// Add XML documentation information to this provided constructor, where the computation of the documentation is delayed until necessary 163 | member AddXmlDocDelayed: xmlDocFunction: (unit -> string) -> unit 164 | 165 | /// Add XML documentation information to this provided constructor, where the computation of the documentation is delayed until necessary 166 | /// The documentation is re-computed every time it is required. 167 | member AddXmlDocComputed: xmlDocFunction: (unit -> string) -> unit 168 | 169 | /// Get or set a flag indicating if the property is static. 170 | member IsStatic: bool 171 | 172 | /// Add definition location information to the provided type definition. 173 | member AddDefinitionLocation: line:int * column:int * filePath:string -> unit 174 | 175 | /// Add a custom attribute to the provided property definition. 176 | member AddCustomAttribute: CustomAttributeData -> unit 177 | 178 | 179 | /// Represents an erased provided property. 180 | [] 181 | type ProvidedEvent = 182 | inherit EventInfo 183 | 184 | /// Create a new provided event. It is not initially associated with any specific provided type definition. 185 | new: eventName: string * eventHandlerType: Type * adderCode: (Expr list -> Expr) * removerCode: (Expr list -> Expr) * ?isStatic: bool -> ProvidedEvent 186 | 187 | /// Add XML documentation information to this provided constructor 188 | member AddXmlDoc: xmlDoc: string -> unit 189 | 190 | /// Add XML documentation information to this provided constructor, where the computation of the documentation is delayed until necessary 191 | member AddXmlDocDelayed: xmlDocFunction: (unit -> string) -> unit 192 | 193 | /// Add XML documentation information to this provided constructor, where the computation of the documentation is delayed until necessary 194 | /// The documentation is re-computed every time it is required. 195 | member AddXmlDocComputed: xmlDocFunction: (unit -> string) -> unit 196 | 197 | /// Get a flag indicating if the property is static. 198 | member IsStatic: bool with get 199 | 200 | /// Add definition location information to the provided type definition. 201 | member AddDefinitionLocation: line:int * column:int * filePath:string -> unit 202 | 203 | 204 | /// Represents an erased provided field. 205 | [] 206 | type ProvidedField = 207 | inherit FieldInfo 208 | 209 | /// Create a new provided field. It is not initially associated with any specific provided type definition. 210 | new: fieldName: string * fieldType: Type -> ProvidedField 211 | 212 | /// Add a 'Obsolete' attribute to this provided field 213 | member AddObsoleteAttribute: message: string * ?isError: bool -> unit 214 | 215 | /// Add XML documentation information to this provided field 216 | member AddXmlDoc: xmlDoc: string -> unit 217 | 218 | /// Add XML documentation information to this provided field, where the computation of the documentation is delayed until necessary 219 | member AddXmlDocDelayed: xmlDocFunction: (unit -> string) -> unit 220 | 221 | /// Add XML documentation information to this provided field, where the computation of the documentation is delayed until necessary 222 | /// The documentation is re-computed every time it is required. 223 | member AddXmlDocComputed: xmlDocFunction: (unit -> string) -> unit 224 | 225 | /// Add definition location information to the provided field definition. 226 | member AddDefinitionLocation: line:int * column:int * filePath:string -> unit 227 | 228 | member SetFieldAttributes: attributes: FieldAttributes -> unit 229 | 230 | /// Create a new provided literal field. It is not initially associated with any specific provided type definition. 231 | static member Literal : fieldName: string * fieldType: Type * literalValue:obj -> ProvidedField 232 | 233 | 234 | /// Represents an array or other symbolic type involving a provided type as the argument. 235 | /// See the type provider spec for the methods that must be implemented. 236 | /// Note that the type provider specification does not require us to implement pointer-equality for provided types. 237 | [] 238 | type ProvidedTypeSymbol = 239 | inherit TypeDelegator 240 | 241 | /// For example, kg 242 | member IsFSharpTypeAbbreviation: bool 243 | 244 | /// For example, int or int 245 | member IsFSharpUnitAnnotated: bool 246 | 247 | /// Helpers to build symbolic provided types 248 | [] 249 | type ProvidedTypeBuilder = 250 | 251 | /// Like typ.MakeGenericType, but will also work with unit-annotated types 252 | static member MakeGenericType: genericTypeDefinition: Type * genericArguments: Type list -> Type 253 | 254 | /// Like methodInfo.MakeGenericMethod, but will also work with unit-annotated types and provided types 255 | static member MakeGenericMethod: genericMethodDefinition: MethodInfo * genericArguments: Type list -> MethodInfo 256 | 257 | /// Like FsharpType.MakeTupleType, but will also work with unit-annotated types and provided types 258 | static member MakeTupleType: args: Type list -> Type 259 | 260 | 261 | /// Helps create erased provided unit-of-measure annotations. 262 | [] 263 | type ProvidedMeasureBuilder = 264 | 265 | /// Gets the measure indicating the "1" unit of measure, that is the unitless measure. 266 | static member One: Type 267 | 268 | /// Returns the measure indicating the product of two units of measure, e.g. kg * m 269 | static member Product: measure1: Type * measure2: Type -> Type 270 | 271 | /// Returns the measure indicating the inverse of two units of measure, e.g. 1 / s 272 | static member Inverse: denominator: Type -> Type 273 | 274 | /// Returns the measure indicating the ratio of two units of measure, e.g. kg / m 275 | static member Ratio: numerator: Type * denominator: Type -> Type 276 | 277 | /// Returns the measure indicating the square of a unit of measure, e.g. m * m 278 | static member Square: ``measure``: Type -> Type 279 | 280 | /// Returns the measure for an SI unit from the F# core library, where the string is in capitals and US spelling, e.g. Meter 281 | static member SI: unitName:string -> Type 282 | 283 | /// Returns a type where the type has been annotated with the given types and/or units-of-measure. 284 | /// e.g. float, Vector 285 | static member AnnotateType: basic: Type * argument: Type list -> Type 286 | 287 | 288 | /// Represents a provided type definition. 289 | [] 290 | type ProvidedTypeDefinition = 291 | inherit TypeDelegator 292 | 293 | /// When making a cross-targeting type provider, use this method instead of the corresponding ProvidedTypeDefinition constructor from ProvidedTypes 294 | new: className: string * baseType: Type option * ?hideObjectMethods: bool * ?nonNullable: bool * ?isErased: bool * ?isSealed: bool * ?isInterface: bool -> ProvidedTypeDefinition 295 | 296 | /// When making a cross-targeting type provider, use this method instead of the corresponding ProvidedTypeDefinition constructor from ProvidedTypes 297 | new: assembly: Assembly * namespaceName: string * className: string * baseType: Type option * ?hideObjectMethods: bool * ?nonNullable: bool * ?isErased: bool * ?isSealed: bool * ?isInterface: bool -> ProvidedTypeDefinition 298 | 299 | /// Add the given type as an implemented interface. 300 | member AddInterfaceImplementation: interfaceType: Type -> unit 301 | 302 | /// Add the given function as a set of on-demand computed interfaces. 303 | member AddInterfaceImplementationsDelayed: interfacesFunction:(unit -> Type list)-> unit 304 | 305 | /// Specifies that the given method body implements the given method declaration. 306 | member DefineMethodOverride: methodInfoBody: ProvidedMethod * methodInfoDeclaration: MethodInfo -> unit 307 | 308 | /// Specifies that the given method bodies implement the given method declarations 309 | member DefineMethodOverridesDelayed: (unit -> (ProvidedMethod * MethodInfo) list) -> unit 310 | 311 | /// Add a 'Obsolete' attribute to this provided type definition 312 | member AddObsoleteAttribute: message: string * ?isError: bool -> unit 313 | 314 | /// Add XML documentation information to this provided constructor 315 | member AddXmlDoc: xmlDoc: string -> unit 316 | 317 | /// Set the base type 318 | member SetBaseType: Type -> unit 319 | 320 | /// Set the base type to a lazily evaluated value. Use this to delay realization of the base type as late as possible. 321 | member SetBaseTypeDelayed: baseTypeFunction:(unit -> Type) -> unit 322 | 323 | /// Set underlying type for generated enums 324 | member SetEnumUnderlyingType: Type -> unit 325 | 326 | /// Add XML documentation information to this provided constructor, where the computation of the documentation is delayed until necessary. 327 | /// The documentation is only computed once. 328 | member AddXmlDocDelayed: xmlDocFunction: (unit -> string) -> unit 329 | 330 | /// Add XML documentation information to this provided constructor, where the computation of the documentation is delayed until necessary 331 | /// The documentation is re-computed every time it is required. 332 | member AddXmlDocComputed: xmlDocFunction: (unit -> string) -> unit 333 | 334 | /// Set the attributes on the provided type. This fully replaces the default TypeAttributes. 335 | member SetAttributes: TypeAttributes -> unit 336 | 337 | /// Add a method, property, nested type or other member to a ProvidedTypeDefinition 338 | member AddMember: memberInfo:MemberInfo -> unit 339 | 340 | /// Add a set of members to a ProvidedTypeDefinition 341 | member AddMembers: memberInfos:list<#MemberInfo> -> unit 342 | 343 | /// Add a member to a ProvidedTypeDefinition, delaying computation of the members until required by the compilation context. 344 | member AddMemberDelayed: memberFunction:(unit -> #MemberInfo) -> unit 345 | 346 | /// Add a set of members to a ProvidedTypeDefinition, delaying computation of the members until required by the compilation context. 347 | member AddMembersDelayed: membersFunction:(unit -> list<#MemberInfo>) -> unit 348 | 349 | /// Add the types of the generated assembly as generative types, where types in namespaces get hierarchically positioned as nested types. 350 | member AddAssemblyTypesAsNestedTypesDelayed: assemblyFunction:(unit -> Assembly) -> unit 351 | 352 | /// Define the static parameters available on a statically parameterized type 353 | member DefineStaticParameters: parameters: ProvidedStaticParameter list * instantiationFunction: (string -> obj[] -> ProvidedTypeDefinition) -> unit 354 | 355 | /// Add definition location information to the provided type definition. 356 | member AddDefinitionLocation: line:int * column:int * filePath:string -> unit 357 | 358 | /// Suppress Object entries in intellisense menus in instances of this provided type 359 | member HideObjectMethods: bool 360 | 361 | /// Disallows the use of the null literal. 362 | member NonNullable: bool 363 | 364 | /// Get a flag indicating if the ProvidedTypeDefinition is erased 365 | member IsErased: bool 366 | 367 | /// Get or set a flag indicating if the ProvidedTypeDefinition has type-relocation suppressed 368 | [] 369 | member SuppressRelocation: bool with get,set 370 | 371 | // This method is used by Debug.fs 372 | member ApplyStaticArguments: name:string * args:obj[] -> ProvidedTypeDefinition 373 | 374 | /// Add a custom attribute to the provided type definition. 375 | member AddCustomAttribute: CustomAttributeData -> unit 376 | 377 | /// Emulate the F# type provider type erasure mechanism to get the 378 | /// actual (erased) type. We erase ProvidedTypes to their base type 379 | /// and we erase array of provided type to array of base type. In the 380 | /// case of generics all the generic type arguments are also recursively 381 | /// replaced with the erased-to types 382 | static member EraseType: typ:Type -> Type 383 | 384 | /// Get or set a utility function to log the creation of root Provided Type. Used to debug caching/invalidation. 385 | static member Logger: (string -> unit) option ref 386 | 387 | 388 | #if !NO_GENERATIVE 389 | /// A provided generated assembly 390 | type ProvidedAssembly = 391 | 392 | inherit Assembly 393 | 394 | /// Create a provided generated assembly 395 | new: assemblyName: AssemblyName * assemblyFileName:string -> ProvidedAssembly 396 | 397 | /// Create a provided generated assembly using a temporary file as the interim assembly storage 398 | new: unit -> ProvidedAssembly 399 | 400 | /// Emit the given provided type definitions as part of the assembly 401 | /// and adjust the 'Assembly' property of all provided type definitions to return that 402 | /// assembly. 403 | /// 404 | /// The assembly is only emitted when the Assembly property on the root type is accessed for the first time. 405 | /// The host F# compiler does this when processing a generative type declaration for the type. 406 | member AddTypes: types: ProvidedTypeDefinition list -> unit 407 | 408 | /// 409 | /// Emit the given nested provided type definitions as part of the assembly. 410 | /// and adjust the 'Assembly' property of all provided type definitions to return that 411 | /// assembly. 412 | /// 413 | /// A path of type names to wrap the generated types. The generated types are then generated as nested types. 414 | member AddNestedTypes: types: ProvidedTypeDefinition list * enclosingGeneratedTypeNames: string list -> unit 415 | 416 | #endif 417 | 418 | 419 | 420 | [] 421 | /// Represents the context for which code is to be generated. Normally you should not need to use this directly. 422 | type ProvidedTypesContext = 423 | 424 | /// Try to find the given target assembly in the context 425 | member TryBindAssemblyNameToTarget: aref: AssemblyName -> Choice 426 | 427 | /// Try to find the given target assembly in the context 428 | member TryBindSimpleAssemblyNameToTarget: assemblyName: string -> Choice 429 | 430 | /// Get the list of referenced assemblies determined by the type provider configuration 431 | member ReferencedAssemblyPaths: string list 432 | 433 | /// Get the resolved referenced assemblies determined by the type provider configuration 434 | member GetTargetAssemblies : unit -> Assembly[] 435 | 436 | /// Get the set of design-time assemblies available to use as a basis for authoring provided types. 437 | member GetSourceAssemblies : unit -> Assembly[] 438 | 439 | /// Add an assembly to the set of design-time assemblies available to use as a basis for authoring provided types 440 | member AddSourceAssembly : Assembly -> unit 441 | 442 | /// Try to get the version of FSharp.Core referenced. May raise an exception if FSharp.Core has not been correctly resolved 443 | member FSharpCoreAssemblyVersion: Version 444 | 445 | /// Returns a type from the referenced assemblies that corresponds to the given design-time type. Normally 446 | /// this method should not be used directly when authoring a type provider. 447 | member ConvertSourceTypeToTarget: Type -> Type 448 | 449 | /// Returns the design-time type that corresponds to the given type from the target referenced assemblies. Normally 450 | /// this method should not be used directly when authoring a type provider. 451 | member ConvertTargetTypeToSource: Type -> Type 452 | 453 | /// Returns a quotation rebuilt with resepct to the types from the target referenced assemblies. Normally 454 | /// this method should not be used directly when authoring a type provider. 455 | member ConvertSourceExprToTarget: Expr -> Expr 456 | 457 | /// Read the assembly related to this context 458 | member ReadRelatedAssembly: fileName: string -> Assembly 459 | 460 | /// Read the assembly related to this context 461 | member ReadRelatedAssembly: bytes: byte[] -> Assembly 462 | 463 | /// A base type providing default implementations of type provider functionality. 464 | type TypeProviderForNamespaces = 465 | 466 | /// Initializes a type provider to provide the types in the given namespace. 467 | /// 468 | /// Optionally specify the design-time assemblies available to use as a basis for authoring provided types. 469 | /// The transitive dependencies of these assemblies are also included. By default 470 | /// Assembly.GetCallingAssembly() and its transitive dependencies are used. 471 | /// 472 | /// 473 | /// 474 | /// Optionally specify a map of assembly names from source model to referenced assemblies. 475 | /// 476 | /// 477 | /// 478 | /// Optionally specify that the location of the type provider design-time component should be used to resolve failing assembly resolutions. 479 | /// This flag or an equivalent call to RegisterProbingFolder is generally needed for any type provider design-time components loaded into .NET Core tooling. 480 | /// 481 | new: config: TypeProviderConfig * namespaceName:string * types: ProvidedTypeDefinition list * ?sourceAssemblies: Assembly list * ?assemblyReplacementMap: (string * string) list * ?addDefaultProbingLocation: bool -> TypeProviderForNamespaces 482 | 483 | /// Initializes a type provider. 484 | /// 485 | /// Optionally specify the design-time assemblies available to use as a basis for authoring provided types. 486 | /// The transitive dependencies of these assemblies are also included. By default 487 | /// Assembly.GetCallingAssembly() and its transitive dependencies are used. 488 | /// 489 | /// 490 | /// 491 | /// Optionally specify a map of assembly names from source model to referenced assemblies. 492 | /// 493 | /// 494 | /// 495 | /// Optionally specify that the location of the type provider design-time component should be used to resolve failing assembly resolutions. 496 | /// This flag or an equivalent call to RegisterProbingFolder is generally needed for any type provider design-time components loaded into .NET Core tooling. 497 | /// 498 | new: config: TypeProviderConfig * ?sourceAssemblies: Assembly list * ?assemblyReplacementMap: (string * string) list * ?addDefaultProbingLocation: bool -> TypeProviderForNamespaces 499 | 500 | /// Invoked by the type provider to add a namespace of provided types in the specification of the type provider. 501 | member AddNamespace: namespaceName:string * types: ProvidedTypeDefinition list -> unit 502 | 503 | /// Invoked by the type provider to get all provided namespaces with their provided types. 504 | member Namespaces: IProvidedNamespace[] 505 | 506 | /// Invoked by the type provider to invalidate the information provided by the provider 507 | member Invalidate: unit -> unit 508 | 509 | /// Invoked by the host of the type provider to get the static parameters for a method. 510 | member GetStaticParametersForMethod: MethodBase -> ParameterInfo[] 511 | 512 | /// Invoked by the host of the type provider to apply the static argumetns for a method. 513 | member ApplyStaticArgumentsForMethod: MethodBase * string * obj[] -> MethodBase 514 | 515 | #if !FX_NO_LOCAL_FILESYSTEM 516 | /// AssemblyResolve handler. Default implementation searches .dll file in registered folders 517 | abstract ResolveAssembly: ResolveEventArgs -> Assembly 518 | default ResolveAssembly: ResolveEventArgs -> Assembly 519 | 520 | /// Registers custom probing path that can be used for probing assemblies 521 | member RegisterProbingFolder: folder: string -> unit 522 | 523 | /// Registers location of RuntimeAssembly (from TypeProviderConfig) as probing folder 524 | member RegisterRuntimeAssemblyLocationAsProbingFolder: config: TypeProviderConfig -> unit 525 | 526 | #endif 527 | 528 | #if !NO_GENERATIVE 529 | /// Register that a given file is a provided generated target assembly, e.g. an assembly produced by an external 530 | /// code generation tool. This assembly should be a target assembly, i.e. use the same asssembly references 531 | /// as given by TargetContext.ReferencedAssemblyPaths 532 | member RegisterGeneratedTargetAssembly: fileName: string -> Assembly 533 | #endif 534 | 535 | [] 536 | member Disposing: IEvent 537 | 538 | /// The context for which code is eventually to be generated. You should not normally 539 | /// need to use this property directly, as translation from the compiler-hosted context to 540 | /// the design-time context will normally be performed automatically. 541 | member TargetContext: ProvidedTypesContext 542 | 543 | interface ITypeProvider 544 | 545 | 546 | module internal UncheckedQuotations = 547 | 548 | type Expr with 549 | static member NewDelegateUnchecked: ty:Type * vs:Var list * body:Expr -> Expr 550 | static member NewObjectUnchecked: cinfo:ConstructorInfo * args:Expr list -> Expr 551 | static member NewArrayUnchecked: elementType:Type * elements:Expr list -> Expr 552 | static member CallUnchecked: minfo:MethodInfo * args:Expr list -> Expr 553 | static member CallUnchecked: obj:Expr * minfo:MethodInfo * args:Expr list -> Expr 554 | static member ApplicationUnchecked: f:Expr * x:Expr -> Expr 555 | static member PropertyGetUnchecked: pinfo:PropertyInfo * args:Expr list -> Expr 556 | static member PropertyGetUnchecked: obj:Expr * pinfo:PropertyInfo * ?args:Expr list -> Expr 557 | static member PropertySetUnchecked: pinfo:PropertyInfo * value:Expr * ?args:Expr list -> Expr 558 | static member PropertySetUnchecked: obj:Expr * pinfo:PropertyInfo * value:Expr * ?args:Expr list -> Expr 559 | static member FieldGetUnchecked: pinfo:FieldInfo -> Expr 560 | static member FieldGetUnchecked: obj:Expr * pinfo:FieldInfo -> Expr 561 | static member FieldSetUnchecked: pinfo:FieldInfo * value:Expr -> Expr 562 | static member FieldSetUnchecked: obj:Expr * pinfo:FieldInfo * value:Expr -> Expr 563 | static member TupleGetUnchecked: e:Expr * n:int -> Expr 564 | static member LetUnchecked: v:Var * e:Expr * body:Expr -> Expr 565 | static member NewRecordUnchecked : ty:Type * args:Expr list -> Expr 566 | 567 | type Shape 568 | val ( |ShapeCombinationUnchecked|ShapeVarUnchecked|ShapeLambdaUnchecked| ): e:Expr -> Choice<(Shape * Expr list),Var, (Var * Expr)> 569 | val RebuildShapeCombinationUnchecked: Shape * args:Expr list -> Expr 570 | 571 | 572 | 573 | -------------------------------------------------------------------------------- /TensorflowTypeProvider/TensorflowTypeProvider/TensorflowTypeProvider.fs: -------------------------------------------------------------------------------- 1 | namespace TensorflowTypeProvider 2 | 3 | open FSharp.Core.CompilerServices 4 | open FSharp.Quotations 5 | open NPYReaderWriter 6 | open ProviderImplementation.ProvidedTypes 7 | open System 8 | open System.IO 9 | open System.Numerics 10 | open System.Reflection 11 | open Tensorflow 12 | open Microsoft.FSharp.Quotations 13 | open Microsoft.FSharp.Quotations.Patterns 14 | 15 | // NHWC 16 | // offset_nhwc(n, c, h, w) = n * HWC + h * WC + w * C + c 17 | module Array2D = 18 | /// Converts an array of either Rank 1 or 2 to a Array2D 19 | let ofArray<'a> (H : int) (W : int) (arr : Array) = 20 | // C row major is assumed 21 | if arr.Rank = 1 then 22 | if arr.Length = H * W then 23 | Array2D.init H W (fun h w -> arr.GetValue(h*W + w) :?> 'a) 24 | else 25 | failwithf "Shape missmatch. Expected count of %i found %i" (H*W) arr.Length 26 | elif arr.Rank = 2 then 27 | if arr.GetLength(0) = H && arr.GetLength(1) = W then 28 | arr :?> 'a[,] 29 | else failwithf "Shape missmatch. Expected (%i,%i) found (%i,%i)" H W (arr.GetLength(0)) (arr.GetLength(1)) 30 | else failwithf "Rank 1 or 2 expected, rank %i found" arr.Rank 31 | 32 | module Array3D = 33 | /// Converts an array of either Rank 1 or 3 to a Array3D 34 | let ofArray<'a> (H : int) (W : int) (C : int) (arr : Array) = 35 | // C row major is assumed 36 | if arr.Rank = 1 then 37 | if arr.Length = H * W * C then 38 | Array3D.init H W C (fun h w c-> arr.GetValue(h*W*C + w*C + c) :?> 'a) 39 | else 40 | failwithf "Shape missmatch. Expected count of %i found %i" (H*W*C) arr.Length 41 | elif arr.Rank = 3 then 42 | if arr.GetLength(0) = H && arr.GetLength(1) = W && arr.GetLength(2) = C then 43 | arr :?> 'a[,,] 44 | else failwithf "Shape missmatch. Expected (%i,%i,%i) found (%i,%i,%i)" H W C (arr.GetLength(0)) (arr.GetLength(1)) (arr.GetLength(2)) 45 | else failwithf "Rank 1 or 3 expected, rank %i found" arr.Rank 46 | 47 | module Array4D = 48 | /// Converts an array of either Rank 1 or 4 to a Array4D 49 | let ofArray<'a> (N : int) (H : int) (W : int) (C : int) (arr : Array) = 50 | // C row major is assumed 51 | if arr.Rank = 1 then 52 | if arr.Length = N * H * W * C then 53 | Array4D.init N H W C (fun n h w c -> arr.GetValue(n*H*W*C + h*W*C + w*C + c) :?> 'a) 54 | else 55 | failwithf "Shape missmatch. Expected count of %i found %i" (N*H*W*C) arr.Length 56 | elif arr.Rank = 4 then 57 | if arr.GetLength(0) = N && arr.GetLength(1) = H && arr.GetLength(2) = W && arr.GetLength(3) = C then 58 | arr :?> 'a[,,,] 59 | else failwithf "Shape missmatch. Expected (%i,%i,%i,%i) found (%i,%i,%i,%i)" 60 | N H W C (arr.GetLength(0)) (arr.GetLength(1)) (arr.GetLength(2)) (arr.GetLength(3)) 61 | else failwithf "Rank 1 or 4 expected, rank %i found" arr.Rank 62 | 63 | module String = 64 | /// Returns up to the first N characters of the string 65 | let truncate N (xs : string) = if xs.Length > N then xs.Substring(0, N) else xs 66 | 67 | 68 | /// Multi-way tree 69 | type RoseTree<'a> = | Node of 'a*RoseTree<'a>[] 70 | 71 | type TreeNames = { full : string; local : string; display : string } 72 | 73 | module RoseTree = 74 | /// Builds a new RoseTree whose elements are the result of applying the given function to each of the elements 75 | let rec map (f : 'a->'b) (tree : RoseTree<'a>) = 76 | match tree with 77 | | Node(x, [||]) -> Node(f(x), [||]) 78 | | Node(x, xs) -> Node(f(x), xs |> Array.map (map f)) 79 | 80 | let print (f : 'a->string) (roseTrees : RoseTree<'a>[]) = 81 | let rec printRoseTree (depth : int) (roseTrees : RoseTree<'a>[]) = 82 | for roseTree in roseTrees do 83 | match roseTree with 84 | | Node(x, xs) -> 85 | printfn "%s %s" ("".PadLeft(depth * 2)) (f(x)) 86 | printRoseTree (depth + 1) xs 87 | printRoseTree 0 roseTrees 88 | 89 | [] 90 | module Utils = 91 | 92 | /// This parses binary data to produce an Arrays 93 | /// These Arrays have a Rank of 1, in this context it is expected that these arrays will be 94 | /// reshaped by another function 95 | /// There are faster methods 96 | let parseData(data : byte[], dt : DataType) : Array = 97 | let byteSize(dt:DataType) = 98 | match dt with 99 | | DataType.DtFloat -> 4 100 | | DataType.DtDouble ->8 101 | | DataType.DtUint8 -> 1 102 | | DataType.DtUint16 -> 2 103 | | DataType.DtUint32 -> 4 104 | | DataType.DtUint64 -> 8 105 | | DataType.DtInt8 -> 1 106 | | DataType.DtInt16 -> 2 107 | | DataType.DtInt32 -> 4 108 | | DataType.DtInt64 -> 8 109 | | DataType.DtString -> failwith "string should not have a byte size" 110 | | DataType.DtComplex128 -> 16 111 | | DataType.DtBool -> 1 112 | | _ -> failwith "unsuported" 113 | let bs = byteSize(dt) 114 | if data.Length % bs <> 0 then failwith "Data size is not a multiple of DataType ByteSize" 115 | let count = data.Length / bs; 116 | match dt with 117 | | DataType.DtFloat -> 118 | [| for i in 0 .. count - 1 -> BitConverter.ToSingle(data, i*bs)|] :> Array 119 | | DataType.DtDouble -> 120 | [| for i in 0 .. count - 1 -> BitConverter.ToDouble(data, i*bs)|] :> Array 121 | | DataType.DtUint8 -> 122 | data |> Array.map uint8 :> Array 123 | | DataType.DtUint16 -> 124 | [| for i in 0 .. count - 1 -> BitConverter.ToUInt16(data, i*bs)|] :> Array 125 | | DataType.DtUint32 -> 126 | [| for i in 0 .. count - 1 -> BitConverter.ToUInt32(data, i*bs)|] :> Array 127 | | DataType.DtUint64 -> 128 | [| for i in 0 .. count - 1 -> BitConverter.ToUInt64(data, i*bs)|] :> Array 129 | | DataType.DtInt8 -> 130 | data |> Array.map int8 :> Array 131 | | DataType.DtInt16 -> 132 | [| for i in 0 .. count - 1 -> BitConverter.ToInt16(data, i*bs)|] :> Array 133 | | DataType.DtInt32 -> 134 | [| for i in 0 .. count - 1 -> BitConverter.ToInt32(data, i*bs)|] :> Array 135 | | DataType.DtInt64 -> 136 | [| for i in 0 .. count - 1 -> BitConverter.ToInt64(data, i*bs)|] :> Array 137 | | DataType.DtBool -> 138 | data |> Array.map ((<>) 0uy) :> Array 139 | | DataType.DtComplex128 140 | | _ -> failwith "unsuported" 141 | 142 | // NOTE: This is not an invertable mapping so be careful to handle duplicates 143 | /// Replaces chars which require backticks in an effort to make the members easier to use 144 | let replaceBacktickChars = 145 | let prohibitedChars = Set(""" .,+$&[]/\*", `@""".ToCharArray()) 146 | fun (name : string) -> 147 | if String.IsNullOrWhiteSpace(name) then "_" 148 | else 149 | name |> String.map (fun c -> if prohibitedChars.Contains(c) then '_' else c) 150 | |> fun name -> if Char.IsLetter(name.[0]) then name else sprintf "_%s" name 151 | 152 | /// This function generates functions which will produce a unique string with a suffix 153 | /// representing the number of entries if there is more than one 154 | /// This approximates the Tensorflow Make Unique pattern 155 | let getMakeUnique() = 156 | let mutable m = Map.empty 157 | fun s -> 158 | let count = match m.TryFind(s) with | None -> 0 | Some(count) -> count 159 | m <- m.Add(s, count+1) 160 | if count = 0 then s 161 | else sprintf "%s_%i" s count |> fun s' -> m <- m.Add(s', 1); s' 162 | 163 | /// This takes in a list of Operation names and produces a Rooted Tree (Array of RoseTrees) 164 | let getDistinctNames (separator : char) (names : string[]) : RoseTree[] = 165 | let distinctDisplayNames(xs : TreeNames[]) : TreeNames[] = 166 | xs |> Array.scan (fun (_, acc : Map) x -> 167 | match acc.TryFind(x.display) with 168 | | None -> (x, acc.Add(x.display, 1)) 169 | | Some(count) -> 170 | let z' = sprintf "%s_%i" x.display count 171 | ({x with display = z'}, acc.Add(z', 0).Add(x.display, count+1)) 172 | ) ({full = ""; local = ""; display = ""}, Map.empty) 173 | |> Array.skip 1 174 | |> Array.map fst 175 | let rec mapChildren (f : 'a[] -> 'a[]) (rt : RoseTree<'a>) : RoseTree<'a> = 176 | match rt with 177 | | Node(x, [||]) -> Node(x, [||]) 178 | | Node(x, xs) -> 179 | let xs, yss = xs |> Array.map (function | Node(x, ys) -> (x, ys)) |> Array.unzip 180 | let childNodes = yss |> Array.map (fun ys -> ys |> Array.map (mapChildren f)) 181 | // Possible array problem with different sizes?? 182 | Node(x, (f(xs), childNodes) ||> Array.zip |> Array.map (fun (x, ys) -> Node(x, ys))) 183 | let rec group (xss : string list list) : RoseTree[] = 184 | match xss with 185 | | [] -> [||] 186 | | [[]] -> [||] 187 | | _ -> 188 | xss 189 | |> List.groupBy (function | [] -> "" | h::_ -> h) 190 | |> List.map (fun (k, xs) -> Node(k, group (xs |> List.map (function | [] -> [] | _::tail -> tail) |> List.distinct))) 191 | |> List.toArray 192 | let rec mapAcc (f : 'a list ->'b) (acc : 'a list) (tree : RoseTree<'a>) = 193 | match tree with 194 | | Node(x, [||]) -> Node(f(x::acc), [||]) 195 | | Node(x, xs) -> Node(f(x::acc), xs |> Array.map (mapAcc f (x::acc))) 196 | names 197 | |> Array.map (fun x -> x.Split(separator) |> List.ofArray) 198 | |> Array.toList |> group 199 | |> Array.map ( 200 | mapAcc ( 201 | function | [] -> failwith "never - given the group function" 202 | | (head::tail) -> 203 | { full = (head::tail) |> List.rev |> String.concat "/"; 204 | display = head; 205 | local = Path.GetFileNameWithoutExtension(head) |> replaceBacktickChars}) []) 206 | |> Array.map (mapChildren distinctDisplayNames) 207 | 208 | /// This returns the unique RoseTree matching the query if one is found. 209 | let rec queryRoseTrees (query : 'b list) (f : 'a -> 'b) (roseTrees : RoseTree<'a>[]) : RoseTree<'a> option = 210 | match query with 211 | | [] -> None 212 | | (head::tail) -> 213 | roseTrees 214 | |> Array.tryFind (function Node(x, _) -> f(x) = head) 215 | |> Option.bind ( 216 | function Node(_, xs) as n-> match tail with | [] -> Some(n) | _ -> queryRoseTrees tail f xs) 217 | 218 | module Map = 219 | /// This returns a new Map where the Keys and Values have been swapped. 220 | let invert (x : Map<'a, 'b[]>) = 221 | [| for KeyValue(k, vs) in x do for v in vs do yield (v, k) |] 222 | |> Array.groupBy fst 223 | |> Array.map (fun (x, xs) -> (x, xs |> Array.map snd)) 224 | |> Map 225 | 226 | [] 227 | module TF = 228 | 229 | type DataType with 230 | member this.Name = 231 | match this with 232 | | DataType.DtHalf -> "float16" 233 | | DataType.DtFloat -> "float32" 234 | | DataType.DtDouble -> "float64" 235 | | DataType.DtUint8 -> "uint8" 236 | | DataType.DtUint16 -> "uint16" 237 | | DataType.DtUint32 -> "uint32" 238 | | DataType.DtUint64 -> "uint64" 239 | | DataType.DtInt8 -> "int8" 240 | | DataType.DtInt16 -> "int16" 241 | | DataType.DtInt32 -> "int32" 242 | | DataType.DtInt64 -> "int64" 243 | | DataType.DtString -> "string" 244 | | DataType.DtComplex64 -> "complex64" 245 | | DataType.DtComplex128 -> "complex128" 246 | | DataType.DtBool -> "bool" 247 | | DataType.DtBfloat16 -> "bfloat16" 248 | | DataType.DtResource -> "resource" 249 | | DataType.DtVariant -> "variant" 250 | | DataType.DtHalfRef -> "float16 ref" 251 | | DataType.DtFloatRef -> "float32 ref" 252 | | DataType.DtDoubleRef -> "float64 ref" 253 | | DataType.DtUint8Ref -> "uint8 ref" 254 | | DataType.DtUint16Ref -> "uint16 ref" 255 | | DataType.DtUint32Ref -> "uint32 ref" 256 | | DataType.DtUint64Ref -> "uint64 ref" 257 | | DataType.DtInt8Ref -> "int8 ref" 258 | | DataType.DtInt16Ref -> "int16 ref" 259 | | DataType.DtInt32Ref -> "int32 ref" 260 | | DataType.DtInt64Ref -> "int64 ref" 261 | | DataType.DtStringRef -> "string ref" 262 | | DataType.DtComplex64Ref -> "complex64 ref" 263 | | DataType.DtComplex128Ref -> "complex128 ref" 264 | | DataType.DtBoolRef -> "bool ref" 265 | | DataType.DtBfloat16Ref -> "bfloat16 ref" 266 | | DataType.DtResourceRef -> "resource ref" 267 | | DataType.DtVariantRef -> "variant ref" 268 | | _ -> String.Empty 269 | 270 | member this.ToType = 271 | match this with 272 | | DataType.DtFloat -> typedefof 273 | | DataType.DtDouble -> typedefof 274 | | DataType.DtUint8 -> typedefof 275 | | DataType.DtUint16 -> typedefof 276 | | DataType.DtUint32 -> typedefof 277 | | DataType.DtUint64 -> typedefof 278 | | DataType.DtInt8 -> typedefof 279 | | DataType.DtInt16 -> typedefof 280 | | DataType.DtInt32 -> typedefof 281 | | DataType.DtInt64 -> typedefof 282 | | DataType.DtString -> typedefof // TFString? 283 | | DataType.DtComplex128 -> typedefof 284 | | DataType.DtBool -> typedefof 285 | | DataType.DtFloatRef -> typedefof 286 | | DataType.DtDoubleRef -> typedefof 287 | | DataType.DtUint8Ref -> typedefof 288 | | DataType.DtUint16Ref -> typedefof 289 | | DataType.DtUint32Ref -> typedefof 290 | | DataType.DtUint64Ref -> typedefof 291 | | DataType.DtInt8Ref -> typedefof 292 | | DataType.DtInt16Ref -> typedefof 293 | | DataType.DtInt32Ref -> typedefof 294 | | DataType.DtInt64Ref -> typedefof 295 | | DataType.DtStringRef -> typedefof // TFString? 296 | | DataType.DtComplex128Ref -> typedefof 297 | | DataType.DtBoolRef -> typedefof 298 | | _ -> typedefof 299 | 300 | type TensorShapeProto with 301 | member this.FriendlyName = 302 | if this.UnknownRank then "unknown" 303 | // TODO Consider how to handle named dimensions. 304 | else [| for x in this.Dim -> if x.Size < 0L then "?" else sprintf "%i" x.Size|] 305 | |> String.concat "," |> sprintf "(%s)" 306 | 307 | type INPYTensor = 308 | abstract member dtype : NPYDType with get 309 | abstract member shape : int[] with get 310 | abstract member data : Array with get 311 | abstract member fortran_order : bool with get 312 | abstract member path : string option with get 313 | abstract member name : string option with get 314 | 315 | type NPYTensor(path:string) = 316 | let npy = lazy (NPYReaderWriter.readNumpy(File.ReadAllBytes(path)) |> Result.requireOk) 317 | interface INPYTensor with 318 | member this.dtype : NPYDType = npy.Force() |> fst |> fun x -> x.npyDType 319 | member this.shape : int[] = npy.Force() |> fst |> fun x -> x.shape 320 | member this.data : Array = npy.Force() |> snd 321 | member this.fortran_order : bool = npy.Force() |> fst |> fun x -> x.fortran_order 322 | member this.path = Some(path) 323 | member this.name = None 324 | 325 | type NPZTensor(desc : NPYDescription, array : Array, path : string, name: string) = 326 | interface INPYTensor with 327 | member this.dtype : NPYDType = desc.npyDType 328 | member this.shape : int[] = desc.shape 329 | member this.data : Array = array 330 | member this.fortran_order : bool = desc.fortran_order 331 | member this.path = Some(path) 332 | member this.name = Some(name) 333 | 334 | type SomeRuntimeHelper() = 335 | static member Help() = "help" 336 | 337 | /// Proxies the TFGraph metadata and precomputed mappings 338 | /// This is to minimize memory usage via a static cache 339 | type TFGraphCache(path : string, separator : char) = 340 | let file = File.OpenRead(path) 341 | let mutable graph = Some(GraphDef.Parser.ParseFrom(file)) 342 | let indexMap = Map(graph.Value.Node |> Seq.mapi (fun i x -> (x.Name, i))) 343 | let keys = graph.Value.Node |> Seq.map (fun x -> x.Name) |> Seq.toArray 344 | let inputMap = Map(graph.Value.Node |> Seq.map (fun x -> (x.Name, x.Input |> Seq.toArray))) 345 | let outputMap = inputMap |> Map.invert 346 | let getNodeByName(name : string) = graph.Value.Node |> Seq.find (fun x -> x.Name = name) 347 | let ops = 348 | graph.Value.Node 349 | |> Seq.map (fun x -> (x.Op, x.Name)) 350 | |> Seq.toArray 351 | |> Array.groupBy fst 352 | |> Array.map (fun (x, xs) -> (x, xs |>Array.map snd)) 353 | |> Map 354 | let names = graph.Value.Node |> Seq.map (fun x -> x.Name) |> Seq.toArray 355 | let roseTrees = names |> getDistinctNames separator 356 | let splitPath (x : string) = x.Split([|separator|], StringSplitOptions.RemoveEmptyEntries) 357 | 358 | let expandRoseTree (path : string) (map : Map) : (int*RoseTree*string[])[]= 359 | 360 | /// An attempt at improving useability 361 | //let filterRoot (xs : string[]) = xs |> Array.filter (fun x -> x.Contains(sprintf "%c" '/')) 362 | let expand (keys : string[]) (map : Map) (prefix : string) = 363 | keys 364 | |> Array.filter (fun x -> x.StartsWith(prefix)) 365 | |> Array.collect (fun x -> 366 | match map.TryFind(x) with 367 | | Some(xs) -> xs 368 | | _ -> failwith "the provided map should contain all keys") 369 | |> Array.filter (fun x -> not(x.StartsWith(prefix))) 370 | //|> filterRoot // NOTE: It's often that the Consts are root 371 | let xs = path |> splitPath 372 | [| 373 | let findRelative (xs : string[]) (ys : string[]) = 374 | let zs = 375 | if ys.Length < xs.Length then ys 376 | else 377 | let common = 378 | (xs, ys |> Array.take (xs.Length)) 379 | ||> Array.zip |> Array.takeWhile (fun (x, y) -> x=y) |> Array.map fst 380 | if common.Length < ys.Length 381 | then [| yield! common; yield ys.[common.Length]|] 382 | else common // This shouldn't happen 383 | (zs, zs.Length - xs.Length) 384 | let queryDepthPairs = 385 | [|for target in expand keys map path -> findRelative xs (target |> splitPath)|] 386 | |> Array.groupBy fst |> Array.map (fun (query, xs) -> (query, snd xs.[0])) 387 | for (query, relDepth) in queryDepthPairs do 388 | match queryRoseTrees (query |> Array.toList) (fun (x : TreeNames) -> x.local) roseTrees with 389 | | Some(rt) -> yield (relDepth, rt, query) 390 | | _ -> () 391 | |] 392 | 393 | let attrs = [|for x in graph.Value.Node -> 394 | [|for KeyValue(k, v) in x.Attr do 395 | if v.ValueCase <> AttrValue.ValueOneofCase.Tensor then yield (k, v)|]|] 396 | 397 | let tensors = [|for x in graph.Value.Node -> 398 | [|for KeyValue(k, v) in x.Attr do 399 | if v.ValueCase = AttrValue.ValueOneofCase.Tensor then yield (k, (v.Tensor.Dtype, v.Tensor.TensorShape))|]|] 400 | do 401 | graph <- None 402 | file.Dispose() 403 | member this.GetAttr(index : int) = attrs.[index] 404 | member this.GetTensors(index : int) = tensors.[index] 405 | member this.IndexMap = indexMap 406 | member this.InputMap = inputMap 407 | member this.OutputMap = outputMap 408 | member this.GetNodeByName(name) = getNodeByName(name) 409 | member this.Keys = keys 410 | member this.Ops = ops 411 | member this.Names = names 412 | member this.RoseTrees = roseTrees 413 | member this.SplitPath(path) = splitPath(path) 414 | member this.ExpandRoseTree (path : string) (map : Map) = expandRoseTree path map 415 | 416 | /// A clearable cache which requires access to provide an intilization 417 | type Cache<'k, 'v when 'k : comparison>() = 418 | let mutable cache = Map.empty<'k, 'v> 419 | member this.TryFetch(key : 'k, f : unit->'v) : 'v = 420 | cache.TryFind(key) |> Option.defaultWith (fun () -> f() |> fun v -> cache <- cache.Add(key, v); v ) 421 | member this.Clear() = cache <- Map.empty 422 | 423 | [] 424 | type TFProviderA (config : TypeProviderConfig) as this = 425 | inherit TypeProviderForNamespaces (config, addDefaultProbingLocation=true) 426 | let ns = "Tensorflow.FSharp" 427 | let asm = Assembly.GetExecutingAssembly() 428 | /// This approximately applies TensorFlow rules for finding a unique name 429 | let makeUnique = getMakeUnique() 430 | static let mutable tfGraphCache = Cache() 431 | static let mutable npyCache = Cache() 432 | static let mutable npzCache = Cache>() 433 | 434 | // check we contain a copy of runtime files, and are not referencing the runtime DLL 435 | do assert (typeof.Assembly.GetName().Name = asm.GetName().Name) 436 | 437 | // NOTE: Should be possible to have a general method which takes in a shape of int[] and returns the correctly shaped array 438 | let mis = 439 | let rec getMethodInfo expr = 440 | match expr with 441 | | Call(_, mi, _) -> Some(mi) 442 | | Lambda(_, expr) -> getMethodInfo expr 443 | | _ -> None 444 | [| <@@ Array2D.ofArray @@>; <@@ Array3D.ofArray @@>; <@@ Array4D.ofArray @@> |] 445 | |> Array.map (fun x -> (x |> getMethodInfo |> Option.get).DeclaringType.GetMember("ofArray").[0] :?> MethodInfo) 446 | 447 | /// This adds a 'Values' property to the type definition with the type and shape derived from the NPYDesciption 448 | let addValues(myType : ProvidedTypeDefinition, desc) = 449 | let rank = desc.shape.Length 450 | match desc.npyDType.TryToType() with 451 | | None -> () 452 | | Some(t) -> 453 | let getTypedData(args : Expr list) = 454 | match rank with 455 | | 2 | 3 | 4-> 456 | let shape = desc.shape 457 | Expr.Call(mis.[rank-2].MakeGenericMethod(t), 458 | [yield! [for r in 0..rank - 1 -> Expr.Value(shape.[r])] 459 | yield <@@ (%%args.[0] : INPYTensor).data @@>]) 460 | | 1 -> <@@ (%%args.[0] : INPYTensor).data @@> 461 | | _ -> failwithf "usupported rank %i" rank 462 | let pp = ProvidedProperty("Values", t.MakeArrayType(rank), getterCode = getTypedData) 463 | pp.AddXmlDoc(sprintf "%O" desc) 464 | myType.AddMember(pp) 465 | 466 | /// Root Type 467 | let t = 468 | let t = ProvidedTypeDefinition(asm, ns, "TFProvider", Some typeof, isErased=true) 469 | t.DefineStaticParameters( [ 470 | ProvidedStaticParameter("Path", typeof) 471 | ProvidedStaticParameter("Separator", typeof,'/') 472 | ] ,(fun typeName args -> 473 | match args with 474 | | [| :? string as path; :? char as separator |] -> 475 | if not(File.Exists(path)) then failwithf "File %s not found" path 476 | else 477 | match System.IO.Path.GetExtension(path).ToLower() with 478 | | ".npy" -> 479 | // NPY files are quite simple with only one Tensor 480 | let myType = ProvidedTypeDefinition(asm, ns, typeName, Some typeof, isErased=true) 481 | myType.AddMember(ProvidedConstructor([], invokeCode = (fun _ -> <@@ NPYTensor(path) :> INPYTensor @@>))) 482 | let getDesc path = File.ReadAllBytes(path) |> NPYReaderWriter.readNumpy |> Result.requireOk |> fst 483 | let desc = npyCache.TryFetch(path, fun () -> getDesc path) 484 | addValues(myType, desc) 485 | myType.AddXmlDoc(sprintf "%O" desc) 486 | myType 487 | | ".npz" -> 488 | // NPZ files are zip files where each tensor has a name 489 | // It is common but not required that the names for a filesytem style hierarchy 490 | let myType = ProvidedTypeDefinition(asm, ns, typeName, Some typeof>, isErased=true) 491 | let expr = <@@ 492 | File.ReadAllBytes(path) 493 | |> NPYReaderWriter.readFromNPZ 494 | |> Map.map (fun k (desc, arr) -> NPZTensor(desc, arr, path, k) :> INPYTensor) 495 | @@> 496 | let ctor = ProvidedConstructor([], invokeCode = (fun args -> expr)) 497 | myType.AddMember(ctor) 498 | let getDescMap path = 499 | File.ReadAllBytes(path) 500 | |> NPYReaderWriter.readFromNPZ 501 | |> Map.map (fun k (v, _) -> v) 502 | let descMap = npzCache.TryFetch(path, fun () -> getDescMap path) 503 | let names = [|for KeyValue(k, _) in descMap -> k|] 504 | let roseTrees = names |> getDistinctNames separator 505 | /// This recursivly builds the tree structure object model 506 | let rec procNodes (myType : ProvidedTypeDefinition) (nodes : RoseTree[]) = 507 | for node in nodes do 508 | match node with 509 | // leaf nodes are of type Tensor which has the 'Values' attribute added for typed access to the array 510 | | Node(x, [||]) -> 511 | let xFull = x.full 512 | let xDisplay = Path.GetFileNameWithoutExtension(x.display) 513 | let subType = ProvidedTypeDefinition(asm, ns, xDisplay, Some typeof, isErased=true) 514 | let pp = ProvidedProperty(xDisplay, subType, getterCode = 515 | (fun args -> <@@ (%%args.[0] : Map).Item(xFull) @@>)) 516 | pp.AddXmlDoc(sprintf "%s %O" x.local descMap.[x.full]) 517 | addValues(subType, descMap.[x.full]) 518 | myType.AddMembers([subType :> MemberInfo; pp :> MemberInfo]) 519 | // the root and inner nodes return a scoped Map of tensors 520 | // sub innder nodes and root nodes are recursively added to this node 521 | | Node(x, xs) -> 522 | let xFull = x.full 523 | let subType = ProvidedTypeDefinition(asm, ns, x.display, Some typeof>, isErased=true) 524 | procNodes subType xs 525 | myType.AddMember(subType) 526 | let f (args : Expr list) = 527 | <@@ 528 | (%%args.[0] : Map) 529 | |> Map.filter (fun k _ -> k.StartsWith(xFull)) 530 | @@> 531 | let pp = ProvidedProperty(x.display, subType, getterCode = f) 532 | pp.AddXmlDoc(sprintf "%s" x.full) 533 | myType.AddMember(pp) 534 | procNodes myType roseTrees 535 | myType 536 | | ".pb" -> 537 | let cache = tfGraphCache.TryFetch((path, separator), fun () -> TFGraphCache(path, separator)) 538 | let myType = ProvidedTypeDefinition(asm, ns, typeName, Some typeof, isErased=true) 539 | myType.AddMember(ProvidedConstructor([], invokeCode = (fun _ -> <@@ GraphDef.Parser.ParseFrom(File.ReadAllBytes(path)) @@>))) 540 | 541 | /// helper function returns itself (a boxed object) as a named member. 542 | /// This is used to represent a hierarchial structure without changing the underlying type 543 | let recPropGraph(name) = 544 | let ptd = ProvidedTypeDefinition(asm, ns, makeUnique(name), Some typeof, isErased=true, hideObjectMethods=true) 545 | let pp = ProvidedProperty(name, ptd, getterCode = (fun args -> <@@ box(%%args.[0]) @@>)) 546 | (ptd, pp) 547 | 548 | /// helper function to box GraphDef type to hide the types members 549 | let recPropGraphDefToObj(name) = 550 | let ptd = ProvidedTypeDefinition(asm, ns, makeUnique(name), Some typeof, isErased=true, hideObjectMethods=true) 551 | let pp = ProvidedProperty(name, ptd, getterCode = (fun args -> <@@ box(%%args.[0] : GraphDef) @@>)) 552 | (ptd, pp) 553 | 554 | /// helper function to box NodeDef type to hide the types members 555 | let recPropNodeDefToObj(name) = 556 | let ptd = ProvidedTypeDefinition(asm, ns, makeUnique(name), Some typeof, isErased=true, hideObjectMethods=true) 557 | let pp = ProvidedProperty(name, ptd, getterCode = (fun args -> <@@ box(%%args.[0] : NodeDef) @@>)) 558 | (ptd, pp) 559 | 560 | /// Provides a Leaf Node definition with the attributes added to an AttrList member 561 | let rec nodeFromGraph(displayName : string, fullName : string) = 562 | let index = cache.IndexMap.[fullName] 563 | let ptd = ProvidedTypeDefinition(asm, ns, makeUnique(displayName), Some typeof, isErased=true, hideObjectMethods=true) 564 | let pp = ProvidedProperty(displayName, ptd, getterCode = (fun args -> <@@ ((%%args.[0] : obj) :?> GraphDef).Node.[index] @@>)) 565 | ptd.AddMembersDelayed(fun () -> 566 | let ptd = ProvidedTypeDefinition(asm, ns, "AttrList", Some typeof, isErased=true, hideObjectMethods=true) 567 | let pp = ProvidedProperty("AttrList", ptd, getterCode = (fun args -> <@@ box(%%args.[0] : NodeDef) @@>)) 568 | ptd.AddMembersDelayed(fun () -> [ 569 | for (k, v) in cache.GetAttr(index) do 570 | match v.ValueCase with 571 | | AttrValue.ValueOneofCase.B -> 572 | yield ProvidedProperty(k, typeof, getterCode = 573 | fun args -> <@@ ((%%args.[0] : obj) :?> NodeDef).Attr.[k].B @@>) 574 | |> fun x -> x.AddXmlDoc(sprintf "bool: %b" v.B); x :> MemberInfo 575 | | AttrValue.ValueOneofCase.F -> 576 | yield ProvidedProperty(k, typeof, getterCode = 577 | fun args -> <@@ ((%%args.[0] : obj) :?> NodeDef).Attr.[k].F @@>) 578 | |> fun x -> x.AddXmlDoc(sprintf "float32: %f" v.F); x :> MemberInfo 579 | | AttrValue.ValueOneofCase.Func -> 580 | //v.Func.Name 581 | //v.Func.Attr 582 | () // NOTE: Typed access to the Func attribute in this manor is not expected to be useful 583 | | AttrValue.ValueOneofCase.List -> 584 | if v.List.Type.Count > 0 then 585 | yield ProvidedProperty(k, typeof, getterCode = 586 | fun args -> <@@ ((%%args.[0] : obj) :?> NodeDef).Attr.[k].List.Type |> Seq.toList @@>) :> MemberInfo 587 | elif v.List.B.Count > 0 then 588 | yield ProvidedProperty(k, typeof, getterCode = 589 | fun args -> <@@ ((%%args.[0] : obj) :?> NodeDef).Attr.[k].List.B |> Seq.toList @@>) :> MemberInfo 590 | elif v.List.F.Count > 0 then 591 | yield ProvidedProperty(k, typeof, getterCode = 592 | fun args -> <@@ ((%%args.[0] : obj) :?> NodeDef).Attr.[k].List.F |> Seq.toList @@>) :> MemberInfo 593 | elif v.List.S.Count > 0 then 594 | yield ProvidedProperty(k, typeof, getterCode = 595 | fun args -> <@@ ((%%args.[0] : obj) :?> NodeDef).Attr.[k].List.S |> Seq.toList @@>) :> MemberInfo 596 | elif v.List.Shape.Count > 0 then 597 | yield ProvidedProperty(k, typeof, getterCode = 598 | fun args -> <@@ ((%%args.[0] : obj) :?> NodeDef).Attr.[k].List.Shape |> Seq.toList @@>) :> MemberInfo 599 | elif v.List.I.Count > 0 then 600 | yield ProvidedProperty(k, typeof, getterCode = 601 | fun args -> <@@ ((%%args.[0] : obj) :?> NodeDef).Attr.[k].List.I |> Seq.toList @@>) :> MemberInfo 602 | elif v.List.Tensor.Count > 0 then 603 | yield ProvidedProperty(k, typeof, getterCode = 604 | fun args -> <@@ ((%%args.[0] : obj) :?> NodeDef).Attr.[k].List.Tensor |> Seq.toList @@>) :> MemberInfo 605 | elif v.List.Func.Count > 0 then () 606 | else () // ignore 607 | | AttrValue.ValueOneofCase.Placeholder -> 608 | yield ProvidedProperty(k, typeof, getterCode = 609 | fun args -> <@@ ((%%args.[0] : obj) :?> NodeDef).Attr.[k].Placeholder @@>) 610 | |> fun x -> x.AddXmlDoc(sprintf "Placeholder: %s" v.Placeholder); x :> MemberInfo 611 | | AttrValue.ValueOneofCase.I -> 612 | yield ProvidedProperty(k, typeof, getterCode = 613 | fun args -> <@@ ((%%args.[0] : obj) :?> NodeDef).Attr.[k].I@@>) 614 | |> fun x -> x.AddXmlDoc(sprintf "int64: %i" v.I); x :> MemberInfo 615 | | AttrValue.ValueOneofCase.None -> () 616 | | AttrValue.ValueOneofCase.S -> 617 | yield ProvidedProperty(k, typeof, getterCode = 618 | fun args -> <@@ ((%%args.[0] : obj) :?> NodeDef).Attr.[k].S@@>) 619 | |> fun x -> 620 | // Truncation is required here as these byte strings can be very long in the case of image data 621 | // We estimate that most human readable strings used in attributes are under 400 characters 622 | x.AddXmlDoc(sprintf "ByteString: %s" (v.S.ToStringUtf8() |> String.truncate 400)); x :> MemberInfo 623 | | AttrValue.ValueOneofCase.Shape -> 624 | yield ProvidedProperty(k, typeof, getterCode = 625 | fun args -> <@@ ((%%args.[0] : obj) :?> NodeDef).Attr.[k].Shape@@>) 626 | |> fun x -> x.AddXmlDoc(sprintf "Shape: %s" (v.Shape.FriendlyName)); x :> MemberInfo 627 | | AttrValue.ValueOneofCase.Tensor -> 628 | failwith "This should not happen as it is filtered out at an earlier stage" 629 | | AttrValue.ValueOneofCase.Type -> 630 | yield ProvidedProperty(k, typeof, getterCode = 631 | fun args -> <@@ ((%%args.[0] : obj) :?> NodeDef).Attr.[k].Type@@>) 632 | |> fun x -> x.AddXmlDoc(sprintf "DataType: %s" (v.Type.Name)); x :> MemberInfo 633 | | _ -> () //failwith "error" 634 | for (k, (dt, shape)) in cache.GetTensors(index) do 635 | let ptd = ProvidedTypeDefinition(asm, ns, "Tensor", Some typeof, isErased=true) 636 | yield ptd :> MemberInfo 637 | let docString = sprintf "Tensor: %s %s" (dt.Name) (shape.FriendlyName) 638 | yield ProvidedProperty(k, ptd, getterCode = 639 | fun args -> <@@ ((%%args.[0] : obj) :?> NodeDef).Attr.[k].Tensor@@>) 640 | |> fun x -> x.AddXmlDoc(docString); x :> MemberInfo 641 | // Add Extra "Values" member if we can 642 | if not(shape.UnknownRank) && not(shape.Dim |> Seq.exists (fun d -> d.Size < 0L)) then 643 | let dims = shape.Dim |> Seq.toArray |> Array.map (fun d -> int d.Size) 644 | //let size = dims |> Array.fold (*) 1 645 | let rank = dims.Length 646 | let t = dt.ToType 647 | if t <> typeof then 648 | let getTypedData(args : Expr list) = 649 | let arrExpr = <@@ parseData((%%args.[0] : TensorProto).TensorContent.ToByteArray(), dt) @@> 650 | match rank with 651 | | 2 | 3 | 4-> 652 | Expr.Call(mis.[rank-2].MakeGenericMethod(t), 653 | [yield! [for r in 0..rank - 1 -> Expr.Value(dims.[r])]; 654 | yield arrExpr]) 655 | | 1 -> arrExpr 656 | | _ -> failwithf "usupported rank %i" rank 657 | let pp = ProvidedProperty("Values", t.MakeArrayType(rank), getterCode = getTypedData) 658 | pp.AddXmlDoc(docString) 659 | ptd.AddMember(pp) 660 | ]) 661 | 662 | // TODO If we want to return to the Graph we must keep a reference to the graph which means 663 | // proxying the NodeDef type 664 | // Feedback on usage is needed to see if the added complexity of navigation though Input and Outputs members will not confuse end users 665 | // let f (name : string) (map : Map) : MemberInfo[] = 666 | // let (subTypeDef, subPropDef) = recPropGraph(name) 667 | // subTypeDef.AddMembersDelayed(fun () -> [ 668 | // //let full = query |> String.concat (sprintf "%c" separator) 669 | // let full = fullName 670 | // for (depth, rt, query) in cache.ExpandRoseTree full map do 671 | // let depthPrefix = if depth = 0 then "" else "../" |> String.replicate -depth 672 | // match rt with 673 | // | RoseTree.Node(x, [||]) -> 674 | // yield! nodeFromGraph(depthPrefix + x.display, x.full) 675 | // | RoseTree.Node(x, xs) -> 676 | // yield! (threadGraphDefThroughRoseTree false query (depthPrefix + x.display) xs) 677 | // ]) 678 | // [|subTypeDef :> MemberInfo; subPropDef :> MemberInfo|] 679 | [ 680 | yield ptd :> MemberInfo 681 | yield upcast pp 682 | // yield! f "Inputs" cache.InputMap 683 | // yield! f "Outputs" cache.OutputMap 684 | ]) 685 | [|ptd :> MemberInfo; upcast pp|] 686 | /// This creates either a root node or inner node to the hierarchial structure 687 | /// The hierarchial structure can be navigatied starting at the root nodes, up and down the inner nodes, and terminating at the leaf nodes 688 | and threadGraphDefThroughRoseTree (isFirst : bool) (query : string[]) (name : string) (rts : RoseTree[]) = 689 | let (ptd, pp) = 690 | if isFirst then recPropGraphDefToObj(name) else recPropGraph(name) 691 | ptd.AddMembersDelayed(fun () -> [ 692 | for rt in rts do 693 | match rt with 694 | | RoseTree.Node(x, [||]) -> 695 | yield! nodeFromGraph(x.display, x.full) 696 | | RoseTree.Node(x, xs) -> 697 | yield! (threadGraphDefThroughRoseTree false [|yield! query; yield x.local|] x.display xs) 698 | let f (name : string) (map : Map) : MemberInfo[] = 699 | let (subTypeDef, subPropDef) = recPropGraph(name) 700 | subTypeDef.AddMembersDelayed(fun () -> [ 701 | let full = query |> String.concat (sprintf "%c" separator) 702 | for (depth, rt, query) in cache.ExpandRoseTree full map do 703 | let depthPrefix = if depth = 0 then "" else "../" |> String.replicate -depth 704 | match rt with 705 | | RoseTree.Node(x, [||]) -> 706 | yield! nodeFromGraph(depthPrefix + x.display, x.full) 707 | | RoseTree.Node(x, xs) -> 708 | yield! (threadGraphDefThroughRoseTree false query (depthPrefix + x.display) xs) 709 | ]) 710 | [|subTypeDef :> MemberInfo; subPropDef :> MemberInfo|] 711 | yield! f "Inputs" cache.InputMap 712 | yield! f "Outputs" cache.OutputMap 713 | ]) 714 | [|ptd :> MemberInfo; pp :> MemberInfo|] 715 | myType.AddMembers( 716 | /// All Nodes list all of the nodes by their full name. 717 | let (allNodesType, allNodesProperty) = recPropGraphDefToObj("AllNodes") 718 | allNodesType.AddMembers([for name in cache.Names do yield! nodeFromGraph(name, name)]) 719 | /// This Groups all of the nodes byte their OpType 720 | let (allOpsType, allOpsProperty) = recPropGraphDefToObj("AllOps") 721 | allOpsType.AddMembers([ 722 | for KeyValue(op, names) in cache.Ops do 723 | let (opType, opProperty) = recPropGraph(op) 724 | yield opType :> MemberInfo 725 | yield opProperty :> MemberInfo 726 | opType.AddMembers([ for name in names do yield! nodeFromGraph(name, name)]) 727 | ]) 728 | [ 729 | yield allNodesType :> MemberInfo 730 | yield upcast allNodesProperty 731 | yield upcast allOpsType 732 | yield upcast allOpsProperty 733 | yield! threadGraphDefThroughRoseTree true [||] "RootNodes" cache.RoseTrees 734 | ] 735 | ) 736 | myType 737 | | ext -> failwithf "Unsupported file extension %s" ext 738 | | _ -> failwith "Unsupported arguments" 739 | )) 740 | t 741 | do this.AddNamespace(ns, [t]) 742 | 743 | [] 744 | do () 745 | 746 | -------------------------------------------------------------------------------- /TensorflowTypeProvider/TensorflowTypeProvider/TensorflowTypeProvider.fsproj: -------------------------------------------------------------------------------- 1 |  2 | 3 | 4 | netstandard2.0 5 | true 6 | Matthew Moloney 7 | 8 | Tensorflow Type Provider 9 | Provides typed access to Operations in a Tensorflow Graph in F#. This is useful for transfer learning and other operations which would typically require 'magic strings' that are often hard to find. 10 | Tensorflow FSharp TypeProvider 11 | Apache-2.0 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | -------------------------------------------------------------------------------- /TensorflowTypeProviderTest/TensorflowTypeProviderTest.sln: -------------------------------------------------------------------------------- 1 |  2 | Microsoft Visual Studio Solution File, Format Version 12.00 3 | # Visual Studio Version 16 4 | VisualStudioVersion = 16.0.29326.143 5 | MinimumVisualStudioVersion = 10.0.40219.1 6 | Project("{F2A71F9B-5D33-465A-A702-920D77279786}") = "TensorflowTypeProviderTest", "TensorflowTypeProviderTest\TensorflowTypeProviderTest.fsproj", "{71315965-7B62-4DE8-870E-0C71985A0431}" 7 | EndProject 8 | Global 9 | GlobalSection(SolutionConfigurationPlatforms) = preSolution 10 | Debug|Any CPU = Debug|Any CPU 11 | Release|Any CPU = Release|Any CPU 12 | EndGlobalSection 13 | GlobalSection(ProjectConfigurationPlatforms) = postSolution 14 | {71315965-7B62-4DE8-870E-0C71985A0431}.Debug|Any CPU.ActiveCfg = Debug|Any CPU 15 | {71315965-7B62-4DE8-870E-0C71985A0431}.Debug|Any CPU.Build.0 = Debug|Any CPU 16 | {71315965-7B62-4DE8-870E-0C71985A0431}.Release|Any CPU.ActiveCfg = Release|Any CPU 17 | {71315965-7B62-4DE8-870E-0C71985A0431}.Release|Any CPU.Build.0 = Release|Any CPU 18 | EndGlobalSection 19 | GlobalSection(SolutionProperties) = preSolution 20 | HideSolutionNode = FALSE 21 | EndGlobalSection 22 | GlobalSection(ExtensibilityGlobals) = postSolution 23 | SolutionGuid = {E568E90F-3DFB-48E0-94FB-E0903790AF7C} 24 | EndGlobalSection 25 | EndGlobal 26 | -------------------------------------------------------------------------------- /TensorflowTypeProviderTest/TensorflowTypeProviderTest/App.config: -------------------------------------------------------------------------------- 1 |  2 | 3 | 4 | 5 | 6 | -------------------------------------------------------------------------------- /TensorflowTypeProviderTest/TensorflowTypeProviderTest/AssemblyInfo.fs: -------------------------------------------------------------------------------- 1 | namespace TensorflowTypeProviderTest.AssemblyInfo 2 | 3 | open System.Reflection 4 | open System.Runtime.CompilerServices 5 | open System.Runtime.InteropServices 6 | 7 | // General Information about an assembly is controlled through the following 8 | // set of attributes. Change these attribute values to modify the information 9 | // associated with an assembly. 10 | [] 11 | [] 12 | [] 13 | [] 14 | [] 15 | [] 16 | [] 17 | [] 18 | 19 | // Setting ComVisible to false makes the types in this assembly not visible 20 | // to COM components. If you need to access a type in this assembly from 21 | // COM, set the ComVisible attribute to true on that type. 22 | [] 23 | 24 | // The following GUID is for the ID of the typelib if this project is exposed to COM 25 | [] 26 | 27 | // Version information for an assembly consists of the following four values: 28 | // 29 | // Major Version 30 | // Minor Version 31 | // Build Number 32 | // Revision 33 | // 34 | // You can specify all the values or you can default the Build and Revision Numbers 35 | // by using the '*' as shown below: 36 | // [] 37 | [] 38 | [] 39 | 40 | do 41 | () -------------------------------------------------------------------------------- /TensorflowTypeProviderTest/TensorflowTypeProviderTest/Program.fs: -------------------------------------------------------------------------------- 1 | // Learn more about F# at https://fsharp.org 2 | // See the 'F# Tutorial' project for more help. 3 | 4 | [] 5 | let main argv = 6 | printfn "%A" argv 7 | 0 // return an integer exit code 8 | -------------------------------------------------------------------------------- /TensorflowTypeProviderTest/TensorflowTypeProviderTest/TensorflowTypeProviderTest.fsproj: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | Debug 6 | AnyCPU 7 | 2.0 8 | 71315965-7b62-4de8-870e-0c71985a0431 9 | Exe 10 | TensorflowTypeProviderTest 11 | TensorflowTypeProviderTest 12 | v4.7.2 13 | true 14 | true 15 | 3239;$(WarningsAsErrors) 16 | TensorflowTypeProviderTest 17 | 18 | 19 | true 20 | full 21 | false 22 | false 23 | bin\$(Configuration)\ 24 | DEBUG;TRACE 25 | 3 26 | AnyCPU 27 | bin\$(Configuration)\$(AssemblyName).XML 28 | true 29 | 30 | 31 | pdbonly 32 | true 33 | true 34 | bin\$(Configuration)\ 35 | TRACE 36 | 3 37 | AnyCPU 38 | bin\$(Configuration)\$(AssemblyName).XML 39 | true 40 | 41 | 42 | 11 43 | 44 | 45 | $(MSBuildExtensionsPath32)\Microsoft\VisualStudio\v$(VisualStudioVersion)\FSharp\Microsoft.FSharp.Targets 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | ..\packages\FSharp.Core.4.7.0\lib\net45\FSharp.Core.dll 58 | 59 | 60 | 61 | 62 | 63 | 64 | True 65 | 66 | 67 | 74 | -------------------------------------------------------------------------------- /TensorflowTypeProviderTest/TensorflowTypeProviderTest/Test.fsx: -------------------------------------------------------------------------------- 1 | #r "netstandard" 2 | #r @"..\lib\System.Reflection.dll" 3 | #r @"..\lib\System.Runtime.dll" 4 | #r @"..\lib\Google.Protobuf.dll" 5 | #r @"..\lib\TensorFlow.NET.dll" 6 | #r @"..\lib\TensorflowTypeProvider.dll" 7 | 8 | let [] testData = __SOURCE_DIRECTORY__ + @"\..\..\TestData\" 9 | 10 | open Tensorflow.FSharp 11 | 12 | 13 | let [] float_8x2Npy = testData + "float32_8x2.npy" 14 | let [] float_16Npy = testData + "float32_16.npy" 15 | let [] intNpy = testData + "int32.npy" 16 | 17 | type NPY = TFProvider 18 | let npy = NPY() 19 | let v = npy.Values.[0] + 10.f 20 | 21 | 22 | 23 | 24 | let [] testDataNpz = testData + "test_data.npz" 25 | type TestDataNPZ = TFProvider 26 | let y = TestDataNPZ() 27 | [for KeyValue(k,v) in y -> k] 28 | [for KeyValue(k,v) in y.float32 -> k] 29 | y.float32.shape_16.Values.[0] 30 | y.float32.shape_8x2.Values.[0,0] 31 | 32 | 33 | 34 | let [] fashionTFGraph = testData + "FashionMNIST\FashionFrozen.pb" 35 | type FashionTFGraph = TFProvider 36 | let z = FashionTFGraph() 37 | z.AllNodes.``conv2d/bias``.AttrList.value.Values 38 | z.AllOps.Const.``conv2d/bias``.AttrList.value.Values 39 | let conv2d = z.AllNodes.``conv2d/Conv2D`` 40 | conv2d.AttrList.data_format 41 | conv2d.AttrList.padding 42 | 43 | 44 | 45 | -------------------------------------------------------------------------------- /TensorflowTypeProviderTest/TensorflowTypeProviderTest/packages.config: -------------------------------------------------------------------------------- 1 |  2 | 3 | 4 | 5 | -------------------------------------------------------------------------------- /TensorflowTypeProviderTest/lib/Google.Protobuf.dll: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fsprojects/TensorflowTypeProvider/8802bea0dbd356355e3f310b5d5c101e2b3eb7e8/TensorflowTypeProviderTest/lib/Google.Protobuf.dll -------------------------------------------------------------------------------- /TensorflowTypeProviderTest/lib/System.Reflection.dll: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fsprojects/TensorflowTypeProvider/8802bea0dbd356355e3f310b5d5c101e2b3eb7e8/TensorflowTypeProviderTest/lib/System.Reflection.dll -------------------------------------------------------------------------------- /TensorflowTypeProviderTest/lib/System.Runtime.dll: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fsprojects/TensorflowTypeProvider/8802bea0dbd356355e3f310b5d5c101e2b3eb7e8/TensorflowTypeProviderTest/lib/System.Runtime.dll -------------------------------------------------------------------------------- /TensorflowTypeProviderTest/lib/TensorFlow.NET.dll: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fsprojects/TensorflowTypeProvider/8802bea0dbd356355e3f310b5d5c101e2b3eb7e8/TensorflowTypeProviderTest/lib/TensorFlow.NET.dll -------------------------------------------------------------------------------- /TensorflowTypeProviderTest/lib/TensorflowTypeProvider.dll: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fsprojects/TensorflowTypeProvider/8802bea0dbd356355e3f310b5d5c101e2b3eb7e8/TensorflowTypeProviderTest/lib/TensorflowTypeProvider.dll -------------------------------------------------------------------------------- /TensorflowTypeProviderTest/lib/netstandard.dll: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fsprojects/TensorflowTypeProvider/8802bea0dbd356355e3f310b5d5c101e2b3eb7e8/TensorflowTypeProviderTest/lib/netstandard.dll -------------------------------------------------------------------------------- /TestData/FashionMNIST/FashionFrozen.pb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fsprojects/TensorflowTypeProvider/8802bea0dbd356355e3f310b5d5c101e2b3eb7e8/TestData/FashionMNIST/FashionFrozen.pb -------------------------------------------------------------------------------- /TestData/FashionMNIST/checkpoint: -------------------------------------------------------------------------------- 1 | model_checkpoint_path: "model.ckpt" 2 | all_model_checkpoint_paths: "model.ckpt" 3 | -------------------------------------------------------------------------------- /TestData/FashionMNIST/model.ckpt.data-00000-of-00001: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fsprojects/TensorflowTypeProvider/8802bea0dbd356355e3f310b5d5c101e2b3eb7e8/TestData/FashionMNIST/model.ckpt.data-00000-of-00001 -------------------------------------------------------------------------------- /TestData/FashionMNIST/model.ckpt.index: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fsprojects/TensorflowTypeProvider/8802bea0dbd356355e3f310b5d5c101e2b3eb7e8/TestData/FashionMNIST/model.ckpt.index -------------------------------------------------------------------------------- /TestData/FashionMNIST/model.ckpt.meta: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fsprojects/TensorflowTypeProvider/8802bea0dbd356355e3f310b5d5c101e2b3eb7e8/TestData/FashionMNIST/model.ckpt.meta -------------------------------------------------------------------------------- /TestData/float32.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fsprojects/TensorflowTypeProvider/8802bea0dbd356355e3f310b5d5c101e2b3eb7e8/TestData/float32.npy -------------------------------------------------------------------------------- /TestData/float32_16.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fsprojects/TensorflowTypeProvider/8802bea0dbd356355e3f310b5d5c101e2b3eb7e8/TestData/float32_16.npy -------------------------------------------------------------------------------- /TestData/float32_8x2.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fsprojects/TensorflowTypeProvider/8802bea0dbd356355e3f310b5d5c101e2b3eb7e8/TestData/float32_8x2.npy -------------------------------------------------------------------------------- /TestData/int32.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fsprojects/TensorflowTypeProvider/8802bea0dbd356355e3f310b5d5c101e2b3eb7e8/TestData/int32.npy -------------------------------------------------------------------------------- /TestData/test_data.npz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fsprojects/TensorflowTypeProvider/8802bea0dbd356355e3f310b5d5c101e2b3eb7e8/TestData/test_data.npz -------------------------------------------------------------------------------- /TestData/test_data.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fsprojects/TensorflowTypeProvider/8802bea0dbd356355e3f310b5d5c101e2b3eb7e8/TestData/test_data.zip -------------------------------------------------------------------------------- /images/NPY.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fsprojects/TensorflowTypeProvider/8802bea0dbd356355e3f310b5d5c101e2b3eb7e8/images/NPY.gif -------------------------------------------------------------------------------- /images/NPZ.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fsprojects/TensorflowTypeProvider/8802bea0dbd356355e3f310b5d5c101e2b3eb7e8/images/NPZ.gif -------------------------------------------------------------------------------- /images/TF.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fsprojects/TensorflowTypeProvider/8802bea0dbd356355e3f310b5d5c101e2b3eb7e8/images/TF.gif --------------------------------------------------------------------------------