├── LICENSE ├── README.md ├── docs └── TmStorage.docx ├── master ├── LICENSE ├── README.md ├── Storage │ ├── Common.cs │ ├── Exceptions.cs │ ├── MasterStream.cs │ ├── Properties │ │ └── AssemblyInfo.cs │ ├── Segment.cs │ ├── SegmentExtent.cs │ ├── Storage.cs │ ├── StorageMetadata.cs │ ├── StorageStatistics.cs │ ├── StorageStream.cs │ ├── StorageStreamMetadata.cs │ ├── StreamTable.cs │ ├── StreamTableMap.cs │ ├── TmStorage.csproj │ ├── Tools.cs │ ├── Transactions │ │ ├── Segment.cs │ │ ├── SegmentMetadata.cs │ │ ├── TransactionLogStreamMetadata.cs │ │ └── TransactionStream.cs │ └── WriteBufferedStream.cs └── TmStorage.sln └── test_apps └── SimpleImageStorage ├── App.xaml ├── App.xaml.cs ├── ControlCommand.cs ├── Lib ├── Fluent.dll └── Images │ ├── Add.ico │ ├── Close.ico │ ├── Create.ico │ ├── Delete.ico │ ├── Open.ico │ └── Replace.ico ├── MainWindow.xaml ├── MainWindow.xaml.cs ├── ProgressDialog.xaml ├── ProgressDialog.xaml.cs ├── Properties ├── AssemblyInfo.cs ├── Resources.Designer.cs ├── Resources.resx ├── Settings.Designer.cs └── Settings.settings ├── SimpleImageStorage.csproj ├── SimpleImageStorage.sln └── app.config /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2014 tomazk8 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | 23 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | TmStorage 2 | ========= 3 | 4 | Storage engine (virtual file system) for .NET 5 | 6 | TmStorage is a virtual file system written in .NET. Within it, streams of data can be stored. TmStorage allocates data when 7 | streams are enlarged and deallocates it when they are shrank. TmStorage storage stores streams in a flat structure where 8 | each stream is referenced by stream Id which is of type GUID. 9 | 10 | TmStorage supports full transactions. In future it will also support caching and snapshots 11 | 12 | It can be used in the simple form as it is, or, because of it's simple flat structure, a more complex storage engines or 13 | file systems with custom internal structure can be developed. Such derived storages or file systems automatically get all 14 | the features in TmStorage (transactions, caching, snapshots...). 15 | -------------------------------------------------------------------------------- /docs/TmStorage.docx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tomazk8/TmStorage/5d72efe5fad3c78e6519ebc2eee71f4c9cad6876/docs/TmStorage.docx -------------------------------------------------------------------------------- /master/LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2012 Tomaz Koritnik 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | 23 | -------------------------------------------------------------------------------- /master/README.md: -------------------------------------------------------------------------------- 1 | TmStorage 2 | ========= 3 | 4 | Storage engine (virtual file system) for .NET 5 | 6 | TmStorage is a virtual file system written in .NET. Within it, streams of data can be stored. TmStorage allocates data when 7 | streams are enlarged and deallocates it when they are shrank. TmStorage storage stores streams in a flat structure where 8 | each stream is referenced by stream Id which is of type GUID. 9 | 10 | TmStorage supports full transactions. In future it will also support caching and snapshots 11 | 12 | It can be used in the simple form as it is, or, because of it's simple flat structure, a more complex storage engines or 13 | file systems with custom internal structure can be developed. Such derived storages or file systems automatically get all 14 | the features in TmStorage (transactions, caching, snapshots...). 15 | -------------------------------------------------------------------------------- /master/Storage/Common.cs: -------------------------------------------------------------------------------- 1 | //Copyright (c) 2012 Tomaz Koritnik 2 | 3 | //Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation 4 | //files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, 5 | //modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the 6 | //Software is furnished to do so, subject to the following conditions: 7 | 8 | //The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 9 | 10 | //THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE 11 | //WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 12 | //COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 13 | //ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 14 | 15 | using System; 16 | 17 | namespace TmFramework.TmStorage 18 | { 19 | /// 20 | /// Class holding system stream Id's 21 | /// 22 | internal static class SystemStreamId 23 | { 24 | public static Guid StorageMetadata = new Guid(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1); 25 | public static Guid EmptySpace = new Guid(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2); 26 | public static Guid StreamTable = new Guid(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3); 27 | 28 | public static bool IsSystemStreamId(Guid streamId) 29 | { 30 | return streamId == StorageMetadata || streamId == EmptySpace || streamId == StreamTable; 31 | } 32 | } 33 | 34 | /// 35 | /// Argument to TransactionStateChanged event 36 | /// 37 | public class TransactionStateChangedEventArgs : EventArgs 38 | { 39 | public TransactionStateChangeType TransactionStateChangeType { get; private set; } 40 | 41 | public TransactionStateChangedEventArgs(TransactionStateChangeType transactionStateChangeType) 42 | { 43 | this.TransactionStateChangeType = transactionStateChangeType; 44 | } 45 | } 46 | 47 | /// 48 | /// Type of transaction state change 49 | /// 50 | public enum TransactionStateChangeType 51 | { 52 | /// 53 | /// Transaction has started 54 | /// 55 | Start, 56 | /// 57 | /// Transaction has been commited 58 | /// 59 | Commit, 60 | /// 61 | /// Transaction has been rolled back 62 | /// 63 | Rollback 64 | } 65 | } 66 | -------------------------------------------------------------------------------- /master/Storage/Exceptions.cs: -------------------------------------------------------------------------------- 1 | //Copyright (c) 2012 Tomaz Koritnik 2 | 3 | //Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation 4 | //files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, 5 | //modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the 6 | //Software is furnished to do so, subject to the following conditions: 7 | 8 | //The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 9 | 10 | //THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE 11 | //WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 12 | //COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 13 | //ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 14 | 15 | using System; 16 | using System.Collections.Generic; 17 | using System.Linq; 18 | using System.Text; 19 | 20 | namespace TmFramework.TmStorage 21 | { 22 | /// 23 | /// Base storage exception 24 | /// 25 | public class StorageException : Exception 26 | { 27 | public StorageException(string message) : base(message) { } 28 | } 29 | 30 | public class UnknownErrorException : StorageException 31 | { 32 | public UnknownErrorException(string message) : base(message) { } 33 | } 34 | 35 | public class InvalidStreamIdException : StorageException 36 | { 37 | public InvalidStreamIdException() : base("Invalid stream Id. It must be larger or equal to zero.") { } 38 | } 39 | 40 | public class StorageCorruptException : StorageException 41 | { 42 | public StorageCorruptException(string message) : base("Storage is corrupt (" + message + ")") { } 43 | } 44 | 45 | public class StorageClosedException : StorageException 46 | { 47 | public StorageClosedException() : base("Storage is closed") { } 48 | } 49 | 50 | public class StreamExistsException : StorageException 51 | { 52 | public StreamExistsException() : base("Stream already exists") { } 53 | } 54 | 55 | public class StreamNotFoundException : StorageException 56 | { 57 | public StreamNotFoundException() : base("Stream could not be found") { } 58 | } 59 | 60 | public class StreamClosedException : StorageException 61 | { 62 | public StreamClosedException() : base("Stream is closed") { } 63 | } 64 | 65 | public class InvalidSegmentException : StorageException 66 | { 67 | public InvalidSegmentException() : base("Error loading segment") { } 68 | } 69 | 70 | public class WritingOutsideOfTransactionException : StorageException 71 | { 72 | public WritingOutsideOfTransactionException() : base("Cannot perform write operation outside the transaction") { } 73 | } 74 | 75 | public class UnableToOpenStorageException : StorageException 76 | { 77 | public UnableToOpenStorageException(string reason) : base("Unable to open storage: " + reason) { } 78 | } 79 | } 80 | -------------------------------------------------------------------------------- /master/Storage/MasterStream.cs: -------------------------------------------------------------------------------- 1 | //Copyright (c) 2012 Tomaz Koritnik 2 | 3 | //Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation 4 | //files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, 5 | //modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the 6 | //Software is furnished to do so, subject to the following conditions: 7 | 8 | //The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 9 | 10 | //THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE 11 | //WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 12 | //COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 13 | //ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 14 | 15 | using System; 16 | using System.IO; 17 | 18 | namespace TmFramework.TmStorage 19 | { 20 | /// 21 | /// Represents a main stream where all of the storage data is stored. 22 | /// 23 | internal class MasterStream : Stream 24 | { 25 | #region Fields 26 | private Stream stream; 27 | private WriteBufferedStream bufferStream; 28 | private bool inTransaction = false; 29 | private bool transactionsEnabled = true; 30 | #endregion 31 | 32 | #region Properties 33 | private long bytesWritten = 0; 34 | /// 35 | /// Total amount of bytes written 36 | /// 37 | public long BytesWritten 38 | { 39 | get { return bytesWritten; } 40 | } 41 | 42 | private long bytesRead = 0; 43 | /// 44 | /// Total amount of bytes read 45 | /// 46 | public long BytesRead 47 | { 48 | get { return bytesRead; } 49 | } 50 | 51 | /// 52 | /// Read/Write cursor position 53 | /// 54 | public override long Position 55 | { 56 | get { return stream.Position; } 57 | set { stream.Position = value; } 58 | } 59 | /// 60 | /// Returns true if stream can be read 61 | /// 62 | public override bool CanRead 63 | { 64 | get { return stream.CanRead; } 65 | } 66 | /// 67 | /// Returns true if stream supports seek operation 68 | /// 69 | public override bool CanSeek 70 | { 71 | get { return stream.CanSeek; } 72 | } 73 | /// 74 | /// Returns true if stream can be written 75 | /// 76 | public override bool CanWrite 77 | { 78 | get { return stream.CanWrite; } 79 | } 80 | /// 81 | /// Length of the stream 82 | /// 83 | public override long Length 84 | { 85 | get { return stream.Length; } 86 | } 87 | #endregion 88 | 89 | #region Constructor 90 | /// 91 | /// Creates the stream 92 | /// 93 | /// Stream to wrap 94 | public MasterStream(Stream stream, bool bufferData) 95 | { 96 | if (stream == null) 97 | throw new ArgumentNullException("stream"); 98 | 99 | if (bufferData) 100 | { 101 | bufferStream = new WriteBufferedStream(stream, 512); 102 | this.stream = bufferStream; 103 | } 104 | else 105 | this.stream = stream; 106 | } 107 | #endregion 108 | 109 | #region Public methods 110 | /// 111 | /// Closes the stream 112 | /// 113 | public override void Close() 114 | { 115 | stream.Close(); 116 | } 117 | /// 118 | /// Flushes buffered data 119 | /// 120 | public override void Flush() 121 | { 122 | if (bufferStream != null) 123 | bufferStream.FlushBufferedData(); 124 | else 125 | stream.Flush(); 126 | } 127 | /// 128 | /// Seek to position 129 | /// 130 | /// Offset from origin (positive or negative direction) 131 | /// Seek origin 132 | /// New stream position 133 | public override long Seek(long offset, SeekOrigin origin) 134 | { 135 | return stream.Seek(offset, origin); 136 | } 137 | /// 138 | /// Sets the length of the stream 139 | /// 140 | /// New length 141 | public override void SetLength(long value) 142 | { 143 | stream.SetLength(value); 144 | } 145 | 146 | /// 147 | /// Read data from stream to buffer 148 | /// 149 | /// Buffer to store read data 150 | /// Position in buffer where writing starts 151 | /// Number of bytes to read 152 | /// Number of bytes read 153 | public override int Read(byte[] buffer, int offset, int count) 154 | { 155 | long pos = stream.Position; 156 | int l = stream.Read(buffer, offset, count); 157 | bytesRead += l; 158 | if (DataRead != null) 159 | DataRead(this, new ReadWriteEventArgs(buffer, pos, l)); 160 | return l; 161 | } 162 | /// 163 | /// Write data from buffer to stream 164 | /// 165 | /// Source buffer 166 | /// Position in buffer where reading starts 167 | /// Amount of data written 168 | public override void Write(byte[] buffer, int offset, int count) 169 | { 170 | if (!inTransaction && transactionsEnabled) 171 | throw new WritingOutsideOfTransactionException(); 172 | 173 | long pos = stream.Position; 174 | bytesWritten += count; 175 | stream.Write(buffer, offset, count); 176 | if (DataWrite != null) 177 | DataWrite(this, new ReadWriteEventArgs(buffer, pos, count)); 178 | } 179 | 180 | public void StartTransaction() 181 | { 182 | if (inTransaction) 183 | throw new InvalidOperationException("Stream is already in transaction"); 184 | 185 | inTransaction = true; 186 | } 187 | public void CommitTransaction() 188 | { 189 | if (!inTransaction) 190 | throw new InvalidOperationException("Stream is not in transaction"); 191 | 192 | inTransaction = false; 193 | } 194 | public void RollbackTransaction() 195 | { 196 | if (!inTransaction) 197 | throw new InvalidOperationException("Stream is not in transaction"); 198 | 199 | WriteBufferedStream bufferedStream = stream as WriteBufferedStream; 200 | 201 | if (bufferedStream != null) 202 | { 203 | bufferedStream.DiscardBufferedData(); 204 | } 205 | 206 | inTransaction = false; 207 | } 208 | #endregion 209 | 210 | #region Events 211 | /// 212 | /// Occurs when data is written to master stream 213 | /// 214 | public event EventHandler DataWrite; 215 | /// 216 | /// Occurs when data is read from master stream 217 | /// 218 | public event EventHandler DataRead; 219 | #endregion 220 | } 221 | 222 | #region ReadWriteEventArgs 223 | internal class ReadWriteEventArgs : EventArgs 224 | { 225 | public byte[] Buf { get; private set; } 226 | public long Location { get; private set; } 227 | public int Length { get; private set; } 228 | 229 | public ReadWriteEventArgs(byte[] buf, long location, int length) 230 | { 231 | Buf = buf; 232 | Location = location; 233 | Length = length; 234 | } 235 | } 236 | #endregion 237 | } 238 | -------------------------------------------------------------------------------- /master/Storage/Properties/AssemblyInfo.cs: -------------------------------------------------------------------------------- 1 | using System.Reflection; 2 | using System.Runtime.CompilerServices; 3 | using System.Runtime.InteropServices; 4 | 5 | // General Information about an assembly is controlled through the following 6 | // set of attributes. Change these attribute values to modify the information 7 | // associated with an assembly. 8 | [assembly: AssemblyTitle("TmStorage")] 9 | [assembly: AssemblyDescription("Virtual file system")] 10 | [assembly: AssemblyConfiguration("")] 11 | [assembly: AssemblyCompany("")] 12 | [assembly: AssemblyProduct("TmStorage")] 13 | [assembly: AssemblyCopyright("Copyright © Tomaz Koritnik 2012")] 14 | [assembly: AssemblyTrademark("")] 15 | [assembly: AssemblyCulture("")] 16 | 17 | // Setting ComVisible to false makes the types in this assembly not visible 18 | // to COM components. If you need to access a type in this assembly from 19 | // COM, set the ComVisible attribute to true on that type. 20 | [assembly: ComVisible(false)] 21 | 22 | // The following GUID is for the ID of the typelib if this project is exposed to COM 23 | [assembly: Guid("7f795a02-b422-4fc1-8b7d-bcd5f07d4ac6")] 24 | 25 | // Version information for an assembly consists of the following four values: 26 | // 27 | // Major Version 28 | // Minor Version 29 | // Build Number 30 | // Revision 31 | // 32 | // You can specify all the values or you can default the Build and Revision Numbers 33 | // by using the '*' as shown below: 34 | // [assembly: AssemblyVersion("1.0.*")] 35 | [assembly: AssemblyVersion("0.9.0.0")] 36 | [assembly: AssemblyFileVersion("0.9.0.0")] 37 | -------------------------------------------------------------------------------- /master/Storage/Segment.cs: -------------------------------------------------------------------------------- 1 | //Copyright (c) 2012 Tomaz Koritnik 2 | 3 | //Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation 4 | //files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, 5 | //modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the 6 | //Software is furnished to do so, subject to the following conditions: 7 | 8 | //The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 9 | 10 | //THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE 11 | //WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 12 | //COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 13 | //ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 14 | 15 | using System; 16 | using System.IO; 17 | using System.Collections.Generic; 18 | 19 | namespace TmFramework.TmStorage 20 | { 21 | /// 22 | /// Represents an area of the master stream that is allocated to a specific stream 23 | /// 24 | internal class Segment 25 | { 26 | private bool isModified = true; 27 | 28 | private long location; 29 | /// 30 | /// Location of segment in master stream 31 | /// 32 | public long Location 33 | { 34 | get { return location; } 35 | private set 36 | { 37 | if (location != value) 38 | { 39 | location = value; 40 | isModified = true; 41 | } 42 | } 43 | } 44 | 45 | private long size; 46 | /// 47 | /// Total size of segment 48 | /// 49 | public long Size 50 | { 51 | get { return size; } 52 | set 53 | { 54 | if (size != value) 55 | { 56 | size = value; 57 | isModified = true; 58 | } 59 | } 60 | } 61 | 62 | private long? nextLocation; 63 | /// 64 | /// Pointer to location where next segment in chain is located or null of it's the last one 65 | /// 66 | public long? NextLocation 67 | { 68 | get { return nextLocation; } 69 | set 70 | { 71 | if (nextLocation != value) 72 | { 73 | nextLocation = value; 74 | isModified = true; 75 | } 76 | } 77 | } 78 | 79 | /// 80 | /// Size of this structure when 81 | /// 82 | public static long StructureSize 83 | { 84 | get { return 20; } 85 | } 86 | /// 87 | /// Size of data area size 88 | /// 89 | public long DataAreaSize 90 | { 91 | get { return Size - StructureSize; } 92 | } 93 | /// 94 | /// Location when data area starts 95 | /// 96 | public long DataAreaStart 97 | { 98 | get { return Location + StructureSize; } 99 | } 100 | /// 101 | /// Location where data area ends (same location where segment ends) 102 | /// 103 | public long DataAreaEnd 104 | { 105 | get { return Location + Size; } 106 | } 107 | 108 | private Segment() 109 | { 110 | } 111 | 112 | /// 113 | /// Splits this segment by making it smaller. 114 | /// 115 | /// Required total size of the new segment 116 | /// If true, new segment is created from the end of this one. This is false only for empty space stream. 117 | /// New segment 118 | public Segment Split(long sizeToRemove, bool splitAtEnd) 119 | { 120 | Segment segment; 121 | 122 | if (sizeToRemove > Size) 123 | throw new InvalidOperationException("Unable to split because size to split is larger than the segment itself"); 124 | 125 | long newSize = this.Size - sizeToRemove; 126 | 127 | if (splitAtEnd) 128 | { 129 | segment = Segment.Create(Location + Size - sizeToRemove, sizeToRemove, null); 130 | } 131 | else 132 | { 133 | segment = Segment.Create(Location, sizeToRemove, null); 134 | this.Location += sizeToRemove; 135 | } 136 | 137 | this.Size = newSize; 138 | 139 | return segment; 140 | } 141 | 142 | /// 143 | /// Creates a segment 144 | /// 145 | /// Segment location 146 | /// Segment size 147 | /// Next segment pointer 148 | public static Segment Create(long location, long size, long? nextSegmentLocation) 149 | { 150 | if (size <= 0) 151 | throw new StorageException("Segment size can't be zero or less"); 152 | 153 | return new Segment 154 | { 155 | Location = location, 156 | Size = size, 157 | NextLocation = nextSegmentLocation 158 | }; 159 | } 160 | /// 161 | /// Loads segment from stream 162 | /// 163 | /// Reader 164 | /// Location of segment in stream 165 | public static Segment Load(Stream stream, long location) 166 | { 167 | stream.Seek(location, SeekOrigin.Begin); 168 | stream.Read(Tools.Buffer, 0, (int)Segment.StructureSize); 169 | Tools.BufferReader.BaseStream.Position = 0; 170 | 171 | long size = Tools.BufferReader.ReadInt64(); 172 | long tmpNextSegmentLocation = Tools.BufferReader.ReadInt64(); 173 | int hash = Tools.BufferReader.ReadInt32(); 174 | int calculatedHash = Tools.CalculateHash(size, tmpNextSegmentLocation); 175 | 176 | if (hash != calculatedHash || size == 0) 177 | throw new InvalidSegmentException(); 178 | 179 | return Segment.Create(location, size, tmpNextSegmentLocation != 0 ? tmpNextSegmentLocation : (long?)null); 180 | } 181 | /// 182 | /// Saves a segment to stream 183 | /// 184 | /// 185 | public void Save(Stream stream) 186 | { 187 | if (isModified) 188 | { 189 | Tools.BufferWriter.BaseStream.Position = 0; 190 | Tools.BufferWriter.Write(Size); 191 | 192 | long nextSegmentValue = NextLocation.HasValue ? NextLocation.Value : (long)0; 193 | Tools.BufferWriter.Write(nextSegmentValue); 194 | 195 | int hash = Tools.CalculateHash(Size, nextSegmentValue); 196 | Tools.BufferWriter.Write(hash); 197 | 198 | stream.Position = Location; 199 | stream.Write(Tools.Buffer, 0, (int)Segment.StructureSize); 200 | 201 | isModified = false; 202 | } 203 | } 204 | } 205 | } 206 | -------------------------------------------------------------------------------- /master/Storage/SegmentExtent.cs: -------------------------------------------------------------------------------- 1 | //Copyright (c) 2012 Tomaz Koritnik 2 | 3 | //Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation 4 | //files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, 5 | //modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the 6 | //Software is furnished to do so, subject to the following conditions: 7 | 8 | //The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 9 | 10 | //THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE 11 | //WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 12 | //COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 13 | //ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 14 | 15 | using System; 16 | using System.Collections.Generic; 17 | using System.Linq; 18 | using System.Text; 19 | 20 | namespace TmFramework.TmStorage 21 | { 22 | /// 23 | /// Segment extent in the master stream 24 | /// 25 | public class SegmentExtent 26 | { 27 | /// 28 | /// Segment location 29 | /// 30 | public readonly long Location; 31 | /// 32 | /// Segment size 33 | /// 34 | public readonly long Size; 35 | 36 | internal SegmentExtent(long location, long size) 37 | { 38 | this.Location = location; 39 | this.Size = size; 40 | } 41 | } 42 | } 43 | -------------------------------------------------------------------------------- /master/Storage/Storage.cs: -------------------------------------------------------------------------------- 1 | //Copyright (c) 2012 Tomaz Koritnik 2 | 3 | //Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation 4 | //files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, 5 | //modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the 6 | //Software is furnished to do so, subject to the following conditions: 7 | 8 | //The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 9 | 10 | //THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE 11 | //WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 12 | //COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 13 | //ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 14 | 15 | using System; 16 | using System.Collections.Generic; 17 | using System.IO; 18 | using System.Linq; 19 | using TmFramework.Transactions; 20 | 21 | namespace TmFramework.TmStorage 22 | { 23 | /// 24 | /// Storage 25 | /// 26 | public class Storage 27 | { 28 | #region Fields 29 | // System streams 30 | private StorageStream storageMetadataStream; 31 | private StorageStreamMetadata streamTableStreamMetadata; 32 | private StorageStream streamTableStream; 33 | internal StorageStream FreeSpaceStream { get; private set; } 34 | 35 | // Transaction support 36 | private TransactionStream transactionStream; 37 | private int transactionLevel = 0; 38 | private List streamsChangedDuringTransaction = new List(); 39 | private List streamsCreatedDuringTransaction = new List(); 40 | 41 | // Stream table 42 | private StreamTable streamTable; 43 | 44 | // List of opened streams 45 | private Dictionary> openedStreams = new Dictionary>(); 46 | #endregion 47 | 48 | #region Properties 49 | /// 50 | /// Storage metadata 51 | /// 52 | public StorageMetadata StorageMetadata { get; private set; } 53 | 54 | private bool isClosed = false; 55 | /// 56 | /// Flag indicated whether storage is closed 57 | /// 58 | public bool IsClosed 59 | { 60 | get { return isClosed; } 61 | } 62 | /// 63 | /// Returns true if storage is in transaction 64 | /// 65 | public bool InTransaction 66 | { 67 | get { return transactionLevel > 0; } 68 | } 69 | 70 | public StorageStatistics Statistics { get; private set; } 71 | 72 | /// 73 | /// Master stream where all of the storage data is stored 74 | /// 75 | internal MasterStream MasterStream { get; private set; } 76 | 77 | internal StreamTable StreamTable 78 | { 79 | get { return streamTable; } 80 | } 81 | internal int OpenedStreamsCount 82 | { 83 | get { return openedStreams.Count; } 84 | } 85 | 86 | private const uint blockSize = 512; 87 | internal uint BlockSize 88 | { 89 | get { return blockSize; } 90 | } 91 | #endregion 92 | 93 | #region Construction 94 | /// 95 | /// Constructor 96 | /// 97 | public Storage(Stream stream, Stream transactionLogStream) 98 | { 99 | this.Statistics = new StorageStatistics(this); 100 | 101 | if (stream.Length == 0) 102 | { 103 | CreateStorage(stream); 104 | } 105 | 106 | this.transactionStream = transactionLogStream != null ? new TransactionStream(stream, transactionLogStream, blockSize) : null; 107 | this.MasterStream = new MasterStream(transactionStream != null ? transactionStream : stream, false); 108 | 109 | OpenStorage(); 110 | } 111 | /// 112 | /// Constructor 113 | /// 114 | public Storage(string filename, string transactionLogFilename) 115 | : this(File.Open(filename, FileMode.OpenOrCreate), 116 | transactionLogFilename != null ? File.Open(transactionLogFilename, FileMode.OpenOrCreate) : null) 117 | { 118 | } 119 | #endregion 120 | 121 | #region Public methods 122 | /// 123 | /// Creates a stream 124 | /// 125 | /// Stream Id 126 | public StorageStream CreateStream(Guid streamId, int tag = 0) 127 | { 128 | CheckClosed(); 129 | 130 | if (SystemStreamId.IsSystemStreamId(streamId)) 131 | throw new InvalidStreamIdException(); 132 | if (ContainsStream(streamId)) 133 | throw new StreamExistsException(); 134 | 135 | StartTransaction(); 136 | try 137 | { 138 | streamTable.Add(streamId, tag); 139 | CommitTransaction(); 140 | streamsCreatedDuringTransaction.Add(streamId); 141 | } 142 | catch 143 | { 144 | RollbackTransaction(); 145 | throw; 146 | } 147 | 148 | return OpenStream(streamId); 149 | } 150 | /// 151 | /// Opens a stream 152 | /// 153 | /// Stream Id 154 | public StorageStream OpenStream(Guid streamId) 155 | { 156 | CheckClosed(); 157 | 158 | if (SystemStreamId.IsSystemStreamId(streamId)) 159 | throw new InvalidStreamIdException(); 160 | 161 | StartTransaction(); 162 | try 163 | { 164 | StorageStream tmpStream = null; 165 | WeakReference streamRef; 166 | 167 | // Check if stream is already opened 168 | if (openedStreams.TryGetValue(streamId, out streamRef)) 169 | { 170 | if (!streamRef.TryGetTarget(out tmpStream)) 171 | { 172 | tmpStream = null; 173 | openedStreams.Remove(streamId); 174 | } 175 | } 176 | 177 | // Open stream 178 | if (tmpStream == null) 179 | { 180 | var streamMetadata = streamTable.Get(streamId); 181 | 182 | if (streamMetadata == null) 183 | throw new StreamNotFoundException(); 184 | 185 | tmpStream = new StorageStream(streamMetadata, this); 186 | //tmpStream.Changed += StorageStream_Changed; 187 | 188 | openedStreams.Add(streamId, new WeakReference(tmpStream)); 189 | } 190 | tmpStream.Position = 0; 191 | 192 | CommitTransaction(); 193 | 194 | return tmpStream; 195 | } 196 | catch 197 | { 198 | RollbackTransaction(); 199 | throw; 200 | } 201 | } 202 | /// 203 | /// Deletes a stream 204 | /// 205 | /// Stream Id 206 | public void DeleteStream(Guid streamId) 207 | { 208 | CheckClosed(); 209 | 210 | if (SystemStreamId.IsSystemStreamId(streamId)) 211 | throw new InvalidStreamIdException(); 212 | 213 | StartTransaction(); 214 | try 215 | { 216 | // Before deleting, set stream size to zero to deallocate all of the space it occupies 217 | StorageStream tmpStream = OpenStream(streamId); 218 | tmpStream.SetLength(0); 219 | tmpStream.Close(); 220 | 221 | openedStreams.Remove(streamId); 222 | streamTable.Remove(streamId); 223 | 224 | // Remove stream from list of changed streams 225 | tmpStream = streamsChangedDuringTransaction.SingleOrDefault(x => x.StreamId == streamId); 226 | if (tmpStream != null) 227 | streamsChangedDuringTransaction.Remove(tmpStream); 228 | // Remove stream from list of created streams 229 | if (streamsCreatedDuringTransaction.Contains(streamId)) 230 | streamsCreatedDuringTransaction.Remove(streamId); 231 | 232 | CommitTransaction(); 233 | } 234 | catch 235 | { 236 | RollbackTransaction(); 237 | throw; 238 | } 239 | } 240 | /// 241 | /// Checks if storage contains specified stream 242 | /// 243 | public bool ContainsStream(Guid streamId) 244 | { 245 | CheckClosed(); 246 | 247 | if (SystemStreamId.IsSystemStreamId(streamId)) 248 | throw new InvalidStreamIdException(); 249 | 250 | return streamTable.Contains(streamId); 251 | } 252 | /// 253 | /// Gets areas where specified stream segments are located 254 | /// 255 | public List GetStreamExtents(Guid streamId) 256 | { 257 | CheckClosed(); 258 | 259 | if (SystemStreamId.IsSystemStreamId(streamId)) 260 | throw new InvalidStreamIdException(); 261 | 262 | StorageStream stream = OpenStream(streamId); 263 | return stream.GetStreamExtents(); 264 | } 265 | /// 266 | /// Gets areas where empty space segments are located 267 | /// 268 | public IEnumerable GetFreeSpaceExtents() 269 | { 270 | CheckClosed(); 271 | if (FreeSpaceStream != null) 272 | { 273 | return FreeSpaceStream.Segments 274 | .Select(x => new SegmentExtent(x.Location, x.Size)).ToList(); 275 | } 276 | else 277 | return new List(); 278 | } 279 | /// 280 | /// Closes the storage 281 | /// 282 | public void Close() 283 | { 284 | if (transactionLevel > 0) 285 | { 286 | InternalRollbackTransaction(); 287 | throw new StorageException("Unable to close storage while transaction is pending"); 288 | } 289 | 290 | if (!isClosed) 291 | { 292 | lock (openedStreams) 293 | { 294 | //cacheCleanupTimer.Dispose(); 295 | //cacheCleanupTimer = null; 296 | 297 | RollbackTransaction(); 298 | 299 | // Cache stream table into empty space stream 300 | MasterStream.Flush(); 301 | MasterStream.Close(); 302 | openedStreams.Clear(); 303 | streamsChangedDuringTransaction.Clear(); 304 | isClosed = true; 305 | } 306 | } 307 | } 308 | /// 309 | /// Gets all of the stream Id's 310 | /// 311 | public IEnumerable GetStreams() 312 | { 313 | return GetStreams(null); 314 | } 315 | /// 316 | /// Gets all of the stream Id's 317 | /// 318 | /// If specified, only streams with specified tag are returned 319 | public IEnumerable GetStreams(int? tag) 320 | { 321 | return streamTable.Get() 322 | .Where(x => !SystemStreamId.IsSystemStreamId(x.StreamId)) 323 | .Where(x => !tag.HasValue || x.Tag == tag.Value) 324 | .Select(x => x.StreamId) 325 | .ToList(); 326 | } 327 | /// 328 | /// Trim the master file to the location where data ends 329 | /// 330 | public void TrimStorage() 331 | { 332 | Segment lastSegment = FreeSpaceStream.Segments.SingleOrDefault(x => !x.NextLocation.HasValue); 333 | 334 | if (lastSegment != null) 335 | { 336 | MasterStream.SetLength(lastSegment.DataAreaStart); 337 | } 338 | } 339 | 340 | /// 341 | /// Start a transaction 342 | /// 343 | public void StartTransaction() 344 | { 345 | try 346 | { 347 | CheckClosed(); 348 | transactionLevel++; 349 | 350 | if (transactionLevel == 1) 351 | { 352 | if (streamsChangedDuringTransaction.Count > 0) 353 | throw new StorageException("At the begining of transaction there should be no changed streams"); 354 | 355 | NotifyTransactionChanging(TransactionStateChangeType.Start); 356 | 357 | MasterStream.StartTransaction(); 358 | 359 | if (transactionStream != null) 360 | { 361 | // Make a list of extents that doesn't need to be backed up 362 | IEnumerable list = FreeSpaceStream != null ? FreeSpaceStream.Segments.Select(x => new Transactions.Segment(x.DataAreaStart, x.DataAreaSize)) : null; 363 | transactionStream.BeginTransaction(list); 364 | } 365 | 366 | NotifyTransactionChanged(TransactionStateChangeType.Start); 367 | } 368 | } 369 | catch 370 | { 371 | InternalRollbackTransaction(); 372 | throw; 373 | } 374 | } 375 | /// 376 | /// Commits a transaction 377 | /// 378 | public void CommitTransaction() 379 | { 380 | try 381 | { 382 | CheckClosed(); 383 | if (transactionLevel == 1) 384 | { 385 | NotifyTransactionChanging(TransactionStateChangeType.Commit); 386 | 387 | SaveChanges(); 388 | if (transactionStream != null) 389 | transactionStream.EndTransaction(); 390 | 391 | streamsCreatedDuringTransaction.Clear(); 392 | MasterStream.Flush(); 393 | MasterStream.CommitTransaction(); 394 | Statistics.TransactionsCommited++; 395 | } 396 | 397 | if (transactionLevel > 0) 398 | { 399 | transactionLevel--; 400 | 401 | if (transactionLevel == 0) 402 | NotifyTransactionChanged(TransactionStateChangeType.Commit); 403 | } 404 | } 405 | catch 406 | { 407 | InternalRollbackTransaction(); 408 | throw; 409 | } 410 | } 411 | /// 412 | /// Rollbacks a transaction 413 | /// 414 | public void RollbackTransaction() 415 | { 416 | CheckClosed(); 417 | 418 | if (transactionStream != null) 419 | { 420 | NotifyTransactionChanging(TransactionStateChangeType.Rollback); 421 | 422 | InternalRollbackTransaction(); 423 | 424 | transactionLevel = 0; 425 | Statistics.TransactionsRolledBack++; 426 | NotifyTransactionChanged(TransactionStateChangeType.Rollback); 427 | } 428 | else 429 | { 430 | CommitTransaction(); 431 | } 432 | } 433 | 434 | public void TruncateStorage() 435 | { 436 | throw new NotImplementedException(); 437 | } 438 | #endregion 439 | 440 | #region Private methods 441 | private void InternalRollbackTransaction() 442 | { 443 | if (transactionLevel > 0) 444 | { 445 | // Remove opened streams created during transaction 446 | lock (openedStreams) 447 | { 448 | foreach (Guid streamId in streamsCreatedDuringTransaction) 449 | { 450 | WeakReference reference; 451 | if (openedStreams.TryGetValue(streamId, out reference)) 452 | { 453 | StorageStream tmpStream; 454 | if (reference.TryGetTarget(out tmpStream)) 455 | { 456 | if (tmpStream != null) 457 | { 458 | tmpStream.InternalClose(); 459 | } 460 | } 461 | 462 | openedStreams.Remove(streamId); 463 | } 464 | } 465 | streamsCreatedDuringTransaction.Clear(); 466 | 467 | // Rollback data 468 | transactionStream.RollbackTransaction(); 469 | MasterStream.RollbackTransaction(); 470 | streamsChangedDuringTransaction.Clear(); 471 | 472 | // Rollback changes in stream table 473 | streamTableStream.ReloadSegmentsOnRollback(streamTableStreamMetadata); 474 | streamTable.RollbackTransaction(); 475 | 476 | // Reload segments in system and opened streams because segments has changed 477 | foreach (var item in openedStreams.Values.ToList()) 478 | { 479 | StorageStream tmpStream; 480 | if (item.TryGetTarget(out tmpStream)) 481 | { 482 | if (streamTable.Contains(tmpStream.StreamId)) 483 | { 484 | StorageStreamMetadata tmpStreamMetadata = streamTable.Get(tmpStream.StreamId); 485 | tmpStream.ReloadSegmentsOnRollback(tmpStreamMetadata); 486 | } 487 | else 488 | { 489 | tmpStream.InternalClose(); 490 | } 491 | } 492 | } 493 | 494 | // Reload empty space segments 495 | var freeSpaceStreamMetadata = streamTable.Get(SystemStreamId.EmptySpace); 496 | FreeSpaceStream.ReloadSegmentsOnRollback(freeSpaceStreamMetadata); 497 | } 498 | } 499 | } 500 | /// 501 | /// Creates a storage 502 | /// 503 | private void CreateStorage(Stream stream) 504 | { 505 | this.MasterStream = new MasterStream(stream, false); 506 | 507 | // Initialize storage metadata 508 | Segment metadataStreamSegment = Segment.Create(0, blockSize, null); 509 | metadataStreamSegment.Save(stream); 510 | 511 | StorageStream metadataStream = new StorageStream(new StorageStreamMetadata(null) 512 | { 513 | FirstSegmentPosition = 0, 514 | InitializedLength = blockSize - Segment.StructureSize, 515 | Length = blockSize - Segment.StructureSize, 516 | StreamId = SystemStreamId.StorageMetadata, 517 | StreamTableIndex = -1 518 | }, this); 519 | StorageMetadata storageMetadata = new StorageMetadata("[TmStorage 1.0]"); // Set metadata again because above, stream was not specified 520 | storageMetadata.Save(metadataStream); 521 | metadataStream.Close(); 522 | 523 | // Initialize stream table 524 | long streamTableSegmentSize = 1000 / ((int)blockSize / StorageStreamMetadata.StructureSize) * blockSize; 525 | Segment streamTableSegment = Segment.Create(blockSize, streamTableSegmentSize, null); 526 | stream.Position = metadataStreamSegment.DataAreaEnd; 527 | streamTableSegment.Save(stream); 528 | 529 | StorageStream streamTableStream = new StorageStream(new StorageStreamMetadata(null) 530 | { 531 | FirstSegmentPosition = blockSize, 532 | InitializedLength = streamTableSegmentSize - Segment.StructureSize, 533 | Length = streamTableSegmentSize - Segment.StructureSize, 534 | StreamId = SystemStreamId.StreamTable, 535 | StreamTableIndex = -1 536 | }, this); 537 | 538 | // Initialize empty space stream 539 | Segment emptyStreamSegment = Segment.Create(streamTableSegment.DataAreaEnd, long.MaxValue - streamTableSegment.DataAreaEnd, null); 540 | stream.Position = streamTableSegment.DataAreaEnd; 541 | emptyStreamSegment.Save(stream); 542 | 543 | // Write empty space stream metadata to stream table 544 | StorageStreamMetadata emptySpaceStreamMetadata = new StorageStreamMetadata(streamTableStream) 545 | { 546 | FirstSegmentPosition = emptyStreamSegment.Location, 547 | InitializedLength = emptyStreamSegment.DataAreaSize, 548 | Length = emptyStreamSegment.DataAreaSize, 549 | StreamId = SystemStreamId.EmptySpace, 550 | StreamTableIndex = 0 551 | }; 552 | emptySpaceStreamMetadata.Save(); 553 | 554 | this.MasterStream = null; 555 | } 556 | /// 557 | /// Opens the storage 558 | /// 559 | private void OpenStorage() 560 | { 561 | StartTransaction(); 562 | try 563 | { 564 | // For metadata assume block size of 512 because blockSize is unknown at this point. 565 | // 512 is the smallest block size so it will work as long as storage metadata is not 566 | // longer than 512 bytes 567 | storageMetadataStream = new StorageStream(new StorageStreamMetadata(null) 568 | { 569 | FirstSegmentPosition = 0, 570 | InitializedLength = 512 - Segment.StructureSize, 571 | Length = 512 - Segment.StructureSize, 572 | StreamId = SystemStreamId.StorageMetadata, 573 | StreamTableIndex = -1 574 | }, this); 575 | StorageMetadata = StorageMetadata.Load(storageMetadataStream); 576 | 577 | streamTableStreamMetadata = new StorageStreamMetadata(storageMetadataStream) 578 | { 579 | FirstSegmentPosition = blockSize, 580 | StreamId = SystemStreamId.StreamTable, 581 | StreamTableIndex = -1 582 | }; 583 | streamTableStream = new StorageStream(streamTableStreamMetadata, this); 584 | streamTable = new StreamTable(streamTableStream); 585 | 586 | var freeSpaceStreamMetadata = streamTable.Get(SystemStreamId.EmptySpace); 587 | FreeSpaceStream = new StorageStream(freeSpaceStreamMetadata, this); 588 | 589 | CommitTransaction(); 590 | } 591 | catch 592 | { 593 | RollbackTransaction(); 594 | throw; 595 | } 596 | } 597 | /// 598 | /// Saves changes of all changed streams during transaction 599 | /// 600 | private void SaveChanges() 601 | { 602 | foreach (var stream in streamsChangedDuringTransaction) 603 | { 604 | stream.Save(); 605 | } 606 | if (streamTable != null) 607 | streamTable.SaveChanges(); 608 | streamsChangedDuringTransaction.Clear(); 609 | } 610 | private void CheckClosed() 611 | { 612 | if (isClosed) 613 | throw new StorageClosedException(); 614 | } 615 | private void NotifyTransactionChanged(TransactionStateChangeType transactionStateChangeType) 616 | { 617 | if (TransactionStateChanged != null) 618 | TransactionStateChanged(this, new TransactionStateChangedEventArgs(transactionStateChangeType)); 619 | } 620 | private void NotifyTransactionChanging(TransactionStateChangeType transactionStateChangeType) 621 | { 622 | if (TransactionStateChanging != null) 623 | TransactionStateChanging(this, new TransactionStateChangedEventArgs(transactionStateChangeType)); 624 | } 625 | #endregion 626 | 627 | #region Internal methods 628 | internal void StreamChanged(StorageStreamChangeType changeType, StorageStream stream) 629 | { 630 | if (!SystemStreamId.IsSystemStreamId(stream.StreamId) || stream.StreamId == SystemStreamId.EmptySpace) 631 | { 632 | switch (changeType) 633 | { 634 | case StorageStreamChangeType.SegmentsAndMetadata: 635 | if (!streamsChangedDuringTransaction.Contains(stream)) 636 | streamsChangedDuringTransaction.Add(stream); 637 | break; 638 | case StorageStreamChangeType.Closing: 639 | if (streamsChangedDuringTransaction.Contains(stream)) 640 | streamsChangedDuringTransaction.Remove(stream); 641 | 642 | openedStreams.Remove(stream.StreamId); 643 | //e.Stream.Changed -= StorageStream_Changed; 644 | break; 645 | } 646 | } 647 | } 648 | #endregion Internal methods 649 | 650 | #region Event handlers 651 | /*private void StorageStream_Changed(object sender, StorageStreamChangedArgs e) 652 | { 653 | switch (e.ChangeType) 654 | { 655 | case StorageStreamChangeType.SegmentsAndMetadata: 656 | if (!streamsChangedDuringTransaction.Contains(e.Stream)) 657 | streamsChangedDuringTransaction.Add(e.Stream); 658 | break; 659 | case StorageStreamChangeType.Closing: 660 | if (streamsChangedDuringTransaction.Contains(e.Stream)) 661 | streamsChangedDuringTransaction.Remove(e.Stream); 662 | openedStreams.Remove(e.Stream.StreamId); 663 | //e.Stream.Changed -= StorageStream_Changed; 664 | break; 665 | } 666 | }*/ 667 | #endregion 668 | 669 | #region Events 670 | /// 671 | /// Triggered after transaction state has changed 672 | /// 673 | public event EventHandler TransactionStateChanged; 674 | /// 675 | /// Triggered before changing transaction state 676 | /// 677 | public event EventHandler TransactionStateChanging; 678 | #endregion 679 | } 680 | } 681 | -------------------------------------------------------------------------------- /master/Storage/StorageMetadata.cs: -------------------------------------------------------------------------------- 1 | //Copyright (c) 2012 Tomaz Koritnik 2 | 3 | //Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation 4 | //files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, 5 | //modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the 6 | //Software is furnished to do so, subject to the following conditions: 7 | 8 | //The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 9 | 10 | //THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE 11 | //WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 12 | //COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 13 | //ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 14 | 15 | using System; 16 | using System.IO; 17 | 18 | namespace TmFramework.TmStorage 19 | { 20 | /// 21 | /// Hold a metadata for the storage 22 | /// 23 | public class StorageMetadata 24 | { 25 | public string Version { get; private set; } 26 | 27 | internal StorageMetadata(string version) 28 | { 29 | this.Version = version; 30 | } 31 | 32 | internal static StorageMetadata Load(Stream stream) 33 | { 34 | stream.Position = 0; 35 | BinaryReader reader = new BinaryReader(stream); 36 | 37 | string version = reader.ReadString(); 38 | int hash = reader.ReadInt32(); 39 | int calculatedHash = Tools.CalculateHash(version); 40 | 41 | if (hash != calculatedHash) 42 | throw new StorageCorruptException("Metadata has check failed"); 43 | 44 | StorageMetadata metadata = new StorageMetadata(version); 45 | return metadata; 46 | } 47 | internal void Save(Stream stream) 48 | { 49 | stream.Position = 0; 50 | BinaryWriter writer = new BinaryWriter(stream); 51 | writer.Write(Version); 52 | 53 | int hash = Tools.CalculateHash(Version); 54 | writer.Write(hash); 55 | } 56 | } 57 | } 58 | -------------------------------------------------------------------------------- /master/Storage/StorageStatistics.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections.Generic; 3 | using System.Linq; 4 | using System.Text; 5 | using System.Threading.Tasks; 6 | 7 | namespace TmFramework.TmStorage 8 | { 9 | public sealed class StorageStatistics 10 | { 11 | private Storage storage; 12 | 13 | public StorageStatistics(Storage storage) 14 | { 15 | this.storage = storage; 16 | } 17 | 18 | public long StorageSize 19 | { 20 | get { return storage.MasterStream.Length; } 21 | } 22 | public long BytesRead 23 | { 24 | get { return storage.MasterStream.BytesRead; } 25 | } 26 | public long BytesWritten 27 | { 28 | get { return storage.MasterStream.BytesWritten; } 29 | } 30 | public int TotalStreamCount 31 | { 32 | get { return storage.StreamTable.Count; } 33 | } 34 | public int OpenedStreamsCount 35 | { 36 | get { return storage.OpenedStreamsCount; } 37 | } 38 | public long TransactionsCommited { get; internal set; } 39 | public long TransactionsRolledBack { get; internal set; } 40 | } 41 | } 42 | -------------------------------------------------------------------------------- /master/Storage/StorageStream.cs: -------------------------------------------------------------------------------- 1 | //Copyright (c) 2012 Tomaz Koritnik 2 | 3 | //Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation 4 | //files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, 5 | //modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the 6 | //Software is furnished to do so, subject to the following conditions: 7 | 8 | //The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 9 | 10 | //THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE 11 | //WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 12 | //COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 13 | //ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 14 | 15 | using System; 16 | using System.Collections.Generic; 17 | using System.IO; 18 | using System.Linq; 19 | 20 | namespace TmFramework.TmStorage 21 | { 22 | /// 23 | /// Represents a stream stored inside the storage 24 | /// 25 | public class StorageStream : Stream 26 | { 27 | #region Fields 28 | // Chain of segments holding stream data 29 | private LinkedList segments = new LinkedList(); 30 | private StorageStreamMetadata metadata; 31 | private Storage storage; 32 | private bool isClosed = false; 33 | private bool changeNotified = false; 34 | #endregion 35 | 36 | #region Construction 37 | /// 38 | /// Constructor that loads the segments from master stream 39 | /// 40 | internal StorageStream(StorageStreamMetadata metadata, Storage storage) 41 | { 42 | if (storage == null) 43 | throw new ArgumentNullException("storage"); 44 | 45 | this.storage = storage; 46 | 47 | LoadStream(metadata); 48 | } 49 | private void LoadStream(StorageStreamMetadata metadata) 50 | { 51 | // Load segments 52 | this.metadata = metadata; 53 | long? segmentPosition = metadata.FirstSegmentPosition; 54 | segments.Clear(); 55 | 56 | while (segmentPosition.HasValue) 57 | { 58 | Segment segment = Segment.Load(storage.MasterStream, segmentPosition.Value); 59 | 60 | segments.AddLast(segment); 61 | segmentPosition = segment.NextLocation; 62 | } 63 | 64 | // Manually adjust stream length and initializedSized for stream table stream 65 | if (metadata.StreamId == SystemStreamId.StreamTable) 66 | { 67 | metadata.Length = segments.Sum(x => x.DataAreaSize); 68 | metadata.InitializedLength = metadata.Length; 69 | } 70 | } 71 | #endregion 72 | 73 | #region Properties 74 | private long position = 0; 75 | /// 76 | /// Read/write cursor position inside stream 77 | /// 78 | public override long Position 79 | { 80 | get 81 | { 82 | CheckClosed(); 83 | return position; 84 | } 85 | set 86 | { 87 | CheckClosed(); 88 | position = value; 89 | } 90 | } 91 | /// 92 | /// Gets the stream Id 93 | /// 94 | public Guid StreamId 95 | { 96 | get { return metadata.StreamId; } 97 | } 98 | /// 99 | /// Gets the tag associated with stream 100 | /// 101 | public int Tag 102 | { 103 | get { return metadata.Tag; } 104 | } 105 | /// 106 | /// Gets if stream can be read. 107 | /// 108 | public override bool CanRead 109 | { 110 | get { return true; } 111 | } 112 | /// 113 | /// Gets if stream can be seeked 114 | /// 115 | public override bool CanSeek 116 | { 117 | get { return true; } 118 | } 119 | /// 120 | /// Gets if stream can be written 121 | /// 122 | public override bool CanWrite 123 | { 124 | get { return true; } 125 | } 126 | /// 127 | /// Gets the stream length 128 | /// 129 | public override long Length 130 | { 131 | get 132 | { 133 | CheckClosed(); 134 | 135 | return metadata.Length; 136 | } 137 | } 138 | 139 | internal IEnumerable Segments 140 | { 141 | get { return segments; } 142 | } 143 | #endregion 144 | 145 | #region Public methods 146 | /// 147 | /// Flushes all changes 148 | /// 149 | public override void Flush() 150 | { 151 | CheckClosed(); 152 | } 153 | /// 154 | /// Reads from stream 155 | /// 156 | public override int Read(byte[] buffer, int offset, int count) 157 | { 158 | CheckClosed(); 159 | return ReadWriteData(buffer, offset, count, false); 160 | } 161 | /// 162 | /// Writes to stream 163 | /// 164 | public override void Write(byte[] buffer, int offset, int count) 165 | { 166 | CheckClosed(); 167 | 168 | storage.StartTransaction(); 169 | try 170 | { 171 | long newLength = position + count; 172 | if (newLength > metadata.Length) 173 | SetLength(newLength); 174 | 175 | // If write position is placed after initialized size, stream must be initialized up to the write position 176 | if (position > metadata.InitializedLength) 177 | { 178 | int bytesToWrite = (int)(position - metadata.InitializedLength); 179 | 180 | position = metadata.InitializedLength; 181 | while (bytesToWrite > 0) 182 | { 183 | int bytesWritten = Math.Min(Tools.EmptyBuffer.Length, bytesToWrite); 184 | Write(Tools.EmptyBuffer, 0, bytesWritten); 185 | bytesToWrite -= bytesWritten; 186 | } 187 | } 188 | 189 | ReadWriteData(buffer, offset, count, true); 190 | storage.CommitTransaction(); 191 | } 192 | catch 193 | { 194 | storage.RollbackTransaction(); 195 | throw; 196 | } 197 | } 198 | /// 199 | /// Helper method doing reading or writing 200 | /// 201 | private int ReadWriteData(byte[] buffer, int offset, int count, bool doWrite) 202 | { 203 | count = Math.Min(buffer.Length - offset, count); 204 | // Limit amount of read data to stream length 205 | if (!doWrite) 206 | count = (int)Math.Min(count, metadata.Length - position); 207 | // Read up to initialized size, then fill output with zeros 208 | int realCount = doWrite ? count : (int)Math.Min(count, metadata.InitializedLength - position); 209 | int fillCount = count - realCount; 210 | long positionInSegment = position; 211 | bool canReadOrWrite = false; 212 | 213 | var node = segments.First; 214 | while (realCount > 0) 215 | { 216 | if (canReadOrWrite) 217 | { 218 | storage.MasterStream.Position = node.Value.DataAreaStart + positionInSegment; 219 | int bytesToReadOrWrite = Math.Min(realCount, (int)(node.Value.DataAreaEnd - (node.Value.DataAreaStart + positionInSegment))); 220 | if (doWrite) 221 | storage.MasterStream.Write(buffer, offset, bytesToReadOrWrite); 222 | else 223 | storage.MasterStream.Read(buffer, offset, bytesToReadOrWrite); 224 | 225 | realCount -= bytesToReadOrWrite; 226 | offset += bytesToReadOrWrite; 227 | 228 | node = node.Next; 229 | positionInSegment = 0; 230 | } 231 | else 232 | { 233 | // Check if position is witheen current segment 234 | if (positionInSegment < node.Value.DataAreaSize) 235 | { 236 | canReadOrWrite = true; 237 | } 238 | else 239 | { 240 | positionInSegment -= node.Value.DataAreaSize; 241 | node = node.Next; 242 | } 243 | } 244 | } 245 | 246 | if (!doWrite) 247 | { 248 | // Fill buffer with zeros only when reading 249 | while (fillCount > 0) 250 | { 251 | int bytesToCopy = Math.Min(fillCount, Tools.EmptyBuffer.Length); 252 | Array.Copy(Tools.EmptyBuffer, 0, buffer, offset, bytesToCopy); 253 | fillCount -= bytesToCopy; 254 | } 255 | } 256 | 257 | position += count; 258 | 259 | if (doWrite && position > metadata.InitializedLength) 260 | { 261 | metadata.InitializedLength = position; 262 | NotifyChanged(StorageStreamChangeType.SegmentsAndMetadata); 263 | } 264 | 265 | return count; 266 | } 267 | /// 268 | /// Seek the stream 269 | /// 270 | public override long Seek(long offset, SeekOrigin origin) 271 | { 272 | CheckClosed(); 273 | switch (origin) 274 | { 275 | case SeekOrigin.Begin: 276 | position = offset; 277 | break; 278 | case SeekOrigin.Current: 279 | position += offset; 280 | break; 281 | case SeekOrigin.End: 282 | position = metadata.Length - offset; 283 | break; 284 | } 285 | 286 | return position; 287 | } 288 | /// 289 | /// Sets stream length 290 | /// 291 | public override void SetLength(long value) 292 | { 293 | CheckClosed(); 294 | 295 | if (value == metadata.Length) 296 | return; 297 | 298 | storage.StartTransaction(); 299 | 300 | try 301 | { 302 | if (value > metadata.Length) 303 | { 304 | var list = storage.FreeSpaceStream.DeallocateSpace(value - metadata.Length); 305 | 306 | AddSegments(list); 307 | } 308 | else if (value < metadata.Length) 309 | { 310 | if (value == 0) 311 | { 312 | // Move all segments to free space stream 313 | storage.FreeSpaceStream.AddSegments(segments.ToList()); 314 | segments.Clear(); 315 | metadata.Length = 0; 316 | 317 | RebuildChain(); 318 | } 319 | else 320 | { 321 | var list = DeallocateSpace(metadata.Length - value); 322 | storage.FreeSpaceStream.AddSegments(list); 323 | } 324 | } 325 | // Stream table size is always sum of the segments because itself has no entry in stream table 326 | // and thus has no stream metadata record to store length in 327 | if (StreamId == SystemStreamId.StreamTable) 328 | metadata.Length = segments.Sum(x => x.DataAreaSize); 329 | else 330 | metadata.Length = value; 331 | storage.CommitTransaction(); 332 | } 333 | catch 334 | { 335 | storage.RollbackTransaction(); 336 | throw; 337 | } 338 | } 339 | /// 340 | /// Closes the stream 341 | /// 342 | public override void Close() 343 | { 344 | storage.StartTransaction(); 345 | try 346 | { 347 | if (!isClosed) 348 | { 349 | Save(); 350 | InternalClose(); 351 | } 352 | storage.CommitTransaction(); 353 | } 354 | catch 355 | { 356 | storage.RollbackTransaction(); 357 | throw; 358 | } 359 | } 360 | #endregion 361 | 362 | #region Internal methods 363 | /// 364 | /// Saves the changes for segments and metadata 365 | /// 366 | internal void Save() 367 | { 368 | CheckClosed(); 369 | 370 | foreach (var segment in segments) 371 | { 372 | segment.Save(storage.MasterStream); 373 | } 374 | metadata.Save(); 375 | changeNotified = false; 376 | } 377 | /// 378 | /// Gets the extents of stream segments 379 | /// 380 | internal List GetStreamExtents() 381 | { 382 | return segments.Select(x => new SegmentExtent(x.Location, x.Size)).ToList(); 383 | } 384 | // Closes stream because it was created during transaction 385 | internal void InternalClose() 386 | { 387 | NotifyChanged(StorageStreamChangeType.Closing); 388 | segments.Clear(); 389 | isClosed = true; 390 | 391 | } 392 | // Reloads segments from storage 393 | internal void ReloadSegmentsOnRollback(StorageStreamMetadata metadata) 394 | { 395 | CheckClosed(); 396 | changeNotified = false; 397 | 398 | // Reaload metadata and segments 399 | LoadStream(metadata); 400 | } 401 | #endregion 402 | 403 | #region Private methods 404 | /// 405 | /// Calculates size of splitted segment based on split location (beginning or end) and size of 406 | /// resulting segments. Segment sizes are always a multiple of block size. 407 | /// When removing space from beginning (only when segment belongs to empty space stream), splitted segment 408 | /// data area size must be equal or more than the amount because there must be enough space for the amount 409 | /// of data. 410 | /// When removing from end (only when segment belongs to any other stream and not empty space stream), 411 | /// segment data area must be equal or less than the amount because more than the required amount would 412 | /// be removed. 413 | /// 414 | private static SplitData CalculateSplittedSegmentSize(Segment segmentToSplit, long amountToRemove, bool splitAtEnd, long blockSize) 415 | { 416 | long newSegmentSize = splitAtEnd ? amountToRemove - Segment.StructureSize : amountToRemove + Segment.StructureSize; 417 | bool isRounded = newSegmentSize % blockSize == 0; 418 | // Round size down to multiple of block size 419 | newSegmentSize = (newSegmentSize / blockSize) * blockSize; 420 | 421 | // Round size up to multiple of block size 422 | if (!splitAtEnd && !isRounded) 423 | { 424 | newSegmentSize += blockSize; 425 | } 426 | 427 | long leftoverSegmentSize = segmentToSplit.Size - newSegmentSize; 428 | 429 | if (leftoverSegmentSize < blockSize) 430 | { 431 | // take whole segment if leftover segment is smaller than block size 432 | return new SplitData { SplittedSegmentSize = segmentToSplit.Size, TakeWholeSegment = true }; 433 | } 434 | else 435 | { 436 | return new SplitData { SplittedSegmentSize = newSegmentSize, TakeWholeSegment = false }; 437 | } 438 | } 439 | /// 440 | /// Deallocates space from this stream 441 | /// 442 | /// Amount to take away from stream length 443 | /// Segments taken from the stream 444 | private List DeallocateSpace(long amount) 445 | { 446 | List list = new List(); 447 | 448 | if (amount > Length) 449 | amount = Length; 450 | 451 | if (metadata.StreamId == SystemStreamId.EmptySpace) 452 | { 453 | var node = segments.First; 454 | var nextNode = node.Next; 455 | 456 | while (amount > 0) 457 | { 458 | SplitData splitData = CalculateSplittedSegmentSize(node.Value, amount, false, storage.BlockSize); 459 | 460 | if (splitData.TakeWholeSegment) 461 | { 462 | // deallocate whole block 463 | list.Add(node.Value); 464 | amount -= node.Value.DataAreaSize; 465 | segments.Remove(node); 466 | node = nextNode; 467 | nextNode = node != null ? node.Next : null; 468 | } 469 | else 470 | { 471 | Segment splitSegment = node.Value.Split(splitData.SplittedSegmentSize, false); 472 | 473 | amount -= splitSegment.DataAreaSize; 474 | list.Add(splitSegment); 475 | } 476 | } 477 | } 478 | else 479 | { 480 | var node = segments.Last; 481 | var prevNode = node.Previous; 482 | 483 | while (amount > 0) 484 | { 485 | SplitData splitData = CalculateSplittedSegmentSize(node.Value, amount, true, storage.BlockSize); 486 | 487 | // If zero, segment can't be split because resulting segment size would be less than block size 488 | if (splitData.SplittedSegmentSize == 0) 489 | break; 490 | 491 | if (splitData.TakeWholeSegment) 492 | { 493 | // deallocate whole block 494 | list.Add(node.Value); 495 | amount -= node.Value.DataAreaSize; 496 | segments.Remove(node); 497 | node = prevNode; 498 | prevNode = node != null ? node.Previous : null; 499 | } 500 | else 501 | { 502 | Segment splitSegment = node.Value.Split(splitData.SplittedSegmentSize, true); 503 | amount -= splitSegment.DataAreaSize; 504 | list.Add(splitSegment); 505 | } 506 | } 507 | } 508 | 509 | RebuildChain(); 510 | 511 | return list; 512 | } 513 | /// 514 | /// Adds segments to the stream to make it longer 515 | /// 516 | private void AddSegments(List list) 517 | { 518 | // For empty space segments must be sorted by location. For other streams segments are added 519 | // to the end because existing segments must be preserved 520 | if (metadata.StreamId == SystemStreamId.EmptySpace) 521 | { 522 | // Sort new segments 523 | list = list 524 | .OrderBy(x => x.Location) 525 | .ToList(); 526 | 527 | // Insert new segments into the chain so that segments are sorted by location 528 | var node = segments.First; 529 | int listIndex = 0; 530 | while (listIndex < list.Count) 531 | { 532 | if (node != null) 533 | { 534 | if (list[listIndex].Location < node.Value.Location) 535 | { 536 | segments.AddBefore(node, list[listIndex]); 537 | listIndex++; 538 | } 539 | else 540 | node = node.Next; 541 | } 542 | else 543 | { 544 | segments.AddLast(list[listIndex]); 545 | listIndex++; 546 | } 547 | } 548 | } 549 | else 550 | { 551 | list.ForEach(x => segments.AddLast(x)); 552 | } 553 | 554 | RebuildChain(); 555 | } 556 | /// 557 | /// Rebuilds segment chain after changes where done. This method adjusts pointes to the next segment, 558 | /// merges adjacent segments together and updates stream metadata. 559 | /// 560 | private void RebuildChain() 561 | { 562 | // Merge adjacent segments 563 | var node = segments.First; 564 | // Update metadata 565 | metadata.FirstSegmentPosition = node != null ? node.Value.Location : (long?)null; 566 | 567 | while (node != null) 568 | { 569 | // Chek if this and next segment can be merged 570 | if (node.Next != null && node.Value.DataAreaEnd == node.Next.Value.Location) 571 | { 572 | node.Value.Size += node.Next.Value.Size; 573 | segments.Remove(node.Next); 574 | } 575 | else 576 | { 577 | node.Value.NextLocation = node.Next != null ? node.Next.Value.Location : (long?)null; 578 | node = node.Next; 579 | } 580 | } 581 | 582 | NotifyChanged(StorageStreamChangeType.SegmentsAndMetadata); 583 | } 584 | private void CheckClosed() 585 | { 586 | if (isClosed) 587 | throw new StreamClosedException(); 588 | } 589 | 590 | /// 591 | /// Informs listener that stream has changed 592 | /// 593 | private void NotifyChanged(StorageStreamChangeType changeType) 594 | { 595 | switch (changeType) 596 | { 597 | case StorageStreamChangeType.SegmentsAndMetadata: 598 | if (!changeNotified) 599 | { 600 | changeNotified = true; 601 | 602 | //if (Changed != null) 603 | // Changed(this, new StorageStreamChangedArgs(this, changeType)); 604 | storage.StreamChanged(changeType, this); 605 | } 606 | break; 607 | case StorageStreamChangeType.Closing: 608 | // When closing, always notify the storage 609 | storage.StreamChanged(changeType, this); 610 | break; 611 | } 612 | } 613 | #endregion 614 | 615 | #region Events 616 | //internal event EventHandler Changed; 617 | #endregion 618 | } 619 | 620 | internal enum StorageStreamChangeType { SegmentsAndMetadata, Closing } 621 | 622 | internal struct SplitData 623 | { 624 | public long SplittedSegmentSize; 625 | public bool TakeWholeSegment; 626 | } 627 | } 628 | -------------------------------------------------------------------------------- /master/Storage/StorageStreamMetadata.cs: -------------------------------------------------------------------------------- 1 | //Copyright (c) 2012 Tomaz Koritnik 2 | 3 | //Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation 4 | //files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, 5 | //modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the 6 | //Software is furnished to do so, subject to the following conditions: 7 | 8 | //The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 9 | 10 | //THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE 11 | //WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 12 | //COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 13 | //ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 14 | 15 | using System; 16 | using System.IO; 17 | 18 | namespace TmFramework.TmStorage 19 | { 20 | /// 21 | /// Holds stream metadata 22 | /// 23 | internal class StorageStreamMetadata 24 | { 25 | #region Fields 26 | private Stream ownerStream; 27 | #endregion 28 | 29 | #region Properties 30 | /// 31 | /// Length up to which stream data is initialized 32 | /// 33 | public long InitializedLength { get; set; } 34 | /// 35 | /// Stream length 36 | /// 37 | public long Length { get; set; } 38 | /// 39 | /// Stream Id 40 | /// 41 | public Guid StreamId { get; set; } 42 | /// 43 | /// Location of the first stream segment or null if stream is empty 44 | /// 45 | public long? FirstSegmentPosition { get; set; } 46 | /// 47 | /// Custom information associated with the stream 48 | /// 49 | public int Tag { get; set; } 50 | /// 51 | /// Size of this structure 52 | /// 53 | public static int StructureSize 54 | { 55 | get { return 48; } 56 | } 57 | 58 | /// 59 | /// Index of stream table entry 60 | /// 61 | public int StreamTableIndex { get; set; } 62 | #endregion 63 | 64 | #region Construction 65 | /// 66 | /// Constructor 67 | /// 68 | /// 69 | internal StorageStreamMetadata(Stream ownerStream) 70 | { 71 | this.ownerStream = ownerStream; 72 | } 73 | #endregion 74 | 75 | #region Public methods 76 | /// 77 | /// Loads metadata 78 | /// 79 | public static StorageStreamMetadata Load(Stream streamTableStream, int streamTableIndex) 80 | { 81 | streamTableStream.Position = streamTableIndex * StorageStreamMetadata.StructureSize; 82 | streamTableStream.Read(Tools.Buffer, 0, StorageStreamMetadata.StructureSize); 83 | Tools.BufferReader.BaseStream.Position = 0; 84 | 85 | StorageStreamMetadata metadata = new StorageStreamMetadata(streamTableStream); 86 | 87 | metadata.StreamTableIndex = streamTableIndex; 88 | metadata.StreamId = new Guid(Tools.BufferReader.ReadBytes(16)); 89 | metadata.Length = Tools.BufferReader.ReadInt64(); 90 | metadata.InitializedLength = Tools.BufferReader.ReadInt64(); 91 | long firstSegmentPos = Tools.BufferReader.ReadInt64(); 92 | metadata.FirstSegmentPosition = firstSegmentPos != 0 ? firstSegmentPos : (long?)null; 93 | metadata.Tag = Tools.BufferReader.ReadInt32(); 94 | int hash = Tools.BufferReader.ReadInt32(); 95 | int calculatedHash = Tools.CalculateHash(metadata.StreamId, metadata.Length, metadata.InitializedLength, firstSegmentPos); 96 | if (hash != calculatedHash) 97 | throw new StorageException("Error loading stream metadata"); 98 | 99 | return metadata; 100 | } 101 | /// 102 | /// Saves metadata 103 | /// 104 | public void Save() 105 | { 106 | if (ownerStream != null && StreamTableIndex != -1) 107 | { 108 | Tools.BufferWriter.Seek(0, SeekOrigin.Begin); 109 | 110 | Tools.BufferWriter.Write(StreamId.ToByteArray()); 111 | Tools.BufferWriter.Write(Length); 112 | Tools.BufferWriter.Write(InitializedLength); 113 | long firstSegmentPos = FirstSegmentPosition.HasValue ? FirstSegmentPosition.Value : (long)0; 114 | Tools.BufferWriter.Write(firstSegmentPos); 115 | Tools.BufferWriter.Write(Tag); 116 | 117 | int hash = Tools.CalculateHash(StreamId, Length, InitializedLength, firstSegmentPos); 118 | Tools.BufferWriter.Write(hash); 119 | 120 | ownerStream.Position = StreamTableIndex * StorageStreamMetadata.StructureSize; 121 | ownerStream.Write(Tools.Buffer, 0, StorageStreamMetadata.StructureSize); 122 | } 123 | } 124 | #endregion 125 | } 126 | } 127 | -------------------------------------------------------------------------------- /master/Storage/StreamTable.cs: -------------------------------------------------------------------------------- 1 | //Copyright (c) 2012 Tomaz Koritnik 2 | 3 | //Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation 4 | //files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, 5 | //modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the 6 | //Software is furnished to do so, subject to the following conditions: 7 | 8 | //The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 9 | 10 | //THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE 11 | //WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 12 | //COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 13 | //ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 14 | 15 | using System; 16 | using System.Collections.Generic; 17 | using System.IO; 18 | 19 | namespace TmFramework.TmStorage 20 | { 21 | /// 22 | /// Table that maps stream Id to location of first segment in the master stream and some additional data 23 | /// 24 | internal class StreamTable 25 | { 26 | #region Fields 27 | // Maps StreamId to index in stream table entry 28 | private Dictionary items = new Dictionary(); 29 | // Stores bits that determine if table entry is used or not 30 | private StreamTableMap map = new StreamTableMap(); 31 | // Stream holding stream table 32 | private StorageStream stream; 33 | private BinaryWriter writer; 34 | private Dictionary entriesAddedInTransaction = new Dictionary(); 35 | #endregion 36 | 37 | #region Construction 38 | /// 39 | /// Create stream table 40 | /// 41 | /// Owner storage 42 | public StreamTable(StorageStream stream) 43 | { 44 | this.stream = stream; 45 | 46 | writer = new BinaryWriter(stream); 47 | LoadStreamTable(); 48 | } 49 | #endregion 50 | 51 | #region Properties 52 | /// 53 | /// Returns number of entries in the table 54 | /// 55 | public int Count 56 | { 57 | get { return items.Count; } 58 | } 59 | #endregion 60 | 61 | #region Public methods 62 | /// 63 | /// Gets first segment location for specified streamId or nulll if not found. 64 | /// 65 | public StorageStreamMetadata Get(Guid streamId) 66 | { 67 | StorageStreamMetadata result = null; 68 | 69 | int index; 70 | // Search through entries in memory 71 | if (!entriesAddedInTransaction.TryGetValue(streamId, out result)) 72 | { 73 | // Load it from stream table stream 74 | if (items.TryGetValue(streamId, out index)) 75 | { 76 | result = StorageStreamMetadata.Load(stream, index); 77 | } 78 | } 79 | 80 | return result; 81 | } 82 | /// 83 | /// Returns table entries 84 | /// 85 | public IEnumerable Get() 86 | { 87 | StorageStreamMetadata streamMetadata; 88 | 89 | foreach (StorageStreamMetadata entry in entriesAddedInTransaction.Values) 90 | { 91 | yield return entry; 92 | } 93 | for (int i = 0; i < stream.Length / StorageStreamMetadata.StructureSize; i++) 94 | { 95 | streamMetadata = StorageStreamMetadata.Load(stream, i); 96 | if (streamMetadata.StreamId != Guid.Empty) 97 | yield return streamMetadata; 98 | } 99 | } 100 | /// 101 | /// Adds new entry 102 | /// 103 | public StorageStreamMetadata Add(Guid streamId, int tag = 0) 104 | { 105 | // Get the position of first empty entry 106 | int index = map.FindFirstEmptyEntry(); 107 | long streamPosition = index * StorageStreamMetadata.StructureSize; 108 | 109 | // Resize stream is needed 110 | if ((streamPosition + StorageStreamMetadata.StructureSize) > stream.Length) 111 | { 112 | int count = (int)stream.Length / StorageStreamMetadata.StructureSize; 113 | count += Math.Min(Math.Max((int)(count * 1.5), 512), 50000); 114 | 115 | long oldLength = stream.Length; 116 | stream.SetLength(count * StorageStreamMetadata.StructureSize); 117 | 118 | // Write zeros to newly allocated space 119 | long bytesToWrite = stream.Length - oldLength; 120 | stream.Position = oldLength; 121 | while (bytesToWrite > 0) 122 | { 123 | int amount = (int)Math.Min(bytesToWrite, Tools.EmptyBuffer.Length); 124 | stream.Write(Tools.EmptyBuffer, 0, amount); 125 | bytesToWrite -= amount; 126 | } 127 | 128 | stream.Save(); 129 | } 130 | 131 | StorageStreamMetadata streamMetadata = new StorageStreamMetadata(stream) 132 | { 133 | FirstSegmentPosition = null, 134 | StreamId = streamId, 135 | Tag = tag, 136 | StreamTableIndex = index 137 | }; 138 | entriesAddedInTransaction.Add(streamId, streamMetadata); 139 | //streamMetadata.Save(); 140 | 141 | map.Set(index, true); 142 | items.Add(streamId, index); 143 | 144 | return streamMetadata; 145 | } 146 | /// 147 | /// Removes entry 148 | /// 149 | /// 150 | public void Remove(Guid streamId) 151 | { 152 | int index = items[streamId]; 153 | 154 | if (entriesAddedInTransaction.ContainsKey(streamId)) 155 | { 156 | entriesAddedInTransaction.Remove(streamId); 157 | } 158 | else 159 | { 160 | // Clear entry 161 | stream.Position = index * StorageStreamMetadata.StructureSize; 162 | 163 | byte[] buf = new byte[48]; 164 | writer.Write(buf, 0, buf.Length); 165 | } 166 | 167 | map.Set(index, false); 168 | items.Remove(streamId); 169 | } 170 | /// 171 | /// Returns whether stream exists in table 172 | /// 173 | public bool Contains(Guid streamId) 174 | { 175 | return items.ContainsKey(streamId); 176 | } 177 | /// 178 | /// Rolls back transaction 179 | /// 180 | public void RollbackTransaction() 181 | { 182 | entriesAddedInTransaction.Clear(); 183 | LoadStreamTable(); 184 | } 185 | public void SaveChanges() 186 | { 187 | foreach (var entry in entriesAddedInTransaction.Values) 188 | { 189 | entry.Save(); 190 | } 191 | entriesAddedInTransaction.Clear(); 192 | } 193 | #endregion 194 | 195 | #region Private methods 196 | private void LoadStreamTable() 197 | { 198 | StorageStreamMetadata streamMetadata; 199 | 200 | stream.Position = 0; 201 | int index = 0; 202 | map.Clear(); 203 | 204 | items.Clear(); 205 | 206 | int cnt = (int)stream.Length / StorageStreamMetadata.StructureSize; 207 | for (int i = 0; i < cnt; i++) 208 | { 209 | streamMetadata = StorageStreamMetadata.Load(stream, i); 210 | 211 | if (streamMetadata.StreamId != Guid.Empty) 212 | { 213 | map.Set(index, true); 214 | items.Add(streamMetadata.StreamId, index); 215 | } 216 | 217 | index++; 218 | } 219 | } 220 | #endregion 221 | } 222 | } 223 | -------------------------------------------------------------------------------- /master/Storage/StreamTableMap.cs: -------------------------------------------------------------------------------- 1 | //Copyright (c) 2012 Tomaz Koritnik 2 | 3 | //Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation 4 | //files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, 5 | //modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the 6 | //Software is furnished to do so, subject to the following conditions: 7 | 8 | //The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 9 | 10 | //THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE 11 | //WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 12 | //COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 13 | //ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 14 | 15 | using System.Collections.Generic; 16 | 17 | namespace TmFramework.TmStorage 18 | { 19 | /// 20 | /// Provides an information which entries in stream table are used and which are not 21 | /// 22 | internal class StreamTableMap 23 | { 24 | private List entries = new List(); 25 | private LinkedList freeEntries = new LinkedList(); 26 | private int count = 0; 27 | 28 | /*public bool Get(int index) 29 | { 30 | // Just return that it's not set if index is above number of items - behave as infinite size list 31 | if (index > count - 1) 32 | return false; 33 | 34 | int entryIndex = index / 32; 35 | int entryOffset = index % 32; 36 | 37 | int value = entries[entryIndex]; 38 | int mask = 1 << entryOffset; 39 | bool result = (value & mask) == mask; 40 | 41 | return result; 42 | }*/ 43 | public void Set(int index, bool newValue) 44 | { 45 | // Enlarge list 46 | if (index > count - 1) 47 | { 48 | while (index > count - 1) 49 | { 50 | int amount = 12500; // make room for another 100.000 entries 51 | count += amount; 52 | 53 | entries.Capacity += amount; 54 | for (int i = 0; i < amount; i++) 55 | { 56 | entries.Add(0); 57 | } 58 | } 59 | } 60 | 61 | // Set bit 62 | int entryIndex = index / 32; 63 | int entryOffset = index % 32; 64 | 65 | int value = entries[entryIndex]; 66 | int mask = 1 << entryOffset; 67 | 68 | if (newValue) 69 | { 70 | value = value | mask; 71 | } 72 | else 73 | { 74 | // invert mask 75 | int maskWithAllBitsSet = -1; 76 | mask = maskWithAllBitsSet ^ mask; 77 | 78 | value = value & mask; 79 | } 80 | 81 | entries[entryIndex] = value; 82 | } 83 | public void Clear() 84 | { 85 | entries.Clear(); 86 | freeEntries.Clear(); 87 | count = 0; 88 | } 89 | 90 | public int FindFirstEmptyEntry() 91 | { 92 | // Search free entries for empty entry 93 | LinkedListNode node = freeEntries.First; 94 | while (node != null) 95 | { 96 | var nextNode = node.Next; 97 | 98 | if (entries[node.Value] != -1) 99 | break; 100 | else 101 | freeEntries.Remove(node); // Entry can be full so remove it 102 | 103 | node = nextNode; 104 | } 105 | 106 | // If empty entry is not found, refill the list 107 | if (node == null) 108 | { 109 | FillFreeEntries(); 110 | node = freeEntries.First; 111 | } 112 | 113 | if (node != null) 114 | { 115 | int index = node.Value; 116 | 117 | for (int bitIndex = 0; bitIndex < 32; bitIndex++) 118 | { 119 | int mask = 1 << bitIndex; 120 | int maskWithAllBitsSet = -1; 121 | mask = maskWithAllBitsSet ^ mask; 122 | 123 | if ((entries[index] & mask) == entries[index]) 124 | { 125 | return index * 32 + bitIndex; 126 | } 127 | } 128 | } 129 | 130 | // All are set so return index of last entry + 1 131 | return count; 132 | } 133 | 134 | private void FillFreeEntries() 135 | { 136 | for (int i = 0; i < entries.Count; i++) 137 | { 138 | if (freeEntries.Count >= 20) 139 | break; 140 | 141 | if (entries[i] != -1) 142 | freeEntries.AddLast(i); 143 | } 144 | } 145 | } 146 | } 147 | -------------------------------------------------------------------------------- /master/Storage/TmStorage.csproj: -------------------------------------------------------------------------------- 1 |  2 | 3 | 4 | 5 | Debug 6 | AnyCPU 7 | {7121554B-164C-454C-8CBE-374BCA12AB08} 8 | Library 9 | Properties 10 | TmFramework.TmStorage 11 | TmFramework.TmStorage 12 | v4.5 13 | 512 14 | 15 | 16 | true 17 | full 18 | false 19 | bin\Debug\ 20 | DEBUG;TRACE 21 | prompt 22 | 4 23 | AnyCPU 24 | false 25 | AllRules.ruleset 26 | 27 | 28 | pdbonly 29 | true 30 | bin\Release\ 31 | TRACE 32 | prompt 33 | 4 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 73 | -------------------------------------------------------------------------------- /master/Storage/Tools.cs: -------------------------------------------------------------------------------- 1 | //Copyright (c) 2012 Tomaz Koritnik 2 | 3 | //Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation 4 | //files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, 5 | //modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the 6 | //Software is furnished to do so, subject to the following conditions: 7 | 8 | //The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 9 | 10 | //THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE 11 | //WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 12 | //COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 13 | //ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 14 | 15 | using System; 16 | using System.IO; 17 | 18 | namespace TmFramework.TmStorage 19 | { 20 | internal static class Tools 21 | { 22 | public readonly static byte[] EmptyBuffer = new byte[32768]; 23 | 24 | #region Buffering 25 | public static byte[] Buffer; 26 | private static MemoryStream bufferStream; 27 | public static BinaryReader BufferReader { get; private set; } 28 | public static BinaryWriter BufferWriter { get; private set; } 29 | #endregion Buffering 30 | 31 | static Tools() 32 | { 33 | Buffer = new byte[128]; 34 | bufferStream = new MemoryStream(Buffer); 35 | BufferReader = new BinaryReader(bufferStream); 36 | BufferWriter = new BinaryWriter(bufferStream); 37 | } 38 | 39 | /// 40 | /// Calculates hash over many parameters 41 | /// 42 | /// 43 | /// 44 | public static int CalculateHash(params object[] values) 45 | { 46 | int hash = 0; 47 | foreach (var value in values) 48 | { 49 | hash ^= value.GetHashCode(); 50 | } 51 | 52 | return hash; 53 | } 54 | 55 | /*/// 56 | /// Saves stream to file 57 | /// 58 | /// 59 | /// 60 | public static void Debug_SaveStream(Stream stream, string filename) 61 | { 62 | stream.Position = 0; 63 | using (FileStream f = new FileStream(filename, FileMode.Create)) 64 | { 65 | byte[] buf = new byte[stream.Length]; 66 | stream.Read(buf, 0, buf.Length); 67 | f.Write(buf, 0, buf.Length); 68 | f.Close(); 69 | } 70 | } 71 | 72 | public static void Debug_ListSegments(StorageStream stream) 73 | { 74 | if (stream != null) 75 | { 76 | foreach (var segment in stream.Segments) 77 | { 78 | System.Diagnostics.Debug.Write(string.Format("Loc: {0} Size: {1} # ", segment.Location, segment.Size)); 79 | System.Diagnostics.Debug.WriteLine(string.Empty); 80 | } 81 | } 82 | }*/ 83 | } 84 | } 85 | -------------------------------------------------------------------------------- /master/Storage/Transactions/Segment.cs: -------------------------------------------------------------------------------- 1 | //Copyright (c) 2012 Tomaz Koritnik 2 | 3 | //Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation 4 | //files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, 5 | //modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the 6 | //Software is furnished to do so, subject to the following conditions: 7 | 8 | //The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 9 | 10 | //THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE 11 | //WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 12 | //COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 13 | //ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 14 | 15 | using System; 16 | 17 | namespace TmFramework.Transactions 18 | { 19 | /// 20 | /// Segment holds information about area that have already been backed up or doesn't require backup 21 | /// 22 | internal class Segment 23 | { 24 | #region Public fields 25 | /// 26 | /// Segment start location in bytes 27 | /// 28 | public long Location; 29 | /// 30 | /// Segment size in bytes 31 | /// 32 | public long Size; 33 | #endregion 34 | 35 | #region Construction 36 | /// 37 | /// COnstruct a segment 38 | /// 39 | public Segment(long location, long size) 40 | { 41 | this.Location = location; 42 | this.Size = size; 43 | } 44 | #endregion 45 | } 46 | } 47 | -------------------------------------------------------------------------------- /master/Storage/Transactions/SegmentMetadata.cs: -------------------------------------------------------------------------------- 1 | //Copyright (c) 2012 Tomaz Koritnik 2 | 3 | //Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation 4 | //files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, 5 | //modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the 6 | //Software is furnished to do so, subject to the following conditions: 7 | 8 | //The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 9 | 10 | //THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE 11 | //WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 12 | //COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 13 | //ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 14 | 15 | using System; 16 | using System.IO; 17 | 18 | namespace TmFramework.Transactions 19 | { 20 | /// 21 | /// Holds the data about 22 | /// 23 | class SegmentMetadata 24 | { 25 | private Guid transactionId; 26 | public Guid TransactionId 27 | { 28 | get { return transactionId; } 29 | set 30 | { 31 | transactionId = value; 32 | } 33 | } 34 | 35 | public long Position { get; set; } 36 | public int Size { get; set; } 37 | 38 | public void Save(Stream stream) 39 | { 40 | BinaryWriter writer = new BinaryWriter(stream); 41 | writer.Write(TransactionId.ToByteArray()); 42 | writer.Write(Position); 43 | writer.Write(Size); 44 | } 45 | public static SegmentMetadata Load(Stream stream) 46 | { 47 | BinaryReader reader = new BinaryReader(stream); 48 | SegmentMetadata sh = new SegmentMetadata 49 | { 50 | TransactionId = new Guid(reader.ReadBytes(16)), 51 | Position = reader.ReadInt64(), 52 | Size = reader.ReadInt32() 53 | }; 54 | return sh; 55 | } 56 | } 57 | } 58 | -------------------------------------------------------------------------------- /master/Storage/Transactions/TransactionLogStreamMetadata.cs: -------------------------------------------------------------------------------- 1 | //Copyright (c) 2012 Tomaz Koritnik 2 | 3 | //Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation 4 | //files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, 5 | //modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the 6 | //Software is furnished to do so, subject to the following conditions: 7 | 8 | //The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 9 | 10 | //THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE 11 | //WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 12 | //COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 13 | //ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 14 | 15 | using System; 16 | using System.IO; 17 | 18 | namespace TmFramework.Transactions 19 | { 20 | /// 21 | /// Metadata about transaction log stream 22 | /// 23 | class TransactionLogStreamMetadata 24 | { 25 | #region Public properties 26 | /// 27 | /// Id of transaction currently being used 28 | /// 29 | public Guid TransactionId { get; set; } 30 | /// 31 | /// Stores the length of master stream when transaction started. Stream size is set to this value when rollback is performed. 32 | /// 33 | public long StreamLength { get; set; } 34 | /// 35 | /// Number of segments in the transaction log 36 | /// 37 | public long SegmentCount { get; set; } 38 | /// 39 | /// Flag determines whether transaction has completed 40 | /// 41 | public bool TransactionCompleted { get; set; } 42 | /// 43 | /// Size of the structure in bytes 44 | /// 45 | public static int StructureSize 46 | { 47 | get { return 16 + 2 * sizeof(long) + sizeof(bool); } 48 | } 49 | #endregion 50 | 51 | #region Public properties 52 | /// 53 | /// Saves metadata to the beginning of the stream 54 | /// 55 | /// 56 | public void Save(Stream stream) 57 | { 58 | stream.Position = 0; 59 | BinaryWriter writer = new BinaryWriter(stream); 60 | writer.Write(TransactionId.ToByteArray()); 61 | writer.Write(StreamLength); 62 | writer.Write(SegmentCount); 63 | writer.Write(TransactionCompleted); 64 | } 65 | /// 66 | /// Loads the metadata from the beginning of the stream 67 | /// 68 | public static TransactionLogStreamMetadata Load(Stream stream) 69 | { 70 | BinaryReader reader = new BinaryReader(stream); 71 | TransactionLogStreamMetadata lsh = new TransactionLogStreamMetadata 72 | { 73 | TransactionId = new Guid(reader.ReadBytes(16)), 74 | StreamLength = reader.ReadInt64(), 75 | SegmentCount = reader.ReadInt64(), 76 | TransactionCompleted = reader.ReadBoolean() 77 | }; 78 | return lsh; 79 | } 80 | #endregion 81 | } 82 | } 83 | -------------------------------------------------------------------------------- /master/Storage/Transactions/TransactionStream.cs: -------------------------------------------------------------------------------- 1 | //Copyright (c) 2012 Tomaz Koritnik 2 | 3 | //Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation 4 | //files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, 5 | //modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the 6 | //Software is furnished to do so, subject to the following conditions: 7 | 8 | //The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 9 | 10 | //THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE 11 | //WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 12 | //COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 13 | //ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 14 | 15 | using System; 16 | using System.Collections.Generic; 17 | using System.IO; 18 | using System.Linq; 19 | 20 | namespace TmFramework.Transactions 21 | { 22 | /// 23 | /// Stream that backs up overwritten data into transaction log stream. 24 | /// 25 | internal class TransactionStream : Stream 26 | { 27 | #region Fields 28 | // Size of the minimum block size that will be backed up 29 | private long blockSize; 30 | // Segments represent space already backed up 31 | private LinkedList segments = new LinkedList(); 32 | // Stream where backed up data is stored 33 | private Stream logStream; 34 | // Log stream header 35 | private TransactionLogStreamMetadata logStreamHeader = null; 36 | // Set to true when creating the transaction stream and log stream contains unfinished transaction. 37 | private bool rollbackNeeded = false; 38 | #endregion 39 | 40 | #region Construction 41 | /// 42 | /// Consrtructs a transaction stream on top of the specified master stream 43 | /// 44 | /// Stream for which backup si performed 45 | /// Stream where backed up data is stored 46 | /// Size of the minimum block size that will be backed up 47 | public TransactionStream(Stream masterStream, Stream logStream, long blockSize) 48 | { 49 | this.masterStream = masterStream; 50 | this.logStream = logStream; 51 | this.blockSize = blockSize; 52 | 53 | // Check if previous transaction has completed 54 | if (logStream != null && logStream.Length > TransactionLogStreamMetadata.StructureSize) 55 | { 56 | logStreamHeader = TransactionLogStreamMetadata.Load(logStream); 57 | rollbackNeeded = !logStreamHeader.TransactionCompleted; 58 | if (logStreamHeader.SegmentCount == 0 || logStreamHeader.TransactionCompleted) 59 | logStreamHeader = null; 60 | } 61 | } 62 | #endregion 63 | 64 | #region Properties 65 | private Stream masterStream; 66 | /*/// 67 | /// Stream for which backup si performed 68 | /// 69 | public Stream MasterStream 70 | { 71 | get { return masterStream; } 72 | } 73 | /// 74 | /// Current transaction id when in transaction or null when transaction is active. 75 | /// 76 | public Guid? CurrentTransactionId 77 | { 78 | get { return logStreamHeader != null ? logStreamHeader.TransactionId : (Guid?)null; } 79 | }*/ 80 | #endregion 81 | 82 | #region Private methods 83 | // Checks if rollback is needed 84 | private void CheckState() 85 | { 86 | if (rollbackNeeded) 87 | throw new InvalidOperationException("Previous transaction did not complete. Rollback needed."); 88 | } 89 | // Find current or next segment at specified stream position 90 | private LinkedListNode GetCurrentSegment(long position, LinkedListNode startNode) 91 | { 92 | LinkedListNode node = startNode != null ? startNode : segments.First; 93 | while (node != null) 94 | { 95 | if (node.Value.Location + node.Value.Size > position) 96 | return node; 97 | else 98 | node = node.Next; 99 | } 100 | return null; 101 | } 102 | // Checks for overlapping blocks and merges if needed 103 | private void CheckSegments() 104 | { 105 | LinkedListNode node = segments.First; 106 | 107 | while (node != null) 108 | { 109 | LinkedListNode next = node.Next; 110 | 111 | if (next != null && (next.Value.Location <= node.Value.Location + node.Value.Size)) 112 | { 113 | node.Value.Size = Math.Max(node.Value.Location + node.Value.Size, next.Value.Location + next.Value.Size) - node.Value.Location; 114 | segments.Remove(next); 115 | } 116 | else 117 | node = next; 118 | } 119 | } 120 | // Converts size in bytes to number of blocks by rounding up 121 | private long ToBlocksRoundUp(long size) 122 | { 123 | long result = (size / blockSize) * blockSize; 124 | return size % blockSize > 0 ? result + blockSize : result; 125 | } 126 | // Converts size in bytes to number of blocks by rounding down 127 | private long ToBlocksRoundDown(long size) 128 | { 129 | return (size / blockSize) * blockSize; 130 | } 131 | // Backs up specified part of master stream. Parts already backed up are skipped. 132 | private void BackupSegment(long location, long size) 133 | { 134 | long endLocation = ToBlocksRoundUp(location + size); 135 | location = ToBlocksRoundDown(location); 136 | size = endLocation - location; 137 | 138 | LinkedListNode node = GetCurrentSegment(location, null); 139 | while (size > 0) 140 | { 141 | bool backedUpPosition = node != null ? ((location >= node.Value.Location) && (location < node.Value.Location + node.Value.Size)) : false; 142 | long amount; 143 | 144 | if (backedUpPosition) 145 | { 146 | amount = node != null ? Math.Min(node.Value.Location + node.Value.Size - location, size) : size; 147 | } 148 | else 149 | { 150 | amount = node != null ? Math.Min(node.Value.Location - location, size) : size; 151 | 152 | // Backup data 153 | BackupData(location, (int)amount); 154 | 155 | // Add backupped segment 156 | Segment newSegment = new Segment(location, amount); 157 | if (node != null) 158 | segments.AddBefore(node, newSegment); 159 | else 160 | { 161 | segments.AddLast(newSegment); 162 | node = segments.First; 163 | } 164 | } 165 | size -= amount; 166 | location += amount; 167 | 168 | node = GetCurrentSegment(location, node); 169 | } 170 | } 171 | // Copies a section of data from master stream to the log stream 172 | private void BackupData(long location, int size) 173 | { 174 | logStream.Seek(0, SeekOrigin.End); 175 | 176 | // Write segment header 177 | SegmentMetadata sh = new SegmentMetadata 178 | { 179 | TransactionId = logStreamHeader.TransactionId, 180 | Position = location, 181 | Size = size 182 | }; 183 | long metadataPos = logStream.Position; 184 | sh.Save(logStream); 185 | 186 | // Copy data 187 | masterStream.Position = location; 188 | 189 | int bytesCopied = CopyData(masterStream, size, logStream); 190 | 191 | // If bytes copied differ from size in header, update the header 192 | if (bytesCopied != size) 193 | { 194 | logStream.Position = metadataPos; 195 | 196 | sh.Size = bytesCopied; 197 | sh.Save(logStream); 198 | } 199 | 200 | logStream.Flush(); 201 | 202 | // Update log stream header 203 | logStreamHeader.SegmentCount++; 204 | logStreamHeader.Save(logStream); 205 | logStream.Flush(); 206 | } 207 | // Copies data from source to destination stream. 208 | private static int CopyData(Stream src, int size, Stream dst) 209 | { 210 | byte[] buf = new byte[32763]; 211 | int bytesCopied = 0; 212 | 213 | while (size > 0) 214 | { 215 | int bytesRead = src.Read(buf, 0, Math.Min(buf.Length, size)); 216 | if (bytesRead == 0) 217 | break; 218 | 219 | dst.Write(buf, 0, bytesRead); 220 | size -= bytesRead; 221 | bytesCopied += bytesRead; 222 | } 223 | 224 | return bytesCopied; 225 | } 226 | #endregion 227 | 228 | #region Public 229 | /// 230 | /// Start transaction 231 | /// 232 | /// Areas in master stream which don't require backing up. Can be null. 233 | public Guid BeginTransaction(IEnumerable initialSegments = null) 234 | { 235 | CheckState(); 236 | if (logStream == null) 237 | throw new InvalidOperationException("Log stream not specified"); 238 | if (logStreamHeader != null) 239 | throw new InvalidOperationException("Unable to start new transaction while existing transaction is in progress"); 240 | 241 | // Add custom segments that will not be backed up 242 | segments.Clear(); 243 | if (initialSegments != null) 244 | { 245 | IOrderedEnumerable orderedList = initialSegments.OrderBy(x => x.Location); 246 | foreach (var segment in orderedList) 247 | { 248 | segments.AddLast(segment); 249 | } 250 | } 251 | // Add unallocated space after the end of stream 252 | segments.AddLast(new Segment(masterStream.Length, long.MaxValue - masterStream.Length)); 253 | CheckSegments(); 254 | 255 | // Truncate transaction log 256 | logStream.SetLength(0); 257 | logStream.Flush(); 258 | 259 | // Initialize transaction log 260 | logStreamHeader = new TransactionLogStreamMetadata 261 | { 262 | TransactionId = Guid.NewGuid(), 263 | StreamLength = masterStream.Length, 264 | SegmentCount = 0, 265 | TransactionCompleted = false 266 | }; 267 | logStreamHeader.Save(logStream); 268 | logStream.Flush(); 269 | 270 | return logStreamHeader.TransactionId; 271 | } 272 | /// 273 | /// Marks in the log file that transaction has ended 274 | /// 275 | public void EndTransaction() 276 | { 277 | CheckState(); 278 | if (logStreamHeader != null) 279 | { 280 | masterStream.Flush(); 281 | 282 | logStreamHeader.TransactionCompleted = true; 283 | logStreamHeader.Save(logStream); 284 | logStream.Flush(); 285 | 286 | logStreamHeader = null; 287 | 288 | CheckSegments(); 289 | } 290 | } 291 | /// 292 | /// Rolls back transaction by copying all backed up data bask to master stream and closes the transaction. 293 | /// 294 | public void RollbackTransaction() 295 | { 296 | if (logStreamHeader != null) 297 | { 298 | if (logStreamHeader.TransactionCompleted) 299 | throw new InvalidOperationException("Can't rollback completed transaction"); 300 | logStream.Position = TransactionLogStreamMetadata.StructureSize; 301 | 302 | // Copy segments from log stream back to stream 303 | for (int i = 0; i < logStreamHeader.SegmentCount; i++) 304 | { 305 | SegmentMetadata sh = SegmentMetadata.Load(logStream); 306 | if (!sh.TransactionId.Equals(logStreamHeader.TransactionId)) 307 | throw new InvalidOperationException("Wrong segment found in transaction log"); 308 | masterStream.Position = sh.Position; 309 | CopyData(logStream, sh.Size, masterStream); 310 | } 311 | 312 | // Set back original stream length 313 | if (logStreamHeader.StreamLength < masterStream.Length) 314 | masterStream.SetLength(logStreamHeader.StreamLength); 315 | 316 | masterStream.Flush(); 317 | logStream.SetLength(0); 318 | logStream.Flush(); 319 | logStreamHeader = null; 320 | rollbackNeeded = false; 321 | } 322 | } 323 | #endregion 324 | 325 | #region Stream 326 | public override bool CanRead 327 | { 328 | get { return masterStream.CanRead; } 329 | } 330 | public override bool CanSeek 331 | { 332 | get { return masterStream.CanSeek; } 333 | } 334 | public override bool CanWrite 335 | { 336 | get { return masterStream.CanWrite; } 337 | } 338 | public override void Flush() 339 | { 340 | CheckState(); 341 | masterStream.Flush(); 342 | } 343 | public override long Length 344 | { 345 | get { return masterStream.Length; } 346 | } 347 | public override long Position 348 | { 349 | get { return masterStream.Position; } 350 | set { masterStream.Position = value; } 351 | } 352 | 353 | public override long Seek(long offset, SeekOrigin origin) 354 | { 355 | return masterStream.Seek(offset, origin); 356 | } 357 | public override void SetLength(long value) 358 | { 359 | CheckState(); 360 | if (logStreamHeader != null && value < masterStream.Length) 361 | BackupSegment(value, Length - value); 362 | masterStream.SetLength(value); 363 | } 364 | public override int Read(byte[] buffer, int offset, int count) 365 | { 366 | CheckState(); 367 | return masterStream.Read(buffer, offset, count); 368 | } 369 | public override void Write(byte[] buffer, int offset, int count) 370 | { 371 | CheckState(); 372 | long pos = masterStream.Position; 373 | if (logStreamHeader != null) 374 | BackupSegment(masterStream.Position, count); 375 | masterStream.Position = pos; 376 | masterStream.Write(buffer, offset, count); 377 | } 378 | #endregion 379 | } 380 | } 381 | -------------------------------------------------------------------------------- /master/Storage/WriteBufferedStream.cs: -------------------------------------------------------------------------------- 1 | //Copyright (c) 2012 Tomaz Koritnik 2 | 3 | //Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation 4 | //files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, 5 | //modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the 6 | //Software is furnished to do so, subject to the following conditions: 7 | 8 | //The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 9 | 10 | //THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE 11 | //WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 12 | //COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 13 | //ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 14 | 15 | using System; 16 | using System.Collections.Generic; 17 | using System.IO; 18 | 19 | namespace TmFramework.TmStorage 20 | { 21 | /// 22 | /// Purpose of this stream is to buffer writes during transaction. They are flushed to the underlying stream when 23 | /// transaction ends or when buffer size is exceeded. 24 | /// 25 | internal class WriteBufferedStream : Stream 26 | { 27 | #region Fields 28 | private Stream stream; 29 | private int blockSize; 30 | // These blocks hold modified data 31 | private LinkedList blocks = new LinkedList(); 32 | //private int maxBufferSize; 33 | //private int bufferedAmount = 0; 34 | #endregion 35 | 36 | #region Construction 37 | public WriteBufferedStream(Stream stream, int blockSize) 38 | { 39 | if (blockSize < 512) 40 | throw new ArgumentException("Block size cannot be less than 512 bytes"); 41 | 42 | this.stream = stream; 43 | this.blockSize = blockSize; 44 | this.length = stream.Length; 45 | //this.maxBufferSize = Math.Max(maxBufferSize, 100000); 46 | } 47 | #endregion 48 | 49 | #region Properties 50 | public override bool CanRead 51 | { 52 | get { return stream.CanRead; } 53 | } 54 | 55 | public override bool CanSeek 56 | { 57 | get { return stream.CanSeek; } 58 | } 59 | 60 | public override bool CanWrite 61 | { 62 | get { return stream.CanWrite; } 63 | } 64 | 65 | private long length = 0; 66 | public override long Length 67 | { 68 | get { return length; } 69 | } 70 | 71 | private long position = 0; 72 | public override long Position 73 | { 74 | get { return position; } 75 | set { position = value; } 76 | } 77 | 78 | private bool bufferingEnabled = true; 79 | /*public bool BufferingEnabled 80 | { 81 | get { return bufferingEnabled; } 82 | set 83 | { 84 | if (value == false && blocks.Count > 0) 85 | throw new InvalidOperationException("Can't disable buffering when data is already buffered"); 86 | 87 | bufferingEnabled = value; 88 | } 89 | }*/ 90 | #endregion 91 | 92 | #region Public methods 93 | public override void Flush() 94 | { 95 | } 96 | 97 | public override int Read(byte[] buffer, int offset, int count) 98 | { 99 | if (bufferingEnabled) 100 | { 101 | // Find block at current position. If none exists, block ahead of current position is returned. 102 | var node = GetCurrentOrNextNode(position); 103 | // Trim read amount if it exceeds stream length 104 | count = Math.Min(count, (int)(length - position)); 105 | 106 | while (count > 0) 107 | { 108 | int amount; 109 | 110 | if (node != null) 111 | { 112 | if (position < node.Value.Position) 113 | { 114 | // Position is located before start of the current block, read from base stream 115 | amount = Math.Min(count, (int)(node.Value.Position - position)); 116 | stream.Position = position; 117 | stream.Read(buffer, offset, amount); 118 | } 119 | else 120 | { 121 | // Position is inside block, read from block 122 | amount = Math.Min(count, (int)(node.Value.EndPosition - position)); 123 | Array.Copy(node.Value.Data, position - node.Value.Position, buffer, offset, amount); 124 | node = node.Next; 125 | } 126 | } 127 | else 128 | { 129 | // No block on current position or in front, read all from base stream 130 | amount = count; 131 | stream.Position = position; 132 | stream.Read(buffer, offset, amount); 133 | } 134 | 135 | offset += amount; 136 | count -= amount; 137 | position += amount; 138 | } 139 | 140 | return count; 141 | } 142 | else 143 | { 144 | stream.Position = position; 145 | int amount = stream.Read(buffer, offset, count); 146 | position = stream.Position; 147 | return amount; 148 | } 149 | } 150 | public override void Write(byte[] buffer, int offset, int count) 151 | { 152 | if (bufferingEnabled) 153 | { 154 | LinkedListNode node; 155 | 156 | if (position > length) 157 | { 158 | // Find block at current position. If none exists, block ahead of current position is returned. 159 | node = GetCurrentOrNextNode(length); 160 | 161 | // Create a block to fill the space from length to position 162 | Block newBlock = null; 163 | if (node != null) 164 | { 165 | int blockCount = CalculateNumberOfBlocks(node.Value.EndPosition, position); 166 | if (blockCount > 0) 167 | newBlock = CreateBlock(node.Value.EndPosition, blockCount); 168 | } 169 | else 170 | { 171 | int blockCount = CalculateNumberOfBlocks(length, position); 172 | if (blockCount > 0) 173 | newBlock = CreateBlock(length, blockCount); 174 | } 175 | if (newBlock != null) 176 | { 177 | blocks.AddLast(newBlock); 178 | //bufferedAmount += newBlock.Data.Length; 179 | } 180 | 181 | // Length can now be increased since space has been initialized 182 | length = position; 183 | } 184 | 185 | length = Math.Max(position + count, length); 186 | node = GetCurrentOrNextNode(position); 187 | 188 | while (count > 0) 189 | { 190 | int amount; 191 | 192 | if (node == null || (position < node.Value.Position)) 193 | { 194 | // Position is before start of the block. Allocate new block. 195 | var newNode = new LinkedListNode(CreateBlock(position, 1)); 196 | if (node != null) 197 | blocks.AddBefore(node, newNode); 198 | else 199 | blocks.AddLast(newNode); 200 | 201 | //bufferedAmount += newNode.Value.Data.Length; 202 | node = newNode; 203 | } 204 | 205 | // Write to current block 206 | amount = Math.Min(count, (int)(node.Value.EndPosition - position)); 207 | Array.Copy(buffer, offset, node.Value.Data, position - node.Value.Position, amount); 208 | node = node.Next; 209 | 210 | offset += amount; 211 | count -= amount; 212 | position += amount; 213 | } 214 | } 215 | else 216 | { 217 | stream.Position = position; 218 | stream.Write(buffer, offset, count); 219 | length = Math.Max(position + count, length); 220 | position = stream.Position; 221 | } 222 | } 223 | 224 | public override long Seek(long offset, SeekOrigin origin) 225 | { 226 | switch (origin) 227 | { 228 | case SeekOrigin.Begin: 229 | position = offset; 230 | break; 231 | case SeekOrigin.Current: 232 | position += offset; 233 | break; 234 | case SeekOrigin.End: 235 | position = length - offset; 236 | break; 237 | } 238 | 239 | return position; 240 | } 241 | 242 | /// 243 | /// Sets the length of buffered data 244 | /// 245 | /// 246 | public override void SetLength(long value) 247 | { 248 | throw new NotImplementedException(); 249 | } 250 | /// 251 | /// Flushes buffered data to the underlying stream 252 | /// 253 | public void FlushBufferedData() 254 | { 255 | foreach (var block in blocks) 256 | { 257 | stream.Position = block.Position; 258 | stream.Write(block.Data, 0, block.Data.Length); 259 | } 260 | blocks.Clear(); 261 | //bufferedAmount = 0; 262 | } 263 | /// 264 | /// Discards all buffered data 265 | /// 266 | public void DiscardBufferedData() 267 | { 268 | blocks.Clear(); 269 | //bufferedAmount = 0; 270 | } 271 | #endregion 272 | 273 | #region Private 274 | private LinkedListNode GetCurrentOrNextNode(long position) 275 | { 276 | LinkedListNode node = blocks.First; 277 | 278 | while (node != null) 279 | { 280 | if (node.Value.EndPosition > position) 281 | break; 282 | 283 | node = node.Next; 284 | } 285 | 286 | return node; 287 | } 288 | private Block CreateBlock(long pos, int groupSize) 289 | { 290 | // Check if buffer is almost full. If yes, flush 10% of blocks. 291 | /*if (bufferedAmount > maxBufferSize) 292 | { 293 | int newSize = (int)((double)maxBufferSize * 0.90); 294 | var node = blocks.First; 295 | 296 | while (bufferedAmount > newSize && node != null) 297 | { 298 | stream.Position = node.Value.Position; 299 | stream.Write(node.Value.Data, 0, node.Value.Data.Length); 300 | bufferedAmount -= node.Value.Data.Length; 301 | 302 | var nextNode = node.Next; 303 | blocks.Remove(node); 304 | node = node.Next; 305 | } 306 | 307 | if (blocks.Count == 0) 308 | bufferedAmount = 0; 309 | }*/ 310 | 311 | long blockPos = (pos / blockSize) * blockSize; 312 | Block block = new Block { Data = new byte[blockSize * groupSize], Position = blockPos }; 313 | 314 | // Fill block with data from base stream 315 | stream.Position = block.Position; 316 | stream.Read(block.Data, 0, block.Data.Length); 317 | 318 | return block; 319 | } 320 | /// 321 | /// Calculates number of blocks required to cover the range specified 322 | /// 323 | /// 324 | /// 325 | /// 326 | private int CalculateNumberOfBlocks(long pos1, long pos2) 327 | { 328 | pos1 = pos1 / (long)blockSize; 329 | 330 | long remainder = pos2 % (long)blockSize; 331 | pos2 = pos2 / (long)blockSize; 332 | if (remainder != 0) 333 | pos2++; 334 | 335 | return (int)(pos2 - pos1); 336 | } 337 | #endregion 338 | 339 | #region Classes 340 | class Block 341 | { 342 | public Block() 343 | { 344 | } 345 | 346 | public byte[] Data; 347 | public long Position; 348 | public long EndPosition 349 | { 350 | get { return Position + Data.Length; } 351 | } 352 | } 353 | #endregion Classes 354 | } 355 | } 356 | -------------------------------------------------------------------------------- /master/TmStorage.sln: -------------------------------------------------------------------------------- 1 |  2 | Microsoft Visual Studio Solution File, Format Version 12.00 3 | # Visual Studio 2013 4 | VisualStudioVersion = 12.0.21005.1 5 | MinimumVisualStudioVersion = 10.0.40219.1 6 | Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "TmStorage", "Storage\TmStorage.csproj", "{7121554B-164C-454C-8CBE-374BCA12AB08}" 7 | EndProject 8 | Global 9 | GlobalSection(SolutionConfigurationPlatforms) = preSolution 10 | Debug|Any CPU = Debug|Any CPU 11 | Release|Any CPU = Release|Any CPU 12 | EndGlobalSection 13 | GlobalSection(ProjectConfigurationPlatforms) = postSolution 14 | {7121554B-164C-454C-8CBE-374BCA12AB08}.Debug|Any CPU.ActiveCfg = Debug|Any CPU 15 | {7121554B-164C-454C-8CBE-374BCA12AB08}.Debug|Any CPU.Build.0 = Debug|Any CPU 16 | {7121554B-164C-454C-8CBE-374BCA12AB08}.Release|Any CPU.ActiveCfg = Release|Any CPU 17 | {7121554B-164C-454C-8CBE-374BCA12AB08}.Release|Any CPU.Build.0 = Release|Any CPU 18 | EndGlobalSection 19 | GlobalSection(SolutionProperties) = preSolution 20 | HideSolutionNode = FALSE 21 | EndGlobalSection 22 | EndGlobal 23 | -------------------------------------------------------------------------------- /test_apps/SimpleImageStorage/App.xaml: -------------------------------------------------------------------------------- 1 |  5 | 6 | 7 | 8 | 9 | 10 | -------------------------------------------------------------------------------- /test_apps/SimpleImageStorage/App.xaml.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections.Generic; 3 | using System.Configuration; 4 | using System.Data; 5 | using System.Linq; 6 | using System.Windows; 7 | 8 | namespace SimpleImageStorage 9 | { 10 | /// 11 | /// Interaction logic for App.xaml 12 | /// 13 | public partial class App : Application 14 | { 15 | } 16 | } 17 | -------------------------------------------------------------------------------- /test_apps/SimpleImageStorage/ControlCommand.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections.Generic; 3 | using System.Linq; 4 | using System.Text; 5 | using System.Windows.Input; 6 | 7 | namespace SimpleImageStorage 8 | { 9 | public class ControlCommand : ICommand 10 | { 11 | private Action action; 12 | 13 | public ControlCommand(Action action) 14 | { 15 | this.action = action; 16 | } 17 | 18 | public bool CanExecute(object parameter) 19 | { 20 | return true; 21 | } 22 | 23 | public event EventHandler CanExecuteChanged; 24 | 25 | public void Execute(object parameter) 26 | { 27 | if (action != null) 28 | { 29 | action(parameter); 30 | } 31 | } 32 | } 33 | } 34 | -------------------------------------------------------------------------------- /test_apps/SimpleImageStorage/Lib/Fluent.dll: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tomazk8/TmStorage/5d72efe5fad3c78e6519ebc2eee71f4c9cad6876/test_apps/SimpleImageStorage/Lib/Fluent.dll -------------------------------------------------------------------------------- /test_apps/SimpleImageStorage/Lib/Images/Add.ico: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tomazk8/TmStorage/5d72efe5fad3c78e6519ebc2eee71f4c9cad6876/test_apps/SimpleImageStorage/Lib/Images/Add.ico -------------------------------------------------------------------------------- /test_apps/SimpleImageStorage/Lib/Images/Close.ico: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tomazk8/TmStorage/5d72efe5fad3c78e6519ebc2eee71f4c9cad6876/test_apps/SimpleImageStorage/Lib/Images/Close.ico -------------------------------------------------------------------------------- /test_apps/SimpleImageStorage/Lib/Images/Create.ico: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tomazk8/TmStorage/5d72efe5fad3c78e6519ebc2eee71f4c9cad6876/test_apps/SimpleImageStorage/Lib/Images/Create.ico -------------------------------------------------------------------------------- /test_apps/SimpleImageStorage/Lib/Images/Delete.ico: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tomazk8/TmStorage/5d72efe5fad3c78e6519ebc2eee71f4c9cad6876/test_apps/SimpleImageStorage/Lib/Images/Delete.ico -------------------------------------------------------------------------------- /test_apps/SimpleImageStorage/Lib/Images/Open.ico: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tomazk8/TmStorage/5d72efe5fad3c78e6519ebc2eee71f4c9cad6876/test_apps/SimpleImageStorage/Lib/Images/Open.ico -------------------------------------------------------------------------------- /test_apps/SimpleImageStorage/Lib/Images/Replace.ico: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tomazk8/TmStorage/5d72efe5fad3c78e6519ebc2eee71f4c9cad6876/test_apps/SimpleImageStorage/Lib/Images/Replace.ico -------------------------------------------------------------------------------- /test_apps/SimpleImageStorage/MainWindow.xaml: -------------------------------------------------------------------------------- 1 |  8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 22 | 23 | 29 | 30 | 35 | 36 | 39 | 40 | 41 | 42 | 47 | 48 | 53 | 54 | 59 | 60 | 61 | 62 | 65 | 68 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 | 85 | 86 | 87 | 88 | 89 | 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | 98 | 99 | 100 | 101 | 102 | 103 | 104 | 105 | 106 | 107 | 108 | 111 | 115 | 118 | 119 | 120 | 121 | 122 | -------------------------------------------------------------------------------- /test_apps/SimpleImageStorage/MainWindow.xaml.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections.Generic; 3 | using System.Linq; 4 | using System.Text; 5 | using System.Windows; 6 | using System.Windows.Controls; 7 | using System.Windows.Data; 8 | using System.Windows.Documents; 9 | using System.Windows.Input; 10 | using System.Windows.Media; 11 | using System.Windows.Media.Imaging; 12 | using System.Windows.Navigation; 13 | using System.Windows.Shapes; 14 | using Fluent; 15 | using TmFramework.TmStorage; 16 | using Microsoft.Win32; 17 | using System.Runtime.Serialization.Formatters.Binary; 18 | using System.IO; 19 | using System.Windows.Threading; 20 | 21 | namespace SimpleImageStorage 22 | { 23 | /// 24 | /// Interaction logic for MainWindow.xaml 25 | /// 26 | public partial class MainWindow : Fluent.RibbonWindow 27 | { 28 | #region Fields 29 | private Storage storage = null; 30 | private List items = null; 31 | private DispatcherTimer timer; 32 | private Guid fileListId = new Guid(1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0); 33 | #endregion 34 | 35 | #region Construction 36 | public MainWindow() 37 | { 38 | InitializeComponent(); 39 | 40 | imageList.DisplayMemberPath = "DisplayName"; 41 | imageList.SelectedValuePath = "StreamId"; 42 | UpdateTransactionButtonStates(); 43 | 44 | timer = new DispatcherTimer( 45 | new TimeSpan(0, 0, 1), 46 | DispatcherPriority.Normal, 47 | (o, e) => 48 | { 49 | if (storage != null) 50 | { 51 | amountReadTextBlock.Text = storage.Statistics.BytesRead.ToString("N0"); 52 | amountWrittenTextBlock.Text = storage.Statistics.BytesWritten.ToString("N0"); 53 | } 54 | else 55 | { 56 | amountReadTextBlock.Text = ""; 57 | amountWrittenTextBlock.Text = ""; 58 | } 59 | }, 60 | this.Dispatcher); 61 | } 62 | #endregion 63 | 64 | #region Private methods 65 | private void UpdateUI() 66 | { 67 | if (storage != null && !storage.IsClosed) 68 | { 69 | if (storage.ContainsStream(fileListId)) 70 | { 71 | imageList.ItemsSource = null; 72 | imageList.ItemsSource = items; 73 | } 74 | } 75 | else 76 | { 77 | imageList.ItemsSource = null; 78 | imageContainer.Source = null; 79 | } 80 | } 81 | private void SaveImageList() 82 | { 83 | if (items != null) 84 | { 85 | storage.StartTransaction(); 86 | 87 | BinaryFormatter f = new BinaryFormatter(); 88 | if (!storage.ContainsStream(fileListId)) 89 | storage.CreateStream(fileListId); 90 | 91 | using (MemoryStream ms = new MemoryStream()) 92 | { 93 | using (Stream s = storage.OpenStream(fileListId)) 94 | { 95 | s.Position = 0; 96 | f.Serialize(ms, items); 97 | 98 | byte[] buf = ms.GetBuffer(); 99 | s.Write(buf, 0, (int)ms.Length); 100 | } 101 | } 102 | 103 | storage.CommitTransaction(); 104 | } 105 | } 106 | private void LoadImageList() 107 | { 108 | if (storage.ContainsStream(fileListId)) 109 | { 110 | // Load stream table 111 | BinaryFormatter f = new BinaryFormatter(); 112 | using (Stream s = storage.OpenStream(fileListId)) 113 | { 114 | if (s.Length > 0) 115 | { 116 | s.Position = 0; 117 | items = (List)f.Deserialize(s); 118 | } 119 | else 120 | items = new List(); 121 | } 122 | } 123 | } 124 | private void AddImages(IEnumerable files) 125 | { 126 | if (files.Count() > 0) 127 | { 128 | if (items == null) 129 | items = new List(); 130 | 131 | //IsEnabled = false; 132 | //ProgressDialog dialog = new ProgressDialog("Progress", "Adding files..."); 133 | //dialog.Show(); 134 | try 135 | { 136 | int count = files.Count(); 137 | int index = 1; 138 | 139 | storage.StartTransaction(); 140 | foreach (string file in files) 141 | { 142 | using (StorageStream s = storage.CreateStream(Guid.NewGuid())) 143 | { 144 | using (FileStream fs = File.OpenRead(file)) 145 | { 146 | byte[] buf = new byte[65536]; 147 | 148 | int l = fs.Read(buf, 0, buf.Length); 149 | while (l > 0) 150 | { 151 | s.Write(buf, 0, l); 152 | l = fs.Read(buf, 0, buf.Length); 153 | } 154 | } 155 | 156 | items.Add(new ImageInfo() 157 | { 158 | Name = System.IO.Path.GetFileName(file), 159 | Size = (int)s.Length, 160 | StreamId = s.StreamId 161 | }); 162 | } 163 | //dialog.Progress = (double)index * 100d / (double)count; 164 | index++; 165 | } 166 | 167 | SaveImageList(); 168 | UpdateUI(); 169 | RefreshMap(imageList.SelectedValue as Guid?); 170 | 171 | storage.CommitTransaction(); 172 | } 173 | finally 174 | { 175 | //dialog.Close(); 176 | //IsEnabled = true; 177 | } 178 | } 179 | } 180 | private void ReplaceImage(string file, Guid streamId) 181 | { 182 | if (file.Count() > 0) 183 | { 184 | if (items == null) 185 | items = new List(); 186 | 187 | using (StorageStream s = storage.OpenStream(streamId)) 188 | { 189 | using (FileStream fs = File.OpenRead(file)) 190 | { 191 | byte[] buf = new byte[65536]; 192 | 193 | int l = fs.Read(buf, 0, buf.Length); 194 | while (l > 0) 195 | { 196 | s.Write(buf, 0, l); 197 | l = fs.Read(buf, 0, buf.Length); 198 | } 199 | } 200 | 201 | ImageInfo ii = items 202 | .Where(i => i.StreamId == streamId) 203 | .First(); 204 | 205 | ii.Name = System.IO.Path.GetFileName(file); 206 | ii.Size = (int)s.Length; 207 | 208 | s.SetLength(s.Position); 209 | } 210 | 211 | SaveImageList(); 212 | UpdateUI(); 213 | RefreshMap(imageList.SelectedValue as Guid?); 214 | } 215 | } 216 | private void RefreshMap(Guid? streamId) 217 | { 218 | mapCanvas.Children.Clear(); 219 | 220 | if (storage != null) 221 | { 222 | mapCanvas.Background = Brushes.Gray; 223 | 224 | List extents = storage.GetFreeSpaceExtents(); 225 | DrawExtents(extents, Brushes.Green); 226 | 227 | if (streamId.HasValue && storage.ContainsStream(streamId.Value)) 228 | { 229 | extents = storage.GetStreamExtents(streamId.Value); 230 | 231 | DrawExtents(extents, Brushes.Navy); 232 | } 233 | } 234 | else 235 | { 236 | mapCanvas.Background = Brushes.White; 237 | } 238 | 239 | } 240 | private void DrawExtents(List extents, Brush brush) 241 | { 242 | if (storage.Statistics.StorageSize == 0) 243 | return; 244 | 245 | double k = (double)mapCanvas.ActualWidth / (double)storage.Statistics.StorageSize; 246 | 247 | foreach (var e in extents) 248 | { 249 | double x = (double)e.Location * k; 250 | double w = Math.Max((double)e.Size * k, 1d); 251 | 252 | Rectangle rect = new Rectangle(); 253 | rect.Width = w; 254 | rect.Height = mapCanvas.ActualHeight; 255 | rect.SetValue(Canvas.LeftProperty, x); 256 | rect.SetValue(Canvas.TopProperty, 0d); 257 | rect.RadiusX = 6; 258 | rect.RadiusY = 3; 259 | rect.Fill = brush; 260 | 261 | mapCanvas.Children.Add(rect); 262 | } 263 | } 264 | private void DeleteSelectedImage() 265 | { 266 | Guid? streamId = imageList.SelectedValue as Guid?; 267 | 268 | if (streamId.HasValue) 269 | { 270 | int index = imageList.SelectedIndex; 271 | storage.DeleteStream(streamId.Value); 272 | ImageInfo ii = items 273 | .Where(i => i.StreamId == streamId.Value) 274 | .First(); 275 | items.Remove(ii); 276 | SaveImageList(); 277 | UpdateUI(); 278 | imageList.SelectedIndex = index; 279 | RefreshMap(imageList.SelectedValue as Guid?); 280 | } 281 | } 282 | #endregion 283 | 284 | #region Event handlers 285 | private void CreateStorageButton_Click(object sender, RoutedEventArgs e) 286 | { 287 | SaveFileDialog dialog = new SaveFileDialog(); 288 | dialog.Filter = "Storage files|*.storage"; 289 | dialog.CheckFileExists = false; 290 | dialog.Title = "Create image storage"; 291 | bool? result = dialog.ShowDialog(); 292 | 293 | if (result.HasValue && result.Value) 294 | { 295 | if (storage != null) 296 | storage.Close(); 297 | 298 | string logFilename = dialog.FileName + "log"; 299 | 300 | if (File.Exists(dialog.FileName)) 301 | File.Delete(dialog.FileName); 302 | if (File.Exists(logFilename)) 303 | File.Delete(logFilename); 304 | 305 | storage = new Storage(dialog.FileName, logFilename); 306 | storage.TransactionStateChanged += Storage_TransactionStateShanged; 307 | storage.CreateStream(fileListId); 308 | 309 | RefreshMap(null); 310 | UpdateTransactionButtonStates(); 311 | } 312 | } 313 | 314 | private void Storage_TransactionStateShanged(object sender, TransactionStateChangedEventArgs e) 315 | { 316 | if (e.TransactionStateChangeType == TransactionStateChangeType.Rollback) 317 | { 318 | LoadImageList(); 319 | UpdateUI(); 320 | RefreshMap(imageList.SelectedValue as Guid?); 321 | } 322 | 323 | UpdateTransactionButtonStates(); 324 | } 325 | 326 | private void UpdateTransactionButtonStates() 327 | { 328 | StartTransactionButton.IsEnabled = storage != null && !storage.InTransaction; 329 | CommitTransactionButton.IsEnabled = storage != null && storage.InTransaction; 330 | RollbackTransactionButton.IsEnabled = storage != null && storage.InTransaction; 331 | } 332 | private void OpenStorageButton_Click(object sender, RoutedEventArgs e) 333 | { 334 | OpenFileDialog dialog = new OpenFileDialog(); 335 | dialog.Filter = "Storage files|*.storage"; 336 | dialog.CheckFileExists = true; 337 | dialog.Title = "Open image storage"; 338 | bool? result = dialog.ShowDialog(); 339 | 340 | if (result.HasValue && result.Value) 341 | { 342 | if (storage != null) 343 | storage.Close(); 344 | 345 | storage = new Storage(dialog.FileName, dialog.FileName + "log"); 346 | storage.TransactionStateChanged += Storage_TransactionStateShanged; 347 | 348 | LoadImageList(); 349 | UpdateUI(); 350 | RefreshMap(null); 351 | UpdateTransactionButtonStates(); 352 | } 353 | } 354 | private void imageList_SelectionChanged(object sender, SelectionChangedEventArgs e) 355 | { 356 | if (storage != null) 357 | { 358 | Guid? streamid = imageList.SelectedValue as Guid?; 359 | if (streamid.HasValue && storage.ContainsStream(streamid.Value)) 360 | { 361 | Stream storageStream = storage.OpenStream(streamid.Value); 362 | try 363 | { 364 | // Read all bytes first to memory, then load image from it 365 | // If data is not read into memory first, then image won't load correctly but only partially. I don't know why 366 | // this happens but there seems that this is not a bug in TmStorage. In both cases, when loading directly 367 | // from stream or from memory, first 48 bytes read are exactly the same. The next reads are different. 368 | // When reading from memory only one read is executed to read all the image data. If loading directly from 369 | // stream, many reads are made in small quentities and it never reaches the end of the stream. 370 | byte[] buf = new byte[(int)storageStream.Length]; 371 | storageStream.Read(buf, 0, buf.Length); 372 | 373 | MemoryStream tmpStream = new MemoryStream(buf); 374 | 375 | BitmapImage img = new BitmapImage(); 376 | storageStream.Position = 0; 377 | img.BeginInit(); 378 | img.StreamSource = tmpStream;// storageStream; 379 | img.EndInit(); 380 | imageContainer.Source = img; 381 | imageContainer.Stretch = Stretch.Uniform; 382 | } 383 | catch 384 | { 385 | imageContainer.Source = null; 386 | } 387 | 388 | RefreshMap(streamid); 389 | } 390 | else 391 | imageContainer.Source = null; 392 | } 393 | } 394 | private void AddImagesButton_Click(object sender, RoutedEventArgs e) 395 | { 396 | if (storage != null) 397 | { 398 | OpenFileDialog d = new OpenFileDialog(); 399 | d.Filter = "Images(bmp,jpg,png,tif)|*.jpg;*.bmp;*.png;*.tif"; 400 | d.Multiselect = true; 401 | bool? r = d.ShowDialog(); 402 | if (r.HasValue && r.Value) 403 | { 404 | AddImages(d.FileNames); 405 | } 406 | } 407 | } 408 | private void RibbonWindow_SizeChanged(object sender, SizeChangedEventArgs e) 409 | { 410 | RefreshMap(imageList.SelectedValue as Guid?); 411 | } 412 | private void DeleteImageButton_Click(object sender, RoutedEventArgs e) 413 | { 414 | DeleteSelectedImage(); 415 | } 416 | private void imageList_KeyDown(object sender, KeyEventArgs e) 417 | { 418 | if (e.Key == Key.Delete) 419 | DeleteSelectedImage(); 420 | } 421 | private void ReplaceImageButton_Click(object sender, RoutedEventArgs e) 422 | { 423 | Guid? streamId = imageList.SelectedValue as Guid?; 424 | 425 | if (streamId.HasValue) 426 | { 427 | OpenFileDialog d = new OpenFileDialog(); 428 | d.Filter = "Images(bmp,jpg,png,tif)|*.jpg;*.bmp;*.png;*.tif"; 429 | bool? r = d.ShowDialog(); 430 | if (r.HasValue && r.Value) 431 | { 432 | ReplaceImage(d.FileName, streamId.Value); 433 | } 434 | 435 | UpdateUI(); 436 | RefreshMap(imageList.SelectedValue as Guid?); 437 | } 438 | } 439 | private void StartTransactionButton_Click(object sender, RoutedEventArgs e) 440 | { 441 | if (storage != null) 442 | storage.StartTransaction(); 443 | } 444 | private void CommitTransactionButton_Click(object sender, RoutedEventArgs e) 445 | { 446 | if (storage != null) 447 | storage.CommitTransaction(); 448 | } 449 | private void RollbackTransactionButton_Click(object sender, RoutedEventArgs e) 450 | { 451 | if (storage != null) 452 | storage.RollbackTransaction(); 453 | } 454 | private void TruncateStorageButton_Click(object sender, RoutedEventArgs e) 455 | { 456 | storage.TruncateStorage(); 457 | RefreshMap(imageList.SelectedValue as Guid?); 458 | } 459 | private void RibbonWindow_Closing(object sender, System.ComponentModel.CancelEventArgs e) 460 | { 461 | if (storage != null) 462 | storage.Close(); 463 | } 464 | #endregion 465 | 466 | private void imageList_DragEnter(object sender, DragEventArgs e) 467 | { 468 | e.Effects = DragDropEffects.Copy; 469 | } 470 | 471 | private void imageList_Drop(object sender, DragEventArgs e) 472 | { 473 | if (e.Data.GetDataPresent(DataFormats.FileDrop)) 474 | { 475 | string[] files = (string[])e.Data.GetData(DataFormats.FileDrop); 476 | 477 | AddImages(files); 478 | } 479 | } 480 | } 481 | 482 | [Serializable] 483 | class ImageInfo 484 | { 485 | public Guid StreamId { get; set; } 486 | public string Name { get; set; } 487 | public int Size { get; set; } 488 | public string DisplayName 489 | { 490 | get 491 | { 492 | return string.Format("{0} ({1:N0} kB)", Name, Size / 1024); 493 | } 494 | } 495 | } 496 | } 497 | -------------------------------------------------------------------------------- /test_apps/SimpleImageStorage/ProgressDialog.xaml: -------------------------------------------------------------------------------- 1 |  5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | -------------------------------------------------------------------------------- /test_apps/SimpleImageStorage/ProgressDialog.xaml.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections.Generic; 3 | using System.Linq; 4 | using System.Text; 5 | using System.Windows; 6 | using System.Windows.Controls; 7 | using System.Windows.Data; 8 | using System.Windows.Documents; 9 | using System.Windows.Input; 10 | using System.Windows.Media; 11 | using System.Windows.Media.Imaging; 12 | using System.Windows.Shapes; 13 | 14 | namespace SimpleImageStorage 15 | { 16 | /// 17 | /// Interaction logic for ProgressDialog.xaml 18 | /// 19 | public partial class ProgressDialog : Window 20 | { 21 | public ProgressDialog(string title, string text) 22 | { 23 | InitializeComponent(); 24 | 25 | Title = title; 26 | progressTextBlock.Text = text; 27 | } 28 | 29 | public double Progress 30 | { 31 | get { return progressBar.Value; } 32 | set 33 | { 34 | progressBar.SetValue(ProgressBar.ValueProperty, value); 35 | //progressBar.Value = value; 36 | } 37 | } 38 | } 39 | } 40 | -------------------------------------------------------------------------------- /test_apps/SimpleImageStorage/Properties/AssemblyInfo.cs: -------------------------------------------------------------------------------- 1 | using System.Reflection; 2 | using System.Resources; 3 | using System.Runtime.CompilerServices; 4 | using System.Runtime.InteropServices; 5 | using System.Windows; 6 | 7 | // General Information about an assembly is controlled through the following 8 | // set of attributes. Change these attribute values to modify the information 9 | // associated with an assembly. 10 | [assembly: AssemblyTitle("SimpleImageStorage")] 11 | [assembly: AssemblyDescription("")] 12 | [assembly: AssemblyConfiguration("")] 13 | [assembly: AssemblyCompany("HP")] 14 | [assembly: AssemblyProduct("SimpleImageStorage")] 15 | [assembly: AssemblyCopyright("Copyright © HP 2012")] 16 | [assembly: AssemblyTrademark("")] 17 | [assembly: AssemblyCulture("")] 18 | 19 | // Setting ComVisible to false makes the types in this assembly not visible 20 | // to COM components. If you need to access a type in this assembly from 21 | // COM, set the ComVisible attribute to true on that type. 22 | [assembly: ComVisible(false)] 23 | 24 | //In order to begin building localizable applications, set 25 | //CultureYouAreCodingWith in your .csproj file 26 | //inside a . For example, if you are using US english 27 | //in your source files, set the to en-US. Then uncomment 28 | //the NeutralResourceLanguage attribute below. Update the "en-US" in 29 | //the line below to match the UICulture setting in the project file. 30 | 31 | //[assembly: NeutralResourcesLanguage("en-US", UltimateResourceFallbackLocation.Satellite)] 32 | 33 | 34 | [assembly: ThemeInfo( 35 | ResourceDictionaryLocation.None, //where theme specific resource dictionaries are located 36 | //(used if a resource is not found in the page, 37 | // or application resource dictionaries) 38 | ResourceDictionaryLocation.SourceAssembly //where the generic resource dictionary is located 39 | //(used if a resource is not found in the page, 40 | // app, or any theme specific resource dictionaries) 41 | )] 42 | 43 | 44 | // Version information for an assembly consists of the following four values: 45 | // 46 | // Major Version 47 | // Minor Version 48 | // Build Number 49 | // Revision 50 | // 51 | // You can specify all the values or you can default the Build and Revision Numbers 52 | // by using the '*' as shown below: 53 | // [assembly: AssemblyVersion("1.0.*")] 54 | [assembly: AssemblyVersion("1.0.0.0")] 55 | [assembly: AssemblyFileVersion("1.0.0.0")] 56 | -------------------------------------------------------------------------------- /test_apps/SimpleImageStorage/Properties/Resources.Designer.cs: -------------------------------------------------------------------------------- 1 | //------------------------------------------------------------------------------ 2 | // 3 | // This code was generated by a tool. 4 | // Runtime Version:4.0.30319.18408 5 | // 6 | // Changes to this file may cause incorrect behavior and will be lost if 7 | // the code is regenerated. 8 | // 9 | //------------------------------------------------------------------------------ 10 | 11 | namespace SimpleImageStorage.Properties { 12 | using System; 13 | 14 | 15 | /// 16 | /// A strongly-typed resource class, for looking up localized strings, etc. 17 | /// 18 | // This class was auto-generated by the StronglyTypedResourceBuilder 19 | // class via a tool like ResGen or Visual Studio. 20 | // To add or remove a member, edit your .ResX file then rerun ResGen 21 | // with the /str option, or rebuild your VS project. 22 | [global::System.CodeDom.Compiler.GeneratedCodeAttribute("System.Resources.Tools.StronglyTypedResourceBuilder", "4.0.0.0")] 23 | [global::System.Diagnostics.DebuggerNonUserCodeAttribute()] 24 | [global::System.Runtime.CompilerServices.CompilerGeneratedAttribute()] 25 | internal class Resources { 26 | 27 | private static global::System.Resources.ResourceManager resourceMan; 28 | 29 | private static global::System.Globalization.CultureInfo resourceCulture; 30 | 31 | [global::System.Diagnostics.CodeAnalysis.SuppressMessageAttribute("Microsoft.Performance", "CA1811:AvoidUncalledPrivateCode")] 32 | internal Resources() { 33 | } 34 | 35 | /// 36 | /// Returns the cached ResourceManager instance used by this class. 37 | /// 38 | [global::System.ComponentModel.EditorBrowsableAttribute(global::System.ComponentModel.EditorBrowsableState.Advanced)] 39 | internal static global::System.Resources.ResourceManager ResourceManager { 40 | get { 41 | if (object.ReferenceEquals(resourceMan, null)) { 42 | global::System.Resources.ResourceManager temp = new global::System.Resources.ResourceManager("SimpleImageStorage.Properties.Resources", typeof(Resources).Assembly); 43 | resourceMan = temp; 44 | } 45 | return resourceMan; 46 | } 47 | } 48 | 49 | /// 50 | /// Overrides the current thread's CurrentUICulture property for all 51 | /// resource lookups using this strongly typed resource class. 52 | /// 53 | [global::System.ComponentModel.EditorBrowsableAttribute(global::System.ComponentModel.EditorBrowsableState.Advanced)] 54 | internal static global::System.Globalization.CultureInfo Culture { 55 | get { 56 | return resourceCulture; 57 | } 58 | set { 59 | resourceCulture = value; 60 | } 61 | } 62 | } 63 | } 64 | -------------------------------------------------------------------------------- /test_apps/SimpleImageStorage/Properties/Resources.resx: -------------------------------------------------------------------------------- 1 |  2 | 3 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 | 85 | 86 | 87 | 88 | 89 | 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | 98 | 99 | 100 | 101 | 102 | 103 | 104 | 105 | 106 | text/microsoft-resx 107 | 108 | 109 | 2.0 110 | 111 | 112 | System.Resources.ResXResourceReader, System.Windows.Forms, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 113 | 114 | 115 | System.Resources.ResXResourceWriter, System.Windows.Forms, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 116 | 117 | -------------------------------------------------------------------------------- /test_apps/SimpleImageStorage/Properties/Settings.Designer.cs: -------------------------------------------------------------------------------- 1 | //------------------------------------------------------------------------------ 2 | // 3 | // This code was generated by a tool. 4 | // Runtime Version:4.0.30319.18408 5 | // 6 | // Changes to this file may cause incorrect behavior and will be lost if 7 | // the code is regenerated. 8 | // 9 | //------------------------------------------------------------------------------ 10 | 11 | namespace SimpleImageStorage.Properties { 12 | 13 | 14 | [global::System.Runtime.CompilerServices.CompilerGeneratedAttribute()] 15 | [global::System.CodeDom.Compiler.GeneratedCodeAttribute("Microsoft.VisualStudio.Editors.SettingsDesigner.SettingsSingleFileGenerator", "12.0.0.0")] 16 | internal sealed partial class Settings : global::System.Configuration.ApplicationSettingsBase { 17 | 18 | private static Settings defaultInstance = ((Settings)(global::System.Configuration.ApplicationSettingsBase.Synchronized(new Settings()))); 19 | 20 | public static Settings Default { 21 | get { 22 | return defaultInstance; 23 | } 24 | } 25 | } 26 | } 27 | -------------------------------------------------------------------------------- /test_apps/SimpleImageStorage/Properties/Settings.settings: -------------------------------------------------------------------------------- 1 |  2 | 3 | 4 | 5 | 6 | 7 | -------------------------------------------------------------------------------- /test_apps/SimpleImageStorage/SimpleImageStorage.csproj: -------------------------------------------------------------------------------- 1 |  2 | 3 | 4 | Debug 5 | x86 6 | 8.0.30703 7 | 2.0 8 | {D42DC174-1E97-4B3D-913C-DB5521548D53} 9 | WinExe 10 | Properties 11 | SimpleImageStorage 12 | SimpleImageStorage 13 | v4.5 14 | 15 | 16 | 512 17 | {60dc8134-eba5-43b8-bcc9-bb4bc16c2548};{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC} 18 | 4 19 | 20 | 21 | x86 22 | true 23 | full 24 | false 25 | bin\Debug\ 26 | DEBUG;TRACE 27 | prompt 28 | 4 29 | false 30 | 31 | 32 | x86 33 | pdbonly 34 | true 35 | bin\Release\ 36 | TRACE 37 | prompt 38 | 4 39 | false 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 4.0 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | MSBuild:Compile 60 | Designer 61 | 62 | 63 | MSBuild:Compile 64 | Designer 65 | 66 | 67 | App.xaml 68 | Code 69 | 70 | 71 | 72 | MainWindow.xaml 73 | Code 74 | 75 | 76 | Designer 77 | MSBuild:Compile 78 | 79 | 80 | 81 | 82 | ProgressDialog.xaml 83 | 84 | 85 | Code 86 | 87 | 88 | True 89 | True 90 | Resources.resx 91 | 92 | 93 | True 94 | Settings.settings 95 | True 96 | 97 | 98 | ResXFileCodeGenerator 99 | Resources.Designer.cs 100 | 101 | 102 | 103 | SettingsSingleFileGenerator 104 | Settings.Designer.cs 105 | 106 | 107 | 108 | 109 | 110 | 111 | 112 | 113 | 114 | 115 | 116 | 117 | 118 | 125 | -------------------------------------------------------------------------------- /test_apps/SimpleImageStorage/SimpleImageStorage.sln: -------------------------------------------------------------------------------- 1 |  2 | Microsoft Visual Studio Solution File, Format Version 12.00 3 | # Visual Studio 2013 4 | VisualStudioVersion = 12.0.31101.0 5 | MinimumVisualStudioVersion = 10.0.40219.1 6 | Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "SimpleImageStorage", "SimpleImageStorage.csproj", "{D42DC174-1E97-4B3D-913C-DB5521548D53}" 7 | EndProject 8 | Global 9 | GlobalSection(SolutionConfigurationPlatforms) = preSolution 10 | Debug|Any CPU = Debug|Any CPU 11 | Debug|Mixed Platforms = Debug|Mixed Platforms 12 | Debug|x86 = Debug|x86 13 | Release|Any CPU = Release|Any CPU 14 | Release|Mixed Platforms = Release|Mixed Platforms 15 | Release|x86 = Release|x86 16 | EndGlobalSection 17 | GlobalSection(ProjectConfigurationPlatforms) = postSolution 18 | {D42DC174-1E97-4B3D-913C-DB5521548D53}.Debug|Any CPU.ActiveCfg = Debug|x86 19 | {D42DC174-1E97-4B3D-913C-DB5521548D53}.Debug|Mixed Platforms.ActiveCfg = Debug|x86 20 | {D42DC174-1E97-4B3D-913C-DB5521548D53}.Debug|Mixed Platforms.Build.0 = Debug|x86 21 | {D42DC174-1E97-4B3D-913C-DB5521548D53}.Debug|x86.ActiveCfg = Debug|x86 22 | {D42DC174-1E97-4B3D-913C-DB5521548D53}.Debug|x86.Build.0 = Debug|x86 23 | {D42DC174-1E97-4B3D-913C-DB5521548D53}.Release|Any CPU.ActiveCfg = Release|x86 24 | {D42DC174-1E97-4B3D-913C-DB5521548D53}.Release|Mixed Platforms.ActiveCfg = Release|x86 25 | {D42DC174-1E97-4B3D-913C-DB5521548D53}.Release|Mixed Platforms.Build.0 = Release|x86 26 | {D42DC174-1E97-4B3D-913C-DB5521548D53}.Release|x86.ActiveCfg = Release|x86 27 | {D42DC174-1E97-4B3D-913C-DB5521548D53}.Release|x86.Build.0 = Release|x86 28 | EndGlobalSection 29 | GlobalSection(SolutionProperties) = preSolution 30 | HideSolutionNode = FALSE 31 | EndGlobalSection 32 | EndGlobal 33 | -------------------------------------------------------------------------------- /test_apps/SimpleImageStorage/app.config: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | --------------------------------------------------------------------------------