├── .DS_Store ├── CONTRIBUTING.MD ├── LICENSE.md ├── README.md ├── docs ├── .DS_Store └── design_patterns │ ├── .DS_Store │ ├── AsyncAPIPattern.md │ ├── SecureSyncPattern.md │ ├── UIResponsivenessPattern.md │ └── images │ ├── .DS_Store │ ├── architecture.png │ ├── async-images │ ├── fig1.png │ └── fig2.png │ ├── fig1.png │ ├── fig2.png │ ├── fig3.png │ ├── nonBlocking_images │ ├── blockingui-black.png │ ├── non-blockingui-black.png │ ├── screen_1.png │ ├── screen_10.png │ ├── screen_2.png │ ├── screen_3.png │ ├── screen_4.png │ ├── screen_5.png │ ├── screen_6.png │ ├── screen_7.png │ ├── screen_8.png │ ├── screen_9.png │ └── temp │ └── securesync-images │ ├── fig-1.png │ └── fig-2.png └── faq.md /.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/IBM/BlockchainDevelopmentDesignPatterns/6cf82348739f239c23d67aeebc9f022d1f1ab2af/.DS_Store -------------------------------------------------------------------------------- /CONTRIBUTING.MD: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/IBM/BlockchainDevelopmentDesignPatterns/6cf82348739f239c23d67aeebc9f022d1f1ab2af/CONTRIBUTING.MD -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/IBM/BlockchainDevelopmentDesignPatterns/6cf82348739f239c23d67aeebc9f022d1f1ab2af/LICENSE.md -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Design patterns implemented for Blockchain applications 2 | 3 | **Owner: [Varun Ojha](https://github.ibm.com/varun-ojha)** 4 | 5 | [![Build Status](https://travis.ibm.com/BlockchainAB/BlockchainDevelopmentDesignPatterns.svg?token=sQbGqm1RNTX4ZceCsxf9&branch=master)](https://travis.ibm.com/BlockchainAB/BlockchainDevelopmentDesignPatterns) 6 | 7 | 8 | # Design patterns for Blockchain based application development 9 | The purpose of this repository is to document design patterns and development best practices when developing end to end applications using Blockchain. 10 | To start with I would be adding approaches, architecture and code snippets for the following design patterns. Please feel free to reach out to me for collaboration on any of the patterns. If you have some specific problems/patterns in mind that you do not see on the list please reach out and I would be happy to add them. 11 | 12 | - Interactive UI design for Blockchain based applications 13 | - Asynchronous API design for Blockchain based applications 14 | - Secure sync and off chain query pattern for Blockchain based applications 15 | - Chaincode and client side application development for Blockchain Multi version concurrency control 16 | - Attribute based Access control 17 | - Higher throughput and latency with Batching 18 | 19 | 20 | # Introduction 21 | 22 | Design patterns are the best formalized practices a programmer can use to 23 | solve common problems when designing an application or system. 24 | 25 | Design patterns can speed up the development process by providing tested, proven 26 | development paradigms. 27 | 28 | Reusing design patterns help prevent subtle issues which cause major 29 | problems, and it also improves code readability for coders and architects who 30 | are familiar with the patterns. 31 | 32 | # Getting started 33 | 34 | Before you dive into the material, you should be familiar with various 35 | Programming/Software Design Principles. 36 | 37 | All designs should be as simple as possible. You should start with KISS, YAGNI, 38 | and Do The Simplest Thing That Could Possibly Work principles. Complexity and 39 | patterns should only be introduced when they are needed for practical 40 | extensibility. 41 | 42 | Once you are familiar with these concepts you can start drilling down into 43 | patterns by any of the following approaches 44 | 45 | - Using difficulty tags, `Difficulty-Beginner`, `Difficulty-Intermediate` & `Difficulty-Expert`. 46 | - Using pattern categories, `UI`, `Front-end` and others. 47 | - Search for a specific pattern. Can't find one? Please report a new pattern [here](xxxxx). 48 | 49 | 50 | ## Blockchain Patterns 51 | 52 | | Pattern | Description | Status | 53 | |:-------:|:----------- |:------:| 54 | | [NonBlockingUIPattern](./docs/design_patterns/UIResponsivenessPattern.md) | Provides design and architecture for creating an interactive and non blocking Blockchain application UI| ✔ | 55 | | [AsyncPattern](./docs/design_patterns/AsyncPattern.md) | Discusses the API design and the interaction with other components such as off chain store and the events service | ✔ | 56 | | [SecureSyncPattern](./docs/design_patterns/SecureSyncPattern.md) | It covers the design of the event service that can be used to securely sync Blockchain with an application’s off chain store | ✔ | 57 | 58 | 59 | 60 | # How to contribute 61 | 62 | If you are willing to contribute to the project you will find the relevant information in our [developer wiki](https://github.com/BlockchainAB/BlockchainDevelopmentDesignPatterns/wiki). We will help you and answer your questions in the [Slack Channel](https://app.slack.com/client/T05119A3Z/C05119AH7). 63 | 64 | ## License 65 | This code pattern is licensed under the Apache Software License, Version 2. Separate third-party code objects invoked within this code pattern are licensed by their respective providers pursuant to their own separate licenses. Contributions are subject to the [Developer Certificate of Origin, Version 1.1 (DCO)](https://developercertificate.org/) and the [Apache Software License, Version 2](https://www.apache.org/licenses/LICENSE-2.0.txt). 66 | 67 | [Apache Software License (ASL) FAQ](https://www.apache.org/foundation/license-faq.html#WhatDoesItMEAN) 68 | -------------------------------------------------------------------------------- /docs/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/IBM/BlockchainDevelopmentDesignPatterns/6cf82348739f239c23d67aeebc9f022d1f1ab2af/docs/.DS_Store -------------------------------------------------------------------------------- /docs/design_patterns/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/IBM/BlockchainDevelopmentDesignPatterns/6cf82348739f239c23d67aeebc9f022d1f1ab2af/docs/design_patterns/.DS_Store -------------------------------------------------------------------------------- /docs/design_patterns/AsyncAPIPattern.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: pattern 3 | title: AsyncAPI 4 | folder: async-api-pattern 5 | categories: UI 6 | tags: 7 | - Blockchain 8 | - Difficulty-Intermediate 9 | - UI 10 | --- 11 | 12 | # Async API Pattern 13 | 14 | ## Overview 15 | 16 | This design pattern is a logical continuation of the non-blocking UI design pattern. It represents the second part of the Blockchain design pattern series. The non-blocking UI pattern touched upon the role of the asynchronous RESTful API and how it fits into the end to end architecture of a distributed blockchain application. This part will dive deeper into the asynchronous backed API and discuss the API design and its interaction with other components such as off chain store and the events service. This design pattern uses Hyperledger Fabric 1.4 as its Blockchain reference. 17 | 18 | 19 | ## Problem Statement 20 | 21 | In traditional applications when the user makes an API request such as creating a new resource, the user generally makes a blocking call to a backed API. Upon completion of the API request, the user receives the appropriate response payload based on whether the request was successful or not. Normally this model works quite well assuming the backend API has a fast response time. But when dealing with certain Blockchain based applications we cannot expect millisecond response times (As per current level of maturity). Listed below are some considerations that are relevant for complex enterprise grade Blockchain based applications. 22 | 23 | ### User interaction with the API 24 | Suppose a user creates a request for generating an invoice and submits it via the API interface. The invoice content needs to be stored on the Blockchain. At the very minimum the backend application would need to perform the following operations 25 | * Send transaction proposals and fetch the corresponding endorsements 26 | * Submit the transaction proposal responses to the orderer(s) 27 | * Wait for a Block/Chaincode event from the event service 28 | * Return the appropriate success/failure response to the user interface 29 | 30 | During this time the user is blocked waiting for the invoice creation request to complete. 31 | 32 | The scenario above is a simplification. Production grade microservices based solutions are quite complex and additionally need to account for the following 33 | * Microservices architecture leading to multiple network calls 34 | * Retries owing to MVCC (Multi version concurrency control) errors from the Blockchain 35 | * Distributed transactions across services 36 | * Resiliency/Recovery scenarios leading to multiple retries for network/connectivity errors 37 | * Network latency 38 | 39 | All the factors above along with Blockchain’s transaction paradigm, which includes asynchronous transaction submission make for a fairly complex backend API with a high response time. 40 | 41 | ### What the user wants? 42 | * Ideally, dealing with synchronous/blocking APIs makes for simpler client applications. The user’s client application makes an API request and receives a response indicating whether the request was successful or not along with the appropriate response payload. But when dealing with complex APIs that have higher response times, a blocking implementation can hamper performance due to resources being tied up waiting for dependent tasks to complete. 43 | * For UI clients the asynchronous API is quite useful as demonstrated in the non-blocking UI design pattern 44 | 45 | ### What the API developer wants? 46 | 47 | * The API developer wants to provide a stable and fast experience to the API consumer. 48 | * For APIs with high response times implementing a blocking request/response paradigm does not make sense. The resources on the API server are blocked waiting for other asynchronous tasks to complete (E.g a thread serving the API request is blocked waiting for Blockchain interaction to complete) 49 | 50 | 51 | 52 | ## Architecture Diagram 53 | 54 | The diagram below shows the different components of a Blockchain based application. This pattern will cover the design of the application layer and its interaction with the off chain cache, event service and the Blockchain. 55 | 56 |   57 | 58 |
59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 |
Fig.1
68 |
69 | 70 |   71 | 72 | The architecture above covers one possible approach for designing a distributed application that talks to Blockchain and caters to user interface requests. The next section will dive deeper into each of the components. 73 | 74 | 75 | ## Solution 76 | 77 | ### RESTful API 78 | 79 | This pattern takes the example of RESTful communication to demonstrate how the asynchronous backend API layer can be designed. Other options such as web sockets, gRPC etc could also be used to implement the communication between the client and server/APIs, but these will covered in subsequent iterations of this pattern. 80 | 81 | * The API layer needs to support asynchronous request execution so as to respond to the client immediately with transaction/request information for further reconciliation. 82 | 83 | * The client can poll the transaction endpoint or be notified via a webhook or message queue of the status of the transaction. 84 | 85 | * The API layer will make use of an off chain persistence store that will be used to replicate the data on the Blockchain for faster and more efficient querying 86 | 87 | * The API layer will leverage the events service to be notified of new blocks being added to the Blockchain. 88 | 89 | * These block events will be used to reconcile the state of the transactions invoked via the application and also to update its off chain persistence with the data that was added to the world state as a part of these block commits. 90 | 91 | ##### Endpoints 92 | 93 | #### POST / invoices 94 | 95 | Sample endpoint for creating an invoice resource using the RESTful API exposed by the application layer. Shown below is a sample response from the said API capturing the resource id, which in this case is the invoice id and the transaction id, which can be leveraged to poll the transactions endpoint for transaction status 96 | 97 | ``` 98 | Response Status: 202, Accepted 99 | 100 | Response payload: 101 | 102 | { 103 | 104 | "invoiceId": "invoice123", 105 | "transactionId": "tx123" 106 | } 107 | ``` 108 | 109 | #### GET /transactions?invoiceId=invoice123&status=Completed 110 | 111 | Sample endpoint for fetching all transactions invoked by a specific entity, for e.g a specific organization or user etc. The endpoint can have optional query parameters to filter the transaction list by the resource/object on which it acted and or status etc. By providing the invoiceId as a query parameter the caller can fetch all the transactions related to that particular invoice 112 | 113 | ``` 114 | Response Status: 200, Ok 115 | 116 | Response payload: 117 | 118 | [{ 119 | "transactionId": "tx123", 120 | 121 | "progress": { 122 | 123 | "status": "InProgress | Completed | Failed", 124 | "message": "helpful message for the user" 125 | 126 | }, 127 | 128 | "objectId": "invoice123", 129 | 130 | "objectType": "invoice", 131 | 132 | "action": "create" 133 | 134 | }, 135 | 136 | { 137 | 138 | "transactionId": "tx456", 139 | 140 | "progress": { 141 | 142 | "status": "InProgress | Completed | Failed", 143 | 144 | "message": "helpful message for the user" 145 | 146 | }, 147 | 148 | "objectId": "invoice123", 149 | 150 | "objectType": "invoice", 151 | 152 | "action": "update" 153 | 154 | } 155 | 156 | ] 157 | 158 | ``` 159 | 160 | #### GET /transactions/{id} 161 | 162 | The sample endpoint can be used for fetching a specific transaction using its transaction id. The client can poll this endpoint to figure out the status of a particular transaction using the transaction id that was returned as a part of the create resource/invoice response. The API endpoint could implement HTTP long polling to reduce the network traffic between the client and server. Certain server side runtime environments such as Node.js are more conducive to implement long polling given their inherent asynchronous programming paradigm. Web sockets, gRPC could be other alternatives for implementation. 163 | 164 | ``` 165 | Response Status: 200, Ok 166 | 167 | Response payload: 168 | 169 | { 170 | 171 | "transactionId": "tx123", 172 | 173 | "progress": { 174 | 175 | "status": "InProgress | Completed | Failed", 176 | 177 | "message": "helpful message for the user" 178 | 179 | }, 180 | 181 | "objectId": "invoice123", 182 | 183 | "objectType": "invoice", 184 | 185 | "action": "create" 186 | 187 | } 188 | 189 | ``` 190 | 191 | ### Off Chain Persistence Store 192 | 193 | The off chain persistence can be implemented using any mechanism that suits the application’s design and data structures. The design of the off chain cache will be taken up in more detail in the third part of this design pattern series, which will cover the secure sync pattern. The purpose of the off chain store is to be a replica of the data on the Blockchain that is relevant to the owning distributed application/stack. All queries can then be served directly from the off chain store rather than going to the Blockchain directly. This provides faster reads and more powerful search capabilities along with more granular administrative control over the persistence layer. 194 | 195 | ### Events Service 196 | 197 | The event service is meant to listen in for block events from the Blockchain peers and then serve them to the application layer. The design of the distributed event service will be taken up in more detail in the third part of this design pattern series, which will cover the secure sync pattern. 198 | 199 | ### Interaction Flow 200 | 201 | The figure below demonstrates the request execution flow between the different components described above. As mentioned earlier the pattern assumes a RESTful API implementation for the application stack 202 | 203 | 204 |   205 | 206 | 207 | 208 | 209 | 210 | 211 | 212 | 213 | 214 | 215 | 216 | 217 | 218 |   219 | 220 | 221 |
    222 |
  1. The user makes a request to the API for creating a new invoice
    2. 223 |
  2. The API performs basic validation of the request
    3. 224 |
  3. Post successful validation the API creates a transaction record in the off chain store to manage the lifecycle of the request. The status of the transaction is set as InProgress
    4. 225 |
  4. The API layer now submits the transaction proposal for endorsement to the required peer(s)
    5. 226 |
  5. If the proposal response indicates successful endorsement then the Blockchain transaction Id is updated in the transaction record created in step 3 and a HTTP 202, Accepted response is returned to the user, which contains the invoice and local transaction record id
    6. 227 |
  6. If the proposal response indicates failure due to chaincode validation errors or other server side errors an appropriate error response is returned to the user with the appropriate HTTP status code
    7. 228 |
  7. Assuming the user got back a 202 with the transaction id, the user client is now free to start polling the transaction endpoint to check the status of the transaction (GET /transactions/{id})
    8. 229 |
  8. Any GET calls to the API to fetch the invoice details (GET /invoices/{id}) while the transaction is InProgress or has Failed will result in a 404 HTTP Status code to indicate that the resource being requested does not exist on the server
    9. 230 |
  9. After returning the success or failure response to the user as indicated in steps 6/7, the API now submits the transaction whose proposal response was received in step 4 to the orderer(s). The API could use a queue to pick up InProgress transactions that need to be moved along in their lifecycle or it could poll the off chain store to fetch the transactions that need to be acted upon. This implementation decision is left to the user
    10. 231 |
  10. Post submission of the transaction to the orderer(s), the API layer will receive an event corresponding to the said transaction indicating whether it was committed on the Blockchain or not
    11. 232 |
  11. This event mechanism is managed by the event service component to which the API application is subscribed to
    12. 233 |
  12. If the transaction was successfully committed then the status of the local transaction record in the off chain store will be updated to Complete
    13. 234 |
  13. In addition the payload from the transaction event can then be leveraged to update the off chain store with the data that was committed to the Blockchain as a part of this transaction
    14. 235 |
  14. E.g the invoice object that now resides on the Blockchain would also be stored on the local off chain store. This will enable faster gets and searches of the invoice
    15. 236 |
  15. A GET /transactions/{id} call on the API now will indicate that the transaction is Complete
    16. 237 |
  16. The user can now fetch the invoice data by querying the API endpoint. The API will in turn fetch the invoice from its off chain store rather than going to the Blockchain
    17. 238 |
239 | 240 | ## Usage/Examples 241 | 242 | ### Example Implementations 243 | -------------------------------------------------------------------------------- /docs/design_patterns/SecureSyncPattern.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: pattern 3 | title: SecureSync 4 | folder: async-pattern 5 | categories: UI 6 | tags: 7 | - Blockchain 8 | - Difficulty-Intermediate 9 | - UI 10 | --- 11 | 12 | # SecureSync Pattern 13 | 14 | ## Overview 15 | 16 | This is the third part of the design patterns series. It covers the design of the event service that can be used to securely sync Blockchain with an application’s off chain store. This off chain store can then be leveraged for running functional queries/searches. The event service can also be used as an integration point to setup a data warehouse or provide streaming data as input to a computing framework such as Apache spark for learning and analytics. This pattern will cover the high level architecture of the event service along with the solution. 17 | 18 | 19 | ## Problem Statement 20 | 21 | ### Transaction lifecycle refresher 22 | 23 | Transaction commit on the Blockchain is an asynchronous operation. The client application submits a transaction to the orderer(s), which in turn returns a response indicating a successful submission. But the response from the orderer does not imply that the said transaction has actually been committed on the ledger. The transaction will make its way into a block, which will be distributed to all the peers on the related channel. Each peer will run the validation logic to decide whether the block will be committed on the peer’s ledger or not. Once a block is successfully committed on a peer’s ledger, the peer will emit a block event. The block event will contain information about all the transactions that were included as a part of that block. 24 | 25 | ### What the application developer wants? 26 | 27 | * The client application, which submitted the transaction to the orderer(s) needs to know whether the transaction was committed to the ledger or not 28 | 29 | * The client application needs to use complex queries and attribute based search for the data on the blockchain in an efficient and fast manner 30 | 31 | * The client application wants to securely extract the data from the Blockchain for learning and analytics in an efficient manner 32 | 33 | 34 | 35 | ## Architecture Diagram 36 | 37 | The diagram below shows the different components of a Blockchain based distributed application that leverages the event service and how the components interact with each other 38 | 39 |   40 | 41 |
42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 |
Fig.1
54 |
55 | 56 |   57 | 58 | The architecture above demonstrates how the event service can be used to publish block events to a queue of choice. These block events can be subsequently picked up by different consumers for functional application integration, learning and analytics. The next section will dive deeper into each of the components. 59 | 60 | ## Solution 61 | 62 | ### Event Service 63 | 64 | The major responsibilities of the Event service are outlined below 65 | 66 | * Registering with and listening in for block events from the channel event service 67 | 68 | * Do a possible data transformation on the block events for easier consumer consumption in addition to what filtered blocks provide 69 | 70 | * Distribute these events to interested consumers with atleast once delivery semantics 71 | 72 | * Be crash fault tolerant and support load balancing on the producer and consumer side 73 | 74 | 75 | #### Deployment 76 | 77 | The event service is meant to be part of a distributed application stack that belongs to a peer owning organization/member. This is in line with the design of other application components so as to avoid point of centralization that gets to see all the events/data on the Blockchain. The event service would be able to listen in for block events on channels that the organization/member is a part of. 78 | 79 | #### Registration 80 | 81 | The event service could leverage the default registration mechanism provided by the chosen SDK or directly write a gRPC client to interact with the Deliver and DeliverFiltered event services provided by the fabric for a more customized implementation. The block events received from the channel event service could then be transformed or pushed as is on a messaging queue of choice. This design pattern will take Kafka as an example. The event service could expose a set of REST APIs, which could be used for configuring/creating subscriptions with event services of different channels. Alternately, the event service could read the necessary configuration on start up from traditional config files and go register with the required channel event services. 82 | 83 | #### Event Producer 84 | 85 | As mentioned earlier, once the event service receives block events from the channel event service, it can push these events (with optional transformation) to a messaging queue of choice. For the sake of discussion this pattern will take Kafka as an example. 86 | 87 | * The event producer would push the block events to a kafka topic 88 | 89 | * This topic can be further subdivided into partitions by whatever partitioning logic suits the application needs. E.g Partitions could be created based on whether the block number is even or odd. Partitions could be created based on chaincodes or chaincode event types etc. The idea is to have a reasonable number of partitions so as to facilitate load balancing when writing to a topic from the producer side 90 | 91 | * The topic should have a reasonable replication factor so that the topic data is replicated across multiple kafka brokers 92 | 93 | * The event producer should also keep track of the blocks that have been successfully pushed on to the topic. This will help recover from unplanned crashes since the event producer can start processing blocks from where it left off before the crash 94 | 95 | #### Event Consumer 96 | 97 | Kafka consumer groups can be used to listen in for the block events from the consuming application. 98 | 99 | * The consumer group will help facilitate load balancing on the consumer side since each partition will be assigned to one consumer in the consumer group 100 | 101 | * In case a consumer crashes, kafka will reassign the partition to a different consumer in the consumer group ensuring that the event gets consumed 102 | 103 | * Number of consumers in the consumer group should be less than or equal to the number partitions in the topic being consumed off. 104 | 105 | * The consuming application can be horizontally scaled so as to create the targeted number of consumers in a consumer group 106 | 107 | #### Delivery Semantics 108 | 109 | Exactly once ordered delivery would be the holy grail. But a more practical implementation would be able to achieve at least once unordered delivery semantics. At least once semantics mean that a block event would not be lost but it may be delivered to the consumer multiple times. Unordered delivery means that the block events may reach the consumer out of order. The consuming application would need to handle both these cases 110 | 111 | ### Consumer Application 112 | 113 | The consumer application design was discussed in detail in the second part of the design pattern series, which covered the async pattern. In a nutshell the application layer is responsible for interacting with the Blockchain for submitting transactions and also providing the API layer for the end consumers to interact with. In addition, the consumer application is also responsible for implementing the client/consumer that can read data/block events off the queue being published to by the event service. The data provided by the event service can then be used to populate the off chain store. This in turn will help create a suitable representation of the data that is available on the ledger. The consumer application can then always directly query the off chain store instead of going to the ledger for fetching the data. 114 | 115 | As per the delivery semantics covered earlier, the consumer application would need to take care of the following 116 | 117 | Handling duplicate block event delivery 118 | 119 | Unless block event processing on the consumer side is idempotent, the consuming application would need to keep track of the blocks it has already consumed. This way the consuming application can ignore any duplicate block events 120 | 121 | Handling unordered block event delivery 122 | 123 | Here is a sample approach to handle out of order block events. It can be adapted to suit an application’s needs. 124 | 125 | For every object/resource that is being acted upon in the chaincode and which will be replicated in the off chain store, the consuming application should maintain the following lookup data 126 | 127 | ``` 128 | { 129 | "objectId": "obj123", 130 | "lastBlockSeen": 3 131 | } 132 | ``` 133 | 134 | Every time the consuming application sees a block related to an object, it should update the lookup data to reflect that. The consuming application can decide to process a transaction in the block only if the object being acted upon has not already seen a later block 135 | 136 | Now let’s see how this approach pans out with an example scenario. Let us assume that the following transactions were committed on to the ledger 137 | 138 | Transaction 1, Block 1: Creates a new object (obj123) on the ledger 139 | 140 | Transaction 2, Block 2: Updates the object (obj123) on the ledger 141 | 142 | Transaction 3, Block 3: Deletes the object (obj123) on the ledger 143 | 144 | Let’s assume the consumer application see’s these blocks in reverse order (3,2,1). 145 | 146 | * The consumer application received block 3 147 | 148 | * It extracts transaction 3 from the block 149 | 150 | * It determines from the transaction payload that the object to be deleted is obj123 151 | 152 | * It does a lookup for obj123. There is no entry for this object. This indicates that the object is yet to be created and there is nothing to delete in the off chain store. 153 | 154 | * The consumer creates an entry for obj123 indicating that the last block seen is 3 155 | 156 | * Now the consumer app receives block 2. It extracts the transaction and determines that object being updated is obj123 157 | 158 | * The consumer performs the lookup for obj123 and finds that an operation corresponding to block 3 has already been performed for this object. The consumer application ignores obj123 as a part of block event 2 159 | 160 | * Same process is repeated when the consumer application receives block 1 161 | 162 | The flowchart below captures the logical flow to follow when the consumer application receives a block event 163 | 164 |   165 | 166 |
167 | 168 | 169 | 170 | 171 | 172 | 173 | 174 | 175 | 176 | 177 | 178 |
Fig.2
179 |
180 | 181 |   182 | 183 | #### Considerations 184 | 185 | * Capture the action being performed (INSERT/UPDATE/DELETE) in the transaction payload or via a chaincode event payload. The consumer app would need to understand the action being performed to handle it appropriately 186 | 187 | * In case of an update, make sure to include the whole data object in the transaction payload or chaincode event. This would ensure that if an update transaction is received before an insert transaction, the app can simply create/insert the said object into the off chain store. If the update transaction included only the fields that were updated then the consumer app would not be able to process this update on the off chain store since the insert transaction has not been processed yet. 188 | 189 | * The logical flow above deals with a single object in isolation. But there could be relationships between different objects, where in one object references another object. For e.g object2 needs to access/read certain information from object1 for it to be processed. Now let’s assume that object2 got created as a part of block2 and object1 was a part of block1. But the consumer application receives block2 before block1. In this case the application needs to be aware of this dependency and can proactively go and fetch the block1 data from the ledger and update the off chain cache. This will facilitate the processing of object2 now since the dependency has been resolved. 190 | 191 | ### Off Chain Persistence Store 192 | 193 | The off chain persistence can be implemented using any mechanism that suits the application’s design and data structures. The purpose of the off chain store is to be a replica of the data on the Blockchain that is relevant to the owning distributed application/stack. All queries can then be served directly from the off chain store rather than going to the Blockchain directly. This provides faster reads and more powerful search capabilities along with more granular administrative control over the persistence layer. 194 | 195 | #### Considerations 196 | 197 | * The off chain store should be distributed and scoped to a distributed application stack and not centralized 198 | 199 | * The off chain store is meant to store the data that is relevant to the owning application stack 200 | 201 | * All reads via the corresponding application’s stack can now go directly to the off chain storage rather than Blockchain 202 | 203 | ### Analytics 204 | 205 | The data being pushed by the event service can also be used for analytics and learning. 206 | 207 | * A dedicated consumer can be used to read data off the queue for analytics and learning purpose 208 | 209 | * Assuming the data is in JSON format, the consumer could then transform data into a traditional tabular relational format or 210 | appropriate format and populate a traditional data warehouse 211 | 212 | * The data warehouse could then be used for standard business analytics, visualization, reporting etc 213 | 214 | * The data warehouse could also be used as a source for a computing framework such as Apache spark, which could use historical 215 | data as a part of its machine learning models 216 | 217 | * Another consumer could be used to read the data off the queue and stream it into Apache spark directly as well. 218 | 219 | ## Usage/Examples 220 | 221 | ### Example Implementations 222 | -------------------------------------------------------------------------------- /docs/design_patterns/UIResponsivenessPattern.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: pattern 3 | title: UIResponsiveness 4 | folder: ui-responsiveness-pattern 5 | categories: UI 6 | tags: 7 | - Blockchain 8 | - Difficulty-Intermediate 9 | - UI 10 | --- 11 | 12 | # Non Blocking UI for Complex Blockchain based applications 13 | 14 | ## Overview 15 | 16 | This design pattern starts by taking the reader through the problems that plague a traditional user interface when interacting with complex backend APIs with high response times. The design pattern then presents some key concepts/approaches that can be used to alleviate the problem at hand and follows up by providing an end to end architecture and solution overview of how the different solution components can come together to create an interactive user interface. This design pattern uses Hyperledger Fabric 1.4 as its Blockchain reference. 17 | 18 | 19 | ## Problem Statement 20 | 21 | In a traditional user interface when the user performs an action such as submitting a form, the user interface generally makes a blocking call to a backed API. The user generally sees the page refresh or a progress bar to indicate that the operation is in progress. Once the user interface receives a response from the backend API, the appropriate success or failure message is then displayed to the user. Normally this model works quite well assuming the backend API has a fast response time. But when dealing with certain Blockchain based applications we cannot expect millisecond response times (As per current level of maturity). Listed below are some considerations that are relevant for complex enterprise grade Blockchain based applications 22 | 23 | ### User interaction with the UI 24 | Suppose a user fills up an invoice creation form and submits it via the user interface. The invoice content needs to be stored on the Blockchain. At the very minimum the backed application would need to perform the following operations 25 | * Send transaction proposals and fetch the corresponding endorsements 26 | * Submit the transaction proposal responses to the orderer(s) 27 | * Wait for a Block/Chaincode event from the event service 28 | * Return the appropriate success/failure response to the user interface 29 | 30 | During this time the user is blocked waiting for the invoice creation request to complete and is watching a progress bar. 31 | 32 | The scenario above is a simplification. Production grade microservices based solutions are quite complex and additionally need to account for the following 33 | * Microservices architecture leading to multiple network calls 34 | * Retries owing to MVCC (Multi version concurrency control) errors from the Blockchain 35 | * Distributed transactions across services 36 | * Resiliency/Recovery scenarios leading to multiple retries for network/connectivity errors 37 | * Network latency 38 | 39 | All the factors above along with Blockchain’s transaction paradigm, which includes asynchronous transaction submission make for a fairly complex backend API with a high response time. 40 | 41 | ### What the user wants? 42 | * A fast and responsive UI that does not block the user from interacting with other facets of the user interface while an operation is in progress 43 | * Fast and trusted reads 44 | 45 | 46 | ## Architecture Diagram 47 | 48 | HTTP Long polling and websockets are two possible approaches that can be used to implement the communication between the client and the server for this pattern. This design pattern uses the HTTP long polling technique as demonstrated in the architecture below. Part 2 of this pattern will demonstrate the websockets approach 49 | 50 |   51 | 52 |
53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 |
Fig.1
65 |
66 | 67 |   68 | 69 | The architecture above covers one possible approach for designing a distributed application that talks to Blockchain and caters to user interface requests. The next section will dive deeper into each of the components. 70 | 71 | 72 | ## Solution 73 | 74 | ### User Interface 75 | 76 | This design pattern will take the example of a web based user interface. 77 | * **SPA (Single Page Application)** with client side MVC: This is a pretty standard way of architecting user interfaces nowadays. Angular JS is a popular framework that is used by UI developers to implement client side MVC pattern. This ensures the burden of view generation is now on the client browser and simplifies the backend APIs to concern itself purely with the data. 78 | 79 | * **AJAX**: Asynchronous JavaScript. Again, this is a fairly standard technique used by the client to fetch information from the server in an asynchronous manner in the background. The user can continue to interact with the user interface in the meantime as the entire page need not be refreshed 80 | 81 | * **Non-Blocking UI**: The user should not be blocked and made to stare at a progress bar while a request is being processed by the backend APIs. API calls with high response times such as 82 | 83 | * Writes to the Blockchain 84 | * File uploads etc 85 | 86 | Should be asynchronous so as to respond to the client immediately with transaction/request information for further reconciliation 87 | 88 |   89 | 90 |
91 | 92 | 93 | 94 | 95 | 96 | 97 | 98 | 99 | 100 | 101 | 102 |
Fig.2
103 |
104 | 105 |   106 | 107 | Asynchronous Backend API 108 | In order to implement a non-blocking user interface the backend data APIs need to support asynchronous request execution. The sequence diagram below captures the interaction between the user interface and the server side data APIs 109 | 110 | 111 |   112 | 113 | 114 | 115 | 116 | 117 | 118 | 119 | 120 | 121 | 122 | 123 | 124 | 125 |   126 | 127 |
    128 |
  1. The user fills up and submits an invoice generation form in the user interface
    2. 129 |
  2. The user interface sends an AJAX request to the server side REST APIs
    3. 130 |
  3. The REST API receives the request and performs some basic validations. If the validations were successful the API responds back with a HTTP 202 Accepted response. This response also contains a transactionId. This indicates that the server has accepted the request and is processing it.
    4. 131 |
  4. The user interface on receiving this response from the server can immediately notify the user that the request for invoice generation is in progress and that the user would be notified once it completes. In the meantime the user is free to interact with the user interface for any other tasks
    5. 132 |
  5. The user interface now starts polling the server transaction API endpoint with the transactionId as a query parameter
    6. (GET ../transactions?transactionId=tx123) 133 | 134 |
  6. The server’s transaction API receives this request and starts polling its transaction store for the status of this transaction. The server keeps the connection open till the transaction status has reached a terminal state (Completed/Failed) or if a timeout has occurred. Once the transaction reaches terminal state or timeout occurs the server responds back to the client with the appropriate transaction information, including status. This is essentially long polling
    7. 135 |
  7. If the user interface receives a transaction failed/completed status from the API then the user interface can notify the user that the invoice has been committed on the Blockchain or in case of failure recommend the appropriate next action
    8. 136 |
  8. In case the transaction status received from the API still says InProgress then the user interface can continue to poll the transaction API endpoint till a terminal state is reached or the user interface decides to give up.
    9. 137 |
  9. If user interface has exhausted all of its polling attempts and the transaction is still InProgress then the user interface could stop polling and wait for the transaction information to be fetched as a part of a user triggered action such as Get All Invoices or View pending invoices etc
    10. 138 |
139 | 140 | For a deeper understanding of how to design an asynchronous RESTful API that talks to Blockchain please refer the [design pattern for Asynchronous API](https://github.com/IBM/BlockchainDevelopmentDesignPatterns/blob/master/docs/design_patterns/AsyncAPIPattern.md) 141 | 142 | For a deeper understanding of and code samples for Long polling please refer to the [Asynchronous API design pattern](https://github.com/IBM/BlockchainDevelopmentDesignPatterns/blob/master/docs/design_patterns/AsyncAPIPattern.md) 143 | 144 | ### Off Chain storage 145 | 146 | An off chain storage can help reduce the query calls to Blockchain and also provide richer search capabilities and faster reads. By leveraging the off chain storage the API can get data to the client quicker, hence improving the responsiveness of the UI and subsequently user experience. 147 | 148 | ### Considerations 149 | 150 | * The off chain storage should be distributed and scoped to a distributed application stack and not centralized 151 | * The off chain storage can be used to store all outgoing transactions from the corresponding application stack 152 | * The off chain storage can be synced with the Blockchain via a secure sync event service and replicate the data that exists on Blockchain (transactions submitted by other application stacks) 153 | * All reads via the corresponding application’s stack can now go directly to the off chain storage rather than Blockchain 154 | 155 | 156 | ## Usage/Examples 157 | 158 |   159 | 160 | 161 | 162 | 163 | 164 | 165 | 166 | 167 | 168 | 169 | 170 | 171 | 172 |   173 | 174 |   175 | 176 | 177 | 178 | 179 | 180 | 181 | 182 | 183 | 184 | 185 | 186 | 187 | 188 |   189 | 190 |   191 | 192 | 193 | 194 | 195 | 196 | 197 | 198 | 199 | 200 | 201 | 202 | 203 | 204 |   205 | 206 |   207 | 208 | 209 | 210 | 211 | 212 | 213 | 214 | 215 | 216 | 217 | 218 | 219 | 220 |   221 | 222 |   223 | 224 | 225 | 226 | 227 | 228 | 229 | 230 | 231 | 232 | 233 | 234 | 235 | 236 |   237 | 238 |   239 | 240 | 241 | 242 | 243 | 244 | 245 | 246 | 247 | 248 | 249 | 250 | 251 | 252 |   253 | 254 |   255 | 256 | 257 | 258 | 259 | 260 | 261 | 262 | 263 | 264 | 265 | 266 | 267 | 268 |   269 | 270 |   271 | 272 | 273 | 274 | 275 | 276 | 277 | 278 | 279 | 280 | 281 | 282 | 283 | 284 |   285 | 286 |   287 | 288 | 289 | 290 | 291 | 292 | 293 | 294 | 295 | 296 | 297 | 298 | 299 | 300 |   301 | 302 |   303 | 304 | 305 | 306 | 307 | 308 | 309 | 310 | 311 | 312 | 313 | 314 | 315 | 316 |   317 | 318 |   319 | 320 | 321 | 322 | 323 | 324 | 325 | 326 | 327 | 328 | 329 | 330 | 331 | 332 |   333 | 334 |   335 | 336 | 337 | 338 | 339 | 340 | 341 | 342 | 343 | 344 | 345 | 346 | 347 | 348 |   349 | ### Example Implementations 350 | 351 | 352 | -------------------------------------------------------------------------------- /docs/design_patterns/images/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/IBM/BlockchainDevelopmentDesignPatterns/6cf82348739f239c23d67aeebc9f022d1f1ab2af/docs/design_patterns/images/.DS_Store -------------------------------------------------------------------------------- /docs/design_patterns/images/architecture.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/IBM/BlockchainDevelopmentDesignPatterns/6cf82348739f239c23d67aeebc9f022d1f1ab2af/docs/design_patterns/images/architecture.png -------------------------------------------------------------------------------- /docs/design_patterns/images/async-images/fig1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/IBM/BlockchainDevelopmentDesignPatterns/6cf82348739f239c23d67aeebc9f022d1f1ab2af/docs/design_patterns/images/async-images/fig1.png -------------------------------------------------------------------------------- /docs/design_patterns/images/async-images/fig2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/IBM/BlockchainDevelopmentDesignPatterns/6cf82348739f239c23d67aeebc9f022d1f1ab2af/docs/design_patterns/images/async-images/fig2.png -------------------------------------------------------------------------------- /docs/design_patterns/images/fig1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/IBM/BlockchainDevelopmentDesignPatterns/6cf82348739f239c23d67aeebc9f022d1f1ab2af/docs/design_patterns/images/fig1.png -------------------------------------------------------------------------------- /docs/design_patterns/images/fig2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/IBM/BlockchainDevelopmentDesignPatterns/6cf82348739f239c23d67aeebc9f022d1f1ab2af/docs/design_patterns/images/fig2.png -------------------------------------------------------------------------------- /docs/design_patterns/images/fig3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/IBM/BlockchainDevelopmentDesignPatterns/6cf82348739f239c23d67aeebc9f022d1f1ab2af/docs/design_patterns/images/fig3.png -------------------------------------------------------------------------------- /docs/design_patterns/images/nonBlocking_images/blockingui-black.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/IBM/BlockchainDevelopmentDesignPatterns/6cf82348739f239c23d67aeebc9f022d1f1ab2af/docs/design_patterns/images/nonBlocking_images/blockingui-black.png -------------------------------------------------------------------------------- /docs/design_patterns/images/nonBlocking_images/non-blockingui-black.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/IBM/BlockchainDevelopmentDesignPatterns/6cf82348739f239c23d67aeebc9f022d1f1ab2af/docs/design_patterns/images/nonBlocking_images/non-blockingui-black.png -------------------------------------------------------------------------------- /docs/design_patterns/images/nonBlocking_images/screen_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/IBM/BlockchainDevelopmentDesignPatterns/6cf82348739f239c23d67aeebc9f022d1f1ab2af/docs/design_patterns/images/nonBlocking_images/screen_1.png -------------------------------------------------------------------------------- /docs/design_patterns/images/nonBlocking_images/screen_10.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/IBM/BlockchainDevelopmentDesignPatterns/6cf82348739f239c23d67aeebc9f022d1f1ab2af/docs/design_patterns/images/nonBlocking_images/screen_10.png -------------------------------------------------------------------------------- /docs/design_patterns/images/nonBlocking_images/screen_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/IBM/BlockchainDevelopmentDesignPatterns/6cf82348739f239c23d67aeebc9f022d1f1ab2af/docs/design_patterns/images/nonBlocking_images/screen_2.png -------------------------------------------------------------------------------- /docs/design_patterns/images/nonBlocking_images/screen_3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/IBM/BlockchainDevelopmentDesignPatterns/6cf82348739f239c23d67aeebc9f022d1f1ab2af/docs/design_patterns/images/nonBlocking_images/screen_3.png -------------------------------------------------------------------------------- /docs/design_patterns/images/nonBlocking_images/screen_4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/IBM/BlockchainDevelopmentDesignPatterns/6cf82348739f239c23d67aeebc9f022d1f1ab2af/docs/design_patterns/images/nonBlocking_images/screen_4.png -------------------------------------------------------------------------------- /docs/design_patterns/images/nonBlocking_images/screen_5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/IBM/BlockchainDevelopmentDesignPatterns/6cf82348739f239c23d67aeebc9f022d1f1ab2af/docs/design_patterns/images/nonBlocking_images/screen_5.png -------------------------------------------------------------------------------- /docs/design_patterns/images/nonBlocking_images/screen_6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/IBM/BlockchainDevelopmentDesignPatterns/6cf82348739f239c23d67aeebc9f022d1f1ab2af/docs/design_patterns/images/nonBlocking_images/screen_6.png -------------------------------------------------------------------------------- /docs/design_patterns/images/nonBlocking_images/screen_7.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/IBM/BlockchainDevelopmentDesignPatterns/6cf82348739f239c23d67aeebc9f022d1f1ab2af/docs/design_patterns/images/nonBlocking_images/screen_7.png -------------------------------------------------------------------------------- /docs/design_patterns/images/nonBlocking_images/screen_8.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/IBM/BlockchainDevelopmentDesignPatterns/6cf82348739f239c23d67aeebc9f022d1f1ab2af/docs/design_patterns/images/nonBlocking_images/screen_8.png -------------------------------------------------------------------------------- /docs/design_patterns/images/nonBlocking_images/screen_9.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/IBM/BlockchainDevelopmentDesignPatterns/6cf82348739f239c23d67aeebc9f022d1f1ab2af/docs/design_patterns/images/nonBlocking_images/screen_9.png -------------------------------------------------------------------------------- /docs/design_patterns/images/nonBlocking_images/temp: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /docs/design_patterns/images/securesync-images/fig-1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/IBM/BlockchainDevelopmentDesignPatterns/6cf82348739f239c23d67aeebc9f022d1f1ab2af/docs/design_patterns/images/securesync-images/fig-1.png -------------------------------------------------------------------------------- /docs/design_patterns/images/securesync-images/fig-2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/IBM/BlockchainDevelopmentDesignPatterns/6cf82348739f239c23d67aeebc9f022d1f1ab2af/docs/design_patterns/images/securesync-images/fig-2.png -------------------------------------------------------------------------------- /faq.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: page 3 | title: FAQ 4 | permalink: /faq/ 5 | icon: fa-question 6 | page-index: 2 7 | --- 8 | 9 | ### Q1: What is the difference between ui responsiveness and xxxxx ? {#Q1} 10 | 11 | While the implementation is similar they solve different problems. xxxxxx 12 | 13 | ### Q2: What is the performance difference with using UI Responsiveness? {#Q2} 14 | 15 | This implementation...... --------------------------------------------------------------------------------