├── images ├── .gitkeep ├── active-remote-sequence-diagram.png ├── many-subscribers-sequence-diagram.png └── synchronous-and-event-driven-platform.png ├── .gitignore ├── 000-cover.md ├── 010-chapter-00.md ├── README.md ├── 002-who-is-this-book-for.md ├── 007-copyright.md ├── 004-what-you-need.md ├── 006-acknowledgements.md ├── 003-whats-in-this-book.md ├── 150-chapter-14.md ├── 005-online-resources.md ├── 999-scratchpad.md ├── 008-table-of-contents.md ├── 001-preface.md ├── 999-outline.md ├── 110-chapter-10.md ├── 040-chapter-03.md ├── 050-chapter-04.md ├── 060-chapter-05.md ├── 020-chapter-01.md ├── 120-chapter-11.md ├── 080-chapter-07.md ├── 070-chapter-06.md ├── 030-chapter-02.md ├── 090-chapter-08.md ├── 100-chapter-09.md ├── 130-chapter-12.md └── 140-chapter-13.md /images/.gitkeep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .bash_history 2 | .viminfo 3 | .DS_Store 4 | 5 | -------------------------------------------------------------------------------- /images/active-remote-sequence-diagram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kevinwatson/rails-microservices-book/HEAD/images/active-remote-sequence-diagram.png -------------------------------------------------------------------------------- /images/many-subscribers-sequence-diagram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kevinwatson/rails-microservices-book/HEAD/images/many-subscribers-sequence-diagram.png -------------------------------------------------------------------------------- /images/synchronous-and-event-driven-platform.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kevinwatson/rails-microservices-book/HEAD/images/synchronous-and-event-driven-platform.png -------------------------------------------------------------------------------- /000-cover.md: -------------------------------------------------------------------------------- 1 | # Building Distributed Rails Applications 2 | 3 | ## Using Protocol Buffers, NATS and RabbitMQ 4 | 5 | ### Kevin Watson 6 | 7 | [Next >>](001-preface.md) 8 | -------------------------------------------------------------------------------- /010-chapter-00.md: -------------------------------------------------------------------------------- 1 | > We who cut mere stones must always be envisioning cathedrals. - Andrew Hunt, The Pragmatic Programmer: From Journeyman to Master 2 | 3 | [Next >>](020-chapter-01.md) 4 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Building Distributed Rails Applications 2 | 3 | ## Using Protocol Buffers, NATS and RabbitMQ 4 | 5 | Start at the beginning: [Cover](000-cover.md) 6 | 7 | Preface: [Preface](001-preface.md) 8 | 9 | Table of Contents: [Table of Contents](008-table-of-contents.md) 10 | 11 | Related code: https://github.com/kevinwatson/rails-microservices-sample-code 12 | -------------------------------------------------------------------------------- /002-who-is-this-book-for.md: -------------------------------------------------------------------------------- 1 | ## Who Is This Book For? 2 | 3 | This book is for anyone who is interested in building a distributed platform consisting of multiple applications, also known as a microservice platform. While all of the examples in this book are written in Ruby using the Ruby on Rails framework, a software developer using any programming language could gain insight and translate the functionality using their programming language and framework of choice. 4 | 5 | [Next >>](003-whats-in-this-book.md) 6 | -------------------------------------------------------------------------------- /007-copyright.md: -------------------------------------------------------------------------------- 1 | ### Building Distributed Rails Applications 2 | 3 | #### Using Protocol Buffers and NATS 4 | 5 | All rights reserved 6 | 7 | Copyright © Kevin J. Watson 8 | 9 | The information and techniques applied by the author in this book are believed to be reliable. However, the author does not guarantee the accuracy, or completeness of the information supplied and is not responsible for errors, omissions, or the results obtained from the use of such information. 10 | 11 | [Next >>](008-table-of-contents.md) 12 | -------------------------------------------------------------------------------- /004-what-you-need.md: -------------------------------------------------------------------------------- 1 | ## What You Need 2 | 3 | You'll need a computer that can run Docker Desktop. You'll also need an Internet connection if you want to download the code samples. 4 | 5 | You can use all of the tools presented in this book for free. The software we'll use is either free or open source software that you can freely download and install. 6 | 7 | The projects presented in this book were developed on a MacBook in a Docker Desktop container environment. The commands should work the same whether you're using Docker on Linux, Windows or macOS. 8 | 9 | [Next >>](005-online-resources.md) 10 | -------------------------------------------------------------------------------- /006-acknowledgements.md: -------------------------------------------------------------------------------- 1 | ## Acknowledgements 2 | 3 | I'd first like to thank my wife, Monica for her patience and understanding when I pursue my hobbies such as writing. 4 | 5 | I'd like to thank my employer, [MX](https://mx.com), and the many people who have built this company for their expertise and many contributions to the Ruby on Rails ecosystem. They have published documentation and numerous Ruby gems which make building microservices on Ruby on Rails easy. 6 | 7 | Special thanks goes to Chris Miller who has been my Ruby on Rails mentor over the course of several job changes. I'd also like to thank Mohamed Asan, Igor Morgunov, Naoto Suzuki, Luilver Garces Brinas, Mark Oveson and Sheldon Teerlink for their feedback and contributions to this book. 8 | 9 | [Next >>](007-copyright.md) 10 | -------------------------------------------------------------------------------- /003-whats-in-this-book.md: -------------------------------------------------------------------------------- 1 | ## What's in This Book? 2 | 3 | This book has several sections. We'll discuss several architectural patterns for building distributed or not-so-distributed systems. We'll discuss various methods of moving data between applications. 4 | 5 | We'll discuss the Ruby on Rails framework, and the building blocks that provide access to the data your application will process. We'll discuss messaging platforms. We'll discuss modeling data entities and their relationships in a distributed environment. 6 | 7 | We'll walk through all of the steps required to spin up a new environment that consists of a NATS server and two Ruby on Rails applications, one having a model backed by a database and the other having a model that remotely accesses the data from the first application. 8 | 9 | We'll discuss event-driven messaging and when it is appropriate. We'll build two applications that communicate via RabbitMQ. 10 | 11 | Finally, we'll build a platform that uses both synchronous and event-driven architectural patterns to share data between services. 12 | 13 | [Next >>](004-what-you-need.md) 14 | -------------------------------------------------------------------------------- /150-chapter-14.md: -------------------------------------------------------------------------------- 1 | ### Chapter 14 - Summary 2 | 3 | Thank you for coming with me on this journey of *Building Distributed Rails Applications.* I hope you had as much fun reading as I did writing this book. My hope is that this book has taken the mystery out of building distributed services and can help you design and implement your own scalable platform. 4 | 5 | In this book, we've discussed a microservice architecture, serialization, messaging systems and different ways to share data. We've built microservices using the Ruby on Rails framework, which allowed us to quickly spin up new applications with just a few terminal commands. We've used open-sourced gems that allow our applications to be easily configured and share data using both real-time and event-driven patterns. With this architecture, you will have the flexibility to scale the parts of the system that grow at a faster pace or require additional computing resources. 6 | 7 | The learning does not stop here! Spin up a proof of concept project at work, get involved in a local user group and contribute to this or other documentation. As technologists, we're all on similar journeys, doing something we love and at the same time building our businesses. 8 | -------------------------------------------------------------------------------- /005-online-resources.md: -------------------------------------------------------------------------------- 1 | ## Online Resources 2 | 3 | The code referenced in this book is available on the book's website at https://github.com/kevinwatson/rails-microservices-sample-code. You can either download and run the sample code, or build the same applications from scratch by following along in Chapters 9, 12 and 13. 4 | 5 | In addition to the sample code on the GitHub website, the following websites will be referenced in this book: 6 | 7 | #### Active Publisher and Action Subscriber 8 | 9 | * https://github.com/mxenabled/action_subscriber 10 | * https://github.com/mxenabled/active_publisher 11 | 12 | #### Active Remote 13 | 14 | * https://github.com/liveh2o/active_remote 15 | * https://github.com/liveh2o/protobuf-activerecord 16 | 17 | #### Docker 18 | 19 | * https://docker.com 20 | * https://docs.docker.com/compose 21 | 22 | #### NATS 23 | 24 | * https://nats.io 25 | 26 | #### Protocol Buffers (Protobuf) 27 | 28 | * https://github.com/mxenabled/protobuf-nats 29 | * https://github.com/ruby-protobuf 30 | * https://github.com/ruby-protobuf/protobuf/wiki/Serialization 31 | 32 | #### RabbitMQ 33 | 34 | * https://www.rabbitmq.com 35 | 36 | #### Ruby and Ruby on Rails 37 | 38 | * https://www.ruby-lang.org 39 | * https://rubygems.org 40 | * https://rubyonrails.org 41 | 42 | [Next >>](006-acknowledgements.md) 43 | -------------------------------------------------------------------------------- /999-scratchpad.md: -------------------------------------------------------------------------------- 1 | # Scratch Pad 2 | 3 | ## Generating protobuf classes 4 | 5 | ```console 6 | $ docker-compose -f docker-compose.builder.yml run builder bash 7 | # cd protobuf 8 | # rake protobuf:compile 9 | ``` 10 | 11 | ## Encoding protobuf messages 12 | 13 | ```console 14 | # irb 15 | > require_relative 'lib/person_message.pb' 16 | > PersonMessage.encode(id: 1, first_name: "George", last_name: "Costanza") 17 | > PeopleMessageList.encode(records: [PersonMessage.new(id: 1, first_name: "George", last_name: "Costanza"), PersonMessage.new(id: 2, first_name: "Elaine", last_name: "Benes")]) 18 | ``` 19 | 20 | ## Diagrams 21 | 22 | Figure 9-1 23 | 24 | ``` 25 | sequenceDiagram 26 | participant Remote as Active Remote App 27 | participant N as NATS 28 | participant Record as Active Record App 29 | Remote->>N: Employee proto 30 | N->>Record: Employee proto 31 | Record->>N: Employee proto 32 | N->>Remote: Employee proto 33 | ``` 34 | 35 | Figure 12-1 36 | 37 | ``` 38 | sequenceDiagram 39 | participant Record as Active Record/Publisher App 40 | participant R as RabbitMQ 41 | participant S1 as Action Subscriber App 1 42 | participant S2 as Action Subscriber App 2 43 | participant S3 as Action Subscriber App 3 44 | Record->>R: Employee proto 45 | R->>S1: Employee proto 46 | R->>S2: Employee proto 47 | R->>S3: Employee proto 48 | ``` 49 | 50 | Figure 13-1 51 | 52 | ``` 53 | sequenceDiagram 54 | participant Remote as Active Remote App 55 | participant N as NATS 56 | participant Record as Active Record/Publisher App 57 | participant R as RabbitMQ 58 | participant S1 as Action Subscriber App 1 59 | participant S2 as Action Subscriber App 2 60 | Remote->>N: Employee proto 61 | N->>Record: Employee proto 62 | Record->>N: Employee proto 63 | N->>Remote: Employee proto 64 | Record->>R: Employee proto 65 | R->>S1: Employee proto 66 | R->>S2: Employee proto 67 | ``` 68 | -------------------------------------------------------------------------------- /008-table-of-contents.md: -------------------------------------------------------------------------------- 1 | ### TABLE OF CONTENTS 2 | 3 | * [Chapter 1 - Microservices](020-chapter-01.md) 4 | * Service Architectures 5 | * Tell me more about microservices 6 | * Why should I use microservices? 7 | * Resources 8 | * Wrap-up 9 | * [Chapter 2 - Service Communications](030-chapter-02.md) 10 | * Introduction 11 | * Protocols 12 | * Data Serialization 13 | * Messaging Systems 14 | * Wrap-up 15 | * [Chapter 3 - Ruby and Ruby on Rails](040-chapter-03.md) 16 | * Ruby 17 | * Ruby on Rails 18 | * Resources 19 | * Wrap-up 20 | * [Chapter 4 - Active Record and Active Model](050-chapter-04.md) 21 | * Introduction 22 | * Resources 23 | * Wrap-up 24 | * [Chapter 5 - Active Remote](060-chapter-05.md) 25 | * Introduction 26 | * Philosophy 27 | * Design 28 | * Implementation 29 | * Resources 30 | * Wrap-up 31 | * [Chapter 6 - Messaging Systems - NATS](070-chapter-06.md) 32 | * Introduction 33 | * Let's Run It 34 | * Resources 35 | * Wrap-up 36 | * [Chapter 7 - Data Relationships](080-chapter-07.md) 37 | * Introduction 38 | * Primary and Foreign Keys 39 | * Natural vs Surrogate Keys 40 | * Database vs App Generated 41 | * When to use each 42 | * Resources 43 | * Wrap-up 44 | * [Chapter 8 - Protocol Buffers (Protobuf)](090-chapter-08.md) 45 | * Introduction 46 | * Philosophy 47 | * Implementation 48 | * Resources 49 | * Wrap-up 50 | * [Chapter 9 - Active Remote Microservice Sandbox](100-chapter-09.md) 51 | * Introduction 52 | * Install Docker 53 | * Implementation 54 | * Resources 55 | * Wrap-up 56 | * [Chapter 10 - Event Driven Messaging](110-chapter-10.md) 57 | * Introduction 58 | * Implementation 59 | * Wrap-up 60 | * [Chapter 11 - Messaging Systems - Rabbit MQ](120-chapter-11.md) 61 | * Introduction 62 | * Let's Run It 63 | * Resources 64 | * Wrap-up 65 | * [Chapter 12 - Event Driven Messaging Sandbox](130-chapter-12.md) 66 | * Introduction 67 | * What We'll Need 68 | * Implementation 69 | * Resources 70 | * Wrap-up 71 | * [Chapter 13 - Active Remote with Events Sandbox](140-chapter-13.md) 72 | * Introduction 73 | * What We'll Need 74 | * Implementation 75 | * Resources 76 | * Wrap-up 77 | * [Chapter 14 - Summary](150-chapter-14.md) 78 | 79 | [Next >>](010-chapter-00.md) 80 | -------------------------------------------------------------------------------- /001-preface.md: -------------------------------------------------------------------------------- 1 | ## Preface 2 | 3 | Welcome to *Building Distributed Rails Applications.* 4 | 5 | Ruby on Rails is a framework built on the Ruby programming language. The Rails framework provides you with the tools you need to easily build a database-backed application. Rails has achieved widespread popularity - many popular websites run on Rails, including Shopify, Basecamp, and GitHub. 6 | 7 | Distributed application architecture defines separate, specialized modules or components of a system which, as a whole, provide the functionality that your users require. Distributed applications can be configured to scale up and down as needed, specifically for modules that require extra computing power. For example, the module that renders the login form may not require much compute power, but the module that optimizes your customer's uploaded photos may need to scale out each time a user uploads a fresh batch of files. 8 | 9 | This book will walk you through building distributed applications in Ruby on Rails. We will discuss monolithic applications, breaking those applications into smaller units (microservices), and describe ways to share data between these services. We'll use a handful of Ruby gems, generously open sourced by the people at [MX](https://mx.com). MX is a financial services company based in Utah. The people at MX have built a distributed, heterogeneous platform from the ground up that processes and analyzes billions of financial transactions every month. 10 | 11 | MX's generous contributions to Protobuf, RabbitMQ and NATS open standards includes (but is not limited to) the following gems: `active_remote`, `protobuf-nats`, `action_subscriber`, `active_publisher` and `protobuf-activerecord`. We'll discuss each gem in detail in this book. 12 | 13 | Ruby continues to evolve and grow in popularity. MX's open source contributions help ensure that Ruby continues to be a viable choice when designing and deploying modern distributed systems. 14 | 15 | Because MX's platform is built on top of open standards (e.g. Protobuf, RabbitMQ and NATS), new services can be spun up in any programming language. As long as a service can understand Protobuf messages and connect to NATS and RabbitMQ, it can respond to messages from any application written in any supported language. 16 | 17 | Even if your go-to language isn't Ruby, hopefully this book will give you an overview of how to design and build distributed services. 18 | 19 | [Next >>](002-who-is-this-book-for.md) 20 | -------------------------------------------------------------------------------- /999-outline.md: -------------------------------------------------------------------------------- 1 | ## Outline 2 | 3 | #### Title: Building Distributed Rails Applications 4 | 5 | 1. Cover 6 | 1. Preface 7 | 1. Who this book is for 8 | 1. What's in this book 9 | 1. What you need 10 | 1. Online resources 11 | 1. Acknowledgements 12 | 1. Chapter 0 - Setup 13 | 1. Why Docker? 14 | 1. Install Docker Desktop 15 | 1. Install Ruby 16 | 1. Wrap-up 17 | 1. Chapter 1 - Microservices Overview 18 | 1. Monoliths vs microservices vs function as a service 19 | 1. What is a microservice? 20 | 1. Why should I design microservices? 21 | 1. Wrap-up 22 | 1. Chapter 2 - Service Communications 23 | 1. Introduction 24 | 1. JSON 25 | 1. XML 26 | 1. Message Bus 27 | 1. Shared Database 28 | 1. Wrap-up 29 | 1. Chapter 3 - Ruby on Rails 30 | 1. Why Ruby on Rails? 31 | 1. The Ruby language 32 | 1. Package management with Bundler 33 | 1. Popularity 34 | 1. Rails generators 35 | 1. Included features 36 | 1. Rails in the enterprise 37 | 1. Wrap-up 38 | 1. Chapter 4 - ActiveRecord and ActiveModel 39 | 1. ActiveModel 40 | 1. ActiveRecord 41 | 1. ActiveResource 42 | 1. What's the difference 43 | 1. Wrap-up 44 | 1. Chapter 5 - ActiveRemote 45 | 1. Introduction 46 | 1. Philosophy 47 | 1. Wrap-up 48 | 1. Chapter 6 - Messaging Systems 49 | 1. Overview 50 | 1. NATS Server 51 | 1. Rabbit MQ ?? 52 | 1. Setup 53 | 1. Testing 54 | 1. Monitoring 55 | 1. Wrap-up 56 | 1. Chapter 7 - Data Relationships 57 | 1. Overview 58 | 1. Primary and Foreign Keys 59 | 1. Natural vs Surrogate 60 | 1. Database vs App Generated 61 | 1. Globally Unique Identifier vs Universally Unique Identifier 62 | 1. When to use each 63 | 1. Wrap-up 64 | 1. Chapter 8 - Protobuf 65 | 1. Introduction 66 | 1. Philosophy 67 | 1. Wrap-up 68 | 1. Chapter 9 - Active Remote Sandbox 69 | 1. Setup 70 | 1. Exposed ports 71 | 1. Testing 72 | 1. Monitoring 73 | 1. Wrap-up 74 | 1. Chapter 10 - Event Driven Messaging 75 | 1. Introduction 76 | 1. Wrap-up 77 | 1. Chapter 11 - Rabbit MQ 78 | 1. Introduction 79 | 1. Setup 80 | 1. Testing 81 | 1. Monitoring 82 | 1. Wrap-up 83 | 1. Chapter 12 - Event Driven Messaging Sandbox 84 | 1. Introduction 85 | 1. Philosophy 86 | 1. Setup 87 | 1. Testing 88 | 1. Wrap-up 89 | 1. Chapter 13 - Event Driven Sandbox 90 | 1. Introduction 91 | 1. Setup 92 | 1. Exposed ports 93 | 1. Testing 94 | 1. Monitoring 95 | 1. Wrap-up 96 | 1. Chapter 14 - Combining Active Remote and Action Subscriber 97 | 1. Introduction 98 | 1. Setup 99 | 1. Testing 100 | 1. Wrap-up 101 | 1. Chapter 15 - Wrap Up 102 | -------------------------------------------------------------------------------- /110-chapter-10.md: -------------------------------------------------------------------------------- 1 | ### Chapter 10 - Event Driven Messaging 2 | 3 | ## Introduction 4 | 5 | Up to this point we have built a platform that communicates between services in a synchronous manner. A messaging system sits between the apps, but the apps are issuing requests and expecting immediate responses. This pattern works great for retrieving and modifying data. What if you want your services to be notified when something happens in another service? Enter the event-driven architecture. 6 | 7 | An event-driven architecture can be demonstrated by the following example: Imagine you have two services, one is an human resources system and another is a payroll system. In a synchronous system, a new employee record could be created in the human resources record, and then in a synchronous workflow (a Active Model callback, a call from a service object, etc) a call could then be made to pass some info to the payroll system to create a new employee payroll record. One advantage of building software this way is that the process is synchronous, and errors can be reported immediately to the user. A disadvantage is that these two services are tightly coupled. If the payroll service was offline for some reason, the human resource service would appear to be having issues, and the user may perceive that the entire system is having issues and may report incorrect info when reporting the issue. 8 | 9 | There are a number of advantages to building on this architecture. One is that the services can be loosely coupled. If the services were built on an event-driven architecture and the human resource system was online, the user could add new employees and the payroll (and other services watching for the same employee created events) would perform their processing in the background. The perceived user experience would be better because the user is interacting with a small service that performs few tasks - because it has less responsibility, we might expect it to be a more responsive application. What is happening in the background is not the user's concern nor are they required to wait for all of the other services to complete processing before control is returned to the user. 10 | 11 | Another advantage to asynchronous processing on an event-driven platform is that additional services can be added to watch for the same events, such as the employee created event mentioned above. For example, if it is later decided that all new employees should receive a weekly dining gift card, a new service could be added to the platform that watches for the employee created event. Adding this new service to watch for an existing event would require zero configuration and zero downtime for existing services. 12 | 13 | ## Implementation 14 | 15 | The software that provides event-driven messaging is sometimes called a message broker. Some of the most popular brokers include, but are not limited to: Apache Kafka, RabbitMQ and Amazon Simple Queue Service (Amazon SQS). 16 | 17 | The gems we'll use to implement our event-driven architecture in the next couple of chapters are designed around RabbitMQ, so from here we'll focus on the features provided by the RabbitMQ message broker. 18 | 19 | ## Wrap-up 20 | 21 | Event driven messaging architectures provide numerous advantages. Some of these advantages are loose-coupling of services, zero downtime when adding new services and in some cases a better user experience. 22 | 23 | Earlier, we implemented the NATS service to route messages between our services. In the next chapter, we'll spin up a RabbitMQ service, subscribe to and publish events to see how a message broker works. 24 | 25 | [Next >>](120-chapter-11.md) 26 | -------------------------------------------------------------------------------- /040-chapter-03.md: -------------------------------------------------------------------------------- 1 | ### Chapter 3 - Ruby and Ruby on Rails 2 | 3 | > What if every creative idea that someone has is unconsciously borrowed from that person's experiences in another reality? Maybe all ideas are plagiarized without us knowing it, because they come to us through some cryptic and unprovable reality slippage? - Elan Mastai, All Our Wrong Todays 4 | 5 | ## Ruby 6 | 7 | The Ruby programming language was designed and developed by Yukihiro "Matz" Matsumoto in the mid 1990s. The language was relatively obscure until David Heinemeier Hansson published the Ruby on Rails framework in July of 2004. Ruby on Rails' popularity drove the development of the Ruby language. 8 | 9 | Ruby is a high-level language. It is high-level because it has a strong abstraction from the details of the computer hardware. The benefit is that the developer has more freedom to write code that the Ruby interpreter will interpret and in some cases optimize for the hardware. The downside is that the developer has more freedom to write code that may not be optimized for the hardware. 10 | 11 | Ruby is also an interpreted language. Interpreted languages (depending on the implementation) run on top of a virtual machine. The virtual machine layer runs on top of the native processor. In contrast, compiled languages are compiled to bytecode and run directly on the native processor. Because of the extra virtual machine layer, interpreted languages are generally slower. 12 | 13 | ## Ruby on Rails 14 | 15 | By the simple fact that you're reading this book, you're most likely familiar with the benefits of the Ruby language and the Ruby on Rails framework. If not, my take is that the Ruby language and the Ruby on Rails framework provide the tools a developer needs to be highly productive while building web applications. New applications can be spun up in a matter of seconds. There are a large number of libraries available (known in the Ruby world as gems), that can be used to extend your application's functionality. For example, if you need to run background processes, there's a gem for that: Sidekiq. If your app needs to manage money and currencies, there's a gem for that: Money. I could go on, but you get the point. 16 | 17 | For more information about why you should use the Rails framework, please review the official Ruby on Rails documentation at https://rubyonrails.org. 18 | 19 | ### Interpreters 20 | 21 | There are a few of Ruby interpreters available. We'll discuss a few of them below. 22 | 23 | #### MRI & YARV 24 | 25 | The Matz's Ruby Interpreter (MRI) was the default Ruby interpreter until Ruby 1.8. When Ruby 1.9 was released, Yet another Ruby VM (YARV) replaced MRI. YARV is the interpreter through Ruby 2+. YARV only supports green threads (which aren't recognized by the operating system and aren't scheduled as tasks among its cores). 26 | 27 | The good news is, there are other interpreters available that can help optimize your hardware for the custom apps you write. 28 | 29 | #### JRuby 30 | 31 | JRuby is an implementation of Ruby that compiles Ruby code to Java bytecode. Some of the benefits are true multithreading, the stability of the Java platform, the ability to call native Java classes, and in some cases, better performance. One of the downsides is increased memory consumption (this is Java, after all). 32 | 33 | ## Resources 34 | 35 | * https://rubygems.org/gems/money 36 | * https://rubygems.org/gems/sidekiq 37 | * https://rubyonrails.org 38 | * https://www.jruby.org 39 | * https://www.ruby-lang.org 40 | 41 | ## Wrap-up 42 | 43 | Ruby is a language that was designed with developer productivity in mind. Ruby on Rails is a framework that provides the toolset a developer needs to quickly spin up an application. There are a wide variety of gems (libraries) available to extend your application's features. 44 | 45 | In the next chapter, we'll discuss two popular gems, Active Record and Active Model. These gems are used to manage and persist the data in your application. 46 | 47 | [Next >>](050-chapter-04.md) 48 | -------------------------------------------------------------------------------- /050-chapter-04.md: -------------------------------------------------------------------------------- 1 | ### Chapter 4 - Active Record and Active Model 2 | 3 | > An object that wraps a row in a database table or view, encapsulates the database access, and adds domain logic on that data. - Martin Fowler, Patterns of Enterprise Architecture 4 | 5 | ## Introduction 6 | 7 | The model architectural pattern provides the serialization and deserialization of data for your application. 8 | 9 | #### Active Record 10 | 11 | The core of a Rails application is its data. Rails applications (as are many framework patterns) are built in multiple layers. Rails itself follows the model-view-controller (MVC) architecture pattern to separate programming logic into separate elements. Active Record is the model part of the MVC pattern in Rails. Active Record is where you add the behavior and persist the data that is required by your application. 12 | 13 | Using Active Record, whether stand-alone or in a Rails app, provides the following benefits: 14 | 15 | * A way to represent models and data 16 | * Associations between models 17 | * Hierarchies through related models 18 | * Data validation 19 | * Database access using objects 20 | 21 | #### Active Model 22 | 23 | Active Model is a library of modules that provides various methods to Active Record. Active Model's modules can also be included in your classes to provide the same functionality. Some of the included modules are below. 24 | 25 | - AttributeMethods - adds custom prefixes and suffixes on methods of a class 26 | - Callbacks - adds before, after and around methods 27 | - Conversion - adds `to_model`, `to_key` and `to_param` methods to an object 28 | - Dirty - adds methods to determine whether the object has been changed or not 29 | - Validations - adds validation methods to an object 30 | - Naming - adds class methods that provide singular and plural versions of the class name 31 | - Model - adds class methods for validations, translations, conversions, etc, and the ability to initialize an object with a hash of attributes 32 | - Serialization - adds functionality to make it easy to serialize and deserialize an object to and from a hash or JSON object 33 | - Translation - adds methods for internationalization using the i18n framework 34 | - Lint Tests - adds functionality to test whether your object is compliant with the Active Model API 35 | - SecurePassword - adds methods to store passwords or other data securely using the `bcrypt` gem 36 | 37 | #### What's the Difference? 38 | 39 | The most common way to persist and retrieve data in a Rails application is with Active Record. Active Record is a wrapper for database tables and their relationships. Active Resource is a wrapper for RESTful resources. 40 | 41 | Active Model is used by Active Record for data validation, serialization and a number of other features. 42 | 43 | Not all models need to be Active Record models that map to a database. We can add models to our app where some models are backed by a database, while another model could map to an API endpoint. This leads us to Active Remote - models that map to a remote service over a message bus. We'll discuss this in detail in the next chapter. 44 | 45 | ## Resources 46 | 47 | * https://guides.rubyonrails.org/active_record_basics.html 48 | * https://guides.rubyonrails.org/active_model_basics.html 49 | 50 | ## Wrap-up 51 | 52 | There are a variety of Ruby gems available that can help you build your application's model layer. If you follow some type of Model architectural pattern, you can create models that are backed by a RESTful resource, a database table, or something else. In the next chapter, we're going to discuss a new model type: Active Remote. Active Remote allows you to share data between applications efficiently. 53 | 54 | In the next chapter, we'll discuss Active Remote, an alternative to Active Record which provides a model for our application, whose data is retrieved from another service. It allows us to continue to utilize the Rails MVC pattern, but retrieve and manage data owned by another service. 55 | 56 | [Next >>](060-chapter-05.md) 57 | -------------------------------------------------------------------------------- /060-chapter-05.md: -------------------------------------------------------------------------------- 1 | ### Chapter 5 - Active Remote 2 | 3 | > ...Senator Amidala, the former Queen of Naboo, is returning to the Galactic Senate to vote on the critical issue of creating an ARMY OF THE REPUBLIC to assist the overwhelmed Jedi.... - Star Wars: Episode II: Attack of the Clones opening crawl 4 | 5 | ## Introduction 6 | 7 | Active Remote is a Ruby gem that can replace Active Record models in your application to provide access to other models that exist in other applications on your network. Similar in philosophy to Active Resource, Active Remote provides data access for remote resources. 8 | 9 | The difference is that while Active Resource provides access to RESTful resources, Active Remote provides access using more durable and efficient methods (e.g. using a message bus for durability, and Protobuf for efficient data serialization and deserialization). 10 | 11 | ## Philosophy 12 | 13 | Active Remote attempts to provide a solution for accessing and managing distributed resources by providing a model which can be implemented with a minimal amount of code. Whether the model's data is persisted locally or somewhere else is of no concern to the rest of the application. 14 | 15 | Further, because Active Remote implements a pub-sub messaging system, clients do not need to be configured with details about which servers own and respond to specific resources. To publish, clients need only specify the message broker and the subject. Some other server will respond to their requests. 16 | 17 | ## Design 18 | 19 | During application initialization, Active Record models read the database schema and generate all of the getters, setters and methods which reduces the amount of boilerplate code that needs to be added to your models which inherit from ActiveRecord::Base. Because Active Remote doesn't have direct access to the database, on the client side, you'll need to declare the Active Remote model's attributes using the `attribute` method. On the server side, where you want to share the Active Record data, you'll need to create a Service class for each model that will define endpoints to allow for searching, creating, updating, deleting, etc. 20 | 21 | ## Implementation 22 | 23 | Active Remote is packaged as a Ruby gem. The Active Remote gem provides a DSL (domain-specific language), handles primary key guid fields, handles serialization, among a number of other features. The Active Remote gem depends on the Protobuf gem, so that gem will get installed automatically when you install or include the Active Remote gem. 24 | 25 | To share data between services, you'll need to include the Protobuf NATS gem. For the client Rails app, the Active Remote and Protobuf NATS are the two gems you'll need to include with your application. In the server Rails app, you'll want to include the Active Remote, Protobuf NATS and the Protobuf Active Record gems. The Protobuf Active Record gem glues together Protobuf and Active Record, providing features such as linking your Protobuf messages to your Active Remote classes. 26 | 27 | ## Resources 28 | 29 | * https://github.com/mxenabled/protobuf-nats 30 | * https://github.com/liveh2o/active_remote 31 | * https://github.com/liveh2o/protobuf-activerecord 32 | * https://github.com/rails/activeresource 33 | * https://github.com/ruby-protobuf/protobuf 34 | 35 | ## Wrap-up 36 | 37 | Active Remote allows you to build a durable and efficient communication platform between microservices. It also allows you to follow established architectural patterns such as MVC. 38 | 39 | Because Active Remote implements a message bus to communicate between services, it gives your services durability. As long as the message bus service remains online, your Rails apps can send a message to another service, and eventually get a reply when the other service comes back online. 40 | 41 | Active Remote also implements Protobuf, an efficient serialization and deserialization. As your platform grows, minimizing the amount of data traveling over the wire will pay dividends as you continue to scale your platform. 42 | 43 | In the next chapter, we'll discuss messaging queues. We'll spin up a NATS server and send and receive simple messages via the telnet protocol. 44 | 45 | [Next >>](070-chapter-06.md) 46 | -------------------------------------------------------------------------------- /020-chapter-01.md: -------------------------------------------------------------------------------- 1 | ### Chapter 1 - Microservices 2 | 3 | > I need you to be clever, Bean. I need you to think of solutions to problems we haven't seen yet. I want you to try things that no one has ever tried because they're absolutely stupid. - Orson Scott Card, Ender's Game 4 | 5 | ## Service Architecture 6 | 7 | First, a few definitions: 8 | 9 | * **API:** Application programming interface - in the context of this book, an application that provides access to its data. An API provides application-to-application communication. 10 | * **Function as a service (FaaS):** A small unit of code that can be published and consumed without also building and maintaining the surrounding infrastructure. Considered one way to build a "serverless" architecture. 11 | * **Microservice:** An application that provides specific functionality and data. 12 | * **Monolith:** A single application that provides all or a large portion of the functionality your company requires. 13 | * **Serverless computing:** The cloud provider provisions and can dynamically manage the server resources as needed. Small units of code, such as function as a service are deployed to a serverless environment. 14 | * **Services:** A service, loosely defined, is a stand-alone application that provides some piece of functionality. Examples include a website, an API, a database server, etc. 15 | 16 | When you're first designing a service, the requirements usually start out small. As time goes on, you and your team add features and the application grows organically into a larger system. We call this a monolith. There is nothing wrong with monoliths, as long as they can handle the processing load. Monoliths are the simplest architecture because they are easy to maintain by small teams, all of the code is in one place, and communication between modules is instantaneous (no network overhead). 17 | 18 | ## Tell me more about microservices 19 | 20 | A microservice is a small app that provides a limited set of functionality. In the UNIX philosophy (as documented by Doug McIlroy), one tenet is to make each program do one thing and do it well. In terms of a microservice architecture, your goal is to build small programs or services that provide a specific set of features. As this feature set becomes more wide used in your organization, you can scale up and out that particular feature to keep up with the needs of the business. 21 | 22 | ## Why should I use microservices? 23 | 24 | By nature, building a microservices architecture incurs additional overhead that may not be worth the time in the early stages of a project. Once the app becomes popular and you can identify bottlenecks in the process, then it may be time to identify and carve out specific functionality into its own service. 25 | 26 | As your development team grows, you may want to split the code base into smaller units that can be maintained and deployed separately. This can have other benefits, too. These benefits include shorter development cycles, test and release cycles. Less code means less regression testing for the quality assurance team. 27 | 28 | If you have different uptime requirements for functionality of your application, your codebase may be a candidate for breaking it up into smaller microservices. This will allow you to meet your service level requirements on a service by service basis. The added benefit is that the services can be designed so that if a less critical piece of the platform goes down, the rest of the platform remains unaffected. 29 | 30 | ## Resources 31 | 32 | * https://en.wikipedia.org/wiki/Unix_philosophy 33 | * https://martinfowler.com/articles/microservices.html 34 | 35 | ## Wrap-up 36 | 37 | There are many ways to provide the logic that meets your business requirements. Most business applications start small, but grow to meet the needs of your business. Most of the time, business applications grow into a large, monolithic app. While there are many reasons to keep the monolith, as your business and team grow, you may want to split your codebase into smaller, easier to maintain units. 38 | 39 | In the next chapter, we'll discuss communications between microservices. We'll cover the pros and cons of various protocols, and some of the serialization methods that can be used to encode our data. 40 | 41 | [Next >>](030-chapter-02.md) 42 | -------------------------------------------------------------------------------- /120-chapter-11.md: -------------------------------------------------------------------------------- 1 | ### Chapter 11 - Messaging Systems - Rabbit MQ 2 | 3 | ## Introduction 4 | 5 | RabbitMQ is one of many implementations of message brokers. It provides many features, such as message queuing, support for multiple messaging protocols and delivery acknowledgement. We'll discuss some of these features in this chapter. 6 | 7 | We're going to run RabbitMQ the same way we ran NATS in chapter 6. Docker makes it easy. We'll follow most of the same steps but with a slightly different docker-compose.yml file. 8 | 9 | ## Let's Run It 10 | 11 | Let's run a local RabbitMQ server and send messages to it. We'll include a Ruby image so we can build a Ruby client to send messages to and receive messages from queues on the RabbitMQ server. With NATS we used telnet to send and receive simple test-based messages - with RabbitMQ we'll need to construct binary messages that we'll send to RabbitMQ. 12 | 13 | _**Listing 11-1**_ Docker Compose file with RabbitMQ and Ruby 14 | 15 | ```yml 16 | # ~/projects/rabbitmq/docker-compose.yml 17 | # usage: docker-compose up 18 | 19 | version: "3.4" 20 | 21 | services: 22 | rabbit: 23 | image: rabbitmq:latest 24 | ports: 25 | - 5672:5672 26 | stdin_open: true 27 | ruby: 28 | image: ruby:2.6.5 29 | stdin_open: true 30 | ``` 31 | 32 | Save the file with the filename `docker-compose.yml`. Let's now switch to that directory and run the containers. The versions of the software and output may differ from what you see in your terminal. 33 | 34 | _**Listing 11-2**_ Start RabbitMQ and Ruby 35 | 36 | ```console 37 | $ cd ~/projects/rabbitmq 38 | $ docker-compose up 39 | Starting rabbitmq_ruby_1 ... done 40 | Creating rabbitmq_rabbit_1 ... done 41 | Attaching to rabbitmq_ruby_1, rabbitmq_rabbit_1 42 | ruby_1 | Switch to inspect mode. 43 | rabbit_1 | Starting RabbitMQ 3.8.2 on Erlang 22.2.3 44 | ... 45 | rabbit_1 | ## ## RabbitMQ 3.8.2 46 | rabbit_1 | ## ## 47 | rabbit_1 | ########## Copyright (c) 2007-2019 Pivotal Software, Inc. 48 | rabbit_1 | ###### ## 49 | rabbit_1 | ########## Licensed under the MPL 1.1. Website: https://rabbitmq.com 50 | ... 51 | rabbit_1 | Starting broker...2020-01-25 14:18:45.535 [info] <0.267.0> 52 | ... 53 | rabbit_1 | 2020-01-25 14:18:46.099 [info] <0.8.0> Server startup complete; 0 plugins started. 54 | ``` 55 | 56 | Creating a producer is simple. We'll connect to the Ruby container and run IRB to create a connection, a channel and a queue. Let's create a message in a new terminal window. 57 | 58 | _**Listing 11-3**_ Creating a message 59 | 60 | ```console 61 | $ docker-compose exec ruby bash 62 | /# gem install bunny # install the bunny gem 63 | ... 64 | /# irb 65 | irb(main):001:0> require 'bunny' 66 | irb(main):002:0> connection = Bunny.new(hostname: 'rabbit') # the 'rabbit' hostname was defined as the service name in the docker-compose.yml file 67 | irb(main):003:0> connection.start 68 | irb(main):004:0> channel = connection.create_channel # create a new channel 69 | irb(main):005:0> queue = channel.queue('hello') # create a queue 70 | irb(main):006:0> channel.default_exchange.publish('Hello World', routing_key: queue.name) # encode a string to a byte array and publish the message to the 'hello' queue 71 | ``` 72 | 73 | Creating a consumer is just about as simple as creating a producer. We'll create a connection, a channel, and a queue. The new queue command will subscribe to queues with the same name, or if it doesn't exist, create a new queue. 74 | 75 | Let's open another terminal window and run the commands in listing 11-4. 76 | 77 | _**Listing 11-4**_ Consuming messages 78 | 79 | ```console 80 | $ docker-compose exec ruby bash 81 | /# irb 82 | irb(main):001:0> require 'bunny' 83 | irb(main):002:0> connection = Bunny.new(hostname: 'rabbit') # the 'rabbit' hostname was defined as the service name in the docker-compose.yml file 84 | irb(main):003:0> connection.start 85 | irb(main):004:0> channel = connection.create_channel # create a new channel 86 | irb(main):005:0> queue = channel.queue('hello') # create a queue (or connect to an existing queue if it already exists) 87 | irb(main):006:0> queue.subscribe(block: true) do |_delivery_info, _properties, body| 88 | irb(main):007:1* puts "- Received #{body}" 89 | irb(main):008:0> end 90 | - Received Hello World 91 | ``` 92 | 93 | You should see the `- Received Hello World` message in the terminal window where we consumed the message (Listing 11-4). This demonstrates that we have RabbitMQ server running, we published a message to a queue, and our consumer received the message. Let's switch back to the IRB session we started in listing 11-3 and publish another message. 94 | 95 | _**Listing 11-5**_ Publish a second message 96 | 97 | ```console 98 | irb(main):007:0> channel.default_exchange.publish('We thought you were a toad!', routing_key: queue.name) # encode and publish another message 99 | irb(main):008:0> connection.close # the message has been sent, so let's close the connection 100 | ``` 101 | 102 | If you switch back to the terminal in Listing 11-4 where we created a consumer, we should see both the '- Received Hello World' and the '- Received We thought you were a toad!' messages. Congratulations! You've successfully started a RabbitMQ server and a couple of Ruby clients to publish and consume messages. 103 | 104 | When you're done, you can press `Ctrl-C` to exit the RabbitMQ consumer terminal, and then `Ctrl-D` or type `exit` to return to the host machine's command prompt. 105 | 106 | ## Resources 107 | 108 | * https://www.rabbitmq.com/tutorials 109 | 110 | ## Wrap-up 111 | 112 | Event-driven messaging systems are a layer in a system architecture that allows you to build a platform that is asynchronous, reliable, decoupled and scalable. Along with NATS, RabbitMQ is one such messaging system that is simple to configure and use. 113 | 114 | In this chapter, we successfully spun up a local RabbitMQ server, created a queue, then published and consumed to messages on that queue. 115 | 116 | In the next chapter, we'll discuss a couple of Ruby gems that make it easy to configure and set up our RabbitMQ clients. 117 | 118 | [Next >>](130-chapter-12.md) 119 | -------------------------------------------------------------------------------- /080-chapter-07.md: -------------------------------------------------------------------------------- 1 | ### Chapter 7 - Data Relationships 2 | 3 | ## Introduction 4 | 5 | Your application will most likely be designed on top of one or more database tables, whether they are stored in a relational database or a NoSQL database. As the application grows, you'll define relationships between entities that will need to be defined both in the application and in the database. 6 | 7 | ## Primary and Foreign Keys 8 | 9 | Primary keys are attributes used to identify the uniqueness of a row in a relation (also known as a table in database terminology). Foreign keys are used to associate child or dependent records to another relation. 10 | 11 | ## Natural vs Surrogate Keys 12 | 13 | The rows in most of your relations should have a unique identifier. A unique identifier could be a single attribute, or two or more attributes. 14 | 15 | A natural key is defined as one or more attributes which uniquely identify a row. For example, in a relation where you store your employee's information, you could use their first and last name attributes as a natural key. But what happens when your company hires another person with the same first and last name? When that time comes, you'll have wished you picked a better natural key, like their Social Security number. You could use the Social Security number attribute as the natural key. Social Security numbers are unique and are assigned to a person for their entire life. Because of this, it makes a great natural key because it uniquely identifies an employee, but doesn't change like their name could. 16 | 17 | > Note: I'm not advocating that you store Social Security numbers in your database, but this serves as a great example of a unique value that comes from an external source. Another reason not to depend on Social Security numbers as unique identifiers is that employees with green cards, etc won't have a number assigned by the Social Security Administration. 18 | 19 | Surrogate keys are auto generated by your database system. Most surrogate keys are an integer type that usually starts at 1 and increases (+1) for each new row that is inserted. 20 | 21 | In theory, natural keys are preferred to surrogate keys because they are composed of the data that is stored in the database. In practice, however, surrogate keys are generally easier to use and can usually be optimized for better application and database performance. 22 | 23 | ## Database vs App Generated 24 | 25 | Surrogate keys can be generated by either the database or the application. Keys that are integer types are most commonly generated by the database, and database concurrency ensures that each new record gets a unique value in it's primary key column, whether you have one or twenty application instances inserting records into the same table. 26 | 27 | ### Database Generated Integers 28 | 29 | The most common surrogate keys generated by a database system are integer values. When designing your database, you'll most likely need to add integer-type surrogate keys so that the data can be managed by Active Record. When designing your tables, keep in mind that you should build your data structures for the long term. By default, Rails scaffolding generates migrations that use 32 bit integer data types. For most tables, especially lookup or domain tables, this integer type is perfectly fine. For some of your tables, after a while (it could be months or years later) your application could come to a screeching halt when the 32 bit integer primary key hits its maximum value. 30 | 31 | When you're designing your database, if you can identify which tables could continue to grow in size over time, it would be better to start with a 64 bit integer - bigint (MySQL) or bigserial (PostgreSQL) data types. If you do use a 64 bit integer for some of your tables, remember to match the type for foreign keys. 32 | 33 | ### Unique Identifiers 34 | 35 | One way to generate surrogate keys which also keeps the surrogate keys unique is to generate unique identifiers in the application. These unique identifiers can be shared with other services. Your design could allow the database to also generate an incrementing primary key, but those values should never be shared with systems outside of the application. 36 | 37 | Unique identifiers are sometimes called globally unique identifiers (GUIDs), or universally unique identifiers (UUIDs). UUIDs are a subset of GUIDs. GUIDs can be generated by the application before the record is persisted to the database. For all practical purposes, each GUID is unique. An example GUID is `83efd88b-ec78-4e84-b8c7-dad8421d42d4`. GUIDs are usually represented by hex digits separated by dashes. These values can be shared as a unique identifier for a specific database row or object between applications. They can also be used as foreign keys to map parent and child objects. 38 | 39 | ## When to use each 40 | 41 | If your application is monolithic (a single application running on top of a single database), there is usually no need to use anything other than the database's auto-incrementing primary key. By default, Rails applications handle the database's surrogate key implementation quite well. 42 | 43 | It's at the point when you decide to split your database and share data between systems that you'll need to implement some type of unique identifier that can be shared between systems. Once you reach this point, your data no longer lives in a single database or application, it is sent and shared across the network. Also, if you implement both a database-generated integer primary key and a UUID key as your microservice platform grows, the database-generated integer value has no meaning outside of its own database and application. There is no need to share the integer value, only the UUID. 44 | 45 | ## Resources 46 | 47 | * https://en.wikipedia.org/wiki/Database_normalization 48 | * https://en.wikipedia.org/wiki/Universally_unique_identifier 49 | 50 | ## Wrap-up 51 | 52 | Database normalization requires you to identify a primary key. This key could be a natural key (one or more columns that define the data), or a surrogate key that is generated by the application or the database. Rails is designed to use database-generated primary keys out of the box. As your infrastructure grows and as you begin to share data with other systems, using a UUID is an option that makes your data portable and uniquely identifiable. 53 | 54 | In the next chapter, we'll discuss a way to serialize the data that will be shared between our microservices. One way to efficiently serialize the data via Protocol Buffers (aka Protobuf). 55 | 56 | [Next >>](090-chapter-08.md) 57 | -------------------------------------------------------------------------------- /070-chapter-06.md: -------------------------------------------------------------------------------- 1 | ### Chapter 6 - Messaging Systems - NATS 2 | 3 | > There are around seven octillion atoms in a human body. That's a lot of goddamn atoms to disassemble, shoot back through time and space, and reassemble in perfect order. - Elan Mastai, All Our Wrong Todays 4 | 5 | ## Introduction 6 | 7 | Messaging systems are a critical component when designing distributed systems. They are used to provide a layer in your platform architecture which is used to shuffle messages between services. A message layer provides a endpoint for services to communicate. Each service only needs to know how to communicate with the message queue, which queues to subscribe to, and which queues to listen on. 8 | 9 | NATS is one such messaging system that provides security, resiliency, is scalable and can meet the performance requirements of most platforms. As of the time of this writing, NATS has clients written in over 30 programming languages. 10 | 11 | In this chapter, we'll spin up a NATS server. We'll test it by publishing and subscribing to messages using a telnet client. 12 | 13 | ## Let's Run It 14 | 15 | Let's use Docker to run a local NATS server and send messages to it. We'll include a BusyBox image so we can run telnet commands to test NATS. 16 | 17 | **Listing 6-1** Docker compose with NATS and BusyBox 18 | 19 | ```yml 20 | # ~/projects/nats/docker-compose.yml 21 | # usage: docker-compose up 22 | 23 | version: "3.4" 24 | 25 | services: 26 | nats: 27 | image: nats:latest 28 | ports: 29 | - 4222:4222 30 | - 8222:8222 31 | stdin_open: true 32 | busybox: 33 | image: busybox:latest 34 | stdin_open: true 35 | ``` 36 | 37 | Save the file with the filename `docker-compose.yml`. Let's now switch to that directory and run the containers. The versions of the software and output may differ from what you see in your terminal. 38 | 39 | **Listing 6-2** Start NATS and Busybox 40 | 41 | ```console 42 | $ cd ~/projects/nats 43 | $ docker-compose up 44 | Starting nats_nats_1 ... done 45 | Starting nats_busybox_1 ... done 46 | Attaching to nats_busybox_1, nats_nats_1 47 | nats_1 | [1] 2019/10/07 13:53:36.029873 [INF] Starting nats-server version 2.0.2 48 | ... 49 | nats_1 | [1] 2019/10/07 13:53:36.032328 [INF] Listening for client connections on 0.0.0.0:4222 50 | ... 51 | nats_1 | [1] 2019/10/07 13:53:36.033766 [INF] Server is ready 52 | ``` 53 | 54 | Creating a subscriber is simple. We'll open a NATS session with telnet. Telnet is a client application that will allow us to issue text-based commands to NATS. We'll provide a subject (in example 6-3 we'll create a subject named 'messages') and also provide a _subscription identifier_. The subscription identifier can be a number or a string. We'll use the keyword `sub` to create and subscribe to a subject. Docker Compose provides a convenient `exec` command to connect and ssh into to a running container. We'll use the `exec` command to log into the running BusyBox container and subscribe via telnet. 55 | 56 | **Listing 6-3** Subscribing to a Subject 57 | 58 | ```console 59 | $ docker-compose exec busybox sh 60 | / # telnet nats 4222 # you'll need to type this line 61 | ... 62 | sub messages 1 # and this line 63 | +OK # this is the acknowledgement from NATS 64 | ``` 65 | 66 | Let's open a new terminal and create a publisher. The publishing client will need to provide the name of the subject it wishes to publish the message on. Along with the subject, the client will also provide the number of bytes that will be published. If the number of bytes is missing or incorrect, the publisher is not following the NATS protocol and the message will be rejected. 67 | 68 | Let's run a telnet command to publish messages to NATS. 69 | 70 | **Listing 6-4** Publishing to a Subject 71 | 72 | ```console 73 | $ docker-compose exec busybox sh 74 | / # telnet nats 4222 # you'll need to type this line 75 | ... 76 | pub messages 12 # and this line 77 | Hello WORLD! # and this line 78 | +OK 79 | ``` 80 | 81 | You should see the `Hello WORLD!` message in the terminal window where we subscribed to the subject (Listing 6-3). This demonstrates that we have NATS server running, we published a message to a subject, and our subscriber received the message. You can press `Ctrl-C` and then the letter `e` to exit the telnet session, and then `Ctrl-D` or type `exit` to return to the host machine's command prompt. 82 | 83 | NATS also provides a monitoring API which we can query to keep tabs on how many messages are sent through the server, etc. Because we're exposing NATS port 8222 outside the Docker environment (see the `docker-compose.yml` file in Listing 6-2), we can view the instrumentation by opening the browser on our host machine at the following address: [http://localhost:8222](http://localhost:8222). A page should render in your browser, with a handful of links. If we were to set up a cluster of NATS servers, additional links would appear. 84 | 85 | As of the time of this writing, there are 5 links on the page. Let's briefly look at each of them: 86 | 87 | * [varz](http://localhost:8222/varz) - General information about the server state and configuration 88 | * [connz](http://localhost:8222/connz) - More detailed information on current and recently closed connections 89 | * [routez](http://localhost:8222/routez) - Information on active routes for a cluster 90 | * [subsz](http://localhost:8222/subsz) - Detailed information about the current subscriptions and the routing data structure 91 | * [help](https://docs.nats.io/nats-server/configuration/monitoring) - A link to the NATS documentation 92 | 93 | Some of the endpoints above also have querystring parameters that can be passed, e.g. http://localhost:8222/connz?sort=start, which will sort the connections by the start time. Check out the NATS documentation for more information about these endpoints and their options. 94 | 95 | ## Resources 96 | 97 | * https://docs.docker.com/compose 98 | * https://hub.docker.com/_/nats 99 | * https://nats.io 100 | * https://docs.nats.io/nats-server/configuration/monitoring 101 | 102 | ## Wrap-up 103 | 104 | Messaging systems are a layer in a system architecture that allows you to build a platform that is asynchronous, reliable, decoupled and scalable. NATS is one messaging system that is simple to configure and use. 105 | 106 | In this chapter, we successfully spun up a local NATS server, created a subject, then published and subscribed to messages to that subject. We also learned about the instrumentation that NATS provides. 107 | 108 | In the next chapter, we'll discuss structured data and what types of keys to use to share data between systems. In chapter 9, we'll set up a microservice environment consisting of two Rails applications that will use NATS to share data. 109 | 110 | [Next >>](080-chapter-07.md) 111 | -------------------------------------------------------------------------------- /030-chapter-02.md: -------------------------------------------------------------------------------- 1 | ### Chapter 2 - Service Communications 2 | 3 | ## Introduction 4 | 5 | As your service infrastructure grows, you'll need to find a communication protocol that is a good balance of development, maintenance, speed and resilency. 6 | 7 | ## Protocols 8 | 9 | Various protocols can be used to move data between services. Each has its advantages and disadvantages. For example, HTTP is one of the most widely used protocols for web pages and RESTful APIs. HTTP provides many useful features such as authentication, but also sends header data with each request. Sending header data with each request could cause undesired network congestion when we're designing a platform that, in order to scale, requires each message to be a small as possible. 10 | 11 | _**Table 2-1**_ Network protocols 12 | 13 | | Protocol | Advantages | Disadvantages | Example uses | 14 | |---|---|---|---| 15 | | AMQP | A binary format that provides queuing, routing, reliability | Binary only | Passing messages to and from RabbitMQ | 16 | | HTTP(S) | Runs on top of TCP, provides request methods, authentication and persistent connections | Some processing overhead is required to provide some of its features, headers are also sent over the wire with each request | World Wide Web, Email, RESTful APIs | 17 | | NATS | Text-based, so clients are available for a wide variety of programming languages | Only used to connect to a NATS server | Publishing to or listening on queues on a NATS server | 18 | | TCP | One of the most popular protocols on the Internet, used for establishing the connection between server and client, guarantees that the data was delivered to the client, provides error checking and resends lost packets | Slower than other protocols such as UDP | SSH, World Wide Web, Email | 19 | | UDP | Its connection-less design is for speed and efficiency | Does not provide error checking or any guarantees that the client received the data | Video streaming, DNS | 20 | 21 | ## Data Serialization 22 | 23 | The data that is sent over the wire needs to be packaged for delivery. A few ways to package this data are in the table below: 24 | 25 | _**Table 2-2**_ Data serialization formats 26 | 27 | | Format | Text/Binary | Advantages | Disadvantages | 28 | |---|---|---|---| 29 | | JSON | Text | Structured, human readable | Keys are present in each object which inflates the message size | 30 | | Protocol Buffers (Protobuf) | Binary | Small footprint | Both client and server need to know the structure of the encoded message | 31 | | XML | Text | Structured, human readable | Opening and closing tags around each field which inflates the message size | 32 | 33 | ### Examples 34 | 35 | Here are examples of serialized data in each format. 36 | 37 | #### JSON 38 | 39 | JSON (JavaScript Object Notation) is a human-readable, text-based format. Its structure consists of name-value pairs. Because of its simple structure, it has become a popular option for sharing data between services. 40 | 41 | ```json 42 | [ 43 | { 44 | "id": 1, 45 | "first_name": "George", 46 | "last_name": "Costanza" 47 | }, 48 | { 49 | "id": 2, 50 | "first_name": "Elaine", 51 | "last_name": "Benes" 52 | } 53 | ] 54 | ``` 55 | 56 | #### Protocol Buffers 57 | 58 | Protocol Buffers (Profobuf) are a language-independent format that is used to generate language-specific code to produce very small messages that are sent over the network. The advantage is network efficiency, the disadvantage is both the sender and receiver need to agree to the message structure in advance. 59 | 60 | Other formats such as JSON use name/value pairs to describe each piece of data. Protobuf uses a field position to define the fields as they are encoded and decoded from a binary format. 61 | 62 | The Person message 63 | 64 | ```protobuf 65 | message PersonMessage { 66 | int32 id = 1; 67 | string first_name = 2; 68 | string last_name = 3; 69 | } 70 | ``` 71 | 72 | A list of people in a single message 73 | 74 | ```protobuf 75 | message PeopleMessageList { 76 | repeated PersonMessage records = 1; 77 | } 78 | ``` 79 | 80 | ##### Ruby Implementation 81 | 82 | The Person class 83 | 84 | ```ruby 85 | class PersonMessage < ::Protobuf::Message 86 | optional :int32, :id, 1 87 | optional :string, :first_name, 2 88 | optional :string, :last_name, 3 89 | end 90 | ``` 91 | 92 | A class to hold a list of people 93 | 94 | ```ruby 95 | class PeopleMessageList < ::Protobuf::Message 96 | repeated ::PersonMessage, :records, 1 97 | end 98 | ``` 99 | 100 | *Serialized Data* 101 | 102 | The data below is a string representation of the binary encoding. 103 | 104 | ```console 105 | # Person 1 106 | \b\x01\x12\x06George\x1A\bCostanza 107 | 108 | # Person 2 109 | \b\x02\x12\x06Elaine\x1A\x05Benes 110 | 111 | # Both 112 | \n\x14\b\x01\x12\x06George\x1A\bCostanza\n\x11\b\x02\x12\x06Elaine\x1A\x05Benes 113 | ``` 114 | 115 | #### XML 116 | 117 | XML (Extensible Markup Language) is a human-readable, text-based format. Like JSON, XML defines both the structure and the data in the same message body. XML is a popular choice for exchanging data over the Internet and between systems. 118 | 119 | ```xml 120 | 121 | 122 | 1 123 | George 124 | Costanza 125 | 126 | 127 | 2 128 | Elaine 129 | Benes 130 | 131 | 132 | ``` 133 | 134 | ## Messaging Systems 135 | 136 | So far we've described protocols for transfering and packaging our data. Now let's discuss established architectures that we can use to communicate between systems. Microservices are small, independent applications that can reach out to other applications to perform some work. 137 | 138 | Services can directly or indirectly call other services. When a service directly calls another service, the caller expects the other service to be available at the endpoint that the caller already knows about. For example, if an API hosts an `HTTP` endpoint at `http://humanresources.internal/employees`, I may write a service that acts as a client that can call that endpoint. I would expect to receive a list of employees, encoded in some format, such as `JSON`. 139 | 140 | Indirectly calling a service means that there is some system between the two services. Examples of systems that acts as an intermediary include a proxy server which may return a cached copy of the data or a message queue server which can queue up requests and smooth out the workload of the server that provides the data. 141 | 142 | Services can also make request-reply or fire-and-forget requests. The request-reply architecture pattern is used when the service sends a request to another service and expects a response. The fire-and-forget pattern is used to send a message to an intermediary service which then notifies all interested services. The sender of the fire-and-forget message doesn't wait for any responses, only that the message was sent. 143 | 144 | Our business needs will drive the microservice architectural patterns that we ultimately implement. These patterns are not mutually exclusive per service. These patterns can be combined in a single service, as we'll show in chapter 13. 145 | 146 | ## References 147 | 148 | * https://www.amqp.org 149 | * https://developers.google.com/protocol-buffers 150 | * https://docs.nats.io/nats-protocol/nats-protocol 151 | * https://www.json.org 152 | * https://www.w3.org/XML 153 | 154 | ## Wrap-up 155 | 156 | In this chapter, we discussed various protocols and message formats that can be used to share data between services. In the next chapter, we'll cover the Ruby language and the Ruby on Rails framework. 157 | 158 | [Next >>](040-chapter-03.md) 159 | -------------------------------------------------------------------------------- /090-chapter-08.md: -------------------------------------------------------------------------------- 1 | ### Chapter 8 - Protocol Buffers (Protobuf) 2 | 3 | > Humans had developed a sequential mode of awareness, while heptapods had developed a simultaneous mode of awareness. We experienced events in an order, and perceived their relationship as cause and effect. They experienced all events at once, and perceived a purpose underlying them all. A minimizing, maximizing purpose. - Ted Chiang, Stories of Your Life and Others 4 | 5 | ## Introduction 6 | 7 | Why did you build your app? Most likely the reason was that you needed to track some data. You or someone at your company may have started with a spreadsheet, but over time, realized that tracking the data in a spreadsheet became cumbersome and no longer met your needs. So, an app was born. Your fresh out-of-the-box app then started to grow, with relationships between data entities. As the amount of data and the number of relationships and the processing requirements grew, you decided that you needed to split your app into separate services. When a piece of data needs to be shared between applications, we need to make a couple of decisions. What attributes will be shared? How will clients access the data? Which app will own and persist the data? How can we make it easy to extend or add new attributes to our entities that are still backwards compatible? 8 | 9 | When you build a microservice platform, you need to make several decisions, one of them being how do we share data between services. As discussed in chapter 2, protocol buffers (aka protobuf) is one of those options. 10 | 11 | Developed by Google, protobuf is a method of serializing data in a binary format that allows for performance over flexibility. Protobuf has a standard definition structure that is used to define messages. Compilers are available to convert the definitions to classes or structures that are language specific. For example, the same definition file can be used to generate classes or structures for both Java and Ruby, so that apps written in both languages can share the same message. 12 | 13 | ## Philosophy 14 | 15 | Protobuf serializes data to a binary format that is not self-describing. In contrast, a JSON or XML object is usually human readable and human editable, and each object can be inspected and the developer can view the field names and their values. Protobuf is a binary format. It implements an ordered field format and both of the services sharing the message need to know the structure of the message. 16 | 17 | There are many ways to encode and share data. XML, JSON and other formats are well-defined and easy to generate and consume. But what if you want better encoding and decoding performance? What if you want to reduce the network load of messages being passed between systems? What if your platform, the number of developers, and the number of messages passed between systems grows overnight? 18 | 19 | Protobuf attempts to solve these and other problems by encoding data to a binary format (which of course is much smaller than a XML or JSON encoded object). Protobuf definitions consist of one or more uniquely numbered fields. Each encoded field is assigned a field number and a value. This field number is what differentiates Protobuf from other objects. The field number is used to encode and decode the message attributes, reduces the amount of data that needs to be encoded by leaving out the attribute name, and allows for extendability so developers can add fields to a definition without having to upgrade all of the apps that consume that message at the same time. This is possible because existing apps will ignore new fields on any messages they receive and decode. 20 | 21 | ## Implementation 22 | 23 | An example Protobuf definition is below. Protobuf files have the file extension `.proto`. 24 | 25 | **Listing 8-1** Employee protobuf message 26 | 27 | ```proto 28 | // file employee.proto 29 | 1 syntax = "proto3"; 30 | 2 31 | 3 message Employee { 32 | 4 string guid = 1; 33 | 5 string first_name = 2; 34 | 6 string last_name = 3; 35 | 7 } 36 | ``` 37 | 38 | Let's inspect each line. 39 | 40 | Line 1 defines the version of the Protobuf syntax we'd like to use. 41 | 42 | Line 3 is the beginning of our message declaration. 43 | 44 | Lines 4-6 are the field definitions. Each line has a type (the guid field is a string type). Line 4 has an attribute name of `guid`, and the field number of 1. 45 | 46 | This Protobuf definition is by itself not used in your application. What we do next is compile this `employee.proto` field to a class or structure file in the same language your app is written in, whether that's Java, C#, Go or Ruby. If you support a heterogeneous platform with multiple languages, you may want to build scripts which will automatically compile your `.proto` files to the required languages each time you add a new `.proto` file or add a new field to one of your existing definitions. 47 | 48 | We briefly covered the [Ruby implementation in chapter 2](https://github.com/kevinwatson/rails-microservices-book/blob/master/030-chapter-02.md#protocol-buffers), but let's review and go into more detail here. 49 | 50 | The example below is the output for the Ruby implementation (additional setup and details can be found in [chapter 9](https://github.com/kevinwatson/rails-microservices-book/blob/master/100-chapter-09.md)). After defining the `.proto` definition file and running the `rake protobuf:compile` command, we will now have files similar to the following: 51 | 52 | **Listing 8-2** Employee Ruby protobuf class 53 | 54 | ```ruby 55 | # file employee.pb.rb 56 | class Employee < ::Protobuf::Message 57 | optional :string, :guid, 1 58 | optional :string, :first_name, 2 59 | optional :string, :last_name, 3 60 | end 61 | ``` 62 | 63 | **Serialized Data** 64 | 65 | Note that the data below is a string representation of the binary encoding. 66 | 67 | **Listing 8-3** Employee protobuf encoding 68 | 69 | ```console 70 | # Employee 71 | \n$d4b3c75c-2b0c-4f74-87d7-651c5ac284aa\x12\x06George\x1A\bCostanza 72 | ``` 73 | 74 | There are a couple of things to note in this string. The first is that no space is wasted in defining the field names. The numbers at the end of the line in both the `.proto` and `.rb` files indicates the field index. When the data in the protobuf message is serialized, the data is packed in a sequential order without the field names. A delimiter is used to separate the fields, which will always be in the same order. Occasionally, we may need to deprecate or remove a field. Because the fields are indexed, the index of the field that needs to be removed will always take that slot and we should never reuse that index number. If we were to reuse the index number, services which are still using the old definition would misinterpret the data in that position and things can go south especially when the data type is modified but the field index is reused (e.g. if the data type changes from an int32 to a bool). 75 | 76 | It's up to the receiver to know the indexes and their related field names when deserializing the message. This has the advantage of requiring less network bandwidth to deliver the message when compared to other message envelopes such as JSON or XML. Another advantage is that when the protobuf message is deserialized, extra fields that are not defined in the protobuf class are ignored. This makes the platform maintainable, because the senders and receivers can be updated and deployed independently. 77 | 78 | For example, a sender can have a newer `::Protobuf::Message` class which adds new fields to a protobuf message. When this message is received by another service, the new fields will be ignored by any receivers that are using an older version of the `::Protobuf::Message` class. A receiver can also be modified independently to expect a new field but if it's not defined in the sender's proto message, the field is marked as `nil` (or the language's equivalent zero or `nil` value). In these examples there is a chance that data in the new fields will be lost, so you may want to update the receivers before updating the senders. This design allows you to update the services independently without the risk of breaking the receiving apps because they aren't ready to receive the newly defined fields. 79 | 80 | ## Resources 81 | 82 | * https://developers.google.com/protocol-buffers 83 | * https://github.com/ruby-protobuf/protobuf/wiki/Compiling-Definitions 84 | 85 | ## Wrap-up 86 | 87 | Protobufs are an efficient way to package and share data between services. They are language agnostic and extendable. Because they are language agnostic, you are not constrainted to building services in a single programming language on your platform. For example, you can write your internal line-of-business applications in Rails while writing your data-crunching algorithms in R. 88 | 89 | In the next chapter, we'll spin up a development sandbox with NATS and Rails. We'll create two Rails applications, one that owns a database and shares the data via Protobuf and Active Remote and another that acts as a client that can retrieve and modify the data in the first app. 90 | 91 | [Next >>](100-chapter-09.md) 92 | -------------------------------------------------------------------------------- /100-chapter-09.md: -------------------------------------------------------------------------------- 1 | ### Chapter 9 - Active Remote Microservice Sandbox 2 | 3 | > I try to formulate a plan but my thoughts are a toxic fizz of regret, panic, and self-loathing, as if someone shook up a bottle of carbonated soda and uncapped it inside my brain. - Elan Mastai, All Our Wrong Todays 4 | 5 | ## Introduction 6 | 7 | Before we can dive into building our distributed environment, we'll first need to set up our development environment. We'll use Docker to quickly get up and running. Docker also provides its own isolated network that won't interfere with processes already running on your development machine. 8 | 9 | We'll use the terms images and containers. While these terms are sometimes used interchangeably, there are distinct differences. Docker images are the file structure of an application on your hard drive (think of an image as the files in your project folder). A Docker container is a running instance of the image (think of a container as the instance of your application). One or more containers can be spun up from a single image, as long as separate service names have been provided. Another example is you can run multiple instances of a Rails app from the same directory by specifying a different port number (e.g. `rails server -p 3000` and `rails server -p 3001`). 10 | 11 | The sandbox environment we'll build in this chapter will use the Active Remote gem to spin up a client (we'll call this app `active-remote`) that can access data in the service that owns the data (we'll call this app `active-record` because it will use Active Record to persist the data to a database that only it has access to). 12 | 13 | In this environment, we'll use NATS to pass messages between our microservices. A request to create a new employee will be sent from the Active Remote app to NATS, and NATS will forward the request to any services that have subscribed to that route. The Active Record app that we build will subscribe and respond with the newly created employee entity wrapped in a Protobuf message. 14 | 15 | _**Figure 9-1**_ Active Remote Message Passing 16 | 17 | ![alt text](images/active-remote-sequence-diagram.png "Active Remote message passing between applications") 18 | 19 | ## Install Docker 20 | 21 | If you already have Docker and Docker Compose installed, great! If not, you'll need to follow the steps below. 22 | 23 | If you're using Windows or macOS, download and install Docker Desktop. Download links and instructions can be found here: https://www.docker.com/products/docker-desktop. 24 | 25 | We'll also use Docker Compose to configure and run several applications from a single configuration file. Docker Compose is included in Docker Desktop for macOS and Windows. If you're running Linux, you'll need to install Docker separately and then follow the Docker Compose installation instructions found here: https://docs.docker.com/compose/install. 26 | 27 | ## Implementation 28 | 29 | ### What we'll need 30 | 31 | * NATS 32 | * Ruby 33 | * Ruby gems 34 | * Active Remote 35 | * Protobuf 36 | * Rails 37 | * SQLite 38 | 39 | Because we installed Docker Desktop, there is no need to install Ruby, the Ruby on Rails framework, NATS or SQLite on your computer. They will be installed inside of the Docker images and containers that we will spin up next. 40 | 41 | #### Testing our Docker and Docker Compose installation 42 | 43 | We can test our installation by running the `docker version` and the `docker-compose --version` commands. The versions you see in your output may differ from the versions you see below. 44 | 45 | **Listing 9-2** Docker environment check 46 | 47 | ```console 48 | $ docker version 49 | Client: 50 | Cloud integration: v1.0.29 51 | Version: 20.10.17 52 | API version: 1.41 53 | ... 54 | Server: Docker Desktop 4.12.0 (85629) 55 | Engine: 56 | Version: 20.10.17 57 | API version: 1.41 (minimum version 1.12) 58 | 59 | $ docker compose version 60 | Docker Compose version v2.10.2 61 | ``` 62 | 63 | If you see any errors, check your Docker Desktop installation. 64 | 65 | ### Project Directory Structure 66 | 67 | Now we'll need to create a directory for our project. As you follow along, you'll create three project sub-directories, one for our shared Protobuf messages, one for our ActiveRecord Ruby on Rails server application that stores the data in a SQLite database and one for our ActiveRemote client application that will provide a front-end for our ActiveRecord service. 68 | 69 | Following along in this tutorial, you should end up with the following directories (and many files and directories in each directory). The directory `rails-microservices-sample-code` is named after the GitHub repo found at https://github.com/kevinwatson/rails-microservices-sample-code. 70 | 71 | * rails-microservices-sample-code 72 | * chapter-09 73 | * active-record 74 | * active-remote 75 | * protobuf 76 | 77 | ### Set up a development environment 78 | 79 | Let's get started by creating a builder Dockerfile and Docker Compose file. We'll use the Dockerfile file to build an image with the command-line apps we need, and we'll use a Docker Compose configuration file to reduce the number of parameters we'll need to use to run each command. The alternative is to simply use a Dockerfile and related `docker` commands. 80 | 81 | Create the following Dockerfile file in the `rails-microservices-sample-code` directory. We'll use the name `Dockerfile.builder` to differentiate the Dockerfile we'll use to generate new rails services vs the Dockerfile we'll use to build and run our Rails applications. 82 | 83 | Note: The first line of these files is a comment and is used to indicate the file path and file name. This line can be omitted from the file. 84 | 85 | _**Listing 9-3**_ Dockerfile used to create an image that we'll use to generate our Rails application 86 | 87 | ```dockerfile 88 | # rails-microservices-sample-code/Dockerfile.builder 89 | 90 | FROM ruby:3.0.6 91 | 92 | RUN apt-get update && apt-get install -qq -y --no-install-recommends \ 93 | build-essential \ 94 | protobuf-compiler \ 95 | nodejs \ 96 | vim 97 | 98 | WORKDIR /home/root 99 | 100 | RUN gem install rails -v 6.1 101 | RUN gem install protobuf 102 | ``` 103 | 104 | Create the following `docker-compose.builder.yml` file in the `rails-microservices-sample-code` directory. We'll use this configuration file to start our development environment with all of the command-line tools that we'll need. 105 | 106 | _**Listing 9-4**_ Docker Compose file to start the container we'll use to generate our Rails application 107 | 108 | ```yaml 109 | # rails-microservices-sample-code/docker-compose.builder.yml 110 | 111 | version: "3.4" 112 | 113 | services: 114 | builder: 115 | build: 116 | context: . 117 | dockerfile: Dockerfile.builder 118 | volumes: 119 | - .:/home/root 120 | stdin_open: true 121 | tty: true 122 | ``` 123 | 124 | Let's start and log into the builder container. We'll then run the Rails generate commands from the container, which will create two Rails apps. Because we've mapped a volume in the `.yml` file above, the files that are generated will be saved to the `rails-microservices-sample-code` directory. If we didn't map a volume, the files we generate would only exist inside the container, and each time we stop and restart the container they would need to be regenerated. Mapping a volume to a directory on the host computer's will serve files through the container's environment, which includes a specific version of Ruby, Rails and the gems we'll need to run our apps. 125 | 126 | ```console 127 | $ docker compose -f docker-compose.builder.yml run builder bash 128 | ``` 129 | 130 | The `run` Docker Compose command will build the image (if it wasn't built already), start the container, ssh into the running container and give us a command prompt using the `bash` shell. 131 | 132 | You should now see that you're logged in as the root user in the container (you'll see a prompt starting with a hash `#`). Logging in as the root user is usually ok inside a container, because the isolation of the container environment limits what the root user can do. 133 | 134 | ### Protobuf 135 | 136 | Now let's create a Protobuf message and compile the `.proto` file to generate the related Ruby file, containing the classes that will be copied to each of our Ruby on Rails apps. This file will define the Protobuf message, requests and remote procedure call definitions. 137 | 138 | Create a couple of directories for our input and output files. The `mkdir -p` command below will create directories with the following structure: 139 | 140 | * protobuf 141 | * definitions 142 | * lib 143 | 144 | ```console 145 | $ mkdir -p protobuf/{definitions,lib} 146 | ``` 147 | 148 | Our Protobuf definition file: 149 | 150 | _**Listing 9-5**_ Employee message protobuf file 151 | 152 | ```protobuf 153 | # rails-microservices-sample-code/protobuf/definitions/employee_message.proto 154 | 155 | syntax = "proto3"; 156 | 157 | message EmployeeMessage { 158 | string guid = 1; 159 | string first_name = 2; 160 | string last_name = 3; 161 | } 162 | 163 | message EmployeeMessageRequest { 164 | string guid = 1; 165 | string first_name = 2; 166 | string last_name = 3; 167 | } 168 | 169 | message EmployeeMessageList { 170 | repeated EmployeeMessage records = 1; 171 | } 172 | 173 | service EmployeeMessageService { 174 | rpc Search (EmployeeMessageRequest) returns (EmployeeMessageList); 175 | rpc Create (EmployeeMessage) returns (EmployeeMessage); 176 | rpc Update (EmployeeMessage) returns (EmployeeMessage); 177 | rpc Destroy (EmployeeMessage) returns (EmployeeMessage); 178 | } 179 | ``` 180 | 181 | To compile the `.proto` files, we'll use a Rake task provided by the `protobuf` gem. To access the `protobuf` gem's Rake tasks, we'll need to create a `Rakefile`. Let's do that now. 182 | 183 | _**Listing 9-6**_ Rakefile 184 | 185 | ```ruby 186 | # rails-microservices-sample-code/protobuf/Rakefile 187 | 188 | require "protobuf/tasks" 189 | ``` 190 | 191 | Now we can run the `compile` Rake task to generate the file. 192 | 193 | ```console 194 | $ mkdir chapter-09 # create a directory for this chapter 195 | $ docker-compose -f docker-compose.builder.yml run builder bash 196 | # cd protobuf 197 | # rake protobuf:compile 198 | ``` 199 | 200 | This will generate a file named `employee_message.pb.rb` file in the `protobuf/lib` directory. We'll copy this file into the `app/lib` directory in the Rails apps we'll create next. 201 | 202 | ### Create a Rails App with a Database 203 | 204 | The first Rails app we'll generate will have an Active Record model and will be able to persist the records to a SQLite database. We'll add the `active_remote`, `protobuf-nats` and `protobuf-activerecord` gems to the `Gemfile` file. We'll then run the `bundle` command to retrieve the gems from https://rubygems.org. After retrieving the gems, we'll create scaffolding for an Employee entity and generate an `employees` table in the SQLite database. We could connect our app to a PostgreSQL or MySQL database, but for the purposes of this demo app, the file-based SQLite database is sufficient for our purposes. Of course, the Active Remote app we generate will not know nor will it care how the data is persisted (or if the data is persisted at all). 205 | 206 | Let's generate the Rails app that will act as the server and owner of the data. As the owner of the data, it can persist the data to a database. We'll call this app `active-record`. 207 | 208 | ```console 209 | # cd chapter-09 210 | # rails new active-record 211 | # cd active-record 212 | # echo "gem 'active_remote'" >> Gemfile 213 | # echo "gem 'protobuf-nats'" >> Gemfile 214 | # echo "gem 'protobuf-activerecord'" >> Gemfile 215 | # bundle 216 | # rails generate scaffold Employee guid:string first_name:string last_name:string 217 | # rails db:migrate 218 | # exit 219 | ``` 220 | 221 | Be sure to inspect the output of each of the commands above, looking for errors. If errors are encountered, please double-check each command for typos or extra characters. 222 | 223 | Let's customize the app to serve our Employee entity via Protobuf. We'll need an `app/lib` directory, and then we'll copy the generated `employee_message.pb.rb` file to this directory. 224 | 225 | ```console 226 | $ mkdir chapter-09/active-record/app/lib 227 | $ cp protobuf/lib/employee_message.pb.rb chapter-09/active-record/app/lib/ 228 | ``` 229 | 230 | Next, we'll need to create a service class to define how to handle the remote procedure call service endpoints we defined in the `.proto` file. We'll need to create an `app/services` directory. We'll then add a `app/services/employee_message_service.rb` file to re-open the `EmployeeMessageService` class defined in our `app/lib/employee_message.pb.rb` file to provide implementation details. Lastly, we'll define some scopes and field_scopes in our `app/models/employee.rb` to wire up existing model attributes with protobuf attributes. 231 | 232 | _**Listing 9-6**_ Employee Active Record model 233 | 234 | ```ruby 235 | # rails-microservices-sample-code/chapter-09/active-record/app/models/employee.rb 236 | 237 | require 'protobuf' 238 | 239 | class Employee < ApplicationRecord 240 | protobuf_message :employee_message 241 | 242 | scope :by_guid, lambda { |*values| where(guid: values) } 243 | scope :by_first_name, lambda { |*values| where(first_name: values) } 244 | scope :by_last_name, lambda { |*values| where(last_name: values) } 245 | 246 | field_scope :guid 247 | field_scope :first_name 248 | field_scope :last_name 249 | end 250 | ``` 251 | 252 | ```console 253 | $ mkdir chapter-09/active-record/app/services 254 | ``` 255 | 256 | _**Listing 9-7**_ Employee message service class 257 | 258 | ```ruby 259 | # rails-microservices-sample-code/chapter-09/active-record/app/services/employee_message_service.rb 260 | 261 | class EmployeeMessageService 262 | def search 263 | records = ::Employee.search_scope(request).map(&:to_proto) 264 | 265 | respond_with records: records 266 | end 267 | 268 | def create 269 | record = ::Employee.create(request) 270 | 271 | respond_with record 272 | end 273 | 274 | def update 275 | record = ::Employee.where(guid: request.guid).first 276 | record.assign_attributes(request) 277 | record.save! 278 | 279 | respond_with record 280 | end 281 | 282 | def destroy 283 | record = ::Employee.where(guid: request.guid).first 284 | 285 | record.delete 286 | respond_with record.to_proto 287 | end 288 | end 289 | ``` 290 | 291 | We'll also need to add a few more details. Because the `app/lib/employee_message.pb.rb` file contains multiple classes, only the class that matches the file name is loaded. In development mode, Rails can lazy load files as long as the file name can be inferred from the class name, e.g. code requiring the class `EmployeeMessageService` will try to lazy load a file named `employee_message_service.rb`, and throw an error if the file is not found. We can either separate the classes in the `app/lib/employee_message.pb.rb` file into separate files, or enable eager loading in the config. For the purposes of this demo, let's enable eager loading. 292 | 293 | Rails 6 now uses zeitwerk to autoload files. Zeitwerk expects that filenames ending with `.rb` match the convention of one class per file, but the employee_message.pb.rb file we generated above doesn't follow this convention. For this reason, we'll need to change the default autoloader configuration back to classic mode by setting `autoloader = :classic`. 294 | 295 | _**Listing 9-8**_ Development configuration file 296 | 297 | ```ruby 298 | # rails-microservices-sample-code/chapter-09/active-record/config/environments/development.rb 299 | 300 | ... 301 | config.eager_load = true 302 | ... 303 | config.autoloader = :classic 304 | ... 305 | ``` 306 | 307 | The last change we need to make to the `active-record` app is to add a `protobuf_nats.yml` config file to configure the code provided by the `protobuf-nats` gem. 308 | 309 | _**Listing 9-9**_ Protobuf Nats config file 310 | 311 | ```yml 312 | # rails-microservices-sample-code/chapter-09/active-record/config/protobuf_nats.yml 313 | 314 | default: &default 315 | servers: 316 | - "nats://nats:4222" 317 | 318 | development: 319 | <<: *default 320 | ``` 321 | 322 | ### Create a Rails App without a Database 323 | 324 | Now it's time to create our second Rails app. We'll call this one `active-remote`. It will have a model, but the model classes will inherit from `ActiveRemote::Base` instead of the default `ApplicationRecord` (which inherits from `ActiveRecord::Base`). In other words, these models will interact with the `active-remote`'s models by sending messages via the NATS server. 325 | 326 | Let's generate the `active-remote` app. We won't need the Active Record persistence layer, so we'll use the `--skip-active-record` flag. We'll need the `active_remote` and `protobuf-nats` gems, but not the `protobuf-activerecord` gem that we included in the `active-record` app. We'll use Rails scaffolding to generate a model, controller and views to view and manage our Employee entity that will be shared between the two apps. 327 | 328 | ```console 329 | $ docker-compose -f docker-compose.builder.yml run builder bash 330 | # cd chapter-09 331 | # rails new active-remote --skip-active-record --skip-webpack-install 332 | # cd active-remote 333 | # echo "gem 'active_remote'" >> Gemfile 334 | # echo "gem 'protobuf-nats'" >> Gemfile 335 | # bundle 336 | # rails generate scaffold Employee guid:string first_name:string last_name:string 337 | # exit 338 | ``` 339 | 340 | We'll need to make a couple of changes to the `active-remote` app. First, let's copy the Protobuf file. 341 | 342 | ```console 343 | $ mkdir chapter-09/active-remote/app/lib 344 | $ cp protobuf/lib/employee_message.pb.rb chapter-09/active-remote/app/lib/ 345 | ``` 346 | 347 | Let's now add a model that inherits from Active Remote. 348 | 349 | _**Listing 9-10**_ Employee Active Remote class 350 | 351 | ```ruby 352 | # rails-microservices-sample-code/chapter-09/active-remote/app/models/employee.rb 353 | 354 | class Employee < ActiveRemote::Base 355 | service_class ::EmployeeMessageService 356 | 357 | attribute :guid 358 | attribute :first_name 359 | attribute :last_name 360 | end 361 | ``` 362 | 363 | Now let's edit the `config/environments/development.rb` file to enable eager loading and use the classic autoloader for the same reasons listed above. 364 | 365 | _**Listing 9-11**_ Development configuration file 366 | 367 | ```ruby 368 | # rails-microservices-sample-code/chapter-09/active-remote/config/environments/development.rb 369 | 370 | ... 371 | config.eager_load = true 372 | ... 373 | config.autoloader = :classic 374 | ... 375 | ``` 376 | 377 | Let's add the `protobuf_nats.yml` file. 378 | 379 | _**Listing 9-12**_ Protobuf Nats config file 380 | 381 | ```yml 382 | # rails-microservices-sample-code/chapter-09/active-remote/config/protobuf_nats.yml 383 | 384 | default: &default 385 | servers: 386 | - "nats://nats:4222" 387 | 388 | development: 389 | <<: *default 390 | ``` 391 | 392 | Because we don't need webpacker or JavaScript for our test app, we'll need to disable pre-defined calls to the runtime. We can do this by removing the `javascript_pack_tag` line from the `application.html.erb` file. 393 | 394 | _**Listing 9-13**_ Remove javascript_pack_tag 395 | 396 | ```ruby 397 | # rails-microservices-sample-code/chapter-09/active-remote/app/views/layouts/application.html.erb 398 | # remove or comment out this line 399 | 400 | <%= javascript_pack_tag 'application', 'data-turbolinks-track': 'reload' %> 401 | ``` 402 | 403 | The last thing we need to do is change a couple of method calls in the `employees_controller.rb` file to change the way that our Protobuf messages are retrieved and instantiated. We need to use the `search` method instead of the default `all` and `find` Active Record methods. Also, because we're using uuids (guids) as the unique key between services, we'll generate a new uuid each time the `new` action is called. 404 | 405 | _**Listing 9-14**_ Employee controller class 406 | 407 | ```ruby 408 | # rails-microservices-sample-code/chapter-09/active-remote/controllers/employees_controller.rb 409 | 410 | def index 411 | @employees = Employee.search({}) 412 | end 413 | 414 | ... 415 | 416 | def new 417 | @employee = Employee.new(guid: SecureRandom.uuid) 418 | end 419 | 420 | ... 421 | 422 | def set_employee 423 | @employee = Employee.search(guid: params[:id]).first 424 | end 425 | ``` 426 | 427 | ### Create and Configure Our Environment 428 | 429 | Last but not least, let's add a `Dockerfile` and `docker-compose.yml` file to create an image and spin up containers and link our services together. 430 | 431 | _**Listing 9-15**_ Sandbox Dockerfile 432 | 433 | ```dockerfile 434 | # rails-microservices-sample-code/Dockerfile 435 | 436 | FROM ruby:3.0.6 437 | 438 | RUN apt-get update && apt-get install -qq -y --no-install-recommends build-essential nodejs 439 | 440 | ENV INSTALL_PATH /usr/src/service 441 | ENV HOME=$INSTALL_PATH PATH=$INSTALL_PATH/bin:$PATH 442 | RUN mkdir -p $INSTALL_PATH 443 | WORKDIR $INSTALL_PATH 444 | 445 | RUN gem install rails -v 6.1 446 | 447 | ADD Gemfile* ./ 448 | RUN set -ex && bundle install --no-deployment 449 | ``` 450 | 451 | _**Listing 9-16**_ Sandbox Docker Compose file 452 | 453 | ```yml 454 | # rails-microservices-sample-code/chapter-09/docker-compose.yml 455 | 456 | version: "3.4" 457 | 458 | services: 459 | active-record: 460 | environment: 461 | - PB_SERVER_TYPE=protobuf/nats/runner 462 | build: 463 | context: ./active-record 464 | dockerfile: ../../Dockerfile 465 | command: bundle exec rpc_server start -p 9399 -o active-record ./config/environment.rb 466 | volumes: 467 | - ./active-record:/usr/src/service 468 | depends_on: 469 | - nats 470 | active-remote: 471 | environment: 472 | - PB_CLIENT_TYPE=protobuf/nats/client 473 | build: 474 | context: ./active-remote 475 | dockerfile: ../../Dockerfile 476 | command: bundle exec puma -C config/puma.rb 477 | ports: 478 | - 3000:3000 479 | volumes: 480 | - ./active-remote:/usr/src/service 481 | depends_on: 482 | - nats 483 | nats: 484 | image: nats:latest 485 | ports: 486 | - 8222:8222 487 | ``` 488 | 489 | ### Run the Apps 490 | 491 | Congratulations! Your apps are now configured and ready to run in a Docker container. Run the following command to download the required images, build a new image that will be used by both Rails containers, and start three services: `active-record`, `active-remote` and `nats`. 492 | 493 | ```console 494 | $ cd chapter-09 495 | $ docker-compose up 496 | ``` 497 | 498 | It may take a few minutes, but once all of the containers are up and running, we can browse to http://localhost:3000/employees. The Rails app running on port 3000 is the Active Remote app. 499 | 500 | ### Monitoring 501 | 502 | If we review the log output in the console where you ran the `docker-compose up` command, we should see output like the following: 503 | 504 | ```console 505 | active-remote_1 | I, [2019-12-28T00:35:06.460838 #1] INFO -- : [CLT] - 6635f4080982 - 2aca3d71d6d0 - EmployeeMessageService#search - 48B/75B - 0.0647s - OK - 2019-12-28T00:35:06+00:00 506 | ``` 507 | 508 | This indicates that the `EmployeeMessageService#search` method was called. Not all output from the services is displayed in the output. 509 | 510 | Go ahead and click the `New Employee` link. Fill out the First name and Last name fields and click the `Create Employee` button to create a new Employee record. Review the logs again. You should see a message like the one below. 511 | 512 | ```console 513 | active-remote_1 | I, [2019-12-28T00:40:43.597089 #1] INFO -- : [CLT] - 0d6886451aa0 - 3f910c005424 - EmployeeMessageService#create 514 | ``` 515 | 516 | We can also check the NATS connection info to verify that data is being passed over the NATS server. Browse to http://localhost:8222 and click the 'connz' link. Clicking links to pull data on the http://localhost:3000/employees page will pass additional messages to the `active-record` app through the NATS server. Refreshing the http://localhost:8222/connz page will display incrementing counters on the `num_connections` and the `num_connections/in_msgs` and `num_connections/out_msgs` fields. 517 | 518 | ## Resources 519 | 520 | * https://docs.docker.com/compose 521 | * https://nats.io 522 | * https://www.sqlite.org 523 | 524 | ## Wrap-up 525 | 526 | Now that you have configured and spun up two new services that can communicate and share data via Protobuf, feel free to experiement by adding new Protobuf messages, additional remote procedure calls, etc. 527 | 528 | After completing the exercises in this chapter, we've built a synchronous platform. In other words, when the service asks for a specific Active Remote object, it expects a quick response from one of the services that own the Active Record model. In the next chapter, we'll discuss the event-driven architectural pattern. This pattern allows us to add one or more of services that can each perform an action when an event is detected. 529 | 530 | [Next >>](110-chapter-10.md) 531 | -------------------------------------------------------------------------------- /130-chapter-12.md: -------------------------------------------------------------------------------- 1 | ### Chapter 12 - Event Driven Messaging Sandbox 2 | 3 | > I'll press your flesh, you dimwitted sumbitch! You don't tell your pappy how to court the electorate. We ain't one-at-a-timin' here. We're MASS communicating! - Pappy O'Daniel, O Brother Where Art Thou? 4 | 5 | ## Introduction 6 | 7 | In chapter 9, we set up a sandbox environment to experiment with sending Protobuf messages over a NATS queue. In this chapter, we'll use Docker and Docker Compose to create an environment and walk through exactly which dependencies we'll need to get a publisher and a consumer up and running. The pub-sub architecture pattern allows us to publish events to 0 or more subscribers, without knowing anything about the individual recipients. 8 | 9 | In this sandbox environment, we'll create a publisher and a single subscriber. We're not limited to a single subscriber, as Figure 12-1 illustrates. We'll be using the fire-and-forget pattern to publish a message and all interested parties will be notified. 10 | 11 | _**Figure 12-1**_ Fire and forget 12 | 13 | ![alt text](images/many-subscribers-sequence-diagram.png "Publisher with many subscribers receiving an employee create message") 14 | 15 | For this sandbox environment, we'll create a single publisher and a single subscriber. 16 | 17 | ## What We'll Need 18 | 19 | * RabbitMQ 20 | * Ruby 21 | * Ruby gems 22 | * Active Publisher 23 | * Action Subscriber 24 | * Protobuf 25 | * Rails 26 | * SQLite 27 | 28 | Active Publisher is a gem that makes it easy to configure a RabbitMQ publisher in a Rails app. It depends on the bunny gem we used for testing in chapter 11. 29 | 30 | Action Subscriber is another gem we'll use that makes it easy to configure a RabbitMQ consumer in a Rails app. ActionSubscriber also provides a domain specific language (DSL) that makes it easy to define and subscribe to queues on the RabbitMQ server. 31 | 32 | We'll use the Protobuf gem to encode and decode our data as described in chapter 8. 33 | 34 | ## Implementation 35 | 36 | ### Project Directory Structure 37 | 38 | Let's create a directory for our project. We'll need three project sub-directories, one for our shared Protobuf messages, one for our Active Publisher Ruby on Rails application that we'll use to publish messages, and a consumer. You could create multiple consumers to demonstrate that multiple clients can listen for the same events published over the same queue. 39 | 40 | In chapter 9, we created a `rails-microservices-sample-code` directory in our home directory. The specific path is not important, but if you've been following along, we can reuse some of the code we generated in chapter 9. Following the tutorial in this chapter, you should end up with the following directories (and many files and directories in each directory). 41 | 42 | * rails-microservices-sample-code 43 | * chapter-12 44 | * active-publisher 45 | * action-subscriber 46 | * protobuf 47 | 48 | ### Set Up a Development Environment 49 | 50 | Some of the steps below are the same as the steps covered in chapter 9. We'll reuse some of the same Dockerfiles which will keep our Ruby versions consistent. I'll include them here, just so we don't have to jump back and forth between chapters. If you followed along in chapter 9 and created these files, you can skip some of these steps. 51 | 52 | Let's create a builder Dockerfile and Docker Compose file. We'll use the Dockerfile file to build an image with the command-line apps we need, and we'll use a Docker Compose configuration file to reduce the number of parameters we'll need to use to run each command. 53 | 54 | Create the following Dockerfile file in the `rails-microservices-sample-code` directory. We'll use the name `Dockerfile.builder` to differentiate the Dockerfile we'll use to generate new rails services vs the Dockerfile we'll use to build and run our Rails applications. 55 | 56 | _**Listing 12-1**_ Dockerfile used to create an image that we'll use to generate our Rails application 57 | 58 | ```dockerfile 59 | # rails-microservices-sample-code/Dockerfile.builder 60 | 61 | FROM ruby:3.0.6 62 | 63 | RUN apt-get update && apt-get install -qq -y --no-install-recommends \ 64 | build-essential \ 65 | protobuf-compiler \ 66 | nodejs \ 67 | vim 68 | 69 | WORKDIR /home/root 70 | 71 | RUN gem install rails -v 6.1 72 | RUN gem install protobuf 73 | ``` 74 | 75 | Create the following `docker-compose.builder.yml` file in the `rails-microservices-sample-code` directory. We'll use this configuration file to start our development environment with all of the command-line tools that we'll need. 76 | 77 | _**Listing 12-2**_ Docker Compose file to start the container we'll use to generate our Rails application 78 | 79 | ```yaml 80 | # rails-microservices-sample-code/docker-compose.builder.yml 81 | 82 | version: "3.4" 83 | 84 | services: 85 | builder: 86 | build: 87 | context: . 88 | dockerfile: Dockerfile.builder 89 | volumes: 90 | - .:/home/root 91 | stdin_open: true 92 | tty: true 93 | ``` 94 | 95 | Let's start and log into the builder container. We'll then run the Rails generate commands from the container, which will create two Rails apps. Because we've mapped a volume in the `.yml` file above, the files that are generated will be saved to the `rails-microservices-sample-code` directory. If we didn't map a volume, the files we generate would only exist inside the container, and each time we stop and restart the container they would need to be regenerated. Mapping a volume to a directory on the host computer's will serve files through the container's environment, which includes a specific version of Ruby, Rails and the gems we'll need to run our apps. 96 | 97 | _**Listing 12-3**_ Starting our builder container 98 | 99 | ```console 100 | $ docker-compose -f docker-compose.builder.yml run builder bash 101 | ``` 102 | 103 | The `run` Docker Compose command will build the image (if it wasn't built already), start the container, ssh into the running container and give us a command prompt using the `bash` shell. 104 | 105 | You should now see that you're logged in as the root user in the container (you'll see a prompt starting with a hash `#`). Logging in as the root user is usually ok inside a container, because the isolation of the container environment limits what the root user can do. You can now type `exit` to shut down the container. We'll start it back up later to generate our rails apps. 106 | 107 | ### Protobuf 108 | 109 | Now let's create a Protobuf message and compile the `.proto` file to generate the related Ruby file, containing the classes that will be copied to each of our Ruby on Rails apps. This file will define the Protobuf message, requests and remote procedure call definitions. 110 | 111 | Create a couple of directories for our input and output files. The `mkdir -p` command below will create directories with the following structure: 112 | 113 | * protobuf 114 | * definitions 115 | * lib 116 | 117 | _**Listing 12-4**_ Create protobuf directories (if you created these in chapter 9 you won't need to run this command again) 118 | 119 | ```console 120 | $ mkdir -p protobuf/{definitions,lib} 121 | ``` 122 | 123 | Our Protobuf definition file: 124 | 125 | _**Listing 12-5**_ Employee message protobuf file (if you created and compiled this file in chapter 9 you can skip ahead to the 'Create a Rails Message Publisher' section below) 126 | 127 | ```protobuf 128 | # rails-microservices-sample-code/protobuf/definitions/employee_message.proto 129 | 130 | syntax = "proto3"; 131 | 132 | message EmployeeMessage { 133 | string guid = 1; 134 | string first_name = 2; 135 | string last_name = 3; 136 | } 137 | 138 | message EmployeeMessageRequest { 139 | string guid = 1; 140 | string first_name = 2; 141 | string last_name = 3; 142 | } 143 | 144 | message EmployeeMessageList { 145 | repeated EmployeeMessage records = 1; 146 | } 147 | 148 | # The EmployeeMessageService service was used for ActiveRemote in chapter 9, but is not necessary here. If you have this service already defined, you can leave it here if you wish. 149 | 150 | ``` 151 | 152 | To compile the `.proto` files, we'll use a Rake task provided by the `protobuf` gem. To access the `protobuf` gem's Rake tasks, we'll need to create a `Rakefile`. Let's do that now. 153 | 154 | _**Listing 12-6**_ Rakefile 155 | 156 | ```ruby 157 | # rails-microservices-sample-code/protobuf/Rakefile 158 | 159 | require "protobuf/tasks" 160 | ``` 161 | 162 | Now we can run the `compile` Rake task to generate the file. 163 | 164 | _**Listing 12-7**_ Starting the builder container and compiling the protobuf definition 165 | 166 | ```console 167 | $ docker-compose -f docker-compose.builder.yml run builder bash 168 | # cd protobuf 169 | # rake protobuf:compile 170 | ``` 171 | 172 | This will generate a file named `employee_message.pb.rb` file in the `protobuf/lib` directory. We'll copy this file into the `app/lib` directory in the Rails apps we'll create next. 173 | 174 | ### Create a Rails Message Publisher 175 | 176 | The first Rails app we'll generate will use the ActivePublisher gem to publish messages to RabbitMQ. We'll add the `active_publisher` gem to the `Gemfile` file. We'll then run the `bundle` command to retrieve the gems from https://rubygems.org. After retrieving the gems, we'll create scaffolding for an Employee entity. This app will store the data in a SQLite database so we can experiment with create and update events. 177 | 178 | Let's generate the Rails app that will act as the publisher of the events. We'll call this app `active-publisher`. We'll also add the Protobuf Active Record gem so we can serialize our Active Record object to a Protobuf message. 179 | 180 | _**Listing 12-8**_ Generating the Rails apps and necessary files 181 | 182 | ```console 183 | $ mkdir chapter-12 # create a directory for this chapter 184 | $ docker-compose -f docker-compose.builder.yml run builder bash 185 | # cd chapter-12 186 | # rails new active-publisher --skip-webpack-install 187 | # cd active-publisher 188 | # echo "gem 'active_publisher'" >> Gemfile 189 | # echo "gem 'protobuf-activerecord'" >> Gemfile 190 | # bundle 191 | # rails generate scaffold Employee guid:string first_name:string last_name:string 192 | # rails db:migrate 193 | # exit 194 | ``` 195 | 196 | Be sure to inspect the output of each of the commands above, looking for errors. If errors are encountered, please double-check each command for typos or extra characters. 197 | 198 | Let's customize the app to serve our Employee entity via Protobuf. We'll need an `app/lib` directory, and then we'll copy the generated `employee_message.pb.rb` file to this directory. 199 | 200 | _**Listing 12-9**_ Setting up the app/lib directory 201 | 202 | ```console 203 | $ mkdir chapter-12/active-publisher/app/lib 204 | $ cp protobuf/lib/employee_message.pb.rb chapter-12/active-publisher/app/lib/ 205 | ``` 206 | 207 | Next, we'll add an `active_publisher` configuration file to the `config` directory. This file will define how our app should connect to the RabbitMQ server. The `rabbit` host will be defined in the `docker-compose` file we'll define in a couple of minutes. 208 | 209 | _**Listing 12-10**_ Active Publisher configuration 210 | 211 | ```yml 212 | # rails-microservices-sample-code/chapter-12/active-publisher/config/active_publisher.yml 213 | 214 | default: &default 215 | host: rabbit 216 | username: guest 217 | password: guest 218 | 219 | development: 220 | <<: *default 221 | ``` 222 | 223 | Now let's create an initializer for Active Publisher. This will load the gem, set the adapter, and load the configuration file. Let's create this file in the `config/initializers` directory. 224 | 225 | _**Listing 12-11**_ Active Publisher initializer 226 | 227 | ```ruby 228 | # rails-microservices-sample-code/chapter-12/active-publisher/config/initializers/active_publisher.rb 229 | 230 | require "active_publisher" 231 | 232 | ::ActivePublisher::Configuration.configure_from_yaml_and_cli 233 | ``` 234 | 235 | Next, let's modify the employee model so we can send the employee Profobuf object to RabbitMQ. We'll use Active Record callbacks to publish messages to separate `created` and `updated` queues after an employee record has been created or modified. Open the `app/models/employee.rb` file and add the following code. 236 | 237 | _**Listing 12-12**_ Employee Active Record model 238 | 239 | ```ruby 240 | # rails-microservices-sample-code/chapter-12/active-publisher/app/models/employee.rb 241 | 242 | require 'protobuf' 243 | 244 | class Employee < ApplicationRecord 245 | protobuf_message :employee_message 246 | 247 | after_create :publish_created 248 | after_update :publish_updated 249 | 250 | def publish_created 251 | Rails.logger.info "Publishing employee object #{self.inspect} on the employee.created queue." 252 | ::ActivePublisher.publish("employee.created", self.to_proto.encode, "events", {}) 253 | end 254 | 255 | def publish_updated 256 | Rails.logger.info "Publishing employee object #{self.inspect} on the employee.updated queue." 257 | ::ActivePublisher.publish("employee.updated", self.to_proto.encode, "events", {}) 258 | end 259 | end 260 | ``` 261 | 262 | Our simple app won't need webpacker or JavaScript, so we'll need to disable pre-defined calls to the runtime. We can do this by removing the `javascript_pack_tag` line from the `application.html.erb` file. 263 | 264 | _**Listing 9-13**_ Remove javascript_pack_tag 265 | 266 | ```ruby 267 | # rails-microservices-sample-code/chapter-12/active-publisher/app/views/layouts/application.html.erb 268 | # remove or comment out this line 269 | 270 | <%= javascript_pack_tag 'application', 'data-turbolinks-track': 'reload' %> 271 | ``` 272 | 273 | Because we're using GUIDs to uniquely identify objects that we're serializing and passing between services, let's modify the controller's `new` action so that it will generate a new GUID. 274 | 275 | _**Listing 12-13**_ Employee controller 276 | 277 | ```ruby 278 | # rails-microservices-sample-code/chapter-12/active-publisher/controllers/employees_controller.rb 279 | 280 | def new 281 | @employee = Employee.new(guid: SecureRandom.uuid) 282 | end 283 | ``` 284 | 285 | We'll also need to add a few more details. Because the `app/lib/employee_message.pb.rb` file contains multiple classes, only the class that matches the file name is loaded. In development mode, Rails can lazy load files as long as the file name can be inferred from the class name, e.g. code requiring the class `EmployeeMessageService` will try to lazy load a file named `employee_message_service.rb`, and throw an error if the file is not found. We can either separate the classes in the `app/lib/employee_message.pb.rb` file into separate files, or enable eager loading in the config. For the purposes of this demo, let's enable eager loading and also cache classes. We'll also need to configure the logger to send output to Docker logs. 286 | 287 | As we covered in chapter 9, Rails 6 now uses zeitwerk to autoload files by default, so we'll also want to change the default and set `autoloader = :classic` in our environment file. 288 | 289 | _**Listing 12-14**_ Development configuration 290 | 291 | ```ruby 292 | # rails-microservices-sample-code/chapter-12/active-publisher/config/environments/development.rb 293 | 294 | Rails.application.configure do 295 | ... 296 | config.cache_classes = true 297 | ... 298 | config.eager_load = true 299 | ... 300 | config.autoloader = :classic 301 | ... 302 | logger = ActiveSupport::Logger.new(STDOUT) 303 | logger.formatter = config.log_formatter 304 | config.logger = ActiveSupport::TaggedLogging.new(logger) 305 | end 306 | ``` 307 | 308 | That's it. Now let's build our subscriber. 309 | 310 | ### Create a Message Subscriber 311 | 312 | Let's create the `action-subscriber` app. It will subscribe to the employee created and updated message queues and simply log that it received a message on the queue. 313 | 314 | _**Listing 12-15**_ Generating the Rails apps and necessary files 315 | 316 | ```console 317 | $ docker-compose -f docker-compose.builder.yml run builder bash 318 | # cd chapter-12 319 | # rails new action-subscriber --skip-active-record 320 | # cd action-subscriber 321 | # echo "gem 'action_subscriber'" >> Gemfile 322 | # echo "gem 'protobuf'" >> Gemfile 323 | # bundle 324 | # exit 325 | ``` 326 | 327 | Now let's set up Action Subscriber to listen for events. We'll need to add a `EmployeeSubscriber` class and add routes via the `ActionSubscriber.draw_routes` method. 328 | 329 | We'll want to put our subscriber classes in their own `subscribers` directory. We'll also need the `lib` directory where we'll copy our Employee Protobuf class. Let's create these directories and copy the files to one of those directories: 330 | 331 | _**Listing 12-16**_ Generating Rails app directories and copying the message class 332 | 333 | ```console 334 | $ mkdir chapter-12/action-subscriber/app/{lib,subscribers} 335 | $ cp protobuf/lib/employee_message.pb.rb chapter-12/action-subscriber/app/lib/ 336 | ``` 337 | 338 | Now let's add the subscriber class. For the purposes of our playground we'll keep it simple - just log that we received the message. 339 | 340 | _**Listing 12-17**_ Employee subscriber class 341 | 342 | ```ruby 343 | # rails-microservices-sample-code/chapter-12/action-subscriber/app/subscribers/employee_subscriber.rb 344 | 345 | class EmployeeSubscriber < ::ActionSubscriber::Base 346 | def created 347 | Rails.logger.info "Received created message: #{EmployeeMessage.decode(payload).inspect}" 348 | end 349 | 350 | def updated 351 | Rails.logger.info "Received updated message: #{EmployeeMessage.decode(payload).inspect}" 352 | end 353 | end 354 | ``` 355 | 356 | Our app needs to know which queues to subscribe to, so we use the `default_routes_for` method which will read our `EmployeeSubscriber` class and generate queues for each of our public methods or subscribe to those queues if they already exist. The hostname `host.docker.internal` is a special Docker hostname, it points to the ip address of the host machine. 357 | 358 | _**Listing 12-18**_ Action Subscriber initializer 359 | 360 | ```ruby 361 | # rails-microservices-sample-code/chapter-12/action-subscriber/config/initializers/action_subscriber.rb 362 | 363 | ActionSubscriber.draw_routes do 364 | default_routes_for EmployeeSubscriber 365 | end 366 | 367 | ActionSubscriber.configure do |config| 368 | config.hosts = ["host.docker.internal"] 369 | config.port = 5672 370 | end 371 | ``` 372 | 373 | We'll need to enable the `cache_classes` and `eager_load` settings, the same way we did for the publisher. We'll also want to use classic mode for `autoloader`. We'll also need to set up a logger so that we can see the log output from our Docker container. 374 | 375 | _**Listing 12-19**_ Development configuration 376 | 377 | ```ruby 378 | # rails-microservices-sample-code/chapter-12/action-subscriber/config/environments/development.rb 379 | 380 | config.cache_classes = true 381 | ... 382 | config.eager_load = true 383 | ... 384 | config.autoloader = :classic 385 | ... 386 | logger = ActiveSupport::Logger.new(STDOUT) 387 | logger.formatter = config.log_formatter 388 | config.logger = ActiveSupport::TaggedLogging.new(logger) 389 | ``` 390 | 391 | ### Create and Configure Our Environment 392 | 393 | Last but not least, let's add a `Dockerfile` and `docker-compose.yml` file to build an image and spin up our Rails and RabbitMQ containers. The `Dockerfile` may already exist from the sandbox we built in chapter 9, but if not, it has the same content here. The `docker-compose.yml` file is new. 394 | 395 | _**Listing 12-20**_ Sandbox Dockerfile (if you created this file in chapter 9 no additional changes are needed) 396 | 397 | ```dockerfile 398 | # rails-microservices-sample-code/Dockerfile 399 | 400 | FROM ruby:3.0.6 401 | 402 | RUN apt-get update && apt-get install -qq -y --no-install-recommends build-essential nodejs 403 | 404 | ENV INSTALL_PATH /usr/src/service 405 | ENV HOME=$INSTALL_PATH PATH=$INSTALL_PATH/bin:$PATH 406 | RUN mkdir -p $INSTALL_PATH 407 | WORKDIR $INSTALL_PATH 408 | 409 | RUN gem install rails -v 6.1 410 | 411 | ADD Gemfile* ./ 412 | RUN set -ex && bundle install --no-deployment 413 | ``` 414 | 415 | The following Docker Compose file includes an instance of RabbitMQ and our new `active-publisher` and `action-subscriber` Rails apps. We'll expose the web app on port 3001. RabbitMQ can take a few seconds to start, so we'll our `action-subscriber` service to restart if it can't connect. In a real-world application we would want to check the response from RabbitMQ before we started up the subscriber. 416 | 417 | Normally, we would add the subscriber to the same Docker Compose file, but, because the Action Subscriber service tries to connect immediately and RabbitMQ can take a few seconds to load, we'll run the subscriber process from a separate Docker Compose file. We'll also need to expose port 5672 to the host machine so we can connect from another Compose environment. 418 | 419 | _**Listing 12-21**_ Sandbox Docker Compose file 420 | 421 | ```yml 422 | # rails-microservices-sample-code/chapter-12/docker-compose.yml 423 | # Usage: docker-compose up 424 | 425 | version: "3.4" 426 | 427 | services: 428 | active-publisher: 429 | build: 430 | context: ./active-publisher 431 | dockerfile: ../../Dockerfile 432 | command: bundle exec puma -C config/puma.rb 433 | volumes: 434 | - ./active-publisher:/usr/src/service 435 | ports: 436 | - 3001:3000 437 | depends_on: 438 | - rabbit 439 | rabbit: 440 | image: rabbitmq:latest 441 | ports: 442 | - 5672:5672 443 | ``` 444 | 445 | Now let's add the `action-subscriber` configuration file. Note that because the Action Subscriber executable spawns a child process to listen for events from RabbmitMQ, we lose the log output if we start the container using the `up` command. To view all of the log info in the terminal, we'll us the Docker Compose `run` command to start a bash shell and run our `action_subscriber` executable there. 446 | 447 | _**Listing 12-22**_ Sandbox Docker Compose subscriber file 448 | 449 | ```yml 450 | # rails-microservices-sample-code/chapter-12/docker-compose-subscriber.yml 451 | # Usage: docker-compose -f docker-compose-subscriber.yml run action-subscriber bash -c 'bundle exec action_subscriber start' 452 | 453 | version: "3.4" 454 | 455 | services: 456 | action-subscriber: 457 | build: 458 | context: ./action-subscriber 459 | dockerfile: ../../Dockerfile 460 | volumes: 461 | - ./action-subscriber:/usr/src/service 462 | 463 | ``` 464 | 465 | Now that everything's in place, let's start our sandbox environment. Because we may already have a `docker-compose.yml` file in the directory, we named our new config files `docker-compose.yml` and `docker-compose-subscriber.yml`. If we ran the shortest version of the `docker-compose up` command, it would by default look for and load the `docker-compose.yml` file. We can use the `-f` flag to specify that we want to use other configuration files instead. Let's run those commands now. 466 | 467 | _**Listing 12-23**_ Starting the sandbox 468 | 469 | ```console 470 | $ cd chapter-12 471 | $ docker-compose up 472 | ``` 473 | 474 | Once you see lines like this, RabbitMQ has started and the Active Publisher Rails app has successfully connected. 475 | 476 | _**Listing 12-24**_ Sandbox logging 477 | 478 | ```console 479 | rabbit_1 | 2020-02-09 22:54:02.253 [info] <0.8.0> Server startup complete; 0 plugins started. 480 | rabbit_1 | completed with 0 plugins. 481 | 72.30.0.2:5672) 482 | rabbit_1 | 2020-02-09 22:54:37.395 [info] <0.641.0> connection <0.641.0> (172.30.0.1:53140 -> 172.30.0.2:5672): user 'guest' authenticated and granted access to vhost '/' 483 | ``` 484 | 485 | Now let's start the subscriber in another terminal window. 486 | 487 | _**Listing 12-25**_ Starting the subscriber sandbox 488 | 489 | ```console 490 | $ cd chapter-12 491 | $ docker-compose -f docker-compose-subscriber.yml run action-subscriber bash -c 'bundle exec action_subscriber start' 492 | ``` 493 | 494 | You should see output like the following. 495 | 496 | _**Listing 12-26**_ Subscriber sandbox logging 497 | 498 | ```console 499 | I, [2020-02-09T22:54:53.900735 #1] INFO -- : Loading configuration... 500 | I, [2020-02-09T22:54:53.902758 #1] INFO -- : Requiring app... 501 | I, [2020-02-09T22:54:59.308155 #1] INFO -- : Starting server... 502 | I, [2020-02-09T22:54:59.374240 #1] INFO -- : Rabbit Hosts: ["host.docker.internal"] 503 | Rabbit Port: 5672 504 | Threadpool Size: 1 505 | Low Priority Subscriber: false 506 | Decoders: 507 | --application/json 508 | --text/plain 509 | 510 | I, [2020-02-09T22:54:59.374419 #1] INFO -- : Middlewares [ 511 | I, [2020-02-09T22:54:59.374488 #1] INFO -- : [ActionSubscriber::Middleware::ErrorHandler, [], nil] 512 | I, [2020-02-09T22:54:59.374542 #1] INFO -- : [ActionSubscriber::Middleware::Decoder, [], nil] 513 | I, [2020-02-09T22:54:59.374944 #1] INFO -- : ] 514 | I, [2020-02-09T22:54:59.375946 #1] INFO -- : EmployeeSubscriber 515 | I, [2020-02-09T22:54:59.376504 #1] INFO -- : -- method: created 516 | I, [2020-02-09T22:54:59.376856 #1] INFO -- : -- threadpool: default (1 threads) 517 | I, [2020-02-09T22:54:59.378129 #1] INFO -- : -- exchange: events 518 | I, [2020-02-09T22:54:59.379231 #1] INFO -- : -- queue: actionsubscriber.employee.created 519 | I, [2020-02-09T22:54:59.379911 #1] INFO -- : -- routing_key: employee.created 520 | I, [2020-02-09T22:54:59.380686 #1] INFO -- : -- prefetch: 2 521 | I, [2020-02-09T22:54:59.382130 #1] INFO -- : -- method: updated 522 | I, [2020-02-09T22:54:59.382702 #1] INFO -- : -- threadpool: default (1 threads) 523 | I, [2020-02-09T22:54:59.383237 #1] INFO -- : -- exchange: events 524 | I, [2020-02-09T22:54:59.383626 #1] INFO -- : -- queue: actionsubscriber.employee.updated 525 | I, [2020-02-09T22:54:59.384405 #1] INFO -- : -- routing_key: employee.updated 526 | I, [2020-02-09T22:54:59.384667 #1] INFO -- : -- prefetch: 2 527 | I, [2020-02-09T22:54:59.393366 #1] INFO -- : Action Subscriber connected 528 | ``` 529 | 530 | These log lines indicate that the subscriber has connected to the server successfully, connected to two queues and is now listening for events. 531 | 532 | Let's create some events. Open your browser and browse to http://localhost:3001/employees. Port 3001 is the port we exposed from the Active Publisher Rails app in the `docker-compose.yml` file. You should see a simple web page with the title **Employees** and a 'New Employee' link. Let's go ahead and click the link. You should now be able to create a new employee record in the web form. Once you fill it out and click the 'Create Employee' button, several things will happen. First, the form data will be sent back to the Active Publisher Rails app. The controller will pass that data on to Active Record, which will create a new record in the SQLite database. Next, the `after_create` callback will run, encoding our Protobuf message and placing it on the `actionsubscriber.employee.created` queue. RabbitMQ will notify subscribers of a specific queue of any new messages. Our Action Subscriber Rails app is one such subscriber. In our `EmployeeSubscriber#created` event handler method, we wrote code to log that we received a message. If you inspect the output from the terminal window where we started the Action Subscriber Rails app, you should see output like the output below. 533 | 534 | _**Listing 12-27**_ More subscriber sandbox logging 535 | 536 | ```console 537 | I, [2020-02-09T23:14:31.163127 #1] INFO -- : RECEIVED 7a99f6 from actionsubscriber.employee.created 538 | I, [2020-02-09T23:14:31.163758 #1] INFO -- : START 7a99f6 EmployeeSubscriber#created 539 | Received created message: # 540 | I, [2020-02-09T23:14:31.164414 #1] INFO -- : FINISHED 7a99f6 541 | ``` 542 | 543 | Congratulations! You have successfully built a messaging platform that can publish and respond to events. Try editing the record you just created. You should see similar output in the Action Subscriber Rails app as it receives and processes the event and the data. For even more fun, try spinning up a second or third subscriber service that listen to the same queues and watch as all of them respond simultaneously to the same published message. 544 | 545 | ## Resources 546 | 547 | * https://docs.docker.com/compose/reference/run 548 | * https://github.com/mxenabled/active_publisher 549 | * https://github.com/ruby-protobuf/protobuf/wiki/Serialization 550 | 551 | ## Wrap-up 552 | 553 | In this chapter, we started a RabbitMQ service, built an Active Publisher Rails application and an Action Subscriber Rails application. We spun those services using two separate Docker environments, and published and consumed messages. We reviewed the logs where we could see that the Protobuf messages were sent successfully from one Rails application to another. 554 | 555 | In the next chapter, we're going to combine the two environments to create a platform that can retrieve data from other services as needed. But that's not all! We'll also add event listeners to our services to respond to events from other services. 556 | 557 | [Next >>](140-chapter-13.md) 558 | -------------------------------------------------------------------------------- /140-chapter-13.md: -------------------------------------------------------------------------------- 1 | ### Chapter 13 - Active Remote with Events Sandbox 2 | 3 | ## Introduction 4 | 5 | So far we've built two Rails services that can communicate with each other via Active Remote and NATS. We've also built two different Rails services that communicate with each via Active Publisher, Action Subscriber and RabbitMQ. As your business requirements grow, you may find the need to use both in your environment - Active Remote for real-time communication between services and Active Subscriber for event-driven messaging. 6 | 7 | Lucky for us, we've already laid the groundwork for this type of platform. In this chapter, we'll spin up a new sandbox environment that uses both NATS and RabbitMQ to communicate and publish messages. 8 | 9 | _**Figure 13-1**_ Creating an employee and notifying all interested parties 10 | 11 | ![alt text](images/synchronous-and-event-driven-platform.png "Publishing an employee created message via Active Remote and Active Publisher") 12 | 13 | ## What We'll Need 14 | 15 | * NATS 16 | * RabbitMQ 17 | * Ruby 18 | * Ruby gems 19 | * Active Publisher 20 | * Active Remote 21 | * Action Subscriber 22 | * Protobuf 23 | * Rails 24 | * SQLite 25 | 26 | ## Implementation 27 | 28 | ### Project Directory Structure 29 | 30 | Let's create a directory for our project. We'll need three project sub-directories, one for our shared Protobuf messages, one for our Active Publisher Ruby on Rails application that we'll use to publish messages, and a consumer. You could create multiple consumers to demonstrate that multiple clients can listen for the same events published over the same queue. 31 | 32 | In chapter 9, we created a `rails-microservices-sample-code` directory in our home directory. The specific path is not important, but if you've been following along, we can reuse some of the code we generated in chapter 9. Following the tutorial in this chapter, you should end up with the following directories (and many files and directories in each directory). 33 | 34 | * rails-microservices-sample-code 35 | * chapter-13 36 | * active-record 37 | * active-remote-publisher 38 | * action-subscriber 39 | * protobuf 40 | 41 | ### Set Up a Development Environment 42 | 43 | Some of the steps below are the same as the steps covered in chapters 9 and 12. We'll reuse some of the same Dockerfiles which will keep our Ruby versions consistent. I'll include them here, just so we don't have to jump back and forth between chapters. If you followed along in chapters 9 and 12 and created these files, you can skip some of these steps. 44 | 45 | Let's create a builder Dockerfile and Docker Compose file. We'll use the Dockerfile file to build an image with the command-line apps we need, and we'll use a Docker Compose configuration file to reduce the number of parameters we'll need to use to run each command. 46 | 47 | Create the following Dockerfile file in the `rails-microservices-sample-code` directory. We'll use the name `Dockerfile.builder` to differentiate the Dockerfile we'll use to generate new rails services vs the Dockerfile we'll use to build and run our Rails applications. 48 | 49 | _**Listing 13-2**_ Dockerfile used to create an image that we'll use to generate our Rails application 50 | 51 | ```dockerfile 52 | # rails-microservices-sample-code/Dockerfile.builder 53 | 54 | FROM ruby:2.6.5 55 | 56 | RUN apt-get update && apt-get install -qq -y --no-install-recommends \ 57 | build-essential \ 58 | protobuf-compiler \ 59 | nodejs \ 60 | vim 61 | 62 | WORKDIR /home/root 63 | 64 | RUN gem install rails -v 5.2.4 65 | RUN gem install protobuf 66 | ``` 67 | 68 | Create the following `docker-compose.builder.yml` file in the `rails-microservices-sample-code` directory. We'll use this configuration file to start our development environment with all of the command-line tools that we'll need. 69 | 70 | _**Listing 13-3**_ Docker Compose file to start the container we'll use to generate our Rails application 71 | 72 | ```yaml 73 | # rails-microservices-sample-code/docker-compose.builder.yml 74 | 75 | version: "3.4" 76 | 77 | services: 78 | builder: 79 | build: 80 | context: . 81 | dockerfile: Dockerfile.builder 82 | volumes: 83 | - .:/home/root 84 | stdin_open: true 85 | tty: true 86 | ``` 87 | 88 | Let's start and log into the builder container. We'll then run the Rails generate commands from the container, which will create two Rails apps. Because we've mapped a volume in the `.yml` file above, the files that are generated will be saved to the `rails-microservices-sample-code` directory. If we didn't map a volume, the files we generate would only exist inside the container, and each time we stop and restart the container they would need to be regenerated. Mapping a volume to a directory on the host computer's will serve files through the container's environment, which includes a specific version of Ruby, Rails and the gems we'll need to run our apps. 89 | 90 | _**Listing 13-4**_ Starting our builder container 91 | 92 | ```console 93 | $ docker-compose -f docker-compose.builder.yml run builder bash 94 | ``` 95 | 96 | The `run` Docker Compose command will build the image (if it wasn't built already), start the container, ssh into the running container and give us a command prompt using the `bash` shell. 97 | 98 | You should now see that you're logged in as the root user in the container (you'll see a prompt starting with a hash `#`). Logging in as the root user is usually ok inside a container, because the isolation of the container environment limits what the root user can do. 99 | 100 | ### Protobuf 101 | 102 | Now let's create a Protobuf message and compile the `.proto` file to generate the related Ruby file, containing the classes that will be copied to each of our Ruby on Rails apps. This file will define the Protobuf message, requests and remote procedure call definitions. 103 | 104 | Create a couple of directories for our input and output files. The `mkdir -p` command below will create directories with the following structure: 105 | 106 | * protobuf 107 | * definitions 108 | * lib 109 | 110 | _**Listing 13-5**_ Creating the necessary directories 111 | 112 | ```console 113 | $ mkdir -p protobuf/{definitions,lib} 114 | ``` 115 | 116 | Our Protobuf definition file: 117 | 118 | _**Listing 13-6**_ Employee message protobuf file 119 | 120 | ```protobuf 121 | # rails-microservices-sample-code/protobuf/definitions/employee_message.proto 122 | 123 | syntax = "proto3"; 124 | 125 | message EmployeeMessage { 126 | string guid = 1; 127 | string first_name = 2; 128 | string last_name = 3; 129 | } 130 | 131 | message EmployeeMessageRequest { 132 | string guid = 1; 133 | string first_name = 2; 134 | string last_name = 3; 135 | } 136 | 137 | message EmployeeMessageList { 138 | repeated EmployeeMessage records = 1; 139 | } 140 | 141 | service EmployeeMessageService { 142 | rpc Search (EmployeeMessageRequest) returns (EmployeeMessageList); 143 | rpc Create (EmployeeMessage) returns (EmployeeMessage); 144 | rpc Update (EmployeeMessage) returns (EmployeeMessage); 145 | rpc Destroy (EmployeeMessage) returns (EmployeeMessage); 146 | } 147 | ``` 148 | 149 | To compile the `.proto` files, we'll use a Rake task provided by the `protobuf` gem. To access the `protobuf` gem's Rake tasks, we'll need to create a `Rakefile`. Let's do that now. 150 | 151 | _**Listing 13-7**_ Rakefile 152 | 153 | ```ruby 154 | # rails-microservices-sample-code/protobuf/Rakefile 155 | 156 | require "protobuf/tasks" 157 | ``` 158 | 159 | Now we can run the `compile` Rake task to generate the file. 160 | 161 | _**Listing 13-8**_ Starting the builder container and compiling the protobuf definition 162 | 163 | ```console 164 | $ docker-compose -f docker-compose.builder.yml run builder bash 165 | # cd protobuf 166 | # rake protobuf:compile 167 | ``` 168 | 169 | This will generate a file named `employee_message.pb.rb` file in the `protobuf/lib` directory. We'll copy this file into the `app/lib` directory in the Rails apps we'll create next. 170 | 171 | ### Create a Rails App without a Database 172 | 173 | We'll call this first app `active-remote`. It will have a model, but the model classes will inherit from `ActiveRemote::Base` instead of the default `ApplicationRecord` (which inherits from `ActiveRecord::Base`). In other words, these models will interact with the `active-remote`'s models by sending messages via the NATS server. 174 | 175 | Let's generate the `active-remote` app. We won't need the Active Record persistence layer, so we'll use the `--skip-active-record` flag. We'll need the `active_remote` and `protobuf-nats` gems, but not the `protobuf-activerecord` gem that we included in the `active-record` app. We'll use Rails scaffolding to generate a model, controller and views to view and manage our Employee protobuf that will be shared between our apps. 176 | 177 | _**Listing 13-9**_ Generating the Rails apps and necessary files 178 | 179 | ```console 180 | $ mkdir chapter-13 # create a directory for this chapter 181 | $ docker-compose -f docker-compose.builder.yml run builder bash 182 | # cd chapter-13 183 | # rails new active-remote --skip-active-record 184 | # cd active-remote 185 | # echo "gem 'active_remote'" >> Gemfile 186 | # echo "gem 'protobuf-nats'" >> Gemfile 187 | # bundle 188 | # rails generate scaffold Employee guid:string first_name:string last_name:string 189 | # exit 190 | ``` 191 | 192 | We'll need to make a couple of changes to the `active-remote` app. First, let's copy the Protobuf file. 193 | 194 | _**Listing 13-10**_ Setting up the app/lib directory and copying the proto class 195 | 196 | ```console 197 | $ mkdir chapter-13/active-remote/app/lib 198 | $ cp protobuf/lib/employee_message.pb.rb chapter-13/active-remote/app/lib/ 199 | ``` 200 | 201 | Now let's edit the `config/environments/development.rb` file to enable eager loading. 202 | 203 | _**Listing 13-11**_ Development configuration 204 | 205 | ```ruby 206 | # rails-microservices-sample-code/chapter-13/active-remote/config/environments/development.rb 207 | 208 | ... 209 | config.eager_load = true 210 | ... 211 | ``` 212 | 213 | Let's add the `protobuf_nats.yml` file. 214 | 215 | _**Listing 13-12**_ Protbuf Nats configuration 216 | 217 | ```yml 218 | # rails-microservices-sample-code/chapter-13/active-remote/config/protobuf_nats.yml 219 | 220 | default: &default 221 | servers: 222 | - "nats://nats:4222" 223 | 224 | development: 225 | <<: *default 226 | ``` 227 | 228 | Let's now add a model that inherits from Active Remote. 229 | 230 | _**Listing 13-13**_ Employee Active Remote model 231 | 232 | ```ruby 233 | # rails-microservices-sample-code/chapter-13/active-remote/app/models/employee.rb 234 | 235 | class Employee < ActiveRemote::Base 236 | service_class ::EmployeeMessageService 237 | 238 | attribute :guid 239 | attribute :first_name 240 | attribute :last_name 241 | end 242 | ``` 243 | 244 | The last thing we need to do is change a couple of method calls in the `employees_controller.rb` file to change the way that our Protobuf messages are retrieved and instantiated. We need to use the `search` method instead of the default `all` and `find` Active Record methods. Also, because we're using uuids (guids) as the unique key between services, we'll generate a new uuid each time the `new` action is called. 245 | 246 | _**Listing 13-14**_ Employee controller 247 | 248 | ```ruby 249 | # rails-microservices-sample-code/chapter-13/active-remote/controllers/employees_controller.rb 250 | 251 | def index 252 | @employees = Employee.search({}) 253 | end 254 | 255 | ... 256 | 257 | def new 258 | @employee = Employee.new(guid: SecureRandom.uuid) 259 | end 260 | 261 | ... 262 | 263 | def set_employee 264 | @employee = Employee.search(guid: params[:id]).first 265 | end 266 | ``` 267 | 268 | ### Create a Rails App with a Database 269 | 270 | The second Rails app we'll create will have its own database and listen for messages over NATS via Active Remote. It will also use the Active Publisher gem to publish messages to RabbitMQ. We'll add the required gems to the `Gemfile` file. We'll then run the `bundle` command to retrieve the gems from https://rubygems.org. After retrieving the gems, we'll create scaffolding for an Employee entity. This app will store the data in a SQLite database so we can set up the create and update events. 271 | 272 | _**Listing 13-15**_ Generating the Rails app and necessary files 273 | 274 | ```console 275 | $ docker-compose -f docker-compose.builder.yml run builder bash 276 | # cd chapter-13 277 | # rails new active-record-publisher 278 | # cd active-record-publisher 279 | # echo "gem 'active_remote'" >> Gemfile 280 | # echo "gem 'active_publisher'" >> Gemfile 281 | # echo "gem 'protobuf-activerecord'" >> Gemfile 282 | # echo "gem 'protobuf-nats'" >> Gemfile 283 | # bundle 284 | # rails generate scaffold Employee guid:string first_name:string last_name:string 285 | # rails db:migrate 286 | # exit 287 | ``` 288 | 289 | Be sure to inspect the output of each of the commands above, looking for errors. If errors are encountered, please double-check each command for typos or extra characters. 290 | 291 | Let's customize the app to serve our Employee entity via Protobuf. We'll need an `app/lib` directory, and then we'll copy the generated `employee_message.pb.rb` file to this directory. 292 | 293 | _**Listing 13-16**_ Generating Rails app directories and copying the message class 294 | 295 | ```console 296 | $ mkdir chapter-13/active-record-publisher/app/lib 297 | $ cp protobuf/lib/employee_message.pb.rb chapter-13/active-record-publisher/app/lib/ 298 | ``` 299 | 300 | Next, we'll add an `active_publisher` configuration file to the `config` directory. This file will define how our app should connect to the RabbitMQ server. The `rabbit` host will be defined in the `docker-compose` file we'll define in a couple of minutes. 301 | 302 | _**Listing 13-17**_ Active Publisher configuration 303 | 304 | ```yml 305 | # rails-microservices-sample-code/chapter-13/active-record-publisher/config/active_publisher.yml 306 | 307 | default: &default 308 | host: rabbit 309 | username: guest 310 | password: guest 311 | 312 | development: 313 | <<: *default 314 | ``` 315 | 316 | Let's add the `protobuf_nats.yml` file. 317 | 318 | _**Listing 13-18**_ Protobuf Nats configuration 319 | 320 | ```yml 321 | # rails-microservices-sample-code/chapter-13/active-record-publisher/config/protobuf_nats.yml 322 | 323 | default: &default 324 | servers: 325 | - "nats://nats:4222" 326 | 327 | development: 328 | <<: *default 329 | ``` 330 | 331 | Now let's create an initializer for Active Publisher. This will load the gem, set the adapter, and load the configuration file. Let's create this file in the `config/initializers` directory. 332 | 333 | _**Listing 13-19**_ Active Publisher initializer 334 | 335 | ```ruby 336 | # rails-microservices-sample-code/chapter-13/active-record-publisher/config/initializers/active_publisher.rb 337 | 338 | require "active_publisher" 339 | 340 | ::ActivePublisher::Configuration.configure_from_yaml_and_cli 341 | ``` 342 | 343 | Next, let's modify the employee model so we can send the employee Profobuf object to RabbitMQ. We'll use Active Record callbacks to publish messages to separate `created` and `updated` queues after an employee record has been created or modified. Open the `app/models/employee.rb` file and add the following code. 344 | 345 | _**Listing 13-20**_ Employee model 346 | 347 | ```ruby 348 | # rails-microservices-sample-code/chapter-13/active-record-publisher/app/models/employee.rb 349 | 350 | require 'protobuf' 351 | 352 | class Employee < ApplicationRecord 353 | protobuf_message :employee_message 354 | 355 | after_create :publish_created 356 | after_update :publish_updated 357 | 358 | scope :by_guid, lambda { |*values| where(guid: values) } 359 | scope :by_first_name, lambda { |*values| where(first_name: values) } 360 | scope :by_last_name, lambda { |*values| where(last_name: values) } 361 | 362 | field_scope :guid 363 | field_scope :first_name 364 | field_scope :last_name 365 | 366 | def publish_created 367 | Rails.logger.info "Publishing employee object #{self.inspect} on the employee.created queue." 368 | ::ActivePublisher.publish("employee.created", self.to_proto.encode, "events", {}) 369 | end 370 | 371 | def publish_updated 372 | Rails.logger.info "Publishing employee object #{self.inspect} on the employee.updated queue." 373 | ::ActivePublisher.publish("employee.updated", self.to_proto.encode, "events", {}) 374 | end 375 | end 376 | ``` 377 | We'll need to add a service directory and class to listen for Active Remote messages published via NATS. 378 | 379 | _**Listing 13-21**_ Creating the app/services directory 380 | 381 | ```console 382 | $ mkdir chapter-13/active-record-publisher/app/services 383 | ``` 384 | 385 | _**Listing 13-22**_ Employee message service class 386 | 387 | ```ruby 388 | # rails-microservices-sample-code/chapter-13/active-record-publisher/app/services/employee_message_service.rb 389 | 390 | class EmployeeMessageService 391 | def search 392 | records = ::Employee.search_scope(request).map(&:to_proto) 393 | 394 | respond_with records: records 395 | end 396 | 397 | def create 398 | record = ::Employee.create(request) 399 | 400 | respond_with record 401 | end 402 | 403 | def update 404 | record = ::Employee.where(guid: request.guid).first 405 | record.assign_attributes(request) 406 | record.save! 407 | 408 | respond_with record 409 | end 410 | 411 | def destroy 412 | record = ::Employee.where(guid: request.guid).first 413 | 414 | record.delete 415 | respond_with record.to_proto 416 | end 417 | end 418 | ``` 419 | 420 | ### Create a Message Subscriber 421 | 422 | Let's create the `action-subscriber` app. It will subscribe to the employee created and updated message queues and simply log that it received a message on the queue. 423 | 424 | _**Listing 13-23**_ Starting our builder container 425 | 426 | ```console 427 | $ docker-compose -f docker-compose.builder.yml run builder bash 428 | # cd chapter-13 429 | # rails new action-subscriber --skip-active-record 430 | # cd action-subscriber 431 | # echo "gem 'action_subscriber'" >> Gemfile 432 | # echo "gem 'protobuf'" >> Gemfile 433 | # bundle 434 | # exit 435 | ``` 436 | 437 | Now let's set up Action Subscriber to listen for events. We'll need to add a `EmployeeSubscriber` class and add routes via the `ActionSubscriber.draw_routes` method. 438 | 439 | We'll want to put our subscriber classes in their own `subscribers` directory. We'll also need the `lib` directory where we'll copy our Employee Protobuf class. Let's create these directories and copy the generated `employee_message.pb.rb` file to this directory. 440 | 441 | _**Listing 13-24**_ Setting up the app/lib directory 442 | 443 | ```console 444 | $ mkdir chapter-13/action-subscriber/app/{lib,subscribers} 445 | $ cp protobuf/lib/employee_message.pb.rb chapter-13/action-subscriber/app/lib/ 446 | ``` 447 | 448 | Now let's add the subscriber class. For our the purposes of our playground we'll keep it simple - just log that we received the message. 449 | 450 | _**Listing 13-25**_ Employee subscriber class 451 | 452 | ```ruby 453 | # rails-microservices-sample-code/chapter-13/action-subscriber/app/subscribers/employee_subscriber.rb 454 | 455 | class EmployeeSubscriber < ::ActionSubscriber::Base 456 | def created 457 | Rails.logger.info "Received created message: #{EmployeeMessage.decode(payload).inspect}" 458 | end 459 | 460 | def updated 461 | Rails.logger.info "Received updated message: #{EmployeeMessage.decode(payload).inspect}" 462 | end 463 | end 464 | ``` 465 | 466 | Our app needs to know which queues to subscribe to, so we use the `default_routes_for` method which will read our `EmployeeSubscriber` class and generate queues for each of our public methods or subscribe to those queues if they already exist. The hostname `host.docker.internal` is a special Docker hostname, it points to the ip address of the host machine. 467 | 468 | _**Listing 13-26**_ Action Subscriber initializer 469 | 470 | ```ruby 471 | # rails-microservices-sample-code/chapter-13/action-subscriber/config/initializers/action_subscriber.rb 472 | 473 | ActionSubscriber.draw_routes do 474 | default_routes_for EmployeeSubscriber 475 | end 476 | 477 | ActionSubscriber.configure do |config| 478 | config.hosts = ["host.docker.internal"] 479 | config.port = 5672 480 | end 481 | ``` 482 | 483 | We'll need to enable the `cache_classes` and `eager_load` settings, the same way we did for the publisher. We'll also need to set up a logger so that we can see the log output from our Docker container. 484 | 485 | _**Listing 13-27**_ Development configuration 486 | 487 | ```ruby 488 | # rails-microservices-sample-code/chapter-13/action-subscriber/config/environments/development.rb 489 | 490 | config.cache_classes = true 491 | ... 492 | config.eager_load = true 493 | ... 494 | logger = ActiveSupport::Logger.new(STDOUT) 495 | logger.formatter = config.log_formatter 496 | config.logger = ActiveSupport::TaggedLogging.new(logger) 497 | ``` 498 | 499 | ### Create and Configure Our Environment 500 | 501 | Last but not least, let's add a `Dockerfile` and two Compose files: `docker-compose.yml` and `docker-compose-subscriber.yml`. These files will be used to build an image and spin up our Rails, NATS and RabbitMQ containers. The `Dockerfile` may already exist from the sandbox we built in chapters 9 and 12, but if not, it has the same content here. 502 | 503 | _**Listing 13-28**_ Sandbox Dockerfile 504 | 505 | ```dockerfile 506 | # rails-microservices-sample-code/Dockerfile 507 | 508 | FROM ruby:2.6.5 509 | 510 | RUN apt-get update && apt-get install -qq -y --no-install-recommends build-essential nodejs 511 | 512 | ENV INSTALL_PATH /usr/src/service 513 | ENV HOME=$INSTALL_PATH PATH=$INSTALL_PATH/bin:$PATH 514 | RUN mkdir -p $INSTALL_PATH 515 | WORKDIR $INSTALL_PATH 516 | 517 | RUN gem install rails -v 5.2.4 518 | 519 | ADD Gemfile* ./ 520 | RUN set -ex && bundle install --no-deployment 521 | ``` 522 | 523 | The following Docker Compose file includes an instance of RabbitMQ and our new `active-record-publisher` and `active-remote` Rails apps. We'll expose the `active-remote` web app on port 3002. 524 | 525 | Normally, we would add the subscriber to the same Docker Compose file, but, because the Action Subscriber service tries to connect immediately and RabbitMQ can take a few seconds to load, we'll run the subscriber process from a separate Docker Compose file. We'll also need to expose port 5672 to the host machine so we can connect from the other Compose environment. 526 | 527 | _**Listing 13-29**_ Sandbox Docker Compose file 528 | 529 | ```yml 530 | # rails-microservices-sample-code/chapter-13/docker-compose.yml 531 | # Usage: docker-compose up 532 | 533 | version: "3.4" 534 | 535 | services: 536 | active-record-publisher: 537 | environment: 538 | - PB_SERVER_TYPE=protobuf/nats/runner 539 | build: 540 | context: ./active-record-publisher 541 | dockerfile: ../../Dockerfile 542 | command: bundle exec rpc_server start -p 9399 -o active-record-publisher ./config/environment.rb 543 | volumes: 544 | - ./active-record-publisher:/usr/src/service 545 | depends_on: 546 | - nats 547 | - rabbit 548 | active-remote: 549 | environment: 550 | - PB_CLIENT_TYPE=protobuf/nats/client 551 | build: 552 | context: ./active-remote 553 | dockerfile: ../../Dockerfile 554 | command: bundle exec puma -C config/puma.rb 555 | ports: 556 | - 3002:3000 557 | volumes: 558 | - ./active-remote:/usr/src/service 559 | depends_on: 560 | - nats 561 | rabbit: 562 | image: rabbitmq:latest 563 | ports: 564 | - 5672:5672 565 | nats: 566 | image: nats:latest 567 | ``` 568 | 569 | Now let's add the `action-subscriber` configuration file. Note that because the Action Subscriber executable spawns a child process to listen for events from RabbmitMQ, we lose the log output if we start the container using the `up` command. To view all of the log info in the terminal, we'll us the Docker Compose `run` command to start a bash shell and run our `action_subscriber` executable there. 570 | 571 | _**Listing 13-30**_ Sandbox Docker Compose subscriber file 572 | 573 | ```yml 574 | # rails-microservices-sample-code/chapter-13/docker-compose-subscriber.yml 575 | # Usage: docker-compose -f docker-compose-subscriber.yml run action-subscriber bash -c 'bundle exec action_subscriber start' 576 | 577 | version: "3.4" 578 | 579 | services: 580 | action-subscriber: 581 | build: 582 | context: ./action-subscriber 583 | dockerfile: ../../Dockerfile 584 | volumes: 585 | - ./action-subscriber:/usr/src/service 586 | ``` 587 | 588 | Now that everything's in place, let's start our active-remote and active-record-publisher apps. 589 | 590 | _**Listing 13-31**_ Starting the sandbox 591 | 592 | ```console 593 | $ cd chapter-13 594 | $ docker-compose up 595 | ``` 596 | 597 | Once you see lines like this, RabbitMQ has started and the Active Publisher Rails app has successfully connected. 598 | 599 | _**Listing 13-32**_ Sandbox logging 600 | 601 | ```console 602 | rabbit_1 | 2020-02-09 22:54:02.253 [info] <0.8.0> Server startup complete; 0 plugins started. 603 | rabbit_1 | completed with 0 plugins. 604 | 72.30.0.2:5672) 605 | rabbit_1 | 2020-02-09 22:54:37.395 [info] <0.641.0> connection <0.641.0> (172.30.0.1:53140 -> 172.30.0.2:5672): user 'guest' authenticated and granted access to vhost '/' 606 | ``` 607 | 608 | Now let's start the subscriber in another terminal window. We can use the `-f` flag to specify that we want to use the `docker-compose-subscriber.yml` file. Let's run this command now. 609 | 610 | _**Listing 13-33**_ Starting the subscriber sandbox 611 | 612 | ```console 613 | $ docker-compose -f docker-compose-subscriber.yml run action-subscriber bash -c 'bundle exec action_subscriber start' 614 | ``` 615 | 616 | You should see output like the following. 617 | 618 | _**Listing 13-34**_ Subscriber sandbox logging 619 | 620 | ```console 621 | I, [2020-03-09T02:54:50.619645 #1] INFO -- : Starting server... 622 | I, [2020-03-09T02:54:50.667362 #1] INFO -- : Rabbit Hosts: ["host.docker.internal"] 623 | Rabbit Port: 5672 624 | Threadpool Size: 8 625 | Low Priority Subscriber: false 626 | Decoders: 627 | --application/json 628 | --text/plain 629 | 630 | I, [2020-03-09T02:54:50.667541 #1] INFO -- : Middlewares [ 631 | I, [2020-03-09T02:54:50.667632 #1] INFO -- : [ActionSubscriber::Middleware::ErrorHandler, [], nil] 632 | I, [2020-03-09T02:54:50.667665 #1] INFO -- : [ActionSubscriber::Middleware::Decoder, [], nil] 633 | I, [2020-03-09T02:54:50.667681 #1] INFO -- : ] 634 | I, [2020-03-09T02:54:50.667708 #1] INFO -- : EmployeeSubscriber 635 | I, [2020-03-09T02:54:50.667766 #1] INFO -- : -- method: created 636 | I, [2020-03-09T02:54:50.667818 #1] INFO -- : -- threadpool: default (8 threads) 637 | I, [2020-03-09T02:54:50.667872 #1] INFO -- : -- exchange: events 638 | I, [2020-03-09T02:54:50.668058 #1] INFO -- : -- queue: actionsubscriber.employee.created 639 | I, [2020-03-09T02:54:50.668253 #1] INFO -- : -- routing_key: employee.created 640 | I, [2020-03-09T02:54:50.668350 #1] INFO -- : -- prefetch: 2 641 | I, [2020-03-09T02:54:50.668401 #1] INFO -- : -- method: updated 642 | I, [2020-03-09T02:54:50.668603 #1] INFO -- : -- threadpool: default (8 threads) 643 | I, [2020-03-09T02:54:50.668649 #1] INFO -- : -- exchange: events 644 | I, [2020-03-09T02:54:50.668682 #1] INFO -- : -- queue: actionsubscriber.employee.updated 645 | I, [2020-03-09T02:54:50.668711 #1] INFO -- : -- routing_key: employee.updated 646 | I, [2020-03-09T02:54:50.668737 #1] INFO -- : -- prefetch: 2 647 | I, [2020-03-09T02:54:50.674268 #1] INFO -- : Action Subscriber connected 648 | ``` 649 | 650 | These log lines indicate that the subscriber has connected to the server successfully, connected to two queues and is now listening for events. 651 | 652 | Let's create an event. Open your browser and browse to http://localhost:3002/employees. Port 3002 is the port we exposed from the `active-remote` Rails app in the `docker-compose.yml` file. You should see a simple web page with the title **Employees** and a 'New Employee' link. Let's go ahead and click the link. You should now be able to create a new employee record in the web form. Once you fill it out and click the 'Create Employee' button, several things will happen. First, the form data will be sent back to the `active-remote` Rails app. The controller will pass that data on to the Active Remote Employee model, which will send the data via NATS to our `active-record-publisher` Rails app. The `active-record-publisher` app will create a new record in the SQLite database. In the `active-record-publisher` model, the `after_create` callback will run, encoding our Protobuf message and placing it on the `actionsubscriber.employee.created` queue. RabbitMQ will notify all subscribers listening on a specific queue of any new messages. Our `action-subscriber` Rails app is one such subscriber. In our `EmployeeSubscriber#created` event handler method, we wrote code to log that we received a message. If you inspect the output from the terminal window where we started the `action-subscriber` Rails app, you should see logging info similar to the output below. 653 | 654 | _**Listing 13-35**_ More subscriber sandbox logging 655 | 656 | ```console 657 | I, [2020-03-09T03:03:05.876609 #1] INFO -- : RECEIVED 35f733 from actionsubscriber.employee.created 658 | I, [2020-03-09T03:03:05.877003 #1] INFO -- : START 35f733 EmployeeSubscriber#created 659 | Received created message: # 660 | I, [2020-03-09T03:03:05.877295 #1] INFO -- : FINISHED 35f733 661 | ``` 662 | 663 | Congratulations! You have successfully built a messaging platform that can publish messages via Active Remote and apps that can respond to events. Try editing the record you just created. You should see similar output in the Action Subscriber Rails app as it receives and processes the event and the data. For even more fun, try spinning up a second or third subscriber service that listen to the same queues and watch as all of them respond simultaneously to the same published message. 664 | 665 | ## Resources 666 | 667 | * https://github.com/kevinwatson/rails-microservices-sample-code 668 | 669 | ## Wrap-up 670 | 671 | In this chapter, we have built three Rails applications that share data using patterns that provide both real-time and event-driven communications. Each pattern has its place. Sometimes communication needs to be real-time, such as when a microservice needs access to one or more records. Other times, requests between services doesn't need to be real-time, but event-driven, such as when a service needs to be notified and a long-running process needs to be started, but the caller doesn't need to know when the process has finished. Examining your business requirements can help you determine when each is necessary. 672 | 673 | This chapter wraps up our journey of building microservices using Ruby on Rails. We've covered a lot of ground. In our next and final chapter, we'll summarize what we've learned so far. We'll also discuss a few ways to continue our journey. 674 | 675 | [Next >>](150-chapter-14.md) 676 | --------------------------------------------------------------------------------