├── .gitignore ├── CODEOWNERS ├── CODE_OF_CONDUCT.md ├── LICENSE ├── README.md ├── SECURITY.md ├── go ├── README.md ├── common │ └── common.go ├── examples │ ├── publish │ │ └── main.go │ ├── publishStream │ │ └── main.go │ ├── schema │ │ └── main.go │ ├── subscribe │ │ └── main.go │ └── topic │ │ └── main.go ├── go.mod ├── go.sum ├── grpcclient │ └── grpcclient.go ├── oauth │ └── oauth.go └── proto │ ├── pubsub_api.pb.go │ └── pubsub_api_grpc.pb.go ├── java ├── README.md ├── pom.xml ├── run.sh └── src │ └── main │ ├── java │ ├── accountupdateapp │ │ ├── AccountListener.java │ │ ├── AccountUpdateAppUtil.java │ │ ├── AccountUpdater.java │ │ └── README.md │ ├── genericpubsub │ │ ├── GetSchema.java │ │ ├── GetTopic.java │ │ ├── ManagedSubscribe.java │ │ ├── Publish.java │ │ ├── PublishStream.java │ │ └── Subscribe.java │ └── utility │ │ ├── APISessionCredentials.java │ │ ├── CommonContext.java │ │ ├── EventParser.java │ │ ├── ExampleConfigurations.java │ │ ├── SessionTokenService.java │ │ └── XClientTraceIdClientInterceptor.java │ ├── proto │ └── pubsub_api.proto │ └── resources │ ├── arguments.yaml │ └── logback.xml ├── pubsub_api.proto └── python ├── InventoryAppExample ├── InventoryApp.py ├── PubSub.py ├── README.md └── SalesforceListener.py └── util ├── ChangeEventHeaderUtility.py └── README.md /.gitignore: -------------------------------------------------------------------------------- 1 | # MacOS system files 2 | .DS_Store 3 | 4 | # Windows system files 5 | Thumbs.db 6 | ehthumbs.db 7 | [Dd]esktop.ini 8 | $RECYCLE.BIN/ 9 | 10 | # Eclipse project files 11 | .project 12 | .idea 13 | 14 | # Dependency directory for Go examples 15 | go/vendor/ 16 | 17 | */target/ 18 | 19 | # Python compiled files 20 | *.pyo 21 | *.pyc 22 | -------------------------------------------------------------------------------- /CODEOWNERS: -------------------------------------------------------------------------------- 1 | # Comment line immediately above ownership line is reserved for related gus information. Please be careful while editing. 2 | #ECCN:Open Source 3 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Salesforce Open Source Community Code of Conduct 2 | 3 | ## About the Code of Conduct 4 | 5 | Equality is a core value at Salesforce. We believe a diverse and inclusive 6 | community fosters innovation and creativity, and are committed to building a 7 | culture where everyone feels included. 8 | 9 | Salesforce open-source projects are committed to providing a friendly, safe, and 10 | welcoming environment for all, regardless of gender identity and expression, 11 | sexual orientation, disability, physical appearance, body size, ethnicity, nationality, 12 | race, age, religion, level of experience, education, socioeconomic status, or 13 | other similar personal characteristics. 14 | 15 | The goal of this code of conduct is to specify a baseline standard of behavior so 16 | that people with different social values and communication styles can work 17 | together effectively, productively, and respectfully in our open source community. 18 | It also establishes a mechanism for reporting issues and resolving conflicts. 19 | 20 | All questions and reports of abusive, harassing, or otherwise unacceptable behavior 21 | in a Salesforce open-source project may be reported by contacting the Salesforce 22 | Open Source Conduct Committee at ossconduct@salesforce.com. 23 | 24 | ## Our Pledge 25 | 26 | In the interest of fostering an open and welcoming environment, we as 27 | contributors and maintainers pledge to making participation in our project and 28 | our community a harassment-free experience for everyone, regardless of gender 29 | identity and expression, sexual orientation, disability, physical appearance, 30 | body size, ethnicity, nationality, race, age, religion, level of experience, education, 31 | socioeconomic status, or other similar personal characteristics. 32 | 33 | ## Our Standards 34 | 35 | Examples of behavior that contributes to creating a positive environment 36 | include: 37 | 38 | - Using welcoming and inclusive language 39 | - Being respectful of differing viewpoints and experiences 40 | - Gracefully accepting constructive criticism 41 | - Focusing on what is best for the community 42 | - Showing empathy toward other community members 43 | 44 | Examples of unacceptable behavior by participants include: 45 | 46 | - The use of sexualized language or imagery and unwelcome sexual attention or 47 | advances 48 | - Personal attacks, insulting/derogatory comments, or trolling 49 | - Public or private harassment 50 | - Publishing, or threatening to publish, others' private information—such as 51 | a physical or electronic address—without explicit permission 52 | - Other conduct which could reasonably be considered inappropriate in a 53 | professional setting 54 | - Advocating for or encouraging any of the above behaviors 55 | 56 | ## Our Responsibilities 57 | 58 | Project maintainers are responsible for clarifying the standards of acceptable 59 | behavior and are expected to take appropriate and fair corrective action in 60 | response to any instances of unacceptable behavior. 61 | 62 | Project maintainers have the right and responsibility to remove, edit, or 63 | reject comments, commits, code, wiki edits, issues, and other contributions 64 | that are not aligned with this Code of Conduct, or to ban temporarily or 65 | permanently any contributor for other behaviors that they deem inappropriate, 66 | threatening, offensive, or harmful. 67 | 68 | ## Scope 69 | 70 | This Code of Conduct applies both within project spaces and in public spaces 71 | when an individual is representing the project or its community. Examples of 72 | representing a project or community include using an official project email 73 | address, posting via an official social media account, or acting as an appointed 74 | representative at an online or offline event. Representation of a project may be 75 | further defined and clarified by project maintainers. 76 | 77 | ## Enforcement 78 | 79 | Instances of abusive, harassing, or otherwise unacceptable behavior may be 80 | reported by contacting the Salesforce Open Source Conduct Committee 81 | at ossconduct@salesforce.com. All complaints will be reviewed and investigated 82 | and will result in a response that is deemed necessary and appropriate to the 83 | circumstances. The committee is obligated to maintain confidentiality with 84 | regard to the reporter of an incident. Further details of specific enforcement 85 | policies may be posted separately. 86 | 87 | Project maintainers who do not follow or enforce the Code of Conduct in good 88 | faith may face temporary or permanent repercussions as determined by other 89 | members of the project's leadership and the Salesforce Open Source Conduct 90 | Committee. 91 | 92 | ## Attribution 93 | 94 | This Code of Conduct is adapted from the [Contributor Covenant][contributor-covenant-home], 95 | version 1.4, available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html. 96 | It includes adaptions and additions from [Go Community Code of Conduct][golang-coc], 97 | [CNCF Code of Conduct][cncf-coc], and [Microsoft Open Source Code of Conduct][microsoft-coc]. 98 | 99 | This Code of Conduct is licensed under the [Creative Commons Attribution 3.0 License][cc-by-3-us]. 100 | 101 | [contributor-covenant-home]: https://www.contributor-covenant.org 'https://www.contributor-covenant.org/' 102 | [golang-coc]: https://golang.org/conduct 103 | [cncf-coc]: https://github.com/cncf/foundation/blob/master/code-of-conduct.md 104 | [microsoft-coc]: https://opensource.microsoft.com/codeofconduct/ 105 | [cc-by-3-us]: https://creativecommons.org/licenses/by/3.0/us/ 106 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Creative Commons Legal Code 2 | 3 | CC0 1.0 Universal 4 | 5 | CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE 6 | LEGAL SERVICES. DISTRIBUTION OF THIS DOCUMENT DOES NOT CREATE AN 7 | ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS 8 | INFORMATION ON AN "AS-IS" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES 9 | REGARDING THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS 10 | PROVIDED HEREUNDER, AND DISCLAIMS LIABILITY FOR DAMAGES RESULTING FROM 11 | THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS PROVIDED 12 | HEREUNDER. 13 | 14 | Statement of Purpose 15 | 16 | The laws of most jurisdictions throughout the world automatically confer 17 | exclusive Copyright and Related Rights (defined below) upon the creator 18 | and subsequent owner(s) (each and all, an "owner") of an original work of 19 | authorship and/or a database (each, a "Work"). 20 | 21 | Certain owners wish to permanently relinquish those rights to a Work for 22 | the purpose of contributing to a commons of creative, cultural and 23 | scientific works ("Commons") that the public can reliably and without fear 24 | of later claims of infringement build upon, modify, incorporate in other 25 | works, reuse and redistribute as freely as possible in any form whatsoever 26 | and for any purposes, including without limitation commercial purposes. 27 | These owners may contribute to the Commons to promote the ideal of a free 28 | culture and the further production of creative, cultural and scientific 29 | works, or to gain reputation or greater distribution for their Work in 30 | part through the use and efforts of others. 31 | 32 | For these and/or other purposes and motivations, and without any 33 | expectation of additional consideration or compensation, the person 34 | associating CC0 with a Work (the "Affirmer"), to the extent that he or she 35 | is an owner of Copyright and Related Rights in the Work, voluntarily 36 | elects to apply CC0 to the Work and publicly distribute the Work under its 37 | terms, with knowledge of his or her Copyright and Related Rights in the 38 | Work and the meaning and intended legal effect of CC0 on those rights. 39 | 40 | 1. Copyright and Related Rights. A Work made available under CC0 may be 41 | protected by copyright and related or neighboring rights ("Copyright and 42 | Related Rights"). Copyright and Related Rights include, but are not 43 | limited to, the following: 44 | 45 | i. the right to reproduce, adapt, distribute, perform, display, 46 | communicate, and translate a Work; 47 | ii. moral rights retained by the original author(s) and/or performer(s); 48 | iii. publicity and privacy rights pertaining to a person's image or 49 | likeness depicted in a Work; 50 | iv. rights protecting against unfair competition in regards to a Work, 51 | subject to the limitations in paragraph 4(a), below; 52 | v. rights protecting the extraction, dissemination, use and reuse of data 53 | in a Work; 54 | vi. database rights (such as those arising under Directive 96/9/EC of the 55 | European Parliament and of the Council of 11 March 1996 on the legal 56 | protection of databases, and under any national implementation 57 | thereof, including any amended or successor version of such 58 | directive); and 59 | vii. other similar, equivalent or corresponding rights throughout the 60 | world based on applicable law or treaty, and any national 61 | implementations thereof. 62 | 63 | 2. Waiver. To the greatest extent permitted by, but not in contravention 64 | of, applicable law, Affirmer hereby overtly, fully, permanently, 65 | irrevocably and unconditionally waives, abandons, and surrenders all of 66 | Affirmer's Copyright and Related Rights and associated claims and causes 67 | of action, whether now known or unknown (including existing as well as 68 | future claims and causes of action), in the Work (i) in all territories 69 | worldwide, (ii) for the maximum duration provided by applicable law or 70 | treaty (including future time extensions), (iii) in any current or future 71 | medium and for any number of copies, and (iv) for any purpose whatsoever, 72 | including without limitation commercial, advertising or promotional 73 | purposes (the "Waiver"). Affirmer makes the Waiver for the benefit of each 74 | member of the public at large and to the detriment of Affirmer's heirs and 75 | successors, fully intending that such Waiver shall not be subject to 76 | revocation, rescission, cancellation, termination, or any other legal or 77 | equitable action to disrupt the quiet enjoyment of the Work by the public 78 | as contemplated by Affirmer's express Statement of Purpose. 79 | 80 | 3. Public License Fallback. Should any part of the Waiver for any reason 81 | be judged legally invalid or ineffective under applicable law, then the 82 | Waiver shall be preserved to the maximum extent permitted taking into 83 | account Affirmer's express Statement of Purpose. In addition, to the 84 | extent the Waiver is so judged Affirmer hereby grants to each affected 85 | person a royalty-free, non transferable, non sublicensable, non exclusive, 86 | irrevocable and unconditional license to exercise Affirmer's Copyright and 87 | Related Rights in the Work (i) in all territories worldwide, (ii) for the 88 | maximum duration provided by applicable law or treaty (including future 89 | time extensions), (iii) in any current or future medium and for any number 90 | of copies, and (iv) for any purpose whatsoever, including without 91 | limitation commercial, advertising or promotional purposes (the 92 | "License"). The License shall be deemed effective as of the date CC0 was 93 | applied by Affirmer to the Work. Should any part of the License for any 94 | reason be judged legally invalid or ineffective under applicable law, such 95 | partial invalidity or ineffectiveness shall not invalidate the remainder 96 | of the License, and in such case Affirmer hereby affirms that he or she 97 | will not (i) exercise any of his or her remaining Copyright and Related 98 | Rights in the Work or (ii) assert any associated claims and causes of 99 | action with respect to the Work, in either case contrary to Affirmer's 100 | express Statement of Purpose. 101 | 102 | 4. Limitations and Disclaimers. 103 | 104 | a. No trademark or patent rights held by Affirmer are waived, abandoned, 105 | surrendered, licensed or otherwise affected by this document. 106 | b. Affirmer offers the Work as-is and makes no representations or 107 | warranties of any kind concerning the Work, express, implied, 108 | statutory or otherwise, including without limitation warranties of 109 | title, merchantability, fitness for a particular purpose, non 110 | infringement, or the absence of latent or other defects, accuracy, or 111 | the present or absence of errors, whether or not discoverable, all to 112 | the greatest extent permissible under applicable law. 113 | c. Affirmer disclaims responsibility for clearing rights of other persons 114 | that may apply to the Work or any use thereof, including without 115 | limitation any person's Copyright and Related Rights in the Work. 116 | Further, Affirmer disclaims responsibility for obtaining any necessary 117 | consents, permissions or other rights required for any use of the 118 | Work. 119 | d. Affirmer understands and acknowledges that Creative Commons is not a 120 | party to this document and has no duty or obligation with respect to 121 | this CC0 or use of the Work. 122 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Getting Started with the Pub/Sub API 2 | 3 | - [About Pub/Sub API](#about-pubsub-api) 4 | - [gRPC](#grpc) 5 | - [Documentation and Blog Posts](#documentation-and-blog-post) 6 | - [Code Samples](#code-samples-from-salesforce) 7 | 8 | ## About Pub/Sub API 9 | Welcome to Pub/Sub API! Pub/Sub API provides a single interface for publishing and subscribing to platform events, including real-time event monitoring events, and change data capture events. Based on [gRPC](https://grpc.io/docs/what-is-grpc/introduction/) and HTTP/2, Pub/Sub API enables efficient delivery of binary event messages in the Apache Avro format. 10 | 11 | This repo contains the critical [proto 12 | file](https://github.com/developerforce/pub-sub-api/blob/main/pubsub_api.proto) that you will need to use the API. 13 | 14 | ## gRPC 15 | gRPC [officially supports 11 languages](https://grpc.io/docs/languages/), but 16 | there is unofficial community support in more. To encode and decode events, an 17 | Avro library for your language of choice will be needed. See below for which 18 | officially supported languages have well-supported Avro libraries: 19 | 20 | |Supported gRPC Language|Avro Libraries| 21 | |-----------------------|--------------| 22 | |C# | [AvroConvert](https://github.com/AdrianStrugala/AvroConvert)
[Apache Avro C#](https://avro.apache.org/docs/current/api/csharp/html/index.html) (docs are not great)| 23 | |C++|[Apache Avro C++](https://avro.apache.org/docs/current/api/cpp/html/index.html)| 24 | |Dart|[avro-dart](https://github.com/sqs/avro-dart) (last updated 2012)| 25 | |Go|[goavro](https://github.com/linkedin/goavro)| 26 | |Java|[Apache Avro Java](https://avro.apache.org/docs/current/getting-started-java/)| 27 | |Kotlin|[avro4k](https://github.com/avro-kotlin/avro4k)| 28 | |Node|[avro-js](https://www.npmjs.com/package/avro-js)| 29 | |Objective C|[ObjectiveAvro](https://github.com/jlawton/ObjectiveAvro) (but read [this](https://stackoverflow.com/questions/57216446/data-serialisation-in-objective-c-avro-alternative))| 30 | |PHP|[avro-php](https://github.com/wikimedia/avro-php)| 31 | |Python|[Apache Avro Python](https://avro.apache.org/docs/current/getting-started-python/)| 32 | |Ruby|[AvroTurf](https://github.com/dasch/avro_turf)| 33 | 34 | ## Documentation, Blog Post and Videos 35 | - [Pub/Sub API Developer Guide](https://developer.salesforce.com/docs/platform/pub-sub-api/overview) 36 | - [Salesforce Architects Blog Post](https://medium.com/salesforce-architects/announcing-pub-sub-api-generally-available-3980c9eaf0b7) 37 | - [Introducing the New gRPC-based Pub Sub API YouTube Developer Quick Takes](https://youtu.be/g9P87_loVVA) 38 | 39 | ## Code Samples from Salesforce 40 | Salesforce provides these samples for demonstration purposes. They aren't meant to be used in production code. Before you use these samples in production, make sure you perform thorough functional and performance testing. 41 | - [Java Quick Start in the Developer Guide](https://developer.salesforce.com/docs/platform/pub-sub-api/guide/qs-java-quick-start.html) 42 | - [Python Quick Start in the Developer Guide](https://developer.salesforce.com/docs/platform/pub-sub-api/guide/qs-python-quick-start.html) 43 | - [Python Code Examples](python/) 44 | - [Go Code Examples](go/) 45 | - [Java Code Examples](java/) 46 | 47 | ## Code Samples from the Developer Community 48 | These examples are developed by the community. They aren't supported by Salesforce. Use at your own discretion. 49 | - [E-Bikes Sample Application](https://github.com/trailheadapps/ebikes-lwc) 50 | - [Pub/Sub API Node Client](https://github.com/pozil/pub-sub-api-node-client) 51 | - [.NET Code Examples](https://github.com/Meyce/pub-sub-api/tree/main/.Net) 52 | - [Ruby Pub/Sub API Example](https://github.com/RenoFi/salesforce-pub-sub-rb) 53 | 54 | If you have a code sample for Pub/Sub API that you would like to add a link to in this section, submit a PR with the modified readme page. We don't guarantee that we can link to all samples. Priority will be given to samples implemented in a programming language that is not represented in this repository's samples. 55 | -------------------------------------------------------------------------------- /SECURITY.md: -------------------------------------------------------------------------------- 1 | ## Security 2 | 3 | Please report any security issue to [security@salesforce.com](mailto:security@salesforce.com) 4 | as soon as it is discovered. These libraries limit their runtime dependencies in 5 | order to reduce the total cost of ownership as much as can be, but all consumers 6 | should remain vigilant and have their security stakeholders review all third-party 7 | products (3PP) like these examples and their dependencies. 8 | -------------------------------------------------------------------------------- /go/README.md: -------------------------------------------------------------------------------- 1 | # Purpose 2 | This directory contains some simple Go examples that can be used with Pub/Sub API. The examples show the general flow for each of the remote procedure calls (RPCs) supported by Pub/Sub API. Note that the examples are provided as a learning resource to demonstrate how to use the RPC methods in the Go language. They weren't performance tested and aren't production ready. See the `Limitations` section below for more details. 3 | 4 | # Project Structure 5 | * `common/` - This directory contains a set of common variables that are shared between all examples. Before running any of the examples, manually update these variables to ensure you're passing the correct username, password, etc. See the `Running the Examples` section below for instructions on how to prepare this file. 6 | * `examples/` - This directory contains standalone examples for each RPC method defined in the proto file. When running an example you will use one of the `main.go` files nested in this directory as your entry point. In general, each example creates a gRPC client, fetches the necessary authentication token, and then calls the necessary RPC method(s) to demonstrate usage. 7 | * `grpcclient/` - This directory contains a wrapper struct that is used by the examples to interact with the gRPC server. The majority of the logic for these examples lives in this wrapper struct. 8 | * `oauth/` - This directory contains basic helper functions to fetch an OAuth token and user data. 9 | * `proto/` - This directory contains the Go-specific `*.pb.go` files that are generated from the `pubsub_api.proto` file found at the root of 10 | this project. The `*.pb.go` files contain helper functions and structs that will be used by the Go examples to interact with Pub/Sub API. 11 | 12 | # Running the Examples 13 | ## Prerequisites 14 | 1. Install [Go](https://go.dev/doc/install). 15 | 2. Clone this project. 16 | 3. Run `go mod vendor` from the `go` directory to fetch dependencies. 17 | - NOTE: At this time the `vendor` directory has not been committed to this repo so you need to manually run the `go mod vendor` command to fetch dependencies. If the `vendor` directory is committed to the repo in the future, this step can be skipped. 18 | 4. Use a Salesforce org. Get the username, password, and login URL. 19 | 5. Create a custom CarMaintenance [platform event](https://developer.salesforce.com/docs/atlas.en-us.platform_events.meta/platform_events/platform_events_define_ui.htm) in your Salesforce org. Ensure your CarMaintenance platform event matches the following structure: 20 | - Standard Fields 21 | - Label: CarMaintenance 22 | - Plural Label: CarMaintenances 23 | - Custom Fields 24 | - Cost (Number) 25 | - Mileage (Number) 26 | - WorkDescription (Text, 200) 27 | 6. Create a connected app and enable OAuth in your Salesforce org. See [Quick Start Prerequisites](https://developer.salesforce.com/docs/atlas.en-us.api_rest.meta/api_rest/quickstart_prereq.htm) in the REST API Developer Guide. Save the consumer key and consumer secret. 28 | 29 | ## Execution 30 | 1. Update the relevant variables in the `common/common.go` file. These configs will apply to all examples. 31 | * `` - Fill this value in with the consumer key you received after setting up OAuth for your Salesforce org. 32 | * `` - Fill this value in with the consumer secret you received after setting up OAuth for your Salesforce org. 33 | * `` - Fill this value in with the username for your Salesforce org. 34 | * `` - Fill this value in with the password for your Salesforce org. 35 | * `` - Fill this value in with the login URL for your Salesforce org. 36 | 2. Choose an example to run from the `examples` directory. Each subdirectory is named after the RPC call flow it demonstrates and you can run the corresponding example with a `go run` command. For example, if you want to run the example for the GetTopic RPC, run `go run examples/topic/main.go` from the `go` directory. 37 | 38 | ## Subscription Options and Configuration 39 | Pub/Sub API allows clients using the `Subscribe` RPC to specify one of the following subscription replay options: 40 | * `LATEST` 41 | * `CUSTOM` 42 | * `EARLIEST` 43 | 44 | For a description of these replay options, refer to the "Replaying an Event Stream" section in the [Pub/Sub API docs](https://developer.salesforce.com/docs/platform/pub-sub-api/references/methods/subscribe-rpc.html#replaying-an-event-stream). 45 | 46 | The subscribe example in this repo is compatible with all three options outlined above. To set a subscription option, update the `ReplayPreset` and `ReplayId` options in the `common.go` file. See the following examples: 47 | * `LATEST` 48 | - set `ReplayPreset = proto.ReplayPreset_LATEST` 49 | - set `ReplayId []byte = nil` 50 | * `CUSTOM` 51 | - set `ReplayPreset = proto.ReplayPreset_CUSTOM` 52 | - set `ReplayId = []byte{0, 0, 0, 0, 0, 37, 132, 136}` 53 | - Note that this is just an example ReplayId intended to show the appropriate formatting. You need to specify a valid ReplayId in the same format demonstrated here. 54 | * `EARLIEST` 55 | - set `ReplayPreset = proto.ReplayPreset_EARLIEST` 56 | - set `ReplayId []byte = nil` 57 | 58 | # Creating Your Own Project 59 | ## Prerequisites 60 | 1. Install protoc, protoc-gen-go, and protoc-gen-go-grpc and ensure the binaries are available in your PATH. See the [gRPC quick start docs](https://grpc.io/docs/languages/go/quickstart/#prerequisites) for more info. 61 | 62 | ## Generating the *.pb.go Files 63 | 1. Fetch the most current [proto file](https://github.com/developerforce/pub-sub-api/blob/main/pubsub_api.proto) and copy it to your own project. 64 | 2. Modify the `option go_package` line to match your desired import path. 65 | 3. Generate the Go files from the proto file with the `protoc` command. As an example, if you copied the proto file to `proto/pubsub_api.proto` then you can generate the `*.pb.go` files by running the following command from the root of your project: 66 | ```bash 67 | protoc --go_out=. --go_opt=paths=source_relative \ 68 | --go-grpc_out=. --go-grpc_opt=paths=source_relative \ 69 | proto/pubsub_api.proto 70 | ``` 71 | 72 | ## Implementation 73 | This repo can be used as a reference point for clients looking to create a Go app to integrate with Pub/Sub API. Note that the project structure and the examples included in this repo are intended for demo purposes; clients are free to implement their own Go apps in any way they see fit. A client may want to consider whether or not the error handling, retries, etc. included in these examples will be appropriate for their specific use case. 74 | 75 | # Limitations 76 | 1. No support for auth token refreshes - At some point the authentication token will expire. At this time, these examples do not handle re-authentication. 77 | 2. No guarantees that streams will remain open - Pub/Sub API has idle timeouts and will close idle streams. If a stream is closed while running these examples, you will most likely need to stop and restart. 78 | * NOTE: This point primarily applies to the `Subscribe` and `PublishStream` RPC examples as these RPCs rely on long-lived streams. 79 | 3. No support for republishing on error - If an error occurs while publishing the relevant examples will surface the error but will not attempt to republish the event. 80 | 4. No security guarantees - Teams using these examples for reference will need to do their own security audits to ensure the dependencies introduced here can be safely used. 81 | 5. No performance testing - These examples have not been perf tested. 82 | 6. No timeouts for streaming calls - The `Subscribe` and `PublishStream` RPC examples do not have timeouts configured. -------------------------------------------------------------------------------- /go/common/common.go: -------------------------------------------------------------------------------- 1 | package common 2 | 3 | import ( 4 | "time" 5 | 6 | "github.com/developerforce/pub-sub-api/go/proto" 7 | ) 8 | 9 | var ( 10 | // topic and subscription-related variables 11 | TopicName = "/event/CarMaintenance__e" 12 | ReplayPreset = proto.ReplayPreset_EARLIEST 13 | ReplayId []byte = nil 14 | Appetite int32 = 5 15 | 16 | // gRPC server variables 17 | GRPCEndpoint = "api.pubsub.salesforce.com:7443" 18 | GRPCDialTimeout = 5 * time.Second 19 | GRPCCallTimeout = 5 * time.Second 20 | 21 | // OAuth header variables 22 | GrantType = "password" 23 | ClientId = "" 24 | ClientSecret = "" 25 | Username = "" 26 | Password = "" 27 | 28 | // OAuth server variables 29 | OAuthEndpoint = "" 30 | OAuthDialTimeout = 5 * time.Second 31 | ) 32 | -------------------------------------------------------------------------------- /go/examples/publish/main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "log" 5 | 6 | "github.com/developerforce/pub-sub-api/go/common" 7 | "github.com/developerforce/pub-sub-api/go/grpcclient" 8 | ) 9 | 10 | func main() { 11 | log.Printf("Creating gRPC client...") 12 | client, err := grpcclient.NewGRPCClient() 13 | if err != nil { 14 | log.Fatalf("could not create gRPC client: %v", err) 15 | } 16 | defer client.Close() 17 | 18 | log.Printf("Populating auth token...") 19 | err = client.Authenticate() 20 | if err != nil { 21 | client.Close() 22 | log.Fatalf("could not authenticate: %v", err) 23 | } 24 | 25 | log.Printf("Populating user info...") 26 | err = client.FetchUserInfo() 27 | if err != nil { 28 | client.Close() 29 | log.Fatalf("could not fetch user info: %v", err) 30 | } 31 | 32 | log.Printf("Making GetTopic request...") 33 | topic, err := client.GetTopic() 34 | if err != nil { 35 | client.Close() 36 | log.Fatalf("could not fetch topic: %v", err) 37 | } 38 | 39 | if !topic.GetCanPublish() { 40 | client.Close() 41 | log.Fatalf("this user is not allowed to publish to the following topic: %s", common.TopicName) 42 | } 43 | 44 | log.Printf("Making GetSchema request...") 45 | schema, err := client.GetSchema(topic.GetSchemaId()) 46 | if err != nil { 47 | client.Close() 48 | log.Fatalf("could not fetch schema: %v", err) 49 | } 50 | 51 | err = client.Publish(schema) 52 | if err != nil { 53 | client.Close() 54 | log.Fatalf("could not publish event: %v", err) 55 | } 56 | 57 | log.Printf("successfully published event") 58 | } 59 | -------------------------------------------------------------------------------- /go/examples/publishStream/main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "log" 5 | 6 | "github.com/developerforce/pub-sub-api/go/common" 7 | "github.com/developerforce/pub-sub-api/go/grpcclient" 8 | ) 9 | 10 | func main() { 11 | log.Printf("Creating gRPC client...") 12 | client, err := grpcclient.NewGRPCClient() 13 | if err != nil { 14 | log.Fatalf("could not create gRPC client: %v", err) 15 | } 16 | defer client.Close() 17 | 18 | log.Printf("Populating auth token...") 19 | err = client.Authenticate() 20 | if err != nil { 21 | client.Close() 22 | log.Fatalf("could not authenticate: %v", err) 23 | } 24 | 25 | log.Printf("Populating user info...") 26 | err = client.FetchUserInfo() 27 | if err != nil { 28 | client.Close() 29 | log.Fatalf("could not fetch user info: %v", err) 30 | } 31 | 32 | log.Printf("Making GetTopic request...") 33 | topic, err := client.GetTopic() 34 | if err != nil { 35 | client.Close() 36 | log.Fatalf("could not fetch topic: %v", err) 37 | } 38 | 39 | if !topic.GetCanPublish() { 40 | client.Close() 41 | log.Fatalf("this user is not allowed to publish to the following topic: %s", common.TopicName) 42 | } 43 | 44 | log.Printf("Making GetSchema request...") 45 | schema, err := client.GetSchema(topic.GetSchemaId()) 46 | if err != nil { 47 | client.Close() 48 | log.Fatalf("could not fetch schema: %v", err) 49 | } 50 | 51 | err = client.PublishStream(schema) 52 | if err != nil { 53 | client.Close() 54 | log.Fatalf("could not publish events: %v", err) 55 | } 56 | } 57 | -------------------------------------------------------------------------------- /go/examples/schema/main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "log" 5 | 6 | "github.com/developerforce/pub-sub-api/go/grpcclient" 7 | ) 8 | 9 | func main() { 10 | log.Printf("Creating gRPC client...") 11 | client, err := grpcclient.NewGRPCClient() 12 | if err != nil { 13 | log.Fatalf("could not create gRPC client: %v", err) 14 | } 15 | defer client.Close() 16 | 17 | log.Printf("Populating auth token...") 18 | err = client.Authenticate() 19 | if err != nil { 20 | client.Close() 21 | log.Fatalf("could not authenticate: %v", err) 22 | } 23 | 24 | log.Printf("Populating user info...") 25 | err = client.FetchUserInfo() 26 | if err != nil { 27 | client.Close() 28 | log.Fatalf("could not fetch user info: %v", err) 29 | } 30 | 31 | log.Printf("Making GetTopic request...") 32 | topic, err := client.GetTopic() 33 | if err != nil { 34 | client.Close() 35 | log.Fatalf("could not fetch topic: %v", err) 36 | } 37 | 38 | log.Printf("Making GetSchema request...") 39 | schema, err := client.GetSchema(topic.GetSchemaId()) 40 | if err != nil { 41 | client.Close() 42 | log.Fatalf("could not fetch schema: %v", err) 43 | } 44 | 45 | log.Printf("schema info: %+v", schema) 46 | } 47 | -------------------------------------------------------------------------------- /go/examples/subscribe/main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "log" 5 | 6 | "github.com/developerforce/pub-sub-api/go/common" 7 | "github.com/developerforce/pub-sub-api/go/grpcclient" 8 | "github.com/developerforce/pub-sub-api/go/proto" 9 | ) 10 | 11 | func main() { 12 | if common.ReplayPreset == proto.ReplayPreset_CUSTOM && common.ReplayId == nil { 13 | log.Fatalf("the replayId variable must be populated when the replayPreset variable is set to CUSTOM") 14 | } else if common.ReplayPreset != proto.ReplayPreset_CUSTOM && common.ReplayId != nil { 15 | log.Fatalf("the replayId variable must not be populated when the replayPreset variable is set to EARLIEST or LATEST") 16 | } 17 | 18 | log.Printf("Creating gRPC client...") 19 | client, err := grpcclient.NewGRPCClient() 20 | if err != nil { 21 | log.Fatalf("could not create gRPC client: %v", err) 22 | } 23 | defer client.Close() 24 | 25 | log.Printf("Populating auth token...") 26 | err = client.Authenticate() 27 | if err != nil { 28 | client.Close() 29 | log.Fatalf("could not authenticate: %v", err) 30 | } 31 | 32 | log.Printf("Populating user info...") 33 | err = client.FetchUserInfo() 34 | if err != nil { 35 | client.Close() 36 | log.Fatalf("could not fetch user info: %v", err) 37 | } 38 | 39 | log.Printf("Making GetTopic request...") 40 | topic, err := client.GetTopic() 41 | if err != nil { 42 | client.Close() 43 | log.Fatalf("could not fetch topic: %v", err) 44 | } 45 | 46 | if !topic.GetCanSubscribe() { 47 | client.Close() 48 | log.Fatalf("this user is not allowed to subscribe to the following topic: %s", common.TopicName) 49 | } 50 | 51 | curReplayId := common.ReplayId 52 | for { 53 | log.Printf("Subscribing to topic...") 54 | 55 | // use the user-provided ReplayPreset by default, but if the curReplayId variable has a non-nil value then assume that we want to 56 | // consume from a custom offset. The curReplayId will have a non-nil value if the user explicitly set the ReplayId or if a previous 57 | // subscription attempt successfully processed at least one event before crashing 58 | replayPreset := common.ReplayPreset 59 | if curReplayId != nil { 60 | replayPreset = proto.ReplayPreset_CUSTOM 61 | } 62 | 63 | // In the happy path the Subscribe method should never return, it will just process events indefinitely. In the unhappy path 64 | // (i.e., an error occurred) the Subscribe method will return both the most recently processed ReplayId as well as the error message. 65 | // The error message will be logged for the user to see and then we will attempt to re-subscribe with the ReplayId on the next iteration 66 | // of this for loop 67 | curReplayId, err = client.Subscribe(replayPreset, curReplayId) 68 | if err != nil { 69 | log.Printf("error occurred while subscribing to topic: %v", err) 70 | } 71 | } 72 | } 73 | -------------------------------------------------------------------------------- /go/examples/topic/main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "log" 5 | 6 | "github.com/developerforce/pub-sub-api/go/grpcclient" 7 | ) 8 | 9 | func main() { 10 | log.Printf("Creating gRPC client...") 11 | client, err := grpcclient.NewGRPCClient() 12 | if err != nil { 13 | log.Fatalf("could not create gRPC client: %v", err) 14 | } 15 | defer client.Close() 16 | 17 | log.Printf("Populating auth token...") 18 | err = client.Authenticate() 19 | if err != nil { 20 | client.Close() 21 | log.Fatalf("could not authenticate: %v", err) 22 | } 23 | 24 | log.Printf("Populating user info...") 25 | err = client.FetchUserInfo() 26 | if err != nil { 27 | client.Close() 28 | log.Fatalf("could not fetch user info: %v", err) 29 | } 30 | 31 | log.Printf("Making GetTopic request...") 32 | topic, err := client.GetTopic() 33 | if err != nil { 34 | client.Close() 35 | log.Fatalf("could not fetch topic: %v", err) 36 | } 37 | 38 | log.Printf("topic info: %v", topic) 39 | } 40 | -------------------------------------------------------------------------------- /go/go.mod: -------------------------------------------------------------------------------- 1 | module github.com/developerforce/pub-sub-api/go 2 | 3 | go 1.18 4 | 5 | require ( 6 | github.com/linkedin/goavro/v2 v2.11.0 7 | google.golang.org/grpc v1.37.0 8 | google.golang.org/protobuf v1.26.0 9 | ) 10 | 11 | require ( 12 | github.com/golang/protobuf v1.5.0 // indirect 13 | github.com/golang/snappy v0.0.1 // indirect 14 | golang.org/x/net v0.0.0-20190311183353-d8887717615a // indirect 15 | golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a // indirect 16 | golang.org/x/text v0.3.0 // indirect 17 | google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013 // indirect 18 | ) 19 | -------------------------------------------------------------------------------- /go/go.sum: -------------------------------------------------------------------------------- 1 | cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= 2 | github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= 3 | github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU= 4 | github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw= 5 | github.com/cncf/udpa/go v0.0.0-20201120205902-5459f2c99403/go.mod h1:WmhPx2Nbnhtbo57+VJT5O0JRkEi1Wbu0z5j0R8u5Hbk= 6 | github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 7 | github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4= 8 | github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4= 9 | github.com/envoyproxy/go-control-plane v0.9.9-0.20210217033140-668b12f5399d/go.mod h1:cXg6YxExXjJnVBQHBLXeUAgxn2UodCpnH306RInaBQk= 10 | github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c= 11 | github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= 12 | github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= 13 | github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= 14 | github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= 15 | github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8= 16 | github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA= 17 | github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs= 18 | github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w= 19 | github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0= 20 | github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QDs8UjoX8= 21 | github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI= 22 | github.com/golang/protobuf v1.5.0 h1:LUVKkCeviFUMKqHa4tXIIij/lbhnMbP7Fn5wKdKkRh4= 23 | github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk= 24 | github.com/golang/snappy v0.0.1 h1:Qgr9rKW7uDUkrbSmQeiDsGa8SjGyCOGtuasMWwvp2P4= 25 | github.com/golang/snappy v0.0.1/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q= 26 | github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M= 27 | github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= 28 | github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= 29 | github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= 30 | github.com/google/go-cmp v0.5.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= 31 | github.com/google/go-cmp v0.5.5 h1:Khx7svrCpmxxtHBq5j2mp/xVjsi8hQMfNLvJFAlrGgU= 32 | github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= 33 | github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= 34 | github.com/linkedin/goavro/v2 v2.11.0 h1:AlU/NR32ESbC/dlzbhTjyqybwESupUCc3SrrHg2qdTg= 35 | github.com/linkedin/goavro/v2 v2.11.0/go.mod h1:UgQUb2N/pmueQYH9bfqFioWxzYCZXSfF8Jw03O5sjqA= 36 | github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= 37 | github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= 38 | github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= 39 | github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA= 40 | golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= 41 | golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= 42 | golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= 43 | golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU= 44 | golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= 45 | golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= 46 | golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= 47 | golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= 48 | golang.org/x/net v0.0.0-20190311183353-d8887717615a h1:oWX7TPOiFAMXLq8o0ikBYfCJVlRHBcsciT5bXOrH628= 49 | golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= 50 | golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= 51 | golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 52 | golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 53 | golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 54 | golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 55 | golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a h1:1BGLXjeY4akVXGgbC9HugT3Jv3hCI0z56oJR5vAMgBU= 56 | golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 57 | golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg= 58 | golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= 59 | golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= 60 | golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY= 61 | golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= 62 | golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= 63 | golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4= 64 | golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 65 | google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= 66 | google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= 67 | google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc= 68 | google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc= 69 | google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013 h1:+kGHl1aib/qcwaRi1CbqBZ1rk19r85MNUf8HaBghugY= 70 | google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo= 71 | google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c= 72 | google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg= 73 | google.golang.org/grpc v1.25.1/go.mod h1:c3i+UQWmh7LiEpx4sFZnkU36qjEYZ0imhYfXVyQciAY= 74 | google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk= 75 | google.golang.org/grpc v1.37.0 h1:uSZWeQJX5j11bIQ4AJoj+McDBo29cY1MCoC1wO3ts+c= 76 | google.golang.org/grpc v1.37.0/go.mod h1:NREThFqKR1f3iQ6oBuvc5LadQuXVGo9rkm5ZGrQdJfM= 77 | google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8= 78 | google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0= 79 | google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM= 80 | google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE= 81 | google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo= 82 | google.golang.org/protobuf v1.22.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= 83 | google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= 84 | google.golang.org/protobuf v1.23.1-0.20200526195155-81db48ad09cc/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= 85 | google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlbajtzgsN7c= 86 | google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw= 87 | google.golang.org/protobuf v1.26.0 h1:bxAC2xTBsZGibn2RTntX0oH50xLsqy1OxA9tTL3p/lk= 88 | google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= 89 | gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= 90 | gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= 91 | honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= 92 | honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= 93 | -------------------------------------------------------------------------------- /go/grpcclient/grpcclient.go: -------------------------------------------------------------------------------- 1 | package grpcclient 2 | 3 | import ( 4 | "context" 5 | "crypto/x509" 6 | "fmt" 7 | "io" 8 | "log" 9 | "sync" 10 | "time" 11 | 12 | "github.com/developerforce/pub-sub-api/go/common" 13 | "github.com/developerforce/pub-sub-api/go/oauth" 14 | "github.com/developerforce/pub-sub-api/go/proto" 15 | "github.com/linkedin/goavro/v2" 16 | "google.golang.org/grpc" 17 | "google.golang.org/grpc/credentials" 18 | "google.golang.org/grpc/credentials/insecure" 19 | "google.golang.org/grpc/metadata" 20 | ) 21 | 22 | const ( 23 | tokenHeader = "accesstoken" 24 | instanceHeader = "instanceurl" 25 | tenantHeader = "tenantid" 26 | ) 27 | 28 | type PubSubClient struct { 29 | accessToken string 30 | instanceURL string 31 | 32 | userID string 33 | orgID string 34 | 35 | conn *grpc.ClientConn 36 | pubSubClient proto.PubSubClient 37 | 38 | schemaCache map[string]*goavro.Codec 39 | } 40 | 41 | // Closes the underlying connection to the gRPC server 42 | func (c *PubSubClient) Close() { 43 | c.conn.Close() 44 | } 45 | 46 | // Makes a call to the OAuth server to fetch credentials. Credentials are stored as part of the PubSubClient object so that they can be 47 | // referenced later in other methods 48 | func (c *PubSubClient) Authenticate() error { 49 | resp, err := oauth.Login() 50 | if err != nil { 51 | return err 52 | } 53 | 54 | c.accessToken = resp.AccessToken 55 | c.instanceURL = resp.InstanceURL 56 | 57 | return nil 58 | } 59 | 60 | // Makes a call to the OAuth server to fetch user info. User info is stored as part of the PubSubClient object so that it can be referenced 61 | // later in other methods 62 | func (c *PubSubClient) FetchUserInfo() error { 63 | resp, err := oauth.UserInfo(c.accessToken) 64 | if err != nil { 65 | return err 66 | } 67 | 68 | c.userID = resp.UserID 69 | c.orgID = resp.OrganizationID 70 | 71 | return nil 72 | } 73 | 74 | // Wrapper function around the GetTopic RPC. This will add the OAuth credentials and make a call to fetch data about a specific topic 75 | func (c *PubSubClient) GetTopic() (*proto.TopicInfo, error) { 76 | var trailer metadata.MD 77 | 78 | req := &proto.TopicRequest{ 79 | TopicName: common.TopicName, 80 | } 81 | 82 | ctx, cancelFn := context.WithTimeout(c.getAuthContext(), common.GRPCCallTimeout) 83 | defer cancelFn() 84 | 85 | resp, err := c.pubSubClient.GetTopic(ctx, req, grpc.Trailer(&trailer)) 86 | printTrailer(trailer) 87 | 88 | if err != nil { 89 | return nil, err 90 | } 91 | 92 | return resp, nil 93 | } 94 | 95 | // Wrapper function around the GetSchema RPC. This will add the OAuth credentials and make a call to fetch data about a specific schema 96 | func (c *PubSubClient) GetSchema(schemaId string) (*proto.SchemaInfo, error) { 97 | var trailer metadata.MD 98 | 99 | req := &proto.SchemaRequest{ 100 | SchemaId: schemaId, 101 | } 102 | 103 | ctx, cancelFn := context.WithTimeout(c.getAuthContext(), common.GRPCCallTimeout) 104 | defer cancelFn() 105 | 106 | resp, err := c.pubSubClient.GetSchema(ctx, req, grpc.Trailer(&trailer)) 107 | printTrailer(trailer) 108 | 109 | if err != nil { 110 | return nil, err 111 | } 112 | 113 | return resp, nil 114 | } 115 | 116 | // Wrapper function around the Subscribe RPC. This will add the OAuth credentials and create a separate streaming client that will be used to 117 | // fetch data from the topic. This method will continuously consume messages unless an error occurs; if an error does occur then this method will 118 | // return the last successfully consumed ReplayId as well as the error message. If no messages were successfully consumed then this method will return 119 | // the same ReplayId that it originally received as a parameter 120 | func (c *PubSubClient) Subscribe(replayPreset proto.ReplayPreset, replayId []byte) ([]byte, error) { 121 | ctx, cancelFn := context.WithCancel(c.getAuthContext()) 122 | defer cancelFn() 123 | 124 | subscribeClient, err := c.pubSubClient.Subscribe(ctx) 125 | if err != nil { 126 | return replayId, err 127 | } 128 | defer subscribeClient.CloseSend() 129 | 130 | initialFetchRequest := &proto.FetchRequest{ 131 | TopicName: common.TopicName, 132 | ReplayPreset: replayPreset, 133 | NumRequested: common.Appetite, 134 | } 135 | if replayPreset == proto.ReplayPreset_CUSTOM && replayId != nil { 136 | initialFetchRequest.ReplayId = replayId 137 | } 138 | 139 | err = subscribeClient.Send(initialFetchRequest) 140 | // If the Send call returns an EOF error then print a log message but do not return immediately. Instead, let the Recv call (below) determine 141 | // if there's a more specific error that can be returned 142 | // See the SendMsg description at https://pkg.go.dev/google.golang.org/grpc#ClientStream 143 | if err == io.EOF { 144 | log.Printf("WARNING - EOF error returned from initial Send call, proceeding anyway") 145 | } else if err != nil { 146 | return replayId, err 147 | } 148 | 149 | requestedEvents := initialFetchRequest.NumRequested 150 | 151 | // NOTE: the replayId should be stored in a persistent data store rather than being stored in a variable 152 | curReplayId := replayId 153 | for { 154 | log.Printf("Waiting for events...") 155 | resp, err := subscribeClient.Recv() 156 | if err == io.EOF { 157 | printTrailer(subscribeClient.Trailer()) 158 | return curReplayId, fmt.Errorf("stream closed") 159 | } else if err != nil { 160 | printTrailer(subscribeClient.Trailer()) 161 | return curReplayId, err 162 | } 163 | 164 | for _, event := range resp.Events { 165 | codec, err := c.fetchCodec(event.GetEvent().GetSchemaId()) 166 | if err != nil { 167 | return curReplayId, err 168 | } 169 | 170 | parsed, _, err := codec.NativeFromBinary(event.GetEvent().GetPayload()) 171 | if err != nil { 172 | return curReplayId, err 173 | } 174 | 175 | body, ok := parsed.(map[string]interface{}) 176 | if !ok { 177 | return curReplayId, fmt.Errorf("error casting parsed event: %v", body) 178 | } 179 | 180 | // Again, this should be stored in a persistent external datastore instead of a variable 181 | curReplayId = event.GetReplayId() 182 | 183 | log.Printf("event body: %+v\n", body) 184 | 185 | // decrement our counter to keep track of how many events have been requested but not yet processed. If we're below our configured 186 | // batch size then proactively request more events to stay ahead of the processor 187 | requestedEvents-- 188 | if requestedEvents < common.Appetite { 189 | log.Printf("Sending next FetchRequest...") 190 | fetchRequest := &proto.FetchRequest{ 191 | TopicName: common.TopicName, 192 | NumRequested: common.Appetite, 193 | } 194 | 195 | err = subscribeClient.Send(fetchRequest) 196 | // If the Send call returns an EOF error then print a log message but do not return immediately. Instead, let the Recv call (above) determine 197 | // if there's a more specific error that can be returned 198 | // See the SendMsg description at https://pkg.go.dev/google.golang.org/grpc#ClientStream 199 | if err == io.EOF { 200 | log.Printf("WARNING - EOF error returned from subsequent Send call, proceeding anyway") 201 | } else if err != nil { 202 | return curReplayId, err 203 | } 204 | 205 | requestedEvents += fetchRequest.NumRequested 206 | } 207 | } 208 | } 209 | } 210 | 211 | // Unexported helper function to retrieve the cached codec from the PubSubClient's schema cache. If the schema ID is not found in the cache 212 | // then a GetSchema call is made and the corresponding codec is cached for future use 213 | func (c *PubSubClient) fetchCodec(schemaId string) (*goavro.Codec, error) { 214 | codec, ok := c.schemaCache[schemaId] 215 | if ok { 216 | log.Printf("Fetched cached codec...") 217 | return codec, nil 218 | } 219 | 220 | log.Printf("Making GetSchema request for uncached schema...") 221 | schema, err := c.GetSchema(schemaId) 222 | if err != nil { 223 | return nil, err 224 | } 225 | 226 | log.Printf("Creating codec from uncached schema...") 227 | codec, err = goavro.NewCodec(schema.GetSchemaJson()) 228 | if err != nil { 229 | return nil, err 230 | } 231 | 232 | c.schemaCache[schemaId] = codec 233 | 234 | return codec, nil 235 | } 236 | 237 | // Wrapper function around the Publish RPC. This will add the OAuth credentials and produce a single hardcoded event to the specified topic. 238 | func (c *PubSubClient) Publish(schema *proto.SchemaInfo) error { 239 | log.Printf("Creating codec from schema...") 240 | codec, err := goavro.NewCodec(schema.SchemaJson) 241 | if err != nil { 242 | return err 243 | } 244 | 245 | sampleEvent := map[string]interface{}{ 246 | "CreatedDate": time.Now().Unix(), 247 | "CreatedById": c.userID, 248 | "Mileage__c": goavro.Union("double", 95443.0), 249 | "Cost__c": goavro.Union("double", 99.40), 250 | "WorkDescription__c": goavro.Union("string", "Replaced front brakes"), 251 | } 252 | 253 | payload, err := codec.BinaryFromNative(nil, sampleEvent) 254 | if err != nil { 255 | return err 256 | } 257 | 258 | var trailer metadata.MD 259 | 260 | req := &proto.PublishRequest{ 261 | TopicName: common.TopicName, 262 | Events: []*proto.ProducerEvent{ 263 | { 264 | SchemaId: schema.GetSchemaId(), 265 | Payload: payload, 266 | }, 267 | }, 268 | } 269 | 270 | ctx, cancelFn := context.WithTimeout(c.getAuthContext(), common.GRPCCallTimeout) 271 | defer cancelFn() 272 | 273 | pubResp, err := c.pubSubClient.Publish(ctx, req, grpc.Trailer(&trailer)) 274 | printTrailer(trailer) 275 | 276 | if err != nil { 277 | return err 278 | } 279 | 280 | result := pubResp.GetResults() 281 | if result == nil { 282 | return fmt.Errorf("nil result returned when publishing to %s", common.TopicName) 283 | } 284 | 285 | if err := result[0].GetError(); err != nil { 286 | return fmt.Errorf(result[0].GetError().GetMsg()) 287 | } 288 | 289 | return nil 290 | } 291 | 292 | // Wrapper function around the PublishStream RPC. This will add the OAuth credentials and produce an event to the topic every five seconds 293 | func (c *PubSubClient) PublishStream(schema *proto.SchemaInfo) error { 294 | log.Printf("Creating codec from schema...") 295 | codec, err := goavro.NewCodec(schema.SchemaJson) 296 | if err != nil { 297 | return err 298 | } 299 | 300 | ctx, cancelFn := context.WithCancel(c.getAuthContext()) 301 | defer cancelFn() 302 | 303 | publishClient, err := c.pubSubClient.PublishStream(ctx) 304 | if err != nil { 305 | return err 306 | } 307 | 308 | sampleEvent := map[string]interface{}{ 309 | "CreatedDate": time.Now().Unix(), 310 | "CreatedById": c.userID, 311 | "Mileage__c": goavro.Union("double", 95443.0), 312 | "Cost__c": goavro.Union("double", 99.40), 313 | "WorkDescription__c": goavro.Union("string", "Replaced front brakes"), 314 | } 315 | 316 | payload, err := codec.BinaryFromNative(nil, sampleEvent) 317 | if err != nil { 318 | return err 319 | } 320 | 321 | publishRequest := &proto.PublishRequest{ 322 | TopicName: common.TopicName, 323 | Events: []*proto.ProducerEvent{ 324 | { 325 | SchemaId: schema.GetSchemaId(), 326 | Payload: payload, 327 | }, 328 | }, 329 | } 330 | 331 | err = publishClient.Send(publishRequest) 332 | // If the Send call returns an EOF error then print a log message but do not return immediately. Instead, let the Recv call (below) determine 333 | // if there's a more specific error that can be returned 334 | // See the SendMsg description at https://pkg.go.dev/google.golang.org/grpc#ClientStream 335 | if err == io.EOF { 336 | log.Printf("WARNING - EOF error returned from initial Send call, proceeding anyway") 337 | } else if err != nil { 338 | return err 339 | } 340 | 341 | log.Printf("Entering event loop...") 342 | 343 | var resErrMutex sync.Mutex 344 | var resErr error 345 | 346 | shutdownGoroutine := func(err error) { 347 | cancelFn() 348 | 349 | resErrMutex.Lock() 350 | defer resErrMutex.Unlock() 351 | 352 | // only capture the first error returned 353 | if resErr == nil { 354 | resErr = err 355 | } 356 | } 357 | 358 | wg := sync.WaitGroup{} 359 | wg.Add(2) 360 | 361 | // sender goroutine. This goroutine will attempt to publish a new event every 5 seconds. This goroutine will run until one of the following 362 | // conditions is met: 363 | // 1. the receiver goroutine returned an error and exited 364 | // 2. this goroutine encounters an error while publishing 365 | go func() { 366 | defer wg.Done() 367 | defer publishClient.CloseSend() 368 | 369 | for { 370 | select { 371 | case <-ctx.Done(): 372 | return 373 | default: 374 | time.Sleep(5 * time.Second) 375 | 376 | log.Printf("Sending next PublishRequest...") 377 | sampleEvent["CreatedDate"] = time.Now().Unix() 378 | 379 | payload, sendErr := codec.BinaryFromNative(nil, sampleEvent) 380 | if sendErr != nil { 381 | shutdownGoroutine(sendErr) 382 | return 383 | } 384 | 385 | publishRequest := &proto.PublishRequest{ 386 | TopicName: common.TopicName, 387 | Events: []*proto.ProducerEvent{ 388 | { 389 | SchemaId: schema.GetSchemaId(), 390 | Payload: payload, 391 | }, 392 | }, 393 | } 394 | 395 | sendErr = publishClient.Send(publishRequest) 396 | // if we encounter an EOF error from the Send method then exit this goroutine without canceling the context or recording the error. 397 | // The Recv method called in the receiver goroutine may return a more specific error explaining why the stream was closed. 398 | // See the SendMsg description at https://pkg.go.dev/google.golang.org/grpc#ClientStream 399 | if sendErr == io.EOF { 400 | log.Printf("WARNING - EOF error returned from subsequent Send call, proceeding anyway") 401 | return 402 | } else if sendErr != nil { 403 | shutdownGoroutine(sendErr) 404 | return 405 | } 406 | } 407 | } 408 | }() 409 | 410 | // receiver goroutine. This goroutine will attempt to receive the PublishStream responses as they are sent back from the Pub/Sub API. This 411 | // goroutine will run until one of the following conditions is met: 412 | // 1. the sender goroutine returned an error and exited 413 | // 2. this goroutine either encounters an error while receiving or the PublishStream response indicates an error occurred while publishing 414 | go func() { 415 | defer wg.Done() 416 | 417 | for { 418 | select { 419 | case <-ctx.Done(): 420 | return 421 | default: 422 | pubResp, recvErr := publishClient.Recv() 423 | if recvErr == io.EOF { 424 | printTrailer(publishClient.Trailer()) 425 | shutdownGoroutine(fmt.Errorf("stream closed")) 426 | return 427 | } else if recvErr != nil { 428 | printTrailer(publishClient.Trailer()) 429 | shutdownGoroutine(recvErr) 430 | return 431 | } 432 | 433 | results := pubResp.GetResults() 434 | if results == nil { 435 | shutdownGoroutine(fmt.Errorf("nil results received")) 436 | return 437 | } 438 | 439 | for _, res := range results { 440 | if res.GetError() != nil { 441 | shutdownGoroutine(fmt.Errorf(res.GetError().GetMsg())) 442 | return 443 | } 444 | } 445 | 446 | log.Printf("successfully published event") 447 | } 448 | } 449 | }() 450 | 451 | wg.Wait() 452 | 453 | return resErr 454 | } 455 | 456 | // Returns a new context with the necessary authentication parameters for the gRPC server 457 | func (c *PubSubClient) getAuthContext() context.Context { 458 | return metadata.NewOutgoingContext(context.Background(), metadata.Pairs( 459 | tokenHeader, c.accessToken, 460 | instanceHeader, c.instanceURL, 461 | tenantHeader, c.orgID, 462 | )) 463 | } 464 | 465 | // Creates a new connection to the gRPC server and returns the wrapper struct 466 | func NewGRPCClient() (*PubSubClient, error) { 467 | dialOpts := []grpc.DialOption{ 468 | grpc.WithBlock(), 469 | } 470 | 471 | if common.GRPCEndpoint == "localhost:7011" { 472 | dialOpts = append(dialOpts, grpc.WithTransportCredentials(insecure.NewCredentials())) 473 | } else { 474 | certs := getCerts() 475 | creds := credentials.NewClientTLSFromCert(certs, "") 476 | dialOpts = append(dialOpts, grpc.WithTransportCredentials(creds)) 477 | } 478 | 479 | ctx, cancelFn := context.WithTimeout(context.Background(), common.GRPCDialTimeout) 480 | defer cancelFn() 481 | 482 | conn, err := grpc.DialContext(ctx, common.GRPCEndpoint, dialOpts...) 483 | if err != nil { 484 | return nil, err 485 | } 486 | 487 | return &PubSubClient{ 488 | conn: conn, 489 | pubSubClient: proto.NewPubSubClient(conn), 490 | schemaCache: make(map[string]*goavro.Codec), 491 | }, nil 492 | } 493 | 494 | // Fetches system certs and returns them if possible. If unable to fetch system certs then an empty cert pool is returned instead 495 | func getCerts() *x509.CertPool { 496 | if certs, err := x509.SystemCertPool(); err == nil { 497 | return certs 498 | } 499 | 500 | return x509.NewCertPool() 501 | } 502 | 503 | // Helper function to display trailers on the console in a more readable format 504 | func printTrailer(trailer metadata.MD) { 505 | if len(trailer) == 0 { 506 | log.Printf("no trailers returned") 507 | return 508 | } 509 | 510 | log.Printf("beginning of trailers") 511 | for key, val := range trailer { 512 | log.Printf("[trailer] = %s, [value] = %s", key, val) 513 | } 514 | log.Printf("end of trailers") 515 | } 516 | -------------------------------------------------------------------------------- /go/oauth/oauth.go: -------------------------------------------------------------------------------- 1 | package oauth 2 | 3 | import ( 4 | "context" 5 | "encoding/json" 6 | "fmt" 7 | "net/http" 8 | "net/url" 9 | "strings" 10 | 11 | "github.com/developerforce/pub-sub-api/go/common" 12 | ) 13 | 14 | const ( 15 | loginEndpoint = "/services/oauth2/token" 16 | userInfoEndpoint = "/services/oauth2/userinfo" 17 | ) 18 | 19 | type LoginResponse struct { 20 | AccessToken string `json:"access_token"` 21 | InstanceURL string `json:"instance_url"` 22 | ID string `json:"id"` 23 | TokenType string `json:"token_type"` 24 | IssuedAt string `json:"issued_at"` 25 | Signature string `json:"signature"` 26 | } 27 | 28 | type UserInfoResponse struct { 29 | UserID string `json:"user_id"` 30 | OrganizationID string `json:"organization_id"` 31 | } 32 | 33 | func Login() (*LoginResponse, error) { 34 | body := url.Values{} 35 | body.Set("grant_type", common.GrantType) 36 | body.Set("client_id", common.ClientId) 37 | body.Set("client_secret", common.ClientSecret) 38 | body.Set("username", common.Username) 39 | body.Set("password", common.Password) 40 | 41 | ctx, cancelFn := context.WithTimeout(context.Background(), common.OAuthDialTimeout) 42 | defer cancelFn() 43 | 44 | req, err := http.NewRequestWithContext(ctx, http.MethodPost, common.OAuthEndpoint+loginEndpoint, strings.NewReader(body.Encode())) 45 | if err != nil { 46 | return nil, err 47 | } 48 | 49 | req.Header.Set("Content-Type", "application/x-www-form-urlencoded") 50 | 51 | httpResp, err := http.DefaultClient.Do(req) 52 | if err != nil { 53 | return nil, err 54 | } 55 | 56 | defer httpResp.Body.Close() 57 | 58 | if httpResp.StatusCode != http.StatusOK { 59 | return nil, fmt.Errorf("non-200 status code returned on OAuth authentication call: %v", httpResp.StatusCode) 60 | } 61 | 62 | var loginResponse LoginResponse 63 | err = json.NewDecoder(httpResp.Body).Decode(&loginResponse) 64 | if err != nil { 65 | return nil, err 66 | } 67 | 68 | return &loginResponse, nil 69 | } 70 | 71 | func UserInfo(accessToken string) (*UserInfoResponse, error) { 72 | ctx, cancelFn := context.WithTimeout(context.Background(), common.OAuthDialTimeout) 73 | defer cancelFn() 74 | 75 | req, err := http.NewRequestWithContext(ctx, http.MethodGet, common.OAuthEndpoint+userInfoEndpoint, nil) 76 | if err != nil { 77 | return nil, err 78 | } 79 | 80 | req.Header.Set("Authorization", fmt.Sprintf("Bearer %s", accessToken)) 81 | 82 | httpResp, err := http.DefaultClient.Do(req) 83 | if err != nil { 84 | return nil, err 85 | } 86 | 87 | defer httpResp.Body.Close() 88 | 89 | if httpResp.StatusCode != http.StatusOK { 90 | return nil, fmt.Errorf("non-200 status code returned on OAuth user info call: %v", httpResp.StatusCode) 91 | } 92 | 93 | var userInfoResponse UserInfoResponse 94 | err = json.NewDecoder(httpResp.Body).Decode(&userInfoResponse) 95 | if err != nil { 96 | return nil, err 97 | } 98 | 99 | return &userInfoResponse, nil 100 | } 101 | -------------------------------------------------------------------------------- /java/README.md: -------------------------------------------------------------------------------- 1 | # Pub/Sub API Java Examples 2 | 3 | ## Overview 4 | This directory contains some Java examples that can be used with the Pub/Sub API. These examples range from generic Publish, Subscribe, ManagedSubscribe (beta), processing CustomEventHeaders in change events, and a specific example of updating the Salesforce Account standard object. It is important to note that these examples are not performance tested nor are they production ready. They are meant to be used as a learning resource or a starting point to understand the flows of each of the Remote Procedure Calls (RPCs) of Pub/Sub API. There are some limitations to these examples as well mentioned below. 5 | 6 | ## Project Structure 7 | In the `src/main` directory of the project, you will find several sub-directories as follows: 8 | * `java/`: This directory contains the main source code for all the examples grouped into separate packages: 9 | * `accountupdateapp/`: This package contains the examples for updating an Account standard object with an AccountNumber. 10 | * `genericpubsub/`: This package contains the examples covering the general flows of all RPCs of Pub/Sub API. 11 | * `processchangeeventheader/`: This package contains an example for extracting the changed fields from a bitmap value in a change event. 12 | * `utility`: This package contains a list of utility classes used across all the examples. 13 | * `proto/` - This directory contains the same `pubsub_api.proto` file found at the root of this repo. The plugin used to generate the sources requires for this proto file to be present in the `src` directory. 14 | * `resources/` - This directory contains a list of resources needed for running the examples. 15 | 16 | ## Running the Examples 17 | ### Prerequisites 18 | 1. Install [Java 11](https://www.oracle.com/java/technologies/javase/jdk11-archive-downloads.html), [Maven](https://maven.apache.org/install.html). 19 | 2. Clone this project. 20 | 3. Run `mvn clean install` from the `java` directory to build the project and generate required sources from the proto file. 21 | 4. The `arguments.yaml` file in the `src/main/resources` sub-directory contains a list of required and optional configurations needed to run the examples. The file contains detailed comments on how to set the configurations. 22 | 5. Get the username, password, and login URL of the Salesforce org you wish to use. 23 | 6. For the examples in `genericpubsub` package, a custom **_Order Event_** [platform event](https://developer.salesforce.com/docs/atlas.en-us.platform_events.meta/platform_events/platform_events_define_ui.htm) has to be created in the Salesforce org. Ensure your `Order Event` platform event matches the following structure: 24 | - Standard Fields 25 | - Label: `Order Event` 26 | - Plural Label: `Order Events` 27 | - Custom Fields 28 | - `Order Number` (Text, 18) 29 | - `City` (Text, 50) 30 | - `Amount` (Number, (16,2)) 31 | 7. For the examples in the `accountupdateapp` package, another custom **_NewAccount_** [platform event](https://developer.salesforce.com/docs/atlas.en-us.platform_events.meta/platform_events/platform_events_define_ui.htm) has to be created in the Salesforce org. [More info here](src/main/java/accountupdateapp/README.md). 32 | 33 | ### Execution 34 | 1. Update the configurations in the `src/main/resources/arguments.yaml` file. The required configurations will apply to all the examples and the optional ones depends on which example is being executed. The configurations include: 35 | 1. Required configurations: 36 | * `PUBSUB_HOST`: Specify the Pub/Sub API endpoint to be used. 37 | * `PUBSUB_PORT`: Specify the Pub/Sub API port to be used (usually 7443). 38 | * `LOGIN_URL`: Specify the login url of the Salesforce org being used to run the examples. 39 | * `USERNAME` & `PASSWORD`: For authentication via username and password, you will need to specify the username and password of the Salesforce org. 40 | * `ACCESS_TOKEN` & `TENANT_ID`: For authentication via session token and tenant ID, you will need to specify the sessionToken and tenant ID of the Salesforce org. 41 | * When using managed event subscriptions (beta), one of these configurations is required. 42 | * `MANAGED_SUB_DEVELOPER_NAME`: Specify the developer name of ManagedEventSubscription. This parameter is used in ManagedSubscribe.java. 43 | * `MANAGED_SUB_ID`: Specify the ID of the ManagedEventSubscription Tooling API record. This parameter is used in ManagedSubscribe.java. 44 | 45 | 2. Optional Parameters: 46 | * `TOPIC`: Specify the topic for which you wish to publish/subscribe. 47 | * `NUMBER_OF_EVENTS_TO_PUBLISH`: Specify the number of events to publish while using the PublishStream RPC. 48 | * `SINGLE_PUBLISH_REQUEST`: Specify if you want to publish the events in a single or multiple PublishRequests. 49 | * `NUMBER_OF_EVENTS_IN_FETCHREQUEST`: Specify the number of events that the Subscribe RPC requests from the server in each FetchRequest. The example fetches at most 5 events in each Subscribe request. If you pass in more than 5, it sends multiple Subscribe requests with at most 5 events requested in FetchRequest each. For more information about requesting events, see [Pull Subscription and Flow Control](https://developer.salesforce.com/docs/platform/pub-sub-api/guide/flow-control.html) in the Pub/Sub API documentation. 50 | * `PROCESS_CHANGE_EVENT_HEADER_FIELDS`: Specify whether the Subscribe or ManagedSubscribe client should process the change data capture event bitmap fields in `ChangeEventHeader`. In this sample, only the `changedFields` field is expanded. To expand the `diffFields` and `nulledFields` header fields, modify the sample code. See [Event Deserialization Considerations](https://developer.salesforce.com/docs/platform/pub-sub-api/guide/event-deserialization-considerations.html). 51 | * `REPLAY_PRESET`: Specify the ReplayPreset for subscribe examples. 52 | * If a subscription has to be started using the CUSTOM replay preset, the `REPLAY_ID` parameter is mandatory. 53 | * The `REPLAY_ID` is a byte array and must be specified in this format: `[]`. Please enter the values as is within the square brackets and without any quotes. 54 | * Example: `[0, 1, 2, 3, 4, -5, 6, 7, -8]` 55 | 56 | 2. After setting up the configurations, any example can be executed using the `./run.sh` file available at the parent directory. 57 | * Format for running the examples: `./run.sh .` 58 | * Example: `./run.sh genericpubsub.PublishStream` 59 | 60 | ## Implementation 61 | - This repo can be used as a reference point for clients looking to create a Java app to integrate with Pub/Sub API. Note that the project structure and the examples included in this repo are intended for demo purposes only and clients are free to implement their own Java apps in any way they see fit. 62 | - The Generic Subscribe and ManagedSubscribe (beta) RPC examples create a long-lived subscription. After all requested events are received, Subscribe sends a new `FetchRequest` and ManagedSubscribe sends a new `ManagedFetchRequest` to keep the subscription alive and the client listening to new events. 63 | - The Generic Subscribe and ManagedSubscribe (beta) RPC examples demonstrate a basic flow control strategy where a new `FetchRequest` or `ManagedFetchRequest` is sent only after the requested number of events in the previous requests are received. The ManagedSubscribe RPC example also shows how to commit a Replay ID by sending commit requests. Custom flow control strategies can be implemented as needed. More info on flow control available [here](https://developer.salesforce.com/docs/platform/pub-sub-api/guide/flow-control.html). 64 | - The Generic Subscribe RPC example demonstrates error handling. After an exception occurs, it attempts to resubscribe after the last received event by implementing Binary Exponential Backoff. The example processes events and sends the retry requests asynchronously. If the error is an invalid replay ID, it tries to resubscribe since the earliest stored event in the event bus. See the `onError()` method in `Subscribe.java`. 65 | 66 | # Limitations 67 | 1. No guarantees that streams will remain open with `PublishStream` examples - Pub/Sub API has idle timeouts and will close idle streams. If a stream is closed while running these examples, you will most likely need to stop and restart. 68 | 2. No support for republishing on error - If an error occurs while publishing the relevant examples will surface the error but will not attempt to republish the event. 69 | 3. No security guarantees - Teams using these examples for reference will need to do their own security audits to ensure the dependencies introduced here can be safely used. 70 | 4. No performance testing - These examples have not been perf tested. 71 | -------------------------------------------------------------------------------- /java/pom.xml: -------------------------------------------------------------------------------- 1 | 2 | 5 | 4.0.0 6 | 7 | org.example 8 | pubsub-java 9 | 1.0-SNAPSHOT 10 | 11 | 12 | 11 13 | 11 14 | 11 15 | 1.11.0 16 | 1.64.0 17 | 3.25.3 18 | 1.64.0 19 | 20 | 21 | 22 | 23 | javax.annotation 24 | javax.annotation-api 25 | 1.3.2 26 | 27 | 28 | io.grpc 29 | grpc-netty 30 | ${grpc.version} 31 | 32 | 33 | io.grpc 34 | grpc-protobuf 35 | ${grpc.version} 36 | 37 | 38 | io.grpc 39 | grpc-stub 40 | ${grpc.version} 41 | 42 | 43 | ch.qos.logback 44 | logback-classic 45 | 1.2.9 46 | 47 | 48 | org.apache.avro 49 | avro 50 | ${avro.version} 51 | 52 | 53 | commons-cli 54 | commons-cli 55 | 1.5.0 56 | 57 | 58 | org.eclipse.jetty 59 | jetty-client 60 | 10.0.10 61 | 62 | 63 | org.yaml 64 | snakeyaml 65 | 1.21 66 | 67 | 68 | com.googlecode.json-simple 69 | json-simple 70 | 1.1.1 71 | 72 | 73 | 74 | 75 | 76 | 77 | kr.motd.maven 78 | os-maven-plugin 79 | 1.4.1.Final 80 | 81 | 82 | 83 | 84 | org.xolstice.maven.plugins 85 | protobuf-maven-plugin 86 | 0.5.0 87 | 88 | com.google.protobuf:protoc:${protoc.version}:exe:${os.detected.classifier} 89 | grpc-java 90 | io.grpc:protoc-gen-grpc-java:${protocJava.version}:exe:${os.detected.classifier} 91 | 92 | 93 | 94 | 95 | compile 96 | compile-custom 97 | 98 | 99 | 100 | 101 | 102 | org.apache.maven.plugins 103 | maven-shade-plugin 104 | 3.3.0 105 | 106 | 107 | package 108 | 109 | shade 110 | 111 | 112 | false 113 | 114 | 115 | 116 | 117 | 118 | 119 | 120 | 121 | 122 | 123 | -------------------------------------------------------------------------------- /java/run.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # 3 | # A convenience script that runs examples based on locally built JARs. Usage: 4 | # mvn clean package 5 | # ./run.sh 6 | # 7 | 8 | EXAMPLE=$1 9 | if [ "x$EXAMPLE" = "x" ]; then 10 | echo "Please specify one of the example class names from the package com.salesforce.eventbusclient.example" 11 | exit -1 12 | fi 13 | java -cp target/pubsub-java-1.0-SNAPSHOT.jar $EXAMPLE $@ -------------------------------------------------------------------------------- /java/src/main/java/accountupdateapp/AccountListener.java: -------------------------------------------------------------------------------- 1 | package accountupdateapp; 2 | 3 | import static accountupdateapp.AccountUpdateAppUtil.*; 4 | import static utility.CommonContext.*; 5 | 6 | import java.io.IOException; 7 | 8 | import org.apache.avro.Schema; 9 | import org.apache.avro.generic.GenericRecord; 10 | import org.slf4j.Logger; 11 | import org.slf4j.LoggerFactory; 12 | 13 | import com.salesforce.eventbus.protobuf.ConsumerEvent; 14 | import com.salesforce.eventbus.protobuf.FetchResponse; 15 | 16 | import genericpubsub.Publish; 17 | import genericpubsub.Subscribe; 18 | import io.grpc.stub.StreamObserver; 19 | import utility.CommonContext; 20 | import utility.ExampleConfigurations; 21 | 22 | /** 23 | * AccountListener 24 | * A subscribe client that listens to the Change Data Capture (CDC) events of the Account object 25 | * and publishes events of the `/event/NewAccount__e` custom platform event. 26 | * 27 | * Example: 28 | * ./run.sh accountupdateapp.AccountListener 29 | * 30 | * @author sidd0610 31 | * @since v1.0 32 | */ 33 | 34 | public class AccountListener { 35 | 36 | protected static final Logger logger = LoggerFactory.getLogger(AccountListener.class.getClass()); 37 | 38 | protected Subscribe subscriber; 39 | protected Publish publisher; 40 | 41 | private static final String SUBSCRIBER_TOPIC = "/data/AccountChangeEvent"; 42 | private static final String PUBLISHER_TOPIC = "/event/NewAccount__e"; 43 | 44 | public AccountListener(ExampleConfigurations requiredParams) { 45 | logger.info("Setting up the Subscriber"); 46 | ExampleConfigurations subscriberParams = setupSubscriberParameters(requiredParams, SUBSCRIBER_TOPIC, 100); 47 | this.subscriber = new Subscribe(subscriberParams, getAccountListenerResponseObserver()); 48 | logger.info("Setting up the Publisher"); 49 | ExampleConfigurations publisherParams = setupPublisherParameters(requiredParams, PUBLISHER_TOPIC); 50 | this.publisher = new Publish(publisherParams); 51 | } 52 | 53 | /** 54 | * Custom StreamObserver for the AccountListener. 55 | * 56 | * @return StreamObserver 57 | */ 58 | private StreamObserver getAccountListenerResponseObserver() { 59 | return new StreamObserver() { 60 | @Override 61 | public void onNext(FetchResponse fetchResponse) { 62 | for(ConsumerEvent ce: fetchResponse.getEventsList()) { 63 | try { 64 | Schema writerSchema = subscriber.getSchema(ce.getEvent().getSchemaId()); 65 | GenericRecord eventPayload = CommonContext.deserialize(writerSchema, ce.getEvent().getPayload()); 66 | subscriber.updateReceivedEvents(1); 67 | for (String recordId : getRecordIdsOfAccountCDCEvent(eventPayload)) { 68 | logger.info("New Account was Created"); 69 | publisher.publish(createNewAccountProducerEvent(publisher.getSchema(), publisher.getSchemaInfo(), recordId)); 70 | } 71 | } catch (Exception e) { 72 | logger.info(e.toString()); 73 | } 74 | } 75 | if (fetchResponse.getPendingNumRequested() == 0) { 76 | subscriber.fetchMore(subscriber.getBatchSize()); 77 | } 78 | } 79 | 80 | @Override 81 | public void onError(Throwable t) { 82 | printStatusRuntimeException("Error during SubscribeStream", (Exception) t); 83 | subscriber.isActive.set(false); 84 | } 85 | 86 | @Override 87 | public void onCompleted() { 88 | logger.info("Received requested number of events! Call completed by server."); 89 | subscriber.isActive.set(false); 90 | } 91 | }; 92 | } 93 | 94 | // Helper function to start the app. 95 | public void startApp() throws InterruptedException { 96 | subscriber.startSubscription(); 97 | } 98 | 99 | // Helper function to stop the app. 100 | public void stopApp() { 101 | subscriber.close(); 102 | publisher.close(); 103 | } 104 | 105 | public static void main(String[] args) throws IOException { 106 | // For this example specifying only the required configurations in the arguments.yaml is enough. 107 | ExampleConfigurations requiredParameters = new ExampleConfigurations("arguments.yaml"); 108 | try { 109 | AccountListener ac = new AccountListener(requiredParameters); 110 | ac.startApp(); 111 | ac.stopApp(); 112 | } catch (Exception e) { 113 | printStatusRuntimeException("Error during AccountListener", e); 114 | } 115 | } 116 | } 117 | -------------------------------------------------------------------------------- /java/src/main/java/accountupdateapp/AccountUpdateAppUtil.java: -------------------------------------------------------------------------------- 1 | package accountupdateapp; 2 | 3 | import java.io.ByteArrayOutputStream; 4 | import java.io.IOException; 5 | import java.util.ArrayList; 6 | import java.util.List; 7 | import java.util.UUID; 8 | 9 | import org.apache.avro.Schema; 10 | import org.apache.avro.generic.GenericDatumWriter; 11 | import org.apache.avro.generic.GenericRecord; 12 | import org.apache.avro.generic.GenericRecordBuilder; 13 | import org.apache.avro.io.BinaryEncoder; 14 | import org.apache.avro.io.EncoderFactory; 15 | import org.eclipse.jetty.client.HttpClient; 16 | import org.eclipse.jetty.client.api.ContentResponse; 17 | import org.eclipse.jetty.client.api.Request; 18 | import org.eclipse.jetty.client.util.StringContentProvider; 19 | import org.json.simple.JSONArray; 20 | import org.json.simple.JSONObject; 21 | import org.json.simple.parser.JSONParser; 22 | import org.json.simple.parser.ParseException; 23 | import org.slf4j.Logger; 24 | 25 | import com.google.protobuf.ByteString; 26 | import com.salesforce.eventbus.protobuf.ProducerEvent; 27 | import com.salesforce.eventbus.protobuf.SchemaInfo; 28 | 29 | import utility.ExampleConfigurations; 30 | 31 | /** 32 | * The AccountUpdateAppUtil class provides helper functions such as creating NewAccount records, 33 | * NewAccount ProducerEvents, updating AccountRecord using REST API required for running the examples 34 | * in the AccountUpdateApp. 35 | */ 36 | public class AccountUpdateAppUtil { 37 | 38 | private static String changeEventHeaderFieldName = "ChangeEventHeader"; 39 | 40 | /** 41 | * Creates the record/payload of the NewAccount custom platform event. 42 | * 43 | * @param schema 44 | * @param accountRecordId 45 | * @return 46 | */ 47 | protected static GenericRecord createNewAccountRecord(Schema schema, String accountRecordId) { 48 | return new GenericRecordBuilder(schema).set("CreatedDate", System.currentTimeMillis() / 1000) 49 | .set("CreatedById", "").set("AccountRecordId__c", accountRecordId).build(); 50 | } 51 | 52 | /** 53 | * Creates the ProducerEvent of the NewAccount custom platform event. 54 | * 55 | * @param schema 56 | * @param schemaInfo 57 | * @param accountRecordId 58 | * @return 59 | * @throws IOException 60 | */ 61 | protected static ProducerEvent createNewAccountProducerEvent(Schema schema, SchemaInfo schemaInfo, String accountRecordId) throws IOException { 62 | GenericRecord event = createNewAccountRecord(schema, accountRecordId); 63 | 64 | // Convert to byte array 65 | GenericDatumWriter writer = new GenericDatumWriter<>(event.getSchema()); 66 | ByteArrayOutputStream buffer = new ByteArrayOutputStream(); 67 | BinaryEncoder encoder = EncoderFactory.get().directBinaryEncoder(buffer, null); 68 | writer.write(event, encoder); 69 | 70 | return ProducerEvent.newBuilder().setSchemaId(schemaInfo.getSchemaId()) 71 | .setPayload(ByteString.copyFrom(buffer.toByteArray())).build(); 72 | 73 | } 74 | 75 | /** 76 | * Parses the Account CDC event received to obtain the recordIds of the Account. 77 | * 78 | * @param eventRecord 79 | * @return 80 | * @throws ParseException 81 | */ 82 | protected static List getRecordIdsOfAccountCDCEvent(GenericRecord eventRecord) throws ParseException { 83 | List recordIds = new ArrayList<>(); 84 | JSONParser parser = new JSONParser(); 85 | JSONObject changeEventHeaderJson; 86 | JSONArray recordIdsJson = new JSONArray(); 87 | try { 88 | changeEventHeaderJson = (JSONObject) parser.parse(eventRecord.get(changeEventHeaderFieldName).toString()); 89 | if (changeEventHeaderJson.get("changeType").toString().equals("CREATE")) { 90 | recordIdsJson = (JSONArray) parser.parse(changeEventHeaderJson.get("recordIds").toString()); 91 | } else { 92 | return recordIds; 93 | } 94 | } catch (Exception e) { 95 | throw e; 96 | } 97 | for(Object o : recordIdsJson) { 98 | recordIds.add(o.toString()); 99 | } 100 | return recordIds; 101 | } 102 | 103 | /** 104 | * Updates the Account Object's AccountNumber field with a generated UUID using the Salesforce REST API. 105 | * 106 | * @param subParams 107 | * @param accountRecordId 108 | * @param token 109 | * @param logger 110 | * @throws Exception 111 | */ 112 | public static void updateAccountRecord(ExampleConfigurations subParams, String accountRecordId, String token, Logger logger) throws Exception { 113 | HttpClient client = new HttpClient(); 114 | client.start(); 115 | 116 | String accountNumber = UUID.randomUUID().toString(); 117 | 118 | Request req = client.POST(subParams.getLoginUrl()+"/services/data/v52.0/sobjects/Account/" + accountRecordId + "?_HttpMethod=PATCH"); 119 | 120 | req.header("Authorization", "Bearer " + token); 121 | req.header("Content-Type", "application/json"); 122 | req.content(new StringContentProvider("{\"AccountNumber\": \"" + accountNumber + "\"}", "utf-8")); 123 | 124 | ContentResponse response = req.send(); 125 | int res = response.getStatus(); 126 | 127 | if (res > 299) { 128 | logger.info("Unable to update Account Record."); 129 | logger.info(response.getContentAsString()); 130 | } else { 131 | logger.info("Successfully updated Account Record. Updated AccountNumber: " + accountNumber); 132 | } 133 | client.stop(); 134 | } 135 | } 136 | -------------------------------------------------------------------------------- /java/src/main/java/accountupdateapp/AccountUpdater.java: -------------------------------------------------------------------------------- 1 | package accountupdateapp; 2 | 3 | 4 | import static accountupdateapp.AccountUpdateAppUtil.*; 5 | import static utility.CommonContext.*; 6 | 7 | import java.io.IOException; 8 | 9 | import org.apache.avro.Schema; 10 | import org.apache.avro.generic.GenericRecord; 11 | import org.slf4j.Logger; 12 | import org.slf4j.LoggerFactory; 13 | 14 | import com.salesforce.eventbus.protobuf.ConsumerEvent; 15 | import com.salesforce.eventbus.protobuf.FetchResponse; 16 | 17 | import genericpubsub.Subscribe; 18 | import io.grpc.stub.StreamObserver; 19 | import utility.CommonContext; 20 | import utility.ExampleConfigurations; 21 | 22 | /** 23 | * AccountUpdater 24 | * A subscribe client that listens to the `/event/NewAccount__e` custom platform events and updates 25 | * the appropriate Account Record with an AccountNumber using the Salesforce REST API. 26 | * 27 | * Example: 28 | * ./run.sh accountupdateapp.AccountUpdater 29 | * 30 | * @author sidd0610 31 | */ 32 | 33 | public class AccountUpdater { 34 | 35 | protected static final Logger logger = LoggerFactory.getLogger(AccountUpdater.class.getClass()); 36 | 37 | protected Subscribe subscriber; 38 | private ExampleConfigurations subscriberParams; 39 | 40 | private static final String SUBSCRIBER_TOPIC = "/event/NewAccount__e"; 41 | 42 | public AccountUpdater(ExampleConfigurations requiredParams) { 43 | logger.info("Setting Up Subscriber"); 44 | this.subscriberParams = setupSubscriberParameters(requiredParams, SUBSCRIBER_TOPIC, 100); 45 | this.subscriber = new Subscribe(subscriberParams, getAccountUpdaterResponseObserver()); 46 | } 47 | 48 | /** 49 | * Custom StreamObserver for the AccountUpdater. 50 | * 51 | * @return StreamObserver 52 | */ 53 | private StreamObserver getAccountUpdaterResponseObserver() { 54 | return new StreamObserver() { 55 | @Override 56 | public void onNext(FetchResponse fetchResponse) { 57 | for(ConsumerEvent ce: fetchResponse.getEventsList()) { 58 | try { 59 | Schema writerSchema = subscriber.getSchema(ce.getEvent().getSchemaId()); 60 | GenericRecord eventPayload = CommonContext.deserialize(writerSchema, ce.getEvent().getPayload()); 61 | subscriber.updateReceivedEvents(1); 62 | String accountRecordId = eventPayload.get("AccountRecordId__c").toString(); 63 | updateAccountRecord(subscriberParams, accountRecordId, subscriber.getSessionToken(), logger); 64 | } catch (Exception e) { 65 | logger.info(e.toString()); 66 | } 67 | } 68 | if (fetchResponse.getPendingNumRequested() == 0) { 69 | subscriber.fetchMore(subscriber.getBatchSize()); 70 | } 71 | } 72 | 73 | @Override 74 | public void onError(Throwable t) { 75 | printStatusRuntimeException("Error during SubscribeStream", (Exception) t); 76 | subscriber.isActive.set(false); 77 | } 78 | 79 | @Override 80 | public void onCompleted() { 81 | logger.info("Received requested number of events! Call completed by server."); 82 | subscriber.isActive.set(false); 83 | } 84 | }; 85 | } 86 | 87 | // Helper function to start the app. 88 | public void startApp() throws InterruptedException { 89 | subscriber.startSubscription(); 90 | } 91 | 92 | public void stopApp() { 93 | subscriber.close(); 94 | } 95 | 96 | public static void main(String[] args) throws IOException { 97 | // For this example specifying only the required configurations in the arguments.yaml is enough. 98 | ExampleConfigurations requiredParameters = new ExampleConfigurations("arguments.yaml"); 99 | try { 100 | AccountUpdater ac = new AccountUpdater(requiredParameters); 101 | ac.startApp(); 102 | ac.stopApp(); 103 | } catch (Exception e) { 104 | printStatusRuntimeException("Error during AccountUpdate", e); 105 | } 106 | } 107 | } 108 | -------------------------------------------------------------------------------- /java/src/main/java/accountupdateapp/README.md: -------------------------------------------------------------------------------- 1 | # Account Update App 2 | 3 | This example subscribes to change events corresponding to the creation of [Account](https://developer.salesforce.com/docs/atlas.en-us.object_reference.meta/object_reference/sforce_api_objects_account.htm) records. Also, it updates a field in the created records using a custom platform event as a mediator between the two processes. 4 | 5 | ## Prerequisites: 6 | 1. The `Account` entity needs to be selected in order to generate change events whenever there is any action taken wrt Account objects. Steps to enable this: 7 | * Go to the `Setup Home` in your Salesforce org 8 | * Under the `Platform Tools` section, click on `Change Data Capture` 9 | * Search for the `Account` object and click on the right arrow in the middle of the screen to select the entity. 10 | * Click on the `Save` button to update the changes. 11 | 2. The `NewAccount` custom platform needs to be created with the following fields: 12 | - Platform Event Name 13 | - Label: `NewAccount` 14 | - Plural Label: `NewAccounts` 15 | - Custom Fields 16 | - `AccountRecordId` (Text, 20) 17 | 3. Only the required configurations need to be specified in the `arguments.yaml` file while running this example. You can specify the other optional configurations, but the optional configurations required for this example will be overwritten while running the examples. 18 | 19 | ## Flow Overview: 20 | * User creates an `Account` standard object which triggers an `AccountChangeEvent` event. 21 | * When the user creates an `Account` object, this generates a change event on the /data/AccountChangeEvent topic. 22 | * A listener listens to this `AccountChangeEvent` event and publishes a `NewAccount` custom platform event 23 | * The listener subscribes to the events on the `/data/AccountChangeEvent` topic and only in the case when a new `Account` is created, it will publish an event on the `/event/NewAccount__e` custom platform event topic with the recordId of the created `Account` object. 24 | * The updater listens to this `NewAccount` event and updates the `Account` object with a randomly generated `AccountNumber`. 25 | * The updater subscribes to the `/event/NewAccount__e` topic and when an event is received, it will update the appropriate `Account` object with a randomly generated `AccountNumber` using the Salesforce REST API. 26 | 27 | ## Running the examples: 28 | 1. Run the `AccountUpdater` first by running the following command: 29 | ``` 30 | ./run.sh accountupdateapp.AccountUpdater 31 | ``` 32 | 2. Run the `AccountListener` next by running the following command: 33 | ``` 34 | ./run.sh accountupdateapp.AccountListener 35 | ``` 36 | 37 | ## Notes: 38 | * Please use the `my domain` URL for your org for running these examples. You can find the my domain URL through Developer Console. 39 | * Open Developer Console 40 | * Click on the Debug menu and select Open Execute Anonymous Window. 41 | * Key in the following in the window: `System.debug(System.url.getOrgDomainUrl());` and execute the same. 42 | * Once done, in the Logs tab below open the logs recently executed code. 43 | * In the logs, get the `my domain` URL from the USER_DEBUG event. 44 | * Subscribers in both the `AccountUpdater` and `AccountListener` subscribe with the ReplayPreset set to LATEST. Therefore, only events generated once the examples have started running will be processed. 45 | * The `AccountUpdater` logs the `AccountNumber` that has been added to the `Account` record which can be used to verify if the update is correct. -------------------------------------------------------------------------------- /java/src/main/java/genericpubsub/GetSchema.java: -------------------------------------------------------------------------------- 1 | package genericpubsub; 2 | 3 | import java.io.IOException; 4 | 5 | import org.apache.avro.Schema; 6 | 7 | import com.salesforce.eventbus.protobuf.SchemaInfo; 8 | import com.salesforce.eventbus.protobuf.SchemaRequest; 9 | import com.salesforce.eventbus.protobuf.TopicInfo; 10 | import com.salesforce.eventbus.protobuf.TopicRequest; 11 | 12 | import utility.CommonContext; 13 | import utility.ExampleConfigurations; 14 | 15 | /** 16 | * An example that retrieves the Schema of a single-topic. 17 | * 18 | * Example: 19 | * ./run.sh genericpubsub.GetSchema 20 | * 21 | * @author sidd0610 22 | */ 23 | public class GetSchema extends CommonContext { 24 | 25 | public GetSchema(final ExampleConfigurations options) { 26 | super(options); 27 | } 28 | 29 | private void getSchema(String topicName) { 30 | // Use the GetTopic RPC to get the topic info for the given topicName. 31 | // Used to retrieve the schema id in this example. 32 | TopicInfo topicInfo = blockingStub.getTopic(TopicRequest.newBuilder().setTopicName(topicName).build()); 33 | logger.info("GetTopic Call RPC ID: " + topicInfo.getRpcId()); 34 | 35 | topicInfo.getAllFields().entrySet().forEach(entry -> { 36 | logger.info(entry.getKey() + " : " + entry.getValue()); 37 | }); 38 | 39 | SchemaRequest schemaRequest = SchemaRequest.newBuilder().setSchemaId(topicInfo.getSchemaId()).build(); 40 | 41 | // Use the GetSchema RPC to get the schema info of the topic. 42 | SchemaInfo schemaResponse = blockingStub.getSchema(schemaRequest); 43 | logger.info("GetSchema Call RPC ID: " + schemaResponse.getRpcId()); 44 | Schema schema = new Schema.Parser().parse(schemaResponse.getSchemaJson()); 45 | 46 | // Printing the topic schema 47 | logger.info("Schema of topic " + topicName + ": " + schema.toString(true)); 48 | } 49 | 50 | public static void main(String[] args) throws IOException { 51 | ExampleConfigurations exampleConfigurations = new ExampleConfigurations("arguments.yaml"); 52 | 53 | // Using the try-with-resource statement. The CommonContext class implements AutoCloseable in 54 | // order to close the resources used. 55 | try (GetSchema example = new GetSchema(exampleConfigurations)) { 56 | example.getSchema(exampleConfigurations.getTopic()); 57 | } catch (Exception e) { 58 | printStatusRuntimeException("Getting schema", e); 59 | } 60 | } 61 | 62 | } 63 | -------------------------------------------------------------------------------- /java/src/main/java/genericpubsub/GetTopic.java: -------------------------------------------------------------------------------- 1 | package genericpubsub; 2 | 3 | import java.io.IOException; 4 | 5 | import com.salesforce.eventbus.protobuf.TopicInfo; 6 | import com.salesforce.eventbus.protobuf.TopicRequest; 7 | 8 | import utility.CommonContext; 9 | import utility.ExampleConfigurations; 10 | 11 | /** 12 | * An example that retrieves the topic info of a single-topic. 13 | * 14 | * Example: 15 | * ./run.sh genericpubsub.GetTopic 16 | * 17 | * @author sidd0610 18 | */ 19 | public class GetTopic extends CommonContext { 20 | 21 | public GetTopic(final ExampleConfigurations options) { 22 | super(options); 23 | } 24 | 25 | private void getTopic(String topicName) { 26 | // Use the GetTopic RPC to get the topic info for the given topicName. 27 | TopicInfo topicInfo = blockingStub.getTopic(TopicRequest.newBuilder().setTopicName(topicName).build()); 28 | 29 | logger.info("Topic Details:"); 30 | topicInfo.getAllFields().entrySet().forEach(item -> { 31 | logger.info(item.getKey() + " : " + item.getValue()); 32 | }); 33 | } 34 | 35 | public static void main(String[] args) throws IOException { 36 | ExampleConfigurations exampleConfigurations = new ExampleConfigurations("arguments.yaml"); 37 | 38 | // Using the try-with-resource statement. The CommonContext class implements AutoCloseable in 39 | // order to close the resources used. 40 | try (GetTopic example = new GetTopic(exampleConfigurations)) { 41 | example.getTopic(exampleConfigurations.getTopic()); 42 | } catch (Exception e) { 43 | printStatusRuntimeException("Error while Getting Topic", e); 44 | } 45 | } 46 | } 47 | -------------------------------------------------------------------------------- /java/src/main/java/genericpubsub/ManagedSubscribe.java: -------------------------------------------------------------------------------- 1 | package genericpubsub; 2 | 3 | import java.io.IOException; 4 | import java.util.Map; 5 | import java.util.Objects; 6 | import java.util.UUID; 7 | import java.util.concurrent.*; 8 | import java.util.concurrent.atomic.AtomicBoolean; 9 | import java.util.concurrent.atomic.AtomicInteger; 10 | 11 | import org.apache.avro.Schema; 12 | import org.apache.avro.generic.GenericRecord; 13 | 14 | import com.google.protobuf.ByteString; 15 | import com.salesforce.eventbus.protobuf.*; 16 | 17 | import io.grpc.stub.StreamObserver; 18 | import utility.CommonContext; 19 | import utility.ExampleConfigurations; 20 | 21 | /** 22 | * A single-topic subscriber that consumes events using Event Bus API ManagedSubscribe RPC. The example demonstrates how to: 23 | * - implement a long-lived subscription to a single topic 24 | * - a basic flow control strategy 25 | * - a basic commits strategy. 26 | *

27 | * Example: 28 | * ./run.sh genericpubsub.ManagedSubscribe 29 | * 30 | * @author jalaya 31 | */ 32 | public class ManagedSubscribe extends CommonContext implements StreamObserver { 33 | private static int BATCH_SIZE; 34 | private StreamObserver serverStream; 35 | private Map schemaCache = new ConcurrentHashMap<>(); 36 | private final CountDownLatch serverOnCompletedLatch = new CountDownLatch(1); 37 | public static AtomicBoolean isActive = new AtomicBoolean(false); 38 | private AtomicInteger receivedEvents = new AtomicInteger(0); 39 | private String developerName; 40 | private String managedSubscriptionId; 41 | private final boolean processChangedFields; 42 | 43 | public ManagedSubscribe(ExampleConfigurations exampleConfigurations) { 44 | super(exampleConfigurations); 45 | isActive.set(true); 46 | this.managedSubscriptionId = exampleConfigurations.getManagedSubscriptionId(); 47 | this.developerName = exampleConfigurations.getDeveloperName(); 48 | this.BATCH_SIZE = exampleConfigurations.getNumberOfEventsToSubscribeInEachFetchRequest(); 49 | this.processChangedFields = exampleConfigurations.getProcessChangedFields(); 50 | } 51 | 52 | /** 53 | * Function to start the ManagedSubscription, and send first ManagedFetchRequest. 54 | */ 55 | public void startManagedSubscription() { 56 | serverStream = asyncStub.managedSubscribe(this); 57 | ManagedFetchRequest.Builder builder = ManagedFetchRequest.newBuilder().setNumRequested(BATCH_SIZE); 58 | 59 | if (Objects.nonNull(managedSubscriptionId)) { 60 | builder.setSubscriptionId(managedSubscriptionId); 61 | logger.info("Starting managed subscription with ID {}", managedSubscriptionId); 62 | } else if (Objects.nonNull(developerName)) { 63 | builder.setDeveloperName(developerName); 64 | logger.info("Starting managed subscription with developer name {}", developerName); 65 | } else { 66 | logger.warn("No ID or developer name specified"); 67 | } 68 | 69 | serverStream.onNext(builder.build()); 70 | 71 | // Thread being blocked here for demonstration of this specific example. Blocking the thread in production is not recommended. 72 | while(isActive.get()) { 73 | waitInMillis(5_000); 74 | logger.info("Subscription Active. Received a total of " + receivedEvents.get() + " events."); 75 | } 76 | } 77 | 78 | /** 79 | * Helps keep the subscription active by sending FetchRequests at regular intervals. 80 | * 81 | * @param numOfRequestedEvents 82 | */ 83 | private void fetchMore(int numOfRequestedEvents) { 84 | logger.info("Fetching more events: {}", numOfRequestedEvents); 85 | ManagedFetchRequest fetchRequest = ManagedFetchRequest 86 | .newBuilder() 87 | .setNumRequested(numOfRequestedEvents) 88 | .build(); 89 | serverStream.onNext(fetchRequest); 90 | } 91 | 92 | /** 93 | * Helper function to process the events received. 94 | */ 95 | private void processEvent(ManagedFetchResponse response) throws IOException { 96 | if (response.getEventsCount() > 0) { 97 | for (ConsumerEvent event : response.getEventsList()) { 98 | String schemaId = event.getEvent().getSchemaId(); 99 | logger.info("processEvent - EventID: {} SchemaId: {}", event.getEvent().getId(), schemaId); 100 | Schema writerSchema = getSchema(schemaId); 101 | GenericRecord record = deserialize(writerSchema, event.getEvent().getPayload()); 102 | logger.info("Received event: {}", record.toString()); 103 | if (processChangedFields) { 104 | // This example expands the changedFields bitmap field in ChangeEventHeader. 105 | // To expand the other bitmap fields, i.e., diffFields and nulledFields, replicate or modify this code. 106 | processAndPrintBitmapFields(writerSchema, record, "changedFields"); 107 | } 108 | } 109 | logger.info("Processed batch of {} event(s)", response.getEventsList().size()); 110 | } 111 | 112 | // Commit the replay after processing batch of events or commit the latest replay on an empty batch 113 | if (!response.hasCommitResponse()) { 114 | doCommitReplay(response.getLatestReplayId()); 115 | } 116 | } 117 | 118 | /** 119 | * Helper function to commit the latest replay received from the server. 120 | */ 121 | private void doCommitReplay(ByteString commitReplayId) { 122 | String newKey = UUID.randomUUID().toString(); 123 | ManagedFetchRequest.Builder fetchRequestBuilder = ManagedFetchRequest.newBuilder(); 124 | CommitReplayRequest commitRequest = CommitReplayRequest.newBuilder() 125 | .setCommitRequestId(newKey) 126 | .setReplayId(commitReplayId) 127 | .build(); 128 | fetchRequestBuilder.setCommitReplayIdRequest(commitRequest); 129 | 130 | logger.info("Sending CommitRequest with CommitReplayRequest ID: {}" , newKey); 131 | serverStream.onNext(fetchRequestBuilder.build()); 132 | } 133 | 134 | /** 135 | * Helper function to inspect the status of a commitRequest. 136 | */ 137 | private void checkCommitResponse(ManagedFetchResponse fetchResponse) { 138 | CommitReplayResponse ce = fetchResponse.getCommitResponse(); 139 | try { 140 | if (ce.hasError()) { 141 | logger.info("Failed Commit CommitRequestID: {} with error: {} with process time: {}", 142 | ce.getCommitRequestId(), ce.getError().getMsg(), ce.getProcessTime()); 143 | return; 144 | } 145 | logger.info("Successfully committed replay with CommitRequestId: {} with process time: {}", 146 | ce.getCommitRequestId(), ce.getProcessTime()); 147 | } catch (Exception e) { 148 | logger.warn(e.getMessage()); 149 | abort(new RuntimeException("Client received error. Closing Call." + e)); 150 | } 151 | } 152 | 153 | @Override 154 | public void onNext(ManagedFetchResponse fetchResponse) { 155 | int batchSize = fetchResponse.getEventsList().size(); 156 | logger.info("ManagedFetchResponse batch of {} events pending requested: {}", batchSize, fetchResponse.getPendingNumRequested()); 157 | logger.info("RPC ID: {}", fetchResponse.getRpcId()); 158 | 159 | if (fetchResponse.hasCommitResponse()) { 160 | checkCommitResponse(fetchResponse); 161 | } 162 | try { 163 | processEvent(fetchResponse); 164 | } catch (IOException e) { 165 | logger.warn(e.getMessage()); 166 | abort(new RuntimeException("Client received error. Closing Call." + e)); 167 | } 168 | 169 | synchronized (this) { 170 | receivedEvents.addAndGet(batchSize); 171 | this.notifyAll(); 172 | if (!isActive.get()) { 173 | return; 174 | } 175 | } 176 | 177 | if (fetchResponse.getPendingNumRequested() == 0) { 178 | fetchMore(BATCH_SIZE); 179 | } 180 | } 181 | 182 | @Override 183 | public void onError(Throwable throwable) { 184 | printStatusRuntimeException("Error during subscribe stream", (Exception) throwable); 185 | 186 | // onError from server closes stream. notify waiting thread that subscription is no longer active. 187 | synchronized (this) { 188 | isActive.set(false); 189 | this.notifyAll(); 190 | } 191 | } 192 | 193 | @Override 194 | public void onCompleted() { 195 | logger.info("Call completed by Server"); 196 | synchronized (this) { 197 | isActive.set(false); 198 | this.notifyAll(); 199 | } 200 | serverOnCompletedLatch.countDown(); 201 | } 202 | 203 | /** 204 | * Helper function to get the schema of an event if it does not already exist in the schema cache. 205 | */ 206 | private Schema getSchema(String schemaId) { 207 | return schemaCache.computeIfAbsent(schemaId, id -> { 208 | SchemaRequest request = SchemaRequest.newBuilder().setSchemaId(id).build(); 209 | String schemaJson = blockingStub.getSchema(request).getSchemaJson(); 210 | return (new Schema.Parser()).parse(schemaJson); 211 | }); 212 | } 213 | 214 | /** 215 | * Closes the connection when the task is complete. 216 | */ 217 | @Override 218 | public synchronized void close() { 219 | if (Objects.nonNull(serverStream)) { 220 | try { 221 | if (isActive.get()) { 222 | isActive.set(false); 223 | this.notifyAll(); 224 | serverStream.onCompleted(); 225 | } 226 | serverOnCompletedLatch.await(6, TimeUnit.SECONDS); 227 | } catch (InterruptedException e) { 228 | logger.warn("interrupted while waiting to close ", e); 229 | } 230 | } 231 | super.close(); 232 | } 233 | 234 | /** 235 | * Helper function to terminate the client on errors. 236 | */ 237 | private synchronized void abort(Exception e) { 238 | serverStream.onError(e); 239 | isActive.set(false); 240 | this.notifyAll(); 241 | } 242 | 243 | /** 244 | * Helper function to halt the current thread. 245 | */ 246 | public void waitInMillis(long duration) { 247 | synchronized (this) { 248 | try { 249 | this.wait(duration); 250 | } catch (InterruptedException e) { 251 | throw new RuntimeException(e); 252 | } 253 | } 254 | } 255 | 256 | public static void main(String args[]) throws IOException { 257 | ExampleConfigurations exampleConfigurations = new ExampleConfigurations("arguments.yaml"); 258 | 259 | // Using the try-with-resource statement. The CommonContext class implements AutoCloseable in 260 | // order to close the resources used. 261 | try (ManagedSubscribe subscribe = new ManagedSubscribe(exampleConfigurations)) { 262 | subscribe.startManagedSubscription(); 263 | } catch (Exception e) { 264 | printStatusRuntimeException("Error during ManagedSubscribe", e); 265 | } 266 | } 267 | } 268 | -------------------------------------------------------------------------------- /java/src/main/java/genericpubsub/Publish.java: -------------------------------------------------------------------------------- 1 | package genericpubsub; 2 | 3 | import java.io.ByteArrayOutputStream; 4 | import java.io.IOException; 5 | import java.util.List; 6 | 7 | import org.apache.avro.Schema; 8 | import org.apache.avro.generic.GenericDatumWriter; 9 | import org.apache.avro.generic.GenericRecord; 10 | import org.apache.avro.io.BinaryEncoder; 11 | import org.apache.avro.io.EncoderFactory; 12 | 13 | import com.google.protobuf.ByteString; 14 | import com.salesforce.eventbus.protobuf.*; 15 | import utility.CommonContext; 16 | import utility.ExampleConfigurations; 17 | 18 | /** 19 | * A single-topic publisher that creates an Order Event event and publishes it. This example uses 20 | * Pub/Sub API's Publish RPC to publish events. 21 | * 22 | * Example: 23 | * ./run.sh genericpubsub.Publish 24 | * 25 | * @author sidd0610 26 | */ 27 | public class Publish extends CommonContext { 28 | 29 | private Schema schema; 30 | 31 | public Publish(ExampleConfigurations exampleConfigurations) { 32 | super(exampleConfigurations); 33 | setupTopicDetails(exampleConfigurations.getTopic(), true, true); 34 | schema = new Schema.Parser().parse(schemaInfo.getSchemaJson()); 35 | } 36 | 37 | /** 38 | * Helper function for creating the ProducerEvent to be published 39 | * 40 | * @return ProducerEvent 41 | * @throws IOException 42 | */ 43 | private ProducerEvent generateProducerEvent() throws IOException { 44 | Schema schema = new Schema.Parser().parse(schemaInfo.getSchemaJson()); 45 | GenericRecord event = createEventMessage(schema); 46 | 47 | // Convert to byte array 48 | GenericDatumWriter writer = new GenericDatumWriter<>(event.getSchema()); 49 | ByteArrayOutputStream buffer = new ByteArrayOutputStream(); 50 | BinaryEncoder encoder = EncoderFactory.get().directBinaryEncoder(buffer, null); 51 | writer.write(event, encoder); 52 | 53 | return ProducerEvent.newBuilder().setSchemaId(schemaInfo.getSchemaId()) 54 | .setPayload(ByteString.copyFrom(buffer.toByteArray())).build(); 55 | } 56 | 57 | /** 58 | * Helper function to generate the PublishRequest with the generated ProducerEvent to be sent 59 | * using the Publish RPC 60 | * 61 | * @return PublishRequest 62 | * @throws IOException 63 | */ 64 | private PublishRequest generatePublishRequest() throws IOException { 65 | ProducerEvent e = generateProducerEvent(); 66 | return PublishRequest.newBuilder().setTopicName(busTopicName).addEvents(e).build(); 67 | } 68 | 69 | /** 70 | * Helper function to publish the event using Publish RPC 71 | */ 72 | public ByteString publish() throws Exception { 73 | PublishResponse response = blockingStub.publish(generatePublishRequest()); 74 | return validatePublishResponse(response); 75 | } 76 | 77 | /** 78 | * Helper function for other examples to publish the event using Publish RPC 79 | * 80 | * @param event 81 | * @return 82 | * @throws Exception 83 | */ 84 | public PublishResponse publish(ProducerEvent event) throws Exception { 85 | PublishRequest publishRequest = PublishRequest.newBuilder().setTopicName(busTopicName).addEvents(event).build(); 86 | PublishResponse response = blockingStub.publish(publishRequest); 87 | validatePublishResponse(response); 88 | return response; 89 | } 90 | 91 | /** 92 | * Helper function to validate the PublishResponse received. Also prints the RPC id of the call. 93 | * 94 | * @param response 95 | * @return 96 | */ 97 | private ByteString validatePublishResponse(PublishResponse response) { 98 | ByteString lastPublishedReplayId = null; 99 | List resultList = response.getResultsList(); 100 | if (resultList.size() != 1) { 101 | String errorMsg = "[ERROR] Error during Publish, received: " + resultList.size() + " events instead of expected 1"; 102 | logger.error(errorMsg); 103 | throw new RuntimeException(errorMsg); 104 | } else { 105 | PublishResult result = resultList.get(0); 106 | if (result.hasError()) { 107 | logger.error("[ERROR] Publishing batch failed with rpcId: " + response.getRpcId()); 108 | logger.error("[ERROR] Error during Publish, event with correlationKey: {} failed with: {}", 109 | response.getResults(0).getCorrelationKey(), result.getError().getMsg()); 110 | } else { 111 | lastPublishedReplayId = result.getReplayId(); 112 | logger.info("Publish Call RPC ID: " + response.getRpcId()); 113 | logger.info("Successfully published an event with correlationKey: {} at {} for tenant {}.", 114 | response.getResults(0).getCorrelationKey(), busTopicName, tenantGuid); 115 | } 116 | } 117 | 118 | return lastPublishedReplayId; 119 | } 120 | 121 | /** 122 | * General getters. 123 | */ 124 | public Schema getSchema() { 125 | return schema; 126 | } 127 | 128 | public SchemaInfo getSchemaInfo() { 129 | return schemaInfo; 130 | } 131 | 132 | public static void main(String[] args) throws IOException { 133 | ExampleConfigurations exampleConfigurations = new ExampleConfigurations("arguments.yaml"); 134 | 135 | // Using the try-with-resource statement. The CommonContext class implements AutoCloseable in 136 | // order to close the resources used. 137 | try (Publish example = new Publish(exampleConfigurations)) { 138 | example.publish(); 139 | } catch (Exception e) { 140 | CommonContext.printStatusRuntimeException("Publishing events", e); 141 | } 142 | } 143 | } -------------------------------------------------------------------------------- /java/src/main/java/genericpubsub/PublishStream.java: -------------------------------------------------------------------------------- 1 | package genericpubsub; 2 | 3 | import java.io.ByteArrayOutputStream; 4 | import java.io.IOException; 5 | import java.util.List; 6 | import java.util.UUID; 7 | import java.util.concurrent.CountDownLatch; 8 | import java.util.concurrent.TimeUnit; 9 | import java.util.concurrent.atomic.AtomicInteger; 10 | import java.util.concurrent.atomic.AtomicReference; 11 | 12 | import org.apache.avro.Schema; 13 | import org.apache.avro.generic.GenericDatumWriter; 14 | import org.apache.avro.generic.GenericRecord; 15 | import org.apache.avro.io.BinaryEncoder; 16 | import org.apache.avro.io.EncoderFactory; 17 | 18 | import com.google.common.collect.Lists; 19 | import com.google.protobuf.ByteString; 20 | import com.salesforce.eventbus.protobuf.*; 21 | 22 | import io.grpc.Status; 23 | import io.grpc.stub.ClientCallStreamObserver; 24 | import io.grpc.stub.StreamObserver; 25 | import utility.CommonContext; 26 | import utility.ExampleConfigurations; 27 | 28 | /** 29 | * A single-topic publisher that creates Order Event events and publishes them. This example 30 | * uses Pub/Sub API's PublishStream RPC to publish events. 31 | * 32 | * Example: 33 | * ./run.sh genericpubsub.PublishStream 34 | * 35 | * @author sidd0610 36 | */ 37 | public class PublishStream extends CommonContext { 38 | private final int TIMEOUT_SECONDS = 30; // Max time we'll wait to finish streaming 39 | 40 | ClientCallStreamObserver requestObserver = null; 41 | 42 | private ByteString lastPublishedReplayId; 43 | 44 | public PublishStream(ExampleConfigurations exampleConfigurations) { 45 | super(exampleConfigurations); 46 | setupTopicDetails(exampleConfigurations.getTopic(), true, true); 47 | } 48 | 49 | /** 50 | * Publishes specified number of events using the PublishStream RPC. 51 | * 52 | * @param numEventsToPublish 53 | * @return ByteString 54 | * @throws Exception 55 | */ 56 | public void publishStream(int numEventsToPublish, Boolean singlePublishRequest) throws Exception { 57 | CountDownLatch finishLatch = new CountDownLatch(1); 58 | AtomicReference finishLatchRef = new AtomicReference<>(finishLatch); 59 | final int numExpectedPublishResponses = singlePublishRequest ? 1 : numEventsToPublish; 60 | final List publishResponses = Lists.newArrayListWithExpectedSize(numExpectedPublishResponses); 61 | AtomicInteger failed = new AtomicInteger(0); 62 | StreamObserver pubObserver = getDefaultPublishStreamObserver(finishLatchRef, 63 | numExpectedPublishResponses, publishResponses, failed); 64 | 65 | // construct the stream 66 | requestObserver = (ClientCallStreamObserver) asyncStub.publishStream(pubObserver); 67 | 68 | if (singlePublishRequest == false) { 69 | // Publish each event in a separate batch 70 | for (int i = 0; i < numEventsToPublish; i++) { 71 | requestObserver.onNext(generatePublishRequest(i, singlePublishRequest)); 72 | } 73 | } else { 74 | // Publish all events in one batch 75 | requestObserver.onNext(generatePublishRequest(numEventsToPublish, singlePublishRequest)); 76 | } 77 | 78 | validatePublishResponse(finishLatch, numExpectedPublishResponses, publishResponses, failed, numEventsToPublish); 79 | requestObserver.onCompleted(); 80 | } 81 | 82 | /** 83 | * Helper function to validate the PublishResponse received. Also prints the RPC id of the call. 84 | * 85 | * @param errorStatus 86 | * @param finishLatch 87 | * @param expectedResponseCount 88 | * @param publishResponses 89 | * @return 90 | * @throws Exception 91 | */ 92 | private void validatePublishResponse(CountDownLatch finishLatch, 93 | int expectedResponseCount, List publishResponses, AtomicInteger failed, int expectedNumEventsPublished) throws Exception { 94 | String exceptionMsg; 95 | boolean failedPublish = false; 96 | if (!finishLatch.await(TIMEOUT_SECONDS, TimeUnit.SECONDS)) { 97 | failedPublish = true; 98 | exceptionMsg = "[ERROR] publishStream timed out after: " + TIMEOUT_SECONDS + "sec"; 99 | logger.error(exceptionMsg); 100 | } 101 | 102 | if (expectedResponseCount != publishResponses.size()) { 103 | failedPublish = true; 104 | exceptionMsg = "[ERROR] PublishStream received: " + publishResponses.size() + " PublishResponses instead of expected " 105 | + expectedResponseCount; 106 | logger.error(exceptionMsg); 107 | } 108 | 109 | if (failed.get() != 0) { 110 | failedPublish = true; 111 | exceptionMsg = "[ERROR] Failed to publish all events. " + failed + " failed out of " 112 | + expectedNumEventsPublished; 113 | logger.error(exceptionMsg); 114 | } 115 | 116 | if (failedPublish) { 117 | throw new RuntimeException("Failed to publish events."); 118 | } 119 | } 120 | 121 | /** 122 | * Creates a ProducerEvent to be published in a PublishRequest. 123 | * 124 | * @param counter 125 | * @return 126 | * @throws IOException 127 | */ 128 | private ProducerEvent generateProducerEvent(int counter) throws IOException { 129 | Schema schema = new Schema.Parser().parse(schemaInfo.getSchemaJson()); 130 | GenericRecord event = createEventMessage(schema, counter); 131 | 132 | // Convert to byte array 133 | GenericDatumWriter writer = new GenericDatumWriter<>(event.getSchema()); 134 | ByteArrayOutputStream buffer = new ByteArrayOutputStream(); 135 | BinaryEncoder encoder = EncoderFactory.get().directBinaryEncoder(buffer, null); 136 | writer.write(event, encoder); 137 | 138 | return ProducerEvent.newBuilder().setSchemaId(schemaInfo.getSchemaId()) 139 | .setPayload(ByteString.copyFrom(buffer.toByteArray())).build(); 140 | } 141 | 142 | /** 143 | * Creates an array of ProducerEvents to be published in a PublishRequest. 144 | * 145 | * @param count 146 | * @return 147 | * @throws IOException 148 | */ 149 | private ProducerEvent[] generateProducerEvents(int count) throws IOException { 150 | Schema schema = new Schema.Parser().parse(schemaInfo.getSchemaJson()); 151 | List events = createEventMessages(schema, count); 152 | 153 | ProducerEvent[] prodEvents = new ProducerEvent[count]; 154 | GenericDatumWriter writer = new GenericDatumWriter<>(events.get(0).getSchema()); 155 | 156 | for(int i=0; i getDefaultPublishStreamObserver(AtomicReference finishLatchRef, int expectedResponseCount, 202 | List publishResponses, AtomicInteger failed) { 203 | return new StreamObserver<>() { 204 | @Override 205 | public void onNext(PublishResponse publishResponse) { 206 | publishResponses.add(publishResponse); 207 | 208 | logger.info("Publish Call rpcId: " + publishResponse.getRpcId()); 209 | 210 | for (PublishResult publishResult : publishResponse.getResultsList()) { 211 | if (publishResult.hasError()) { 212 | failed.incrementAndGet(); 213 | logger.error("[ERROR] Publishing event with correlationKey: " + publishResult.getCorrelationKey() + 214 | " failed with error: " + publishResult.getError().getMsg()); 215 | } else { 216 | logger.info("Event published with correlationKey: " + publishResult.getCorrelationKey()); 217 | lastPublishedReplayId = publishResult.getReplayId(); 218 | } 219 | } 220 | if (publishResponses.size() == expectedResponseCount) { 221 | finishLatchRef.get().countDown(); 222 | } 223 | } 224 | 225 | @Override 226 | public void onError(Throwable t) { 227 | logger.error("[ERROR] Unexpected error status: " + Status.fromThrowable(t)); 228 | printStatusRuntimeException("Error during PublishStream", (Exception) t); 229 | finishLatchRef.get().countDown(); 230 | } 231 | 232 | @Override 233 | public void onCompleted() { 234 | logger.info("Successfully published events for topic " + busTopicName + " for tenant " + tenantGuid); 235 | finishLatchRef.get().countDown(); 236 | } 237 | }; 238 | } 239 | 240 | public static void main(String[] args) throws IOException { 241 | ExampleConfigurations exampleConfigurations = new ExampleConfigurations("arguments.yaml"); 242 | 243 | // Using the try-with-resource statement. The CommonContext class implements AutoCloseable in 244 | // order to close the resources used. 245 | try (PublishStream example = new PublishStream(exampleConfigurations)) { 246 | example.publishStream(exampleConfigurations.getNumberOfEventsToPublish(), 247 | exampleConfigurations.getSinglePublishRequest()); 248 | } catch (Exception e) { 249 | printStatusRuntimeException("Error During PublishStream", e); 250 | } 251 | } 252 | } 253 | -------------------------------------------------------------------------------- /java/src/main/java/genericpubsub/Subscribe.java: -------------------------------------------------------------------------------- 1 | package genericpubsub; 2 | 3 | import java.io.IOException; 4 | import java.util.Map; 5 | import java.util.concurrent.*; 6 | import java.util.concurrent.atomic.AtomicBoolean; 7 | import java.util.concurrent.atomic.AtomicInteger; 8 | 9 | import io.grpc.Metadata; 10 | import io.grpc.StatusRuntimeException; 11 | import org.apache.avro.Schema; 12 | import org.apache.avro.generic.GenericRecord; 13 | 14 | import com.google.protobuf.ByteString; 15 | import com.salesforce.eventbus.protobuf.*; 16 | 17 | import io.grpc.stub.StreamObserver; 18 | import utility.CommonContext; 19 | import utility.ExampleConfigurations; 20 | 21 | /** 22 | * A single-topic subscriber that consumes events using Event Bus API Subscribe RPC. The example demonstrates how to: 23 | * - implement a long-lived subscription to a single topic 24 | * - a basic flow control strategy 25 | * - a basic retry strategy. 26 | * 27 | * Example: 28 | * ./run.sh genericpubsub.Subscribe 29 | * 30 | * @author sidd0610 31 | */ 32 | public class Subscribe extends CommonContext { 33 | 34 | public static int BATCH_SIZE; 35 | public static int MAX_RETRIES = 3; 36 | public static String ERROR_REPLAY_ID_VALIDATION_FAILED = "fetch.replayid.validation.failed"; 37 | public static String ERROR_REPLAY_ID_INVALID = "fetch.replayid.corrupted"; 38 | public static String ERROR_SERVICE_UNAVAILABLE = "service.unavailable"; 39 | public static int SERVICE_UNAVAILABLE_WAIT_BEFORE_RETRY_SECONDS = 5; 40 | public static ExampleConfigurations exampleConfigurations; 41 | public static AtomicBoolean isActive = new AtomicBoolean(false); 42 | public static AtomicInteger retriesLeft = new AtomicInteger(MAX_RETRIES); 43 | private StreamObserver serverStream; 44 | private Map schemaCache = new ConcurrentHashMap<>(); 45 | private AtomicInteger receivedEvents = new AtomicInteger(0); 46 | private final StreamObserver responseStreamObserver; 47 | private final ReplayPreset replayPreset; 48 | private final ByteString customReplayId; 49 | private final ScheduledExecutorService retryScheduler; 50 | // Replay should be stored in replay store as bytes since replays are opaque. 51 | private volatile ByteString storedReplay; 52 | private final boolean processChangedFields; 53 | 54 | public Subscribe(ExampleConfigurations exampleConfigurations) { 55 | super(exampleConfigurations); 56 | isActive.set(true); 57 | this.exampleConfigurations = exampleConfigurations; 58 | this.BATCH_SIZE = exampleConfigurations.getNumberOfEventsToSubscribeInEachFetchRequest(); 59 | this.responseStreamObserver = getDefaultResponseStreamObserver(); 60 | this.setupTopicDetails(exampleConfigurations.getTopic(), false, false); 61 | this.replayPreset = exampleConfigurations.getReplayPreset(); 62 | this.customReplayId = exampleConfigurations.getReplayId(); 63 | this.retryScheduler = Executors.newScheduledThreadPool(1); 64 | this.processChangedFields = exampleConfigurations.getProcessChangedFields(); 65 | } 66 | 67 | public Subscribe(ExampleConfigurations exampleConfigurations, StreamObserver responseStreamObserver) { 68 | super(exampleConfigurations); 69 | isActive.set(true); 70 | this.exampleConfigurations = exampleConfigurations; 71 | this.BATCH_SIZE = exampleConfigurations.getNumberOfEventsToSubscribeInEachFetchRequest(); 72 | this.responseStreamObserver = responseStreamObserver; 73 | this.setupTopicDetails(exampleConfigurations.getTopic(), false, false); 74 | this.replayPreset = exampleConfigurations.getReplayPreset(); 75 | this.customReplayId = exampleConfigurations.getReplayId(); 76 | this.retryScheduler = Executors.newScheduledThreadPool(1); 77 | this.processChangedFields = exampleConfigurations.getProcessChangedFields(); 78 | } 79 | 80 | /** 81 | * Function to start the subscription. 82 | */ 83 | public void startSubscription() { 84 | logger.info("Subscription started for topic: " + busTopicName + "."); 85 | fetch(BATCH_SIZE, busTopicName, replayPreset, customReplayId); 86 | // Thread being blocked here for demonstration of this specific example. Blocking the thread in production is not recommended. 87 | while(isActive.get()) { 88 | waitInMillis(5_000); 89 | logger.info("Subscription Active. Received a total of " + receivedEvents.get() + " events."); 90 | } 91 | } 92 | 93 | /** Helper function to send FetchRequests. 94 | * @param providedBatchSize 95 | * @param providedTopicName 96 | * @param providedReplayPreset 97 | * @param providedReplayId 98 | */ 99 | public void fetch(int providedBatchSize, String providedTopicName, ReplayPreset providedReplayPreset, ByteString providedReplayId) { 100 | serverStream = asyncStub.subscribe(this.responseStreamObserver); 101 | FetchRequest.Builder fetchRequestBuilder = FetchRequest.newBuilder() 102 | .setNumRequested(providedBatchSize) 103 | .setTopicName(providedTopicName) 104 | .setReplayPreset(providedReplayPreset); 105 | if (providedReplayPreset == ReplayPreset.CUSTOM) { 106 | logger.info("Subscription has Replay Preset set to CUSTOM. In this case, the events will be delivered from provided ReplayId."); 107 | fetchRequestBuilder.setReplayId(providedReplayId); 108 | } 109 | serverStream.onNext(fetchRequestBuilder.build()); 110 | } 111 | 112 | /** 113 | * Function to decide the delay (in ms) in sending FetchRequests using 114 | * Binary Exponential Backoff - Waits for 2^(Max Number of Retries - Retries Left) * 1000. 115 | */ 116 | public long getBackoffWaitTime() { 117 | long waitTime = (long) (Math.pow(2, MAX_RETRIES - retriesLeft.get()) * 1000); 118 | return waitTime; 119 | } 120 | 121 | /** 122 | * Helper function to halt the current thread. 123 | */ 124 | public void waitInMillis(long duration) { 125 | synchronized (this) { 126 | try { 127 | this.wait(duration); 128 | } catch (InterruptedException e) { 129 | throw new RuntimeException(e); 130 | } 131 | } 132 | } 133 | 134 | /** 135 | * Creates a StreamObserver for handling the incoming FetchResponse messages from the server. 136 | * 137 | * @return 138 | */ 139 | private StreamObserver getDefaultResponseStreamObserver() { 140 | return new StreamObserver() { 141 | @Override 142 | public void onNext(FetchResponse fetchResponse) { 143 | logger.info("Received batch of " + fetchResponse.getEventsList().size() + " events"); 144 | logger.info("RPC ID: " + fetchResponse.getRpcId()); 145 | for(ConsumerEvent ce : fetchResponse.getEventsList()) { 146 | try { 147 | processEvent(ce); 148 | } catch (Exception e) { 149 | logger.info(e.toString()); 150 | } 151 | receivedEvents.addAndGet(1); 152 | } 153 | // Latest replayId stored for any future FetchRequests with CUSTOM ReplayPreset. 154 | // NOTE: Replay IDs are opaque in nature and should be stored and used as bytes without any conversion. 155 | storedReplay = fetchResponse.getLatestReplayId(); 156 | 157 | // Reset retry count 158 | if (retriesLeft.get() != MAX_RETRIES) { 159 | retriesLeft.set(MAX_RETRIES); 160 | } 161 | 162 | // Implementing a basic flow control strategy where the next fetchRequest is sent only after the 163 | // requested number of events in the previous fetchRequest(s) are received. 164 | // NOTE: This block may need to be implemented before the processing of events if event processing takes 165 | // a long time. There is a 70s timeout period during which, if pendingNumRequested is 0 and no events are 166 | // further requested then the stream will be closed. 167 | if (fetchResponse.getPendingNumRequested() == 0) { 168 | fetchMore(BATCH_SIZE); 169 | } 170 | } 171 | 172 | @Override 173 | public void onError(Throwable t) { 174 | printStatusRuntimeException("Error during Subscribe", (Exception) t); 175 | logger.info("Retries remaining: " + retriesLeft.get()); 176 | if (retriesLeft.get() == 0) { 177 | logger.info("Exhausted all retries. Closing Subscription."); 178 | isActive.set(false); 179 | } else { 180 | retriesLeft.decrementAndGet(); 181 | Metadata trailers = ((StatusRuntimeException)t).getTrailers() != null ? ((StatusRuntimeException)t).getTrailers() : null; 182 | String errorCode = (trailers != null && trailers.get(Metadata.Key.of("error-code", Metadata.ASCII_STRING_MARSHALLER)) != null) ? 183 | trailers.get(Metadata.Key.of("error-code", Metadata.ASCII_STRING_MARSHALLER)) : null; 184 | 185 | // Closing the old stream for sanity 186 | serverStream.onCompleted(); 187 | 188 | ReplayPreset retryReplayPreset = ReplayPreset.LATEST; 189 | ByteString retryReplayId = null; 190 | long retryDelay = getBackoffWaitTime(); 191 | 192 | // Retry strategies that can be implemented based on the error type. 193 | if(errorCode != null && !errorCode.isEmpty()) { 194 | if (errorCode.contains(ERROR_REPLAY_ID_VALIDATION_FAILED) || errorCode.contains(ERROR_REPLAY_ID_INVALID)) { 195 | logger.info("Invalid or no replayId provided in FetchRequest for CUSTOM Replay. Trying again with EARLIEST Replay."); 196 | retryReplayPreset = ReplayPreset.EARLIEST; 197 | } else if (errorCode.contains(ERROR_SERVICE_UNAVAILABLE)) { 198 | logger.info("Service currently unavailable. Trying again with LATEST Replay."); 199 | retryDelay = SERVICE_UNAVAILABLE_WAIT_BEFORE_RETRY_SECONDS * 1000; 200 | } else { 201 | if (storedReplay != null) { 202 | logger.info("Retrying with Stored Replay."); 203 | retryReplayPreset = ReplayPreset.CUSTOM; 204 | retryReplayId = getStoredReplay(); 205 | } else { 206 | logger.info("Retrying with LATEST Replay."); 207 | } 208 | 209 | } 210 | } else { 211 | logger.info("Unknown error. Retrying with LATEST Replay."); 212 | } 213 | logger.info(String.format("Retrying in %s ms.", retryDelay)); 214 | retryScheduler.schedule(new RetryRequestSender(retryReplayPreset, retryReplayId), retryDelay, TimeUnit.MILLISECONDS); 215 | } 216 | } 217 | 218 | @Override 219 | public void onCompleted() { 220 | logger.info("Call completed by server. Closing Subscription."); 221 | isActive.set(false); 222 | } 223 | }; 224 | } 225 | 226 | /** 227 | * A Runnable class that is used to send the FetchRequests by making a new Subscribe call while retrying on 228 | * receiving an error. This is done in order to avoid blocking the thread while waiting for retries. This class is 229 | * passed to the ScheduledExecutorService which will asynchronously send the FetchRequests during retries. 230 | */ 231 | private class RetryRequestSender implements Runnable { 232 | private ReplayPreset retryReplayPreset; 233 | private ByteString retryReplayId; 234 | public RetryRequestSender(ReplayPreset replayPreset, ByteString replayId) { 235 | this.retryReplayPreset = replayPreset; 236 | this.retryReplayId = replayId; 237 | } 238 | 239 | @Override 240 | public void run() { 241 | fetch(BATCH_SIZE, busTopicName, retryReplayPreset, retryReplayId); 242 | logger.info("Retry FetchRequest Sent."); 243 | } 244 | } 245 | 246 | /** 247 | * Helper function to process the events received. 248 | */ 249 | private void processEvent(ConsumerEvent ce) throws IOException { 250 | Schema writerSchema = getSchema(ce.getEvent().getSchemaId()); 251 | this.storedReplay = ce.getReplayId(); 252 | GenericRecord record = deserialize(writerSchema, ce.getEvent().getPayload()); 253 | logger.info("Received event with payload: " + record.toString() + " with schema name: " + writerSchema.getName()); 254 | if (processChangedFields) { 255 | // This example expands the changedFields bitmap field in ChangeEventHeader. 256 | // To expand the other bitmap fields, i.e., diffFields and nulledFields, replicate or modify this code. 257 | processAndPrintBitmapFields(writerSchema, record, "changedFields"); 258 | } 259 | } 260 | 261 | /** 262 | * Helper function to get the schema of an event if it does not already exist in the schema cache. 263 | */ 264 | public Schema getSchema(String schemaId) { 265 | return schemaCache.computeIfAbsent(schemaId, id -> { 266 | SchemaRequest request = SchemaRequest.newBuilder().setSchemaId(id).build(); 267 | String schemaJson = blockingStub.getSchema(request).getSchemaJson(); 268 | return (new Schema.Parser()).parse(schemaJson); 269 | }); 270 | } 271 | 272 | /** 273 | * Helps keep the subscription active by sending FetchRequests at regular intervals. 274 | * 275 | * @param numEvents 276 | */ 277 | public void fetchMore(int numEvents) { 278 | FetchRequest fetchRequest = FetchRequest.newBuilder().setTopicName(this.busTopicName) 279 | .setNumRequested(numEvents).build(); 280 | serverStream.onNext(fetchRequest); 281 | } 282 | 283 | /** 284 | * General getters and setters. 285 | */ 286 | public AtomicInteger getReceivedEvents() { 287 | return receivedEvents; 288 | } 289 | 290 | public void updateReceivedEvents(int delta) { 291 | receivedEvents.addAndGet(delta); 292 | } 293 | 294 | public int getBatchSize() { 295 | return BATCH_SIZE; 296 | } 297 | public ByteString getStoredReplay() { 298 | return storedReplay; 299 | } 300 | 301 | public void setStoredReplay(ByteString storedReplay) { 302 | this.storedReplay = storedReplay; 303 | } 304 | 305 | /** 306 | * Closes the connection when the task is complete. 307 | */ 308 | @Override 309 | public synchronized void close() { 310 | try { 311 | if (serverStream != null) { 312 | serverStream.onCompleted(); 313 | } 314 | if (retryScheduler != null) { 315 | retryScheduler.shutdown(); 316 | } 317 | } catch (Exception e) { 318 | logger.info(e.toString()); 319 | } 320 | super.close(); 321 | } 322 | 323 | public static void main(String args[]) throws IOException { 324 | ExampleConfigurations exampleConfigurations = new ExampleConfigurations("arguments.yaml"); 325 | 326 | // Using the try-with-resource statement. The CommonContext class implements AutoCloseable in 327 | // order to close the resources used. 328 | try (Subscribe subscribe = new Subscribe(exampleConfigurations)) { 329 | subscribe.startSubscription(); 330 | } catch (Exception e) { 331 | printStatusRuntimeException("Error during Subscribe", e); 332 | } 333 | } 334 | } 335 | -------------------------------------------------------------------------------- /java/src/main/java/utility/APISessionCredentials.java: -------------------------------------------------------------------------------- 1 | package utility; 2 | 3 | import java.util.UUID; 4 | import java.util.concurrent.Executor; 5 | 6 | import org.slf4j.Logger; 7 | import org.slf4j.LoggerFactory; 8 | 9 | import io.grpc.CallCredentials; 10 | import io.grpc.Metadata; 11 | 12 | /** 13 | * The APISessionCredentials class extends the CallCredentials class of gRPC to add important 14 | * credential information, i.e., tenantId, accessToken and instanceUrl to every request made to 15 | * Pub/Sub API. 16 | */ 17 | public class APISessionCredentials extends CallCredentials { 18 | 19 | // Instance url of the customer org 20 | public static final Metadata.Key INSTANCE_URL_KEY = keyOf("instanceUrl"); 21 | // Session token of the customer 22 | public static final Metadata.Key SESSION_TOKEN_KEY = keyOf("accessToken"); 23 | // Tenant Id of the customer org 24 | public static final Metadata.Key TENANT_ID_KEY = keyOf("tenantId"); 25 | 26 | private String instanceURL; 27 | private String tenantId; 28 | private String token; 29 | 30 | private static final Logger log = LoggerFactory.getLogger(APISessionCredentials.class); 31 | 32 | public APISessionCredentials(String tenantId, String instanceURL, String token) { 33 | this.instanceURL = instanceURL; 34 | this.tenantId = tenantId; 35 | this.token = token; 36 | } 37 | 38 | @Override 39 | public void applyRequestMetadata(RequestInfo requestInfo, Executor executor, MetadataApplier metadataApplier) { 40 | log.debug("API session credentials applied to " + requestInfo.getMethodDescriptor()); 41 | Metadata headers = new Metadata(); 42 | headers.put(INSTANCE_URL_KEY, instanceURL); 43 | headers.put(TENANT_ID_KEY, tenantId); 44 | headers.put(SESSION_TOKEN_KEY, token); 45 | metadataApplier.apply(headers); 46 | } 47 | 48 | @Override 49 | public void thisUsesUnstableApi() { 50 | 51 | } 52 | 53 | private static Metadata.Key keyOf(String name) { 54 | return Metadata.Key.of(name, Metadata.ASCII_STRING_MARSHALLER); 55 | } 56 | 57 | public String getToken() { 58 | return token; 59 | } 60 | } -------------------------------------------------------------------------------- /java/src/main/java/utility/CommonContext.java: -------------------------------------------------------------------------------- 1 | package utility; 2 | 3 | import java.io.*; 4 | import java.nio.ByteBuffer; 5 | import java.util.ArrayList; 6 | import java.util.List; 7 | import java.util.Map; 8 | import java.util.concurrent.TimeUnit; 9 | 10 | import org.apache.avro.Schema; 11 | import org.apache.avro.generic.GenericData; 12 | import org.apache.avro.generic.GenericDatumReader; 13 | import org.apache.avro.generic.GenericRecord; 14 | import org.apache.avro.generic.GenericRecordBuilder; 15 | import org.apache.avro.io.BinaryDecoder; 16 | import org.apache.avro.io.DatumReader; 17 | import org.apache.avro.io.DecoderFactory; 18 | import org.eclipse.jetty.client.HttpClient; 19 | import org.eclipse.jetty.client.HttpProxy; 20 | import org.slf4j.Logger; 21 | import org.slf4j.LoggerFactory; 22 | 23 | import com.google.common.base.CaseFormat; 24 | import com.google.protobuf.ByteString; 25 | import com.salesforce.eventbus.protobuf.*; 26 | 27 | import io.grpc.*; 28 | 29 | import static utility.EventParser.getFieldListFromBitmap; 30 | 31 | /** 32 | * The CommonContext class provides a list of member variables and functions that is used across 33 | * all examples for various purposes like setting up the HttpClient, CallCredentials, stubs for 34 | * sending requests, generating events etc. 35 | */ 36 | public class CommonContext implements AutoCloseable { 37 | 38 | protected static final Logger logger = LoggerFactory.getLogger(CommonContext.class.getClass()); 39 | 40 | protected final ManagedChannel channel; 41 | protected final PubSubGrpc.PubSubStub asyncStub; 42 | protected final PubSubGrpc.PubSubBlockingStub blockingStub; 43 | 44 | protected final HttpClient httpClient; 45 | protected final SessionTokenService sessionTokenService; 46 | protected final CallCredentials callCredentials; 47 | 48 | protected String tenantGuid; 49 | protected String busTopicName; 50 | protected TopicInfo topicInfo; 51 | protected SchemaInfo schemaInfo; 52 | protected String sessionToken; 53 | 54 | public CommonContext(final ExampleConfigurations options) { 55 | String grpcHost = options.getPubsubHost(); 56 | int grpcPort = options.getPubsubPort(); 57 | logger.info("Using grpcHost {} and grpcPort {}", grpcHost, grpcPort); 58 | 59 | if (options.usePlaintextChannel()) { 60 | channel = ManagedChannelBuilder.forAddress(grpcHost, grpcPort).usePlaintext().build(); 61 | } else { 62 | channel = ManagedChannelBuilder.forAddress(grpcHost, grpcPort).build(); 63 | } 64 | 65 | httpClient = setupHttpClient(); 66 | sessionTokenService = new SessionTokenService(httpClient); 67 | 68 | callCredentials = setupCallCredentials(options); 69 | sessionToken = ((APISessionCredentials) callCredentials).getToken(); 70 | 71 | Channel interceptedChannel = ClientInterceptors.intercept(channel, new XClientTraceIdClientInterceptor()); 72 | 73 | asyncStub = PubSubGrpc.newStub(interceptedChannel).withCallCredentials(callCredentials); 74 | blockingStub = PubSubGrpc.newBlockingStub(interceptedChannel).withCallCredentials(callCredentials); 75 | } 76 | 77 | /** 78 | * Helper function to setup the HttpClient used for sending requests. 79 | */ 80 | private HttpClient setupHttpClient() { 81 | HttpClient httpClient = new HttpClient(); 82 | Map env = System.getenv(); 83 | 84 | String httpProxy = env.get("HTTP_PROXY"); 85 | if (httpProxy != null) { 86 | String[] httpProxyParts = httpProxy.split(":"); 87 | httpClient.getProxyConfiguration().getProxies() 88 | .add(new HttpProxy(httpProxyParts[0], Integer.parseInt(httpProxyParts[1]))); 89 | } 90 | 91 | try { 92 | httpClient.start(); 93 | } catch (Exception e) { 94 | logger.error("cannot create HTTP client", e); 95 | } 96 | return httpClient; 97 | } 98 | 99 | /** 100 | * Helper function to setup the CallCredentials of the requests. 101 | * 102 | * @param options Command line arguments passed. 103 | * @return CallCredentials 104 | */ 105 | public CallCredentials setupCallCredentials(ExampleConfigurations options) { 106 | if (options.getAccessToken() != null) { 107 | try { 108 | return sessionTokenService.loginWithAccessToken(options.getLoginUrl(), 109 | options.getAccessToken(), options.getTenantId()); 110 | } catch (Exception e) { 111 | close(); 112 | throw new IllegalArgumentException("cannot log in with access token", e); 113 | } 114 | } else if (options.getUsername() != null && options.getPassword() != null) { 115 | try { 116 | return sessionTokenService.login(options.getLoginUrl(), 117 | options.getUsername(), options.getPassword(), options.useProvidedLoginUrl()); 118 | } catch (Exception e) { 119 | close(); 120 | throw new IllegalArgumentException("cannot log in with username/password", e); 121 | } 122 | } else { 123 | logger.warn("Please use either username/password or session token for authentication"); 124 | close(); 125 | return null; 126 | } 127 | } 128 | 129 | /** 130 | * Helper function to setup the topic details in the PublishUnary, PublishStream and 131 | * SubscribeStream examples. Function also checks whether the topic under consideration 132 | * can publish or subscribe. 133 | * 134 | * @param topicName name of the topic 135 | * @param pubOrSubMode publish mode if true, subscribe mode if false 136 | * @param fetchSchema specify whether schema info has to be fetched 137 | */ 138 | protected void setupTopicDetails(final String topicName, final boolean pubOrSubMode, final boolean fetchSchema) { 139 | if (topicName != null && !topicName.isEmpty()) { 140 | try { 141 | topicInfo = blockingStub.getTopic(TopicRequest.newBuilder().setTopicName(topicName).build()); 142 | tenantGuid = topicInfo.getTenantGuid(); 143 | busTopicName = topicInfo.getTopicName(); 144 | 145 | if (pubOrSubMode && !topicInfo.getCanPublish()) { 146 | throw new IllegalArgumentException( 147 | "Topic " + topicInfo.getTopicName() + " is not available for publish"); 148 | } 149 | 150 | if (!pubOrSubMode && !topicInfo.getCanSubscribe()) { 151 | throw new IllegalArgumentException( 152 | "Topic " + topicInfo.getTopicName() + " is not available for subscribe"); 153 | } 154 | 155 | if (fetchSchema) { 156 | SchemaRequest schemaRequest = SchemaRequest.newBuilder().setSchemaId(topicInfo.getSchemaId()) 157 | .build(); 158 | schemaInfo = blockingStub.getSchema(schemaRequest); 159 | } 160 | } catch (final Exception ex) { 161 | logger.error("Error during fetching topic", ex); 162 | close(); 163 | throw ex; 164 | } 165 | } 166 | } 167 | 168 | /** 169 | * Helper function to convert the replayId in long to ByteString type. 170 | * 171 | * @param replayValue value of the replayId in long 172 | * @return ByteString value of the replayId 173 | */ 174 | public static ByteString getReplayIdFromLong(long replayValue) { 175 | ByteBuffer buffer = ByteBuffer.allocate(8); 176 | buffer.putLong(replayValue); 177 | buffer.flip(); 178 | 179 | return ByteString.copyFrom(buffer); 180 | } 181 | 182 | /** 183 | * Helper function to create an event. 184 | * Currently generates event message for the topic "Order Event". Modify the fields 185 | * accordingly for an event of your choice. 186 | * 187 | * @param schema schema of the topic 188 | * @return 189 | */ 190 | public GenericRecord createEventMessage(Schema schema) { 191 | // Update CreatedById with the appropriate User Id from your org. 192 | return new GenericRecordBuilder(schema).set("CreatedDate", System.currentTimeMillis()) 193 | .set("CreatedById", "").set("Order_Number__c", "1") 194 | .set("City__c", "Los Angeles").set("Amount__c", 35.0).build(); 195 | } 196 | 197 | /** 198 | * Helper function to create an event with a counter appended to 199 | * the end of a Text field. Used while publishing multiple events. 200 | * Currently generates event message for the topic "Order Event". Modify the fields 201 | * accordingly for an event of your choice. 202 | * 203 | * @param schema schema of the topic 204 | * @param counter counter to be appended towards the end of any Text Field 205 | * @return 206 | */ 207 | public GenericRecord createEventMessage(Schema schema, final int counter) { 208 | // Update CreatedById with the appropriate User Id from your org. 209 | return new GenericRecordBuilder(schema).set("CreatedDate", System.currentTimeMillis()) 210 | .set("CreatedById", "").set("Order_Number__c", String.valueOf(counter+1)) 211 | .set("City__c", "Los Angeles").set("Amount__c", 35.0).build(); 212 | } 213 | 214 | public List createEventMessages(Schema schema, final int numEvents) { 215 | 216 | String[] orderNumbers = {"99","100","101","102","103"}; 217 | String[] cities = {"Los Angeles", "New York", "San Francisco", "San Jose", "Boston"}; 218 | Double[] amounts = {35.0, 20.0, 2.0, 123.0, 180.0}; 219 | 220 | // Update CreatedById with the appropriate User Id from your org. 221 | List events = new ArrayList<>(); 222 | for (int i=0; i").set("Order_Number__c", orderNumbers[i % 5]) 225 | .set("City__c", cities[i % 5]).set("Amount__c", amounts[i % 5]).build()); 226 | } 227 | 228 | return events; 229 | } 230 | 231 | 232 | /** 233 | * Helper function to print the gRPC exception and trailers while a 234 | * StatusRuntimeException is caught 235 | * 236 | * @param context 237 | * @param e 238 | */ 239 | public static final void printStatusRuntimeException(final String context, final Exception e) { 240 | logger.error(context); 241 | 242 | if (e instanceof StatusRuntimeException) { 243 | final StatusRuntimeException expected = (StatusRuntimeException)e; 244 | logger.error(" === GRPC Exception ===", e); 245 | Metadata trailers = ((StatusRuntimeException)e).getTrailers(); 246 | logger.error(" === Trailers ==="); 247 | trailers.keys().stream().forEach(t -> { 248 | logger.error("[Trailer] = " + t + " [Value] = " 249 | + trailers.get(Metadata.Key.of(t, Metadata.ASCII_STRING_MARSHALLER))); 250 | }); 251 | } else { 252 | logger.error(" === Exception ===", e); 253 | } 254 | } 255 | 256 | /** 257 | * Helper function to deserialize the event payload received in bytes. 258 | * 259 | * @param schema 260 | * @param payload 261 | * @return 262 | * @throws IOException 263 | */ 264 | public static GenericRecord deserialize(Schema schema, ByteString payload) throws IOException { 265 | DatumReader reader = new GenericDatumReader(schema); 266 | ByteArrayInputStream in = new ByteArrayInputStream(payload.toByteArray()); 267 | BinaryDecoder decoder = DecoderFactory.get().directBinaryDecoder(in, null); 268 | return reader.read(null, decoder); 269 | } 270 | 271 | /** 272 | * Helper function to process and print bitmap fields 273 | * 274 | * @param schema 275 | * @param record 276 | * @param bitmapField 277 | * @return 278 | */ 279 | public static void processAndPrintBitmapFields(Schema schema, GenericRecord record, String bitmapField) { 280 | String bitmapFieldPascal = CaseFormat.LOWER_CAMEL.to(CaseFormat.UPPER_CAMEL, bitmapField); 281 | try { 282 | List changedFields = getFieldListFromBitmap(schema, 283 | (GenericData.Record) record.get("ChangeEventHeader"), bitmapField); 284 | if (!changedFields.isEmpty()) { 285 | logger.info("============================"); 286 | logger.info(" " + bitmapFieldPascal + " "); 287 | logger.info("============================"); 288 | for (String field : changedFields) { 289 | logger.info(field); 290 | } 291 | logger.info("============================\n"); 292 | } else { 293 | logger.info("No " + bitmapFieldPascal + " found\n"); 294 | } 295 | } catch (Exception e) { 296 | logger.info("Trying to process " + bitmapFieldPascal + " on unsupported events or no " + 297 | bitmapFieldPascal + " found. Error: " + e.getMessage() + "\n"); 298 | } 299 | } 300 | 301 | /** 302 | * Helper function to setup Subscribe configurations in some examples. 303 | * 304 | * @param requiredParams 305 | * @param topic 306 | * @return 307 | */ 308 | public static ExampleConfigurations setupSubscriberParameters(ExampleConfigurations requiredParams, String topic, int numberOfEvents) { 309 | ExampleConfigurations subParams = new ExampleConfigurations(); 310 | setCommonParameters(subParams, requiredParams); 311 | subParams.setTopic(topic); 312 | subParams.setReplayPreset(ReplayPreset.LATEST); 313 | subParams.setNumberOfEventsToSubscribeInEachFetchRequest(numberOfEvents); 314 | return subParams; 315 | } 316 | 317 | /** 318 | * Helper function to setup Publish configurations in some examples. 319 | * 320 | * @param requiredParams 321 | * @param topic 322 | * @return 323 | */ 324 | public static ExampleConfigurations setupPublisherParameters(ExampleConfigurations requiredParams, String topic) { 325 | ExampleConfigurations pubParams = new ExampleConfigurations(); 326 | setCommonParameters(pubParams, requiredParams); 327 | pubParams.setTopic(topic); 328 | return pubParams; 329 | } 330 | 331 | /** 332 | * Helper function to setup common configurations for publish and subscribe operations. 333 | * 334 | * @param ep 335 | * @param requiredParams 336 | */ 337 | private static void setCommonParameters(ExampleConfigurations ep, ExampleConfigurations requiredParams) { 338 | ep.setLoginUrl(requiredParams.getLoginUrl()); 339 | ep.setPubsubHost(requiredParams.getPubsubHost()); 340 | ep.setPubsubPort(requiredParams.getPubsubPort()); 341 | if (requiredParams.getUsername() != null && requiredParams.getPassword() != null) { 342 | ep.setUsername(requiredParams.getUsername()); 343 | ep.setPassword(requiredParams.getPassword()); 344 | } else { 345 | ep.setAccessToken(requiredParams.getAccessToken()); 346 | ep.setTenantId(requiredParams.getTenantId()); 347 | } 348 | ep.setPlaintextChannel(requiredParams.usePlaintextChannel()); 349 | } 350 | 351 | /** 352 | * General getters. 353 | */ 354 | public String getSessionToken() { 355 | return sessionToken; 356 | } 357 | 358 | /** 359 | * Implementation of the close() function from AutoCloseable interface for relinquishing the 360 | * resources used in the try-with-resource blocks in the examples and the resources used 361 | * in this class. 362 | */ 363 | @Override 364 | public void close() { 365 | if (httpClient != null) { 366 | try { 367 | httpClient.stop(); 368 | } catch (Throwable t) { 369 | logger.warn("Cannot stop session HTTP client", t); 370 | } 371 | } 372 | 373 | try { 374 | channel.shutdown().awaitTermination(20, TimeUnit.SECONDS); 375 | } catch (Throwable t) { 376 | logger.warn("Cannot shutdown GRPC channel", t); 377 | } 378 | } 379 | } -------------------------------------------------------------------------------- /java/src/main/java/utility/EventParser.java: -------------------------------------------------------------------------------- 1 | package utility; 2 | 3 | import java.io.IOException; 4 | import java.util.ArrayList; 5 | import java.util.BitSet; 6 | import java.util.List; 7 | import java.util.ListIterator; 8 | 9 | import org.apache.avro.Schema; 10 | import org.apache.avro.Schema.Type; 11 | import org.apache.avro.generic.GenericDatumReader; 12 | import org.apache.avro.generic.GenericRecord; 13 | import org.apache.avro.generic.GenericData.Array; 14 | import org.apache.avro.generic.GenericData.Record; 15 | import org.apache.avro.io.DatumReader; 16 | import org.apache.avro.util.Utf8; 17 | 18 | /** 19 | * A utility class used to generate the field names from bitmap encoded values. 20 | * 21 | * @author pozil 22 | */ 23 | public class EventParser { 24 | 25 | private Schema schema; 26 | private DatumReader datumReader; 27 | 28 | public EventParser(Schema schema) { 29 | this.schema = schema; 30 | this.datumReader = new GenericDatumReader(schema); 31 | } 32 | 33 | /** 34 | * Retrieves the list of fields from a bitmap encoded value. 35 | * 36 | * @param schema 37 | * @param eventHeader 38 | * @param fieldName 39 | * @return 40 | * @throws IOException 41 | */ 42 | public static List getFieldListFromBitmap(Schema schema, Record eventHeader, String fieldName) 43 | throws IOException { 44 | @SuppressWarnings("unchecked") 45 | Array utf8Values = (Array) eventHeader.get(fieldName); 46 | List values = new ArrayList<>(); 47 | for (Utf8 utf8Value : utf8Values) { 48 | values.add(utf8Value.toString()); 49 | } 50 | expandBitmap(schema, values); 51 | return values; 52 | } 53 | 54 | /** 55 | * Translate a bitmap-compressed field list into its expanded representation as 56 | * a list of field names 57 | */ 58 | public static void expandBitmap(Schema schema, List val) { 59 | if (val != null && !val.isEmpty()) { 60 | // replace top field level bitmap with list of fields 61 | if (val.get(0).startsWith("0x")) { 62 | String bitMap = val.get(0); 63 | val.addAll(0, fieldNamesFromBitmap(schema, bitMap)); 64 | val.remove(bitMap); 65 | } 66 | // replace parentPos-nestedNulledBitMap with list of fields too 67 | if ((val.get(val.size() - 1)).contains("-")) { 68 | for (ListIterator itr = val.listIterator(); itr.hasNext();) { 69 | 70 | String[] bitmapMapStrings = (itr.next()).split("-"); 71 | if (bitmapMapStrings.length < 2) 72 | continue; // that's the first top level field bitmap; 73 | 74 | // interpret the parent field name from mapping of parentFieldPos -> 75 | // childFieldbitMap 76 | Schema.Field parentField = schema.getFields().get(Integer.valueOf(bitmapMapStrings[0])); 77 | Schema childSchema = getValueSchema(parentField.schema()); 78 | 79 | if (childSchema.getType().equals(Schema.Type.RECORD)) { // make sure we're really dealing with 80 | // compound field 81 | int nestedSize = childSchema.getFields().size(); 82 | String parentFieldName = parentField.name(); 83 | 84 | // interpret the child field names from mapping of parentFieldPos -> 85 | // childFieldbitMap 86 | List fullFieldNames = new ArrayList<>(); 87 | fieldNamesFromBitmap(childSchema, bitmapMapStrings[1]).stream() 88 | .map(col -> parentFieldName + "." + col).forEach(fullFieldNames::add); 89 | if (fullFieldNames.size() > 0) { 90 | itr.remove(); 91 | // when all nested fields under a compound got nulled out at once by customer, 92 | // we recognize the top level field instead of trying to list every single 93 | // nested field 94 | if (fullFieldNames.size() == nestedSize) { 95 | itr.add(parentFieldName); 96 | } else { 97 | fullFieldNames.stream().forEach(itr::add); 98 | } 99 | } 100 | } 101 | } 102 | } 103 | } 104 | } 105 | 106 | /** 107 | * Convert bitmap representation into list of fields based on Avro schema 108 | * 109 | * @param schema schema 110 | * @param bitmap bitmap of nulled fields 111 | * @return list of fields corresponding to bitmap 112 | */ 113 | private static List fieldNamesFromBitmap(Schema schema, String bitmap) { 114 | BitSet bitSet = convertHexStringToBitSet(bitmap); 115 | List fieldList = new ArrayList<>(); 116 | bitSet.stream().mapToObj(pos -> schema.getFields().get(pos).name()) 117 | .forEach(fieldName -> fieldList.add(fieldName)); 118 | return fieldList; 119 | } 120 | 121 | /** 122 | * Converts a hexadecimal string into a BitSet 123 | * 124 | * @param hex 125 | * @return BitSet 126 | */ 127 | private static BitSet convertHexStringToBitSet(String hex) { 128 | // Parse hex string as bytes 129 | String s = hex.substring(2); // Strip out 0x prefix 130 | int len = s.length(); 131 | byte[] bytes = new byte[len / 2]; 132 | for (int i = 0; i < len; i += 2) { 133 | // using left shift operator on every character 134 | bytes[i / 2] = (byte) ((Character.digit(s.charAt(i), 16) << 4) + Character.digit(s.charAt(i + 1), 16)); 135 | } 136 | // Reverse bytes 137 | len /= 2; 138 | byte[] reversedBytes = new byte[len]; 139 | for (int i = 0; i < len; i++) { 140 | reversedBytes[i] = bytes[len - i - 1]; 141 | } 142 | // Return value as BitSet 143 | return BitSet.valueOf(reversedBytes); 144 | } 145 | 146 | /** 147 | * Get the value type of an "optional" schema, which is a union of [null, 148 | * valueSchema] 149 | * 150 | * @param schema 151 | * @return value schema or the original schema if it does not look like optional 152 | */ 153 | private static Schema getValueSchema(Schema schema) { 154 | if (schema.getType() == Schema.Type.UNION) { 155 | List types = schema.getTypes(); 156 | if (types.size() == 2 && types.get(0).getType() == Type.NULL) { 157 | // Optional is a union of (null, ), return the underlying type 158 | return types.get(1); 159 | } else if (types.size() == 2 && types.get(0).getType() == Type.STRING) { 160 | // for required Switchable_PersonName 161 | return schema.getTypes().get(1); 162 | } else if (types.size() == 3 && types.get(0).getType() == Type.NULL 163 | && types.get(1).getType() == Type.STRING) { 164 | // for optional Switchable_PersonName 165 | return schema.getTypes().get(2); 166 | } 167 | } 168 | return schema; 169 | } 170 | } -------------------------------------------------------------------------------- /java/src/main/java/utility/ExampleConfigurations.java: -------------------------------------------------------------------------------- 1 | package utility; 2 | 3 | import java.io.FileInputStream; 4 | import java.io.IOException; 5 | import java.io.InputStream; 6 | import java.util.HashMap; 7 | 8 | import org.yaml.snakeyaml.Yaml; 9 | 10 | import com.google.protobuf.ByteString; 11 | import com.salesforce.eventbus.protobuf.ReplayPreset; 12 | 13 | /** 14 | * The ExampleConfigurations class is used for setting up the configurations for running the examples. 15 | * The configurations can be read from a YAML file or created directly via an object. It also sets 16 | * default values when an optional configuration is not specified. 17 | */ 18 | public class ExampleConfigurations { 19 | private String username; 20 | private String password; 21 | private String loginUrl; 22 | private String tenantId; 23 | private String accessToken; 24 | private String pubsubHost; 25 | private Integer pubsubPort; 26 | private String topic; 27 | private Integer numberOfEventsToPublish; 28 | private Boolean singlePublishRequest; 29 | private Integer numberOfEventsToSubscribeInEachFetchRequest; 30 | private Boolean processChangedFields; 31 | private Boolean plaintextChannel; 32 | private Boolean providedLoginUrl; 33 | private ReplayPreset replayPreset; 34 | private ByteString replayId; 35 | private String managedSubscriptionId; 36 | private String developerName; 37 | 38 | public ExampleConfigurations() { 39 | this(null, null, null, null, null, 40 | null, null, null, 5, false, 5, false, 41 | false, false, ReplayPreset.LATEST, null, null, null); 42 | } 43 | public ExampleConfigurations(String filename) throws IOException { 44 | 45 | Yaml yaml = new Yaml(); 46 | InputStream inputStream = new FileInputStream("src/main/resources/"+filename); 47 | HashMap obj = yaml.load(inputStream); 48 | 49 | // Reading Required Parameters 50 | this.loginUrl = obj.get("LOGIN_URL").toString(); 51 | this.pubsubHost = obj.get("PUBSUB_HOST").toString(); 52 | this.pubsubPort = Integer.parseInt(obj.get("PUBSUB_PORT").toString()); 53 | 54 | // Reading Optional Parameters 55 | this.username = obj.get("USERNAME") == null ? null : obj.get("USERNAME").toString(); 56 | this.password = obj.get("PASSWORD") == null ? null : obj.get("PASSWORD").toString(); 57 | this.topic = obj.get("TOPIC") == null ? "/event/Order_Event__e" : obj.get("TOPIC").toString(); 58 | this.tenantId = obj.get("TENANT_ID") == null ? null : obj.get("TENANT_ID").toString(); 59 | this.accessToken = obj.get("ACCESS_TOKEN") == null ? null : obj.get("ACCESS_TOKEN").toString(); 60 | this.numberOfEventsToPublish = obj.get("NUMBER_OF_EVENTS_TO_PUBLISH") == null ? 61 | 5 : Integer.parseInt(obj.get("NUMBER_OF_EVENTS_TO_PUBLISH").toString()); 62 | this.singlePublishRequest = obj.get("SINGLE_PUBLISH_REQUEST") == null ? 63 | false : Boolean.parseBoolean(obj.get("SINGLE_PUBLISH_REQUEST").toString()); 64 | this.numberOfEventsToSubscribeInEachFetchRequest = obj.get("NUMBER_OF_EVENTS_IN_FETCHREQUEST") == null ? 65 | 5 : Integer.parseInt(obj.get("NUMBER_OF_EVENTS_IN_FETCHREQUEST").toString()); 66 | this.processChangedFields = obj.get("PROCESS_CHANGE_EVENT_HEADER_FIELDS") == null ? 67 | false : Boolean.parseBoolean(obj.get("PROCESS_CHANGE_EVENT_HEADER_FIELDS").toString()); 68 | this.plaintextChannel = obj.get("USE_PLAINTEXT_CHANNEL") != null && Boolean.parseBoolean(obj.get("USE_PLAINTEXT_CHANNEL").toString()); 69 | this.providedLoginUrl = obj.get("USE_PROVIDED_LOGIN_URL") != null && Boolean.parseBoolean(obj.get("USE_PROVIDED_LOGIN_URL").toString()); 70 | 71 | if (obj.get("REPLAY_PRESET") != null) { 72 | if (obj.get("REPLAY_PRESET").toString().equals("EARLIEST")) { 73 | this.replayPreset = ReplayPreset.EARLIEST; 74 | } else if (obj.get("REPLAY_PRESET").toString().equals("CUSTOM")) { 75 | this.replayPreset = ReplayPreset.CUSTOM; 76 | this.replayId = getByteStringFromReplayIdInputString(obj.get("REPLAY_ID").toString()); 77 | } else { 78 | this.replayPreset = ReplayPreset.LATEST; 79 | } 80 | } else { 81 | this.replayPreset = ReplayPreset.LATEST; 82 | } 83 | 84 | this.developerName = obj.get("MANAGED_SUB_DEVELOPER_NAME") == null ? null : obj.get("MANAGED_SUB_DEVELOPER_NAME").toString(); 85 | this.managedSubscriptionId = obj.get("MANAGED_SUB_ID") == null ? null : obj.get("MANAGED_SUB_ID").toString(); 86 | } 87 | 88 | public ExampleConfigurations(String username, String password, String loginUrl, 89 | String pubsubHost, int pubsubPort, String topic) { 90 | this(username, password, loginUrl, null, null, pubsubHost, pubsubPort, topic, 91 | 5, false, Integer.MAX_VALUE, false, false, false, ReplayPreset.LATEST, null, null, null); 92 | } 93 | 94 | public ExampleConfigurations(String username, String password, String loginUrl, String tenantId, String accessToken, 95 | String pubsubHost, Integer pubsubPort, String topic, Integer numberOfEventsToPublish, 96 | Boolean singlePublishRequest, Integer numberOfEventsToSubscribeInEachFetchRequest, 97 | Boolean processChangedFields, Boolean plaintextChannel, Boolean providedLoginUrl, 98 | ReplayPreset replayPreset, ByteString replayId, String devName, String managedSubId) { 99 | this.username = username; 100 | this.password = password; 101 | this.loginUrl = loginUrl; 102 | this.tenantId = tenantId; 103 | this.accessToken = accessToken; 104 | this.pubsubHost = pubsubHost; 105 | this.pubsubPort = pubsubPort; 106 | this.topic = topic; 107 | this.singlePublishRequest = singlePublishRequest; 108 | this.numberOfEventsToPublish = numberOfEventsToPublish; 109 | this.numberOfEventsToSubscribeInEachFetchRequest = numberOfEventsToSubscribeInEachFetchRequest; 110 | this.processChangedFields = processChangedFields; 111 | this.plaintextChannel = plaintextChannel; 112 | this.providedLoginUrl = providedLoginUrl; 113 | this.replayPreset = replayPreset; 114 | this.replayId = replayId; 115 | this.developerName = devName; 116 | this.managedSubscriptionId = managedSubId; 117 | } 118 | 119 | public String getUsername() { 120 | return username; 121 | } 122 | 123 | public void setUsername(String username) { 124 | this.username = username; 125 | } 126 | 127 | public String getPassword() { 128 | return password; 129 | } 130 | 131 | public void setPassword(String password) { 132 | this.password = password; 133 | } 134 | 135 | public String getLoginUrl() { 136 | return loginUrl; 137 | } 138 | 139 | public void setLoginUrl(String loginUrl) { 140 | this.loginUrl = loginUrl; 141 | } 142 | 143 | public String getTenantId() { 144 | return tenantId; 145 | } 146 | 147 | public void setTenantId(String tenantId) { 148 | this.tenantId = tenantId; 149 | } 150 | 151 | public String getAccessToken() { 152 | return accessToken; 153 | } 154 | 155 | public void setAccessToken(String accessToken) { 156 | this.accessToken = accessToken; 157 | } 158 | 159 | public String getPubsubHost() { 160 | return pubsubHost; 161 | } 162 | 163 | public void setPubsubHost(String pubsubHost) { 164 | this.pubsubHost = pubsubHost; 165 | } 166 | 167 | public int getPubsubPort() { 168 | return pubsubPort; 169 | } 170 | 171 | public void setPubsubPort(int pubsubPort) { 172 | this.pubsubPort = pubsubPort; 173 | } 174 | 175 | public Integer getNumberOfEventsToPublish() { 176 | return numberOfEventsToPublish; 177 | } 178 | 179 | public void setNumberOfEventsToPublish(Integer numberOfEventsToPublish) { 180 | this.numberOfEventsToPublish = numberOfEventsToPublish; 181 | } 182 | 183 | public Boolean getSinglePublishRequest() { 184 | return singlePublishRequest; 185 | } 186 | 187 | public void setSinglePublishRequest(Boolean singlePublishRequest) { 188 | this.singlePublishRequest = singlePublishRequest; 189 | } 190 | 191 | public int getNumberOfEventsToSubscribeInEachFetchRequest() { 192 | return numberOfEventsToSubscribeInEachFetchRequest; 193 | } 194 | 195 | public void setNumberOfEventsToSubscribeInEachFetchRequest(int numberOfEventsToSubscribeInEachFetchRequest) { 196 | this.numberOfEventsToSubscribeInEachFetchRequest = numberOfEventsToSubscribeInEachFetchRequest; 197 | } 198 | 199 | public Boolean getProcessChangedFields() { 200 | return processChangedFields; 201 | } 202 | 203 | public void setProcessChangedFields(Boolean processChangedFields) { 204 | this.processChangedFields = processChangedFields; 205 | } 206 | 207 | public boolean usePlaintextChannel() { 208 | return plaintextChannel; 209 | } 210 | 211 | public void setPlaintextChannel(boolean plaintextChannel) { 212 | this.plaintextChannel = plaintextChannel; 213 | } 214 | 215 | public Boolean useProvidedLoginUrl() { 216 | return providedLoginUrl; 217 | } 218 | 219 | public String getTopic() { 220 | return topic; 221 | } 222 | 223 | public void setTopic(String topic) { 224 | this.topic = topic; 225 | } 226 | 227 | public void setProvidedLoginUrl(Boolean providedLoginUrl) { 228 | this.providedLoginUrl = providedLoginUrl; 229 | } 230 | 231 | public ReplayPreset getReplayPreset() { 232 | return replayPreset; 233 | } 234 | 235 | public void setReplayPreset(ReplayPreset replayPreset) { 236 | this.replayPreset = replayPreset; 237 | } 238 | 239 | public ByteString getReplayId() { 240 | return replayId; 241 | } 242 | 243 | public void setReplayId(ByteString replayId) { 244 | this.replayId = replayId; 245 | } 246 | 247 | public String getManagedSubscriptionId() { 248 | return managedSubscriptionId; 249 | } 250 | 251 | public void setManagedSubscriptionId(String managedSubscriptionId) { 252 | this.managedSubscriptionId = managedSubscriptionId; 253 | } 254 | 255 | public String getDeveloperName() { 256 | return developerName; 257 | } 258 | 259 | public void setDeveloperName(String developerName) { 260 | this.developerName = developerName; 261 | } 262 | 263 | 264 | /** 265 | * NOTE: replayIds are meant to be opaque (See docs: https://developer.salesforce.com/docs/platform/pub-sub-api/guide/intro.html) 266 | * and this is used for example purposes only. A long-lived subscription client will use the stored replay to 267 | * resubscribe on failure. The stored replay should be in bytes and not in any other form. 268 | */ 269 | public ByteString getByteStringFromReplayIdInputString(String input) { 270 | ByteString replayId; 271 | String[] values = input.substring(1, input.length()-2).split(","); 272 | byte[] b = new byte[values.length]; 273 | int i=0; 274 | for (String x : values) { 275 | if (x.strip().length() != 0) { 276 | b[i++] = (byte)Integer.parseInt(x.strip()); 277 | } 278 | } 279 | replayId = ByteString.copyFrom(b); 280 | return replayId; 281 | } 282 | } 283 | -------------------------------------------------------------------------------- /java/src/main/java/utility/SessionTokenService.java: -------------------------------------------------------------------------------- 1 | package utility; 2 | 3 | import java.io.ByteArrayInputStream; 4 | import java.io.UnsupportedEncodingException; 5 | import java.net.ConnectException; 6 | import java.net.URL; 7 | import java.nio.ByteBuffer; 8 | 9 | import javax.xml.parsers.SAXParser; 10 | import javax.xml.parsers.SAXParserFactory; 11 | 12 | import org.eclipse.jetty.client.HttpClient; 13 | import org.eclipse.jetty.client.api.ContentResponse; 14 | import org.eclipse.jetty.client.api.Request; 15 | import org.eclipse.jetty.client.util.ByteBufferContentProvider; 16 | import org.slf4j.Logger; 17 | import org.slf4j.LoggerFactory; 18 | import org.xml.sax.Attributes; 19 | import org.xml.sax.SAXException; 20 | import org.xml.sax.helpers.DefaultHandler; 21 | 22 | /** 23 | * The SessionTokenService class is used for logging into the org used by the customer using the 24 | * Salesforce SOAP API and retrieving the tenandId and session token which will be used to create 25 | * CallCredentials. It also has a static subclass that parses the LoginResponse. 26 | */ 27 | public class SessionTokenService { 28 | private static final Logger LOGGER = LoggerFactory.getLogger(SessionTokenService.class); 29 | 30 | private static final String ENV_END = ""; 31 | private static final String ENV_START = 32 | ""; 35 | 36 | // The enterprise SOAP API endpoint used for the login call 37 | private static final String SERVICES_SOAP_PARTNER_ENDPOINT = "/services/Soap/u/43.0/"; 38 | 39 | // HttpClient is thread safe and meant to be shared; assume callers are managing its life cycle correctly 40 | private final HttpClient httpClient; 41 | 42 | public SessionTokenService(HttpClient httpClient) { 43 | if (httpClient == null) { 44 | throw new IllegalArgumentException("HTTP client cannot be null"); 45 | } 46 | 47 | this.httpClient = httpClient; 48 | } 49 | 50 | /** 51 | * Function to login with the username/password of the client. 52 | * 53 | * @param loginEndpoint 54 | * @param user 55 | * @param pwd 56 | * @param useProvidedLoginUrl 57 | * @return 58 | * @throws Exception 59 | */ 60 | public APISessionCredentials login(String loginEndpoint, String user, String pwd, boolean useProvidedLoginUrl) throws Exception { 61 | URL endpoint; 62 | endpoint = new URL(new URL(loginEndpoint), SERVICES_SOAP_PARTNER_ENDPOINT); 63 | LOGGER.trace("requesting session token from {}", endpoint); 64 | Request post = httpClient.POST(endpoint.toURI()); 65 | post.content(new ByteBufferContentProvider("text/xml", ByteBuffer.wrap(soapXmlForLogin(user, pwd)))); 66 | post.header("SOAPAction", "''"); 67 | post.header("PrettyPrint", "Yes"); 68 | ContentResponse response = post.send(); 69 | LoginResponseParser parser = parse(response); 70 | 71 | final String token = parser.sessionId; 72 | if (token == null || parser.serverUrl == null) { 73 | throw new ConnectException(String.format("Unable to login: %s", parser.faultstring)); 74 | } 75 | 76 | if (null == parser.organizationId) { 77 | throw new ConnectException( 78 | String.format("Unable to login: organization id is not found in the response")); 79 | } 80 | 81 | String url; 82 | if (useProvidedLoginUrl) { 83 | url = loginEndpoint; 84 | } else { 85 | // Form url to this format: https://na44.stmfa.stm.salesforce.com 86 | URL soapEndpoint = new URL(parser.serverUrl); 87 | url = soapEndpoint.getProtocol() + "://" + soapEndpoint.getHost(); 88 | // Adding port info for local app setup 89 | if (soapEndpoint.getPort() > -1) { 90 | url += ":" + soapEndpoint.getPort(); 91 | } 92 | } 93 | 94 | LOGGER.debug("created session token credentials for {} from {}", parser.organizationId, url); 95 | return new APISessionCredentials(parser.organizationId, url, token); 96 | } 97 | 98 | /** 99 | * Function to login with the tenantId and session token of the client. 100 | * 101 | * @param loginEndpoint 102 | * @param accessToken 103 | * @param tenantId 104 | * @return 105 | */ 106 | public APISessionCredentials loginWithAccessToken(String loginEndpoint, String accessToken, String tenantId) { 107 | return new APISessionCredentials(tenantId, loginEndpoint, accessToken); 108 | } 109 | 110 | private static class LoginResponseParser extends DefaultHandler { 111 | 112 | private String buffer; 113 | private String faultstring; 114 | 115 | private boolean reading = false; 116 | private String serverUrl; 117 | private String sessionId; 118 | private String organizationId; 119 | 120 | @Override 121 | public void characters(char[] ch, int start, int length) { 122 | if (reading) { 123 | buffer = new String(ch, start, length); 124 | } 125 | } 126 | 127 | @Override 128 | public void endElement(String uri, String localName, String qName) { 129 | reading = false; 130 | switch (localName) { 131 | case "organizationId": 132 | organizationId = buffer; 133 | break; 134 | case "sessionId": 135 | sessionId = buffer; 136 | break; 137 | case "serverUrl": 138 | serverUrl = buffer; 139 | break; 140 | case "faultstring": 141 | faultstring = buffer; 142 | break; 143 | default: 144 | } 145 | buffer = null; 146 | } 147 | 148 | @Override 149 | public void startElement(String uri, String localName, String qName, Attributes attributes) { 150 | switch (localName) { 151 | case "sessionId": 152 | case "serverUrl": 153 | case "faultstring": 154 | case "organizationId": 155 | reading = true; 156 | break; 157 | default: 158 | } 159 | } 160 | } 161 | 162 | private static LoginResponseParser parse(ContentResponse response) throws Exception { 163 | try { 164 | SAXParserFactory spf = SAXParserFactory.newInstance(); 165 | spf.setFeature("http://xml.org/sax/features/external-general-entities", false); 166 | spf.setFeature("http://xml.org/sax/features/external-parameter-entities", false); 167 | spf.setFeature("http://apache.org/xml/features/nonvalidating/load-external-dtd", false); 168 | spf.setFeature("http://apache.org/xml/features/disallow-doctype-decl", true); 169 | spf.setNamespaceAware(true); 170 | SAXParser saxParser = spf.newSAXParser(); 171 | 172 | LoginResponseParser parser = new LoginResponseParser(); 173 | 174 | saxParser.parse(new ByteArrayInputStream(response.getContent()), parser); 175 | 176 | return parser; 177 | } catch (SAXException e) { 178 | throw new Exception(String.format("Unable to login: %s::%s", response.getStatus(), response.getReason())); 179 | } 180 | } 181 | 182 | private static byte[] soapXmlForLogin(String username, String password) throws UnsupportedEncodingException { 183 | return (ENV_START + " " + " " + username + "" + " " 184 | + password + "" + " " + ENV_END).getBytes("UTF-8"); 185 | } 186 | } 187 | -------------------------------------------------------------------------------- /java/src/main/java/utility/XClientTraceIdClientInterceptor.java: -------------------------------------------------------------------------------- 1 | package utility; 2 | 3 | import java.util.UUID; 4 | 5 | import org.slf4j.Logger; 6 | import org.slf4j.LoggerFactory; 7 | 8 | import io.grpc.*; 9 | 10 | public class XClientTraceIdClientInterceptor implements ClientInterceptor { 11 | private static final Logger logger = LoggerFactory.getLogger(XClientTraceIdClientInterceptor.class.getClass()); 12 | private static final Metadata.Key X_CLIENT_TRACE_ID = Metadata.Key.of("x-client-trace-id", Metadata.ASCII_STRING_MARSHALLER); 13 | 14 | @Override 15 | public ClientCall interceptCall(MethodDescriptor method, 16 | CallOptions callOptions, Channel next) { 17 | return new ForwardingClientCall.SimpleForwardingClientCall<>(next.newCall(method, callOptions)) { 18 | 19 | @Override 20 | public void start(Listener responseListener, Metadata headers) { 21 | String xClientTraceId = UUID.randomUUID().toString(); 22 | headers.put(X_CLIENT_TRACE_ID, xClientTraceId); 23 | logger.info("sending request for xClientTraceId {}", xClientTraceId); 24 | 25 | super.start(new ForwardingClientCallListener.SimpleForwardingClientCallListener<>(responseListener) { 26 | @Override 27 | public void onClose(Status status, Metadata trailers) { 28 | logger.info("request completed for xClientTraceId {} with status {}", xClientTraceId, status); 29 | super.onClose(status, trailers); 30 | } 31 | }, headers); 32 | } 33 | }; 34 | } 35 | } 36 | -------------------------------------------------------------------------------- /java/src/main/resources/arguments.yaml: -------------------------------------------------------------------------------- 1 | # 'arguments.yaml' contains the required and optional configurations for running the examples. 2 | # 3 | # Note: Please ensure to specify a value of `null` to all optional configurations when 4 | # you do not wish to specify a value for the same. Some optional configurations will be 5 | # initialised with default values specified below. 6 | 7 | # ========================= 8 | # Required Configurations: 9 | # ========================= 10 | # Pub/Sub API Endpoint 11 | PUBSUB_HOST: api.pubsub.salesforce.com 12 | # Pub/Sub API Host 13 | PUBSUB_PORT: 7443 14 | # Your Salesforce Login URL 15 | LOGIN_URL: null 16 | 17 | # For authentication, you can use either username/password or accessToken/tenantId types. 18 | # Either one of the combinations is required. Please specify `null` values to the unused type. 19 | # Your Salesforce Username 20 | USERNAME: null 21 | # Your Salesforce Password 22 | PASSWORD: null 23 | # Your Salesforce org Tenant ID 24 | TENANT_ID: null 25 | # Your Salesforce Session Token 26 | ACCESS_TOKEN: null 27 | 28 | # ========================= 29 | # Optional Configurations: 30 | # ========================= 31 | # Topic to publish/subscribe to (default: /event/Order_Event__e) 32 | TOPIC: null 33 | # Number of Events to publish in single or separate batches (default: 5) 34 | # Used only by PublishStream.java 35 | NUMBER_OF_EVENTS_TO_PUBLISH: null 36 | # Indicates whether to add events to a single PublishRequest (true) or 37 | # in different PublishRequests (default: false) 38 | # Used only by PublishStream.java 39 | SINGLE_PUBLISH_REQUEST: null 40 | # Number of events to subscribe to in each FetchRequest/ManagedFetchRequest (default: 5) 41 | NUMBER_OF_EVENTS_IN_FETCHREQUEST: null 42 | # ReplayPreset (Accepted Values: {EARLIEST, LATEST (default), CUSTOM}) 43 | REPLAY_PRESET: null 44 | # Replay ID in ByteString 45 | REPLAY_ID: null 46 | # Flag to enable/disable processing of bitmap fields in ChangeEventHeader in Subscribe and 47 | # ManagedSubscribe examples for change data capture events (default: false) 48 | PROCESS_CHANGE_EVENT_HEADER_FIELDS: null 49 | 50 | # ManagedSubscribe RPC parameters 51 | # For ManagedSubscribe.java, either supply the developer name or the ID of ManagedEventSubscription 52 | MANAGED_SUB_DEVELOPER_NAME: null 53 | MANAGED_SUB_ID: null -------------------------------------------------------------------------------- /java/src/main/resources/logback.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | %d [%thread] %logger{36} - %msg%n 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | -------------------------------------------------------------------------------- /python/InventoryAppExample/InventoryApp.py: -------------------------------------------------------------------------------- 1 | """ 2 | InventoryApp.py 3 | 4 | This is a subscriber client that listens for Change Data Capture events for the 5 | Opportunity object and publishes `/event/NewOrderConfirmation__e` events. In 6 | the example, this file would be hosted somewhere outside of Salesforce. The `if 7 | __debug__` conditionals are to slow down the speed of the app for demoing 8 | purposes. 9 | """ 10 | import os, sys, avro 11 | 12 | dir_path = os.path.dirname(os.path.realpath(__file__)) 13 | parent_dir_path = os.path.abspath(os.path.join(dir_path, os.pardir)) 14 | sys.path.insert(0, parent_dir_path) 15 | 16 | from datetime import datetime, timedelta 17 | import logging 18 | 19 | from PubSub import PubSub 20 | import pubsub_api_pb2 as pb2 21 | from utils.ClientUtil import command_line_input 22 | import time 23 | from util.ChangeEventHeaderUtility import process_bitmap 24 | 25 | my_publish_topic = '/event/NewOrderConfirmation__e' 26 | 27 | 28 | def make_publish_request(schema_id, record_id, obj): 29 | """ 30 | Creates a PublishRequest per the proto file. 31 | """ 32 | req = pb2.PublishRequest( 33 | topic_name=my_publish_topic, 34 | events=generate_producer_events(schema_id, record_id, obj)) 35 | return req 36 | 37 | 38 | def generate_producer_events(schema_id, record_id, obj): 39 | """ 40 | Encodes the data to be sent in the event and creates a ProducerEvent per 41 | the proto file. 42 | """ 43 | schema = obj.get_schema_json(schema_id) 44 | dt = datetime.now() + timedelta(days=5) 45 | payload = { 46 | "CreatedDate": int(datetime.now().timestamp()), 47 | "CreatedById": '005R0000000cw06IAA', 48 | "OpptyRecordId__c": record_id, 49 | "EstimatedDeliveryDate__c": int(dt.timestamp()), 50 | "Weight__c": 58.2} 51 | req = { 52 | "schema_id": schema_id, 53 | "payload": obj.encode(schema, payload), 54 | } 55 | return [req] 56 | 57 | 58 | def process_order(event, pubsub): 59 | """ 60 | This is a callback that gets passed to the `PubSub.subscribe()` method. It 61 | decodes the payload of the received event and extracts the opportunity ID. 62 | Next, it calls a helper function to publish the 63 | `/event/NewOrderConfirmation__e` event. For simplicity, this sample uses an 64 | estimated delivery date of five days from the current date. When no events 65 | are received within a certain time period, the API's subscribe method sends 66 | keepalive messages and the latest replay ID through this callback. 67 | """ 68 | if event.events: 69 | print("Number of events received in FetchResponse: ", len(event.events)) 70 | # If all requested events are delivered, release the semaphore 71 | # so that a new FetchRequest gets sent by `PubSub.fetch_req_stream()`. 72 | if event.pending_num_requested == 0: 73 | pubsub.release_subscription_semaphore() 74 | 75 | for evt in event.events: 76 | payload_bytes = evt.event.payload 77 | schema_id = evt.event.schema_id 78 | json_schema = pubsub.get_schema_json(schema_id) 79 | decoded_event = pubsub.decode(pubsub.get_schema_json(schema_id), 80 | payload_bytes) 81 | 82 | print("Received event payload: \n", decoded_event) 83 | # A change event contains the ChangeEventHeader field. Check if received event is a change event. 84 | if 'ChangeEventHeader' in decoded_event: 85 | # Decode the bitmap fields contained within the ChangeEventHeader. For example, decode the 'changedFields' field. 86 | # An example to process bitmap in 'changedFields' 87 | changed_fields = decoded_event['ChangeEventHeader']['changedFields'] 88 | converted_changed_fields = process_bitmap(avro.schema.parse(json_schema), changed_fields) 89 | print("Change Type: " + decoded_event['ChangeEventHeader']['changeType']) 90 | print("=========== Changed Fields =============") 91 | print(converted_changed_fields) 92 | print("=========================================") 93 | # Do not process updates made by the SalesforceListener app to the opportunity record delivery date 94 | if decoded_event['ChangeEventHeader']['changeOrigin'].find('client=SalesforceListener') != -1: 95 | print("Skipping change event because it is an update to the delivery date by SalesforceListener.") 96 | return 97 | 98 | record_id = decoded_event['ChangeEventHeader']['recordIds'][0] 99 | 100 | if __debug__: 101 | time.sleep(10) 102 | print("> Received new order! Processing order...") 103 | if __debug__: 104 | time.sleep(4) 105 | print(" Done! Order replicated in inventory system.") 106 | if __debug__: 107 | time.sleep(2) 108 | print("> Calculating estimated delivery date...") 109 | if __debug__: 110 | time.sleep(2) 111 | print(" Done! Sending estimated delivery date back to Salesforce.") 112 | if __debug__: 113 | time.sleep(10) 114 | 115 | topic_info = pubsub.get_topic(topic_name=my_publish_topic) 116 | 117 | # Publish NewOrderConfirmation__e event 118 | res = pubsub.stub.Publish(make_publish_request(topic_info.schema_id, record_id, pubsub), 119 | metadata=pubsub.metadata) 120 | if res.results[0].replay_id: 121 | print("> Event published successfully.") 122 | else: 123 | print("> Failed publishing event.") 124 | else: 125 | print("[", time.strftime('%b %d, %Y %l:%M%p %Z'), "] The subscription is active.") 126 | 127 | # The replay_id is used to resubscribe after this position in the stream if the client disconnects. 128 | # Implement storage of replay for resubscribe!!! 129 | event.latest_replay_id 130 | 131 | 132 | def run(argument_dict): 133 | cdc_listener = PubSub(argument_dict) 134 | cdc_listener.auth() 135 | 136 | # Subscribe to Opportunity CDC events 137 | cdc_listener.subscribe('/data/OpportunityChangeEvent', "LATEST", "", 1, process_order) 138 | 139 | 140 | if __name__ == '__main__': 141 | argument_dict = command_line_input(sys.argv[1:]) 142 | logging.basicConfig() 143 | run(argument_dict) 144 | -------------------------------------------------------------------------------- /python/InventoryAppExample/PubSub.py: -------------------------------------------------------------------------------- 1 | """ 2 | PubSub.py 3 | 4 | This file defines the class `PubSub`, which contains common functionality for 5 | both publisher and subscriber clients. 6 | """ 7 | 8 | import io 9 | import threading 10 | import xml.etree.ElementTree as et 11 | from datetime import datetime 12 | 13 | import avro.io 14 | import avro.schema 15 | import certifi 16 | import grpc 17 | import requests 18 | 19 | import pubsub_api_pb2 as pb2 20 | import pubsub_api_pb2_grpc as pb2_grpc 21 | from urllib.parse import urlparse 22 | from utils.ClientUtil import load_properties 23 | 24 | properties = load_properties("../resources/application.properties") 25 | 26 | with open(certifi.where(), 'rb') as f: 27 | secure_channel_credentials = grpc.ssl_channel_credentials(f.read()) 28 | 29 | 30 | def get_argument(key, argument_dict): 31 | if key in argument_dict.keys(): 32 | return argument_dict[key] 33 | else: 34 | return properties.get(key) 35 | 36 | 37 | class PubSub(object): 38 | """ 39 | Class with helpers to use the Salesforce Pub/Sub API. 40 | """ 41 | 42 | json_schema_dict = {} 43 | 44 | def __init__(self, argument_dict): 45 | self.url = get_argument('url', argument_dict) 46 | self.username = get_argument('username', argument_dict) 47 | self.password = get_argument('password', argument_dict) 48 | self.metadata = None 49 | grpc_host = get_argument('grpcHost', argument_dict) 50 | grpc_port = get_argument('grpcPort', argument_dict) 51 | pubsub_url = grpc_host + ":" + grpc_port 52 | channel = grpc.secure_channel(pubsub_url, secure_channel_credentials) 53 | self.stub = pb2_grpc.PubSubStub(channel) 54 | self.session_id = None 55 | self.pb2 = pb2 56 | self.topic_name = get_argument('topic', argument_dict) 57 | # If the API version is not provided as an argument, use a default value 58 | if get_argument('apiVersion', argument_dict) == None: 59 | self.apiVersion = '57.0' 60 | else: 61 | # Otherwise, get the version from the argument 62 | self.apiVersion = get_argument('apiVersion', argument_dict) 63 | """ 64 | Semaphore used for subscriptions. This keeps the subscription stream open 65 | to receive events and to notify when to send the next FetchRequest. 66 | See Python Quick Start for more information. 67 | https://developer.salesforce.com/docs/platform/pub-sub-api/guide/qs-python-quick-start.html 68 | There is probably a better way to do this. This is only sample code. Please 69 | use your own discretion when writing your production Pub/Sub API client. 70 | Make sure to use only one semaphore per subscribe call if you are planning 71 | to share the same instance of PubSub. 72 | """ 73 | self.semaphore = threading.Semaphore(1) 74 | 75 | def auth(self): 76 | """ 77 | Sends a login request to the Salesforce SOAP API to retrieve a session 78 | token. The session token is bundled with other identifying information 79 | to create a tuple of metadata headers, which are needed for every RPC 80 | call. 81 | """ 82 | url_suffix = '/services/Soap/u/' + self.apiVersion + '/' 83 | headers = {'content-type': 'text/xml', 'SOAPAction': 'Login'} 84 | xml = "" + \ 87 | "" 90 | res = requests.post(self.url + url_suffix, data=xml, headers=headers) 91 | res_xml = et.fromstring(res.content.decode('utf-8'))[0][0][0] 92 | 93 | try: 94 | url_parts = urlparse(res_xml[3].text) 95 | self.url = "{}://{}".format(url_parts.scheme, url_parts.netloc) 96 | self.session_id = res_xml[4].text 97 | except IndexError: 98 | print("An exception occurred. Check the response XML below:", 99 | res.__dict__) 100 | 101 | # Get org ID from UserInfo 102 | uinfo = res_xml[6] 103 | # Org ID 104 | self.tenant_id = uinfo[8].text; 105 | 106 | # Set metadata headers 107 | self.metadata = (('accesstoken', self.session_id), 108 | ('instanceurl', self.url), 109 | ('tenantid', self.tenant_id)) 110 | 111 | def release_subscription_semaphore(self): 112 | """ 113 | Release semaphore so FetchRequest can be sent 114 | """ 115 | self.semaphore.release() 116 | 117 | def make_fetch_request(self, topic, replay_type, replay_id, num_requested): 118 | """ 119 | Creates a FetchRequest per the proto file. 120 | """ 121 | replay_preset = None 122 | match replay_type: 123 | case "LATEST": 124 | replay_preset = pb2.ReplayPreset.LATEST 125 | case "EARLIEST": 126 | replay_preset = pb2.ReplayPreset.EARLIEST 127 | case "CUSTOM": 128 | replay_preset = pb2.ReplayPreset.CUSTOM 129 | case _: 130 | raise ValueError('Invalid Replay Type ' + replay_type) 131 | return pb2.FetchRequest( 132 | topic_name=topic, 133 | replay_preset=replay_preset, 134 | replay_id=bytes.fromhex(replay_id), 135 | num_requested=num_requested) 136 | 137 | def fetch_req_stream(self, topic, replay_type, replay_id, num_requested): 138 | """ 139 | Returns a FetchRequest stream for the Subscribe RPC. 140 | """ 141 | while True: 142 | # Only send FetchRequest when needed. Semaphore release indicates need for new FetchRequest 143 | self.semaphore.acquire() 144 | print("Sending Fetch Request") 145 | yield self.make_fetch_request(topic, replay_type, replay_id, num_requested) 146 | 147 | def encode(self, schema, payload): 148 | """ 149 | Uses Avro and the event schema to encode a payload. The `encode()` and 150 | `decode()` methods are helper functions to serialize and deserialize 151 | the payloads of events that clients will publish and receive using 152 | Avro. If you develop an implementation with a language other than 153 | Python, you will need to find an Avro library in that language that 154 | helps you encode and decode with Avro. When publishing an event, the 155 | plaintext payload needs to be Avro-encoded with the event schema for 156 | the API to accept it. When receiving an event, the Avro-encoded payload 157 | needs to be Avro-decoded with the event schema for you to read it in 158 | plaintext. 159 | """ 160 | schema = avro.schema.parse(schema) 161 | buf = io.BytesIO() 162 | encoder = avro.io.BinaryEncoder(buf) 163 | writer = avro.io.DatumWriter(schema) 164 | writer.write(payload, encoder) 165 | return buf.getvalue() 166 | 167 | def decode(self, schema, payload): 168 | """ 169 | Uses Avro and the event schema to decode a serialized payload. The 170 | `encode()` and `decode()` methods are helper functions to serialize and 171 | deserialize the payloads of events that clients will publish and 172 | receive using Avro. If you develop an implementation with a language 173 | other than Python, you will need to find an Avro library in that 174 | language that helps you encode and decode with Avro. When publishing an 175 | event, the plaintext payload needs to be Avro-encoded with the event 176 | schema for the API to accept it. When receiving an event, the 177 | Avro-encoded payload needs to be Avro-decoded with the event schema for 178 | you to read it in plaintext. 179 | """ 180 | schema = avro.schema.parse(schema) 181 | buf = io.BytesIO(payload) 182 | decoder = avro.io.BinaryDecoder(buf) 183 | reader = avro.io.DatumReader(schema) 184 | ret = reader.read(decoder) 185 | return ret 186 | 187 | def get_topic(self, topic_name): 188 | return self.stub.GetTopic(pb2.TopicRequest(topic_name=topic_name), 189 | metadata=self.metadata) 190 | 191 | def get_schema_json(self, schema_id): 192 | """ 193 | Uses GetSchema RPC to retrieve schema given a schema ID. 194 | """ 195 | # If the schema is not found in the dictionary, get the schema and store it in the dictionary 196 | if schema_id not in self.json_schema_dict or self.json_schema_dict[schema_id]==None: 197 | res = self.stub.GetSchema(pb2.SchemaRequest(schema_id=schema_id), metadata=self.metadata) 198 | self.json_schema_dict[schema_id] = res.schema_json 199 | 200 | return self.json_schema_dict[schema_id] 201 | 202 | def generate_producer_events(self, schema, schema_id): 203 | """ 204 | Encodes the data to be sent in the event and creates a ProducerEvent per 205 | the proto file. Change the below payload to match the schema used. 206 | """ 207 | payload = { 208 | "CreatedDate": int(datetime.now().timestamp()), 209 | "CreatedById": '005R0000000cw06IAA', # Your user ID 210 | "textt__c": 'Hello World' 211 | } 212 | req = { 213 | "schema_id": schema_id, 214 | "payload": self.encode(schema, payload) 215 | } 216 | return [req] 217 | 218 | def subscribe(self, topic, replay_type, replay_id, num_requested, callback): 219 | """ 220 | Calls the Subscribe RPC defined in the proto file and accepts a 221 | client-defined callback to handle any events that are returned by the 222 | API. It uses a semaphore to prevent the Python client from closing the 223 | connection prematurely (this is due to the way Python's GRPC library is 224 | designed and may not be necessary for other languages--Java, for 225 | example, does not need this). 226 | """ 227 | sub_stream = self.stub.Subscribe(self.fetch_req_stream(topic, replay_type, replay_id, num_requested), metadata=self.metadata) 228 | print("> Subscribed to", topic) 229 | for event in sub_stream: 230 | callback(event, self) 231 | 232 | def publish(self, topic_name, schema, schema_id): 233 | """ 234 | Publishes events to the specified Platform Event topic. 235 | """ 236 | 237 | return self.stub.Publish(self.pb2.PublishRequest( 238 | topic_name=topic_name, 239 | events=self.generate_producer_events(schema, 240 | schema_id)), 241 | metadata=self.metadata) -------------------------------------------------------------------------------- /python/InventoryAppExample/README.md: -------------------------------------------------------------------------------- 1 | # Pub/Sub API Example - Inventory App 2 | 3 | This example of the Pub/Sub API is meant to be a conceptual example only—the 4 | code is not a template, not meant for copying and pasting, and not intended to 5 | serve as anything other than a read-only learning resource. We encourage you to 6 | read through the code and this README in order to understand the logic 7 | underpinning the example, so that you can take the learnings and apply them to 8 | your own implementations. Also note that the way this example is structured is 9 | but one way to interact with the Pub/Sub API using Python. You are free to 10 | mirror the structure in your own code, but it is far from the only way to 11 | engage with the API. 12 | 13 | The example imagines a scenario in which salespeople closing opportunities in 14 | Salesforce need an "Estimated Delivery Date" field filled in by an integration 15 | between Salesforce and an external inventory app. When an opportunity is closed 16 | in Salesforce, a Change Data Capture event gets published by Salesforce. This 17 | event gets consumed by an inventory app (`InventoryApp.py`) hosted in an 18 | external system like AWS, which sets off the inventory process for the order, 19 | like packaging, shipping, etc. Once the inventory app has calculated the 20 | estimated delivery date for the order, it sends that information back to 21 | Salesforce in the payload of a `NewOrderConfirmation` event. On the Salesforce 22 | side, a subscriber client (`SalesforceListener.py`) receives the 23 | `NewOrderConfirmation` event and uses the date contained in the payload to 24 | update the very opportunity that just closed with its estimated delivery date. 25 | In this scenario, this enables the salesperson who closed the deal to report 26 | the estimated delivery date to their customer right away—the integration acts 27 | so quickly that the salesperson can see the estimated delivery date almost 28 | instantaneously after they close the opportunity. 29 | 30 | A video demonstrating this app in action can be found on the 31 | [TrailheaDX](https://www.salesforce.com/trailheadx) website. After 32 | registering/logging in, go to [Product & Partner 33 | Demos](https://www.salesforce.com/trailheadx) and click `Integrations & 34 | Analytics` > `Platform APIs` to watch it. 35 | 36 | The proto file for the API can be found [here](https://github.com/developerforce/pub-sub-api/blob/main/pubsub_api.proto). 37 | 38 | This example uses Python features that require Python version 3.10 or later, such as the `match` statement. 39 | 40 | To build a working client example in Python please follow [the Python Quick Start Guide.](https://developer.salesforce.com/docs/platform/pub-sub-api/guide/qs-python-quick-start.html) 41 | -------------------------------------------------------------------------------- /python/InventoryAppExample/SalesforceListener.py: -------------------------------------------------------------------------------- 1 | """ 2 | SalesforceListener.py 3 | 4 | This is a subscriber client that listens for `/event/NewOrderConfirmation__e` 5 | events published by the inventory app (`InventoryApp.py`). The `if __debug__` 6 | conditionals are to slow down the speed of the app for demoing purposes. 7 | """ 8 | 9 | import os, sys, avro 10 | 11 | dir_path = os.path.dirname(os.path.realpath(__file__)) 12 | parent_dir_path = os.path.abspath(os.path.join(dir_path, os.pardir)) 13 | sys.path.insert(0, parent_dir_path) 14 | 15 | from util.ChangeEventHeaderUtility import process_bitmap 16 | from datetime import datetime 17 | import json 18 | import logging 19 | import requests 20 | import time 21 | 22 | from PubSub import PubSub 23 | from utils.ClientUtil import command_line_input 24 | 25 | 26 | def process_confirmation(event, pubsub): 27 | """ 28 | This is a callback that gets passed to the `PubSub.subscribe()` method. It 29 | decodes the payload of the received event and extracts the opportunity ID 30 | and estimated delivery date. Using those two pieces of information, it 31 | updates the relevant opportunity with its estimated delivery date using the 32 | REST API. When no events are received within a certain time period, the 33 | API's subscribe method sends keepalive messages and the latest replay ID 34 | through this callback. 35 | """ 36 | 37 | if event.events: 38 | print("Number of events received in FetchResponse: ", len(event.events)) 39 | # If all requested events are delivered, release the semaphore 40 | # so that a new FetchRequest gets sent by `PubSub.fetch_req_stream()`. 41 | if event.pending_num_requested == 0: 42 | pubsub.release_subscription_semaphore() 43 | 44 | for evt in event.events: 45 | # Get the event payload and schema, then decode the payload 46 | payload_bytes = evt.event.payload 47 | json_schema = pubsub.get_schema_json(evt.event.schema_id) 48 | decoded_event = pubsub.decode(json_schema, payload_bytes) 49 | # print(decoded_event) 50 | # A change event contains the ChangeEventHeader field. Check if received event is a change event. 51 | if 'ChangeEventHeader' in decoded_event: 52 | # Decode the bitmap fields contained within the ChangeEventHeader. For example, decode the 'changedFields' field. 53 | # An example to process bitmap in 'changedFields' 54 | changed_fields = decoded_event['ChangeEventHeader']['changedFields'] 55 | print("Change Type: " + decoded_event['ChangeEventHeader']['changeType']) 56 | print("=========== Changed Fields =============") 57 | print(process_bitmap(avro.schema.parse(json_schema), changed_fields)) 58 | print("=========================================") 59 | print("> Received order confirmation! Updating estimated delivery date...") 60 | if __debug__: 61 | time.sleep(2) 62 | # Update the Desription field of the opportunity with the estimated delivery date with a REST request 63 | day = datetime.fromtimestamp(decoded_event['EstimatedDeliveryDate__c']).strftime('%Y-%m-%d') 64 | res = requests.patch(pubsub.url + "/services/data/v" + pubsub.apiVersion + "/sobjects/Opportunity/" 65 | + decoded_event['OpptyRecordId__c'], json.dumps({"Description": "Estimated Delivery Date: " + day}), 66 | headers={"Authorization": "Bearer " + pubsub.session_id, 67 | "Content-Type": "application/json", 68 | "Sforce-Call-Options": "client=SalesforceListener"}) 69 | print(" Done!", res) 70 | else: 71 | print("[", time.strftime('%b %d, %Y %l:%M%p %Z'), "] The subscription is active.") 72 | 73 | # The replay_id is used to resubscribe after this position in the stream if the client disconnects. 74 | # Implement storage of replay for resubscribe!!! 75 | event.latest_replay_id 76 | 77 | def run(argument_dict): 78 | sfdc_updater = PubSub(argument_dict) 79 | sfdc_updater.auth() 80 | 81 | # Subscribe to /event/NewOrderConfirmation__e events 82 | sfdc_updater.subscribe('/event/NewOrderConfirmation__e', "LATEST", "", 1, process_confirmation) 83 | 84 | 85 | if __name__ == '__main__': 86 | argument_dict = command_line_input(sys.argv[1:]) 87 | logging.basicConfig() 88 | run(argument_dict) 89 | -------------------------------------------------------------------------------- /python/util/ChangeEventHeaderUtility.py: -------------------------------------------------------------------------------- 1 | """ 2 | ChangeEventHeaderUtility.py 3 | 4 | This class provides the utility method to decode the bitmap fields (eg: changedFields) and return the avro schema field values represented by the bitmap. 5 | To understand the process of bitmap conversion, see "Event Deserialization Considerations" in the Pub/Sub API documentation at https://developer.salesforce.com/docs/platform/pub-sub-api/guide/event-deserialization-considerations.html. 6 | """ 7 | 8 | from avro.schema import Schema 9 | from bitstring import BitArray 10 | 11 | 12 | def process_bitmap(avro_schema: Schema, bitmap_fields: list): 13 | fields = [] 14 | if len(bitmap_fields) != 0: 15 | # replace top field level bitmap with list of fields 16 | if bitmap_fields[0].startswith("0x"): 17 | bitmap = bitmap_fields[0] 18 | fields = fields + get_fieldnames_from_bitstring(bitmap, avro_schema) 19 | bitmap_fields.remove(bitmap) 20 | # replace parentPos-nested Nulled BitMap with list of fields too 21 | if len(bitmap_fields) != 0 and "-" in str(bitmap_fields[-1]): 22 | for bitmap_field in bitmap_fields: 23 | if bitmap_field is not None and "-" in str(bitmap_field): 24 | bitmap_strings = bitmap_field.split("-") 25 | # interpret the parent field name from mapping of parentFieldPos -> childFieldbitMap 26 | parent_field = avro_schema.fields[int(bitmap_strings[0])] 27 | child_schema = get_value_schema(parent_field.type) 28 | # make sure we're really dealing with compound field 29 | if child_schema.type is not None and child_schema.type == 'record': 30 | nested_size = len(child_schema.fields) 31 | parent_field_name = parent_field.name 32 | # interpret the child field names from mapping of parentFieldPos -> childFieldbitMap 33 | full_field_names = get_fieldnames_from_bitstring(bitmap_strings[1], child_schema) 34 | full_field_names = append_parent_name(parent_field_name, full_field_names) 35 | if len(full_field_names) > 0: 36 | # when all nested fields under a compound got nulled out at once by customer, we recognize the top level field instead of trying to list every single nested field 37 | fields = fields + full_field_names 38 | return fields 39 | 40 | 41 | def convert_hexbinary_to_bitset(bitmap): 42 | bit_array = BitArray(hex=bitmap[2:]) 43 | binary_string = bit_array.bin 44 | return binary_string[::-1] 45 | 46 | 47 | def append_parent_name(parent_field_name, full_field_names): 48 | for index in range(len(full_field_names)): 49 | full_field_names[index] = parent_field_name + "." + full_field_names[index] 50 | return full_field_names 51 | 52 | 53 | def get_fieldnames_from_bitstring(bitmap, avro_schema: Schema): 54 | bitmap_field_name = [] 55 | fields_list = list(avro_schema.fields) 56 | binary_string = convert_hexbinary_to_bitset(bitmap) 57 | indexes = find('1', binary_string) 58 | for index in indexes: 59 | bitmap_field_name.append(fields_list[index].name) 60 | return bitmap_field_name 61 | 62 | 63 | # Get the value type of an "optional" schema, which is a union of [null, valueSchema] 64 | def get_value_schema(parent_field): 65 | if parent_field.type == 'union': 66 | schemas = parent_field.schemas 67 | if len(schemas) == 2 and schemas[0].type == 'null': 68 | return schemas[1] 69 | if len(schemas) == 2 and schemas[0].type == 'string': 70 | return schemas[1] 71 | if len(schemas) == 3 and schemas[0].type == 'null' and schemas[1].type == 'string': 72 | return schemas[2] 73 | return parent_field 74 | 75 | 76 | # Find the positions of 1 in the bit string 77 | def find(to_find, binary_string): 78 | return [i for i, x in enumerate(binary_string) if x == to_find] -------------------------------------------------------------------------------- /python/util/README.md: -------------------------------------------------------------------------------- 1 | # Pub/Sub API Examples - Utility Code 2 | 3 | 4 | ## ChangeEventHeaderUtility.py 5 | Because the Pub/Sub API is a binary API, delivered events are formatted as raw 6 | Avro binary and sometimes contain fields that are not plaintext-readable. This 7 | manifests in Change Data Capture events, causing them to look different from 8 | how they are delivered when subscribing via Streaming API. 9 | 10 | For example, a Change Data Capture event received via Pub/Sub API might look 11 | like this: 12 | 13 | ``` 14 | {'ChangeEventHeader': 15 | { 16 | ... 17 | 'nulledFields': [], 18 | 'diffFields': [], 19 | 'changedFields': ['0x650004E0'] 20 | }, 21 | ... 22 | } 23 | ``` 24 | 25 | In this example, `changedFields` is encoded as a bitmap string. This method is 26 | more space efficient than using a list of field names. A bit set to 1 in the 27 | `changedFields` bitmap value indicates that the field at the corresponding 28 | position in the Avro schema was changed. More information about bitmap fields 29 | can be found in [Event Deserialization Considerations](https://developer.salesforce.com/docs/platform/pub-sub-api/guide/event-deserialization-considerations.html) in the Pub/Sub API documentation. 30 | 31 | We have provided this example to demonstrate how bitmap values can be decoded 32 | so that they are human-readable and you can process the event by using the 33 | values in the bitmap fields. 34 | --------------------------------------------------------------------------------