() {
55 | @Override
56 | public void onNext(FetchResponse fetchResponse) {
57 | for(ConsumerEvent ce: fetchResponse.getEventsList()) {
58 | try {
59 | Schema writerSchema = subscriber.getSchema(ce.getEvent().getSchemaId());
60 | GenericRecord eventPayload = CommonContext.deserialize(writerSchema, ce.getEvent().getPayload());
61 | subscriber.updateReceivedEvents(1);
62 | String accountRecordId = eventPayload.get("AccountRecordId__c").toString();
63 | updateAccountRecord(subscriberParams, accountRecordId, subscriber.getSessionToken(), logger);
64 | } catch (Exception e) {
65 | logger.info(e.toString());
66 | }
67 | }
68 | if (fetchResponse.getPendingNumRequested() == 0) {
69 | subscriber.fetchMore(subscriber.getBatchSize());
70 | }
71 | }
72 |
73 | @Override
74 | public void onError(Throwable t) {
75 | printStatusRuntimeException("Error during SubscribeStream", (Exception) t);
76 | subscriber.isActive.set(false);
77 | }
78 |
79 | @Override
80 | public void onCompleted() {
81 | logger.info("Received requested number of events! Call completed by server.");
82 | subscriber.isActive.set(false);
83 | }
84 | };
85 | }
86 |
87 | // Helper function to start the app.
88 | public void startApp() throws InterruptedException {
89 | subscriber.startSubscription();
90 | }
91 |
92 | public void stopApp() {
93 | subscriber.close();
94 | }
95 |
96 | public static void main(String[] args) throws IOException {
97 | // For this example specifying only the required configurations in the arguments.yaml is enough.
98 | ExampleConfigurations requiredParameters = new ExampleConfigurations("arguments.yaml");
99 | try {
100 | AccountUpdater ac = new AccountUpdater(requiredParameters);
101 | ac.startApp();
102 | ac.stopApp();
103 | } catch (Exception e) {
104 | printStatusRuntimeException("Error during AccountUpdate", e);
105 | }
106 | }
107 | }
108 |
--------------------------------------------------------------------------------
/java/src/main/java/accountupdateapp/README.md:
--------------------------------------------------------------------------------
1 | # Account Update App
2 |
3 | This example subscribes to change events corresponding to the creation of [Account](https://developer.salesforce.com/docs/atlas.en-us.object_reference.meta/object_reference/sforce_api_objects_account.htm) records. Also, it updates a field in the created records using a custom platform event as a mediator between the two processes.
4 |
5 | ## Prerequisites:
6 | 1. The `Account` entity needs to be selected in order to generate change events whenever there is any action taken wrt Account objects. Steps to enable this:
7 | * Go to the `Setup Home` in your Salesforce org
8 | * Under the `Platform Tools` section, click on `Change Data Capture`
9 | * Search for the `Account` object and click on the right arrow in the middle of the screen to select the entity.
10 | * Click on the `Save` button to update the changes.
11 | 2. The `NewAccount` custom platform needs to be created with the following fields:
12 | - Platform Event Name
13 | - Label: `NewAccount`
14 | - Plural Label: `NewAccounts`
15 | - Custom Fields
16 | - `AccountRecordId` (Text, 20)
17 | 3. Only the required configurations need to be specified in the `arguments.yaml` file while running this example. You can specify the other optional configurations, but the optional configurations required for this example will be overwritten while running the examples.
18 |
19 | ## Flow Overview:
20 | * User creates an `Account` standard object which triggers an `AccountChangeEvent` event.
21 | * When the user creates an `Account` object, this generates a change event on the /data/AccountChangeEvent topic.
22 | * A listener listens to this `AccountChangeEvent` event and publishes a `NewAccount` custom platform event
23 | * The listener subscribes to the events on the `/data/AccountChangeEvent` topic and only in the case when a new `Account` is created, it will publish an event on the `/event/NewAccount__e` custom platform event topic with the recordId of the created `Account` object.
24 | * The updater listens to this `NewAccount` event and updates the `Account` object with a randomly generated `AccountNumber`.
25 | * The updater subscribes to the `/event/NewAccount__e` topic and when an event is received, it will update the appropriate `Account` object with a randomly generated `AccountNumber` using the Salesforce REST API.
26 |
27 | ## Running the examples:
28 | 1. Run the `AccountUpdater` first by running the following command:
29 | ```
30 | ./run.sh accountupdateapp.AccountUpdater
31 | ```
32 | 2. Run the `AccountListener` next by running the following command:
33 | ```
34 | ./run.sh accountupdateapp.AccountListener
35 | ```
36 |
37 | ## Notes:
38 | * Please use the `my domain` URL for your org for running these examples. You can find the my domain URL through Developer Console.
39 | * Open Developer Console
40 | * Click on the Debug menu and select Open Execute Anonymous Window.
41 | * Key in the following in the window: `System.debug(System.url.getOrgDomainUrl());` and execute the same.
42 | * Once done, in the Logs tab below open the logs recently executed code.
43 | * In the logs, get the `my domain` URL from the USER_DEBUG event.
44 | * Subscribers in both the `AccountUpdater` and `AccountListener` subscribe with the ReplayPreset set to LATEST. Therefore, only events generated once the examples have started running will be processed.
45 | * The `AccountUpdater` logs the `AccountNumber` that has been added to the `Account` record which can be used to verify if the update is correct.
--------------------------------------------------------------------------------
/java/src/main/java/genericpubsub/GetSchema.java:
--------------------------------------------------------------------------------
1 | package genericpubsub;
2 |
3 | import java.io.IOException;
4 |
5 | import org.apache.avro.Schema;
6 |
7 | import com.salesforce.eventbus.protobuf.SchemaInfo;
8 | import com.salesforce.eventbus.protobuf.SchemaRequest;
9 | import com.salesforce.eventbus.protobuf.TopicInfo;
10 | import com.salesforce.eventbus.protobuf.TopicRequest;
11 |
12 | import utility.CommonContext;
13 | import utility.ExampleConfigurations;
14 |
15 | /**
16 | * An example that retrieves the Schema of a single-topic.
17 | *
18 | * Example:
19 | * ./run.sh genericpubsub.GetSchema
20 | *
21 | * @author sidd0610
22 | */
23 | public class GetSchema extends CommonContext {
24 |
25 | public GetSchema(final ExampleConfigurations options) {
26 | super(options);
27 | }
28 |
29 | private void getSchema(String topicName) {
30 | // Use the GetTopic RPC to get the topic info for the given topicName.
31 | // Used to retrieve the schema id in this example.
32 | TopicInfo topicInfo = blockingStub.getTopic(TopicRequest.newBuilder().setTopicName(topicName).build());
33 | logger.info("GetTopic Call RPC ID: " + topicInfo.getRpcId());
34 |
35 | topicInfo.getAllFields().entrySet().forEach(entry -> {
36 | logger.info(entry.getKey() + " : " + entry.getValue());
37 | });
38 |
39 | SchemaRequest schemaRequest = SchemaRequest.newBuilder().setSchemaId(topicInfo.getSchemaId()).build();
40 |
41 | // Use the GetSchema RPC to get the schema info of the topic.
42 | SchemaInfo schemaResponse = blockingStub.getSchema(schemaRequest);
43 | logger.info("GetSchema Call RPC ID: " + schemaResponse.getRpcId());
44 | Schema schema = new Schema.Parser().parse(schemaResponse.getSchemaJson());
45 |
46 | // Printing the topic schema
47 | logger.info("Schema of topic " + topicName + ": " + schema.toString(true));
48 | }
49 |
50 | public static void main(String[] args) throws IOException {
51 | ExampleConfigurations exampleConfigurations = new ExampleConfigurations("arguments.yaml");
52 |
53 | // Using the try-with-resource statement. The CommonContext class implements AutoCloseable in
54 | // order to close the resources used.
55 | try (GetSchema example = new GetSchema(exampleConfigurations)) {
56 | example.getSchema(exampleConfigurations.getTopic());
57 | } catch (Exception e) {
58 | printStatusRuntimeException("Getting schema", e);
59 | }
60 | }
61 |
62 | }
63 |
--------------------------------------------------------------------------------
/java/src/main/java/genericpubsub/GetTopic.java:
--------------------------------------------------------------------------------
1 | package genericpubsub;
2 |
3 | import java.io.IOException;
4 |
5 | import com.salesforce.eventbus.protobuf.TopicInfo;
6 | import com.salesforce.eventbus.protobuf.TopicRequest;
7 |
8 | import utility.CommonContext;
9 | import utility.ExampleConfigurations;
10 |
11 | /**
12 | * An example that retrieves the topic info of a single-topic.
13 | *
14 | * Example:
15 | * ./run.sh genericpubsub.GetTopic
16 | *
17 | * @author sidd0610
18 | */
19 | public class GetTopic extends CommonContext {
20 |
21 | public GetTopic(final ExampleConfigurations options) {
22 | super(options);
23 | }
24 |
25 | private void getTopic(String topicName) {
26 | // Use the GetTopic RPC to get the topic info for the given topicName.
27 | TopicInfo topicInfo = blockingStub.getTopic(TopicRequest.newBuilder().setTopicName(topicName).build());
28 |
29 | logger.info("Topic Details:");
30 | topicInfo.getAllFields().entrySet().forEach(item -> {
31 | logger.info(item.getKey() + " : " + item.getValue());
32 | });
33 | }
34 |
35 | public static void main(String[] args) throws IOException {
36 | ExampleConfigurations exampleConfigurations = new ExampleConfigurations("arguments.yaml");
37 |
38 | // Using the try-with-resource statement. The CommonContext class implements AutoCloseable in
39 | // order to close the resources used.
40 | try (GetTopic example = new GetTopic(exampleConfigurations)) {
41 | example.getTopic(exampleConfigurations.getTopic());
42 | } catch (Exception e) {
43 | printStatusRuntimeException("Error while Getting Topic", e);
44 | }
45 | }
46 | }
47 |
--------------------------------------------------------------------------------
/java/src/main/java/genericpubsub/ManagedSubscribe.java:
--------------------------------------------------------------------------------
1 | package genericpubsub;
2 |
3 | import java.io.IOException;
4 | import java.util.Map;
5 | import java.util.Objects;
6 | import java.util.UUID;
7 | import java.util.concurrent.*;
8 | import java.util.concurrent.atomic.AtomicBoolean;
9 | import java.util.concurrent.atomic.AtomicInteger;
10 |
11 | import org.apache.avro.Schema;
12 | import org.apache.avro.generic.GenericRecord;
13 |
14 | import com.google.protobuf.ByteString;
15 | import com.salesforce.eventbus.protobuf.*;
16 |
17 | import io.grpc.stub.StreamObserver;
18 | import utility.CommonContext;
19 | import utility.ExampleConfigurations;
20 |
21 | /**
22 | * A single-topic subscriber that consumes events using Event Bus API ManagedSubscribe RPC. The example demonstrates how to:
23 | * - implement a long-lived subscription to a single topic
24 | * - a basic flow control strategy
25 | * - a basic commits strategy.
26 | *
27 | * Example:
28 | * ./run.sh genericpubsub.ManagedSubscribe
29 | *
30 | * @author jalaya
31 | */
32 | public class ManagedSubscribe extends CommonContext implements StreamObserver {
33 | private static int BATCH_SIZE;
34 | private StreamObserver serverStream;
35 | private Map schemaCache = new ConcurrentHashMap<>();
36 | private final CountDownLatch serverOnCompletedLatch = new CountDownLatch(1);
37 | public static AtomicBoolean isActive = new AtomicBoolean(false);
38 | private AtomicInteger receivedEvents = new AtomicInteger(0);
39 | private String developerName;
40 | private String managedSubscriptionId;
41 | private final boolean processChangedFields;
42 |
43 | public ManagedSubscribe(ExampleConfigurations exampleConfigurations) {
44 | super(exampleConfigurations);
45 | isActive.set(true);
46 | this.managedSubscriptionId = exampleConfigurations.getManagedSubscriptionId();
47 | this.developerName = exampleConfigurations.getDeveloperName();
48 | this.BATCH_SIZE = exampleConfigurations.getNumberOfEventsToSubscribeInEachFetchRequest();
49 | this.processChangedFields = exampleConfigurations.getProcessChangedFields();
50 | }
51 |
52 | /**
53 | * Function to start the ManagedSubscription, and send first ManagedFetchRequest.
54 | */
55 | public void startManagedSubscription() {
56 | serverStream = asyncStub.managedSubscribe(this);
57 | ManagedFetchRequest.Builder builder = ManagedFetchRequest.newBuilder().setNumRequested(BATCH_SIZE);
58 |
59 | if (Objects.nonNull(managedSubscriptionId)) {
60 | builder.setSubscriptionId(managedSubscriptionId);
61 | logger.info("Starting managed subscription with ID {}", managedSubscriptionId);
62 | } else if (Objects.nonNull(developerName)) {
63 | builder.setDeveloperName(developerName);
64 | logger.info("Starting managed subscription with developer name {}", developerName);
65 | } else {
66 | logger.warn("No ID or developer name specified");
67 | }
68 |
69 | serverStream.onNext(builder.build());
70 |
71 | // Thread being blocked here for demonstration of this specific example. Blocking the thread in production is not recommended.
72 | while(isActive.get()) {
73 | waitInMillis(5_000);
74 | logger.info("Subscription Active. Received a total of " + receivedEvents.get() + " events.");
75 | }
76 | }
77 |
78 | /**
79 | * Helps keep the subscription active by sending FetchRequests at regular intervals.
80 | *
81 | * @param numOfRequestedEvents
82 | */
83 | private void fetchMore(int numOfRequestedEvents) {
84 | logger.info("Fetching more events: {}", numOfRequestedEvents);
85 | ManagedFetchRequest fetchRequest = ManagedFetchRequest
86 | .newBuilder()
87 | .setNumRequested(numOfRequestedEvents)
88 | .build();
89 | serverStream.onNext(fetchRequest);
90 | }
91 |
92 | /**
93 | * Helper function to process the events received.
94 | */
95 | private void processEvent(ManagedFetchResponse response) throws IOException {
96 | if (response.getEventsCount() > 0) {
97 | for (ConsumerEvent event : response.getEventsList()) {
98 | String schemaId = event.getEvent().getSchemaId();
99 | logger.info("processEvent - EventID: {} SchemaId: {}", event.getEvent().getId(), schemaId);
100 | Schema writerSchema = getSchema(schemaId);
101 | GenericRecord record = deserialize(writerSchema, event.getEvent().getPayload());
102 | logger.info("Received event: {}", record.toString());
103 | if (processChangedFields) {
104 | // This example expands the changedFields bitmap field in ChangeEventHeader.
105 | // To expand the other bitmap fields, i.e., diffFields and nulledFields, replicate or modify this code.
106 | processAndPrintBitmapFields(writerSchema, record, "changedFields");
107 | }
108 | }
109 | logger.info("Processed batch of {} event(s)", response.getEventsList().size());
110 | }
111 |
112 | // Commit the replay after processing batch of events or commit the latest replay on an empty batch
113 | if (!response.hasCommitResponse()) {
114 | doCommitReplay(response.getLatestReplayId());
115 | }
116 | }
117 |
118 | /**
119 | * Helper function to commit the latest replay received from the server.
120 | */
121 | private void doCommitReplay(ByteString commitReplayId) {
122 | String newKey = UUID.randomUUID().toString();
123 | ManagedFetchRequest.Builder fetchRequestBuilder = ManagedFetchRequest.newBuilder();
124 | CommitReplayRequest commitRequest = CommitReplayRequest.newBuilder()
125 | .setCommitRequestId(newKey)
126 | .setReplayId(commitReplayId)
127 | .build();
128 | fetchRequestBuilder.setCommitReplayIdRequest(commitRequest);
129 |
130 | logger.info("Sending CommitRequest with CommitReplayRequest ID: {}" , newKey);
131 | serverStream.onNext(fetchRequestBuilder.build());
132 | }
133 |
134 | /**
135 | * Helper function to inspect the status of a commitRequest.
136 | */
137 | private void checkCommitResponse(ManagedFetchResponse fetchResponse) {
138 | CommitReplayResponse ce = fetchResponse.getCommitResponse();
139 | try {
140 | if (ce.hasError()) {
141 | logger.info("Failed Commit CommitRequestID: {} with error: {} with process time: {}",
142 | ce.getCommitRequestId(), ce.getError().getMsg(), ce.getProcessTime());
143 | return;
144 | }
145 | logger.info("Successfully committed replay with CommitRequestId: {} with process time: {}",
146 | ce.getCommitRequestId(), ce.getProcessTime());
147 | } catch (Exception e) {
148 | logger.warn(e.getMessage());
149 | abort(new RuntimeException("Client received error. Closing Call." + e));
150 | }
151 | }
152 |
153 | @Override
154 | public void onNext(ManagedFetchResponse fetchResponse) {
155 | int batchSize = fetchResponse.getEventsList().size();
156 | logger.info("ManagedFetchResponse batch of {} events pending requested: {}", batchSize, fetchResponse.getPendingNumRequested());
157 | logger.info("RPC ID: {}", fetchResponse.getRpcId());
158 |
159 | if (fetchResponse.hasCommitResponse()) {
160 | checkCommitResponse(fetchResponse);
161 | }
162 | try {
163 | processEvent(fetchResponse);
164 | } catch (IOException e) {
165 | logger.warn(e.getMessage());
166 | abort(new RuntimeException("Client received error. Closing Call." + e));
167 | }
168 |
169 | synchronized (this) {
170 | receivedEvents.addAndGet(batchSize);
171 | this.notifyAll();
172 | if (!isActive.get()) {
173 | return;
174 | }
175 | }
176 |
177 | if (fetchResponse.getPendingNumRequested() == 0) {
178 | fetchMore(BATCH_SIZE);
179 | }
180 | }
181 |
182 | @Override
183 | public void onError(Throwable throwable) {
184 | printStatusRuntimeException("Error during subscribe stream", (Exception) throwable);
185 |
186 | // onError from server closes stream. notify waiting thread that subscription is no longer active.
187 | synchronized (this) {
188 | isActive.set(false);
189 | this.notifyAll();
190 | }
191 | }
192 |
193 | @Override
194 | public void onCompleted() {
195 | logger.info("Call completed by Server");
196 | synchronized (this) {
197 | isActive.set(false);
198 | this.notifyAll();
199 | }
200 | serverOnCompletedLatch.countDown();
201 | }
202 |
203 | /**
204 | * Helper function to get the schema of an event if it does not already exist in the schema cache.
205 | */
206 | private Schema getSchema(String schemaId) {
207 | return schemaCache.computeIfAbsent(schemaId, id -> {
208 | SchemaRequest request = SchemaRequest.newBuilder().setSchemaId(id).build();
209 | String schemaJson = blockingStub.getSchema(request).getSchemaJson();
210 | return (new Schema.Parser()).parse(schemaJson);
211 | });
212 | }
213 |
214 | /**
215 | * Closes the connection when the task is complete.
216 | */
217 | @Override
218 | public synchronized void close() {
219 | if (Objects.nonNull(serverStream)) {
220 | try {
221 | if (isActive.get()) {
222 | isActive.set(false);
223 | this.notifyAll();
224 | serverStream.onCompleted();
225 | }
226 | serverOnCompletedLatch.await(6, TimeUnit.SECONDS);
227 | } catch (InterruptedException e) {
228 | logger.warn("interrupted while waiting to close ", e);
229 | }
230 | }
231 | super.close();
232 | }
233 |
234 | /**
235 | * Helper function to terminate the client on errors.
236 | */
237 | private synchronized void abort(Exception e) {
238 | serverStream.onError(e);
239 | isActive.set(false);
240 | this.notifyAll();
241 | }
242 |
243 | /**
244 | * Helper function to halt the current thread.
245 | */
246 | public void waitInMillis(long duration) {
247 | synchronized (this) {
248 | try {
249 | this.wait(duration);
250 | } catch (InterruptedException e) {
251 | throw new RuntimeException(e);
252 | }
253 | }
254 | }
255 |
256 | public static void main(String args[]) throws IOException {
257 | ExampleConfigurations exampleConfigurations = new ExampleConfigurations("arguments.yaml");
258 |
259 | // Using the try-with-resource statement. The CommonContext class implements AutoCloseable in
260 | // order to close the resources used.
261 | try (ManagedSubscribe subscribe = new ManagedSubscribe(exampleConfigurations)) {
262 | subscribe.startManagedSubscription();
263 | } catch (Exception e) {
264 | printStatusRuntimeException("Error during ManagedSubscribe", e);
265 | }
266 | }
267 | }
268 |
--------------------------------------------------------------------------------
/java/src/main/java/genericpubsub/Publish.java:
--------------------------------------------------------------------------------
1 | package genericpubsub;
2 |
3 | import java.io.ByteArrayOutputStream;
4 | import java.io.IOException;
5 | import java.util.List;
6 |
7 | import org.apache.avro.Schema;
8 | import org.apache.avro.generic.GenericDatumWriter;
9 | import org.apache.avro.generic.GenericRecord;
10 | import org.apache.avro.io.BinaryEncoder;
11 | import org.apache.avro.io.EncoderFactory;
12 |
13 | import com.google.protobuf.ByteString;
14 | import com.salesforce.eventbus.protobuf.*;
15 | import utility.CommonContext;
16 | import utility.ExampleConfigurations;
17 |
18 | /**
19 | * A single-topic publisher that creates an Order Event event and publishes it. This example uses
20 | * Pub/Sub API's Publish RPC to publish events.
21 | *
22 | * Example:
23 | * ./run.sh genericpubsub.Publish
24 | *
25 | * @author sidd0610
26 | */
27 | public class Publish extends CommonContext {
28 |
29 | private Schema schema;
30 |
31 | public Publish(ExampleConfigurations exampleConfigurations) {
32 | super(exampleConfigurations);
33 | setupTopicDetails(exampleConfigurations.getTopic(), true, true);
34 | schema = new Schema.Parser().parse(schemaInfo.getSchemaJson());
35 | }
36 |
37 | /**
38 | * Helper function for creating the ProducerEvent to be published
39 | *
40 | * @return ProducerEvent
41 | * @throws IOException
42 | */
43 | private ProducerEvent generateProducerEvent() throws IOException {
44 | Schema schema = new Schema.Parser().parse(schemaInfo.getSchemaJson());
45 | GenericRecord event = createEventMessage(schema);
46 |
47 | // Convert to byte array
48 | GenericDatumWriter writer = new GenericDatumWriter<>(event.getSchema());
49 | ByteArrayOutputStream buffer = new ByteArrayOutputStream();
50 | BinaryEncoder encoder = EncoderFactory.get().directBinaryEncoder(buffer, null);
51 | writer.write(event, encoder);
52 |
53 | return ProducerEvent.newBuilder().setSchemaId(schemaInfo.getSchemaId())
54 | .setPayload(ByteString.copyFrom(buffer.toByteArray())).build();
55 | }
56 |
57 | /**
58 | * Helper function to generate the PublishRequest with the generated ProducerEvent to be sent
59 | * using the Publish RPC
60 | *
61 | * @return PublishRequest
62 | * @throws IOException
63 | */
64 | private PublishRequest generatePublishRequest() throws IOException {
65 | ProducerEvent e = generateProducerEvent();
66 | return PublishRequest.newBuilder().setTopicName(busTopicName).addEvents(e).build();
67 | }
68 |
69 | /**
70 | * Helper function to publish the event using Publish RPC
71 | */
72 | public ByteString publish() throws Exception {
73 | PublishResponse response = blockingStub.publish(generatePublishRequest());
74 | return validatePublishResponse(response);
75 | }
76 |
77 | /**
78 | * Helper function for other examples to publish the event using Publish RPC
79 | *
80 | * @param event
81 | * @return
82 | * @throws Exception
83 | */
84 | public PublishResponse publish(ProducerEvent event) throws Exception {
85 | PublishRequest publishRequest = PublishRequest.newBuilder().setTopicName(busTopicName).addEvents(event).build();
86 | PublishResponse response = blockingStub.publish(publishRequest);
87 | validatePublishResponse(response);
88 | return response;
89 | }
90 |
91 | /**
92 | * Helper function to validate the PublishResponse received. Also prints the RPC id of the call.
93 | *
94 | * @param response
95 | * @return
96 | */
97 | private ByteString validatePublishResponse(PublishResponse response) {
98 | ByteString lastPublishedReplayId = null;
99 | List resultList = response.getResultsList();
100 | if (resultList.size() != 1) {
101 | String errorMsg = "[ERROR] Error during Publish, received: " + resultList.size() + " events instead of expected 1";
102 | logger.error(errorMsg);
103 | throw new RuntimeException(errorMsg);
104 | } else {
105 | PublishResult result = resultList.get(0);
106 | if (result.hasError()) {
107 | logger.error("[ERROR] Publishing batch failed with rpcId: " + response.getRpcId());
108 | logger.error("[ERROR] Error during Publish, event with correlationKey: {} failed with: {}",
109 | response.getResults(0).getCorrelationKey(), result.getError().getMsg());
110 | } else {
111 | lastPublishedReplayId = result.getReplayId();
112 | logger.info("Publish Call RPC ID: " + response.getRpcId());
113 | logger.info("Successfully published an event with correlationKey: {} at {} for tenant {}.",
114 | response.getResults(0).getCorrelationKey(), busTopicName, tenantGuid);
115 | }
116 | }
117 |
118 | return lastPublishedReplayId;
119 | }
120 |
121 | /**
122 | * General getters.
123 | */
124 | public Schema getSchema() {
125 | return schema;
126 | }
127 |
128 | public SchemaInfo getSchemaInfo() {
129 | return schemaInfo;
130 | }
131 |
132 | public static void main(String[] args) throws IOException {
133 | ExampleConfigurations exampleConfigurations = new ExampleConfigurations("arguments.yaml");
134 |
135 | // Using the try-with-resource statement. The CommonContext class implements AutoCloseable in
136 | // order to close the resources used.
137 | try (Publish example = new Publish(exampleConfigurations)) {
138 | example.publish();
139 | } catch (Exception e) {
140 | CommonContext.printStatusRuntimeException("Publishing events", e);
141 | }
142 | }
143 | }
--------------------------------------------------------------------------------
/java/src/main/java/genericpubsub/PublishStream.java:
--------------------------------------------------------------------------------
1 | package genericpubsub;
2 |
3 | import java.io.ByteArrayOutputStream;
4 | import java.io.IOException;
5 | import java.util.List;
6 | import java.util.UUID;
7 | import java.util.concurrent.CountDownLatch;
8 | import java.util.concurrent.TimeUnit;
9 | import java.util.concurrent.atomic.AtomicInteger;
10 | import java.util.concurrent.atomic.AtomicReference;
11 |
12 | import org.apache.avro.Schema;
13 | import org.apache.avro.generic.GenericDatumWriter;
14 | import org.apache.avro.generic.GenericRecord;
15 | import org.apache.avro.io.BinaryEncoder;
16 | import org.apache.avro.io.EncoderFactory;
17 |
18 | import com.google.common.collect.Lists;
19 | import com.google.protobuf.ByteString;
20 | import com.salesforce.eventbus.protobuf.*;
21 |
22 | import io.grpc.Status;
23 | import io.grpc.stub.ClientCallStreamObserver;
24 | import io.grpc.stub.StreamObserver;
25 | import utility.CommonContext;
26 | import utility.ExampleConfigurations;
27 |
28 | /**
29 | * A single-topic publisher that creates Order Event events and publishes them. This example
30 | * uses Pub/Sub API's PublishStream RPC to publish events.
31 | *
32 | * Example:
33 | * ./run.sh genericpubsub.PublishStream
34 | *
35 | * @author sidd0610
36 | */
37 | public class PublishStream extends CommonContext {
38 | private final int TIMEOUT_SECONDS = 30; // Max time we'll wait to finish streaming
39 |
40 | ClientCallStreamObserver requestObserver = null;
41 |
42 | private ByteString lastPublishedReplayId;
43 |
44 | public PublishStream(ExampleConfigurations exampleConfigurations) {
45 | super(exampleConfigurations);
46 | setupTopicDetails(exampleConfigurations.getTopic(), true, true);
47 | }
48 |
49 | /**
50 | * Publishes specified number of events using the PublishStream RPC.
51 | *
52 | * @param numEventsToPublish
53 | * @return ByteString
54 | * @throws Exception
55 | */
56 | public void publishStream(int numEventsToPublish, Boolean singlePublishRequest) throws Exception {
57 | CountDownLatch finishLatch = new CountDownLatch(1);
58 | AtomicReference finishLatchRef = new AtomicReference<>(finishLatch);
59 | final int numExpectedPublishResponses = singlePublishRequest ? 1 : numEventsToPublish;
60 | final List publishResponses = Lists.newArrayListWithExpectedSize(numExpectedPublishResponses);
61 | AtomicInteger failed = new AtomicInteger(0);
62 | StreamObserver pubObserver = getDefaultPublishStreamObserver(finishLatchRef,
63 | numExpectedPublishResponses, publishResponses, failed);
64 |
65 | // construct the stream
66 | requestObserver = (ClientCallStreamObserver) asyncStub.publishStream(pubObserver);
67 |
68 | if (singlePublishRequest == false) {
69 | // Publish each event in a separate batch
70 | for (int i = 0; i < numEventsToPublish; i++) {
71 | requestObserver.onNext(generatePublishRequest(i, singlePublishRequest));
72 | }
73 | } else {
74 | // Publish all events in one batch
75 | requestObserver.onNext(generatePublishRequest(numEventsToPublish, singlePublishRequest));
76 | }
77 |
78 | validatePublishResponse(finishLatch, numExpectedPublishResponses, publishResponses, failed, numEventsToPublish);
79 | requestObserver.onCompleted();
80 | }
81 |
82 | /**
83 | * Helper function to validate the PublishResponse received. Also prints the RPC id of the call.
84 | *
85 | * @param errorStatus
86 | * @param finishLatch
87 | * @param expectedResponseCount
88 | * @param publishResponses
89 | * @return
90 | * @throws Exception
91 | */
92 | private void validatePublishResponse(CountDownLatch finishLatch,
93 | int expectedResponseCount, List publishResponses, AtomicInteger failed, int expectedNumEventsPublished) throws Exception {
94 | String exceptionMsg;
95 | boolean failedPublish = false;
96 | if (!finishLatch.await(TIMEOUT_SECONDS, TimeUnit.SECONDS)) {
97 | failedPublish = true;
98 | exceptionMsg = "[ERROR] publishStream timed out after: " + TIMEOUT_SECONDS + "sec";
99 | logger.error(exceptionMsg);
100 | }
101 |
102 | if (expectedResponseCount != publishResponses.size()) {
103 | failedPublish = true;
104 | exceptionMsg = "[ERROR] PublishStream received: " + publishResponses.size() + " PublishResponses instead of expected "
105 | + expectedResponseCount;
106 | logger.error(exceptionMsg);
107 | }
108 |
109 | if (failed.get() != 0) {
110 | failedPublish = true;
111 | exceptionMsg = "[ERROR] Failed to publish all events. " + failed + " failed out of "
112 | + expectedNumEventsPublished;
113 | logger.error(exceptionMsg);
114 | }
115 |
116 | if (failedPublish) {
117 | throw new RuntimeException("Failed to publish events.");
118 | }
119 | }
120 |
121 | /**
122 | * Creates a ProducerEvent to be published in a PublishRequest.
123 | *
124 | * @param counter
125 | * @return
126 | * @throws IOException
127 | */
128 | private ProducerEvent generateProducerEvent(int counter) throws IOException {
129 | Schema schema = new Schema.Parser().parse(schemaInfo.getSchemaJson());
130 | GenericRecord event = createEventMessage(schema, counter);
131 |
132 | // Convert to byte array
133 | GenericDatumWriter writer = new GenericDatumWriter<>(event.getSchema());
134 | ByteArrayOutputStream buffer = new ByteArrayOutputStream();
135 | BinaryEncoder encoder = EncoderFactory.get().directBinaryEncoder(buffer, null);
136 | writer.write(event, encoder);
137 |
138 | return ProducerEvent.newBuilder().setSchemaId(schemaInfo.getSchemaId())
139 | .setPayload(ByteString.copyFrom(buffer.toByteArray())).build();
140 | }
141 |
142 | /**
143 | * Creates an array of ProducerEvents to be published in a PublishRequest.
144 | *
145 | * @param count
146 | * @return
147 | * @throws IOException
148 | */
149 | private ProducerEvent[] generateProducerEvents(int count) throws IOException {
150 | Schema schema = new Schema.Parser().parse(schemaInfo.getSchemaJson());
151 | List events = createEventMessages(schema, count);
152 |
153 | ProducerEvent[] prodEvents = new ProducerEvent[count];
154 | GenericDatumWriter writer = new GenericDatumWriter<>(events.get(0).getSchema());
155 |
156 | for(int i=0; i getDefaultPublishStreamObserver(AtomicReference finishLatchRef, int expectedResponseCount,
202 | List publishResponses, AtomicInteger failed) {
203 | return new StreamObserver<>() {
204 | @Override
205 | public void onNext(PublishResponse publishResponse) {
206 | publishResponses.add(publishResponse);
207 |
208 | logger.info("Publish Call rpcId: " + publishResponse.getRpcId());
209 |
210 | for (PublishResult publishResult : publishResponse.getResultsList()) {
211 | if (publishResult.hasError()) {
212 | failed.incrementAndGet();
213 | logger.error("[ERROR] Publishing event with correlationKey: " + publishResult.getCorrelationKey() +
214 | " failed with error: " + publishResult.getError().getMsg());
215 | } else {
216 | logger.info("Event published with correlationKey: " + publishResult.getCorrelationKey());
217 | lastPublishedReplayId = publishResult.getReplayId();
218 | }
219 | }
220 | if (publishResponses.size() == expectedResponseCount) {
221 | finishLatchRef.get().countDown();
222 | }
223 | }
224 |
225 | @Override
226 | public void onError(Throwable t) {
227 | logger.error("[ERROR] Unexpected error status: " + Status.fromThrowable(t));
228 | printStatusRuntimeException("Error during PublishStream", (Exception) t);
229 | finishLatchRef.get().countDown();
230 | }
231 |
232 | @Override
233 | public void onCompleted() {
234 | logger.info("Successfully published events for topic " + busTopicName + " for tenant " + tenantGuid);
235 | finishLatchRef.get().countDown();
236 | }
237 | };
238 | }
239 |
240 | public static void main(String[] args) throws IOException {
241 | ExampleConfigurations exampleConfigurations = new ExampleConfigurations("arguments.yaml");
242 |
243 | // Using the try-with-resource statement. The CommonContext class implements AutoCloseable in
244 | // order to close the resources used.
245 | try (PublishStream example = new PublishStream(exampleConfigurations)) {
246 | example.publishStream(exampleConfigurations.getNumberOfEventsToPublish(),
247 | exampleConfigurations.getSinglePublishRequest());
248 | } catch (Exception e) {
249 | printStatusRuntimeException("Error During PublishStream", e);
250 | }
251 | }
252 | }
253 |
--------------------------------------------------------------------------------
/java/src/main/java/genericpubsub/Subscribe.java:
--------------------------------------------------------------------------------
1 | package genericpubsub;
2 |
3 | import java.io.IOException;
4 | import java.util.Map;
5 | import java.util.concurrent.*;
6 | import java.util.concurrent.atomic.AtomicBoolean;
7 | import java.util.concurrent.atomic.AtomicInteger;
8 |
9 | import io.grpc.Metadata;
10 | import io.grpc.StatusRuntimeException;
11 | import org.apache.avro.Schema;
12 | import org.apache.avro.generic.GenericRecord;
13 |
14 | import com.google.protobuf.ByteString;
15 | import com.salesforce.eventbus.protobuf.*;
16 |
17 | import io.grpc.stub.StreamObserver;
18 | import utility.CommonContext;
19 | import utility.ExampleConfigurations;
20 |
21 | /**
22 | * A single-topic subscriber that consumes events using Event Bus API Subscribe RPC. The example demonstrates how to:
23 | * - implement a long-lived subscription to a single topic
24 | * - a basic flow control strategy
25 | * - a basic retry strategy.
26 | *
27 | * Example:
28 | * ./run.sh genericpubsub.Subscribe
29 | *
30 | * @author sidd0610
31 | */
32 | public class Subscribe extends CommonContext {
33 |
34 | public static int BATCH_SIZE;
35 | public static int MAX_RETRIES = 3;
36 | public static String ERROR_REPLAY_ID_VALIDATION_FAILED = "fetch.replayid.validation.failed";
37 | public static String ERROR_REPLAY_ID_INVALID = "fetch.replayid.corrupted";
38 | public static String ERROR_SERVICE_UNAVAILABLE = "service.unavailable";
39 | public static int SERVICE_UNAVAILABLE_WAIT_BEFORE_RETRY_SECONDS = 5;
40 | public static ExampleConfigurations exampleConfigurations;
41 | public static AtomicBoolean isActive = new AtomicBoolean(false);
42 | public static AtomicInteger retriesLeft = new AtomicInteger(MAX_RETRIES);
43 | private StreamObserver serverStream;
44 | private Map schemaCache = new ConcurrentHashMap<>();
45 | private AtomicInteger receivedEvents = new AtomicInteger(0);
46 | private final StreamObserver responseStreamObserver;
47 | private final ReplayPreset replayPreset;
48 | private final ByteString customReplayId;
49 | private final ScheduledExecutorService retryScheduler;
50 | // Replay should be stored in replay store as bytes since replays are opaque.
51 | private volatile ByteString storedReplay;
52 | private final boolean processChangedFields;
53 |
54 | public Subscribe(ExampleConfigurations exampleConfigurations) {
55 | super(exampleConfigurations);
56 | isActive.set(true);
57 | this.exampleConfigurations = exampleConfigurations;
58 | this.BATCH_SIZE = exampleConfigurations.getNumberOfEventsToSubscribeInEachFetchRequest();
59 | this.responseStreamObserver = getDefaultResponseStreamObserver();
60 | this.setupTopicDetails(exampleConfigurations.getTopic(), false, false);
61 | this.replayPreset = exampleConfigurations.getReplayPreset();
62 | this.customReplayId = exampleConfigurations.getReplayId();
63 | this.retryScheduler = Executors.newScheduledThreadPool(1);
64 | this.processChangedFields = exampleConfigurations.getProcessChangedFields();
65 | }
66 |
67 | public Subscribe(ExampleConfigurations exampleConfigurations, StreamObserver responseStreamObserver) {
68 | super(exampleConfigurations);
69 | isActive.set(true);
70 | this.exampleConfigurations = exampleConfigurations;
71 | this.BATCH_SIZE = exampleConfigurations.getNumberOfEventsToSubscribeInEachFetchRequest();
72 | this.responseStreamObserver = responseStreamObserver;
73 | this.setupTopicDetails(exampleConfigurations.getTopic(), false, false);
74 | this.replayPreset = exampleConfigurations.getReplayPreset();
75 | this.customReplayId = exampleConfigurations.getReplayId();
76 | this.retryScheduler = Executors.newScheduledThreadPool(1);
77 | this.processChangedFields = exampleConfigurations.getProcessChangedFields();
78 | }
79 |
80 | /**
81 | * Function to start the subscription.
82 | */
83 | public void startSubscription() {
84 | logger.info("Subscription started for topic: " + busTopicName + ".");
85 | fetch(BATCH_SIZE, busTopicName, replayPreset, customReplayId);
86 | // Thread being blocked here for demonstration of this specific example. Blocking the thread in production is not recommended.
87 | while(isActive.get()) {
88 | waitInMillis(5_000);
89 | logger.info("Subscription Active. Received a total of " + receivedEvents.get() + " events.");
90 | }
91 | }
92 |
93 | /** Helper function to send FetchRequests.
94 | * @param providedBatchSize
95 | * @param providedTopicName
96 | * @param providedReplayPreset
97 | * @param providedReplayId
98 | */
99 | public void fetch(int providedBatchSize, String providedTopicName, ReplayPreset providedReplayPreset, ByteString providedReplayId) {
100 | serverStream = asyncStub.subscribe(this.responseStreamObserver);
101 | FetchRequest.Builder fetchRequestBuilder = FetchRequest.newBuilder()
102 | .setNumRequested(providedBatchSize)
103 | .setTopicName(providedTopicName)
104 | .setReplayPreset(providedReplayPreset);
105 | if (providedReplayPreset == ReplayPreset.CUSTOM) {
106 | logger.info("Subscription has Replay Preset set to CUSTOM. In this case, the events will be delivered from provided ReplayId.");
107 | fetchRequestBuilder.setReplayId(providedReplayId);
108 | }
109 | serverStream.onNext(fetchRequestBuilder.build());
110 | }
111 |
112 | /**
113 | * Function to decide the delay (in ms) in sending FetchRequests using
114 | * Binary Exponential Backoff - Waits for 2^(Max Number of Retries - Retries Left) * 1000.
115 | */
116 | public long getBackoffWaitTime() {
117 | long waitTime = (long) (Math.pow(2, MAX_RETRIES - retriesLeft.get()) * 1000);
118 | return waitTime;
119 | }
120 |
121 | /**
122 | * Helper function to halt the current thread.
123 | */
124 | public void waitInMillis(long duration) {
125 | synchronized (this) {
126 | try {
127 | this.wait(duration);
128 | } catch (InterruptedException e) {
129 | throw new RuntimeException(e);
130 | }
131 | }
132 | }
133 |
134 | /**
135 | * Creates a StreamObserver for handling the incoming FetchResponse messages from the server.
136 | *
137 | * @return
138 | */
139 | private StreamObserver getDefaultResponseStreamObserver() {
140 | return new StreamObserver() {
141 | @Override
142 | public void onNext(FetchResponse fetchResponse) {
143 | logger.info("Received batch of " + fetchResponse.getEventsList().size() + " events");
144 | logger.info("RPC ID: " + fetchResponse.getRpcId());
145 | for(ConsumerEvent ce : fetchResponse.getEventsList()) {
146 | try {
147 | processEvent(ce);
148 | } catch (Exception e) {
149 | logger.info(e.toString());
150 | }
151 | receivedEvents.addAndGet(1);
152 | }
153 | // Latest replayId stored for any future FetchRequests with CUSTOM ReplayPreset.
154 | // NOTE: Replay IDs are opaque in nature and should be stored and used as bytes without any conversion.
155 | storedReplay = fetchResponse.getLatestReplayId();
156 |
157 | // Reset retry count
158 | if (retriesLeft.get() != MAX_RETRIES) {
159 | retriesLeft.set(MAX_RETRIES);
160 | }
161 |
162 | // Implementing a basic flow control strategy where the next fetchRequest is sent only after the
163 | // requested number of events in the previous fetchRequest(s) are received.
164 | // NOTE: This block may need to be implemented before the processing of events if event processing takes
165 | // a long time. There is a 70s timeout period during which, if pendingNumRequested is 0 and no events are
166 | // further requested then the stream will be closed.
167 | if (fetchResponse.getPendingNumRequested() == 0) {
168 | fetchMore(BATCH_SIZE);
169 | }
170 | }
171 |
172 | @Override
173 | public void onError(Throwable t) {
174 | printStatusRuntimeException("Error during Subscribe", (Exception) t);
175 | logger.info("Retries remaining: " + retriesLeft.get());
176 | if (retriesLeft.get() == 0) {
177 | logger.info("Exhausted all retries. Closing Subscription.");
178 | isActive.set(false);
179 | } else {
180 | retriesLeft.decrementAndGet();
181 | Metadata trailers = ((StatusRuntimeException)t).getTrailers() != null ? ((StatusRuntimeException)t).getTrailers() : null;
182 | String errorCode = (trailers != null && trailers.get(Metadata.Key.of("error-code", Metadata.ASCII_STRING_MARSHALLER)) != null) ?
183 | trailers.get(Metadata.Key.of("error-code", Metadata.ASCII_STRING_MARSHALLER)) : null;
184 |
185 | // Closing the old stream for sanity
186 | serverStream.onCompleted();
187 |
188 | ReplayPreset retryReplayPreset = ReplayPreset.LATEST;
189 | ByteString retryReplayId = null;
190 | long retryDelay = getBackoffWaitTime();
191 |
192 | // Retry strategies that can be implemented based on the error type.
193 | if(errorCode != null && !errorCode.isEmpty()) {
194 | if (errorCode.contains(ERROR_REPLAY_ID_VALIDATION_FAILED) || errorCode.contains(ERROR_REPLAY_ID_INVALID)) {
195 | logger.info("Invalid or no replayId provided in FetchRequest for CUSTOM Replay. Trying again with EARLIEST Replay.");
196 | retryReplayPreset = ReplayPreset.EARLIEST;
197 | } else if (errorCode.contains(ERROR_SERVICE_UNAVAILABLE)) {
198 | logger.info("Service currently unavailable. Trying again with LATEST Replay.");
199 | retryDelay = SERVICE_UNAVAILABLE_WAIT_BEFORE_RETRY_SECONDS * 1000;
200 | } else {
201 | if (storedReplay != null) {
202 | logger.info("Retrying with Stored Replay.");
203 | retryReplayPreset = ReplayPreset.CUSTOM;
204 | retryReplayId = getStoredReplay();
205 | } else {
206 | logger.info("Retrying with LATEST Replay.");
207 | }
208 |
209 | }
210 | } else {
211 | logger.info("Unknown error. Retrying with LATEST Replay.");
212 | }
213 | logger.info(String.format("Retrying in %s ms.", retryDelay));
214 | retryScheduler.schedule(new RetryRequestSender(retryReplayPreset, retryReplayId), retryDelay, TimeUnit.MILLISECONDS);
215 | }
216 | }
217 |
218 | @Override
219 | public void onCompleted() {
220 | logger.info("Call completed by server. Closing Subscription.");
221 | isActive.set(false);
222 | }
223 | };
224 | }
225 |
226 | /**
227 | * A Runnable class that is used to send the FetchRequests by making a new Subscribe call while retrying on
228 | * receiving an error. This is done in order to avoid blocking the thread while waiting for retries. This class is
229 | * passed to the ScheduledExecutorService which will asynchronously send the FetchRequests during retries.
230 | */
231 | private class RetryRequestSender implements Runnable {
232 | private ReplayPreset retryReplayPreset;
233 | private ByteString retryReplayId;
234 | public RetryRequestSender(ReplayPreset replayPreset, ByteString replayId) {
235 | this.retryReplayPreset = replayPreset;
236 | this.retryReplayId = replayId;
237 | }
238 |
239 | @Override
240 | public void run() {
241 | fetch(BATCH_SIZE, busTopicName, retryReplayPreset, retryReplayId);
242 | logger.info("Retry FetchRequest Sent.");
243 | }
244 | }
245 |
246 | /**
247 | * Helper function to process the events received.
248 | */
249 | private void processEvent(ConsumerEvent ce) throws IOException {
250 | Schema writerSchema = getSchema(ce.getEvent().getSchemaId());
251 | this.storedReplay = ce.getReplayId();
252 | GenericRecord record = deserialize(writerSchema, ce.getEvent().getPayload());
253 | logger.info("Received event with payload: " + record.toString() + " with schema name: " + writerSchema.getName());
254 | if (processChangedFields) {
255 | // This example expands the changedFields bitmap field in ChangeEventHeader.
256 | // To expand the other bitmap fields, i.e., diffFields and nulledFields, replicate or modify this code.
257 | processAndPrintBitmapFields(writerSchema, record, "changedFields");
258 | }
259 | }
260 |
261 | /**
262 | * Helper function to get the schema of an event if it does not already exist in the schema cache.
263 | */
264 | public Schema getSchema(String schemaId) {
265 | return schemaCache.computeIfAbsent(schemaId, id -> {
266 | SchemaRequest request = SchemaRequest.newBuilder().setSchemaId(id).build();
267 | String schemaJson = blockingStub.getSchema(request).getSchemaJson();
268 | return (new Schema.Parser()).parse(schemaJson);
269 | });
270 | }
271 |
272 | /**
273 | * Helps keep the subscription active by sending FetchRequests at regular intervals.
274 | *
275 | * @param numEvents
276 | */
277 | public void fetchMore(int numEvents) {
278 | FetchRequest fetchRequest = FetchRequest.newBuilder().setTopicName(this.busTopicName)
279 | .setNumRequested(numEvents).build();
280 | serverStream.onNext(fetchRequest);
281 | }
282 |
283 | /**
284 | * General getters and setters.
285 | */
286 | public AtomicInteger getReceivedEvents() {
287 | return receivedEvents;
288 | }
289 |
290 | public void updateReceivedEvents(int delta) {
291 | receivedEvents.addAndGet(delta);
292 | }
293 |
294 | public int getBatchSize() {
295 | return BATCH_SIZE;
296 | }
297 | public ByteString getStoredReplay() {
298 | return storedReplay;
299 | }
300 |
301 | public void setStoredReplay(ByteString storedReplay) {
302 | this.storedReplay = storedReplay;
303 | }
304 |
305 | /**
306 | * Closes the connection when the task is complete.
307 | */
308 | @Override
309 | public synchronized void close() {
310 | try {
311 | if (serverStream != null) {
312 | serverStream.onCompleted();
313 | }
314 | if (retryScheduler != null) {
315 | retryScheduler.shutdown();
316 | }
317 | } catch (Exception e) {
318 | logger.info(e.toString());
319 | }
320 | super.close();
321 | }
322 |
323 | public static void main(String args[]) throws IOException {
324 | ExampleConfigurations exampleConfigurations = new ExampleConfigurations("arguments.yaml");
325 |
326 | // Using the try-with-resource statement. The CommonContext class implements AutoCloseable in
327 | // order to close the resources used.
328 | try (Subscribe subscribe = new Subscribe(exampleConfigurations)) {
329 | subscribe.startSubscription();
330 | } catch (Exception e) {
331 | printStatusRuntimeException("Error during Subscribe", e);
332 | }
333 | }
334 | }
335 |
--------------------------------------------------------------------------------
/java/src/main/java/utility/APISessionCredentials.java:
--------------------------------------------------------------------------------
1 | package utility;
2 |
3 | import java.util.UUID;
4 | import java.util.concurrent.Executor;
5 |
6 | import org.slf4j.Logger;
7 | import org.slf4j.LoggerFactory;
8 |
9 | import io.grpc.CallCredentials;
10 | import io.grpc.Metadata;
11 |
12 | /**
13 | * The APISessionCredentials class extends the CallCredentials class of gRPC to add important
14 | * credential information, i.e., tenantId, accessToken and instanceUrl to every request made to
15 | * Pub/Sub API.
16 | */
17 | public class APISessionCredentials extends CallCredentials {
18 |
19 | // Instance url of the customer org
20 | public static final Metadata.Key INSTANCE_URL_KEY = keyOf("instanceUrl");
21 | // Session token of the customer
22 | public static final Metadata.Key SESSION_TOKEN_KEY = keyOf("accessToken");
23 | // Tenant Id of the customer org
24 | public static final Metadata.Key TENANT_ID_KEY = keyOf("tenantId");
25 |
26 | private String instanceURL;
27 | private String tenantId;
28 | private String token;
29 |
30 | private static final Logger log = LoggerFactory.getLogger(APISessionCredentials.class);
31 |
32 | public APISessionCredentials(String tenantId, String instanceURL, String token) {
33 | this.instanceURL = instanceURL;
34 | this.tenantId = tenantId;
35 | this.token = token;
36 | }
37 |
38 | @Override
39 | public void applyRequestMetadata(RequestInfo requestInfo, Executor executor, MetadataApplier metadataApplier) {
40 | log.debug("API session credentials applied to " + requestInfo.getMethodDescriptor());
41 | Metadata headers = new Metadata();
42 | headers.put(INSTANCE_URL_KEY, instanceURL);
43 | headers.put(TENANT_ID_KEY, tenantId);
44 | headers.put(SESSION_TOKEN_KEY, token);
45 | metadataApplier.apply(headers);
46 | }
47 |
48 | @Override
49 | public void thisUsesUnstableApi() {
50 |
51 | }
52 |
53 | private static Metadata.Key keyOf(String name) {
54 | return Metadata.Key.of(name, Metadata.ASCII_STRING_MARSHALLER);
55 | }
56 |
57 | public String getToken() {
58 | return token;
59 | }
60 | }
--------------------------------------------------------------------------------
/java/src/main/java/utility/CommonContext.java:
--------------------------------------------------------------------------------
1 | package utility;
2 |
3 | import java.io.*;
4 | import java.nio.ByteBuffer;
5 | import java.util.ArrayList;
6 | import java.util.List;
7 | import java.util.Map;
8 | import java.util.concurrent.TimeUnit;
9 |
10 | import org.apache.avro.Schema;
11 | import org.apache.avro.generic.GenericData;
12 | import org.apache.avro.generic.GenericDatumReader;
13 | import org.apache.avro.generic.GenericRecord;
14 | import org.apache.avro.generic.GenericRecordBuilder;
15 | import org.apache.avro.io.BinaryDecoder;
16 | import org.apache.avro.io.DatumReader;
17 | import org.apache.avro.io.DecoderFactory;
18 | import org.eclipse.jetty.client.HttpClient;
19 | import org.eclipse.jetty.client.HttpProxy;
20 | import org.slf4j.Logger;
21 | import org.slf4j.LoggerFactory;
22 |
23 | import com.google.common.base.CaseFormat;
24 | import com.google.protobuf.ByteString;
25 | import com.salesforce.eventbus.protobuf.*;
26 |
27 | import io.grpc.*;
28 |
29 | import static utility.EventParser.getFieldListFromBitmap;
30 |
31 | /**
32 | * The CommonContext class provides a list of member variables and functions that is used across
33 | * all examples for various purposes like setting up the HttpClient, CallCredentials, stubs for
34 | * sending requests, generating events etc.
35 | */
36 | public class CommonContext implements AutoCloseable {
37 |
38 | protected static final Logger logger = LoggerFactory.getLogger(CommonContext.class.getClass());
39 |
40 | protected final ManagedChannel channel;
41 | protected final PubSubGrpc.PubSubStub asyncStub;
42 | protected final PubSubGrpc.PubSubBlockingStub blockingStub;
43 |
44 | protected final HttpClient httpClient;
45 | protected final SessionTokenService sessionTokenService;
46 | protected final CallCredentials callCredentials;
47 |
48 | protected String tenantGuid;
49 | protected String busTopicName;
50 | protected TopicInfo topicInfo;
51 | protected SchemaInfo schemaInfo;
52 | protected String sessionToken;
53 |
54 | public CommonContext(final ExampleConfigurations options) {
55 | String grpcHost = options.getPubsubHost();
56 | int grpcPort = options.getPubsubPort();
57 | logger.info("Using grpcHost {} and grpcPort {}", grpcHost, grpcPort);
58 |
59 | if (options.usePlaintextChannel()) {
60 | channel = ManagedChannelBuilder.forAddress(grpcHost, grpcPort).usePlaintext().build();
61 | } else {
62 | channel = ManagedChannelBuilder.forAddress(grpcHost, grpcPort).build();
63 | }
64 |
65 | httpClient = setupHttpClient();
66 | sessionTokenService = new SessionTokenService(httpClient);
67 |
68 | callCredentials = setupCallCredentials(options);
69 | sessionToken = ((APISessionCredentials) callCredentials).getToken();
70 |
71 | Channel interceptedChannel = ClientInterceptors.intercept(channel, new XClientTraceIdClientInterceptor());
72 |
73 | asyncStub = PubSubGrpc.newStub(interceptedChannel).withCallCredentials(callCredentials);
74 | blockingStub = PubSubGrpc.newBlockingStub(interceptedChannel).withCallCredentials(callCredentials);
75 | }
76 |
77 | /**
78 | * Helper function to setup the HttpClient used for sending requests.
79 | */
80 | private HttpClient setupHttpClient() {
81 | HttpClient httpClient = new HttpClient();
82 | Map env = System.getenv();
83 |
84 | String httpProxy = env.get("HTTP_PROXY");
85 | if (httpProxy != null) {
86 | String[] httpProxyParts = httpProxy.split(":");
87 | httpClient.getProxyConfiguration().getProxies()
88 | .add(new HttpProxy(httpProxyParts[0], Integer.parseInt(httpProxyParts[1])));
89 | }
90 |
91 | try {
92 | httpClient.start();
93 | } catch (Exception e) {
94 | logger.error("cannot create HTTP client", e);
95 | }
96 | return httpClient;
97 | }
98 |
99 | /**
100 | * Helper function to setup the CallCredentials of the requests.
101 | *
102 | * @param options Command line arguments passed.
103 | * @return CallCredentials
104 | */
105 | public CallCredentials setupCallCredentials(ExampleConfigurations options) {
106 | if (options.getAccessToken() != null) {
107 | try {
108 | return sessionTokenService.loginWithAccessToken(options.getLoginUrl(),
109 | options.getAccessToken(), options.getTenantId());
110 | } catch (Exception e) {
111 | close();
112 | throw new IllegalArgumentException("cannot log in with access token", e);
113 | }
114 | } else if (options.getUsername() != null && options.getPassword() != null) {
115 | try {
116 | return sessionTokenService.login(options.getLoginUrl(),
117 | options.getUsername(), options.getPassword(), options.useProvidedLoginUrl());
118 | } catch (Exception e) {
119 | close();
120 | throw new IllegalArgumentException("cannot log in with username/password", e);
121 | }
122 | } else {
123 | logger.warn("Please use either username/password or session token for authentication");
124 | close();
125 | return null;
126 | }
127 | }
128 |
129 | /**
130 | * Helper function to setup the topic details in the PublishUnary, PublishStream and
131 | * SubscribeStream examples. Function also checks whether the topic under consideration
132 | * can publish or subscribe.
133 | *
134 | * @param topicName name of the topic
135 | * @param pubOrSubMode publish mode if true, subscribe mode if false
136 | * @param fetchSchema specify whether schema info has to be fetched
137 | */
138 | protected void setupTopicDetails(final String topicName, final boolean pubOrSubMode, final boolean fetchSchema) {
139 | if (topicName != null && !topicName.isEmpty()) {
140 | try {
141 | topicInfo = blockingStub.getTopic(TopicRequest.newBuilder().setTopicName(topicName).build());
142 | tenantGuid = topicInfo.getTenantGuid();
143 | busTopicName = topicInfo.getTopicName();
144 |
145 | if (pubOrSubMode && !topicInfo.getCanPublish()) {
146 | throw new IllegalArgumentException(
147 | "Topic " + topicInfo.getTopicName() + " is not available for publish");
148 | }
149 |
150 | if (!pubOrSubMode && !topicInfo.getCanSubscribe()) {
151 | throw new IllegalArgumentException(
152 | "Topic " + topicInfo.getTopicName() + " is not available for subscribe");
153 | }
154 |
155 | if (fetchSchema) {
156 | SchemaRequest schemaRequest = SchemaRequest.newBuilder().setSchemaId(topicInfo.getSchemaId())
157 | .build();
158 | schemaInfo = blockingStub.getSchema(schemaRequest);
159 | }
160 | } catch (final Exception ex) {
161 | logger.error("Error during fetching topic", ex);
162 | close();
163 | throw ex;
164 | }
165 | }
166 | }
167 |
168 | /**
169 | * Helper function to convert the replayId in long to ByteString type.
170 | *
171 | * @param replayValue value of the replayId in long
172 | * @return ByteString value of the replayId
173 | */
174 | public static ByteString getReplayIdFromLong(long replayValue) {
175 | ByteBuffer buffer = ByteBuffer.allocate(8);
176 | buffer.putLong(replayValue);
177 | buffer.flip();
178 |
179 | return ByteString.copyFrom(buffer);
180 | }
181 |
182 | /**
183 | * Helper function to create an event.
184 | * Currently generates event message for the topic "Order Event". Modify the fields
185 | * accordingly for an event of your choice.
186 | *
187 | * @param schema schema of the topic
188 | * @return
189 | */
190 | public GenericRecord createEventMessage(Schema schema) {
191 | // Update CreatedById with the appropriate User Id from your org.
192 | return new GenericRecordBuilder(schema).set("CreatedDate", System.currentTimeMillis())
193 | .set("CreatedById", "").set("Order_Number__c", "1")
194 | .set("City__c", "Los Angeles").set("Amount__c", 35.0).build();
195 | }
196 |
197 | /**
198 | * Helper function to create an event with a counter appended to
199 | * the end of a Text field. Used while publishing multiple events.
200 | * Currently generates event message for the topic "Order Event". Modify the fields
201 | * accordingly for an event of your choice.
202 | *
203 | * @param schema schema of the topic
204 | * @param counter counter to be appended towards the end of any Text Field
205 | * @return
206 | */
207 | public GenericRecord createEventMessage(Schema schema, final int counter) {
208 | // Update CreatedById with the appropriate User Id from your org.
209 | return new GenericRecordBuilder(schema).set("CreatedDate", System.currentTimeMillis())
210 | .set("CreatedById", "").set("Order_Number__c", String.valueOf(counter+1))
211 | .set("City__c", "Los Angeles").set("Amount__c", 35.0).build();
212 | }
213 |
214 | public List createEventMessages(Schema schema, final int numEvents) {
215 |
216 | String[] orderNumbers = {"99","100","101","102","103"};
217 | String[] cities = {"Los Angeles", "New York", "San Francisco", "San Jose", "Boston"};
218 | Double[] amounts = {35.0, 20.0, 2.0, 123.0, 180.0};
219 |
220 | // Update CreatedById with the appropriate User Id from your org.
221 | List events = new ArrayList<>();
222 | for (int i=0; i").set("Order_Number__c", orderNumbers[i % 5])
225 | .set("City__c", cities[i % 5]).set("Amount__c", amounts[i % 5]).build());
226 | }
227 |
228 | return events;
229 | }
230 |
231 |
232 | /**
233 | * Helper function to print the gRPC exception and trailers while a
234 | * StatusRuntimeException is caught
235 | *
236 | * @param context
237 | * @param e
238 | */
239 | public static final void printStatusRuntimeException(final String context, final Exception e) {
240 | logger.error(context);
241 |
242 | if (e instanceof StatusRuntimeException) {
243 | final StatusRuntimeException expected = (StatusRuntimeException)e;
244 | logger.error(" === GRPC Exception ===", e);
245 | Metadata trailers = ((StatusRuntimeException)e).getTrailers();
246 | logger.error(" === Trailers ===");
247 | trailers.keys().stream().forEach(t -> {
248 | logger.error("[Trailer] = " + t + " [Value] = "
249 | + trailers.get(Metadata.Key.of(t, Metadata.ASCII_STRING_MARSHALLER)));
250 | });
251 | } else {
252 | logger.error(" === Exception ===", e);
253 | }
254 | }
255 |
256 | /**
257 | * Helper function to deserialize the event payload received in bytes.
258 | *
259 | * @param schema
260 | * @param payload
261 | * @return
262 | * @throws IOException
263 | */
264 | public static GenericRecord deserialize(Schema schema, ByteString payload) throws IOException {
265 | DatumReader reader = new GenericDatumReader(schema);
266 | ByteArrayInputStream in = new ByteArrayInputStream(payload.toByteArray());
267 | BinaryDecoder decoder = DecoderFactory.get().directBinaryDecoder(in, null);
268 | return reader.read(null, decoder);
269 | }
270 |
271 | /**
272 | * Helper function to process and print bitmap fields
273 | *
274 | * @param schema
275 | * @param record
276 | * @param bitmapField
277 | * @return
278 | */
279 | public static void processAndPrintBitmapFields(Schema schema, GenericRecord record, String bitmapField) {
280 | String bitmapFieldPascal = CaseFormat.LOWER_CAMEL.to(CaseFormat.UPPER_CAMEL, bitmapField);
281 | try {
282 | List changedFields = getFieldListFromBitmap(schema,
283 | (GenericData.Record) record.get("ChangeEventHeader"), bitmapField);
284 | if (!changedFields.isEmpty()) {
285 | logger.info("============================");
286 | logger.info(" " + bitmapFieldPascal + " ");
287 | logger.info("============================");
288 | for (String field : changedFields) {
289 | logger.info(field);
290 | }
291 | logger.info("============================\n");
292 | } else {
293 | logger.info("No " + bitmapFieldPascal + " found\n");
294 | }
295 | } catch (Exception e) {
296 | logger.info("Trying to process " + bitmapFieldPascal + " on unsupported events or no " +
297 | bitmapFieldPascal + " found. Error: " + e.getMessage() + "\n");
298 | }
299 | }
300 |
301 | /**
302 | * Helper function to setup Subscribe configurations in some examples.
303 | *
304 | * @param requiredParams
305 | * @param topic
306 | * @return
307 | */
308 | public static ExampleConfigurations setupSubscriberParameters(ExampleConfigurations requiredParams, String topic, int numberOfEvents) {
309 | ExampleConfigurations subParams = new ExampleConfigurations();
310 | setCommonParameters(subParams, requiredParams);
311 | subParams.setTopic(topic);
312 | subParams.setReplayPreset(ReplayPreset.LATEST);
313 | subParams.setNumberOfEventsToSubscribeInEachFetchRequest(numberOfEvents);
314 | return subParams;
315 | }
316 |
317 | /**
318 | * Helper function to setup Publish configurations in some examples.
319 | *
320 | * @param requiredParams
321 | * @param topic
322 | * @return
323 | */
324 | public static ExampleConfigurations setupPublisherParameters(ExampleConfigurations requiredParams, String topic) {
325 | ExampleConfigurations pubParams = new ExampleConfigurations();
326 | setCommonParameters(pubParams, requiredParams);
327 | pubParams.setTopic(topic);
328 | return pubParams;
329 | }
330 |
331 | /**
332 | * Helper function to setup common configurations for publish and subscribe operations.
333 | *
334 | * @param ep
335 | * @param requiredParams
336 | */
337 | private static void setCommonParameters(ExampleConfigurations ep, ExampleConfigurations requiredParams) {
338 | ep.setLoginUrl(requiredParams.getLoginUrl());
339 | ep.setPubsubHost(requiredParams.getPubsubHost());
340 | ep.setPubsubPort(requiredParams.getPubsubPort());
341 | if (requiredParams.getUsername() != null && requiredParams.getPassword() != null) {
342 | ep.setUsername(requiredParams.getUsername());
343 | ep.setPassword(requiredParams.getPassword());
344 | } else {
345 | ep.setAccessToken(requiredParams.getAccessToken());
346 | ep.setTenantId(requiredParams.getTenantId());
347 | }
348 | ep.setPlaintextChannel(requiredParams.usePlaintextChannel());
349 | }
350 |
351 | /**
352 | * General getters.
353 | */
354 | public String getSessionToken() {
355 | return sessionToken;
356 | }
357 |
358 | /**
359 | * Implementation of the close() function from AutoCloseable interface for relinquishing the
360 | * resources used in the try-with-resource blocks in the examples and the resources used
361 | * in this class.
362 | */
363 | @Override
364 | public void close() {
365 | if (httpClient != null) {
366 | try {
367 | httpClient.stop();
368 | } catch (Throwable t) {
369 | logger.warn("Cannot stop session HTTP client", t);
370 | }
371 | }
372 |
373 | try {
374 | channel.shutdown().awaitTermination(20, TimeUnit.SECONDS);
375 | } catch (Throwable t) {
376 | logger.warn("Cannot shutdown GRPC channel", t);
377 | }
378 | }
379 | }
--------------------------------------------------------------------------------
/java/src/main/java/utility/EventParser.java:
--------------------------------------------------------------------------------
1 | package utility;
2 |
3 | import java.io.IOException;
4 | import java.util.ArrayList;
5 | import java.util.BitSet;
6 | import java.util.List;
7 | import java.util.ListIterator;
8 |
9 | import org.apache.avro.Schema;
10 | import org.apache.avro.Schema.Type;
11 | import org.apache.avro.generic.GenericDatumReader;
12 | import org.apache.avro.generic.GenericRecord;
13 | import org.apache.avro.generic.GenericData.Array;
14 | import org.apache.avro.generic.GenericData.Record;
15 | import org.apache.avro.io.DatumReader;
16 | import org.apache.avro.util.Utf8;
17 |
18 | /**
19 | * A utility class used to generate the field names from bitmap encoded values.
20 | *
21 | * @author pozil
22 | */
23 | public class EventParser {
24 |
25 | private Schema schema;
26 | private DatumReader datumReader;
27 |
28 | public EventParser(Schema schema) {
29 | this.schema = schema;
30 | this.datumReader = new GenericDatumReader(schema);
31 | }
32 |
33 | /**
34 | * Retrieves the list of fields from a bitmap encoded value.
35 | *
36 | * @param schema
37 | * @param eventHeader
38 | * @param fieldName
39 | * @return
40 | * @throws IOException
41 | */
42 | public static List getFieldListFromBitmap(Schema schema, Record eventHeader, String fieldName)
43 | throws IOException {
44 | @SuppressWarnings("unchecked")
45 | Array utf8Values = (Array) eventHeader.get(fieldName);
46 | List values = new ArrayList<>();
47 | for (Utf8 utf8Value : utf8Values) {
48 | values.add(utf8Value.toString());
49 | }
50 | expandBitmap(schema, values);
51 | return values;
52 | }
53 |
54 | /**
55 | * Translate a bitmap-compressed field list into its expanded representation as
56 | * a list of field names
57 | */
58 | public static void expandBitmap(Schema schema, List val) {
59 | if (val != null && !val.isEmpty()) {
60 | // replace top field level bitmap with list of fields
61 | if (val.get(0).startsWith("0x")) {
62 | String bitMap = val.get(0);
63 | val.addAll(0, fieldNamesFromBitmap(schema, bitMap));
64 | val.remove(bitMap);
65 | }
66 | // replace parentPos-nestedNulledBitMap with list of fields too
67 | if ((val.get(val.size() - 1)).contains("-")) {
68 | for (ListIterator itr = val.listIterator(); itr.hasNext();) {
69 |
70 | String[] bitmapMapStrings = (itr.next()).split("-");
71 | if (bitmapMapStrings.length < 2)
72 | continue; // that's the first top level field bitmap;
73 |
74 | // interpret the parent field name from mapping of parentFieldPos ->
75 | // childFieldbitMap
76 | Schema.Field parentField = schema.getFields().get(Integer.valueOf(bitmapMapStrings[0]));
77 | Schema childSchema = getValueSchema(parentField.schema());
78 |
79 | if (childSchema.getType().equals(Schema.Type.RECORD)) { // make sure we're really dealing with
80 | // compound field
81 | int nestedSize = childSchema.getFields().size();
82 | String parentFieldName = parentField.name();
83 |
84 | // interpret the child field names from mapping of parentFieldPos ->
85 | // childFieldbitMap
86 | List fullFieldNames = new ArrayList<>();
87 | fieldNamesFromBitmap(childSchema, bitmapMapStrings[1]).stream()
88 | .map(col -> parentFieldName + "." + col).forEach(fullFieldNames::add);
89 | if (fullFieldNames.size() > 0) {
90 | itr.remove();
91 | // when all nested fields under a compound got nulled out at once by customer,
92 | // we recognize the top level field instead of trying to list every single
93 | // nested field
94 | if (fullFieldNames.size() == nestedSize) {
95 | itr.add(parentFieldName);
96 | } else {
97 | fullFieldNames.stream().forEach(itr::add);
98 | }
99 | }
100 | }
101 | }
102 | }
103 | }
104 | }
105 |
106 | /**
107 | * Convert bitmap representation into list of fields based on Avro schema
108 | *
109 | * @param schema schema
110 | * @param bitmap bitmap of nulled fields
111 | * @return list of fields corresponding to bitmap
112 | */
113 | private static List fieldNamesFromBitmap(Schema schema, String bitmap) {
114 | BitSet bitSet = convertHexStringToBitSet(bitmap);
115 | List fieldList = new ArrayList<>();
116 | bitSet.stream().mapToObj(pos -> schema.getFields().get(pos).name())
117 | .forEach(fieldName -> fieldList.add(fieldName));
118 | return fieldList;
119 | }
120 |
121 | /**
122 | * Converts a hexadecimal string into a BitSet
123 | *
124 | * @param hex
125 | * @return BitSet
126 | */
127 | private static BitSet convertHexStringToBitSet(String hex) {
128 | // Parse hex string as bytes
129 | String s = hex.substring(2); // Strip out 0x prefix
130 | int len = s.length();
131 | byte[] bytes = new byte[len / 2];
132 | for (int i = 0; i < len; i += 2) {
133 | // using left shift operator on every character
134 | bytes[i / 2] = (byte) ((Character.digit(s.charAt(i), 16) << 4) + Character.digit(s.charAt(i + 1), 16));
135 | }
136 | // Reverse bytes
137 | len /= 2;
138 | byte[] reversedBytes = new byte[len];
139 | for (int i = 0; i < len; i++) {
140 | reversedBytes[i] = bytes[len - i - 1];
141 | }
142 | // Return value as BitSet
143 | return BitSet.valueOf(reversedBytes);
144 | }
145 |
146 | /**
147 | * Get the value type of an "optional" schema, which is a union of [null,
148 | * valueSchema]
149 | *
150 | * @param schema
151 | * @return value schema or the original schema if it does not look like optional
152 | */
153 | private static Schema getValueSchema(Schema schema) {
154 | if (schema.getType() == Schema.Type.UNION) {
155 | List types = schema.getTypes();
156 | if (types.size() == 2 && types.get(0).getType() == Type.NULL) {
157 | // Optional is a union of (null, ), return the underlying type
158 | return types.get(1);
159 | } else if (types.size() == 2 && types.get(0).getType() == Type.STRING) {
160 | // for required Switchable_PersonName
161 | return schema.getTypes().get(1);
162 | } else if (types.size() == 3 && types.get(0).getType() == Type.NULL
163 | && types.get(1).getType() == Type.STRING) {
164 | // for optional Switchable_PersonName
165 | return schema.getTypes().get(2);
166 | }
167 | }
168 | return schema;
169 | }
170 | }
--------------------------------------------------------------------------------
/java/src/main/java/utility/ExampleConfigurations.java:
--------------------------------------------------------------------------------
1 | package utility;
2 |
3 | import java.io.FileInputStream;
4 | import java.io.IOException;
5 | import java.io.InputStream;
6 | import java.util.HashMap;
7 |
8 | import org.yaml.snakeyaml.Yaml;
9 |
10 | import com.google.protobuf.ByteString;
11 | import com.salesforce.eventbus.protobuf.ReplayPreset;
12 |
13 | /**
14 | * The ExampleConfigurations class is used for setting up the configurations for running the examples.
15 | * The configurations can be read from a YAML file or created directly via an object. It also sets
16 | * default values when an optional configuration is not specified.
17 | */
18 | public class ExampleConfigurations {
19 | private String username;
20 | private String password;
21 | private String loginUrl;
22 | private String tenantId;
23 | private String accessToken;
24 | private String pubsubHost;
25 | private Integer pubsubPort;
26 | private String topic;
27 | private Integer numberOfEventsToPublish;
28 | private Boolean singlePublishRequest;
29 | private Integer numberOfEventsToSubscribeInEachFetchRequest;
30 | private Boolean processChangedFields;
31 | private Boolean plaintextChannel;
32 | private Boolean providedLoginUrl;
33 | private ReplayPreset replayPreset;
34 | private ByteString replayId;
35 | private String managedSubscriptionId;
36 | private String developerName;
37 |
38 | public ExampleConfigurations() {
39 | this(null, null, null, null, null,
40 | null, null, null, 5, false, 5, false,
41 | false, false, ReplayPreset.LATEST, null, null, null);
42 | }
43 | public ExampleConfigurations(String filename) throws IOException {
44 |
45 | Yaml yaml = new Yaml();
46 | InputStream inputStream = new FileInputStream("src/main/resources/"+filename);
47 | HashMap obj = yaml.load(inputStream);
48 |
49 | // Reading Required Parameters
50 | this.loginUrl = obj.get("LOGIN_URL").toString();
51 | this.pubsubHost = obj.get("PUBSUB_HOST").toString();
52 | this.pubsubPort = Integer.parseInt(obj.get("PUBSUB_PORT").toString());
53 |
54 | // Reading Optional Parameters
55 | this.username = obj.get("USERNAME") == null ? null : obj.get("USERNAME").toString();
56 | this.password = obj.get("PASSWORD") == null ? null : obj.get("PASSWORD").toString();
57 | this.topic = obj.get("TOPIC") == null ? "/event/Order_Event__e" : obj.get("TOPIC").toString();
58 | this.tenantId = obj.get("TENANT_ID") == null ? null : obj.get("TENANT_ID").toString();
59 | this.accessToken = obj.get("ACCESS_TOKEN") == null ? null : obj.get("ACCESS_TOKEN").toString();
60 | this.numberOfEventsToPublish = obj.get("NUMBER_OF_EVENTS_TO_PUBLISH") == null ?
61 | 5 : Integer.parseInt(obj.get("NUMBER_OF_EVENTS_TO_PUBLISH").toString());
62 | this.singlePublishRequest = obj.get("SINGLE_PUBLISH_REQUEST") == null ?
63 | false : Boolean.parseBoolean(obj.get("SINGLE_PUBLISH_REQUEST").toString());
64 | this.numberOfEventsToSubscribeInEachFetchRequest = obj.get("NUMBER_OF_EVENTS_IN_FETCHREQUEST") == null ?
65 | 5 : Integer.parseInt(obj.get("NUMBER_OF_EVENTS_IN_FETCHREQUEST").toString());
66 | this.processChangedFields = obj.get("PROCESS_CHANGE_EVENT_HEADER_FIELDS") == null ?
67 | false : Boolean.parseBoolean(obj.get("PROCESS_CHANGE_EVENT_HEADER_FIELDS").toString());
68 | this.plaintextChannel = obj.get("USE_PLAINTEXT_CHANNEL") != null && Boolean.parseBoolean(obj.get("USE_PLAINTEXT_CHANNEL").toString());
69 | this.providedLoginUrl = obj.get("USE_PROVIDED_LOGIN_URL") != null && Boolean.parseBoolean(obj.get("USE_PROVIDED_LOGIN_URL").toString());
70 |
71 | if (obj.get("REPLAY_PRESET") != null) {
72 | if (obj.get("REPLAY_PRESET").toString().equals("EARLIEST")) {
73 | this.replayPreset = ReplayPreset.EARLIEST;
74 | } else if (obj.get("REPLAY_PRESET").toString().equals("CUSTOM")) {
75 | this.replayPreset = ReplayPreset.CUSTOM;
76 | this.replayId = getByteStringFromReplayIdInputString(obj.get("REPLAY_ID").toString());
77 | } else {
78 | this.replayPreset = ReplayPreset.LATEST;
79 | }
80 | } else {
81 | this.replayPreset = ReplayPreset.LATEST;
82 | }
83 |
84 | this.developerName = obj.get("MANAGED_SUB_DEVELOPER_NAME") == null ? null : obj.get("MANAGED_SUB_DEVELOPER_NAME").toString();
85 | this.managedSubscriptionId = obj.get("MANAGED_SUB_ID") == null ? null : obj.get("MANAGED_SUB_ID").toString();
86 | }
87 |
88 | public ExampleConfigurations(String username, String password, String loginUrl,
89 | String pubsubHost, int pubsubPort, String topic) {
90 | this(username, password, loginUrl, null, null, pubsubHost, pubsubPort, topic,
91 | 5, false, Integer.MAX_VALUE, false, false, false, ReplayPreset.LATEST, null, null, null);
92 | }
93 |
94 | public ExampleConfigurations(String username, String password, String loginUrl, String tenantId, String accessToken,
95 | String pubsubHost, Integer pubsubPort, String topic, Integer numberOfEventsToPublish,
96 | Boolean singlePublishRequest, Integer numberOfEventsToSubscribeInEachFetchRequest,
97 | Boolean processChangedFields, Boolean plaintextChannel, Boolean providedLoginUrl,
98 | ReplayPreset replayPreset, ByteString replayId, String devName, String managedSubId) {
99 | this.username = username;
100 | this.password = password;
101 | this.loginUrl = loginUrl;
102 | this.tenantId = tenantId;
103 | this.accessToken = accessToken;
104 | this.pubsubHost = pubsubHost;
105 | this.pubsubPort = pubsubPort;
106 | this.topic = topic;
107 | this.singlePublishRequest = singlePublishRequest;
108 | this.numberOfEventsToPublish = numberOfEventsToPublish;
109 | this.numberOfEventsToSubscribeInEachFetchRequest = numberOfEventsToSubscribeInEachFetchRequest;
110 | this.processChangedFields = processChangedFields;
111 | this.plaintextChannel = plaintextChannel;
112 | this.providedLoginUrl = providedLoginUrl;
113 | this.replayPreset = replayPreset;
114 | this.replayId = replayId;
115 | this.developerName = devName;
116 | this.managedSubscriptionId = managedSubId;
117 | }
118 |
119 | public String getUsername() {
120 | return username;
121 | }
122 |
123 | public void setUsername(String username) {
124 | this.username = username;
125 | }
126 |
127 | public String getPassword() {
128 | return password;
129 | }
130 |
131 | public void setPassword(String password) {
132 | this.password = password;
133 | }
134 |
135 | public String getLoginUrl() {
136 | return loginUrl;
137 | }
138 |
139 | public void setLoginUrl(String loginUrl) {
140 | this.loginUrl = loginUrl;
141 | }
142 |
143 | public String getTenantId() {
144 | return tenantId;
145 | }
146 |
147 | public void setTenantId(String tenantId) {
148 | this.tenantId = tenantId;
149 | }
150 |
151 | public String getAccessToken() {
152 | return accessToken;
153 | }
154 |
155 | public void setAccessToken(String accessToken) {
156 | this.accessToken = accessToken;
157 | }
158 |
159 | public String getPubsubHost() {
160 | return pubsubHost;
161 | }
162 |
163 | public void setPubsubHost(String pubsubHost) {
164 | this.pubsubHost = pubsubHost;
165 | }
166 |
167 | public int getPubsubPort() {
168 | return pubsubPort;
169 | }
170 |
171 | public void setPubsubPort(int pubsubPort) {
172 | this.pubsubPort = pubsubPort;
173 | }
174 |
175 | public Integer getNumberOfEventsToPublish() {
176 | return numberOfEventsToPublish;
177 | }
178 |
179 | public void setNumberOfEventsToPublish(Integer numberOfEventsToPublish) {
180 | this.numberOfEventsToPublish = numberOfEventsToPublish;
181 | }
182 |
183 | public Boolean getSinglePublishRequest() {
184 | return singlePublishRequest;
185 | }
186 |
187 | public void setSinglePublishRequest(Boolean singlePublishRequest) {
188 | this.singlePublishRequest = singlePublishRequest;
189 | }
190 |
191 | public int getNumberOfEventsToSubscribeInEachFetchRequest() {
192 | return numberOfEventsToSubscribeInEachFetchRequest;
193 | }
194 |
195 | public void setNumberOfEventsToSubscribeInEachFetchRequest(int numberOfEventsToSubscribeInEachFetchRequest) {
196 | this.numberOfEventsToSubscribeInEachFetchRequest = numberOfEventsToSubscribeInEachFetchRequest;
197 | }
198 |
199 | public Boolean getProcessChangedFields() {
200 | return processChangedFields;
201 | }
202 |
203 | public void setProcessChangedFields(Boolean processChangedFields) {
204 | this.processChangedFields = processChangedFields;
205 | }
206 |
207 | public boolean usePlaintextChannel() {
208 | return plaintextChannel;
209 | }
210 |
211 | public void setPlaintextChannel(boolean plaintextChannel) {
212 | this.plaintextChannel = plaintextChannel;
213 | }
214 |
215 | public Boolean useProvidedLoginUrl() {
216 | return providedLoginUrl;
217 | }
218 |
219 | public String getTopic() {
220 | return topic;
221 | }
222 |
223 | public void setTopic(String topic) {
224 | this.topic = topic;
225 | }
226 |
227 | public void setProvidedLoginUrl(Boolean providedLoginUrl) {
228 | this.providedLoginUrl = providedLoginUrl;
229 | }
230 |
231 | public ReplayPreset getReplayPreset() {
232 | return replayPreset;
233 | }
234 |
235 | public void setReplayPreset(ReplayPreset replayPreset) {
236 | this.replayPreset = replayPreset;
237 | }
238 |
239 | public ByteString getReplayId() {
240 | return replayId;
241 | }
242 |
243 | public void setReplayId(ByteString replayId) {
244 | this.replayId = replayId;
245 | }
246 |
247 | public String getManagedSubscriptionId() {
248 | return managedSubscriptionId;
249 | }
250 |
251 | public void setManagedSubscriptionId(String managedSubscriptionId) {
252 | this.managedSubscriptionId = managedSubscriptionId;
253 | }
254 |
255 | public String getDeveloperName() {
256 | return developerName;
257 | }
258 |
259 | public void setDeveloperName(String developerName) {
260 | this.developerName = developerName;
261 | }
262 |
263 |
264 | /**
265 | * NOTE: replayIds are meant to be opaque (See docs: https://developer.salesforce.com/docs/platform/pub-sub-api/guide/intro.html)
266 | * and this is used for example purposes only. A long-lived subscription client will use the stored replay to
267 | * resubscribe on failure. The stored replay should be in bytes and not in any other form.
268 | */
269 | public ByteString getByteStringFromReplayIdInputString(String input) {
270 | ByteString replayId;
271 | String[] values = input.substring(1, input.length()-2).split(",");
272 | byte[] b = new byte[values.length];
273 | int i=0;
274 | for (String x : values) {
275 | if (x.strip().length() != 0) {
276 | b[i++] = (byte)Integer.parseInt(x.strip());
277 | }
278 | }
279 | replayId = ByteString.copyFrom(b);
280 | return replayId;
281 | }
282 | }
283 |
--------------------------------------------------------------------------------
/java/src/main/java/utility/SessionTokenService.java:
--------------------------------------------------------------------------------
1 | package utility;
2 |
3 | import java.io.ByteArrayInputStream;
4 | import java.io.UnsupportedEncodingException;
5 | import java.net.ConnectException;
6 | import java.net.URL;
7 | import java.nio.ByteBuffer;
8 |
9 | import javax.xml.parsers.SAXParser;
10 | import javax.xml.parsers.SAXParserFactory;
11 |
12 | import org.eclipse.jetty.client.HttpClient;
13 | import org.eclipse.jetty.client.api.ContentResponse;
14 | import org.eclipse.jetty.client.api.Request;
15 | import org.eclipse.jetty.client.util.ByteBufferContentProvider;
16 | import org.slf4j.Logger;
17 | import org.slf4j.LoggerFactory;
18 | import org.xml.sax.Attributes;
19 | import org.xml.sax.SAXException;
20 | import org.xml.sax.helpers.DefaultHandler;
21 |
22 | /**
23 | * The SessionTokenService class is used for logging into the org used by the customer using the
24 | * Salesforce SOAP API and retrieving the tenandId and session token which will be used to create
25 | * CallCredentials. It also has a static subclass that parses the LoginResponse.
26 | */
27 | public class SessionTokenService {
28 | private static final Logger LOGGER = LoggerFactory.getLogger(SessionTokenService.class);
29 |
30 | private static final String ENV_END = "";
31 | private static final String ENV_START =
32 | "";
35 |
36 | // The enterprise SOAP API endpoint used for the login call
37 | private static final String SERVICES_SOAP_PARTNER_ENDPOINT = "/services/Soap/u/43.0/";
38 |
39 | // HttpClient is thread safe and meant to be shared; assume callers are managing its life cycle correctly
40 | private final HttpClient httpClient;
41 |
42 | public SessionTokenService(HttpClient httpClient) {
43 | if (httpClient == null) {
44 | throw new IllegalArgumentException("HTTP client cannot be null");
45 | }
46 |
47 | this.httpClient = httpClient;
48 | }
49 |
50 | /**
51 | * Function to login with the username/password of the client.
52 | *
53 | * @param loginEndpoint
54 | * @param user
55 | * @param pwd
56 | * @param useProvidedLoginUrl
57 | * @return
58 | * @throws Exception
59 | */
60 | public APISessionCredentials login(String loginEndpoint, String user, String pwd, boolean useProvidedLoginUrl) throws Exception {
61 | URL endpoint;
62 | endpoint = new URL(new URL(loginEndpoint), SERVICES_SOAP_PARTNER_ENDPOINT);
63 | LOGGER.trace("requesting session token from {}", endpoint);
64 | Request post = httpClient.POST(endpoint.toURI());
65 | post.content(new ByteBufferContentProvider("text/xml", ByteBuffer.wrap(soapXmlForLogin(user, pwd))));
66 | post.header("SOAPAction", "''");
67 | post.header("PrettyPrint", "Yes");
68 | ContentResponse response = post.send();
69 | LoginResponseParser parser = parse(response);
70 |
71 | final String token = parser.sessionId;
72 | if (token == null || parser.serverUrl == null) {
73 | throw new ConnectException(String.format("Unable to login: %s", parser.faultstring));
74 | }
75 |
76 | if (null == parser.organizationId) {
77 | throw new ConnectException(
78 | String.format("Unable to login: organization id is not found in the response"));
79 | }
80 |
81 | String url;
82 | if (useProvidedLoginUrl) {
83 | url = loginEndpoint;
84 | } else {
85 | // Form url to this format: https://na44.stmfa.stm.salesforce.com
86 | URL soapEndpoint = new URL(parser.serverUrl);
87 | url = soapEndpoint.getProtocol() + "://" + soapEndpoint.getHost();
88 | // Adding port info for local app setup
89 | if (soapEndpoint.getPort() > -1) {
90 | url += ":" + soapEndpoint.getPort();
91 | }
92 | }
93 |
94 | LOGGER.debug("created session token credentials for {} from {}", parser.organizationId, url);
95 | return new APISessionCredentials(parser.organizationId, url, token);
96 | }
97 |
98 | /**
99 | * Function to login with the tenantId and session token of the client.
100 | *
101 | * @param loginEndpoint
102 | * @param accessToken
103 | * @param tenantId
104 | * @return
105 | */
106 | public APISessionCredentials loginWithAccessToken(String loginEndpoint, String accessToken, String tenantId) {
107 | return new APISessionCredentials(tenantId, loginEndpoint, accessToken);
108 | }
109 |
110 | private static class LoginResponseParser extends DefaultHandler {
111 |
112 | private String buffer;
113 | private String faultstring;
114 |
115 | private boolean reading = false;
116 | private String serverUrl;
117 | private String sessionId;
118 | private String organizationId;
119 |
120 | @Override
121 | public void characters(char[] ch, int start, int length) {
122 | if (reading) {
123 | buffer = new String(ch, start, length);
124 | }
125 | }
126 |
127 | @Override
128 | public void endElement(String uri, String localName, String qName) {
129 | reading = false;
130 | switch (localName) {
131 | case "organizationId":
132 | organizationId = buffer;
133 | break;
134 | case "sessionId":
135 | sessionId = buffer;
136 | break;
137 | case "serverUrl":
138 | serverUrl = buffer;
139 | break;
140 | case "faultstring":
141 | faultstring = buffer;
142 | break;
143 | default:
144 | }
145 | buffer = null;
146 | }
147 |
148 | @Override
149 | public void startElement(String uri, String localName, String qName, Attributes attributes) {
150 | switch (localName) {
151 | case "sessionId":
152 | case "serverUrl":
153 | case "faultstring":
154 | case "organizationId":
155 | reading = true;
156 | break;
157 | default:
158 | }
159 | }
160 | }
161 |
162 | private static LoginResponseParser parse(ContentResponse response) throws Exception {
163 | try {
164 | SAXParserFactory spf = SAXParserFactory.newInstance();
165 | spf.setFeature("http://xml.org/sax/features/external-general-entities", false);
166 | spf.setFeature("http://xml.org/sax/features/external-parameter-entities", false);
167 | spf.setFeature("http://apache.org/xml/features/nonvalidating/load-external-dtd", false);
168 | spf.setFeature("http://apache.org/xml/features/disallow-doctype-decl", true);
169 | spf.setNamespaceAware(true);
170 | SAXParser saxParser = spf.newSAXParser();
171 |
172 | LoginResponseParser parser = new LoginResponseParser();
173 |
174 | saxParser.parse(new ByteArrayInputStream(response.getContent()), parser);
175 |
176 | return parser;
177 | } catch (SAXException e) {
178 | throw new Exception(String.format("Unable to login: %s::%s", response.getStatus(), response.getReason()));
179 | }
180 | }
181 |
182 | private static byte[] soapXmlForLogin(String username, String password) throws UnsupportedEncodingException {
183 | return (ENV_START + " " + " " + username + "" + " "
184 | + password + "" + " " + ENV_END).getBytes("UTF-8");
185 | }
186 | }
187 |
--------------------------------------------------------------------------------
/java/src/main/java/utility/XClientTraceIdClientInterceptor.java:
--------------------------------------------------------------------------------
1 | package utility;
2 |
3 | import java.util.UUID;
4 |
5 | import org.slf4j.Logger;
6 | import org.slf4j.LoggerFactory;
7 |
8 | import io.grpc.*;
9 |
10 | public class XClientTraceIdClientInterceptor implements ClientInterceptor {
11 | private static final Logger logger = LoggerFactory.getLogger(XClientTraceIdClientInterceptor.class.getClass());
12 | private static final Metadata.Key X_CLIENT_TRACE_ID = Metadata.Key.of("x-client-trace-id", Metadata.ASCII_STRING_MARSHALLER);
13 |
14 | @Override
15 | public ClientCall interceptCall(MethodDescriptor method,
16 | CallOptions callOptions, Channel next) {
17 | return new ForwardingClientCall.SimpleForwardingClientCall<>(next.newCall(method, callOptions)) {
18 |
19 | @Override
20 | public void start(Listener responseListener, Metadata headers) {
21 | String xClientTraceId = UUID.randomUUID().toString();
22 | headers.put(X_CLIENT_TRACE_ID, xClientTraceId);
23 | logger.info("sending request for xClientTraceId {}", xClientTraceId);
24 |
25 | super.start(new ForwardingClientCallListener.SimpleForwardingClientCallListener<>(responseListener) {
26 | @Override
27 | public void onClose(Status status, Metadata trailers) {
28 | logger.info("request completed for xClientTraceId {} with status {}", xClientTraceId, status);
29 | super.onClose(status, trailers);
30 | }
31 | }, headers);
32 | }
33 | };
34 | }
35 | }
36 |
--------------------------------------------------------------------------------
/java/src/main/resources/arguments.yaml:
--------------------------------------------------------------------------------
1 | # 'arguments.yaml' contains the required and optional configurations for running the examples.
2 | #
3 | # Note: Please ensure to specify a value of `null` to all optional configurations when
4 | # you do not wish to specify a value for the same. Some optional configurations will be
5 | # initialised with default values specified below.
6 |
7 | # =========================
8 | # Required Configurations:
9 | # =========================
10 | # Pub/Sub API Endpoint
11 | PUBSUB_HOST: api.pubsub.salesforce.com
12 | # Pub/Sub API Host
13 | PUBSUB_PORT: 7443
14 | # Your Salesforce Login URL
15 | LOGIN_URL: null
16 |
17 | # For authentication, you can use either username/password or accessToken/tenantId types.
18 | # Either one of the combinations is required. Please specify `null` values to the unused type.
19 | # Your Salesforce Username
20 | USERNAME: null
21 | # Your Salesforce Password
22 | PASSWORD: null
23 | # Your Salesforce org Tenant ID
24 | TENANT_ID: null
25 | # Your Salesforce Session Token
26 | ACCESS_TOKEN: null
27 |
28 | # =========================
29 | # Optional Configurations:
30 | # =========================
31 | # Topic to publish/subscribe to (default: /event/Order_Event__e)
32 | TOPIC: null
33 | # Number of Events to publish in single or separate batches (default: 5)
34 | # Used only by PublishStream.java
35 | NUMBER_OF_EVENTS_TO_PUBLISH: null
36 | # Indicates whether to add events to a single PublishRequest (true) or
37 | # in different PublishRequests (default: false)
38 | # Used only by PublishStream.java
39 | SINGLE_PUBLISH_REQUEST: null
40 | # Number of events to subscribe to in each FetchRequest/ManagedFetchRequest (default: 5)
41 | NUMBER_OF_EVENTS_IN_FETCHREQUEST: null
42 | # ReplayPreset (Accepted Values: {EARLIEST, LATEST (default), CUSTOM})
43 | REPLAY_PRESET: null
44 | # Replay ID in ByteString
45 | REPLAY_ID: null
46 | # Flag to enable/disable processing of bitmap fields in ChangeEventHeader in Subscribe and
47 | # ManagedSubscribe examples for change data capture events (default: false)
48 | PROCESS_CHANGE_EVENT_HEADER_FIELDS: null
49 |
50 | # ManagedSubscribe RPC parameters
51 | # For ManagedSubscribe.java, either supply the developer name or the ID of ManagedEventSubscription
52 | MANAGED_SUB_DEVELOPER_NAME: null
53 | MANAGED_SUB_ID: null
--------------------------------------------------------------------------------
/java/src/main/resources/logback.xml:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 |
6 |
7 | %d [%thread] %logger{36} - %msg%n
8 |
9 |
10 |
11 |
12 |
13 |
14 |
15 |
16 |
--------------------------------------------------------------------------------
/python/InventoryAppExample/InventoryApp.py:
--------------------------------------------------------------------------------
1 | """
2 | InventoryApp.py
3 |
4 | This is a subscriber client that listens for Change Data Capture events for the
5 | Opportunity object and publishes `/event/NewOrderConfirmation__e` events. In
6 | the example, this file would be hosted somewhere outside of Salesforce. The `if
7 | __debug__` conditionals are to slow down the speed of the app for demoing
8 | purposes.
9 | """
10 | import os, sys, avro
11 |
12 | dir_path = os.path.dirname(os.path.realpath(__file__))
13 | parent_dir_path = os.path.abspath(os.path.join(dir_path, os.pardir))
14 | sys.path.insert(0, parent_dir_path)
15 |
16 | from datetime import datetime, timedelta
17 | import logging
18 |
19 | from PubSub import PubSub
20 | import pubsub_api_pb2 as pb2
21 | from utils.ClientUtil import command_line_input
22 | import time
23 | from util.ChangeEventHeaderUtility import process_bitmap
24 |
25 | my_publish_topic = '/event/NewOrderConfirmation__e'
26 |
27 |
28 | def make_publish_request(schema_id, record_id, obj):
29 | """
30 | Creates a PublishRequest per the proto file.
31 | """
32 | req = pb2.PublishRequest(
33 | topic_name=my_publish_topic,
34 | events=generate_producer_events(schema_id, record_id, obj))
35 | return req
36 |
37 |
38 | def generate_producer_events(schema_id, record_id, obj):
39 | """
40 | Encodes the data to be sent in the event and creates a ProducerEvent per
41 | the proto file.
42 | """
43 | schema = obj.get_schema_json(schema_id)
44 | dt = datetime.now() + timedelta(days=5)
45 | payload = {
46 | "CreatedDate": int(datetime.now().timestamp()),
47 | "CreatedById": '005R0000000cw06IAA',
48 | "OpptyRecordId__c": record_id,
49 | "EstimatedDeliveryDate__c": int(dt.timestamp()),
50 | "Weight__c": 58.2}
51 | req = {
52 | "schema_id": schema_id,
53 | "payload": obj.encode(schema, payload),
54 | }
55 | return [req]
56 |
57 |
58 | def process_order(event, pubsub):
59 | """
60 | This is a callback that gets passed to the `PubSub.subscribe()` method. It
61 | decodes the payload of the received event and extracts the opportunity ID.
62 | Next, it calls a helper function to publish the
63 | `/event/NewOrderConfirmation__e` event. For simplicity, this sample uses an
64 | estimated delivery date of five days from the current date. When no events
65 | are received within a certain time period, the API's subscribe method sends
66 | keepalive messages and the latest replay ID through this callback.
67 | """
68 | if event.events:
69 | print("Number of events received in FetchResponse: ", len(event.events))
70 | # If all requested events are delivered, release the semaphore
71 | # so that a new FetchRequest gets sent by `PubSub.fetch_req_stream()`.
72 | if event.pending_num_requested == 0:
73 | pubsub.release_subscription_semaphore()
74 |
75 | for evt in event.events:
76 | payload_bytes = evt.event.payload
77 | schema_id = evt.event.schema_id
78 | json_schema = pubsub.get_schema_json(schema_id)
79 | decoded_event = pubsub.decode(pubsub.get_schema_json(schema_id),
80 | payload_bytes)
81 |
82 | print("Received event payload: \n", decoded_event)
83 | # A change event contains the ChangeEventHeader field. Check if received event is a change event.
84 | if 'ChangeEventHeader' in decoded_event:
85 | # Decode the bitmap fields contained within the ChangeEventHeader. For example, decode the 'changedFields' field.
86 | # An example to process bitmap in 'changedFields'
87 | changed_fields = decoded_event['ChangeEventHeader']['changedFields']
88 | converted_changed_fields = process_bitmap(avro.schema.parse(json_schema), changed_fields)
89 | print("Change Type: " + decoded_event['ChangeEventHeader']['changeType'])
90 | print("=========== Changed Fields =============")
91 | print(converted_changed_fields)
92 | print("=========================================")
93 | # Do not process updates made by the SalesforceListener app to the opportunity record delivery date
94 | if decoded_event['ChangeEventHeader']['changeOrigin'].find('client=SalesforceListener') != -1:
95 | print("Skipping change event because it is an update to the delivery date by SalesforceListener.")
96 | return
97 |
98 | record_id = decoded_event['ChangeEventHeader']['recordIds'][0]
99 |
100 | if __debug__:
101 | time.sleep(10)
102 | print("> Received new order! Processing order...")
103 | if __debug__:
104 | time.sleep(4)
105 | print(" Done! Order replicated in inventory system.")
106 | if __debug__:
107 | time.sleep(2)
108 | print("> Calculating estimated delivery date...")
109 | if __debug__:
110 | time.sleep(2)
111 | print(" Done! Sending estimated delivery date back to Salesforce.")
112 | if __debug__:
113 | time.sleep(10)
114 |
115 | topic_info = pubsub.get_topic(topic_name=my_publish_topic)
116 |
117 | # Publish NewOrderConfirmation__e event
118 | res = pubsub.stub.Publish(make_publish_request(topic_info.schema_id, record_id, pubsub),
119 | metadata=pubsub.metadata)
120 | if res.results[0].replay_id:
121 | print("> Event published successfully.")
122 | else:
123 | print("> Failed publishing event.")
124 | else:
125 | print("[", time.strftime('%b %d, %Y %l:%M%p %Z'), "] The subscription is active.")
126 |
127 | # The replay_id is used to resubscribe after this position in the stream if the client disconnects.
128 | # Implement storage of replay for resubscribe!!!
129 | event.latest_replay_id
130 |
131 |
132 | def run(argument_dict):
133 | cdc_listener = PubSub(argument_dict)
134 | cdc_listener.auth()
135 |
136 | # Subscribe to Opportunity CDC events
137 | cdc_listener.subscribe('/data/OpportunityChangeEvent', "LATEST", "", 1, process_order)
138 |
139 |
140 | if __name__ == '__main__':
141 | argument_dict = command_line_input(sys.argv[1:])
142 | logging.basicConfig()
143 | run(argument_dict)
144 |
--------------------------------------------------------------------------------
/python/InventoryAppExample/PubSub.py:
--------------------------------------------------------------------------------
1 | """
2 | PubSub.py
3 |
4 | This file defines the class `PubSub`, which contains common functionality for
5 | both publisher and subscriber clients.
6 | """
7 |
8 | import io
9 | import threading
10 | import xml.etree.ElementTree as et
11 | from datetime import datetime
12 |
13 | import avro.io
14 | import avro.schema
15 | import certifi
16 | import grpc
17 | import requests
18 |
19 | import pubsub_api_pb2 as pb2
20 | import pubsub_api_pb2_grpc as pb2_grpc
21 | from urllib.parse import urlparse
22 | from utils.ClientUtil import load_properties
23 |
24 | properties = load_properties("../resources/application.properties")
25 |
26 | with open(certifi.where(), 'rb') as f:
27 | secure_channel_credentials = grpc.ssl_channel_credentials(f.read())
28 |
29 |
30 | def get_argument(key, argument_dict):
31 | if key in argument_dict.keys():
32 | return argument_dict[key]
33 | else:
34 | return properties.get(key)
35 |
36 |
37 | class PubSub(object):
38 | """
39 | Class with helpers to use the Salesforce Pub/Sub API.
40 | """
41 |
42 | json_schema_dict = {}
43 |
44 | def __init__(self, argument_dict):
45 | self.url = get_argument('url', argument_dict)
46 | self.username = get_argument('username', argument_dict)
47 | self.password = get_argument('password', argument_dict)
48 | self.metadata = None
49 | grpc_host = get_argument('grpcHost', argument_dict)
50 | grpc_port = get_argument('grpcPort', argument_dict)
51 | pubsub_url = grpc_host + ":" + grpc_port
52 | channel = grpc.secure_channel(pubsub_url, secure_channel_credentials)
53 | self.stub = pb2_grpc.PubSubStub(channel)
54 | self.session_id = None
55 | self.pb2 = pb2
56 | self.topic_name = get_argument('topic', argument_dict)
57 | # If the API version is not provided as an argument, use a default value
58 | if get_argument('apiVersion', argument_dict) == None:
59 | self.apiVersion = '57.0'
60 | else:
61 | # Otherwise, get the version from the argument
62 | self.apiVersion = get_argument('apiVersion', argument_dict)
63 | """
64 | Semaphore used for subscriptions. This keeps the subscription stream open
65 | to receive events and to notify when to send the next FetchRequest.
66 | See Python Quick Start for more information.
67 | https://developer.salesforce.com/docs/platform/pub-sub-api/guide/qs-python-quick-start.html
68 | There is probably a better way to do this. This is only sample code. Please
69 | use your own discretion when writing your production Pub/Sub API client.
70 | Make sure to use only one semaphore per subscribe call if you are planning
71 | to share the same instance of PubSub.
72 | """
73 | self.semaphore = threading.Semaphore(1)
74 |
75 | def auth(self):
76 | """
77 | Sends a login request to the Salesforce SOAP API to retrieve a session
78 | token. The session token is bundled with other identifying information
79 | to create a tuple of metadata headers, which are needed for every RPC
80 | call.
81 | """
82 | url_suffix = '/services/Soap/u/' + self.apiVersion + '/'
83 | headers = {'content-type': 'text/xml', 'SOAPAction': 'Login'}
84 | xml = "" + \
87 | ""
90 | res = requests.post(self.url + url_suffix, data=xml, headers=headers)
91 | res_xml = et.fromstring(res.content.decode('utf-8'))[0][0][0]
92 |
93 | try:
94 | url_parts = urlparse(res_xml[3].text)
95 | self.url = "{}://{}".format(url_parts.scheme, url_parts.netloc)
96 | self.session_id = res_xml[4].text
97 | except IndexError:
98 | print("An exception occurred. Check the response XML below:",
99 | res.__dict__)
100 |
101 | # Get org ID from UserInfo
102 | uinfo = res_xml[6]
103 | # Org ID
104 | self.tenant_id = uinfo[8].text;
105 |
106 | # Set metadata headers
107 | self.metadata = (('accesstoken', self.session_id),
108 | ('instanceurl', self.url),
109 | ('tenantid', self.tenant_id))
110 |
111 | def release_subscription_semaphore(self):
112 | """
113 | Release semaphore so FetchRequest can be sent
114 | """
115 | self.semaphore.release()
116 |
117 | def make_fetch_request(self, topic, replay_type, replay_id, num_requested):
118 | """
119 | Creates a FetchRequest per the proto file.
120 | """
121 | replay_preset = None
122 | match replay_type:
123 | case "LATEST":
124 | replay_preset = pb2.ReplayPreset.LATEST
125 | case "EARLIEST":
126 | replay_preset = pb2.ReplayPreset.EARLIEST
127 | case "CUSTOM":
128 | replay_preset = pb2.ReplayPreset.CUSTOM
129 | case _:
130 | raise ValueError('Invalid Replay Type ' + replay_type)
131 | return pb2.FetchRequest(
132 | topic_name=topic,
133 | replay_preset=replay_preset,
134 | replay_id=bytes.fromhex(replay_id),
135 | num_requested=num_requested)
136 |
137 | def fetch_req_stream(self, topic, replay_type, replay_id, num_requested):
138 | """
139 | Returns a FetchRequest stream for the Subscribe RPC.
140 | """
141 | while True:
142 | # Only send FetchRequest when needed. Semaphore release indicates need for new FetchRequest
143 | self.semaphore.acquire()
144 | print("Sending Fetch Request")
145 | yield self.make_fetch_request(topic, replay_type, replay_id, num_requested)
146 |
147 | def encode(self, schema, payload):
148 | """
149 | Uses Avro and the event schema to encode a payload. The `encode()` and
150 | `decode()` methods are helper functions to serialize and deserialize
151 | the payloads of events that clients will publish and receive using
152 | Avro. If you develop an implementation with a language other than
153 | Python, you will need to find an Avro library in that language that
154 | helps you encode and decode with Avro. When publishing an event, the
155 | plaintext payload needs to be Avro-encoded with the event schema for
156 | the API to accept it. When receiving an event, the Avro-encoded payload
157 | needs to be Avro-decoded with the event schema for you to read it in
158 | plaintext.
159 | """
160 | schema = avro.schema.parse(schema)
161 | buf = io.BytesIO()
162 | encoder = avro.io.BinaryEncoder(buf)
163 | writer = avro.io.DatumWriter(schema)
164 | writer.write(payload, encoder)
165 | return buf.getvalue()
166 |
167 | def decode(self, schema, payload):
168 | """
169 | Uses Avro and the event schema to decode a serialized payload. The
170 | `encode()` and `decode()` methods are helper functions to serialize and
171 | deserialize the payloads of events that clients will publish and
172 | receive using Avro. If you develop an implementation with a language
173 | other than Python, you will need to find an Avro library in that
174 | language that helps you encode and decode with Avro. When publishing an
175 | event, the plaintext payload needs to be Avro-encoded with the event
176 | schema for the API to accept it. When receiving an event, the
177 | Avro-encoded payload needs to be Avro-decoded with the event schema for
178 | you to read it in plaintext.
179 | """
180 | schema = avro.schema.parse(schema)
181 | buf = io.BytesIO(payload)
182 | decoder = avro.io.BinaryDecoder(buf)
183 | reader = avro.io.DatumReader(schema)
184 | ret = reader.read(decoder)
185 | return ret
186 |
187 | def get_topic(self, topic_name):
188 | return self.stub.GetTopic(pb2.TopicRequest(topic_name=topic_name),
189 | metadata=self.metadata)
190 |
191 | def get_schema_json(self, schema_id):
192 | """
193 | Uses GetSchema RPC to retrieve schema given a schema ID.
194 | """
195 | # If the schema is not found in the dictionary, get the schema and store it in the dictionary
196 | if schema_id not in self.json_schema_dict or self.json_schema_dict[schema_id]==None:
197 | res = self.stub.GetSchema(pb2.SchemaRequest(schema_id=schema_id), metadata=self.metadata)
198 | self.json_schema_dict[schema_id] = res.schema_json
199 |
200 | return self.json_schema_dict[schema_id]
201 |
202 | def generate_producer_events(self, schema, schema_id):
203 | """
204 | Encodes the data to be sent in the event and creates a ProducerEvent per
205 | the proto file. Change the below payload to match the schema used.
206 | """
207 | payload = {
208 | "CreatedDate": int(datetime.now().timestamp()),
209 | "CreatedById": '005R0000000cw06IAA', # Your user ID
210 | "textt__c": 'Hello World'
211 | }
212 | req = {
213 | "schema_id": schema_id,
214 | "payload": self.encode(schema, payload)
215 | }
216 | return [req]
217 |
218 | def subscribe(self, topic, replay_type, replay_id, num_requested, callback):
219 | """
220 | Calls the Subscribe RPC defined in the proto file and accepts a
221 | client-defined callback to handle any events that are returned by the
222 | API. It uses a semaphore to prevent the Python client from closing the
223 | connection prematurely (this is due to the way Python's GRPC library is
224 | designed and may not be necessary for other languages--Java, for
225 | example, does not need this).
226 | """
227 | sub_stream = self.stub.Subscribe(self.fetch_req_stream(topic, replay_type, replay_id, num_requested), metadata=self.metadata)
228 | print("> Subscribed to", topic)
229 | for event in sub_stream:
230 | callback(event, self)
231 |
232 | def publish(self, topic_name, schema, schema_id):
233 | """
234 | Publishes events to the specified Platform Event topic.
235 | """
236 |
237 | return self.stub.Publish(self.pb2.PublishRequest(
238 | topic_name=topic_name,
239 | events=self.generate_producer_events(schema,
240 | schema_id)),
241 | metadata=self.metadata)
--------------------------------------------------------------------------------
/python/InventoryAppExample/README.md:
--------------------------------------------------------------------------------
1 | # Pub/Sub API Example - Inventory App
2 |
3 | This example of the Pub/Sub API is meant to be a conceptual example only—the
4 | code is not a template, not meant for copying and pasting, and not intended to
5 | serve as anything other than a read-only learning resource. We encourage you to
6 | read through the code and this README in order to understand the logic
7 | underpinning the example, so that you can take the learnings and apply them to
8 | your own implementations. Also note that the way this example is structured is
9 | but one way to interact with the Pub/Sub API using Python. You are free to
10 | mirror the structure in your own code, but it is far from the only way to
11 | engage with the API.
12 |
13 | The example imagines a scenario in which salespeople closing opportunities in
14 | Salesforce need an "Estimated Delivery Date" field filled in by an integration
15 | between Salesforce and an external inventory app. When an opportunity is closed
16 | in Salesforce, a Change Data Capture event gets published by Salesforce. This
17 | event gets consumed by an inventory app (`InventoryApp.py`) hosted in an
18 | external system like AWS, which sets off the inventory process for the order,
19 | like packaging, shipping, etc. Once the inventory app has calculated the
20 | estimated delivery date for the order, it sends that information back to
21 | Salesforce in the payload of a `NewOrderConfirmation` event. On the Salesforce
22 | side, a subscriber client (`SalesforceListener.py`) receives the
23 | `NewOrderConfirmation` event and uses the date contained in the payload to
24 | update the very opportunity that just closed with its estimated delivery date.
25 | In this scenario, this enables the salesperson who closed the deal to report
26 | the estimated delivery date to their customer right away—the integration acts
27 | so quickly that the salesperson can see the estimated delivery date almost
28 | instantaneously after they close the opportunity.
29 |
30 | A video demonstrating this app in action can be found on the
31 | [TrailheaDX](https://www.salesforce.com/trailheadx) website. After
32 | registering/logging in, go to [Product & Partner
33 | Demos](https://www.salesforce.com/trailheadx) and click `Integrations &
34 | Analytics` > `Platform APIs` to watch it.
35 |
36 | The proto file for the API can be found [here](https://github.com/developerforce/pub-sub-api/blob/main/pubsub_api.proto).
37 |
38 | This example uses Python features that require Python version 3.10 or later, such as the `match` statement.
39 |
40 | To build a working client example in Python please follow [the Python Quick Start Guide.](https://developer.salesforce.com/docs/platform/pub-sub-api/guide/qs-python-quick-start.html)
41 |
--------------------------------------------------------------------------------
/python/InventoryAppExample/SalesforceListener.py:
--------------------------------------------------------------------------------
1 | """
2 | SalesforceListener.py
3 |
4 | This is a subscriber client that listens for `/event/NewOrderConfirmation__e`
5 | events published by the inventory app (`InventoryApp.py`). The `if __debug__`
6 | conditionals are to slow down the speed of the app for demoing purposes.
7 | """
8 |
9 | import os, sys, avro
10 |
11 | dir_path = os.path.dirname(os.path.realpath(__file__))
12 | parent_dir_path = os.path.abspath(os.path.join(dir_path, os.pardir))
13 | sys.path.insert(0, parent_dir_path)
14 |
15 | from util.ChangeEventHeaderUtility import process_bitmap
16 | from datetime import datetime
17 | import json
18 | import logging
19 | import requests
20 | import time
21 |
22 | from PubSub import PubSub
23 | from utils.ClientUtil import command_line_input
24 |
25 |
26 | def process_confirmation(event, pubsub):
27 | """
28 | This is a callback that gets passed to the `PubSub.subscribe()` method. It
29 | decodes the payload of the received event and extracts the opportunity ID
30 | and estimated delivery date. Using those two pieces of information, it
31 | updates the relevant opportunity with its estimated delivery date using the
32 | REST API. When no events are received within a certain time period, the
33 | API's subscribe method sends keepalive messages and the latest replay ID
34 | through this callback.
35 | """
36 |
37 | if event.events:
38 | print("Number of events received in FetchResponse: ", len(event.events))
39 | # If all requested events are delivered, release the semaphore
40 | # so that a new FetchRequest gets sent by `PubSub.fetch_req_stream()`.
41 | if event.pending_num_requested == 0:
42 | pubsub.release_subscription_semaphore()
43 |
44 | for evt in event.events:
45 | # Get the event payload and schema, then decode the payload
46 | payload_bytes = evt.event.payload
47 | json_schema = pubsub.get_schema_json(evt.event.schema_id)
48 | decoded_event = pubsub.decode(json_schema, payload_bytes)
49 | # print(decoded_event)
50 | # A change event contains the ChangeEventHeader field. Check if received event is a change event.
51 | if 'ChangeEventHeader' in decoded_event:
52 | # Decode the bitmap fields contained within the ChangeEventHeader. For example, decode the 'changedFields' field.
53 | # An example to process bitmap in 'changedFields'
54 | changed_fields = decoded_event['ChangeEventHeader']['changedFields']
55 | print("Change Type: " + decoded_event['ChangeEventHeader']['changeType'])
56 | print("=========== Changed Fields =============")
57 | print(process_bitmap(avro.schema.parse(json_schema), changed_fields))
58 | print("=========================================")
59 | print("> Received order confirmation! Updating estimated delivery date...")
60 | if __debug__:
61 | time.sleep(2)
62 | # Update the Desription field of the opportunity with the estimated delivery date with a REST request
63 | day = datetime.fromtimestamp(decoded_event['EstimatedDeliveryDate__c']).strftime('%Y-%m-%d')
64 | res = requests.patch(pubsub.url + "/services/data/v" + pubsub.apiVersion + "/sobjects/Opportunity/"
65 | + decoded_event['OpptyRecordId__c'], json.dumps({"Description": "Estimated Delivery Date: " + day}),
66 | headers={"Authorization": "Bearer " + pubsub.session_id,
67 | "Content-Type": "application/json",
68 | "Sforce-Call-Options": "client=SalesforceListener"})
69 | print(" Done!", res)
70 | else:
71 | print("[", time.strftime('%b %d, %Y %l:%M%p %Z'), "] The subscription is active.")
72 |
73 | # The replay_id is used to resubscribe after this position in the stream if the client disconnects.
74 | # Implement storage of replay for resubscribe!!!
75 | event.latest_replay_id
76 |
77 | def run(argument_dict):
78 | sfdc_updater = PubSub(argument_dict)
79 | sfdc_updater.auth()
80 |
81 | # Subscribe to /event/NewOrderConfirmation__e events
82 | sfdc_updater.subscribe('/event/NewOrderConfirmation__e', "LATEST", "", 1, process_confirmation)
83 |
84 |
85 | if __name__ == '__main__':
86 | argument_dict = command_line_input(sys.argv[1:])
87 | logging.basicConfig()
88 | run(argument_dict)
89 |
--------------------------------------------------------------------------------
/python/util/ChangeEventHeaderUtility.py:
--------------------------------------------------------------------------------
1 | """
2 | ChangeEventHeaderUtility.py
3 |
4 | This class provides the utility method to decode the bitmap fields (eg: changedFields) and return the avro schema field values represented by the bitmap.
5 | To understand the process of bitmap conversion, see "Event Deserialization Considerations" in the Pub/Sub API documentation at https://developer.salesforce.com/docs/platform/pub-sub-api/guide/event-deserialization-considerations.html.
6 | """
7 |
8 | from avro.schema import Schema
9 | from bitstring import BitArray
10 |
11 |
12 | def process_bitmap(avro_schema: Schema, bitmap_fields: list):
13 | fields = []
14 | if len(bitmap_fields) != 0:
15 | # replace top field level bitmap with list of fields
16 | if bitmap_fields[0].startswith("0x"):
17 | bitmap = bitmap_fields[0]
18 | fields = fields + get_fieldnames_from_bitstring(bitmap, avro_schema)
19 | bitmap_fields.remove(bitmap)
20 | # replace parentPos-nested Nulled BitMap with list of fields too
21 | if len(bitmap_fields) != 0 and "-" in str(bitmap_fields[-1]):
22 | for bitmap_field in bitmap_fields:
23 | if bitmap_field is not None and "-" in str(bitmap_field):
24 | bitmap_strings = bitmap_field.split("-")
25 | # interpret the parent field name from mapping of parentFieldPos -> childFieldbitMap
26 | parent_field = avro_schema.fields[int(bitmap_strings[0])]
27 | child_schema = get_value_schema(parent_field.type)
28 | # make sure we're really dealing with compound field
29 | if child_schema.type is not None and child_schema.type == 'record':
30 | nested_size = len(child_schema.fields)
31 | parent_field_name = parent_field.name
32 | # interpret the child field names from mapping of parentFieldPos -> childFieldbitMap
33 | full_field_names = get_fieldnames_from_bitstring(bitmap_strings[1], child_schema)
34 | full_field_names = append_parent_name(parent_field_name, full_field_names)
35 | if len(full_field_names) > 0:
36 | # when all nested fields under a compound got nulled out at once by customer, we recognize the top level field instead of trying to list every single nested field
37 | fields = fields + full_field_names
38 | return fields
39 |
40 |
41 | def convert_hexbinary_to_bitset(bitmap):
42 | bit_array = BitArray(hex=bitmap[2:])
43 | binary_string = bit_array.bin
44 | return binary_string[::-1]
45 |
46 |
47 | def append_parent_name(parent_field_name, full_field_names):
48 | for index in range(len(full_field_names)):
49 | full_field_names[index] = parent_field_name + "." + full_field_names[index]
50 | return full_field_names
51 |
52 |
53 | def get_fieldnames_from_bitstring(bitmap, avro_schema: Schema):
54 | bitmap_field_name = []
55 | fields_list = list(avro_schema.fields)
56 | binary_string = convert_hexbinary_to_bitset(bitmap)
57 | indexes = find('1', binary_string)
58 | for index in indexes:
59 | bitmap_field_name.append(fields_list[index].name)
60 | return bitmap_field_name
61 |
62 |
63 | # Get the value type of an "optional" schema, which is a union of [null, valueSchema]
64 | def get_value_schema(parent_field):
65 | if parent_field.type == 'union':
66 | schemas = parent_field.schemas
67 | if len(schemas) == 2 and schemas[0].type == 'null':
68 | return schemas[1]
69 | if len(schemas) == 2 and schemas[0].type == 'string':
70 | return schemas[1]
71 | if len(schemas) == 3 and schemas[0].type == 'null' and schemas[1].type == 'string':
72 | return schemas[2]
73 | return parent_field
74 |
75 |
76 | # Find the positions of 1 in the bit string
77 | def find(to_find, binary_string):
78 | return [i for i, x in enumerate(binary_string) if x == to_find]
--------------------------------------------------------------------------------
/python/util/README.md:
--------------------------------------------------------------------------------
1 | # Pub/Sub API Examples - Utility Code
2 |
3 |
4 | ## ChangeEventHeaderUtility.py
5 | Because the Pub/Sub API is a binary API, delivered events are formatted as raw
6 | Avro binary and sometimes contain fields that are not plaintext-readable. This
7 | manifests in Change Data Capture events, causing them to look different from
8 | how they are delivered when subscribing via Streaming API.
9 |
10 | For example, a Change Data Capture event received via Pub/Sub API might look
11 | like this:
12 |
13 | ```
14 | {'ChangeEventHeader':
15 | {
16 | ...
17 | 'nulledFields': [],
18 | 'diffFields': [],
19 | 'changedFields': ['0x650004E0']
20 | },
21 | ...
22 | }
23 | ```
24 |
25 | In this example, `changedFields` is encoded as a bitmap string. This method is
26 | more space efficient than using a list of field names. A bit set to 1 in the
27 | `changedFields` bitmap value indicates that the field at the corresponding
28 | position in the Avro schema was changed. More information about bitmap fields
29 | can be found in [Event Deserialization Considerations](https://developer.salesforce.com/docs/platform/pub-sub-api/guide/event-deserialization-considerations.html) in the Pub/Sub API documentation.
30 |
31 | We have provided this example to demonstrate how bitmap values can be decoded
32 | so that they are human-readable and you can process the event by using the
33 | values in the bitmap fields.
34 |
--------------------------------------------------------------------------------