| property | 66 |default | 67 |description | 68 |
|---|---|---|
| metadata.broker.list | 72 |73 | | (required) A comma separated list of seed brokers to connect to in order 74 | to get metadata about the cluster. | 75 |
| queue.buffering.max.ms | 79 |5000 | 80 |Maximum time to buffer data. For example a setting of 100 will try to 81 | batch together 100ms of messages to send at once. This will improve 82 | throughput but adds message delivery latency due to the buffering. | 83 |
| request.required.acks | 87 |1 | 88 |This value controls when a produce request is considered completed.
89 | Specifically, how many other brokers must have committed the data to their
90 | log and acknowledged this to the leader? Typical values are
91 |
|
107 |
| request.timeout.ms | 111 |10000 | 112 |The amount of time the broker will wait trying to meet the 113 | request.required.acks requirement before sending back an error to the client. 114 | | 115 |
| message.send.max.retries | 119 |3 | 120 |This property will cause the producer to automatically retry a failed 121 | send request. This property specifies the number of retries when such 122 | failures occur. Note that setting a non-zero value here can lead to 123 | duplicates in the case of network errors that cause a message to be sent but 124 | the acknowledgement to be lost. | 125 |
| retry.backoff.ms | 129 |100 | 130 |Before each retry, the producer refreshes the metadata of relevant topics 131 | to see if a new leader has been elected. Since leader election takes a bit of 132 | time, this property specifies the amount of time that the producer waits 133 | before refreshing the metadata. | 134 |
| topic.metadata.refresh.interval.ms | 138 |600 * 1000 | 139 |The producer generally refreshes the topic metadata from brokers when 140 | there is a failure (partition missing, leader not available...). It will also 141 | poll regularly (default: every 10min so 600000ms). If you set this to a 142 | negative value, metadata will only get refreshed on failure. If you set this 143 | to zero, the metadata will get refreshed after each message sent (not 144 | recommended). Important note: the refresh happen only AFTER the message is 145 | sent, so if the producer never sends a message the metadata is never 146 | refreshed | 147 |
| message.buffer.size | 151 |1024*1024 | 152 |The size of each buffer that is used to store raw messages before they 153 | are sent. Since a full buffer is sent at once, don't make this too big. | 154 |
| num.buffers | 158 |2 | 159 |The number of buffers to use. At any given time, there is up to one 160 | buffer being filled with new data, up to one buffer having its data sent to 161 | the broker, and any number of buffers waiting to be filled and/or sent. 162 | 163 | Essentially, the limit of the amount of data that can be queued at at any 164 | given time is message.buffer.size * num.buffers. Although, in reality, you 165 | won't get buffers to 100% full each time. | 166 |
| send.buffer.size | 170 |message.buffer.size + 200 | 171 |Size of the byte buffer used to store the final (with headers and 172 | compression applied) data to be sent to the broker. | 173 |
| compression.codec | 177 |none | 178 |This parameter allows you to specify the compression codec for all data 179 | generated by this producer. Valid values are "none", "gzip" and "snappy". | 180 |
| gzip.compression.level | 184 |java.util.zip.Deflater.DEFAULT_COMPRESSION | 185 |If compression.codec is set to gzip, then this allows configuration of
186 | the compression level.
187 |
|
193 |
| queue.enqueue.timeout.ms | 197 |-1 | 198 |The amount of time to block before dropping messages when all buffers are 199 | full. If set to 0 events will be enqueued immediately or dropped if the queue 200 | is full (the producer send call will never block). If set to -1 the producer 201 | will block indefinitely and never willingly drop a send. | 202 |
| property | 214 |default | 215 |description | 216 |
|---|---|---|
| metadata.broker.list | 220 |221 | | (required) A list of seed brokers to connect to in order to get 222 | information about the Kafka broker cluster. | 223 |
| fetch.message.max.bytes | 227 |1024 * 1024 | 228 |The number of byes of messages to attempt to fetch for each 229 | topic-partition in each fetch request. These bytes will be read into memory 230 | for each partition, so this helps control the memory used by the consumer. 231 | The fetch request size must be at least as large as the maximum message size 232 | the server allows or else it is possible for the producer to send messages 233 | larger than the consumer can fetch. | 234 |
| fetch.wait.max.ms | 238 |100 | 239 |The maximum amount of time the server will block before answering the 240 | fetch request if there isn't sufficient data to immediately satisfy 241 | fetch.min.bytes | 242 |
| fetch.min.bytes | 246 |1 | 247 |The minimum amount of data the server should return for a fetch request. 248 | If insufficient data is available the request will wait for that much data to 249 | accumulate before answering the request. | 250 |
| socket.receive.buffer.bytes | 254 |64 * 1024 | 255 |The socket receive buffer for network requests | 256 |
| auto.offset.reset | 260 |largest | 261 |What to do when there is no initial offset in ZooKeeper or if an offset
262 | is out of range:
263 |
|
269 |
-1.
38 | *
39 | * @param src
40 | * source byte array.
41 | * @param srcPos
42 | * start position of data in the source byte array.
43 | * @param length
44 | * length of data in the source byte array.
45 | * @param dest
46 | * destination byte array
47 | * @param destPos
48 | * position in destination byte array to write to
49 | * @return the number of bytes written to the destination array, or
50 | * -1 if there was not enough room to write the
51 | * compressed data.
52 | * @throws IOException
53 | */
54 | public int compress(byte[] src, int srcPos, int length, byte[] dest,
55 | int destPos) throws IOException;
56 |
57 | }
58 |
--------------------------------------------------------------------------------
/src/main/java/com/blackberry/bdp/krackle/compression/Decompressor.java:
--------------------------------------------------------------------------------
1 | /**
2 | * Copyright 2014 BlackBerry, Limited.
3 | *
4 | * Licensed under the Apache License, Version 2.0 (the "License");
5 | * you may not use this file except in compliance with the License.
6 | * You may obtain a copy of the License at
7 | *
8 | * http://www.apache.org/licenses/LICENSE-2.0
9 | *
10 | * Unless required by applicable law or agreed to in writing, software
11 | * distributed under the License is distributed on an "AS IS" BASIS,
12 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13 | * See the License for the specific language governing permissions and
14 | * limitations under the License.
15 | */
16 | package com.blackberry.bdp.krackle.compression;
17 |
18 | import java.io.IOException;
19 |
20 | /**
21 | * Interface for decompressors used to compress data for sending to the broker.
22 | */
23 | public interface Decompressor {
24 |
25 | /**
26 | * Return the attribute value associated with this compression method.
27 | *
28 | * @return the attribute value associated with this compression method.
29 | */
30 | public byte getAttribute();
31 |
32 | /**
33 | * Decompresses the data from the source array into the destination array.
34 | *
35 | * If the destination array is (potentially) not big enough to hold the
36 | * decompressed data, then the decompress method will not decompress anything
37 | * and return -1.
38 | *
39 | * @param src
40 | * source byte array.
41 | * @param srcPos
42 | * position in source byte array to start from.
43 | * @param length
44 | * length of data to decompress.
45 | * @param dest
46 | * destination byte array.
47 | * @param destPos
48 | * position in destination byte array to write to.
49 | * @param maxLength
50 | * max length of decompressed data.
51 | * @return the number of bytes written to the destination, or -1
52 | * if maxLength was not big enough to hold the data.
53 | * @throws IOException
54 | */
55 | public int decompress(byte[] src, int srcPos, int length, byte[] dest,
56 | int destPos, int maxLength) throws IOException;
57 |
58 | }
59 |
--------------------------------------------------------------------------------
/src/main/java/com/blackberry/bdp/krackle/compression/GzipCompressor.java:
--------------------------------------------------------------------------------
1 | /**
2 | * Copyright 2014 BlackBerry, Limited.
3 | *
4 | * Licensed under the Apache License, Version 2.0 (the "License");
5 | * you may not use this file except in compliance with the License.
6 | * You may obtain a copy of the License at
7 | *
8 | * http://www.apache.org/licenses/LICENSE-2.0
9 | *
10 | * Unless required by applicable law or agreed to in writing, software
11 | * distributed under the License is distributed on an "AS IS" BASIS,
12 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13 | * See the License for the specific language governing permissions and
14 | * limitations under the License.
15 | */
16 | package com.blackberry.bdp.krackle.compression;
17 |
18 | import java.io.IOException;
19 | import java.nio.ByteBuffer;
20 | import java.nio.ByteOrder;
21 | import java.util.zip.CRC32;
22 | import java.util.zip.Deflater;
23 |
24 | import com.blackberry.bdp.krackle.Constants;
25 |
26 | /**
27 | * Compressor implementation that used the GZIP algorithm.
28 | */
29 | public class GzipCompressor implements Compressor {
30 |
31 | private static final byte[] HEADER_BYTES = new byte[] //
32 | {(byte) 0x1f, (byte) 0x8b, // Magic number
33 | 8, // Deflate
34 | 0, // All flags zero
35 | 0, 0, 0, 0, // Set MTIME to zero, for ease of use
36 | 0, // No extra flags
37 | 3 // UNIX OS
38 | };
39 | private final Deflater deflater;
40 | private final CRC32 crc;
41 | private final ByteBuffer bb;
42 | private int compressedSize;
43 | private int maxOutputSize;
44 | private int deflaterOutputSize;
45 | private byte[] testBytes = new byte[1];
46 |
47 | /**
48 | * New instance with default compression level.
49 | */
50 | public GzipCompressor() {
51 | this(Deflater.DEFAULT_COMPRESSION);
52 | }
53 |
54 | /**
55 | * New instance with the given compression level
56 | *
57 | * @param compressionLevel
58 | * requested compression level. Valid values are -1
59 | * (default compression), 0 (no compression),
60 | * 1-9.
61 | */
62 | public GzipCompressor(int compressionLevel) {
63 | deflater = new Deflater(compressionLevel, true);
64 | crc = new CRC32();
65 | bb = ByteBuffer.allocate(8);
66 | bb.order(ByteOrder.LITTLE_ENDIAN);
67 | }
68 |
69 | @Override
70 | public byte getAttribute() {
71 | return Constants.GZIP;
72 | }
73 |
74 | @Override
75 | public int compress(byte[] src, int srcPos, int length, byte[] dest,
76 | int destPos) throws IOException {
77 | System.arraycopy(HEADER_BYTES, 0, dest, destPos, 10);
78 | compressedSize = 10;
79 |
80 | deflater.reset();
81 | deflater.setInput(src, srcPos, length);
82 | deflater.finish();
83 |
84 | // The output can't exceed the bytes we have to work with, less 10 bytes for
85 | // headers and 8 bytes for footers.
86 | maxOutputSize = dest.length - destPos - 10 - 8;
87 | deflaterOutputSize = deflater.deflate(dest, destPos + compressedSize,
88 | maxOutputSize);
89 | if (deflaterOutputSize == maxOutputSize) {
90 | // We just filled the output buffer! Either we have more to decompress, or
91 | // we don't. If we do, then that's an error. If we don't then that's fine.
92 | // So let's check.
93 | if (deflater.deflate(testBytes, 0, 1) == 1) {
94 | // We couldn't fit everything in the output buffer.
95 | return -1;
96 | }
97 | }
98 | compressedSize += deflaterOutputSize;
99 |
100 | crc.reset();
101 | crc.update(src, srcPos, length);
102 | bb.clear();
103 | bb.putInt((int) crc.getValue());
104 | bb.putInt(length);
105 | bb.rewind();
106 | bb.get(dest, destPos + compressedSize, 8);
107 |
108 | compressedSize += 8;
109 |
110 | return compressedSize;
111 | }
112 |
113 | }
114 |
--------------------------------------------------------------------------------
/src/main/java/com/blackberry/bdp/krackle/compression/GzipDecompressor.java:
--------------------------------------------------------------------------------
1 | /**
2 | * Copyright 2014 BlackBerry, Limited.
3 | *
4 | * Licensed under the Apache License, Version 2.0 (the "License");
5 | * you may not use this file except in compliance with the License.
6 | * You may obtain a copy of the License at
7 | *
8 | * http://www.apache.org/licenses/LICENSE-2.0
9 | *
10 | * Unless required by applicable law or agreed to in writing, software
11 | * distributed under the License is distributed on an "AS IS" BASIS,
12 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13 | * See the License for the specific language governing permissions and
14 | * limitations under the License.
15 | */
16 | package com.blackberry.bdp.krackle.compression;
17 |
18 | import java.io.IOException;
19 | import java.util.zip.DataFormatException;
20 | import java.util.zip.Inflater;
21 |
22 | import com.blackberry.bdp.krackle.Constants;
23 |
24 | /**
25 | * Decompressor implementation that used the GZIP algorithm.
26 | */
27 | public class GzipDecompressor implements Decompressor {
28 |
29 | // various fields, as named in rfc 1952
30 |
31 | private static final byte ID1 = (byte) 0x1f;
32 | private static final byte ID2 = (byte) 0x8b;
33 | private static final byte CM = 8; // Compression Method = Deflate
34 |
35 | // masks for various flags
36 | // FTEXT flag is ignored, but we keep it here for completeness
37 | @SuppressWarnings("unused")
38 | private static final byte FLG_FTEXT = 0x01;
39 | private static final byte FLG_FHCRC = 0x02;
40 | private static final byte FLG_FEXTRA = 0x04;
41 | private static final byte FLG_FNAME = 0x08;
42 | private static final byte FLG_COMMENT = 0x10;
43 |
44 | private byte flags;
45 | private boolean fhcrc;
46 | private boolean fextra;
47 | private boolean fname;
48 | private boolean comment;
49 |
50 | private Inflater inflater;
51 | private int pos;
52 | private short extraLength;
53 | private int decompressedLength;
54 |
55 | /**
56 | * Constructor.
57 | */
58 | public GzipDecompressor() {
59 | inflater = new Inflater(true);
60 | }
61 |
62 | @Override
63 | public byte getAttribute() {
64 | return Constants.GZIP;
65 | }
66 |
67 | @Override
68 | public int decompress(byte[] src, int srcPos, int length, byte[] dest,
69 | int destPos, int maxLength) throws IOException {
70 | pos = srcPos;
71 |
72 | // Check the magic number and compression method
73 | if (src[pos] != ID1 || src[pos + 1] != ID2) {
74 | throw new IOException("Wrong gzip magic number.");
75 | }
76 | if (src[pos + 2] != CM) {
77 | throw new IOException("Unrecognized compession method.");
78 | }
79 | pos += 3;
80 |
81 | // read flags
82 | flags = src[pos];
83 | // ftext = (FLG_FTEXT == (FLG_FTEXT & flags));
84 | fhcrc = (FLG_FHCRC == (FLG_FHCRC & flags));
85 | fextra = (FLG_FEXTRA == (FLG_FEXTRA & flags));
86 | fname = (FLG_FNAME == (FLG_FNAME & flags));
87 | comment = (FLG_COMMENT == (FLG_COMMENT & flags));
88 | pos++;
89 |
90 | // ignore the timestamp. 4 bytes
91 | // ignore extra flags. 1 byte
92 | // ignore OS. 1 byte
93 | pos += 6;
94 |
95 | if (fextra) {
96 | // skip it
97 | extraLength = readShort(src, pos);
98 | pos += 2 + extraLength;
99 | }
100 |
101 | if (fname) {
102 | // skip it
103 | while (src[pos] != 0x00) {
104 | pos++;
105 | }
106 | pos++;
107 | }
108 |
109 | if (comment) {
110 | // skip it
111 | while (src[pos] != 0x00) {
112 | pos++;
113 | }
114 | pos++;
115 | }
116 |
117 | if (fhcrc) {
118 | // skip it
119 | pos += 2;
120 | }
121 |
122 | inflater.reset();
123 | inflater.setInput(src, pos, length - (pos - srcPos) - 8);
124 | try {
125 | decompressedLength = inflater.inflate(dest, destPos, maxLength);
126 |
127 | // Sometimes we get truncated input. So we may not be 'finished'
128 | // inflating, but we still want to return success. So only fail if we have
129 | // the buffer full, and are not finished.
130 | if (inflater.finished() == false && decompressedLength == maxLength) {
131 | // There was not enough room to write the output
132 |
133 | return -1;
134 | }
135 | } catch (DataFormatException e) {
136 | throw new IOException("Error decompressing data.", e);
137 | }
138 |
139 | return decompressedLength;
140 | }
141 |
142 | private short readShort(byte[] src, int pos) {
143 | // little endian!
144 | return (short) ((src[pos] & 0xFF) | (src[pos + 1] & 0xFF) << 8);
145 | }
146 |
147 | }
148 |
--------------------------------------------------------------------------------
/src/main/java/com/blackberry/bdp/krackle/compression/SnappyCompressor.java:
--------------------------------------------------------------------------------
1 | /**
2 | * Copyright 2014 BlackBerry, Limited.
3 | *
4 | * Licensed under the Apache License, Version 2.0 (the "License");
5 | * you may not use this file except in compliance with the License.
6 | * You may obtain a copy of the License at
7 | *
8 | * http://www.apache.org/licenses/LICENSE-2.0
9 | *
10 | * Unless required by applicable law or agreed to in writing, software
11 | * distributed under the License is distributed on an "AS IS" BASIS,
12 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13 | * See the License for the specific language governing permissions and
14 | * limitations under the License.
15 | */
16 | package com.blackberry.bdp.krackle.compression;
17 |
18 | import java.io.IOException;
19 |
20 | import org.xerial.snappy.Snappy;
21 |
22 | import com.blackberry.bdp.krackle.Constants;
23 |
24 | /**
25 | * Compressor implementation that used the Snappy algorithm.
26 | */
27 | public class SnappyCompressor implements Compressor {
28 |
29 | private final static byte[] header = new byte[]{ //
30 | -126, 'S', 'N', 'A', 'P', 'P', 'Y', 0, // Magic number
31 | 0, 0, 0, 1, // version
32 | 0, 0, 0, 1 // min compatable version
33 | };
34 | private final static int headerLength = header.length;
35 |
36 | private int compressedLength;
37 | private int maxCompressedSize;
38 |
39 | @Override
40 | public byte getAttribute() {
41 | return Constants.SNAPPY;
42 | }
43 |
44 | @Override
45 | public int compress(byte[] src, int srcPos, int length, byte[] dest,
46 | int destPos) throws IOException {
47 | System.arraycopy(header, 0, dest, destPos, headerLength);
48 |
49 | // Compressed size cannot be greater than what we have available
50 | maxCompressedSize = dest.length - destPos - headerLength - 4;
51 | if (Snappy.maxCompressedLength(length) > maxCompressedSize) {
52 | return -1;
53 | }
54 |
55 | compressedLength = Snappy.compress(src, srcPos, length, dest, destPos
56 | + headerLength + 4);
57 | writeInt(compressedLength, dest, destPos + headerLength);
58 |
59 | return headerLength + 4 + compressedLength;
60 | }
61 |
62 | private void writeInt(int i, byte[] dest, int pos) {
63 | dest[pos] = (byte) (i >> 24);
64 | dest[pos + 1] = (byte) (i >> 16);
65 | dest[pos + 2] = (byte) (i >> 8);
66 | dest[pos + 3] = (byte) i;
67 | }
68 |
69 | }
70 |
--------------------------------------------------------------------------------
/src/main/java/com/blackberry/bdp/krackle/compression/SnappyDecompressor.java:
--------------------------------------------------------------------------------
1 | /**
2 | * Copyright 2014 BlackBerry, Limited.
3 | *
4 | * Licensed under the Apache License, Version 2.0 (the "License");
5 | * you may not use this file except in compliance with the License.
6 | * You may obtain a copy of the License at
7 | *
8 | * http://www.apache.org/licenses/LICENSE-2.0
9 | *
10 | * Unless required by applicable law or agreed to in writing, software
11 | * distributed under the License is distributed on an "AS IS" BASIS,
12 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13 | * See the License for the specific language governing permissions and
14 | * limitations under the License.
15 | */
16 | package com.blackberry.bdp.krackle.compression;
17 |
18 | import java.io.IOException;
19 |
20 | import org.xerial.snappy.Snappy;
21 |
22 | import com.blackberry.bdp.krackle.Constants;
23 |
24 | /**
25 | * Decompressor implementation that used the Snappy algorithm.
26 | */
27 | public class SnappyDecompressor implements Decompressor {
28 |
29 | private static final byte[] MAGIC_NUMBER = new byte[]{ //
30 | -126, 'S', 'N', 'A', 'P', 'P', 'Y', 0};
31 |
32 | private byte[] src;
33 | private int pos;
34 |
35 | private int blockLength;
36 | private int decompressedLength;
37 |
38 | private int uncompressedBlockLength;
39 |
40 | @Override
41 | public byte getAttribute() {
42 | return Constants.SNAPPY;
43 | }
44 |
45 | @Override
46 | public int decompress(byte[] src, int srcPos, int length, byte[] dest,
47 | int destPos, int maxLength) throws IOException {
48 | this.src = src;
49 | decompressedLength = 0;
50 |
51 | // Check for magic number
52 | if (src[srcPos] == MAGIC_NUMBER[0] //
53 | || src[srcPos + 1] == MAGIC_NUMBER[1] //
54 | || src[srcPos + 2] == MAGIC_NUMBER[2] //
55 | || src[srcPos + 3] == MAGIC_NUMBER[3] //
56 | || src[srcPos + 4] == MAGIC_NUMBER[4] //
57 | || src[srcPos + 5] == MAGIC_NUMBER[5] //
58 | || src[srcPos + 6] == MAGIC_NUMBER[6] //
59 | || src[srcPos + 7] == MAGIC_NUMBER[7]) {
60 |
61 | // advance past the magic number
62 | // assume the version (4 bytes), min compatable version (4 bytes) are fine
63 | pos = srcPos + 8 + 8;
64 |
65 | // TODO: limit the decompressed length
66 | while (pos < srcPos + length) {
67 | blockLength = readInt();
68 |
69 | // Check to see if this will exceed maxLength
70 | uncompressedBlockLength = Snappy.uncompressedLength(src, pos,
71 | blockLength);
72 | if (decompressedLength + uncompressedBlockLength > maxLength) {
73 | return -1;
74 | }
75 |
76 | decompressedLength += Snappy.uncompress(src, pos, blockLength, dest,
77 | destPos + decompressedLength);
78 | pos += blockLength;
79 | }
80 |
81 | return decompressedLength;
82 | } else {
83 | // Assume it's just straight compressed
84 | return Snappy.uncompress(src, pos, blockLength, dest, destPos);
85 | }
86 | }
87 |
88 | private int readInt() {
89 | pos += 4;
90 | return src[pos - 4] & 0xFF << 24 //
91 | | (src[pos - 3] & 0xFF) << 16 //
92 | | (src[pos - 2] & 0xFF) << 8 //
93 | | (src[pos - 1] & 0xFF);
94 | }
95 |
96 | }
97 |
--------------------------------------------------------------------------------
/src/main/java/com/blackberry/bdp/krackle/consumer/BrokerUnavailableException.java:
--------------------------------------------------------------------------------
1 | /**
2 | * Copyright 2014 BlackBerry, Limited.
3 | *
4 | * Licensed under the Apache License, Version 2.0 (the "License");
5 | * you may not use this file except in compliance with the License.
6 | * You may obtain a copy of the License at
7 | *
8 | * http://www.apache.org/licenses/LICENSE-2.0
9 | *
10 | * Unless required by applicable law or agreed to in writing, software
11 | * distributed under the License is distributed on an "AS IS" BASIS,
12 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13 | * See the License for the specific language governing permissions and
14 | * limitations under the License.
15 | */
16 | package com.blackberry.bdp.krackle.consumer;
17 |
18 | /**
19 | *
20 | * @author dariens
21 | */
22 | public class BrokerUnavailableException extends Exception {
23 |
24 | public BrokerUnavailableException(String error, Exception e) {
25 | super(error, e);
26 | }
27 |
28 | }
29 |
--------------------------------------------------------------------------------
/src/main/java/com/blackberry/bdp/krackle/consumer/Consumer.java:
--------------------------------------------------------------------------------
1 | /**
2 | * Copyright 2014 BlackBerry, Limited.
3 | *
4 | * Licensed under the Apache License, Version 2.0 (the "License");
5 | * you may not use this file except in compliance with the License.
6 | * You may obtain a copy of the License at
7 | *
8 | * http://www.apache.org/licenses/LICENSE-2.0
9 | *
10 | * Unless required by applicable law or agreed to in writing, software
11 | * distributed under the License is distributed on an "AS IS" BASIS,
12 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13 | * See the License for the specific language governing permissions and
14 | * limitations under the License.
15 | */
16 | package com.blackberry.bdp.krackle.consumer;
17 |
18 | import com.blackberry.bdp.common.jmx.MetricRegistrySingleton;
19 | import java.io.IOException;
20 | import java.io.InputStream;
21 | import java.io.OutputStream;
22 | import java.net.Socket;
23 | import java.nio.ByteBuffer;
24 | import java.nio.charset.Charset;
25 |
26 | import org.slf4j.Logger;
27 | import org.slf4j.LoggerFactory;
28 |
29 | import com.blackberry.bdp.krackle.Constants;
30 | import com.blackberry.bdp.krackle.KafkaError;
31 | import com.blackberry.bdp.krackle.auth.AuthenticatedSocketSingleton;
32 | import com.blackberry.bdp.krackle.meta.Broker;
33 | import com.blackberry.bdp.krackle.meta.MetaData;
34 | import com.codahale.metrics.Meter;
35 | import com.codahale.metrics.MetricRegistry;
36 | import java.net.SocketTimeoutException;
37 |
38 | /**
39 | * An implementation of the Kafka 0.9 consumer.
40 | *
41 | * This class acts as a consumer of data from a cluster of Kafka brokers.
42 | * Each instance only reads data from a single partition of a single topic.
43 | * If you need to read more than that, then instantiate more instances.
44 | *
45 | * This class was designed to be very light weight. The standard Java
46 | * client creates a lot of objects, and therefore causes a lot of garbage
47 | * collection that leads to a major slowdown in performance. This client
48 | * creates no new objects during steady state running, and so avoids
49 | * all garbage collection overhead.
50 | */
51 | public class Consumer {
52 |
53 | private static final Logger LOG = LoggerFactory.getLogger(Consumer.class);
54 | private static final Charset UTF8 = Charset.forName("UTF8");
55 |
56 | private ConsumerConfiguration conf;
57 |
58 | private String clientId;
59 | private byte[] clientIdBytes;
60 | private short clientIdLength;
61 |
62 | private String topic;
63 | private byte[] topicBytes;
64 | private short topicLength;
65 | private Broker broker;
66 | private int partition;
67 |
68 | private final MessageSetReader messageSetReader = new MessageSetReader();
69 |
70 | private long offset;
71 | private long lastOffset;
72 | private long highWaterMark;
73 |
74 | private byte[] offsetRequestBytes;
75 | private ByteBuffer offsetRequestBuffer;
76 |
77 | private byte[] offsetResponseBytes;
78 | private ByteBuffer offsetResponseBuffer;
79 |
80 | private byte[] requestBytes;
81 | private ByteBuffer requestBuffer;
82 |
83 | private int fetchMessageMaxBytes;
84 | private byte[] responseBytes;
85 | private ByteBuffer responseBuffer;
86 |
87 | private int fetchWaitMaxMs;
88 | private int fetchMinBytes;
89 |
90 | private MetricRegistry metrics;
91 | private Meter mMessageRequests = null;
92 | private Meter mMessageRequestsTotal = null;
93 | private Meter mMessagesReturned = null;
94 | private Meter mMessagesReturnedTotal = null;
95 | private Meter mBytesReturned = null;
96 | private Meter mBytesReturnedTotal = null;
97 | private Meter mMessageRequestsNoData = null;
98 | private Meter mMessageRequestsNoDataTotal = null;
99 | private Meter mBrokerReadAttempts = null;
100 | private Meter mBrokerReadAttemptsTotal = null;
101 | private Meter mBrokerReadSuccess = null;
102 | private Meter mBrokerReadSuccessTotal = null;
103 | private Meter mBrokerReadFailure = null;
104 | private Meter mBrokerReadFailureTotal = null;
105 |
106 | private int bytesReturned = 0;
107 | private Socket brokerSocket = null;
108 | private InputStream brokerIn = null;
109 | private OutputStream brokerOut = null;
110 | private int correlationId = 0;
111 | private int bytesRead;
112 | private int responseLength;
113 | private int responseCorrelationId;
114 | private short errorCode;
115 | private int messageSetSize;
116 |
117 | /*
118 | * Create a new consumer that reads from a given consumer. It attempts to start at offset 0.
119 | *
120 | * @param conf ConsumerConfiguration for this consumer.
121 | * @param clientId clientId to be send with requests to Kafka.
122 | * @param topic topic to read from.
123 | * @param partition id of the partition to read from.
124 | */
125 | public Consumer(ConsumerConfiguration conf, String clientId, String topic, int partition) throws BrokerUnavailableException {
126 | this(conf, clientId, topic, partition, 0L);
127 | }
128 |
129 | /*
130 | * Create a new consumer that reads from a given consumer that attempts to start reading at the given offset.
131 | *
132 | * @param conf ConsumerConfiguration for this consumer.
133 | * @param clientId clientId to be send with requests to Kafka.
134 | * @param topic topic to read from.
135 | * @param partition id of the partition to read from.
136 | * @param offset the offset to start reading from.
137 | */
138 | public Consumer(ConsumerConfiguration conf, String clientId, String topic, int partition, long offset) throws BrokerUnavailableException {
139 | this(conf, clientId, topic, partition, offset, null);
140 | }
141 |
142 | /*
143 | * Create a new consumer that reads from a given consumer that attempts to start reading at the given offset.
144 | *
145 | * Metrics are reported using the given instance of MetricRegistry instead the internal singleton instance.
146 | *
147 | * @param conf ConsumerConfiguration for this consumer.
148 | * @param clientId clientId to be send with requests to Kafka.
149 | * @param topic topic to read from.
150 | * @param partition id of the partition to read from.
151 | * @param offset the offset to start reading from.
152 | * @param metrics the instance of MetricRegistry to use for reporting metrics.
153 | */
154 | public Consumer(ConsumerConfiguration conf, String clientId, String topic, int partition, long offset, MetricRegistry metrics) throws BrokerUnavailableException {
155 | LOG.info("[{}-{}] creating consumer from offset {}", topic, partition, offset);
156 |
157 | this.conf = conf;
158 |
159 | if (metrics == null) {
160 | this.metrics = MetricRegistrySingleton.getInstance().getMetricsRegistry();
161 | MetricRegistrySingleton.getInstance().enableJmx();
162 | MetricRegistrySingleton.getInstance().enableConsole();
163 | } else {
164 | this.metrics = metrics;
165 | }
166 |
167 | this.clientId = clientId;
168 | clientIdBytes = clientId.getBytes(UTF8);
169 | clientIdLength = (short) clientIdBytes.length;
170 |
171 | this.topic = topic;
172 | topicBytes = topic.getBytes(UTF8);
173 | topicLength = (short) topicBytes.length;
174 |
175 | this.partition = partition;
176 | this.offset = offset;
177 |
178 | initializeMetrics();
179 |
180 | offsetRequestBytes = new byte[44 + clientIdLength + topicLength];
181 | offsetRequestBuffer = ByteBuffer.wrap(offsetRequestBytes);
182 |
183 | offsetResponseBytes = new byte[32 + topicLength];
184 | offsetResponseBuffer = ByteBuffer.wrap(offsetResponseBytes);
185 |
186 | requestBytes = new byte[52 + clientIdLength + topicLength];
187 | requestBuffer = ByteBuffer.wrap(requestBytes);
188 |
189 | fetchMessageMaxBytes = conf.getFetchMessageMaxBytes();
190 | responseBytes = new byte[fetchMessageMaxBytes + 32 + topicLength];
191 | responseBuffer = ByteBuffer.wrap(responseBytes);
192 |
193 | fetchWaitMaxMs = conf.getFetchWaitMaxMs();
194 | fetchMinBytes = conf.getFetchMinBytes();
195 |
196 | LOG.info("[{}-{}] connecting to broker", topic, partition);
197 | connectToBroker();
198 | }
199 |
200 | private void initializeMetrics() {
201 | String name = topic + "-" + partition;
202 |
203 | mMessageRequests = this.metrics.meter("krackle:consumer:partitions:" + name + ":message requests");
204 | mMessageRequestsTotal = this.metrics.meter("krackle:consumer:total:message requests");
205 | mMessagesReturned = this.metrics.meter("krackle:consumer:partitions:" + name + ":message returned");
206 | mMessagesReturnedTotal = this.metrics.meter("krackle:consumer:total:message returned");
207 | mBytesReturned = this.metrics.meter("krackle:consumer:partitions:" + name + ":bytes returned");
208 | mBytesReturnedTotal = this.metrics.meter("krackle:consumer:total:bytes returned");
209 | mMessageRequestsNoData = this.metrics.meter("krackle:consumer:partitions:" + name + ":no message returned");
210 | mMessageRequestsNoDataTotal = this.metrics.meter("krackle:consumer:total:no message returned");
211 | mBrokerReadAttempts = this.metrics.meter("krackle:consumer:partitions:" + name + ":broker consume attempts");
212 | mBrokerReadAttemptsTotal = this.metrics.meter("krackle:consumer:total:broker consume attempts");
213 | mBrokerReadSuccess = this.metrics.meter("krackle:consumer:partitions:" + name + ":broker consume success");
214 | mBrokerReadSuccessTotal = this.metrics.meter("krackle:consumer:total:broker consume success");
215 | mBrokerReadFailure = this.metrics.meter("krackle:consumer:partitions:" + name + ":broker consume failure");
216 | mBrokerReadFailureTotal = this.metrics.meter("krackle:consumer:total:broker consume failure");
217 | }
218 |
219 | /*
220 | * Read in a message from Kafka into the given byte array.
221 | *
222 | * If the size of the message exceeds maxLength, it will be truncated to fit.
223 | *
224 | * @param buffer the byte array to write into.
225 | * @param pos the position in the byte array to write to.
226 | * @param maxLength the max size of the message to write.
227 | * @return the number of bytes writen, or -1 if no data was returned.
228 | * @throws IOException
229 | */
230 | public int getMessage(byte[] buffer, int pos, int maxLength) throws IOException, BrokerUnavailableException {
231 | mMessageRequests.mark();
232 | mMessageRequestsTotal.mark();
233 |
234 | try {
235 | if (messageSetReader == null || messageSetReader.isReady() == false) {
236 | readFromBroker();
237 |
238 | if (messageSetReader == null || messageSetReader.isReady() == false) {
239 | mMessageRequestsNoData.mark();
240 | mMessageRequestsNoDataTotal.mark();
241 | return -1;
242 | }
243 | }
244 |
245 | bytesReturned = messageSetReader.getMessage(buffer, pos, maxLength);
246 |
247 | if (bytesReturned == -1) {
248 | mMessageRequestsNoData.mark();
249 | mMessageRequestsNoDataTotal.mark();
250 | return -1;
251 | }
252 |
253 | lastOffset = messageSetReader.getOffset();
254 | offset = messageSetReader.getNextOffset();
255 |
256 | //LOG.info("message received from messageSetReader latOffset {} offset {}" , lastOffset, offset);
257 | mMessagesReturned.mark();
258 | mMessagesReturnedTotal.mark();
259 | mBytesReturned.mark(bytesReturned);
260 | mBytesReturnedTotal.mark(bytesReturned);
261 |
262 | return bytesReturned;
263 | } catch (SocketTimeoutException e) {
264 | LOG.error("[{}-{}] socket timeout to {}: {}", topic, partition, broker.getNiceDescription(), e);
265 | connectToBroker();
266 | return -1;
267 | }
268 | }
269 |
270 | private void readFromBroker() throws IOException, BrokerUnavailableException {
271 | mBrokerReadAttempts.mark();
272 | mBrokerReadAttemptsTotal.mark();
273 |
274 | if (brokerSocket == null || brokerSocket.isClosed()) {
275 | LOG.info("[{}-{}] Connecting to broker", topic, partition);
276 | connectToBroker();
277 | }
278 |
279 | try {
280 | correlationId++;
281 |
282 | sendConsumeRequest(correlationId);
283 | receiveConsumeResponse(correlationId);
284 |
285 | mBrokerReadSuccess.mark();
286 | mBrokerReadSuccessTotal.mark();
287 | } catch (SocketTimeoutException e) {
288 | LOG.error("[{}-{}] socket timeout to {}", topic, partition, broker.getNiceDescription());
289 | connectToBroker();
290 | } catch (OffsetOutOfRangeException e) {
291 | mBrokerReadFailure.mark();
292 | mBrokerReadFailureTotal.mark();
293 |
294 | if (conf.getAutoOffsetReset().equals("smallest")) {
295 | LOG.warn("[{}-{}] offset {} out of range. Resetting to the earliest offset available {}", topic, partition, offset, getEarliestOffset());
296 | offset = getEarliestOffset();
297 | } else {
298 | if (conf.getAutoOffsetReset().equals("largest")) {
299 | LOG.warn("[{}-{}] Offset {} out of range. Resetting to the latest offset available {}", topic, partition, offset, getLatestOffset());
300 | offset = getLatestOffset();
301 | } else {
302 | LOG.error("[{}-{}] offset out of range and not configured to auto reset", topic, partition);
303 | throw e;
304 | }
305 | }
306 | } catch (Exception e) {
307 | LOG.error("[{}-{}] error getting data from broker: ", topic, partition, e);
308 |
309 | if (brokerSocket != null) {
310 | try {
311 | brokerSocket.close();
312 | } catch (IOException e1) {
313 | LOG.error("[{}-{}] error closing socket: ", topic, partition, e1);
314 | }
315 | }
316 |
317 | brokerSocket = null;
318 | mBrokerReadFailure.mark();
319 | mBrokerReadFailureTotal.mark();
320 | }
321 | }
322 |
323 | public long getEarliestOffset() throws BrokerUnavailableException {
324 | try {
325 | correlationId++;
326 | sendOffsetRequest(Constants.EARLIEST_OFFSET, correlationId);
327 | return getOffsetResponse(correlationId);
328 | } catch (SocketTimeoutException e) {
329 | LOG.error("[{}-{}] socket timeout to {}: {}", topic, partition, broker.getNiceDescription(), e);
330 | connectToBroker();
331 | } catch (IOException e) {
332 | LOG.error("[{}-{}] error getting earliest offset: ", topic, partition);
333 | }
334 | return 0L;
335 | }
336 |
337 | public long getLatestOffset() throws BrokerUnavailableException {
338 | try {
339 | correlationId++;
340 | sendOffsetRequest(Constants.LATEST_OFFSET, correlationId);
341 | return getOffsetResponse(correlationId);
342 | } catch (SocketTimeoutException e) {
343 | LOG.error("[{}-{}] socket timeout to {}: {}", topic, partition, broker.getNiceDescription(), e);
344 | connectToBroker();
345 | } catch (IOException e) {
346 | LOG.error("[{}-{}] error getting latest offset: ", topic, partition);
347 | }
348 | return Long.MAX_VALUE;
349 | }
350 |
351 | private void sendOffsetRequest(long time, int correlationId) throws IOException {
352 | LOG.debug("Sending request for offset. correlation id = {}, time = {}", correlationId, time);
353 |
354 | offsetRequestBuffer.clear();
355 |
356 | // skip 4 bytes for length
357 | offsetRequestBuffer.position(4);
358 |
359 | // API key
360 | offsetRequestBuffer.putShort(Constants.APIKEY_OFFSET_REQUEST);
361 |
362 | // API Version
363 | offsetRequestBuffer.putShort(Constants.API_VERSION);
364 |
365 | // Correlation Id
366 | offsetRequestBuffer.putInt(correlationId);
367 |
368 | // ClientId
369 | offsetRequestBuffer.putShort(clientIdLength);
370 | offsetRequestBuffer.put(clientIdBytes);
371 |
372 | // replica id is always -1
373 | offsetRequestBuffer.putInt(-1);
374 |
375 | // Only requesting for 1 topic
376 | offsetRequestBuffer.putInt(1);
377 |
378 | // Topic Name
379 | offsetRequestBuffer.putShort(topicLength);
380 | offsetRequestBuffer.put(topicBytes);
381 |
382 | // Only requesting for 1 partition
383 | offsetRequestBuffer.putInt(1);
384 |
385 | // Partition
386 | offsetRequestBuffer.putInt(partition);
387 |
388 | // Time for offset
389 | offsetRequestBuffer.putLong(time);
390 |
391 | // We only need one offset
392 | offsetRequestBuffer.putInt(1);
393 |
394 | // Add the length to the start
395 | offsetRequestBuffer.putInt(0, offsetRequestBuffer.position() - 4);
396 |
397 | brokerOut.write(offsetRequestBytes, 0, offsetRequestBuffer.position());
398 | }
399 |
400 | private long getOffsetResponse(int correlationId) throws IOException, BrokerUnavailableException {
401 | LOG.debug("[{}-{}] waiting for response. correlation id = {}", topic, partition, correlationId);
402 |
403 | try {
404 | // read the length of the response
405 | bytesRead = 0;
406 | while (bytesRead < 4) {
407 | bytesRead += brokerIn.read(offsetResponseBytes, bytesRead, 4 - bytesRead);
408 | }
409 |
410 | offsetResponseBuffer.clear();
411 | responseLength = offsetResponseBuffer.getInt();
412 |
413 | bytesRead = 0;
414 | while (bytesRead < responseLength) {
415 | bytesRead += brokerIn.read(offsetResponseBytes, bytesRead, responseLength - bytesRead);
416 | LOG.debug("[{}-{}] read {} bytes", topic, partition, bytesRead);
417 | }
418 |
419 | offsetResponseBuffer.clear();
420 |
421 | // Check correlation Id
422 | responseCorrelationId = offsetResponseBuffer.getInt();
423 |
424 | if (responseCorrelationId != correlationId) {
425 | LOG.error("[{}-{}] correlation id mismatch. Expected {}, got {}", topic, partition, correlationId, responseCorrelationId);
426 | throw new IOException("Correlation ID mismatch. Expected " + correlationId + ". Got " + responseCorrelationId + ".");
427 | }
428 |
429 | // We can skip a bunch of stuff here.
430 | // There is 1 topic (4 bytes), then the topic name (2 + topicLength
431 | // bytes), then the number of partitions (which is 1) (4 bytes),
432 | // then the partition id (4 bytes)
433 | offsetResponseBuffer.position(offsetResponseBuffer.position() + 4 + 2 + topicLength + 4 + 4);
434 |
435 | // Next is the error code.
436 | errorCode = offsetResponseBuffer.getShort();
437 |
438 | if (errorCode == KafkaError.OffsetOutOfRange.getCode()) {
439 | throw new OffsetOutOfRangeException();
440 | } else {
441 | if (errorCode != KafkaError.NoError.getCode()) {
442 | throw new IOException("Error from Kafka. (" + errorCode + ") " + KafkaError.getMessage(errorCode));
443 | }
444 | }
445 |
446 | // Finally, the offset. There is an array of one (skip 4 bytes)
447 | offsetResponseBuffer.position(offsetResponseBuffer.position() + 4);
448 | LOG.debug("Succeeded in request. correlation id = {}", correlationId);
449 |
450 | return offsetResponseBuffer.getLong();
451 |
452 | } catch (SocketTimeoutException e) {
453 | LOG.error("[{}-{}] socket timeout to {}: {}", topic, partition, broker.getNiceDescription(), e);
454 | connectToBroker();
455 | return -1;
456 | } finally {
457 | // Clean out any other data that is sitting on the socket to be read.
458 | // It's useless to us, but may through off future transactions if we
459 | // leave it there.
460 |
461 | bytesRead = 0;
462 | while (brokerIn.available() > 0) {
463 | bytesRead += brokerIn.read(offsetResponseBytes, bytesRead, offsetResponseBytes.length);
464 | }
465 | }
466 | }
467 |
468 | private void sendConsumeRequest(int correlationId) throws IOException {
469 | LOG.debug("[{}-{}] sending consume request. correlation id = {}", topic, partition, correlationId);
470 |
471 | requestBuffer.clear();
472 |
473 | // Skip 4 bytes for the request size
474 | requestBuffer.position(requestBuffer.position() + 4);
475 |
476 | // API key
477 | requestBuffer.putShort(Constants.APIKEY_FETCH_REQUEST);
478 |
479 | // API Version
480 | requestBuffer.putShort(Constants.API_VERSION);
481 |
482 | // Correlation Id
483 | requestBuffer.putInt(correlationId);
484 |
485 | // ClientId
486 | requestBuffer.putShort(clientIdLength);
487 | requestBuffer.put(clientIdBytes);
488 |
489 | // Replica ID is always -1
490 | requestBuffer.putInt(-1);
491 |
492 | // Max wait time
493 | requestBuffer.putInt(fetchWaitMaxMs);
494 |
495 | // Min bytes
496 | requestBuffer.putInt(fetchMinBytes);
497 |
498 | // Only requesting for 1 topic
499 | requestBuffer.putInt(1);
500 |
501 | // Topic Name
502 | requestBuffer.putShort(topicLength);
503 | requestBuffer.put(topicBytes);
504 |
505 | // Only requesting for 1 partition
506 | requestBuffer.putInt(1);
507 |
508 | // Partition
509 | requestBuffer.putInt(partition);
510 |
511 | // FetchOffset
512 | requestBuffer.putLong(offset);
513 |
514 | // MaxBytes
515 | requestBuffer.putInt(fetchMessageMaxBytes);
516 |
517 | /* Fill in missing data */
518 | // Full size
519 | requestBuffer.putInt(0, requestBuffer.position() - 4);
520 |
521 | // Send!
522 | brokerOut.write(requestBytes, 0, requestBuffer.position());
523 | }
524 |
525 | private void receiveConsumeResponse(int correlationId) throws IOException {
526 | LOG.debug("[{}-{}] waiting for response. correlation id = {}", topic, partition, correlationId);
527 |
528 | try {
529 | // read the length of the response
530 | bytesRead = 0;
531 | while (bytesRead < 4) {
532 | bytesRead += brokerIn.read(responseBytes, bytesRead, 4 - bytesRead);
533 | }
534 |
535 | responseBuffer.clear();
536 | responseLength = responseBuffer.getInt();
537 | bytesRead = 0;
538 |
539 | while (bytesRead < responseLength) {
540 | bytesRead += brokerIn.read(responseBytes, bytesRead, responseLength - bytesRead);
541 | }
542 |
543 | responseBuffer.clear();
544 |
545 | // Next is the corelation ID
546 | responseCorrelationId = responseBuffer.getInt();
547 |
548 | if (responseCorrelationId != correlationId) {
549 | LOG.error("[{}-{}] correlation id mismatch. Expected {}, got {}", topic, partition, correlationId, responseCorrelationId);
550 | throw new IOException("Correlation ID mismatch. Expected " + correlationId + ". Got " + responseCorrelationId + ".");
551 | }
552 |
553 | // We can skip a bunch of stuff here.
554 | // There is 1 topic (4 bytes), then the topic name (2 + topicLength
555 | // bytes), then the number of partitions (which is 1) (4 bytes),
556 | // then the partition id (4 bytes)
557 | responseBuffer.position(responseBuffer.position() + 4 + 2 + topicLength + 4 + 4);
558 |
559 | // Next is the error code.
560 | errorCode = responseBuffer.getShort();
561 |
562 | if (errorCode == KafkaError.OffsetOutOfRange.getCode()) {
563 | throw new OffsetOutOfRangeException();
564 | } else {
565 | if (errorCode != KafkaError.NoError.getCode()) {
566 | throw new IOException("Error from Kafka. (" + errorCode + ") " + KafkaError.getMessage(errorCode));
567 | }
568 | }
569 |
570 | // Next is the high watermark
571 | highWaterMark = responseBuffer.getLong();
572 |
573 | LOG.debug("[{}-{}] receiveConsumeRequest offset {} highWaterMark {}", topic, partition, offset, highWaterMark);
574 |
575 | // Message set size
576 | messageSetSize = responseBuffer.getInt();
577 |
578 | messageSetReader.init(responseBytes, responseBuffer.position(), messageSetSize);
579 |
580 | LOG.debug("[{}-{}] succeeded in request. correlation id = {}", topic, partition, correlationId);
581 |
582 | } finally {
583 | // Clean out any other data that is sitting on the socket to be read.
584 | // It's useless to us, but may through off future transactions if we
585 | // leave it there.
586 |
587 | bytesRead = 0;
588 |
589 | while (brokerIn.available() > 0) {
590 | bytesRead += brokerIn.read(responseBytes, bytesRead, responseBytes.length);
591 | }
592 | }
593 | }
594 |
595 | private void connectToBroker() throws BrokerUnavailableException {
596 | long backoff = 1000;
597 | long retries = 5;
598 | long attempt = 1;
599 | Boolean stopping = false;
600 |
601 | while (!stopping) {
602 | try {
603 | MetaData meta = MetaData.getMetaData(conf.getMetadataBrokerList(),
604 | topic,
605 | clientId);
606 | broker = meta.getTopic(topic).getPartition(partition).getLeader();
607 | LOG.info("[{}-{}] connecting to broker {}", topic, partition, broker.getNiceDescription());
608 |
609 | brokerSocket = AuthenticatedSocketSingleton.getInstance().build(broker.getHost(), broker.getPort());
610 | brokerSocket.setSoTimeout(conf.getSocketTimeoutMs());
611 | brokerSocket.setKeepAlive(true);
612 | brokerSocket.setReceiveBufferSize(conf.getSocketReceiveBufferBytes());
613 | brokerIn = brokerSocket.getInputStream();
614 | brokerOut = brokerSocket.getOutputStream();
615 |
616 | LOG.info("[{}-{}] successfully connected to broker {} and set a timeout of {}",
617 | topic, partition, broker.getNiceDescription(), conf.getSocketTimeoutMs());
618 |
619 | break;
620 | } catch (Exception e) {
621 | if (attempt < retries) {
622 | LOG.error("[{}-{}] error connecting to broker on attempt {}/{}, retrying in {} seconds... error message was: ",
623 | topic, partition, attempt, retries, (backoff / 1000), e);
624 | try {
625 | Thread.sleep(backoff);
626 | } catch (InterruptedException ie) {
627 | stopping = true;
628 | }
629 |
630 | backoff *= 2;
631 | attempt++;
632 | } else {
633 | throw new BrokerUnavailableException("Failed to connect to broker, no retries left--giving up", e);
634 | }
635 | }
636 | }
637 | }
638 |
639 | public long getLastOffset() {
640 | return lastOffset;
641 | }
642 |
643 | public long getNextOffset() {
644 | return offset;
645 | }
646 |
647 | public long getHighWaterMark() {
648 | return highWaterMark;
649 | }
650 |
651 | public void setNextOffset(long nextOffset) throws IOException, BrokerUnavailableException {
652 | LOG.info("[{}-{}] request to set the next offset to {} received", topic, partition, nextOffset);
653 |
654 | this.offset = nextOffset;
655 |
656 | correlationId++;
657 |
658 | try {
659 | sendConsumeRequest(correlationId);
660 | receiveConsumeResponse(correlationId);
661 | } catch (SocketTimeoutException e) {
662 | LOG.error("[{}-{}] socket timeout to {}", topic, partition, broker.getNiceDescription());
663 | connectToBroker();
664 | }
665 |
666 | LOG.info("[{}-{}] successfully set the next offset to {} via correlation ID {}", topic, partition, nextOffset, correlationId);
667 | }
668 |
669 | }
670 |
--------------------------------------------------------------------------------
/src/main/java/com/blackberry/bdp/krackle/consumer/ConsumerConfiguration.java:
--------------------------------------------------------------------------------
1 | /**
2 | * Copyright 2015 BlackBerry, Limited.
3 | *
4 | * Licensed under the Apache License, Version 2.0 (the "License");
5 | * you may not use this file except in compliance with the License.
6 | * You may obtain a copy of the License at
7 | *
8 | * http://www.apache.org/licenses/LICENSE-2.0
9 | *
10 | * Unless required by applicable law or agreed to in writing, software
11 | * distributed under the License is distributed on an "AS IS" BASIS,
12 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13 | * See the License for the specific language governing permissions and
14 | * limitations under the License.
15 | */
16 | package com.blackberry.bdp.krackle.consumer;
17 |
18 | import com.blackberry.bdp.krackle.auth.AuthenticatedSocketSingleton;
19 | import java.util.ArrayList;
20 | import java.util.List;
21 | import java.util.Properties;
22 |
23 | import org.slf4j.Logger;
24 | import org.slf4j.LoggerFactory;
25 |
26 | /**
27 | * Configuration for a consumer.
28 | *
29 | * Many of these properties are the same as those in the standard Java client, as documented at http://kafka.apache.org/documentation.html#consumerconfigs
30 | *
31 | * Valid properties are
32 | * | property | 36 | *default | 37 | *description | 38 | *
|---|---|---|
| metadata.broker.list | 42 | *43 | * | (required) A list of seed brokers to connect to in order to get information about the Kafka broker cluster. | 44 | *
| fetch.message.max.bytes | 48 | *1024 * 1024 | 49 | *The number of byes of messages to attempt to fetch for each topic-partition in each fetch request. These bytes will be read into memory for each partition, so this helps control the memory used by the consumer. The fetch request size must be at least as large as the maximum message size the server allows or else it is possible for the producer to send messages larger than the consumer can fetch. | 50 | *
| fetch.wait.max.ms | 54 | *100 | 55 | *The maximum amount of time the server will block before answering the fetch request if there isn't sufficient data to immediately satisfy fetch.min.bytes | 56 | *
| fetch.min.bytes | 60 | *1 | 61 | *The minimum amount of data the server should return for a fetch request. If insufficient data is available the request will wait for that much data to accumulate before answering the request. | 62 | *
| socket.receive.buffer.bytes | 66 | *64 * 1024 | 67 | *The socket receive buffer for network requests | 68 | *
| auto.offset.reset | 72 | *largest | 73 | *What to do when there is no initial offset in ZooKeeper or if an offset is out of range:
74 | *
|
80 | *
-1 if no data was returned.
95 | * @throws IOException
96 | */
97 | public int getMessage(byte[] dest, int pos, int maxLength) throws IOException {
98 | if (messageSetReader != null && messageSetReader.isReady()) {
99 | bytesCopied = messageSetReader.getMessage(dest, pos, maxLength);
100 | offset = messageSetReader.getOffset();
101 |
102 | if (!messageSetReader.isReady() && !buffer.hasRemaining()) {
103 | ready = false;
104 | } else {
105 | ready = true;
106 | }
107 | } else {
108 | // There are occasional truncated messages. If we don't have enough,
109 | // then return -1 and go not-ready
110 | // This will cover the offset, message size and crc
111 | if (buffer.remaining() < 8 + 4) {
112 | ready = false;
113 | return -1;
114 | }
115 |
116 | // offset
117 | offset = buffer.getLong();
118 |
119 | // messageSize
120 | messageSize = buffer.getInt();
121 |
122 | // This should be the last size check we need to do.
123 | if (buffer.remaining() < messageSize) {
124 | ready = false;
125 | return -1;
126 | }
127 |
128 | // Crc => int32
129 | crc = buffer.getInt();
130 |
131 | // check that the crc is correct
132 | crc32.reset();
133 | crc32.update(bytes, buffer.position(), messageSize - 4);
134 |
135 | if (crc != (int) crc32.getValue()) {
136 | LOG.error("CRC value mismatch.");
137 | ready = false;
138 | return -1;
139 | }
140 |
141 | // MagicByte => int8
142 | magicByte = buffer.get();
143 |
144 | if (magicByte != Constants.MAGIC_BYTE) {
145 | LOG.error("Incorrect magic byte.");
146 | ready = false;
147 | return -1;
148 | }
149 |
150 | // Attributes => int8
151 | attributes = buffer.get();
152 | compression = (byte) (attributes & Constants.COMPRESSION_MASK);
153 |
154 | // Key => bytes
155 | keyLength = buffer.getInt();
156 |
157 | if (keyLength == -1) {
158 | // null key
159 | } else {
160 | // ignore the key
161 | buffer.position(buffer.position() + keyLength);
162 | }
163 |
164 | // Value => bytes
165 | valueLength = buffer.getInt();
166 |
167 | if (valueLength == -1) {
168 | // null value. return -1, but we may still be ready.
169 | if (!buffer.hasRemaining()) {
170 | ready = false;
171 | }
172 | return -1;
173 | }
174 |
175 | if (compression == Constants.NO_COMPRESSION) {
176 | bytesCopied = Math.min(maxLength, valueLength);
177 | if (bytesCopied < valueLength) {
178 | LOG.warn("Truncating message from {} to {} bytes.", valueLength, maxLength);
179 | }
180 | System.arraycopy(bytes, buffer.position(), dest, pos, bytesCopied);
181 | } else {
182 | if (compression == Constants.SNAPPY) {
183 | decompressedSize = decompress(getSnappyDecompressor());
184 | ensureMessageSetReader();
185 | messageSetReader.init(decompressionBytes, 0, decompressedSize);
186 |
187 | if (messageSetReader.isReady()) {
188 | bytesCopied = messageSetReader.getMessage(dest, pos, maxLength);
189 | offset = messageSetReader.getOffset();
190 | }
191 | } else {
192 | if (compression == Constants.GZIP) {
193 | decompressedSize = decompress(getGzipDecompressor());
194 | ensureMessageSetReader();
195 | messageSetReader.init(decompressionBytes, 0, decompressedSize);
196 |
197 | if (messageSetReader.isReady()) {
198 | bytesCopied = messageSetReader.getMessage(dest, pos, maxLength);
199 | offset = messageSetReader.getOffset();
200 | }
201 | }
202 | }
203 | }
204 |
205 | buffer.position(buffer.position() + valueLength);
206 |
207 | if ((messageSetReader == null || !messageSetReader.isReady()) && !buffer.hasRemaining()) {
208 | ready = false;
209 | } else {
210 | ready = true;
211 | }
212 | }
213 |
214 | return bytesCopied;
215 | }
216 |
217 | private int decompress(Decompressor decompressor) throws IOException {
218 | while (true) {
219 | decompressedSize = decompressor.decompress(bytes, buffer.position(), valueLength, decompressionBytes, 0, decompressionBytes.length);
220 | if (decompressedSize == -1) {
221 | // Our output buffer was not big enough. So increase our
222 | // buffers and retry. This should be very rare.
223 |
224 | LOG.debug("Expanding decompression buffer from {} to {}", decompressionBytes.length, 2 * decompressionBytes.length);
225 |
226 | decompressionBytes = new byte[2 * decompressionBytes.length];
227 | } else {
228 | // we didn't fill the buffer. So our buffer was big enough.
229 | break;
230 | }
231 | }
232 |
233 | return decompressedSize;
234 | }
235 |
236 | private void ensureMessageSetReader() {
237 | if (messageSetReader == null) {
238 | messageSetReader = new MessageSetReader();
239 | }
240 | }
241 |
242 | private SnappyDecompressor getSnappyDecompressor() {
243 | if (snappyDecompressor == null) {
244 | snappyDecompressor = new SnappyDecompressor();
245 | }
246 | return snappyDecompressor;
247 | }
248 |
249 | private GzipDecompressor getGzipDecompressor() {
250 | if (gzipDecompressor == null) {
251 | gzipDecompressor = new GzipDecompressor();
252 | }
253 | return gzipDecompressor;
254 | }
255 |
256 | /**
257 | * Checks if this message set ready is ready.
258 | *
259 | * Ready means that it is initialized and has not reached the end of its data. This does not guarantee that the next request will succeed, since the remaining data could be a truncated message.
260 | *
261 | * @return true if this MessageSetReader is ready, otherwise false.
262 | */
263 | public boolean isReady() {
264 | return ready;
265 | }
266 |
267 | /**
268 | * Gets the offset of the last message read.
269 | *
270 | * @return the offset of the last message read.
271 | */
272 | public long getOffset() {
273 | return offset;
274 | }
275 |
276 | /**
277 | * Gets the offset of the next message that would be returned.
278 | *
279 | * @return the offset of the next message that would be returned.
280 | */
281 | public long getNextOffset() {
282 | return offset + 1;
283 | }
284 |
285 | }
286 |
--------------------------------------------------------------------------------
/src/main/java/com/blackberry/bdp/krackle/consumer/OffsetOutOfRangeException.java:
--------------------------------------------------------------------------------
1 | /**
2 | * Copyright 2014 BlackBerry, Limited.
3 | *
4 | * Licensed under the Apache License, Version 2.0 (the "License");
5 | * you may not use this file except in compliance with the License.
6 | * You may obtain a copy of the License at
7 | *
8 | * http://www.apache.org/licenses/LICENSE-2.0
9 | *
10 | * Unless required by applicable law or agreed to in writing, software
11 | * distributed under the License is distributed on an "AS IS" BASIS,
12 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13 | * See the License for the specific language governing permissions and
14 | * limitations under the License.
15 | */
16 | package com.blackberry.bdp.krackle.consumer;
17 |
18 | import java.io.IOException;
19 |
20 | /**
21 | * Exception used to handle offset out of range errors being returned by a
22 | * broker.
23 | */
24 | public class OffsetOutOfRangeException extends IOException {
25 |
26 | private static final long serialVersionUID = 1L;
27 |
28 | }
29 |
--------------------------------------------------------------------------------
/src/main/java/com/blackberry/bdp/krackle/exceptions/AuthenticationException.java:
--------------------------------------------------------------------------------
1 | /**
2 | * Copyright 2015 BlackBerry, Limited.
3 | *
4 | * Licensed under the Apache License, Version 2.0 (the "License");
5 | * you may not use this file except in compliance with the License.
6 | * You may obtain a copy of the License at
7 | *
8 | * http://www.apache.org/licenses/LICENSE-2.0
9 | *
10 | * Unless required by applicable law or agreed to in writing, software
11 | * distributed under the License is distributed on an "AS IS" BASIS,
12 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13 | * See the License for the specific language governing permissions and
14 | * limitations under the License.
15 | */
16 | package com.blackberry.bdp.krackle.exceptions;
17 |
18 | public class AuthenticationException extends Exception {
19 | public AuthenticationException(String message) {
20 | super(message);
21 | }
22 | }
23 |
--------------------------------------------------------------------------------
/src/main/java/com/blackberry/bdp/krackle/exceptions/InvalidConfigurationTypeException.java:
--------------------------------------------------------------------------------
1 | /**
2 | * Copyright 2015 BlackBerry, Limited.
3 | *
4 | * Licensed under the Apache License, Version 2.0 (the "License");
5 | * you may not use this file except in compliance with the License.
6 | * You may obtain a copy of the License at
7 | *
8 | * http://www.apache.org/licenses/LICENSE-2.0
9 | *
10 | * Unless required by applicable law or agreed to in writing, software
11 | * distributed under the License is distributed on an "AS IS" BASIS,
12 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13 | * See the License for the specific language governing permissions and
14 | * limitations under the License.
15 | */
16 | package com.blackberry.bdp.krackle.exceptions;
17 |
18 | public class InvalidConfigurationTypeException extends Exception {
19 | public InvalidConfigurationTypeException(String message) {
20 | super(message);
21 | }
22 | }
23 |
--------------------------------------------------------------------------------
/src/main/java/com/blackberry/bdp/krackle/exceptions/MissingConfigurationException.java:
--------------------------------------------------------------------------------
1 | /**
2 | * Copyright 2015 BlackBerry, Limited.
3 | *
4 | * Licensed under the Apache License, Version 2.0 (the "License");
5 | * you may not use this file except in compliance with the License.
6 | * You may obtain a copy of the License at
7 | *
8 | * http://www.apache.org/licenses/LICENSE-2.0
9 | *
10 | * Unless required by applicable law or agreed to in writing, software
11 | * distributed under the License is distributed on an "AS IS" BASIS,
12 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13 | * See the License for the specific language governing permissions and
14 | * limitations under the License.
15 | */
16 | package com.blackberry.bdp.krackle.exceptions;
17 |
18 | public class MissingConfigurationException extends Exception {
19 | public MissingConfigurationException(String message) {
20 | super(message);
21 | }
22 | }
23 |
--------------------------------------------------------------------------------
/src/main/java/com/blackberry/bdp/krackle/jaas/Login.java:
--------------------------------------------------------------------------------
1 | /**
2 | * Copyright 2015 BlackBerry, Limited.
3 | *
4 | * Licensed under the Apache License, Version 2.0 (the "License");
5 | * you may not use this file except in compliance with the License.
6 | * You may obtain a copy of the License at
7 | *
8 | * http://www.apache.org/licenses/LICENSE-2.0
9 | *
10 | * Unless required by applicable law or agreed to in writing, software
11 | * distributed under the License is distributed on an "AS IS" BASIS,
12 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13 | * See the License for the specific language governing permissions and
14 | * limitations under the License.
15 | */
16 | package com.blackberry.bdp.krackle.jaas;
17 |
18 | /**
19 | * This class was copied from the Apache ZooKeeper project:
20 | *
21 | * git@github.com:apache/zookeeper.git
22 | * 92707a6a84a7965df2d7d8ead0acb721b6e62878
23 | *
24 | * Adapted as required to work with Krackle and it's implementation
25 | * of AuthenticatedSocketBuilder where needed. The following
26 | * major changes were performed:
27 | *
28 | * 1) Package name was changed
29 | * 2) Use of a login via ticket cache/kinit command disabled
30 | * 3) Use of org.apache.zookeeper.common.Time removed
31 | * 4) JavaDoc added where missing
32 | * 5) Required the principal to be found in the JAAS config
33 | * 6) Added getPrincipal()
34 | * 7) Added a client call back handler
35 | *
36 | */
37 |
38 | /**
39 | * This class is responsible for refreshing Kerberos credentials for
40 | * logins for both Zookeeper client and server.
41 | * See ZooKeeperSaslServer for server-side usage.
42 | * See ZooKeeperSaslClient for client-side usage.
43 | */
44 | import com.blackberry.bdp.krackle.Time;
45 | import javax.security.auth.kerberos.KerberosPrincipal;
46 | import javax.security.auth.login.AppConfigurationEntry;
47 | import javax.security.auth.login.Configuration;
48 | import javax.security.auth.login.LoginContext;
49 | import javax.security.auth.login.LoginException;
50 | import javax.security.auth.callback.CallbackHandler;
51 |
52 | import org.slf4j.Logger;
53 | import org.slf4j.LoggerFactory;
54 |
55 | import javax.security.auth.kerberos.KerberosTicket;
56 | import javax.security.auth.Subject;
57 |
58 | import java.util.Date;
59 | import java.util.Random;
60 | import java.util.Set;
61 | import javax.security.auth.callback.Callback;
62 | import javax.security.auth.callback.NameCallback;
63 | import javax.security.auth.callback.PasswordCallback;
64 | import javax.security.auth.callback.UnsupportedCallbackException;
65 | import javax.security.sasl.AuthorizeCallback;
66 | import javax.security.sasl.RealmCallback;
67 |
68 | public class Login {
69 |
70 | private static final Logger LOG = LoggerFactory.getLogger(Login.class);
71 | public CallbackHandler callbackHandler;
72 |
73 | // LoginThread will sleep until 80% of time from last refresh to
74 | // ticket's expiry has been reached, at which time it will wake
75 | // and try to renew the ticket.
76 | private static final float TICKET_RENEW_WINDOW = 0.80f;
77 |
78 | /**
79 | * Percentage of random jitter added to the renewal time
80 | */
81 | private static final float TICKET_RENEW_JITTER = 0.05f;
82 |
83 | // Regardless of TICKET_RENEW_WINDOW setting above and the ticket expiry time,
84 | // thread will not sleep between refresh attempts any less than 1 minute (60*1000 milliseconds = 1 minute).
85 | // Change the '1' to e.g. 5, to change this to 5 minutes.
86 | private static final long MIN_TIME_BEFORE_RELOGIN = 1 * 60 * 1000L;
87 |
88 | private Subject subject = null;
89 | private Thread t = null;
90 | private boolean isKrbTicket = false;
91 | private boolean isUsingTicketCache = false;
92 |
93 | /** Random number generator */
94 | private static final Random rng = new Random();
95 |
96 | private LoginContext login = null;
97 | private String loginContextName = null;
98 | private String principal = null;
99 |
100 | // Initialize 'lastLogin' to do a login at first time
101 | private long lastLogin = Time.currentElapsedTime() - MIN_TIME_BEFORE_RELOGIN;
102 |
103 | /**
104 | * LoginThread constructor. The constructor starts the thread used
105 | * to periodically re-login to the Kerberos Ticket Granting Server.
106 | * @param loginContextName
107 | * name of section in JAAS file that will be use to login.
108 | * Passed as first param to javax.security.auth.login.LoginContext().
109 | *
110 | * @param callbackHandler
111 | * Passed as second param to javax.security.auth.login.LoginContext().
112 | * @throws javax.security.auth.login.LoginException
113 | * Thrown if authentication fails.
114 | */
115 | public Login(final String loginContextName, CallbackHandler callbackHandler)
116 | throws LoginException {
117 | this.callbackHandler = callbackHandler;
118 | login = login(loginContextName);
119 | this.loginContextName = loginContextName;
120 | subject = login.getSubject();
121 | isKrbTicket = !subject.getPrivateCredentials(KerberosTicket.class).isEmpty();
122 | AppConfigurationEntry entries[] = Configuration.getConfiguration().getAppConfigurationEntry(loginContextName);
123 | for (AppConfigurationEntry entry : entries) {
124 | // there will only be a single entry, so this for() loop will only be iterated through once.
125 | if (entry.getOptions().get("useTicketCache") != null) {
126 | String val = (String) entry.getOptions().get("useTicketCache");
127 | if (val.equals("true")) {
128 | isUsingTicketCache = true;
129 | }
130 | }
131 | if (entry.getOptions().get("principal") != null) {
132 | principal = (String) entry.getOptions().get("principal");
133 | } else {
134 | throw new LoginException("could not determine principal from login context configuration");
135 | }
136 | break;
137 | }
138 |
139 | if (!isKrbTicket) {
140 | // if no TGT, do not bother with ticket management.
141 | return;
142 | }
143 |
144 | // Refresh the Ticket Granting Ticket (TGT) periodically. How often to refresh is determined by the
145 | // TGT's existing expiry date and the configured MIN_TIME_BEFORE_RELOGIN. For testing and development,
146 | // you can decrease the interval of expiration of tickets (for example, to 3 minutes) by running :
147 | // "modprinc -maxlife 3mins NOTE: Every single one of these properties can be overwritten for a specific topic by using the following property patten:
33 | * 34 | *source.<topic>.<property>
35 | * 36 | *Valid properties are
37 | * 38 | *| property | 42 | *default | 43 | *description | 44 | *
|---|---|---|
| metadata.broker.list | 48 | *49 | * | (required) A comma separated list of seed brokers to connect to in order to get metadata about the cluster. | 50 | *
| queue.buffering.max.ms | 54 | *5000 | 55 | *Maximum time to buffer data. For example a setting of 100 will try to batch together 100ms of messages to send at once. This will improve throughput but adds message delivery latency due to the buffering. | 56 | *
| request.required.acks | 60 | *1 | 61 | *This value controls when a produce request is considered completed. Specifically, how many other brokers must have committed the data to their log and acknowledged this to the leader? Typical values are
62 | *
|
68 | *
| request.timeout.ms | 72 | *10000 | 73 | *The amount of time the broker will wait trying to meet the request.required.acks requirement before sending back an error to the client. 74 | * | 75 | *
| message.send.max.retries | 79 | *3 | 80 | *This property will cause the producer to automatically retry a failed send request. This property specifies the number of retries when such failures occur. Note that setting a non-zero value here can lead to duplicates in the case of network errors that cause a message to be sent but the acknowledgement to be lost. | 81 | *
| retry.backoff.ms | 85 | *100 | 86 | *Before each retry, the producer refreshes the metadata of relevant topics to see if a new leader has been elected. Since leader election takes a bit of time, this property specifies the amount of time that the producer waits before refreshing the metadata. | 87 | *
| topic.metadata.refresh.interval.ms | 91 | *60 * 10 * 1000 | 92 | *The producer generally refreshes the topic metadata from brokers when there is a failure (partition missing, leader not available...). It will also poll regularly (default: every 10min so 600000ms). If you set this to a negative value, metadata will only get refreshed on failure. If you set this to zero, the metadata will get refreshed after each message sent (not recommended). Important note: the refresh happen only AFTER the message is sent, so if the producer never sends a message the metadata is never refreshed | 93 | *
| message.buffer.size | 97 | *1024*1024 | 98 | *The size of each buffer that is used to store raw messages before they are sent. Since a full buffer is sent at once, don't make this too big. | 99 | *
| use.shared.buffers | 103 | *false | 104 | *If this is set to true, then there will be one set of buffers that is used by all Producer instances. In that case, ensure that num.buffers is large enough to accommodate this. | 105 | *
| num.buffers | 109 | *2 | 110 | *The number of buffers to use. At any given time, there is up to one buffer being filled with new data, up to one buffer having its data sent to the broker, and any number of buffers waiting to be filled and/or sent. 111 | * 112 | * Essentially, the limit of the amount of data that can be queued at at any given time is message.buffer.size num.buffers. Although, in reality, you won't get buffers to 100% full each time. 113 | * 114 | * If use.shared.buffers=false, then this many buffers will be allocated per Producer. If use.shared.buffers=true then this is the total for the JVM. | 115 | *
| send.buffer.size | 119 | *1.5 * 1024 * 1024 | 120 | *Size of the byte buffer used to store the final (with headers and compression applied) data to be sent to the broker. | 121 | * 122 | *
| sender.threads | 125 | *1 | 126 | *How many threads to use for sending to the broker. A larger number can be useful for higher latency high volume topics however message order cannot be guaranteed | 127 | *
| compression.codec | 131 | *none | 132 | *This parameter allows you to specify the compression codec for all data generated by this producer. Valid values are "none", "gzip" and "snappy". | 133 | *
| gzip.compression.level | 137 | *java.util.zip.Deflater.DEFAULT_COMPRESSION | 138 | *If compression.codec is set to gzip, then this allows configuration of the compression level.
139 | *
|
145 | *
| queue.enqueue.timeout.ms | 149 | *-1 | 150 | *The amount of time to block before dropping messages when all buffers are full. If set to 0 events will be enqueued immediately or dropped if the queue is full (the producer send call will never block). If set to -1 the producer will block indefinitely and never willingly drop a send. | 151 | *
NOTE: Quick rotate, rotate, and partition rotation in General
156 | 157 | *Quick rotate is no longer a supported configuration item. Instead all topic meta data refreshes will rotate partitions and if quicker rotation is required than topic.metadata.refresh.interval.ms can be configured accordingly. Regular topic specific overrides are possible as well for topics that require faster rotaiton.
158 | 159 | * 160 | */ 161 | public class ProducerConfiguration { 162 | 163 | private static final Logger LOG = LoggerFactory.getLogger(ProducerConfiguration.class); 164 | 165 | protected static final int ONE_MB = 1024 * 1024; 166 | private final Properties props; 167 | public String topicName; 168 | 169 | // Options matching the producer client 170 | private List