├── .github └── workflows │ └── semgrep.yml ├── Makefile ├── README ├── TODO ├── c ├── entry.pb-c.c └── entry.pb-c.h ├── common.h ├── cpp ├── entry.pb.cc └── entry.pb.h ├── entry.proto ├── examples ├── data.sql ├── getting_started ├── helper.sql ├── import_data.cc └── importer.json ├── pg_kc.c └── pg_kc.sql /.github/workflows/semgrep.yml: -------------------------------------------------------------------------------- 1 | 2 | on: 3 | pull_request: {} 4 | workflow_dispatch: {} 5 | push: 6 | branches: 7 | - main 8 | - master 9 | name: Semgrep config 10 | jobs: 11 | semgrep: 12 | name: semgrep/ci 13 | runs-on: ubuntu-20.04 14 | env: 15 | SEMGREP_APP_TOKEN: ${{ secrets.SEMGREP_APP_TOKEN }} 16 | SEMGREP_URL: https://cloudflare.semgrep.dev 17 | SEMGREP_APP_URL: https://cloudflare.semgrep.dev 18 | SEMGREP_VERSION_CHECK_URL: https://cloudflare.semgrep.dev/api/check-version 19 | container: 20 | image: returntocorp/semgrep 21 | steps: 22 | - uses: actions/checkout@v3 23 | - run: semgrep ci 24 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | PG_INCLUDE=`pg_config --includedir-server` 2 | PG_INSTALL=`pg_config --pkglibdir` 3 | 4 | INCLUDE=-I$(PG_INCLUDE) 5 | DEST_DIR=$(PG_INSTALL) 6 | OPT_LEVEL=-O2 7 | CFLAGS=$(OPT_LEVEL) -W -Wall -Wpointer-arith -Wno-unused-parameter -Wunused-function -Wunused-variable -Wunused-value -Werror 8 | LIBS=-lkyotocabinet -lprotobuf -lprotobuf-c 9 | 10 | .c.o: 11 | gcc -g -c $(INCLUDE) $(CFLAGS) -fpic c/entry.pb-c.c $< 12 | 13 | .cc.o: 14 | gcc -g -c $(INCLUDE) $(CFLAGS) -fpic cpp/entry.pb.cc $< 15 | 16 | all: libpg_kc.so 17 | 18 | gen: 19 | protoc-c --c_out=c entry.proto 20 | protoc --cpp_out=cpp entry.proto 21 | 22 | clean: 23 | rm *.o 24 | rm *.so 25 | 26 | libpg_kc.so: entry.proto pg_kc.o entry.pb-c.o 27 | gcc --library-path=/usr/local/lib --shared -o libpg_kc.so $(LIBS) -fpic entry.pb-c.o pg_kc.o 28 | 29 | install: libpg_kc.so 30 | install -s libpg_kc.so $(DEST_DIR) -------------------------------------------------------------------------------- /README: -------------------------------------------------------------------------------- 1 | 2 | SortaSQL 3 | 4 | An extension to PostgreSQL allowing Kyoto Cabinets to be used as a backing data store. 5 | Currently very customized for CloudFlare's usage pattern of online log analysis. 6 | 7 | Install 8 | 9 | Update the paths in common.h 10 | 11 | protoc-c --c_out=c entry.proto 12 | protoc --cpp_out=cpp entry.proto 13 | make 14 | sudo make install 15 | 16 | About 17 | 18 | Google has BigTable and GFS. 19 | Facebook has HBase and Cassandra. 20 | And then there's the up and coming set of Membase, CouchDB, MongoDB, and so on and so forth. 21 | 22 | These all scale by eliminating flexibility, presenting a user with a key-value interface where keys are ordered to allow efficient iteration. 23 | But by throwing out the relational part of a relational database, you lose a lot of value and flexibility in looking at data, not to 24 | mention all of the nice language bindings that old school RDBMs have. 25 | Compounding this, NoSQL DBs are also all new, all have bugs, and no one has that much experience managing them. 26 | Its a brave new world out there. 27 | 28 | At CloudFlare, we spent quite a while evaluating different NoSQL solutions. 29 | We were looking for a reliable and cheap way to store Big Data down the road, while storing Medium Data right now. 30 | We need to be able to scale with one technology, going from a single machine to two machines, to four and so on to full on a cluster. 31 | The catch is we don't want to buy that cluster right now. 32 | 33 | We messed with HBase and Hive in particular, but weren't able to get reliable performance from these while running on a scaled down infrastructure. 34 | For a while then we fell back on Postgres, but pretty soon this was showing signs of creaking under the load. 35 | But we still weren't ready to make the jump to a big rack full of servers. 36 | 37 | Needing a middle ground, we created a hybrid model using both SQL and noSQL technologies. 38 | In our existing Postgres DB, we created a custom data type which acts as a pointer into an embedded Kyoto Cabinet DB. 39 | Kyoto Cabinet is a collection of C functions which allow lookup and sets into a simple data file containing records, each of these consisting of a key/value pair. 40 | Adding a few functions and operators to Postgres allows searching of these KC files both using Postgres's explicit indexes and KC's implicit indexes (keys are ordered and can be stored via a B-Tree). 41 | Storing values serialized using Google's Protocol Buffers allows complex structures to be added as values, while not needing much diskspace. 42 | 43 | See also: 44 | 45 | http://www.postgresql.org/ 46 | http://fallabs.com/kyotocabinet/ 47 | 48 | Copyright 2011 CloudFlare, Inc. 49 | -------------------------------------------------------------------------------- /TODO: -------------------------------------------------------------------------------- 1 | Features which need to be worked on: 2 | 3 | * Working symetrical functions to support the PG callback functions input, output, receive, send. 4 | 5 | Right now all input runs via the bulk uploader. It would be good to enable adding data via the PG interface. 6 | This includes the text based input/output functions (symetrical here means that output from output() can be fed right back to input()), 7 | and their binary equivalents recieve/send. 8 | 9 | * Support for INSERT, UPDATE statements. 10 | 11 | * Smarter sharding/replication layer, moving data at the KC/PG level across boxes rather than at the user level. 12 | 13 | * Possible integration with a caching filesystem (http://en.gentoo-wiki.com/wiki/CacheFS) to imporve performance. 14 | 15 | * More examples. 16 | -------------------------------------------------------------------------------- /c/entry.pb-c.c: -------------------------------------------------------------------------------- 1 | /* Generated by the protocol buffer compiler. DO NOT EDIT! */ 2 | 3 | #include "entry.pb-c.h" 4 | void cloudflare__zone_time_bucket__counter__init 5 | (Cloudflare__ZoneTimeBucket__Counter *message) 6 | { 7 | static Cloudflare__ZoneTimeBucket__Counter init_value = CLOUDFLARE__ZONE_TIME_BUCKET__COUNTER__INIT; 8 | *message = init_value; 9 | } 10 | void cloudflare__zone_time_bucket__init 11 | (Cloudflare__ZoneTimeBucket *message) 12 | { 13 | static Cloudflare__ZoneTimeBucket init_value = CLOUDFLARE__ZONE_TIME_BUCKET__INIT; 14 | *message = init_value; 15 | } 16 | size_t cloudflare__zone_time_bucket__get_packed_size 17 | (const Cloudflare__ZoneTimeBucket *message) 18 | { 19 | PROTOBUF_C_ASSERT (message->base.descriptor == &cloudflare__zone_time_bucket__descriptor); 20 | return protobuf_c_message_get_packed_size ((const ProtobufCMessage*)(message)); 21 | } 22 | size_t cloudflare__zone_time_bucket__pack 23 | (const Cloudflare__ZoneTimeBucket *message, 24 | uint8_t *out) 25 | { 26 | PROTOBUF_C_ASSERT (message->base.descriptor == &cloudflare__zone_time_bucket__descriptor); 27 | return protobuf_c_message_pack ((const ProtobufCMessage*)message, out); 28 | } 29 | size_t cloudflare__zone_time_bucket__pack_to_buffer 30 | (const Cloudflare__ZoneTimeBucket *message, 31 | ProtobufCBuffer *buffer) 32 | { 33 | PROTOBUF_C_ASSERT (message->base.descriptor == &cloudflare__zone_time_bucket__descriptor); 34 | return protobuf_c_message_pack_to_buffer ((const ProtobufCMessage*)message, buffer); 35 | } 36 | Cloudflare__ZoneTimeBucket * 37 | cloudflare__zone_time_bucket__unpack 38 | (ProtobufCAllocator *allocator, 39 | size_t len, 40 | const uint8_t *data) 41 | { 42 | return (Cloudflare__ZoneTimeBucket *) 43 | protobuf_c_message_unpack (&cloudflare__zone_time_bucket__descriptor, 44 | allocator, len, data); 45 | } 46 | void cloudflare__zone_time_bucket__free_unpacked 47 | (Cloudflare__ZoneTimeBucket *message, 48 | ProtobufCAllocator *allocator) 49 | { 50 | PROTOBUF_C_ASSERT (message->base.descriptor == &cloudflare__zone_time_bucket__descriptor); 51 | protobuf_c_message_free_unpacked ((ProtobufCMessage*)message, allocator); 52 | } 53 | static const ProtobufCFieldDescriptor cloudflare__zone_time_bucket__counter__field_descriptors[2] = 54 | { 55 | { 56 | "key", 57 | 1, 58 | PROTOBUF_C_LABEL_REQUIRED, 59 | PROTOBUF_C_TYPE_STRING, 60 | 0, /* quantifier_offset */ 61 | PROTOBUF_C_OFFSETOF(Cloudflare__ZoneTimeBucket__Counter, key), 62 | NULL, 63 | NULL, 64 | NULL,NULL /* reserved1, reserved2 */ 65 | }, 66 | { 67 | "value", 68 | 2, 69 | PROTOBUF_C_LABEL_REQUIRED, 70 | PROTOBUF_C_TYPE_INT64, 71 | 0, /* quantifier_offset */ 72 | PROTOBUF_C_OFFSETOF(Cloudflare__ZoneTimeBucket__Counter, value), 73 | NULL, 74 | NULL, 75 | NULL,NULL /* reserved1, reserved2 */ 76 | }, 77 | }; 78 | static const unsigned cloudflare__zone_time_bucket__counter__field_indices_by_name[] = { 79 | 0, /* field[0] = key */ 80 | 1, /* field[1] = value */ 81 | }; 82 | static const ProtobufCIntRange cloudflare__zone_time_bucket__counter__number_ranges[1 + 1] = 83 | { 84 | { 1, 0 }, 85 | { 0, 2 } 86 | }; 87 | const ProtobufCMessageDescriptor cloudflare__zone_time_bucket__counter__descriptor = 88 | { 89 | PROTOBUF_C_MESSAGE_DESCRIPTOR_MAGIC, 90 | "cloudflare.ZoneTimeBucket.Counter", 91 | "Counter", 92 | "Cloudflare__ZoneTimeBucket__Counter", 93 | "cloudflare", 94 | sizeof(Cloudflare__ZoneTimeBucket__Counter), 95 | 2, 96 | cloudflare__zone_time_bucket__counter__field_descriptors, 97 | cloudflare__zone_time_bucket__counter__field_indices_by_name, 98 | 1, cloudflare__zone_time_bucket__counter__number_ranges, 99 | (ProtobufCMessageInit) cloudflare__zone_time_bucket__counter__init, 100 | NULL,NULL,NULL /* reserved[123] */ 101 | }; 102 | static const ProtobufCFieldDescriptor cloudflare__zone_time_bucket__field_descriptors[10] = 103 | { 104 | { 105 | "map_name", 106 | 1, 107 | PROTOBUF_C_LABEL_REQUIRED, 108 | PROTOBUF_C_TYPE_STRING, 109 | 0, /* quantifier_offset */ 110 | PROTOBUF_C_OFFSETOF(Cloudflare__ZoneTimeBucket, map_name), 111 | NULL, 112 | NULL, 113 | NULL,NULL /* reserved1, reserved2 */ 114 | }, 115 | { 116 | "classification", 117 | 2, 118 | PROTOBUF_C_LABEL_REQUIRED, 119 | PROTOBUF_C_TYPE_STRING, 120 | 0, /* quantifier_offset */ 121 | PROTOBUF_C_OFFSETOF(Cloudflare__ZoneTimeBucket, classification), 122 | NULL, 123 | NULL, 124 | NULL,NULL /* reserved1, reserved2 */ 125 | }, 126 | { 127 | "doctype", 128 | 3, 129 | PROTOBUF_C_LABEL_REQUIRED, 130 | PROTOBUF_C_TYPE_STRING, 131 | 0, /* quantifier_offset */ 132 | PROTOBUF_C_OFFSETOF(Cloudflare__ZoneTimeBucket, doctype), 133 | NULL, 134 | NULL, 135 | NULL,NULL /* reserved1, reserved2 */ 136 | }, 137 | { 138 | "pop", 139 | 4, 140 | PROTOBUF_C_LABEL_REQUIRED, 141 | PROTOBUF_C_TYPE_STRING, 142 | 0, /* quantifier_offset */ 143 | PROTOBUF_C_OFFSETOF(Cloudflare__ZoneTimeBucket, pop), 144 | NULL, 145 | NULL, 146 | NULL,NULL /* reserved1, reserved2 */ 147 | }, 148 | { 149 | "psource", 150 | 5, 151 | PROTOBUF_C_LABEL_REQUIRED, 152 | PROTOBUF_C_TYPE_STRING, 153 | 0, /* quantifier_offset */ 154 | PROTOBUF_C_OFFSETOF(Cloudflare__ZoneTimeBucket, psource), 155 | NULL, 156 | NULL, 157 | NULL,NULL /* reserved1, reserved2 */ 158 | }, 159 | { 160 | "result_id", 161 | 6, 162 | PROTOBUF_C_LABEL_REQUIRED, 163 | PROTOBUF_C_TYPE_STRING, 164 | 0, /* quantifier_offset */ 165 | PROTOBUF_C_OFFSETOF(Cloudflare__ZoneTimeBucket, result_id), 166 | NULL, 167 | NULL, 168 | NULL,NULL /* reserved1, reserved2 */ 169 | }, 170 | { 171 | "db_key", 172 | 7, 173 | PROTOBUF_C_LABEL_REQUIRED, 174 | PROTOBUF_C_TYPE_STRING, 175 | 0, /* quantifier_offset */ 176 | PROTOBUF_C_OFFSETOF(Cloudflare__ZoneTimeBucket, db_key), 177 | NULL, 178 | NULL, 179 | NULL,NULL /* reserved1, reserved2 */ 180 | }, 181 | { 182 | "db_path", 183 | 8, 184 | PROTOBUF_C_LABEL_REQUIRED, 185 | PROTOBUF_C_TYPE_STRING, 186 | 0, /* quantifier_offset */ 187 | PROTOBUF_C_OFFSETOF(Cloudflare__ZoneTimeBucket, db_path), 188 | NULL, 189 | NULL, 190 | NULL,NULL /* reserved1, reserved2 */ 191 | }, 192 | { 193 | "kv_map_file", 194 | 9, 195 | PROTOBUF_C_LABEL_OPTIONAL, 196 | PROTOBUF_C_TYPE_STRING, 197 | 0, /* quantifier_offset */ 198 | PROTOBUF_C_OFFSETOF(Cloudflare__ZoneTimeBucket, kv_map_file), 199 | NULL, 200 | NULL, 201 | NULL,NULL /* reserved1, reserved2 */ 202 | }, 203 | { 204 | "map_entry", 205 | 10, 206 | PROTOBUF_C_LABEL_REPEATED, 207 | PROTOBUF_C_TYPE_MESSAGE, 208 | PROTOBUF_C_OFFSETOF(Cloudflare__ZoneTimeBucket, n_map_entry), 209 | PROTOBUF_C_OFFSETOF(Cloudflare__ZoneTimeBucket, map_entry), 210 | &cloudflare__zone_time_bucket__counter__descriptor, 211 | NULL, 212 | NULL,NULL /* reserved1, reserved2 */ 213 | }, 214 | }; 215 | static const unsigned cloudflare__zone_time_bucket__field_indices_by_name[] = { 216 | 1, /* field[1] = classification */ 217 | 6, /* field[6] = db_key */ 218 | 7, /* field[7] = db_path */ 219 | 2, /* field[2] = doctype */ 220 | 8, /* field[8] = kv_map_file */ 221 | 9, /* field[9] = map_entry */ 222 | 0, /* field[0] = map_name */ 223 | 3, /* field[3] = pop */ 224 | 4, /* field[4] = psource */ 225 | 5, /* field[5] = result_id */ 226 | }; 227 | static const ProtobufCIntRange cloudflare__zone_time_bucket__number_ranges[1 + 1] = 228 | { 229 | { 1, 0 }, 230 | { 0, 10 } 231 | }; 232 | const ProtobufCMessageDescriptor cloudflare__zone_time_bucket__descriptor = 233 | { 234 | PROTOBUF_C_MESSAGE_DESCRIPTOR_MAGIC, 235 | "cloudflare.ZoneTimeBucket", 236 | "ZoneTimeBucket", 237 | "Cloudflare__ZoneTimeBucket", 238 | "cloudflare", 239 | sizeof(Cloudflare__ZoneTimeBucket), 240 | 10, 241 | cloudflare__zone_time_bucket__field_descriptors, 242 | cloudflare__zone_time_bucket__field_indices_by_name, 243 | 1, cloudflare__zone_time_bucket__number_ranges, 244 | (ProtobufCMessageInit) cloudflare__zone_time_bucket__init, 245 | NULL,NULL,NULL /* reserved[123] */ 246 | }; 247 | -------------------------------------------------------------------------------- /c/entry.pb-c.h: -------------------------------------------------------------------------------- 1 | /* Generated by the protocol buffer compiler. DO NOT EDIT! */ 2 | 3 | #ifndef PROTOBUF_C_entry_2eproto__INCLUDED 4 | #define PROTOBUF_C_entry_2eproto__INCLUDED 5 | 6 | #include 7 | 8 | PROTOBUF_C_BEGIN_DECLS 9 | 10 | 11 | typedef struct _Cloudflare__ZoneTimeBucket Cloudflare__ZoneTimeBucket; 12 | typedef struct _Cloudflare__ZoneTimeBucket__Counter Cloudflare__ZoneTimeBucket__Counter; 13 | 14 | 15 | /* --- enums --- */ 16 | 17 | 18 | /* --- messages --- */ 19 | 20 | struct _Cloudflare__ZoneTimeBucket__Counter 21 | { 22 | ProtobufCMessage base; 23 | char *key; 24 | int64_t value; 25 | }; 26 | #define CLOUDFLARE__ZONE_TIME_BUCKET__COUNTER__INIT \ 27 | { PROTOBUF_C_MESSAGE_INIT (&cloudflare__zone_time_bucket__counter__descriptor) \ 28 | , NULL, 0 } 29 | 30 | 31 | struct _Cloudflare__ZoneTimeBucket 32 | { 33 | ProtobufCMessage base; 34 | char *map_name; 35 | char *classification; 36 | char *doctype; 37 | char *pop; 38 | char *psource; 39 | char *result_id; 40 | char *db_key; 41 | char *db_path; 42 | char *kv_map_file; 43 | size_t n_map_entry; 44 | Cloudflare__ZoneTimeBucket__Counter **map_entry; 45 | }; 46 | #define CLOUDFLARE__ZONE_TIME_BUCKET__INIT \ 47 | { PROTOBUF_C_MESSAGE_INIT (&cloudflare__zone_time_bucket__descriptor) \ 48 | , NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, 0,NULL } 49 | 50 | 51 | /* Cloudflare__ZoneTimeBucket__Counter methods */ 52 | void cloudflare__zone_time_bucket__counter__init 53 | (Cloudflare__ZoneTimeBucket__Counter *message); 54 | /* Cloudflare__ZoneTimeBucket methods */ 55 | void cloudflare__zone_time_bucket__init 56 | (Cloudflare__ZoneTimeBucket *message); 57 | size_t cloudflare__zone_time_bucket__get_packed_size 58 | (const Cloudflare__ZoneTimeBucket *message); 59 | size_t cloudflare__zone_time_bucket__pack 60 | (const Cloudflare__ZoneTimeBucket *message, 61 | uint8_t *out); 62 | size_t cloudflare__zone_time_bucket__pack_to_buffer 63 | (const Cloudflare__ZoneTimeBucket *message, 64 | ProtobufCBuffer *buffer); 65 | Cloudflare__ZoneTimeBucket * 66 | cloudflare__zone_time_bucket__unpack 67 | (ProtobufCAllocator *allocator, 68 | size_t len, 69 | const uint8_t *data); 70 | void cloudflare__zone_time_bucket__free_unpacked 71 | (Cloudflare__ZoneTimeBucket *message, 72 | ProtobufCAllocator *allocator); 73 | /* --- per-message closures --- */ 74 | 75 | typedef void (*Cloudflare__ZoneTimeBucket__Counter_Closure) 76 | (const Cloudflare__ZoneTimeBucket__Counter *message, 77 | void *closure_data); 78 | typedef void (*Cloudflare__ZoneTimeBucket_Closure) 79 | (const Cloudflare__ZoneTimeBucket *message, 80 | void *closure_data); 81 | 82 | /* --- services --- */ 83 | 84 | 85 | /* --- descriptors --- */ 86 | 87 | extern const ProtobufCMessageDescriptor cloudflare__zone_time_bucket__descriptor; 88 | extern const ProtobufCMessageDescriptor cloudflare__zone_time_bucket__counter__descriptor; 89 | 90 | PROTOBUF_C_END_DECLS 91 | 92 | 93 | #endif /* PROTOBUF_entry_2eproto__INCLUDED */ 94 | -------------------------------------------------------------------------------- /common.h: -------------------------------------------------------------------------------- 1 | #ifndef CF_COMMON_KCPG 2 | #define CF_COMMON_KCPG 3 | 4 | #include 5 | 6 | static const char *BASE_DB_DIR = "/mnt/ssd/pg/kyoto"; 7 | static const char *DB_TYPE = "kct"; 8 | static const mode_t NEW_DIR_MODE = 0777; 9 | static const mode_t NEW_DIR_MASK = 0777; 10 | static const mode_t FILE_MODE = 0666; 11 | static const char *CF_LABEL_SEP = ":"; 12 | static const int MAX_KEYS_BEFORE_KV_MAP = 512; // How many keys to keep as a counter, before placing them in KC. 13 | //static const int MAX_KEYS_BEFORE_KV_MAP = 1; // How many keys to keep as a counter, before placing them in KC. 14 | static const int KC_MAX_ENTRIES_PER_RID = 64; 15 | 16 | // Where to stick kv dbs. 17 | static const char *CF_KV_MAP_BASE = "/mnt/ssd_new/pg/kyoto_kvs"; 18 | 19 | #endif 20 | -------------------------------------------------------------------------------- /cpp/entry.pb.cc: -------------------------------------------------------------------------------- 1 | // Generated by the protocol buffer compiler. DO NOT EDIT! 2 | 3 | #define INTERNAL_SUPPRESS_PROTOBUF_FIELD_DEPRECATION 4 | #include "entry.pb.h" 5 | #include 6 | #include 7 | #include 8 | #include 9 | #include 10 | #include 11 | // @@protoc_insertion_point(includes) 12 | 13 | namespace cloudflare { 14 | 15 | namespace { 16 | 17 | const ::google::protobuf::Descriptor* ZoneTimeBucket_descriptor_ = NULL; 18 | const ::google::protobuf::internal::GeneratedMessageReflection* 19 | ZoneTimeBucket_reflection_ = NULL; 20 | const ::google::protobuf::Descriptor* ZoneTimeBucket_Counter_descriptor_ = NULL; 21 | const ::google::protobuf::internal::GeneratedMessageReflection* 22 | ZoneTimeBucket_Counter_reflection_ = NULL; 23 | 24 | } // namespace 25 | 26 | 27 | void protobuf_AssignDesc_entry_2eproto() { 28 | protobuf_AddDesc_entry_2eproto(); 29 | const ::google::protobuf::FileDescriptor* file = 30 | ::google::protobuf::DescriptorPool::generated_pool()->FindFileByName( 31 | "entry.proto"); 32 | GOOGLE_CHECK(file != NULL); 33 | ZoneTimeBucket_descriptor_ = file->message_type(0); 34 | static const int ZoneTimeBucket_offsets_[10] = { 35 | GOOGLE_PROTOBUF_GENERATED_MESSAGE_FIELD_OFFSET(ZoneTimeBucket, map_name_), 36 | GOOGLE_PROTOBUF_GENERATED_MESSAGE_FIELD_OFFSET(ZoneTimeBucket, classification_), 37 | GOOGLE_PROTOBUF_GENERATED_MESSAGE_FIELD_OFFSET(ZoneTimeBucket, doctype_), 38 | GOOGLE_PROTOBUF_GENERATED_MESSAGE_FIELD_OFFSET(ZoneTimeBucket, pop_), 39 | GOOGLE_PROTOBUF_GENERATED_MESSAGE_FIELD_OFFSET(ZoneTimeBucket, psource_), 40 | GOOGLE_PROTOBUF_GENERATED_MESSAGE_FIELD_OFFSET(ZoneTimeBucket, result_id_), 41 | GOOGLE_PROTOBUF_GENERATED_MESSAGE_FIELD_OFFSET(ZoneTimeBucket, db_key_), 42 | GOOGLE_PROTOBUF_GENERATED_MESSAGE_FIELD_OFFSET(ZoneTimeBucket, db_path_), 43 | GOOGLE_PROTOBUF_GENERATED_MESSAGE_FIELD_OFFSET(ZoneTimeBucket, kv_map_file_), 44 | GOOGLE_PROTOBUF_GENERATED_MESSAGE_FIELD_OFFSET(ZoneTimeBucket, map_entry_), 45 | }; 46 | ZoneTimeBucket_reflection_ = 47 | new ::google::protobuf::internal::GeneratedMessageReflection( 48 | ZoneTimeBucket_descriptor_, 49 | ZoneTimeBucket::default_instance_, 50 | ZoneTimeBucket_offsets_, 51 | GOOGLE_PROTOBUF_GENERATED_MESSAGE_FIELD_OFFSET(ZoneTimeBucket, _has_bits_[0]), 52 | GOOGLE_PROTOBUF_GENERATED_MESSAGE_FIELD_OFFSET(ZoneTimeBucket, _unknown_fields_), 53 | -1, 54 | ::google::protobuf::DescriptorPool::generated_pool(), 55 | ::google::protobuf::MessageFactory::generated_factory(), 56 | sizeof(ZoneTimeBucket)); 57 | ZoneTimeBucket_Counter_descriptor_ = ZoneTimeBucket_descriptor_->nested_type(0); 58 | static const int ZoneTimeBucket_Counter_offsets_[2] = { 59 | GOOGLE_PROTOBUF_GENERATED_MESSAGE_FIELD_OFFSET(ZoneTimeBucket_Counter, key_), 60 | GOOGLE_PROTOBUF_GENERATED_MESSAGE_FIELD_OFFSET(ZoneTimeBucket_Counter, value_), 61 | }; 62 | ZoneTimeBucket_Counter_reflection_ = 63 | new ::google::protobuf::internal::GeneratedMessageReflection( 64 | ZoneTimeBucket_Counter_descriptor_, 65 | ZoneTimeBucket_Counter::default_instance_, 66 | ZoneTimeBucket_Counter_offsets_, 67 | GOOGLE_PROTOBUF_GENERATED_MESSAGE_FIELD_OFFSET(ZoneTimeBucket_Counter, _has_bits_[0]), 68 | GOOGLE_PROTOBUF_GENERATED_MESSAGE_FIELD_OFFSET(ZoneTimeBucket_Counter, _unknown_fields_), 69 | -1, 70 | ::google::protobuf::DescriptorPool::generated_pool(), 71 | ::google::protobuf::MessageFactory::generated_factory(), 72 | sizeof(ZoneTimeBucket_Counter)); 73 | } 74 | 75 | namespace { 76 | 77 | GOOGLE_PROTOBUF_DECLARE_ONCE(protobuf_AssignDescriptors_once_); 78 | inline void protobuf_AssignDescriptorsOnce() { 79 | ::google::protobuf::GoogleOnceInit(&protobuf_AssignDescriptors_once_, 80 | &protobuf_AssignDesc_entry_2eproto); 81 | } 82 | 83 | void protobuf_RegisterTypes(const ::std::string&) { 84 | protobuf_AssignDescriptorsOnce(); 85 | ::google::protobuf::MessageFactory::InternalRegisterGeneratedMessage( 86 | ZoneTimeBucket_descriptor_, &ZoneTimeBucket::default_instance()); 87 | ::google::protobuf::MessageFactory::InternalRegisterGeneratedMessage( 88 | ZoneTimeBucket_Counter_descriptor_, &ZoneTimeBucket_Counter::default_instance()); 89 | } 90 | 91 | } // namespace 92 | 93 | void protobuf_ShutdownFile_entry_2eproto() { 94 | delete ZoneTimeBucket::default_instance_; 95 | delete ZoneTimeBucket_reflection_; 96 | delete ZoneTimeBucket_Counter::default_instance_; 97 | delete ZoneTimeBucket_Counter_reflection_; 98 | } 99 | 100 | void protobuf_AddDesc_entry_2eproto() { 101 | static bool already_here = false; 102 | if (already_here) return; 103 | already_here = true; 104 | GOOGLE_PROTOBUF_VERIFY_VERSION; 105 | 106 | ::google::protobuf::DescriptorPool::InternalAddGeneratedFile( 107 | "\n\013entry.proto\022\ncloudflare\"\220\002\n\016ZoneTimeBu" 108 | "cket\022\020\n\010map_name\030\001 \002(\t\022\026\n\016classification" 109 | "\030\002 \002(\t\022\017\n\007doctype\030\003 \002(\t\022\013\n\003pop\030\004 \002(\t\022\017\n\007" 110 | "psource\030\005 \002(\t\022\021\n\tresult_id\030\006 \002(\t\022\016\n\006db_k" 111 | "ey\030\007 \002(\t\022\017\n\007db_path\030\010 \002(\t\022\023\n\013kv_map_file" 112 | "\030\t \001(\t\0225\n\tmap_entry\030\n \003(\0132\".cloudflare.Z" 113 | "oneTimeBucket.Counter\032%\n\007Counter\022\013\n\003key\030" 114 | "\001 \002(\t\022\r\n\005value\030\002 \002(\003", 300); 115 | ::google::protobuf::MessageFactory::InternalRegisterGeneratedFile( 116 | "entry.proto", &protobuf_RegisterTypes); 117 | ZoneTimeBucket::default_instance_ = new ZoneTimeBucket(); 118 | ZoneTimeBucket_Counter::default_instance_ = new ZoneTimeBucket_Counter(); 119 | ZoneTimeBucket::default_instance_->InitAsDefaultInstance(); 120 | ZoneTimeBucket_Counter::default_instance_->InitAsDefaultInstance(); 121 | ::google::protobuf::internal::OnShutdown(&protobuf_ShutdownFile_entry_2eproto); 122 | } 123 | 124 | // Force AddDescriptors() to be called at static initialization time. 125 | struct StaticDescriptorInitializer_entry_2eproto { 126 | StaticDescriptorInitializer_entry_2eproto() { 127 | protobuf_AddDesc_entry_2eproto(); 128 | } 129 | } static_descriptor_initializer_entry_2eproto_; 130 | 131 | 132 | // =================================================================== 133 | 134 | const ::std::string ZoneTimeBucket_Counter::_default_key_; 135 | #ifndef _MSC_VER 136 | const int ZoneTimeBucket_Counter::kKeyFieldNumber; 137 | const int ZoneTimeBucket_Counter::kValueFieldNumber; 138 | #endif // !_MSC_VER 139 | 140 | ZoneTimeBucket_Counter::ZoneTimeBucket_Counter() 141 | : ::google::protobuf::Message() { 142 | SharedCtor(); 143 | } 144 | 145 | void ZoneTimeBucket_Counter::InitAsDefaultInstance() { 146 | } 147 | 148 | ZoneTimeBucket_Counter::ZoneTimeBucket_Counter(const ZoneTimeBucket_Counter& from) 149 | : ::google::protobuf::Message() { 150 | SharedCtor(); 151 | MergeFrom(from); 152 | } 153 | 154 | void ZoneTimeBucket_Counter::SharedCtor() { 155 | _cached_size_ = 0; 156 | key_ = const_cast< ::std::string*>(&_default_key_); 157 | value_ = GOOGLE_LONGLONG(0); 158 | ::memset(_has_bits_, 0, sizeof(_has_bits_)); 159 | } 160 | 161 | ZoneTimeBucket_Counter::~ZoneTimeBucket_Counter() { 162 | SharedDtor(); 163 | } 164 | 165 | void ZoneTimeBucket_Counter::SharedDtor() { 166 | if (key_ != &_default_key_) { 167 | delete key_; 168 | } 169 | if (this != default_instance_) { 170 | } 171 | } 172 | 173 | void ZoneTimeBucket_Counter::SetCachedSize(int size) const { 174 | GOOGLE_SAFE_CONCURRENT_WRITES_BEGIN(); 175 | _cached_size_ = size; 176 | GOOGLE_SAFE_CONCURRENT_WRITES_END(); 177 | } 178 | const ::google::protobuf::Descriptor* ZoneTimeBucket_Counter::descriptor() { 179 | protobuf_AssignDescriptorsOnce(); 180 | return ZoneTimeBucket_Counter_descriptor_; 181 | } 182 | 183 | const ZoneTimeBucket_Counter& ZoneTimeBucket_Counter::default_instance() { 184 | if (default_instance_ == NULL) protobuf_AddDesc_entry_2eproto(); return *default_instance_; 185 | } 186 | 187 | ZoneTimeBucket_Counter* ZoneTimeBucket_Counter::default_instance_ = NULL; 188 | 189 | ZoneTimeBucket_Counter* ZoneTimeBucket_Counter::New() const { 190 | return new ZoneTimeBucket_Counter; 191 | } 192 | 193 | void ZoneTimeBucket_Counter::Clear() { 194 | if (_has_bits_[0 / 32] & (0xffu << (0 % 32))) { 195 | if (_has_bit(0)) { 196 | if (key_ != &_default_key_) { 197 | key_->clear(); 198 | } 199 | } 200 | value_ = GOOGLE_LONGLONG(0); 201 | } 202 | ::memset(_has_bits_, 0, sizeof(_has_bits_)); 203 | mutable_unknown_fields()->Clear(); 204 | } 205 | 206 | bool ZoneTimeBucket_Counter::MergePartialFromCodedStream( 207 | ::google::protobuf::io::CodedInputStream* input) { 208 | #define DO_(EXPRESSION) if (!(EXPRESSION)) return false 209 | ::google::protobuf::uint32 tag; 210 | while ((tag = input->ReadTag()) != 0) { 211 | switch (::google::protobuf::internal::WireFormatLite::GetTagFieldNumber(tag)) { 212 | // required string key = 1; 213 | case 1: { 214 | if (::google::protobuf::internal::WireFormatLite::GetTagWireType(tag) == 215 | ::google::protobuf::internal::WireFormatLite::WIRETYPE_LENGTH_DELIMITED) { 216 | DO_(::google::protobuf::internal::WireFormatLite::ReadString( 217 | input, this->mutable_key())); 218 | ::google::protobuf::internal::WireFormat::VerifyUTF8String( 219 | this->key().data(), this->key().length(), 220 | ::google::protobuf::internal::WireFormat::PARSE); 221 | } else { 222 | goto handle_uninterpreted; 223 | } 224 | if (input->ExpectTag(16)) goto parse_value; 225 | break; 226 | } 227 | 228 | // required int64 value = 2; 229 | case 2: { 230 | if (::google::protobuf::internal::WireFormatLite::GetTagWireType(tag) == 231 | ::google::protobuf::internal::WireFormatLite::WIRETYPE_VARINT) { 232 | parse_value: 233 | DO_((::google::protobuf::internal::WireFormatLite::ReadPrimitive< 234 | ::google::protobuf::int64, ::google::protobuf::internal::WireFormatLite::TYPE_INT64>( 235 | input, &value_))); 236 | _set_bit(1); 237 | } else { 238 | goto handle_uninterpreted; 239 | } 240 | if (input->ExpectAtEnd()) return true; 241 | break; 242 | } 243 | 244 | default: { 245 | handle_uninterpreted: 246 | if (::google::protobuf::internal::WireFormatLite::GetTagWireType(tag) == 247 | ::google::protobuf::internal::WireFormatLite::WIRETYPE_END_GROUP) { 248 | return true; 249 | } 250 | DO_(::google::protobuf::internal::WireFormat::SkipField( 251 | input, tag, mutable_unknown_fields())); 252 | break; 253 | } 254 | } 255 | } 256 | return true; 257 | #undef DO_ 258 | } 259 | 260 | void ZoneTimeBucket_Counter::SerializeWithCachedSizes( 261 | ::google::protobuf::io::CodedOutputStream* output) const { 262 | // required string key = 1; 263 | if (_has_bit(0)) { 264 | ::google::protobuf::internal::WireFormat::VerifyUTF8String( 265 | this->key().data(), this->key().length(), 266 | ::google::protobuf::internal::WireFormat::SERIALIZE); 267 | ::google::protobuf::internal::WireFormatLite::WriteString( 268 | 1, this->key(), output); 269 | } 270 | 271 | // required int64 value = 2; 272 | if (_has_bit(1)) { 273 | ::google::protobuf::internal::WireFormatLite::WriteInt64(2, this->value(), output); 274 | } 275 | 276 | if (!unknown_fields().empty()) { 277 | ::google::protobuf::internal::WireFormat::SerializeUnknownFields( 278 | unknown_fields(), output); 279 | } 280 | } 281 | 282 | ::google::protobuf::uint8* ZoneTimeBucket_Counter::SerializeWithCachedSizesToArray( 283 | ::google::protobuf::uint8* target) const { 284 | // required string key = 1; 285 | if (_has_bit(0)) { 286 | ::google::protobuf::internal::WireFormat::VerifyUTF8String( 287 | this->key().data(), this->key().length(), 288 | ::google::protobuf::internal::WireFormat::SERIALIZE); 289 | target = 290 | ::google::protobuf::internal::WireFormatLite::WriteStringToArray( 291 | 1, this->key(), target); 292 | } 293 | 294 | // required int64 value = 2; 295 | if (_has_bit(1)) { 296 | target = ::google::protobuf::internal::WireFormatLite::WriteInt64ToArray(2, this->value(), target); 297 | } 298 | 299 | if (!unknown_fields().empty()) { 300 | target = ::google::protobuf::internal::WireFormat::SerializeUnknownFieldsToArray( 301 | unknown_fields(), target); 302 | } 303 | return target; 304 | } 305 | 306 | int ZoneTimeBucket_Counter::ByteSize() const { 307 | int total_size = 0; 308 | 309 | if (_has_bits_[0 / 32] & (0xffu << (0 % 32))) { 310 | // required string key = 1; 311 | if (has_key()) { 312 | total_size += 1 + 313 | ::google::protobuf::internal::WireFormatLite::StringSize( 314 | this->key()); 315 | } 316 | 317 | // required int64 value = 2; 318 | if (has_value()) { 319 | total_size += 1 + 320 | ::google::protobuf::internal::WireFormatLite::Int64Size( 321 | this->value()); 322 | } 323 | 324 | } 325 | if (!unknown_fields().empty()) { 326 | total_size += 327 | ::google::protobuf::internal::WireFormat::ComputeUnknownFieldsSize( 328 | unknown_fields()); 329 | } 330 | GOOGLE_SAFE_CONCURRENT_WRITES_BEGIN(); 331 | _cached_size_ = total_size; 332 | GOOGLE_SAFE_CONCURRENT_WRITES_END(); 333 | return total_size; 334 | } 335 | 336 | void ZoneTimeBucket_Counter::MergeFrom(const ::google::protobuf::Message& from) { 337 | GOOGLE_CHECK_NE(&from, this); 338 | const ZoneTimeBucket_Counter* source = 339 | ::google::protobuf::internal::dynamic_cast_if_available( 340 | &from); 341 | if (source == NULL) { 342 | ::google::protobuf::internal::ReflectionOps::Merge(from, this); 343 | } else { 344 | MergeFrom(*source); 345 | } 346 | } 347 | 348 | void ZoneTimeBucket_Counter::MergeFrom(const ZoneTimeBucket_Counter& from) { 349 | GOOGLE_CHECK_NE(&from, this); 350 | if (from._has_bits_[0 / 32] & (0xffu << (0 % 32))) { 351 | if (from._has_bit(0)) { 352 | set_key(from.key()); 353 | } 354 | if (from._has_bit(1)) { 355 | set_value(from.value()); 356 | } 357 | } 358 | mutable_unknown_fields()->MergeFrom(from.unknown_fields()); 359 | } 360 | 361 | void ZoneTimeBucket_Counter::CopyFrom(const ::google::protobuf::Message& from) { 362 | if (&from == this) return; 363 | Clear(); 364 | MergeFrom(from); 365 | } 366 | 367 | void ZoneTimeBucket_Counter::CopyFrom(const ZoneTimeBucket_Counter& from) { 368 | if (&from == this) return; 369 | Clear(); 370 | MergeFrom(from); 371 | } 372 | 373 | bool ZoneTimeBucket_Counter::IsInitialized() const { 374 | if ((_has_bits_[0] & 0x00000003) != 0x00000003) return false; 375 | 376 | return true; 377 | } 378 | 379 | void ZoneTimeBucket_Counter::Swap(ZoneTimeBucket_Counter* other) { 380 | if (other != this) { 381 | std::swap(key_, other->key_); 382 | std::swap(value_, other->value_); 383 | std::swap(_has_bits_[0], other->_has_bits_[0]); 384 | _unknown_fields_.Swap(&other->_unknown_fields_); 385 | std::swap(_cached_size_, other->_cached_size_); 386 | } 387 | } 388 | 389 | ::google::protobuf::Metadata ZoneTimeBucket_Counter::GetMetadata() const { 390 | protobuf_AssignDescriptorsOnce(); 391 | ::google::protobuf::Metadata metadata; 392 | metadata.descriptor = ZoneTimeBucket_Counter_descriptor_; 393 | metadata.reflection = ZoneTimeBucket_Counter_reflection_; 394 | return metadata; 395 | } 396 | 397 | 398 | // ------------------------------------------------------------------- 399 | 400 | const ::std::string ZoneTimeBucket::_default_map_name_; 401 | const ::std::string ZoneTimeBucket::_default_classification_; 402 | const ::std::string ZoneTimeBucket::_default_doctype_; 403 | const ::std::string ZoneTimeBucket::_default_pop_; 404 | const ::std::string ZoneTimeBucket::_default_psource_; 405 | const ::std::string ZoneTimeBucket::_default_result_id_; 406 | const ::std::string ZoneTimeBucket::_default_db_key_; 407 | const ::std::string ZoneTimeBucket::_default_db_path_; 408 | const ::std::string ZoneTimeBucket::_default_kv_map_file_; 409 | #ifndef _MSC_VER 410 | const int ZoneTimeBucket::kMapNameFieldNumber; 411 | const int ZoneTimeBucket::kClassificationFieldNumber; 412 | const int ZoneTimeBucket::kDoctypeFieldNumber; 413 | const int ZoneTimeBucket::kPopFieldNumber; 414 | const int ZoneTimeBucket::kPsourceFieldNumber; 415 | const int ZoneTimeBucket::kResultIdFieldNumber; 416 | const int ZoneTimeBucket::kDbKeyFieldNumber; 417 | const int ZoneTimeBucket::kDbPathFieldNumber; 418 | const int ZoneTimeBucket::kKvMapFileFieldNumber; 419 | const int ZoneTimeBucket::kMapEntryFieldNumber; 420 | #endif // !_MSC_VER 421 | 422 | ZoneTimeBucket::ZoneTimeBucket() 423 | : ::google::protobuf::Message() { 424 | SharedCtor(); 425 | } 426 | 427 | void ZoneTimeBucket::InitAsDefaultInstance() { 428 | } 429 | 430 | ZoneTimeBucket::ZoneTimeBucket(const ZoneTimeBucket& from) 431 | : ::google::protobuf::Message() { 432 | SharedCtor(); 433 | MergeFrom(from); 434 | } 435 | 436 | void ZoneTimeBucket::SharedCtor() { 437 | _cached_size_ = 0; 438 | map_name_ = const_cast< ::std::string*>(&_default_map_name_); 439 | classification_ = const_cast< ::std::string*>(&_default_classification_); 440 | doctype_ = const_cast< ::std::string*>(&_default_doctype_); 441 | pop_ = const_cast< ::std::string*>(&_default_pop_); 442 | psource_ = const_cast< ::std::string*>(&_default_psource_); 443 | result_id_ = const_cast< ::std::string*>(&_default_result_id_); 444 | db_key_ = const_cast< ::std::string*>(&_default_db_key_); 445 | db_path_ = const_cast< ::std::string*>(&_default_db_path_); 446 | kv_map_file_ = const_cast< ::std::string*>(&_default_kv_map_file_); 447 | ::memset(_has_bits_, 0, sizeof(_has_bits_)); 448 | } 449 | 450 | ZoneTimeBucket::~ZoneTimeBucket() { 451 | SharedDtor(); 452 | } 453 | 454 | void ZoneTimeBucket::SharedDtor() { 455 | if (map_name_ != &_default_map_name_) { 456 | delete map_name_; 457 | } 458 | if (classification_ != &_default_classification_) { 459 | delete classification_; 460 | } 461 | if (doctype_ != &_default_doctype_) { 462 | delete doctype_; 463 | } 464 | if (pop_ != &_default_pop_) { 465 | delete pop_; 466 | } 467 | if (psource_ != &_default_psource_) { 468 | delete psource_; 469 | } 470 | if (result_id_ != &_default_result_id_) { 471 | delete result_id_; 472 | } 473 | if (db_key_ != &_default_db_key_) { 474 | delete db_key_; 475 | } 476 | if (db_path_ != &_default_db_path_) { 477 | delete db_path_; 478 | } 479 | if (kv_map_file_ != &_default_kv_map_file_) { 480 | delete kv_map_file_; 481 | } 482 | if (this != default_instance_) { 483 | } 484 | } 485 | 486 | void ZoneTimeBucket::SetCachedSize(int size) const { 487 | GOOGLE_SAFE_CONCURRENT_WRITES_BEGIN(); 488 | _cached_size_ = size; 489 | GOOGLE_SAFE_CONCURRENT_WRITES_END(); 490 | } 491 | const ::google::protobuf::Descriptor* ZoneTimeBucket::descriptor() { 492 | protobuf_AssignDescriptorsOnce(); 493 | return ZoneTimeBucket_descriptor_; 494 | } 495 | 496 | const ZoneTimeBucket& ZoneTimeBucket::default_instance() { 497 | if (default_instance_ == NULL) protobuf_AddDesc_entry_2eproto(); return *default_instance_; 498 | } 499 | 500 | ZoneTimeBucket* ZoneTimeBucket::default_instance_ = NULL; 501 | 502 | ZoneTimeBucket* ZoneTimeBucket::New() const { 503 | return new ZoneTimeBucket; 504 | } 505 | 506 | void ZoneTimeBucket::Clear() { 507 | if (_has_bits_[0 / 32] & (0xffu << (0 % 32))) { 508 | if (_has_bit(0)) { 509 | if (map_name_ != &_default_map_name_) { 510 | map_name_->clear(); 511 | } 512 | } 513 | if (_has_bit(1)) { 514 | if (classification_ != &_default_classification_) { 515 | classification_->clear(); 516 | } 517 | } 518 | if (_has_bit(2)) { 519 | if (doctype_ != &_default_doctype_) { 520 | doctype_->clear(); 521 | } 522 | } 523 | if (_has_bit(3)) { 524 | if (pop_ != &_default_pop_) { 525 | pop_->clear(); 526 | } 527 | } 528 | if (_has_bit(4)) { 529 | if (psource_ != &_default_psource_) { 530 | psource_->clear(); 531 | } 532 | } 533 | if (_has_bit(5)) { 534 | if (result_id_ != &_default_result_id_) { 535 | result_id_->clear(); 536 | } 537 | } 538 | if (_has_bit(6)) { 539 | if (db_key_ != &_default_db_key_) { 540 | db_key_->clear(); 541 | } 542 | } 543 | if (_has_bit(7)) { 544 | if (db_path_ != &_default_db_path_) { 545 | db_path_->clear(); 546 | } 547 | } 548 | } 549 | if (_has_bits_[8 / 32] & (0xffu << (8 % 32))) { 550 | if (_has_bit(8)) { 551 | if (kv_map_file_ != &_default_kv_map_file_) { 552 | kv_map_file_->clear(); 553 | } 554 | } 555 | } 556 | map_entry_.Clear(); 557 | ::memset(_has_bits_, 0, sizeof(_has_bits_)); 558 | mutable_unknown_fields()->Clear(); 559 | } 560 | 561 | bool ZoneTimeBucket::MergePartialFromCodedStream( 562 | ::google::protobuf::io::CodedInputStream* input) { 563 | #define DO_(EXPRESSION) if (!(EXPRESSION)) return false 564 | ::google::protobuf::uint32 tag; 565 | while ((tag = input->ReadTag()) != 0) { 566 | switch (::google::protobuf::internal::WireFormatLite::GetTagFieldNumber(tag)) { 567 | // required string map_name = 1; 568 | case 1: { 569 | if (::google::protobuf::internal::WireFormatLite::GetTagWireType(tag) == 570 | ::google::protobuf::internal::WireFormatLite::WIRETYPE_LENGTH_DELIMITED) { 571 | DO_(::google::protobuf::internal::WireFormatLite::ReadString( 572 | input, this->mutable_map_name())); 573 | ::google::protobuf::internal::WireFormat::VerifyUTF8String( 574 | this->map_name().data(), this->map_name().length(), 575 | ::google::protobuf::internal::WireFormat::PARSE); 576 | } else { 577 | goto handle_uninterpreted; 578 | } 579 | if (input->ExpectTag(18)) goto parse_classification; 580 | break; 581 | } 582 | 583 | // required string classification = 2; 584 | case 2: { 585 | if (::google::protobuf::internal::WireFormatLite::GetTagWireType(tag) == 586 | ::google::protobuf::internal::WireFormatLite::WIRETYPE_LENGTH_DELIMITED) { 587 | parse_classification: 588 | DO_(::google::protobuf::internal::WireFormatLite::ReadString( 589 | input, this->mutable_classification())); 590 | ::google::protobuf::internal::WireFormat::VerifyUTF8String( 591 | this->classification().data(), this->classification().length(), 592 | ::google::protobuf::internal::WireFormat::PARSE); 593 | } else { 594 | goto handle_uninterpreted; 595 | } 596 | if (input->ExpectTag(26)) goto parse_doctype; 597 | break; 598 | } 599 | 600 | // required string doctype = 3; 601 | case 3: { 602 | if (::google::protobuf::internal::WireFormatLite::GetTagWireType(tag) == 603 | ::google::protobuf::internal::WireFormatLite::WIRETYPE_LENGTH_DELIMITED) { 604 | parse_doctype: 605 | DO_(::google::protobuf::internal::WireFormatLite::ReadString( 606 | input, this->mutable_doctype())); 607 | ::google::protobuf::internal::WireFormat::VerifyUTF8String( 608 | this->doctype().data(), this->doctype().length(), 609 | ::google::protobuf::internal::WireFormat::PARSE); 610 | } else { 611 | goto handle_uninterpreted; 612 | } 613 | if (input->ExpectTag(34)) goto parse_pop; 614 | break; 615 | } 616 | 617 | // required string pop = 4; 618 | case 4: { 619 | if (::google::protobuf::internal::WireFormatLite::GetTagWireType(tag) == 620 | ::google::protobuf::internal::WireFormatLite::WIRETYPE_LENGTH_DELIMITED) { 621 | parse_pop: 622 | DO_(::google::protobuf::internal::WireFormatLite::ReadString( 623 | input, this->mutable_pop())); 624 | ::google::protobuf::internal::WireFormat::VerifyUTF8String( 625 | this->pop().data(), this->pop().length(), 626 | ::google::protobuf::internal::WireFormat::PARSE); 627 | } else { 628 | goto handle_uninterpreted; 629 | } 630 | if (input->ExpectTag(42)) goto parse_psource; 631 | break; 632 | } 633 | 634 | // required string psource = 5; 635 | case 5: { 636 | if (::google::protobuf::internal::WireFormatLite::GetTagWireType(tag) == 637 | ::google::protobuf::internal::WireFormatLite::WIRETYPE_LENGTH_DELIMITED) { 638 | parse_psource: 639 | DO_(::google::protobuf::internal::WireFormatLite::ReadString( 640 | input, this->mutable_psource())); 641 | ::google::protobuf::internal::WireFormat::VerifyUTF8String( 642 | this->psource().data(), this->psource().length(), 643 | ::google::protobuf::internal::WireFormat::PARSE); 644 | } else { 645 | goto handle_uninterpreted; 646 | } 647 | if (input->ExpectTag(50)) goto parse_result_id; 648 | break; 649 | } 650 | 651 | // required string result_id = 6; 652 | case 6: { 653 | if (::google::protobuf::internal::WireFormatLite::GetTagWireType(tag) == 654 | ::google::protobuf::internal::WireFormatLite::WIRETYPE_LENGTH_DELIMITED) { 655 | parse_result_id: 656 | DO_(::google::protobuf::internal::WireFormatLite::ReadString( 657 | input, this->mutable_result_id())); 658 | ::google::protobuf::internal::WireFormat::VerifyUTF8String( 659 | this->result_id().data(), this->result_id().length(), 660 | ::google::protobuf::internal::WireFormat::PARSE); 661 | } else { 662 | goto handle_uninterpreted; 663 | } 664 | if (input->ExpectTag(58)) goto parse_db_key; 665 | break; 666 | } 667 | 668 | // required string db_key = 7; 669 | case 7: { 670 | if (::google::protobuf::internal::WireFormatLite::GetTagWireType(tag) == 671 | ::google::protobuf::internal::WireFormatLite::WIRETYPE_LENGTH_DELIMITED) { 672 | parse_db_key: 673 | DO_(::google::protobuf::internal::WireFormatLite::ReadString( 674 | input, this->mutable_db_key())); 675 | ::google::protobuf::internal::WireFormat::VerifyUTF8String( 676 | this->db_key().data(), this->db_key().length(), 677 | ::google::protobuf::internal::WireFormat::PARSE); 678 | } else { 679 | goto handle_uninterpreted; 680 | } 681 | if (input->ExpectTag(66)) goto parse_db_path; 682 | break; 683 | } 684 | 685 | // required string db_path = 8; 686 | case 8: { 687 | if (::google::protobuf::internal::WireFormatLite::GetTagWireType(tag) == 688 | ::google::protobuf::internal::WireFormatLite::WIRETYPE_LENGTH_DELIMITED) { 689 | parse_db_path: 690 | DO_(::google::protobuf::internal::WireFormatLite::ReadString( 691 | input, this->mutable_db_path())); 692 | ::google::protobuf::internal::WireFormat::VerifyUTF8String( 693 | this->db_path().data(), this->db_path().length(), 694 | ::google::protobuf::internal::WireFormat::PARSE); 695 | } else { 696 | goto handle_uninterpreted; 697 | } 698 | if (input->ExpectTag(74)) goto parse_kv_map_file; 699 | break; 700 | } 701 | 702 | // optional string kv_map_file = 9; 703 | case 9: { 704 | if (::google::protobuf::internal::WireFormatLite::GetTagWireType(tag) == 705 | ::google::protobuf::internal::WireFormatLite::WIRETYPE_LENGTH_DELIMITED) { 706 | parse_kv_map_file: 707 | DO_(::google::protobuf::internal::WireFormatLite::ReadString( 708 | input, this->mutable_kv_map_file())); 709 | ::google::protobuf::internal::WireFormat::VerifyUTF8String( 710 | this->kv_map_file().data(), this->kv_map_file().length(), 711 | ::google::protobuf::internal::WireFormat::PARSE); 712 | } else { 713 | goto handle_uninterpreted; 714 | } 715 | if (input->ExpectTag(82)) goto parse_map_entry; 716 | break; 717 | } 718 | 719 | // repeated .cloudflare.ZoneTimeBucket.Counter map_entry = 10; 720 | case 10: { 721 | if (::google::protobuf::internal::WireFormatLite::GetTagWireType(tag) == 722 | ::google::protobuf::internal::WireFormatLite::WIRETYPE_LENGTH_DELIMITED) { 723 | parse_map_entry: 724 | DO_(::google::protobuf::internal::WireFormatLite::ReadMessageNoVirtual( 725 | input, add_map_entry())); 726 | } else { 727 | goto handle_uninterpreted; 728 | } 729 | if (input->ExpectTag(82)) goto parse_map_entry; 730 | if (input->ExpectAtEnd()) return true; 731 | break; 732 | } 733 | 734 | default: { 735 | handle_uninterpreted: 736 | if (::google::protobuf::internal::WireFormatLite::GetTagWireType(tag) == 737 | ::google::protobuf::internal::WireFormatLite::WIRETYPE_END_GROUP) { 738 | return true; 739 | } 740 | DO_(::google::protobuf::internal::WireFormat::SkipField( 741 | input, tag, mutable_unknown_fields())); 742 | break; 743 | } 744 | } 745 | } 746 | return true; 747 | #undef DO_ 748 | } 749 | 750 | void ZoneTimeBucket::SerializeWithCachedSizes( 751 | ::google::protobuf::io::CodedOutputStream* output) const { 752 | // required string map_name = 1; 753 | if (_has_bit(0)) { 754 | ::google::protobuf::internal::WireFormat::VerifyUTF8String( 755 | this->map_name().data(), this->map_name().length(), 756 | ::google::protobuf::internal::WireFormat::SERIALIZE); 757 | ::google::protobuf::internal::WireFormatLite::WriteString( 758 | 1, this->map_name(), output); 759 | } 760 | 761 | // required string classification = 2; 762 | if (_has_bit(1)) { 763 | ::google::protobuf::internal::WireFormat::VerifyUTF8String( 764 | this->classification().data(), this->classification().length(), 765 | ::google::protobuf::internal::WireFormat::SERIALIZE); 766 | ::google::protobuf::internal::WireFormatLite::WriteString( 767 | 2, this->classification(), output); 768 | } 769 | 770 | // required string doctype = 3; 771 | if (_has_bit(2)) { 772 | ::google::protobuf::internal::WireFormat::VerifyUTF8String( 773 | this->doctype().data(), this->doctype().length(), 774 | ::google::protobuf::internal::WireFormat::SERIALIZE); 775 | ::google::protobuf::internal::WireFormatLite::WriteString( 776 | 3, this->doctype(), output); 777 | } 778 | 779 | // required string pop = 4; 780 | if (_has_bit(3)) { 781 | ::google::protobuf::internal::WireFormat::VerifyUTF8String( 782 | this->pop().data(), this->pop().length(), 783 | ::google::protobuf::internal::WireFormat::SERIALIZE); 784 | ::google::protobuf::internal::WireFormatLite::WriteString( 785 | 4, this->pop(), output); 786 | } 787 | 788 | // required string psource = 5; 789 | if (_has_bit(4)) { 790 | ::google::protobuf::internal::WireFormat::VerifyUTF8String( 791 | this->psource().data(), this->psource().length(), 792 | ::google::protobuf::internal::WireFormat::SERIALIZE); 793 | ::google::protobuf::internal::WireFormatLite::WriteString( 794 | 5, this->psource(), output); 795 | } 796 | 797 | // required string result_id = 6; 798 | if (_has_bit(5)) { 799 | ::google::protobuf::internal::WireFormat::VerifyUTF8String( 800 | this->result_id().data(), this->result_id().length(), 801 | ::google::protobuf::internal::WireFormat::SERIALIZE); 802 | ::google::protobuf::internal::WireFormatLite::WriteString( 803 | 6, this->result_id(), output); 804 | } 805 | 806 | // required string db_key = 7; 807 | if (_has_bit(6)) { 808 | ::google::protobuf::internal::WireFormat::VerifyUTF8String( 809 | this->db_key().data(), this->db_key().length(), 810 | ::google::protobuf::internal::WireFormat::SERIALIZE); 811 | ::google::protobuf::internal::WireFormatLite::WriteString( 812 | 7, this->db_key(), output); 813 | } 814 | 815 | // required string db_path = 8; 816 | if (_has_bit(7)) { 817 | ::google::protobuf::internal::WireFormat::VerifyUTF8String( 818 | this->db_path().data(), this->db_path().length(), 819 | ::google::protobuf::internal::WireFormat::SERIALIZE); 820 | ::google::protobuf::internal::WireFormatLite::WriteString( 821 | 8, this->db_path(), output); 822 | } 823 | 824 | // optional string kv_map_file = 9; 825 | if (_has_bit(8)) { 826 | ::google::protobuf::internal::WireFormat::VerifyUTF8String( 827 | this->kv_map_file().data(), this->kv_map_file().length(), 828 | ::google::protobuf::internal::WireFormat::SERIALIZE); 829 | ::google::protobuf::internal::WireFormatLite::WriteString( 830 | 9, this->kv_map_file(), output); 831 | } 832 | 833 | // repeated .cloudflare.ZoneTimeBucket.Counter map_entry = 10; 834 | for (int i = 0; i < this->map_entry_size(); i++) { 835 | ::google::protobuf::internal::WireFormatLite::WriteMessageMaybeToArray( 836 | 10, this->map_entry(i), output); 837 | } 838 | 839 | if (!unknown_fields().empty()) { 840 | ::google::protobuf::internal::WireFormat::SerializeUnknownFields( 841 | unknown_fields(), output); 842 | } 843 | } 844 | 845 | ::google::protobuf::uint8* ZoneTimeBucket::SerializeWithCachedSizesToArray( 846 | ::google::protobuf::uint8* target) const { 847 | // required string map_name = 1; 848 | if (_has_bit(0)) { 849 | ::google::protobuf::internal::WireFormat::VerifyUTF8String( 850 | this->map_name().data(), this->map_name().length(), 851 | ::google::protobuf::internal::WireFormat::SERIALIZE); 852 | target = 853 | ::google::protobuf::internal::WireFormatLite::WriteStringToArray( 854 | 1, this->map_name(), target); 855 | } 856 | 857 | // required string classification = 2; 858 | if (_has_bit(1)) { 859 | ::google::protobuf::internal::WireFormat::VerifyUTF8String( 860 | this->classification().data(), this->classification().length(), 861 | ::google::protobuf::internal::WireFormat::SERIALIZE); 862 | target = 863 | ::google::protobuf::internal::WireFormatLite::WriteStringToArray( 864 | 2, this->classification(), target); 865 | } 866 | 867 | // required string doctype = 3; 868 | if (_has_bit(2)) { 869 | ::google::protobuf::internal::WireFormat::VerifyUTF8String( 870 | this->doctype().data(), this->doctype().length(), 871 | ::google::protobuf::internal::WireFormat::SERIALIZE); 872 | target = 873 | ::google::protobuf::internal::WireFormatLite::WriteStringToArray( 874 | 3, this->doctype(), target); 875 | } 876 | 877 | // required string pop = 4; 878 | if (_has_bit(3)) { 879 | ::google::protobuf::internal::WireFormat::VerifyUTF8String( 880 | this->pop().data(), this->pop().length(), 881 | ::google::protobuf::internal::WireFormat::SERIALIZE); 882 | target = 883 | ::google::protobuf::internal::WireFormatLite::WriteStringToArray( 884 | 4, this->pop(), target); 885 | } 886 | 887 | // required string psource = 5; 888 | if (_has_bit(4)) { 889 | ::google::protobuf::internal::WireFormat::VerifyUTF8String( 890 | this->psource().data(), this->psource().length(), 891 | ::google::protobuf::internal::WireFormat::SERIALIZE); 892 | target = 893 | ::google::protobuf::internal::WireFormatLite::WriteStringToArray( 894 | 5, this->psource(), target); 895 | } 896 | 897 | // required string result_id = 6; 898 | if (_has_bit(5)) { 899 | ::google::protobuf::internal::WireFormat::VerifyUTF8String( 900 | this->result_id().data(), this->result_id().length(), 901 | ::google::protobuf::internal::WireFormat::SERIALIZE); 902 | target = 903 | ::google::protobuf::internal::WireFormatLite::WriteStringToArray( 904 | 6, this->result_id(), target); 905 | } 906 | 907 | // required string db_key = 7; 908 | if (_has_bit(6)) { 909 | ::google::protobuf::internal::WireFormat::VerifyUTF8String( 910 | this->db_key().data(), this->db_key().length(), 911 | ::google::protobuf::internal::WireFormat::SERIALIZE); 912 | target = 913 | ::google::protobuf::internal::WireFormatLite::WriteStringToArray( 914 | 7, this->db_key(), target); 915 | } 916 | 917 | // required string db_path = 8; 918 | if (_has_bit(7)) { 919 | ::google::protobuf::internal::WireFormat::VerifyUTF8String( 920 | this->db_path().data(), this->db_path().length(), 921 | ::google::protobuf::internal::WireFormat::SERIALIZE); 922 | target = 923 | ::google::protobuf::internal::WireFormatLite::WriteStringToArray( 924 | 8, this->db_path(), target); 925 | } 926 | 927 | // optional string kv_map_file = 9; 928 | if (_has_bit(8)) { 929 | ::google::protobuf::internal::WireFormat::VerifyUTF8String( 930 | this->kv_map_file().data(), this->kv_map_file().length(), 931 | ::google::protobuf::internal::WireFormat::SERIALIZE); 932 | target = 933 | ::google::protobuf::internal::WireFormatLite::WriteStringToArray( 934 | 9, this->kv_map_file(), target); 935 | } 936 | 937 | // repeated .cloudflare.ZoneTimeBucket.Counter map_entry = 10; 938 | for (int i = 0; i < this->map_entry_size(); i++) { 939 | target = ::google::protobuf::internal::WireFormatLite:: 940 | WriteMessageNoVirtualToArray( 941 | 10, this->map_entry(i), target); 942 | } 943 | 944 | if (!unknown_fields().empty()) { 945 | target = ::google::protobuf::internal::WireFormat::SerializeUnknownFieldsToArray( 946 | unknown_fields(), target); 947 | } 948 | return target; 949 | } 950 | 951 | int ZoneTimeBucket::ByteSize() const { 952 | int total_size = 0; 953 | 954 | if (_has_bits_[0 / 32] & (0xffu << (0 % 32))) { 955 | // required string map_name = 1; 956 | if (has_map_name()) { 957 | total_size += 1 + 958 | ::google::protobuf::internal::WireFormatLite::StringSize( 959 | this->map_name()); 960 | } 961 | 962 | // required string classification = 2; 963 | if (has_classification()) { 964 | total_size += 1 + 965 | ::google::protobuf::internal::WireFormatLite::StringSize( 966 | this->classification()); 967 | } 968 | 969 | // required string doctype = 3; 970 | if (has_doctype()) { 971 | total_size += 1 + 972 | ::google::protobuf::internal::WireFormatLite::StringSize( 973 | this->doctype()); 974 | } 975 | 976 | // required string pop = 4; 977 | if (has_pop()) { 978 | total_size += 1 + 979 | ::google::protobuf::internal::WireFormatLite::StringSize( 980 | this->pop()); 981 | } 982 | 983 | // required string psource = 5; 984 | if (has_psource()) { 985 | total_size += 1 + 986 | ::google::protobuf::internal::WireFormatLite::StringSize( 987 | this->psource()); 988 | } 989 | 990 | // required string result_id = 6; 991 | if (has_result_id()) { 992 | total_size += 1 + 993 | ::google::protobuf::internal::WireFormatLite::StringSize( 994 | this->result_id()); 995 | } 996 | 997 | // required string db_key = 7; 998 | if (has_db_key()) { 999 | total_size += 1 + 1000 | ::google::protobuf::internal::WireFormatLite::StringSize( 1001 | this->db_key()); 1002 | } 1003 | 1004 | // required string db_path = 8; 1005 | if (has_db_path()) { 1006 | total_size += 1 + 1007 | ::google::protobuf::internal::WireFormatLite::StringSize( 1008 | this->db_path()); 1009 | } 1010 | 1011 | } 1012 | if (_has_bits_[8 / 32] & (0xffu << (8 % 32))) { 1013 | // optional string kv_map_file = 9; 1014 | if (has_kv_map_file()) { 1015 | total_size += 1 + 1016 | ::google::protobuf::internal::WireFormatLite::StringSize( 1017 | this->kv_map_file()); 1018 | } 1019 | 1020 | } 1021 | // repeated .cloudflare.ZoneTimeBucket.Counter map_entry = 10; 1022 | total_size += 1 * this->map_entry_size(); 1023 | for (int i = 0; i < this->map_entry_size(); i++) { 1024 | total_size += 1025 | ::google::protobuf::internal::WireFormatLite::MessageSizeNoVirtual( 1026 | this->map_entry(i)); 1027 | } 1028 | 1029 | if (!unknown_fields().empty()) { 1030 | total_size += 1031 | ::google::protobuf::internal::WireFormat::ComputeUnknownFieldsSize( 1032 | unknown_fields()); 1033 | } 1034 | GOOGLE_SAFE_CONCURRENT_WRITES_BEGIN(); 1035 | _cached_size_ = total_size; 1036 | GOOGLE_SAFE_CONCURRENT_WRITES_END(); 1037 | return total_size; 1038 | } 1039 | 1040 | void ZoneTimeBucket::MergeFrom(const ::google::protobuf::Message& from) { 1041 | GOOGLE_CHECK_NE(&from, this); 1042 | const ZoneTimeBucket* source = 1043 | ::google::protobuf::internal::dynamic_cast_if_available( 1044 | &from); 1045 | if (source == NULL) { 1046 | ::google::protobuf::internal::ReflectionOps::Merge(from, this); 1047 | } else { 1048 | MergeFrom(*source); 1049 | } 1050 | } 1051 | 1052 | void ZoneTimeBucket::MergeFrom(const ZoneTimeBucket& from) { 1053 | GOOGLE_CHECK_NE(&from, this); 1054 | map_entry_.MergeFrom(from.map_entry_); 1055 | if (from._has_bits_[0 / 32] & (0xffu << (0 % 32))) { 1056 | if (from._has_bit(0)) { 1057 | set_map_name(from.map_name()); 1058 | } 1059 | if (from._has_bit(1)) { 1060 | set_classification(from.classification()); 1061 | } 1062 | if (from._has_bit(2)) { 1063 | set_doctype(from.doctype()); 1064 | } 1065 | if (from._has_bit(3)) { 1066 | set_pop(from.pop()); 1067 | } 1068 | if (from._has_bit(4)) { 1069 | set_psource(from.psource()); 1070 | } 1071 | if (from._has_bit(5)) { 1072 | set_result_id(from.result_id()); 1073 | } 1074 | if (from._has_bit(6)) { 1075 | set_db_key(from.db_key()); 1076 | } 1077 | if (from._has_bit(7)) { 1078 | set_db_path(from.db_path()); 1079 | } 1080 | } 1081 | if (from._has_bits_[8 / 32] & (0xffu << (8 % 32))) { 1082 | if (from._has_bit(8)) { 1083 | set_kv_map_file(from.kv_map_file()); 1084 | } 1085 | } 1086 | mutable_unknown_fields()->MergeFrom(from.unknown_fields()); 1087 | } 1088 | 1089 | void ZoneTimeBucket::CopyFrom(const ::google::protobuf::Message& from) { 1090 | if (&from == this) return; 1091 | Clear(); 1092 | MergeFrom(from); 1093 | } 1094 | 1095 | void ZoneTimeBucket::CopyFrom(const ZoneTimeBucket& from) { 1096 | if (&from == this) return; 1097 | Clear(); 1098 | MergeFrom(from); 1099 | } 1100 | 1101 | bool ZoneTimeBucket::IsInitialized() const { 1102 | if ((_has_bits_[0] & 0x000000ff) != 0x000000ff) return false; 1103 | 1104 | for (int i = 0; i < map_entry_size(); i++) { 1105 | if (!this->map_entry(i).IsInitialized()) return false; 1106 | } 1107 | return true; 1108 | } 1109 | 1110 | void ZoneTimeBucket::Swap(ZoneTimeBucket* other) { 1111 | if (other != this) { 1112 | std::swap(map_name_, other->map_name_); 1113 | std::swap(classification_, other->classification_); 1114 | std::swap(doctype_, other->doctype_); 1115 | std::swap(pop_, other->pop_); 1116 | std::swap(psource_, other->psource_); 1117 | std::swap(result_id_, other->result_id_); 1118 | std::swap(db_key_, other->db_key_); 1119 | std::swap(db_path_, other->db_path_); 1120 | std::swap(kv_map_file_, other->kv_map_file_); 1121 | map_entry_.Swap(&other->map_entry_); 1122 | std::swap(_has_bits_[0], other->_has_bits_[0]); 1123 | _unknown_fields_.Swap(&other->_unknown_fields_); 1124 | std::swap(_cached_size_, other->_cached_size_); 1125 | } 1126 | } 1127 | 1128 | ::google::protobuf::Metadata ZoneTimeBucket::GetMetadata() const { 1129 | protobuf_AssignDescriptorsOnce(); 1130 | ::google::protobuf::Metadata metadata; 1131 | metadata.descriptor = ZoneTimeBucket_descriptor_; 1132 | metadata.reflection = ZoneTimeBucket_reflection_; 1133 | return metadata; 1134 | } 1135 | 1136 | 1137 | // @@protoc_insertion_point(namespace_scope) 1138 | 1139 | } // namespace cloudflare 1140 | 1141 | // @@protoc_insertion_point(global_scope) 1142 | -------------------------------------------------------------------------------- /cpp/entry.pb.h: -------------------------------------------------------------------------------- 1 | // Generated by the protocol buffer compiler. DO NOT EDIT! 2 | // source: entry.proto 3 | 4 | #ifndef PROTOBUF_entry_2eproto__INCLUDED 5 | #define PROTOBUF_entry_2eproto__INCLUDED 6 | 7 | #include 8 | 9 | #include 10 | 11 | #if GOOGLE_PROTOBUF_VERSION < 2003000 12 | #error This file was generated by a newer version of protoc which is 13 | #error incompatible with your Protocol Buffer headers. Please update 14 | #error your headers. 15 | #endif 16 | #if 2003000 < GOOGLE_PROTOBUF_MIN_PROTOC_VERSION 17 | #error This file was generated by an older version of protoc which is 18 | #error incompatible with your Protocol Buffer headers. Please 19 | #error regenerate this file with a newer version of protoc. 20 | #endif 21 | 22 | #include 23 | #include 24 | #include 25 | #include 26 | // @@protoc_insertion_point(includes) 27 | 28 | namespace cloudflare { 29 | 30 | // Internal implementation detail -- do not call these. 31 | void protobuf_AddDesc_entry_2eproto(); 32 | void protobuf_AssignDesc_entry_2eproto(); 33 | void protobuf_ShutdownFile_entry_2eproto(); 34 | 35 | class ZoneTimeBucket; 36 | class ZoneTimeBucket_Counter; 37 | 38 | // =================================================================== 39 | 40 | class ZoneTimeBucket_Counter : public ::google::protobuf::Message { 41 | public: 42 | ZoneTimeBucket_Counter(); 43 | virtual ~ZoneTimeBucket_Counter(); 44 | 45 | ZoneTimeBucket_Counter(const ZoneTimeBucket_Counter& from); 46 | 47 | inline ZoneTimeBucket_Counter& operator=(const ZoneTimeBucket_Counter& from) { 48 | CopyFrom(from); 49 | return *this; 50 | } 51 | 52 | inline const ::google::protobuf::UnknownFieldSet& unknown_fields() const { 53 | return _unknown_fields_; 54 | } 55 | 56 | inline ::google::protobuf::UnknownFieldSet* mutable_unknown_fields() { 57 | return &_unknown_fields_; 58 | } 59 | 60 | static const ::google::protobuf::Descriptor* descriptor(); 61 | static const ZoneTimeBucket_Counter& default_instance(); 62 | 63 | void Swap(ZoneTimeBucket_Counter* other); 64 | 65 | // implements Message ---------------------------------------------- 66 | 67 | ZoneTimeBucket_Counter* New() const; 68 | void CopyFrom(const ::google::protobuf::Message& from); 69 | void MergeFrom(const ::google::protobuf::Message& from); 70 | void CopyFrom(const ZoneTimeBucket_Counter& from); 71 | void MergeFrom(const ZoneTimeBucket_Counter& from); 72 | void Clear(); 73 | bool IsInitialized() const; 74 | 75 | int ByteSize() const; 76 | bool MergePartialFromCodedStream( 77 | ::google::protobuf::io::CodedInputStream* input); 78 | void SerializeWithCachedSizes( 79 | ::google::protobuf::io::CodedOutputStream* output) const; 80 | ::google::protobuf::uint8* SerializeWithCachedSizesToArray(::google::protobuf::uint8* output) const; 81 | int GetCachedSize() const { return _cached_size_; } 82 | private: 83 | void SharedCtor(); 84 | void SharedDtor(); 85 | void SetCachedSize(int size) const; 86 | public: 87 | 88 | ::google::protobuf::Metadata GetMetadata() const; 89 | 90 | // nested types ---------------------------------------------------- 91 | 92 | // accessors ------------------------------------------------------- 93 | 94 | // required string key = 1; 95 | inline bool has_key() const; 96 | inline void clear_key(); 97 | static const int kKeyFieldNumber = 1; 98 | inline const ::std::string& key() const; 99 | inline void set_key(const ::std::string& value); 100 | inline void set_key(const char* value); 101 | inline void set_key(const char* value, size_t size); 102 | inline ::std::string* mutable_key(); 103 | 104 | // required int64 value = 2; 105 | inline bool has_value() const; 106 | inline void clear_value(); 107 | static const int kValueFieldNumber = 2; 108 | inline ::google::protobuf::int64 value() const; 109 | inline void set_value(::google::protobuf::int64 value); 110 | 111 | // @@protoc_insertion_point(class_scope:cloudflare.ZoneTimeBucket.Counter) 112 | private: 113 | ::google::protobuf::UnknownFieldSet _unknown_fields_; 114 | mutable int _cached_size_; 115 | 116 | ::std::string* key_; 117 | static const ::std::string _default_key_; 118 | ::google::protobuf::int64 value_; 119 | friend void protobuf_AddDesc_entry_2eproto(); 120 | friend void protobuf_AssignDesc_entry_2eproto(); 121 | friend void protobuf_ShutdownFile_entry_2eproto(); 122 | 123 | ::google::protobuf::uint32 _has_bits_[(2 + 31) / 32]; 124 | 125 | // WHY DOES & HAVE LOWER PRECEDENCE THAN != !? 126 | inline bool _has_bit(int index) const { 127 | return (_has_bits_[index / 32] & (1u << (index % 32))) != 0; 128 | } 129 | inline void _set_bit(int index) { 130 | _has_bits_[index / 32] |= (1u << (index % 32)); 131 | } 132 | inline void _clear_bit(int index) { 133 | _has_bits_[index / 32] &= ~(1u << (index % 32)); 134 | } 135 | 136 | void InitAsDefaultInstance(); 137 | static ZoneTimeBucket_Counter* default_instance_; 138 | }; 139 | // ------------------------------------------------------------------- 140 | 141 | class ZoneTimeBucket : public ::google::protobuf::Message { 142 | public: 143 | ZoneTimeBucket(); 144 | virtual ~ZoneTimeBucket(); 145 | 146 | ZoneTimeBucket(const ZoneTimeBucket& from); 147 | 148 | inline ZoneTimeBucket& operator=(const ZoneTimeBucket& from) { 149 | CopyFrom(from); 150 | return *this; 151 | } 152 | 153 | inline const ::google::protobuf::UnknownFieldSet& unknown_fields() const { 154 | return _unknown_fields_; 155 | } 156 | 157 | inline ::google::protobuf::UnknownFieldSet* mutable_unknown_fields() { 158 | return &_unknown_fields_; 159 | } 160 | 161 | static const ::google::protobuf::Descriptor* descriptor(); 162 | static const ZoneTimeBucket& default_instance(); 163 | 164 | void Swap(ZoneTimeBucket* other); 165 | 166 | // implements Message ---------------------------------------------- 167 | 168 | ZoneTimeBucket* New() const; 169 | void CopyFrom(const ::google::protobuf::Message& from); 170 | void MergeFrom(const ::google::protobuf::Message& from); 171 | void CopyFrom(const ZoneTimeBucket& from); 172 | void MergeFrom(const ZoneTimeBucket& from); 173 | void Clear(); 174 | bool IsInitialized() const; 175 | 176 | int ByteSize() const; 177 | bool MergePartialFromCodedStream( 178 | ::google::protobuf::io::CodedInputStream* input); 179 | void SerializeWithCachedSizes( 180 | ::google::protobuf::io::CodedOutputStream* output) const; 181 | ::google::protobuf::uint8* SerializeWithCachedSizesToArray(::google::protobuf::uint8* output) const; 182 | int GetCachedSize() const { return _cached_size_; } 183 | private: 184 | void SharedCtor(); 185 | void SharedDtor(); 186 | void SetCachedSize(int size) const; 187 | public: 188 | 189 | ::google::protobuf::Metadata GetMetadata() const; 190 | 191 | // nested types ---------------------------------------------------- 192 | 193 | typedef ZoneTimeBucket_Counter Counter; 194 | 195 | // accessors ------------------------------------------------------- 196 | 197 | // required string map_name = 1; 198 | inline bool has_map_name() const; 199 | inline void clear_map_name(); 200 | static const int kMapNameFieldNumber = 1; 201 | inline const ::std::string& map_name() const; 202 | inline void set_map_name(const ::std::string& value); 203 | inline void set_map_name(const char* value); 204 | inline void set_map_name(const char* value, size_t size); 205 | inline ::std::string* mutable_map_name(); 206 | 207 | // required string classification = 2; 208 | inline bool has_classification() const; 209 | inline void clear_classification(); 210 | static const int kClassificationFieldNumber = 2; 211 | inline const ::std::string& classification() const; 212 | inline void set_classification(const ::std::string& value); 213 | inline void set_classification(const char* value); 214 | inline void set_classification(const char* value, size_t size); 215 | inline ::std::string* mutable_classification(); 216 | 217 | // required string doctype = 3; 218 | inline bool has_doctype() const; 219 | inline void clear_doctype(); 220 | static const int kDoctypeFieldNumber = 3; 221 | inline const ::std::string& doctype() const; 222 | inline void set_doctype(const ::std::string& value); 223 | inline void set_doctype(const char* value); 224 | inline void set_doctype(const char* value, size_t size); 225 | inline ::std::string* mutable_doctype(); 226 | 227 | // required string pop = 4; 228 | inline bool has_pop() const; 229 | inline void clear_pop(); 230 | static const int kPopFieldNumber = 4; 231 | inline const ::std::string& pop() const; 232 | inline void set_pop(const ::std::string& value); 233 | inline void set_pop(const char* value); 234 | inline void set_pop(const char* value, size_t size); 235 | inline ::std::string* mutable_pop(); 236 | 237 | // required string psource = 5; 238 | inline bool has_psource() const; 239 | inline void clear_psource(); 240 | static const int kPsourceFieldNumber = 5; 241 | inline const ::std::string& psource() const; 242 | inline void set_psource(const ::std::string& value); 243 | inline void set_psource(const char* value); 244 | inline void set_psource(const char* value, size_t size); 245 | inline ::std::string* mutable_psource(); 246 | 247 | // required string result_id = 6; 248 | inline bool has_result_id() const; 249 | inline void clear_result_id(); 250 | static const int kResultIdFieldNumber = 6; 251 | inline const ::std::string& result_id() const; 252 | inline void set_result_id(const ::std::string& value); 253 | inline void set_result_id(const char* value); 254 | inline void set_result_id(const char* value, size_t size); 255 | inline ::std::string* mutable_result_id(); 256 | 257 | // required string db_key = 7; 258 | inline bool has_db_key() const; 259 | inline void clear_db_key(); 260 | static const int kDbKeyFieldNumber = 7; 261 | inline const ::std::string& db_key() const; 262 | inline void set_db_key(const ::std::string& value); 263 | inline void set_db_key(const char* value); 264 | inline void set_db_key(const char* value, size_t size); 265 | inline ::std::string* mutable_db_key(); 266 | 267 | // required string db_path = 8; 268 | inline bool has_db_path() const; 269 | inline void clear_db_path(); 270 | static const int kDbPathFieldNumber = 8; 271 | inline const ::std::string& db_path() const; 272 | inline void set_db_path(const ::std::string& value); 273 | inline void set_db_path(const char* value); 274 | inline void set_db_path(const char* value, size_t size); 275 | inline ::std::string* mutable_db_path(); 276 | 277 | // optional string kv_map_file = 9; 278 | inline bool has_kv_map_file() const; 279 | inline void clear_kv_map_file(); 280 | static const int kKvMapFileFieldNumber = 9; 281 | inline const ::std::string& kv_map_file() const; 282 | inline void set_kv_map_file(const ::std::string& value); 283 | inline void set_kv_map_file(const char* value); 284 | inline void set_kv_map_file(const char* value, size_t size); 285 | inline ::std::string* mutable_kv_map_file(); 286 | 287 | // repeated .cloudflare.ZoneTimeBucket.Counter map_entry = 10; 288 | inline int map_entry_size() const; 289 | inline void clear_map_entry(); 290 | static const int kMapEntryFieldNumber = 10; 291 | inline const ::cloudflare::ZoneTimeBucket_Counter& map_entry(int index) const; 292 | inline ::cloudflare::ZoneTimeBucket_Counter* mutable_map_entry(int index); 293 | inline ::cloudflare::ZoneTimeBucket_Counter* add_map_entry(); 294 | inline const ::google::protobuf::RepeatedPtrField< ::cloudflare::ZoneTimeBucket_Counter >& 295 | map_entry() const; 296 | inline ::google::protobuf::RepeatedPtrField< ::cloudflare::ZoneTimeBucket_Counter >* 297 | mutable_map_entry(); 298 | 299 | // @@protoc_insertion_point(class_scope:cloudflare.ZoneTimeBucket) 300 | private: 301 | ::google::protobuf::UnknownFieldSet _unknown_fields_; 302 | mutable int _cached_size_; 303 | 304 | ::std::string* map_name_; 305 | static const ::std::string _default_map_name_; 306 | ::std::string* classification_; 307 | static const ::std::string _default_classification_; 308 | ::std::string* doctype_; 309 | static const ::std::string _default_doctype_; 310 | ::std::string* pop_; 311 | static const ::std::string _default_pop_; 312 | ::std::string* psource_; 313 | static const ::std::string _default_psource_; 314 | ::std::string* result_id_; 315 | static const ::std::string _default_result_id_; 316 | ::std::string* db_key_; 317 | static const ::std::string _default_db_key_; 318 | ::std::string* db_path_; 319 | static const ::std::string _default_db_path_; 320 | ::std::string* kv_map_file_; 321 | static const ::std::string _default_kv_map_file_; 322 | ::google::protobuf::RepeatedPtrField< ::cloudflare::ZoneTimeBucket_Counter > map_entry_; 323 | friend void protobuf_AddDesc_entry_2eproto(); 324 | friend void protobuf_AssignDesc_entry_2eproto(); 325 | friend void protobuf_ShutdownFile_entry_2eproto(); 326 | 327 | ::google::protobuf::uint32 _has_bits_[(10 + 31) / 32]; 328 | 329 | // WHY DOES & HAVE LOWER PRECEDENCE THAN != !? 330 | inline bool _has_bit(int index) const { 331 | return (_has_bits_[index / 32] & (1u << (index % 32))) != 0; 332 | } 333 | inline void _set_bit(int index) { 334 | _has_bits_[index / 32] |= (1u << (index % 32)); 335 | } 336 | inline void _clear_bit(int index) { 337 | _has_bits_[index / 32] &= ~(1u << (index % 32)); 338 | } 339 | 340 | void InitAsDefaultInstance(); 341 | static ZoneTimeBucket* default_instance_; 342 | }; 343 | // =================================================================== 344 | 345 | 346 | // =================================================================== 347 | 348 | // ZoneTimeBucket_Counter 349 | 350 | // required string key = 1; 351 | inline bool ZoneTimeBucket_Counter::has_key() const { 352 | return _has_bit(0); 353 | } 354 | inline void ZoneTimeBucket_Counter::clear_key() { 355 | if (key_ != &_default_key_) { 356 | key_->clear(); 357 | } 358 | _clear_bit(0); 359 | } 360 | inline const ::std::string& ZoneTimeBucket_Counter::key() const { 361 | return *key_; 362 | } 363 | inline void ZoneTimeBucket_Counter::set_key(const ::std::string& value) { 364 | _set_bit(0); 365 | if (key_ == &_default_key_) { 366 | key_ = new ::std::string; 367 | } 368 | key_->assign(value); 369 | } 370 | inline void ZoneTimeBucket_Counter::set_key(const char* value) { 371 | _set_bit(0); 372 | if (key_ == &_default_key_) { 373 | key_ = new ::std::string; 374 | } 375 | key_->assign(value); 376 | } 377 | inline void ZoneTimeBucket_Counter::set_key(const char* value, size_t size) { 378 | _set_bit(0); 379 | if (key_ == &_default_key_) { 380 | key_ = new ::std::string; 381 | } 382 | key_->assign(reinterpret_cast(value), size); 383 | } 384 | inline ::std::string* ZoneTimeBucket_Counter::mutable_key() { 385 | _set_bit(0); 386 | if (key_ == &_default_key_) { 387 | key_ = new ::std::string; 388 | } 389 | return key_; 390 | } 391 | 392 | // required int64 value = 2; 393 | inline bool ZoneTimeBucket_Counter::has_value() const { 394 | return _has_bit(1); 395 | } 396 | inline void ZoneTimeBucket_Counter::clear_value() { 397 | value_ = GOOGLE_LONGLONG(0); 398 | _clear_bit(1); 399 | } 400 | inline ::google::protobuf::int64 ZoneTimeBucket_Counter::value() const { 401 | return value_; 402 | } 403 | inline void ZoneTimeBucket_Counter::set_value(::google::protobuf::int64 value) { 404 | _set_bit(1); 405 | value_ = value; 406 | } 407 | 408 | // ------------------------------------------------------------------- 409 | 410 | // ZoneTimeBucket 411 | 412 | // required string map_name = 1; 413 | inline bool ZoneTimeBucket::has_map_name() const { 414 | return _has_bit(0); 415 | } 416 | inline void ZoneTimeBucket::clear_map_name() { 417 | if (map_name_ != &_default_map_name_) { 418 | map_name_->clear(); 419 | } 420 | _clear_bit(0); 421 | } 422 | inline const ::std::string& ZoneTimeBucket::map_name() const { 423 | return *map_name_; 424 | } 425 | inline void ZoneTimeBucket::set_map_name(const ::std::string& value) { 426 | _set_bit(0); 427 | if (map_name_ == &_default_map_name_) { 428 | map_name_ = new ::std::string; 429 | } 430 | map_name_->assign(value); 431 | } 432 | inline void ZoneTimeBucket::set_map_name(const char* value) { 433 | _set_bit(0); 434 | if (map_name_ == &_default_map_name_) { 435 | map_name_ = new ::std::string; 436 | } 437 | map_name_->assign(value); 438 | } 439 | inline void ZoneTimeBucket::set_map_name(const char* value, size_t size) { 440 | _set_bit(0); 441 | if (map_name_ == &_default_map_name_) { 442 | map_name_ = new ::std::string; 443 | } 444 | map_name_->assign(reinterpret_cast(value), size); 445 | } 446 | inline ::std::string* ZoneTimeBucket::mutable_map_name() { 447 | _set_bit(0); 448 | if (map_name_ == &_default_map_name_) { 449 | map_name_ = new ::std::string; 450 | } 451 | return map_name_; 452 | } 453 | 454 | // required string classification = 2; 455 | inline bool ZoneTimeBucket::has_classification() const { 456 | return _has_bit(1); 457 | } 458 | inline void ZoneTimeBucket::clear_classification() { 459 | if (classification_ != &_default_classification_) { 460 | classification_->clear(); 461 | } 462 | _clear_bit(1); 463 | } 464 | inline const ::std::string& ZoneTimeBucket::classification() const { 465 | return *classification_; 466 | } 467 | inline void ZoneTimeBucket::set_classification(const ::std::string& value) { 468 | _set_bit(1); 469 | if (classification_ == &_default_classification_) { 470 | classification_ = new ::std::string; 471 | } 472 | classification_->assign(value); 473 | } 474 | inline void ZoneTimeBucket::set_classification(const char* value) { 475 | _set_bit(1); 476 | if (classification_ == &_default_classification_) { 477 | classification_ = new ::std::string; 478 | } 479 | classification_->assign(value); 480 | } 481 | inline void ZoneTimeBucket::set_classification(const char* value, size_t size) { 482 | _set_bit(1); 483 | if (classification_ == &_default_classification_) { 484 | classification_ = new ::std::string; 485 | } 486 | classification_->assign(reinterpret_cast(value), size); 487 | } 488 | inline ::std::string* ZoneTimeBucket::mutable_classification() { 489 | _set_bit(1); 490 | if (classification_ == &_default_classification_) { 491 | classification_ = new ::std::string; 492 | } 493 | return classification_; 494 | } 495 | 496 | // required string doctype = 3; 497 | inline bool ZoneTimeBucket::has_doctype() const { 498 | return _has_bit(2); 499 | } 500 | inline void ZoneTimeBucket::clear_doctype() { 501 | if (doctype_ != &_default_doctype_) { 502 | doctype_->clear(); 503 | } 504 | _clear_bit(2); 505 | } 506 | inline const ::std::string& ZoneTimeBucket::doctype() const { 507 | return *doctype_; 508 | } 509 | inline void ZoneTimeBucket::set_doctype(const ::std::string& value) { 510 | _set_bit(2); 511 | if (doctype_ == &_default_doctype_) { 512 | doctype_ = new ::std::string; 513 | } 514 | doctype_->assign(value); 515 | } 516 | inline void ZoneTimeBucket::set_doctype(const char* value) { 517 | _set_bit(2); 518 | if (doctype_ == &_default_doctype_) { 519 | doctype_ = new ::std::string; 520 | } 521 | doctype_->assign(value); 522 | } 523 | inline void ZoneTimeBucket::set_doctype(const char* value, size_t size) { 524 | _set_bit(2); 525 | if (doctype_ == &_default_doctype_) { 526 | doctype_ = new ::std::string; 527 | } 528 | doctype_->assign(reinterpret_cast(value), size); 529 | } 530 | inline ::std::string* ZoneTimeBucket::mutable_doctype() { 531 | _set_bit(2); 532 | if (doctype_ == &_default_doctype_) { 533 | doctype_ = new ::std::string; 534 | } 535 | return doctype_; 536 | } 537 | 538 | // required string pop = 4; 539 | inline bool ZoneTimeBucket::has_pop() const { 540 | return _has_bit(3); 541 | } 542 | inline void ZoneTimeBucket::clear_pop() { 543 | if (pop_ != &_default_pop_) { 544 | pop_->clear(); 545 | } 546 | _clear_bit(3); 547 | } 548 | inline const ::std::string& ZoneTimeBucket::pop() const { 549 | return *pop_; 550 | } 551 | inline void ZoneTimeBucket::set_pop(const ::std::string& value) { 552 | _set_bit(3); 553 | if (pop_ == &_default_pop_) { 554 | pop_ = new ::std::string; 555 | } 556 | pop_->assign(value); 557 | } 558 | inline void ZoneTimeBucket::set_pop(const char* value) { 559 | _set_bit(3); 560 | if (pop_ == &_default_pop_) { 561 | pop_ = new ::std::string; 562 | } 563 | pop_->assign(value); 564 | } 565 | inline void ZoneTimeBucket::set_pop(const char* value, size_t size) { 566 | _set_bit(3); 567 | if (pop_ == &_default_pop_) { 568 | pop_ = new ::std::string; 569 | } 570 | pop_->assign(reinterpret_cast(value), size); 571 | } 572 | inline ::std::string* ZoneTimeBucket::mutable_pop() { 573 | _set_bit(3); 574 | if (pop_ == &_default_pop_) { 575 | pop_ = new ::std::string; 576 | } 577 | return pop_; 578 | } 579 | 580 | // required string psource = 5; 581 | inline bool ZoneTimeBucket::has_psource() const { 582 | return _has_bit(4); 583 | } 584 | inline void ZoneTimeBucket::clear_psource() { 585 | if (psource_ != &_default_psource_) { 586 | psource_->clear(); 587 | } 588 | _clear_bit(4); 589 | } 590 | inline const ::std::string& ZoneTimeBucket::psource() const { 591 | return *psource_; 592 | } 593 | inline void ZoneTimeBucket::set_psource(const ::std::string& value) { 594 | _set_bit(4); 595 | if (psource_ == &_default_psource_) { 596 | psource_ = new ::std::string; 597 | } 598 | psource_->assign(value); 599 | } 600 | inline void ZoneTimeBucket::set_psource(const char* value) { 601 | _set_bit(4); 602 | if (psource_ == &_default_psource_) { 603 | psource_ = new ::std::string; 604 | } 605 | psource_->assign(value); 606 | } 607 | inline void ZoneTimeBucket::set_psource(const char* value, size_t size) { 608 | _set_bit(4); 609 | if (psource_ == &_default_psource_) { 610 | psource_ = new ::std::string; 611 | } 612 | psource_->assign(reinterpret_cast(value), size); 613 | } 614 | inline ::std::string* ZoneTimeBucket::mutable_psource() { 615 | _set_bit(4); 616 | if (psource_ == &_default_psource_) { 617 | psource_ = new ::std::string; 618 | } 619 | return psource_; 620 | } 621 | 622 | // required string result_id = 6; 623 | inline bool ZoneTimeBucket::has_result_id() const { 624 | return _has_bit(5); 625 | } 626 | inline void ZoneTimeBucket::clear_result_id() { 627 | if (result_id_ != &_default_result_id_) { 628 | result_id_->clear(); 629 | } 630 | _clear_bit(5); 631 | } 632 | inline const ::std::string& ZoneTimeBucket::result_id() const { 633 | return *result_id_; 634 | } 635 | inline void ZoneTimeBucket::set_result_id(const ::std::string& value) { 636 | _set_bit(5); 637 | if (result_id_ == &_default_result_id_) { 638 | result_id_ = new ::std::string; 639 | } 640 | result_id_->assign(value); 641 | } 642 | inline void ZoneTimeBucket::set_result_id(const char* value) { 643 | _set_bit(5); 644 | if (result_id_ == &_default_result_id_) { 645 | result_id_ = new ::std::string; 646 | } 647 | result_id_->assign(value); 648 | } 649 | inline void ZoneTimeBucket::set_result_id(const char* value, size_t size) { 650 | _set_bit(5); 651 | if (result_id_ == &_default_result_id_) { 652 | result_id_ = new ::std::string; 653 | } 654 | result_id_->assign(reinterpret_cast(value), size); 655 | } 656 | inline ::std::string* ZoneTimeBucket::mutable_result_id() { 657 | _set_bit(5); 658 | if (result_id_ == &_default_result_id_) { 659 | result_id_ = new ::std::string; 660 | } 661 | return result_id_; 662 | } 663 | 664 | // required string db_key = 7; 665 | inline bool ZoneTimeBucket::has_db_key() const { 666 | return _has_bit(6); 667 | } 668 | inline void ZoneTimeBucket::clear_db_key() { 669 | if (db_key_ != &_default_db_key_) { 670 | db_key_->clear(); 671 | } 672 | _clear_bit(6); 673 | } 674 | inline const ::std::string& ZoneTimeBucket::db_key() const { 675 | return *db_key_; 676 | } 677 | inline void ZoneTimeBucket::set_db_key(const ::std::string& value) { 678 | _set_bit(6); 679 | if (db_key_ == &_default_db_key_) { 680 | db_key_ = new ::std::string; 681 | } 682 | db_key_->assign(value); 683 | } 684 | inline void ZoneTimeBucket::set_db_key(const char* value) { 685 | _set_bit(6); 686 | if (db_key_ == &_default_db_key_) { 687 | db_key_ = new ::std::string; 688 | } 689 | db_key_->assign(value); 690 | } 691 | inline void ZoneTimeBucket::set_db_key(const char* value, size_t size) { 692 | _set_bit(6); 693 | if (db_key_ == &_default_db_key_) { 694 | db_key_ = new ::std::string; 695 | } 696 | db_key_->assign(reinterpret_cast(value), size); 697 | } 698 | inline ::std::string* ZoneTimeBucket::mutable_db_key() { 699 | _set_bit(6); 700 | if (db_key_ == &_default_db_key_) { 701 | db_key_ = new ::std::string; 702 | } 703 | return db_key_; 704 | } 705 | 706 | // required string db_path = 8; 707 | inline bool ZoneTimeBucket::has_db_path() const { 708 | return _has_bit(7); 709 | } 710 | inline void ZoneTimeBucket::clear_db_path() { 711 | if (db_path_ != &_default_db_path_) { 712 | db_path_->clear(); 713 | } 714 | _clear_bit(7); 715 | } 716 | inline const ::std::string& ZoneTimeBucket::db_path() const { 717 | return *db_path_; 718 | } 719 | inline void ZoneTimeBucket::set_db_path(const ::std::string& value) { 720 | _set_bit(7); 721 | if (db_path_ == &_default_db_path_) { 722 | db_path_ = new ::std::string; 723 | } 724 | db_path_->assign(value); 725 | } 726 | inline void ZoneTimeBucket::set_db_path(const char* value) { 727 | _set_bit(7); 728 | if (db_path_ == &_default_db_path_) { 729 | db_path_ = new ::std::string; 730 | } 731 | db_path_->assign(value); 732 | } 733 | inline void ZoneTimeBucket::set_db_path(const char* value, size_t size) { 734 | _set_bit(7); 735 | if (db_path_ == &_default_db_path_) { 736 | db_path_ = new ::std::string; 737 | } 738 | db_path_->assign(reinterpret_cast(value), size); 739 | } 740 | inline ::std::string* ZoneTimeBucket::mutable_db_path() { 741 | _set_bit(7); 742 | if (db_path_ == &_default_db_path_) { 743 | db_path_ = new ::std::string; 744 | } 745 | return db_path_; 746 | } 747 | 748 | // optional string kv_map_file = 9; 749 | inline bool ZoneTimeBucket::has_kv_map_file() const { 750 | return _has_bit(8); 751 | } 752 | inline void ZoneTimeBucket::clear_kv_map_file() { 753 | if (kv_map_file_ != &_default_kv_map_file_) { 754 | kv_map_file_->clear(); 755 | } 756 | _clear_bit(8); 757 | } 758 | inline const ::std::string& ZoneTimeBucket::kv_map_file() const { 759 | return *kv_map_file_; 760 | } 761 | inline void ZoneTimeBucket::set_kv_map_file(const ::std::string& value) { 762 | _set_bit(8); 763 | if (kv_map_file_ == &_default_kv_map_file_) { 764 | kv_map_file_ = new ::std::string; 765 | } 766 | kv_map_file_->assign(value); 767 | } 768 | inline void ZoneTimeBucket::set_kv_map_file(const char* value) { 769 | _set_bit(8); 770 | if (kv_map_file_ == &_default_kv_map_file_) { 771 | kv_map_file_ = new ::std::string; 772 | } 773 | kv_map_file_->assign(value); 774 | } 775 | inline void ZoneTimeBucket::set_kv_map_file(const char* value, size_t size) { 776 | _set_bit(8); 777 | if (kv_map_file_ == &_default_kv_map_file_) { 778 | kv_map_file_ = new ::std::string; 779 | } 780 | kv_map_file_->assign(reinterpret_cast(value), size); 781 | } 782 | inline ::std::string* ZoneTimeBucket::mutable_kv_map_file() { 783 | _set_bit(8); 784 | if (kv_map_file_ == &_default_kv_map_file_) { 785 | kv_map_file_ = new ::std::string; 786 | } 787 | return kv_map_file_; 788 | } 789 | 790 | // repeated .cloudflare.ZoneTimeBucket.Counter map_entry = 10; 791 | inline int ZoneTimeBucket::map_entry_size() const { 792 | return map_entry_.size(); 793 | } 794 | inline void ZoneTimeBucket::clear_map_entry() { 795 | map_entry_.Clear(); 796 | } 797 | inline const ::cloudflare::ZoneTimeBucket_Counter& ZoneTimeBucket::map_entry(int index) const { 798 | return map_entry_.Get(index); 799 | } 800 | inline ::cloudflare::ZoneTimeBucket_Counter* ZoneTimeBucket::mutable_map_entry(int index) { 801 | return map_entry_.Mutable(index); 802 | } 803 | inline ::cloudflare::ZoneTimeBucket_Counter* ZoneTimeBucket::add_map_entry() { 804 | return map_entry_.Add(); 805 | } 806 | inline const ::google::protobuf::RepeatedPtrField< ::cloudflare::ZoneTimeBucket_Counter >& 807 | ZoneTimeBucket::map_entry() const { 808 | return map_entry_; 809 | } 810 | inline ::google::protobuf::RepeatedPtrField< ::cloudflare::ZoneTimeBucket_Counter >* 811 | ZoneTimeBucket::mutable_map_entry() { 812 | return &map_entry_; 813 | } 814 | 815 | 816 | // @@protoc_insertion_point(namespace_scope) 817 | 818 | } // namespace cloudflare 819 | 820 | #ifndef SWIG 821 | namespace google { 822 | namespace protobuf { 823 | 824 | 825 | } // namespace google 826 | } // namespace protobuf 827 | #endif // SWIG 828 | 829 | // @@protoc_insertion_point(global_scope) 830 | 831 | #endif // PROTOBUF_entry_2eproto__INCLUDED 832 | -------------------------------------------------------------------------------- /entry.proto: -------------------------------------------------------------------------------- 1 | package cloudflare; 2 | 3 | message ZoneTimeBucket { 4 | 5 | message Counter { 6 | required string key = 1; 7 | required int64 value = 2; 8 | } 9 | 10 | required string map_name = 1; 11 | required string classification = 2; 12 | required string doctype = 3; 13 | required string pop = 4; 14 | required string psource = 5; 15 | required string result_id = 6; 16 | required string db_key = 7; 17 | required string db_path = 8; 18 | optional string kv_map_file = 9; // If not null, use the kv_map within, not #10 19 | repeated Counter map_entry = 10; // Use a built in list of counters? 20 | 21 | } 22 | -------------------------------------------------------------------------------- /examples/getting_started: -------------------------------------------------------------------------------- 1 | 2 | Sample Project #1 (log processing) 3 | 4 | Take the data defined in data.sql, add it to a SortaSQL DB, then query the data. 5 | 6 | 1) Compile and install the code as shown in the README file. You will first want to look in common.h, setting file paths. 7 | You should now have the library pg_kc.so installed 8 | 9 | 2) In Postgres, run the files pg_kc.sql and helper.sql. (\. path/to/file.sql) Make sure that you don't get any errors. 10 | You should now have a number of functions definined in Postgres, including get_page_views(). Look at the definition of this function (in helper.sql) to get a sense of how to access SortaSQL rows. 11 | 12 | 3) Compile import_data.cc 13 | g++ import_data.cc -o import_data ../cpp/entry.pb.cc -I../ -ljson_linux-gcc-4.4.4_libmt -lboost_iostreams -lpcrecpp -lkyotocabinet -lprotobuf 14 | 15 | This program takes data in a Postgres COPY statement and adds it to a SortaSQL DB. 16 | 17 | You will need to have installed 18 | * http://jsoncpp.sourceforge.net/ 19 | * boost iostreams 20 | * pcrecpp 21 | 22 | Adjust the config directives in importer.json to make sense, making sure the directories listed exist. 23 | 24 | Import the data: 25 | 26 | ./import_data example.sql 27 | 28 | This will output a file which needs to be loaded in Postgres while populating the proper KC DBs. 29 | The PG file will now just have the metadata needed for accessing these KC DBs. 30 | Load this file into Postgres. 31 | 32 | Finally, access the data. For example, to get the number of page views for the zone -99999 in the last 1000000 hours, try: 33 | 34 | select * from get_page_views(1000000, -99999); 35 | 36 | To make custom functions, study the definition of the function get_page_views in helper.sql. 37 | -------------------------------------------------------------------------------- /examples/helper.sql: -------------------------------------------------------------------------------- 1 | CREATE OR REPLACE FUNCTION get_page_views(IN __hours INT, IN __zid INT, OUT __st timestamp, OUT __val bigint) RETURNS setof RECORD AS $_$ 2 | DECLARE 3 | r RECORD; 4 | BEGIN 5 | FOR r IN select start_time, sum(val) as val from ( select date_trunc('hour', start_time) as start_time, kc_key(kc) as key, kc_val(kc) as val, kc_doctype(kc) as doc from (select start_time, kcx('basic', kcts(start_time, oid), result_id) as kc from aggregate_meta_data where zid = __zid and start_time >= now() - (__hours || ' hour')::interval and period = '15 min' ) a ) b where key = 'num_rows' and doc = 'HTML' group by start_time order by start_time asc 6 | LOOP 7 | __st := r.start_time; 8 | __val := r.val; 9 | RETURN NEXT; 10 | END LOOP; 11 | END; 12 | $_$ Language 'plpgsql' IMMUTABLE STRICT; 13 | 14 | CREATE TABLE aggregate_meta_data ( 15 | result_id VARCHAR(128) PRIMARY KEY, 16 | zid BIGINT NOT NULL, 17 | oid BIGINT NOT NULL, 18 | start_time TIMESTAMPTZ NOT NULL, 19 | period INTERVAL NOT NULL, 20 | zone_name TEXT NOT NULL, 21 | CONSTRAINT cf_logs_aggregate_uq UNIQUE(zid, start_time, period) 22 | ); -------------------------------------------------------------------------------- /examples/import_data.cc: -------------------------------------------------------------------------------- 1 | /** 2 | Goals here: 3 | 4 | Read in a merged sql dump file. 5 | Write the sql file to postgres. 6 | Strip out the k/v info, instead writting this info directly into KC. 7 | 8 | @author ian@cloudflare.com 9 | */ 10 | 11 | #include 12 | #include 13 | #include 14 | #include 15 | #include 16 | #include 17 | #include 18 | 19 | #include 20 | #include 21 | #include 22 | #include 23 | #include 24 | 25 | #include 26 | #include 27 | 28 | #include 29 | #include 30 | #include 31 | #include 32 | 33 | #include "cpp/entry.pb.h" 34 | #include "common.h" 35 | 36 | using namespace kyotocabinet; 37 | using namespace std; 38 | 39 | // Do we want to clear out any cruft from the DB? 40 | //static const uint32_t g_sql_db_flags = PolyDB::OWRITER | PolyDB::OCREATE | PolyDB::OTRUNCATE; 41 | static const uint32_t g_sql_db_flags = PolyDB::OWRITER | PolyDB::OCREATE; 42 | 43 | // States 44 | enum merge_state_t { DELETE = 10, AGG = 20, MAP = 40 }; 45 | enum merge_map_t { MAP_RID, MAP_CLASSIFICATION, MAP_DOCTYPE, MAP_POP, MAP_PSOURCE, MAP_NAME, MAP_KEY, MAP_VALUE }; 46 | 47 | // And how to tell what state we're in ... 48 | static const pcrecpp::RE CF_STATE_RE("^(BEGIN)|^(COPY)|(\\\\.)|(COMMIT)"); 49 | 50 | // Pull out the proper start time. 51 | static const char *DEFAULT_UID_DIR = "-1"; 52 | static const char *DEFAULT_TIME_DIR = "default_time"; 53 | static const pcrecpp::RE CF_START_TIME_RE("\'(.*?) (.*)"); 54 | static const pcrecpp::RE CF_START_TIME_DUMP_RE("(.*?) (.*)"); 55 | static const pcrecpp::RE CF_RID_RE("^(.*?)\\s"); 56 | static const pcrecpp::RE CF_ALL_RE("(.*?)\\s"); 57 | 58 | // And how fields are seperated. 59 | typedef boost::tokenizer > tokenizer; 60 | static boost::char_separator CF_MAP_SEP("\t"); 61 | 62 | // Handle the closing of dbs as we go though. 63 | static const unsigned int MAP_DBS_TO_PRUNE = 200; 64 | static const unsigned int MAX_MAP_SIZE = 400; 65 | 66 | static int64_t g_next_db = 0; 67 | 68 | // Macros to do simple things. 69 | inline void set_kv_path(cloudflare::ZoneTimeBucket &msg, string &map_db_key_str) { 70 | string map_dir(CF_KV_MAP_BASE); 71 | size_t found = map_dir.length(); 72 | struct stat path_buf; 73 | 74 | map_dir = map_dir + "/" + msg.db_path(); 75 | found = map_dir.find_first_of("/", found); 76 | while (found != string::npos) { 77 | string tmp_found = map_dir.substr(0, found); 78 | if (stat(tmp_found.c_str(), &path_buf) != 0) { 79 | mkdir (tmp_found.c_str(), NEW_DIR_MASK); 80 | chmod (tmp_found.c_str(), NEW_DIR_MODE); 81 | } 82 | found = map_dir.find_first_of("/", found + 1); 83 | } 84 | 85 | // And finish off the last one. 86 | if (stat(map_dir.c_str(), &path_buf) != 0) { 87 | mkdir (map_dir.c_str(), NEW_DIR_MASK); 88 | chmod (map_dir.c_str(), NEW_DIR_MODE); 89 | } 90 | 91 | msg.set_kv_map_file(map_dir + "/" + map_db_key_str + "." + DB_TYPE); 92 | } 93 | 94 | inline void close_kv_db(PolyDB *current_map_kv, bool &db_open) { 95 | if (db_open) { 96 | current_map_kv->end_transaction(1); 97 | if (current_map_kv->close()) { 98 | db_open = false; 99 | } else { 100 | cerr << "close error: " << current_map_kv->error().name() << endl; 101 | } 102 | } 103 | } 104 | 105 | inline void open_kv_db(PolyDB *current_map_kv, cloudflare::ZoneTimeBucket &msg, bool &db_open) { 106 | 107 | // Do nothing if the db is already open to the right one. 108 | if (db_open && current_map_kv->path() == msg.kv_map_file()) { 109 | return; 110 | } 111 | 112 | // Always close before opening. 113 | close_kv_db(current_map_kv, db_open); 114 | 115 | if (!db_open) { 116 | if (current_map_kv->open(msg.kv_map_file(), g_sql_db_flags)) { 117 | chmod (msg.kv_map_file().c_str(), FILE_MODE); 118 | db_open = true; 119 | current_map_kv->begin_transaction(); 120 | } else { 121 | cerr << msg.kv_map_file() << " -- open error: " << current_map_kv->error().name() << endl; 122 | } 123 | } 124 | } 125 | 126 | inline void prune_map(map &dbs, PolyDB *dbs2queue, cloudflare::ZoneTimeBucket &msg) { 127 | 128 | // Close all of the agg map dbs. 129 | DB::Cursor* cur = dbs2queue->cursor(); 130 | cur->jump(); 131 | unsigned int count = 0; 132 | string value; 133 | string old_db = msg.db_path(); 134 | while ((count < MAP_DBS_TO_PRUNE) && ((cur->get_value(&value, false)))) { 135 | if (value.compare(old_db) != 0) { 136 | dbs[value]->end_transaction(1); 137 | dbs[value]->close(); 138 | delete dbs[value]; 139 | dbs.erase(value); 140 | cur->remove(); 141 | } else { // Make sure that we ALWAYS step the cursor here. 142 | cur->step(); 143 | } 144 | count++; 145 | } 146 | delete cur; 147 | } 148 | 149 | inline void open_main_db (PolyDB *db, string &map_name, string &time_dir, string &uid_dir) { 150 | 151 | string base_dir(BASE_DB_DIR); 152 | struct stat path_buf; 153 | base_dir = (base_dir + "/") + time_dir; 154 | if (stat(base_dir.c_str(), &path_buf) != 0) { 155 | mkdir (base_dir.c_str(), NEW_DIR_MODE); 156 | chmod (base_dir.c_str(), NEW_DIR_MODE); 157 | } 158 | base_dir = (base_dir + "/") + uid_dir; 159 | if (stat(base_dir.c_str(), &path_buf) != 0) { 160 | mkdir (base_dir.c_str(), NEW_DIR_MODE); 161 | chmod (base_dir.c_str(), NEW_DIR_MODE); 162 | } 163 | 164 | string tune_opts = "#opts=cl#bnum=1000#psiz=32768"; 165 | if (!db->open(base_dir + "/pg_agg_map_" + map_name + "." + DB_TYPE + tune_opts, g_sql_db_flags)) { 166 | cerr << "open error: " << db->error().name() << endl; 167 | } 168 | db->begin_transaction(); 169 | // And make sure that everyone can write to this DB. 170 | chmod ((base_dir + "/pg_agg_map_" + map_name + "." + DB_TYPE).c_str(), FILE_MODE); 171 | } 172 | 173 | // For when we've tripped the size limit -- need to move all values from map_entry to the new db. 174 | inline void dump_to_kc (cloudflare::ZoneTimeBucket &msg, PolyDB *current_map_kv, string &key, int64_t value) { 175 | for (int j = 0; j < msg.map_entry_size(); j++) { 176 | current_map_kv->increment(msg.map_entry(j).key(), msg.map_entry(j).value()); 177 | } 178 | msg.clear_map_entry(); 179 | current_map_kv->increment(key, value); 180 | } 181 | 182 | inline int handle_delete(PolyDB *deldb, string &line) { 183 | deldb->set(line, "1"); 184 | return 0; 185 | } 186 | 187 | inline int handle_agg(PolyDB *aggdb, PolyDB *rid2tsdb, string &line, bool is_regular) { 188 | 189 | tokenizer tokens(line, CF_MAP_SEP); 190 | string rid; 191 | string uid; 192 | string time; 193 | int place = 0; 194 | bool found = true; 195 | 196 | for (tokenizer::iterator tok_iter = tokens.begin(); tok_iter != tokens.end(); ++tok_iter) { 197 | switch (place) { 198 | case 0: rid = (*tok_iter); 199 | case 2: uid = (*tok_iter); 200 | case 3: if (!is_regular) { CF_START_TIME_DUMP_RE.PartialMatch((*tok_iter), &time); } 201 | else { CF_START_TIME_RE.PartialMatch((*tok_iter), &time); } 202 | } 203 | place++; 204 | } 205 | 206 | if (found) { 207 | rid2tsdb->set(rid, time + "\t" + uid); 208 | } 209 | aggdb->set(line, "1"); 210 | return 0; 211 | } 212 | 213 | inline int handle_map(PolyDB *mapdb, string &line) { 214 | 215 | int place = MAP_RID; 216 | tokenizer tokens(line, CF_MAP_SEP); 217 | stringstream map_key; 218 | int64_t map_value = 0; 219 | 220 | for (tokenizer::iterator tok_iter = tokens.begin(); tok_iter != tokens.end(); ++tok_iter) { 221 | switch (place) { 222 | case MAP_RID: map_key << (*tok_iter) << "\t"; break; 223 | case MAP_CLASSIFICATION: map_key << (*tok_iter) << "\t"; break; 224 | case MAP_DOCTYPE: map_key << (*tok_iter) << "\t"; break; 225 | case MAP_POP: map_key << (*tok_iter) << "\t"; break; 226 | case MAP_PSOURCE: map_key << (*tok_iter) << "\t"; break; 227 | case MAP_NAME: map_key << (*tok_iter) << "\t"; break; 228 | case MAP_KEY: map_key << (*tok_iter); break; 229 | case MAP_VALUE: map_value = kyotocabinet::atoi(tok_iter->c_str()); break; 230 | } 231 | place++; 232 | } 233 | 234 | if (map_value > 0) { 235 | mapdb->increment(map_key.str(), map_value); 236 | } 237 | return 0; 238 | } 239 | 240 | int load_file (PolyDB *deldb, PolyDB *aggdb, PolyDB *mapdb, PolyDB *ip_deldb, PolyDB *ipdb, PolyDB *rid2tsdb, string &file, 241 | int start_state = 0) { 242 | 243 | 244 | ifstream infile; 245 | string line; 246 | int lines = 0; 247 | int current_state = start_state; 248 | boost::iostreams::filtering_istream in; 249 | boost::iostreams::zlib_params p; 250 | 251 | if (file.substr(file.length() - 3, 3) == ".gz") { 252 | infile.open(file.c_str(), ios_base::in | ios_base::binary); 253 | p.window_bits = 16 + MAX_WBITS; 254 | in.push(boost::iostreams::zlib_decompressor(p)); 255 | in.push(infile); 256 | } else { 257 | infile.open(file.c_str(), ifstream::in); 258 | in.push(infile); 259 | } 260 | 261 | while (in.good()) { 262 | lines++; 263 | getline (in, line); 264 | if (CF_STATE_RE.PartialMatch(line)) { 265 | current_state += 10; 266 | } else { 267 | switch (current_state) { 268 | case DELETE: 269 | handle_delete(deldb, line); break; 270 | case AGG: 271 | handle_agg(aggdb, rid2tsdb, line, (start_state == 0)); break; 272 | case MAP: 273 | handle_map(mapdb, line); break; 274 | } 275 | } 276 | } 277 | 278 | infile.close(); 279 | return lines; 280 | } 281 | 282 | int handle_map_merge(map &dbs, Json::Value &config, 283 | string &line, int64_t value, cloudflare::ZoneTimeBucket &msg, 284 | PolyDB *current_map_kv, bool &db_open, PolyDB *rid2tsdb, PolyDB *dbs2queue) { 285 | 286 | stringstream map_db_key; 287 | stringstream kv_valwriter; 288 | char sizewriter[64]; 289 | 290 | string classification; 291 | string rid; 292 | string map_name; 293 | string doctype; 294 | string pop; 295 | string psource; 296 | string key; 297 | 298 | tokenizer tokens(line, CF_MAP_SEP); 299 | int place = MAP_RID; 300 | map::iterator it; 301 | 302 | string map_db_key_str; 303 | string msg_value; 304 | string msg_ser; 305 | string time_dir; 306 | string uid_dir; 307 | string main_db_path; 308 | cloudflare::ZoneTimeBucket_Counter *counter; 309 | string uid_time_dir; 310 | 311 | for (tokenizer::iterator tok_iter = tokens.begin(); tok_iter != tokens.end(); ++tok_iter) { 312 | switch (place) { 313 | case MAP_RID: rid = (*tok_iter); break; 314 | case MAP_CLASSIFICATION: classification = (*tok_iter); break; 315 | case MAP_DOCTYPE: doctype = (*tok_iter); map_db_key << (*tok_iter) << CF_LABEL_SEP; break; 316 | case MAP_POP: pop = (*tok_iter); map_db_key << (*tok_iter) << CF_LABEL_SEP; break; 317 | case MAP_PSOURCE: psource = (*tok_iter); map_db_key << (*tok_iter) << CF_LABEL_SEP; break; 318 | case MAP_NAME: map_name = (*tok_iter); map_db_key << (*tok_iter); break; 319 | case MAP_KEY: key = (*tok_iter); break; 320 | } 321 | place++; 322 | } 323 | 324 | // Do we need to make a new db? 325 | 326 | if (rid2tsdb->get(rid, &uid_time_dir)) { 327 | size_t tab_pos = uid_time_dir.find_first_of("\t"); 328 | time_dir = uid_time_dir.substr(0, tab_pos); 329 | uid_dir = uid_time_dir.substr(tab_pos+1); 330 | } else { 331 | time_dir = DEFAULT_TIME_DIR; 332 | uid_dir = DEFAULT_UID_DIR; 333 | } 334 | 335 | main_db_path = map_name + "/" + time_dir + "/" + uid_dir; 336 | it = dbs.find(main_db_path); 337 | if (it == dbs.end()) { 338 | if (dbs.size() > MAX_MAP_SIZE) { 339 | prune_map(dbs, dbs2queue, msg); 340 | } 341 | dbs[main_db_path] = new PolyDB(); // Need to clear out this listing periodically. 342 | open_main_db (dbs[main_db_path], map_name, time_dir, uid_dir); 343 | snprintf(sizewriter, 64, "%020ld", g_next_db); 344 | g_next_db++; 345 | dbs2queue->set(sizewriter, main_db_path); 346 | } 347 | 348 | map_db_key_str = rid + CF_LABEL_SEP + classification + CF_LABEL_SEP + map_db_key.str(); 349 | //cout << "Going with " << map_db_key_str << endl; 350 | 351 | // If we are dealing with the same damn PB from before, use it. 352 | if (msg.has_db_key() && msg.db_key().compare(map_db_key_str) == 0) { 353 | 354 | //cout << "Reusing" << endl; 355 | 356 | // Clearing old values and reseting based on kc info from the merge. 357 | if (msg.has_kv_map_file()) { 358 | 359 | open_kv_db(current_map_kv, msg, db_open); 360 | current_map_kv->increment(key, value); 361 | // cout << "Reusing current KC" << endl; 362 | 363 | } else if (msg.map_entry_size() > MAX_KEYS_BEFORE_KV_MAP) { 364 | // Set the map db path. 365 | set_kv_path(msg, map_db_key_str); 366 | open_kv_db(current_map_kv, msg, db_open); 367 | dump_to_kc (msg, current_map_kv, key, value); 368 | 369 | // cout << "Dumping" << endl; 370 | 371 | } else { 372 | // cout << "Setting " << key << endl; 373 | 374 | counter = msg.add_map_entry(); 375 | counter->set_key(key); 376 | counter->set_value(value); 377 | } 378 | } else { 379 | // Save the old and get the new. 380 | // Save the old values (if they exist) 381 | if (msg.has_db_key()) { 382 | msg.SerializeToString(&msg_ser); 383 | dbs[msg.db_path()]->set(msg.db_key(), msg_ser); 384 | close_kv_db(current_map_kv, db_open); 385 | } 386 | 387 | // Load the old values if such a thing exists, updating the map keys accordingly. 388 | 389 | 390 | if (dbs[main_db_path]->get(map_db_key_str, &msg_value)) { 391 | // Remove the old info from the map. 392 | msg.clear_map_entry(); 393 | if (msg.ParseFromString(msg_value)) { 394 | // Clearing old values and reseting based on kc info from the merge. 395 | if (msg.has_kv_map_file()) { 396 | 397 | //cout << "Opening" << msg.kv_map_file() << endl; 398 | 399 | open_kv_db(current_map_kv, msg, db_open); 400 | current_map_kv->increment(key, value); 401 | } else if (msg.map_entry_size() > MAX_KEYS_BEFORE_KV_MAP) { 402 | set_kv_path(msg, map_db_key_str); 403 | open_kv_db(current_map_kv, msg, db_open); 404 | 405 | //cout << "Dumping from rebuild" << endl; 406 | 407 | dump_to_kc (msg, current_map_kv, key, value); 408 | } else { 409 | 410 | //cout << "Adding counter " << key << endl; 411 | counter = msg.add_map_entry(); 412 | counter->set_key(key); 413 | counter->set_value(value); 414 | } 415 | } 416 | } else { 417 | 418 | //cout << "Adding counter fresh msg " << key << endl; 419 | 420 | msg.set_result_id(rid); 421 | msg.set_map_name(map_name); 422 | msg.set_classification(classification); 423 | msg.set_doctype(doctype); 424 | msg.set_pop(pop); 425 | msg.set_psource(psource); 426 | msg.set_db_key(map_db_key_str); 427 | msg.set_db_path(main_db_path); 428 | 429 | // Assumed to be a new entry, so free to just go crazy. 430 | msg.clear_map_entry(); 431 | msg.clear_kv_map_file(); 432 | counter = msg.add_map_entry(); 433 | counter->set_key(key); 434 | counter->set_value(value); 435 | } 436 | } 437 | 438 | return 1; // Why one? why not? 439 | } 440 | 441 | int output_merged (PolyDB *deldb, PolyDB *aggdb, PolyDB *mapdb, PolyDB *ip_deldb, PolyDB *ipdb, PolyDB *rid2tsdb, PolyDB *dbs2queue, 442 | Json::Value &config, string &tmp_file, string &final_file) { 443 | 444 | ofstream outfile (tmp_file.c_str(), ios_base::out | ios_base::trunc); 445 | DB::Cursor *cur = deldb->cursor(); 446 | string rec_key; 447 | string map_key; 448 | map dbs; 449 | map::iterator it; 450 | cloudflare::ZoneTimeBucket current_msg; 451 | string current_msg_ser; 452 | 453 | outfile << "BEGIN;\n"; 454 | 455 | cur->jump(); 456 | while (cur->get_key(&rec_key, true)) { 457 | outfile << rec_key << endl; 458 | } 459 | delete cur; 460 | 461 | outfile << "COPY aggregate_meta_data (result_id, zid, oid, start_time, period, zone_name) FROM stdin WITH DELIMITER E'\\t';" 462 | << endl; 463 | 464 | cur = aggdb->cursor(); 465 | cur->jump(); 466 | while (cur->get_key(&rec_key, true)) { 467 | outfile << rec_key << endl; 468 | } 469 | delete cur; 470 | 471 | outfile << "\\.\n"; 472 | 473 | // Move the map values into a kcdb here. 474 | cur = mapdb->cursor(); 475 | cur->jump(); 476 | 477 | PolyDB kv_map_db; 478 | bool db_open = false; 479 | 480 | while (cur->get_key(&map_key, true)) { 481 | handle_map_merge(dbs, config, map_key, mapdb->increment(map_key, 0), current_msg, &kv_map_db, db_open, 482 | rid2tsdb, dbs2queue); 483 | } 484 | delete cur; 485 | 486 | close_kv_db(&kv_map_db, db_open); 487 | 488 | // Save the last value to the db. 489 | if (current_msg.has_map_name()) { 490 | current_msg.SerializeToString(¤t_msg_ser); 491 | dbs[current_msg.db_path()]->set(current_msg.db_key(), current_msg_ser); 492 | } 493 | 494 | outfile << "COMMIT;\n"; 495 | 496 | // Close all of the agg map dbs. 497 | for ( it = dbs.begin() ; it != dbs.end(); it++ ) { 498 | it->second->end_transaction(1); 499 | it->second->close(); 500 | delete it->second; 501 | } 502 | 503 | outfile.close(); 504 | 505 | // Now, rename to final. 506 | rename (tmp_file.c_str(), final_file.c_str()); 507 | 508 | return 1; 509 | } 510 | 511 | inline void open_dbs(PolyDB &deldb, PolyDB &aggdb, PolyDB &mapdb, PolyDB &ip_deldb, PolyDB &ipdb, PolyDB &rid2tsdb, PolyDB &dbs2queue, 512 | Json::Value &config, const uint32_t db_flags) { 513 | 514 | if (!deldb.open(config.get("LOG_MERGE_DELETE", "orig_del.kct").asString(), db_flags)) { 515 | cerr << "open error (deldb): " << deldb.error().name() << endl; 516 | } 517 | if (!aggdb.open(config.get("LOG_MERGE_AGG", "orig_agg.kct").asString(), db_flags)) { 518 | cerr << "open error (aggdb): " << aggdb.error().name() << endl; 519 | } 520 | if (!mapdb.open(config.get("LOG_MERGE_MAP", "orig_map.kct").asString(), db_flags)) { 521 | cerr << "open error (mapdb): " << mapdb.error().name() << endl; 522 | } 523 | if (!ip_deldb.open(config.get("LOG_MERGE_DELETE_IPS", "orig_del_ips.kct").asString(), db_flags)) { 524 | cerr << "open error (ip_deldb): " << ip_deldb.error().name() << endl; 525 | } 526 | if (!ipdb.open(config.get("LOG_MERGE_IPS", "orig_ipdb.kct").asString(), db_flags)) { 527 | cerr << "open error (ipdb): " << ipdb.error().name() << endl; 528 | } 529 | if (!rid2tsdb.open("%", db_flags)) { 530 | cerr << "open error: " << rid2tsdb.error().name() << endl; 531 | } 532 | if (!dbs2queue.open("%", db_flags)) { 533 | cerr << "open error: " << dbs2queue.error().name() << endl; 534 | } 535 | } 536 | 537 | inline void close_dbs(PolyDB &deldb, PolyDB &aggdb, PolyDB &mapdb, PolyDB &ip_deldb, PolyDB &ipdb, PolyDB &rid2tsdb, PolyDB &dbs2queue) { 538 | deldb.close(); 539 | aggdb.close(); 540 | mapdb.close(); 541 | ip_deldb.close(); 542 | ipdb.close(); 543 | rid2tsdb.close(); 544 | dbs2queue.close(); 545 | } 546 | 547 | int main(int argc, char** argv) { 548 | 549 | const char *config_file = "importer.json"; 550 | const char *lock_file = "/tmp/import.LOCK"; // This is importent. 551 | int c; 552 | struct stat path_buf; 553 | string current_file; 554 | const uint32_t db_flags = PolyDB::OWRITER | PolyDB::OCREATE | PolyDB::OTRUNCATE; 555 | string tmp_file; 556 | string final_file; 557 | stringstream file_namer; 558 | int start_state = 0; 559 | 560 | while (-1 != (c = getopt(argc, argv, "c:d"))) { 561 | switch (c) { 562 | case 'd': 563 | start_state = DELETE; 564 | break; 565 | case 'c': 566 | config_file = optarg; 567 | break; 568 | default: 569 | exit(-22); 570 | } 571 | } 572 | 573 | // DBs here for the merging part. 574 | PolyDB deldb; 575 | PolyDB aggdb; 576 | PolyDB mapdb; 577 | PolyDB ip_deldb; 578 | PolyDB ipdb; 579 | PolyDB rid2tsdb; 580 | PolyDB dbs2queue; 581 | 582 | // Grab the config file. 583 | Json::Value config; 584 | Json::Reader reader; 585 | ifstream inconf(config_file, ifstream::in); 586 | 587 | // Check the lock file. 588 | if (stat(lock_file, &path_buf) == 0) { 589 | cerr << "ERROR: Lockfile " << lock_file << " present." << endl; 590 | exit(-22); 591 | } else { 592 | ofstream lockstr; 593 | lockstr.open (lock_file); 594 | lockstr.close(); 595 | } 596 | 597 | if (!reader.parse(inconf, config)) { 598 | return -1; 599 | } 600 | inconf.close(); 601 | 602 | // Make sure that the proto bufs are happy. 603 | GOOGLE_PROTOBUF_VERIFY_VERSION; 604 | 605 | open_dbs(deldb, aggdb, mapdb, ip_deldb, ipdb, rid2tsdb, dbs2queue, config, db_flags); 606 | 607 | // Parse the file(s) 608 | int nextarg = optind; 609 | if (argc > optind) { 610 | while (nextarg < argc) { 611 | if (stat(argv[nextarg], &path_buf) == 0) { 612 | if (S_ISREG(path_buf.st_mode)) { 613 | current_file = string(argv[nextarg]); 614 | load_file(&deldb, &aggdb, &mapdb, &ip_deldb, &ipdb, &rid2tsdb, current_file, start_state); 615 | } else if (S_ISDIR(path_buf.st_mode)) { 616 | DIR *dir; 617 | struct dirent *ent; 618 | string tmp_file; 619 | 620 | current_file = string(argv[nextarg]); 621 | dir = opendir (argv[nextarg]); 622 | if (dir != NULL) { 623 | while ((ent = readdir (dir)) != NULL) { 624 | tmp_file = current_file + "/" + ent->d_name; 625 | load_file(&deldb, &aggdb, &mapdb, &ip_deldb, &ipdb, &rid2tsdb, tmp_file, start_state); 626 | } 627 | closedir (dir); 628 | } 629 | } 630 | } 631 | nextarg++; 632 | } 633 | } 634 | 635 | // Write out the new file 636 | file_namer << config.get("SQL_FILE_EXTRA_NEW_DIR", "/tmp").asString() << "/tmp_" << time(NULL) << ".sql"; 637 | tmp_file = file_namer.str(); 638 | file_namer.str(""); 639 | 640 | file_namer << config.get("SQL_NEW_FILE_DIR", "/tmp").asString() << "/final_" << time(NULL) << ".sql"; 641 | final_file = file_namer.str(); 642 | 643 | output_merged(&deldb, &aggdb, &mapdb, &ip_deldb, &ipdb, &rid2tsdb, &dbs2queue, config, tmp_file, final_file); 644 | cout << final_file << endl; 645 | 646 | // And the PB. 647 | google::protobuf::ShutdownProtobufLibrary(); 648 | 649 | close_dbs(deldb, aggdb, mapdb, ip_deldb, ipdb, rid2tsdb, dbs2queue); 650 | 651 | // And the lock file. 652 | unlink (lock_file); 653 | } 654 | -------------------------------------------------------------------------------- /examples/importer.json: -------------------------------------------------------------------------------- 1 | { 2 | "AGG_MAP_BASE":"/tmp/", 3 | "LOG_MERGE_DELETE":"/mnt/logs/kyoto/merge_del.kch", 4 | "LOG_MERGE_AGG":"/mnt/logs/kyoto/merge_agg.kch", 5 | "LOG_MERGE_MAP":"/mnt/logs/kyoto/merge_map.kch", 6 | "LOG_MERGE_IPS":"/mnt/logs/kyoto/merge_ips.kch", 7 | "OWNER_ID_DIR":"/mnt/logs/kyoto/owners", 8 | "SQL_FILE_DIR":"/logger/outgoing/sql", 9 | "SQL_FILE_EXTRA_NEW_DIR":"/logger/outgoing/sql_new", 10 | "TMP_FILE_DIR":"/mnt/logs/tmp", 11 | "SQL_NEW_FILE_DIR":"/mnt/logs/sql", 12 | "MERGE_LOCK_FILE":"/tmp/merge.lock" 13 | } 14 | -------------------------------------------------------------------------------- /pg_kc.c: -------------------------------------------------------------------------------- 1 | /** 2 | 3 | Copywrite 2011 CloudFlare, Inc. 4 | @author Ian Pye 5 | 6 | Provides a bridge from PostgreSQL to Kyoto Cabinet. 7 | 8 | Use at your own risk. 9 | 10 | */ 11 | 12 | #include 13 | 14 | #include 15 | #include 16 | #include 17 | #include 18 | #include 19 | #include 20 | 21 | #include 22 | #include 23 | #include 24 | #include 25 | #include 26 | 27 | #include "c/entry.pb-c.h" 28 | #include "common.h" 29 | 30 | #ifdef PG_MODULE_MAGIC 31 | PG_MODULE_MAGIC; 32 | #endif 33 | 34 | // Print debugging notes? 35 | //#define CF_DUBUG 1; 36 | //#define CF_NO_DB_IS_ERR 1; 37 | 38 | // General constant defines 39 | static const int MAX_JUMP_KEY_LEN = 256; 40 | static const int MIN_ARGS = 3; 41 | static const int START_VARIABLE_INDEX = 3; 42 | static const int END_VARIABLE_INDEX = 7; 43 | static const int MAX_KC_ROW_ENTRY = 256; 44 | static const int KC_MAX_ROW_SER = 1028; 45 | static const int KC_MAX_RID = 1028; 46 | 47 | // Hold the DB and its asc. info. 48 | typedef struct { 49 | KCDB *db; 50 | char *db_path; 51 | KCCUR *cur; 52 | char *jump_key; 53 | Cloudflare__ZoneTimeBucket *msg; 54 | unsigned int next_map; 55 | KCDB *kv_db; 56 | KCCUR *kv_cur; 57 | } KC_ENTRY; 58 | 59 | typedef struct { 60 | char *map_name; 61 | char *start_time; 62 | char *kv_map_name; 63 | } KC_DB_INFO; 64 | 65 | // Row type 66 | typedef struct { 67 | int32 vl_len_; /* varlena header (do not touch directly!) */ 68 | char *doctype; 69 | char *classification; 70 | char *pop; 71 | char *psource; 72 | char *key; 73 | int64_t value; 74 | //KC_DB_INFO db_info; // @TODO -- should this be set, allowing for working in/out functions? 75 | } KC_ROW; 76 | 77 | PG_FUNCTION_INFO_V1(kc_row_in); 78 | 79 | // @TODO -- make this happy. 80 | Datum kc_row_in(PG_FUNCTION_ARGS) { 81 | //char *kv = PG_GETARG_CSTRING(0); 82 | KC_ROW *out; 83 | 84 | size_t size = sizeof(KC_ROW); 85 | out = (KC_ROW *)palloc(size); 86 | memset(out, '0', size); 87 | SET_VARSIZE(out, size); 88 | out->doctype = (char *)palloc(sizeof(char) * MAX_KC_ROW_ENTRY); 89 | out->classification = (char *)palloc(sizeof(char) * MAX_KC_ROW_ENTRY); 90 | out->pop = (char *)palloc(sizeof(char) * MAX_KC_ROW_ENTRY); 91 | out->psource = (char *)palloc(sizeof(char) * MAX_KC_ROW_ENTRY); 92 | out->key = (char *)palloc(sizeof(char) * MAX_KC_ROW_ENTRY); 93 | 94 | strncpy(out->key, "lolkey -- fill me in", MAX_KC_ROW_ENTRY); 95 | out->value = 0; 96 | 97 | PG_RETURN_POINTER(out); 98 | } 99 | 100 | PG_FUNCTION_INFO_V1(kc_row_out); 101 | 102 | Datum kc_row_out(PG_FUNCTION_ARGS) { 103 | KC_ROW *in = (KC_ROW *) PG_GETARG_POINTER(0); 104 | char *result; 105 | 106 | result = (char *) palloc(256); 107 | snprintf(result, KC_MAX_ROW_SER, "{%s,%s,%s,%s,%s,%ld}", in->classification, in->doctype, 108 | in->pop, in->psource, in->key, in->value); 109 | PG_RETURN_CSTRING(result); 110 | } 111 | 112 | PG_FUNCTION_INFO_V1(kc_key); 113 | 114 | Datum kc_key(PG_FUNCTION_ARGS) { 115 | KC_ROW *in = (KC_ROW *) PG_GETARG_POINTER(0); 116 | text *tout; 117 | 118 | tout = cstring_to_text(in->key); 119 | PG_RETURN_TEXT_P(tout); 120 | } 121 | 122 | PG_FUNCTION_INFO_V1(kc_val); 123 | 124 | Datum kc_val(PG_FUNCTION_ARGS) { 125 | KC_ROW *in = (KC_ROW *) PG_GETARG_POINTER(0); 126 | int64_t result = in->value; 127 | PG_RETURN_INT64(result); 128 | } 129 | 130 | PG_FUNCTION_INFO_V1(kc_doctype); 131 | 132 | Datum kc_doctype(PG_FUNCTION_ARGS) { 133 | KC_ROW *in = (KC_ROW *) PG_GETARG_POINTER(0); 134 | text *tout; 135 | 136 | tout = cstring_to_text(in->doctype); 137 | PG_RETURN_TEXT_P(tout); 138 | } 139 | 140 | PG_FUNCTION_INFO_V1(kc_class); 141 | 142 | Datum kc_class(PG_FUNCTION_ARGS) { 143 | KC_ROW *in = (KC_ROW *) PG_GETARG_POINTER(0); 144 | text *tout; 145 | 146 | tout = cstring_to_text(in->classification); 147 | PG_RETURN_TEXT_P(tout); 148 | } 149 | 150 | PG_FUNCTION_INFO_V1(kc_pop); 151 | 152 | Datum kc_pop(PG_FUNCTION_ARGS) { 153 | KC_ROW *in = (KC_ROW *) PG_GETARG_POINTER(0); 154 | text *tout; 155 | 156 | tout = cstring_to_text(in->pop); 157 | PG_RETURN_TEXT_P(tout); 158 | } 159 | 160 | PG_FUNCTION_INFO_V1(kc_psource); 161 | 162 | Datum kc_psource(PG_FUNCTION_ARGS) { 163 | KC_ROW *in = (KC_ROW *) PG_GETARG_POINTER(0); 164 | text *tout; 165 | 166 | tout = cstring_to_text(in->psource); 167 | PG_RETURN_TEXT_P(tout); 168 | } 169 | 170 | PG_FUNCTION_INFO_V1(kc_sum); 171 | 172 | Datum kc_sum(PG_FUNCTION_ARGS) { 173 | int64_t total = PG_GETARG_INT64(0); 174 | KC_ROW *in = (KC_ROW *) PG_GETARG_POINTER(1); 175 | int64_t res = total + in->value; 176 | 177 | PG_RETURN_INT64(res); 178 | } 179 | 180 | // Open a DB. 181 | bool open_db (KCDB *db, char *map_type, char *start_time) { 182 | // Figure out what db to open. 183 | char db_buffer[512]; 184 | snprintf(db_buffer, 256, "%s/%s/pg_agg_map_%s.%s", BASE_DB_DIR, start_time, map_type, DB_TYPE); 185 | if (!kcdbopen(db, db_buffer, KCOWRITER)) { 186 | #ifdef CF_NO_DB_IS_ERR 187 | ereport(ERROR, 188 | (errcode(ERRCODE_EXTERNAL_ROUTINE_EXCEPTION), 189 | errmsg("Error opening db: \"%s\", \"%s\". Make sure that the map_name is valid.", 190 | db_buffer, kcecodename(kcdbecode(db))))); 191 | #endif 192 | #ifdef CF_DUBUG 193 | ereport(NOTICE, 194 | (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), 195 | errmsg("Error opening db: \"%s\", \"%s\". Make sure that the map_name is valid.", 196 | db_buffer, kcecodename(kcdbecode(db))))); 197 | #endif 198 | return false; 199 | } 200 | 201 | #ifdef CF_DUBUG 202 | ereport(NOTICE, 203 | (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), 204 | errmsg("Opened db -- %s.", db_buffer))); 205 | #endif 206 | 207 | return true; 208 | } 209 | 210 | /** 211 | General idea is to provide an interface into a map_name:doctype:pop:psource key stored tree table. 212 | */ 213 | 214 | PG_FUNCTION_INFO_V1(kc_expand); 215 | 216 | Datum kc_expand(PG_FUNCTION_ARGS) { 217 | 218 | KC_ENTRY *search; 219 | FuncCallContext *funcctx; 220 | int call_cntr; 221 | char *kbuf; 222 | size_t ksiz, vsiz; 223 | const char *cvbuf; 224 | char *kv_kbuf = NULL; 225 | size_t kv_ksiz; 226 | int done; 227 | 228 | /* stuff done only on the first call of the function */ 229 | if (SRF_IS_FIRSTCALL()) { 230 | MemoryContext oldcontext; 231 | 232 | /* create a function context for cross-call persistence */ 233 | funcctx = SRF_FIRSTCALL_INIT(); 234 | 235 | /* switch to memory context appropriate for multiple function calls */ 236 | oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx); 237 | 238 | // Make sure that there are enough args. 239 | if (PG_NARGS() < MIN_ARGS) { 240 | ereport(ERROR, 241 | (errcode(ERRCODE_EXTERNAL_ROUTINE_INVOCATION_EXCEPTION), 242 | errmsg("Must run expand with at least %d args!", MIN_ARGS))); 243 | } 244 | 245 | /* Make the kcdb here. */ 246 | search = (KC_ENTRY *)palloc(sizeof(KC_ENTRY)); 247 | search->db = kcdbnew(); 248 | if (open_db (search->db, text_to_cstring(PG_GETARG_TEXT_PP(0)), text_to_cstring(PG_GETARG_TEXT_PP(1)))) { 249 | 250 | // Set the key to jump into: 251 | // Call with -- map_name, result_id, class, doctype, pop, psource 252 | // Here, map_name describes a db to open. 253 | // Otherwise, result_id:class:doctype:pop:psource 254 | (search->jump_key) = (char *) palloc(MAX_JUMP_KEY_LEN * sizeof(char)); 255 | 256 | int index_point; 257 | search->jump_key = text_to_cstring(PG_GETARG_TEXT_PP(2)); 258 | int size_left = MAX_JUMP_KEY_LEN; 259 | for (index_point = START_VARIABLE_INDEX; index_point < END_VARIABLE_INDEX; index_point++) { 260 | if (PG_NARGS() > index_point) { 261 | char *next_idx = text_to_cstring(PG_GETARG_TEXT_PP(index_point)); 262 | if (next_idx != NULL) { 263 | size_left = size_left - (2 + strlen(next_idx)); 264 | strncat (search->jump_key, CF_LABEL_SEP, size_left); 265 | strncat (search->jump_key, next_idx, size_left); 266 | } 267 | } 268 | } 269 | 270 | #ifdef CF_DUBUG 271 | ereport(NOTICE, 272 | (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), 273 | errmsg("Setting jump buffer -- [%s]", search->jump_key))); 274 | #endif 275 | 276 | // Create a cursor, and set it to the base point looking for entries. 277 | search->cur = kcdbcursor(search->db); 278 | kccurjumpkey(search->cur, search->jump_key, MAX_JUMP_KEY_LEN); 279 | } else { 280 | search->db = NULL; 281 | } 282 | 283 | search->next_map = 0; 284 | search->msg = NULL; 285 | 286 | // Save the search struct for the subsequent calls. 287 | funcctx->user_fctx = search; 288 | 289 | MemoryContextSwitchTo(oldcontext); 290 | } 291 | 292 | /* stuff done on every call of the function */ 293 | funcctx = SRF_PERCALL_SETUP(); 294 | 295 | call_cntr = funcctx->call_cntr; 296 | search = (KC_ENTRY *) funcctx->user_fctx; 297 | 298 | // If no current msg, try to get the next one. 299 | done = 1; 300 | 301 | #ifdef CF_DUBUG 302 | ereport(NOTICE, 303 | (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), 304 | errmsg("beginning run"))); 305 | #endif 306 | 307 | if (search->msg) { 308 | 309 | #ifdef CF_DUBUG 310 | ereport(NOTICE, 311 | (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), 312 | errmsg("Incrementing next from map %d -- %zu", search->next_map, search->msg->n_map_entry))); 313 | #endif 314 | 315 | // Case if we are using the external cursor running over kv map. 316 | // Ready the next 317 | if (search->msg->kv_map_file) { 318 | 319 | if ((kv_kbuf = kccurgetkey(search->kv_cur, &kv_ksiz, 1)) == NULL) { 320 | done = 1; 321 | kccurdel(search->kv_cur); 322 | kcdbendtran (search->kv_db, 1); 323 | if (!kcdbclose(search->kv_db)) { 324 | ereport(ERROR, 325 | (errcode(ERRCODE_EXTERNAL_ROUTINE_EXCEPTION), 326 | errmsg("Error Closeing db: \"%s\"", kcecodename(kcdbecode(search->kv_db))))); 327 | } 328 | 329 | // Also need to free this. 330 | cloudflare__zone_time_bucket__free_unpacked(search->msg, NULL); 331 | search->msg = NULL; 332 | 333 | } else { 334 | done = 0; 335 | } 336 | 337 | 338 | } else { 339 | if (search->next_map >= search->msg->n_map_entry) { 340 | // Done with this msg -- move on to the next one. 341 | cloudflare__zone_time_bucket__free_unpacked(search->msg, NULL); 342 | search->msg = NULL; 343 | } else { 344 | done = 0; 345 | } 346 | } 347 | } 348 | 349 | if (search->db && !search->msg) { 350 | 351 | #ifdef CF_DUBUG 352 | ereport(NOTICE, 353 | (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), 354 | errmsg("Getting new buf -- %s", search->jump_key))); 355 | #endif 356 | 357 | if ((kbuf = kccurget(search->cur, &ksiz, &cvbuf, &vsiz, 1)) != NULL) { 358 | // Pull up the PB and expand it. 359 | search->msg = cloudflare__zone_time_bucket__unpack(NULL, vsiz, (const uint8_t *)cvbuf); 360 | if (search->msg == NULL) { // Something failed 361 | ereport(ERROR, 362 | (errcode(ERRCODE_EXTERNAL_ROUTINE_EXCEPTION), 363 | errmsg("error unpacking incoming message"))); 364 | done = 1; 365 | } else { 366 | // Does the buffer match the searched for string? 367 | // @TODO -- bound this? 368 | if (strstr(search->msg->db_key, search->jump_key)) { 369 | done = 0; 370 | search->next_map = 0; 371 | 372 | // And load the kvkc if needed. 373 | if (search->msg->kv_map_file) { 374 | 375 | #ifdef CF_DUBUG 376 | ereport(NOTICE, 377 | (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), 378 | errmsg("Switching to kvs %s", search->msg->kv_map_file))); 379 | #endif 380 | 381 | search->kv_db = kcdbnew(); 382 | 383 | if (!kcdbopen(search->kv_db, search->msg->kv_map_file, KCOWRITER)) { 384 | #ifdef CF_NO_DB_IS_ERR 385 | ereport(ERROR, 386 | (errcode(ERRCODE_EXTERNAL_ROUTINE_EXCEPTION), 387 | errmsg("Error opening db: \"%s\", \"%s\". Make sure that the map_name is valid.", 388 | search->msg->kv_map_file, kcecodename(kcdbecode(search->kv_db))))); 389 | #endif 390 | #ifdef CF_DUBUG 391 | ereport(NOTICE, 392 | (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), 393 | errmsg("Error opening db: \"%s\", \"%s\". Make sure that the map_name is valid.", 394 | search->msg->kv_map_file, kcecodename(kcdbecode(search->kv_db))))); 395 | #endif 396 | done = 1; 397 | } else { 398 | kcdbbegintran (search->kv_db, 0); 399 | search->kv_cur = kcdbcursor(search->kv_db); 400 | kccurjump(search->kv_cur); 401 | 402 | if ((kv_kbuf = kccurgetkey(search->kv_cur, &kv_ksiz, 1)) == NULL) { 403 | done = 1; 404 | kccurdel(search->kv_cur); 405 | kcdbendtran (search->kv_db, 1); 406 | if (!kcdbclose(search->kv_db)) { 407 | ereport(ERROR, 408 | (errcode(ERRCODE_EXTERNAL_ROUTINE_EXCEPTION), 409 | errmsg("Error Closeing db: \"%s\"", kcecodename(kcdbecode(search->kv_db))))); 410 | } 411 | } else { 412 | done = 0; 413 | } 414 | } 415 | } 416 | } else { 417 | done = 1; 418 | } 419 | } 420 | kcfree(kbuf); 421 | } else { 422 | #ifdef CF_DUBUG 423 | ereport(NOTICE, 424 | (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), 425 | errmsg("no msg to find"))); 426 | #endif 427 | done = 1; 428 | } 429 | } 430 | 431 | #ifdef CF_DUBUG 432 | ereport(NOTICE, 433 | (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), 434 | errmsg("Done? %d -- next buf -- %d", done, search->next_map))); 435 | #endif 436 | 437 | // Take the next itteration over the cursor. If the next is NULL or else not matching the resultid passed in 438 | // End. Otherwise, parse the value, populating the next row of the returning tuple. 439 | if (!done) { 440 | KC_ROW *out; 441 | Datum result; 442 | 443 | size_t size = sizeof(KC_ROW); 444 | out = (KC_ROW *)palloc(size); 445 | memset(out, '0', size); 446 | SET_VARSIZE(out, size); 447 | 448 | out->classification = (char *)palloc(MAX_KC_ROW_ENTRY * sizeof(char)); 449 | out->doctype = (char *)palloc(MAX_KC_ROW_ENTRY * sizeof(char)); 450 | out->pop = (char *)palloc(MAX_KC_ROW_ENTRY * sizeof(char)); 451 | out->psource = (char *)palloc(MAX_KC_ROW_ENTRY * sizeof(char)); 452 | out->key = (char *)palloc(MAX_KC_ROW_ENTRY * sizeof(char)); 453 | 454 | strncpy(out->classification, search->msg->classification, MAX_KC_ROW_ENTRY); 455 | strncpy(out->doctype, search->msg->doctype, MAX_KC_ROW_ENTRY); 456 | strncpy(out->pop, search->msg->pop, MAX_KC_ROW_ENTRY); 457 | strncpy(out->psource, search->msg->psource, MAX_KC_ROW_ENTRY); 458 | 459 | if (search->msg->kv_map_file) { 460 | 461 | #ifdef CF_DUBUG 462 | ereport(NOTICE, 463 | (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), 464 | errmsg("getting val from -- [%s]", search->msg->kv_map_file))); 465 | #endif 466 | 467 | snprintf(out->key, MAX_KC_ROW_ENTRY, "%s", kv_kbuf); 468 | out->value = kcdbincrint (search->kv_db, kv_kbuf, kv_ksiz, 0); 469 | 470 | if (out->value == INT64_MIN) { 471 | ereport(NOTICE, 472 | (errcode(ERRCODE_EXTERNAL_ROUTINE_EXCEPTION), 473 | errmsg("ERROR Getting val from key -- [%s], %s", kv_kbuf, kcecodename(kcdbecode(search->kv_db))))); 474 | } 475 | 476 | kcfree(kv_kbuf); 477 | } else { 478 | 479 | #ifdef CF_DUBUG 480 | ereport(NOTICE, 481 | (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), 482 | errmsg("Loading %s %ld", search->msg->map_entry[search->next_map]->key, 483 | search->msg->map_entry[search->next_map]->value))); 484 | #endif 485 | 486 | snprintf(out->key, MAX_KC_ROW_ENTRY, "%s", search->msg->map_entry[search->next_map]->key); 487 | out->value = search->msg->map_entry[search->next_map]->value; 488 | } 489 | 490 | result = PointerGetDatum(out); 491 | 492 | /* clean up (this is not really necessary) */ 493 | pfree(out->classification); 494 | pfree(out->doctype); 495 | pfree(out->pop); 496 | pfree(out->psource); 497 | pfree(out->key); 498 | pfree(out); 499 | 500 | // Remember that we are going to the next step. 501 | search->next_map++; 502 | 503 | SRF_RETURN_NEXT(funcctx, result); 504 | } else { /* do when there is no more left */ 505 | if (search->db) { 506 | kccurdel(search->cur); 507 | if (!kcdbclose(search->db)) { 508 | ereport(ERROR, 509 | (errcode(ERRCODE_EXTERNAL_ROUTINE_EXCEPTION), 510 | errmsg("Error Closeing db: \"%s\"", kcecodename(kcdbecode(search->db))))); 511 | } 512 | 513 | if (search->msg != NULL) { 514 | cloudflare__zone_time_bucket__free_unpacked(search->msg, NULL); 515 | } 516 | 517 | pfree(search->jump_key); 518 | } 519 | pfree(search); 520 | 521 | #ifdef CF_DUBUG 522 | ereport(NOTICE, 523 | (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), 524 | errmsg("Done with run"))); 525 | #endif 526 | 527 | // Don't delete db, this leads to segfaults. 528 | SRF_RETURN_DONE(funcctx); 529 | } 530 | } 531 | 532 | inline void set_kv_path(Cloudflare__ZoneTimeBucket *msg, char *map_name, char *start_time_uid, KCDB* db) { 533 | 534 | struct stat path_buf; 535 | char dir_buf[KC_MAX_RID]; 536 | char *found; 537 | 538 | msg->kv_map_file = (char *)palloc(KC_MAX_RID * sizeof(char)); 539 | snprintf(msg->kv_map_file, KC_MAX_RID, "%s/%s/%s/%s.%s", 540 | CF_KV_MAP_BASE, 541 | start_time_uid, 542 | map_name, 543 | msg->db_key, 544 | DB_TYPE); 545 | 546 | snprintf(dir_buf, KC_MAX_RID, "%s", CF_KV_MAP_BASE); 547 | if (stat(dir_buf, &path_buf) != 0) { 548 | mkdir (dir_buf, NEW_DIR_MASK); 549 | } 550 | chmod (dir_buf, NEW_DIR_MODE); 551 | 552 | found = strtok (start_time_uid, "/"); 553 | while (found != NULL) { 554 | strncat (dir_buf, "/", KC_MAX_RID); 555 | strncat (dir_buf, found, KC_MAX_RID); 556 | if (stat(dir_buf, &path_buf) != 0) { 557 | mkdir (dir_buf, NEW_DIR_MODE); 558 | } 559 | chmod (dir_buf, NEW_DIR_MODE); 560 | found = strtok (NULL, "/"); 561 | } 562 | strncat (dir_buf, "/", KC_MAX_RID); 563 | strncat (dir_buf, map_name, KC_MAX_RID); 564 | if (stat(dir_buf, &path_buf) != 0) { 565 | mkdir (dir_buf, NEW_DIR_MODE); 566 | } 567 | chmod (dir_buf, NEW_DIR_MODE); 568 | 569 | if (!kcdbopen(db, msg->kv_map_file, KCOWRITER | KCOCREATE)) { 570 | ereport(ERROR, 571 | (errcode(ERRCODE_EXTERNAL_ROUTINE_EXCEPTION), 572 | errmsg("KV open error for: %s %s\n", msg->kv_map_file, kcecodename(kcdbecode(db))))); 573 | } 574 | kcdbbegintran (db, 0); 575 | kcdbclear (db); 576 | chmod (msg->kv_map_file, FILE_MODE); 577 | } 578 | 579 | int merge_using_kv_map(Cloudflare__ZoneTimeBucket *msg_old, Cloudflare__ZoneTimeBucket *msg_new, KCDB* db) { 580 | 581 | unsigned int i; 582 | int num_new = 0; 583 | if (msg_new->kv_map_file) { 584 | // Open a db for the new map file. 585 | KCCUR* cur; 586 | KCDB* new_db; 587 | new_db = kcdbnew(); 588 | /* open the database */ 589 | if (!kcdbopen(new_db, msg_new->kv_map_file, KCOWRITER)) { 590 | ereport(ERROR, 591 | (errcode(ERRCODE_EXTERNAL_ROUTINE_EXCEPTION), 592 | errmsg("open error: %s", kcecodename(kcdbecode(new_db))))); 593 | } 594 | 595 | cur = kcdbcursor(new_db); 596 | kccurjump(cur); 597 | 598 | char *kv_kbuf; 599 | size_t kv_ksiz; 600 | while ((kv_kbuf = kccurgetkey(cur, &kv_ksiz, 1)) != NULL) { 601 | int64_t value = kcdbincrint (new_db, kv_kbuf, kv_ksiz, 0); 602 | 603 | #ifdef CF_DUBUG 604 | ereport(NOTICE, 605 | (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), 606 | errmsg("Saving: %s -- %ld", kv_kbuf, value))); 607 | #endif 608 | 609 | kcdbincrint (db, kv_kbuf, kv_ksiz, value); 610 | num_new++; 611 | kcfree(kv_kbuf); 612 | } 613 | kccurdel(cur); 614 | kcdbclose(new_db); 615 | } else { 616 | for (i = 0; i < msg_new->n_map_entry; i++) { 617 | kcdbincrint (db, msg_new->map_entry[i]->key, strlen(msg_new->map_entry[i]->key), msg_new->map_entry[i]->value); 618 | num_new++; 619 | } 620 | } 621 | 622 | return num_new; 623 | } 624 | 625 | int merge_messages_basic(Cloudflare__ZoneTimeBucket *msg_old, Cloudflare__ZoneTimeBucket *msg_new) { 626 | // Iterate over the list of the new, adding to the old. 627 | unsigned int i,j,last_entry; 628 | int num_new = 0; 629 | bool found = false; 630 | 631 | last_entry = msg_old->n_map_entry; 632 | for (i = 0; i < msg_new->n_map_entry; i++) { 633 | found = false; 634 | for (j = 0; j < last_entry; j++) { 635 | if (strcmp(msg_old->map_entry[j]->key, msg_new->map_entry[i]->key) == 0) { 636 | msg_old->map_entry[j]->value += msg_new->map_entry[i]->value; 637 | found = true; 638 | break; 639 | } 640 | } 641 | 642 | // If we get here, it is a new key. Copy over the info. 643 | if (!found) { 644 | msg_old->map_entry[msg_old->n_map_entry] = palloc(sizeof(Cloudflare__ZoneTimeBucket__Counter)); 645 | memcpy (msg_old->map_entry[msg_old->n_map_entry], msg_new->map_entry[i], sizeof(Cloudflare__ZoneTimeBucket__Counter)); 646 | msg_old->map_entry[msg_old->n_map_entry]->key = palloc(strlen(msg_new->map_entry[i]->key) + 1); 647 | strcpy(msg_old->map_entry[msg_old->n_map_entry]->key, msg_new->map_entry[i]->key); 648 | msg_old->n_map_entry++; 649 | num_new++; 650 | } 651 | } 652 | return num_new; 653 | } 654 | 655 | // Now, a function to shrink KC values down. Need to open in write mode for this ... 656 | // map_name, new rid, array of rids, class, doctype, pop, psource 657 | // Idea is to: 658 | // Make a new buffer, with the passed in value. 659 | // Iterate over the list of rids, doing rid:class:doctype:pop:psource. Get buffer. 660 | // Go over hashmap, adding to the buffers map. 661 | // Save using the new buffer info. 662 | 663 | PG_FUNCTION_INFO_V1(kc_shrink); 664 | 665 | Datum kc_shrink(PG_FUNCTION_ARGS) { 666 | 667 | char *map_name = text_to_cstring(PG_GETARG_TEXT_PP(0)); 668 | char *start_time = text_to_cstring(PG_GETARG_TEXT_PP(1)); // Start time + uid!!! 669 | char *new_rid = text_to_cstring(PG_GETARG_TEXT_PP(2)); 670 | ArrayType *old_rids = PG_GETARG_ARRAYTYPE_P(3); 671 | char *classification = text_to_cstring(PG_GETARG_TEXT_PP(4)); 672 | char *doctype = text_to_cstring(PG_GETARG_TEXT_PP(5)); 673 | char *pop = text_to_cstring(PG_GETARG_TEXT_PP(6)); 674 | char *psource = text_to_cstring(PG_GETARG_TEXT_PP(7)); 675 | text *tout; 676 | int i,j; 677 | Datum *rid_datums; 678 | bool *rid_nulls; 679 | int rid_count; 680 | char *next_rid; 681 | KCDB *main_db; 682 | char *vbuf; 683 | size_t vsiz; 684 | 685 | // Open our DB. 686 | main_db = kcdbnew(); 687 | if (!open_db (main_db, map_name, start_time)) { 688 | tout = cstring_to_text(new_rid); 689 | PG_RETURN_TEXT_P(tout); 690 | } 691 | kcdbbegintran (main_db, 0); 692 | 693 | // First fill in what we can from the input. 694 | Cloudflare__ZoneTimeBucket msg = CLOUDFLARE__ZONE_TIME_BUCKET__INIT; 695 | 696 | msg.map_name = (char *)palloc(MAX_KC_ROW_ENTRY * sizeof(char)); 697 | msg.doctype = (char *)palloc(MAX_KC_ROW_ENTRY * sizeof(char)); 698 | msg.classification = (char *)palloc(MAX_KC_ROW_ENTRY * sizeof(char)); 699 | msg.pop = (char *)palloc(MAX_KC_ROW_ENTRY * sizeof(char)); 700 | msg.psource = (char *)palloc(MAX_KC_ROW_ENTRY * sizeof(char)); 701 | msg.result_id = (char *)palloc(MAX_KC_ROW_ENTRY * sizeof(char)); 702 | msg.db_key = (char *)palloc(MAX_KC_ROW_ENTRY * sizeof(char)); 703 | msg.db_path = (char *)palloc(MAX_KC_ROW_ENTRY * sizeof(char)); 704 | msg.map_entry = palloc(MAX_KEYS_BEFORE_KV_MAP * sizeof(Cloudflare__ZoneTimeBucket__Counter)); 705 | msg.n_map_entry = 0; 706 | 707 | strncpy(msg.map_name, map_name, MAX_KC_ROW_ENTRY); 708 | strncpy(msg.classification, classification, MAX_KC_ROW_ENTRY); 709 | strncpy(msg.doctype, doctype, MAX_KC_ROW_ENTRY); 710 | strncpy(msg.pop, pop, MAX_KC_ROW_ENTRY); 711 | strncpy(msg.psource, psource, MAX_KC_ROW_ENTRY); 712 | strncpy(msg.result_id, new_rid, KC_MAX_RID); 713 | snprintf(msg.db_path, MAX_KC_ROW_ENTRY, "%s%s%s", 714 | map_name, "/", start_time); 715 | 716 | snprintf(msg.db_key, KC_MAX_RID, "%s%s%s%s%s%s%s%s%s%s%s", 717 | new_rid, CF_LABEL_SEP, 718 | classification, CF_LABEL_SEP, 719 | doctype, CF_LABEL_SEP, 720 | pop, CF_LABEL_SEP, 721 | psource, CF_LABEL_SEP, 722 | map_name); 723 | 724 | // Now run over the array. 725 | deconstruct_array(old_rids, TEXTOID, -1, false, 'i', 726 | &rid_datums, &rid_nulls, &rid_count); 727 | if (ARR_HASNULL(old_rids)) { 728 | ereport(ERROR, 729 | (errcode(ERRCODE_ARRAY_ELEMENT_ERROR), 730 | errmsg("cannot work with arrays containing NULLs"))); 731 | } 732 | 733 | int num_new_keys = 0; 734 | int num_entries = 0; 735 | char keys_to_use[rid_count][KC_MAX_RID]; 736 | Cloudflare__ZoneTimeBucket *msg_new[rid_count]; 737 | j=0; 738 | for (i = 0; i < rid_count; i++) { 739 | next_rid = TextDatumGetCString(rid_datums[i]); 740 | snprintf(keys_to_use[i], KC_MAX_RID, "%s%s%s%s%s%s%s%s%s%s%s", 741 | next_rid, CF_LABEL_SEP, 742 | classification, CF_LABEL_SEP, 743 | doctype, CF_LABEL_SEP, 744 | pop, CF_LABEL_SEP, 745 | psource, CF_LABEL_SEP, 746 | map_name); 747 | 748 | vbuf = kcdbget(main_db, keys_to_use[i], strlen(keys_to_use[i]), &vsiz); 749 | if (vbuf) { 750 | msg_new[j] = cloudflare__zone_time_bucket__unpack(NULL, vsiz, (const uint8_t *)vbuf); 751 | if (msg_new[j] == NULL) { // Something failed 752 | ereport(ERROR, 753 | (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), 754 | errmsg("error unpacking incoming message"))); 755 | } else { 756 | if (msg_new[j]->kv_map_file) { 757 | num_entries = MAX_KEYS_BEFORE_KV_MAP + 1; 758 | } else { 759 | num_entries += msg_new[j]->n_map_entry; 760 | } 761 | j++; 762 | } 763 | kcfree(vbuf); 764 | } else { 765 | #ifdef CF_DUBUG 766 | ereport(NOTICE, 767 | (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), 768 | errmsg("get error on %s -- %s", keys_to_use[i], kcecodename(kcdbecode(main_db))))); 769 | #endif 770 | } 771 | } 772 | 773 | // Now merge the buffers. 774 | KCDB* msg_db = NULL; 775 | if (num_entries > MAX_KEYS_BEFORE_KV_MAP) { 776 | msg_db = kcdbnew(); 777 | set_kv_path(&msg, map_name, start_time, msg_db); 778 | } 779 | 780 | for (i = 0; i < j; i++) { 781 | if (num_entries > MAX_KEYS_BEFORE_KV_MAP) { 782 | num_new_keys += merge_using_kv_map(&msg, msg_new[i], msg_db); 783 | } else { 784 | num_new_keys += merge_messages_basic(&msg, msg_new[i]); 785 | } 786 | cloudflare__zone_time_bucket__free_unpacked(msg_new[i], NULL); 787 | } 788 | 789 | if (num_entries > MAX_KEYS_BEFORE_KV_MAP) { 790 | // Close the db. 791 | kcdbendtran (msg_db, 1); 792 | kcdbclose(msg_db); 793 | } 794 | 795 | #ifdef CF_DUBUG 796 | ereport(NOTICE, 797 | (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), 798 | errmsg("saving: num map entries: %zu -- writting with %d keys", msg.n_map_entry, num_new_keys))); 799 | #endif 800 | 801 | // Save the updated buffer. 802 | if (num_new_keys > 0) { 803 | unsigned int len; 804 | void *buf; 805 | len = cloudflare__zone_time_bucket__get_packed_size (&msg); 806 | buf = palloc (len); 807 | cloudflare__zone_time_bucket__pack (&msg, buf); 808 | if(!kcdbset(main_db, msg.db_key, strlen(msg.db_key), buf, len)) { 809 | ereport(ERROR, 810 | (errcode(ERRCODE_EXTERNAL_ROUTINE_EXCEPTION), 811 | errmsg("set error: %s\n", kcecodename(kcdbecode(main_db))))); 812 | } 813 | pfree (buf); 814 | } 815 | 816 | // Done! 817 | kcdbendtran (main_db, 1); 818 | if (!kcdbclose(main_db)) { 819 | ereport(ERROR, 820 | (errcode(ERRCODE_EXTERNAL_ROUTINE_EXCEPTION), 821 | errmsg("Error Closeing db: \"%s\"", kcecodename(kcdbecode(main_db))))); 822 | } 823 | 824 | tout = cstring_to_text(new_rid); 825 | PG_RETURN_TEXT_P(tout); 826 | } 827 | 828 | 829 | // Delete all entries in the db for the given map and prefix + any rid passed in. 830 | 831 | PG_FUNCTION_INFO_V1(kc_delete); 832 | 833 | Datum kc_delete(PG_FUNCTION_ARGS) { 834 | 835 | char *map_name = text_to_cstring(PG_GETARG_TEXT_PP(0)); 836 | char *start_time = text_to_cstring(PG_GETARG_TEXT_PP(1)); 837 | ArrayType *rids = PG_GETARG_ARRAYTYPE_P(2); 838 | int i; 839 | Datum *rid_datums; 840 | bool *rid_nulls; 841 | int rid_count; 842 | char *next_rid; 843 | KCDB *main_db; 844 | char *vbuf; 845 | size_t vsiz; 846 | int64_t num_keys_to_run; 847 | int64_t num_keys_deleted; 848 | char **keys_to_use; 849 | Cloudflare__ZoneTimeBucket *msg_new; 850 | 851 | // Open our DB. 852 | main_db = kcdbnew(); 853 | if (!open_db (main_db, map_name, start_time)) { 854 | PG_RETURN_INT64(0); 855 | } 856 | kcdbbegintran (main_db, 0); 857 | 858 | // Now run over the array. 859 | deconstruct_array(rids, TEXTOID, -1, false, 'i', 860 | &rid_datums, &rid_nulls, &rid_count); 861 | if (ARR_HASNULL(rids)) { 862 | ereport(ERROR, 863 | (errcode(ERRCODE_ARRAY_ELEMENT_ERROR), 864 | errmsg("cannot work with arrays containing NULLs"))); 865 | } 866 | 867 | keys_to_use = (char **)palloc(KC_MAX_ENTRIES_PER_RID * sizeof(char)); 868 | num_keys_deleted = 0; 869 | char prefixes_to_use[rid_count][KC_MAX_RID]; 870 | 871 | for (i = 0; i < rid_count; i++) { 872 | next_rid = TextDatumGetCString(rid_datums[i]); 873 | snprintf(prefixes_to_use[i], KC_MAX_RID, "%s%s", next_rid, CF_LABEL_SEP); 874 | num_keys_to_run = kcdbmatchprefix (main_db, prefixes_to_use[i], keys_to_use, KC_MAX_ENTRIES_PER_RID); 875 | if (num_keys_to_run != -1) { 876 | num_keys_deleted += num_keys_to_run; 877 | int next_key; 878 | for (next_key=0; next_key < num_keys_to_run; next_key++) { 879 | vbuf = kcdbget(main_db, keys_to_use[next_key], strlen(keys_to_use[next_key]), &vsiz); 880 | if (vbuf) { 881 | msg_new = cloudflare__zone_time_bucket__unpack(NULL, vsiz, (const uint8_t *)vbuf); 882 | if (msg_new == NULL) { // Something failed 883 | ereport(ERROR, 884 | (errcode(ERRCODE_EXTERNAL_ROUTINE_EXCEPTION), 885 | errmsg("error unpacking incoming message"))); 886 | } else { 887 | if (msg_new->kv_map_file) { 888 | unlink(msg_new->kv_map_file); 889 | } 890 | kcdbremove (main_db, keys_to_use[next_key], strlen(keys_to_use[next_key])); 891 | } 892 | cloudflare__zone_time_bucket__free_unpacked(msg_new, NULL); 893 | kcfree(vbuf); 894 | kcfree(keys_to_use[next_key]); 895 | } else { 896 | ereport(NOTICE, 897 | (errcode(ERRCODE_EXTERNAL_ROUTINE_EXCEPTION), 898 | errmsg("get error on %s -- %s", keys_to_use[next_key], kcecodename(kcdbecode(main_db))))); 899 | } 900 | } 901 | } else { 902 | ereport(NOTICE, 903 | (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), 904 | errmsg("prefix error on %s -- %s", prefixes_to_use[i], kcecodename(kcdbecode(main_db))))); 905 | } 906 | } 907 | 908 | pfree(keys_to_use); 909 | 910 | // Done! 911 | kcdbendtran (main_db, 1); 912 | if (!kcdbclose(main_db)) { 913 | ereport(ERROR, 914 | (errcode(ERRCODE_EXTERNAL_ROUTINE_EXCEPTION), 915 | errmsg("Error Closeing db: \"%s\"", kcecodename(kcdbecode(main_db))))); 916 | } 917 | 918 | PG_RETURN_INT64(num_keys_deleted); 919 | } 920 | -------------------------------------------------------------------------------- /pg_kc.sql: -------------------------------------------------------------------------------- 1 | CREATE LANGUAGE plpgsql; 2 | CREATE TYPE kc_row; 3 | 4 | DROP FUNCTION kc_row_in(cstring) CASCADE; 5 | CREATE FUNCTION kc_row_in(cstring) 6 | RETURNS kc_row 7 | AS 'libpg_kc', 'kc_row_in' 8 | LANGUAGE C IMMUTABLE STRICT; 9 | 10 | DROP FUNCTION kc_row_out(kc_row) CASCADE; 11 | CREATE FUNCTION kc_row_out(kc_row) 12 | RETURNS cstring 13 | AS 'libpg_kc' , 'kc_row_out' 14 | LANGUAGE C IMMUTABLE STRICT; 15 | 16 | CREATE TYPE kc_row ( 17 | INTERNALLENGTH = variable, 18 | INPUT = kc_row_in, 19 | OUTPUT = kc_row_out, 20 | ALIGNMENT = double, 21 | STORAGE = plain 22 | ); 23 | 24 | DROP FUNCTION kc_key(kc_row); 25 | CREATE FUNCTION kc_key(kc_row) 26 | RETURNS text 27 | AS 'libpg_kc', 'kc_key' 28 | LANGUAGE C IMMUTABLE STRICT; 29 | 30 | CREATE FUNCTION kc_val(kc_row) 31 | RETURNS bigint 32 | AS 'libpg_kc', 'kc_val' 33 | LANGUAGE C IMMUTABLE STRICT; 34 | 35 | DROP FUNCTION kc_class(kc_row); 36 | CREATE FUNCTION kc_class(kc_row) 37 | RETURNS text 38 | AS 'libpg_kc', 'kc_class' 39 | LANGUAGE C IMMUTABLE STRICT; 40 | 41 | DROP FUNCTION kc_doctype(kc_row); 42 | CREATE FUNCTION kc_doctype(kc_row) 43 | RETURNS text 44 | AS 'libpg_kc', 'kc_doctype' 45 | LANGUAGE C IMMUTABLE STRICT; 46 | 47 | DROP FUNCTION kc_pop(kc_row); 48 | CREATE FUNCTION kc_pop(kc_row) 49 | RETURNS text 50 | AS 'libpg_kc', 'kc_pop' 51 | LANGUAGE C IMMUTABLE STRICT; 52 | 53 | DROP FUNCTION kc_psource(kc_row); 54 | CREATE FUNCTION kc_psource(kc_row) 55 | RETURNS text 56 | AS 'libpg_kc', 'kc_psource' 57 | LANGUAGE C IMMUTABLE STRICT; 58 | 59 | CREATE OR REPLACE FUNCTION kc_sum(bigint, kc_row) 60 | RETURNS bigint 61 | AS 'libpg_kc', 'kc_sum' 62 | LANGUAGE C IMMUTABLE; 63 | 64 | CREATE AGGREGATE sum ( kc_row ) ( 65 | SFUNC = kc_sum, 66 | INITCOND = 0, 67 | STYPE = bigint 68 | ); 69 | 70 | CREATE OR REPLACE FUNCTION kcts(__start_time TIMESTAMPTZ, __oid BIGINT) RETURNS text AS $_$ 71 | BEGIN 72 | RETURN to_char(__start_time, 'YYYY-MM-DD') || '/' || __oid; 73 | END; 74 | $_$ Language 'plpgsql' IMMUTABLE STRICT; 75 | 76 | /** 77 | Returns multiple rows, being the expanded rows for every map found by the passed in inputs. 78 | Indexed by: result_id:map_name:doctype:pop:psource 79 | 80 | Call with -- result_id, map_name, doctype, pop, psource 81 | Everything from map_name down can be null. 82 | 83 | Return map_name doctype pop psource key value list 84 | */ 85 | 86 | DROP FUNCTION kcx(text, text, text); 87 | CREATE OR REPLACE FUNCTION kcx(text, text, text) 88 | RETURNS SETOF kc_row 89 | AS 'libpg_kc', 'kc_expand' 90 | LANGUAGE C STABLE STRICT; 91 | 92 | DROP FUNCTION kcx(text, text, text, text); 93 | CREATE OR REPLACE FUNCTION kcx(IN text, IN text, IN text, IN text) 94 | RETURNS SETOF kc_row 95 | AS 'libpg_kc', 'kc_expand' 96 | LANGUAGE C STABLE STRICT; 97 | 98 | DROP FUNCTION kcx(text, text, text, text, text); 99 | CREATE OR REPLACE FUNCTION kcx(IN text, IN text, IN text, IN text, IN text) 100 | RETURNS SETOF kc_row 101 | AS 'libpg_kc', 'kc_expand' 102 | LANGUAGE C STABLE STRICT; 103 | 104 | DROP FUNCTION kcx(text, text, text, text, text, text); 105 | CREATE OR REPLACE FUNCTION kcx(IN text, IN text, IN text, IN text, IN text, IN text) 106 | RETURNS SETOF kc_row 107 | AS 'libpg_kc', 'kc_expand' 108 | LANGUAGE C STABLE STRICT; 109 | 110 | DROP FUNCTION kcx(text, text, text, text, text, text, text); 111 | CREATE OR REPLACE FUNCTION kcx(IN text, IN text, IN text, IN text, IN text, IN text, IN text) 112 | RETURNS SETOF kc_row 113 | AS 'libpg_kc', 'kc_expand' 114 | LANGUAGE C STABLE STRICT; 115 | 116 | /** 117 | Given the fixed values, writes (overrites) the kc entry. 118 | map_name, start_time, new rid, array of rids, class, doctype, pop, psource 119 | returns new kcid. 120 | */ 121 | DROP FUNCTION kcs (text, text, text, text array, text, text, text, text); 122 | CREATE FUNCTION kcs (text, text, text, text array, text, text, text, text) 123 | RETURNS text 124 | AS 'libpg_kc', 'kc_shrink' 125 | LANGUAGE C STABLE STRICT; 126 | 127 | /** 128 | Also need a way to delete values in the kcdb 129 | */ 130 | DROP FUNCTION kcd (text, text, text array); 131 | CREATE FUNCTION kcd (text, text, text array) 132 | RETURNS bigint 133 | AS 'libpg_kc', 'kc_delete' 134 | LANGUAGE C STABLE STRICT; 135 | --------------------------------------------------------------------------------