├── .gitignore ├── CHANGES ├── LICENSE ├── README.md ├── VERSION ├── zeek-agent ├── __load__.zeek ├── examples │ ├── auditd │ │ ├── __load__.zeek │ │ ├── process-start.zeek │ │ └── socket-open.zeek │ ├── endpointsecurity │ │ ├── __load__.zeek │ │ └── process-start.zeek │ └── osquery │ │ ├── __load__.zeek │ │ ├── interfaces.zeek │ │ ├── listening-ports.zeek │ │ ├── mounts.zeek │ │ └── users.zeek └── framework │ ├── __load__.zeek │ ├── bro_backend.zeek │ ├── configuration.zeek │ ├── framework_commons.zeek │ ├── hosts_send.zeek │ ├── zeek_agent_framework.zeek │ ├── zeek_agent_hosts.zeek │ ├── zeek_agent_subscriptions.zeek │ ├── zeek_logger.zeek │ └── zeek_logger_log.zeek └── zkg.meta /.gitignore: -------------------------------------------------------------------------------- 1 | build 2 | tmp 3 | -------------------------------------------------------------------------------- /CHANGES: -------------------------------------------------------------------------------- 1 | 2 | 0.4-22 | 2020-02-27 11:07:08 -0500 3 | 4 | * Update & extend example scripts. (Seth Hall, Corelight) 5 | 6 | * Framework cleanup. (Seth Hall, Corelight) 7 | 8 | 0.4-10 | 2020-01-08 11:42:42 +0000 9 | 10 | * Rename from osquery framework to zeek-agent. (Robin Sommer, Corelight) 11 | 12 | * Rewrite the process_open_sockets scripts. (Alessandro Gario) 13 | 14 | * Print Zeek Agent version and edition on connection. (Alessandro Gario) 15 | 16 | * Updates the table scripts for socket_events, zeek_logger. (Alessandro Gario) 17 | 18 | * Update README. (Robin Sommer, Corelight) 19 | 20 | 0.4 | 2019-09-23 10:28:33 +0000 21 | 22 | * Initial import. 23 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2019, International Computer Science Institute. All rights reserved. 2 | 3 | Based on code contributed to the Zeek project by Steffen Haas and Corelight, Inc. 4 | 5 | Redistribution and use in source and binary forms, with or without 6 | modification, are permitted provided that the following conditions are met: 7 | 8 | (1) Redistributions of source code must retain the above copyright 9 | notice, this list of conditions and the following disclaimer. 10 | 11 | (2) Redistributions in binary form must reproduce the above copyright 12 | notice, this list of conditions and the following disclaimer in the 13 | documentation and/or other materials provided with the distribution. 14 | 15 | (3) Neither the name of the International Computer Science Institute, 16 | the Zeek project, nor the names of contributors may be used to 17 | endorse or promote products derived from this software without 18 | specific prior written permission. 19 | 20 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 21 | "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 22 | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 23 | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 24 | OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 25 | SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 26 | LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 27 | DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 28 | THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 29 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 30 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 31 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | # About 3 | 4 | This Zeek script framework communicates with the [Zeek 5 | Agent](https://github.com/zeek/zeek-agent) to perform live 6 | queries against the agent's tables and then incorporate the results 7 | back into Zeek's processing & logging. In addition to tables built in, 8 | the agent can connect to [Osquery](https://osquery.io) to retrieve any 9 | of the host data provided there. 10 | 11 | *Note*: This framework is still a work in progress and expected to 12 | change further in terms of API, functionality, and implementation. 13 | 14 | # Prerequisites 15 | 16 | The framework requires Zeek 3.0+, which you can download and install 17 | per the instructions on the [Zeek web site](https://zeek.org/download). 18 | 19 | You will also need to install the Zeek Agent itself, as well as 20 | optionally Osquery, according to [these 21 | instructions](https://github.com/zeek/zeek-agent-framework). 22 | 23 | # Installation 24 | 25 | The easiest way to install the `zeek-agent` framework is through the 26 | [Zeek package 27 | manager](https://docs.zeek.org/projects/package-manager/en/stable/index.html). 28 | If you have not installed the package manager yet, do that first: 29 | 30 | # pip install zkg 31 | # zkg autoconfig 32 | 33 | # zkg install zeek/zeek-agent-framework 34 | 35 | Alternatively, you can clone the repository manually and copy it over 36 | into Zeek's `site` folder: 37 | 38 | # git clone https://github.com/zeek/zeek-agent-framework 39 | # cp -a zeek-agent-framework/zeek-agent $(zeek-config --site_dir) 40 | 41 | If you'd rather run it directly out of the local repository clone 42 | (rather than `site`), set your `ZEEKPATH` accordingly: 43 | 44 | # export ZEEKPATH=:$(zeek-config --zeekpath) 45 | 46 | # Usage 47 | 48 | Using any of the three installation methods above, you can now load 49 | the framework when you start Zeek: 50 | 51 | # zeek zeek-agent 52 | 53 | Once you start up any agents, you should start seeing a new Zeek log 54 | file `zeek-agent.log` that records the hosts connecting to Zeek: 55 | 56 | # cat zeek-agent.log 57 | #fields ts source peer level message 58 | 1576768875.018249 local ZeekMaster info Subscribing to zeek announce topic /zeek/zeek-agent/zeek_announce 59 | 1576768875.018249 local ZeekMaster info Subscribing to zeek individual topic /zeek/zeek-agent/zeek/C6EAF3CFDF46831E2D9103E5A1C48F78AD873A00#10223 60 | 1576768877.709030 local ZeekMaster info Incoming connection established from C6EAF3CFDF46831E2D9103E5A1C48F78AD873A3C#7503 61 | 62 | You won't see much more at first as there's nothing sending queries to 63 | the endhost yet. Check out the `examples/` directory for scripts that 64 | are using the built in (currently Linux audit based) and Osquery based 65 | functionality. 66 | 67 | # Examples 68 | 69 | The framework ships with examples that currently use Osquery derived tables 70 | and Linux auditd based tables. Use the follow lines to load all of the 71 | associated examples. 72 | 73 | To load the Osquery examples: 74 | 75 | @load zeek-agent/examples/osquery 76 | 77 | To load the auditd examples: 78 | 79 | @load zeek-agent/examples/auditd 80 | 81 | To load the EndpointSecurity (MacOS) examples: 82 | 83 | @load zeek-agent/examples/endpointsecurity 84 | 85 | 86 | # Credits 87 | 88 | This Zeek framework is based on an earlier implementation by [Steffen 89 | Haas](https://github.com/iBigQ), with recent work contributed by 90 | [Corelight](https://www.corelight.com) and [Trail of 91 | Bits](https://www.trailofbits.com). 92 | -------------------------------------------------------------------------------- /VERSION: -------------------------------------------------------------------------------- 1 | 0.4-22 2 | -------------------------------------------------------------------------------- /zeek-agent/__load__.zeek: -------------------------------------------------------------------------------- 1 | @load ./framework 2 | -------------------------------------------------------------------------------- /zeek-agent/examples/auditd/__load__.zeek: -------------------------------------------------------------------------------- 1 | 2 | # Load all the tables that are built into zeek-agent. 3 | @load ./process-start 4 | @load ./socket-open 5 | -------------------------------------------------------------------------------- /zeek-agent/examples/auditd/process-start.zeek: -------------------------------------------------------------------------------- 1 | ##! Logs process events activity 2 | 3 | @load zeek-agent 4 | 5 | module Agent_ProcessStart; 6 | 7 | export { 8 | redef enum Log::ID += { LOG }; 9 | 10 | type Info: record { 11 | ts: time &log; 12 | host_ts: time &log; 13 | host: string &log; 14 | hostname: string &log; 15 | pid: int &log; 16 | path: string &log; 17 | cmdline: string &log; 18 | cwd: string &log; 19 | uid: int &log; 20 | gid: int &log; 21 | parent: int &log; 22 | }; 23 | } 24 | 25 | event Agent_ProcessStart::process_start(result: ZeekAgent::Result, 26 | pid: int, path: string, cmdline: string, cwd: string, 27 | uid: int, gid: int, host_time: int, parent: int) 28 | { 29 | if ( result$utype != ZeekAgent::ADD ) 30 | return; 31 | 32 | local host_ts = double_to_time(host_time); 33 | local info = Info($ts = network_time(), 34 | $host_ts = host_ts, 35 | $host = result$host, 36 | $hostname = ZeekAgent::getHostInfo(result$host)$hostname, 37 | $pid = pid, 38 | $path = path, 39 | $cmdline = cmdline, 40 | $cwd = cwd, 41 | $uid = uid, 42 | $gid = gid, 43 | $parent = parent); 44 | 45 | Log::write(LOG, info); 46 | } 47 | 48 | event zeek_init() &priority=10 49 | { 50 | Log::create_stream(LOG, [$columns=Info, $path="agent-process_events"]); 51 | 52 | local query = ZeekAgent::Query($ev=Agent_ProcessStart::process_start, 53 | $query="SELECT pid, path, cmdline, cwd, uid, gid, time, parent FROM process_events", 54 | $utype=ZeekAgent::ADD); 55 | ZeekAgent::subscribe(query); 56 | } 57 | -------------------------------------------------------------------------------- /zeek-agent/examples/auditd/socket-open.zeek: -------------------------------------------------------------------------------- 1 | ##! Logs socket events activity 2 | 3 | @load zeek-agent 4 | 5 | module Agent_SocketOpen; 6 | 7 | export { 8 | redef enum Log::ID += { LOG }; 9 | 10 | type Info: record { 11 | ts: time &log; 12 | host_ts: time &log; 13 | host: string &log; 14 | hostname: string &log; 15 | action: string &log; 16 | pid: int &log; 17 | fd: int &log; 18 | path: string &log; 19 | local_address: addr &log &default=0.0.0.0; 20 | remote_address: addr &log &default=0.0.0.0; 21 | local_port: int &log; 22 | remote_port: int &log; 23 | success: int &log; 24 | }; 25 | } 26 | 27 | event Agent_SocketOpen::socket_open(result: ZeekAgent::Result, 28 | action: string, pid: int, fd: int, path: string, 29 | local_address: string, remote_address: string, 30 | local_port: int, remote_port: int, 31 | host_time: int, success: int) 32 | { 33 | if ( result$utype != ZeekAgent::ADD ) 34 | return; 35 | 36 | local host_ts = double_to_time(host_time); 37 | local info = Info($ts = network_time(), 38 | $host_ts = host_ts, 39 | $host = result$host, 40 | $hostname = ZeekAgent::getHostInfo(result$host)$hostname, 41 | $pid = pid, 42 | $action = action, 43 | $fd = fd, 44 | $path = path, 45 | $local_port = local_port, 46 | $remote_port = remote_port, 47 | $success = success); 48 | 49 | if ( local_address != "" ) 50 | info$local_address = to_addr(local_address); 51 | 52 | if ( remote_address != "" ) 53 | info$remote_address = to_addr(remote_address); 54 | 55 | Log::write(LOG, info); 56 | } 57 | 58 | event zeek_init() &priority=10 59 | { 60 | Log::create_stream(LOG, [$columns=Info, $path="agent-sockets_opening"]); 61 | 62 | local query = ZeekAgent::Query($ev=Agent_SocketOpen::socket_open, 63 | $query="SELECT action, pid, fd, path, local_address, remote_address, local_port, remote_port, time, success FROM socket_events WHERE family=2", 64 | $utype=ZeekAgent::ADD); 65 | ZeekAgent::subscribe(query); 66 | } 67 | -------------------------------------------------------------------------------- /zeek-agent/examples/endpointsecurity/__load__.zeek: -------------------------------------------------------------------------------- 1 | 2 | # Load all the tables that are built into zeek-agent. 3 | @load ./process-start 4 | -------------------------------------------------------------------------------- /zeek-agent/examples/endpointsecurity/process-start.zeek: -------------------------------------------------------------------------------- 1 | ##! Logs process events activity 2 | 3 | @load zeek-agent 4 | 5 | module Agent_ProcessStart; 6 | 7 | export { 8 | redef enum Log::ID += { LOG }; 9 | 10 | type Info: record { 11 | ts: time &log; 12 | host_ts: time &log; 13 | host: string &log; 14 | hostname: string &log; 15 | pid: int &log; 16 | path: string &log; 17 | cmdline: string &log; 18 | uid: int &log; 19 | gid: int &log; 20 | parent: int &log; 21 | platform_binary: int &log; 22 | }; 23 | } 24 | 25 | event Agent_ProcessStart::process_start(result: ZeekAgent::Result, 26 | pid: int, path: string, cmdline: string, 27 | uid: int, gid: int, host_time: int, parent: int, platform_binary: int) 28 | { 29 | if ( result$utype != ZeekAgent::ADD ) 30 | return; 31 | 32 | local host_ts = double_to_time(host_time); 33 | local info = Info($ts = network_time(), 34 | $host_ts = host_ts, 35 | $host = result$host, 36 | $hostname = ZeekAgent::getHostInfo(result$host)$hostname, 37 | $pid = pid, 38 | $path = path, 39 | $cmdline = cmdline, 40 | $uid = uid, 41 | $gid = gid, 42 | $parent = parent, 43 | $platform_binary = platform_binary); 44 | 45 | Log::write(LOG, info); 46 | } 47 | 48 | 49 | event zeek_init() &priority=10 50 | { 51 | Log::create_stream(LOG, [$columns=Info, $path="agent-process_events"]); 52 | 53 | local query = ZeekAgent::Query($ev=Agent_ProcessStart::process_start, 54 | $query="SELECT process_id, path, cmdline, user_id, group_id, timestamp, parent_process_id, platform_binary FROM process_events", 55 | $utype=ZeekAgent::ADD); 56 | ZeekAgent::subscribe(query); 57 | } 58 | -------------------------------------------------------------------------------- /zeek-agent/examples/osquery/__load__.zeek: -------------------------------------------------------------------------------- 1 | 2 | # Load all of the example OSquery based scripts. 3 | 4 | @load ./interfaces 5 | @load ./listening-ports 6 | @load ./mounts 7 | @load ./users 8 | -------------------------------------------------------------------------------- /zeek-agent/examples/osquery/interfaces.zeek: -------------------------------------------------------------------------------- 1 | ##! Query interfaces activity. 2 | 3 | @load zeek-agent 4 | 5 | module AgentInterfaces; 6 | 7 | export { 8 | 9 | redef enum Log::ID += { LOG }; 10 | 11 | type Info: record { 12 | ts: time &log; 13 | host: string &log; 14 | hostname: string &log; 15 | interface: string &log; 16 | mac: string &log &optional; 17 | ip: addr &log; 18 | mask: addr &log &optional; 19 | }; 20 | } 21 | 22 | 23 | event AgentInterfaces::new_interface(result: ZeekAgent::Result, interface: string, mac: string, ip: string, mask: string) 24 | { 25 | # Remove interface name from IP and turn the string into an ip address 26 | local clean_ip = to_addr(split_string(ip, /\%/)[0]); 27 | 28 | local info = Info($ts = network_time(), 29 | $host = result$host, 30 | $hostname = ZeekAgent::getHostInfo(result$host)$hostname, 31 | $interface = interface, 32 | $ip = clean_ip); 33 | 34 | if ( mac != "00:00:00:00:00:00" ) 35 | info$mac = mac; 36 | 37 | if ( mask != "" ) 38 | info$mask = to_addr(mask); 39 | 40 | Log::write(LOG, info); 41 | } 42 | 43 | 44 | event zeek_init() &priority=10 45 | { 46 | Log::create_stream(LOG, [$columns=Info, $path="agent-interfaces"]); 47 | 48 | local query = ZeekAgent::Query($ev=AgentInterfaces::new_interface, 49 | $query="SELECT d.interface, d.mac, a.address, a.mask FROM interface_addresses AS a INNER JOIN interface_details AS d ON a.interface=d.interface", 50 | $utype=ZeekAgent::ADD); 51 | ZeekAgent::subscribe(query); 52 | } 53 | -------------------------------------------------------------------------------- /zeek-agent/examples/osquery/listening-ports.zeek: -------------------------------------------------------------------------------- 1 | #! Query listening ports activity 2 | 3 | @load zeek-agent 4 | 5 | module AgentListeningPorts; 6 | 7 | export { 8 | redef enum Log::ID += { LOG }; 9 | 10 | type Info: record { 11 | ts: time &log; 12 | host: string &log; 13 | hostname: string &log; 14 | ## Indicate if the port was "opened" or "closed". 15 | action: string &log; 16 | address: addr &log; 17 | listen_port: int &log; 18 | protocol: string &log; 19 | pid: int &log; 20 | process_name: string &log; 21 | }; 22 | 23 | const proto_lookup: table[int] of string = { 24 | [6] = "tcp", 25 | [17] = "udp", 26 | [132] = "sctp", 27 | } &redef &default=function(i: int): string { return fmt("unknown-%d", i); }; 28 | } 29 | 30 | event AgentListeningPorts::listening_port(result: ZeekAgent::Result, pid: int, process_name: string, protocol: int, local_addr: string, local_port: int) 31 | { 32 | # Don't log existing open ports. We only want the moment ports are opened or closed. 33 | if ( result$utype == ZeekAgent::INITIAL ) 34 | return; 35 | 36 | # Remove interface name from IP and turn the string into an ip address 37 | local clean_addr = to_addr(split_string(local_addr, /\%/)[0]); 38 | 39 | local info = Info($ts = network_time(), 40 | $host = result$host, 41 | $hostname = ZeekAgent::getHostInfo(result$host)$hostname, 42 | $action = result$utype == ZeekAgent::ADD ? "opened" : "closed", 43 | $pid = pid, 44 | $process_name = process_name, 45 | $protocol = proto_lookup[protocol], 46 | $address = clean_addr, 47 | $listen_port = local_port); 48 | 49 | Log::write(LOG, info); 50 | } 51 | 52 | event zeek_init() &priority=10 53 | { 54 | Log::create_stream(LOG, [$columns=Info, $path="agent-listening-ports"]); 55 | 56 | # Family 2 is INET, so we're only watching for that. 57 | local query = ZeekAgent::Query($ev=AgentListeningPorts::listening_port, 58 | #$query="SELECT pid, protocol, address, port FROM listening_ports WHERE family=2", 59 | $query="SELECT listening_ports.pid, name, protocol, address, port FROM listening_ports LEFT JOIN processes WHERE processes.pid=listening_ports.pid AND family=2 AND address!='127.0.0.1' AND address!='::1';", 60 | $utype=ZeekAgent::BOTH); 61 | 62 | ZeekAgent::subscribe(query); 63 | } 64 | -------------------------------------------------------------------------------- /zeek-agent/examples/osquery/mounts.zeek: -------------------------------------------------------------------------------- 1 | #! Query mounts activity. 2 | 3 | @load zeek-agent 4 | 5 | module AgentMounts; 6 | 7 | export { 8 | redef enum Log::ID += { LOG }; 9 | 10 | type Info: record { 11 | ts: time &log; 12 | host: string &log; 13 | hostname: string &log; 14 | ## One of "existing_mount", "new_mount" or "unmount". 15 | ## "mount" implies that the disk was already mounted when the agent reported in. 16 | ## "new_mount" implies that the disk was just mounted when the event was sent. 17 | ## "umount" implies the disk was unmounted when the event was sent. 18 | action: string &log; 19 | path: string &log; 20 | }; 21 | } 22 | 23 | const mount_lookup = { 24 | [ZeekAgent::INITIAL] = "existing_mount", 25 | [ZeekAgent::ADD] = "new_mount", 26 | [ZeekAgent::REMOVE] = "unmount", 27 | }; 28 | 29 | event AgentMounts::change(result: ZeekAgent::Result, path: string) 30 | { 31 | local info = Info($ts = network_time(), 32 | $host = result$host, 33 | $hostname = ZeekAgent::getHostInfo(result$host)$hostname, 34 | $action = mount_lookup[result$utype], 35 | $path = path); 36 | 37 | Log::write(LOG, info); 38 | } 39 | 40 | event zeek_init() &priority=10 41 | { 42 | Log::create_stream(LOG, [$columns=Info, $path="agent-mounts"]); 43 | 44 | local ev = ZeekAgent::Query($ev=AgentMounts::change, 45 | $query="SELECT path FROM mounts", 46 | $utype=ZeekAgent::BOTH); 47 | ZeekAgent::subscribe(ev); 48 | } 49 | -------------------------------------------------------------------------------- /zeek-agent/examples/osquery/users.zeek: -------------------------------------------------------------------------------- 1 | #! Query users activity. 2 | 3 | @load zeek-agent 4 | 5 | module AgentUsers; 6 | 7 | export { 8 | redef enum Log::ID += { LOG }; 9 | 10 | type Info: record { 11 | ts: time &log; 12 | host: string &log; 13 | hostname: string &log; 14 | action: string &log; 15 | username: string &log; 16 | uid: int &log; 17 | gid: int &log; 18 | description: string &log; 19 | home_dir: string &log; 20 | shell: string &log; 21 | }; 22 | } 23 | 24 | event AgentUsers::change(result: ZeekAgent::Result, uid: int, gid: int, username: string, description: string, directory: string, shell: string) 25 | { 26 | # Don't log existing users. We only want the moment users are added or removed. 27 | if ( result$utype == ZeekAgent::INITIAL ) 28 | return; 29 | 30 | local info = Info($ts = network_time(), 31 | $host = result$host, 32 | $hostname = ZeekAgent::getHostInfo(result$host)$hostname, 33 | $action = (result$utype == ZeekAgent::ADD ? "add" : "remove"), 34 | $username = username, 35 | $uid = uid, 36 | $gid = gid, 37 | $description = description, 38 | $home_dir = directory, 39 | $shell = shell); 40 | 41 | Log::write(LOG, info); 42 | } 43 | 44 | event zeek_init() &priority=10 45 | { 46 | Log::create_stream(LOG, [$columns=Info, $path="agent-users"]); 47 | 48 | local query = ZeekAgent::Query($ev=AgentUsers::change, 49 | $query="SELECT uid_signed, gid_signed, username, description, directory, shell FROM users", 50 | $utype=ZeekAgent::BOTH); 51 | ZeekAgent::subscribe(query); 52 | } 53 | -------------------------------------------------------------------------------- /zeek-agent/framework/__load__.zeek: -------------------------------------------------------------------------------- 1 | @load base/frameworks/cluster 2 | 3 | @load ./configuration 4 | @load ./framework_commons 5 | @load ./hosts_send 6 | @load ./zeek_agent_subscriptions 7 | 8 | @if ( !Cluster::is_enabled() || Cluster::local_node_type() == Cluster::MANAGER ) 9 | @load ./zeek_agent_hosts 10 | @load ./bro_backend 11 | @load ./zeek_agent_framework 12 | @endif 13 | 14 | @load ./zeek_logger 15 | @load ./zeek_logger_log 16 | -------------------------------------------------------------------------------- /zeek-agent/framework/bro_backend.zeek: -------------------------------------------------------------------------------- 1 | @load base/frameworks/broker 2 | @load base/frameworks/logging 3 | 4 | @load ./framework_commons 5 | 6 | module ZeekAgent; 7 | 8 | export { 9 | ## Share subscription to an event with other zeek nodes. 10 | ## 11 | ## q: The query to subscribe to. 12 | ## host_list: Specific hosts to address per query (optional). 13 | ## group_list: Specific groups to address per query (optional). 14 | global share_subscription: function(q: ZeekAgent::Query, host_list: vector of string &default=vector(""), group_list: vector of string &default=vector("")); 15 | 16 | ## Revoke sharing of subscription to an events. 17 | ## 18 | ## q: The query to revoke. 19 | ## host_list: Specific hosts to address per query (optional). 20 | ## group_list: Specific groups to address per query (optional). 21 | global unshare_subscription: function(q: ZeekAgent::Query, host_list: vector of string &default=vector(""), group_list: vector of string &default=vector("")); 22 | 23 | ## Share a one-time query to all currently connected clients. 24 | ## 25 | ## 26 | ## q: The query to execute. 27 | ## host_list: Specific hosts to address per query (optional). 28 | ## group_list: Specific groups to address per query (optional). 29 | global share_execution: function(q: ZeekAgent::Query, host_list: vector of string &default=vector(""), group_list: vector of string &default=vector("")); 30 | 31 | ## Share a grouping for organizing hosts in groups. 32 | ## 33 | ## range_list: the subnets that are addressed. 34 | ## group: the group hosts should join. 35 | global share_grouping: function(range_list: vector of subnet, group: string); 36 | 37 | ## Revoke sharing a grouping for organizing hosts in groups. 38 | ## 39 | ## range_list: the subnets that are addressed. 40 | ## group: the group hosts should leave. 41 | global unshare_grouping: function(range_list: vector of subnet, group: string); 42 | } 43 | 44 | global zeek_new: event(peer_name: string, zeek_id: string, init: bool); 45 | 46 | # Sent to share subscribing to an event. 47 | global zeek_subscribe: event(group_flood: bool, via_peer_id: string, q: ZeekAgent::Query, host_list: vector of string, group_list: vector of string); 48 | 49 | # Sent to revoke subscribing to an event. 50 | global zeek_unsubscribe: event(group_flood: bool, via_peer_id: string, q: ZeekAgent::Query, host_list: vector of string, group_list: vector of string); 51 | 52 | # Sent to share one-time query execution. 53 | global zeek_execute: event(group_flood: bool, via_peer_id: string, q: ZeekAgent::Query, host_list: vector of string, group_list: vector of string); 54 | 55 | # Sent to share groupings hosts for join a group. 56 | global zeek_join: event(group_flood: bool, via_peer_id: string, range_list: vector of subnet, group: string); 57 | 58 | # Sent by us to hosts for leaving a group. 59 | global zeek_leave: event(group_flood: bool, via_peer_id: string, range_list: vector of subnet, group: string); 60 | 61 | # Internal table for tracking incoming subscriptions from remote 62 | global zeek_subscriptions: table[string] of ZeekAgent::Subscriptions; 63 | 64 | # Internal table for tracking incoming assignments from remote 65 | global zeek_groupings: table[string] of ZeekAgent::Groupings; 66 | 67 | # Internal mapping of broker id (peer_name) to zeek-agent (host_id) 68 | global peer_to_zeek: table[string] of string; 69 | 70 | function delete_zeek_subscription(query: ZeekAgent::Query) 71 | { 72 | local peer_name = cat(Broker::node_id()); 73 | local found = -1; 74 | 75 | # Find idx to delete 76 | for ( idx in zeek_subscriptions[peer_name] ) 77 | { 78 | if ( ZeekAgent::same_event(zeek_subscriptions[peer_name][idx]$query, query) ) 79 | { 80 | found = idx; 81 | break; 82 | } 83 | } 84 | 85 | if ( idx == -1 ) 86 | { 87 | # TODO Log 88 | return; 89 | } 90 | 91 | # New vector of new size 92 | local new_subscriptions: ZeekAgent::Subscriptions; 93 | for ( idx in zeek_subscriptions[peer_name] ) 94 | { 95 | if ( idx == found ) 96 | next; 97 | 98 | new_subscriptions += zeek_subscriptions[peer_name][idx]; 99 | } 100 | 101 | zeek_subscriptions[peer_name] = new_subscriptions; 102 | } 103 | 104 | function delete_zeek_grouping(group: string) 105 | { 106 | local peer_name = cat(Broker::node_id()); 107 | local found = -1; 108 | 109 | # Find idx to delete 110 | for ( idx in zeek_groupings[peer_name] ) 111 | { 112 | if ( zeek_groupings[peer_name][idx]$group == group ) 113 | { 114 | found = idx; 115 | break; 116 | } 117 | } 118 | 119 | if ( idx == -1 ) 120 | { 121 | # TODO Log 122 | return; 123 | } 124 | 125 | # New vector of new size 126 | local new_groupings: ZeekAgent::Groupings; 127 | for ( idx in zeek_groupings[peer_name] ) 128 | { 129 | if ( idx == found ) 130 | next; 131 | 132 | new_groupings += zeek_groupings[peer_name][idx]; 133 | } 134 | 135 | zeek_groupings[peer_name] = new_groupings; 136 | } 137 | 138 | function send_subscription(topic: string, ev: any, group_flood: bool, q: ZeekAgent::Query, host_list: vector of string &default=vector(""), group_list: vector of string &default=vector("")) 139 | { 140 | local ev_name = split_string(fmt("%s", ev), /\n/)[0]; 141 | ZeekAgent::log_zeek("debug", topic, fmt("%s event %s() for query '%s'", "Forwarding", ev_name, q$query)); 142 | 143 | local ev_args = Broker::make_event(ev, group_flood, fmt("%s",Broker::node_id()), q, host_list, group_list); 144 | Broker::publish(topic, ev_args); 145 | } 146 | 147 | function send_grouping(topic: string, ev: any, group_flood: bool, range_list: vector of subnet, group: string) 148 | { 149 | local ev_name = split_string(fmt("%s", ev), /\n/)[0]; 150 | ZeekAgent::log_zeek("debug", topic, fmt("%s event %s() for group '%s'", "Forwarding", ev_name, group)); 151 | 152 | local ev_args = Broker::make_event(ev, group_flood, cat(Broker::node_id()), range_list, group); 153 | Broker::publish(topic, ev_args); 154 | } 155 | 156 | function share_subscription(q: ZeekAgent::Query, host_list: vector of string &default=vector(""), group_list: vector of string &default=vector("")) 157 | { 158 | local peer_name = cat(Broker::node_id()); 159 | if ( peer_name !in zeek_subscriptions ) 160 | zeek_subscriptions[peer_name] = vector(); 161 | 162 | zeek_subscriptions[peer_name] += Subscription($query=q, $hosts=host_list, $groups=group_list); 163 | send_subscription(ZeekAgent::ZeekBroadcastTopic, zeek_subscribe, T, q, host_list, group_list); 164 | } 165 | 166 | function unshare_subscription(q: ZeekAgent::Query, host_list: vector of string &default=vector(""), group_list: vector of string &default=vector("")) 167 | { 168 | delete_zeek_subscription(q); 169 | send_subscription(ZeekAgent::ZeekBroadcastTopic, zeek_unsubscribe, T, q, host_list, group_list); 170 | } 171 | 172 | function share_execution(q: ZeekAgent::Query, host_list: vector of string &default=vector(""), group_list: vector of string &default=vector("")) 173 | { 174 | send_subscription(ZeekAgent::ZeekBroadcastTopic, zeek_execute, T, q, host_list, group_list); 175 | } 176 | 177 | function share_grouping(range_list: vector of subnet, group: string) 178 | { 179 | local peer_name = cat(Broker::node_id()); 180 | if ( peer_name !in zeek_groupings ) 181 | zeek_groupings[peer_name] = vector(); 182 | 183 | zeek_groupings[peer_name] += Grouping($group=group, $ranges=range_list); 184 | send_grouping(ZeekAgent::ZeekBroadcastTopic, zeek_join, T, range_list, group); 185 | } 186 | 187 | function unshare_grouping(range_list: vector of subnet, group: string) 188 | { 189 | delete_zeek_grouping(group); 190 | send_grouping(ZeekAgent::ZeekBroadcastTopic, zeek_leave, T, range_list, group); 191 | } 192 | 193 | event ZeekAgent::zeek_subscribe(group_flood: bool, via_peer_id: string, q: ZeekAgent::Query, host_list: vector of string, group_list: vector of string) 194 | { 195 | # Keep state about the direction the subscription came from 196 | local topic: string; 197 | if ( via_peer_id !in zeek_subscriptions ) 198 | { 199 | topic = fmt("%s/%s", ZeekAgent::ZeekIndividualTopic, via_peer_id); 200 | ZeekAgent::log_zeek("warning", topic, fmt("Unexpected event %s from unknown Zeek for query %s", "zeek_subscribe", q$query)); 201 | return; 202 | } 203 | 204 | ZeekAgent::insert_subscription(q, host_list, group_list); 205 | zeek_subscriptions[via_peer_id] += Subscription($query=q, $hosts=host_list, $groups=group_list); 206 | 207 | # Group Flooding will be done automatically in Broker 208 | if ( group_flood ) 209 | return; 210 | 211 | # Forward subscription manually to all other neighbors 212 | local peer_name: string; 213 | for ( peer_name in peer_to_zeek ) 214 | { 215 | if (peer_name == via_peer_id) 216 | next; 217 | 218 | topic = fmt("%s/%s", ZeekAgent::ZeekIndividualTopic, peer_name); 219 | send_subscription(topic, zeek_subscribe, F, q, host_list, group_list); 220 | } 221 | } 222 | 223 | event ZeekAgent::zeek_unsubscribe(group_flood: bool, via_peer_id: string, q: ZeekAgent::Query, host_list: vector of string, group_list: vector of string) 224 | { 225 | # Remove state about the direction the subscription came from 226 | local topic: string; 227 | if ( via_peer_id !in zeek_subscriptions ) 228 | { 229 | topic = fmt("%s/%s", ZeekAgent::ZeekIndividualTopic, via_peer_id); 230 | ZeekAgent::log_zeek("warning", topic, fmt("Unexpected event %s from unkown Zeek for query %s", "zeek_unsubscribe", q$query)); 231 | return; 232 | } 233 | 234 | ZeekAgent::remove_subscription(q, host_list, group_list); 235 | delete_zeek_subscription(q); 236 | 237 | # Group Flooding will be done automatically in Broker 238 | if ( group_flood ) 239 | return; 240 | 241 | # Forward unsubscription manually to all other neighbors 242 | local peer_name: string; 243 | for ( peer_name in peer_to_zeek ) 244 | { 245 | if ( peer_name == via_peer_id ) 246 | next; 247 | 248 | topic = fmt("%s/%s", ZeekAgent::ZeekIndividualTopic, peer_name); 249 | send_subscription(topic, zeek_unsubscribe, F, q, host_list, group_list); 250 | } 251 | } 252 | 253 | event ZeekAgent::zeek_execute(group_flood: bool, via_peer_id: string, q: ZeekAgent::Query, host_list: vector of string, group_list: vector of string) 254 | { 255 | # Apply execution locallzeek_agentsert_execution(q, host_list, group_list); 256 | 257 | # Group Flooding will be done automatically in Broker 258 | if ( group_flood ) 259 | return; 260 | 261 | # Forward subscription manually to all other neighbors 262 | for ( peer_name in peer_to_zeek ) 263 | { 264 | if ( peer_name == via_peer_id ) 265 | next; 266 | 267 | local topic = fmt("%s/%s", ZeekAgent::ZeekIndividualTopic, peer_name); 268 | send_subscription(topic, zeek_subscribe, F, q, host_list, group_list); 269 | } 270 | } 271 | 272 | event ZeekAgent::zeek_join(group_flood: bool, via_peer_id: string, range_list: vector of subnet, group: string) 273 | { 274 | # Keep state about the direction the subscription came from 275 | local topic: string; 276 | if ( via_peer_id !in zeek_groupings ) 277 | { 278 | topic = fmt("%s/%s", ZeekAgent::ZeekIndividualTopic, via_peer_id); 279 | ZeekAgent::log_zeek("warning", topic, fmt("Unexpected event %s from unkown Zeek for group %s", "zeek_join", group)); 280 | return; 281 | } 282 | 283 | ZeekAgent::insert_grouping(range_list, group); 284 | zeek_groupings[via_peer_id] += Grouping($group=group, $ranges=range_list); 285 | 286 | # Group Flooding will be done automatically in Broker 287 | if ( group_flood ) 288 | return; 289 | 290 | # Forward grouping manually to all other neighbors 291 | for ( peer_name in peer_to_zeek ) 292 | { 293 | if ( peer_name == via_peer_id ) 294 | next; 295 | 296 | topic = fmt("%s/%s", ZeekAgent::ZeekIndividualTopic, peer_name); 297 | send_grouping(topic, zeek_join, F, range_list, group); 298 | } 299 | } 300 | 301 | event ZeekAgent::zeek_leave(group_flood: bool, via_peer_id: string, range_list: vector of subnet, group: string) 302 | { 303 | # Remove state about the direction the subscription came from 304 | local topic: string; 305 | if ( via_peer_id !in zeek_groupings ) 306 | { 307 | topic = fmt("%s/%s", ZeekAgent::ZeekIndividualTopic, via_peer_id); 308 | ZeekAgent::log_zeek("warning", topic, fmt("Unexpected event %s from unknown Zeek for group %s", "zeek_leave", group)); 309 | return; 310 | } 311 | 312 | ZeekAgent::remove_grouping(range_list, group); 313 | delete_zeek_grouping(group); 314 | 315 | # Group Flooding will be done automatically in Broker 316 | if ( group_flood ) 317 | return; 318 | 319 | # Forward grouping manually to all other neighbors 320 | local peer_name: string; 321 | for ( peer_name in peer_to_zeek ) 322 | { 323 | if ( peer_name == via_peer_id ) 324 | next; 325 | 326 | topic = fmt("%s/%s", ZeekAgent::ZeekIndividualTopic, peer_name); 327 | send_grouping(topic, zeek_leave, F, range_list, group); 328 | } 329 | } 330 | 331 | event ZeekAgent::zeek_new(peer_name: string, zeek_id: string, init: bool) 332 | { 333 | ZeekAgent::log_zeek("info", zeek_id, fmt("Zeek Backend connected (%s announced as %s)", peer_name, zeek_id)); 334 | local topic = fmt("%s/%s", ZeekAgent::ZeekIndividualTopic, peer_name); 335 | 336 | # Zeek already known? 337 | if ( peer_name in peer_to_zeek ) 338 | { 339 | local topic_from = fmt("%s/%s", ZeekAgent::ZeekIndividualTopic, peer_name); 340 | ZeekAgent::log_zeek("warning", topic_from, fmt("Peer %s with ID %s already known as Zeek", peer_name, zeek_id)); 341 | } 342 | 343 | # Internal client tracking 344 | peer_to_zeek[peer_name] = zeek_id; 345 | zeek_subscriptions[peer_name] = vector(); 346 | zeek_groupings[peer_name] = vector(); 347 | 348 | # Also announce back to retrieve their state 349 | if ( init ) 350 | { 351 | local ev_args = Broker::make_event(zeek_new, fmt("%s",Broker::node_id()), ZeekAgent::Zeek_ID_Topic, F); 352 | Broker::publish(topic, ev_args); 353 | } 354 | 355 | # Send own subscriptions 356 | local s: ZeekAgent::Subscription; 357 | for ( p_name in zeek_subscriptions ) 358 | { 359 | if ( p_name == peer_name ) 360 | next; 361 | 362 | for ( i in zeek_subscriptions[p_name] ) 363 | { 364 | s = zeek_subscriptions[p_name][i]; 365 | send_subscription(topic, zeek_subscribe, F, s$query, s$hosts, s$groups); 366 | } 367 | } 368 | 369 | # Send own groupings 370 | local g: ZeekAgent::Grouping; 371 | for ( p_name in zeek_groupings ) 372 | { 373 | if ( p_name == peer_name ) 374 | next; 375 | 376 | for ( i in zeek_groupings[p_name] ) 377 | { 378 | g = zeek_groupings[p_name][i]; 379 | send_grouping(topic, zeek_join, F, g$ranges, g$group); 380 | } 381 | } 382 | 383 | # raise event for new zeek 384 | event ZeekAgent::bro_connected(zeek_id); 385 | } 386 | 387 | function revoke_subscriptions(peer_name: string, disconnected: bool &default=T) 388 | { 389 | for ( i in zeek_subscriptions[peer_name] ) 390 | { 391 | local s = zeek_subscriptions[peer_name][i]; 392 | 393 | # Remove locally 394 | ZeekAgent::remove_subscription(s$query, s$hosts, s$groups); 395 | 396 | # Generate unsubscribe caused by disconnect 397 | if ( disconnected ) 398 | { 399 | # Safe to flood the unsubscribe 400 | local topic = ZeekAgent::ZeekBroadcastTopic; 401 | send_subscription(topic, zeek_unsubscribe, T, s$query, s$hosts, s$groups); 402 | } 403 | else 404 | { 405 | # TODO: Remove manually by individual messages to other neighbors 406 | } 407 | } 408 | 409 | # Remove State 410 | delete zeek_subscriptions[peer_name]; 411 | } 412 | 413 | function revoke_groupings(peer_name: string, disconnected: bool &default=T) 414 | { 415 | local g: ZeekAgent::Grouping; 416 | for ( i in zeek_groupings[peer_name] ) 417 | { 418 | g = zeek_groupings[peer_name][i]; 419 | 420 | # Remove locally 421 | ZeekAgent::remove_grouping(g$ranges, g$group); 422 | 423 | # Generate unsubscribe caused by disconnect 424 | if ( disconnected ) 425 | { 426 | # Safe to flood the unsubscribe 427 | local topic: string = ZeekAgent::ZeekBroadcastTopic; 428 | send_grouping(topic, zeek_leave, T, g$ranges, g$group); 429 | } 430 | else 431 | { 432 | # TODO: Remove manually by individual messages to other neighbors 433 | } 434 | } 435 | 436 | # Remove State 437 | local g_list: vector of ZeekAgent::Grouping; 438 | zeek_groupings[peer_name] = g_list; 439 | } 440 | 441 | event Broker::peer_added(endpoint: Broker::EndpointInfo, msg: string) 442 | { 443 | # Only for outgoing connections 444 | if ( msg != "received handshake from remote core" ) 445 | return; 446 | 447 | local peer_name: string = {endpoint$id}; 448 | local topic: string = fmt("%s/%s", ZeekAgent::ZeekIndividualTopic, peer_name); 449 | 450 | # Send announce message to the remote peer 451 | local ev_args = Broker::make_event(zeek_new, fmt("%s",Broker::node_id()), ZeekAgent::Zeek_ID_Topic, T); 452 | Broker::publish(topic, ev_args); 453 | } 454 | 455 | event Broker::peer_removed(endpoint: Broker::EndpointInfo, msg: string) 456 | { 457 | local peer_name: string = {endpoint$id}; 458 | if ( peer_name !in zeek_subscriptions ) 459 | return; 460 | 461 | local zeek_id: string = peer_to_zeek[peer_name]; 462 | ZeekAgent::log_zeek("info", zeek_id, "Zeek disconnected"); 463 | 464 | # Revoke all subscriptions that came in via this peer 465 | revoke_subscriptions(peer_name); 466 | revoke_groupings(peer_name); 467 | delete zeek_subscriptions[peer_name]; 468 | delete zeek_groupings[peer_name]; 469 | delete peer_to_zeek[peer_name]; 470 | 471 | # raise event for disconnected bro 472 | event ZeekAgent::bro_disconnected(zeek_id); 473 | } 474 | 475 | event Broker::peer_lost(endpoint: Broker::EndpointInfo, msg: string) 476 | { 477 | local peer_name: string = {endpoint$id}; 478 | if ( peer_name !in zeek_subscriptions ) 479 | return; 480 | 481 | local zeek_id: string = peer_to_zeek[peer_name]; 482 | ZeekAgent::log_zeek("info", zeek_id, "Zeek disconnected"); 483 | 484 | # Revoke all subscriptions that came in via this peer 485 | revoke_subscriptions(peer_name); 486 | revoke_groupings(peer_name); 487 | delete zeek_subscriptions[peer_name]; 488 | delete zeek_groupings[peer_name]; 489 | delete peer_to_zeek[peer_name]; 490 | 491 | # raise event for disconnected zeek 492 | event ZeekAgent::bro_disconnected(zeek_id); 493 | } 494 | 495 | event zeek_init() 496 | { 497 | # Listen on Zeek announce topic 498 | local topic: string = ZeekAgent::ZeekAnnounceTopic; 499 | ZeekAgent::log_local("info", fmt("Subscribing to Zeek announce topic %s", topic)); 500 | Broker::subscribe(topic); 501 | 502 | # Listen on Zeek individual topic 503 | topic = fmt("%s/%s", ZeekAgent::ZeekIndividualTopic, Broker::node_id()); 504 | ZeekAgent::log_local("info", fmt("Subscribing to Zeek individual topic %s", topic)); 505 | Broker::subscribe(topic); 506 | 507 | # Connect to remote Zeek 508 | if ( |ZeekAgent::backend_ip| != 0 && 509 | ZeekAgent::backend_ip != "0.0.0.0" ) 510 | { 511 | Broker::peer(ZeekAgent::backend_ip, ZeekAgent::backend_port, 10sec); 512 | } 513 | } 514 | -------------------------------------------------------------------------------- /zeek-agent/framework/configuration.zeek: -------------------------------------------------------------------------------- 1 | 2 | module ZeekAgent; 3 | 4 | export { 5 | # Interval in seconds to execute scheduled queries on hosts 6 | const default_query_interval = 10secs &redef; 7 | } 8 | -------------------------------------------------------------------------------- /zeek-agent/framework/framework_commons.zeek: -------------------------------------------------------------------------------- 1 | module ZeekAgent; 2 | 3 | export { 4 | ## The local IP Broker is listening on 5 | const broker_ip = "0.0.0.0" &redef; 6 | 7 | ## The local port Broker is listening on 8 | const broker_port = 9999/tcp &redef; 9 | 10 | ## The local name of Zeek 11 | const endpoint_name = "ZeekMaster" &redef; 12 | 13 | ## The remote IP Broker is connecting to 14 | const backend_ip = "0.0.0.0" &redef; 15 | 16 | ## The remote port Broker is connecting to 17 | const backend_port = 9999/tcp &redef; 18 | 19 | # Topic prefix used for all topics in zeek-agent communication 20 | const TopicPrefix = "/zeek/zeek-agent" &redef; 21 | 22 | # Topic for individual zeeks 23 | const ZeekIndividualTopic = fmt("%s/zeek",TopicPrefix) &redef; 24 | # Topic to address all zeeks 25 | const ZeekBroadcastTopic = fmt("%s/zeeks",TopicPrefix) &redef; 26 | # Individual channel of this zeek instance 27 | const Zeek_ID_Topic = fmt("%s/%s",ZeekIndividualTopic,endpoint_name) &redef; 28 | 29 | # Topic for individual hosts 30 | const HostIndividualTopic = fmt("%s/host",TopicPrefix) &redef; 31 | # Topic for groups 32 | const HostGroupTopic = fmt("%s/group",TopicPrefix) &redef; 33 | # Topic to address all hosts 34 | const HostBroadcastTopic = fmt("%s/hosts",TopicPrefix) &redef; 35 | 36 | # Topic to which zeeks send announce messages 37 | const ZeekAnnounceTopic = fmt("%s/zeek_announce",TopicPrefix) &redef; 38 | # Topic to which hosts send announce messages 39 | const HostAnnounceTopic = fmt("%s/host_announce",TopicPrefix) &redef; 40 | 41 | ## The zeek-agent logging stream identifier. 42 | redef enum Log::ID += { LOG }; 43 | 44 | ## A record type containing the column fields of the zeek-agent log. 45 | type Info: record { 46 | ## The network time at which a zeek-agent activity occurred. 47 | ts: time &log; 48 | ## The scope of the message. Can be 'local' to indicating a message relevant for 49 | ## this node only. 'zeek' indicates interfaction with other zeek nodes and 50 | ## 'zeek-agent' indicates interaction with zeek-agents. 51 | source: string &log; 52 | ## The peer name (if any) with which a communication event is concerned. 53 | peer: string &log &optional; 54 | ## The severity of the communication event message. 55 | level: string &log &optional; 56 | ## The main log message. 57 | message: string &log; 58 | }; 59 | 60 | ## Type defining the type of zeek-agent change we are interested in. 61 | type UpdateType: enum { 62 | ## Report the initial set of results. For purposes of scheduled queries, 63 | ## this behaves the same as "BOTH". It is primarily used when recieiving 64 | INITIAL, 65 | ADD, ##< Report new elements. 66 | REMOVE, ##< Report removed element. 67 | BOTH, ##< Report both new and removed elements. 68 | SNAPSHOT ##< Report the current status at query time. 69 | }; 70 | 71 | ## Type defining a SQL query and schedule/execution parameters to be send to hosts. 72 | type Query: record { 73 | ## The zeek-agent SQL query selecting the activity to subscribe to. 74 | query: string; 75 | ## The type of update to report. 76 | utype: UpdateType &default=BOTH; 77 | ## The interval of the query 78 | inter: interval &optional &default=ZeekAgent::default_query_interval; 79 | ## The Broker topic THEY send the query result to 80 | resT: string &default=Zeek_ID_Topic; 81 | ## The Zeek event to execute when receiving updates. 82 | ev: any &optional; 83 | ## A cookie we can set to match the result event 84 | cookie: string &default=""; 85 | }; 86 | 87 | ## Type defining the event header of responses 88 | type Result: record { 89 | host: string; 90 | utype: UpdateType; 91 | cookie: string &optional; 92 | }; 93 | 94 | ## Event that signals the connection of a new zeek-agent 95 | ## 96 | ## client_id: An id that uniquely identifies an zeek-agent 97 | global host_connected: event (host_id: string); 98 | 99 | ## Event that signals the disconnection of an zeek-agent 100 | ## 101 | ## client_id: An id that uniquely identifies an zeek-agent 102 | global host_disconnected: event (host_id: string); 103 | 104 | ## Event that signals the connection of a new bro 105 | ## 106 | ## client_id: An id that uniquely identifies a bro 107 | global bro_connected: event (zeek_id: string); 108 | 109 | ## Event that signals the disconnection of a bro 110 | ## 111 | ## client_id: An id that uniquely identifies a bro 112 | global bro_disconnected: event (zeek_id: string); 113 | 114 | ## Log a message of local scope for this zeek node 115 | ## 116 | ## level: the severity of the message 117 | ## msg: the message content 118 | global log_local: function(level: string, msg: string, log: any &default=LOG); 119 | 120 | ## Log a message with scope including other zeek nodes 121 | ## 122 | ## level: the severity of the message 123 | ## peer: the identifier of the other zeek 124 | ## msg: the message content 125 | global log_zeek: function(level: string, peer: string, msg: string, log: any &default=LOG); 126 | 127 | ## Log a message with scope including zeek-agent nodes 128 | ## 129 | ## level: the severity of the message 130 | ## peer: the identifier for the zeek-agent or group 131 | ## msg: the message content 132 | global ZeekAgent::log: function(level: string, peer: string, msg: string, log: any &default=LOG); 133 | 134 | ## Comparison of two queries to be equal 135 | global same_event: function (q1: Query, q2: Query): bool; 136 | } 137 | 138 | function log_local(level: string, msg: string, log: any) 139 | { 140 | Log::write(log, Info($ts = network_time(), 141 | $level = level, 142 | $source = "local", 143 | $peer = endpoint_name, 144 | $message = msg)); 145 | } 146 | 147 | function log_zeek(level: string, peer: string, msg: string, log: any) 148 | { 149 | Log::write(log, Info($ts = network_time(), 150 | $level = level, 151 | $source = "zeek", 152 | $peer = peer, 153 | $message = msg)); 154 | } 155 | 156 | function ZeekAgent::log(level: string, peer: string, msg: string, log: any) 157 | { 158 | Log::write(log, Info($ts = network_time(), 159 | $level = level, 160 | $source = "host", 161 | $peer = peer, 162 | $message = msg)); 163 | } 164 | 165 | function same_event(q1: Query, q2: Query) : bool 166 | { 167 | if ( q1$query != q2$query ) 168 | return F; 169 | if ( q1?$ev != q2?$ev ) 170 | return F; 171 | if ( q1?$ev && fmt("%s", q1$ev) != fmt("%s", q2$ev) ) 172 | return F; 173 | if ( q1?$utype != q2?$utype ) 174 | return F; 175 | if ( q1?$utype && q1$utype != q2$utype ) 176 | return F; 177 | if ( q1?$resT != q2?$resT ) 178 | return F; 179 | if ( q1?$resT && q1$resT != q2$resT ) 180 | return F; 181 | if ( q1?$inter != q2?$inter ) 182 | return F; 183 | if ( q1?$inter && q1$inter != q2$inter ) 184 | return F; 185 | 186 | return T; 187 | } 188 | 189 | event zeek_init() &priority=10 190 | { 191 | Log::create_stream(LOG, [$columns=Info, $path="zeek-agent"]); 192 | } 193 | -------------------------------------------------------------------------------- /zeek-agent/framework/hosts_send.zeek: -------------------------------------------------------------------------------- 1 | 2 | @load base/frameworks/broker 3 | @load base/frameworks/logging 4 | 5 | module ZeekAgent; 6 | 7 | export { 8 | ## The zeek-agent logging stream identifier. 9 | redef enum Log::ID += { LOG_SEND }; 10 | 11 | ## Send a message to an zeek-agent or group of hosts to subscribe to a query 12 | ## 13 | ## topic: The topic of the host or group to address 14 | ## query: The query to subscribe to 15 | global send_subscribe: function(topic: string, query: ZeekAgent::Query); 16 | 17 | ## Send a message to an zeek-agent or group of hosts to unsubscribe from a query 18 | ## 19 | ## topic: The topic of the host or group to address 20 | ## query: The query to unsubscribe from 21 | global send_unsubscribe: function(topic: string, query: ZeekAgent::Query); 22 | 23 | ## Send a message to an zeek-agent or group of hosts to execute a query 24 | ## 25 | ## topic: The topic of the host or group to address 26 | ## query: The query to execute 27 | global send_execute: function(topic: string, q: ZeekAgent::Query); 28 | 29 | ## Send a message to an zeek-agent or group of hosts to join a group 30 | ## 31 | ## host_topic: The topic of the host or group to address 32 | ## group: The group to join 33 | global send_join: function(host_topic: string, group: string); 34 | 35 | ## Send a message to an zeek-agent or group of hosts to leave a group 36 | ## 37 | ## host_topic: The topic of the host or group to address 38 | ## group: The group to leave 39 | global send_leave: function(host_topic: string, group: string); 40 | } 41 | 42 | # Sent by us to hosts for subscribing to an event. 43 | global host_subscribe: event(ev: string, query: string, cookie: string, resT: string, utype: string, inter: count); 44 | 45 | # Sent by us to hosts for unsubscribing from an event. 46 | global host_unsubscribe: event(ev: string, query: string, cookie: string, resT: string, utype: string, inter: count); 47 | 48 | # Sent by us to hosts for one-time query execution. 49 | global host_execute: event(ev: string, query: string, cookie: string, resT: string, utype: string); 50 | 51 | # Sent by us to hosts for join a group. 52 | global host_join: event(group: string); 53 | 54 | # Sent by us to hosts for leaving a group. 55 | global host_leave: event(group: string); 56 | 57 | function send_subscribe(topic: string, query: ZeekAgent::Query) 58 | { 59 | local ev_name = split_string(fmt("%s", query$ev), /\n/)[0]; 60 | local host_topic = topic; 61 | 62 | ZeekAgent::log("debug", topic, fmt("%s event %s() for query '%s'", "Subscribing to", ev_name, query$query), LOG_SEND); 63 | 64 | local update_type = "BOTH"; 65 | if ( query$utype == ZeekAgent::ADD ) 66 | update_type = "ADDED"; 67 | 68 | if ( query$utype == ZeekAgent::REMOVE ) 69 | update_type = "REMOVED"; 70 | 71 | local cookie = query$cookie; 72 | 73 | local resT = topic; 74 | if ( query?$resT ) 75 | resT = query$resT; 76 | Broker::subscribe(resT); 77 | 78 | local inter = double_to_count(interval_to_double(query$inter)); 79 | local ev_args = Broker::make_event(host_subscribe, ev_name, query$query, cookie, resT, update_type, inter); 80 | Broker::publish(host_topic, ev_args); 81 | } 82 | 83 | function send_unsubscribe(topic: string, query: ZeekAgent::Query) 84 | { 85 | local ev_name = split_string(fmt("%s", query$ev), /\n/)[0]; 86 | local host_topic = topic; 87 | 88 | ZeekAgent::log("debug", topic, fmt("%s event %s() for query '%s'", "Unsubscribing from", ev_name, query$query), LOG_SEND); 89 | 90 | local update_type = "BOTH"; 91 | if ( query$utype == ZeekAgent::ADD ) 92 | update_type = "ADDED"; 93 | 94 | if ( query$utype == ZeekAgent::REMOVE ) 95 | update_type = "REMOVED"; 96 | 97 | local cookie = query$cookie; 98 | 99 | local resT = topic; 100 | if ( query?$resT ) 101 | resT = query$resT; 102 | 103 | local inter = double_to_count(interval_to_double(query$inter)); 104 | local ev_args = Broker::make_event(host_unsubscribe, ev_name, query$query, cookie, resT, update_type, inter); 105 | Broker::publish(host_topic, ev_args); 106 | } 107 | 108 | function send_execute(topic: string, q: ZeekAgent::Query) 109 | { 110 | local ev_name = split_string(fmt("%s", q$ev), /\n/)[0]; 111 | local host_topic = topic; 112 | 113 | ZeekAgent::log("debug", topic, fmt("%s event %s() for query '%s'", "Executing", ev_name, q$query), LOG_SEND); 114 | 115 | local cookie = q$cookie; 116 | 117 | local resT = topic; 118 | if ( q?$resT ) 119 | resT = q$resT; 120 | Broker::subscribe(resT); 121 | 122 | local ev_args = Broker::make_event(host_execute, ev_name, q$query, cookie, resT, "SNAPSHOT"); 123 | Broker::publish(host_topic, ev_args); 124 | } 125 | 126 | function send_join(host_topic: string, group: string) 127 | { 128 | ZeekAgent::log("info", host_topic, fmt("%s group '%s'", "Joining", group), LOG_SEND); 129 | local ev_args = Broker::make_event(host_join, group); 130 | Broker::publish(host_topic, ev_args); 131 | } 132 | 133 | function send_leave(host_topic: string, group: string) 134 | { 135 | ZeekAgent::log("info", host_topic, fmt("%s group '%s'", "Leaving", group), LOG_SEND); 136 | local ev_args = Broker::make_event(host_leave, group); 137 | Broker::publish(host_topic, ev_args); 138 | } 139 | 140 | event zeek_init() 141 | { 142 | Log::create_stream(LOG_SEND, [$columns=ZeekAgent::Info, $path="zeek-agent-hosts"]); 143 | } 144 | -------------------------------------------------------------------------------- /zeek-agent/framework/zeek_agent_framework.zeek: -------------------------------------------------------------------------------- 1 | 2 | @load base/frameworks/broker 3 | @load base/frameworks/logging 4 | 5 | module ZeekAgent; 6 | 7 | export { 8 | ## Subscribe to an event from clients. Whenever an zeek-agent connects to us, we'll subscribe to all matching 9 | ## activity from it. 10 | ## 11 | ## The query is a mandatory parameter. It is send to a specific host and/or group (if specified). Otherwise (if 12 | ## neither hosts nor group is given) the query is send to the broadcast group, such that all hosts will receive it. 13 | ## 14 | ## q: The query to subscribe to. 15 | ## host: A specific host to address (optional). 16 | ## group: A specific group to address (optional). 17 | global subscribe: function(q: Query, host: string &default="", group: string &default=""); 18 | 19 | ## Unsubscribe to an event from clients. This is sent to all clients that are currently connected and would match a 20 | ## similar subscribe call. 21 | ## 22 | ## The query is a mandatory parameter. It is send to a specific host and/or group (if specified). Otherwise (if 23 | ## neither hosts nor group is given) the query is send to the broadcast group, such that all hosts will receive it. 24 | ## 25 | ## q: The query to revoke. 26 | ## host: A specific host to address (optional). 27 | ## group: A specific group to address (optional). 28 | global unsubscribe: function(q: Query, host: string &default="", group: string &default=""); 29 | 30 | ## Subscribe to multiple events. Whenever an zeek-agent connects to us, we'll subscribe to all matching activity 31 | ## from it. 32 | ## 33 | ## The queries is an mandatory parameter and contains 1 or more queries. Each of them is send to the specified hosts 34 | ## and the specified groups. If neither is given, each query is broadcasted to all hosts. 35 | ## 36 | ## q: The query to subscribe to. 37 | ## host_list: Specific hosts to address per query (optional). 38 | ## group_list: Specific groups to address per query (optional). 39 | global subscribe_multiple: function(q: Query, host_list: vector of string &default=vector(""), group_list: vector of string &default=vector("")); 40 | 41 | ## Unsubscribe from multiple events. This will get sent to all clients that are currently connected and would match 42 | ## a similar subscribe call. 43 | ## 44 | ## The queries is an mandatory parameter and contains 1 or more queries. Each of them is send to the specified hosts 45 | ## and the specified groups. If neither is given, each query is broadcasted to all hosts. 46 | ## 47 | ## q: The query to revoke. 48 | ## host_list: Specific hosts to address per query (optional). 49 | ## group_list: Specific groups to address per query (optional). 50 | global unsubscribe_multiple: function(q: Query, host_list: vector of string &default=vector(""), group_list: vector of string &default=vector("")); 51 | 52 | ## Send a one-time query to all currently connected clients. 53 | ## 54 | ## The query is a mandatory parameter. It is send to a specific host and/or group (if specified). Otherwise (if 55 | ## neither hosts nor group is given) the query is send to the broadcast group, such that all hosts will receive it. 56 | ## 57 | ## q: The query to execute. 58 | ## host: A specific host to address (optional). 59 | ## group: A specific group to address (optional). 60 | ## 61 | ## topic: The topic where the subscription is send to. All hosts in this group will 62 | ## get the subscription. 63 | global execute: function(q: Query, host: string &default="", group: string &default=""); 64 | 65 | ## Send multiple one-time queries to all currently connected clients. 66 | ## 67 | ## The queries is an mandatory parameter and contains 1 or more queries. Each of them is send to the specified hosts 68 | ## and the specified groups. If neither is given, each query is broadcasted to all hosts. 69 | ## 70 | ## q: The queriy to execute. 71 | ## host_list: Specific hosts to address per query (optional). 72 | ## group_list: Specific groups to address per query (optional). 73 | global execute_multiple: function(q: Query, host_list: vector of string &default=vector(""), group_list: vector of string &default=vector("")); 74 | 75 | ## Make a subnet to be addressed by a group. Whenever an zeek-agent connects to us, we'll instruct it to join 76 | ## the given group. 77 | ## 78 | ## range: the subnet that is addressed. 79 | ## group: the group hosts should join. 80 | global join: function(range: subnet, group: string); 81 | 82 | ## Revoke a grouping that a specific subnet should join a group. 83 | ## 84 | ## range: the subnet that is addressed. 85 | ## group: the group hosts should leave. 86 | global leave: function(range: subnet, group: string); 87 | 88 | ## Make a subnets to be addressed by a group. Whenever an zeek-agent connects to us, we'll instruct it to join 89 | ## the given groups. 90 | ## 91 | ## range_list: the subnets that are addressed. 92 | ## group: the group hosts should join. 93 | global join_multiple: function(range_list: vector of subnet, group: string); 94 | 95 | ## Revoke a grouping that specific subnets should join a group. 96 | ## 97 | ## range_list: the subnet that is addressed. 98 | ## group: the group hosts should leave. 99 | global leave_mutiple: function(range_list: vector of subnet, group: string); 100 | } 101 | 102 | function subscribe(q: Query, host: string, group: string) 103 | { 104 | local host_list = vector(host); 105 | local group_list = vector(group); 106 | subscribe_multiple(q, host_list, group_list); 107 | } 108 | 109 | function subscribe_multiple(q: Query, host_list: vector of string, group_list: vector of string) 110 | { 111 | ZeekAgent::share_subscription(q, host_list, group_list); 112 | ZeekAgent::insert_subscription(q, host_list, group_list); 113 | } 114 | 115 | function unsubscribe(q: Query, host: string, group: string) 116 | { 117 | local host_list = vector(host); 118 | local group_list = vector(group); 119 | unsubscribe_multiple(q, host_list, group_list); 120 | } 121 | 122 | function unsubscribe_multiple(q: Query, host_list: vector of string, group_list: vector of string) 123 | { 124 | ZeekAgent::unshare_subscription(q, host_list, group_list); 125 | ZeekAgent::remove_subscription(q, host_list, group_list); 126 | } 127 | 128 | function execute(q: Query, host: string, group: string) 129 | { 130 | local host_list = vector(host); 131 | local group_list = vector(group); 132 | execute_multiple(q, host_list, group_list); 133 | } 134 | 135 | function execute_multiple(q: Query, host_list: vector of string, group_list: vector of string) 136 | { 137 | ZeekAgent::share_execution(q, host_list, group_list); 138 | ZeekAgent::insert_execution(q, host_list, group_list); 139 | } 140 | 141 | function join(range: subnet, group: string) 142 | { 143 | local range_list = vector(range); 144 | join_multiple(range_list, group); 145 | } 146 | 147 | function join_multiple(range_list: vector of subnet, group: string) 148 | { 149 | ZeekAgent::share_grouping(range_list, group); 150 | ZeekAgent::insert_grouping(range_list, group); 151 | } 152 | 153 | function leave(range: subnet, group: string) 154 | { 155 | local range_list = vector(range); 156 | join_multiple(range_list, group); 157 | } 158 | 159 | function leave_multiple(range_list: vector of subnet, group: string) 160 | { 161 | ZeekAgent::unshare_grouping(range_list, group); 162 | ZeekAgent::remove_grouping(range_list, group); 163 | } 164 | 165 | event Broker::peer_added(endpoint: Broker::EndpointInfo, msg: string) 166 | { 167 | local peer_name = endpoint$id; 168 | 169 | if ( msg == "received handshake from remote core" ) 170 | log_local("info", fmt("Outgoing connection established to %s", peer_name)); 171 | else if ( msg == "handshake successful" ) 172 | log_local("info", fmt("Incoming connection established from %s", peer_name)); 173 | else 174 | log_local("info", fmt("Unkown connection established with %s", peer_name)); 175 | } 176 | 177 | event Broker::peer_removed(endpoint: Broker::EndpointInfo, msg: string) 178 | { 179 | local peer_name = endpoint$id; 180 | 181 | log_local("info", fmt("Removed connection with %s", peer_name)); 182 | } 183 | 184 | event Broker::peer_lost(endpoint: Broker::EndpointInfo, msg: string) 185 | { 186 | local peer_name = endpoint$id; 187 | 188 | log_local("info", fmt("Lost connection with %s", peer_name)); 189 | } 190 | 191 | event Broker::status(endpoint: Broker::EndpointInfo, msg: string) 192 | { 193 | log_local("info", fmt("Status: %s", msg)); 194 | } 195 | 196 | event zeek_init() &priority=10 197 | { 198 | local topic = Zeek_ID_Topic; 199 | log_local("info", fmt("Subscribing to Broker topic %s", topic)); 200 | Broker::subscribe(topic); 201 | 202 | log_local("info", fmt("Accepting incoming broker connections on IP %s and port %s", broker_ip, broker_port)); 203 | Broker::listen(broker_ip, broker_port); 204 | } 205 | -------------------------------------------------------------------------------- /zeek-agent/framework/zeek_agent_hosts.zeek: -------------------------------------------------------------------------------- 1 | @load base/frameworks/broker 2 | @load base/frameworks/logging 3 | 4 | module ZeekAgent; 5 | 6 | export { 7 | ## Checks the new ip address of the given host against the groupings and makes it to join respective groups. 8 | ## 9 | ## host_id: the id of the host 10 | ## ip: the new ip address of the host 11 | global new_host_address: function(host_id: string, ip: addr): vector of string; 12 | 13 | ## Checks the new group of the given host against the subscriptions and makes it to schedule respective queries. 14 | ## 15 | ## host_id: the id of the host 16 | ## group: the new group of the host 17 | global new_host_group: function(host_id: string, group: string); 18 | 19 | ## Hook that can be called by others to indicate that an IP address was added to a host 20 | global add_host_addr: hook(host_id: string, ip: addr); 21 | 22 | ## Hook that can be called by others to indicate that an IP address was removed from a host 23 | global remove_host_addr: hook(host_id: string, ip: addr); 24 | } 25 | 26 | global connect_balance: table[string] of count; 27 | 28 | ## Sends current subscriptions to the new zeek-agent (given by client_id). 29 | ## 30 | ## This checks if any subscription matches the host restriction (or broadcast) 31 | ## 32 | ## client_id: The client ID 33 | function send_subscriptions_new_host(host_id: string) 34 | { 35 | local host_topic = fmt("%s/%s", ZeekAgent::HostIndividualTopic, host_id); 36 | for ( i in subscriptions ) 37 | { 38 | local s = subscriptions[i]; 39 | local skip_subscription = F; 40 | 41 | if ( ! s$query?$ev ) 42 | { 43 | # Skip Subscription because it was deleted"; 44 | next; 45 | } 46 | 47 | # Check for broadcast 48 | local sub_hosts: vector of string = s$hosts; 49 | local sub_groups: vector of string = s$groups; 50 | if ( |sub_hosts| <= 1 && sub_hosts[0] == "" && 51 | |sub_groups| <= 1 && sub_groups[0] == "" ) 52 | { 53 | # To all if nothing specified 54 | ZeekAgent::send_subscribe(host_topic, s$query); 55 | skip_subscription = T; 56 | } 57 | 58 | if ( skip_subscription ) 59 | next; 60 | 61 | # Check the hosts in the Subscriptions 62 | for ( j in sub_hosts ) 63 | { 64 | local sub_host = sub_hosts[j]; 65 | if ( host_id == sub_host ) 66 | { 67 | ZeekAgent::send_subscribe(host_topic, s$query); 68 | skip_subscription = T; 69 | break; 70 | } 71 | } 72 | 73 | if ( skip_subscription ) 74 | next; 75 | 76 | # Check the groups in the Subscriptions 77 | for ( j in host_groups[host_id] ) 78 | { 79 | local host_group = host_groups[host_id][j]; 80 | for ( k in sub_groups ) 81 | { 82 | local sub_group = sub_groups[k]; 83 | if ( |host_group| <= |sub_group| && 84 | host_group == sub_group[:|host_group|] ) 85 | { 86 | ZeekAgent::send_subscribe(host_topic, s$query); 87 | skip_subscription = T; 88 | break; 89 | } 90 | } 91 | if ( skip_subscription ) 92 | break; 93 | } 94 | if ( skip_subscription ) 95 | next; 96 | } 97 | } 98 | 99 | 100 | ## Checks for subscriptions that match the recently joined group 101 | function send_subscriptions_new_group(host_id: string, group: string) 102 | { 103 | local host_topic = fmt("%s/%s", ZeekAgent::HostIndividualTopic, host_id); 104 | for ( i in subscriptions ) 105 | { 106 | local s = subscriptions[i]; 107 | 108 | if ( ! s$query?$ev ) 109 | { 110 | # Skip Subscription because it was deleted"; 111 | next; 112 | } 113 | 114 | # Check the groups in the Subscriptions 115 | local sub_groups: vector of string = s$groups; 116 | for ( k in sub_groups ) 117 | { 118 | local sub_group = sub_groups[k]; 119 | if ( group == sub_group ) 120 | { 121 | if ( |group| <= |sub_group| && group == sub_group[:|group|] ) 122 | { 123 | ZeekAgent::send_subscribe(host_topic, s$query); 124 | break; 125 | } 126 | } 127 | } 128 | } 129 | } 130 | 131 | ## Checks for groups that match the recently added address 132 | function send_joins_new_address(host_id: string, ip: addr) 133 | { 134 | local host_topic = fmt("%s/%s", ZeekAgent::HostIndividualTopic,host_id); 135 | local new_groups: vector of string; 136 | for ( i in groupings ) 137 | { 138 | local c = groupings[i]; 139 | 140 | if ( c$group == "" ) 141 | { 142 | # Skip because Collection was deleted 143 | next; 144 | } 145 | 146 | for ( k in c$ranges ) 147 | { 148 | local range = c$ranges[k]; 149 | if ( ip in range ) 150 | { 151 | local new_group: string = c$group; 152 | ZeekAgent::log("info", host_id, fmt("joining new group %s", new_group)); 153 | ZeekAgent::send_join( host_topic, new_group ); 154 | host_groups[host_id] += new_group; 155 | new_groups += new_group; 156 | break; 157 | } 158 | } 159 | } 160 | 161 | for ( g in new_groups ) 162 | { 163 | local group = new_groups[g]; 164 | send_subscriptions_new_group(host_id, group); 165 | } 166 | } 167 | 168 | hook ZeekAgent::add_host_addr(host_id: string, ip: addr) 169 | { 170 | send_joins_new_address(host_id, ip); 171 | } 172 | 173 | hook ZeekAgent::add_host_addr(host_id: string, ip: addr) 174 | { 175 | #TODO 176 | } 177 | 178 | @if ( !Cluster::is_enabled() || Cluster::local_node_type() == Cluster::MANAGER ) 179 | event ZeekAgent::host_new(peer_name: string, hostname: string, host_id: string, group_list: vector of string, zeek_agent_version: string, zeek_agent_edition: string) 180 | { 181 | # 'standalone' edition only comes with built-in tables 182 | # 'osquery' edition can export additional tables IF osquery is connected 183 | ZeekAgent::log("info", host_id, fmt("Zeek Agent v%s (%s) host connected (%s announced as: %s)", zeek_agent_version, zeek_agent_edition, peer_name, host_id)); 184 | 185 | #TODO: Do not load the osquery tables if the client edition is set to 'standalone' 186 | # ... 187 | 188 | # Internal client tracking 189 | peer_to_host[peer_name] = host_id; 190 | hosts[host_id] = [$uuid = host_id, $hostname=hostname]; 191 | for ( i in group_list ) 192 | { 193 | add groups[group_list[i]]; 194 | } 195 | 196 | host_groups[host_id] = group_list; 197 | #TODO: that is only the topic prefix 198 | host_groups[host_id] += ZeekAgent::HostIndividualTopic; 199 | 200 | # Make host to join group and to schedule queries 201 | send_subscriptions_new_host(host_id); 202 | 203 | # raise event for new host 204 | event ZeekAgent::host_connected(host_id); 205 | } 206 | 207 | function _reset_peer(peer_name: string) 208 | { 209 | if ( peer_name !in peer_to_host ) 210 | return; 211 | 212 | local host_id: string = peer_to_host[peer_name]; 213 | ZeekAgent::log("info", host_id, "Zeek Agent host disconnected"); 214 | 215 | # Check if anyone else is left in the groups 216 | local others_groups: set[string]; 217 | # Collect set of groups others are in 218 | for ( i in host_groups ) 219 | { 220 | if ( i != host_id ) 221 | { 222 | for ( j in host_groups[i] ) 223 | { 224 | add others_groups[host_groups[i][j]]; 225 | } 226 | } 227 | } 228 | 229 | # Remove group if no one else has the group 230 | for ( k in host_groups[host_id] ) 231 | { 232 | local host_g: string = host_groups[host_id][k]; 233 | if ( host_g !in others_groups ) 234 | { 235 | delete groups[host_g]; 236 | } 237 | } 238 | } 239 | 240 | function _remove_peer(peer_name: string) 241 | { 242 | if ( peer_name !in peer_to_host ) 243 | return; 244 | 245 | local host_id: string = peer_to_host[peer_name]; 246 | delete peer_to_host[peer_name]; 247 | 248 | # Internal client tracking 249 | delete hosts[host_id]; 250 | delete host_groups[host_id]; 251 | } 252 | 253 | event Broker::peer_added(endpoint: Broker::EndpointInfo, msg: string) 254 | { 255 | local peer_name = endpoint$id; 256 | 257 | # Connect balance 258 | if ( peer_name !in connect_balance ) 259 | { 260 | connect_balance[peer_name] = 0; 261 | } 262 | 263 | if ( connect_balance[peer_name] > 0 ) 264 | { 265 | _reset_peer(peer_name); 266 | } 267 | connect_balance[peer_name] += 1; 268 | } 269 | 270 | 271 | event Broker::peer_lost(endpoint: Broker::EndpointInfo, msg: string) 272 | { 273 | local peer_name = endpoint$id; 274 | 275 | if ( peer_name in peer_to_host ) 276 | { 277 | local host_id: string = peer_to_host[peer_name]; 278 | ZeekAgent::log("info", host_id, "Zeek Agent host disconnected"); 279 | 280 | # raise event for the disconnected host 281 | event ZeekAgent::host_disconnected(host_id); 282 | } 283 | 284 | # Connect balance 285 | if ( connect_balance[peer_name] == 1 ) 286 | { 287 | _reset_peer(peer_name); 288 | _remove_peer(peer_name); 289 | } 290 | connect_balance[peer_name] -= 1; 291 | 292 | if ( connect_balance[peer_name] == 0 ) 293 | { 294 | delete connect_balance[peer_name]; 295 | } 296 | } 297 | 298 | event zeek_init() 299 | { 300 | # Listen on host announce topic 301 | local topic: string = ZeekAgent::HostAnnounceTopic; 302 | ZeekAgent::log_local("info", fmt("Subscribing to host announce topic %s", topic)); 303 | Broker::subscribe(topic); 304 | } 305 | @endif 306 | -------------------------------------------------------------------------------- /zeek-agent/framework/zeek_agent_subscriptions.zeek: -------------------------------------------------------------------------------- 1 | module ZeekAgent; 2 | 3 | export { 4 | ## Subscribe to an event. Whenever an zeek-agent connects to us, we'll subscribe to all matching activity 5 | ## from it. 6 | ## 7 | ## The query is an mandatory parameter and contains one query. It is send to the specified hosts 8 | ## and the specified groups. If neither is given, the query is broadcasted to all hosts. 9 | ## 10 | ## q: The queries to subscribe to. 11 | ## host_list: Specific hosts to address per query (optional). 12 | ## group_list: Specific groups to address per query (optional). 13 | global insert_subscription: function(q: ZeekAgent::Query, host_list: vector of string &default=vector(""), group_list: vector of string &default=vector("")); 14 | 15 | ## Unsubscribe from an events. This will get sent to all clients that are currently connected and would match 16 | ## a similar subscribe call. 17 | ## 18 | ## The query is an mandatory parameter and contains one query. It is send to the specified hosts 19 | ## and the specified groups. If neither is given, the query is broadcasted to all hosts. 20 | ## 21 | ## q: The queries to revoke. 22 | ## host_list: Specific hosts to address per query (optional). 23 | ## group_list: Specific groups to address per query (optional). 24 | global remove_subscription: function(q: ZeekAgent::Query, host_list: vector of string &default=vector(""), group_list: vector of string &default=vector("")); 25 | 26 | ## Send a one-time query to all currently connected clients. 27 | ## 28 | ## The query is an mandatory parameter and contains one query. It is send to the specified hosts 29 | ## and the specified groups. If neither is given, the query is broadcasted to all hosts. 30 | ## 31 | ## q: The queries to execute. 32 | ## host_list: Specific hosts to address per query (optional). 33 | ## group_list: Specific groups to address per query (optional). 34 | global insert_execution: function(q: ZeekAgent::Query, host_list: vector of string &default=vector(""), group_list: vector of string &default=vector("")); 35 | 36 | ## Make subnets to be addressed by a group. Whenever an zeek-agent connects to us, we'll instruct it to join 37 | ## the given group. 38 | ## 39 | ## range_list: the subnets that are addressed. 40 | ## group: the group hosts should join. 41 | global insert_grouping: function(range_list: vector of subnet, group: string); 42 | 43 | ## Make subnets to be no longer addressed by a group. This will get sent to all clients that are currently connected and would match 44 | ## a similar join call 45 | ## 46 | ## range_list: the subnets that are addressed. 47 | ## group: the group hosts should leave. 48 | global remove_grouping: function(range_list: vector of subnet, group: string); 49 | 50 | ## Hook to retrieve the IP addresses of a host by its id 51 | ## 52 | ## host_id: The identifier of the host 53 | global getIPsOfHost: hook(host_id: string, addresses: vector of addr); 54 | 55 | ## Peer Status of host 56 | ## 57 | ## host_id: The identifier of the host 58 | global isHostAlive: function(host_id: string): bool; 59 | 60 | type HostInfo: record { 61 | uuid: string; ##< UUID of host 62 | hostname: string &default=""; ##< self-reported name of host 63 | }; 64 | 65 | ## Get meta data about a connected host. If 66 | ## called for a host with isHostAlive() = false, will return 67 | ## a dummy record. 68 | global getHostInfo: function(host_id: string) : HostInfo; 69 | 70 | # Internal record for tracking a subscription. 71 | type Subscription: record { 72 | query: ZeekAgent::Query; 73 | hosts: vector of string &default=vector(); 74 | groups: vector of string &default=vector(); 75 | }; 76 | 77 | type Subscriptions: vector of Subscription; 78 | 79 | # Internal record for tracking groups 80 | type Grouping: record { 81 | group: string; 82 | ranges: vector of subnet &default=vector(); 83 | }; 84 | 85 | type Groupings: vector of Grouping; 86 | } 87 | 88 | # Internal vector of subscriptions 89 | global subscriptions: Subscriptions; 90 | 91 | # Internal vector of host groupins 92 | global groupings: Groupings; 93 | 94 | # Internal set for tracing client ids 95 | global hosts: table[string] of HostInfo; 96 | 97 | # Internal set for groups of clients 98 | global groups: set[string] = {ZeekAgent::HostBroadcastTopic}; 99 | 100 | # Internal table for tracking client (ids) and their respective groups 101 | global host_groups: table[string] of vector of string; 102 | 103 | # Internal mapping of broker id (peer_name) to zeek-agent id (host_id) 104 | global peer_to_host: table[string] of string; 105 | 106 | function insert_subscription(q: ZeekAgent::Query, host_list: vector of string, group_list: vector of string) 107 | { 108 | # Include new Subscription in the vector 109 | subscriptions += Subscription($query=q, $hosts=host_list, $groups=group_list); 110 | if ( |host_list| <= 1 && host_list[0] == "" && 111 | |group_list| <= 1 && group_list[0] == "" ) 112 | { 113 | # To all if nothing specified 114 | ZeekAgent::send_subscribe(ZeekAgent::HostBroadcastTopic, q); 115 | } 116 | else 117 | { 118 | # To specific host 119 | for ( j in host_list ) 120 | { 121 | if ( host_list[j] != "" ) 122 | { 123 | ZeekAgent::send_subscribe(fmt("%s/%s", ZeekAgent::HostIndividualTopic,host_list[j]), q); 124 | } 125 | } 126 | 127 | # To specific group 128 | for ( j in group_list ) 129 | { 130 | if ( group_list[j] != "" ) 131 | { 132 | ZeekAgent::send_subscribe(fmt("%s/%s", ZeekAgent::HostGroupTopic,group_list[j]), q); 133 | } 134 | } 135 | } 136 | } 137 | 138 | function remove_subscription(q: ZeekAgent::Query, host_list: vector of string, group_list: vector of string) 139 | { 140 | # Cancel internal subscription 141 | for ( j in subscriptions ) 142 | { 143 | if ( ZeekAgent::same_event(subscriptions[j]$query, q) ) 144 | { 145 | # Don't have a delete for vector, so set it to no-op by leaving the event empty. 146 | subscriptions[j]$query = Query($query=""); 147 | } 148 | } 149 | 150 | # Send unsubscribe 151 | if ( |host_list| <= 1 && host_list[0] == "" && 152 | |group_list| <= 1 && group_list[0] == "" ) 153 | { 154 | # To all if nothing specified 155 | ZeekAgent::send_unsubscribe(ZeekAgent::HostBroadcastTopic, q); 156 | } 157 | else 158 | { 159 | # To specific host 160 | for ( j in host_list ) 161 | { 162 | if ( host_list[j] != "" ) 163 | { 164 | ZeekAgent::send_unsubscribe(fmt("%s/%s", ZeekAgent::HostIndividualTopic,host_list[j]), q); 165 | } 166 | } 167 | 168 | # To specific group 169 | for ( j in group_list ) 170 | { 171 | if ( group_list[j] != "" ) 172 | { 173 | ZeekAgent::send_unsubscribe(fmt("%s/%s", ZeekAgent::HostGroupTopic,group_list[j]), q); 174 | } 175 | } 176 | } 177 | } 178 | 179 | function insert_execution(q: ZeekAgent::Query, host_list: vector of string, group_list: vector of string) 180 | { 181 | if ( |host_list| <= 1 && host_list[0] == "" && 182 | |group_list| <= 1 && group_list[0] == "" ) 183 | { 184 | # To all if nothing specified 185 | ZeekAgent::send_execute(ZeekAgent::HostBroadcastTopic, q); 186 | } 187 | else 188 | { 189 | # To specific host 190 | for ( j in host_list ) 191 | { 192 | if ( host_list[j] != "" ) 193 | { 194 | ZeekAgent::send_execute(fmt("%s/%s", ZeekAgent::HostIndividualTopic, host_list[j]), q); 195 | } 196 | } 197 | 198 | # To specific group 199 | for ( j in group_list ) 200 | { 201 | if ( group_list[j] != "" ) 202 | { 203 | ZeekAgent::send_execute(fmt("%s/%s", ZeekAgent::HostGroupTopic, group_list[j]), q); 204 | } 205 | } 206 | } 207 | } 208 | 209 | function insert_grouping(range_list: vector of subnet, group: string) 210 | { 211 | # Include new Grouping in the vector 212 | groupings += Grouping($group=group, $ranges=range_list); 213 | 214 | for ( host in hosts ) 215 | { 216 | local host_topic = fmt("%s/%s", ZeekAgent::HostIndividualTopic,host); 217 | local skip_host = F; 218 | 219 | local hostIPs: vector of addr; 220 | hook ZeekAgent::getIPsOfHost(host, hostIPs); 221 | for ( j in hostIPs ) 222 | { 223 | if ( skip_host ) 224 | break; 225 | 226 | for ( i in range_list ) 227 | { 228 | if ( hostIPs[j] in range_list[i] ) 229 | { 230 | local new_group = group; 231 | ZeekAgent::log("info", host, fmt("Joining new group %s", new_group)); 232 | ZeekAgent::send_join(host_topic, new_group); 233 | host_groups[host] += new_group; 234 | add groups[new_group]; 235 | skip_host = T; 236 | break; 237 | } 238 | } 239 | } 240 | } 241 | } 242 | 243 | function remove_grouping(range_list: vector of subnet, group: string) 244 | { 245 | #TODO notImplemented 246 | } 247 | 248 | function isHostAlive(host_id: string): bool 249 | { 250 | return host_id in hosts; 251 | } 252 | 253 | function getHostInfo(host_id: string) : HostInfo 254 | { 255 | if ( host_id in hosts ) 256 | return hosts[host_id]; 257 | else 258 | return [$uuid=host_id]; 259 | } 260 | 261 | 262 | 263 | -------------------------------------------------------------------------------- /zeek-agent/framework/zeek_logger.zeek: -------------------------------------------------------------------------------- 1 | #! Query processes activity. 2 | 3 | module ZeekAgent; 4 | 5 | export { 6 | global log_added: event(t: time, host_id: string, event_time: int, severity: string, message: string); 7 | global log_removed: event(t: time, host_id: string, event_time: int, severity: string, message: string); 8 | } 9 | 10 | event ZeekAgent::table_logger(result: ZeekAgent::Result, event_time: int, severity: string, message: string) 11 | { 12 | if ( result$utype == ZeekAgent::ADD ) 13 | event ZeekAgent::log_added(network_time(), result$host, event_time, severity, message); 14 | 15 | if ( result$utype == ZeekAgent::REMOVE) 16 | event ZeekAgent::log_removed(network_time(), result$host, event_time, severity, message); 17 | } 18 | 19 | event zeek_init() 20 | { 21 | local query = ZeekAgent::Query($ev=ZeekAgent::table_logger, 22 | $query="SELECT time, severity, message FROM zeek_logger", 23 | $utype=ZeekAgent::BOTH); 24 | ZeekAgent::subscribe(query); 25 | } 26 | -------------------------------------------------------------------------------- /zeek-agent/framework/zeek_logger_log.zeek: -------------------------------------------------------------------------------- 1 | 2 | #! Logs socket events activity 3 | 4 | module ZeekAgent::logging::table_logger; 5 | 6 | export { 7 | redef enum Log::ID += { LOG }; 8 | 9 | type Info: record { 10 | ts: time &log; 11 | host: string &log; 12 | event_time: int &log; 13 | severity: string &log; 14 | message: string &log; 15 | }; 16 | } 17 | 18 | @if ( !Cluster::is_enabled() || Cluster::local_node_type() == Cluster::MANAGER ) 19 | event ZeekAgent::log_added(ts: time, host_id: string, event_time :int, severity :string, message :string) 20 | { 21 | local info = Info($ts=ts, 22 | $host=host_id, 23 | $event_time=event_time, 24 | $severity=severity, 25 | $message=message); 26 | 27 | Log::write(LOG, info); 28 | } 29 | @endif 30 | 31 | event zeek_init() &priority=10 32 | { 33 | Log::create_stream(LOG, [$columns=Info, $path="zeek-agent-logger"]); 34 | } 35 | -------------------------------------------------------------------------------- /zkg.meta: -------------------------------------------------------------------------------- 1 | [package] 2 | version = 0.4-dev 3 | description = Script framework for communicating with zeek-agent endpoints 4 | tags = zeek-agent,osquery 5 | script_dir = zeek-agent 6 | depends = 7 | zeek >=3.0.0 8 | --------------------------------------------------------------------------------