├── .gitattributes
├── ElasticMARC Guide.docx
├── README.md
├── elasticsearch
    └── config
    │   ├── elasticsearch.yml
    │   └── jvm.options
├── kibana
    ├── config
    │   └── kibana.yml
    ├── logs
    │   └── kibana.log
    └── visuals
    │   ├── DMARCDashboards.json
    │   ├── DMARCSearches.json
    │   └── DMARCVisuals.json
├── logstash
    ├── bin
    │   ├── DMARCNMA.ps1
    │   ├── DmarcUnattended.ps1
    │   └── dmarcscript.ps1
    ├── config
    │   ├── jvm.options
    │   ├── logstash.yml
    │   ├── pipelines.yml
    │   └── pipelines
    │   │   ├── beatspipeline.yml
    │   │   └── dmarcpipeline.yml
    └── templates
    │   └── dmarcxmltemplate.json
└── samplereport.xml


/.gitattributes:
--------------------------------------------------------------------------------
1 | # Auto detect text files and perform LF normalization
2 | * text=auto
3 | 


--------------------------------------------------------------------------------
/ElasticMARC Guide.docx:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Grunticus03/ElasticMARC/4231d9f1f77e3e3acf97b959511b3eee1354764e/ElasticMARC Guide.docx


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
  1 | ![Logo](https://github.com/wwalker0307/ElasticMARC/blob/assets/ElasticMARC.JPG?raw=true)
  2 | 
  3 | ElasticMARC is a solution developed using Elastic's [Elastic Stack](https://www.elastic.co/products) to ingest, enrich, and visualize DMARC aggregate report data.  The primary focus of ElasticMARC is to provide a simple, guided setup utilizing a Windows platform.  While Linux platforms can utilize most of this setup, a PowerShell script is used to modify the structure of the XML reports prior to being ingested by Elastic Stack.
  4 | 
  5 | ![alt text](https://github.com/wwalker0307/ElasticMARC/blob/assets/Dashboard.JPG?raw=true)
  6 | 
  7 | ![alt text](https://github.com/wwalker0307/ElasticMARC/blob/assets/Event.JPG?raw=true)
  8 | 
  9 | Required Software
 10 | ------
 11 | •	[Elasticsearch](https://www.elastic.co/downloads/elasticsearch)
 12 | 
 13 | •	[Logstash](https://www.elastic.co/downloads/logstash)
 14 | 
 15 | •	[Kibana](https://www.elastic.co/downloads/kibana)
 16 | 
 17 | •	[Non-Sucking Service Manager (NSSM)](https://nssm.cc/download)
 18 | 
 19 | •	[Notepad++ (Optional)](https://notepad-plus-plus.org/download/v7.5.6.html)
 20 | 
 21 | <br/>
 22 | 
 23 | Pre-requisites
 24 | ------
 25 | Prior to installing the Elastic Stack, the following modifications are required.
 26 | 
 27 | ### Disable Windows Page File (Swap Disk)
 28 | Failure to disable the Page File can have a significant impact on the performance and reliability of the Elastic Stack.
 29 | 1.	Open the System Properties window located in the control panel.
 30 | 2.	Select Advanced System Settings
 31 | 3.	On the Advanced tab, select Settings… under the Performance section.
 32 | 4.	Select the Advanced tab on the following window and then Change…
 33 | 5.	Uncheck Automatically manage paging file size for all drives, select the No paging file button, and click Set.6
 34 | 6.	Reboot computer.
 35 | 
 36 | ### Set JAVA Environment Variable
 37 | The Elasticstack relies on Java.  Current releases include Java in their download package.  However, you need to set an environment variable to tell the applications where to look for Java.
 38 | 
 39 | | Variable Name | Variable Value |
 40 | | :--- | :--- |
 41 | | JAVA_HOME | JAVAROOTFOLDER (E.G. D:\Elastic\elasticsearch\jdk) |
 42 | 
 43 | Miscellaneous Considerations
 44 | ------
 45 | ### Hardware Requirements
 46 | For the purposes of this implementation; one CPU and 6 GB RAM should be sufficient.  If you intend to ingest more data, such as with Beats agents, you may need to allocate more resources.
 47 | 
 48 | ### Path Referencing in This Guide
 49 | For simplicity and consistency, when referring to the installation location of an application, root shall imply drive letter and path to the application.  For example, D:\Elastic Stack\Elasticsearch\bin will be root\bin.
 50 | 
 51 | ### Example Configurations
 52 | Prebuilt example configuration files are included for each Elastic Stack application.  These files require minimal modifications and are intended to get you up and running on a basic Elastic Stack implementation.
 53 | 
 54 | ### Time Stamping
 55 | When creating an Index, you have three options for the Time Filter field.  The decision on which field to use is dependent on your organization’s desires.  Be aware, once you’ve selected a field, you cannot change it without first recreating the index and then removing all previously indexed data.
 56 | 
 57 | | Field | Purpose |
 58 | | :--- | :--- |
 59 | | @timestamp | This will tag each event with the date and time that it was processed by Logstash. |
 60 | | Report.start | Start time for the reporting period defined in each XML |
 61 | | Report.end | End time for the reporting period defined in each XML |
 62 | 
 63 | ### Elastic Stack Applications
 64 | The Elastic Stack applications do not have an installation process or executable.  Wherever you decompress the archives effectively becomes the installation location.  Ensure that you place the files in the proper location prior to configuration.
 65 | 
 66 | ### Recommended Editor
 67 | It is highly recommended that you use a text editor like Notepad++ to maintain proper encoding of the configuration files.  It also generally just makes for a friendlier method of working with configuration files.
 68 | <br/>
 69 | 
 70 | Elasticsearch Installation
 71 | ------
 72 | 1.	Decompress Elasticsearch to your intended installation location.
 73 | 2.	Download and Decompress ElasticMARC to a temporary location
 74 | 3.	Copy the contents of ElasticMARC\elasticsearch to the Elasticsearch directory, overwriting any existing files.
 75 | 4.	Open root\config\elasticsearch.yml and modify the following:
 76 | 
 77 | | Setting | Value | Default |
 78 | | :--- | :--- | :--- |
 79 | | Node.name: | HostnameOfComputer |  |
 80 | | Network.host: | IPv4 address Elasticsearch will listen on, use 0.0.0.0 to listen on all addresses. |  |
 81 | | http.port: | Port Elasticsearch will listen on | 9200 |
 82 | | (Optional) path.data: | Where Elasticsearch will store indexed data. | root\data|
 83 | | (Optional) path.logs: | Where Elasticsearch will store logs. | root\logs |
 84 | <br/>
 85 | <br/>
 86 | 5.	Open root\config\jvm.options and modify the following, if necessary:
 87 | 
 88 | | Setting | Value | Default |
 89 | | :--- | :--- | :--- |
 90 | | -Xms | Initial RAM Elasticsearch JVM will use. | 1g |
 91 | | -Xmx | Max RAM Elasticsearch JVM will use. | 1g |
 92 | *   Xms and Xmx should be set to the same size.  If they are not, you may experience performance issues.  These values represent the amount of RAM the Elasticsearch JVM will allocate.  For the purposes of this guide, 1GB is sufficient.
 93 | <br/>
 94 | 6.	Open an administrative CMD window and enter the following commands: <br/>
 95 | Root\bin\elasticsearch-service.bat install<br/>
 96 | Root\bin\elasticsearch-service.bat manager<br/>
 97 | 7.	In the window that appears, modify the following:
 98 | 
 99 | | Setting | Value |
100 | | :--- | :--- |
101 | | (Optional) Display Name: | I prefer to remove the version information |
102 | | Startup Type: | Automatic |
103 | <br/>
104 | 8.	Select apply, start the service, and close the service manager window.
105 | <br/>
106 | ***Elasticsearch installation is now complete!***
107 | 
108 | <br/>
109 | 
110 | Kibana Installation
111 | ------
112 | 1.	Decompress Kibana to your intended installation location.
113 | 2.	Copy the contents of ElasticMARC\kibana to the Kibana directory, overwriting any existing files.
114 | 3.	Open root\config\kibana.yml and modify the following:
115 | 
116 | | Setting | Value | Default |
117 | | :--- | :--- | :--- |
118 | | Server.port: | Port to listen on | 5601 |
119 | | Server.host: | Server hostname |  |
120 | | Server.name: | Server hostname |  |
121 | | Elasticsearch.url: | http://SERVERHOSTNAME:IP |  |
122 | | Logging.dest: | File and path for logging.  Folder must exist, file will be created, preserve double quotes | root\log |
123 | *   If you want to change the logging level, change the appropriate logging line value to true.
124 | *   Kibana does not have a service installer, we will utilize NSSM to create a service for Kibana.
125 | <br/>
126 | 4.	Decompress NSSM to your intended installation location.</br>
127 | 5.	Open an administrative CMD prompt and enter the following command:</br>
128 | 		Root\nssm\win64\nssm.exe install Kibana</br>
129 | 6.	On the Application tab, set the following:
130 | 
131 | | Setting | Value |
132 | | :--- | :--- |
133 | | Path: | Root\bin\kibana.bat |
134 | | Startup Directory: | root\bin |
135 | <br/>
136 | <br/>
137 | 7.	On the Details tab, set the following
138 | 
139 | | Setting | Value |
140 | | :--- | :--- |
141 | | Display Name: | Kibana |
142 | | (Optional) Description: | Kibana VER (I.E. Kibana 6.2.2) |
143 | | Startup Type: | Automatic |
144 | <br/>
145 | <br/>
146 | 8.	Select Install Service and click OK to finish.</br>
147 | 9.	In the administrative CMD prompt enter the following to start the Kibana service.</br>
148 | 		Powershell -c Start-Service Kibana</br>
149 | 10.	After a few moments, you can verify Kibana’s functionality by opening a browser and pointing it to http://hostname:port as configured in Kibana.yml’s server.host and server.port properties.</br>
150 |  <br/>
151 | 
152 | Logstash Installation
153 | ------
154 | 1.	Decompress Logstash to your intended installation location.
155 | 2.	Copy the contents of ElasticMARC\logstashto the logstash directory, overwriting any existing files.
156 | 3.	Create a folder that will be the ingest point for the DMARC Aggregate reports.
157 | 4.	Open root\config\logastash.yml and modify the following:
158 | 
159 | | Setting | Value |
160 | | :--- | :--- |
161 | | Node.name: | Server hostname |
162 | | http.host: | IPv4 Address of Logstash server |
163 | | http.port: | Port to listen on |
164 | | (Optional) Log.level: | Uncomment and set to desired level. Trace is most detailed but very chatty.  Debug is usually sufficient for troubleshooting |
165 | <br/>
166 | <br/>
167 | 5.	Open root\config\jvm.options and modify the following:
168 | 
169 | | Setting | Value | Default |
170 | | :--- | :--- | :--- |
171 | | -Xms | Initial RAM used by Logstash JVM | 1g |
172 | | -Xmx | Max RAM used by Logstash JVM | 1g |
173 | *   Xms and Xmx should be set to the same size.  If they are not, you may experience performance issues.  These values represent the amount of RAM the Logstash JVM will allocate.  For the purposes of this guide, 1GB is sufficient.
174 | <br/>
175 | <br/>
176 | 6.	Open root\config\pipelines.yml and modify the following:
177 | 
178 | | Setting | Value |
179 | | :--- | :--- |
180 | | Path.config: | /root/config/pipelines/dmarcpipeline.yml. Do not use a drive letter, use forward slashes, preserve double quotes |
181 | *   (Optional) If you’d like to implement Beats data ingesting, you can uncomment the second set of pipeline values that are pre-configured for this purpose.
182 | <br/>
183 | <br/>
184 | 7.	Open root\config\pipelines\dmarcpipeline.yml and modify the following:
185 | 
186 | | Setting | Value |
187 | | :--- | :--- |
188 | | Line 3 id => | Cosmetic tag assigned to input of pipeline. Set to folder ingesting the XML files, preserve double quotes |
189 | | Line 4 path => | Folder Logstash monitors for files to ingest. Use forward slashes, preserve double quotes, use *.xml after folder path |
190 | | Line 95 hosts => | ServerName:Port Logstash sends data to once it’s been processed. Preserve brackets and double quotes |
191 | | Line 98 template => | Location of Elasticsearch template, use drive letter, forward slashes in path, preserve quotes |
192 | <br/>
193 | <br/>
194 | 8.	(Optional) If implementing Beats, open root\config\pipelines\beatspipeline.yml and modify the following:
195 | 
196 | | Setting | Value |
197 | | :--- | :--- |
198 | | Line 12 hosts => | ServerName:Port Logstash sends data to once it’s been processed. Preserve brackets and double quotes |</br>
199 | *   Logstash does not have a service installer, we will utilize NSSM to create a service for Logstash. In the following steps, root refers to the location that NSSM has been extracted to.<br/>
200 | 
201 | 9.	Open an administrative CMD prompt and enter the following command:<br/>
202 | 		Root\win64\nssm.exe install Logstash
203 | 10.	On the Application tab, enter the following:
204 | 
205 | | Setting | Value |
206 | | :--- | :--- |
207 | | Path: | root\bin\logstash.bat |
208 | | Startup Directory: | root\bin |
209 | <br/>
210 | <br/>
211 | 11.	On the Details tab, enter the following:
212 | 
213 | | Setting | Value |
214 | | :--- | :--- |
215 | | Display Name: | Logstash |
216 | | (Optional) Description: | Logstash VER (I.E. Logstash 6.2.2) |
217 | | Startup Type: | Automatic |
218 | <br/>
219 | <br/>
220 | 12.	Select, Install Service and click OK to finish.<br/>
221 | 13.	In the administrative CMD prompt enter the following to start the Logstash service.<br/>
222 | 		Powershell -c Start-Service Logstash<br/><br/>
223 | ***Logstash installation is now complete!***
224 | <br/>
225 | 
226 | Configuring Kibana
227 | ------
228 | At this point, the Elastic Stack installation is complete and ready to start ingesting data.  Before we start visualizing the reports, we need to ingest some sample data.  This will allow us to create an index pattern and import the preconfigured visualizations and dashboards that are included.   A sample report is included and exists alongside where this report was extracted to.
229 | ### Basic Kibana Configuration
230 | URLs in Kibana can get large as you start manipulating data and especially when loading a dashboard with many visualizations.  For this reason, I recommend changing Kibana to store the URL with the session.
231 | 1.	Open a browser and go to your Kibana instance
232 | 2.	Select Management from the menu on the left, then Advanced Settings.
233 | 3.	Set state:storeInSessionStorage to true
234 | 4.	I recommend going through the remaining settings in this section, but take caution as these settings can break your installation if improperly configured.
235 | 
236 | ### Ingest Sample Data
237 | 1.	Open a Powershell window and execute the following:
238 | 2.	LogstashRoot\bin\dmarcscript.ps1
239 | 3.	Enter the folder path containing the sample report XML.
240 | 4.	Enter the folder path that Logstash is monitoring.
241 | 5.	Assuming all pre-requisites are met, PowerShell will modify the XML structure and save the modified file to the specified ingest folder.  From here, Logstash will ingest, parse, and output the data to Elasticsearch.
242 | 
243 | ### Index Pattern Creation
244 | 1.	Open a browser and navigate to your Kibana instance
245 | 2.	Click Management on the left side, then Index Patterns.
246 | 3.	You will see a list of indexes that have been created.  If this is a new install, there should be only one named dmarcxml-YYYY.MM.dd.
247 | 4.	Enter dmarcxml-* for the index pattern and click Next Step
248 | 5.	Select a Time Filter field name
249 | *	See Miscellaneous Considerations near the top of this guide for an explanation of these fields.
250 | 6.	Expand Show advanced options and enter dmarcxml-* as a custom index pattern ID
251 | 7.	Click Create Index Pattern to finish index creation.
252 | 
253 | ### Visualizations & Dashboard Import
254 | Sample dashboards and visualizations have been created to assist in familiarization of the Kibana interface and get new users up and running quickly.
255 | 1.	Open a browser and navigate to your Kibana instance.
256 | 2.	Select Management on the left side, then Saved Objects.
257 | 3.	Click the Import button at the top right of this page.
258 | 4.	Navigate to the kibana\visuals folder and select DMARCsearches.json
259 | 5.	If prompted, select Yes, to overwrite all saved objects.
260 | 6.	Repeat step 4 to import DMARCVisuals.json and DMARCDashboards.json.
261 | 7.	To view the preconfigured dashboards, select Dashboard on the left side of the page.
262 | 8.	To view individual visualizations, select Visualize on the left side of the page.
263 | 
264 | ### Optional Field Formatting
265 | Kibana provides the ability to format fields in a variety of ways.  In particular, you can create links on fields utilizing the field value as part of the URL.  Process to do this is outlined below.
266 | 1.	Open a browser and navigate to your Kibana instance.
267 | 2.	Select Management on the left side, then Index Patterns.
268 | 3.	Locate the auth_result.spf_domain field and click the pencil icon in the controls column.
269 | 4.	Use the following values:
270 | 
271 | | Setting | Value |
272 | | :--- | :--- |
273 | | Format: | URL |
274 | | Type: | Link |
275 | | URL Template: | `https://dig.whois.com.au/whois/{{value}}` |
276 | | Label Template: | {{value}} |
277 | *   In addition, you can also use `https://www.google.com/maps/place/{{value}}` on many of the geographic fields, including the coordinates keyword field to link to Google Maps.
278 | 


--------------------------------------------------------------------------------
/elasticsearch/config/elasticsearch.yml:
--------------------------------------------------------------------------------
1 | node.name: ServerHostname
2 | bootstrap.memory_lock: true
3 | network.host: 0.0.0.0
4 | http.port: 9200
5 | #path.data: C:/ElasticStack/Data/ElasticSearch
6 | 


--------------------------------------------------------------------------------
/elasticsearch/config/jvm.options:
--------------------------------------------------------------------------------
  1 | ## JVM configuration
  2 | 
  3 | ################################################################
  4 | ## IMPORTANT: JVM heap size
  5 | ################################################################
  6 | ##
  7 | ## You should always set the min and max JVM heap
  8 | ## size to the same value. For example, to set
  9 | ## the heap to 4 GB, set:
 10 | ##
 11 | ## -Xms4g
 12 | ## -Xmx4g
 13 | ##
 14 | ## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html
 15 | ## for more information
 16 | ##
 17 | ################################################################
 18 | 
 19 | # Xms represents the initial size of total heap space
 20 | # Xmx represents the maximum size of total heap space
 21 | 
 22 | -Xms1g
 23 | -Xmx1g
 24 | 
 25 | ################################################################
 26 | ## Expert settings
 27 | ################################################################
 28 | ##
 29 | ## All settings below this section are considered
 30 | ## expert settings. Don't tamper with them unless
 31 | ## you understand what you are doing
 32 | ##
 33 | ################################################################
 34 | 
 35 | ## GC configuration
 36 | -XX:+UseConcMarkSweepGC
 37 | -XX:CMSInitiatingOccupancyFraction=75
 38 | -XX:+UseCMSInitiatingOccupancyOnly
 39 | 
 40 | ## optimizations
 41 | 
 42 | # pre-touch memory pages used by the JVM during initialization
 43 | -XX:+AlwaysPreTouch
 44 | 
 45 | ## basic
 46 | 
 47 | # explicitly set the stack size
 48 | -Xss1m
 49 | 
 50 | # set to headless, just in case
 51 | -Djava.awt.headless=true
 52 | 
 53 | # ensure UTF-8 encoding by default (e.g. filenames)
 54 | -Dfile.encoding=UTF-8
 55 | 
 56 | # use our provided JNA always versus the system one
 57 | -Djna.nosys=true
 58 | 
 59 | # turn off a JDK optimization that throws away stack traces for common
 60 | # exceptions because stack traces are important for debugging
 61 | -XX:-OmitStackTraceInFastThrow
 62 | 
 63 | # flags to configure Netty
 64 | -Dio.netty.noUnsafe=true
 65 | -Dio.netty.noKeySetOptimization=true
 66 | -Dio.netty.recycler.maxCapacityPerThread=0
 67 | 
 68 | # log4j 2
 69 | -Dlog4j.shutdownHookEnabled=false
 70 | -Dlog4j2.disable.jmx=true
 71 | 
 72 | -Djava.io.tmpdir=${ES_TMPDIR}
 73 | 
 74 | ## heap dumps
 75 | 
 76 | # generate a heap dump when an allocation from the Java heap fails
 77 | # heap dumps are created in the working directory of the JVM
 78 | -XX:+HeapDumpOnOutOfMemoryError
 79 | 
 80 | # specify an alternative path for heap dumps
 81 | # ensure the directory exists and has sufficient space
 82 | #-XX:HeapDumpPath=/heap/dump/path
 83 | 
 84 | ## JDK 8 GC logging
 85 | 
 86 | 8:-XX:+PrintGCDetails
 87 | 8:-XX:+PrintGCDateStamps
 88 | 8:-XX:+PrintTenuringDistribution
 89 | 8:-XX:+PrintGCApplicationStoppedTime
 90 | 8:-Xloggc:logs/gc.log
 91 | 8:-XX:+UseGCLogFileRotation
 92 | 8:-XX:NumberOfGCLogFiles=32
 93 | 8:-XX:GCLogFileSize=64m
 94 | 
 95 | # JDK 9+ GC logging
 96 | 9-:-Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m
 97 | # due to internationalization enhancements in JDK 9 Elasticsearch need to set the provider to COMPAT otherwise
 98 | # time/date parsing will break in an incompatible way for some date patterns and locals
 99 | 9-:-Djava.locale.providers=COMPAT
100 | 


--------------------------------------------------------------------------------
/kibana/config/kibana.yml:
--------------------------------------------------------------------------------
1 | server:
2 |   port: 80
3 |   host: "ServerHostname"
4 |   name: "ServerHostname"
5 | elasticsearch.url: "http://ServerHostname:9200"
6 | 


--------------------------------------------------------------------------------
/kibana/logs/kibana.log:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Grunticus03/ElasticMARC/4231d9f1f77e3e3acf97b959511b3eee1354764e/kibana/logs/kibana.log


--------------------------------------------------------------------------------
/kibana/visuals/DMARCDashboards.json:
--------------------------------------------------------------------------------
 1 | [
 2 |   {
 3 |     "_id": "faaa0cf0-279c-11e8-b656-99c73f50969b",
 4 |     "_type": "dashboard",
 5 |     "_source": {
 6 |       "title": "DMARC",
 7 |       "hits": 0,
 8 |       "description": "",
 9 |       "panelsJSON": "[{\"embeddableConfig\":{\"mapCenter\":[14.26438308756265,3.5156250000000004],\"mapZoom\":2},\"gridData\":{\"h\":5,\"i\":\"1\",\"w\":6,\"x\":0,\"y\":0},\"id\":\"55fe84b0-0fb9-11e8-8dc5-e141d701899b\",\"panelIndex\":\"1\",\"title\":\"World\",\"type\":\"visualization\",\"version\":\"6.2.2\"},{\"embeddableConfig\":{\"mapCenter\":[37.09023980307208,-96.767578125],\"mapZoom\":4},\"gridData\":{\"h\":5,\"i\":\"2\",\"w\":6,\"x\":6,\"y\":0},\"id\":\"2a1cf250-1441-11e8-8dc5-e141d701899b\",\"panelIndex\":\"2\",\"title\":\"United States\",\"type\":\"visualization\",\"version\":\"6.2.2\"},{\"gridData\":{\"h\":2,\"i\":\"3\",\"w\":3,\"x\":0,\"y\":5},\"id\":\"bc238b40-2608-11e8-b656-99c73f50969b\",\"panelIndex\":\"3\",\"title\":\"\",\"type\":\"visualization\",\"version\":\"6.2.2\"},{\"gridData\":{\"h\":2,\"i\":\"4\",\"w\":3,\"x\":3,\"y\":5},\"id\":\"05ebb890-0fbd-11e8-8dc5-e141d701899b\",\"panelIndex\":\"4\",\"title\":\"\",\"type\":\"visualization\",\"version\":\"6.2.2\"},{\"gridData\":{\"h\":3,\"i\":\"5\",\"w\":3,\"x\":6,\"y\":5},\"id\":\"17d5b590-1f4b-11e8-9724-59524d75d35b\",\"panelIndex\":\"5\",\"title\":\"Top Senders (Country)\",\"type\":\"visualization\",\"version\":\"6.2.2\"},{\"gridData\":{\"h\":3,\"i\":\"6\",\"w\":3,\"x\":9,\"y\":5},\"id\":\"71737e10-1f4c-11e8-9724-59524d75d35b\",\"panelIndex\":\"6\",\"title\":\"Top Senders (Domain)\",\"type\":\"visualization\",\"version\":\"6.2.2\"},{\"gridData\":{\"h\":3,\"i\":\"7\",\"w\":3,\"x\":0,\"y\":7},\"id\":\"b7466350-0fb9-11e8-8dc5-e141d701899b\",\"panelIndex\":\"7\",\"title\":\"SPF Result\",\"type\":\"visualization\",\"version\":\"6.2.2\"},{\"gridData\":{\"h\":3,\"i\":\"8\",\"w\":3,\"x\":3,\"y\":7},\"id\":\"94996f50-0fb9-11e8-8dc5-e141d701899b\",\"panelIndex\":\"8\",\"title\":\"DKIM Result\",\"type\":\"visualization\",\"version\":\"6.2.2\"},{\"gridData\":{\"h\":3,\"i\":\"9\",\"w\":3,\"x\":0,\"y\":10},\"id\":\"b8693ba0-0fbd-11e8-8dc5-e141d701899b\",\"panelIndex\":\"9\",\"title\":\"Top Reporting Orgs\",\"type\":\"visualization\",\"version\":\"6.2.2\"},{\"gridData\":{\"h\":3,\"i\":\"10\",\"w\":3,\"x\":6,\"y\":8},\"id\":\"9627e210-0fba-11e8-8dc5-e141d701899b\",\"panelIndex\":\"10\",\"type\":\"visualization\",\"version\":\"6.2.2\"},{\"gridData\":{\"h\":3,\"i\":\"11\",\"w\":3,\"x\":9,\"y\":8},\"id\":\"7f45e1c0-59e1-11e8-8e48-d7c271d057a2\",\"panelIndex\":\"11\",\"type\":\"visualization\",\"version\":\"6.2.3\"}]",
10 |       "optionsJSON": "{\"darkTheme\":true,\"hidePanelTitles\":false,\"useMargins\":true}",
11 |       "version": 1,
12 |       "timeRestore": false,
13 |       "kibanaSavedObjectMeta": {
14 |         "searchSourceJSON": "{\"query\":{\"language\":\"lucene\",\"query\":\"\"},\"filter\":[],\"highlightAll\":true,\"version\":true}"
15 |       }
16 |     }
17 |   },
18 |   {
19 |     "_id": "f562dc20-2a05-11e8-ab27-8fbecdf8c5f8",
20 |     "_type": "dashboard",
21 |     "_source": {
22 |       "title": "DMARC - Searchable",
23 |       "hits": 0,
24 |       "description": "",
25 |       "panelsJSON": "[{\"panelIndex\":\"1\",\"gridData\":{\"x\":0,\"y\":0,\"w\":6,\"h\":8,\"i\":\"1\"},\"title\":\"\",\"version\":\"6.2.2\",\"type\":\"visualization\",\"id\":\"2a1cf250-1441-11e8-8dc5-e141d701899b\",\"embeddableConfig\":{\"mapZoom\":5,\"mapCenter\":[39.232253141714914,-97.64648437500001]}},{\"panelIndex\":\"2\",\"gridData\":{\"x\":6,\"y\":0,\"w\":6,\"h\":6,\"i\":\"2\"},\"title\":\"\",\"version\":\"6.2.2\",\"type\":\"search\",\"id\":\"285422c0-144d-11e8-8dc5-e141d701899b\"},{\"panelIndex\":\"3\",\"gridData\":{\"x\":6,\"y\":6,\"w\":2,\"h\":2,\"i\":\"3\"},\"title\":\"\",\"version\":\"6.2.2\",\"type\":\"visualization\",\"id\":\"05ebb890-0fbd-11e8-8dc5-e141d701899b\"},{\"panelIndex\":\"4\",\"gridData\":{\"x\":8,\"y\":6,\"w\":2,\"h\":2,\"i\":\"4\"},\"title\":\"Top Reporting Organizations\",\"version\":\"6.2.2\",\"type\":\"visualization\",\"id\":\"b8693ba0-0fbd-11e8-8dc5-e141d701899b\"},{\"panelIndex\":\"5\",\"gridData\":{\"x\":10,\"y\":6,\"w\":2,\"h\":2,\"i\":\"5\"},\"title\":\"SPF Result\",\"version\":\"6.2.2\",\"type\":\"visualization\",\"id\":\"b7466350-0fb9-11e8-8dc5-e141d701899b\"},{\"title\":\"Top Senders (City)\",\"panelIndex\":\"6\",\"gridData\":{\"x\":0,\"y\":8,\"w\":4,\"h\":3,\"i\":\"6\"},\"version\":\"6.2.2\",\"type\":\"visualization\",\"id\":\"a1fc8b00-1f4f-11e8-9724-59524d75d35b\"},{\"title\":\"Top Senders (Region)\",\"panelIndex\":\"7\",\"gridData\":{\"x\":4,\"y\":8,\"w\":4,\"h\":3,\"i\":\"7\"},\"version\":\"6.2.2\",\"type\":\"visualization\",\"id\":\"b3851180-1f4f-11e8-9724-59524d75d35b\"},{\"title\":\"Top Senders (Country)\",\"panelIndex\":\"8\",\"gridData\":{\"x\":8,\"y\":8,\"w\":4,\"h\":3,\"i\":\"8\"},\"version\":\"6.2.2\",\"type\":\"visualization\",\"id\":\"17d5b590-1f4b-11e8-9724-59524d75d35b\"}]",
26 |       "optionsJSON": "{\"darkTheme\":true,\"useMargins\":true,\"hidePanelTitles\":false}",
27 |       "version": 1,
28 |       "timeRestore": false,
29 |       "kibanaSavedObjectMeta": {
30 |         "searchSourceJSON": "{\"query\":{\"query\":\"\",\"language\":\"lucene\"},\"filter\":[],\"highlightAll\":true,\"version\":true}"
31 |       }
32 |     }
33 |   }
34 | ]


--------------------------------------------------------------------------------
/kibana/visuals/DMARCSearches.json:
--------------------------------------------------------------------------------
 1 | [
 2 |   {
 3 |     "_id": "37c99200-0fbe-11e8-8dc5-e141d701899b",
 4 |     "_type": "search",
 5 |     "_source": {
 6 |       "title": "DMARC - Reported Policies",
 7 |       "description": "",
 8 |       "hits": 0,
 9 |       "columns": [
10 |         "DMARC Policy Action",
11 |         "DKIM Mode",
12 |         "SPF Mode",
13 |         "Application Percentage"
14 |       ],
15 |       "sort": [
16 |         "@timestamp",
17 |         "desc"
18 |       ],
19 |       "version": 1,
20 |       "kibanaSavedObjectMeta": {
21 |         "searchSourceJSON": "{\"index\":\"dmarcxml-*\",\"highlightAll\":true,\"version\":true,\"query\":{\"query\":\"\",\"language\":\"lucene\"},\"filter\":[{\"meta\":{\"index\":\"dmarcxml-*\",\"negate\":false,\"disabled\":false,\"alias\":null,\"type\":\"exists\",\"key\":\"DMARC Policy Action\",\"value\":\"exists\"},\"exists\":{\"field\":\"DMARC Policy Action\"},\"$state\":{\"store\":\"appState\"}}]}"
22 |       }
23 |     }
24 |   },
25 |   {
26 |     "_id": "285422c0-144d-11e8-8dc5-e141d701899b",
27 |     "_type": "search",
28 |     "_source": {
29 |       "title": "DMARC - City, Region, Nation",
30 |       "description": "",
31 |       "hits": 0,
32 |       "columns": [
33 |         "email.source_ip",
34 |         "geoip.city_name",
35 |         "geoip.region_name",
36 |         "geoip.country_name"
37 |       ],
38 |       "sort": [
39 |         "@timestamp",
40 |         "desc"
41 |       ],
42 |       "version": 1,
43 |       "kibanaSavedObjectMeta": {
44 |         "searchSourceJSON": "{\"index\":\"dmarcxml-*\",\"highlightAll\":true,\"version\":true,\"query\":{\"language\":\"lucene\",\"query\":\"\"},\"filter\":[]}"
45 |       }
46 |     }
47 |   },
48 |   {
49 |     "_id": "7ff146f0-65a8-11e8-8e18-21bbcec4a3b8",
50 |     "_type": "search",
51 |     "_source": {
52 |       "title": "DMARC - SPF Results",
53 |       "description": "",
54 |       "hits": 0,
55 |       "columns": [
56 |         "email.source_ip",
57 |         "email.count",
58 |         "email.header_from",
59 |         "authresult.spf_domain",
60 |         "email.spf_evaluation",
61 |         "report.org"
62 |       ],
63 |       "sort": [
64 |         "report.start",
65 |         "desc"
66 |       ],
67 |       "version": 1,
68 |       "kibanaSavedObjectMeta": {
69 |         "searchSourceJSON": "{\"index\":\"dmarcxml-*\",\"highlightAll\":true,\"version\":true,\"query\":{\"query\":\"\",\"language\":\"lucene\"},\"filter\":[]}"
70 |       }
71 |     }
72 |   }
73 | ]


--------------------------------------------------------------------------------
/kibana/visuals/DMARCVisuals.json:
--------------------------------------------------------------------------------
  1 | [
  2 |   {
  3 |     "_id": "35ed5cc0-1444-11e8-8dc5-e141d701899b",
  4 |     "_type": "visualization",
  5 |     "_source": {
  6 |       "title": "DMARC - Coordinate Map - MetroArea - CustomMap - Heatmap",
  7 |       "visState": "{\"title\":\"DMARC - Coordinate Map - MetroArea - CustomMap - Heatmap\",\"type\":\"tile_map\",\"params\":{\"mapType\":\"Heatmap\",\"isDesaturated\":true,\"addTooltip\":true,\"heatClusterSize\":1.5,\"legendPosition\":\"bottomright\",\"mapZoom\":2,\"mapCenter\":[0,0],\"wms\":{\"enabled\":true,\"options\":{\"format\":\"image/png\",\"transparent\":true,\"layers\":\"TOPO-OSM-WMS\",\"version\":\"1.3.0\",\"attribution\":\"Mundialis Mapping Service\"},\"url\":\"http://ows.mundialis.de/services/service?\"}},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{\"customLabel\":\"Emails\"}},{\"id\":\"2\",\"enabled\":true,\"type\":\"geohash_grid\",\"schema\":\"segment\",\"params\":{\"field\":\"geoip.location.coordinates\",\"autoPrecision\":true,\"isFilteredByCollar\":true,\"useGeocentroid\":true,\"precision\":7,\"customLabel\":\"Coordinates\"}}]}",
  8 |       "uiStateJSON": "{\"mapZoom\":15,\"mapCenter\":[38.62842946020773,-90.1922607421875]}",
  9 |       "description": "",
 10 |       "version": 1,
 11 |       "kibanaSavedObjectMeta": {
 12 |         "searchSourceJSON": "{\"index\":\"dmarcxml-*\",\"filter\":[],\"query\":{\"query\":\"\",\"language\":\"lucene\"}}"
 13 |       }
 14 |     }
 15 |   },
 16 |   {
 17 |     "_id": "4723d300-1441-11e8-8dc5-e141d701899b",
 18 |     "_type": "visualization",
 19 |     "_source": {
 20 |       "title": "DMARC - Coordinate Map - US - DefaultMap - HeatMap",
 21 |       "visState": "{\"title\":\"DMARC - Coordinate Map - US - DefaultMap - HeatMap\",\"type\":\"tile_map\",\"params\":{\"mapType\":\"Heatmap\",\"isDesaturated\":true,\"addTooltip\":true,\"heatClusterSize\":1.5,\"legendPosition\":\"bottomright\",\"mapZoom\":2,\"mapCenter\":[0,0],\"wms\":{\"enabled\":false,\"options\":{\"format\":\"image/png\",\"transparent\":true}}},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"enabled\":true,\"type\":\"geohash_grid\",\"schema\":\"segment\",\"params\":{\"field\":\"geoip.location.coordinates\",\"autoPrecision\":false,\"isFilteredByCollar\":false,\"useGeocentroid\":true,\"precision\":3}}]}",
 22 |       "uiStateJSON": "{\"mapZoom\":5,\"mapCenter\":[35.44277092585766,-92.373046875]}",
 23 |       "description": "",
 24 |       "version": 1,
 25 |       "kibanaSavedObjectMeta": {
 26 |         "searchSourceJSON": "{\"index\":\"dmarcxml-*\",\"filter\":[],\"query\":{\"query\":\"\",\"language\":\"lucene\"}}"
 27 |       }
 28 |     }
 29 |   },
 30 |   {
 31 |     "_id": "a1fc8b00-1f4f-11e8-9724-59524d75d35b",
 32 |     "_type": "visualization",
 33 |     "_source": {
 34 |       "title": "DMARC - Top Senders (City)",
 35 |       "visState": "{\"title\":\"DMARC - Top Senders (City)\",\"type\":\"horizontal_bar\",\"params\":{\"type\":\"histogram\",\"grid\":{\"categoryLines\":false,\"style\":{\"color\":\"#eee\"},\"valueAxis\":\"ValueAxis-1\"},\"categoryAxes\":[{\"id\":\"CategoryAxis-1\",\"type\":\"category\",\"position\":\"left\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\"},\"labels\":{\"show\":true,\"rotate\":0,\"filter\":false,\"truncate\":200},\"title\":{}}],\"valueAxes\":[{\"id\":\"ValueAxis-1\",\"name\":\"LeftAxis-1\",\"type\":\"value\",\"position\":\"bottom\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\",\"mode\":\"normal\"},\"labels\":{\"show\":true,\"rotate\":75,\"filter\":true,\"truncate\":100},\"title\":{\"text\":\"Emails\"}}],\"seriesParams\":[{\"show\":true,\"type\":\"histogram\",\"mode\":\"normal\",\"data\":{\"label\":\"Emails\",\"id\":\"1\"},\"valueAxis\":\"ValueAxis-1\",\"drawLinesBetweenPoints\":true,\"showCircles\":true}],\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"times\":[],\"addTimeMarker\":false,\"orderBucketsBySum\":true},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{\"customLabel\":\"Emails\"}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"geoip.city_name.keyword\",\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\",\"customLabel\":\"City\"}}]}",
 36 |       "uiStateJSON": "{}",
 37 |       "description": "",
 38 |       "version": 1,
 39 |       "kibanaSavedObjectMeta": {
 40 |         "searchSourceJSON": "{\"index\":\"dmarcxml-*\",\"filter\":[],\"query\":{\"query\":\"\",\"language\":\"lucene\"}}"
 41 |       }
 42 |     }
 43 |   },
 44 |   {
 45 |     "_id": "7d87ac50-0fb9-11e8-8dc5-e141d701899b",
 46 |     "_type": "visualization",
 47 |     "_source": {
 48 |       "title": "DMARC - Region Map - US - DefaultMap",
 49 |       "visState": "{\"title\":\"DMARC - Region Map - US - DefaultMap\",\"type\":\"region_map\",\"params\":{\"legendPosition\":\"bottomright\",\"addTooltip\":true,\"colorSchema\":\"Reds\",\"selectedLayer\":{\"attribution\":\"<p><a href=\\\"http://www.naturalearthdata.com/about/terms-of-use\\\">Made with NaturalEarth</a> | <a href=\\\"https://www.elastic.co/elastic-maps-service\\\">Elastic Maps Service</a></p>&#10;\",\"name\":\"US States\",\"format\":\"geojson\",\"url\":\"https://layers.geo.elastic.co/blob/5086441721823232?elastic_tile_service_tos=agree&my_app_version=6.1.1\",\"fields\":[{\"name\":\"postal\",\"description\":\"Two letter abbreviation\"},{\"name\":\"name\",\"description\":\"State name\"}],\"created_at\":\"2017-04-26T19:45:22.377820\",\"tags\":[],\"id\":5086441721823232,\"layerId\":\"elastic_maps_service.US States\"},\"selectedJoinField\":{\"name\":\"name\",\"description\":\"State name\"},\"isDisplayWarning\":false,\"wms\":{\"enabled\":false,\"options\":{\"format\":\"image/png\",\"transparent\":true,\"layers\":\"TOPO-OSM-WMS\",\"version\":\"1.3.0\",\"attribution\":\"Mundialis Mapping Service\"},\"url\":\"http://ows.mundialis.de/services/service?\"},\"mapZoom\":2,\"mapCenter\":[0,0]},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{\"customLabel\":\"Messages Sent\"}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"geoip.region_name.keyword\",\"size\":20000,\"order\":\"desc\",\"orderBy\":\"1\",\"customLabel\":\"State\"}}]}",
 50 |       "uiStateJSON": "{\"mapZoom\":5,\"mapCenter\":[37.80544394934274,-94.39453125000001]}",
 51 |       "description": "",
 52 |       "version": 1,
 53 |       "kibanaSavedObjectMeta": {
 54 |         "searchSourceJSON": "{\"index\":\"dmarcxml-*\",\"filter\":[],\"query\":{\"query\":\"\",\"language\":\"lucene\"}}"
 55 |       }
 56 |     }
 57 |   },
 58 |   {
 59 |     "_id": "d5fb9fa0-1440-11e8-8dc5-e141d701899b",
 60 |     "_type": "visualization",
 61 |     "_source": {
 62 |       "title": "DMARC - Region Map - US - CustomMap",
 63 |       "visState": "{\"title\":\"DMARC - Region Map - US - CustomMap\",\"type\":\"region_map\",\"params\":{\"legendPosition\":\"bottomright\",\"addTooltip\":true,\"colorSchema\":\"Reds\",\"selectedLayer\":{\"attribution\":\"<p><a href=\\\"http://www.naturalearthdata.com/about/terms-of-use\\\">Made with NaturalEarth</a> | <a href=\\\"https://www.elastic.co/elastic-maps-service\\\">Elastic Maps Service</a></p>&#10;\",\"name\":\"US States\",\"format\":\"geojson\",\"url\":\"https://layers.geo.elastic.co/blob/5086441721823232?elastic_tile_service_tos=agree&my_app_version=6.1.1\",\"fields\":[{\"name\":\"postal\",\"description\":\"Two letter abbreviation\"},{\"name\":\"name\",\"description\":\"State name\"}],\"created_at\":\"2017-04-26T19:45:22.377820\",\"tags\":[],\"id\":5086441721823232,\"layerId\":\"elastic_maps_service.US States\"},\"selectedJoinField\":{\"name\":\"name\",\"description\":\"State name\"},\"isDisplayWarning\":false,\"wms\":{\"enabled\":true,\"options\":{\"format\":\"image/png\",\"transparent\":true,\"layers\":\"TOPO-OSM-WMS\",\"version\":\"1.3.0\",\"attribution\":\"Mundialis Mapping Service\"},\"url\":\"http://ows.mundialis.de/services/service?\"},\"mapZoom\":2,\"mapCenter\":[0,0]},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{\"customLabel\":\"Messages Sent\"}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"geoip.region_name.keyword\",\"size\":20000,\"order\":\"desc\",\"orderBy\":\"1\",\"customLabel\":\"State\"}}]}",
 64 |       "uiStateJSON": "{\"mapZoom\":5,\"mapCenter\":[37.80544394934274,-94.39453125000001]}",
 65 |       "description": "",
 66 |       "version": 1,
 67 |       "kibanaSavedObjectMeta": {
 68 |         "searchSourceJSON": "{\"index\":\"dmarcxml-*\",\"filter\":[],\"query\":{\"query\":\"\",\"language\":\"lucene\"}}"
 69 |       }
 70 |     }
 71 |   },
 72 |   {
 73 |     "_id": "b10896c0-1441-11e8-8dc5-e141d701899b",
 74 |     "_type": "visualization",
 75 |     "_source": {
 76 |       "title": "DMARC - Coordinate Map - MetroArea - DefaultMap - ShadedCircle",
 77 |       "visState": "{\"title\":\"DMARC - Coordinate Map - MetroArea - DefaultMap - ShadedCircle\",\"type\":\"tile_map\",\"params\":{\"mapType\":\"Shaded Circle Markers\",\"isDesaturated\":true,\"addTooltip\":true,\"heatClusterSize\":1.6,\"legendPosition\":\"bottomright\",\"mapZoom\":2,\"mapCenter\":[0,0],\"wms\":{\"enabled\":false,\"options\":{\"format\":\"image/png\",\"transparent\":true}}},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"enabled\":true,\"type\":\"geohash_grid\",\"schema\":\"segment\",\"params\":{\"field\":\"geoip.location.coordinates\",\"autoPrecision\":true,\"isFilteredByCollar\":true,\"useGeocentroid\":true,\"precision\":5}}]}",
 78 |       "uiStateJSON": "{\"mapZoom\":10,\"mapCenter\":[38.58520989992525,-90.06591796875]}",
 79 |       "description": "",
 80 |       "version": 1,
 81 |       "kibanaSavedObjectMeta": {
 82 |         "searchSourceJSON": "{\"index\":\"dmarcxml-*\",\"filter\":[],\"query\":{\"query\":\"\",\"language\":\"lucene\"}}"
 83 |       }
 84 |     }
 85 |   },
 86 |   {
 87 |     "_id": "e28b8980-1443-11e8-8dc5-e141d701899b",
 88 |     "_type": "visualization",
 89 |     "_source": {
 90 |       "title": "DMARC - Coordinate Map - MetroArea - CustomMap - ShadedCircle",
 91 |       "visState": "{\"title\":\"DMARC - Coordinate Map - MetroArea - CustomMap - ShadedCircle\",\"type\":\"tile_map\",\"params\":{\"mapType\":\"Shaded Circle Markers\",\"isDesaturated\":true,\"addTooltip\":true,\"heatClusterSize\":1.5,\"legendPosition\":\"bottomright\",\"mapZoom\":2,\"mapCenter\":[0,0],\"wms\":{\"enabled\":true,\"options\":{\"format\":\"image/png\",\"transparent\":true,\"layers\":\"TOPO-OSM-WMS\",\"version\":\"1.3.0\",\"attribution\":\"Mundialis Mapping Service\"},\"url\":\"http://ows.mundialis.de/services/service?\"}},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{\"customLabel\":\"Emails\"}},{\"id\":\"2\",\"enabled\":true,\"type\":\"geohash_grid\",\"schema\":\"segment\",\"params\":{\"field\":\"geoip.location.coordinates\",\"autoPrecision\":true,\"isFilteredByCollar\":true,\"useGeocentroid\":true,\"precision\":7,\"customLabel\":\"Coordinates\"}}]}",
 92 |       "uiStateJSON": "{\"mapZoom\":15,\"mapCenter\":[38.62842946020773,-90.1922607421875]}",
 93 |       "description": "",
 94 |       "version": 1,
 95 |       "kibanaSavedObjectMeta": {
 96 |         "searchSourceJSON": "{\"index\":\"dmarcxml-*\",\"filter\":[],\"query\":{\"query\":\"\",\"language\":\"lucene\"}}"
 97 |       }
 98 |     }
 99 |   },
100 |   {
101 |     "_id": "b8693ba0-0fbd-11e8-8dc5-e141d701899b",
102 |     "_type": "visualization",
103 |     "_source": {
104 |       "title": "DMARC - Top Reporting Orgs",
105 |       "visState": "{\"title\":\"DMARC - Top Reporting Orgs\",\"type\":\"pie\",\"params\":{\"type\":\"pie\",\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"isDonut\":true,\"labels\":{\"show\":true,\"values\":false,\"last_level\":true,\"truncate\":100}},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"report.org.keyword\",\"size\":7,\"order\":\"desc\",\"orderBy\":\"1\",\"customLabel\":\"Reporting Organization\"}}]}",
106 |       "uiStateJSON": "{\"vis\":{\"legendOpen\":false}}",
107 |       "description": "",
108 |       "version": 1,
109 |       "kibanaSavedObjectMeta": {
110 |         "searchSourceJSON": "{\"index\":\"dmarcxml-*\",\"filter\":[],\"query\":{\"query\":\"\",\"language\":\"lucene\"}}"
111 |       }
112 |     }
113 |   },
114 |   {
115 |     "_id": "b3851180-1f4f-11e8-9724-59524d75d35b",
116 |     "_type": "visualization",
117 |     "_source": {
118 |       "title": "DMARC - Top Senders (Region)",
119 |       "visState": "{\"title\":\"DMARC - Top Senders (Region)\",\"type\":\"horizontal_bar\",\"params\":{\"type\":\"histogram\",\"grid\":{\"categoryLines\":false,\"style\":{\"color\":\"#eee\"},\"valueAxis\":\"ValueAxis-1\"},\"categoryAxes\":[{\"id\":\"CategoryAxis-1\",\"type\":\"category\",\"position\":\"left\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\"},\"labels\":{\"show\":true,\"rotate\":0,\"filter\":false,\"truncate\":200},\"title\":{}}],\"valueAxes\":[{\"id\":\"ValueAxis-1\",\"name\":\"LeftAxis-1\",\"type\":\"value\",\"position\":\"bottom\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\",\"mode\":\"normal\"},\"labels\":{\"show\":true,\"rotate\":75,\"filter\":true,\"truncate\":100},\"title\":{\"text\":\"Emails\"}}],\"seriesParams\":[{\"show\":true,\"type\":\"histogram\",\"mode\":\"normal\",\"data\":{\"label\":\"Emails\",\"id\":\"1\"},\"valueAxis\":\"ValueAxis-1\",\"drawLinesBetweenPoints\":true,\"showCircles\":true}],\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"times\":[],\"addTimeMarker\":false,\"orderBucketsBySum\":true},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{\"customLabel\":\"Emails\"}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"geoip.region_name.keyword\",\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\",\"customLabel\":\"City\"}}]}",
120 |       "uiStateJSON": "{}",
121 |       "description": "",
122 |       "version": 1,
123 |       "kibanaSavedObjectMeta": {
124 |         "searchSourceJSON": "{\"index\":\"dmarcxml-*\",\"filter\":[],\"query\":{\"query\":\"\",\"language\":\"lucene\"}}"
125 |       }
126 |     }
127 |   },
128 |   {
129 |     "_id": "bc238b40-2608-11e8-b656-99c73f50969b",
130 |     "_type": "visualization",
131 |     "_source": {
132 |       "title": "DMARC - Ingested Reports",
133 |       "visState": "{\"title\":\"DMARC - Ingested Reports\",\"type\":\"metric\",\"params\":{\"addTooltip\":true,\"addLegend\":false,\"type\":\"metric\",\"metric\":{\"percentageMode\":false,\"useRanges\":false,\"colorSchema\":\"Green to Red\",\"metricColorMode\":\"None\",\"colorsRange\":[{\"from\":0,\"to\":10000}],\"labels\":{\"show\":true},\"invertColors\":false,\"style\":{\"bgFill\":\"#000\",\"bgColor\":false,\"labelColor\":false,\"subText\":\"\",\"fontSize\":50}}},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"cardinality\",\"schema\":\"metric\",\"params\":{\"field\":\"path.keyword\",\"customLabel\":\"Ingested Reports\"}}]}",
134 |       "uiStateJSON": "{}",
135 |       "description": "",
136 |       "version": 1,
137 |       "kibanaSavedObjectMeta": {
138 |         "searchSourceJSON": "{\"index\":\"dmarcxml-*\",\"filter\":[],\"query\":{\"query\":\"\",\"language\":\"lucene\"}}"
139 |       }
140 |     }
141 |   },
142 |   {
143 |     "_id": "55fe84b0-0fb9-11e8-8dc5-e141d701899b",
144 |     "_type": "visualization",
145 |     "_source": {
146 |       "title": "DMARC - Country Heatmap",
147 |       "visState": "{\"title\":\"DMARC - Country Heatmap\",\"type\":\"region_map\",\"params\":{\"legendPosition\":\"bottomright\",\"addTooltip\":true,\"colorSchema\":\"Reds\",\"selectedLayer\":{\"attribution\":\"<p><a href=\\\"http://www.naturalearthdata.com/about/terms-of-use\\\">Made with NaturalEarth</a> | <a href=\\\"https://www.elastic.co/elastic-maps-service\\\">Elastic Maps Service</a></p>&#10;\",\"name\":\"World Countries\",\"format\":\"geojson\",\"url\":\"https://layers.geo.elastic.co/blob/5659313586569216?elastic_tile_service_tos=agree&my_app_version=6.1.1\",\"fields\":[{\"name\":\"iso2\",\"description\":\"Two letter abbreviation\"},{\"name\":\"name\",\"description\":\"Country name\"},{\"name\":\"iso3\",\"description\":\"Three letter abbreviation\"}],\"created_at\":\"2017-04-26T17:12:15.978370\",\"tags\":[],\"id\":5659313586569216,\"layerId\":\"elastic_maps_service.World Countries\"},\"selectedJoinField\":{\"name\":\"iso2\",\"description\":\"Two letter abbreviation\"},\"isDisplayWarning\":false,\"wms\":{\"enabled\":false,\"options\":{\"format\":\"image/png\",\"transparent\":true,\"layers\":\"TOPO-OSM-WMS\",\"version\":\"1.3.0\",\"attribution\":\"Mundialis Mapping Service\"},\"url\":\"http://ows.mundialis.de/services/service?\",\"baseLayersAreLoaded\":{},\"tmsLayers\":[{\"id\":\"road_map\",\"url\":\"https://tiles.maps.elastic.co/v2/default/{z}/{x}/{y}.png?elastic_tile_service_tos=agree&my_app_name=kibana&my_app_version=6.2.2&license=a6b33275-612d-4eee-adc9-260c3f630677\",\"minZoom\":0,\"maxZoom\":18,\"attribution\":\"<p>&#169; <a href=\\\"http://www.openstreetmap.org/copyright\\\">OpenStreetMap</a> contributors | <a href=\\\"https://www.elastic.co/elastic-maps-service\\\">Elastic Maps Service</a></p>&#10;\",\"subdomains\":[]}],\"selectedTmsLayer\":{\"id\":\"road_map\",\"url\":\"https://tiles.maps.elastic.co/v2/default/{z}/{x}/{y}.png?elastic_tile_service_tos=agree&my_app_name=kibana&my_app_version=6.2.2&license=a6b33275-612d-4eee-adc9-260c3f630677\",\"minZoom\":0,\"maxZoom\":18,\"attribution\":\"<p>&#169; <a href=\\\"http://www.openstreetmap.org/copyright\\\">OpenStreetMap</a> contributors | <a href=\\\"https://www.elastic.co/elastic-maps-service\\\">Elastic Maps Service</a></p>&#10;\",\"subdomains\":[]}},\"mapZoom\":2,\"mapCenter\":[0,0],\"outlineWeight\":1,\"showAllShapes\":true},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"sum\",\"schema\":\"metric\",\"params\":{\"field\":\"email.count\",\"customLabel\":\"Messages Sent\"}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"geoip.country_code2.keyword\",\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"size\":100000,\"order\":\"desc\",\"orderBy\":\"1\",\"customLabel\":\"Country\"}}]}",
148 |       "uiStateJSON": "{\"mapZoom\":3,\"mapCenter\":[42.42345651793833,-0.17578125]}",
149 |       "description": "",
150 |       "version": 1,
151 |       "kibanaSavedObjectMeta": {
152 |         "searchSourceJSON": "{\"index\":\"dmarcxml-*\",\"filter\":[],\"query\":{\"query\":\"\",\"language\":\"lucene\"}}"
153 |       }
154 |     }
155 |   },
156 |   {
157 |     "_id": "2a1cf250-1441-11e8-8dc5-e141d701899b",
158 |     "_type": "visualization",
159 |     "_source": {
160 |       "title": "DMARC - Coordinate Map - US - DefaultMap - ScaledCircles",
161 |       "visState": "{\"title\":\"DMARC - Coordinate Map - US - DefaultMap - ScaledCircles\",\"type\":\"tile_map\",\"params\":{\"mapType\":\"Scaled Circle Markers\",\"isDesaturated\":true,\"addTooltip\":true,\"heatClusterSize\":1.5,\"legendPosition\":\"bottomleft\",\"mapZoom\":2,\"mapCenter\":[0,0],\"wms\":{\"enabled\":false,\"options\":{\"format\":\"image/png\",\"transparent\":true,\"layers\":\"TOPO-OSM-WMS\",\"version\":\"1.3.0\",\"attribution\":\"Mundialis Mapping Service\"},\"baseLayersAreLoaded\":{},\"tmsLayers\":[{\"id\":\"road_map\",\"url\":\"https://tiles.maps.elastic.co/v2/default/{z}/{x}/{y}.png?elastic_tile_service_tos=agree&my_app_name=kibana&my_app_version=6.2.2&license=7dafc3ad-6752-498d-bb49-42725ba12530\",\"minZoom\":0,\"maxZoom\":18,\"attribution\":\"<p>&#169; <a href=\\\"http://www.openstreetmap.org/copyright\\\">OpenStreetMap</a> contributors | <a href=\\\"https://www.elastic.co/elastic-maps-service\\\">Elastic Maps Service</a></p>&#10;\",\"subdomains\":[]}],\"selectedTmsLayer\":{\"id\":\"road_map\",\"url\":\"https://tiles.maps.elastic.co/v2/default/{z}/{x}/{y}.png?elastic_tile_service_tos=agree&my_app_name=kibana&my_app_version=6.2.2&license=7dafc3ad-6752-498d-bb49-42725ba12530\",\"minZoom\":0,\"maxZoom\":18,\"attribution\":\"<p>&#169; <a href=\\\"http://www.openstreetmap.org/copyright\\\">OpenStreetMap</a> contributors | <a href=\\\"https://www.elastic.co/elastic-maps-service\\\">Elastic Maps Service</a></p>&#10;\",\"subdomains\":[]},\"url\":\"http://ows.mundialis.de/services/service?\"}},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"enabled\":true,\"type\":\"geohash_grid\",\"schema\":\"segment\",\"params\":{\"field\":\"geoip.location.coordinates\",\"autoPrecision\":true,\"isFilteredByCollar\":true,\"useGeocentroid\":true,\"precision\":3}}]}",
162 |       "uiStateJSON": "{\"mapZoom\":5,\"mapCenter\":[37.75334401310659,-90.87890625000001]}",
163 |       "description": "",
164 |       "version": 1,
165 |       "kibanaSavedObjectMeta": {
166 |         "searchSourceJSON": "{\"index\":\"dmarcxml-*\",\"filter\":[],\"query\":{\"query\":\"\",\"language\":\"lucene\"}}"
167 |       }
168 |     }
169 |   },
170 |   {
171 |     "_id": "9627e210-0fba-11e8-8dc5-e141d701899b",
172 |     "_type": "visualization",
173 |     "_source": {
174 |       "title": "DMARC - SPF Pass Count",
175 |       "visState": "{\"title\":\"DMARC - SPF Pass Count\",\"type\":\"table\",\"params\":{\"perPage\":5,\"showPartialRows\":false,\"showMeticsAtAllLevels\":false,\"sort\":{\"columnIndex\":null,\"direction\":null},\"showTotal\":true,\"totalFunc\":\"sum\"},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"sum\",\"schema\":\"metric\",\"params\":{\"field\":\"email.count\",\"customLabel\":\"Count\"}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"bucket\",\"params\":{\"field\":\"email.source_ip\",\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"exclude\":\"\",\"size\":100000,\"order\":\"desc\",\"orderBy\":\"1\",\"customLabel\":\"IP Address\"}}]}",
176 |       "uiStateJSON": "{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":null,\"direction\":null}}},\"spy\":null}",
177 |       "description": "",
178 |       "version": 1,
179 |       "kibanaSavedObjectMeta": {
180 |         "searchSourceJSON": "{\"index\":\"dmarcxml-*\",\"filter\":[{\"meta\":{\"index\":\"dmarcxml-*\",\"type\":\"phrases\",\"key\":\"email.spf_evaluation\",\"value\":\"pass, Pass\",\"params\":[\"pass\",\"Pass\"],\"negate\":false,\"disabled\":false,\"alias\":null},\"query\":{\"bool\":{\"should\":[{\"match_phrase\":{\"email.spf_evaluation\":\"pass\"}},{\"match_phrase\":{\"email.spf_evaluation\":\"Pass\"}}],\"minimum_should_match\":1}},\"$state\":{\"store\":\"appState\"}}],\"query\":{\"query\":\"\",\"language\":\"lucene\"}}"
181 |       }
182 |     }
183 |   },
184 |   {
185 |     "_id": "7f45e1c0-59e1-11e8-8e48-d7c271d057a2",
186 |     "_type": "visualization",
187 |     "_source": {
188 |       "title": "DMARC - Top Senders (IP)",
189 |       "visState": "{\"title\":\"DMARC - Top Senders (IP)\",\"type\":\"table\",\"params\":{\"perPage\":11,\"showPartialRows\":false,\"showMeticsAtAllLevels\":false,\"sort\":{\"columnIndex\":null,\"direction\":null},\"showTotal\":true,\"totalFunc\":\"sum\"},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"sum\",\"schema\":\"metric\",\"params\":{\"field\":\"email.count\",\"customLabel\":\"Count\"}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"bucket\",\"params\":{\"field\":\"email.source_ip\",\"otherBucket\":true,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"exclude\":\"\",\"size\":10,\"order\":\"desc\",\"orderBy\":\"1\",\"customLabel\":\"IP Address\"}}]}",
190 |       "uiStateJSON": "{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":null,\"direction\":null}}},\"spy\":null}",
191 |       "description": "",
192 |       "version": 1,
193 |       "kibanaSavedObjectMeta": {
194 |         "searchSourceJSON": "{\"index\":\"dmarcxml-*\",\"filter\":[],\"query\":{\"query\":\"\",\"language\":\"lucene\"}}"
195 |       }
196 |     }
197 |   },
198 |   {
199 |     "_id": "94996f50-0fb9-11e8-8dc5-e141d701899b",
200 |     "_type": "visualization",
201 |     "_source": {
202 |       "title": "DMARC - DKIM Result",
203 |       "visState": "{\"title\":\"DMARC - DKIM Result\",\"type\":\"pie\",\"params\":{\"type\":\"pie\",\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"isDonut\":true,\"labels\":{\"show\":true,\"values\":true,\"last_level\":true,\"truncate\":100}},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"sum\",\"schema\":\"metric\",\"params\":{\"field\":\"email.count\"}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"email.dkim_evaluation.keyword\",\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\",\"customLabel\":\"DKIM Result\"}}]}",
204 |       "uiStateJSON": "{\"vis\":{\"legendOpen\":false}}",
205 |       "description": "",
206 |       "version": 1,
207 |       "kibanaSavedObjectMeta": {
208 |         "searchSourceJSON": "{\"index\":\"dmarcxml-*\",\"filter\":[],\"query\":{\"query\":\"\",\"language\":\"lucene\"}}"
209 |       }
210 |     }
211 |   },
212 |   {
213 |     "_id": "b7466350-0fb9-11e8-8dc5-e141d701899b",
214 |     "_type": "visualization",
215 |     "_source": {
216 |       "title": "DMARC - SPF Result",
217 |       "visState": "{\"title\":\"DMARC - SPF Result\",\"type\":\"pie\",\"params\":{\"type\":\"pie\",\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"isDonut\":false,\"labels\":{\"show\":true,\"values\":true,\"last_level\":true,\"truncate\":100}},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"sum\",\"schema\":\"metric\",\"params\":{\"field\":\"email.count\"}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"email.spf_evaluation.keyword\",\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\",\"customLabel\":\"SPF Result\"}}]}",
218 |       "uiStateJSON": "{\"vis\":{\"legendOpen\":false}}",
219 |       "description": "",
220 |       "version": 1,
221 |       "kibanaSavedObjectMeta": {
222 |         "searchSourceJSON": "{\"index\":\"dmarcxml-*\",\"filter\":[],\"query\":{\"query\":\"\",\"language\":\"lucene\"}}"
223 |       }
224 |     }
225 |   },
226 |   {
227 |     "_id": "05ebb890-0fbd-11e8-8dc5-e141d701899b",
228 |     "_type": "visualization",
229 |     "_source": {
230 |       "title": "DMARC - Email Count",
231 |       "visState": "{\"title\":\"DMARC - Email Count\",\"type\":\"metric\",\"params\":{\"addTooltip\":true,\"addLegend\":false,\"type\":\"metric\",\"metric\":{\"percentageMode\":false,\"useRanges\":false,\"colorSchema\":\"Green to Red\",\"metricColorMode\":\"None\",\"colorsRange\":[{\"from\":0,\"to\":10000000}],\"labels\":{\"show\":true},\"invertColors\":false,\"style\":{\"bgFill\":\"#000\",\"bgColor\":false,\"labelColor\":false,\"subText\":\"\",\"fontSize\":50}}},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"sum\",\"schema\":\"metric\",\"params\":{\"field\":\"email.count\",\"customLabel\":\"Emails Sent\"}}]}",
232 |       "uiStateJSON": "{}",
233 |       "description": "",
234 |       "version": 1,
235 |       "kibanaSavedObjectMeta": {
236 |         "searchSourceJSON": "{\"index\":\"dmarcxml-*\",\"filter\":[],\"query\":{\"query\":\"\",\"language\":\"lucene\"}}"
237 |       }
238 |     }
239 |   },
240 |   {
241 |     "_id": "17d5b590-1f4b-11e8-9724-59524d75d35b",
242 |     "_type": "visualization",
243 |     "_source": {
244 |       "title": "DMARC - Top Senders (Country)",
245 |       "visState": "{\"title\":\"DMARC - Top Senders (Country)\",\"type\":\"horizontal_bar\",\"params\":{\"type\":\"histogram\",\"grid\":{\"categoryLines\":false,\"style\":{\"color\":\"#eee\"},\"valueAxis\":\"ValueAxis-1\"},\"categoryAxes\":[{\"id\":\"CategoryAxis-1\",\"type\":\"category\",\"position\":\"left\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\"},\"labels\":{\"show\":true,\"rotate\":0,\"filter\":false,\"truncate\":100},\"title\":{}}],\"valueAxes\":[{\"id\":\"ValueAxis-1\",\"name\":\"BottomAxis-1\",\"type\":\"value\",\"position\":\"bottom\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\",\"mode\":\"normal\",\"defaultYExtents\":false,\"setYExtents\":false},\"labels\":{\"show\":true,\"rotate\":75,\"filter\":false,\"truncate\":100},\"title\":{\"text\":\"Number of Emails\"}}],\"seriesParams\":[{\"show\":true,\"type\":\"histogram\",\"mode\":\"normal\",\"data\":{\"label\":\"Number of Emails\",\"id\":\"1\"},\"valueAxis\":\"ValueAxis-1\",\"drawLinesBetweenPoints\":true,\"showCircles\":true}],\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"times\":[],\"addTimeMarker\":false,\"radiusRatio\":51,\"orderBucketsBySum\":true},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"sum\",\"schema\":\"metric\",\"params\":{\"field\":\"email.count\",\"customLabel\":\"Number of Emails\"}},{\"id\":\"3\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"geoip.country_name.keyword\",\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\",\"customLabel\":\"Country\"}}]}",
246 |       "uiStateJSON": "{\"vis\":{\"legendOpen\":false},\"spy\":null}",
247 |       "description": "",
248 |       "version": 1,
249 |       "kibanaSavedObjectMeta": {
250 |         "searchSourceJSON": "{\"index\":\"dmarcxml-*\",\"filter\":[],\"query\":{\"query\":\"\",\"language\":\"lucene\"}}"
251 |       }
252 |     }
253 |   },
254 |   {
255 |     "_id": "71737e10-1f4c-11e8-9724-59524d75d35b",
256 |     "_type": "visualization",
257 |     "_source": {
258 |       "title": "DMARC - Top Senders (Domain)",
259 |       "visState": "{\"title\":\"DMARC - Top Senders (Domain)\",\"type\":\"horizontal_bar\",\"params\":{\"type\":\"histogram\",\"grid\":{\"categoryLines\":false,\"style\":{\"color\":\"#eee\"},\"valueAxis\":\"ValueAxis-1\"},\"categoryAxes\":[{\"id\":\"CategoryAxis-1\",\"type\":\"category\",\"position\":\"left\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\"},\"labels\":{\"show\":true,\"rotate\":0,\"filter\":false,\"truncate\":200},\"title\":{}}],\"valueAxes\":[{\"id\":\"ValueAxis-1\",\"name\":\"LeftAxis-1\",\"type\":\"value\",\"position\":\"bottom\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\",\"mode\":\"normal\"},\"labels\":{\"show\":true,\"rotate\":75,\"filter\":true,\"truncate\":100},\"title\":{\"text\":\"Number of Emails\"}}],\"seriesParams\":[{\"show\":true,\"type\":\"histogram\",\"mode\":\"normal\",\"data\":{\"label\":\"Number of Emails\",\"id\":\"1\"},\"valueAxis\":\"ValueAxis-1\",\"drawLinesBetweenPoints\":true,\"showCircles\":true}],\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"times\":[],\"addTimeMarker\":false,\"orderBucketsBySum\":true},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"sum\",\"schema\":\"metric\",\"params\":{\"field\":\"email.count\",\"customLabel\":\"Number of Emails\"}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"authresult.spf_domain.keyword\",\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"exclude\":\"\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\",\"customLabel\":\"Sender's Domain\"}}]}",
260 |       "uiStateJSON": "{\"vis\":{\"legendOpen\":false}}",
261 |       "description": "",
262 |       "version": 1,
263 |       "kibanaSavedObjectMeta": {
264 |         "searchSourceJSON": "{\"index\":\"dmarcxml-*\",\"filter\":[],\"query\":{\"query\":\"\",\"language\":\"lucene\"}}"
265 |       }
266 |     }
267 |   }
268 | ]


--------------------------------------------------------------------------------
/logstash/bin/DMARCNMA.ps1:
--------------------------------------------------------------------------------
  1 | #DMARC No Mailbox Access Script
  2 | #Use this script if you are pulling DMARC aggregate reports from a non-Microsoft Exchange hosted mailbox.
  3 | #This script will decompress .zip/.gz compressed archives, modify the XML structure, and save the
  4 | #resulting file.
  5 | 
  6 | $downloadDirectory = Read-Host "Folder path to downloaded attachments"
  7 | $outfile = Read-Host "Folder Path to save extracted files to"
  8 | $Ingest = Read-Host "Folder Path to save modified XML reports to"
  9 | $Cleanup = Read-Host "Cleanup attachments and unmodified reports? (y|n)"
 10 | Clear-Host
 11 | 
 12 | ###########Decompress Archives###########
 13 | Function DeGZip-File{
 14 |     Param(
 15 |         $infile,
 16 |         $outfile       
 17 |         )
 18 | 
 19 |     $input = New-Object System.IO.FileStream $inFile, ([IO.FileMode]::Open), ([IO.FileAccess]::Read), ([IO.FileShare]::Read)
 20 |     $output = New-Object System.IO.FileStream $outFile, ([IO.FileMode]::Create), ([IO.FileAccess]::Write), ([IO.FileShare]::None)
 21 |     $gzipStream = New-Object System.IO.Compression.GzipStream $input, ([IO.Compression.CompressionMode]::Decompress)
 22 | 
 23 |     $buffer = New-Object byte[](1024)
 24 |     while($true){
 25 |         $read = $gzipstream.Read($buffer, 0, 1024)
 26 |         if ($read -le 0){break}
 27 |         $output.Write($buffer, 0, $read)
 28 |         }
 29 | 
 30 |     $gzipStream.Close()
 31 |     $output.Close()
 32 |     $input.Close()
 33 | }
 34 | $degz = Get-ChildItem $downloadDirectory | Where-Object {$_.name -match ".*\.gz$"}
 35 | $dezip = Get-ChildItem $downloadDirectory | Where-Object {$_.name -match ".*\.zip$"}
 36 | #Moved to line 5
 37 | if ((Test-Path $outfile) -eq $false) {
 38 |   New-Item -ItemType Directory -Force -Path $outfile | Out-Null
 39 | }
 40 | foreach ($file in $degz) {
 41 |     $of = $file.name.Replace(".xml.gz","")
 42 |     DeGZip-File  $downloadDirectory\$file $outfile\$of".xml"
 43 |     $gz++
 44 |     Write-Progress -activity "Decompressing GZ files" -status "Decompressed: $gz of $($degz.Count)"
 45 |     }
 46 | foreach ($infile in $dezip) {
 47 |     Expand-Archive -Path $downloadDirectory\$infile -DestinationPath $outfile
 48 |     $gzi++
 49 |     Write-Progress -activity "Decompressing zip files" -status "Decompressed: $gzi of $($dezip.Count)"
 50 | }
 51 | 
 52 | 
 53 | ###########Modify XML Files###########
 54 | #Moved to line 6
 55 | if ((Test-Path $Ingest) -eq $false) {
 56 |   New-Item -ItemType Directory -Force -Path $Ingest | Out-Null
 57 | }
 58 | #XML Restructure
 59 | $XMLRepo = Get-Childitem $outfile -Recurse | Where-Object {$_.Name -match ".*\.xml$"}
 60 | foreach ($xmlfile in $xmlrepo) {
 61 |   [xml]$xml = Get-Content -Path $xmlfile.VersionInfo.FileName
 62 |   $xmlrecord = $xml.feedback.record
 63 |   foreach ($record in $xmlrecord) {
 64 |     $xmlreport = $xml.SelectSingleNode("//feedback/report_metadata").Clone()
 65 |     $xmlpolicy = $xml.SelectSingleNode("//feedback/policy_published").Clone()
 66 |     $record.AppendChild($xmlreport) | Out-Null
 67 |     $record.AppendChild($xmlpolicy) | Out-Null
 68 |   }
 69 |   $xmlpolicy = $xml.SelectSingleNode("//feedback/policy_published")
 70 |   $xmlreport = $xml.SelectSingleNode("//feedback/report_metadata")
 71 |   $xml.feedback.RemoveChild($xmlreport) | Out-Null
 72 |   $xml.feedback.RemoveChild($xmlpolicy) | Out-Null
 73 | #Save to ingest point
 74 |   $xml.Save("$Ingest\$xmlfile")
 75 |   $xmlprogress++
 76 |   Write-Progress -activity "Modifying XML Structures" -status "Modified: $xmlprogress of $($xmlrepo.Count)"
 77 | }
 78 | Clear-Host
 79 | Write-Host "Modification process complete!"
 80 | Start-Sleep -Seconds 3
 81 | 
 82 | ###########File Cleanup###########
 83 | #Moved to line 7
 84 | if ($cleanup -eq "y") {
 85 |     #Remove downloaded email attachments
 86 |     Get-ChildItem $downloadDirectory | Where-Object {$_.name -match ".*\.gz$|.*\.zip$"} | Remove-Item -Force -Confirm:$false
 87 |     $dlc = Get-ChildItem $downloadDirectory
 88 |     if ($dlc.count -lt 1) {
 89 |         Remove-Item $downloadDirectory
 90 |     }
 91 |     #Remove decompressed, unmodified XML files
 92 |     Get-ChildItem $outfile | Where-Object {$_.name -match ".*\.xml$"} | Remove-Item -Force -Confirm:$false
 93 |     $dcx = Get-ChildItem $outfile
 94 |     if ($dcx.count -lt 1) {
 95 |         Remove-Item $outfile
 96 |     }
 97 | }
 98 | if ($cleanup -eq "n") {
 99 | exit
100 | }
101 | #Remove modified XML files
102 | $Cleanup = Read-Host "Remove modified files? (y|n)"
103 | if ($cleanup -eq "y") {
104 |     Get-ChildItem $ingest | Where-Object {$_.name -match ".*\.xml$"} | Remove-Item -Force -Confirm:$false
105 |     $modx = Get-ChildItem $ingest
106 |     if ($modx.count -lt 1) {
107 |         Remove-Item $ingest
108 |     }
109 | }
110 | Clear-Host
111 | Write-Host "Cleanup process complete!"
112 | Start-Sleep -Seconds 3


--------------------------------------------------------------------------------
/logstash/bin/DmarcUnattended.ps1:
--------------------------------------------------------------------------------
  1 | #Set variables on lines 5,7,9,11,13,15 - retain double quotes.
  2 | #Ingested files are not deleted until 10 minutes after they've been written to disk.
  3 | 
  4 | #Email address of mailbox to connect to.
  5 | $MailboxName = "DMARC_RUA@example.com"
  6 | #FQDN of CAS server.
  7 | $urireq = "https://mail.example.com"
  8 | #Where to save attachments.
  9 | $downloadDirectory = "D:\Temp\Attachments"
 10 | #Where to save extracted XML files.
 11 | $outfile = "D:\Temp\Extracted"
 12 | #Where to save modified XML files.
 13 | $Ingest = "D:\Temp\DMARC"
 14 | #Where to move emails after report has been downloaded.
 15 | $FolderDest = "Review Completed"
 16 | 
 17 | ###########Download messages from mailbox###########
 18 | $Provider=New-Object Microsoft.CSharp.CSharpCodeProvider
 19 | $Compiler=$Provider.CreateCompiler()
 20 | $Params=New-Object System.CodeDom.Compiler.CompilerParameters
 21 | $Params.GenerateExecutable=$False
 22 | $Params.GenerateInMemory=$True
 23 | $Params.IncludeDebugInformation=$False
 24 | $Params.ReferencedAssemblies.Add("System.DLL") | Out-Null
 25 | $TASource=@'
 26 |     namespace Local.ToolkitExtensions.Net.CertificatePolicy{
 27 |         public class TrustAll : System.Net.ICertificatePolicy {
 28 |             public TrustAll() {
 29 |             }
 30 |             public bool CheckValidationResult(System.Net.ServicePoint sp,
 31 |             System.Security.Cryptography.X509Certificates.X509Certificate cert,
 32 |             System.Net.WebRequest req, int problem) {
 33 |                 return true;
 34 |             }
 35 |         }
 36 |     }
 37 | '@
 38 | $TAResults=$Provider.CompileAssemblyFromSource($Params,$TASource)
 39 | $TAAssembly=$TAResults.CompiledAssembly
 40 | $TrustAll=$TAAssembly.CreateInstance("Local.ToolkitExtensions.Net.CertificatePolicy.TrustAll")
 41 | [System.Net.ServicePointManager]::CertificatePolicy=$TrustAll
 42 | if ((Test-Path "C:\Program Files\Microsoft\Exchange\Web Services\2.2\Microsoft.Exchange.WebServices.dll") -eq $true) {
 43 |     $API22 = $true
 44 |     Add-Type -Path "C:\Program Files\Microsoft\Exchange\Web Services\2.2\Microsoft.Exchange.WebServices.dll"
 45 | }
 46 | if ($API22 -ne $true -and (Test-Path "C:\Program Files\Microsoft\Exchange\Web Services\1.2\Microsoft.Exchange.WebServices.dll") -eq $true) {
 47 |     $API12= $true
 48 |     Add-Type -Path "C:\Program Files\Microsoft\Exchange\Web Services\1.2\Microsoft.Exchange.WebServices.dll"
 49 | }
 50 | if ($API22 -ne $true -and $API12 -ne $true) {
 51 |     Exit
 52 | }
 53 | $ExchangeVersion = [Microsoft.Exchange.WebServices.Data.ExchangeVersion]::Exchange2013_SP1
 54 | $service = New-Object Microsoft.Exchange.WebServices.Data.ExchangeService($ExchangeVersion)
 55 | $uri = [System.URI] "$urireq/ews/Exchange.asmx"
 56 | $service.Url = $uri
 57 | $Sfha = new-object Microsoft.Exchange.WebServices.Data.SearchFilter+IsEqualTo([Microsoft.Exchange.WebServices.Data.EmailMessageSchema]::HasAttachments, $true)
 58 | $folderid= new-object Microsoft.Exchange.WebServices.Data.FolderId([Microsoft.Exchange.WebServices.Data.WellKnownFolderName]::Inbox,"$MailboxName")
 59 | $Inbox = [Microsoft.Exchange.WebServices.Data.Folder]::Bind($service,$folderid)
 60 | $ivItemView = New-Object Microsoft.Exchange.WebServices.Data.ItemView(200)
 61 | if ((Test-Path $downloadDirectory) -eq $false) {
 62 |     New-Item -ItemType Directory -Force -Path $downloadDirectory | Out-Null
 63 | }
 64 | $findItemsResults = $Inbox.FindItems($Sfha,$ivItemView)
 65 | foreach($miMailItems in $findItemsResults.Items){
 66 |     $miMailItems.Load()
 67 |     foreach($attach in $miMailItems.Attachments){
 68 |         $attach.Load()
 69 |         $fiFile = new-object System.IO.FileStream(($downloadDirectory + "\" + $attach.Name.ToString()), [System.IO.FileMode]::Create)
 70 |         $fiFile.Write($attach.Content, 0, $attach.Content.Length)
 71 |         $fiFile.Close()
 72 |     }
 73 | }
 74 | $fvFolderView =  New-Object Microsoft.Exchange.WebServices.Data.FolderView(100)
 75 | $fvFolderView.Traversal = [Microsoft.Exchange.WebServices.Data.FolderTraversal]::Shallow;
 76 | $SfSearchFilter = new-object Microsoft.Exchange.WebServices.Data.SearchFilter+IsEqualTo([Microsoft.Exchange.WebServices.Data.FolderSchema]::DisplayName,"$FolderDest")
 77 | $findFolderResults = $Inbox.FindFolders($SfSearchFilter,$fvFolderView)
 78 | $ivItemView =  New-Object Microsoft.Exchange.WebServices.Data.ItemView(200)
 79 | $fiItems = $null
 80 | do
 81 | {
 82 |     $fiItems = $Inbox.FindItems($Sfha,$ivItemView)
 83 |     foreach($Item in $fiItems.Items) {
 84 |         $Item.IsRead = $true
 85 |         $item.Update("AlwaysOverwrite")
 86 |         $Item.Move($findFolderResults.Folders[0].Id) | Out-Null
 87 |     }
 88 |     $ivItemView.Offset += $fiItems.Items.Count
 89 | }while($fiItems.MoreAvailable -eq $true)
 90 | ###########Decompress Archives###########
 91 | Function DeGZip-File{
 92 |     Param(
 93 |         $infile,
 94 |         $outfile       
 95 |     )
 96 | 
 97 |     $input = New-Object System.IO.FileStream $inFile, ([IO.FileMode]::Open), ([IO.FileAccess]::Read), ([IO.FileShare]::Read)
 98 |     $output = New-Object System.IO.FileStream $outFile, ([IO.FileMode]::Create), ([IO.FileAccess]::Write), ([IO.FileShare]::None)
 99 |     $gzipStream = New-Object System.IO.Compression.GzipStream $input, ([IO.Compression.CompressionMode]::Decompress)
100 | 
101 |     $buffer = New-Object byte[](1024)
102 |     while($true){
103 |         $read = $gzipstream.Read($buffer, 0, 1024)
104 |         if ($read -le 0){break}
105 |     $output.Write($buffer, 0, $read)
106 |     }
107 | 
108 |     $gzipStream.Close()
109 |     $output.Close()
110 |     $input.Close()
111 | }
112 | $degz = Get-ChildItem $downloadDirectory | Where-Object {$_.name -match ".*\.gz$"}
113 | $dezip = Get-ChildItem $downloadDirectory | Where-Object {$_.name -match ".*\.zip$"}
114 | if ((Test-Path $outfile) -eq $false) {
115 |     New-Item -ItemType Directory -Force -Path $outfile | Out-Null
116 | }
117 | foreach ($file in $degz) {
118 |     $of = $file.name.Replace(".xml.gz","")
119 |     DeGZip-File  $downloadDirectory\$file $outfile\$of".xml"
120 | }
121 | foreach ($infile in $dezip) {
122 |     Expand-Archive -Path $downloadDirectory\$infile -DestinationPath $outfile
123 | }
124 | ###########Modify XML Files###########
125 | if ((Test-Path $Ingest) -eq $false) {
126 |     New-Item -ItemType Directory -Force -Path $Ingest | Out-Null
127 | }
128 | $XMLRepo = Get-Childitem $outfile -Recurse | Where-Object {$_.Name -match ".*\.xml$"}
129 | foreach ($xmlfile in $xmlrepo) {
130 |     [xml]$xml = Get-Content -Path $xmlfile.VersionInfo.FileName
131 |     $xmlrecord = $xml.feedback.record
132 |     foreach ($record in $xmlrecord) {
133 |         $xmlreport = $xml.SelectSingleNode("//feedback/report_metadata").Clone()
134 |         $xmlpolicy = $xml.SelectSingleNode("//feedback/policy_published").Clone()
135 |         $record.AppendChild($xmlreport) | Out-Null
136 |         $record.AppendChild($xmlpolicy) | Out-Null
137 |     }
138 |     $xmlpolicy = $xml.SelectSingleNode("//feedback/policy_published")
139 |     $xmlreport = $xml.SelectSingleNode("//feedback/report_metadata")
140 |     $xml.feedback.RemoveChild($xmlreport) | Out-Null
141 |     $xml.feedback.RemoveChild($xmlpolicy) | Out-Null
142 |     $xml.Save("$Ingest\$xmlfile")
143 | }
144 | ###########File Cleanup###########
145 | Get-ChildItem $downloadDirectory | Where-Object {$_.name -match ".*\.gz$|.*\.zip$"} | Remove-Item -Force -Confirm:$false
146 | $dlc = Get-ChildItem $downloadDirectory
147 | if ($dlc.count -lt 1) {
148 |     Remove-Item $downloadDirectory
149 | }
150 | Get-ChildItem $outfile | Where-Object {$_.name -match ".*\.xml$"} | Remove-Item -Force -Confirm:$false
151 | $dcx = Get-ChildItem $outfile
152 | if ($dcx.count -lt 1) {
153 |     Remove-Item $outfile
154 | }
155 | #Deletes files from the ingest folder that are older than 10 minutes
156 | Get-ChildItem $ingest | Where-Object {$_.name -match ".*\.xml$" -and $_.LastWriteTime -lt ((Get-Date).AddMinutes(-10))} | Remove-Item -Force -Confirm:$false


--------------------------------------------------------------------------------
/logstash/bin/dmarcscript.ps1:
--------------------------------------------------------------------------------
  1 | #Email address to retrieve aggregate reports from, user running process must have Full Access rights to mailbox.
  2 | $MailboxName = Read-Host "E-mail address of aggregate mailbox"
  3 | #FQDN to a CAS server, CAS array URL works.
  4 | $urireq = Read-Host "Enter FQDN to CAS(https://mail.example.com)"
  5 | $downloadDirectory = Read-Host "Path to save attachments to"
  6 | $outfile = Read-Host "Path to save extracted files"
  7 | $Ingest = Read-Host "Path to save modified XML reports"
  8 | $FolderDest = Read-Host "Name of mailbox folder to move message to (must exist)"
  9 | $Cleanup = Read-Host "Cleanup attachments and unmodified reports? (y|n)"
 10 | Clear-Host
 11 | ###########Download messages from mailbox###########
 12 | #Accept any certificates presented by the CAS
 13 | 
 14 | #Create a compilation environment
 15 | $Provider=New-Object Microsoft.CSharp.CSharpCodeProvider
 16 | $Compiler=$Provider.CreateCompiler()
 17 | $Params=New-Object System.CodeDom.Compiler.CompilerParameters
 18 | $Params.GenerateExecutable=$False
 19 | $Params.GenerateInMemory=$True
 20 | $Params.IncludeDebugInformation=$False
 21 | $Params.ReferencedAssemblies.Add("System.DLL") | Out-Null
 22 | $TASource=@'
 23 |     namespace Local.ToolkitExtensions.Net.CertificatePolicy{
 24 |         public class TrustAll : System.Net.ICertificatePolicy {
 25 |             public TrustAll() {
 26 |             }
 27 |             public bool CheckValidationResult(System.Net.ServicePoint sp,
 28 |             System.Security.Cryptography.X509Certificates.X509Certificate cert,
 29 |             System.Net.WebRequest req, int problem) {
 30 |                 return true;
 31 |             }
 32 |         }
 33 |     }
 34 | '@
 35 | $TAResults=$Provider.CompileAssemblyFromSource($Params,$TASource)
 36 | $TAAssembly=$TAResults.CompiledAssembly
 37 | 
 38 | #We now create an instance of the TrustAll and attach it to the ServicePointManager
 39 | $TrustAll=$TAAssembly.CreateInstance("Local.ToolkitExtensions.Net.CertificatePolicy.TrustAll")
 40 | [System.Net.ServicePointManager]::CertificatePolicy=$TrustAll
 41 | 
 42 | #Load the EWS API and connect to the CAS/EWS
 43 | #EWS API is found at: https://www.microsoft.com/en-us/download/details.aspx?id=42951
 44 | #Load Managed API dll
 45 | if ((Test-Path "C:\Program Files\Microsoft\Exchange\Web Services\2.2\Microsoft.Exchange.WebServices.dll") -eq $true) {
 46 |   $API22 = $true
 47 |   Add-Type -Path "C:\Program Files\Microsoft\Exchange\Web Services\2.2\Microsoft.Exchange.WebServices.dll"
 48 |   Write-Host "EWS API 2.2 loaded"
 49 | }
 50 | if ($API22 -ne $true -and (Test-Path "C:\Program Files\Microsoft\Exchange\Web Services\1.2\Microsoft.Exchange.WebServices.dll") -eq $true) {
 51 |   $API12= $true
 52 |   Add-Type -Path "C:\Program Files\Microsoft\Exchange\Web Services\1.2\Microsoft.Exchange.WebServices.dll"
 53 |   Write-Host "EWS API 1.2 loaded"
 54 | }
 55 | if ($API22 -ne $true -and $API12 -ne $true) {
 56 |   Write-Host "EWS API not detected, quitting in five seconds..."
 57 |   Write-Host "EWS API 2.2 can be downloaded from https://www.microsoft.com/en-us/download/details.aspx?id=42951"
 58 |   Start-Sleep -Seconds 5
 59 |   Exit
 60 | }
 61 | 
 62 | #Set Exchange Version (Values found here: https://msdn.microsoft.com/en-us/library/microsoft.exchange.webservices.data.exchangeversion(v=exchg.80).aspx)
 63 | $ExchangeVersion = [Microsoft.Exchange.WebServices.Data.ExchangeVersion]::Exchange2013_SP1
 64 | 
 65 | #Create Exchange Service Object
 66 | $service = New-Object Microsoft.Exchange.WebServices.Data.ExchangeService($ExchangeVersion)
 67 | 
 68 | #Set Credentials to use two options are available Option 1 to use explicit credentials or Option 2 use the Default (logged On) credentials
 69 | #Credentials Option 1 using UPN for the windows Account
 70 | #$creds = New-Object System.Net.NetworkCredential("USERNAME","PASSWORD","DOMAIN")
 71 | #$service.Credentials = $creds
 72 | 
 73 | #Credentials Option 2
 74 | #service.UseDefaultCredentials = $true
 75 | 
 76 | #Set the URL of the CAS (Client Access Server) to use three options are available to use Autodiscover to find the CAS URL, Hardcode the CAS to use, or
 77 | #prompt user for FQDN.
 78 | #Specify mailbox to connect to - Moved to line 1
 79 | 
 80 | #CAS URL Option 1 Autodiscover
 81 | #$service.AutodiscoverUrl($MailboxName,{$true})
 82 | #"Using CAS Server : " + $Service.url
 83 | 
 84 | #CAS URL Option 2 Hardcoded
 85 | #$uri=[system.URI] "https://owa.stlouisco.com/ews/Exchange.asmx"
 86 | #$service.Url = $uri
 87 | 
 88 | #CAS URL Option 3 User Prompt
 89 | #Moved to line 2
 90 | $uri = [System.URI] "$urireq/ews/Exchange.asmx"
 91 | $service.Url = $uri
 92 | 
 93 | #Bind to the Inbox folder
 94 | $Sfha = new-object Microsoft.Exchange.WebServices.Data.SearchFilter+IsEqualTo([Microsoft.Exchange.WebServices.Data.EmailMessageSchema]::HasAttachments, $true)
 95 | $folderid= new-object Microsoft.Exchange.WebServices.Data.FolderId([Microsoft.Exchange.WebServices.Data.WellKnownFolderName]::Inbox,"$MailboxName")
 96 | $Inbox = [Microsoft.Exchange.WebServices.Data.Folder]::Bind($service,$folderid)
 97 | 
 98 | #Find attachments and copy them to the download directory
 99 | $ivItemView = New-Object Microsoft.Exchange.WebServices.Data.ItemView(200)
100 | #Moved to line 3
101 | if ((Test-Path $downloadDirectory) -eq $false) {
102 |   New-Item -ItemType Directory -Force -Path $downloadDirectory | Out-Null
103 | }
104 | $findItemsResults = $Inbox.FindItems($Sfha,$ivItemView)
105 | foreach($miMailItems in $findItemsResults.Items){
106 |     $miMailItems.Load()
107 |     foreach($attach in $miMailItems.Attachments){
108 |         $attach.Load()
109 |         $fiFile = new-object System.IO.FileStream(($downloadDirectory + "\" + $attach.Name.ToString()), [System.IO.FileMode]::Create)
110 |         $fiFile.Write($attach.Content, 0, $attach.Content.Length)
111 |         $fiFile.Close()
112 |         $i++
113 |         Write-Progress -activity "Downloading attachments..." -status "Downloaded: $i of $($findItemsResults.Subject.Count)"
114 |     }
115 | }
116 | 
117 | #This section moves emails from the Inbox to a subfolder of "Inbox" called "Review Completed", make sure to create the folder.
118 | #Get the ID of the folder to move to
119 | #Moved to line 4
120 | $fvFolderView =  New-Object Microsoft.Exchange.WebServices.Data.FolderView(100)
121 | $fvFolderView.Traversal = [Microsoft.Exchange.WebServices.Data.FolderTraversal]::Shallow;
122 | $SfSearchFilter = new-object Microsoft.Exchange.WebServices.Data.SearchFilter+IsEqualTo([Microsoft.Exchange.WebServices.Data.FolderSchema]::DisplayName,"$FolderDest")
123 | $findFolderResults = $Inbox.FindFolders($SfSearchFilter,$fvFolderView)
124 | 
125 | #Define ItemView to retrieve just 200 Items
126 | $ivItemView =  New-Object Microsoft.Exchange.WebServices.Data.ItemView(200)
127 | $fiItems = $null
128 | do
129 | {
130 |     $fiItems = $Inbox.FindItems($Sfha,$ivItemView)
131 |     #[Void]$service.LoadPropertiesForItems($fiItems,$psPropset)
132 |     foreach($Item in $fiItems.Items) {
133 |         #Mark item as read
134 |         $Item.IsRead = $true
135 |         $item.Update("AlwaysOverwrite")
136 |         #Move the Message
137 |         $Item.Move($findFolderResults.Folders[0].Id) | Out-Null
138 |         $p++
139 |         Write-Progress -activity "Moving messages" -status "Moved: $p of $($fiItems.Subject.Count)"
140 |     }
141 |     $ivItemView.Offset += $fiItems.Items.Count
142 | }while($fiItems.MoreAvailable -eq $true)
143 | 
144 | 
145 | ###########Decompress Archives###########
146 | Function DeGZip-File{
147 |     Param(
148 |         $infile,
149 |         $outfile       
150 |         )
151 | 
152 |     $input = New-Object System.IO.FileStream $inFile, ([IO.FileMode]::Open), ([IO.FileAccess]::Read), ([IO.FileShare]::Read)
153 |     $output = New-Object System.IO.FileStream $outFile, ([IO.FileMode]::Create), ([IO.FileAccess]::Write), ([IO.FileShare]::None)
154 |     $gzipStream = New-Object System.IO.Compression.GzipStream $input, ([IO.Compression.CompressionMode]::Decompress)
155 | 
156 |     $buffer = New-Object byte[](1024)
157 |     while($true){
158 |         $read = $gzipstream.Read($buffer, 0, 1024)
159 |         if ($read -le 0){break}
160 |         $output.Write($buffer, 0, $read)
161 |         }
162 | 
163 |     $gzipStream.Close()
164 |     $output.Close()
165 |     $input.Close()
166 | }
167 | $degz = Get-ChildItem $downloadDirectory | Where-Object {$_.name -match ".*\.gz$"}
168 | $dezip = Get-ChildItem $downloadDirectory | Where-Object {$_.name -match ".*\.zip$"}
169 | #Moved to line 5
170 | if ((Test-Path $outfile) -eq $false) {
171 |   New-Item -ItemType Directory -Force -Path $outfile | Out-Null
172 | }
173 | foreach ($file in $degz) {
174 |     $of = $file.name.Replace(".xml.gz","")
175 |     DeGZip-File  $downloadDirectory\$file $outfile\$of".xml"
176 |     $gz++
177 |     Write-Progress -activity "Decompressing GZ files" -status "Decompressed: $gz of $($degz.Count)"
178 |     }
179 | foreach ($infile in $dezip) {
180 |     Expand-Archive -Path $downloadDirectory\$infile -DestinationPath $outfile
181 |     $gzi++
182 |     Write-Progress -activity "Decompressing zip files" -status "Decompressed: $gzi of $($dezip.Count)"
183 | }
184 | 
185 | 
186 | ###########Modify XML Files###########
187 | #Moved to line 6
188 | if ((Test-Path $Ingest) -eq $false) {
189 |   New-Item -ItemType Directory -Force -Path $Ingest | Out-Null
190 | }
191 | #XML Restructure
192 | $XMLRepo = Get-Childitem $outfile -Recurse | Where-Object {$_.Name -match ".*\.xml$"}
193 | foreach ($xmlfile in $xmlrepo) {
194 |   [xml]$xml = Get-Content -Path $xmlfile.VersionInfo.FileName
195 |   $xmlrecord = $xml.feedback.record
196 |   foreach ($record in $xmlrecord) {
197 |     $xmlreport = $xml.SelectSingleNode("//feedback/report_metadata").Clone()
198 |     $xmlpolicy = $xml.SelectSingleNode("//feedback/policy_published").Clone()
199 |     $record.AppendChild($xmlreport) | Out-Null
200 |     $record.AppendChild($xmlpolicy) | Out-Null
201 |   }
202 |   $xmlpolicy = $xml.SelectSingleNode("//feedback/policy_published")
203 |   $xmlreport = $xml.SelectSingleNode("//feedback/report_metadata")
204 |   $xml.feedback.RemoveChild($xmlreport) | Out-Null
205 |   $xml.feedback.RemoveChild($xmlpolicy) | Out-Null
206 | #Save to ingest point
207 |   $xml.Save("$Ingest\$xmlfile")
208 |   $xmlprogress++
209 |   Write-Progress -activity "Modifying XML Structures" -status "Modified: $xmlprogress of $($xmlrepo.Count)"
210 | }
211 | Clear-Host
212 | Write-Host "Modification process complete!"
213 | Start-Sleep -Seconds 3
214 | 
215 | ###########File Cleanup###########
216 | #Moved to line 7
217 | if ($cleanup -eq "y") {
218 |     #Remove downloaded email attachments
219 |     Get-ChildItem $downloadDirectory | Where-Object {$_.name -match ".*\.gz$|.*\.zip$"} | Remove-Item -Force -Confirm:$false
220 |     $dlc = Get-ChildItem $downloadDirectory
221 |     if ($dlc.count -lt 1) {
222 |         Remove-Item $downloadDirectory
223 |     }
224 |     #Remove decompressed, unmodified XML files
225 |     Get-ChildItem $outfile | Where-Object {$_.name -match ".*\.xml$"} | Remove-Item -Force -Confirm:$false
226 |     $dcx = Get-ChildItem $outfile
227 |     if ($dcx.count -lt 1) {
228 |         Remove-Item $outfile
229 |     }
230 | }
231 | if ($cleanup -eq "n") {
232 | exit
233 | }
234 | #Remove modified XML files
235 | $Cleanup = Read-Host "Remove modified files? (y|n)"
236 | if ($cleanup -eq "y") {
237 |     Get-ChildItem $ingest | Where-Object {$_.name -match ".*\.xml$"} | Remove-Item -Force -Confirm:$false
238 |     $modx = Get-ChildItem $ingest
239 |     if ($modx.count -lt 1) {
240 |         Remove-Item $ingest
241 |     }
242 | }
243 | Clear-Host
244 | Write-Host "Cleanup process complete!"
245 | Start-Sleep -Seconds 3
246 | 


--------------------------------------------------------------------------------
/logstash/config/jvm.options:
--------------------------------------------------------------------------------
 1 | ## JVM configuration
 2 | 
 3 | # Xms represents the initial size of total heap space
 4 | # Xmx represents the maximum size of total heap space
 5 | 
 6 | -Xms1g
 7 | -Xmx1g
 8 | 
 9 | ################################################################
10 | ## Expert settings
11 | ################################################################
12 | ##
13 | ## All settings below this section are considered
14 | ## expert settings. Don't tamper with them unless
15 | ## you understand what you are doing
16 | ##
17 | ################################################################
18 | 
19 | ## GC configuration
20 | -XX:+UseParNewGC
21 | -XX:+UseConcMarkSweepGC
22 | -XX:CMSInitiatingOccupancyFraction=75
23 | -XX:+UseCMSInitiatingOccupancyOnly
24 | 
25 | ## Locale
26 | # Set the locale language
27 | #-Duser.language=en
28 | 
29 | # Set the locale country
30 | #-Duser.country=US
31 | 
32 | # Set the locale variant, if any
33 | #-Duser.variant=
34 | 
35 | ## basic
36 | 
37 | # set the I/O temp directory
38 | #-Djava.io.tmpdir=$HOME
39 | 
40 | # set to headless, just in case
41 | -Djava.awt.headless=true
42 | 
43 | # ensure UTF-8 encoding by default (e.g. filenames)
44 | -Dfile.encoding=UTF-8
45 | 
46 | # use our provided JNA always versus the system one
47 | #-Djna.nosys=true
48 | 
49 | # Turn on JRuby invokedynamic
50 | -Djruby.compile.invokedynamic=true
51 | # Force Compilation
52 | -Djruby.jit.threshold=0
53 | 
54 | ## heap dumps
55 | 
56 | # generate a heap dump when an allocation from the Java heap fails
57 | # heap dumps are created in the working directory of the JVM
58 | -XX:+HeapDumpOnOutOfMemoryError
59 | 
60 | # specify an alternative path for heap dumps
61 | # ensure the directory exists and has sufficient space
62 | #-XX:HeapDumpPath=${LOGSTASH_HOME}/heapdump.hprof
63 | 
64 | ## GC logging
65 | #-XX:+PrintGCDetails
66 | #-XX:+PrintGCTimeStamps
67 | #-XX:+PrintGCDateStamps
68 | #-XX:+PrintClassHistogram
69 | #-XX:+PrintTenuringDistribution
70 | #-XX:+PrintGCApplicationStoppedTime
71 | 
72 | # log GC status to a file with time stamps
73 | # ensure the directory exists
74 | #-Xloggc:${LS_GC_LOG_FILE}
75 | 
76 | # Entropy source for randomness
77 | -Djava.security.egd=file:/dev/urandom
78 | 


--------------------------------------------------------------------------------
/logstash/config/logstash.yml:
--------------------------------------------------------------------------------
1 |  node.name: ServerHostname
2 |  config.reload.automatic: true
3 |  config.reload.interval: 5s
4 |  http.host: "ServerIPv4Address"
5 |  http.port: 9600
6 | # path.data: "c:/ElasticStack/Data/Logstash"
7 | # path.logs: "c:/ElasticStack/Logs/Logstash"
8 | 


--------------------------------------------------------------------------------
/logstash/config/pipelines.yml:
--------------------------------------------------------------------------------
1 |  - pipeline.id: dmarcxml
2 | #  Do not include drive letter in below path.
3 |    path.config: "/logstash/config/pipelines/dmarcpipeline.yml"
4 | #
5 | # Uncomment the below to use a preconfigured beats pipeline.
6 | # - pipeline.id: beats
7 | #   path.config: "/logstash/config/pipelines/beatspipeline.yml"


--------------------------------------------------------------------------------
/logstash/config/pipelines/beatspipeline.yml:
--------------------------------------------------------------------------------
 1 | input {
 2 |   beats {
 3 |     id => "Beats Input"
 4 |     host => "0.0.0.0"
 5 |     port => 5044
 6 |     include_codec_tag => false
 7 |   }
 8 | }
 9 | output {
10 |   elasticsearch{
11 |     id => "Output to Elasticsearch"
12 |     hosts => [ "http://ElasticServerHostnameOrIP:port" ]
13 |     index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
14 |     manage_template => false
15 | # Uncomment and configure the below for xpack integration
16 | #   user: "elastic"
17 | #   password: "changeme"
18 |   }
19 | }


--------------------------------------------------------------------------------
/logstash/config/pipelines/dmarcpipeline.yml:
--------------------------------------------------------------------------------
  1 | input {
  2 |   file {
  3 |     id => "Ingest_DMARC"
  4 |     path => ["/Ingest/DMARC/*"]
  5 |     discover_interval => 5
  6 |     mode => "read"
  7 |     codec => multiline {
  8 |       negate => true
  9 |       pattern => "<record>"
 10 |       what => "previous"
 11 |       multiline_tag => ""
 12 |     }
 13 |   }
 14 | }
 15 | filter {
 16 |   xml {
 17 |     id => "Field Extraction"
 18 |     store_xml => false
 19 |     force_array => false
 20 |     source => "message"
 21 |     xpath => [
 22 |       "record/report_metadata/org_name/text()", "report.org",
 23 |       "record/report_metadata/email/text()", "report.org_contact",
 24 |       "record/report_metadata/extra_contact_info/text()", "report.additional_contact",
 25 |       "record/report_metadata/report_id/text()", "report.id",
 26 |       "record/report_metadata/date_range/begin/text()", "report.start",
 27 |       "record/report_metadata/date_range/end/text()", "report.end",
 28 |       "record/report_metadata/error/text()", "report.error",
 29 |       "record/policy_published/domain/text()", "policy.domain",
 30 |       "record/policy_published/aspf/text()", "policy.spf_mode",
 31 |       "record/policy_published/adkim/text()", "policy.dkim_mode",
 32 |       "record/policy_published/p/text()", "policy.dmarc.domain_action",
 33 |       "record/policy_published/sp/text()", "policy.dmarc.subdomain_action",
 34 |       "record/policy_published/pct/text()", "policy.percentage",
 35 |       "record/policy_published/fo/text()", "policy.forensic_reporting",
 36 |       "record/row/source_ip/text()", "email.source_ip",
 37 |       "record/row/count/text()", "email.count",
 38 |       "record/row/policy_evaluated/disposition/text()", "email.dmarc_action",
 39 |       "record/row/policy_evaluated/spf/text()", "email.spf_evaluation",
 40 |       "record/row/policy_evaluated/dkim/text()", "email.dkim_evaluation",
 41 |       "record/row/policy_evaluated/reason/type/text()", "dmarc.override_type",
 42 |       "record/row/policy_evaluated/reason/comment/text()", "dmarc.override_comment",
 43 |       "record/identifiers/envelope_to/text()", "email.envelope_to",
 44 |       "record/identifiers/envelope_from/text()", "email.envelope_from",
 45 |       "record/identifiers/header_from/text()", "email.header_from",
 46 |       "record/auth_results/dkim/domain/text()", "authresult.dkim_domain",
 47 |       "record/auth_results/dkim/selector/text()", "authresult.dkim_selector",
 48 |       "record/auth_results/dkim/result/text()", "authresult.dkim_result",
 49 |       "record/auth_results/dkim/human_result/text()", "authresult.dkim_human_result",
 50 |       "record/auth_results/spf/domain/text()", "authresult.spf_domain",
 51 |       "record/auth_results/spf/scope/text()", "authresult.spf_scope",
 52 |       "record/auth_results/spf/result/text()", "authresult.spf_result"
 53 |     ]
 54 |   }
 55 |   mutate {
 56 |     id => "Field Normalize"
 57 |     strip => [
 58 |       "report.end",
 59 |       "report.start"
 60 |     ]
 61 |     lowercase => [
 62 |       "email.dkim_evaluation",
 63 |       "email.dmarc_action",
 64 |       "email.spf_evaluation",
 65 |       "policy.dmarc.domain_action",
 66 |       "policy.dmarc.subdomain_action",
 67 |       "policy.dkim_mode",
 68 |       "policy.spf_mode"
 69 |     ]
 70 |     gsub => [
 71 |       "policy.dkim_mode", "r", "Relaxed",
 72 |       "policy.dkim_mode", "s", "Strict",
 73 |       "policy.spf_mode", "r", "Relaxed",
 74 |       "policy.spf_mode", "s", "Strict",
 75 |       "policy.forensic_reporting", "0", "All Fail",
 76 |       "policy.forensic_reporting", "1", "Any Fail",
 77 |       "policy.forensic_reporting", "d", "DKIM Fail",
 78 |       "policy.forensic_reporting", "s", "SPF Fail"
 79 |     ]
 80 |   }
 81 |   geoip {
 82 |     id => "IP Geo-mapping"
 83 |     source => "email.source_ip"
 84 |   }
 85 |   if "_geoip_lookup_failure" not in [tags] {
 86 |     mutate {
 87 |       id => "Create coordinates field"
 88 |       add_field => {
 89 |         "[geoip][location][coordinates]" => "%{[geoip][location][lat]}, %{[geoip][location][lon]}"
 90 |       }
 91 |       remove_field => ["host"]
 92 |     }
 93 |   }
 94 |   fingerprint {
 95 |     id => "Duplicate Protection"
 96 |     source => "message"
 97 |     target => "[@metadata][fingerprint]"
 98 |     method => "MURMUR3"
 99 |   }
100 |   mutate {
101 |     add_field => {
102 |       "ingest_time" => "%{+YYYY.MM.dd HH:mm:ss}"
103 |     }
104 |   }
105 | }
106 | output {
107 |   elasticsearch {
108 |     id => "Send to Elasticsearch"
109 |     hosts => ["ELASTICSEARCHHOST:9200"]
110 |     document_id => "%{[@metadata][fingerprint]}"
111 |     template => "/Logstash/templates/dmarcxmltemplate.json"
112 |     template_name => "dmarcxml"
113 |     index => "dmarcxml-%{+YYYY.MM.dd}"
114 |   }
115 | }
116 | 


--------------------------------------------------------------------------------
/logstash/templates/dmarcxmltemplate.json:
--------------------------------------------------------------------------------
  1 | {
  2 | 	"index_patterns": ["dmarcxml-*"],
  3 | 	"settings": {
  4 | 		"index": {
  5 | 			"refresh_interval": "5s",
  6 | 			"number_of_shards": 1,
  7 | 			"number_of_replicas": 0
  8 | 		}
  9 | 	},
 10 | 	"mappings": {
 11 | 		"doc": {
 12 | 			"properties": {
 13 | 				"@timestamp": {
 14 | 					"type": "date"
 15 | 				},
 16 | 				"@version": {
 17 | 					"type": "text",
 18 | 					"fields": {
 19 | 						"keyword": {
 20 | 							"type": "keyword"
 21 | 						}
 22 | 					}
 23 | 				},
 24 | 				"auth_result": {
 25 | 					"properties": {
 26 | 						"dkim_domain": {
 27 | 							"type": "text",
 28 | 							"fields": {
 29 | 								"keyword": {
 30 | 									"type": "keyword"
 31 | 								}
 32 | 							}
 33 | 						},
 34 | 						"dkim_result": {
 35 | 							"type": "text",
 36 | 							"fields": {
 37 | 								"keyword": {
 38 | 									"type": "keyword"
 39 | 								}
 40 | 							}
 41 | 						},
 42 | 						"dkim_human_result": {
 43 | 							"type": "text",
 44 | 							"fields": {
 45 | 								"keyword": {
 46 | 									"type": "keyword"
 47 | 								}
 48 | 							}
 49 | 						},
 50 | 						"dkim_selector": {
 51 | 							"type": "text",
 52 | 							"fields": {
 53 | 								"keyword": {
 54 | 									"type": "keyword"
 55 | 								}
 56 | 							}
 57 | 						},
 58 | 						"spf_domain": {
 59 | 							"type": "text",
 60 | 							"fields": {
 61 | 								"keyword": {
 62 | 									"type": "keyword"
 63 | 								}
 64 | 							}
 65 | 						},
 66 | 						"spf_result": {
 67 | 							"type": "text",
 68 | 							"fields": {
 69 | 								"keyword": {
 70 | 									"type": "keyword"
 71 | 								}
 72 | 							}
 73 | 						},
 74 | 						"spf_scope": {
 75 | 							"type": "text",
 76 | 							"fields": {
 77 | 								"keyword": {
 78 | 									"type": "keyword"
 79 | 								}
 80 | 							}
 81 | 						}
 82 | 					}
 83 | 				},
 84 | 				"dmarc": {
 85 | 					"properties": {
 86 | 						"override_comment": {
 87 | 							"type": "text",
 88 | 							"fields": {
 89 | 								"keyword": {
 90 | 									"type": "keyword"
 91 | 								}
 92 | 							}
 93 | 						},
 94 | 						"override_type": {
 95 | 							"type": "text",
 96 | 							"fields": {
 97 | 								"keyword": {
 98 | 									"type": "keyword"
 99 | 								}
100 | 							}
101 | 						}
102 | 					}
103 | 				},
104 | 				"email": {
105 | 					"properties": {
106 | 						"count": {
107 | 							"type": "short"
108 | 						},
109 | 						"dkim_evaluation": {
110 | 							"type": "text",
111 | 							"fields": {
112 | 								"keyword": {
113 | 									"type": "keyword"
114 | 								}
115 | 							}
116 | 						},
117 | 						"dmarc_action": {
118 | 							"type": "text",
119 | 							"fields": {
120 | 								"keyword": {
121 | 									"type": "keyword"
122 | 								}
123 | 							}
124 | 						},
125 | 						"envelope_from": {
126 | 							"type": "text",
127 | 							"fields": {
128 | 								"keyword": {
129 | 									"type": "keyword"
130 | 								}
131 | 							}
132 | 						},
133 | 						"envelope_to": {
134 | 							"type": "text",
135 | 							"fields": {
136 | 								"keyword": {
137 | 									"type": "keyword"
138 | 								}
139 | 							}
140 | 						},
141 | 						"header_from": {
142 | 							"type": "text",
143 | 							"fields": {
144 | 								"keyword": {
145 | 									"type": "keyword"
146 | 								}
147 | 							}
148 | 						},
149 | 						"source_ip": {
150 | 							"type": "ip"
151 | 						},
152 | 						"spf_evaluation": {
153 | 							"type": "text",
154 | 							"fields": {
155 | 								"keyword": {
156 | 									"type": "keyword"
157 | 								}
158 | 							}
159 | 						}
160 | 					}
161 | 				},
162 | 				"geoip": {
163 | 					"properties": {
164 | 						"city_name": {
165 | 						"type": "text",
166 | 						"fields": {
167 | 							"keyword": {
168 | 								"type": "keyword"
169 | 							}
170 | 						}
171 | 					},
172 | 						"continent_code": {
173 | 						"type": "text",
174 | 						"fields": {
175 | 							"keyword": {
176 | 								"type": "keyword"
177 | 							}
178 | 						}
179 | 					},
180 | 						"country_code2": {
181 | 						"type": "text",
182 | 						"fields": {
183 | 							"keyword": {
184 | 								"type": "keyword"
185 | 							}
186 | 						}
187 | 					},
188 | 						"country_code3": {
189 | 						"type": "text",
190 | 						"fields": {
191 | 							"keyword": {
192 | 								"type": "keyword"
193 | 							}
194 | 						}
195 | 					},
196 | 						"country_name": {
197 | 						"type": "text",
198 | 						"fields": {
199 | 							"keyword": {
200 | 								"type": "keyword"
201 | 							}
202 | 						}
203 | 					},
204 | 						"dma_code": {
205 | 						"type": "long"
206 | 					},
207 | 						"ip": {
208 | 						"type": "ip",
209 | 						"fields": {
210 | 							"keyword": {
211 | 								"type": "keyword"
212 | 							}
213 | 						}
214 | 					},
215 | 						"latitude": {
216 | 							"type": "float"
217 | 						},
218 | 						"location": {
219 | 							"properties": {
220 | 								"coordinates": {
221 | 									"type": "geo_point",
222 | 									"fields": {
223 | 										"keyword": {
224 | 											"type": "keyword"
225 | 										}
226 | 									}
227 | 								},
228 | 								"lat": {
229 | 									"type": "float"
230 | 								},
231 | 								"lon": {
232 | 									"type": "float"
233 | 								},
234 | 								"longitude": {
235 | 									"type": "float"
236 | 								}
237 | 							}
238 | 						},
239 | 						"postal_code": {
240 | 							"type": "text",
241 | 							"fields": {
242 | 								"keyword": {
243 | 								"type": "keyword"
244 | 								}
245 | 							}
246 | 						},
247 | 						"region_code": {
248 | 							"type": "text",
249 | 							"fields": {
250 | 								"keyword": {
251 | 									"type": "keyword"
252 | 								}
253 | 							}
254 | 						},
255 | 						"region_name": {
256 | 							"type": "text",
257 | 							"fields": {
258 | 								"keyword": {
259 | 									"type": "keyword"
260 | 								}
261 | 							}
262 | 						},
263 | 						"timezone": {
264 | 							"type": "text",
265 | 							"fields": {
266 | 								"keyword": {
267 | 									"type": "keyword"
268 | 								}
269 | 							}
270 | 						}
271 | 					}
272 | 				},
273 | 				"message": {
274 | 					"type": "text",
275 | 					"fields": {
276 | 						"keyword": {
277 | 							"type": "keyword"
278 | 						}
279 | 					}
280 | 				},
281 | 				"path": {
282 | 				"type": "text",
283 | 					"fields": {
284 | 						"keyword": {
285 | 						"type": "keyword"
286 | 						}
287 | 					}
288 | 				},
289 | 				"policy": {
290 | 					"properties": {
291 | 						"dkim_alignment": {
292 | 							"type": "text",
293 | 							"fields": {
294 | 								"keyword": {
295 | 									"type": "keyword"
296 | 								}
297 | 							}
298 | 						},
299 | 						"dmarc": {
300 | 							"properties": {
301 | 								"domain_action": {
302 | 									"type": "text",
303 | 									"fields": {
304 | 										"keyword": {
305 | 											"type": "keyword"
306 | 										}
307 | 									}
308 | 								},
309 | 								"subdomain_action": {
310 | 									"type": "text",
311 | 									"fields": {
312 | 										"keyword": {
313 | 											"type": "keyword"
314 | 										}
315 | 									}
316 | 								}
317 | 							}
318 | 						},
319 | 						"domain": {
320 | 							"type": "text",
321 | 							"fields": {
322 | 								"keyword": {
323 | 									"type": "keyword"
324 | 								}
325 | 							}
326 | 						},
327 | 						"percentage": {
328 | 							"type": "short",
329 | 							"fields": {
330 | 								"keyword": {
331 | 									"type": "keyword"
332 | 								}
333 | 							}
334 | 						},
335 | 						"forensic_reporting": {
336 | 							"type": "text",
337 | 							"fields": {
338 | 								"keyword": {
339 | 									"type": "keyword"
340 | 								}
341 | 							}
342 | 						},
343 | 						"spf_alignment": {
344 | 							"type": "text",
345 | 							"fields": {
346 | 								"keyword": {
347 | 									"type": "keyword"
348 | 								}
349 | 							}
350 | 						}
351 | 					}
352 | 				},
353 | 				"report": {
354 | 					"properties": {
355 | 						"org_contact_additional": {
356 | 							"type": "text",
357 | 							"fields": {
358 | 								"keyword": {
359 | 									"type": "keyword"
360 | 								}
361 | 							}
362 | 						},
363 | 						"end": {
364 | 							"type": "date",
365 | 							"format": "epoch_second",
366 | 							"fields": {
367 | 								"keyword": {
368 | 									"type": "keyword"
369 | 								}
370 | 							}
371 | 						},
372 | 						"id": {
373 | 							"type": "text",
374 | 							"fields": {
375 | 								"keyword": {
376 | 									"type": "keyword"
377 | 								}
378 | 							}
379 | 						},
380 | 						"org": {
381 | 							"type": "text",
382 | 							"fields": {
383 | 								"keyword": {
384 | 									"type": "keyword"
385 | 								}
386 | 							}
387 | 						},
388 | 						"org_contact": {
389 | 							"type": "text",
390 | 							"fields": {
391 | 								"keyword": {
392 | 									"type": "keyword"
393 | 								}
394 | 							}
395 | 						},
396 | 						"error": {
397 | 							"type": "text",
398 | 							"fields": {
399 | 								"keyword": {
400 | 									"type": "keyword"
401 | 								}
402 | 							}
403 | 						},
404 | 						"start": {
405 | 							"type": "date",
406 | 							"format": "epoch_second",
407 | 							"fields": {
408 | 								"keyword": {
409 | 									"type": "keyword"
410 | 								}
411 | 							}
412 | 						}
413 | 					}
414 | 				},
415 | 				"tags": {
416 | 					"type": "text",
417 | 					"fields": {
418 | 						"keyword": {
419 | 							"type": "keyword"
420 | 						}
421 | 					}
422 | 				}
423 | 			}
424 | 		}
425 | 	}
426 | }


--------------------------------------------------------------------------------
/samplereport.xml:
--------------------------------------------------------------------------------
 1 | <?xml version="1.0" encoding="UTF-8" ?>
 2 | <feedback>
 3 |   <report_metadata>
 4 |     <org_name>acme.com</org_name>
 5 |     <email>noreply-dmarc-support@acme.com</email>
 6 |     <extra_contact_info>http://acme.com/dmarc/support</extra_contact_info>
 7 |     <report_id>9391651994964116463</report_id>
 8 |     <error>There was a sample error.</error>
 9 |     <date_range>
10 |       <begin>1335571200</begin>
11 |       <end>1335657599</end>
12 |     </date_range>
13 |   </report_metadata>
14 |   <policy_published>
15 |     <domain>example.com</domain>
16 |     <adkim>r</adkim>
17 |     <aspf>r</aspf>
18 |     <p>none</p>
19 |     <sp>none</sp>
20 |     <pct>100</pct>
21 |     <fo>1</fo>
22 |   </policy_published>
23 |   <record>
24 |     <row>
25 |       <source_ip>72.150.241.94</source_ip>
26 |       <count>2</count>
27 |       <policy_evaluated>
28 |         <disposition>none</disposition>
29 |         <dkim>fail</dkim>
30 |         <spf>pass</spf>
31 |         <reason>
32 |           <type>Other</type>
33 |           <comment>DMARC Policy overridden for incoherent example.</comment>
34 | 		</reason>
35 |       </policy_evaluated>
36 |     </row>
37 |     <identifiers>
38 |       <header_from>example.com</header_from>
39 |       <envelope_from>example.com</envelope_from>
40 |       <envelope_to>acme.com</envelope_to>
41 |     </identifiers>
42 |     <auth_results>
43 |       <dkim>
44 |         <domain>example.com</domain>
45 |         <selector>ExamplesSelector</selector>
46 |         <result>fail</result>
47 |         <human_result>Incoherent example</human_result>
48 |       </dkim>
49 |       <spf>
50 |         <domain>example.com</domain>
51 |         <scope>helo</scope>
52 |         <result>pass</result>
53 |       </spf>
54 |     </auth_results>
55 |   </record>
56 | </feedback>


--------------------------------------------------------------------------------