9 | =====================================================================================================
10 |
11 | `anomalyDetection` implements procedures to aid in detecting network log anomalies. By combining various multivariate analytic approaches relevant to network anomaly detection, it provides cyber analysts efficient means to detect suspected anomalies requiring further evaluation.
12 |
13 | Installation
14 | ------------
15 |
16 | You can install `anomalyDetection` two ways.
17 |
18 | - Using the latest released version from CRAN:
19 |
20 |
21 |
22 | install.packages("anomalyDetection")
23 |
24 | - Using the latest development version from GitHub:
25 |
26 |
27 |
28 | if (packageVersion("devtools") < 1.6) {
29 | install.packages("devtools")
30 | }
31 |
32 | devtools::install_github("koalaverse/anomalyDetection", build_vignettes = TRUE)
33 |
34 | Learning
35 | --------
36 |
37 | To get started with `anomalyDetection`, read the intro [vignette](https://cran.r-project.org/web/packages/anomalyDetection/vignettes/Introduction.html): `vignette("Introduction", package = "anomalyDetection")`. This will provide a thorough introduction to the functions provided in the package.
38 |
39 | References
40 | ----------
41 |
42 | Gutierrez, R.J., Boehmke, B.C., Bauer, K.W., Saie, C.M. & Bihl, T.J. (2017) "`anomalyDetection`: Implementation of augmented network log anomaly detection procedures." The R Journal, 9(2), 354-365. [link](https://journal.r-project.org/archive/2017/RJ-2017-039/index.html)
43 |
--------------------------------------------------------------------------------
/man/mahalanobis_distance.Rd:
--------------------------------------------------------------------------------
1 | % Generated by roxygen2: do not edit by hand
2 | % Please edit documentation in R/mahalanobis_distance.R
3 | \name{mahalanobis_distance}
4 | \alias{mahalanobis_distance}
5 | \alias{mahalanobis_distance.matrix}
6 | \alias{mahalanobis_distance.data.frame}
7 | \title{Mahalanobis Distance}
8 | \usage{
9 | mahalanobis_distance(data, output = c("md", "bd", "both"),
10 | normalize = FALSE)
11 |
12 | \method{mahalanobis_distance}{matrix}(data, output = c("md", "bd", "both"),
13 | normalize = FALSE)
14 |
15 | \method{mahalanobis_distance}{data.frame}(data, output = c("md", "bd",
16 | "both"), normalize = FALSE)
17 | }
18 | \arguments{
19 | \item{data}{A matrix or data frame. Data frames will be converted to matrices
20 | via \code{data.matrix}.}
21 |
22 | \item{output}{Character string specifying which distance metric(s) to
23 | compute. Current options include: \code{"md"} for Mahalanobis distance
24 | (default); \code{"bd"} for absolute breakdown distance (used to see which
25 | columns drive the Mahalanobis distance); and \code{"both"} to return both
26 | distance metrics.}
27 |
28 | \item{normalize}{Logical indicating whether or not to normalize the breakdown
29 | distances within each column (so that breakdown distances across columns can
30 | be compared).}
31 | }
32 | \value{
33 | If \code{output = "md"}, then a vector containing the Mahalanobis
34 | distances is returned. Otherwise, a matrix.
35 | }
36 | \description{
37 | Calculates the distance between the elements in a data set and the mean
38 | vector of the data for outlier detection. Values are independent of the scale
39 | between variables.
40 | }
41 | \examples{
42 | \dontrun{
43 | # Simulate some data
44 | x <- data.frame(C1 = rnorm(100), C2 = rnorm(100), C3 = rnorm(100))
45 |
46 | # Add Mahalanobis distances
47 | x \%>\% dplyr::mutate(MD = mahalanobis_distance(x))
48 |
49 | # Add Mahalanobis and breakdown distances
50 | x \%>\% cbind(mahalanobis_distance(x, output = "both"))
51 |
52 | # Add Mahalanobis and normalized breakdown distances
53 | x \%>\% cbind(mahalanobis_distance(x, output = "both", normalize = TRUE))
54 | }
55 | }
56 | \references{
57 | W. Wang and R. Battiti, "Identifying Intrusions in Computer Networks with
58 | Principal Component Analysis," in First International Conference on
59 | Availability, Reliability and Security, 2006.
60 | }
61 |
--------------------------------------------------------------------------------
/docs/docsearch.js:
--------------------------------------------------------------------------------
1 | $(function() {
2 |
3 | // register a handler to move the focus to the search bar
4 | // upon pressing shift + "/" (i.e. "?")
5 | $(document).on('keydown', function(e) {
6 | if (e.shiftKey && e.keyCode == 191) {
7 | e.preventDefault();
8 | $("#search-input").focus();
9 | }
10 | });
11 |
12 | $(document).ready(function() {
13 | // do keyword highlighting
14 | /* modified from https://jsfiddle.net/julmot/bL6bb5oo/ */
15 | var mark = function() {
16 |
17 | var referrer = document.URL ;
18 | var paramKey = "q" ;
19 |
20 | if (referrer.indexOf("?") !== -1) {
21 | var qs = referrer.substr(referrer.indexOf('?') + 1);
22 | var qs_noanchor = qs.split('#')[0];
23 | var qsa = qs_noanchor.split('&');
24 | var keyword = "";
25 |
26 | for (var i = 0; i < qsa.length; i++) {
27 | var currentParam = qsa[i].split('=');
28 |
29 | if (currentParam.length !== 2) {
30 | continue;
31 | }
32 |
33 | if (currentParam[0] == paramKey) {
34 | keyword = decodeURIComponent(currentParam[1].replace(/\+/g, "%20"));
35 | }
36 | }
37 |
38 | if (keyword !== "") {
39 | $(".contents").unmark({
40 | done: function() {
41 | $(".contents").mark(keyword);
42 | }
43 | });
44 | }
45 | }
46 | };
47 |
48 | mark();
49 | });
50 | });
51 |
52 | /* Search term highlighting ------------------------------*/
53 |
54 | function matchedWords(hit) {
55 | var words = [];
56 |
57 | var hierarchy = hit._highlightResult.hierarchy;
58 | // loop to fetch from lvl0, lvl1, etc.
59 | for (var idx in hierarchy) {
60 | words = words.concat(hierarchy[idx].matchedWords);
61 | }
62 |
63 | var content = hit._highlightResult.content;
64 | if (content) {
65 | words = words.concat(content.matchedWords);
66 | }
67 |
68 | // return unique words
69 | var words_uniq = [...new Set(words)];
70 | return words_uniq;
71 | }
72 |
73 | function updateHitURL(hit) {
74 |
75 | var words = matchedWords(hit);
76 | var url = "";
77 |
78 | if (hit.anchor) {
79 | url = hit.url_without_anchor + '?q=' + escape(words.join(" ")) + '#' + hit.anchor;
80 | } else {
81 | url = hit.url + '?q=' + escape(words.join(" "));
82 | }
83 |
84 | return url;
85 | }
86 |
--------------------------------------------------------------------------------
/R/bd_row.R:
--------------------------------------------------------------------------------
1 | #' @title Breakdown for Mahalanobis Distance
2 | #'
3 | #' @description
4 | #' \code{bd_row} indicates which variables in data are driving the Mahalanobis
5 | #' distance for a specific row \code{r}, relative to the mean vector of the data.
6 | #'
7 | #' @param data numeric data
8 | #' @param row row of interest
9 | #' @param n number of values to return. By default, will return all variables
10 | #' (columns) with their respective differences. However, you can choose to view
11 | #' only the top \code{n} variables by setting the \code{n} value.
12 | #'
13 | #' @return
14 | #'
15 | #' Returns a vector indicating the variables in \code{data} that are driving the
16 | #' Mahalanobis distance for the respective row.
17 | #'
18 | #' @seealso
19 | #'
20 | #' \code{\link{mahalanobis_distance}} for computing the Mahalanobis Distance values
21 | #'
22 | #' @examples
23 | #' \dontrun{
24 | #' x = matrix(rnorm(200*3), ncol = 10)
25 | #' colnames(x) = paste0("C", 1:ncol(x))
26 | #'
27 | #' # compute the relative differences for row 5 and return all variables
28 | #' x %>%
29 | #' mahalanobis_distance("bd", normalize = TRUE) %>%
30 | #' bd_row(5)
31 | #'
32 | #' # compute the relative differences for row 5 and return the top 3 variables
33 | #' # that are influencing the Mahalanobis Distance the most
34 | #' x %>%
35 | #' mahalanobis_distance("bd", normalize = TRUE) %>%
36 | #' bd_row(5, 3)
37 | #' }
38 | #'
39 | #' @export
40 |
41 | bd_row <- function(data, row, n = NULL) {
42 |
43 | # return error if parameters are missing or invalid
44 | if(missing(data)) {
45 | stop("Missing data argument", call. = FALSE)
46 | }
47 | if(! row %in% seq_len(nrow(data))) {
48 | stop("Invalid row value", call. = FALSE)
49 | }
50 | if(length(row) != 1) {
51 | stop("row value must be a single integer", call. = FALSE)
52 | }
53 | if(!isTRUE(n %in% seq_len(ncol(data))) && !is.null(n)) {
54 | stop("Invalid n value", call. = FALSE)
55 | }
56 |
57 | C <- stats::cov(data)
58 | CM <- as.matrix(colMeans(data))
59 | D <- (data[row,] - t(CM))
60 | bd <- D /sqrt(diag(C))
61 | bd_abs <- abs(bd)
62 | tmp <- sort(bd_abs, decreasing = TRUE, index.return = TRUE)
63 | bd_sort <- tmp$x
64 | bd_index <- tmp$ix
65 |
66 | output <- bd_sort
67 | names(output) <- colnames(data)[bd_index]
68 |
69 | if(!is.null(n)) {
70 | output <- output[seq_len(n)]
71 | }
72 |
73 | return(output)
74 |
75 | }
76 |
77 |
--------------------------------------------------------------------------------
/README.Rmd:
--------------------------------------------------------------------------------
1 | ---
2 | output: github_document
3 | ---
4 |
5 |
6 |
7 | ```{r, echo = FALSE}
8 | knitr::opts_chunk$set(
9 | collapse = TRUE,
10 | comment = "#>",
11 | fig.path = "README-"
12 | )
13 | ```
14 |
15 | [](https://cran.r-project.org/package=anomalyDetection)
16 | [](https://travis-ci.org/koalaverse/anomalyDetection)
17 | [](https://ci.appveyor.com/project/bradleyboehmke/anomalyDetection)
18 | [](https://codecov.io/gh/koalaverse/anomalyDetection)
19 | [](http://cranlogs.r-pkg.org/badges/anomalyDetection)
20 | [](http://cranlogs.r-pkg.org/badges/grand-total/anomalyDetection)
21 |
22 | # anomalyDetection
23 |
24 | `anomalyDetection` implements procedures to aid in detecting network log anomalies. By combining various multivariate analytic approaches relevant to network anomaly detection, it provides cyber analysts efficient means to detect suspected anomalies requiring further evaluation.
25 |
26 |
27 | ## Installation
28 |
29 | You can install `anomalyDetection` two ways.
30 |
31 | - Using the latest released version from CRAN:
32 |
33 | ```
34 | install.packages("anomalyDetection")
35 | ```
36 |
37 | - Using the latest development version from GitHub:
38 |
39 | ```
40 | if (packageVersion("devtools") < 1.6) {
41 | install.packages("devtools")
42 | }
43 |
44 | devtools::install_github("koalaverse/anomalyDetection", build_vignettes = TRUE)
45 | ```
46 |
47 | ## Learning
48 |
49 | To get started with `anomalyDetection`, read the intro [vignette](https://cran.r-project.org/web/packages/anomalyDetection/vignettes/Introduction.html): `vignette("Introduction", package = "anomalyDetection")`. This will provide a thorough introduction to the functions provided in the package.
50 |
51 | ## References
52 |
53 | Gutierrez, R.J., Boehmke, B.C., Bauer, K.W., Saie, C.M. & Bihl, T.J. (2017) "`anomalyDetection`: Implementation of augmented network log anomaly detection procedures." The R Journal, 9(2), 354-365. [link](https://journal.r-project.org/archive/2017/RJ-2017-039/index.html)
54 |
--------------------------------------------------------------------------------
/docs/jquery.sticky-kit.min.js:
--------------------------------------------------------------------------------
1 | /*
2 | Sticky-kit v1.1.2 | WTFPL | Leaf Corcoran 2015 | http://leafo.net
3 | */
4 | (function(){var b,f;b=this.jQuery||window.jQuery;f=b(window);b.fn.stick_in_parent=function(d){var A,w,J,n,B,K,p,q,k,E,t;null==d&&(d={});t=d.sticky_class;B=d.inner_scrolling;E=d.recalc_every;k=d.parent;q=d.offset_top;p=d.spacer;w=d.bottoming;null==q&&(q=0);null==k&&(k=void 0);null==B&&(B=!0);null==t&&(t="is_stuck");A=b(document);null==w&&(w=!0);J=function(a,d,n,C,F,u,r,G){var v,H,m,D,I,c,g,x,y,z,h,l;if(!a.data("sticky_kit")){a.data("sticky_kit",!0);I=A.height();g=a.parent();null!=k&&(g=g.closest(k));
5 | if(!g.length)throw"failed to find stick parent";v=m=!1;(h=null!=p?p&&a.closest(p):b(""))&&h.css("position",a.css("position"));x=function(){var c,f,e;if(!G&&(I=A.height(),c=parseInt(g.css("border-top-width"),10),f=parseInt(g.css("padding-top"),10),d=parseInt(g.css("padding-bottom"),10),n=g.offset().top+c+f,C=g.height(),m&&(v=m=!1,null==p&&(a.insertAfter(h),h.detach()),a.css({position:"",top:"",width:"",bottom:""}).removeClass(t),e=!0),F=a.offset().top-(parseInt(a.css("margin-top"),10)||0)-q,
6 | u=a.outerHeight(!0),r=a.css("float"),h&&h.css({width:a.outerWidth(!0),height:u,display:a.css("display"),"vertical-align":a.css("vertical-align"),"float":r}),e))return l()};x();if(u!==C)return D=void 0,c=q,z=E,l=function(){var b,l,e,k;if(!G&&(e=!1,null!=z&&(--z,0>=z&&(z=E,x(),e=!0)),e||A.height()===I||x(),e=f.scrollTop(),null!=D&&(l=e-D),D=e,m?(w&&(k=e+u+c>C+n,v&&!k&&(v=!1,a.css({position:"fixed",bottom:"",top:c}).trigger("sticky_kit:unbottom"))),e
50 |
51 |
105 |
106 |
107 |
108 |
109 |
137 |
138 |
139 |
140 |
141 |
142 |
143 |
--------------------------------------------------------------------------------
/docs/pkgdown.css:
--------------------------------------------------------------------------------
1 | /* Sticky footer */
2 |
3 | /**
4 | * Basic idea: https://philipwalton.github.io/solved-by-flexbox/demos/sticky-footer/
5 | * Details: https://github.com/philipwalton/solved-by-flexbox/blob/master/assets/css/components/site.css
6 | *
7 | * .Site -> body > .container
8 | * .Site-content -> body > .container .row
9 | * .footer -> footer
10 | *
11 | * Key idea seems to be to ensure that .container and __all its parents__
12 | * have height set to 100%
13 | *
14 | */
15 |
16 | html, body {
17 | height: 100%;
18 | }
19 |
20 | body > .container {
21 | display: flex;
22 | height: 100%;
23 | flex-direction: column;
24 |
25 | padding-top: 60px;
26 | }
27 |
28 | body > .container .row {
29 | flex: 1 0 auto;
30 | }
31 |
32 | footer {
33 | margin-top: 45px;
34 | padding: 35px 0 36px;
35 | border-top: 1px solid #e5e5e5;
36 | color: #666;
37 | display: flex;
38 | flex-shrink: 0;
39 | }
40 | footer p {
41 | margin-bottom: 0;
42 | }
43 | footer div {
44 | flex: 1;
45 | }
46 | footer .pkgdown {
47 | text-align: right;
48 | }
49 | footer p {
50 | margin-bottom: 0;
51 | }
52 |
53 | img.icon {
54 | float: right;
55 | }
56 |
57 | img {
58 | max-width: 100%;
59 | }
60 |
61 | /* Typographic tweaking ---------------------------------*/
62 |
63 | .contents h1.page-header {
64 | margin-top: calc(-60px + 1em);
65 | }
66 |
67 | /* Section anchors ---------------------------------*/
68 |
69 | a.anchor {
70 | margin-left: -30px;
71 | display:inline-block;
72 | width: 30px;
73 | height: 30px;
74 | visibility: hidden;
75 |
76 | background-image: url(./link.svg);
77 | background-repeat: no-repeat;
78 | background-size: 20px 20px;
79 | background-position: center center;
80 | }
81 |
82 | .hasAnchor:hover a.anchor {
83 | visibility: visible;
84 | }
85 |
86 | @media (max-width: 767px) {
87 | .hasAnchor:hover a.anchor {
88 | visibility: hidden;
89 | }
90 | }
91 |
92 |
93 | /* Fixes for fixed navbar --------------------------*/
94 |
95 | .contents h1, .contents h2, .contents h3, .contents h4 {
96 | padding-top: 60px;
97 | margin-top: -40px;
98 | }
99 |
100 | /* Static header placement on mobile devices */
101 | @media (max-width: 767px) {
102 | .navbar-fixed-top {
103 | position: absolute;
104 | }
105 | .navbar {
106 | padding: 0;
107 | }
108 | }
109 |
110 |
111 | /* Sidebar --------------------------*/
112 |
113 | #sidebar {
114 | margin-top: 30px;
115 | }
116 | #sidebar h2 {
117 | font-size: 1.5em;
118 | margin-top: 1em;
119 | }
120 |
121 | #sidebar h2:first-child {
122 | margin-top: 0;
123 | }
124 |
125 | #sidebar .list-unstyled li {
126 | margin-bottom: 0.5em;
127 | }
128 |
129 | .orcid {
130 | height: 16px;
131 | vertical-align: middle;
132 | }
133 |
134 | /* Reference index & topics ----------------------------------------------- */
135 |
136 | .ref-index th {font-weight: normal;}
137 |
138 | .ref-index td {vertical-align: top;}
139 | .ref-index .alias {width: 40%;}
140 | .ref-index .title {width: 60%;}
141 |
142 | .ref-index .alias {width: 40%;}
143 | .ref-index .title {width: 60%;}
144 |
145 | .ref-arguments th {text-align: right; padding-right: 10px;}
146 | .ref-arguments th, .ref-arguments td {vertical-align: top;}
147 | .ref-arguments .name {width: 20%;}
148 | .ref-arguments .desc {width: 80%;}
149 |
150 | /* Nice scrolling for wide elements --------------------------------------- */
151 |
152 | table {
153 | display: block;
154 | overflow: auto;
155 | }
156 |
157 | /* Syntax highlighting ---------------------------------------------------- */
158 |
159 | pre {
160 | word-wrap: normal;
161 | word-break: normal;
162 | border: 1px solid #eee;
163 | }
164 |
165 | pre, code {
166 | background-color: #f8f8f8;
167 | color: #333;
168 | }
169 |
170 | pre code {
171 | overflow: auto;
172 | word-wrap: normal;
173 | white-space: pre;
174 | }
175 |
176 | pre .img {
177 | margin: 5px 0;
178 | }
179 |
180 | pre .img img {
181 | background-color: #fff;
182 | display: block;
183 | height: auto;
184 | }
185 |
186 | code a, pre a {
187 | color: #375f84;
188 | }
189 |
190 | a.sourceLine:hover {
191 | text-decoration: none;
192 | }
193 |
194 | .fl {color: #1514b5;}
195 | .fu {color: #000000;} /* function */
196 | .ch,.st {color: #036a07;} /* string */
197 | .kw {color: #264D66;} /* keyword */
198 | .co {color: #888888;} /* comment */
199 |
200 | .message { color: black; font-weight: bolder;}
201 | .error { color: orange; font-weight: bolder;}
202 | .warning { color: #6A0366; font-weight: bolder;}
203 |
204 | /* Clipboard --------------------------*/
205 |
206 | .hasCopyButton {
207 | position: relative;
208 | }
209 |
210 | .btn-copy-ex {
211 | position: absolute;
212 | right: 0;
213 | top: 0;
214 | visibility: hidden;
215 | }
216 |
217 | .hasCopyButton:hover button.btn-copy-ex {
218 | visibility: visible;
219 | }
220 |
221 | /* mark.js ----------------------------*/
222 |
223 | mark {
224 | background-color: rgba(255, 255, 51, 0.5);
225 | border-bottom: 2px solid rgba(255, 153, 51, 0.3);
226 | padding: 1px;
227 | }
228 |
229 | /* vertical spacing after htmlwidgets */
230 | .html-widget {
231 | margin-bottom: 10px;
232 | }
233 |
--------------------------------------------------------------------------------
/R/factor_analysis.R:
--------------------------------------------------------------------------------
1 | #' @title Factor Analysis with Varimax Rotation
2 | #'
3 | #' @description
4 | #' \code{factor_analysis} reduces the structure of the data by relating the
5 | #' correlation between variables to a set of factors, using the eigen-decomposition
6 | #' of the correlation matrix.
7 | #'
8 | #' @param data numeric data
9 | #' @param hc_points vector of eigenvalues [designed to use output from \code{\link{horns_curve}}]
10 | #'
11 | #' @return A list containing:
12 | #' \enumerate{
13 | #' \item \code{fa_loadings}: numerical matrix with the original factor loadings
14 | #' \item \code{fa_scores}: numerical matrix with the row scores for each factor
15 | #' \item \code{fa_loadings_rotated}: numerical matrix with the varimax rotated factor loadings
16 | #' \item \code{fa_scores_rotated}: numerical matrix with the row scores for each varimax rotated factor
17 | #' \item \code{num_factors}: numeric vector identifying the number of factors
18 | #' }
19 | #'
20 | #' @references
21 | #'
22 | #' H. F. Kaiser, "The Application of Electronic Computers to Factor Analysis,"
23 | #' Educational and Psychological Measurement, 1960.
24 | #'
25 | #' @seealso
26 | #'
27 | #' \code{\link{horns_curve}} for computing the average eigenvalues used for \code{hc_points} argument
28 | #'
29 | #' @examples
30 | #'
31 | #' # Perform Factor Analysis with matrix \code{x}
32 | #' x <- matrix(rnorm(200*3), ncol = 10)
33 | #'
34 | #' x %>%
35 | #' horns_curve() %>%
36 | #' factor_analysis(x, hc_points = .)
37 | #'
38 | #' @export
39 |
40 | factor_analysis <- function(data, hc_points) {
41 |
42 | # return error if parameters are missing
43 | if(missing(data)) {
44 | stop("Missing argument: data argument", call. = FALSE)
45 | }
46 | if(missing(hc_points)) {
47 | stop("Missing argument: hc_points argument", call. = FALSE)
48 | }
49 |
50 | data <- as.matrix(data)
51 | N <- nrow(data)
52 | M <- ncol(data)
53 | R <- stats::cor(data)
54 | tmp <- eigen(R)
55 | tmp2 <- sort(tmp$values, decreasing = TRUE, index.return = TRUE)
56 | eigval <- tmp2$x
57 | eigvec <- tmp$vectors[, order(tmp2$ix)]
58 |
59 | # dimensionality assessment - finds the number of factors needed to account for
60 | # the variance in the data
61 | num_factors <- sum(eigval >= hc_points)
62 |
63 | xbar <- colMeans(data)
64 | Xd <- data - matrix(1, N, 1) %*% t(xbar)
65 | v <- suppressWarnings(diag(M) * diag(1 / sqrt(stats::var(data))))
66 | Xs <- Xd %*% v
67 |
68 | eigval2 <- eigval[seq_len(num_factors)]
69 | eigvec2 <- eigvec[,seq_len(num_factors)]
70 |
71 | lambda_mat <- vapply(seq_len(num_factors), function(i) sqrt(eigval2[i]) %*% eigvec2[ , i], numeric(M))
72 |
73 | # generalized inverse is necessary to avoid matrix close to singularity
74 | fa_scores <- Xs %*% MASS::ginv(R) %*% lambda_mat
75 |
76 | rotation <- stats::varimax(lambda_mat)
77 | B <- lambda_mat %*% rotation$rotmat
78 | fa_scores_rotated <- Xs %*% MASS::ginv(R) %*% B
79 |
80 | output <- list(fa_loadings = lambda_mat,
81 | fa_scores = fa_scores,
82 | fa_loadings_rotated = B,
83 | fa_scores_rotated = fa_scores_rotated,
84 | num_factors = num_factors)
85 |
86 | return(output)
87 |
88 | }
89 |
90 |
91 | #' Easy Access to Factor Analysis Results
92 | #'
93 | #' \code{factor_analysis_result} Provides easy access to factor analysis results
94 | #'
95 | #' @param data list output from \code{factor_analysis}
96 | #' @param results factor analysis results to extract. Can use either results
97 | #' name or number (i.e. fa_scores or 2)::
98 | #' \enumerate{
99 | #' \item \code{fa_loadings} (default)
100 | #' \item \code{fa_scores}
101 | #' \item \code{fa_loadings_rotated}
102 | #' \item \code{fa_scores_rotated}
103 | #' \item \code{num_factors}
104 | #' }
105 | #'
106 | #'
107 | #' @return Returns the one of the selected results:
108 | #' \enumerate{
109 | #' \item \code{fa_loadings}: numerical matrix with the original factor loadings
110 | #' \item \code{fa_scores}: numerical matrix with the row scores for each factor
111 | #' \item \code{fa_loadings_rotated}: numerical matrix with the varimax rotated factor loadings
112 | #' \item \code{fa_scores_rotated}: numerical matrix with the row scores for each varimax rotated factor
113 | #' \item \code{num_factors}: numeric vector identifying the number of factors
114 | #' }
115 | #'
116 | #' @seealso
117 | #'
118 | #' \code{\link{factor_analysis}} for computing the factor analysis results
119 | #'
120 | #' @examples
121 | #'
122 | #' # An efficient means for getting factor analysis results
123 | #' x <- matrix(rnorm(200*3), ncol = 10)
124 | #' N <- nrow(x)
125 | #' p <- ncol(x)
126 | #'
127 | #' x %>%
128 | #' horns_curve() %>%
129 | #' factor_analysis(x, hc_points = .) %>%
130 | #' factor_analysis_results(fa_scores_rotated)
131 | #'
132 | #' @export
133 |
134 | factor_analysis_results <- function(data, results = 1) {
135 |
136 | result_input <- deparse(substitute(results))
137 | result_options <- names(data)
138 |
139 | # return error if parameters are missing
140 | if(missing(data)) {
141 | stop("Missing argument: data argument", call. = FALSE)
142 | }
143 | if(missing(result_input)) {
144 | stop("Missing argument: results argument", call. = FALSE)
145 | }
146 |
147 | if(result_input %in% as.character(1:5)) {
148 | data[[results]]
149 | } else if(result_input %in% result_options) {
150 | data[[result_input]]
151 | } else {
152 | stop("Invalid results argument: see ?factor_analysis_results for options", call. = FALSE)
153 | }
154 | }
155 |
156 |
157 |
158 |
159 |
--------------------------------------------------------------------------------
/docs/reference/anomalyDetection.html:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
9 |
110 |
125 |
126 |
136 |
111 |
124 |
112 |
114 |
115 | Articles
113 |
116 |
123 | All vignettes
117 | 118 | 119 |-
120 |
- Introduction 121 |
56 |
57 |
111 |
112 |
113 |
114 |
115 |
153 |
154 |
155 |
156 |
157 |
158 |
159 |
--------------------------------------------------------------------------------
/R/pca.R:
--------------------------------------------------------------------------------
1 | #' @title Principal Component Analysis
2 | #'
3 | #' @description
4 | #' \code{principal_components} relates the data to a set of a components through
5 | #' the eigen-decomposition of the correlation matrix, where each component explains
6 | #' some variance of the data and returns the results as an object of class prcomp.
7 | #'
8 | #' @param data numeric data.
9 | #' @param retx a logical value indicating whether the rotated variables should be returned.
10 | #' @param center a logical value indicating whether the variables should be shifted to be
11 | #' zero centered. Alternately, a vector of length equal the number of columns of x can
12 | #' be supplied. The value is passed to scale.
13 | #' @param scale. a logical value indicating whether the variables should be scaled to have
14 | #' unit variance before the analysis takes place. The default is FALSE for consistency
15 | #' with S, but in general scaling is advisable. Alternatively, a vector of length equal
16 | #' the number of columns of \code{data} can be supplied. The value is passed to scale.
17 | #' @param tol a value indicating the magnitude below which components should be omitted.
18 | #' (Components are omitted if their standard deviations are less than or equal to tol
19 | #' times the standard deviation of the first component.) With the default null setting,
20 | #' no components are omitted. Other settings for tol could be \code{tol = 0} or
21 | #' \code{tol = sqrt(.Machine$double.eps)}, which would omit essentially constant components.
22 | #' @param ... arguments passed to or from other methods.
23 | #'
24 | #' @details
25 | #'
26 | #' The calculation is done by a singular value decomposition of the (centered and
27 | #' possibly scaled) data matrix, not by using eigen on the covariance matrix.
28 | #' This is generally the preferred method for numerical accuracy
29 | #'
30 | #' @return \code{principal_components} returns a list containing the following components:
31 | #' \enumerate{
32 | #' \item \code{pca_sdev}: the standard deviations of the principal components (i.e., the square roots of the eigenvalues of the correlation matrix, though the calculation is actually done with the singular values of the data matrix).
33 | #' \item \code{pca_loadings}: the matrix of variable loadings (i.e., a matrix whose columns contain the eigenvectors).
34 | #' \item \code{pca_rotated}: if \code{retx} is \code{TRUE} the value of the rotated data (the centred (and scaled if requested) data multiplied by the rotation matrix) is returned. Hence, \code{cov(x)} is the diagonal matrix \code{diag(sdev^2)}.
35 | #' \item \code{pca_center}: the centering used
36 | #' \item \code{pca_scale}: whether scaling was used
37 | #' }
38 | #'
39 | #' @seealso
40 | #'
41 | #' \code{\link{prcomp}}, \code{\link{biplot.prcomp}}, \code{\link{screeplot}}, \code{\link{cor}},
42 | #' \code{\link{cov}}, \code{\link{svd}}, \code{\link{eigen}}
43 | #'
44 | #'
45 | #' @examples
46 | #' x <- matrix(rnorm(200 * 3), ncol = 10)
47 | #' principal_components(x)
48 | #' principal_components(x, scale = TRUE)
49 | #'
50 | #' @export
51 |
52 | principal_components <- function(data, retx = TRUE, center = TRUE, scale. = FALSE, tol = NULL, ...) {
53 |
54 | pca <- stats::prcomp(data, retx = TRUE, center = TRUE, scale. = FALSE, tol = NULL, ...)
55 | list(
56 | pca_sdev = pca[[1]],
57 | pca_loadings = pca[[2]],
58 | pca_rotated = pca[[5]],
59 | pca_center = pca[[3]],
60 | pca_scale = pca[[4]]
61 | )
62 |
63 | }
64 |
65 | #' Easy Access to Principal Component Analysis Results
66 | #'
67 | #' \code{principal_components_result} Provides easy access to principal
68 | #' component analysis results
69 | #'
70 | #' @param data list output from \code{principal_components}
71 | #' @param results principal component analysis results to extract. Can use either
72 | #' results name or number (i.e. pca_loadings or 2):
73 | #' \enumerate{
74 | #' \item \code{pca_sdev}
75 | #' \item \code{pca_loadings} (default)
76 | #' \item \code{pca_rotated}
77 | #' \item \code{pca_center}
78 | #' \item \code{pca_scale}
79 | #' }
80 | #'
81 | #'
82 | #' @return Returns one of the selected results:
83 | #' \enumerate{
84 | #' \item \code{pca_sdev}: the standard deviations of the principal components (i.e., the square roots of the eigenvalues of the correlation matrix, though the calculation is actually done with the singular values of the data matrix).
85 | #' \item \code{pca_loadings}: the matrix of variable loadings (i.e., a matrix whose columns contain the eigenvectors).
86 | #' \item \code{pca_rotated}: if \code{retx} is \code{TRUE} the value of the rotated data (the centred (and scaled if requested) data multiplied by the rotation matrix) is returned. Hence, \code{cov(x)} is the diagonal matrix \code{diag(sdev^2)}.
87 | #' \item \code{pca_center}: the centering used
88 | #' \item \code{pca_scale}: whether scaling was used
89 | #' }
90 | #'
91 | #' @seealso
92 | #'
93 | #' \code{\link{principal_components}} for computing the principal components results
94 | #'
95 | #' @examples
96 | #'
97 | #' # An efficient means for getting principal component analysis results
98 | #' x <- matrix(rnorm(200 * 3), ncol = 10)
99 | #'
100 | #' principal_components(x) %>%
101 | #' principal_components_result(pca_loadings)
102 | #'
103 | #' @export
104 |
105 | principal_components_result <- function(data, results = 2) {
106 |
107 | result_input <- deparse(substitute(results))
108 | result_options <- names(data)
109 |
110 | # return error if parameters are missing
111 | if(missing(data)) {
112 | stop("Missing argument: data argument", call. = FALSE)
113 | }
114 | if(missing(result_input)) {
115 | stop("Missing argument: results argument", call. = FALSE)
116 | }
117 |
118 | if(result_input %in% as.character(1:5)) {
119 | data[[results]]
120 | } else if(result_input %in% result_options) {
121 | data[[result_input]]
122 | } else {
123 | stop("Invalid results argument: see ?principal_components_result for options", call. = FALSE)
124 | }
125 | }
126 |
--------------------------------------------------------------------------------
/docs/authors.html:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
9 |
116 |
141 |
142 |
152 |
117 |
134 |
140 |
118 |
122 |
123 |
124 | anomalyDetection: An R package for implementing augmented network log anomoly 119 | detection procedures.
120 | Source:R/anomalyDetection.R
121 | anomalyDetection.Rd
125 |
126 |
130 |
131 |
132 |
133 | anomalyDetection: An R package for implementing augmented network log anomoly 127 | detection procedures.
128 | 129 |
50 |
51 |
105 |
106 |
107 |
108 |
109 |
169 |
170 |
171 |
172 |
173 |
174 |
175 |
--------------------------------------------------------------------------------
/docs/reference/security_logs.html:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
9 |
110 |
156 |
157 |
158 |
168 |
111 |
154 |
155 |
112 |
114 |
115 | Citation
113 |Gutierrez RJ, Boehmke BC, Bauer KW, Saie CM, Bihl TJ (2017). 116 | “anomalyDetection: Implementation of augmented network log anomaly detection procedures.” 117 | The R Journal, 9(2), 354–365. 118 | https://journal.r-project.org/archive/2017/RJ-2017-039/index.html. 119 |
120 |@Article{,
121 | title = {anomalyDetection: Implementation of augmented network log anomaly detection procedures.},
122 | author = {Robert J. Gutierrez and Bradley C. Boehmke and Kenneth W. Bauer and Cade M. Saie and Trevor J. Bihl},
123 | journal = {The R Journal},
124 | year = {2017},
125 | volume = {9},
126 | number = {2},
127 | pages = {354--365},
128 | url = {https://journal.r-project.org/archive/2017/RJ-2017-039/index.html},
129 | }
130 |
131 |
133 |
134 | Authors
132 |-
135 |
-
136 |
Bradley Boehmke. Author, maintainer.
138 |
137 |
139 | -
140 |
Brandon Greenwell. Author.
142 |
141 |
143 | -
144 |
Jason Freels. Author. 145 |
146 |
147 | -
148 |
Robert Gutierrez. Author. 149 |
150 |
151 |
53 |
54 |
108 |
109 |
110 |
111 |
112 |
166 |
167 |
168 |
169 |
170 |
171 |
172 |
--------------------------------------------------------------------------------
/docs/news/index.html:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
9 |
113 |
154 |
155 |
165 |
114 |
119 |
120 |
145 |
153 |
121 |
122 |
125 |
126 | A mock dataset containing common information that appears in security logs.
123 | 124 |security_logs
127 |
128 | Format
129 | 130 |A data frame with 300 rows and 10 variables:
-
131 |
- Device_Vendor
Company who made the device
132 | - Device_Product
Name of the security device
133 | - Device_Action
Outcome result of access
134 | - Src_IP
IP address of the source
135 | - Dst_IP
IP address of the destination
136 | - Src_Port
Port identifier of the source
137 | - Dst_Port
Port identifier of the destination
138 | - Protocol
Transport protocol used
139 | - Country_Src
Country of the source
140 | - Bytes_TRF
Number of bytes transferred
141 |
50 |
51 |
105 |
106 |
107 |
108 |
109 |
177 |
178 |
179 |
180 |
181 |
182 |
183 |
--------------------------------------------------------------------------------
/docs/reference/get_all_factors.html:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
9 |
110 |
165 |
166 |
176 |
111 |
152 |
153 |
163 |
164 |
112 |
115 |
116 | Changelog
113 | Source:NEWS.md
114 |
117 |
124 | 118 | anomalyDetection 0.2.6 Unreleased 119 |
120 |-
121 |
- Tested and fixed any issues with new tibble version dependency 122 |
125 |
136 | 126 | anomalyDetection 0.2.5 2018-03-07 127 |
128 |-
129 |
- Tested and fixed any issues with new tidyr version dependency 130 |
- Updated maintainership and active authors 131 |
- Updated URL for new Github organization holding this package 132 |
- Updated vignette package loading in vignette to resolve 1 note 133 |
- Added reference in the Description field 134 |
137 |
151 | 138 | anomalyDetection 0.2.4 2017-09-22 139 |
140 |-
141 |
- Added
NEWSfile.
142 | - Better tolerance in
mahalanobis_distancewhen inverting covariance matrices.
143 | -
144 |
mahalanobis_distanceandhorns_curvehave been rewritten in C++ using theRcppArmadillopackage. This greatly improved the speed (and accuracy) of these functions.
145 | -
146 |
tabulate_state_vectorhas been rewritten using thedplyrpackage, greatly improving the speed of this function. Greater traceability is now also present for missing values and numeric variables.
147 | - Producing histogram matrix to visually display anomalous blocks made easier with addition of
hmatfunction
148 | - Properly registered native routines and disabled symbol search. 149 |
54 |
55 |
109 |
110 |
111 |
112 |
113 |
184 |
185 |
186 |
187 |
188 |
189 |
190 |
--------------------------------------------------------------------------------
/R/hmat.R:
--------------------------------------------------------------------------------
1 | #' @title Plot a Histogram Matrix
2 | #'
3 | #' @description Display a histogram matrix for visual inspection of anomalous
4 | #' observation detection. The color of the blocks represents how anomalous each
5 | #' block is, where a lighter blue represents a more anomalous block. The size
6 | #' of the points indicate which values are driving the anomaly, with larger
7 | #' blocks representing more anomalous values.
8 | #'
9 | #' @param data the data set (data frame or matrix)
10 | #' @param input the type of input data being passed to the function. \code{data} for
11 | #' a raw categorical data set, \code{SV} for a state vector input, and \code{MD} if the
12 | #' input has already had the Mahalanobis distances calculated
13 | #' @param top how many of the most anomalous blocks you would like to display
14 | #' (default 20)
15 | #' @param order whether to show the anomalous blocks in numeric order or in order of
16 | #' most anomalous to least anomalous (default is "numeric", other choice is "anomaly")
17 | #' @param block_length argument fed into \code{tabulate_state_vector}, necessary if
18 | #' \code{input = data}
19 | #' @param level_limit argument fed into \code{tabulate_state_vector}, if the
20 | #' number of unique categories for a variable exceeds this number, only keep
21 | #' a limited number of the most popular values (default 50)
22 | #' @param level_keep argument fed into \code{tabulate_state_vector}, if \code{level_limit}
23 | #' is exceeded, keep this many of the most popular values (default 10)
24 | #' @param partial_block argument fed into \code{tabulate_state_vector}, if the number of
25 | #' entries is not divisible by the \code{block_length}, this logical decides
26 | #' whether to keep the smaller last block (default \code{TRUE})
27 | #' @param na.rm whether to keep track of missing values as part of the analysis or
28 | #' ignore them (default \code{FALSE})
29 | #' @param min_var argument fed into \code{mc_adjust}, if a column in the state
30 | #' vector has variance less than this value, remove it (default 0.1)
31 | #' @param max_cor argument fed into \code{mc_adjust}, if a column in the state
32 | #' vector has correlation greater than this value, remove it (default 0.9)
33 | #' @param action argument fed into \code{mc_adjust}, if a column does not fall in
34 | #' the specified range, determine what to do with it (default "exclude")
35 | #' @param output argument fed into \code{mahalanobis_distance} that decides
36 | #' whether to add a column for the Mahalanobis Distance ('MD'), the breakdown
37 | #' distances ('BD') or both (default "both")
38 | #' @param normalize argument fed into \code{mahalanobis_distance} that decides
39 | #' whether to normalize the values by column (default = FALSE)
40 | #'
41 | #' @examples
42 | #' \dontrun{
43 | #' # Data set input
44 | #' hmat(security_logs,block_length = 8)
45 | #'
46 | #' # Data Set input with top 10 blocks displayed
47 | #' hmat(security_logs, top = 10, block_length = 5)
48 | #'
49 | #' # State Vector Input
50 | #' tabulate_state_vector(security_logs, block_length = 6, level_limit = 20) %>%
51 | #' hmat(input = "SV")
52 | #' }
53 | #'
54 | #' @export
55 |
56 | hmat <- function(data, input = "data", top = 20, order = "numeric", block_length = NULL,
57 | level_limit = 50, level_keep = 10, partial_block = TRUE, na.rm = FALSE,
58 | min_var = 0.1, max_cor = 0.9, action = "exclude",
59 | output = "both", normalize = FALSE) {
60 |
61 | # if the input is a vector or NULL, throw a warning
62 | if (is.vector(data)) {
63 | stop("data must be a matrix or data frame")
64 | }
65 | if (is.null(data)) {
66 | stop("data is NULL")
67 | }
68 |
69 | # if the order input is not "numeric" or "anomaly", stop
70 | if (order != "numeric" & order != "anomaly") {
71 | stop("Invalid Input: argument order must be either 'numeric' or 'anomaly'")
72 | }
73 |
74 |
75 | # if the input is a raw data set, not a state vector
76 | if (input == "data") {
77 |
78 | if(is.null(block_length)) {
79 | stop("Please supply a block_length argument")
80 | }
81 |
82 | suppressWarnings(
83 | suppressMessages(
84 | temp <- dplyr::mutate_(
85 | tibble::as.tibble(
86 | anomalyDetection::mahalanobis_distance(
87 | anomalyDetection::mc_adjust(
88 | anomalyDetection::tabulate_state_vector(data,block_length,
89 | level_limit,
90 | level_keep,
91 | partial_block,
92 | na.rm)
93 | ,min_var, max_cor, action)
94 | ,output, normalize)
95 | )
96 | ,.dots = list(Block = quote(as.factor(1:n()))))
97 | )
98 | )
99 | temp$Ranked <- rank(dplyr::desc(temp$MD), ties.method = "random")
100 | temp <- dplyr::filter_(
101 | tidyr::gather_(temp,"Variable","BD",names(temp)[-c(1,length(names(temp))-1,length(names(temp)))])
102 | ,.dots = ~ Ranked <= top)
103 | temp$Variable <- substr(temp$Variable,1,nchar(temp$Variable)-3)
104 | if (order == "anomaly") {
105 | temp$Block <- stats::reorder(temp$Block, temp$Ranked)
106 | }
107 | return(
108 | ggplot2::ggplot(temp,
109 | ggplot2::aes_string(x = "Block", y = "Variable",
110 | color = "MD", size = "BD")) +
111 | ggplot2::geom_point()
112 | )
113 |
114 |
115 | } else if (input == "SV") {
116 |
117 | suppressWarnings(
118 | suppressMessages(
119 | temp <- dplyr::mutate_(
120 | tibble::as.tibble(
121 | anomalyDetection::mahalanobis_distance(
122 | anomalyDetection::mc_adjust(data,min_var, max_cor, action)
123 | ,output, normalize)
124 | )
125 | ,.dots = list(Block = quote(as.factor(1:n()))))
126 | )
127 | )
128 | temp$Ranked <- rank(dplyr::desc(temp$MD), ties.method = "random")
129 | temp <- dplyr::filter_(
130 | tidyr::gather_(temp,"Variable","BD",names(temp)[-c(1,length(names(temp))-1,length(names(temp)))])
131 | ,.dots = ~ Ranked <= top)
132 | temp$Variable <- substr(temp$Variable,1,nchar(temp$Variable)-3)
133 | if (order == "anomaly") {
134 | temp$Block <- stats::reorder(temp$Block, temp$Ranked)
135 | }
136 | return(
137 | ggplot2::ggplot(temp,
138 | ggplot2::aes_string(x = "Block", y = "Variable",
139 | color = "MD", size = "BD")) +
140 | ggplot2::geom_point()
141 | )
142 |
143 | } else if (input == "MD") {
144 |
145 | suppressWarnings(
146 | suppressMessages(
147 | temp <- dplyr::mutate_(
148 | tibble::as.tibble(data)
149 | ,.dots = list(Block = quote(as.factor(1:n()))))
150 | )
151 | )
152 | temp$Ranked <- rank(dplyr::desc(temp$MD), ties.method = "random")
153 | temp <- dplyr::filter_(
154 | tidyr::gather_(temp,"Variable","BD",names(temp)[-c(1,length(names(temp))-1,length(names(temp)))])
155 | ,.dots = ~ Ranked <= top)
156 | temp$Variable <- substr(temp$Variable,1,nchar(temp$Variable)-3)
157 | if (order == "anomaly") {
158 | temp$Block <- stats::reorder(temp$Block, temp$Ranked)
159 | }
160 | return(
161 | ggplot2::ggplot(temp,
162 | ggplot2::aes_string(x = "Block", y = "Variable",
163 | color = "MD", size = "BD")) +
164 | ggplot2::geom_point()
165 | )
166 |
167 | } else {
168 |
169 | stop("Invalid Input: input should be set as 'data','SV', or 'MD'.")
170 |
171 | }
172 | }
173 |
--------------------------------------------------------------------------------
/R/tabulate_state_vector.R:
--------------------------------------------------------------------------------
1 | #' @title Tabulate State Vector
2 | #'
3 | #' @description
4 | #' \code{tabulate_state_vector} employs a tabulated vector approach to transform
5 | #' security log data into unique counts of data attributes based on time blocks.
6 | #' Taking a contingency table approach, this function separates variables of type
7 | #' character or factor into their unique levels and counts the number of occurrences
8 | #' for those levels within each block. Due to the large number of unique IP addresses,
9 | #' this function allows for the user to determine how many IP addresses they would
10 | #' like to investigate. The function tabulates the most popular IP addresses.
11 | #'
12 | #' @param data data
13 | #' @param block_length integer value to divide data by
14 | #' @param level_limit integer value to determine the cutoff for the number of
15 | #' factors in a column to display before being reduced to show the number of
16 | #' levels to keep (default is 50)
17 | #' @param level_keep integer value indicating the top number of factor levels to
18 | #' retain if a column has more than the level limit (default is 10)
19 | #' @param partial_block a logical which determines whether incomplete blocks are kept in
20 | #' the analysis in the case where the number of log entries isn't evenly
21 | #' divisible by the \code{block_length}
22 | #' @param na.rm whether to keep track of missing values as part of the analysis or
23 | #' ignore them
24 | #'
25 | #' @return A data frame where each row represents one block and the columns count
26 | #' the number of occurrences that character/factor level occurred in that block
27 | #'
28 | #' @examples
29 | #' tabulate_state_vector(security_logs, 30)
30 | #'
31 | #' @export
32 |
33 | tabulate_state_vector <- function(data, block_length, level_limit = 50L,
34 | level_keep = 10L, partial_block = FALSE,
35 | na.rm = FALSE) {
36 |
37 | # return error if parameters are missing
38 | if(missing(data)) {
39 | stop("Missing argument: data argument", call. = FALSE)
40 | }
41 | if(missing(block_length)) {
42 | stop("Missing argument: block_length argument", call. = FALSE)
43 | }
44 |
45 | # return error if arguments are wrong type
46 | if(!is.data.frame(data) & !is.matrix(data)) {
47 | stop("data must be a data frame or matrix", call. = FALSE)
48 | }
49 | if(is.null(nrow(data)) | isTRUE(nrow(data) < block_length)) {
50 | stop("Your data input does not have sufficient number of rows",
51 | call. = FALSE)
52 | }
53 | if(!is.numeric(block_length)) {
54 | stop("block_length must be a numeric input", call. = FALSE)
55 | }
56 | if(!is.numeric(level_limit)) {
57 | stop("level_limit must be a numeric input", call. = FALSE)
58 | }
59 | if(!is.numeric(level_keep)) {
60 | stop("level_keep must be a numeric input", call. = FALSE)
61 | }
62 | if(!is.logical(partial_block)) {
63 | stop("partial_block must be a logical input", call. = FALSE)
64 | }
65 | if(!is.logical(na.rm)) {
66 | stop("na.rm must be a logical input", call. = FALSE)
67 | }
68 |
69 | # Calculate the number of leftover log entries
70 | left <- nrow(data) %% block_length
71 |
72 | # Calculate number of blocks necessary and throw a warning if
73 | # observations are removed
74 | if (partial_block == TRUE) {
75 | nblocks <- ceiling(nrow(data)/block_length)
76 | if (left != 0) {
77 | message(paste("The last block contains only",as.character(left),
78 | "observations"))
79 | }
80 | } else {
81 | nblocks <- floor(nrow(data)/block_length)
82 | data <- data[1:(nrow(data)-left),]
83 | if (left != 0) {
84 | message(paste("You have removed the last",as.character(left),
85 | "observations from the data set"))
86 | }
87 | }
88 |
89 | # Determine which variables have more categories than level_limit
90 | counts <- purrr::map_dbl(purrr::map(data,unique),length)
91 | overflowvars <- names(subset(counts,counts > level_limit))
92 |
93 | # Only evaluate if there is an issue with overflow variables
94 | if (length(overflowvars) > 0) {
95 |
96 | # Throw warning indicating there is overflow
97 | message("Some variables contain more than ",as.character(level_limit),
98 | " levels. Only the ",as.character(level_keep)," most popular",
99 | " levels of these variables will be tabulated.")
100 |
101 | # Find and store 10 most popular levels of each overflow variable
102 | suppressMessages(
103 | popvars <- dplyr::select_(
104 | tidyr::spread_(
105 | dplyr::mutate_(
106 | dplyr::group_by_(
107 | dplyr::top_n(
108 | dplyr::count_(
109 | dplyr::group_by_(
110 | dplyr::filter_(
111 | stats::na.omit(
112 | tidyr::gather_(
113 | dplyr::select_(data, ~overflowvars)
114 | ,"Variables","Values",overflowvars)
115 | )
116 | ,.dots = ~ substr(Values,1,1) != "0")
117 | ,"Variables")
118 | ,"Values")
119 | ,level_keep)
120 | ,"Variables")
121 | ,.dots = list(n = quote(1:n())))
122 | ,"Variables","Values")
123 | , ~overflowvars)
124 | )
125 |
126 | # replace unpopular categories with "0"
127 | suppressWarnings(
128 | for (i in overflowvars) {
129 | data[[i]][!is.na(data[[i]]) & !(data[[i]] %in% popvars[[i]])] <- "0"
130 | }
131 | )
132 | }
133 |
134 |
135 | # construct state vector
136 | if (na.rm == TRUE) {
137 |
138 | # Remove NAs
139 | temp <- dplyr::mutate_(data, .dots = list(BLK = quote(as.numeric(floor((1:n()-1)/block_length)+1))))
140 | temp <- tidyr::spread_(
141 | tibble::as.tibble(
142 | base::table(
143 | dplyr::group_by_(
144 | dplyr::select_(
145 | dplyr::mutate_(
146 | dplyr::filter_(
147 | stats::na.omit(
148 | tidyr::gather_(temp,"Variables","Values",names(temp)[-length(names(temp))])
149 | )
150 | ,.dots = ~ Values != "0")
151 | ,Values = ~ dplyr::if_else(grepl("[A-Za-z]", Values),
152 | Values,
153 | paste0(Variables,"_",Values)))
154 | ,~ c(BLK,Values))
155 | ,"BLK")
156 | )
157 | )
158 | ,"Values","n")
159 | empty <- as.data.frame(matrix(0,nrow = (nblocks - nrow(temp)),
160 | ncol = ncol(temp)))
161 | colnames(empty) <- colnames(temp)
162 | empty$BLK <- setdiff(1:nblocks,temp$BLK)
163 | temp <- dplyr::arrange_(rbind(temp,empty),.dots = ~ as.numeric(BLK))
164 | return(
165 | tibble::as.tibble(
166 | purrr::map_df(
167 | dplyr::select_(temp, ~ names(temp)[-1])
168 | ,base::as.numeric)
169 | )
170 | )
171 |
172 |
173 |
174 | } else {
175 |
176 | # Do not remove NAs
177 | temp <- dplyr::mutate_(data, .dots = list(BLK = quote(as.numeric(floor((1:n()-1)/block_length)+1))))
178 | temp <- tidyr::spread_(
179 | tibble::as.tibble(
180 | base::table(
181 | dplyr::group_by_(
182 | dplyr::select_(
183 | dplyr::mutate_(
184 | dplyr::filter_(
185 | dplyr::mutate_(
186 | tidyr::gather_(temp,"Variables","Values",names(temp)[-length(names(temp))])
187 | , Values = ~ dplyr::if_else(is.na(Values),
188 | paste0(Variables,"_NA"),
189 | Values))
190 | ,.dots = ~ Values != "0")
191 | ,Values = ~ dplyr::if_else(grepl("[A-Za-z]", Values),
192 | Values,
193 | paste0(Variables,"_",Values)))
194 | ,~ c(BLK,Values))
195 | ,"BLK")
196 | )
197 | )
198 | ,"Values","n")
199 | empty <- as.data.frame(matrix(0,nrow = (nblocks - nrow(temp)),
200 | ncol = ncol(temp)))
201 | colnames(empty) <- colnames(temp)
202 | empty$BLK <- setdiff(1:nblocks,temp$BLK)
203 | temp <- dplyr::arrange_(rbind(temp,empty),.dots = ~ as.numeric(BLK))
204 | return(
205 | tibble::as.tibble(
206 | purrr::map_df(
207 | dplyr::select_(temp, ~ names(temp)[-1])
208 | ,base::as.numeric)
209 | )
210 | )
211 |
212 | }
213 |
214 | }
215 |
--------------------------------------------------------------------------------
/docs/reference/horns_curve.html:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
9 |
114 |
172 |
173 |
183 |
115 |
120 |
121 |
132 |
138 |
139 |
158 |
171 |
122 |
123 |
127 |
128 | get_all_factors finds all factor pairs for a given integer (i.e. a number
124 | that divides evenly into another number).
get_all_factors(n)129 | 130 |
Arguments
131 || n | 135 |number to be factored |
136 |
|---|
Source
140 | 141 |http://stackoverflow.com/a/6425597/3851274
142 | 143 |Value
144 | 145 |A list containing the integer vector(s) containing all factors for the given
146 | n inputs.
Examples
150 |157 |151 | # Find all the factors of 39304 152 | get_all_factors(39304)#> $`39304` 153 | #> [1] 1 2 4 8 17 34 68 136 289 578 1156 2312 154 | #> [13] 4913 9826 19652 39304 155 | #>156 |
55 |
56 |
110 |
111 |
112 |
113 |
114 |
199 |
200 |
201 |
202 |
203 |
204 |
205 |
--------------------------------------------------------------------------------
/docs/reference/index.html:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
9 |
115 |
187 |
188 |
198 |
116 |
121 |
122 |
134 |
153 |
154 |
173 |
186 |
123 |
124 |
129 |
130 | Computes the average eigenvalues produced by a Monte Carlo simulation that
125 | randomly generates a large number of nxp matrices of standard
126 | normal deviates.
horns_curve(data, n, p, nsim = 1000L)131 | 132 |
Arguments
133 || data | 137 |A matrix or data frame. |
138 |
|---|---|
| n | 141 |Integer specifying the number of rows. |
142 |
| p | 145 |Integer specifying the number of columns. |
146 |
| nsim | 149 |Integer specifying the number of Monte Carlo simulations to run.
150 | Default is |
151 |
Value
155 | 156 |A vector of length p containing the averaged eigenvalues. The
157 | values can then be plotted or compared to the true eigenvalues from a dataset
158 | for a dimensionality reduction assessment.
References
161 | 162 |J. L. Horn, "A rationale and test for the number of factors in factor 163 | analysis," Psychometrika, vol. 30, no. 2, pp. 179-185, 1965.
164 | 165 | 166 |Examples
167 |172 |# Perform Horn's Parallel analysis with matrix n x p dimensions 168 | x <- matrix(rnorm(200 * 10), ncol = 10) 169 | horns_curve(x)#> [1] 1.4101301 1.2833549 1.1824088 1.0973557 1.0211361 0.9482947 0.8806428 170 | #> [8] 0.8098271 0.7381270 0.6545347horns_curve(n = 200, p = 10)#> [1] 1.4106276 1.2772393 1.1798234 1.0963995 1.0196483 0.9475956 0.8772685 171 | #> [8] 0.8093003 0.7386109 0.6543157plot(horns_curve(x)) # scree plot
50 |
51 |
105 |
106 |
107 |
108 |
109 |
251 |
252 |
253 |
254 |
255 |
256 |
257 |
--------------------------------------------------------------------------------
/docs/reference/mc_adjust.html:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
9 |
110 |
239 |
240 |
250 |
111 |
231 |
232 |
238 |
112 |
114 |
115 | Reference
113 |
53 |
54 |
108 |
109 |
110 |
111 |
112 |
204 |
205 |
206 |
207 |
208 |
209 |
210 |
--------------------------------------------------------------------------------
/docs/reference/bd_row.html:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
9 |
113 |
192 |
193 |
203 |
114 |
119 |
120 |
130 |
152 |
153 |
178 |
191 |
121 |
122 |
125 |
126 | mc_adjust handles issues with multi-collinearity.
mc_adjust(data, min_var = 0.1, max_cor = 0.9, action = "exclude")127 | 128 |
Arguments
129 || data | 133 |named numeric data object (either data frame or matrix) |
134 |
|---|---|
| min_var | 137 |numeric value between 0-1 for the minimum acceptable variance (default = 0.1) |
138 |
| max_cor | 141 |numeric value between 0-1 for the maximum acceptable correlation (default = 0.9) |
142 |
| action | 145 |select action for handling columns causing multi-collinearity issues
|
150 |
Value
154 | 155 |mc_adjust returns the numeric data object supplied minus variables
156 | violating the minimum acceptable variance (min_var) and the
157 | maximum acceptable correlation (max_cor) levels.
Details
160 | 161 |mc_adjust handles issues with multi-collinearity by first removing
162 | any columns whose variance is close to or less than min_var. Then, it
163 | removes linearly dependent columns. Finally, it removes any columns that have
164 | a high absolute correlation value equal to or greater than max_cor.
Examples
168 |# NOT RUN { 169 | x <- matrix(runif(100), ncol = 10) 170 | x %>% 171 | mc_adjust() 172 | 173 | x %>% 174 | mc_adjust(min_var = .15, max_cor = .75, action = "select") 175 | # }177 |176 |
54 |
55 |
109 |
110 |
111 |
112 |
113 |
204 |
205 |
206 |
207 |
208 |
209 |
210 |
--------------------------------------------------------------------------------
/docs/reference/kaisers_index.html:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
9 |
114 |
192 |
193 |
203 |
115 |
120 |
121 |
132 |
148 |
149 |
157 |
158 |
159 |
178 |
191 |
122 |
123 |
127 |
128 | bd_row indicates which variables in data are driving the Mahalanobis
124 | distance for a specific row r, relative to the mean vector of the data.
bd_row(data, row, n = NULL)129 | 130 |
Arguments
131 || data | 135 |numeric data |
136 |
|---|---|
| row | 139 |row of interest |
140 |
| n | 143 |number of values to return. By default, will return all variables
144 | (columns) with their respective differences. However, you can choose to view
145 | only the top |
146 |
Value
150 | 151 |Returns a vector indicating the variables in data that are driving the
152 | Mahalanobis distance for the respective row.
See also
155 | 156 |mahalanobis_distance for computing the Mahalanobis Distance values
Examples
160 |# NOT RUN { 161 | x = matrix(rnorm(200*3), ncol = 10) 162 | colnames(x) = paste0("C", 1:ncol(x)) 163 | 164 | # compute the relative differences for row 5 and return all variables 165 | x %>% 166 | mahalanobis_distance("bd", normalize = TRUE) %>% 167 | bd_row(5) 168 | 169 | # compute the relative differences for row 5 and return the top 3 variables 170 | # that are influencing the Mahalanobis Distance the most 171 | x %>% 172 | mahalanobis_distance("bd", normalize = TRUE) %>% 173 | bd_row(5, 3) 174 | 175 | # }177 |176 |
67 |
68 |
122 |
123 |
124 |
125 |
126 |
217 |
218 |
219 |
220 |
221 |
222 |
223 |
--------------------------------------------------------------------------------
/docs/index.html:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
127 |
205 |
206 |
216 |
128 |
158 |
164 |
165 |
176 |
177 |
178 |
189 |
204 |
129 |
132 |
133 |
134 | Kaiser's Index of Factorial Simplicity
130 | Source:R/kaisers_index.R
131 | kaisers_index.Rd
135 |
136 |
153 |
154 | kaisers_index computes scores designed to assess the quality of a factor
137 | analysis solution. It measures the tendency towards unifactoriality for both
138 | a given row and the entire matrix as a whole. Kaiser proposed the evaluations
139 | of the score shown below:
-
141 |
In the .90s: Marvelous
142 | In the .80s: Meritorious
143 | In the .70s: Middling
144 | In the .60s: Mediocre
145 | In the .50s: Miserable
146 | < .50: Unacceptable
147 |
Use as basis for selecting original or rotated loadings/scores in
150 | factor_analysis.
kaisers_index(loadings)155 | 156 |
Arguments
157 || loadings | 161 |numerical matrix of the factor loadings |
162 |
|---|
Value
166 | 167 |Vector containing the computed score
168 | 169 |References
170 | 171 |H. F. Kaiser, "An index of factorial simplicity," Psychometrika, vol. 39, no. 1, pp. 31-36, 1974.
172 | 173 |See also
174 | 175 |factor_analysis for computing the factor analysis loadings
Examples
179 |188 |# Perform Factor Analysis with matrix \code{x} 180 | x <- matrix(rnorm(200*3), ncol = 10) 181 | 182 | x %>% 183 | horns_curve() %>% 184 | factor_analysis(x, hc_points = .) %>% 185 | factor_analysis_results(fa_loadings_rotated) %>% 186 | kaisers_index()#> [1] 0.8162322187 |
25 |
79 |
80 |
81 |
82 |
185 |
186 |
187 |
188 |
189 |
190 |
--------------------------------------------------------------------------------
83 |
173 |
174 |
175 |
184 |
84 |
85 |
86 |
87 |
88 |
89 |
90 |
124 |
125 |
171 |
172 |
91 |
94 |
123 | 
93 | anomalyDetection implements procedures to aid in detecting network log anomalies. By combining various multivariate analytic approaches relevant to network anomaly detection, it provides cyber analysts efficient means to detect suspected anomalies requiring further evaluation.
96 |
112 | 97 | Installation
98 |You can install anomalyDetection two ways.
-
100 |
- Using the latest released version from CRAN: 101 |
install.packages("anomalyDetection")-
104 |
- Using the latest development version from GitHub: 105 |
if (packageVersion("devtools") < 1.6) {
107 | install.packages("devtools")
108 | }
109 |
110 | devtools::install_github("koalaverse/anomalyDetection", build_vignettes = TRUE)
113 |
117 | 114 | Learning
115 |To get started with anomalyDetection, read the intro vignette: vignette("Introduction", package = "anomalyDetection"). This will provide a thorough introduction to the functions provided in the package.
118 |
122 | 119 | References
120 |Gutierrez, R.J., Boehmke, B.C., Bauer, K.W., Saie, C.M. & Bihl, T.J. (2017) “anomalyDetection: Implementation of augmented network log anomaly detection procedures.” The R Journal, 9(2), 354-365. link
