├── .gitignore
├── README.md
├── data
├── groc.arules.tsv
├── groc.arules.xml
├── iris.glm.tsv
├── iris.glm.xml
├── iris.hc.tsv
├── iris.hc.xml
├── iris.kmeans.tsv
├── iris.kmeans.xml
├── iris.lm_p.tsv
├── iris.lm_p.xml
├── iris.multinom.tsv
├── iris.multinom.xml
├── iris.nn.tsv
├── iris.nn.xml
├── iris.rf.tsv
├── iris.rf.xml
├── iris.rpart.tsv
├── iris.rpart.xml
├── iris.svm.tsv
├── iris.svm.xml
├── orders.tsv
├── sample.rf.xml
└── sample.tsv
├── pom.xml
└── src
├── main
└── java
│ └── pattern
│ ├── Classifier.java
│ ├── ClassifierFunction.java
│ ├── ClassifierSplitFunction.java
│ ├── PMML.java
│ ├── PatternException.java
│ ├── Schema.java
│ ├── XPathReader.java
│ ├── datafield
│ ├── CategoricalDataField.java
│ ├── ContinuousDataField.java
│ ├── DataField.java
│ ├── DataFieldFactory.java
│ └── PortableBitSet.java
│ ├── model
│ ├── MiningModel.java
│ ├── Model.java
│ ├── clust
│ │ ├── ClusteringModel.java
│ │ └── Exemplar.java
│ ├── glm
│ │ ├── GeneralizedRegressionModel.java
│ │ ├── LinkFunction.java
│ │ ├── PCell.java
│ │ ├── PPCell.java
│ │ ├── PPMatrix.java
│ │ └── ParamMatrix.java
│ ├── lm
│ │ └── RegressionModel.java
│ └── tree
│ │ ├── Context.java
│ │ ├── Edge.java
│ │ ├── Tree.java
│ │ ├── TreeModel.java
│ │ └── Vertex.java
│ └── predictor
│ ├── CategoricalPredictor.java
│ ├── NumericPredictor.java
│ ├── Predictor.java
│ └── PredictorFactory.java
├── py
├── gen_orders.py
└── rf_eval.py
├── r
├── pmml_models.R
├── rattle_pmml.R
└── rf_pmml.R
└── test
└── java
└── pattern
└── model
├── KMeansTest.java
├── ModelTest.java
└── RandomForestTest.java
/.gitignore:
--------------------------------------------------------------------------------
1 | build/
2 | dot/
3 | out/
4 | misc/
5 | .gradle
6 | Rplots.pdf
7 | *.class
8 | *.log
9 | *~
10 |
11 | # IntelliJ Files #
12 | *.ipr
13 | *.iml
14 | *.iws
15 |
16 | # Package Files #
17 | *.jar
18 | *.war
19 | *.ear
20 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | #Storm.pattern
2 |
3 | This project is based on the cascading.pattern project. The pattern sub-project for http://Cascading.org/ which uses flows as
4 | containers for machine learning models, importing [PMML](http://en.wikipedia.org/wiki/Predictive_Model_Markup_Language) model descriptions from _R_, _SAS_, _Weka_, _RapidMiner_, _KNIME_,
5 | _SQL Server_, etc.
6 |
7 | All Credit to Chris and Paco for the excellent work!
8 |
9 | Current support for PMML includes:
10 |
11 | * [Random Forest](http://en.wikipedia.org/wiki/Random_forest) in [PMML 4.0+](http://www.dmg.org/v4-0-1/MultipleModels.html) exported from [R/Rattle](http://cran.r-project.org/web/packages/rattle/index.html)
12 | * [Linear Regression](http://en.wikipedia.org/wiki/Linear_regression) in [PMML 1.1+](http://www.dmg.org/v1-1/generalregression.html)
13 | * [Hierarchical Clustering](http://en.wikipedia.org/wiki/Hierarchical_clustering) and [K-Means Clustering](http://en.wikipedia.org/wiki/K-means_clustering) in [PMML 2.0+](http://www.dmg.org/v2-0/ClusteringModel.html)
14 | * [Logistic Regression](http://en.wikipedia.org/wiki/Logistic_regression) in [PMML 4.0.1+](http://www.dmg.org/v4-0-1/Regression.html)
15 |
16 |
17 | Use in Storm Topology
18 | ---------------------
19 |
20 | First include the clojars repo in your POM (or project.clj or sbt or wherever):
21 |
22 |
23 |
24 | clojars.org
25 | http://clojars.org/repo
26 |
27 |
28 |
29 | And then add the dependency:
30 |
31 |
32 | com.github.quintona
33 | storm-pattern
34 | 0.0.2-SNAPSHOT
35 |
36 |
37 | I have created a very simple trident topology to illustrate the usage, it is available from [here.](https://github.com/quintona/pattern-demo-topology). At a high level, this is all that is required:
38 |
39 | topology.newStream("valueStream", spout)
40 | .each(new Fields(fields), new ClassifierFunction(pmml_file),
41 | new Fields("prediction"))
42 | .each(new Fields("prediction"), new PrintlnFunction(),
43 | new Fields());
44 |
45 | You simply need to create the Classifier function and pass in the model.
46 |
47 | ## Build Instructions (if you are extending storm-pattern)
48 | ------------------
49 | To build and then run its unit tests:
50 |
51 | mvn clean install
52 |
53 | The following scripts generate a baseline (model+data) for the _Random
54 | Forest_ algorithm. This baseline includes a reference data set --
55 | 1000 independent variables, 500 rows of simulated ecommerce orders --
56 | plus a predictive model in PMML:
57 |
58 | ./src/py/gen_orders.py 500 1000 > orders.tsv
59 | R --vanilla < ./src/r/rf_pmml.R > model.log
60 |
61 | This will generate `huge.rf.xml` as the PMML export for a Random
62 | Forest classifier plus `huge.tsv` as a baseline data set for
63 | regression testing.
64 |
65 | Example Models
66 | --------------
67 | Check the `src/r/rattle_pmml.R` script for examples of predictive
68 | models which are created in R, then exported using _Rattle_.
69 | These examples use the popular
70 | [Iris](http://en.wikipedia.org/wiki/Iris_flower_data_set) data set.
71 |
72 | * random forest (rf)
73 | * linear regression (lm)
74 | * hierarchical clustering (hclust)
75 | * k-means clustering (kmeans)
76 | * logistic regression (glm)
77 | * multinomial model (multinom)
78 | * single hidden-layer neural network (nnet)
79 | * support vector machine (ksvm)
80 | * recursive partition classification tree (rpart)
81 | * association rules
82 |
83 | To execute the R script:
84 |
85 | R --vanilla < src/r/rattle_pmml.R
86 |
87 | It is possible to extend PMML support for other kinds of modeling in R
88 | and other analytics platforms. Contact the developers to discuss on
89 | the [cascading-user](https://groups.google.com/forum/?fromgroups#!forum/cascading-user)
90 | email forum.
91 |
92 |
93 | PMML Resources
94 | --------------
95 | * [Data Mining Group](http://www.dmg.org/) XML standards and supported vendors
96 | * [PMML In Action](http://www.amazon.com/dp/1470003244) book
97 | * [PMML validator](http://www.zementis.com/pmml_tools.htm)
98 |
--------------------------------------------------------------------------------
/data/groc.arules.xml:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 |
6 | 2013-01-25 10:01:55
7 |
8 |
9 |
10 |
11 |
12 |
13 |
14 |
15 |
16 |
17 |