├── Calaca
├── CHANGELOG.md
├── LICENSE
├── README.md
├── _site
│ ├── bower.json
│ ├── bower_components
│ │ ├── angular-animate
│ │ │ ├── .bower.json
│ │ │ ├── README.md
│ │ │ ├── angular-animate.min.js
│ │ │ ├── angular-animate.min.js.map
│ │ │ ├── bower.json
│ │ │ └── package.json
│ │ ├── angular
│ │ │ ├── .bower.json
│ │ │ ├── README.md
│ │ │ ├── angular-csp.css
│ │ │ ├── angular.min.js
│ │ │ ├── angular.min.js.map
│ │ │ ├── bower.json
│ │ │ └── package.json
│ │ └── elasticsearch
│ │ │ ├── .bower.json
│ │ │ ├── README.md
│ │ │ ├── bower.json
│ │ │ ├── elasticsearch.angular.min.js
│ │ │ └── package.json
│ ├── css
│ │ ├── calaca.css
│ │ └── reset.css
│ ├── index.html
│ └── js
│ │ ├── app.js
│ │ ├── config.js
│ │ ├── controllers.js
│ │ └── services.js
└── plugin-descriptor.properties
├── LICENSE
├── README.md
├── create-template.sh
├── delete-es.sh
├── docker-run.sh
├── es-query-examples.sh
├── flush-redis.py
├── github-es.py
├── github-redis.py
├── kibana-dashboard.json
├── requirements.txt
└── scraper.py
/Calaca/CHANGELOG.md:
--------------------------------------------------------------------------------
1 | # Changelog
2 |
3 | ## 1.2.1 - Apr 7, 2016 -- Alexander Liu https://github.com/xros
4 | - Add support for ElasticSearch 2.x
5 | - Add a new branch 2.x
6 |
7 | ## 1.2.0 - May 17, 2015
8 | - Add support for delayed searches
9 | - Simplify configs
10 | - Plugin support
11 | - Update documentation
12 | - Bug fixes/cleanup
13 |
14 | ## 1.1.1 - Dec 9, 2014
15 | - Add support for lucene queries
16 | - Add support for specifying protocol
17 | - Bug fix for port config
18 |
19 | ## 1.1.0 - Oct 19 2014
20 | - Add pagination
21 | - Add time taken and hits count for query
22 | - Add bower.json
23 | - Split js files
24 |
25 | ## 1.0.1 - Jul 24 2014
26 | - Add documentation to show how to configure results.
27 | - Update the required configs/documentation
28 | - Update css class names
29 | - Add monitoring plug
30 | - Add changelog
31 |
32 | ## 1.0.0 - Jun 15 2014
33 | - Initial release
34 |
--------------------------------------------------------------------------------
/Calaca/LICENSE:
--------------------------------------------------------------------------------
1 | The MIT License (MIT)
2 |
3 | Copyright (c) 2015 Roman Sanchez
4 |
5 | Permission is hereby granted, free of charge, to any person obtaining a copy
6 | of this software and associated documentation files (the "Software"), to deal
7 | in the Software without restriction, including without limitation the rights
8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9 | copies of the Software, and to permit persons to whom the Software is
10 | furnished to do so, subject to the following conditions:
11 |
12 | The above copyright notice and this permission notice shall be included in all
13 | copies or substantial portions of the Software.
14 |
15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21 | SOFTWARE.
--------------------------------------------------------------------------------
/Calaca/README.md:
--------------------------------------------------------------------------------
1 | Calaca
2 | =========
3 |
4 | Calaca is a beautiful, easy to use, search UI for Elasticsearch. It's made for you if you need to do quick searches for your documents and don't need anything hard to setup, use.
5 | - Quick, easy and instant search
6 | - Looks good
7 | - Simple configs
8 | - Query metrics(hits counts, time took)
9 | - Pagination
10 | - Lucene queries(boolean, range, etc)
11 | - Plugin install or stand-alone
12 |
13 | Demo
14 | =========
15 | 
16 |
17 | Dependencies(Already included)
18 | ----
19 | - angular.js
20 | - angular-animate.js
21 | - elasticsearch.angular.js
22 |
23 | Get Started
24 | ----
25 | In **config.js** change the configs to match your Elasticsearch cluster.
26 | ```js
27 | /* Configs */
28 | url: "http://localhost:9200" //Cluster http url
29 | index_name: "twitter" //Index name or comma-separated list
30 | type: "tweet" //Type
31 | size: 10 //Number of results displayed at a time
32 | search_delay: 500 //Delay between actual search request in ms
33 | ```
34 |
35 | In **index.html** append to ```result.``` the field name you want to show from your es document.
36 | Using dot notation, you can access nested fields like such ```result.transactions.time ```.
37 | ```html
38 |
{{result.name}}
39 |
{{result.description}}
40 | ```
41 |
42 | Plugin
43 | ----
44 | You can also install as an elasticsearch plugin from elasticsearch home directory. Same config updates are required to **config.js** and **index.html**.
45 |
46 | Here it supports ElasticSearch version 2.x
47 | ```bash
48 | bin/plugin install romansanchez/calaca
49 | ```
50 |
51 | Should now be accessible by visiting url: http://your-host:9200/_plugin/calaca/
52 |
53 |
54 | Styling
55 | ----
56 | You can easily change the look and feel of Calaca by implementing the below CSS classes.
57 | ```css
58 | .title
59 | .search-box
60 | .no-results
61 | .results
62 | .result
63 | ```
64 |
65 | Common Issues
66 | ----
67 | * No 'Access-Control-Allow-Origin' header is present on the requested resource.
68 | * Add ```http.cors.enabled: true``` and ```http.cors.allow-origin: "*"``` to your ```elasticsearch.yml```
69 |
70 | Version
71 | ----
72 |
73 | 1.2.1
74 |
75 | Author
76 | ----
77 |
78 | [@rooomansanchez]
79 |
80 | Contributors
81 | ----
82 |
83 | [xros](https://github.com/xros)
84 |
85 |
86 | License
87 | ----
88 |
89 | MIT
90 |
91 | [romansanchez.me]:http://romansanchez.me
92 | [@rooomansanchez]:http://twitter.com/rooomansanchez
93 | [Pulse]:http://www.espulse.com
94 |
--------------------------------------------------------------------------------
/Calaca/_site/bower.json:
--------------------------------------------------------------------------------
1 | {
2 | "name": "Calaca",
3 | "version": "1.2.0",
4 | "homepage": "https://github.com/romansanchez/Calaca",
5 | "authors": [
6 | "Roman Sanchez http://romansanchez.me"
7 | ],
8 | "description": "Search UI for Elasticsearch",
9 | "main": "index.html",
10 | "keywords": [
11 | "elasticsearch",
12 | "search"
13 | ],
14 | "license": "MIT",
15 | "ignore": [
16 | "**/.*",
17 | "node_modules",
18 | "bower_components",
19 | "test",
20 | "tests"
21 | ],
22 | "dependencies": {
23 | "angular": "1.2.26",
24 | "elasticsearch": "~2.2.0",
25 | "angular-animate": "1.2.26"
26 | }
27 | }
28 |
--------------------------------------------------------------------------------
/Calaca/_site/bower_components/angular-animate/.bower.json:
--------------------------------------------------------------------------------
1 | {
2 | "name": "angular-animate",
3 | "version": "1.2.26",
4 | "main": "./angular-animate.js",
5 | "dependencies": {
6 | "angular": "1.2.26"
7 | },
8 | "homepage": "https://github.com/angular/bower-angular-animate",
9 | "_release": "1.2.26",
10 | "_resolution": {
11 | "type": "version",
12 | "tag": "v1.2.26",
13 | "commit": "453c5ac3ad0beb28e8296655c2e4079b5e0f31da"
14 | },
15 | "_source": "git://github.com/angular/bower-angular-animate.git",
16 | "_target": "1.2.26",
17 | "_originalSource": "angular-animate"
18 | }
--------------------------------------------------------------------------------
/Calaca/_site/bower_components/angular-animate/README.md:
--------------------------------------------------------------------------------
1 | # bower-angular-animate
2 |
3 | This repo is for distribution on `bower`. The source for this module is in the
4 | [main AngularJS repo](https://github.com/angular/angular.js/tree/master/src/ngAnimate).
5 | Please file issues and pull requests against that repo.
6 |
7 | ## Install
8 |
9 | Install with `bower`:
10 |
11 | ```shell
12 | bower install angular-animate
13 | ```
14 |
15 | Add a `
19 | ```
20 |
21 | And add `ngAnimate` as a dependency for your app:
22 |
23 | ```javascript
24 | angular.module('myApp', ['ngAnimate']);
25 | ```
26 |
27 | ## Documentation
28 |
29 | Documentation is available on the
30 | [AngularJS docs site](http://docs.angularjs.org/api/ngAnimate).
31 |
32 | ## License
33 |
34 | The MIT License
35 |
36 | Copyright (c) 2010-2012 Google, Inc. http://angularjs.org
37 |
38 | Permission is hereby granted, free of charge, to any person obtaining a copy
39 | of this software and associated documentation files (the "Software"), to deal
40 | in the Software without restriction, including without limitation the rights
41 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
42 | copies of the Software, and to permit persons to whom the Software is
43 | furnished to do so, subject to the following conditions:
44 |
45 | The above copyright notice and this permission notice shall be included in
46 | all copies or substantial portions of the Software.
47 |
48 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
49 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
50 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
51 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
52 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
53 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
54 | THE SOFTWARE.
55 |
--------------------------------------------------------------------------------
/Calaca/_site/bower_components/angular-animate/angular-animate.min.js:
--------------------------------------------------------------------------------
1 | /*
2 | AngularJS v1.2.26
3 | (c) 2010-2014 Google, Inc. http://angularjs.org
4 | License: MIT
5 | */
6 | (function(F,e,O){'use strict';e.module("ngAnimate",["ng"]).directive("ngAnimateChildren",function(){return function(G,s,g){g=g.ngAnimateChildren;e.isString(g)&&0===g.length?s.data("$$ngAnimateChildren",!0):G.$watch(g,function(e){s.data("$$ngAnimateChildren",!!e)})}}).factory("$$animateReflow",["$$rAF","$document",function(e,s){return function(g){return e(function(){g()})}}]).config(["$provide","$animateProvider",function(G,s){function g(e){for(var g=0;g=y&&b>=v&&e()}var h=g(b);a=b.data(q);if(-1!=h.getAttribute("class").indexOf(d)&&a){var m="";u(d.split(" "),function(a,b){m+=(0",
21 | "license": "MIT",
22 | "bugs": {
23 | "url": "https://github.com/angular/angular.js/issues"
24 | },
25 | "homepage": "http://angularjs.org"
26 | }
27 |
--------------------------------------------------------------------------------
/Calaca/_site/bower_components/angular/.bower.json:
--------------------------------------------------------------------------------
1 | {
2 | "name": "angular",
3 | "version": "1.2.26",
4 | "main": "./angular.js",
5 | "dependencies": {},
6 | "homepage": "https://github.com/angular/bower-angular",
7 | "_release": "1.2.26",
8 | "_resolution": {
9 | "type": "version",
10 | "tag": "v1.2.26",
11 | "commit": "7308d8d650b2b9948796035cbf6f3b175d45efe0"
12 | },
13 | "_source": "git://github.com/angular/bower-angular.git",
14 | "_target": "1.2.26",
15 | "_originalSource": "angular"
16 | }
--------------------------------------------------------------------------------
/Calaca/_site/bower_components/angular/README.md:
--------------------------------------------------------------------------------
1 | # bower-angular
2 |
3 | This repo is for distribution on `bower`. The source for this module is in the
4 | [main AngularJS repo](https://github.com/angular/angular.js).
5 | Please file issues and pull requests against that repo.
6 |
7 | ## Install
8 |
9 | Install with `bower`:
10 |
11 | ```shell
12 | bower install angular
13 | ```
14 |
15 | Add a `
19 | ```
20 |
21 | ## Documentation
22 |
23 | Documentation is available on the
24 | [AngularJS docs site](http://docs.angularjs.org/).
25 |
26 | ## License
27 |
28 | The MIT License
29 |
30 | Copyright (c) 2010-2012 Google, Inc. http://angularjs.org
31 |
32 | Permission is hereby granted, free of charge, to any person obtaining a copy
33 | of this software and associated documentation files (the "Software"), to deal
34 | in the Software without restriction, including without limitation the rights
35 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
36 | copies of the Software, and to permit persons to whom the Software is
37 | furnished to do so, subject to the following conditions:
38 |
39 | The above copyright notice and this permission notice shall be included in
40 | all copies or substantial portions of the Software.
41 |
42 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
43 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
44 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
45 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
46 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
47 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
48 | THE SOFTWARE.
49 |
--------------------------------------------------------------------------------
/Calaca/_site/bower_components/angular/angular-csp.css:
--------------------------------------------------------------------------------
1 | /* Include this file in your html if you are using the CSP mode. */
2 |
3 | @charset "UTF-8";
4 |
5 | [ng\:cloak], [ng-cloak], [data-ng-cloak], [x-ng-cloak],
6 | .ng-cloak, .x-ng-cloak,
7 | .ng-hide {
8 | display: none !important;
9 | }
10 |
11 | ng\:form {
12 | display: block;
13 | }
14 |
15 | .ng-animate-block-transitions {
16 | transition:0s all!important;
17 | -webkit-transition:0s all!important;
18 | }
19 |
20 | /* show the element during a show/hide animation when the
21 | * animation is ongoing, but the .ng-hide class is active */
22 | .ng-hide-add-active, .ng-hide-remove {
23 | display: block!important;
24 | }
25 |
--------------------------------------------------------------------------------
/Calaca/_site/bower_components/angular/angular.min.js:
--------------------------------------------------------------------------------
1 | /*
2 | AngularJS v1.2.26
3 | (c) 2010-2014 Google, Inc. http://angularjs.org
4 | License: MIT
5 | */
6 | (function(W,X,t){'use strict';function C(b){return function(){var a=arguments[0],c,a="["+(b?b+":":"")+a+"] http://errors.angularjs.org/1.2.26/"+(b?b+"/":"")+a;for(c=1;c").append(b).html();try{return 3===b[0].nodeType?K(c):c.match(/^(<[^>]+>)/)[1].replace(/^<([\w\-]+)/,function(a,b){return"<"+K(b)})}catch(d){return K(c)}}function dc(b){try{return decodeURIComponent(b)}catch(a){}}function ec(b){var a={},c,d;r((b||"").split("&"),function(b){b&&(c=b.replace(/\+/g,"%20").split("="),d=dc(c[0]),y(d)&&(b=y(c[1])?dc(c[1]):!0,kb.call(a,d)?J(a[d])?a[d].push(b):a[d]=[a[d],b]:a[d]=b))});return a}function Cb(b){var a=
16 | [];r(b,function(b,d){J(b)?r(b,function(b){a.push(Ca(d,!0)+(!0===b?"":"="+Ca(b,!0)))}):a.push(Ca(d,!0)+(!0===b?"":"="+Ca(b,!0)))});return a.length?a.join("&"):""}function lb(b){return Ca(b,!0).replace(/%26/gi,"&").replace(/%3D/gi,"=").replace(/%2B/gi,"+")}function Ca(b,a){return encodeURIComponent(b).replace(/%40/gi,"@").replace(/%3A/gi,":").replace(/%24/g,"$").replace(/%2C/gi,",").replace(/%20/g,a?"%20":"+")}function Xc(b,a){function c(a){a&&d.push(a)}var d=[b],e,f,g=["ng:app","ng-app","x-ng-app",
17 | "data-ng-app"],k=/\sng[:\-]app(:\s*([\w\d_]+);?)?\s/;r(g,function(a){g[a]=!0;c(X.getElementById(a));a=a.replace(":","\\:");b.querySelectorAll&&(r(b.querySelectorAll("."+a),c),r(b.querySelectorAll("."+a+"\\:"),c),r(b.querySelectorAll("["+a+"]"),c))});r(d,function(a){if(!e){var b=k.exec(" "+a.className+" ");b?(e=a,f=(b[2]||"").replace(/\s+/g,",")):r(a.attributes,function(b){!e&&g[b.name]&&(e=a,f=b.value)})}});e&&a(e,f?[f]:[])}function fc(b,a){var c=function(){b=w(b);if(b.injector()){var c=b[0]===X?
18 | "document":ia(b);throw Ta("btstrpd",c.replace(/,"<").replace(/>/,">"));}a=a||[];a.unshift(["$provide",function(a){a.value("$rootElement",b)}]);a.unshift("ng");c=gc(a);c.invoke(["$rootScope","$rootElement","$compile","$injector","$animate",function(a,b,c,d,e){a.$apply(function(){b.data("$injector",d);c(b)(a)})}]);return c},d=/^NG_DEFER_BOOTSTRAP!/;if(W&&!d.test(W.name))return c();W.name=W.name.replace(d,"");Va.resumeBootstrap=function(b){r(b,function(b){a.push(b)});c()}}function mb(b,a){a=
19 | a||"_";return b.replace(Yc,function(b,d){return(d?a:"")+b.toLowerCase()})}function Db(b,a,c){if(!b)throw Ta("areq",a||"?",c||"required");return b}function Wa(b,a,c){c&&J(b)&&(b=b[b.length-1]);Db(P(b),a,"not a function, got "+(b&&"object"===typeof b?b.constructor.name||"Object":typeof b));return b}function Da(b,a){if("hasOwnProperty"===b)throw Ta("badname",a);}function hc(b,a,c){if(!a)return b;a=a.split(".");for(var d,e=b,f=a.length,g=0;g "+e[1]+a.replace(me,"<$1>$2>")+e[2];d.removeChild(d.firstChild);for(a=e[0];a--;)d=d.lastChild;a=0;for(e=d.childNodes.length;a=Q?(c.preventDefault=null,c.stopPropagation=null,c.isDefaultPrevented=null):(delete c.preventDefault,delete c.stopPropagation,delete c.isDefaultPrevented)};c.elem=b;return c}function Ka(b,a){var c=typeof b,d;"function"==c||"object"==c&&null!==b?"function"==typeof(d=
32 | b.$$hashKey)?d=b.$$hashKey():d===t&&(d=b.$$hashKey=(a||hb)()):d=b;return c+":"+d}function bb(b,a){if(a){var c=0;this.nextUid=function(){return++c}}r(b,this.put,this)}function sc(b){var a,c;"function"===typeof b?(a=b.$inject)||(a=[],b.length&&(c=b.toString().replace(pe,""),c=c.match(qe),r(c[1].split(re),function(b){b.replace(se,function(b,c,d){a.push(d)})})),b.$inject=a):J(b)?(c=b.length-1,Wa(b[c],"fn"),a=b.slice(0,c)):Wa(b,"fn",!0);return a}function gc(b){function a(a){return function(b,c){if(T(b))r(b,
33 | $b(a));else return a(b,c)}}function c(a,b){Da(a,"service");if(P(b)||J(b))b=n.instantiate(b);if(!b.$get)throw cb("pget",a);return l[a+k]=b}function d(a,b){return c(a,{$get:b})}function e(a){var b=[],c,d,f,k;r(a,function(a){if(!h.get(a)){h.put(a,!0);try{if(v(a))for(c=Ya(a),b=b.concat(e(c.requires)).concat(c._runBlocks),d=c._invokeQueue,f=0,k=d.length;f 4096 bytes)!"));else{if(m.cookie!==
41 | da)for(da=m.cookie,d=da.split("; "),O={},f=0;fh&&this.remove(p.key),b},get:function(a){if(h").parent()[0])});var f=L(a,b,a,c,d,e);ca(a,"ng-scope");return function(b,c,d,e){Db(b,"scope");var g=c?La.clone.call(a):a;r(d,function(a,b){g.data("$"+b+"Controller",a)});d=0;for(var m=g.length;darguments.length&&
52 | (b=a,a=t);K&&(c=da);return p(a,b,c)}var u,M,z,O,I,B,da={},rb;u=c===f?d:ha(d,new Ob(w(f),d.$attr));M=u.$$element;if(L){var Na=/^\s*([@=&])(\??)\s*(\w*)\s*$/;B=e.$new(!0);!H||H!==L&&H!==L.$$originalDirective?M.data("$isolateScopeNoTemplate",B):M.data("$isolateScope",B);ca(M,"ng-isolate-scope");r(L.scope,function(a,c){var d=a.match(Na)||[],f=d[3]||c,g="?"==d[2],d=d[1],m,l,n,p;B.$$isolateBindings[c]=d+f;switch(d){case "@":u.$observe(f,function(a){B[c]=a});u.$$observers[f].$$scope=e;u[f]&&(B[c]=b(u[f])(e));
53 | break;case "=":if(g&&!u[f])break;l=q(u[f]);p=l.literal?Aa:function(a,b){return a===b||a!==a&&b!==b};n=l.assign||function(){m=B[c]=l(e);throw ja("nonassign",u[f],L.name);};m=B[c]=l(e);B.$watch(function(){var a=l(e);p(a,B[c])||(p(a,m)?n(e,a=B[c]):B[c]=a);return m=a},null,l.literal);break;case "&":l=q(u[f]);B[c]=function(a){return l(e,a)};break;default:throw ja("iscp",L.name,c,a);}})}rb=p&&F;R&&r(R,function(a){var b={$scope:a===L||a.$$isolateScope?B:e,$element:M,$attrs:u,$transclude:rb},c;I=a.controller;
54 | "@"==I&&(I=u[a.name]);c=s(I,b);da[a.name]=c;K||M.data("$"+a.name+"Controller",c);a.controllerAs&&(b.$scope[a.controllerAs]=c)});g=0;for(z=m.length;gG.priority)break;if(V=G.scope)O=O||G,G.templateUrl||(db("new/isolated scope",L,G,Z),T(V)&&(L=G));C=G.name;!G.templateUrl&&G.controller&&(V=G.controller,R=R||{},db("'"+C+"' controller",R[C],G,Z),R[C]=G);if(V=G.transclude)x=!0,G.$$tlb||
56 | (db("transclusion",fa,G,Z),fa=G),"element"==V?(K=!0,u=G.priority,V=Z,Z=d.$$element=w(X.createComment(" "+C+": "+d[C]+" ")),c=Z[0],Na(f,Ba.call(V,0),c),S=z(V,e,u,g&&g.name,{nonTlbTranscludeDirective:fa})):(V=w(Kb(c)).contents(),Z.empty(),S=z(V,e));if(G.template)if(D=!0,db("template",H,G,Z),H=G,V=P(G.template)?G.template(Z,d):G.template,V=W(V),G.replace){g=G;V=Ib.test(V)?w(aa(V)):[];c=V[0];if(1!=V.length||1!==c.nodeType)throw ja("tplrt",C,"");Na(f,Z,c);qa={$attr:{}};V=da(c,[],qa);var $=a.splice(Fa+
57 | 1,a.length-(Fa+1));L&&y(V);a=a.concat(V).concat($);E(d,qa);qa=a.length}else Z.html(V);if(G.templateUrl)D=!0,db("template",H,G,Z),H=G,G.replace&&(g=G),N=ue(a.splice(Fa,a.length-Fa),Z,d,f,x&&S,m,n,{controllerDirectives:R,newIsolateScopeDirective:L,templateDirective:H,nonTlbTranscludeDirective:fa}),qa=a.length;else if(G.compile)try{Q=G.compile(Z,d,S),P(Q)?F(null,Q,U,Y):Q&&F(Q.pre,Q.post,U,Y)}catch(ve){l(ve,ia(Z))}G.terminal&&(N.terminal=!0,u=Math.max(u,G.priority))}N.scope=O&&!0===O.scope;N.transcludeOnThisElement=
58 | x;N.templateOnThisElement=D;N.transclude=S;p.hasElementTranscludeDirective=K;return N}function y(a){for(var b=0,c=a.length;bp.priority)&&-1!=p.restrict.indexOf(f)&&(q&&(p=bc(p,{$$start:q,$$end:n})),b.push(p),h=p)}catch(F){l(F)}}return h}function E(a,b){var c=b.$attr,d=a.$attr,e=a.$$element;r(a,function(d,e){"$"!=
59 | e.charAt(0)&&(b[e]&&b[e]!==d&&(d+=("style"===e?";":" ")+b[e]),a.$set(e,d,!0,c[e]))});r(b,function(b,f){"class"==f?(ca(e,b),a["class"]=(a["class"]?a["class"]+" ":"")+b):"style"==f?(e.attr("style",e.attr("style")+";"+b),a.style=(a.style?a.style+";":"")+b):"$"==f.charAt(0)||a.hasOwnProperty(f)||(a[f]=b,d[f]=c[f])})}function ue(a,b,c,d,e,f,g,h){var m=[],l,q,s=b[0],u=a.shift(),F=D({},u,{templateUrl:null,transclude:null,replace:null,$$originalDirective:u}),N=P(u.templateUrl)?u.templateUrl(b,c):u.templateUrl;
60 | b.empty();n.get(A.getTrustedResourceUrl(N),{cache:p}).success(function(n){var p,A;n=W(n);if(u.replace){n=Ib.test(n)?w(aa(n)):[];p=n[0];if(1!=n.length||1!==p.nodeType)throw ja("tplrt",u.name,N);n={$attr:{}};Na(d,b,p);var z=da(p,[],n);T(u.scope)&&y(z);a=z.concat(a);E(c,n)}else p=s,b.html(n);a.unshift(F);l=H(a,p,c,e,b,u,f,g,h);r(d,function(a,c){a==p&&(d[c]=b[0])});for(q=L(b[0].childNodes,e);m.length;){n=m.shift();A=m.shift();var R=m.shift(),I=m.shift(),z=b[0];if(A!==s){var B=A.className;h.hasElementTranscludeDirective&&
61 | u.replace||(z=Kb(p));Na(R,w(A),z);ca(w(z),B)}A=l.transcludeOnThisElement?O(n,l.transclude,I):I;l(q,n,z,d,A)}m=null}).error(function(a,b,c,d){throw ja("tpload",d.url);});return function(a,b,c,d,e){a=e;m?(m.push(b),m.push(c),m.push(d),m.push(a)):(l.transcludeOnThisElement&&(a=O(b,l.transclude,e)),l(q,b,c,d,a))}}function x(a,b){var c=b.priority-a.priority;return 0!==c?c:a.name!==b.name?a.namea.status?d:n.reject(d)}var c={method:"get",transformRequest:e.transformRequest,transformResponse:e.transformResponse},d=function(a){var b=e.headers,c=D({},a.headers),d,f,b=D({},b.common,b[K(a.method)]);
71 | a:for(d in b){a=K(d);for(f in c)if(K(f)===a)continue a;c[d]=b[d]}(function(a){var b;r(a,function(c,d){P(c)&&(b=c(),null!=b?a[d]=b:delete a[d])})})(c);return c}(a);D(c,a);c.headers=d;c.method=Ia(c.method);var f=[function(a){d=a.headers;var c=xc(a.data,wc(d),a.transformRequest);x(c)&&r(d,function(a,b){"content-type"===K(b)&&delete d[b]});x(a.withCredentials)&&!x(e.withCredentials)&&(a.withCredentials=e.withCredentials);return s(a,c,d).then(b,b)},t],g=n.when(c);for(r(A,function(a){(a.request||a.requestError)&&
72 | f.unshift(a.request,a.requestError);(a.response||a.responseError)&&f.push(a.response,a.responseError)});f.length;){a=f.shift();var m=f.shift(),g=g.then(a,m)}g.success=function(a){g.then(function(b){a(b.data,b.status,b.headers,c)});return g};g.error=function(a){g.then(null,function(b){a(b.data,b.status,b.headers,c)});return g};return g}function s(c,f,g){function h(a,b,c,e){I&&(200<=a&&300>a?I.put(w,[a,b,vc(c),e]):I.remove(w));p(b,a,c,e);d.$$phase||d.$apply()}function p(a,b,d,e){b=Math.max(b,0);(200<=
73 | b&&300>b?A.resolve:A.reject)({data:a,status:b,headers:wc(d),config:c,statusText:e})}function s(){var a=Ra(q.pendingRequests,c);-1!==a&&q.pendingRequests.splice(a,1)}var A=n.defer(),r=A.promise,I,H,w=F(c.url,c.params);q.pendingRequests.push(c);r.then(s,s);!c.cache&&!e.cache||(!1===c.cache||"GET"!==c.method&&"JSONP"!==c.method)||(I=T(c.cache)?c.cache:T(e.cache)?e.cache:u);if(I)if(H=I.get(w),y(H)){if(H&&P(H.then))return H.then(s,s),H;J(H)?p(H[1],H[0],ha(H[2]),H[3]):p(H,200,{},"OK")}else I.put(w,r);x(H)&&
74 | ((H=Pb(c.url)?b.cookies()[c.xsrfCookieName||e.xsrfCookieName]:t)&&(g[c.xsrfHeaderName||e.xsrfHeaderName]=H),a(c.method,w,f,h,g,c.timeout,c.withCredentials,c.responseType));return r}function F(a,b){if(!b)return a;var c=[];Tc(b,function(a,b){null===a||x(a)||(J(a)||(a=[a]),r(a,function(a){T(a)&&(a=ta(a)?a.toISOString():na(a));c.push(Ca(b)+"="+Ca(a))}))});0=Q&&(!b.match(/^(get|post|head|put|delete|options)$/i)||
76 | !W.XMLHttpRequest))return new W.ActiveXObject("Microsoft.XMLHTTP");if(W.XMLHttpRequest)return new W.XMLHttpRequest;throw C("$httpBackend")("noxhr");}function Vd(){this.$get=["$browser","$window","$document",function(b,a,c){return ye(b,xe,b.defer,a.angular.callbacks,c[0])}]}function ye(b,a,c,d,e){function f(a,b,c){var f=e.createElement("script"),g=null;f.type="text/javascript";f.src=a;f.async=!0;g=function(a){$a(f,"load",g);$a(f,"error",g);e.body.removeChild(f);f=null;var k=-1,s="unknown";a&&("load"!==
77 | a.type||d[b].called||(a={type:"error"}),s=a.type,k="error"===a.type?404:200);c&&c(k,s)};sb(f,"load",g);sb(f,"error",g);8>=Q&&(f.onreadystatechange=function(){v(f.readyState)&&/loaded|complete/.test(f.readyState)&&(f.onreadystatechange=null,g({type:"load"}))});e.body.appendChild(f);return g}var g=-1;return function(e,m,h,l,n,p,q,s){function F(){A=g;R&&R();z&&z.abort()}function u(a,d,e,f,g){L&&c.cancel(L);R=z=null;0===d&&(d=e?200:"file"==ua(m).protocol?404:0);a(1223===d?204:d,e,f,g||"");b.$$completeOutstandingRequest(E)}
78 | var A;b.$$incOutstandingRequestCount();m=m||b.url();if("jsonp"==K(e)){var N="_"+(d.counter++).toString(36);d[N]=function(a){d[N].data=a;d[N].called=!0};var R=f(m.replace("JSON_CALLBACK","angular.callbacks."+N),N,function(a,b){u(l,a,d[N].data,"",b);d[N]=E})}else{var z=a(e);z.open(e,m,!0);r(n,function(a,b){y(a)&&z.setRequestHeader(b,a)});z.onreadystatechange=function(){if(z&&4==z.readyState){var a=null,b=null,c="";A!==g&&(a=z.getAllResponseHeaders(),b="response"in z?z.response:z.responseText);A===g&&
79 | 10>Q||(c=z.statusText);u(l,A||z.status,b,a,c)}};q&&(z.withCredentials=!0);if(s)try{z.responseType=s}catch(ca){if("json"!==s)throw ca;}z.send(h||null)}if(0
35 |
36 |
37 |
38 |
39 |
40 |
41 |
42 |
43 |
44 |
45 |
46 |
47 |
48 |
49 |
50 |
51 |
--------------------------------------------------------------------------------
/Calaca/_site/js/app.js:
--------------------------------------------------------------------------------
1 | /*
2 | * Calaca - Search UI for Elasticsearch
3 | * https://github.com/romansanchez/Calaca
4 | * http://romansanchez.me
5 | * @rooomansanchez
6 | *
7 | * v1.2.0
8 | * MIT License
9 | */
10 |
11 | /* Module */
12 | window.Calaca = angular.module('calaca', ['elasticsearch', 'ngAnimate'],
13 | ['$locationProvider', function($locationProvider){
14 | $locationProvider.html5Mode(true);
15 | }]
16 | );
--------------------------------------------------------------------------------
/Calaca/_site/js/config.js:
--------------------------------------------------------------------------------
1 | /*
2 | * Calaca - Search UI for Elasticsearch
3 | * https://github.com/romansanchez/Calaca
4 | * http://romansanchez.me
5 | * @rooomansanchez
6 | *
7 | * v1.2.0
8 | * MIT License
9 | */
10 |
11 | /* Configs */
12 | /**
13 | *
14 | * url - Cluster http url
15 | * index_name - Index name or comma-separated list
16 | * type - Type
17 | * size - Number of results to display at a time when pagination is enabled.
18 | * search_delay - Delay between actual search request in ms. Reduces number of queries to cluster by not making a request on each keystroke.
19 | */
20 |
21 | var CALACA_CONFIGS = {
22 | url: "http://localhost:9200",
23 | index_name: "websites",
24 | type: "sites",
25 | size: 50,
26 | search_delay: 10
27 | }
28 |
--------------------------------------------------------------------------------
/Calaca/_site/js/controllers.js:
--------------------------------------------------------------------------------
1 | /*
2 | * Calaca - Search UI for Elasticsearch
3 | * https://github.com/romansanchez/Calaca
4 | * http://romansanchez.me
5 | * @rooomansanchez
6 | *
7 | * v1.2.0
8 | * MIT License
9 | */
10 |
11 | /* Calaca Controller
12 | *
13 | * On change in search box, search() will be called, and results are bind to scope as results[]
14 | *
15 | */
16 | Calaca.controller('calacaCtrl', ['calacaService', '$scope', '$location', function(results, $scope, $location){
17 |
18 | //Init empty array
19 | $scope.results = [];
20 |
21 | //Init offset
22 | $scope.offset = 0;
23 |
24 | var paginationTriggered;
25 | var maxResultsSize = CALACA_CONFIGS.size;
26 | var searchTimeout;
27 |
28 | $scope.delayedSearch = function(mode) {
29 | clearTimeout(searchTimeout);
30 | searchTimeout = setTimeout(function() {
31 | $scope.search(mode)
32 | }, CALACA_CONFIGS.search_delay);
33 | }
34 |
35 | //On search, reinitialize array, then perform search and load results
36 | $scope.search = function(m){
37 | $scope.results = [];
38 | $scope.offset = m == 0 ? 0 : $scope.offset;//Clear offset if new query
39 | $scope.loading = m == 0 ? false : true;//Reset loading flag if new query
40 |
41 | if(m == -1 && paginationTriggered) {
42 | if ($scope.offset - maxResultsSize >= 0 ) $scope.offset -= maxResultsSize;
43 | }
44 | if(m == 1 && paginationTriggered) {
45 | $scope.offset += maxResultsSize;
46 | }
47 | $scope.paginationLowerBound = $scope.offset + 1;
48 | $scope.paginationUpperBound = ($scope.offset == 0) ? maxResultsSize : $scope.offset + maxResultsSize;
49 | $scope.loadResults(m);
50 | };
51 |
52 | //Load search results into array
53 | $scope.loadResults = function(m) {
54 | results.search($scope.query, m, $scope.offset).then(function(a) {
55 |
56 | //Load results
57 | var i = 0;
58 | for(;i < a.hits.length; i++){
59 | $scope.results.push(a.hits[i]);
60 | }
61 |
62 | //Set time took
63 | $scope.timeTook = a.timeTook;
64 |
65 | //Set total number of hits that matched query
66 | $scope.hits = a.hitsCount;
67 |
68 | //Pluralization
69 | $scope.resultsLabel = ($scope.hits != 1) ? "results" : "result";
70 |
71 | //Check if pagination is triggered
72 | paginationTriggered = $scope.hits > maxResultsSize ? true : false;
73 |
74 | //Set loading flag if pagination has been triggered
75 | if(paginationTriggered) {
76 | $scope.loading = true;
77 | }
78 | });
79 | };
80 |
81 | $scope.paginationEnabled = function() {
82 | return paginationTriggered ? true : false;
83 | };
84 |
85 |
86 | }]
87 | );
--------------------------------------------------------------------------------
/Calaca/_site/js/services.js:
--------------------------------------------------------------------------------
1 | /*
2 | * Calaca - Search UI for Elasticsearch
3 | * https://github.com/romansanchez/Calaca
4 | * http://romansanchez.me
5 | * @rooomansanchez
6 | *
7 | * v1.2.0
8 | * MIT License
9 | */
10 |
11 | /* Service to Elasticsearch */
12 | Calaca.factory('calacaService', ['$q', 'esFactory', '$location', function($q, elasticsearch, $location){
13 |
14 | //Set default url if not configured
15 | CALACA_CONFIGS.url = (CALACA_CONFIGS.url.length > 0) ? CALACA_CONFIGS.url : $location.protocol() + '://' +$location.host() + ":9200";
16 |
17 | var client = elasticsearch({ host: CALACA_CONFIGS.url });
18 |
19 | var search = function(query, mode, offset){
20 |
21 | var deferred = $q.defer();
22 |
23 | if (query.length == 0) {
24 | deferred.resolve({ timeTook: 0, hitsCount: 0, hits: [] });
25 | return deferred.promise;
26 | }
27 |
28 | client.search({
29 | "index": CALACA_CONFIGS.index_name,
30 | "type": CALACA_CONFIGS.type,
31 | "body": {
32 | "size": CALACA_CONFIGS.size,
33 | "from": offset,
34 | "query": {
35 | "query_string": {
36 | "query": query
37 | }
38 | }
39 | }
40 | }).then(function(result) {
41 |
42 | var i = 0, hitsIn, hitsOut = [], source;
43 | hitsIn = (result.hits || {}).hits || [];
44 | for(;i < hitsIn.length; i++){
45 | source = hitsIn[i]._source;
46 | source._id = hitsIn[i]._id;
47 | source._index = hitsIn[i]._index;
48 | source._type = hitsIn[i]._type;
49 | source._score = hitsIn[i]._score;
50 | hitsOut.push(source);
51 | }
52 | deferred.resolve({ timeTook: result.took, hitsCount: result.hits.total, hits: hitsOut });
53 | }, deferred.reject);
54 |
55 | return deferred.promise;
56 | };
57 |
58 | return {
59 | "search": search
60 | };
61 |
62 | }]
63 | );
64 |
--------------------------------------------------------------------------------
/Calaca/plugin-descriptor.properties:
--------------------------------------------------------------------------------
1 | name=calaca
2 | description=a search UI for elasticsearch
3 | version=1.2.1
4 | site=true
5 |
--------------------------------------------------------------------------------
/LICENSE:
--------------------------------------------------------------------------------
1 | Apache License
2 | Version 2.0, January 2004
3 | http://www.apache.org/licenses/
4 |
5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
6 |
7 | 1. Definitions.
8 |
9 | "License" shall mean the terms and conditions for use, reproduction,
10 | and distribution as defined by Sections 1 through 9 of this document.
11 |
12 | "Licensor" shall mean the copyright owner or entity authorized by
13 | the copyright owner that is granting the License.
14 |
15 | "Legal Entity" shall mean the union of the acting entity and all
16 | other entities that control, are controlled by, or are under common
17 | control with that entity. For the purposes of this definition,
18 | "control" means (i) the power, direct or indirect, to cause the
19 | direction or management of such entity, whether by contract or
20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the
21 | outstanding shares, or (iii) beneficial ownership of such entity.
22 |
23 | "You" (or "Your") shall mean an individual or Legal Entity
24 | exercising permissions granted by this License.
25 |
26 | "Source" form shall mean the preferred form for making modifications,
27 | including but not limited to software source code, documentation
28 | source, and configuration files.
29 |
30 | "Object" form shall mean any form resulting from mechanical
31 | transformation or translation of a Source form, including but
32 | not limited to compiled object code, generated documentation,
33 | and conversions to other media types.
34 |
35 | "Work" shall mean the work of authorship, whether in Source or
36 | Object form, made available under the License, as indicated by a
37 | copyright notice that is included in or attached to the work
38 | (an example is provided in the Appendix below).
39 |
40 | "Derivative Works" shall mean any work, whether in Source or Object
41 | form, that is based on (or derived from) the Work and for which the
42 | editorial revisions, annotations, elaborations, or other modifications
43 | represent, as a whole, an original work of authorship. For the purposes
44 | of this License, Derivative Works shall not include works that remain
45 | separable from, or merely link (or bind by name) to the interfaces of,
46 | the Work and Derivative Works thereof.
47 |
48 | "Contribution" shall mean any work of authorship, including
49 | the original version of the Work and any modifications or additions
50 | to that Work or Derivative Works thereof, that is intentionally
51 | submitted to Licensor for inclusion in the Work by the copyright owner
52 | or by an individual or Legal Entity authorized to submit on behalf of
53 | the copyright owner. For the purposes of this definition, "submitted"
54 | means any form of electronic, verbal, or written communication sent
55 | to the Licensor or its representatives, including but not limited to
56 | communication on electronic mailing lists, source code control systems,
57 | and issue tracking systems that are managed by, or on behalf of, the
58 | Licensor for the purpose of discussing and improving the Work, but
59 | excluding communication that is conspicuously marked or otherwise
60 | designated in writing by the copyright owner as "Not a Contribution."
61 |
62 | "Contributor" shall mean Licensor and any individual or Legal Entity
63 | on behalf of whom a Contribution has been received by Licensor and
64 | subsequently incorporated within the Work.
65 |
66 | 2. Grant of Copyright License. Subject to the terms and conditions of
67 | this License, each Contributor hereby grants to You a perpetual,
68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable
69 | copyright license to reproduce, prepare Derivative Works of,
70 | publicly display, publicly perform, sublicense, and distribute the
71 | Work and such Derivative Works in Source or Object form.
72 |
73 | 3. Grant of Patent License. Subject to the terms and conditions of
74 | this License, each Contributor hereby grants to You a perpetual,
75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable
76 | (except as stated in this section) patent license to make, have made,
77 | use, offer to sell, sell, import, and otherwise transfer the Work,
78 | where such license applies only to those patent claims licensable
79 | by such Contributor that are necessarily infringed by their
80 | Contribution(s) alone or by combination of their Contribution(s)
81 | with the Work to which such Contribution(s) was submitted. If You
82 | institute patent litigation against any entity (including a
83 | cross-claim or counterclaim in a lawsuit) alleging that the Work
84 | or a Contribution incorporated within the Work constitutes direct
85 | or contributory patent infringement, then any patent licenses
86 | granted to You under this License for that Work shall terminate
87 | as of the date such litigation is filed.
88 |
89 | 4. Redistribution. You may reproduce and distribute copies of the
90 | Work or Derivative Works thereof in any medium, with or without
91 | modifications, and in Source or Object form, provided that You
92 | meet the following conditions:
93 |
94 | (a) You must give any other recipients of the Work or
95 | Derivative Works a copy of this License; and
96 |
97 | (b) You must cause any modified files to carry prominent notices
98 | stating that You changed the files; and
99 |
100 | (c) You must retain, in the Source form of any Derivative Works
101 | that You distribute, all copyright, patent, trademark, and
102 | attribution notices from the Source form of the Work,
103 | excluding those notices that do not pertain to any part of
104 | the Derivative Works; and
105 |
106 | (d) If the Work includes a "NOTICE" text file as part of its
107 | distribution, then any Derivative Works that You distribute must
108 | include a readable copy of the attribution notices contained
109 | within such NOTICE file, excluding those notices that do not
110 | pertain to any part of the Derivative Works, in at least one
111 | of the following places: within a NOTICE text file distributed
112 | as part of the Derivative Works; within the Source form or
113 | documentation, if provided along with the Derivative Works; or,
114 | within a display generated by the Derivative Works, if and
115 | wherever such third-party notices normally appear. The contents
116 | of the NOTICE file are for informational purposes only and
117 | do not modify the License. You may add Your own attribution
118 | notices within Derivative Works that You distribute, alongside
119 | or as an addendum to the NOTICE text from the Work, provided
120 | that such additional attribution notices cannot be construed
121 | as modifying the License.
122 |
123 | You may add Your own copyright statement to Your modifications and
124 | may provide additional or different license terms and conditions
125 | for use, reproduction, or distribution of Your modifications, or
126 | for any such Derivative Works as a whole, provided Your use,
127 | reproduction, and distribution of the Work otherwise complies with
128 | the conditions stated in this License.
129 |
130 | 5. Submission of Contributions. Unless You explicitly state otherwise,
131 | any Contribution intentionally submitted for inclusion in the Work
132 | by You to the Licensor shall be under the terms and conditions of
133 | this License, without any additional terms or conditions.
134 | Notwithstanding the above, nothing herein shall supersede or modify
135 | the terms of any separate license agreement you may have executed
136 | with Licensor regarding such Contributions.
137 |
138 | 6. Trademarks. This License does not grant permission to use the trade
139 | names, trademarks, service marks, or product names of the Licensor,
140 | except as required for reasonable and customary use in describing the
141 | origin of the Work and reproducing the content of the NOTICE file.
142 |
143 | 7. Disclaimer of Warranty. Unless required by applicable law or
144 | agreed to in writing, Licensor provides the Work (and each
145 | Contributor provides its Contributions) on an "AS IS" BASIS,
146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
147 | implied, including, without limitation, any warranties or conditions
148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
149 | PARTICULAR PURPOSE. You are solely responsible for determining the
150 | appropriateness of using or redistributing the Work and assume any
151 | risks associated with Your exercise of permissions under this License.
152 |
153 | 8. Limitation of Liability. In no event and under no legal theory,
154 | whether in tort (including negligence), contract, or otherwise,
155 | unless required by applicable law (such as deliberate and grossly
156 | negligent acts) or agreed to in writing, shall any Contributor be
157 | liable to You for damages, including any direct, indirect, special,
158 | incidental, or consequential damages of any character arising as a
159 | result of this License or out of the use or inability to use the
160 | Work (including but not limited to damages for loss of goodwill,
161 | work stoppage, computer failure or malfunction, or any and all
162 | other commercial damages or losses), even if such Contributor
163 | has been advised of the possibility of such damages.
164 |
165 | 9. Accepting Warranty or Additional Liability. While redistributing
166 | the Work or Derivative Works thereof, You may choose to offer,
167 | and charge a fee for, acceptance of support, warranty, indemnity,
168 | or other liability obligations and/or rights consistent with this
169 | License. However, in accepting such obligations, You may act only
170 | on Your own behalf and on Your sole responsibility, not on behalf
171 | of any other Contributor, and only if You agree to indemnify,
172 | defend, and hold each Contributor harmless for any liability
173 | incurred by, or claims asserted against, such Contributor by reason
174 | of your accepting any such warranty or additional liability.
175 |
176 | END OF TERMS AND CONDITIONS
177 |
178 | APPENDIX: How to apply the Apache License to your work.
179 |
180 | To apply the Apache License to your work, attach the following
181 | boilerplate notice, with the fields enclosed by brackets "{}"
182 | replaced with your own identifying information. (Don't include
183 | the brackets!) The text should be enclosed in the appropriate
184 | comment syntax for the file format. We also recommend that a
185 | file or class name and description of purpose be included on the
186 | same "printed page" as the copyright notice for easier
187 | identification within third-party archives.
188 |
189 | Copyright {yyyy} {name of copyright owner}
190 |
191 | Licensed under the Apache License, Version 2.0 (the "License");
192 | you may not use this file except in compliance with the License.
193 | You may obtain a copy of the License at
194 |
195 | http://www.apache.org/licenses/LICENSE-2.0
196 |
197 | Unless required by applicable law or agreed to in writing, software
198 | distributed under the License is distributed on an "AS IS" BASIS,
199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
200 | See the License for the specific language governing permissions and
201 | limitations under the License.
202 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # pydatadc-2016
2 | PyData DC 2016 - Elastic and Redis
3 |
--------------------------------------------------------------------------------
/create-template.sh:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 |
3 | curl -XPUT localhost:9200/_template/template_1 -d '
4 | {
5 | "template":"*",
6 | "settings":{
7 | "index.refresh_interval":"5s"
8 | },
9 | "mappings":{
10 | "_default_":{
11 | "_all":{
12 | "enabled":true
13 | },
14 | "dynamic_templates":[
15 | {
16 | "string_fields":{
17 | "match":"*",
18 | "match_mapping_type":"string",
19 | "mapping":{
20 | "type":"string",
21 | "index":"not_analyzed",
22 | "omit_norms":true
23 | }
24 | }
25 | }
26 | ]
27 | }
28 | }
29 | }'
30 |
--------------------------------------------------------------------------------
/delete-es.sh:
--------------------------------------------------------------------------------
1 | curl --noproxy "*" -XDELETE http://localhost:9200/*
2 |
--------------------------------------------------------------------------------
/docker-run.sh:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 |
3 | docker run -d --net=host --name redis redis
4 | docker run -d --net=host --name elastic elasticsearch
5 | docker run --name kibana --net=host -e ELASTICSEARCH_URL=http://localhost:9200 -d kibana
6 |
--------------------------------------------------------------------------------
/es-query-examples.sh:
--------------------------------------------------------------------------------
1 | curl "http://localhost:9200/_search?q=title:Marshy%20Ski&size=10&pretty=true"
2 |
3 | curl http://localhost:9200/_search -d '{ "query": { "match": { "title": "Marshy Ski" } } }'
4 |
--------------------------------------------------------------------------------
/flush-redis.py:
--------------------------------------------------------------------------------
1 | import redis
2 |
3 | db = redis.StrictRedis('localhost', 6379, 1)
4 | db.flushall()
5 |
--------------------------------------------------------------------------------
/github-es.py:
--------------------------------------------------------------------------------
1 | import requests
2 | import elasticsearch
3 | import json
4 | import os
5 | from requests.auth import HTTPBasicAuth
6 |
7 | # Connect to ElasticSearch
8 | es = elasticsearch.Elasticsearch("127.0.0.1", port=9200)
9 |
10 | # Delete & Create index
11 | es.indices.delete(index="github", ignore=[400, 404])
12 | es.indices.create(index="github", ignore=400)
13 |
14 | token = os.environ['TOKEN']
15 |
16 | url = "https://api.github.com/search/users?q=tim"
17 |
18 | # Get a response of searched users named Tim
19 | resp = requests.get(url, auth=HTTPBasicAuth("marshyski", token)).json()["items"]
20 |
21 | people = []
22 |
23 | # Create a list of people from response
24 | for p in resp:
25 | people.append(p["login"])
26 |
27 | # Iterate over people list
28 | for p in people:
29 | url = "https://api.github.com/users/" + p + "/repos"
30 | resp = requests.get(url, auth=HTTPBasicAuth("marshyski", token)).json()
31 | langs = []
32 |
33 | # Iterate over json array and put language in list
34 | for l in resp:
35 | if l["language"] != None:
36 | langs.append(l["language"])
37 |
38 | # Get favorite language
39 | favlang = max(langs, key=langs.count)
40 | person = {}
41 | person["name"] = p
42 | person["language"] = favlang
43 | data = json.dumps(person)
44 |
45 | # POST to ElasticSearch
46 | res = es.index(index="github", doc_type="people", id=p, body=data)
47 | print "Created", res['created'], "\n"
--------------------------------------------------------------------------------
/github-redis.py:
--------------------------------------------------------------------------------
1 | import requests
2 | import redis
3 |
4 | db = redis.StrictRedis('localhost', 6379, 0)
5 |
6 | url = "https://api.github.com/users/marshyski/repos"
7 |
8 | # Get response back in JSON
9 | resp = requests.get(url).json()
10 |
11 | langs = []
12 |
13 | # Iterate over json array and put language in list
14 | for l in resp:
15 | if l["language"] != None:
16 | langs.append(l["language"])
17 |
18 | # Get max of highest key in the list of elements
19 | favlang = max(langs, key=langs.count)
20 |
21 | # Set value of language
22 | db.set("lang", favlang)
23 |
24 | # Set hash timlang with dict
25 | db.hmset("timlang", {"name": "timski", "language": favlang})
26 |
27 | # Get value from key lang
28 | print "Favorite language is", db.get("lang")
29 |
30 | # Get hash timlang
31 | print db.hgetall("timlang")
32 |
33 | # Get hash and value of name
34 | print db.hgetall("timlang")["name"]
--------------------------------------------------------------------------------
/kibana-dashboard.json:
--------------------------------------------------------------------------------
1 | [
2 | {
3 | "_id": "pydata",
4 | "_type": "dashboard",
5 | "_source": {
6 | "title": "pydata",
7 | "hits": 0,
8 | "description": "",
9 | "panelsJSON": "[{\"col\":1,\"id\":\"datatables\",\"panelIndex\":1,\"row\":1,\"size_x\":5,\"size_y\":3,\"type\":\"visualization\"},{\"col\":6,\"id\":\"pie\",\"panelIndex\":2,\"row\":1,\"size_x\":7,\"size_y\":3,\"type\":\"visualization\"},{\"col\":1,\"id\":\"total\",\"panelIndex\":3,\"row\":4,\"size_x\":3,\"size_y\":2,\"type\":\"visualization\"},{\"col\":4,\"id\":\"count-languages\",\"panelIndex\":4,\"row\":4,\"size_x\":3,\"size_y\":2,\"type\":\"visualization\"}]",
10 | "optionsJSON": "{\"darkTheme\":false}",
11 | "uiStateJSON": "{}",
12 | "version": 1,
13 | "timeRestore": false,
14 | "kibanaSavedObjectMeta": {
15 | "searchSourceJSON": "{\"filter\":[{\"query\":{\"query_string\":{\"query\":\"*\",\"analyze_wildcard\":true}}}]}"
16 | }
17 | }
18 | },
19 | {
20 | "_id": "pie",
21 | "_type": "visualization",
22 | "_source": {
23 | "title": "pie",
24 | "visState": "{\"title\":\"New Visualization\",\"type\":\"pie\",\"params\":{\"shareYAxis\":true,\"addTooltip\":true,\"addLegend\":true,\"isDonut\":false},\"aggs\":[{\"id\":\"1\",\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"language\",\"size\":100,\"order\":\"desc\",\"orderBy\":\"1\"}}],\"listeners\":{}}",
25 | "uiStateJSON": "{}",
26 | "description": "",
27 | "version": 1,
28 | "kibanaSavedObjectMeta": {
29 | "searchSourceJSON": "{\"index\":\"github\",\"query\":{\"query_string\":{\"query\":\"*\",\"analyze_wildcard\":true}},\"filter\":[]}"
30 | }
31 | }
32 | },
33 | {
34 | "_id": "datatables",
35 | "_type": "visualization",
36 | "_source": {
37 | "title": "datatables",
38 | "visState": "{\"title\":\"New Visualization\",\"type\":\"table\",\"params\":{\"perPage\":10,\"showPartialRows\":false,\"showMeticsAtAllLevels\":false},\"aggs\":[{\"id\":\"1\",\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"type\":\"terms\",\"schema\":\"bucket\",\"params\":{\"field\":\"name\",\"size\":100,\"order\":\"desc\",\"orderBy\":\"1\"}},{\"id\":\"3\",\"type\":\"terms\",\"schema\":\"bucket\",\"params\":{\"field\":\"language\",\"size\":100,\"order\":\"desc\",\"orderBy\":\"1\"}}],\"listeners\":{}}",
39 | "uiStateJSON": "{}",
40 | "description": "",
41 | "version": 1,
42 | "kibanaSavedObjectMeta": {
43 | "searchSourceJSON": "{\"index\":\"github\",\"query\":{\"query_string\":{\"query\":\"*\",\"analyze_wildcard\":true}},\"filter\":[]}"
44 | }
45 | }
46 | },
47 | {
48 | "_id": "total",
49 | "_type": "visualization",
50 | "_source": {
51 | "title": "total",
52 | "visState": "{\"title\":\"New Visualization\",\"type\":\"metric\",\"params\":{\"handleNoResults\":true,\"fontSize\":60},\"aggs\":[{\"id\":\"1\",\"type\":\"count\",\"schema\":\"metric\",\"params\":{}}],\"listeners\":{}}",
53 | "uiStateJSON": "{}",
54 | "description": "",
55 | "version": 1,
56 | "kibanaSavedObjectMeta": {
57 | "searchSourceJSON": "{\"index\":\"github\",\"query\":{\"query_string\":{\"query\":\"*\",\"analyze_wildcard\":true}},\"filter\":[]}"
58 | }
59 | }
60 | },
61 | {
62 | "_id": "count-languages",
63 | "_type": "visualization",
64 | "_source": {
65 | "title": "count languages",
66 | "visState": "{\"title\":\"New Visualization\",\"type\":\"metric\",\"params\":{\"handleNoResults\":true,\"fontSize\":60},\"aggs\":[{\"id\":\"1\",\"type\":\"cardinality\",\"schema\":\"metric\",\"params\":{\"field\":\"language\"}}],\"listeners\":{}}",
67 | "uiStateJSON": "{}",
68 | "description": "",
69 | "version": 1,
70 | "kibanaSavedObjectMeta": {
71 | "searchSourceJSON": "{\"index\":\"github\",\"query\":{\"query_string\":{\"query\":\"*\",\"analyze_wildcard\":true}},\"filter\":[]}"
72 | }
73 | }
74 | }
75 | ]
--------------------------------------------------------------------------------
/requirements.txt:
--------------------------------------------------------------------------------
1 | redis==2.10.5
2 | elasticsearch==2.4.0
3 | requests==2.7.0
4 | beautifulsoup4==4.5.1
--------------------------------------------------------------------------------
/scraper.py:
--------------------------------------------------------------------------------
1 | import elasticsearch
2 | import requests, json, datetime, time, re
3 | from bs4 import BeautifulSoup
4 |
5 | requests.packages.urllib3.disable_warnings()
6 |
7 | # Configurations
8 | sites = ['http://marshyski.com', 'http://marshyski.com/man-behind-the-keyboard',
9 | 'http://marshyski.com/music', 'http://www.npr.org/sections/news/', 'https://news.google.com/']
10 |
11 | index_name = "websites"
12 | index_type = "sites"
13 | elastic_host = "127.0.0.1"
14 | elastic_port = "9200"
15 |
16 | # Connect to ElasticSearch
17 | es = elasticsearch.Elasticsearch(elastic_host, port=elastic_port)
18 |
19 | # Delete & Create index
20 | es.indices.delete(index=index_name, ignore=[400, 404])
21 | es.indices.create(index=index_name, ignore=400)
22 |
23 | # Make requests in configuration
24 | for URL in sites:
25 |
26 | try:
27 | r = requests.get(URL, verify=False)
28 | r_text = r.text
29 | soup = BeautifulSoup(r_text, "lxml")
30 | title = soup.title.string
31 | if r.status_code == 200:
32 |
33 | # Get only text from request body
34 | site_text = soup.get_text()
35 |
36 | # Strip HTML and join new lines
37 | lines = (line.strip() for line in site_text.splitlines())
38 | chunks = (phrase.strip() for line in lines for phrase in line.split(" "))
39 | join = ' '.join(chunk for chunk in chunks if chunk)
40 | text = re.sub(r'[^a-zA-Z\d\s]', '', join)
41 |
42 | for script in soup(["script", "style", "title", "a", "footer"]) + \
43 | soup.findAll('div', attrs={'class': 'footer'}) + \
44 | soup.findAll('div', attrs={'id': 'sidebar'}):
45 | script.extract()
46 |
47 | site_text = soup.get_text().replace('"', "'")
48 |
49 | lines = (line.strip() for line in site_text.splitlines())
50 | chunks = (phrase.strip() for line in lines for phrase in line.split(" "))
51 | text = ' '.join(chunk for chunk in chunks if chunk)
52 |
53 | date_time = datetime.datetime.strftime(datetime.datetime.now(), '%Y%m%d%H%M%S')
54 |
55 | # Build request to ElasticSearch
56 | js = {"title":title, "url":URL, "date":date_time, "body":text}
57 | data = json.dumps(js)
58 |
59 | print data
60 |
61 | # POST to ElasticSearch
62 | res = es.index(index=index_name, doc_type=index_type, id=date_time, body=data)
63 | print "Created", res['created'], "\n"
64 | time.sleep(1.0)
65 |
66 | else:
67 |
68 | print URL, r.status_code, "[FAIL]"
69 |
70 | except:
71 | print URL, "[FAIL]"
72 |
--------------------------------------------------------------------------------