├── .gitignore ├── README.md ├── accounts_manager └── accounts_manager.go ├── adaptative_bootstrap_tree ├── rec_tree.go └── rec_tree_test.go ├── api └── api.go ├── billing ├── billing.go └── billing_test.go ├── bin ├── pit-cli.go └── pit.go ├── cfg ├── cfg.go ├── cfg_test.go └── etc │ └── config_dev.ini ├── debian ├── changelog ├── control ├── postinst └── rules ├── debian_static ├── changelog ├── control ├── postinst └── rules ├── etc ├── pit_dev.ini └── pit_pro.ini ├── license.txt ├── log ├── log.go └── log_test.go ├── makefile ├── models ├── instances │ ├── instances.go │ └── instances_test.go ├── shard_info │ ├── shard_info.go │ └── shard_info_test.go └── users │ ├── users.go │ └── users_test.go ├── recommender ├── recomender.go └── recomender_test.go ├── shards_manager └── shards_manager.go └── static ├── account-logs.html ├── account-panel.html ├── api.html ├── billing.html ├── cases-of-use.html ├── contact-form.html ├── css ├── bootstrap-theme.css ├── bootstrap-theme.css.map ├── bootstrap-theme.min.css ├── bootstrap.css ├── bootstrap.css.map ├── bootstrap.min.css ├── dashboard.css ├── highlightjs.css ├── sidebar.css └── styles.css ├── img ├── 60-lines.png ├── brick.png ├── bright-squares.png ├── concrete_seamless.png ├── confectionary.png ├── favicon.png ├── graph_bg.png ├── grey_wash_wall.png ├── how_works.png ├── light_grey.png ├── logo_orange.png ├── logo_white.png ├── logo_white_r.png ├── manage_pannel.png └── old_map.png ├── index.html ├── js ├── bootstrap.js ├── bootstrap.min.js ├── ekko-lightbox.min.js ├── handlebars-v3.0.1.js ├── highcharts.js ├── highlight.pack.js ├── npm.js └── pit │ ├── bootstrap.js │ └── controllers │ ├── account.js │ ├── billing.js │ ├── contact-form.js │ ├── groups.js │ └── login.js ├── pricing.html ├── privacy.html ├── security.html ├── terms.html ├── try-it.html └── website-terms.html /.gitignore: -------------------------------------------------------------------------------- 1 | test_training_set/* 2 | tmp 3 | dist_package 4 | pit.deb 5 | pit_static.deb 6 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## Pitia - Real-Time & Highly Scalable Recommender System 2 | 3 | ![Pitia](https://raw.githubusercontent.com/alonsovidales/pit/master/static/img/favicon.png) 4 | 5 | ### Description 6 | Pitia is an open source recommender system developed using Go and and based in an improved version of the algorithm described in the "Adaptive Bootstrapping of Recommender Systems Using Decision Trees" paper by Yahoo. 7 | 8 | After test the recomendations algorithm we got more than a 95% of precision using the Netflix Prize dataset, you can read more about how the tests was performed on [our blog](http://blog.pitia.info/2015/06/more-than-95-of-precision-obtained.html). 9 | 10 | Pitia provides an easy to use [HTTP API](http://pitia.info/api) that can be integrated on almost any client. 11 | 12 | This project is designed as a horizontally scalable system based on the concept of virtual shards inside instances. The system was designed to be deployed on an array of instances behind a load balancer in order to distribute randomly the requests across the nodes. There is no a master instance in the cluster and all the new instances are registered automatically being registered, so to scale the system just add new instances, is recomended to add autoscaling based in the CPU and memory usage of the nodes. 13 | 14 | Dynamo DB is used to coordinate the virtual shards distribution, architecture of the cluster, and to store the accounts information. 15 | 16 | #### Intances, groups, shards and accounts 17 | The system contains *User Accounts*, each user account contains *Groups* and each group, *Virtual Shards* 18 | 19 | Each group has to be used for a single purpose, for instance we can have groups to perform recommendations of movies, books, etc. Each different use case has to be isolated in a separate group. 20 | 21 | For instance, a group can store book classifications in order to be used to perform recommendations of books based on the books that the users had been read, but other group can contain items of a store in general, in order to perform recommendations of items to buy based in the elements that the user had bought before. 22 | 23 | Each group contains number of Virtual Shards, up to the number of available instances, defined by the user, since each shard is going to be allocated in a different inscante, the shards can be of different types (see [Pricing section](http://pitia.info/pricing)) , the type will define the number of requests per second and elements that can be stored on each shard, this properties can be configured in the INI file. 24 | 25 | Since each shard is allocated in a different node in case of one of the nodes goes down the shards allocated by this node are going to be acquired by another nodes. In order to grant high availability, it is not recommended to define less than two shards by group. 26 | 27 | #### Shard adquisition 28 | In order to distribute the shards across the cluster instances, the system uses a bidding strategy, each instance try to acquire a group if have enough resources to allocate it, to bid for a shard, the instance inscribes itself in a DynamoDB table, after some seconds, time enough for the other instances to claim that shard, the shard with more resources available that is claiming this shard will acquire it. 29 | 30 | If an instance goes down, the shards are released after a period of time that can be defined in the INI config file being them released, and the other nodes are going to start with the bidding strategy to claim this free shards. 31 | 32 | The information stored on each shard is not shared with another shards of the same group since the purpose of this system is to perform recommendations and based in the idea that the load balancer is going to distribute randomly the incoming requests across all the available instances we can consider that the quality of the predictions is the same for all the shards. 33 | 34 | #### Data storage 35 | Each shard is going to dump all the information in memory periodically into S3 encoded as JSON, and each time a new shard is adquired the memory will be restored using the last available backup on S3. 36 | 37 | ### Installation and configuration 38 | The configuration of each of the cluster nodes is defined in two places, the /etc/pit_\.ini file, and some environment variables, the INI file contains the most general configuration parameters and this file can be upload to any public repository without security risks, the environment variables contains security related variables. 39 | The environment variables to be present on the system are the next: 40 | 41 | ``` 42 | AWS_DEFAULT_REGION = AWS region where the S3 and DynamoDB information will be stored 43 | AWS_ACCESS_KEY_ID = AWS AMI key ID , this key has to have access to the S3 bucked and to DynamoDB 44 | AWS_SECRET_ACCESS_KEY = AWS AMI key 45 | PIT_MAIL_PASS = Password of the e-mail used to send notifications 46 | ``` 47 | 48 | #### Deployment 49 | There is a MakeFile that will help you out with the most common tasks like: 50 | 51 | * **deps:** Downloads and installs all the Go reqiered dependencies 52 | * **updatedeps:** In case of have the dependencies already installed, this script will update them to the last available version, is recommended to use GoDeps in order to avoid problems with versions, etc, this scripts are designed to help during the developement process 53 | * **format:** Goes through all the files using "go fmt" auto formating them 54 | * **test:** Lauches all the test suites for all the packages 55 | * **deb:** Compiles the aplication for amd64 architecture building a debian package that can be used to install the aplication on all the environments 56 | * **static_deb:** Generates a debian package that contains all the static content used by the https://wwww.pitia.info website and contained in the *static* directory. 57 | * **deploy_dev:** Generates the debian package using the "deb" script, uploads and deploys it into all the machines specified on the env var PIT_DEV_SERVERS , use spaces as separator for the machine names, like export PIT_DEV_SERVERS="machine1 machihne2 ... machineN" 58 | * **deploy_pro:** Generates the debian package using the "deb" script, uploads and deploys it into all the machines specified on the env var PIT_PRO_SERVERS , use spaces as separator for the machine names, like export PIT_PRO_SERVERS="machine1 machihne2 ... machineN" 59 | * **deploy_static_pro:** Generates the debian package for static content only using the "static_deb" script and uploads and deploys it into all the machines specified on the env var PIT_PRO_SERVERS , use spaces as separator for the machine names, like export PIT_PRO_SERVERS="machine1 machihne2 ... machineN" 60 | 61 | In order to start the system after install it in a node, execute: 62 | ``` 63 | pit 64 | ``` 65 | Where env is the corresponding name for the environment, this var will define what file to use of the available INI files on the machine, for instance, *pit pro* will get the configuration from */etc/pit_pro.ini* 66 | 67 | ### Management 68 | The system can be managed from the Web panel, this panel provides a high level management, and allows to: 69 | 70 | * Create accounts 71 | * Update account information 72 | * Retrieve account passwords 73 | * Verify account identity 74 | * Create groups of shards 75 | * Configure groups of shards 76 | * Visualize the current status of the shards, elements stored on each one, load that are receiving, status, etc. 77 | * Regenerate the secret key for each group 78 | * Access to the account logs to know all the different changes that had been performed on the user account 79 | * Access to the billing information 80 | 81 | ![Panel](http://pitia.info/img/manage_pannel.png) 82 | 83 | A lower level management can be performed using the *pit-cli* terminal client, in order to use this client is necessary to have the environment configured on the machine where it is going to be executed, this means the corresponding INI file and environment variables. 84 | 85 | This client provides a super user access, that means that you can work with any account registered on the system, and allows to perform the next operations: 86 | 87 | * List all the instances on the cluster 88 | * List all the registered users 89 | * Add new user accounts 90 | * Show all the information regarding to a user 91 | * Enable / disable user accounts 92 | * List all the available groups of shards in the cluster 93 | * Remove groups of shards 94 | * Change the configuration for a group of shards 95 | 96 | ``` 97 | root@pit-pro-004:~# pit-cli --env pro --help 98 | usage: pit-cli --env=ENV [] [ ...] 99 | 100 | Pit-cli is a tool to manage all the different features of the recommener system Pit 101 | 102 | Flags: 103 | --help Show help. 104 | --env=ENV Configuration environment: pro, pre, dev 105 | 106 | Commands: 107 | help [] 108 | Show help for a command. 109 | 110 | instances list 111 | Lists all the instances on the clusted 112 | 113 | users list 114 | lists all the users 115 | 116 | users add 117 | Adds a new user 118 | 119 | users show 120 | Shows all the sotred information for a specific user 121 | 122 | users enable 123 | Enables a disabled user account 124 | 125 | users disable 126 | Disables an enabled user account 127 | 128 | groups list [] 129 | lists all the shards in the system 130 | 131 | groups del 132 | Removes one of the groups 133 | 134 | groups update --max-score=MAX-SCORE --num-shards=NUM-SHARDS --num-elems=NUM-ELEMS --max-req-sec=MAX-REQ-SEC --max-ins-req-sec=MAX-INS-REQ-SEC --user-id=USER-ID --group-id=GROUP-ID 135 | Adds or updates an existing shard 136 | ``` 137 | 138 | ### License 139 | 140 | Use of this source code is governed by the [GPL license](https://github.com/alonsovidales/pit/blob/master/license.txt). These programs and documents are distributed without any warranty, express or implied. All use of these programs is entirely at the user's own risk. 141 | 142 | 143 | 144 | 145 | 146 | 147 | 148 | 149 | 150 | 151 | 152 | -------------------------------------------------------------------------------- /accounts_manager/accounts_manager.go: -------------------------------------------------------------------------------- 1 | package accountsmanager 2 | 3 | import ( 4 | "crypto/sha1" 5 | "encoding/json" 6 | "fmt" 7 | "github.com/alonsovidales/pit/log" 8 | "github.com/alonsovidales/pit/models/users" 9 | "net/http" 10 | "net/smtp" 11 | "net/url" 12 | "os" 13 | "strconv" 14 | "strings" 15 | "time" 16 | ) 17 | 18 | const ( 19 | // CBillingInfo Endpoint for the service that will provide the client 20 | // billing info 21 | CBillingInfo = "/billing_info" 22 | // CRegisterPath Endpoint to be used to register new system accounts 23 | CRegisterPath = "/account_register" 24 | // CVerifyPath Endpoint to be used to verify the identity of an account 25 | CVerifyPath = "/account_verify" 26 | // CLogsPath Endpoint that will return the logs for an account 27 | CLogsPath = "/account_logs" 28 | // CDisablePath Endpoint to be used to disable an account 29 | CDisablePath = "/disable_account" 30 | // CRecoverPassPath Endpoint to be used to recover for account password 31 | // recovery 32 | CRecoverPassPath = "/recover_pass" 33 | // CChangePass Endpoint to be used to change an account password 34 | CChangePass = "/change_pass" 35 | // CResetPass Endpoint to be used to reset the password for an account 36 | CResetPass = "/reset_pass" 37 | 38 | // cConfirmEmailTTL Time in seconds that a recovery password token is 39 | // going to be valid 40 | cConfirmEmailTTL = 24 * 3600 41 | ) 42 | 43 | // Manager Strcuture that will be used to manage all the accounts in the system 44 | type Manager struct { 45 | usersModel users.ModelInt 46 | baseURL string 47 | secret string 48 | mailFromAddr string 49 | mailServerAddr string 50 | mailServerPort int64 51 | } 52 | 53 | // Init Initialize a Manager struct with the corresponding parameters and 54 | // returns it 55 | func Init(baseURL, mailFromAddr, mailServerAddr string, mailServerPort int64, usersModel users.ModelInt) (mg *Manager) { 56 | mg = &Manager{ 57 | baseURL: baseURL, 58 | secret: os.Getenv("PIT_SECRET"), 59 | mailServerAddr: mailServerAddr, 60 | mailFromAddr: mailFromAddr, 61 | mailServerPort: mailServerPort, 62 | usersModel: usersModel, 63 | } 64 | 65 | return 66 | } 67 | 68 | // BillingInfo Returns as JSON the billing info for the given account 69 | func (mg *Manager) BillingInfo(w http.ResponseWriter, r *http.Request) { 70 | w.Header().Set("Access-Control-Allow-Origin", "*") 71 | 72 | uid := r.FormValue("u") 73 | key := r.FormValue("k") 74 | 75 | if userInfo := mg.usersModel.GetUserInfo(uid, key); userInfo != nil { 76 | logs := userInfo.GetBillingInfo() 77 | logsJSON, err := json.Marshal(logs) 78 | if err != nil { 79 | w.WriteHeader(500) 80 | w.Write([]byte("User billing logs can't be converted to JSON")) 81 | return 82 | } 83 | 84 | w.WriteHeader(200) 85 | w.Write(logsJSON) 86 | } else { 87 | w.WriteHeader(401) 88 | w.Write([]byte(fmt.Sprint("Unauthorized"))) 89 | } 90 | } 91 | 92 | // Register a new account on the system 93 | func (mg *Manager) Register(w http.ResponseWriter, r *http.Request) { 94 | w.Header().Set("Access-Control-Allow-Origin", "*") 95 | 96 | // Sanitize e-mail addr removin all the + Chars in order to avoid fake 97 | // duplicated accounts 98 | uid := strings.Replace(r.FormValue("uid"), "+", "", -1) 99 | key := r.FormValue("key") 100 | 101 | if uid == "" || key == "" { 102 | w.WriteHeader(400) 103 | w.Write([]byte("The uid and key parameters are required")) 104 | return 105 | } 106 | 107 | if mg.usersModel.AdminGetUserInfoByID(uid) != nil { 108 | w.WriteHeader(422) 109 | w.Write([]byte("The email address you have entered is already registered")) 110 | return 111 | } 112 | 113 | ttl := time.Now().Unix() + cConfirmEmailTTL 114 | keyHash := mg.usersModel.HashPassword(key) 115 | 116 | v := url.Values{} 117 | v.Set("u", uid) 118 | v.Set("k", keyHash) 119 | v.Set("t", fmt.Sprintf("%d", ttl)) 120 | v.Set("s", mg.getSignature(uid, keyHash, ttl)) 121 | 122 | verifURL := fmt.Sprintf( 123 | "%s/%s?%s", 124 | mg.baseURL, 125 | CVerifyPath, 126 | v.Encode()) 127 | 128 | emailSent := mg.SendEmail( 129 | uid, 130 | fmt.Sprintf( 131 | "Hello from Pitia!,\n\tPlease, click on the next link in order to verify you account: %s\n\nBest!,", 132 | verifURL), 133 | "Account verification from Pitia") 134 | 135 | if !emailSent { 136 | w.WriteHeader(500) 137 | w.Write([]byte("Problem trying to send the verification e-mail")) 138 | 139 | return 140 | } 141 | 142 | w.WriteHeader(200) 143 | w.Write([]byte("Verification e-mail sent, please check your e-mail!")) 144 | } 145 | 146 | // Verify This method will be called in order to verify the identity of an 147 | // account, using the link from the e-mail provided during the registration 148 | func (mg *Manager) Verify(w http.ResponseWriter, r *http.Request) { 149 | w.Header().Set("Access-Control-Allow-Origin", "*") 150 | 151 | uid := r.FormValue("u") 152 | key := r.FormValue("k") 153 | ttl, err := strconv.ParseInt(r.FormValue("t"), 10, 64) 154 | if err != nil { 155 | w.WriteHeader(500) 156 | w.Write([]byte("The privided timestamp can't be parsed as integer")) 157 | return 158 | } 159 | sign := r.FormValue("s") 160 | 161 | if sign != mg.getSignature(uid, key, ttl) { 162 | w.WriteHeader(403) 163 | w.Write([]byte("Signature error")) 164 | return 165 | } 166 | 167 | if ttl < time.Now().Unix() { 168 | w.WriteHeader(403) 169 | w.Write([]byte("Verification e-mail expired, please register again your account")) 170 | return 171 | } 172 | 173 | if user, err := mg.usersModel.RegisterUserPlainKey(uid, key, r.RemoteAddr); err != nil { 174 | w.WriteHeader(500) 175 | w.Write([]byte(fmt.Sprint(err))) 176 | } else { 177 | user.AddActivityLog(users.CActivityAccountType, "Account verified", r.RemoteAddr) 178 | w.WriteHeader(200) 179 | w.Write([]byte("Account verified")) 180 | } 181 | } 182 | 183 | // Logs Returns the logs for a provided account 184 | func (mg *Manager) Logs(w http.ResponseWriter, r *http.Request) { 185 | w.Header().Set("Access-Control-Allow-Origin", "*") 186 | 187 | uid := r.FormValue("u") 188 | key := r.FormValue("k") 189 | 190 | if userInfo := mg.usersModel.GetUserInfo(uid, key); userInfo != nil { 191 | logs := userInfo.GetAllActivity() 192 | logsJSON, err := json.Marshal(logs) 193 | if err != nil { 194 | w.WriteHeader(500) 195 | w.Write([]byte("User logs can't be converted to JSON")) 196 | return 197 | } 198 | 199 | w.WriteHeader(200) 200 | w.Write(logsJSON) 201 | } else { 202 | w.WriteHeader(401) 203 | w.Write([]byte(fmt.Sprint("Unauthorized"))) 204 | } 205 | } 206 | 207 | // Disable Disables an account disallowing any future access to it 208 | func (mg *Manager) Disable(w http.ResponseWriter, r *http.Request) { 209 | w.Header().Set("Access-Control-Allow-Origin", "*") 210 | 211 | uid := r.FormValue("u") 212 | key := r.FormValue("k") 213 | 214 | if userInfo := mg.usersModel.GetUserInfo(uid, key); userInfo != nil { 215 | userInfo.DisableUser() 216 | userInfo.AddActivityLog(users.CActivityAccountType, "Account disabled", r.RemoteAddr) 217 | w.WriteHeader(200) 218 | w.Write([]byte("OK")) 219 | } else { 220 | w.WriteHeader(401) 221 | w.Write([]byte(fmt.Sprint("Unauthorized"))) 222 | } 223 | } 224 | 225 | // ChangePass Verify and changes the password for a speciied account 226 | func (mg *Manager) ChangePass(w http.ResponseWriter, r *http.Request) { 227 | w.Header().Set("Access-Control-Allow-Origin", "*") 228 | 229 | var recSignatue string 230 | 231 | uid := r.FormValue("u") 232 | key := r.FormValue("k") 233 | newKey := r.FormValue("nk") 234 | signature := r.FormValue("s") 235 | if signature != "" { 236 | ttl, err := strconv.ParseInt(r.FormValue("t"), 10, 64) 237 | if err != nil { 238 | w.WriteHeader(500) 239 | w.Write([]byte("Internal Server Error")) 240 | return 241 | } 242 | recSignatue = mg.getSignature(uid, "recovery", ttl) 243 | } 244 | 245 | if userInfo := mg.usersModel.GetUserInfo(uid, key); userInfo != nil || (signature != "" && signature == recSignatue) { 246 | if userInfo.UpdateUser(newKey) { 247 | userInfo.AddActivityLog(users.CActivityAccountType, "Password changed", r.RemoteAddr) 248 | w.WriteHeader(200) 249 | w.Write([]byte("OK")) 250 | } else { 251 | w.WriteHeader(500) 252 | w.Write([]byte("Internal Server Error")) 253 | } 254 | } else { 255 | w.WriteHeader(401) 256 | w.Write([]byte(fmt.Sprint("Unauthorized"))) 257 | } 258 | } 259 | 260 | // RecoverPass This method generates a "recovery token" that is sent to the 261 | // account associated e-mail, from the link provided on the e-mail the user can 262 | // restore the account password 263 | func (mg *Manager) RecoverPass(w http.ResponseWriter, r *http.Request) { 264 | w.Header().Set("Access-Control-Allow-Origin", "*") 265 | 266 | uid := r.FormValue("u") 267 | 268 | if userInfo := mg.usersModel.AdminGetUserInfoByID(uid); userInfo != nil { 269 | ttl := time.Now().Unix() + cConfirmEmailTTL 270 | 271 | v := url.Values{} 272 | v.Set("u", uid) 273 | v.Set("t", fmt.Sprintf("%d", ttl)) 274 | v.Set("s", mg.getSignature(uid, "recovery", ttl)) 275 | 276 | verifURL := fmt.Sprintf( 277 | "%s/%s?%s", 278 | mg.baseURL, 279 | CResetPass, 280 | v.Encode()) 281 | 282 | body := fmt.Sprintf( 283 | "Hi!,\n\tYou have requested password recovery, please click the following link to reset your password: %s\n\nBest,", 284 | verifURL) 285 | 286 | if mg.SendEmail(uid, body, "Pitia: Password Recovery") { 287 | userInfo.AddActivityLog(users.CActivityAccountType, "Password recovery sent", r.RemoteAddr) 288 | w.WriteHeader(200) 289 | w.Write([]byte("OK")) 290 | return 291 | } 292 | 293 | w.WriteHeader(500) 294 | w.Write([]byte("KO")) 295 | return 296 | } 297 | 298 | // Don't return any clue about if the user is or is not registered 299 | w.WriteHeader(200) 300 | w.Write([]byte("OK")) 301 | return 302 | } 303 | 304 | // getSignature Generates a signature from the provided parameters that can be 305 | // used to verify or sign this 306 | func (mg *Manager) getSignature(uid, keyHash string, ttl int64) string { 307 | return fmt.Sprintf("%x", sha1.Sum([]byte(fmt.Sprintf("%s:%s:%d:%s", uid, keyHash, ttl, mg.secret)))) 308 | } 309 | 310 | // SendEmail Sends an e-mail to the specified addess with the specified subject 311 | // and body 312 | func (mg *Manager) SendEmail(to, body, subject string) (success bool) { 313 | auth := smtp.PlainAuth( 314 | mg.mailFromAddr, 315 | mg.mailFromAddr, 316 | os.Getenv("PIT_MAIL_PASS"), 317 | mg.mailServerAddr, 318 | ) 319 | 320 | body = fmt.Sprintf( 321 | "Subject: %s\r\n\r\n\r\n%s", 322 | subject, 323 | []byte(body)) 324 | 325 | err := smtp.SendMail( 326 | fmt.Sprintf("%s:%d", mg.mailServerAddr, mg.mailServerPort), 327 | auth, 328 | mg.mailFromAddr, 329 | []string{to}, 330 | []byte(body), 331 | ) 332 | 333 | if err != nil { 334 | log.Error("Problem trying to send a verification e-mail:", err) 335 | return false 336 | } 337 | return true 338 | } 339 | -------------------------------------------------------------------------------- /adaptative_bootstrap_tree/rec_tree_test.go: -------------------------------------------------------------------------------- 1 | package rectree 2 | 3 | import ( 4 | "bufio" 5 | "encoding/json" 6 | "github.com/alonsovidales/pit/log" 7 | "math" 8 | "os" 9 | "runtime" 10 | "strconv" 11 | "strings" 12 | "testing" 13 | ) 14 | 15 | const ( 16 | TESTSET = "../test_training_set/training_set.info" 17 | MAXSCORE = 5 18 | ) 19 | 20 | func TestCollabInsertion(t *testing.T) { 21 | runtime.GOMAXPROCS(runtime.NumCPU()) 22 | 23 | f, err := os.Open(TESTSET) 24 | if err != nil { 25 | log.Error("Can't read the the test set file:", TESTSET, "Error:", err) 26 | t.Fail() 27 | } 28 | r := bufio.NewReader(f) 29 | s, e := Readln(r) 30 | records := []map[uint64]uint8{} 31 | log.Info("Parsing test file...") 32 | for i := 0; e == nil && i < 10000; i++ { 33 | //for i := 0; e == nil && i < 480187; i++ { 34 | s, e = Readln(r) 35 | _, scores := parseLine(s) 36 | records = append(records, scores) 37 | } 38 | log.Info("Generating tree...") 39 | tr, _ := ProcessNewTrees(records, 50, MAXSCORE, 3) 40 | tr.setTestMode() 41 | log.Info("Tree generated...") 42 | 43 | quadError := 0.0 44 | comparedItems := 0 45 | for i := 0; e == nil && i < 1000; i++ { 46 | s, e = Readln(r) 47 | _, scores := parseLine(s) 48 | elements := tr.GetBestRecommendation(scores, 10) 49 | 50 | for _, elemID := range elements { 51 | if score, rated := scores[elemID]; rated { 52 | quadError += (1.0 - (float64(score) / MAXSCORE)) * (1.0 - (float64(score) / MAXSCORE)) 53 | comparedItems++ 54 | } 55 | } 56 | } 57 | 58 | // Estimate the Root-mean-square deviation, we will use 0.3 for this test because the training set and the number of trees is too low 59 | rmsd := math.Sqrt(quadError / float64(comparedItems)) 60 | if rmsd > 0.3 { 61 | t.Error("The RMSD is bigger than the expected, obtained:", rmsd) 62 | } 63 | 64 | return 65 | } 66 | 67 | func Readln(r *bufio.Reader) (string, error) { 68 | var isPrefix = true 69 | var err error 70 | var line, ln []byte 71 | for isPrefix && err == nil { 72 | line, isPrefix, err = r.ReadLine() 73 | ln = append(ln, line...) 74 | } 75 | 76 | return string(ln), err 77 | } 78 | 79 | func parseLine(line string) (recordID uint64, values map[uint64]uint8) { 80 | parts := strings.SplitN(line, ":", 2) 81 | recordIDOrig, _ := strconv.ParseInt(parts[0], 10, 64) 82 | recordID = uint64(recordIDOrig) 83 | 84 | valuesAux := make(map[string]uint8) 85 | if len(parts) < 2 { 86 | log.Fatal(line) 87 | } 88 | json.Unmarshal([]byte(parts[1]), &valuesAux) 89 | values = make(map[uint64]uint8) 90 | for k, v := range valuesAux { 91 | kI, _ := strconv.ParseInt(k, 10, 64) 92 | values[uint64(kI)] = uint8(v) 93 | } 94 | 95 | return 96 | } 97 | -------------------------------------------------------------------------------- /api/api.go: -------------------------------------------------------------------------------- 1 | package api 2 | 3 | import ( 4 | "fmt" 5 | "github.com/alonsovidales/pit/accounts_manager" 6 | "github.com/alonsovidales/pit/cfg" 7 | "github.com/alonsovidales/pit/log" 8 | "github.com/alonsovidales/pit/shards_manager" 9 | "net/http" 10 | ) 11 | 12 | const ( 13 | cHealtyPath = "/check_healty" 14 | cContact = "/contact" 15 | ) 16 | 17 | // API Structure that manage the HTTP API 18 | type API struct { 19 | shardsManager *shardsmanager.Manager 20 | accountsManager *accountsmanager.Manager 21 | staticPath string 22 | 23 | muxHTTPServer *http.ServeMux 24 | } 25 | 26 | // Init Initializes the API and starts listening on the specified ports serving 27 | // both the HTTP API and the static content 28 | func Init(shardsManager *shardsmanager.Manager, accountsManager *accountsmanager.Manager, staticPath string, httpPort, httpsPort int, cert, key string) (api *API, sslAPI *API) { 29 | api = &API{ 30 | shardsManager: shardsManager, 31 | accountsManager: accountsManager, 32 | muxHTTPServer: http.NewServeMux(), 33 | staticPath: staticPath, 34 | } 35 | api.registerAPIs(false) 36 | log.Info("Starting API server on port:", httpPort) 37 | go http.ListenAndServe(fmt.Sprintf(":%d", httpPort), api.muxHTTPServer) 38 | 39 | // SSL Server, will not serve the /rec method by performance issues 40 | sslAPI = &API{ 41 | shardsManager: shardsManager, 42 | accountsManager: accountsManager, 43 | muxHTTPServer: http.NewServeMux(), 44 | staticPath: staticPath, 45 | } 46 | sslAPI.registerAPIs(true) 47 | log.Info("Starting SSL API server on port:", httpsPort) 48 | go http.ListenAndServeTLS(fmt.Sprintf(":%d", httpsPort), cert, key, sslAPI.muxHTTPServer) 49 | 50 | return 51 | } 52 | 53 | func (api *API) contact(w http.ResponseWriter, r *http.Request) { 54 | w.Header().Set("Access-Control-Allow-Origin", "*") 55 | 56 | email := r.FormValue("mail") 57 | content := r.FormValue("content") 58 | 59 | if email != "" && content != "" { 60 | if api.accountsManager.SendEmail(cfg.GetStr("mail", "addr"), content, fmt.Sprintf("Contact form from: %s", email)) { 61 | w.WriteHeader(200) 62 | w.Write([]byte("OK")) 63 | return 64 | } 65 | } 66 | 67 | w.WriteHeader(500) 68 | w.Write([]byte("KO")) 69 | return 70 | } 71 | 72 | // registerAPIs Recister all the handles into the corresponding endpoints 73 | func (api *API) registerAPIs(ssl bool) { 74 | if !ssl { 75 | api.muxHTTPServer.HandleFunc(shardsmanager.CRecPath, api.shardsManager.ScoresAPIHandler) 76 | api.muxHTTPServer.HandleFunc(shardsmanager.CScoresPath, api.shardsManager.ScoresAPIHandler) 77 | } 78 | 79 | api.muxHTTPServer.HandleFunc(shardsmanager.CGroupInfoPath, api.shardsManager.GroupInfoAPIHandler) 80 | 81 | api.muxHTTPServer.HandleFunc(cHealtyPath, func(w http.ResponseWriter, r *http.Request) { 82 | w.WriteHeader(200) 83 | w.Write([]byte("OK")) 84 | }) 85 | 86 | api.muxHTTPServer.HandleFunc(shardsmanager.CRegenerateGroupKey, api.shardsManager.RegenerateGroupKey) 87 | api.muxHTTPServer.HandleFunc(shardsmanager.CDelGroup, api.shardsManager.DelGroup) 88 | api.muxHTTPServer.HandleFunc(shardsmanager.CGetGroupsByUser, api.shardsManager.GetGroupsByUser) 89 | api.muxHTTPServer.HandleFunc(shardsmanager.CAddUpdateGroup, api.shardsManager.AddUpdateGroup) 90 | api.muxHTTPServer.HandleFunc(shardsmanager.CSetShardsGroup, api.shardsManager.SetShards) 91 | api.muxHTTPServer.HandleFunc(shardsmanager.CRemoveShardsContent, api.shardsManager.RemoveShardsContent) 92 | 93 | api.muxHTTPServer.HandleFunc(accountsmanager.CBillingInfo, api.accountsManager.BillingInfo) 94 | api.muxHTTPServer.HandleFunc(accountsmanager.CRegisterPath, api.accountsManager.Register) 95 | api.muxHTTPServer.HandleFunc(accountsmanager.CVerifyPath, api.accountsManager.Verify) 96 | api.muxHTTPServer.HandleFunc(accountsmanager.CLogsPath, api.accountsManager.Logs) 97 | api.muxHTTPServer.HandleFunc(accountsmanager.CRecoverPassPath, api.accountsManager.RecoverPass) 98 | api.muxHTTPServer.HandleFunc(accountsmanager.CChangePass, api.accountsManager.ChangePass) 99 | api.muxHTTPServer.HandleFunc(accountsmanager.CDisablePath, api.accountsManager.Disable) 100 | 101 | api.muxHTTPServer.HandleFunc(cContact, api.contact) 102 | 103 | api.muxHTTPServer.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { 104 | filePath := r.URL.Path[1:] 105 | path := api.staticPath + filePath 106 | lastPosSlash := -1 107 | lastPosDot := -1 108 | 109 | for i := 0; i < len(path); i++ { 110 | switch path[i] { 111 | case '/': 112 | lastPosSlash = i 113 | case '.': 114 | lastPosDot = i 115 | } 116 | } 117 | 118 | if filePath != "" && lastPosDot < lastPosSlash { 119 | path += ".html" 120 | } 121 | 122 | http.ServeFile(w, r, path) 123 | }) 124 | } 125 | -------------------------------------------------------------------------------- /billing/billing.go: -------------------------------------------------------------------------------- 1 | package billing 2 | 3 | import ( 4 | "encoding/json" 5 | "fmt" 6 | "github.com/alonsovidales/pit/log" 7 | "io/ioutil" 8 | "net/http" 9 | "strings" 10 | "time" 11 | ) 12 | 13 | // Billing Structure used to manage all the communication with the billing 14 | // provider, in this case PayPal 15 | type Billing struct { 16 | token string 17 | tokenExpireTs int64 18 | url string 19 | baseURI string 20 | clientID string 21 | secret string 22 | email string 23 | bussinessName string 24 | addrAddr string 25 | addrCity string 26 | addrState string 27 | addrZip string 28 | addrCountryCode string 29 | } 30 | 31 | // InvMerchantInfo Information of the client 32 | type InvMerchantInfo struct { 33 | Email string `json:"email"` 34 | BusinessName string `json:"business_name"` 35 | Address map[string]string `json:"address"` 36 | } 37 | 38 | // InvItemPrize Price for the item to be billed 39 | type InvItemPrize struct { 40 | Currency string `json:"currency"` 41 | Value int `json:"value"` 42 | } 43 | 44 | // InvItem Item name and quantity 45 | type InvItem struct { 46 | Name string `json:"name"` 47 | Quantity float64 `json:"quantity"` 48 | UnitPrice *InvItemPrize `json:"unit_price"` 49 | } 50 | 51 | // Invoice Information for an invoice 52 | type Invoice struct { 53 | MerchantInfo *InvMerchantInfo `json:"merchant_info"` 54 | BillingInfo []map[string]string `json:"billing_info"` 55 | Items []*InvItem `json:"items"` 56 | Note string `json:"note"` 57 | } 58 | 59 | // GetBilling Returns an object that allows to manage all the information of 60 | // the billing provider 61 | func GetBilling(baseURI, clientID, secret, email, bussinessName, addrAddr, addrCity, addrState, addrZip, addrCountryCode string) (bi *Billing) { 62 | bi = &Billing{ 63 | baseURI: baseURI, 64 | clientID: clientID, 65 | secret: secret, 66 | email: email, 67 | bussinessName: bussinessName, 68 | addrAddr: addrAddr, 69 | addrCity: addrCity, 70 | addrState: addrState, 71 | addrZip: addrZip, 72 | addrCountryCode: addrCountryCode, 73 | } 74 | 75 | return 76 | } 77 | 78 | // SendNewBill Generates and sends a new bill to the provider to be processed 79 | func (bi *Billing) SendNewBill(invTitle, targetEmail string, items []*InvItem) (billID string, err error) { 80 | inv := &Invoice{ 81 | MerchantInfo: &InvMerchantInfo{ 82 | Email: bi.email, 83 | BusinessName: bi.addrAddr, 84 | Address: map[string]string{ 85 | "line1": bi.addrAddr, 86 | "city": bi.addrCity, 87 | "state": bi.addrState, 88 | "postal_code": bi.addrZip, 89 | "country_code": bi.addrCountryCode, 90 | }, 91 | }, 92 | BillingInfo: []map[string]string{ 93 | map[string]string{ 94 | "email": targetEmail, 95 | }, 96 | }, 97 | Items: items, 98 | Note: invTitle, 99 | } 100 | 101 | invStr, _ := json.Marshal(inv) 102 | 103 | if body, err := bi.callToAPI("invoicing/invoices", "POST", string(invStr)); err == nil { 104 | responseInfo := make(map[string]interface{}) 105 | err = json.Unmarshal(body, &responseInfo) 106 | if err != nil { 107 | log.Error("Problem trying to response from PayPal API, Error:", err) 108 | } 109 | 110 | billID = responseInfo["id"].(string) 111 | sendRes, err := bi.callToAPI(fmt.Sprintf("invoicing/invoices/%s/send", billID), "POST", "") 112 | 113 | log.Info("Sent!!!:", fmt.Sprintf("invoicing/invoices/%s/send", billID), string(sendRes), err) 114 | } 115 | 116 | return 117 | } 118 | 119 | // getToken Retreves and returns the token from the provider 120 | func (bi *Billing) getToken() { 121 | client := &http.Client{} 122 | 123 | req, err := http.NewRequest("POST", bi.baseURI+"oauth2/token", strings.NewReader("grant_type=client_credentials")) 124 | req.Header.Add("Accept", `application/json`) 125 | req.Header.Add("Accept-Language", `en_US`) 126 | req.SetBasicAuth(bi.clientID, bi.secret) 127 | resp, err := client.Do(req) 128 | if err != nil { 129 | log.Error("Problem trying to request a new token from the PayPal API, Error:", err) 130 | } 131 | 132 | body, err := ioutil.ReadAll(resp.Body) 133 | if err != nil { 134 | log.Error("Problem trying to parse the response form the PayPal API, Error:", err) 135 | } 136 | 137 | responseInfo := make(map[string]interface{}) 138 | err = json.Unmarshal(body, &responseInfo) 139 | if err != nil { 140 | log.Error("Problem trying to read token from PayPal API, Error:", err) 141 | } 142 | 143 | bi.token = responseInfo["access_token"].(string) 144 | bi.tokenExpireTs = time.Now().Unix() + int64(responseInfo["expires_in"].(float64)) 145 | } 146 | 147 | // callToAPI Performs an HTTP request to the billing provider API 148 | func (bi *Billing) callToAPI(uri string, method string, reqBody string) (body []byte, err error) { 149 | if time.Now().Unix() > bi.tokenExpireTs { 150 | bi.getToken() 151 | } 152 | 153 | client := &http.Client{} 154 | 155 | url := bi.baseURI + uri 156 | log.Info(reqBody) 157 | req, err := http.NewRequest(method, url, strings.NewReader(reqBody)) 158 | req.Header.Add("Content-Type", `application/json`) 159 | req.Header.Add("Authorization", "Bearer "+bi.token) 160 | resp, err := client.Do(req) 161 | if err != nil { 162 | log.Error("Problem trying to do a request to the PayPal API, Url:", url, "Error:", err) 163 | } 164 | 165 | body, err = ioutil.ReadAll(resp.Body) 166 | if err != nil { 167 | log.Error("Problem trying to parse the response form the PayPal API, Error:", err) 168 | } 169 | 170 | return 171 | } 172 | -------------------------------------------------------------------------------- /billing/billing_test.go: -------------------------------------------------------------------------------- 1 | package billing 2 | 3 | import ( 4 | "testing" 5 | ) 6 | 7 | func TestDebugLevel(t *testing.T) { 8 | bi := GetBilling( 9 | "https://api.sandbox.paypal.com/v1/", 10 | "-wTuNOeVl", 11 | "eV6y4frwKhDFjcCSO4Bv8NYL2okskviYOP646H8", 12 | "info-facilitator@pitia.info", 13 | "Pitia.info", 14 | "1234 Main St.", 15 | "Portland", 16 | "OR", 17 | "97217", 18 | "US", 19 | ) 20 | 21 | bi.SendNewBill("test bill", "info@pitia.info", []*InvItem{ 22 | &InvItem{ 23 | Name: "Test Item", 24 | Quantity: 10.5, 25 | UnitPrice: &InvItemPrize{ 26 | Currency: "USD", 27 | Value: 1, 28 | }, 29 | }, 30 | &InvItem{ 31 | Name: "Test Item 2", 32 | Quantity: 12.5, 33 | UnitPrice: &InvItemPrize{ 34 | Currency: "USD", 35 | Value: 2, 36 | }, 37 | }, 38 | }) 39 | } 40 | -------------------------------------------------------------------------------- /bin/pit-cli.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "fmt" 5 | "github.com/alonsovidales/pit/cfg" 6 | "github.com/alonsovidales/pit/models/instances" 7 | "github.com/alonsovidales/pit/models/shard_info" 8 | "github.com/alonsovidales/pit/models/users" 9 | "gopkg.in/alecthomas/kingpin.v1" 10 | "os" 11 | "strings" 12 | "text/tabwriter" 13 | ) 14 | 15 | // Terminal text colours 16 | const ( 17 | // CLR0 Default terminal colour 18 | CLR0 = "\x1b[30;1m" 19 | // CLRR Red colour 20 | CLRR = "\x1b[31;1m" 21 | // CLRG Green colour 22 | CLRG = "\x1b[32;1m" 23 | // CLRY Yellow colour 24 | CLRY = "\x1b[33;1m" 25 | // CLRB Blue colour 26 | CLRB = "\x1b[34;1m" 27 | // CLRN Neutral colour 28 | CLRN = "\x1b[0m" 29 | ) 30 | 31 | func main() { 32 | app := kingpin.New("pit-cli", "Pit-cli is a tool to manage all the different features of the recommener system Pit") 33 | 34 | env := app.Flag("env", "Configuration environment: pro, pre, dev").Required().Enum("pro", "pre", "dev") 35 | 36 | cmdInstances := app.Command("instances", "Manage all the instances of the cluster") 37 | cmdInstancesList := cmdInstances.Command("list", "Lists all the instances on the clusted") 38 | 39 | cmdUsers := app.Command("users", "Users management") 40 | cmdUsersList := cmdUsers.Command("list", "lists all the users") 41 | 42 | cmdUsersAdd := cmdUsers.Command("add", "Adds a new user") 43 | cmdUsersAddUID := cmdUsersAdd.Arg("user-ID", "User ID").Required().String() 44 | cmdUsersAddKey := cmdUsersAdd.Arg("key", "User Password").Required().String() 45 | 46 | cmdUsersShow := cmdUsers.Command("show", "Shows all the sotred information for a specific user") 47 | cmdUsersShowUID := cmdUsersShow.Arg("user-ID", "User ID").Required().String() 48 | cmdUsersEnable := cmdUsers.Command("enable", "Enables a disabled user account") 49 | cmdUsersEnableUID := cmdUsersEnable.Arg("user-ID", "User ID").Required().String() 50 | cmdUsersDisable := cmdUsers.Command("disable", "Disables an enabled user account") 51 | cmdUsersDisableUID := cmdUsersDisable.Arg("user-ID", "User ID").Required().String() 52 | 53 | cmdGroups := app.Command("groups", "Manage all the recommendation groups allocated by Pit") 54 | cmdGroupsList := cmdGroups.Command("list", "lists all the shards in the system") 55 | cmdGroupsListUser := cmdGroupsList.Flag("userid", `List the instances for this user`).String() 56 | 57 | cmdGroupsDel := cmdGroups.Command("del", "Removes one of the groups") 58 | cmdGroupsDelGroupID := cmdGroupsDel.Arg("group-id", `ID of the group to be removed`).Required().String() 59 | 60 | cmdGroupsAdd := cmdGroups.Command("update", "Adds or updates an existing shard") 61 | cmdGroupsAddMaxScore := cmdGroupsAdd.Flag("max-score", `Max possible score`).Required().Int() 62 | cmdGroupsAddNumShards := cmdGroupsAdd.Flag("num-shards", `Total number of shards`).Required().Int() 63 | cmdGroupsAddMaxElements := cmdGroupsAdd.Flag("num-elems", `Max number of elements that can be allocated by shard`).Required().Int() 64 | cmdGroupsAddMaxReqSec := cmdGroupsAdd.Flag("max-req-sec", `Max number of requests by second`).Required().Int() 65 | cmdGroupsAddMaxInsertReqSec := cmdGroupsAdd.Flag("max-ins-req-sec", `Max number of insert requests`).Required().Int() 66 | cmdGroupsAddUserID := cmdGroupsAdd.Flag("user-id", `User ID of the owner of this group`).Required().String() 67 | cmdGroupsAddGroupID := cmdGroupsAdd.Flag("group-id", `ID of the group to be updated or added`).Required().String() 68 | 69 | kingpin.MustParse(app.Parse(os.Args[1:])) 70 | cfg.Init("pit", *env) 71 | switch kingpin.MustParse(app.Parse(os.Args[1:])) { 72 | case cmdInstancesList.FullCommand(): 73 | listInstances() 74 | 75 | case cmdGroupsList.FullCommand(): 76 | listGroups(*cmdGroupsListUser) 77 | 78 | case cmdGroupsDel.FullCommand(): 79 | delGroup(*cmdGroupsDelGroupID) 80 | 81 | case cmdGroupsAdd.FullCommand(): 82 | addGroup( 83 | *cmdGroupsAddUserID, 84 | *cmdGroupsAddGroupID, 85 | *cmdGroupsAddNumShards, 86 | uint64(*cmdGroupsAddMaxElements), 87 | uint64(*cmdGroupsAddMaxReqSec), 88 | uint64(*cmdGroupsAddMaxInsertReqSec), 89 | uint8(*cmdGroupsAddMaxScore)) 90 | 91 | case cmdUsersAdd.FullCommand(): 92 | addUser(*cmdUsersAddUID, *cmdUsersAddKey) 93 | 94 | case cmdUsersList.FullCommand(): 95 | listUsers() 96 | 97 | case cmdUsersShow.FullCommand(): 98 | showUserInfo(*cmdUsersShowUID) 99 | 100 | case cmdUsersEnable.FullCommand(): 101 | enableUser(*cmdUsersEnableUID) 102 | 103 | case cmdUsersDisable.FullCommand(): 104 | disableUser(*cmdUsersDisableUID) 105 | 106 | default: 107 | fmt.Printf("Not command specified, use: \"%s --help\" to get help\n", strings.Join(os.Args, " ")) 108 | } 109 | } 110 | 111 | func listUsers() { 112 | md := users.GetModel( 113 | cfg.GetStr("aws", "prefix"), 114 | cfg.GetStr("aws", "region")) 115 | 116 | w := new(tabwriter.Writer) 117 | w.Init(os.Stdout, 0, 8, 3, '\t', 0) 118 | fmt.Fprintln(w, "Uid\tEnabled\tRegTs\tRegIP\tLogLines") 119 | fmt.Fprintln(w, "---\t-------\t-----\t-----\t--------") 120 | 121 | for uid, user := range md.GetRegisteredUsers() { 122 | lines := 0 123 | for _, v := range user.GetAllActivity() { 124 | lines += len(v) 125 | } 126 | fmt.Fprintf( 127 | w, 128 | "%s\t%s\t%d\t%s\t%d\n", 129 | uid, 130 | user.Enabled, 131 | user.RegTs, 132 | user.RegIP, 133 | lines) 134 | } 135 | w.Flush() 136 | } 137 | 138 | func addUser(cmdUsersAddUID string, cmdUsersAddKey string) { 139 | md := users.GetModel( 140 | cfg.GetStr("aws", "prefix"), 141 | cfg.GetStr("aws", "region")) 142 | 143 | if _, err := md.RegisterUser(cmdUsersAddUID, cmdUsersAddKey, "127.0.0.1"); err != nil { 144 | fmt.Println("Problem trying to register the user:", err) 145 | } else { 146 | fmt.Println("User registered") 147 | } 148 | } 149 | 150 | func showUserInfo(uid string) { 151 | md := users.GetModel( 152 | cfg.GetStr("aws", "prefix"), 153 | cfg.GetStr("aws", "region")) 154 | 155 | user := md.AdminGetUserInfoByID(uid) 156 | if user == nil { 157 | fmt.Println("User Not Found") 158 | return 159 | } 160 | 161 | fmt.Println("User info:") 162 | fmt.Println("ID:", uid) 163 | fmt.Println("Enabled:", user.Enabled) 164 | fmt.Println("Registered TS:", user.RegTs) 165 | fmt.Println("Registered IP:", user.RegIP) 166 | fmt.Println("Activity Logs:") 167 | 168 | w := new(tabwriter.Writer) 169 | w.Init(os.Stdout, 0, 8, 3, '\t', 0) 170 | fmt.Fprintln(w, "Timestamp\tType\tDescripton") 171 | fmt.Fprintln(w, "---------\t----\t----------") 172 | 173 | for _, lines := range user.GetAllActivity() { 174 | for _, line := range lines { 175 | fmt.Fprintf( 176 | w, 177 | "%d\t%s\t%s\n", 178 | line.Ts, 179 | line.LogType, 180 | line.Desc) 181 | } 182 | } 183 | w.Flush() 184 | } 185 | 186 | func disableUser(uid string) { 187 | fmt.Println("The user with user ID:", uid, "will be disabled") 188 | if askForConfirmation() { 189 | md := users.GetModel( 190 | cfg.GetStr("aws", "prefix"), 191 | cfg.GetStr("aws", "region")) 192 | 193 | user := md.AdminGetUserInfoByID(uid) 194 | if user != nil { 195 | user.DisableUser() 196 | fmt.Println("User Disabled") 197 | } else { 198 | fmt.Println("User not found") 199 | } 200 | } 201 | } 202 | 203 | func enableUser(uid string) { 204 | fmt.Println("The user with user ID:", uid, "will be enabled") 205 | if askForConfirmation() { 206 | md := users.GetModel( 207 | cfg.GetStr("aws", "prefix"), 208 | cfg.GetStr("aws", "region")) 209 | 210 | user := md.AdminGetUserInfoByID(uid) 211 | if user != nil { 212 | user.EnableUser() 213 | fmt.Println("User Enabled") 214 | } else { 215 | fmt.Println("User not found") 216 | } 217 | } 218 | } 219 | 220 | func listInstances() { 221 | md := instances.InitAndKeepAlive( 222 | cfg.GetStr("aws", "prefix"), 223 | cfg.GetStr("aws", "region"), 224 | false) 225 | 226 | fmt.Println(CLRG + "Instances" + CLRN) 227 | fmt.Println(CLRG + "---------" + CLRN) 228 | for _, instance := range md.GetInstances() { 229 | fmt.Println(instance) 230 | } 231 | } 232 | 233 | func listGroups(userID string) { 234 | md := shardinfo.GetModel( 235 | cfg.GetStr("aws", "prefix"), 236 | cfg.GetStr("aws", "region"), 237 | "") 238 | 239 | w := new(tabwriter.Writer) 240 | w.Init(os.Stdout, 0, 8, 3, '\t', 0) 241 | fmt.Fprintln(w, "User ID\tSecret\tGroupID\tMax Score\tTotal Shards\tMax Elements\tMax req sec\tMax Insert Req Sec\tShard owners") 242 | fmt.Fprintln(w, "-------\t------\t-------\t---------\t------------\t------------\t-----------\t------------------\t------------") 243 | 244 | groups := md.GetAllGroups() 245 | for _, groups := range groups { 246 | for _, group := range groups { 247 | shardOwners := "" 248 | for _, shard := range group.Shards { 249 | shardOwners += fmt.Sprintf("%s %d\t", shard.Addr, shard.LastTs) 250 | } 251 | 252 | fmt.Fprintf( 253 | w, 254 | "%s\t%s\t%s\t%d\t%d\t%d\t%d\t%d\t%s\n", 255 | group.UserID, 256 | group.Secret, 257 | group.GroupID, 258 | group.MaxScore, 259 | group.NumShards, 260 | group.MaxElements, 261 | group.MaxReqSec, 262 | group.MaxInsertReqSec, 263 | shardOwners) 264 | } 265 | } 266 | w.Flush() 267 | } 268 | 269 | func delGroup(groupID string) { 270 | fmt.Println("The next group will be deleted:") 271 | md := shardinfo.GetModel( 272 | cfg.GetStr("aws", "prefix"), 273 | cfg.GetStr("aws", "region"), 274 | "") 275 | 276 | group := md.GetGroupByID(groupID) 277 | if group == nil { 278 | fmt.Println("Group not found with ID:", groupID) 279 | return 280 | } 281 | 282 | if askForConfirmation() { 283 | md.RemoveGroup(groupID) 284 | fmt.Println("Group removed") 285 | } 286 | } 287 | 288 | func addGroup(userID, groupID string, numShards int, maxElements, maxReqSec, maxInsertReqSec uint64, maxScore uint8) { 289 | md := shardinfo.GetModel( 290 | cfg.GetStr("aws", "prefix"), 291 | cfg.GetStr("aws", "region"), 292 | "") 293 | 294 | fmt.Println(CLRG + "The next group will be added:" + CLRN) 295 | fmt.Println("User ID:", userID) 296 | fmt.Println("Group ID:", groupID) 297 | fmt.Println("Num Shards:", numShards) 298 | fmt.Println("Max elements:", maxElements) 299 | fmt.Println("Max requests by sec / shard:", maxReqSec) 300 | fmt.Println("Max Insert requests by sec / shard:", maxInsertReqSec) 301 | fmt.Println("Max score:", maxScore) 302 | 303 | if askForConfirmation() { 304 | _, key, err := md.AddUpdateGroup("Custom", userID, groupID, numShards, maxElements, maxReqSec, maxInsertReqSec, maxScore) 305 | if err != nil { 306 | fmt.Println("Problem adding a new group, Error:", err) 307 | } else { 308 | fmt.Println("Group added, key:", key) 309 | } 310 | } 311 | } 312 | 313 | func askForConfirmation() bool { 314 | var response string 315 | 316 | fmt.Print(CLRR + "\nAre you completly sure? (y/n): " + CLRN) 317 | if _, err := fmt.Scanln(&response); err != nil { 318 | return false 319 | } 320 | 321 | possibleAnsw := map[string]bool{ 322 | "y": true, 323 | "Y": true, 324 | "yes": true, 325 | "Yes": true, 326 | "YES": true, 327 | } 328 | 329 | _, ok := possibleAnsw[response] 330 | 331 | return ok 332 | } 333 | -------------------------------------------------------------------------------- /bin/pit.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "github.com/alonsovidales/pit/accounts_manager" 5 | "github.com/alonsovidales/pit/api" 6 | "github.com/alonsovidales/pit/cfg" 7 | "github.com/alonsovidales/pit/log" 8 | "github.com/alonsovidales/pit/models/users" 9 | "github.com/alonsovidales/pit/shards_manager" 10 | "os" 11 | "os/signal" 12 | "runtime" 13 | "syscall" 14 | ) 15 | 16 | func main() { 17 | if len(os.Args) > 1 { 18 | cfg.Init("pit", os.Args[1]) 19 | 20 | log.SetLogger( 21 | log.Levels[cfg.GetStr("logger", "level")], 22 | cfg.GetStr("logger", "log_file"), 23 | cfg.GetInt("logger", "max_log_size_mb"), 24 | ) 25 | } else { 26 | cfg.Init("pit", "dev") 27 | } 28 | runtime.GOMAXPROCS(runtime.NumCPU()) 29 | 30 | usersModel := users.GetModel( 31 | cfg.GetStr("aws", "prefix"), 32 | cfg.GetStr("aws", "region")) 33 | 34 | accountsManager := accountsmanager.Init( 35 | cfg.GetStr("rec-api", "base-url"), 36 | cfg.GetStr("mail", "addr"), 37 | cfg.GetStr("mail", "server"), 38 | cfg.GetInt("mail", "port"), 39 | usersModel) 40 | 41 | shardsManager := shardsmanager.Init( 42 | cfg.GetStr("aws", "prefix"), 43 | cfg.GetStr("aws", "region"), 44 | cfg.GetStr("aws", "s3-backups-path"), 45 | int(cfg.GetInt("rec-api", "port")), 46 | usersModel, 47 | cfg.GetStr("mail", "addr")) 48 | 49 | api.Init( 50 | shardsManager, 51 | accountsManager, 52 | cfg.GetStr("rec-api", "static"), 53 | int(cfg.GetInt("rec-api", "port")), 54 | int(cfg.GetInt("rec-api", "ssl-port")), 55 | cfg.GetStr("rec-api", "ssl-cert"), 56 | cfg.GetStr("rec-api", "ssl-key")) 57 | 58 | log.Info("System started...") 59 | c := make(chan os.Signal, 1) 60 | signal.Notify(c, os.Interrupt, os.Kill, syscall.SIGTERM) 61 | // Block until a signal is received. 62 | <-c 63 | 64 | log.Info("Stopping all the services") 65 | shardsManager.Stop() 66 | } 67 | -------------------------------------------------------------------------------- /cfg/cfg.go: -------------------------------------------------------------------------------- 1 | package cfg 2 | 3 | // Package designed as easy to use interface for parse INI files 4 | 5 | import ( 6 | "fmt" 7 | "github.com/alonsovidales/pit/log" 8 | "github.com/alyu/configparser" 9 | "strconv" 10 | ) 11 | 12 | var cfg *configparser.Configuration 13 | var sections = make(map[string]*configparser.Section) 14 | 15 | // Init Loads a INI file onto memory, first try to liad the config file from 16 | // the etc/ directory on the current path, and if the file can't be found, try 17 | // to load it from the /etc/ directory 18 | // The name of the file to be used has to be the specified "appName_env.ini" 19 | func Init(appName, env string) (err error) { 20 | // Trying to read the config file form the /etc directory on a first 21 | // instance 22 | if cfg, err = configparser.Read(fmt.Sprintf("etc/%s_%s.ini", appName, env)); err != nil { 23 | cfg, err = configparser.Read(fmt.Sprintf("/etc/%s_%s.ini", appName, env)) 24 | } 25 | 26 | return 27 | } 28 | 29 | // GetStr Returns the value of the section, subsection as string 30 | func GetStr(sec, subsec string) string { 31 | return loadSection(sec).ValueOf(subsec) 32 | } 33 | 34 | // GetUint64 Returns the value of the section, subsection as uint64 35 | func GetUint64(sec, subsec string) (v uint64) { 36 | return uint64(GetInt(sec, subsec)) 37 | } 38 | 39 | // GetInt Returns the value of the section, subsection as int64 40 | func GetInt(sec, subsec string) (v int64) { 41 | if v, err := strconv.ParseInt(loadSection(sec).ValueOf(subsec), 10, 64); err == nil { 42 | return v 43 | } 44 | 45 | log.Error("Configuration parameter:", sec, subsec, "can't be parsed as integer") 46 | return 47 | } 48 | 49 | // GetFloat Returns the value of the section, subsection as float 50 | func GetFloat(sec, subsec string) (v float64) { 51 | if v, err := strconv.ParseFloat(loadSection(sec).ValueOf(subsec), 64); err == nil { 52 | return v 53 | } 54 | 55 | log.Error("Configuration parameter:", sec, subsec, "can't be parsed as integer") 56 | return 57 | } 58 | 59 | // GetBool Returns the value of the section, subsection as boolean 60 | func GetBool(sec, subsec string) (v bool) { 61 | vSec := loadSection(sec).ValueOf(subsec) 62 | 63 | return vSec == "1" || vSec == "true" 64 | } 65 | 66 | // loadSection loads a section of the config file 67 | func loadSection(name string) (section *configparser.Section) { 68 | if section, ok := sections[name]; ok { 69 | return section 70 | } 71 | 72 | if cfg == nil { 73 | log.Fatal("Configuration file not yet loaded, call to the Init method before try to use the config manager") 74 | } 75 | 76 | if sec, err := cfg.Section(name); err == nil { 77 | sections[name] = sec 78 | 79 | } else { 80 | log.Fatal("Configuration subsection:", name, "can't be parsed") 81 | } 82 | 83 | return sections[name] 84 | } 85 | -------------------------------------------------------------------------------- /cfg/cfg_test.go: -------------------------------------------------------------------------------- 1 | package cfg 2 | 3 | import ( 4 | "testing" 5 | ) 6 | 7 | func TestDebugLevel(t *testing.T) { 8 | if err := Init("config", "dev"); err != nil { 9 | t.Error("Test config file can't be loaded") 10 | t.Fail() 11 | } 12 | 13 | if GetStr("section1", "val_str") != "test" { 14 | t.Error("Expected value for section \"section1\" and \"val_str\" filed was \"test\"") 15 | } 16 | 17 | if GetInt("section1", "val_int") != 123 { 18 | t.Error("Expected value for section \"section1\" and \"val_int\" filed was \"123\"") 19 | } 20 | 21 | if !GetBool("section1", "val_bool_true") { 22 | t.Error("Expected value for section \"section1\" and \"val_bool_true\" filed was \"true\"") 23 | } 24 | 25 | if GetBool("section1", "val_bool_false") { 26 | t.Error("Expected value for section \"section1\" and \"val_bool_false\" filed was \"false\"") 27 | } 28 | 29 | if !GetBool("section2", "val_bool_true") { 30 | t.Error("Expected value for section \"section2\" and \"val_bool_true\" filed was \"true\"") 31 | } 32 | 33 | if GetBool("section2", "val_bool_false") { 34 | t.Error("Expected value for section \"section2\" and \"val_bool_false\" filed was \"false\"") 35 | } 36 | } 37 | -------------------------------------------------------------------------------- /cfg/etc/config_dev.ini: -------------------------------------------------------------------------------- 1 | [section1] 2 | val_int=123 3 | val_str=test 4 | val_bool_true=1 5 | val_bool_false=0 6 | [section2] 7 | val_bool_true=true 8 | val_bool_false=false 9 | -------------------------------------------------------------------------------- /debian/changelog: -------------------------------------------------------------------------------- 1 | pit (0.0.1) unstable; urgency=low 2 | 3 | * Release candidate 0.0.1 4 | 5 | -- Alonso Vidales Fri, 21 Mar 2015 23:01:00 +0000 6 | -------------------------------------------------------------------------------- /debian/control: -------------------------------------------------------------------------------- 1 | Maintainer: Alonso Vidales 2 | Priority: optional 3 | Section: misc 4 | Source: pit 5 | Package: pit 6 | Architecture: amd64 7 | Depends: supervisor 8 | Description: Pit is a hight performance distributed recommender system 9 | Version: 0.0.1 10 | -------------------------------------------------------------------------------- /debian/postinst: -------------------------------------------------------------------------------- 1 | echo "Package installed, restarting supervisor..." 2 | supervisorctl restart all 3 | -------------------------------------------------------------------------------- /debian/rules: -------------------------------------------------------------------------------- 1 | #!/usr/bin/make -f 2 | # See debhelper(7) (uncomment to enable) 3 | # output every command that modifies files on the build system. 4 | #DH_VERBOSE = 1 5 | 6 | # see EXAMPLES in dpkg-buildflags(1) and read /usr/share/dpkg/* 7 | DPKG_EXPORT_BUILDFLAGS = 1 8 | include /usr/share/dpkg/default.mk 9 | 10 | # see FEATURE AREAS in dpkg-buildflags(1) 11 | #export DEB_BUILD_MAINT_OPTIONS = hardening=+all 12 | 13 | # see ENVIRONMENT in dpkg-buildflags(1) 14 | # package maintainers to append CFLAGS 15 | #export DEB_CFLAGS_MAINT_APPEND = -Wall -pedantic 16 | # package maintainers to append LDFLAGS 17 | #export DEB_LDFLAGS_MAINT_APPEND = -Wl,--as-needed 18 | 19 | # main packaging script based on dh7 syntax 20 | %: 21 | dh $@ 22 | -------------------------------------------------------------------------------- /debian_static/changelog: -------------------------------------------------------------------------------- 1 | pit (0.0.1) unstable; urgency=low 2 | 3 | * Release candidate 0.0.1 4 | 5 | -- Alonso Vidales Fri, 21 Mar 2015 23:01:00 +0000 6 | -------------------------------------------------------------------------------- /debian_static/control: -------------------------------------------------------------------------------- 1 | Maintainer: Alonso Vidales 2 | Priority: optional 3 | Section: misc 4 | Source: pit 5 | Package: pit-static 6 | Architecture: amd64 7 | Description: Pit is a hight performance distributed recommender system 8 | Version: 0.0.1 9 | -------------------------------------------------------------------------------- /debian_static/postinst: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alonsovidales/pit/85f0585734756163bceb43dd3307a140bbe37b38/debian_static/postinst -------------------------------------------------------------------------------- /debian_static/rules: -------------------------------------------------------------------------------- 1 | #!/usr/bin/make -f 2 | # See debhelper(7) (uncomment to enable) 3 | # output every command that modifies files on the build system. 4 | #DH_VERBOSE = 1 5 | 6 | # see EXAMPLES in dpkg-buildflags(1) and read /usr/share/dpkg/* 7 | DPKG_EXPORT_BUILDFLAGS = 1 8 | include /usr/share/dpkg/default.mk 9 | 10 | # see FEATURE AREAS in dpkg-buildflags(1) 11 | #export DEB_BUILD_MAINT_OPTIONS = hardening=+all 12 | 13 | # see ENVIRONMENT in dpkg-buildflags(1) 14 | # package maintainers to append CFLAGS 15 | #export DEB_CFLAGS_MAINT_APPEND = -Wall -pedantic 16 | # package maintainers to append LDFLAGS 17 | #export DEB_LDFLAGS_MAINT_APPEND = -Wl,--as-needed 18 | 19 | # main packaging script based on dh7 syntax 20 | %: 21 | dh $@ 22 | -------------------------------------------------------------------------------- /etc/pit_dev.ini: -------------------------------------------------------------------------------- 1 | [rec-api] 2 | port=80 3 | instance-mem-gb=15 4 | records-by-gb=2000000 5 | 6 | [aws] 7 | prefix=dev 8 | region=eu-west-1 9 | s3-backups-path=/backups_dev 10 | 11 | [logger] 12 | level=DEBUG 13 | log_file=/var/log/pit/pit.log 14 | max_log_size_mb=100 15 | -------------------------------------------------------------------------------- /etc/pit_pro.ini: -------------------------------------------------------------------------------- 1 | [rec-api] 2 | ssl-port=443 3 | port=80 4 | base-url=http://api.pitia.info 5 | static=/var/www/ 6 | ssl-cert=/etc/certs/pitia.cert 7 | ssl-key=/etc/certs/pitia.key 8 | 9 | [mail] 10 | addr=info@pitia.info 11 | server=smtp.gmail.com 12 | port=25 13 | 14 | [mem] 15 | instance-mem-gb=15 16 | records-by-gb=8000000 17 | 18 | [aws] 19 | prefix=pro 20 | region=eu-west-1 21 | s3-backups-path=/backups_pro 22 | 23 | [logger] 24 | level=INFO 25 | log_file=/var/log/pit/pit.log 26 | max_log_size_mb=100 27 | 28 | [paypal] 29 | api-url=https://api.sandbox.paypal.com/v1/oauth2/token 30 | 31 | [group-types] 32 | small-reqs=50 33 | small-records=2000000 34 | small-large-cost-hour=0.0097 35 | 36 | medium-reqs=150 37 | medium-records=6000000 38 | medium-cost-hour=0.0222 39 | 40 | large-reqs=250 41 | large-records=10000000 42 | large-cost-hour=0.0361 43 | 44 | x-large-reqs=500 45 | x-large-records=20000000 46 | x-large-cost-hour=0.0625 47 | 48 | xx-large-reqs=1500 49 | xx-large-records=30000000 50 | xx-large-cost-hour=0.1194 51 | -------------------------------------------------------------------------------- /log/log.go: -------------------------------------------------------------------------------- 1 | package log 2 | 3 | import ( 4 | "fmt" 5 | logger "log" 6 | "os" 7 | "runtime" 8 | "strings" 9 | "sync" 10 | "time" 11 | ) 12 | 13 | const ( 14 | // DEBUG used to specify the "debug" level in order to print all the log messages 15 | DEBUG = 1 16 | // INFO used to specify the "info" level in order to print the info + error + fatal messages 17 | INFO = 2 18 | // ERROR used to specify the "error" level in order to print the error + fatal messages 19 | ERROR = 3 20 | // FATAL used to specify the "fatal" level in order to print only the fatal log lines 21 | FATAL = 4 22 | ) 23 | 24 | var level = 0 25 | var file *os.File 26 | var path string 27 | var maxSize int64 28 | var mutex = new(sync.Mutex) 29 | 30 | // Levels Different allowed debugging levels, the allowed levels are: DEBUG, 31 | // INFO, ERROR, FATAL 32 | var Levels = map[string]int{ 33 | "DEBUG": DEBUG, 34 | "INFO": INFO, 35 | "ERROR": ERROR, 36 | "FATAL": FATAL, 37 | } 38 | 39 | // SetLogger Sets the global logger level, and the path and size of the log 40 | // file to be used as output for the logs, in case of this method is not 41 | // called, all the logs will be print on the standar output 42 | func SetLogger(newLevel int, filePath string, maxSizeMB int64) { 43 | level = newLevel 44 | maxSize = maxSizeMB * 1024000 45 | setLogFile(filePath) 46 | } 47 | 48 | // Debug Adds a new log line to the logs file in case of being in a DEBUG level 49 | // or higer 50 | func Debug(v ...interface{}) { 51 | if level <= DEBUG { 52 | _, file, line, _ := runtime.Caller(1) 53 | fileParts := strings.Split(file, "/") 54 | newLog(fmt.Sprintf("DEBUG: <%s:%d> ", fileParts[len(fileParts)-1], line), v...) 55 | } 56 | } 57 | 58 | // Info Adds a new log line to the logs file in case of being in a INFO level 59 | // or higer 60 | func Info(v ...interface{}) { 61 | if level <= INFO { 62 | newLog("INFO: ", v...) 63 | } 64 | } 65 | 66 | // Error Adds a new log line to the logs file in case of being in a ERROR level 67 | // or higer 68 | func Error(v ...interface{}) { 69 | if level <= ERROR { 70 | _, file, line, _ := runtime.Caller(1) 71 | fileParts := strings.Split(file, "/") 72 | newLog(fmt.Sprintf("ERROR: <%s:%d> ", fileParts[len(fileParts)-1], line), v...) 73 | } 74 | } 75 | 76 | // Fatal Adds a new log line to the logs file and interrupts the execution of 77 | // the application 78 | func Fatal(v ...interface{}) { 79 | if level <= FATAL { 80 | _, file, line, _ := runtime.Caller(1) 81 | fileParts := strings.Split(file, "/") 82 | newLog(fmt.Sprintf("FATAL: <%s:%d> ", fileParts[len(fileParts)-1], line), v...) 83 | } 84 | } 85 | 86 | // setLogFile Sets the specified path as new log file, in case of have defined 87 | // a previous log file, rotates this 88 | func setLogFile(filePath string) { 89 | if file != nil { 90 | file.Close() 91 | os.Rename(path, fmt.Sprintf("%s_%d.old", path, int32(time.Now().Unix()))) 92 | } 93 | 94 | path = filePath 95 | if outFile, err := os.OpenFile(filePath, os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666); err == nil { 96 | file = outFile 97 | logger.SetOutput(file) 98 | } else { 99 | Fatal("Can't open the log file:", filePath) 100 | } 101 | } 102 | 103 | // newLog Adds a new log line to the logger file with the specified level at 104 | // the begging 105 | func newLog(l string, v ...interface{}) { 106 | if file != nil { 107 | mutex.Lock() 108 | fStat, err := file.Stat() 109 | if err != nil { 110 | Fatal("Can't stat logger file") 111 | } 112 | if fStat.Size() > maxSize { 113 | fmt.Println("ROTATE", fStat.Size(), maxSize) 114 | logger.Print("Rotating log file") 115 | setLogFile(path) 116 | } 117 | mutex.Unlock() 118 | } 119 | logger.Print(l, fmt.Sprintln(v...)) 120 | } 121 | -------------------------------------------------------------------------------- /log/log_test.go: -------------------------------------------------------------------------------- 1 | package log 2 | 3 | import ( 4 | "bufio" 5 | "os" 6 | "testing" 7 | ) 8 | 9 | func TestDebugLevel(t *testing.T) { 10 | test := "/tmp/levels.log" 11 | os.Remove(test) 12 | SetLogger(DEBUG, test, 10000) 13 | Debug("test Debug") 14 | Info("test Info") 15 | Error("test Error") 16 | 17 | f, err := os.Open(test) 18 | if err != nil { 19 | t.Error("the logger file:", test, "was not generated, or can't be accessed") 20 | t.Fail() 21 | } 22 | defer f.Close() 23 | 24 | scanner := bufio.NewScanner(f) 25 | l := 0 26 | for scanner.Scan() { 27 | l++ 28 | } 29 | 30 | if l != 3 { 31 | t.Error("expected lines: 3, but:", l, "obtained") 32 | } 33 | } 34 | 35 | func TestInfoLevel(t *testing.T) { 36 | test := "/tmp/levels.log" 37 | os.Remove(test) 38 | SetLogger(INFO, test, 10000) 39 | Debug("test Debug") 40 | Info("test Info") 41 | Error("test Error") 42 | 43 | f, err := os.Open(test) 44 | if err != nil { 45 | t.Error("the logger file:", test, "was not generated, or can't be accessed") 46 | t.Fail() 47 | } 48 | defer f.Close() 49 | 50 | scanner := bufio.NewScanner(f) 51 | l := 0 52 | for scanner.Scan() { 53 | l++ 54 | } 55 | 56 | if l != 2 { 57 | t.Error("Expected lines: 2, but:", l, "obtained") 58 | } 59 | } 60 | 61 | func TestErrorLevel(t *testing.T) { 62 | test := "/tmp/levels.log" 63 | os.Remove(test) 64 | SetLogger(ERROR, test, 10000) 65 | Debug("test Debug") 66 | Info("test Info") 67 | Error("test Error") 68 | 69 | f, err := os.Open(test) 70 | if err != nil { 71 | t.Error("the logger file:", test, "was not generated, or can't be accessed") 72 | t.Fail() 73 | } 74 | defer f.Close() 75 | 76 | scanner := bufio.NewScanner(f) 77 | l := 0 78 | for scanner.Scan() { 79 | l++ 80 | } 81 | 82 | if l != 1 { 83 | t.Error("Expected lines: 1, but:", l, "obtained") 84 | } 85 | } 86 | -------------------------------------------------------------------------------- /makefile: -------------------------------------------------------------------------------- 1 | NO_COLOR=\033[0m 2 | OK_COLOR=\033[32;01m 3 | ERROR_COLOR=\033[31;01m 4 | WARN_COLOR=\033[33;01m 5 | DEPS = $(go list -f '{{range .TestImports}}{{.}} {{end}}' ./... | sort | uniq) 6 | 7 | deps: 8 | @echo "$(OK_COLOR)==> deps" 9 | @echo "$(OK_COLOR)==> Installing dependencies$(NO_COLOR)" 10 | @go get -d -v ./... 11 | @echo $(DEPS) | xargs -n1 go get -d 12 | 13 | updatedeps: 14 | @echo "$(OK_COLOR)==> updatedeps" 15 | @echo "$(OK_COLOR)==> Updating all dependencies$(NO_COLOR)" 16 | @go get -d -v -u ./... 17 | @echo $(DEPS) | xargs -n1 go get -d -u 18 | 19 | format: 20 | @echo "$(OK_COLOR)==> format" 21 | @echo "$(OK_COLOR)==> Formatting$(NO_COLOR)" 22 | go fmt ./... 23 | 24 | test: 25 | @echo "$(OK_COLOR)==> test" 26 | @echo "$(OK_COLOR)==> Testing$(NO_COLOR)" 27 | @find * -maxdepth 0 -mindepth 0 -type d -not -path "*.*" | awk '{print "./" $$0 "/..."}' | xargs go test 28 | 29 | lint: 30 | @echo "$(OK_COLOR)==> lint" 31 | @echo "$(OK_COLOR)==> Linting$(NO_COLOR)" 32 | golint ./... 33 | 34 | deb: 35 | @echo "$(OK_COLOR)==> Building DEB$(NO_COLOR)" 36 | rm -rf dist_package 37 | mkdir -p dist_package/usr/local/bin 38 | mkdir -p dist_package/var/log/pit 39 | GOOS=linux GOARCH=amd64 go build -o dist_package/usr/local/bin/pit bin/pit.go 40 | GOOS=linux GOARCH=amd64 go build -o dist_package/usr/local/bin/pit-cli bin/pit-cli.go 41 | rm -rf tmp 42 | mkdir tmp 43 | cp -a etc dist_package/ 44 | 45 | cd dist_package; tar czvf ../tmp/data.tar.gz * 46 | 47 | cd debian; tar czvf ../tmp/control.tar.gz * 48 | echo 2.0 > tmp/debian-binary 49 | ar -r pit.deb tmp/debian-binary tmp/control.tar.gz tmp/data.tar.gz 50 | rm -rf tmp 51 | @echo "$(OK_COLOR)==> Package created: pit.deb$(NO_COLOR)" 52 | 53 | static_deb: 54 | @echo "$(OK_COLOR)==> Building Static DEB$(NO_COLOR)" 55 | rm -rf dist_package 56 | mkdir -p dist_package/var/www 57 | rm -rf tmp 58 | mkdir tmp 59 | cp -a static/* dist_package/var/www/ 60 | 61 | cd dist_package; tar czvf ../tmp/data.tar.gz * 62 | 63 | cd debian_static; tar czvf ../tmp/control.tar.gz * 64 | echo 2.0 > tmp/debian-binary 65 | ar -r pit_static.deb tmp/debian-binary tmp/control.tar.gz tmp/data.tar.gz 66 | rm -rf tmp 67 | @echo "$(OK_COLOR)==> Static Package created: pit_static.deb$(NO_COLOR)" 68 | 69 | deploy_dev: deb 70 | @ for SERVER in $$PIT_DEV_SERVERS ; do \ 71 | echo "Uploading code to server: $(OK_COLOR)$$SERVER$(NO_COLOR)"; \ 72 | scp -i $$HOME/.ssh/id_rsa_dev_pit pit.deb ubuntu@$$SERVER:/tmp/pit.deb ; \ 73 | done 74 | @ for SERVER in $$PIT_DEV_SERVERS ; do \ 75 | echo "Deploying new code on server: $(OK_COLOR)$$SERVER$(NO_COLOR)"; \ 76 | ssh -i $$HOME/.ssh/id_rsa_dev_pit ubuntu@$$SERVER "sudo dpkg -i /tmp/pit.deb" ; \ 77 | done 78 | 79 | deploy_pro: deb 80 | ssh-add $$HOME/.ssh/id_rsa_pro_pit 81 | @ for SERVER in $$PIT_PRO_SERVERS ; do \ 82 | echo "Uploading code to server: $(OK_COLOR)$$SERVER$(NO_COLOR)"; \ 83 | scp -i $$HOME/.ssh/id_rsa_pro_pit pit.deb root@$$SERVER:/tmp/pit.deb ; \ 84 | done 85 | @ for SERVER in $$PIT_PRO_SERVERS ; do \ 86 | echo "Deploying new code on server: $(OK_COLOR)$$SERVER$(NO_COLOR)"; \ 87 | ssh -i $$HOME/.ssh/id_rsa_pro_pit root@$$SERVER "dpkg -i /tmp/pit.deb" ; \ 88 | done 89 | 90 | deploy_static_pro: static_deb 91 | ssh-add $$HOME/.ssh/id_rsa_pro_pit 92 | @ for SERVER in $$PIT_PRO_SERVERS ; do \ 93 | echo "Uploading code to server: $(OK_COLOR)$$SERVER$(NO_COLOR)"; \ 94 | scp -i $$HOME/.ssh/id_rsa_pro_pit pit_static.deb root@$$SERVER:/tmp/pit_static.deb ; \ 95 | done 96 | @ for SERVER in $$PIT_PRO_SERVERS ; do \ 97 | echo "Deploying new code on server: $(OK_COLOR)$$SERVER$(NO_COLOR)"; \ 98 | ssh -i $$HOME/.ssh/id_rsa_pro_pit root@$$SERVER "dpkg -i /tmp/pit_static.deb" ; \ 99 | done 100 | -------------------------------------------------------------------------------- /models/instances/instances.go: -------------------------------------------------------------------------------- 1 | package instances 2 | 3 | import ( 4 | "fmt" 5 | "github.com/alonsovidales/pit/log" 6 | "github.com/goamz/goamz/aws" 7 | "github.com/goamz/goamz/dynamodb" 8 | "os" 9 | "sort" 10 | "strconv" 11 | "sync" 12 | "time" 13 | ) 14 | 15 | const ( 16 | // cTable DynamoDB table to be used 17 | cTable = "instances" 18 | // cPrimKey Primary key to be used 19 | cPrimKey = "hostName" 20 | // cDefaultWRCapacity The capacity to be provisioned when the table is 21 | // created 22 | cDefaultWRCapacity = 5 23 | // cTTL time in seconds to wait until set as instance as abandoned 24 | cTTL = 30 25 | ) 26 | 27 | // ModelInt Interface used to control the content on the persistence table 28 | type ModelInt interface { 29 | // GetTotalInstances Returns the total number of active instances 30 | GetTotalInstances() int 31 | // GetInstances Returns the host name of all the active instances 32 | GetInstances() (instances []string) 33 | // GetMaxShardsToAcquire Returns the max number of shards that still 34 | // has to be adquired for a group 35 | GetMaxShardsToAcquire(totalShards int) int 36 | } 37 | 38 | // Model Manages the accesses to the DynamoDB table 39 | type Model struct { 40 | prefix string 41 | table *dynamodb.Table 42 | instancesAlive []string 43 | conn *dynamodb.Server 44 | tableName string 45 | mutex sync.Mutex 46 | } 47 | 48 | type byName []string 49 | 50 | func (a byName) Len() int { return len(a) } 51 | func (a byName) Swap(i, j int) { a[i], a[j] = a[j], a[i] } 52 | func (a byName) Less(i, j int) bool { return a[i] < a[j] } 53 | 54 | var hostName string 55 | 56 | func init() { 57 | var err error 58 | hostName, err = os.Hostname() 59 | if err != nil { 60 | log.Fatal("Can't get the hostname of the local machine:", err) 61 | } 62 | } 63 | 64 | // GetHostName Returns the hostname of hte current host machine 65 | func GetHostName() string { 66 | return hostName 67 | } 68 | 69 | // SetHostname Used for testing proposals only, force the host name to be the 70 | // one specified 71 | func SetHostname(hn string) { 72 | hostName = hn 73 | } 74 | 75 | // InitAndKeepAlive Initializes the table, connection, etc and keeps a process 76 | // in background to update all the information 77 | func InitAndKeepAlive(prefix string, awsRegion string, keepAlive bool) (im *Model) { 78 | if awsAuth, err := aws.EnvAuth(); err == nil { 79 | im = &Model{ 80 | prefix: prefix, 81 | tableName: fmt.Sprintf("%s_%s", prefix, cTable), 82 | conn: &dynamodb.Server{ 83 | Auth: awsAuth, 84 | Region: aws.Regions[awsRegion], 85 | }, 86 | } 87 | im.initTable() 88 | 89 | if keepAlive { 90 | im.registerHostName(hostName) 91 | } 92 | im.updateInstances() 93 | if keepAlive { 94 | go func() { 95 | for { 96 | im.registerHostName(hostName) 97 | im.updateInstances() 98 | time.Sleep(time.Second) 99 | } 100 | }() 101 | } 102 | } else { 103 | log.Error("Problem trying to connect with DynamoDB, Error:", err) 104 | return 105 | } 106 | 107 | return 108 | } 109 | 110 | // GetMaxShardsToAcquire Returns the max number of shards that still has to be 111 | // adquired for a group 112 | func (im *Model) GetMaxShardsToAcquire(totalShards int) (total int) { 113 | im.mutex.Lock() 114 | defer im.mutex.Unlock() 115 | 116 | log.Debug("Instances alive:", im.instancesAlive) 117 | if len(im.instancesAlive) == 0 { 118 | return 0 119 | } 120 | 121 | total = totalShards / len(im.instancesAlive) 122 | if im.instancesAlive[len(im.instancesAlive)-1] == hostName { 123 | total += totalShards % len(im.instancesAlive) 124 | } 125 | 126 | return 127 | } 128 | 129 | // GetTotalInstances Returns the total number of active instances 130 | func (im *Model) GetTotalInstances() int { 131 | if len(im.instancesAlive) == 0 { 132 | return 1 133 | } 134 | 135 | return len(im.instancesAlive) 136 | } 137 | 138 | // GetInstances Returns the host name of all the active instances 139 | func (im *Model) GetInstances() (instances []string) { 140 | im.mutex.Lock() 141 | instances = make([]string, len(im.instancesAlive)) 142 | copy(instances, im.instancesAlive) 143 | im.mutex.Unlock() 144 | 145 | return 146 | } 147 | 148 | func (im *Model) delTable() { 149 | if tableDesc, err := im.conn.DescribeTable(im.tableName); err == nil { 150 | if _, err = im.conn.DeleteTable(*tableDesc); err != nil { 151 | log.Error("Can't remove Dynamo table:", im.tableName, "Error:", err) 152 | } 153 | } else { 154 | log.Error("Can't remove Dynamo table:", im.tableName, "Error:", err) 155 | } 156 | } 157 | 158 | func (im *Model) registerHostName(hostName string) { 159 | attribs := []dynamodb.Attribute{ 160 | *dynamodb.NewStringAttribute(cPrimKey, hostName), 161 | *dynamodb.NewStringAttribute("ts", fmt.Sprintf("%d", time.Now().Unix())), 162 | } 163 | 164 | if _, err := im.table.PutItem(hostName, cPrimKey, attribs); err != nil { 165 | log.Fatal("The hostname can't be registered on the instances table, Error:", err) 166 | } 167 | } 168 | 169 | func (im *Model) updateInstances() { 170 | if rows, err := im.table.Scan(nil); err == nil { 171 | instances := []string{} 172 | for _, row := range rows { 173 | lastTs, _ := strconv.ParseInt(row["ts"].Value, 10, 64) 174 | if lastTs, _ = strconv.ParseInt(row["ts"].Value, 10, 64); lastTs+cTTL > time.Now().Unix() { 175 | instances = append(instances, row[cPrimKey].Value) 176 | } else if row[cPrimKey].Value != hostName { 177 | log.Info("Outdated instance detected, removing it, name:", row[cPrimKey].Value) 178 | attKey := &dynamodb.Key{ 179 | HashKey: row[cPrimKey].Value, 180 | RangeKey: "", 181 | } 182 | 183 | _, err = im.table.DeleteItem(attKey) 184 | if err != nil { 185 | log.Error("The instance:", row[cPrimKey].Value, "can't be removed, Error:", err) 186 | } 187 | } 188 | } 189 | 190 | sort.Sort(byName(instances)) 191 | im.mutex.Lock() 192 | im.instancesAlive = instances 193 | im.mutex.Unlock() 194 | } else { 195 | log.Error("Problem trying to get the list of instances from Dynamo DB, Error:", err) 196 | } 197 | } 198 | 199 | func (im *Model) initTable() { 200 | pKey := dynamodb.PrimaryKey{dynamodb.NewStringAttribute(cPrimKey, ""), nil} 201 | im.table = im.conn.NewTable(im.tableName, pKey) 202 | 203 | res, err := im.table.DescribeTable() 204 | if err != nil { 205 | log.Info("Creating a new table on DynamoDB:", im.tableName) 206 | td := dynamodb.TableDescriptionT{ 207 | TableName: im.tableName, 208 | AttributeDefinitions: []dynamodb.AttributeDefinitionT{ 209 | dynamodb.AttributeDefinitionT{cPrimKey, "S"}, 210 | }, 211 | KeySchema: []dynamodb.KeySchemaT{ 212 | dynamodb.KeySchemaT{cPrimKey, "HASH"}, 213 | }, 214 | ProvisionedThroughput: dynamodb.ProvisionedThroughputT{ 215 | ReadCapacityUnits: cDefaultWRCapacity, 216 | WriteCapacityUnits: cDefaultWRCapacity, 217 | }, 218 | } 219 | 220 | if _, err := im.conn.CreateTable(td); err != nil { 221 | log.Error("Error trying to create a table on Dynamo DB, table:", im.tableName, "Error:", err) 222 | } 223 | if res, err = im.table.DescribeTable(); err != nil { 224 | log.Error("Error trying to describe a table on Dynamo DB, table:", im.tableName, "Error:", err) 225 | } 226 | } 227 | for "ACTIVE" != res.TableStatus { 228 | if res, err = im.table.DescribeTable(); err != nil { 229 | log.Error("Can't describe Dynamo DB instances table, Error:", err) 230 | } 231 | log.Debug("Waiting for active table, current status:", res.TableStatus) 232 | time.Sleep(time.Second) 233 | } 234 | } 235 | -------------------------------------------------------------------------------- /models/instances/instances_test.go: -------------------------------------------------------------------------------- 1 | package instances 2 | 3 | import ( 4 | "os" 5 | "testing" 6 | "time" 7 | ) 8 | 9 | var im *Model 10 | 11 | func TestMain(m *testing.M) { 12 | im = InitAndKeepAlive("test", "eu-west-1", true) 13 | time.Sleep(time.Second) 14 | 15 | retCode := m.Run() 16 | 17 | im.delTable() 18 | 19 | os.Exit(retCode) 20 | } 21 | 22 | func TestAddGetNoInstance(t *testing.T) { 23 | if len(im.GetInstances()) != 0 { 24 | t.Error("The test shouldn't return any instance, but:", im.GetInstances(), "was returned") 25 | } 26 | } 27 | 28 | func TestAddGetInstances(t *testing.T) { 29 | im.registerHostName("test1") 30 | im.registerHostName("test2") 31 | im.registerHostName("test3") 32 | time.Sleep(time.Second) 33 | if len(im.GetInstances()) != 3 { 34 | t.Error("The test should to return 3 instances, but:", im.GetInstances(), "was returned") 35 | } 36 | 37 | time.Sleep(cTTL * 2 * time.Second) 38 | if len(im.GetInstances()) != 0 { 39 | t.Error("The test should to return 0 instances, but:", im.GetInstances(), "was returned") 40 | } 41 | } 42 | -------------------------------------------------------------------------------- /models/shard_info/shard_info_test.go: -------------------------------------------------------------------------------- 1 | package shardinfo 2 | 3 | import ( 4 | //"github.com/alonsovidales/pit/log" 5 | //"github.com/alonsovidales/pit/models/instances" 6 | "os" 7 | "reflect" 8 | "testing" 9 | "time" 10 | ) 11 | 12 | var md *Model 13 | 14 | func TestMain(m *testing.M) { 15 | md = GetModel("test", "eu-west-1") 16 | time.Sleep(time.Second * 20) 17 | 18 | retCode := m.Run() 19 | 20 | md.delTables() 21 | 22 | os.Exit(retCode) 23 | } 24 | 25 | func TestAddGroupPersistAndRead(t *testing.T) { 26 | var err error 27 | 28 | originalGroups := make([]*GroupInfo, 3) 29 | originalGroups[0], err = md.AddGroup("userID", "secret", "groupId", 1, 1000000, 100, 1000, 5) 30 | if err != nil { 31 | t.Error("Problem trying to insert a new group, Error:", err) 32 | t.Fail() 33 | } 34 | 35 | originalGroups[1], err = md.AddGroup("userID1", "secret1", "groupId1", 2, 1000400, 110, 160, 6) 36 | if err != nil { 37 | t.Error("Problem trying to insert a new group, Error:", err) 38 | t.Fail() 39 | } 40 | 41 | originalGroups[2], err = md.AddGroup("userID2", "secret2", "groupId2", 3, 100000, 300, 1500, 10) 42 | if err != nil { 43 | t.Error("Problem trying to insert a new group, Error:", err) 44 | t.Fail() 45 | } 46 | 47 | time.Sleep(time.Second * (cUpdatePeriod * 2)) 48 | 49 | grByID, err := md.GetGroupByUserKeyID("dljvnekw", "secret", "123") 50 | if err != CErrGroupUserNotFound || grByID != nil { 51 | t.Error("Trying to get a group for an unexisting user, but the system didn't return the corresponding error, error returned:", err) 52 | } 53 | 54 | grByID, err = md.GetGroupByUserKeyID("userID", "secret", "123") 55 | if err != CErrGroupNotFound || grByID != nil { 56 | t.Error("Trying to get a unexisting group, but the system didn't return the corresponding error, error returned:", err) 57 | } 58 | 59 | grByID, err = md.GetGroupByUserKeyID("userID", "asd", "groupId") 60 | if err != CErrAuth || grByID != nil { 61 | t.Error("Trying to get a group using unvalid credentials, but the system didn't return the corresponding error, error returned:", err) 62 | } 63 | 64 | _, err = md.AddGroup("userID", "secret", "groupId", 3, 1000000, 100, 1000, 10) 65 | if err != CErrGroupInUse { 66 | t.Error("Trying to add a duplicated group ID, but the system didn't return the corresponding error, error returned:", err) 67 | } 68 | 69 | for _, gr := range originalGroups { 70 | grToCompare, err := md.GetGroupByUserKeyID(gr.UserID, gr.Secret, gr.GroupID) 71 | if err != nil || grToCompare == nil { 72 | t.Error("The group can't be obtained from the model, Error:", err) 73 | t.Fail() 74 | } 75 | 76 | for k := range gr.Shards { 77 | if _, ok := grToCompare.Shards[k]; !ok || !reflect.DeepEqual(gr.Shards[k], grToCompare.Shards[k]) { 78 | t.Error("After store and read a group, the shards contained are not equal:", gr.Shards[k], grToCompare.Shards[k]) 79 | } 80 | } 81 | 82 | gr.md = nil 83 | grToCompare.md = nil 84 | gr.Shards = nil 85 | grToCompare.Shards = nil 86 | 87 | if gr.UserID != grToCompare.UserID || 88 | gr.Secret != grToCompare.Secret || 89 | gr.GroupID != grToCompare.GroupID || 90 | gr.MaxScore != grToCompare.MaxScore || 91 | gr.NumShards != grToCompare.NumShards || 92 | gr.MaxReqSec != grToCompare.MaxReqSec || 93 | gr.MaxInsertReqSec != grToCompare.MaxInsertReqSec || 94 | gr.MaxElements != grToCompare.MaxElements { 95 | t.Error("After store and read a group, the result is not equal to the inserted group", gr, grToCompare) 96 | } 97 | } 98 | } 99 | -------------------------------------------------------------------------------- /models/users/users_test.go: -------------------------------------------------------------------------------- 1 | package users 2 | 3 | import ( 4 | "os" 5 | "reflect" 6 | "testing" 7 | "time" 8 | ) 9 | 10 | var um *Model 11 | 12 | func TestMain(m *testing.M) { 13 | um = GetModel("test", "eu-west-1") 14 | time.Sleep(time.Second) 15 | 16 | retCode := m.Run() 17 | 18 | um.delTable() 19 | 20 | os.Exit(retCode) 21 | } 22 | 23 | func TestAddGetListUsers(t *testing.T) { 24 | user, err := um.RegisterUser("uid", "key", "127.0.0.1") 25 | if user == nil || err != nil { 26 | t.Error("A new user can't be registered") 27 | } 28 | time.Sleep(time.Second) 29 | if userAux, err := um.RegisterUser("uid", "key", "127.0.0.1"); err == nil || userAux != nil { 30 | t.Error("Duplicated user registration") 31 | } 32 | 33 | if user.uid != "uid" { 34 | t.Error("The user information stored doesn't corresponds with the returned, User:", user) 35 | } 36 | 37 | time.Sleep(time.Second) 38 | 39 | u1, err := um.RegisterUser("uid1", "key1", "127.0.0.2") 40 | if err != nil { 41 | t.Error("A new user can't be registered") 42 | } 43 | u2, err := um.RegisterUser("uid2", "key2", "127.0.0.3") 44 | if err != nil { 45 | t.Error("A new user can't be registered") 46 | } 47 | usersToBeReturned := map[string]*User{ 48 | "uid": user, 49 | "uid1": u1, 50 | "uid2": u2, 51 | } 52 | 53 | usersRegistered := um.GetRegisteredUsers() 54 | for k, v := range usersToBeReturned { 55 | if !reflect.DeepEqual(usersRegistered[k], v) { 56 | t.Error("The returned registered users are not equal to the inserted", usersRegistered[k], v) 57 | } 58 | } 59 | 60 | secUser := um.GetUserInfo("uid", "key") 61 | 62 | if !reflect.DeepEqual(user, secUser) { 63 | t.Error("The returned used is not equal to the inserted", user, secUser) 64 | } 65 | 66 | secUser.DisableUser() 67 | time.Sleep(time.Second) 68 | secUser = um.GetUserInfo("uid", "key") 69 | if secUser != nil { 70 | t.Error("The user wasn't disabled") 71 | } 72 | } 73 | 74 | func TestUpdateUser(t *testing.T) { 75 | user, err := um.RegisterUser("uid99", "key99", "127.0.0.1") 76 | if user == nil || err != nil { 77 | t.Error("A new user can't be registered") 78 | } 79 | user.AddActivityLog("test", "testing...", "127.0.0.1") 80 | 81 | userFromDb := um.GetUserInfo("uid99", "key99") 82 | 83 | if _, ok := userFromDb.logs["test"]; !ok || !reflect.DeepEqual(userFromDb.logs, user.logs) { 84 | t.Error("The inserted logs doesn't match with the returned from DB:", userFromDb.logs, user.logs) 85 | } 86 | } 87 | -------------------------------------------------------------------------------- /recommender/recomender_test.go: -------------------------------------------------------------------------------- 1 | package recommender 2 | 3 | import ( 4 | "bufio" 5 | "encoding/json" 6 | "github.com/alonsovidales/pit/log" 7 | "os" 8 | "runtime" 9 | "strconv" 10 | "strings" 11 | "testing" 12 | "time" 13 | ) 14 | 15 | const ( 16 | TESTSET = "../test_training_set/training_set.info" 17 | ) 18 | 19 | func TestCompression(t *testing.T) { 20 | aux := "This is a test..." 21 | rc := &Recommender{} 22 | if string(rc.uncompress(rc.compress([]byte(aux)))) != aux { 23 | t.Error("Problem trying to compress and uncompress data") 24 | } 25 | } 26 | 27 | func TestRecommenderLoadNoBackup(t *testing.T) { 28 | sh := NewShard("/testing", "test_collab_insertion_no_baackup", 10, 5, "eu-west-1") 29 | if sh.LoadBackup() { 30 | t.Error("The method LoadBackup can't return true when a backup doesn't exist") 31 | } 32 | } 33 | 34 | func TestRecommenderSaveLoad(t *testing.T) { 35 | maxClassifications := uint64(1000000) 36 | runtime.GOMAXPROCS(runtime.NumCPU()) 37 | 38 | sh := NewShard("/testing", "test_collab_insertion", maxClassifications, 5, "eu-west-1") 39 | 40 | f, err := os.Open(TESTSET) 41 | if err != nil { 42 | log.Error("Can't read the the test set file:", TESTSET, "Error:", err) 43 | t.Fail() 44 | } 45 | r := bufio.NewReader(f) 46 | s, e := Readln(r) 47 | i := 0 48 | for e == nil && i < 100000 { 49 | s, e = Readln(r) 50 | recID, scores := parseLine(s) 51 | sh.AddRecord(recID, scores) 52 | i++ 53 | if i%1000 == 0 { 54 | log.Debug("Lines processed:", i) 55 | } 56 | } 57 | 58 | time.Sleep(time.Second) 59 | 60 | if sh.totalClassif > maxClassifications { 61 | t.Error( 62 | "Problem with the garbage collection, the total number of stored stores are:", 63 | sh.totalClassif, "and the max defined boundary is:", maxClassifications) 64 | } 65 | 66 | if sh.status != StatusStarting { 67 | t.Error("The expectede status was:", StatusStarting, "but the actual one is:", sh.status) 68 | } 69 | 70 | log.Debug("Processing tree...") 71 | sh.RecalculateTree() 72 | 73 | if sh.status != StatusActive { 74 | t.Error("The expectede status was:", StatusActive, "but the actual one is:", sh.status) 75 | } 76 | 77 | s, e = Readln(r) 78 | recID, scores := parseLine(s) 79 | recomendationsBef := sh.CalcScores(recID, scores, 10) 80 | if len(recomendationsBef) != 10 { 81 | t.Error("The expected recommendations was 10, but:", len(recomendationsBef), "obtained.") 82 | } 83 | 84 | prevScores := sh.totalClassif 85 | sh.SaveBackup() 86 | 87 | sh = NewShard("/testing", "test_collab_insertion", maxClassifications, 5, "eu-west-1") 88 | sh.RecalculateTree() 89 | 90 | if sh.status != StatusNoRecords { 91 | t.Error("The expectede status was:", StatusNoRecords, "but the actual one is:", sh.status) 92 | } 93 | 94 | sh.LoadBackup() 95 | 96 | sh.RecalculateTree() 97 | 98 | if prevScores != sh.totalClassif { 99 | t.Error( 100 | "Before store a backup the number of records was:", prevScores, 101 | "but after load the backup is:", sh.totalClassif) 102 | } 103 | 104 | recomendationsAfter := sh.CalcScores(recID, scores, 10) 105 | if len(recomendationsAfter) != 10 { 106 | t.Error("The expected recommendations was 10, but:", len(recomendationsAfter), "obtained.") 107 | } 108 | 109 | log.Debug("Classifications:", sh.maxClassif) 110 | } 111 | 112 | func Readln(r *bufio.Reader) (string, error) { 113 | var err error 114 | var line, ln []byte 115 | 116 | isPrefix := true 117 | for isPrefix && err == nil { 118 | line, isPrefix, err = r.ReadLine() 119 | ln = append(ln, line...) 120 | } 121 | 122 | return string(ln), err 123 | } 124 | 125 | func parseLine(line string) (recordID uint64, values map[uint64]uint8) { 126 | parts := strings.SplitN(line, ":", 2) 127 | recordIDOrig, _ := strconv.ParseInt(parts[0], 10, 64) 128 | recordID = uint64(recordIDOrig) 129 | 130 | valuesAux := make(map[string]uint8) 131 | if len(parts) < 2 { 132 | log.Fatal(line) 133 | } 134 | json.Unmarshal([]byte(parts[1]), &valuesAux) 135 | values = make(map[uint64]uint8) 136 | for k, v := range valuesAux { 137 | kI, _ := strconv.ParseInt(k, 10, 64) 138 | values[uint64(kI)] = v 139 | } 140 | 141 | return 142 | } 143 | -------------------------------------------------------------------------------- /static/account-logs.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | Highly Scalable Recommender System | Pitia 9 | 10 | 11 | 12 | 13 | 14 | 18 | 19 | 20 | 21 | 22 | 102 | 103 |
104 |
105 | 110 |
111 |
112 |

Account Information

113 | E-mail: 114 |
115 |
116 |

Change Password

117 |
118 | Old Password 119 | 120 | 121 | New Password 122 | 123 | Repeat Password 124 | 125 | 126 |
127 | 128 | 129 |
130 |
131 |
132 |
133 |
134 |
Activity Logs:
135 | 136 | 137 | 138 | 139 | 140 | 141 | 142 | 143 | 144 | 145 | 146 | 147 |
#TypeDateFromDesc
148 |
149 |
150 |
151 |
152 | 153 | 154 | 155 | 156 | 157 | 158 | 159 | 160 | 161 | 162 | 163 | 164 | 175 | 176 | 177 | -------------------------------------------------------------------------------- /static/account-panel.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | Highly Scalable Recommender System | Pitia 9 | 10 | 11 | 12 | 13 | 14 | 18 | 19 | 20 | 21 | 22 | 102 | 103 |
104 |
105 | 110 |
111 |
112 |

Groups / Shards Manager

113 | 114 | 115 | 118 | 119 | 159 | 160 |
161 |
162 |
163 |
164 | 165 | 211 | 212 | 245 | 246 | 247 | 248 | 249 | 250 | 251 | 252 | 253 | 254 | 255 | 256 | 267 | 268 | 269 | -------------------------------------------------------------------------------- /static/billing.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | Highly Scalable Recommender System | Pitia 9 | 10 | 11 | 12 | 13 | 14 | 18 | 19 | 20 | 21 | 22 | 102 | 103 |
104 |
105 | 110 |
111 |
112 |

Billing

113 | 114 |

Billing History

115 | 116 |

Month-to-date Cost

117 |
118 |
119 |
120 | Billing history:
121 |
122 | 123 | 124 | 125 | 126 | 127 | 128 | 129 | 130 | 131 | 132 | 133 | 134 |
#FromToAmounthPaid
135 |
136 |
137 |
138 | Historic of charges:
139 |
140 | 141 |
142 | 143 | 144 | 145 | 146 | 147 | 148 | 149 | 150 | 151 | 152 | 153 | 154 | 155 | 156 | 157 |
#Group* InstancesTypeFromToHoursTotal cost
158 | * In order to warranty the best of the services, each group counts with at least two instances, but we only bill by one of the first to instances on this group 159 |
160 |
161 |
162 |
163 | 164 | 165 | 166 | 167 | 168 | 169 | 170 | 171 | 172 | 173 | 174 | 185 | 186 | 187 | -------------------------------------------------------------------------------- /static/cases-of-use.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | Highly Scalable Recommender System | Pitia 9 | 10 | 11 | 12 | 13 | 17 | 18 | 19 | 20 | 21 | 108 | 109 |
110 |
111 |

Cases Of Use

112 | 113 |

114 | Item recommendations based on direct measurements 115 |

116 | This kind of interest measurement on the products is one of the most commons on some streaming applications, online stores, etc.
117 | For instance, you could add a rating system based on a number of stars, like a movies classification based on a 0 to 5 stars system. Based on the previous user classifications for the movies we will be able to predict what are the best movies for each user.
118 |
119 | This is an easy and direct way, but sometimes the users lie about its preferences or prefer not to classify some controverted items. In order to improve the recommendations for this cases you can use indirect measurements. 120 | 121 |

122 | Item recommendations based on indirect measurements 123 |

124 | If you don't have any rating system implemented in your site, don't worry you can use any kind of indirect measurement of the user interest for the items, usually this kind of measurement is more precise than a direct measurement, behavior never lies.

125 | This kind of actions can determine the interest of a user in the product: 126 |
    127 |
  • 128 | Time spent on the products Send to us the time spent by the users watching each of the products during the last week, and we will be able to predict what other products can be of interest for this user 129 |
    • 130 |
  • 131 | Number of visits to the products Another sign of interest for an item could be the number of times that a user has visited it, so we can predict the interest of the user for items based on this information 132 |
    • 133 |
  • 134 | Actions on a product If the user opens the attached images of the product, access to the full description of an article and so on, that could be an indicative of the grade of interest on the product, you can assign a score for each possible action, and send to us the sum of this scores by item 135 |
    • 136 |
  • 137 | ... Any other measurable indicator of interest can be analysed by Pitia in order to provide a great user experience to your customers! 138 |
    • 139 |
140 |
141 | Note: Take care of the bots, a bot like Google, Bing, etc can have an arbitrary behaviour that doesn't give any value to the collected data, try to avoid sending this information to Pitia. 142 |
143 |
144 |
145 | 146 | 166 | 167 | 168 | 169 | 170 | 171 | 172 | 173 | 174 | 183 | 184 | 185 | -------------------------------------------------------------------------------- /static/contact-form.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | Highly Scalable Recommender System | Pitia 9 | 10 | 11 | 12 | 13 | 17 | 18 | 19 | 20 | 21 | 108 | 109 |
110 |
111 | 112 |

Real-Time & Highly Scalable Recommender System

113 |

Improve the way your users explore your products!

114 |
115 |
116 |

Interested?


117 | We hope to open the system to the public soon, but in the meantime if you want to try Pitia, and you explain us how you would plan to use it, we could grant you enough shards to fulfill expectations! 118 |
119 |
120 |
121 |
122 | 123 |
124 |
125 | 126 |
127 |
128 | 129 |
130 |
131 |
132 |
133 |
134 | 135 | 155 | 156 | 157 | 158 | 159 | 160 | 161 | 162 | 163 | 164 | 165 | 171 | 172 | 181 | 182 | 183 | -------------------------------------------------------------------------------- /static/css/dashboard.css: -------------------------------------------------------------------------------- 1 | /* 2 | * Base structure 3 | */ 4 | 5 | /* Move down content because we have a fixed navbar that is 50px tall */ 6 | body { 7 | padding-top: 50px; 8 | } 9 | 10 | 11 | /* 12 | * Global add-ons 13 | */ 14 | 15 | .sub-header { 16 | padding-bottom: 10px; 17 | border-bottom: 1px solid #eee; 18 | } 19 | 20 | /* 21 | * Top navigation 22 | * Hide default border to remove 1px line. 23 | */ 24 | .navbar-fixed-top { 25 | border: 0; 26 | } 27 | 28 | /* 29 | * Sidebar 30 | */ 31 | 32 | /* Hide for mobile, show later */ 33 | .sidebar { 34 | display: none; 35 | } 36 | @media (min-width: 768px) { 37 | .sidebar { 38 | position: fixed; 39 | top: 51px; 40 | bottom: 0; 41 | left: 0; 42 | z-index: 1000; 43 | display: block; 44 | padding: 20px; 45 | overflow-x: hidden; 46 | overflow-y: auto; /* Scrollable contents if viewport is shorter than content. */ 47 | background-color: #f5f5f5; 48 | border-right: 1px solid #eee; 49 | } 50 | } 51 | 52 | /* Sidebar navigation */ 53 | .nav-sidebar { 54 | margin-right: -21px; /* 20px padding + 1px border */ 55 | margin-bottom: 20px; 56 | margin-left: -20px; 57 | } 58 | .nav-sidebar > li > a { 59 | padding-right: 20px; 60 | padding-left: 20px; 61 | } 62 | .nav-sidebar > .active > a, 63 | .nav-sidebar > .active > a:hover, 64 | .nav-sidebar > .active > a:focus { 65 | color: #fff; 66 | background-color: #428bca; 67 | } 68 | 69 | 70 | /* 71 | * Main content 72 | */ 73 | 74 | .main { 75 | padding: 20px; 76 | } 77 | @media (min-width: 768px) { 78 | .main { 79 | padding-right: 40px; 80 | padding-left: 40px; 81 | } 82 | } 83 | .main .page-header { 84 | margin-top: 0; 85 | } 86 | 87 | 88 | /* 89 | * Placeholder dashboard ideas 90 | */ 91 | 92 | .placeholders { 93 | margin-bottom: 30px; 94 | text-align: center; 95 | } 96 | .placeholders h4 { 97 | margin-bottom: 0; 98 | } 99 | .placeholder { 100 | margin-bottom: 20px; 101 | } 102 | .placeholder img { 103 | display: inline-block; 104 | border-radius: 50%; 105 | } 106 | -------------------------------------------------------------------------------- /static/css/highlightjs.css: -------------------------------------------------------------------------------- 1 | /* 2 | 3 | Original style from softwaremaniacs.org (c) Ivan Sagalaev 4 | 5 | */ 6 | 7 | .hljs { 8 | display: block; 9 | overflow-x: auto; 10 | padding: 0.5em; 11 | background: #f0f0f0; 12 | -webkit-text-size-adjust: none; 13 | } 14 | 15 | .hljs, 16 | .hljs-subst, 17 | .hljs-tag .hljs-title, 18 | .nginx .hljs-title { 19 | color: black; 20 | } 21 | 22 | .hljs-string, 23 | .hljs-title, 24 | .hljs-constant, 25 | .hljs-parent, 26 | .hljs-tag .hljs-value, 27 | .hljs-rule .hljs-value, 28 | .hljs-preprocessor, 29 | .hljs-pragma, 30 | .hljs-name, 31 | .haml .hljs-symbol, 32 | .ruby .hljs-symbol, 33 | .ruby .hljs-symbol .hljs-string, 34 | .hljs-template_tag, 35 | .django .hljs-variable, 36 | .smalltalk .hljs-class, 37 | .hljs-addition, 38 | .hljs-flow, 39 | .hljs-stream, 40 | .bash .hljs-variable, 41 | .pf .hljs-variable, 42 | .apache .hljs-tag, 43 | .apache .hljs-cbracket, 44 | .tex .hljs-command, 45 | .tex .hljs-special, 46 | .erlang_repl .hljs-function_or_atom, 47 | .asciidoc .hljs-header, 48 | .markdown .hljs-header, 49 | .coffeescript .hljs-attribute { 50 | color: #800; 51 | } 52 | 53 | .smartquote, 54 | .hljs-comment, 55 | .hljs-annotation, 56 | .diff .hljs-header, 57 | .hljs-chunk, 58 | .asciidoc .hljs-blockquote, 59 | .markdown .hljs-blockquote { 60 | color: #888; 61 | } 62 | 63 | .hljs-number, 64 | .hljs-date, 65 | .hljs-regexp, 66 | .hljs-literal, 67 | .hljs-hexcolor, 68 | .smalltalk .hljs-symbol, 69 | .smalltalk .hljs-char, 70 | .go .hljs-constant, 71 | .hljs-change, 72 | .lasso .hljs-variable, 73 | .makefile .hljs-variable, 74 | .asciidoc .hljs-bullet, 75 | .markdown .hljs-bullet, 76 | .asciidoc .hljs-link_url, 77 | .markdown .hljs-link_url { 78 | color: #080; 79 | } 80 | 81 | .hljs-label, 82 | .hljs-javadoc, 83 | .ruby .hljs-string, 84 | .hljs-decorator, 85 | .hljs-filter .hljs-argument, 86 | .hljs-localvars, 87 | .hljs-array, 88 | .hljs-attr_selector, 89 | .hljs-important, 90 | .hljs-pseudo, 91 | .hljs-pi, 92 | .haml .hljs-bullet, 93 | .hljs-doctype, 94 | .hljs-deletion, 95 | .hljs-envvar, 96 | .hljs-shebang, 97 | .apache .hljs-sqbracket, 98 | .nginx .hljs-built_in, 99 | .tex .hljs-formula, 100 | .erlang_repl .hljs-reserved, 101 | .hljs-prompt, 102 | .asciidoc .hljs-link_label, 103 | .markdown .hljs-link_label, 104 | .vhdl .hljs-attribute, 105 | .clojure .hljs-attribute, 106 | .asciidoc .hljs-attribute, 107 | .lasso .hljs-attribute, 108 | .coffeescript .hljs-property, 109 | .hljs-phony { 110 | color: #88f; 111 | } 112 | 113 | .hljs-keyword, 114 | .hljs-id, 115 | .hljs-title, 116 | .hljs-built_in, 117 | .css .hljs-tag, 118 | .hljs-javadoctag, 119 | .hljs-phpdoc, 120 | .hljs-dartdoc, 121 | .hljs-yardoctag, 122 | .smalltalk .hljs-class, 123 | .hljs-winutils, 124 | .bash .hljs-variable, 125 | .pf .hljs-variable, 126 | .apache .hljs-tag, 127 | .hljs-type, 128 | .hljs-typename, 129 | .tex .hljs-command, 130 | .asciidoc .hljs-strong, 131 | .markdown .hljs-strong, 132 | .hljs-request, 133 | .hljs-status { 134 | font-weight: bold; 135 | } 136 | 137 | .asciidoc .hljs-emphasis, 138 | .markdown .hljs-emphasis { 139 | font-style: italic; 140 | } 141 | 142 | .nginx .hljs-built_in { 143 | font-weight: normal; 144 | } 145 | 146 | .coffeescript .javascript, 147 | .javascript .xml, 148 | .lasso .markup, 149 | .tex .hljs-formula, 150 | .xml .javascript, 151 | .xml .vbscript, 152 | .xml .css, 153 | .xml .hljs-cdata { 154 | opacity: 0.5; 155 | } 156 | -------------------------------------------------------------------------------- /static/css/sidebar.css: -------------------------------------------------------------------------------- 1 | .group { 2 | background: yellow; 3 | width: 200px; 4 | height: 500px; 5 | } 6 | .group .subgroup { 7 | background: orange; 8 | width: 150px; 9 | height: 200px; 10 | } 11 | .fixed { 12 | position: fixed; 13 | } 14 | 15 | /* sidebar */ 16 | .bs-docs-sidebar { 17 | padding-left: 20px; 18 | margin-top: 20px; 19 | margin-bottom: 20px; 20 | } 21 | 22 | /* all links */ 23 | .bs-docs-sidebar .nav>li>a { 24 | color: #eeeeee; 25 | border-left: 2px solid transparent; 26 | padding: 4px 20px; 27 | font-size: 13px; 28 | font-weight: 400; 29 | } 30 | 31 | /* nested links */ 32 | .bs-docs-sidebar .nav .nav>li>a { 33 | padding-top: 1px; 34 | padding-bottom: 1px; 35 | padding-left: 30px; 36 | font-size: 12px; 37 | } 38 | 39 | /* active & hover links */ 40 | .bs-docs-sidebar .nav>.active>a, 41 | .bs-docs-sidebar .nav>li>a:hover, 42 | .bs-docs-sidebar .nav>li>a:focus { 43 | color: #d79d00; 44 | text-decoration: none; 45 | background-color: transparent; 46 | border-left-color: #d79d00; 47 | } 48 | /* all active links */ 49 | .bs-docs-sidebar .nav>.active>a, 50 | .bs-docs-sidebar .nav>.active:hover>a, 51 | .bs-docs-sidebar .nav>.active:focus>a { 52 | font-weight: 700; 53 | } 54 | /* nested active links */ 55 | .bs-docs-sidebar .nav .nav>.active>a, 56 | .bs-docs-sidebar .nav .nav>.active:hover>a, 57 | .bs-docs-sidebar .nav .nav>.active:focus>a { 58 | font-weight: 500; 59 | } 60 | 61 | /* hide inactive nested list */ 62 | .bs-docs-sidebar .nav ul.nav { 63 | display: none; 64 | } 65 | /* show active nested list */ 66 | .bs-docs-sidebar .nav>.active>ul.nav { 67 | display: block; 68 | } 69 | -------------------------------------------------------------------------------- /static/css/styles.css: -------------------------------------------------------------------------------- 1 | * { 2 | margin: 0; 3 | } 4 | 5 | html, body { 6 | background-color: #eaeaea; 7 | height: 100%; 8 | } 9 | 10 | .top-nav-bar { 11 | background-color: #1a1a1a; 12 | } 13 | 14 | .main-title { 15 | background-image: url("../img/graph_bg.png"); 16 | background-size: cover; 17 | padding-top: 80px; 18 | color: #ffffff; 19 | background-color: #ce9a0b; 20 | height: 380px; 21 | } 22 | 23 | .main-page-logo { 24 | background-image: url("../img/logo_white.png"); 25 | background-size: contain; 26 | background-repeat: no-repeat; 27 | margin: 0 auto; 28 | height: 150px; 29 | width: 100px; 30 | } 31 | 32 | .logo-header { 33 | width: 100px; 34 | background-image: url("../img/logo_white_r.png"); 35 | height: 30px; 36 | margin-top: 10px; 37 | background-size: contain; 38 | background-repeat: no-repeat; 39 | } 40 | 41 | #login-incorrect { 42 | display: none; 43 | } 44 | 45 | .login-dropdown-menu { 46 | padding: 0 10px 0 0; 47 | } 48 | 49 | .login-form-group { 50 | margin: 10px auto; 51 | } 52 | 53 | #login-button { 54 | float: right; 55 | } 56 | 57 | #account-pannel-link { 58 | display: none; 59 | } 60 | 61 | #account-name-top { 62 | color: #cccccc; 63 | } 64 | 65 | #logout-container { 66 | display: none; 67 | } 68 | 69 | footer { 70 | position: fixed; 71 | bottom: 0; 72 | width: 100%; 73 | } 74 | 75 | .footer-links a { 76 | color: #878787; 77 | } 78 | 79 | .footer-links { 80 | padding-top: 10px; 81 | height: 50px; 82 | color: #878787; 83 | background-color: #1c1c1c; 84 | font-size: 10px; 85 | padding-right: 100px; 86 | } 87 | 88 | .logo-foot { 89 | background-image: url("../img/logo_white_r.png"); 90 | margin-top: 10px; 91 | height: 30px; 92 | background-size: contain; 93 | background-repeat: no-repeat; 94 | } 95 | 96 | .footer-logo { 97 | background-color: #1c1c1c; 98 | padding-top: 5px; 99 | padding-left: 100px; 100 | height: 50px; 101 | } 102 | 103 | .main-how-works-wrapper { 104 | padding: 1% 4% 2% 4%; 105 | background-color: #000000; 106 | min-height: 200px; 107 | color: #ffffff; 108 | font-size: 1.2em; 109 | padding-bottom: 10px; 110 | } 111 | 112 | ul.main-how-works-ul { 113 | padding: 10px; 114 | list-style-type: none; 115 | } 116 | 117 | .yellow-text { 118 | color: #deaa1b; 119 | } 120 | 121 | .main-high-availability-wrapper { 122 | color: #0f4e70; 123 | background-image: url(../img/60-lines.png); 124 | background-color: #eccb47; 125 | padding: 1% 4% 3% 4%; 126 | font-size: 1.2em; 127 | } 128 | 129 | .red-text { 130 | color: #cd3f26; 131 | } 132 | 133 | .main-management { 134 | background-color: #ac354f; 135 | color: #fff; 136 | font-size: 1.3em; 137 | padding: 2% 0 1% 3%; 138 | min-height: 340px; 139 | } 140 | 141 | .main-pricing-wrapper { 142 | background-color: #2563ab; 143 | color: #eeeeee; 144 | } 145 | 146 | .info>.h1, .h2, .h3, h1, h2, h3 { 147 | margin-top: 0; 148 | padding-top: 10px; 149 | } 150 | 151 | .main-pricing-cols { 152 | padding-bottom: 20px; 153 | } 154 | 155 | .main-instance-pannel { 156 | width: 50%; 157 | margin: 0 auto; 158 | width: 500px; 159 | text-align: left; 160 | } 161 | 162 | ul.main-instance-pannel-info { 163 | padding-top: 10px; 164 | } 165 | 166 | ul.main-instance-pannel-info li { 167 | color: #444444; 168 | font-size: 0.75em; 169 | list-style-type: none; 170 | } 171 | 172 | .main-suscribe-column { 173 | background-image: url(../img/60-lines.png); 174 | background-color: #609572; 175 | min-height: 330px; 176 | color: #ffffff; 177 | font-size: 1.2em; 178 | padding: 60px 0 0 10%; 179 | } 180 | 181 | .main-suscribe-form { 182 | padding: 66px 0 0 1%; 183 | } 184 | 185 | .special-text { 186 | color: #deaa1b; 187 | font-weight: bold; 188 | font-size: 2em; 189 | } 190 | 191 | .suscribe-form-elem { 192 | width: 400px; 193 | } 194 | 195 | .cases-of-use-content { 196 | background-color: #fff; 197 | color: #383838; 198 | font-size: 1.1em; 199 | padding: 70px 5%; 200 | padding-bottom: 50px; 201 | } 202 | 203 | .how_works_image { 204 | background-image: url("../img/how_works.png"); 205 | background-size: contain; 206 | background-repeat: no-repeat; 207 | max-width: 750px; 208 | height: 150px; 209 | padding-top: 80px; 210 | margin: 0 auto; 211 | } 212 | 213 | .prices-disclamer { 214 | margin-top: 40px; 215 | } 216 | 217 | .pricing-page-body { 218 | padding: 50px 5% 0 5%; 219 | color: #ffffff; 220 | min-height: 500px; 221 | font-size: 1.1em; 222 | background-color: #0f4e70; 223 | color: #eeeeee; 224 | background-color: #2563ab; 225 | } 226 | 227 | .pricing-head { 228 | padding-bottom: 20px; 229 | } 230 | 231 | .pricing-body { 232 | height: 100%; 233 | min-height: 500px; 234 | } 235 | 236 | .api-page-body { 237 | padding-top: 50px; 238 | } 239 | 240 | .api { 241 | background-color: #262c4a; 242 | font-size: 1.2em; 243 | height: 3920px; 244 | color: #ffffff; 245 | padding-top: 20px; 246 | padding-right: 50px; 247 | background-image: url(../img/60-lines.png); 248 | } 249 | 250 | .api-code { 251 | margin-left: 30px; 252 | padding: 5px; 253 | } 254 | 255 | .json-examples-pre { 256 | width: 170px; 257 | height: 180px; 258 | background-color: white; 259 | padding: 0; 260 | margin-left: 60px; 261 | } 262 | 263 | .json-examples { 264 | background-color: white; 265 | padding: 0; 266 | padding-left: 10px; 267 | margin: 0; 268 | } 269 | 270 | .contact-form-wrapper { 271 | background-color: #609572; 272 | min-height: 300px; 273 | color: #ffffff; 274 | font-size: 1.2em; 275 | padding: 20px 0 0 10%; 276 | } 277 | 278 | .main-title.contact-title { 279 | height: 320px; 280 | } 281 | 282 | ul.group-desc { 283 | padding: 10px; 284 | list-style-type: none; 285 | } 286 | 287 | .group-shards-by-group { 288 | width: 260px; 289 | } 290 | 291 | .shards-pannel { 292 | width: 100%; 293 | margin-top: 10px; 294 | } 295 | 296 | .status-shard-label { 297 | float: right; 298 | } 299 | 300 | .progress-bar-shard { 301 | width: 50%; 302 | display: inline; 303 | float: right; 304 | margin-right: 20px; 305 | } 306 | 307 | .animated-char { 308 | width: 95%; 309 | margin: 0; 310 | height: 200px; 311 | } 312 | 313 | .shards-container { 314 | padding-left: 10px; 315 | } 316 | 317 | #shads-container { 318 | min-height: 400px; 319 | } 320 | 321 | .special-text-small { 322 | font-size: 1em; 323 | } 324 | 325 | .json-examples-scores-data { 326 | width: 150px; 327 | height: 220px; 328 | background-color: white; 329 | padding: 0; 330 | margin-left: 60px; 331 | } 332 | 333 | .api-page-header { 334 | color: #ff2222; 335 | } 336 | 337 | .contact { 338 | background-color: #609572; 339 | } 340 | 341 | #top-menu-option a { 342 | color: #fff; 343 | } 344 | 345 | #top-menu-option li.active a { 346 | color: #555; 347 | } 348 | 349 | .paid-charges { 350 | margin-top: 6px; 351 | margin-right: 5px; 352 | float: left; 353 | width: 10px; 354 | height: 10px; 355 | background-color: #ccf0d9; 356 | } 357 | 358 | .no-paid-charges { 359 | margin-top: 6px; 360 | margin-right: 5px; 361 | float: left; 362 | width: 10px; 363 | height: 10px; 364 | background-color: #dff5f2; 365 | } 366 | 367 | .paid-legend { 368 | font-size: 10px; 369 | } 370 | 371 | .billing_row_paid { 372 | background-color: #ccf0d9; 373 | } 374 | 375 | .billing_row_nopaid { 376 | background-color: #dff5f2; 377 | } 378 | 379 | #month-do-date-cost { 380 | font-weight: bold; 381 | } 382 | 383 | .free-text { 384 | color: #deaa1b; 385 | } 386 | 387 | .precission-link, .precission-link:hover { 388 | text-decoration: none; 389 | color: #deaa1b; 390 | } 391 | 392 | .precission-div { 393 | text-align: center; 394 | font-weight: bold; 395 | font-size: 22px; 396 | margin: 0 auto; 397 | } 398 | -------------------------------------------------------------------------------- /static/img/60-lines.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alonsovidales/pit/85f0585734756163bceb43dd3307a140bbe37b38/static/img/60-lines.png -------------------------------------------------------------------------------- /static/img/brick.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alonsovidales/pit/85f0585734756163bceb43dd3307a140bbe37b38/static/img/brick.png -------------------------------------------------------------------------------- /static/img/bright-squares.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alonsovidales/pit/85f0585734756163bceb43dd3307a140bbe37b38/static/img/bright-squares.png -------------------------------------------------------------------------------- /static/img/concrete_seamless.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alonsovidales/pit/85f0585734756163bceb43dd3307a140bbe37b38/static/img/concrete_seamless.png -------------------------------------------------------------------------------- /static/img/confectionary.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alonsovidales/pit/85f0585734756163bceb43dd3307a140bbe37b38/static/img/confectionary.png -------------------------------------------------------------------------------- /static/img/favicon.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alonsovidales/pit/85f0585734756163bceb43dd3307a140bbe37b38/static/img/favicon.png -------------------------------------------------------------------------------- /static/img/graph_bg.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alonsovidales/pit/85f0585734756163bceb43dd3307a140bbe37b38/static/img/graph_bg.png -------------------------------------------------------------------------------- /static/img/grey_wash_wall.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alonsovidales/pit/85f0585734756163bceb43dd3307a140bbe37b38/static/img/grey_wash_wall.png -------------------------------------------------------------------------------- /static/img/how_works.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alonsovidales/pit/85f0585734756163bceb43dd3307a140bbe37b38/static/img/how_works.png -------------------------------------------------------------------------------- /static/img/light_grey.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alonsovidales/pit/85f0585734756163bceb43dd3307a140bbe37b38/static/img/light_grey.png -------------------------------------------------------------------------------- /static/img/logo_orange.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alonsovidales/pit/85f0585734756163bceb43dd3307a140bbe37b38/static/img/logo_orange.png -------------------------------------------------------------------------------- /static/img/logo_white.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alonsovidales/pit/85f0585734756163bceb43dd3307a140bbe37b38/static/img/logo_white.png -------------------------------------------------------------------------------- /static/img/logo_white_r.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alonsovidales/pit/85f0585734756163bceb43dd3307a140bbe37b38/static/img/logo_white_r.png -------------------------------------------------------------------------------- /static/img/manage_pannel.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alonsovidales/pit/85f0585734756163bceb43dd3307a140bbe37b38/static/img/manage_pannel.png -------------------------------------------------------------------------------- /static/img/old_map.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alonsovidales/pit/85f0585734756163bceb43dd3307a140bbe37b38/static/img/old_map.png -------------------------------------------------------------------------------- /static/js/ekko-lightbox.min.js: -------------------------------------------------------------------------------- 1 | /*! 2 | * Lightbox for Bootstrap 3 by @ashleydw 3 | * https://github.com/ashleydw/lightbox 4 | * 5 | * License: https://github.com/ashleydw/lightbox/blob/master/LICENSE 6 | */ 7 | (function(){"use strict";var a,b;a=jQuery,b=function(b,c){var d,e,f,g=this;return this.options=a.extend({title:null,footer:null,remote:null},a.fn.ekkoLightbox.defaults,c||{}),this.$element=a(b),d="",this.modal_id=this.options.modal_id?this.options.modal_id:"ekkoLightbox-"+Math.floor(1e3*Math.random()+1),f='",e='",a(document.body).append('"),this.modal=a("#"+this.modal_id),this.modal_dialog=this.modal.find(".modal-dialog").first(),this.modal_content=this.modal.find(".modal-content").first(),this.modal_body=this.modal.find(".modal-body").first(),this.lightbox_container=this.modal_body.find(".ekko-lightbox-container").first(),this.lightbox_body=this.lightbox_container.find("> div:first-child").first(),this.showLoading(),this.modal_arrows=null,this.border={top:parseFloat(this.modal_dialog.css("border-top-width"))+parseFloat(this.modal_content.css("border-top-width"))+parseFloat(this.modal_body.css("border-top-width")),right:parseFloat(this.modal_dialog.css("border-right-width"))+parseFloat(this.modal_content.css("border-right-width"))+parseFloat(this.modal_body.css("border-right-width")),bottom:parseFloat(this.modal_dialog.css("border-bottom-width"))+parseFloat(this.modal_content.css("border-bottom-width"))+parseFloat(this.modal_body.css("border-bottom-width")),left:parseFloat(this.modal_dialog.css("border-left-width"))+parseFloat(this.modal_content.css("border-left-width"))+parseFloat(this.modal_body.css("border-left-width"))},this.padding={top:parseFloat(this.modal_dialog.css("padding-top"))+parseFloat(this.modal_content.css("padding-top"))+parseFloat(this.modal_body.css("padding-top")),right:parseFloat(this.modal_dialog.css("padding-right"))+parseFloat(this.modal_content.css("padding-right"))+parseFloat(this.modal_body.css("padding-right")),bottom:parseFloat(this.modal_dialog.css("padding-bottom"))+parseFloat(this.modal_content.css("padding-bottom"))+parseFloat(this.modal_body.css("padding-bottom")),left:parseFloat(this.modal_dialog.css("padding-left"))+parseFloat(this.modal_content.css("padding-left"))+parseFloat(this.modal_body.css("padding-left"))},this.modal.on("show.bs.modal",this.options.onShow.bind(this)).on("shown.bs.modal",function(){return g.modal_shown(),g.options.onShown.call(g)}).on("hide.bs.modal",this.options.onHide.bind(this)).on("hidden.bs.modal",function(){return g.gallery&&a(document).off("keydown.ekkoLightbox"),g.modal.remove(),g.options.onHidden.call(g)}).modal("show",c),this.modal},b.prototype={modal_shown:function(){var b,c=this;return this.options.remote?(this.gallery=this.$element.data("gallery"),this.gallery&&(this.gallery_items="document.body"===this.options.gallery_parent_selector||""===this.options.gallery_parent_selector?a(document.body).find('*[data-toggle="lightbox"][data-gallery="'+this.gallery+'"]'):this.$element.parents(this.options.gallery_parent_selector).first().find('*[data-toggle="lightbox"][data-gallery="'+this.gallery+'"]'),this.gallery_index=this.gallery_items.index(this.$element),a(document).on("keydown.ekkoLightbox",this.navigate.bind(this)),this.options.directional_arrows&&this.gallery_items.length>1&&(this.lightbox_container.append('
'),this.modal_arrows=this.lightbox_container.find("div.ekko-lightbox-nav-overlay").first(),this.lightbox_container.find("a"+this.strip_spaces(this.options.left_arrow_class)).on("click",function(a){return a.preventDefault(),c.navigate_left()}),this.lightbox_container.find("a"+this.strip_spaces(this.options.right_arrow_class)).on("click",function(a){return a.preventDefault(),c.navigate_right()}))),this.options.type?"image"===this.options.type?this.preloadImage(this.options.remote,!0):"youtube"===this.options.type&&(b=this.getYoutubeId(this.options.remote))?this.showYoutubeVideo(b):"vimeo"===this.options.type?this.showVimeoVideo(this.options.remote):"instagram"===this.options.type?this.showInstagramVideo(this.options.remote):"url"===this.options.type?this.loadRemoteContent(this.options.remote):"video"===this.options.type?this.showVideoIframe(this.options.remote):this.error('Could not detect remote target type. Force the type using data-type="image|youtube|vimeo|instagram|url|video"'):this.detectRemoteType(this.options.remote)):this.error("No remote target given")},strip_stops:function(a){return a.replace(/\./g,"")},strip_spaces:function(a){return a.replace(/\s/g,"")},isImage:function(a){return a.match(/(^data:image\/.*,)|(\.(jp(e|g|eg)|gif|png|bmp|webp|svg)((\?|#).*)?$)/i)},isSwf:function(a){return a.match(/\.(swf)((\?|#).*)?$/i)},getYoutubeId:function(a){var b;return b=a.match(/^.*(youtu.be\/|v\/|u\/\w\/|embed\/|watch\?v=|\&v=)([^#\&\?]*).*/),b&&11===b[2].length?b[2]:!1},getVimeoId:function(a){return a.indexOf("vimeo")>0?a:!1},getInstagramId:function(a){return a.indexOf("instagram")>0?a:!1},navigate:function(a){if(a=a||window.event,39===a.keyCode||37===a.keyCode){if(39===a.keyCode)return this.navigate_right();if(37===a.keyCode)return this.navigate_left()}},navigateTo:function(b){var c,d;return 0>b||b>this.gallery_items.length-1?this:(this.showLoading(),this.gallery_index=b,this.$element=a(this.gallery_items.get(this.gallery_index)),this.updateTitleAndFooter(),d=this.$element.attr("data-remote")||this.$element.attr("href"),this.detectRemoteType(d,this.$element.attr("data-type")||!1),this.gallery_index+1'+this.options.loadingMessage+""),this},showYoutubeVideo:function(a){var b,c;return c=this.checkDimensions(this.$element.data("width")||560),b=c/(560/315),this.showVideoIframe("//www.youtube.com/embed/"+a+"?badge=0&autoplay=1&html5=1",c,b)},showVimeoVideo:function(a){var b,c;return c=this.checkDimensions(this.$element.data("width")||560),b=c/(500/281),this.showVideoIframe(a+"?autoplay=1",c,b)},showInstagramVideo:function(a){var b,c;return c=this.checkDimensions(this.$element.data("width")||612),this.resize(c),b=c+80,this.lightbox_body.html(''),this.options.onContentLoaded.call(this),this.modal_arrows?this.modal_arrows.css("display","none"):void 0},showVideoIframe:function(a,b,c){return c=c||b,this.resize(b),this.lightbox_body.html('
'),this.options.onContentLoaded.call(this),this.modal_arrows&&this.modal_arrows.css("display","none"),this},loadRemoteContent:function(b){var c,d,e=this;return d=this.$element.data("width")||560,this.resize(d),c=this.$element.data("disableExternalCheck")||!1,c||this.isExternal(b)?(this.lightbox_body.html(''),this.options.onContentLoaded.call(this)):this.lightbox_body.load(b,a.proxy(function(){return e.$element.trigger("loaded.bs.modal")})),this.modal_arrows&&this.modal_arrows.css("display","none"),this},isExternal:function(a){var b;return b=a.match(/^([^:\/?#]+:)?(?:\/\/([^\/?#]*))?([^?#]+)?(\?[^#]*)?(#.*)?/),"string"==typeof b[1]&&b[1].length>0&&b[1].toLowerCase()!==location.protocol?!0:"string"==typeof b[2]&&b[2].length>0&&b[2].replace(new RegExp(":("+{"http:":80,"https:":443}[location.protocol]+")?$"),"")!==location.host?!0:!1},error:function(a){return this.lightbox_body.html(a),this},preloadImage:function(b,c){var d,e=this;return d=new Image,(null==c||c===!0)&&(d.onload=function(){var b;return b=a(""),b.attr("src",d.src),b.addClass("img-responsive"),e.lightbox_body.html(b),e.modal_arrows&&e.modal_arrows.css("display","block"),e.resize(d.width),e.options.onContentLoaded.call(e)},d.onerror=function(){return e.error("Failed to load image: "+b)}),d.src=b,d},resize:function(b){var c;return c=b+this.border.left+this.padding.left+this.padding.right+this.border.right,this.modal_dialog.css("width","auto").css("max-width",c),this.lightbox_container.find("a").css("line-height",function(){return a(this).parent().height()+"px"}),this},checkDimensions:function(a){var b,c;return c=a+this.border.left+this.padding.left+this.padding.right+this.border.right,b=document.body.clientWidth,c>b&&(a=this.modal_body.width()),a},close:function(){return this.modal.modal("hide")},addTrailingSlash:function(a){return"/"!==a.substr(-1)&&(a+="/"),a}},a.fn.ekkoLightbox=function(c){return this.each(function(){var d;return d=a(this),c=a.extend({remote:d.attr("data-remote")||d.attr("href"),gallery_parent_selector:d.attr("data-parent"),type:d.attr("data-type")},c,d.data()),new b(this,c),this})},a.fn.ekkoLightbox.defaults={gallery_parent_selector:"document.body",left_arrow_class:".glyphicon .glyphicon-chevron-left",right_arrow_class:".glyphicon .glyphicon-chevron-right",directional_arrows:!0,type:null,always_show_close:!0,loadingMessage:"Loading...",onShow:function(){},onShown:function(){},onHide:function(){},onHidden:function(){},onNavigate:function(){},onContentLoaded:function(){}}}).call(this); -------------------------------------------------------------------------------- /static/js/npm.js: -------------------------------------------------------------------------------- 1 | // This file is autogenerated via the `commonjs` Grunt task. You can require() this file in a CommonJS environment. 2 | require('../../js/transition.js') 3 | require('../../js/alert.js') 4 | require('../../js/button.js') 5 | require('../../js/carousel.js') 6 | require('../../js/collapse.js') 7 | require('../../js/dropdown.js') 8 | require('../../js/modal.js') 9 | require('../../js/tooltip.js') 10 | require('../../js/popover.js') 11 | require('../../js/scrollspy.js') 12 | require('../../js/tab.js') 13 | require('../../js/affix.js') -------------------------------------------------------------------------------- /static/js/pit/bootstrap.js: -------------------------------------------------------------------------------- 1 | var cAPIBase = "https://pitia.info" 2 | 3 | var Bootstrap = function () { 4 | console.log('Boostrap'); 5 | }; 6 | 7 | $(function() { 8 | new Bootstrap(); 9 | }); 10 | -------------------------------------------------------------------------------- /static/js/pit/controllers/account.js: -------------------------------------------------------------------------------- 1 | var AccountController = (function() { 2 | var user = LoginController.getUser(); 3 | var key = LoginController.getKey(); 4 | var logsTableBody = $("#logs-table-body"); 5 | 6 | var my = { 7 | init: function() { 8 | $("#change-pass-form").submit(function(event) { 9 | var oldPass = $("#old-pass").val(); 10 | var newPass = $("#new-pass").val(); 11 | var repNewPass = $("#repeat-new-pass").val(); 12 | 13 | $("#repeat-pass-incorrect").hide(); 14 | $("#old-pass-incorrect").hide(); 15 | $("#pass-updated").hide(); 16 | 17 | if (newPass !== repNewPass) { 18 | $("#repeat-pass-incorrect").show(); 19 | } else { 20 | $.ajax({ 21 | type: 'POST', 22 | url: cAPIBase + '/change_pass', 23 | data: { 24 | u: LoginController.getUser(), 25 | k: oldPass, 26 | nk: newPass 27 | }, 28 | success: function() { 29 | localStorage.setItem("key", newPass); 30 | key = newPass; 31 | $("#pass-updated").show(); 32 | }, 33 | error: function() { 34 | $("#old-pass-incorrect").show(); 35 | }, 36 | }); 37 | } 38 | event.preventDefault(); 39 | }); 40 | 41 | $("#email-addr").text(user); 42 | 43 | $.ajax({ 44 | type: 'POST', 45 | url: cAPIBase + '/account_logs', 46 | data: { 47 | u: user, 48 | k: key, 49 | }, 50 | success: function(data) { 51 | var i = 1; 52 | $.each(data, function(type, logs) { 53 | $.each(logs, function(_, line) { 54 | var d = new Date(line.ts * 1000); 55 | 56 | logsTableBody.append($("\ 57 | \ 58 | " + i++ + "\ 59 | " + line.type + "\ 60 | " + d.toDateString() + " " + d.toTimeString() + "\ 61 | " + line.ip.split(':')[0] + "\ 62 | " + line.desc + "\ 63 | \ 64 | ")); 65 | }); 66 | }); 67 | }, 68 | dataType: 'json' 69 | }); 70 | } 71 | }; 72 | 73 | return my; 74 | })(); 75 | -------------------------------------------------------------------------------- /static/js/pit/controllers/billing.js: -------------------------------------------------------------------------------- 1 | var BillingController = (function() { 2 | function round(num) { 3 | return Math.round(num * 1000) / 1000 4 | } 5 | 6 | var my = { 7 | init: function() { 8 | $.ajax({ 9 | type: 'POST', 10 | dataType: 'json', 11 | url: cAPIBase + '/billing_info', 12 | data: { 13 | u: LoginController.getUser(), 14 | k: LoginController.getKey(), 15 | }, 16 | success: function(data) { 17 | var billingTable = $('#billing-pending-table-body'); 18 | var billsTable = $('#billing-history-table-body'); 19 | $('#month-do-date-cost').text('$' + round(data.to_pay)); 20 | console.log('Result'); 21 | console.log(data); 22 | console.log(data.history); 23 | var i = 0; 24 | $.each(data.history.reverse(), function(_, line) { 25 | if (line.paid) { 26 | var paidClass = 'billing_row_paid'; 27 | } else { 28 | var paidClass = 'billing_row_nopaid'; 29 | } 30 | billingTable.append($("\ 31 | \ 32 | " + (data.history.length - ++i) + "\ 33 | " + line.group + "\ 34 | " + line.instances + "\ 35 | " + line.type + "\ 36 | " + new Date(line.from * 1000) + "\ 37 | " + new Date(line.to * 1000) + "\ 38 | " + round((line.to - line.from) / 3600) + "\ 39 | " + round(line.price) + "\ 40 | \ 41 | ")); 42 | }); 43 | 44 | i = 0; 45 | $.each(data.bills.reverse(), function(_, line) { 46 | var paypalButton = ''; 47 | if (!line.paid) { 48 | // TODO Add paypal button 49 | } 50 | billsTable.append($("\ 51 | \ 52 | " + (data.bills.length - ++i) + "\ 53 | " + new Date(line.from) + "\ 54 | " + new Date(line.to) + "\ 55 | " + line.amount + "\ 56 | " + line.paid + " " + paypalButton + "\ 57 | \ 58 | ")); 59 | }); 60 | }, 61 | error: function() { 62 | alert("The billing information can't be read"); 63 | }, 64 | }); 65 | } 66 | }; 67 | 68 | return my; 69 | })(); 70 | -------------------------------------------------------------------------------- /static/js/pit/controllers/contact-form.js: -------------------------------------------------------------------------------- 1 | var contactForm = (function () { 2 | $("#contact-form").submit(function() { 3 | $.ajax({ 4 | type: 'POST', 5 | url: cAPIBase + '/contact', 6 | data: { 7 | mail: $("#contact-email").val(), 8 | content: $("#contact-content").val() 9 | }, 10 | success: function() { 11 | var contactButton = $("#contact-submit"); 12 | contactButton.text("Sent!"); 13 | contactButton.removeClass("btn-success"); 14 | contactButton.addClass("btn-primary"); 15 | }, 16 | }); 17 | event.preventDefault(); 18 | }); 19 | })(); 20 | -------------------------------------------------------------------------------- /static/js/pit/controllers/groups.js: -------------------------------------------------------------------------------- 1 | var GroupsController = (function() { 2 | var groupsSecrets = {}; 3 | var charts = {}; 4 | 5 | var my = { 6 | getGroupsInfo: function(successCallback, errorCallback) { 7 | $.ajax({ 8 | type: 'POST', 9 | url: cAPIBase + '/get_groups_by_user', 10 | data: { 11 | u: LoginController.getUser(), 12 | uk: LoginController.getKey(), 13 | }, 14 | success: successCallback, 15 | error: errorCallback, 16 | dataType: 'json' 17 | }); 18 | }, 19 | 20 | init: function() { 21 | $('#new-group-form').submit(function (e) { 22 | e.preventDefault(); 23 | addNewGroup(); 24 | }); 25 | 26 | this.getGroupsInfo(plotGroupInfo); 27 | } 28 | }; 29 | 30 | var sanitize = function(k) { 31 | return k.replace(/\./g, "-").replace(/:/g, "-"); 32 | }; 33 | 34 | var addNewGroup = function() { 35 | $.ajax({ 36 | type: 'POST', 37 | url: cAPIBase + '/add_group', 38 | data: { 39 | u: LoginController.getUser(), 40 | uk: LoginController.getKey(), 41 | guid: $('#new-group-name').val(), 42 | gt: $('#new-group-type').val(), 43 | shards: $('#new-group-shards').val(), 44 | maxscore: $('#new-group-max-score').val() 45 | }, 46 | success: function() { 47 | location.reload(); 48 | }, 49 | error: function(msg) { 50 | alert(msg.responseText); 51 | }, 52 | dataType: 'json' 53 | }); 54 | }; 55 | 56 | var getShardsInfo = function(groupId, callback) { 57 | $.ajax({ 58 | type: 'POST', 59 | url: cAPIBase + '/info', 60 | data: { 61 | uid: LoginController.getUser(), 62 | key: groupsSecrets[groupId], 63 | group: groupId 64 | }, 65 | success: callback, 66 | dataType: 'json' 67 | }); 68 | } 69 | 70 | var roundTwoDec = function(n) { 71 | return Math.round(n * 100) / 100 72 | }; 73 | 74 | var updateGroupKey = function(groupId) { 75 | $.ajax({ 76 | type: 'POST', 77 | url: cAPIBase + '/generate_group_key', 78 | data: { 79 | u: LoginController.getUser(), 80 | uk: LoginController.getKey(), 81 | g: groupId, 82 | k: groupsSecrets[groupId] 83 | }, 84 | success: function(newKey) { 85 | $("#group-" + sanitize(groupId) + "-key").text(newKey); 86 | groupsSecrets[groupId] = newKey; 87 | } 88 | }); 89 | }; 90 | 91 | var updateShardsInfo = function (groupId) { 92 | var prevInfo = null; 93 | setInterval(function () { 94 | getShardsInfo(groupId, function (data) { 95 | if (prevInfo != null && prevInfo !== Object.keys(data).length) { 96 | location.reload(); 97 | } 98 | prevInfo = Object.keys(data).length; 99 | var x = (new Date()).getTime() 100 | $.each(data, function(k, v) { 101 | var shardIdSanit = sanitize(k); 102 | var sanitGroupID = sanitize(groupId); 103 | var secVal = v.queries_by_sec[v.queries_by_sec.length-1]; 104 | var perc = roundTwoDec(secVal / ~~$("#group-" + sanitGroupID + "-req-sec").text() * 100); 105 | if (perc > 100) { 106 | perc = 100; 107 | } 108 | $("#shard-req-sec-" + sanitGroupID + "-" + shardIdSanit).text(secVal); 109 | var barDiv = $("#shard-req-sec-prog-bar-" + sanitGroupID + "-" + shardIdSanit); 110 | barDiv.css("width", perc + "%"); 111 | barDiv.text(perc + "%"); 112 | 113 | $("#shard-elems-stored-" + sanitGroupID + "-" + shardIdSanit).text(v.stored_elements); 114 | var percElems = roundTwoDec(v.stored_elements / ~~$("#group-" + sanitGroupID + "-max-elements").text() * 100); 115 | var barDivElems = $("#shard-elems-stored-prog-bar-" + sanitGroupID + "-" + shardIdSanit); 116 | barDivElems.css("width", percElems + "%"); 117 | barDivElems.text(percElems + "%"); 118 | 119 | $("#shard-status-" + sanitGroupID + "-" + shardIdSanit).text(v.rec_tree_status); 120 | 121 | if (perc > 80) { 122 | barDiv.addClass("progress-bar-danger"); 123 | barDiv.removeClass("progress-bar-warning"); 124 | } else if (perc > 65) { 125 | barDiv.addClass("progress-bar-warning"); 126 | barDiv.removeClass("progress-bar-danger"); 127 | } else { 128 | barDiv.removeClass("progress-bar-danger"); 129 | barDiv.removeClass("progress-bar-warning"); 130 | } 131 | 132 | charts[sanitGroupID + "-sec-" + shardIdSanit].addPoint([x, secVal], true, true); 133 | if (Math.round(x/1000) % 60 === 0) { 134 | var minVal = v.queries_by_min[v.queries_by_min.length-1]; 135 | charts[sanitGroupID + "-min-" + shardIdSanit].addPoint([x, minVal], true, true); 136 | } 137 | }); 138 | }); 139 | }, 1000); 140 | }; 141 | 142 | var addAnimatedChar = function(title, targetDiv, id, initialInfo, timeMult) { 143 | if (initialInfo.length > 1000) { 144 | initialInfo = initialInfo.slice(initialInfo.length-1000); 145 | } 146 | for (var i = initialInfo.length; i < 60; i++) { 147 | initialInfo.push(0); 148 | } 149 | targetDiv.highcharts({ 150 | chart: { 151 | type: 'spline', 152 | animation: Highcharts.svg, 153 | marginRight: 10, 154 | events: { 155 | load: function () { 156 | charts[id] = this.series[0]; 157 | } 158 | } 159 | }, 160 | title: { 161 | text: null, 162 | }, 163 | xAxis: { 164 | type: 'datetime', 165 | tickPixelInterval: 150 166 | }, 167 | yAxis: { 168 | title: { 169 | text: null, 170 | }, 171 | plotLines: [{ 172 | value: 0, 173 | width: 1, 174 | color: '#808080' 175 | }] 176 | }, 177 | tooltip: { 178 | formatter: function () { 179 | return '' + this.series.name + '
' + 180 | Highcharts.dateFormat('%Y-%m-%d %H:%M:%S', this.x) + '
' + 181 | Highcharts.numberFormat(this.y, 2); 182 | } 183 | }, 184 | legend: { 185 | enabled: false 186 | }, 187 | exporting: { 188 | enabled: false 189 | }, 190 | series: [{ 191 | name: title, 192 | data: (function () { 193 | var data = [], 194 | time = (new Date()).getTime(), 195 | i; 196 | 197 | for (i = 0; i < initialInfo.length; i++) { 198 | data.push({ 199 | x: time - (initialInfo.length - i) * (timeMult * 1000), 200 | y: initialInfo[i], 201 | }); 202 | } 203 | return data; 204 | }()) 205 | }] 206 | }); 207 | } 208 | 209 | var removeGroup = function(groupId) { 210 | $.ajax({ 211 | type: 'POST', 212 | url: cAPIBase + '/del_group', 213 | data: { 214 | u: LoginController.getUser(), 215 | uk: LoginController.getKey(), 216 | k: groupsSecrets[groupId], 217 | g: groupId, 218 | }, 219 | success: function(newKey) { 220 | setTimeout(function() { 221 | location.reload(); 222 | }, 2000); 223 | } 224 | }); 225 | }; 226 | 227 | var removeShardsContent = function(groupId) { 228 | $.ajax({ 229 | type: 'POST', 230 | url: cAPIBase + '/remove_group_shards_content', 231 | data: { 232 | u: LoginController.getUser(), 233 | uk: LoginController.getKey(), 234 | k: groupsSecrets[groupId], 235 | g: groupId, 236 | }, 237 | success: function(newKey) { 238 | alert("Content removed, this action can take some time to have effect, please be pattient"); 239 | } 240 | }); 241 | }; 242 | 243 | var updateShardsGroup = function(groupId, numShards) { 244 | $.ajax({ 245 | type: 'POST', 246 | url: cAPIBase + '/set_shards_group', 247 | data: { 248 | u: LoginController.getUser(), 249 | uk: LoginController.getKey(), 250 | g: groupId, 251 | k: groupsSecrets[groupId], 252 | s: numShards 253 | }, 254 | success: function(newKey) { 255 | //alert("Updated"); 256 | } 257 | }); 258 | }; 259 | 260 | var plotGroupInfo = function(data) { 261 | Highcharts.setOptions({ 262 | global: { 263 | useUTC: false 264 | } 265 | }); 266 | var groupsTemplate = Handlebars.compile($("#groups-template").html()); 267 | var shardsTemplate = Handlebars.compile($("#shards-template").html()); 268 | var containerDiv = $("#shads-container"); 269 | 270 | containerDiv.html(''); 271 | 272 | if (data === null) { 273 | data = []; 274 | $('#shads-container').text('No groups defined'); 275 | } 276 | $.each(data, function (k, v) { 277 | v.group_name = v.group_id; 278 | v.group_id = sanitize(v.group_id); 279 | var html = groupsTemplate(v); 280 | 281 | groupsSecrets[k] = v.secret; 282 | containerDiv.append(html); 283 | 284 | $("#shards-update-buttn-" + sanitize(k)).click(function() { 285 | updateShardsGroup(k, ~~$("#shards-update-txt-" + sanitize(k)).val()); 286 | }); 287 | 288 | $("#group-button-" + sanitize(k) + "-del-group").click(function() { 289 | if (confirm('This action will remove all the content from the shards, stored backups and configuration, this action can\'t be undone. Are you completly sure that you want to perform this action?')) { 290 | removeGroup(k); 291 | } 292 | }); 293 | 294 | $("#group-button-" + sanitize(k) + "-remove-all").click(function() { 295 | if (confirm('This action will remove all the content from the shards and stored backups, this action can\'t be undone. Are you completly sure that you want to perform this action?')) { 296 | removeShardsContent(k); 297 | } 298 | }); 299 | 300 | $("#group-button-" + sanitize(k) + "-key").click(function() { 301 | if (confirm('Are you sure that you want to regenerate the key for this group?, remember change it on all the clients')) { 302 | updateGroupKey(k); 303 | } 304 | }); 305 | 306 | getShardsInfo(k, function(shardsData) { 307 | var shardsGroupContainer = $("#group-shards-" + sanitize(k)); 308 | $.each(shardsData, function(host, shardInfo) { 309 | var statusLevel; 310 | 311 | switch(shardInfo.rec_tree_status) { 312 | case "STARTING": 313 | statusLevel = "primary"; 314 | break; 315 | case "LOADING": 316 | statusLevel = "info"; 317 | break; 318 | case "ACTIVE": 319 | statusLevel = "success"; 320 | break; 321 | case "NO_RECORDS": 322 | statusLevel = "warning"; 323 | break; 324 | } 325 | 326 | var reqsSec = shardInfo.queries_by_sec[shardInfo.queries_by_sec.length-1]; 327 | var hostSanit = sanitize(host); 328 | var templateInfo = { 329 | group_id: sanitize(k), 330 | shard_id_full: host, 331 | shard_id: hostSanit, 332 | elems_stored: shardInfo.stored_elements, 333 | reqs_sec: reqsSec, 334 | status_level: statusLevel, 335 | perc_stored: roundTwoDec((shardInfo.stored_elements / v.max_elems) * 100), 336 | perc_reqs: roundTwoDec((reqsSec / v.max_req_sec) * 100), 337 | shard_status: shardInfo.rec_tree_status 338 | }; 339 | shardsGroupContainer.append(shardsTemplate(templateInfo)); 340 | 341 | // Add the animated chart at the bottom 342 | addAnimatedChar("Req/sec", $("#req-sec-stats-" + sanitize(k) + "-" + hostSanit), sanitize(k) + "-sec-" + hostSanit, shardInfo.queries_by_sec, 1); 343 | addAnimatedChar("Req/min", $("#req-min-stats-" + sanitize(k) + "-" + hostSanit), sanitize(k) + "-min-" + hostSanit, shardInfo.queries_by_min, 60); 344 | }); 345 | }); 346 | 347 | updateShardsInfo(k); 348 | }); 349 | }; 350 | 351 | return my; 352 | })(); 353 | -------------------------------------------------------------------------------- /static/js/pit/controllers/login.js: -------------------------------------------------------------------------------- 1 | var LoginController = (function() { 2 | var user = localStorage.getItem("user"); 3 | var key = localStorage.getItem("key"); 4 | 5 | var loggedIn = function() { 6 | return (user && key && user !== "undefined" && key !== "undefined"); 7 | }; 8 | 9 | var loginForm = $("#login-dropdown"); 10 | var logOutDiv = $("#logout-container"); 11 | var loginButton = $("#login-button"); 12 | var logOutButton = $("#log-out-button"); 13 | var loginIncorrect = $("#login-incorrect"); 14 | var loginEmail = $("#login-email"); 15 | var loginPass = $("#login-pass"); 16 | var accountName = $("#account-name-top"); 17 | var accountPannelLink = $("#account-pannel-link"); 18 | 19 | var loginQuery = function(loginUser, loginKey) { 20 | user = loginUser; 21 | key = loginKey; 22 | GroupsController.getGroupsInfo(function(data) { 23 | loginEmail.val(''); 24 | loginPass.val(''); 25 | loginIncorrect.hide(); 26 | doLogin(); 27 | 28 | accountPannelLink.show(); 29 | }, function () { 30 | loginIncorrect.show(); 31 | }); 32 | }; 33 | 34 | var doLogin = function () { 35 | localStorage.setItem("user", user); 36 | localStorage.setItem("key", key); 37 | accountName.text(user); 38 | accountName.show(); 39 | logOutDiv.show(); 40 | loginForm.hide(); 41 | }; 42 | 43 | var logOut = function () { 44 | localStorage.removeItem("user"); 45 | localStorage.removeItem("key"); 46 | accountName.hide(); 47 | logOutDiv.hide(); 48 | loginForm.show(); 49 | window.location = 'index.html'; 50 | }; 51 | 52 | logOutButton.click(logOut); 53 | 54 | if (loggedIn()) { 55 | $(function() { 56 | $('#try-it-button').hide(); 57 | doLogin(); 58 | loginQuery(user, key); 59 | }); 60 | } else { 61 | loginForm.show(); 62 | loginForm.submit(function(event) { 63 | loginQuery(loginEmail.val(), loginPass.val()); 64 | event.preventDefault(); 65 | }); 66 | } 67 | 68 | return { 69 | getUser: function() { 70 | return user; 71 | }, 72 | getKey: function() { 73 | return key; 74 | } 75 | }; 76 | })(); 77 | -------------------------------------------------------------------------------- /static/pricing.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | Highly Scalable Recommender System | Pitia 9 | 10 | 11 | 12 | 13 | 17 | 18 | 19 | 20 | 21 | 108 | 109 |
110 |
111 |

Pricing

112 |

* Try Pitia For Free!

113 |

114 | Thanks to Pitia is a high performance implementation our servers are able to manage thousands of requests per second resulting in very competitive prices.
115 | The prices below are expressed by Shard, each Shard is able to handle up to the specified request per second, and store up to the specified number of elements.
116 | The elements stored in a Shard refers to the number of classifications stored on it, for instance, if 20 different users that have classified an average of 5 elements, we will have 100 elements stored on this Shard.
Pitia uses a LRU policy in order to store in memory the newest classifications removing the old classifications when the limit of the number of elements are reached, on this way the system is able to adapt the recommendations to the last trends in your application. 117 |

118 |
119 |
120 |
121 |
122 | Small - $ 0,0097 / hour 123 |
124 |
    125 |
  • 126 | Requests per second: 50 127 |
    • 128 |
  • 129 | Insert Requests per second: 200 130 |
    • 131 |
  • 132 | Max number of stored elements: 2.000.000 133 |
    • 134 |
135 |
136 |
137 |
138 |
139 | Medium - $ 0,0222 / month 140 |
141 |
    142 |
  • 143 | Requests per second: 150 144 |
    • 145 |
  • 146 | Insert Requests per second: 600 147 |
    • 148 |
  • 149 | Max number of stored elements: 6.000.000 150 |
    • 151 |
152 |
153 |
154 |
155 |
156 | Large - $ 0,0361 / month 157 |
158 |
    159 |
  • 160 | Requests per second: 250 161 |
    • 162 |
  • 163 | Insert Requests per second: 1.000 164 |
    • 165 |
  • 166 | Max number of stored elements: 10.000.000 167 |
    • 168 |
169 |
170 |
171 |
172 |
173 |
174 | X-Large - $ 0,0625 / hour 175 |
176 |
    177 |
  • 178 | Requests per second: 500 179 |
    • 180 |
  • 181 | Insert Requests per second: 2.000 182 |
    • 183 |
  • 184 | Max number of stored elements: 20.000.000 185 |
    • 186 |
187 |
188 |
189 |
190 |
191 | XX-Large $ 0,1194 / hour 192 |
193 |
    194 |
  • 195 | Requests per second: 1.000 196 |
    • 197 |
  • 198 | Insert Requests per second: 2.000 199 |
    • 200 |
  • 201 | Max number of stored elements: 30.000.000 202 |
    • 203 |
204 |
205 |
206 | * During the first month you can use two Small instances with no charge!
207 | All the prices are expressed by Shard. Remember that the first Shard of a group is always composed by two Shards but you will be charged just by one. Please have in consideration that one of the Shards can be off-line, in order to provide the best user experience to your clients, please try to reserve enough Shards to cover all the needs when one of the Shards is down. 208 |
209 |
210 |
211 | 212 | 232 | 233 | 234 | 235 | 236 | 237 | 238 | 239 | 240 | 249 | 250 | 251 | -------------------------------------------------------------------------------- /static/privacy.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | Highly Scalable Recommender System | Pitia 9 | 10 | 11 | 12 | 13 | 17 | 18 | 19 | 20 | 21 | 98 | 99 |
100 |
101 | 102 |

Real-Time & Highly Scalable Recommender System

103 |

Improve the way your users explores your products.

104 |
105 |
106 |
We are working really hard, but this section is still not ready, sorry :'(

107 | but you can contact us with any question, and we will be more than happy to help you! 108 |
109 |
110 |
111 |
112 | 113 |
114 |
115 | 116 |
117 |
118 | 119 |
120 |
121 |
122 |
123 | 124 | 144 | 145 | 146 | 147 | 148 | 149 | 150 | 151 | 152 | 153 | 154 | 160 | 161 | 170 | 171 | 172 | -------------------------------------------------------------------------------- /static/security.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | Highly Scalable Recommender System | Pitia 9 | 10 | 11 | 12 | 13 | 17 | 18 | 19 | 20 | 21 | 98 | 99 |
100 |
101 | 102 |

Real-Time & Highly Scalable Recommender System

103 |

Improve the way your users explores your products.

104 |
105 |
106 |
We are working really hard, but this section is still not ready, sorry :'(

107 | but you can contact us with any question, and we will be more than happy to help you! 108 |
109 |
110 |
111 |
112 | 113 |
114 |
115 | 116 |
117 |
118 | 119 |
120 |
121 |
122 |
123 | 124 | 144 | 145 | 146 | 147 | 148 | 149 | 150 | 151 | 152 | 153 | 154 | 160 | 161 | 170 | 171 | 172 | -------------------------------------------------------------------------------- /static/terms.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | Highly Scalable Recommender System | Pitia 9 | 10 | 11 | 12 | 13 | 17 | 18 | 19 | 20 | 21 | 98 | 99 |
100 |
101 | 102 |

Real-Time & Highly Scalable Recommender System

103 |

Improve the way your users explores your products.

104 |
105 |
106 |
We are working really hard, but this section is still not ready, sorry :'(

107 | but you can contact us with any question, and we will be more than happy to help you! 108 |
109 |
110 |
111 |
112 | 113 |
114 |
115 | 116 |
117 |
118 | 119 |
120 |
121 |
122 |
123 | 124 | 144 | 145 | 146 | 147 | 148 | 149 | 150 | 151 | 152 | 153 | 154 | 160 | 161 | 170 | 171 | 172 | -------------------------------------------------------------------------------- /static/try-it.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | Highly Scalable Recommender System | Pitia 9 | 10 | 11 | 12 | 13 | 17 | 18 | 19 | 20 | 21 | 105 | 106 |
107 |
108 | 109 |

Real-Time & Highly Scalable Recommender System

110 |

Improve the way your users explore your products!

111 |
112 |
113 |

Interested?


114 | If you want to use Pitia, explain us how you would plan to use it, we could grant you enough shards to fulfill expectations! For the first month you can have two Small instances for free! 115 |
116 |
117 |
118 |
119 | 120 |
121 |
122 | 123 |
124 |
125 | 126 |
127 |
128 |
129 |
130 |
131 | 132 | 152 | 153 | 154 | 155 | 156 | 157 | 158 | 159 | 160 | 161 | 162 | 168 | 169 | 178 | 179 | 180 | -------------------------------------------------------------------------------- /static/website-terms.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | Highly Scalable Recommender System | Pitia 9 | 10 | 11 | 12 | 13 | 17 | 18 | 19 | 20 | 21 | 98 | 99 |
100 |
101 | 102 |

Real-Time & Highly Scalable Recommender System

103 |

Improve the way your users explores your products.

104 |
105 |
106 |
We are working really hard, but this section is still not ready, sorry :'(

107 | but you can contact us with any question, and we will be more than happy to help you! 108 |
109 |
110 |
111 |
112 | 113 |
114 |
115 | 116 |
117 |
118 | 119 |
120 |
121 |
122 |
123 | 124 | 144 | 145 | 146 | 147 | 148 | 149 | 150 | 151 | 152 | 153 | 154 | 160 | 161 | 170 | 171 | 172 | --------------------------------------------------------------------------------