├── LICENSE.md ├── README.md ├── cache.php └── plugins └── yapcache └── plugin.php /LICENSE.md: -------------------------------------------------------------------------------- 1 | Copyright (c) 2011, Ian Barber, (c) 2016 Chris Hastie 2 | 3 | All rights reserved. 4 | 5 | Redistribution and use in source and binary forms, with or without 6 | modification, are permitted provided that the following conditions are 7 | met: 8 | 9 | * Redistributions of source code must retain the above copyright 10 | notice, this list of conditions and the following disclaimer. 11 | 12 | * Redistributions in binary form must reproduce the above copyright 13 | notice, this list of conditions and the following disclaimer in the 14 | documentation and/or other materials provided with the distribution. 15 | 16 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 17 | "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 18 | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 19 | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 20 | HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 21 | SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 22 | LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 23 | DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 24 | THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 25 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 26 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 27 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | YAPCache 2 | ======== 3 | 4 | YAPCache is an APC based caching plugin for the [YOURLS](http://yourls.org/) URL shortener. 5 | 6 | YAPCache is designed to remove a lot of the database traffic from YOURLS, primarily the write load from doing the logging and click tracking. We have attempted to strike a balance between keeping most information, but spilling it in some cases in the name of higher performance. YAPCache will write data back to the database based on either the time since the last write or the amount of data currently cached. YAPCache also adds an [API call](#flushing-the-cache-with-an-api-call) to YOURLS that can be used to force a write out to the database. 7 | 8 | YAPCache is a fork of [Ian Barber's YOURLS APC Cache](https://github.com/ianbarber/Yourls-APC-Cache), with a few changes [listed below](#difference-from-yourls-apc-cache). You should not try to install both plugins at the same time! 9 | 10 | 11 | Installation 12 | ------------ 13 | 14 | 0. If you previously used another APC cache with YOURLS, uninstall it 15 | 1. Download the latest version of YAPCache 16 | 2. Copy the `plugins/yapcache` folder into your `user/plugins` folder for YOURLS 17 | 3. Set up the parameters for YAPCache in your YOURLS configuration file, `user/config.php` ([see below](#configuration)) 18 | 4. Copy the `cache.php` file into `user/` 19 | 5. There is no need to activate this plugin (by the same token, deactivating it via the admin panel will not disable it—to do that remove or rename `user/cache.php`) 20 | 21 | A recent version of APC is required. 22 | 23 | Operation 24 | --------- 25 | 26 | ### Caches 27 | 28 | There are four separate caches operated by the plugin: 29 | 30 | **Keyword Cache**: A read cache which caches the keyword -> long URL look up in APC. This is done on request, and by default is cached for about one hour. This period is configurable with `YAPC_READ_CACHE_TIMEOUT`. This cache will be destroyed if updates are made to the keyword. 31 | 32 | **Options Cache**: A read cache which caches YOURLS options in APC to avoid the need to retrieve these from the database at every request. Cache period is defined by `YAPC_READ_CACHE_TIMEOUT` and is one hour by default. This cache will be destroyed if any options are updated. 33 | 34 | **Click caching**: A write cache for records of clicks. Rather than writing directly to the database, we write clicks to APC. We keep a record of how long it is since we last wrote clicks to the database and if that period exceeds `YAPC_WRITE_CACHE_TIMEOUT` seconds (default 120) we write all cached clicks to the database. We also keep an eye on how many URLs we are caching click information for and will write to the database if this figure exceeds `YAPC_MAX_UPDATES` (default 200). Additionally, if the number of clicks stored for a single URL exceeds `YAPC_MAX_CLICKS` all clicks are written to the database. Since database writes can involve a lot of updates in quick succession, in either case, if the current server load exceeds `YAPC_MAX_LOAD` we delay the write. We bundle update queries up in a single transaction, which will reduce the overhead involved considerably as long as your table supports transactions. 35 | 36 | **Log caching**: A write cache similar to the click tracking, but tracking the log entries for each request. Note that each request for the same URL will increase the number of log table records being cached. In contrast, multiple requests for the same URL will not increase the number of records being cached, only the number of clicks recorded in a single record. The consequence of this is that log caching is likely to reach `YAPC_MAX_UPDATES` faster than click caching. 37 | 38 | ### Flushing the cache with an API call 39 | 40 | It is also possible to manually flush the cache out to the database using an API call. If your YOURLS installation is private you will need to authenticate in the usual way. If `YAPC_API_USER` is defined and not empty only the user defined can flush the cache. To flush you [construct an API call in the usual way](http://yourls.org/#API) using the action `flushcache`. eg 41 | 42 | ``` 43 | http://your-own-domain-here.com/yourls-api.php?action=flushcache&signature=1002a612b4 44 | ``` 45 | 46 | The API call is useful if you want to be sure that the cache will be written out at a defined interval even if there are periods when few requests are coming in. You can, for example, call it every five minutes from crontab with something likely 47 | 48 | ``` 49 | */5 * * * * curl -s "https://your-own-domain-here.com/yourls-api.php?action=flushcache&signature=1002a612b4" > /dev/null 50 | ``` 51 | 52 | You might also consider flushing the cache before restarts of the webserver. Many log rotation scripts, for example, will restart Apache after rotating the log, so it can be useful to use a script to flush the cache immediately before the log is rotated. 53 | 54 | It is possible to disable writing to the database as part of a normal request by setting both `YAPC_WRITE_CACHE_TIMEOUT` and `YAPC_MAX_UPDATES` to 0. Writes to the database will then only be triggered by the flushcache API call. As database writes can be slow this approach may improve user experience by ensuring that redirects are never delayed by writing data out to the database. 55 | 56 | ### Will you loose clicks? 57 | Almost certainly. Whilst we've taken care to try to minimise this there will be times when clicks and logs cached in APC disappear before they have been written to the database. APC is not ideal for holding volatile data that isn't stored elsewhere yet. If it runs low on memory it will start pruning its cached data and that could mean clicks and logs. Webserver restarts will also clear APC's cache, and on many systems these happen regularly when the logs are rotated. In deciding on suitable values for the various configuration options you will need to balance performance against the risk of loosing data. The longer you cache writes for, the more likely you are to loose data, and the more data you are likely to loose. 58 | 59 | If loosing the odd click is unacceptable to you you probably shouldn't use this plugin. 60 | 61 | Configuration 62 | ------------- 63 | 64 | The plugin comes with working defaults, but you will probably want to adjust things to your particular needs. The following constants can be defined in `user/config.php` to alter the plugin's behaviour, eg 65 | 66 | ```php 67 | define("YAPC_READ_CACHE_TIMEOUT", 1800); 68 | ``` 69 | 70 | ### YAPC_WRITE_CACHE_TIMEOUT 71 | _Interger. Default: 120._ 72 | Number of seconds to cache data for before writing it out to the database. A value of 0 will disable writing based on time 73 | 74 | ### YAPC_READ_CACHE_TIMEOUT 75 | _Interger. Default: 3600._ 76 | Number of seconds to cache reads from the database. We do try to delete cached data that has been changed, but not all YOURLS functions that change data have hooks we can use, so it is possible that for a time some stats etc will appear out of date in the admin interface. Redirects should be current, however 77 | 78 | ### YAPC_LONG_TIMEOUT 79 | _Interger. Default: 86400._ 80 | A timeout used for a few long lasting indexes. You probably don't need to change this 81 | 82 | ### YAPC_LOCK_TIMEOUT 83 | _Interger. Default: 30._ 84 | Maximum number of seconds to hold a lock on the indexes whilst doing a database write. If things are working this shouldn't take anywhere near 30 seconds (maybe 2 maximum) 85 | 86 | ### YAPC_MAX_LOAD 87 | _Interger. Default: 0.7._ 88 | If the system load exceeds this value apc-cache will delay writes to the database. A sensible value for this will depend, amongst other things, on how many processors you have. A value of 0 means don't check the load before writing. This setting has no effect on Windows. 89 | 90 | When the time cached exceeds YAPC_WRITE_CACHE_HARD_TIMEOUT writes will be done no matter what the load. 91 | 92 | ### YAPC_WRITE_CACHE_HARD_TIMEOUT 93 | _Interger. Default: 600._ 94 | Number of seconds before a write of cached data to the database is forced, even if the load exceeds YAPC_MAX_LOAD. This setting has no effect if YAPC_WRITE_CACHE_TIMEOUT is set to 0. 95 | 96 | ### YAPC_BACKOFF_TIME 97 | _Interger. Default: 30._ 98 | Number of seconds to delay the next attempt to write to the database if the load exceeds YAPC_MAX_LOAD 99 | 100 | ### YAPC_MAX_UPDATES 101 | _Interger. Default: 200._ 102 | The maximum number of updates of each type (clicks and logs) to hold in the cache before writing out to the database. When the number of cached updates exceeds this value they will be written to the database, irrespective of how long they have been cached. However, they won't be written out if the load exceeds YAPC_MAX_LOAD. A value of 0 means never write out on the basis of the number of cached updates. 103 | 104 | ### YAPC_MAX_CLICKS 105 | _Interger. Default: 30._ 106 | The maximum number of clicks that will be stored for a single URL. If a single URL has had more than this number of clicks since the last time the click cache was written to the database then a write will be performed. However, no write will be done if the load exceeds YAPC_MAX_LOAD. A value of 0 disables this test. 107 | 108 | ### YAPC_API_USER 109 | _String. Default: empty string_ 110 | The name of a user who is allowed to use the `flushcache` API call to force a write to the database. If set to false, the `flushcache` API call is disabled. If set to an empty string, any user may force a database write. 111 | 112 | ### YAPC_STATS_SHUNT 113 | _String. Default: Undefined._ 114 | If set to `none` this will cause the caching of the clicks and logredirects to be disabled, and the queries logged as normal. This is handy if you want to keep the URL caching, but still have 100% accurate stats (though the benefit of the plugin will be pretty small then). 115 | 116 | If set to `drop` this will cause the clicks and log redirect information to be dropped completely (a more aggressive NOSTATS). There will be no clicks or logredirects logged. 117 | 118 | ### YAPC_SKIP_CLICKTRACK 119 | _Boolean. Default: false._ 120 | If true this will cause the plugin to take no action on click/redirect logging (if this is being handled by another plugin for example). 121 | 122 | ### YAPC_ID 123 | _String. Default: ycache-_ 124 | A string which is prepended to all APC keys. This is useful if you run two or more instances of YOURLS on the same server and need to avoid their APC cache keys clashing. 125 | 126 | ### YAPC_DEBUG 127 | _Boolean. Default: false._ 128 | If true additional debug information is sent using PHP's `error_log()` function. You will find this wherever your server is configured to send PHP errors to. 129 | 130 | ### YAPC_REDIRECT_FIRST 131 | _Boolean. Default: false._ 132 | Set this to true to send redirects to the client first, and deal with logging and database updates later. This is experimental and highly likely to interact badly with certain other plugins. In particular, it is known to not work with some plugins which change the default HTTP status code to 302. To work around this use the YAPC_REDIRECT_FIRST_CODE setting instead. 133 | 134 | If you are potentially caching large numbers of updates, a request that triggers a database write may result in a slow response as normally the database writes are performed first, then a response is sent to the client. This option allows you to respond to the client first and do the slow stuff later. If you are only doing updates based on an API call this setting probably has no benefit and is best avoided. 135 | 136 | ### YAPC_REDIRECT_FIRST_CODE 137 | _Interger. Default: 301_ 138 | The HTTP status code to send with redirects when YAPC_REDIRECT_FIRST is true. Defaults to 301 (moved permanantly). 302 is the most likely alternative (although 303 or 307 are possible). Has no effect if YAPC_REDIRECT_FIRST is false 139 | 140 | Difference from Yourls-APC-Cache 141 | -------------------------------- 142 | 143 | The main differences between YAPCache and Ian Barber's original Yourls-APC-Cache are summarised below 144 | 145 | * YAPCache uses a different strategy for caching clicks. Instead of one timer for each URL, YAPCache uses a single timer for all URLs. This is somewhat more aggressive, ie clicks are more likely to be cached. It also means that multiple URLs are updated in the database at the same time. By wrapping these in a transaction the transaction overhead is reduced (ie one transaction for multiple updates, rather than the one transaction per update implied by autocommit) 146 | 147 | * YAPCache uses a different approach to timers. YAPCache writes the time into an APC key and then checks that, rather than relying on the key's TTL. This allows a bit more flexibility in the logic of when to write out to the database, allowing some other changes, including 148 | 149 | * An option to do writes on the basis of the number of records cached as well as / instead of the time since the last write 150 | 151 | * Writes can be delayed if the server load exceeds a threshold 152 | 153 | * YAPCache provides an API call that can be used to trigger a write out to the database. 154 | 155 | * YAPCache includes an experimental option to send the redirect to the client first and delay the slower work of updated the database until afterwards 156 | 157 | * A few minor bugs have been fixed 158 | 159 | ### Which one should you use? 160 | 161 | It's your choice. YAPCache has a few extra features—if you need these, use YAPCache. Yourls-APC-Cache is older and has been in use for several years longer. It is more tested (although not without bugs). YAPCache is based on this mature code, but it does include some substantial changes that as yet have had only limited testing in a production environment. 162 | 163 | 164 | -------------------------------------------------------------------------------- /cache.php: -------------------------------------------------------------------------------- 1 | option = apc_fetch($key); 96 | $ydb->installed = apc_fetch(YAPC_YOURLS_INSTALLED); 97 | return true; 98 | } 99 | 100 | return false; 101 | } 102 | 103 | /** 104 | * Cache all_options data. 105 | * 106 | * @param array $options 107 | * @return array options 108 | */ 109 | function yapc_get_all_options($option) { 110 | apc_store(YAPC_ALL_OPTIONS, $option, YAPC_READ_CACHE_TIMEOUT); 111 | // Set timeout on installed property twice as long as the options as otherwise there could be a split second gap 112 | apc_store(YAPC_YOURLS_INSTALLED, true, (2 * YAPC_READ_CACHE_TIMEOUT)); 113 | return $option; 114 | } 115 | 116 | /** 117 | * Clear the options cache if an option is altered 118 | * This covers changes to plugins too 119 | * 120 | * @param string $plugin 121 | */ 122 | function yapc_option_change($args) { 123 | apc_delete(YAPC_ALL_OPTIONS); 124 | } 125 | 126 | /** 127 | * If the URL data is in the cache, stick it back into the global DB object. 128 | * 129 | * @param string $args 130 | */ 131 | function yapc_pre_get_keyword($args) { 132 | global $ydb; 133 | $keyword = $args[0]; 134 | $use_cache = isset($args[1]) ? $args[1] : true; 135 | 136 | // Lookup in cache 137 | if($use_cache && apc_exists(yapc_get_keyword_key($keyword))) { 138 | $ydb->infos[$keyword] = apc_fetch(yapc_get_keyword_key($keyword)); 139 | } 140 | } 141 | 142 | /** 143 | * Store the keyword info in the cache 144 | * 145 | * @param array $info 146 | * @param string $keyword 147 | */ 148 | function yapc_get_keyword_infos($info, $keyword) { 149 | // Store in cache 150 | apc_store(yapc_get_keyword_key($keyword), $info, YAPC_READ_CACHE_TIMEOUT); 151 | return $info; 152 | } 153 | 154 | /** 155 | * Delete a cache entry for a keyword if that keyword is edited. 156 | * 157 | * @param array $return 158 | * @param string $url 159 | * @param string $keyword 160 | * @param string $newkeyword 161 | * @param string $title 162 | * @param bool $new_url_already_there 163 | * @param bool $keyword_is_ok 164 | */ 165 | function yapc_edit_link( $return, $url, $keyword, $newkeyword, $title, $new_url_already_there, $keyword_is_ok ) { 166 | if($return['status'] != 'fail') { 167 | apc_delete(yapc_get_keyword_key($keyword)); 168 | } 169 | return $return; 170 | } 171 | 172 | /** 173 | * Update the number of clicks in a performant manner. This manner of storing does 174 | * mean we are pretty much guaranteed to lose a few clicks. 175 | * 176 | * @param string $keyword 177 | */ 178 | function yapc_shunt_update_clicks($false, $keyword) { 179 | 180 | // initalize the timer. 181 | if(!apc_exists(YAPC_CLICK_TIMER)) { 182 | apc_add(YAPC_CLICK_TIMER, time()); 183 | } 184 | 185 | if(defined('YAPC_STATS_SHUNT')) { 186 | if(YAPC_STATS_SHUNT == "drop") { 187 | return true; 188 | } else if(YAPC_STATS_SHUNT == "none"){ 189 | return false; 190 | } 191 | } 192 | 193 | $keyword = yourls_sanitize_string( $keyword ); 194 | $key = YAPC_CLICK_KEY_PREFIX . $keyword; 195 | 196 | // Store in cache 197 | $added = false; 198 | $clicks = 1; 199 | if(!apc_exists($key)) { 200 | $added = apc_add($key, $clicks); 201 | } 202 | if(!$added) { 203 | $clicks = yapc_key_increment($key); 204 | } 205 | 206 | /* we need to keep a record of which keywords we have 207 | * data cached for. We do this in an associative array 208 | * stored at YAPC_CLICK_INDEX, with keyword as the keyword 209 | */ 210 | $idxkey = YAPC_CLICK_INDEX; 211 | yapc_lock_click_index(); 212 | if(apc_exists($idxkey)) { 213 | $clickindex = apc_fetch($idxkey); 214 | } else { 215 | $clickindex = array(); 216 | } 217 | $clickindex[$keyword] = 1; 218 | apc_store ( $idxkey, $clickindex); 219 | yapc_unlock_click_index(); 220 | 221 | if(yapc_write_needed('click', $clicks)) { 222 | yapc_write_clicks(); 223 | } 224 | 225 | return true; 226 | } 227 | 228 | /** 229 | * write any cached clicks out to the database 230 | */ 231 | function yapc_write_clicks() { 232 | global $ydb; 233 | yapc_debug("write_clicks: Writing clicks to database"); 234 | $updates = 0; 235 | // set up a lock so that another hit doesn't start writing too 236 | if(!apc_add(YAPC_CLICK_UPDATE_LOCK, 1, YAPC_LOCK_TIMEOUT)) { 237 | yapc_debug("write_clicks: Could not lock the click index. Abandoning write", true); 238 | return $updates; 239 | } 240 | 241 | if(apc_exists(YAPC_CLICK_INDEX)) { 242 | yapc_lock_click_index(); 243 | $clickindex = apc_fetch(YAPC_CLICK_INDEX); 244 | if($clickindex === false || !apc_delete(YAPC_CLICK_INDEX)) { 245 | // if apc_delete fails it's because the key went away. We probably have a race condition 246 | yapc_unlock_click_index(); 247 | yapc_debug("write_clicks: Index key disappeared. Abandoning write", true); 248 | apc_store(YAPC_CLICK_TIMER, time()); 249 | return $updates; 250 | } 251 | yapc_unlock_click_index(); 252 | 253 | /* as long as the tables support transactions, it's much faster to wrap all the updates 254 | * up into a single transaction. Reduces the overhead of starting a transaction for each 255 | * query. The down side is that if one query errors we'll loose the log 256 | */ 257 | $ydb->query("START TRANSACTION"); 258 | foreach ($clickindex as $keyword => $z) { 259 | $key = YAPC_CLICK_KEY_PREFIX . $keyword; 260 | $value = 0; 261 | if(!apc_exists($key)) { 262 | yapc_debug("write_clicks: Click key $key dissappeared. Possible data loss!", true); 263 | continue; 264 | } 265 | $value += yapc_key_zero($key); 266 | yapc_debug("write_clicks: Adding $value clicks for $keyword"); 267 | // Write value to DB 268 | $ydb->query("UPDATE `" . 269 | YOURLS_DB_TABLE_URL. 270 | "` SET `clicks` = clicks + " . $value . 271 | " WHERE `keyword` = '" . $keyword . "'"); 272 | $updates++; 273 | } 274 | yapc_debug("write_clicks: Committing changes"); 275 | $ydb->query("COMMIT"); 276 | } 277 | apc_store(YAPC_CLICK_TIMER, time()); 278 | apc_delete(YAPC_CLICK_UPDATE_LOCK); 279 | yapc_debug("write_clicks: Updated click records for $updates URLs"); 280 | return $updates; 281 | } 282 | 283 | /** 284 | * Update the log in a performant way. There is a reasonable chance of losing a few log entries. 285 | * This is a good trade off for us, but may not be for everyone. 286 | * 287 | * @param string $keyword 288 | */ 289 | function yapc_shunt_log_redirect($false, $keyword) { 290 | 291 | if(defined('YAPC_STATS_SHUNT')) { 292 | if(YAPC_STATS_SHUNT == "drop") { 293 | return true; 294 | } else if(YAPC_STATS_SHUNT == "none"){ 295 | return false; 296 | } 297 | } 298 | // respect setting in YOURLS_NOSTATS. Why you'd want to enable the plugin and 299 | // set YOURLS_NOSTATS true I don't know ;) 300 | if ( !yourls_do_log_redirect() ) 301 | return true; 302 | 303 | // Initialise the time. 304 | if(!apc_exists(YAPC_LOG_TIMER)) { 305 | apc_add(YAPC_LOG_TIMER, time()); 306 | } 307 | $ip = yourls_get_IP(); 308 | $args = array( 309 | date( 'Y-m-d H:i:s' ), 310 | yourls_sanitize_string( $keyword ), 311 | ( isset( $_SERVER['HTTP_REFERER'] ) ? yourls_sanitize_url( $_SERVER['HTTP_REFERER'] ) : 'direct' ), 312 | yourls_get_user_agent(), 313 | $ip, 314 | yourls_geo_ip_to_countrycode( $ip ) 315 | ); 316 | 317 | // Separated out the calls to make a bit more readable here 318 | $key = YAPC_LOG_INDEX; 319 | $logindex = 0; 320 | $added = false; 321 | 322 | if(!apc_exists($key)) { 323 | $added = apc_add($key, 0); 324 | } 325 | 326 | 327 | $logindex = yapc_key_increment($key); 328 | 329 | 330 | // We now have a reserved logindex, so lets cache 331 | apc_store(yapc_get_logindex($logindex), $args, YAPC_LONG_TIMEOUT); 332 | 333 | // If we've been caching for over a certain amount do write 334 | if(yapc_write_needed('log')) { 335 | // We can add, so lets flush the log cache 336 | yapc_write_log(); 337 | } 338 | 339 | return true; 340 | } 341 | 342 | /** 343 | * write any cached log entries out to the database 344 | */ 345 | function yapc_write_log() { 346 | global $ydb; 347 | $updates = 0; 348 | // set up a lock so that another hit doesn't start writing too 349 | if(!apc_add(YAPC_LOG_UPDATE_LOCK, 1, YAPC_LOCK_TIMEOUT)) { 350 | yapc_debug("write_log: Could not lock the log index. Abandoning write", true); 351 | return $updates; 352 | } 353 | yapc_debug("write_log: Writing log to database"); 354 | 355 | $key = YAPC_LOG_INDEX; 356 | $index = apc_fetch($key); 357 | if($index === false) { 358 | yapc_debug("write_log: key $key has disappeared. Abandoning write."); 359 | apc_store(YAPC_LOG_TIMER, time()); 360 | apc_delete(YAPC_LOG_UPDATE_LOCK); 361 | return $updates; 362 | } 363 | $fetched = 0; 364 | $n = 0; 365 | $loop = true; 366 | $values = array(); 367 | 368 | // Retrieve all items and reset the counter 369 | while($loop) { 370 | for($i = $fetched+1; $i <= $index; $i++) { 371 | $row = apc_fetch(yapc_get_logindex($i)); 372 | if($row === false) { 373 | yapc_debug("write_log: log entry " . yapc_get_logindex($i) . " disappeared. Possible data loss!!", true); 374 | } else { 375 | $values[] = $row; 376 | } 377 | } 378 | 379 | $fetched = $index; 380 | $n++; 381 | 382 | if(apc_cas($key, $index, 0)) { 383 | $loop = false; 384 | } else { 385 | usleep(500); 386 | $index = apc_fetch($key); 387 | } 388 | } 389 | yapc_debug("write_log: $fetched log entries retrieved; index reset after $n tries"); 390 | // Insert all log message - we're assuming input filtering happened earlier 391 | $query = ""; 392 | 393 | foreach($values as $value) { 394 | if(!is_array($value)) { 395 | yapc_debug("write_log: log row is not an array. Skipping"); 396 | continue; 397 | } 398 | if(strlen($query)) { 399 | $query .= ","; 400 | } 401 | $row = "('" . 402 | $value[0] . "', '" . 403 | $value[1] . "', '" . 404 | $value[2] . "', '" . 405 | $value[3] . "', '" . 406 | $value[4] . "', '" . 407 | $value[5] . "')"; 408 | yapc_debug("write_log: row: $row"); 409 | $query .= $row; 410 | $updates++; 411 | } 412 | $ydb->query( "INSERT INTO `" . YOURLS_DB_TABLE_LOG . "` 413 | (click_time, shorturl, referrer, user_agent, ip_address, country_code) 414 | VALUES " . $query); 415 | apc_store(YAPC_LOG_TIMER, time()); 416 | apc_delete(YAPC_LOG_UPDATE_LOCK); 417 | yapc_debug("write_log: Added $updates entries to log"); 418 | return $updates; 419 | 420 | } 421 | 422 | /** 423 | * Helper function to return a cache key for the log index. 424 | * 425 | * @param string $key 426 | * @return string 427 | */ 428 | function yapc_get_logindex($key) { 429 | return YAPC_LOG_INDEX . "-" . $key; 430 | } 431 | 432 | /** 433 | * Helper function to return a keyword key. 434 | * 435 | * @param string $key 436 | * @return string 437 | */ 438 | function yapc_get_keyword_key($keyword) { 439 | return YAPC_KEYWORD_PREFIX . $keyword; 440 | } 441 | 442 | /** 443 | * Helper function to do an atomic increment to a variable, 444 | * 445 | * 446 | * @param string $key 447 | * @return void 448 | */ 449 | function yapc_key_increment($key) { 450 | $n = 1; 451 | while(!$result = apc_inc($key)) { 452 | usleep(500); 453 | $n++; 454 | } 455 | if($n > 1) yapc_debug("key_increment: took $n tries on key $key"); 456 | return $result; 457 | } 458 | 459 | /** 460 | * Reset a key to 0 in a atomic manner 461 | * 462 | * @param string $key 463 | * @return old value before the reset 464 | */ 465 | function yapc_key_zero($key) { 466 | $old = 0; 467 | $n = 1; 468 | $old = apc_fetch($key); 469 | if($old == 0) { 470 | return $old; 471 | } 472 | while(!apc_cas($key, $old, 0)) { 473 | usleep(500); 474 | $n++; 475 | $old = apc_fetch($key); 476 | if($old == 0) { 477 | yapc_debug("key_zero: Key zeroed by someone else. Try $n. Key $key"); 478 | return $old; 479 | } 480 | } 481 | if($n > 1) yapc_debug("key_zero: Key $key zeroed from $old after $n tries"); 482 | return $old; 483 | } 484 | 485 | /** 486 | * Helper function to manage a voluntary lock on YAPC_CLICK_INDEX 487 | * 488 | * @return true when locked 489 | */ 490 | function yapc_lock_click_index() { 491 | $n = 1; 492 | // we always unlock as soon as possilbe, so a TTL of 1 should be fine 493 | while(!apc_add(YAPC_CLICK_INDEX_LOCK, 1, 1)) { 494 | $n++; 495 | usleep(500); 496 | } 497 | if($n > 1) yapc_debug("lock_click_index: Locked click index in $n tries"); 498 | return true; 499 | } 500 | 501 | /** 502 | * Helper function to unlock a voluntary lock on YAPC_CLICK_INDEX 503 | * 504 | * @return void 505 | */ 506 | function yapc_unlock_click_index() { 507 | apc_delete(YAPC_CLICK_INDEX_LOCK); 508 | } 509 | 510 | /** 511 | * Send debug messages to PHP's error log 512 | * 513 | * @param string $msg 514 | * @param bool $important 515 | * @return void 516 | */ 517 | function yapc_debug ($msg, $important=false) { 518 | if ($important || (defined('YAPC_DEBUG') && YAPC_DEBUG)) { 519 | error_log("yourls_apc_cache: " . $msg); 520 | } 521 | } 522 | 523 | /** 524 | * Check if the server load is above our maximum threshold for doing DB writes 525 | * 526 | * @return bool true if load exceeds threshold, false otherwise 527 | */ 528 | function yapc_load_too_high() { 529 | if(YAPC_MAX_LOAD == 0) 530 | // YAPC_MAX_LOAD of 0 means don't do load check 531 | return false; 532 | if (stristr(PHP_OS, 'win')) 533 | // can't get load on Windows, so just assume it's OK 534 | return false; 535 | $load = sys_getloadavg(); 536 | if ($load[0] < YAPC_MAX_LOAD) 537 | return false; 538 | return true; 539 | } 540 | 541 | /** 542 | * Count number of click updates that are cached 543 | * 544 | * @return int number of keywords with cached clicks 545 | */ 546 | function yapc_click_updates_count() { 547 | $count = 0; 548 | if(apc_exists(YAPC_CLICK_INDEX)) { 549 | $clickindex = apc_fetch(YAPC_CLICK_INDEX); 550 | $count = count($clickindex); 551 | } 552 | return $count; 553 | } 554 | 555 | 556 | /** 557 | * Check if we need to do a write to DB yet 558 | * Considers time since last write, system load etc 559 | * 560 | * @param string $type either 'click' or 'log' 561 | * @param int $clicks number of clicks cached for current URL 562 | * @return bool true if a DB write is due, false otherwise 563 | */ 564 | function yapc_write_needed($type, $clicks=0) { 565 | 566 | if($type == 'click') { 567 | $timerkey = YAPC_CLICK_TIMER; 568 | $count = yapc_click_updates_count(); 569 | } elseif ($type = 'log') { 570 | $timerkey = YAPC_LOG_TIMER; 571 | $count = apc_fetch(YAPC_LOG_INDEX); 572 | } else { 573 | return false; 574 | } 575 | if (empty($count)) $count = 0; 576 | yapc_debug("write_needed: Info: $count $type updates in cache"); 577 | 578 | if (!empty($clicks)) yapc_debug("write_needed: Info: current URL has $clicks cached clicks"); 579 | 580 | if(apc_exists($timerkey)) { 581 | $lastupdate = apc_fetch($timerkey); 582 | $elapsed = time() - $lastupdate; 583 | yapc_debug("write_needed: Info: Last $type write $elapsed seconds ago at " . strftime("%T" , $lastupdate)); 584 | 585 | /** 586 | * in the tests below YAPC_WRITE_CACHE_TIMEOUT of 0 means never do a write on the basis of 587 | * time elapsed, YAPC_MAX_UPDATES of 0 means never do a write on the basis of number 588 | * of queued updates, YAPC_MAX_CLICKS of 0 means never write on the basis of the number 589 | * clicks pending 590 | **/ 591 | 592 | // if we reached YAPC_WRITE_CACHE_HARD_TIMEOUT force a write out no matter what 593 | if ( !empty(YAPC_WRITE_CACHE_TIMEOUT) && $elapsed > YAPC_WRITE_CACHE_HARD_TIMEOUT) { 594 | yapc_debug("write_needed: True: Reached hard timeout (" . YAPC_WRITE_CACHE_HARD_TIMEOUT ."). Forcing write for $type after $elapsed seconds"); 595 | return true; 596 | } 597 | 598 | // if we've backed off because of server load, don't write 599 | if( apc_exists(YAPC_BACKOFF_KEY)) { 600 | yapc_debug("write_needed: False: Won't do write for $type during backoff period"); 601 | return false; 602 | } 603 | 604 | // have we either reached YAPC_WRITE_CACHE_TIMEOUT or exceeded YAPC_MAX_UPDATES or YAPC_MAX_CLICKS 605 | if(( !empty(YAPC_WRITE_CACHE_TIMEOUT) && $elapsed > YAPC_WRITE_CACHE_TIMEOUT ) 606 | || ( !empty(YAPC_MAX_UPDATES) && $count > YAPC_MAX_UPDATES ) 607 | || (!empty(YAPC_MAX_CLICKS) && !empty($clicks) && $clicks > YAPC_MAX_CLICKS) ) { 608 | // if server load is high, delay the write and set a backoff so we won't try again 609 | // for a short while 610 | if(yapc_load_too_high()) { 611 | yapc_debug("write_needed: False: System load too high. Won't try writing to database for $type", true); 612 | apc_add(YAPC_BACKOFF_KEY, time(), YAPC_BACKOFF_TIME); 613 | return false; 614 | } 615 | yapc_debug("write_needed: True: type: $type; count: $count; elapsed: $elapsed; clicks: $clicks; YAPC_WRITE_CACHE_TIMEOUT: " . YAPC_WRITE_CACHE_TIMEOUT . "; YAPC_MAX_UPDATES: " . YAPC_MAX_UPDATES . "; YAPC_MAX_CLICKS: " . YAPC_MAX_CLICKS); 616 | return true; 617 | } 618 | 619 | return false; 620 | } 621 | 622 | // The timer key went away. Better do an update to be safe 623 | yapc_debug("write_needed: True: reason: no $type timer found"); 624 | return true; 625 | 626 | } 627 | 628 | /** 629 | * Add the flushcache method to the API 630 | * 631 | * @param array $api_action 632 | * @return array $api_action 633 | */ 634 | function yapc_api_filter($api_actions) { 635 | $api_actions['flushcache'] = 'yapc_force_flush'; 636 | return $api_actions; 637 | } 638 | 639 | /** 640 | * Force a write of both clicks and logs to the database 641 | * 642 | * @return array $return status of updates 643 | */ 644 | function yapc_force_flush() { 645 | /* YAPC_API_USER of false means disable API. 646 | * YAPC_API_USER of empty string means allow 647 | * any user to use API. Otherwise only the specified 648 | * user is allowed 649 | */ 650 | $user = defined( 'YOURLS_USER' ) ? YOURLS_USER : '-1'; 651 | if(YAPC_API_USER === false) { 652 | yapc_debug("force_flush: Attempt to use API flushcache function whilst it is disabled. User: $user", true); 653 | $return = array( 654 | 'simple' => 'Error: The flushcache function is disabled', 655 | 'message' => 'Error: The flushcache function is disabled', 656 | 'errorCode' => 403, 657 | ); 658 | } 659 | elseif(!empty(YAPC_API_USER) && YAPC_API_USER != $user) { 660 | yapc_debug("force_flush: Unauthorised attempt to use API flushcache function by $user", true); 661 | $return = array( 662 | 'simple' => 'Error: User not authorised to use the flushcache function', 663 | 'message' => 'Error: User not authorised to use the flushcache function', 664 | 'errorCode' => 403, 665 | ); 666 | } else { 667 | yapc_debug("force_flush: Forcing write to database from API call"); 668 | $start = microtime(true); 669 | $log_updates = yapc_write_log(); 670 | $log_time = sprintf("%01.3f", 1000*(microtime(true) - $start)); 671 | $click_updates = yapc_write_clicks(); 672 | $click_time = sprintf("%01.3f", 1000*(microtime(true) - $start)); 673 | $return = array( 674 | 'clicksUpdated' => $click_updates, 675 | 'clickUpdateTime' => $click_time, 676 | 'logsUpdated' => $log_updates, 677 | 'logUpdateTime' => $log_time, 678 | 'statusCode' => 200, 679 | 'simple' => "Updated clicks for $click_updates URLs in ${click_time} ms. Logged $log_updates hits in ${log_time} ms.", 680 | 'message' => 'Success', 681 | ); 682 | } 683 | return $return; 684 | } 685 | 686 | /** 687 | * Replaces yourls_redirect. Does redirect first, then does logging and click 688 | * recording afterwards so that redirect is not delayed 689 | * This is somewhat fragile and may be broken by other plugins that hook on 690 | * pre_redirect, redirect_location or redirect_code 691 | * 692 | */ 693 | function yapc_redirect_shorturl( $args ) { 694 | $code = defined('YAPC_REDIRECT_FIRST_CODE')?YAPC_REDIRECT_FIRST_CODE:301; 695 | $location = $args[0]; 696 | $keyword = $args[1]; 697 | yourls_do_action( 'pre_redirect', $location, $code ); 698 | $location = yourls_apply_filter( 'redirect_location', $location, $code ); 699 | $code = yourls_apply_filter( 'redirect_code', $code, $location ); 700 | // Redirect, either properly if possible, or via Javascript otherwise 701 | if( !headers_sent() ) { 702 | yourls_status_header( $code ); 703 | header( "Location: $location" ); 704 | // force the headers to be sent 705 | echo "Redirecting to $location\n"; 706 | @ob_end_flush(); 707 | @ob_flush(); 708 | flush(); 709 | } else { 710 | yourls_redirect_javascript( $location ); 711 | } 712 | 713 | $start = microtime(true); 714 | // Update click count in main table 715 | $update_clicks = yourls_update_clicks( $keyword ); 716 | 717 | // Update detailed log for stats 718 | $log_redirect = yourls_log_redirect( $keyword ); 719 | $lapsed = sprintf("%01.3f", 1000*(microtime(true) - $start)); 720 | yapc_debug("redirect_shorturl: Database updates took $lapsed ms after sending redirect"); 721 | 722 | die(); 723 | } 724 | --------------------------------------------------------------------------------