├── .gitignore ├── COPYING ├── LICENSE ├── Makefile.am ├── README.rst ├── autogen.sh ├── configure.ac ├── m4 └── README └── src ├── Makefile.am ├── tests ├── test01.vtc └── test02.vtc ├── vmod_boltsort.c └── vmod_boltsort.vcc /.gitignore: -------------------------------------------------------------------------------- 1 | Makefile 2 | Makefile.in 3 | .deps/ 4 | .libs/ 5 | *.o 6 | *.lo 7 | *.la 8 | *~ 9 | *.[1-9] 10 | 11 | /aclocal.m4 12 | /autom4te.cache/ 13 | /compile 14 | /config.guess 15 | /config.h 16 | /config.h.in 17 | /config.log 18 | /config.status 19 | /config.sub 20 | /configure 21 | /depcomp 22 | /install-sh 23 | /libtool 24 | /ltmain.sh 25 | /missing 26 | /stamp-h1 27 | 28 | /src/vcc_if.c 29 | /src/vcc_if.h 30 | -------------------------------------------------------------------------------- /COPYING: -------------------------------------------------------------------------------- 1 | Copyright (c) 2012 Vimeo LLC 2 | ... 3 | See LICENSE for details. 4 | 5 | You're free to use and distribute this under terms in the 6 | LICENSE. Please add your relevant copyright statements. 7 | 8 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Redistribution and use in source and binary forms, with or without 2 | modification, are permitted provided that the following conditions 3 | are met: 4 | 1. Redistributions of source code must retain the above copyright 5 | notice, this list of conditions and the following disclaimer. 6 | 2. Redistributions in binary form must reproduce the above copyright 7 | notice, this list of conditions and the following disclaimer in the 8 | documentation and/or other materials provided with the distribution. 9 | 10 | THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND 11 | ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 12 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 13 | ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE 14 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 15 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS 16 | OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) 17 | HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 18 | LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY 19 | OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 20 | SUCH DAMAGE. 21 | -------------------------------------------------------------------------------- /Makefile.am: -------------------------------------------------------------------------------- 1 | ACLOCAL_AMFLAGS = -I m4 2 | 3 | SUBDIRS = src 4 | 5 | EXTRA_DIST = README.rst 6 | 7 | dist_man_MANS = vmod_boltsort.3 8 | MAINTAINERCLEANFILES = $(dist_man_MANS) 9 | 10 | vmod_boltsort.3: README.rst 11 | 12 | %.1 %.2 %.3 %.4 %.5 %.6 %.7 %.8 %.9: 13 | if HAVE_RST2MAN 14 | ${RST2MAN} $< $@ 15 | else 16 | @echo "========================================" 17 | @echo "You need rst2man installed to make dist" 18 | @echo "========================================" 19 | @false 20 | endif 21 | 22 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | ============= 2 | vmod_boltsort 3 | ============= 4 | 5 | ----------------------- 6 | Varnish boltsort Module 7 | ----------------------- 8 | 9 | :Author: Naren Venkataraman 10 | :Date: 2012-09-01 11 | :Version: 1.0 12 | :Manual section: 3 13 | 14 | SYNOPSIS 15 | ======== 16 | 17 | At Vimeo we use Varnish Cache to cache our player traffic. It has 18 | worked out great for us. 19 | 20 | Varnish at this point has no built in module to sort querystrings. 21 | Say I have two of these: 22 | 23 | * /video/48088296?title=0&byline=0&portrait=0&color=51a516 24 | * /video/48088296?byline=0&color=51a516&portrait=0&title=0 25 | 26 | Boltsort sorts the querystring and ensures that Varnish treats the 27 | above one and the same. 28 | 29 | DESCRIPTION 30 | =========== 31 | 32 | The boltsort Varnish vmod sorts querystring. 33 | 34 | * Insertion sort on tokenized querystring params makes it twice as fast as current qsort implementations 35 | * Custom param compare instead of storing param lengths for each querystring param means lesser stack usage 36 | 37 | 38 | FUNCTIONS 39 | ========= 40 | 41 | sort 42 | ----- 43 | 44 | Prototype 45 | :: 46 | 47 | sort(STRING S) 48 | Return value 49 | STRING 50 | Description 51 | Returns url with querystring sorted 52 | Example 53 | :: 54 | 55 | set req.url = boltsort.sort(req.url); 56 | 57 | INSTALLATION 58 | ============ 59 | Same as libvmod-example installation instructions 60 | 61 | Usage:: 62 | 63 | yum install python-docutils or apt-get install python-docutils 64 | ./autogen.sh 65 | ./configure VARNISHSRC=DIR [VMODDIR=DIR] 66 | 67 | `VARNISHSRC` is the directory of the Varnish source tree for which to 68 | compile your vmod. Both the `VARNISHSRC` and `VARNISHSRC/include` 69 | will be added to the include search paths for your module. 70 | 71 | Optionally you can also set the vmod install directory by adding 72 | `VMODDIR=DIR` (defaults to the pkg-config discovered directory from your 73 | Varnish installation). 74 | 75 | Make targets: 76 | 77 | * make - builds the vmod 78 | * make install - installs your vmod in `VMODDIR` 79 | * make check - runs the unit tests in ``src/tests/*.vtc`` 80 | 81 | In your VCL you could then use this vmod along the following lines:: 82 | 83 | import boltsort; 84 | 85 | sub vcl_hash { 86 | # sort the querysting and use the sorted one for hashing 87 | set req.url = boltsort.sort(req.url); 88 | } 89 | 90 | COPYRIGHT 91 | ========= 92 | 93 | This document is licensed under the same license as the 94 | libvmod-boltsort project. See LICENSE for details. 95 | 96 | * Copyright (c) 2012 Vimeo 97 | -------------------------------------------------------------------------------- /autogen.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | warn() { 4 | echo "WARNING: $@" 1>&2 5 | } 6 | 7 | case `uname -s` in 8 | Darwin) 9 | LIBTOOLIZE=glibtoolize 10 | ;; 11 | FreeBSD) 12 | LIBTOOLIZE=libtoolize 13 | ;; 14 | Linux) 15 | LIBTOOLIZE=libtoolize 16 | ;; 17 | SunOS) 18 | LIBTOOLIZE=libtoolize 19 | ;; 20 | *) 21 | warn "unrecognized platform:" `uname -s` 22 | LIBTOOLIZE=libtoolize 23 | esac 24 | 25 | automake_version=`automake --version | tr ' ' '\n' | egrep '^[0-9]\.[0-9a-z.-]+'` 26 | if [ -z "$automake_version" ] ; then 27 | warn "unable to determine automake version" 28 | else 29 | case $automake_version in 30 | 0.*|1.[0-8]|1.[0-8][.-]*) 31 | warn "automake ($automake_version) detected; 1.9 or newer recommended" 32 | ;; 33 | *) 34 | ;; 35 | esac 36 | fi 37 | 38 | set -ex 39 | 40 | aclocal -I m4 41 | $LIBTOOLIZE --copy --force 42 | autoheader 43 | automake --add-missing --copy --foreign 44 | autoconf 45 | -------------------------------------------------------------------------------- /configure.ac: -------------------------------------------------------------------------------- 1 | AC_PREREQ(2.59) 2 | AC_COPYRIGHT([Copyright (c) 2011 Varnish Software AS]) 3 | AC_INIT([libvmod-boltsort], [trunk]) 4 | AC_CONFIG_MACRO_DIR([m4]) 5 | AC_CONFIG_SRCDIR(src/vmod_boltsort.vcc) 6 | AM_CONFIG_HEADER(config.h) 7 | 8 | AC_CANONICAL_SYSTEM 9 | AC_LANG(C) 10 | 11 | AM_INIT_AUTOMAKE([foreign]) 12 | 13 | AC_GNU_SOURCE 14 | AC_PROG_CC 15 | AC_PROG_CC_STDC 16 | if test "x$ac_cv_prog_cc_c99" = xno; then 17 | AC_MSG_ERROR([Could not find a C99 compatible compiler]) 18 | fi 19 | AC_PROG_CPP 20 | 21 | AC_PROG_INSTALL 22 | AC_PROG_LIBTOOL 23 | AC_PROG_MAKE_SET 24 | 25 | # Check for rst utilities 26 | AC_CHECK_PROGS(RST2MAN, [rst2man rst2man.py], "no") 27 | if test "x$RST2MAN" = "xno"; then 28 | AC_MSG_WARN([rst2man not found - not building man pages]) 29 | fi 30 | AM_CONDITIONAL(HAVE_RST2MAN, [test "x$RST2MAN" != "xno"]) 31 | 32 | # Check for pkg-config 33 | PKG_PROG_PKG_CONFIG 34 | 35 | # Checks for header files. 36 | AC_HEADER_STDC 37 | AC_CHECK_HEADERS([sys/stdlib.h]) 38 | 39 | # Check for python 40 | AC_CHECK_PROGS(PYTHON, [python3 python3.1 python3.2 python2.7 python2.6 python2.5 python2 python], [AC_MSG_ERROR([Python is needed to build this vmod, please install python.])]) 41 | 42 | # Varnish source tree 43 | AC_ARG_VAR([VARNISHSRC], [path to Varnish source tree (mandatory)]) 44 | if test "x$VARNISHSRC" = x; then 45 | AC_MSG_ERROR([No Varnish source tree specified]) 46 | fi 47 | VARNISHSRC=`cd $VARNISHSRC && pwd` 48 | AC_CHECK_FILE([$VARNISHSRC/include/varnishapi.h], 49 | [], 50 | [AC_MSG_FAILURE(["$VARNISHSRC" is not a Varnish source directory])] 51 | ) 52 | 53 | # Check that varnishtest is built in the varnish source directory 54 | AC_CHECK_FILE([$VARNISHSRC/bin/varnishtest/varnishtest], 55 | [], 56 | [AC_MSG_FAILURE([Can't find "$VARNISHSRC/bin/varnishtest/varnishtest". Please build your varnish source directory])] 57 | ) 58 | 59 | # vmod installation dir 60 | AC_ARG_VAR([VMODDIR], [vmod installation directory @<:@LIBDIR/varnish/vmods@:>@]) 61 | if test "x$VMODDIR" = x; then 62 | VMODDIR=`pkg-config --variable=vmoddir varnishapi` 63 | if test "x$VMODDIR" = x; then 64 | AC_MSG_FAILURE([Can't determine vmod installation directory]) 65 | fi 66 | fi 67 | 68 | AC_CONFIG_FILES([ 69 | Makefile 70 | src/Makefile 71 | ]) 72 | AC_OUTPUT 73 | -------------------------------------------------------------------------------- /m4/README: -------------------------------------------------------------------------------- 1 | required by autogen.sh 2 | -------------------------------------------------------------------------------- /src/Makefile.am: -------------------------------------------------------------------------------- 1 | INCLUDES = -I$(VARNISHSRC)/include -I$(VARNISHSRC) 2 | 3 | vmoddir = $(VMODDIR) 4 | vmod_LTLIBRARIES = libvmod_boltsort.la 5 | 6 | libvmod_boltsort_la_LDFLAGS = -module -export-dynamic -avoid-version 7 | 8 | libvmod_boltsort_la_SOURCES = \ 9 | vcc_if.c \ 10 | vcc_if.h \ 11 | vmod_boltsort.c 12 | 13 | vcc_if.c vcc_if.h: $(VARNISHSRC)/lib/libvmod_std/vmod.py $(top_srcdir)/src/vmod_boltsort.vcc 14 | @PYTHON@ $(VARNISHSRC)/lib/libvmod_std/vmod.py $(top_srcdir)/src/vmod_boltsort.vcc 15 | 16 | VMOD_TESTS = tests/*.vtc 17 | .PHONY: $(VMOD_TESTS) 18 | 19 | tests/*.vtc: 20 | $(VARNISHSRC)/bin/varnishtest/varnishtest -Dvarnishd=$(VARNISHSRC)/bin/varnishd/varnishd -Dvmod_topbuild=$(abs_top_builddir) $@ 21 | 22 | check: $(VMOD_TESTS) 23 | 24 | EXTRA_DIST = \ 25 | vmod_boltsort.vcc \ 26 | $(VMOD_TESTS) 27 | 28 | CLEANFILES = $(builddir)/vcc_if.c $(builddir)/vcc_if.h 29 | -------------------------------------------------------------------------------- /src/tests/test01.vtc: -------------------------------------------------------------------------------- 1 | varnishtest "Test boltsort vmod" 2 | 3 | server s1 { 4 | 5 | rxreq 6 | txresp 7 | 8 | rxreq 9 | txresp 10 | 11 | rxreq 12 | txresp 13 | 14 | rxreq 15 | txresp 16 | 17 | rxreq 18 | txresp 19 | 20 | rxreq 21 | txresp 22 | 23 | rxreq 24 | txresp 25 | 26 | rxreq 27 | txresp 28 | 29 | } -start 30 | 31 | varnish v1 -vcl+backend { 32 | 33 | import boltsort from "${vmod_topbuild}/src/.libs/libvmod_boltsort.so"; 34 | sub vcl_deliver { 35 | set resp.http.naren = boltsort.sort(req.url); 36 | } 37 | 38 | } -start 39 | 40 | client c1 { 41 | 42 | txreq -url "/video/44505073?title=0&byline=0&portrait=0&color=51a516" 43 | rxresp 44 | expect resp.http.naren == "/video/44505073?byline=0&color=51a516&portrait=0&title=0" 45 | 46 | txreq -url "/video/44505073?byline=0&&&&&" 47 | rxresp 48 | expect resp.http.naren == "/video/44505073?byline=0" 49 | 50 | txreq -url "/video/2?&" 51 | rxresp 52 | expect resp.http.naren == "/video/2?" 53 | 54 | txreq -url "/video/2" 55 | rxresp 56 | expect resp.http.naren == "/video/2" 57 | 58 | txreq -url "/video/2?cod=cape&cape=cod" 59 | rxresp 60 | expect resp.http.naren == "/video/2?cape=cod&cod=cape" 61 | 62 | txreq -url "/" 63 | rxresp 64 | expect resp.http.naren == "/" 65 | 66 | } 67 | 68 | client c1 -run 69 | -------------------------------------------------------------------------------- /src/tests/test02.vtc: -------------------------------------------------------------------------------- 1 | varnishtest "Test boltsort req.url in vcl_hash" 2 | 3 | server s1 { 4 | 5 | rxreq 6 | txresp 7 | rxreq 8 | txresp 9 | 10 | } -start 11 | 12 | varnish v1 -vcl+backend { 13 | 14 | import boltsort from "${vmod_topbuild}/src/.libs/libvmod_boltsort.so"; 15 | 16 | sub vcl_hash { 17 | set req.url = boltsort.sort(req.url); 18 | } 19 | 20 | sub vcl_deliver { 21 | if (obj.hits > 0) { 22 | set resp.http.X-Cache = "HIT"; 23 | } 24 | else { 25 | set resp.http.X-Cache = "MISS"; 26 | } 27 | } 28 | 29 | } -start 30 | 31 | client c1 { 32 | 33 | txreq -url "/video/47013255?title=0&byline=0&portrait=0&autoplay=1" 34 | rxresp 35 | expect resp.status == 200 36 | expect resp.http.X-Cache == "MISS" 37 | 38 | txreq -url "/video/47013255?title=0&byline=0&portrait=0&autoplay=1" 39 | rxresp 40 | expect resp.status == 200 41 | expect resp.http.X-Cache == "HIT" 42 | 43 | } 44 | 45 | client c2 { 46 | 47 | txreq -url "/video/47013255?autoplay=1&title=0&byline=0&portrait=0" 48 | rxresp 49 | expect resp.status == 200 50 | expect resp.http.X-Cache == "HIT" 51 | 52 | txreq -url "autoplay=1&title=0&byline=0&portrait=0&&&&" 53 | rxresp 54 | expect resp.status == 200 55 | expect resp.http.X-Cache == "HIT" 56 | 57 | } 58 | 59 | client c2 { 60 | 61 | txreq -url "/video/47013255?autoplay=1&title=0&byline=0&portrait=0" 62 | rxresp 63 | expect resp.status == 200 64 | expect resp.http.X-Cache == "HIT" 65 | 66 | txreq -url "/video/47013255?autoplay=1&title=0&byline=0&portrait=0&&&&" 67 | rxresp 68 | expect resp.status == 200 69 | expect resp.http.X-Cache == "HIT" 70 | 71 | } 72 | 73 | 74 | client c1 -run 75 | client c2 -run 76 | -------------------------------------------------------------------------------- /src/vmod_boltsort.c: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | 5 | #include "vrt.h" 6 | #include "bin/varnishd/cache.h" 7 | 8 | #define MAX_PARAM_COUNT 32 9 | #define EQUALS(c, h) ((c == h) || (c == '\0' && h == '&') || (c == '&' && h == '\0')) 10 | #define ENDS(s) (s == '&' || s == '\0') 11 | #define NOT_ENDS(s) (s != '&' && s != '\0') 12 | 13 | static const char TERMINATORS[2] = {'\0', '&'}; 14 | 15 | //since we dont store param length, we have to evaluate everytime 16 | static inline int param_compare (char *s, char *t) 17 | { 18 | 19 | for ( ;EQUALS(*s, *t); s++, t++) { 20 | if (ENDS(*s)) { 21 | return 0; 22 | } 23 | } 24 | return *s - *t; 25 | 26 | } 27 | 28 | //end of param is either first occurance of & or '\0' 29 | static inline int param_copy(char *dst, char *src, char *last_param) 30 | { 31 | 32 | int len = strchr(src, TERMINATORS[(src != last_param)]) - src; 33 | memcpy(dst, src, len); 34 | return len; 35 | 36 | } 37 | 38 | //Varnish vmod requires this 39 | int init_function(struct vmod_priv *priv, const struct VCL_conf *conf) 40 | { 41 | 42 | return 0; 43 | 44 | } 45 | 46 | //sort query string 47 | const char * vmod_sort(struct sess *sp, const char *url) 48 | { 49 | 50 | if (url == NULL) { 51 | return NULL; 52 | } 53 | 54 | int qs_index = 0; 55 | int param_count = 0; 56 | 57 | char *dst_url = NULL; 58 | char *qs = NULL; 59 | 60 | //To avoid 1 pass for count calculations, assuming MAX_PARAM_COUNT as max 61 | char* params[MAX_PARAM_COUNT]; 62 | 63 | int i, p; 64 | char *param = NULL; 65 | 66 | qs = strchr(url, '?'); 67 | if(!qs) { 68 | return url; 69 | } 70 | 71 | //add first param and increment count 72 | params[param_count++] = ++qs; 73 | qs_index = qs - url; 74 | 75 | //Continue to find query string 76 | while((qs = strchr(qs, '&')) != NULL) { 77 | param = ++qs; 78 | 79 | for(p = 0; p < param_count; p++) { 80 | //if incoming param is < param at position then place it at p and then move up rest 81 | if(param[0] < params[p][0] || param_compare(param, params[p]) < 0) { 82 | for(i = param_count; i > p; i--) { 83 | params[i] = params[i-1]; 84 | } 85 | break; 86 | } 87 | } 88 | params[p] = param; 89 | param_count++; 90 | 91 | //if it exceed max params return as is 92 | if (param_count == MAX_PARAM_COUNT) { 93 | return url; 94 | } 95 | } 96 | 97 | //there is nothing after & 98 | //eg: http://127.0.0.1/?me=1& 99 | if (param_count == 1) { 100 | return url; 101 | } 102 | 103 | //allocate space for sorted url 104 | struct ws *ws = sp->wrk->ws; 105 | dst_url = WS_Alloc(ws, strchr(param, '\0') - url + 1); 106 | WS_Assert(ws); 107 | 108 | //if alloc fails return as is 109 | if(dst_url == NULL) { 110 | return url; 111 | } 112 | 113 | //copy data before query string 114 | char* cp = memcpy(dst_url, url, qs_index) + qs_index; 115 | 116 | //get rid of all empty params /test/?a&&& 117 | for(p = 0; p < param_count - 1; p++) { 118 | if (params[p][0] != '\0' && params[p][0] != '&') { 119 | break; 120 | } 121 | } 122 | 123 | //copy sorted params 124 | for(; p < param_count - 1; p++) { 125 | //copy and increment 126 | cp += param_copy(cp, params[p], param); 127 | *cp++ = '&'; 128 | } 129 | 130 | //copy the last param 131 | cp += param_copy(cp, params[p], param); 132 | *cp = '\0'; 133 | 134 | return dst_url; 135 | 136 | } 137 | -------------------------------------------------------------------------------- /src/vmod_boltsort.vcc: -------------------------------------------------------------------------------- 1 | Module boltsort 2 | Init init_function 3 | Function STRING sort(STRING) 4 | --------------------------------------------------------------------------------