├── .gitignore ├── LICENSE ├── Makefile ├── README.md ├── project.clj ├── resources └── log4j.properties ├── src └── lq │ └── core.clj └── test └── lq └── core_test.clj /.gitignore: -------------------------------------------------------------------------------- 1 | /target 2 | /classes 3 | /checkouts 4 | pom.xml 5 | pom.xml.asc 6 | *.jar 7 | *.class 8 | /.lein-* 9 | /.nrepl-port 10 | .hgignore 11 | .hg/ 12 | .DS_Store 13 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2017 Junegunn Choi 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in 13 | all copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 21 | THE SOFTWARE. 22 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | all: 2 | lein uberjar 3 | 4 | repl: 5 | lein ring server-headless 6 | 7 | .PHONY: all repl 8 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # LQ 2 | 3 | LQ is a simple HTTP server that manages named queues of lines of text in 4 | memory. By using plain-text request and response bodies, it aims to aid shell 5 | scripting scenarios in distributed environments where it's not feasible to set 6 | up proper development tools across the nodes (e.g. all you have is curl). 7 | 8 | The underlying data structure for each named queue is [*LinkedHashSet*][lhs], 9 | so the lines in each queue are unique, which may or may not be desirable 10 | depending on your requirement. So in essense, LQ is just a map of unbounded 11 | *LinkedHashSet*s exposed via HTTP API. No extra effort has been made to make 12 | it more efficient or robust. 13 | 14 | [lhs]: https://docs.oracle.com/javase/8/docs/api/java/util/LinkedHashSet.html 15 | 16 | ## Usage 17 | 18 | ### Starting the server 19 | 20 | ```bash 21 | # Starts LQ server at port 1234 22 | java -jar lq.jar 1234 23 | ``` 24 | 25 | ### Using LQ with curl 26 | 27 | ```bash 28 | # Add lines to the queue named "foobar" 29 | ls -al | curl -XPOST --data-binary @- localhost:1234/foobar 30 | 31 | # Look inside the queue 32 | curl localhost:1234/foobar 33 | 34 | # Remove the first line in the queue and return it 35 | line="$(curl -XPOST localhost:1234/foobar/shift)" 36 | ``` 37 | 38 | ## API Endpoints 39 | 40 | | Path | Method | Request | Response | Description | 41 | | ------------------- | ------ | ------- | ---------------------------------- | ---------------------------------------------------------- | 42 | | `/` | GET | | Names of queues with their counts | List the queues | 43 | | `/` | DELETE | | Number of deleted lines | Remove all queues | 44 | | `/:name` | GET | | Lines in the queue | Return the lines stored in the queue | 45 | | `/:name` | GET | Lines | Matched lines in the queue | Return the matched lines in the queue (containment test) | 46 | | `/:name` | PUT | Lines | Number of lines in the new queue | Recreate queue with the lines | 47 | | `/:name` | POST | Lines | Number of lines added to the queue | Append the lines to the queue | 48 | | `/:name` | DELETE | | Number of lines deleted | Delete the queue | 49 | | `/:name` | DELETE | Lines | Number of lines deleted | Delete the lines from the queue | 50 | | `/:name/shift` | POST | | Removed line | Remove the first line in the queue | 51 | | `/:name1/to/:name2` | POST | | Moved line | Move the first line of the first queue to the second queue | 52 | | `/:name1/to/:name2` | POST | Lines | Moved lines | Move the lines of the first queue to the second queue | 53 | 54 | ## Recipes 55 | 56 | ### Simple task queue 57 | 58 | ```bash 59 | LQ=lq.server.host 60 | 61 | # Upload the list of URLs to the queue named "urls" 62 | curl -XPOST --data-binary @urls.txt $LQ/urls 63 | 64 | # Process each URL in the queue in order 65 | while [ true ]; do 66 | url="$(curl -XPOST --silent $LQ/urls/shift)" 67 | [ -z "$url" ] && break 68 | echo "Processing $url" 69 | # ... 70 | done 71 | ``` 72 | 73 | ### Polling 74 | 75 | ```bash 76 | while [ true ]; do 77 | url="$(curl -XPOST --silent $LQ/urls/shift)" 78 | if [ -z "$url" ]; then 79 | sleep 5 80 | continue 81 | fi 82 | 83 | # ... 84 | done 85 | ``` 86 | 87 | ### Task state transition 88 | 89 | 1. Take the first line in `todo` and add it to `ongoing` 90 | 1. When the task for the line is complete, add it to `complete` 91 | 1. When `todo` becomes empty, check if there are incomplete tasks left in 92 | `ongoing` due to unexpected errors. 93 | 1. Move every line in `ongoing` back to `todo` 94 | 1. Repeat 95 | 96 | ```bash 97 | LQ=lq.server 98 | 99 | # Reset LQ server 100 | curl -XDELETE $LQ/ 101 | 102 | # Build task queue 103 | cat tasks.txt | curl -XPOST --data-binary @- $LQ/todo 104 | 105 | # Process each task 106 | while [ true ]; do 107 | # Move one task to ongoing 108 | url="$(curl -XPOST --silent $LQ/todo/to/ongoing)" 109 | [ -z "$task" ] && break 110 | 111 | # Process the task ... 112 | 113 | # Move the task to done 114 | curl -XPOST --data-binary "$task" $LQ/ongoing/to/complete 115 | done 116 | 117 | # Check the number of completed tasks, are we done? 118 | curl $LQ/complete | wc -l 119 | 120 | # Copy lines in ongoing back to todo 121 | curl $LQ/ongoing | curl -XPOST --data-binary @- $LQ/todo 122 | 123 | # Delete ongoing 124 | curl -XDELETE $LQ/ongoing 125 | 126 | # Repeat until all is done 127 | ``` 128 | 129 | ## Development 130 | 131 | ```sh 132 | # Start nREPL + Ring server 133 | make repl 134 | 135 | # Build uberjar 136 | make 137 | ``` 138 | 139 | ## License 140 | 141 | MIT 142 | -------------------------------------------------------------------------------- /project.clj: -------------------------------------------------------------------------------- 1 | (defproject lq "0.1.0-SNAPSHOT" 2 | :description "A simple HTTP server for queuing lines of text" 3 | :url "https://github.com/junegunn/lq" 4 | :license {:name "MIT"} 5 | :dependencies [[org.clojure/clojure "1.8.0"] 6 | [ring/ring-core "1.5.1"] 7 | [ring/ring-jetty-adapter "1.5.1"] 8 | [ring/ring-defaults "0.2.3"] 9 | [compojure "1.5.2"] 10 | [org.clojure/tools.logging "0.3.1"] 11 | [org.slf4j/slf4j-log4j12 "1.7.25"]] 12 | :plugins [[lein-ring "0.9.7"]] 13 | :main lq.core 14 | :ring {:handler lq.core/api 15 | :nrepl {:start? true}} 16 | :uberjar-name "lq.jar" 17 | :target-path "target/%s" 18 | :profiles {:uberjar {:aot :all}}) 19 | -------------------------------------------------------------------------------- /resources/log4j.properties: -------------------------------------------------------------------------------- 1 | log4j.rootLogger=INFO, STDOUT 2 | log4j.appender.STDOUT=org.apache.log4j.ConsoleAppender 3 | log4j.appender.STDOUT.layout=org.apache.log4j.PatternLayout 4 | log4j.appender.STDOUT.layout.ConversionPattern=%d: %m%n 5 | -------------------------------------------------------------------------------- /src/lq/core.clj: -------------------------------------------------------------------------------- 1 | (ns lq.core 2 | "LQ is a simple HTTP server that manages named queues of lines of text in 3 | memory. By using plain-text request and response bodies, it aims to aid shell 4 | scripting scenarios in distributed environments where it's not feasible to 5 | set up proper development tools across the nodes (e.g. all you have is curl). 6 | 7 | The underlying data structure for each named queue is LinkedHashSet, so the 8 | lines in each queue are unique, which may or may not be desirable depending 9 | on your requirement. So in essense, LQ is just a map of unbounded 10 | LinkedHashSets exposed via HTTP API. No extra effort has been made to make it 11 | more efficient or robust." 12 | (:require [clojure.string :as str] 13 | [clojure.tools.logging :as log] 14 | [compojure.core :refer [DELETE GET POST PUT defroutes]] 15 | [compojure.route :as route] 16 | [ring.adapter.jetty :as jetty] 17 | [ring.middleware.defaults :refer [api-defaults wrap-defaults]] 18 | [ring.util.response :as resp]) 19 | (:import (java.util Iterator LinkedHashSet)) 20 | (:gen-class)) 21 | 22 | (defonce 23 | ^{:doc "Managed queues indexed by their names. TODO: We currently don't 24 | remove queues from the index when they become empty. This can 25 | theoretically be a source of memory leak."} 26 | queues 27 | (atom {})) 28 | 29 | (defonce 30 | ^{:doc "Debug log is disabled by default"} 31 | debug? false) 32 | 33 | (defmacro debug 34 | "Debug macro" 35 | [& args] 36 | (letfn [(fmt [arg] (if (vector? arg) arg [arg]))] 37 | `(when debug? 38 | (let [tail# ~(last args)] 39 | (if (coll? tail#) 40 | (doseq [expr# (if (seq tail#) tail# [""])] 41 | (log/info ~@(map fmt (butlast args)) expr#)) 42 | (log/info ~@(map fmt args))))))) 43 | 44 | (defn ^java.util.Set create-queue 45 | "Creates a new data structured to be used as a queue. We currently use 46 | LinkedHashSet as it's insertion-ordered and it allows fast lookups." 47 | ([] 48 | (LinkedHashSet.)) 49 | ([^java.util.Collection lines] 50 | (LinkedHashSet. lines))) 51 | 52 | (defn ^java.util.Set get-queue 53 | "Returns the queue for the given name. When create is set, creates a new 54 | queue if not found." 55 | ([topic] 56 | (get-queue topic false)) 57 | ([topic create] 58 | (let [got (volatile! nil)] 59 | (swap! queues 60 | (fn [queues] 61 | (if-let [queue (queues topic)] 62 | (do (vreset! got queue) 63 | queues) 64 | (if create 65 | (assoc queues topic (vreset! got (create-queue))) 66 | queues)))) 67 | @got))) 68 | 69 | (defmacro with-queue 70 | "Executes the body with the queue synchronously with explicit locking" 71 | [[name topic & {:keys [create else]}] & body] 72 | `(if-let [~name (get-queue ~topic ~create)] 73 | (locking ~name 74 | ~@body) 75 | ~else)) 76 | 77 | (defn body->lines 78 | "Splits the request body into lines" 79 | [body] 80 | (when body 81 | (remove empty? (str/split (slurp body) #"\n+")))) 82 | 83 | (defn ->line 84 | "Optionally appends a new line character to the string representation of the 85 | given object." 86 | [elem] 87 | (if elem 88 | (str elem "\n") 89 | "")) 90 | 91 | (defn ^String ->lines 92 | "Returns the concatenated string of the lines" 93 | [lines] 94 | (apply str (map ->line lines))) 95 | 96 | (defn index 97 | "Lists all queues with their counts" 98 | [] 99 | (for [[key queue] (sort-by key @queues) 100 | :let [queue-count (locking queue (count queue))] 101 | :when (pos? queue-count)] 102 | (str key " " queue-count))) 103 | 104 | (defn replace! 105 | "Replaces the queue for the given name" 106 | [topic lines] 107 | (let [new-queue (create-queue lines) 108 | new-size (count new-queue)] 109 | (swap! queues assoc topic new-queue) 110 | new-size)) 111 | 112 | (defn push! 113 | "Appends lines to the queue with the given name" 114 | [topic lines] 115 | (with-queue [queue topic :create true] 116 | (count (filter #(.add queue %) lines)))) 117 | 118 | (defn delete! 119 | "Deletes lines from the queues with the given name" 120 | [topic lines] 121 | (with-queue [queue topic :else 0] 122 | (count (filter #(.remove queue %) lines)))) 123 | 124 | (defn clear! 125 | "Empties the queue" 126 | [topic] 127 | (with-queue [queue topic :else 0] 128 | (let [prev-count (count queue)] 129 | (.clear queue) 130 | prev-count))) 131 | 132 | (defn clear-all! 133 | "Removes all data" 134 | [] 135 | (let [deleted (volatile! 0)] 136 | (swap! queues 137 | (fn [queues] 138 | (vreset! deleted 139 | (reduce + (map #(locking % (count %)) 140 | (vals queues)))) 141 | {})) 142 | @deleted)) 143 | 144 | (defn head 145 | "Returns the first element in the Set" 146 | [^java.util.Set queue] 147 | (when-let [^Iterator iterator (some-> queue .iterator)] 148 | (when (.hasNext iterator) 149 | (.next iterator)))) 150 | 151 | (defn shift! 152 | "Removes the first line from the queue and returns the line. If the queue 153 | becomes empty, it will be removed from the index." 154 | [topic] 155 | (with-queue [queue topic] 156 | (let [first-line (head queue)] 157 | (.remove queue first-line) 158 | first-line))) 159 | 160 | (defn move-impl! 161 | "Removes lines from the first queue according to remove-fn and adds the 162 | removed lines to the second queue. remove-fn should return the list of 163 | removed lines." 164 | [topic1 topic2 remove-fn] 165 | (when-let [removed (seq (with-queue [queue1 topic1] (remove-fn queue1)))] 166 | (with-queue [queue2 topic2 :create true] 167 | (.addAll queue2 removed)) 168 | removed)) 169 | 170 | (defn move! 171 | "Removes lines from the first queue and adds them to the second queue" 172 | [topic1 topic2 lines] 173 | (move-impl! topic1 topic2 174 | (fn [^java.util.Set queue] 175 | (and queue (filter #(.remove queue %) lines))))) 176 | 177 | (defn shift-over! 178 | "Removes the first line from the first queue and adds it to the second" 179 | [topic1 topic2] 180 | (move-impl! topic1 topic2 181 | (fn [^java.util.Set queue] 182 | (when-let [first-line (head queue)] 183 | (when (.remove queue first-line) 184 | [first-line]))))) 185 | 186 | (defroutes api-routes 187 | (GET "/" [] 188 | (->lines (index))) 189 | 190 | (DELETE "/" {:keys [remote-addr]} 191 | (debug remote-addr :clear) 192 | (->line (clear-all!))) 193 | 194 | (GET "/:topic" [topic :as {:keys [body]}] 195 | (->lines 196 | (if-let [lines (seq (body->lines body))] 197 | (with-queue [queue topic] 198 | (filter #(.contains queue %) lines)) 199 | (with-queue [queue topic] 200 | (into [] queue))))) 201 | 202 | (POST "/:topic" [topic :as {:keys [remote-addr body]}] 203 | (let [lines (body->lines body)] 204 | (debug remote-addr topic :append lines) 205 | (->line (push! topic lines)))) 206 | 207 | (PUT "/:topic" [topic :as {:keys [remote-addr body]}] 208 | (let [lines (body->lines body)] 209 | (debug remote-addr topic :recreate lines) 210 | (->line (replace! topic lines)))) 211 | 212 | (DELETE "/:topic" [topic :as {:keys [remote-addr body]}] 213 | (->line 214 | (let [lines (body->lines body)] 215 | (debug remote-addr topic :delete lines) 216 | (if-let [lines (seq lines)] 217 | (delete! topic lines) 218 | (clear! topic))))) 219 | 220 | (POST "/:topic/shift" [topic :as {:keys [remote-addr]}] 221 | (let [line (->line (shift! topic))] 222 | (debug remote-addr topic :shift [line]) 223 | line)) 224 | 225 | (POST "/:topic1/to/:topic2" [topic1 topic2 :as {:keys [remote-addr body]}] 226 | (->lines 227 | (let [lines (body->lines body)] 228 | (debug remote-addr [topic1 topic2] :to lines) 229 | (if-let [lines (seq lines)] 230 | (move! topic1 topic2 lines) 231 | (shift-over! topic1 topic2))))) 232 | 233 | (route/not-found "")) 234 | 235 | (defn wrap-plain-text-response 236 | "A middleware that consistently sets the content type of the response to 237 | text/plain" 238 | [handler] 239 | (fn [request] 240 | (resp/content-type 241 | (handler request) 242 | "text/plain; charset=UTF-8"))) 243 | 244 | (def api 245 | "API handler" 246 | (-> api-routes 247 | (wrap-defaults (assoc-in api-defaults [:params :urlencoded] false)) 248 | wrap-plain-text-response)) 249 | 250 | (defn parse-args 251 | "Parses command-line arguments and returns port number. May throw 252 | IllegalArgumentException." 253 | [args] 254 | (let [{flags true args false} (group-by #(str/starts-with? % "-") args)] 255 | {:debug (some? (some #{"-d" "--debug"} flags)) 256 | :port (if-let [port (last args)] 257 | (try (Integer/parseInt port) 258 | (catch NumberFormatException e 259 | (throw (IllegalArgumentException. 260 | (str "Invalid port number: " port))))) 261 | (throw (IllegalArgumentException. "Port number required")))})) 262 | 263 | (defn -main 264 | [& args] 265 | (try 266 | (let [{:keys [debug port]} (parse-args args)] 267 | (when debug (def debug? true)) 268 | (log/info "Start LQ server at port" port) 269 | (jetty/run-jetty api {:port port})) 270 | (catch IllegalArgumentException e 271 | (log/error (.getMessage e)) 272 | (log/info "usage: java -jar lq.jar [-d] PORT") 273 | (System/exit 1)))) 274 | -------------------------------------------------------------------------------- /test/lq/core_test.clj: -------------------------------------------------------------------------------- 1 | (ns lq.core-test 2 | (:require [clojure.test :refer :all] 3 | [lq.core :refer :all]) 4 | (:import (java.io ByteArrayInputStream))) 5 | 6 | (defn ->req 7 | ([method uri] 8 | {:remote-addr "localhost" 9 | :request-method method 10 | :uri uri 11 | :body (ByteArrayInputStream. (byte-array 0))}) 12 | ([method uri lines] 13 | {:remote-addr "localhost" 14 | :request-method method 15 | :uri uri 16 | :body (ByteArrayInputStream. (.getBytes (->lines lines)))})) 17 | 18 | (defn req 19 | [& args] 20 | (-> (apply ->req args) api :body)) 21 | 22 | (def abc ["a" "b" "c"]) 23 | 24 | (def cde ["c" "d" "e"]) 25 | 26 | (deftest api-test 27 | (testing "Lifecyle" 28 | (testing "Delete everything" 29 | (is (re-matches #"[0-9]+\n" (req :delete "/")))) 30 | 31 | (testing "LQ should be empty at this point" 32 | (is (= "" (req :get "/")))) 33 | 34 | (testing "Create a queue with PUT" 35 | (is (= "3\n" (req :put "/foo" abc)))) 36 | 37 | (testing "Put should be idempotent" 38 | (is (= "3\n" (req :put "/foo" abc)))) 39 | 40 | (testing "foo has 3 lines" 41 | (is (= "foo 3\n" (req :get "/")))) 42 | 43 | (testing "foo has a, b, and c" 44 | (is (= (->lines abc) (req :get "/foo")))) 45 | 46 | (testing "bar is empty" 47 | (is (= "" (req :get "/bar")))) 48 | 49 | (testing "Add lines to bar with POST" 50 | (is (= "3\n" (req :post "/bar" abc)))) 51 | 52 | (testing "bar is not empty" 53 | (is (= (->lines abc) (req :get "/bar")))) 54 | 55 | (testing "Add more lines to bar with POST. Two lines added excluding the duplicate line." 56 | (is (= "2\n" (req :post "/bar" cde)))) 57 | 58 | (testing "Already there" 59 | (is (= "0\n" (req :post "/bar" cde)))) 60 | 61 | (testing "bar has 5 lines and foo has 3 lines" 62 | (is (= "bar 5\nfoo 3\n" (req :get "/")))) 63 | 64 | (testing "Shift foo three times" 65 | (doseq [expected abc] 66 | (is (= (->line expected) (req :post "/foo/shift"))))) 67 | 68 | (testing "foo is now empty" 69 | (is (= "" (req :get "/foo")))) 70 | 71 | (testing "You can't shift anymore" 72 | (is (= "" (req :post "/foo/shift")))) 73 | 74 | (testing "bar still has 5 lines and foo is no longer displayed" 75 | (is (= "bar 5\n" (req :get "/")))) 76 | 77 | (testing "Shift from bar and push to foo" 78 | (is (= "a\n" (req :post "/bar/to/foo"))) 79 | (is (= "b\n" (req :post "/bar/to/foo")))) 80 | 81 | (testing "bar now has 3 lines and foo got 2" 82 | (is (= (->lines (take 2 abc)) (req :get "/foo"))) 83 | (is (= (->lines cde) (req :get "/bar"))) 84 | (is (= "bar 3\nfoo 2\n" (req :get "/")))) 85 | 86 | (testing "Containment check" 87 | (is (= "e\nc\n" (req :get "/bar" ["e" "x" "c"])))) 88 | 89 | (testing "Containment check: no result" 90 | (is (= "" (req :get "/bar" ["x" "y" "z"])))) 91 | 92 | ;; foo: a b => a b c e 93 | ;; bar: c d e => d 94 | (testing "Move designated lines" 95 | (is (= "e\nc\n" (req :post "/bar/to/foo" ["e" "x" "c"])))) 96 | 97 | (testing "Delete topic bar" 98 | (is (= "1\n" (req :delete "/bar"))) 99 | (is (= "foo 4\n" (req :get "/")))) 100 | 101 | ;; foo: a b c e 102 | (testing "Delete matching lines" 103 | (is (= "2\n" (req :delete "/foo" ["b" "e" "x"]))) 104 | (is (= "foo 2\n" (req :get "/"))) 105 | (is (= "2\n" (req :delete "/foo" abc))) 106 | (is (= "" (req :get "/")))))) 107 | --------------------------------------------------------------------------------