├── data
├── projects
│ └── raw_clojurellm_projects.json
├── synthetic
│ └── raw_clojurellm_synthetic.json
├── web_crawl
│ └── raw_clojurellm_web_crawl.json
├── stackoverflow
│ └── raw_clojurellm_stackoverflow.json
├── clojurians_chat
│ └── raw_clojurellm_clojurians_chat.json
├── clojurians_forum
│ └── raw_clojurellm_clojurians_forum.json
├── general_programming
│ └── raw_clojurellm_general_programming.json
└── raw_clojurellm_jsonline_sample_dataset.json
├── .gitignore
├── deps.edn
├── LICENSE
├── src
└── clojure_llm
│ └── collect
│ ├── email_groups
│ ├── clojure.clj
│ ├── clojurescript.clj
│ └── common.clj
│ └── data
│ ├── toxic.clj
│ └── names.clj
└── README.md
/data/projects/raw_clojurellm_projects.json:
--------------------------------------------------------------------------------
1 |
--------------------------------------------------------------------------------
/data/synthetic/raw_clojurellm_synthetic.json:
--------------------------------------------------------------------------------
1 |
--------------------------------------------------------------------------------
/data/web_crawl/raw_clojurellm_web_crawl.json:
--------------------------------------------------------------------------------
1 |
--------------------------------------------------------------------------------
/data/stackoverflow/raw_clojurellm_stackoverflow.json:
--------------------------------------------------------------------------------
1 |
--------------------------------------------------------------------------------
/data/clojurians_chat/raw_clojurellm_clojurians_chat.json:
--------------------------------------------------------------------------------
1 |
--------------------------------------------------------------------------------
/data/clojurians_forum/raw_clojurellm_clojurians_forum.json:
--------------------------------------------------------------------------------
1 |
--------------------------------------------------------------------------------
/data/general_programming/raw_clojurellm_general_programming.json:
--------------------------------------------------------------------------------
1 |
--------------------------------------------------------------------------------
/.gitignore:
--------------------------------------------------------------------------------
1 | pom.xml
2 | pom.xml.asc
3 | *.jar
4 | *.class
5 | /lib/
6 | /classes/
7 | /target/
8 | /checkouts/
9 | .lein-deps-sum
10 | .lein-repl-history
11 | .lein-plugins/
12 | .lein-failures
13 | .nrepl-port
14 | .cpcache/
15 | .clj-kondo
16 | .lsp
17 | .calva
18 | .portal
19 |
--------------------------------------------------------------------------------
/deps.edn:
--------------------------------------------------------------------------------
1 | {:deps
2 | {skyscraper/skyscraper {:mvn/version "0.3.4"}
3 | enlive/enlive {:mvn/version "1.1.6"}
4 | reaver/reaver {:mvn/version "0.1.3"}
5 | net.clojars.john/injest {:mvn/version "0.1.0-beta.8"}
6 | org.clojure/data.json {:mvn/version "2.4.0"}}
7 | :aliases
8 | {:email-clojure-main
9 | {:exec-fn clojure-llm.collect.email-groups.clojure/-main}
10 | :email-clojure-sample
11 | {:exec-fn clojure-llm.collect.email-groups.clojure/-sample-main}
12 | :email-clojurescript-main
13 | {:exec-fn clojure-llm.collect.email-groups.clojurescript/-main}
14 | :email-clojurescript-sample
15 | {:exec-fn clojure-llm.collect.email-groups.clojurescript/-sample-main}}}
16 |
--------------------------------------------------------------------------------
/data/raw_clojurellm_jsonline_sample_dataset.json:
--------------------------------------------------------------------------------
1 | [
2 | {"instruction": "Create an vector of length 5 which contains all even numbers between 1 and 10.", "input": "", "output": "(def v [2 4 6 8 10])"},
3 | {"instruction": "Formulate an equation to calculate the height of a triangle given the angle, side lengths and opposite side length.", "input": "", "output": "Height of triangle = opposite side length * sin (angle) / side length" },
4 | {"instruction": "A number is \"perfect\" if the sum of its divisors equal the number itself. 6 is a perfect number because 1+2+3=6. Write a function which returns true for perfect numbers and false otherwise.", "input": "", "output": "(fn [n]\n (= n \n (reduce + (filter #(= 0 (mod n %)) (map inc (range (- n 1))))))\n )" }
5 | ]
6 |
--------------------------------------------------------------------------------
/LICENSE:
--------------------------------------------------------------------------------
1 | MIT License
2 |
3 | Copyright (c) 2023 ruped
4 |
5 | Permission is hereby granted, free of charge, to any person obtaining a copy
6 | of this software and associated documentation files (the "Software"), to deal
7 | in the Software without restriction, including without limitation the rights
8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9 | copies of the Software, and to permit persons to whom the Software is
10 | furnished to do so, subject to the following conditions:
11 |
12 | The above copyright notice and this permission notice shall be included in all
13 | copies or substantial portions of the Software.
14 |
15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21 | SOFTWARE.
22 |
--------------------------------------------------------------------------------
/src/clojure_llm/collect/email_groups/clojure.clj:
--------------------------------------------------------------------------------
1 | (ns clojure-llm.collect.email-groups.clojure
2 | (:require
3 | [clojure.data.json :as json]
4 | [clojure.edn :as edn]
5 | [clojure.string :as str]
6 | [clojure-llm.collect.data.names :as names]
7 | [clojure-llm.collect.data.toxic :as toxic]
8 | [clojure-llm.collect.email-groups.common :as c]
9 | [reaver]
10 | [injest.path :refer [x>> =>>]]
11 | [skyscraper.core :as core :refer [defprocessor]]))
12 |
13 | (def base-page "https://www.mail-archive.com/clojure@googlegroups.com/")
14 |
15 | (def first-page (str base-page "msg107207.html"))
16 |
17 | (def seed
18 | [{:url first-page
19 | :processor :landing-page
20 | :page 1}])
21 |
22 | (def p-atom (atom #{}))
23 |
24 | (defn update-url [url]
25 | (let [base-count (if (str/starts-with? url "https://") (count base-page) 0)
26 | old-url (->> url
27 | (drop-last 5)
28 | (drop (+ 3 base-count))
29 | (apply str))
30 | ends-with-hash? (str/ends-with? old-url "#")
31 | cleaned-url (-> old-url (str/replace "#" ""))
32 | p (->> cleaned-url edn/read-string dec)
33 | p (if ends-with-hash? (dec p) p)
34 | p (if (@p-atom p) (dec p) p)]
35 | (swap! p-atom conj p)
36 | (when (< 10000 p)
37 | (let [new-url (str "msg" p ".html")]
38 | new-url))))
39 |
40 | (defprocessor :landing-page
41 | :cache-template "clojure-google-mail-group/:page"
42 | :process-fn (fn [doc context]
43 | (concat
44 | (reaver/extract-from doc ".content"
45 | [:subject :body]
46 | ".msgHead" reaver/text
47 | ".msgBody" reaver/text)
48 | (when-let [next-page (-> context :url update-url)]
49 | (let [next-page (if-not (= next-page "#")
50 | next-page
51 | (-> context :url update-url))]
52 | [{:url (str base-page next-page)
53 | :processor :landing-page
54 | :page (inc (:page context))}])))))
55 |
56 | (defn run []
57 | (core/scrape seed
58 | :html-cache true
59 | :download-error-handler
60 | (fn [_ _ c]
61 | [(assoc c :url (str base-page (->> c :url update-url)))])
62 | :parse-fn core/parse-reaver))
63 |
64 | (def sample-raw-data-file-name "data/clojure_mailgroup/clojure_llm_clojure_mailgroup_sample.json")
65 | (def raw-data-file-name "data/clojure_mailgroup/clojure_llm_clojure_mailgroup.json")
66 | (def data-file-name "data/clojure_mailgroup/clojure_llm_clojure_mailgroup_prompts.json")
67 | (def sample-data-file-name "data/clojure_mailgroup/clojure_llm_clojure_mailgroup_prompts_sample.json")
68 |
69 | (defn -download-emails []
70 | (reset! p-atom #{})
71 | (let [res (run)]
72 | (spit raw-data-file-name "[\n")
73 | (->> res
74 | (interpose ",")
75 | (partition-all 2)
76 | (map (fn [[p c]]
77 | (str (json/write-str (dissoc p :page)) c \newline)))
78 | (mapv #(spit raw-data-file-name % :append true)))
79 | (spit raw-data-file-name "]" :append true)))
80 |
81 | #_(-download-emails)
82 |
83 |
84 | (defn -process-emails []
85 | (let [res (c/mk-data (c/get-mail raw-data-file-name))]
86 | (spit data-file-name "[\n")
87 | (x>> res
88 | (interpose ",")
89 | (partition-all 2)
90 | (map (fn [[p c]]
91 | (str (json/write-str (dissoc p :page)) c \newline)))
92 | (mapv #(spit data-file-name % :append true)))
93 | (spit data-file-name "]" :append true)))
94 |
95 | #_(-process-emails)
96 |
97 | (defn -main [& args]
98 | (-download-emails)
99 | (-process-emails))
100 |
101 | #_(-main)
102 |
103 | (defn create-sample []
104 | (reset! p-atom #{})
105 | (let [res (run)
106 | res2 (=>> res
107 | (take 1000)
108 | (interpose ",")
109 | (partition-all 2)
110 | (map (fn [[p c]]
111 | (if-not c
112 | (str (json/write-str (dissoc p :page)) \newline)
113 | (str (json/write-str (dissoc p :page)) c \newline)))))
114 | ]
115 | (spit sample-raw-data-file-name "[\n")
116 | (->> res2
117 | (mapv #(spit sample-raw-data-file-name % :append true)))
118 | (spit sample-raw-data-file-name "]" :append true)))
119 |
120 | #_(create-sample)
121 |
122 | (defn -process-sample-emails [& _args]
123 | (let [res (c/mk-data (c/get-mail sample-raw-data-file-name))]
124 | (spit sample-data-file-name "[\n")
125 | (=>> res
126 | (interpose ",")
127 | (partition-all 2)
128 | (map (fn [[p c]]
129 | (str (json/write-str (dissoc p :page)) c \newline)))
130 | (map #(spit sample-data-file-name % :append true)))
131 | (spit sample-data-file-name "]" :append true)))
132 |
133 | #_(-process--sample-emails)
134 |
135 |
136 | (defn -sample-main [& args]
137 | (create-sample)
138 | (-process-sample-emails))
139 |
140 | ;; 22 seconds
141 | ;; user: 0m52.903s
--------------------------------------------------------------------------------
/src/clojure_llm/collect/email_groups/clojurescript.clj:
--------------------------------------------------------------------------------
1 | (ns clojure-llm.collect.email-groups.clojurescript
2 | (:require
3 | [clojure.data.json :as json]
4 | [clojure.edn :as edn]
5 | [clojure.string :as str]
6 | [clojure-llm.collect.data.names :as names]
7 | [clojure-llm.collect.data.toxic :as toxic]
8 | [clojure-llm.collect.email-groups.common :as c]
9 | [reaver]
10 | [injest.path :refer [x>> =>>]]
11 | [skyscraper.core :as core :refer [defprocessor]]))
12 |
13 | (def base-page "https://www.mail-archive.com/clojurescript@googlegroups.com/")
14 |
15 | (def first-page (str base-page "msg08581.html"))
16 |
17 | (def seed
18 | [{:url first-page
19 | :processor :landing-page
20 | :page 1}])
21 |
22 | (def p-atom (atom #{}))
23 |
24 | (defn pad-zeros [n]
25 | (let [c (count (str n))]
26 | (if-not (< c 5)
27 | (str n)
28 | (let [diff (- 5 c)
29 | nstr (str (apply str (take diff (cycle ["0"]))) n)]
30 | nstr))))
31 |
32 | (defn update-url [url]
33 | (let [base-count (if (str/starts-with? url "https://") (count base-page) 0)
34 | old-url (->> url
35 | (drop-last 5)
36 | (drop (+ 3 base-count))
37 | (apply str))
38 | ends-with-hash? (str/ends-with? old-url "#")
39 | cleaned-url (-> old-url (str/replace "#" ""))
40 | p (->> cleaned-url Long/valueOf dec)
41 | p (if ends-with-hash? (dec p) p)
42 | p (if (@p-atom p) (dec p) p)]
43 | (swap! p-atom conj p)
44 | (when (< -1 p)
45 | (let [new-url (str "msg" (pad-zeros p) ".html")]
46 | new-url))))
47 |
48 | (defprocessor :landing-page
49 | :cache-template "clojurescript-google-mail-group/:page"
50 | :process-fn (fn [doc context]
51 | (concat
52 | (reaver/extract-from doc ".content"
53 | [:subject :body]
54 | ".msgHead" reaver/text
55 | ".msgBody" reaver/text)
56 | (when-let [next-page (-> context :url update-url)]
57 | (let [next-page (if-not (= next-page "#")
58 | next-page
59 | (-> context :url update-url))]
60 | [{:url (str base-page next-page)
61 | :processor :landing-page
62 | :page (inc (:page context))}])))))
63 |
64 | (defn run []
65 | (core/scrape seed
66 | :html-cache true
67 | :download-error-handler
68 | (fn [_ _ c]
69 | [(assoc c :url (str base-page (->> c :url update-url)))])
70 | :parse-fn core/parse-reaver))
71 |
72 | (def raw-data-file-name "data/clojurescript_mailgroup/clojure_llm_clojurescript_mailgroup.json")
73 | (def data-file-name "data/clojurescript_mailgroup/clojure_llm_clojurescript_mailgroup_prompts.json")
74 |
75 | (def sample-raw-data-file-name "data/clojurescript_mailgroup/clojure_llm_clojurescript_mailgroup_sample.json")
76 | (def sample-data-file-name "data/clojurescript_mailgroup/clojure_llm_clojurescript_mailgroup_prompts_sample.json")
77 |
78 | (defn -download-emails []
79 | (reset! p-atom #{})
80 | (let [res (run)]
81 | (spit raw-data-file-name "[\n")
82 | (->> res
83 | (interpose ",")
84 | (partition-all 2)
85 | (map (fn [[p c]]
86 | (str (json/write-str (dissoc p :page)) c \newline)))
87 | (mapv #(spit raw-data-file-name % :append true)))
88 | (spit raw-data-file-name "]" :append true)))
89 |
90 | #_(-download-emails)
91 |
92 | (defn -process-emails []
93 | (let [res (c/mk-data (c/get-mail raw-data-file-name))]
94 | (spit data-file-name "[\n")
95 | (x>> res
96 | (interpose ",")
97 | (partition-all 2)
98 | (map (fn [[p c]]
99 | (str (json/write-str (dissoc p :page)) c \newline)))
100 | (mapv #(spit data-file-name % :append true)))
101 | (spit data-file-name "]" :append true)))
102 |
103 | #_(-process-emails)
104 |
105 | (defn -main [& args]
106 | (-download-emails)
107 | (-process-emails))
108 |
109 | #_(-main)
110 |
111 | (defn create-sample []
112 | (reset! p-atom #{})
113 | (let [res (run)
114 | res2 (=>> res
115 | (take 1000)
116 | (interpose ",")
117 | (partition-all 2)
118 | (map (fn [[p c]]
119 | (if-not c
120 | (str (json/write-str (dissoc p :page)) \newline)
121 | (str (json/write-str (dissoc p :page)) c \newline)))))]
122 | (spit sample-raw-data-file-name "[\n")
123 | (->> res2
124 | (mapv #(spit sample-raw-data-file-name % :append true)))
125 | (spit sample-raw-data-file-name "]" :append true)))
126 |
127 | #_(create-sample)
128 |
129 | (defn -process-sample-emails [& _args]
130 | (let [res (c/mk-data (c/get-mail sample-raw-data-file-name))]
131 | (spit sample-data-file-name "[\n")
132 | (=>> res
133 | (interpose ",")
134 | (partition-all 2)
135 | (map (fn [[p c]]
136 | (str (json/write-str (dissoc p :page)) c \newline)))
137 | (map #(spit sample-data-file-name % :append true)))
138 | (spit sample-data-file-name "]" :append true)))
139 |
140 | #_(-process-sample-emails)
141 |
142 |
143 | (defn -sample-main [& args]
144 | (create-sample)
145 | (-process-sample-emails))
146 |
147 | #_(-sample-main)
148 |
--------------------------------------------------------------------------------
/src/clojure_llm/collect/email_groups/common.clj:
--------------------------------------------------------------------------------
1 | (ns clojure-llm.collect.email-groups.common
2 | (:require
3 | [clojure.data.json :as json]
4 | [clojure.edn :as edn]
5 | [clojure.string :as str]
6 | [clojure-llm.collect.data.names :as names]
7 | [clojure-llm.collect.data.toxic :as toxic]
8 | [reaver]
9 | [injest.path :refer [x>> =>>]]
10 | [skyscraper.core :as core :refer [defprocessor]]))
11 |
12 | (defn remove-message [body]
13 | (first (str/split body #"You received this message")))
14 |
15 | (defn remove-dashes [body]
16 | (first (str/split body #"--~--~---------")))
17 |
18 | (defn get-non-space-tokens [s]
19 | (=>> s
20 | (partition-by (complement #{\space \newline}))
21 | (filter (complement #(= \space (first %))))
22 | (map #(apply str %))))
23 |
24 | (defn might-have-code? [body]
25 | (when body
26 | (and
27 | (str/includes? body "(")
28 | (str/includes? body ")"))))
29 |
30 | (defn attach-data [m]
31 | (let [s1 (get m "subject")
32 | s2 (-> s1 (str/replace "[ClojureScript]" ""))
33 | reply? (or (str/starts-with? s2 "Re:")
34 | (str/starts-with? s2 "RE:")
35 | (str/starts-with? s2 "re:")
36 | (str/starts-with? s2 " Re:")
37 | (str/starts-with? s2 " RE:")
38 | (str/starts-with? s2 " re:"))
39 | subject0 (->> s2 (drop-last 32) (apply str))
40 | subject1 (if-not reply?
41 | subject0
42 | (apply str (drop 4 subject0)))
43 | subject-tokens (get-non-space-tokens subject1)
44 | short? (< (count subject-tokens) 3)
45 | [subject author] (if short?
46 | [(first subject-tokens) (last subject-tokens)]
47 | [(apply str (interpose " " subject-tokens)) (apply str (interpose " " (take-last 2 subject-tokens)))])
48 | [author-first-name author-last-name] (get-non-space-tokens author)
49 | body (some->> (get m "body") remove-message remove-dashes)
50 | toxic-body? (seq (filter toxic/bad-names (get-non-space-tokens body)))] ;; check for toxicity
51 | (-> m
52 | (assoc :timestamp (->> s2 (take-last 31) (apply str)))
53 | (assoc :og-subj subject0)
54 | (assoc :subject subject)
55 | (assoc :sub-match (apply str (take 15 subject))) ;; match on first 15 characters -- we don't have stable subject end strings
56 | (assoc :author author)
57 | (assoc :toxic? (boolean (first toxic-body?)))
58 | (assoc :author-first-name author-first-name)
59 | (assoc :author-last-name author-last-name)
60 | (assoc :might-have-code? (might-have-code? body))
61 | (assoc :body body)
62 | (dissoc "subject" "body")
63 | (merge (if reply?
64 | {:reply? true :original? false}
65 | {:reply? false :original? true})))))
66 |
67 | (defn not-derrida [m]
68 | (and (or (:original? m) (:might-have-code? m)) ;; if it's a reply, let's make sure it at least has some code
69 | (not (:toxic? m)) ;; let's check for toxic language
70 | (not
71 | (or ;; a bunch of words in subjects indicative of announcements
72 | (str/includes? (:subject m) "Call for Contributions")
73 | (str/includes? (:subject m) "Call for Participation")
74 | (str/includes? (:subject m) "Job")
75 | (str/includes? (:subject m) "JOB")
76 | (str/includes? (:subject m) "CfP")
77 | (str/includes? (:subject m) "CFP")
78 | (str/includes? (:subject m) "Call for Papers")
79 | (str/includes? (:subject m) "is now available")
80 | (str/includes? (:subject m) "hiring")
81 | (str/includes? (:subject m) "meeting")
82 | (str/includes? (:subject m) "meetup")
83 | (str/includes? (:subject m) "Clojurists Together")
84 | (str/includes? (:subject m) "Online Meetup")
85 | (str/includes? (:subject m) "Online meetup")
86 | (str/starts-with? (:subject m) "[Blog]")
87 | (str/starts-with? (:subject m) "[BLOG]")
88 | (str/starts-with? (:subject m) "[CFP]")
89 | (str/starts-with? (:subject m) "[JOB]")
90 | (str/starts-with? (:subject m) "{JOB}")
91 | (str/starts-with? (:subject m) "[ANN]")
92 | (str/starts-with? (:subject m) "RFC")
93 | (str/starts-with? (:subject m) "ANN:")
94 | (str/starts-with? (:subject m) "Announcement")
95 | (str/starts-with? (:subject m) "Ann:")))))
96 |
97 | (defn clean-groups [groups]
98 | (->> groups
99 | (map (fn [[k v]]
100 | (let [original (->> v (filter :original?) first)
101 | original-author (:author original)
102 | non-author-replies (->> v
103 | (filter #(not= original-author (:author %)))
104 | (into [original]))]
105 | [k non-author-replies]))) ;; we only care about the original question and direct answers to the original question (not from the original author)
106 | (into {})))
107 |
108 | (defn pair-groups [groups]
109 | (->> groups
110 | (map (fn [[k v]]
111 | (let [original (->> v (filterv :original?) first)
112 | [psuedo-first-name psuedo-last-name] (names/rand-first-last) ;; anonymize original author per conversation thread
113 | others (->> v (filterv :reply?))
114 | original-author-first-name (:author-first-name original)
115 | original-author-last-name (:author-last-name original)
116 | original-prompt (str (:subject original) "\n"
117 | (:body original))
118 | prompt (->> original-prompt
119 | get-non-space-tokens
120 | (map #(cond
121 | (str/includes? (or % "") (or original-author-first-name "")) psuedo-first-name
122 | (str/includes? (or % "") (or original-author-last-name "")) psuedo-last-name
123 | :else %))
124 | (interpose " ")
125 | (apply str))]
126 | [k
127 | (->> others
128 | (mapv (fn [reply]
129 | (let [reply-author-first-name (:author-first-name reply)
130 | reply-author-last-name (:author-last-name reply)
131 | [psuedo-reply-first-name psuedo-reply-last-name] (names/rand-first-last) ;; anonymize reply author per convo prompt pair
132 | response (->> reply
133 | :body
134 | get-non-space-tokens
135 | (map #(cond
136 | (str/includes? (or % "") (or original-author-first-name "")) psuedo-first-name
137 | (str/includes? (or % "") (or original-author-last-name "")) psuedo-last-name
138 | (str/includes? (or % "") (or reply-author-first-name "")) psuedo-reply-first-name
139 | (str/includes? (or % "") (or reply-author-last-name "")) psuedo-reply-last-name
140 | :else %))
141 | (interpose " ")
142 | (apply str))]
143 | {:prompt prompt
144 | :response response}))))])))
145 | (into {})))
146 |
147 | (defn mk-data [mail]
148 | (x>> mail
149 | (map attach-data)
150 | (filter not-derrida)
151 | (group-by :sub-match)
152 | (filter #(and (->> % second count (not= 1))
153 | (->> % second (filter :original?) first)))
154 | clean-groups
155 | pair-groups
156 | (map second)
157 | (filter #(> 4 (count %))) ;; limit conversation size to 3 replies
158 | (apply concat)
159 | (filterv (comp (complement nil?) seq))))
160 |
161 | (defn get-mail [raw-data-file-name]
162 | (json/read-str (slurp raw-data-file-name)))
163 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # clojurellm-data
2 | Clojure LLM - Dataset curation for fine tuning an LLM for Clojure.
3 |
4 | | Dataset | Location | Size | Launch Main | Launch Sample |
5 | | ------------- | ------------- | ------------- | ------------- | ------------- |
6 | | Clojure email groups | [data/clojure_mailgroup](/data/clojure_mailgroup) | 25.31MB | clj -X:email-clojure-main | clj -X:email-clojure-sample |
7 | | Clojurescript email groups | [data/clojurescript_mailgroup](./data/clojurescript_mailgroup) | 3.1MB | clj -X:email-clojurescript-main | clj -X:email-clojurescript-sample |
8 | | Clojurians chat logs | [data/clojurians_chat](./data/clojurians_chat) | N/A | N/A | N/A |
9 | | Clojurians forum | [data/clojurians_forum](./data/clojurians_forum) | N/A | N/A | N/A |
10 | | General programming | [data/general_programming](./data/general_programming) | N/A | N/A | N/A |
11 | | Clojure/script Projects | [data/projects](./data/projects) | N/A | N/A | N/A |
12 | | Stackoverflow | [data/stackoverflow](./data/stackoverflow) | N/A | N/A | N/A |
13 | | Synthetic Clojure | [data/synthetic](./data/synthetic) | N/A | N/A | N/A |
14 | | Clojure web crawl | [data/web_crawl](./data/web_crawl) | N/A | N/A | N/A |
15 |
16 |
17 | ## ClojureLLM Data Management Policy
18 |
19 | 1. Intro
20 |
21 | 1. What
22 |
23 | The purpose of the data in this repository is for the fine-tuning of an LLM for a Clojure coding assistant.
24 |
25 | The purpose of this document is to outline the policies and procedures by which data custodians acting on behalf of the ClojureLLM project manage the security and safety of the data being used in the project, as well as other information useful to different stakeholders. These documents are sometimes called "datasheets."
26 | Who
29 |
30 | ClojureLLM Data is developed and supported by members of the Clojure community for the benefit of the Clojure community.
31 |
32 | In this document, any ClojureLLM developer working on the data in this repository shall be referred to as a "Data Custodian."
33 | Support and Funding
36 |
37 | Infrastructure for this project is currently funded by the ClojureLLM team but we will have a method for folks to contribute funds for training runs soon.
38 | Data Sources
45 |
46 | 1. Clojure Code Data Sources
47 |
48 | ClojureLLM will use the following sources of Clojure code for data.
49 |
50 | - Github/gitlab Clojure project crawl
51 | - https://github.com/phronmophobic/dewey
52 | - Synthetically generated clojure code projects and code conversations
53 | - Santacoder fine-tune stack for Clojure
54 | - https://huggingface.co/mrm8488/santacoder-finetuned-the-stack-clojure
55 |
56 | Not all of these sources may be used and others may be added to this list over time.
57 | Clojure Conversation Data Sources
60 |
61 | ClojureLLM will use the following sources of Clojure code for data.
62 |
63 | - Clojurians Slack message (as question answer pairs)
64 | - https://clojurians-log.clojureverse.org/
65 | - Clojureverse forum logs
66 | - https://clojureverse.org/
67 | - Stackoverflow questions tagged with Clojure
68 | - https://stackoverflow.com/questions/tagged/clojure
69 | - Clojure/script mailing lists
70 | - https://www.mail-archive.com/clojure@googlegroups.com/
71 | - https://www.mail-archive.com/clojurescript@googlegroups.com/
72 | - Crawl Clojure RSS feed history
73 | - https://planet.clojure.in/atom.xml
74 |
75 | Not all of these sources may be used and others may be added to this list over time.
76 | Non-Clojure Code Data Sources
79 |
80 | ClojureLLM may leverage some existing and/or future datasets, made available in the larger open source community, so as to facilitate the translation of programming concepts from other languages into Clojure.
81 | - programming datasets
82 | - https://www.bigcode-project.org/
83 | - programming instruction sets from other coding assistants
84 | - https://github.com/yaodongC/awesome-instruction-dataset
85 | Collection
92 |
93 | 1. Scraping
94 |
95 | ClojureLLM Data Custodian may use any Clojure web scraping tool they'd like, but [skyscraper](https://github.com/nathell/skyscraper) is recommended.
96 |
97 | Before scraping any given site, ensure the copyright of the site does not prohibit the usage of its code-related data for LLM training for any reason.
98 |
99 | The script for a given dataset should be added as a launch alias in the project `deps.edn` in order to run the script.
100 |
101 | Storing
104 |
105 | Due to storage constraints on Github, ClojureLLM will not be storing entire datasets in the repo. User's running particular pipelines will execute the download/scraping scripts for the dataset they're working on instead.
106 |
107 | However, it is advised to store a small _sample_ of the dataset that the scripts will produce, so that folks can experiment without having to run the scrape.
108 |
109 | > Note: We'd like to keep the repo under 100MB in general
110 | Sanatization
115 |
116 | 1. Remove Garbage
117 |
118 | We're only interested in the Clojure code and the human langauge related to the Clojure code. However, those values will usually be embedded within HTML, JSON and various document formats. That data should be purged from the dataset.
119 | Deduplication
122 |
123 | It is possible that some the code or conversation data exists in more than one location on the internet. So it's possible for there to be duplicates the scraped data. Therefore, it is the responsibility of the data custodian defining the download script to eliminate duplication of data both within their dateset as well as the rest of the datasets in the repo.
124 | Remove ClojureLLM Outputs
127 |
128 | ClojureLLM outputs may end up in chat logs and we don't want to waste test space allocated for human training data. This likely won't be a huge problem - just be sure to avoid including massive amounts of outputs, especially from ClojureLLM.
129 |
130 | We may eventually develop some data watermarks and tools to later help automatically detect ClojureLLM code in text for possible elision from the dataset.
131 | Toxicity and Bias
134 |
135 | Make an effort to remove toxicity, sarcasm, hyperbole, bias, personal opinions, jokes, or anything not related to Clojure code or advice around the usage and understanding of Clojure code and other related programming technologies.
136 |
137 | We plan on having LLM based sentiment/semantic classification tools in the future to help automate the detection of toxicity and general divergence from the target content for ClojureLLM. Different datasets will then be able to leverage those tools.
138 | PII
141 |
142 | Data custodians should make an effort to remove Personally Identifiable Information, including but not limited to:
143 | - credit card numbers
144 | - personal names (except library authors)
145 | - emails (except library/solution contact info)
146 | - home addresses
147 | - social security numbers
148 | - phone numbers
149 | - financial data
150 | - publicly accessble IP addresses (not local)
151 | - employer of speaker
152 | - social network handles
153 | - anyone mentioning their name explicitly
154 |
155 | We plan to provide PII scanning tools for datasets that can be used generically from all of the dataset collection scripts.
156 |
157 |
158 | 6. Injection Attack Detection
159 |
160 | One potential danger of LLMs is the ability for an attacker to surruptitiously poisons public datasets with data that either corrupts the data or injects prompts or information into the data that produces undesirable inference or side-effects in the LLM training on and infering on the data.
161 |
162 | We are still learning about this mode of attack, but as our understanding increases, we plan to automate the detection and removal of these instances from ClojureLLM datasets.
163 | User Anonymization
166 |
167 | Data custodians should make an effort to anonymize the users associated with Clojure code and conversations around Clojure code.
168 |
169 | This includes:
170 | - cross-conversation anonymization
171 | - psuedo names will be wellknown names
172 | - ["Bob" "Alice" "Jamal" "Myleen" "Oliver" etc]
173 | - 50% male / 50% famale names (open to comment)
174 | - psuedo names will be ethnically / culturally diverse
175 | - Redact descriptions of human likenesses ("oh, no, I have green eyes")
176 | - Redact any mentioning children or family members
177 |
178 | These items are open to feedback and expansion. In general, we want to represent a diversity of backgrounds for a dataset that is helpful to existing and future Clojurists around the world.
179 |
180 | Tools for anonymizing users will be shared across the different dataset pipelines as they are built out.
181 |
182 | Data Enrichment
189 |
190 | 1. Clojure Code Generation
191 |
192 | A large part of this project will involve the synthetic generation of large amounts of Clojure code, so as to give the ClojureLLM a very deep intuition around how the Clojure compiler behaves.
193 |
194 | This is open question and we hope the community will give feedback on how best to accomplish this.
195 |
196 | Eventually we may use Clojure code generation tools to help grow out the other datasets in this repository as well.
197 | Conversation Grammar
200 |
201 | There is often grammatical and syntactical errors in common language between humans. We can correct these errors though with tools that will automatically fix those mistakes, which can increase the comprehensibility of the training data.
202 |
203 | Again, data custodians that build tools for cleaning up grammar in a particular dataset should make an effort to make those same tools available in the rest of the datasets in this repository.
204 | Usage
209 |
210 | 1. Code Completion
211 |
212 | One model will be used primarily for Code Completion. This model will be smaller, fit in more applications and will execute faster, for more immediate feedback while the Clojurist is typing.
213 | Code Conversation / Pair Programming
216 |
217 | Another model will be used for pair programming, asking ClojureLLM questions and getting a written response in natural language explaining the answer.
218 |
219 | This model will be necessarily larger, to understand more general natural language concepts, translating between them and code concepts. It will also be slower, as the Clojurist will see the words in the response be written out in realtime.
220 | Distribution
225 |
226 | 1. Open / Restricted
227 |
228 | Some models that we'll be starting off with may have licenses that restrict what we can do with them. Some allow for commercial use, others do not. We intend on supporting and working on both.
229 |
230 | If the best model available, that can provide the best experience for Clojure devs, is a restricted model, we may still want to use that in some projects, like an open source LSP server that can use an LLM. Because an open source project like that is not commercial, it is free to use models that have a commercial restriction. We're not going to go with a lesser model for that purpose, just because it cannot be used commercially.
231 |
232 | That being said, a stated purpose of this project is to also make available the development of commercial Clojure applications on top of LLM-based technologies.
233 | Large / Small
236 |
237 | As stated above, code completion models will likely need to be smaller, in order to be fast and useful. Conversational models will likely need to be larger.
238 |
239 | That being said, this space is evolving fast and smaller and larger models with different performance characteristics will continue coming out and we intend to experiment with many of them.
240 | Maintenance
245 |
246 | 1. Community Feedback
247 |
248 | The direction of this project is an open community effort and it is likely to change as things progress, so we encourage everyone to engage and provide feedback on what can be improved and where you'd like to see things go.
249 |
250 | Feel free to file a PR to update this document or file an issue if you have an questions or concerns.
251 | Dataset Versioning and Updates
254 |
255 | Some datasets will be made available in the Releases of this project. A zip file of all the datasets will be made available on huggingface.
256 |