![\"[Digital](\"/images/static_banner.gif\")
",
102 | }
103 |
104 | for _, killstr := range removestrings {
105 | htmlpage = strings.Replace(htmlpage, killstr, "", -1)
106 | }
107 |
108 | // Now to start rewriting things with things that are in the DB
109 | lines := strings.Split(htmlpage, "\n")
110 | for _, line := range lines {
111 | if strings.Contains(line, "![[AltaVista]](/images/av_personal.gif)
11 | ![[Advanced]](/images/av_advanced.gif)
12 | ![[Simple]](/images/av_simple.gif)
13 | ![[Help]](/images/av_help.gif)
14 | 16 |
 |
35 | Digital Equipment Corporation
36 | Copyright 1997 © All Rights Reserved |
37 |  |
38 |
65 | 66 | 67 |  68 | 69 | The search engine works: 70 | 71 |  72 | 73 | As NeoZeed found out, the search interface only listens on loopback (hardcoded), which is very annoying if you want to expose it to the wider world! To solve this [stunnel](https://www.stunnel.org/index.html) 74 | was deployed to listen on * and relay connections back to the local instance, with added SSL! [Using a pretty questionable (but CT logged) default SSL certificate too](https://crt.sh/?id=130496527) [(A)](http://archive.is/WJNYb)! 75 | 76 | ## Providing data to the indexer 77 | 78 | Because the indexer and backend is a Windows 98 QEMU VM, there has to be a way of giving that VM approximately 5GB of tiny files for the AltaVista indexer to see and include in the index. For this, I chose to make a FAT32 file system every 24 hours as a snapshot of the crawled data, and then restart the crawling VM to see the new files. This worked really quite well in testing with a small amount of files, however a few issues became apparent: first of all FAT32 has file limits that need to be paid attention to. For example, FAT32 is not able to have more than 255 files in the root directory, so you have to be sure to spread out your files in folder structures. 79 | 80 | Another issue to keep in mind is that the maximum drive size for FAT32 is 32GB (approx). This means the amount of textual content can't go bigger than that (or we would have to spawn more virtual drives). Fortunately, the size of the crawled content is far below that so this is a non issue. 81 | 82 | While crawling a "production size" data set, the system would reset at a random point during the indexing program running: 83 | 84 |  85 | 86 | After tweaking the settings involving the disk caches, a slightly more constructive error was obtained: 87 | 88 |  89 | 90 | This is good! I can search for `exception 05 has occurred at 0028:C2920074`! Right? As it happens, there is very little information about this kind of crash on the internet (it may have existed at some point in the past, but since been removed; after all, the OS is 20 years old), however the one piece of information I could gather from searching is that it was VFAT driver related. Suspecting a bad combo between high IO load and QEMU's [INT_13](https://en.wikipedia.org/wiki/INT_13H) implementation, I went for the only other file system/data input system available, CD/DVD ROM! 91 | 92 | After doing a small 500MB test (a test that FAT32 could not pass) we had a small index! 93 | 94 |  95 | 96 | At this point we had to scale up the solution to the 300k / 4 GB of files. I discovered that Windows 98 does support DVDs, even though the UI can only display that the drive is 2GB, even if the drive is much larger than that. Despite that, all content was accessible on the drive and an initial index was (slowly) built. 97 | 98 | ## Sanitise the index interface 99 | 100 | The only problem with using a 20 year old indexer, is that it's likely a **very** bad idea to expose directly to the internet. The other issue is that on most of the pages the interface serves references local (as in, `file://`) assets, meaning that a simple reverse proxy would not work. 101 | 102 | In addition, local paths are not very useful to people searching. For this `alta-sanitise` was written to provide a sane front end to it, while still keeping the Windows 98 AltaVista index as it's backend. 103 | 104 | To do this, I produce a file system containing all the files that were downloaded, and name them by their database ID: 105 | 106 |  107 | 108 | However in alta-sanitise, we use the database we formed using crawling, to rewrite the URL's into something viewable: 109 | 110 |  111 | 112 | To ensure the VM could be used for more than one project, lighttpd was put in front as a reverse proxy of `alta-sanitise`, and Cloudflare used for cache. This leaves the final flow looking like this: 113 | 114 |  115 | 116 | # Monitoring Windows 98 117 | 118 | Most of my servers are monitored using collectd. Unfortunately there is no Windows 98 client for collectd (!?), so I decided to make one. 119 | 120 | A simple Visual Basic 6 application will poll every 10 seconds and output [collectd command strings](https://collectd.org/wiki/index.php/Plain_text_protocol#PUTVAL) over the serial port (where it can be passed on to collectd on the hypervisor): 121 | 122 |  123 | 124 |  125 | 126 | This code can be found separately at: https://github.com/benjojo/win32-collectd 127 | 128 | (Some troubled soul may find this useful outside of a gopher crawler.) 129 | 130 | # Giving back to the community 131 | 132 | Now that I have a sizeable index of the gopher space, I feel like I should give back to gopher space: 133 | 134 |  135 | 136 | You can now find my blog on gopher at `gopher.blog.benjojo.co.uk` (** only accessible on gopher, Lynx supports gopher if that helps **). 137 | 138 | The search engine can be used at: http://gophervista.benjojo.co.uk 139 | 140 | And the code can be found at: https://github.com/benjojo/gophervista -------------------------------------------------------------------------------- /blog/bsod.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/benjojo/gophervista/0778fcb538da8532fa01652dc290c551aa1543c6/blog/bsod.png -------------------------------------------------------------------------------- /blog/demo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/benjojo/gophervista/0778fcb538da8532fa01652dc290c551aa1543c6/blog/demo.png -------------------------------------------------------------------------------- /blog/example-search.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/benjojo/gophervista/0778fcb538da8532fa01652dc290c551aa1543c6/blog/example-search.png -------------------------------------------------------------------------------- /blog/flow.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/benjojo/gophervista/0778fcb538da8532fa01652dc290c551aa1543c6/blog/flow.png -------------------------------------------------------------------------------- /blog/grafana.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/benjojo/gophervista/0778fcb538da8532fa01652dc290c551aa1543c6/blog/grafana.png -------------------------------------------------------------------------------- /blog/index-built.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/benjojo/gophervista/0778fcb538da8532fa01652dc290c551aa1543c6/blog/index-built.png -------------------------------------------------------------------------------- /blog/installer.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/benjojo/gophervista/0778fcb538da8532fa01652dc290c551aa1543c6/blog/installer.png -------------------------------------------------------------------------------- /blog/random-crash.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/benjojo/gophervista/0778fcb538da8532fa01652dc290c551aa1543c6/blog/random-crash.gif -------------------------------------------------------------------------------- /blog/rewritten-search.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/benjojo/gophervista/0778fcb538da8532fa01652dc290c551aa1543c6/blog/rewritten-search.png -------------------------------------------------------------------------------- /blog/simple-search.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/benjojo/gophervista/0778fcb538da8532fa01652dc290c551aa1543c6/blog/simple-search.png -------------------------------------------------------------------------------- /blog/tty-my-blog.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/benjojo/gophervista/0778fcb538da8532fa01652dc290c551aa1543c6/blog/tty-my-blog.gif -------------------------------------------------------------------------------- /blog/tty-of-gopher.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/benjojo/gophervista/0778fcb538da8532fa01652dc290c551aa1543c6/blog/tty-of-gopher.gif -------------------------------------------------------------------------------- /crawler/crawl.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "encoding/base32" 5 | "fmt" 6 | "io/ioutil" 7 | "log" 8 | "os" 9 | "time" 10 | 11 | "github.com/benjojo/gophervista/rfc1436" 12 | ) 13 | 14 | func crawl(Queue crawlerQueue, datadir string) { 15 | var workers map[string]chan string 16 | workers = make(map[string]chan string) 17 | indexn := 0 18 | skiphost := "impossibleaaaaaaaaaaaaaaaaaa" 19 | 20 | for { 21 | time.Sleep(time.Millisecond * 5) 22 | var asset string 23 | asset, indexn = Queue.GetItemToCrawlFromIndex(indexn, skiphost) 24 | if asset == "" { 25 | log.Print("Nothing to crawl...") 26 | indexn = 0 27 | continue 28 | } 29 | indexn++ 30 | 31 | canCrawl, _ := Queue.CheckItemPerms(asset) 32 | 33 | if !canCrawl { 34 | log.Printf("Can't crawl %s because of restriction", asset) 35 | Queue.FlagItemAsCrawled(asset) 36 | continue 37 | } 38 | 39 | _, hn, _, err := rfc1436.ParseURI(asset) 40 | if err != nil { 41 | log.Printf("Invalid URI, %s", err.Error()) 42 | continue 43 | } 44 | 45 | if workers[hn] == nil { 46 | ch := make(chan string, 1) 47 | workers[hn] = ch 48 | go crawlWorker(ch, Queue, datadir) 49 | go crawlerFeeder(ch, hn, Queue) 50 | } 51 | select { 52 | case workers[hn] <- asset: // ignore if it's full. 53 | skiphost = "impossibleaaaaaaaaaaa" 54 | default: 55 | skiphost = hn 56 | } 57 | 58 | } 59 | } 60 | 61 | func crawlerFeeder(assetChan chan string, hn string, Queue crawlerQueue) { 62 | indexn := 0 63 | totallydry := 0 64 | 65 | for { 66 | time.Sleep(time.Millisecond * 500) 67 | var asset string 68 | asset, indexn = Queue.GetItemToCrawlFromIndexFromHost(indexn, hn) 69 | if asset == "" { 70 | log.Printf("Nothing to crawl, Reseting pointer for %s", hn) 71 | indexn = 0 72 | totallydry++ 73 | if totallydry > 1 { 74 | time.Sleep(time.Minute) 75 | log.Printf("Really lothing left to crawl for %s, Chilling back for a bit", hn) 76 | } 77 | continue 78 | } 79 | indexn++ 80 | 81 | canCrawl, _ := Queue.CheckItemPerms(asset) 82 | 83 | if !canCrawl { 84 | log.Printf("Can't crawl %s because of restriction", asset) 85 | Queue.FlagItemAsCrawled(asset) 86 | continue 87 | } 88 | 89 | assetChan <- asset 90 | totallydry = 0 91 | } 92 | } 93 | 94 | func crawlWorker(assetChan chan string, Queue crawlerQueue, datadir string) { 95 | for asset := range assetChan { 96 | if asset == "" { 97 | log.Print("Nothing to crawl...") 98 | continue 99 | } 100 | 101 | _, canLearn := Queue.CheckItemPerms(asset) 102 | 103 | time.Sleep(time.Second) 104 | 105 | d, ci, err := rfc1436.Get(asset) 106 | log.Printf("Grabbing %s", asset) 107 | if err != nil { 108 | log.Printf("Failed to crawl '%s' - %s", asset, err.Error()) 109 | Queue.FlagItemAsCrawled(asset) 110 | continue 111 | } 112 | 113 | storefolder := fmt.Sprintf("%s/raw/%s-%d/", datadir, ci.Hostname, ci.Port) 114 | err = os.MkdirAll(storefolder, 0755) 115 | if err != nil && err != os.ErrExist { 116 | log.Printf("Unable to make directory structure (%s) to store responce in, %s", storefolder, err.Error()) 117 | } 118 | 119 | ioutil.WriteFile(storefolder+base32shortcut(asset), d.Raw, 0755) 120 | 121 | for _, ln := range d.Items { 122 | if ln.Type == rfc1436.TypeMenuEntity || 123 | ln.Type == rfc1436.TypeTextFile { 124 | newpath := fmt.Sprintf("gopher://%s:%d%s", ln.Host, ln.Port, ln.Path) 125 | if canLearn { 126 | Queue.FlagItem(newpath, ln.Type) 127 | } else { 128 | log.Printf("Can't add %s to crawl list, Domain has restriction in place", newpath) 129 | } 130 | } 131 | } 132 | Queue.FlagItemAsCrawled(asset) 133 | } 134 | } 135 | 136 | func base32shortcut(path string) string { 137 | return base32.StdEncoding.EncodeToString([]byte(path)) 138 | } 139 | -------------------------------------------------------------------------------- /crawler/database.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "database/sql" 5 | "fmt" 6 | "log" 7 | "regexp" 8 | "sync" 9 | "time" 10 | ) 11 | 12 | type crawlerQueue struct { 13 | db *sql.DB 14 | dblock sync.Mutex 15 | flagchan chan FlagRequest 16 | } 17 | 18 | func (cq *crawlerQueue) Init(databasepath string) error { 19 | db, err := sql.Open("sqlite3", databasepath) 20 | if err != nil { 21 | return err 22 | } 23 | 24 | err = db.Ping() // database do odd things 25 | if err != nil { 26 | return err 27 | } 28 | 29 | cq.db = db 30 | 31 | cq.maketableifnotexists("assets", 32 | "CREATE TABLE assets (id INTEGER PRIMARY KEY, timestamp NUMERIC, path TEXT, type TEXT, lastcrawled NUMERIC);") 33 | 34 | cq.maketableifnotexists("restrictions", 35 | "CREATE TABLE `restrictions` (`id` INTEGER,`pattern` TEXT,`nolearn` INTEGER DEFAULT 0,`nocrawl` INTEGER DEFAULT 0,PRIMARY KEY(id));") 36 | 37 | cq.flagchan = make(chan FlagRequest, 1) 38 | go cq.FlagChannelConsumer(cq.flagchan) 39 | 40 | return nil 41 | } 42 | 43 | func (cq *crawlerQueue) maketableifnotexists(table, makestring string) { 44 | tablename := "" 45 | cq.db.QueryRow("SELECT name FROM sqlite_master WHERE type='table' AND name=?;", table).Scan(&tablename) 46 | 47 | if tablename != table { 48 | log.Printf("Made a new %s table", table) 49 | cq.db.Exec(makestring) 50 | } 51 | 52 | } 53 | 54 | func (cq *crawlerQueue) GetItemToCrawl() string { 55 | cq.dblock.Lock() 56 | defer cq.dblock.Unlock() 57 | asset := "" 58 | err := cq.db.QueryRow("SELECT path FROM assets WHERE lastcrawled < ? LIMIT 1", time.Now().Unix()-604800).Scan(&asset) 59 | if err != nil { 60 | return "" 61 | } 62 | return asset 63 | } 64 | 65 | func (cq *crawlerQueue) GetItemToCrawlFromIndex(indexin int, skiphost string) (asset string, index int) { 66 | cq.dblock.Lock() 67 | defer cq.dblock.Unlock() 68 | err := cq.db.QueryRow("SELECT path,id FROM assets WHERE lastcrawled < ? AND id > ? AND path NOT LIKE ? LIMIT 1", 69 | time.Now().Unix()-604800, indexin, fmt.Sprintf("%%%s%%", skiphost)).Scan(&asset, &index) 70 | if err != nil { 71 | return "", 0 72 | } 73 | return asset, index 74 | } 75 | 76 | func (cq *crawlerQueue) GetItemToCrawlFromIndexFromHost(indexin int, targethost string) (asset string, index int) { 77 | cq.dblock.Lock() 78 | defer cq.dblock.Unlock() 79 | err := cq.db.QueryRow("SELECT path,id FROM assets WHERE lastcrawled < ? AND id > ? AND path LIKE ? LIMIT 1", 80 | time.Now().Unix()-604800, indexin, fmt.Sprintf("%%%s%%", targethost)).Scan(&asset, &index) 81 | if err != nil { 82 | return "", 0 83 | } 84 | return asset, index 85 | } 86 | 87 | func (cq *crawlerQueue) FlagItemAsCrawled(path string) { 88 | _, err := cq.db.Exec("UPDATE assets SET lastcrawled = ? WHERE path = ?", time.Now().Unix(), path) 89 | if err != nil { 90 | log.Printf("Unable to flag asset as crawled? %s", err.Error()) 91 | if err.Error() == "database is locked" { // aaaa a aaaa aaaaaaaa terrible i'm so sorry 92 | log.Printf("Retrying in a moment due to %s", err.Error()) 93 | go func() { 94 | // Well we are already doing sins here, so what's one more? 95 | time.Sleep(time.Second) 96 | cq.FlagItemAsCrawled(path) 97 | }() 98 | } 99 | } 100 | } 101 | 102 | type FlagRequest struct { 103 | Path string 104 | GTypeStr string 105 | } 106 | 107 | func (cq *crawlerQueue) FlagItem(path string, gtypestr string) { 108 | cq.flagchan <- FlagRequest{ 109 | Path: path, 110 | GTypeStr: gtypestr, 111 | } 112 | } 113 | 114 | func (cq *crawlerQueue) FlagChannelConsumer(in chan FlagRequest) { 115 | for rq := range in { 116 | cq.dblock.Lock() 117 | test := 0 118 | err := cq.db.QueryRow("SELECT COUNT(*) path FROM assets WHERE path = ?;", rq.Path).Scan(&test) 119 | if err != nil { 120 | log.Printf("Huh, unable to see if an item to be should be crawled - %s", err.Error()) 121 | continue 122 | } 123 | 124 | if test == 0 { 125 | log.Printf("Found item to add %s", rq.Path) 126 | 127 | // Fantastic, We can add it 128 | _, err = cq.db.Exec("INSERT INTO `assets` (`id`,`timestamp`,`path`,`type`,`lastcrawled`) VALUES (NULL,?,?,?,0);", time.Now().Unix(), rq.Path, rq.GTypeStr) 129 | if err != nil { 130 | log.Printf("Huh, unable to flag to be should be crawled - %s", err.Error()) 131 | continue 132 | } 133 | } 134 | cq.dblock.Unlock() 135 | } 136 | } 137 | 138 | type permissionRow struct { 139 | Pattern *regexp.Regexp 140 | CanLearn bool 141 | CanCrawl bool 142 | } 143 | 144 | var permissionsCache []permissionRow 145 | 146 | func (cq *crawlerQueue) CheckItemPerms(path string) (canCrawl bool, canLearn bool) { 147 | if len(permissionsCache) == 0 || permissionsCache == nil { 148 | log.Printf("Loading perms") 149 | permissionsCache = make([]permissionRow, 0) 150 | rows, err := cq.db.Query("SELECT pattern,nolearn,nocrawl FROM restrictions;") 151 | if err != nil { 152 | log.Fatalf("Unable to load crawling restrictions/perms, %s", err.Error()) 153 | } 154 | 155 | for rows.Next() { 156 | var iCanLearn, iCanCrawl int 157 | var RERaw string 158 | 159 | err := rows.Scan(&RERaw, &canLearn, &canCrawl) 160 | if err != nil { 161 | log.Fatalf("Unable to process crawling restrictions/perms, %s", err.Error()) 162 | } 163 | 164 | RE := regexp.MustCompile(RERaw) 165 | 166 | n := permissionRow{ 167 | Pattern: RE, 168 | CanCrawl: iCanCrawl == 0, 169 | CanLearn: iCanLearn == 0, 170 | } 171 | 172 | permissionsCache = append(permissionsCache, n) 173 | } 174 | log.Printf("%d perm restrictions loaded", len(permissionsCache)) 175 | } 176 | 177 | // Now that it's all loaded... 178 | 179 | for _, testcase := range permissionsCache { 180 | if testcase.Pattern.MatchString(path) { 181 | return !testcase.CanCrawl, !testcase.CanLearn 182 | } 183 | } 184 | 185 | return true, true 186 | } 187 | -------------------------------------------------------------------------------- /crawler/main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "flag" 5 | "log" 6 | "os" 7 | 8 | _ "github.com/mattn/go-sqlite3" 9 | ) 10 | 11 | var ( 12 | databasepath = flag.String("dbpath", "./crawl.db", "Where to store the metadata of crawling") 13 | datapath = flag.String("datadir", "./data", "where to download assets to") 14 | ) 15 | 16 | func main() { 17 | flag.Parse() 18 | 19 | ledger := crawlerQueue{} 20 | err := ledger.Init(*databasepath) 21 | if err != nil { 22 | log.Fatalf("Unable to open database. %s", err.Error()) 23 | } 24 | 25 | crawl(ledger, *datapath) 26 | os.Exit(1) 27 | } 28 | -------------------------------------------------------------------------------- /filesystem-gen/main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "database/sql" 5 | "encoding/base32" 6 | "flag" 7 | "fmt" 8 | "io" 9 | "log" 10 | "os" 11 | "path/filepath" 12 | 13 | _ "github.com/mattn/go-sqlite3" 14 | ) 15 | 16 | func main() { 17 | dbpath := flag.String("dbpath", "./crawl.db", "Where to find the database") 18 | opath := flag.String("outputpath", "./cds/", "Where to put the folders") 19 | flag.Parse() 20 | log.Printf("Making filesystem...") 21 | 22 | db, err := sql.Open("sqlite3", *dbpath) 23 | if err != nil { 24 | log.Fatalf("unable to open db %s", err.Error()) 25 | } 26 | 27 | err = db.Ping() // database do odd things 28 | if err != nil { 29 | log.Fatalf("unable to check db %s", err.Error()) 30 | } 31 | 32 | filesmoved := 0 33 | cd := 0 34 | bytesOnDisk := 0 35 | 36 | os.MkdirAll(*opath+fmt.Sprint(cd)+"/0/", 0777) 37 | 38 | filepath.Walk("./data", func(path string, info os.FileInfo, err error) error { 39 | if err != nil { 40 | return nil 41 | } 42 | if info.IsDir() { 43 | return nil 44 | } 45 | 46 | encodedfilename := filepath.Base(path) 47 | gopherurl, err := base32.StdEncoding.DecodeString(encodedfilename) 48 | if err != nil { 49 | return nil 50 | } 51 | 52 | var dbid int 53 | err = db.QueryRow("SELECT id FROM assets WHERE path=? LIMIT 1", string(gopherurl)).Scan(&dbid) 54 | 55 | if err != nil || dbid == 0 { 56 | log.Printf("%s", err.Error()) 57 | return nil 58 | } 59 | 60 | if bytesOnDisk > 1900000000 { 61 | bytesOnDisk = 0 62 | cd++ 63 | os.MkdirAll(fmt.Sprintf("%s%d/", *opath, cd), 0777) 64 | } 65 | bytesOnDisk += int(info.Size()) 66 | 67 | if filesmoved%1000 == 0 { 68 | os.MkdirAll(fmt.Sprintf("%s%d/%d/", *opath, cd, filesmoved/1000), 0777) 69 | } 70 | 71 | dstpath := fmt.Sprintf("%s%d/%d/%d.txt", *opath, cd, filesmoved/1000, dbid) 72 | err = copy(path, dstpath) 73 | if err != nil { 74 | log.Printf("failed to copy file over to %s - %s", dstpath, err.Error()) 75 | } 76 | 77 | filesmoved++ 78 | 79 | return nil 80 | }) 81 | } 82 | 83 | func copy(src, dst string) error { 84 | in, err := os.Open(src) 85 | if err != nil { 86 | return err 87 | } 88 | defer in.Close() 89 | out, err := os.Create(dst) 90 | if err != nil { 91 | return err 92 | } 93 | defer out.Close() 94 | _, err = io.Copy(out, in) 95 | cerr := out.Close() 96 | if err != nil { 97 | return err 98 | } 99 | return cerr 100 | } 101 | -------------------------------------------------------------------------------- /filesystem-gen/mkfs.sh: -------------------------------------------------------------------------------- 1 | #/bin/bash 2 | set -e 3 | 4 | cp crawl.db snapshot.db 5 | 6 | # if ! df -h | grep -Fq loop0; then 7 | # echo "Need to make FS" 8 | # losetup -d /dev/loop0 || echo "Non fatal, continuing" 9 | # dd if=/dev/zero of=filesystem.img bs=1 count=0 seek=8G 10 | # sfdisk filesystem.img < filesystem-gen/layout.sfdisk 11 | # losetup /dev/loop0 filesystem.img 12 | # mkfs.fat /dev/loop0p1 13 | # mkdir fat32 || echo "Unable to make fat32 dir, probs because it exists" 14 | # mount /dev/loop0p1 ./fat32/ 15 | # fi 16 | 17 | mkdir fat32 || echo "Non fatal, continuing" 18 | 19 | cd fat32 && rm -r * || echo "wasnt able to remove things in the FS, probs because it was empty anyway" 20 | 21 | # # Zero out the drive so that it compresses better 22 | # for i in {1..8} 23 | # do 24 | # dd if=/dev/zero of=$i.stuff bs=1 count=0 seek=1G || echo "." 25 | # done 26 | # rm *.stuff 27 | # cd .. 28 | ./bin/filesystem-gen -dbpath snapshot.db 29 | #umount ./fat32 30 | #losetup -d /dev/loop0 31 | #qemu-img convert -O qcow2 filesystem.img image.qcow2 32 | rm mega.iso || echo "." 33 | xorrisofs -o mega.iso fat32/ -------------------------------------------------------------------------------- /go.mod: -------------------------------------------------------------------------------- 1 | module github.com/benjojo/gophervista 2 | 3 | go 1.16 4 | 5 | require github.com/mattn/go-sqlite3 v1.14.10 6 | -------------------------------------------------------------------------------- /go.sum: -------------------------------------------------------------------------------- 1 | github.com/mattn/go-sqlite3 v1.14.10 h1:MLn+5bFRlWMGoSRmJour3CL1w/qL96mvipqpwQW/Sfk= 2 | github.com/mattn/go-sqlite3 v1.14.10/go.mod h1:NyWgC/yNuGj7Q9rpYnZvas74GogHl5/Z4A/KQRfk6bU= 3 | -------------------------------------------------------------------------------- /rfc1436/gopher.go: -------------------------------------------------------------------------------- 1 | package rfc1436 2 | 3 | import ( 4 | "fmt" 5 | "io/ioutil" 6 | "net" 7 | "time" 8 | ) 9 | 10 | func Get(uri string) (out Document, info GopherConnectionInfo, err error) { 11 | bin, ci, err := GetBinary(uri) 12 | if err != nil { 13 | return out, GopherConnectionInfo{}, err 14 | } 15 | 16 | Doc, err := parseDocument(bin) 17 | if err != nil { 18 | return out, ci, err 19 | } 20 | 21 | return Doc, ci, err 22 | } 23 | 24 | func GetBinary(uri string) (out []byte, info GopherConnectionInfo, err error) { 25 | path, hostname, port, err := ParseURI(uri) 26 | if err != nil { 27 | return out, GopherConnectionInfo{}, err 28 | } 29 | 30 | return requestRaw(path, hostname, port) 31 | } 32 | 33 | func requestRaw(path, hostname string, port int) (out []byte, info GopherConnectionInfo, err error) { 34 | conn, err := net.DialTimeout("tcp", fmt.Sprintf("%s:%d", hostname, port), time.Second*5) 35 | 36 | cinfo := GopherConnectionInfo{ 37 | Path: path, 38 | Hostname: hostname, 39 | Port: port, 40 | } 41 | 42 | if err != nil { 43 | return out, cinfo, err 44 | } 45 | conn.SetReadDeadline(time.Now().Add(time.Second * 10)) 46 | 47 | defer conn.Close() 48 | 49 | if path == "/" { 50 | path = "" 51 | } 52 | 53 | payload := fmt.Sprintf("%s\r\n", path) 54 | _, err = conn.Write([]byte(payload)) 55 | if err != nil { 56 | return out, cinfo, err 57 | } 58 | 59 | out, err = ioutil.ReadAll(conn) 60 | return out, cinfo, err 61 | } 62 | -------------------------------------------------------------------------------- /rfc1436/parse.go: -------------------------------------------------------------------------------- 1 | package rfc1436 2 | 3 | import ( 4 | "strconv" 5 | "strings" 6 | ) 7 | 8 | // Gopher Message type codes 9 | // See: https://www.ietf.org/rfc/rfc1436.txt 10 | const ( 11 | TypeTextFile = "0" 12 | TypeMenuEntity = "1" 13 | TypeCSOProtocol = "2" 14 | TypeErr = "3" 15 | TypeMacintoshBinHex = "4" 16 | TypePCDOS = "5" 17 | Typeuuencoded = "6" 18 | TypeIndexServer = "7" 19 | TypeTelnetSession = "8" 20 | TypeBinaryFile = "9" 21 | TypeDuplicatedServer = "+" 22 | TypeGIF = "g" 23 | TypeImage = "I" 24 | TypeTN3270 = "T" 25 | TypeHTML = "h" 26 | TypeMessage = "i" 27 | TypeWAVSound = "s" 28 | ) 29 | 30 | // Document repersents a Gopher document that has been 31 | // parsed and ready to be processed for either 32 | // reading or rendering 33 | type Document struct { 34 | Raw []byte 35 | Items []Item 36 | } 37 | 38 | type Item struct { 39 | Type string 40 | Content string 41 | Host string 42 | Port int 43 | Path string 44 | } 45 | 46 | func parseDocument(in []byte) (D Document, err error) { 47 | D.Raw = in 48 | D.Items = make([]Item, 0) 49 | 50 | lines := strings.Split(string(in), "\r\n") 51 | for _, line := range lines { 52 | if line == "." { 53 | break 54 | } 55 | 56 | if !strings.Contains(line, "\t") { 57 | continue 58 | } 59 | 60 | if len(line) < 5 { 61 | // too short to be a gopher listing 62 | continue 63 | } 64 | 65 | if !isValidGopherType(string(line[:1])) { 66 | // is unparsable 67 | continue 68 | } 69 | 70 | e := Item{} 71 | e.Type = string(line[:1]) 72 | parts := strings.Split(line[1:], "\t") 73 | if len(parts) != 4 { 74 | // Unexpected amounts of tabs 75 | continue 76 | } 77 | 78 | e.Content = parts[0] 79 | if !strings.HasPrefix(parts[1], "/") { 80 | parts[1] = "/" + parts[1] 81 | } 82 | e.Path = parts[1] 83 | e.Host = parts[2] 84 | pn, err := strconv.ParseInt(parts[3], 10, 16) 85 | if err != nil { 86 | // invalid port number 87 | continue 88 | } 89 | e.Port = int(pn) 90 | 91 | D.Items = append(D.Items, e) 92 | } 93 | return D, nil 94 | } 95 | 96 | func isValidGopherType(in string) bool { 97 | if in == "0" || in == "1" || in == "2" || in == "3" || 98 | in == "4" || in == "5" || in == "6" || in == "7" || 99 | in == "8" || in == "9" || in == "+" || in == "g" || 100 | in == "I" || in == "T" || in == "h" || in == "i" || 101 | in == "s" { 102 | return true 103 | } 104 | return false 105 | } 106 | -------------------------------------------------------------------------------- /rfc1436/uri.go: -------------------------------------------------------------------------------- 1 | package rfc1436 2 | 3 | import ( 4 | "fmt" 5 | "net/url" 6 | "strconv" 7 | "strings" 8 | ) 9 | 10 | type GopherConnectionInfo struct { 11 | Path string 12 | Hostname string 13 | Port int 14 | } 15 | 16 | func ParseURI(inputURI string) (path, hostname string, port int, err error) { 17 | if strings.HasPrefix(inputURI, "gopher://") { 18 | U, err := url.Parse(inputURI) 19 | if err != nil { 20 | return "", "", 0, err 21 | } 22 | 23 | // Now figure the port out 24 | if strings.Contains(U.Host, ":") { 25 | parts := strings.Split(U.Host, ":") 26 | if len(parts) != 2 { 27 | return "", "", 0, fmt.Errorf("the URI provided has more than one colon") 28 | } 29 | 30 | p, err := strconv.ParseUint(parts[1], 10, 16) 31 | if err != nil { 32 | return "", "", 0, fmt.Errorf("port on URI is invalid") 33 | } 34 | return U.Path, parts[0], int(p), nil 35 | } 36 | return U.Path, U.Host, 70, nil 37 | } 38 | return ParseURI("gopher://" + inputURI) 39 | } 40 | -------------------------------------------------------------------------------- /vendor/github.com/mattn/go-sqlite3/.codecov.yml: -------------------------------------------------------------------------------- 1 | coverage: 2 | status: 3 | project: off 4 | patch: off 5 | -------------------------------------------------------------------------------- /vendor/github.com/mattn/go-sqlite3/.gitignore: -------------------------------------------------------------------------------- 1 | *.db 2 | *.exe 3 | *.dll 4 | *.o 5 | 6 | # VSCode 7 | .vscode 8 | 9 | # Exclude from upgrade 10 | upgrade/*.c 11 | upgrade/*.h 12 | 13 | # Exclude upgrade binary 14 | upgrade/upgrade 15 | -------------------------------------------------------------------------------- /vendor/github.com/mattn/go-sqlite3/LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2014 Yasuhiro Matsumoto 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /vendor/github.com/mattn/go-sqlite3/README.md: -------------------------------------------------------------------------------- 1 | go-sqlite3 2 | ========== 3 | 4 | [](http://godoc.org/github.com/mattn/go-sqlite3) 5 | [](https://github.com/mattn/go-sqlite3/actions?query=workflow%3AGo) 6 | [](https://opencollective.com/mattn-go-sqlite3) 7 | [](https://codecov.io/gh/mattn/go-sqlite3) 8 | [](https://goreportcard.com/report/github.com/mattn/go-sqlite3) 9 | 10 | Latest stable version is v1.14 or later, not v2. 11 | 12 | ~~**NOTE:** The increase to v2 was an accident. There were no major changes or features.~~ 13 | 14 | # Description 15 | 16 | A sqlite3 driver that conforms to the built-in database/sql interface. 17 | 18 | Supported Golang version: See [.github/workflows/go.yaml](./.github/workflows/go.yaml). 19 | 20 | This package follows the official [Golang Release Policy](https://golang.org/doc/devel/release.html#policy). 21 | 22 | ### Overview 23 | 24 | - [go-sqlite3](#go-sqlite3) 25 | - [Description](#description) 26 | - [Overview](#overview) 27 | - [Installation](#installation) 28 | - [API Reference](#api-reference) 29 | - [Connection String](#connection-string) 30 | - [DSN Examples](#dsn-examples) 31 | - [Features](#features) 32 | - [Usage](#usage) 33 | - [Feature / Extension List](#feature--extension-list) 34 | - [Compilation](#compilation) 35 | - [Android](#android) 36 | - [ARM](#arm) 37 | - [Cross Compile](#cross-compile) 38 | - [Google Cloud Platform](#google-cloud-platform) 39 | - [Linux](#linux) 40 | - [Alpine](#alpine) 41 | - [Fedora](#fedora) 42 | - [Ubuntu](#ubuntu) 43 | - [Mac OSX](#mac-osx) 44 | - [Windows](#windows) 45 | - [Errors](#errors) 46 | - [User Authentication](#user-authentication) 47 | - [Compile](#compile) 48 | - [Usage](#usage-1) 49 | - [Create protected database](#create-protected-database) 50 | - [Password Encoding](#password-encoding) 51 | - [Available Encoders](#available-encoders) 52 | - [Restrictions](#restrictions) 53 | - [Support](#support) 54 | - [User Management](#user-management) 55 | - [SQL](#sql) 56 | - [Examples](#examples) 57 | - [*SQLiteConn](#sqliteconn) 58 | - [Attached database](#attached-database) 59 | - [Extensions](#extensions) 60 | - [Spatialite](#spatialite) 61 | - [FAQ](#faq) 62 | - [License](#license) 63 | - [Author](#author) 64 | 65 | # Installation 66 | 67 | This package can be installed with the `go get` command: 68 | 69 | go get github.com/mattn/go-sqlite3 70 | 71 | _go-sqlite3_ is *cgo* package. 72 | If you want to build your app using go-sqlite3, you need gcc. 73 | However, after you have built and installed _go-sqlite3_ with `go install github.com/mattn/go-sqlite3` (which requires gcc), you can build your app without relying on gcc in future. 74 | 75 | ***Important: because this is a `CGO` enabled package, you are required to set the environment variable `CGO_ENABLED=1` and have a `gcc` compile present within your path.*** 76 | 77 | # API Reference 78 | 79 | API documentation can be found [here](http://godoc.org/github.com/mattn/go-sqlite3). 80 | 81 | Examples can be found under the [examples](./_example) directory. 82 | 83 | # Connection String 84 | 85 | When creating a new SQLite database or connection to an existing one, with the file name additional options can be given. 86 | This is also known as a DSN (Data Source Name) string. 87 | 88 | Options are append after the filename of the SQLite database. 89 | The database filename and options are separated by an `?` (Question Mark). 90 | Options should be URL-encoded (see [url.QueryEscape](https://golang.org/pkg/net/url/#QueryEscape)). 91 | 92 | This also applies when using an in-memory database instead of a file. 93 | 94 | Options can be given using the following format: `KEYWORD=VALUE` and multiple options can be combined with the `&` ampersand. 95 | 96 | This library supports DSN options of SQLite itself and provides additional options. 97 | 98 | Boolean values can be one of: 99 | * `0` `no` `false` `off` 100 | * `1` `yes` `true` `on` 101 | 102 | | Name | Key | Value(s) | Description | 103 | |------|-----|----------|-------------| 104 | | UA - Create | `_auth` | - | Create User Authentication, for more information see [User Authentication](#user-authentication) | 105 | | UA - Username | `_auth_user` | `string` | Username for User Authentication, for more information see [User Authentication](#user-authentication) | 106 | | UA - Password | `_auth_pass` | `string` | Password for User Authentication, for more information see [User Authentication](#user-authentication) | 107 | | UA - Crypt | `_auth_crypt` |I never thought I be doing this again, and yet, here we are! Join me in the adventures of "oh god we are installing windows 98 again" pic.twitter.com/Ub83PQ4sV7
— Ben Cox (@Benjojo12) May 6, 2017
553 |
554 | ### Financial Contributors
555 |
556 | Become a financial contributor and help us sustain our community. [[Contribute here](https://opencollective.com/mattn-go-sqlite3/contribute)].
557 |
558 | #### Individuals
559 |
560 | 
561 |
562 | #### Organizations
563 |
564 | Support this project with your organization. Your logo will show up here with a link to your website. [[Contribute](https://opencollective.com/mattn-go-sqlite3/contribute)]
565 |
566 | 
567 | 
568 | 
569 | 
570 | 
571 | 
572 | 
573 | 
574 | 
575 | 
576 |
577 | # License
578 |
579 | MIT: http://mattn.mit-license.org/2018
580 |
581 | sqlite3-binding.c, sqlite3-binding.h, sqlite3ext.h
582 |
583 | The -binding suffix was added to avoid build failures under gccgo.
584 |
585 | In this repository, those files are an amalgamation of code that was copied from SQLite3. The license of that code is the same as the license of SQLite3.
586 |
587 | # Author
588 |
589 | Yasuhiro Matsumoto (a.k.a mattn)
590 |
591 | G.J.R. Timmer
592 |
--------------------------------------------------------------------------------
/vendor/github.com/mattn/go-sqlite3/backup.go:
--------------------------------------------------------------------------------
1 | // Copyright (C) 2019 Yasuhiro Matsumoto