├── .github └── workflows │ └── go.yaml ├── .gitignore ├── Makefile ├── README.md ├── cmd └── elephanttalk │ ├── main.go │ └── test.lisp ├── go.mod ├── go.sum ├── opencv └── lisp.go └── talk ├── calibration.go ├── db.go ├── detect.go ├── geometry.go ├── lisp.go ├── matrix.go ├── page.go ├── print.go ├── talk.lisp └── vision.go /.github/workflows/go.yaml: -------------------------------------------------------------------------------- 1 | name: Go 2 | 3 | on: push 4 | 5 | jobs: 6 | 7 | build: 8 | runs-on: ubuntu-latest 9 | steps: 10 | - uses: actions/checkout@v3 11 | 12 | - name: Set up Go 13 | uses: actions/setup-go@v4 14 | with: 15 | go-version: 1.20.x 16 | 17 | # commented out until we actually have tests, and I figure out gocv workflows 18 | # - name: Test 19 | # run: go test -v -count=1 -short ./... 20 | 21 | - name: Fmt 22 | run: go fmt ./... 23 | 24 | - name: Diff 25 | run: git diff --exit-code . 26 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *.png 2 | build 3 | calibration_v0.json 4 | calibration_v1.json 5 | calibration.json 6 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | run: 2 | go run cmd/elephanttalk/main.go 3 | 4 | test: 5 | go test ./... -test.short -count=1 6 | 7 | clean: 8 | rm -rf build 9 | 10 | build: clean 11 | go build -o ./build/elephanttalk ./cmd/elephanttalk/ 12 | 13 | install: 14 | go install ./cmd/elephanttalk 15 | 16 | fmt: 17 | go fmt ./... 18 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Elephant Talk 2 | 3 | Trying to figure out how Bret Victor's [Dynamicland](https://dynamicland.org/) works by trying to recreate some of it. 4 | 5 | ## What it is and what it is not 6 | 7 | ElephantTalk is an approximation of Realtalk, a language, and RealtalkOS, an operating system, both of which make up a large part of Dynamicland. A good description of what it is like to work in such a system can be found [here](https://omar.website/posts/notes-from-dynamicland-geokit/) and in youtube videos/twitter threads here and there on the internet. The best way to experience it is to go to the physical research lab in Oakland, which I have not done. Hence I am trying to describe an elephant like so many blind men that have only touched or seen parts of it. 8 | 9 | This project notably misses most of the collaborative environment such a system is built for. ElephantTalk is built to run on one machine, with one projector and one camera. In the future, it will support live editing of pages, but right now the main way of interacting is still through a little rectangular screen in order to set it up. Multiple projectors/screens/tables and interactions between RealtalkOS instances is out of scope. 10 | 11 | ElephantTalk supports scripting pieces of paper (pages) through [Lisp](https://github.com/deosjr/whistle) (using an experimental interpreter I wrote in Golang) whereas Realtalk uses Lua. Image detection is done using [gocv](https://github.com/hybridgroup/gocv). The lisp package includes a very simple datalog instance, which powers the claim/wish/when model of Realtalk. There are no other dependencies: everything together is a single Go program that can run on a laptop. 12 | 13 | I won't claim to fully understand Realtalk or dynamicland: this project's whole purpose is to tangle with their concepts and play around with them. If you find any major differences or have been to Dynamicland, please reach out and let me know! Information is still scarce, and Oakland is far away from where I live. 14 | 15 | ## How to run 16 | The `make run` command starts the project and should open two windows. One shows camera output: this is the debug window. The other shows a mostly black screen: this is the projector window. The projector window should be moved to a second screen projected onto a wall or floor (surface). The camera should be placed such that it mostly captures the surface and is rectilinear to it. Before we can play with pages, we need to calibrate the system. For this, you will want to print out a calibration page which contains four coloured dots. 17 | 18 | ### Calibration 19 | You should now have the projector showing a red cross on the surface. Place the calibration page so that the cross is in the center of the four dots, and with the debug window in focus press any key. This will be the center of the projection space. The more this is off from the center of the camera (which you can see on the debug window), the more we need to correct for it: this is why we are calibrating. 20 | Now you should see another prompt to place the page, but off to the right of where it was previously. Place the calibration page so that again the cross is in the middle of the dots, and press any key. If everything is well and good, you should see a blue outline projected on top of the calibration page. Press any key one more time: this concludes calibration. 21 | 22 | ### Scripting 23 | 24 | From now on, each frame the program will attempt to detect pages identified by coloured dots. Each page is unique and associated with a script, which runs each frame the page is detected. A database of pages is hardcoded in `main.go`. Adding pages dynamically is next on the todo list. 25 | Currently no state is persisted between frames! Interacting with Realtalk is hard to describe in text, I suggest watching a few videos like this one https://www.youtube.com/watch?v=PvHddfHX9hc 26 | 27 | For a good first idea of what the scripting language tries to provide beyond basic lisp, see [this image](https://omar.website/posts/notes-from-dynamicland-geokit/realtalk-cheat-sheet.png) (via [Omar Rizwan](https://twitter.com/rsnous), credited to [Tabitha Yong](https://twitter.com/telogram)). 28 | -------------------------------------------------------------------------------- /cmd/elephanttalk/main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | _ "embed" 5 | 6 | "github.com/deosjr/elephanttalk/talk" 7 | ) 8 | 9 | //go:embed test.lisp 10 | var testpage string 11 | 12 | func main() { 13 | // instead of using all coloured dots to identify pages, only use the corner dots 14 | talk.UseSimplifiedIDs() 15 | 16 | //page1 17 | //talk.AddPageFromShorthand("ygybr", "brgry", "gbgyg", "bgryy", `(claim this 'outlined 'blue)`) 18 | talk.AddPageFromShorthand("ygybr", "brgry", "gbgyg", "bgryy", `(claim this 'pointing 30)`) 19 | 20 | //page2 21 | talk.AddPageFromShorthand("yggyg", "rgyrb", "bybbg", "brgrg", `(claim this 'highlighted 'red)`) 22 | 23 | //page that always counts as recognised but doesnt have to be present physically 24 | talk.AddBackgroundPage(testpage) 25 | 26 | talk.Run() 27 | } 28 | -------------------------------------------------------------------------------- /cmd/elephanttalk/test.lisp: -------------------------------------------------------------------------------- 1 | (begin 2 | #| should be counterclockwise, somehow isnt; fixed for now by negating angle |# 3 | (define rotateAround (lambda (pivot point angle) 4 | (let ((s (sin (- 0 angle))) 5 | (c (cos (- 0 angle))) 6 | (px (car point)) 7 | (py (cdr point)) 8 | (cx (car pivot)) 9 | (cy (cdr pivot))) 10 | (let ((x (- px cx)) 11 | (y (- py cy))) 12 | (cons 13 | (+ cx (- (* c x) (* s y))) 14 | (+ cy (+ (* s x) (* c y)))))))) 15 | 16 | (define point-add (lambda (p q) 17 | (cons 18 | (+ (car p) (car q)) 19 | (+ (cdr p) (cdr q))))) 20 | 21 | (define point-sub (lambda (p q) 22 | (cons 23 | (- (car p) (car q)) 24 | (- (cdr p) (cdr q))))) 25 | 26 | (define point-mul (lambda (p n) 27 | (cons (* (car p) n) (* (cdr p) n)))) 28 | 29 | (define point-div (lambda (p n) 30 | (cons (/ (car p) n) (/ (cdr p) n)))) 31 | 32 | (define midpoint (lambda (points) 33 | (point-div (foldl point-add points (cons 0 0)) (length points)))) 34 | 35 | (define points->rect (lambda (points) 36 | (let ((rects (map (lambda (p) 37 | (let ((min (point-add p (cons -1 -1))) (max (point-add p (cons 1 1)))) 38 | (make-rectangle (car min) (cdr min) (car max) (cdr max)))) points))) 39 | #| (foldl rects rect:union (car rects)) |# 40 | (rect:union (rect:union (rect:union (car rects) (car (cdr rects))) (car (cdr (cdr rects)))) (car (cdr (cdr (cdr rects))))) 41 | ))) 42 | 43 | #| TODO: illu (ie gocv.Mat) is not hashable, so cant store it in claim in db. pass by ref? |# 44 | 45 | (when ((highlighted ,?page ,?color) ((page points) ,?page ,?points) ((page angle) ,?page ,?angle)) do 46 | (let ((center (midpoint (quote ,?points))) 47 | (unangle (* -360 (/ ,?angle (* 2 pi)))) 48 | (illu (make-illumination))) 49 | (let ((rotated (map (lambda (p) (rotateAround center p ,?angle)) (quote ,?points))) 50 | (m (gocv:rotation_matrix2D (car center) (cdr center) unangle 1.0))) 51 | #| TODO: make p->center a unit vector, add in projector-space inches instead of a percentage |# 52 | (define inset (lambda (p) (point-add p (point-mul (point-sub center p) 0.2)))) 53 | (gocv:rect illu (points->rect (map inset rotated)) ,?color -1) 54 | (gocv:text illu "TEST" (point2d (car center) (cdr center)) 0.5 green 2) 55 | #| might not work because it doesnt support inplace |# 56 | (gocv:warp_affine illu illu m 1280 720) 57 | (claim ,?page 'has-illumination 'illu)))) 58 | 59 | (when ((outlined ,?page ,?color) ((page points) ,?page ,?points)) do 60 | (let ((pts (quote ,?points)) 61 | (illu (make-illumination))) 62 | (let ((ulhc (car pts)) 63 | (urhc (car (cdr pts))) 64 | (lrhc (car (cdr (cdr pts)))) 65 | (llhc (car (cdr (cdr (cdr pts)))))) 66 | (let ((ulhc (point2d (car ulhc) (cdr ulhc))) 67 | (urhc (point2d (car urhc) (cdr urhc))) 68 | (lrhc (point2d (car lrhc) (cdr lrhc))) 69 | (llhc (point2d (car llhc) (cdr llhc)))) 70 | (gocv:line illu ulhc urhc ,?color 5) 71 | (gocv:line illu urhc lrhc ,?color 5) 72 | (gocv:line illu lrhc llhc ,?color 5) 73 | (gocv:line illu llhc ulhc ,?color 5))))) 74 | 75 | (when ((pointing ,?page ,?cm) ((page points) ,?page ,?points)) do 76 | (let ((pts (quote ,?points)) 77 | (illu (make-illumination))) 78 | (let ((ulhc (car pts)) 79 | (urhc (car (cdr pts))) 80 | (lrhc (car (cdr (cdr pts))))) 81 | (let ((mid (point-div (point-add ulhc urhc) 2)) 82 | (line (point-sub lrhc urhc))) 83 | (let ((norm (point-div line (sqrt (+ (* (car line) (car line)) (* (cdr line) (cdr line))))))) 84 | (let ((end (point-add mid (point-mul norm (* pixelsPerCM ,?cm))))) 85 | (gocv:line illu (point2d (car mid) (cdr mid)) (point2d (car end) (cdr end)) green 5) 86 | #| TODO: if end is contained in another page, assert claim pointing-at page otherpage |# 87 | )))))) 88 | ) 89 | -------------------------------------------------------------------------------- /go.mod: -------------------------------------------------------------------------------- 1 | module github.com/deosjr/elephanttalk 2 | 3 | go 1.21 4 | 5 | toolchain go1.22.2 6 | 7 | require ( 8 | github.com/deosjr/whistle v0.0.0-20230606141022-90a4546b49c5 9 | gocv.io/x/gocv v0.36.1-chessboard 10 | ) 11 | 12 | require gonum.org/v1/gonum v0.15.0 // indirect 13 | 14 | // replace gocv.io/x/gocv => ../../coert/gocv 15 | replace gocv.io/x/gocv => ../gocv 16 | -------------------------------------------------------------------------------- /go.sum: -------------------------------------------------------------------------------- 1 | github.com/deosjr/whistle v0.0.0-20230603075734-1dc18394c635 h1:BvMErYC4GXUlYmVfBWGFHe1E5cp6uS+Sg5o+A7QnENg= 2 | github.com/deosjr/whistle v0.0.0-20230603075734-1dc18394c635/go.mod h1:AqJExKTUYXaa+5DDBu0J+d4ajZbabQEiafNepi/Jwvk= 3 | github.com/deosjr/whistle v0.0.0-20230604162944-34851f1db745 h1:TLicDWljBcTIBOVRUHqWiIj8+qCpeL2NGPRlfBQEkG0= 4 | github.com/deosjr/whistle v0.0.0-20230604162944-34851f1db745/go.mod h1:AqJExKTUYXaa+5DDBu0J+d4ajZbabQEiafNepi/Jwvk= 5 | github.com/deosjr/whistle v0.0.0-20230606141022-90a4546b49c5 h1:aSuo5bPaU/iE4sp08cWaua0BLHEPK26yIvHWFeQhaGs= 6 | github.com/deosjr/whistle v0.0.0-20230606141022-90a4546b49c5/go.mod h1:AqJExKTUYXaa+5DDBu0J+d4ajZbabQEiafNepi/Jwvk= 7 | github.com/pascaldekloe/goe v0.1.0/go.mod h1:lzWF7FIEvWOWxwDKqyGYQf6ZUaNfKdP144TG7ZOy1lc= 8 | gocv.io/x/gocv v0.27.0 h1:3X8I74ULsWHd4m7DQRv2Nqx5VkKscfUFnKgLNodiboI= 9 | gocv.io/x/gocv v0.27.0/go.mod h1:n4LnYjykU6y9gn48yZf4eLCdtuSb77XxSkW6g0wGf/A= 10 | -------------------------------------------------------------------------------- /opencv/lisp.go: -------------------------------------------------------------------------------- 1 | package opencv 2 | 3 | import ( 4 | "image" 5 | "image/color" 6 | "math" 7 | 8 | "github.com/deosjr/whistle/lisp" 9 | "gocv.io/x/gocv" 10 | ) 11 | 12 | var ( 13 | beamerWidth, beamerHeight = 1280, 720 14 | ) 15 | 16 | // wrapper around gocv for use in lisp code 17 | 18 | func Load(env *lisp.Env) { 19 | // colors 20 | env.Add("black", lisp.NewPrimitive(color.RGBA{})) 21 | env.Add("white", lisp.NewPrimitive(color.RGBA{255, 255, 255, 0})) 22 | env.Add("red", lisp.NewPrimitive(color.RGBA{255, 0, 0, 0})) 23 | env.Add("green", lisp.NewPrimitive(color.RGBA{0, 255, 0, 0})) 24 | env.Add("blue", lisp.NewPrimitive(color.RGBA{0, 0, 255, 0})) 25 | 26 | // illumination is a gocv mat 27 | env.AddBuiltin("make-illumination", newIllumination) 28 | // TODO: once we explore declarations in projectionspace vs rotation a bit more 29 | //env.AddBuiltin("ill:rectangle", illuRectangle) 30 | 31 | // gocv drawing, might be replaced by ill:draw funcs at some point 32 | env.AddBuiltin("gocv:line", gocvLine) 33 | env.AddBuiltin("gocv:rect", gocvRectangle) 34 | env.AddBuiltin("gocv:text", gocvText) 35 | env.AddBuiltin("gocv:rotation_matrix2D", rotationMatrix) 36 | env.AddBuiltin("gocv:warp_affine", warpAffine) 37 | 38 | // golang image lib for 2d primitives 39 | // TODO: if we reason only in projector space or even page space 40 | // some of this might become less relevant or even confusing 41 | env.AddBuiltin("point2d", newPoint2D) 42 | env.AddBuiltin("make-rectangle", newRectangle) 43 | env.AddBuiltin("rect:union", rectUnion) 44 | 45 | // missing math builtins 46 | env.AddBuiltin("sin", sine) 47 | env.AddBuiltin("cos", cosine) 48 | env.AddBuiltin("sqrt", sqrt) 49 | } 50 | 51 | // TODO: defer close? memory leak otherwise? 52 | // dont want to write close in lisp, so for now we'll need some memory mngment 53 | // outside of it (ie keeping track in Go of created mats and closing them) 54 | // NOTE: gocv.Mat is unhashable, so we cant even store it in datalog anyways 55 | var Illus = []gocv.Mat{} 56 | 57 | func newIllumination(args []lisp.SExpression) (lisp.SExpression, error) { 58 | illu := gocv.NewMatWithSize(beamerHeight, beamerWidth, gocv.MatTypeCV8UC3) 59 | Illus = append(Illus, illu) 60 | return lisp.NewPrimitive(illu), nil 61 | } 62 | 63 | // (gocv:line illu p q color fill) 64 | func gocvLine(args []lisp.SExpression) (lisp.SExpression, error) { 65 | illu := args[0].AsPrimitive().(gocv.Mat) 66 | p := args[1].AsPrimitive().(image.Point) 67 | q := args[2].AsPrimitive().(image.Point) 68 | c := args[3].AsPrimitive().(color.RGBA) 69 | fill := int(args[4].AsNumber()) 70 | gocv.Line(&illu, p, q, c, fill) 71 | return lisp.NewPrimitive(illu), nil 72 | } 73 | 74 | // (gocv:rect illu rect color fill) 75 | func gocvRectangle(args []lisp.SExpression) (lisp.SExpression, error) { 76 | illu := args[0].AsPrimitive().(gocv.Mat) 77 | rect := args[1].AsPrimitive().(image.Rectangle) 78 | c := args[2].AsPrimitive().(color.RGBA) 79 | fill := int(args[3].AsNumber()) 80 | gocv.Rectangle(&illu, rect, c, fill) 81 | return lisp.NewPrimitive(illu), nil 82 | } 83 | 84 | // TODO: cant pick a font yet 85 | // NOTE: text cant be drawn at an angle, so has to be drawn then rotated 86 | // (gocv:text illu text origin scale color fill) 87 | func gocvText(args []lisp.SExpression) (lisp.SExpression, error) { 88 | illu := args[0].AsPrimitive().(gocv.Mat) 89 | txt := args[1].AsPrimitive().(string) 90 | origin := args[2].AsPrimitive().(image.Point) 91 | scale := args[3].AsNumber() 92 | c := args[4].AsPrimitive().(color.RGBA) 93 | fill := int(args[5].AsNumber()) 94 | gocv.PutText(&illu, txt, origin, gocv.FontHersheySimplex, scale, c, fill) 95 | return lisp.NewPrimitive(illu), nil 96 | } 97 | 98 | // (gocv:rotation_matrix2D cx cy degrees scale) 99 | func rotationMatrix(args []lisp.SExpression) (lisp.SExpression, error) { 100 | x, y := int(args[0].AsNumber()), int(args[1].AsNumber()) 101 | degrees := args[2].AsNumber() 102 | scale := args[3].AsNumber() 103 | return lisp.NewPrimitive(gocv.GetRotationMatrix2D(image.Pt(x, y), degrees, scale)), nil 104 | } 105 | 106 | // (gocv:warp_affine src dst m sx sy) 107 | func warpAffine(args []lisp.SExpression) (lisp.SExpression, error) { 108 | src := args[0].AsPrimitive().(gocv.Mat) 109 | dst := args[1].AsPrimitive().(gocv.Mat) 110 | m := args[2].AsPrimitive().(gocv.Mat) 111 | x, y := int(args[3].AsNumber()), int(args[4].AsNumber()) 112 | gocv.WarpAffine(src, &dst, m, image.Pt(x, y)) 113 | return lisp.NewPrimitive(true), nil 114 | } 115 | 116 | // (point2D x y) -> image.Point primitive 117 | func newPoint2D(args []lisp.SExpression) (lisp.SExpression, error) { 118 | x, y := int(args[0].AsNumber()), int(args[1].AsNumber()) 119 | return lisp.NewPrimitive(image.Pt(x, y)), nil 120 | } 121 | 122 | // (make-rectangle minx miny maxx maxy) -> image.Rectangle primitive 123 | func newRectangle(args []lisp.SExpression) (lisp.SExpression, error) { 124 | px, py := int(args[0].AsNumber()), int(args[1].AsNumber()) 125 | qx, qy := int(args[2].AsNumber()), int(args[3].AsNumber()) 126 | r := image.Rectangle{image.Pt(px, py), image.Pt(qx, qy)} 127 | return lisp.NewPrimitive(r), nil 128 | } 129 | 130 | func rectUnion(args []lisp.SExpression) (lisp.SExpression, error) { 131 | r1 := args[0].AsPrimitive().(image.Rectangle) 132 | r2 := args[1].AsPrimitive().(image.Rectangle) 133 | return lisp.NewPrimitive(r1.Union(r2)), nil 134 | } 135 | 136 | func sine(args []lisp.SExpression) (lisp.SExpression, error) { 137 | return lisp.NewPrimitive(math.Sin(args[0].AsNumber())), nil 138 | } 139 | 140 | func cosine(args []lisp.SExpression) (lisp.SExpression, error) { 141 | return lisp.NewPrimitive(math.Cos(args[0].AsNumber())), nil 142 | } 143 | 144 | func sqrt(args []lisp.SExpression) (lisp.SExpression, error) { 145 | return lisp.NewPrimitive(math.Sqrt(args[0].AsNumber())), nil 146 | } 147 | -------------------------------------------------------------------------------- /talk/calibration.go: -------------------------------------------------------------------------------- 1 | package talk 2 | 3 | import ( 4 | "encoding/json" 5 | "fmt" 6 | "image" 7 | "image/color" 8 | "math" 9 | "os" 10 | 11 | "gocv.io/x/gocv" 12 | ) 13 | 14 | const WEBCAM_HEIGHT = 720 15 | const WEBCAM_WIDTH = 1280 16 | 17 | const STRAIGHT_W = 600 // width of the straight chessboard 18 | const STRAIGHT_H = 331 // height of the straight chessboard 19 | var colorBlack = color.RGBA{0, 0, 0, 255} 20 | var colorWhite = color.RGBA{255, 255, 255, 255} 21 | var colorRed = color.RGBA{255, 0, 0, 255} 22 | var colorGreen = color.RGBA{0, 255, 0, 255} 23 | var colorBlue = color.RGBA{0, 0, 255, 255} 24 | var colorYellow = color.RGBA{255, 255, 0, 255} 25 | var colorCyan = color.RGBA{0, 255, 255, 255} 26 | var colorMagenta = color.RGBA{255, 0, 255, 255} 27 | 28 | type calibrationResults struct { 29 | pixelsPerCM float64 30 | displacement point 31 | displayRatio float64 32 | referenceColors []color.RGBA 33 | scChsBrd straightChessboard 34 | } 35 | 36 | func calibration(webcam *gocv.VideoCapture, debugwindow, projection *gocv.Window) calibrationResults { 37 | //calibrationPage() 38 | 39 | // Step 1: set up beamer + webcam at a surface 40 | // Step 2: drag projector window to the beamer 41 | // Step 3: beamer projects midpoint -> position calibration pattern centered on midpoint 42 | // Step 4: recognise pattern and calculate pixel distances 43 | 44 | // TODO probably rename these 45 | // img: debug window output from camera 46 | // cimg: projector window 47 | scChsBrd := loadCalibration("calibration.json") 48 | 49 | img := gocv.NewMat() 50 | defer img.Close() 51 | 52 | cimg := gocv.NewMatWithSize(beamerHeight, beamerWidth, gocv.MatTypeCV8UC3) 53 | defer cimg.Close() 54 | red := color.RGBA{255, 0, 0, 0} 55 | green := color.RGBA{0, 255, 0, 0} 56 | blue := color.RGBA{0, 0, 255, 0} 57 | yellow := color.RGBA{255, 255, 0, 0} 58 | 59 | w, h := beamerWidth/2., beamerHeight/2. 60 | gocv.Line(&cimg, image.Pt(w-5, h), image.Pt(w+5, h), red, 2) 61 | gocv.Line(&cimg, image.Pt(w, h-5), image.Pt(w, h+5), red, 2) 62 | gocv.PutText(&cimg, "Place calibration pattern", image.Pt(w-100, h+50), 0, .5, color.RGBA{255, 255, 255, 0}, 2) 63 | 64 | var pattern []circle 65 | 66 | fi := frameInput{ 67 | webcam: webcam, 68 | debugWindow: debugwindow, 69 | projection: projection, 70 | img: img, 71 | cimg: cimg, 72 | scChsBrd: scChsBrd, 73 | } 74 | 75 | if err := frameloop(fi, func(_ image.Image, spatialPartition map[image.Rectangle][]circle) { 76 | // find calibration pattern, draw around it 77 | for k, v := range spatialPartition { 78 | if !findCalibrationPattern(v) { 79 | continue 80 | } 81 | sortCirclesAsCorners(v) 82 | gocv.Rectangle(&img, k, red, 2) 83 | r := image.Rectangle{ 84 | v[0].mid.add(point{-v[0].r, -v[0].r}).toIntPt(), 85 | v[3].mid.add(point{v[3].r, v[3].r}).toIntPt(), 86 | } 87 | gocv.Rectangle(&img, r, blue, 2) 88 | 89 | // TODO: draw indicators for horizontal/vertical align 90 | pattern = v 91 | } 92 | 93 | }, nil, 100); err != nil { 94 | return calibrationResults{} 95 | } 96 | 97 | // keypress breaks the loop, assume pattern is found over midpoint 98 | // draw conclusions about colors and distances 99 | webcamMid := circlesMidpoint(pattern) 100 | // average over all 4 distances to circles in pattern 101 | dpixels := euclidian(pattern[0].mid.sub(webcamMid)) 102 | dpixels += euclidian(pattern[1].mid.sub(webcamMid)) 103 | dpixels += euclidian(pattern[2].mid.sub(webcamMid)) 104 | dpixels += euclidian(pattern[3].mid.sub(webcamMid)) 105 | dpixels = dpixels / 4. 106 | dcm := math.Sqrt(1.5*1.5 + 1.5*1.5) 107 | 108 | // just like for printing 1cm = 118px, we need a new ratio for projections 109 | // NOTE: pixPerCM lives in webcamspace, NOT beamerspace 110 | pixPerCM := dpixels / dcm 111 | 112 | // beamer midpoint vs webcam midpoint displacement 113 | beamerMid := point{float64(w), float64(h)} 114 | displacement := beamerMid.sub(webcamMid) 115 | 116 | // get color samples of the four dots as reference values 117 | var colorSamples []color.RGBA 118 | for { 119 | if ok := webcam.Read(&img); !ok { 120 | fmt.Printf("cannot read device\n") 121 | return calibrationResults{} 122 | } 123 | if img.Empty() { 124 | continue 125 | } 126 | 127 | actualImage, _ := img.ToImage() 128 | if colorSamples == nil { 129 | // TODO: average within the circle? 130 | colorSamples = make([]color.RGBA, 4) 131 | for i, circle := range pattern { 132 | c := actualImage.At(int(circle.mid.x), int(circle.mid.y)) 133 | rr, gg, bb, _ := c.RGBA() 134 | colorSamples[i] = color.RGBA{uint8(rr), uint8(gg), uint8(bb), 0} 135 | } 136 | } 137 | break 138 | } 139 | 140 | // project another reference point and calculate diff between webcam-space and projector-space 141 | // ratio between webcam and beamer 142 | displayRatio := 1.0 143 | 144 | if err := frameloop(fi, func(_ image.Image, spatialPartition map[image.Rectangle][]circle) { 145 | gocv.Rectangle(&cimg, image.Rect(0, 0, beamerWidth, beamerHeight), color.RGBA{}, -1) 146 | gocv.Line(&cimg, image.Pt(w-5+200, h), image.Pt(w+5+200, h), red, 2) 147 | gocv.Line(&cimg, image.Pt(w+200., h-5), image.Pt(w+200, h+5), red, 2) 148 | gocv.PutText(&cimg, "Place calibration pattern", image.Pt(w-100+200, h+50), 0, .5, color.RGBA{255, 255, 255, 0}, 2) 149 | 150 | // find calibration pattern, draw around it 151 | for k, v := range spatialPartition { 152 | if !findCalibrationPattern(v) { 153 | continue 154 | } 155 | sortCirclesAsCorners(v) 156 | gocv.Rectangle(&img, k, red, 2) 157 | r := image.Rectangle{ 158 | v[0].mid.add(point{-v[0].r, -v[0].r}).toIntPt(), 159 | v[3].mid.add(point{v[3].r, v[3].r}).toIntPt(), 160 | } 161 | gocv.Rectangle(&img, r, blue, 2) 162 | 163 | midpoint := circlesMidpoint(v) 164 | // assume Y component stays 0 (i.e. we are horizontally aligned between webcam and beamer) 165 | displayRatio = midpoint.sub(webcamMid).x / 200.0 166 | 167 | // projecting the draw ratio difference 168 | withoutRatio := midpoint.add(displacement).toIntPt() 169 | gocv.Line(&cimg, beamerMid.toIntPt(), withoutRatio, blue, 2) 170 | 171 | // TODO: draw indicators for horizontal/vertical align 172 | pattern = v 173 | } 174 | 175 | gocv.Circle(&img, image.Pt(10, 10), 10, colorSamples[0], -1) 176 | gocv.Circle(&img, image.Pt(30, 10), 10, colorSamples[1], -1) 177 | gocv.Circle(&img, image.Pt(50, 10), 10, colorSamples[2], -1) 178 | gocv.Circle(&img, image.Pt(70, 10), 10, colorSamples[3], -1) 179 | 180 | }, colorSamples, 100); err != nil { 181 | return calibrationResults{} 182 | } 183 | 184 | if err := frameloop(fi, func(actualImage image.Image, spatialPartition map[image.Rectangle][]circle) { 185 | gocv.Rectangle(&cimg, image.Rect(0, 0, beamerWidth, beamerHeight), color.RGBA{}, -1) 186 | 187 | for k, v := range spatialPartition { 188 | if !findCalibrationPattern(v) { 189 | continue 190 | } 191 | sortCirclesAsCorners(v) 192 | 193 | colorDiff := make([]float64, 4) 194 | for i, circle := range v { 195 | c := actualImage.At(int(circle.mid.x), int(circle.mid.y)) 196 | colorDiff[i] = colorDistance(c, colorSamples[i]) 197 | } 198 | // experimentally, all diffs under 10k means we are good (paper rightway up) 199 | // unless ofc lighting changes drastically 200 | 201 | gocv.Rectangle(&img, k, red, 2) 202 | r := image.Rectangle{ 203 | v[0].mid.add(point{-v[0].r, -v[0].r}).toIntPt(), 204 | v[3].mid.add(point{v[3].r, v[3].r}).toIntPt(), 205 | } 206 | gocv.Rectangle(&img, r, blue, 2) 207 | 208 | gocv.Circle(&img, v[0].mid.toIntPt(), int(v[0].r), red, 2) 209 | gocv.Circle(&img, v[1].mid.toIntPt(), int(v[1].r), green, 2) 210 | gocv.Circle(&img, v[2].mid.toIntPt(), int(v[2].r), blue, 2) 211 | gocv.Circle(&img, v[3].mid.toIntPt(), int(v[3].r), yellow, 2) 212 | 213 | // now we project around the whole A4 containing the calibration pattern 214 | // a4 in cm: 21 x 29.7 215 | a4hpx := (29.7 * pixPerCM) / 2. 216 | a4wpx := (21.0 * pixPerCM) / 2. 217 | midpoint := circlesMidpoint(v) 218 | min := midpoint.add(point{-a4wpx, -a4hpx}) 219 | max := midpoint.add(point{a4wpx, a4hpx}) 220 | a4 := image.Rectangle{min.toIntPt(), max.toIntPt()} 221 | gocv.Rectangle(&img, a4, blue, 4) 222 | 223 | // adjust for displacement and display ratio 224 | a4 = image.Rectangle{translate(min, displacement, displayRatio).toIntPt(), translate(max, displacement, displayRatio).toIntPt()} 225 | gocv.Rectangle(&cimg, a4, blue, 4) 226 | } 227 | 228 | gocv.Circle(&img, image.Pt(10, 10), 10, colorSamples[0], -1) 229 | gocv.Circle(&img, image.Pt(30, 10), 10, colorSamples[1], -1) 230 | gocv.Circle(&img, image.Pt(50, 10), 10, colorSamples[2], -1) 231 | gocv.Circle(&img, image.Pt(70, 10), 10, colorSamples[3], -1) 232 | }, colorSamples, 100); err != nil { 233 | return calibrationResults{} 234 | } 235 | 236 | // TODO: happy (y/n) ? if no return to start of calibration 237 | return calibrationResults{pixPerCM, displacement, displayRatio, colorSamples, scChsBrd} 238 | } 239 | 240 | type straightChessboard struct { 241 | Rotation gocv.Mat 242 | Translation gocv.Mat 243 | Camera gocv.Mat 244 | Distortion gocv.Mat 245 | MapX gocv.Mat 246 | MapY gocv.Mat 247 | Roi image.Rectangle 248 | M gocv.Mat 249 | Csf float64 250 | ColorModels []gocv.Mat 251 | } 252 | 253 | func (sc *straightChessboard) UnMarshalJSON(data []byte) error { 254 | type Alias straightChessboard 255 | aux := &struct { 256 | Translation []float64 257 | Rotation []float64 258 | Camera []float64 259 | Distortion []float64 260 | MapX []float64 261 | MapY []float64 262 | Roi string 263 | M []float64 264 | Csf float64 265 | ColorModels [][]float64 266 | *Alias 267 | }{ 268 | Alias: (*Alias)(sc), 269 | } 270 | 271 | if err := json.Unmarshal(data, &aux); err != nil { 272 | return err 273 | } 274 | 275 | sc.Translation, _ = doubleSliceToMat64F(aux.Translation, 3, 1, 1) 276 | sc.Rotation, _ = doubleSliceToMat64F(aux.Rotation, 3, 1, 1) 277 | sc.Camera, _ = doubleSliceToMat64F(aux.Camera, 3, 3, 1) 278 | sc.Distortion, _ = doubleSliceToMat64F(aux.Distortion, 5, 1, 1) 279 | sc.MapX, _ = floatSliceToMat32F(aux.MapX, WEBCAM_HEIGHT, WEBCAM_WIDTH, 1) 280 | sc.MapY, _ = floatSliceToMat32F(aux.MapY, WEBCAM_HEIGHT, WEBCAM_WIDTH, 1) 281 | sc.Roi = stringToRect(aux.Roi) 282 | sc.M, _ = doubleSliceToMat64F(aux.M, 3, 3, 1) 283 | // sc.Csf = aux.Csf 284 | // sizes := []int{HIST_SIZE, HIST_SIZE, HIST_SIZE} 285 | // sc.ColorModels, _ = floatToMats32F(aux.ColorModels, sizes, NB_CLRD_CHCKRS) 286 | 287 | return nil 288 | } 289 | 290 | func loadCalibration(file string) straightChessboard { 291 | b, err := os.ReadFile(file) 292 | if err != nil { 293 | fmt.Println(err) 294 | } 295 | 296 | scChsBrd := straightChessboard{} 297 | err = scChsBrd.UnMarshalJSON(b) 298 | if err != nil { 299 | fmt.Println(err) 300 | } 301 | 302 | return scChsBrd 303 | } 304 | 305 | // Helper function to convert image.Rectangle to string 306 | func rectToString(r image.Rectangle) string { 307 | return fmt.Sprintf("%d,%d,%d,%d", r.Min.X, r.Min.Y, r.Max.X, r.Max.Y) 308 | } 309 | 310 | func stringToRect(s string) image.Rectangle { 311 | var r image.Rectangle 312 | fmt.Sscanf(s, "%d,%d,%d,%d", &r.Min.X, &r.Min.Y, &r.Max.X, &r.Max.Y) 313 | return r 314 | } 315 | 316 | func beamerToChessboard(frame gocv.Mat, sc straightChessboard) gocv.Mat { 317 | dstSize := image.Pt(STRAIGHT_W, STRAIGHT_H) 318 | 319 | canvas := gocv.NewMat() 320 | defer canvas.Close() 321 | gocv.Remap(frame, &canvas, sc.MapX, sc.MapY, gocv.InterpolationLinear, gocv.BorderConstant, colorBlack) 322 | 323 | cbRegion := canvas.Region(sc.Roi) 324 | defer cbRegion.Close() 325 | 326 | scbRegion := gocv.NewMat() 327 | gocv.WarpPerspectiveWithParams(cbRegion, &scbRegion, sc.M, dstSize, 328 | gocv.InterpolationLinear, gocv.BorderConstant, colorBlack) 329 | 330 | return scbRegion 331 | } 332 | 333 | func chessboardToBeamer(frame gocv.Mat, sc straightChessboard) gocv.Mat { 334 | scbRegion := gocv.NewMat() 335 | gocv.WarpPerspectiveWithParams(frame, &scbRegion, sc.M, image.Pt(sc.MapX.Cols(), sc.MapX.Rows()), 336 | gocv.InterpolationLinear|gocv.WarpInverseMap, gocv.BorderConstant, colorBlack) 337 | return scbRegion 338 | } 339 | -------------------------------------------------------------------------------- /talk/db.go: -------------------------------------------------------------------------------- 1 | package talk 2 | 3 | import ( 4 | "strings" 5 | ) 6 | 7 | // TODO: a proper database solution, inmem is good enough for now 8 | var pageDB = map[uint32]page{} 9 | 10 | var backgroundPages = []page{} 11 | 12 | // AddBackgroundPage adds a virtual page to the db which always counts as recognised in a frame. 13 | // This means its code will always get executed. 'this' is not supported since pageID doesnt have strong guarantees 14 | func AddBackgroundPage(code string) { 15 | backgroundPages = append(backgroundPages, page{code: code}) 16 | } 17 | 18 | // AddPageFromShorthand lets you add a page to the database when you already know its corners 19 | // used while the database doesnt persist across sessions, so we dont print new pages all the time 20 | func AddPageFromShorthand(ulhc, urhc, lrhc, llhc, code string) bool { 21 | return addToDB(page{ 22 | ulhc: cornerShorthand(ulhc), 23 | urhc: cornerShorthand(urhc), 24 | llhc: cornerShorthand(llhc), 25 | lrhc: cornerShorthand(lrhc), 26 | code: code, 27 | }) 28 | } 29 | 30 | // Each 3 consecutive corners have their own partial ID 31 | // We store all 4 of those for each page, and each has to be unique! 32 | // This allows us to find a page with only 3 corners detected 33 | func addToDB(p page) bool { 34 | id1 := pagePartialID(p.ulhc.id(), p.urhc.id(), p.lrhc.id()) 35 | id2 := pagePartialID(p.urhc.id(), p.lrhc.id(), p.llhc.id()) 36 | id3 := pagePartialID(p.lrhc.id(), p.llhc.id(), p.ulhc.id()) 37 | id4 := pagePartialID(p.llhc.id(), p.ulhc.id(), p.urhc.id()) 38 | if _, ok := pageDB[id1]; ok { 39 | return false 40 | } 41 | if _, ok := pageDB[id2]; ok { 42 | return false 43 | } 44 | if _, ok := pageDB[id3]; ok { 45 | return false 46 | } 47 | if _, ok := pageDB[id4]; ok { 48 | return false 49 | } 50 | p.id = pageID(p.ulhc.id(), p.urhc.id(), p.lrhc.id(), p.llhc.id()) 51 | pageDB[id1] = p 52 | pageDB[id2] = p 53 | pageDB[id3] = p 54 | pageDB[id4] = p 55 | return true 56 | } 57 | 58 | func cornerShorthand(debug string) corner { 59 | s := "rgby" 60 | return corner{ 61 | ll: dot{c: dotColor(strings.IndexRune(s, rune(debug[0])))}, 62 | l: dot{c: dotColor(strings.IndexRune(s, rune(debug[1])))}, 63 | m: dot{c: dotColor(strings.IndexRune(s, rune(debug[2])))}, 64 | r: dot{c: dotColor(strings.IndexRune(s, rune(debug[3])))}, 65 | rr: dot{c: dotColor(strings.IndexRune(s, rune(debug[4])))}, 66 | } 67 | } 68 | -------------------------------------------------------------------------------- /talk/detect.go: -------------------------------------------------------------------------------- 1 | package talk 2 | 3 | import ( 4 | "image" 5 | "image/color" 6 | "math" 7 | "sort" 8 | 9 | "gocv.io/x/gocv" 10 | ) 11 | 12 | func detect(img gocv.Mat, actualImage image.Image, ref []color.RGBA) map[image.Rectangle][]circle { 13 | cimg := gocv.NewMat() 14 | defer cimg.Close() 15 | 16 | gocv.GaussianBlur(img, &cimg, image.Pt(9, 9), 2.0, 2.0, gocv.BorderDefault) 17 | 18 | gocv.CvtColor(cimg, &cimg, gocv.ColorRGBToGray) 19 | 20 | circleMat := gocv.NewMat() 21 | defer circleMat.Close() 22 | 23 | gocv.HoughCirclesWithParams( 24 | cimg, 25 | &circleMat, 26 | gocv.HoughGradient, 27 | 1, // dp 28 | float64(img.Rows()/64), // minDistance between centers 29 | 75, // param1 30 | 20, // param2 31 | 1, // minRadius 32 | 50, // maxRadius 33 | ) 34 | 35 | spatialPartition := map[image.Rectangle][]circle{} 36 | // webcam is 1280x720, 16x9 times 80 37 | // TODO: more than one size, hierarchical division? 38 | //square := 80 39 | square := 130 40 | square2 := square / 2. 41 | for x := 0; x < 32; x++ { 42 | for y := 0; y < 18; y++ { 43 | ulhc := image.Pt(x*square2, y*square2) 44 | urhc := image.Pt(x*square2+square, y*square2+square) 45 | spatialPartition[image.Rectangle{ulhc, urhc}] = []circle{} 46 | } 47 | } 48 | 49 | for i := 0; i < circleMat.Cols(); i++ { 50 | v := circleMat.GetVecfAt(0, i) 51 | // if circles are found 52 | if len(v) > 2 { 53 | x := float64(v[0]) 54 | y := float64(v[1]) 55 | r := float64(v[2]) 56 | 57 | c := actualImage.At(int(x), int(y)) 58 | // if we have sampled colors, only consider circles with color 'close' to a reference 59 | // TODO: we could use gocv.InRange using NewMatFromScalar for lower/upper bounds then bitwiseOr img per color 60 | // then join back(?) the four color-filtered versions of the image and only test Hough against that? 61 | /* 62 | if ref != nil { 63 | closeEnough := false 64 | for _, refC := range ref { 65 | if colorDistance(c, refC) < 1000000 { 66 | closeEnough = true 67 | } 68 | } 69 | if !closeEnough { 70 | continue 71 | } 72 | } 73 | */ 74 | 75 | mid := image.Pt(int(x), int(y)) 76 | for rect, list := range spatialPartition { 77 | if mid.In(rect) { 78 | spatialPartition[rect] = append(list, circle{point{x, y}, r, c}) 79 | } 80 | } 81 | 82 | gocv.Circle(&img, mid, int(r), color.RGBA{0, 0, 255, 0}, 2) 83 | gocv.Circle(&img, mid, 2, color.RGBA{255, 0, 0, 0}, 3) 84 | } 85 | } 86 | return spatialPartition 87 | } 88 | 89 | // calibration pattern is four circles in a rectangle 90 | // check if they are equidistant to their midpoint 91 | func findCalibrationPattern(v []circle) bool { 92 | if len(v) != 4 { 93 | return false 94 | } 95 | midpoint := circlesMidpoint(v) 96 | 97 | dist := euclidian(midpoint.sub(v[0].mid)) 98 | for _, p := range v[1:] { 99 | ddist := euclidian(midpoint.sub(p.mid)) 100 | if !equalWithMargin(ddist, dist, 2.0) { 101 | return false 102 | } 103 | } 104 | return true 105 | } 106 | 107 | func findCorners(v []circle, ref []color.RGBA) (corner, bool) { 108 | // first detect lines 109 | lines := [][]circle{} 110 | for i, c := range v { 111 | dists := map[int][]int{} 112 | for j, o := range v { 113 | if i == j { 114 | continue 115 | } 116 | // magic number? bucketing distances is hard 117 | d := int(euclidian(c.mid.sub(o.mid)) / 10) 118 | dists[d] = append(dists[d], j) 119 | } 120 | var candidate []int 121 | for _, indices := range dists { 122 | if len(indices) == 2 { 123 | candidate = indices 124 | break 125 | } 126 | } 127 | if candidate == nil { 128 | continue 129 | } 130 | line1 := v[candidate[0]].mid.sub(c.mid) 131 | line2 := v[candidate[1]].mid.sub(c.mid) 132 | dot := line1.x*line2.x + line1.y*line2.y 133 | angle := math.Acos(dot / (euclidian(line1) * euclidian(line2))) 134 | epsilon := math.Abs(angle - math.Pi) 135 | if epsilon < 0.2 { 136 | lines = append(lines, []circle{v[candidate[0]], c, v[candidate[1]]}) 137 | } 138 | } 139 | if len(lines) != 2 { 140 | return corner{}, false 141 | } 142 | 143 | line1 := lines[0] 144 | line2 := lines[1] 145 | var top, end1, end2 circle 146 | switch { 147 | case line1[0] == line2[0]: 148 | top = line1[0] 149 | end1, end2 = line1[2], line2[2] 150 | case line1[2] == line2[2]: 151 | top = line1[2] 152 | end1, end2 = line1[0], line2[0] 153 | case line1[0] == line2[2]: 154 | top = line1[0] 155 | end1, end2 = line1[2], line2[0] 156 | case line1[2] == line2[0]: 157 | top = line1[2] 158 | end1, end2 = line1[0], line2[2] 159 | default: 160 | return corner{}, false 161 | } 162 | 163 | mid1, mid2 := line1[1], line2[1] 164 | v = []circle{end1, mid1, top, mid2, end2} 165 | 166 | // midpoint test 167 | midpoint := circlesMidpoint(v) 168 | 169 | sortedDistances := []float64{} 170 | for _, p := range v { 171 | sortedDistances = append(sortedDistances, euclidian(midpoint.sub(p.mid))) 172 | } 173 | sort.Float64s(sortedDistances) 174 | // first 3 are roughly equal, last 2 are roughly x2 175 | // middle one is the 'top' of the 'arrow' 176 | first3 := (sortedDistances[0] + sortedDistances[1] + sortedDistances[2]) / 3.0 177 | last2 := (sortedDistances[3] + sortedDistances[4]) / 2.0 178 | if !equalWithMargin(first3*2, last2, 5.0) { 179 | return corner{}, false 180 | } 181 | if !equalWithMargin(sortedDistances[0], sortedDistances[1], 3.0) { 182 | return corner{}, false 183 | } 184 | if !equalWithMargin(sortedDistances[0], sortedDistances[2], 6.0) { 185 | return corner{}, false 186 | } 187 | if !equalWithMargin(sortedDistances[1], sortedDistances[2], 6.0) { 188 | return corner{}, false 189 | } 190 | if !equalWithMargin(sortedDistances[3], sortedDistances[4], 6.0) { 191 | return corner{}, false 192 | } 193 | 194 | // Rotate both ends around top by a quarter. One ends on top of the other: this is _left_ 195 | rot1 := rotateAround(top.mid, end1.mid, math.Pi/2.) 196 | rot2 := rotateAround(top.mid, end2.mid, math.Pi/2.) 197 | 198 | var left, leftmid, right, rightmid circle 199 | 200 | if euclidian(rot1.sub(end2.mid)) < 10 { 201 | left = end1 202 | leftmid = mid1 203 | rightmid = mid2 204 | right = end2 205 | } else if euclidian(rot2.sub(end1.mid)) < 10 { 206 | left = end2 207 | leftmid = mid2 208 | rightmid = mid1 209 | right = end1 210 | } else { 211 | return corner{}, false 212 | } 213 | 214 | v = []circle{left, leftmid, top, rightmid, right} 215 | 216 | colors := make([]dotColor, 5) 217 | for i, c := range v { 218 | sample := c.c 219 | dist := math.MaxFloat64 220 | for j, refC := range ref { 221 | if d := colorDistance(sample, refC); d < dist { 222 | dist = d 223 | colors[i] = dotColor(j) 224 | } 225 | } 226 | rr, gg, bb, _ := sample.RGBA() 227 | rr = rr >> 8 228 | gg = gg >> 8 229 | bb = bb >> 8 230 | switch { 231 | case rr < 80 && gg < 80 && bb < 80: 232 | colors[i] = blueDot 233 | case gg > rr && gg > bb: 234 | colors[i] = greenDot 235 | case rr > 2*gg && gg > bb+20: 236 | colors[i] = yellowDot 237 | case rr > 2*gg && rr > 3*bb: 238 | colors[i] = redDot 239 | } 240 | } 241 | return corner{ 242 | ll: dot{p: left.mid, c: colors[0]}, 243 | l: dot{p: leftmid.mid, c: colors[1]}, 244 | m: dot{p: top.mid, c: colors[2]}, 245 | r: dot{p: rightmid.mid, c: colors[3]}, 246 | rr: dot{p: right.mid, c: colors[4]}, 247 | }, true 248 | } 249 | 250 | func equalWithMargin(x, y, margin float64) bool { 251 | return !(x-margin > y || x+margin < y) 252 | } 253 | -------------------------------------------------------------------------------- /talk/geometry.go: -------------------------------------------------------------------------------- 1 | package talk 2 | 3 | import ( 4 | "image" 5 | "image/color" 6 | "math" 7 | "sort" 8 | ) 9 | 10 | type point struct { 11 | x, y float64 12 | } 13 | 14 | func (p point) add(q point) point { 15 | return point{p.x + q.x, p.y + q.y} 16 | } 17 | 18 | func (p point) sub(q point) point { 19 | return point{p.x - q.x, p.y - q.y} 20 | } 21 | 22 | func (p point) div(n float64) point { 23 | return point{p.x / n, p.y / n} 24 | } 25 | 26 | func (p point) toIntPt() image.Point { 27 | return image.Pt(int(p.x), int(p.y)) 28 | } 29 | 30 | type circle struct { 31 | mid point 32 | r float64 33 | c color.Color 34 | } 35 | 36 | func euclidian(p point) float64 { 37 | return math.Sqrt(p.x*p.x + p.y*p.y) 38 | } 39 | 40 | func translate(p, delta point, ratio float64) point { 41 | // first we add the difference from webcam to beamer midpoints 42 | q := p.add(delta) 43 | // then we boost from midpoint by missing ratio 44 | beamerMid := point{float64(beamerWidth) / 2., float64(beamerHeight) / 2.} 45 | deltaV := q.sub(beamerMid) 46 | factor := 0. 47 | if ratio != 0 { 48 | factor = (1. / ratio) - 1. 49 | } 50 | adjust := point{deltaV.x * factor, deltaV.y * factor} 51 | return q.add(adjust) 52 | } 53 | 54 | // counterclockwise rotation 55 | // TODO: ???? expected counterclockwise but getting clockwise ???? 56 | // 'fixed' by flipping sign on angle in sin/cos, shouldnt be there 57 | func rotateAround(pivot, p point, radians float64) point { 58 | s := math.Sin(-radians) 59 | c := math.Cos(-radians) 60 | 61 | x := p.x - pivot.x 62 | y := p.y - pivot.y 63 | 64 | xNew := (c*x - s*y) + pivot.x 65 | yNew := (s*x + c*y) + pivot.y 66 | return point{xNew, yNew} 67 | } 68 | 69 | func angleBetween(u, v point) float64 { 70 | dot := u.x*v.x + u.y*v.y 71 | return math.Acos(dot / (euclidian(u) * euclidian(v))) 72 | } 73 | 74 | func sortCirclesAsCorners(circles []circle) { 75 | // ulhc, urhc, llhc, lrhc 76 | sort.Slice(circles, func(i, j int) bool { 77 | return circles[i].mid.x+circles[i].mid.y < circles[j].mid.x+circles[j].mid.y 78 | }) 79 | // since origin is upperleft, ulhc is first and lrhc is last 80 | // urhc and llhc is unordered yet; urhc is assumed to be higher up 81 | if circles[1].mid.y > circles[2].mid.y { 82 | circles[1], circles[2] = circles[2], circles[1] 83 | } 84 | } 85 | 86 | func circlesMidpoint(circles []circle) point { 87 | mid := circles[0].mid 88 | for _, c := range circles[1:] { 89 | mid = mid.add(c.mid) 90 | } 91 | return mid.div(float64(len(circles))) 92 | } 93 | 94 | func ptsToRect(pts []point) image.Rectangle { 95 | r := image.Rectangle{ 96 | pts[0].add(point{-1, -1}).toIntPt(), 97 | pts[0].add(point{1, 1}).toIntPt(), 98 | } 99 | for _, p := range pts { 100 | r = r.Union(image.Rectangle{ 101 | p.add(point{-1, -1}).toIntPt(), 102 | p.add(point{1, 1}).toIntPt(), 103 | }) 104 | } 105 | return r 106 | } 107 | 108 | // calculate diff with reference color naively as a euclidian distance in color space 109 | func colorDistance(sample, reference color.Color) float64 { 110 | rr, gg, bb, _ := sample.RGBA() 111 | refR, refG, refB, _ := reference.RGBA() 112 | dR := float64(rr>>8) - float64(refR>>8) 113 | dG := float64(gg>>8) - float64(refG>>8) 114 | dB := float64(bb>>8) - float64(refB>>8) 115 | return dR*dR + dG*dG + dB*dB 116 | } 117 | -------------------------------------------------------------------------------- /talk/lisp.go: -------------------------------------------------------------------------------- 1 | package talk 2 | 3 | import ( 4 | _ "embed" 5 | "fmt" 6 | 7 | "github.com/deosjr/elephanttalk/opencv" 8 | "github.com/deosjr/whistle/datalog" 9 | "github.com/deosjr/whistle/kanren" 10 | "github.com/deosjr/whistle/lisp" 11 | ) 12 | 13 | //go:embed talk.lisp 14 | var elephanttalk string 15 | 16 | func LoadRealTalk() lisp.Lisp { 17 | l := lisp.New() 18 | kanren.Load(l) 19 | datalog.Load(l) 20 | if err := l.Load(elephanttalk); err != nil { 21 | panic(err) 22 | } 23 | opencv.Load(l.Env) 24 | return l 25 | } 26 | 27 | // clear datalog db global vars at start of each frame 28 | func clear(l lisp.Lisp) { 29 | l.Eval("(set! dl_edb (make-hashmap))") 30 | l.Eval("(set! dl_idb (make-hashmap))") 31 | l.Eval("(set! dl_rdb (quote ()))") 32 | l.Eval("(set! dl_idx_entity (make-hashmap))") 33 | l.Eval("(set! dl_idx_attr (make-hashmap))") 34 | l.Eval("(set! dl_counter 0)") 35 | } 36 | 37 | // write a recognised page to lisp, storing it in datalog 38 | // returns an int identifier for this page, which is unique in this frame only 39 | func page2lisp(l lisp.Lisp, p page, pts []point) int { 40 | lisppoints := fmt.Sprintf("(list (cons %f %f) (cons %f %f) (cons %f %f) (cons %f %f))", pts[0].x, pts[0].y, pts[1].x, pts[1].y, pts[2].x, pts[2].y, pts[3].x, pts[3].y) 41 | dID, _ := l.Eval(fmt.Sprintf(`(dl_record 'page 42 | ('id %d) 43 | ('points %s) 44 | ('angle %f) 45 | ('code %q) 46 | )`, p.id, lisppoints, p.angle, p.code)) 47 | return int(dID.AsNumber()) 48 | } 49 | 50 | func evalPages(l lisp.Lisp, pages map[uint64]page, datalogIDs map[uint64]int) { 51 | for _, page := range backgroundPages { 52 | _, err := l.Eval(page.code) 53 | if err != nil { 54 | fmt.Println(page.id, err) 55 | } 56 | } 57 | 58 | for _, page := range pages { 59 | // v1 of claim/wish/when 60 | // run each pages' code, including claims, wishes and whens 61 | // set 'this to the page's id 62 | _, err := l.Eval(fmt.Sprintf("(define this %d)", datalogIDs[page.id])) 63 | if err != nil { 64 | fmt.Println(err) 65 | } 66 | _, err = l.Eval(page.code) 67 | if err != nil { 68 | fmt.Println(page.id, err) 69 | } 70 | } 71 | 72 | _, err := l.Eval("(dl_fixpoint)") 73 | if err != nil { 74 | fmt.Println("fixpoint", err) 75 | } 76 | } 77 | -------------------------------------------------------------------------------- /talk/matrix.go: -------------------------------------------------------------------------------- 1 | package talk 2 | 3 | import ( 4 | "encoding/binary" 5 | "fmt" 6 | "unsafe" 7 | 8 | "gocv.io/x/gocv" 9 | ) 10 | 11 | const EPSILON = 0.0000001 12 | 13 | func matToDoubleSlice64F(mat gocv.Mat) []float64 { 14 | if !(mat.Type() == gocv.MatTypeCV64F || mat.Type() == gocv.MatTypeCV64FC3) { 15 | fmt.Println("matToDoubleSlice64F mat type", mat.Type()) 16 | return nil 17 | } 18 | data, _ := mat.DataPtrFloat64() 19 | return data 20 | } 21 | func matsToDoubleSlice64F(mats []gocv.Mat) [][]float64 { 22 | var data [][]float64 23 | for _, mat := range mats { 24 | data = append(data, matToDoubleSlice64F(mat)) 25 | } 26 | return data 27 | } 28 | 29 | func matToFloatSlice32F(mat gocv.Mat) []float32 { 30 | if !(mat.Type() == gocv.MatTypeCV32F || mat.Type() == gocv.MatTypeCV32FC3) { 31 | fmt.Println("matToFloatSlice32F mat type", mat.Type()) 32 | return nil 33 | } 34 | data, _ := mat.DataPtrFloat32() 35 | return data 36 | } 37 | 38 | func matsToFloatSlice32F(mats []gocv.Mat) [][]float32 { 39 | var data [][]float32 40 | for _, mat := range mats { 41 | data = append(data, matToFloatSlice32F(mat)) 42 | } 43 | return data 44 | } 45 | 46 | func matToFloatSlice16S(mat gocv.Mat) []int32 { 47 | var data []int32 48 | rows := mat.Rows() 49 | cols := mat.Cols() 50 | for c := 0; c < cols; c++ { 51 | for r := 0; r < rows; r++ { 52 | value := mat.GetIntAt(r, c) 53 | data = append(data, value) 54 | } 55 | } 56 | return data 57 | } 58 | 59 | // Print3DMatValues prints all the values of a 3D gocv.Mat in a structured tabular format 60 | func PrintMatValues64F(mat gocv.Mat) { 61 | // Assume mat is a 3D matrix where each point is a single float 62 | rows := mat.Rows() 63 | cols := mat.Cols() 64 | 65 | fmt.Println("Rows", rows, "Cols", cols) 66 | fmt.Println("Channels", mat.Channels(), "Type", mat.Type()) 67 | 68 | for c := 0; c < cols; c++ { 69 | for r := 0; r < rows; r++ { 70 | val := mat.GetDoubleAt(r, c) 71 | fmt.Printf("%9.8f ", val) 72 | } 73 | fmt.Println() // New line for each row 74 | } 75 | } 76 | 77 | func PrintMatValues64FC(mat gocv.Mat) { 78 | // Assume mat is a 3D matrix where each point is a single float 79 | rows := mat.Rows() 80 | cols := mat.Cols() 81 | chns := mat.Channels() 82 | 83 | fmt.Println("Rows", rows, "Cols", cols) 84 | fmt.Println("Channels", chns, "Type", mat.Type()) 85 | 86 | mat_data, _ := mat.DataPtrFloat64() 87 | for c := 0; c < cols; c++ { 88 | for r := 0; r < rows; r++ { 89 | fmt.Printf("[") 90 | for ch := 0; ch < chns; ch++ { 91 | val := mat_data[mat.Cols()*r*3+c*3+ch] 92 | fmt.Printf("%9.8f ", val) 93 | } 94 | fmt.Println("]") 95 | } 96 | fmt.Println() // New line for each row 97 | } 98 | } 99 | 100 | func PrintMatValues32F(mat gocv.Mat) { 101 | // Assume mat is a 3D matrix where each point is a single float 102 | rows := mat.Rows() 103 | cols := mat.Cols() 104 | 105 | for c := 0; c < cols; c++ { 106 | for r := 0; r < rows; r++ { 107 | val := mat.GetFloatAt(r, c) 108 | fmt.Printf("%9.8f ", val) 109 | } 110 | fmt.Println() // New line for each row 111 | } 112 | } 113 | 114 | func PrintMatValues32FC(mat gocv.Mat) { 115 | // Assume mat is a 3D matrix where each point is a single float 116 | rows := mat.Size()[0] 117 | cols := mat.Size()[1] 118 | depth := mat.Size()[2] // Assuming the third dimension size is retrievable like this 119 | 120 | for d := 0; d < depth; d++ { 121 | fmt.Printf("Slice %d:\n", d) 122 | for c := 0; c < cols; c++ { 123 | for r := 0; r < rows; r++ { 124 | val := mat.GetFloatAt3(d, c, r) 125 | fmt.Printf("%9.8f ", val) 126 | } 127 | fmt.Println() // New line for each row 128 | } 129 | fmt.Println() // Extra new line after each depth slice 130 | } 131 | } 132 | 133 | func matToFloatSlice32FC(mat gocv.Mat) []float32 { 134 | if mat.Type() != gocv.MatTypeCV32F { 135 | fmt.Println("mat type", mat.Type()) 136 | return nil 137 | } 138 | data, _ := mat.DataPtrFloat32() 139 | return data 140 | } 141 | 142 | func matsToFloatSlice32FC(mats []gocv.Mat) [][]float32 { 143 | var data [][]float32 144 | for _, mat := range mats { 145 | data = append(data, matToFloatSlice32FC(mat)) 146 | } 147 | return data 148 | } 149 | 150 | func doubleSliceToMat64F(data []float64, rows, cols, channels int) (gocv.Mat, error) { 151 | data_bytes := make([]byte, len(data)*8) 152 | for i := 0; i < len(data); i++ { 153 | bits := *(*uint64)(unsafe.Pointer(&data[i])) 154 | binary.LittleEndian.PutUint64(data_bytes[i*8:i*8+8], bits) 155 | } 156 | if channels == 1 { 157 | return gocv.NewMatFromBytes(rows, cols, gocv.MatTypeCV64F, data_bytes) 158 | } else if channels == 3 { 159 | return gocv.NewMatFromBytes(rows, cols, gocv.MatTypeCV64FC3, data_bytes) 160 | } 161 | return gocv.NewMat(), fmt.Errorf("invalid number of channels") 162 | } 163 | 164 | func doubleSliceToNDMat64F(data []float64, sizes []int) (gocv.Mat, error) { 165 | data_bytes := make([]byte, len(data)*8) 166 | for i := 0; i < len(data); i++ { 167 | bits := *(*uint64)(unsafe.Pointer(&data[i])) 168 | binary.LittleEndian.PutUint64(data_bytes[i*8:i*8+8], bits) 169 | } 170 | return gocv.NewMatWithSizesFromBytes(sizes, gocv.MatTypeCV64F, data_bytes) 171 | 172 | } 173 | 174 | func floatSliceToMat32F(data []float64, rows, cols, channels int) (gocv.Mat, error) { 175 | mat, err := doubleSliceToMat64F(data, rows, cols, channels) 176 | if err != nil { 177 | fmt.Println("Error creating mat") 178 | return gocv.NewMat(), err 179 | } 180 | mat.ConvertTo(&mat, gocv.MatTypeCV32F) 181 | return mat, err 182 | } 183 | 184 | func floatToMats32F(data [][]float64, sizes []int, nbMats int) ([]gocv.Mat, error) { 185 | mats := make([]gocv.Mat, nbMats) 186 | for i := 0; i < nbMats; i++ { 187 | mat, err := doubleSliceToNDMat64F(data[i], sizes) 188 | if err == nil { 189 | mats[i] = gocv.NewMat() 190 | mat.ConvertTo(&mats[i], gocv.MatTypeCV32F) 191 | } else { 192 | fmt.Println("Error creating mat", i) 193 | errMsg := fmt.Errorf("error creating mat %d", i) 194 | return nil, errMsg 195 | } 196 | } 197 | return mats, nil 198 | } 199 | 200 | func PrintMatValues32I(mat gocv.Mat) { 201 | // Assume mat is a 3D matrix where each point is a single float 202 | rows := mat.Rows() 203 | cols := mat.Cols() 204 | 205 | for r := 0; r < rows; r++ { 206 | for c := 0; c < cols; c++ { 207 | val := mat.GetIntAt(r, c) 208 | fmt.Printf("%d ", val) 209 | } 210 | fmt.Println() // New line for each row 211 | } 212 | } 213 | func PrintMatValues8UC3(mat gocv.Mat) { 214 | // Assume mat is a 3D matrix where each point is a single float 215 | rows := mat.Size()[0] 216 | cols := mat.Size()[1] 217 | chns := mat.Channels() 218 | 219 | fmt.Println("Rows", rows, "Cols", cols) 220 | fmt.Println("Channels", chns, "Type", mat.Type()) 221 | 222 | mat_data, _ := mat.DataPtrUint8() 223 | for r := 0; r < rows; r++ { 224 | for c := 0; c < cols; c++ { 225 | fmt.Printf("[") 226 | for ch := 0; ch < chns; ch++ { 227 | val := mat_data[mat.Cols()*r*3+c*3+ch] 228 | fmt.Printf("%d ", val) 229 | } 230 | fmt.Println("]") 231 | } 232 | fmt.Println() 233 | } 234 | fmt.Println() // Extra new line after each depth slice 235 | } 236 | 237 | func PrintMatValues8U(mat gocv.Mat) { 238 | // Assume mat is a 3D matrix where each point is a single float 239 | rows := mat.Rows() 240 | cols := mat.Cols() 241 | chns := mat.Channels() 242 | 243 | fmt.Println("Rows", rows, "Cols", cols) 244 | fmt.Println("Channels", chns, "Type", mat.Type()) 245 | 246 | for r := 0; r < rows; r++ { 247 | for c := 0; c < cols; c++ { 248 | val := mat.GetUCharAt(r, c) 249 | fmt.Printf("%d ", val) 250 | } 251 | fmt.Println() // New line for each row 252 | } 253 | } 254 | 255 | func sumMat(mat gocv.Mat) (float64, error) { 256 | if mat.Type() != gocv.MatTypeCV32F { 257 | return 0, fmt.Errorf("mat is not of type CV_32F") 258 | } 259 | 260 | data := mat.ToBytes() 261 | var sum float64 262 | for i := 0; i < len(data); i += 4 { 263 | bits := binary.LittleEndian.Uint32(data[i : i+4]) 264 | val := *(*float32)(unsafe.Pointer(&bits)) 265 | sum += float64(val) 266 | } 267 | return sum, nil 268 | } 269 | 270 | func ensureNonZero(mat gocv.Mat) (gocv.Mat, error) { 271 | if mat.Type() != gocv.MatTypeCV32F { 272 | return gocv.Mat{}, fmt.Errorf("mat is not of type CV_32F") 273 | } 274 | 275 | data := mat.ToBytes() 276 | outData := make([]byte, len(data)) 277 | for i := 0; i < len(data); i += 4 { 278 | bits := binary.LittleEndian.Uint32(data[i : i+4]) 279 | val := *(*float32)(unsafe.Pointer(&bits)) 280 | if val == 0 { 281 | val = EPSILON 282 | } 283 | binary.LittleEndian.PutUint32(outData[i:i+4], *(*uint32)(unsafe.Pointer(&val))) 284 | } 285 | 286 | // Create a new Mat with the same dimensions as the input but with the modified data 287 | result, err := gocv.NewMatWithSizesFromBytes(mat.Size(), gocv.MatTypeCV32F, outData) 288 | if err != nil { 289 | return gocv.Mat{}, err 290 | } 291 | 292 | return result, nil 293 | } 294 | -------------------------------------------------------------------------------- /talk/page.go: -------------------------------------------------------------------------------- 1 | package talk 2 | 3 | import ( 4 | "image/color" 5 | ) 6 | 7 | // using CIELAB color picker and comparing with reference material from dynamicland 8 | // red6, green7, purple8, orange6 9 | var cielabRed = color.RGBA{245, 34, 45, 0} 10 | var cielabGreen = color.RGBA{56, 158, 13, 0} 11 | var cielabBlue = color.RGBA{57, 16, 133, 0} 12 | var cielabYellow = color.RGBA{250, 140, 22, 0} 13 | 14 | // If true, leave out the furthest corner dots 15 | // This _drastically_ reduces amount of unique possible pages, but gives stability in detection 16 | var simpleIDs bool 17 | 18 | func UseSimplifiedIDs() { 19 | simpleIDs = true 20 | } 21 | 22 | type page struct { 23 | id uint64 24 | ulhc, urhc, lrhc, llhc corner 25 | angle float64 26 | code string 27 | } 28 | 29 | // to define left and right under rotation: 30 | // left arm of the corner can make a 90 degree counterclockwise rotation 31 | // and end up on top of the right arm, 'closing' the corner 32 | type corner struct { 33 | ll, l, m, r, rr dot 34 | } 35 | 36 | func (c corner) debugPrint() string { 37 | s := []string{"r", "g", "b", "y"} 38 | return s[c.ll.c] + s[c.l.c] + s[c.m.c] + s[c.r.c] + s[c.rr.c] 39 | } 40 | 41 | // each dotColor stores 2 bits of info 42 | // one corner therefore has 10 bits of information 43 | func (c corner) id() uint16 { 44 | var out uint16 45 | if simpleIDs { 46 | out |= uint16(c.r.c) 47 | out |= uint16(c.m.c) << 2 48 | out |= uint16(c.l.c) << 4 49 | return out 50 | } 51 | out |= uint16(c.rr.c) 52 | out |= uint16(c.r.c) << 2 53 | out |= uint16(c.m.c) << 4 54 | out |= uint16(c.l.c) << 6 55 | out |= uint16(c.ll.c) << 8 56 | return out 57 | } 58 | 59 | // one page has 4 corners, therefore a 40 bit unique id in theory 60 | // however, we want to still recognise a paper when one corner is covered 61 | // practically this means each paper has 4 unique 30 bit ids (related by a 10-bit shift) 62 | // this takes 2 bits out of the space of unique 30 bit pageIDs, so 2**28 remain 63 | func pageID(ulhc, urhc, lrhc, llhc uint16) uint64 { 64 | var out uint64 65 | if simpleIDs { 66 | out |= uint64(llhc) 67 | out |= uint64(lrhc) << 6 68 | out |= uint64(urhc) << 12 69 | out |= uint64(ulhc) << 18 70 | return out 71 | } 72 | out |= uint64(llhc) 73 | out |= uint64(lrhc) << 10 74 | out |= uint64(urhc) << 20 75 | out |= uint64(ulhc) << 30 76 | return out 77 | } 78 | 79 | func pagePartialID(x, y, z uint16) uint32 { 80 | var out uint32 81 | if simpleIDs { 82 | out |= uint32(x) 83 | out |= uint32(y) << 6 84 | out |= uint32(z) << 12 85 | return out 86 | } 87 | out |= uint32(z) 88 | out |= uint32(y) << 10 89 | out |= uint32(x) << 20 90 | return out 91 | } 92 | 93 | type dot struct { 94 | p point 95 | c dotColor 96 | } 97 | 98 | type dotColor uint8 99 | 100 | const ( 101 | redDot dotColor = iota 102 | greenDot 103 | blueDot 104 | yellowDot 105 | ) 106 | -------------------------------------------------------------------------------- /talk/print.go: -------------------------------------------------------------------------------- 1 | package talk 2 | 3 | import ( 4 | "image" 5 | "image/color" 6 | 7 | "gocv.io/x/gocv" 8 | ) 9 | 10 | // TODO: call from examples folder? 11 | func PrintCalibrationPage() { 12 | w, h := 2480, 3508 // 300 ppi/dpi 13 | // a4 in cm: 21 x 29.7 14 | // which means 1cm in pixels = 2480/21 =~ 118 15 | img := gocv.NewMatWithSize(h, w, gocv.MatTypeCV8UC3) 16 | defer img.Close() 17 | 18 | red := cielabRed 19 | green := cielabGreen 20 | blue := cielabBlue 21 | yellow := cielabYellow 22 | white := color.RGBA{255, 255, 255, 0} 23 | 24 | midw, midh := w/2., h/2. 25 | d := int(1.5 * 118) // circle radius = 1, circle distance = 1 26 | gocv.Rectangle(&img, image.Rect(0, 0, w, h), white, -1) 27 | gocv.Circle(&img, image.Pt(midw-d, midh-d), 1*118, red, -1) 28 | gocv.Circle(&img, image.Pt(midw+d, midh-d), 1*118, green, -1) 29 | gocv.Circle(&img, image.Pt(midw-d, midh+d), 1*118, blue, -1) 30 | gocv.Circle(&img, image.Pt(midw+d, midh+d), 1*118, yellow, -1) 31 | 32 | gocv.IMWrite("out.png", img) 33 | } 34 | 35 | func PrintPageFromShorthand(ulhc, urhc, lrhc, llhc, code string) { 36 | PrintPage(page{ 37 | ulhc: cornerShorthand(ulhc), 38 | urhc: cornerShorthand(urhc), 39 | llhc: cornerShorthand(llhc), 40 | lrhc: cornerShorthand(lrhc), 41 | code: code, 42 | }) 43 | } 44 | 45 | func PrintPage(p page) { 46 | w, h := 2480, 3508 // 300 ppi/dpi 47 | // a4 in cm: 21 x 29.7 48 | // which means 1cm in pixels = 2480/21 =~ 118 49 | img := gocv.NewMatWithSize(h, w, gocv.MatTypeCV8UC3) 50 | defer img.Close() 51 | 52 | red := cielabRed 53 | green := cielabGreen 54 | blue := cielabBlue 55 | yellow := cielabYellow 56 | white := color.RGBA{255, 255, 255, 0} 57 | 58 | colors := []color.RGBA{red, green, blue, yellow} 59 | 60 | r := 118 61 | d := r / 2 62 | 63 | gocv.Rectangle(&img, image.Rect(0, 0, w, h), white, -1) 64 | 65 | gocv.Circle(&img, image.Pt(d+r, 3*d+5*r), r, colors[int(p.ulhc.ll.c)], -1) 66 | gocv.Circle(&img, image.Pt(d+r, 2*d+3*r), r, colors[int(p.ulhc.l.c)], -1) 67 | gocv.Circle(&img, image.Pt(d+r, d+r), r, colors[int(p.ulhc.m.c)], -1) 68 | gocv.Circle(&img, image.Pt(2*d+3*r, d+r), r, colors[int(p.ulhc.r.c)], -1) 69 | gocv.Circle(&img, image.Pt(3*d+5*r, d+r), r, colors[int(p.ulhc.rr.c)], -1) 70 | 71 | gocv.Circle(&img, image.Pt(w-(3*d+5*r), d+r), r, colors[int(p.urhc.ll.c)], -1) 72 | gocv.Circle(&img, image.Pt(w-(2*d+3*r), d+r), r, colors[int(p.urhc.l.c)], -1) 73 | gocv.Circle(&img, image.Pt(w-(d+r), d+r), r, colors[int(p.urhc.m.c)], -1) 74 | gocv.Circle(&img, image.Pt(w-(d+r), 2*d+3*r), r, colors[int(p.urhc.r.c)], -1) 75 | gocv.Circle(&img, image.Pt(w-(d+r), 3*d+5*r), r, colors[int(p.urhc.rr.c)], -1) 76 | 77 | gocv.Circle(&img, image.Pt(w-(d+r), h-(3*d+5*r)), r, colors[int(p.lrhc.ll.c)], -1) 78 | gocv.Circle(&img, image.Pt(w-(d+r), h-(2*d+3*r)), r, colors[int(p.lrhc.l.c)], -1) 79 | gocv.Circle(&img, image.Pt(w-(d+r), h-(d+r)), r, colors[int(p.lrhc.m.c)], -1) 80 | gocv.Circle(&img, image.Pt(w-(2*d+3*r), h-(d+r)), r, colors[int(p.lrhc.r.c)], -1) 81 | gocv.Circle(&img, image.Pt(w-(3*d+5*r), h-(d+r)), r, colors[int(p.lrhc.rr.c)], -1) 82 | 83 | gocv.Circle(&img, image.Pt(3*d+5*r, h-(d+r)), r, colors[int(p.llhc.ll.c)], -1) 84 | gocv.Circle(&img, image.Pt(2*d+3*r, h-(d+r)), r, colors[int(p.llhc.l.c)], -1) 85 | gocv.Circle(&img, image.Pt(d+r, h-(d+r)), r, colors[int(p.llhc.m.c)], -1) 86 | gocv.Circle(&img, image.Pt(d+r, h-(2*d+3*r)), r, colors[int(p.llhc.r.c)], -1) 87 | gocv.Circle(&img, image.Pt(d+r, h-(3*d+5*r)), r, colors[int(p.llhc.rr.c)], -1) 88 | 89 | gocv.IMWrite("out.png", img) 90 | } 91 | -------------------------------------------------------------------------------- /talk/talk.lisp: -------------------------------------------------------------------------------- 1 | #| v2 version of claim/wish/when model 2 | samples (claims/wishes) are no longer the same: a claim is a hard assert into db, 3 | but a wish is a special kind of assertion to be used in 'when' 4 | example: when /someone/ wishes x: 5 | after running fixpoint analysis once per frame, some claims are still picked up 6 | outside of db and executed upon, mostly illumination-related (blit) 7 | TODO: insertion of var 'this' does not work properly? execution context is not correct 8 | solution: insert (define this ?id) at start of each codeblock? |# 9 | (define-syntax claim 10 | (syntax-rules (dl_assert this claims list) 11 | ((_ id attr value) (begin 12 | (dl_assert this 'claims (list id attr value)) 13 | (dl_assert id attr value))))) 14 | 15 | (define-syntax wish 16 | (syntax-rules (dl_assert this wishes) 17 | ((_ x) (dl_assert this 'wishes (quote x))))) 18 | 19 | #| 'when' makes a rule and includes code execution 20 | this code execution is handled by hacking into the datalog implementation (see below) 21 | code can include further claims/wishes or even other when-statements 22 | NOTE: for now, this is going to be executing on every fixpoint iteration that matches, 23 | so it better be idempotent / not too inefficient! 24 | if conditions match, assert a fact (?id 'code ?code) where ?code already has vars replaced 25 | (when (is-a (unquote ?page) window) do (wish ((unquote ?page) highlighted blue))) |# 26 | (define-syntax when 27 | (syntax-rules (wishes do code this dl_rule :- begin) 28 | ((_ (condition ...) do statement ...) 29 | (dl_rule (code this (begin statement ...)) :- condition ...)) 30 | ((_ someone wishes w do statement ...) 31 | (dl_rule (code this (begin statement ...)) :- (wishes someone w))))) 32 | 33 | #| overwrite part of datalog naive fixpoint implementation 34 | to include code execution in when-blocks! 35 | NOTE: assumes all rules are ((code id (stmt ...)) :- condition ...) 36 | runs each newly found code to run using map eval 37 | NOTE: order is _not_ guaranteed but once code includes bindings, so same rule should only run once per set of bindings 38 | (due to key equivalence being checked on the FULL sexpression code) 39 | TODO: do we even need to update indices? |# 40 | (define dl_fixpoint_iterate (lambda () 41 | (let ((new (hashmap-keys (set_difference (foldl (lambda (x y) (set-extend! y x)) (map dl_apply_rule dl_rdb) (make-hashmap)) dl_idb)))) 42 | (set-extend! dl_idb new) 43 | (map dl_update_indices new) 44 | (map (lambda (c) (eval (car (cdr (cdr c))))) new) 45 | (if (not (null? new)) (dl_fixpoint_iterate))))) 46 | -------------------------------------------------------------------------------- /talk/vision.go: -------------------------------------------------------------------------------- 1 | package talk 2 | 3 | import ( 4 | "fmt" 5 | "image" 6 | "image/color" 7 | "math" 8 | "time" 9 | 10 | "github.com/deosjr/elephanttalk/opencv" 11 | "gocv.io/x/gocv" 12 | ) 13 | 14 | var ( 15 | // detected from webcam output instead! 16 | //webcamWidth, webcamHeight = 1280, 720 17 | beamerWidth, beamerHeight = 1280, 720 18 | ) 19 | 20 | func Run() { 21 | webcam, err := gocv.VideoCaptureDevice(0) 22 | if err != nil { 23 | panic(err) 24 | } 25 | defer webcam.Close() 26 | 27 | debugwindow := gocv.NewWindow("debug") 28 | defer debugwindow.Close() 29 | projection := gocv.NewWindow("projector") 30 | defer projection.Close() 31 | 32 | cResults := calibration(webcam, debugwindow, projection) 33 | fmt.Println(cResults) 34 | /* 35 | cResults := calibrationResults{ 36 | pixelsPerCM: 8.33666, 37 | displacement: image.Pt(93, 0), 38 | displayRatio: 0.93, 39 | referenceColors: []color.RGBA{{201, 66, 67, 0}, {88, 101, 65, 0}, {74, 57, 88, 0}, {217, 109, 72, 0}}, 40 | } 41 | */ 42 | vision(webcam, debugwindow, projection, cResults) 43 | } 44 | 45 | type frameInput struct { 46 | webcam *gocv.VideoCapture 47 | debugWindow *gocv.Window 48 | projection *gocv.Window 49 | // TODO: should these be passed as ptrs? 50 | img gocv.Mat 51 | cimg gocv.Mat 52 | scChsBrd straightChessboard 53 | } 54 | 55 | func frameloop(fi frameInput, f func(image.Image, map[image.Rectangle][]circle), ref []color.RGBA, waitMillis int) error { 56 | for { 57 | start := time.Now() 58 | if ok := fi.webcam.Read(&fi.img); !ok { 59 | return fmt.Errorf("cannot read device\n") 60 | } 61 | if fi.img.Empty() { 62 | continue 63 | } 64 | 65 | straightImage := beamerToChessboard(fi.img, fi.scChsBrd) 66 | 67 | // since detect draws in img, we take a snapshot first 68 | actualImage, _ := straightImage.ToImage() //fi.img.ToImage() 69 | spatialPartition := detect(fi.img, actualImage, ref) 70 | 71 | f(actualImage, spatialPartition) 72 | 73 | fps := time.Second / time.Since(start) 74 | gocv.PutText(&straightImage, fmt.Sprintf("FPS: %d", fps), image.Pt(0, 20), 0, .5, color.RGBA{}, 2) 75 | 76 | fi.debugWindow.IMShow(straightImage) 77 | fi.projection.IMShow(fi.cimg) 78 | key := fi.debugWindow.WaitKey(waitMillis) 79 | if key >= 0 { 80 | return nil 81 | } 82 | } 83 | } 84 | 85 | func vision(webcam *gocv.VideoCapture, debugwindow, projection *gocv.Window, cResults calibrationResults) { 86 | img := gocv.NewMat() 87 | defer img.Close() 88 | cimg := gocv.NewMatWithSize(beamerHeight, beamerWidth, gocv.MatTypeCV8UC3) 89 | defer cimg.Close() 90 | 91 | straightener := loadCalibration("calibration.json") 92 | 93 | l := LoadRealTalk() 94 | // translate to beamerspace 95 | pixPerCM := cResults.pixelsPerCM 96 | if cResults.displayRatio != 0 { 97 | pixPerCM *= (1. / cResults.displayRatio) - 1. 98 | } 99 | l.Eval(fmt.Sprintf("(define pixelsPerCM %f)", pixPerCM)) 100 | 101 | fi := frameInput{ 102 | webcam: webcam, 103 | debugWindow: debugwindow, 104 | projection: projection, 105 | img: img, 106 | cimg: cimg, 107 | scChsBrd: straightener, 108 | } 109 | 110 | // ttl in frames; essentially buffering page location for flaky detection 111 | type persistPage struct { 112 | id uint64 113 | ttl int 114 | } 115 | 116 | persistCorners := map[corner]persistPage{} 117 | 118 | if err := frameloop(fi, func(_ image.Image, spatialPartition map[image.Rectangle][]circle) { 119 | clear(l) 120 | datalogIDs := map[uint64]int{} 121 | 122 | for k, v := range persistCorners { 123 | if v.ttl == 0 { 124 | delete(persistCorners, k) 125 | continue 126 | } 127 | persistCorners[k] = persistPage{v.id, v.ttl - 1} 128 | } 129 | 130 | gocv.Circle(&img, image.Pt(5, 5), 5, cResults.referenceColors[0], -1) 131 | gocv.Circle(&img, image.Pt(15, 5), 5, cResults.referenceColors[1], -1) 132 | gocv.Circle(&img, image.Pt(25, 5), 5, cResults.referenceColors[2], -1) 133 | gocv.Circle(&img, image.Pt(35, 5), 5, cResults.referenceColors[3], -1) 134 | 135 | red := color.RGBA{255, 0, 0, 0} 136 | green := color.RGBA{0, 255, 0, 0} 137 | blue := color.RGBA{0, 0, 255, 0} 138 | yellow := color.RGBA{255, 255, 0, 0} 139 | 140 | gocv.Rectangle(&cimg, image.Rect(0, 0, beamerWidth, beamerHeight), color.RGBA{}, -1) 141 | 142 | // TODO: this is cheating, will work for now 143 | // deduplication due to overlapping detection regions 144 | cornersByTop := map[point]corner{} 145 | corners := []corner{} 146 | 147 | // find corners 148 | for k, v := range spatialPartition { 149 | corner, ok := findCorners(v, cResults.referenceColors) 150 | if !ok { 151 | continue 152 | } 153 | if _, ok := cornersByTop[corner.m.p]; ok { 154 | // dont care if this corner is detected in multiple overlapping regions 155 | continue 156 | } 157 | 158 | gocv.Rectangle(&img, k, red, 2) 159 | gocv.Line(&img, corner.m.p.toIntPt(), corner.ll.p.toIntPt(), blue, 2) 160 | gocv.Line(&img, corner.m.p.toIntPt(), corner.rr.p.toIntPt(), blue, 2) 161 | 162 | cs := []color.RGBA{red, green, blue, yellow} 163 | gocv.Circle(&img, corner.ll.p.toIntPt(), 8, cs[int(corner.ll.c)], -1) 164 | gocv.Circle(&img, corner.l.p.toIntPt(), 8, cs[int(corner.l.c)], -1) 165 | gocv.Circle(&img, corner.m.p.toIntPt(), 8, cs[int(corner.m.c)], -1) 166 | gocv.Circle(&img, corner.r.p.toIntPt(), 8, cs[int(corner.r.c)], -1) 167 | gocv.Circle(&img, corner.rr.p.toIntPt(), 8, cs[int(corner.rr.c)], -1) 168 | 169 | cornersByTop[corner.m.p] = corner 170 | corners = append(corners, corner) 171 | } 172 | 173 | // attempt to update corners if their colors dont match corner that was really close to it previous frame 174 | // persisted corners are guaranteed to have matched an existing page 175 | for i, c := range corners { 176 | for o := range persistCorners { 177 | if euclidian(c.m.p.sub(o.m.p)) < 5.0 { 178 | corners[i] = corner{ 179 | ll: dot{c.ll.p, o.ll.c}, 180 | l: dot{c.l.p, o.l.c}, 181 | m: dot{c.m.p, o.m.c}, 182 | r: dot{c.r.p, o.r.c}, 183 | rr: dot{c.rr.p, o.rr.c}, 184 | } 185 | break 186 | } 187 | } 188 | } 189 | 190 | cornersClockwise := map[corner]corner{} 191 | cornersCounterClockwise := map[corner]corner{} 192 | // compare each corner against all others (TODO: can be more efficient ofc) 193 | // try to find another corner: the one clockwise in order that would form a page 194 | for _, c := range corners { 195 | for _, o := range corners { 196 | if c.m.p == o.m.p { 197 | continue 198 | } 199 | right := c.rr.p.sub(c.m.p) 200 | toO := o.m.p.sub(c.m.p) 201 | angle1 := angleBetween(right, toO) 202 | if angle1 > 0.05 { 203 | continue 204 | } 205 | left := o.ll.p.sub(o.m.p) 206 | toC := c.m.p.sub(o.m.p) 207 | angle2 := angleBetween(left, toC) 208 | if angle2 > 0.05 { 209 | continue 210 | } 211 | prev, ok := cornersClockwise[c] 212 | if ok { 213 | // overwrite previously found corner if this one is closer 214 | if euclidian(c.m.p.sub(prev.m.p)) > euclidian(c.m.p.sub(o.m.p)) { 215 | cornersClockwise[c] = o 216 | cornersCounterClockwise[o] = c 217 | } 218 | } else { 219 | cornersClockwise[c] = o 220 | cornersCounterClockwise[o] = c 221 | } 222 | } 223 | } 224 | 225 | // parse corners into pages 226 | pages := map[uint64]page{} 227 | for len(corners) > 0 { 228 | c := corners[0] 229 | next := cornersClockwise[c] 230 | corners = corners[1:] 231 | 232 | cs := []corner{c, next} 233 | // only picking potential pages, those with at least 3 corners recognised 234 | for i := 0; i < 3; i++ { 235 | n, ok := cornersClockwise[next] 236 | if !ok { 237 | break 238 | } 239 | cs = append(cs, n) 240 | c, next = next, n 241 | } 242 | if !(len(cs) == 3) && !(len(cs) == 5 && cs[0].m.p == cs[4].m.p) { 243 | // either we have 3 corners, or we have 5 since the last one is guaranteed to point at the first 244 | continue 245 | } 246 | // because cs[0] = cs[4], remove one instance of that corner 247 | if len(cs) == 5 { 248 | cs = cs[:4] 249 | } 250 | 251 | // if we detect four corners but one is wrong, we should attempt getting page from other configurations 252 | // if we detect only three, we attempt to find by those 3 corners only 253 | var p page 254 | if len(cs) == 3 { 255 | pID := pagePartialID(cs[0].id(), cs[1].id(), cs[2].id()) 256 | pg, ok := pageDB[pID] 257 | if !ok { 258 | continue 259 | } 260 | if _, ok := pages[p.id]; ok { 261 | continue 262 | } 263 | p = pg 264 | missingMid := cs[2].m.p.add(cs[0].m.p.sub(cs[1].m.p)) 265 | // TODO: fill in missing dots positions on missing corner? 266 | missingCorner := corner{m: dot{p: missingMid}} 267 | cs = append(cs, missingCorner) 268 | } else if len(cs) == 4 { 269 | found := false 270 | for i := 0; i < 4; i++ { 271 | cs = []corner{cs[1], cs[2], cs[3], cs[0]} 272 | pID := pagePartialID(cs[0].id(), cs[1].id(), cs[2].id()) 273 | pg, ok := pageDB[pID] 274 | if !ok { 275 | continue 276 | } 277 | if _, ok := pages[p.id]; ok { 278 | continue 279 | } 280 | p = pg 281 | found = true 282 | break 283 | } 284 | if !found { 285 | continue 286 | } 287 | } 288 | // TODO: if ulhc is not properly detected, this will cause issues 289 | for i := 0; i < 4; i++ { 290 | if cs[0].id() == p.ulhc.id() { 291 | break 292 | } 293 | cs = []corner{cs[1], cs[2], cs[3], cs[0]} 294 | } 295 | // error correct colors on corners because 1 might be wrong 296 | // in which case we would persist wrong corner across frames and error correction will not work 297 | //p.ulhc, p.urhc, p.lrhc, p.llhc = cs[0], cs[1], cs[2], cs[3] 298 | p.ulhc = corner{ 299 | ll: dot{cs[0].ll.p, p.ulhc.ll.c}, 300 | l: dot{cs[0].l.p, p.ulhc.l.c}, 301 | m: dot{cs[0].m.p, p.ulhc.m.c}, 302 | r: dot{cs[0].r.p, p.ulhc.r.c}, 303 | rr: dot{cs[0].rr.p, p.ulhc.rr.c}, 304 | } 305 | p.urhc = corner{ 306 | ll: dot{cs[1].ll.p, p.urhc.ll.c}, 307 | l: dot{cs[1].l.p, p.urhc.l.c}, 308 | m: dot{cs[1].m.p, p.urhc.m.c}, 309 | r: dot{cs[1].r.p, p.urhc.r.c}, 310 | rr: dot{cs[1].rr.p, p.urhc.rr.c}, 311 | } 312 | p.lrhc = corner{ 313 | ll: dot{cs[2].ll.p, p.lrhc.ll.c}, 314 | l: dot{cs[2].l.p, p.lrhc.l.c}, 315 | m: dot{cs[2].m.p, p.lrhc.m.c}, 316 | r: dot{cs[2].r.p, p.lrhc.r.c}, 317 | rr: dot{cs[2].rr.p, p.lrhc.rr.c}, 318 | } 319 | p.llhc = corner{ 320 | ll: dot{cs[3].ll.p, p.llhc.ll.c}, 321 | l: dot{cs[3].l.p, p.llhc.l.c}, 322 | m: dot{cs[3].m.p, p.llhc.m.c}, 323 | r: dot{cs[3].r.p, p.llhc.r.c}, 324 | rr: dot{cs[3].rr.p, p.llhc.rr.c}, 325 | } 326 | 327 | rightArm := p.ulhc.rr.p.sub(p.ulhc.m.p) 328 | rightAbs := p.ulhc.m.p.add(point{100, 0}).sub(p.ulhc.m.p) 329 | angle := angleBetween(rightArm, rightAbs) 330 | if p.ulhc.rr.p.y < p.ulhc.m.p.y { 331 | angle = 2*math.Pi - angle 332 | } 333 | p.angle = angle 334 | pages[p.id] = p 335 | 336 | // persist this page across frames 337 | pagePersist := persistPage{id: p.id, ttl: 10} 338 | persistCorners[p.ulhc] = pagePersist 339 | persistCorners[p.urhc] = pagePersist 340 | persistCorners[p.lrhc] = pagePersist 341 | persistCorners[p.llhc] = pagePersist 342 | 343 | // Clockwise from upper left hand corner 344 | pts := []point{p.ulhc.m.p, p.urhc.m.p, p.lrhc.m.p, p.llhc.m.p} 345 | center := pts[0].add(pts[1]).add(pts[2]).add(pts[3]).div(4) 346 | r := ptsToRect([]point{ 347 | rotateAround(center, pts[0], angle), 348 | rotateAround(center, pts[1], angle), 349 | rotateAround(center, pts[2], angle), 350 | rotateAround(center, pts[3], angle), 351 | }) 352 | gocv.Rectangle(&img, r, green, 2) 353 | 354 | aabb := ptsToRect(pts) 355 | gocv.Rectangle(&img, aabb, blue, 2) 356 | 357 | // in lisp we store the points already translated to beamerspace instead of webcamspace 358 | // NOTE: this means distances between papers in inches should use a conversion as well! 359 | //for i, pt := range pts { 360 | // pts[i] = translate(pt, cResults.displacement, cResults.displayRatio) 361 | //} 362 | 363 | dID := page2lisp(l, p, pts) 364 | datalogIDs[p.id] = dID 365 | } 366 | 367 | evalPages(l, pages, datalogIDs) 368 | 369 | for _, illu := range opencv.Illus { 370 | blit(&illu, &cimg) 371 | illu.Close() 372 | } 373 | opencv.Illus = []gocv.Mat{} 374 | 375 | }, cResults.referenceColors, 10); err != nil { 376 | fmt.Println(err) 377 | } 378 | } 379 | 380 | // TODO: only works if area to be colored is still black 381 | // smth like 'set nonblack area in 'from' to white, use that as mask, blacken 'to' area with mask first?' 382 | func blit(from, to *gocv.Mat) { 383 | gocv.BitwiseOr(*from, *to, to) 384 | } 385 | --------------------------------------------------------------------------------