├── example ├── Setup.hs ├── image.jpg ├── friday-report-example.cabal └── Main.hs ├── example.png ├── bench_results.png ├── SUMMARY.md ├── .gitignore ├── README.md ├── 5. Perspectives and conclusion.md ├── 1. Introduction.md ├── 4. A composable image framework.md ├── 2. A taste of Haskell.md └── 3. A typed image processing framework.md /example/Setup.hs: -------------------------------------------------------------------------------- 1 | import Distribution.Simple 2 | main = defaultMain 3 | -------------------------------------------------------------------------------- /example.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RaphaelJ/friday-report/HEAD/example.png -------------------------------------------------------------------------------- /bench_results.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RaphaelJ/friday-report/HEAD/bench_results.png -------------------------------------------------------------------------------- /example/image.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RaphaelJ/friday-report/HEAD/example/image.jpg -------------------------------------------------------------------------------- /SUMMARY.md: -------------------------------------------------------------------------------- 1 | # Summary 2 | 3 | * [Introduction](README.md) 4 | * [A taste of Haskell](2. A taste of Haskell.md) 5 | * [A typed image processing framework](3. A typed image processing framework.md) 6 | * [A composable image framework](4. A composable image framework.md) 7 | * [Perspectives and conclusion](5. Perspectives and conclusion.md) 8 | 9 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Node rules: 2 | ## Grunt intermediate storage (http://gruntjs.com/creating-plugins#storing-task-files) 3 | .grunt 4 | 5 | ## Dependency directory 6 | ## Commenting this out is preferred by some people, see 7 | ## https://docs.npmjs.com/misc/faq#should-i-check-my-node_modules-folder-into-git 8 | node_modules 9 | 10 | # Book build output 11 | _book 12 | 13 | # eBook build output 14 | *.epub 15 | *.mobi 16 | *.pdf -------------------------------------------------------------------------------- /example/friday-report-example.cabal: -------------------------------------------------------------------------------- 1 | name: friday-report-example 2 | version: 0.1.0.0 3 | synopsis: An image processing example for the friday-report 4 | 5 | homepage: https://github.com/RaphaelJ/friday-report 6 | 7 | author: Raphael Javaux 8 | maintainer: raphaeljavaux@gmail.com 9 | 10 | category: Graphics 11 | build-type: Simple 12 | 13 | cabal-version: >=1.10 14 | 15 | extra-source-files: image.jpg 16 | 17 | executable example 18 | main-is: Main.hs 19 | ghc-options: -Wall -O2 -rtsopts -threaded "-with-rtsopts=-N -A10m" 20 | default-language: Haskell2010 21 | 22 | build-depends: base >= 4 && < 5 23 | , friday >= 0.2.2 && < 0.3 24 | , friday-devil >= 0.1 && < 0.2 25 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | This repository contains a set of documents I wrote in the context of a personal 2 | project I had to present to get my Master's degree in Computer Science from the 3 | University of Liège, Belgium. It's not related to my Master's thesis. 4 | 5 | They describe how I implemented *[friday](https://github.com/RaphaelJ/friday)*, 6 | **an image processing library for Haskell**. The library uses quite advanced 7 | functional programming tricks to implement features like **automatic 8 | parallelization of algorithms** or **the ability to fuse a chain of image 9 | processing algorithms in single efficient loops**. 10 | 11 | The target audience was composed of professors who didn't had any extensive 12 | experience with functional programming. **I wrote these documents in such a way 13 | that they can be understood by programmers having only experience with 14 | statically typed programming languages** (like *C++*, *Java* or *C#*). I think 15 | it could be an interesting short read (about 20 printed pages) for such people 16 | who are curious about what can be achieved with functional programming 17 | languages. 18 | 19 | Pardon my English, I'm not a native speaker. 20 | 21 | ## Contents 22 | 23 | 1. [Introduction](1. Introduction.md) (not this document) 24 | 2. [A taste of Haskell](2. A taste of Haskell.md) 25 | 3. [A typed image processing framework](3. A typed image processing framework.md) 26 | 4. [A composable image framework](4. A composable image framework.md) 27 | 5. [Perspectives and conclusion](5. Perspectives and conclusion.md) 28 | 29 | I hope you will enjoy the read ! -------------------------------------------------------------------------------- /example/Main.hs: -------------------------------------------------------------------------------- 1 | {-# LANGUAGE FlexibleContexts 2 | , ScopedTypeVariables #-} 3 | 4 | import Prelude hiding (map) 5 | import System.Environment (getArgs) 6 | 7 | import Vision.Image ( 8 | RGB, InterpolMethod (Bilinear) 9 | , compute, computeP, crop, delayed, shape, resize 10 | ) 11 | import Vision.Image.Storage.DevIL (Autodetect (..), load, save) 12 | import Vision.Primitive (Z (..), (:.) (..), Rect (..), ix2) 13 | 14 | -- Reads an image from a file, applies a composition of transformations to 15 | -- create a centred and squared miniature and then writes the result to a file: 16 | -- 17 | -- usage: ./delayed input.png output.png 18 | main :: IO () 19 | main = do 20 | [inputPath, outputPath] <- getArgs 21 | 22 | -- Loads the image. Automatically infers the format. 23 | io <- load Autodetect inputPath 24 | 25 | case io of 26 | Left err -> do 27 | putStrLn "Unable to load the image:" 28 | print err 29 | Right (input :: RGB) -> do 30 | let -- Gets the size of the image. 31 | Z :. h :. w = shape input 32 | 33 | -- Creates a Rect object which will be used to define how we 34 | -- will crop our image. The rectangle is centered on the largest 35 | -- side of the image. 36 | rect | w > h = Rect ((w - h) `quot` 2) 0 h h 37 | | otherwise = Rect 0 ((h - w) `quot` 2) w w 38 | 39 | -- Crops the image. Doesn't compute the image into a "real" 40 | -- image: by using a delayed representation, this intermediate 41 | -- image will not exist in the computer memory as a large array. 42 | cropped = delayed (crop rect input) 43 | 44 | -- Resizes the image. By using the delayed representation of the 45 | -- cropped image, our compiler should be able to fuse these two 46 | -- transformations into a single loop. 47 | resized = delayed (resize Bilinear (ix2 1000 1000) cropped) 48 | 49 | -- let output = compute resized :: RGB 50 | output <- computeP resized :: IO RGB 51 | 52 | mErr <- save Autodetect outputPath output 53 | case mErr of 54 | Nothing -> 55 | putStrLn "Success." 56 | Just err -> do 57 | putStrLn "Unable to save the image:" 58 | print err 59 | -------------------------------------------------------------------------------- /5. Perspectives and conclusion.md: -------------------------------------------------------------------------------- 1 | # Perspectives and conclusions 2 | 3 | ## Future 4 | 5 | A third major version of *friday* is currently developed and available in the 6 | *v0.3* branch of the GitHub source repository. 7 | 8 | The main (breaking) change is that *friday* is now able to fuse more complex 9 | algorithms: some algorithms are more efficient if they can "remember" some 10 | values when generating pixels. 11 | 12 | As an example, here is the pseudo-code of a simple algorithm that flip an 13 | image over its vertical axis (like in a water reflection): 14 | 15 | for y in [0..h-1]: 16 | src_y = h - 1 - y 17 | 18 | for x in [0..w-1]: 19 | dst[y][x] = src[src_y][x] 20 | 21 | The `src_y` value is constant for each line. It's not possible to "cache" such 22 | value with a generating function as simple as `fromFunction`. 23 | 24 | For this reason, *friday* has since the first version a more advanced generating 25 | function that accepts two additional functions, one that generates constant line 26 | values and one that generates column constant values. Manifest images were able 27 | to cache these values but they were ignored by delayed images. 28 | 29 | Fusing these two functions is way harder than fusing the pixel generating 30 | function. You can't just give the output of the previous function in the 31 | pipeline to the next function in the pipeline, because of index issues. By now, 32 | it works in the developpement branch for simple algorithms (resizing, 33 | thresholding ...) but it doesn't for filters (blurs, derivatives ...). Getting 34 | every type right while remaining generic is quite an headache as a lot of 35 | constraints are involved. 36 | 37 | ## Maintaining a public project is time consuming 38 | 39 | Even if the project is quite young and experimental (the first public version 40 | has been released one year ago), it's already used by some people. 41 | 42 | I already received a dozen bug reports via the GitHub repository. There has not 43 | yet been a single "true" bug. The received reports were about feature requests 44 | or errors when the library was compiled on older systems. Some fixes required 45 | a few hours of work and were rarely requested when I had time to spend on the 46 | project. 47 | 48 | I received two significant external contributions to integrate two new image 49 | processing algorithms. They required a significant amount of time to be merged 50 | in the main codebase as I needed to rewrite some parts to fit the library's way 51 | of doing things. 52 | 53 | To deal with breaking changes, I had to start maintaining a *change log* and I 54 | had to start to develop in a separate *experimental* branch while maintaining a 55 | *stable* branch. -------------------------------------------------------------------------------- /1. Introduction.md: -------------------------------------------------------------------------------- 1 | # Introduction 2 | 3 | This document will describe a project I'm working on for about two years. This 4 | is not the sole personal project I worked on during this period, but it is 5 | probably the largest and the most serious one. 6 | 7 | The project is named **friday** and is a *image processing framework* written 8 | in the strongly typed functional programming language **Haskell**. 9 | 10 | I started thinking about a functional image processing library while doing an 11 | internship in computer vision three years ago. I noticed that most bugs I 12 | encountered were avoidable if the library we used (*OpenCV*) was written in a 13 | more functional way, and with a stronger type system. 14 | 15 | While a lot of image processing libraries already exist, designing one from 16 | scratch for a functional programming language is actually quite interesting. 17 | 18 | The four distinguishing features of this project are: 19 | 20 | * A reliance on a strongly typed programming language to detect programming 21 | errors directly at compile time. I will give in the course of this document 22 | a few examples of bugs which can be avoided at compile time with *friday* 23 | while they will silently occur at runtime with other libraries. 24 | * The ability to fuse image transformations. 25 | For example, if one wishes to apply a rotation on a resized image, he can 26 | use *friday* and the *Haskell* compiler to generate a single loop that will 27 | automatically combine the resizing and the rotating operations into a single 28 | algorithm, removing the need to store the intermediate resized image [1]. 29 | * The ability to automatically parallelize algorithms to use multiple 30 | processors. 31 | The library is able to parallelize upon request image transformations, even 32 | these which have been generated by the fusion mechanism. 33 | * Being extremely generic. One who wants to create new algorithms, new 34 | pixel color-spaces or new ways to store an image [2] will be able to reuse 35 | the advanced type checking features and both the fusion and automatic 36 | parallelization mechanisms. 37 | 38 | The library is more like a set of *building blocks to write image processing 39 | algorithms in a functional programming style* than a collection of image 40 | processing algorithms. 41 | Because of that, the expression *image processing framework* seems more 42 | appropriate than *image processing library*. 43 | 44 | [1] This feature might seem odd for programmers unfamiliar with functional 45 | programming. The technique involved, named *deforestation* or *stream fusion*, 46 | is however well known to functional programmers. A similar feature exists in 47 | some list, array and stream libraries. 48 | 49 | [2] One use case involving the creation of a new image type could be to 50 | interface with an existing library or hardware (GPU ...). 51 | 52 | ## Other features 53 | 54 | The library currently supports four color-spaces: RGB, RGBA, HSV and grey-scale 55 | pixels. Images can be converted between these color-spaces. 56 | 57 | The following features and algorithms have been implemented: 58 | 59 | * various image transformations: resize, crop, vertical and horizontal 60 | flip, flood fill ... 61 | * various filters: morphological transformations (dilation and erosion), 62 | blurring (mean and Gaussian blurs) and derivative operators (Sobel and Scharr 63 | operators), with a generic interface to create new filters from kernels. 64 | * non-adaptive, adaptive, Otsu and SCW thresholding methods. 65 | * edge detection using Canny's algorithm. 66 | * full type-safe support for masked images. 67 | * histogram computation, histogram comparisons and image equalization by their 68 | histograms. 69 | 70 | Loading and saving images from and to disk is delayed to a C library, with 71 | wrappers provided by 72 | [the separate friday-devil package](https://github.com/RaphaelJ/friday-devil). 73 | 74 | ## Availability and source code 75 | 76 | The library was released for the first time with its online documentation on 77 | the [official *Haskell*'s package 78 | repository](https://hackage.haskell.org/package/friday) in August 2014 and a 79 | second major version was released in January 2015. 80 | A third experimental version is currently developed and is available in the 81 | *v0.3* branch of the [GitHub source 82 | repository](http://github.com/RaphaelJ/friday). 83 | 84 | According to the *Haskell*'s package repository statistics, about 150 users 85 | compile and install the library each month, making it the third most used image 86 | processing package in the repository. 87 | 88 | ## Document structure 89 | 90 | I assume that the reader has no knowledge of the *Haskell* programming 91 | language. The following section gives some *Haskell* basics and 92 | introduces the language's syntax and type system. This is mandatory 93 | to understand the other parts of the document as *Haskell* is a very unique 94 | programming language. 95 | 96 | The second section exposes how *friday* design diverges from others image 97 | processing libraries by relying on a strong and type system, and how this 98 | enables safety and reusability. 99 | 100 | The third section talks about the fusion and parallelization mechanisms and how 101 | they are implemented. Both mechanisms are very tightly related. The section also 102 | briefly addresses performances. 103 | -------------------------------------------------------------------------------- /4. A composable image framework.md: -------------------------------------------------------------------------------- 1 | # A composable image framework 2 | 3 | This section will present two cool features of *friday*. As I said in the 4 | introduction, the library has the property that you can compose algorithms and 5 | expect them to be *fused in a single and efficient loop* or *automatically 6 | parallelized*. 7 | 8 | The main motivation when designing this feature was to be able to design complex 9 | but efficient transformations from a small set of simple functions (like `map`). 10 | 11 | ## The image and function isomorphism 12 | 13 | To understand how the compiler and the library are allowed to fuse and 14 | parallelize algorithms, one must remember *referential transparency*. 15 | Referential transparency is the property which asserts that *pure function 16 | calls can be replaced by the values they generate without changing the output 17 | of the program*. 18 | 19 | This property has an interesting effect on images. Images (and more generally 20 | arrays and vectors), in a *pure* functional programming context, may be thought 21 | of as functions whose domains are restricted to a contiguous subset of integers 22 | (the valid coordinates of the pixels in the case of an image). In other words, 23 | you can replace the image indexing operator by the function which generates the 24 | pixels of the image *without* changing the output of the program. 25 | 26 | The main advantage of considering images as functions instead of arrays is 27 | that functions are composable. Function composability is the idea which propels 28 | the fusion mechanism: *friday considers images as functions and composes them 29 | into larger functions*. 30 | 31 | ## A low level image creation interface 32 | 33 | Image types implement a type class interface named `FromFunction` which can be 34 | used to build images. 35 | 36 | *The type class provides a function to construct an image instance from a pure 37 | function*. The function's signature looks like this one: 38 | 39 | ```Haskell 40 | fromFunction :: Size -> (Point -> ImagePixel i) -> i 41 | ``` 42 | 43 | The function accepts the size of the image to create and a pure function which 44 | generates pixels from their coordinates. 45 | 46 | You usually don't need to use `fromFunction` directly. Functions like `map` 47 | provide an higher level interface and are implemented using `fromFunction`. 48 | 49 | ## Two image representations 50 | 51 | Two image types implement the previously described `FromFunction` type class. 52 | One is named the **manifest representation** while the other is named the 53 | **delayed representation**. 54 | 55 | * The *manifest* representation uses an in memory vector to store the pixels 56 | values. The vector is entirely initialized when this image is constructed 57 | with `fromFunction`. 58 | * The *delayed* representation only stores the size and the function which 59 | constructs the image. 60 | 61 | In other words, manifest images are memory resident images while delayed images 62 | are only composed of the function that generate pixel values. 63 | 64 | Both image types also implement the `Image` type class which provides an 65 | interface to get pixel values. They can therefore be freely substituted as you 66 | usually only access to these two using this interface. 67 | 68 | ## Fusing delayed images 69 | 70 | Knowing about the manifest and the delayed representations, we can now write a 71 | pipeline of transformations and only have the original and the resulting image 72 | to exist in memory. 73 | 74 | Let's write a program that creates a black and white thumbnail by 75 | 76 | 1. converting the image to grey-scale. 77 | 2. applying our `blackAndWhite` function to convert grey-scale pixels to black 78 | and white values. We defined the `blackAndWhite` function in the course of 79 | the previous section using the `map` function. 80 | 3. removing edges so the image is a square as large as its smallest side. 81 | 4. resizing the resulting image to be 250 pixels wide. 82 | 83 | Here is the source code of the program which created this thumbnail: 84 | 85 | ```Haskell 86 | -- Gets the size of the input image. 87 | (h, w) = shape input 88 | 89 | -- Gets a grey scale delayed image from the input image. 90 | grey = delayed (convert input) 91 | 92 | -- Applies our `blackAndWhite` function to the grey-scale image. 93 | black = delayed (blackAndWhite grey) 94 | 95 | -- Creates a rectangle (Rect) object which will be used to define how we 96 | -- will crop/cut our image and remove edges. 97 | -- 98 | -- The rectangle is centered on the largest side of the image. 99 | rect = if w > h then Rect ((w - h) / 2) 0 h h 100 | else Rect 0 ((h - w) / 2) w w 101 | 102 | -- Crops the image. 103 | cropped = delayed (crop rect black) 104 | 105 | -- Resizes the image. 106 | -- 107 | -- By using the delayed representations, our compiler should be able to 108 | -- fuse these four transformations into a single loop. 109 | resized = manifest (resize Bilinear (250, 250) cropped) 110 | ``` 111 | 112 | Notice the use of the two `manifest` and `delayed` functions. These two do 113 | nothing except giving hints to the compiler on what image representation we 114 | want to generate. If we don't write these functions, the *Haskell*'s compiler 115 | will fail and complain that it can't unambiguously determine what image types to 116 | generate. 117 | 118 | ## Implementation 119 | 120 | The *Haskell*'s compiler will generate an efficient loop from the previous 121 | program. 122 | 123 | Functions which accept delayed images or which return delayed images are 124 | requested to be inlined. In the case of the previous example, `convert` will be 125 | inlined in `blackAndWhite`, which will be inlined in `crop`, which will be 126 | inlined in `resize`, which will be inlined. 127 | 128 | The `fromFunction` call inside the `resize` function will also be inlined. As 129 | the `fromFunction` implementation to generate manifest images is a loop which 130 | calls the pixel generation function for the entire image, this will result in a 131 | specialized loop body incorporating all the inlined pipeline. 132 | 133 | It's quite difficult to figure out why this works if you're not used to this 134 | kind of pattern (we ask the compiler to inline functions which receive inline 135 | functions as arguments). But trust me, it does work. 136 | 137 | If an algorithm is called with manifest images (like in other image processing 138 | libraries), it will not be inlined. The compiler will however be requested to create a specialized instance for the requested pixel type(s). 139 | 140 | ## Automatic parallelization 141 | 142 | The library also provide an useful function named `computeP`, for "*compute* in 143 | *p*arallel". 144 | 145 | The function is actually really simple: it computes a manifest image from 146 | a delayed image. But it uses available processors to share the work. 147 | 148 | Say that we want to parallelize the thumbnail program we just wrote, we just 149 | need to replace this line: 150 | 151 | resized = manifest (resize Bilinear (250, 250) cropped) 152 | 153 | By this line: 154 | 155 | resized = computeP (resize Bilinear (250, 250) cropped) 156 | 157 | ... and the compiler will generate a parallel loop for our entire pipeline of 158 | transformations ! 159 | 160 | We will not notice an huge speedup because of the small size of the thumbnails 161 | we generate (250 x 250 pixels), but it gives an almost linear speedup on 162 | programs that work on larger images. 163 | 164 | You can freely substitute calls to `manifest` for calls to `computeP` without 165 | risking any race condition as *pure functions are always thread safe*. 166 | 167 | 168 | ## Mutable images 169 | 170 | *There is some image processing algorithms that can't be expressed efficiently without mutable states* (i.e. they can't be efficiently written with 171 | `fromFunction`). 172 | 173 | Drawing routines (e.g. drawing a rectangle in an image or the flood fill 174 | algorithm) are such algorithms. 175 | 176 | For this reason, *manifest images can also be created in a context where mutable 177 | states are allowed*. When in this context, the programmer is able to update 178 | image values exactly as he would do in an imperative programming language. 179 | 180 | Functions written in this context are still deterministic as you can't do side 181 | effect nor use uninitialized variables. A mutable state is also not allowed to 182 | leak outside this context (the context must return immutable values). Those 183 | functions appear *pure* from the outside. We rely on the type checker to 184 | enforce this closed containment, by using a functional pattern named *monads*. 185 | Monads are a far too advanced concept to be introduced in this document. 186 | 187 | Delayed images can't be created in this mutable context. Thus *image 188 | transformations written this way can't be fused nor automatically parallelized*. 189 | 190 | ## Performances 191 | 192 | The library has been reasonably optimized and could be used to a limited extent 193 | for real-time applications. 194 | 195 | The graph in Appendix shows how the library compares to the *OpenCV* and the 196 | *ImageMagick* libraries, both written in C. Multi-core parallelism was 197 | *disabled* on all three libraries. The fastest implementation for each 198 | algorithm is taken as reference. 199 | 200 | *friday* is about one order of magnitude slower than *OpenCV* but does well 201 | against *ImageMagick*. 202 | 203 | *OpenCV* performances are explained by hand-written specialized algorithms which 204 | make an intensive use of vectorial instructions (SSE, AVX and the like) for 205 | common image types. 206 | -------------------------------------------------------------------------------- /2. A taste of Haskell.md: -------------------------------------------------------------------------------- 1 | # A taste of Haskell 2 | 3 | *Haskell* is a functional and strongly typed programming language. 4 | 5 | The first part of this section explains that functional programming is mostly 6 | about programming without mutation and side effect. 7 | 8 | The second part gives a brief introduction to *Haskell* syntax and type system. 9 | 10 | ## What is functional programming ? 11 | 12 | Functional programming is, in its most universal definition, a programming style 13 | which tries to *avoid mutating state* and mutable data structures, by 14 | *emphasing on the composition of expressions* in place of the execution of 15 | commands and instructions. 16 | 17 | See the following C imperative program which manipulates a rectangle: 18 | 19 | ```C 20 | struct rect_t { 21 | int x, y, width, height; 22 | } 23 | 24 | void move(rect_t *rect, int dx, int dy) 25 | { 26 | rect->x += dx; 27 | rect->y += dy; 28 | } 29 | 30 | rect_t my_rectangle(void) 31 | { 32 | rect_t rect = { 33 | .x = 0, .y = 0, 34 | .width = 100, .height = 100 35 | }; 36 | 37 | move(&rect, 10, 10); 38 | 39 | return rect; 40 | } 41 | ``` 42 | 43 | And now compare it with this other C program written in a functional style: 44 | 45 | ```C 46 | struct rect_t { 47 | int x, y, width, height; 48 | } 49 | 50 | rect_t move(rect_t rect, int dx, int dy) 51 | { 52 | return { 53 | .x = rect.x + dx, 54 | .y = rect.y + dy, 55 | .width = rect.width, 56 | .height = rect.height 57 | }; 58 | } 59 | 60 | rect_t my_rectangle() 61 | { 62 | rect_t rect = { 63 | .x = 0, .y = 0, 64 | .width = 100, .height = 100 65 | }; 66 | 67 | return move(rect, 10, 10); 68 | } 69 | ``` 70 | 71 | The main difference is that in the functional style, *we never modify 72 | any data previously created*. In our example, when the original rectangle needs 73 | to be moved a *new* rectangle will be created. In functional programming there 74 | is no *destructive update* and data are *immutable*. 75 | 76 | With this absence of destructive updates, variables are used in a mathematical 77 | sense, as identifiers to immutable expressions or values. In the `my_rectangle` 78 | function of the example, `rect` is associated to the value of an expression, 79 | and this value will never change. 80 | 81 | Functions which are written in this functional style, without using any 82 | mutable state, are called *pure* and are very close to the mathematical 83 | definition of a function. Pure functions accepts a set of immutable values as 84 | input and computes a new immutable value as output, doing nothing else. 85 | 86 | ### Removing mutable states does matter 87 | 88 | Prohibiting mutable states has the effect of *removing all side effects*. 89 | Pure functions can't call `printf()` or write to a file, as this can be seen 90 | as modifying the state of the runtime environment. Nor they can read from a 91 | file this is equivalent to read from a mutable state. 92 | 93 | *Pure functions are deterministic*. A pure function call can be replaced by the 94 | value it will return. This is called *referential transparency*. Referential 95 | transparency is the key concept to understand why *friday* and the *Haskell* 96 | compiler are able to safely fuse two image transformations into a single loop 97 | and to automatically paralellize image processing algorithms. 98 | 99 | While time is important in imperative programming (it determines the order in 100 | which instructions are executed), it is an implicit concept in functional 101 | programming. 102 | As pure expressions always return the same values, the time at which they are 103 | executed does not matter. In *Haskell*, expressions and function calls are 104 | evaluated in a dependency graph: think about a spreadsheet, you do not define 105 | the order in which cells are computed, they are evaluated as needed. 106 | 107 | Pure functions are also easier to test, thread-safe by design and more 108 | reusable. Code written in a functional style is usualy easier to reason about as 109 | you only need to find the definition of a variable to get its actual value 110 | (this doesn't hold in an imperative program as values change over time). 111 | 112 | ### The world needs side effects 113 | 114 | A sharp-minded reader will argue that a program that does not do any side 115 | effect is useless, as it has no way to communicate with the outside world. And 116 | indeed, programmers write programs for their side effects. 117 | 118 | *Haskell* allows you to write functions which do side effects but they will 119 | be tagged differently by the type system. Here is also the idea of 120 | *contamination*: any function which call a function tagged as having side 121 | effects will be tagged as well. Similary, a function which has been typed as not 122 | having side effect will not be able to call a function with side effects. 123 | 124 | Good functional programmers will try to write as few functions with side 125 | effects as possible, to benefit from the advantages of pure functions. 126 | 127 | ## Haskell 128 | 129 | Now that you know what functional programming is about, let me introduce 130 | *Haskell*. 131 | 132 | You don't need to know a lot of things about the language to understand the 133 | remaining of this document. 134 | 135 | From a syntactic point of view, *Haskell* programs are made out of 136 | *equations*, not *instructions*. An equation has a left hand side and a 137 | right hand side, with an `=` sign in between. The left hand side is the 138 | identifier of the equation while the right hand side is an expression 139 | denoting its value. 140 | 141 | ```Haskell 142 | -- This is a comment 143 | 144 | x = y * z + 5 145 | y = 10 146 | z = 15 147 | ``` 148 | 149 | As you can see, equations don't need to be defined in order. The language 150 | evaluates them when their values are needed. 151 | 152 | Equations can accept arguments, in which case they are functions. There is no 153 | parenthesis nor comma between function parameters. Likewise, there is no 154 | parenthesis nor comma between arguments when making a function call. 155 | Parenthesis are needed when the order of operations does not fit, like in 156 | arithmetic. Function call is the "operator" with the highest priority. You 157 | don't need to worry too much about that, it feels natural: 158 | 159 | ```Haskell 160 | -- This function squares a number. 161 | square n = n * n 162 | 163 | -- This function accepts two numbers and computes the hypotenuse of a 164 | -- triangle. 165 | pythagoras a b = sqrt (square a + square b) 166 | ``` 167 | 168 | ### Haskell's type system 169 | 170 | While I told you several times that *Haskell* has a strong and advanced type 171 | system, I didn't write any type nor function signature in the previous 172 | examples. 173 | 174 | This may feel strange at first, but these examples were actually entirely 175 | statically typed. *Haskell* has a program-wise *type inference* engine. It's is 176 | a feature which makes the compiler able to deduce expression, function and 177 | equation types by itself. 178 | 179 | We can ask the type inference engine for the type signatures he find while 180 | analysing our equations. For example, if we ask the type signature of the 181 | `pythagoras` function, we will get: 182 | 183 | ```Haskell 184 | pythagoras :: Floating a 185 | => a -> a -> a 186 | ``` 187 | 188 | What we see on the second line is that the function accepts two values of type 189 | `a` and that it returns a value of this *same* type `a`. Argument types are 190 | separated by the `->` operator and the type at the right of the last `->` is 191 | always the one of the returned value [1]. 192 | 193 | `a` is not a "real" type. It's a *type variable*. Type variables in 194 | signatures are "free types" in the sense that they can be replaced by a "real" 195 | type. If we were in *C++* or *Java* and if we were using templates or generics, 196 | type variables would be called *type parameters* (the `T` in `vector`). 197 | 198 | In *Haskell*, type variables always start with a lower case letter while 199 | "real" types start with an upper case letter. `Int` is a real type while 200 | `a_type` is a type variable. 201 | 202 | Constraints can be applied to type variables. Constraints in a type signature 203 | are located after the `::` operator but before the `=>` operator. Our 204 | `pythagoras` function has one constraint, `Floating a`. This constraint is 205 | saying that our type variable `a` must be an instance of the `Floating` *type 206 | class*. Type classes are similar to *Java* interfaces or *C++* abstract 207 | classes. This `Floating` type class is implemented by floating point numbers. 208 | 209 | To summarize, this signature is telling us that our `pythagoras` function 210 | accepts two floating point numbers of the same type and that it returns a 211 | floating point number of this very same type. The type inference engine always 212 | tries to generalize signatures as much as possible. 213 | 214 | This kind of polymorphism which relies on type classes as constraints is called 215 | *parametric polymorphism* and is similar to what can be done with *Java* 216 | generics or *C++* templates. It's however very different from the traditional 217 | Object Oriented polymorphism in the sense that every function call can be 218 | resolved at compile time (there is *no late binding*). This is an important 219 | feature for this project as you can write generic algorithms which don't suffer 220 | from reduced performances when compared to their specialized counterparts. 221 | 222 | [1] There is a practical reason for why function argument types are separated 223 | by this `->` operator and not another, but this goes out of the scope of this 224 | report. Take a look after *Currying* on the Internet if you are interested. 225 | -------------------------------------------------------------------------------- /3. A typed image processing framework.md: -------------------------------------------------------------------------------- 1 | # A typed image processing framework 2 | 3 | To understand why you can benefit from a typed image processing framework, one 4 | must be aware of how existing image processing libraries are designed. 5 | 6 | Images in these libraries must be generic data structures relative to the 7 | pixel's type, as users will expect to use the same interface, whatever the image 8 | content is; e.g. you should expect to use the same functions when you are 9 | working with an grey-scale or an RGB image. 10 | 11 | If you take a look at *OpenCV* --- which is probably the most popular image 12 | processing library --- you will observe that their images are stored in a data 13 | structure similar to this *C++* type: 14 | 15 | ```C++ 16 | class Mat { 17 | int type; 18 | int rows, cols; 19 | uchar *data; 20 | 21 | // Returns a pixel's value. 22 | templace 23 | inline T at(int x, int y) 24 | { 25 | return *((T *) data + cols * y + x); 26 | } 27 | } 28 | ``` 29 | 30 | They implement pixel polymorphism using (unsafe) pointer castings, a familiar 31 | technique to C programmers. They also add a `type` field so programmers can 32 | discriminate pixel types at runtime and they provide a function which carries 33 | out the pointer arithmetic to get a pixel value. 34 | 35 | The first major issue with this approach is that **neither the compiler nor the 36 | runtime are aware of the types of the data they handle**. i.e. a grey-scale 37 | image (made of 8 bits pixels) has exactly the same type as an RGBA image (made 38 | of 32 bits pixels). 39 | 40 | You're entirely free to write something like this: 41 | 42 | ```C++ 43 | Mat grey_image = get_grey_image(); 44 | grey_image.at(10, 15); 45 | ``` 46 | 47 | ... which will fail **silently** at runtime, generating silly values. 48 | This is not even dynamic typing. Dynamic typing implies type-checking at 49 | runtime, but there is no type checking in this case: it is literally untyped 50 | programming. In OpenCV, it is the programmer's responsibility to check that 51 | images are correctly typed by looking at the `type` field of the `Mat` 52 | data-structure. 53 | 54 | Function signatures are also too weak to tell exactly which image types 55 | functions can accept as input or generate as output. 56 | Function authors are required to write a documentation telling exactly what 57 | image types are handled, and they need to keep that documentation up to date. 58 | It is common in practice to find undocumented or incorrectly documented 59 | functions, even in a library as used as *OpenCV*. 60 | 61 | 62 | The second major issue is that **you do not have true genericity**. It is quite 63 | hard --- and in most cases impossible --- to write efficient algorithms that 64 | work on any pixel type. 65 | 66 | In the case of *OpenCV*, algorithms are often written in multiple 67 | instances, each specialized for a pixel type. For example, the thresholding 68 | function starts with a large conditional expression which dispatches the 69 | execution to different instances of the algorithm, depending on the pixel type. 70 | That also means that you can not expect the function to work with a new pixel 71 | type. 72 | 73 | ## What friday does 74 | 75 | *friday* is entirely type-safe and generic in respect to pixel and image types. 76 | Images are accessed and generated using two type classes which do not restrict 77 | how images are represented. 78 | 79 | *You can write algorithms and expect the type inference engine to find the most 80 | generic signature for it*. 81 | 82 | As an example, let's write a function that convert a grey-scale image into 83 | a black and white image: 84 | 85 | ```Haskell 86 | -- First, we create a simple function to convert grey-scale values (in the 87 | -- range [0; 255]) into black (0) and white (255) values. 88 | f val = if val > 127 then 255 else 0 89 | 90 | -- Then we use the `map` function to define our black and white 91 | -- transformation. 92 | -- 93 | -- `map` is a function that applies the given function to each pixel 94 | -- of an image, returning a new image. 95 | blackAndWhite img = map f img 96 | ``` 97 | 98 | When we ask the compiler what is the type of `blackAndWhite`, it gives us 99 | the following signature: 100 | 101 | ```Haskell 102 | blackAndWhite :: (Num (ImagePixel src), Num (ImagePixel res), 103 | Ord (ImagePixel src), FunctorImage src res) 104 | => src -> res 105 | ``` 106 | 107 | The function has two type variables: `src` and `res`, corresponding to the 108 | source 109 | and the generated images, respectively. Four constraints are applied on these 110 | two image types: 111 | 112 | * `Num (ImagePixel src)` and `Num (ImagePixel res)` require the pixel types of 113 | both images to be numbers. 114 | * `Ord (ImagePixel src)` require our source image to be composed of comparable 115 | pixels. This is because of the call to the `>` operator in our `f` function. 116 | * `FunctorImage src res` means that we must be able to apply the `map` function 117 | to generate an image of type `res` from an image of type `src`. 118 | 119 | The function obviously works if we use it on a grey-scale image. But if we 120 | give it an RGB image, the compiler will fail with the following error: 121 | 122 | Could not deduce (Num RGBPixel) 123 | 124 | This is the compiler way of telling us that an `RGBPixel` is not an number, and 125 | that thereby we can't compare it to another number (127 in our example). 126 | 127 | *friday* uses *type signatures as a kind of documentation that the compiler can 128 | generate, understand and enforce*. This is really different from what is 129 | happening in libraries like *OpenCV* where the programmer is expected to 130 | document and enforce types. 131 | 132 | ### Type families 133 | 134 | Readers who are used to work with generics or templates in other programming 135 | languages (like *Java*, *C++* or *C\#*) could argue that the previous example 136 | could be as well implemented in these languages. That's true, but this first 137 | example was rather simple. 138 | 139 | Yet *friday* makes use of more advanced *Haskell*'s type system features 140 | that doesn't exist in these languages. The most important feature is called 141 | *type families*. Type families are like type system functions. The concept is 142 | too wide to be explained in this document but can be easily guessed with an 143 | example: 144 | 145 | Let's talk about *image histograms*. Image histograms are the color 146 | distribution of images --- i.e. the number of pixels for each color. One 147 | interesting thing about histograms is that they have as many dimensions as the 148 | image they are computed from has channels. A grey-scale histogram will be a 149 | single dimension vector, an RGB histogram will be a three dimensions array, an 150 | RGBA histogram will be a four dimensions array and so on ... For example, the 151 | value of an RGB histogram at index `(0, 255, 0)` will give you the number of 152 | green pixels in the image. 153 | 154 | *friday* uses type families to enforce type safety when computing histograms 155 | from images. 156 | 157 | The following expression computes a three dimensions histogram from an RGB 158 | image: 159 | 160 | ```Haskell 161 | hist :: Histogram DIM3 162 | hist = histogram rgbImg 163 | ``` 164 | 165 | As we can see in the type signature, the `Histogram` data type receives a type 166 | parameter, namely `DIM3` (in *Java* or *C++*, it would have been written 167 | `Histogram`). `DIM3` is an alias for a tuple of three integers. This tuple 168 | is used to index values in the histogram. 169 | 170 | But what if we tried to compute a single dimension histogram from this very same 171 | image ? 172 | 173 | ```Haskell 174 | hist :: Histogram DIM1 175 | hist = histogram rgbImg 176 | ``` 177 | 178 | The compiler will fail with the following message: 179 | 180 | Couldn't match type 'DIM3' with 'DIM1' 181 | 182 | That's because there is a type system constraint on the `histogram` function 183 | which uniquely determines the dimension of an histogram from an image pixel 184 | type. 185 | 186 | This kind of constraint is more complex to express that those from the previous 187 | example and requires an advanced type system. In *Haskell*, it's achieved with 188 | type families. 189 | 190 | ### Empty types 191 | 192 | *friday* relies heavily on another type system trick named *empty types*. 193 | 194 | Empty types are types that do not hold any value. An empty type could be 195 | defined in C as: 196 | 197 | ```C 198 | struct my_empty_type { }; 199 | ``` 200 | 201 | In Haskell, they are defined like this: 202 | 203 | ```Haskell 204 | data MyEmptyType 205 | ``` 206 | 207 | I used them in several places as an alternative to enumerations. 208 | 209 | For example, the library provides the `load` or the `save` functions to load 210 | and save images from and to the hard drive. You are required to specify the 211 | image format (*BMP*, *JPEG*, *PNG* ...). However, some formats (like *GIF*) can 212 | be read from but can't be saved to. The `load` function also accepts the 213 | special `Autodetect` format which detects the image type using the file's 214 | headers. 215 | 216 | Instead of using an enumeration of constants to specify file formats like one 217 | would have done in *C++*, each format is its own empty type: 218 | 219 | <<<<<<< HEAD 220 | ======= 221 | 222 | >>>>>>> 06470e8857308fcfb0ea6e6691a36a8c28796a94 223 | ```Haskell 224 | data Autodetect 225 | data BMP 226 | data JPEG 227 | data PNG 228 | data GIF 229 | ... 230 | ``` 231 | 232 | Then you have two type classes, `LoadImageType` and `SaveImageType`. Formats 233 | which can be used with `load` are instance of `LoadImageType` and formats 234 | which can be used with `save` are instance of `SaveImageType`. 235 | 236 | If you try to save an image as a *GIF* file, you will get the following error at 237 | *compile time*: 238 | 239 | No instance for (SaveImageType GIF) 240 | 241 | As a side note, empty types don't exist at runtime and have no memory 242 | footprint. They are only known to the compiler. 243 | 244 | ### Masked images 245 | 246 | Another interesting feature of *friday* is how it manages *masked images*. 247 | 248 | Masked images are images which are not defined for every coordinates in their 249 | shape. They are used when you want to ignore some areas of an image. For 250 | example, you will use a masked image if you don't want algorithms to be applied 251 | on the background part of the image. 252 | 253 | With *OpenCV*, functions which can work with masked images accept an additional 254 | "mask" parameter which is a bitmap image specifying which pixels should be 255 | ignored. That means that you need to treat masked images explicitly 256 | when you write new algorithms. 257 | 258 | In *friday*, masked images are a distinct image type which shares some type 259 | classes with traditional image types. So far as you don't use unmasked-specific 260 | functions, the compiler is able to generalize your algorithms to masked images. 261 | The `blackAndWhite` function we wrote earlier also works with masked images ! 262 | --------------------------------------------------------------------------------