├── .hgignore
├── README.md
├── slides
├── Makefile
├── slides.html
└── slides.md
├── src
├── Download.hs
├── Links.hs
├── Main.hs
├── RankPages.hs
└── Spider.hs
└── strange-loop-2011.cabal
/.hgignore:
--------------------------------------------------------------------------------
1 | ^(?:dist|cabal-dev)$
2 | \.(?:aux|eventlog|h[ip]|log|[oa]|orig|prof|ps|rej|swp)$
3 | ~$
4 | syntax: glob
5 | .\#*
6 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # Slides from the Haskell workshop at Strange Loop 2011
2 |
3 | I gave a 3-hour "intro to Haskell" workshop at the Strange Loop
4 | conference, on September 18, 2011. These are the slides I used,
5 | with a couple of post-workshop bugs fixed.
6 |
7 | You can [read the slides online
8 | here](http://bos.github.com/strange-loop-2011/slides/slides.html).
9 |
10 | They're formatted from Markdown to HTML using John MacFarlane's
11 | fabulous [pandoc tool](http://johnmacfarlane.net/pandoc/).
12 |
13 | Feel free to use them yourself, but if you do, please give attribution
14 | to me. Thanks!
15 |
--------------------------------------------------------------------------------
/slides/Makefile:
--------------------------------------------------------------------------------
1 | all: slides.html
2 |
3 | slides.html: slides.md
4 | pandoc --offline -s -t slidy -o $@ $<
5 |
6 | clean:
7 | -rm -f slides.html
8 |
--------------------------------------------------------------------------------
/slides/slides.html:
--------------------------------------------------------------------------------
1 |
2 |
4 |
5 |
6 |
7 |
8 |
9 |
10 | Haskell: Functional Programming, Solid Code, Big Data
11 |
30 |
33 |
36 |
37 |
38 |
39 |
Haskell: Functional Programming, Solid Code, Big Data
40 |
41 | Bryan O'Sullivan
42 |
43 |
2011-09-18
44 |
46 |
47 |
Welcome!
48 |
main = putStrLn "hello!"
49 |
50 | My name is Bryan O'Sullivan
I started using Haskell in 1993
I wrote a book about it, "Real World Haskell"
53 | I write lots of open source code
57 | My company invests heavily and openly in Haskell
61 |
65 |
68 |
69 |
Copy these slides (if you want)
70 |
git clone https://github.com/bos/strange-loop-2011
71 |
72 |
75 |
76 |
My Haskell background
77 |
78 | Began using Haskell at university
When I graduated in 1995, I switched to more mainstream languages
80 |
81 | - C, C++, Java, Python, Perl, etc.
82 |
My interest reawakened around 2005
I've found Haskell to be a great general-purpose language
The community of people is amazing
87 |
90 |
91 |
What to expect 1
92 |
93 | Haskell is a fairly big language
Since so much is unfamiliar to newcomers, expect to wander far from your comfort zone
I'm going to teach you interesting things, but not everything
97 |
100 |
101 |
What to expect 2
102 |
103 | This is a hands-on workshop: you'll be writing code!
There will be a short break every 45 minutes
Don't be afraid to ask questions!
107 |
110 |
111 |
Your tools
112 |
125 |
128 |
129 |
What else is needed?
130 |
131 | A text editor
A terminal window
134 |
137 |
138 |
Problem definition
139 |
Given a web site, we want to scrape it and find important web pages.
140 |
This involves a lot of figuring stuff out!
141 |
142 | Learn Haskell
Download one web page
Extract links from a page, so we can find more pages to download
Once we're done, compute which ones are important
Make it all fast?
148 |
151 |
152 |
Let's get started!
153 |
Create a file named Hello.hs and give it the following contents:
154 |
main = putStrLn "hello, world!"
155 |
The suffix .hs is the standard for Haskell source files.
156 |
File names start with a capital letter, and everyone uses CamelCase.
157 |
160 |
161 |
Building it
162 |
This command will look for Hello.hs in the current directory, and compile it:
163 |
ghc --make Hello
164 |
165 |
The generated executable will be named Hello (Hello.exe on Windows).
166 |
167 | - That
--make option up there tells GHC to automatically deal with dependencies on source files and packages.
168 |
169 |
172 |
173 |
Checking in
174 |
Is everyone able to build and run their Hello executable?
175 |
178 |
179 |
Something a little more convenient
180 |
It's nice to have fast, native code at our fingertips.
181 |
But when I'm working, I expect a few things:
182 |
186 |
For these circumstances, a full compiler is a bit slow.
187 |
Instead, I often use the interactive interpreter, ghci.
188 |
191 |
192 |
Let's start GHCi
193 |
Easily done:
194 |
ghci
195 |
196 |
It will display a startup banner, followed by a prompt:
197 |
Prelude>
198 |
199 |
This default prompt tells us which modules are available to play with.
200 |
203 |
204 |
Play around
205 |
The ghci interpreter evaluates expressions interactively.
206 |
Try it out:
207 |
2 + 2
208 |
123456781234567812345678 * 87654321876543
209 |
"foo" ++ "bar"
210 |
(That ++ is the "append" operator.)
211 |
214 |
215 |
Directives
216 |
All interpreter directives start with a ":" character.
217 |
Let's load our source file into ghci:
218 |
:load Hello.hs
219 |
220 |
Now the ghci prompt changes:
221 |
*Main>
222 |
223 |
226 |
227 |
Running our code in ghci
228 |
We defined a function named main, so let's invoke it:
229 |
main
230 |
231 |
Did that work for you?
232 |
What about this?
233 |
putStrLn "hi mom!"
234 |
237 |
238 |
A few more useful directives
239 |
Remember, all ghci directives start with a ":".
240 |
241 | :help tells us what directives are available.
:reload reloads the file we last :loaded.
:edit launches our text editor on the file you most recently :loaded. (Does not automatically :reload.)
:quit exits back to the shell.
246 |
249 |
250 |
Final ghci efficiency tips
251 |
We can abbreviate directives:
252 |
253 | :e is treated as :edit
:r is :reload
256 |
We also have command line history and editing.
257 |
258 | On Unix, compatible with readline.
On Windows, the same as cmd.exe.
261 |
264 |
265 |
Getting used to the cycle
266 |
Use :edit or your text editor to change the "hello" text.
267 |
Use :reload to reload the source file.
268 |
Test out your redefinition of main.
269 |
270 | - For practice, hit the "up arrow" key to cycle through your command history until you get back to the last time you typed
main.
271 |
272 |
275 |
276 |
Lists and strings
277 |
[1,2,3,4]
278 |
['h','e','l','l','o']
279 |
Double quotes are just syntactic sugar for the longer form:
280 |
"hello"
281 |
What does this print?
282 |
"foo" == ['f','o','o']
283 |
286 |
287 |
Calling functions: 1
288 |
We use white space to separate a function from its argument:
289 |
head "foo"
290 |
head [1,2,3]
291 |
tail [1,2,3]
292 |
295 |
296 |
Calling functions: 2
297 |
If a function takes multiple arguments, we separate them with white space:
298 |
min 3 4
299 |
If an argument is a compound expression, wrap it in parentheses:
300 |
compare (3+5) (2+7)
301 |
max (min 3 4) 5
302 |
305 |
306 |
Quick exercises: 1
307 |
Use ghci as a calculator.
308 |
The ** operator performs exponentiation.
309 |
310 | - If I invest 5 quatloos at 3% compound interest per annum, how many quatloos will I have after 10 years?
311 |
312 |
315 |
316 |
Quick exercises: 2
317 |
The notation ['a'..'z'] generates a list from start to end, inclusive.
318 |
The sum function adds the elements of a list.
319 |
320 | - What is the sum of the numbers between 9 and 250, inclusive, minus 2?
321 |
322 |
325 |
326 |
Quick exercises: 3
327 |
The show function renders a value as a string. Try it!
328 |
show (1 == 2)
329 |
The length function tells us how many elements are in a list.
330 |
length [1,2,3]
331 |
332 | - How many digits are in the product of all numbers between 0xBE and 0xEF, inclusive?
333 |
334 |
337 |
338 |
Defining a function
339 |
It is pretty simple to define a new function.
340 |
Open up your text editor, create a new file with a .hs extension, and get writing!
341 |
isOdd x = (rem x 2) == 1
342 |
343 | We start with the name of the function.
Next come the names we want to give its parameter(s), separated by white space.
After those come a single = character, with the body of the function following.
347 |
Load your source file into ghci and give myOdd a try.
348 |
351 |
352 |
Making life more interesting
353 |
Now we can define very simple functions, but we're missing some important building blocks for more fun.
354 |
So let's get to it!
355 |
358 |
359 |
Conditional execution
360 |
Q: What does the familiar if look like in Haskell?
361 |
A: Familiar!
362 |
gcd a b = if b == 0
then a
else gcd b (rem a b)
363 |
We have the following elements:
364 |
369 |
372 |
373 |
Finally! A tiny bit about types
374 |
The two possible results of an if expression must have the same type.
375 |
If then evaluates to a String, well else must too!
376 |
For instance, this makes no sense:
377 |
if True
then 3.14
else "wombat"
378 |
We are forbidden from writing ill-typed expressions like this.
379 |
382 |
383 |
What about else?
384 |
In imperative languages, we can usually leave out the else clause after an if.
385 |
Not so in Haskell.
386 |
Why does this make sense for imperative languages, but not Haskell?
387 |
390 |
391 |
A nearly trivial exercise
392 |
Write a function that appends ", world" to its argument if the argument is "hello", or just returns its argument unmodified otherwise.
393 |
394 | - Remember, the "append" function is an operator named
++.
395 |
396 |
399 |
400 |
Lists in Haskell
401 |
We already know what a list looks like in Haskell:
402 |
[1,2,3]
403 |
And of course there's the syntactic sugar for strings:
404 |
"foo" == ['f','o','o']
405 |
But is this everything there is to know?
406 |
409 |
410 |
List constructors
411 |
Supposing we want to construct a list from first principles.
412 |
413 | We write the empty list as [].
Given an existing list, we can add another element to the front of the list using the : operator.
416 |
419 |
420 |
Type this into ghci
421 |
Add an element to an empty list:
422 |
1 : []
423 |
426 |
427 |
From single-element lists onwards
428 |
What about extending that list?
429 |
2 : (1 : [])
430 |
You're probably guessing now that [2,1] is syntactic sugar for 2:(1:[]). And you're right!
431 |
What is the result of this expression?
432 |
5 : 8 : [] == [5,8]
433 |
436 |
437 |
Constructors
438 |
We refer to [] and : as constructors, because we use them to construct lists.
439 |
When you create a list, the Haskell runtime has to remember which constructors you used, and where.
440 |
So the value [5,8] is represented as:
441 |
442 | A : constructor, with 5 as its first parameter, and as its second ...
Another : constructor, this time with 8 as its first parameter, and now as its second ...
A [] constructor
446 |
449 |
450 |
What did we see?
451 |
Depending on your background, I bet you're thinking something like this:
452 |
456 |
Right on.
457 |
460 |
461 |
Why do we care about constructors?
462 |
Of course Haskell has to remember what a list is constructed of.
463 |
It also lets us inspect a list, to see which constructors were used.
464 |
How do we do this?
465 |
import Data.Char
isCapitalized name
= case name of
(first:rest) -> isUpper first
[] -> False
466 |
469 |
470 |
Welcome to the case expression
471 |
A case expression allows us to inspect a structure to see how it was constructed.
472 |
isCapitalized name
= case name of
[] -> False
(first:rest) -> isUpper first
473 |
474 | In between case and of is the expression we are inspecting.
If the constructor used was the empty-list constructor [], then clearly the name we're inspecting is empty, hence not capitalized.
477 |
If the constructor used was the "add to the front" : operator, then things get more interesting.
478 |
479 | Whatever was the first parameter of the : constructor is bound to the name first.
The second parameter of the : constructor (i.e. everything in the list after the first element) is bound to the name rest.
The expression following the -> is evaluated with these values.
483 |
486 |
487 |
Pattern matching
488 |
The case expression performs what we call pattern matching.
489 |
490 | Patterns are checked from top to bottom.
As soon as a match is found, its right hand side (after the ->) is used as the result of the entire case expression.
If no match succeeds, an exception is thrown.
494 |
497 |
498 |
A worked example
499 |
Let's step through the machinery of what happens if we evaluate this expression.
500 |
isCapitalized "Ann"
501 |
504 |
505 |
Whew! A few exercises!
506 |
Finally! We can write slightly more complex functions.
507 |
Now that you can inspect the front of a list, you should be able to process an entire list recursively.
508 |
First, please write a function named myLength that computes the number of elements in a list.
509 |
Next, write a function named countCaps that calculates the number of capital letters in a string.
510 |
countCaps "Monkey Butter" == 2
511 |
514 |
515 |
Counting capital letters
516 |
Wow, that countCaps function was a pain, right?
517 |
Here's my definition that uses only the machinery we've learned so far:
518 |
countCaps string =
case string of
[] -> 0
(x:xs) -> if isUpper x
then 1 + countCaps xs
else countCaps xs
519 |
522 |
523 |
Huh.
524 |
I thought Haskell was all about concision!?
525 |
528 |
529 |
Conciseness 1: top-level pattern matching
530 |
countCaps [] = 0
countCaps (x:xs) =
if isUpper x
then 1 + countCaps xs
else countCaps xs
531 |
We can define a function as a series of equations, each containing a pattern match.
532 |
This is nice syntactic sugar for case.
533 |
536 |
537 |
Conciseness 2: guards
538 |
countCaps [] = 0
countCaps (x:xs)
| isUpper x = 1 + countCaps xs
| otherwise = countCaps xs
539 |
After each | is a guard.
540 |
541 | If a pattern matches, we evaluate each Boolean guard expression from top to bottom.
When one succeeds, we evaluate the RHS as the body of the function.
544 |
(Yes, patterns in a case can have guards too.)
545 |
548 |
549 |
Before
550 |
Like the original version, but with use of case stripped out:
551 |
countCaps xs =
if null xs
then 0
else if isUpper (head xs)
then 1 + countCaps (tail xs)
else countCaps (tail xs)
552 |
555 |
556 |
After
557 |
Both shorter and easier to follow:
558 |
countCaps [] = 0
countCaps (x:xs)
| isUpper x = 1 + countCaps xs
| otherwise = countCaps xs
559 |
562 |
563 |
Another approach
564 |
Write a new version of countCaps as follows:
565 |
566 | Write a function that goes through a list, and which generates a new list that contains only its capital letters.
Use length to count the number of elements.
569 |
This should give the same result as your first function. Right?
570 |
573 |
574 |
A change of specification
575 |
Suppose we want to count the number of lowercase letters in a string.
576 |
This seems almost the same as our function for counting uppercase letters.
577 |
What can we do with this observation?
578 |
581 |
582 |
Higher order functions
583 |
Higher order function: a function that accepts another function as a parameter.
584 |
filter pred [] = []
filter pred (x:xs)
| pred x = x : filter pred xs
| otherwise = filter pred xs
585 |
How can we use this to define countLowerCase?
586 |
589 |
590 |
Data in, data out
591 |
By now, we've seen several definitions like this:
592 |
countLowerCase string =
length (filter isLower string)
593 |
This is a recurring pattern:
594 |
595 | A function of one argument
It's being fed the result of ...
... another function of one argument
599 |
602 |
603 |
Function composition
604 |
Haskell doesn't limit us to giving functions alphanumeric names.
605 |
Here, we define a function named simply ".", which we can use as an operator:
606 |
(f . g) x = f (g x)
607 |
How to use this?
608 |
countLowerCase = length . filter isLower
609 |
612 |
613 |
Understanding composition
614 |
If that seemed hard to follow, let's make it clearer.
615 |
We'll plug the arguments into the RHS of our function definition:
616 |
(f . g) x = f (g x)
617 |
We had length as the first argument to ".", and filter isLower as the second:
618 |
(length . filter isLower) x
= length (filter isLower x)
619 |
622 |
623 |
Local variables
624 |
Inside an expression, we can introduce new variables using let.
625 |
let x = 2
y = 4
in x + y
626 |
630 |
633 |
634 |
White space
635 |
Haskell is sensitive to white space!
636 |
637 | A top-level definition starts in the leftmost column.
After the beginning of a definition, if the next non-empty line is indented further, it is treated as a continuation of that definition.
Never use tabs in your source files.
641 |
644 |
645 |
White space and local variables
646 |
If you're defining local variables, they must start in the same column.
647 |
This is good:
648 |
let x = 2
y = 4
in x + y
649 |
But this will lead to a compiler error:
650 |
let x = 2
y = 4
in x + y
651 |
654 |
655 |
Composition exercise
656 |
Using function composition wherever you can, write a function that accepts a string and returns a new string containing only the words that begin with vowels.
657 |
658 | - You'll want to play with the
words and unwords functions before you start.
659 |
660 |
Example:
661 |
disemvowel "I think, therefore I am."
== "I I am."
662 |
665 |
666 |
My solution
667 |
Here's how I wrote disemvowel:
668 |
disemvowel =
let isVowel c = toLower c `elem` "aeiou"
in unwords . filter (isVowel . head) . words
669 |
Does this remind you of a Unix shell pipeline, only right-to-left?
670 |
673 |
674 |
Problem definition, once again
675 |
Given a web site, we want to scrape it and find important web pages.
676 |
We're now Haskell experts, right?
677 |
678 | - Download one web page
679 |
680 |
683 |
684 |
Let's download a web page!
685 |
We'd really like to rely on a library to download a web page for us.
686 |
At times like this, there's a very handy central repository of open source Haskell software:
687 |
691 |
Go there now!
692 |
Click on the Packages link at the top of the page to browse packages.
693 |
Alas, the list is overwhelmingly long, but we can find libraries for all kinds of tasks if we're patient.
694 |
Are we patient?
695 |
698 |
699 |
Ugh!
700 |
Scrolling through thousands of libraries is hard - surely there's a better way?
701 |
Enter the cabal command!
702 |
Run this command in a terminal window:
703 |
cabal update
704 |
705 |
This downloads the latest index of all software on Hackage.
706 |
With the index updated, we can search it:
707 |
cabal list http
708 |
709 |
That still gives us 20+ packages to comb through, but at least it's better than the 3,400 on the Packages web page.
710 |
713 |
714 |
Short-cutting the search
715 |
The best HTTP client library is named http-enumerator.
716 |
We can read about it online:
717 |
720 |
That landing page for a package is intimidating, but look towards the bottom, at the section labeled "Modules".
721 |
What do you see?
722 |
725 |
726 |
Installing a package
727 |
Before we can use http-enumerator, we must install it.
728 |
To install the http-enumerator package, we just issue a single command:
729 |
cabal install http-enumerator
730 |
731 |
This command figures out all the other libraries that http-enumerator depends on, and downloads, compiles, and installs the whole lot.
732 |
Expect it to take a few minutes and print a lot of output.
733 |
736 |
737 |
Reading docs: packages and modules
738 |
While we're waiting for the http-enumerator package and all of its dependencies to install, let's try to figure out how we should use it.
739 |
Remember the link to API documentation at the end of the package's web page? Click through to the API docs.
740 |
An API page begins with a title that looks something like this:
741 |
Network.HTTP.Enumerator
742 |
743 |
This is the name of a module.
744 |
A module is a collection of related code.
745 |
A package is a collection of related modules.
746 |
(This will sound familiar if you know Python.)
747 |
750 |
751 |
Reading docs: the rest
752 |
After the initial blurb, a module's docs consists of type signatures and descriptions.
753 |
Here is a really simple type signature:
754 |
foo :: String
755 |
756 |
How the heck do we read this?
757 |
The name of the thing being defined comes before the :: characters.
758 |
Its type follows after the ::.
759 |
This means "the value named foo has the type String".
760 |
763 |
764 |
Haskell's type system
765 |
Up until now, we have not bothered talking about types or type signatures.
766 |
Every expression and value in Haskell has a single type.
767 |
Those types can almost always be inferred automatically by the compiler or interpreter.
768 |
771 |
772 |
The most common basic types
773 |
774 | Bool
Int
Char
Double
779 |
782 |
783 |
A function signature
784 |
Here's another type signature:
785 |
words :: String -> [String]
786 |
787 |
Here we see a new symbol, ->, which means "this is a function".
788 |
The type after the last -> is the return type of the function.
789 |
All of its predecessors are argument types.
790 |
So this is a function that takes one String argument, and returns... what?
791 |
794 |
795 |
List notation
796 |
The notation [a] means "a list of values, all of some type a".
797 |
So [String] means "a list of values, all of type String".
798 |
801 |
802 |
Type synonyms
803 |
What's a String?
804 |
805 | - It's not special, just a synonym for
[Char], i.e. "a list of Char".
806 |
807 |
We can introduce new synonyms of our own.
808 |
type Dollars = Int
809 |
A type synonym can be handy for documenting an intended use for an existing type.
810 |
813 |
814 |
Words
815 |
words :: String -> [String]
816 |
817 |
We can now read that this function accepts a string as argument, and returns a list of strings.
818 |
From reading its name and type signature, can you guess what words might do?
819 |
822 |
823 |
Another signature
824 |
Tell me about this signature:
825 |
mystery :: [String] -> String
826 |
827 |
What are some reasonable possible behaviours for this function?
828 |
831 |
832 |
Reading real-world docs
833 |
Here is the very first signature from http-enumerator:
834 |
simpleHttp
835 | :: (MonadIO m, Failure HttpException m) =>
836 | String -> m ByteString
837 |
838 |
This is more complex! How the heck do we read it?
839 |
The bits between :: and '=>' are constraints on where we can use simpleHttp - but let's ignore constraints for now.
840 |
841 | - Important: it's often safe to gloss over things we don't (yet) understand.
842 |
843 |
We'll also ignore that mysterious lowercase m for a bit.
844 |
What can we tell about this function?
845 |
848 |
849 |
ByteString
850 |
A ByteString is a blob of binary data.
851 |
Unlike String, it is not represented as a list, but as a packed array.
852 |
However, it contains binary bytes, not text!
853 |
854 | - Don't use
ByteString for working with data that you have to manipulate as text.
855 |
856 |
859 |
860 |
Let's play in ghci!
861 |
Does everyone have http-enumerator installed now?
862 |
Fire up ghci, and let's play with the module:
863 |
import Network.HTTP.Enumerator
864 |
865 |
Notice that after we type this, the prompt changes:
866 |
Prelude Network.HTTP.Enumerator>
867 |
868 |
This tells us that the module has loaded and is available.
869 |
872 |
873 |
Wait! Are you on Windows?
874 |
On Windows, we have to set up Winsock before any networking will work.
875 |
First, let's load the lowest-level networking module:
876 |
import Network.Socket
877 |
878 |
And here's how we initialize Winsock:
879 |
withSocketsDo (return ())
880 |
881 |
(It's harmless to do this on Unix.)
882 |
885 |
886 |
With that out of the way ...
887 |
Finally - let's load a web page!
888 |
simpleHttp "http://example.com/"
889 |
890 |
Did that just print a ton of HTML in the terminal window? All right!
891 |
894 |
895 |
From binary to text
896 |
Now we have a ByteString, which we need to turn into text for manipulating.
897 |
Let's cheat, and assume that all web pages are encoded in UTF-8.
898 |
901 |
902 |
Pure code
903 |
So far, all of the code we have written has been "pure".
904 |
905 | The behaviour of all of our functions has depended only on their inputs.
All of our data is immutable.
There's thus no way to change a global variable and modify the behaviour of a function.
909 |
912 |
913 |
Impure code
914 |
And yet ... somehow we downloaded a web page!
915 |
916 | - Web pages clearly are not pure.
917 |
918 |
So can we write code like this?
919 |
length (simpleHttp "http://x.org/")
920 |
NO.
921 |
The type system segregates code that must be pure from code that may have side effects ("impure" code).
922 |
925 |
926 |
Are we stuck?
927 |
Well, let's look at a simpler example than simpleHttp.
928 |
Type this in ghci:
929 |
:type readFile
930 |
931 |
This will tell us what the type of readFile is.
932 |
935 |
936 |
IO
937 |
The :type directive should print something like this:
938 |
readFile :: FilePath -> IO String
939 |
Notice that IO on the result type?
940 |
It means "this function may have side effects".
941 |
We often refer to impure functions, with IO in the result type, as actions.
942 |
943 | - This helps to distinguish them from pure functions.
944 |
945 |
948 |
949 |
Mixing IO and other stuff
950 |
The type system keeps track of which functions have IO in their types, and keeps us honest.
951 |
We can still mix pure and impure code in a natural way:
952 |
charCount fileName = do
contents <- readFile fileName
return (length contents)
953 |
956 |
957 |
"do" notation
958 |
Critical to what we just saw was the do keyword at the beginning of the function definition.
959 |
This introduces a series of IO actions, one per line.
960 |
963 |
964 |
Capturing the results of impure code
965 |
To capture the result of an IO action, we use <- instead of =.
966 |
contents <- readFile fileName
967 |
The result (contents) is pure - it does not have the IO type.
968 |
This is how we supply pure code with data returned from impure code.
969 |
972 |
973 |
The "return" action
974 |
This is not the return type you're used to!
975 |
It takes a pure value (without IO in its type), and wraps it with the IO type.
976 |
Pure code can't call impure code, but it can thread data back into the impure world using return.
977 |
980 |
981 |
Haskell programs and IO
982 |
When you write a Haskell program, its entry point must be named main.
983 |
The type of main must be:
984 |
main :: IO ()
985 |
() is named "unit", and means more or less the same thing as void in C or Java.
986 |
What this means is that all Haskell programs are impure!
987 |
990 |
991 |
Binary to text
992 |
Remember we were planning to cheat earlier?
993 |
We had this:
994 |
simpleHttp :: String -> IO ByteString
995 |
We need something whose result is an IO String instead.
996 |
How should that look?
997 |
1000 |
1001 |
UTF-8 conversion
1002 |
To do the conversion, let's grab a package named utf8-string.
1003 |
cabal install utf8-string
1004 |
1005 |
That contains a package named Data.ByteString.Lazy.UTF8.
1006 |
import Data.ByteString.Lazy.UTF8
1007 |
It defines a function named toString:
1008 |
toString :: ByteString -> String
1009 |
1012 |
1013 |
UTF-8 conversion exercise
1014 |
Write an action that downloads a URL and converts it from a ByteString to a String using toString.
1015 |
Write a type signature for the action.
1016 |
1017 | Haskell definitions usually don't require type signatures.
Nevertheless, we write them for documentation on almost all top-level definitions.
1020 |
1023 |
1024 |
Downloading and saving a web page
1025 |
Use your download function to save a local copy of the page you just wrote.
1026 |
saveAs :: String -> Int -> IO ()
1027 |
For simplicity, let's save the local files as names containing numbers:
1028 |
makeFileName :: Int -> FilePath
makeFileName k = "download-" ++ show k ++ ".html"
1029 |
To save a local copy of a file, you'll need the writeFile action.
1030 |
1033 |
1034 |
Shoveling through HTML
1035 |
Two truisms:
1036 |
1040 |
So! Let's use another library.
1041 |
cabal install tagsoup
1042 |
1043 |
The tagsoup package can parse arbitrarily messy HTML.
1044 |
It will feed us a list of events, like a SAX parser.
1045 |
1048 |
1049 |
Dealing with problems
1050 |
Try this:
1051 |
head [1]
1052 |
Now try this:
1053 |
head []
1054 |
1057 |
1058 |
Oops
1059 |
If we pass an empty list, the head function throws an exception.
1060 |
Suppose we need a version of head that will not throw an exception.
1061 |
safeHead :: [a] -> ????
1062 |
What should the ???? be?
1063 |
Let's invent something.
1064 |
safeHead (x:xs) = Some x
safeHead [] = None
1065 |
1068 |
1069 |
Some? None?
1070 |
1074 |
To bring these constructors into existence, we need to declare a new type.
1075 |
data Perhaps a = Some a
| None
1076 |
The | character separates the constructors. We can read it as:
1077 |
1078 | The Perhaps type has two constructors:
Some followed by a single argument
or None with no arguments
1082 |
1085 |
1086 |
Maybe
1087 |
Actually, Haskell already has a Perhaps type.
1088 |
data Maybe a = Just a
| Nothing
1089 |
The a is a type parameter, meaning that when we write this type, we have to supply another type as a parameter:
1090 |
1091 | Maybe Int
Maybe String
1094 |
1097 |
1098 |
Using constructors
1099 |
If we want to construct a Maybe Int using the Just constructor, we must pass it an Int.
1100 |
Just 1 :: Maybe Int
Nothing :: Maybe Int
1101 |
This will not work, because the types don't match:
1102 |
Just [1] :: Maybe String
1103 |
1106 |
1107 |
Pattern matching over constructors
1108 |
We can pattern match over the constructors for Maybe just as we did for lists.
1109 |
case foo of
Just x -> x
Nothing -> bar
1110 |
1113 |
1114 |
Tags
1115 |
The tagsoup package defines the following type:
1116 |
data Tag = TagOpen String [Attribute]
| TagClose String
| TagText String
| TagComment String
| TagWarning String
| TagPosition Row Column
1117 |
What do you think the constructors mean?
1118 |
1121 |
1122 |
Pattern matching on a Tag
1123 |
Suppose we want to write a predicate that will tell is if a Tag is an opening tag.
1124 |
1128 |
1131 |
1132 |
Don't care!
1133 |
Our first body looked like this:
1134 |
isOpenTag (TagOpen x y) = True
isOpenTag (TagClose x) = False
isOpenTag (TagText x) = False
isOpenTag (TagComment x) = False
isOpenTag (TagWarning x) = False
isOpenTag (TagPosition x y) = False
1135 |
Concise, but ugly.
1136 |
1140 |
1143 |
1144 |
The wild card pattern
1145 |
We can write "I don't care what this pattern or variable is" using the "_" character.
1146 |
isOpenTag (TagOpen _ _) = True
isOpenTag _ = False
1147 |
The wild card pattern always matches.
1148 |
1149 | Since we don't care about x or y, we can state that explicitly using _.
Since we don't care about any constructor except TagOpen, we can match all the others using _.
1152 |
1155 |
1156 |
Just a quick question
1157 |
Why don't we write the function like this?
1158 |
isOpenTag _ = False
isOpenTag (TagOpen _ _) = True
1159 |
1162 |
1163 |
Extracting links from a web page
1164 |
Suppose we have a page in memory already.
1165 |
1166 | Browse the tagsoup docs, in the Text.HTML.TagSoup module.
Find a function that will parse a web page into a series of tags.
1169 |
1172 |
1173 |
Let's use it!
1174 |
processPage url = do
page <- download url
return (parseTags page)
1175 |
1178 |
1179 |
Tidying tags up
1180 |
Parsed tags can contain a mixture of tag names.
1181 |
<A HREF="...">
1182 |
1183 |
<a hrEF="...">
1184 |
1185 |
1186 | - Find a
tagsoup function that will turn tag names and attributes to lower case.
1187 |
1188 |
1191 |
1192 |
Canonical tags
1193 |
Let's use our function to clean up the result of parseTags.
1194 |
processPage url = do
page <- download url
return
(canonicalizeTags
(parseTags page))
1195 |
1198 |
1199 |
Extracting links
1200 |
We only care about open tags that are links, so <a> tags.
1201 |
1202 | How would we write the type of a function that will indicate whether a Tag is an open tag with the correct name?
How would we use this function to extract only the open tags from a list of parsed tags?
1205 |
1208 |
1209 |
Whee!
1210 |
This cascade is getting a bit ridiculous.
1211 |
processPage url = do
page <- download url
return
(filter (isTagOpenName "a")
(canonicalizeTags
(parseTags page)))
1212 |
Two observations:
1213 |
1217 |
1220 |
1221 |
A rewriting exercise
1222 |
Take this function and split it into pure and impure parts.
1223 |
Write the pure part using function composition.
1224 |
processPage url = do
page <- download url
return
(filter (isTagOpenName "a")
(canonicalizeTags
(parseTags page)))
1225 |
1228 |
1229 |
My solution
1230 |
processPage url = do
page <- download url
return (process page)
process =
filter (isTagOpenName "a") .
canonicalizeTags .
parseTags
1231 |
1234 |
1235 |
More stuff to filter out
1236 |
Let's skip nofollow links.
1237 |
We want to get the "rel" attribute of a tag.
1238 |
1239 | - Find a function that extracts an attribute from a tag.
1240 |
1241 |
1244 |
1245 |
No following
1246 |
nofollow tag = fromAttrib "rel" tag == "nofollow"
1247 |
process =
filter (not . nofollow) .
filter (isTagOpenName "a") .
canonicalizeTags .
parseTags
1248 |
1251 |
1252 |
We have a list of <a> tags
1253 |
How would we extract the "href" attribute from every element of the list?
1254 |
1257 |
1258 |
Only non-empty <a href> tags
1259 |
process =
filter (not . null) .
map (fromAttrib "href") .
filter (not . nofollow) .
filter (isTagOpenName "a") .
canonicalizeTags .
parseTags
1260 |
1263 |
1264 |
Canonical URLs
1265 |
Links can be absolute, relative, or invalid garbage, and we only want valid-looking absolute links.
1266 |
To properly create an absolute link, we need to know the absolute URL of the page we're looking at.
1267 |
canonicalizeLink :: String -> String -> Maybe String
1268 |
1271 |
1272 |
Working with URIs
1273 |
The Network.URI package contains some functions we might find handy.
1274 |
parseURI :: String -> Maybe URI
parseURIReference :: String -> Maybe URI
uriToString id "" :: URI -> String
nonStrictRelativeTo :: URI -> URI -> Maybe URI
1275 |
1278 |
1279 |
A monster of indentation
1280 |
This is really hard to read!
1281 |
import Network.URI
canon :: String -> String -> Maybe String
canon referer path =
case parseURI referer of
Nothing -> Nothing
Just r ->
case parseURIReference path of
Nothing -> Nothing
Just p ->
case nonStrictRelativeTo p r of
Nothing -> Nothing
Just u ->
Just (uriToString id u "")
1282 |
Surely there's a better way.
1283 |
1286 |
1287 |
Stair stepping
1288 |
Notice that that function was a series of case inspections of Maybe values?
1289 |
Suppose we had a function that accepted a normal value, and returned a Maybe value.
1290 |
a -> Maybe b
1291 |
And suppose we had a concise syntax for writing an anonymous function.
1292 |
\a -> "hi mom! " ++ a
1293 |
The \ is pronounced "lambda".
1294 |
1297 |
1298 |
Observation
1299 |
The case analysis is quite verbose. Suppose we had a function that performed it, and called another function if our value was Just.
1300 |
bind :: Maybe a -> (a -> Maybe b) -> Maybe b
bind Nothing _ = Nothing
bind (Just value) action = action value
1301 |
1304 |
1305 |
Using bind
1306 |
How could we use this?
1307 |
canon1 referer path =
parseURI referer `bind`
\r -> parseURIReference path `bind`
\p -> nonStrictRelativeTo p r `bind`
\u -> Just (uriToString id u "")
1308 |
If we enclose a function name in backticks, we can use the function as an infix operator.
1309 |
1312 |
1313 |
Reformatting the code
1314 |
canon referer path =
parseURI referer `bind` \r ->
parseURIReference path `bind` \p ->
nonStrictRelativeTo p r `bind` \u ->
Just (uriToString id u "")
1315 |
1318 |
1319 |
A built-in name for bind
1320 |
The >>= operator is a more general version of our bind function.
1321 |
canon referer path =
parseURI referer >>= \r ->
parseURIReference path >>= \p ->
nonStrictRelativeTo p r >>= \u ->
Just (uriToString id u "")
1322 |
1325 |
1326 |
Using syntactic sugar
1327 |
Here's some tidier syntax that should look familiar.
1328 |
canonicalize :: String -> String -> Maybe String
canonicalize referer path = do
r <- parseURI referer
p <- parseURIReference path
u <- nonStrictRelativeTo p r
return (uriToString id u "")
1329 |
1332 |
1333 |
Nearly there
1334 |
process url =
map (canonicalize url) .
filter (not . null) .
map (fromAttrib "href") .
filter (\t -> fromAttrib "rel" t /= "nofollow") .
filter (isTagOpenName "a") .
canonicalizeTags .
parseTags
1335 |
One awkward thing: what is the type of this function?
1336 |
1339 |
1340 |
From [Maybe a] to [a]
1341 |
Go to this web site:
1342 |
1345 |
Type this into the search box:
1346 |
[Maybe a] -> [a]
1347 |
What does the first result say?
1348 |
1351 |
1352 |
We're there!
1353 |
import Data.Maybe
import Network.URI
links url =
catMaybes .
map (canonicalize url) .
filter (not . null) .
map (fromAttrib "href") .
filter (\t -> fromAttrib "rel" t /= "nofollow") .
filter (isTagOpenName "a") .
canonicalizeTags .
parseTags
1354 |
1357 |
1358 |
From links to spidering
1359 |
If we can download the links from one page, we can easily write a spider to follow those links.
1360 |
To keep things simple, let's set a limit on the number of pages we'll download.
1361 |
What information do we want to generate?
1362 |
What do we need to track along the way?
1363 |
1366 |
1367 |
What we need to track
1368 |
Here's the state we need to maintain:
1369 |
1370 | The number of pages we have downloaded
A collection of pages we have seen links to, but haven't downloaded
A collection of pages and their outbound links
1374 |
1377 |
1378 |
Tracking what we've seen
1379 |
For any given page, we need to remember both it and all the pages it links to.
1380 |
One possibility for associating the two is a tuple:
1381 |
("http://x.org/", ["http://microsoft.com/"])
1382 |
Tuples are useful any time we want mixed-type data without the hassle of creating a new type.
1383 |
Speaking of a new type, here's how we'd define one:
1384 |
data Link = Link String [String]
-- Let's define some accessors, too.
linkFrom (Link url _) = url
linkTo (Link _ links) = links
1385 |
1388 |
1389 |
Avoiding duplication
1390 |
We don't want to visit any URL twice.
1391 |
How do we avoid this?
1392 |
visited url = elem url . map linkTo
1393 |
This function has a problem - what is that problem?
1394 |
1397 |
1398 |
Better performance
1399 |
We really want a structure with a fast lookup operation.
1400 |
What would you use in your language?
1401 |
1404 |
1405 |
Maps and importing
1406 |
In Haskell, we have mutable hash tables, but we don't use them.
1407 |
Instead, we use immutable key-value maps.
1408 |
We must perform fancy module importing tricks because the Data.Map module defines a lot of names that would otherwise overlap with built-in names.
1409 |
This means "only import the name Map from Data.Map":
1410 |
import Data.Map (Map)
1411 |
And this means "import everything from Data.Map, but all those names must be prefixed with Map.":
1412 |
import qualified Data.Map as Map
1413 |
1416 |
1417 |
What use is an immutable data structure?
1418 |
Everyone knows how to add a key and value to a hash table, right?
1419 |
And that seems like a fundamental operation.
1420 |
What do we do with maps?
1421 |
1422 | - Create a new map that is identical to the one we supply, with the requested element added.
1423 |
1424 |
How can this possibly work? Is it efficient?
1425 |
1428 |
1429 |
A fistful of dollars
1430 |
Here's a surprisingly handy built-in operator:
1431 |
f $ x = f x
1432 |
Why is this useful? Because it lets us eliminate parentheses.
1433 |
Before:
1434 |
explode k = error ("failed on " ++ show k)
1435 |
After:
1436 |
explode k = error $ "failed on " ++ show k
1437 |
1440 |
1441 |
Partial application
1442 |
This is annoying to write:
1443 |
increment k = 1 + k
1444 |
Almost as bad:
1445 |
\k -> 1 + k
1446 |
Much handier, and identical:
1447 |
(1+)
1448 |
In fact, this is valid:
1449 |
increment = (1+)
1450 |
1453 |
1454 |
Spidering, in all its glory
1455 |
spider :: Int -> URL -> IO (Map URL [URL])
spider count url0 = go 0 Map.empty (Set.singleton url0)
where
go k seen queue0
| k >= count = return seen
| otherwise =
case Set.minView queue0 of
Nothing -> return seen
Just (url, queue) -> do
page <- download url
let ls = links url page
newSeen = Map.insert url ls seen
notSeen = Set.fromList .
filter (`Map.notMember` newSeen) $ ls
newQueue = queue `Set.union` notSeen
go (k+1) newSeen newQueue
1456 |
1459 |
1460 |
Where do we stand?
1461 |
We can now:
1462 |
1467 |
What remains?
1468 |
1472 |
1475 |
1476 |
Fin
1477 |
At this point, if we have miraculously not run out of time, we're going on a choose-your-own-adventure session in Emacs.
1478 |
Thanks for sticking with the slideshow so far!
1479 |