├── README.rst
├── build_dict.py
├── data
    └── dict.pkl
├── doc
    ├── api-objects.txt
    ├── build_dict-module.html
    ├── build_dict-pysrc.html
    ├── class-tree.html
    ├── class_hierarchy_for_faststemme.gif
    ├── class_hierarchy_for_htmlreader.gif
    ├── class_hierarchy_for_multitag.gif
    ├── class_hierarchy_for_naiverater.gif
    ├── class_hierarchy_for_rater.gif
    ├── class_hierarchy_for_reader.gif
    ├── class_hierarchy_for_simpleread.gif
    ├── class_hierarchy_for_stemmer.gif
    ├── class_hierarchy_for_tag.gif
    ├── class_hierarchy_for_unicoderea.gif
    ├── crarr.png
    ├── epydoc.css
    ├── epydoc.js
    ├── extras-module.html
    ├── extras-pysrc.html
    ├── extras.FastStemmer-class.html
    ├── extras.HTMLReader-class.html
    ├── extras.NaiveRater-class.html
    ├── extras.SimpleReader-class.html
    ├── extras.UnicodeReader-class.html
    ├── frames.html
    ├── help.html
    ├── identifier-index.html
    ├── index.html
    ├── module-tree.html
    ├── redirect.html
    ├── tagger-module.html
    ├── tagger-pysrc.html
    ├── tagger.MultiTag-class.html
    ├── tagger.Rater-class.html
    ├── tagger.Reader-class.html
    ├── tagger.Stemmer-class.html
    ├── tagger.Tag-class.html
    ├── tagger.Tagger-class.html
    ├── toc-build_dict-module.html
    ├── toc-everything.html
    ├── toc-extras-module.html
    ├── toc-tagger-module.html
    └── toc.html
├── extras.py
├── tagger.py
├── test_ui.py
└── tests
    ├── bbc1.txt
    ├── bbc2.txt
    ├── bbc3.txt
    ├── guardian1.txt
    ├── guardian2.txt
    ├── post1.txt
    ├── wikipedia1.txt
    ├── wikipedia2.txt
    └── wikipedia3.txt


/README.rst:
--------------------------------------------------------------------------------
  1 | ======
  2 | tagger
  3 | ======
  4 | 
  5 | Module for extracting tags from text documents.
  6 |                    
  7 | Copyright (C) 2011 by Alessandro Presta
  8 | 
  9 | Configuration
 10 | =============
 11 | 
 12 | Dependencies:
 13 | python2.7, stemming, nltk (optional), lxml (optional), tkinter (optional)
 14 | 
 15 | You can install the stemming package with::
 16 | 
 17 |     $ easy_install stemming
 18 | 
 19 | Usage
 20 | =====
 21 | 
 22 | Tagging a text document from Python::
 23 | 
 24 |     import tagger
 25 |     weights = pickle.load(open('data/dict.pkl', 'rb')) # or your own dictionary
 26 |     myreader = tagger.Reader() # or your own reader class
 27 |     mystemmer = tagger.Stemmer() # or your own stemmer class
 28 |     myrater = tagger.Rater(weights) # or your own... (you got the idea)
 29 |     mytagger = Tagger(myreader, mystemmer, myrater)
 30 |     best_3_tags = mytagger(text_string, 3)
 31 | 
 32 | Running the module as a script::
 33 | 
 34 |     $ ./tagger.py <text document(s) to tag>
 35 | 
 36 | Example::
 37 | 
 38 |     $ ./tagger.py tests/*
 39 |     Loading dictionary... 
 40 |     Tags for  tests/bbc1.txt :
 41 |     ['bin laden', 'obama', 'pakistan', 'killed', 'raid']
 42 |     Tags for  tests/bbc2.txt :
 43 |     ['jo yeates', 'bristol', 'vincent tabak', 'murder', 'strangled']
 44 |     Tags for  tests/bbc3.txt :
 45 |     ['snp', 'party', 'election', 'scottish', 'labour']
 46 |     Tags for  tests/guardian1.txt :
 47 |     ['bin laden', 'al-qaida', 'killed', 'pakistan', 'al-fawwaz']
 48 |     Tags for  tests/guardian2.txt :
 49 |     ['clegg', 'tory', 'lib dem', 'party', 'coalition']
 50 |     Tags for  tests/post1.txt :
 51 |     ['sony', 'stolen', 'playstation network', 'hacker attack', 'lawsuit']
 52 |     Tags for  tests/wikipedia1.txt :
 53 |     ['universe', 'anthropic principle', 'observed', 'cosmological', 'theory']
 54 |     Tags for  tests/wikipedia2.txt :
 55 |     ['beetroot', 'beet', 'betaine', 'blood pressure', 'dietary nitrate']
 56 |     Tags for  tests/wikipedia3.txt :
 57 |     ['the lounge lizards', 'jazz', 'john lurie', 'musical', 'albums']
 58 | 
 59 | A brief explanation
 60 | ===================
 61 | 
 62 | Extracting tags from a text document involves at least three steps: splitting the document into words, grouping together variants of the same word, and ranking them according to their relevance.
 63 | These three tasks are carried out respectively by the **Reader**, **Stemmer** and **Rater** classes, and their work is put together by the **Tagger** class.
 64 | 
 65 | A **Reader** object may accept as input a document in some format, perform some normalisation of the text (such as turning everything into lower case), analyse the structure of the phrases and punctuation, and return a list of words respecting the order in the text, perhaps with some additional information such as which ones look like proper nouns, or are at the end of a phrase.
 66 | A very straightforward way of doing this would be to just match all the words with a regular expression, and this is indeed what the **SimpleReader** class does.
 67 | 
 68 | The **Stemmer** tries to recognise the root of a word, in order to identify slightly different forms. This is already a quite complicated task, and it's clearly language-specific.
 69 | The *stem* module in the NLTK package provides algorithms for many languages
 70 | and integrates nicely with the tagger::
 71 |     
 72 |     import nltk
 73 |     # an English stemmer using Lancaster's algorithm
 74 |     mystemmer = Stemmer(nltk.stem.LancasterStemmer)
 75 |     # an Italian stemmer
 76 |     class MyItalianStemmer(Stemmer):
 77 |         def __init__(self):
 78 |             Stemmer.__init__(self, nltk.stem.ItalianStemmer)
 79 |         def preprocess(self, string):
 80 |             # do something with the string before passing it to nltk's stemmer
 81 | 
 82 | The **Rater** takes the list of words contained in the document, together with any additional information gathered at the previous stages, and returns a list of tags (i.e. words or small units of text) ordered by some idea of "relevance".
 83 | 
 84 | It turns out that just working on the information contained in the document itself is not enough, because it says nothing about the frequency of a term in the language. For this reason, an early "off-line" phase of the algorithm consists in analysing a *corpus* (i.e. a sample of documents written in the same language) to build a dictionary of known words. This is taken care by the **build_dict()** function.
 85 | It is advised to build your own dictionaries, and the **build_dict_from_nltk()** function in the *extras* module enables you to use the corpora included in NLTK::
 86 |     
 87 |     build_dict_from_nltk(output_file, nltk.corpus.brown, 
 88 |                          nltk.corpus.stopwords.words('english'), measure='ICF')
 89 | 
 90 | So far, we may define the relevance of a word as the product of two distinct functions: one that depends on the document itself, and one that depends on the corpus.
 91 | A standard measure in information retrieval is TF-IDF (*term frequency-inverse
 92 | document frequency*): the frequency of the word in the document multiplied by
 93 | the (logarithm of) the inverse of its frequency in the corpus (i.e. the cardinality of the corpus divided by the number of documents where the word is found).
 94 | If we treat the whole corpus as a single document, and count the total occurrences of the term instead, we obtain ICF (*inverse collection frequency*).
 95 | Both of these are implemented in the *build_dict* module, and any other reasonable measure should be fine, provided that it is normalised in the interval [0,1]. The dictionary is passed to the **Rater** object as the *weights* argument in its constructor.
 96 | We might also want to define the first term of the product in a different way, and this is done by overriding the **rate_tags()** method (which by default calculates TF for each word and multiplies it by its weight)::
 97 | 
 98 |     class MyRater(Rater):
 99 |         def rate_tags(self, tags):
100 |             # set each tag's rating as you wish
101 | 
102 | If we were not too picky about the results, these few bits would already make an acceptable tagger.
103 | However, it's a matter of fact that tags formed only by single words are quite limited: while "obama" and "barack obama" are both reasonable tags (and it is quite easy to treat cases like this in order to regard them as equal), having "laden" and "bin" as two separate tags is definitely not acceptable and misleading.
104 | Compare the results on the same document using the **NaiveRater** class (defined in the module *extras*) instead of the standard one.
105 | 
106 | The *multitag_size* parameter in the **Rater**'s constructor defines the maximum number of words that can constitute a tag. Multitags are generated in the **create_multitags()** method; if additional information about the position of a word in the phrase is available (i.e. the **terminal** member of the class **Tag**), this can be done in a more accurate way.
107 | The rating of a **MultiTag** is computed from the ratings of its unit tags.
108 | By default, the **combined_rating()** method uses the geometric mean, with a special treatment of proper nouns if that information is available too (in the **proper** member).
109 | This method can be overridden too, so there is room for experimentation.
110 | 
111 | With a few "common sense" heuristics the results are greatly improved.
112 | The final stage of the default rating algorithm involves discarding redundant tags (i.e. tags that contain or are contained in other, less relevant tags).
113 | 
114 | It should be stressed that the default implementation doesn't make any assumption on the type of document that is being tagged (except for it being written in English) and on the kinds of tags that should be given priority (which sometimes can be a matter of taste or depend on the particular task we are using the tags for).
115 | With some additional assumptions and an accurate treatment of corner cases, the tagger can be tailored to suit the user's needs.
116 | 
117 | This is proof-of-concept software and extensive experimentation is encouraged. The design of the base classes should allow for this, and the few examples in the *extras* module are a good starting point for customising the algorithm.
118 | 


--------------------------------------------------------------------------------
/build_dict.py:
--------------------------------------------------------------------------------
  1 | #!/usr/bin/env python
  2 | 
  3 | # Copyright (C) 2011 by Alessandro Presta
  4 | 
  5 | # Permission is hereby granted, free of charge, to any person obtaining a copy
  6 | # of this software and associated documentation files (the "Software"), to deal
  7 | # in the Software without restriction, including without limitation the rights
  8 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
  9 | # copies of the Software, and to permit persons to whom the Software is
 10 | # furnished to do so, subject to the following conditions:
 11 | 
 12 | # The above copyright notice and this permission notice shall be included in
 13 | # all copies or substantial portions of the Software.
 14 | 
 15 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 16 | # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 17 | # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 18 | # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 19 | # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 20 | # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 21 | # THE SOFTWARE
 22 | 
 23 | 
 24 | '''
 25 | Usage: build_dict.py -o <output file> -s <stopwords file> <list of files>
 26 | '''
 27 | 
 28 | from tagger import Stemmer
 29 | from extras import SimpleReader
 30 | 
 31 | 
 32 | def build_dict(corpus, stopwords=None, measure='IDF'):
 33 |     '''
 34 |     @param corpus:    a list of documents, represented as lists of (stemmed)
 35 |                       words
 36 |     @param stopwords: the list of (stemmed) words that should have zero weight
 37 |     @param measure:   the measure used to compute the weights ('IDF'
 38 |                       i.e. 'inverse document frequency' or 'ICF' i.e.
 39 |                       'inverse collection frequency'; defaults to 'IDF')
 40 | 
 41 |     @returns: a dictionary of weights in the interval [0,1]
 42 |     '''
 43 | 
 44 |     import collections
 45 |     import math
 46 | 
 47 |     dictionary = {}
 48 | 
 49 |     if measure == 'ICF':
 50 |         words = [w for doc in corpus for w in doc]
 51 |         
 52 |         term_count = collections.Counter(words)
 53 |         total_count = float(len(words))
 54 |         scale = math.log(total_count)
 55 |     
 56 |         for w, cnt in term_count.iteritems():
 57 |             dictionary[w] = math.log(total_count / (cnt + 1)) / scale
 58 | 
 59 |     elif measure == 'IDF':
 60 |         corpus_size = float(len(corpus))
 61 |         scale = math.log(corpus_size)
 62 | 
 63 |         term_count = collections.defaultdict(int)
 64 | 
 65 |         for doc in corpus:
 66 |             words = set(doc)
 67 |             for w in words:
 68 |                 term_count[w] += 1
 69 | 
 70 |         for w, cnt in term_count.iteritems():
 71 |             dictionary[w] = math.log(corpus_size / (cnt + 1)) / scale
 72 |             
 73 |     if stopwords:
 74 |         for w in stopwords:
 75 |             dictionary[w] = 0.0
 76 |     
 77 |     return dictionary
 78 | 
 79 | 
 80 | def build_dict_from_files(output_file, corpus_files, stopwords_file=None,
 81 |                           reader=SimpleReader(), stemmer=Stemmer(),
 82 |                           measure='IDF', verbose=False):
 83 |     '''
 84 |     @param output_file:    the binary stream where the dictionary should be
 85 |                            saved
 86 |     @param corpus_files:   a list of streams with words to process
 87 |     @param stopwords_file: a stream containing a list of stopwords
 88 |     @param reader:         the L{Reader} object to be used
 89 |     @param stemmer:        the L{Stemmer} object to be used
 90 |     @param measure:        the measure used to compute the weights ('IDF'
 91 |                            i.e. 'inverse document frequency' or 'ICF' i.e.
 92 |                            'inverse collection frequency'; defaults to 'IDF')
 93 |     @param verbose:        whether information on the progress should be
 94 |                            printed on screen
 95 |     '''
 96 | 
 97 |     import pickle
 98 | 
 99 |     if verbose: print 'Processing corpus...'
100 |     corpus = []
101 |     for doc in corpus_files:
102 |         corpus.append(reader(doc.read()))
103 |     corpus = [[w.stem for w in map(stemmer, doc)] for doc in corpus]
104 | 
105 |     stopwords = None
106 |     if stopwords_file:
107 |         if verbose: print 'Processing stopwords...'
108 |         stopwords = reader(stopwords_file.read())
109 |         stopwords = [w.stem for w in map(stemmer, stopwords)]
110 | 
111 |     if verbose: print 'Building dictionary... '
112 |     dictionary = build_dict(corpus, stopwords, measure)
113 |     pickle.dump(dictionary, output_file, -1) 
114 |     
115 | 
116 | if __name__ == '__main__':
117 | 
118 |     import getopt
119 |     import sys
120 |     
121 |     try:
122 |         options = getopt.getopt(sys.argv[1:], 'o:s:')
123 |         output_file = options[0][0][1]
124 |         stopwords_file = options[0][1][1]
125 |         corpus = options[1]
126 |     except:
127 |         print __doc__
128 |         exit(1)
129 |     
130 |     corpus = [open(doc, 'r') for doc in corpus]
131 |     stopwords_file = open(stopwords_file, 'r')
132 |     output_file = open(output_file, 'wb')
133 | 
134 |     build_dict_from_files(output_file, corpus, stopwords_file, verbose=True)
135 | 
136 |     output_file.close()
137 |     stopwords_file.close()
138 |     for doc in corpus:
139 |         doc.close()
140 |     
141 |                
142 | 
143 | 


--------------------------------------------------------------------------------
/data/dict.pkl:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/kevinmcmahon/tagger/69432971b6af1f2649f2af5b623f156ffeb0d1c3/data/dict.pkl


--------------------------------------------------------------------------------
/doc/api-objects.txt:
--------------------------------------------------------------------------------
 1 | build_dict	build_dict-module.html
 2 | build_dict.build_dict_from_files	build_dict-module.html#build_dict_from_files
 3 | build_dict.build_dict	build_dict-module.html#build_dict
 4 | build_dict.__package__	build_dict-module.html#__package__
 5 | extras	extras-module.html
 6 | extras.__package__	extras-module.html#__package__
 7 | extras.build_dict_from_nltk	extras-module.html#build_dict_from_nltk
 8 | tagger	tagger-module.html
 9 | tagger.__package__	tagger-module.html#__package__
10 | extras.FastStemmer	extras.FastStemmer-class.html
11 | tagger.Stemmer.preprocess	tagger.Stemmer-class.html#preprocess
12 | tagger.Stemmer.__call__	tagger.Stemmer-class.html#__call__
13 | extras.FastStemmer.__init__	extras.FastStemmer-class.html#__init__
14 | tagger.Stemmer.match_contractions	tagger.Stemmer-class.html#match_contractions
15 | extras.HTMLReader	extras.HTMLReader-class.html
16 | tagger.Reader.match_phrases	tagger.Reader-class.html#match_phrases
17 | tagger.Reader.match_words	tagger.Reader-class.html#match_words
18 | tagger.Reader.match_paragraphs	tagger.Reader-class.html#match_paragraphs
19 | tagger.Reader.preprocess	tagger.Reader-class.html#preprocess
20 | extras.HTMLReader.__call__	extras.HTMLReader-class.html#__call__
21 | tagger.Reader.match_apostrophes	tagger.Reader-class.html#match_apostrophes
22 | extras.NaiveRater	extras.NaiveRater-class.html
23 | extras.NaiveRater.__call__	extras.NaiveRater-class.html#__call__
24 | tagger.Rater.create_multitags	tagger.Rater-class.html#create_multitags
25 | tagger.Rater.__init__	tagger.Rater-class.html#__init__
26 | tagger.Rater.rate_tags	tagger.Rater-class.html#rate_tags
27 | extras.SimpleReader	extras.SimpleReader-class.html
28 | tagger.Reader.match_phrases	tagger.Reader-class.html#match_phrases
29 | tagger.Reader.match_words	tagger.Reader-class.html#match_words
30 | tagger.Reader.match_paragraphs	tagger.Reader-class.html#match_paragraphs
31 | tagger.Reader.preprocess	tagger.Reader-class.html#preprocess
32 | extras.SimpleReader.__call__	extras.SimpleReader-class.html#__call__
33 | tagger.Reader.match_apostrophes	tagger.Reader-class.html#match_apostrophes
34 | extras.UnicodeReader	extras.UnicodeReader-class.html
35 | tagger.Reader.match_phrases	tagger.Reader-class.html#match_phrases
36 | tagger.Reader.match_words	tagger.Reader-class.html#match_words
37 | tagger.Reader.match_paragraphs	tagger.Reader-class.html#match_paragraphs
38 | tagger.Reader.preprocess	tagger.Reader-class.html#preprocess
39 | extras.UnicodeReader.__call__	extras.UnicodeReader-class.html#__call__
40 | tagger.Reader.match_apostrophes	tagger.Reader-class.html#match_apostrophes
41 | tagger.MultiTag	tagger.MultiTag-class.html
42 | tagger.MultiTag.combined_rating	tagger.MultiTag-class.html#combined_rating
43 | tagger.Tag.__repr__	tagger.Tag-class.html#__repr__
44 | tagger.Tag.__hash__	tagger.Tag-class.html#__hash__
45 | tagger.Tag.__lt__	tagger.Tag-class.html#__lt__
46 | tagger.Tag.__eq__	tagger.Tag-class.html#__eq__
47 | tagger.MultiTag.__init__	tagger.MultiTag-class.html#__init__
48 | tagger.Rater	tagger.Rater-class.html
49 | tagger.Rater.__call__	tagger.Rater-class.html#__call__
50 | tagger.Rater.create_multitags	tagger.Rater-class.html#create_multitags
51 | tagger.Rater.__init__	tagger.Rater-class.html#__init__
52 | tagger.Rater.rate_tags	tagger.Rater-class.html#rate_tags
53 | tagger.Reader	tagger.Reader-class.html
54 | tagger.Reader.match_phrases	tagger.Reader-class.html#match_phrases
55 | tagger.Reader.match_words	tagger.Reader-class.html#match_words
56 | tagger.Reader.match_paragraphs	tagger.Reader-class.html#match_paragraphs
57 | tagger.Reader.__call__	tagger.Reader-class.html#__call__
58 | tagger.Reader.preprocess	tagger.Reader-class.html#preprocess
59 | tagger.Reader.match_apostrophes	tagger.Reader-class.html#match_apostrophes
60 | tagger.Stemmer	tagger.Stemmer-class.html
61 | tagger.Stemmer.preprocess	tagger.Stemmer-class.html#preprocess
62 | tagger.Stemmer.__call__	tagger.Stemmer-class.html#__call__
63 | tagger.Stemmer.__init__	tagger.Stemmer-class.html#__init__
64 | tagger.Stemmer.match_contractions	tagger.Stemmer-class.html#match_contractions
65 | tagger.Tag	tagger.Tag-class.html
66 | tagger.Tag.__hash__	tagger.Tag-class.html#__hash__
67 | tagger.Tag.__lt__	tagger.Tag-class.html#__lt__
68 | tagger.Tag.__eq__	tagger.Tag-class.html#__eq__
69 | tagger.Tag.__repr__	tagger.Tag-class.html#__repr__
70 | tagger.Tag.__init__	tagger.Tag-class.html#__init__
71 | tagger.Tagger	tagger.Tagger-class.html
72 | tagger.Tagger.__call__	tagger.Tagger-class.html#__call__
73 | tagger.Tagger.__init__	tagger.Tagger-class.html#__init__
74 | 


--------------------------------------------------------------------------------
/doc/class-tree.html:
--------------------------------------------------------------------------------
  1 | <?xml version="1.0" encoding="ascii"?>
  2 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  3 |           "DTD/xhtml1-transitional.dtd">
  4 | <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
  5 | <head>
  6 |   <title>Class Hierarchy</title>
  7 |   <link rel="stylesheet" href="epydoc.css" type="text/css" />
  8 |   <script type="text/javascript" src="epydoc.js"></script>
  9 | </head>
 10 | 
 11 | <body bgcolor="white" text="black" link="blue" vlink="#204080"
 12 |       alink="#204080">
 13 | <!-- ==================== NAVIGATION BAR ==================== -->
 14 | <table class="navbar" border="0" width="100%" cellpadding="0"
 15 |        bgcolor="#a0c0ff" cellspacing="0">
 16 |   <tr valign="middle">
 17 | 
 18 |   <!-- Tree link -->
 19 |       <th bgcolor="#70b0f0" class="navbar-select"
 20 |           >&nbsp;&nbsp;&nbsp;Trees&nbsp;&nbsp;&nbsp;</th>
 21 | 
 22 |   <!-- Index link -->
 23 |       <th>&nbsp;&nbsp;&nbsp;<a
 24 |         href="identifier-index.html">Indices</a>&nbsp;&nbsp;&nbsp;</th>
 25 | 
 26 |   <!-- Help link -->
 27 |       <th>&nbsp;&nbsp;&nbsp;<a
 28 |         href="help.html">Help</a>&nbsp;&nbsp;&nbsp;</th>
 29 | 
 30 |   <!-- Project homepage -->
 31 |       <th class="navbar" align="right" width="100%">
 32 |         <table border="0" cellpadding="0" cellspacing="0">
 33 |           <tr><th class="navbar" align="center"
 34 |             ><a class="navbar" target="_top" href="http://github.com/apresta/tagger">tagger</a></th>
 35 |           </tr></table></th>
 36 |   </tr>
 37 | </table>
 38 | <table width="100%" cellpadding="0" cellspacing="0">
 39 |   <tr valign="top">
 40 |     <td width="100%">&nbsp;</td>
 41 |     <td>
 42 |       <table cellpadding="0" cellspacing="0">
 43 |         <!-- hide/show private -->
 44 |         <tr><td align="right"><span class="options">[<a href="javascript:void(0);" class="privatelink"
 45 |     onclick="toggle_private();">hide&nbsp;private</a>]</span></td></tr>
 46 |         <tr><td align="right"><span class="options"
 47 |             >[<a href="frames.html" target="_top">frames</a
 48 |             >]&nbsp;|&nbsp;<a href="class-tree.html"
 49 |             target="_top">no&nbsp;frames</a>]</span></td></tr>
 50 |       </table>
 51 |     </td>
 52 |   </tr>
 53 | </table>
 54 | <center><b>
 55 |  [ <a href="module-tree.html">Module Hierarchy</a>
 56 |  | <a href="class-tree.html">Class Hierarchy</a> ]
 57 | </b></center><br />
 58 | <h1 class="epydoc">Class Hierarchy</h1>
 59 | <ul class="nomargin-top">
 60 |     <li> <strong class="uidlink"><a href="tagger.Rater-class.html">tagger.Rater</a></strong>:
 61 |       <em class="summary">Class for estimating the relevance of tags</em>
 62 |     <ul>
 63 |     <li> <strong class="uidlink"><a href="extras.NaiveRater-class.html">extras.NaiveRater</a></strong>:
 64 |       <em class="summary">Rater subclass that jusk ranks single-word tags by their frequency 
 65 |         and weight</em>
 66 |     </li>
 67 |     </ul>
 68 |     </li>
 69 |     <li> <strong class="uidlink"><a href="tagger.Reader-class.html">tagger.Reader</a></strong>:
 70 |       <em class="summary">Class for parsing a string of text to obtain tags</em>
 71 |     <ul>
 72 |     <li> <strong class="uidlink"><a href="extras.SimpleReader-class.html">extras.SimpleReader</a></strong>:
 73 |       <em class="summary">Reader subclass that doesn't perform any advanced analysis of the 
 74 |         text</em>
 75 |     </li>
 76 |     <li> <strong class="uidlink"><a href="extras.UnicodeReader-class.html">extras.UnicodeReader</a></strong>:
 77 |       <em class="summary">Reader subclass that converts Unicode strings to a close ASCII 
 78 |         representation</em>
 79 |     <ul>
 80 |     <li> <strong class="uidlink"><a href="extras.HTMLReader-class.html">extras.HTMLReader</a></strong>:
 81 |       <em class="summary">Reader subclass that can parse HTML code from the input</em>
 82 |     </li>
 83 |     </ul>
 84 |     </li>
 85 |     </ul>
 86 |     </li>
 87 |     <li> <strong class="uidlink"><a href="tagger.Stemmer-class.html">tagger.Stemmer</a></strong>:
 88 |       <em class="summary">Class for extracting the stem of a word</em>
 89 |     <ul>
 90 |     <li> <strong class="uidlink"><a href="extras.FastStemmer-class.html">extras.FastStemmer</a></strong>:
 91 |       <em class="summary">Stemmer subclass that uses a much faster, but less correct 
 92 |         algorithm</em>
 93 |     </li>
 94 |     </ul>
 95 |     </li>
 96 |     <li> <strong class="uidlink"><a href="tagger.Tag-class.html">tagger.Tag</a></strong>:
 97 |       <em class="summary">General class for tags (small units of text)</em>
 98 |     <ul>
 99 |     <li> <strong class="uidlink"><a href="tagger.MultiTag-class.html">tagger.MultiTag</a></strong>:
100 |       <em class="summary">Class for aggregates of tags (usually next to each other in the 
101 |         document)</em>
102 |     </li>
103 |     </ul>
104 |     </li>
105 |     <li> <strong class="uidlink"><a href="tagger.Tagger-class.html">tagger.Tagger</a></strong>:
106 |       <em class="summary">Master class for tagging text documents</em>
107 |     </li>
108 | </ul>
109 | <!-- ==================== NAVIGATION BAR ==================== -->
110 | <table class="navbar" border="0" width="100%" cellpadding="0"
111 |        bgcolor="#a0c0ff" cellspacing="0">
112 |   <tr valign="middle">
113 | 
114 |   <!-- Tree link -->
115 |       <th bgcolor="#70b0f0" class="navbar-select"
116 |           >&nbsp;&nbsp;&nbsp;Trees&nbsp;&nbsp;&nbsp;</th>
117 | 
118 |   <!-- Index link -->
119 |       <th>&nbsp;&nbsp;&nbsp;<a
120 |         href="identifier-index.html">Indices</a>&nbsp;&nbsp;&nbsp;</th>
121 | 
122 |   <!-- Help link -->
123 |       <th>&nbsp;&nbsp;&nbsp;<a
124 |         href="help.html">Help</a>&nbsp;&nbsp;&nbsp;</th>
125 | 
126 |   <!-- Project homepage -->
127 |       <th class="navbar" align="right" width="100%">
128 |         <table border="0" cellpadding="0" cellspacing="0">
129 |           <tr><th class="navbar" align="center"
130 |             ><a class="navbar" target="_top" href="http://github.com/apresta/tagger">tagger</a></th>
131 |           </tr></table></th>
132 |   </tr>
133 | </table>
134 | <table border="0" cellpadding="0" cellspacing="0" width="100%%">
135 |   <tr>
136 |     <td align="left" class="footer">
137 |     Generated by Epydoc 3.0.1 on Fri May 13 11:13:02 2011
138 |     </td>
139 |     <td align="right" class="footer">
140 |       <a target="mainFrame" href="http://epydoc.sourceforge.net"
141 |         >http://epydoc.sourceforge.net</a>
142 |     </td>
143 |   </tr>
144 | </table>
145 | 
146 | <script type="text/javascript">
147 |   <!--
148 |   // Private objects are initially displayed (because if
149 |   // javascript is turned off then we want them to be
150 |   // visible); but by default, we want to hide them.  So hide
151 |   // them unless we have a cookie that says to show them.
152 |   checkCookie();
153 |   // -->
154 | </script>
155 | </body>
156 | </html>
157 | 


--------------------------------------------------------------------------------
/doc/class_hierarchy_for_faststemme.gif:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/kevinmcmahon/tagger/69432971b6af1f2649f2af5b623f156ffeb0d1c3/doc/class_hierarchy_for_faststemme.gif


--------------------------------------------------------------------------------
/doc/class_hierarchy_for_htmlreader.gif:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/kevinmcmahon/tagger/69432971b6af1f2649f2af5b623f156ffeb0d1c3/doc/class_hierarchy_for_htmlreader.gif


--------------------------------------------------------------------------------
/doc/class_hierarchy_for_multitag.gif:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/kevinmcmahon/tagger/69432971b6af1f2649f2af5b623f156ffeb0d1c3/doc/class_hierarchy_for_multitag.gif


--------------------------------------------------------------------------------
/doc/class_hierarchy_for_naiverater.gif:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/kevinmcmahon/tagger/69432971b6af1f2649f2af5b623f156ffeb0d1c3/doc/class_hierarchy_for_naiverater.gif


--------------------------------------------------------------------------------
/doc/class_hierarchy_for_rater.gif:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/kevinmcmahon/tagger/69432971b6af1f2649f2af5b623f156ffeb0d1c3/doc/class_hierarchy_for_rater.gif


--------------------------------------------------------------------------------
/doc/class_hierarchy_for_reader.gif:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/kevinmcmahon/tagger/69432971b6af1f2649f2af5b623f156ffeb0d1c3/doc/class_hierarchy_for_reader.gif


--------------------------------------------------------------------------------
/doc/class_hierarchy_for_simpleread.gif:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/kevinmcmahon/tagger/69432971b6af1f2649f2af5b623f156ffeb0d1c3/doc/class_hierarchy_for_simpleread.gif


--------------------------------------------------------------------------------
/doc/class_hierarchy_for_stemmer.gif:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/kevinmcmahon/tagger/69432971b6af1f2649f2af5b623f156ffeb0d1c3/doc/class_hierarchy_for_stemmer.gif


--------------------------------------------------------------------------------
/doc/class_hierarchy_for_tag.gif:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/kevinmcmahon/tagger/69432971b6af1f2649f2af5b623f156ffeb0d1c3/doc/class_hierarchy_for_tag.gif


--------------------------------------------------------------------------------
/doc/class_hierarchy_for_unicoderea.gif:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/kevinmcmahon/tagger/69432971b6af1f2649f2af5b623f156ffeb0d1c3/doc/class_hierarchy_for_unicoderea.gif


--------------------------------------------------------------------------------
/doc/crarr.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/kevinmcmahon/tagger/69432971b6af1f2649f2af5b623f156ffeb0d1c3/doc/crarr.png


--------------------------------------------------------------------------------
/doc/epydoc.js:
--------------------------------------------------------------------------------
  1 | function toggle_private() {
  2 |         // Search for any private/public links on this page.  Store
  3 |         // their old text in "cmd," so we will know what action to
  4 |         // take; and change their text to the opposite action.
  5 |         var cmd = "?";
  6 |         var elts = document.getElementsByTagName("a");
  7 |         for(var i=0; i<elts.length; i++) {
  8 |           if (elts[i].className == "privatelink") {
  9 |             cmd = elts[i].innerHTML;
 10 |             elts[i].innerHTML = ((cmd && cmd.substr(0,4)=="show")?
 11 |                                     "hide&nbsp;private":"show&nbsp;private");
 12 |           }
 13 |         }
 14 |         // Update all DIVs containing private objects.
 15 |         var elts = document.getElementsByTagName("div");
 16 |         for(var i=0; i<elts.length; i++) {
 17 |           if (elts[i].className == "private") {
 18 |             elts[i].style.display = ((cmd && cmd.substr(0,4)=="hide")?"none":"block");
 19 |           }
 20 |           else if (elts[i].className == "public") {
 21 |             elts[i].style.display = ((cmd && cmd.substr(0,4)=="hide")?"block":"none");
 22 |           }
 23 |         }
 24 |         // Update all table rows containing private objects.  Note, we
 25 |         // use "" instead of "block" becaue IE & firefox disagree on what
 26 |         // this should be (block vs table-row), and "" just gives the
 27 |         // default for both browsers.
 28 |         var elts = document.getElementsByTagName("tr");
 29 |         for(var i=0; i<elts.length; i++) {
 30 |           if (elts[i].className == "private") {
 31 |             elts[i].style.display = ((cmd && cmd.substr(0,4)=="hide")?"none":"");
 32 |           }
 33 |         }
 34 |         // Update all list items containing private objects.
 35 |         var elts = document.getElementsByTagName("li");
 36 |         for(var i=0; i<elts.length; i++) {
 37 |           if (elts[i].className == "private") {
 38 |             elts[i].style.display = ((cmd && cmd.substr(0,4)=="hide")?
 39 |                                         "none":"");
 40 |           }
 41 |         }
 42 |         // Update all list items containing private objects.
 43 |         var elts = document.getElementsByTagName("ul");
 44 |         for(var i=0; i<elts.length; i++) {
 45 |           if (elts[i].className == "private") {
 46 |             elts[i].style.display = ((cmd && cmd.substr(0,4)=="hide")?"none":"block");
 47 |           }
 48 |         }
 49 |         // Set a cookie to remember the current option.
 50 |         document.cookie = "EpydocPrivate="+cmd;
 51 |       }
 52 | function show_private() {
 53 |         var elts = document.getElementsByTagName("a");
 54 |         for(var i=0; i<elts.length; i++) {
 55 |           if (elts[i].className == "privatelink") {
 56 |             cmd = elts[i].innerHTML;
 57 |             if (cmd && cmd.substr(0,4)=="show")
 58 |                 toggle_private();
 59 |           }
 60 |         }
 61 |       }
 62 | function getCookie(name) {
 63 |         var dc = document.cookie;
 64 |         var prefix = name + "=";
 65 |         var begin = dc.indexOf("; " + prefix);
 66 |         if (begin == -1) {
 67 |           begin = dc.indexOf(prefix);
 68 |           if (begin != 0) return null;
 69 |         } else
 70 |         { begin += 2; }
 71 |         var end = document.cookie.indexOf(";", begin);
 72 |         if (end == -1)
 73 |         { end = dc.length; }
 74 |         return unescape(dc.substring(begin + prefix.length, end));
 75 |       }
 76 | function setFrame(url1, url2) {
 77 |           parent.frames[1].location.href = url1;
 78 |           parent.frames[2].location.href = url2;
 79 |       }
 80 | function checkCookie() {
 81 |         var cmd=getCookie("EpydocPrivate");
 82 |         if (cmd && cmd.substr(0,4)!="show" && location.href.indexOf("#_") < 0)
 83 |             toggle_private();
 84 |       }
 85 | function toggleCallGraph(id) {
 86 |         var elt = document.getElementById(id);
 87 |         if (elt.style.display == "none")
 88 |             elt.style.display = "block";
 89 |         else
 90 |             elt.style.display = "none";
 91 |       }
 92 | function expand(id) {
 93 |   var elt = document.getElementById(id+"-expanded");
 94 |   if (elt) elt.style.display = "block";
 95 |   var elt = document.getElementById(id+"-expanded-linenums");
 96 |   if (elt) elt.style.display = "block";
 97 |   var elt = document.getElementById(id+"-collapsed");
 98 |   if (elt) { elt.innerHTML = ""; elt.style.display = "none"; }
 99 |   var elt = document.getElementById(id+"-collapsed-linenums");
100 |   if (elt) { elt.innerHTML = ""; elt.style.display = "none"; }
101 |   var elt = document.getElementById(id+"-toggle");
102 |   if (elt) { elt.innerHTML = "-"; }
103 | }
104 | 
105 | function collapse(id) {
106 |   var elt = document.getElementById(id+"-expanded");
107 |   if (elt) elt.style.display = "none";
108 |   var elt = document.getElementById(id+"-expanded-linenums");
109 |   if (elt) elt.style.display = "none";
110 |   var elt = document.getElementById(id+"-collapsed-linenums");
111 |   if (elt) { elt.innerHTML = "<br />"; elt.style.display="block"; }
112 |   var elt = document.getElementById(id+"-toggle");
113 |   if (elt) { elt.innerHTML = "+"; }
114 |   var elt = document.getElementById(id+"-collapsed");
115 |   if (elt) {
116 |     elt.style.display = "block";
117 |     
118 |     var indent = elt.getAttribute("indent");
119 |     var pad = elt.getAttribute("pad");
120 |     var s = "<tt class='py-lineno'>";
121 |     for (var i=0; i<pad.length; i++) { s += "&nbsp;" }
122 |     s += "</tt>";
123 |     s += "&nbsp;&nbsp;<tt class='py-line'>";
124 |     for (var i=0; i<indent.length; i++) { s += "&nbsp;" }
125 |     s += "<a href='#' onclick='expand(\"" + id;
126 |     s += "\");return false'>...</a></tt><br />";
127 |     elt.innerHTML = s;
128 |   }
129 | }
130 | 
131 | function toggle(id) {
132 |   elt = document.getElementById(id+"-toggle");
133 |   if (elt.innerHTML == "-")
134 |       collapse(id); 
135 |   else
136 |       expand(id);
137 |   return false;
138 | }
139 | 
140 | function highlight(id) {
141 |   var elt = document.getElementById(id+"-def");
142 |   if (elt) elt.className = "py-highlight-hdr";
143 |   var elt = document.getElementById(id+"-expanded");
144 |   if (elt) elt.className = "py-highlight";
145 |   var elt = document.getElementById(id+"-collapsed");
146 |   if (elt) elt.className = "py-highlight";
147 | }
148 | 
149 | function num_lines(s) {
150 |   var n = 1;
151 |   var pos = s.indexOf("\n");
152 |   while ( pos > 0) {
153 |     n += 1;
154 |     pos = s.indexOf("\n", pos+1);
155 |   }
156 |   return n;
157 | }
158 | 
159 | // Collapse all blocks that mave more than `min_lines` lines.
160 | function collapse_all(min_lines) {
161 |   var elts = document.getElementsByTagName("div");
162 |   for (var i=0; i<elts.length; i++) {
163 |     var elt = elts[i];
164 |     var split = elt.id.indexOf("-");
165 |     if (split > 0)
166 |       if (elt.id.substring(split, elt.id.length) == "-expanded")
167 |         if (num_lines(elt.innerHTML) > min_lines)
168 |           collapse(elt.id.substring(0, split));
169 |   }
170 | }
171 | 
172 | function expandto(href) {
173 |   var start = href.indexOf("#")+1;
174 |   if (start != 0 && start != href.length) {
175 |     if (href.substring(start, href.length) != "-") {
176 |       collapse_all(4);
177 |       pos = href.indexOf(".", start);
178 |       while (pos != -1) {
179 |         var id = href.substring(start, pos);
180 |         expand(id);
181 |         pos = href.indexOf(".", pos+1);
182 |       }
183 |       var id = href.substring(start, href.length);
184 |       expand(id);
185 |       highlight(id);
186 |     }
187 |   }
188 | }
189 | 
190 | function kill_doclink(id) {
191 |   var parent = document.getElementById(id);
192 |   parent.removeChild(parent.childNodes.item(0));
193 | }
194 | function auto_kill_doclink(ev) {
195 |   if (!ev) var ev = window.event;
196 |   if (!this.contains(ev.toElement)) {
197 |     var parent = document.getElementById(this.parentID);
198 |     parent.removeChild(parent.childNodes.item(0));
199 |   }
200 | }
201 | 
202 | function doclink(id, name, targets_id) {
203 |   var elt = document.getElementById(id);
204 | 
205 |   // If we already opened the box, then destroy it.
206 |   // (This case should never occur, but leave it in just in case.)
207 |   if (elt.childNodes.length > 1) {
208 |     elt.removeChild(elt.childNodes.item(0));
209 |   }
210 |   else {
211 |     // The outer box: relative + inline positioning.
212 |     var box1 = document.createElement("div");
213 |     box1.style.position = "relative";
214 |     box1.style.display = "inline";
215 |     box1.style.top = 0;
216 |     box1.style.left = 0;
217 |   
218 |     // A shadow for fun
219 |     var shadow = document.createElement("div");
220 |     shadow.style.position = "absolute";
221 |     shadow.style.left = "-1.3em";
222 |     shadow.style.top = "-1.3em";
223 |     shadow.style.background = "#404040";
224 |     
225 |     // The inner box: absolute positioning.
226 |     var box2 = document.createElement("div");
227 |     box2.style.position = "relative";
228 |     box2.style.border = "1px solid #a0a0a0";
229 |     box2.style.left = "-.2em";
230 |     box2.style.top = "-.2em";
231 |     box2.style.background = "white";
232 |     box2.style.padding = ".3em .4em .3em .4em";
233 |     box2.style.fontStyle = "normal";
234 |     box2.onmouseout=auto_kill_doclink;
235 |     box2.parentID = id;
236 | 
237 |     // Get the targets
238 |     var targets_elt = document.getElementById(targets_id);
239 |     var targets = targets_elt.getAttribute("targets");
240 |     var links = "";
241 |     target_list = targets.split(",");
242 |     for (var i=0; i<target_list.length; i++) {
243 |         var target = target_list[i].split("=");
244 |         links += "<li><a href='" + target[1] + 
245 |                "' style='text-decoration:none'>" +
246 |                target[0] + "</a></li>";
247 |     }
248 |   
249 |     // Put it all together.
250 |     elt.insertBefore(box1, elt.childNodes.item(0));
251 |     //box1.appendChild(box2);
252 |     box1.appendChild(shadow);
253 |     shadow.appendChild(box2);
254 |     box2.innerHTML =
255 |         "Which <b>"+name+"</b> do you want to see documentation for?" +
256 |         "<ul style='margin-bottom: 0;'>" +
257 |         links + 
258 |         "<li><a href='#' style='text-decoration:none' " +
259 |         "onclick='kill_doclink(\""+id+"\");return false;'>"+
260 |         "<i>None of the above</i></a></li></ul>";
261 |   }
262 |   return false;
263 | }
264 | 
265 | function get_anchor() {
266 |           var href = location.href;
267 |           var start = href.indexOf("#")+1;
268 |           if ((start != 0) && (start != href.length))
269 |               return href.substring(start, href.length);
270 |       }
271 | function redirect_url(dottedName) {
272 |           // Scan through each element of the "pages" list, and check
273 |           // if "name" matches with any of them.
274 |           for (var i=0; i<pages.length; i++) {
275 | 
276 |               // Each page has the form "<pagename>-m" or "<pagename>-c";
277 |               // extract the <pagename> portion & compare it to dottedName.
278 |               var pagename = pages[i].substring(0, pages[i].length-2);
279 |               if (pagename == dottedName.substring(0,pagename.length)) {
280 | 
281 |                   // We've found a page that matches `dottedName`;
282 |                   // construct its URL, using leftover `dottedName`
283 |                   // content to form an anchor.
284 |                   var pagetype = pages[i].charAt(pages[i].length-1);
285 |                   var url = pagename + ((pagetype=="m")?"-module.html":
286 |                                                         "-class.html");
287 |                   if (dottedName.length > pagename.length)
288 |                       url += "#" + dottedName.substring(pagename.length+1,
289 |                                                         dottedName.length);
290 |                   return url;
291 |               }
292 |           }
293 |       }
294 | 


--------------------------------------------------------------------------------
/doc/extras-module.html:
--------------------------------------------------------------------------------
  1 | <?xml version="1.0" encoding="ascii"?>
  2 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  3 |           "DTD/xhtml1-transitional.dtd">
  4 | <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
  5 | <head>
  6 |   <title>extras</title>
  7 |   <link rel="stylesheet" href="epydoc.css" type="text/css" />
  8 |   <script type="text/javascript" src="epydoc.js"></script>
  9 | </head>
 10 | 
 11 | <body bgcolor="white" text="black" link="blue" vlink="#204080"
 12 |       alink="#204080">
 13 | <!-- ==================== NAVIGATION BAR ==================== -->
 14 | <table class="navbar" border="0" width="100%" cellpadding="0"
 15 |        bgcolor="#a0c0ff" cellspacing="0">
 16 |   <tr valign="middle">
 17 | 
 18 |   <!-- Tree link -->
 19 |       <th>&nbsp;&nbsp;&nbsp;<a
 20 |         href="module-tree.html">Trees</a>&nbsp;&nbsp;&nbsp;</th>
 21 | 
 22 |   <!-- Index link -->
 23 |       <th>&nbsp;&nbsp;&nbsp;<a
 24 |         href="identifier-index.html">Indices</a>&nbsp;&nbsp;&nbsp;</th>
 25 | 
 26 |   <!-- Help link -->
 27 |       <th>&nbsp;&nbsp;&nbsp;<a
 28 |         href="help.html">Help</a>&nbsp;&nbsp;&nbsp;</th>
 29 | 
 30 |   <!-- Project homepage -->
 31 |       <th class="navbar" align="right" width="100%">
 32 |         <table border="0" cellpadding="0" cellspacing="0">
 33 |           <tr><th class="navbar" align="center"
 34 |             ><a class="navbar" target="_top" href="http://github.com/apresta/tagger">tagger</a></th>
 35 |           </tr></table></th>
 36 |   </tr>
 37 | </table>
 38 | <table width="100%" cellpadding="0" cellspacing="0">
 39 |   <tr valign="top">
 40 |     <td width="100%">
 41 |       <span class="breadcrumbs">
 42 |         Module&nbsp;extras
 43 |       </span>
 44 |     </td>
 45 |     <td>
 46 |       <table cellpadding="0" cellspacing="0">
 47 |         <!-- hide/show private -->
 48 |         <tr><td align="right"><span class="options">[<a href="javascript:void(0);" class="privatelink"
 49 |     onclick="toggle_private();">hide&nbsp;private</a>]</span></td></tr>
 50 |         <tr><td align="right"><span class="options"
 51 |             >[<a href="frames.html" target="_top">frames</a
 52 |             >]&nbsp;|&nbsp;<a href="extras-module.html"
 53 |             target="_top">no&nbsp;frames</a>]</span></td></tr>
 54 |       </table>
 55 |     </td>
 56 |   </tr>
 57 | </table>
 58 | <!-- ==================== MODULE DESCRIPTION ==================== -->
 59 | <h1 class="epydoc">Module extras</h1><p class="nomargin-top"><span class="codelink"><a href="extras-pysrc.html">source&nbsp;code</a></span></p>
 60 | <!-- ==================== CLASSES ==================== -->
 61 | <a name="section-Classes"></a>
 62 | <table class="summary" border="1" cellpadding="3"
 63 |        cellspacing="0" width="100%" bgcolor="white">
 64 | <tr bgcolor="#70b0f0" class="table-header">
 65 |   <td colspan="2" class="table-header">
 66 |     <table border="0" cellpadding="0" cellspacing="0" width="100%">
 67 |       <tr valign="top">
 68 |         <td align="left"><span class="table-header">Classes</span></td>
 69 |         <td align="right" valign="top"
 70 |          ><span class="options">[<a href="#section-Classes"
 71 |          class="privatelink" onclick="toggle_private();"
 72 |          >hide private</a>]</span></td>
 73 |       </tr>
 74 |     </table>
 75 |   </td>
 76 | </tr>
 77 | <tr>
 78 |     <td width="15%" align="right" valign="top" class="summary">
 79 |       <span class="summary-type">&nbsp;</span>
 80 |     </td><td class="summary">
 81 |         <a href="extras.UnicodeReader-class.html" class="summary-name">UnicodeReader</a><br />
 82 |       Reader subclass that converts Unicode strings to a close ASCII 
 83 |         representation
 84 |     </td>
 85 |   </tr>
 86 | <tr>
 87 |     <td width="15%" align="right" valign="top" class="summary">
 88 |       <span class="summary-type">&nbsp;</span>
 89 |     </td><td class="summary">
 90 |         <a href="extras.HTMLReader-class.html" class="summary-name">HTMLReader</a><br />
 91 |       Reader subclass that can parse HTML code from the input
 92 |     </td>
 93 |   </tr>
 94 | <tr>
 95 |     <td width="15%" align="right" valign="top" class="summary">
 96 |       <span class="summary-type">&nbsp;</span>
 97 |     </td><td class="summary">
 98 |         <a href="extras.SimpleReader-class.html" class="summary-name">SimpleReader</a><br />
 99 |       Reader subclass that doesn't perform any advanced analysis of the 
100 |         text
101 |     </td>
102 |   </tr>
103 | <tr>
104 |     <td width="15%" align="right" valign="top" class="summary">
105 |       <span class="summary-type">&nbsp;</span>
106 |     </td><td class="summary">
107 |         <a href="extras.FastStemmer-class.html" class="summary-name">FastStemmer</a><br />
108 |       Stemmer subclass that uses a much faster, but less correct 
109 |         algorithm
110 |     </td>
111 |   </tr>
112 | <tr>
113 |     <td width="15%" align="right" valign="top" class="summary">
114 |       <span class="summary-type">&nbsp;</span>
115 |     </td><td class="summary">
116 |         <a href="extras.NaiveRater-class.html" class="summary-name">NaiveRater</a><br />
117 |       Rater subclass that jusk ranks single-word tags by their frequency 
118 |         and weight
119 |     </td>
120 |   </tr>
121 | </table>
122 | <!-- ==================== FUNCTIONS ==================== -->
123 | <a name="section-Functions"></a>
124 | <table class="summary" border="1" cellpadding="3"
125 |        cellspacing="0" width="100%" bgcolor="white">
126 | <tr bgcolor="#70b0f0" class="table-header">
127 |   <td colspan="2" class="table-header">
128 |     <table border="0" cellpadding="0" cellspacing="0" width="100%">
129 |       <tr valign="top">
130 |         <td align="left"><span class="table-header">Functions</span></td>
131 |         <td align="right" valign="top"
132 |          ><span class="options">[<a href="#section-Functions"
133 |          class="privatelink" onclick="toggle_private();"
134 |          >hide private</a>]</span></td>
135 |       </tr>
136 |     </table>
137 |   </td>
138 | </tr>
139 | <tr>
140 |     <td width="15%" align="right" valign="top" class="summary">
141 |       <span class="summary-type">&nbsp;</span>
142 |     </td><td class="summary">
143 |       <table width="100%" cellpadding="0" cellspacing="0" border="0">
144 |         <tr>
145 |           <td><span class="summary-sig"><a href="extras-module.html#build_dict_from_nltk" class="summary-sig-name">build_dict_from_nltk</a>(<span class="summary-sig-arg">output_file</span>,
146 |         <span class="summary-sig-arg">corpus</span>=<span class="summary-sig-default">None</span>,
147 |         <span class="summary-sig-arg">stopwords</span>=<span class="summary-sig-default">None</span>,
148 |         <span class="summary-sig-arg">stemmer</span>=<span class="summary-sig-default">Stemmer()</span>,
149 |         <span class="summary-sig-arg">measure</span>=<span class="summary-sig-default"><code class="variable-quote">'</code><code class="variable-string">IDF</code><code class="variable-quote">'</code></span>,
150 |         <span class="summary-sig-arg">verbose</span>=<span class="summary-sig-default">False</span>)</span></td>
151 |           <td align="right" valign="top">
152 |             <span class="codelink"><a href="extras-pysrc.html#build_dict_from_nltk">source&nbsp;code</a></span>
153 |             
154 |           </td>
155 |         </tr>
156 |       </table>
157 |       
158 |     </td>
159 |   </tr>
160 | </table>
161 | <!-- ==================== VARIABLES ==================== -->
162 | <a name="section-Variables"></a>
163 | <table class="summary" border="1" cellpadding="3"
164 |        cellspacing="0" width="100%" bgcolor="white">
165 | <tr bgcolor="#70b0f0" class="table-header">
166 |   <td colspan="2" class="table-header">
167 |     <table border="0" cellpadding="0" cellspacing="0" width="100%">
168 |       <tr valign="top">
169 |         <td align="left"><span class="table-header">Variables</span></td>
170 |         <td align="right" valign="top"
171 |          ><span class="options">[<a href="#section-Variables"
172 |          class="privatelink" onclick="toggle_private();"
173 |          >hide private</a>]</span></td>
174 |       </tr>
175 |     </table>
176 |   </td>
177 | </tr>
178 | <tr>
179 |     <td width="15%" align="right" valign="top" class="summary">
180 |       <span class="summary-type">&nbsp;</span>
181 |     </td><td class="summary">
182 |         <a name="__package__"></a><span class="summary-name">__package__</span> = <code title="None">None</code>
183 |     </td>
184 |   </tr>
185 | </table>
186 | <!-- ==================== FUNCTION DETAILS ==================== -->
187 | <a name="section-FunctionDetails"></a>
188 | <table class="details" border="1" cellpadding="3"
189 |        cellspacing="0" width="100%" bgcolor="white">
190 | <tr bgcolor="#70b0f0" class="table-header">
191 |   <td colspan="2" class="table-header">
192 |     <table border="0" cellpadding="0" cellspacing="0" width="100%">
193 |       <tr valign="top">
194 |         <td align="left"><span class="table-header">Function Details</span></td>
195 |         <td align="right" valign="top"
196 |          ><span class="options">[<a href="#section-FunctionDetails"
197 |          class="privatelink" onclick="toggle_private();"
198 |          >hide private</a>]</span></td>
199 |       </tr>
200 |     </table>
201 |   </td>
202 | </tr>
203 | </table>
204 | <a name="build_dict_from_nltk"></a>
205 | <div>
206 | <table class="details" border="1" cellpadding="3"
207 |        cellspacing="0" width="100%" bgcolor="white">
208 | <tr><td>
209 |   <table width="100%" cellpadding="0" cellspacing="0" border="0">
210 |   <tr valign="top"><td>
211 |   <h3 class="epydoc"><span class="sig"><span class="sig-name">build_dict_from_nltk</span>(<span class="sig-arg">output_file</span>,
212 |         <span class="sig-arg">corpus</span>=<span class="sig-default">None</span>,
213 |         <span class="sig-arg">stopwords</span>=<span class="sig-default">None</span>,
214 |         <span class="sig-arg">stemmer</span>=<span class="sig-default">Stemmer()</span>,
215 |         <span class="sig-arg">measure</span>=<span class="sig-default"><code class="variable-quote">'</code><code class="variable-string">IDF</code><code class="variable-quote">'</code></span>,
216 |         <span class="sig-arg">verbose</span>=<span class="sig-default">False</span>)</span>
217 |   </h3>
218 |   </td><td align="right" valign="top"
219 |     ><span class="codelink"><a href="extras-pysrc.html#build_dict_from_nltk">source&nbsp;code</a></span>&nbsp;
220 |     </td>
221 |   </tr></table>
222 |   
223 |   
224 |   <dl class="fields">
225 |     <dt>Parameters:</dt>
226 |     <dd><ul class="nomargin-top">
227 |         <li><strong class="pname"><code>output_file</code></strong> - the binary stream where the dictionary should be saved</li>
228 |         <li><strong class="pname"><code>corpus</code></strong> - the NLTK corpus to use (defaults to nltk.corpus.reuters)</li>
229 |         <li><strong class="pname"><code>stopwords</code></strong> - a list of (not stemmed) stopwords (defaults to 
230 |           nltk.corpus.reuters.words('stopwords'))</li>
231 |         <li><strong class="pname"><code>stemmer</code></strong> - the <a href="tagger.Stemmer-class.html" class="link">Stemmer</a> 
232 |           object to be used</li>
233 |         <li><strong class="pname"><code>measure</code></strong> - the measure used to compute the weights ('IDF' i.e. 'inverse 
234 |           document frequency' or 'ICF' i.e. 'inverse collection frequency';
235 |           defaults to 'IDF')</li>
236 |         <li><strong class="pname"><code>verbose</code></strong> - whether information on the progress should be printed on screen</li>
237 |     </ul></dd>
238 |   </dl>
239 | </td></tr></table>
240 | </div>
241 | <br />
242 | <!-- ==================== NAVIGATION BAR ==================== -->
243 | <table class="navbar" border="0" width="100%" cellpadding="0"
244 |        bgcolor="#a0c0ff" cellspacing="0">
245 |   <tr valign="middle">
246 | 
247 |   <!-- Tree link -->
248 |       <th>&nbsp;&nbsp;&nbsp;<a
249 |         href="module-tree.html">Trees</a>&nbsp;&nbsp;&nbsp;</th>
250 | 
251 |   <!-- Index link -->
252 |       <th>&nbsp;&nbsp;&nbsp;<a
253 |         href="identifier-index.html">Indices</a>&nbsp;&nbsp;&nbsp;</th>
254 | 
255 |   <!-- Help link -->
256 |       <th>&nbsp;&nbsp;&nbsp;<a
257 |         href="help.html">Help</a>&nbsp;&nbsp;&nbsp;</th>
258 | 
259 |   <!-- Project homepage -->
260 |       <th class="navbar" align="right" width="100%">
261 |         <table border="0" cellpadding="0" cellspacing="0">
262 |           <tr><th class="navbar" align="center"
263 |             ><a class="navbar" target="_top" href="http://github.com/apresta/tagger">tagger</a></th>
264 |           </tr></table></th>
265 |   </tr>
266 | </table>
267 | <table border="0" cellpadding="0" cellspacing="0" width="100%%">
268 |   <tr>
269 |     <td align="left" class="footer">
270 |     Generated by Epydoc 3.0.1 on Fri May 13 11:13:02 2011
271 |     </td>
272 |     <td align="right" class="footer">
273 |       <a target="mainFrame" href="http://epydoc.sourceforge.net"
274 |         >http://epydoc.sourceforge.net</a>
275 |     </td>
276 |   </tr>
277 | </table>
278 | 
279 | <script type="text/javascript">
280 |   <!--
281 |   // Private objects are initially displayed (because if
282 |   // javascript is turned off then we want them to be
283 |   // visible); but by default, we want to hide them.  So hide
284 |   // them unless we have a cookie that says to show them.
285 |   checkCookie();
286 |   // -->
287 | </script>
288 | </body>
289 | </html>
290 | 


--------------------------------------------------------------------------------
/doc/extras.FastStemmer-class.html:
--------------------------------------------------------------------------------
  1 | <?xml version="1.0" encoding="ascii"?>
  2 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  3 |           "DTD/xhtml1-transitional.dtd">
  4 | <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
  5 | <head>
  6 |   <title>extras.FastStemmer</title>
  7 |   <link rel="stylesheet" href="epydoc.css" type="text/css" />
  8 |   <script type="text/javascript" src="epydoc.js"></script>
  9 | </head>
 10 | 
 11 | <body bgcolor="white" text="black" link="blue" vlink="#204080"
 12 |       alink="#204080">
 13 | <!-- ==================== NAVIGATION BAR ==================== -->
 14 | <table class="navbar" border="0" width="100%" cellpadding="0"
 15 |        bgcolor="#a0c0ff" cellspacing="0">
 16 |   <tr valign="middle">
 17 | 
 18 |   <!-- Tree link -->
 19 |       <th>&nbsp;&nbsp;&nbsp;<a
 20 |         href="module-tree.html">Trees</a>&nbsp;&nbsp;&nbsp;</th>
 21 | 
 22 |   <!-- Index link -->
 23 |       <th>&nbsp;&nbsp;&nbsp;<a
 24 |         href="identifier-index.html">Indices</a>&nbsp;&nbsp;&nbsp;</th>
 25 | 
 26 |   <!-- Help link -->
 27 |       <th>&nbsp;&nbsp;&nbsp;<a
 28 |         href="help.html">Help</a>&nbsp;&nbsp;&nbsp;</th>
 29 | 
 30 |   <!-- Project homepage -->
 31 |       <th class="navbar" align="right" width="100%">
 32 |         <table border="0" cellpadding="0" cellspacing="0">
 33 |           <tr><th class="navbar" align="center"
 34 |             ><a class="navbar" target="_top" href="http://github.com/apresta/tagger">tagger</a></th>
 35 |           </tr></table></th>
 36 |   </tr>
 37 | </table>
 38 | <table width="100%" cellpadding="0" cellspacing="0">
 39 |   <tr valign="top">
 40 |     <td width="100%">
 41 |       <span class="breadcrumbs">
 42 |         <a href="extras-module.html">Module&nbsp;extras</a> ::
 43 |         Class&nbsp;FastStemmer
 44 |       </span>
 45 |     </td>
 46 |     <td>
 47 |       <table cellpadding="0" cellspacing="0">
 48 |         <!-- hide/show private -->
 49 |         <tr><td align="right"><span class="options">[<a href="javascript:void(0);" class="privatelink"
 50 |     onclick="toggle_private();">hide&nbsp;private</a>]</span></td></tr>
 51 |         <tr><td align="right"><span class="options"
 52 |             >[<a href="frames.html" target="_top">frames</a
 53 |             >]&nbsp;|&nbsp;<a href="extras.FastStemmer-class.html"
 54 |             target="_top">no&nbsp;frames</a>]</span></td></tr>
 55 |       </table>
 56 |     </td>
 57 |   </tr>
 58 | </table>
 59 | <!-- ==================== CLASS DESCRIPTION ==================== -->
 60 | <h1 class="epydoc">Class FastStemmer</h1><p class="nomargin-top"><span class="codelink"><a href="extras-pysrc.html#FastStemmer">source&nbsp;code</a></span></p>
 61 | <center>
 62 | <center>  <map id="class_hierarchy_for_faststemme" name="class_hierarchy_for_faststemme">
 63 | <area shape="rect" id="node1" href="extras.FastStemmer-class.html" title="FastStemmer" alt="" coords="13,61,120,88"/>
 64 | <area shape="rect" id="node2" href="tagger.Stemmer-class.html" title="tagger.Stemmer" alt="" coords="5,5,128,32"/>
 65 | </map>
 66 |   <img src="class_hierarchy_for_faststemme.gif" alt='' usemap="#class_hierarchy_for_faststemme" ismap="ismap" class="graph-without-title" />
 67 | </center>
 68 | </center>
 69 | <hr />
 70 | <p>Stemmer subclass that uses a much faster, but less correct 
 71 |   algorithm</p>
 72 | 
 73 | <!-- ==================== INSTANCE METHODS ==================== -->
 74 | <a name="section-InstanceMethods"></a>
 75 | <table class="summary" border="1" cellpadding="3"
 76 |        cellspacing="0" width="100%" bgcolor="white">
 77 | <tr bgcolor="#70b0f0" class="table-header">
 78 |   <td colspan="2" class="table-header">
 79 |     <table border="0" cellpadding="0" cellspacing="0" width="100%">
 80 |       <tr valign="top">
 81 |         <td align="left"><span class="table-header">Instance Methods</span></td>
 82 |         <td align="right" valign="top"
 83 |          ><span class="options">[<a href="#section-InstanceMethods"
 84 |          class="privatelink" onclick="toggle_private();"
 85 |          >hide private</a>]</span></td>
 86 |       </tr>
 87 |     </table>
 88 |   </td>
 89 | </tr>
 90 | <tr>
 91 |     <td width="15%" align="right" valign="top" class="summary">
 92 |       <span class="summary-type">&nbsp;</span>
 93 |     </td><td class="summary">
 94 |       <table width="100%" cellpadding="0" cellspacing="0" border="0">
 95 |         <tr>
 96 |           <td><span class="summary-sig"><a href="extras.FastStemmer-class.html#__init__" class="summary-sig-name">__init__</a>(<span class="summary-sig-arg">self</span>)</span><br />
 97 |       Returns:
 98 |       a new <a href="tagger.Stemmer-class.html" class="link">Stemmer</a> 
 99 |       object</td>
100 |           <td align="right" valign="top">
101 |             <span class="codelink"><a href="extras-pysrc.html#FastStemmer.__init__">source&nbsp;code</a></span>
102 |             
103 |           </td>
104 |         </tr>
105 |       </table>
106 |       
107 |     </td>
108 |   </tr>
109 |   <tr>
110 |     <td colspan="2" class="summary">
111 |     <p class="indent-wrapped-lines"><b>Inherited from <code><a href="tagger.Stemmer-class.html">tagger.Stemmer</a></code></b>:
112 |       <code><a href="tagger.Stemmer-class.html#__call__">__call__</a></code>,
113 |       <code><a href="tagger.Stemmer-class.html#preprocess">preprocess</a></code>
114 |       </p>
115 |     </td>
116 |   </tr>
117 | </table>
118 | <!-- ==================== CLASS VARIABLES ==================== -->
119 | <a name="section-ClassVariables"></a>
120 | <table class="summary" border="1" cellpadding="3"
121 |        cellspacing="0" width="100%" bgcolor="white">
122 | <tr bgcolor="#70b0f0" class="table-header">
123 |   <td colspan="2" class="table-header">
124 |     <table border="0" cellpadding="0" cellspacing="0" width="100%">
125 |       <tr valign="top">
126 |         <td align="left"><span class="table-header">Class Variables</span></td>
127 |         <td align="right" valign="top"
128 |          ><span class="options">[<a href="#section-ClassVariables"
129 |          class="privatelink" onclick="toggle_private();"
130 |          >hide private</a>]</span></td>
131 |       </tr>
132 |     </table>
133 |   </td>
134 | </tr>
135 |   <tr>
136 |     <td colspan="2" class="summary">
137 |     <p class="indent-wrapped-lines"><b>Inherited from <code><a href="tagger.Stemmer-class.html">tagger.Stemmer</a></code></b>:
138 |       <code><a href="tagger.Stemmer-class.html#match_contractions">match_contractions</a></code>
139 |       </p>
140 |     </td>
141 |   </tr>
142 | </table>
143 | <!-- ==================== METHOD DETAILS ==================== -->
144 | <a name="section-MethodDetails"></a>
145 | <table class="details" border="1" cellpadding="3"
146 |        cellspacing="0" width="100%" bgcolor="white">
147 | <tr bgcolor="#70b0f0" class="table-header">
148 |   <td colspan="2" class="table-header">
149 |     <table border="0" cellpadding="0" cellspacing="0" width="100%">
150 |       <tr valign="top">
151 |         <td align="left"><span class="table-header">Method Details</span></td>
152 |         <td align="right" valign="top"
153 |          ><span class="options">[<a href="#section-MethodDetails"
154 |          class="privatelink" onclick="toggle_private();"
155 |          >hide private</a>]</span></td>
156 |       </tr>
157 |     </table>
158 |   </td>
159 | </tr>
160 | </table>
161 | <a name="__init__"></a>
162 | <div>
163 | <table class="details" border="1" cellpadding="3"
164 |        cellspacing="0" width="100%" bgcolor="white">
165 | <tr><td>
166 |   <table width="100%" cellpadding="0" cellspacing="0" border="0">
167 |   <tr valign="top"><td>
168 |   <h3 class="epydoc"><span class="sig"><span class="sig-name">__init__</span>(<span class="sig-arg">self</span>)</span>
169 |     <br /><em class="fname">(Constructor)</em>
170 |   </h3>
171 |   </td><td align="right" valign="top"
172 |     ><span class="codelink"><a href="extras-pysrc.html#FastStemmer.__init__">source&nbsp;code</a></span>&nbsp;
173 |     </td>
174 |   </tr></table>
175 |   
176 |   
177 |   <dl class="fields">
178 |     <dt>Parameters:</dt>
179 |     <dd><ul class="nomargin-top">
180 |         <li><strong class="pname"><code>stemmer</code></strong> - an object or module with a 'stem' method (defaults to 
181 |           stemming.porter2)</li>
182 |     </ul></dd>
183 |     <dt>Returns:</dt>
184 |         <dd>a new <a href="tagger.Stemmer-class.html" 
185 |           class="link">Stemmer</a> object</dd>
186 |     <dt>Overrides:
187 |         <a href="tagger.Stemmer-class.html#__init__">tagger.Stemmer.__init__</a>
188 |         <dd><em class="note">(inherited documentation)</em></dd>
189 |     </dt>
190 |   </dl>
191 | </td></tr></table>
192 | </div>
193 | <br />
194 | <!-- ==================== NAVIGATION BAR ==================== -->
195 | <table class="navbar" border="0" width="100%" cellpadding="0"
196 |        bgcolor="#a0c0ff" cellspacing="0">
197 |   <tr valign="middle">
198 | 
199 |   <!-- Tree link -->
200 |       <th>&nbsp;&nbsp;&nbsp;<a
201 |         href="module-tree.html">Trees</a>&nbsp;&nbsp;&nbsp;</th>
202 | 
203 |   <!-- Index link -->
204 |       <th>&nbsp;&nbsp;&nbsp;<a
205 |         href="identifier-index.html">Indices</a>&nbsp;&nbsp;&nbsp;</th>
206 | 
207 |   <!-- Help link -->
208 |       <th>&nbsp;&nbsp;&nbsp;<a
209 |         href="help.html">Help</a>&nbsp;&nbsp;&nbsp;</th>
210 | 
211 |   <!-- Project homepage -->
212 |       <th class="navbar" align="right" width="100%">
213 |         <table border="0" cellpadding="0" cellspacing="0">
214 |           <tr><th class="navbar" align="center"
215 |             ><a class="navbar" target="_top" href="http://github.com/apresta/tagger">tagger</a></th>
216 |           </tr></table></th>
217 |   </tr>
218 | </table>
219 | <table border="0" cellpadding="0" cellspacing="0" width="100%%">
220 |   <tr>
221 |     <td align="left" class="footer">
222 |     Generated by Epydoc 3.0.1 on Fri May 13 11:13:07 2011
223 |     </td>
224 |     <td align="right" class="footer">
225 |       <a target="mainFrame" href="http://epydoc.sourceforge.net"
226 |         >http://epydoc.sourceforge.net</a>
227 |     </td>
228 |   </tr>
229 | </table>
230 | 
231 | <script type="text/javascript">
232 |   <!--
233 |   // Private objects are initially displayed (because if
234 |   // javascript is turned off then we want them to be
235 |   // visible); but by default, we want to hide them.  So hide
236 |   // them unless we have a cookie that says to show them.
237 |   checkCookie();
238 |   // -->
239 | </script>
240 | </body>
241 | </html>
242 | 


--------------------------------------------------------------------------------
/doc/extras.HTMLReader-class.html:
--------------------------------------------------------------------------------
  1 | <?xml version="1.0" encoding="ascii"?>
  2 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  3 |           "DTD/xhtml1-transitional.dtd">
  4 | <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
  5 | <head>
  6 |   <title>extras.HTMLReader</title>
  7 |   <link rel="stylesheet" href="epydoc.css" type="text/css" />
  8 |   <script type="text/javascript" src="epydoc.js"></script>
  9 | </head>
 10 | 
 11 | <body bgcolor="white" text="black" link="blue" vlink="#204080"
 12 |       alink="#204080">
 13 | <!-- ==================== NAVIGATION BAR ==================== -->
 14 | <table class="navbar" border="0" width="100%" cellpadding="0"
 15 |        bgcolor="#a0c0ff" cellspacing="0">
 16 |   <tr valign="middle">
 17 | 
 18 |   <!-- Tree link -->
 19 |       <th>&nbsp;&nbsp;&nbsp;<a
 20 |         href="module-tree.html">Trees</a>&nbsp;&nbsp;&nbsp;</th>
 21 | 
 22 |   <!-- Index link -->
 23 |       <th>&nbsp;&nbsp;&nbsp;<a
 24 |         href="identifier-index.html">Indices</a>&nbsp;&nbsp;&nbsp;</th>
 25 | 
 26 |   <!-- Help link -->
 27 |       <th>&nbsp;&nbsp;&nbsp;<a
 28 |         href="help.html">Help</a>&nbsp;&nbsp;&nbsp;</th>
 29 | 
 30 |   <!-- Project homepage -->
 31 |       <th class="navbar" align="right" width="100%">
 32 |         <table border="0" cellpadding="0" cellspacing="0">
 33 |           <tr><th class="navbar" align="center"
 34 |             ><a class="navbar" target="_top" href="http://github.com/apresta/tagger">tagger</a></th>
 35 |           </tr></table></th>
 36 |   </tr>
 37 | </table>
 38 | <table width="100%" cellpadding="0" cellspacing="0">
 39 |   <tr valign="top">
 40 |     <td width="100%">
 41 |       <span class="breadcrumbs">
 42 |         <a href="extras-module.html">Module&nbsp;extras</a> ::
 43 |         Class&nbsp;HTMLReader
 44 |       </span>
 45 |     </td>
 46 |     <td>
 47 |       <table cellpadding="0" cellspacing="0">
 48 |         <!-- hide/show private -->
 49 |         <tr><td align="right"><span class="options">[<a href="javascript:void(0);" class="privatelink"
 50 |     onclick="toggle_private();">hide&nbsp;private</a>]</span></td></tr>
 51 |         <tr><td align="right"><span class="options"
 52 |             >[<a href="frames.html" target="_top">frames</a
 53 |             >]&nbsp;|&nbsp;<a href="extras.HTMLReader-class.html"
 54 |             target="_top">no&nbsp;frames</a>]</span></td></tr>
 55 |       </table>
 56 |     </td>
 57 |   </tr>
 58 | </table>
 59 | <!-- ==================== CLASS DESCRIPTION ==================== -->
 60 | <h1 class="epydoc">Class HTMLReader</h1><p class="nomargin-top"><span class="codelink"><a href="extras-pysrc.html#HTMLReader">source&nbsp;code</a></span></p>
 61 | <center>
 62 | <center>  <map id="class_hierarchy_for_htmlreader" name="class_hierarchy_for_htmlreader">
 63 | <area shape="rect" id="node1" href="extras.HTMLReader-class.html" title="HTMLReader" alt="" coords="12,117,116,144"/>
 64 | <area shape="rect" id="node2" href="extras.UnicodeReader-class.html" title="UnicodeReader" alt="" coords="5,61,123,88"/>
 65 | <area shape="rect" id="node3" href="tagger.Reader-class.html" title="tagger.Reader" alt="" coords="9,5,119,32"/>
 66 | </map>
 67 |   <img src="class_hierarchy_for_htmlreader.gif" alt='' usemap="#class_hierarchy_for_htmlreader" ismap="ismap" class="graph-without-title" />
 68 | </center>
 69 | </center>
 70 | <hr />
 71 | <p>Reader subclass that can parse HTML code from the input</p>
 72 | 
 73 | <!-- ==================== INSTANCE METHODS ==================== -->
 74 | <a name="section-InstanceMethods"></a>
 75 | <table class="summary" border="1" cellpadding="3"
 76 |        cellspacing="0" width="100%" bgcolor="white">
 77 | <tr bgcolor="#70b0f0" class="table-header">
 78 |   <td colspan="2" class="table-header">
 79 |     <table border="0" cellpadding="0" cellspacing="0" width="100%">
 80 |       <tr valign="top">
 81 |         <td align="left"><span class="table-header">Instance Methods</span></td>
 82 |         <td align="right" valign="top"
 83 |          ><span class="options">[<a href="#section-InstanceMethods"
 84 |          class="privatelink" onclick="toggle_private();"
 85 |          >hide private</a>]</span></td>
 86 |       </tr>
 87 |     </table>
 88 |   </td>
 89 | </tr>
 90 | <tr>
 91 |     <td width="15%" align="right" valign="top" class="summary">
 92 |       <span class="summary-type">&nbsp;</span>
 93 |     </td><td class="summary">
 94 |       <table width="100%" cellpadding="0" cellspacing="0" border="0">
 95 |         <tr>
 96 |           <td><span class="summary-sig"><a href="extras.HTMLReader-class.html#__call__" class="summary-sig-name">__call__</a>(<span class="summary-sig-arg">self</span>,
 97 |         <span class="summary-sig-arg">html</span>)</span><br />
 98 |       Returns:
 99 |       a list of tags respecting the order in the text</td>
100 |           <td align="right" valign="top">
101 |             <span class="codelink"><a href="extras-pysrc.html#HTMLReader.__call__">source&nbsp;code</a></span>
102 |             
103 |           </td>
104 |         </tr>
105 |       </table>
106 |       
107 |     </td>
108 |   </tr>
109 |   <tr>
110 |     <td colspan="2" class="summary">
111 |     <p class="indent-wrapped-lines"><b>Inherited from <code><a href="tagger.Reader-class.html">tagger.Reader</a></code></b>:
112 |       <code><a href="tagger.Reader-class.html#preprocess">preprocess</a></code>
113 |       </p>
114 |     </td>
115 |   </tr>
116 | </table>
117 | <!-- ==================== CLASS VARIABLES ==================== -->
118 | <a name="section-ClassVariables"></a>
119 | <table class="summary" border="1" cellpadding="3"
120 |        cellspacing="0" width="100%" bgcolor="white">
121 | <tr bgcolor="#70b0f0" class="table-header">
122 |   <td colspan="2" class="table-header">
123 |     <table border="0" cellpadding="0" cellspacing="0" width="100%">
124 |       <tr valign="top">
125 |         <td align="left"><span class="table-header">Class Variables</span></td>
126 |         <td align="right" valign="top"
127 |          ><span class="options">[<a href="#section-ClassVariables"
128 |          class="privatelink" onclick="toggle_private();"
129 |          >hide private</a>]</span></td>
130 |       </tr>
131 |     </table>
132 |   </td>
133 | </tr>
134 |   <tr>
135 |     <td colspan="2" class="summary">
136 |     <p class="indent-wrapped-lines"><b>Inherited from <code><a href="tagger.Reader-class.html">tagger.Reader</a></code></b>:
137 |       <code><a href="tagger.Reader-class.html#match_apostrophes">match_apostrophes</a></code>,
138 |       <code><a href="tagger.Reader-class.html#match_paragraphs">match_paragraphs</a></code>,
139 |       <code><a href="tagger.Reader-class.html#match_phrases">match_phrases</a></code>,
140 |       <code><a href="tagger.Reader-class.html#match_words">match_words</a></code>
141 |       </p>
142 |     </td>
143 |   </tr>
144 | </table>
145 | <!-- ==================== METHOD DETAILS ==================== -->
146 | <a name="section-MethodDetails"></a>
147 | <table class="details" border="1" cellpadding="3"
148 |        cellspacing="0" width="100%" bgcolor="white">
149 | <tr bgcolor="#70b0f0" class="table-header">
150 |   <td colspan="2" class="table-header">
151 |     <table border="0" cellpadding="0" cellspacing="0" width="100%">
152 |       <tr valign="top">
153 |         <td align="left"><span class="table-header">Method Details</span></td>
154 |         <td align="right" valign="top"
155 |          ><span class="options">[<a href="#section-MethodDetails"
156 |          class="privatelink" onclick="toggle_private();"
157 |          >hide private</a>]</span></td>
158 |       </tr>
159 |     </table>
160 |   </td>
161 | </tr>
162 | </table>
163 | <a name="__call__"></a>
164 | <div>
165 | <table class="details" border="1" cellpadding="3"
166 |        cellspacing="0" width="100%" bgcolor="white">
167 | <tr><td>
168 |   <table width="100%" cellpadding="0" cellspacing="0" border="0">
169 |   <tr valign="top"><td>
170 |   <h3 class="epydoc"><span class="sig"><span class="sig-name">__call__</span>(<span class="sig-arg">self</span>,
171 |         <span class="sig-arg">html</span>)</span>
172 |     <br /><em class="fname">(Call operator)</em>
173 |   </h3>
174 |   </td><td align="right" valign="top"
175 |     ><span class="codelink"><a href="extras-pysrc.html#HTMLReader.__call__">source&nbsp;code</a></span>&nbsp;
176 |     </td>
177 |   </tr></table>
178 |   
179 |   
180 |   <dl class="fields">
181 |     <dt>Parameters:</dt>
182 |     <dd><ul class="nomargin-top">
183 |         <li><strong class="pname"><code>text</code></strong> - the string of text to be tagged</li>
184 |     </ul></dd>
185 |     <dt>Returns:</dt>
186 |         <dd>a list of tags respecting the order in the text</dd>
187 |     <dt>Overrides:
188 |         <a href="tagger.Reader-class.html#__call__">tagger.Reader.__call__</a>
189 |         <dd><em class="note">(inherited documentation)</em></dd>
190 |     </dt>
191 |   </dl>
192 | </td></tr></table>
193 | </div>
194 | <br />
195 | <!-- ==================== NAVIGATION BAR ==================== -->
196 | <table class="navbar" border="0" width="100%" cellpadding="0"
197 |        bgcolor="#a0c0ff" cellspacing="0">
198 |   <tr valign="middle">
199 | 
200 |   <!-- Tree link -->
201 |       <th>&nbsp;&nbsp;&nbsp;<a
202 |         href="module-tree.html">Trees</a>&nbsp;&nbsp;&nbsp;</th>
203 | 
204 |   <!-- Index link -->
205 |       <th>&nbsp;&nbsp;&nbsp;<a
206 |         href="identifier-index.html">Indices</a>&nbsp;&nbsp;&nbsp;</th>
207 | 
208 |   <!-- Help link -->
209 |       <th>&nbsp;&nbsp;&nbsp;<a
210 |         href="help.html">Help</a>&nbsp;&nbsp;&nbsp;</th>
211 | 
212 |   <!-- Project homepage -->
213 |       <th class="navbar" align="right" width="100%">
214 |         <table border="0" cellpadding="0" cellspacing="0">
215 |           <tr><th class="navbar" align="center"
216 |             ><a class="navbar" target="_top" href="http://github.com/apresta/tagger">tagger</a></th>
217 |           </tr></table></th>
218 |   </tr>
219 | </table>
220 | <table border="0" cellpadding="0" cellspacing="0" width="100%%">
221 |   <tr>
222 |     <td align="left" class="footer">
223 |     Generated by Epydoc 3.0.1 on Fri May 13 11:13:07 2011
224 |     </td>
225 |     <td align="right" class="footer">
226 |       <a target="mainFrame" href="http://epydoc.sourceforge.net"
227 |         >http://epydoc.sourceforge.net</a>
228 |     </td>
229 |   </tr>
230 | </table>
231 | 
232 | <script type="text/javascript">
233 |   <!--
234 |   // Private objects are initially displayed (because if
235 |   // javascript is turned off then we want them to be
236 |   // visible); but by default, we want to hide them.  So hide
237 |   // them unless we have a cookie that says to show them.
238 |   checkCookie();
239 |   // -->
240 | </script>
241 | </body>
242 | </html>
243 | 


--------------------------------------------------------------------------------
/doc/extras.NaiveRater-class.html:
--------------------------------------------------------------------------------
  1 | <?xml version="1.0" encoding="ascii"?>
  2 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  3 |           "DTD/xhtml1-transitional.dtd">
  4 | <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
  5 | <head>
  6 |   <title>extras.NaiveRater</title>
  7 |   <link rel="stylesheet" href="epydoc.css" type="text/css" />
  8 |   <script type="text/javascript" src="epydoc.js"></script>
  9 | </head>
 10 | 
 11 | <body bgcolor="white" text="black" link="blue" vlink="#204080"
 12 |       alink="#204080">
 13 | <!-- ==================== NAVIGATION BAR ==================== -->
 14 | <table class="navbar" border="0" width="100%" cellpadding="0"
 15 |        bgcolor="#a0c0ff" cellspacing="0">
 16 |   <tr valign="middle">
 17 | 
 18 |   <!-- Tree link -->
 19 |       <th>&nbsp;&nbsp;&nbsp;<a
 20 |         href="module-tree.html">Trees</a>&nbsp;&nbsp;&nbsp;</th>
 21 | 
 22 |   <!-- Index link -->
 23 |       <th>&nbsp;&nbsp;&nbsp;<a
 24 |         href="identifier-index.html">Indices</a>&nbsp;&nbsp;&nbsp;</th>
 25 | 
 26 |   <!-- Help link -->
 27 |       <th>&nbsp;&nbsp;&nbsp;<a
 28 |         href="help.html">Help</a>&nbsp;&nbsp;&nbsp;</th>
 29 | 
 30 |   <!-- Project homepage -->
 31 |       <th class="navbar" align="right" width="100%">
 32 |         <table border="0" cellpadding="0" cellspacing="0">
 33 |           <tr><th class="navbar" align="center"
 34 |             ><a class="navbar" target="_top" href="http://github.com/apresta/tagger">tagger</a></th>
 35 |           </tr></table></th>
 36 |   </tr>
 37 | </table>
 38 | <table width="100%" cellpadding="0" cellspacing="0">
 39 |   <tr valign="top">
 40 |     <td width="100%">
 41 |       <span class="breadcrumbs">
 42 |         <a href="extras-module.html">Module&nbsp;extras</a> ::
 43 |         Class&nbsp;NaiveRater
 44 |       </span>
 45 |     </td>
 46 |     <td>
 47 |       <table cellpadding="0" cellspacing="0">
 48 |         <!-- hide/show private -->
 49 |         <tr><td align="right"><span class="options">[<a href="javascript:void(0);" class="privatelink"
 50 |     onclick="toggle_private();">hide&nbsp;private</a>]</span></td></tr>
 51 |         <tr><td align="right"><span class="options"
 52 |             >[<a href="frames.html" target="_top">frames</a
 53 |             >]&nbsp;|&nbsp;<a href="extras.NaiveRater-class.html"
 54 |             target="_top">no&nbsp;frames</a>]</span></td></tr>
 55 |       </table>
 56 |     </td>
 57 |   </tr>
 58 | </table>
 59 | <!-- ==================== CLASS DESCRIPTION ==================== -->
 60 | <h1 class="epydoc">Class NaiveRater</h1><p class="nomargin-top"><span class="codelink"><a href="extras-pysrc.html#NaiveRater">source&nbsp;code</a></span></p>
 61 | <center>
 62 | <center>  <map id="class_hierarchy_for_naiverater" name="class_hierarchy_for_naiverater">
 63 | <area shape="rect" id="node1" href="extras.NaiveRater-class.html" title="NaiveRater" alt="" coords="9,61,100,88"/>
 64 | <area shape="rect" id="node2" href="tagger.Rater-class.html" title="tagger.Rater" alt="" coords="5,5,104,32"/>
 65 | </map>
 66 |   <img src="class_hierarchy_for_naiverater.gif" alt='' usemap="#class_hierarchy_for_naiverater" ismap="ismap" class="graph-without-title" />
 67 | </center>
 68 | </center>
 69 | <hr />
 70 | <p>Rater subclass that jusk ranks single-word tags by their frequency and
 71 |   weight</p>
 72 | 
 73 | <!-- ==================== INSTANCE METHODS ==================== -->
 74 | <a name="section-InstanceMethods"></a>
 75 | <table class="summary" border="1" cellpadding="3"
 76 |        cellspacing="0" width="100%" bgcolor="white">
 77 | <tr bgcolor="#70b0f0" class="table-header">
 78 |   <td colspan="2" class="table-header">
 79 |     <table border="0" cellpadding="0" cellspacing="0" width="100%">
 80 |       <tr valign="top">
 81 |         <td align="left"><span class="table-header">Instance Methods</span></td>
 82 |         <td align="right" valign="top"
 83 |          ><span class="options">[<a href="#section-InstanceMethods"
 84 |          class="privatelink" onclick="toggle_private();"
 85 |          >hide private</a>]</span></td>
 86 |       </tr>
 87 |     </table>
 88 |   </td>
 89 | </tr>
 90 | <tr>
 91 |     <td width="15%" align="right" valign="top" class="summary">
 92 |       <span class="summary-type">&nbsp;</span>
 93 |     </td><td class="summary">
 94 |       <table width="100%" cellpadding="0" cellspacing="0" border="0">
 95 |         <tr>
 96 |           <td><span class="summary-sig"><a href="extras.NaiveRater-class.html#__call__" class="summary-sig-name">__call__</a>(<span class="summary-sig-arg">self</span>,
 97 |         <span class="summary-sig-arg">tags</span>)</span><br />
 98 |       Returns:
 99 |       a list of unique (multi)tags sorted by relevance</td>
100 |           <td align="right" valign="top">
101 |             <span class="codelink"><a href="extras-pysrc.html#NaiveRater.__call__">source&nbsp;code</a></span>
102 |             
103 |           </td>
104 |         </tr>
105 |       </table>
106 |       
107 |     </td>
108 |   </tr>
109 |   <tr>
110 |     <td colspan="2" class="summary">
111 |     <p class="indent-wrapped-lines"><b>Inherited from <code><a href="tagger.Rater-class.html">tagger.Rater</a></code></b>:
112 |       <code><a href="tagger.Rater-class.html#__init__">__init__</a></code>,
113 |       <code><a href="tagger.Rater-class.html#create_multitags">create_multitags</a></code>,
114 |       <code><a href="tagger.Rater-class.html#rate_tags">rate_tags</a></code>
115 |       </p>
116 |     </td>
117 |   </tr>
118 | </table>
119 | <!-- ==================== METHOD DETAILS ==================== -->
120 | <a name="section-MethodDetails"></a>
121 | <table class="details" border="1" cellpadding="3"
122 |        cellspacing="0" width="100%" bgcolor="white">
123 | <tr bgcolor="#70b0f0" class="table-header">
124 |   <td colspan="2" class="table-header">
125 |     <table border="0" cellpadding="0" cellspacing="0" width="100%">
126 |       <tr valign="top">
127 |         <td align="left"><span class="table-header">Method Details</span></td>
128 |         <td align="right" valign="top"
129 |          ><span class="options">[<a href="#section-MethodDetails"
130 |          class="privatelink" onclick="toggle_private();"
131 |          >hide private</a>]</span></td>
132 |       </tr>
133 |     </table>
134 |   </td>
135 | </tr>
136 | </table>
137 | <a name="__call__"></a>
138 | <div>
139 | <table class="details" border="1" cellpadding="3"
140 |        cellspacing="0" width="100%" bgcolor="white">
141 | <tr><td>
142 |   <table width="100%" cellpadding="0" cellspacing="0" border="0">
143 |   <tr valign="top"><td>
144 |   <h3 class="epydoc"><span class="sig"><span class="sig-name">__call__</span>(<span class="sig-arg">self</span>,
145 |         <span class="sig-arg">tags</span>)</span>
146 |     <br /><em class="fname">(Call operator)</em>
147 |   </h3>
148 |   </td><td align="right" valign="top"
149 |     ><span class="codelink"><a href="extras-pysrc.html#NaiveRater.__call__">source&nbsp;code</a></span>&nbsp;
150 |     </td>
151 |   </tr></table>
152 |   
153 |   
154 |   <dl class="fields">
155 |     <dt>Parameters:</dt>
156 |     <dd><ul class="nomargin-top">
157 |         <li><strong class="pname"><code>tags</code></strong> - a list of (preferably stemmed) tags</li>
158 |     </ul></dd>
159 |     <dt>Returns:</dt>
160 |         <dd>a list of unique (multi)tags sorted by relevance</dd>
161 |     <dt>Overrides:
162 |         <a href="tagger.Rater-class.html#__call__">tagger.Rater.__call__</a>
163 |         <dd><em class="note">(inherited documentation)</em></dd>
164 |     </dt>
165 |   </dl>
166 | </td></tr></table>
167 | </div>
168 | <br />
169 | <!-- ==================== NAVIGATION BAR ==================== -->
170 | <table class="navbar" border="0" width="100%" cellpadding="0"
171 |        bgcolor="#a0c0ff" cellspacing="0">
172 |   <tr valign="middle">
173 | 
174 |   <!-- Tree link -->
175 |       <th>&nbsp;&nbsp;&nbsp;<a
176 |         href="module-tree.html">Trees</a>&nbsp;&nbsp;&nbsp;</th>
177 | 
178 |   <!-- Index link -->
179 |       <th>&nbsp;&nbsp;&nbsp;<a
180 |         href="identifier-index.html">Indices</a>&nbsp;&nbsp;&nbsp;</th>
181 | 
182 |   <!-- Help link -->
183 |       <th>&nbsp;&nbsp;&nbsp;<a
184 |         href="help.html">Help</a>&nbsp;&nbsp;&nbsp;</th>
185 | 
186 |   <!-- Project homepage -->
187 |       <th class="navbar" align="right" width="100%">
188 |         <table border="0" cellpadding="0" cellspacing="0">
189 |           <tr><th class="navbar" align="center"
190 |             ><a class="navbar" target="_top" href="http://github.com/apresta/tagger">tagger</a></th>
191 |           </tr></table></th>
192 |   </tr>
193 | </table>
194 | <table border="0" cellpadding="0" cellspacing="0" width="100%%">
195 |   <tr>
196 |     <td align="left" class="footer">
197 |     Generated by Epydoc 3.0.1 on Fri May 13 11:13:08 2011
198 |     </td>
199 |     <td align="right" class="footer">
200 |       <a target="mainFrame" href="http://epydoc.sourceforge.net"
201 |         >http://epydoc.sourceforge.net</a>
202 |     </td>
203 |   </tr>
204 | </table>
205 | 
206 | <script type="text/javascript">
207 |   <!--
208 |   // Private objects are initially displayed (because if
209 |   // javascript is turned off then we want them to be
210 |   // visible); but by default, we want to hide them.  So hide
211 |   // them unless we have a cookie that says to show them.
212 |   checkCookie();
213 |   // -->
214 | </script>
215 | </body>
216 | </html>
217 | 


--------------------------------------------------------------------------------
/doc/extras.SimpleReader-class.html:
--------------------------------------------------------------------------------
  1 | <?xml version="1.0" encoding="ascii"?>
  2 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  3 |           "DTD/xhtml1-transitional.dtd">
  4 | <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
  5 | <head>
  6 |   <title>extras.SimpleReader</title>
  7 |   <link rel="stylesheet" href="epydoc.css" type="text/css" />
  8 |   <script type="text/javascript" src="epydoc.js"></script>
  9 | </head>
 10 | 
 11 | <body bgcolor="white" text="black" link="blue" vlink="#204080"
 12 |       alink="#204080">
 13 | <!-- ==================== NAVIGATION BAR ==================== -->
 14 | <table class="navbar" border="0" width="100%" cellpadding="0"
 15 |        bgcolor="#a0c0ff" cellspacing="0">
 16 |   <tr valign="middle">
 17 | 
 18 |   <!-- Tree link -->
 19 |       <th>&nbsp;&nbsp;&nbsp;<a
 20 |         href="module-tree.html">Trees</a>&nbsp;&nbsp;&nbsp;</th>
 21 | 
 22 |   <!-- Index link -->
 23 |       <th>&nbsp;&nbsp;&nbsp;<a
 24 |         href="identifier-index.html">Indices</a>&nbsp;&nbsp;&nbsp;</th>
 25 | 
 26 |   <!-- Help link -->
 27 |       <th>&nbsp;&nbsp;&nbsp;<a
 28 |         href="help.html">Help</a>&nbsp;&nbsp;&nbsp;</th>
 29 | 
 30 |   <!-- Project homepage -->
 31 |       <th class="navbar" align="right" width="100%">
 32 |         <table border="0" cellpadding="0" cellspacing="0">
 33 |           <tr><th class="navbar" align="center"
 34 |             ><a class="navbar" target="_top" href="http://github.com/apresta/tagger">tagger</a></th>
 35 |           </tr></table></th>
 36 |   </tr>
 37 | </table>
 38 | <table width="100%" cellpadding="0" cellspacing="0">
 39 |   <tr valign="top">
 40 |     <td width="100%">
 41 |       <span class="breadcrumbs">
 42 |         <a href="extras-module.html">Module&nbsp;extras</a> ::
 43 |         Class&nbsp;SimpleReader
 44 |       </span>
 45 |     </td>
 46 |     <td>
 47 |       <table cellpadding="0" cellspacing="0">
 48 |         <!-- hide/show private -->
 49 |         <tr><td align="right"><span class="options">[<a href="javascript:void(0);" class="privatelink"
 50 |     onclick="toggle_private();">hide&nbsp;private</a>]</span></td></tr>
 51 |         <tr><td align="right"><span class="options"
 52 |             >[<a href="frames.html" target="_top">frames</a
 53 |             >]&nbsp;|&nbsp;<a href="extras.SimpleReader-class.html"
 54 |             target="_top">no&nbsp;frames</a>]</span></td></tr>
 55 |       </table>
 56 |     </td>
 57 |   </tr>
 58 | </table>
 59 | <!-- ==================== CLASS DESCRIPTION ==================== -->
 60 | <h1 class="epydoc">Class SimpleReader</h1><p class="nomargin-top"><span class="codelink"><a href="extras-pysrc.html#SimpleReader">source&nbsp;code</a></span></p>
 61 | <center>
 62 | <center>  <map id="class_hierarchy_for_simpleread" name="class_hierarchy_for_simpleread">
 63 | <area shape="rect" id="node1" href="extras.SimpleReader-class.html" title="SimpleReader" alt="" coords="5,61,115,88"/>
 64 | <area shape="rect" id="node2" href="tagger.Reader-class.html" title="tagger.Reader" alt="" coords="5,5,115,32"/>
 65 | </map>
 66 |   <img src="class_hierarchy_for_simpleread.gif" alt='' usemap="#class_hierarchy_for_simpleread" ismap="ismap" class="graph-without-title" />
 67 | </center>
 68 | </center>
 69 | <hr />
 70 | <p>Reader subclass that doesn't perform any advanced analysis of the 
 71 |   text</p>
 72 | 
 73 | <!-- ==================== INSTANCE METHODS ==================== -->
 74 | <a name="section-InstanceMethods"></a>
 75 | <table class="summary" border="1" cellpadding="3"
 76 |        cellspacing="0" width="100%" bgcolor="white">
 77 | <tr bgcolor="#70b0f0" class="table-header">
 78 |   <td colspan="2" class="table-header">
 79 |     <table border="0" cellpadding="0" cellspacing="0" width="100%">
 80 |       <tr valign="top">
 81 |         <td align="left"><span class="table-header">Instance Methods</span></td>
 82 |         <td align="right" valign="top"
 83 |          ><span class="options">[<a href="#section-InstanceMethods"
 84 |          class="privatelink" onclick="toggle_private();"
 85 |          >hide private</a>]</span></td>
 86 |       </tr>
 87 |     </table>
 88 |   </td>
 89 | </tr>
 90 | <tr>
 91 |     <td width="15%" align="right" valign="top" class="summary">
 92 |       <span class="summary-type">&nbsp;</span>
 93 |     </td><td class="summary">
 94 |       <table width="100%" cellpadding="0" cellspacing="0" border="0">
 95 |         <tr>
 96 |           <td><span class="summary-sig"><a href="extras.SimpleReader-class.html#__call__" class="summary-sig-name">__call__</a>(<span class="summary-sig-arg">self</span>,
 97 |         <span class="summary-sig-arg">text</span>)</span><br />
 98 |       Returns:
 99 |       a list of tags respecting the order in the text</td>
100 |           <td align="right" valign="top">
101 |             <span class="codelink"><a href="extras-pysrc.html#SimpleReader.__call__">source&nbsp;code</a></span>
102 |             
103 |           </td>
104 |         </tr>
105 |       </table>
106 |       
107 |     </td>
108 |   </tr>
109 |   <tr>
110 |     <td colspan="2" class="summary">
111 |     <p class="indent-wrapped-lines"><b>Inherited from <code><a href="tagger.Reader-class.html">tagger.Reader</a></code></b>:
112 |       <code><a href="tagger.Reader-class.html#preprocess">preprocess</a></code>
113 |       </p>
114 |     </td>
115 |   </tr>
116 | </table>
117 | <!-- ==================== CLASS VARIABLES ==================== -->
118 | <a name="section-ClassVariables"></a>
119 | <table class="summary" border="1" cellpadding="3"
120 |        cellspacing="0" width="100%" bgcolor="white">
121 | <tr bgcolor="#70b0f0" class="table-header">
122 |   <td colspan="2" class="table-header">
123 |     <table border="0" cellpadding="0" cellspacing="0" width="100%">
124 |       <tr valign="top">
125 |         <td align="left"><span class="table-header">Class Variables</span></td>
126 |         <td align="right" valign="top"
127 |          ><span class="options">[<a href="#section-ClassVariables"
128 |          class="privatelink" onclick="toggle_private();"
129 |          >hide private</a>]</span></td>
130 |       </tr>
131 |     </table>
132 |   </td>
133 | </tr>
134 |   <tr>
135 |     <td colspan="2" class="summary">
136 |     <p class="indent-wrapped-lines"><b>Inherited from <code><a href="tagger.Reader-class.html">tagger.Reader</a></code></b>:
137 |       <code><a href="tagger.Reader-class.html#match_apostrophes">match_apostrophes</a></code>,
138 |       <code><a href="tagger.Reader-class.html#match_paragraphs">match_paragraphs</a></code>,
139 |       <code><a href="tagger.Reader-class.html#match_phrases">match_phrases</a></code>,
140 |       <code><a href="tagger.Reader-class.html#match_words">match_words</a></code>
141 |       </p>
142 |     </td>
143 |   </tr>
144 | </table>
145 | <!-- ==================== METHOD DETAILS ==================== -->
146 | <a name="section-MethodDetails"></a>
147 | <table class="details" border="1" cellpadding="3"
148 |        cellspacing="0" width="100%" bgcolor="white">
149 | <tr bgcolor="#70b0f0" class="table-header">
150 |   <td colspan="2" class="table-header">
151 |     <table border="0" cellpadding="0" cellspacing="0" width="100%">
152 |       <tr valign="top">
153 |         <td align="left"><span class="table-header">Method Details</span></td>
154 |         <td align="right" valign="top"
155 |          ><span class="options">[<a href="#section-MethodDetails"
156 |          class="privatelink" onclick="toggle_private();"
157 |          >hide private</a>]</span></td>
158 |       </tr>
159 |     </table>
160 |   </td>
161 | </tr>
162 | </table>
163 | <a name="__call__"></a>
164 | <div>
165 | <table class="details" border="1" cellpadding="3"
166 |        cellspacing="0" width="100%" bgcolor="white">
167 | <tr><td>
168 |   <table width="100%" cellpadding="0" cellspacing="0" border="0">
169 |   <tr valign="top"><td>
170 |   <h3 class="epydoc"><span class="sig"><span class="sig-name">__call__</span>(<span class="sig-arg">self</span>,
171 |         <span class="sig-arg">text</span>)</span>
172 |     <br /><em class="fname">(Call operator)</em>
173 |   </h3>
174 |   </td><td align="right" valign="top"
175 |     ><span class="codelink"><a href="extras-pysrc.html#SimpleReader.__call__">source&nbsp;code</a></span>&nbsp;
176 |     </td>
177 |   </tr></table>
178 |   
179 |   
180 |   <dl class="fields">
181 |     <dt>Parameters:</dt>
182 |     <dd><ul class="nomargin-top">
183 |         <li><strong class="pname"><code>text</code></strong> - the string of text to be tagged</li>
184 |     </ul></dd>
185 |     <dt>Returns:</dt>
186 |         <dd>a list of tags respecting the order in the text</dd>
187 |     <dt>Overrides:
188 |         <a href="tagger.Reader-class.html#__call__">tagger.Reader.__call__</a>
189 |         <dd><em class="note">(inherited documentation)</em></dd>
190 |     </dt>
191 |   </dl>
192 | </td></tr></table>
193 | </div>
194 | <br />
195 | <!-- ==================== NAVIGATION BAR ==================== -->
196 | <table class="navbar" border="0" width="100%" cellpadding="0"
197 |        bgcolor="#a0c0ff" cellspacing="0">
198 |   <tr valign="middle">
199 | 
200 |   <!-- Tree link -->
201 |       <th>&nbsp;&nbsp;&nbsp;<a
202 |         href="module-tree.html">Trees</a>&nbsp;&nbsp;&nbsp;</th>
203 | 
204 |   <!-- Index link -->
205 |       <th>&nbsp;&nbsp;&nbsp;<a
206 |         href="identifier-index.html">Indices</a>&nbsp;&nbsp;&nbsp;</th>
207 | 
208 |   <!-- Help link -->
209 |       <th>&nbsp;&nbsp;&nbsp;<a
210 |         href="help.html">Help</a>&nbsp;&nbsp;&nbsp;</th>
211 | 
212 |   <!-- Project homepage -->
213 |       <th class="navbar" align="right" width="100%">
214 |         <table border="0" cellpadding="0" cellspacing="0">
215 |           <tr><th class="navbar" align="center"
216 |             ><a class="navbar" target="_top" href="http://github.com/apresta/tagger">tagger</a></th>
217 |           </tr></table></th>
218 |   </tr>
219 | </table>
220 | <table border="0" cellpadding="0" cellspacing="0" width="100%%">
221 |   <tr>
222 |     <td align="left" class="footer">
223 |     Generated by Epydoc 3.0.1 on Fri May 13 11:13:08 2011
224 |     </td>
225 |     <td align="right" class="footer">
226 |       <a target="mainFrame" href="http://epydoc.sourceforge.net"
227 |         >http://epydoc.sourceforge.net</a>
228 |     </td>
229 |   </tr>
230 | </table>
231 | 
232 | <script type="text/javascript">
233 |   <!--
234 |   // Private objects are initially displayed (because if
235 |   // javascript is turned off then we want them to be
236 |   // visible); but by default, we want to hide them.  So hide
237 |   // them unless we have a cookie that says to show them.
238 |   checkCookie();
239 |   // -->
240 | </script>
241 | </body>
242 | </html>
243 | 


--------------------------------------------------------------------------------
/doc/extras.UnicodeReader-class.html:
--------------------------------------------------------------------------------
  1 | <?xml version="1.0" encoding="ascii"?>
  2 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  3 |           "DTD/xhtml1-transitional.dtd">
  4 | <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
  5 | <head>
  6 |   <title>extras.UnicodeReader</title>
  7 |   <link rel="stylesheet" href="epydoc.css" type="text/css" />
  8 |   <script type="text/javascript" src="epydoc.js"></script>
  9 | </head>
 10 | 
 11 | <body bgcolor="white" text="black" link="blue" vlink="#204080"
 12 |       alink="#204080">
 13 | <!-- ==================== NAVIGATION BAR ==================== -->
 14 | <table class="navbar" border="0" width="100%" cellpadding="0"
 15 |        bgcolor="#a0c0ff" cellspacing="0">
 16 |   <tr valign="middle">
 17 | 
 18 |   <!-- Tree link -->
 19 |       <th>&nbsp;&nbsp;&nbsp;<a
 20 |         href="module-tree.html">Trees</a>&nbsp;&nbsp;&nbsp;</th>
 21 | 
 22 |   <!-- Index link -->
 23 |       <th>&nbsp;&nbsp;&nbsp;<a
 24 |         href="identifier-index.html">Indices</a>&nbsp;&nbsp;&nbsp;</th>
 25 | 
 26 |   <!-- Help link -->
 27 |       <th>&nbsp;&nbsp;&nbsp;<a
 28 |         href="help.html">Help</a>&nbsp;&nbsp;&nbsp;</th>
 29 | 
 30 |   <!-- Project homepage -->
 31 |       <th class="navbar" align="right" width="100%">
 32 |         <table border="0" cellpadding="0" cellspacing="0">
 33 |           <tr><th class="navbar" align="center"
 34 |             ><a class="navbar" target="_top" href="http://github.com/apresta/tagger">tagger</a></th>
 35 |           </tr></table></th>
 36 |   </tr>
 37 | </table>
 38 | <table width="100%" cellpadding="0" cellspacing="0">
 39 |   <tr valign="top">
 40 |     <td width="100%">
 41 |       <span class="breadcrumbs">
 42 |         <a href="extras-module.html">Module&nbsp;extras</a> ::
 43 |         Class&nbsp;UnicodeReader
 44 |       </span>
 45 |     </td>
 46 |     <td>
 47 |       <table cellpadding="0" cellspacing="0">
 48 |         <!-- hide/show private -->
 49 |         <tr><td align="right"><span class="options">[<a href="javascript:void(0);" class="privatelink"
 50 |     onclick="toggle_private();">hide&nbsp;private</a>]</span></td></tr>
 51 |         <tr><td align="right"><span class="options"
 52 |             >[<a href="frames.html" target="_top">frames</a
 53 |             >]&nbsp;|&nbsp;<a href="extras.UnicodeReader-class.html"
 54 |             target="_top">no&nbsp;frames</a>]</span></td></tr>
 55 |       </table>
 56 |     </td>
 57 |   </tr>
 58 | </table>
 59 | <!-- ==================== CLASS DESCRIPTION ==================== -->
 60 | <h1 class="epydoc">Class UnicodeReader</h1><p class="nomargin-top"><span class="codelink"><a href="extras-pysrc.html#UnicodeReader">source&nbsp;code</a></span></p>
 61 | <center>
 62 | <center>  <map id="class_hierarchy_for_unicoderea" name="class_hierarchy_for_unicoderea">
 63 | <area shape="rect" id="node1" href="extras.HTMLReader-class.html" title="HTMLReader" alt="" coords="12,117,116,144"/>
 64 | <area shape="rect" id="node2" href="extras.UnicodeReader-class.html" title="UnicodeReader" alt="" coords="5,61,123,88"/>
 65 | <area shape="rect" id="node3" href="tagger.Reader-class.html" title="tagger.Reader" alt="" coords="9,5,119,32"/>
 66 | </map>
 67 |   <img src="class_hierarchy_for_unicoderea.gif" alt='' usemap="#class_hierarchy_for_unicoderea" ismap="ismap" class="graph-without-title" />
 68 | </center>
 69 | </center>
 70 | <hr />
 71 | <p>Reader subclass that converts Unicode strings to a close ASCII 
 72 |   representation</p>
 73 | 
 74 | <!-- ==================== INSTANCE METHODS ==================== -->
 75 | <a name="section-InstanceMethods"></a>
 76 | <table class="summary" border="1" cellpadding="3"
 77 |        cellspacing="0" width="100%" bgcolor="white">
 78 | <tr bgcolor="#70b0f0" class="table-header">
 79 |   <td colspan="2" class="table-header">
 80 |     <table border="0" cellpadding="0" cellspacing="0" width="100%">
 81 |       <tr valign="top">
 82 |         <td align="left"><span class="table-header">Instance Methods</span></td>
 83 |         <td align="right" valign="top"
 84 |          ><span class="options">[<a href="#section-InstanceMethods"
 85 |          class="privatelink" onclick="toggle_private();"
 86 |          >hide private</a>]</span></td>
 87 |       </tr>
 88 |     </table>
 89 |   </td>
 90 | </tr>
 91 | <tr>
 92 |     <td width="15%" align="right" valign="top" class="summary">
 93 |       <span class="summary-type">&nbsp;</span>
 94 |     </td><td class="summary">
 95 |       <table width="100%" cellpadding="0" cellspacing="0" border="0">
 96 |         <tr>
 97 |           <td><span class="summary-sig"><a href="extras.UnicodeReader-class.html#__call__" class="summary-sig-name">__call__</a>(<span class="summary-sig-arg">self</span>,
 98 |         <span class="summary-sig-arg">text</span>)</span><br />
 99 |       Returns:
100 |       a list of tags respecting the order in the text</td>
101 |           <td align="right" valign="top">
102 |             <span class="codelink"><a href="extras-pysrc.html#UnicodeReader.__call__">source&nbsp;code</a></span>
103 |             
104 |           </td>
105 |         </tr>
106 |       </table>
107 |       
108 |     </td>
109 |   </tr>
110 |   <tr>
111 |     <td colspan="2" class="summary">
112 |     <p class="indent-wrapped-lines"><b>Inherited from <code><a href="tagger.Reader-class.html">tagger.Reader</a></code></b>:
113 |       <code><a href="tagger.Reader-class.html#preprocess">preprocess</a></code>
114 |       </p>
115 |     </td>
116 |   </tr>
117 | </table>
118 | <!-- ==================== CLASS VARIABLES ==================== -->
119 | <a name="section-ClassVariables"></a>
120 | <table class="summary" border="1" cellpadding="3"
121 |        cellspacing="0" width="100%" bgcolor="white">
122 | <tr bgcolor="#70b0f0" class="table-header">
123 |   <td colspan="2" class="table-header">
124 |     <table border="0" cellpadding="0" cellspacing="0" width="100%">
125 |       <tr valign="top">
126 |         <td align="left"><span class="table-header">Class Variables</span></td>
127 |         <td align="right" valign="top"
128 |          ><span class="options">[<a href="#section-ClassVariables"
129 |          class="privatelink" onclick="toggle_private();"
130 |          >hide private</a>]</span></td>
131 |       </tr>
132 |     </table>
133 |   </td>
134 | </tr>
135 |   <tr>
136 |     <td colspan="2" class="summary">
137 |     <p class="indent-wrapped-lines"><b>Inherited from <code><a href="tagger.Reader-class.html">tagger.Reader</a></code></b>:
138 |       <code><a href="tagger.Reader-class.html#match_apostrophes">match_apostrophes</a></code>,
139 |       <code><a href="tagger.Reader-class.html#match_paragraphs">match_paragraphs</a></code>,
140 |       <code><a href="tagger.Reader-class.html#match_phrases">match_phrases</a></code>,
141 |       <code><a href="tagger.Reader-class.html#match_words">match_words</a></code>
142 |       </p>
143 |     </td>
144 |   </tr>
145 | </table>
146 | <!-- ==================== METHOD DETAILS ==================== -->
147 | <a name="section-MethodDetails"></a>
148 | <table class="details" border="1" cellpadding="3"
149 |        cellspacing="0" width="100%" bgcolor="white">
150 | <tr bgcolor="#70b0f0" class="table-header">
151 |   <td colspan="2" class="table-header">
152 |     <table border="0" cellpadding="0" cellspacing="0" width="100%">
153 |       <tr valign="top">
154 |         <td align="left"><span class="table-header">Method Details</span></td>
155 |         <td align="right" valign="top"
156 |          ><span class="options">[<a href="#section-MethodDetails"
157 |          class="privatelink" onclick="toggle_private();"
158 |          >hide private</a>]</span></td>
159 |       </tr>
160 |     </table>
161 |   </td>
162 | </tr>
163 | </table>
164 | <a name="__call__"></a>
165 | <div>
166 | <table class="details" border="1" cellpadding="3"
167 |        cellspacing="0" width="100%" bgcolor="white">
168 | <tr><td>
169 |   <table width="100%" cellpadding="0" cellspacing="0" border="0">
170 |   <tr valign="top"><td>
171 |   <h3 class="epydoc"><span class="sig"><span class="sig-name">__call__</span>(<span class="sig-arg">self</span>,
172 |         <span class="sig-arg">text</span>)</span>
173 |     <br /><em class="fname">(Call operator)</em>
174 |   </h3>
175 |   </td><td align="right" valign="top"
176 |     ><span class="codelink"><a href="extras-pysrc.html#UnicodeReader.__call__">source&nbsp;code</a></span>&nbsp;
177 |     </td>
178 |   </tr></table>
179 |   
180 |   
181 |   <dl class="fields">
182 |     <dt>Parameters:</dt>
183 |     <dd><ul class="nomargin-top">
184 |         <li><strong class="pname"><code>text</code></strong> - the string of text to be tagged</li>
185 |     </ul></dd>
186 |     <dt>Returns:</dt>
187 |         <dd>a list of tags respecting the order in the text</dd>
188 |     <dt>Overrides:
189 |         <a href="tagger.Reader-class.html#__call__">tagger.Reader.__call__</a>
190 |         <dd><em class="note">(inherited documentation)</em></dd>
191 |     </dt>
192 |   </dl>
193 | </td></tr></table>
194 | </div>
195 | <br />
196 | <!-- ==================== NAVIGATION BAR ==================== -->
197 | <table class="navbar" border="0" width="100%" cellpadding="0"
198 |        bgcolor="#a0c0ff" cellspacing="0">
199 |   <tr valign="middle">
200 | 
201 |   <!-- Tree link -->
202 |       <th>&nbsp;&nbsp;&nbsp;<a
203 |         href="module-tree.html">Trees</a>&nbsp;&nbsp;&nbsp;</th>
204 | 
205 |   <!-- Index link -->
206 |       <th>&nbsp;&nbsp;&nbsp;<a
207 |         href="identifier-index.html">Indices</a>&nbsp;&nbsp;&nbsp;</th>
208 | 
209 |   <!-- Help link -->
210 |       <th>&nbsp;&nbsp;&nbsp;<a
211 |         href="help.html">Help</a>&nbsp;&nbsp;&nbsp;</th>
212 | 
213 |   <!-- Project homepage -->
214 |       <th class="navbar" align="right" width="100%">
215 |         <table border="0" cellpadding="0" cellspacing="0">
216 |           <tr><th class="navbar" align="center"
217 |             ><a class="navbar" target="_top" href="http://github.com/apresta/tagger">tagger</a></th>
218 |           </tr></table></th>
219 |   </tr>
220 | </table>
221 | <table border="0" cellpadding="0" cellspacing="0" width="100%%">
222 |   <tr>
223 |     <td align="left" class="footer">
224 |     Generated by Epydoc 3.0.1 on Fri May 13 11:13:08 2011
225 |     </td>
226 |     <td align="right" class="footer">
227 |       <a target="mainFrame" href="http://epydoc.sourceforge.net"
228 |         >http://epydoc.sourceforge.net</a>
229 |     </td>
230 |   </tr>
231 | </table>
232 | 
233 | <script type="text/javascript">
234 |   <!--
235 |   // Private objects are initially displayed (because if
236 |   // javascript is turned off then we want them to be
237 |   // visible); but by default, we want to hide them.  So hide
238 |   // them unless we have a cookie that says to show them.
239 |   checkCookie();
240 |   // -->
241 | </script>
242 | </body>
243 | </html>
244 | 


--------------------------------------------------------------------------------
/doc/frames.html:
--------------------------------------------------------------------------------
 1 | <?xml version="1.0" encoding="iso-8859-1"?>
 2 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Frameset//EN"
 3 |           "DTD/xhtml1-frameset.dtd">
 4 | <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 5 | <head>
 6 |   <title> tagger </title>
 7 | </head>
 8 | <frameset cols="20%,80%">
 9 |   <frameset rows="30%,70%">
10 |     <frame src="toc.html" name="moduleListFrame"
11 |            id="moduleListFrame" />
12 |     <frame src="toc-everything.html" name="moduleFrame"
13 |            id="moduleFrame" />
14 |   </frameset>
15 |   <frame src="module-tree.html" name="mainFrame" id="mainFrame" />
16 | </frameset>
17 | </html>
18 | 


--------------------------------------------------------------------------------
/doc/help.html:
--------------------------------------------------------------------------------
  1 | <?xml version="1.0" encoding="ascii"?>
  2 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  3 |           "DTD/xhtml1-transitional.dtd">
  4 | <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
  5 | <head>
  6 |   <title>Help</title>
  7 |   <link rel="stylesheet" href="epydoc.css" type="text/css" />
  8 |   <script type="text/javascript" src="epydoc.js"></script>
  9 | </head>
 10 | 
 11 | <body bgcolor="white" text="black" link="blue" vlink="#204080"
 12 |       alink="#204080">
 13 | <!-- ==================== NAVIGATION BAR ==================== -->
 14 | <table class="navbar" border="0" width="100%" cellpadding="0"
 15 |        bgcolor="#a0c0ff" cellspacing="0">
 16 |   <tr valign="middle">
 17 | 
 18 |   <!-- Tree link -->
 19 |       <th>&nbsp;&nbsp;&nbsp;<a
 20 |         href="module-tree.html">Trees</a>&nbsp;&nbsp;&nbsp;</th>
 21 | 
 22 |   <!-- Index link -->
 23 |       <th>&nbsp;&nbsp;&nbsp;<a
 24 |         href="identifier-index.html">Indices</a>&nbsp;&nbsp;&nbsp;</th>
 25 | 
 26 |   <!-- Help link -->
 27 |       <th bgcolor="#70b0f0" class="navbar-select"
 28 |           >&nbsp;&nbsp;&nbsp;Help&nbsp;&nbsp;&nbsp;</th>
 29 | 
 30 |   <!-- Project homepage -->
 31 |       <th class="navbar" align="right" width="100%">
 32 |         <table border="0" cellpadding="0" cellspacing="0">
 33 |           <tr><th class="navbar" align="center"
 34 |             ><a class="navbar" target="_top" href="http://github.com/apresta/tagger">tagger</a></th>
 35 |           </tr></table></th>
 36 |   </tr>
 37 | </table>
 38 | <table width="100%" cellpadding="0" cellspacing="0">
 39 |   <tr valign="top">
 40 |     <td width="100%">&nbsp;</td>
 41 |     <td>
 42 |       <table cellpadding="0" cellspacing="0">
 43 |         <!-- hide/show private -->
 44 |         <tr><td align="right"><span class="options">[<a href="javascript:void(0);" class="privatelink"
 45 |     onclick="toggle_private();">hide&nbsp;private</a>]</span></td></tr>
 46 |         <tr><td align="right"><span class="options"
 47 |             >[<a href="frames.html" target="_top">frames</a
 48 |             >]&nbsp;|&nbsp;<a href="help.html"
 49 |             target="_top">no&nbsp;frames</a>]</span></td></tr>
 50 |       </table>
 51 |     </td>
 52 |   </tr>
 53 | </table>
 54 | 
 55 | <h1 class="epydoc"> API Documentation </h1>
 56 | 
 57 | <p> This document contains the API (Application Programming Interface)
 58 | documentation for tagger.  Documentation for the Python
 59 | objects defined by the project is divided into separate pages for each
 60 | package, module, and class.  The API documentation also includes two
 61 | pages containing information about the project as a whole: a trees
 62 | page, and an index page.  </p>
 63 | 
 64 | <h2> Object Documentation </h2>
 65 | 
 66 |   <p>Each <strong>Package Documentation</strong> page contains: </p>
 67 |   <ul>
 68 |     <li> A description of the package. </li>
 69 |     <li> A list of the modules and sub-packages contained by the
 70 |     package.  </li>
 71 |     <li> A summary of the classes defined by the package. </li>
 72 |     <li> A summary of the functions defined by the package. </li>
 73 |     <li> A summary of the variables defined by the package. </li>
 74 |     <li> A detailed description of each function defined by the
 75 |     package. </li>
 76 |     <li> A detailed description of each variable defined by the
 77 |     package. </li>
 78 |   </ul>
 79 |   
 80 |   <p>Each <strong>Module Documentation</strong> page contains:</p>
 81 |   <ul>
 82 |     <li> A description of the module. </li>
 83 |     <li> A summary of the classes defined by the module. </li>
 84 |     <li> A summary of the functions defined by the module. </li>
 85 |     <li> A summary of the variables defined by the module. </li>
 86 |     <li> A detailed description of each function defined by the
 87 |     module. </li>
 88 |     <li> A detailed description of each variable defined by the
 89 |     module. </li>
 90 |   </ul>
 91 |   
 92 |   <p>Each <strong>Class Documentation</strong> page contains: </p>
 93 |   <ul>
 94 |     <li> A class inheritance diagram. </li>
 95 |     <li> A list of known subclasses. </li>
 96 |     <li> A description of the class. </li>
 97 |     <li> A summary of the methods defined by the class. </li>
 98 |     <li> A summary of the instance variables defined by the class. </li>
 99 |     <li> A summary of the class (static) variables defined by the
100 |     class. </li> 
101 |     <li> A detailed description of each method defined by the
102 |     class. </li>
103 |     <li> A detailed description of each instance variable defined by the
104 |     class. </li> 
105 |     <li> A detailed description of each class (static) variable defined
106 |     by the class. </li> 
107 |   </ul>
108 | 
109 | <h2> Project Documentation </h2>
110 | 
111 |   <p> The <strong>Trees</strong> page contains the module and class hierarchies: </p>
112 |   <ul>
113 |     <li> The <em>module hierarchy</em> lists every package and module, with
114 |     modules grouped into packages.  At the top level, and within each
115 |     package, modules and sub-packages are listed alphabetically. </li>
116 |     <li> The <em>class hierarchy</em> lists every class, grouped by base
117 |     class.  If a class has more than one base class, then it will be
118 |     listed under each base class.  At the top level, and under each base
119 |     class, classes are listed alphabetically. </li>
120 |   </ul>
121 |   
122 |   <p> The <strong>Index</strong> page contains indices of terms and
123 |   identifiers: </p>
124 |   <ul>
125 |     <li> The <em>term index</em> lists every term indexed by any object's
126 |     documentation.  For each term, the index provides links to each
127 |     place where the term is indexed. </li>
128 |     <li> The <em>identifier index</em> lists the (short) name of every package,
129 |     module, class, method, function, variable, and parameter.  For each
130 |     identifier, the index provides a short description, and a link to
131 |     its documentation. </li>
132 |   </ul>
133 | 
134 | <h2> The Table of Contents </h2>
135 | 
136 | <p> The table of contents occupies the two frames on the left side of
137 | the window.  The upper-left frame displays the <em>project
138 | contents</em>, and the lower-left frame displays the <em>module
139 | contents</em>: </p>
140 | 
141 | <table class="help summary" border="1" cellspacing="0" cellpadding="3">
142 |   <tr style="height: 30%">
143 |     <td align="center" style="font-size: small">
144 |        Project<br />Contents<hr />...</td>
145 |     <td align="center" style="font-size: small" rowspan="2" width="70%">
146 |       API<br />Documentation<br />Frame<br /><br /><br />
147 |     </td>
148 |   </tr>
149 |   <tr>
150 |     <td align="center" style="font-size: small">
151 |       Module<br />Contents<hr />&nbsp;<br />...<br />&nbsp;
152 |     </td>
153 |   </tr>
154 | </table><br />
155 | 
156 | <p> The <strong>project contents frame</strong> contains a list of all packages
157 | and modules that are defined by the project.  Clicking on an entry
158 | will display its contents in the module contents frame.  Clicking on a
159 | special entry, labeled "Everything," will display the contents of
160 | the entire project. </p>
161 | 
162 | <p> The <strong>module contents frame</strong> contains a list of every
163 | submodule, class, type, exception, function, and variable defined by a
164 | module or package.  Clicking on an entry will display its
165 | documentation in the API documentation frame.  Clicking on the name of
166 | the module, at the top of the frame, will display the documentation
167 | for the module itself. </p>
168 | 
169 | <p> The "<strong>frames</strong>" and "<strong>no frames</strong>" buttons below the top
170 | navigation bar can be used to control whether the table of contents is
171 | displayed or not. </p>
172 | 
173 | <h2> The Navigation Bar </h2>
174 | 
175 | <p> A navigation bar is located at the top and bottom of every page.
176 | It indicates what type of page you are currently viewing, and allows
177 | you to go to related pages.  The following table describes the labels
178 | on the navigation bar.  Note that not some labels (such as
179 | [Parent]) are not displayed on all pages. </p>
180 | 
181 | <table class="summary" border="1" cellspacing="0" cellpadding="3" width="100%">
182 | <tr class="summary">
183 |   <th>Label</th>
184 |   <th>Highlighted when...</th>
185 |   <th>Links to...</th>
186 | </tr>
187 |   <tr><td valign="top"><strong>[Parent]</strong></td>
188 |       <td valign="top"><em>(never highlighted)</em></td>
189 |       <td valign="top"> the parent of the current package </td></tr>
190 |   <tr><td valign="top"><strong>[Package]</strong></td>
191 |       <td valign="top">viewing a package</td>
192 |       <td valign="top">the package containing the current object
193 |       </td></tr>
194 |   <tr><td valign="top"><strong>[Module]</strong></td>
195 |       <td valign="top">viewing a module</td>
196 |       <td valign="top">the module containing the current object
197 |       </td></tr> 
198 |   <tr><td valign="top"><strong>[Class]</strong></td>
199 |       <td valign="top">viewing a class </td>
200 |       <td valign="top">the class containing the current object</td></tr>
201 |   <tr><td valign="top"><strong>[Trees]</strong></td>
202 |       <td valign="top">viewing the trees page</td>
203 |       <td valign="top"> the trees page </td></tr>
204 |   <tr><td valign="top"><strong>[Index]</strong></td>
205 |       <td valign="top">viewing the index page</td>
206 |       <td valign="top"> the index page </td></tr>
207 |   <tr><td valign="top"><strong>[Help]</strong></td>
208 |       <td valign="top">viewing the help page</td>
209 |       <td valign="top"> the help page </td></tr>
210 | </table>
211 | 
212 | <p> The "<strong>show private</strong>" and "<strong>hide private</strong>" buttons below
213 | the top navigation bar can be used to control whether documentation
214 | for private objects is displayed.  Private objects are usually defined
215 | as objects whose (short) names begin with a single underscore, but do
216 | not end with an underscore.  For example, "<code>_x</code>",
217 | "<code>__pprint</code>", and "<code>epydoc.epytext._tokenize</code>"
218 | are private objects; but "<code>re.sub</code>",
219 | "<code>__init__</code>", and "<code>type_</code>" are not.  However,
220 | if a module defines the "<code>__all__</code>" variable, then its
221 | contents are used to decide which objects are private. </p>
222 | 
223 | <p> A timestamp below the bottom navigation bar indicates when each
224 | page was last updated. </p>
225 | <!-- ==================== NAVIGATION BAR ==================== -->
226 | <table class="navbar" border="0" width="100%" cellpadding="0"
227 |        bgcolor="#a0c0ff" cellspacing="0">
228 |   <tr valign="middle">
229 | 
230 |   <!-- Tree link -->
231 |       <th>&nbsp;&nbsp;&nbsp;<a
232 |         href="module-tree.html">Trees</a>&nbsp;&nbsp;&nbsp;</th>
233 | 
234 |   <!-- Index link -->
235 |       <th>&nbsp;&nbsp;&nbsp;<a
236 |         href="identifier-index.html">Indices</a>&nbsp;&nbsp;&nbsp;</th>
237 | 
238 |   <!-- Help link -->
239 |       <th bgcolor="#70b0f0" class="navbar-select"
240 |           >&nbsp;&nbsp;&nbsp;Help&nbsp;&nbsp;&nbsp;</th>
241 | 
242 |   <!-- Project homepage -->
243 |       <th class="navbar" align="right" width="100%">
244 |         <table border="0" cellpadding="0" cellspacing="0">
245 |           <tr><th class="navbar" align="center"
246 |             ><a class="navbar" target="_top" href="http://github.com/apresta/tagger">tagger</a></th>
247 |           </tr></table></th>
248 |   </tr>
249 | </table>
250 | <table border="0" cellpadding="0" cellspacing="0" width="100%%">
251 |   <tr>
252 |     <td align="left" class="footer">
253 |     Generated by Epydoc 3.0.1 on Fri May 13 11:13:02 2011
254 |     </td>
255 |     <td align="right" class="footer">
256 |       <a target="mainFrame" href="http://epydoc.sourceforge.net"
257 |         >http://epydoc.sourceforge.net</a>
258 |     </td>
259 |   </tr>
260 | </table>
261 | 
262 | <script type="text/javascript">
263 |   <!--
264 |   // Private objects are initially displayed (because if
265 |   // javascript is turned off then we want them to be
266 |   // visible); but by default, we want to hide them.  So hide
267 |   // them unless we have a cookie that says to show them.
268 |   checkCookie();
269 |   // -->
270 | </script>
271 | </body>
272 | </html>
273 | 


--------------------------------------------------------------------------------
/doc/index.html:
--------------------------------------------------------------------------------
 1 | <?xml version="1.0" encoding="iso-8859-1"?>
 2 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Frameset//EN"
 3 |           "DTD/xhtml1-frameset.dtd">
 4 | <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 5 | <head>
 6 |   <title> tagger </title>
 7 | </head>
 8 | <frameset cols="20%,80%">
 9 |   <frameset rows="30%,70%">
10 |     <frame src="toc.html" name="moduleListFrame"
11 |            id="moduleListFrame" />
12 |     <frame src="toc-everything.html" name="moduleFrame"
13 |            id="moduleFrame" />
14 |   </frameset>
15 |   <frame src="module-tree.html" name="mainFrame" id="mainFrame" />
16 | </frameset>
17 | </html>
18 | 


--------------------------------------------------------------------------------
/doc/module-tree.html:
--------------------------------------------------------------------------------
  1 | <?xml version="1.0" encoding="ascii"?>
  2 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  3 |           "DTD/xhtml1-transitional.dtd">
  4 | <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
  5 | <head>
  6 |   <title>Module Hierarchy</title>
  7 |   <link rel="stylesheet" href="epydoc.css" type="text/css" />
  8 |   <script type="text/javascript" src="epydoc.js"></script>
  9 | </head>
 10 | 
 11 | <body bgcolor="white" text="black" link="blue" vlink="#204080"
 12 |       alink="#204080">
 13 | <!-- ==================== NAVIGATION BAR ==================== -->
 14 | <table class="navbar" border="0" width="100%" cellpadding="0"
 15 |        bgcolor="#a0c0ff" cellspacing="0">
 16 |   <tr valign="middle">
 17 | 
 18 |   <!-- Tree link -->
 19 |       <th bgcolor="#70b0f0" class="navbar-select"
 20 |           >&nbsp;&nbsp;&nbsp;Trees&nbsp;&nbsp;&nbsp;</th>
 21 | 
 22 |   <!-- Index link -->
 23 |       <th>&nbsp;&nbsp;&nbsp;<a
 24 |         href="identifier-index.html">Indices</a>&nbsp;&nbsp;&nbsp;</th>
 25 | 
 26 |   <!-- Help link -->
 27 |       <th>&nbsp;&nbsp;&nbsp;<a
 28 |         href="help.html">Help</a>&nbsp;&nbsp;&nbsp;</th>
 29 | 
 30 |   <!-- Project homepage -->
 31 |       <th class="navbar" align="right" width="100%">
 32 |         <table border="0" cellpadding="0" cellspacing="0">
 33 |           <tr><th class="navbar" align="center"
 34 |             ><a class="navbar" target="_top" href="http://github.com/apresta/tagger">tagger</a></th>
 35 |           </tr></table></th>
 36 |   </tr>
 37 | </table>
 38 | <table width="100%" cellpadding="0" cellspacing="0">
 39 |   <tr valign="top">
 40 |     <td width="100%">&nbsp;</td>
 41 |     <td>
 42 |       <table cellpadding="0" cellspacing="0">
 43 |         <!-- hide/show private -->
 44 |         <tr><td align="right"><span class="options">[<a href="javascript:void(0);" class="privatelink"
 45 |     onclick="toggle_private();">hide&nbsp;private</a>]</span></td></tr>
 46 |         <tr><td align="right"><span class="options"
 47 |             >[<a href="frames.html" target="_top">frames</a
 48 |             >]&nbsp;|&nbsp;<a href="module-tree.html"
 49 |             target="_top">no&nbsp;frames</a>]</span></td></tr>
 50 |       </table>
 51 |     </td>
 52 |   </tr>
 53 | </table>
 54 | <center><b>
 55 |  [ <a href="module-tree.html">Module Hierarchy</a>
 56 |  | <a href="class-tree.html">Class Hierarchy</a> ]
 57 | </b></center><br />
 58 | <h1 class="epydoc">Module Hierarchy</h1>
 59 | <ul class="nomargin-top">
 60 |     <li> <strong class="uidlink"><a href="build_dict-module.html">build_dict</a></strong>: <em class="summary">Usage: build_dict.py -o &lt;output file&gt; -s &lt;stopwords 
 61 |         file&gt; &lt;list of files&gt;</em>    </li>
 62 |     <li> <strong class="uidlink"><a href="extras-module.html">extras</a></strong>    </li>
 63 |     <li> <strong class="uidlink"><a href="tagger-module.html">tagger</a></strong>: <em class="summary">====== tagger ======</em>    </li>
 64 | </ul>
 65 | <!-- ==================== NAVIGATION BAR ==================== -->
 66 | <table class="navbar" border="0" width="100%" cellpadding="0"
 67 |        bgcolor="#a0c0ff" cellspacing="0">
 68 |   <tr valign="middle">
 69 | 
 70 |   <!-- Tree link -->
 71 |       <th bgcolor="#70b0f0" class="navbar-select"
 72 |           >&nbsp;&nbsp;&nbsp;Trees&nbsp;&nbsp;&nbsp;</th>
 73 | 
 74 |   <!-- Index link -->
 75 |       <th>&nbsp;&nbsp;&nbsp;<a
 76 |         href="identifier-index.html">Indices</a>&nbsp;&nbsp;&nbsp;</th>
 77 | 
 78 |   <!-- Help link -->
 79 |       <th>&nbsp;&nbsp;&nbsp;<a
 80 |         href="help.html">Help</a>&nbsp;&nbsp;&nbsp;</th>
 81 | 
 82 |   <!-- Project homepage -->
 83 |       <th class="navbar" align="right" width="100%">
 84 |         <table border="0" cellpadding="0" cellspacing="0">
 85 |           <tr><th class="navbar" align="center"
 86 |             ><a class="navbar" target="_top" href="http://github.com/apresta/tagger">tagger</a></th>
 87 |           </tr></table></th>
 88 |   </tr>
 89 | </table>
 90 | <table border="0" cellpadding="0" cellspacing="0" width="100%%">
 91 |   <tr>
 92 |     <td align="left" class="footer">
 93 |     Generated by Epydoc 3.0.1 on Fri May 13 11:13:02 2011
 94 |     </td>
 95 |     <td align="right" class="footer">
 96 |       <a target="mainFrame" href="http://epydoc.sourceforge.net"
 97 |         >http://epydoc.sourceforge.net</a>
 98 |     </td>
 99 |   </tr>
100 | </table>
101 | 
102 | <script type="text/javascript">
103 |   <!--
104 |   // Private objects are initially displayed (because if
105 |   // javascript is turned off then we want them to be
106 |   // visible); but by default, we want to hide them.  So hide
107 |   // them unless we have a cookie that says to show them.
108 |   checkCookie();
109 |   // -->
110 | </script>
111 | </body>
112 | </html>
113 | 


--------------------------------------------------------------------------------
/doc/redirect.html:
--------------------------------------------------------------------------------
 1 | <html><head><title>Epydoc Redirect Page</title>
 2 | <meta http-equiv="cache-control" content="no-cache" />
 3 | <meta http-equiv="expires" content="0" />
 4 | <meta http-equiv="pragma" content="no-cache" />
 5 |   <script type="text/javascript" src="epydoc.js"></script>
 6 | </head>
 7 | <body>
 8 | <script type="text/javascript">
 9 | <!--
10 | var pages = ["extras.UnicodeReader-c", "extras.SimpleReader-c", "extras.FastStemmer-c", "extras.HTMLReader-c", "extras.NaiveRater-c", "tagger.MultiTag-c", "tagger.Stemmer-c", "tagger.Reader-c", "tagger.Tagger-c", "tagger.Rater-c", "build_dict-m", "tagger.Tag-c", "extras-m", "tagger-m"];
11 | var dottedName = get_anchor();
12 | if (dottedName) {
13 |     var target = redirect_url(dottedName);
14 |     if (target) window.location.replace(target);
15 | }
16 | // -->
17 | </script>
18 | 
19 | <h3>Epydoc Auto-redirect page</h3>
20 | 
21 | <p>When javascript is enabled, this page will redirect URLs of
22 | the form <tt>redirect.html#<i>dotted.name</i></tt> to the
23 | documentation for the object with the given fully-qualified
24 | dotted name.</p>
25 | <p><a id="message"> &nbsp; </a></p>
26 | 
27 | <script type="text/javascript">
28 | <!--
29 | if (dottedName) {
30 |     var msg = document.getElementById("message");
31 |     msg.innerHTML = "No documentation found for <tt>"+
32 |                     dottedName+"</tt>";
33 | }
34 | // -->
35 | </script>
36 | 
37 | </body>
38 | </html>
39 | 


--------------------------------------------------------------------------------
/doc/tagger-module.html:
--------------------------------------------------------------------------------
  1 | <?xml version="1.0" encoding="ascii"?>
  2 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  3 |           "DTD/xhtml1-transitional.dtd">
  4 | <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
  5 | <head>
  6 |   <title>tagger</title>
  7 |   <link rel="stylesheet" href="epydoc.css" type="text/css" />
  8 |   <script type="text/javascript" src="epydoc.js"></script>
  9 | </head>
 10 | 
 11 | <body bgcolor="white" text="black" link="blue" vlink="#204080"
 12 |       alink="#204080">
 13 | <!-- ==================== NAVIGATION BAR ==================== -->
 14 | <table class="navbar" border="0" width="100%" cellpadding="0"
 15 |        bgcolor="#a0c0ff" cellspacing="0">
 16 |   <tr valign="middle">
 17 | 
 18 |   <!-- Tree link -->
 19 |       <th>&nbsp;&nbsp;&nbsp;<a
 20 |         href="module-tree.html">Trees</a>&nbsp;&nbsp;&nbsp;</th>
 21 | 
 22 |   <!-- Index link -->
 23 |       <th>&nbsp;&nbsp;&nbsp;<a
 24 |         href="identifier-index.html">Indices</a>&nbsp;&nbsp;&nbsp;</th>
 25 | 
 26 |   <!-- Help link -->
 27 |       <th>&nbsp;&nbsp;&nbsp;<a
 28 |         href="help.html">Help</a>&nbsp;&nbsp;&nbsp;</th>
 29 | 
 30 |   <!-- Project homepage -->
 31 |       <th class="navbar" align="right" width="100%">
 32 |         <table border="0" cellpadding="0" cellspacing="0">
 33 |           <tr><th class="navbar" align="center"
 34 |             ><a class="navbar" target="_top" href="http://github.com/apresta/tagger">tagger</a></th>
 35 |           </tr></table></th>
 36 |   </tr>
 37 | </table>
 38 | <table width="100%" cellpadding="0" cellspacing="0">
 39 |   <tr valign="top">
 40 |     <td width="100%">
 41 |       <span class="breadcrumbs">
 42 |         Module&nbsp;tagger
 43 |       </span>
 44 |     </td>
 45 |     <td>
 46 |       <table cellpadding="0" cellspacing="0">
 47 |         <!-- hide/show private -->
 48 |         <tr><td align="right"><span class="options">[<a href="javascript:void(0);" class="privatelink"
 49 |     onclick="toggle_private();">hide&nbsp;private</a>]</span></td></tr>
 50 |         <tr><td align="right"><span class="options"
 51 |             >[<a href="frames.html" target="_top">frames</a
 52 |             >]&nbsp;|&nbsp;<a href="tagger-module.html"
 53 |             target="_top">no&nbsp;frames</a>]</span></td></tr>
 54 |       </table>
 55 |     </td>
 56 |   </tr>
 57 | </table>
 58 | <!-- ==================== MODULE DESCRIPTION ==================== -->
 59 | <h1 class="epydoc">Module tagger</h1><p class="nomargin-top"><span class="codelink"><a href="tagger-pysrc.html">source&nbsp;code</a></span></p>
 60 | <p>====== tagger ======</p>
 61 |   <p>Module for extracting tags from text documents.</p>
 62 |   <p>Copyright (C) 2011 by Alessandro Presta</p>
 63 |   <h1 class="heading">Configuration</h1>
 64 |     <p>Dependencies: python2.7, stemming, nltk (optional), lxml (optional),
 65 |     tkinter (optional)</p>
 66 |     <p>You can install the stemming package with:</p>
 67 | <pre class="literalblock">
 68 |    $ easy_install stemming
 69 | </pre>
 70 |   <h1 class="heading">Usage</h1>
 71 |     <p>Tagging a text document from Python:</p>
 72 | <pre class="literalblock">
 73 |    import tagger
 74 |    weights = pickle.load(open('data/dict.pkl', 'rb')) # or your own dictionary
 75 |    myreader = tagger.Reader() # or your own reader class
 76 |    mystemmer = tagger.Stemmer() # or your own stemmer class
 77 |    myrater = tagger.Rater(weights) # or your own... (you got the idea)
 78 |    mytagger = Tagger(myreader, mystemmer, myrater)
 79 |    best_3_tags = mytagger(text_string, 3)
 80 | </pre>
 81 |     <p>Running the module as a script:</p>
 82 | <pre class="literalblock">
 83 |    $ ./tagger.py &lt;text document(s) to tag&gt;
 84 | </pre>
 85 |     <p>Example:</p>
 86 | <pre class="literalblock">
 87 |    $ ./tagger.py tests/*
 88 |    Loading dictionary... 
 89 |    Tags for  tests/bbc1.txt :
 90 |    ['bin laden', 'obama', 'pakistan', 'killed', 'raid']
 91 |    Tags for  tests/bbc2.txt :
 92 |    ['jo yeates', 'bristol', 'vincent tabak', 'murder', 'strangled']
 93 |    Tags for  tests/bbc3.txt :
 94 |    ['snp', 'party', 'election', 'scottish', 'labour']
 95 |    Tags for  tests/guardian1.txt :
 96 |    ['bin laden', 'al-qaida', 'killed', 'pakistan', 'al-fawwaz']
 97 |    Tags for  tests/guardian2.txt :
 98 |    ['clegg', 'tory', 'lib dem', 'party', 'coalition']
 99 |    Tags for  tests/post1.txt :
100 |    ['sony', 'stolen', 'playstation network', 'hacker attack', 'lawsuit']
101 |    Tags for  tests/wikipedia1.txt :
102 |    ['universe', 'anthropic principle', 'observed', 'cosmological', 'theory']
103 |    Tags for  tests/wikipedia2.txt :
104 |    ['beetroot', 'beet', 'betaine', 'blood pressure', 'dietary nitrate']
105 |    Tags for  tests/wikipedia3.txt :
106 |    ['the lounge lizards', 'jazz', 'john lurie', 'musical', 'albums']
107 | </pre>
108 | 
109 | <!-- ==================== CLASSES ==================== -->
110 | <a name="section-Classes"></a>
111 | <table class="summary" border="1" cellpadding="3"
112 |        cellspacing="0" width="100%" bgcolor="white">
113 | <tr bgcolor="#70b0f0" class="table-header">
114 |   <td colspan="2" class="table-header">
115 |     <table border="0" cellpadding="0" cellspacing="0" width="100%">
116 |       <tr valign="top">
117 |         <td align="left"><span class="table-header">Classes</span></td>
118 |         <td align="right" valign="top"
119 |          ><span class="options">[<a href="#section-Classes"
120 |          class="privatelink" onclick="toggle_private();"
121 |          >hide private</a>]</span></td>
122 |       </tr>
123 |     </table>
124 |   </td>
125 | </tr>
126 | <tr>
127 |     <td width="15%" align="right" valign="top" class="summary">
128 |       <span class="summary-type">&nbsp;</span>
129 |     </td><td class="summary">
130 |         <a href="tagger.Tag-class.html" class="summary-name">Tag</a><br />
131 |       General class for tags (small units of text)
132 |     </td>
133 |   </tr>
134 | <tr>
135 |     <td width="15%" align="right" valign="top" class="summary">
136 |       <span class="summary-type">&nbsp;</span>
137 |     </td><td class="summary">
138 |         <a href="tagger.MultiTag-class.html" class="summary-name">MultiTag</a><br />
139 |       Class for aggregates of tags (usually next to each other in the 
140 |         document)
141 |     </td>
142 |   </tr>
143 | <tr>
144 |     <td width="15%" align="right" valign="top" class="summary">
145 |       <span class="summary-type">&nbsp;</span>
146 |     </td><td class="summary">
147 |         <a href="tagger.Reader-class.html" class="summary-name">Reader</a><br />
148 |       Class for parsing a string of text to obtain tags
149 |     </td>
150 |   </tr>
151 | <tr>
152 |     <td width="15%" align="right" valign="top" class="summary">
153 |       <span class="summary-type">&nbsp;</span>
154 |     </td><td class="summary">
155 |         <a href="tagger.Stemmer-class.html" class="summary-name">Stemmer</a><br />
156 |       Class for extracting the stem of a word
157 |     </td>
158 |   </tr>
159 | <tr>
160 |     <td width="15%" align="right" valign="top" class="summary">
161 |       <span class="summary-type">&nbsp;</span>
162 |     </td><td class="summary">
163 |         <a href="tagger.Rater-class.html" class="summary-name">Rater</a><br />
164 |       Class for estimating the relevance of tags
165 |     </td>
166 |   </tr>
167 | <tr>
168 |     <td width="15%" align="right" valign="top" class="summary">
169 |       <span class="summary-type">&nbsp;</span>
170 |     </td><td class="summary">
171 |         <a href="tagger.Tagger-class.html" class="summary-name">Tagger</a><br />
172 |       Master class for tagging text documents
173 |     </td>
174 |   </tr>
175 | </table>
176 | <!-- ==================== VARIABLES ==================== -->
177 | <a name="section-Variables"></a>
178 | <table class="summary" border="1" cellpadding="3"
179 |        cellspacing="0" width="100%" bgcolor="white">
180 | <tr bgcolor="#70b0f0" class="table-header">
181 |   <td colspan="2" class="table-header">
182 |     <table border="0" cellpadding="0" cellspacing="0" width="100%">
183 |       <tr valign="top">
184 |         <td align="left"><span class="table-header">Variables</span></td>
185 |         <td align="right" valign="top"
186 |          ><span class="options">[<a href="#section-Variables"
187 |          class="privatelink" onclick="toggle_private();"
188 |          >hide private</a>]</span></td>
189 |       </tr>
190 |     </table>
191 |   </td>
192 | </tr>
193 | <tr>
194 |     <td width="15%" align="right" valign="top" class="summary">
195 |       <span class="summary-type">&nbsp;</span>
196 |     </td><td class="summary">
197 |         <a name="__package__"></a><span class="summary-name">__package__</span> = <code title="None">None</code>
198 |     </td>
199 |   </tr>
200 | </table>
201 | <!-- ==================== NAVIGATION BAR ==================== -->
202 | <table class="navbar" border="0" width="100%" cellpadding="0"
203 |        bgcolor="#a0c0ff" cellspacing="0">
204 |   <tr valign="middle">
205 | 
206 |   <!-- Tree link -->
207 |       <th>&nbsp;&nbsp;&nbsp;<a
208 |         href="module-tree.html">Trees</a>&nbsp;&nbsp;&nbsp;</th>
209 | 
210 |   <!-- Index link -->
211 |       <th>&nbsp;&nbsp;&nbsp;<a
212 |         href="identifier-index.html">Indices</a>&nbsp;&nbsp;&nbsp;</th>
213 | 
214 |   <!-- Help link -->
215 |       <th>&nbsp;&nbsp;&nbsp;<a
216 |         href="help.html">Help</a>&nbsp;&nbsp;&nbsp;</th>
217 | 
218 |   <!-- Project homepage -->
219 |       <th class="navbar" align="right" width="100%">
220 |         <table border="0" cellpadding="0" cellspacing="0">
221 |           <tr><th class="navbar" align="center"
222 |             ><a class="navbar" target="_top" href="http://github.com/apresta/tagger">tagger</a></th>
223 |           </tr></table></th>
224 |   </tr>
225 | </table>
226 | <table border="0" cellpadding="0" cellspacing="0" width="100%%">
227 |   <tr>
228 |     <td align="left" class="footer">
229 |     Generated by Epydoc 3.0.1 on Fri May 13 11:13:02 2011
230 |     </td>
231 |     <td align="right" class="footer">
232 |       <a target="mainFrame" href="http://epydoc.sourceforge.net"
233 |         >http://epydoc.sourceforge.net</a>
234 |     </td>
235 |   </tr>
236 | </table>
237 | 
238 | <script type="text/javascript">
239 |   <!--
240 |   // Private objects are initially displayed (because if
241 |   // javascript is turned off then we want them to be
242 |   // visible); but by default, we want to hide them.  So hide
243 |   // them unless we have a cookie that says to show them.
244 |   checkCookie();
245 |   // -->
246 | </script>
247 | </body>
248 | </html>
249 | 


--------------------------------------------------------------------------------
/doc/tagger.MultiTag-class.html:
--------------------------------------------------------------------------------
  1 | <?xml version="1.0" encoding="ascii"?>
  2 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  3 |           "DTD/xhtml1-transitional.dtd">
  4 | <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
  5 | <head>
  6 |   <title>tagger.MultiTag</title>
  7 |   <link rel="stylesheet" href="epydoc.css" type="text/css" />
  8 |   <script type="text/javascript" src="epydoc.js"></script>
  9 | </head>
 10 | 
 11 | <body bgcolor="white" text="black" link="blue" vlink="#204080"
 12 |       alink="#204080">
 13 | <!-- ==================== NAVIGATION BAR ==================== -->
 14 | <table class="navbar" border="0" width="100%" cellpadding="0"
 15 |        bgcolor="#a0c0ff" cellspacing="0">
 16 |   <tr valign="middle">
 17 | 
 18 |   <!-- Tree link -->
 19 |       <th>&nbsp;&nbsp;&nbsp;<a
 20 |         href="module-tree.html">Trees</a>&nbsp;&nbsp;&nbsp;</th>
 21 | 
 22 |   <!-- Index link -->
 23 |       <th>&nbsp;&nbsp;&nbsp;<a
 24 |         href="identifier-index.html">Indices</a>&nbsp;&nbsp;&nbsp;</th>
 25 | 
 26 |   <!-- Help link -->
 27 |       <th>&nbsp;&nbsp;&nbsp;<a
 28 |         href="help.html">Help</a>&nbsp;&nbsp;&nbsp;</th>
 29 | 
 30 |   <!-- Project homepage -->
 31 |       <th class="navbar" align="right" width="100%">
 32 |         <table border="0" cellpadding="0" cellspacing="0">
 33 |           <tr><th class="navbar" align="center"
 34 |             ><a class="navbar" target="_top" href="http://github.com/apresta/tagger">tagger</a></th>
 35 |           </tr></table></th>
 36 |   </tr>
 37 | </table>
 38 | <table width="100%" cellpadding="0" cellspacing="0">
 39 |   <tr valign="top">
 40 |     <td width="100%">
 41 |       <span class="breadcrumbs">
 42 |         <a href="tagger-module.html">Module&nbsp;tagger</a> ::
 43 |         Class&nbsp;MultiTag
 44 |       </span>
 45 |     </td>
 46 |     <td>
 47 |       <table cellpadding="0" cellspacing="0">
 48 |         <!-- hide/show private -->
 49 |         <tr><td align="right"><span class="options">[<a href="javascript:void(0);" class="privatelink"
 50 |     onclick="toggle_private();">hide&nbsp;private</a>]</span></td></tr>
 51 |         <tr><td align="right"><span class="options"
 52 |             >[<a href="frames.html" target="_top">frames</a
 53 |             >]&nbsp;|&nbsp;<a href="tagger.MultiTag-class.html"
 54 |             target="_top">no&nbsp;frames</a>]</span></td></tr>
 55 |       </table>
 56 |     </td>
 57 |   </tr>
 58 | </table>
 59 | <!-- ==================== CLASS DESCRIPTION ==================== -->
 60 | <h1 class="epydoc">Class MultiTag</h1><p class="nomargin-top"><span class="codelink"><a href="tagger-pysrc.html#MultiTag">source&nbsp;code</a></span></p>
 61 | <center>
 62 | <center>  <map id="class_hierarchy_for_multitag" name="class_hierarchy_for_multitag">
 63 | <area shape="rect" id="node1" href="tagger.MultiTag-class.html" title="MultiTag" alt="" coords="5,61,80,88"/>
 64 | <area shape="rect" id="node2" href="tagger.Tag-class.html" title="Tag" alt="" coords="20,5,65,32"/>
 65 | </map>
 66 |   <img src="class_hierarchy_for_multitag.gif" alt='' usemap="#class_hierarchy_for_multitag" ismap="ismap" class="graph-without-title" />
 67 | </center>
 68 | </center>
 69 | <hr />
 70 | <p>Class for aggregates of tags (usually next to each other in the 
 71 |   document)</p>
 72 | 
 73 | <!-- ==================== INSTANCE METHODS ==================== -->
 74 | <a name="section-InstanceMethods"></a>
 75 | <table class="summary" border="1" cellpadding="3"
 76 |        cellspacing="0" width="100%" bgcolor="white">
 77 | <tr bgcolor="#70b0f0" class="table-header">
 78 |   <td colspan="2" class="table-header">
 79 |     <table border="0" cellpadding="0" cellspacing="0" width="100%">
 80 |       <tr valign="top">
 81 |         <td align="left"><span class="table-header">Instance Methods</span></td>
 82 |         <td align="right" valign="top"
 83 |          ><span class="options">[<a href="#section-InstanceMethods"
 84 |          class="privatelink" onclick="toggle_private();"
 85 |          >hide private</a>]</span></td>
 86 |       </tr>
 87 |     </table>
 88 |   </td>
 89 | </tr>
 90 | <tr>
 91 |     <td width="15%" align="right" valign="top" class="summary">
 92 |       <span class="summary-type">&nbsp;</span>
 93 |     </td><td class="summary">
 94 |       <table width="100%" cellpadding="0" cellspacing="0" border="0">
 95 |         <tr>
 96 |           <td><span class="summary-sig"><a href="tagger.MultiTag-class.html#__init__" class="summary-sig-name">__init__</a>(<span class="summary-sig-arg">self</span>,
 97 |         <span class="summary-sig-arg">tail</span>,
 98 |         <span class="summary-sig-arg">head</span>=<span class="summary-sig-default">None</span>)</span><br />
 99 |       Returns:
100 |       a new <a href="tagger.MultiTag-class.html" class="link">MultiTag</a> 
101 |       object</td>
102 |           <td align="right" valign="top">
103 |             <span class="codelink"><a href="tagger-pysrc.html#MultiTag.__init__">source&nbsp;code</a></span>
104 |             
105 |           </td>
106 |         </tr>
107 |       </table>
108 |       
109 |     </td>
110 |   </tr>
111 | <tr>
112 |     <td width="15%" align="right" valign="top" class="summary">
113 |       <span class="summary-type">&nbsp;</span>
114 |     </td><td class="summary">
115 |       <table width="100%" cellpadding="0" cellspacing="0" border="0">
116 |         <tr>
117 |           <td><span class="summary-sig"><a href="tagger.MultiTag-class.html#combined_rating" class="summary-sig-name">combined_rating</a>(<span class="summary-sig-arg">self</span>)</span><br />
118 |       Method that computes the multitag's rating from the ratings of unit 
119 |       subtags</td>
120 |           <td align="right" valign="top">
121 |             <span class="codelink"><a href="tagger-pysrc.html#MultiTag.combined_rating">source&nbsp;code</a></span>
122 |             
123 |           </td>
124 |         </tr>
125 |       </table>
126 |       
127 |     </td>
128 |   </tr>
129 |   <tr>
130 |     <td colspan="2" class="summary">
131 |     <p class="indent-wrapped-lines"><b>Inherited from <code><a href="tagger.Tag-class.html">Tag</a></code></b>:
132 |       <code><a href="tagger.Tag-class.html#__eq__">__eq__</a></code>,
133 |       <code><a href="tagger.Tag-class.html#__hash__">__hash__</a></code>,
134 |       <code><a href="tagger.Tag-class.html#__lt__">__lt__</a></code>,
135 |       <code><a href="tagger.Tag-class.html#__repr__">__repr__</a></code>
136 |       </p>
137 |     </td>
138 |   </tr>
139 | </table>
140 | <!-- ==================== METHOD DETAILS ==================== -->
141 | <a name="section-MethodDetails"></a>
142 | <table class="details" border="1" cellpadding="3"
143 |        cellspacing="0" width="100%" bgcolor="white">
144 | <tr bgcolor="#70b0f0" class="table-header">
145 |   <td colspan="2" class="table-header">
146 |     <table border="0" cellpadding="0" cellspacing="0" width="100%">
147 |       <tr valign="top">
148 |         <td align="left"><span class="table-header">Method Details</span></td>
149 |         <td align="right" valign="top"
150 |          ><span class="options">[<a href="#section-MethodDetails"
151 |          class="privatelink" onclick="toggle_private();"
152 |          >hide private</a>]</span></td>
153 |       </tr>
154 |     </table>
155 |   </td>
156 | </tr>
157 | </table>
158 | <a name="__init__"></a>
159 | <div>
160 | <table class="details" border="1" cellpadding="3"
161 |        cellspacing="0" width="100%" bgcolor="white">
162 | <tr><td>
163 |   <table width="100%" cellpadding="0" cellspacing="0" border="0">
164 |   <tr valign="top"><td>
165 |   <h3 class="epydoc"><span class="sig"><span class="sig-name">__init__</span>(<span class="sig-arg">self</span>,
166 |         <span class="sig-arg">tail</span>,
167 |         <span class="sig-arg">head</span>=<span class="sig-default">None</span>)</span>
168 |     <br /><em class="fname">(Constructor)</em>
169 |   </h3>
170 |   </td><td align="right" valign="top"
171 |     ><span class="codelink"><a href="tagger-pysrc.html#MultiTag.__init__">source&nbsp;code</a></span>&nbsp;
172 |     </td>
173 |   </tr></table>
174 |   
175 |   
176 |   <dl class="fields">
177 |     <dt>Parameters:</dt>
178 |     <dd><ul class="nomargin-top">
179 |         <li><strong class="pname"><code>tail</code></strong> - the <a href="tagger.Tag-class.html" class="link">Tag</a> object 
180 |           to add to the first part (head)</li>
181 |         <li><strong class="pname"><code>head</code></strong> - the (eventually absent) <a href="tagger.MultiTag-class.html" 
182 |           class="link">MultiTag</a> to be extended</li>
183 |     </ul></dd>
184 |     <dt>Returns:</dt>
185 |         <dd>a new <a href="tagger.MultiTag-class.html" 
186 |           class="link">MultiTag</a> object</dd>
187 |     <dt>Overrides:
188 |         <a href="tagger.Tag-class.html#__init__">Tag.__init__</a>
189 |     </dt>
190 |   </dl>
191 | </td></tr></table>
192 | </div>
193 | <a name="combined_rating"></a>
194 | <div>
195 | <table class="details" border="1" cellpadding="3"
196 |        cellspacing="0" width="100%" bgcolor="white">
197 | <tr><td>
198 |   <table width="100%" cellpadding="0" cellspacing="0" border="0">
199 |   <tr valign="top"><td>
200 |   <h3 class="epydoc"><span class="sig"><span class="sig-name">combined_rating</span>(<span class="sig-arg">self</span>)</span>
201 |   </h3>
202 |   </td><td align="right" valign="top"
203 |     ><span class="codelink"><a href="tagger-pysrc.html#MultiTag.combined_rating">source&nbsp;code</a></span>&nbsp;
204 |     </td>
205 |   </tr></table>
206 |   
207 |   <p>Method that computes the multitag's rating from the ratings of unit 
208 |   subtags</p>
209 |   <p>(the default implementation uses the geometric mean - with a special 
210 |   treatment for proper nouns - but this method can be overridden)</p>
211 |   <dl class="fields">
212 |     <dt>Returns:</dt>
213 |         <dd>the rating of the multitag</dd>
214 |   </dl>
215 | </td></tr></table>
216 | </div>
217 | <br />
218 | <!-- ==================== NAVIGATION BAR ==================== -->
219 | <table class="navbar" border="0" width="100%" cellpadding="0"
220 |        bgcolor="#a0c0ff" cellspacing="0">
221 |   <tr valign="middle">
222 | 
223 |   <!-- Tree link -->
224 |       <th>&nbsp;&nbsp;&nbsp;<a
225 |         href="module-tree.html">Trees</a>&nbsp;&nbsp;&nbsp;</th>
226 | 
227 |   <!-- Index link -->
228 |       <th>&nbsp;&nbsp;&nbsp;<a
229 |         href="identifier-index.html">Indices</a>&nbsp;&nbsp;&nbsp;</th>
230 | 
231 |   <!-- Help link -->
232 |       <th>&nbsp;&nbsp;&nbsp;<a
233 |         href="help.html">Help</a>&nbsp;&nbsp;&nbsp;</th>
234 | 
235 |   <!-- Project homepage -->
236 |       <th class="navbar" align="right" width="100%">
237 |         <table border="0" cellpadding="0" cellspacing="0">
238 |           <tr><th class="navbar" align="center"
239 |             ><a class="navbar" target="_top" href="http://github.com/apresta/tagger">tagger</a></th>
240 |           </tr></table></th>
241 |   </tr>
242 | </table>
243 | <table border="0" cellpadding="0" cellspacing="0" width="100%%">
244 |   <tr>
245 |     <td align="left" class="footer">
246 |     Generated by Epydoc 3.0.1 on Fri May 13 11:13:08 2011
247 |     </td>
248 |     <td align="right" class="footer">
249 |       <a target="mainFrame" href="http://epydoc.sourceforge.net"
250 |         >http://epydoc.sourceforge.net</a>
251 |     </td>
252 |   </tr>
253 | </table>
254 | 
255 | <script type="text/javascript">
256 |   <!--
257 |   // Private objects are initially displayed (because if
258 |   // javascript is turned off then we want them to be
259 |   // visible); but by default, we want to hide them.  So hide
260 |   // them unless we have a cookie that says to show them.
261 |   checkCookie();
262 |   // -->
263 | </script>
264 | </body>
265 | </html>
266 | 


--------------------------------------------------------------------------------
/doc/tagger.Tag-class.html:
--------------------------------------------------------------------------------
  1 | <?xml version="1.0" encoding="ascii"?>
  2 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  3 |           "DTD/xhtml1-transitional.dtd">
  4 | <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
  5 | <head>
  6 |   <title>tagger.Tag</title>
  7 |   <link rel="stylesheet" href="epydoc.css" type="text/css" />
  8 |   <script type="text/javascript" src="epydoc.js"></script>
  9 | </head>
 10 | 
 11 | <body bgcolor="white" text="black" link="blue" vlink="#204080"
 12 |       alink="#204080">
 13 | <!-- ==================== NAVIGATION BAR ==================== -->
 14 | <table class="navbar" border="0" width="100%" cellpadding="0"
 15 |        bgcolor="#a0c0ff" cellspacing="0">
 16 |   <tr valign="middle">
 17 | 
 18 |   <!-- Tree link -->
 19 |       <th>&nbsp;&nbsp;&nbsp;<a
 20 |         href="module-tree.html">Trees</a>&nbsp;&nbsp;&nbsp;</th>
 21 | 
 22 |   <!-- Index link -->
 23 |       <th>&nbsp;&nbsp;&nbsp;<a
 24 |         href="identifier-index.html">Indices</a>&nbsp;&nbsp;&nbsp;</th>
 25 | 
 26 |   <!-- Help link -->
 27 |       <th>&nbsp;&nbsp;&nbsp;<a
 28 |         href="help.html">Help</a>&nbsp;&nbsp;&nbsp;</th>
 29 | 
 30 |   <!-- Project homepage -->
 31 |       <th class="navbar" align="right" width="100%">
 32 |         <table border="0" cellpadding="0" cellspacing="0">
 33 |           <tr><th class="navbar" align="center"
 34 |             ><a class="navbar" target="_top" href="http://github.com/apresta/tagger">tagger</a></th>
 35 |           </tr></table></th>
 36 |   </tr>
 37 | </table>
 38 | <table width="100%" cellpadding="0" cellspacing="0">
 39 |   <tr valign="top">
 40 |     <td width="100%">
 41 |       <span class="breadcrumbs">
 42 |         <a href="tagger-module.html">Module&nbsp;tagger</a> ::
 43 |         Class&nbsp;Tag
 44 |       </span>
 45 |     </td>
 46 |     <td>
 47 |       <table cellpadding="0" cellspacing="0">
 48 |         <!-- hide/show private -->
 49 |         <tr><td align="right"><span class="options">[<a href="javascript:void(0);" class="privatelink"
 50 |     onclick="toggle_private();">hide&nbsp;private</a>]</span></td></tr>
 51 |         <tr><td align="right"><span class="options"
 52 |             >[<a href="frames.html" target="_top">frames</a
 53 |             >]&nbsp;|&nbsp;<a href="tagger.Tag-class.html"
 54 |             target="_top">no&nbsp;frames</a>]</span></td></tr>
 55 |       </table>
 56 |     </td>
 57 |   </tr>
 58 | </table>
 59 | <!-- ==================== CLASS DESCRIPTION ==================== -->
 60 | <h1 class="epydoc">Class Tag</h1><p class="nomargin-top"><span class="codelink"><a href="tagger-pysrc.html#Tag">source&nbsp;code</a></span></p>
 61 | <center>
 62 | <center>  <map id="class_hierarchy_for_tag" name="class_hierarchy_for_tag">
 63 | <area shape="rect" id="node1" href="tagger.MultiTag-class.html" title="MultiTag" alt="" coords="5,61,80,88"/>
 64 | <area shape="rect" id="node2" href="tagger.Tag-class.html" title="Tag" alt="" coords="20,5,65,32"/>
 65 | </map>
 66 |   <img src="class_hierarchy_for_tag.gif" alt='' usemap="#class_hierarchy_for_tag" ismap="ismap" class="graph-without-title" />
 67 | </center>
 68 | </center>
 69 | <hr />
 70 | <p>General class for tags (small units of text)</p>
 71 | 
 72 | <!-- ==================== INSTANCE METHODS ==================== -->
 73 | <a name="section-InstanceMethods"></a>
 74 | <table class="summary" border="1" cellpadding="3"
 75 |        cellspacing="0" width="100%" bgcolor="white">
 76 | <tr bgcolor="#70b0f0" class="table-header">
 77 |   <td colspan="2" class="table-header">
 78 |     <table border="0" cellpadding="0" cellspacing="0" width="100%">
 79 |       <tr valign="top">
 80 |         <td align="left"><span class="table-header">Instance Methods</span></td>
 81 |         <td align="right" valign="top"
 82 |          ><span class="options">[<a href="#section-InstanceMethods"
 83 |          class="privatelink" onclick="toggle_private();"
 84 |          >hide private</a>]</span></td>
 85 |       </tr>
 86 |     </table>
 87 |   </td>
 88 | </tr>
 89 | <tr>
 90 |     <td width="15%" align="right" valign="top" class="summary">
 91 |       <span class="summary-type">&nbsp;</span>
 92 |     </td><td class="summary">
 93 |       <table width="100%" cellpadding="0" cellspacing="0" border="0">
 94 |         <tr>
 95 |           <td><span class="summary-sig"><a href="tagger.Tag-class.html#__init__" class="summary-sig-name">__init__</a>(<span class="summary-sig-arg">self</span>,
 96 |         <span class="summary-sig-arg">string</span>,
 97 |         <span class="summary-sig-arg">stem</span>=<span class="summary-sig-default">None</span>,
 98 |         <span class="summary-sig-arg">rating</span>=<span class="summary-sig-default">1.0</span>,
 99 |         <span class="summary-sig-arg">proper</span>=<span class="summary-sig-default">False</span>,
100 |         <span class="summary-sig-arg">terminal</span>=<span class="summary-sig-default">False</span>)</span><br />
101 |       Returns:
102 |       a new <a href="tagger.Tag-class.html" class="link">Tag</a> object</td>
103 |           <td align="right" valign="top">
104 |             <span class="codelink"><a href="tagger-pysrc.html#Tag.__init__">source&nbsp;code</a></span>
105 |             
106 |           </td>
107 |         </tr>
108 |       </table>
109 |       
110 |     </td>
111 |   </tr>
112 | <tr>
113 |     <td width="15%" align="right" valign="top" class="summary">
114 |       <span class="summary-type">&nbsp;</span>
115 |     </td><td class="summary">
116 |       <table width="100%" cellpadding="0" cellspacing="0" border="0">
117 |         <tr>
118 |           <td><span class="summary-sig"><a name="__eq__"></a><span class="summary-sig-name">__eq__</span>(<span class="summary-sig-arg">self</span>,
119 |         <span class="summary-sig-arg">other</span>)</span></td>
120 |           <td align="right" valign="top">
121 |             <span class="codelink"><a href="tagger-pysrc.html#Tag.__eq__">source&nbsp;code</a></span>
122 |             
123 |           </td>
124 |         </tr>
125 |       </table>
126 |       
127 |     </td>
128 |   </tr>
129 | <tr>
130 |     <td width="15%" align="right" valign="top" class="summary">
131 |       <span class="summary-type">&nbsp;</span>
132 |     </td><td class="summary">
133 |       <table width="100%" cellpadding="0" cellspacing="0" border="0">
134 |         <tr>
135 |           <td><span class="summary-sig"><a name="__repr__"></a><span class="summary-sig-name">__repr__</span>(<span class="summary-sig-arg">self</span>)</span></td>
136 |           <td align="right" valign="top">
137 |             <span class="codelink"><a href="tagger-pysrc.html#Tag.__repr__">source&nbsp;code</a></span>
138 |             
139 |           </td>
140 |         </tr>
141 |       </table>
142 |       
143 |     </td>
144 |   </tr>
145 | <tr>
146 |     <td width="15%" align="right" valign="top" class="summary">
147 |       <span class="summary-type">&nbsp;</span>
148 |     </td><td class="summary">
149 |       <table width="100%" cellpadding="0" cellspacing="0" border="0">
150 |         <tr>
151 |           <td><span class="summary-sig"><a name="__lt__"></a><span class="summary-sig-name">__lt__</span>(<span class="summary-sig-arg">self</span>,
152 |         <span class="summary-sig-arg">other</span>)</span></td>
153 |           <td align="right" valign="top">
154 |             <span class="codelink"><a href="tagger-pysrc.html#Tag.__lt__">source&nbsp;code</a></span>
155 |             
156 |           </td>
157 |         </tr>
158 |       </table>
159 |       
160 |     </td>
161 |   </tr>
162 | <tr>
163 |     <td width="15%" align="right" valign="top" class="summary">
164 |       <span class="summary-type">&nbsp;</span>
165 |     </td><td class="summary">
166 |       <table width="100%" cellpadding="0" cellspacing="0" border="0">
167 |         <tr>
168 |           <td><span class="summary-sig"><a name="__hash__"></a><span class="summary-sig-name">__hash__</span>(<span class="summary-sig-arg">self</span>)</span></td>
169 |           <td align="right" valign="top">
170 |             <span class="codelink"><a href="tagger-pysrc.html#Tag.__hash__">source&nbsp;code</a></span>
171 |             
172 |           </td>
173 |         </tr>
174 |       </table>
175 |       
176 |     </td>
177 |   </tr>
178 | </table>
179 | <!-- ==================== METHOD DETAILS ==================== -->
180 | <a name="section-MethodDetails"></a>
181 | <table class="details" border="1" cellpadding="3"
182 |        cellspacing="0" width="100%" bgcolor="white">
183 | <tr bgcolor="#70b0f0" class="table-header">
184 |   <td colspan="2" class="table-header">
185 |     <table border="0" cellpadding="0" cellspacing="0" width="100%">
186 |       <tr valign="top">
187 |         <td align="left"><span class="table-header">Method Details</span></td>
188 |         <td align="right" valign="top"
189 |          ><span class="options">[<a href="#section-MethodDetails"
190 |          class="privatelink" onclick="toggle_private();"
191 |          >hide private</a>]</span></td>
192 |       </tr>
193 |     </table>
194 |   </td>
195 | </tr>
196 | </table>
197 | <a name="__init__"></a>
198 | <div>
199 | <table class="details" border="1" cellpadding="3"
200 |        cellspacing="0" width="100%" bgcolor="white">
201 | <tr><td>
202 |   <table width="100%" cellpadding="0" cellspacing="0" border="0">
203 |   <tr valign="top"><td>
204 |   <h3 class="epydoc"><span class="sig"><span class="sig-name">__init__</span>(<span class="sig-arg">self</span>,
205 |         <span class="sig-arg">string</span>,
206 |         <span class="sig-arg">stem</span>=<span class="sig-default">None</span>,
207 |         <span class="sig-arg">rating</span>=<span class="sig-default">1.0</span>,
208 |         <span class="sig-arg">proper</span>=<span class="sig-default">False</span>,
209 |         <span class="sig-arg">terminal</span>=<span class="sig-default">False</span>)</span>
210 |     <br /><em class="fname">(Constructor)</em>
211 |   </h3>
212 |   </td><td align="right" valign="top"
213 |     ><span class="codelink"><a href="tagger-pysrc.html#Tag.__init__">source&nbsp;code</a></span>&nbsp;
214 |     </td>
215 |   </tr></table>
216 |   
217 |   
218 |   <dl class="fields">
219 |     <dt>Parameters:</dt>
220 |     <dd><ul class="nomargin-top">
221 |         <li><strong class="pname"><code>string</code></strong> - the actual representation of the tag</li>
222 |         <li><strong class="pname"><code>stem</code></strong> - the internal (usually stemmed) representation; tags with the same
223 |           stem are regarded as equal</li>
224 |         <li><strong class="pname"><code>rating</code></strong> - a measure of the tag's relevance in the interval [0,1]</li>
225 |         <li><strong class="pname"><code>proper</code></strong> - whether the tag is a proper noun</li>
226 |         <li><strong class="pname"><code>terminal</code></strong> - set to True if the tag is at the end of a phrase (or anyway it 
227 |           cannot be logically merged to the following one)</li>
228 |     </ul></dd>
229 |     <dt>Returns:</dt>
230 |         <dd>a new <a href="tagger.Tag-class.html" class="link">Tag</a> object</dd>
231 |   </dl>
232 | </td></tr></table>
233 | </div>
234 | <br />
235 | <!-- ==================== NAVIGATION BAR ==================== -->
236 | <table class="navbar" border="0" width="100%" cellpadding="0"
237 |        bgcolor="#a0c0ff" cellspacing="0">
238 |   <tr valign="middle">
239 | 
240 |   <!-- Tree link -->
241 |       <th>&nbsp;&nbsp;&nbsp;<a
242 |         href="module-tree.html">Trees</a>&nbsp;&nbsp;&nbsp;</th>
243 | 
244 |   <!-- Index link -->
245 |       <th>&nbsp;&nbsp;&nbsp;<a
246 |         href="identifier-index.html">Indices</a>&nbsp;&nbsp;&nbsp;</th>
247 | 
248 |   <!-- Help link -->
249 |       <th>&nbsp;&nbsp;&nbsp;<a
250 |         href="help.html">Help</a>&nbsp;&nbsp;&nbsp;</th>
251 | 
252 |   <!-- Project homepage -->
253 |       <th class="navbar" align="right" width="100%">
254 |         <table border="0" cellpadding="0" cellspacing="0">
255 |           <tr><th class="navbar" align="center"
256 |             ><a class="navbar" target="_top" href="http://github.com/apresta/tagger">tagger</a></th>
257 |           </tr></table></th>
258 |   </tr>
259 | </table>
260 | <table border="0" cellpadding="0" cellspacing="0" width="100%%">
261 |   <tr>
262 |     <td align="left" class="footer">
263 |     Generated by Epydoc 3.0.1 on Fri May 13 11:13:08 2011
264 |     </td>
265 |     <td align="right" class="footer">
266 |       <a target="mainFrame" href="http://epydoc.sourceforge.net"
267 |         >http://epydoc.sourceforge.net</a>
268 |     </td>
269 |   </tr>
270 | </table>
271 | 
272 | <script type="text/javascript">
273 |   <!--
274 |   // Private objects are initially displayed (because if
275 |   // javascript is turned off then we want them to be
276 |   // visible); but by default, we want to hide them.  So hide
277 |   // them unless we have a cookie that says to show them.
278 |   checkCookie();
279 |   // -->
280 | </script>
281 | </body>
282 | </html>
283 | 


--------------------------------------------------------------------------------
/doc/tagger.Tagger-class.html:
--------------------------------------------------------------------------------
  1 | <?xml version="1.0" encoding="ascii"?>
  2 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  3 |           "DTD/xhtml1-transitional.dtd">
  4 | <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
  5 | <head>
  6 |   <title>tagger.Tagger</title>
  7 |   <link rel="stylesheet" href="epydoc.css" type="text/css" />
  8 |   <script type="text/javascript" src="epydoc.js"></script>
  9 | </head>
 10 | 
 11 | <body bgcolor="white" text="black" link="blue" vlink="#204080"
 12 |       alink="#204080">
 13 | <!-- ==================== NAVIGATION BAR ==================== -->
 14 | <table class="navbar" border="0" width="100%" cellpadding="0"
 15 |        bgcolor="#a0c0ff" cellspacing="0">
 16 |   <tr valign="middle">
 17 | 
 18 |   <!-- Tree link -->
 19 |       <th>&nbsp;&nbsp;&nbsp;<a
 20 |         href="module-tree.html">Trees</a>&nbsp;&nbsp;&nbsp;</th>
 21 | 
 22 |   <!-- Index link -->
 23 |       <th>&nbsp;&nbsp;&nbsp;<a
 24 |         href="identifier-index.html">Indices</a>&nbsp;&nbsp;&nbsp;</th>
 25 | 
 26 |   <!-- Help link -->
 27 |       <th>&nbsp;&nbsp;&nbsp;<a
 28 |         href="help.html">Help</a>&nbsp;&nbsp;&nbsp;</th>
 29 | 
 30 |   <!-- Project homepage -->
 31 |       <th class="navbar" align="right" width="100%">
 32 |         <table border="0" cellpadding="0" cellspacing="0">
 33 |           <tr><th class="navbar" align="center"
 34 |             ><a class="navbar" target="_top" href="http://github.com/apresta/tagger">tagger</a></th>
 35 |           </tr></table></th>
 36 |   </tr>
 37 | </table>
 38 | <table width="100%" cellpadding="0" cellspacing="0">
 39 |   <tr valign="top">
 40 |     <td width="100%">
 41 |       <span class="breadcrumbs">
 42 |         <a href="tagger-module.html">Module&nbsp;tagger</a> ::
 43 |         Class&nbsp;Tagger
 44 |       </span>
 45 |     </td>
 46 |     <td>
 47 |       <table cellpadding="0" cellspacing="0">
 48 |         <!-- hide/show private -->
 49 |         <tr><td align="right"><span class="options">[<a href="javascript:void(0);" class="privatelink"
 50 |     onclick="toggle_private();">hide&nbsp;private</a>]</span></td></tr>
 51 |         <tr><td align="right"><span class="options"
 52 |             >[<a href="frames.html" target="_top">frames</a
 53 |             >]&nbsp;|&nbsp;<a href="tagger.Tagger-class.html"
 54 |             target="_top">no&nbsp;frames</a>]</span></td></tr>
 55 |       </table>
 56 |     </td>
 57 |   </tr>
 58 | </table>
 59 | <!-- ==================== CLASS DESCRIPTION ==================== -->
 60 | <h1 class="epydoc">Class Tagger</h1><p class="nomargin-top"><span class="codelink"><a href="tagger-pysrc.html#Tagger">source&nbsp;code</a></span></p>
 61 | <p>Master class for tagging text documents</p>
 62 |   <p>(this is a simple interface that should allow convenient 
 63 |   experimentation by using different classes as building blocks)</p>
 64 | 
 65 | <!-- ==================== INSTANCE METHODS ==================== -->
 66 | <a name="section-InstanceMethods"></a>
 67 | <table class="summary" border="1" cellpadding="3"
 68 |        cellspacing="0" width="100%" bgcolor="white">
 69 | <tr bgcolor="#70b0f0" class="table-header">
 70 |   <td colspan="2" class="table-header">
 71 |     <table border="0" cellpadding="0" cellspacing="0" width="100%">
 72 |       <tr valign="top">
 73 |         <td align="left"><span class="table-header">Instance Methods</span></td>
 74 |         <td align="right" valign="top"
 75 |          ><span class="options">[<a href="#section-InstanceMethods"
 76 |          class="privatelink" onclick="toggle_private();"
 77 |          >hide private</a>]</span></td>
 78 |       </tr>
 79 |     </table>
 80 |   </td>
 81 | </tr>
 82 | <tr>
 83 |     <td width="15%" align="right" valign="top" class="summary">
 84 |       <span class="summary-type">&nbsp;</span>
 85 |     </td><td class="summary">
 86 |       <table width="100%" cellpadding="0" cellspacing="0" border="0">
 87 |         <tr>
 88 |           <td><span class="summary-sig"><a href="tagger.Tagger-class.html#__init__" class="summary-sig-name">__init__</a>(<span class="summary-sig-arg">self</span>,
 89 |         <span class="summary-sig-arg">reader</span>,
 90 |         <span class="summary-sig-arg">stemmer</span>,
 91 |         <span class="summary-sig-arg">rater</span>)</span><br />
 92 |       Returns:
 93 |       a new <a href="tagger.Tagger-class.html" class="link">Tagger</a> 
 94 |       object</td>
 95 |           <td align="right" valign="top">
 96 |             <span class="codelink"><a href="tagger-pysrc.html#Tagger.__init__">source&nbsp;code</a></span>
 97 |             
 98 |           </td>
 99 |         </tr>
100 |       </table>
101 |       
102 |     </td>
103 |   </tr>
104 | <tr>
105 |     <td width="15%" align="right" valign="top" class="summary">
106 |       <span class="summary-type">&nbsp;</span>
107 |     </td><td class="summary">
108 |       <table width="100%" cellpadding="0" cellspacing="0" border="0">
109 |         <tr>
110 |           <td><span class="summary-sig"><a href="tagger.Tagger-class.html#__call__" class="summary-sig-name">__call__</a>(<span class="summary-sig-arg">self</span>,
111 |         <span class="summary-sig-arg">text</span>,
112 |         <span class="summary-sig-arg">tags_number</span>=<span class="summary-sig-default">5</span>)</span></td>
113 |           <td align="right" valign="top">
114 |             <span class="codelink"><a href="tagger-pysrc.html#Tagger.__call__">source&nbsp;code</a></span>
115 |             
116 |           </td>
117 |         </tr>
118 |       </table>
119 |       
120 |     </td>
121 |   </tr>
122 | </table>
123 | <!-- ==================== METHOD DETAILS ==================== -->
124 | <a name="section-MethodDetails"></a>
125 | <table class="details" border="1" cellpadding="3"
126 |        cellspacing="0" width="100%" bgcolor="white">
127 | <tr bgcolor="#70b0f0" class="table-header">
128 |   <td colspan="2" class="table-header">
129 |     <table border="0" cellpadding="0" cellspacing="0" width="100%">
130 |       <tr valign="top">
131 |         <td align="left"><span class="table-header">Method Details</span></td>
132 |         <td align="right" valign="top"
133 |          ><span class="options">[<a href="#section-MethodDetails"
134 |          class="privatelink" onclick="toggle_private();"
135 |          >hide private</a>]</span></td>
136 |       </tr>
137 |     </table>
138 |   </td>
139 | </tr>
140 | </table>
141 | <a name="__init__"></a>
142 | <div>
143 | <table class="details" border="1" cellpadding="3"
144 |        cellspacing="0" width="100%" bgcolor="white">
145 | <tr><td>
146 |   <table width="100%" cellpadding="0" cellspacing="0" border="0">
147 |   <tr valign="top"><td>
148 |   <h3 class="epydoc"><span class="sig"><span class="sig-name">__init__</span>(<span class="sig-arg">self</span>,
149 |         <span class="sig-arg">reader</span>,
150 |         <span class="sig-arg">stemmer</span>,
151 |         <span class="sig-arg">rater</span>)</span>
152 |     <br /><em class="fname">(Constructor)</em>
153 |   </h3>
154 |   </td><td align="right" valign="top"
155 |     ><span class="codelink"><a href="tagger-pysrc.html#Tagger.__init__">source&nbsp;code</a></span>&nbsp;
156 |     </td>
157 |   </tr></table>
158 |   
159 |   
160 |   <dl class="fields">
161 |     <dt>Parameters:</dt>
162 |     <dd><ul class="nomargin-top">
163 |         <li><strong class="pname"><code>reader</code></strong> - a <a href="tagger.Reader-class.html" class="link">Reader</a> 
164 |           object</li>
165 |         <li><strong class="pname"><code>stemmer</code></strong> - a <a href="tagger.Stemmer-class.html" class="link">Stemmer</a> 
166 |           object</li>
167 |         <li><strong class="pname"><code>rater</code></strong> - a <a href="tagger.Rater-class.html" class="link">Rater</a> object</li>
168 |     </ul></dd>
169 |     <dt>Returns:</dt>
170 |         <dd>a new <a href="tagger.Tagger-class.html" class="link">Tagger</a> 
171 |           object</dd>
172 |   </dl>
173 | </td></tr></table>
174 | </div>
175 | <a name="__call__"></a>
176 | <div>
177 | <table class="details" border="1" cellpadding="3"
178 |        cellspacing="0" width="100%" bgcolor="white">
179 | <tr><td>
180 |   <table width="100%" cellpadding="0" cellspacing="0" border="0">
181 |   <tr valign="top"><td>
182 |   <h3 class="epydoc"><span class="sig"><span class="sig-name">__call__</span>(<span class="sig-arg">self</span>,
183 |         <span class="sig-arg">text</span>,
184 |         <span class="sig-arg">tags_number</span>=<span class="sig-default">5</span>)</span>
185 |     <br /><em class="fname">(Call operator)</em>
186 |   </h3>
187 |   </td><td align="right" valign="top"
188 |     ><span class="codelink"><a href="tagger-pysrc.html#Tagger.__call__">source&nbsp;code</a></span>&nbsp;
189 |     </td>
190 |   </tr></table>
191 |   
192 |   
193 |   <dl class="fields">
194 |     <dt>Parameters:</dt>
195 |     <dd><ul class="nomargin-top">
196 |         <li><strong class="pname"><code>text</code></strong> - the string of text to be tagged</li>
197 |         <li><strong class="pname"><code>tags_number</code></strong> - number of best tags to be returned
198 |           <p>Returns: a list of (hopefully) relevant tags</p></li>
199 |     </ul></dd>
200 |   </dl>
201 | </td></tr></table>
202 | </div>
203 | <br />
204 | <!-- ==================== NAVIGATION BAR ==================== -->
205 | <table class="navbar" border="0" width="100%" cellpadding="0"
206 |        bgcolor="#a0c0ff" cellspacing="0">
207 |   <tr valign="middle">
208 | 
209 |   <!-- Tree link -->
210 |       <th>&nbsp;&nbsp;&nbsp;<a
211 |         href="module-tree.html">Trees</a>&nbsp;&nbsp;&nbsp;</th>
212 | 
213 |   <!-- Index link -->
214 |       <th>&nbsp;&nbsp;&nbsp;<a
215 |         href="identifier-index.html">Indices</a>&nbsp;&nbsp;&nbsp;</th>
216 | 
217 |   <!-- Help link -->
218 |       <th>&nbsp;&nbsp;&nbsp;<a
219 |         href="help.html">Help</a>&nbsp;&nbsp;&nbsp;</th>
220 | 
221 |   <!-- Project homepage -->
222 |       <th class="navbar" align="right" width="100%">
223 |         <table border="0" cellpadding="0" cellspacing="0">
224 |           <tr><th class="navbar" align="center"
225 |             ><a class="navbar" target="_top" href="http://github.com/apresta/tagger">tagger</a></th>
226 |           </tr></table></th>
227 |   </tr>
228 | </table>
229 | <table border="0" cellpadding="0" cellspacing="0" width="100%%">
230 |   <tr>
231 |     <td align="left" class="footer">
232 |     Generated by Epydoc 3.0.1 on Fri May 13 11:13:08 2011
233 |     </td>
234 |     <td align="right" class="footer">
235 |       <a target="mainFrame" href="http://epydoc.sourceforge.net"
236 |         >http://epydoc.sourceforge.net</a>
237 |     </td>
238 |   </tr>
239 | </table>
240 | 
241 | <script type="text/javascript">
242 |   <!--
243 |   // Private objects are initially displayed (because if
244 |   // javascript is turned off then we want them to be
245 |   // visible); but by default, we want to hide them.  So hide
246 |   // them unless we have a cookie that says to show them.
247 |   checkCookie();
248 |   // -->
249 | </script>
250 | </body>
251 | </html>
252 | 


--------------------------------------------------------------------------------
/doc/toc-build_dict-module.html:
--------------------------------------------------------------------------------
 1 | <?xml version="1.0" encoding="ascii"?>
 2 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
 3 |           "DTD/xhtml1-transitional.dtd">
 4 | <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 5 | <head>
 6 |   <title>build_dict</title>
 7 |   <link rel="stylesheet" href="epydoc.css" type="text/css" />
 8 |   <script type="text/javascript" src="epydoc.js"></script>
 9 | </head>
10 | 
11 | <body bgcolor="white" text="black" link="blue" vlink="#204080"
12 |       alink="#204080">
13 | <h1 class="toc">Module build_dict</h1>
14 | <hr />
15 |   <h2 class="toc">Functions</h2>
16 |     <a target="mainFrame" href="build_dict-module.html#build_dict"
17 |      >build_dict</a><br />    <a target="mainFrame" href="build_dict-module.html#build_dict_from_files"
18 |      >build_dict_from_files</a><br />  <h2 class="toc">Variables</h2>
19 |     <a target="mainFrame" href="build_dict-module.html#__package__"
20 |      >__package__</a><br /><hr />
21 | <span class="options">[<a href="javascript:void(0);" class="privatelink"
22 |     onclick="toggle_private();">hide&nbsp;private</a>]</span>
23 | 
24 | <script type="text/javascript">
25 |   <!--
26 |   // Private objects are initially displayed (because if
27 |   // javascript is turned off then we want them to be
28 |   // visible); but by default, we want to hide them.  So hide
29 |   // them unless we have a cookie that says to show them.
30 |   checkCookie();
31 |   // -->
32 | </script>
33 | </body>
34 | </html>
35 | 


--------------------------------------------------------------------------------
/doc/toc-everything.html:
--------------------------------------------------------------------------------
 1 | <?xml version="1.0" encoding="ascii"?>
 2 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
 3 |           "DTD/xhtml1-transitional.dtd">
 4 | <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 5 | <head>
 6 |   <title>Everything</title>
 7 |   <link rel="stylesheet" href="epydoc.css" type="text/css" />
 8 |   <script type="text/javascript" src="epydoc.js"></script>
 9 | </head>
10 | 
11 | <body bgcolor="white" text="black" link="blue" vlink="#204080"
12 |       alink="#204080">
13 | <h1 class="toc">Everything</h1>
14 | <hr />
15 |   <h2 class="toc">All Classes</h2>
16 |     <a target="mainFrame" href="extras.FastStemmer-class.html"
17 |      >extras.FastStemmer</a><br />    <a target="mainFrame" href="extras.HTMLReader-class.html"
18 |      >extras.HTMLReader</a><br />    <a target="mainFrame" href="extras.NaiveRater-class.html"
19 |      >extras.NaiveRater</a><br />    <a target="mainFrame" href="extras.SimpleReader-class.html"
20 |      >extras.SimpleReader</a><br />    <a target="mainFrame" href="extras.UnicodeReader-class.html"
21 |      >extras.UnicodeReader</a><br />    <a target="mainFrame" href="tagger.MultiTag-class.html"
22 |      >tagger.MultiTag</a><br />    <a target="mainFrame" href="tagger.Rater-class.html"
23 |      >tagger.Rater</a><br />    <a target="mainFrame" href="tagger.Reader-class.html"
24 |      >tagger.Reader</a><br />    <a target="mainFrame" href="tagger.Stemmer-class.html"
25 |      >tagger.Stemmer</a><br />    <a target="mainFrame" href="tagger.Tag-class.html"
26 |      >tagger.Tag</a><br />    <a target="mainFrame" href="tagger.Tagger-class.html"
27 |      >tagger.Tagger</a><br />  <h2 class="toc">All Functions</h2>
28 |     <a target="mainFrame" href="build_dict-module.html#build_dict"
29 |      >build_dict.build_dict</a><br />    <a target="mainFrame" href="build_dict-module.html#build_dict_from_files"
30 |      >build_dict.build_dict_from_files</a><br />    <a target="mainFrame" href="extras-module.html#build_dict_from_nltk"
31 |      >extras.build_dict_from_nltk</a><br />  <h2 class="toc">All Variables</h2>
32 |     <a target="mainFrame" href="build_dict-module.html#__package__"
33 |      >build_dict.__package__</a><br />    <a target="mainFrame" href="extras-module.html#__package__"
34 |      >extras.__package__</a><br />    <a target="mainFrame" href="tagger-module.html#__package__"
35 |      >tagger.__package__</a><br /><hr />
36 | <span class="options">[<a href="javascript:void(0);" class="privatelink"
37 |     onclick="toggle_private();">hide&nbsp;private</a>]</span>
38 | 
39 | <script type="text/javascript">
40 |   <!--
41 |   // Private objects are initially displayed (because if
42 |   // javascript is turned off then we want them to be
43 |   // visible); but by default, we want to hide them.  So hide
44 |   // them unless we have a cookie that says to show them.
45 |   checkCookie();
46 |   // -->
47 | </script>
48 | </body>
49 | </html>
50 | 


--------------------------------------------------------------------------------
/doc/toc-extras-module.html:
--------------------------------------------------------------------------------
 1 | <?xml version="1.0" encoding="ascii"?>
 2 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
 3 |           "DTD/xhtml1-transitional.dtd">
 4 | <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 5 | <head>
 6 |   <title>extras</title>
 7 |   <link rel="stylesheet" href="epydoc.css" type="text/css" />
 8 |   <script type="text/javascript" src="epydoc.js"></script>
 9 | </head>
10 | 
11 | <body bgcolor="white" text="black" link="blue" vlink="#204080"
12 |       alink="#204080">
13 | <h1 class="toc">Module extras</h1>
14 | <hr />
15 |   <h2 class="toc">Classes</h2>
16 |     <a target="mainFrame" href="extras.FastStemmer-class.html"
17 |      >FastStemmer</a><br />    <a target="mainFrame" href="extras.HTMLReader-class.html"
18 |      >HTMLReader</a><br />    <a target="mainFrame" href="extras.NaiveRater-class.html"
19 |      >NaiveRater</a><br />    <a target="mainFrame" href="extras.SimpleReader-class.html"
20 |      >SimpleReader</a><br />    <a target="mainFrame" href="extras.UnicodeReader-class.html"
21 |      >UnicodeReader</a><br />  <h2 class="toc">Functions</h2>
22 |     <a target="mainFrame" href="extras-module.html#build_dict_from_nltk"
23 |      >build_dict_from_nltk</a><br />  <h2 class="toc">Variables</h2>
24 |     <a target="mainFrame" href="extras-module.html#__package__"
25 |      >__package__</a><br /><hr />
26 | <span class="options">[<a href="javascript:void(0);" class="privatelink"
27 |     onclick="toggle_private();">hide&nbsp;private</a>]</span>
28 | 
29 | <script type="text/javascript">
30 |   <!--
31 |   // Private objects are initially displayed (because if
32 |   // javascript is turned off then we want them to be
33 |   // visible); but by default, we want to hide them.  So hide
34 |   // them unless we have a cookie that says to show them.
35 |   checkCookie();
36 |   // -->
37 | </script>
38 | </body>
39 | </html>
40 | 


--------------------------------------------------------------------------------
/doc/toc-tagger-module.html:
--------------------------------------------------------------------------------
 1 | <?xml version="1.0" encoding="ascii"?>
 2 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
 3 |           "DTD/xhtml1-transitional.dtd">
 4 | <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 5 | <head>
 6 |   <title>tagger</title>
 7 |   <link rel="stylesheet" href="epydoc.css" type="text/css" />
 8 |   <script type="text/javascript" src="epydoc.js"></script>
 9 | </head>
10 | 
11 | <body bgcolor="white" text="black" link="blue" vlink="#204080"
12 |       alink="#204080">
13 | <h1 class="toc">Module tagger</h1>
14 | <hr />
15 |   <h2 class="toc">Classes</h2>
16 |     <a target="mainFrame" href="tagger.MultiTag-class.html"
17 |      >MultiTag</a><br />    <a target="mainFrame" href="tagger.Rater-class.html"
18 |      >Rater</a><br />    <a target="mainFrame" href="tagger.Reader-class.html"
19 |      >Reader</a><br />    <a target="mainFrame" href="tagger.Stemmer-class.html"
20 |      >Stemmer</a><br />    <a target="mainFrame" href="tagger.Tag-class.html"
21 |      >Tag</a><br />    <a target="mainFrame" href="tagger.Tagger-class.html"
22 |      >Tagger</a><br />  <h2 class="toc">Variables</h2>
23 |     <a target="mainFrame" href="tagger-module.html#__package__"
24 |      >__package__</a><br /><hr />
25 | <span class="options">[<a href="javascript:void(0);" class="privatelink"
26 |     onclick="toggle_private();">hide&nbsp;private</a>]</span>
27 | 
28 | <script type="text/javascript">
29 |   <!--
30 |   // Private objects are initially displayed (because if
31 |   // javascript is turned off then we want them to be
32 |   // visible); but by default, we want to hide them.  So hide
33 |   // them unless we have a cookie that says to show them.
34 |   checkCookie();
35 |   // -->
36 | </script>
37 | </body>
38 | </html>
39 | 


--------------------------------------------------------------------------------
/doc/toc.html:
--------------------------------------------------------------------------------
 1 | <?xml version="1.0" encoding="ascii"?>
 2 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
 3 |           "DTD/xhtml1-transitional.dtd">
 4 | <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 5 | <head>
 6 |   <title>Table of Contents</title>
 7 |   <link rel="stylesheet" href="epydoc.css" type="text/css" />
 8 |   <script type="text/javascript" src="epydoc.js"></script>
 9 | </head>
10 | 
11 | <body bgcolor="white" text="black" link="blue" vlink="#204080"
12 |       alink="#204080">
13 | <h1 class="toc">Table&nbsp;of&nbsp;Contents</h1>
14 | <hr />
15 |   <a target="moduleFrame" href="toc-everything.html">Everything</a>
16 |   <br />
17 |   <h2 class="toc">Modules</h2>
18 |     <a target="moduleFrame" href="toc-build_dict-module.html"
19 |      onclick="setFrame('toc-build_dict-module.html','build_dict-module.html');"     >build_dict</a><br />    <a target="moduleFrame" href="toc-extras-module.html"
20 |      onclick="setFrame('toc-extras-module.html','extras-module.html');"     >extras</a><br />    <a target="moduleFrame" href="toc-tagger-module.html"
21 |      onclick="setFrame('toc-tagger-module.html','tagger-module.html');"     >tagger</a><br /><hr />
22 |   <span class="options">[<a href="javascript:void(0);" class="privatelink"
23 |     onclick="toggle_private();">hide&nbsp;private</a>]</span>
24 | 
25 | <script type="text/javascript">
26 |   <!--
27 |   // Private objects are initially displayed (because if
28 |   // javascript is turned off then we want them to be
29 |   // visible); but by default, we want to hide them.  So hide
30 |   // them unless we have a cookie that says to show them.
31 |   checkCookie();
32 |   // -->
33 | </script>
34 | </body>
35 | </html>
36 | 


--------------------------------------------------------------------------------
/extras.py:
--------------------------------------------------------------------------------
  1 | # Copyright (C) 2011 by Alessandro Presta
  2 | 
  3 | # Permission is hereby granted, free of charge, to any person obtaining a copy
  4 | # of this software and associated documentation files (the "Software"), to deal
  5 | # in the Software without restriction, including without limitation the rights
  6 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
  7 | # copies of the Software, and to permit persons to whom the Software is
  8 | # furnished to do so, subject to the following conditions:
  9 | 
 10 | # The above copyright notice and this permission notice shall be included in
 11 | # all copies or substantial portions of the Software.
 12 | 
 13 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 14 | # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 15 | # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 16 | # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 17 | # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 18 | # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 19 | # THE SOFTWARE
 20 | 
 21 | 
 22 | from tagger import *
 23 | 
 24 | 
 25 | class UnicodeReader(Reader):
 26 |     '''
 27 |     Reader subclass that converts Unicode strings to a close ASCII
 28 |     representation
 29 |     '''
 30 | 
 31 |     def __call__(self, text):
 32 |         import unicodedata
 33 | 
 34 |         text = unicode(text)
 35 |         text = unicodedata.normalize('NFKD', text).encode('ascii', 'ignore')
 36 |         return Reader.__call__(self, text)
 37 | 
 38 | 
 39 | class HTMLReader(UnicodeReader):
 40 |     '''
 41 |     Reader subclass that can parse HTML code from the input
 42 |     '''
 43 | 
 44 |     def __call__(self, html):
 45 |         import lxml.html
 46 | 
 47 |         text = lxml.html.fromstring(html).text_content().encode('utf-8')
 48 |         return UnicodeReader.__call__(self, text)
 49 | 
 50 |     
 51 | class SimpleReader(Reader):
 52 |     '''
 53 |     Reader subclass that doesn't perform any advanced analysis of the text
 54 |     '''
 55 |     
 56 |     def __call__(self, text):
 57 |         text = text.lower()
 58 |         text = self.preprocess(text)
 59 |         words = self.match_words.findall(text)
 60 |         tags = [Tag(w) for w in words]
 61 |         return tags
 62 | 
 63 | 
 64 | class FastStemmer(Stemmer):
 65 |     '''
 66 |     Stemmer subclass that uses a much faster, but less correct algorithm
 67 |     '''
 68 | 
 69 |     def __init__(self):
 70 |         from stemming import porter
 71 |         
 72 |         Stemmer.__init__(self, porter)
 73 | 
 74 | 
 75 | class NaiveRater(Rater):
 76 |     '''
 77 |     Rater subclass that jusk ranks single-word tags by their frequency and
 78 |     weight
 79 |     '''
 80 | 
 81 |     def __call__(self, tags):
 82 |         self.rate_tags(tags)
 83 |         # we still get rid of one-character tags
 84 |         unique_tags = set(t for t in tags if len(t.string) > 1)
 85 |         return sorted(unique_tags)
 86 |     
 87 |         
 88 | def build_dict_from_nltk(output_file, corpus=None, stopwords=None,
 89 |                          stemmer=Stemmer(), measure='IDF', verbose=False):
 90 |     '''
 91 |     @param output_file: the binary stream where the dictionary should be saved
 92 |     @param corpus:      the NLTK corpus to use (defaults to nltk.corpus.reuters)
 93 |     @param stopwords:   a list of (not stemmed) stopwords (defaults to
 94 |                         nltk.corpus.reuters.words('stopwords'))
 95 |     @param stemmer:     the L{Stemmer} object to be used
 96 |     @param measure:     the measure used to compute the weights ('IDF'
 97 |                         i.e. 'inverse document frequency' or 'ICF' i.e.
 98 |                         'inverse collection frequency'; defaults to 'IDF')
 99 |     @param verbose:     whether information on the progress should be printed
100 |                         on screen
101 |     '''
102 |     
103 |     from build_dict import build_dict
104 |     import nltk
105 |     import pickle
106 | 
107 |     if not (corpus and stopwords):
108 |         nltk.download('reuters')
109 |         
110 |     corpus = corpus or nltk.corpus.reuters
111 |     stopwords = stopwords or nltk.corpus.reuters.words('stopwords')
112 | 
113 |     corpus_list = []
114 |     
115 |     if verbose: print 'Processing corpus...'
116 |     for file in corpus.fileids():
117 |         doc = [stemmer(Tag(w.lower())).stem for w in corpus.words(file)
118 |                if w[0].isalpha()]
119 |         corpus_list.append(doc)
120 | 
121 |     if verbose: print 'Processing stopwords...'
122 |     stopwords = [stemmer(Tag(w.lower())).stem for w in stopwords]
123 | 
124 |     if verbose: print 'Building dictionary... '
125 |     dictionary = build_dict(corpus_list, stopwords, measure)
126 |     pickle.dump(dictionary, output_file, -1) 
127 | 
128 | 
129 | 
130 |     
131 | 
132 |     
133 | 
134 |      
135 | 
136 |     
137 | 
138 |         
139 | 
140 |     
141 | 
142 | 
143 | 
144 | 
145 | 
146 |         
147 | 


--------------------------------------------------------------------------------
/test_ui.py:
--------------------------------------------------------------------------------
 1 | #!/usr/bin/env python
 2 | 
 3 | from Tkinter import *
 4 | import tkMessageBox
 5 | import ScrolledText
 6 | 
 7 | from tagger import *
 8 | from extras import UnicodeReader
 9 | 
10 | import pickle
11 | 
12 | with open('data/dict.pkl', 'rb') as f:
13 |    weights = pickle.load(f)
14 | tagger = Tagger(UnicodeReader(), Stemmer(), Rater(weights))
15 | 
16 | top = Tk()
17 | top.title('tagger')
18 | 
19 | st = ScrolledText.ScrolledText(top)
20 | st.pack()
21 | 
22 | def tag():
23 |    tags = tagger(st.get(1.0, END))
24 |    output = ', '.join(t.string for t in tags)
25 |    tkMessageBox.showinfo('Tags:', output)
26 |    st.delete(1.0, END)
27 | 
28 | b = Button(top, text ='TAG!', command=tag)
29 | 
30 | b.pack()
31 | top.mainloop()
32 | 


--------------------------------------------------------------------------------
/tests/bbc1.txt:
--------------------------------------------------------------------------------
 1 | Obama lays wreath at Ground Zero
 2 | US President Barack Obama is visiting Ground Zero, the site of the 9/11 attacks in New York, four days after US forces killed al-Qaeda head Osama Bin Laden in Pakistan.
 3 | Bin Laden was believed to be the mastermind of the 9/11 attacks in 2001.
 4 | Mr Obama laid a wreath in memory of the nearly 3,000 victims and spoke to relatives at the site.
 5 | He earlier told New York firefighters: "When we say we will never forget, we mean what we say."
 6 | The visit comes a day after the US president said graphic photographs of Bin Laden's dead body would not be made public.
 7 | The al-Qaeda leader was killed by US special forces in northern Pakistan on Monday. His body was then buried at sea from a US aircraft carrier.
 8 | The Pakistani military on Thursday admitted "shortcomings" for failing to locate Bin Laden and has said it will launch an investigation.
 9 | But it also warned it would review co-operation with the US if there were any more unilateral raids such as the one that killed Bin Laden.
10 | Moment of reflection
11 | Mr Obama's first stop in New York was a fire station in midtown Manhattan. He told firefighters: "We are going to make sure that the perpetrators of that horrible act will see justice."
12 | Mr Obama met families of the victims at the site of Ground Zero, where he laid a wreath in red, white and blue.
13 | He made no public comments at the scene.
14 | The BBC's Barbara Plett, in New York, says the Obama administration is very sensitive to accusations that the president is politicising his visit.
15 | Thousands of people gathered at Ground Zero on Sunday night, waving flags and climbing street signs, as the news emerged that Bin Laden had been killed.
16 | On Monday, Mr Obama said he had made it his top national security priority to find Bin Laden.
17 | Ground Zero is now a building site, with construction scheduled for completion in 2013.
18 | As well as several office towers, the area will also house the National September 11 Memorial & Museum, which comprises a museum, waterfalls and a park.
19 | Mr Obama has decided not to publish photos of Bin Laden's body, saying the images could pose a national security risk.
20 | "It is important for us to make sure that very graphic photos of somebody who was shot in the head are not floating around as an incitement to additional violence, as a propaganda tool. That's not who we are," Mr Obama said.
21 | Our correspondent says there are mixed feelings in New York about the decision not to publish the pictures. While some want proof that it was Bin Laden who was killed, for others, the photos would reopen painful memories.
22 | Meanwhile, White House officials have again changed their account of the raid on Bin Laden's compound.
23 | They told respected media outlets, including the New York Times, that only one individual - a courier for Bin Laden - fired at US special forces. He was killed at the start of the raid.
24 | The Al Qaeda leader was elsewhere in compound and unarmed when killed.
25 | As recently as Tuesday, the President's spokesman had spoken of a "highly volatile" fire-fight, lasting throughout the forty minute operation. And the White House had given the impression that an armed Osama Bin Laden had been killed in a shoot-out.
26 | Critics have raised concerns about the legality of the operation, after the US revised its account to acknowledge Bin Laden was unarmed when shot dead.
27 | But the US has said Bin Laden was a lawful military target, whose killing was "an act of national self-defence".
28 | Anger in Pakistan
29 | In Pakistan, there are continuing recriminations over the failure to arrest or locate Bin Laden.
30 | A senior Pakistani military official said one of Bin Laden's wives told investigators she had been living in the same room for five years, along with her husband.
31 | On Thursday, the head of Pakistan's diplomatic service, Salman Bashir, again dismissed allegations his country's secret services had links to al-Qaeda, and said the investigation into the presence of Bin Laden in Abbottabad would reveal what failures there were.
32 | Pakistan's army has long been seen as the most effective institution in an unstable country. However, Pakistani public opinion has been critical of the perceived violation of national sovereignty by the US raid.
33 | Referring to the raid, Chief of Army Staff Gen Ashfaq Kayani was quoted by the Reuters news agency as saying: "Any similar action violating the sovereignty of Pakistan will warrant a review on the level of military/intelligence cooperation with the United States."
34 | The BBC's Syed Shoaib Hasan, in Rawalpindi, said Pakistan's forces again denied all prior knowledge of the raid and Bin Laden's whereabouts.
35 | However, he says there were some contradictions: the military claimed Pakistan contributed information leading to the capture of Bin Laden by providing details of phone calls made from Pakistan to Saudi Arabia.
36 | The calls - from a Saudi man - concerned financial transactions. The last one was traced to Bin Laden's compound, the military said.
37 | However, our correspondent in Rawalpindi says the military did not explain why, after this call, the compound was not raided or investigated.
38 | 


--------------------------------------------------------------------------------
/tests/bbc2.txt:
--------------------------------------------------------------------------------
 1 | Jo Yeates: Vincent Tabak admits manslaughter
 2 | A man charged with the murder of Jo Yeates, whose body was found near Bristol on Christmas Day, has admitted manslaughter but denied murder.
 3 | Vincent Tabak, 33, is accused of killing the landscape architect who disappeared on 17 December after going for drinks with colleagues.
 4 | The prosecution has refused to accept Tabak's manslaughter plea and a murder trial will go ahead on 4 October.
 5 | Tabak, a Dutch national, lived next to Jo Yeates in Clifton, Bristol.
 6 | He appeared via video link from HMP Long Lartin in Worcestershire.
 7 | Vincent Tabak spoke only to confirm his name and to say he was content for proceedings to continue in English without an interpreter, before entering his pleas.
 8 | He was remanded in custody to face trial at Bristol Crown Court.
 9 | Strangled
10 | Miss Yeates's parents David and Theresa were in court for the hearing.
11 | Their daughter, who grew up in Ampfield, Hampshire, spent the evening of 17 December in the Bristol Ram with colleagues before visiting Tesco Express to buy a pizza on her way home to the Clifton area of Bristol.
12 | She was reported missing by her boyfriend Greg Reardon on 19 December.
13 | Jo Yeates's snow-covered body was found in Longwood Lane, Failand, by dog walkers on Christmas Day, eight days after she was reported missing.
14 | A post-mortem test revealed she had been strangled.
15 | Her funeral took place at St Mark's Church in Hampshire, where she was christened, on 11 February.
16 | Her body was carried into the church in a wicker coffin in front of her parents, who said her death had been "traumatic".
17 | Miss Yeates's coffin was adorned with daffodils, small sunflowers and assorted other flowers.
18 | 


--------------------------------------------------------------------------------
/tests/bbc3.txt:
--------------------------------------------------------------------------------
 1 | Scottish election: SNP profile
 2 | 
 3 | By Andrew Black
 4 | Political reporter, BBC Scotland
 5 | Over the years, the world has seen the rise and fall of many single-issue groups and minor parties, yet only a handful go on to achieve their goals.
 6 | The Scottish National Party is one of those.
 7 | The story of the SNP is one of success and failure, highs and lows, rogues and visionaries - but, most of all, it's the story of a party which started life on the fringes and moved in to claim political success.
 8 | Despite the party's turbulent history, it is now set to realise its vision for an independence referendum, after first emerging as the government of Scotland in 2007.
 9 | The SNP's spring conference in March last year was a bittersweet one.
10 | The party was on a high, but it was also mourning the death, at 86, of one of its former leaders, Billy Wolfe.
11 | This was the man who finally transformed the SNP into a serious party, guiding it to its greatest Westminster electoral success in 1974.
12 | The case for Scottish home rule goes right back to its unification with England in 1707.
13 | The view that the Scots who put their names to the Act of Union had been bribed, famously spurred Robert Burns to write: "We are bought and sold for English gold. Such a parcel of rogues in a nation."
14 | Many years later, the realisation that a pro-independence, election-fighting party was the way to go eventually led to the creation in 1934 of the Scottish National Party, through the amalgamation of the Scottish Party and the National Party of Scotland.
15 | Election-fighting
16 | But for years the SNP struggled to make an impact, party due to the on-going debate between those who wanted to concentrate on independence - the fundamentalists - and those who wanted to achieve it through policies such as devolution - the gradualists.
17 | The young Nationalist party's other problem was that, put simply, it just was not any good at fighting elections, because of its lack of funding, organisation and policies beyond independence.
18 | In its first test, the 1935 General Election, the SNP contested eight seats and won none.
19 | It was not until a decade later and the Motherwell and Wishaw by-election when the party finally got a break.
20 | When the contest was announced following the death of sitting Labour MP James Walker, the Nationalists sent in one of their up-and-comers, Robert McIntyre, to fight the seat.
21 | After standing largely on a platform of Labour failures in post-war reconstruction, the SNP took the seat with 50% of the vote, but lost it months later in the election.
22 | Even though this brief victory provided much excitement over what the party could achieve, it failed to make progress in subsequent elections and disquiet set in.
23 | But it was this disquiet which forced the party to reorganise - a move which would help the SNP to one of its most famous achievements.
24 | The Hamilton by-election should have been a breeze for Labour, but, as the party's vote collapsed, the SNP's Winnie Ewing romped home on 46%, famously declaring: "Stop the world, Scotland wants to get on."
25 | The 1970s was the decade of boom and bust for the SNP. They failed to hang on in Hamilton, but 1970 brought the SNP its first UK election gain, in the Western Isles.
26 | That same decade also saw the beginnings of the party's "It's Scotland's Oil" strategy, which sought to demonstrate Scotland was seeing little direct benefit of the tax wealth brought by North Sea oil.
27 | More success followed in 1973, when Margo MacDonald, "the blonde bombshell" won the Glasgow Govan by-election and, the following year, an under-fire Tory government called an election, which it lost.
28 | The SNP gained six seats and retained the Western Isles, but lost Govan - however, there were to be further gains.
29 | With Labour in power as a minority government, the party had little choice but to call a second election in 1974 - but not before committing to support for a Scottish Assembly.
30 | Even so, the SNP gained a further four seats, hitting its all time Westminster high of 11.
31 | Despite the success, tensions began to develop between those in the SNP who were elected and those who were not.
32 | 'Tartan Tories'
33 | Then came 1979 - the year which provided two killer blows to the SNP.
34 | Not only did Scots voters fail to support the establishment of a Scottish Assembly in a referendum, but Margaret Thatcher's Tories swept to power - meaning the constitutional issue was not only off the table, but had been completely blown out of the water.
35 | The SNP had also come under a period of heavy fire from rival parties, portrayed by Labour as the "Tartan Tories" and "Separatists" by the Conservatives.
36 | With a post-election SNP slashed back to two MPs, the party needed a serious jump-start - but that jump-start dragged the party into a period which could have finished it off for good.
37 | The start of the 80s was a torrid time for the SNP. Many in the party felt bitter that it had come so far but was now practically back at square one in terms of its performance and the independence argument.
38 | The deep divisions gave rise to two notorious splinter groups.
39 | One was the ultra-nationalist group Siol Nan Gaidheal - branded "proto fascists" by the then SNP leader Gordon Wilson - whose members used to march around in Highland dress.
40 | The other was the Interim Committee for Political Discussion - more commonly known as the '79 Group.
41 | Formed to sharpen the party's message and appeal to dissident Labour voters, the group also embarked on a campaign of civil disobedience, spearheaded by the former Labour MP Jim Sillars, who had founded the Scottish Labour Party before defecting to the SNP in 1980.
42 | The campaign took a radical turn when Sillars, with several other group members, broke into Edinburgh's old Royal High School building.
43 | Then, a leak of '79 Group minutes to the media raised the prospect of links with the Provisional Sinn Fein.
44 | Despite claims the leaked version was inaccurate, Mr Wilson had had enough.
45 | His view that the party had to unite or die led to a ban on organised groups, but when the '79 Group refused to go quietly, seven of its members were briefly expelled from the party.
46 | They included Scotland's future justice secretary, Kenny MacAskill, and one Alex Salmond.
47 | The 1987 election saw another bad SNP performance. The party emerged with only three seats - but with the collapse in the Conservative vote, the constitutional issue was back.
48 | The SNP needed new blood at the top, and it came in the form of Alex Salmond.
49 | Despite previous form with the '79 Group, Mr Salmond had risen through the SNP ranks, becoming MP for Banff and Buchan and deputy leader of the party.
50 | Independence case
51 | Mr Salmond did not have his work to seek on becoming leader in 1990.
52 | As well as having to deal with on-going issues over the party's independence policy - future minister Alex Neil had declared Scotland would be "free by '93" - there was an election to fight.
53 | In 1992, the SNP increased its vote, but the party was only able to retain the three seats it already had, and lost Govan, which Mr Sillars re-took for the party in a 1988 by-election.
54 | Mr Salmond moved to modernise the SNP, repositioning the party as more socially democratic and pro-European and pushing the economic case for independence.
55 | Labour's commitment to a Scottish Parliament, delivered in 1999, was both a blessing and a curse for the Nationalists.
56 | Although devolution presented a great opportunity for the SNP, many questioned how relevant a pro-independence party would be - George Robertson famously quipped devolution would "kill nationalism stone dead".
57 | The SNP won 35 seats in the first election and also had two MEPs and six MPs.
58 | But the best it could manage in 1999 was becoming the main opposition to the Labour-Lib Dem coalition government.
59 | Mr Salmond's decision to quit as leader and an MSP came as a surprise.
60 | Despite much speculation over his reasons for returning to Westminster, ultimately, after a decade in the job, he decided it was time to step aside.
61 | His successor in 2000 was John Swinney, but, despite being among the party's brightest talent, his four-year tenure was plagued by dissenters from within.
62 | The party dropped a seat in 2001, and, despite a slick 2003 election campaign, the SNP once again ended up as the opposition.
63 | Later that year, a little-known SNP activist called Bill Wilson challenged Mr Swinney for the leadership, accusing him of ducking responsibility for a "plummeting" SNP vote.
64 | Mr Swinney won a decisive victory but was left weakened, and, at Holyrood, SNP MSPs Bruce McFee and Adam Ingram declared they would not support Mr Swinney in a leadership ballot.
65 | Another, Campbell Martin, was flung out of the party after bosses found his criticism of the Swinney leadership damaged its interests in the run-up to the SNP's poor European election showing in 2004, where it failed to overtake Labour.
66 | Mr Swinney quit as leader, accepting responsibility for failing to sell the party's message - but warned SNP members over the damage caused by "the loose and dangerous talk of the few".
67 | Close result
68 | When the leaderless party turned to Mr Salmond, he drew on a quote from US civil war leader General Sherman to declare: "If nominated I'll decline. If drafted I'll defer. And if elected I'll resign."
69 | Then, in a move almost as surprising as his decision to quit, Mr Salmond launched a successful leadership campaign on a joint ticket with Nicola Sturgeon, winning a decisive victory.
70 | Nobody thought the 2007 Scottish election result could be so close.
71 | In the end, the SNP won the election by one seat, while Mr Salmond returned to Holyrood as MSP for Gordon.
72 | With the SNP's pro-independence stance ruling out a coalition, the party forged ahead as a minority government.
73 | The SNP government had promised to seek consensus on an issue-by-issue basis, but when the opposition parties thought the government was being disingenuous, they converged to reject the Scottish budget in 2009.
74 | It was passed on the second attempt, but served as a reminder to the SNP the delicate position it was in.
75 | Other key manifesto commitments also ran into trouble - plans to replace council tax with local income tax were dropped due to lack of support, while ambitious plans to cut class sizes in the early primary school years ran into problems.
76 | Eventually, the bill on an independence referendum was dropped.
77 | Such is life in minority government.
78 | Although the SNP's focus had become the Scottish government, it was keen not to lose sight of its status beyond the Holyrood bubble and, in 2009, won the largest share of the Scottish vote in the European election for the first time.
79 | Continuing its knack for winning safe Labour seats in by-elections, the SNP delivered a crushing blow to Labour, winning Glasgow East by overturning a majority of 13,507 to win by just a few hundred votes.
80 | But the party failed to repeat this success a few months later in the Glenrothes by-election and, later, in Glasgow North East.
81 | In a story that bore echoes of the past for the SNP, the 2010 UK election saw Labour regain Glasgow East, while the Nationalists concluded that, with a resurgent Tory party on course for victory, Scots voters came out in their droves to back Labour.
82 | The 2011 Holyrood election was Labour's to lose. In the event, that is exactly what happened.
83 | Despite polls predicting a Labour lead over the SNP of up to 15 points, the Nationalists threw themselves into the campaign.
84 | They say their positive campaign, versus Labour's negativity, was what won it for them.
85 | The SNP's 2007 win was rightly described as a historic one - but, four years later, the has re-written the history books again.
86 | Its jaw-dropping victory means it will be Scotland's first majority government - and the independence referendum will happen.
87 | The SNP has truly come a long way since the fringes of 1934.
88 | 


--------------------------------------------------------------------------------
/tests/guardian1.txt:
--------------------------------------------------------------------------------
 1 | Osama bin Laden death: Al-Qaida vows to carry out revenge attacks on US
 2 | White House says US is being 'extremely vigilant' after al-Qaida declares Bin Laden's death a curse on the US
 3 | 
 4 | Al-Qaida conceded in an 11-paragraph statement that Osama Bin Laden had been killed. Photograph: Rahimullah Yousafzai/AP
 5 | Al-Qaida has vowed to carry out revenge attacks on the US and its allies over the killing of Osama bin Laden, warning that celebrations in the west would be replaced by sorrow and blood.
 6 | 
 7 | The statement on a jihadist website was the first by al-Qaida since Bin Laden's death, which it said would become "a curse that hunts the Americans and their collaborators, and chases them outside and inside their country".
 8 | 
 9 | The 11-paragraph statement, dated Tuesday, confirmed that Bin Laden was dead, disappointing conspiracy theorists who refuse to believe he has been killed.
10 | 
11 | The White House spokesman, Jay Carney, said of the al-Qaida statement: "We are aware of it. What it does, obviously, is acknowledge the obvious, which is that Osama bin Laden was killed on Sunday night by US forces. We're being extremely vigilant. We're quite aware of the potential for activity and are highly vigilant on that matter for that reason. US security, both at home and at embassies and bases overseas, has been on high alert since Sunday."
12 | 
13 | The Department of Homeland Security has warned US train operators to be especially careful after officials said that among computers, hard disks and other material taken from the Abbottabad compound they found a vague plan to attack the US rail network on this year's 10th anniversary of 9/11. One proposal was to demolish part of a rail track so that a train would fall into a river or valley, according to US officials.
14 | 
15 | Carney said: "One of the things we saw, I think, was the notice that DHS put out with regard to the information collected about the consideration at least of a terrorist plot against American railways back in February of 2010.
16 | 
17 | "The fact that the world's most wanted terrorist might have been considering further terror plots against the United States is not a surprise, but it reminds us, of course, that we need to remain ever vigilant."
18 | 
19 | In its statement, al-Qaida said: "We stress that the blood of the holy warrior sheikh Osama bin Laden, God bless him, is precious to us and to all Muslims and will not go in vain. We will remain, God willing, a curse chasing the Americans and their agents, following them outside and inside their countries.
20 | 
21 | "Soon, God willing, their happiness will turn to sadness, their blood will be mingled with their tears."
22 | 
23 | It said Bin Laden's death would not deflect al-Qaida from its war against the US and its allies, which include the Pakistani government. It called on Pakistan to rise up against the "traitors".
24 | 
25 | The discovery of Bin Laden's hide–away so close to the capital, Islamabad, has strained relations between the US and Pakistan. Carl Levin, chairman of the Senate armed services committee and a Democrat, ordered an investigation into whether the Pakistani government and intelligence services knew of his whereabouts. "We need these questions about whether or not the top level of the Pakistan government knew or was told by the ISI, their intelligence service, about anything about this suspicious activity for years in a very, very centralised place," Levin said.
26 | 
27 | The senator, who is usually guarded in his public statements, hinted that he believed some senior figures in Pakistani intelligence knew where Bin Laden was hiding – comments that will further inflame the Pakistani government.
28 | 
29 | "I think at high levels – high levels being the intelligence service – they knew it," Levin said. "I can't prove it. I just think it's counterintuitive not to."
30 | 
31 | He raised doubts about continuing the billions of dollars in aid to Pakistan, which requires congressional approval.
32 | 
33 | The Obama administration so far has been reluctant to criticise the Pakistani government and has opted instead to stress the positive aspects of the ties. The strategy seems to be to try to use Pakistan's embarrassment to prise out other al-Qaida or Taliban figures who may be living in Pakistan, such as the Taliban leader, Mullah Omar, and Bin Laden's deputy, Ayman al-Zawahiri.
34 | 
35 | United Nations human rights investigators have called on Washington to disclose whether there had been any plan to capture Bin Laden. While they acknowledged the difficulties involved in such terrorist-related missions, they raised questions about the legality of the killing.
36 | 
37 | The UN's special rapporteur on extrajudicial, summary or arbitrary executions, Christof Heyns, and the special rapporteur on the promotion and protection of human rights and fundamental freedoms while countering terrorism, Martin Scheinin, said the US "should disclose the supporting facts to allow an assessment in terms of international human rights law standards", adding: "For instance, it will be particularly important to know if the planning of the mission allowed an effort to capture Bin Laden." There has been relatively little debate in the US so far about the legality of the raid.
38 | 
39 | Meanwhile the New York Times reported that the US may have targeted one of the men named as a possible successor to Bin Laden. Quoting American officials, the paper said a missile strike from an American military drone in a remote region of Yemen on Thursday was aimed at killing Anwar al-Awlaki, the radical American-born cleric believed to be hiding in the country.
40 | 
41 | Separately it was reported that a Saudi man accused of conspiring with Bin Laden in the bombings of two US embassies expects to be extradited in the next few months to face charges after more than 12 years in British custody, according to documents which have emerged from a US court.
42 | 
43 | Prosecutors in New York have charged Khalid al-Fawwaz with helping al-Qaida to orchestrate the 1998 car bombings of the US embassies in Kenya and Tanzania, which killed 224 people.
44 | 
45 | A letter from a lawyer seeking to be appointed as al-Fawwaz's US defence counsel, said: "He [al-Fawwaz] anticipates extradition from the United Kingdom to the United States within the next few months to face these charges."
46 | 


--------------------------------------------------------------------------------
/tests/guardian2.txt:
--------------------------------------------------------------------------------
 1 | After election battle, bruised Clegg seeks distance from Tory partners
 2 | There is no serious move against the Lib Dem leader so far – but he is under pressure to make his party stand out again
 3 | 
 4 | The local election rout and AV defeat leaves Liberal Democrat leader Nick Clegg facing the extraordinary challenge of turning around his party's fortunes by the next election. Photograph: Toby Melville/Reuters
 5 | Nick Clegg has moved to reassure shattered Liberal Democrats that he could engineer a political recovery by the time of the next election, and that the current mood of anger at his "betrayal" would dissipate over two to three years.
 6 | 
 7 | As the economy recovers, voters will slowly and grudgingly recognise the party's difficult role in saving the country from crisis, he feels.
 8 | 
 9 | Senior party figures were predicting the Lib Dems would suffer at least two more years in the doldrums. Despite the loss of more than 500 English council seats, 13 Scottish parliament seats and one in the Welsh assembly, there is no sign of any half-serious move against Clegg's leadership. The judgment across the party remains that Clegg had no choice but to form a coalition with the Tories last May.
10 | 
11 | But the pressure he has been under from within his party for months to highlight the its distinctiveness is now increasing. It will be a key test of Clegg's still developing leadership how he balances these competing demands.
12 | 
13 | In the first sign that Clegg understands he needs to do more to distance his party from its coalition partners, he argued in a change of tone that one role for his party would be to protect the country from a return to the unfairness of Thatcherism. He is implying that the ideology of an unalloyed majority Conservative administration would be well to the right.
14 | 
15 | The Lib Dem federal committee will meet shortly to set out the specific ways in which it expects the party to do more to differentiate itself from the Tories in line with a lengthy motion passed at the party's spring conference in Sheffield.
16 | 
17 | Senior party figures almost universally predicted the Lib Dems were now entering "a transactional business relationship" with their coalition partners – a phrase first urged on the deputy prime minister by Vince Cable, the business secretary, last autumn.
18 | 
19 | Many Lib Dem activists have been demanding distinctiveness and a willingness by their leadership to spell out what they have stopped, as well as what they have achieved, by being in coalition. Clegg will explain more about his approach to coalition politics in a speech next week.
20 | 
21 | Cable warned at his party conference in the autumn that "to hold our own we need to maintain our party's identity and our authentic voice".
22 | 
23 | That point was echoed strongly by Evan Harris, the former MP and vice-chairman of the party's federal policy committee and the authentic voice of the social liberal left in the party.
24 | 
25 | He said no one was turning on Clegg inside the party, but his approach had to be less "collegiate" towards the prime minister. "He has got to deliver a strategy change, which is to do more than point out what we have achieved but also point out what bits of the programme come from a Conservative philosophy that we do not share," Harris said.
26 | 
27 | Disgruntled senior Lib Dems, knowing the beating was coming, were privately spitting at what they regard as an ill-judged attempt by some in Clegg's circle to project the coalition as some kind of new ideological fusion of JS Mill and Friedrich Hayek.
28 | 
29 | One Clegg aide said: "We are not going to behave like an opposition in the government, but we will have greater latitude to talk about when we disagree, as we already have over multiculturalism. That set the template. We are not going suddenly to Defcon 2 [a reference to the US armed forces expression for a defence-ready condition], or have poisonous rubbish briefed into the papers.
30 | 
31 | ""We cannot play silly buggers or spring surprises on the Tories. It has to be managed and agreed. Nor can you have a German system where ministers from different sides of the coalition go on television to set out their differences. We are not ready for that culturally."
32 | 
33 | But it was being argued that too many Lib Dem ministers in departments of state have been happy, in the words of the former leader Sir Menzies Campbell, to give the impression that "they get on like a house on fire with their Tory secretaries of state".
34 | 
35 | One Lib Dem official admitted the "power dynamics" of the coalition had changed after Thursday's vote. "I guess people that voted Tory last time sort of knew what they were getting. They were hardly surprised by what Cameron did, and did not feel the need to punish him," the official said.
36 | 
37 | "If you had to say on the basis of these results which party is going to form an overall majority at the next election, it is the Conservatives."
38 | 
39 | For the moment, Clegg will just have to go through the grind of government as it undertakes plans to shake up health, the police, welfare and the House of Lords and introduces a more radical form of recall for MPs guilty of wrongdoing than previously envisaged.
40 | 
41 | There is a deeper malaise for his party, however. The battle wounds of the alternative vote referendum campaign are going to leave permanent scars. For some, the anger is directed at Cameron for implicitly endorsing attacks by the NOtoAV campaign that trashed Clegg's leadership.
42 | 
43 | Neil O'Brien, director of right-leaning thinktank Policy Exchange, said: "The real threat is that the coalition will be crippled inside. When trust and goodwill flow, the coalition can make real progress. Each partner is prepared to swallow decisions that are unpalatable, knowing that, in the not too distant future, the favour will be returned. That could get harder now."
44 | 
45 | There is a deeper disappointment for some. One wing of the Tory party and the Liberal Democrats had been harbouring hopes that Cameron could actually have embraced the alternative vote, as Michael Gove, his education secretary, intended to do. In 1911, the Tory party had split on Lords reform between the hedgers and ditchers. There was optimistic talk in some circles that Cameron would prove to be a hedger.
46 | 
47 | If AV went through, it would have been possible for the Tories and Lib Dems to come to second preference arrangements at the next election. Cameron would then have achieved a realignment of politics. He chose to retain the existing divisions.
48 | 


--------------------------------------------------------------------------------
/tests/post1.txt:
--------------------------------------------------------------------------------
 1 | Sony aims to fully restore PlayStation Network, down by hacker attack, by end of May
 2 | 
 3 | By Associated Press, Updated: Tuesday, May 10, 11:56 AM
 4 | 
 5 | TOKYO — Sony said Tuesday it aims to fully restore its PlayStation Network, shut down after a massive security breach affecting over 100 million online accounts, by the end of May.
 6 | 
 7 | Sony also confirmed that personal data from 24.6 million user accounts was stolen in the hacker attack last month. Personal data, including credit card numbers, might have been stolen from another 77 million PlayStation accounts, said Sony Computer Entertainment Inc. spokesman Satoshi Fukuoka.
 8 | 
 9 | He said Sony has not received any reports of illegal uses of stolen information, and the company is continuing its probe into the hacker attack. He declined to give details on the investigation.
10 | 
11 | Sony shut down the PlayStation network, a system that links gamers worldwide in live play, on April 20 after discovering the security breach. The network also allows users to upgrade and download games and other content.
12 | 
13 | Sony was under heavy criticism over its handling of the network intrusion. The company did not notify consumers of the breach until April 26 even though it began investigating unusual activity on the network since April 19.
14 | 
15 | Last month, U.S. lawyers filed a lawsuit against Sony on behalf of lead plaintiff Kristopher Johns for negligent protection of personal data and failure to inform players in a timely fashion that their credit card information may have been stolen. The lawsuit seeks class-action status.
16 | 
17 | Fukuoka declined to comment on the lawsuit.
18 | 


--------------------------------------------------------------------------------
/tests/wikipedia2.txt:
--------------------------------------------------------------------------------
 1 | Beetroot
 2 | From Wikipedia, the free encyclopedia
 3 | 
 4 | Beetroots at a grocery store
 5 | 
 6 | The beetroot, also known as the table beet, garden beet, red beet or informally simply as beet, is one of the many cultivated varieties of beets (Beta vulgaris) and arguably the most commonly encountered variety in North America and Britain.
 7 | 
 8 | Uses
 9 | 
10 | As a root vegetable
11 | 
12 | The usually deep-red roots of beetroot are eaten boiled either as a cooked vegetable, or cold as a salad after cooking and adding oil and vinegar, or raw and shredded, either alone or combined with any salad vegetable. A large proportion of the commercial production is processed into boiled and sterilised beets or into pickles. In Eastern Europe beet soup, such as borscht, is a popular dish. Yellow-coloured beetroots are grown on a very small scale for home consumption.[1]
13 | 
14 | As a leaf vegetable
15 | 
16 | The green leafy portion of the beet is also edible. It is most commonly served boiled or steamed, in which case it has a taste and texture similar to spinach.
17 | 
18 | Health benefits
19 | 
20 | Beetroots are a rich source of potent antioxidants and nutrients, including magnesium, sodium, potassium and vitamin C, and betaine, which is important for cardiovascular health. It functions by acting with other nutrients to reduce the concentration of homocysteine, a homologue of the naturally occurring amino acid cysteine, which can be harmful to blood vessels and thus contribute to the development of heart disease, stroke, and peripheral vascular disease. Betaine functions in conjunction with S-adenosylmethionine, folic acid, and vitamins B6 and B12 to carry out this function.[2]
21 | 
22 | Additionally, several preliminary studies on both rats and humans have shown betaine may protect against liver disease, particularly the build up of fatty deposits in the liver caused by alcohol abuse, protein deficiency, or diabetes, among other causes. The nutrient also helps individuals with hypochlorhydria, a condition causing abnormally low levels of stomach acid, by increasing stomach acidity.[2]
23 | 
24 | Beetroot juice has been shown to lower blood pressure and thus help prevent cardiovascular problems. Research published in the American Heart Association journal Hypertension showed drinking 500 ml of beetroot juice led to a reduction in blood pressure within one hour. The reduction was more pronounced after three to four hours, and was measurable up to 24 hours after drinking the juice. The effect is attributed to the high nitrate content of the beetroot. The study correlated high nitrate concentrations in the blood following ingestion of the beetroot juice and the drop in blood pressure. Dietary nitrate, such as that found in the beetroot, is thought to be a source for the biological messenger nitric oxide, which is used by the endothelium to signal smooth muscle, triggering it to relax. This induces vasodilation and increased blood flow.[3]
25 | 
26 | Other studies have found the positive effects beetroot juice can have on human exercise and performances. In studies conducted by the Exeter University, scientists found cyclists who drank a half-litre of beetroot juice several hours before setting off were able to ride up to 20 per cent longer than those who drank a placebo blackcurrant juice. [4]
27 | 
28 | As a dye
29 | 
30 | Betanin, obtained from the roots, is used industrially as red food colourants, e.g. to improve the color and flavor of tomato paste, sauces, desserts, jams and jellies, ice cream, sweets and breakfast cereals.[1] Within older bulbs of beetroot, the colour is a deep crimson and the flesh is much softer. Beetroot dye may also be used in ink.
31 | 
32 | Betanin is not broken down in the body, and in higher concentration can temporarily cause urine (termed beeturia) and stool to assume a reddish color. This effect can cause distress and concern due to the visual similarity to bloody stools or urine, but is completely harmless and will subside once the food is out of the system.
33 | 
34 | As a traditional remedy
35 | 
36 | It is a rich source of the element boron. Field Marshal Montgomery is reputed to have exhorted his troops to 'take favours in the beetroot fields', a euphemism for visiting prostitutes.[5] From the Middle Ages, beetroot was used as a treatment for a variety of conditions, especially illnesses relating to digestion and the blood. Bartolomeo Platina recommended taking beetroot with garlic to nullify the effects of 'garlic-breath'.[6]
37 | 
38 | References
39 | 
40 | ^ a b Grubben, G.J.H. & Denton, O.A. (2004) Plant Resources of Tropical Africa 2. Vegetables. PROTA Foundation, Wageningen; Backhuys, Leiden; CTA, Wageningen.
41 | ^ a b A.D.A.M., Inc., ed. (2002), Betaine, University of Maryland Medical Center
42 | ^ Webb, Andrew J.; Nakul Patel; Stavros Loukogeorgakis; Mike Okorie; Zainab Aboud; Shivani Misra; Rahim Rashid; Philip Miall; John Deanfield; Nigel Benjamin; Raymond MacAllister; Adrian J. Hobbs; Amrita Ahluwalia; Patel, N; Loukogeorgakis, S; Okorie, M; Aboud, Z; Misra, S; Rashid, R; Miall, P et al. (2008), "Acute Blood Pressure Lowering, Vasoprotective, and Antiplatelet Properties of Dietary Nitrate via Bioconversion to Nitrite", Hypertension 51 (3): 784–790, doi:10.1161/HYPERTENSIONAHA.107.103523, PMC 2839282, PMID 18250365
43 | ^ "Beet your personal best". Sydney Morning Hearld. 4 October 2010. Retrieved 5 October 2010.
44 | ^ Stephen Nottingham (2004) (E-book), Beetroot
45 | ^ Platina De Honesta Voluptate et Valetudine, 3.14
46 | 


--------------------------------------------------------------------------------
/tests/wikipedia3.txt:
--------------------------------------------------------------------------------
 1 | The Lounge Lizards
 2 | From Wikipedia, the free encyclopedia
 3 | 
 4 | The Lounge Lizards are a jazz group formed in 1978 by saxophone player John Lurie.
 5 | Initially a tongue in cheek "fake jazz" combo, drawing on punk rock and no wave as much as jazz, The Lounge Lizards have since become respected for their creative and distinctive sound.
 6 | Contents [hide]
 7 | 1 History
 8 | 2 Past members
 9 | 3 Discography
10 | 3.1 Studio albums
11 | 3.2 Live albums
12 | [edit]History
13 | 
14 | The Lounge Lizards were founded on June 4th, 1979 with John Lurie, his brother Evan (piano and organ), Arto Lindsay (guitar), Steve Piccolo (bass guitar), and Anton Fier (drums). They were initially a punk or fake jazz group but soon evolved into something quite special. Taking music from all corners of the globe and synthesizing it into something truly organic and unique. The New York Times' Robert Palmer wrote (October 7,1986): "The Lounge Lizards are not faking anything. They have staked their claim to a musical territory that lies somewhere west of Charles Mingus and east of Bernard Hermann and made it their own."
15 | In 1984 the new group consists of Erik Sanko, Curtis Fowlkes, Marc Ribot, Evan Lurie, Roy Nathanson, Dougie Bowne, E.J Rodriguez. This edition of The Lounge Lizards recorded three albums in two years, and demonstrated John Lurie's increasingly sophisticated and multi-layered compositions that often stray rather far from conventional jazz: He was able to integrate elements of various world musics (he often favors tango-inspired passages in his songs), which retain a distinctive flavor, but avoid gimmickry. One critic notes traces of "Erik Satie and Kurt Weill" [1].
16 | John Lurie formed a new version of The Lounge Lizards in the early 1990s; prominent members included Steven Bernstein (trumpet), Michael Blake (saxophone), Oren Bloedow (bass guitar), David Tronzo (guitar), Calvin Weston (drums) and Billy Martin (percussion).
17 | Recent years have found The Lounge Lizards less active; John Lurie has been increasingly occupied with painting, while Evan Lurie has worked on The Backyardigans, a children's show that highlights multiple musical genres.
18 | 


--------------------------------------------------------------------------------