2 |
3 | # Expug
4 |
5 | > Indented shorthand HTML templates for Elixir
6 |
7 | [](https://github.com/rstacruz/expug)
8 | [](https://travis-ci.org/rstacruz/expug "See test builds")
9 |
10 | Expug is a template language based on [Pug][] (formerly known as [Jade][]).
11 | It lets you write HTML as indented shorthand, inspired by Haml, Slim, Pug/Jade, and so on.
12 |
13 |
14 |
15 | ```jade
16 | doctype html
17 | html
18 | meta(charset="utf-8")
19 | title Hello, world!
20 | body
21 | a(href=@link)
22 | | This is a link
23 | ```
24 |
25 | [Pug]: http://pugjs.org/
26 | [Jade]: http://jade-lang.com/
27 |
28 | ## Installation
29 |
30 | Add expug to your list of dependencies in `mix.exs`:
31 |
32 | ```elixir
33 | def deps do
34 | [{:expug, "~> 0.9"}]
35 | end
36 | ```
37 |
38 | Also see [phoenix_expug](https://github.com/rstacruz/phoenix_expug) for Phoenix integration.
39 |
40 | ## Syntax
41 |
42 | Use CSS-like selectors for elements, and express your nesting through indentations.
43 |
44 | ```jade
45 | ul.links
46 | li
47 | a(href="/") This is a link
48 | ```
49 |
50 | Read more: [Syntax](https://hexdocs.pm/expug/syntax.html)
51 |
52 | ## Why should I use it?
53 |
54 | Read on [this comparison](https://hexdocs.pm/expug/prior_art.html) to see how Expug compares to other similar libraries.
55 |
56 | ## Thanks
57 |
58 | **expug** © 2016-2017, Rico Sta. Cruz. Released under the [MIT] License.59 | Authored and maintained by Rico Sta. Cruz with help from contributors ([list][contributors]). 60 | 61 | > [ricostacruz.com](http://ricostacruz.com) · 62 | > GitHub [@rstacruz](https://github.com/rstacruz) · 63 | > Twitter [@rstacruz](https://twitter.com/rstacruz) 64 | 65 | [MIT]: http://mit-license.org/ 66 | [contributors]: http://github.com/rstacruz/expug/contributors 67 | 68 | [](https://github.com/rstacruz) 69 | [](https://twitter.com/rstacruz) 70 | -------------------------------------------------------------------------------- /config/config.exs: -------------------------------------------------------------------------------- 1 | # This file is responsible for configuring your application 2 | # and its dependencies with the aid of the Mix.Config module. 3 | use Mix.Config 4 | 5 | # This configuration is loaded before any dependency and is restricted 6 | # to this project. If another project depends on this project, this 7 | # file won't be loaded nor affect the parent project. For this reason, 8 | # if you want to provide default values for your application for 9 | # 3rd-party users, it should be done in your "mix.exs" file. 10 | 11 | # You can configure for your application as: 12 | # 13 | # config :expug, key: :value 14 | # 15 | # And access this configuration in your application as: 16 | # 17 | # Application.get_env(:expug, :key) 18 | # 19 | # Or configure a 3rd-party app: 20 | # 21 | # config :logger, level: :info 22 | # 23 | 24 | config :ex_unit, :capture_log, true 25 | 26 | # It is also possible to import configuration files, relative to this 27 | # directory. For example, you can emulate configuration per environment 28 | # by uncommenting the line below and defining dev.exs, test.exs and such. 29 | # Configuration from the imported file will override the ones defined 30 | # here (which is why it is important to import them last). 31 | # 32 | # import_config "#{Mix.env}.exs" 33 | -------------------------------------------------------------------------------- /docs/misc/how_it_works.md: -------------------------------------------------------------------------------- 1 | # Misc: How it works 2 | 3 | Expug converts a `.pug` template into an EEx string: 4 | 5 | ```elixir 6 | iex> Expug.to_eex!(~s[div(role="alert")= @message]) 7 | "
><%= @message %>"
8 | ```
9 |
10 | See the module `Expug` for details.
11 |
--------------------------------------------------------------------------------
/docs/misc/line_number_preservation.md:
--------------------------------------------------------------------------------
1 | # Misc: Line number preservation
2 |
3 | Eex has no provisions for source maps, so we'll have to emulate this by outputing EEx that matches line numbers *exactly* with the source `.pug` files.
4 |
5 | ```jade
6 | div
7 | | Hello,
8 | = @name
9 |
10 | button.btn
11 | | Save
12 | ```
13 |
14 | ```html
15 |
85 | > assign @xyz not available in eex template. 86 | 87 | [calliope]: https://github.com/nurugger07/calliope 88 | [slime]: https://github.com/slime-lang/slime 89 | -------------------------------------------------------------------------------- /docs/syntax/code.md: -------------------------------------------------------------------------------- 1 | # Syntax: Code 2 | 3 | ## Unbuffered code 4 | Unbuffered code starts with `-` does not add any output directly. 5 | 6 | ```jade 7 | - name = assigns.name 8 | div(id="name-#{name}") 9 | ``` 10 | 11 | ## Bufferred code 12 | 13 | Buffered code starts with `=` and outputs the result of evaluating the Elixir expression in the template. For security, it is first HTML escaped. 14 | 15 | ```jade 16 | p= "Hello, #{name}" 17 | ``` 18 | 19 | ## Unescaped code 20 | 21 | Buffered code may be unescaped by using `!=`. This skips the HTML escaping. 22 | 23 | ```jade 24 | div!= markdown_to_html(@article.body) |> sanitize() 25 | ``` 26 | 27 | ## Conditionals and Loops 28 | 29 | For `if`, `cond`, `try`, `for`, an `end` statement is automatically inferred. 30 | 31 | ```jade 32 | = if assigns.name do 33 | = "Hello, #{@name}" 34 | ``` 35 | 36 | They also need to begin with `=`, not `-`. Except for `else`, `rescue` and so on. 37 | 38 | ```jade 39 | = if assigns.current_user do 40 | | Welcome. 41 | - else 42 | | You are not signed in. 43 | ``` 44 | 45 | ## Multiline 46 | 47 | If a line ends in one of these characters: `,` `(` `{` `[`, the next line is considered to be part of the Elixir expression. 48 | 49 | ```jade 50 | = render App.PageView, 51 | "index.html", 52 | conn: @conn 53 | ``` 54 | 55 | You may also force multiline by starting a line with `=` immediately followed by a newline. Any text indented after this will be treated as an Elixir expression, regardless of what each line ends in. 56 | 57 | ```jade 58 | = 59 | render App.PageView, 60 | "index.html", 61 | [conn: @conn] ++ 62 | assigns 63 | ``` 64 | -------------------------------------------------------------------------------- /docs/syntax/comments.md: -------------------------------------------------------------------------------- 1 | # Syntax: Comments 2 | 3 | Comments begin with `//-`. 4 | 5 | ```jade 6 | //- This is a comment 7 | ``` 8 | 9 | You may nest under it, and those lines will be ignored. 10 | 11 | ```jade 12 | //- everything here is ignored: 13 | a(href="/") 14 | | Link 15 | ``` 16 | 17 | `-#` is also supported to keep consistency with Elixir. 18 | 19 | ```jade 20 | -# This is also a comment 21 | ``` 22 | 23 | HTML comments 24 | ------------- 25 | 26 | HTML comments begin with `//` (no hyphen). They will be rendered as ``. 27 | 28 | ```jade 29 | // This is a comment 30 | ``` 31 | 32 | Also see 33 | -------- 34 | 35 | -><%= \"\\n\" %><%= @message %><%= \"\\n\" %>
\n"
37 |
38 | ## Internal notes
39 |
40 | `Expug.to_eex/2` pieces together 4 steps into a pipeline:
41 |
42 | - `tokenize/2` - turns source into tokens.
43 | - `compile/2` - turns tokens into an AST.
44 | - `build/2` - turns an AST into a line map.
45 | - `stringify/2` - turns a line map into an EEx template.
46 |
47 | ## Also see
48 |
49 | - `Expug.Tokenizer`
50 | - `Expug.Compiler`
51 | - `Expug.Builder`
52 | - `Expug.Stringifier`
53 | """
54 |
55 | defdelegate tokenize(source, opts), to: Expug.Tokenizer
56 | defdelegate compile(tokens, opts), to: Expug.Compiler
57 | defdelegate build(ast, opts), to: Expug.Builder
58 | defdelegate stringify(lines, opts), to: Expug.Stringifier
59 |
60 | @doc ~S"""
61 | Compiles an Expug template to an EEx template.
62 |
63 | Returns `{:ok, result}`, where `result` is an EEx string. On error, it will
64 | return `{:error, ...}`.
65 |
66 | ## Options
67 | All options are optional.
68 |
69 | * `attr_helper` (String) - the attribute helper to use (default: `"Expug.Runtime.attr"`)
70 | * `raw_helper` (String) - the raw helper to use (default: `"raw"`)
71 | """
72 | def to_eex(source, opts \\ []) do
73 | try do
74 | eex = source
75 | |> tokenize(opts)
76 | |> compile(opts)
77 | |> build(opts)
78 | |> stringify(opts)
79 | {:ok, eex}
80 | catch %{type: _type} = err->
81 | {:error, err}
82 | end
83 | end
84 |
85 | @doc ~S"""
86 | Compiles an Expug template to an EEx template and raises errors on failure.
87 |
88 | Returns the EEx string on success. On failure, it raises `Expug.Error`.
89 | """
90 | def to_eex!(source, opts \\ []) do
91 | case to_eex(source, opts) do
92 | {:ok, eex} ->
93 | eex
94 | {:error, err} ->
95 | err = err |> Map.put(:source, source)
96 | raise Expug.Error.exception(err)
97 | end
98 | end
99 | end
100 |
--------------------------------------------------------------------------------
/lib/expug/builder.ex:
--------------------------------------------------------------------------------
1 | defmodule Expug.Builder do
2 | @moduledoc ~S"""
3 | Builds lines from an AST.
4 |
5 | iex> source = "div\n | Hello"
6 | iex> with tokens <- Expug.Tokenizer.tokenize(source),
7 | ...> ast <- Expug.Compiler.compile(tokens),
8 | ...> lines <- Expug.Builder.build(ast),
9 | ...> do: lines
10 | %{
11 | :lines => 2,
12 | 1 => ["
16 | Hello,
17 | <%= @name %>
18 | <%
19 | %><%="\n"%>
21 | ```
22 |
23 | ## Internal notes
24 |
25 | `Expug.Builder` brings this output:
26 |
27 | ```js
28 | lines = %{
29 | :lines => 6,
30 | 1 => [ "" ],
31 | 2 => [ "Hello," ],
32 | 3 => [ "<%= @name %>" ],
33 |
34 | 5 => [ "", "
" ]
36 | }
37 | ```
38 |
39 | `Expug.Stringifier` will take this and yield a final EEx string. The rules it follows are:
40 |
41 | - Multiline lines (like 6) will be joined with a fake newline (`<%= "\n" %>`).
42 | - Empty lines (like line 4) will start with `<%`, with a final `%>` in the next line that has something.
43 |
--------------------------------------------------------------------------------
/docs/misc/prior_art.md:
--------------------------------------------------------------------------------
1 | # Misc: Prior art
2 |
3 | > a.k.a., "Why should I use Expug over other template engines?"
4 |
5 | There's [calliope] and [slime] that brings Haml and Slim to Elixir, respectively. Expug offers a bit more:
6 |
7 | ## Pug/Jade syntax!
8 |
9 | The Pug syntax is something I personally find more sensible than Slim, and less noisy than Haml.
10 |
11 | ```
12 | # Expug
13 | p.alert(align="center") Hello!
14 |
15 | # HAML
16 | %p.alert{align: "center"} Hello!
17 |
18 | # Slime
19 | p.alert align="center" Hello!
20 | ```
21 |
22 | Expug tries to infer what you mean based on balanced parentheses. In contrast, you're forced to use `"#{...}"` in slime.
23 |
24 | ```
25 | # Expug
26 | script(src=static_path(@conn, "/js/app.js") type="text/javascript")
27 | # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
28 |
29 | # Slime
30 | script[src="#{static_path(@conn, "/js/app.js")}" type="text/javascript"]
31 | ```
32 |
33 | Also notice that you're forced to use `[` in Slime if your attributes have `(` in it. Expug doesn't have this restriction.
34 |
35 | ```
36 | # Slime
37 | a(href="/")
38 | a(href="/link")
39 | a[href="/link" onclick="alert('Are you sure?')"]
40 | ```
41 |
42 | Slime has optional braces, which leads to a lot of confusion. In Expug, parentheses are required.
43 |
44 | ```
45 | # Slime
46 | strong This is bold text.
47 | strong color="blue" This is also valid, but confusing.
48 |
49 | # Expug
50 | strong(color="blue") Easier and less confusing!
51 | ```
52 |
53 |
54 | ## True multilines
55 |
56 | Expug has a non-line-based tokenizer that can figure out multiline breaks.
57 |
58 | ```
59 | # Expug
60 | = render App.UserView,
61 | "show.html",
62 | conn: @conn
63 |
64 | div(
65 | style="font-weight: bold"
66 | role="alert"
67 | )
68 | ```
69 |
70 | Using brace-matching, Expug's parser can reliably figure out what you mean.
71 |
72 | ```
73 | # Expug
74 | script(
75 | src=static_path(
76 | @conn,
77 | "/js/app.js"))
78 | ```
79 |
80 | ## Correct line number errors
81 |
82 | Errors in Expug will always map to the correct source line numbers.
83 |
84 | > CompileError in show.html.pug (line 2):85 | > assign @xyz not available in eex template. 86 | 87 | [calliope]: https://github.com/nurugger07/calliope 88 | [slime]: https://github.com/slime-lang/slime 89 | -------------------------------------------------------------------------------- /docs/syntax/code.md: -------------------------------------------------------------------------------- 1 | # Syntax: Code 2 | 3 | ## Unbuffered code 4 | Unbuffered code starts with `-` does not add any output directly. 5 | 6 | ```jade 7 | - name = assigns.name 8 | div(id="name-#{name}") 9 | ``` 10 | 11 | ## Bufferred code 12 | 13 | Buffered code starts with `=` and outputs the result of evaluating the Elixir expression in the template. For security, it is first HTML escaped. 14 | 15 | ```jade 16 | p= "Hello, #{name}" 17 | ``` 18 | 19 | ## Unescaped code 20 | 21 | Buffered code may be unescaped by using `!=`. This skips the HTML escaping. 22 | 23 | ```jade 24 | div!= markdown_to_html(@article.body) |> sanitize() 25 | ``` 26 | 27 | ## Conditionals and Loops 28 | 29 | For `if`, `cond`, `try`, `for`, an `end` statement is automatically inferred. 30 | 31 | ```jade 32 | = if assigns.name do 33 | = "Hello, #{@name}" 34 | ``` 35 | 36 | They also need to begin with `=`, not `-`. Except for `else`, `rescue` and so on. 37 | 38 | ```jade 39 | = if assigns.current_user do 40 | | Welcome. 41 | - else 42 | | You are not signed in. 43 | ``` 44 | 45 | ## Multiline 46 | 47 | If a line ends in one of these characters: `,` `(` `{` `[`, the next line is considered to be part of the Elixir expression. 48 | 49 | ```jade 50 | = render App.PageView, 51 | "index.html", 52 | conn: @conn 53 | ``` 54 | 55 | You may also force multiline by starting a line with `=` immediately followed by a newline. Any text indented after this will be treated as an Elixir expression, regardless of what each line ends in. 56 | 57 | ```jade 58 | = 59 | render App.PageView, 60 | "index.html", 61 | [conn: @conn] ++ 62 | assigns 63 | ``` 64 | -------------------------------------------------------------------------------- /docs/syntax/comments.md: -------------------------------------------------------------------------------- 1 | # Syntax: Comments 2 | 3 | Comments begin with `//-`. 4 | 5 | ```jade 6 | //- This is a comment 7 | ``` 8 | 9 | You may nest under it, and those lines will be ignored. 10 | 11 | ```jade 12 | //- everything here is ignored: 13 | a(href="/") 14 | | Link 15 | ``` 16 | 17 | `-#` is also supported to keep consistency with Elixir. 18 | 19 | ```jade 20 | -# This is also a comment 21 | ``` 22 | 23 | HTML comments 24 | ------------- 25 | 26 | HTML comments begin with `//` (no hyphen). They will be rendered as ``. 27 | 28 | ```jade 29 | // This is a comment 30 | ``` 31 | 32 | Also see 33 | -------- 34 | 35 | -
\nHello<%= \"\\n\" %>
\n"}
10 |
11 | `to_eex!/2` is the same, and instead returns the result or throws an
12 | `Expug.Error`.
13 |
14 | iex> source = "div\n | Hello"
15 | iex> Expug.to_eex!(source)
16 | "\nHello<%= \"\\n\" %>
\n"
17 |
18 | ## Errors
19 | `to_eex/2` will give you this in case of an error:
20 |
21 | {:error, %{
22 | type: :parse_error,
23 | position: {3, 2}, # line/col
24 | ... # other metadata
25 | }}
26 |
27 | Internally, the other classes will throw `%{type, position, ...}` which will
28 | be caught here.
29 |
30 | ## The `raw` helper
31 | Note that it needs `raw/1`, something typically provided by
32 | [Phoenix.HTML](http://devdocs.io/phoenix/phoenix_html/phoenix.html#raw/1).
33 | You don't need Phoenix.HTML however; a binding with `raw/1` would do.
34 |
35 | iex> Expug.to_eex!(~s[div(role="alert")= @message])
36 | ""],
13 | 2 => ["Hello", "
"]
14 | }
15 |
16 | This gives you a map of lines that the `Stringifier` will work on.
17 |
18 | ## Also see
19 | - `Expug.Compiler` builds the AST used by this builder.
20 | - `Expug.Stringifier` takes this builder's output.
21 | """
22 |
23 | require Logger
24 |
25 | # See: http://www.w3.org/TR/html5/syntax.html#void-elements
26 | @void_elements ["area", "base", "br", "col", "embed", "hr", "img", "input",
27 | "keygen", "link", "meta", "param", "source", "track", "wbr"]
28 |
29 | @defaults %{
30 | attr_helper: "Expug.Runtime.attr",
31 | raw_helper: "raw"
32 | }
33 |
34 | def build(ast, opts \\ []) do
35 | opts = Enum.into(opts, @defaults)
36 |
37 | %{lines: 0, options: opts, doctype: nil}
38 | |> make(ast)
39 | |> Map.delete(:options)
40 | end
41 |
42 | @doc """
43 | Builds elements.
44 | """
45 | def make(doc, %{type: :document} = node) do
46 | doc
47 | |> Map.put(:doctype, :html)
48 | |> make(node[:doctype])
49 | |> children(node[:children])
50 | |> Map.delete(:doctype)
51 | end
52 |
53 | def make(doc, %{type: :doctype, value: "html5"} = node) do
54 | doc
55 | |> put(node, "")
56 | end
57 |
58 | def make(doc, %{type: :doctype, value: "xml"} = node) do
59 | doc
60 | |> Map.put(:doctype, :xml)
61 | |> put(node, ~s())
62 | end
63 |
64 | def make(doc, %{type: :doctype, value: value} = node) do
65 | doc
66 | |> put(node, "")
67 | end
68 |
69 | @doc """
70 | Builds elements.
71 | """
72 | def make(doc, %{type: :element, children: list} = node) do
73 | doc
74 | |> put(node, element(doc, node))
75 | |> children(list)
76 | |> put_last(" node[:name] <> ">")
77 | end
78 |
79 | def make(doc, %{type: :element} = node) do
80 | doc
81 | |> put(node, self_closing_element(doc, node))
82 | end
83 |
84 | def make(doc, %{type: :statement, value: value, children: [_|_] = list} = node) do
85 | doc
86 | |> put(node, "<% #{value} %>")
87 | |> put_collapse(node)
88 | |> children(list)
89 | |> add_closing(node)
90 | end
91 |
92 | def make(doc, %{type: :statement, value: value} = node) do
93 | doc
94 | |> put(node, "<% #{value} %>")
95 | end
96 |
97 | @doc """
98 | Builds text.
99 | """
100 | def make(doc, %{type: :raw_text, value: value} = node) do
101 | doc
102 | |> put(node, "#{value}")
103 | end
104 |
105 | def make(doc, %{type: :buffered_text, value: value, children: [_|_] = list} = node) do
106 | doc
107 | |> put(node, "<%= #{value} %>")
108 | |> put_collapse(node)
109 | |> children(list)
110 | |> add_closing(node)
111 | end
112 |
113 | def make(doc, %{type: :buffered_text, value: value} = node) do
114 | doc
115 | |> put(node, "<%= #{value} %>")
116 | end
117 |
118 | def make(doc, %{type: :html_comment, value: value} = node) do
119 | doc
120 | |> put(node, "")
121 | end
122 |
123 | # Handle `!= for item <- list do` (has children)
124 | def make(doc, %{type: :unescaped_text, value: value, children: [_|_] = list} = node) do
125 | %{options: %{raw_helper: raw}} = doc
126 | doc
127 | |> put(node, "<%= #{raw}(#{value} %>")
128 | |> put_collapse(node)
129 | |> children(list)
130 | |> add_closing(node, ")")
131 | end
132 |
133 | # Handle `!= @hello`
134 | def make(doc, %{type: :unescaped_text, value: value} = node) do
135 | %{options: %{raw_helper: raw}} = doc
136 | case node[:open] do
137 | true ->
138 | doc
139 | |> put(node, "<%= #{raw}(#{value} %>")
140 | _ ->
141 | doc
142 | |> put(node, "<%= #{raw}(#{value}) %>")
143 | end
144 | end
145 |
146 | def make(doc, %{type: :block_text, value: value} = node) do
147 | doc
148 | |> put(node, value)
149 | end
150 |
151 | def make(doc, nil) do
152 | doc
153 | end
154 |
155 | def make(_doc, %{type: type, token: {position, _, _}}) do
156 | throw %{
157 | type: :cant_build_node,
158 | node_type: type,
159 | position: position
160 | }
161 | end
162 |
163 | def add_closing(doc, node, suffix \\ "")
164 | def add_closing(doc, %{close: close}, suffix) do
165 | doc
166 | |> put_last_no_space("<% #{close}#{suffix} %>")
167 | end
168 |
169 | def add_closing(doc, _, _), do: doc
170 |
171 | @doc """
172 | Builds a list of nodes.
173 | """
174 | def children(doc, nil) do
175 | doc
176 | end
177 |
178 | def children(doc, list) do
179 | Enum.reduce list, doc, fn node, doc ->
180 | make(doc, node)
181 | end
182 | end
183 |
184 | @doc """
185 | Builds an element opening tag.
186 | """
187 |
188 | def element(doc, node) do
189 | "<" <> node[:name] <> attributes(doc, node[:attributes]) <> ">"
190 | end
191 |
192 | def self_closing_element(doc, node) do
193 | tag = node[:name] <> attributes(doc, node[:attributes])
194 | cond do
195 | doc[:doctype] == :xml ->
196 | "<#{tag} />"
197 | self_closable?(node) ->
198 | "<#{tag}>"
199 | true ->
200 | "<#{tag}>"
201 | end
202 | end
203 |
204 | def self_closable?(node) do
205 | Enum.any?(@void_elements, &(&1 == node[:name])) && true
206 | end
207 |
208 | @doc ~S"""
209 | Stringifies an attributes map.
210 |
211 | iex> doc = %{options: %{}}
212 | iex> Expug.Builder.attributes(doc, %{ "src" => [{:text, "image.jpg"}] })
213 | " src=\"image.jpg\""
214 |
215 | #iex> doc = %{options: %{}}
216 | #iex> Expug.Builder.attributes(doc, %{ "class" => [{:text, "a"}, {:text, "b"}] })
217 | #" class=\"a b\""
218 |
219 | iex> doc = %{options: %{attr_helper: "attr", raw_helper: "raw"}}
220 | iex> Expug.Builder.attributes(doc, %{ "src" => [{:eval, "@image"}] })
221 | "<%= raw(attr(\"src\", @image)) %>"
222 |
223 | iex> doc = %{options: %{attr_helper: "attr", raw_helper: "raw"}}
224 | iex> Expug.Builder.attributes(doc, %{ "class" => [{:eval, "@a"}, {:eval, "@b"}] })
225 | "<%= raw(attr(\"class\", Enum.join([@a, @b], \" \"))) %>"
226 | """
227 | def attributes(_doc, nil), do: ""
228 |
229 | def attributes(doc, %{} = attributes) do
230 | Enum.reduce attributes, "", fn {key, values}, acc ->
231 | acc <> valueify(doc, key, values)
232 | end
233 | end
234 |
235 | def valueify(doc, key, [{:eval, value}]) do
236 | %{options: %{attr_helper: attr, raw_helper: raw}} = doc
237 | "<%= #{raw}(#{attr}(#{inspect(key)}, #{value})) %>"
238 | end
239 |
240 | def valueify(_doc, key, [{:text, value}]) do
241 | Expug.Runtime.attr(key, value)
242 | end
243 |
244 | def valueify(doc, key, values) when length(values) > 1 do
245 | %{options: %{attr_helper: attr, raw_helper: raw}} = doc
246 | inside = Enum.reduce values, "", fn
247 | {:eval, value}, acc ->
248 | acc |> str_join(value, ", ")
249 | {:text, value}, acc ->
250 | acc |> str_join(Expug.Runtime.attr_value(value), ", ")
251 | end
252 |
253 | "<%= #{raw}(#{attr}(#{inspect(key)}, Enum.join([#{inside}], \" \"))) %>"
254 | end
255 |
256 | def str_join(left, str, sep \\ " ")
257 | def str_join("", str, _sep), do: str
258 | def str_join(left, str, sep), do: left <> sep <> str
259 |
260 | @doc """
261 | Adds a line based on a token's location.
262 | """
263 | def put(%{lines: max} = doc, %{token: {{line, _col}, _, _}}, str) do
264 | doc
265 | |> update_line_count(line, max)
266 | |> Map.update(line, [str], &(&1 ++ [str]))
267 | end
268 |
269 | @doc """
270 | Adds a line to the end of a document.
271 | Used for closing tags.
272 | """
273 | def put_last(%{lines: line} = doc, str) do
274 | doc
275 | |> Map.update(line, [str], &(&1 ++ [str]))
276 | end
277 |
278 | @doc """
279 | Puts a collapser on the lane after the given token.
280 | Used for if...end statements.
281 | """
282 | def put_collapse(%{lines: max} = doc, %{token: {{line, _col}, _, _}}) do
283 | doc
284 | |> update_line_count(line + 1, max)
285 | |> Map.update(line + 1, [:collapse], &(&1 ++ [:collapse]))
286 | end
287 |
288 | @doc """
289 | Adds a line to the end of a document, but without a newline before it.
290 | Used for closing `<% end %>`.
291 | """
292 | def put_last_no_space(%{lines: line} = doc, str) do
293 | doc
294 | |> Map.update(line, [str], fn segments ->
295 | List.update_at(segments, -1, &(&1 <> str))
296 | end)
297 | end
298 |
299 | @doc """
300 | Updates the `:lines` count if the latest line is beyond the current max.
301 | """
302 | def update_line_count(doc, line, max) when line > max do
303 | Map.put(doc, :lines, line)
304 | end
305 |
306 | def update_line_count(doc, _line, _max) do
307 | doc
308 | end
309 | end
310 |
--------------------------------------------------------------------------------
/lib/expug/compiler.ex:
--------------------------------------------------------------------------------
1 | defmodule Expug.Compiler do
2 | @moduledoc """
3 | Compiles tokens into an AST.
4 |
5 | ## How it works
6 | Nodes are maps with a `:type` key. They are then filled up using a function
7 | with the same name as the type:
8 |
9 | node = %{type: :document}
10 | document({node, tokens})
11 |
12 | This function returns another `{node, tokens}` tuple, where `node` is the
13 | updated node, and `tokens` are the rest of the tokens to parse.
14 |
15 | The functions (`document/1`) here can do 1 of these things:
16 |
17 | - Spawn a child, say, `%{type: :element}`, then delegate to its function (eg, `element()`).
18 | - Simply return a `{node, tokens}` - no transformation here.
19 |
20 | The functions `indent()` and `statement()` are a little different. It can
21 | give you an element, or a text node, or whatever.
22 |
23 | ## Also see
24 | - `Expug.Tokenizer` is used to build the tokens used by this compiler.
25 | - `Expug.Builder` uses the AST made by this compiler.
26 | """
27 |
28 | require Logger
29 |
30 | @doc """
31 | Compiles tokens. Returns `{:ok, ast}` on success.
32 |
33 | On failure, it returns `{:error, [type: type, position: {line, col}]}`.
34 | """
35 | def compile(tokens, _opts \\ []) do
36 | tokens = Enum.reverse(tokens)
37 | node = %{type: :document}
38 |
39 | try do
40 | {node, _tokens} = document({node, tokens})
41 | node = Expug.Transformer.transform(node)
42 | node
43 | catch
44 | {:compile_error, type, [{pos, token, _} | _]} ->
45 | # TODO: create an EOF token
46 | throw %{ type: type, position: pos, token_type: token }
47 | {:compile_error, type, []} ->
48 | throw %{ type: type }
49 | end
50 | end
51 |
52 | @doc """
53 | A document.
54 | """
55 | def document({node, [{_, :doctype, type} = t | tokens]}) do
56 | node = Map.put(node, :doctype, %{
57 | type: :doctype,
58 | value: type,
59 | token: t
60 | })
61 | indent({node, tokens}, [0])
62 | end
63 |
64 | def document({node, tokens}) do
65 | indent({node, tokens}, [0]) # optional
66 | end
67 |
68 | @doc """
69 | Indentation. Called with `depth` which is the current level its at.
70 | """
71 | def indent({node, [{_, :indent, subdepth} | [_|_] = tokens]}, [d | _] = depths)
72 | when subdepth > d do
73 | if node[:children] == nil do
74 | throw {:compile_error, :unexpected_indent, hd(tokens)}
75 | end
76 |
77 | # Found children, start a new subtree.
78 | [child | rest] = Enum.reverse(node[:children] || [])
79 | {child, tokens} = statement({child, tokens}, [ subdepth | depths ])
80 | |> indent([ subdepth | depths ])
81 |
82 | # Go back to our tree.
83 | children = Enum.reverse([child | rest])
84 | node = Map.put(node, :children, children)
85 | {node, tokens}
86 | |> indent(depths)
87 | end
88 |
89 | def indent({node, [{_, :indent, subdepth} | [_|_] = tokens]}, [d | _] = depths)
90 | when subdepth == d do
91 | {node, tokens}
92 | |> statement(depths)
93 | |> indent(depths)
94 | end
95 |
96 | def indent({node, [{_, :indent, subdepth} | [_|_]] = tokens}, [d | _])
97 | when subdepth < d do
98 | # throw {:compile_error, :ambiguous_indentation, token}
99 | {node, tokens}
100 | end
101 |
102 | # End of file, no tokens left.
103 | def indent({node, []}, _depth) do
104 | {node, []}
105 | end
106 |
107 | def indent({_node, tokens}, _depth) do
108 | throw {:compile_error, :unexpected_token, tokens}
109 | end
110 |
111 | @doc """
112 | A statement after an `:indent`.
113 | Can consume these:
114 |
115 | :element_name
116 | :element_class
117 | :element_id
118 | [:attribute_open [...] :attribute_close]
119 | [:buffered_text | :unescaped_text | :raw_text | :block_text]
120 | """
121 | def statement({node, [{_, :line_comment, _} | [{_, :subindent, _} | _] = tokens]}, _depths) do
122 | # Pretend to be an element and capture stuff into it; discard it afterwards.
123 | # This is wrong anyway; it should be tokenized differently.
124 | subindent({node, tokens})
125 | end
126 |
127 | def statement({node, [{_, :line_comment, _} | tokens]}, _depths) do
128 | {node, tokens}
129 | end
130 |
131 | def statement({node, [{_, :html_comment, value} = t | tokens]}, _depths) do
132 | child = %{type: :html_comment, value: value, token: t}
133 | {child, tokens} = append_subindent({child, tokens})
134 | node = add_child(node, child)
135 | {node, tokens}
136 | end
137 |
138 | def statement({node, [{_, :element_name, _} = t | _] = tokens}, depths) do
139 | add_element(node, t, tokens, depths)
140 | end
141 |
142 | def statement({node, [{_, :element_class, _} = t | _] = tokens}, depths) do
143 | add_element(node, t, tokens, depths)
144 | end
145 |
146 | def statement({node, [{_, :element_id, _} = t | _] = tokens}, depths) do
147 | add_element(node, t, tokens, depths)
148 | end
149 |
150 | def statement({node, [{_, :raw_text, value} = t | tokens]}, _depth) do
151 | child = %{type: :raw_text, value: value, token: t}
152 | node = add_child(node, child)
153 | {node, tokens}
154 | end
155 |
156 | def statement({node, [{_, :buffered_text, value} = t | tokens]}, _depth) do
157 | child = %{type: :buffered_text, value: value, token: t}
158 | {child, tokens} = append_subindent({child, tokens})
159 | node = add_child(node, child)
160 | {node, tokens}
161 | end
162 |
163 | def statement({node, [{_, :unescaped_text, value} = t | tokens]}, _depth) do
164 | child = %{type: :unescaped_text, value: value, token: t}
165 | {child, tokens} = append_subindent({child, tokens})
166 | node = add_child(node, child)
167 | {node, tokens}
168 | end
169 |
170 | def statement({node, [{_, :statement, value} = t | tokens]}, _depth) do
171 | child = %{type: :statement, value: value, token: t}
172 | {child, tokens} = append_subindent({child, tokens})
173 | node = add_child(node, child)
174 | {node, tokens}
175 | end
176 |
177 | def statement({_node, tokens}, _depths) do
178 | throw {:compile_error, :unexpected_token, tokens}
179 | end
180 |
181 | @doc """
182 | Consumes `:subindent` tokens and adds them to the `value` of `node`.
183 | """
184 | def append_subindent({node, [{_, :subindent, value} | tokens]}) do
185 | node = node
186 | |> Map.update(:value, value, &(&1 <> "\n#{value}"))
187 | {node, tokens}
188 | |> append_subindent()
189 | end
190 |
191 | def append_subindent({node, tokens}) do
192 | {node, tokens}
193 | end
194 |
195 | def add_element(node, t, tokens, depth) do
196 | child = %{type: :element, name: "div", token: t}
197 | {child, rest} = element({child, tokens}, node, depth)
198 | node = add_child(node, child)
199 | {node, rest}
200 | end
201 |
202 | @doc """
203 | Parses an element.
204 | Returns a `%{type: :element}` node.
205 | """
206 | def element({node, tokens}, parent, depths) do
207 | case tokens do
208 | [{_, :element_name, value} | rest] ->
209 | node = Map.put(node, :name, value)
210 | element({node, rest}, parent, depths)
211 |
212 | [{_, :element_id, value} | rest] ->
213 | attr_list = add_attribute(node[:attributes] || %{}, "id", {:text, value})
214 | node = Map.put(node, :attributes, attr_list)
215 | element({node, rest}, parent, depths)
216 |
217 | [{_, :element_class, value} | rest] ->
218 | attr_list = add_attribute(node[:attributes] || %{}, "class", {:text, value})
219 | node = Map.put(node, :attributes, attr_list)
220 | element({node, rest}, parent, depths)
221 |
222 | [{_, :raw_text, value} = t | rest] ->
223 | # should be in children
224 | child = %{type: :raw_text, value: value, token: t}
225 | node = add_child(node, child)
226 | element({node, rest}, parent, depths)
227 |
228 | [{_, :buffered_text, value} = t | rest] ->
229 | child = %{type: :buffered_text, value: value, token: t}
230 | {child, rest} = append_subindent({child, rest})
231 | node = add_child(node, child)
232 | element({node, rest}, parent, depths)
233 |
234 | [{_, :unescaped_text, value} = t | rest] ->
235 | child = %{type: :unescaped_text, value: value, token: t}
236 | {child, rest} = append_subindent({child, rest})
237 | node = add_child(node, child)
238 | element({node, rest}, parent, depths)
239 |
240 | [{_, :block_text, _} | rest] ->
241 | t = hd(rest)
242 | {rest, lines} = subindent_capture(rest)
243 | child = %{type: :block_text, value: Enum.join(lines, "\n"), token: t}
244 | node = add_child(node, child)
245 | element({node, rest}, parent, depths)
246 |
247 | [{_, :attribute_open, _} | rest] ->
248 | {attr_list, rest} = attribute({node[:attributes] || %{}, rest})
249 | node = Map.put(node, :attributes, attr_list)
250 | element({node, rest}, parent, depths)
251 |
252 | tokens ->
253 | {node, tokens}
254 | end
255 | end
256 |
257 | @doc """
258 | Returns a list of `[type: :attribute]` items.
259 | """
260 | def attribute({attr_list, tokens}) do
261 | case tokens do
262 | [{_, :attribute_key, key}, {_, :attribute_value, value} | rest] ->
263 | attr_list = add_attribute(attr_list, key, {:eval, value})
264 | {attr_list, rest}
265 | |> attribute()
266 |
267 | [{_, :attribute_key, key} | rest] ->
268 | attr_list = add_attribute(attr_list, key, {:eval, true})
269 | {attr_list, rest}
270 | |> attribute()
271 |
272 | [{_, :attribute_close, _} | rest] ->
273 | {attr_list, rest}
274 |
275 | rest ->
276 | {attr_list, rest}
277 | end
278 | end
279 |
280 | def add_attribute(list, key, value) do
281 | Map.update(list, key, [value], &(&1 ++ [value]))
282 | end
283 |
284 | @doc """
285 | Adds a child to a Node.
286 |
287 | iex> Expug.Compiler.add_child(%{}, %{type: :a})
288 | %{children: [%{type: :a}]}
289 |
290 | iex> src = %{children: [%{type: :a}]}
291 | ...> Expug.Compiler.add_child(src, %{type: :b})
292 | %{children: [%{type: :a}, %{type: :b}]}
293 | """
294 | def add_child(node, child) do
295 | Map.update(node, :children, [child], &(&1 ++ [child]))
296 | end
297 |
298 | @doc """
299 | Matches `:subindent` tokens and discards them. Used for line comments (`-#`).
300 | """
301 | def subindent({node, [{_, :subindent, _} | rest]}) do
302 | subindent({node, rest})
303 | end
304 |
305 | def subindent({node, rest}) do
306 | {node, rest}
307 | end
308 |
309 | def subindent_capture(tokens, lines \\ [])
310 | def subindent_capture([{_, :subindent, line} | rest], lines) do
311 | lines = lines ++ [line]
312 | subindent_capture(rest, lines)
313 | end
314 |
315 | def subindent_capture(rest, lines) do
316 | {rest, lines}
317 | end
318 | end
319 |
--------------------------------------------------------------------------------
/lib/expug/expression_tokenizer.ex:
--------------------------------------------------------------------------------
1 | defmodule Expug.ExpressionTokenizer do
2 | @moduledoc ~S"""
3 | Tokenizes an expression.
4 | This is used by `Expug.Tokenizer` to match attribute values and support multiline.
5 |
6 | `expression/2` is used to capture an expression token.
7 |
8 | state
9 | |> Expug.ExpressionTokenizer.expression(:attribute_value)
10 |
11 | ## Valid expressions
12 | Expressions are combination of one or more of these:
13 |
14 | - a word without spaces
15 | - a balanced `(` ... `)` pair (or `[`, or `{`)
16 | - a string with single quotes `'...'` or double quotes `"..."`
17 |
18 | A balanced pair can have balanced pairs, words, and strings inside them.
19 | Double-quote strings can have `#{...}` interpolation inside them.
20 |
21 | ## Examples
22 | These are valid expressions:
23 |
24 | hello
25 | hello(1 + 2)
26 | "Hello world" # strings
27 | (hello world) # balanced (...) pair
28 |
29 | These aren't:
30 |
31 | hello world # spaces
32 | hello(world[) # pairs not balanced
33 | "hello #{foo(}" # not balanced inside an interpolation
34 | """
35 |
36 | import Expug.TokenizerTools
37 |
38 | def expression(state, token_name) do
39 | state
40 | |> start_empty(token_name)
41 | |> many_of(&expression_fragment/1)
42 | end
43 |
44 | def expression_fragment(state) do
45 | state
46 | |> one_of([
47 | &balanced_parentheses/1,
48 | &balanced_braces/1,
49 | &balanced_brackets/1,
50 | &double_quote_string/1,
51 | &single_quote_string/1,
52 | &expression_term/1
53 | ])
54 | end
55 |
56 | @doc """
57 | Matches simple expressions like `xyz` or even `a+b`.
58 | """
59 | def expression_term(state) do
60 | state
61 | |> append(~r/^[^\(\)\[\]\{\}"', \n\t]+/)
62 | end
63 |
64 | @doc """
65 | Matches simple expressions like `xyz`, but only for inside parentheses.
66 | These can have spaces.
67 | """
68 | def expression_term_inside(state) do
69 | state
70 | |> append(~r/^[^\(\)\[\]\{\}"']+/)
71 | end
72 |
73 | @doc """
74 | Matches balanced `(...)` fragments
75 | """
76 | def balanced_parentheses(state) do
77 | state
78 | |> balanced_pairs(~r/^\(/, ~r/^\)/)
79 | end
80 |
81 | @doc """
82 | Matches balanced `{...}` fragments
83 | """
84 | def balanced_braces(state) do
85 | state
86 | |> balanced_pairs(~r/^\{/, ~r/^\}/)
87 | end
88 |
89 | @doc """
90 | Matches balanced `[...]` fragments
91 | """
92 | def balanced_brackets(state) do
93 | state
94 | |> balanced_pairs(~r/^\[/, ~r/^\]/)
95 | end
96 |
97 | @doc """
98 | Underlying implementation for `balanced_*` functions
99 | """
100 | def balanced_pairs(state, left, right) do
101 | state
102 | |> append(left)
103 | |> optional(fn s -> s
104 | |> many_of(fn s -> s
105 | |> one_of([
106 | &expression_fragment/1,
107 | &expression_term_inside/1
108 | ])
109 | end)
110 | end)
111 | |> append(right)
112 | end
113 |
114 | @doc """
115 | Matches an entire double-quoted string, taking care of interpolation and escaping
116 | """
117 | def double_quote_string(state) do
118 | state
119 | |> append(~r/^"/)
120 | |> optional_many_of(fn s -> s
121 | |> one_of([
122 | &(&1 |> append(~r/^#/) |> balanced_braces()),
123 | &(&1 |> append(~r/^(?:(?:\\")|[^"])/))
124 | ])
125 | end)
126 | |> append(~r/^"/)
127 | end
128 |
129 | @doc """
130 | Matches an entire double-quoted string, taking care of escaping
131 | """
132 | def single_quote_string(state) do
133 | state
134 | |> append(~r/^'/)
135 | |> optional_many_of(&(&1 |> append(~r/^(?:(?:\\')|[^'])/)))
136 | |> append(~r/^'/)
137 | end
138 | end
139 |
--------------------------------------------------------------------------------
/lib/expug/expug_error.ex:
--------------------------------------------------------------------------------
1 | defmodule Expug.Error do
2 | @moduledoc """
3 | A parse error
4 | """
5 |
6 | defexception [:message, :line, :column]
7 |
8 | def exception(%{type: type, position: {ln, col}, source: source} = err) do
9 | {message, description} = exception_message(type, err)
10 | line_source = source |> String.split("\n") |> Enum.at(ln - 1)
11 | indent = repeat_string(col - 1, " ")
12 |
13 | %Expug.Error{
14 | message:
15 | "#{message} on line #{ln}\n\n"
16 | <> " #{line_source}\n"
17 | <> " #{indent}^\n\n"
18 | <> description,
19 | line: ln,
20 | column: col
21 | }
22 | end
23 |
24 | def exception(err) do
25 | %Expug.Error{
26 | message: "Error #{inspect(err)}"
27 | }
28 | end
29 |
30 | def repeat_string(times, string \\ " ") do
31 | 1..times |> Enum.reduce("", fn _, acc -> acc <> string end)
32 | end
33 |
34 | def exception_message(:parse_error, %{expected: _expected}) do
35 | {
36 | "Parse error",
37 | """
38 | Expug encountered a character it didn't expect.
39 | """
40 | }
41 | end
42 |
43 | def exception_message(:unexpected_indent, _) do
44 | {
45 | "Unexpected indentation",
46 | """
47 | Expug found spaces when it didn't expect any.
48 | """
49 | }
50 | end
51 |
52 | def exception_message(:ambiguous_indentation, _) do
53 | {
54 | "Ambiguous indentation",
55 | """
56 | Expug found spaces when it didn't expect any.
57 | """
58 | }
59 | end
60 |
61 | def exception_message(type, _) do
62 | {
63 | "#{type} error",
64 | """
65 | Expug encountered a #{type} error.
66 | """
67 | }
68 | end
69 | end
70 |
--------------------------------------------------------------------------------
/lib/expug/runtime.ex:
--------------------------------------------------------------------------------
1 | defmodule Expug.Runtime do
2 | @moduledoc """
3 | Functions used by Expug-compiled templates at runtime.
4 |
5 | ```eex
6 | >
7 | ```
8 | """
9 |
10 | @doc """
11 | Quotes a given `str` for use as an HTML attribute.
12 | """
13 | def attr_value(str) do
14 | "\"#{attr_value_escape(str)}\""
15 | end
16 |
17 | def attr_value_escape(str) do
18 | str
19 | |> String.replace("&", "&")
20 | |> String.replace("\"", """)
21 | |> String.replace("<", "<")
22 | |> String.replace(">", ">")
23 | end
24 |
25 | def attr(key, true) do
26 | " " <> key
27 | end
28 |
29 | def attr(_key, false) do
30 | ""
31 | end
32 |
33 | def attr(_key, nil) do
34 | ""
35 | end
36 |
37 | def attr(key, value) do
38 | " " <> key <> "=" <> attr_value(value)
39 | end
40 | end
41 |
--------------------------------------------------------------------------------
/lib/expug/stringifier.ex:
--------------------------------------------------------------------------------
1 | defmodule Expug.Stringifier do
2 | @moduledoc """
3 | Stringifies builder output.
4 |
5 | ## Also see
6 | - `Expug.Builder` builds the line map used by this stringifier.
7 | - `Expug.to_eex/1` is the main entry point that uses this stringifier.
8 | """
9 |
10 | def stringify(%{} = doc, _opts \\ []) do
11 | {max, doc} = Map.pop(doc, :lines)
12 | doc = doc
13 | |> Map.delete(:doctype)
14 | |> Map.delete(:options)
15 | list = doc |> Map.to_list() |> Enum.sort()
16 |
17 | case render_lines(list, 0, max) do
18 | # Move the newline to the end
19 | "\n" <> rest -> rest <> "\n"
20 | rest -> rest
21 | end
22 | end
23 |
24 | # Works on a list of `{2, [""]}` tuples.
25 | # Each pass works on one line.
26 | #
27 | # %{
28 | # :lines => 2,
29 | # 1 => ["![]()
"]
28 | }
29 | end
30 |
31 | test "self-closing xml" do
32 | eex = build("doctype xml\nimg")
33 | assert eex == %{
34 | :lines => 2,
35 | 1 => [""],
36 | 2 => ["![]()
"]
37 | }
38 | end
39 |
40 | test "single element" do
41 | eex = build("div")
42 | assert eex == %{
43 | :lines => 1,
44 | 1 => [""]
45 | }
46 | end
47 |
48 | test "single element with attributes" do
49 | eex = build("div(id=foo)")
50 | assert eex == %{
51 | :lines => 1,
52 | 1 => [">
"]
53 | }
54 | end
55 |
56 | test "value-less attributes" do
57 | eex = build("div(src)")
58 | assert eex == %{
59 | :lines => 1,
60 | 1 => [""],
30 | # 2 => ["", "
"]
31 | # }
32 | #
33 | # Renders into these in 2 passes:
34 | #
35 | # "\n"
36 | # "\n<%= "\n" %>
"
37 | #
38 | defp render_lines([{line, elements} | rest], last, max) do
39 | {padding, meat} = render_elements(elements, line, last)
40 | cursor = line + count_newlines(meat)
41 |
42 | padding <> meat <> render_lines(rest, cursor, max)
43 | end
44 |
45 | defp render_lines([], _last, _max) do
46 | ""
47 | end
48 |
49 | # Renders a line. If it starts with :collapse, don't give
50 | # the `\n`
51 | defp render_elements([:collapse | elements], line, last) do
52 | { padding(line, last - 1),
53 | Enum.join(elements, ~S[<%= "\n" %>]) }
54 | end
55 |
56 | defp render_elements(elements, line, last) do
57 | { "\n" <> padding(line, last),
58 | Enum.join(elements, ~S[<%= "\n" %>]) }
59 | end
60 |
61 | # Counts the amount of newlines in a string
62 | defp count_newlines(str) do
63 | length(Regex.scan(~r/\n/, str))
64 | end
65 |
66 | # Contructs `<% .. %>` padding. Used to fill in blank lines
67 | # in the source.
68 | defp padding(line, last) when line - last - 1 <= 0 do
69 | ""
70 | end
71 |
72 | defp padding(line, last) do
73 | "<%" <> newlines(line - last - 1) <> "%>"
74 | end
75 |
76 | # Gives `n` amounts of newlines.
77 | def newlines(n) when n <= 0 do
78 | ""
79 | end
80 |
81 | def newlines(n) do
82 | "\n" <> newlines(n - 1)
83 | end
84 | end
85 |
--------------------------------------------------------------------------------
/lib/expug/tokenizer.ex:
--------------------------------------------------------------------------------
1 | defmodule Expug.Tokenizer do
2 | @moduledoc ~S"""
3 | Tokenizes a Pug template into a list of tokens. The main entry point is
4 | `tokenize/1`.
5 |
6 | iex> Expug.Tokenizer.tokenize("title= name")
7 | [
8 | {{1, 8}, :buffered_text, "name"},
9 | {{1, 1}, :element_name, "title"},
10 | {{1, 1}, :indent, 0}
11 | ]
12 |
13 | Note that the tokens are reversed! It's easier to append to the top of a list
14 | rather than to the end, making it more efficient.
15 |
16 | This output is the consumed next by `Expug.Compiler`, which turns them into
17 | an Abstract Syntax Tree.
18 |
19 | ## Token types
20 |
21 | ```
22 | div.blue#box
23 | ```
24 |
25 | - `:indent` - 0
26 | - `:element_name` - `"div"`
27 | - `:element_class` - `"blue"`
28 | - `:element_id` - `"box"`
29 |
30 | ```
31 | div(name="en")
32 | ```
33 |
34 | - `:attribute_open` - `"("`
35 | - `:attribute_key` - `"name"`
36 | - `:attribute_value` - `"\"en\""`
37 | - `:attribute_close` - `")"`
38 |
39 | ```
40 | div= hello
41 | ```
42 |
43 | - `:buffered_text` - `hello`
44 |
45 | ```
46 | div!= hello
47 | ```
48 |
49 | - `:unescaped_text` - `hello`
50 |
51 | ```
52 | div hello
53 | ```
54 |
55 | - `:raw_text` - `"hello"`
56 |
57 | ```
58 | | Hello there
59 | ```
60 |
61 | - `:raw_text` - `"Hello there"`
62 |
63 | ```
64 | = Hello there
65 | ```
66 |
67 | - `:buffered_text` - `"Hello there"`
68 |
69 | ```
70 | - foo = bar
71 | ```
72 |
73 | - `:statement` - `foo = bar`
74 |
75 | ```
76 | doctype html5
77 | ```
78 |
79 | - `:doctype` - `html5`
80 |
81 | ```
82 | -# comment
83 | more comments
84 | ```
85 |
86 | - `:line_comment` - `comment`
87 | - `:subindent` - `more comments`
88 |
89 | ```
90 | // comment
91 | more comments
92 | ```
93 |
94 | - `:html_comment` - `comment`
95 | - `:subindent` - `more comments`
96 |
97 | ## Also see
98 | - `Expug.TokenizerTools` has the functions used by this tokenizer.
99 | - `Expug.Compiler` uses the output of this tokenizer to build an AST.
100 | - `Expug.ExpressionTokenizer` is used to tokenize expressions.
101 | """
102 |
103 | import Expug.TokenizerTools
104 | alias Expug.TokenizerTools.State
105 |
106 | @doc """
107 | Tokenizes a string.
108 | Returns a list of tokens. Each token is in the format `{position, token, value}`.
109 | """
110 | def tokenize(source, opts \\ []) do
111 | source = trim_trailing(source)
112 | run(source, opts, &document/1)
113 | end
114 |
115 | @doc """
116 | Matches an entire document.
117 | """
118 | def document(state) do
119 | state
120 | |> optional(&newlines/1)
121 | |> optional(&doctype/1)
122 | |> many_of(
123 | &(&1 |> element_or_text() |> newlines()),
124 | &(&1 |> element_or_text()))
125 | end
126 |
127 | @doc """
128 | Matches `doctype html`.
129 | """
130 | def doctype(state) do
131 | state
132 | |> discard(~r/^doctype/, :doctype_prelude)
133 | |> whitespace()
134 | |> eat(~r/^[^\n]+/, :doctype)
135 | |> optional(&newlines/1)
136 | end
137 |
138 | @doc """
139 | Matches an HTML element, text node, or, you know... the basic statements.
140 | I don't know what to call this.
141 | """
142 | def element_or_text(state) do
143 | state
144 | |> indent()
145 | |> one_of([
146 | &line_comment/1, # `-# hello`
147 | &html_comment/1, # `// hello`
148 | &buffered_text/1, # `= hello`
149 | &unescaped_text/1, # `!= hello`
150 | &raw_text/1, # `| hello`
151 | &statement/1, # `- hello`
152 | &element/1 # `div.blue hello`
153 | ])
154 | end
155 |
156 | @doc """
157 | Matches any number of blank newlines. Whitespaces are accounted for.
158 | """
159 | def newlines(state) do
160 | state
161 | |> discard(~r/^\n(?:[ \t]*\n)*/, :newlines)
162 | end
163 |
164 | @doc """
165 | Matches an indentation. Gives a token that looks like `{_, :indent, 2}`
166 | where the last number is the number of spaces/tabs.
167 |
168 | Doesn't really care if you use spaces or tabs; a tab is treated like a single
169 | space.
170 | """
171 | def indent(state) do
172 | state
173 | |> eat(~r/^\s*/, :indent, &[{&3, :indent, String.length(&2)} | &1])
174 | end
175 |
176 | @doc """
177 | Matches `div.foo[id="name"]= Hello world`
178 | """
179 | def element(state) do
180 | state
181 | |> element_descriptor()
182 | |> optional(&attributes_block/1)
183 | |> optional(fn s -> s
184 | |> one_of([
185 | &sole_buffered_text/1,
186 | &sole_unescaped_text/1,
187 | &sole_raw_text/1,
188 | &block_text/1
189 | ])
190 | end)
191 | end
192 |
193 | @doc """
194 | Matches `div`, `div.foo` `div.foo.bar#baz`, etc
195 | """
196 | def element_descriptor(state) do
197 | state
198 | |> one_of([
199 | &element_descriptor_full/1,
200 | &element_name/1,
201 | &element_class_or_id_list/1
202 | ])
203 | end
204 |
205 | @doc """
206 | Matches `div.foo.bar#baz`
207 | """
208 | def element_descriptor_full(state) do
209 | state
210 | |> element_name()
211 | |> element_class_or_id_list()
212 | end
213 |
214 | @doc """
215 | Matches `.foo.bar#baz`
216 | """
217 | def element_class_or_id_list(state) do
218 | state
219 | |> many_of(&element_class_or_id/1)
220 | end
221 |
222 | @doc """
223 | Matches `.foo` or `#id` (just one)
224 | """
225 | def element_class_or_id(state) do
226 | state
227 | |> one_of([ &element_class/1, &element_id/1 ])
228 | end
229 |
230 | @doc """
231 | Matches `.foo`
232 | """
233 | def element_class(state) do
234 | state
235 | |> discard(~r/^\./, :dot)
236 | |> eat(~r/^[A-Za-z0-9_\-]+/, :element_class)
237 | end
238 |
239 | @doc """
240 | Matches `#id`
241 | """
242 | def element_id(state) do
243 | state
244 | |> discard(~r/^#/, :hash)
245 | |> eat(~r/^[A-Za-z0-9_\-]+/, :element_id)
246 | end
247 |
248 | @doc """
249 | Matches `[name='foo' ...]`
250 | """
251 | def attributes_block(state) do
252 | state
253 | |> optional_whitespace()
254 | |> one_of([
255 | &attribute_bracket/1,
256 | &attribute_paren/1,
257 | &attribute_brace/1
258 | ])
259 | end
260 |
261 | def attribute_bracket(state) do
262 | state
263 | |> eat(~r/^\[/, :attribute_open)
264 | |> optional_whitespace()
265 | |> optional(&attribute_list/1)
266 | |> eat(~r/^\]/, :attribute_close)
267 | end
268 |
269 | def attribute_paren(state) do
270 | state
271 | |> eat(~r/^\(/, :attribute_open)
272 | |> optional_whitespace()
273 | |> optional(&attribute_list/1)
274 | |> eat(~r/^\)/, :attribute_close)
275 | end
276 |
277 | def attribute_brace(state) do
278 | state
279 | |> eat(~r/^\{/, :attribute_open)
280 | |> optional_whitespace()
281 | |> optional(&attribute_list/1)
282 | |> eat(~r/^\}/, :attribute_close)
283 | end
284 |
285 | @doc """
286 | Matches `foo='val' bar='val'`
287 | """
288 | def attribute_list(state) do
289 | state
290 | |> optional_whitespace_or_newline()
291 | |> many_of(
292 | &(&1 |> attribute() |> attribute_separator() |> whitespace_or_newline()),
293 | &(&1 |> attribute()))
294 | |> optional_whitespace_or_newline()
295 | end
296 |
297 | @doc """
298 | Matches an optional comma in between attributes.
299 |
300 | div(id=a class=b)
301 | div(id=a, class=b)
302 | """
303 | def attribute_separator(state) do
304 | state
305 | |> discard(~r/^,?/, :comma)
306 | end
307 |
308 | @doc """
309 | Matches `foo='val'` or `foo`
310 | """
311 | def attribute(state) do
312 | state
313 | |> one_of([
314 | &attribute_key_value/1,
315 | &attribute_key/1
316 | ])
317 | end
318 |
319 | def attribute_key_value(state) do
320 | state
321 | |> attribute_key()
322 | |> optional_whitespace()
323 | |> attribute_equal()
324 | |> optional_whitespace()
325 | |> attribute_value()
326 | end
327 |
328 | def attribute_key(state) do
329 | state
330 | |> eat(~r/^[A-Za-z][A-Za-z\-0-9:]*/, :attribute_key)
331 | end
332 |
333 | def attribute_value(state) do
334 | state
335 | |> Expug.ExpressionTokenizer.expression(:attribute_value)
336 | end
337 |
338 | def attribute_equal(state) do
339 | state
340 | |> discard(~r/^=/, :eq)
341 | end
342 |
343 | @doc "Matches whitespace; no tokens emitted"
344 | def whitespace(state) do
345 | state
346 | |> discard(~r/^[ \t]+/, :whitespace)
347 | end
348 |
349 | @doc "Matches whitespace or newline; no tokens emitted"
350 | def whitespace_or_newline(state) do
351 | state
352 | |> discard(~r/^[ \t\n]+/, :whitespace_or_newline)
353 | end
354 |
355 | def optional_whitespace(state) do
356 | state
357 | |> discard(~r/^[ \t]*/, :whitespace)
358 | end
359 |
360 | def optional_whitespace_or_newline(state) do
361 | state
362 | |> discard(~r/^[ \t\n]*/, :whitespace_or_newline)
363 | end
364 |
365 | @doc "Matches `=`"
366 | def sole_buffered_text(state) do
367 | state
368 | |> optional_whitespace()
369 | |> buffered_text()
370 | end
371 |
372 | @doc "Matches `!=`"
373 | def sole_unescaped_text(state) do
374 | state
375 | |> optional_whitespace()
376 | |> unescaped_text()
377 | end
378 |
379 | @doc "Matches text"
380 | def sole_raw_text(state) do
381 | state
382 | |> whitespace()
383 | |> eat(~r/^[^\n]+/, :raw_text)
384 | end
385 |
386 | @doc "Matches `title` in `title= hello`"
387 | def element_name(state) do
388 | state
389 | |> eat(~r/^[A-Za-z_][A-Za-z0-9:_\-]*/, :element_name)
390 | end
391 |
392 | def line_comment(state) do
393 | state
394 | |> one_of([
395 | &(&1 |> discard(~r/^\/\/-/, :line_comment)),
396 | &(&1 |> discard(~r/^-\s*(?:#|\/\/)/, :line_comment))
397 | ])
398 | |> optional_whitespace()
399 | |> eat(~r/^[^\n]*/, :line_comment)
400 | |> optional(&subindent_block/1)
401 | end
402 |
403 | def block_text(state) do
404 | state
405 | |> eat(~r/^\./, :block_text)
406 | |> subindent_block()
407 | end
408 |
409 | def subindent_block(state) do
410 | sublevel = state |> get_next_indent()
411 | state
412 | |> many_of(& &1 |> newlines() |> subindent(sublevel))
413 | end
414 |
415 | def subindent(state, level) do
416 | state
417 | |> discard(~r/^[ \t]{#{level}}/, :whitespace)
418 | |> eat(~r/^[^\n]*/, :subindent)
419 | end
420 |
421 | def get_indent([{_, :indent, text} | _]) do
422 | text
423 | end
424 |
425 | def get_indent([_ | rest]) do
426 | get_indent(rest)
427 | end
428 |
429 | def get_indent([]) do
430 | ""
431 | end
432 |
433 | def html_comment(state) do
434 | state
435 | |> discard(~r[^//], :html_comment)
436 | |> optional_whitespace()
437 | |> eat(~r/^[^\n$]*/, :html_comment)
438 | |> optional(&subindent_block/1)
439 | end
440 |
441 | def buffered_text(state) do
442 | state
443 | |> one_of([
444 | &one_line_buffered_text/1,
445 | &multiline_buffered_text/1
446 | ])
447 | end
448 |
449 | def one_line_buffered_text(state) do
450 | state
451 | |> discard(~r/^=/, :eq)
452 | |> optional_whitespace()
453 | |> eat(~r/^(?:[,\[\(\{]\s*\n|[^\n$])+/, :buffered_text)
454 | end
455 |
456 | def multiline_buffered_text(state) do
457 | state
458 | |> discard(~r/^=/, :eq)
459 | |> start_empty(:buffered_text)
460 | |> subindent_block()
461 | end
462 |
463 | def unescaped_text(state) do
464 | state
465 | |> one_of([
466 | &one_line_unescaped_text/1,
467 | &multiline_unescaped_text/1
468 | ])
469 | end
470 |
471 | def one_line_unescaped_text(state) do
472 | state
473 | |> discard(~r/^!=/, :bang_eq)
474 | |> optional_whitespace()
475 | |> eat(~r/^(?:[,\[\(\{]\s*\n|[^\n$])+/, :unescaped_text)
476 | end
477 |
478 | def multiline_unescaped_text(state) do
479 | state
480 | |> discard(~r/^!=/, :bang_eq)
481 | |> start_empty(:unescaped_text)
482 | |> subindent_block()
483 | end
484 |
485 | def raw_text(state) do
486 | state
487 | |> discard(~r/^\|/, :pipe)
488 | |> optional_whitespace()
489 | |> eat(~r/^[^\n]+/, :raw_text)
490 | end
491 |
492 | def statement(state) do
493 | state
494 | |> one_of([
495 | &one_line_statement/1,
496 | &multiline_statement/1
497 | ])
498 | end
499 |
500 | def one_line_statement(state) do
501 | state
502 | |> discard(~r/^\-/, :dash)
503 | |> optional_whitespace()
504 | |> eat(~r/^(?:[,\[\(\{]\s*\n|[^\n$])+/, :statement)
505 | end
506 |
507 | def multiline_statement(state) do
508 | state
509 | |> discard(~r/^\-/, :dash)
510 | |> start_empty(:statement)
511 | |> subindent_block()
512 | end
513 |
514 | @doc ~S"""
515 | Returns the next indentation level after some newlines.
516 | Infers the last indentation level based on `doc`.
517 |
518 | iex> source = "-#\n span"
519 | iex> doc = [{0, :indent, 0}]
520 | iex> Expug.Tokenizer.get_next_indent(%{tokens: doc, source: source, position: 2}, 0)
521 | 2
522 | """
523 | def get_next_indent(%State{tokens: doc} = state) do
524 | level = get_indent(doc)
525 | get_next_indent(state, level)
526 | end
527 |
528 | @doc ~S"""
529 | Returns the next indentation level after some newlines.
530 |
531 | iex> source = "-#\n span"
532 | iex> Expug.Tokenizer.get_next_indent(%{tokens: [], source: source, position: 2}, 0)
533 | 2
534 |
535 | iex> source = "-#\n\n\n span"
536 | iex> Expug.Tokenizer.get_next_indent(%{tokens: [], source: source, position: 2}, 0)
537 | 2
538 | """
539 | def get_next_indent(state, level) do
540 | %{tokens: [{_, :indent, sublevel} |_], position: pos} =
541 | state |> newlines() |> indent()
542 | if sublevel <= level, do: throw {:parse_error, pos, [:indent]}
543 | sublevel
544 | end
545 |
546 | # Shim for String.trim_trailing/1, which doesn't exist in Elixir 1.2.6. It
547 | # falls back to String.rstrip/1 in these cases.
548 | if Keyword.has_key?(String.__info__(:functions), :trim_trailing) do
549 | defp trim_trailing(source) do
550 | String.trim_trailing(source)
551 | end
552 | else
553 | defp trim_trailing(source) do
554 | String.rstrip(source)
555 | end
556 | end
557 | end
558 |
--------------------------------------------------------------------------------
/lib/expug/tokenizer_tools.ex:
--------------------------------------------------------------------------------
1 | defmodule Expug.TokenizerTools do
2 | @moduledoc """
3 | Builds tokenizers.
4 |
5 | defmodule MyTokenizer do
6 | import Expug.TokenizerTools
7 |
8 | def tokenizer(source)
9 | run(source, [], &document/1)
10 | end
11 |
12 | def document(state)
13 | state
14 | |> discard(%r/^doctype /, :doctype_prelude)
15 | |> eat(%r/^[a-z0-9]+/, :doctype_value)
16 | end
17 | end
18 |
19 | ## The state
20 |
21 | `Expug.TokenizerTools.State` is a struct from the `source` and `opts` given to `run/3`.
22 |
23 | %{ tokens: [], source: "...", position: 0, options: ... }
24 |
25 | `run/3` creates the state and invokes a function you give it.
26 |
27 | source = "doctype html"
28 | run(source, [], &document/1)
29 |
30 | `eat/3` tries to find the given regexp from the `source` at position `pos`.
31 | If it matches, it returns a new state: a new token is added (`:open_quote` in
32 | this case), and the position `pos` is advanced.
33 |
34 | eat(state, ~r/^"/, :open_quote)
35 |
36 | If it fails to match, it'll throw a `{:parse_error, pos, [:open_quote]}`.
37 | Roughly this translates to "parse error in position *pos*, expected to find
38 | *:open_quote*".
39 |
40 | ## Mixing and matching
41 |
42 | `eat/3` will normally be wrapped into functions for most token types.
43 |
44 | def doctype(state)
45 | state
46 | |> discard(%r/^doctype/, :doctype_prelude)
47 | |> whitespace()
48 | |> eat(%r/^[a-z0-9]+/, :doctype_value)
49 | end
50 |
51 | def whitespace(state)
52 | state
53 | |> eat(^r/[ \s\t]+, :whitespace, :nil)
54 | end
55 |
56 | `one_of/3`, `optional/2`, `many_of/2` can then be used to compose these functions.
57 |
58 | state
59 | |> one_of([ &doctype/1, &foobar/1 ])
60 | |> optional(&doctype/1)
61 | |> many_of(&doctype/1)
62 | """
63 |
64 | alias Expug.TokenizerTools.State
65 |
66 | @doc """
67 | Turns a State into a final result.
68 |
69 | Returns either `{:ok, doc}` or `{:parse_error, %{type, position, expected}}`.
70 | Guards against unexpected end-of-file.
71 | """
72 | def finalize(%State{tokens: doc, source: source, position: position}) do
73 | if String.slice(source, position..-1) != "" do
74 | expected = Enum.uniq_by(get_parse_errors(doc), &(&1))
75 | throw {:parse_error, position, expected}
76 | else
77 | doc
78 | |> scrub_parse_errors()
79 | |> convert_positions(source)
80 | end
81 | end
82 |
83 | @doc """
84 | Runs; catches parse errors and throws them properly.
85 | """
86 | def run(source, opts, fun) do
87 | state = %State{tokens: [], source: source, position: 0, options: opts}
88 | try do
89 | fun.(state)
90 | |> finalize()
91 | catch {:parse_error, position, expected} ->
92 | position = convert_positions(position, source)
93 | throw %{type: :parse_error, position: position, expected: expected}
94 | end
95 | end
96 |
97 | @doc """
98 | Extracts the last parse errors that happened.
99 |
100 | In case of failure, `run/3` will check the last parse errors
101 | that happened. Returns a list of atoms of the expected tokens.
102 | """
103 | def get_parse_errors([{_, :parse_error, expected} | rest]) do
104 | expected ++ get_parse_errors(rest)
105 | end
106 |
107 | def get_parse_errors(_) do
108 | []
109 | end
110 |
111 | @doc """
112 | Gets rid of the `:parse_error` hints in the document.
113 | """
114 | def scrub_parse_errors(doc) do
115 | Enum.reject doc, fn {_, type, _} ->
116 | type == :parse_error
117 | end
118 | end
119 |
120 | @doc """
121 | Finds any one of the given token-eater functions.
122 |
123 | state |> one_of([ &brackets/1, &braces/1, &parens/1 ])
124 | """
125 | def one_of(state, funs, expected \\ [])
126 | def one_of(%State{} = state, [fun | rest], expected) do
127 | try do
128 | fun.(state)
129 | catch {:parse_error, _, expected_} ->
130 | one_of(state, rest, expected ++ expected_)
131 | end
132 | end
133 |
134 | def one_of(%State{position: pos}, [], expected) do
135 | throw {:parse_error, pos, expected}
136 | end
137 |
138 | @doc """
139 | An optional argument.
140 |
141 | state |> optional(&text/1)
142 | """
143 | def optional(state, fun) do
144 | try do
145 | fun.(state)
146 | catch
147 | {:parse_error, _, [nil | _]} ->
148 | # These are append errors, don't bother with it
149 | state
150 |
151 | {:parse_error, err_pos, expected} ->
152 | # Add a parse error pseudo-token to the document. They will be scrubbed
153 | # later on, but it will be inspected in case of a parse error.
154 | next = {err_pos, :parse_error, expected}
155 | Map.update(state, :tokens, [next], &[next | &1])
156 | end
157 | end
158 |
159 | @doc """
160 | Checks many of a certain token.
161 | """
162 | def many_of(state, head) do
163 | many_of(state, head, head)
164 | end
165 |
166 | @doc """
167 | Checks many of a certain token, and lets you provide a different `tail`.
168 | """
169 | def many_of(state = %State{source: source, position: pos}, head, tail) do
170 | if String.slice(source, pos..-1) == "" do
171 | state
172 | else
173 | try do
174 | state |> head.() |> many_of(head, tail)
175 | catch {:parse_error, _, _} ->
176 | state |> tail.()
177 | end
178 | end
179 | end
180 |
181 | @doc """
182 | Checks many of a certain token.
183 |
184 | Syntactic sugar for `optional(s, many_of(s, ...))`.
185 | """
186 | def optional_many_of(state, head) do
187 | state
188 | |> optional(&(&1 |> many_of(head)))
189 | end
190 |
191 | @doc """
192 | Consumes a token.
193 |
194 | See `eat/4`.
195 | """
196 | def eat(state, expr) do
197 | eat(state, expr, nil, fn doc, _, _ -> doc end)
198 | end
199 |
200 | @doc """
201 | Consumes a token.
202 |
203 | state
204 | |> eat(~r/[a-z]+/, :key)
205 | |> discard(~r/\s*=\s*/, :equal)
206 | |> eat(~r/[a-z]+/, :value)
207 | """
208 | def eat(state, expr, token_name) do
209 | eat(state, expr, token_name, &([{&3, token_name, &2} | &1]))
210 | end
211 |
212 | @doc """
213 | Consumes a token, but doesn't push it to the State.
214 |
215 | state
216 | |> eat(~r/[a-z]+/, :key)
217 | |> discard(~r/\s*=\s*/, :equal)
218 | |> eat(~r/[a-z]+/, :value)
219 | """
220 | def discard(state, expr, token_name) do
221 | eat state, expr, token_name, fn state, _, _ -> state end
222 | end
223 |
224 | @doc """
225 | Consumes a token.
226 |
227 | eat state, ~r/.../, :document
228 |
229 | Returns a `State`. Available parameters are:
230 |
231 | * `state` - assumed to be a state map (given by `run/3`).
232 | * `expr` - regexp expression.
233 | * `token_name` (atom, optional) - token name.
234 | * `reducer` (function, optional) - a function.
235 |
236 | ## Reducers
237 |
238 | If `reducer` is a function, `tokens` is transformed using that function.
239 |
240 | eat state, ~r/.../, :document, &[{&3, :document, &2} | &1]
241 |
242 | # &1 == tokens in current State
243 | # &2 == matched String
244 | # &3 == position
245 |
246 | ## Also see
247 |
248 | `discard/3` will consume a token, but not push it to the State.
249 |
250 | state
251 | |> discard(~r/\s+/, :whitespace) # discard it
252 | """
253 | def eat(%{tokens: doc, source: source, position: pos} = state, expr, token_name, fun) do
254 | remainder = String.slice(source, pos..-1)
255 | case match(expr, remainder) do
256 | [term] ->
257 | length = String.length(term)
258 | state
259 | |> Map.put(:position, pos + length)
260 | |> Map.put(:tokens, fun.(doc, term, pos))
261 | nil ->
262 | throw {:parse_error, pos, [token_name]}
263 | end
264 | end
265 |
266 | @doc """
267 | Creates an token with a given `token_name`.
268 |
269 | This is functionally the same as `|> eat(~r//, :token_name)`, but using
270 | `start_empty()` can make your code more readable.
271 |
272 | state
273 | |> start_empty(:quoted_string)
274 | |> append(~r/^"/)
275 | |> append(~r/[^"]+/)
276 | |> append(~r/^"/)
277 | """
278 | def start_empty(%State{position: pos} = state, token_name) do
279 | token = {pos, token_name, ""}
280 | state
281 | |> Map.update(:tokens, [token], &[token | &1])
282 | end
283 |
284 | @doc """
285 | Like `eat/4`, but instead of creating a token, it appends to the last token.
286 |
287 | Useful alongside `start_empty()`.
288 |
289 | state
290 | |> start_empty(:quoted_string)
291 | |> append(~r/^"/)
292 | |> append(~r/[^"]+/)
293 | |> append(~r/^"/)
294 | """
295 | def append(state, expr) do
296 | # parse_error will trip here; the `nil` token name ensures parse errors
297 | # will not make it to the document.
298 | state
299 | |> eat(expr, nil, fn [ {pos, token_name, left} | rest ], right, _pos ->
300 | [ {pos, token_name, left <> right} | rest ]
301 | end)
302 | end
303 |
304 | @doc ~S"""
305 | Converts numeric positions into `{line, col}` tuples.
306 |
307 | iex> source = "div\n body"
308 | iex> doc = [
309 | ...> { 0, :indent, "" },
310 | ...> { 0, :element_name, "div" },
311 | ...> { 4, :indent, " " },
312 | ...> { 6, :element_name, "body" }
313 | ...> ]
314 | iex> Expug.TokenizerTools.convert_positions(doc, source)
315 | [
316 | { {1, 1}, :indent, "" },
317 | { {1, 1}, :element_name, "div" },
318 | { {2, 1}, :indent, " " },
319 | { {2, 3}, :element_name, "body" }
320 | ]
321 | """
322 | def convert_positions(doc, source) do
323 | offsets = String.split(source, "\n")
324 | |> Stream.map(&(String.length(&1) + 1))
325 | |> Stream.scan(&(&1 + &2))
326 | |> Enum.to_list
327 | offsets = [ 0 | offsets ]
328 | convert_position(doc, offsets)
329 | end
330 |
331 | # Converts a position number `n` to a tuple `{line, col}`.
332 | defp convert_position(pos, offsets) when is_number(pos) do
333 | line = Enum.find_index(offsets, &(pos < &1))
334 | offset = Enum.at(offsets, line - 1)
335 | col = pos - offset
336 | {line, col + 1}
337 | end
338 |
339 | defp convert_position({pos, a, b}, offsets) do
340 | {convert_position(pos, offsets), a, b}
341 | end
342 |
343 | defp convert_position([ token | rest ], offsets) do
344 | [ convert_position(token, offsets) | convert_position(rest, offsets) ]
345 | end
346 |
347 | defp convert_position([], _offsets) do
348 | []
349 | end
350 |
351 | defp match(expr, remainder) do
352 | Regex.run(expr, remainder)
353 | end
354 | end
355 |
--------------------------------------------------------------------------------
/lib/expug/tokenizer_tools/state.ex:
--------------------------------------------------------------------------------
1 | defmodule Expug.TokenizerTools.State do
2 | @moduledoc """
3 | The state used by the tokenizer.
4 |
5 | %{ tokens: [], source: "...", position: 0, options: ... }
6 |
7 | ## Also see
8 |
9 | - `Expug.TokenizerTools`
10 | """
11 | defstruct [:tokens, :source, :position, :options]
12 | end
13 |
14 |
--------------------------------------------------------------------------------
/lib/expug/transformer.ex:
--------------------------------------------------------------------------------
1 | defmodule Expug.Transformer do
2 | @moduledoc """
3 | Transforms a node after compilation.
4 | """
5 |
6 | alias Expug.Visitor
7 |
8 | # Helper for later
9 | defmacrop statement?(type) do
10 | quote do
11 | unquote(type) == :buffered_text or
12 | unquote(type) == :unescaped_text or
13 | unquote(type) == :statement
14 | end
15 | end
16 |
17 | @doc """
18 | Transforms a node.
19 | """
20 | def transform(node) do
21 | node
22 | |> Visitor.visit_children(&close_clauses/1)
23 | end
24 |
25 | @doc """
26 | Finds out what clauses can follow a given clause.
27 |
28 | iex> Expug.Transformer.clause_after("if")
29 | ["else"]
30 |
31 | iex> Expug.Transformer.clause_after("try")
32 | ["catch", "rescue", "after"]
33 |
34 | iex> Expug.Transformer.clause_after("cond")
35 | [] # nothing can follow cond
36 | """
37 | def clause_after("if"), do: ["else"]
38 | def clause_after("unless"), do: ["else"]
39 | def clause_after("try"), do: ["catch", "rescue", "after"]
40 | def clause_after("catch"), do: ["catch", "after"]
41 | def clause_after("rescue"), do: ["rescue", "after"]
42 | def clause_after(_), do: []
43 | def clause_roots(), do: ["if", "unless", "try"]
44 |
45 | @doc """
46 | Closes all possible clauses in the given `children`.
47 | """
48 | def close_clauses(children) do
49 | {_, children} = close_clause(children, clause_roots())
50 | children
51 | end
52 |
53 | @doc """
54 | Closes all a given `next` clause in the given `children`.
55 |
56 | Returns a tuple of `{status, children}` where `:status` depicts what happened
57 | on the first node given to it. `:multi` means it was matched for a multi-clause,
58 | `:single` means it was matched for a single clause, `:ok` otherwise.
59 | """
60 | def close_clause([node | children], next) do
61 | pre = prelude(node)
62 |
63 | cond do
64 | # it's a multi-clause thing (eg, if-else-end, try-rescue-after-end)
65 | # See if we're at `if`...
66 | statement?(node.type) and Enum.member?(next, pre) ->
67 | # Then check if the next one is `else`...
68 | case close_clause(children, clause_after(pre)) do
69 | {:multi, children} ->
70 | # the next one IS else, don't close and proceed
71 | node = node |> Map.put(:open, true)
72 | {:multi, [node | children]}
73 |
74 | {_, children} ->
75 | # the next one is not else, so close us up and proceed
76 | node = node
77 | |> Map.put(:open, true)
78 | |> Map.put(:close, "end")
79 | {:multi, [node | close_clauses(children)]}
80 | end
81 |
82 | # it's a single-clause thing (eg, cond do)
83 | statement?(node.type) and open?(node.value) and !Enum.member?(clause_roots(), pre) ->
84 | node = node
85 | |> Map.put(:open, true)
86 | |> Map.put(:close, "end")
87 | {:single, [node | close_clauses(children)]}
88 |
89 | # Else, just reset the chain
90 | true ->
91 | {:ok, [node | close_clauses(children)]}
92 | end
93 | end
94 |
95 | def close_clause([], _upcoming) do
96 | {:ok, []} # The last child is `if`
97 | end
98 |
99 | def close_clause(children, [] = _upcoming) do
100 | {:ok, children} # Already closed end, but there's still more
101 | end
102 |
103 | @doc """
104 | Get the prelude of a given node
105 |
106 | iex> Expug.Transformer.prelude(%{value: "if foo"})
107 | "if"
108 |
109 | iex> Expug.Transformer.prelude(%{value: "case derp"})
110 | "case"
111 |
112 | iex> Expug.Transformer.prelude(%{value: "1 + 2"})
113 | nil
114 | """
115 | def prelude(%{value: statement}) do
116 | case Regex.run(~r/\s*([a-z]+)/, statement) do
117 | [_, prelude] -> prelude
118 | _ -> nil
119 | end
120 | end
121 |
122 | def prelude(_) do
123 | nil
124 | end
125 |
126 | # Checks if a given statement is open.
127 | defp open?(statement) do
128 | has_do = Regex.run(~r/[^A-Za-z0-9_]do\s*$/, statement)
129 | has_do = has_do || Regex.run(~r/[^A-Za-z0-9_]fn.*->$/, statement)
130 | has_do && true || false
131 | end
132 | end
133 |
--------------------------------------------------------------------------------
/lib/expug/visitor.ex:
--------------------------------------------------------------------------------
1 | defmodule Expug.Visitor do
2 | @moduledoc """
3 | Internal helper for traversing an AST.
4 |
5 | iex> node = %{
6 | ...> title: "Hello",
7 | ...> children: [
8 | ...> %{title: "fellow"},
9 | ...> %{title: "humans"}
10 | ...> ]
11 | ...> }
12 | iex> Expug.Visitor.visit(node, fn node ->
13 | ...> {:ok, Map.update(node, :title, ".", &(&1 <> "."))}
14 | ...> end)
15 | %{
16 | title: "Hello.",
17 | children: [
18 | %{title: "fellow."},
19 | %{title: "humans."}
20 | ]
21 | }
22 | """
23 |
24 | @doc """
25 | Returns a function `fun` recursively across `node` and its descendants.
26 | """
27 | def visit(node, fun) do
28 | {continue, node} = fun.(node)
29 | if continue == :ok do
30 | visit_recurse(node, fun)
31 | else
32 | node
33 | end
34 | end
35 |
36 | @doc """
37 | Visits all children lists recursively across `node` and its descendants.
38 |
39 | Works just like `visit/2`, but instead of operating on nodes, it operates on
40 | node children (lists).
41 | """
42 | def visit_children(node, fun) do
43 | visit node, fn
44 | %{children: children} = node ->
45 | children = fun.(children)
46 | node = put_in(node.children, children)
47 | {:ok, node}
48 | node ->
49 | {:ok, node}
50 | end
51 | end
52 |
53 | defp visit_recurse(%{children: children} = node, fun) do
54 | Map.put(node, :children, (for c <- children, do: visit(c, fun)))
55 | end
56 |
57 | defp visit_recurse(node, _) do
58 | node
59 | end
60 | end
61 |
--------------------------------------------------------------------------------
/mix.exs:
--------------------------------------------------------------------------------
1 | defmodule Expug.Mixfile do
2 | use Mix.Project
3 |
4 | @version "0.9.2"
5 | @description """
6 | Indented shorthand templates for HTML. (pre-release)
7 | """
8 |
9 | def project do
10 | [app: :expug,
11 | version: @version,
12 | description: @description,
13 | elixir: "~> 1.2",
14 | elixirc_paths: elixirc_paths(Mix.env),
15 | build_embedded: Mix.env == :prod,
16 | start_permanent: Mix.env == :prod,
17 | source_url: "https://github.com/rstacruz/expug",
18 | homepage_url: "https://github.com/rstacruz/expug",
19 | docs: docs(),
20 | package: package(),
21 | deps: deps()]
22 | end
23 |
24 | def application do
25 | [applications: [:logger]]
26 | end
27 |
28 | defp deps do
29 | [
30 | {:earmark, "~> 1.2.3", only: :dev},
31 | {:ex_doc, "~> 0.18.1", only: :dev}
32 | ]
33 | end
34 |
35 | defp elixirc_paths(:test), do: ["lib", "test/support"]
36 | defp elixirc_paths(_), do: ["lib"]
37 |
38 | def package do
39 | [
40 | maintainers: ["Rico Sta. Cruz"],
41 | licenses: ["MIT"],
42 | files: ["lib", "mix.exs", "README.md"],
43 | links: %{github: "https://github.com/rstacruz/expug"}
44 | ]
45 | end
46 |
47 | def docs do
48 | [
49 | source_ref: "v#{@version}",
50 | main: "readme",
51 | extras:
52 | Path.wildcard("*.md") ++
53 | Path.wildcard("docs/**/*.md")
54 | ]
55 | end
56 | end
57 |
--------------------------------------------------------------------------------
/mix.lock:
--------------------------------------------------------------------------------
1 | %{"calliope": {:hex, :calliope, "0.4.0", "cdba8ae42b225de1c906bcb511b1fa3a8dd601bda3b2005743111f7ec92cd809", [:mix], []},
2 | "earmark": {:hex, :earmark, "1.2.3", "206eb2e2ac1a794aa5256f3982de7a76bf4579ff91cb28d0e17ea2c9491e46a4", [:mix], [], "hexpm"},
3 | "ex_doc": {:hex, :ex_doc, "0.18.1", "37c69d2ef62f24928c1f4fdc7c724ea04aecfdf500c4329185f8e3649c915baf", [:mix], [{:earmark, "~> 1.1", [hex: :earmark, repo: "hexpm", optional: false]}], "hexpm"},
4 | "phoenix_html": {:hex, :phoenix_html, "2.5.1", "631053f9e345fecb5c87d9e0ccd807f7266d27e2ee4269817067af425fd81ba8", [:mix], [{:plug, "~> 0.13 or ~> 1.0", [hex: :plug, optional: false]}]},
5 | "plug": {:hex, :plug, "1.1.5", "de5645c18170415a72b18cc3d215c05321ddecac27a15acb923742156e98278b", [:mix], [{:cowboy, "~> 1.0", [hex: :cowboy, optional: true]}]}}
6 |
--------------------------------------------------------------------------------
/test/builder_test.exs:
--------------------------------------------------------------------------------
1 | defmodule BuilderTest do
2 | use ExUnit.Case
3 | doctest Expug.Builder
4 |
5 | def build(source) do
6 | with \
7 | tokens <- Expug.Tokenizer.tokenize(source),
8 | ast <- Expug.Compiler.compile(tokens) do
9 | Expug.Builder.build(ast)
10 | end
11 | end
12 |
13 | test "build" do
14 | eex = build("doctype html\ndiv Hello")
15 | assert eex == %{
16 | :lines => 2,
17 | 1 => [""],
18 | 2 => ["", "Hello", "
"]
19 | }
20 | end
21 |
22 | test "self-closing img" do
23 | eex = build("doctype html\nimg")
24 | assert eex == %{
25 | :lines => 2,
26 | 1 => [""],
27 | 2 => ["