├── .gitignore ├── README.rst ├── django_pancake ├── __init__.py ├── __init__.pyc ├── flatten.py └── make_pancakes.py ├── setup.py └── tests ├── __init__.py ├── __init__.pyc └── test_pancake.py /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | *.pyo 3 | build -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | ============== 2 | django-pancake 3 | ============== 4 | 5 | By Adrian Holovaty 6 | 7 | Library for "flattening" Django templates. 8 | 9 | Run ``make_pancakes.py`` with an output directory name, and it will fill that 10 | directory with a "flat" version of each template -- a pancake. A pancake is a 11 | template in which: 12 | 13 | * Template inheritance is fully expanded. ``{% extends %}`` and ``{% block %}`` 14 | tags are gone, and the parent templates are fully folded in. 15 | 16 | * ``{% include %}`` tags are gone, with their contents fully folded in. 17 | 18 | * Template comments are removed: ``{% comment %}`` syntax and 19 | ``{# short syntax #}``. 20 | 21 | Pancakes behave exactly the same as the original templates. But they might 22 | render more quickly, because they avoid the overhead of template inheritance 23 | and template includes at rendering time. (Whether they do or don't actually 24 | render more quickly depends on other factors such as whether you're using 25 | template caching.) 26 | 27 | Think of it as "denormalizing" your templates, as a database administrator 28 | might denormalize SQL tables for performance. You give up the DRY principle 29 | but make up for it in faster application speed. 30 | 31 | You can also use pancakes as a learning tool. You can examine a pancake to 32 | see how all of your template blocks fit together in the resulting "compiled" 33 | template. 34 | 35 | Pros and cons 36 | ============= 37 | 38 | Pros: 39 | 40 | * Might make run-time template rendering faster. In some cases, the rendering 41 | is significantly faster. See "How fast is the speed improvement?" below. 42 | 43 | * Can help you understand how a complex template inheritance structure works. 44 | You can examine a pancake to see how all of the template blocks fit together. 45 | 46 | * Not a huge commitment. Give it a shot and see whether it makes things faster 47 | for you. 48 | 49 | Cons: 50 | 51 | * Might not actually result in a significant speed increase. 52 | 53 | * Makes your deployment more complex, as you now have to manage generated 54 | templates and run django-pancake to generate them whenever you deploy. 55 | 56 | * May require you to change the way you write templates, specifically by 57 | removing dynamic ``{% include %}`` and ``{% extends %}`` tags. See 58 | "Limitations" below. 59 | 60 | * (Philosophical.) Django really should do this in memory rather than compiling 61 | to templates on the filesystem. See "Related projects" below. 62 | 63 | How fast is the speed improvement? 64 | ================================== 65 | 66 | It depends on what you're doing in templates, and it depends on whether you 67 | have template caching enabled. 68 | 69 | If you're just using basic template inheritance and includes, you should expect 70 | a tiny/negligible performance improvement -- on the order of 10 milliseconds 71 | for a single template render. In this case, it may not be worth the added 72 | complexity it introduces to your deployment environment. 73 | 74 | But if you're doing crazy things -- say, having a template with a template tag 75 | that loads other templates, within a loop, with each subtemplate being in an 76 | inheritance structure three levels deep -- then you might see a significant 77 | benefit that makes it worth the complexity. 78 | 79 | At EveryBlock, I found it sped a certain type of page up by 200 milliseconds, 80 | which is pretty great. But that was on my laptop, and unfortunately, the speed 81 | increase went away when the code was deployed onto the production servers 82 | (which were running the cached template loader). So django-pancake basically 83 | had no positive effect for us, and I disabled it. But, who knows, it might help 84 | somebody else. 85 | 86 | Usage 87 | ===== 88 | 89 | 1. Generate the pancakes. Pass it the directory that contains your source 90 | templates and the directory you want pancakes to be generated in:: 91 | 92 | python make_pancakes.py /path/to/source/directory /path/to/pancake/directory 93 | 94 | 2. Point Django at the pancake directory:: 95 | 96 | TEMPLATE_DIRS = [ 97 | '/path/to/pancake/directory', 98 | ] 99 | 100 | Limitations 101 | =========== 102 | 103 | If you want django-pancake to work with your templates, make sure your 104 | templates do the following: 105 | 106 | * Avoid using ``block.super`` in anything but a standalone variable. This is 107 | OK:: 108 | 109 | {{ block.super }} 110 | 111 | But these statements are not:: 112 | 113 | {% if block.super %} 114 | {{ block.super|lower }} 115 | {% some_other_tag block.super %} 116 | 117 | If you use ``block.super`` in one of these prohibited ways, django-pancake 118 | will not detect it and will generate your templates as if everything is OK. 119 | But you'll likely get odd behavior when the template is rendered. The problem 120 | is that ``block`` is no longer a variable in the pancakes, so it'll be 121 | evaluated as ``False``. 122 | 123 | * Avoid dynamic ``{% extends %}`` tags -- that is, when the parent template 124 | name is a variable. Example: ``{% extends my_template_name %}`` (note the 125 | lack of quotes around ``my_template_name``). If you do this, django-pancake 126 | will raise a ``PancakeFail`` exception. 127 | 128 | * Likewise, avoid dynamic ``{% include %}`` tags. Example: 129 | ``{% include some_include %}``. If you do this, django-pancake will raise a 130 | ``PancakeFail`` exception. 131 | 132 | * Don't use the "only" keyword in ``{% include %}`` tags. If you do, 133 | django-pancake won't raise an exception, but it'll merely output the same 134 | ``{% include %}`` tag, so you don't get the benefit of flattening. 135 | 136 | Other notes 137 | =========== 138 | 139 | Obviously, pancakes are very redundant -- each of them includes all of the 140 | markup from the base template(s), etc. You shouldn't check pancakes into 141 | revision control or hand-edit them. They're purely for performance and should 142 | be handled as any automatically generated code. 143 | 144 | Related projects 145 | ================ 146 | 147 | Ideally, django-pancake wouldn't exist, and Django would do this "flattening" 148 | itself, along with providing a more robust template compilation step. That's a 149 | significantly harder problem, and we're working on it. The goal is for 150 | django-pancake not to have to exist. 151 | 152 | But, in the meantime, here are some related projects that have gone down that 153 | road: 154 | 155 | * templatetk (https://github.com/mitsuhiko/templatetk/) 156 | 157 | * django-template-preprocessor (https://github.com/citylive/django-template-preprocessor/) 158 | -------------------------------------------------------------------------------- /django_pancake/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/adrianholovaty/django-pancake/4fa757f5818d50292084e7f7eea5e3e395a5b21d/django_pancake/__init__.py -------------------------------------------------------------------------------- /django_pancake/__init__.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/adrianholovaty/django-pancake/4fa757f5818d50292084e7f7eea5e3e395a5b21d/django_pancake/__init__.pyc -------------------------------------------------------------------------------- /django_pancake/flatten.py: -------------------------------------------------------------------------------- 1 | from django.template.base import Lexer, TOKEN_BLOCK, TOKEN_TEXT, TOKEN_VAR 2 | import os 3 | import re 4 | 5 | class PancakeFail(Exception): 6 | pass 7 | 8 | class ASTNode(object): 9 | "A node in the AST." 10 | def __init__(self, name): 11 | self.name = name 12 | self.leaves = [] # Each leaf can be a string or another ASTNode. 13 | 14 | def __repr__(self): 15 | return '<%s %s>' % (self.__class__.__name__, self.name) 16 | 17 | def sub_text(self): 18 | for leaf in self.leaves: 19 | if isinstance(leaf, ASTNode): 20 | for subleaf in leaf.sub_text(): 21 | yield subleaf 22 | else: 23 | yield leaf 24 | 25 | def sub_nodes(self): 26 | for leaf in self.leaves: 27 | if isinstance(leaf, ASTNode): 28 | yield leaf 29 | for subleaf in leaf.sub_nodes(): 30 | yield subleaf 31 | 32 | class Template(ASTNode): 33 | "Root node of the AST. Represents a template, which may or may not have a parent." 34 | def __init__(self, name): 35 | super(Template, self).__init__(name) 36 | self.parent = None # Template object for the parent template, if there's a parent. 37 | self.blocks = {} # Maps block names to objects in self.leaves for quick lookup. 38 | self.loads = set() # Template libraries to load. 39 | 40 | class Block(ASTNode): 41 | "Represents a {% block %}." 42 | pass 43 | 44 | class TemplateDirectory(object): 45 | "Dictionary-like object that maps template names to template strings." 46 | def __init__(self, directory): 47 | self.directory = directory 48 | 49 | def __getitem__(self, template_name): 50 | filename = os.path.join(self.directory, template_name) 51 | return open(filename).read() 52 | 53 | class Parser(object): 54 | def __init__(self, fail_gracefully=True): 55 | self.fail_gracefully = fail_gracefully 56 | 57 | def parse(self, template_name, templates): 58 | """ 59 | Creates an AST for the given template. Returns a Template object. 60 | """ 61 | self.templates = templates # Maps template names to template sources. 62 | self.root = Template(template_name) 63 | self.stack = [self.root] 64 | self.current = self.root 65 | self.tokens = Lexer(self.templates[template_name], 'django-pancake').tokenize() 66 | _TOKEN_TEXT, _TOKEN_VAR, _TOKEN_BLOCK = TOKEN_TEXT, TOKEN_VAR, TOKEN_BLOCK 67 | while self.tokens: 68 | token = self.next_token() 69 | 70 | if token.token_type == _TOKEN_TEXT: 71 | self.current.leaves.append(token.contents) 72 | 73 | elif token.token_type == _TOKEN_VAR: 74 | if token.contents == 'block.super': 75 | if self.root.parent is None: 76 | raise PancakeFail('Got {{ block.super }} in a template that has no parent') 77 | 78 | super_block_name = self.stack[-1].name 79 | current_par = self.root.parent 80 | while current_par is not None: 81 | if super_block_name in current_par.blocks: 82 | self.current.leaves.extend(current_par.blocks[super_block_name].leaves) 83 | break 84 | current_par = current_par.parent 85 | else: 86 | self.current.leaves.append('{{ %s }}' % token.contents) 87 | 88 | elif token.token_type == _TOKEN_BLOCK: 89 | try: 90 | tag_name, arg = token.contents.split(None, 1) 91 | except ValueError: 92 | tag_name, arg = token.contents.strip(), None 93 | method_name = 'do_%s' % tag_name 94 | if hasattr(self, method_name): 95 | getattr(self, method_name)(arg) 96 | else: 97 | self.current.leaves.append('{%% %s %%}' % token.contents) 98 | 99 | return self.root 100 | 101 | def next_token(self): 102 | return self.tokens.pop(0) 103 | 104 | def do_block(self, text): 105 | if not text: 106 | raise PancakeFail('{% block %} without a name') 107 | self.current.leaves.append(Block(text)) 108 | self.root.blocks[text] = self.current = self.current.leaves[-1] 109 | self.stack.append(self.current) 110 | 111 | def do_endblock(self, text): 112 | self.stack.pop() 113 | self.current = self.stack[-1] 114 | 115 | def do_extends(self, text): 116 | if not text: 117 | raise PancakeFail('{%% extends %%} without an argument (file: %r)' % self.root.name) 118 | if text[0] in ('"', "'"): 119 | parent_name = text[1:-1] 120 | self.root.parent = Parser().parse(parent_name, self.templates) 121 | else: 122 | raise PancakeFail('Variable {%% extends %%} tags are not supported (file: %r)' % self.root.name) 123 | 124 | def do_comment(self, text): 125 | # Consume all tokens until 'endcomment' 126 | while self.tokens: 127 | token = self.next_token() 128 | if token.token_type == TOKEN_BLOCK: 129 | try: 130 | tag_name, arg = token.contents.split(None, 1) 131 | except ValueError: 132 | tag_name, arg = token.contents.strip(), None 133 | if tag_name == 'endcomment': 134 | break 135 | 136 | def do_load(self, text): 137 | # Keep track of which template libraries have been loaded, 138 | # so that we can pass them up to the root. 139 | self.root.loads.update(text.split()) 140 | 141 | def do_include(self, text): 142 | if ' only' in text: 143 | if self.fail_gracefully: 144 | self.current.leaves.append('{%% include %s %%}' % text) 145 | return 146 | else: 147 | raise PancakeFail('{%% include %%} tags containing "only" are not supported (file: %r)' % self.root.name) 148 | try: 149 | template_name, rest = text.split(None, 1) 150 | except ValueError: 151 | template_name, rest = text, '' 152 | if not template_name[0] in ('"', "'"): 153 | if self.fail_gracefully: 154 | self.current.leaves.append('{%% include %s %%}' % text) 155 | return 156 | else: 157 | raise PancakeFail('Variable {%% include %%} tags are not supported (file: %r)' % self.root.name) 158 | template_name = template_name[1:-1] 159 | if rest.startswith('with '): 160 | rest = rest[5:] 161 | 162 | include_node = Parser().parse(template_name, self.templates) 163 | 164 | # Add {% load %} tags from the included template. 165 | self.root.loads.update(include_node.loads) 166 | 167 | if rest: 168 | self.current.leaves.append('{%% with %s %%}' % rest) 169 | self.current.leaves.extend(include_node.leaves) 170 | if rest: 171 | self.current.leaves.append('{% endwith %}') 172 | 173 | def flatten_ast(template): 174 | "Given an AST as returned by the parser, returns a string of flattened template text." 175 | # First, make a list from the template inheritance structure -- the family. 176 | # This will be in order from broad to specific. 177 | family = [] 178 | while 1: 179 | family.insert(0, template) 180 | if template.parent is None: 181 | break 182 | template = template.parent 183 | 184 | # Now, starting with the base template, loop downward over the child 185 | # templates (getting more specific). For each child template, fill in the 186 | # blocks in the parent template. 187 | master = family[0] 188 | for child in family[1:]: 189 | for block in child.sub_nodes(): 190 | if block.name in master.blocks: 191 | master.blocks[block.name].leaves = block.leaves 192 | # For all blocks that this NEW template defined, update 193 | # master.blocks so that any subsequent children can access and 194 | # override the right thing. 195 | for child_leaf in block.sub_nodes(): 196 | master.blocks[child_leaf.name] = child_leaf 197 | master.loads.update(child.loads) 198 | 199 | # Add the {% load %} statements from all children. 200 | # Put them in alphabetical order to be consistent. 201 | if master.loads: 202 | loads = sorted(master.loads) 203 | master.leaves.insert(0, '{%% load %s %%}' % ' '.join(loads)) 204 | 205 | return master 206 | 207 | def flatten(template_name, templates): 208 | p = Parser() 209 | template = p.parse(template_name, templates) 210 | flat = flatten_ast(template) 211 | return ''.join(flat.sub_text()) 212 | 213 | if __name__ == "__main__": 214 | import sys 215 | print flatten(sys.argv[1], TemplateDirectory(sys.argv[2])) 216 | -------------------------------------------------------------------------------- /django_pancake/make_pancakes.py: -------------------------------------------------------------------------------- 1 | from flatten import flatten, TemplateDirectory 2 | import os 3 | 4 | def template_names(input_dir, prefix=''): 5 | for filename in os.listdir(input_dir): 6 | template_name = os.path.join(prefix, filename) 7 | full_name = os.path.join(input_dir, filename) 8 | if os.path.isdir(full_name): 9 | for name in template_names(full_name, template_name): 10 | yield name 11 | else: 12 | yield template_name 13 | 14 | def make_pancakes(input_dir, output_dir): 15 | templates = TemplateDirectory(input_dir) 16 | for template_name in template_names(input_dir): 17 | print "Writing %s" % template_name 18 | pancake = flatten(template_name, templates) 19 | outfile = os.path.join(output_dir, template_name) 20 | try: 21 | os.makedirs(os.path.dirname(outfile)) 22 | except OSError: # Already exists. 23 | pass 24 | with open(outfile, 'w') as fp: 25 | fp.write(pancake) 26 | 27 | if __name__ == "__main__": 28 | import sys 29 | make_pancakes(sys.argv[1], sys.argv[2]) 30 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | from distutils.core import setup 2 | 3 | setup( 4 | name='django-pancake', 5 | version='0.1', 6 | description='Library for "flattening" Django templates.', 7 | author='Adrian Holovaty', 8 | author_email='adrian@holovaty.com', 9 | url='https://github.com/adrianholovaty/django-pancake', 10 | license='MIT', 11 | classifiers = [ 12 | 'Framework :: Django', 13 | 'Intended Audience :: Developers', 14 | 'License :: OSI Approved :: MIT License', 15 | 'Programming Language :: Python', 16 | 'Topic :: Internet :: WWW/HTTP', 17 | ], 18 | packages=['django_pancake'], 19 | ) 20 | -------------------------------------------------------------------------------- /tests/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/adrianholovaty/django-pancake/4fa757f5818d50292084e7f7eea5e3e395a5b21d/tests/__init__.py -------------------------------------------------------------------------------- /tests/__init__.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/adrianholovaty/django-pancake/4fa757f5818d50292084e7f7eea5e3e395a5b21d/tests/__init__.pyc -------------------------------------------------------------------------------- /tests/test_pancake.py: -------------------------------------------------------------------------------- 1 | from django_pancake.flatten import flatten 2 | 3 | # template_name: (template_source, pancake_source), 4 | TESTS = { 5 | # No inheritance. 6 | 'test1': ('Some text here', 'Some text here'), 7 | 'test2': ('{% unknown_block_tag %}Text here', '{% unknown_block_tag %}Text here'), 8 | 9 | # Basic inheritance. 10 | 'base1': ('{% block title %}{% endblock %}', ''), 11 | 'child1': ('{% extends "base1" %} {% block title %}It worked{% endblock %}', 'It worked'), 12 | 13 | # Empty child. 14 | 'emptychild1': ('{% extends "base1" %}', ''), 15 | 16 | # Junk in child templates. 17 | 'junk1': ('{% extends "base1" %} Junk goes here', ''), 18 | 'junk2': ('{% extends "base1" %} Junk goes here {% some_tag %}', ''), 19 | 'junk3': ('{% extends "base1" %} Junk goes here {% some_tag %} {% block title %}Worked{% endblock %}', 'Worked'), 20 | 21 | # Inheritance with default values in blocks. 22 | 'defaultbase1': ('{% block title %}Default{% endblock %}', 'Default'), 23 | 'default1': ('{% extends "defaultbase1" %}', 'Default'), 24 | 'default2': ('{% extends "defaultbase1" %}{% block title %}No default{% endblock %}', 'No default'), 25 | 26 | # Blocks within blocks. 27 | 'withinbase1': ('{% block fulltitle %}{% block title %}Welcome{% endblock %} | Example.com{% endblock %}', 'Welcome | Example.com'), 28 | 'within1': ('{% extends "withinbase1" %} {% block fulltitle %}Yay{% endblock %}', 'Yay'), 29 | 'within2': ('{% extends "withinbase1" %} {% block title %}Some page{% endblock %}', 'Some page | Example.com'), 30 | 31 | # Blocks within blocks, overriding both blocks. 32 | 'bothblocks1': ('{% extends "withinbase1" %}{% block fulltitle %}Outer{% endblock %}{% block title %}Inner{% endblock %}', 'Outer'), 33 | 'bothblocks2': ('{% extends "withinbase1" %}{% block title %}Inner{% endblock %}{% block fulltitle %}Outer{% endblock %}', 'Outer'), 34 | 35 | # Three-level inheritance structure. 36 | '3levelbase1': ('{% block content %}Welcome!
{% block header %}{% endblock %}{% endblock %}', 'Welcome!
'), 37 | '3levelbase2': ('{% extends "3levelbase1" %}{% block header %}

{% block h1 %}{% endblock %}

{% endblock %}', 'Welcome!

'), 38 | '3level1': ('{% extends "3levelbase2" %}{% block h1 %}Title goes here{% endblock %}', 'Welcome!

Title goes here

'), 39 | 40 | # Wacky bug: Four-level inheritance structure. 41 | 'wackylevel1': ('{% block content %}{% endblock %}', ''), 42 | 'wackylevel2': ('{% extends "wackylevel1" %} {% block content %}
{% block canvas %}{% endblock %}
{% block rail %}{% endblock %}
{% endblock %}', 43 | '
'), 44 | 'wackylevel3': ('{% extends "wackylevel2" %} {% block content %}
{% block rail %}{% endblock %}
{% block canvas %}{% endblock %}
{% endblock %}', 45 | '
'), 46 | 'wackylevel4': ('{% extends "wackylevel3" %}{% block rail %}Rail{% endblock %}{% block canvas %}Canvas{% endblock %}', 47 | '
Rail
Canvas
'), 48 | 49 | # Inheritance, skipping a level. 50 | 'skiplevel1': ('{% block header %}{% block h1 %}{% endblock %}

Header

{% endblock %}', '

Header

'), 51 | 'skiplevel2': ('{% extends "skiplevel1" %}', '

Header

'), 52 | 'skiplevel3': ('{% extends "skiplevel2" %}{% block h1 %}

Title

{% endblock %}', '

Title

Header

'), 53 | 54 | # {% load %} statements get bubbled up for inheritance. 55 | 'loadbase1': ('{% load webdesign %}{% block title %}{% endblock %}', '{% load webdesign %}'), 56 | 'load1': ('{% extends "base1" %}{% load humanize %}{% block title %}Load 1{% endblock %}', '{% load humanize %}Load 1'), 57 | 'load2': ('{% extends "loadbase1" %}{% load humanize %}{% block title %}Load 2{% endblock %}', '{% load humanize webdesign %}Load 2'), 58 | 59 | # {% load %} statements get bubbled up for includes. 60 | 'loadinclude1': ('{% load webdesign %}Hello', '{% load webdesign %}Hello'), 61 | 'load3': ('{% load foo %}{% include "loadinclude1" %} there', '{% load foo webdesign %}Hello there'), 62 | 63 | # block.super. 64 | 'super1': ('{% extends "withinbase1" %}{% block title %}{{ block.super }} to the site{% endblock %}', 'Welcome to the site | Example.com'), 65 | 'super2': ('{% extends "withinbase1" %}{% block title %}{{ block.super }} {{ block.super }} {{ block.super }}{% endblock %}', 'Welcome Welcome Welcome | Example.com'), 66 | 67 | # block.super, skipping a level. 68 | 'superskip1': ('{% block header %}{% block h1 %}{% endblock %}

Header

{% endblock %}', '

Header

'), 69 | 'superskip2': ('{% extends "superskip1" %}', '

Header

'), 70 | 'superskip3': ('{% extends "superskip2" %}{% block header %}Here: {{ block.super }}{% endblock %}', 'Here:

Header

'), 71 | 72 | # Include tag. 73 | 'include1': ('{% include "defaultbase1" %}', 'Default'), 74 | 'include2': ('{% include "defaultbase1" with foo=bar %}', '{% with foo=bar %}Default{% endwith %}'), 75 | 'include3': ('{% include "defaultbase1" with foo=bar baz=3 %}', '{% with foo=bar baz=3 %}Default{% endwith %}'), 76 | 77 | # Include tag with 'only'. 78 | 'includeonly1': ('{% include "defaultbase1" only %}', '{% include "defaultbase1" only %}'), 79 | 80 | # Include tag with variable argument. 81 | 'includevariable1': ('{% include some_template %}', '{% include some_template %}'), 82 | 83 | # Remove template comments. 84 | 'comments1': ('lo{% comment %}Long-style comment{% endcomment %}ve', 'love'), 85 | 'comments2': ('lo{# Short-style comment #}ve', 'love'), 86 | 'comments3': ('lo{% comment %}{% if foo %}foo{% else %}bar{% endif %}{# Inner comment #}Some other stuff{% endcomment %}ve', 'love'), 87 | } 88 | TEMPLATES = dict((k, v[0]) for k, v in TESTS.items()) 89 | 90 | def test_flatten(): 91 | for template_name, (template_source, pancake_source) in TESTS.items(): 92 | yield check_flatten, template_name, pancake_source 93 | 94 | def check_flatten(template_name, expected): 95 | result = flatten(template_name, TEMPLATES) 96 | assert result == expected, 'expected %r, got %r' % (expected, result) 97 | --------------------------------------------------------------------------------