Add despacing to DTL to enable removal of blank lines and floating bits of text in generated HTML

[donparakin]

Code of Conduct

• I agree to follow Django's Code of Conduct

Feature Description

Add two regular expressions to DTL's core loader to enable removing excess whitespace by using delimiters like {#- and -#}. These will remove excessive blank lines and floating text from generated HTML.

Problem

HTML generated by DTL is, um, not beautiful or even okay looking: it often has lots and lots of blank lines and bits of text floating in space.

I know not everyone does a view-source, but I do. I also want to add lots of comments to my DTL templates without adding lots of blank lines to the generated HTML.

Request or proposal

proposal

Additional Details

For a more detailed description, see django-despacer.

I'm a frugal pragmatic perfectionist (with deadlines).

As a perfectionist, I'm horrified by the HTML generated by DTL: so many empty lines and bits of text floating in space. Being pragmatic, I accept that perfect whitespace control that generates perfectly indented HTML is unrealistic; so I'll settle for merely better looking HTML. Being frugal, I won't repeatedly pay a price for it (on every request) but I'm willing to pay just once (at template load time).

I'm pretty sure this won't break existing templates. I don't believe a - immediately after an opening delimiter or before a closing delimiter is currently valid. So, backward compatibility should be preserved.

Implementation Suggestions

For demonstrating and testing the idea, I have implemented it as a package on PyPI: django-despacer. It works on my machine 😄.

Rather than it remaining a package, I'm hoping it can be included in core so it's awkward setup (requiring configuration of "loaders") is not necessary.

[github-actions]

Thank you donparakin for sharing your idea! We have a lot of them so please be patient. You can see the current queue here.

Community instructions

For commenters, please use the emoji reactions on the issue to express support, and/or concern easily. Please use the comments to ask questions or contribute knowledge about the idea. It is unhelpful to post comments of "I'd love this" or "What's the state of this?"

Reaction Guide

• 👍 This is something I would use
• 👎 This is something that would cause problems for me or Django
• 😕 I’m indifferent to this
• 🎉 This is an easy win

[LilyFirefly]

Interesting idea! There's also prior art for this in Jinja2, which Django also supports.

I think within Django we could integrate this more naturally, which would mean we could leave the behaviour on in debug mode.

[donparakin]

Interesting idea! There's also prior art for this in Jinja2.

Thanks. Yes, there's definitely prior art also including Golang templates and PHP's Twig. I definitely cannot claim credit for the idea. I've done a lot of Jinja2, PHP, and Golang templates over the years so I cringe whenever I do DTL. I was surprised how easy it was to add to a DTL loader.

[donparakin]

FYI, I upgraded django-despacer to 0.1.3. Nothing about the despacing feature changed. Just how I shoehorn it into the template loading process. If it's in Django core, no loaders need changing.

While I'm here ;), for those who haven't read more at django-despacer, here's what the HTML looks like with & without despacing:

<div>
  {#- I can now add lots of comments to my template -#}
  {#- ... without adding a ton of empty lines! -#}
  {#- the {%- on the `if` below removes empty lines -#}
  {#- I added for readability -#}

  {%- if not user.is_authenticated %}
    Howdy, stranger!
  {%- else %}
    Welcome back, {{ user.name }}.
  {%- endif -%}

  <button>
    {#- Despacing before and after collapses <button> down to a one liner -#}
    {%- translate "Save and Publish" -%}
    {#- Isn't it just great to be able to add lots of comments! -#}
  </button>
</div>

Without django-despacer (and without the -), the HTML would look like:

<div>
  
  




    Howdy, stranger!
  


  <button>
    
    Save and Publish
    
  </button>
</div>

... but with django-despacer:

<div>
    Howdy, stranger!
  <button>Save and Publish</button>
</div>

[medmunds]

Should despacing be ignored in a {% verbatim %} block, like other Django template syntax? (I think it probably should, but that would complicate the implementation a bit.)

        Incidentally,

               one 

other good use for despacing flags
   is in       plain-text     templates
like are often used for rendering

        email  

. (E.g., Django's own
   password_reset_email.html
)

[donparakin]

Should despacing be ignored in a {% verbatim %} block, ...

Good point. While on vacation, I gave this some thought. Despacing is ignored inside a {% verbatim %} block if there are not any despacing delimiters ({%-, -%}, {{-, -}}, {#-, -#}) within the block (of course). I believe that {% verbatim %} blocks are rare and that ones with a despacing delimiter inside would be extremely rare.

Handling this extremely rare case would add non-trivial complexity and processing. Adding it would punish the 99.999999999% of DTL templates that don't have this rarity. I considered writing simpler code to just detect this rare case and, if found, skip all despacing. But the detection code would still punish the 99.999999999%, although less.

I'm now thinking the way handle this is not with more code. Despacing is a cosmetic nicety, not a necessity. Instead of more code, the documentation would tell the template developer to add {# nodespacing #} if she/he have this extremely rare case. Okay, okay, a trivial amount of more code would be needed to detect {# nodespacing #}.

FYI, I've upgraded django-despacer to 0.1.4 to include {# nodespacing #}.

[medmunds]

I believe that {% verbatim %} blocks are rare and that ones with a despacing delimiter inside would be extremely rare.

I think you're probably right about that. Fwiw, in some older projects I've used {% verbatim %} to embed template language blocks rendered client side (e.g., Handlebars) in my Django templates. I think wanting whitespace control would be as likely in that as it would be in DTL itself, for all the same reasons. Of course, this is really only a problem with client template languages that share DTL's syntax, namely, Nunjucks. Agreed that the added complexity to ignore despacing within {% verbatim %} doesn't seem worth it.

Despacing is a cosmetic nicety, not a necessity.

I would push back on that. There are several cases where whitespace is significant. Two examples I've run into in real-world projects:

• In HTML, composing textual lists, where punctuation like commas needs to hug the text item. E.g: {% if forloop.last and forloop.length > 1 %}and {% endif %}{{ item }}{% if (forloop.length > 2 or forloop.length > 1 and USE_OXFORD_COMMA) and not forloop.last %}, {% endif %}. (There are several other cases in HTML, some solvable with CSS, but many not.)
• Templates for plain-text email, where random line breaks, indents, and excessive spacing make the result appear broken. (Somewhat facetious example in my earlier comment, but based on a true story.)

I currently solve these with some combination of {% spaceless %} and lengthy, hard-to-read source lines. Despacing would be much better.

[medmunds]

I just noticed that the current django-despacer implementation behaves differently in DEBUG mode:

When DEBUG is True (dev), when a DTL template is loaded: [whitespace] is not removed; only the -. Why? The line numbers of the original DTL source code must match the line numbers in the compiler's input to have correct line numbers in any error messages. That's important in development.

That seems like a problem, because despacing is not merely cosmetic (see previous comment). I think users would find it confusing if whitespace removal is turned off in DEBUG.

Rather than a regex preprocessor, despacing would probably need to be implemented in DTL's lexer, where it can be applied consistently while preserving source line numbers for error messages. (That's essentially how Jinja handles it.)

[jacobtylerwalls]

If we add this feature, do we need to update the design philosophies document?

The template system shouldn’t do magic things with whitespace. If a template includes whitespace, the system should treat the whitespace as it treats text – just display it. Any whitespace that’s not in a template tag should be displayed.

[medmunds]

If we add this feature, do we need to update the design philosophies document?

The template system shouldn’t do magic things with whitespace. If a template includes whitespace, the system should treat the whitespace as it treats text – just display it. Any whitespace that’s not in a template tag should be displayed.

As someone who's spent a lot of time editing plain-text templates, I guess I'd argue that philosophy is already at odds with this one earlier in the list: "The template system shouldn’t be designed so that it only outputs HTML. It should be equally good at generating other text-based formats, or just plain text." [emphasis mine] (Also, the philosphy probably needs a footnote for {% spaceless %} 😜)

Any sufficiently documented syntax is distinguishable from magic. So I'm +1 on both despacing and updating the philosophy.

[donparakin]

... There are several cases where whitespace is significant. ...

Notice that despacing only occurs where you explicitly want it to occur. In places where whitespace is significant, do not use an explicit despacing delimiter ({{-, -}}, {%-, -%}, {#-, -#}).

[donparakin]

I think users would find it confusing if whitespace removal is turned off in DEBUG.

On the other hand, I think users would find it extremely confusing if the line numbers in template errors do not match the line numbers of the template source code. Correct line numbers is especially important during DEBUG (development). There are several other documented things that are turned off in DEBUG. We'll just have to document no despacing too.