Tangling code from Hugo content with Raku

I could just use Org mode, but noo that’s too easy

You know what else I can tangle? Yarn!
You know what else I can tangle? Yarn!

Let’s say I have a file. The one you’re reading, perhaps. Well, its original Markdown content.

It has a shortcode in it.

{{< code file="hello.py" >}}
print("Hello")
{{< /code >}}

I based {{< code >}} here on a shortcode from the hugo docs. It presents highlighted code with additional context.

hello.py

print("Hello")

Really handy when you’re writing about code. Thing is, now I have two copies. There’s one here in the shortcode, and another in a hello.py file that I’m writing about. I’d prefer there was only a single copy. That way they don’t get out of sync.

I could use Hugo’s readFile function in a new shortcode, including the contents of hello.py in this Markdown file. Something like this:

{{< include file="hello.py" >}}

But that still breaks up the writing flow a little bit. I’m writing the code over here, and writing about it over there. It’s a tiny complaint, but working with Org mode has spoiled me. I get to write the code in the same document that I’m writing about it in. Everything stays in sync, more or less.

What I want is to write about hello.py here, and with a command have hello.py appear on my filesystem, containing the Python code I’ve been describing.

And I want to do it without disturbing Hugo. Let it turn Markdown into HTML.

Tangling

This process is called “tangling,” and it’s popular in the admittedly small world of Literate Programming. The code is interleaved throughout some kind of document, and a tool like noweb or Babel parses the document to create code files. Could be any kind of file, really. The process can get fancy.

But the start is not fancy: given a text file containing a {{< code file="(something)" >}}, write the contents of that shortcode to the named file.

tangle.raku

sub MAIN() {
  my $filename = "index.md";
  my $opener = '{{< ';
  my $closer = ' >}}';
  my regex shortcode {
    $opener
      code \s
      'file="' $<filename> = .+? '"'  # Remember the filename
      .*?
    $closer
    \n                # Ignore leading newline
    $<content> = .+?  # Remember everything else in the block
    \n                # Ignore leading newline
    $opener '/code' $closer
  }

  my $markdown = slurp $filename;

  if $markdown.match(/ <shortcode> /) {
    my $tangle-file = $/<shortcode><filename>;
    my $tangle-content = $/<shortcode><content>;
    spurt $tangle-file, $tangle-content;
    say "Tangled to $tangle-file";
  }
}

I love Raku’s approach to regular expressions. For starters, the syntax looks a bit more like describing a grammar. I can break the funny regex characters up with spaces, and clarify them with comments. In fact, I could someday build this up to a real grammar.

Secondly, it addresses the fact that most text we look at these days contains multiple lines. I didn’t have to worry about any special multiline flags to get this working.

Finally, getting at the named captures was — I wouldn’t say “obvious,” but at least “coherent.” I can treat the match variable $/ as a nested Hash. The important bits look something like this:

shortcode =>
  filename => 「hello.py」
  content => 「print("Hello")」

I can grab the named capture filename of my matched shortcode regex with $/<shortcode><filename> — or ~$<shortcode><filename>, depending on your preferred syntax.

This is all possible in languages like Perl with assorted flags, but I haven’t seen parsing treated so well by default since maybe REBOL.

Anyways, let’s run this thing.

$ raku tangle.raku
Tangled to hello.py
$ bat hello.py
───────┬────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: hello.py
───────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ print("Hello")
───────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────

Sweet.

Except — this Markdown file I’m writing. It has two file code blocks now. I want to tangle both of them.

Multiple output files

This requires a couple changes, since I’m writing code about Hugo shortcodes in a Hugo post.

To show shortcode directives without Hugo evaluating them, they need to look like shortcode comments. Their contents will get passed straight through as part of your post. To show {{< shortcode >}} in a post, your Hugo content needs {{</* shortcode */>}}.

So that’s lovely and all, but can be a headache of its own for this specific situation of extracting code from a blog post.

I need to remember this commented shortcode syntax.

Define commented shortcodes

  my $commented-opener = '{{' ~ '</* ';
  my $commented-closer = ' */>' ~ '}}';

That way I can replace those commented shortcode delimiters with their normal counterparts when I tangle later.

Replace commented shortcodes

      my $tangle-content = $block<shortcode><content>
        .subst(:global, / $commented-opener /, $opener)
        .subst(:global, / $commented-closer /, $closer);

Now that I have that particular detail out of the way, tangle every block? Sure! Make a regular expression match :global and it returns a list containing every match.

Tangle every block

  my $markdown  = slurp $filename;
  my @fragments = $markdown.match(/<shortcode>/, :global);

  for @fragments -> $block {
    my $tangle-file = $block<shortcode><filename>;
    «replace-commented-shortcodes»
    spurt $tangle-file, $tangle-content;
    say "Tangled to $tangle-file";
  }

I think that about covers it. The shortcode recognition logic can stay the same.

tangle-multi.raku

sub MAIN() {
  my $filename = "index.md";
  my $opener = '{{< ';
  my $closer = ' >}}';

  my regex shortcode {
    $opener
      code \h
      'file="' $<filename> = .+? '"'  # Remember the filename
      .*?
    $closer
    \n                # Ignore leading newline
    $<content> = .+?  # Remember everything else in the block
    \n                # Ignore trailing newline
    $opener '/code' $closer
  }

  «define-commented-shortcodes»

  «tangle-every-block»
}

And it works!

$ raku tangle-multi.raku
Tangled to hello.py
Tangled to tangle.raku
$ bat tangle.raku
───────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: tangle.raku
───────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ sub MAIN() {
   2   │   my $filename = "index.md";
   3   │   my $opener = '{{< ';
   4   │   my $closer = ' >}}';
   5   │   my regex shortcode {
   6   │     $opener
   7   │       code \s
   8   │       'file="' $<filename> = .+? '"'  # Remember the filename
   9   │       .*?
  10   │     $closer
  11   │     \n                # Ignore leading newline
  12   │     $<content> = .+?  # Remember everything else in the block
  13   │     \n                # Ignore leading newline
  14   │     $opener '/code' $closer
  15   │   }
  16   │
  17   │   my $markdown = slurp $filename;
  18   │
  19   │   if $markdown.match(/ <shortcode> /) {
  20   │     my $tangle-file = $/<shortcode><filename>;
  21   │     my $tangle-content = $/<shortcode><content>;
  22   │     spurt $tangle-file, $tangle-content;
  23   │     say "Tangled to $tangle-file";
  24   │   }
  25   │ }
───────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────

Unfortunately, I’m not quite done yet.

Multiple fragments

I’m not done yet because I don’t like to describe my code a full file at a time. I’d rather talk about this bit here, explain that bit over there, then mash it all up in the end.

Consistency counts, so I need to pick a syntax. Well — you’ve been reading along. You can see that I already made my choice. I got used to <<fragment-text>> in Babel, where the attribute is called name. Might as well keep doing that over here. Oh but hang on. I want it to stand out a bit. I’ll use angle quotes «‥».

Note

On a US keyboard using Vim or Neovim, « is a digraph which can be entered via Control-k followed by < <. Or if you’ve set up a Compose key, it’s Compose followed by < < in any editor.

» is the same, but > > instead.

Or you can use <<…>> in your code and ignore my recent obsession with fancy characters.

Yes, I know I could practically write it all with fancy characters in Raku. One step at a time.

Let’s go back to the Python code because it’s still so small.

Say I want to demonstrate the delightful Rich terminal library for Python.

Import libraries

from rich import print
from rich.panel import Panel
from rich.markdown import Markdown

But before I really use it in my code, I spend 1,500 words singing its praises.

It’s nice. I like it.

Okay, done singing. Time to write the rest of the program.

rich-hello.py

«import-libraries»

md = Markdown("**Hello**, *World*.")
print(Panel(md))

I identify the fragment with a name attribute:

{{< code name="import-libraries" lang="python" >}}
from rich import print
from rich.panel import Panel
from rich.markdown import Markdown
{{< /code >}}

My code block references the import-libraries fragment by name when I’m ready for it.

{{< code file="rich-hello.py" >}}
«import-libraries»

md = Markdown("**Hello**, *World*.")
print(Panel(md))
{{< /code >}}

Rounding up fragments to tangle

Recognizing an additional parameter doesn’t make my regular expression that much more complicated, but I can see things getting more complex — or me finding a better pattern later — so let’s give the params their own named regex for some encapsulation.

Shortcode params regex

  my regex params {
      'file="' $<filename> = .+? '"'
      ||
      'name="' $<fragment> = .+? '"'
  }

That way I can drop it in shortcode to say “oh and look for params while you’re at it please.”

Nested shortcode regex

  «shortcode-params-regex»
  my $opener = '{{< ';
  my $closer = ' >}}';

  my regex shortcode {
    $opener code \s <params> .*? $closer
    \n                # Ignore leading newline
    $<content> = .+?  # Remember everything else in the block
    \n                # Ignore trailing newline
    $opener '/code' $closer
  }

Okay, we recognize file and name parameters. What do we do with them? We gather them!

Gather fragments and files

    my %fragment-for;
    my @filenames;
    my $markdown = slurp $filename;

    for $markdown.match(/<shortcode>/, :global) -> $block {
      my $tangle-content = $block<shortcode><content>;
      my $params = $block<shortcode><params>;
      my $fragment = $params<fragment> || $params<filename>;

      if $fragment {
        say "fragment: $fragment";
        %fragment-for{ $fragment.Str } = $tangle-content;
      }

      if my $filename = $params<filename> {
        @filenames.push($filename.Str);
      }
    }

Tangling my fragments

Let’s see here. I know before I can write any files, I need to make sure everything’s tangled. Trying to keep fragments easy to identify. They sit on a line by themselves, possibly with some leading whitespace.

Tangle fragments

    my regex fragment { ^^ \h*? "«" $<name> = .+? "»" $$ }
    my %tangle-for;

    «tangle-function»

    for %fragment-for.keys -> $name { tangle($name); }

Note

Raku functions are lexically scoped, which means it’s perfectly okay to declare a function inside another function. Though next time I revisit this, I may want to think about a class or something to hold the complexity.

But what does that function need to look like? I’m still not sure I got it quite right. I mean I know the basic shape of it.

Tangle function

    sub tangle(Str $name) {
      «tangle-error-checking»

      «tangle-text»
    }

It needs some error checking. I know that much. Oh, and if it’s already been tangled I should avoid going through it again.

Tangle error checking

      return "" unless $name;

      if %tangle-for{ $name } {
        return %tangle-for{ $name }.Str;
      }

      my $content = %fragment-for{ $name };
      unless $content {
        die "«$name» is not a valid fragment";
      }

The idea of the thing is clear enough. Find and recursively tangle each fragment found in this text, replacing the fragment references with their tangled text. Once that’s all done, cache and return the tangled text.

Tangle text

      for $content.match(/ <fragment> /, :global) -> $match {
        my $fragment-ref = $match.Str;
        my $fragment-name = $match<fragment><name>.Str;
        say "$name ← «$fragment-name»";
        $content.subst-mutate(/$fragment-ref/, tangle( $fragment-name));
      }

      %tangle-for{ $name } = $content;

I flailed while tangling fragments. Lots of complaints from Raku about the difference between a Match and a String. There must be better ways. But the most important thing? I got it to work eventually.

Writing tangled files

After all that, writing the tangled files felt easy.

Write tangled files

    «define-commented-shortcodes»

    for @filenames -> $tangle-file {
      my $tangle-content = %tangle-for{ $tangle-file }
        .subst(:global, / $commented-opener /, $opener)
        .subst(:global, / $commented-closer /, $closer);
      spurt $tangle-file, $tangle-content;
      say "Tangled to $tangle-file";
    }

Then — theoretically — all these fragments I wrote will make a useful code tangler!

Might as well make it so this script can look at more than just the file I’m editing right now.

tangle-fragments.raku

sub MAIN(Str $filename) {
  «nested-shortcode-regex»

  «gather-fragments-and-files»

  «tangle-fragments»

  «write-tangled-files»
}

Easiest CLI I ever wrote, by the way. See?

$ raku tangle-fragments.raku
Usage:
  tangle-fragments.raku <filename>

Time for the real thing. I’m nervous. I shouldn’t be nervous. I know how this story ends. Then again I keep rewriting the middle.

$ raku tangle-fragments.raku index.md
fragment: hello.py
fragment: tangle.raku
fragment: define-commented-shortcodes
fragment: replace-commented-shortcodes
fragment: tangle-every-block
fragment: tangle-multi.raku
fragment: import-libraries
fragment: rich-hello.py
fragment: shortcode-params-regex
fragment: nested-shortcode-regex
fragment: gather-fragments-and-files
fragment: tangle-fragments
fragment: tangle-function
fragment: tangle-error-checking
fragment: tangle-text
fragment: write-tangled-files
fragment: tangle-fragments.raku
tangle-function <-- (tangle-error-checking)
tangle-function <-- (tangle-text)
nested-shortcode-regex <-- (shortcode-params-regex)
tangle-every-block <-- (replace-commented-shortcodes)
tangle-fragments <-- (tangle-function)
write-tangled-files <-- (define-commented-shortcodes)
tangle-fragments.raku <-- (nested-shortcode-regex)
tangle-fragments.raku <-- (gather-fragments-and-files)
tangle-fragments.raku <-- (tangle-fragments)
tangle-fragments.raku <-- (write-tangled-files)
rich-hello.py <-- (import-libraries)
tangle-multi.raku <-- (define-commented-shortcodes)
tangle-multi.raku <-- (tangle-every-block)
Tangled to hello.py
Tangled to tangle.raku
Tangled to tangle-multi.raku
Tangled to rich-hello.py
Tangled to tangle-fragments.raku

That overwrote my test version of tangle-fragments.raku. Scary. Ran the new version to keep myself honest. It produced the same output, and appears to have correctly tangled my fragments!

$ bat rich.hello.py
───────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: rich-hello.py
───────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ from rich import print
   2   │ from rich.panel import Panel
   3   │ from rich.markdown import Markdown
   4   │
   5   │ md = Markdown("**Hello**, *World*.")
   6   │ print(Panel(md))
───────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────

Running rich-hello.py looks more interesting with a screenshot than a text block:

terminal screenshot
formatted output using Rich

Okay. Now I’m done.

I could have done this in Python. There are decent parsing libraries out there. But Raku did this on its own, without pulling in any extra — without pulling in any libraries.

Done? You barely started!

My tangle script is no competition for Org mode’s Babel.

  • It needs more error checking
    • circular fragment references
    • unreachable files (path, permissions)
  • smart handling of whitespace and indentation, to keep Python from becoming a chore
  • rendering fragment names in such a way that syntax highlighters can do something pretty with them (Especially when writing code in a language that Chroma has heard of)
  • hidden blocks
  • code evaluation and display of results

But it’ll do for now.