You know what else I can tangle? Yarn!
I spend a while using Raku to extract code from a Hugo post.
2020-09-03
@codesections@fosstodon.org found a typo! I forgot to show the target file name once command line arguments are in place. It should go
raku tangle-fragments.raku index.md
.
Let’s say I have a file. The one you’re reading, perhaps. Well, its original Markdown content.
It has a Hugo shortcode in it.
I based {{< code >}}
here on a shortcode from the Hugo docs. It presents highlighted code with additional context.
Really handy when you’re writing about code. Thing is, now I have two copies. There’s one here in the shortcode, and another in a hello.py
file that I’m writing about. I’d prefer there was only a single copy. That way they don’t get out of sync.
I could use Hugo’s readFile function in a new shortcode, including the contents of hello.py
in this Markdown file. Something like this:
NOTE
Actual shortcode logic left as an exercise for the reader.
But that still breaks up the writing flow a little bit. I’m writing the code over here, and writing about it over there. It’s a tiny complaint, but working with Org has spoiled me. I get to write the code in the same document that I’m writing about it in. Everything stays in sync, more or less.
What I want is to write about hello.py
here, and with a command have hello.py
appear on my filesystem, containing the Python code I’ve been describing.
And I want to do it without disturbing Hugo. Let it turn Markdown into HTML.
Tangling
This process is called “tangling,” and it’s popular in the admittedly small world of Literate Programming. The code is interleaved throughout some kind of document. A tool like noweb or Babel parses the document to create code files. Could be any kind of file, really. The process can get fancy.
But the start is not fancy: given a text file containing a {{< code file="(something)" >}}
, write the contents of that shortcode to the named file.
I love Raku’s approach to regular expressions. For starters, the syntax looks a bit more like describing a grammar. I can break the funny regex characters up with spaces, and clarify them with comments. In fact, I could someday build this up to a real grammar.
Secondly, it addresses the fact that most text we look at these days contains multiple lines. I didn’t have to worry about any special multiline flags to get this working.
Finally, getting at the named captures was — I wouldn’t say “obvious,” but at least “coherent.” I can treat the match variable $/
as a nested Hash. The important bits look something like this::
I can grab the named capture filename
of my matched shortcode
regex with $/<shortcode><filename>
— or ~$<shortcode><filename>
, depending on your preferred syntax.
This is all possible in languages like Perl with assorted flags, but I haven’t seen parsing treated so well by default since maybe REBOL.
Anyways, let’s run this thing.
Sweet.
Except — this Markdown file I’m writing. It has two file code blocks now. I want to tangle both of them.
Multiple output files
This requires a couple changes, since I’m writing code about Hugo shortcodes in a Hugo post.
To show shortcode directives without Hugo evaluating them, they need to look like shortcode comments. Their contents will get passed straight through as part of your post. To show {{< shortcode >}}
in a post, your Hugo content needs {{</* shortcode */>}}
.
So that’s lovely and all, but can be a headache of its own for this specific situation of extracting code from a blog post.
I need to remember this commented shortcode syntax.
NOTE
Goodness, that looks silly. Well, I’m writing this blog post as a test case for the code. I couldn’t figure out how to cleanly present the commented shortcode delimiters without Hugo and my code getting into a fierce argument.
If I wasn’t writing the code in this post, I could use something simpler, like this:
But that’s not the path I chose. It’s not easy to write programs that write themselves. Sometimes you must help them along.
That way I can replace those commented shortcode delimiters with their normal counterparts when I tangle later.
Now that I have that particular detail out of the way, tangle every block? Sure! Make a regular expression match :global
and it returns a list containing every match.
I think that about covers it. The shortcode recognition logic can stay the same.
And it works!
Unfortunately, I’m not quite done yet.
Multiple fragments
I’m not done yet because I don’t like to describe my code a full file at a time. I’d rather talk about this bit here, explain that bit over there, then mash it all up in the end.
Consistency counts, so I need to pick a syntax. Well — you’ve been reading along. You can see that I already made my choice. I got used to <<fragment-name>>
in Babel, where the attribute is called name
. Might as well keep doing that over here. Oh but hang on. I want it to stand out a bit. I’ll use angle quotes «‥»
.
NOTE
On a US keyboard using Vim or Neovim,
«
is a digraph which can be entered via Control-k followed by <<. Or if you’ve set up a Compose key, it’s Compose followed by << in any editor.
»
is the same, but >> instead.Or you can use
<<…>>
in your code and ignore my recent obsession with fancy characters.Yes, I know I could practically write it all with fancy characters in Raku. One step at a time.
Let’s go back to the Python code because it’s still so small.
Say I want to demonstrate the delightful Rich terminal library for Python.
But before I really use it in my code, I spend 1,500 words singing its praises.
It’s nice. I like it.
Okay, done singing. Time to write the rest of the program.
I identify the fragment with a name
attribute:
My code
block references the import-libraries
fragment by name when I’m ready for it.
I might spend some time talking about the code
shortcode in another post, but I dislike Go’s templating enough that this does not sound like fun.
Rounding up fragments to tangle
Recognizing an additional parameter doesn’t make my regular expression that much more complicated, but I can see things getting ore complex. I could even find a better pattern later. Let’s give the params their own named regex for some encapsulation.
That way I can drop it in shortcode
to say “oh and look for params
while you’re at it please.”
Okay, we recognize file
and name
parameters. What do we do with them? We gather them!
Tangling my fragments
Let’s see here. I know before I can write any files, I need to make sure everything’s tangled Trying to keep fragments easy to identify. They sit on a line by themselves, possibly with some leading whitespace.
Raku functions are lexically scoped, which means it’s perfectly okay to declare a function inside another function. Though next time I revisit his, I may want to think about a class or something to hold the complexity.
But what does that function need to look like? I’m still not sure I got it quite right. I mean I know the basic shape of it.
It needs some error checking. I know that much. Oh, and if it’s already been tangled I should avoid going through it again.
The idea of the thing is clear enough. Find and recursively tangle
each fragment found in this text, replacing the fragment references with their tangled text. Once that’s all done, cache and return the tangled text.
I flailed while tangling fragments. Lots of complaints from Raku about the difference between a Match
and a String
. There must be better ways. But the most important thing? I got it to work eventually.
Writing tangled files
After all that, writing the tangled files felt easy.
Then — theoretically — all these fragments I wrote will make a useful code tangler!
Might as well make it so this script can look at more than just the file I’m editing right now.
Easiest CLI I ever wrote, by the way. See?
Time for the real thing. I’m nervous. I shouldn’t be nervous. I know how this story ends. Then again I keep rewriting the middle.
That overwrote my test version of tangle-fragments.raku
. Scary. Ran the new version to keep myself honest. It produced the same output, and appears to have correctly tangled my fragments!
Running rich-hello.py
looks more interesting with a screenshot than a text block:
Okay. Now I’m done.
I could have done this in Python. There are decent parsing libraries out there. But Raku did this on its own, without pulling in any extra — without pulling in any libraries.
Done? You barely started!
My tangle script is no competition for Org mode’s Babel.
- it needs more error checking
- circular fragment references
- unreachable files (path, permissions)
- smart handling of whitespace and indentation to keep Python from becoming a chore
- rendering fragment names in such a way that syntax highlighters can do something pretty with them
- especially when writing code in a language that Chroma has heard of
- hidden blocks
- code evaluation and display of results
But it’ll do for now.
Backlinks
Got a comment? A question? More of a comment than a question?
Talk to me about this page on: mastodon
Added to vault 2024-01-15. Updated on 2024-02-01