JJA

Auto-instrument your code with OTel

2025-12-16T00:00:00+00:00

“Hot Chocolate” by David Thompson, CC BY-NC-SA 2.0

A whole year had gone by since the elves at Santa’s workshop had started using OpenTelemetry to collect and export telemetry from their multiple services, and things had been going well. But in a meeting room deep under the icy cover of the North Pole, trouble was brewing.

“Well what do you suggest, then?”, asked Duende Juniorsson, the junior elf in the team.

“I don’t have a solution, I just know where the problem is”, answered Gnomo Knullpointer perhaps a little more exasperated than he should be. “I know that OpenTelemetry has been a game changer, and that we want to have more of it. But instrumenting code has a cost, and I’m just wondering how much longer we can keep paying it”.

The elves had been relying on instrumentation libraries to generate telemetry without having to make changes to their own codebase. This “zero-code instrumentation” had been the key selling point when they started using OpenTelemetry. But it meant that they were limited either by the libraries that were available, or by the resources they could dedicate to writing their own.

“If we want to instrument something, and there is no instrumentation library for it, and we don’t have the resources to write our own, then the only other option is to instrument it manually”, said Duende.

“Sure, but that is only marginally less work. And anything you touch will have to be tested”, said Tess ‘t Moor, the QA elf. “So we might be saving time for you developer elves, but only because I’ll be the one testing things”.

There was a pause in the conversation, as each elf stopped to gather their thoughts. Santa, who was sitting at one end of the table looking nervous, did not like this. He did not like his elves getting upset and forgetting what Christmas was all about, but more than anything, he did not like being in a meeting where nobody was talking. It made him feel like maybe they were waiting for him to say something, and under pressure the only thing he could ever think of saying was “Ho ho ho”. He was pretty sure this was not the right time to say it.

Ada Slashdóttir, the team’s senior elf, cleared her throat and absentmindedly started tapping her cup of hot chocolate with a candy cane. “What if we get Perl to write the instrumentation code for us?”.

Auto-auto-instrumentation

“Most OpenTelemetry instrumentation libraries look very similar”, she continued. “They declare methods in a package that need to be instrumented, and then monkey-patch them to generate the necessary telemetry.”

“But not all telemetry is the same”, replied Gnomo. “OpenTelemetry specifies which attributes need to be set in what kinds of spans: a span representing an HTTP transaction needs to specify the URL it is for, one for a database operation needs to specify what database it is for, etc. We cannot have a one-size-fits-all approach”.

“We can probably get 90% of the way with a generic approach”, continued Ada. “And worry about the last 10% when we get there”.

“Isn’t this making the problem worse?”, asked Duende. “A minute ago we didn’t have the resources to write an instrumentation library, and now we are talking about writing a library to write instrumentation libraries?”.

“Ah, but we don’t need to write it ourselves”, said Ada. “We can use the new OpenTelemetry::Instrumentation::namespace”.

The elves loaded up the documentation and dove in.

Ada brought up her terminal. “Say you have some code that calls some package to do something”:

#!perl
use v5.36;
use UUID4::Tiny 'create_uuid_string';

say create_uuid_string;

“That will print out a UUID”, said Duende.

“Exactly”, replied Ada. “And if we add OpenTelemetry?”.

#!perl
use v5.36;

# Dump spans to STDERR in prettified JSON format
BEGIN {
    $ENV{OTEL_TRACES_EXPORTER}              = 'console';
    $ENV{OTEL_PERL_EXPORTER_CONSOLE_FORMAT} = 'json,pretty=1';
}

use OpenTelemetry;
use OpenTelemetry::SDK;

use UUID4::Tiny 'create_uuid_string';

say create_uuid_string;

“Nothing will change”, said Gnomo. “We loaded the API and initialised the SDK, but since we are not generating any telemetry, nothing is different”.

“Now let’s try loading the namespace instrumentation”, said Ada.

#!perl
use v5.36;

# Dump spans to STDERR in prettified JSON format
BEGIN {
    $ENV{OTEL_TRACES_EXPORTER}              = 'console';
    $ENV{OTEL_PERL_EXPORTER_CONSOLE_FORMAT} = 'json,pretty=1';
}

use OpenTelemetry;
use OpenTelemetry::SDK;
use OpenTelemetry::Instrumentation namespace => [
    'UUID4::Tiny' => 1,
];

use UUID4::Tiny 'create_uuid_string';

say create_uuid_string;

The output of the script had changed! The UUID was still being printed at the bottom of the screen, but above it were several lines with JSON encoded OpenTelemetry spans. Slightly edited for brevity, the output looked a little bit like this:

{
   "end_timestamp" : 1765582438.54618,
   "instrumentation_scope" : {
      "name" : "UUID4::Tiny",
      "version" : "0.003"
   },
   "name" : "UUID4::Tiny::create_uuid",
   "parent_span_id" : "dbed67eb5146795b",
   "span_id" : "c2f03b363333f610",
   "start_timestamp" : 1765582438.54615,
   "trace_id" : "386e98e4fa576fc5280e8da1f59d3031",
   ...
}
{
   "end_timestamp" : 1765582438.54669,
   "instrumentation_scope" : {
      "name" : "UUID4::Tiny",
      "version" : "0.003"
   },
   "name" : "UUID4::Tiny::is_uuid_string",
   "parent_span_id" : "e266cf2b01725042",
   "span_id" : "6165bab8a0b6f448",
   "start_timestamp" : 1765582438.54666,
   "trace_id" : "386e98e4fa576fc5280e8da1f59d3031",
   ...
}
{
   "end_timestamp" : 1765582438.54678,
   "instrumentation_scope" : {
      "name" : "UUID4::Tiny",
      "version" : "0.003"
   },
   "name" : "UUID4::Tiny::uuid_to_string",
   "parent_span_id" : "dbed67eb5146795b",
   "span_id" : "e266cf2b01725042",
   "start_timestamp" : 1765582438.54654,
   "trace_id" : "386e98e4fa576fc5280e8da1f59d3031",
   ...
}
{
   "end_timestamp" : 1765582438.54686,
   "instrumentation_scope" : {
      "name" : "UUID4::Tiny",
      "version" : "0.003"
   },
   "name" : "UUID4::Tiny::create_uuid_string",
   "parent_span_id" : "0000000000000000",
   "span_id" : "dbed67eb5146795b",
   "start_timestamp" : 1765582438.54598,
   "trace_id" : "386e98e4fa576fc5280e8da1f59d3031",
   ...
}
4c9debb9-6370-4d7c-94f2-a5f1799475ce

“Hey, this is much more like it”, said Duende, who had got pretty familiar with OpenTelemetry spans. “You can see that they all share a trace_id, and can track what span created which other span by looking at the span_id and the parent_span_id”.

“Yeah, and look at the names”, said Gnomo. “They are the fully-qualified subroutine name of the code that executed. So you can tell that when we called create_uuid_string it called uuid_to_string, which itself called is_uuid_string”.

“And all of that without using a pre-made instrumentation library for UUID4::Tiny, or manually instrumenting anything”, said Ada.

“How does this even work?”, asked Duende after looking at the code a little more carefully. “We load the instrumentation before loading the code to be instrumented. How does it know what to instrument before we import it?”.

“Good question!”, said Ada excitedly. “Let’s dive deeper.”

Looking under the hood

“When you load the namespace instrumentation with a rule like the one we gave it, it will look among the modules that have been loaded to see if any match the one we want to instrument”, said Ada. “In this case, since we haven’t loaded UUID4::Tiny, it won’t find it”.

“So how does it know when to instrument it?”, asked Gnomo.

“It uses a require hook that will execute when a module of interest is loaded. At that point, it installs the necessary instrumentation”, explained Ada.

“Is that safe? Won’t that make everything slower?”, asked Duende.

“To some extent, yes”, replied Ada. “Which is why this is not really made as a substitute for writing instrumentation libraries. It is meant as a way to facilitate the instrumentation of existing codebases, or as a stopgap measure to use when exploring what to instrument. But it’s hard to predict the impact in the abstract: we’ll always just have to benchmark things to see what the real impact is, and decide whether it makes sense to pay the price”.

“That seems reasonable, but why does it need the hook at all?”, asked Gnomo. “Wouldn’t it be simpler to just make sure to load it after the import?”.

“Ah, but there we hit another interesting caveat with this instrumentation”, replied Ada. “Let’s try it: what happens if we do that?”. Ada modified the code to look like this and re-ran it.

#!perl
use v5.36;

# Dump spans to STDERR in prettified JSON format
BEGIN {
    $ENV{OTEL_TRACES_EXPORTER}              = 'console';
    $ENV{OTEL_PERL_EXPORTER_CONSOLE_FORMAT} = 'json,pretty=1';
}

# Import the function first, then instrument
# This probably won't do what you want!
use UUID4::Tiny 'create_uuid_string';

use OpenTelemetry;
use OpenTelemetry::SDK;
use OpenTelemetry::Instrumentation namespace => [
    'UUID4::Tiny' => 1,
];

say create_uuid_string;

“It still works”, said Gnomo. “I can see the OpenTelemetry traces”.

“But there’s one span missing!”, said Duende. “We lost the one for create_uuid_string”.

“Precisely”, said Ada. “A case like this, where we load a package and import a function from that namespace into ours, is very common. But if we load the instrumentation after we’ve imported it, when we execute the imported symbol from our own namespace we’ll be running the uninstrumented code. To solve this we have to instrument the code before importing it, but then we have to use the require hook”.

“This suddenly feels very Perlish”, said Duende. “In that it is very powerful but also a little risky”.

“It’s all about taking managed risks”, replied Ada. “This instrumentation works best when used in a targeted way, aimed at having the smallest impact possible while still being useful. That’s why it tries to give you plenty of options to control what gets instrumented and when: you can match packages with literal strings or with regular expressions, and while we haven’t been doing it here, you can also do the same for individual functions within any package of interest”.

“Is that why the options are in an array reference?”, asked Duende.

“Yes”, answered Ada. “The instrumentation will process the rules in order. You can pass them in an array reference to control that order, or in a hash reference if you don’t care about the order. But the array reference is more predictable”.

Here a use, there a use, everywhere a yule use

“I can see how this would be useful”, chimed in Tess. “But I could see this list of ‘rules’ getting unwieldy, if we need to have them all in the same place and they include package and subroutine matchers, etc”.

“Yes, that can become a problem”, agreed Ada. “So you don’t actually have to have everything in the same place. You can load the instrumentation multiple times with different rules, and each invocation will be independent. This is particularly true of options, which can be passed together with the rules to modify how that particular set of rules are interpreted, or to load the rules from an external source.”

“That’s useful”, said Gnomo. “But even then, if the code we are instrumenting is our own code, it might be awkward to have the instrumentation live far away from the code itself. In those cases, manual instrumentation might still be better”.

“Yeah, I agree. A small scope is always better”, replied Ada. “Luckily, there’s a related instrumentation that you can use in those cases: OpenTelemetry::Instrumentation::caller. Let me show you how using that one looks like”.

The instrumentation is coming from inside the package

“Say we have some utility package where we have some functions”, said Ada.

#!perl
package Santa::Workshop;

use v5.36;

sub allocate_gifts { ... }

sub is_naughty { ... }

# Sensitive data. Do not reveal!
sub dump_naughty_list { ... }

“If we want to auto-instrument this code”, said Ada, “but we don’t want (or can’t) touch it, we can load the caller instrumentation. We just need to make sure we load it after the functions have been defined”.

“Does it also take the same options and rules as the namespace instrumentation?”, asked Duende.

“Yes”, said Ada, “but since we are already ‘inside’ a package, so to speak, we skip the package-level matchers and go straight to the subroutine-level ones.”

#!perl
package Santa::Workshop;

use v5.36;

sub allocate_gifts { ... }

sub is_naughty { ... }

# Sensitive data. Do not reveal!
sub dump_naughty_list { ... }

use OpenTelemetry::Instrumentation caller => [
    dump_naughty_list => 0,    # Ignore sensitive function
    qr/.*/            => 1,    # Instrument the rest
];

Wrapping up

The elves were happy to have at least a path forward. They understood that these instrumentations were experimental, and that the best way to help was to use them and see where they felt awkward. And that’s exactly what they aimed to do.

It was another successful day at Santa’s workshop, making hard things possible, and Santa was happy to see that his elves had once again managed to figure things out by themselves. He was proud of his elves. Proud to see them grow and learn and try new things. And proud as well to see that, even when deep in the weeds of some technical problem, they knew that what mattered was what they were working towards. The rest was just the tools for the job.

Oh-oh. Santa had been lovingly staring at the elves for too long, and now they were staring back, as if waiting for him to say something. But this time, he knew exactly what to say.

“Ho ho ho!”, laughed Santa, and everybody smiled.

London Perl & Raku Workshop 2025

2025-12-02T00:00:00+00:00

This year’s London Perl & Raku Workshop took place on 29 November, a couple of days before I write these lines. I first attended this conference in 2018 when I gave a talk about starting my life as a programmer, and I’ve attended every iteration since then.

But this year was special, because it was the first time I was in the organising committee. And this was not just my first time organising this conference in particular, but the first time I ever helped organise an event of this type at all.

I knew that organising a conference would not be easy, and indeed there are a lot of things that I would now have done differently. There’s a lot to digest, and I know my feelings will change as time passes and the dust settles. But I wanted to leave here a collection of my fresh thoughts about the experience and its impact, even if only for future reference.

The experience was a success. It was hard, and there were times when I would not have put my money on the event even taking place, but I’m glad that we stuck with it and pushed forward regardless. I think the venue worked well, and I think the post-event social was good, with plenty of attendees and a comfortable room for ourselves.
The community that participates in this event is incredible. We announced the event late in the year, with less than a month’s head start, and were floored by the number of talk submissions and the attendance. All in all, we had 22 talks, 7 of which were lightning talks, from 18 different speakers. There were 57 registered attendees all in all.
While we can have a successful conference despite the organising team’s failings (because the community is amazing), we should not abuse this. I know there were people who care deeply about this community in general and this event in particular who could not attend or did not have time to prepare the talks they would have liked. I cannot apologise enough for this, and I at least will try to avoid this in the future. We owe the community that much. 🙇
Even in his absence, Matt Trout’s presence was clearly felt. He was explicitly present in Stevan Little’s “YAARP” talk and in several conversations during the social, but his contributions had a much broader silent impact. We build on the shoulders of giants. Matt, you are already missed.
Remote talks are hard. Not only for the obvious technical reasons, but also because they are harder to manage as an organiser. They are good in principle, because they can accommodate more of people’s varying circumstances. But even when everything goes to plan it is difficult to make them feel like they fit with the rest. I do not think this is a technology problem, and I’m not sure it can be solved.
Swag is not that important. Or at least not important enough for anyone to mention to us that they wished they had received a hoodie or a scarf or what have you. That said, I do feel that having some tangible reminder that the event took place would have been nice.
I missed having some sort of catering. It was awkward to have a “coffee break” which meant “go outside and find yourselves some coffee”. And I don’t know what to say: I like biscuits. I want biscuits.

All in all, I am happy that I participated in this year’s event, and I look forward to helping bring the next iteration of this event to fruition next year. More than anything, I am happy to be able to help build this community.

Thank you to Andrew Mehta, Lee Johnson, Julien Fiegehenn, John Davies and very specially to all of those who were there. You made it all worth it.

Step into the cookie jar

2024-05-23T00:00:00+00:00

I’ve written my fair number of Raku distributions, but I think the one that seems to be the most successful, in that it appears to have the most use, is HTTP::Tiny, a port of the Perl library of the same name. This was released a couple of years ago already, so you might be familiar with it. You can read the post where it was first announced, but as a summary, here are some of the things that it can do:

it’s a fully HTTP/1.1 compliant library
it has no dependencies, although some of its functionality (eg. HTTPS support) depends on optional libraries
it supports both HTTP and HTTPS proxies
it supports streaming requests and responses, form uploads, ranged requests

I had a lot of fun working on it, and I learned loads while doing so as well. But perhaps more importantly, it filled a niche that I think was missing, and judging on its usage, I’m not the only one who thinks this.

Overall, I’m pretty happy with how that one turned out

However, for all the things it does support, there are some things that it can’t do which I really want. And some of those, specifically timeouts and cookie support, have been on the to-do list from the very beginning…

Timeouts are missing because of some limitations with the handles that it uses under the hood, so this will be a hard problem to crack. But from HTTP::Tiny v0.2.0, cookie support is no longer a limitation, and I wanted to talk a little bit about how that came to be.

I’ve been thinking about HTTP cookies quite a bit while working on this, but I know that not everybody has had the pleasure of reading the RFC. So just in case, let’s do a very quick cookie primer.

The core idea of HTTP cookies is to give the server the ability to store state in the client. It’s a simple idea, but as you can imagine, it has all sorts of ramifications and security implications, some of which we’ll discuss below.

They’ve been used for a long time. They started being used experimentally in 1994, and since then there’s been several RFCs that have specified what you may call different “versions” of cookies: RFC 2109 published in 1997, RFC 2965 published in 2000, and RFC 6265 published in 2011, which is the most current one (at the time of writing). The next version is currently being drafted and has seen several revisions, so it’s very likely that it will end up being published Real Soon Now.

But since RFC 6265 is the one that is currently active, that’s the one that I’m targeting and the one we’ll be focusing on.

Their shape is very simple: it’s just a key/value pair with a set of additional attributes. They’re sent by the server in a special header called Set-Cookie with the key and the value, followed by the cookie’s attributes.

Every RFC so far has specified a list of recognised attributes with specific meanings, but this list is constantly changing, with new attributes being added and old ones being removed. But users have always used attributes to extend the specification and to experiment with additional features. Sometimes these become well-received and spread, becoming de-facto standards (like the SameSite attribute, which has yet to be specified in any RFC).

All of the attributes that are currently recognised by RFC 6265 are there to govern what the scope of the cookie should be. Some examples of existing attributes are:

Secure and HttpOnly, which restrict the protocols of the cookie
Domain and Path, which restrict the URLs the cookie is applicable to
Expires and Max-Age, which restrict the lifespan of the cookie

Using these attributes, the client determines what cookie is applicable for a request, and then sends it back to the server in the Cookie header (although the cookie that the client sends back to the server does not contain any attributes, only the value).

Domains and paths

Back to the cookie attributes, the Domain and Path attributes are interesting because there’s some risks involved.

Let’s take the following four cookies as an example:

A=1; Domain=    domain.com; Path=/
B=1; Domain=    domain.com; Path=/path
C=1; Domain=sub.domain.com; Path=/
D=1; Domain=sub.domain.com; Path=/path

Each of these cookies has a separate combination of Domain and Path, and they will serve to illustrate how these attributes interact. The table below illustrates which cookies the client can set (in the Cookie header) for any given request, and which can be set by the server (in the Set-Cookie header) for the same requests.

Domain	Path	Cookie				Set-Cookie
`domain.com`	`/`	A				A	B
`domain.com`	`/path`	A	B			A	B
`sub.domain.com`	`/`	A		C		A	B	C	D
`sub.domain.com`	`/path`	A	B	C	D	A	B	C	D

Cookies that the client and server can set for different requests

So, for example, with the cookies above, a client making a request to domain.com/ can only send cookie A along with the request. All other cookies either apply to a subdomain of the domain the request is for, or to a path that is under the one that is being requested.

On the other hand, a request to sub.domain.com/path can be sent by a client with cookies A, B, C, and D, because they all apply to a domain that is either the one being requested or to a domain that is above it, and to either the path that is being requested or one above it.

Note, however, that the rules that specify which cookies the client can send with a given request are not the same as those that specify which cookies the server can set for the same request.

Taking the same requests as in the previous examples, in response to a request to domain.com/ the server can set cookies A and B, even though B applies to a path that sits below the one that was requested. And the same goes for a response to a request from sub.domain.com/path: in that case, the server can set all four of the example cookies, including cookies that apply to domain.com, which sits above the one that was requested.

Supercookies

And here’s where we bump into the first real security consideration around cookies.

If we go back to the table above, we can see that a cookie that applies to domain.com applies also to any domain under domain.com. But this means that there’s technically nothing that stops you from setting a cookie with a Domain set to plain com. And if you did, that cookie would technically apply to any domain that ended with .com.

Because of their ability to apply so widely, any cookie with a Domain set to a top-level domain is called a “supercookie”, and they present a risk that needs to be accounted for.

Before we get too far down the rabbit-hole, let’s put a pin on this and get back to our main topic. We’ll come back to supercookies later.

Throughout the discussion of cookie applicability above I was careful to use the word “can”: “the client can send such-and-such cookie”. Why is that? This is because cookies are entirely optional (indeed, this is why we can have HTTP user agents that are compliant with HTTP/1.1 but do not support cookies, like HTTP::Tiny).

And even if the client supports cookies, there are good reasons why the client can choose not to send them, or the server not to set them: for the client, this requires cookies to be stored somewhere, which uses memory, and this may be limited. And for the server, setting cookies requires sending data on the request’s headers, which themselves have limits, and will have some overhead, etc.

The goal of this project was to create a library that would take on that responsibility on the client side. It would retrieve cookies from a response, determine whether they were being legitimately set by the server, find storage space for them, and determine also which of the cookies that have been stored are applicable to any new requests. This is traditionally called a “cookie jar”.

Furthermore, since the idea was for this was for it to be usable with HTTP::Tiny, I wanted this cookie jar to follow the same principles as that library: with minimal dependencies, and compliant with the specification, in this case RFC 6265.

Prior art

When I looked in the Raku ecosystem I could find two cookie jars: HTTP::Cookies, which seems to be a port of a Perl library, and Cro::HTTP::Client::CookieJar. But both of these had issues that made them unattractive from my perspective.

Poor interoperability

The first issue was that both of these are actually distributed with an HTTP client. This might not sound like a big problem, but remember the goal here is to have a cookie jar that can be used with HTTP::Tiny, and that is a distribution which is supposed to have no dependencies. If in order to use cookies with it you need to install another user agent, then the exercise becomes a little pointless.

A related issue is that, because they are distributed with these user agents, they rely on those frameworks to work. For example, the HTTP::Cookies module has a add-cookie-header method that will generate a Cookie header to send with a request. But in order to use it, you need to give it an HTTP::Request object, and the method will inject the header into the request directly, there is no way to access that header without using that object.

The Cro::HTTP::Client::CookieJar add-to-request method has a similar requirement, but this time using a different class to represent a request: this one needs a Cro::HTTP::Request object, as well as a Cro::Uri object.

What this means is that in order to use these libraries, you have to buy in to the rest of their frameworks, which makes interoperability more difficult (and makes them unsuitable for this particular project).

Non-compliant behaviours

But even if I had decided that this was not a problem, or I had worked around it with some wrapper code or something of that sort, none of these were as compliant with RFC 6265 as I would have liked.

Some of these were relatively minor. For example, the spec specifies an order in which the cookies should be serialised in the Cookie header, but Cro::HTTP::Client::CookieJar does not follow this order. This is “relatively minor” because even though this goes against the spec, this point is only a “should”, and the spec also states that servers “should not” rely on the order of the cookies.

But some of these can have more serious consequences. For example, HTTP::Cookies does not do path restriction for cookies, and in this case this is a “must”.

And although not in the spec, perhaps their most serious limitation in my eyes was that neither of them had a way to protect the user against the supercookies mentioned above (we’ll get back to this point later).

Introducing Cookie::Jar

Since none of the existing tools were suitable for my goals, this project resulted in the release of a distribution called Cookie::Jar. In the spirit of HTTP::Tiny, Cookie::Jar is a minimalist HTTP cookie jar which takes inspiration from existing Perl libraries¹, and is compliant with RFC 6265. It does have one dependency on IDNA::Punycode to support international domains, but it has no other functional dependencies, by which I mean that that is the only distribution that it needs in order to function.

Other than that, the main difference between this cookie jar and the ones discussed above is that its interface relies only on Raku built-in types, to make it usable in the largest possible set of contexts.

But in order to protect against supercookies there is still one more problem we need to solve.

The Public Suffix List

We said supercookies are cookies that have their Domain attribute set to a top-level domain. But this begs the questions: what is a top-level domain?

Some of them are pretty easy to spot: .com, .org, .net. Those we all know.

In recent years, however, the list has grown significantly with the release of newer top-level domains: we now have .pizza, .香港. And new ones get added all the time.

More seriously, you cannot simply assume that the last component of the domain will be top-level domain, since there are combined top-level domains, like .co.uk, .ne.jp or even cases like .pvt.k12.ma.us.

I do not want to get into the discussion about how close to the top a domain needs to be to be a “top-level domain”. Not only because it’s difficult, but mainly because, luckily for us, that question has already been answered by a thing called the Public Suffix List.

The Public Suffix List is a project initially started by Mozilla, and currently maintained by a community of volunteers who is constantly keeping an eye on what new top-level domains appear. It’s available online, and it’s used very widely by a large number of different projects.

The list includes both domains that have been registered by ICANN, as well as “private” domains that have been submitted by certain domain owners who want to treat their domains as a top-level domain. This includes cases like .blogspot.com, .github.io, or .googleapis.com (at time of writing).

This list has become the way to protect against supercookies, so if I wanted to be able to identify supercookies, I needed a way to get access to the list.

Like with the cookie jar, I tried looking for prior art in the Raku ecosystem, but surprisingly there was nothing. This explains why none of the other cookie jars offer this feature.

Using the list

The list has three types of entries. There’s literal top-level domains, like kumamoto.jp; patterns that match all domains under a given literal, like *.yokohama.jp; and exceptions that restrict those patterns, like !city.yokohama.jp. Since the list has been designed to be machine-readable, parsing it and using it is not actually the hardest part of this problem.

The real challenge comes from the fact that this list is constantly being updated, sometimes several times per week. Which raised the question about how a library that uses this list can be kept up to date.

One possible solution would be to curate the changes, so that every month or so I could look at the list and decide whether the most recent changes were worthy of a release. However, I did not want to commit to that, and I also did not want to take on the responsibility of making that decision.

Another possibility would be to shift that responsibility to the user, and either require or allow them to provide a list that the library can then use. This is indeed the approach taken by some of the existing Perl libraries I looked at as a reference. However, this meant that the list could not be pre-compiled into the module, and had to be parsed every time the module was loaded. This was something I was trying to avoid.

The third possibility was an idea I got from the psl Rust library, which every day checks for changes in the upstream list, and mints a new release if anything has changed. The first time I saw this I thought it was madness, but this is actually the method I ended up going with.

Introducing PublicSuffix

This resulted in a new distribution called PublicSuffix which was released very soon after Cookie::Jar. This is a very small distribution that parses the list, stores it on compilation, and gets updated automatically whenever a new version of the list is made available.

It is not clear to me whether this is the best solution to this problem, but it did mean that I could offer a good balance between allowing users to control what version of the list they are on, while still supporting fast execution times by compiling the list into the module². That said, I am open to the idea of adding support for users providing their own lists if this proves to be a wanted feature.

Putting it all together

We’ve now delved several levels deep, so it’s a good idea to try to go back to the surface and see how all these pieces fit together.

The PublicSuffix module is now available on the Raku ecosystem, and can be used on its own for whatever you think might benefit from it.

use PublicSuffix;

say public-suffix '福.個人.香港';
# OUTPUT: 個人.香港

say public-suffix 'test.pvt.k12.ma.us';
# OUTPUT: pvt.k12.ma.us

say registrable-domain 'raku.land';
# OUTPUT: raku.land

By importing it, it will export two functions. The main function is public-suffix, which takes a string representing a valid host and returns a string with the top-level domain of that host, or the type object if the host is an IP address. It will also export the registrable-domain function, which also takes a valid host as a string and returns the “registrable domain” of the host, or “the host’s public suffix and the domain label preceding it, if any”.

Both of them support hosts in either ASCII “punycode” or in unicode, and they will attempt to return a value in the same format that it was provided in.

In order to use it with Cookie::Jar, all you have to do is have the module installed. As long as is it is usable, Cookie::Jar will load it and use it to reject any supercookies that may have been received from a response.

use Cookie::Jar;

my $c = Cookie::Jar.new;
$c.add: 'https://foo.com', 'foo=123; Domain=foo.com';
$c.add: 'https://foo.com', 'bar=234';

# Rejected if PublicSuffix is available
$c.add: 'https://foo.com', 'super=1; Domain=com';

say "{ .name } -> { .value }"
    for $c.get: 'GET', 'https://foo.com';
# OUTPUT:
# foo -> 123
# bar -> 234

# The 'bar' cookie does not apply to this subdomain
say $c.header: 'GET', 'https://www.foo.com';
# OUTPUT:
# foo=123

As for the cookie jar itself, as you can see in the code snippet above, you can add cookies with the URL that issues the response, and the value from the Set-Cookie header. The jar will know how to parse that value, it will determine which cookies are valid and can be set by this response, and will know where to store them for future use.

To allow for inspection or for any extra processing you may want to do on the cookies, you can retrieve them from the jar. In this case, however, you have access to read-only versions of the cookies, to ensure that the cookies in the jar remain the ones that were received from the server.

Alternatively, you can give it the request method, and the URL the request is for, and the jar will return a the value for the Cookie header that can be sent along with that request.

And coming all the way to the top, starting from HTTP::Tiny version 0.2.0 the constructor accepts a new cookie-jar parameter which can be set to an object to use to process cookies. Importantly, the client does not internally check that the cookie jar object is an instance of Cookie::Jar: as long as the class of the cookie jar provides an interface that is compatible, HTTP::Tiny is happy to use it

use HTTP::Tiny;
use Cookie::Jar;

my $cookie-jar = Cookie::Jar.new;
$cookie-jar.load: 'cookie.jar' if 'cookie.jar'.IO.e;

my $client = HTTP::Tiny.new: :$cookie-jar;

... # Client will use the cookie jar

$cookie-jar.save: 'cookie.jar';

Parting thoughts

Before wrapping up, I thought it would be good to go over a couple of things that this project brought to mind.

The first is the now almost commonplace idea that Raku is “a young language with a long youth”. A couple of times now, when starting to use the language in earnest for a serious task, I bump into holes in surprising places. For a specific example, when setting out to implement cookie support, I did not expect to find no tool to check the public suffix list and to have to implement my own. It caught me by surprise to have to go that low-level.

The good thing, as I think has also been said by others before me, is that Raku is very versatile, so the tools you need to fill in those holes are often available. I find that experiences like this one have a value not only in their specific outcomes, but also in the holes that get covered along the way.

A related but different aspect is that even when the tools are available, it is not always easy to find them, or to evaluate which of them are fit for purpose. The process of finding existing cookie jars, and then realising why they were not useful for my task, took a considerable amount of effort. And there may be other cookie jars that I couldn’t find, which only makes this more serious.

I think the Raku ecosystem is good. I think there’s plenty of good in there, but to a large extent because of that “long youth”, there’s a lot of stuff that we need to curate, and building the tools that allow us to find the real gems in that ecosystem is going to be hard work, but very valuable.

Thank you.

Specially HTTP::CookieJar, HTTP::Cookies, and Mojo::UserAgent::CookieJar. My gratitude to the authors and maintainers of those distributions. 🙇 ↩
Note that this is just my expectation, and has not been verified by a benchmark. ↩

Santa Kotlin is coming to Perl

2023-12-24T00:00:00+00:00

This post starts with a long introduction to explain why I’m writing, but if you want to skip all of that you can jump directly to the meat and potatoes.

Stay awhile and listen

I grew up as a Nintendo kid. Some of the first video games I played were on an NES, and the first console I ever owned was an SNES. I have very fond memories of those times and of the games I got to know and play.

Although at the time I don’t think I considered applying that label to myself, it was very much a part of how I understood myself and my place among my peers. I was a Nintendo kid, and they… well, they were something else. Sega kids, maybe.

This misguided sense of identity paired well with a misguided sense of loyalty, which made it so I found it difficult to enjoy both at the same time. If I was a “Nintendo” kid, what would it mean if I enjoyed Sega… things? What would it mean to own one?

This was not only related to video games, either. I remember very easily falling into this trap with all sorts of similar “contrasts”. I was a Beatles kid, so I couldn’t like The Rolling Stones. I was a Star Wars kid, so I couldn’t enjoy Star Trek. The list was tragically endless.

This was not so hard when my team was inarguably better than the other. But I remember how hard it got to be a “Nintendo kid” when the PlayStation came around. It was gritty, it was powerful, it was exciting… and it felt inaccessible.

Am I still reading the Perl advent calendar?

Ah, yes. Perl.

I’ve always loved Perl. It was not my very first programming language (you and me, BASIC, for life), but it was the first one where I felt like I could write real programs. The first that I felt was worth mastering, and the one I’m most comfortable with, even today.

So, surprise surprise, I was a Perl kid.

And there have been times when being a Perl kid has not been easy.

I am fortunately past the time when I look at the world in terms of clubs that you belong to because of the things you like. I will have you know I can listen to both Radiohead and Coldplay without breaking a sweat (I take no responsibility for deciding what contrasted with what).

But to this day, there are aspects of this worldview that remain in me.

Perl’s PlayStation

I imagine this largely depends on my particular interests, but for the Perl kid in me, it was hard to see how easy the other kids had it when they wanted to integrate with other languages.

To me, this was the PlayStation to Perl’s Nintendo.

I remember several attempts trying to teach my teenager-self how to write XS, so I could bind to this or that library. I remember feeling frustrated and defeated. I remember wondering if this meant that Perl was holding me back…

The answer is “no”. If I was being held back, it was me who was doing so by again thinking in terms of clubs.

But even if I had continued to see the world through that lens, the Perl we have at our disposal today is miles from the Perl I learned as a kid. There are still, I am sure, plenty of areas where I think Perl has to catch up. But we are at a moment where Perl is positively blooming with new features and tools, that make catching up possible, if not outright easy.

In the last two versions alone (at the time of writing, 5.36 and 5.38) we have n-at-a-time iteration, a native try with finally support (finally!), the new defer blocks, native booleans, the new builtin namespace, and a powerful new syntax for defining classes. Not to mention other recent native features (like sub signatures and the isa operator), or the things made possible via CPAN: async/await support, the renewed efforts into PDL, and what I might consider the jewel of modern Perl: FFI::Platypus.

Time will tell, but I feel like this is what it must feel like to live during a renaissance.

Any chance of having actual code in this post?

Yes, I’m getting to that. Now that I’ve finished with the introduction we can get to the meat and potatoes of this post. I hope I didn’t lose too many of you along the way.

Binding to Kotlin from Perl

What motivated this post in the first place was a task at work where I was asked to look into the feasibility of integrating with a third-party that provided SDKs for several languages… but not Perl.

Lucky for me, they had made the code of those SDKs publicly available, so I could examine it. And while looking through them I realised that most of the heavy lifting was done by binding to a shared C library. My teenager-self would have had a traumatic flashback sequence at this point, but this is modern Perl. We have FFI::Platypus. “This will be easy”, I thought.

The challenge came when I realised that the library was originally written in Kotlin via what they know as “Kotlin/Native”, which generates header files with some ad-hoc hoops for us to jump through. As an attempt at simplifying things, I’ve put together a repository with a sort of sample distribution that you can play around with as an illustration. The code examples below will be taken from it.

In any case, the native Kotlin extension will take code that looks like this:

package example

fun reverseString(str: String) : String {
    return str.reversed()
}

and eventually wrap it in a C struct which will look like the one below:

typedef struct {
  /* Service functions. */
  // ... Snipped 28 fields with fields pointing to service functions

  /* User functions. */
  struct {
    struct {
      struct {
        const char* (*reverseString)(const char* str);
      } example;
    } root;
  } kotlin;
} libexample_ExportedSymbols;

extern libexample_ExportedSymbols* libexample_symbols(void);

Which, to summarise, is exposing a global libexample_symbols function which returns a pointer to a struct where the last field (named kotlin) holds a pointer to a struct with a field (named root) which holds a pointer to a struct with a field (named example) which holds a pointer to the function that you wrote.

That’s a mouthful.

When I first saw this, and saw that doing it in eg. Ruby (the SDK I was looking at for guidance) was not only possible, but relatively simple-looking, I got pangs of that PlayStation feeling.

But as it turns out, FFI::Platypus already gives us all the tools to deal with something like this.

The first thing will be to define the nested structs, and for that we will need FFI::C (remember that you can look at the whole file these snippets are taken from in the sample repository):

package
    Santa::Kotlin::Example {
    FFI::C->struct( Example => [
        reverseString => 'opaque',
    ]);
}

package
    Santa::Kotlin::Root {
    FFI::C->struct( Root => [ example => 'Example' ]);
}

package
    Santa::Kotlin::Kotlin {
    FFI::C->struct( Kotlin => [ root => 'Root' ]);
}

package
    Santa::Kotlin::Symbols {

    FFI::C->struct( Symbols => [
        # ... 28 skipped fields which we must have here too ...
        kotlin => 'Kotlin',
    ]);
}

These packages are only for internal use, so that’s why they have a newline after the package keyword: it makes it so that if this code is ever put on CPAN, these packages will not be indexed.

When defining a struct with FFI::C, the first parameter is a name that can be referred to later, which is why these are defined from the inside (the ones most deeply nested) going out. It means I can refer to the types of the inner fields when defining the outer structs, like in the root field of type Root in the struct for the Santa::Kotlin::Kotlin package: since it is of type Root, its value will automatically be cast into a Santa::Kotlin::Root object.

We still need to get our hands on an instance of this outermost struct, and for that we have to bind to that global libexample_symbols function:

# Register $ffi with FFI::C, so new types become available
FFI::C->ffi($ffi);

# ...

my $symbols = $ffi
    ->function( libexample_symbols => ['void'] => 'Symbols' )
    ->();

Since we’ve told FFI::C that it should register any types it creates with this instance of FFI::Platypus, we can use the Symbols type (which corresponds to the Santa::Kotlin::Symbols package defined above) as the return value of this function.

Note also that we are not attaching this function, because we are not going to expose it to our users. We only want to be able to call it once so we can get a reference to the struct it returns, which we store in $symbols.

Once we’ve done all this preparation, we are ready to attach any functions in our example Kotlin package to our Santa::Kotlin Perl package, and we do this by using the memory addresses of the functions we are interested in:

$ffi->attach(
    #          we look up the address    and give it a Perl name
    #                                \               \
    [ $symbols->kotlin->root->example->reverseString, 'reverse_string' ],
    ['string'] => 'string',
);

At this point, we are ready to call this function as we would any other from our perl code:

use Santa::Kotlin;
say Santa::Kotlin::reverse_string('lrep ot gnimoc si niltok atnas');
# OUTPUT: santa kotlin is coming to perl

These are good times to be a Perl kid, so happy holidays to all the good ones out there.

Happy hacking!

Bundling specific modules

2021-04-28T00:00:00+00:00

The last time I wrote about bundling vendored modules in Perl, I was a little annoyed by the fact that the tools I was using did not allow me to bundle a specific dependency. They made it very easy to bundle all of them. But every time I’ve needed to do this in the past, I’ve always only needed to bundle some dependencies (the ones that I’m vendoring for whatever reason), being happy to fetch all the rest from upstream public sources.

I said as much that time:

If you only want to bundle some dependencies with your project you’ll have to manually remove all the ones you don’t care about (from both vendor/cache and your 02packages.details.txt.gz).

As it turns out, doing this is much simpler than I thought.

Honorable mentions

Considering the TIMTOWTDI Perl ethos, it should come as no surprise that there are a bunch of tools already available on CPAN to do more or less what I was looking for. So before going through what I actually ended up using, let’s go through some of the alternatives I considered… and why I didn’t use them.

Carton

This was the tool I initially used in my previous post. It is great for what it does, but does not allow you to bundle specific distributions. As far as I’m concerned, it’s the best tool for this sort of thing… unless you’re trying to do what I wanted.

Pinto

I’ve heard about Pinto in a couple of places, and I really wanted to like it. However, it’s not only massive, it also seems to unfortunately be rather unmaintained.

I tried installing Pinto on a fresh Perl install and it meant installing a whooping 175 distributions. And looking through the list of dependencies, that hardly comes as a surprise: the list includes heavy hitters like Moose, LWP::UserAgent, DBIx::Class, Plack, and Starman.

It might be that it’s the right tool for some task, but not for this.

CPAN::Mini

CPAN::Mini is a fairly minimal distribution that provides a minicpan command line utility to create a local clone of CPAN. Pair this with something like CPAN::Mini::Inject and its mcpani and you can now create ether a CPAN mirror or a superset of CPAN.

If I wanted to create a DarkPAN that served all the code in CPAN as well as some extra distributions, these two are probably the tools I’d go with.

Still, this was a little more than what I had bargained for, since I only wanted those extra non-CPAN modules, skipping what could already be obtained upstream.

And the winner is…

CPAN::Repository

In the end, CPAN::Repository did everything I wanted, and with a fairly small number of dependencies.¹ The absolute minimal way to do this is with something like the following:

perl -MCPAN::Repository -E '
    CPAN::Repository->new( dir => "vendor/cache" )
        ->add_author_distribution(@ARGV)
' SOMEAUTHOR Some-Distro-1.337.tar.gz

This will generate the following file structure:

vendor/
└── cache
    ├── authors
    │  ├── 01mailrc.txt
    │  ├── 01mailrc.txt.gz
    │  └── id
    │      └── S
    │          └── SO
    │              └── SOMEAUTHOR
    │                  └── Some-Distro-1.337.tar.gz
    └── modules
        ├── 02packages.details.txt
        ├── 02packages.details.txt.gz
        └── 02STAMP

which can then be used following the same instructions as in my initial post.

The use case

What motivated this was work on a codebase that depends on some CPAN distributions that are currently undermaintained.

When possible, the first choice is of course to move away from those dependencies to other alternatives. Maybe even alternatives maintained by yourself or the organisation you work for.

But in some cases, moving away from a dependency can be too much work, or you might be unwilling or unable to commit to maintaining a new distribution (and having two unmaintained distributions instead of one helps no one).

In this case, the solution I was investigating was whether we could create a fork of the distributions in question, and go through the regular release cycle… except without a publicly indexed release.

This has all the good things we know that come with a proper release: version numbers that can be tracked and maintained using your regular cpanfile; auditable change logs; etc. But all of that without the commitment to third parties.

To this end, I ended up writing a small script that would wrap around those CPAN::Repository calls and make things a little more extensible and robust.

The script takes a mandatory list of possible URIs (which can either point to external repositories on providers like Github or GitLab, or directly to distribution tarballs) and some options to define how that distribution should be indexed. It adds some checks to encourage the use of proper releases, but these are designed to be easily skippable in case you feel like your foot could use a bullet-hole.

And that’s pretty much it! All I need now is to convince the rest of my team that this is a reasonable way to go. But even if we end up not using this, I’m sure this will come in handy.

All in all, I’m glad I got a chance to take another stab at this problem, and that the solution was as simple as it was.

Installing on a fresh Perl like with Pinto, it installed 56 distributions in total. This is not tiny but it’s also not the end of the world. ↩

Porting from Go

2020-11-22T00:00:00+00:00

In the past year or so, every time I’ve wanted to use hot-reloading¹ for some service I’m writing, I’ve reached for a tool called reflex. This tool is not particularly original, but it is a reasonably good implementation, and it ships as a standalone-binary, which makes it very easy to move about.

It is also written in a language I have some familiarity with, and one that I’m trying to learn more about.

So of course I decided to implement my own.

The motivation was basically the same as the one that lead to HTTP::Tiny: a desire to understand how the code worked, and a search for real-world problems to attempt to solve using Raku. I might write more in a later post about the specifics of lorea, but in this post I want to focus on one aspect in particular, which I think showcases some of the strengths of Raku… and some of its weaknesses.

Some Go code to get going

One of the nice features of reflex is that it batches file-system changes to limit how often the command it needs to re-run gets executed. This is how that feature is explained in the documentation:

Part of what reflex does is apply some heuristics to batch together file changes. There are many reasons that files change on disk, and these changes frequently come in large bursts. For instance, when you save a file in your editor, it probably makes a tempfile and then copies it over the target, leading to several different changes. Reflex hides this from you by batching some changes together.

Simple enough.

Now let’s look at the implementation:

func (r *Reflex) batch(out chan<- string, in <-chan string) {
    const silenceInterval = 300 * time.Millisecond
    for name := range in {
        r.backlog.Add(name)
        timer := time.NewTimer(silenceInterval)
    outer:
        for {
            select {
            case name := <-in:
                r.backlog.Add(name)
                if !timer.Stop() {
                        <-timer.C
                }
                timer.Reset(silenceInterval)
            case <-timer.C:
                for {
                    select {
                    case name := <-in:
                        r.backlog.Add(name)
                    case out <- r.backlog.Next():
                        if r.backlog.RemoveOne() {
                            break outer
                        }
                    }
                }
            }
        }
    }
}

Yikes.

That block comes with a comment on top with these notes:

batch receives file notification events and batches them up. It’s a bit tricky, but here’s what it accomplishes:

When we initially get a message, wait a bit and batch messages before trying to send anything. This is because the file events come in bursts.

Once it’s time to send, don’t do it until the out channel is unblocked. In the meantime, keep batching. When we’ve sent off all the batched messages, go back to the beginning.

When I came across this piece of code, I had to re-read it several times, and cross-reference that to the comment, and read through a couple of pages of documentation before I could say I understood it. And now, to write this post, I had to spend a good couple of minutes more staring at it before I could remember how it worked.

In short, this is waiting on file-system changes to come in from a channel, and it adds them to a backlog as they come in. It continues to do this until there is a break between the incoming changes of at least 300 milliseconds, and it then processes them.

The key here is the interaction between the time.Timer to keep track of that time window, and the incoming and outgoing channels, which in Go are blocking. The nested loops are there to make it so that, if an event comes in while some other part of the system is running, it will also get processed.

Porting it to Raku

As it turns out, porting code from Go to Raku is normally quite painless. Like Go, Raku has superb support for asynchronous programming, and helper classes like channels are built-in.

However, when it came to porting this bit of code I ran into a bit of a complication: Raku has no equivalent to time.Timer.

For this particular case, the Go timer was being used to schedule an event that would trigger in the future, unless it was re-scheduled to happen further in the future via the timer.Reset call.

Now, while Raku comes with classes to represent moments in time, as well as events in the future, these are not particularly well suited for this case. For one, even if you can schedule an event in the future (eg. with Promise.new: $delay) this promise will take place. Even if you are not paying attention to it, any code you schedule will run.

While it does look like there is some intention of implementing cancellations for Promises in the future, we’re not there yet. So we’ve got to make do with what we have.

Luckily, Raku is a language that is built on the notion that you should have enough rope to shoot yourself in the foot, and that it does. Enter Timer::Breakable, which implements something alike to breakable promises, and my very own Timer::Stopwatch, which uses them to implement basically the same idea as the Go timer.

The resulting code

With that set aside, let’s look at how that Go code compares to the equivalent code in Raku:

method run ( --> Promise ) {
    use Timer::Stopwatch;
    my Timer::Stopwatch $timer .= new;

    start react {
        whenever $!supply {
            $timer.reset: 0.3;
            $!queue.add: .path;
        }
        whenever $timer { start self!process-queue }
    }
}

The beauty here is that, unlike in Go, I’m using a Supply which offers the same asynchronous guarantees as a channel with the benefit that reading and writing from it does not block, so the nested loops to make sure that we catch events that happen while we are waiting are gone.

Trade-offs, trade-offs

It’s possible that there are some subtle differences between the two versions of this code, but they are functionally identical, and I was happy to get rid of the added complexity to get code that I can understand at a glance.

It did highlight in my eyes one of the limitations in Raku, however: that the standard library, while huge in some respects, still lacks tools to do some of the things that you might argue are pretty basic.

Luckily, this kind of thing can be remedied by libraries and modules that extend the language, like the ones shown here. And as more of these real-world cases are explored, one can hope that cases like these, where we lack the tools to get the job done, become a rarity.

I’ll be happy to continue to contribute to that end.

“Hot reloading” is a feature made popular particularly by web frameworks wherein any change in the source code of the service will make the development version of the server reload.

More generally, however, this can be applied to anything that will trigger some command when a file on disk has changed. And this is precisely what reflex does. ↩

Typesetting ToBI

2020-11-15T00:00:00+00:00

A lifetime ago,¹ back when I was still convinced I was on track to become a speech researcher, I spent a considerable amount of time dealing with ToBI tone markers.

In case you need a refresher, or come from an entirely different walk of life, ToBI stands for “Tones and Break Indices”, and is a standard for speech researchers to mark intonation, or prosody. This sort of annotation is pretty essential for research, because it allows researchers to record these data, share them with their colleagues, and process it as a whole.

This last point is important, and is one of the reasons why the standard has been this successful: the transcription uses only ASCII characters that were and continue to be trivial to input, and easy to read by machines. This is in fact something that was identified as a strength in the original publication that described it:

[We believe ToBI will become a standard for prosodic transcription because, among other reasons,] it defines ASCII formats for machine-readable representations of the transcriptions which are in principle independent of the pitch extraction and signal available to the researcher.

In brief, ToBI tone markers are largely made of H and L characters, some of which might be marked as primary stresses with a * character, and sometimes combined with + to make up complex tones.

Some basic ToBI tones

The problem

The problem with these tones, as I imagine some of you might have immediately noticed, is that they are butt-ugly, and have absolutely zero chance of looking good in a paragraph of text.

The typical typesetting of ToBI tones in context

And here is where the rubber meets the road: we want tone markers that use only ASCII characters so they are machine readable to be able to process our data. But once they’ve been processed, and we want to report the results, we are going to be putting those tones markers in an entirely different context: one where humans, and not machines, are the primary readers, and where they will be surrounded by more of those pesky human-readable sentences.

I’d posit that using the standard machine-readable notation in prose is ultimately wrong.

Before anyone gets the wrong idea: none of this, including the example above, is an indictment on the quality of the academic work that typesets these markers this way. I’m pretty sure that most people in academia does this because that’s how they’ve always seen it in print, and because they have better things to do with their time than obsess over the spacing of the characters in their publications.²

Then again, when I got around to writing my dissertation, I was procrastinating pretty heavily, and a lot of it was pretty much exactly obsessing over kerning and trying to make sure anything I wrote would simultaneously make Bringhurst and Tufte proud.

Which is to say: there was no way I was going to just print that in my thesis.

My solution

So I came up with a different way to do it that seemed better in my eyes:

Typesetting single tones

The fundamental idea was twofold. First, use small-caps instead of the upper-case for H and the L to make the text blend into the surrounding text. Second, recognise that the * character was marking a property of the tone, which meant that anything that made this link easier to see should be in scope.

The result, as seen above, makes the markers have less vertical movement (meaning that they can be read from left to right with the eye largely following stable horizontal line), and allows each of them to occupy the same amount of horizontal space, which promotes the idea that the tones are the fundamental unit at play.

This was easily accomplished in LaTeX.³ First we define some basic components:

\newcommand{\high}{\textsc{h}}
\newcommand{\low}{\textsc{l}}
\newcommand{\ToBIminus}{-}
\newcommand{\ToBIplus}{+}

Which can then be combined to generate the tones shown above:

\newcommand{\Lstar}{\low\kern-0.465em\raisebox{0.15em}{*}\kern0.09em}
\newcommand{\Hstar}{\high\kern-0.485em\raisebox{0.15em}{*}\kern0.11em}

What about compound tones

Typesetting compound tones

For compound tones, it’s just a matter of defining the rest of the relevant combinations:

\newcommand{\LHstar}{\mbox{\low\ToBIplus\Hstar}}
\newcommand{\LstarH}{\mbox{\Lstar\ToBIplus\high}}
\newcommand{\HLstar}{\mbox{\high\ToBIplus\Lstar}}
\newcommand{\HstarL}{\mbox{\Hstar\ToBIplus\low}}

I am pretty sure that the values used by me to do this are subject to a lot of improvement. And I’m sure also that, as much as I’m interested in typography, others are probably better than me at evaluating where I might have gone too far or not far enough.

Still, I’m very happy with the result as it looks in the document I was producing, and if I can share this and expose other people to the possibility of these tones not making your document look like someone coughed a bunch of letters on your page, I’ll consider my job here done. And I’d say that the results speak for themselves:

My alternative ToBI tones in context

The best part of this? Because of the characters I used, the end results are still completely searchable using a standard PDF reader, which really highlights the fact that the storage of machine-readable information, and its display are different things, and we should not sacrifice the latter for the former.

References

Silverman, K. et al. (1992) ToBI: A Standard for Labeling English Prosody In: The Second International Conference on Spoken Language Processing, ICSLP 1992, Banff, Alberta, Canada, October 13-16, 1992

Or at least what feels like a lifetime ago… ↩
After all, that strongly worded response to Reviewer #2’s passive-aggressive remarks is not going to write itself. ↩
In hindsight, my getting into LaTeX in order to write my dissertation, and the amount of time I spent not actually writing my dissertation, seems like a pretty clear symptom that I was dissatisfied with a purely academic career. ↩

Introducing HTTP::Tiny

2020-11-08T00:00:00+00:00

For the past couple of weeks I’ve been working on a minimal HTTP client library for Raku, and it has now been released. Say hello to HTTP::Tiny!

This might feel like a bit of an unnecessary thing to do, since Raku already has several usable HTTP clients.

Why?

Initially, the project started as an excuse for me to learn about using Raku in the real world. After the project was chosen, a secondary goal was to learn about the HTTP protocol.¹

That second part was not as courageous as you might think because, in case you are not coming from the Perl world, there is already a Perl library named HTTP::Tiny which I was going to use as a model. So rather than implementing the HTTP spec from scratch, the idea was to port an existing and well tested library.

The more I worked on it, however, the more I realised that this might actually be useful to have in the Raku world as well, and these are reflected in the abstract of the original Perl library, which I’ve inherited for the Raku one as well:

A small, simple, correct HTTP/1.1 client

Small

The “small” in the abstract is related to the ::Tiny in the name, which comes from an idea first proposed by Adam Kennedy for CPAN modules: to write “tiny” versions of useful modules implemented in as little code as possible, with “no non-core dependencies”.

This reference to a “core” makes more sense in Perl because that language has a clearly defined distribution of “core” modules that can be considered “part of the language”. It is not common, and indeed difficult, to have a working version of Perl that does not have the core modules installed.

And although Raku also has Rakudo Star, which works as a sort of de-facto core distribution, this is by no means a “core” set, but more a recommended set of modules to accomplish common tasks.

And this is important, because particularly when trying to bring Raku into environments where Raku is not already established, any external dependency is a barrier.

My challenge when writing the Raku version of HTTP::Tiny was to use only language built-ins. To write a minimal HTTP client that was usable, and had no dependencies.

As for “as little code as possible”, HTTP::Tiny has just over 1000 lines of code. Deciding if that’s too much is left as an exercise for the beholder.

Simple

Simplicity in software can be understood in a number of different ways, and some of them are related to the point above: the more code you use, the more complexity it hides. In that sense, HTTP::Tiny is as simple as I’ve been able to make it.

But a different way to understand “simple” is in the simplicity of use, which relates to the interface that is available for the user.

At its core, HTTP::Tiny has one method: request. Every other method available (with the exception of the can-ssl utility one) calls this one method to perform the core of its work. The signature is shown below:

method request (
    Str $method,
    Str $url,
       :%headers,
       :$content,
       :&data-callback,
       :&trailer-callback,
) returns Hash

The only required parameters are the method and the URL, with the remaining ones existing to support the multiple different kinds of requests available.

Every effort was made to keep this interface small and predictable, but ultimately, whether this interface is simple or not will also be up to the users of this library. To see what this means, the documentation is probably the best place to check, and a cookbook-style document with examples of common requests is also available to see what common requests look like.

Correct

More importantly, HTTP::Tiny aims to be a correct implementation of the HTTP/1.1 transport, and to my surprise, this might be the area where this library might make the most significant contribution.

HTTP::Tiny of course supports the most common use cases, including form and file uploads. But it also supports streaming requests, multipart ranged responses, and both HTTP and HTTPS proxied connections. It also handles unexpected 1XX responses, request redirection, and trailers in chunked responses.²

Needless to say, it might strive for correctness, but it is far from perfect. A full list of known limitations is available in the documentation, and that list will undoubtedly grow to include the limitations that will only become apparent with use.

But for that it needs users, so please consider giving HTTP::Tiny a shot.

Yes, the P stands for “protocol” already. Thanks. ↩
Some of these things might seem obvious, but I found that support for all of them is unexpectedly hard to come by in the existing libraries. The ones that were easily resolved became upstream contributions (because we’re not competitors but colleagues), but not all of them have been resolved. ↩

Dissecting a Raku snippet

2020-10-26T00:00:00+00:00

A couple of days ago, while prototyping the solution to a problem I had, I wrote a bit of code that brought immense joy to me. It was the kind of code that makes you giggle each time you see it run just because of how clever and elegant the solution seems, and because of the excitement of proving that, against your expectations, it actually works.

Wanting to share the excitement, I shared it online. The snippet looked like this:

sub foo ( $a, $b ) { $a ~ $b }

my $wh;

$wh = &foo.wrap: {
    LEAVE &foo.unwrap: $wh;
    callwith( $^b, $^a );
}

say foo( 1, 2 ) for 1 .. 3;

This is very dense.

And a bit cryptic.

And hella fun.

So let’s break it down. To do this, we’ll start from the end, and work our way back to the beginning.

The results

When the code above gets executed, it prints out the following lines:

21
12
12

This is the result of the final line:

say foo( 1, 2 ) for 1 .. 3;

which prints the result of calling foo( 1, 2 ) three times. This line will be familiar to Perl programmers because Raku supports “statement modifiers”, which it inherited from Perl: the for at the end of the statement modifies the statement that comes before, and executes it once for each element in the 1 .. 3 range (which is actually a Range object).

But if we are calling foo( 1, 2 ) three times, then why does it output 21 the first time we call it, and 12 the rest of the times?

Wrapping code

That’s because of the previous block:

my $wh;

$wh = &foo.wrap: {
    LEAVE &foo.unwrap: $wh;
    callwith( $^b, $^a );
}

This bit of code is where the core of the magic happens. ✨

The first bit that matters is the call to wrap This is called on the foo subroutine object (which is why we reference it with the & sigil, otherwise we might call the function rather than get a reference to it).¹

The wrap method allows us to attach a block of code to foo, so that calling that function will execute the block of code instead.

Re-dispatch

Inside the block we are using to wrap foo we use callwith to be able to call the wrapped subroutine. We could use a number of different alternatives depending on whether we wanted to get a return value and whether we wanted to modify the arguments the underlying function gets called with.

In this case, callwith allows us to modify the arguments, but it never returns. And we modify the arguments by calling it like callwith( $^b, $^a ).

But where did these variables come from?

Self-declared positionals

They are self-declared positional arguments.

When a block does not define an explicit list of arguments, any variables used within its scope that use the ^ twigil (= any variable with a ^ after the sigil) declares an implicit positional argument. Each positional the block receives will get assigned to these variables in alphabetical order.

In the example above, then, the call callwith( $^b, $^a ) will re-dispatch to the function we are wrapping, with the first and second arguments switched. Since the original subroutine concatenates both arguments (with the ~ operator), this explains the first line of output: we call foo( 1, 2 ) and this re-dispatches to foo( 2, 1 ) which results in the 21 line.

Progress!

Now for the truly magic bit.

Self-unwrapping wrappers

Before actually re-dispatching to the original code, we have another line:

LEAVE &foo.unwrap: $wh;

This line uses a LEAVE phaser to register a statement to be called when execution leaves the current block. So after switching the arguments and re-dispatching to the original subroutine, execution leaves the block and this statement finally gets executed.

When it does, it calls the unwrap method on foo to remove the wrapping code (it does this by passing in the wrapping handle that was stored in the $wh variable).

Wrap-up

And that’s it! The snippet defines a foo subroutine that takes two arguments and concatenates them. It then wraps it into a block of code that swaps the arguments before calling the original subroutine, and then removes itself as if it was never there.

This explains the output: the first time we call foo( 1, 2 ) the arguments get swapped and we print 21, but on subsequent calls there is no wrapping code anymore, so we get lines like 12.

What a mouthful! But all of that in seven lines of code. Raku can really pack a punch. ✊

That’s nice, but what’s the real use?

Incidentally, this was something that I was prototyping to solve a real problem: I had a callback that would get called repeatedly, and I wanted to hook into the first time that callback was executed so I could run a preliminary check. But once the check had passed, there was no point in doing it every time. Unwrap to the rescue.

Incidentally, this makes use of another useful Raku feature: in Raku everything is an object. This is what allows us to, for example, seamlessly call methods on subroutines. ↩

Sharing command line parameters in Raku

2020-09-28T00:00:00+00:00

Raku is full of neat little tricks and gems that can make the programmer’s life simpler (or at least a whole more entertaining). And as a language that rewards mastery, it’s not a surprise that these can combine into sophisticated solutions to common problems.

This post will illustrate this point with one such example: sharing parameters in a command line application. But first, we’ll need to cover some of the neat little building blocks that allow for this.

Command line interfaces

Raku has built-in support for writing command line interfaces. If a file has a MAIN subroutine, it will called automatically when the file is directly executed. And more interestingly will use that subroutine’s signature to automatically parse the command line arguments.

That means that an executable file named cli that looks like this:

#!/usr/bin/env raku

sub MAIN ( Str $command, Str $path, Bool :$debug ) {
    note "Working on $path" if $debug;

    given $command {
        when 'grep' {
            .say for $path.IO.lines.grep: /raku/;
        }
        when 'count' {
            say "$_ has { .IO.lines.elems } lines" given $path;
        }
        default {
            say "Unknown command: $command";
            say $*USAGE;
        }
    }
}

will result in the following output:

$ ./cli grep ./cli
#!/usr/bin/env raku
            .say for $path.IO.lines.grep: /raku/;
$ ./cli count ./cli
/home/user/cli has 17 lines
$ ./cli
Usage:
  /home/user/cli [--debug] <command> <path>
$ ./cli reverse ./cli
Unknown command: reverse
Usage:
  /home/user/cli [--debug] <command> <path>

Multiple dispatch

This is made possible by the signature in the MAIN subroutine, which specifies what parameters that subroutine can take (and of course, which ones it cannot, like that --fake-option one I used).

But the compiler not only can determine whether a function call is valid or not based on its signature, it can also decide which function to call based the arguments used.

We can use this to expand our application:

#!/usr/bin/env raku

multi sub MAIN ( 'grep', Str $path, Bool :$debug ) {
    note "Working on $path" if $debug;
    .say for $path.IO.lines.grep: /raku/;
}

multi sub MAIN ( 'count', Str $path, Bool :$debug ) {
    note "Working on $path" if $debug;
    say "$_ has { .IO.lines.elems } lines" given $path;
}

multi sub MAIN (
    Str $command, Str $path, Bool :$debug,
) is hidden-from-USAGE {
    note "Working on $path" if $debug;
    say "Unknown command: $command";
    say $*USAGE;
}

which will keep the behaviour for all the above calls, but will now give us a different help message:¹

$ ./cli --help
Usage:
  /home/user/cli [--debug] grep <path>
  /home/user/cli [--debug] count <path>

This new version keeps different behaviours separate, which makes implementing a new command a lot simpler. But it also has some unwanted issues, like forcing us to repeat the shared parameters on each entry, which can get pretty unwieldy with more complex applications.

Luckily, we have another ace up our sleeves.

Subroutine prototypes

When declaring multi subroutines we can declare common behaviours by declaring a proto. This can be useful when we want to make sure that we don’t accidentally make the $path into an Int, for example. But we can also use it for our application:²

#!/usr/bin/env raku

proto sub MAIN ( $, Str $path, Bool :$debug, | ) {
    note "Working on $path" if $debug;
    {*}
}

multi sub MAIN ( 'grep', Str $path, *% ) {
    .say for $path.IO.lines.grep: /raku/;
}

multi sub MAIN ( 'count', Str $path, *% ) {
    say "$_ has { .IO.lines.elems } lines" given $path;
}

multi sub MAIN ( Str $command, Str $path, *% ) is hidden-from-USAGE {
    say "Unknown command: $command";
    say $*USAGE;
}

The proto defines a common portion of MAIN in which we can place any shared behaviours (including for example any initialisation we may want to do with our now global parameters). Unfortunately, it does mean that our automatically generated usage message gets slightly less nice:³

$ ./cli --help
Usage:
  /home/user/cli grep <path>
  /home/user/cli count <path>

TIMTOWTDI

As always, there is more than one way to do it,⁴ and it’s up to you as the developer to decide which one is the most appropriate for your needs.

I’m just glad that Raku has all these tools at my disposal, and gives me enough ~~rope~~ room to grow.

Note how we can use the is-hidden-from-USAGE trait to tell Raku to ignore a particular subroutine when generating the usage message. This is another of those nice little tricks Raku is littered with. ↩
The | at the end of the proto declaration declares a capture parameter, and in this case has the effect of saying that the specific implementations of this proto may have other parameters (like sub-command-specific options, for example).

The *% I used in the MAIN subs is a slurpy parameter which slurps in any non-specified parameters to lets that signature match even if I don’t bother to specify a :$debug named parameter, for example. I could have used a | instead, but that has some effects on the usage message that I wanted to avoid. ↩
Fortunately, there are ways in which we can solve this by either expanding the built-in message available in $*USAGE or replacing it entirely by defining our own USAGE subroutine. ↩
You could also us unit to define file-wide MAINs for each separete command, and leave the top level file to dispatch to the right file! ↩