January 2, 2023
Matchertext: an escape route from language-embedding hell?
We often need to embed strings written in one programming language
into code written in another.
we routinely embed
within shell scripts
or string literals in C-like languages.
routinely contain embedded
We often need to embed one
such as to formulate a query to a Web service
or otherwise refers to other websites.
Whenever we embed strings, however,
we traditionally encounter numerous flavors
of a general problem I call language-embedding hell
The host language must have a way to find the end of the embedded string,
which implies constraining the embedded string's syntax.
Quoted string literals in C-like languages, for example,
require any quote or backslash characters within the embedded string to be
Escaping increases the length of the embedded string,
is annoying and error-prone when we must do it manually,
and yields critical security flaws
when automated incorrectly.
The ideal of verbatim cross-language embedding
Imagine for the moment an ideal world in which all programming languages were
within each other.
In this fictional world,
valid string in any
could be embedded within the same or any other language,
without any escaping, obfuscation, or length expansion.
To embed a string, you ideally would just “copy-and-paste” it
into any suitable context,
such as a quoted string in the host language.
Wouldn't that be nice?
Our tradition of expressing programming languages as
arbitrary “flat” sequences of
unfortunately renders this ideal of verbatim embedding
impossible in its pure form.
To find the end of an embedded plain text string unambiguously,
a host language must either escape or outright forbid
certain character sequences (e.g., the designated closing quote or end tag),
or else must prefix each embedded string with its precise length.
The latter practice works fine in machine-to-machine protocols,
but would be unbearably tedious and error-prone in a language
intended for humans to write.
Matchertext: a pragmatic path out of embedding hell
What if we could design and gradually deploy
a not-too-painful “upgrade” to our traditional plain text foundation
for programming languages,
which would eventually allow us to achieve a reasonable approximation
to the above ideal of verbatim, “copy-and-paste” embedding?
This goal is the essence of matchertext
an idea detailed in
a new preprint
The pragmatic essence of the matchertext idea is simple.
First, we define six particular ASCII characters as matchers
namely the open and close parentheses
the square brackets
and the curly braces
We call these characters matchers
because their traditional, already-ubiquitous purpose
is to be used in matching pairs to surround and delimit other text.
Now we define matchertext
as any plain text string
conforming to one additional rule or “syntactic discipline”:
namely that matchers must match
throughout any matchertext string, without exception.
Nesting is allowed, but must use corresponding matchers.
For example, the string ‘
’ is valid matchertext,
but strings like ‘
’ are not matchertext.
Most of today's programming languages are already “matchertext-compatible”,
in the sense that many (perhaps most) of the strings we already tend to write,
and might want to embed,
will already happen to conform to this syntactic discipline.
Strings in today's languages that aren't matchertext —
i.e., strings containing unmatched matchers —
can usually be rewritten with a bit of effort
to conform to this discipline.
Why should we bother?
Because by following a few simple language-neutral rules,
any “matchertext-aware” language can host embedded strings
written in any other
without escaping, expansion, or other obfuscation of embedded strings.
Embedded matchertext strings can nest to any depth,
and there are no disallowed character sequences or other constraints,
the basic rule that matchers must match.
As long as you know (or your tooling can check) that a string is matchertext,
you can just “copy-and-paste” it into any matchertext embedding context,
with no escaping or other fuss, as in the ideal world above.
While the matchertext idea was motivated by the embedding problem,
I have already noticed at least one side benefit —
or “icing on the cake” —
to adopting this discipline.
When writing matchertext,
I find it convenient that highlighting text editors like
no longer ever
about which open matcher is associated with which close matcher,
even when the editor has no specific knowledge of the language in question.
Matchertext is merely about strictly
enforce an existing discipline
that we already mostly
and rigorously enforcing a useful programming discipline
often has unanticipated benefits of this kind.
Incremental language extensions for matchertext
Of course, matchertext will never be useful
unless at least some language designers and developers
are willing to “take the plunge” and try implementing and using it.
explores what this could mean in the context of various popular languages,
including C-like languages,
SGML-derived languages such as HTML and XML,
and embedding-oriented “little languages”
like regular expressions and URIs.
In general, today's languages can be evolved gracefully and incrementally
to become more “matchertext-aware”,
via two main classes of language extensions:
and embedding extensions
Both classes of extensions can readily be designed and deployed
so as to preserve backward compatibility with existing code.
Hosting extensions make matchertext useful
by allowing the verbatim embedding of any matchertext string
in a suitable host-language construct such as a string literal.
In C-like languages, for example,
I suggest new backslash-escape sequences like
where matchertext is an arbitrary matchertext string.
SGML-derived markup languages might be enhanced
with backward-compatible syntax extensions such as
<name attributes [matchertext]>
for a tag containing arbitrary matchertext content.
Embedding extensions reduce the (hopefully already-modest) pain
of writing matchertext-compliant code,
particularly by providing new alternative ways to escape unmatched matchers.
A statement like
printf("[") is invalid matchertext,
and simple backslash-escaping as in
printf("\[") does not work
because the matchertext discipline
is oblivious to language-specific escaping rules.
Numeric escaping like
printf("\x5B") solves the problem
and already works fine,
but is a bit cumbersome.
Thus, I suggest potential new backward-compatble escapes
that are more visual while being matchertext-compliant:
\c for unmatched open and close brackets
in C-like languages,
[>] for unmatched brackets
in regular expressions or SGML-style markup languages.
Practical experimentation with matchertext
As of this writing,
most of the work of implementing and experimenting with the matchertext idea
remains to be done.
To succeed it will need plenty of time (years at least, if not decades),
and will eventually require effort from interested people across
many programming languages.
The current preprint is intended to be only a modest initial starting point,
to be updated occasionally if and when interest materializes
and we gain further experience worth reflecting in the paper.
As a concrete initial experiment,
I have made the matchertext idea central to the design of
a more concise alternative to HTML or XML syntax.
MinML attempts to bring HTML-style syntax
closer to the convenience of Markdown and its many variants,
while preserving the full power and generality of HTML or XML.
MinML embodies the matchertext discipline
and uses it to handle nested “example code” constructs
more cleanly and sanely than in existing markup languages.
Since MinML is only a single matchertext-aware language,
it can by no means unlock the full potential of the matchertext idea alone.
I hope that others — maybe you —
will try proposing, implementing, and using the matchertext idea
in new or existing programming languages of your choice.
If you do,
then well-considered pull requests to the
matchertext paper source
and significant contributions to future versions of the paper
will be acknowledged appropriately.
Thanks for reading!