6 Qi Macros

Qi may be extended in much the same way as Racket – using macros. Qi macros are indistinguishable from built-in Qi forms during the macro expansion phase, just as user-defined Racket macros are indistinguishable from macros that are part of the Racket language. This allows us to have the same syntactic freedom with Qi as we are used to with Racket, from being able to add new language features to implementing entire new languages in Qi.

This "first class" macro extensibility of Qi follows the general approach described in Macros for Domain-Specific Languages (Ballantyne et. al.).

6.1 Defining Macros

These Qi macro definition forms mirror the corresponding forms for defining Racket macros. Note that if you use syntax patterns or syntax classes in your macro definition, or if you are manipulating syntax objects directly, you may need to (require (for-syntax syntax/parse racket/base)), just as you would in writing similar Racket macros.

However, if the binding you define in this way collides with an identifier in Racket (for instance, if you call it cond), it would override the Racket version (unlike using define-qi-syntax-rule or define-qi-syntax-parser where they exist in a distinct binding space). To avoid this, use define-qi-syntax instead of define-syntax.

Note that the type constructor qi-macro is all that is publicly exported for this struct type (and only in the syntax phase), since the details of its implementation are considered internal to the Qi library.

Note that macros defined using this form must wrap the resulting syntax parser as a qi-macro.

6.2 Using Macros

Note: This section is about using Qi macros. If you are looking for information on using macros of another language (such as Racket or another DSL) together with Qi, see Using Racket Macros as Flows.

Qi macros are bindings just like Racket macros. In order to use them, simply define them, and if necessary, provide, and require them in the relevant modules, with the proviso below regarding "binding spaces." Once defined and in scope, Qi macros are indistinguishable from built-in Qi forms, and may be used in any flow expression just like the built-in forms.

In order to ensure that Qi macros are only usable within a Qi context and do not interfere with Racket macros that may happen to share the same name, Qi macros are defined so that they exist in their own binding space. This means that you must use the provide subform for-space in order to make Qi macros available for use in other modules. They may be required in the same way as any other bindings, however, i.e. indicating for-space with require is not necessary.

To illustrate, the providing module would resemble this:

(provide (for-space qi pare))

(define-qi-syntax-rule (pare car-flo cdr-flo)

(group 1 car-flo cdr-flo))

And assuming the module defining the Qi macro pare is called mac-module, then any of the following (among other variations) would import it into scope.

(require mac-module)

(require (only-in mac-module pare))

6.2.1 Racket Version Compatibility

As binding spaces were added to Racket in version 8.3, older versions of Racket will not be able to use the macros described here, but can still use the legacy qi:-prefixed macros.

6.3 Adding New Language Features

When you consider that Racket’s class-based object system for object-oriented programming is implemented with Racket macros in terms of the underlying struct type system, it gives you some idea of the extent to which macros enable the addition of new language features, both great and small. In this section we’ll look at a few examples of what Qi macros can do.

6.3.1 Write Yourself a Maybe Monad for Great Good

In functional languages such as Haskell, a popular way to do (or rather avoid) exception handling is to use the Maybe monad. Qi doesn’t include monads out of the box yet, but you could implement a version of the Maybe monad yourself by using macros. But first, let’s quickly review why you might want to in the first place.

Earlier, we drew a distinction between two paradigms employed in programming languages: one organized around the flow of control and another organized around the flow of data. A way to manage possible errors in code along the lines of the former ("control") paradigm is to handle exceptions that may occur at each stage, and take appropriate action – for instance, abort the remainder of the computation. A second way to handle errors, more along the lines of the "flow of data" paradigm, is for the "failing" computation to simply produce a sentinel value that signifies an error, so that the sequence of operations does not actually fail but merely generates and propagates a value signifying failure. The trick is, how to do this in such a way that downstream computations are aware of the sentinel error value so that they don’t attempt to perform computations on it that they might do on a "normal" value? This is where the Maybe monad comes in.

We want to thread values through a number of flows, and if any of those flows raises an exception, we’d like the entire flow to generate no values. Typically, we compose flows in series by using the ~> form. For flows that may fail, we need a similar form, but one that (1) handles failure of a particular flow by producing no values, and (2) composes flows so that the entire flow fails (i.e. produces no values) if any component fails.

Let’s write each of these in turn and then put them together.

For the first, we write a macro that wraps any Qi flow with the exception handling logic to generate no values.

(define-qi-syntax-rule (get flo)

(try flo [exn? ⏚]))

This uses Qi’s try form to catch any exceptions raised during execution of the flow, handling them by simply generating no values as the result.

Now for the second part, in the binary case of two flows f and g, either of which may fail to produce values, the composition could be defined as:

(define-qi-syntax-rule (mcomp f g)

(~> f (when live? g)))

... which only feeds the output of the first flow to the second if there is any. Now, let’s put these together to write our failure-aware threading form, that is to say, our Maybe monad.

(define-qi-syntax-parser maybe~>

[(_ flo)

#'(get flo)]

[(_ flo1 flo ...)

#'(mcomp (get flo1) (maybe~> flo ...))])

This form is just like ~>, except that it does two additional things: (1) It wraps each component flow with the get macro so that an exception would result in the flow generating no values, and (2) it checks whether there are values flowing at all before attempting to invoke the next flow on the outputs. Thus, if there is a failure at any point, the entire rest of the computation is short-circuited.

Note that short-circuiting isn’t essential here as long as our composition ensures that the result is still well-defined if downstream flow components are invoked with no values upon failure of an upstream component (and they should produce no values in this case). But as we already know the result at the first point of failure, it is more performant to avoid invoking subsequent flows at all rather than rely on repeated composition in a computation destined to produce no values, and indeed, most Maybe implementations do short-circuit in this manner.

((☯ (maybe~> (/ 2) sqr add1)) 10)

((☯ (maybe~> (/ 0) sqr add1)) 10)

And there you have it, you’ve implemented the Maybe monad in about nine lines of Qi macros.

6.3.2 Translating Foreign Macros

Qi expects components of a flow to be flows, which at the lowest level are functions. This means that Qi cannot naively be used with forms from the host language (or another DSL) that are macros. If we didn’t have define-qi-foreign-syntaxes to register such "foreign-language macros" with Qi in a convenient way, we could still implement this feature ourselves, by writing corresponding Qi macros to wrap the foreign macros. The following example demonstrates how this might work.

In Converting a Macro to a Flow, we learned that Racket macros could be used from Qi by employing esc and wrapping the foreign macro invocation in a lambda. To avoid doing this manually each time, we could write a Qi macro to make this syntactic transformation invisible. For instance:

Note that the Qi macros can have the same name as the Racket macros since they exist in different binding spaces and therefore don’t interfere with one another.

Of course, writing Qi macros for such cases in practice is unnecessary as there is define-qi-foreign-syntaxes instead, which does this for you and in a robust and generally applicable way.

6.4 Writing Languages in Qi

Just as Racket macros allow us to write new languages in Racket, Qi macros allow us to write new languages in Qi.

You may prefer to use Qi as your starting point if your language deals with the flow of data, or if the semantics of the language are more easily expressed in Qi than in Racket. By starting from Qi, you inherit access to all of Qi’s forms, extensions, and tools that have been designed with the flow of data in mind – so you can focus on the specifics of your domain rather than the generalities of data flow.

In general, macros that define new languages are called interface macros, since they form the interface between two languages. Languages fall into two classes depending on their use of interface macros. We’ll learn about these two classes and then go over some examples to get a sense for when each type of language is called for.

6.4.1 Embedded Languages

One class of language has as many interface macros as there are forms in the language, so that the language seamlessly extends the host language. Such languages are called embedded languages or embedded DSLs. Examples of embedded languages in the Racket ecosystem include Deta, Sawzall, Racket’s built-in contract DSL, Social Contract, and Megaparsack.

Embedded languages implicitly inherit the semantics of the host language (but may define and employ custom semantics, even predominantly). With Qi as the host language, this means that such languages are inherently flow-oriented, and could range from general-purpose "dialects" of Qi to specialized DSLs. They are perhaps the most common type of language one might write in Qi.

If your language would employ flows in a general way but with specialized data structures or idioms, then it may be a good candidate for implementation as an embedded Qi DSL.

If there is an existing such language already implemented in Racket that you’d like to treat as a Qi DSL, you can embed it into Qi by using define-qi-foreign-syntaxes, but note that this "extrinsic" embedding would not benefit from any flow optimizations that may eventually be part of the Qi compiler, and incurs some administrative overhead.

6.4.2 Hosted Languages

It is also possible to implement your language as a single macro or a small set of mutually reliant macros, with the bulk of the forms of the language specified as expansion rules within these macros. Such a language is called a hosted language or hosted DSL, and each of the interface macros it is made up of could be considered to be hosted sublanguages. Examples of hosted languages include Racket’s match, and Qi itself, and typically (as in these examples) they are defined via a single interface macro containing all of the rules of the language.

The advantage of writing a hosted DSL, in general, is that by introducing a level of indirection between your code and the host-language (e.g. Racket or Qi) expander, you gain access to a distinct namespace that does not interfere with the names in the host language, allowing you greater syntactic freedom (for instance, to name your forms and and if, which would otherwise collide with forms of the same name in the host language). In addition, you gain control over the expansion process, allowing you to, for instance, add a custom compiler to optimize the expanded forms of your language before host language expansion takes over.

If you are interested in writing a hosted language that you’d like to use from within Qi, there are two options. You could either write the language as a Qi macro, or as a Racket macro and leverage it via Qi’s esc. In the latter case, you could even write a Qi "bridge" macro that transparently employs esc. These two options are functionally equivalent, but if your language is data-oriented it may make more sense for it to compile to Qi so that it can leverage any flow optimizations that may eventually be part of the Qi compiler.

6.4.3 Embedding a Hosted Language

You can always embed a hosted language into the host language by implementing a set of macros corresponding to each form of the language. For languages that are large enough, this may be the best option to gain the advantages of a hosted language while also retaining the convenience of an embedded one for special cases. For instance, for a small embedded version of Qi, you could do:

(define-syntax-parse-rule (~> (arg ...) flo ...)

(on (arg ...) (~> flo ...)))

(define-syntax-parse-rule (>< (arg ...) flo)

(on (arg ...) (>< flo)))

(define-syntax-parse-rule (-< (arg ...) flo ...)

(on (arg ...) (-< flo ...)))

(define-syntax-parse-rule (== (arg ...) flo ...)

(on (arg ...) (== flo ...)))

And this would allow you to use Qi forms directly in Racket – indeed, the forms in the Language Interface are such embeddings of Qi into Racket. The same approach would also work to embed a hosted DSL into Qi, whether that DSL is hosted on Qi or Racket.

6.4.3.1 Exercise: Pattern Matching

Let’s add some pattern matching to Qi by embedding Racket’s pattern matching language into Qi, using this approach.

First, the simplest possible embedding of match is to just write a Qi macro corresponding to the Racket macro.

(define-qi-syntax-rule (match [pats body] ...)

(esc (λ args

(match/values (apply values args)

[pats (apply (flow body) args)]

...))))

This converts the foreign macro to a flow in the usual way, i.e. by wrapping it in a lambda and using it via esc, as discussed earlier. Note that it expects the body of the match clauses to be Qi rather than Racket, and any identifiers bound by pattern matching would be in scope in these flows since that is what match does. Let’s use it:

(~> (5 (list 1 2 3))

(match

[(n (list a b c)) (gen n (+ a b c))]

[(n (cons v vs)) 'something-else]))

This is great, but in practice we are often interested in using pattern matching just for destructuring the input, and already know the pattern it is going to match. It would be nice to have a more convenient form to use in such cases. We can do this by writing a second macro to embed this narrower functionality into Qi.

(define-qi-syntax-rule (pat pat-clause body ...)

(match [pat-clause body ...]))

And now:

(~> (5 (list 1 2 3))

(pat (n (list a b c)) (gen n (+ a b c))))

(~> (1 2 3) (pat (_ (? number?) x) +))

Similarly, we could write more such embeddings to simplify other common cases, such as matching against a single input value. Thus, the features provided by one language may be embedded into another language.