7 Field Guide

This section contains practical advice on using Qi. It includes recipes for doing various things, advice on gotchas, troubleshooting commonly encountered errors, and other tips you may find useful "in the field."

7.1 Writing Flows

7.1.1 Start by Drawing a Circuit Diagram

Before you write a flow, consider drawing out a "circuit diagram" on paper. Start by drawing wires corresponding to the inputs, and then draw boxes for each transformation and trace out what happens to the outputs. This practice is the Qi equivalent of writing "pseudocode" with other languages, and is especially useful when writing complex flows entailing folds and loops. With practice, this can become second nature and can be a very helpful recourse.

7.1.2 Use Small Building Blocks

Decompose your flow into its smallest components, and name each so that they are independent flows. Qi flows, by virtue of being functions, are highly composable, and are, by the same token, eminently decomposable. This tends to make refactoring flows a much more reliable undertaking than it typically is in other languages.

7.1.3 Carry Your Toolbox

A journeyman of one’s craft – a woodworker, electrician, or a plumber, say – always goes to work with a trusty toolbox that contains the tools of the trade, some perhaps even of their own design. An electrician, for instance, may have a voltage tester, a multimeter, and a continuity tester in her toolbox. Although these are "debugging" tools, they aren’t just for identifying bugs – by providing rapid feedback, they enable her to explore and find creative solutions quickly and reliably. It’s the same with Qi. Learn to use the debugging tools, and use them often.

7.1.4 Be Intentional About Effects

Qi encourages a style that avoids "accidental" effects. A flow should either be pure (that is, it should be free of "side effects" such as printing to the screen or writing to a file), or its entire purpose should be to fulfill a side effect. It is considered inadvisable to have a function with sane inputs and outputs (resembling a pure function) that also performs a side effect. It would be better to decouple the effect from the rest of your function (splitting it into smaller functions, as necessary) and perform the effect explicitly via the effect form, or otherwise escape from Qi using something like esc (note that function identifiers used in a flow context are implicitly escaped) in order to perform the effect. This will ensure that there are no surprises with regard to order of effects.

7.2 Debugging

There are three prominent debugging strategies which may be used independently or in tandem – side effects, probing, and fixtures.

7.2.1 Using Side Effects

The most lightweight way to debug your code is to use side effects, as this allows you to check values at various points in flows without affecting their functioning in any way. You can use this debugging approach always, even in functional Racket code that isn’t using Qi.

This approach involves using the side-effect form, effect (or ε) at a particular point (or several points) in the flow in order to see or manipulate the values there. To use it in general Racket code, just wrap the Racket code with ☯ to employ Qi there (and therefore side effects).

Side effects are a natural fit for debugging functional code in general, as the example below shows.

Example: Racket’s regexp-replace* function transforms a string into another based on a regex-based replacement rule. It accepts a pattern, a string, and a replacement rule (as a function), and then constructs the output string by parsing the input string and calling your replacement rule function each time there is a match to the pattern. With regexes, things don’t usually work until you’ve gone through multiple cycles of debugging, so in this case, in order to see what arguments are being supplied to your replacement rule function, you could simply add a side effect using Qi.

(regexp-replace* PATTERN

str

(☯ (ε (>< println) replace-rule)))

7.2.2 Using a Probe

Qi includes a "circuit tester" style debugger, which you can use to check the values at arbitrary points in the flow. It can be used even if the flow is raising an error – the tester can help you find the error. It offers similar functionality to debug but is specialized for functional debugging and Qi flows.

To use it, first wrap the entire expression invoking the flow with a probe form. Then, you can place a literal readout anywhere within the flow definition to cause the entire expression to evaluate to the values flowing at that point. This works even if your flow is defined elsewhere (even in another file) and only used at the invocation site by name.

Note that probe is a Racket (rather than Qi) form, and it must wrap a flow invocation rather than a flow definition. The readout, on the other hand, is a Qi expression and must be placed somewhere within the flow definition.

(~> (5) sqr (* 2) add1)

(probe (~> (5) readout sqr (* 2) add1))

(probe (~> (5) sqr readout (* 2) add1))

(probe (~> (5) sqr (* 2) readout add1))

(probe (~> (5) sqr (* 2) add1 readout))

(probe (~> (5) sqr (if (~> (> 20) readout) _ (* 2)) add1))

(define-flow my-flow

(~> sqr readout (* 3) add1))

(probe (my-flow 5))

When the flow you’d like to debug is a named flow that is not defined inline at the invocation site, you’ll need to take some extra steps to ensure that you can place a readout at the definition site even though the probe itself is placed at the invocation site.

To do this, either wrap the entire body of the definition, or a subflow in the definition, with qi:probe, or alternatively, use define-probed-flow instead of define-flow, which transparently does this for you. Now, you can place a (distinct) probe at the invocation site, as usual, and it will receive the readout that you indicate at the definition site.

(define-probed-flow name body) is equivalent to (define-flow name (qi:probe body)) or (define name (flow (qi:probe body))).

(define-probed-flow my-flow

(~> sqr readout (* 3) add1))

(probe (my-flow 5))

(define my-flow-too

(☯ (qi:probe (~> sqr readout (* 3) add1))))

(probe (my-flow-too 5))

7.2.3 Using Fixtures

The probe debugger allows you to check values at specific points in the flow, that is, essentially, the output of the upstream components at that point. It is sometimes also useful to fix the input to downstream components. In unit testing, fixing inputs to functions to test their behavior in a known environment is referred to as writing "fixtures." It’s the same idea.

The basic way to do it is to insert a gen form at the point of interest in the flow, as gen ignores its inputs and just produces whatever values you specify.

(~> (2) sqr (gen 9) add1 (* 2))

Methodical use of gen together with the probe debugger allows you to isolate bugs to specific sections of the flow, and then triangulate further using the same approach until you find the exact problem.

(probe (~> (3) (-< _ "5") (gen 3 5) + sqr readout (* 2) add1))

7.2.4 Common Errors and What They Mean

7.2.4.1 Expected Number of Values Not Received

; result arity mismatch;

;  expected number of values not received

;   expected: 1

;   received: 2

Meaning: A flow is either returning more or fewer values than the continuation of the flow is expecting. See Multiple Return Values for general information about this.

Common example: Attempting to assign the result of a multi-valued flow to a single variable. Use define-values instead of define here, or consider decomposing the flow into multiple flows that each return a single value.

Common example: Attempting to invoke a function with arguments produced by a multi-valued flow, something like (+ (~> ((range 10)) △)). Function application syntax in Racket expects a single argument in each argument position, and cannot receive them all from a flow in this way. You could use call-with-values to do it, but it is much simpler to just use Qi’s invocation syntax via a threading form, e.g. (~> ((range 10)) △ +).

Common example: Attempting to employ a Racket expression producing multiple values in an expression where the continuation expects one value, e.g. (~> () (gen (values 1 2 3)) +). Whether the expression is Racket or Qi, the number of values returned must be the number of values expected by the continuation – and typically, that’s one. In this example, you could simply write the values directly as separate expressions, each producing one value, e.g. (~> (1 2 3) +) or (~> () (gen 1 2 3) +). If you must use a single Racket expression to produce the values, then you could use (~> () (esc (λ _ (values 1 2 3))) +), instead.

Common example: Using the threading form ~> without wrapping the input arguments in parentheses. Remember that, unlike Racket’s usual threading macro, input arguments to Qi’s threading form must be wrapped in parentheses.

7.2.4.2 Wildcard Not Allowed as an Expression

;  _: wildcard not allowed as an expression

;   in: _

Meaning: _ is a valid Qi expression but an invalid Racket expression. Somewhere in the course of evaluation of your code, the interpreter received _ and was asked to evaluate it as a Racket expression. It doesn’t like this.

Common example: Trying to evaluate a Qi expression without having required the Qi library, so that the expression is evaluated as a Racket expression. (require qi) should do it.

Common example: Trying to use a template inside a nested application. For instance, (~> (1) (* 3 (+ _ 2))) is invalid because, within the (* ...) template, the language is Racket rather than Qi, and you can’t use a Qi template (i.e. (+ _ 2)) there. You might try sequencing the flow, something like (~> (1) (+ _ 2) (* 3)).

Common example: Trying to use a Racket macro (rather than a function), or a macro from another DSL, as a flow without first registering it via define-qi-foreign-syntaxes. In general, Qi expects flows to be functions unless otherwise explicitly signaled.

7.2.4.3 ~@ Not Allowed as an Expression

Meaning: Somewhere in the course of evaluation of your code, the interpreter received ~@ at runtime and was asked to evaluate it as an expression. But ~@ is not a valid Racket expression outside of syntax templates in the expansion phase, where it is used to work with sequences.

Common example: Using ~@ in a macro template without having (require (for-syntax racket/base)). As a result, ~@ is left unchanged through expansion instead of being treated as modulating how expansion is done. Then at runtime, the evaluator complains that it got ~@.

7.2.4.4 Bad Syntax

; lambda: bad syntax

;   in: lambda

Meaning: The Racket interpreter received syntax, in this case simply "lambda", that it considers to be invalid. Note that if it received something it didn’t know anything about, it would say "undefined" rather than "bad syntax." Bad syntax indicates known syntax used in an incorrect way.

Common example: A Racket expression has not been properly escaped within a Qi context. For instance, (flow (lambda (x) x)) is invalid because the wrapped expression is Racket rather than Qi. To fix this, use esc, as in (flow (esc (lambda (x) x))).

Common example: Trying to use a Racket macro (rather than a function), or a macro from another DSL, as a flow without first registering it via define-qi-foreign-syntaxes. In general, Qi expects flows to be functions unless otherwise explicitly signaled.

7.2.4.5 Use Does Not Match Pattern

; m: use does not match pattern: (m x y)

;   in: m

Meaning: A macro was used in a way that doesn’t match any declared syntax patterns.

Common example: Trying to use a Racket macro (rather than a function), or a macro from another DSL, as a flow without first registering it via define-qi-foreign-syntaxes. In general, Qi expects flows to be functions unless otherwise explicitly signaled.

7.2.4.6 Expected Identifier Not Starting With Character

; syntax-parser: expected identifier not starting with ~ character

;   at: ~optional

Meaning: A macro attempted to use a syntax pattern (which are commonly prefixed with the ~ character) but the parser thinks it’s an identifier and doesn’t like its name.

Common example: Syntax patterns are defined in the syntax/parse library. If you are using them in Qi macros, you will need to (require syntax/parse) at the appropriate phase level.

7.2.4.7 Identifier’s Binding is Ambiguous

; count: identifier's binding is ambiguous

;   in: count

Meaning: The expander attempted to resolve a reference and found more than one possible binding.

Common example: Having a Racket function in scope that has the same name as a Qi form, and attempting to use this unqualified identifier as a flow. To avoid the issue, rename the Racket function to something else, or use an explicit esc to indicate the Racket binding.

7.2.4.8 Not Defined as Syntax Class

; syntax-parser: not defined as syntax class

;   at: expr

Meaning: A macro attempted to use a syntax class that the expander doesn’t know about.

Common example: Common syntax classes are defined in the syntax/parse library. If you are using them in Qi macros, you will need to (require syntax/parse) at the appropriate phase level (e.g. (require (for-syntax syntax/parse)).

7.2.4.9 Too Many Ellipses in Template

; syntax: too many ellipses in template

;   at: ...

Meaning: A macro template attempted to refer to a syntax pattern matching many datums but the parser claims there is no such pattern.

Common example: Attempting to use a syntax pattern like ...+ without requiring the syntax/parse library. When writing Qi macros, you will often need (require (for-syntax syntax/parse)), the same as when writing Racket macros.

7.2.4.10 Syntax: Unbound Identifier

; syntax: unbound identifier;

; also, no #%app syntax transformer is bound in the transformer phase

Meaning: A macro attempted to manipulate a syntax object but the expander doesn’t know what that even is.

Common example: When writing Qi macros, you will often need (require (for-syntax racket/base)), the same as when writing Racket macros.

7.2.4.11 Undefined

; mac: undefined;

;  cannot reference an identifier before its definition

Meaning: An identifier appears unbound in your code.

Common example: Attempting to use a Qi macro in one module without providing it from the module where it is defined – note that Qi macros must be provided as (provide (for-space qi mac)). See Using Macros for more on this.

7.2.4.12 Compose: Contract Violation

; compose: contract violation

;   expected: procedure?

;   given: '()

Meaning: The interpreter attempted to compose functions but encountered a value that was not a function.

Common example: Attempting to use null as if it were a literal. Use '() or (gen null) instead. See null is Not a Literal for more.

7.2.4.13 List Arity Mismatch

; list?: arity mismatch;

;  the expected number of arguments does not match the given number

;   expected: 1

;   given: 2

Meaning: The predicate list? was invoked, and it is complaining that it expects a single input argument but received many.

Common example: Attempting to separate multiple input values using △. This form separates a single input list into component values, and will produce the above error if it receives more than one input.

7.2.4.14 Fancy-app Arity Mismatch

; .../fancy-app/main.rkt:28:19: arity mismatch;

;  the expected number of arguments does not match the given number

;   expected: 2

;   given: 1

Meaning: Qi uses fancy-app to handle partial application templates. Fancy-app is complaining that it was told to expect a certain number of arguments in the function invocation but received a different number.

Common example: You have a template like (f _ _) with a certain number of arguments indicated, but it was invoked with more or fewer arguments.

7.2.4.15 Application: Not a Procedure

; application: not a procedure;

;  expected a procedure that can be applied to arguments

;   given: 5

Meaning: The interpreter attempted to invoke a function but it found something other than a function (in this case, the value 5).

Common example: Attempting to partially apply a function with the same name as a Qi form, for instance, using a Named let with the name loop and attempting to partially apply the recursive invocation using Qi. Since loop is the name of a Qi form, in order to use the Racket function in scope, you need to either use esc or rename the function (in the case of loop, consider go instead) to avoid the collision.

7.2.5 Gotchas

7.2.5.1 null is Not a Literal

In Racket, null is another way to indicate the empty list '(), so that null and '() are typically interchangeable. But note that '() is a literal, while null is an identifier whose value is '(). Therefore, as Qi interprets literals as functions generating them, '() in Qi is treated as a flow that produces the value '(). On the other hand, as Qi expects identifiers to be function-valued, and as null isn’t a function, using it on its own is an error.

7.2.5.2 There’s No Escaping esc

If you have a function that returns another function that you’d like to use as a flow (e.g. perhaps parametrized by the first function over some argument), the usual way to do it is something like this:

(~> (3) (esc (get-f 1)))

But in an idle moment, this clever shortcut may tempt you:

(~> (3) ((get-f 1)))

That is, since Qi typically interprets parenthesized expressions as partial application templates, you might expect that this would pass the value 3 to the function resulting from (get-f 1). In fact, that isn’t what happens, and an error is raised instead. As there is only one datum within the outer pair of parentheses in ((get-f 1)), the usual interpretation as partial application would not typically be useful, so Qi opts to treat it as invalid syntax.

One way to dodge this is by using an explicit template:

(~> (3) ((get-f 1) _))

This works in most cases, but it has different semantics than the version using esc, as that version evaluates the escaped expression first to yield the flow that will be applied to inputs, while this one only evaluates the (up to that point, incomplete) expression when it is actually invoked with arguments. In the most common cases there will be no difference to the result, but if the flow is invoked multiple times (for instance, if it were first defined as (define-flow my-flow (☯ ((get-f 1) _)))), then the expression too would be evaluated multiple times, producing different functions each time. This may be computationally more expensive than using esc, and also, if either get-f or the function it produces is stateful in any way (for instance, if it is a closure or if there is any randomness involved), then this version would also produce different results than the esc version.

So in sum, it’s perhaps best to rely on esc in such cases to be as explicit as possible about what you mean, rather than rely on quirks of the implementation that are revealed at this boundary between two languages.

7.2.5.3 Mutable Values Defy the Laws of Flows

Qi is designed to model flows of immutable values. Using a mutable value in a flow is fine as long as you don’t mutate it, or if you mutate it only in a side effect. Otherwise, such values can produce unexpected behavior. For instance, consider the following flow involving a mutable vector:

(define vv (vector 1 2 3))

(~> (vv) (-< _ _) (== (vector-set! 0 5) (vector-set! 2 10)) vector-append)

You might expect this flow to produce a vector (vector 5 2 3 1 2 10), but in fact, it raises an error.

This is because vector-set! mutates the vector and produces not the result of the operation but (void), since the mutation that is the intent of the function has been performed and there is no need to produce a fresh value. Indeed, mutating interfaces typically produce (void), which is not a useful value that could be used in a flow. As a result, the output of the relay is two values, but not the ones we expect, but rather, values that are both (void). These are received by vector-append, which then complains that it expects vectors but was given (void).

Worse still, even though this computation raises an error, we find that the original vector that we started with, vv, has been mutated to (vector 5 2 10), since the same vector is operated on in both channels of the relay prior to the error being encountered.

So in general, use mutable values with caution. Such values can be useful as side effects, for instance to capture some idea of statefulness, perhaps keeping track of the number of times a flow was invoked. But they should generally not be used as inputs to a flow, especially if they are to be mutated.

7.2.5.4 Order of Effects

Qi flows may exhibit a different order of effects (in the functional programming sense) than equivalent Racket functions.

Consider the Racket expression: (map sqr (filter odd? (list 1 2 3 4 5))). As this invokes odd? on all of the elements of the input list, followed by sqr on all of the elements of the intermediate list, if we imagine that odd? and sqr print their inputs as a side effect before producing their results, then executing this program would print the numbers in the sequence 1 ,2 ,3 ,4 ,5 ,1 ,3 ,5.

The equivalent Qi flow is (~> ((list 1 2 3 4 5)) (filter odd?) (map sqr)). As this sequence is "deforested" by Qi’s compiler to avoid multiple passes over the data and the memory overhead of intermediate representations, it invokes the functions in sequence on each element rather than on all of the elements of each list in turn. The printed sequence with Qi would be 1 ,1 ,2 ,3 ,3 ,4 ,5 ,5.

Yet, either implementation produces the same output: (list 1 9 25).

So, to reiterate, while the output of Qi flows will be the same as the output of equivalent Racket expressions, they may nevertheless exhibit a different order of effects.

7.3 Effectively Using Feedback Loops

feedback is Qi’s most powerful looping form, useful for arbitrary recursion. As it encourages quite a different way of thinking than Racket’s usual looping forms do, here are some tips on "grokking" it.

In essence, the feedback loop is very simple –- all it does is pass the same inputs through a flow over and over again until a condition is met, at which point these inputs just flow out of the loop. Nothing complicated at all! The subtlety comes in, though, when we treat some inputs as "control" inputs that determine attributes of the flow or as "scratch" inputs that encode computations done on the flow, while treating the remaining inputs as the data that are actually acted upon. By doing this, we can do pretty much anything we’d like to, i.e. it can be used for general recursion.

7.3.1 Control Values and Data Values

Prior to entering the feedback loop, augment the data values by starting "channels" (that is, simply input values passed at a specific index to the feedback flow) for control or scratch values that the loop will need (although control and scratch inputs are not quite the same (see above), we can use the terms interchangeably for our purposes here). In some common cases, this may include a counter channel which keeps track of number of iterations, a result channel which accumulates an output, or something of this nature. In addition to these control channels, the loop will, of course, also receive all of the input data in the form of multiple values following the control values. The control inputs must always come first, so that we know where to find them (since we have no idea how many data values there will be at any stage of the loop), so that we can consistently refer to them using e.g. 1> and 2>.

7.3.2 Input Tracing

For each input, think about just one cycle of the loop: what must happen to it in this cycle before it is fed forward to the next cycle of the loop? Trace each input in this way and ensure that the corresponding output of the present cycle represents the correct input value for the next cycle. For instance, if there is a simple counter in the first input position, ensure that the first output of the present cycle is the counter incremented by one. We also need to ensure that the same number of control values flow to the next cycle as are used in the present cycle. There are no constraints on the number of data values, and often, this will change from one cycle to the next.

7.3.3 Keeping It Tidy

Use the then clause to ensure that the feedback loop produces only its computed output and not the "scratch" values used in guiding the flow, i.e., these should be blocked in the then clause (using, for instance, block or another appropriate form).

7.4 Idioms and Transforms

7.4.1 Nested Applications are Sequential Flows

A nested function application can always be converted to a sequential flow.

7.4.2 Converting a Function to a Closure

7.4.2.1 Basic Recipe

Sometimes you may find you want to go from something like (~> f1 f2) to a similar flow except that one of the functions is itself parameterized by an input, i.e. it is a closure. If f1 is the one that needs to be a closure, you can do it like this: (~> (==* (clos f1) _) apply f2), assuming that the closed-over argument to f1 is passed in as the first input, and the remaining inputs are the data inputs to the flow. Closures are useful in a wide variety of situations, however, and this isn’t a one-size-fits-all formula.

7.4.2.2 Definition vs Invocation Inputs

A flow defined using clos retains all of the inputs from the definition site when it is applied at the invocation site. Referring to inputs within the flow definition thus means these combined "definition site + invocation site" inputs (in that order, unless modulated by a containing threading form), not the inputs available at the definition site alone. So for instance, if you only need access to a subset of definition-site inputs rather than all of them, these should be filtered prior to passing them in to clos as, otherwise, you would unwittingly be filtering out invocation site inputs too. The following example illustrates this.

If you are interested in creating a string-append-derived function that prepends a prefix to an input, where the prefix is itself determined at runtime, you might attempt it this way:

> (~> ("prefix-" "something")       (-< (clos (~> 1> string-append))           2>)       apply)

"prefix-"

... but this does not produce the expected output for the aforementioned reason, that the body of the clos form refers to the inputs ("prefix-", "something", "something"), so that selecting the first one with 1> means there is only one input being passed to string-append in producing the result. Instead, the following is what we want:

> (~> ("prefix-" "something")       (-< (~> 1> (clos string-append))           2>)       apply)

"prefix-something"

Of course, in this particular example, using a relay instead of a tee junction would avoid the issue altogether.

> (~> ("prefix-" "something")       (== (clos string-append)           _)       apply)

"prefix-something"

7.4.3 Converting a Macro to a Flow

Flows are expected to be function-valued at runtime, and so you cannot naively use a macro as a flow. You can always convert a macro into a function by employing an esc form and wrapping the macro in a lambda.

But this can be cumbersome for anything other than a one-off use of a macro, and it also doesn’t take advantage of the syntactic conveniences (such as templates) that Qi already offers. You could write Qi macros to wrap these "foreign" macros and provide all of Qi’s usual syntactic behavior, but luckily, you don’t need to! Simply use define-qi-foreign-syntaxes to "register" any such foreign macros (i.e. macros in any language other than Qi, including Racket) as Qi forms, and then you can use them in the same way as any other function, except that the catch-all __ template isn’t supported.

Using this approach, you would need to register each such foreign macro using define-qi-foreign-syntaxes prior to use. Even though you can register as many as you like with a single declaration, this may feel like an impedance, especially for deep integrations with other DSLs where there may be a large number of such forms. See Writing a Qi Dialect for yet another approach.

7.4.4 Bindings are an Alternative to Nonlinearity

In some cases, we’d prefer to think of a nonlinear flow as a linear sequence on a subset of arguments that happens to need the remainder of the arguments somewhere down the line. In such cases, it is advisable to employ bindings so that the flow can be defined on this subset of them and employ the remainder by name.

For example, these are equivalent:

(define-flow make-document

(~> (== _

(~>> file-contents

(parse-result document/p)

△))

document))

(define (make-document name file)

(~>> (file)

file-contents

(parse-result document/p)

△

(document name)))

Adding bindings can eliminate nonlinearities, and by the same token, introducing nonlinearity can eliminate bindings.