Learning to follow commands from demonstrations
A continual learning system on gSCAN and ReaSCAN, without pretraining or gradient descent
Abstract. We trained Wakeline's Continual Learning System (CLS) on gSCAN and ReaSCAN, two benchmarks in which an agent in a grid world must turn a written command into an exact sequence of actions. The CLS learns online from the training demonstrations, without pretraining or gradient descent, and trains on gSCAN in 9–13 minutes in one CPU process. It reaches 100 % exact match on every test split of both benchmarks, on three training orders. This includes the splits where the best published learned models reach 43.6 % (gSCAN D), 11.94 % (gSCAN G) and 27.3 % (ReaSCAN C2). The CLS learns its frame of reference, a navigation policy, the adverb patterns and the size comparison. It is given step-by-step execution, symbolic perception and, for ReaSCAN's nested clauses, a locality prior. Two counting baselines on the same interface reach at most 42.2 %.
What we wanted to find out
Our working hypothesis at Wakeline is that a learner's environment carries enough information for it to build, quickly, the structure it needs. With that structure, it can understand and act in its environment as well as pretrained systems, or better. One particular interest of ours is how language can be learned not from large text corpora but the way a toddler learns it: by pointing at things and naming them. We built a training environment in which our Continual Learning System (CLS) constructs and recognizes objects from moving pixels alone, associates heard words with them, and uses that knowledge to carry out tasks. The CLS learns incrementally from what it observes, without pretraining or gradient descent.
To see how well the system performs at its current capability level, we evaluated it on gSCAN and ReaSCAN. gSCAN (Ruis et al., 2020) tests whether commands interpreted in a grid world generalize to held-out combinations, and since its release it has been addressed by many approaches, several of them with supplied structure (Appendix C). ReaSCAN (Wu et al., 2021) was built to be harder. Its commands pick out objects through relative clauses, so a model has to read the structure of the command, and its test splits hold out new combinations of objects, relations and clause structures. On its nested-clause split (C2) the best published learned model reaches 27.3 %.
The results came out very positive for the CLS, so we wanted to understand whether this was due to a) pure luck, b) too much pre-programmed scaffolding or c) a learning approach that actually works.
The two tasks
gSCAN. An agent stands in a 6 × 6 grid with several objects. Each object has a shape (circle, square or cylinder), a color (red, green, blue or yellow) and a size from 1 to 4. A command names a verb ("walk to", "push" or "pull"), an object, and sometimes an adverb ("hesitantly", "while spinning", "cautiously" or "while zigzagging"). The object is named by its shape, sometimes with its color and a size word; "small" and "big" mean the smallest or biggest of the objects that fit the rest of the description. The answer is the exact action sequence a demonstrator would produce, from six actions: walk, turn left, turn right, stay, push and pull. The agent always starts facing east. A push or pull continues until the object is blocked by a wall or another object, and a heavy object (size 3 or 4) takes two pushes or pulls per cell. The adverbs change the sequence: "hesitantly" adds a stay after each step, "while spinning" four left turns before each step, "cautiously" a look to both sides before each step, and "while zigzagging" alternates steps along rows and columns.
Split A is a random test split. Each other split holds back one kind of combination from training:
- B and C: a known color on a known shape as the target (yellow squares, red squares).
- D: a target south-west of the agent.
- E: a size-2 circle called "small", which training never shows.
- F: pushing a size-3 square.
- G: "cautiously", with a single training demonstration.
- H: "pull" together with "while spinning".
- I: action sequences longer than any in training.
ReaSCAN. The same world and actions, with descriptions that need reading. A command picks out its target through relative clauses about other objects. The relations are same row, same column, same color, same shape, same size, and inside a box; boxes are outlined objects that other objects can lie inside. Each scene contains distractors, so that every clause is needed to find the target. Training commands have up to two clauses, joined by "and". The zero-shot splits hold out new combinations of color, shape and size (A1–A3), objects or relations never seen together (B1, B2), three clauses (C1), and a clause inside a clause (C2).
The commands in the figures and the video were drawn at random from each split with a fixed seed, among those with 8 to 18 actions.
What we did
On gSCAN and ReaSCAN, in contrast to our testing environment, the CLS does not see pixels, but the symbolic world state the benchmarks provide, as the published models do. So we took out the part of the perception system that learns objects from low-level value streams in pixels and instead fed a tuple of discrete values into the higher-level learning layer.
We trained the CLS on the training demonstrations of gSCAN (367,933) and ReaSCAN (539,722), in three training orders each, and evaluated it on the full test sets. For each piece the model learns, we ran a control that takes the learned piece away and shows which split depends on it. What the CLS is given is listed below; the model itself is described in Appendix A, and Appendix C compares what each system is given.
To separate the learning rule from the interface, we ran two counting baselines that keep the CLS's interface and replace its learning with a plain table. As a reference, we trained a Transformer from scratch on the same gSCAN data.
Every scored run had its protocol and predictions committed to version control before it ran.
What we found
- The CLS generalizes exactly on both benchmarks. It reaches 100 % exact match on every split of ReaSCAN and of gSCAN, on three training orders. On ReaSCAN's nested clauses (C2), of which the training data contains none, the result rests on a locality prior we supply.
- It learns mappings and choices inside representations we supply. The CLS learns its frame of reference, a step-by-step navigation policy, the segmentation of adverb patterns, the comparison behind "small" and "big", and where an "and" clause attaches. It needs no annotation of which object a training demonstration is about. It is given step-by-step execution, symbolic perception, the kinds of situation each learner distinguishes and, for nested clauses, a locality prior.
- Simple counting on the same interface does not reach these results. With the CLS's learning replaced by a table of the last outcome seen, exact match is at most 15.9 % on any split. With a table of the most frequent outcome, it is at most 42.2 %, and 0 % on E and H.
- It is cheap to train. The CLS uses no pretraining, trains on gSCAN in 9–13 minutes in one CPU process and produces an action sequence in about 1 ms.
- The Transformer reproduces prior results: 99.85 % on random test commands and 0 % on D.
gSCAN
| Split | Baseline | Best prior result | CLS (ours) |
|---|---|---|---|
| A: Random | 97.69 ± 0.22 | 99.95 (Qiu et al., 2021) | 100.00 |
| B: Yellow squares | 54.96 ± 39.39 | 99.90 (Qiu et al., 2021) | 100.00 |
| C: Red squares | 23.51 ± 21.82 | 99.9 (Sikarwar et al., 2022) | 100.00 |
| D: Novel direction | 0.00 ± 0.00 | 43.60 (Setzler et al., 2022) | 100.00 |
| E: Relativity | 35.02 ± 2.35 | 99.8 (Sikarwar et al., 2022) | 100.00 |
| F: Class inference | 92.52 ± 6.75 | 99.98 (Qiu et al., 2021) | 100.00 |
| G: Adverb k = 1 | 0.00 ± 0.00 | 11.94 (Kuo et al., 2021) | 100.00 |
| H: Adverb to verb | 22.70 ± 4.59 | 98.4 (Hein & Diepold, 2022) | 100.00 |
| I: Length | 2.10 ± 0.05 | 84.42 (Setzler et al., 2022) | 100.00 |
- Evaluation: exact match over the full action sequence, computed with the official gSCAN evaluation code. It agrees with our own scoring on all 1,752,003 predictions (dev, A–H and the length split, three training orders).
- End to end: every reported score uses the model's own choice of target object, made from the command and the scene. The dataset's target annotation is used neither at test time nor in training. The replay of an exact sequence ends on the same object as the demonstration, so target selection is also 100 %.
- Best prior results: the D and I results of Setzler et al. (2022) are from an arXiv preprint; the best peer-reviewed result on D is 16 (Spilsbury et al., 2024). The G result of Kuo et al. is the version on the gSCAN leaderboard.
- Systems with hand-written components: Yang et al. (2023) report 100 % on the first 1,000 examples of each split, using GPT-3 as a semantic parser with hand-written logic rules. Appendix C lists what each system is given.
ReaSCAN
| Split | GCN-LSTM (Gao et al., 2020) | GroCoT | CLS (ours) |
|---|---|---|---|
| Compositional test (updated) | 97.10 ± 0.38 | – | 100.00 |
| P1: simple (updated) | 99.38 ± 0.13 | – | 100.00 |
| P2: one relative clause (updated) | 97.71 ± 0.56 | – | 100.00 |
| P3: two relative clauses (updated) | 95.87 ± 0.40 | – | 100.00 |
| A1: yellow squares | 92.3 | 99.6 | 100.00 |
| A2: red squares | 42.1 | 93.1 | 100.00 |
| A3: small cylinders | 87.5 | 98.9 | 100.00 |
| B1: novel co-occurrence of objects | 69.7 | 93.9 | 100.00 |
| B2: novel co-occurrence of relations | 52.8 | 86.0 | 100.00 |
| C1: three clauses | 57.0 | 76.3 | 100.00 |
| C2: nested relative clauses | 22.1 | 27.3 | 100.00 |
- CLS model: the ReaSCAN model reuses the gSCAN model's navigation and manner learners and adds a learner for relational clauses, trained on the 539,722 training demonstrations.
- Data: ReaSCAN v1.1 from the authors' repository (archive sha256
679900cf751a1af1e82628b4acb95e73183d21834a40e548ab0f91811d177d24). The authors replaced the non-generalization test sets after the original paper, because the earlier ones could contain examples identical to training examples. The compositional test set in this release is the updated one: it is exactly the union of the updated P1–P3 test sets (5,753 commands). Table 2 therefore compares it with the authors' updated results for these sets, not with the paper's. The zero-shot splits A1–C2 were not changed. Boxes are objects of the scene when the command is read. The replay world holds one object per cell and leaves boxes out: the target is never a box, and boxes never block movement in any training or dev demonstration. "while spinning" and "while zigzagging" are joined into one token, as in gSCAN. - Evaluation: exact match over the full action sequence, computed by our own scoring code. We did not run an official ReaSCAN scorer. Every reported score is end to end, as on gSCAN. As a separate diagnostic, not part of any reported score, execution with the true target supplied is also exact on every example of every split.
- Given in addition to the gSCAN model: a generic family of relations (same, lower and higher per dimension, and extent units from which "inside" is learned, with size read as a length in cells); one relation word per clause; the requirement that a relation link exceed the base rate of its unit; the hypothesis space for splitting a command into clauses; two candidate readings for where a clause attaches; the prior that a relative clause attaches to the nearest preceding noun phrase, used only where training has no evidence; that a clause restricts the phrase it attaches to, applied recursively; and that the objects a command mentions are distinct. The attachment of "and" clauses is learned. C2 commands contain only row and column relations, so the C2 result does not depend on the extent units.
What the CLS is given
The CLS learns which attribute of an object each word refers to (word-to-world mappings), its own frame of reference, a step-by-step navigation policy, the adverb patterns and where they go, which objects "small" and "big" pick out, and when a push or pull stops (Appendix A). It is given the following.
- Step-by-step execution. It acts one action at a time in a grid simulator and observes the state after each.
- Symbolic attributes. Each object comes as shape, color, size and position; which words name which attributes is learned.
- An opaque heading token and own movement. One of four labels without meaning, and the agent's change of position in grid coordinates after each action.
- The offset to the object it has chosen. At test time the model chooses the object from the words and the scene; the offset is the difference between that object's position and the agent's. In training, the object the demonstration ends on plays this role.
- The four neighboring cells. Whether each is free, holds an object or lies outside the grid.
- Contact and movement signals. The object the agent ends on in a training demonstration, and whether an action moved the agent or an object.
- The kinds of situation each learner can distinguish. For navigation: where the target lies relative to the agent (ahead or behind, left or right, in the learned frame) together with the previous step, with one decision per cell. For adverb patterns: a prediction of the next action that is kept until it is met. For size words: comparisons between the candidate objects a command leaves. For heavy objects: five ways of keying how many pushes a cell takes.
- A preference for the most specific situation that applies, when several learned situations match.
- Readout thresholds. Two fixed constants, the same on every split.
No navigation procedure is given: paths are made one cell at a time by the learned policy.
What the CLS learns, and what depends on it
Each learned piece was scored with a control that takes it away; Appendix D gives every control on every split. Where a control fails, it fails on exactly the commands that need the piece, in whichever split they occur. The compass control fails D and nothing else. The fresh-prediction control fails every command with "while spinning" or "cautiously" and no other. The control without comparisons fails every command with a size word and no other; on every split its score equals the share of commands without one. The heaviness controls lose nothing. Of the two clause controls, the flat reading loses only C2; applying the prior to "and" as well changes what the learner hears and loses commands across the clause splits.
| Learned piece | What it does | Split it concerns | Control | Control's exact match |
|---|---|---|---|---|
| Frame of reference and step policy | Learns ahead, left, right and behind per heading from the agent's own movement, and chooses each step from where the target lies in that frame and from the previous step | D | The same learner with compass directions in place of its own frame | D 0 %, every other split 100 % |
| Segmentation of adverb patterns | Walks the chain's own prediction of the next action along each demonstration, keeping it until it is met; what it does not predict forms the adverb pattern | G, H | The same prediction made afresh at every step | G 0 %, H 0 %, other compositional splits 75–85 %, length split 100 % |
| Size comparison | Learns, per word, comparisons between the candidates of a command ("this one has size 1 where another has size 2"); only words reliably linked to an attribute take part | E | No comparisons between candidates | E 0 %, other splits 33–42 % |
| Pushes per cell for heavy objects | Reads from the data which of five keyings carries the count | F | Each keying alone, and the verb alone with no count | 100 % on every split in every case |
| Clause attachment (ReaSCAN) | Learns where an "and" clause attaches; a locality prior covers the attachment training gives no evidence for | C2 | Nested clause read as another clause on the head | C2 3.6 % |
| Locality prior applied to "and" as well | C2 32.3 %, the single-clause split P2 54.2 % | |||
| Target object in training | Uses the object the demonstration ends on | all | The dataset's annotation of the target object | identical predictions (0 of 1,752,003 differ) |
- Navigation. In the agent's own frame, the first state of a south-west path is unseen, and the policy falls back to less specific situations it has learned ("behind: turn round", "ahead: walk"). Turning round is read from the demonstrations as two left turns, and as two right turns when train and dev are re-demonstrated that way. The only path-changing word it finds is "while zigzagging". Its paths match the demonstrated paths on every training demonstration. The policy does not compose a direction from its parts: it fails a target due north of the start, and two quadrants held out at once (see Limitations). Split I does not depend on the learned frame: with compass directions it is also 100 %, because the length split's training covers every direction.
- Segmentation. The control shows what keeping the prediction does. A prediction made afresh passes through states inside a spin from which the next turn is predicted, so the spin is cut in half and every spinning or cautious command fails. The learned patterns are the four-turn spin for "while spinning", a stay after each step for "hesitantly", and the four-turn look for "cautiously", learned from its single training example.
- Size comparison. "Small" and "big" are the only words that take part, on every training order. The price is a condition outside the released splits: a size word applied to a pair of sizes never seen with that word (see Limitations).
- Heaviness. F rests on world feedback during execution: the agent keeps pushing while the object moves, so the count never decides a released example. Keying on object identity alone would read the wrong count for the red squares of C, which never appear as targets in training. Offering all five keyings reads the right count everywhere.
- Clause attachment. The training data contains no command with a clause inside a clause (0 of 539,722 training commands), so where a nested clause attaches cannot be learned from the training data. Any system that gets C2 right from ReaSCAN's training set carries a prior of this kind. The "and" attachment is chosen on held-out training examples (coordinate 99.7–100 % against nearest 36–38 %). Applying the prior to "and" as well makes the learner hear relation words against the wrong object, so it learns a worse lexicon and fails even single-clause commands.
- Target object. On gSCAN's data every demonstration ends on the object it is about, so training on the end object gives the same tables and the same predictions as the dataset's annotation. At test time the model chooses the object itself.
A counting baseline on the same interface
To isolate the learning rule from the interface, we kept the CLS's structure (execution, percepts, the kinds of situation each learner distinguishes, the places where a learned mapping is read) and replaced every learned mapping with a plain table:
- Last seen: each situation maps to the outcome it was last paired with.
- Most frequent: each situation maps to the outcome it was paired with most often.
Each table replaced every learned mapping at once, on three training orders, and was scored with the official gSCAN scorer, which agrees with our own scoring on every prediction.
| Split | Last seen | Most frequent | CLS |
|---|---|---|---|
| dev | 0.9–1.9 | 41.5–41.9 | 100.0 |
| A: Random | 0.9–1.0 | 28.6–29.0 | 100.0 |
| B: Yellow squares | 0.1–1.5 | 42.2 | 100.0 |
| C: Red squares | 0.2–0.6 | 37.0 | 100.0 |
| D: Novel direction | 0.0–1.7 | 41.7–42.0 | 100.0 |
| E: Relativity | 0.0–1.2 | 0.0 | 100.0 |
| F: Class inference | 0.2–1.1 | 39.6 | 100.0 |
| G: Adverb k = 1 | 0.0 | 33.6–33.9 | 100.0 |
| H: Adverb to verb | 0.0–1.9 | 0.0 | 100.0 |
| Length dev | 1.5–15.9 | 31.9 | 100.0 |
| Length test | 1.4–13.0 | 30.9 | 100.0 |
| I: Length | 0.8–9.6 | 14.7 | 100.0 |
Discussion
On both benchmarks, the CLS reaches 100 % on every split. It learns mappings and choices inside representations we supply: its frame, its navigation policy, the segmentation of adverb patterns, the size comparison and the attachment of "and" clauses, with less supplied structure than Setzler et al. (2022), who give step-by-step execution, and than Hein & Diepold (2022), who give the agent's position and heading, a memory of recent actions and target supervision. It is given step-by-step execution, symbolic perception of the grid, the offset to the object it has chosen, the kinds of situation each learner distinguishes and, on ReaSCAN, a locality prior for clause attachment.
The counting baselines are given exactly the same interface as the CLS. With a table of the last outcome seen they reach at most 15.9 % on any split, and with a table of the most frequent outcome at most 42.2 %.
- Comparability. The systems receive different inputs and structure, so the results are not like for like. The most informative comparisons are with the prior models that were given part of the same structure (Appendix C), and with the counting baseline, which is given exactly the same.
- ReaSCAN C2. The training data contains no nested clause. The CLS's C2 result rests on the locality prior, and we expect any system that reaches C2 from ReaSCAN's training data to carry an equivalent assumption.
- Interaction. The CLS executes each command step by step and, after each step, perceives what lies in the cells around the agent and whether the object moved; from this it has learned when a push or pull stops. A model that produces the whole action sequence at once has to infer from the world state whether an object is blocked.
Limitations
- Two losses outside the released splits. Besides the released splits, we test on seven conditions built from gSCAN's training data with the official generator, one of them a control. The CLS scores 0 % on two of them. A size flip (a size word applied to a pair of sizes never seen with it) fails because a pair never confirmed with the word has no verdict. A target due north of the start fails because the navigation policy falls back to a situation learned from due-south starts and does not compose a direction from its parts. Each loss belongs to one learned piece. Neither touches the released splits.
- Navigation. The learned policy does not handle two quadrants held out at once (50–86 % on plain walks in development). All starts in gSCAN face east, so a leg order relative to the agent and one relative to the grid cannot be told apart.
- Supplied structure remains. The kinds of situation each learner distinguishes, the commitment in the segmentation, the comparison within candidates, the readout constants and the ReaSCAN prior are ours. Three results rest on such structure: F on world feedback during execution, I on step-by-step execution, and ReaSCAN C2 on the locality prior. Whether a learner can arrive at this structure without being given it is not tested here.
- ReaSCAN scoring. The ReaSCAN results use our own exact-match scoring; the official gSCAN scorer was run on all gSCAN predictions.
- Internal results. No outside group has rerun them.
Sources
- Ruis, Andreas, Baroni, Bouchacourt, Lake 2020: A Benchmark for Systematic Generalization in Grounded Language Understanding (NeurIPS)
- Ruis, Lake 2022: Improving Systematic Generalization Through Modularity and Augmentation (CogSci)
- Qiu, Hu, Zhang, Shaw, Sha 2021: Systematic Generalization on gSCAN (EMNLP)
- Setzler, Howland, Phillips 2022: Recursive Decoding (arXiv preprint)
- Hein, Diepold 2022: A Minimal Model for Compositional Generalization on gSCAN (BlackboxNLP)
- Kuo, Katz, Barbu 2021: Compositional Networks Enable Systematic Generalization (Findings of EMNLP) and the gSCAN leaderboard
- Nye, Tessler, Tenenbaum, Lake 2021: Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoning (NeurIPS)
- Spilsbury, Marttinen, Ilin 2024: Generating Demonstrations for In-Context Compositional Generalization (EMNLP)
- Wu, Kreiss, Ong, Potts 2021: ReaSCAN (NeurIPS Datasets and Benchmarks)
- Gao, Huang, Mooney 2020: Systematic Generalization on gSCAN with Language Conditioned Embedding (AACL-IJCNLP)
- Sikarwar, Patel, Goyal 2022: When Can Transformers Ground and Compose (EMNLP), Table 4
- Yang et al. 2023: GPT-3 with answer-set programs on gSCAN (Findings of ACL)
Appendix A: Our model
The CLS learns incrementally: each observation updates what it has learned as it arrives, without epochs, gradient descent or pretrained components, and what it has learned can be inspected directly. In our own environment the same mechanism learns object kinds from pixels, words from sound, and the links between them. On gSCAN and ReaSCAN the model consists of three learners. One identifies the target object. One learns the agent's frame of reference and a policy that chooses each step toward the chosen object. One learns the action patterns of the adverbs, where they go, and when a push or pull stops. Table A1 lists what is learned from the demonstrations and what the model is given.
| Learned from the demonstrations | Given to the model |
|---|---|
| Word-to-world mappings | Step-by-step execution of each action in the world state |
| Which words compare sizes ("small" and "big"), and which candidate each picks out, from comparisons between candidates | Symbolic attributes and position per object, the heading as an opaque token, and the agent's own movement in grid coordinates |
| The agent's frame of reference (ahead, left, right, behind per heading), from its own movement within 13–18 demonstrations | The offset from the agent to the object the model has chosen, from the positions in the scene; in training, the object the demonstration ends on |
| A step policy: which direction to step from where the target lies and from the previous step, and which word changes a path ("while zigzagging") | The contents (free, edge or object) of the four cells around the agent after each step, and signals for contact and for movement of the agent or an object |
| How to turn around, read from the demonstrations | The kinds of situation each learner distinguishes: target side and previous step, with one decision per cell; comparisons within the candidates of a command; five keyings of the count for heavy objects |
| The action pattern of each adverb and on which side of a step it goes, for "cautiously" from its single training example | A prediction of the next action per token, kept until it is met, from which the adverb patterns are segmented |
| Which actions count as steps; when a push or pull stops, from what lies in the cells around the agent; how many pushes a heavy object takes per cell | A preference for the most specific matching situation; walk and turn left as the actions that define the frame's axes; the readout thresholds |
| On ReaSCAN: relation words, the clause split, and where an "and" clause attaches | On ReaSCAN: the relation family, the clause split's hypothesis space, two candidate attachments, the locality prior where training has no evidence, recursive restriction, distinct objects |
Paths are produced by the learned policy one cell at a time, in the learned frame. Several prior models were also given part of this structure; Appendix C compares what each system receives.
Example: one training demonstration as the CLS receives it. Figure A1 shows a record from the gSCAN training set.
At test time the record holds no demonstration. Replaying the demonstration step by step gives the learners the following. The offset is to the object the demonstration ends on; at test time it is to the object the model has chosen.
| Action | Heading | Agent position | Offset to the object (rows, columns) | Moved |
|---|---|---|---|---|
| (start) | east | (0, 3) | (+1, 0) | – |
| turn right | south | (0, 3) | (+1, 0) | nothing |
| walk | south | (1, 3) | (0, 0) | agent |
| stay | south | (1, 3) | (0, 0) | nothing |
| push | south | (2, 3) | (0, 0) | agent and square |
| stay | south | (2, 3) | (0, 0) | nothing |
| push, stay (3 more times) | south | (5, 3) | (0, 0) | agent and square at each push |
From many such demonstrations, one learner links "square" to the shape of the object the demonstration ends on. Another learns that "turn right" turns the agent from east to south, that "walk" moves it one cell ahead, and that with the target ahead in its frame the next step is a walk. The third learns that "hesitantly" adds a stay after each step and that a push repeats until the cell ahead is the edge of the grid or another object.
Appendix B: Compute
| System | Hardware | Pretraining | Training | Time per command |
|---|---|---|---|---|
| CLS, gSCAN | one process on a shared 16-core server | none | 9–13 min on 367,933 demonstrations, data loading included | about 1 ms (0.7 ms on D), test-set loading included |
| Transformer from scratch (2.8 M parameters) | Apple M4 Max GPU | none | 84 min, 60,000 steps | – |
The CLS learns from the full training set. How few demonstrations it needs has not been measured.
Appendix C: Given structure across systems
Step-by-step execution, a given agent pose, target supervision, intermediately supervised modules, a symbolic world model and hand-written rules have each been used in prior work on gSCAN.
| System | Given beyond the world state and the command | Pretraining | D | G | H |
|---|---|---|---|---|---|
| End-to-end models (Ruis et al., 2020; Qiu et al., 2021; Sikarwar et al., 2022) | none | none | 0.0 | 0.0 | 22.2–22.9 |
| Kuo et al. (2021) | network structure that follows the parse of the command | none | 11.5 | 1.1 | 21.0 |
| Nye et al. (2021) | a hand-written symbolic world model and verifier | none | 0.0 | 0.0 | 21.8 |
| Setzler et al. (2022), RD random | step-by-step execution: the world state is re-encoded after every action; random start orientation in training | none | 43.6 | 0.0 | 22.0 |
| Hein & Diepold (2022) | the agent's position and heading, a memory of recent actions, target supervision | none | 2.2 | 0.0 | 98.4 |
| Ruis & Lake (2022) | modules trained on ground-truth intermediate outputs (agent and target position, heading, plan); 150 augmented adverbs | none | 0.0 | – (k = 5: 80.0) | 76.8 |
| Yang et al. (2023) | GPT-3 as a parser; hand-written logic rules (answer-set programs) for the world and the actions | yes | 100 (first 1,000) | 100 (first 1,000) | 100 (first 1,000) |
| Counting baseline (this work) | the same as the CLS, with every learned mapping read from a most-frequent table | none | 41.7–42.0 | 33.6–33.9 | 0.0 |
| CLS (ours) | step-by-step execution; symbolic attributes and positions, an opaque heading token and own movement; the offset to the object the model has chosen, from the positions in the scene; the contents of the four neighboring cells; contact and movement signals; the kinds of situation each learner distinguishes (target side and previous step for navigation, a committed next-action prediction for adverb patterns, comparisons within the candidates for size words); readout thresholds; no navigation procedure | none | 100 | 100 | 100 |
Appendix D: Controls on every split
| Control | dev | A | B | C | D | E | F | G | H | Length dev | Length test | I |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Compass directions in place of the learned frame | 100 | 100 | 100 | 100 | 0.0 | 100 | 100 | 100 | 100 | 100 | 100 | 100 |
| Next-action prediction made afresh at every step | 81.9 | 84.9 | 82.7 | 82.7 | 82.0 | 81.8 | 75.0 | 0.0 | 0.0 | 100 | 100 | 100 |
| No comparisons between candidates | 42.1 | 35.6 | 40.8 | 40.8 | 41.8 | 0.0 | 33.3 | 41.8 | 41.6 | 39.9 | 36.4 | 42.4 |
| Heavy objects: each of the five keyings alone, and the verb alone with no count | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 |
| Control | Compositional test | P1 | P2 | P3 | A1 | A2 | A3 | B1 | B2 | C1 | C2 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Nested clause read as another clause on the head | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 3.6 |
| Locality prior applied to "and" as well | 37.1 | 100 | 54.2 | 2.8 | 39.8 | 28.8 | 42.9 | 1.9 | 0.0 | 0.1 | 32.3 |
This report describes internal research results. It makes no claim about products or their performance.
Investment Opportunities
We are currently engaging with investors and strategic partners interested in long-term technological impact grounded in scientific discipline.
Wakeline represents a fundamentally different approach to artificial intelligence, prioritizing continuous adaptation, reduced compute dependence, and real industrial application.
Conversations focus on collaboration, evidence building, and shared ambition.