GlyphBench: A Playground for Language-Model Reinforcement Learning

We introduce GlyphBench, a game-based environment suite for reinforcement learning (RL) post-training of language-model agents, with over 360 tasks spanning diverse games. GlyphBench renders spatial observations as two-dimensional Unicode grids and connects training, evaluation, and trajectory replay through a unified interface designed to support efficient and reproducible research. We use GlyphBench to study how observation interfaces, reasoning effort, and agent harnesses affect performance, and how RL configurations shape learning dynamics. Our results show that glyph observations outperform native text and pixels in a variety of environments. Post-training on GlyphBench yields gains surpassing those provided by post-training on math- and code-based tasks, providing evidence that reasoning gains from gameplay can yield strong transfer and generalization. Together, these results highlight GlyphBench’s value as a testbed for systematic research on how language-model agents learn, interact, and generalize.

Reinforcement learning (RL) has improved the performance of language models on problems with verifiable rewards, such as mathematics and code¹. Tasks that are interactive by nature are particularly challenging: models must choose actions that shape subsequent observations, act with incomplete information, and account for consequences that may only become apparent later. For example, a software-engineering agent edits code, runs tests, and uses the resulting feedback to guide further decisions². Studying how models acquire these capabilities, and whether they support more general reasoning, requires diverse interactive tasks that researchers can use for training and evaluate under controlled conditions.

Games have been a central testbed for deep RL over the past decade, exposing different learning challenges through repeatable interactions. Benchmarks such as the ALE³ helped establish deep RL⁴, while Procgen⁵, MiniHack⁶, and Craftax⁷ expose distinct challenges in generalization, exploration, and planning. Games have also driven landmark results in Go⁸ and StarCraft II⁹, which have sparked further advances in the field. Collectively, these benchmarks support both targeted studies of individual capabilities and experiments in which several capabilities must work together.

For language-model post-training, however, differences in observation and action interfaces complicate comparisons across environments: performance depends on how information is presented as well as on the decisions a task requires¹⁰. Recent training studies use settings as different as BabyAI-Text¹¹, symbolic multi-turn games¹², and visual arcade games¹³. BALROG¹⁴ and lmgame-Bench¹⁵ bring several games together through shared interfaces, but game collections, observation formats, and training setups still differ across studies. Making these well-studied environments accessible through a common text interface that preserves spatial layout offers a practical way to study learning and generalization across games within a shared experimental framework.

We introduce GlyphBench to make a diverse collection of interactive environments accessible for language-model RL. GlyphBench brings together puzzles, arcade games, and established RL benchmarks through a common text interface: agents observe two-dimensional Unicode grids and interact through named actions. These grids preserve the relative positions of objects in a form that language models can consume directly and researchers can inspect. Researchers can train and evaluate policies across these environments, replay trajectories, and test changes to training or the agent harness (Figure 1). GlyphBench supports this cycle of measuring performance, diagnosing behavior, and testing improvements across diverse tasks.

Figure 1. GlyphBench connects nine game families through spatial text observations and named actions. A shared task interface supports policy evaluation, RL training, and trajectory replay. Six families form the standard track; three provide extended tasks.

We first evaluate models of different sizes and capabilities, comparing task performance, episode length, and token use. Our experiments in Craftax¹⁶ show the benefit of spatial glyph observations: they achieve higher mean reward than native text and pixels for all three models we evaluate. We separately examine how harness design affects the use of additional reasoning. Experiments on BALROG¹⁷ test the interface beyond our task collection and show gains on several environments.

We train a single policy on 100 GlyphBench tasks and observe improvements across all six standard suites on held-out environment seeds. The trained policy improves on held-out Reasoning Gym¹⁸ problems, outperforming a math-trained baseline and a code-trained baseline. These results provide empirical evidence that reasoning gains from gameplay can transfer more effectively to this benchmark than those from math or code RL.

We designed GlyphBench to support broader research on how language models learn from rewards, use information during interaction, and generalize. To illustrate its use for studying training dynamics, we examine learning rate and rollout group size, two central choices in group-based RL. After verifying that single-task policies learn across all six standard suites, we vary these choices in Snake under a fixed interaction budget. Smaller rollout groups allow more updates, but the resulting gains depend strongly on learning rate.

Language-model agents in games. Applying language models to game environments introduces a fundamental design question: how should pretrained knowledge be translated into effective actions? SPRING addresses this through structured reasoning over knowledge extracted from the environment¹⁹, while Voyager generates, refines, and reuses executable skills in Minecraft²⁰. Both illustrate how an agent’s reasoning procedure and supporting components can improve gameplay without updating the underlying model’s parameters. Broader benchmarks examine these design choices across environments. BALROG evaluates language-model and vision-language agents on established RL tasks and identifies limitations in handling complex environments and visual observations²¹. lmgame-Bench studies how perception, memory, and prompt design affect the reliability of game-based evaluation²². Agentick extends comparisons to RL, language-model, and vision-language agents through common tasks with multiple observation modalities²³. These studies show why comparisons must account for the interaction interface alongside the underlying model, motivating our experiments on harness design and inference-time reasoning.

Learning through interaction and transfer. Learning from interaction raises two further questions: how to train policies effectively, and whether the resulting improvements extend beyond the training tasks. GLAM investigates online RL for grounding pretrained language-model policies in textual environments with spatial and navigation tasks²⁴. RAGEN studies training instability and the effects of rollout diversity and freshness in multi-turn RL²⁵, while AgentGym-RL develops a framework for training across interactive environments and a strategy that progressively increases interaction horizons²⁶. These works make the collection and optimization of interaction trajectories central to the study of agent learning. Evidence for transfer from games also highlights the importance of the evaluation domain. The initial lmgame-Bench study reports that RL on Sokoban or Tetris improves performance on other games, planning problems, and WebShop²⁷. ViGaL reports that RL on visual arcade games improves multimodal reasoning on benchmarks including MathVista and MMMU²⁸. These findings motivate investigating which capabilities transfer under different training conditions. We use GlyphBench to study optimization under a fixed interaction budget and multitask training across diverse spatial text games, evaluating both performance on held-out game seeds and transfer to held-out Reasoning Gym problems²⁹. Appendix A expands these connections.

GlyphBench provides over 360 tasks across nine game families through a common framework for language-model training, evaluation, and trajectory replay. Researchers can study learning on individual tasks, train policies across task mixtures, and inspect the decisions behind aggregate performance. Glyph observations and named, task-specific actions provide a consistent interaction protocol across the suite. Standard tasks bound episode lengths and returns to support repeated RL experiments, while extended tasks support longer episodes of exploration and progression.

Each task e is an episodic, partially observed decision process with transition rule Pₑ, observation rule Oₑ, and rewards rₜ. A language model with parameters θ and a harness η induce a policy πθ,η(aₜ | hₜ) over actions given interaction history hₜ. The harness constructs prompts, retains history or memory, and parses model outputs. For task distribution 𝒟, training seeks to maximize

where τ is an episode of length T. This notation distinguishes the variables studied below: the observation rule and harness determine the agent’s inputs, RL updates θ, and the training mixture determines 𝒟.

We place spatial information in a two-dimensional Unicode grid, identify symbols with a legend, and use a heads-up display (HUD) for quantities such as health, inventory, or velocity. Actions use a task-specific discrete vocabulary. This representation makes adjacency and layout explicit in text (Figure 2); Appendix B shows the full interface in replay.

We select game families with an established role in RL research on control, procedural generalization, navigation and memory, and survival with staged progression (Appendix Table 1). Using these familiar challenges gives researchers a basis for interpreting language-model behavior and choosing tasks that test a particular hypothesis.

We adapt their mechanics into 303 standard tasks and bound episode lengths and cumulative returns by design: T ≤ 512 and Rₑ(τ) ∈ [−1, 1]. The episode cap limits the cost of collecting each training example, while the shared return range makes it easier to combine tasks and compare progress across a mixture. The 59 extended tasks support longer episodes through arcade adaptations and interfaces to full Craftax and NetHack. We use standard tasks for RL training and CraftaxFull to study agents that must explore, gather resources, and prepare for challenges across nine floors. Appendix C illustrates these floors, and Appendix B describes the training interfaces and trajectory replay tools.

Figure 2. The same Craftax state in pixels, Craftax’s original text renderer (native text), and GlyphBench’s glyph format. Native text is a comparison interface, not part of GlyphBench. Text excerpts highlight spatial representation; Figure 11 gives both complete observations. Glyph colors aid visualization only.

We evaluate GlyphBench as a setting in which language models can act, learn from environmental feedback, and develop capabilities that extend beyond the training environments. We first compare existing models without additional training on GlyphBench, examining task performance improvement and the effects of observation format, agent harness, and reasoning effort. To measure the learning transfer provided by GlyphBench across tasks, we then train policies on individual tasks and a task mixture. Finally, to determine whether training on GlyphBench produces better reasoning transfer to planning tasks, we test the trained policies on external reasoning problems to evaluate transfer beyond gameplay. Training uses our Prime-RL integration³⁰; Appendix E specifies the objectives and configurations.

Comparing model capabilities. We evaluate Qwen3.5-4B and Qwen3.8-27B on all 303 standard tasks with 25 matched seeds per task (7575 episodes per model). Returns are averaged within tasks and tasks receive equal weight. Episode length and prompt-plus-completion tokens complement return by measuring how much interaction and generation each model uses.

Agent design and inference budgets. We use CraftaxFull to study how agents interpret observations and use information from earlier decisions. We compare two agent harnesses. The Basic harness presents the game instructions and conversation history at each turn. The Guided Memory harness instead organizes information into a compact memory of plans, landmarks, and recent actions, and supplies guidance for the current floor. It updates this memory through separate model calls, giving the agent a way to revisit its plans as the game changes.

For GPT-5.6-{Luna, Terra, Sol}, we first compare native text, glyphs, and pixels with Guided Memory at high reasoning effort. We then compare the two harnesses using glyphs and vary Sol’s reasoning effort to examine how harness design affects the use of additional computation. We report five-episode means with 95% t intervals, scoring reward as 100R/226³¹. We include one exploratory GPT-6-Astra episode with Guided Memory at maximum effort to illustrate how far a stronger model can progress. Appendix C describes the Craftax interfaces and harnesses. We also compare original and glyph interfaces for five models in BabyAI, Baba Is AI, Crafter, and MiniHack from BALROG³².

Establishing learnability. We train 25 separate Qwen3.5-4B policies across all six standard suites to check whether the tasks supply useful reward signals and the common training interface supports improvement. These single-task runs establish a basis for studying how training choices affect learning.

Allocating a fixed interaction budget. We use Snake from GlyphBench to illustrate how learning rate and rollout group size interact during RL. Across its three difficulty levels, we compare frequent updates from small groups with fewer updates from large groups under the same interaction budget. We cross five group sizes (32–512) with four learning rates (10⁻⁶–10⁻⁵), yielding 60 conditions. Each update uses one group, so 6,144 rollouts permit 6,144/G updates at group size G. In Appendices J and K, we examine higher learning rates and longer rollout budgets.

Learning across tasks and testing transfer. We study learning over 100 gradient steps with one Qwen3.5-4B policy trained on 100 GlyphBench tasks. We use group size 32, learning rate 5×10⁻⁶, and 32 groups per update (1,024 rollouts), with Prime-RL’s IPO-style loss. We evaluate on held-out environment seeds every 50 updates. To examine transfer beyond the training environments, we compare the base model and policies obtained after GlyphBench, Math, and Code RL on identical Reasoning Gym problems³³. Appendix I describes the baseline datasets and evaluations; Appendix H reports additional evaluations on MathArena and SWE-bench³⁴³⁵³⁶.

We first characterize performance before additional RL training on GlyphBench. Qwen3.8-27B improves on 270 of 303 tasks, with a mean paired return gain of 0.436. Return distributions shift upward in every suite, while scores remain spread across tasks within each family (Figure 3). The collection therefore distinguishes the two models while retaining tasks on which the stronger model has room to improve, establishing a reference point for the training experiments that follow.

Figure 3. Performance before additional GlyphBench training on 303 tasks with 25 matched seeds each. (a) Task-mean returns, with interquartile ranges and median dots. (b–c) Length and token use over 7575 episodes per model. Tokens include prompts and completions under each model’s tokenizer.

Higher performance also comes with lower interaction cost: mean episode length falls from 72.8 to 23.9 turns and mean total tokens from 2.15M to 596k. This seemingly counterintuitive result is consistent with a stronger policy reaching goals directly, while a weaker one spends longer repeating unproductive actions or reasoning. Researchers can use GlyphBench’s replay interface to examine these behaviors and connect differences in return and token use to individual decisions (Appendix B).

Glyph observations give the highest mean CraftaxFull reward for Luna, Terra, and Sol with Guided Memory at high reasoning effort (Figure 4(a)). For Sol, the mean reaches 11.4% of maximum reward, compared with 3.7% for pixels and 0.3% for native text. Glyphs preserve adjacency and spatial layout while naming entities through a legend. This reduces the need to reconstruct a map from coordinate descriptions or identify objects visually, offering a plausible explanation for the advantage. The result also parallels Agentick’s findings on spatial text observations³⁷.

Richer harness support becomes most useful when paired with sufficient reasoning. At high effort, Guided Memory improves Terra and slightly improves Sol, while Luna achieves a higher mean with Basic (panel (b)). Sol’s effort sweep makes the interaction clearer (panel (c)): performance with Basic levels off near 11%, whereas Guided Memory continues improving to 15.8% at maximum effort. The steeper improvement with Guided Memory suggests that additional reasoning helps Sol use the richer memory and guidance. Harness design therefore affects how much an agent gains from additional inference computation, motivating studies that vary both together³⁸.

Astra reaches 51.4% in the exploratory episode with Guided Memory at maximum reasoning effort, the highest individual return in our Craftax evaluation. This run illustrates how far an agent can progress when a stronger model combines the glyph interface with memory and guidance.

Additionally, we use BALROG³⁹ to examine how glyph rendering affects performance outside the GlyphBench task collection (Figure 12, Appendix D). Glyph rendering increases 12 of 20 model-environment means, leaves five unchanged, and lowers three. All five models improve in mean progression on Crafter, while the largest increase occurs for Astra on MiniHack, from 65% to 85%. These comparisons show that the effect of the observation interface depends on both the environment and the model.

Figure 4. CraftaxFull observation, harness, and reasoning-effort comparisons without task-specific fine-tuning (100R/226). Fixed settings are labelled in each panel. Means have 95% t intervals over five episodes; open markers show individual episodes in (a–b). The star in (b) shows one exploratory Astra episode with Guided Memory at maximum reasoning effort.

We next train separate policies to test whether the shared interface and task rewards support learning across all six standard suites. Figure 5 shows two tasks from each suite, with all 25 runs in Appendix F. Training returns improve across the suites, although the pace and stability of learning differ between tasks. These experiments establish that policies can improve through interaction across the environment suite and provide a basis for examining how optimization choices shape that improvement.

Figure 5. Selected single-task learning curves across all six suites. Pale traces show training reward; dark traces show centered seven-point means. Dashed references show initial-policy rollout means. Axis limits are specific to each task. Appendix F gives the selection rule and the full 25-run figure.

We next examine how a fixed amount of interaction is used for policy improvement. In GlyphBench’s Snake environment, we vary rollout group size and learning rate across three difficulty levels while holding the total number of sampled episodes fixed (Figure 6). These choices jointly control the updates made from that experience: smaller groups permit more frequent updates, while learning rate scales each update. A 6,144-rollout budget gives 192 updates at group size 32, compared with 12 at group size 512.

The benefit of frequent updates depends on how much each update changes the policy. At 10⁻⁶, progress is modest and group sizes remain relatively close. At intermediate rates, smaller groups pull ahead: raising the rate to 5×10⁻⁶ increases group 32’s final return on easy Snake from 0.587 to 0.900, with analogous gains on medium and hard. The precise optimum varies with difficulty, but the common pattern is that learning rate and update frequency must be chosen together.

Larger rates eventually erase these gains: 10⁻⁵ and 10⁻⁴ collapse toward zero (Appendix J). Conversely, extending the budget to 25,600 rollouts at 10⁻⁶ preserves group 32’s advantage over group 512 on all three difficulties (Appendix K). Across these Snake tasks, smaller rollout groups are most effective when paired with a suitable learning rate. The experiment illustrates how GlyphBench can be used to study the interaction between optimization choices and task difficulty under a common interaction budget.

Figure 6. Optimization under a 6,144-rollout budget. The 45 conditions shown vary learning rate (rows), Snake difficulty (columns), and group size (colors). Pale traces show batch returns; solid curves show trailing 512-rollout means; dots mark final observations. Return scales match within columns. Appendix J reports higher rates.

The single-task experiments train a separate policy for each task. We now test whether one policy can improve across a mixture of 100 tasks with different rules and reward structures. Mean return on new instances of these tasks rises from 0.012 to 0.319 within 100 gradient steps (Figure 7). This establishes improvement under joint training across the mixture. Evaluation uses held-out environment seeds for the same task types; the following section tests whether the learned capabilities extend beyond the training environments. Appendix G provides task-level learning curves and evaluations by suite.

Figure 7. Learning across 100 GlyphBench tasks over 100 gradient steps. (a) Training returns and a trailing nine-step mean. (b) Evaluation on new environment seeds for the same tasks.

To relate these gains to other common domains for language-model RL, we apply the same optimization recipe to a set of 100 math problems and a set of 100 code problems. In Appendix I, we show the learning curves for these runs. In the following section, we evaluate the policies obtained after training on held-out reasoning problems to compare how their gains transfer beyond their respective training domains.

We now test whether the improvements acquired through gameplay extend to reasoning problems outside the training environments. We evaluate the base model and policies obtained after GlyphBench, Math, and Code RL on the same 3,100 Reasoning Gym problems with a 32K output-token budget (Figure 8). GlyphBench RL reaches 63.48% accuracy, compared with 62.29% for Math RL, 58.98% for Code RL, and 56.04% for the base model. The gain over Math RL is 1.19 percentage points, with a paired 95% interval of [0.38, 2.02].

Figure 8. Reasoning Gym accuracy at a 32K output-token budget for the base model and policies trained on games, math, or code. Each policy answers the same 3,100 problems, with three attempts per problem. (a) Overall accuracy across 31 tasks. (b) Accuracy by category. Whiskers show 95% problem-bootstrap intervals, resampling within tasks and keeping each problem’s three attempts together.

The gains extend across several forms of reasoning. GlyphBench RL leads on algebra, ARC, games, and graphs, while Math RL leads on cognition and geometry. Training through gameplay can therefore improve performance on problems that require a direct answer, even though the policy learned from sequences of actions and environmental feedback. These results provide empirical evidence that gameplay can yield stronger reasoning transfer than math or code RL in this setting.

In Appendices H and I, we report evaluations at additional output budgets, category results, and generation diagnostics. We also evaluate on MathArena and SWE-bench in Appendix H, where the changes are inconclusive.

GlyphBench connects training, evaluation, and inspection of agent behavior through a common interface for language-model RL. Our experiments illustrate how researchers can use it to study the interactions among observation formats, harness design, reasoning effort, and optimization choices. Training through gameplay also improves reasoning beyond the training environments, outperforming math- and code-trained baselines in the main Reasoning Gym experiment. This evidence motivates further study of the capabilities models acquire through interaction and how those capabilities transfer. Future experiments can vary the need for memory, planning, and adaptation to feedback during training, then test how each affects performance on new tasks and on reasoning problems without environment interaction. Bounded tasks make repeated experiments practical, while extended environments and trajectory replay help researchers connect aggregate gains to the decisions behind them. Together, these tools support broader research on how language models learn from rewards, use information to act, and generalize to new problems.