Self-play Task Generation for the Modal-sdk with Vmax’s Campaign infrastructure

AI research and engineering is increasingly bottlenecked by infrastructure. To increase development velocity, agents that are expert in interacting with infrastructure are required. To evaluate and post-train such agents, we need to run expensive verifiers at scale. For example, when training an agent to set up an inference endpoint or post-training workload, many GPU-based verifiers in sandboxes need to be available simultaneously to grade rollouts.

Modal is an ideal platform for autonomous agents to interact with and to be trained to interact with AI infrastructure. Compute is elastic on Modal, meaning agents can spin up workloads at will and sandboxes with expensive compute resources to grade their proficiency in building AI workloads can be gathered ephemerally. Furthermore, the verifier compute required is not consistent between rollouts, some require access to a single CPUs, multiple CPUs or combinations of CPUs, GPUs and file systems—verification is only possible when these resources can be acquired at will.

Here is an example task that requires a GPU. The solver must port a numpy SVD low-rank approximation to a Modal Function that requests a GPU and computes the decomposition with torch.linalg.svd on a CUDA tensor, matching the numpy results within tolerance. A view of what this rollout would look like in Campaign is embedded below.

Current frontier models can already use the Modal SDK. However, the SDK is fast changing—frontier models are only familiar with SDK features at a snapshot in time of their training corpus. We find that even when supplying frontier models with the latest documentation they can fail to correctly use the SDK. Therefore, to build a model capable of running compute workloads on Modal, tasks need to be crafted that target specific SDK features where models are uninformed. At Vmax, we believe that deciding what a model should learn, and how it should learn it, is an optimization problem to be solved by self-play, rather than one to be solved purely by manual, human engineering.

In this blog, we detail how we generated tasks targeting the failure modes of GPT 5.5 on the Modal SDK. We used algorithms that are integrated into the Campaign task generation infrastructure at Vmax, a self-play task generation system for building tasks tailored towards a given domain. These are results from work we did earlier this summer, which we are sharing as a preview of what Campaign can do.

To build a model capable of translating python code to Modal SDK specific code, we require tasks that target failure modes of LLMs with the Modal SDK and a post-training pipeline for fine-tuning LLMs to overcome these failure modes.

We generate 3 subsets of tasks to build a post-training curriculum of Modal tasks. The didactic task split teaches the syntax of the Modal SDK. The Mulberry task split, where Mulberry is an inference-only self-play method from Campaign, aims to increase the difficulty of tasks in our curriculum while covering the full feature set of the Modal SDK. Finally, our GitHub issue-based task split targets realism, requiring an agent to do Modal translation inspired by issues described in public sources, including Github issues and the Modal docs.

All tasks in our post-training pipeline require translating python code to Modal SDK code. To verify correctness, verifiers check whether three conditions are true. Firstly, the translated code matches the python code for a given set of inputs and outputs. Secondly, it asserts that the translated Modal code uses a given set of SDK contracts that it is instructed to use in its prompt. Examples include: fan-out (.map(), .starmap(), .for_each(), or .spawn_map()on a Modal function), persistent storage (tasks involving modal.Volume) and GPU resources (a valid gpu= request on a reachable Modal function or class). Lastly, we use an LLM as a judge to verify that the compute primitives visible in the AST are actually used to perform computations required by the task's instruction body.

The didactic split was mined agentically, by exploring scientific computing repositories to surface interesting, self-contained workflows that could be constructed as Modal translation tasks. The source repos include NumPy, SciPy, scikit-learn, PyTorch tutorials and examples, NetworkX, scikit-image, xarray, statsmodels, SymPy, pandas, Polars, and Modal’s multinode training guide. This naive task mining pipeline is cheap but does not build a curriculum of increasing complexity. In the initial generated set, prior to filtering for quality control, the didactic training split tasks covered only 12/19 contracts, left 7 contracts at zero coverage, averaged 2.24 contracts per task, and were 91% single remote-function tasks. Representative tasks are small wrappers: numpy-gpu-tiny-attention targets remote execution plus GPU, while secret merge-suffix targets remote execution plus Secrets. To increase complexity and coverage of the tasks generated we developed our Mulberry task generation system.

Mulberry is an inference-only self-play task generation system from Campaign based on our self-play research, including PopuLoRA. Similar in spirit to count-based exploration, Mulberry takes a set of task categories as input, then a setter is directed to cover all categories of tasks, while a solver attempts to solve the tasks generated. Once all categories are covered, setters continue to generate and harden tasks, as measured by their pass@k performance.

Finally, we mine tasks that aim to target failure modes discussed in public sources including Github issues and the Modal docs. This generation requires Github issue-like failure reports and asks the setter to turn them into concrete Modal tasks with verifiable failure modes. Its seed pack contains summaries of Github issues. The generator then takes a domain an input, such as vLLM, checkpointing, logs, or Modal lifecycle behavior, along with a small set of such seed summaries. With this context, the generator is asked to infer a related failure mode and produce a deterministic task using smoke computations for tasks involving large compute workloads.

Starting from a base MoE Qwen3.6-35B-A3B model, we train our agent on 32 H100s with GSPO. We take advantage of the Claude code harness shipped in Slime v0.3.0 to train our agent, removing the subagents and compaction functionality. CISPO is a natural ablation for future runs because it also targets efficient clipped importance-sampling updates for LLM RL, and we recently contributed CISPO support to Slime. We train our agent on a training mix of 45% didactic tasks, 45% Mulberry self-play tasks and 10% GitHub issue-based tasks. We use a relatively small context during training of 28k, and evaluate at that same context as well as at a higher context of 128k. Because Mulberry can generate tasks by hardening previous tasks, we ensure that there are no tasks in the validation set that are hardened versions of the training tasks. Further, no tasks in the validation set share the same workflow example script used as inspiration for task generation as those in training, however they do share the same source repositories (e.g. scikit-learn), stronger generalization claims would require further task generation, ideally in private repositories.

During training, every rollout that passes its task's tests is also checked by an LLM as a judge, a frozen Qwen3.6-35B-A3B model. The judge reads the task's instructions, the Modal capabilities the task requires and the submitted code, and decides whether the code utilizes the Modal structure the task instruction body asks for or only adds Modal decorators (e.g. for a GPU) but does not actually run computations on the GPU. When it flags this sort of shortcut, the rollout's reward is set to zero. At eval time we use an Astra judge with the same prompt and setup from training.

Additionally, we add a modal-docs tool, which lets our agent (and those we compare against) query modal-docs documentation. The tool indexes Modal’s guides, examples, and API reference (derived from modal.com/llms.txt) and exposes two actions: query, which returns the most relevant pages for a set of concept keys, and show, which returns a bounded window of a specific page so long documents can be read incrementally rather than flooding the context. Adding this docs tool ensures that the closed model APIs we compare against (GPT-5.5, GPT-6 Luna and Astra) are able to be used in context learning—allowing us to more accurately ascertain the advantage of post-training versus simply providing the latest SDK documentation to the agent in its context.

We compare the base model, Modal-coder, GPT-5.5, GPT-6 Luna and Astra in the same Claude Code harness on 282 tasks (2 GitHub issue-based, 243 didactic and 37 Mulberry), at 28k and at 128k context. We compare at 28k because this is the context size we used for training to validate the tasks generated. However, we find that for a subset of tasks (particularly in the Mulberry and Github split) there is no practical room for the agents to write their solutions in the tokens available. As a result, we further evaluate at 128k to see the performance of models when context size is not a limiting factor, keeping the task sets consistent between both context settings. At 28k we ran eight rollouts per task for every model, and at 128k we recorded one rollout per task. At 28k, overall mean reward and Pass@1 are shown ± one standard error across the eight runs, where a run is one rollout on every task.

Table 1: Model performance at 28k context, 16k output tokens per turn, eight rollouts per task.

At 28k, Modal-coder is comparable to GPT-5.5 and GPT-6 Luna on overall mean reward (0.567 versus 0.581 versus 0.574), with all models having access to the Modal docs. Astra, at $0.20 per rollout, leads every split. Github Issue-based and Mulberry task splits are the most challenging splits, with Mulberry being the most challenging for Astra.

Table 2: Model performance at 128k context, 16k output tokens per turn, one rollout per task.

† Base Qwen and Modal-coder costs are server costs per attempt. GPT-5.5, Astra and GPT-6 Luna costs are estimated from logged token usage at the rates Azure billed: $5/$30 and $10/$50 per million input/output tokens for GPT-5.5 and Astra, and $0.10/$0.50 for GPT-6 Luna, with cached input at a tenth of the input price and uncached Astra and GPT-6 Luna input billed as cache writes at 1.25 times the input price, using each run's estimated share of cached input. All estimates exclude sandbox, verifier and judge costs, and retried attempts. For the base and trained models, we assume a fully utilized server of 2 H100s at Modal’s list price ($0.001097 per GPU-second, or $0.00265 per second for 2 H100s including the server’s CPU and memory), producing approximately 1,700 output tokens per second in aggregate across concurrent rollouts. This is roughly the throughput we measured running the evaluation tasks at high concurrency, 1,600-2,100 output tokens per second with 32 to 128 agents running at once.

At 128k context, Modal-coder still performs remarkably well given its post-training context limits. However, GPT‑5.5 and GPT-6 Luna do beat the trained Modal-coder model overall (0.694 and 0.658 versus 0.594 mean reward). Astra leads or ties every split again, at 0.924 overall.

Training improved upon base performance on Modal-specific conventions. Across the 26 didactic tasks that require a web endpoint, our model solved 157 of its 208 attempts at 28k, against 23 for GPT‑5.5. An example where GPT 5.5 fails and Modal-coder passes is shown below for the task polars-struct-lookup-endpoint. GPT 5.5 assumes that FastAPI will be installed by default on the image (this used to be true until version 2024.10—released in October 2024). Despite calling the Modal-docs tool it did not conclude that FastAPI would not be installed by default. On the other hand, Modal-coder uses image = modal.Image.debian_slim().pip_install("polars", "fastapi", "uvicorn") to install FastAPI.

A further example shows how the training process imbues Modal specific knowledge is the networkx-wl-sharded-registry task. The base model scored 0/8 on this task while Modal-coder achieved 8/8. The task requires a workflow for parallel deduplication of a set of graphs, in the side by side diff shown below, Modal-coder performs this computation with .starmap, while the base model hallucinates an API “modal.Mapping” that does not exist.

Using our self-play based task generation infrastructure, Campaign, we were able to significantly improve upon the capabilities of the base model which was already post-trained to be a competent software engineer, suggesting that self-play based task generation can enable domain expansion of custom models in unfamiliar engineering environments.

While these results are encouraging, the generated tasks focus on a narrow capability of translation python code to Modal-SDK specific code. A fruitful avenue for future work would be to expose the modal-coder model as a tool call to frontier models when writing Modal code, allowing it to draft workloads and then use Modal-coder to translate the workflow to Modal compatible code, expanding the knowledge of a complete agent system.

Modal provides excellent infrastructure for model post-training, including elastic GPUs and sandboxes for rollouts, read more about Modal’s approach to reinforcement learning here: https://modal.com/blog/reinforcement-learning-infrastructure-problem

If you would like to run a self-play campaign for your own use-case, sign up to Campaign waitlist: https://vmax.ai/.