Published at the Conference on Language Modeling (COLM) 2026

Smarter by the Moment

Environment-Driven Dynamic Policies for Continual LLM Improvement

Ting-Wei Chang1, Po-Chun Chen1, Hen-Hsen Huang2, Hsin-Hsi Chen1,3

  • 1Department of Computer Science and Information Engineering, National Taiwan University
  • 2Institute of Information Science, Academia Sinica
  • 3AI Research Center (AINTU), National Taiwan University

An LLM agent that answers one task after another tends to make the same kind of mistake over and over. Memory helps, but retrieving a similar past question only replays an example; it never says what to do differently. DRPG adds the missing half: a second model reads what the agent got right and wrong, and writes a handful of rules, a policy, that the agent follows on the next question. The policy is rewritten at every step, from a history that keeps growing.

What the agent wrote for itself

Both policies below were generated by llama-4-scout at step 100 of a real run, from its own correct and incorrect answers up to that point. The difference between them is the paper’s main finding about when this mechanism pays off.

Spider, step 100 of the stream

  • Directly address the question. Focus on directly answering the question asked, avoiding unnecessary joins or conditions that do not contribute to answering the query.
  • Validate schema and relationships. Verify the schema and relationships between tables to ensure correct joins, subqueries, and conditions are used.
  • Precise use of SQL constructs. Choose SQL constructs that accurately handle the query requirements, such as using aggregation or grouping when necessary, and handle edge cases like empty result sets or division by zero.
  • Avoid ambiguous or redundant results. Ensure that queries are clear and unambiguous, directly answering the question without providing redundant or unnecessary information.
  • Handle multiple results correctly. Ensure correct usage of subqueries, aggregations, and grouping to handle complex queries with multiple results, considering all required columns and potential duplicates.

Every rule here travels. Schema validation and join discipline apply to the next SQL query whatever it asks about, which is why this run gained 8.5 points over Self-StreamICL on Spider.

How a step works

Nothing is trained. The agent keeps a memory of everything it has answered, tagged with the environment’s verdict, and two retrievals run against it: one feeds the agent examples it got right, the other feeds the policy generator an equal number of successes and failures to compare.

# at time step t
R_t   = retrieve(x_t, correct cases)                  # examples for the agent
R'_t  = retrieve(x_t, correct) ∪ retrieve(x_t, wrong)   # k/2 each, contrastive
P_t   = policy_generator(R'_t)                        # up to five rules
ŷ_t   = agent(x_t, R_t, P_t)
fb_t  = environment(x_t, ŷ_t)                         # correct: 1 or 0
memory.add(x_t, ŷ_t, fb_t)
Diagram of the DRPG framework: an input question goes to past experiences, which feed retrieved examples to the agent and contrastive examples to the policy generator; the generator's policy goes to the agent, whose answer goes to the environment, whose binary feedback returns to past experiences.
The policy generator is a separate call and can be a separate model: a smaller or cross-family generator works about as well, and sometimes better.

What it costs

Two LLM calls per question instead of one, and two retrievals instead of one. On mistral-medium × Spider that comes to 1,676 input and 143 output tokens per question, against 585 and 17 for Self-StreamICL: about three times the tokens, for a method that needs no gradient updates and no labelled training data.

Where it helps, and where it doesn’t

Seven LLMs across three families (Gemini, Llama and Mistral) on six streaming benchmarks. The counts below are how many of the seven models improved under DRPG against each baseline.

DRPG wins, out of 7 models per benchmark.
DRPG against Spider CoSQL BIRD HotpotQA DDXPlus DS-1000
Zero-shot 7/76/77/75/77/73/7
Self-Refine 7/76/77/75/76/74/7
Self-StreamICL 7/76/75/75/73/73/7

Text-to-SQL is where errors are structurally regular, so five rules cover a lot of ground. DS-1000 spans seven Python libraries whose APIs share no common failure mode, and DDXPlus has 49 diagnoses; there the policy collapses into a list of special cases and plain instance retrieval stays ahead. The paper reports full per-cell scores, significance tests, and ablations on retrieval strategy, policy continuity and the environment feedback itself.

Paper, code and contact

The code release contains the DRPG agent, the six benchmark environments, the configs behind every table, and a walkthrough for running it on a model of your own.

BibTeX

@inproceedings{chang2026drpg,
  title     = {Smarter by the Moment: Environment-Driven Dynamic Policies for
               Continual LLM Improvement},
  author    = {Chang, Ting-Wei and Chen, Po-Chun and Huang, Hen-Hsen and
               Chen, Hsin-Hsi},
  booktitle = {Conference on Language Modeling (COLM)},
  year      = {2026}
}
Questions about the work
Ting-Wei Chang on LinkedIn