Preprint · 2026
Memory Anchors for Continual Robot Learning
Corresponding contact: maxjdu@stanford.edu
Abstract
When learning a new task on a robot, it is common to rehearse past experiences to prevent catastrophic forgetting of past skills. This approach works very well across architectures and task sets, and on real robots too. But why does it work so well? Our work shows that a small set of these experiences contributes greatly to anchoring past performance. Withholding even a small fraction of them increases catastrophic forgetting significantly.
Our insight: new, unseen tasks already interact with existing knowledge within the policy. If a new task has high representation overlap with past tasks, learning the task can cause forgetting when the new task requires different actions. A simple example is learning how to manipulate a familiar object in a new way. The policy is most sensitive to the past task data sampled within this conflict region. We call these experiences Memory Anchors.
We present a simple way to find Memory Anchors for any new task by using the current policy's latent structure and action predictions. Excluding only 10% of Memory Anchors before sampling the buffer leads to more than a 4.5x increase in catastrophic forgetting on the LIBERO benchmark suites. Conversely, enriching the replay buffer with Memory Anchors can decrease high-conflict task forgetting by 63%. We show that these trends also hold on VLA models and on two real-world task suites.
Watch Overview Video
01 · Continual Learning Challenge
Data Sensitivity in Continual Robot Learning
To understand Memory Anchors, we first need to take a closer look at why robot policies forget old tasks when learning new ones.
Sequential Task Learning
We explore the problem setting of imitation learning on tasks in sequence. Drag the slider to step through ten tasks. The horizontal axis is the learning progress, and the vertical axis is the evaluated tasks. The lighter the off-diagonal elements, the worse the forgetting.
Visual inspired by Huihan Liu
Rehearsing Past Tasks: Training a new task is nearly guaranteed to destroy past task knowledge without some regularization. In our work, we study a common regularization approach called Experience Replay (ER). The idea is simple: keep a buffer of past experiences and replay them during training.
Measuring Retention: We measure a policy's past task retention through Negative Backward Transfer (NBT), visualized as the shaded area below. Click through different ER buffer sizes to see how they affect this score. Higher is worse, lower is better.
Negative Backward Transfer (NBT) Score: 0.42
Swipe across the figure to read →
Show Experiment Details
Diffusion U-Net policy with Resnet-18 encoder. Results shown on the canonical LIBERO-Goal ordering on the 50 supplied test-time reset states. Buffer percentage represents percentage of the past task data used in the ER buffer, sampled randomly at the start of every task.
New Task Interactions with Existing Tasks
Most robot continual learning works look at this NBT metric, but not many look at how it arises. Does a policy forget past tasks slowly, or all at once? Below, we plot a single task's performance across different learning orders. Look at how this task's performance (task 6, red line) changes as the model learns task 8 (orange region).
Show Experiment Details
Diffusion U-Net policy trained on the first few supplied LIBERO task training orderings where task 8 comes after task 6. Using a 1% buffer size.
What we found: forgetting a task does not occur gradually. When some tasks are learned, other tasks experience a steep decline in performance. To understand why this happens, let's look at our conflicting task pair, 6 and 8. The target objects in the tasks are next to each other, which makes the observations in the two tasks similar at the start.
Task 6 (Old) · Cream Cheese in Bowl
Task 8 (New) · Bowl on Plate
Initial Representation Overlap
If two similar tasks are trained simultaneously, the policy will learn to tell their states apart. But if they are trained in sequence, a policy may incidentally collapse the observation space of the second (new) task into the space of the first one. This happens because the latent space was never constructed with the new task in mind, and broad visual similarities can lead to full overlap.
We indeed see this phenomenon for the Cream Cheese to Bowl and Bowl to Plate tasks described above. In the latent space of a policy trained on Cream Cheese to Bowl, the new Bowl to Plate task has many states in the overlap of the past task representations.
Initially, the current policy's observation encoder collapses Bowl to Plate (new task) into the same latent space as Cream Cheese to Bowl (existing task)
Conflicting Actions
Collapsing new tasks into existing representation spaces can create conflict if the required actions are different. We plot the state novelty of the new Bowl to Plate task relative to all previous tasks by finding the nearest neighbor distance in the policy's latent space. On the same graph, we plot action disagreement by comparing the predicted actions of the policy with the ground truth actions.
New trajectory: Bowl to Plate
The two highlighted regions show areas of high state similarity paired with high action disagreement.
What we found: New task data on a policy has regions of combined high state familiarity and high action disagreement. Two regions are highlighted in the example above. The first region is the conflict between reaching for the Cream Cheese and the Bowl. The second region is the conflict between the Bowl to Plate and previous Bowl-related tasks that arises when the robot moves the bowl towards the plate.
Our claim: Continual learning conflicts arise when the new data contains overlapping state representations but different actions. Past task data similar to these conflict regions is especially important in regulating catastrophic forgetting. We call this subset of old task data: Memory Anchors.
Show Experiment Details
Nearest neighbor plot: computed nearest neighbor from the old task data to the current observation using the policy's own observation latent space. Action disagreement plot: added noise to the action label, denoised it using the diffusion policy, and compared the denoised output to the action label.
02 · Finding Memory Anchors
Finding Memory Anchors
The Three-Step Approach
To test our claim, we must extract the Memory Anchors. We present a three-step filter approach that uses the current policy, old task data, and the new task data.
- State Representation Overlap. Using the latent space of the current policy, find the transitions in the new data that overlap with the old task data.
- Action Prediction Disagreement. Within this state overlap of the new task data, identify transitions where the current policy predicts something significantly different from the action labels.
- Anchor Selection. Using the current policy's latent space again, select the old task data that are closest to the new task data isolated in the previous action disagreement step.
The figure below illustrates the whole process, which is described in detail with interactive visualizations in the sections below.
Swipe across the figure to read →
1) State Representation Overlap
Old and new task data are both placed into the current policy's representation space. Latent distances between new and old data naturally segment the new task data into two groups: inside the representation overlap and outside the overlap. Using clustering, we isolate the new task data within this overlap. Look at how this approach segments the Bowl to Plate task:
Existing Tasks
New Task: Bowl to Plate
Inside Overlap
Placeholder explanation of why this transition falls inside the state-overlap region.
Example 1/12
FAQ · What representations do you use?
You could use any representation that the current policy uses, but we found the most success using the observation embeddings, a combination of visual embeddings and language embeddings. You can compute the stats (mean, median, nth quartiles) of the pairwise distances between new and old embeddings, and use these features in a K-means clustering algorithm.
FAQ · Why not segment by timestep?
While it is true that LIBERO typically starts in state overlap and then diverges, doing so requires tuning the timestep threshold. This heuristic is also not generally true. In our real robot experiment, state overlap is interleaved throughout a new task trajectory.
FAQ · What if the new task is very different?
If the new task is in an entirely different environment (as happens in LIBERO-Long), clustering still separates closer from further states, as an ordering still exists in latent space. We find that Memory Anchors still impact highly heterogeneous task sequences (see results), though the selection process becomes less interpretable.
2) Action Disagreement
In this step, we want to further subsample the new task data by isolating the representation overlap points that require different actions from what the current policy predicts. To measure action disagreement, we add noise to the action labels, run denoising using the diffusion policy, and compare to the original labels. Look at how action disagreement happens in the Bowl to Plate task:
New Task Observation (from State Overlap)
Action Disagreement
Predicted Action Agrees
Placeholder explanation for agree example 1: despite the state overlap, the policy predicts an action consistent with the new task.
Example 1/8
FAQ · Why can you trust the predictions of a policy fed with new task data that it hasn't seen yet?
The new task data fed into the policy in Step #2 has already been filtered to be within the latent space of old task data. This doesn't guarantee that it will be fully in-distribution (e.g. the language description will be new), but it increases the likelihood of the policy producing a meaningful denoising. Even if the new task data is fully out of distribution (e.g. new environment), the result is letting all of the representation overlap data through to anchor selection, which is a valid strategy when the new data is very different.
FAQ · Why not just run inference on the data and compare to action labels?
This approach also works, but it can produce false positives for multimodal data with multiple correct actions. By adding noise, we retain enough action structure to preserve the chosen modality if the policy agrees with the action.
3) Anchor Selection
That action disagreement subset of the new task data is most likely to cause catastrophic forgetting during training. The old data most similar to this subset therefore plays a critical role in anchoring past performance; we call this data the Memory Anchors. Below are examples of Memory Anchors extracted for the Bowl to Plate task. Hover over the images to see the tasks they come from. Notice how they focus on critical decision areas, including the bowl grasp and the initial object reaching.
Frame 10
Frame 43
Frame 14
Frame 59
Frame 45
Frame 13
Frame 62
Frame 52
Frame 12
Frame 14
Frame 4
Frame 12
Frame 11
Frame 12
Memory Anchor
Hover/tap over a Memory Anchor for more details.
FAQ · Is the whole process resource intensive?
The whole extraction process takes up a very small percentage of the training run (~3%) and, unlike other enhanced ER methods, does not require any per-step costs or gradient computations. Extracting Memory Anchors does scale with the size of the past task dataset. If the dataset becomes unwieldy, it can be subsampled and/or the selection processes can be approximated to bound the compute.
03 · Impact of Memory Anchors
Impact of Memory Anchors
In the experiments below, we test our claims about Memory Anchors on the LIBERO continual learning benchmark and real-world robot tasks.
Taking Memory Anchors Away
Memory Anchors are naturally present in a randomly selected ER buffer (RandER). To test their importance, we conduct a simple experiment. We remove some Memory Anchors (top 1%, 5%, 10%) from the sampling process and measure catastrophic forgetting.
What we found: the fewer Memory Anchors available, the worse the forgetting, even as the total number of memories in the ER buffer stays the same.
Swipe across the chart to read →
Show Experiment Details
Using an ER buffer size of 1000 (total, not per task) across all trials. The only difference between all experimental conditions is the past task dataset. In the RandER condition, all of the data is equally likely to be chosen. In the RandER w/o X% condition, the X% best Memory Anchors are removed from the datasets before running RandER sampling. The reported values are average NBT (Negative Backward Transfer), error bars show ±1 SEM.
Adding More Memory Anchors
If taking away Memory Anchors worsened forgetting, then adding more Memory Anchors should reduce it. We test this by adding Memory Anchors into the ER buffer (10% / 20%) and then filling the rest randomly (AnchorER). Below, we graph the worst-case NBT, computed from the six most conflicting task-task pairs identified from RandER baseline runs.
What we found: by enriching the buffer with more Memory Anchor samples, catastrophic forgetting is reduced significantly on the worst task-task interactions.
Swipe across the chart to read →
Show Experiment Details
Using an ER buffer size of 1% of the past data size. The randomly selected Memory Anchors can be enough to support many task-task pairings, which is why we chose to focus on the worst-case interaction subset for this experiment. This subset was found by looking at all of the task-task interactions on a set of RandER runs and picking the 6 interactions with the highest average forgetting across training orders. We report the average forgetting on these interactions for AnchorER and RandER.
FAQ · Why not use all Memory Anchors?
As seen in some of the Memory Anchor examples in the last section, Memory Anchors concentrate around critical areas of high conflict. They are critical for maintaining past task performance (as supported by our subtraction experiment), but they are not the only critical pieces of data in the ER buffer. Having high buffer diversity is also important to fully cover the support of past tasks.
On VLA models
All the results so far have been reported on a diffusion policy trained from scratch. We verify that Memory Anchors still impact larger, pretrained models like the π0.5 VLA. Complementary to the findings in this paper, we did observe that VLAs have near-perfect task retention with sufficient memory budgets. However, at the proportionally smaller ER buffer sizes used in our experiments, the π0.5 VLA actually performed worse than the from-scratch diffusion policy. Under these conditions, adding more Memory Anchors was beneficial. Removing Memory Anchors also further degraded performance.
VLA Continual Learning Results
Click on the two experimental conditions to see examples of how Memory Anchors impact VLA performance. The lighter the off-diagonal cells, the worse the forgetting.
RandER: keeping behind 1% of past data in the ER buffer leads to low task retention.
Reducing Memory Anchors
On the π0.5 VLA, performance degrades as Memory Anchors are removed from the past task data before the ER buffer is sampled.
Enriching Memory Anchors
On the π0.5 VLA, enriching the ER buffer with Memory Anchors (10%, 20% of buffer) reduces forgetting over Random ER.
FAQ · Why are the RandER numbers different in the two experiments?
Consistent with our experiment setup on diffusion policy, we use a 1% buffer size for the addition experiment and a 1000-memory total buffer for the subtraction experiment. We use the larger, fixed-size buffer for the subtraction experiment because RandER does better, showing us that Memory Anchors are still important even at a larger budget. Likewise, we use a smaller buffer on the addition experiment to show that Memory Anchors can improve performance even with limited data. This finding is consistent between VLAs and diffusion policies.
Real World: Jar Opening Task
The simulated LIBERO benchmark shows that Memory Anchors are effective in long sequences of tasks, but do these findings transfer to a real robot? To investigate, we make a diagnostic task suite on a real robot, OpenJar. It consists of three visually similar jars that require different strategies: Counterclockwise, Clockwise, and Lift. Unlike the LIBERO benchmark, which has only a few decision points per task, OpenJar requires the robot to make a decision every time the gripper is moving towards the jar. For the screw-type jars, this happens many times during the task.
Task 1: Counterclockwise
Open the jar by rotating the lid counterclockwise and then lifting.
Task 2: Clockwise
Open the jar by rotating the lid clockwise and then lifting.
Task 3: Lift
Open the jar by directly lifting the lid.
We train a diffusion policy on the jar sequence and compare the success rates with AnchorER and RandER. The videos below show a representative rollout from each of the evaluated tasks in the training order. Hover/tap over a clip to see additional details.
AnchorER (Ours)
RandER (Baseline)
OpenJar Continual Learning Task
Hover/tap over a clip for more details.
What we found: Memory Anchors impact real robot performance, both reducing forgetting and improving the learning of new tasks compared to RandER. Interestingly, the targeted coverage of AnchorER mitigates an overregularization effect caused by the large ER buffer on the second task. The final success rate of AnchorER on the three tasks is 1.7x higher than RandER's. Further results and visualizations are shown in the summary figure below.
Swipe across the figure to read →
Show Experiment Details
Diffusion U-Net policy using a pretrained TIMM image transformer encoder. Using 5000 memories for both ER experimental conditions. Data collected using the UMI interface. See paper appendix for full details. Click here for some uncut evaluation videos.
Show Memory Anchor Examples
Below are the top 100 Memory Anchors extracted from the first task when we start training the second task. Notice how they all focus on the moment when the robot is about to move forward and contact the jar. This is the exact moment that differentiates counterclockwise and clockwise jar rotations.
Swipe across the figure to read →
Real World: Sweater Folding Task
The OpenJar task was designed to be diagnostic of continual learning challenges. We conduct another set of experiments on SweaterFold, a realistic sweater folding scenario where a robot learns a sequence of garment manipulation skills. These tasks have high overlap near the start but require very different actions.
Task 1: Fold Left
Left arm reaches towards the left sleeve and folds it over the chest.
Task 2: Fold Right
Right arm reaches towards the right sleeve and folds it over the chest.
Task 3: Fold Up
Both arms grab the bottom of the sweater and fold it upwards.
As before, we train a diffusion policy with AnchorER and RandER. The videos below show a representative rollout from each of the evaluated tasks in the training order. Hover/tap over a clip to see additional details.
AnchorER (Ours)
RandER (Baseline)
Sweater Folding Continual Learning Task
Hover/tap over a clip for more details.
What we found: Consistent with the OpenJar results, Memory Anchors also impact the continual learning performance of SweaterFold by reducing catastrophic forgetting after learning Fold Up and improving the learning of the Fold Left task. Further results and visualizations are shown in the summary figure below.
Swipe across the figure to read →
Show Experiment Details
Diffusion U-Net policy using a pretrained TIMM image transformer encoder and a CLIP language encoder finetuned during training. Using 125 memories for both ER experimental conditions. Data collected using the UMI interface. See paper appendix for full details.
04 · Conclusion
Conclusion, Limitations, and FAQ
We introduced the idea of Memory Anchors, a small subset of past task data critical for maintaining performance during sequential training with Experience Replay (ER). We proposed a method for identifying Memory Anchors and enriching the ER buffer with them. Through a series of studies in simulation and on a real robot, we demonstrated the importance of Memory Anchors in continual learning for overall task performance, high-conflict tasks, and on large pretrained models.
Limitations
- Data and model access: Finding Memory Anchors requires access to past training data and the model's representation space. For proprietary models, this may not be feasible. A workaround is to extract Memory Anchors from an alternate, open-access dataset and use that dataset for the ER buffer. We leave this for future work.
- Must compute many representations: Identifying Memory Anchors requires distances in latent space, which means computing representations for many data points. For large past datasets, this can be bounded by subsampling the sets.
- Not the only factor: As seen with the AnchorER enrichment, we can't fill an ER buffer with only Memory Anchors, as the diversity suffers and, subsequently, so does the performance. Beyond Memory Anchors, diversity and other data qualities also influence continual learning.
FAQs
We have addressed specific FAQs / experiment details in their relevant sections, and here we address general FAQs about Memory Anchors and Continual Learning.
FAQ · Why do we care about forgetting past tasks?
In some adaptation frameworks, we only care about positive transfer: the ability to leverage past tasks to learn a new one. But if finetuning a model on a new task causes the past tasks to degrade, we are no longer making a generalist policy. And as we saw with our real world results, dealing with catastrophic forgetting means dealing with the relationships between tasks, which also help with forward transfer.
FAQ · Do large pretrained models solve catastrophic forgetting?
With a large enough buffer, large pretrained models indeed have great sequential learning performance. We encourage you to check out this paper. However, at the low ER buffer sizes in our experiments (around 20x smaller than past works on VLA), pretrained models still struggle. In fact, they perform worse than a from-scratch diffusion policy, but they also respond more strongly to Memory Anchor enrichment.
FAQ · Why do you care about small ER buffer sizes?
Having access to a large ER buffer during training can become resource intensive. We experiment with ER buffer sizes that are small relative to the past task experience size, which represents a practical setup for continual learning in the real world with longer task sequences. These frugal buffers also highlight the importance of data and Memory Anchors.
FAQ · I'm trying to do robot continual learning. What are your recommendations?
A randomly selected ER buffer already contains Memory Anchors, so it is always good to start with a random buffer. Random ER may be sufficient if you are using a large buffer (>5% of the total dataset) and training on tasks that are very different from one another (e.g. tasks in completely different environments with different objects). But if you are encountering catastrophic forgetting, consider computing Memory Anchors using our method and seeing if enriching your ER buffer provides some improvement.
05 · BibTeX
BibTeX
@misc{du2026memoryanchorscontinualrobot,
title={Memory Anchors for Continual Robot Learning},
author={Maximilian Du and Zhanyi Sun and Chen Xu and Paarth Shah and Masha Itkina and Shuran Song},
year={2026},
eprint={2608.26545},
archivePrefix={arXiv},
primaryClass={cs.RO},
url={https://arxiv.org/abs/2608.26545},
}