LERa: Replanning with Visual Feedback in Instruction Following

Svyatoslav Pchelintsev1*, Maxim Patratskiy1*, Anatoly Onishchenko1, Alexandr Korchemnyi1, Aleksandr Medvedev2, Uliana Vinogradova2, Ilya Galuzinsky2, Aleksey Postnikov2, Alexey K. Kovalev1,3, Aleksandr I. Panov1,3
1MIPT, Dolgoprudny, Russia, 2Sberbank Robotics Center, Moscow, Russia, 3AIRI, Moscow, Russia
*Equal contribution

LERa is a robust and adaptable solution for error-aware task execution in robotics.

LERa Example Figure

Abstract

Large Language Models are increasingly used in robotics for task planning, but their reliance on textual inputs limits their adaptability to real-world changes and failures. To address these challenges, we propose LERa — Look, Explain, Replan — a Visual Language Model-based replanning approach that utilizes visual feedback. Unlike existing methods, LERa requires only a raw RGB image, a natural language instruction, an initial task plan, and failure detection—without additional information such as object detection or predefined conditions that may be unavailable in a given scenario. The replanning process consists of three steps: (i) Look, where LERa generates a scene description and identifies errors; (ii) Explain, where it provides corrective guidance; and (iii) Replan, where it modifies the plan accordingly. LERa is adaptable to various agent architectures and can handle errors from both dynamic scene changes and task execution failures. We evaluate LERa on the newly introduced ALFRED-ChaOS and VirtualHome-ChaOS datasets, achieving a 40% improvement over baselines in dynamic environments. In tabletop manipulation tasks with a predefined probability of task failure within the PyBullet simulator, LERa improves success rates by up to 67%. Further experiments, including real-world trials with a tabletop manipulator robot, confirm LERa's effectiveness in replanning. We demonstrate that LERa is a robust and adaptable solution for error-aware task execution in robotics.

Experimental Results

Replanning results for different agents and environments.
Agent ALFRED-ChaOS (Seen) ALFRED-ChaOS (Unseen) VirtualHome-ChaOS PyBullet (gpt4o) PyBullet (Gemini)
SR↑GCR↑SRep↑ SR↑GCR↑SRep↑ SR↑GCR↑SRep↑ SR↑GCR↑SRep↑ SR↑GCR↑SRep↑
Oracle 33.0450.04- 31.6551.71- 50.0085.75- 19.0032.50- 19.0032.50-
O-Ra 34.3851.197.69 34.1754.0814.16 50.0084.330.00 53.0061.5839.13 56.0073.6746.39
O-ERa 40.1856.4037.08 42.8161.8133.33 50.0085.920.00 75.0080.5071.24 72.0078.2565.65
O-LRa 34.3851.006.73 33.4553.5711.01 93.0097.0487.00 79.0083.3373.71 87.0092.9285.81
Baseline 33.0450.153.15 32.0151.890.98 52.0085.914.10 67.0074.9259.06 82.0089.0878.90
O-LERa 49.5564.5573.39 53.6070.2374.57 94.0698.1795.03 67.0072.6761.29 86.0089.1784.83

Performance of LERa with different VLMs across different environments.
VLM ALFRED-ChaOS (Seen) ALFRED-ChaOS (Unseen) VirtualHome-ChaOS PyBullet
SR↑GCR↑SRep↑ SR↑GCR↑SRep↑ SR↑GCR↑SRep↑ SR↑GCR↑SRep↑
LLaMA-3.2-11b 35.7152.0111.11 33.8153.819.38 52.0074.1720.00 34.0045.9220.54
LLaMA-3.2-90b 38.8454.8030.58 36.3355.9725.93 54.0084.258.00 64.0071.0857.68
Gemini-Flash-1.5 46.4361.6167.22 51.4468.3867.71 59.4072.5124.00 55.0066.3346.75
Gemini-Pro-1.5 42.1956.5156.16 46.4063.8555.64 65.3587.8741.58 86.0089.1784.83
gpt-4o-mini 43.7559.7146.81 46.0463.9749.12 74.2582.5056.25 48.0062.9237.75
gpt-4o 49.5564.5573.39 53.6070.2374.57 94.0698.1795.03 67.0072.6761.29
The impact of the imperfect checker on replanning.
Agent Seen split Unseen split
SR↑ GSR↑ SRep↑ SR↑ GSR↑ SRep↑
O-15 12.50 35.90 - 12.95 39.75 -
O-10 17.41 40.89 - 17.63 43.38 -
O-05 24.55 45.01 - 24.82 47.54 -
O-FC 33.04 50.04 - 31.65 51.71 -
O-L-15 22.32 44.42 35.48 20.50 45.29 25.00
O-L-10 25.45 46.91 47.32 25.18 48.95 28.70
O-L-05 34.38 52.57 54.16 35.97 55.76 35.43
O-L-FC 44.64 59.00 52.83 46.76 63.46 49.34

Real Robot Experiments

BibTeX

@inproceedings{patratskiy2025lera,
  title={LERa: Replanning with Visual Feedback in Instruction Following},
  author={Pchelintsev, Svyatoslav and Patratskiy, Maxim and Onishchenko, Anatoly and Korchemnyi, Alexandr and Medvedev, Aleksandr and Vinogradova, Uliana and Galuzinsky, Ilya and Postnikov, Aleksey and Kovalev, Alexey K. and Panov, Aleksandr I.},
  booktitle={IROS 2025},
  year={2025}
}