Skill-Latent Rewards: LLM-Guided Reward Formulation for Hierarchical Reinforcement Learning
Skill‑LR uses vision‑language and large language models to generate subgoal‑aware latent rewards, enabling faster, more reliable learning in long‑horizon hierarchical RL tasks.

Introduction
Long‑horizon reinforcement learning (RL) struggles when feedback is sparse or delayed, making credit assignment across many timesteps difficult. Traditional return‑decomposition methods redistribute rewards but treat each state‑action pair in isolation, ignoring the hierarchical subgoal structure that often governs progress in complex environments. Skill‑Latent Rewards (Skill‑LR) tackles this gap by marrying vision‑language models (VLMs) with large language models (LLMs) to produce subgoal‑conditioned latent rewards. The approach yields stage‑specific credit that evolves with the agent’s plan, reducing temporal variance and improving sample efficiency.
How Skill‑LR Works
- Subgoal Identification – At each timestep a VLM processes the agent’s visual observation and extracts a textual description of the currently active subgoal (e.g., “pick up the key” or “open the door”).
- Two‑Stage Prompting – The identified subgoal is fed to an LLM in a carefully crafted prompt. First, the LLM contextualizes the subgoal within the overall task hierarchy; second, it generates a concise latent reward formulation that reflects progress toward that subgoal.
- Reward Injection – The latent reward is combined with any environmental signal and used by the hierarchical RL controller to update both low‑level skill policies and the high‑level planner.
By aligning reward signals with the hierarchical plan, Skill‑LR provides dense, interpretable feedback without hand‑crafting shaping functions.
Empirical Highlights
Skill‑LR was evaluated on multi‑stage MiniGrid and BabyAI environments, including the classic DoorKey and the more demanding Unlock‑to‑Unlock scenarios. Compared to sparse environmental rewards and strong baselines such as RUDDER and LaRe, Skill‑LR:
- Converged in fewer episodes, often halving the sample count needed for comparable performance.
- Showed greater stability across random seeds, reducing variance in final success rates.
- Produced reward signals that were human‑readable, aiding debugging and policy inspection.
These results suggest that subgoal‑aligned latent rewards can serve as a principled alternative to handcrafted shaping, especially when tasks exhibit clear hierarchical structure.
Practical Takeaways
- When to use Skill‑LR – Ideal for environments where subgoals are visually observable and can be described in natural language (e.g., navigation, object manipulation). Less suited for purely abstract state spaces without a clear visual component.
- Implementation tips – Leverage pre‑trained VLMs (such as CLIP‑based models) for robust subgoal extraction, and choose an LLM with strong instruction‑following capabilities. Prompt engineering is crucial: include the overall task description, the current subgoal, and a request for a scalar reward.
- Scalability – The overhead of VLM and LLM inference is modest relative to environment simulation time in many benchmark settings, but may become a bottleneck in real‑time robotics; batching and model distillation can mitigate this.
Conclusion
Skill‑LR demonstrates that large language models can go beyond text generation to serve as dynamic reward designers, bridging the gap between hierarchical planning and credit assignment. By grounding rewards in subgoal semantics, the framework offers a clear path toward more sample‑efficient, interpretable RL in long‑horizon tasks, and opens avenues for future work on automated reward shaping in domains where subgoals are less visually explicit.
Source
Read original sourceWhy we picked this
Combines vision‑language models and LLMs for hierarchical RL reward shaping, notable AI research.