Reinforcement Learning: On-Policy vs Off-Policy

Introduction to Reinforcement Learning

Reinforcement learning is often introduced through a series of algorithms such as SARSA, Q-learning, PPO, DQN, and SAC. Each name seems to denote a distinct method or a particular mathematical formulation. However, these algorithms hinge on a fundamental question: should an agent learn solely from the behavior it is currently employing, or can it also learn from behavior generated in another way? This question lies at the heart of the distinction between on-policy and off-policy learning.

Definition of Key Terms

To understand this distinction, it is essential to define certain terms. In reinforcement learning, a policy is the rule or strategy that an agent uses to decide which action to take in each situation. An on-policy method learns from the same strategy that the agent is currently following. In contrast, an off-policy method separates the two, allowing the agent to act according to one strategy while learning about another.

This distinction goes beyond mere terminology. It influences crucial properties of a learning algorithm: how it explores, how much data it requires, whether it can learn from past experiences, and the stability of the training.

Practical Examples

Consider the example of a robot learning to navigate a busy warehouse. For safety reasons, its behavior during training must remain conservative. An on-policy method directly improves this conservative behavior, while an off-policy method allows for greater flexibility. For instance, the robot can continue to act cautiously while learning, from the experiences collected, a different strategy that could eventually be more effective. This separation between an agent's behavior and what it learns is the key idea behind off-policy learning.

What You Need to Remember

• On-policy methods learn from the same strategy that the agent uses to interact with the environment. They are often more stable and easier to reason about, but they generally cannot leverage past data as effectively. SARSA is the standard example of on-policy learning.

• Off-policy methods learn a target strategy using data collected from a different behavior strategy. This makes them more data-efficient and allows them to learn from replay buffers, recorded data, or the experience of another agent, but training can be less stable. Q-learning is the standard example of off-policy learning.

• Expected SARSA sits between the two by taking an expectation over subsequent actions, which often reduces variance and can be used in either an on-policy or off-policy framework.

This distinction influences some of the most important properties of a reinforcement learning system, including exploration, sample efficiency, stability, and safety during learning.

Understanding What an Agent is Trying to Learn

Before comparing algorithms like SARSA and Q-learning, it is helpful to understand what the agent is actually trying to learn. In most tabular reinforcement learning methods, the agent does not learn directly from actions; it learns estimates of the quality of different actions in various situations.

A central concept in reinforcement learning is the action value function, typically denoted as Q(s, a). In simple terms, this function measures the quality of action (a) in state (s), taking into account not only the immediate reward but also the future rewards that may follow.

Action Value Function

More precisely, under a policy π, the action value function is defined as the expected return when we start in state s, take action a, and then follow policy π thereafter:

[ Q(s, a) = \mathbb{E}[G_t | s_t = s, a_t = a] ]

where G_t is the total discounted return from time t.

Distinction Between On-Policy and Off-Policy

We need to remember two important terms:

• Target policy (π): the policy that the agent is trying to evaluate or improve.
• Behavior policy (b): the policy that actually generates the experience.

With these definitions, we can state the distinction clearly:

• In on-policy learning, the agent learns about the same policy it uses to act. This means that the target policy and the behavior policy are the same: (π = b).

• In off-policy learning, the agent learns about one policy while following another. In this case, the target policy and the behavior policy are different: (π ≠ b).

This difference may seem minor, but it has significant consequences.

Conclusion

In an on-policy method, the agent improves the strategy it is actually using in the environment. In an off-policy method, the agent can act in one way, perhaps cautiously or randomly to explore, while learning about a different strategy in the background. This separation allows off-policy methods to reuse old data, learn from exploratory actions, and even benefit from the experience collected by another agent.