This module introduces temporal logic as a specification language for reinforcement learning. In earlier modules, rewards were treated as numerical signals used to guide behavior. But many intended objectives are not simply properties of one transition. They are properties of whole trajectories: whether something eventually happens, whether something bad is always avoided, whether every request is eventually served, or whether some desirable condition recurs over time. We begin by translating reinforcement-learning trajectories into labeled traces using atomic propositions. We then introduce the syntax and semantics of linear temporal logic. After that, we study common temporal specification patterns, including reachability, safety, response, recurrence, and persistence. The module then explains ω-regular objectives, Büchi automata, product MDPs, and why the choice of automaton matters for reinforcement learning. The goal is not to become logicians. The goal is to gain a precise language for describing trajectory-level behavior in reinforcement learning.

This module develops reward machines as finite-state reward specifications for reinforcement learning. Module 2 introduced temporal logic, ω-regular objectives, Büchi automata, product MDPs, and the limit-reachability bridge from temporal specifications to model-free RL. We now focus on a closely related, but more reward-centered, object: the reward machine. A reward machine is an automaton-like structure that stores task progress and emits rewards. It is useful when the reward depends not only on the current state and action, but also on what has happened before. For example, whether “deliver” is rewarding may depend on whether the package has already been picked up. Whether entering a room is good may depend on whether the key has already been collected. Whether a request has been handled may depend on which request is currently pending. The key idea is simple: environment state + reward-machine state = state with task progress. The environment state records where the agent is. The reward-machine state records what the reward specification remembers. This module treats reward machines as structured reward programs. We define reward machines, explain how they represent non-Markovian rewards, construct product MDPs with reward-machine state, and study examples for common task patterns. We then explain how reward-machine structure supports decomposition, counterfactual experience, interpretability, and auditing. The final lesson surveys richer reward-machine models, including counting, timed, continuous-time, and physics-informed reward machines. The goal is not to repeat the automata theory from Module 2. The goal is to understand reward machines as a practical specification device: a finite-state way to write rewards that depend on task progress.

This module studies how rewards can be made more informative, inferred from evidence, and audited for alignment with intended behavior. So far, we have studied how to specify structured objectives using temporal logic, automata, product MDPs, and reward machines. We now ask a complementary question: how can rewards be shaped, inferred, or learned? A reward can be correct but still difficult to learn from. For example, a sparse reward that gives +1 only when a task is completed may faithfully describe success, but it may provide almost no guidance during early learning. Reward shaping adds additional feedback to make learning easier. However, shaping must be designed carefully: a poorly chosen shaping reward can change the incentives and create new specification-gaming opportunities. The module then turns to methods for learning rewards from evidence. Inverse reinforcement learning infers rewards from demonstrations. Preference-based reinforcement learning infers rewards from comparisons or human feedback. Automata-learning methods can infer structured reward machines from interaction. The final lesson studies how learned rewards should be audited for faithfulness, robustness, and reward hacking. The goal is to understand reward programming as both a design problem and an inference problem

This module studies reward programming when the standard fully observable MDP abstraction is not rich enough. In a standard MDP, the agent observes the state, chooses an action, receives reward, and transitions to a new state. This abstraction is powerful, but many reward-programming problems require more structure. Sometimes reward design is hard because the environment model is missing the right structure. The agent may not observe the full state. The task may require memory, hierarchy, or recursion. Other agents may affect the outcome. Or the system may evolve in continuous time according to physical dynamics. In such cases, a reward that looks complicated may be a symptom of an inadequate model. This module studies richer abstractions for reward programming: partially observable MDPs, beliefs and memory, hierarchical and recursive task structure, multi-agent interaction, and continuous-time dynamical systems. The goal is not to always use the most expressive model. The goal is to choose the simplest model that faithfully captures the task, supports learning, and remains transparent enough to audit.