docs: synchronize missing English translations

Author: sanbuphy

docs/.vitepress/config.mjs

@@ -1819,7 +1819,7 @@ const enSidebar = {
           items: [
             {
               text: '9.1 Deterministic Policy Gradients and DDPG',
-              link: '/en/chapter11_continuous_control/deterministic-policy-gradient-ddpg'
+              link: '/en/chapter11_continuous_control/ddpg'
             },
             {
               text: '9.2 TD3 and SAC',
@@ -2366,8 +2366,8 @@ const enSidebar = {
           collapsed: false,
           items: [
             {
-              text: 'C.1 Paper Reading Guide',
-              link: '/en/appendix_paper_reading/paper-reading-guide'
+              text: 'C.1 Learning Resources and Project Roadmap',
+              link: '/en/appendix_paper_reading/learning-resources'
             },
             {
               text: 'C.2 GPU Hours Estimation Table',

docs/en/appendix_gpu_hours/gpu-hours-estimation.md

@@ -186,7 +186,7 @@ Suitable for learning the full process of RLHF/RLVR/DPO, and can complete a full
 
 - Reproduce the training curves of R1-Zero on GSM8K using GRPO ([Chapter 18](../chapter18_grpo/grpo-practice-and-mechanism))
 - Fine-tune on Anthropic HH-RLHF data using DPO ([Chapter 14](../chapter17_dpo/dpo-objective-derivation))
-- Run SAC/TD3 on CartPole / MuJoCo ([Chapter 9](../chapter11_continuous_control/deterministic-policy-gradient-ddpg))
+- Run SAC/TD3 on CartPole / MuJoCo ([Chapter 9](../chapter11_continuous_control/ddpg))
   :::
 
 ### Multi-GPU Experiments (7B–13B Models)
@@ -301,4 +301,4 @@ Next Steps Recommendation:
 
 - **Do a Baseline Experiment**: Refer to the GRPO/DPO code in [Appendix D Code Quick Reference](../appendix_code_cheatsheet/sft-kl) and run it on a single GPU.
 - **Plan a Mid-Scale Experiment**: Refer to the distributed training and monitoring sections in [Appendix B Engineering Practices](../appendix_industrial_training/training-debugging).
-- **Read the Cost Disclosure in Frontier Papers**: Look for training details in the tech reports in [Appendix F](../appendix_paper_reading/paper-reading-guide).
+- **Read the Cost Disclosure in Frontier Papers**: Look for training details in the tech reports in [Appendix C](../appendix_paper_reading/learning-resources).

…dix_paper_reading/paper-reading-guide.md …ndix_paper_reading/learning-resources.mddocs/en/appendix_paper_reading/paper-reading-guide.md renamed to docs/en/appendix_paper_reading/learning-resources.md docs/en/appendix_paper_reading/paper-reading-guide.md renamed to docs/en/appendix_paper_reading/learning-resources.md

@@ -362,17 +362,29 @@ The shortest path from S to G hugs the cliff edge and takes 11 steps. But it is
   <em>Source: <a href="https://gymnasium.farama.org/environments/toy_text/cliff_walking/" target="_blank" rel="noopener noreferrer">Gymnasium Documentation - Cliff Walking</a>. The original environment is adapted from Sutton and Barto, <em>Reinforcement Learning: An Introduction</em>, Example 6.6.</em>
 </p>
 
-**Q-Learning** (off-policy) uses $\max_{a'} Q(s',a')$ in its target, which assumes that the future always takes the optimal action. Therefore it learns the shortest cliff-hugging path.
+The two algorithms behave differently in CliffWalking because one term in their update targets is different.
 
-**SARSA** (on-policy) uses the target $r + \gamma Q(s', a')$, where $a'$ is the action actually sampled from the current behavior policy (typically $\varepsilon$-greedy). Since SARSA accounts for the risk of exploration-induced mistakes inside its update, it tends to learn a safer path away from the cliff, even if it is longer.
+**Q-Learning** uses
+
+$$
+Q(s,a) \leftarrow Q(s,a) + \alpha\left[r + \gamma\max_{a'}Q(s',a') - Q(s,a)\right].
+$$
+
+Its target is $r+\gamma\max_{a'}Q(s',a')$. Regardless of which next action the behavior policy actually executes, Q-Learning uses the largest Q-value in $s'$. It therefore assumes that the next step will execute the optimal action perfectly. The resulting policy follows the shortest path beside the cliff without accounting for the risk that exploration may move the agent downward.
+
+**SARSA** uses
+
+$$
+Q(s,a) \leftarrow Q(s,a) + \alpha\left[r + \gamma Q(s',a') - Q(s,a)\right].
+$$
+
+Its target is $r+\gamma Q(s',a')$, where $a'$ is the action actually sampled from the current $\varepsilon$-greedy policy. Because SARSA learns from the next action that the behavior policy really selected, its update includes the risk of exploration-induced mistakes. It therefore tends to learn a longer but safer path away from the cliff.
 
 The difference between the two algorithms comes from different assumptions about “the future policy”:
 
 - Q-Learning estimates the value of the **optimal policy**, independent of how the current behavior policy explores.
 - SARSA estimates the value of the **current behavior policy**, so it avoids risks introduced by exploration.
 
-This is not a question of which one is “better”. They answer different questions. Q-Learning answers “if we ignore the randomness of exploration, what is the ideal optimum?” SARSA answers “given the exploration noise we are actually using, what is the safest way to behave?”.
-
 <span id="limitations"></span>
 
 ## Limitations of Tabular Methods

docs/en/chapter03_mdp/value-q.md

@@ -339,7 +339,7 @@ Observations must also be sufficient for decision-making. The current frame or f
 
 Rewards must arrive within a reasonable time. DQN propagates future returns step-by-step through the TD target. If the replay buffer contains only negative samples or meaningless transitions for a long time, the network cannot know which early action was useful. This is not a formula error — it is the learning signal being too far from the action.
 
-Finally, ε-greedy exploration must be able to produce useful experience. When action combinations are too many, episodes too long, and failure feedback too late, random exploration may fail to collect meaningful samples for a long time. In such cases, reducing the action set, designing staged rewards, or switching to methods better suited for long-horizon exploration is needed.
+Finally, $\epsilon$-greedy exploration must be able to produce useful experience. When action combinations are too many, episodes too long, and failure feedback too late, random exploration may fail to collect meaningful samples for a long time. In such cases, reducing the action set, designing staged rewards, or switching to methods better suited for long-horizon exploration is needed.
 
 Not meeting these conditions does not mean DQN completely fails — it means additional design is needed: continuous actions require different algorithms, missing observations require memory, sparse rewards require engineering or task decomposition.