all notebooks are updated and should be working

[altaykerem]

Changes

Sorry if this is a lot of change all at once! I liked the book and as I read it I made some changes to match/fix/explain the code. As I did these, at the end of the book, I though it would be nice to share.

Overall

• Made sure all notebooks are both "new and more detailed"
• Python version updated to 3.14
• Some bugs were fixed
• No library warnings were disregarded
• Listing names matches with the book
• Notebooks are formatted with black formatter
• Cleaned up code that doesn't provide any additional information different to the main notebooks

Appendix

• plt.plot([loss.item() for loss in losses]) instead of plt.plot(losses)

Chapter 2

Listing 2.9

• Reset each time the cell runs.
• Output bandit stats.

Section 2.4 Working with PyTorch

• More details on gradient descent.

Chapter 3

• Import Gridworld from a shared package inside the project
• Added max move to done condition
• Reward is calculated inside makeMove like in a gym environment

Listing 3.2

• Remove repeated lines
• MSELoss needed explicit reduction

Listing 3.3

• Bug in state transition. state, state1 got confused
• Checking done condition was better reward == -1

PyTorch Automatic Differentiation Review

• Linear model missed @ x for no reason

Listing 3.5

• More intuitive control-flow for done and status
• Running mean was implemented but not used for the loss plot

Listing 3.7

• Played with sync frequency, experience replay was already as good for this simple case

Chapter 4

Listing 4.4

• Softmax on wrong dimension, you can see the first training flot not having an upward trend caused by this

Visual Gameplay

• This shows the model can go to 500 easily

Chapter 5

• I worked around the issue of multiprocessing not working with the notebook in some systems by:

  1. Writing the worker code to a python script from the notebook
  2. Calling the script inside the notebook

• Calling scripts from the notebook lost variables so models were pickled

Listing 5.1

• Platform specific import for demonstrating multiprocessing

Listing 5.6

• Added back training logs
• Made the worker function more generic for switching between n_steps and monte carlo

Listing 5.9

• Episode length tracking

Chapter 6

Listing 6.4

• One last evaluate population added to calculate population fit

Chapter 7

Listing 7.14

• Game termination happens at a specific time and when the agent sees the negative reward it associates the reward the moving up. So the longer I trained the worse the model got.
• More specific control flow.
• Added prints for seeing the actual events happening.

Chapter 8

• Fixed listing headers

Listing 8.11

• Fixed confusion between variables, use_extrinsic and use_explicit.

Listing 8.13

• last_x_pos is not actually 0 at the start of the game

Chapter 9

Listing 9.13

• https://github.com/geek-ai/MAgent is no longer maintained, upgraded to https://github.com/Farama-Foundation/MAgent2
• The new package did not allow the rendered video output, so keeping a list of rendered rgd arrays was used for visualizing events inside the notebook.

Listing 9.16

• Actually use l inside the function get_onehot.

Listing 9.20

• Added temperature to team step.

Listing 9.21

• reset step count at the start of each game. this bug prevented anything from happening after 250 steps
• param[0] for team 1 and param[1] for team 2