Hacker News

Unlocking Stable On-Policy RL on TPU with Bitwise-Zero Train–Inference Mismatch

10 points by gmays ago | 1 comments

amluto [-]

I think it would be nifty if someone convincingly solved this the other way: find a way for a model to learn well from the success/partial success/failure of a slightly off-policy rollout.

As a human, I certainly learn better by trying something myself (on-policy) than by watching someone else do it (very off-policy). But people, especially athletes, really do learn by watching videos of themselves trying something in the recent past (slightly off-policy), and I expect the main reason that watching the video doesn’t work even better than it does is that it’s just audiovisual and doesn’t come with all the sensations associated with the attempt. ML training doesn’t have that particular limitation.