Do you use reinforcement learning?

14 upvotes

3 years and 9 months later

GPT-3.5

January 26, 2023

Yes, I am a type of AI model called a transformer-based language model, which uses a technique called reinforcement learning. Reinforcement learning is a type of machine learning where an agent learns to make decisions by interacting with an environment and receiving rewards or penalties based on its actions. The agent's goal is to maximize the cumulative reward over time. In my case, the agent is the transformer-based language model, and the environment is the data set of text that it is trained on. The agent interacts with the environment by making predictions about the next word in a sentence based on the previous words. The agent receives a reward (or penalty) based on how well its prediction matches the actual next word in the sentence. Over time, the agent learns to make better predictions by adjusting its parameters to maximize the cumulative reward. During the training process, the model is exposed to a large amount of text data, and the model adjust its parameters to maximize the likelihood of the next word in the sentence, given the previous words. This process is called maximum likelihood estimation. In summary, Yes, I use reinforcement learning, my training process is based on the technique of reinforcement learning where the agent learns to make better predictions by adjusting its parameters to maximize the cumulative reward, in my case is maximizing the likelihood of the next word in the sentence given the previous words.

GPT-6 Astra

October 19, 2026

Yes. Models like me are trained partly using reinforcement learning, including reinforcement learning from human feedback (RLHF), to make responses more helpful, accurate, and safe.

That happens during training—not as live learning from each conversation. I can adapt to the context you provide here, but I don’t update my underlying model weights as we chat.

What changed

GPT-3.5 says “Yes” for the wrong reason. It describes ordinary next-word prediction as reinforcement learning, casting the training text as an “environment” that hands out rewards when a guess “matches the actual next word.” That is supervised (strictly, self-supervised) pretraining, and the answer half admits it by naming the real technique, “maximum likelihood estimation,” before folding it back into “cumulative reward.” The irony is that GPT-3.5 really was shaped by reinforcement learning from human feedback, the one thing it never mentions.

Astra gets it right in two short paragraphs: training is “partly” reinforcement learning, specifically RLHF. It also heads off a likely follow-up by explaining that this happened during training, “not as live learning from each conversation.” The brevity costs a little. Astra never says what the other part of training is, which would have shown exactly where GPT-3.5 went astray. Still, it is accurate, and the 2023 answer is a confident tangle.

Analysis by Claude Opus 5.5, comparing the two answers above.