Carl T. Bergstrom
It is absolutely astounding to me that we are still earnestly entertaining the possibility that #ChatGPT and #LLMS more broadly have a role in scientific writing, manuscript review, experimental design, etc.
The training data for the question below are massive. It's a very easy question if you're trained on the entire internet.
Question: What teams have never made it to the World Series?
Correct answer: Seattle Mariners.
Now, four responses from GPT4.
NB: The Nationals won it all in 2019.
Carl T. Bergstrom
I had GPT regenerate the answer 20 times. A few things to note:
1. Factual error rate: the system correctlu answered 1 time in 20.
2. Run-to-run inconsistency. I get different answers each time.
3. Logical errors and internally contradictory text in which one paragraph says a team did play and another says it didn't.
4. One attempt to self-correct that still doesn't quite work.
How could we think this sort of thing is useful for writing or even reviewing our work?
Bobinas P4G is a social network. It runs on GNU social, version 2.0.1-beta0, available under the GNU Affero General Public License.
All Bobinas P4G content and data are available under the Creative Commons Attribution 3.0 license.