It can be both.

The value should already be dropping with this news. If this is happening now, it seems likely more will come.

It's a stupid strategy that will put the rest of the world ahead of the US on AI. Anthropic's value will suffer for it.

This sounds like classic "you're using it wrong", if they had said it was done in smaller tasks you would very likely have people here saying that was wrong too.

Maybe a flaw in the labeling, but not the core methodology.

Verbatim code snippets like this imply the model is overfitting to it's training data.

While I probably wouldn't classify it as cheating, it is an even bigger signal of concern for model quality.

Cheating by breaking the rules at least implies some learned patterns.

Repeating training data verbatim for narrow cases like this implies that the model is overfitting.

If we're evaluating a person, rote recall is not necessarily cheating. It's expected, but then you'd expect them to apply that rote-memorized information in a novel way later on and prove they understand how they applied their priors to the new situation.

Models don't actually reason in the same sense, so recalling rote from their training data is "cheating" in the sense that the training data cheated, not the model. So many of those benches have snaked their way into training data to make them less useful benchmarks. That, I think, is going to be a long-term difficulty in quantitatively assessing model quality and "intelligence." So it is cheating, in a sense of what we expect from the models and training data, but not in a human sense.

If it was truely an arm's race to AGI they would've stopped relying on the data/param scaling law BS ages ago.

The way I see it the benefit of benchmark isn't to take Simon's results at face value. It's a template for your own benchmarks that are easy to visually evaluate.

> put him in a classroom setting, for something he doesn't really care about, and he just won't do the work.

Coming from a family of people assumed to be like this, and having friends in similar situations. Rarely are they actually disinterested in the subject, but are disinterested in how it was taught and self conscious of their perceived understanding of the subject. It's easier to say you don't care when it comes up.

> nearly everyone thinks that they know the right way to teach, and most people don't.

It's typical for people to accumulate many examples of how "not to teach", and it's natural to extrapolate those experiences into ideas of "how to teach". To your point though, most people don't know how to do things they aren't practiced in, and some don't even know how to do things they are practiced in.

> They would grasp the subjects quickly when I was speaking, they would ask good questions during class [...] but they didn't want to be there.

> [...] isn't interested in learning.

I highly doubt these students weren't interested in learning. By your own account they were engaged during class.

No teaching style is going to be able to fit to all students equally.

I was taught (when teaching for the military, actually) that leadership/teaching/etc. often talk about toolboxes but neglect a VERY important one.

You shouldn't just have a toolbox of things you've picked up from your best examples, that you've been taught, etc. It's possibly more important to have another toolbox of broken tools from all the terrible bosses, reactions to situations you've witnessed, etc.

This way when you go to do something and it's not in your toolbox, you can pull out that box of broken/bad tools and see if it's there. Otherwise we perpetuate bad leadership (and teaching IS leadership) through intentional ignorance (forgetting the lessons those bad situations gave us).