No need to apply critical or creative thought when the downvote button is right there. Go on, just give it a click and carry on with your day so we don’t have to think about the bad things.
If I interpret "a machine superintelligence" as "a classroom of 300IQ humans," I'm not really sure how this is true? You still have material and energy constraints, you can't think your way out of those.
For the concrete problem we're discussing, you can hack your competitors out of existence, replace all of your knowledge workers to shed costs, hyperoptimise your logistics, etc. It's not just intelligence, it's speed and scale.
Bostrom's Superintelligence (2014) is a bit of a dreary read, and I didn't finish it, but it pulls no punches about the leverage that a superintelligence might have in our highly-connected world.
> For the concrete problem we're discussing, you can hack your competitors out of existence, replace all of your knowledge workers to shed costs, hyperoptimise your logistics, etc. It's not just intelligence, it's speed and scale.
For the concrete problem we're discussing, that hypothetical belongs in a Marvel movie, not reality. In the real world, you can't 'hack your competitors out of existence', and you'll be going to prison very quickly for trying this sort of thing.
> especially if you're willing to break the law / normal operating decorum
in my original post. If you have a superintelligence, you have something that can find and take advantage of every exploitation vector in parallel - technical, social, bureaucratic - and use that to destroy a company from the inside. A superintelligence that is subservient to its operator is an informational superweapon.
I agree that this sounds fanciful, but you can see what existing cyberattacks can do to organisations; it does not take that much imagination to gauge how much worse it could be when the process can be automated and scaled.
> A superintelligence that is subservient to its operator is an informational superweapon.
The five dollar wrench attack will put an end to that operator's use of an informational superweapon.
> I agree that this sounds fanciful, but you can see what existing cyberattacks can do to organisations
What can it do? Generally, a minor disruption to operations.
It consistently does a lot less than what law enforcement can do to you if you start messing with other rich peoples' money, while having enough of a presence to own a super-intelligence and a trillion-dollar data center.
Within a day - well before any legal or societal force could intervene - a superintelligence could make its way into every part of an organisation's internal network and tear it apart from the inside.
Conventional hackers are limited by the serial nature of their work - finding breaches, exploiting them, conducting further exploration of the network, trying not to get detected - in ways that a superintelligence would not be. The latter could be a hundred times as effective, a hundred times as fast, and a hundred times more parallel.
I agree that this is unlikely to happen because the societal bill would come due in time, but my point is that a month's lead is enough to do significant and lasting damage.
the one where i think of a particular piece of work, and i know who did it, then tell a student "oh, see if $author's group published anything else about this."
i'm not using software for this if this is off the top of my head, and it's the sort of thing that, at scale, hurts the forgotten author and their students
There’s a cute study demonstrating this effect by comparing career success in economics and psychology.
The author lists for economics papers are traditionally alphabetized, so more of your output will be known by your name if it occurs early in the alphabet. Abbie Ableson gets lots of mentions as "Ableson et al." while Zhang Zhu will almost always be relegated to the "et al". If name recognition matters, you’d expect successful academic economists to be clustered at the beginning of the alphabet—-and this appears to be true.
I don’t know that everyone would label it like that, but it’s inarguably true that success in academia comes from your reputation/name recognition.
Metrics are often attempts to formalize this but they’re not how most people actually make decisions: nobody is inviting seminar speakers or choosing collaborators because they have a high h-index. If anything, it goes the other way: name recognition gets you invited to speak or collaborate, which makes more people aware of your work, which boosts metrics.
That is false. The first thing everyone (at least everyone in CS---IDK about other fields) looks at are h-indexes, impact factors, number of papers per year, university rankings, and similar metrics. Researchers are most definitely selecting collaborators with a high h-index.
you're thinking of the programs in low-level langs that survived their higher-level-lang competitors; if you plot the programs on your machine by age, how does the low quartile compare on reliability between programs written in each group
my impression is that most CL these days is existing large closed-source codebases, hence the price tag for those compilers (you're not trying it out for a bit, you're funding the compiler devs to work full-time on the issues you're actually having) and relatively little open-source activity for "finished" things -- if you're developing against internal libraries, it's hard to open-source just the part you intend to
(work at a CL shop; mostly SBCL users, but maybe 1/3 of people are die-hard ACL fans)
how does this compare to MoSA (arXiv:2505.00315)? do you require that there's a single contiguous window? and do you literally predict on position, or with a computed feature?
I predict a specific location then put a window around that. Of course you can predict a different location per head or multiple window locations per head as well. The cost is negligible (single linear embx1 size) so attn becomes a fixed cost per token just like traditional windowed attn. Of course this doesn't solve memory consumption because you still have a kv cache unless you only do attn over the initial embeddings at which point you don't need the cache, just the token history. This is the tact I'm taking now since I have other ways of providing long context at deeper layers that remain O(1) for token prediction and are paralellizable like standard attn. I think this kind of architecture is the future, infinite context, fixed size state, O(1) prediction, externalized memory are all possible and break current context, memory and compute problems. It is clear that in the future token caching will be dead once these types of models (mine or someone else's with the same properties) are properly tuned and well trained.
reply