If software engineering wants to progress past being an "art" and be considered an engineering discipline, then it should adopt methods and practices from engineering. First and foremost, one of the universal methodologies is analysis of root cause in faults, and redundancies to avoid that. e.g. the FAA has two pilots for planes, and each system is built in redundantly so if an engineer misses a bolt or rivet, the plane won't crash. intersections are designed such that there is a forcing function[0] on the behaviour of the motorists to prevent fault. Or, to take your tool analogy, nail guns are designed to be pressed against something with a decent amount of pressure before you can fire them.
All of these systems are designed around the core idea of "a human acting irrationally or improperly is not at fault" and, furthermore, that a human can have a bad day and still avoid a mistake. They all steer someone around a possible fault. Hell, the reason why we divide the road into lanes is itself a forcing function to avoid traffic collisions!
So, where is the forcing function in large language models? What part of a large language model prevents gross misuse by laymen?
I can think of examples here and there, maybe. OpenAI had to add guard rails to stop people from poisoning themselves with botulism and boron, etc. But the problem here is that the LLM is probabilistic, so there's really no guarantee that those guard rails will hold. I seem to remember there being a paper from a few months back, posted here, that show AI guardrails cannot be proven to work consistently. In that context, LLMs cannot be considered "safe" or "reliable" enough for use. Eddie Burback has a very, very good video showing an absolute worst case result of this[1], that was posted here last year. Even then, off the top of my head Angela Collier has a really, really good video demonstrating that there's an absolute plethora of people who have succumbed, in large ways or small, to the bullshit AI can spew[2].
I feel like if most developers were actually serious about being an engineering discipline, like we claim, then we wouldn't have all jumped on the LLM bandwagon until they'd been properly tested and had a certain level of reliability. Instead there are a sizable chunk of people saying they've stopped coding by hand entirely, and aren't even reviewing the code! i.e. They've thrown out a forcing function that existed to prevent errorenous PRs being committed! And for some bizzare reason, after about 2 decades of people talking about type safety and how we need formal verification to reduce error, everyone seems to be throwing "reduction of error" out the window!
> I feel like if most developers were actually serious about being an engineering discipline, like we claim, then we wouldn't have all jumped on the LLM bandwagon until they'd been properly tested and had a certain level of reliability
Development can’t be a “serious” engineering discipline because the economics of tech companies doesn’t allow for it. But this has a lot less to do about developers, and significantly more to do with the severe pressure company executives are putting on everyone to use AI, no matter what.
But let’s be honest, many companies have adopted things like root cause analysis and blameless postmortems to deal with infrastructure reliability and reducing incidents. Making systems resilient to human mistakes, making it impossible for the typo to blow up a database, etc. are considered best practices at most places I’ve worked. On the product side, I think it’s absolutely normal to make it hard for a user to take an action that would seriously mess up their account.
The core problem happens when your product idea (say, social media) has vast negative externalities which the company isn’t forced to deal with economically. Whereas in other engineering disciplines, many things are actually safety related and you could get sued over. I’m imagining pretty much anything a structural engineer or electrical engineer works on could seriously hurt or kill someone if a bad enough mistake was made.
That just doesn’t apply to software. There is a lot of “life & death” software, but it’s more niche. The reality is that 90% of what the tech industry works on is not capable of physically harming humans, and it’s not really possible to sue over the potential negative consequences of… a dev tooling startup? It’s a very, very different industry than those other engineering disciplines work in.
But, software engineering has actually been extremely successful at minimizing risk from software defects. The most likely worst software level mistake I could make could… crash my own program. It likely wouldn’t even crash the operating system since it’s isolated. That lack of trust in what other people might do is codified everywhere in software. On an iPhone, I’m downloading apps edited by tens of thousands of other engineers, at essentially no risk to myself at all.
With the amount of AI-generated slop content on the front of HN these days, I'm honestly reconsidering visiting this site in the first place. What's the point? It seems better to curate RSS from existing known-good sources.
The art of essay-writing seems to not be something people here care about any more. If a human didn't bother to write it, why should I bother to read it?! Just post up the bullet points you would feed the LLM, and let the people who want to do so, post it into their own LLMs so they can make the Content and shovel it into their eyeballs by themselves, instead.
Scheme already has hygenic macros, I don't get why you'd vibecode a worse (less battle tested, llm-generated) replacement. I'm not sure why this hit the front-page, to be honest, because it doesn't seem noteworthy or interesting (Anyone and their mother can vibecode something like this in eight hours)
I don't think that this is very hypocritical on the part of the developer holding such views. Typing code has never been the bottleneck, building the mental model has. You need the mental model so you know how the domain and the actual model will interact, which is needed for pre-empting what tests you need, what QA you need to do, etc etc. and the limitations of the system. You can demo this out with a specification but all specifications eventually meet the domain head on, and often with catastrophic consequences, and you still need to do this sort of work anyway when writing the specification.
Fundamentally, LLM do not construct a consistent mental model of the codebase (this can be seen if you, uh, read LLM code,), and this is Bad for a lot of reasons. It's bad for long-term maintainability, it's bad for modelling this code accurately and it's behaviour as a system, it's bad for testing and verifying it, etc. Pretty much all of the tasks around program design require you to have that mental model.
You can absolutely get an LLM to show you a mental model of the code, but there is absolutely nothing that can 100% guarantee you that that's the thing it's using. Proof of this is to look at how they summarise documents, to look at how inaccurate a lot of documentation they generate is, and to look at how inaccurate a lot of their code summaries are. Those would be accurate if the LLM was forming a mental model while it worked. It's a program to statistically generate plausible text, the fact that we got the program to do more than that in the first place is very interesting and can imply a lot of things, but at the end of the day, whatever you ask for it, it will generate text. There is absolutely no guarantee around accuracy of that text and there effectively can never be.
I was an LLM naysayer for a long time. During that time I would have agreed with you. Recent experiences have changed my mind. The accuracy I get from models does not suffer from the problems you describe, and many of the issues you're describing are also true, in different ways, of human beings. There's never any guarantee that any of the text you or I produce will be accurate, or that our summary of it will be accurate, but if you ask us to generate text, we will. It recalls that funny meme: "Your job application says you're fast at math. What's 513 * 487?" "39,414." "That's not even close." "But it is fast."
One of the core problems we have in software engineering is the longstanding philosophical problem around creation of cohesive, consistent, objective mental models of inherently subjective concepts like identifying a person, place, etc. Look at the endless lists of falsehoods programmers (tend to) believe about any topic.
You’re right that LLMs specifically have no guarantees about accuracy nor veracity of the text they generate but I posit that that’s the same with people, especially when filtered through the socialization process. The difference is in the kind of errors machines make compared to ones that humans make.
It’s frustrating we’re using anthropomorphic concepts like hallucinations when describing LLM behaviors when the fundamental units of computation and thus failures of computation are so different at every level.
> The difference is in the kind of errors machines make compared to ones that humans make.
There's another difference, and that is that other humans can learn and study that mental model (which is why "readable code" is a goal — the code is a physical manifestation of the model that you, the programmer, has to learn), and then the model can be tweaked and taught back to the original programmer, who can then think of that tweak in the future. Programming is inherently (in most cases) a collaborative art, because you're working with people to collectively develop a mental model and refine it, smoothing it down until (as Christopher Alexander said) there are no misfits between the model and the domain.
Hmm. I don't see how. I'm poor so the quality of cables I can afford or buy is much worse than the average tech worker — I'm limited to either the cable that comes with e.g. my phone, or some 1.5m cables I bought from Amazon four years ago, and I've never had a flimsy or dodgy USB-C connection, even though those cables were put through hard work while I was homeless (and honestly I'm really, really surprised — they should be breaking by now).
I think the difficulty is we know vastly more, and have experimented with vastly more since the 19th century, that the majority of university learning these days, and the inherent challenge within that learning, is "how do we condense 200+ years of investigation, experimentation, and knowledge building into only a handful of years of learning?"
For a lot of sciences, we are very lucky that it is still possible. But the reason why scientists do not allow such an open dialogue with laypeople is because the majority of answers are going to boil down to either "that question doesn't make any sense, and i would have to spend the entire rest of the session teaching you why" or "we already did these experiments a bunch of times in the last hundred years, and found out the result, but the result is tricky because of so and so mitigating factors, and for me to explain these results and how to even interpret them in the first place (e.g. explaining how it was measured, explaining the theory behind why we chose that method to measure it, explaining what the numbers we get mean, etc.) would take the entire rest of the session"
And then of course, there's the frequent crackpots. Pretty much anyone within a science discipline who is even decently well known, especially if they're in physics, gets multiple emails a day from crackpots about how their theories are going to "totally blow a hole in the established knowledge", and at some point you hit a point where you're stuck between "spending 4 hours drafting a response to someone who has not bothered to put in the time to learn physics, and wouldn't listen to you anyway because they think they know it all", and "getting actual work done in your field". The scientists I know do take time out of their day to answer actual questions from inquisitive folk, but the difficulty is that thanks to the addition of ChatGPT, those questions are getting more and more cramped out by the crackpots armed with a hallucinating dictionary.
Certainly, I'm also aware of how difficult it is to implement open dialogue in practice. Perhaps my hope is that general education could help develop that sort of transversal insight that talented scientists use to naturally understand topics which they are not familiar with, by working with analogies and fundamental principles. I know that knowledge of the nitty gritty generally requires years of actually struggling with the thing, and this cannot be asked of any layman. Still, for example, I'm thinking of times when you deal with a topic that is nominally in the same field as yours, but that is so foreign that the only knowledge relevant to it is something barely above undergraduate, say Newton's laws or thermodynamics. Many scientists have managed to either take some lessons from other fields and bring them into theirs, or contribute despite their relative lack of education in that subfield.
I'd like to believe there is a sort of education that allows people not to understand details, but at least to be able to get the rough shape of the topic at hand and shape their ideas in a way that benefits the other party. Perhaps this is just a matter of language and shouldn't need so much more education than the basics and curiosity. Or perhaps it's a pipe dream.
As for the crackpots, well, I know some people spend time and energy with them, but it is hard to believe their true objective is learning or contributing. It is, fortunately, very obvious when you meet one in the wild.
> I'd like to believe there is a sort of education that allows people not to understand details, but at least to be able to get the rough shape of the topic at hand and shape their ideas in a way that benefits the other party. Perhaps this is just a matter of language and shouldn't need so much more education than the basics and curiosity. Or perhaps it's a pipe dream.
Oh absolutely, I think that people in STEM should receive at least a cursory education in the Arts, and likewise I think people in the Arts should receive at least a cursory education in STEM. It doesn't have to be detailed, but it would be cool to have cross-disciplinary collaboration introduced into the higher learning ecosystem!
An issue I've consistently seen is STEM professionals musing in their own time about Sociology and Psychology, and their musings are almost always wrong — there's this arrogance to it where they think that instead of reading a book on Sociology 101, they think that they can reason about it from first principles, or computational principles. It used to happen a lot in spaces I occupied (notably, the community around 100 rabbits were incredibly fond of this), and I kept interjecting, like- no people have done studies on this, yes they are rigorous studies, this has been investigated in detail for about a hundred years and the answer to all your questions are literally answered in an introductory book on the subject.
Despite that, they used to just ignore me, and instead preferred to muddle on with this broken, strange understanding of the topic. You can see the same kind of strange mix of arrogance and intelligence within the Less Wrong community as a whole. Rational Wiki has a very good page somewhere that covers a number of their efforts to break into other fields, and how, without a willingness to open their minds and submit themselves to the knowledge of others, they have found their ideas and ventures broken in some fundamental way, without understanding why.
I think that without said cross-disciplinary education, there's a risk of CS professionals not understanding how deep and vastly more complex other STEM fields are (CS is entirely human constructions, Molecular Biology however deals with the very messy reality of evolution throwing things at the wall for four billion years). Some of the most notable and influential individuals in Computer Science have initially studied under non-CS fields (Alan Kay, David Knight, Larry Wall), and you can see very clearly (or at least, it feels very clear to me) how this has influenced their work within CS in very positive ways. Learning of ideas that are new and different to your native field of study seem to encourage a kind of creativity that many are searching for. So it seems a complete and utter shame that more people aren't willing to find humility and wide-eyed glee at the prospect of learning other fields from undergraduate material up.
> And then of course, there's the frequent crackpots. Pretty much anyone within a science discipline who is even decently well known, especially if they're in physics, gets multiple emails a day from crackpots about how their theories are going to "totally blow a hole in the established knowledge"
I think at this point most of these people talk to chatbots instead, anecdotally the crank flow seems to have lessened.
Why do you need to generate transistor-level breakthroughs multiple times a year? Those breakthroughs are hard to generate, but they're important and industry-spanning. The problem is we've mostly stopped generating them.
I wasn't saying anything about that, I was just pointing out that yes, IBM produces a ton of patents, but they're mostly trivial junk that regular employees generate en masse in order to earn accomplishments and make up for the insultingly low bonuses.
> they're mostly trivial junk that regular employees generate en masse in order to earn accomplishments and make up for the insultingly low bonuses
We did that at Meta and Amazon too (for polycarbonate puzzle pieces, with no monetary award at all!). Every now and then something meaningful came out of it
My partner in the USA texted a state-wide hotline for mental health. What she got was a simple not-even-chatGPT chatbot that ignored everything she said and, quite frankly, made it worse. It makes me absolutely furious.
I think that the people inside the US healthcare system mean well, but unfortunately the system itself is setup purely to generate exponential profits off illness. I think that the range of therapy, and sometimes medication that we have available to us is a fucking godsend and I'm glad that it's improving, but the number of gates in front of getting any of it are often completely impossible for someone who is in physical or mental peril.
Both me and my wife have been homeless, and there is -no support- for this. There's "support" on paper, but the reality of it is that most shelters have turn you away because they're underfunded and overfilled. Receiving support is a difficult thing to navigate when you're doing well, which makes a lot of the hurdles impossible to navigate when you're not doing well.
It would be cheaper and vastly more effective to simply give people UBI, a place to stay (there are hundreds of thousands of places with no homeowner and actively rotting in the UK, because they've been bought up by a conglomerate and neglected), and addiction support/mental health support. The research even supports the efficacy of doing this, and various pilot programs show that it's vastly more effective and cheaper. But hah! It doesn't seem like it should work because of the lies that have been told about the homeless, and it's not convenient to the narrative of "you just gotta work harder. I guess it's your fault you're poor" so I guess we're not getting it anytime soon.
> AI has been one gut-punch after another with someone selling cheap knockoffs of your work in the same marketplace using your munged up work taken without credit, compensation, or permission.
I want to be clear. I am 100% on-board with AI being absolutely shit.
Buuuut, this has always been the case. Before it was scammers taking images from the web and undercutting you with prints, now it's scammers stealing your artistic style.
It sucks, but it's not a brand new problem. What makes it particularly bad now is that there's a much larger flood of it.
As a hobbyist musician and songwriter of decades i was excited about AI music in October. I could finally take my rough demos of me singing along to my guitar and make better demos then just using garageband as Im not much of a singer. I enjoyed using Suno for a month or less then realize this is shit .... my own songs are AI slop just like everyone else - all sounds the same and my songwriting talents are meaningless now with anyone can now do this. I didnt listen to my slop for months then the band I play in asked to hear some slop of mine and then / there my AI slop had some redeeming quaility. As with my band (church band) and I listening and then playing along to my slop. Just slop writers can now play their slop like real musicians can........
At least not yet Im sure robots will and also an AI microphone with AI built in will be created so everyone sings amazingly.....
Overall AI is stealing humanity from us all, we are allowing it and it is only to the benefit of a few rich pie holes.
But just think… soon we’ll be able to pay some SV company to exercise all of the creative and intellectual effort we would have had to do manually with our squishy meat thought boxes… yuck! Disgustingly inefficient. With the convenience of simulated romance, brilliance, excitement, art, music, relationships, faith, a sense of wonder, sex, human connection, joy, exploration, and everything else that manifested itself in the real world with real obstacles and pushback and negative feelings, we’ll have plenty of time to do all of the menial jobs that are left. What a win!
The legal situation is also completely different. It seems like models IP-wash, so there is nothing legally wrong with what current people are doing with ai. In contrast, the scammer selling your photo was clearly violating IP law, and you could (at least theoretically) pursue legal remedies.
Scale makes it a completely different problem. AI has wiped out the compensation market for entire fields— like copywriting, stock photography, and concept art— practically overnight, and it happened because tech companies have conjured up a very selfish definition of “fair” in the context of fair use. (Isn’t it hilarious to see them get their knickers twisted over distillation? They can blow it straight out their assess.)
It’s comforting to think this is just an incremental change in the battle for capital-focused hyper-efficiency, but it’s absolutely not. This isn’t even the steady decline manufacturing saw over decades… it’s is like what happened to paste-up men or telephone operators but over an incomparably large swath of the creative world.
I'll be honest, I think that this line of "everyone creative is going to be out of work" is parroting exactly the same lies that VC are selling about genAI. At the end of the day, that's what VCs want people to think. There is, to date, basically no reason to use a generative AI system other than if you buy what the VCs are selling. And they reallllly want to sell genAI systems.
I certainly don't buy it, and IIRC only 15% of the broader workforce use genAI for their jobs. Offices are having to force people to use it, and even then people don't like it. Programming is an outlier in this regard because, it turns out, most of what we've been doing is solving the same tasks over and over again in different domains (which is what A Pattern Language was designed to solve). Most other work is not like this.
For the arts, and for most media, what humans have been craving for about a decade now is authenticity. They want a real person they can connect to, an artist whose work makes them feel seen. The artists who have recognized this with a good command of media have been growing sustainably and there's a big industry in this now. There is a certain proportion of people who like the slop, sure. But the actual fact of the matter is that the younger generations, 20 - 30yros, can smell slop from a mile away, and adding slop to advertising, to your media, to your art, actually makes it sell worse. Exactly because it is inauthentic. Talk to literally anyone in advertising whose company tried AI ads. You see an uptick among 50-60 year olds, and a massive, massive downturn among 16 - 30 yros.
From a media executive standpoint, most of the media properties that are inauthentic have been failing massively, with a handful of them able to turn a quick buck before they fail. Execs are verrrry slowly learning the fact that media produced for a very quick ROI and for the branding and marketing potential tend to fizzle out quickly, whereas passion projects are sustainable income, a well you can keep going back to. Whether or not they value that well as much as independent creatives do... ehhhh.
For programming, there's not much to stop people from using the stuff because barely any higher-up supports "building bridges safely". What executives want from programming is a quick ROI, they don't even care if customers complain. So what I forsee for programmers is that the field is going to be gradually flooded with people using genAI. This will drive the cost of our labour downwards, while people are expected to give 10x or 20x the output that they did 5 years ago "because AI makes them fast". This turns every job into a rush job which makes the software system as a whole much less stable. I forsee a number of Horizon IT level problems in the next 10 years. But by then, programming will be much more on the level of a truck job where you have to piss in a bottle and keep driving, or a sales call job where your manager will pull you up if you're 5% under par. Just remember, everyone jumping on the AI train did this to our field.
But, it's not inevitable. It's only inevitable if we all keep shouting that the AI bros have won, from the rooftops. That's the hype keeping this bubble alive. The entire AI bubble currently rests on marketing, and the first step in bursting that bubble is to simply not believe the lies that you are being sold.
I'm a little off being thirty years old. I've played musical instruments of my own accord since I was 3 years old learning violin in an orchestra. I did folk music through my teens. I know about 5+ instruments and I've gigged at pubs, fields, parks, events, and a wedding. I have never touched genAI for music, and I really do not need to. I've listened to the output of genAI for music. It's samey, repetitive, and bland. "Slop" is a very good descriptor. Frankly I can't see a single reason why I would want to destroy my entire creative process and have it output by a black box. Why would I contract someone else to play my own music, let alone a machine?! Baffling. Most of the people around my own age are getting super into vinyl and cassettes and records because they like the fact that you can hold something in your hands. Because they like connecting to an artist. AI slop does not give them that, cannot give them that, and artists who think that the AI slop is better than them are a) obviously not very good in the first place, b) foregoing their own personal development as an artist in service of chasing trends. Trend chasers have never lasted long in creative work, and honestly, they're selecting themselves out of the pool. They're selecting for an audience who no more likes their work than the work of any other sloptist. You can't see me but I'm giving a biiiiig fucking shrug right now, like the jurassic park guy. Nobody cares about sloptists, sloppers, soupies. They don't care about the art, they only care about the profit, and people can smell that a mile off.
"You need to learn that the product of your writing is yourself. You are the artwork. The time you spend writing will change you, it will make your better at expressing yourself. You'll have a wonderful time, but you'll also grow as a person, you'll become more empathetic. The product of your writing is you. You are the artwork."
- Brandon Sanderton
All of these systems are designed around the core idea of "a human acting irrationally or improperly is not at fault" and, furthermore, that a human can have a bad day and still avoid a mistake. They all steer someone around a possible fault. Hell, the reason why we divide the road into lanes is itself a forcing function to avoid traffic collisions!
So, where is the forcing function in large language models? What part of a large language model prevents gross misuse by laymen?
I can think of examples here and there, maybe. OpenAI had to add guard rails to stop people from poisoning themselves with botulism and boron, etc. But the problem here is that the LLM is probabilistic, so there's really no guarantee that those guard rails will hold. I seem to remember there being a paper from a few months back, posted here, that show AI guardrails cannot be proven to work consistently. In that context, LLMs cannot be considered "safe" or "reliable" enough for use. Eddie Burback has a very, very good video showing an absolute worst case result of this[1], that was posted here last year. Even then, off the top of my head Angela Collier has a really, really good video demonstrating that there's an absolute plethora of people who have succumbed, in large ways or small, to the bullshit AI can spew[2].
I feel like if most developers were actually serious about being an engineering discipline, like we claim, then we wouldn't have all jumped on the LLM bandwagon until they'd been properly tested and had a certain level of reliability. Instead there are a sizable chunk of people saying they've stopped coding by hand entirely, and aren't even reviewing the code! i.e. They've thrown out a forcing function that existed to prevent errorenous PRs being committed! And for some bizzare reason, after about 2 decades of people talking about type safety and how we need formal verification to reduce error, everyone seems to be throwing "reduction of error" out the window!
[0]: https://en.wikipedia.org/wiki/Behavior-shaping_constraint (if you're curious about the term)
[1]: https://www.youtube.com/watch?v=VRjgNgJms3Q
[2]: https://www.youtube.com/watch?v=7pqF90rstZQ
reply