Later they shift the blame to Railway for not having scoped creds and other guardrails. I am somewhat sympathetic to that, but they also violated the same rule they give to the agent - they didn't actually verify...
Railway’s “Ship software peacefully” is a good mantra, and they might want to add more protections around very destructive operations.
There’s a lot of blame to be passed around in this story, including OP’s own ways of working. But I agree with them that such destructive operations shouldn’t be in an MCP, or at least be disabled by default.
Note they didn't say "we used scopes but there is a bug that killed us". No, they simply assumed the token would be magically scoped somehow without any justification for doing so:
>Tokens are not scoped by operation, by environment, or by resource at the permission level. There is no role-based access control for the Railway API — every token is effectively root. The Railway community has been asking for scoped tokens for years. It hasn't shipped.
I get that this paragraph is a retrospective realization (I hope, otherwise the argument is even more ludicrous). But like, if the UI didn't ask you to choose scopes for your token then there is no reason to assume they will magically be enforced somehow! And you sure as hell shouldn't trust it to your agent without checking.
They're trying to blame Railway for not having safeguards - which is a fair critique - but they clearly should have known better or at least followed their own instructions.
If they wanted scoped tokens, they should have put on their roadmap an item to move to a SaaS product which has scoped tokens. Or ACLs. And until then, kept it on a list of risks: unscoped token may be misused by developer to delete prod db.
There's no difference in risk between this being done by an LLM vs. a human. Both make mistakes, so if you want to reduce the risk of this happening, you should poka-yoke[0] your systems to make this less likely to happen.
I'm not sure what's more striking about this blog post: that it includes virtually no assumption of blame on the part of the author, or that the author had this happen to them and was so angry with AI that they decided to use AI to write up the post.
Sorry but are you implying that for every system you integrate with, you verify the scope of an API key by checking each CRUD operation on every API endpoint they provide?
I think the suggestion from their "somewhat sympathetic" position is that if you are integrating with something you should (a) find out up front what limits it does or doesn't have on its API keys, so that it's not a nasty surprise later, and (b) absolutely don't give keys without really tight scopes to "agents."
The person here who deleted prod DB with their agent made an assumption that an API key wouldn't have broad permission if there weren't warnings ("We had no idea — and Railway's token-creation flow gave us no warning — that the same token had blanket authority across the entire Railway GraphQL API, including destructive operations like volumeDelete. "). I don't know what the UI looks like exactly, but unless I'm explicitly selecting a specific set of limited permissions, I don't know why I'd assume "this won't do more than I am creating it for". Like "I didn't ask the guy at the gun store to put bullets in, I wouldn't have given the gun to the agent if I'd known there were bullets in it."
I also would be wary of running on an "infrastructure provider" that didn't make things like that very clear.
Is this overly harsh? I don't know. I've had to explain far too many times to people (including other engineers) what makes doing certain things unsafe/foolish (since they initially think I'm wasting time checking things like that). So I think stories like this need to be taken as "absolutely do not make the same mistakes" cautionary tales by as many people as possible.
For every API you publish, do you verify that scoped API keys work as they should before you go live? If so, why would you not do the same for APIs you integrate with? It's all part of "your" system from the user's perspective.
I think the author is being deceptive with this part:
>3. CLI tokens have blanket permissions across environments.
>The Railway CLI token I created to add and remove custom domains had the same volumeDelete permission as a token created for any other purpose. Tokens are not scoped by operation, by environment, or by resource at the permission level. There is no role-based access control for the Railway API — every token is effectively root. The Railway community has been asking for scoped tokens for years. It hasn't shipped.
They're trying to make it sound like there was some misleading design around scopes, but the last sentence gives it away. They simply assumed that a scope would be enforced somehow, even though they never explicitly defined one like you would in a service that actually supports them. (Or worse, they actually knew all this ahead of time and still proceeded).
That said, I haven't used this service so I can't evaluate the UX. I know that in GitHub or cloud IAM there is no ambiguity about what you're granting. And if I didn't have full confidence in the limits of a credential then I sure as hell wouldn't give it to an agent.
“why would you not do the same for APIs you integrate with?”
Who does that? Jira and Salesforce have hundreds of endpoints each. AWS has hundreds of services, and each may have hundreds of endpoints. Who on your team is testing key scopes of every endpoint? Do you do it for each key you generate? After all, that external system could have a bug at any moment in managing scopes. Or they could introduce new endpoints that aren’t handled properly. So for existing keys, how frequently do you re-validate the scope against all the endpoints?
Yes but my original reply was to someone that seemed to imply that this founder was dumb not to verify that Railway’s API key that should have been limited to managing custom domains, truly was limited to managing custom domains. I’ve never used Railway but my pushback is that no one in the real world exhaustively verifies a key is scoped properly against all 3rd party endpoints. We trust vendors to document how they’re scoped and to actually do that.
I think it is meaningful that the author didn't say "there was a bug in scope enforcement" or "the UX is really misleading- look at these screenshots." In fact they even state this a long standing community FR. And they don't even say they only discovered this after the incident!
It actually seems like they knew ahead of time and proceeded anyway, but are just using this critique as a way to shift blame.
No I'm not. But it's clearly stated in the article that the API doesn't have scopes at all... So there was no reason to assume that some would be magically applied!
In GitHub or AWS etc you expect scopes to work because you define them. However if there is no way to define them in the first place, would you assume the system can somehow read your mind about what the client can access??
In fact I now believe this is a deliberate rhetorical sleight of hand. Point out a legit critique of the API design as if it is an excuse. But really any responsible engineer would notice the lack of scopes immediately, and that would be a flashing siren not to trust them to an agent.
That's one of the first instructions in my system prompt when I'm working with an LLM:
> Do not reply in the first person – i.e. do not use the words "I," "Me," "We," and so on – unless you've been asked a direct question about your actions or responses.
It's not bulletproof but it works reasonably well.
Why did a mental healthcare company have the ability to exploit this?
Do you think they accidentally found this 5 seconds before their exploit was launched or do you think they might have actually put some effort into doing this since they are an organization of people.
I am pretty skeptical it’s intentional. Very risky move. If they make apple look bad they can say goodbye to getting featured in the app store, or could even get pulled from the store completely.
I can see a fucked up ceo greenlighting a trick to get their app installed on your phone without asking. I can’t really see them having it repeatedly download.
I suspect it’s a bug, or worst case a backdoor that’s been triggered with a commercial app instead of spyware accidentally or “accidentally”.
I cannot possibly imagine the company as a whole would approve of this, much less anyone at the company who wants to keep their job. If it’s found that they exploited Apple to cause this, Apple might force-remove their app worldwide and definitely will kill their developer account pending any lawsuits. That’s the sort of thing that gets a CMO fired. Seems extremely unlikely, but if their C_O gets fired on Monday or Friday, then we’ll probably know why :D
That's why I was emphasizing "one way" and "can begin".
Clearly viral infection may be a distal factor but it is not the proximate cause of Alzheimer's... which is quite conclusively due to protein aggregates.
Yes, exactly. Whether it APOE mutation, traumatic injury, metabolic dysfunction, Herpes infection, or some other inflammation that upregulates kinases, you end up with hyperphosphorylated tau which in turn forms the neurofibrillary tangles that destroy nutrient transport in neurons and eventually kills them.
"Neuron death by protein aggregates" is the best way to define Alzheimer's. Anything more specific refers to only one of those aforementioned causes; anything less is just "brain don't work no more."
These clickbait headlines are so frustrating, especially since the article itself explains the tau mechanism and all the progress that has been made in understanding the disease.
It very much has been. Which is why it's tiring to see these articles (and HN comments) portraying it otherwise. Search HN for Alzheimer's and you will find lots of updates on the research and lots of the same uninformed comment threads.
Because of Joy, scientists developed a simple, non-invasive, three-minute skin swab test that analyzes sebum to diagnose Parkinson's. In laboratory settings, this test has shown an accuracy rate of over 90%.
Unfortunately that insight hasn't led closer to a cure.
It also turns out sort of bad to tell people they have a horrible neurodegenerative disease 10 years before the major symptoms start.
Why isn't everyone's mind blown? Well for the cynical explanation, look at the state of science education and what people said about Artemis, vaccines, etc. or optimistically, there are too many mind blowing discoveries to treat them all fairly.
To me it is more like software consultant speak than AI booster speak. And it is not exactly surprising that the people in a particular subculture all talk similarly.
Well, I hear it from people who are regular devs and not consultants, although it's more common with people who aren't really working in the trenches anymore.
Like ex-developer turned PM who is now vibe coding everything they can and thinks it's the greatest thing ever.
Typical extraction yield is 18-20%. For a 20g dose that's 4g of material consumed, or about 30 individual beans.
I wonder if you could find similar effects with 4g or broccoli sprouts, or garlic, or ginger, or cumin seed, shiitake mushroom, seaweed, soursop leaf, or...
reply