Ohhhh geee!!! I just applied the patch to my local git copy.
You need to use the model on the PR that he submitted, the model is particular because it has extra information that allows the MTP to happen.
I have two amd gpus, and qwen3.6 27B qk6 does around 20t/s generation... If I run it only on one I get like 35t/s.
But with this patch I saw 46t/s with qwen3.6 27B q8... this is insane, it's 250% faster than the original speed, there was no gpu I could upgrade to get that kind of boost, amazing!
What's "sad" is how slow the ollama folks are being in vendoring newer versions of ggml into their codebase. That attitude just leaves them stranded without access to newer features.
A few days ago I switched again from Qwen3.6 to Gemma 4 - for personal use I've experienced better average performance with the 26B version of the latter than the 27B of the former.
For someone who's been running local models for a long while, these are very very exciting times.
Oh that's fascinating. 3.6 27B is pretty damned good, but slow in wall-clock times on my DGX Spark-alike. It generates huge reams of thinking before it gets the (usually correct!) answer, so wall-clock time is rough for tasks even at ~20tk/s
I'm surprised the 26B-A4B is better? It should be faster too, interesting. I'm excited to try 31B with MTP, because MTP-2 is what makes 27B bearable on the GB10.
What are you using it for? Agent-based coding, or something else?
General purpose, mostly internet research in the form of slow-crawling. (Emphasis on slow - I've ultimately landed on Scrapling's API for seamless content rendering, and I use image support so as not to exclude informative images or weirdly rendered text.)
For coding I don't need image support so I stuff the entire GPU with text-only mode. I don't have a workflow where I send LLMs off to generate thousands of lines of code but what little coding I did I did with Qwen3.6 and it was spectacular, as you likely suggest.
I've been thinking about doing more of this too. What spec machine are you running? And are you using long-running autonomous agents or more of the IDE/co-pilot style of collaboration?
Gemma certainly was trained for tool calling, but the implementation in llama.cpp has been troubled because Gemma uses a different chat template format. The processor from the transformers library works fine though.
I'm using llama.cpp with Gemma and tool calling is mission critical. It's perfectly fine on my end.
There are definitely differences in the eagerness to tool-call that you'll need to manage. And for all local models I've ever used, I've had to micromanage the tools provided by servers to eliminate any possibility that they reach for something wonky or confusing.
> However I find qwen unbeatable for toolcallling. I think gemma wasnt trained on that at all.
Gemma4 chat template seems to had multiple issues, at least with llama.cpp, not sure they're all fixed yet. It assumed simple types for parameters for example.
Thanks for the link,it took qwen3.6-27B-q8 w/256k context on my RTX A6000 from ~20t/s to 55t/s. Prefill is mysteriously slower however, but prefill is so much faster still that I think I'm still bottlenecked on output most of the time.
I don’t exactly know where MTP inference fits within the inference stack, but does someone know whether it’s possible to implement it for the MLX universe?
Why when asking a model to change text in a minor way; are we not asking it to generate the operational transformations necessary to modify the text, and then just executing the ot on the existing text vs reproducing every token? Maybe tools are doing that more than I realize?
The only thing a model can output is tokens; to achieve this, a tool of converting tokens into operational transformations is required. For example, I have an ast-grep skill, it will instruct the model to generate ast-grep rules and run ast-grep to perform file modifications.
I am saying to directly output the operational transformation instructions as the tokens. You’re essentially telling it to “write the diff” and then applying the patch.
A coding agent provides an edit tool to the model for making code changes, with this tool typically just providing a find/replace operation. The model may of course need to do a bunch of work (grepping etc) to figure out what to change, but the actual change will just be sent from model to agent as a "replace X with Y" edit tool request, with this "edit" then done locally by the agent.
It's interesting how the agent (at least in case of Claude Code) is then applying this find/replace "edit" to the requested file... Since the agent wants to be platform independent (Linux/Windows/Max) it uses Node.js for file access, and performs the "edit" by itself using Node.js to read the entire file, make the change, then write back an entire new file.
The simple answer is: because it is not necessary to achieve the same final output. Most LLMs today are trained as autoregressive token predictors. They fundamentally can't work any other way. But we know how to train them really well and they have many applications beyond editing text. Diffusion LLMs exist too, which work a bit closer to what you describe, but they are not yet at the same level of intelligence since training methods are not that mature and they are generally less flexible as well.
So predict the tokens of the operational transformation.
I just asked: Write the operational transformation sequence and command to turn “this is really beautiful” to “this is very very beautiful”
and in return got: You can map this out by moving a virtual cursor across the text and telling it what to keep, remove, or add. You start by retaining the first eight characters to keep "this is " untouched. Then you delete the next six characters to remove the word "really". In that exact spot, you insert the nine characters for "very very". You finish the operation by retaining the final ten characters, which preserves the space and the word "beautiful". You can code this specific command sequence as [retain(8), delete(6), insert("very very"), retain(10)].
In a large paragraph of text I would expect it to be way quicker and cheaper to generate “[retain(800), delete(6), insert("very very"), retain(10000)]” than repredict the entire remainder of the unedited text.
I have no idea when I’m being lied to anymore but allegedly Aider and Cursor work the way I described, although cursor is using a second model to apply the edit.
They all do something similar under the hood. Patching files is not a trivial task when you only have the changed text content and not the actual file structure to work with. It kind of works, but is fundamentally limited by the LLM output architecture.
Cursor has a dedicated merge model. It takes input like this:
class Foo {
// ....
int calculation() {
return 42;
}
// more stuff
}
where the main model emits something that is a sort of casual under-specified diff format and the merge model figures out how to interpret it as a patch.
I've seen Claude use sed to edit files on other hosts instead of copying the file back and forth to edit it. Not quite full blown OT but it's going in that direction.
According to the linked PR, the original model does come with MTP which is another "head" (=output path) in the same model and (supposedly) runs faster.
The current implementation ignores that head but the PR let the tool recognize it, plus does proper integration (run the MTP while running the slower main path then compare the result, I believe.)
The standard way of doing MTP is to run the drafter autoregressively for k steps, and then (not concurrently) use the larger model as a verifier for those k tokens at the same time. The larger model can then accept a prefix of those k tokens, and in any case generates one more token (which is needed in case you accepted zero tokens from the drafter). The larger model can effectively use this k as a "batch" dimension, reducing the penalty of large weight loading. Meanwhile the drafter is much smaller, so it's fine for _it_ to be autoregressive, as long as the main model is parallel.
Yeah important conceptually to remember MTP is kind of just more weights, but speculative decoding is the runtime algorithm that’s a significant add to whatever code is serving the model.
How so? I’m not saying most of work doesn’t go into creating the drafting model or enabling a new head on the primary model, but the point is that however cool it is the result is, more weights. Speculative decoding requires code to be aware of how this works at the inference level.
Ideally the fees would be similar to the Norway model, where some tickets are tied to the income of the driver, in this case the pre-tax earnings of the company that created the driverless car.
That can make sense (opinions differ) for individuals, but it's not like the company is advertising with "we get you there at 1.2x legal speed". They're not competing on that; they're not choosing to do this on purpose like an individual might choose to speed (for example because of economic incentives if their hourly price is high)
If they were, then it makes sense to fine them to some multiple of the benefit they got from this advertising tactic, but as it is, I don't see why it should be different from anyone else's ticket. The company isn't likely to enjoy a flood of this administrative work, besides the cost of the actual fines, so they'll work to minimise them anyway
They may not advertise “getting you there at 1.2x legal speed” but the sooner they drop you off, the sooner they can get another fare. Over a whole fleet it will add up to changing the size of the required fleet.
If getting a ticket one ride in a thousand is cheaper than deploying another 2000 cars to make up for the increased trip time I’d expect them to keep getting tickets.
I’m also not sure they don’t do it on purpose. Tesla self driving has an aggressive mode willing to speed and roll through stop signs. Those were deliberate, law breaking, choices.
Oh, I actually read an article about that this morning in bed but didn't imagine regulators would approve a mode that is programmed to ignore laws. Not sure how it works where you're from but in the Netherlands, where Tesla did the approval process to subsequently apply for an EU license for using the drive assistance mechanism, the regulator "hopes to not need more than a week to approve safety updates. Other updates can take longer". They test every change anew it sounds like, at least at first (there was some talk of relaxing reporting requirements if accident rates are as expected, that is, lower than human drivers when using the presently approved assist methods)
Seems foreign (heh) to me that a regulator would allow software to drive lethal machinery on public roads without checking that it at least stops at a stop sign. Fining them proportional to turnover makes a lot more sense suddenly. Or maybe a competent regulator, like, prevention rather than after the fact action
Just like human_brain.exe right? I miss signs, and notice others missing signs, in unfamiliar areas all the time. So why not the same punishment and thus the same incentive not to do that?
A sibling comment mentioned that they did willfully build a variant that breaks laws btw. If there is apparently no regulator that needs to approve what makes it onto the road, I can understand a lot better why people are calling for insurmountable fines such that they have no choice but to remove that. Why not call for adequate approval mechanisms though, looking at other countries' examples where this problem doesn't exist... I dunno
Isn't Norway only for drunk driving? Finland has it for massive speed excesses, but it is based on net taxable income taking out business expenses for taxi drivers, and Waymo is still negative.
If they become profitable you'd want to normalize by number of miles, unless you just want an incentive system to get more people on the road (extra drivers) and increase chance of humans suffering road injuries to boost employment in an internal service sector.
But even then coming out with a more efficient fleet than a competitor for higher margin would be penalized. You'd rather disincentivize skimping on safety for margin and not disincentivize better maintenance and fuel economy.
Assuming you divide it down to the earnings per car, that makes perfect sense. Of course right now they aren't making any profit at all, and by the time it is relevant it is likely that the cars will commit substantially no violations at all.
Various services have existed, such as portmap(8), though NFS and similar services have often suffered from the "too complicated to debug" problem where devops (then sysadmins) would try turning the system off and then back on again in the hopes of resolving the issue du jour. You might get lucky and determine that node number three (of many) was cursed and leave it switched off for the Season of Mammon, more commonly known as Christmas, and to retire it quietly, later. Hypothetically.
Generally host and port mapping gets shoved somewhere into the configuration management layer and hopefully does not become too complicated (or have too many security holes) as this could vary from "configuration files and a few scripts" to database and services layers that few can debug, especially not a sysadmin at 3 AM in the morning running on an hour of bad sleep. Hypothetically.
this is a nice idea, but
idk why, in macos if i do
`nc -l 127.0.0.1 gopher`
and then try to open url "http://127.0.0.1:gopher/" - safari does not open it, no requests visible in the `nc` output.
* URL rejected: Port number was not a decimal number between 0 and 65535
* Closing connection
curl: (3) URL rejected: Port number was not a decimal number between 0 and 65535
so the ports are named, it is nice, but in practice it does not make life easier.
i chose gopher port just as an example. try with any other service name mapped to a port number from /etc/services and the result will be the same. the OP's goal was to use many http/https services, so we are talking about many http(s) services.
i just wanted to make the point that even if you have service names in /etc/services, it is not possible to use that names easily to host/access http(s) services.
The names are the kind of servers that listen on those ports (by default) like ssh, telnet, http, and smtp. They are not subdomains or for URI parsing.
URI contains ":port" tho, but practically it is only digital number.
the OP made a tool which helps them to avoid using port numbers. people commented in a way that looked like laughing at him, like he reinvented the wheel, and talking about /etc/services. well ok, i decided to try using /etc/services for the purpose of using names instead of port numbers.
would it be possible to add "myapp 60001/tcp" to /etc/services and then work with "http://localhost:myapp"? NO! browsers do not translate these names into port numbers. netcat does. curl does not.
so probably the OP's solution is not that questionable and really solved their need? and "good old friend /etc/services" is not useful for this? i dont know what it is useful for as running services on non-standard ports actually helps with hiding from security/vuln scanners and is practiced widely.
Maybe the joke has gotten too far, but the point that most people have been trying to say is that the issue that the OP is trying to solve has been solved for years. It's called a reverse proxy. Doing the configuration automatically like OP does does not alter the fact that it is a reverse proxy. `/etc/services` serve a different purpose.
Also URL parsing is a completely different matter. Browsers are primarily an HTTP(s) client. If you do not mention the ":port", it will try to connect to 80 (HTTP) or HTTPS (443). Because that the default ports for a web server. Other services have different port. So if your URL has the ftp scheme, the default port would be 21.
I know you're trying to be funny but ... technically it's 100% clear: You should talk HTTP, because that's the URL scheme here. The port makes no difference. You just happened to use a port by name. For all we know I run my HTTP server on some NFS related port so all the script kiddies try all the wrong exploits on it or something ;)
Well, the entire context of this is https so anything else is immaterial. The only reason it would be gopher is if you didn't read the post or don't understand the basics of https.
This is not possible since it is ambiguous. It is impossible to parse “http://127.0.0.1:gopher/“ since that would be valid as either “scheme://user:host/“ or “scheme://host:portname/“.
if you configure sshd to listen on port 443, does it become an https server? i was just trying to demondtrate: pick any port from /etc/services and try to use the name instead of port number. no, it does not work well when trying to use for local-hosting http(s) services. so to address the irony and sarcasm of the messages i was replying to:
zdw: It's like someone should make a file... maybe in /etc ... and put short names for services in it... maybe it could be called /etc/services...
tolciho: And then they might code up some sort of service lookup tool thingy to use on the train wreck that is the modern web.
$ getent services gopher
gopher 70/tcp
Many clients also do not support getservent(3) or portmap or DNS SRV records or NIS or LDAP or ActiveDirectory so one might wonder why there are so many half-baked, failed, or overy complicated attempts at solving whatever the problem is here even before "AI has entered the chat".
i know, but the OP's goal was to host/access http(s) services with names and avoid port numbers, and gopher service name was chosen by me as an example. my point was that /etc/services cannot be used for the OP's need.
if you host an http(s) service on port 11111 you can reach it with url http://127.1:11111, but url http://127.1:vce/ would not work in most software.
Try http://127.0.0.1:hkp instead of http://127.0.0.1:11371 for an OpenPGP HTTP keyserver. HTTP will work, but using the service name won't. Does that make what they're trying to say clearer?
Perhaps we could even make the file the port itself, perhaps calling it a “socket”? A “unix socket” would be a great name. If we could place all these files behind a local reverse proxy then we could use localhost/jekyll or localhost/fastapi. It’s just a dream
That would mean not being able to vibe code up an entire app to deal with something as insurmountable as looking at a list of numbers and post it on HN for those sweet, sweet upvotes. Why would they not do that.
Sure, but they are running web-apps they've vibe-coded (hence the .vibe tld) and for that use-case of many web apps that I run in docker containers I use nginx-proxy [0]. All the container needs is a VIRTUAL_HOST environment variable with the domain and what my router needs is an address entry for the wildcard subdomains. I even have nginx-proxy on a internet-accessible staging server.
Not modern enough. Unix is too low level, antiquated, and discriminates against those who just want to get shit done instead of reading manpages or documentation by hand.
This is exact problem I see with all of those vibe coded software: In few years everything will be super fragmented, everyone will be using their own set of tools, or vibe coding them, themselves. Communication between teams or even between team members will become very hard because of those differences. 'What do you mean production is down? On my vibe coded dashboard everything is green!'
Why do people always assume that change is permanent?
It's never.
After decentralisation we always see decentralisation.
After a period of growth, a decline will follow.
After the vibe coding hype, consolidation will follow.
After rain comes sunshine.
Well, I am forty years old. In twenty years I won't be working. We are in the dawn of AI software. If this will be decentralisation it will be at least 10 years of that. Probably more. So if fifteen or twenty years things will be going back to 'our normal' as we see today, I won't be concerned about all of it that much anymore. Perspective change with age considerably.
I am pretty convinced you need root on most systems to update DNS resolution mechanism system-wide (eg. to edit /etc/hosts or run a local DNS server and put that into /etc/resolv.conf).
Technically you can set the HOSTALIASES variable to point to a custom hosts file, but that only works with programs that use gethostbyname(3). (Which is most of them? IDK.)
The article is about the dude not knowing what service is where so he codes a json mapping. He could just update his /etc/services for the same thing. Oh but wait, he mentioned ai agents that changes everything!
you go and look in etc services for what is bound to port 5009. the article might not be the most useful but these comments are completely off the mark and stupid.
Maybe not hate, but I think that kind of blog, and every single person posting AI slop to LinkedIn, deserve to be shamed publicly for that. It's just that no one does that, and those who do are frowned upon, and down voted to death, like here. The reason I asked "Why?" was to confirm that in fact there are others who think doing that is shameful, that I am not the only one. The outcome is disappointing.
HTTP 1.1 and later will have the browser supply the domain name that was used to access the site, and even though *.localhost all resolve to 127.0.0.1, nginx will pluck out the correct configuration and proxy_pass the correct one.
> It's like someone should make a file... maybe in /etc ... and put short names for services in it... maybe it could be called /etc/services...
People shit-talk container orchestration systems like Kubernetes, but if anything they greatly simplified (if not completely eliminated) the need for this sort of network bookkeeping.
Your comment unironically is something I prefer and one of my biggest pain points with Linux.
As a newb, I'm sure there's something called with a mycommonproblemd name that has a stateful interface. But sometimes that all adds up to make things feel complex. And it let's me make stupid mistakes, like I forgot to close or open a port on firewalld, or I disabled a container but forgot to commit a change to my systemd units.
It's nice to just have a nice file called myservice.nix that tracks the firewall port, name, systemd startup and update scripts.
Hacker News loves to make snarky comments about everything to do with K8s and YAML all the time, and yet in my experience the amount of times an issue was caused by actual YAML can be counted on one hand.
Way more often it’s developers who can’t figure out that their http library only supports 2 concurrent connections, or emit garbage/malformed log lines and then bitch that they can’t see their app logs because we dropped them, or can’t be fucked to do “kubectl describe” in their own developer namespace that they have full permission for.
If you truly experience issues with just using YAML then you need to skill up probably.
Most of the issues with YAML are really issues with people who think that since "configuration as code" is good, that "code as configuration" must also be good.
No, go ahead. Tell me how just using /etc/services does what this does. Because I'm calling bullshit.
But go ahead. /etc/services, please, share with me how it's setup to do thing likes create the HTPS and makes it trusted and sets up the domain. Go ahead.
Go ahead. You can ONLY use /etc/services.
Or, you are admit you don't actually have a clue as to what /etc/services does.
IMO, Kubernetes isn't inevitable, and this seems to paint it as such.
K8s is well suited to dynamically scaling a SaaS product delivered over the web. When you get outside this scenario - for example, on-prem or single node "clusters" that are running K8s just for API compatibility, it seems like either overkill or a bad choice. Even when cloud deployed, K8s mostly functions as a batteries-not-included wrapper around the underlying cloud provider services and APIs.
There are also folks who understand the innards of K8s very well that have legitimate criticisms of it - for example, this one from the MetalLB developer: https://blog.dave.tf/post/new-kubernetes/
Before you deploy something, actually understand what the pros/cons are, and what problem it was made to solve, and if your problem isn't at least mostly a match, keep looking.
This is a need it fails at miserably. k8s reminds me of the raid recentralization anti pattern problem where you fix a hardware failure that never occurs in exchange for knowing simple higher level mistakes or security problems will tank something now too large to fail again.
Kubernetes, in the form of k3s, was a critical success factor for us with the onprem deployment of our SaaS product.
What's the problem with a single-node cluster? We use that for e.g. dev environments, as well as some small onprem deployments.
> Even when cloud deployed, K8s mostly functions as a batteries-not-included wrapper around the underlying cloud provider services and APIs.
Which batteries are not included? The "wrapper around the underlying cloud provider services and APIs" is enormously important. Why would you prefer to use a less well-designed, more vendor-specific set of APIs?
I seriously don't get these criticisms of k8s. K8s abstracts away, and standardizes, an enormous amount of system complexity. The people who object to it just don't have the requirements where it starts making sense, that's all.
> Kubernetes, in the form of k3s, was a critical success factor for us with the onprem deployment of our SaaS product.
What surprises and gotchas did you have to deal with using k3s as a Kubernetes implementation?
Did you use an LB? Which one? I'm assuming all your onprem nodes were just linux servers with very basic equipment (the fanciest networking equipment you used were 10GbE PCIe cards, nothing more special than that?)
We sell to enterprise customers. All of them deploy our solution on internal cloud-style VM clusters. We use the Traefik ingress controller by default.
There really weren't any particular surprises or gotchas at that level.
In this context, I've never had to deal with anything at the level of the type of Ethernet card. That's kind of the point: platforms like k8s abstract away from that.
It's also difficult for data pipelines or data intensive things. At several companies we've run into the "Need to put ML model behind API and pods get killed because health checks via API are basically not compatible with container fully under load but still working"
They previously did the entire famicon library.
reply