To be fair, the money/time tradeoff breaks down pretty quickly - you've almost certainly used all the easy wins by the time you're even earning mid six-figures.
Basically, there are only so many hours of outsourceable stuff in your day. Once you've made the trade for cooking (eating out/chef/stay-home-spouse), cleaning (maid/stay-home-spouse/etc), errands, scheduling, minor research, and other miscellany (secretary), local transport (owning a car) there isn't much more to do.
If you really value your time, you can fly a private jet at the cost of maybe a few thousand dollars per hour saved, I guess.
And Warren Buffet, even in his frugality, does all of these things.
You say "by the time you're even earning mid six-figures", but "mid six-figures" is a whole lot of money. (Of course it's not much relative to Warren Buffett. but then what is?)
... Oh, wait, by "mid six-figures" do you mean $150k or $500k? Even $150k is a very nice salary by most people's standards, but I was assuming you meant $500k. (Six figures: 100k to 999k. Midpoint of that range is $550k. Perhaps the geometric mean is more natural; that's about $300k.)
The point he was supporting was "no matter how rich you are, there's only so much time you can get by trading money for it," in which context $500k/year isn't out of place--especially, as you noticed, with Warren Buffett as the example.
I meant around $500K. I think that's the point where it's reasonably likely you have a secretary/personal-assistant to take care of those last little bits of outsourceable work.
I think frugal here is relative. He doesn't care about status symbols and living a super extravagant life with huge mansions and yachts that other billionaires do. But I'm sure he has a lot of expenses that most middle class would not have.
If his car breaks down, do you really think he'll take it to the shop himself and start taking a bus to work?