When I was there, DigitalOcean was writing a complete replacement for the Ceph S3 gateway because its performance under high concurrency was awful.
They just completely swapped out the whole service from the stack and wrote one in Go because of how much better the concurrency management was, and Ceph's team and codebase C++ was too resistant to change.
Unrelated, but one of the more annoying aspects of whatever software they use now is lack of IPv6 for the CDN layer of DigitalOcean Spaces. It means I need to proxy requests myself. :(
I confirm this, I used SeaweedFS to serve 1M users daily with 56 million images / ~100TB with 2 servers + HDD only, while Minio can't do this. Seaweedfs performance is much better than Minio's.
The only problem is that SeaweedFS documentation is hard to understand.
They seem to have gone all-in on AI, for commits and ticket management. Not interested in interacting with that.
Otherwise, the built in admin on one-executable was nice, and support for tiered storage, but single node parallel write performance was pretty unimpressive and started throwing strange errors (investigating of which led to the AI ticket discovery).
I'm not sure if it even has any sort of cluster consensus algorithm? I can't imagine it not eating committed writes in a multi-node deployment.
Garage and Ceph (well, radosgw) are the only open source S3-compatible object storage which have undergone serious durability/correctness testing. Anything else will most likely eat your data.
Hi there, RustFS team member here! Thanks for taking a look.
To clarify our architecture: RustFS is purpose-built for high-performance object storage. We intentionally avoid relying on general-purpose consensus algorithms like Raft in the data path, as they introduce unnecessary latency for large blobs.
Instead, we rely on Erasure Coding for durability and Quorum-based Strict Consistency for correctness. A write is strictly acknowledged only after the data has been safely persisted to the majority of drives. This means the concern about "eating committed writes" is addressed through strict read-after-write guarantees rather than a background consensus log.
While we avoid heavy consensus for data transfer, we utilize dsync—a custom, lightweight distributed locking mechanism—for coordination. This specific architectural strategy has been proven reliable in production environments at the EiB scale.
Is there a paper or some other architecture document for dsync?
It's really hard to solve this problem without a consensus algorithm in a way that doesn't sacrifice something (usually correctness in edge cases/network partitions). Data availability is easy(ish), but keeping the metadata consistent requires some sort of consensus, either using Raft/Paxos/..., using strictly commutative operations, or similar. I'm curious how RustFS solves this, and I couldn't find any documentation.
EiB scale doesn't mean much - some workloads don't require strict metadata consistency guarantees, but others do.
I’m Elvin from the RustFS team in the U.S. Thanks for sharing the benchmark; it’s helpful to see how RustFS performs in real-world setups.
We know trust matters, especially for a newer project, and we try to earn it through transparency and external validation. we were excited to see RustFS recently added as an optional service in Laravel Sail’s official Docker environment (PR #822). Having our implementation reviewed and accepted by a major ecosystem like Laravel was an encouraging milestone for us.
If the “non-technical reasons” you mentioned are around licensing or governance, I’m happy to discuss our long-term Apache 2.0 commitment and path to a stable GA.
https://www.repoflow.io/blog/benchmarking-self-hosted-s3-com... was useful.
RustFS also looks interesting but for entirely non-technical reasons we had to exclude it.
Anyone have any advice for swapping this in for Minio?