I guess I'm curious if I'm understanding what you mean here, because it seems like there's a huge number of counterexamples. GNU coreutils. The linux kernel. FreeBSD. NFS and iSCSI drivers for either of those kernels. Cgroups in the Linux kernel.
If anything, it seems strange to expect to be able to monetize free-as-in-freedom software. GNU freedom number 0 is "The freedom to run the program as you wish, for any purpose". I don't see anything in there about "except for business purposes", or anything in there about "except for businesses I think can afford to pay me". It seems like a lot of these "open core" cloud companies just have a fundamental misunderstanding about what free software is.
Which isn't to say I have anything against people choosing to monetize their software. I couldn't afford to give all my work away for free, which is why I don't do that. However, I don't feel a lot of sympathy to people who surely use tons of actual libre software without paying for it, when someone uses their libre software without paying.
The trick here is that people may not want to be coding MinIO. It's like... just not that fun of a thing to work on, compared to something more visible, more elevator-pitchy, more sexy. You spend all your spare time donating your labour to a project that... serves files? I the lowly devops bow before you and thank you for your beautiful contribution, but I the person meeting you at a party wonder why you do this in particular with your spare time instead of, well, so many other things.
I've never understood it, but then, that's why I'm not a famous open-source dev, right?
Already a few more ideas I want to code :)
But this might create the problem image models are facing, AI eating itself...
Companies pay their employees to work on Linux because it's valuable to them. Intel wants their hardware well supported. Facebook wants their servers running fast. It's an ecosystem built around free-as-in-freedom software, where a lot of people get paid to make the software better, and everyone can use it for free-as-in-beer
Compare that to the "open core" model where a company generally offers a limited gratis version of their product, but is really organized to funnel leads into their paid offering.
The latter is fine, but I don't really consider it some kind of charity or public service. It's just a company that's decided on a very radical marketing strategy.
It's not clear if RustFS is even implementing a proper distributed consensus mechanism. Erasure Coding with quorum replication alone is not enough for partition tolerance. I can't find anything in their docs.
I recently learned that Ceph also has an object store and have been playing around with microceph. Ceph also is more flexible than garage in terms of aggregating differently sized disks. Since it's also already integrated in Proxmox and has over a decade of enterprise deployments, that's my top contender at the moment. I'm just not sure about the level of S3 API compatibility.
Any opinions on Ceph vs RustFS?
If you're looking for something that won't eat your data in edge cases, Ceph (and perhaps Garage) are your only options.
Why?
I don't think open source projects need a path to monetization in all cases, most don't have that. But if you make such a project your main income, you certainly need money.
If you then restrict the license, you are just developing commercial software, it then has little to do with open source. Developing commercial software is completely fine, but it simply isn't open source.
There is also real open source software with a steady income and they are different than projects that change to commercial software and we should discriminate terms here.
I haver not used it but will be likely a good minio alternative for people who want to run a server and don't use minio just as s3 client.
Recently SeaweedFS is moving fast and added a lot more features, such as: * Server Side Encryption: SSE-S3, SSE-KMS, SSE-C * Object Versioning * Object Lock & Retention * IAM integration * a lot of integration tests
Also, SeaweedFS performance is the best in almost all categories in a user's test https://www.repoflow.io/blog/benchmarking-self-hosted-s3-com... And after that, there is a recent architectural change that increases performance even more, with write latency reduced by 30%.
Thank you for your work. I was in a position where I had to choose between minio and seaweed FS and though seaweed FS was better in every way the lack of an includes dashboard or UI accessibility was a huge factor for me back then. I don't expect or even want you to make any roadmap changes but just wanted to let you know of a possible pain point.
No test suite, plenty of regressions, and data loss bugs on core code paths that should have been battled tested after so many years. There are many moving parts, which is both its strength and its weakness as anything can break - and does break. Even Erasure Coding/Decoding has had problems, but a guy from Proton has contributed a lot of fixes in this area lately.
One of the big positive in my opinion, is the maintainer. He is an extremely friendly and responsive gentleman. Seaweedfs is also the most lightweight storage system you can find, and it is extremely easy to set up, and can run on servers with very little hardware resources.
Many people are happy with it, but you'd better be ready to understand their file format to fix corruption issues by hand. As far as i am concerned, i realized that after watching all these bugs, the idea of using seaweedfs was causing me more anxiety than peace of mind. Since we didn't need to store billions of files yet, not even millions, we went with creating a file storage API in ASP.NET Core in 1 or 2 hours, hosted on a VPS, that we can replicate using rsync without problem. Since i made this decision, i have peace of mind and no longer think about my storage system. Simplicity is often better, and OSes have long been optimized to cache and serve files natively.
If you are not interested in contributing fixes and digging into the file format when a problem occurs, and if your data is important to you, unless you operate at the billions of files scalability tier Seaweedfs shines at, i'd suggest rolling your own boring storage system.
For our use case (3 nodes, 61TB of NVMe) it seems like the best option out of what I looked at (GarageFS, JuiceFS, Ceph). If we had 5+ nodes I'd probably have gone with Ceph though.
Any alternatives besides racking own servers?
*EDIT* Did a little ChatGPT and it recommended tiny t4g.micro then use EBS of type cold HDD (sc1). Not gonna be fast, but for offsite backup will probably do the trick.
https://www.hetzner.com/storage/storage-box/
It's not as fast as local storage of course, but it's cheap!
I'm not saying it's wrong since I don't know what it's for, I'm just wondering what the use-case could be.
For backup to a dedi you don't really need to bother running the object store though.
Not an issue at all until they do.
I can't really say anything about advanced features or operational stability though.
The wording gives an Apache license only to MinIO, not to people who use it. So MinIO can relicense the the contributor code under a commercially viable license, but no one else can. Everyone else will only have access to the contribution under AGPL as part of the whole project.
The closest alternative seems to be RustFS. Has anyone tried it? I was waiting until they support site replication before switching.
I hadn't heard of RustFS and it looks interesting, although I nearly clicked away based on the sheer volume of marketing wank on their main page. The GitHub repo is here: https://github.com/rustfs/rustfs
I don’t begrudge it that. I get the impression that Garage isn’t necessarily focussed on this kind of use case.
After years of using Garage for S3 for the homelab I’d never pick anything else. Absolutely rock solid, no problem whatsoever. There isn’t ONE other piece of software I can say that about, not ONE.
Major kudos to the guys at deuxfleurs. Merci beaucoup!
They have a lot of obviously fake quotes from non-existent people at positions that don’t even mention what company it is. The pictures are misgendered and even contain pictures of kids.
Feels like the whole page is AI generated.
So, arguably worse than MinIO.
This is not that. This is not even a license. They want a full transfer of intellectual property ownership. Sure that enables them to use it in a commercial product, but it also enables them to sue if contributors contribute similarly to other projects. Obviously that would create a shit storm, and there is an exception with the public license, but riddle me this: can you legally make similar contributions to multiple projects that have this type of CLA?
Let us take a step back and instead look where such terms are more common: employment contracts.
You don't need assignment to the project if you are not planning to change project's license.
You do need assignment to the project if you need to ever rugpull the community and close the code
The number of weird incompetent things the Minio people have done is surprisingly high.
This was going to be my next project, as I am currently storing my Kopia/Ente on MinIO in a non-distributed way. MinIO project going to shi*s is a good reason to take on this project faster than later.
I'd take a look at garage (didn't try seaweed yet).
Is it open to the public? I'd like to check it out
It's promising, but definitely check the roadmap before deploying at scale.
The core is stable at this point, but the user/policy management and the web interface is still in the works.
That said, if there was contributed AGPL code, they couldn't change the license on that part of the code w/o a CLA. AGPL also doesn't necessarily mean you have to make the code publicly available, just available to those that you give the program to (I'm assuming AGPL is like the GPL in this regard).
So, that I'd be curious about it is -- (1) is there any contributed AGPL code in the current version? (2) what license is granted to customers of the enterprise version?
Minio can completely use whatever license they want for their code. But, if there was contributed code w/o a CLA, then I'm not sure how a commercial/enterprise license would play with contriubuted AGPL code. It would be an interesting question to find out.
This is the crucial difference between the AGPL and the GPL: the AGPL requires you to make the code available to users for whom you run the code, as well as users you give the program to.
But, what I don't know is -- is there any other AGPL code that minio doesn't own, but that was otherwise contributed to minio? Because, presumably, they aren't actually giving their customers the minio program with an AGPL license, rather they have whatever their enterprise license agreement is. If this is the case, and there is AGPL code that's not owned by Minio, I can foresee problems in the future.
Additionally, in order to CHANGE the license, if others contributed code under that license, you would need their permission, on top of the fact, you cannot retroactively revoke the license for previous versions.
I can't see how this is a defensible position for Minio, but I'm not sure they really care that much at this point.
(I personally choose not to contribute to projects with CLAs, I don't want my contributions to become closed-source in the future.)
https://blog.min.io/weka-violates-minios-open-source-license...
They think they can revoke someone's AGPL license. That's not at all how that license works!
I have zero knowledge about the squabble between MinIO and Weka. I don't know, and don't care, if either of them is in the right. But if Weka isn't complying with the terms of the AGPL, then MinIO has the legal right to tell them they can no longer distribute MinIO's licensed work at all, because nothing else grants them that privilege.
If that weren't true, there'd be no teeth to the A?GPL whatsoever.
> All rights granted under this License are granted for the term of copyright on the Program, and are irrevocable provided the stated conditions are met.
it also says
> You may not propagate or modify a covered work except as expressly provided under this License. Any attempt otherwise to propagate or modify it is void, and will automatically terminate your rights under this License (including any patent licenses granted under the third paragraph of section 11).
> However, if you cease all violation of this License, then your license from a particular copyright holder is reinstated (a) provisionally, unless and until the copyright holder explicitly and finally terminates your license, and (b) permanently, if the copyright holder fails to notify you of the violation by some reasonable means prior to 60 days after the cessation.
> Moreover, your license from a particular copyright holder is reinstated permanently if the copyright holder notifies you of the violation by some reasonable means, this is the first time you have received notice of violation of this License (for any work) from that copyright holder, and you cure the violation prior to 30 days after your receipt of the notice.
This is in common with the GPLv3. It is much longer than the corresponding terms of the GPLv2 to remedy a sort of fragility in the GPLv2 which says your license terminates permanently if you ever violate the GPL, even temporarily and by accident.
I have no knowledge of whether Weka did or didn't violate the license, but if they did violate it and refused to fix it, MinIO's revocation of their license is completely in accordance with the terms of the license as written. I don't think a GPL termination case has yet been litigated.
It has, though it has mainly been under the "breach of contract" approach and not under "copyright infringement" approach. See https://en.wikipedia.org/wiki/Open_source_license_litigation
I do have a separate AGPL project (see github) where I have nearly all of the copyright and have looked into how one would be able to enforce this in the US at some point and it did look pretty bleak -- it is a civil suit where you have to show damages etc. but IANAL.
I did not like the FUD they were spreading about AGPL at the time since it is a good license for end-user applications.
What is the performance like for reads, writes, and deletes?
And just to play devil's advocate: What would you say to someone who argues that you've essentially reimplemented a filesystem?
W.r.t. filesystem: Yes, aware of this. Initially used minio and also implemented the use case directly on XFS as well and only had problems at larger scales (that still fit on a machine) with it. Ceph went into a similar direction with BlueStore (reimplementing the filesystem, but with RocksDB).
Looks like a great alternative.
https://github.com/minio/minio/issues/13308#issuecomment-929...
https://github.com/minio/minio/discussions/13571#discussionc...
> you may be violating AGPLv3 if you are using MinIO to build commercial products or using it for commercial purposes
Should I contact a MinIO salesman to purchase an enterprise license ASAP or is it fine if I license my kids and advent of code solutions under the AGPLv3 license ?
Or just that the users would need to make minio sources, including modifications, freely available?
I guess that's kind of the big question inherent to the AGPL?
If you use Minio or another AGPL licensed service internally to support your own product without a customer ever touching it's API, it should be fine.
So if the minio maintainers (or anybody that forks the project and wants to work it) can fix any security issues that may occur I don't see any problems with using it.
AFIK they removed it only to move it to their paid version, didn't they?
The UI was useful when first configuring the buckets and permissions; if you've got it working (and don't need to change anything) you're good to go. Also, everything can be configured without the UI (not so easily of course).
The concerning language for me is this part that was added:
> Critical security fixes may be evaluated on a case-by-case basis
It seems to imply that any fixes _may_ be merged in, but there's no guarantees.
Minio is/was pretty solid product for places where rack of servers for Ceph wasn't an option (it does have quite a bit higher memory requirements), or you just need a bit of S3 (like we have small local instances that just run as build cache for CI/CD)
But that's not where money is
This is a wild sentence--how can you criticize them for abandoning POSIX support __and__ building a massively over-complicated product? Making a reliable POSIX system is inherently very complex.
POSIX can be complicated, but it puts you in a nice ecosystem, so for some use-cases complex POSIX support is not over complicated. It is just… appropriately complicated.
"foo" and "foo/bar" are valid S3 object names that cannot coexist on a POSIX filesystem
Or are we saying that they no longer support running minio on POSIX systems at all, due to using linux specific syscalls or something else I'm not thinking of? I don't know whether they did this or not.
Those seem like two very different things to me, and when someone says "they don't support POSIX", I assume the latter
I don'T think they would abandon POSIX systems in general, because what sense would that make?
https://github.com/versity/versitygw
Unlike other options like Garage or Minio, it doesn't have any clustering, replication, erasure coding, ...
Your S3 objects are just files on disk, and Versity exposes it. I gather it exists to provide an S3 interface on top of their other project (ScoutFS), but it seems like it should work on any old filesystem.
He also mentioned that the minio-to-versity migration is a straight forward process. Apparently, you just read the data from mino's shadow filesystem and set it as an extended attribute in your file.
Thanks for posting this, as it's the first I've come across their work.
It's a "Misc" endpoint in the Garage docs here: https://garagehq.deuxfleurs.fr/documentation/reference-manua...
Thanks for pointing that out.
If the answer is the latter, seaweedfs is an option:
https://github.com/seaweedfs/seaweedfs?tab=readme-ov-file#qu...
Possibly of interest: s3gw[1] is a modified version of ceph's radosgw that allows it to run standalone. It's geared towards kubernetes (notably part of Rancher's storage solution), but should work as a standalone container.
[0] https://github.com/gaul/s3proxy [1] https://github.com/s3gw-tech/s3gw
It's not a fully featured s3 compatible service, like MinIO, but we used it to great success as a local on-prem s3 read/write cache with AWS as the backing S3 store. This avoided expensive network egress charges as we wanted to process data in both AWS as well as in a non-AWS GPU cluster (i.e. a neocloud)
IANAL, of course
Meanwhile, MinIO's own contributions and the distribution itself (outbound license) were AGPL licensed.
It's effectively a CLA, just a bit weaker, since they're still bound by the terms of Apache 2 vs. a full license assignment like most CLAs.
Choosing AGPL with contributors giving up rights is a huge red flag for "hey, we are going to rug pull".
Just AGPL by companies without even allowing contributor rights is saying, "hey, we are going to attempt to squeeze profit out and don't want competition on our SaaS offering."
I wish companies would stop trying to get free code out of the open source community. There have been so many rug pulls it should be expected now.
What is an AI Stor (e missing on purpose because that is how it is branded: https://www.min.io/product/aistor)
For production you'd need a proper cluster deployed via Helm, but for trying it out locally that setup is easy to get running.
Replication
A trusted identity provider is a
key component to single sign on.
It’s probably just Minio but it costs more money.
fwiw while they do produce Docker containers for it, it's also extremely simple to run without that - it's a single binary and running it with systemd is unsurprisingly simple[1].
0: https://garagehq.deuxfleurs.fr/
1: https://garagehq.deuxfleurs.fr/documentation/cookbook/system...
What if the sponsorships run out?
You shouldn't expect guarantee of any kind.
You can always fork the project, then the questions of sponsorships still remains.
Recently Ghostty is a non profit, which means that it is guaranteed not to turn into a for profit and rugpull like what MinIO has done.
In the end open source allows motivated people to take over the project if you aren't willing to do it yourself but projects can also die of lack of motivated/paid resources.
quay.io/minio/minio:RELEASE.2025-04-22T22-12-26Z
Here's a simple script that does it automagically (you'll need golang installed):
> build-minio-ver.sh
#!/bin/bash
set -e
VERSION=$(git ls-remote --tags https://github.com/minio/minio.git | \
grep -Eo 'RELEASE\.[0-9T-]+Z' | sort | tail -n1)
echo "Building MinIO $VERSION ..."
rm -rf /tmp/minio-build
git clone --depth 1 https://github.com/minio/minio.git /tmp/minio-build
cd /tmp/minio-build
git fetch --tags
git checkout "$VERSION"
echo "Building minio..."
CGO_ENABLED=0 go build -trimpath \
-ldflags "-s -w \
-X github.com/minio/minio/cmd.Version=$VERSION \
-X github.com/minio/minio/cmd.ReleaseTag=$VERSION \
-X github.com/minio/minio/cmd.CommitID=$(git rev-parse HEAD)" \
-o "$OLDPWD/minio"
echo " Binary created at: $(realpath "$OLDPWD/minio")"
"$OLDPWD/minio" --versionIt is useful to remember that one may fork at the commit before a license change.
I assume forks, and software that uses them will be held to the same requirements.
https://opensource.google/documentation/reference/using/agpl...
You create a proprietary piece of software. You license it to Google and negotiate terms. You then negotiate different terms with Microsoft. Nothing so far prevents you from doing this. You can't yank the license from Google unless your contract allows that, but maybe it does. You can in theory then go and release it under a different license to the public. If that license is perpetual and non-revokable then presumably I can use it after you decide to stop offering that license. But if the license is non-transferrable then I can't pass on your software to someone else either by giving them a flash drive with it, or by releasing it under a different license.
Several open source projects have been re-licensed. The main thing that really is the obstacle is that in a popular open source or copyleft project you have many contributors each of which holds the copyright to their patches. So now you have a mess of trying to relicense only some parts of your codebase and replace others for the people resisting the change or those you can't reach. It's a messy process. For example, check out how the Open Street Maps data got relicensed and what that took.
My understanding of what they meant by "retroactively apply a restrictive license" is to apply a restrictive license to previous commits that were already distributed using a FOSS license (the FOSS part being implied by the new license being "restrictive" and because these discussions are usually around license changes for previously FOSS projects such as Terraform).
As allowing redistribution under at least the same license is usually a requirement for a license to be considered FOSS, you can't really change the license of an existing version as anyone who has acquired the version under the previous license can still redistribute it under the same terms.
Edit: s/commit/version/, added "under the same terms" at the end, add that the new license being "restrictive" contributes to the implication that the previous license was FOSS
Copyright law is also a complex matter which differs by country and I am not a lawyer so take this with a grain of salt, but there seem to be "edge cases" where the license can be revoked as seen in the stackexchange page below.
See:
https://lists.opensource.org/pipermail/license-discuss_lists...
https://opensource.stackexchange.com/questions/4012/are-lice...
That got backlash so now it’s just getting dropped entirely?
People get to do whatever they want but bit jarring to go from this is worth something people will pay for to maintenance mode in quick succession
That's literally what the commit shows that they're doing?
> *This project is currently under maintenance and is not accepting new changes.*
> For enterprise support and actively maintained versions, please see MinIO SloppyAISlop (not actual name)
Start open source to use free advertising and community programmer, and then dumps it all for commercial licensing.
I think n8n is next because they finished the release candidate for version 2.0, but there are no changelogs.
Its like GET <namespace>/object, PUT <namespace>/object. To me its the most obvious mapping of HTTP to immutable object key value storage you could imagine.
It is bad that the control plane responses can be malformed XML (e.g keys are not escaped right if you put XML control characters in object paths) but that can be forgiven as an oversight.
Its not perfect but I don't think its a strange API at all.
My browser prints that out to 413 pages with a naive print preview. You can squeeze it to 350 pretty reasonably with a bit of scaling before it starts getting to awfully small type on the page.
Yes, there's a simple API with simple capabilities struggling to get out there, but pointing that out is merely the first step on the thousand-mile journey of determining what, exactly, that is. "Everybody uses 10% of Microsoft Word, the problem is, they all use a different 10%", basically. If you sat down with even 5 relevant stakeholders and tried to define that "simple API" you'd be shocked what you discover and how badly Hyrum's Law will bite you even at that scale.
> My browser prints that out to 413 pages with a naive print preview. You can squeeze it to 350 pretty reasonably with a bit of scaling before it starts getting to awfully small type on the page.
idk why you link to Go SDK docs when you can link to the actual API reference documentation: https://docs.aws.amazon.com/AmazonS3/latest/API/API_Operatio... and its PDF version: https://docs.aws.amazon.com/pdfs/AmazonS3/latest/API/s3-api.... (just 3874 pages)
https://developers.cloudflare.com/r2/api/s3/api/
It's not that much, most of weirder S3 APIs are optional, orthogonal APIs, which is good design.
And many S3-compatible alternatives (probably most but the big ones like Ceph) don't implement all of the features.
For example for lifecycles backblaze have completely different JSON syntax
3500 pages to describe upload and download, basically. That is pretty strange in my book.
Now with the trivial part off the table, let's consder storage classes, security and ACLs, lifecycle management, events, etc.
I’ve seen a lot of bad takes and this is one of them.
Listing keys is weird (is it V1 or V2)?
The authentication relies on an obtuse and idiosyncratic signature algorithm.
And S3 in practice responds with malformed XML, as you point out.
Protocol-wise, I have trouble liking it over WebDAV. And that's depressing.
I think it's only gotten as complicated as it has as new features have been organically added. I'm sure there are good use cases for everything, but it does beg the question -- is a better API possible for object storage? What's the minimal API required? GET/POST/DELETE?
For example, did you know that date filtering in S3 is based on string prefix matching against an ISO8601/RFC3339 style string representation? Want all objects created between 2024-01-01 and 2024-06-30? You'll need to construct six YYYY-MM prefixes (one per month) for datetime and add them as filter array elements.
As a result the service abbreviation is also incorrect these days. Originally the first S stood for "Simple". With all the additions they've had to bolt on, S2 would be far more appropriate a name.
S3 has 3 independent permissions mechanisms.
it's storing a [utf8-string => bytes] mapping with some very minimal metadata. But that can be whatever you want. JSON, CBOR, XML, actual document formats etc.
And it's default encoding for listing, management operations and similar is XML....
> but I feel like we missed an opportunity here for a standardized interface.
except S3 _is_ the de-facto standard interface which most object storage system speaks
but I agree it's kinda a pain
and commonly done partial (both feature wise and partial wrong). E.g. S3 store utf8 strings, not utf8 file paths (like e.g. minio does), that being wrong seems fine but can lead to a lot of problems (not just being incompatible for some applications but also having unexpected perf. characteristics for others) making it only partial S3 compatible. Similar some implementations random features like bulk delete or support `If-Match`/`If-Non-Match` headers can also make them S3 incompatible for some use cases.
So yeah, a new external standard which makes it clear what you should expect to be supported to be standard compatible would be nice.
1. I like that it is written in Go
2. I saw nothing above what Apache Spark+Hadoop with _consistent_ object stores already offers on Amazon (S3), Google Cloud (GCS), and or Microsoft (Azure Storage, ADLS Gen2)
Best of luck, maybe folks should look around for that https://donate.apache.org/ button before the tax year concludes =3
it was very simple to setup, and even if you just leased a bunch of servers off say OVH, far FAR cheaper to run your own than paying any of the big cloud providers.
It also had pretty low requirements, ceph can do all that but setup is more complex and RAM requirements far, far higher
For a proper Ceph setup, even the 45drives budget configuration is still not "hobby" grade.
I will have to dive into the MinIO manual at some point, as the value proposition still seems like a mystery. Cheers =3
But that's kind of advantage only on the small companies and hobbyist market, big company either have enough needs to run big ceph cluster, or to buy it as a service.
Minio is literally "point it at storage(s), done". And at far smaller RAM usage.
Ceph is mon servers, osd servers, then rados gatway server on top of that.
Yes, Ceph is RADOS at its core. However, RADOS != S3. Ceph provides an S3 compatible backend with the RADOS Gateway (RGW).
I don't understand yet exactly what MinIO would add on top of that to make it relevant at any scale. I'll peruse the manual on the weekend, because their main site was not helpful. Thanks for trying though ¯\_(ツ)_/¯
My only blocker for a fork to maintain compatibility and path to upgrade from earlier versions.
At the 1 billion valuation from the previous round, achieving a successful exit requires a company with deep pockets. Right now, Nvidia is probably a suitable buyer for MinIO, which might explain all the recent movements from them. Dell, Broadcom, NetApp, etc, are not going to buy them.
I think it is about 10 years old now and it is really stable.
For example, we were running a 20 node k8s cluster for our Cortex (distributed Prometheus) install, monitoring about 30k servers around the world, and it was generating a bit over a TB of data a day. It was a lot more cost effective and performant to create a minio cluster for that data than to use S3.
Also, you can get durability with minio with multi cluster replication.
We have already worked through all the pain points and it all works smoothly. No reason to change something that isn't a problem.
Self-hosted S3 clones with actual durability guarantees exist, but the only properly engineered open source choices are Ceph + radosgw (single-region, though) or Garage (global replication based on last-writer-wins CRDS conflict resolution).
Edit: some discussion of this here: https://news.ycombinator.com/item?id=46136871
Probably yes.
[0] https://github.com/Barre/ZeroFS
[1] https://canonical-microceph.readthedocs-hosted.com/stable/
The use case is fully stateless infrastructure: your file/database servers become disposable and interchangeable (no "pets"), because all state lives in S3. This dramatically simplifies operations, scaling, and disaster recovery, and it's cheap since S3 (or at least, S3 compatible services) storage costs are very low.
The MinIO article's criticisms don't really apply here because ZeroFS doesn't store files 1:1 to S3. It uses an LSM-tree database backed by S3, which allows it to implement proper POSIX semantics with actual performance.
What about NFS or traditional filesystems on iSCSI block devices? I assume you're not using those because managing/scaling/HA for them is too painful? What about the openstack equivalents of EFS/EBS? Or Ceph's fs/blockdev solutions (although looking into it a bit, it seems like those are based on its object store)?
There are two ways open source projects continue.
1. The creator has a real, solid way to make money (React by Facebook, Go by Google).
2. The project is extremely popular (Linux, PostreSQL).
Is it possible for people to reliably keep working for ~free? In theory yes, but if you expect that, you have a very bad understanding of 98% of human behavior.
There's also tonne of Open Source that isn't as popular but serving niche communities. It's definitely harder but not impossible. OS core and paid hosting with bells and whistles has proven to be a good sustainable model.
Redis, Elasticsearch, Terraform, MongoDB, CockroachDB have all changed their OSS licenses in recent years.
Also, Debian has been around for a few decades, although I do admit that - like the Linux kernel - that wouldn't have been possible without a lot of companies contributing back to the ecosystem.
Need to start reconsidering the approach now and looking for alternatives
Similar to the way Broadcom did with VMware hiking prices astronomically for their largest clients, and basically killing the SME offering.
https://garagehq.deuxfleurs.fr/
Edit: jeez, three of us all at once...
rclone serve s3 path/to/buckets --addr :9000 --auth-key <key-id>,<secret>
`Be wary that an OSD, whether based on a physical device or a file, is resource intensive.`
Can anyone quantify "resource intensive" here? Is it "takes an entire Raspberry Pi to run the minimum set" or is it "takes 4 cores per OSD"?
Edit: This is the specific doc page https://canonical-microceph.readthedocs-hosted.com/stable/ho...
minio was also suited for some smaller use cases (e.g. running a partial S3 compatible storage for integration tests). Ceph isn't really good for it.
But if you ran large minio clusters in production ceph might be a very good alternative.
I haven't tried it though. Seems simple enough to run.
Am forced to use MinIO for certain products now but will eventually move to better eventually. Garage is high on my list of alternatives.
Anyone have any suggestions?
Seaweedfs is more mature and has many interfaces (S3, webdav, SFTP, REST, fuse mount). It's most appropriate for storing lots of small files.
I prefer the command line interface and data/synchronization model of Garage, though. It's easier to manage, probably because the developers aren't biting off more than they can chew.
Like in the old MinIO days, an S3 object is a file on the filesystem, not some replicated blocks. You could always rebuild the full object store content with a few rsync. I appreciate the simplicity.
My main concern was that you couldn't configure it easily through files, you had to use CLI, which wasn't very convenient. I hope this has changed.
Configuration is still through the CLI, though it's fairly simple. If your usecase is similar to the way that the Deuxfleurs organization uses it -- several heterogeneous, geographically distributed nodes that are more or less set-it-and-forget-it -- then it's probably a good fit.
My use case is relatively common : I want small S3 compatible object stores that can be deployed in Kubernetes without manual intervention. The CLI part was a bit in the way last time, this could have been automated but it wasn't straightforward.
Commercial only, they will replace the agpl contributions from external people. (Or at least they will say that)
Naming the product “AIStor” is one of the most blatant forced AI branding pivots I’ve seen.
https://www.seagate.com/products/video-analytics/skyhawk-ai-...
> Gaslight community when rightfully annoyed
> Kill off primary product
> Offer same product with AI slapped on the name to enterprise customers.
Good riddance Minio, and goodbye!
Your brain is full of spiders
You've got garlic in your soul, Mr. Grinch
I wouldn't touch you with a
39 and a half foot pole!
I’ve evaluated both RustFS and Garage, and here’s the breakdown:
Release Cadence: Garage feels a bit slower, while RustFS is shipping updates almost weekly.
Licensing: Garage is on AGPLv3, but RustFS uses the Apache license (which is huge for enterprise adoption).
Stability: Garage currently has the edge in distributed environments.
With MinIO effectively bowing out of the OSS race, my money is on RustFS to take the lead.