As of recently (workflows worked for months) I even have part of my CI on actions that fails with [0]
2026-02-27T10:11:51.1425380Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
2026-02-27T10:11:56.2331271Z ##[error]The operation was canceled.
I had to disable the workflows.
GitHub support response has been
“ We recommend reviewing the specific job step this occurs at to identify any areas where you can lessen parallel operations and CPU/memory consumption at one time.”
That plus other various issues makes me start to think about alternatives, and it would have never occurred to me one year back.
We've jumped ship to self-hosted Jenkins. Woodpecker CI looks cool but Jenkins seemed like a safer bet for us. It's been well worth the effort and it's simplified and sped up our CI massively.
Once we got the email that they were going to charge for self-hosted runners that was the final nail in the coffin for us. They walked it back but we've lost faith entirely in the platform and vision.
CI "simplification" there gets stale fast once plugins age out of sync, upgrades turn into archaeology, and some random maintainer vanishes. Losing faith in GitHub is one thing, but paying an engineer to babysit CI infra every week is usually a worse deal than the SaaS bill.
CI is basically a glorified shell script executor with some env variable/credential injection. We don't use many of the plugins aside from the docker and the agent shell one.
I'm not sure I'm sold on this argument considering the docker image for GH Actions is made by "myoung34" and half the actions are community written, have terrible documentation and feel like abandonware too.
I don’t know what’s worse - in 2026 someone genuinely suggesting Jenkins as a viable GHA alternative, or me agreeing with that.
Jenkins has possibly the worst user experience of any piece of software I’ve had to use in the last few years. It’s slow, brittle, somehow both heavyweight and has no features, littered with security vulns due to architecture, is impossible to navigate, has absolutely no standardisation for usage.
Yep, I went through the exact same pains. We desperately wanted to move to something else and I kept steering us away from Jenkins as I'd experienced its pains at a previous role. We evaluated tons of different options and begrudgingly settled on Jenkins too.
Using nonstandard ports would break the `ssh foo.exe.dev` pattern.
This could also have been solved by requiring users to customize their SSH config (coder does this once per machine, and it applies to all workspaces), but I guess the exe.dev guys are going for a "zero-config, works anywhere" experience.
Zero-config usually means the complexity got shoved somewhere less visible.
An SSH config is fine for one box, but with a pile of ephemeral workspaces it turns into stale cruft fast and half the entries is for hosts you forgot existed.
The port issue is also boringly practical.
A lot of corp envs treat 22 as blessed and anything else as a ticket, so baking the routing into the name is ugly but I can see why they picked it, even if the protocool should have had a target name from day one.
Too bad most SSH clients don't seem to support SRV records, they would've been perfect for this:
;; Domain: mydomain.com.
;; SSH running on port 2999 at host 1.2.3.4
;; A Record
vm1928.mydomain.com. 1 IN A 1.2.3.4
;; SRV Record
_ssh._tcp.vm1928.mydomain.com. 1 IN SRV 0 0 2999 vm1928.mydomain.com.
If supported it would result in just being able to do "ssh vm1928.mydomain.com" without having to add "-p 1928"
Not needing a different port. Middleboxes sometimes block ssh on nonstandard ports. Also, to preserve the alignment between the SSH hostname and the web service hostname, as though the user was accessing a single host at a single public address. Usability is key for them.
Like, I understand the really restrictive ones that only allow web browsing. But why allow outgoing ssh to port 22 but not other ports? Especially when port 22 is arguably the least secure option. At that point let people connect to any port except for a small blacklist.
Asking back, when I limit the outgoing connections from a network, why would I account for any nonstandard port and make the ruleset unwieldy, just in case someone wanted to do something clever?
A simple ruleset would only block a couple dangerous ports and leave everything else connectable. Whitelisting outgoing destination ports is more complicated and more annoying to deal with for no benefit. The only place you should be whitelisting destination ports is when you're looking at incoming connections.
I definitely block outgoing ports on all our servers by default; Established connections, HTTP(S), DNS, NTP, plus infra-specific rules. There is really no legitimate reason to connect to anything else. The benefit is defence against exfiltration.
If you're allowing direct https out, how are you stopping exfiltration?
Maybe https is routed through a monitoring proxy, but in the situation of allowing ssh the ssh wouldn't be going though one. So I still don't see the point of restricting outgoing ports on a machine that's allowed to ssh out.
You can't, reasonably. It's just a heuristic against many exploits using non-standard ports to avoid detection by proxies or traffic inspection utilities.
You can, but you need additional components to do it, like an SSH session broker (i.e. a gateway or proxy). Some of these, like SSH Communications' PrivX suite, can record all traffic running through the proxy. It's not all that different from HTTPS security and auditing proxies.
I don't because that would be impossible. Every business has different rules. But if you (as a business) want to to use this, you will find a way to make the changes to those "middleboxes". It's not your network, it's your business's network.
Large multi-national corporations, by way of their sheer size, tend to force their vendors to bend towards their needs, not to adapt to meet their vendors' unusual networking requirements.
Of all the SSH servers in the world, what percentage are listening on a port other than 22? To answer this question, you can visit https://data-status.shodan.io/ports.html and see for yourself.
By "unusual," I literally mean "not usual/not typical." Not "never happens."
I'll explain it once again, then leave this thread:
Companies frequently put egress network policies in place that confine certain protocols like SSH and HTTP to certain ports. They do this in order to achieve compliance with regulations, to achieve security or operational certifications, or simply because they're paranoid. It's not necessarily the least restrictive means of accomplishing their goals, but that's what they do. And if they're big enough, they're going to use the size of the deal and their brand equity to persuade their vendors, who might ordinarily prefer to offer a service on a nonstandard port, to provide it on the customer's preferred port instead.
If you still don't understand, I'm sorry, but I cannot assist further.
Companies might do that. They have the right to do so. If they still want to use that service, they will find a way to use it. Be it by vendor-coercing or simpler methods.
Just because those companies exist, does not mean that their shitty practices have any imapct on real internet connections. If you as a paying ISP customer want to use a custom port or whatever, it is going to work. So you as a developer don't have any restriction (which you don't know anyway beforehand) if you are developing a solution for a problem.
"Middleboxes" is a hackernews meme that is thrown around because people here work at places who restrict stuff and they can't bother to change that situation but instead complain about it.
The fact that games exist and they use all kind of ports is proof that this is not a problem for normal networks.
ZeroFS [0] is very thankful for what it brought to Linux with the v9fs [1] subsystem which is very nice to work with (network native) compared to fuse :)
I believe that the Windows Subsystem for Linux (WSL, really a Linux subsystem on Windows) uses the Plan 9 network protocol, 9p, to expose the host Windows filesystem to the Linux virtual environment.
Respect to your work on ZeroFS, but I find it kind of off-putting for you to come in and immediately put down JuiceFS, especially with benchmark results that don't make a ton of sense, and are likely making apples-to-oranges comparisons with how JuiceFS works or mount options.
For example, it doesn't really make sense that "92% of data modification operations" would fail on JuiceFS, which makes me question a lot of the methodology in these tests.
I have very limited experiences with object storage, but my humble benchmarks with juicefs + minio/garage [1] showed very bad performance (i.e. total collapse within a few hours) when running lots of small operations (torrents).
I wouldn't be surprised if there's a lot of tuning that can be achieved, but after days of reading docs and experimenting with different settings i just assumed JuiceFS was a very bad fit for archives shared through Bittorrent. I hope to be proven wrong, but in the meantime i'm very glad zerofs was mentioned as an alternative for small files/operations. I'll try to find the time to benchmark it too.
> but I find it kind of off-putting for you to come in and immediately put down JuiceFS, especially with benchmark results that don't make a ton of sense, and are likely making apples-to-oranges comparisons with how JuiceFS works or mount options.
The benchmark suite is trivial and opensource [1].
Is performing benchmarks “putting down” these days?
If you believe that the benchmarks are unfair to juicefs for a reason or for another, please put up a PR with a better methodology or corrected numbers. I’d happily merge it.
EDIT: From your profile, it seems like you are running a VC backed competitor, would be fair to mention that…
The actual code being benchmarked is trivial and open-source, but I don't see the actual JuiceFS setup anywhere in the ZeroFS repository. This means the self-published results don't seem to be reproducible by anyone looking to externally validate the stated claims in more detail. Given the very large performance differences, I have a hard time believing it's an actual apples-to-apples production-quality setup. It seems much more likely that some simple tuning is needed to make them more comparable, in which case the takeaway may be that JuiceFS may have more fiddly configuration without well-rounded defaults, not that it's actually hundreds of times slower when properly tuned for the workload.
(That said, I'd love to be wrong and confidently discover that ZeroFS is indeed that much faster!)
Yes, I'm working in the space too. I think it's fine to do benchmarks, I don't think it's necessary to immediately post them any time a competitor comes up on HN.
I don't want to see the cloud storage sector turn as bitter as the cloud database sector.
I've previously looked through the benchmarking code, and I still have some serious concerns about the way that you're presenting things on your page.
Our product is Archil [1], and we are building our service on top of a durable, distributed SSD storage layer. As a result, we have the ability to: (a) store and use data in S3 in its native format [not a block based format like the other solutions in this thread], (b) durably commit writes to our storage layer with lower latency than products which operate as installable OSS libraries and communicate with S3 directly, and (c) handle multiple writers from different instances like NFS.
Our team spent years working on NFS+Lustre products at Amazon (EFS and FSx for Lustre), so we understand the performance problems that these storage products have traditionally had.
We've built a custom protocol that allows our users to achieve high-performance for small file operations (git -- perfect for coding agents) and highly-parallel HPC workloads (model training, inference).
Obviously, there are tons of storage products because everyone makes different tradeoffs around durability, file size optimizations, etc. We're excited to have an approach that we think can flex around these properties dynamically, while providing best-in-class performance when compared to "true" storage systems like VAST, Weka, and Pure.
> ZeroFS supports running multiple instances on the same storage backend: one read-write instance and multiple read-only instances.
Well that's a big limiting factor that needs to be at the front in any distributed filesystem comparison.
Though I'm confused, the page says things like "ZeroFS makes S3 behave like a regular block device", but in that case how do read-only instances mount it without constantly getting their state corrupted out from under them? Is that implicitly talking about the NBD access, and the other access modes have logic to handle that?
Edit: What I want to see is a ZeroFS versus s3backer comparison.
Let's remember that JuiceFS can be setup very easily to not have a single point of failure (by replicating the metadata engine), meanwhile ZeroFS seems to have exactly that.
Yea, that is a big caveat to ZeroFS. Single point of failure. It is like saying I can write a faster etcd by only having a single node. Sure, that is possible, but the hard part of distributed systems is the coordination, and coordination always makes performance worse.
I personally have went with Ceph for distributed storage. I personally have a lot more confidence in Ceph over JuiceFS and ZeroFS, but realize building and running a ceph cluster is more complex, but with that complexity you get much cheaper S3, block storage, and cephfs.
I'm not an expert in any way, but i personally benchmarked [1] juiceFS performance totalling collapsing under very small files/operations (torrenting). It's good to be skeptical, but it might just be that the bar is very low for this specific usecase (IIRC juiceFS was configured and optimized for block sizes of several MBs).
For a proper comparison, also significant to note that JuiceFS is Apache-2.0 licensed while ZeroFS is dual AGPL-3.0/commercial licensed, significantly limiting the latter's ability to be easily adopted outside of open source projects.
This clarification is helpful, thanks! The README currently implies a slightly different take, perhaps it could be made more clear that it's suitable for use unmodified in closed source products:
> The AGPL license is suitable for open source projects, while commercial licenses are available for organizations requiring different terms.
I was a bit unclear on where the AGPL's network-interaction clause draws its boundaries- so the commercial license would only be needed for closed-source modifications/forks, or if statically linking ZeroFS crate into a larger proprietary Rust program, is that roughly it?
Also worth noting (as a sibling comment pointed out) that despite these assurances the untested legal risks of AGPL-licensed code may still cause difficulties for larger, risk-averse companies. Google notably has a blanket policy [1] banning all AGPL code entirely as "the risks outweigh the benefits", so large organizations are probably another area where the commercial license comes into play.
It’s not great UX on that angle. I am currently working on coordination (through s3, not node to node communication), so that you can just spawn instances without thinking about it.
ZeroFS is a single-writer architecture and therefore has overall bandwidth limited by the box it's running on.
JuiceFS scales out horizontally as each individual client writes/reads directly to/from S3, as long as the metadata engine keeps up it has essentially unlimited bandwidth across many compute nodes.
But as the benchmark shows, it is fiddly especially for workloads with many small files and is pretty wasteful in terms of S3 operations, which for the largest workloads has meaningful cost.
I think both have their place at the moment. But the space of "advanced S3-backed filesystems" is... advancing these days.
LSMs are “for small burst workloads kept in memory”? That’s just incorrect. “Once compaction hits all bets are off” suggests a misunderstanding of what compaction is for.
“Playing with fire,” “not sure about crash consistency,” “all bets are off”
Based on what exactly? ZeroFS has well defined durability semantics, guarantees which are much stronger than local block devices. If there’s a specific correctness issue, name it.
For ZeroFS [0], I went an alternate route with NFS/9P. I am surprised that it’s not more common as this approach has various advantages [1] while being much more workable than fuse.
Interesting! The network first point makes a lot of sense, especially bc you will most likely not access your actual datastore within the process running in the sandbox and instead just call some server that handles db access, access control etc.
What I often do on ZeroFS [0] is to convert issues to discussions, it's a one click operation on github and helps reduce the noise for ill-defined issues.
2026-02-27T10:11:51.1425380Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled. 2026-02-27T10:11:56.2331271Z ##[error]The operation was canceled.
I had to disable the workflows.
GitHub support response has been
“ We recommend reviewing the specific job step this occurs at to identify any areas where you can lessen parallel operations and CPU/memory consumption at one time.”
That plus other various issues makes me start to think about alternatives, and it would have never occurred to me one year back.
[0] https://github.com/Barre/ZeroFS/actions/runs/22480743922/job...
reply