The Scaling Pipeline documentation states:

Atomic writes: All settings except "maximum durability" currently avoid atomic writes — what this means is that if the operating system running Jenkins fails, data that is buffered for writing to disk will not be flushed, it will be lost. This is quite rare, but can happen as a result of container or virtualization operations that halt the operating system or disconnect storage. Usually this data is flushed pretty quickly to disk, so the window for data loss is brief. On Linux this flush-to-disk can be forced by running 'sync'. In some rare cases this can also result in a build that cannot be loaded.

I think this is a bit misleading. I was running with SURVIVABLE_NONATOMIC for a while. My kernels don't panic and my storage doesn't drop writes. But my JVM doesn't always go down cleanly. My Ansible deployment scripts don't POST to /exit before stopping the container. And sometimes the JVM runs out of memory and the Docker container restarts. According to the documentation, those shouldn't be a big deal: after all, my OS and storage are stable.

But JVM restarts are a big deal in SURVIVABLE_NONATOMIC mode. Yes SURVIVABLE_NONATOMIC avoids XmlFile#write which avoids fsync(2), which is great. But SURVIVABLE_NONATOMIC is not so great because rather than writing the XML file to a temporary location and moving it (atomically) on top of the destination once it has been closed, it truncates the destination and then writes the new content to it. That leaves a window of time where, if the JVM goes down, data might be lost in the destination file. "But you didn't want atomicity!" you say. Fair enough, but yet the documentation told me that all I had to do was make sure my OS and storage were stable. It didn't say anything about keeping my JVM from shutting down uncleanly, and that's a tall order. In fact, I explicitly ran sync(1) before (uncleanly) shutting down my JVM in the hope that I wouldn't have any durability problems, but I still ran into durability problems. Looking into the implementation after the fact, it all makes sense now.

What I really want is a mode for Pipeline to stop calling fsync(2) but to keep writing out temporary files and atomically moving them on top of the destination. That way I don't need to worry about my JVM crashing, but I do need to worry about my kernel panicking or my storage dropping writes. I can handle that. Yet there seems to be no such option short of the nuclear -Dhudson.util.AtomicFileWriter.DISABLE_FORCED_FLUSH=true tunable, which disables fsync(2) not just for Pipeline but for all of Jenkins. That is a little more risk than I am willing to take on as well.

What is to be done about all of this? I could just use -Dhudson.util.AtomicFileWriter.DISABLE_FORCED_FLUSH=true and live with the risk, but I think something ought to be done about Pipeline. If we are to treat the current description of SURVIVABLE_NONATOMIC as normative, then I think it should continue skipping the fsync(2) but still use temporary files and atomic moves. But then doing atomic moves with a mode called "non-atomic" makes no sense, so we would likely need a new mode with the description of the current SURVIVABLE_NONATOMIC mode (i.e., protects you from JVM crashes but not OS/storage failures). The current SURVIVABLE_NONATOMIC mode would need to have its description changed to alert users that they are subject to durability issues with unclean JVM shutdowns as well as OS and storage problems.

If there is consensus about the above, it wouldn't be that hard to implement. We could expose a new XmlFile#write(Object, boolean) in core that allows one to disable the fsync(2). Then it's just a matter of defining a new durability mode and plumbing that through to workflow-support's PipelineIOUtils. A lot of plumbing but a relatively simple change. This would allow Pipeline users to opt out of fsync(2) for Pipeline while retaining atomicity and protection from JVM restarts, while also retaining fsync(2) for non-Pipeline use cases.