cc jthompson oleg_nenashev not sure what info you need here.

I really don't have much to suggest. I'm not terribly familiar with the ProcessTree class. This particular part of the code was last changed 12 years ago.

My first guess with things like this is always that something changed with the environment, the system, network or something. It could be something changed with Big Sur, but I don't know that we can really be confident with that yet.

It looks like it could be a memory allocation or consumption issue, but I'm not familiar enough with the ProcessTree code to know. Maybe memory is used differently on Big Sur than before – there are certainly instances of such behavior on significant OS upgrades.

With the inconsistent nature of this problem, it's going to require good troubleshooting to isolate the problem. I don't have any specific troubleshooting ideas, though. If there's any way to make it reproducible that would help.

I will keep an eye on it and see if I can find a pattern – maybe it's concurrent builds or some other thing I haven't noticed yet...

Anyway, if anybody has some ideas for troubleshooting, let me know and I will try!

My understanding of the code from a not too deep read is that, a combination of process arguments and environment variables exceed a megabyte in size.

I'm not sure if this behaviour is normal or if my understanding is correct but perhaps this will narrow our search a little.

a combination of process arguments and environment variables exceed a megabyte in size

Wouldn't that mean that it would be easy to reproduce? I mean, when a build fails with this error, I can just do a rebuild and most of the time it then finishes correctly – with the same arguments and environment. Or am I missing something here?

Update on the issue itself: I rebooted the machine shortly after my last comment and all was quiet for about 4 days. Now it has started again, unfortunately with the same random behavior... Last night even my smallest job on that machine (runs every night, no parameters, just a simple 'cocoapods update') failed with this error. This is an old-school matrix-build, but I don't think that makes a real difference.

Any suggestions as to how I might figure out the size of the arguments and environment on the jobs/runs?

AFAICT you can't change KERN_ARGMAX.

I agree it should be easy to reproduce if that was in fact the case. It could be a more subtle bug.

This has been annoying us more and more, so I went Googling and found another old issue: JENKINS-9634

I also found this: https://stackoverflow.com/questions/6147815/jenkins-throwing-indexoutofboundsexception-at-end-of-build-on-mac, which suggests disabling the ProcessTreeKiller (which apparently has been renamed since – read that somewhere else, but don't have a link atm). Does anybody know if it would be possible to disable this on only the node running Mac or do I need to do it for the whole instance of Jenkins?

Since I am not sure if disabling the PTK has any (negative) effects, I don't really want to do it on the whole instance...

It might be worth trying the latest weekly release, we’ve updated JNA to the latest version recently

I'll give that a try next week – Do I just need the core or also some plugins?

A number of plugins need updating too

I added this when launch jenkins, now only seeing this issue for tasks running on nodes.
   -Dhudson.util.ProcessTreeKiller.disable=true

• could reproduce it with a dump task, say, just echo a string in the build script.  even the node is on the same machine (the server)
• and this happens for both jenkins 1 (lte) and jenkins2

-David

The

-Dhudson.util.ProcessTreeKiller.disable=true

 should be an option of the launch agent on the node side

No errors since one week since option added to the macOS "LaunchAgent" command in a shell script :

java -Dhudson.util.ProcessTreeKiller.disable=true -jar agent.jar -jnlpUrl https://REDACTED/slave-agent.jnlp -secret REDACTED -noCertificateCheck

thanks kenji,  it works when launch agent with that option
(actually need to have it for both, launching server and each node...)

kenji How to add?

simply download it from https://your-jenkins.url/jnlpJars/agent.jar
 

kenji
I tried downloading the agent.jar file from https://localhost:8080/jnlpJars/agent.jar
1. My mac tells me that this file can't be trusted and doesn't let me run this file. Is there any other way to run this file? I tried running it by double clicking it.
2. Where exactly do you add this command on the launch agent?
java -Dhudson.util.ProcessTreeKiller.disable=true -jar agent.jar -jnlpUrl https://REDACTED/slave-agent.jnlp -secret REDACTED -noCertificateCheck
Is there a certain file I should add this to? What is this REDACTED part of the command? Running it in the terminal throws unknown exception REDACTED.

We only have one master agent - no other nodes/slave agent.

You can follow this blog post explaining how to configure a new node in jenkins and how to run a macOS launch agent : https://mgrebenets.github.io/mobile%20ci/2015/02/01/jenkins-remote-node

I prefer running some shell script to have a cleaner launchd plist file.

The REDACTED following -secret is redacted to keep this secret token... secret.

kenji
Is there a way to solve the memory-bound error on the main server (that has the master) without using/setting up nodes? We would prefer to use this machine for all of our builds and not use any Jenkins slaves.

This should be fixed in recent weekly versions and not need these work arounds. It will be in LTS next Wednesday