Alexander Komarov added a comment - 2019-11-18 20:08 - edited Thanks pjdarton for the reminder.  I also added rudimentary configuration for sleep/etc via environment variables.

Gregory PICOT added a comment - 2020-01-15 08:16 - edited Hi, Thank akom for the retry bit, we tried it on our end to secure the jnlp connexion. The retry in itself seems to work great, but since we implemented it, our containers doesn't end properly: The kill -15 sent on job termination is not correctly interpreted by the container, and we have to wait 10 sec for a kill -9 to really put an end to the container. This is problematic because we have a small window where the master believe the container (and the agent related) is free to use. It could then try to start a job in it, with no chance to last long. What I could figure out is that since the entrypoint is the new script, when the $ACTUAL_ENTRYPOINT run, it is not the PID 1, and the jar is not linked to our new entrypoint (since is it started by exec command). When the kill -15 occur, not every process are killed, so the container stays alive. I tried to add the exec command before runnning the $ACTUAL_ENTRYPOINT, it resolve the issue of the sigterm interpretation, but we loose the retry logic... I'm still trying to figure out a solution to torward properly sigterms and keep the retry.

Gregory PICOT added a comment - 2020-01-23 13:36 Hello,   Here's an update regarding the issue between sigterm propagation and retry : We managed to make them work together following this article: https://unix.stackexchange.com/questions/146756/forward-sigterm-to-child-in-bash Instead of using the exec command, we used wait and trap commands to meet our need. I find the Article from Andreas Veithen++  to be very interesting and detailed.   We put back the "usual" jenkins entrypoint in the dockerfile, but instead of starting the jar, it launch our script : exec $JAVA_BIN $JAVA_OPTS $JNLP_PROTOCOL_OPTS -cp /usr/share/jenkins/agent.jar hudson.remoting.jnlp.Main -headless $TUNNEL $URL $WORKDIR $DIRECT $PROTOCOLS $INSTANCE_IDENTITY $OPT_JENKINS_SECRET $OPT_JENKINS_AGENT_NAME "$@" replaced by :   exec /usr/local/bin/jenkins-agent-retry.sh "$@"     Here is what jenkins-agent-retry.sh look like now :   #!/usr/bin/env sh if [ $# -eq 1 ]; then # if `docker run` only has one arguments, we assume user is running alternate command like `bash` to inspect the image exec "$@" fi # Gestion SIGTERM https: //unix.stackexchange.com/questions/146756/forward-sigterm-to-child-in-bash prep_term() { unset term_child_pid unset term_kill_needed trap 'handle_term' TERM INT } handle_term() { if [ "${term_child_pid}" ]; then kill -TERM "${term_child_pid}" 2>/dev/ null else term_kill_needed= "yes" fi } wait_term() { term_child_pid=$! if [ "${term_kill_needed}" ]; then kill -TERM "${term_child_pid}" 2>/dev/ null fi wait ${term_child_pid} trap - TERM INT wait ${term_child_pid} } echo "[INFO] JDK $($JAVA_BIN -version 2>&1|awk '$0~/openjdk version/ {print $3}' ) to connect to master" echo "[INFO] Remoting Version: $($JAVA_BIN -cp /usr/share/jenkins/agent.jar hudson.remoting.jnlp.Main -headless -version)" echo "[INFO] Start Agent command: " $JAVA_BIN $JAVA_OPTS $JNLP_PROTOCOL_OPTS -cp /usr/share/jenkins/agent.jar hudson.remoting.jnlp.Main -headless $TUNNEL $URL $WORKDIR $DIRECT $PROTOCOLS $INSTANCE_IDENTITY $OPT_JENKINS_SECRET $OPT_JENKINS_AGENT_NAME "$@" # Gestion Retry from https: //issues.jenkins-ci.org/browse/JENKINS-59790?focusedCommentId=379913&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-379913 SLEEP=${JNLP_RETRY_SLEEP:-5} # Try to reconnect this many times TRIES=${JNLP_RETRY_COUNT:-3} # Stop retrying after this many seconds regardless MAXTIME=${JNLP_RETRY_MAXTIME:-60} START=$(date +%s) while [ $TRIES -gt 0 ] && [ $(($(date +%s) - $START)) -lt $MAXTIME ]; do prep_term $JAVA_BIN $JAVA_OPTS $JNLP_PROTOCOL_OPTS -cp /usr/share/jenkins/agent.jar hudson.remoting.jnlp.Main -headless $TUNNEL $URL $WORKDIR $DIRECT $PROTOCOLS $INSTANCE_IDENTITY $OPT_JENKINS_SECRET $OPT_JENKINS_AGENT_NAME "$@" & wait_term CODE=$? if [ $CODE -eq 143 ]; then break fi echo "exited [$CODE], waiting $SLEEP seconds and retrying" sleep $SLEEP TRIES=$(($TRIES - 1)) done  

pjdarton added a comment - 2020-08-10 16:07 Note to anyone monitoring this: There's a prospective fix here - https://github.com/jenkinsci/docker-plugin/pull/789 However, it needs to be tested. I don't have this problem in my docker environment so I can't test it; what I need is someone who can reproduce the issue to try a "without this fix" and a "with this fix" comparison and to report back (to the conversation in the PR as I don't have the time to monitor stuff in JIRA as closely as I should). If I can get a decent level of confidence that this change does actually fix the problem then I'll merge it in and push out a maintstream release with the fix.

pjdarton added a comment - 2020-10-09 14:59 Fixed by https://github.com/jenkinsci/docker-plugin/pull/789 which has (just) been merged and will be in the next release.