Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-52808

Jenkins worker is made available to jobs before startup script finishes running

XMLWordPrintable

      According to the wiki (https://wiki.jenkins.io/display/JENKINS/Google+Compute+Engine+Plugin) the worker VM will have the "Startup Script" commands run on it before the worker is made available to jobs

      Startup Script - defines a set of commands that should be run before making the instance available for running your jobs.

       

      In practice it seems the "Startup Script" commands are still running on the VM when a job starts running on it.

      I verified this by having some of the in the "Startup Script" section echo timestamps and logging to a file. The file was still being updated for a couple of minutes after the job started running on the VM.

       

      In my case the "Startup script" is seeding the worker with some caches and I would need for the cache setup to be done before the worker is made available to Jobs.

       

       

          [JENKINS-52808] Jenkins worker is made available to jobs before startup script finishes running

          Rachel Yen added a comment -

          Hi Arash,

          You are correct in practice it is entirely possible that the worker can still be running your startup script while your jenkins build is happening. 

          We expect most consumers would want to run most of their code in the actual build, but based on your use case, perhaps we should implement a long-term solution.

          In the meantime, here are some possible quick/temporary workarounds.

          1. If you look at https://github.com/jenkinsci/google-compute-engine-plugin/blob/master/src/test/java/com/google/jenkins/plugins/computeengine/ComputeEngineCloudIT.java, the string DEB_JAVA_STARTUP_SCRIPT includes two significant lines: /etc/init.d/ssh stop and /etc/init.d/ssh start. It's a way to not let your workers start their build while the startup script is running because your server cannot SSH to your workers.
          2. As mentioned, if possible and pertinent to your use case, move the caching portion to the build job.

           

          Let us know if you have any more questions.

           

          Rachel Yen added a comment - Hi Arash, You are correct in practice it is entirely possible that the worker can still be running your startup script while your jenkins build is happening.  We expect most consumers would want to run most of their code in the actual build, but based on your use case, perhaps we should implement a long-term solution. In the meantime, here are some possible quick/temporary workarounds. If you look at https://github.com/jenkinsci/google-compute-engine-plugin/blob/master/src/test/java/com/google/jenkins/plugins/computeengine/ComputeEngineCloudIT.java,  the string DEB_JAVA_STARTUP_SCRIPT includes two significant lines: /etc/init.d/ssh stop and /etc/init.d/ssh start. It's a way to not let your workers start their build while the startup script is running because your server cannot SSH to your workers. As mentioned, if possible and pertinent to your use case, move the caching portion to the build job.   Let us know if you have any more questions.  

          Rachel Yen added a comment -

          I'm going to close this for now. If there are any more issues, please let us know.

          Rachel Yen added a comment - I'm going to close this for now. If there are any more issues, please let us know.

          Rachel Yen added a comment -

          See comments for solution.

          Rachel Yen added a comment - See comments for solution.

            zombiemoose Rachel Yen
            arash arash m
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: