• Icon: Bug Bug
    • Resolution: Not A Defect
    • Icon: Major Major
    • ec2-plugin
    • None

      Hello,

      I recently moved from the LTS docker image to 2.362-alpine. I had some issues with a plugin being compiled with Java 55, but I don't believe that is related. Once I updated to the new image, it appears the EC2 plugin is now terminating build agents right after they get spawned. Essentially, it spins up an agent, connects, then shuts id down and tries a new one. Rinse and repeat. Nothing on the inf side has changed, and I am able to spin up AMIs perfectly fine by hand. Logs below:

       
       
       
          2022-08-04T05:43:23.616+02:00    2022-08-04 03:43:23.615+0000 [id=328] INFO o.j.p.g.ApiRateLimitChecker$RateLimitCheckerAdapter#checkRateLimit: LocalChecker for rate limit was not set for this thread. Configured using system settings with API URL 'https://api.github.com'.
          2022-08-04T05:43:26.332+02:00    2022-08-04 03:43:26.331+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Connecting to 10.1.49.48 on port 22, with timeout 10000.
          2022-08-04T05:43:26.333+02:00    2022-08-04 03:43:26.332+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Failed to connect via ssh: There was a problem while connecting to 10.1.49.48:22
          2022-08-04T05:43:26.333+02:00    2022-08-04 03:43:26.333+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Waiting for SSH to come up. Sleeping 5.
          2022-08-04T05:43:31.422+02:00    2022-08-04 03:43:31.421+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Connecting to 10.1.49.48 on port 22, with timeout 10000.
          2022-08-04T05:43:31.424+02:00    2022-08-04 03:43:31.423+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Failed to connect via ssh: There was a problem while connecting to 10.1.49.48:22
          2022-08-04T05:43:31.424+02:00    2022-08-04 03:43:31.424+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Waiting for SSH to come up. Sleeping 5.
          2022-08-04T05:43:36.476+02:00    2022-08-04 03:43:36.476+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Connecting to 10.1.49.48 on port 22, with timeout 10000.
          2022-08-04T05:43:36.569+02:00    2022-08-04 03:43:36.569+0000 [id=335] INFO hudson.plugins.ec2.EC2Cloud#log: The SSH key ssh-ed25519 41:18:57:45:15:e5:1d:9b:d5:79:60:30:17:f2:c0:e5 has been automatically trusted for connections to EC2 (EC2 agents) - integration agents (i-0290a7c9b9d5a0c85)
          2022-08-04T05:43:36.581+02:00    2022-08-04 03:43:36.581+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Connected via SSH.
          2022-08-04T05:43:36.647+02:00    2022-08-04 03:43:36.646+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: connect fresh as root
          2022-08-04T05:43:36.725+02:00    2022-08-04 03:43:36.725+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Connecting to 10.1.49.48 on port 22, with timeout 10000.
          2022-08-04T05:43:36.849+02:00    2022-08-04 03:43:36.849+0000 [id=336] INFO hudson.plugins.ec2.EC2Cloud#log: Connection allowed after the host key has been verified
          2022-08-04T05:43:36.865+02:00    2022-08-04 03:43:36.865+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Connected via SSH.
          2022-08-04T05:43:36.953+02:00    2022-08-04 03:43:36.952+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Creating tmp directory (/tmp) if it does not exist
          2022-08-04T05:43:38.918+02:00    2022-08-04 03:43:38.917+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Verifying: java -fullversion
          2022-08-04T05:43:39.281+02:00    2022-08-04 03:43:39.281+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Verifying: which scp
          2022-08-04T05:43:39.286+02:00    2022-08-04 03:43:39.285+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Copying remoting.jar to: /tmp
          2022-08-04T05:43:39.373+02:00    2022-08-04 03:43:39.373+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Launching remoting agent (via Trilead SSH2 Connection): java -jar /tmp/remoting.jar -workDir /var/lib/jenkins
          2022-08-04T05:43:40.378+02:00    2022-08-04 03:43:40.378+0000 [id=177] INFO h.plugins.ec2.EC2OndemandSlave#lambda$terminate$0: Terminated EC2 instance (terminated): i-0290a7c9b9d5a0c85
          2022-08-04T05:43:40.424+02:00    2022-08-04 03:43:40.424+0000 [id=177] INFO h.plugins.ec2.EC2OndemandSlave#lambda$terminate$0: Removed EC2 instance from jenkins controller: i-0290a7c9b9d5a0c85

          [JENKINS-69236] EC2 Plugin Terminates Agent Immediately

          I rolled back to the LTS alpine image and restored from a previous update. Everything is back to normal. I will probably try the upgrade again in a few weeks to step back through it all and see where the issue arises.

          Cameron Stevenson added a comment - I rolled back to the LTS alpine image and restored from a previous update. Everything is back to normal. I will probably try the upgrade again in a few weeks to step back through it all and see where the issue arises.

          I switched my Jenkins image to the jdk-11 version and stepped it up to 2.346.3-2-lts-jdk11. I then tried the 3.57 container so I could update all the plugins via plugin manager as recommended before proceeding past that version. The same issue came back.

          I am not sure if the issue is from the 3.57 version or because I switched to the non-lts image. Problem is that there are no errors when installing the plugins via cli nor when upgrading them manually via the plugin manager.

          For now I will keep using the lts-jdk-11 containers and see if the issue every comes back when upgrading versions.

          Cameron Stevenson added a comment - I switched my Jenkins image to the jdk-11 version and stepped it up to 2.346.3-2-lts-jdk11. I then tried the 3.57 container so I could update all the plugins via plugin manager as recommended before proceeding past that version. The same issue came back. I am not sure if the issue is from the 3.57 version or because I switched to the non-lts image. Problem is that there are no errors when installing the plugins via cli nor when upgrading them manually via the plugin manager. For now I will keep using the lts-jdk-11 containers and see if the issue every comes back when upgrading versions.

          Bart Peeters added a comment -

          I had the same issue.

          My Jenkins slave AMI had Java 8 installed, but when using Jenkins version > v2.3, it needs at least Java 11.

          So if your agent also has Java 8, it is probably the same issue.

          I created a new AMI which has only Java 17 installed and now it works as expected.

          Bart Peeters added a comment - I had the same issue. My Jenkins slave AMI had Java 8 installed, but when using Jenkins version > v2.3, it needs at least Java 11. So if your agent also has Java 8, it is probably the same issue. I created a new AMI which has only Java 17 installed and now it works as expected.

          Oh birtpeeters that's good thinking. I will have to run some tests!

          Cameron Stevenson added a comment - Oh birtpeeters that's good thinking. I will have to run some tests!

          Basil Crow added a comment -

          Note that as of JENKINS-69304 you can specify a custom Java binary in the EC2 plugin. This allows you to, for example, run the Jenkins Remoting process with Java 11 while continuing to use Java 8 for your application builds.

          Basil Crow added a comment - Note that as of JENKINS-69304 you can specify a custom Java binary in the EC2 plugin. This allows you to, for example, run the Jenkins Remoting process with Java 11 while continuing to use Java 8 for your application builds.

          basil thank you for the for that. I will look into it!

          Cameron Stevenson added a comment - basil thank you for the for that. I will look into it!

            thoulen FABRIZIO MANFREDI
            ckstevenson Cameron Stevenson
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: