Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-69236

EC2 Plugin Terminates Agent Immediately

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Not A Defect
    • Icon: Major Major
    • ec2-plugin
    • None

      Hello,

      I recently moved from the LTS docker image to 2.362-alpine. I had some issues with a plugin being compiled with Java 55, but I don't believe that is related. Once I updated to the new image, it appears the EC2 plugin is now terminating build agents right after they get spawned. Essentially, it spins up an agent, connects, then shuts id down and tries a new one. Rinse and repeat. Nothing on the inf side has changed, and I am able to spin up AMIs perfectly fine by hand. Logs below:

       
       
       
          2022-08-04T05:43:23.616+02:00    2022-08-04 03:43:23.615+0000 [id=328] INFO o.j.p.g.ApiRateLimitChecker$RateLimitCheckerAdapter#checkRateLimit: LocalChecker for rate limit was not set for this thread. Configured using system settings with API URL 'https://api.github.com'.
          2022-08-04T05:43:26.332+02:00    2022-08-04 03:43:26.331+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Connecting to 10.1.49.48 on port 22, with timeout 10000.
          2022-08-04T05:43:26.333+02:00    2022-08-04 03:43:26.332+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Failed to connect via ssh: There was a problem while connecting to 10.1.49.48:22
          2022-08-04T05:43:26.333+02:00    2022-08-04 03:43:26.333+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Waiting for SSH to come up. Sleeping 5.
          2022-08-04T05:43:31.422+02:00    2022-08-04 03:43:31.421+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Connecting to 10.1.49.48 on port 22, with timeout 10000.
          2022-08-04T05:43:31.424+02:00    2022-08-04 03:43:31.423+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Failed to connect via ssh: There was a problem while connecting to 10.1.49.48:22
          2022-08-04T05:43:31.424+02:00    2022-08-04 03:43:31.424+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Waiting for SSH to come up. Sleeping 5.
          2022-08-04T05:43:36.476+02:00    2022-08-04 03:43:36.476+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Connecting to 10.1.49.48 on port 22, with timeout 10000.
          2022-08-04T05:43:36.569+02:00    2022-08-04 03:43:36.569+0000 [id=335] INFO hudson.plugins.ec2.EC2Cloud#log: The SSH key ssh-ed25519 41:18:57:45:15:e5:1d:9b:d5:79:60:30:17:f2:c0:e5 has been automatically trusted for connections to EC2 (EC2 agents) - integration agents (i-0290a7c9b9d5a0c85)
          2022-08-04T05:43:36.581+02:00    2022-08-04 03:43:36.581+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Connected via SSH.
          2022-08-04T05:43:36.647+02:00    2022-08-04 03:43:36.646+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: connect fresh as root
          2022-08-04T05:43:36.725+02:00    2022-08-04 03:43:36.725+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Connecting to 10.1.49.48 on port 22, with timeout 10000.
          2022-08-04T05:43:36.849+02:00    2022-08-04 03:43:36.849+0000 [id=336] INFO hudson.plugins.ec2.EC2Cloud#log: Connection allowed after the host key has been verified
          2022-08-04T05:43:36.865+02:00    2022-08-04 03:43:36.865+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Connected via SSH.
          2022-08-04T05:43:36.953+02:00    2022-08-04 03:43:36.952+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Creating tmp directory (/tmp) if it does not exist
          2022-08-04T05:43:38.918+02:00    2022-08-04 03:43:38.917+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Verifying: java -fullversion
          2022-08-04T05:43:39.281+02:00    2022-08-04 03:43:39.281+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Verifying: which scp
          2022-08-04T05:43:39.286+02:00    2022-08-04 03:43:39.285+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Copying remoting.jar to: /tmp
          2022-08-04T05:43:39.373+02:00    2022-08-04 03:43:39.373+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Launching remoting agent (via Trilead SSH2 Connection): java -jar /tmp/remoting.jar -workDir /var/lib/jenkins
          2022-08-04T05:43:40.378+02:00    2022-08-04 03:43:40.378+0000 [id=177] INFO h.plugins.ec2.EC2OndemandSlave#lambda$terminate$0: Terminated EC2 instance (terminated): i-0290a7c9b9d5a0c85
          2022-08-04T05:43:40.424+02:00    2022-08-04 03:43:40.424+0000 [id=177] INFO h.plugins.ec2.EC2OndemandSlave#lambda$terminate$0: Removed EC2 instance from jenkins controller: i-0290a7c9b9d5a0c85

          [JENKINS-69236] EC2 Plugin Terminates Agent Immediately

          Cameron Stevenson created issue -
          Cameron Stevenson made changes -
          Description Original: Hello,

          I recently moved from the LTS docker image to 2.362-alpine. I had some issues with a plugin being compiled with Java 55, but I don't believe that is related. Once I updated to the new image, it appears the EC2 plugin is now terminating build agents right after they get spawned. Essentially, it spins up an agent, connects, then shuts id down and tries a new one. Rinse and repeat. Nothing on the inf side has changed, and I am able to spin up AMIs perfectly fine by hand. Logs below:

           
          | |2022-08-04T05:43:23.616+02:00|2022-08-04 03:43:23.615+0000 [id=328] INFO o.j.p.g.ApiRateLimitChecker$RateLimitCheckerAdapter#checkRateLimit: LocalChecker for rate limit was not set for this thread. Configured using system settings with API URL 'https://api.github.com'.|
          | |

          | |2022-08-04T05:43:26.332+02:00|2022-08-04 03:43:26.331+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Connecting to 10.1.49.48 on port 22, with timeout 10000.|
          | |

          | |2022-08-04T05:43:26.333+02:00|2022-08-04 03:43:26.332+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Failed to connect via ssh: There was a problem while connecting to 10.1.49.48:22|
          | |

          | |2022-08-04T05:43:26.333+02:00|2022-08-04 03:43:26.333+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Waiting for SSH to come up. Sleeping 5.|
          | |

          | |2022-08-04T05:43:31.422+02:00|2022-08-04 03:43:31.421+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Connecting to 10.1.49.48 on port 22, with timeout 10000.|
          | |

          | |2022-08-04T05:43:31.424+02:00|2022-08-04 03:43:31.423+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Failed to connect via ssh: There was a problem while connecting to 10.1.49.48:22|
          | |

          | |2022-08-04T05:43:31.424+02:00|2022-08-04 03:43:31.424+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Waiting for SSH to come up. Sleeping 5.|
          | |

          | |2022-08-04T05:43:36.476+02:00|2022-08-04 03:43:36.476+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Connecting to 10.1.49.48 on port 22, with timeout 10000.|
          | |

          | |2022-08-04T05:43:36.569+02:00|2022-08-04 03:43:36.569+0000 [id=335] INFO hudson.plugins.ec2.EC2Cloud#log: The SSH key ssh-ed25519 41:18:57:45:15:e5:1d:9b:d5:79:60:30:17:f2:c0:e5 has been automatically trusted for connections to EC2 (EC2 agents) - integration agents (i-0290a7c9b9d5a0c85)|
          | |

          | |2022-08-04T05:43:36.581+02:00|2022-08-04 03:43:36.581+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Connected via SSH.|
          | |

          | |2022-08-04T05:43:36.647+02:00|2022-08-04 03:43:36.646+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: connect fresh as root|
          | |

          | |2022-08-04T05:43:36.725+02:00|2022-08-04 03:43:36.725+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Connecting to 10.1.49.48 on port 22, with timeout 10000.|
          | |

          | |2022-08-04T05:43:36.849+02:00|2022-08-04 03:43:36.849+0000 [id=336] INFO hudson.plugins.ec2.EC2Cloud#log: Connection allowed after the host key has been verified|
          | |

          | |2022-08-04T05:43:36.865+02:00|2022-08-04 03:43:36.865+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Connected via SSH.|
          | |

          | |2022-08-04T05:43:36.953+02:00|2022-08-04 03:43:36.952+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Creating tmp directory (/tmp) if it does not exist|
          | |

          | |2022-08-04T05:43:38.918+02:00|2022-08-04 03:43:38.917+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Verifying: java -fullversion|
          | |

          | |2022-08-04T05:43:39.281+02:00|2022-08-04 03:43:39.281+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Verifying: which scp|
          | |

          | |2022-08-04T05:43:39.286+02:00|2022-08-04 03:43:39.285+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Copying remoting.jar to: /tmp|
          | |

          | |2022-08-04T05:43:39.373+02:00|2022-08-04 03:43:39.373+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Launching remoting agent (via Trilead SSH2 Connection): java -jar /tmp/remoting.jar -workDir /var/lib/jenkins|
          | |

          | |2022-08-04T05:43:40.378+02:00|2022-08-04 03:43:40.378+0000 [id=177] INFO h.plugins.ec2.EC2OndemandSlave#lambda$terminate$0: Terminated EC2 instance (terminated): i-0290a7c9b9d5a0c85|
          | |

          | |2022-08-04T05:43:40.424+02:00|2022-08-04 03:43:40.424+0000 [id=177] INFO h.plugins.ec2.EC2OndemandSlave#lambda$terminate$0: Removed EC2 instance from jenkins controller: i-0290a7c9b9d5a0c85|           		New: Hello,

          I recently moved from the LTS docker image to 2.362-alpine. I had some issues with a plugin being compiled with Java 55, but I don't believe that is related. Once I updated to the new image, it appears the EC2 plugin is now terminating build agents right after they get spawned. Essentially, it spins up an agent, connects, then shuts id down and tries a new one. Rinse and repeat. Nothing on the inf side has changed, and I am able to spin up AMIs perfectly fine by hand. Logs below:

           
           
           
              2022-08-04T05:43:23.616+02:00    2022-08-04 03:43:23.615+0000 [id=328] INFO o.j.p.g.ApiRateLimitChecker$RateLimitCheckerAdapter#checkRateLimit: LocalChecker for rate limit was not set for this thread. Configured using system settings with API URL 'https://api.github.com'.
              2022-08-04T05:43:26.332+02:00    2022-08-04 03:43:26.331+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Connecting to 10.1.49.48 on port 22, with timeout 10000.
              2022-08-04T05:43:26.333+02:00    2022-08-04 03:43:26.332+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Failed to connect via ssh: There was a problem while connecting to 10.1.49.48:22
              2022-08-04T05:43:26.333+02:00    2022-08-04 03:43:26.333+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Waiting for SSH to come up. Sleeping 5.
              2022-08-04T05:43:31.422+02:00    2022-08-04 03:43:31.421+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Connecting to 10.1.49.48 on port 22, with timeout 10000.
              2022-08-04T05:43:31.424+02:00    2022-08-04 03:43:31.423+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Failed to connect via ssh: There was a problem while connecting to 10.1.49.48:22
              2022-08-04T05:43:31.424+02:00    2022-08-04 03:43:31.424+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Waiting for SSH to come up. Sleeping 5.
              2022-08-04T05:43:36.476+02:00    2022-08-04 03:43:36.476+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Connecting to 10.1.49.48 on port 22, with timeout 10000.
              2022-08-04T05:43:36.569+02:00    2022-08-04 03:43:36.569+0000 [id=335] INFO hudson.plugins.ec2.EC2Cloud#log: The SSH key ssh-ed25519 41:18:57:45:15:e5:1d:9b:d5:79:60:30:17:f2:c0:e5 has been automatically trusted for connections to EC2 (EC2 agents) - integration agents (i-0290a7c9b9d5a0c85)
              2022-08-04T05:43:36.581+02:00    2022-08-04 03:43:36.581+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Connected via SSH.
              2022-08-04T05:43:36.647+02:00    2022-08-04 03:43:36.646+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: connect fresh as root
              2022-08-04T05:43:36.725+02:00    2022-08-04 03:43:36.725+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Connecting to 10.1.49.48 on port 22, with timeout 10000.
              2022-08-04T05:43:36.849+02:00    2022-08-04 03:43:36.849+0000 [id=336] INFO hudson.plugins.ec2.EC2Cloud#log: Connection allowed after the host key has been verified
              2022-08-04T05:43:36.865+02:00    2022-08-04 03:43:36.865+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Connected via SSH.
              2022-08-04T05:43:36.953+02:00    2022-08-04 03:43:36.952+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Creating tmp directory (/tmp) if it does not exist
              2022-08-04T05:43:38.918+02:00    2022-08-04 03:43:38.917+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Verifying: java -fullversion
              2022-08-04T05:43:39.281+02:00    2022-08-04 03:43:39.281+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Verifying: which scp
              2022-08-04T05:43:39.286+02:00    2022-08-04 03:43:39.285+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Copying remoting.jar to: /tmp
              2022-08-04T05:43:39.373+02:00    2022-08-04 03:43:39.373+0000 [id=180] INFO hudson.plugins.ec2.EC2Cloud#log: Launching remoting agent (via Trilead SSH2 Connection): java -jar /tmp/remoting.jar -workDir /var/lib/jenkins
              2022-08-04T05:43:40.378+02:00    2022-08-04 03:43:40.378+0000 [id=177] INFO h.plugins.ec2.EC2OndemandSlave#lambda$terminate$0: Terminated EC2 instance (terminated): i-0290a7c9b9d5a0c85
              2022-08-04T05:43:40.424+02:00    2022-08-04 03:43:40.424+0000 [id=177] INFO h.plugins.ec2.EC2OndemandSlave#lambda$terminate$0: Removed EC2 instance from jenkins controller: i-0290a7c9b9d5a0c85
          Basil Crow made changes -
          Resolution New: Not A Defect [ 7 ]
          Status Original: Open [ 1 ] New: Resolved [ 5 ]
          Basil Crow made changes -
          Status Original: Resolved [ 5 ] New: Closed [ 6 ]

            thoulen FABRIZIO MANFREDI
            ckstevenson Cameron Stevenson
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: