Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-59216

LinkageError in ec2 spot fleet while performing jenkins UserRequest

    • Icon: Bug Bug
    • Resolution: Not A Defect
    • Icon: Major Major

      Hi everyone, apologize first for any information leak or mis-understanding, this is my first time to report issue in community. And I have toke merely all way I can to deal with it.

       

      [Environment]

      jenkins 2.176.3 with all plugins in latest version
      java version (both jenkins master and spot fleet) 1.8.0
      – plugins –
      ec2-fleet-plugin 1.10.2
      git-client 2.8.2
      junit 1.28
      --------------

      Jenkins master on ec2 r5.large, Amazon linux
      Spot fleet on r4.large / r5.large / r5.xlarge , Amazon linux

       

      [Scenario]

      I use jenkins with spot-fleet plugin for CI

      CI all pass in spot-fleet with initial ec2 mechine

      While ec2 scale out, CI fail in new instance with "remote call on  i-xxxxxx failed"

      Below is the log from instance of spot-fleet (retrieved from ${jenkinsHome/remoting/logs})

      remoting_log.txt 

       

      [Reproduce]

      3 CI job with identical codebase run at the same time
      a ec2-spot-fleet with init 2 instance

      In my case, 1 CI job take 2 ec2 instance, so one job will take the initial two instance, and the other jobs have to run in scaled out instances.

      • Result : CI pass in one job and fail in other two

      [More information]

      1. Below is the jenkinsfile with sensitive information replaced

      Jenkinsfile

      2. I face the problem in two scenario, because docker build and testing in different spot fleet

      scenario A:  

      building spot-fleet rotate ec2 every day, new ec2 instance fail to build 

      scenario B:

      is the reproduce experiment, happen in spot-fleet used in testing stage
      new ec2 (scaled out) fail in CI testing while initial instances success
      you can see pipeline with blueOcean in ec2_N_success/fail.png attachment

       

      3. Slave log viewed in jenkins
      ErrorInScaledOutInstance.pdf

        1. ErrorInScaledOutInstance.pdf
          1.13 MB
        2. ec2_fail_build.png
          ec2_fail_build.png
          105 kB
        3. Jenkinsfile
          14 kB
        4. Jenkinsfile.txt
          14 kB
        5. ec2_3_fail.png
          ec2_3_fail.png
          872 kB
        6. ec2_2_fail.png
          ec2_2_fail.png
          517 kB
        7. ec2_1_success.png
          ec2_1_success.png
          156 kB
        8. remoting_log.txt
          35 kB

          [JENKINS-59216] LinkageError in ec2 spot fleet while performing jenkins UserRequest

          TzuChin Hsu created issue -
          TzuChin Hsu made changes -
          Description Original: Hi everyone, apologize first for any information leak or mis-understanding, this is my first time to report issue in community. And I have toke merely all way I can to deal with it.


          [Scenario]

          I use jenkins with spot-fleet plugin for CI

          CI all pass in spot-fleet with initial ec2 mechine

          While ec2 scale out, CI fail in new instance with "*remote call on  i-xxxxxx failed*"

          Below is the log from instance of spot-fleet (retrieved from ${jenkinsHome/remoting/logs})

          [^remoting_log.txt] 

           

          [Reproduce]

          3 CI job with identical codebase run at the same time
          a ec2-spot-fleet with init 2 instance

          In my case, 1 CI job take 2 ec2 instance, so one job will take the initial two instance, and the other jobs have to run in scaled out instances.
           * Result : CI pass in one job and fail in other two

          !ec2_1_success.png!

          [More information]

          1. Below is the jenkinsfile with sensitive information replaced

          [^Jenkinsfile]

          2. I face the problem in two scenario, because docker build and testing in different spot fleet

          scenario A:  

          building spot-fleet rotate ec2 every day, new ec2 instance fail to build 

          !ec2_fail_build.png!

          scenario B:

          is the reproduce experiment, happen in spot-fleet used in testing stage
          new ec2 (scaled out) fail in CI testing while initial instances success
          you can see pipeline with blueOcean in ec2_N_success/fail.png attachment

           

          3. Slave log viewed in jenkins
          [^ErrorInScaledOutInstance.pdf]
          New: Hi everyone, apologize first for any information leak or mis-understanding, this is my first time to report issue in community. And I have toke merely all way I can to deal with it.

           

          [Environment]

          jenkins 2.176.3 with all plugins in latest version
          java version (both jenkins master and spot fleet) 1.8.0
          -- plugins --
          ec2-fleet-plugin 1.10.2
          git-client 2.8.2
          junit 1.28
          --------------

          Jenkins master on ec2 r5.large, Amazon linux
          Spot fleet on r4.large / r5.large / r5.xlarge , Amazon linux

           

          [Scenario]

          I use jenkins with spot-fleet plugin for CI

          CI all pass in spot-fleet with initial ec2 mechine

          While ec2 scale out, CI fail in new instance with "*remote call on  i-xxxxxx failed*"

          Below is the log from instance of spot-fleet (retrieved from ${jenkinsHome/remoting/logs})

          [^remoting_log.txt] 

           

          [Reproduce]

          3 CI job with identical codebase run at the same time
           a ec2-spot-fleet with init 2 instance

          In my case, 1 CI job take 2 ec2 instance, so one job will take the initial two instance, and the other jobs have to run in scaled out instances.
           * Result : CI pass in one job and fail in other two

          !ec2_1_success.png!

          [More information]

          1. Below is the jenkinsfile with sensitive information replaced

          [^Jenkinsfile]

          2. I face the problem in two scenario, because docker build and testing in different spot fleet

          scenario A:  

          building spot-fleet rotate ec2 every day, new ec2 instance fail to build 

          !ec2_fail_build.png!

          scenario B:

          is the reproduce experiment, happen in spot-fleet used in testing stage
           new ec2 (scaled out) fail in CI testing while initial instances success
           you can see pipeline with blueOcean in ec2_N_success/fail.png attachment

           

          3. Slave log viewed in jenkins
           [^ErrorInScaledOutInstance.pdf]
          TzuChin Hsu made changes -
          Environment Original: jenkins 2.176.3 with all plugins in latest version
          java version (both jenkins master and spot fleet) 1.8.0
          -- plugins --
          ec2-fleet-plugin 1.10.2
          git-client 2.8.2
          junit 1.28
          --------------

          Jenkins master on ec2 r5.large, Amazon linux
          Spot fleet on r4.large / r5.large / r5.xlarge , Amazon linux
          New: jenkins 2.176.3 with all plugins in latest version

          java version (both jenkins master and spot fleet) 1.8.0
          -- plugins --
          ec2-fleet-plugin 1.10.2
          git-client 2.8.2
          junit 1.28
          --------------

          Jenkins master on ec2 r5.large, Amazon linux
          Spot fleet on r4.large / r5.large / r5.xlarge , Amazon linux
          TzuChin Hsu made changes -
          Environment Original: jenkins 2.176.3 with all plugins in latest version

          java version (both jenkins master and spot fleet) 1.8.0
          -- plugins --
          ec2-fleet-plugin 1.10.2
          git-client 2.8.2
          junit 1.28
          --------------

          Jenkins master on ec2 r5.large, Amazon linux
          Spot fleet on r4.large / r5.large / r5.xlarge , Amazon linux
          New: jenkins 2.176.3 with all plugins in latest version\n

          java version (both jenkins master and spot fleet) 1.8.0
          -- plugins --
          ec2-fleet-plugin 1.10.2
          git-client 2.8.2
          junit 1.28
          --------------

          Jenkins master on ec2 r5.large, Amazon linux
          Spot fleet on r4.large / r5.large / r5.xlarge , Amazon linux
          TzuChin Hsu made changes -
          Environment Original: jenkins 2.176.3 with all plugins in latest version\n

          java version (both jenkins master and spot fleet) 1.8.0
          -- plugins --
          ec2-fleet-plugin 1.10.2
          git-client 2.8.2
          junit 1.28
          --------------

          Jenkins master on ec2 r5.large, Amazon linux
          Spot fleet on r4.large / r5.large / r5.xlarge , Amazon linux
          New: jenkins 2.176.3 with all plugins in latest version
          java version (both jenkins master and spot fleet) 1.8.0

          -- plugins --
          ec2-fleet-plugin 1.10.2
          git-client 2.8.2
          junit 1.28
          --------------

          Jenkins master on ec2 r5.large, Amazon linux
          Spot fleet on r4.large / r5.large / r5.xlarge , Amazon linux
          Mark Waite made changes -
          Component/s Original: git-client-plugin [ 17423 ]
          Artem Stasiuk made changes -
          Assignee Original: Chad Schmutzer [ schmutze ] New: Artem Stasiuk [ terma ]
          Resolution New: Not A Defect [ 7 ]
          Status Original: Open [ 1 ] New: Resolved [ 5 ]
          Artem Stasiuk made changes -
          Status Original: Resolved [ 5 ] New: Closed [ 6 ]

            terma Artem Stasiuk
            holis TzuChin Hsu
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: