Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-32588

Unable to launch linux slaves using ec2 plugin using Eucalyptus

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Critical
    • Resolution: Fixed
    • ec2-plugin
    • None
    • Jenkins ver. 1.609.1
      Ec2 plugin 1.30
      Amazon Web Services SDK 1.10.26
      Jackson 2 API Plugin 2.5.4

    Description

      Hi,

      I have followed the guidelines for setting up linux slave.

      https://wiki.jenkins-ci.org/display/JENKINS/Amazon+EC2+Plugin

      I have configured all the details in Cloud plugin section and found success in both "Test Connection" and "Check AMI". But when triggered Jenkin job with the "Label",it is always in "(pending—Waiting for next available executor)" . In jenkins log i would see there is error for every three minutes and not able to launch slave

      "
      WARNING: Exception during provisioning
      com.amazonaws.AmazonServiceException: com.eucalyptus.ws.WebServicesException: Missing required object (Service: AmazonEC2; Status Code: 500; Error Code: InternalFailure; Request ID: d0f8e09c-323c-421b-8468-26851a458ad8)

      "

      Attachments

        1. jenkins_1.31.log
          7 kB
        2. jenkins_1.32.log
          7 kB
        3. jenkins.log
          66 kB

        Activity

          Code changed in jenkins
          User: Francis Upton IV
          Path:
          src/main/java/hudson/plugins/ec2/EC2Cloud.java
          http://jenkins-ci.org/commit/ec2-plugin/c29c5f0a76adb4ae569fe73ef44c6ba2147e17ca
          Log:
          JENKINS-32588 Unable to launch linux slaves using ec2 plugin using Eucalyptus

          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Francis Upton IV Path: src/main/java/hudson/plugins/ec2/EC2Cloud.java http://jenkins-ci.org/commit/ec2-plugin/c29c5f0a76adb4ae569fe73ef44c6ba2147e17ca Log: JENKINS-32588 Unable to launch linux slaves using ec2 plugin using Eucalyptus
          francisu Francis Upton added a comment - Can you try this HPI and see if this resolves your problem? https://jenkins.ci.cloudbees.com/job/plugins/job/ec2-plugin/373/org.jenkins-ci.plugins$ec2/artifact/org.jenkins-ci.plugins/ec2/1.31-SNAPSHOT/ec2-1.31-SNAPSHOT.hpi
          svmkphani Murali Phani added a comment - - edited

          Thanks.I have updated new plugin .but below isssue is there in log

          --clip-
          IWARNING: Exception during provisioning
          com.amazonaws.AmazonServiceException: com.eucalyptus.ws.WebServicesException: Missing required object (Service: AmazonEC2; Status Code: 500; Error Code: InternalFailure; Request ID: 6d8b0b9b-dc77-4695-9ada-55383235fe0c)
          at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1181)
          at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:766)
          at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:485)
          at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:306)
          at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:11901)
          at com.amazonaws.services.ec2.AmazonEC2Client.describeSpotInstanceRequests(AmazonEC2Client.java:6437)

          --clip-

          svmkphani Murali Phani added a comment - - edited Thanks.I have updated new plugin .but below isssue is there in log -- clip - IWARNING: Exception during provisioning com.amazonaws.AmazonServiceException: com.eucalyptus.ws.WebServicesException: Missing required object (Service: AmazonEC2; Status Code: 500; Error Code: InternalFailure; Request ID: 6d8b0b9b-dc77-4695-9ada-55383235fe0c) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1181) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:766) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:485) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:306) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:11901) at com.amazonaws.services.ec2.AmazonEC2Client.describeSpotInstanceRequests(AmazonEC2Client.java:6437) -- clip -
          francisu Francis Upton added a comment -

          I don't see what the problem is from this log entry. And the original problem does seem to be fixed.

          francisu Francis Upton added a comment - I don't see what the problem is from this log entry. And the original problem does seem to be fixed.
          francisu Francis Upton added a comment -

          So the same problem then? Can you attach the log again just to be sure?

          francisu Francis Upton added a comment - So the same problem then? Can you attach the log again just to be sure?
          svmkphani Murali Phani added a comment -

          Yes.Same issue.attached jenkins_1.31.log

          svmkphani Murali Phani added a comment - Yes.Same issue.attached jenkins_1.31.log
          francisu Francis Upton added a comment -

          I'm not familiar with how to use Eucalyptus, nor can I find the offending source code to help me to understand this problem. Can you either point me to the source code or give me steps about how I might be able to reproduce this problem so I can fix it?

          francisu Francis Upton added a comment - I'm not familiar with how to use Eucalyptus, nor can I find the offending source code to help me to understand this problem. Can you either point me to the source code or give me steps about how I might be able to reproduce this problem so I can fix it?
          svmkphani Murali Phani added a comment -

          Sorry,i am also doing on first time checking this plugin.As explained in "Description" I have followed to update plugin.On fresh Jenkins dashboard i have put ec2 plugin and needed dependent plugin and configured cloud details .It has failed to launch slaves when tried to test job with slave name.

          svmkphani Murali Phani added a comment - Sorry,i am also doing on first time checking this plugin.As explained in "Description" I have followed to update plugin.On fresh Jenkins dashboard i have put ec2 plugin and needed dependent plugin and configured cloud details .It has failed to launch slaves when tried to test job with slave name.
          svmkphani Murali Phani added a comment -

          Please provide similar plugin details which works similarly like ec2

          svmkphani Murali Phani added a comment - Please provide similar plugin details which works similarly like ec2
          francisu Francis Upton added a comment -

          Can you try this with version 1.29 of the ec2 plugin?

          francisu Francis Upton added a comment - Can you try this with version 1.29 of the ec2 plugin?
          svmkphani Murali Phani added a comment -

          Great.1.29 worked for me.No issues with 1.29.Thanks for help

          svmkphani Murali Phani added a comment - Great.1.29 worked for me.No issues with 1.29.Thanks for help
          francisu Francis Upton added a comment -

          OK, I see the problem (introduced in 1.30). The DescribeSpotInstancesRequest is called without any filters which is not valid. Shortly I will provide an update for you to try if you don't mind to confirm I fixed it,

          francisu Francis Upton added a comment - OK, I see the problem (introduced in 1.30). The DescribeSpotInstancesRequest is called without any filters which is not valid. Shortly I will provide an update for you to try if you don't mind to confirm I fixed it,
          francisu Francis Upton added a comment -

          Can you please try this hpi of the EC2 plugin and see if it fixes your problem?

          https://www.dropbox.com/s/avu3z43rlm1mci6/ec2.hpi?dl=0

          francisu Francis Upton added a comment - Can you please try this hpi of the EC2 plugin and see if it fixes your problem? https://www.dropbox.com/s/avu3z43rlm1mci6/ec2.hpi?dl=0
          svmkphani Murali Phani added a comment -

          Still same issue .But i noticed one thing that if already instances are created it could able to fetch and use .Only issue with creation(provisioning)

          svmkphani Murali Phani added a comment - Still same issue .But i noticed one thing that if already instances are created it could able to fetch and use .Only issue with creation(provisioning)
          francisu Francis Upton added a comment -

          Thanks, can you upload your log just so I can get additional information? Thanks very much for your quick response and for your patience with this, I hope you don't mind trying a couple more times until we can get this fixed. I will let you know when I have another update.

          francisu Francis Upton added a comment - Thanks, can you upload your log just so I can get additional information? Thanks very much for your quick response and for your patience with this, I hope you don't mind trying a couple more times until we can get this fixed. I will let you know when I have another update.
          svmkphani Murali Phani added a comment -

          Yes,I can check it .I have attached jenkins_1.32.log

          svmkphani Murali Phani added a comment - Yes,I can check it .I have attached jenkins_1.32.log

          Code changed in jenkins
          User: Francis Upton IV
          Path:
          src/main/java/hudson/plugins/ec2/EC2Cloud.java
          http://jenkins-ci.org/commit/ec2-plugin/2d347988974086ddbf89635373c49a51ad9c7d50
          Log:
          JENKINS-32588 Unable to launch linux slaves using ec2 plugin using Eucalyptus

          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Francis Upton IV Path: src/main/java/hudson/plugins/ec2/EC2Cloud.java http://jenkins-ci.org/commit/ec2-plugin/2d347988974086ddbf89635373c49a51ad9c7d50 Log: JENKINS-32588 Unable to launch linux slaves using ec2 plugin using Eucalyptus

          Code changed in jenkins
          User: Francis Upton IV
          Path:
          src/main/java/hudson/plugins/ec2/EC2Cloud.java
          http://jenkins-ci.org/commit/ec2-plugin/0fb5802d317d81ba2684b2f829bb28ef31252ee3
          Log:
          JENKINS-32588 Unable to launch linux slaves using ec2 plugin using Eucalyptus

          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Francis Upton IV Path: src/main/java/hudson/plugins/ec2/EC2Cloud.java http://jenkins-ci.org/commit/ec2-plugin/0fb5802d317d81ba2684b2f829bb28ef31252ee3 Log: JENKINS-32588 Unable to launch linux slaves using ec2 plugin using Eucalyptus
          francisu Francis Upton added a comment - Please try this version: https://jenkins.ci.cloudbees.com/job/plugins/job/ec2-plugin/381/org.jenkins-ci.plugins$ec2/artifact/org.jenkins-ci.plugins/ec2/1.32-SNAPSHOT/ec2-1.32-SNAPSHOT.jar
          svmkphani Murali Phani added a comment -

          can you please share as hpi

          svmkphani Murali Phani added a comment - can you please share as hpi
          francisu Francis Upton added a comment - Oops, sorry: https://jenkins.ci.cloudbees.com/job/plugins/job/ec2-plugin/381/org.jenkins-ci.plugins$ec2/artifact/org.jenkins-ci.plugins/ec2/1.32-SNAPSHOT/ec2-1.32-SNAPSHOT.hpi
          svmkphani Murali Phani added a comment -

          But i have one issue (in 1.29 also same issue) "with only 'root' account i could able to launch instances .if i provide other LDAP account is not able to authenticate and launch instance.

          Remote FS root /root
          Remote user root

          svmkphani Murali Phani added a comment - But i have one issue (in 1.29 also same issue) "with only 'root' account i could able to launch instances .if i provide other LDAP account is not able to authenticate and launch instance. Remote FS root /root Remote user root
          francisu Francis Upton added a comment -

          The issue that you have with 1.29 about the root should be a separate JIRA since it happens in that version. This issue is only the problem introduced since 1.29.

          I looked at the log and there is little in it. I don't see any tracing of exceptions or anything. Do you have more information?

          francisu Francis Upton added a comment - The issue that you have with 1.29 about the root should be a separate JIRA since it happens in that version. This issue is only the problem introduced since 1.29. I looked at the log and there is little in it. I don't see any tracing of exceptions or anything. Do you have more information?
          svmkphani Murali Phani added a comment - - edited

          sorry,the new pluginis working fine.But it is taking too long to create instance.At least 5-10 mins for initiating instance launch

          About root account ,the only log appearing at console.I will attach log .

          svmkphani Murali Phani added a comment - - edited sorry,the new pluginis working fine.But it is taking too long to create instance.At least 5-10 mins for initiating instance launch About root account ,the only log appearing at console.I will attach log .
          svmkphani Murali Phani added a comment - Reported another Jira regarding LDAP https://issues.jenkins-ci.org/browse/JENKINS-32703?filter=-2
          francisu Francis Upton added a comment -

          Is the problem with the long instance creation reproduceable? Can you provide the logs for this?

          francisu Francis Upton added a comment - Is the problem with the long instance creation reproduceable? Can you provide the logs for this?
          svmkphani Murali Phani added a comment -

          Sure.I will provide log .Could you please look into issue JENKINS-32703

          svmkphani Murali Phani added a comment - Sure.I will provide log .Could you please look into issue JENKINS-32703

          Looks to me like this was fixed three months ago in https://github.com/jenkinsci/ec2-plugin/commit/0fb5802d317d81ba2684b2f829bb28ef31252ee3, but still hasn't been released. Are there plans for a new release any time soon?

          gavllew Gavin Llewellyn added a comment - Looks to me like this was fixed three months ago in https://github.com/jenkinsci/ec2-plugin/commit/0fb5802d317d81ba2684b2f829bb28ef31252ee3 , but still hasn't been released. Are there plans for a new release any time soon?
          francisu Francis Upton added a comment -

          Should have a release by tomorrow. We are waiting on one more bug fix.

          francisu Francis Upton added a comment - Should have a release by tomorrow. We are waiting on one more bug fix.
          francisu Francis Upton added a comment -

          Fixed in 1.32 release

          francisu Francis Upton added a comment - Fixed in 1.32 release

          Thanks Francis!

          gavllew Gavin Llewellyn added a comment - Thanks Francis!

          People

            francisu Francis Upton
            svmkphani Murali Phani
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: