Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-23788

EC2 plugin regression: fails to launch nodes with "syntax error near unexpected token `newline'"

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      The latest EC2 plugin seems to have broken node launch.

      With 1.21 nodes used to provision correctly from Manage Nodes or on-demand (though, per Issue JENKINS-23787 they weren't spooled up correctly after stopping on idle).

      Now, launch of new nodes or starting existing nodes is failing with:

      Starting existing instance: i-b848fbfa result:{StartingInstances: [{InstanceId: i-b848fbfa,CurrentState: {Code: 0,Name: pending},PreviousState: {Code: 80,Name: stopped}}]}
      Connecting to 10.0.1.164 on port 22, with timeout 10000.
      Waiting for SSH to come up. Sleeping 5.
      Connecting to 10.0.1.164 on port 22, with timeout 10000.
      Connected via SSH.
      bootstrap()
      Getting keypair...
      Using key: jenkins_ec2_key
      xxxx
      -----BEGIN RSA PRIVATE KEY-----
      xxxx
      Authenticating as ec2-user
      take over connection
      Executing init script
      bash: line 1: syntax error near unexpected token `newline'
      bash: line 1: `<?xml version="1.0" encoding="UTF-8"?>'
      init script failed: exit code=2
      

      It looks like the plugin is trying to execute some XML as a script.

      The Jenkins logs contain:

      Jul 15, 2014 4:05:04 AM SEVERE hudson.remoting.ExportTable unexportByOid
      Trying to unexport an object that's already unexported
      java.lang.IllegalStateException: Invalid object ID 112 iota=82
      	at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:304)
      	at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:327)
      	at hudson.remoting.Channel.unexport(Channel.java:600)
      	at hudson.remoting.UnexportCommand.execute(UnexportCommand.java:38)
      	at hudson.remoting.Channel$2.handle(Channel.java:475)
      	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60)
      Caused by: java.lang.Exception: Object appears to be deallocated at lease before Tue Jul 15 01:55:05 UTC 2014
      	at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:300)
      	... 5 more
      
      Jul 15, 2014 4:05:04 AM SEVERE hudson.remoting.ExportTable unexportByOid
      2nd unexport attempt is here
      Command hudson.remoting.UnexportCommand@7b9fbccc created at
      	at hudson.remoting.Command.<init>(Command.java:67)
      	at hudson.remoting.Command.<init>(Command.java:50)
      	at hudson.remoting.UnexportCommand.<init>(UnexportCommand.java:33)
      	at hudson.remoting.RemoteInvocationHandler.finalize(RemoteInvocationHandler.java:221)
      	at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)
      	at java.lang.ref.Finalizer.runFinalizer(Unknown Source)
      	at java.lang.ref.Finalizer.access$100(Unknown Source)
      	at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)
      

      though it's not clear if that's related.

        Attachments

          Activity

          Hide
          ringerc Craig Ringer added a comment -

          This is not an EC2 plugin bug. It's an unrelated misconfiguration.

          The EC2 access key for the host had recently been revoked and re-generated. While the plugin key was updated, the signed URLs for setup scripts, like:

          #!/bin/bash
          curl -L -s "http://2q-jenkins-resources.s3.amazonaws.com/amazonlinux201209_jenkins_instancesetup.sh?AWSAccessKeyId=AKIAJZPC6FQAU4REUX2A&Expires=1670373735&Signature=nnnwC%2Fjmc8F69ZR5GGMnyf%2BVZ64%3D" | sudo bash
          

          were not. The output of this, when curl fails, is:

          <?xml version="1.0" encoding="UTF-8"?>
          <Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>52BD679ED9E0DB18</RequestId><HostId>cdKQeciGcl+HLd1TLSdxY48LGPk8znlYKGIV15momRZ1rPFgcInjnB2tBMx+EIW5</HostId></Error>
          

          thus the error.

          set -o pipefail
          

          would've helped produce a clearer error, when combined with

          curl -L -s --fail
          

          to produce a command like:

          set -o pipefail
          if ! curl -L -s --fail 'http://2q-jenkins-resources.s3.amazonaws.com/amazonlinux201209_jenkins_instancesetup.sh?AWSAccessKeyId=AKIAJZPC6FQAU4REUX2A&amp;Expires=1670373735&amp;Signature=nnnwC%2Fjmc8F69ZR5GGMnyf%2BVZ64%3D' | bash; then 
            echo "Setup script failed"
          fi
          

          So - PEBKAC, no problem here.

          Show
          ringerc Craig Ringer added a comment - This is not an EC2 plugin bug. It's an unrelated misconfiguration. The EC2 access key for the host had recently been revoked and re-generated. While the plugin key was updated, the signed URLs for setup scripts, like: #!/bin/bash curl -L -s "http: //2q-jenkins-resources.s3.amazonaws.com/amazonlinux201209_jenkins_instancesetup.sh?AWSAccessKeyId=AKIAJZPC6FQAU4REUX2A&Expires=1670373735&Signature=nnnwC%2Fjmc8F69ZR5GGMnyf%2BVZ64%3D" | sudo bash were not. The output of this, when curl fails, is: <?xml version= "1.0" encoding= "UTF-8" ?> <Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>52BD679ED9E0DB18</RequestId><HostId>cdKQeciGcl+HLd1TLSdxY48LGPk8znlYKGIV15momRZ1rPFgcInjnB2tBMx+EIW5</HostId></Error> thus the error. set -o pipefail would've helped produce a clearer error, when combined with curl -L -s --fail to produce a command like: set -o pipefail if ! curl -L -s --fail 'http: //2q-jenkins-resources.s3.amazonaws.com/amazonlinux201209_jenkins_instancesetup.sh?AWSAccessKeyId=AKIAJZPC6FQAU4REUX2A&amp;Expires=1670373735&amp;Signature=nnnwC%2Fjmc8F69ZR5GGMnyf%2BVZ64%3D' | bash; then echo "Setup script failed" fi So - PEBKAC, no problem here.
          Hide
          ringerc Craig Ringer added a comment -

          Per comments, this was user misconfiguration.

          Show
          ringerc Craig Ringer added a comment - Per comments, this was user misconfiguration.

            People

            Assignee:
            ringerc Craig Ringer
            Reporter:
            ringerc Craig Ringer
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: