Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-34684

Jcloud - Unable to create build nodes in Openstack cloud

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Not A Defect
    • jclouds-plugin
    • Jenkins Version: 1.580.1
      jclouds plugin: 2.8.1-1
      Openstack Versions: Icehouse - nova, keystone, neutron; Kilo - glance, cinder

    Description

      In the last few days, our Jenkins master has lost the ability to communicate with our Openstack installation. When jenkins tries to create a slave, or reconnect to existing slaves after a Jenkins restart, there is an error:

      ...
      May 09, 2016 9:17:33 AM org.jclouds.logging.jdk.JDKLogger logError
      SEVERE: Error parsing input
      com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected a string but was BEGIN_ARRAY at line 1 column 930
      	at org.jclouds.json.internal.DeserializationConstructorAndReflectiveTypeAdapterFactory$DeserializeIntoParameterizedConstructor.read(DeserializationConstructorAndReflectiveTypeAdapterFactory.java:181)
      ...
      

      Complete log file is attached as well as a sample response from Openstack.

      Our Openstack engineer has examine the cloud configuration, and has not been able to identify any issues there.

      Restarting the service & the host did not clear up the issue. We did not change any Jenkins config, and this was working normally previously.

      Attachments

        1. jenkins.log.20160509
          1.15 MB
          dave goodine
        2. openstack-response.json
          7 kB
          dave goodine

        Activity

          felfert Fritz Elfert added a comment -

          Obviously, something in the openstack response has changed (some single value was expected, but an array value was returned). Similar exceptions have been noticed before, when some openstack components had been updated.
          To find out the exact reason, please upload the json response again, but NOT beautified but as a single line. Otherwise it is impossible to determine what element started at column 930 originally.

          felfert Fritz Elfert added a comment - Obviously, something in the openstack response has changed (some single value was expected, but an array value was returned). Similar exceptions have been noticed before, when some openstack components had been updated. To find out the exact reason, please upload the json response again, but NOT beautified but as a single line . Otherwise it is impossible to determine what element started at column 930 originally.
          davegoodine dave goodine added a comment -

          felfert - here is the json:

          {"access": {"token": {"issued_at": "2016-05-09T20:31:38.918294", "expires": "2016-05-10T20:31:38Z", "id": "96471d3bdaf245de950a5e8c98dd96cf", "tenant": {"description": "", "enabled": true, "id": "be351b66fff74a25a1e060f101a1889a", "name": "cibuild"}}, "serviceCatalog": [{"endpoints": [{"adminURL": "http://localhost:8888", "region": "regionOne", "internalURL": "http://localhost:888", "id": "7dbfb467567b400d81a3eeb018e0a41a", "publicURL": "http://localhost:8888"}], "endpoints_links": [], "type": "rating", "name": "CloudKitty"}, {"endpoints": [{"adminURL": "http://controller-n02.openstack.drillinginfo.com:8774/v2/be351b66fff74a25a1e060f101a1889a", "region": "regionOne", "internalURL": "http://controller-n02.openstack.drillinginfo.com:8774/v2/be351b66fff74a25a1e060f101a1889a", "id": "33dc4970833847db8280ace6fa7dff32", "publicURL": "http://controller-n02.openstack.drillinginfo.com:8774/v2/be351b66fff74a25a1e060f101a1889a"}], "endpoints_links": [], "type": "compute", "name": "nova"}, {"endpoints": [{"adminURL": "http://controller-n02.openstack.drillinginfo.com:9696", "region": "regionOne", "internalURL": "http://controller-n02.openstack.drillinginfo.com:9696", "id": "5e68688c7d1247d39fa0b5535f323840", "publicURL": "http://controller-n02.openstack.drillinginfo.com:9696"}], "endpoints_links": [], "type": "network", "name": "neutron"}, {"endpoints": [{"adminURL": "http://cinder.openstack.drillinginfo.com:8776/v2/be351b66fff74a25a1e060f101a1889a", "region": "regionOne", "internalURL": "http://cinder.openstack.drillinginfo.com:8776/v2/be351b66fff74a25a1e060f101a1889a", "id": "866d4b882b044387955d1809fc3248df", "publicURL": "http://cinder.openstack.drillinginfo.com:8776/v2/be351b66fff74a25a1e060f101a1889a"}], "endpoints_links": [], "type": "volumev2", "name": "cinderv2"}, {"endpoints": [{"adminURL": "http://controller-n01.openstack.drillinginfo.com:9292", "region": "regionOne", "internalURL": "http://controller-n01.openstack.drillinginfo.com:9292", "id": "4b7000e012a94082bfcc3e1745dbe21e", "publicURL": "http://controller-n01.openstack.drillinginfo.com:9292"}], "endpoints_links": [], "type": "image", "name": "glance"}, {"endpoints": [{"adminURL": "http://controller-n01.openstack.drillinginfo.com:8777", "region": "regionOne", "internalURL": "http://controller-n01.openstack.drillinginfo.com:8777", "id": "3f5714f2514247be8947285c7b951dbb", "publicURL": "http://controller-n01.openstack.drillinginfo.com:8777"}], "endpoints_links": [], "type": "metering", "name": "ceilometer"}, {"endpoints": [{"adminURL": "http://controller-n01.openstack.drillinginfo.com:8000/v1", "region": "regionOne", "internalURL": "http://controller-n01.openstack.drillinginfo.com:8000/v1", "id": "55e2b168ac454e369f4207b5a757c4be", "publicURL": "http://controller-n01.openstack.drillinginfo.com:8000/v1"}], "endpoints_links": [], "type": "cloudformation", "name": "heat-cfn"}, {"endpoints": [{"adminURL": "http://cinder.openstack.drillinginfo.com:8776/v1/be351b66fff74a25a1e060f101a1889a", "region": "regionOne", "internalURL": "http://cinder.openstack.drillinginfo.com:8776/v1/be351b66fff74a25a1e060f101a1889a", "id": "cfa7beecf0064c9181bd4d27aebf31e8", "publicURL": "http://cinder.openstack.drillinginfo.com:8776/v1/be351b66fff74a25a1e060f101a1889a"}], "endpoints_links": [], "type": "volume", "name": "cinder"}, {"endpoints": [{"adminURL": "http://controller-n01.openstack.drillinginfo.com:8004/v1/be351b66fff74a25a1e060f101a1889a", "region": "regionOne", "internalURL": "http://controller-n01.openstack.drillinginfo.com:8004/v1/be351b66fff74a25a1e060f101a1889a", "id": "46b092a9a68a4bec89d9bdbf56257bdd", "publicURL": "http://controller-n01.openstack.drillinginfo.com:8004/v1/be351b66fff74a25a1e060f101a1889a"}], "endpoints_links": [], "type": "orchestration", "name": "heat"}, {"endpoints": [{"adminURL": "http://controller-n02.openstack.drillinginfo.com:35357/v2.0", "region": "regionOne", "internalURL": "http://controller-n02.openstack.drillinginfo.com:5000/v2.0", "id": "5706a782cffa4a8ebfa4b3f884771cc2", "publicURL": "http://controller-n02.openstack.drillinginfo.com:5000/v2.0"}], "endpoints_links": [], "type": "identity", "name": "keystone"}], "user": {"username": "cibuild-openstack", "roles_links": [], "id": "cibuild-openstack", "roles": [{"name": "_member_"}], "name": "cibuild-openstack"}, "metadata": {"is_admin": 0, "roles": ["9fe2ff9ee4384b1894a90878d3e92bab"]}}}
          
          davegoodine dave goodine added a comment - felfert - here is the json: { "access" : { "token" : { "issued_at" : "2016-05-09T20:31:38.918294" , "expires" : "2016-05-10T20:31:38Z" , "id" : "96471d3bdaf245de950a5e8c98dd96cf" , "tenant" : { "description" : "", " enabled ": true , " id ": " be351b66fff74a25a1e060f101a1889a ", " name ": " cibuild "}}, " serviceCatalog ": [{" endpoints ": [{" adminURL ": " http: //localhost:8888 ", " region ": " regionOne ", " internalURL ": " http://localhost:888 ", " id ": " 7dbfb467567b400d81a3eeb018e0a41a ", " publicURL ": " http://localhost:8888 "}], " endpoints_links ": [], " type ": " rating ", " name ": " CloudKitty "}, {" endpoints ": [{" adminURL ": " http://controller-n02.openstack.drillinginfo.com:8774/v2/be351b66fff74a25a1e060f101a1889a ", " region ": " regionOne ", " internalURL ": " http://controller-n02.openstack.drillinginfo.com:8774/v2/be351b66fff74a25a1e060f101a1889a ", " id ": " 33dc4970833847db8280ace6fa7dff32 ", " publicURL ": " http://controller-n02.openstack.drillinginfo.com:8774/v2/be351b66fff74a25a1e060f101a1889a "}], " endpoints_links ": [], " type ": " compute ", " name ": " nova "}, {" endpoints ": [{" adminURL ": " http://controller-n02.openstack.drillinginfo.com:9696 ", " region ": " regionOne ", " internalURL ": " http://controller-n02.openstack.drillinginfo.com:9696 ", " id ": " 5e68688c7d1247d39fa0b5535f323840 ", " publicURL ": " http://controller-n02.openstack.drillinginfo.com:9696 "}], " endpoints_links ": [], " type ": " network ", " name ": " neutron "}, {" endpoints ": [{" adminURL ": " http://cinder.openstack.drillinginfo.com:8776/v2/be351b66fff74a25a1e060f101a1889a ", " region ": " regionOne ", " internalURL ": " http://cinder.openstack.drillinginfo.com:8776/v2/be351b66fff74a25a1e060f101a1889a ", " id ": " 866d4b882b044387955d1809fc3248df ", " publicURL ": " http://cinder.openstack.drillinginfo.com:8776/v2/be351b66fff74a25a1e060f101a1889a "}], " endpoints_links ": [], " type ": " volumev2 ", " name ": " cinderv2 "}, {" endpoints ": [{" adminURL ": " http://controller-n01.openstack.drillinginfo.com:9292 ", " region ": " regionOne ", " internalURL ": " http://controller-n01.openstack.drillinginfo.com:9292 ", " id ": " 4b7000e012a94082bfcc3e1745dbe21e ", " publicURL ": " http://controller-n01.openstack.drillinginfo.com:9292 "}], " endpoints_links ": [], " type ": " image ", " name ": " glance "}, {" endpoints ": [{" adminURL ": " http://controller-n01.openstack.drillinginfo.com:8777 ", " region ": " regionOne ", " internalURL ": " http://controller-n01.openstack.drillinginfo.com:8777 ", " id ": " 3f5714f2514247be8947285c7b951dbb ", " publicURL ": " http://controller-n01.openstack.drillinginfo.com:8777 "}], " endpoints_links ": [], " type ": " metering ", " name ": " ceilometer "}, {" endpoints ": [{" adminURL ": " http://controller-n01.openstack.drillinginfo.com:8000/v1 ", " region ": " regionOne ", " internalURL ": " http://controller-n01.openstack.drillinginfo.com:8000/v1 ", " id ": " 55e2b168ac454e369f4207b5a757c4be ", " publicURL ": " http://controller-n01.openstack.drillinginfo.com:8000/v1 "}], " endpoints_links ": [], " type ": " cloudformation ", " name ": " heat-cfn "}, {" endpoints ": [{" adminURL ": " http://cinder.openstack.drillinginfo.com:8776/v1/be351b66fff74a25a1e060f101a1889a ", " region ": " regionOne ", " internalURL ": " http://cinder.openstack.drillinginfo.com:8776/v1/be351b66fff74a25a1e060f101a1889a ", " id ": " cfa7beecf0064c9181bd4d27aebf31e8 ", " publicURL ": " http://cinder.openstack.drillinginfo.com:8776/v1/be351b66fff74a25a1e060f101a1889a "}], " endpoints_links ": [], " type ": " volume ", " name ": " cinder "}, {" endpoints ": [{" adminURL ": " http://controller-n01.openstack.drillinginfo.com:8004/v1/be351b66fff74a25a1e060f101a1889a ", " region ": " regionOne ", " internalURL ": " http://controller-n01.openstack.drillinginfo.com:8004/v1/be351b66fff74a25a1e060f101a1889a ", " id ": " 46b092a9a68a4bec89d9bdbf56257bdd ", " publicURL ": " http://controller-n01.openstack.drillinginfo.com:8004/v1/be351b66fff74a25a1e060f101a1889a "}], " endpoints_links ": [], " type ": " orchestration ", " name ": " heat "}, {" endpoints ": [{" adminURL ": " http://controller-n02.openstack.drillinginfo.com:35357/v2.0 ", " region ": " regionOne ", " internalURL ": " http://controller-n02.openstack.drillinginfo.com:5000/v2.0 ", " id ": " 5706a782cffa4a8ebfa4b3f884771cc2 ", " publicURL ": " http://controller-n02.openstack.drillinginfo.com:5000/v2.0 "}], " endpoints_links ": [], " type ": " identity ", " name ": " keystone "}], " user ": {" username ": " cibuild-openstack ", " roles_links ": [], " id ": " cibuild-openstack ", " roles ": [{" name ": " _member_ "}], " name ": " cibuild-openstack "}, " metadata ": {" is_admin ": 0, " roles ": [" 9fe2ff9ee4384b1894a90878d3e92bab"]}}}

          felfert Hello, do you have any suggestions?

          marisasawat Marisa Sawatphadungkij added a comment - felfert Hello, do you have any suggestions?
          davegoodine dave goodine added a comment -

          Deleting an image with bad metadata resolved this issue.

          davegoodine dave goodine added a comment - Deleting an image with bad metadata resolved this issue.

          People

            felfert Fritz Elfert
            davegoodine dave goodine
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: