Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-34684

Jcloud - Unable to create build nodes in Openstack cloud

    • Icon: Bug Bug
    • Resolution: Not A Defect
    • Icon: Major Major
    • jclouds-plugin
    • Jenkins Version: 1.580.1
      jclouds plugin: 2.8.1-1
      Openstack Versions: Icehouse - nova, keystone, neutron; Kilo - glance, cinder

      In the last few days, our Jenkins master has lost the ability to communicate with our Openstack installation. When jenkins tries to create a slave, or reconnect to existing slaves after a Jenkins restart, there is an error:

      ...
      May 09, 2016 9:17:33 AM org.jclouds.logging.jdk.JDKLogger logError
      SEVERE: Error parsing input
      com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected a string but was BEGIN_ARRAY at line 1 column 930
      	at org.jclouds.json.internal.DeserializationConstructorAndReflectiveTypeAdapterFactory$DeserializeIntoParameterizedConstructor.read(DeserializationConstructorAndReflectiveTypeAdapterFactory.java:181)
      ...
      

      Complete log file is attached as well as a sample response from Openstack.

      Our Openstack engineer has examine the cloud configuration, and has not been able to identify any issues there.

      Restarting the service & the host did not clear up the issue. We did not change any Jenkins config, and this was working normally previously.

          [JENKINS-34684] Jcloud - Unable to create build nodes in Openstack cloud

          Fritz Elfert added a comment -

          Obviously, something in the openstack response has changed (some single value was expected, but an array value was returned). Similar exceptions have been noticed before, when some openstack components had been updated.
          To find out the exact reason, please upload the json response again, but NOT beautified but as a single line. Otherwise it is impossible to determine what element started at column 930 originally.

          Fritz Elfert added a comment - Obviously, something in the openstack response has changed (some single value was expected, but an array value was returned). Similar exceptions have been noticed before, when some openstack components had been updated. To find out the exact reason, please upload the json response again, but NOT beautified but as a single line . Otherwise it is impossible to determine what element started at column 930 originally.

          dave goodine added a comment -

          felfert - here is the json:

          {"access": {"token": {"issued_at": "2016-05-09T20:31:38.918294", "expires": "2016-05-10T20:31:38Z", "id": "96471d3bdaf245de950a5e8c98dd96cf", "tenant": {"description": "", "enabled": true, "id": "be351b66fff74a25a1e060f101a1889a", "name": "cibuild"}}, "serviceCatalog": [{"endpoints": [{"adminURL": "http://localhost:8888", "region": "regionOne", "internalURL": "http://localhost:888", "id": "7dbfb467567b400d81a3eeb018e0a41a", "publicURL": "http://localhost:8888"}], "endpoints_links": [], "type": "rating", "name": "CloudKitty"}, {"endpoints": [{"adminURL": "http://controller-n02.openstack.drillinginfo.com:8774/v2/be351b66fff74a25a1e060f101a1889a", "region": "regionOne", "internalURL": "http://controller-n02.openstack.drillinginfo.com:8774/v2/be351b66fff74a25a1e060f101a1889a", "id": "33dc4970833847db8280ace6fa7dff32", "publicURL": "http://controller-n02.openstack.drillinginfo.com:8774/v2/be351b66fff74a25a1e060f101a1889a"}], "endpoints_links": [], "type": "compute", "name": "nova"}, {"endpoints": [{"adminURL": "http://controller-n02.openstack.drillinginfo.com:9696", "region": "regionOne", "internalURL": "http://controller-n02.openstack.drillinginfo.com:9696", "id": "5e68688c7d1247d39fa0b5535f323840", "publicURL": "http://controller-n02.openstack.drillinginfo.com:9696"}], "endpoints_links": [], "type": "network", "name": "neutron"}, {"endpoints": [{"adminURL": "http://cinder.openstack.drillinginfo.com:8776/v2/be351b66fff74a25a1e060f101a1889a", "region": "regionOne", "internalURL": "http://cinder.openstack.drillinginfo.com:8776/v2/be351b66fff74a25a1e060f101a1889a", "id": "866d4b882b044387955d1809fc3248df", "publicURL": "http://cinder.openstack.drillinginfo.com:8776/v2/be351b66fff74a25a1e060f101a1889a"}], "endpoints_links": [], "type": "volumev2", "name": "cinderv2"}, {"endpoints": [{"adminURL": "http://controller-n01.openstack.drillinginfo.com:9292", "region": "regionOne", "internalURL": "http://controller-n01.openstack.drillinginfo.com:9292", "id": "4b7000e012a94082bfcc3e1745dbe21e", "publicURL": "http://controller-n01.openstack.drillinginfo.com:9292"}], "endpoints_links": [], "type": "image", "name": "glance"}, {"endpoints": [{"adminURL": "http://controller-n01.openstack.drillinginfo.com:8777", "region": "regionOne", "internalURL": "http://controller-n01.openstack.drillinginfo.com:8777", "id": "3f5714f2514247be8947285c7b951dbb", "publicURL": "http://controller-n01.openstack.drillinginfo.com:8777"}], "endpoints_links": [], "type": "metering", "name": "ceilometer"}, {"endpoints": [{"adminURL": "http://controller-n01.openstack.drillinginfo.com:8000/v1", "region": "regionOne", "internalURL": "http://controller-n01.openstack.drillinginfo.com:8000/v1", "id": "55e2b168ac454e369f4207b5a757c4be", "publicURL": "http://controller-n01.openstack.drillinginfo.com:8000/v1"}], "endpoints_links": [], "type": "cloudformation", "name": "heat-cfn"}, {"endpoints": [{"adminURL": "http://cinder.openstack.drillinginfo.com:8776/v1/be351b66fff74a25a1e060f101a1889a", "region": "regionOne", "internalURL": "http://cinder.openstack.drillinginfo.com:8776/v1/be351b66fff74a25a1e060f101a1889a", "id": "cfa7beecf0064c9181bd4d27aebf31e8", "publicURL": "http://cinder.openstack.drillinginfo.com:8776/v1/be351b66fff74a25a1e060f101a1889a"}], "endpoints_links": [], "type": "volume", "name": "cinder"}, {"endpoints": [{"adminURL": "http://controller-n01.openstack.drillinginfo.com:8004/v1/be351b66fff74a25a1e060f101a1889a", "region": "regionOne", "internalURL": "http://controller-n01.openstack.drillinginfo.com:8004/v1/be351b66fff74a25a1e060f101a1889a", "id": "46b092a9a68a4bec89d9bdbf56257bdd", "publicURL": "http://controller-n01.openstack.drillinginfo.com:8004/v1/be351b66fff74a25a1e060f101a1889a"}], "endpoints_links": [], "type": "orchestration", "name": "heat"}, {"endpoints": [{"adminURL": "http://controller-n02.openstack.drillinginfo.com:35357/v2.0", "region": "regionOne", "internalURL": "http://controller-n02.openstack.drillinginfo.com:5000/v2.0", "id": "5706a782cffa4a8ebfa4b3f884771cc2", "publicURL": "http://controller-n02.openstack.drillinginfo.com:5000/v2.0"}], "endpoints_links": [], "type": "identity", "name": "keystone"}], "user": {"username": "cibuild-openstack", "roles_links": [], "id": "cibuild-openstack", "roles": [{"name": "_member_"}], "name": "cibuild-openstack"}, "metadata": {"is_admin": 0, "roles": ["9fe2ff9ee4384b1894a90878d3e92bab"]}}}
          

          dave goodine added a comment - felfert - here is the json: { "access" : { "token" : { "issued_at" : "2016-05-09T20:31:38.918294" , "expires" : "2016-05-10T20:31:38Z" , "id" : "96471d3bdaf245de950a5e8c98dd96cf" , "tenant" : { "description" : "", " enabled ": true , " id ": " be351b66fff74a25a1e060f101a1889a ", " name ": " cibuild "}}, " serviceCatalog ": [{" endpoints ": [{" adminURL ": " http: //localhost:8888 ", " region ": " regionOne ", " internalURL ": " http://localhost:888 ", " id ": " 7dbfb467567b400d81a3eeb018e0a41a ", " publicURL ": " http://localhost:8888 "}], " endpoints_links ": [], " type ": " rating ", " name ": " CloudKitty "}, {" endpoints ": [{" adminURL ": " http://controller-n02.openstack.drillinginfo.com:8774/v2/be351b66fff74a25a1e060f101a1889a ", " region ": " regionOne ", " internalURL ": " http://controller-n02.openstack.drillinginfo.com:8774/v2/be351b66fff74a25a1e060f101a1889a ", " id ": " 33dc4970833847db8280ace6fa7dff32 ", " publicURL ": " http://controller-n02.openstack.drillinginfo.com:8774/v2/be351b66fff74a25a1e060f101a1889a "}], " endpoints_links ": [], " type ": " compute ", " name ": " nova "}, {" endpoints ": [{" adminURL ": " http://controller-n02.openstack.drillinginfo.com:9696 ", " region ": " regionOne ", " internalURL ": " http://controller-n02.openstack.drillinginfo.com:9696 ", " id ": " 5e68688c7d1247d39fa0b5535f323840 ", " publicURL ": " http://controller-n02.openstack.drillinginfo.com:9696 "}], " endpoints_links ": [], " type ": " network ", " name ": " neutron "}, {" endpoints ": [{" adminURL ": " http://cinder.openstack.drillinginfo.com:8776/v2/be351b66fff74a25a1e060f101a1889a ", " region ": " regionOne ", " internalURL ": " http://cinder.openstack.drillinginfo.com:8776/v2/be351b66fff74a25a1e060f101a1889a ", " id ": " 866d4b882b044387955d1809fc3248df ", " publicURL ": " http://cinder.openstack.drillinginfo.com:8776/v2/be351b66fff74a25a1e060f101a1889a "}], " endpoints_links ": [], " type ": " volumev2 ", " name ": " cinderv2 "}, {" endpoints ": [{" adminURL ": " http://controller-n01.openstack.drillinginfo.com:9292 ", " region ": " regionOne ", " internalURL ": " http://controller-n01.openstack.drillinginfo.com:9292 ", " id ": " 4b7000e012a94082bfcc3e1745dbe21e ", " publicURL ": " http://controller-n01.openstack.drillinginfo.com:9292 "}], " endpoints_links ": [], " type ": " image ", " name ": " glance "}, {" endpoints ": [{" adminURL ": " http://controller-n01.openstack.drillinginfo.com:8777 ", " region ": " regionOne ", " internalURL ": " http://controller-n01.openstack.drillinginfo.com:8777 ", " id ": " 3f5714f2514247be8947285c7b951dbb ", " publicURL ": " http://controller-n01.openstack.drillinginfo.com:8777 "}], " endpoints_links ": [], " type ": " metering ", " name ": " ceilometer "}, {" endpoints ": [{" adminURL ": " http://controller-n01.openstack.drillinginfo.com:8000/v1 ", " region ": " regionOne ", " internalURL ": " http://controller-n01.openstack.drillinginfo.com:8000/v1 ", " id ": " 55e2b168ac454e369f4207b5a757c4be ", " publicURL ": " http://controller-n01.openstack.drillinginfo.com:8000/v1 "}], " endpoints_links ": [], " type ": " cloudformation ", " name ": " heat-cfn "}, {" endpoints ": [{" adminURL ": " http://cinder.openstack.drillinginfo.com:8776/v1/be351b66fff74a25a1e060f101a1889a ", " region ": " regionOne ", " internalURL ": " http://cinder.openstack.drillinginfo.com:8776/v1/be351b66fff74a25a1e060f101a1889a ", " id ": " cfa7beecf0064c9181bd4d27aebf31e8 ", " publicURL ": " http://cinder.openstack.drillinginfo.com:8776/v1/be351b66fff74a25a1e060f101a1889a "}], " endpoints_links ": [], " type ": " volume ", " name ": " cinder "}, {" endpoints ": [{" adminURL ": " http://controller-n01.openstack.drillinginfo.com:8004/v1/be351b66fff74a25a1e060f101a1889a ", " region ": " regionOne ", " internalURL ": " http://controller-n01.openstack.drillinginfo.com:8004/v1/be351b66fff74a25a1e060f101a1889a ", " id ": " 46b092a9a68a4bec89d9bdbf56257bdd ", " publicURL ": " http://controller-n01.openstack.drillinginfo.com:8004/v1/be351b66fff74a25a1e060f101a1889a "}], " endpoints_links ": [], " type ": " orchestration ", " name ": " heat "}, {" endpoints ": [{" adminURL ": " http://controller-n02.openstack.drillinginfo.com:35357/v2.0 ", " region ": " regionOne ", " internalURL ": " http://controller-n02.openstack.drillinginfo.com:5000/v2.0 ", " id ": " 5706a782cffa4a8ebfa4b3f884771cc2 ", " publicURL ": " http://controller-n02.openstack.drillinginfo.com:5000/v2.0 "}], " endpoints_links ": [], " type ": " identity ", " name ": " keystone "}], " user ": {" username ": " cibuild-openstack ", " roles_links ": [], " id ": " cibuild-openstack ", " roles ": [{" name ": " _member_ "}], " name ": " cibuild-openstack "}, " metadata ": {" is_admin ": 0, " roles ": [" 9fe2ff9ee4384b1894a90878d3e92bab"]}}}

          felfert Hello, do you have any suggestions?

          Marisa Sawatphadungkij added a comment - felfert Hello, do you have any suggestions?

          dave goodine added a comment -

          Deleting an image with bad metadata resolved this issue.

          dave goodine added a comment - Deleting an image with bad metadata resolved this issue.

            felfert Fritz Elfert
            davegoodine dave goodine
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: