• Evergreen - Milestone 2

      Problem statement

      The AWS instance I have for testing got broken when updating from 134 to 135. (I'm attaching the updates.auditLog file for deeper analysis.
      In the UI, we get the usual

      java.lang.IllegalArgumentException: No hudson.slaves.Cloud implementation found for amazonEC2
      Possible solution: Try to install 'configuration-as-code-support' plugin
      	at io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.findDescribableBySymbol(HeteroDescribableConfigurator.java:142)
      	at io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.configure(HeteroDescribableConfigurator.java:93)
      	at io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.check(HeteroDescribableConfigurator.java:101)
      	at io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.check(HeteroDescribableConfigurator.java:43)
      	at io.jenkins.plugins.casc.BaseConfigurator.configure(BaseConfigurator.java:322)
      	at io.jenkins.plugins.casc.BaseConfigurator.check(BaseConfigurator.java:265)
      	at io.jenkins.plugins.casc.ConfigurationAsCode.lambda$checkWith$8(ConfigurationAsCode.java:640)
      	at io.jenkins.plugins.casc.ConfigurationAsCode.invokeWith(ConfigurationAsCode.java:605)
      	at io.jenkins.plugins.casc.ConfigurationAsCode.checkWith(ConfigurationAsCode.java:640)
      	at io.jenkins.plugins.casc.ConfigurationAsCode.configureWith(ConfigurationAsCode.java:627)
      	at io.jenkins.plugins.casc.ConfigurationAsCode.configureWith(ConfigurationAsCode.java:539)
      	at io.jenkins.plugins.casc.ConfigurationAsCode.configure(ConfigurationAsCode.java:270)
      	at io.jenkins.plugins.casc.ConfigurationAsCode.init(ConfigurationAsCode.java:262)
      Caused: java.lang.reflect.InvocationTargetException
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104)
      Caused: java.lang.Error
      	at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:110)
      	at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
      	at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
      	at jenkins.model.Jenkins$5.runTask(Jenkins.java:1066)
      	at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
      	at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Caused: org.jvnet.hudson.reactor.ReactorException
      	at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:282)
      	at jenkins.InitReactorRunner.run(InitReactorRunner.java:48)
      	at jenkins.model.Jenkins.executeReactor(Jenkins.java:1100)
      	at jenkins.model.Jenkins.<init>(Jenkins.java:904)
      	at hudson.model.Hudson.<init>(Hudson.java:85)
      	at hudson.model.Hudson.<init>(Hudson.java:81)
      	at hudson.WebAppMain$3.run(WebAppMain.java:233)
      Caused: hudson.util.HudsonFailedToLoad
      	at hudson.WebAppMain$3.run(WebAppMain.java:250)
      

      (Exposed also is another issue with continuously rolling back, but let's process first the fact the instance got borked by, apparently, a wrong computation of plugins to be updated)

      Expected

      The update should go fine, and the result of the instance should be the target UL.

        1. failed-restart.log
          399 kB
        2. ul134.json
          41 kB
        3. ul135.json
          41 kB
        4. updates.auditlog
          71 kB

          [JENKINS-54358] AWS flavor got broken by a wrong update

          So the update from 134 to 135 failed:

           

          cat updates.auditlog| jq .updateLevel | head
          134
          135
          134
          133
          132
          131
          130
          129
          128
          127
           

          So first interesting thing (just retrieved the full json from UL 134 and 135 using INFRA-1769, yey!):

          Despite the only difference between the metrics plugin:

          evergreen_prod=> select commit from updates where id=135;
                            commit
          ------------------------------------------
           6bffe54dc7265b90e714da21f1cb248e1e2f2319
          (1 row)
          
          $ diff ul134.json ul135.json
          2c2
          <   "timestamp": 1540568443089,
          ---
          >   "timestamp": 1540570564042,
          624,625c624,625
          <       "version": "4.0.2.2",
          <       "url": "https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/metrics/4.0.2.2/metrics-4.0.2.2.hpi",
          ---
          >       "version": "4.0.2.3-rc229.d9b0eb3c576d",
          >       "url": "https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/metrics/4.0.2.3-rc229.d9b0eb3c576d/metrics-4.0.2.3-rc229.d9b0eb3c576d.hpi",
          628c628
          <         "signature": "b60b49ab946cd8b960f999876b7da880fadf9ca07dd9524f9f0af63a5d0342eb"
          ---
          >         "signature": "1c1b95da5a0131a1af73ca81d90e4135ee3d05919a7382cc0c4f8195c9133aba"
          

          Here is what was served by the backend to the instance for this update:

          (as we can see, the part about metrics plugin is correct, but there's an chunk about EC2 and aws-java-sdk that shouldn't be here to be analyzed)

          {
            "timestamp": "2018-10-29T14:31:30.363Z",
            "updateLevel": 135,
            "manifest": {
              "meta": {
                "channel": "general",
                "level": 135
              },
              "core": {},
              "plugins": {
                "updates": [
                  {
                    "url": "https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/metrics/4.0.2.3-rc229.d9b0eb3c576d/metrics-4.0.2.3-rc229.d9b0eb3c576d.hpi",
                    "artifactId": "metrics",
                    "checksum": {
                      "type": "sha256",
                      "signature": "1c1b95da5a0131a1af73ca81d90e4135ee3d05919a7382cc0c4f8195c9133aba"
                    }
                  },
                  {
                    "url": "https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/aws-java-sdk/1.11.341/aws-java-sdk-1.11.341.hpi",
                    "artifactId": "aws-java-sdk",
                    "checksum": {
                      "type": "sha256",
                      "signature": "ebe7cca7755dc020d138ef537a7bf7b61e3bbad41f7f0f6be4004917c0769238"
                    }
                  },
                  {
                    "url": "https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/ec2/1.39/ec2-1.39.hpi",
                    "artifactId": "ec2",
                    "checksum": {
                      "type": "sha256",
                      "signature": "fdd8670b1a71bcdacc5d7a358fa67fb8c44efc6f2ecc717d5e572c669934a0a7"
                    }
                  }
                ],
                "deletes": []
              }
            }
          }
          

          Baptiste Mathus added a comment - So the update from 134 to 135 failed:   cat updates.auditlog| jq .updateLevel | head 134 135 134 133 132 131 130 129 128 127 So first interesting thing (just retrieved the full json from UL 134 and 135 using INFRA-1769, yey!): Despite the only difference between the metrics plugin: evergreen_prod=> select commit from updates where id=135; commit ------------------------------------------ 6bffe54dc7265b90e714da21f1cb248e1e2f2319 (1 row) $ diff ul134.json ul135.json 2c2 < "timestamp": 1540568443089, --- > "timestamp": 1540570564042, 624,625c624,625 < "version": "4.0.2.2", < "url": "https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/metrics/4.0.2.2/metrics-4.0.2.2.hpi", --- > "version": "4.0.2.3-rc229.d9b0eb3c576d", > "url": "https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/metrics/4.0.2.3-rc229.d9b0eb3c576d/metrics-4.0.2.3-rc229.d9b0eb3c576d.hpi", 628c628 < "signature": "b60b49ab946cd8b960f999876b7da880fadf9ca07dd9524f9f0af63a5d0342eb" --- > "signature": "1c1b95da5a0131a1af73ca81d90e4135ee3d05919a7382cc0c4f8195c9133aba" Here is what was served by the backend to the instance for this update: (as we can see, the part about metrics plugin is correct, but there's an chunk about EC2 and aws-java-sdk that shouldn't be here to be analyzed) { "timestamp": "2018-10-29T14:31:30.363Z", "updateLevel": 135, "manifest": { "meta": { "channel": "general", "level": 135 }, "core": {}, "plugins": { "updates": [ { "url": "https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/metrics/4.0.2.3-rc229.d9b0eb3c576d/metrics-4.0.2.3-rc229.d9b0eb3c576d.hpi", "artifactId": "metrics", "checksum": { "type": "sha256", "signature": "1c1b95da5a0131a1af73ca81d90e4135ee3d05919a7382cc0c4f8195c9133aba" } }, { "url": "https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/aws-java-sdk/1.11.341/aws-java-sdk-1.11.341.hpi", "artifactId": "aws-java-sdk", "checksum": { "type": "sha256", "signature": "ebe7cca7755dc020d138ef537a7bf7b61e3bbad41f7f0f6be4004917c0769238" } }, { "url": "https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/ec2/1.39/ec2-1.39.hpi", "artifactId": "ec2", "checksum": { "type": "sha256", "signature": "fdd8670b1a71bcdacc5d7a358fa67fb8c44efc6f2ecc717d5e572c669934a0a7" } } ], "deletes": [] } } }

            batmat Baptiste Mathus
            batmat Baptiste Mathus
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: