• Evergreen - Milestone 2

      Problem statement

      The AWS instance I have for testing got broken when updating from 134 to 135. (I'm attaching the updates.auditLog file for deeper analysis.
      In the UI, we get the usual

      java.lang.IllegalArgumentException: No hudson.slaves.Cloud implementation found for amazonEC2
      Possible solution: Try to install 'configuration-as-code-support' plugin
      	at io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.findDescribableBySymbol(HeteroDescribableConfigurator.java:142)
      	at io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.configure(HeteroDescribableConfigurator.java:93)
      	at io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.check(HeteroDescribableConfigurator.java:101)
      	at io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.check(HeteroDescribableConfigurator.java:43)
      	at io.jenkins.plugins.casc.BaseConfigurator.configure(BaseConfigurator.java:322)
      	at io.jenkins.plugins.casc.BaseConfigurator.check(BaseConfigurator.java:265)
      	at io.jenkins.plugins.casc.ConfigurationAsCode.lambda$checkWith$8(ConfigurationAsCode.java:640)
      	at io.jenkins.plugins.casc.ConfigurationAsCode.invokeWith(ConfigurationAsCode.java:605)
      	at io.jenkins.plugins.casc.ConfigurationAsCode.checkWith(ConfigurationAsCode.java:640)
      	at io.jenkins.plugins.casc.ConfigurationAsCode.configureWith(ConfigurationAsCode.java:627)
      	at io.jenkins.plugins.casc.ConfigurationAsCode.configureWith(ConfigurationAsCode.java:539)
      	at io.jenkins.plugins.casc.ConfigurationAsCode.configure(ConfigurationAsCode.java:270)
      	at io.jenkins.plugins.casc.ConfigurationAsCode.init(ConfigurationAsCode.java:262)
      Caused: java.lang.reflect.InvocationTargetException
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104)
      Caused: java.lang.Error
      	at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:110)
      	at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
      	at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
      	at jenkins.model.Jenkins$5.runTask(Jenkins.java:1066)
      	at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
      	at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Caused: org.jvnet.hudson.reactor.ReactorException
      	at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:282)
      	at jenkins.InitReactorRunner.run(InitReactorRunner.java:48)
      	at jenkins.model.Jenkins.executeReactor(Jenkins.java:1100)
      	at jenkins.model.Jenkins.<init>(Jenkins.java:904)
      	at hudson.model.Hudson.<init>(Hudson.java:85)
      	at hudson.model.Hudson.<init>(Hudson.java:81)
      	at hudson.WebAppMain$3.run(WebAppMain.java:233)
      Caused: hudson.util.HudsonFailedToLoad
      	at hudson.WebAppMain$3.run(WebAppMain.java:250)
      

      (Exposed also is another issue with continuously rolling back, but let's process first the fact the instance got borked by, apparently, a wrong computation of plugins to be updated)

      Expected

      The update should go fine, and the result of the instance should be the target UL.

        1. updates.auditlog
          71 kB
          Baptiste Mathus
        2. ul135.json
          41 kB
          Baptiste Mathus
        3. ul134.json
          41 kB
          Baptiste Mathus
        4. failed-restart.log
          399 kB
          Baptiste Mathus

          [JENKINS-54358] AWS flavor got broken by a wrong update

          So the update from 134 to 135 failed:

           

          cat updates.auditlog| jq .updateLevel | head
          134
          135
          134
          133
          132
          131
          130
          129
          128
          127
           

          So first interesting thing (just retrieved the full json from UL 134 and 135 using INFRA-1769, yey!):

          Despite the only difference between the metrics plugin:

          evergreen_prod=> select commit from updates where id=135;
                            commit
          ------------------------------------------
           6bffe54dc7265b90e714da21f1cb248e1e2f2319
          (1 row)
          
          $ diff ul134.json ul135.json
          2c2
          <   "timestamp": 1540568443089,
          ---
          >   "timestamp": 1540570564042,
          624,625c624,625
          <       "version": "4.0.2.2",
          <       "url": "https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/metrics/4.0.2.2/metrics-4.0.2.2.hpi",
          ---
          >       "version": "4.0.2.3-rc229.d9b0eb3c576d",
          >       "url": "https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/metrics/4.0.2.3-rc229.d9b0eb3c576d/metrics-4.0.2.3-rc229.d9b0eb3c576d.hpi",
          628c628
          <         "signature": "b60b49ab946cd8b960f999876b7da880fadf9ca07dd9524f9f0af63a5d0342eb"
          ---
          >         "signature": "1c1b95da5a0131a1af73ca81d90e4135ee3d05919a7382cc0c4f8195c9133aba"
          

          Here is what was served by the backend to the instance for this update:

          (as we can see, the part about metrics plugin is correct, but there's an chunk about EC2 and aws-java-sdk that shouldn't be here to be analyzed)

          {
            "timestamp": "2018-10-29T14:31:30.363Z",
            "updateLevel": 135,
            "manifest": {
              "meta": {
                "channel": "general",
                "level": 135
              },
              "core": {},
              "plugins": {
                "updates": [
                  {
                    "url": "https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/metrics/4.0.2.3-rc229.d9b0eb3c576d/metrics-4.0.2.3-rc229.d9b0eb3c576d.hpi",
                    "artifactId": "metrics",
                    "checksum": {
                      "type": "sha256",
                      "signature": "1c1b95da5a0131a1af73ca81d90e4135ee3d05919a7382cc0c4f8195c9133aba"
                    }
                  },
                  {
                    "url": "https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/aws-java-sdk/1.11.341/aws-java-sdk-1.11.341.hpi",
                    "artifactId": "aws-java-sdk",
                    "checksum": {
                      "type": "sha256",
                      "signature": "ebe7cca7755dc020d138ef537a7bf7b61e3bbad41f7f0f6be4004917c0769238"
                    }
                  },
                  {
                    "url": "https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/ec2/1.39/ec2-1.39.hpi",
                    "artifactId": "ec2",
                    "checksum": {
                      "type": "sha256",
                      "signature": "fdd8670b1a71bcdacc5d7a358fa67fb8c44efc6f2ecc717d5e572c669934a0a7"
                    }
                  }
                ],
                "deletes": []
              }
            }
          }
          

          Baptiste Mathus added a comment - So the update from 134 to 135 failed:   cat updates.auditlog| jq .updateLevel | head 134 135 134 133 132 131 130 129 128 127 So first interesting thing (just retrieved the full json from UL 134 and 135 using INFRA-1769, yey!): Despite the only difference between the metrics plugin: evergreen_prod=> select commit from updates where id=135; commit ------------------------------------------ 6bffe54dc7265b90e714da21f1cb248e1e2f2319 (1 row) $ diff ul134.json ul135.json 2c2 < "timestamp": 1540568443089, --- > "timestamp": 1540570564042, 624,625c624,625 < "version": "4.0.2.2", < "url": "https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/metrics/4.0.2.2/metrics-4.0.2.2.hpi", --- > "version": "4.0.2.3-rc229.d9b0eb3c576d", > "url": "https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/metrics/4.0.2.3-rc229.d9b0eb3c576d/metrics-4.0.2.3-rc229.d9b0eb3c576d.hpi", 628c628 < "signature": "b60b49ab946cd8b960f999876b7da880fadf9ca07dd9524f9f0af63a5d0342eb" --- > "signature": "1c1b95da5a0131a1af73ca81d90e4135ee3d05919a7382cc0c4f8195c9133aba" Here is what was served by the backend to the instance for this update: (as we can see, the part about metrics plugin is correct, but there's an chunk about EC2 and aws-java-sdk that shouldn't be here to be analyzed) { "timestamp": "2018-10-29T14:31:30.363Z", "updateLevel": 135, "manifest": { "meta": { "channel": "general", "level": 135 }, "core": {}, "plugins": { "updates": [ { "url": "https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/metrics/4.0.2.3-rc229.d9b0eb3c576d/metrics-4.0.2.3-rc229.d9b0eb3c576d.hpi", "artifactId": "metrics", "checksum": { "type": "sha256", "signature": "1c1b95da5a0131a1af73ca81d90e4135ee3d05919a7382cc0c4f8195c9133aba" } }, { "url": "https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/aws-java-sdk/1.11.341/aws-java-sdk-1.11.341.hpi", "artifactId": "aws-java-sdk", "checksum": { "type": "sha256", "signature": "ebe7cca7755dc020d138ef537a7bf7b61e3bbad41f7f0f6be4004917c0769238" } }, { "url": "https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/ec2/1.39/ec2-1.39.hpi", "artifactId": "ec2", "checksum": { "type": "sha256", "signature": "fdd8670b1a71bcdacc5d7a358fa67fb8c44efc6f2ecc717d5e572c669934a0a7" } } ], "deletes": [] } } }

            batmat Baptiste Mathus
            batmat Baptiste Mathus
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: