Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-53983

Per-instance tainting still serves the tainted level if you're "coming from" a lower level

    XMLWordPrintable

    Details

    • Similar Issues:
    • Sprint:
      Evergreen - Milestone 2

      Description

      NOTE: I've added a reproducer test case, and this issue matches my experience from testing it locally when I see the rollback work, and then still re-upgrade to the tainted level after going away from it previously. But I'm still not 100% sure of the diagnosis, so keep this in mind below.

      Problem statement

      When an instance taints a given update level (after a failure to upgrade), then it correctly gets the previous non tainted UL.
      But then, when asking the backend again, passing the previous UL as parameter (i.e. the good one, the one before the just tainted one), then the backend will still serve an answer to upgrade to the tainted level.

      From a user standpoint, this will surface through the following behavior:

      • Imagine we are on UL 50.
      • UL51 is pushed
      • Instance updates to UL51 and is unable to restart
      • Instance informs the backend UL51 was a failure (aka taints it)
      • Instance goes back to UL50 from UL51
      • Instance updates to UL51, fails...
      • Loop back to a few lines above, and so on. I.e. the instance will do UL50->UL51->UL50->UL51, etc.

      Expected behavior

      Once per-instance tainted, a given UL should never be proposed again to a given instance.

        Attachments

          Issue Links

            Activity

            batmat Baptiste Mathus created issue -
            batmat Baptiste Mathus made changes -
            Field Original Value New Value
            Assignee R. Tyler Croy [ rtyler ] Baptiste Mathus [ batmat ]
            batmat Baptiste Mathus made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            batmat Baptiste Mathus made changes -
            Link This issue is blocking JENKINS-53273 [ JENKINS-53273 ]
            batmat Baptiste Mathus made changes -
            Description _NOTE: I've added a [reproducer test case|https://github.com/jenkins-infra/evergreen/pull/316/commits/bf5a61c5312642efecb3007cf61a8903d5e59c08], and this issue matches my experience from testing it locally when I see the rollback work, and then still re-upgrade to the tainted level after going away from it previously. But I'm still not 100% sure of the diagnosis, so keep this in mind below._

            h3. Problem statement
            When an instance taints a given update level (after a failure to upgrade), then it correctly gets the previous non tainted UL.
            But then, when asking the backend again, passing the previous UL as parameter (i.e. the good one, the one *before* the just tainted one), then the backend will still serve an answer to upgrade to the tainted level.


            h3. Expected behavior
            Once _per-instance tainted_ a given UL should never be proposed again to a given instance.
            _NOTE: I've added a [reproducer test case|https://github.com/jenkins-infra/evergreen/pull/316/commits/bf5a61c5312642efecb3007cf61a8903d5e59c08], and this issue matches my experience from testing it locally when I see the rollback work, and then still re-upgrade to the tainted level after going away from it previously. But I'm still not 100% sure of the diagnosis, so keep this in mind below._

            h3. Problem statement
            When an instance taints a given update level (after a failure to upgrade), then it correctly gets the previous non tainted UL.
            But then, when asking the backend again, passing the previous UL as parameter (i.e. the good one, the one *before* the just tainted one), then the backend will still serve an answer to upgrade to the tainted level.


            h3. Expected behavior
            Once _per-instance tainted_, a given UL should never be proposed again to a given instance.
            batmat Baptiste Mathus made changes -
            Description _NOTE: I've added a [reproducer test case|https://github.com/jenkins-infra/evergreen/pull/316/commits/bf5a61c5312642efecb3007cf61a8903d5e59c08], and this issue matches my experience from testing it locally when I see the rollback work, and then still re-upgrade to the tainted level after going away from it previously. But I'm still not 100% sure of the diagnosis, so keep this in mind below._

            h3. Problem statement
            When an instance taints a given update level (after a failure to upgrade), then it correctly gets the previous non tainted UL.
            But then, when asking the backend again, passing the previous UL as parameter (i.e. the good one, the one *before* the just tainted one), then the backend will still serve an answer to upgrade to the tainted level.


            h3. Expected behavior
            Once _per-instance tainted_, a given UL should never be proposed again to a given instance.
            _NOTE: I've added a [reproducer test case|https://github.com/jenkins-infra/evergreen/pull/316/commits/bf5a61c5312642efecb3007cf61a8903d5e59c08], and this issue matches my experience from testing it locally when I see the rollback work, and then still re-upgrade to the tainted level after going away from it previously. But I'm still not 100% sure of the diagnosis, so keep this in mind below._

            h3. Problem statement
            When an instance taints a given update level (after a failure to upgrade), then it correctly gets the previous non tainted UL.
            But then, when asking the backend again, passing the previous UL as parameter (i.e. the good one, the one *before* the just tainted one), then the backend will still serve an answer to upgrade to the tainted level.

            From a user standpoint, this will surface through the following behavior:
            * Imagine we are on UL 50.
            * UL51 is pushed
            * Instance updates to UL51 and is unable to restart
            * Instance informs the backend UL51 was a failure (aka _taints_ it)
            * Instance goes back to UL50 from UL51
            * Instance updates to UL51, fails...
            * Loop back to a few lines above, and so on


            h3. Expected behavior
            Once _per-instance tainted_, a given UL should never be proposed again to a given instance.
            batmat Baptiste Mathus made changes -
            Description _NOTE: I've added a [reproducer test case|https://github.com/jenkins-infra/evergreen/pull/316/commits/bf5a61c5312642efecb3007cf61a8903d5e59c08], and this issue matches my experience from testing it locally when I see the rollback work, and then still re-upgrade to the tainted level after going away from it previously. But I'm still not 100% sure of the diagnosis, so keep this in mind below._

            h3. Problem statement
            When an instance taints a given update level (after a failure to upgrade), then it correctly gets the previous non tainted UL.
            But then, when asking the backend again, passing the previous UL as parameter (i.e. the good one, the one *before* the just tainted one), then the backend will still serve an answer to upgrade to the tainted level.

            From a user standpoint, this will surface through the following behavior:
            * Imagine we are on UL 50.
            * UL51 is pushed
            * Instance updates to UL51 and is unable to restart
            * Instance informs the backend UL51 was a failure (aka _taints_ it)
            * Instance goes back to UL50 from UL51
            * Instance updates to UL51, fails...
            * Loop back to a few lines above, and so on


            h3. Expected behavior
            Once _per-instance tainted_, a given UL should never be proposed again to a given instance.
            _NOTE: I've added a [reproducer test case|https://github.com/jenkins-infra/evergreen/pull/316/commits/bf5a61c5312642efecb3007cf61a8903d5e59c08], and this issue matches my experience from testing it locally when I see the rollback work, and then still re-upgrade to the tainted level after going away from it previously. But I'm still not 100% sure of the diagnosis, so keep this in mind below._

            h3. Problem statement
            When an instance taints a given update level (after a failure to upgrade), then it correctly gets the previous non tainted UL.
            But then, when asking the backend again, passing the previous UL as parameter (i.e. the good one, the one *before* the just tainted one), then the backend will still serve an answer to upgrade to the tainted level.

            From a user standpoint, this will surface through the following behavior:
            * Imagine we are on UL 50.
            * UL51 is pushed
            * Instance updates to UL51 and is unable to restart
            * Instance informs the backend UL51 was a failure (aka _taints_ it)
            * Instance goes back to UL50 from UL51
            * Instance updates to UL51, fails...
            * Loop back to a few lines above, and so on. I.e. the instance will do UL50->UL51->UL50->UL51, etc.


            h3. Expected behavior
            Once _per-instance tainted_, a given UL should never be proposed again to a given instance.
            batmat Baptiste Mathus made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            batmat Baptiste Mathus made changes -
            Rank Ranked higher
            batmat Baptiste Mathus made changes -
            Remote Link This issue links to "evergreen PR (Web Link)" [ 21913 ]
            batmat Baptiste Mathus made changes -
            Status In Review [ 10005 ] In Progress [ 3 ]
            batmat Baptiste Mathus made changes -
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Resolved [ 5 ]

              People

              Assignee:
              batmat Baptiste Mathus
              Reporter:
              batmat Baptiste Mathus
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: