Can you give us more details?

The last few jobs and their timestamps, their results, and whether a new build
is being scheduled or not. Also, please paste the cc.xml contents

> The last few jobs and their timestamps,
> their results, and whether a new build
> is being scheduled or not.

I do not know which jobs were run or wether the jobs that were running after the
job in question succeeded or not. Hudson has been restarted since and its output
was not logged to a file. For us the problem persists however, it is not a
temporary phenomenon.

> Also, please paste the cc.xml contents

From http://hudson.ramfelt.se/cc.xml, note the second last project "XBMC TV",
which has been listed with lastBuildStatus="Failure":

<Projects>
<Project activity="Sleeping" lastBuildTime="2007-08-21T20:30:53Z"
webUrl="http://hudson.ramfelt.se/job/Airduel/" lastBuildLabel="10"
lastBuildStatus="Success" name="Airduel"/>

<Project activity="Sleeping" lastBuildTime="2008-02-10T23:03:14Z"
webUrl="http://hudson.ramfelt.se/job/Hudson/" lastBuildLabel="357"
lastBuildStatus="Success" name="Hudson"/>

<Project activity="Sleeping" lastBuildTime="2008-01-17T13:24:59Z"
webUrl="http://hudson.ramfelt.se/job/Hudson free style/" lastBuildLabel="11"
lastBuildStatus="Success" name="Hudson free style"/>

<Project activity="Sleeping" lastBuildTime="2008-02-11T07:01:58Z"
webUrl="http://hudson.ramfelt.se/job/NMaven/" lastBuildLabel="45"
lastBuildStatus="Failure" name="NMaven"/>

<Project activity="Sleeping" lastBuildTime="2008-01-10T07:35:21Z"
webUrl="http://hudson.ramfelt.se/job/NMaven maven job/" lastBuildLabel="25"
lastBuildStatus="Failure" name="NMaven maven job"/>

<Project activity="Sleeping" lastBuildTime="2008-01-29T18:41:52Z"
webUrl="http://hudson.ramfelt.se/job/NUnit 2.0/" lastBuildLabel="5"
lastBuildStatus="Failure" name="NUnit 2.0"/>

<Project activity="Sleeping" lastBuildTime="2007-11-20T20:35:07Z"
webUrl="http://hudson.ramfelt.se/job/XBMC Test/" lastBuildLabel="4"
lastBuildStatus="Success" name="XBMC Test"/>

<Project activity="Sleeping" lastBuildTime="2007-11-20T20:36:21Z"
webUrl="http://hudson.ramfelt.se/job/XBMC TV/" lastBuildLabel="22"
lastBuildStatus="Failure" name="XBMC TV"/>

<Project activity="Sleeping" lastBuildTime="2007-12-30T21:38:24Z"
webUrl="http://hudson.ramfelt.se/job/XBMC TV - Verification/"
lastBuildLabel="11" lastBuildStatus="Success" name="XBMC TV - Verification"/>
</Projects>

Here is the project information from the front page (http://hudson.ramfelt.se/),
note the "XBMC TV" project (second last, again) to be listed as "Unstable":

Success
100%
W Description %
Test Result: 0 tests failing out of a total of 40 tests. 100
Build stability: No recent builds failed. 100
Airduel 5 months 23 days (#10) 6 months 25 days (#1) 26 seconds

Success
100%
W Description %
Test Result: 0 tests failing out of a total of 189 tests. 100
Build stability: No recent builds failed. 100
Hudson 9 hours 48 minutes (#357) 4 days 9 hours (#345) 1 hour 21 minutes

Success
79%
W Description %
Build stability: 1 out of the last 5 builds failed. 79
Test Result: 0 tests failing out of a total of 185 tests. 100
Hudson free style 24 days (#11) 24 days (#10) 13 minutes

Unstable
91%
W Description %
Test Result: 3 tests failing out of a total of 35 tests. 91
Build stability: No recent builds failed. 100
NMaven 1 hour 49 minutes (#45) 6 days 13 hours (#35) 18 minutes

Failed
79%
W Description %
Build stability: 1 out of the last 5 builds failed. 79
Test Result: 0 tests failing out of a total of 17 tests. 100
NMaven maven job 1 month 7 days (#23) 1 month 2 days (#25) 2 minutes 59
seconds

Failed
74%
W Description %
Build stability: 1 out of the last 4 builds failed. 74
Test Result: 12 tests failing out of a total of 1173 tests. 98
NUnit 2.0 15 days (#4) 12 days (#5) 9 minutes 45 seconds

Success
24%
W Description %
Build stability: 3 out of the last 4 builds failed. 24
XBMC Test 2 months 22 days (#4) 2 months 22 days (#3) 1 minute 53 seconds

Unstable
33%
W Description %
Number of pylint violations is 804 33
Test Result: 56 tests failing out of a total of 130 tests. 56
Build stability: 1 out of the last 5 builds failed. 79
XBMC TV 2 months 22 days (#22) 5 months 15 days (#16) 6 minutes 39 seconds

Success
79%
W Description %
Build stability: 1 out of the last 5 builds failed. 79
Test Result: 0 tests failing out of a total of 2 tests. 100
XBMC TV - Verification 1 month 12 days (#11) 6 months 18 days (#7) 59 seconds

Looking at Functions.java:

/**

• Converts the Hudson build status to CruiseControl build status,
• which is either Success, Failure, Exception, or Unknown.
  */
  public static String toCCStatus(Item i) {
  if (i instanceof Job)
  Unknown macro: { Job j = (Job) i; switch (j.getIconColor().noAnime()) { case ABORTED: case RED: case YELLOW: return "Failure"; case BLUE: return "Success"; case DISABLED: case GREY: return "Unknown"; } }

  return "Unknown";
  }

Given this it seems that an unstable build is considered a failure. When Hudson
considers a build successful but unstable when it has only some tests failing
maybe it is "more correct" to interpret it as a "Success" in cc.xml, too.

Created an attachment (id=210)
Patch to consider unstable builds to be successful builds in case of "cc.xml"

Given that cc.xml doesn't have a state that directly corresponds to Hudson's
"unstable" state, we have to make a choice how this unstable state in Hudson is
mapped to a cc.xml state.

The current code makes a choice to map this to a failure. Given how other tools
operate, such as Maven, I think it is correct to classify test failures as a
failure.

If we report this as a success, people will not notice test regressions. That
defeats the purpose of Hudson.

How about code coverage? Our Jenkins instance is configured to mark a build as unstable if cove coverage goes down, which happens very often, especially because is measured using two decimals, so just removing one line of code somewhere can make the build unstable, and even worse to display it as a failure in the all cc.xml consumers.

How can we fix this?

kohsuke, now after 9 years, could we revise that decision and start to expose this information inside cc.xml files? There is nothing preventing us from exposing the extended status as a new property.

The reality is that for big projects it is normal to have unstable builds because you always have bugs that are not sorted. Also this practice is important for test driven development, where we are supposed to add tests for bugs before fixing them. Fixing a bug could take, well, years or forever.