Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46046

Publisher with Standard Mode / Relative Threshold does not work for more than one sourceDataFiles

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • performance-plugin
    • None
    • Jenkins v2.70
      Performance Plugin v3.3-SNAPSHOT
      Windows Server 2012
      Oracle Java 1.8.0_66

    Description

      How to reproduce:

      In either a Freestyle or Pipeline job,

      1. Specify 2 sourceDataFiles, say JMeter csv files: a.csv;b.csv
      2. Run Publisher (this will create serialized versions of the parsed files).
      3. Run once more for a relative comparison.
      4. Run once again but with files in different order: b.csv;a.csv

      This will produce output similar to the following.

      One build will have a relative comparison for just one file's samples, whereas the other build will have no comparison (as it tries to compare a.csv with b.csv), as below.

      [Pipeline] step
       Performance: Recording JMeterCsv reports 'b.csv'
       Performance: Parsing JMeter report file 'E:\Jenkins\jobs\test\builds\30\performance-reports\JMeterCSV\b.csv'.
       Performance: Recording JMeterCsv reports 'a.csv'
       Performance: Parsing JMeter report file 'E:\Jenkins\jobs\test\builds\30\performance-reports\JMeterCSV\a.csv'.
       Performance: Percentage of relative difference outside -10.0 to +10.0 % sets the build as failure
       Performance: Percentage of relative difference outside -5.0 to +5.0 % sets the build as unstable
       Performance: Recording JMeterCsv reports 'b.csv'
       Performance: Recording JMeterCsv reports 'a.csv'
      Comparison build no. - 29 and 30 using 90 Percentile response time
      PrevBuildURI CurrentBuildURI PrevBuildURI90% CurrentBuildURI90% RelativeDiff RelativeDiffPercentage 
       a a 900 900 0.0 0.0
       [Pipeline] }
       [Pipeline] // node
       [Pipeline] End of Pipeline
       Finished: SUCCESS
      

       

      [Pipeline] step
      Performance: Recording JMeterCsv reports 'a.csv'
      Performance: Parsing JMeter report file 'E:\Jenkins\jobs\test\builds\29\performance-reports\JMeterCSV\a.csv'.
      Performance: Recording JMeterCsv reports 'b.csv'
      Performance: Parsing JMeter report file 'E:\Jenkins\jobs\test\builds\29\performance-reports\JMeterCSV\b.csv'.
      Performance: Percentage of relative difference outside -10.0 to +10.0 % sets the build as failure
      Performance: Percentage of relative difference outside -5.0 to +5.0 % sets the build as unstable
      Performance: Recording JMeterCsv reports 'a.csv'
      Performance: Recording JMeterCsv reports 'b.csv'
      
      Comparison build no. - 28 and 29 using 90 Percentile response time
      
      
      PrevBuildURI	CurrentBuildURI		PrevBuildURI90%		CurrentBuildURI90%	RelativeDiff	RelativeDiffPercentage 
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] End of Pipeline
      Finished: SUCCESS

       

      Sample Jenkinsfile:

      node {
          ['a','b'].each { 
              sh "echo timeStamp,elapsed,label,responseCode,success > ${it}.csv"
              10.times { i -> sh "echo ${1500000000000+i*1000},${i*100},$it,200,true >> ${it}.csv" }
          }
      
          perfReport([
              sourceDataFiles: 'a.csv;b.csv',
              modeEvaluation: false,
              modeOfThreshold: true, 
              failBuildIfNoResultFile: true, 
              relativeUnstableThresholdNegative: 5,
              relativeUnstableThresholdPositive: 5,
              relativeFailedThresholdNegative: 10,
              relativeFailedThresholdPositive: 10,
              compareBuildPrevious: true,
              configType: 'PRT'
          ])
      }

       

      Attachments

        Activity

          People

            undera Andrey Pokhilko
            tilln Till Neunast
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: