Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61452

Illegal base64 character in FileLogStorage$1.writeHtmlTo(FileLogStorage.java:203)

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • core
    • workflow-api-plugin 2.36

      Jenkins 2.222.3, workflow-api-plugin 2.40, splunk-devops 1.9.3, splunk-devops-extend 1.9.3
    • 2.420, 2.414.2

      sporadic console error

      java.lang.IllegalArgumentException: Illegal base64 character 5b
      	at java.util.Base64$Decoder.decode0(Base64.java:714)
      	at java.util.Base64$Decoder.decode(Base64.java:526)
      	at hudson.util.UnbufferedBase64InputStream.read(UnbufferedBase64InputStream.java:41)
      	at hudson.util.UnbufferedBase64InputStream.read(UnbufferedBase64InputStream.java:53)
      	at java.io.DataInputStream.readFully(DataInputStream.java:195)
      	at java.io.DataInputStream.readFully(DataInputStream.java:169)
      	at hudson.console.ConsoleNote.readFrom(ConsoleNote.java:248)
      	at hudson.console.ConsoleAnnotationOutputStream.eol(ConsoleAnnotationOutputStream.java:111)
      	at hudson.console.LineTransformationOutputStream.eol(LineTransformationOutputStream.java:60)
      	at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:56)
      	at java.io.FilterOutputStream.write(FilterOutputStream.java:77)
      	at org.jenkinsci.plugins.workflow.log.FileLogStorage$1$1.write(FileLogStorage.java:238)
      	at java.io.FilterOutputStream.write(FilterOutputStream.java:125)
      	at org.apache.commons.io.output.ProxyOutputStream.write(ProxyOutputStream.java:89)
      	at org.kohsuke.stapler.framework.io.LargeText$HeadMark.moveTo(LargeText.java:314)
      	at org.kohsuke.stapler.framework.io.LargeText.writeLogTo(LargeText.java:229)
      	at hudson.console.AnnotatedLargeText.writeRawLogTo(AnnotatedLargeText.java:168)
      	at org.jenkinsci.plugins.workflow.log.FileLogStorage$1.writeHtmlTo(FileLogStorage.java:203)
      	at hudson.console.AnnotatedLargeText.writeLogTo(AnnotatedLargeText.java:147)
      	at org.kohsuke.stapler.framework.io.LargeText.doProgressText(LargeText.java:264)
      	at hudson.console.AnnotatedLargeText.doProgressiveHtml(AnnotatedLargeText.java:95)
      

          [JENKINS-61452] Illegal base64 character in FileLogStorage$1.writeHtmlTo(FileLogStorage.java:203)

          Ted Xiao created issue -

          Ted Xiao added a comment -

          The console page will go back to normal if the job completes, either normally or aborted. probably a caching bug in the FileLogStorage

          Ted Xiao added a comment - The console page will go back to normal if the job completes, either normally or aborted. probably a caching bug in the FileLogStorage
          Mike Ryan made changes -
          Environment Original: workflow-api-plugin 2.36 New: workflow-api-plugin 2.36

          Jenkins 2.222.3, workflow-api-plugin 2.40, splunk-devops 1.9.3, splunk-devops-extend 1.9.3

          Mike Ryan added a comment -

          This began happening in my environment (Jenkins 2.222.3, workflow-api-plugin 2.40, splunk-devops 1.9.3, splunk-devops-extend 1.9.3) after installing the splunk-devops-extend 1.9.3 plugin. 

           

          I've done a bit of debugging on this, and it appears that log lines within a run's console log can be corrupted. ConsoleNote.java attempts to decode base64 data sitting between an ANSI escape PREAMBLE and POSTAMBLE. I get "Illegal base64 character" exception when a log line contains the PREAMBLE but not the POSTAMBLE.

          Like I mentioned, this only started happening after we installed the "splunk-devops-extend" plugin in our environment, which allowed us to start forwarding pipeline step logs to Splunk.

          I can find instances of the bad log lines by greping the original console log (on the jenkins server), looking for lines that contain the preamble, but not the postamble.

          cat some/build/console/log | grep '\[8mha' |grep -v '\[0m'|wc -l 

          .

          Here's an example of what I see in the logs. Please understand that I had to replace the actual base64 encoded string with "base64data" ---- I don't know exactly what is in there, so I don't want to expose anything.

          ESC[8mha:////4[2020-06-02T13:57:07.191Z] 09:57:06.854 stuff is logged here 
          base64database64database64database64database64database64database64database64data+j[2020-06-02T13:57:07.196Z] 09:57:06.856 more stuff is logged here 
          base64database64database64dataESC[0m[Pipeline] // timeout
          

           

          Mike Ryan added a comment - This began happening in my environment (Jenkins 2.222.3, workflow-api-plugin 2.40, splunk-devops 1.9.3, splunk-devops-extend 1.9.3) after installing the splunk-devops-extend 1.9.3 plugin.    I've done a bit of debugging on this, and it appears that log lines within a run's console log can be corrupted. ConsoleNote.java attempts to decode base64 data sitting between an ANSI escape PREAMBLE and POSTAMBLE . I get "Illegal base64 character" exception when a log line contains the PREAMBLE but not the POSTAMBLE. Like I mentioned, this only started happening after we installed the "splunk-devops-extend" plugin in our environment, which allowed us to start forwarding pipeline step logs to Splunk. I can find instances of the bad log lines by greping the original console log (on the jenkins server), looking for lines that contain the preamble, but not the postamble. cat some/build/console/log | grep '\[8mha' |grep -v '\[0m'|wc -l . Here's an example of what I see in the logs. Please understand that I had to replace the actual base64 encoded string with "base64data" ---- I don't know exactly what is in there, so I don't want to expose anything. ESC[8mha:////4[2020-06-02T13:57:07.191Z] 09:57:06.854 stuff is logged here base64database64database64database64database64database64database64database64data+j[2020-06-02T13:57:07.196Z] 09:57:06.856 more stuff is logged here base64database64database64dataESC[0m[Pipeline] // timeout  
          Mike Ryan made changes -
          Component/s New: splunk-devops-plugin [ 21829 ]

          Ted Xiao added a comment -

          mikeryan_tan I think you can reproduce the issue just with workflow-api-plugin 2.40, https://github.com/jenkinsci/workflow-api-plugin/blob/master/src/main/java/org/jenkinsci/plugins/workflow/log/DelayBufferedOutputStream.java  may not flush data in-time and caused partial base64 data written to disk. 

          splunk-devops-extend depends on workflow-api-plugin , I guess when you install the plugin, Jenkins probably upgraded workflow-api-plugin to latest version.

           

          Ted Xiao added a comment - mikeryan_tan I think you can reproduce the issue just with workflow-api-plugin 2.40,  https://github.com/jenkinsci/workflow-api-plugin/blob/master/src/main/java/org/jenkinsci/plugins/workflow/log/DelayBufferedOutputStream.java   may not flush data in-time and caused partial base64 data written to disk.  splunk-devops-extend depends on workflow-api-plugin , I guess when you install the plugin, Jenkins probably upgraded workflow-api-plugin to latest version.  

          Mike Ryan added a comment -

          No, we have definitely had workflow-api-plug-in 2.40 installed for some time, as it that was released in February. splunk-devops-extend was the only plugin that we had installed on May 29th, and we started seeing the exceptions shortly after the restart.

          Perhaps forwarding console logs to splunk introduces enough of a delay into the system to make the buffer flushing issue present more often?

          Mike Ryan added a comment - No, we have definitely had workflow-api-plug-in 2.40 installed for some time, as it that was released in February. splunk-devops-extend was the only plugin that we had installed on May 29th, and we started seeing the exceptions shortly after the restart. Perhaps forwarding console logs to splunk introduces enough of a delay into the system to make the buffer flushing issue present more often?

          Ted Xiao added a comment -

          I can easily reproduce the issue without any splunk plugin installed.
          Assume you have 2 jobs, one is trigger job, and the other one is downstream job called my-downstream

          // downstream job  my-downstream config

          properties([
           parameters([string(defaultValue: '1', description: '', name: 'counter', trim: false)])
          ])
          echo "${env.counter}"
          sleep 600 

          // trigger job config

          import hudson.console.ModelHyperlinkNote
          
          def kickJobs(count){
          	def runs=[:]
             	for(int i=0; i<count; i++){
             	    def jobId = i+1
          		def quietPeriod = 11*jobId.intdiv(9) + jobId%9
          		runs["run_$i"] = {
          		   echo "running $jobId"
          	       def userJob=build job: "my-downstream", parameters:[new StringParameterValue("counter", "job_$jobId")],
          	       quietPeriod: quietPeriod
          	       def consoleMsg = ModelHyperlinkNote.encodeTo("/${userJob.rawBuild.url}console", "Console")
          	       echo consoleMsg
          		}
              }
              parallel runs
              return true
          }
          
          node{
              timestamps{
                  ansiColor{
                    kickJobs(1000)
                  }
              }
          }
          

          Please first run the downstream job manually to update job property so it can be run concurrently by the trigger job, then start the trigger job and wait 1-2 minutes, then you will see the error on console

          the offending line in log looks like

          ^[[8mha:^[[8mha:////4OoLEwBdjfr/tkaLP4ecvUmMRmZlK////4NgeI/uDMeiCxyag5umRs/gNaD0jg/WoUHqu9mv3ESxOAAAAqh+LCAAAAAAAAP9b85aBtbiIQTGjNKU4P08vOT+vOD8nVc83PyU1x6OyILUoJzMv2y+/JJUBAhiZGBgqihhk0NSjKDWzXb3RdlLBUSYGJk8GtpzUvPSSDB8G5tKinBIGIZ+sxLJE/ZzEvHT94JKizLx0a6BxUmjGOUNodHsLgAypEgZF/az8JH2X1KTSdDArt1I3Jb88r7ikKDUxV9/EX0cYK7AAV8Tp3+zAAAAqh+LCAAAAAAAAP9b85aBtbiIQTGjNKU4P08vOT+vOD8nVc83FgfAL+zp+LWAAAA^[[0mDebug
          

          Ted Xiao added a comment - I can easily reproduce the issue without any splunk plugin installed. Assume you have 2 jobs, one is trigger job, and the other one is downstream job called  my-downstream // downstream job  my-downstream config properties([ parameters([string(defaultValue: '1' , description: '', name: ' counter', trim: false )]) ]) echo "${env.counter}" sleep 600 // trigger job config import hudson.console.ModelHyperlinkNote def kickJobs(count){ def runs=[:] for ( int i=0; i<count; i++){ def jobId = i+1 def quietPeriod = 11*jobId.intdiv(9) + jobId%9 runs[ "run_$i" ] = { echo "running $jobId" def userJob=build job: "my-downstream" , parameters:[ new StringParameterValue( "counter" , "job_$jobId" )], quietPeriod: quietPeriod def consoleMsg = ModelHyperlinkNote.encodeTo( "/${userJob.rawBuild.url}console" , "Console" ) echo consoleMsg } } parallel runs return true } node{ timestamps{ ansiColor{ kickJobs(1000) } } } Please first run the downstream job manually to update job property so it can be run concurrently by the trigger job, then start the trigger job and wait 1-2 minutes, then you will see the error on console the offending line in log looks like ^[[8mha:^[[8mha: ////4OoLEwBdjfr/tkaLP4ecvUmMRmZlK////4NgeI/uDMeiCxyag5umRs/gNaD0jg/WoUHqu9mv3ESxOAAAAqh+LCAAAAAAAAP9b85aBtbiIQTGjNKU4P08vOT+vOD8nVc83PyU1x6OyILUoJzMv2y+/JJUBAhiZGBgqihhk0NSjKDWzXb3RdlLBUSYGJk8GtpzUvPSSDB8G5tKinBIGIZ+sxLJE/ZzEvHT94JKizLx0a6BxUmjGOUNodHsLgAypEgZF/az8JH2X1KTSdDArt1I3Jb88r7ikKDUxV9/EX0cYK7AAV8Tp3+zAAAAqh+LCAAAAAAAAP9b85aBtbiIQTGjNKU4P08vOT+vOD8nVc83FgfAL+zp+LWAAAA^[[0mDebug

          Ted Xiao added a comment -

          jglick any suggestion ?

          Ted Xiao added a comment - jglick any suggestion ?
          Ted Xiao made changes -
          Description Original: sporadic console error
          {code}
          java.lang.IllegalArgumentException: Illegal base64 character 5b
          at java.util.Base64$Decoder.decode0(Base64.java:714)
          at java.util.Base64$Decoder.decode(Base64.java:526)
          at hudson.util.UnbufferedBase64InputStream.read(UnbufferedBase64InputStream.java:41)
          at hudson.util.UnbufferedBase64InputStream.read(UnbufferedBase64InputStream.java:53)
          at java.io.DataInputStream.readFully(DataInputStream.java:195)
          at java.io.DataInputStream.readFully(DataInputStream.java:169)
          at hudson.console.ConsoleNote.readFrom(ConsoleNote.java:248)
          at hudson.console.ConsoleAnnotationOutputStream.eol(ConsoleAnnotationOutputStream.java:111)
          at hudson.console.LineTransformationOutputStream.eol(LineTransformationOutputStream.java:60)
          at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:56)
          at java.io.FilterOutputStream.write(FilterOutputStream.java:77)
          at org.jenkinsci.plugins.workflow.log.FileLogStorage$1$1.write(FileLogStorage.java:238)
          at java.io.FilterOutputStream.write(FilterOutputStream.java:125)
          at org.apache.commons.io.output.ProxyOutputStream.write(ProxyOutputStream.java:89)
          at org.kohsuke.stapler.framework.io.LargeText$HeadMark.moveTo(LargeText.java:314)
          at org.kohsuke.stapler.framework.io.LargeText.writeLogTo(LargeText.java:229)
          at hudson.console.AnnotatedLargeText.writeRawLogTo(AnnotatedLargeText.java:168)
          at org.jenkinsci.plugins.workflow.log.FileLogStorage$1.writeHtmlTo(FileLogStorage.java:203)
          at hudson.console.AnnotatedLargeText.writeLogTo(AnnotatedLargeText.java:147)
          at org.kohsuke.stapler.framework.io.LargeText.doProgressText(LargeText.java:264)
          at hudson.console.AnnotatedLargeText.doProgressiveHtml(AnnotatedLargeText.java:95)
          {code}

          it is fine to view console text via /consoleText URL which is using
          {code}
           public void doConsoleText(StaplerRequest req, StaplerResponse rsp) throws IOException {
                  rsp.setContentType("text/plain;charset=UTF-8");
                  try (OutputStream os = rsp.getCompressedOutputStream(req)) {
                      writeLogTo(getLogText()::writeLogTo, os);
                  }
              }
          {code}

          suspect a bug in FileLogStorage writeHtmlTo method
          New: sporadic console error
          {code}
          java.lang.IllegalArgumentException: Illegal base64 character 5b
          at java.util.Base64$Decoder.decode0(Base64.java:714)
          at java.util.Base64$Decoder.decode(Base64.java:526)
          at hudson.util.UnbufferedBase64InputStream.read(UnbufferedBase64InputStream.java:41)
          at hudson.util.UnbufferedBase64InputStream.read(UnbufferedBase64InputStream.java:53)
          at java.io.DataInputStream.readFully(DataInputStream.java:195)
          at java.io.DataInputStream.readFully(DataInputStream.java:169)
          at hudson.console.ConsoleNote.readFrom(ConsoleNote.java:248)
          at hudson.console.ConsoleAnnotationOutputStream.eol(ConsoleAnnotationOutputStream.java:111)
          at hudson.console.LineTransformationOutputStream.eol(LineTransformationOutputStream.java:60)
          at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:56)
          at java.io.FilterOutputStream.write(FilterOutputStream.java:77)
          at org.jenkinsci.plugins.workflow.log.FileLogStorage$1$1.write(FileLogStorage.java:238)
          at java.io.FilterOutputStream.write(FilterOutputStream.java:125)
          at org.apache.commons.io.output.ProxyOutputStream.write(ProxyOutputStream.java:89)
          at org.kohsuke.stapler.framework.io.LargeText$HeadMark.moveTo(LargeText.java:314)
          at org.kohsuke.stapler.framework.io.LargeText.writeLogTo(LargeText.java:229)
          at hudson.console.AnnotatedLargeText.writeRawLogTo(AnnotatedLargeText.java:168)
          at org.jenkinsci.plugins.workflow.log.FileLogStorage$1.writeHtmlTo(FileLogStorage.java:203)
          at hudson.console.AnnotatedLargeText.writeLogTo(AnnotatedLargeText.java:147)
          at org.kohsuke.stapler.framework.io.LargeText.doProgressText(LargeText.java:264)
          at hudson.console.AnnotatedLargeText.doProgressiveHtml(AnnotatedLargeText.java:95)
          {code}

            jglick Jesse Glick
            fengxx Ted Xiao
            Votes:
            1 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: