Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61452

Illegal base64 character in FileLogStorage$1.writeHtmlTo(FileLogStorage.java:203)

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • core
    • workflow-api-plugin 2.36

      Jenkins 2.222.3, workflow-api-plugin 2.40, splunk-devops 1.9.3, splunk-devops-extend 1.9.3
    • 2.420, 2.414.2

      sporadic console error

      java.lang.IllegalArgumentException: Illegal base64 character 5b
      	at java.util.Base64$Decoder.decode0(Base64.java:714)
      	at java.util.Base64$Decoder.decode(Base64.java:526)
      	at hudson.util.UnbufferedBase64InputStream.read(UnbufferedBase64InputStream.java:41)
      	at hudson.util.UnbufferedBase64InputStream.read(UnbufferedBase64InputStream.java:53)
      	at java.io.DataInputStream.readFully(DataInputStream.java:195)
      	at java.io.DataInputStream.readFully(DataInputStream.java:169)
      	at hudson.console.ConsoleNote.readFrom(ConsoleNote.java:248)
      	at hudson.console.ConsoleAnnotationOutputStream.eol(ConsoleAnnotationOutputStream.java:111)
      	at hudson.console.LineTransformationOutputStream.eol(LineTransformationOutputStream.java:60)
      	at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:56)
      	at java.io.FilterOutputStream.write(FilterOutputStream.java:77)
      	at org.jenkinsci.plugins.workflow.log.FileLogStorage$1$1.write(FileLogStorage.java:238)
      	at java.io.FilterOutputStream.write(FilterOutputStream.java:125)
      	at org.apache.commons.io.output.ProxyOutputStream.write(ProxyOutputStream.java:89)
      	at org.kohsuke.stapler.framework.io.LargeText$HeadMark.moveTo(LargeText.java:314)
      	at org.kohsuke.stapler.framework.io.LargeText.writeLogTo(LargeText.java:229)
      	at hudson.console.AnnotatedLargeText.writeRawLogTo(AnnotatedLargeText.java:168)
      	at org.jenkinsci.plugins.workflow.log.FileLogStorage$1.writeHtmlTo(FileLogStorage.java:203)
      	at hudson.console.AnnotatedLargeText.writeLogTo(AnnotatedLargeText.java:147)
      	at org.kohsuke.stapler.framework.io.LargeText.doProgressText(LargeText.java:264)
      	at hudson.console.AnnotatedLargeText.doProgressiveHtml(AnnotatedLargeText.java:95)
      

          [JENKINS-61452] Illegal base64 character in FileLogStorage$1.writeHtmlTo(FileLogStorage.java:203)

          Jesse Glick added a comment -

          the reproduce step (trigger job config) doesn't require splunk plugin installed

          Understood, but this is not obviously the same issue—could be a bug in Pipeline infrastructure, with echo, with ansicolor, with timestamps, etc.

          Jesse Glick added a comment - the reproduce step (trigger job config) doesn't require splunk plugin installed Understood, but this is not obviously the same issue—could be a bug in Pipeline infrastructure, with echo , with ansicolor , with timestamps , etc.

          Ted Xiao added a comment -

          Agrees that ansicolor/timestamps may have bugs and should be fixed.
          What I observed is that workflow job tolerated the issue before, e.g. workflow-api 2.34 + workflow-job 2.25 + ansicolor 0.5.0 works fine (at least /console page is fine), but it throws error on workflow-api 2.39 + workflow-job 2.37 + ansicolor 0.5.0. Hope it is not too much to ask if we catch error and ignore the bad line in workflow-api

          Ted Xiao added a comment - Agrees that ansicolor/timestamps may have bugs and should be fixed. What I observed is that workflow job tolerated the issue before, e.g. workflow-api 2.34 + workflow-job 2.25 + ansicolor 0.5.0 works fine (at least /console page is fine), but it throws error on workflow-api 2.39 + workflow-job 2.37 + ansicolor 0.5.0. Hope it is not too much to ask if we catch error and ignore the bad line in workflow-api

          Ted Xiao added a comment -

          mikeryan_tan splunk JVM issue will be tracked in JENKINS-62663

          Ted Xiao added a comment - mikeryan_tan splunk JVM issue will be tracked in JENKINS-62663

          Jesse Glick added a comment -

          Have not taken the time yet to reproduce and analyze the situation in the debugger, but

          ignore the bad line in workflow-api

          AFAICT this aspect of the bug is in Jenkins core: malformed Base64 results in an IllegalArgumentException, rather than being suppressed quietly in ConsoleAnnotationOutputStream due to a null return value or IOException from ConsoleNote.readFrom. That would be a bug in UnbufferedBase64InputStream and/or ConsoleNote.

          The deeper cause is of course that the annotations are mangled to begin with, which as previously mentioned might have a similar root cause as previously fixed synchronization bugs.

          Jesse Glick added a comment - Have not taken the time yet to reproduce and analyze the situation in the debugger, but ignore the bad line in workflow-api AFAICT this aspect of the bug is in Jenkins core: malformed Base64 results in an IllegalArgumentException , rather than being suppressed quietly in ConsoleAnnotationOutputStream due to a null return value or IOException from ConsoleNote.readFrom . That would be a bug in UnbufferedBase64InputStream and/or ConsoleNote . The deeper cause is of course that the annotations are mangled to begin with, which as previously mentioned might have a similar root cause as previously fixed synchronization bugs.

          Mike Ryan added a comment -

          Just a heads up: the act of viewing a corrupt log file produced by this bug results in Jenkins leaking a file descriptor: JENKINS-62985

          Mike Ryan added a comment - Just a heads up: the act of viewing a corrupt log file produced by this bug results in Jenkins leaking a file descriptor: JENKINS-62985

          Jesse Glick added a comment -

          The visible symptom (IAE) should have been corrected by https://github.com/jenkinsci/jenkins/pull/4636 in 2.231.

          Jesse Glick added a comment - The visible symptom (IAE) should have been corrected by https://github.com/jenkinsci/jenkins/pull/4636 in 2.231.

          Jesse Glick added a comment -

          …and may have been introduced by https://github.com/jenkinsci/jenkins/pull/4169 in 2.194.

          Jesse Glick added a comment - …and may have been introduced by https://github.com/jenkinsci/jenkins/pull/4169 in 2.194.

          Jesse Glick added a comment -

          Jesse Glick added a comment - …or by https://github.com/jenkinsci/jenkins/pull/2958 in 2.73.

          Florian Ramillien added a comment - - edited

          We just see that we got this problem on our Jenkin 2.332.4 too.

          What we see:

          • Logs displayed by stage are always well displayed (BlueOcean).
          • Logs displayed in full consoleText can be truncated or with two lines mixed.
          • Logs available on BlueOcean "Artifacts" page may return a HTTP 500 error with "Illegal base64 character -1"

          We got this error since a long time, but didn't take care about it because we generally use only stage view, not full log view.

          Difference between log by stages and full log look related to 'log-index'. If 'log-index' is used, lines can be mixed because the index is based on byte adresses and not line numbers. Doing this, even if two lines are mixed, index is accurate. Does the full log view use this index to display the full log ?

          Or maybe should we catch this base64 error to be able to display the full log ? In `ConsoleNote.skip()` ?

          Stack trace in our repro case:

          Caused by: java.io.IOException: Illegal base64 character -1
              at java.base/java.util.Base64$DecInputStream.read(Unknown Source)
              at java.base/java.io.DataInputStream.read(Unknown Source)
              at java.base/java.io.DataInputStream.readFully(Unknown Source)
              at hudson.util.IOUtils.skip(IOUtils.java:91)
              at hudson.console.ConsoleNote.skip(ConsoleNote.java:311)
              at hudson.console.PlainTextConsoleOutputStream.eol(PlainTextConsoleOutputStream.java:67)

          Florian Ramillien added a comment - - edited We just see that we got this problem on our Jenkin 2.332.4 too. What we see: Logs displayed by stage are always well displayed (BlueOcean). Logs displayed in full consoleText can be truncated or with two lines mixed. Logs available on BlueOcean "Artifacts" page may return a HTTP 500 error with "Illegal base64 character -1" We got this error since a long time, but didn't take care about it because we generally use only stage view, not full log view. Difference between log by stages and full log look related to 'log-index'. If 'log-index' is used, lines can be mixed because the index is based on byte adresses and not line numbers. Doing this, even if two lines are mixed, index is accurate. Does the full log view use this index to display the full log ? Or maybe should we catch this base64 error to be able to display the full log ? In `ConsoleNote.skip()` ? Stack trace in our repro case: Caused by: java.io.IOException: Illegal base64 character -1     at java.base/java.util.Base64$DecInputStream.read(Unknown Source)     at java.base/java.io.DataInputStream.read(Unknown Source)     at java.base/java.io.DataInputStream.readFully(Unknown Source)     at hudson.util.IOUtils.skip(IOUtils.java:91)     at hudson.console.ConsoleNote.skip(ConsoleNote.java:311)     at hudson.console.PlainTextConsoleOutputStream.eol(PlainTextConsoleOutputStream.java:67)

          Anna added a comment - - edited

          We also got this problem on Jenkins 2.361.1.

          It seems like in full console output two lines are getting mixed.
          As a result, we get:

          java.io.IOException: Illegal base64 character -1
          09:27:09  	at java.base/java.util.Base64$DecInputStream.read(Base64.java:1016)
          09:27:09  	at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
          09:27:09  	at java.base/java.io.DataInputStream.readFully(DataInputStream.java:200)
          09:27:09  	at hudson.util.IOUtils.skip(IOUtils.java:90)
          09:27:09  	at hudson.console.ConsoleNote.skip(ConsoleNote.java:311)
          09:27:09  	at hudson.console.PlainTextConsoleOutputStream.eol(PlainTextConsoleOutputStream.java:67)
          09:27:09  	at hudson.console.LineTransformationOutputStream.eol(LineTransformationOutputStream.java:61)
          09:27:09  	at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:57)
          09:27:09  	at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:75)
          09:27:09  	at org.apache.commons.io.output.ProxyOutputStream.write(ProxyOutputStream.java:92)
          09:27:09  	at org.kohsuke.stapler.framework.io.LargeText$HeadMark.moveTo(LargeText.java:329)
          09:27:09  	at org.kohsuke.stapler.framework.io.LargeText.writeLogTo(LargeText.java:244)
          09:27:09  	at hudson.console.AnnotatedLargeText.writeLogTo(AnnotatedLargeText.java:164) 

          Anna added a comment - - edited We also got this problem on Jenkins 2.361.1. It seems like in full console output two lines are getting mixed. As a result, we get: java.io.IOException: Illegal base64 character -1 09:27:09 at java.base/java.util.Base64$DecInputStream.read(Base64.java:1016) 09:27:09 at java.base/java.io.DataInputStream.read(DataInputStream.java:149) 09:27:09 at java.base/java.io.DataInputStream.readFully(DataInputStream.java:200) 09:27:09 at hudson.util.IOUtils.skip(IOUtils.java:90) 09:27:09 at hudson.console.ConsoleNote.skip(ConsoleNote.java:311) 09:27:09 at hudson.console.PlainTextConsoleOutputStream.eol(PlainTextConsoleOutputStream.java:67) 09:27:09 at hudson.console.LineTransformationOutputStream.eol(LineTransformationOutputStream.java:61) 09:27:09 at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:57) 09:27:09 at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:75) 09:27:09 at org.apache.commons.io.output.ProxyOutputStream.write(ProxyOutputStream.java:92) 09:27:09 at org.kohsuke.stapler.framework.io.LargeText$HeadMark.moveTo(LargeText.java:329) 09:27:09 at org.kohsuke.stapler.framework.io.LargeText.writeLogTo(LargeText.java:244) 09:27:09 at hudson.console.AnnotatedLargeText.writeLogTo(AnnotatedLargeText.java:164)

            jglick Jesse Glick
            fengxx Ted Xiao
            Votes:
            1 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: