This began happening in my environment (Jenkins 2.222.3, workflow-api-plugin 2.40, splunk-devops 1.9.3, splunk-devops-extend 1.9.3) after installing the splunk-devops-extend 1.9.3 plugin.
I've done a bit of debugging on this, and it appears that log lines within a run's console log can be corrupted. ConsoleNote.java attempts to decode base64 data sitting between an ANSI escape PREAMBLE and POSTAMBLE. I get "Illegal base64 character" exception when a log line contains the PREAMBLE but not the POSTAMBLE.
Like I mentioned, this only started happening after we installed the "splunk-devops-extend" plugin in our environment, which allowed us to start forwarding pipeline step logs to Splunk.
I can find instances of the bad log lines by greping the original console log (on the jenkins server), looking for lines that contain the preamble, but not the postamble.
cat some/build/console/log | grep '\[8mha' |grep -v '\[0m'|wc -l
Here's an example of what I see in the logs. Please understand that I had to replace the actual base64 encoded string with "base64data" ---- I don't know exactly what is in there, so I don't want to expose anything.
ESC[8mha:////4[2020-06-02T13:57:07.191Z] 09:57:06.854 stuff is logged here
base64database64database64database64database64database64database64database64data+j[2020-06-02T13:57:07.196Z] 09:57:06.856 more stuff is logged here
base64database64database64dataESC[0m[Pipeline] // timeout