This issue has now been resolved, but I would like to understand how the credentials work "underneath" and the root cause, so that we do not run into this issue anymore.

While running a Jenkins Pipeline job on September 2, 2021, there was an issue exposed within the job: 

secrets.sh -inkey sdma-ng-private.pem
unable to load signing key file
<removed>:error:<removed>:PEM routines:PEM_read_bio:no start line:pem_lib.c:703:Expecting: ANY PRIVATE KEY
total 48

After investigating, there were a few things determined

1. the Jenkins Credential referenced, was actually missing.  There is a secret text used for this pipeline, which was stored in the Jenkins credentials.  I created this jenkins credential on August 19, 2021 with an automatically generated ID of "a1d1b042-2da3-49ec-a4b4-b26fcbfa2e04". When looking for this file on Sept 2, it was no longer found.  To resolve the issue, I created another Jenkins Credential, "31bbb21b-a8d2-45f9-a0c7-1e99404b121e" with a label of  SDMA NG AES_ENCRYPTION_PWD.
2. Another Jenkins Credential, which was a file, seemed to be corrupt.  This pem file was suppose to reference a SDMA certificate with "----BEGIN PRIVATE KEY---- "<encoded certificate> "----END PRIVATE KEY----".  However, it was corrupt in that it had garbage characters through the file, which was validated in the Jenkins job workspace.  This issue was corrected by downloading the latest pem from the shared 1password vault and reuploading to the Jenkins credential.

Can someone explain how this could have happened?

Is this something that could only have been a manual deletion or corruption?

Could a restart of Jenkins have done this?

is there space used by Jenkins credentials that could be corrupted? 

Is there anything that I can check?

Looks like I am using Jenkins Credential plugin version 2.3.15