If your github app private key contains these 3 letters - "RSA", Jenkins wrongly identifies it as PKCS#1 format key and throws the below error.
“Private key must be a PKCS#8 formatted string, to convert it from PKCS#1 use: openssl pkcs8 -topk8 -inform PEM -outform PEM -in current-key.pem -out new-key.pem -nocrypt”

The problem is with the code https://github.com/jenkinsci/github-branch-source-plugin/blob/master/src/main/java/org/jenkinsci/plugins/github_branch_source/JwtHelper.java#L63.
I believe the code should either check for the complete string “BEGIN RSA PRIVATE KEY” or just the RSA word with spaces - “ RSA “.