-
Bug
-
Resolution: Incomplete
-
Major
-
v 2.129
My Jenkins implementation uses 'publish-over-ssh' plugin to do various simple tasks on hosts that are deployment targets, such as stop/start the app server, clean out log files etc. The configurations for that plugin were merely sets of UNIX commands such as :
service foo start
or
cd log_dir rm -rf *
My Jenkins worked fine until I was to upgrade from v2.100 to v2.101. When I upgraded to 2.101, I encountered that the upgrade broke a lot of jobs, in that when I would go to the job config, I would get error messages (can't remember which type of jobs). I reverted to 2.100 and they all worked again. So, for a couple of months, I was scared to upgrade Jenkins because I thought your software upgrades were breaking my implementation, basically by not being reverse compatible and no longer supporting configurations created under previous versions (how else to describe my situation?).
I finally decided to bite the bullet and upgrade from 2.100 to 2.129 hoping you fixed the previous incompatibility. However, I turned out that all the jobs that were utilizing the publish-over-ssh plugin lost their remote command scripts under "Exec command" field and that the SSH Server name defaults to the first SSH server I have defined in my Jenkins configuration, of which there are more than 50. So essentially, I upgraded to 2.129 and a significant portion of the jobs lost their configuration.
Not only did the configuration disappear in the front end but I dug into the JENKINS_HOME directory substructure where the job configs are in XML files (config.xml) and it appears that the upgrade wiped out the SSH commands there as well as though they never had existed. I can luckily find them under logs but not in the same format as they were in the job (so there is some hope of recovery). It looks like the upgrade changed config.xml to remove any trace of the SSH commands and did not back up the old version that does have the commands, at least I can't find any version history.
When I went in the problem jobs to re-enter the lost config, upon saving I got the attached error message, which looks like Java stack trace. That problem went away once I upgraded the SSH publish plugin from 1.17 to 1.19 but the configs were still gone.
This was a sizable disaster for us and a huge disappointment in Jenkins. I urge you to fix this issue ASAP.
- duplicates
-
-
- Closed
-
[JENKINS-52357] After upgrade 2.100 --> 2.129, publish-over-SSH config disappeared
Resolving since it looks like a full duplicate of JENKINS-49124 .
If the upgrade of plugins does not resolve the issue, please reopen the ticket
I upgraded the plugin but that still doesn't change the fact that the configuration for that plugin disappeared when I upgraded Jenkins – not when I upgraded the plugin. You may argue that is because the condition to have the plugin upgraded when Jenkins was being upgraded wasn't met – however, it is a Jenkins problem because the event that wiped out the configs was the Jenkins upgrade.
The issue was only partly resolved by the plugin upgrade – It enabled re-entry of the SSH configs, however, it did not restore the configs that were deleted when Jenkins was upgraded.
Please do not regard this as a plugin/user problem and not a Jenkins problem. The user should not have to be responsible to make sure all of their plugins are up to date when they upgrade Jenkins. You should have a mechanism to not do damage in case of this use case scenario (outdated plugins).
I understand your "recommendation" but it is not realistic to expect the user community to read fine print disclaimers before performing potentially dangerous actions – if that were the industry standard, we would spend entire work days reading disclaimers. Disclaimers are not a substitute for carefully designed software that assumes stupidity on the part of the user (I am sorry but I have a thousand other things to focus on and things like managing Jenkins need not be usurping mental faculties, i.e. needs to be so soft around the edge as to be impossible to make damage unless some documentation of the size of Anna Karenina was read).
So yeah – this is not a user carelessness problem, it is not a plugin problem, it is a Jenkins problem that you all need to fix by making your app provide reasonable safety nets for user and plugin stupidity.
> The issue was only partly resolved by the plugin upgrade – It enabled re-entry of the SSH configs, however, it did not restore the configs that were deleted when Jenkins was upgraded.
This is strange. The dismissed configs should land in Old Data monitor.
> So yeah – this is not a user carelessness problem, it is not a plugin problem, it is a Jenkins problem that you all need to fix by making your app provide reasonable safety nets for user and plugin stupidity.
Here I agree with you. We used all available channels as a part of this story, including blogposts, mailing lists and upgrade guidelines. With 100 affected plugins, we have got less than 200 issues reported, so my assumption is that we pretty effectively reached out to users. But we can do better for sure.
If you have any particular proposals, please put them in the retrospective document: https://docs.google.com/document/d/1KCCIxWh-c44GJbW_AwKooOd7wD4vthWp_KC0r2OJQl0/edit
> The dismissed configs should land in Old Data monitor.
Can I find them somewhere under my $JENKINS_HOME directory on the server filesystem where the Jenkins runs?
For your information, all publish-over-ssh component type JENKINS issues related to the Publish Over SSH plugin have been transferred to Github: https://github.com/jenkinsci/publish-over-ssh-plugin/issues
Here is the direct link to this issue in Github: https://github.com/jenkinsci/publish-over-ssh-plugin/issues/105
And here is the link to a search for related issues: https://github.com/jenkinsci/publish-over-ssh-plugin/issues?q=%22JENKINS-52357%22
(Note: this is an automated bulk comment)
Closing ticket, please use the corresponding Github Issue as linked above.
> This was a sizable disaster for us and a huge disappointment in Jenkins. I urge you to fix this issue ASAP.
Maybe you want to read the https://jenkins.io/redirect/class-filter/ page as suggested in the error message. Note that this issue is described in upgrade guidelines and weekly/LTS changelogs
JENKINS-49124which has exactly the same stacktrace as you reportedI highly recommend to read https://jenkins.io/redirect/class-filter/ and follow the upgrade guidelines there.