-
Bug
-
Resolution: Fixed
-
Major
-
Windows Server 2016, MSFT Docker, OpenJDK 8 Container
-
Powered by SuggestiMate -
Jenkins 2.168
Using IIS reverse proxy, on LTS 2.138 artifacts and workspace were accessible through web UI as expected. Windows Server 2016 using MSFT docker container, OpenJDK8 or 11 (currently 8) to load jenkins.war file on container start.
After upgrade to LTS 2.150.2, or mainstream 2.164, artifacts are no longer visible on the jobs through the web UI, and accessing job workspaces present a 403 error.
HTTP ERROR 403
Problem accessing /job/<job_name_redacted>/ws/. Reason:
Trying to access a file outside of the directory, target:
Powered by Jetty:// 9.4.z-SNAPSHOT
Attempted upgrade of container to OpenJDK 11, same errors occur.
Job artifacts are created and present on local file system. Workspace is created and present on local file system. Job will build project, just won't present the results/workspace through the UI.
Downgrading back to LTS 2.138 and the job artifacts and workspaces are visible/accessible with same IIS configuration.
IIS reverse proxy reference: https://wiki.jenkins.io/display/JENKINS/Running+Jenkins+behind+IIS (instead of localhost and host file, used container IP address directly). Tested as working in 2.138, fails on 2.150.2+, works correctly when reverted to 2.138.
- causes
-
JENKINS-56578 Jenkins 2.168 prevent HTML Publisher to copy html report from agent to master
-
- Open
-
- links to
[JENKINS-56114] Artifacts and workspace not accessible after upgrade from LTS 2.138 to 2.150.2+ (IIS reverse proxy)
Job directory structure was created by Jenkins itself from initial deployment version of 2.73.2. Upgrades through multiple LTS versions to 2.138 worked as expected.
Jenkins uses a Docker bind mount into the container to map c:\jenkins to the external bind mount point on the host file system, and all Jenkins files reside within in the Jenkins created directory tree.
When starting the container, docker variables and mount point:
$HOST_JENKINS_HOME = "d:/data/jenkins"
$JENKINS_HOME = "c:/jenkins"
--mount type=bind,source=${HOST_JENKINS_HOME},target=${JENKINS_HOME}
Jenkins directory listing (from within the container) included in attachment: jenkins_dir_tree_20190213_1.txt
danielbeck wfollonier Is there any additional information required? We are impacted in one of our deployed instances with this configuration currently. Thanks.
Thanks for the information. Both Wadeck and I are generally very busy, so don't expect responses within just a day, or even several days. I recommend you try the workaround https://jenkins.io/security/advisory/2018-12-05/#SECURITY-904 if you need this functionality to be restored.
danielbeck Thanks for the information. I tried adding that to the startup, no change (artifacts and workspace still unavailable).
Startup command via powershell script:
& java.exe -Dhudson.model.DirectoryBrowserSupport.allowSymlinkEscape=true -jar c:/jenkins.war --sessionTimeout=720 --sessionEviction=720
jkam thank you for the report!
Have you tried the escape hatch with version 2.164? We recently made some changes in that area between 2.163 and 2.164.
Could you also provide the stacktrace of the error in the application?
From my PoV the reverse proxy is completely out of the way. I imagine there is a flaw in the sysmlink detection mechanism we are using in your case.
Could you try to go to the Script Console and execute this script? (with putting the real job name in the nameOfYourJob variable) and give us the result (after redaction of private stuff)
def rootDir = new FilePath(new File(".")) println([rootDir.exists(), rootDir.absolutize()]) def jenkinsHome = new FilePath(Jenkins.get().getRootDir()) println([jenkinsHome.exists(), jenkinsHome.absolutize()]) def workspace = new FilePath(new File(Jenkins.get().getRootDir(), "workspace")) println([workspace.exists(), workspace.absolutize(), workspace.isDescendant(".")]) def nameOfYourJob = "<any of your job name starting by Au>" def link = new FilePath(new File(Jenkins.get().getRootDir(), "workspace/$nameOfYourJob")) println([link.exists(), link.absolutize(), link.isDescendant("."), link.isDescendant("version.txt")]) def linkchild = workspace.child(nameOfYourJob) println([linkchild.exists(), linkchild.absolutize(), linkchild.isDescendant("."), linkchild.isDescendant("version.txt")])
Hi wfollonier, here is the output from the script console. Used the same job as all the items captured above for consistency.
I'm currently using version 2.164.
Note: I'm still using the -D directive above, I'll have to restart the instance without that directive if that information is also required.
[true, C:]
[true, c:\jenkins]
[true, c:\jenkins\workspace, false]
[true, c:\jenkins\workspace\Au<redacted>, false, false]
[true, c:\jenkins\workspace\Au<redacted>, false, false]
For stack traces, I dont' see anything on the console when I attempt to access the project (for it's artifacts), and when I access the workspace the console displays:
Feb 15, 2019 12:03:25 PM hudson.model.DirectoryBrowserSupport serveFile
WARNING: Trying to access a file outside of the directory, target:
Let me know which console and/or debug you're after for more detail.
jkam To clarify, in the warning getting logged, it ends after the colon?
CC jimklimov who reported something similar on IRC, also with the message ending at :
danielbeck Correct, both the HTTP 403 and the console end with no information after "target:".
For thoroughness, I restarted the instance removing the -Dhudson.model.DirectoryBrowserSupport.allowSymlinkEscape=true directive from the startup, and ran the script code above:
[true, C:]
[true, c:\jenkins]
[true, c:\jenkins\workspace, false]
[true, c:\jenkins\workspace\Au<redacted>, false, false]
[true, c:\jenkins\workspace\Au<redacted>, false, false]
jkam, I spent some time trying to reproduce your situation.
Using my computer running Windows 10 (not the Server version like you), with docker with Windows container mode.
docker run -it --mount type=bind,source="c:/[...]/docker_2.150.2_mount",target="c:/jenkins" -p 8085:8085 -p 8005:8005 openjdk:8-windowsservercore
Then once inside, run
java -jar -DJENKINS_HOME=c:/jenkins <link-to-jenkins-2.150.2.war> --httpPort=8085
Then I create a job called test, that is writing "test" to the file "version.txt" and then running the script I provided below.
Result:
[true, C:\] [true, c:\jenkins] [true, c:\jenkins\workspace, true] [true, c:\jenkins\workspace\test, true, true] [true, c:\jenkins\workspace\test, true, true]
Then I created a virtual disk to simulate your situation, and binding a folder inside the new D: to my Jenkins but got exactly the same result. And of course (due to script result) the workspace / artifacts are browsable as expected.
In order to understand better your situation:
1) Did you try with a Windows 10 machine?
2) Could you provide the exact command you used to run (2a) the docker and (2b) the war inside that image?
3) Are you using some particular feature only present in Windows Server 2016?
Answers:
1) I did not try this on Windows 10. Jenkins 2.138 worked correctly. Only after upgrade to 2.150.2+ did issues start. If I revert to 2.138, the workspace and artifacts will work again, and then going back to 2.150.2+ they stop working (currently on 2.164 but i usually prefer to track LTS).
Note: All the jobs were build using pre 2.138 versions (I started on 2.73), so those built the directory trees, etc. that have been carried through the LTS upgrade cycles. Only on 2.150.2+ did the workspace/artifacts start having issue.
2) Included
3) I am using Windows Server 2016 Version 1607. Docker was installed through powershell, PSGallery, so it is MSFT Docker.
Install-Module -Name DockerMsftProvider -Repository PSGallery
Docker runner:
$JENKINS_IP = "<actual docker container fixed IP>"
$JENKINS_VERSION = "2.164"
$HOST_JENKINS_HOME = "d:/data/jenkins"
$JENKINS_HOME = "c:/jenkins"
$HOST_BUILD_ARCHIVE = "d:/data/buildarchive"
$JENKINS_BUILD_ARCHIVE = "c:/buildarchive"
docker run -d --ip ${JENKINS_IP} `
--mount type=bind,source=${HOST_JENKINS_HOME},target=${JENKINS_HOME} `
--mount type=bind,source=${HOST_BUILD_ARCHIVE},target=${JENKINS_BUILD_ARCHIVE} `
"<redacted>/jenkins-on-windowsservercore:$JENKINS_VERSION"
C: is the Jenkins point within the container, which mounts from D: outside the container as the docker mount/data drive.
For the web UI, the Jenkins port is exposed via the Dockerfile (EXPOSE 8080).
When I build the container, Dockefile uses a powershell command to create an empty C:/jenkins within the container for the mount points.
RUN
New-Item -ItemType Directory -Force -Path $env:JENKINS_HOME;
New-Item -ItemType Directory -Force -Path $env:JENKINS_BUILD_ARCHIVE;
Staring Jenkins (via powershell startup script):
& java.exe -jar c:/jenkins.war --sessionTimeout=720 --sessionEviction=720
jkam Thank you for the additional information. Unfortunately, the "best" I can reproduce is to throw java.lang.IllegalArgumentException: The parent does not exist. I never get to the point of having the Trying to access a file outside of the directory, target:.
Tested with Windows Server 2016, 1607, using DockerMsftProvider: 1.0.0.6, with docker 18.09.2.
I tried with the regular openjdk:8-windowsservercore and also with a custom built dockerfile, inspired by https://github.com/asmagin/jenkins-on-windowsservercore/ but using your instructions.
The only thing I "discover", is when you are in the docker image, if you use cmd, and type dir, you got something like:
Do you have any other information that could be useful?
And additionally, using the escape hatch, the workspace / artifact are displayed as expected.
Possibly related issues:
- https://github.com/nodejs/node/issues/8897
- https://github.com/moby/moby/issues/27537
- [bug in openjdk] https://bugs.openjdk.java.net/browse/JDK-8172711 <= most interesting
Finally reproduced with success. To have the message and not the exception, the required version is 2.164+ ! The behavior changed between LTS and that weekly around that message / exception.
If I run:
DirectoryBrowserSupport.ALLOW_SYMLINK_ESCAPE = true
In the Script Console, I can access the workspace, but the artifacts are still not displayed.
I will provide a WAR in a moment (still today I hope) so that you can test by yourself if my changes are effectice in your case or not
The WAR will be available in this PR once the build is completed.
Thanks wfollonier, I will try a deployment with the WAR once it becomes available and let you know my results.
jkam As the build is currently failing, you can use this link in the meantime: https://drive.google.com/file/d/1cRd_DKpJF9hezf1q9mM2LDMHUM_vQGCp/view?usp=sharing
I just build the war locally from the branch of the PR, feel free to don't trust and wait for the build to success
The WAR is now available from the build on https://ci.jenkins.io/blue/organizations/jenkins/Core%2Fjenkins/detail/PR-3914/3/artifacts, search for jenkins-war-2.167-rc28022.09203244c3f8.war.
Hi wfollonier, I have retrieved the CI build and deployed to my environment. I can report that the fix is successful. We are able to see the workspace and artifacts now. We have also run several jobs and so far have not encountered other issues.
Hi wfollonier - we ran though various jobs, screens, etc. as well and everything seems to working as expected. Thanks for your work on this and the turnaround on a fix!
jkam That's really appreciated to receive such feedback, thank you The correction will normally be part of the next weekly (~2.168)
wfollonier Looks like a possible regression from your fix.
jkam Please provide the full paths including involved reparse points (symlinks, junctions, etc.) to all directories involved here.