Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-14483

Remaining Xvfb processes in matrix jobs

    XMLWordPrintable

Details

    Description

      Enabling the Xvfb plugin on matrix jobs starts a Xvfb server for the job itself and one for each configuration.

      The Xvfb servers for the configurations are stoppd properly, but the Xvfb server for the job itself keeps running when the job finishes. This ends up in locked screens and a vast number of directories.

      We experienced that on a master only instance as well as in master-slave mode.

      Xvfb plugin version 1.0.2

      Attachments

        Activity

          zregvart zregvart added a comment - - edited

          Thorsten, would you mind testing a snapshot version of the plugin? I've changed the behavior for matrix builds so that the Xvfb is only started for matrix jobs, not the build itself. You'll need to install (upload) the plugin in your Jenkins instance as this is a snapshot version not a released. I will release the plugin as soon as you confirm that it fixes your problem.

          The url for the snapshot version is: https://www.dropbox.com/s/tdakwhdxq9glae5/xvfb.hpi

          thanks!

          Zoran

          zregvart zregvart added a comment - - edited Thorsten, would you mind testing a snapshot version of the plugin? I've changed the behavior for matrix builds so that the Xvfb is only started for matrix jobs, not the build itself. You'll need to install (upload) the plugin in your Jenkins instance as this is a snapshot version not a released. I will release the plugin as soon as you confirm that it fixes your problem. The url for the snapshot version is: https://www.dropbox.com/s/tdakwhdxq9glae5/xvfb.hpi thanks! Zoran

          Hi Zoran,

          I tried version 1.0.3-SNAPSHOT (private-07/21/2012 12:32-zregvart) on Jenkins 1.434 without success: Xvfb is still started on the matrix build too.

          console output:

          ...
          Ignoring Matrix build, waiting for Matrix jobs to follow
          Xvfb starting$ /usr/bin/Xvfb :2 -screen 0 1024x768x24 -fbdir ${JENKINS_HOME}/2012-07-23_16-22-104384832168309393644xvfb
          ...
          
          tkahler Thorsten Kahler added a comment - Hi Zoran, I tried version 1.0.3-SNAPSHOT (private-07/21/2012 12:32-zregvart) on Jenkins 1.434 without success: Xvfb is still started on the matrix build too. console output: ... Ignoring Matrix build, waiting for Matrix jobs to follow Xvfb starting$ /usr/bin/Xvfb :2 -screen 0 1024x768x24 -fbdir ${JENKINS_HOME}/2012-07-23_16-22-104384832168309393644xvfb ...
          zregvart zregvart added a comment -

          Hi Thorsten, sorry for not getting back earlier, the new version is up on Dropbox (same url), this version should fix this issue – only one Xvfb server will be launched: the one for matrix build, for individual matrix jobs no attempt will be made to start more Xvfb servers, when the build/jobs finish Xvfb should stop and the temp directory should be removed.

          If you have the time to test this version I would much appreciate it.

          Thanks!

          zregvart zregvart added a comment - Hi Thorsten, sorry for not getting back earlier, the new version is up on Dropbox (same url), this version should fix this issue – only one Xvfb server will be launched: the one for matrix build, for individual matrix jobs no attempt will be made to start more Xvfb servers, when the build/jobs finish Xvfb should stop and the temp directory should be removed. If you have the time to test this version I would much appreciate it. Thanks!

          Hi Zoran,

          I justed tested the new version with following observations (according to console log) on Jenkins 1.434:

          • Xvfb is only started with matrix job
          • Xvfb is stopped with configuration jobs

          The result is that the first configuration works fine and properly stops the Xvfb process. But then there's no available X server for the following configuration jobs and so they fail (message: webkit_server: cannot connect to X server :16; F-webkit_server: cannot connect to X server :16).
          I've only tested on a standalone Jenkins server but I think that this behaviour will fail especially in master-slave setups: if Xvfb is started on the master and the configuration jobs run on a slave every job will fail.

          Sorry for the bad news

          Cheers,
          Thorsten

          tkahler Thorsten Kahler added a comment - Hi Zoran, I justed tested the new version with following observations (according to console log) on Jenkins 1.434: Xvfb is only started with matrix job Xvfb is stopped with configuration jobs The result is that the first configuration works fine and properly stops the Xvfb process. But then there's no available X server for the following configuration jobs and so they fail (message: webkit_server: cannot connect to X server :16; F-webkit_server: cannot connect to X server :16). I've only tested on a standalone Jenkins server but I think that this behaviour will fail especially in master-slave setups: if Xvfb is started on the master and the configuration jobs run on a slave every job will fail. Sorry for the bad news Cheers, Thorsten
          zregvart zregvart added a comment -

          Thorsten, I've uploaded a new version on Dropbox. I think that I've finally cracked it, although I'm not all too happy about the implementation: Xvfb is started with matrix build, subsequent axis jobs are run and counted, when the matrix job finishes total that count is decreased and if it gets to zero Xvfb is stopped. You see, I could not find a way to get notified that all matrix jobs are done, so counting is the best thing I could think of. Anyways I realized that I needed to test better as well, and it seems to workout nice in my environment.

          Thank you for your patience, I hope that this will finally fix it. If you can test in your environment once again and confirm that I would much appreciate it.

          Thanks!

          zregvart zregvart added a comment - Thorsten, I've uploaded a new version on Dropbox. I think that I've finally cracked it, although I'm not all too happy about the implementation: Xvfb is started with matrix build, subsequent axis jobs are run and counted, when the matrix job finishes total that count is decreased and if it gets to zero Xvfb is stopped. You see, I could not find a way to get notified that all matrix jobs are done, so counting is the best thing I could think of. Anyways I realized that I needed to test better as well, and it seems to workout nice in my environment. Thank you for your patience, I hope that this will finally fix it. If you can test in your environment once again and confirm that I would much appreciate it. Thanks!

          Hi Zoran,

          I'm sorry but it still doesn't work I think Xvfb shouldn't be started with the matrix job itself but only with every configuration - and stopped there as well. For a standalone Jenkins your current approach might work if Xvfb would not be stopped with configuration jobs (see excerpts from console log below). But as mentionend before with a master-slave setup IMHO the only way would be to start and stop Xvfb with the configuration jobs.

          Cheers,
          Thorsten

          Output of my latest test:

          Matrix job:

          Xvfb starting$ /usr/bin/Xvfb :60 -screen 0 1024x768x24 -fbdir /...
          [EnvInject] - Preparing an environment for the job.
          [EnvInject] - Jenkins system variables are kept.
          [EnvInject] - Jenkins build variables are kept.
          Couldn't open RGB_DB '/usr/share/X11/rgb'
          _XSERVTransmkdir: Owner of /tmp/.X11-unix should be set to root
          [EnvInject] - Injecting as environment variables the properties content 
          # ...
          error opening security policy file /usr/lib64/xserver/SecurityPolicy
          [EnvInject] - Variables injected successfully.
          
          Deleting project workspace... done
          
          Triggering <configuration_1>
          FreeFontPath: FPE "unix/:7100" refcount is 2, should be 1; fixing.
          Couldn't open RGB_DB '/usr/share/X11/rgb'
          error opening security policy file /usr/lib64/xserver/SecurityPolicy
          FreeFontPath: FPE "unix/:7100" refcount is 2, should be 1; fixing.
          Couldn't open RGB_DB '/usr/share/X11/rgb'
          error opening security policy file /usr/lib64/xserver/SecurityPolicy
          FreeFontPath: FPE "unix/:7100" refcount is 2, should be 1; fixing.
          Triggering <configuration_2>
          <configuration_2> completed with result FAILURE
          Finished: FAILURE
          

          configuration_1 (touchstone build):

          gnoring Matrix job, Matrix build (the parent of this job) should have started Xvfb
          [EnvInject] - Using environment variables injected by the matrix job.
          
          Deleting project workspace... done
          
          No emails were triggered.
          [<configuration_1>] $ /bin/bash -le /tmp/hudson3288844765556768259.sh
          [EnvInject] - Injecting environment variables from a build step.
          [EnvInject] - Injecting as environment variables the properties file path '/.../<confiugration_1>/build_environment.properties'
          [EnvInject] - Variables injected successfully.
          # ...
          # doing deployment of PHP app
          # running some Jasmine smoke tests
          # ...
          Xvfb stopping
          Archiving artifacts
          # ...
          

          configuration_2:

          Ignoring Matrix job, Matrix build (the parent of this job) should have started Xvfb
          [EnvInject] - Using environment variables injected by the matrix job.
          
          Deleting project workspace... done
          
          No emails were triggered.
          [<confiugration_2>] $ /bin/bash -le /tmp/hudson4841252131230113695.sh
          [EnvInject] - Injecting environment variables from a build step.
          [EnvInject] - Injecting as environment variables the properties file path '/.../<confiugration_2>/build_environment.properties'
          [EnvInject] - Variables injected successfully.
          # ...
          # doing deployment of PHP app
          # trying to run some Jasmine smoke tests:
          Call smoketest (product) for URL http://<configuration_2.domain>/
          Using the ci-smoketest profile...
          webkit_server: cannot connect to X server :60
          F-webkit_server: cannot connect to X server :60
          F
          
          (::) failed steps (::)
          
          # error message
          
          1 scenario (1 failed)
          2 steps (1 failed, 1 skipped)
          0m30.454s
          Build step 'Use builders from another project' marked build as failure
          Xvfb stopping
          Archiving artifacts
          
          tkahler Thorsten Kahler added a comment - Hi Zoran, I'm sorry but it still doesn't work I think Xvfb shouldn't be started with the matrix job itself but only with every configuration - and stopped there as well. For a standalone Jenkins your current approach might work if Xvfb would not be stopped with configuration jobs (see excerpts from console log below). But as mentionend before with a master-slave setup IMHO the only way would be to start and stop Xvfb with the configuration jobs. Cheers, Thorsten Output of my latest test: Matrix job: Xvfb starting$ /usr/bin/Xvfb :60 -screen 0 1024x768x24 -fbdir /... [EnvInject] - Preparing an environment for the job. [EnvInject] - Jenkins system variables are kept. [EnvInject] - Jenkins build variables are kept. Couldn 't open RGB_DB ' /usr/share/X11/rgb' _XSERVTransmkdir: Owner of /tmp/.X11-unix should be set to root [EnvInject] - Injecting as environment variables the properties content # ... error opening security policy file /usr/lib64/xserver/SecurityPolicy [EnvInject] - Variables injected successfully. Deleting project workspace... done Triggering <configuration_1> FreeFontPath: FPE "unix/:7100" refcount is 2, should be 1; fixing. Couldn 't open RGB_DB ' /usr/share/X11/rgb' error opening security policy file /usr/lib64/xserver/SecurityPolicy FreeFontPath: FPE "unix/:7100" refcount is 2, should be 1; fixing. Couldn 't open RGB_DB ' /usr/share/X11/rgb' error opening security policy file /usr/lib64/xserver/SecurityPolicy FreeFontPath: FPE "unix/:7100" refcount is 2, should be 1; fixing. Triggering <configuration_2> <configuration_2> completed with result FAILURE Finished: FAILURE configuration_1 (touchstone build): gnoring Matrix job, Matrix build (the parent of this job) should have started Xvfb [EnvInject] - Using environment variables injected by the matrix job. Deleting project workspace... done No emails were triggered. [<configuration_1>] $ /bin/bash -le /tmp/hudson3288844765556768259.sh [EnvInject] - Injecting environment variables from a build step. [EnvInject] - Injecting as environment variables the properties file path '/.../<confiugration_1>/build_environment.properties' [EnvInject] - Variables injected successfully. # ... # doing deployment of PHP app # running some Jasmine smoke tests # ... Xvfb stopping Archiving artifacts # ... configuration_2: Ignoring Matrix job, Matrix build (the parent of this job) should have started Xvfb [EnvInject] - Using environment variables injected by the matrix job. Deleting project workspace... done No emails were triggered. [<confiugration_2>] $ /bin/bash -le /tmp/hudson4841252131230113695.sh [EnvInject] - Injecting environment variables from a build step. [EnvInject] - Injecting as environment variables the properties file path '/.../<confiugration_2>/build_environment.properties' [EnvInject] - Variables injected successfully. # ... # doing deployment of PHP app # trying to run some Jasmine smoke tests: Call smoketest (product) for URL http: //<configuration_2.domain>/ Using the ci-smoketest profile... webkit_server: cannot connect to X server :60 F-webkit_server: cannot connect to X server :60 F (::) failed steps (::) # error message 1 scenario (1 failed) 2 steps (1 failed, 1 skipped) 0m30.454s Build step 'Use builders from another project' marked build as failure Xvfb stopping Archiving artifacts
          zregvart zregvart added a comment -

          Thorsten, thank you for testing!

          I agree the best way would be to run Xvfb with each matrix job, that was my initial approach – but all matrix jobs share the same BuildWrapper instance, so process handle that is used for Xvfb termination/cleanup would be overwritten with each subsequent matrix job. I have to rethink my options here; I'll ask on jenkins-dev mailing list for starters – I must be missing something.

          Anyways thank you for following this issue for so long and I hope to find the solution for it soon.

          zregvart zregvart added a comment - Thorsten, thank you for testing! I agree the best way would be to run Xvfb with each matrix job, that was my initial approach – but all matrix jobs share the same BuildWrapper instance, so process handle that is used for Xvfb termination/cleanup would be overwritten with each subsequent matrix job. I have to rethink my options here; I'll ask on jenkins-dev mailing list for starters – I must be missing something. Anyways thank you for following this issue for so long and I hope to find the solution for it soon.

          Code changed in jenkins
          User: Zoran Regvart
          Path:
          src/main/java/org/jenkinsci/plugins/xvfb/XvfbBuildWrapper.java
          src/main/java/org/jenkinsci/plugins/xvfb/XvfbEnvironment.java
          http://jenkins-ci.org/commit/xvfb-plugin/b25ee74766b91b268f740640590f985191323a21
          Log:
          JENKINS-14483 Remaining Xvfb processes in matrix jobs

          implemented as kindly suggested by Chris Jonhson on jenkins-dev mailing
          list

          https://groups.google.com/d/topic/jenkinsci-dev/KQVDCV9ATm8/discussion

          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Zoran Regvart Path: src/main/java/org/jenkinsci/plugins/xvfb/XvfbBuildWrapper.java src/main/java/org/jenkinsci/plugins/xvfb/XvfbEnvironment.java http://jenkins-ci.org/commit/xvfb-plugin/b25ee74766b91b268f740640590f985191323a21 Log: JENKINS-14483 Remaining Xvfb processes in matrix jobs implemented as kindly suggested by Chris Jonhson on jenkins-dev mailing list https://groups.google.com/d/topic/jenkinsci-dev/KQVDCV9ATm8/discussion
          zregvart zregvart added a comment -

          Thorsten, Chris Johnson helped me a great deal on the jenkins-dev mailing list (https://groups.google.com/d/topic/jenkinsci-dev/KQVDCV9ATm8/discussion) so I'm pretty confident that the version I've uploaded to Dropbox, finally, solves this issue. Now Xvfb is launched only for individual matrix jobs and should terminate and clean up when the job finishes. If I may be so bold to ask you to test on your end one more time, so that when you confirm that it's OK I can release version 1.0.4.

          Thank you!

          zregvart zregvart added a comment - Thorsten, Chris Johnson helped me a great deal on the jenkins-dev mailing list ( https://groups.google.com/d/topic/jenkinsci-dev/KQVDCV9ATm8/discussion ) so I'm pretty confident that the version I've uploaded to Dropbox, finally, solves this issue. Now Xvfb is launched only for individual matrix jobs and should terminate and clean up when the job finishes. If I may be so bold to ask you to test on your end one more time, so that when you confirm that it's OK I can release version 1.0.4. Thank you!

          Hi Zoran, this is the error message (in matrix job) after installing your latest snapshot version:

          Started by user Thorsten Kahler
          [EnvInject] - Preparing an environment for the job.
          [EnvInject] - Jenkins system variables are kept.
          [EnvInject] - Jenkins build variables are kept.
          [EnvInject] - [ERROR] - SEVERE ERROR occurs: null
          Finished: FAILURE
          

          This might be a compatibility issue with our outdated Jenkins version 1.434?

          tkahler Thorsten Kahler added a comment - Hi Zoran, this is the error message (in matrix job) after installing your latest snapshot version: Started by user Thorsten Kahler [EnvInject] - Preparing an environment for the job. [EnvInject] - Jenkins system variables are kept. [EnvInject] - Jenkins build variables are kept. [EnvInject] - [ERROR] - SEVERE ERROR occurs: null Finished: FAILURE This might be a compatibility issue with our outdated Jenkins version 1.434?
          zregvart zregvart added a comment -

          I'm guessing that env-inject plugin catches NullPointerException (probably here: https://github.com/jenkinsci/envinject-plugin/blob/master/src/main/java/org/jenkinsci/plugins/envinject/EnvInjectListener.java#L60), this could be due to setup method of xvfb plugin now returning null (https://github.com/jenkinsci/xvfb-plugin/blob/master/src/main/java/org/jenkinsci/plugins/xvfb/XvfbBuildWrapper.java#L255) on non unix platforms. Can it be that one of your slave is non-unix (windows)? It would probably be better for backward compatibility that the xvfb plugin does not return null on non-unix platforms, as it did in previous versions. I'll make that change and upload a new version on dropbox in a few minutes.

          zregvart zregvart added a comment - I'm guessing that env-inject plugin catches NullPointerException (probably here: https://github.com/jenkinsci/envinject-plugin/blob/master/src/main/java/org/jenkinsci/plugins/envinject/EnvInjectListener.java#L60 ), this could be due to setup method of xvfb plugin now returning null ( https://github.com/jenkinsci/xvfb-plugin/blob/master/src/main/java/org/jenkinsci/plugins/xvfb/XvfbBuildWrapper.java#L255 ) on non unix platforms. Can it be that one of your slave is non-unix (windows)? It would probably be better for backward compatibility that the xvfb plugin does not return null on non-unix platforms, as it did in previous versions. I'll make that change and upload a new version on dropbox in a few minutes.

          Code changed in jenkins
          User: Zoran Regvart
          Path:
          src/main/java/org/jenkinsci/plugins/xvfb/XvfbBuildWrapper.java
          http://jenkins-ci.org/commit/xvfb-plugin/a53ce0e6cf6acc96ad5a2020d1f0492e9e7b6492
          Log:
          JENKINS-14483 Remaining Xvfb processes in matrix jobs

          return empty environment instead of null from setUp method

          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Zoran Regvart Path: src/main/java/org/jenkinsci/plugins/xvfb/XvfbBuildWrapper.java http://jenkins-ci.org/commit/xvfb-plugin/a53ce0e6cf6acc96ad5a2020d1f0492e9e7b6492 Log: JENKINS-14483 Remaining Xvfb processes in matrix jobs return empty environment instead of null from setUp method
          zregvart zregvart added a comment -

          Thorsten, I've uploaded a new snapshot on Dropbox, if it's OK now I'll release 1.0.4 later this afternoon.

          zregvart zregvart added a comment - Thorsten, I've uploaded a new snapshot on Dropbox, if it's OK now I'll release 1.0.4 later this afternoon.

          Hi Zoran, still the same error.

          tkahler Thorsten Kahler added a comment - Hi Zoran, still the same error.

          Code changed in jenkins
          User: Zoran Regvart
          Path:
          src/main/java/org/jenkinsci/plugins/xvfb/XvfbBuildWrapper.java
          http://jenkins-ci.org/commit/xvfb-plugin/81e7a1ff832b99bf2652b022c56eb1e81c96163c
          Log:
          JENKINS-14483 Remaining Xvfb processes in matrix jobs

          fix NPE with Environment Injector Plugin installed

          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Zoran Regvart Path: src/main/java/org/jenkinsci/plugins/xvfb/XvfbBuildWrapper.java http://jenkins-ci.org/commit/xvfb-plugin/81e7a1ff832b99bf2652b022c56eb1e81c96163c Log: JENKINS-14483 Remaining Xvfb processes in matrix jobs fix NPE with Environment Injector Plugin installed
          zregvart zregvart added a comment -

          Hi Thorsten I was able to reproduce your problem with Environment Injector Plugin installed, luckily it was a very simple fix, and as usual I need your help testing the new version I've just uploaded to Dropbox.

          Thank you

          zregvart zregvart added a comment - Hi Thorsten I was able to reproduce your problem with Environment Injector Plugin installed, luckily it was a very simple fix, and as usual I need your help testing the new version I've just uploaded to Dropbox. Thank you
          zregvart zregvart added a comment -

          fixed in 1.0.4, changes in behavior are noted in the wiki (https://wiki.jenkins-ci.org/display/JENKINS/Xvfb+Plugin)

          zregvart zregvart added a comment - fixed in 1.0.4, changes in behavior are noted in the wiki ( https://wiki.jenkins-ci.org/display/JENKINS/Xvfb+Plugin )

          People

            zregvart zregvart
            tkahler Thorsten Kahler
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: