Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-21548

Xvfb failure does not fail build

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: xvfb-plugin
    • Labels:
    • Environment:
      RHEL 6.2
    • Similar Issues:

      Description

      When using the Xvfb plugin, if the plugin fails to establish a connection, then the build should fail, as any tests that depend on an Xvfb display are bound to fail also. Or, at least there should be an option in the advance settings to fail the job if Xvfb fails.

      Currently, the plugin simply logs an error, and then proceeds with the rest of the build job.

      12:28:14 Xvfb starting$ /usr/bin/Xvfb :1958 -screen 0 1024x768x24 -fbdir /buildstore/jenkins_0c9_slave/2014-01-29_12-28-139011048368266724308xvfb
      12:28:14 _XSERVTransSocketINETCreateListener: ...SocketCreateListener() failed
      12:28:14 _XSERVTransMakeAllCOTSServerListeners: server already running
      12:28:14
      12:28:14 Fatal server error:
      12:28:14 Cannot establish any listening sockets - Make sure an X server isn't already running
      12:28:14 unlink: No such file or directory
      12:28:14 unlink failed, Invalid argument
      12:28:14 ERROR: Xvfb failed to start, consult the lines above for errors
      12:28:29 ha:AAAAWB+LCAAAAAAAAABb85aBtbiIQSmjNKU4P08vOT+vOD8nVc8DzHWtSE4tKMnMz/PLL0ldFVf2c+b/lb5MDAwVRQxSaBqcITRIIQMEMIIUFgAAckCEiWAAAAA=Copied 1 artifact from "ha:AAAAsB+LCAAAAAAAAABb85aBtbiIQSajNKU4P08vOT+vOD8nVc+jsiC1KCczL9svvyTVzHb1RttJBUeZGJg8GdhyUvPSSzJ8GJhLi3JKGIR8shLLEvVzEvPS9YNLijLz0q0rihik0IxzhtAgwxgggJGJgaGiAMjQLmEw0M/KT9L3TSwuSS2KL0lNzDUwjHcsLcnPTSxJTYl3K81LLsnMz0vMiQ9JLS4B2qAPABqeEK21AAA
      ....
      ....
      .... [rest of build job proceeds, and inevitably fails]
      ....

        Attachments

          Activity

          Hide
          zregvart zregvart added a comment -

          Hi Richard, thanks for reporting the issue. Is there anything that I should be aware with regards to your build/steps? I'm asking because since JENKINS-18094 (version 1.0.8) if you receive "Xvfb failed to start, consult the lines above for errors" message in the console log your build will inevitably be aborted due to RunnerAbortedException being thrown immediately after.
          If you indeed are running version 1.0.8 could you create a sample build job and attach it to this issue.

          Thanks

          Zoran

          Show
          zregvart zregvart added a comment - Hi Richard, thanks for reporting the issue. Is there anything that I should be aware with regards to your build/steps? I'm asking because since JENKINS-18094 (version 1.0.8) if you receive "Xvfb failed to start, consult the lines above for errors" message in the console log your build will inevitably be aborted due to RunnerAbortedException being thrown immediately after. If you indeed are running version 1.0.8 could you create a sample build job and attach it to this issue. Thanks Zoran
          Hide
          robotlime Richard Cross added a comment - - edited

          Well, this is embarrassing. I'm having difficulty setting up a test job that replicates the issue. With Xvfb configured and a single, simple shell command as the build task, the build aborts if a display is already taken (I run the same display at the command line.

          In my example above, the issue I was getting is this: http://www.x.org/wiki/FAQErrorMessages/#index6h2 which is slightly harder to replicate manually.

          Show
          robotlime Richard Cross added a comment - - edited Well, this is embarrassing. I'm having difficulty setting up a test job that replicates the issue. With Xvfb configured and a single, simple shell command as the build task, the build aborts if a display is already taken (I run the same display at the command line. In my example above, the issue I was getting is this: http://www.x.org/wiki/FAQErrorMessages/#index6h2 which is slightly harder to replicate manually.
          Hide
          robotlime Richard Cross added a comment - - edited

          Ok - I've replicated the issue. It happens when Xvfb's lock file is removed, but the display is still running.

          1. Start a display on Jenkins Slave: /usr/bin/Xvfb :1972 -screen 0 1024x768x24 &
          2. Remove the corresponding lock file: rm -rf /tmp/.X1972-lock
          3. Run a simply Jenkins job that starts Xvfb on display 1972, with a single build task (e.g. a
          4. Run build job

          Build job passes successfully, even though the Xvfb plugin logs an error.

          Started by user Cross, Richard (Contractor)
          [EnvInject] - Loading node environment variables.
          Building remotely on appduvirtual0c9 in workspace /buildstore/jenkins_appduvirtual0c9_slave/workspace/Xvfb_test_failure_scenario
          Xvfb starting$ /usr/bin/Xvfb :1972 -screen 0 1024x768x24 -fbdir /buildstore/jenkins_appduvirtual0c9_slave/2014-02-03_12-23-504926962666561200040xvfb
          _XSERVTransSocketINETCreateListener: ...SocketCreateListener() failed
          _XSERVTransMakeAllCOTSServerListeners: server already running

          Fatal server error:
          Cannot establish any listening sockets - Make sure an X server isn't already running
          unlink: No such file or directory
          unlink failed, Invalid argument[Xvfb_test_failure_scenario] $ /bin/sh -xe /tmp/hudson7411154595497137988.sh
          + echo This should only run is Xvfb is successful.
          This should only run is Xvfb is successful.
          Xvfb stopping
          Notifying upstream projects of job completion
          Finished: SUCCESS

          Show
          robotlime Richard Cross added a comment - - edited Ok - I've replicated the issue. It happens when Xvfb's lock file is removed, but the display is still running. 1. Start a display on Jenkins Slave: /usr/bin/Xvfb :1972 -screen 0 1024x768x24 & 2. Remove the corresponding lock file: rm -rf /tmp/.X1972-lock 3. Run a simply Jenkins job that starts Xvfb on display 1972, with a single build task (e.g. a 4. Run build job Build job passes successfully, even though the Xvfb plugin logs an error. Started by user Cross, Richard (Contractor) [EnvInject] - Loading node environment variables. Building remotely on appduvirtual0c9 in workspace /buildstore/jenkins_appduvirtual0c9_slave/workspace/Xvfb_test_failure_scenario Xvfb starting$ /usr/bin/Xvfb :1972 -screen 0 1024x768x24 -fbdir /buildstore/jenkins_appduvirtual0c9_slave/2014-02-03_12-23-504926962666561200040xvfb _XSERVTransSocketINETCreateListener: ...SocketCreateListener() failed _XSERVTransMakeAllCOTSServerListeners: server already running Fatal server error: Cannot establish any listening sockets - Make sure an X server isn't already running unlink: No such file or directory unlink failed, Invalid argument [Xvfb_test_failure_scenario] $ /bin/sh -xe /tmp/hudson7411154595497137988.sh + echo This should only run is Xvfb is successful. This should only run is Xvfb is successful. Xvfb stopping Notifying upstream projects of job completion Finished: SUCCESS
          Hide
          zregvart zregvart added a comment -

          Richard, thanks the detailed steps needed to reproduce this. I'm getting more or less the same output on my test setup but unlike in your case the build fails. This is what I got:

          Building remotely on ubuntu
          Xvfb starting$ /usr/bin/Xvfb :1111 -screen 0 1024x768x24 -fbdir /tmp/jenkins/2014-02-03_20-51-241837796366101415022xvfb
          _XSERVTransSocketINETCreateListener: ...SocketCreateListener() failed
          _XSERVTransMakeAllCOTSServerListeners: server already running
          (EE) 
          Fatal server error:
          (EE) Cannot establish any listening sockets - Make sure an X server isn't already running(EE) 
          unlink: No such file or directory
          unlink  failed, Invalid argument
          ERROR: Xvfb failed to start, consult the lines above for errors
          Finished: FAILURE
          

          I'm running this slave on Ubuntu 13.10 with Xvfb from x.org/freedesktop.org version 1.14.3.

          I did notice that your last console log does not contain the "ERROR: Xvfb failed to start, consult the lines above for errors" line, was it missing from the output?

          The way the plugin checks if Xvfb started successfully is to see if the Xvfb process is alive, it could be that in your case there is a race condition between this check and the Xvfb terminating. To help with these situations you have an option to set the timeout value (the "Timeout in seconds" option in the advanced settings), where 0 is no timeout. Try setting timeout to 1 or 2 seconds and see if the launch failure of Xvfb is properly detected.

          Another alternative would be to let Xvfb choose its own display number, the way it does this is to try and find a free port to listen on, so it would not start on the same port as another X server.

          Show
          zregvart zregvart added a comment - Richard, thanks the detailed steps needed to reproduce this. I'm getting more or less the same output on my test setup but unlike in your case the build fails. This is what I got: Building remotely on ubuntu Xvfb starting$ /usr/bin/Xvfb :1111 -screen 0 1024x768x24 -fbdir /tmp/jenkins/2014-02-03_20-51-241837796366101415022xvfb _XSERVTransSocketINETCreateListener: ...SocketCreateListener() failed _XSERVTransMakeAllCOTSServerListeners: server already running (EE) Fatal server error: (EE) Cannot establish any listening sockets - Make sure an X server isn't already running(EE) unlink: No such file or directory unlink failed, Invalid argument ERROR: Xvfb failed to start, consult the lines above for errors Finished: FAILURE I'm running this slave on Ubuntu 13.10 with Xvfb from x.org/freedesktop.org version 1.14.3. I did notice that your last console log does not contain the "ERROR: Xvfb failed to start, consult the lines above for errors" line, was it missing from the output? The way the plugin checks if Xvfb started successfully is to see if the Xvfb process is alive, it could be that in your case there is a race condition between this check and the Xvfb terminating. To help with these situations you have an option to set the timeout value (the "Timeout in seconds" option in the advanced settings), where 0 is no timeout. Try setting timeout to 1 or 2 seconds and see if the launch failure of Xvfb is properly detected. Another alternative would be to let Xvfb choose its own display number, the way it does this is to try and find a free port to listen on, so it would not start on the same port as another X server.
          Hide
          robotlime Richard Cross added a comment -

          Thanks for the suggestion; I'll try it out when I'm back in the office.

          By the way, I only hard coded the display number in this job order to replicate the fault. In our live build jobs of course, the display number is left to auto-select, with a high offset to avoid collisions with other processes that require displays.

          Show
          robotlime Richard Cross added a comment - Thanks for the suggestion; I'll try it out when I'm back in the office. By the way, I only hard coded the display number in this job order to replicate the fault. In our live build jobs of course, the display number is left to auto-select, with a high offset to avoid collisions with other processes that require displays.
          Hide
          zregvart zregvart added a comment -

          Local race condition, closing

          Show
          zregvart zregvart added a comment - Local race condition, closing

            People

            Assignee:
            zregvart zregvart
            Reporter:
            robotlime Richard Cross
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: