• 2.320,2.319.1

      See https://github.com/jenkinsci/jenkins/runs/4107177325
      https://github.com/jenkinsci/jenkins/pull/5881/checks?check_run_id=4107861004

      Logs:
      https://ci.jenkins.io/job/Core/job/jenkins/job/PR-5881/2/execution/node/116/

      java.nio.file.NoSuchFileException: /home/jenkins/workspace/Core_jenkins_PR-5881/ath/athSources/src/main/resources/ath-container/run.sh
      	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
      	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
      	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
      	at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219)
      	at java.base/java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:484)
      	at java.base/java.nio.file.Files.newOutputStream(Files.java:228)
      	at hudson.util.IOUtils.copy(IOUtils.java:52)
      	at hudson.FilePath.readFromTar(FilePath.java:2862)
      Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to EC2 (aws-us-east-2) - High memory ubuntu 20.04 (i-02b6bda105ac6c6d9)
      		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1797)
      		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
      		at hudson.remoting.Channel.call(Channel.java:1001)
      		at hudson.FilePath.act(FilePath.java:1166)
      		at hudson.FilePath.act(FilePath.java:1155)
      		at hudson.FilePath.untar(FilePath.java:617)
      		at org.jenkinsci.plugins.workflow.flow.StashManager.unstash(StashManager.java:161)
      		at org.jenkinsci.plugins.workflow.support.steps.stash.UnstashStep$Execution.run(UnstashStep.java:77)
      		at org.jenkinsci.plugins.workflow.support.steps.stash.UnstashStep$Execution.run(UnstashStep.java:64)
      		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
      		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
      		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
      		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      		at java.base/java.lang.Thread.run(Thread.java:829)
      Caused: java.io.IOException: Failed to extract athSources.tar.gz
      	at hudson.FilePath.readFromTar(FilePath.java:2872)
      	at hudson.FilePath.access$500(FilePath.java:213)
      	at hudson.FilePath$UntarRemote.invoke(FilePath.java:633)
      	at hudson.FilePath$UntarRemote.invoke(FilePath.java:622)
      	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3338)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
      	at hudson.remoting.Request$2.run(Request.java:376)
      	at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
      	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
      	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
      	at java.base/java.lang.Thread.run(Thread.java:833)
      

      Started just after the controller was upgraded to the security release

      cc danielbeck wfollonier jglick

          [JENKINS-67063] Unstash with symlinks fails

          Tim Jacomb created issue -
          Tim Jacomb made changes -
          Description Original: See https://github.com/jenkinsci/jenkins/runs/4107177325
          https://github.com/jenkinsci/jenkins/pull/5881/checks?check_run_id=4107861004

          Started just after the controller was upgraded to the security release

          cc [~danielbeck] [~wfollonier] [~jglick]
          New: See https://github.com/jenkinsci/jenkins/runs/4107177325
          https://github.com/jenkinsci/jenkins/pull/5881/checks?check_run_id=4107861004

          Logs:
          https://ci.jenkins.io/job/Core/job/jenkins/job/PR-5881/2/execution/node/116/
          {code}
          java.nio.file.NoSuchFileException: /home/jenkins/workspace/Core_jenkins_PR-5881/ath/athSources/src/main/resources/ath-container/run.sh
          at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
          at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
          at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
          at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219)
          at java.base/java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:484)
          at java.base/java.nio.file.Files.newOutputStream(Files.java:228)
          at hudson.util.IOUtils.copy(IOUtils.java:52)
          at hudson.FilePath.readFromTar(FilePath.java:2862)
          Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to EC2 (aws-us-east-2) - High memory ubuntu 20.04 (i-02b6bda105ac6c6d9)
          at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1797)
          at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
          at hudson.remoting.Channel.call(Channel.java:1001)
          at hudson.FilePath.act(FilePath.java:1166)
          at hudson.FilePath.act(FilePath.java:1155)
          at hudson.FilePath.untar(FilePath.java:617)
          at org.jenkinsci.plugins.workflow.flow.StashManager.unstash(StashManager.java:161)
          at org.jenkinsci.plugins.workflow.support.steps.stash.UnstashStep$Execution.run(UnstashStep.java:77)
          at org.jenkinsci.plugins.workflow.support.steps.stash.UnstashStep$Execution.run(UnstashStep.java:64)
          at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
          at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
          at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
          at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
          at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
          at java.base/java.lang.Thread.run(Thread.java:829)
          Caused: java.io.IOException: Failed to extract athSources.tar.gz
          at hudson.FilePath.readFromTar(FilePath.java:2872)
          at hudson.FilePath.access$500(FilePath.java:213)
          at hudson.FilePath$UntarRemote.invoke(FilePath.java:633)
          at hudson.FilePath$UntarRemote.invoke(FilePath.java:622)
          at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3338)
          at hudson.remoting.UserRequest.perform(UserRequest.java:211)
          at hudson.remoting.UserRequest.perform(UserRequest.java:54)
          at hudson.remoting.Request$2.run(Request.java:376)
          at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
          at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
          at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
          at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
          at java.base/java.lang.Thread.run(Thread.java:833)
          {code}

          Started just after the controller was upgraded to the security release

          cc [~danielbeck] [~wfollonier] [~jglick]

          Tim Jacomb added a comment -

          Reproduced locally with script extracted from pipeline library,

          should run on anything with maven and git installed, inline pipeline script.
          Recommended plugins + pipeline utility steps

          #!/usr/bin/env groovy
          
          /**
           * Wrapper for running the ATH see README and runATH.txt for full documentation
           */
          
          void stashJenkinsWar(String jenkins, String stashName = "jenkinsWar") {
              def isVersionNumber = (jenkins =~ /^\d+([.]\d+)*(-rc[0-9]+[.][0-9a-f]{12})?$/).matches()
              def isLocalJenkins = jenkins.startsWith("file://")
              def mirror = "http://mirrors.jenkins.io/"
          
              def jenkinsURL
          
              if (jenkins == "latest") {
                  jenkinsURL = mirror + "war/latest/jenkins.war"
              } else if (jenkins == "latest-rc") {
                  jenkinsURL = mirror + "/war-rc/latest/jenkins.war"
              } else if (jenkins == "lts") {
                  jenkinsURL = mirror + "war-stable/latest/jenkins.war"
              } else if (jenkins == "lts-rc") {
                  jenkinsURL = mirror + "war-stable-rc/latest/jenkins.war"
              }
          
              if (isLocalJenkins) {
                  if (!fileExists(jenkins - "file://")) {
                      error "Specified Jenkins file does not exists"
                  }
              }
              if (!isVersionNumber && !isLocalJenkins) {
                  if (jenkinsURL == null) {
                      error "Not sure how to interpret $jenkins as a version, alias, or URL"
                  }
                  echo 'Checking whether Jenkins WAR is available…'
                  sh "curl -ILf ${jenkinsURL}"
              }
          
              if (isVersionNumber) {
                  List<String> downloadCommand = [
                          "dependency:copy",
                          "-Dartifact=org.jenkins-ci.main:jenkins-war:${jenkins}:war",
                          "-DoutputDirectory=.",
                          "-Dmdep.stripVersion=true"
                  ]
                  dir("deps") {
                      runMaven(downloadCommand)
                      sh "cp jenkins-war.war jenkins.war"
                      stash includes: 'jenkins.war', name: stashName
                  }
              } else if (isLocalJenkins) {
                  dir(pwd(tmp: true)) {
                      sh "cp ${jenkins - 'file://'} jenkins.war"
                      stash includes: "*.war", name: "jenkinsWar"
                  }
              } else {
                  sh("curl -o jenkins.war -L ${jenkinsURL}")
                  stash includes: '*.war', name: 'jenkinsWar'
              }
          }
          
          def runATH(Map params = [:]) {
              def athUrl = params.get('athUrl', 'https://github.com/jenkinsci/acceptance-test-harness.git')
              def athRevision = params.get('athRevision', 'master')
              def metadataFile = params.get('metadataFile', 'essentials.yml')
              def jenkins = params.get('jenkins', 'latest')
              def jdks = params.get('jdks', [8])
              def athContainerImageTag = params.get("athImage", "jenkins/ath");
              def configFile = params.get("configFile", null)
              def defaultJavaOptions = params.get('javaOptions', ['--no-transfer-progress'])
          
              def mirror = "http://mirrors.jenkins.io/"
              def defaultCategory = "org.jenkinsci.test.acceptance.junit.SmokeTest"
              def metadata
              def athContainerImage
              def isLocalATH
              def isVersionNumber
          
              def athSourcesFolder = "athSources"
          
              def supportedBrowsers = ["firefox"]
              def supportedJdks = [8, 11]
          
              def skipExecution = false
          
              def localPluginsStashName = env.RUN_ATH_LOCAL_PLUGINS_STASH_NAME ?: "localPlugins"
          
          
                  if (!fileExists(metadataFile)) {
                      echo "Skipping ATH execution because the metadata file does not exist. Current value is ${metadataFile}."
                      skipExecution = true
                      return
                  }
          
              stage("Getting ATH sources and Jenkins war") {
                  // Start validation
                  metadata = readYaml(file: metadataFile)?.ath
                  if (metadata == null) {
                      echo "Skipping ATH execution because the metadata file does not contain an ath section"
                      skipExecution = true
                      return
                  }
                  if (metadata == 'default') {
                      echo "Using default configuration for ATH"
                      metadata = [:]
                  } else if (metadata.browsers == null) {
                      echo "The provided metadata file does not include the browsers property, using firefox as default"
                  }
                  // Allow to override athUrl and athRevision from metadata file
                  athUrl = metadata.athUrl ?: athUrl
                  isLocalATH = athUrl.startsWith("file://")
                  athRevision = metadata.athRevision ?: athRevision
                  athContainerImageTag = metadata.athImage ?: athContainerImageTag
          
                  // Allow override of jenkins version from metadata file
                  jenkins = metadata.jenkins ?: jenkins
                  isVersionNumber = (jenkins =~ /^\d+([.]\d+)*$/).matches()
          
                  // Allow override of JDK version from metadata file
                  jdks = metadata.jdks ?: jdks
          
                  if (!isLocalATH) {
                      echo 'Checking connectivity to ATH sources…'
                      sh "git ls-remote --exit-code -h ${athUrl}"
                  }
                  stashJenkinsWar(jenkins)
                  // Validation ended
          
                  // ATH
                  if (isLocalATH) { // Deal with already existing ATH sources
                      athSourcesFolder = athUrl - "file://"
                  } else {
                      dir(athSourcesFolder) {
                          checkout changelog: true, poll: false, scm: [$class           : 'GitSCM', branches: [[name: athRevision]],
                                                                       userRemoteConfigs: [[url: athUrl]]]
                      }
                  }
                  dir(athSourcesFolder) {
                      // We may need to run things in parallel later, so avoid several checkouts
                      stash name: "athSources"
                  }
          
              }
          
              
                  if (skipExecution) {
                      return
                  }
                  stage("Running ATH") {
                      dir("athSources") {
                          unstash name: "athSources"
                          if (athContainerImageTag == "local") {
                              echo "'local' ATH container image specified, building it"
                              def uid = sh(script: "id -u", returnStdout: true)
                              def gid = sh(script: "id -g", returnStdout: true)
                              athContainerImage = docker.build('jenkins/ath', "--build-arg=uid='$uid' --build-arg=gid='$gid' -f src/main/resources/ath-container/Dockerfile .")
                          } else {
                              echo "No building ATH docker container image. Using ${athContainerImageTag} as specified"
                              athContainerImage = docker.image(athContainerImageTag)
                              athContainerImage.pull() //Use the latest available version
                          }
                      }
          
                      def testsToRun = metadata.tests?.join(",")
                      def categoriesToRun = metadata.categories?.join(",")
                      def browsers = metadata.browsers ?: ["firefox"]
                      def failFast = metadata.failFast ?: false
                      def rerunCount = metadata.rerunFailingTestsCount ?: 0
                      // Elvis fails in case useLocalSnapshots == false in metadata File
                      def localSnapshots = metadata.useLocalSnapshots != null ? metadata.useLocalSnapshots : true
          
                      if (testsToRun == null && categoriesToRun == null) {
                          categoriesToRun = defaultCategory
                      }
          
                      // Shorthand for running all tests
                      if (testsToRun == "all") {
                          testsToRun = categoriesToRun = null
                      }
          
                      def testingbranches = ["failFast": failFast]
                      for (jdk in jdks) {
                          if (supportedJdks.contains(jdk)) {
                              def currentJdk = jdk
                              def javaOptions = defaultJavaOptions.clone()
                              //Add shm-size to avoid selenium.WebDriverException exceptions like 'Failed to decode response from marionette' and webdriver closed
                              def containerArgs = "-v /var/run/docker.sock:/var/run/docker.sock -u ath-user --shm-size 2g"
          
                              if(configFile) {
                                  containerArgs += " -e CONFIG=../${configFile}" // ATH runs are executed in a subfolder, hence path needs to take that into account
                              }
          
                              // Add options for jdks
                              if ( currentJdk  > 8) {
                                  // Add environment variable
                                  containerArgs += " -e java_version=${currentJdk}"
                              }
          
                              for (browser in browsers) {
                                  if (supportedBrowsers.contains(browser)) {
                                      def currentBrowser = browser
          
                                      def commandBase = "./run.sh ${currentBrowser} ./jenkins.war -B -ntp -Dmaven.test.failure.ignore=true -DforkCount=1 -Dsurefire.rerunFailingTestsCount=${rerunCount}"
          
                                      if (testsToRun) {
                                          testingbranches["ATH individual tests-${currentBrowser}-jdk${currentJdk}"] = {
                                              dir("test${currentBrowser}jdk${currentJdk}") {
                                                  def discriminator = "-Dtest=${testsToRun}"
                                                  test(discriminator, commandBase, localSnapshots, localPluginsStashName, containerArgs, athContainerImage, javaOptions)
                                              }
                                          }
                                      }
                                      if (categoriesToRun) {
                                          testingbranches["ATH categories-${currentBrowser}-jdk${currentJdk}"] = {
                                              dir("categories${currentBrowser}jdk${currentJdk}") {
                                                  def discriminator = "-Dgroups=${categoriesToRun}"
                                                  test(discriminator, commandBase, localSnapshots, localPluginsStashName, containerArgs, athContainerImage, javaOptions)
                                              }
                                          }
                                      }
                                      if (testsToRun == null && categoriesToRun == null) {
                                          testingbranches["ATH ${currentBrowser}-jdk${currentJdk}"] = {
                                              dir("ath${currentBrowser}${currentJdk}") {
                                                  def discriminator = ""
                                                  test(discriminator, commandBase, localSnapshots, localPluginsStashName, containerArgs, athContainerImage, javaOptions)
                                              }
                                          }
                                      }
          
                                  } else {
                                      echo "${browser} is not yet supported"
                                  }
                              }
                          } else {
                              echo "${jdk} is not yet supported"
                          }
                      }
          
                      parallel testingbranches
                  }
          }
          
          private void test(discriminator, commandBase, localSnapshots, localPluginsStashName, containerArgs, athContainerImage, javaOptions) {
              unstashResources(localSnapshots, localPluginsStashName)
              athContainerImage.inside(containerArgs) {
                  realtimeJUnit(testResults: 'target/surefire-reports/TEST-*.xml', testDataPublishers: [[$class: 'AttachmentPublisher']]) {
                      /*
                      If you want to compile with java > 8 and have all needed, do it
                      If you want to compile with java > 8 and you lack the set-java script, FAILURE
                      If other case (java8 and lack set-java), write a message in the log
                       */
                      def command = '''
                      if [ -x ./set-java.sh ] && [ -n "$java_version" ] && [ "$java_version" -gt "8" ]; then
                          ./set-java.sh $java_version;
                      elif [ -n "$java_version" ] && [ "$java_version" -gt "8" ] && [ ! -x ./set-java.sh ]; then
                          echo "ERROR: ./set-java.sh not found because you are using old ATH sources and \\$java_version = $java_version specified. We cannot run on this JDK";
                          exit 1;
                      elif [ ! -x ./set-java.sh ]; then
                          echo "INFO: ./set-java.sh not found because you are using old ATH sources, please consider to update ATH sources and docker image";
                      fi
          
                      eval "$(./vnc.sh)" \
                      && export DISPLAY=$BROWSER_DISPLAY \
                      && export SHARED_DOCKER_SERVICE=true \
                      && export EXERCISEDPLUGINREPORTER=textfile \
                      && \
                      '''
          
                      command += prepareCommand(commandBase, discriminator, localSnapshots, localPluginsStashName)
                      if (!javaOptions.isEmpty()) {
                          command = """export JAVA_OPTS="${javaOptions.join(' ')}" && ${command}"""
                      }
          
                      sh command
                  }
              }
          }
          
          private String prepareCommand(commandBase, discriminator, localSnapshots, localPluginsStashName ) {
              def command = commandBase + " ${discriminator}"
              if (localSnapshots && localPluginsStashName) {
                  command = "LOCAL_JARS=${getLocalPluginsList()} " + command
              }
              command
          }
          
          private void unstashResources(localSnapshots, localPluginsStashName) {
              unstash name: "athSources"
              unstash name: "jenkinsWar"
              dir("localPlugins") {
                  if (localSnapshots && localPluginsStashName) {
                      unstash name: localPluginsStashName
                  }
              }
          }
          
          private String getLocalPluginsList() {
              return sh(script : "ls -p -d -1 localPlugins/*.* | tr '\n' ':'| sed 's/.\$//'", returnStdout: true).trim()
          }
          
          node("docker-highmem") {
              // Just to be safe
              deleteDir()
              def fileUri
              def metadataPath
              dir("sources") {
                  git 'https://github.com/jenkinsci/jenkins.git'
                  sh 'mvn --batch-mode --show-version -ntp -Pquick-build -am -pl war package'
                  dir("war/target") {
                      fileUri = "file://" + pwd() + "/jenkins.war"
                  }
                  metadataPath = pwd() + "/essentials.yml"
              }
              dir("ath") {
                  runATH jenkins: fileUri, metadataFile: metadataPath
              }
          }
          

          Tim Jacomb added a comment - Reproduced locally with script extracted from pipeline library, should run on anything with maven and git installed, inline pipeline script. Recommended plugins + pipeline utility steps #!/usr/bin/env groovy /** * Wrapper for running the ATH see README and runATH.txt for full documentation */ void stashJenkinsWar( String jenkins, String stashName = "jenkinsWar" ) { def isVersionNumber = (jenkins =~ /^\d+([.]\d+)*(-rc[0-9]+[.][0-9a-f]{12})?$/).matches() def isLocalJenkins = jenkins.startsWith( "file: //" ) def mirror = "http: //mirrors.jenkins.io/" def jenkinsURL if (jenkins == "latest" ) { jenkinsURL = mirror + "war/latest/jenkins.war" } else if (jenkins == "latest-rc" ) { jenkinsURL = mirror + "/war-rc/latest/jenkins.war" } else if (jenkins == "lts" ) { jenkinsURL = mirror + "war-stable/latest/jenkins.war" } else if (jenkins == "lts-rc" ) { jenkinsURL = mirror + "war-stable-rc/latest/jenkins.war" } if (isLocalJenkins) { if (!fileExists(jenkins - "file: //" )) { error "Specified Jenkins file does not exists" } } if (!isVersionNumber && !isLocalJenkins) { if (jenkinsURL == null ) { error "Not sure how to interpret $jenkins as a version, alias, or URL" } echo 'Checking whether Jenkins WAR is available…' sh "curl -ILf ${jenkinsURL}" } if (isVersionNumber) { List< String > downloadCommand = [ "dependency:copy" , "-Dartifact=org.jenkins-ci.main:jenkins-war:${jenkins}:war" , "-DoutputDirectory=." , "-Dmdep.stripVersion= true " ] dir( "deps" ) { runMaven(downloadCommand) sh "cp jenkins-war.war jenkins.war" stash includes: 'jenkins.war' , name: stashName } } else if (isLocalJenkins) { dir(pwd(tmp: true )) { sh "cp ${jenkins - 'file: //' } jenkins.war" stash includes: "*.war" , name: "jenkinsWar" } } else { sh( "curl -o jenkins.war -L ${jenkinsURL}" ) stash includes: '*.war' , name: 'jenkinsWar' } } def runATH(Map params = [:]) { def athUrl = params.get( 'athUrl' , 'https: //github.com/jenkinsci/acceptance-test-harness.git' ) def athRevision = params.get( 'athRevision' , 'master' ) def metadataFile = params.get( 'metadataFile' , 'essentials.yml' ) def jenkins = params.get( 'jenkins' , 'latest' ) def jdks = params.get( 'jdks' , [8]) def athContainerImageTag = params.get( "athImage" , "jenkins/ath" ); def configFile = params.get( "configFile" , null ) def defaultJavaOptions = params.get( 'javaOptions' , [ '--no-transfer-progress' ]) def mirror = "http: //mirrors.jenkins.io/" def defaultCategory = "org.jenkinsci.test.acceptance.junit.SmokeTest" def metadata def athContainerImage def isLocalATH def isVersionNumber def athSourcesFolder = "athSources" def supportedBrowsers = [ "firefox" ] def supportedJdks = [8, 11] def skipExecution = false def localPluginsStashName = env.RUN_ATH_LOCAL_PLUGINS_STASH_NAME ?: "localPlugins" if (!fileExists(metadataFile)) { echo "Skipping ATH execution because the metadata file does not exist. Current value is ${metadataFile}." skipExecution = true return } stage( "Getting ATH sources and Jenkins war" ) { // Start validation metadata = readYaml(file: metadataFile)?.ath if (metadata == null ) { echo "Skipping ATH execution because the metadata file does not contain an ath section" skipExecution = true return } if (metadata == ' default ' ) { echo "Using default configuration for ATH" metadata = [:] } else if (metadata.browsers == null ) { echo "The provided metadata file does not include the browsers property, using firefox as default " } // Allow to override athUrl and athRevision from metadata file athUrl = metadata.athUrl ?: athUrl isLocalATH = athUrl.startsWith( "file: //" ) athRevision = metadata.athRevision ?: athRevision athContainerImageTag = metadata.athImage ?: athContainerImageTag // Allow override of jenkins version from metadata file jenkins = metadata.jenkins ?: jenkins isVersionNumber = (jenkins =~ /^\d+([.]\d+)*$/).matches() // Allow override of JDK version from metadata file jdks = metadata.jdks ?: jdks if (!isLocalATH) { echo 'Checking connectivity to ATH sources…' sh "git ls-remote --exit-code -h ${athUrl}" } stashJenkinsWar(jenkins) // Validation ended // ATH if (isLocalATH) { // Deal with already existing ATH sources athSourcesFolder = athUrl - "file: //" } else { dir(athSourcesFolder) { checkout changelog: true , poll: false , scm: [$class : 'GitSCM' , branches: [[name: athRevision]], userRemoteConfigs: [[url: athUrl]]] } } dir(athSourcesFolder) { // We may need to run things in parallel later, so avoid several checkouts stash name: "athSources" } } if (skipExecution) { return } stage( "Running ATH" ) { dir( "athSources" ) { unstash name: "athSources" if (athContainerImageTag == "local" ) { echo " 'local' ATH container image specified, building it" def uid = sh(script: "id -u" , returnStdout: true ) def gid = sh(script: "id -g" , returnStdout: true ) athContainerImage = docker.build( 'jenkins/ath' , "--build-arg=uid= '$uid' --build-arg=gid= '$gid' -f src/main/resources/ath-container/Dockerfile ." ) } else { echo "No building ATH docker container image. Using ${athContainerImageTag} as specified" athContainerImage = docker.image(athContainerImageTag) athContainerImage.pull() //Use the latest available version } } def testsToRun = metadata.tests?.join( "," ) def categoriesToRun = metadata.categories?.join( "," ) def browsers = metadata.browsers ?: [ "firefox" ] def failFast = metadata.failFast ?: false def rerunCount = metadata.rerunFailingTestsCount ?: 0 // Elvis fails in case useLocalSnapshots == false in metadata File def localSnapshots = metadata.useLocalSnapshots != null ? metadata.useLocalSnapshots : true if (testsToRun == null && categoriesToRun == null ) { categoriesToRun = defaultCategory } // Shorthand for running all tests if (testsToRun == "all" ) { testsToRun = categoriesToRun = null } def testingbranches = [ "failFast" : failFast] for (jdk in jdks) { if (supportedJdks.contains(jdk)) { def currentJdk = jdk def javaOptions = defaultJavaOptions.clone() //Add shm-size to avoid selenium.WebDriverException exceptions like 'Failed to decode response from marionette' and webdriver closed def containerArgs = "-v / var /run/docker.sock:/ var /run/docker.sock -u ath-user --shm-size 2g" if (configFile) { containerArgs += " -e CONFIG=../${configFile}" // ATH runs are executed in a subfolder, hence path needs to take that into account } // Add options for jdks if ( currentJdk > 8) { // Add environment variable containerArgs += " -e java_version=${currentJdk}" } for (browser in browsers) { if (supportedBrowsers.contains(browser)) { def currentBrowser = browser def commandBase = "./run.sh ${currentBrowser} ./jenkins.war -B -ntp -Dmaven.test.failure.ignore= true -DforkCount=1 -Dsurefire.rerunFailingTestsCount=${rerunCount}" if (testsToRun) { testingbranches[ "ATH individual tests-${currentBrowser}-jdk${currentJdk}" ] = { dir( "test${currentBrowser}jdk${currentJdk}" ) { def discriminator = "-Dtest=${testsToRun}" test(discriminator, commandBase, localSnapshots, localPluginsStashName, containerArgs, athContainerImage, javaOptions) } } } if (categoriesToRun) { testingbranches[ "ATH categories-${currentBrowser}-jdk${currentJdk}" ] = { dir( "categories${currentBrowser}jdk${currentJdk}" ) { def discriminator = "-Dgroups=${categoriesToRun}" test(discriminator, commandBase, localSnapshots, localPluginsStashName, containerArgs, athContainerImage, javaOptions) } } } if (testsToRun == null && categoriesToRun == null ) { testingbranches[ "ATH ${currentBrowser}-jdk${currentJdk}" ] = { dir( "ath${currentBrowser}${currentJdk}" ) { def discriminator = "" test(discriminator, commandBase, localSnapshots, localPluginsStashName, containerArgs, athContainerImage, javaOptions) } } } } else { echo "${browser} is not yet supported" } } } else { echo "${jdk} is not yet supported" } } parallel testingbranches } } private void test(discriminator, commandBase, localSnapshots, localPluginsStashName, containerArgs, athContainerImage, javaOptions) { unstashResources(localSnapshots, localPluginsStashName) athContainerImage.inside(containerArgs) { realtimeJUnit(testResults: 'target/surefire-reports/TEST-*.xml' , testDataPublishers: [[$class: 'AttachmentPublisher' ]]) { /* If you want to compile with java > 8 and have all needed, do it If you want to compile with java > 8 and you lack the set-java script, FAILURE If other case (java8 and lack set-java), write a message in the log */ def command = ''' if [ -x ./set-java.sh ] && [ -n "$java_version" ] && [ "$java_version" -gt "8" ]; then ./set-java.sh $java_version; elif [ -n "$java_version" ] && [ "$java_version" -gt "8" ] && [ ! -x ./set-java.sh ]; then echo "ERROR: ./set-java.sh not found because you are using old ATH sources and \\$java_version = $java_version specified. We cannot run on this JDK" ; exit 1; elif [ ! -x ./set-java.sh ]; then echo "INFO: ./set-java.sh not found because you are using old ATH sources, please consider to update ATH sources and docker image" ; fi eval "$(./vnc.sh)" \ && export DISPLAY=$BROWSER_DISPLAY \ && export SHARED_DOCKER_SERVICE= true \ && export EXERCISEDPLUGINREPORTER=textfile \ && \ ''' command += prepareCommand(commandBase, discriminator, localSnapshots, localPluginsStashName) if (!javaOptions.isEmpty()) { command = """export JAVA_OPTS=" ${javaOptions.join( ' ' )} " && ${command}" "" } sh command } } } private String prepareCommand(commandBase, discriminator, localSnapshots, localPluginsStashName ) { def command = commandBase + " ${discriminator}" if (localSnapshots && localPluginsStashName) { command = "LOCAL_JARS=${getLocalPluginsList()} " + command } command } private void unstashResources(localSnapshots, localPluginsStashName) { unstash name: "athSources" unstash name: "jenkinsWar" dir( "localPlugins" ) { if (localSnapshots && localPluginsStashName) { unstash name: localPluginsStashName } } } private String getLocalPluginsList() { return sh(script : "ls -p -d -1 localPlugins/*.* | tr '\n' ':' | sed 's/.\$ //' " , returnStdout: true ).trim() } node( "docker-highmem" ) { // Just to be safe deleteDir() def fileUri def metadataPath dir( "sources" ) { git 'https: //github.com/jenkinsci/jenkins.git' sh 'mvn --batch-mode --show-version -ntp -Pquick-build -am -pl war package ' dir( "war/target" ) { fileUri = "file: //" + pwd() + "/jenkins.war" } metadataPath = pwd() + "/essentials.yml" } dir( "ath" ) { runATH jenkins: fileUri, metadataFile: metadataPath } }

          Jesse Glick added a comment -
          total 16
          drwxrwxr-x 2 jglick jglick 4096 Nov  4 17:43 .
          drwxrwxr-x 4 jglick jglick 4096 Nov  4 17:43 ..
          -rw-rw-r-- 1 jglick jglick 2730 Nov  4 17:43 Dockerfile
          lrwxrwxrwx 1 jglick jglick   39 Nov  4 17:43 run.sh -> src/main/resources/ath-container/run.sh
          lrwxrwxrwx 1 jglick jglick   44 Nov  4 17:43 set-java.sh -> src/main/resources/ath-container/set-java.sh
          -rwxrwxr-x 1 jglick jglick  752 Nov  4 17:43 vnc.sh
          

          Something amiss with symlinks, perhaps due to the new use of canonicalization.

          Jesse Glick added a comment - total 16 drwxrwxr-x 2 jglick jglick 4096 Nov 4 17:43 . drwxrwxr-x 4 jglick jglick 4096 Nov 4 17:43 .. -rw-rw-r-- 1 jglick jglick 2730 Nov 4 17:43 Dockerfile lrwxrwxrwx 1 jglick jglick 39 Nov 4 17:43 run.sh -> src/main/resources/ath-container/run.sh lrwxrwxrwx 1 jglick jglick 44 Nov 4 17:43 set-java.sh -> src/main/resources/ath-container/set-java.sh -rwxrwxr-x 1 jglick jglick 752 Nov 4 17:43 vnc.sh Something amiss with symlinks, perhaps due to the new use of canonicalization.
          Jesse Glick made changes -
          Status Original: Open [ 1 ] New: In Progress [ 3 ]

          Daniel Beck added a comment - - edited

          Run this twice and it will fail:

          node {
              dir('athSources') {
                  checkout changelog: true, poll: false, scm: [$class: 'GitSCM', branches: [[name: 'master']],
                              userRemoteConfigs: [[url: 'https://github.com/jenkinsci/acceptance-test-harness.git']]]
                  stash name: 'athSources'
              }
              dir('foo2') {
                  unstash name: 'athSources'
              }
          }
          

          Daniel Beck added a comment - - edited Run this twice and it will fail: node { dir('athSources') { checkout changelog: true, poll: false, scm: [$class: 'GitSCM', branches: [[name: 'master']], userRemoteConfigs: [[url: 'https://github.com/jenkinsci/acceptance-test-harness.git']]] stash name: 'athSources' } dir('foo2') { unstash name: 'athSources' } }
          Tim Jacomb made changes -
          Summary Original: ATH tests won't start on Jenkins core build New: Multiple checkout with symlinks fails

          Daniel Beck added a comment -

          Reproduced in a test with the full athSources.tar.gz, but all attempts at recreating a tar file from scratch that has the same problem have failed so far.

          Daniel Beck added a comment - Reproduced in a test with the full athSources.tar.gz , but all attempts at recreating a tar file from scratch that has the same problem have failed so far.

          Daniel Beck added a comment -

          It looks like archiving artifacts from agents is unaffected, 2.303.2 and 2.303.3 have the same archived artifacts from a checkout of jenkinsci/acceptance-test-harness.

          Daniel Beck added a comment - It looks like archiving artifacts from agents is unaffected, 2.303.2 and 2.303.3 have the same archived artifacts from a checkout of jenkinsci/acceptance-test-harness .

          Daniel Beck added a comment -

          It looks like archiving artifacts from agents is unaffected

          TBD actually, I'm seeing some weird behavior in my local environment which may be caused by differences between Linux (breaking) and Mac OS (working). Or it's related to stashing/unstashing on controller or agent, unsure.

          If anyone else has a good reproduction, that would be useful.

          Daniel Beck added a comment - It looks like archiving artifacts from agents is unaffected TBD actually, I'm seeing some weird behavior in my local environment which may be caused by differences between Linux (breaking) and Mac OS (working). Or it's related to stashing/unstashing on controller or agent, unsure. If anyone else has a good reproduction, that would be useful.

            danielbeck Daniel Beck
            timja Tim Jacomb
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: