Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65488

Gradle Artifactory plugin hangs in "run step" with Kubernetes Pipeline

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Hello,

      I have an issue that I'm also following with JFrog for some time now regarding a gradle build pipeline using Kubernetes Pipeline DSL.

       

      To explain the architecture:

      1. I have a Kubernetes cluster where a Gitlab and Jenkins instance are running in pods.
      2. This cluster is offline (does not have permissions to go outside)
      3. We use an Artifactory on-prem instance that is allowed to go outside to retrieve dependencies or store build artifacts
      4. The cluster is allowed connected to Artifactory through 443 port
      5. The kubernetes cluster doesn't have any special rules or network configurations aside the VM's where they are on

      When doing a build using the commands provided with the plugin for Maven with this architecture the build runs smoothly and whether it fails or passes, it finalizes the build. However when using the subset of gradle pipeline code it hangs in the running after the build passes or fails.

       

      From JFrog support, they've suggested that the plugin section of Gradle might have some conflict with Kubernetes plugin.

       

      Snippet of build hanging ~30 min before being aborted:

      11:15:02  2021-04-27T10:15:02.020+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed2021-04-27T10:15:02.080+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 
      11:15:02  2021-04-27T10:15:02.081+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] FAILURE: Build failed with an exception.
      11:15:02  2021-04-27T10:15:02.081+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 
      11:15:02  2021-04-27T10:15:02.081+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * Where:
      11:15:02  2021-04-27T10:15:02.081+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Build file '/home/jenkins/agent/workspace/project-examples-pipeline/build.gradle' line: 58
      11:15:02  2021-04-27T10:15:02.081+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 
      11:15:02  2021-04-27T10:15:02.081+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * What went wrong:
      11:15:02  2021-04-27T10:15:02.081+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] A problem occurred evaluating root project 'project-examples-pipeline'.
      11:15:02  2021-04-27T10:15:02.081+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] > Plugin with id 'maven' not found.
      11:15:02  2021-04-27T10:15:02.081+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 
      11:15:02  2021-04-27T10:15:02.081+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * Try:
      11:15:02  2021-04-27T10:15:02.081+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Run with --stacktrace option to get the stack trace.  Run with --scan to get full insights.
      11:15:02  2021-04-27T10:15:02.082+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 
      11:15:02  2021-04-27T10:15:02.082+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * Get more help at https://help.gradle.org
      11:15:02  2021-04-27T10:15:02.082+0000 [ERROR] [org.gradle.internal.buildevents.BuildResultLogger] 
      11:15:02  2021-04-27T10:15:02.082+0000 [ERROR] [org.gradle.internal.buildevents.BuildResultLogger] BUILD FAILED in 9s
      11:15:02  
      11:15:02  2021-04-27T10:15:02.020+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Register task :services:webservice:test' completed
      11:15:02  2021-04-27T10:15:02.020+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Register task :services:webservice:javadoc' completed
      11:15:02  2021-04-27T10:15:02.020+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Register task :services:webservice:jar' completed
      11:15:02  2021-04-27T10:15:02.020+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed
      11:15:02  2021-04-27T10:15:02.021+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed
      11:15:02  2021-04-27T10:15:02.021+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed
      11:15:02  2021-04-27T10:15:02.021+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed
      11:15:02  2021-04-27T10:15:02.021+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed
      11:15:02  2021-04-27T10:15:02.021+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed
      11:15:02  2021-04-27T10:15:02.021+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed
      11:15:02  2021-04-27T10:15:02.021+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed
      11:15:02  2021-04-27T10:15:02.021+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed
      11:15:02  2021-04-27T10:15:02.021+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed
      11:15:02  2021-04-27T10:15:02.021+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed
      11:15:02  2021-04-27T10:15:02.022+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed
      11:15:02  2021-04-27T10:15:02.022+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed
      11:15:02  2021-04-27T10:15:02.022+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed
      11:15:02  2021-04-27T10:15:02.022+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed
      11:15:02  2021-04-27T10:15:02.022+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed
      11:15:02  2021-04-27T10:15:02.022+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed
      11:15:02  2021-04-27T10:15:02.022+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed
      11:15:02  2021-04-27T10:15:02.025+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed
      11:15:02  2021-04-27T10:15:02.025+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Apply plugin org.gradle.java to project ':services:webservice'' completed
      11:15:02  2021-04-27T10:15:02.026+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Cross-configure project :services:webservice' completed
      11:15:02  2021-04-27T10:15:02.026+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute 'allprojects {}' action' completed
      11:15:02  2021-04-27T10:15:02.038+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Cross-configure project :api' completed
      11:15:02  2021-04-27T10:15:02.038+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute 'subprojects {}' action' completed
      11:15:02  2021-04-27T10:15:02.038+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Apply build file 'build.gradle' to root project 'project-examples-pipeline'' completed
      11:15:02  2021-04-27T10:15:02.038+0000 [DEBUG] [org.gradle.configuration.project.BuildScriptProcessor] Timing: Running the build script took 5.626 secs
      11:15:02  2021-04-27T10:15:02.069+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute Gradle.addProjectEvaluationListener listener' completed
      11:15:02  2021-04-27T10:15:02.069+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Notify afterEvaluate listeners of :' completed
      11:15:02  2021-04-27T10:15:02.069+0000 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Daemon worker: released lock on :
      11:15:02  2021-04-27T10:15:02.070+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Completing Build operation 'Configure project :'
      11:15:02  2021-04-27T10:15:02.070+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Configure project :' completed
      11:15:02  2021-04-27T10:15:02.070+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Completing Build operation 'Configure build'
      11:15:02  2021-04-27T10:15:02.070+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Configure build' completed
      11:15:02  2021-04-27T10:15:02.074+0000 [DEBUG] [org.gradle.internal.work.DefaultWorkerLeaseService] Worker lease root.1 completed (0 worker(s) in use)
      11:15:02  2021-04-27T10:15:02.074+0000 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Daemon worker: released lock on root.1
      11:15:02  2021-04-27T10:15:02.084+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Build finished for file system watching' started
      11:15:02  2021-04-27T10:15:02.084+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Completing Build operation 'Build finished for file system watching'
      11:15:02  2021-04-27T10:15:02.084+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Build finished for file system watching' completed
      11:15:02  2021-04-27T10:15:02.084+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Completing Build operation 'Run build'
      11:15:02  2021-04-27T10:15:02.084+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Run build' completed
      11:15:02  2021-04-27T10:15:02.123+0000 [DEBUG] [org.gradle.cache.internal.LockOnDemandCrossProcessCacheAccess] Releasing file lock for cache directory md-supplier (/home/cicd/jenkins/.gradle/caches/7.0/md-supplier)
      11:15:02  2021-04-27T10:15:02.124+0000 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on cache directory md-supplier (/home/cicd/jenkins/.gradle/caches/7.0/md-supplier).
      11:15:02  2021-04-27T10:15:02.129+0000 [DEBUG] [org.gradle.cache.internal.LockOnDemandCrossProcessCacheAccess] Releasing file lock for Build Output Cleanup Cache (/home/jenkins/agent/workspace/project-examples-pipeline/.gradle/buildOutputCleanup)
      11:15:02  2021-04-27T10:15:02.130+0000 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on Build Output Cleanup Cache (/home/jenkins/agent/workspace/project-examples-pipeline/.gradle/buildOutputCleanup).
      11:15:02  2021-04-27T10:15:02.150+0000 [DEBUG] [org.gradle.cache.internal.DefaultCacheAccess] Cache Configuration Cache (/home/jenkins/agent/workspace/project-examples-pipeline/.gradle/configuration-cache) was closed 0 times.
      11:15:02  2021-04-27T10:15:02.150+0000 [DEBUG] [org.gradle.cache.internal.LockOnDemandCrossProcessCacheAccess] Releasing file lock for cache directory md-rule (/home/cicd/jenkins/.gradle/caches/7.0/md-rule)
      11:15:02  2021-04-27T10:15:02.151+0000 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on cache directory md-rule (/home/cicd/jenkins/.gradle/caches/7.0/md-rule).
      11:15:02  2021-04-27T10:15:02.151+0000 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.resolveengine.store.CachedStoreFactory] Resolution result cache closed. Cache reads: 0, disk reads: 0 (avg: 0.0 secs, total: 0.0 secs)
      11:15:02  2021-04-27T10:15:02.152+0000 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.resolveengine.store.CachedStoreFactory] Resolution result cache closed. Cache reads: 0, disk reads: 0 (avg: 0.0 secs, total: 0.0 secs)
      11:15:02  2021-04-27T10:15:02.152+0000 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.resolveengine.store.ResolutionResultsStoreFactory] Deleted 2 resolution results binary files in 0.0 secs
      11:15:02  2021-04-27T10:15:02.154+0000 [DEBUG] [org.gradle.deployment.internal.DefaultDeploymentRegistry] Stopping 0 deployment handles
      11:15:02  2021-04-27T10:15:02.154+0000 [DEBUG] [org.gradle.deployment.internal.DefaultDeploymentRegistry] Stopped deployment handles
      11:15:02  2021-04-27T10:15:02.155+0000 [DEBUG] [org.gradle.cache.internal.LockOnDemandCrossProcessCacheAccess] Releasing file lock for file hash cache (/home/jenkins/agent/workspace/project-examples-pipeline/.gradle/7.0/fileHashes)
      11:15:02  2021-04-27T10:15:02.155+0000 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on file hash cache (/home/jenkins/agent/workspace/project-examples-pipeline/.gradle/7.0/fileHashes).
      11:15:02  2021-04-27T10:15:02.155+0000 [DEBUG] [org.gradle.cache.internal.LockOnDemandCrossProcessCacheAccess] Releasing file lock for checksums cache (/home/jenkins/agent/workspace/project-examples-pipeline/.gradle/checksums)
      11:15:02  2021-04-27T10:15:02.156+0000 [DEBUG] [org.gradle.cache.internal.btree.BTreePersistentIndexedCache] Closing cache md5-checksums.bin (/home/jenkins/agent/workspace/project-examples-pipeline/.gradle/checksums/md5-checksums.bin)
      11:15:02  2021-04-27T10:15:02.156+0000 [DEBUG] [org.gradle.cache.internal.btree.BTreePersistentIndexedCache] Closing cache sha1-checksums.bin (/home/jenkins/agent/workspace/project-examples-pipeline/.gradle/checksums/sha1-checksums.bin)
      11:15:02  2021-04-27T10:15:02.157+0000 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on checksums cache (/home/jenkins/agent/workspace/project-examples-pipeline/.gradle/checksums).
      11:15:02  2021-04-27T10:15:02.157+0000 [DEBUG] [org.gradle.cache.internal.DefaultCacheAccess] Cache VCS Checkout Cache (/home/jenkins/agent/workspace/project-examples-pipeline/.gradle/vcs-1) was closed 0 times.
      11:15:02  2021-04-27T10:15:02.157+0000 [DEBUG] [org.gradle.cache.internal.DefaultCacheAccess] Cache VCS metadata (/home/jenkins/agent/workspace/project-examples-pipeline/.gradle/7.0/vcsMetadata-1) was closed 0 times.
      11:15:02  2021-04-27T10:15:02.157+0000 [DEBUG] [org.gradle.cache.internal.LockOnDemandCrossProcessCacheAccess] Releasing file lock for dependencies-accessors (/home/jenkins/agent/workspace/project-examples-pipeline/.gradle/7.0/dependencies-accessors)
      11:15:02  2021-04-27T10:15:02.157+0000 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on dependencies-accessors (/home/jenkins/agent/workspace/project-examples-pipeline/.gradle/7.0/dependencies-accessors).
      11:15:02  2021-04-27T10:15:02.167+0000 [DEBUG] [org.gradle.cache.internal.VersionSpecificCacheCleanupAction] Processed version-specific caches at /home/jenkins/agent/workspace/project-examples-pipeline/.gradle for cleanup in 0.008 secs
      11:15:02  2021-04-27T10:15:02.182+0000 [DEBUG] [org.gradle.launcher.daemon.server.exec.ExecuteBuild] The daemon has finished executing the build.
      11:15:02  2021-04-27T10:15:02.245+0000 [DEBUG] [org.gradle.launcher.daemon.client.DaemonClientInputForwarder] Dispatching close input message: org.gradle.launcher.daemon.protocol.CloseInput@2a23b6ab
      11:15:02  2021-04-27T10:15:02.245+0000 [DEBUG] [org.gradle.launcher.daemon.client.DaemonClientConnection] thread 26: dispatching class org.gradle.launcher.daemon.protocol.CloseInput
      11:15:02  2021-04-27T10:15:02.246+0000 [DEBUG] [org.gradle.launcher.daemon.client.DaemonClient] Received result Success[value=org.gradle.launcher.exec.BuildActionResult@536f2a7e] from daemon DaemonInfo{pid=108, address=[eb4fa101-6d2e-4837-b041-61d1300d8176 port:39711, addresses:[/127.0.0.1]], state=Busy, lastBusy=1619518493778, context=DefaultDaemonContext[uid=b067fbf9-3fa5-4111-9adb-624d0aeb6be6,javaHome=/apps/java/jdk1.8.0_74,daemonRegistryDir=/home/cicd/jenkins/.gradle/daemon,pid=108,idleTimeout=10800000,priority=NORMAL,daemonOpts=-XX:MaxMetaspaceSize=256m,-XX:+HeapDumpOnOutOfMemoryError,-Xms256m,-Xmx512m,-Dfile.encoding=UTF-8,-Duser.country,-Duser.language=en,-Duser.variant]} (build should be done).
      11:15:02  2021-04-27T10:15:02.246+0000 [DEBUG] [org.gradle.launcher.daemon.client.DaemonClientConnection] thread 1: dispatching class org.gradle.launcher.daemon.protocol.Finished
      11:15:02  2021-04-27T10:15:02.246+0000 [DEBUG] [org.gradle.launcher.daemon.client.DaemonClientConnection] thread 1: connection stop
      11:15:02  2021-04-27T10:15:02.247+0000 [LIFECYCLE] [org.gradle.launcher.cli.DebugLoggerWarningAction] 
      11:15:02  #############################################################################
      11:15:02     WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING
      11:15:02  
      11:15:02     Debug level logging will leak security sensitive information!
      11:15:02  
      11:15:02     https://docs.gradle.org/7.0/userguide/logging.html#sec:debug_security
      11:15:02  #############################################################################
      11:15:02  
      11:42:04  Aborted by harvest
      11:42:04  [Pipeline] }
      11:42:04  [Pipeline] // dir
      11:42:04  [Pipeline] }
      11:42:04  [Pipeline] // script
      11:42:04  [Pipeline] }
      11:42:04  [Pipeline] // container
      

       

      Jenkins Pipeline gradle code snippet used:

      scriptContext.env.JAVA_HOME = scriptContext.tool(JavaVersion.JAVA_8.name)
      
      def rtGradle = scriptContext.Artifactory.newGradleBuild()
      rtGradle.resolver server: server, repo: 'libs-release'
      
      //rtGradle.useWrapper = true
      rtGradle.usesPlugin = true
      rtGradle.deployer server: server, repo:  configArt.release ? configArt.release : "libs-release-local"
      rtGradle.deployer.deployArtifacts = toPublish
      rtGradle.deployer.mavenCompatible = true
      rtGradle.deployer.deployMavenDescriptors = true
      rtGradle.deployer.deployIvyDescriptors = false
      
      rtGradle.tool = jenkinsGradle
      buildInfo = scriptContext.Artifactory.newBuildInfo()
      buildInfo.name = projectName + "-pipeline"
      buildInfo.env.capture = true
      
      buildFile: 'build.gradle', tasks: configTool.tasks, includeEnvVars: true
      buildToolInfo = rtGradle.run rootDir: scriptContext.WORKSPACE + "/", buildFile: 'build.gradle', tasks: configTool.tasks, includeEnvVars: true
      customLogger.log(LogLevel.INFO, "Gradle Plugin Build Finished")
      
      

      Maven Jenkins Pipeline snippet equivalent:

      scriptContext.env.JAVA_HOME = scriptContext.tool(JavaVersion.JAVA_8.name)
      def pomFile = new MavenTool().getPomPath(configTool)
      rtTool = scriptContext.Artifactory.newMavenBuild()
      rtTool.tool = jenkinsMaven
      
      buildInfo = scriptContext.Artifactory.newBuildInfo()
      buildInfo.name = projectName + "-pipeline"
      buildInfo.env.capture = true
      
      rtTool.deployer server: server,
              releaseRepo: configArt.release ? configArt.release : "",
              snapshotRepo: configArt.snapshot ? configArt.snapshot : "",
              includeEnvVars: true
      
      rtTool.deployer.deployArtifacts = toPublish
      
      buildToolInfo = rtTool.run pom: pomFile, goals: configTool.goals, buildInfo: buildInfo
      customLogger.log(LogLevel.INFO, "Maven Build Finished")
      

      Aditional notes:

      • This problems occurs independently of the gradle version being used,  be it via plugin or wrapper (normally all the builds were done via wrapper)
      • This problem didn't happened when Jenkins was installed in a VM (but that is not a viable solution)
      • This problem started happening when we first moved Jenkins to the Kubernetes instance in 1.10
      • The plugins, jenkins and kubernetes are updated regularly 

       

        Attachments

          Activity

          There are no comments yet on this issue.

            People

            Assignee:
            eyalbe Eyal Ben Moshe
            Reporter:
            ricardo_guerreiro Ricardo
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated: