Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65488

Gradle Artifactory plugin hangs in "run step" with Kubernetes Pipeline

XMLWordPrintable

      Hello,

      I have an issue that I'm also following with JFrog for some time now regarding a gradle build pipeline using Kubernetes Pipeline DSL.

       

      To explain the architecture:

      1. I have a Kubernetes cluster where a Gitlab and Jenkins instance are running in pods.
      2. This cluster is offline (does not have permissions to go outside)
      3. We use an Artifactory on-prem instance that is allowed to go outside to retrieve dependencies or store build artifacts
      4. The cluster is allowed connected to Artifactory through 443 port
      5. The kubernetes cluster doesn't have any special rules or network configurations aside the VM's where they are on

      When doing a build using the commands provided with the plugin for Maven with this architecture the build runs smoothly and whether it fails or passes, it finalizes the build. However when using the subset of gradle pipeline code it hangs in the running after the build passes or fails.

       

      From JFrog support, they've suggested that the plugin section of Gradle might have some conflict with Kubernetes plugin.

       

      Snippet of build hanging ~30 min before being aborted:

      11:15:02  2021-04-27T10:15:02.020+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed2021-04-27T10:15:02.080+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 
11:15:02  2021-04-27T10:15:02.081+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] FAILURE: Build failed with an exception.
11:15:02  2021-04-27T10:15:02.081+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 
11:15:02  2021-04-27T10:15:02.081+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * Where:
11:15:02  2021-04-27T10:15:02.081+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Build file '/home/jenkins/agent/workspace/project-examples-pipeline/build.gradle' line: 58
11:15:02  2021-04-27T10:15:02.081+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 
11:15:02  2021-04-27T10:15:02.081+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * What went wrong:
11:15:02  2021-04-27T10:15:02.081+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] A problem occurred evaluating root project 'project-examples-pipeline'.
11:15:02  2021-04-27T10:15:02.081+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] > Plugin with id 'maven' not found.
11:15:02  2021-04-27T10:15:02.081+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 
11:15:02  2021-04-27T10:15:02.081+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * Try:
11:15:02  2021-04-27T10:15:02.081+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Run with --stacktrace option to get the stack trace.  Run with --scan to get full insights.
11:15:02  2021-04-27T10:15:02.082+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 
11:15:02  2021-04-27T10:15:02.082+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * Get more help at https://help.gradle.org
11:15:02  2021-04-27T10:15:02.082+0000 [ERROR] [org.gradle.internal.buildevents.BuildResultLogger] 
11:15:02  2021-04-27T10:15:02.082+0000 [ERROR] [org.gradle.internal.buildevents.BuildResultLogger] BUILD FAILED in 9s
11:15:02  
11:15:02  2021-04-27T10:15:02.020+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Register task :services:webservice:test' completed
11:15:02  2021-04-27T10:15:02.020+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Register task :services:webservice:javadoc' completed
11:15:02  2021-04-27T10:15:02.020+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Register task :services:webservice:jar' completed
11:15:02  2021-04-27T10:15:02.020+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed
11:15:02  2021-04-27T10:15:02.021+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed
11:15:02  2021-04-27T10:15:02.021+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed
11:15:02  2021-04-27T10:15:02.021+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed
11:15:02  2021-04-27T10:15:02.021+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed
11:15:02  2021-04-27T10:15:02.021+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed
11:15:02  2021-04-27T10:15:02.021+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed
11:15:02  2021-04-27T10:15:02.021+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed
11:15:02  2021-04-27T10:15:02.021+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed
11:15:02  2021-04-27T10:15:02.021+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed
11:15:02  2021-04-27T10:15:02.021+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed
11:15:02  2021-04-27T10:15:02.022+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed
11:15:02  2021-04-27T10:15:02.022+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed
11:15:02  2021-04-27T10:15:02.022+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed
11:15:02  2021-04-27T10:15:02.022+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed
11:15:02  2021-04-27T10:15:02.022+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed
11:15:02  2021-04-27T10:15:02.022+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed
11:15:02  2021-04-27T10:15:02.022+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed
11:15:02  2021-04-27T10:15:02.025+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed
11:15:02  2021-04-27T10:15:02.025+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Apply plugin org.gradle.java to project ':services:webservice'' completed
11:15:02  2021-04-27T10:15:02.026+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Cross-configure project :services:webservice' completed
11:15:02  2021-04-27T10:15:02.026+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute 'allprojects {}' action' completed
11:15:02  2021-04-27T10:15:02.038+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Cross-configure project :api' completed
11:15:02  2021-04-27T10:15:02.038+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute 'subprojects {}' action' completed
11:15:02  2021-04-27T10:15:02.038+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Apply build file 'build.gradle' to root project 'project-examples-pipeline'' completed
11:15:02  2021-04-27T10:15:02.038+0000 [DEBUG] [org.gradle.configuration.project.BuildScriptProcessor] Timing: Running the build script took 5.626 secs
11:15:02  2021-04-27T10:15:02.069+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute Gradle.addProjectEvaluationListener listener' completed
11:15:02  2021-04-27T10:15:02.069+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Notify afterEvaluate listeners of :' completed
11:15:02  2021-04-27T10:15:02.069+0000 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Daemon worker: released lock on :
11:15:02  2021-04-27T10:15:02.070+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Completing Build operation 'Configure project :'
11:15:02  2021-04-27T10:15:02.070+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Configure project :' completed
11:15:02  2021-04-27T10:15:02.070+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Completing Build operation 'Configure build'
11:15:02  2021-04-27T10:15:02.070+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Configure build' completed
11:15:02  2021-04-27T10:15:02.074+0000 [DEBUG] [org.gradle.internal.work.DefaultWorkerLeaseService] Worker lease root.1 completed (0 worker(s) in use)
11:15:02  2021-04-27T10:15:02.074+0000 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Daemon worker: released lock on root.1
11:15:02  2021-04-27T10:15:02.084+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Build finished for file system watching' started
11:15:02  2021-04-27T10:15:02.084+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Completing Build operation 'Build finished for file system watching'
11:15:02  2021-04-27T10:15:02.084+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Build finished for file system watching' completed
11:15:02  2021-04-27T10:15:02.084+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Completing Build operation 'Run build'
11:15:02  2021-04-27T10:15:02.084+0000 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Run build' completed
11:15:02  2021-04-27T10:15:02.123+0000 [DEBUG] [org.gradle.cache.internal.LockOnDemandCrossProcessCacheAccess] Releasing file lock for cache directory md-supplier (/home/cicd/jenkins/.gradle/caches/7.0/md-supplier)
11:15:02  2021-04-27T10:15:02.124+0000 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on cache directory md-supplier (/home/cicd/jenkins/.gradle/caches/7.0/md-supplier).
11:15:02  2021-04-27T10:15:02.129+0000 [DEBUG] [org.gradle.cache.internal.LockOnDemandCrossProcessCacheAccess] Releasing file lock for Build Output Cleanup Cache (/home/jenkins/agent/workspace/project-examples-pipeline/.gradle/buildOutputCleanup)
11:15:02  2021-04-27T10:15:02.130+0000 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on Build Output Cleanup Cache (/home/jenkins/agent/workspace/project-examples-pipeline/.gradle/buildOutputCleanup).
11:15:02  2021-04-27T10:15:02.150+0000 [DEBUG] [org.gradle.cache.internal.DefaultCacheAccess] Cache Configuration Cache (/home/jenkins/agent/workspace/project-examples-pipeline/.gradle/configuration-cache) was closed 0 times.
11:15:02  2021-04-27T10:15:02.150+0000 [DEBUG] [org.gradle.cache.internal.LockOnDemandCrossProcessCacheAccess] Releasing file lock for cache directory md-rule (/home/cicd/jenkins/.gradle/caches/7.0/md-rule)
11:15:02  2021-04-27T10:15:02.151+0000 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on cache directory md-rule (/home/cicd/jenkins/.gradle/caches/7.0/md-rule).
11:15:02  2021-04-27T10:15:02.151+0000 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.resolveengine.store.CachedStoreFactory] Resolution result cache closed. Cache reads: 0, disk reads: 0 (avg: 0.0 secs, total: 0.0 secs)
11:15:02  2021-04-27T10:15:02.152+0000 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.resolveengine.store.CachedStoreFactory] Resolution result cache closed. Cache reads: 0, disk reads: 0 (avg: 0.0 secs, total: 0.0 secs)
11:15:02  2021-04-27T10:15:02.152+0000 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.resolveengine.store.ResolutionResultsStoreFactory] Deleted 2 resolution results binary files in 0.0 secs
11:15:02  2021-04-27T10:15:02.154+0000 [DEBUG] [org.gradle.deployment.internal.DefaultDeploymentRegistry] Stopping 0 deployment handles
11:15:02  2021-04-27T10:15:02.154+0000 [DEBUG] [org.gradle.deployment.internal.DefaultDeploymentRegistry] Stopped deployment handles
11:15:02  2021-04-27T10:15:02.155+0000 [DEBUG] [org.gradle.cache.internal.LockOnDemandCrossProcessCacheAccess] Releasing file lock for file hash cache (/home/jenkins/agent/workspace/project-examples-pipeline/.gradle/7.0/fileHashes)
11:15:02  2021-04-27T10:15:02.155+0000 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on file hash cache (/home/jenkins/agent/workspace/project-examples-pipeline/.gradle/7.0/fileHashes).
11:15:02  2021-04-27T10:15:02.155+0000 [DEBUG] [org.gradle.cache.internal.LockOnDemandCrossProcessCacheAccess] Releasing file lock for checksums cache (/home/jenkins/agent/workspace/project-examples-pipeline/.gradle/checksums)
11:15:02  2021-04-27T10:15:02.156+0000 [DEBUG] [org.gradle.cache.internal.btree.BTreePersistentIndexedCache] Closing cache md5-checksums.bin (/home/jenkins/agent/workspace/project-examples-pipeline/.gradle/checksums/md5-checksums.bin)
11:15:02  2021-04-27T10:15:02.156+0000 [DEBUG] [org.gradle.cache.internal.btree.BTreePersistentIndexedCache] Closing cache sha1-checksums.bin (/home/jenkins/agent/workspace/project-examples-pipeline/.gradle/checksums/sha1-checksums.bin)
11:15:02  2021-04-27T10:15:02.157+0000 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on checksums cache (/home/jenkins/agent/workspace/project-examples-pipeline/.gradle/checksums).
11:15:02  2021-04-27T10:15:02.157+0000 [DEBUG] [org.gradle.cache.internal.DefaultCacheAccess] Cache VCS Checkout Cache (/home/jenkins/agent/workspace/project-examples-pipeline/.gradle/vcs-1) was closed 0 times.
11:15:02  2021-04-27T10:15:02.157+0000 [DEBUG] [org.gradle.cache.internal.DefaultCacheAccess] Cache VCS metadata (/home/jenkins/agent/workspace/project-examples-pipeline/.gradle/7.0/vcsMetadata-1) was closed 0 times.
11:15:02  2021-04-27T10:15:02.157+0000 [DEBUG] [org.gradle.cache.internal.LockOnDemandCrossProcessCacheAccess] Releasing file lock for dependencies-accessors (/home/jenkins/agent/workspace/project-examples-pipeline/.gradle/7.0/dependencies-accessors)
11:15:02  2021-04-27T10:15:02.157+0000 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on dependencies-accessors (/home/jenkins/agent/workspace/project-examples-pipeline/.gradle/7.0/dependencies-accessors).
11:15:02  2021-04-27T10:15:02.167+0000 [DEBUG] [org.gradle.cache.internal.VersionSpecificCacheCleanupAction] Processed version-specific caches at /home/jenkins/agent/workspace/project-examples-pipeline/.gradle for cleanup in 0.008 secs
11:15:02  2021-04-27T10:15:02.182+0000 [DEBUG] [org.gradle.launcher.daemon.server.exec.ExecuteBuild] The daemon has finished executing the build.
11:15:02  2021-04-27T10:15:02.245+0000 [DEBUG] [org.gradle.launcher.daemon.client.DaemonClientInputForwarder] Dispatching close input message: org.gradle.launcher.daemon.protocol.CloseInput@2a23b6ab
11:15:02  2021-04-27T10:15:02.245+0000 [DEBUG] [org.gradle.launcher.daemon.client.DaemonClientConnection] thread 26: dispatching class org.gradle.launcher.daemon.protocol.CloseInput
11:15:02  2021-04-27T10:15:02.246+0000 [DEBUG] [org.gradle.launcher.daemon.client.DaemonClient] Received result Success[value=org.gradle.launcher.exec.BuildActionResult@536f2a7e] from daemon DaemonInfo{pid=108, address=[eb4fa101-6d2e-4837-b041-61d1300d8176 port:39711, addresses:[/127.0.0.1]], state=Busy, lastBusy=1619518493778, context=DefaultDaemonContext[uid=b067fbf9-3fa5-4111-9adb-624d0aeb6be6,javaHome=/apps/java/jdk1.8.0_74,daemonRegistryDir=/home/cicd/jenkins/.gradle/daemon,pid=108,idleTimeout=10800000,priority=NORMAL,daemonOpts=-XX:MaxMetaspaceSize=256m,-XX:+HeapDumpOnOutOfMemoryError,-Xms256m,-Xmx512m,-Dfile.encoding=UTF-8,-Duser.country,-Duser.language=en,-Duser.variant]} (build should be done).
11:15:02  2021-04-27T10:15:02.246+0000 [DEBUG] [org.gradle.launcher.daemon.client.DaemonClientConnection] thread 1: dispatching class org.gradle.launcher.daemon.protocol.Finished
11:15:02  2021-04-27T10:15:02.246+0000 [DEBUG] [org.gradle.launcher.daemon.client.DaemonClientConnection] thread 1: connection stop
11:15:02  2021-04-27T10:15:02.247+0000 [LIFECYCLE] [org.gradle.launcher.cli.DebugLoggerWarningAction] 
11:15:02  #############################################################################
11:15:02     WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING
11:15:02  
11:15:02     Debug level logging will leak security sensitive information!
11:15:02  
11:15:02     https://docs.gradle.org/7.0/userguide/logging.html#sec:debug_security
11:15:02  #############################################################################
11:15:02  
11:42:04  Aborted by harvest
11:42:04  [Pipeline] }
11:42:04  [Pipeline] // dir
11:42:04  [Pipeline] }
11:42:04  [Pipeline] // script
11:42:04  [Pipeline] }
11:42:04  [Pipeline] // container

       

      Jenkins Pipeline gradle code snippet used:

      scriptContext.env.JAVA_HOME = scriptContext.tool(JavaVersion.JAVA_8.name)

def rtGradle = scriptContext.Artifactory.newGradleBuild()
rtGradle.resolver server: server, repo: 'libs-release'

//rtGradle.useWrapper = true
rtGradle.usesPlugin = true
rtGradle.deployer server: server, repo:  configArt.release ? configArt.release : "libs-release-local"
rtGradle.deployer.deployArtifacts = toPublish
rtGradle.deployer.mavenCompatible = true
rtGradle.deployer.deployMavenDescriptors = true
rtGradle.deployer.deployIvyDescriptors = false

rtGradle.tool = jenkinsGradle
buildInfo = scriptContext.Artifactory.newBuildInfo()
buildInfo.name = projectName + "-pipeline"
buildInfo.env.capture = true

buildFile: 'build.gradle', tasks: configTool.tasks, includeEnvVars: true
buildToolInfo = rtGradle.run rootDir: scriptContext.WORKSPACE + "/", buildFile: 'build.gradle', tasks: configTool.tasks, includeEnvVars: true
customLogger.log(LogLevel.INFO, "Gradle Plugin Build Finished")

      Maven Jenkins Pipeline snippet equivalent:

      scriptContext.env.JAVA_HOME = scriptContext.tool(JavaVersion.JAVA_8.name)
def pomFile = new MavenTool().getPomPath(configTool)
rtTool = scriptContext.Artifactory.newMavenBuild()
rtTool.tool = jenkinsMaven

buildInfo = scriptContext.Artifactory.newBuildInfo()
buildInfo.name = projectName + "-pipeline"
buildInfo.env.capture = true

rtTool.deployer server: server,
        releaseRepo: configArt.release ? configArt.release : "",
        snapshotRepo: configArt.snapshot ? configArt.snapshot : "",
        includeEnvVars: true

rtTool.deployer.deployArtifacts = toPublish

buildToolInfo = rtTool.run pom: pomFile, goals: configTool.goals, buildInfo: buildInfo
customLogger.log(LogLevel.INFO, "Maven Build Finished")

      Aditional notes:

      • This problems occurs independently of the gradle version being used,  be it via plugin or wrapper (normally all the builds were done via wrapper)
      • This problem didn't happened when Jenkins was installed in a VM (but that is not a viable solution)
      • This problem started happening when we first moved Jenkins to the Kubernetes instance in 1.10
      • The plugins, jenkins and kubernetes are updated regularly 

       

            eyalbe Eyal Ben Moshe
            ricardo_guerreiro Ricardo
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: