Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46120

OOM: MultiJob and adding value to end of environment variable, eg. path

      Multijob fails to handle (global) environment variable that is adding value to end of variable eg. for path.

      Finally the issue seems to be "minor" but I lefted it open because Multijob was only "project model" to fail on this.

      Global properties in system preference:

      LD_LIBRARY_PATH
      /opt/oracle/instantclient:$LD_LIBRARY_PATH

       

      This is from end to change build name which also fails for OOM heap.

      Set build name.
      13:36:46 FATAL: Java heap space
      13:36:46 java.lang.OutOfMemoryError: Java heap space
      13:36:46 	at java.util.Arrays.copyOf(Arrays.java:3332)
      13:36:46 	at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
      13:36:46 	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
      13:36:46 	at java.lang.StringBuilder.append(StringBuilder.java:136)
      13:36:46 	at hudson.Util.replaceMacro(Util.java:180)
      13:36:46 	at hudson.Util.replaceMacro(Util.java:146)
      13:36:46 	at hudson.EnvVars.resolve(EnvVars.java:343)
      13:36:46 	at hudson.model.AbstractBuild.getEnvironment(AbstractBuild.java:886)
      13:36:46 	at org.jenkinsci.plugins.tokenmacro.TokenMacro.expandAll(TokenMacro.java:225)
      13:36:46 	at org.jenkinsci.plugins.tokenmacro.TokenMacro.expandAll(TokenMacro.java:216)
      13:36:46 	at org.jenkinsci.plugins.tokenmacro.TokenMacro.expandAll(TokenMacro.java:208)
      13:36:46 	at org.jenkinsci.plugins.buildnamesetter.BuildNameSetter.setDisplayName(BuildNameSetter.java:74)
      13:36:46 	at org.jenkinsci.plugins.buildnamesetter.BuildNameSetter.access$000(BuildNameSetter.java:31)
      13:36:46 	at org.jenkinsci.plugins.buildnamesetter.BuildNameSetter$1.tearDown(BuildNameSetter.java:64)
      13:36:46 	at hudson.model.Build$BuildExecution.doRun(Build.java:174)
      13:36:46 	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:490)
      13:36:46 	at com.tikal.jenkins.plugins.multijob.MultiJobBuild$MultiJobRunnerImpl.run(MultiJobBuild.java:136)
      13:36:46 	at hudson.model.Run.execute(Run.java:1737)
      13:36:46 	at com.tikal.jenkins.plugins.multijob.MultiJobBuild.run(MultiJobBuild.java:73)
      13:36:46 	at hudson.model.ResourceController.execute(ResourceController.java:97)
      13:36:46 	at hudson.model.Executor.run(Executor.java:419)
      13:36:59 Notifying upstream projects of job completion
      

        1. Biggest objects 2,2GB for MultijobBuild.png
          Biggest objects 2,2GB for MultijobBuild.png
          315 kB
        2. Class based heap usage.png
          Class based heap usage.png
          175 kB
        3. job.xml
          31 kB
        4. job2.xml
          24 kB
        5. Reachability table of heap dump.png
          Reachability table of heap dump.png
          410 kB

          [JENKINS-46120] OOM: MultiJob and adding value to end of environment variable, eg. path

          Heikki Simperi added a comment - - edited

          Added as attachment second plain multijob project xml file with same result. This is console printout:

           

          12:56:55 Started by user Name
          12:56:55 [EnvInject] - Loading node environment variables.
          12:56:55 Building remotely on hara mini (REMOTE_MINI_JOBS JENKINS_MINI) in workspace /mnt/jenkins/mini/workspace/31-PROJ-CUST-deploy-all
          12:56:55 [WS-CLEANUP] Deleting project workspace...
          12:56:55 [WS-CLEANUP] Done
          12:56:55 Copied 2 artifacts from "00-PROJ-build" build number 7957
          12:56:55 [EnvInject] - Injecting environment variables from a build step.
          12:56:55 [EnvInject] - Injecting as environment variables the properties file path 'version.properties'
          12:56:55 [EnvInject] - Variables injected successfully.
          12:56:55 >> Job status: [31-PROJ-CUST-deploy] the 'build only if scm changes' feature is disabled.
          12:56:55 Starting build job 31-PROJ-CUST-deploy.
          12:56:55 >> Job status: [31-PROJ-CUST-deploy] the 'build only if scm changes' feature is disabled.
          12:56:55 Starting build job 31-PROJ-CUST-deploy.
          12:56:55 >> Job status: [31-PROJ-CUST-deploy] the 'build only if scm changes' feature is disabled.
          12:56:55 Starting build job 31-PROJ-CUST-deploy.
          12:56:55 >> Job status: [31-PROJ-CUST-deploy] the 'build only if scm changes' feature is disabled.
          12:56:55 Starting build job 31-PROJ-CUST-deploy.
          12:56:55 >> Job status: [31-PROJ-CUST-deploy] the 'build only if scm changes' feature is disabled.
          12:56:55 Starting build job 31-PROJ-CUST-deploy.
          12:56:55 >> Job status: [31-PROJ-CUST-deploy] the 'build only if scm changes' feature is disabled.
          12:56:55 Starting build job 31-PROJ-CUST-deploy.
          12:56:55 >> Job status: [31-PROJ-CUST-deploy] the 'build only if scm changes' feature is disabled.
          12:56:55 Starting build job 31-PROJ-CUST-deploy.
          12:56:55 >> Job status: [31-PROJ-CUST-deploy] the 'build only if scm changes' feature is disabled.
          12:56:55 Starting build job 31-PROJ-CUST-deploy.
          12:56:55 >> Job status: [31-PROJ-CUST-deploy] the 'build only if scm changes' feature is disabled.
          12:56:55 Starting build job 31-PROJ-CUST-deploy.
          12:56:55 >> Job status: [31-PROJ-CUST-deploy] the 'build only if scm changes' feature is disabled.
          12:56:56 Starting build job 31-PROJ-CUST-deploy.
          12:56:56 >> Job status: [31-PROJ-CUST-deploy] the 'build only if scm changes' feature is disabled.
          12:56:57 Starting build job 31-PROJ-CUST-deploy.
          12:56:57 >> Job status: [31-PROJ-CUST-deploy] the 'build only if scm changes' feature is disabled.
          12:57:00 Starting build job 31-PROJ-CUST-deploy.
          12:57:01 >> Job status: [31-PROJ-CUST-deploy] the 'build only if scm changes' feature is disabled.
          12:57:10 Starting build job 31-PROJ-CUST-deploy.
          12:57:11 >> Job status: [31-PROJ-CUST-deploy] the 'build only if scm changes' feature is disabled.
          12:57:40 Starting build job 31-PROJ-CUST-deploy.
          12:57:44 >> Job status: Warning: [31-PROJ-CUST-deploy] subjob in the phase configuration is disabled.
          12:59:17 >> Job status: [31-PROJ-CUST-deploy] the 'build only if scm changes' feature is disabled.
          12:59:29 [MultiJob] - [ERROR] - Problems occurs on injecting env vars as a build step: Java heap space
          12:59:29 Starting build job 31-PROJ-CUST-deploy.
          12:59:39 FATAL: Java heap space
          12:59:39 java.lang.OutOfMemoryError: Java heap space
          12:59:40 Notifying upstream projects of job completion
          12:59:40 Finished: FAILURE

          Heikki Simperi added a comment - - edited Added as attachment second plain multijob project xml file with same result. This is console printout:   12:56:55 Started by user Name 12:56:55 [EnvInject] - Loading node environment variables. 12:56:55 Building remotely on hara mini (REMOTE_MINI_JOBS JENKINS_MINI) in workspace /mnt/jenkins/mini/workspace/31-PROJ-CUST-deploy-all 12:56:55 [WS-CLEANUP] Deleting project workspace... 12:56:55 [WS-CLEANUP] Done 12:56:55 Copied 2 artifacts from "00-PROJ-build" build number 7957 12:56:55 [EnvInject] - Injecting environment variables from a build step. 12:56:55 [EnvInject] - Injecting as environment variables the properties file path 'version.properties' 12:56:55 [EnvInject] - Variables injected successfully. 12:56:55 >> Job status: [31-PROJ-CUST-deploy] the 'build only if scm changes' feature is disabled. 12:56:55 Starting build job 31-PROJ-CUST-deploy. 12:56:55 >> Job status: [31-PROJ-CUST-deploy] the 'build only if scm changes' feature is disabled. 12:56:55 Starting build job 31-PROJ-CUST-deploy. 12:56:55 >> Job status: [31-PROJ-CUST-deploy] the 'build only if scm changes' feature is disabled. 12:56:55 Starting build job 31-PROJ-CUST-deploy. 12:56:55 >> Job status: [31-PROJ-CUST-deploy] the 'build only if scm changes' feature is disabled. 12:56:55 Starting build job 31-PROJ-CUST-deploy. 12:56:55 >> Job status: [31-PROJ-CUST-deploy] the 'build only if scm changes' feature is disabled. 12:56:55 Starting build job 31-PROJ-CUST-deploy. 12:56:55 >> Job status: [31-PROJ-CUST-deploy] the 'build only if scm changes' feature is disabled. 12:56:55 Starting build job 31-PROJ-CUST-deploy. 12:56:55 >> Job status: [31-PROJ-CUST-deploy] the 'build only if scm changes' feature is disabled. 12:56:55 Starting build job 31-PROJ-CUST-deploy. 12:56:55 >> Job status: [31-PROJ-CUST-deploy] the 'build only if scm changes' feature is disabled. 12:56:55 Starting build job 31-PROJ-CUST-deploy. 12:56:55 >> Job status: [31-PROJ-CUST-deploy] the 'build only if scm changes' feature is disabled. 12:56:55 Starting build job 31-PROJ-CUST-deploy. 12:56:55 >> Job status: [31-PROJ-CUST-deploy] the 'build only if scm changes' feature is disabled. 12:56:56 Starting build job 31-PROJ-CUST-deploy. 12:56:56 >> Job status: [31-PROJ-CUST-deploy] the 'build only if scm changes' feature is disabled. 12:56:57 Starting build job 31-PROJ-CUST-deploy. 12:56:57 >> Job status: [31-PROJ-CUST-deploy] the 'build only if scm changes' feature is disabled. 12:57:00 Starting build job 31-PROJ-CUST-deploy. 12:57:01 >> Job status: [31-PROJ-CUST-deploy] the 'build only if scm changes' feature is disabled. 12:57:10 Starting build job 31-PROJ-CUST-deploy. 12:57:11 >> Job status: [31-PROJ-CUST-deploy] the 'build only if scm changes' feature is disabled. 12:57:40 Starting build job 31-PROJ-CUST-deploy. 12:57:44 >> Job status: Warning: [31-PROJ-CUST-deploy] subjob in the phase configuration is disabled. 12:59:17 >> Job status: [31-PROJ-CUST-deploy] the 'build only if scm changes' feature is disabled. 12:59:29 [MultiJob] - [ERROR] - Problems occurs on injecting env vars as a build step: Java heap space 12:59:29 Starting build job 31-PROJ-CUST-deploy. 12:59:39 FATAL: Java heap space 12:59:39 java.lang.OutOfMemoryError: Java heap space 12:59:40 Notifying upstream projects of job completion 12:59:40 Finished: FAILURE

          It seems that master node is running out of Heap

          Heikki Simperi added a comment - It seems that master node is running out of Heap

          Added multiple screenshots from heap dump

          Heikki Simperi added a comment - Added multiple screenshots from heap dump

          Finally the heap dump shown me the problem. Late I had added global new environment variable for Oracle libraries which caused the problem by "referring" to it self again and again. Multijob plugin "job" failed to handle situation while others worked fine.

          Heikki Simperi added a comment - Finally the heap dump shown me the problem. Late I had added global new environment variable for Oracle libraries which caused the problem by "referring" to it self again and again. Multijob plugin "job" failed to handle situation while others worked fine.

          Closing issue as part of tikal-multijob-plugin issues cleanup.
          If still relevant, please open a matching issue in https://github.com/jenkinsci/tikal-multijob-plugin/issues (you can refer to this issue in its description)

          Yoram Michaeli added a comment - Closing issue as part of tikal-multijob-plugin issues cleanup. If still relevant, please open a matching issue in https://github.com/jenkinsci/tikal-multijob-plugin/issues (you can refer to this issue in its description)

            Unassigned Unassigned
            heikkisi Heikki Simperi
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: