Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-69233

Too many open files caused by ESTABLISHED TCP connections

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • bitbucket-build-status-notifier-plugin, bitbucket-pullrequest-builder-plugin
    • None

      Hello,

      We have been using Jenkins integrated with Bitbucket Cloud to build pull request. 

      Starting from July 22, our Jenkins crash with error "Too Many Open files". Then I increase the limitation from 1024 to 65530 for Tomcat. But the open files handlers keep increasing.

       

      The lsof return many ESTABLISHED sessions which seems hang up there.

      $ lsof -p `ps aux |grep 'Dcatalina.base=/opt/cmtools/tomcat' | grep -v grep | awk '{print $2}'` | grep 'bitbucket.org' | grep ESTABLISHED | wc -l
2221 

$ lsof -p `ps aux |grep 'Dcatalina.base=/opt/cmtools/tomcat' | grep -v grep | awk '{print $2}'` | grep 'bitbucket.org' | grep ESTABLISHED | tail
lsof: WARNING: can't stat() proc file system /run/docker/netns/df773c7e9cdc
      Output information may be incomplete.
lsof: WARNING: can't stat() proc file system /run/docker/netns/569b30af418b
      Output information may be incomplete.
java    26704 arris 6708u     IPv6          202094267       0t0        TCP servassure-jenkins.internal.cloudapp.net:47125->bitbucket.org:https (ESTABLISHED)
java    26704 arris 6712u     IPv6          202543117       0t0        TCP servassure-jenkins.internal.cloudapp.net:38371->bitbucket.org:https (ESTABLISHED)
java    26704 arris 6713u     IPv6          202458979       0t0        TCP servassure-jenkins.internal.cloudapp.net:48186->bitbucket.org:https (ESTABLISHED)
java    26704 arris 6715u     IPv6          202543119       0t0        TCP servassure-jenkins.internal.cloudapp.net:46762->bitbucket.org:https (ESTABLISHED)
java    26704 arris 6716u     IPv6          202543481       0t0        TCP servassure-jenkins.internal.cloudapp.net:51012->bitbucket.org:https (ESTABLISHED)
java    26704 arris 6721u     IPv6          202544138       0t0        TCP servassure-jenkins.internal.cloudapp.net:36599->bitbucket.org:https (ESTABLISHED)
java    26704 arris 6722u     IPv6          202459069       0t0        TCP servassure-jenkins.internal.cloudapp.net:52891->bitbucket.org:https (ESTABLISHED)
java    26704 arris 6723u     IPv6          202543220       0t0        TCP servassure-jenkins.internal.cloudapp.net:53959->bitbucket.org:https (ESTABLISHED)
java    26704 arris 6726u     IPv6          202543233       0t0        TCP servassure-jenkins.internal.cloudapp.net:33194->bitbucket.org:https (ESTABLISHED)
java    26704 arris 6727u     IPv6          202459070       0t0        TCP servassure-jenkins.internal.cloudapp.net:48945->bitbucket.org:https (ESTABLISHED)

      In the tomcat log, I could see something about the bitbucket.org.

      02-Aug-2022 07:20:01.531 WARNING \[pool-15-thread-1] bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.bitbucket.ApiClient.send Response status: HTTP/1.1 404 Not Found URI: https://bitbucket.org/api/2.0/repositories/workassureonline/servassure-ui/commit/45b0440398fe/statuses/build/jenkins-038f198e8166b3f7be424472547c004602-Aug-2022 08:56:01.673 WARNING \[pool-15-thread-3] bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.bitbucket.ApiClient.send Response st
atus: HTTP/1.1 404 Not Found URI: https://bitbucket.org/api/2.0/repositories/workassureonline/nxt/commit/5b93a6135c33/statuses/build/b3ef82c04d9
f9a769afcb0ea0e261ed879db754702-Aug-2022 15:20:01.560 WARNING \[pool-15-thread-5] bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.bitbucket.ApiClient.send Response st
atus: HTTP/1.1 404 Not Found URI: https://bitbucket.org/api/2.0/repositories/workassureonline/servassure-ui/commit/a0f920844563/statuses/build/j
enkins-038f198e8166b3f7be424472547c004602-Aug-2022 15:12:01.357 WARNING \[pool-15-thread-3] bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.bitbucket.ApiClient.send Response st
atus: HTTP/1.1 404 Not Found URI: https://bitbucket.org/api/2.0/repositories/workassureonline/docsis-platform/commit/6f811a6cfffc/statuses/build
/581f7c385dab837a504002de5dfbca93102ad50502-Aug-2022 16:10:02.204 WARNING \[pool-15-thread-3] bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.bitbucket.ApiClient.send Response status: HTTP/1.1 404 Not Found URI: https://bitbucket.org/api/2.0/repositories/workassureonline/servassure-ui/commit/9833447e4311/statuses/build/jenkins-038f198e8166b3f7be424472547c004602-Aug-2022 16:30:02.043 WARNING \[pool-15-thread-4] bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.bitbucket.ApiClient.send Response status: HTTP/1.1 404 Not Found URI: https://bitbucket.org/api/2.0/repositories/workassureonline/servassure-ui/commit/792b9fd2b70e/statuses/build/jenkins-038f198e8166b3f7be424472547c004602-Aug-2022 17:10:01.558 WARNING \[pool-15-thread-4] bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.bitbucket.ApiClient.send Response status: HTTP/1.1 404 Not Found URI: https://bitbucket.org/api/2.0/repositories/workassureonline/servassure-ui/commit/a090cc173719/statuses/build/jenkins-038f198e8166b3f7be424472547c004602-Aug-2022 17:10:01.938 WARNING \[pool-15-thread-4] bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.bitbucket.ApiClient.send Response status: HTTP/1.1 404 Not Found URI: https://bitbucket.org/api/2.0/repositories/workassureonline/servassure-ui/commit/b0a78850e9e5/statuses/build/jenkins-038f198e8166b3f7be424472547c004602-Aug-2022 17:30:02.212 WARNING \[pool-15-thread-2] bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.bitbucket.ApiClient.send Response status: HTTP/1.1 404 Not Found URI: https://bitbucket.org/api/2.0/repositories/workassureonline/servassure-ui/commit/58d771d4102f/statuses/build/jenkins-038f198e8166b3f7be424472547c004602-Aug-2022 17:40:01.934 WARNING \[pool-15-thread-3] bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.bitbucket.ApiClient.send Response status: HTTP/1.1 404 Not Found URI: https://bitbucket.org/api/2.0/repositories/workassureonline/servassure-ui/commit/f105ef5a3e84/statuses/build/jenkins-038f198e8166b3f7be424472547c004602-Aug-2022 18:10:01.972 WARNING \[pool-15-thread-4] bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.bitbucket.ApiClient.send Response status: HTTP/1.1 404 Not Found URI: https://bitbucket.org/api/2.0/repositories/workassureonline/servassure-ui/commit/15c7cc5cf78f/statuses/build/jenkins-038f198e8166b3f7be424472547c004602-Aug-2022 19:10:01.501 WARNING \[pool-15-thread-1] bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.bitbucket.ApiClient.send Response status: HTTP/1.1 404 Not Found URI: https://bitbucket.org/api/2.0/repositories/workassureonline/servassure-ui/commit/af37358ac2bb/statuses/build/jenkins-038f198e8166b3f7be424472547c004602-Aug-2022 19:40:02.064 WARNING \[pool-15-thread-2] bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.bitbucket.ApiClient.send Response status: HTTP/1.1 404 Not Found URI: https://bitbucket.org/api/2.0/repositories/workassureonline/servassure-ui/commit/6b50c174f1fa/statuses/build/jenkins-038f198e8166b3f7be424472547c004602-Aug-2022 21:40:01.475 WARNING \[pool-15-thread-5] bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.bitbucket.ApiClient.send Response status: HTTP/1.1 404 Not Found URI: https://bitbucket.org/api/2.0/repositories/workassureonline/servassure-ui/commit/9e54a7acd904/statuses/build/jenkins-038f198e8166b3f7be424472547c004602-Aug-2022 23:00:01.499 WARNING \[pool-15-thread-5] bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.bitbucket.ApiClient.send Response status: HTTP/1.1 404 Not Found URI: https://bitbucket.org/api/2.0/repositories/workassureonline/servassure-ui/commit/68bacef32270/statuses/build/jenkins-038f198e8166b3f7be424472547c004602-Aug-2022 23:20:01.506 WARNING \[pool-15-thread-1] bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.bitbucket.ApiClient.send Response status: HTTP/1.1 404 Not Found URI: https://bitbucket.org/api/2.0/repositories/workassureonline/servassure-ui/commit/47a1fc8a4b6d/statuses/build/jenkins-038f198e8166b3f7be424472547c004602-Aug-2022 23:20:01.948 WARNING \[pool-15-thread-1] bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.bitbucket.ApiClient.send Response status: HTTP/1.1 404 Not Found URI: https://bitbucket.org/api/2.0/repositories/workassureonline/servassure-ui/commit/ceae88727458/statuses/build/jenkins-038f198e8166b3f7be424472547c0046

      I have check related two Jenkins plugins https://plugins.jenkins.io/bitbucket-build-status-notifier/ and https://plugins.jenkins.io/bitbucket-pullrequest-builder/ but don't find out any clue.

      I don't understand these log and ESTABLISHED sessions.

      The open file number keep increasing and I need to kill tomcat period then restart it.

      Appreciate for any help on this!

      Thanks,

      Darren

        1. catalina.out
          3.57 MB
        2. jenkins_open_files_num.txt
          3 kB

            flagbit Antonio Mansilla
            dxie Darren Xie
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: