//代码占位符

node("xxx") {
    logstash {
       xxxx
    }
}

 Backgroud

Above code snippet, show how we use jenkins pipeline on logstash-plugin,  every job is configured to use global decorator.

It works fine until local logstash TCP server is running into some network issue, which is caused by "Too many open files" socket issue. 

IMPACT

The connection timeout takes up to 10 minutes to recover for each line of console log, while we has hundreds of builds running at the same time, this gets even worse because each line in every build is stuck, and they blocks each other. It leads to a CI accident.

REQUIREMENT

Since we use logstash to assist out CI, the logstash/ELK should not affect our main CI business and process, we better set a build scoped connection broken flag for each build, if the connection gets broken, we just skip the log for this build.