-
Bug
-
Resolution: Not A Defect
-
Major
Hi everyone, apologize first for any information leak or mis-understanding, this is my first time to report issue in community. And I have toke merely all way I can to deal with it.
[Environment]
jenkins 2.176.3 with all plugins in latest version
java version (both jenkins master and spot fleet) 1.8.0
– plugins –
ec2-fleet-plugin 1.10.2
git-client 2.8.2
junit 1.28
--------------
Jenkins master on ec2 r5.large, Amazon linux
Spot fleet on r4.large / r5.large / r5.xlarge , Amazon linux
[Scenario]
I use jenkins with spot-fleet plugin for CI
CI all pass in spot-fleet with initial ec2 mechine
While ec2 scale out, CI fail in new instance with "remote call on i-xxxxxx failed"
Below is the log from instance of spot-fleet (retrieved from ${jenkinsHome/remoting/logs})
[Reproduce]
3 CI job with identical codebase run at the same time
a ec2-spot-fleet with init 2 instance
In my case, 1 CI job take 2 ec2 instance, so one job will take the initial two instance, and the other jobs have to run in scaled out instances.
- Result : CI pass in one job and fail in other two
[More information]
1. Below is the jenkinsfile with sensitive information replaced
2. I face the problem in two scenario, because docker build and testing in different spot fleet
scenario A:
building spot-fleet rotate ec2 every day, new ec2 instance fail to build
scenario B:
is the reproduce experiment, happen in spot-fleet used in testing stage
new ec2 (scaled out) fail in CI testing while initial instances success
you can see pipeline with blueOcean in ec2_N_success/fail.png attachment
3. Slave log viewed in jenkins
ErrorInScaledOutInstance.pdf
Hi, this problem have been "SOLVED"
That's because "yum update" when jenkins slave have already run in machine,
and some java lib might be affected, so cause "LinakageError", after update AMI this problem has not be seen anymore.