-
Type:
Bug
-
Resolution: Not A Defect
-
Priority:
Critical
-
Component/s: ec2-plugin
Today provisioning of nodes stopped working without any changes on our side with the error
"....AmazonEC2Exception: /dev/sda1 is being assigned to multiple times" (Will attach details in the end.)
The block device setting had worked until today with:
/dev/sda1=:28:true:gp3
We are using T3a Large build nodes based on Amazon's "ubuntu-jammy-22.04-amd64-server-20220912" default image.
Provisioning worked after removing the whole block device line. But of course the root device then was again back to 8GB gp2.
Detailed Log:
Oct 24, 2022 4:52:08 PM INFO hudson.plugins.ec2.EC2Cloud provisionSlaveTemplate{description='SAM', labels='sam'}. Attempting to provision agent needed by excess workload of 1 units
Oct 24, 2022 4:52:08 PM INFO hudson.plugins.ec2.SlaveTemplate getImageGetting image for request {ExecutableUsers: [],Filters: [],ImageIds: [ami-0caef02b518350c8b],Owners: [],}
Oct 24, 2022 4:52:08 PM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfoSlaveTemplate{description='SAM', labels='sam'}. Considering launching
Oct 24, 2022 4:52:08 PM INFO hudson.plugins.ec2.SlaveTemplate setupRootDeviceAMI had /dev/sda1
Oct 24, 2022 4:52:08 PM INFO hudson.plugins.ec2.SlaveTemplate setupRootDevice{DeleteOnTermination: true,SnapshotId: snap-0df1cce044fe2d749,VolumeSize: 8,VolumeType: gp2,Encrypted: false}
Oct 24, 2022 4:52:08 PM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfoSlaveTemplate{description='SAM', labels='sam'}. EBS default encryption value set to: Based on AMI (null)
Oct 24, 2022 4:52:08 PM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfoSlaveTemplate{description='SAM', labels='sam'}. Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Oct 24, 2022 4:52:08 PM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfoSlaveTemplate{description='SAM', labels='sam'}. Looking for existing instances with describe-instance: {Filters: [{Name: image-id,Values: [ami-0caef02b518350c8b]}, {Name: instance-type,Values: [t3a.large]}, {Name: key-name,Values: [ipfolio-devops]}, {Name: tenancy,Values: [default]}, {Name: subnet-id,Values: [subnet-03a03839047540a6b]}, {Name: instance.group-id,Values: [sg-0c023fae74c14c6b2]}, {Name: tag:jenkins_slave_type,Values: [demand_SAM]}, {Name: tag:tr:environment-type,Values: [dev]}, {Name: tag:Name,Values: [jenkins-sam]}, {Name: tag:tr:role,Values: [ci]}, {Name: tag:tr:appFamily,Values: [ipfolio]}, {Name: tag:ca:owner,Values: [unycom-devops@unycom.com]}, {Name: tag:jenkins_server_url,Values: [https://jenkins.dev.ipfolio.com/]}, {Name: tag:tr:appName,Values: [ipfolio]}, {Name: tag:ca:created-by,Values: [jenkins]}, {Name: tag:Type,Values: [ci-runner]}],InstanceIds: [],}
Oct 24, 2022 4:52:09 PM WARNING hudson.plugins.ec2.EC2Cloud provisionSlaveTemplate{description='SAM', labels='sam'}. Exception during provisioning
com.amazonaws.services.ec2.model.AmazonEC2Exception: /dev/sda1 is being assigned to multiple times. (Service: AmazonEC2; Status Code: 400; Error Code: InvalidBlockDeviceMapping; Request ID: d9817a47-2037-4f30-b62c-598f84424175; Proxy: null)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
at com.amazonaws.services.ec2.AmazonEC2Client.doInvoke(AmazonEC2Client.java:34698)
at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:34665)
at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:34654)
at com.amazonaws.services.ec2.AmazonEC2Client.executeRunInstances(AmazonEC2Client.java:33356)
at com.amazonaws.services.ec2.AmazonEC2Client.runInstances(AmazonEC2Client.java:33325)
at hudson.plugins.ec2.SlaveTemplate.provisionOndemand(SlaveTemplate.java:1159)
at hudson.plugins.ec2.SlaveTemplate.provisionOndemand(SlaveTemplate.java:1101)
at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:907)
at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:717)
at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:743)
at hudson.slaves.Cloud.provision(Cloud.java:210)
at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:726)
at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:325)
at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:823)
at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:94)
at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:69)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
