When you configure an EC2 Cloud with an IAM Role in the Arn Role field but do not specify a Session Name, the role is not used at all:

https://github.com/jenkinsci/ec2-plugin/blob/1760.vcc93a_2ec6efe/src/main/java/hudson/plugins/ec2/EC2Cloud.java#L1086-L1094

This is quite misleading. The test connection would still work if you the controller has an auth mechanism within AWS (IRSA with Kubernetes, EC2 Instance Profile, ...). And the EC2 Plugin does not give much information about this.

.h3 Workaround

Always specify a Session Name when using Arn Role.

.h3

The Session Name is mandatory for Arn Role when using the com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.Builder, so we should either reflect this requirement in the UI.. Or handle the failure it would result in, an NPE "You must specify a value for roleArn and roleSessionName"

Another Improvement that could help is to have the Test Connection display the (assumed) identity.