• Icon: Task Task
    • Resolution: Unresolved
    • Icon: Minor Minor
    • ec2-plugin
    • None

      Update the EC2 plugin to use the AWS SDK for Java 2.x as provided by the AWS SDK for Java 2.x API plugin

      Confirm that the EC2 plugin works as expected.

          [JENKINS-73640] Update EC2 plugin to use AWS SDK for Java 2.x

          Basil Crow added a comment -

          ec2-cloud-axis depends on this plugin, so care should be taken to preserve compatibility.

          Basil Crow added a comment - ec2-cloud-axis depends on this plugin, so care should be taken to preserve compatibility.

          Denis Blanchette added a comment - - edited

          I've started work on this. I'm at a point where there are no compilation errors, but some tests fail. For example, CASC configurations are broken because the name of the enums are different for isntance types.

          I'm not sure I'll have time to finish this, but you are welcome to start from where I left of

          I pushed my changes here: 

          https://github.com/coveord/ec2-plugin/tree/feat/aws-sdk-v2

          Denis Blanchette added a comment - - edited I've started work on this. I'm at a point where there are no compilation errors, but some tests fail. For example, CASC configurations are broken because the name of the enums are different for isntance types. I'm not sure I'll have time to finish this, but you are welcome to start from where I left of I pushed my changes here:  https://github.com/coveord/ec2-plugin/tree/feat/aws-sdk-v2

          Basil Crow added a comment -

          Thanks for starting this, dblanchette! I did a little more work on this today and got all the tests passing in https://github.com/jenkinsci/ec2-plugin/pull/1021 by adjusting the CasC tests to expect the new enumeration names. This obviously needs to be improved to be backward-compatible with the old enumeration names before the change can be released, but in the meantime it gives us an incremental build at https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/ec2/1791.vf0f35ec30a_fe/ (Plugin Installation Manager input format: ec2:incrementals;org.jenkins-ci.plugins;1791.vf0f35ec30a_fe) which can be installed as described in https://www.jenkins.io/doc/book/managing/plugins/#advanced-installation for testing purposes. It would be great if you or anyone else could try running that against a real EC2 environment and fix up any remaining problems that are encountered.

          Basil Crow added a comment - Thanks for starting this, dblanchette ! I did a little more work on this today and got all the tests passing in https://github.com/jenkinsci/ec2-plugin/pull/1021 by adjusting the CasC tests to expect the new enumeration names. This obviously needs to be improved to be backward-compatible with the old enumeration names before the change can be released, but in the meantime it gives us an incremental build at https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/ec2/1791.vf0f35ec30a_fe/ (Plugin Installation Manager input format: ec2:incrementals;org.jenkins-ci.plugins;1791.vf0f35ec30a_fe ) which can be installed as described in https://www.jenkins.io/doc/book/managing/plugins/#advanced-installation for testing purposes. It would be great if you or anyone else could try running that against a real EC2 environment and fix up any remaining problems that are encountered.

          Basil Crow added a comment -

          (I also wonder if there might be a compatibility issue with serialized XStream XML data that contains old instance type enumeration values.)

          Basil Crow added a comment - (I also wonder if there might be a compatibility issue with serialized XStream XML data that contains old instance type enumeration values.)

          Basil Crow added a comment -

          Thanks to an additional contribution from dblanchette, there is a new incremental build at https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/ec2/1793.v56038b_2df95c/ (Plugin Installation Manager input format: ec2:incrementals;org.jenkins-ci.plugins;1793.v56038b_2df95c) which is backwards-compatible with XML and CasC data in the old format and can be installed as described in https://www.jenkins.io/doc/book/managing/plugins/#advanced-installation for testing purposes. It would be great if someone could try running that against a real EC2 environment and fix up any remaining problems that are encountered.

          Basil Crow added a comment - Thanks to an additional contribution from dblanchette , there is a new incremental build at https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/ec2/1793.v56038b_2df95c/ (Plugin Installation Manager input format: ec2:incrementals;org.jenkins-ci.plugins;1793.v56038b_2df95c ) which is backwards-compatible with XML and CasC data in the old format and can be installed as described in https://www.jenkins.io/doc/book/managing/plugins/#advanced-installation for testing purposes. It would be great if someone could try running that against a real EC2 environment and fix up any remaining problems that are encountered.

          Denis Blanchette added a comment - - edited

          basil I tested on our former production instance (we do blue-green deployment with CasC)

          It has 10+ Clouds on different AWS accounts. All agents are Linux.

          The config loads correctly, with the new instance type format being used and selected (e.g. m7.large instead of M7Large)

          I get an issue with permissions (Stack trace jenkins_ec2_auth_exception.txt). It seems there is a code path that does not get a client properly before attempting to make calls. The catch at line 567 of EC2Cloud.java also does not seem to be doing anything. I eventually got it to work after using the "Test Connection" and "Check AMI" buttons, but I cannot find a way to do that again after a reboot.

          Denis Blanchette added a comment - - edited basil I tested on our former production instance (we do blue-green deployment with CasC) It has 10+ Clouds on different AWS accounts. All agents are Linux. The config loads correctly, with the new instance type format being used and selected (e.g. m7.large instead of M7Large) I get an issue with permissions (Stack trace jenkins_ec2_auth_exception.txt ). It seems there is a code path that does not get a client properly before attempting to make calls. The catch at line 567 of EC2Cloud.java also does not seem to be doing anything. I eventually got it to work after using the "Test Connection" and "Check AMI" buttons, but I cannot find a way to do that again after a reboot.

          Hi folks, we are interested and motivated to test this plugin on the (new AWS) ci.jenkins.io.

          For context: the Jenkins infra team is currently working on migrating https://ci.jenkins.io from Azure to AWS (for sponsoring reasons).
          It involves switching VM ephemeral agents from Azure VM to EC2 plugin: https://github.com/jenkins-infra/helpdesk/issues/4316

          Since we have successfully created the "blue" ci.jenkins.io in AWS with a first `ec2` Linux template, we can absolutely start using the v2 AWS line in here to live on the edge and provide feedbacks.

          Some elements we'll be able to test:

          • Credential-less IAM authentication (using instance role from the VM IMDS v2 metadatas)
          • Private subnet with private IP
          • Linux instances (and Windows with SSH launcher soon)

          Damien Duportal added a comment - Hi folks, we are interested and motivated to test this plugin on the (new AWS) ci.jenkins.io. For context: the Jenkins infra team is currently working on migrating https://ci.jenkins.io from Azure to AWS (for sponsoring reasons) . It involves switching VM ephemeral agents from Azure VM to EC2 plugin: https://github.com/jenkins-infra/helpdesk/issues/4316 Since we have successfully created the "blue" ci.jenkins.io in AWS with a first `ec2` Linux template, we can absolutely start using the v2 AWS line in here to live on the edge and provide feedbacks. Some elements we'll be able to test: Credential-less IAM authentication (using instance role from the VM IMDS v2 metadatas) Private subnet with private IP Linux instances (and Windows with SSH launcher soon)

          Basil Crow added a comment -

          dblanchette Thanks for testing! It's possible you were running into JENKINS-75014, which I just fixed. If so, you should be able to get the fix by removing the aws-java-sdk2-sts plugin and upgrading the rest to these versions or newer:

          • aws-credentials 239.v6681a_0ea_46ef
          • aws-global-configuration 141.v3a_5c7ded79ee
          • aws-java-sdk2-core 2.29.34-9.v117ff2a_65538
          • aws-java-sdk2-ec2 2.29.34-9.v117ff2a_65538

          Basil Crow added a comment - dblanchette Thanks for testing! It's possible you were running into JENKINS-75014 , which I just fixed. If so, you should be able to get the fix by removing the aws-java-sdk2-sts plugin and upgrading the rest to these versions or newer: aws-credentials 239.v6681a_0ea_46ef aws-global-configuration 141.v3a_5c7ded79ee aws-java-sdk2-core 2.29.34-9.v117ff2a_65538 aws-java-sdk2-ec2 2.29.34-9.v117ff2a_65538

          Basil Crow added a comment -

          laszlog Any interest in testing this? I saw in a different thread that you were eager to upgrade to a release with v2 support. Testing the code in this PR will help us get there.

          Basil Crow added a comment - laszlog Any interest in testing this? I saw in a different thread that you were eager to upgrade to a release with v2 support. Testing the code in this PR will help us get there.

          Basil Crow added a comment -

          dduportal That sounds great! When you're ready, you should be able to install the latest incremental build from https://github.com/jenkinsci/ec2-plugin/pull/1021. Please also ensure that the aws-credentials, aws-global-configuration, and AWS Java SDK 2 plugins are all upgraded to the latest versions (and aws-java-sdk2-sts is uninstalled). I can debug any issues related to the v2 upgrade, but I won't be able to help with issues that are also present in the v1 SDK.

          Basil Crow added a comment - dduportal That sounds great! When you're ready, you should be able to install the latest incremental build from https://github.com/jenkinsci/ec2-plugin/pull/1021 . Please also ensure that the aws-credentials , aws-global-configuration , and AWS Java SDK 2 plugins are all upgraded to the latest versions (and aws-java-sdk2-sts is uninstalled). I can debug any issues related to the v2 upgrade, but I won't be able to help with issues that are also present in the v1 SDK.

          Basil Crow added a comment -

          Deployed version 1793.vd89384144eb_d to Incrementals with the fix for JENKINS-75014. Download link:

          Plugin Installation Manager input format: (documentation): ec2:incrementals;org.jenkins-ci.plugins;1793.vd89384144eb_d

          Basil Crow added a comment - Deployed version 1793.vd89384144eb_d to Incrementals with the fix for JENKINS-75014 . Download link: ec2 Plugin Installation Manager input format: ( documentation ): ec2:incrementals;org.jenkins-ci.plugins;1793.vd89384144eb_d

          basil Unfortunately, that did not work. I get the same error message and I triple-checked that the correct versions were installed and that STS was not.

          I think this might be a cross-account issue, because it works when using the same account the Jenkins controller is located and has an IAM role in, but not in any other account.

          Denis Blanchette added a comment - basil Unfortunately, that did not work. I get the same error message and I triple-checked that the correct versions were installed and that STS was not. I think this might be a cross-account issue, because it works when using the same account the Jenkins controller is located and has an IAM role in, but not in any other account.

          Basil Crow added a comment -

          dblanchette Interesting, I am not very familiar with AWS but I will try to think about this some more. I re-read the code in createCredentialsProvider and found what I think is a bug in how we create the default credentials provider chain. I fixed that in https://github.com/jenkinsci/ec2-plugin/pull/1021/commits/bbe92168a2072d30c1cf72f7f39b37c97cee2014 so maybe you could try again once the build completes and a new incremental is available.

          Basil Crow added a comment - dblanchette Interesting, I am not very familiar with AWS but I will try to think about this some more. I re-read the code in createCredentialsProvider and found what I think is a bug in how we create the default credentials provider chain. I fixed that in https://github.com/jenkinsci/ec2-plugin/pull/1021/commits/bbe92168a2072d30c1cf72f7f39b37c97cee2014 so maybe you could try again once the build completes and a new incremental is available.

          basil It took me a while, but I found the issue. Our controller node is in another region than the agents we are launching. There are places in the code where we pass the default EC2 endpoint (when no override is set). This does not work without a region in a cross-region context.

          I fixed it so that we get the first of: the override endpoint (if provided), the endpoint with the region (if a region has been provided), or the default region-less endpoint.

          I tested a local build with my changes on our former production instance and it works. I tested the "Test Connection" and "Check AMI" buttons, the population of the regions in the dropdown, launching instances in the same account/region (instance profile), and launching instances in a different account and region (instance profile + role).

          My changes are here: https://github.com/coveord/ec2-plugin/tree/fix/JENKINS-73640. Like last time, I made them on top of your branch, so they should merge without issues. Bear in mind I'm more of a Python developer, so I might have made some changes that are not idiomatic. I also left a TODO in the code (I don't know how regions work in Eucalyptus).

          Happy holidays!

          Denis Blanchette added a comment - basil It took me a while, but I found the issue. Our controller node is in another region than the agents we are launching. There are places in the code where we pass the default EC2 endpoint (when no override is set). This does not work without a region in a cross-region context. I fixed it so that we get the first of: the override endpoint (if provided), the endpoint with the region (if a region has been provided), or the default region-less endpoint. I tested a local build with my changes on our former production instance and it works. I tested the "Test Connection" and "Check AMI" buttons, the population of the regions in the dropdown, launching instances in the same account/region (instance profile), and launching instances in a different account and region (instance profile + role). My changes are here: https://github.com/coveord/ec2-plugin/tree/fix/JENKINS-73640 . Like last time, I made them on top of your branch, so they should merge without issues. Bear in mind I'm more of a Python developer, so I might have made some changes that are not idiomatic. I also left a TODO in the code (I don't know how regions work in Eucalyptus). Happy holidays!

          Basil Crow added a comment -

          I also left a TODO in the code (I don't know how regions work in Eucalyptus).

          Eucalyptus seems dead. I filed https://github.com/jenkinsci/ec2-plugin/pull/1027 to flense it. That should eliminate this TODO.

          Basil Crow added a comment - I also left a TODO in the code (I don't know how regions work in Eucalyptus). Eucalyptus seems dead. I filed https://github.com/jenkinsci/ec2-plugin/pull/1027 to flense it. That should eliminate this TODO.

          Basil Crow added a comment -

          dblanchette I cleaned up the region code and deployed version 1796.vdd7f420b_9298 to Incrementals. Can you please give it a try? Download link:

          Plugin Installation Manager input format: (documentation): ec2:incrementals;org.jenkins-ci.plugins;1796.vdd7f420b_9298

          Basil Crow added a comment - dblanchette  I cleaned up the region code and deployed version 1796.vdd7f420b_9298 to Incrementals. Can you please give it a try? Download link: ec2 Plugin Installation Manager input format: ( documentation ): ec2:incrementals;org.jenkins-ci.plugins;1796.vdd7f420b_9298

          basil Thank you! I installed the incremental in our instance and I confirm it works.

          Denis Blanchette added a comment - basil Thank you! I installed the incremental in our instance and I confirm it works.

          Basil Crow added a comment -

          Thanks dblanchette! Is anyone else interested in doing some more real-world testing of this plugin on AWS SDK for Java v2?

          Basil Crow added a comment - Thanks dblanchette ! Is anyone else interested in doing some more real-world testing of this plugin on AWS SDK for Java v2?

          Laszlo Gaal added a comment -

          Hi basil ;

          my apologies for not getting back to you earlier; life has kept me entertained in various other ways.

          However, starting this weekend I'll probably have some bandwidth to test the plugin. Is this one (https://issues.jenkins.io/browse/JENKINS-73640?focusedId=451499&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-451499 ) still the latest version I should grab?
          I'll scan the whole thread for dependency installtion instructions, but I may come back with newbie questions, as I've only used plugins from the official released plugins list so far.

          Last but not least: thank a lot for picking this up and keeping the the gate open for new EC2 functionality, it is highly appreciated!

          Laszlo Gaal added a comment - Hi basil ; my apologies for not getting back to you earlier; life has kept me entertained in various other ways. However, starting this weekend I'll probably have some bandwidth to test the plugin. Is this one ( https://issues.jenkins.io/browse/JENKINS-73640?focusedId=451499&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-451499 ) still the latest version I should grab? I'll scan the whole thread for dependency installtion instructions, but I may come back with newbie questions, as I've only used plugins from the official released plugins list so far. Last but not least: thank a lot for picking this up and keeping the the gate open for new EC2 functionality, it is highly appreciated!

          Thiago added a comment -

          do we have any updates?

          Thiago added a comment - do we have any updates?

          Basil Crow added a comment -

          Last but not least: thank a lot for picking this up and keeping the the gate open for new EC2 functionality, it is highly appreciated!

          laszlog Thank you for your kind words, but to clarify—this ticket is unassigned, and I haven't picked this up. I am simply enabling members of the Jenkins community to test this change by providing them with an incremental build.

          do we have any updates?

          ttomen See https://issues.jenkins.io/browse/JENKINS-73640?focusedId=451602&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-451602. Have you tested the incremental build from https://github.com/jenkinsci/ec2-plugin/pull/1021/checks?check_run_id=37486086102 against a real EC2 account?

          Basil Crow added a comment - Last but not least: thank a lot for picking this up and keeping the the gate open for new EC2 functionality, it is highly appreciated! laszlog Thank you for your kind words, but to clarify—this ticket is unassigned, and I haven't picked this up. I am simply enabling members of the Jenkins community to test this change by providing them with an incremental build. do we have any updates? ttomen See https://issues.jenkins.io/browse/JENKINS-73640?focusedId=451602&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-451602 . Have you tested the incremental build from https://github.com/jenkinsci/ec2-plugin/pull/1021/checks?check_run_id=37486086102 against a real EC2 account?

            Unassigned Unassigned
            markewaite Mark Waite
            Votes:
            1 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated: