Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-40654

Make plugin compatible with storage backends compatible with Amazon S3 (OpenStack Swift...)

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Open (View Workflow)
    • Priority: Minor
    • Resolution: Unresolved
    • Component/s: s3-plugin
    • Labels:
      None
    • Similar Issues:

      Description

      Storage systems that are compatible with Amazon S3 (OpenStack Swift...) have to change the Amazon S3 endpoints configuration.

      To do this with the AWS SDK, you have to provide a file classpath://com/amazonaws/partitions/override/endpoints.json using the JSON format defined in https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/resources/com/amazonaws/partitions/endpoints.json

        Attachments

          Issue Links

            Activity

            Show
            cleclerc Cyrille Le Clerc added a comment - See https://github.com/jenkinsci/s3-plugin/pull/100
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Cyrille Le Clerc
            Path:
            src/main/java/hudson/plugins/s3/ClientHelper.java
            src/main/java/hudson/plugins/s3/Entry.java
            src/main/java/hudson/plugins/s3/S3BucketPublisher.java
            http://jenkins-ci.org/commit/s3-plugin/ff7d34914fbd52a9e44ae7029194d0351c1671c8
            Log:
            JENKINS-40654 Load the list of Amazon S3 regions from c.a.r.RegionMetadataProvider, stop relying on a static lists and constants.

            • Adapt the http proxy logic of the plugin to discover the S3 endpoint hostname with `com.amazonaws.regions.Region#getServiceEndpoint("s3")` instead of the hardcoded constant `s3.amazonaws.com`
            • Replace
            • `com.amazonaws.regions.Regions` by `com.amazonaws.regions.RegionUtils#getRegionsForService("s3")`
            • `com.amazonaws.services.s3.model.Region` by `com.amazonaws.regions.Region`
            • For the default region used in some places of the plugin, introduce the system property `hudson.plugins.s3.DEFAULT_AMAZON_S3_REGION` to override the default `us-east-1`. Note that it would be better to no longer rely on a default AWS Region and to ask the user to specify the desired AWS Region.

            The list of AWS regions can be overridden specifying a file `classpath://com/amazonaws/partitions/override/endpoints.json` matching the format defined in https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/resources/com/amazonaws/partitions/endpoints.json .

            A solution to add this file in the classpath of Jenkins is to use the `java` command line parameter `-Xbootclasspath/a:/path/to/boot/classpath/folder/` and to locate `com/amazonaws/partitions/override/endpoints.json` in `/path/to/boot/classpath/folder/`.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Cyrille Le Clerc Path: src/main/java/hudson/plugins/s3/ClientHelper.java src/main/java/hudson/plugins/s3/Entry.java src/main/java/hudson/plugins/s3/S3BucketPublisher.java http://jenkins-ci.org/commit/s3-plugin/ff7d34914fbd52a9e44ae7029194d0351c1671c8 Log: JENKINS-40654 Load the list of Amazon S3 regions from c.a.r.RegionMetadataProvider , stop relying on a static lists and constants. Adapt the http proxy logic of the plugin to discover the S3 endpoint hostname with `com.amazonaws.regions.Region#getServiceEndpoint("s3")` instead of the hardcoded constant `s3.amazonaws.com` Replace `com.amazonaws.regions.Regions` by `com.amazonaws.regions.RegionUtils#getRegionsForService("s3")` `com.amazonaws.services.s3.model.Region` by `com.amazonaws.regions.Region` For the default region used in some places of the plugin, introduce the system property `hudson.plugins.s3.DEFAULT_AMAZON_S3_REGION` to override the default `us-east-1`. Note that it would be better to no longer rely on a default AWS Region and to ask the user to specify the desired AWS Region. The list of AWS regions can be overridden specifying a file `classpath://com/amazonaws/partitions/override/endpoints.json` matching the format defined in https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/resources/com/amazonaws/partitions/endpoints.json . A solution to add this file in the classpath of Jenkins is to use the `java` command line parameter `-Xbootclasspath/a:/path/to/boot/classpath/folder/` and to locate `com/amazonaws/partitions/override/endpoints.json` in `/path/to/boot/classpath/folder/`.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Cyrille Le Clerc
            Path:
            README.md
            src/main/java/hudson/plugins/s3/ClientHelper.java
            src/main/java/hudson/plugins/s3/Entry.java
            http://jenkins-ci.org/commit/s3-plugin/2d1b795c62f8bc37ac02c69e4fb24bfc5b1092e6
            Log:
            JENKINS-40654 @Jimilian’s recommendations

            • Document `classpath://com/amazonaws/partitions/override/endpoints.json` and `hudson.plugins.s3.DEFAULT_AMAZON_S3_REGION` in README.md
            • Use `@nonnull` and `@nullable` in helper methods (not in the whole ClientHelper class to not change too many things at once)
            • Cleanup imports
            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Cyrille Le Clerc Path: README.md src/main/java/hudson/plugins/s3/ClientHelper.java src/main/java/hudson/plugins/s3/Entry.java http://jenkins-ci.org/commit/s3-plugin/2d1b795c62f8bc37ac02c69e4fb24bfc5b1092e6 Log: JENKINS-40654 @Jimilian’s recommendations Document `classpath://com/amazonaws/partitions/override/endpoints.json` and `hudson.plugins.s3.DEFAULT_AMAZON_S3_REGION` in README.md Use `@nonnull` and `@nullable` in helper methods (not in the whole ClientHelper class to not change too many things at once) Cleanup imports
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Akbashev Alexander
            Path:
            README.md
            src/main/java/hudson/plugins/s3/ClientHelper.java
            src/main/java/hudson/plugins/s3/Entry.java
            src/main/java/hudson/plugins/s3/S3BucketPublisher.java
            http://jenkins-ci.org/commit/s3-plugin/945f3315a246888482561d12406fa39b84f8a3f2
            Log:
            Merge pull request #100 from cyrille-leclerc/master

            JENKINS-40654 Make plugin compatible with storage backends compatible with Amazon S3 (OpenStack Swift...)

            Compare: https://github.com/jenkinsci/s3-plugin/compare/92247d2d099b...945f3315a246

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Akbashev Alexander Path: README.md src/main/java/hudson/plugins/s3/ClientHelper.java src/main/java/hudson/plugins/s3/Entry.java src/main/java/hudson/plugins/s3/S3BucketPublisher.java http://jenkins-ci.org/commit/s3-plugin/945f3315a246888482561d12406fa39b84f8a3f2 Log: Merge pull request #100 from cyrille-leclerc/master JENKINS-40654 Make plugin compatible with storage backends compatible with Amazon S3 (OpenStack Swift...) Compare: https://github.com/jenkinsci/s3-plugin/compare/92247d2d099b...945f3315a246
            Hide
            jamshid Jamshid Afshar added a comment -

            It would be much easier if the S3 endpoint could be specified in the configuration page (just use setEndpoint()), rather than the current endpoints.json classpath solution, which requires changes to jenkins startup.

            Show
            jamshid Jamshid Afshar added a comment - It would be much easier if the S3 endpoint could be specified in the configuration page (just use setEndpoint()), rather than the current endpoints.json classpath solution, which requires changes to jenkins startup.
            Hide
            simonrichter Simon Richter added a comment -

            We use both Amazon S3 and CERN's OpenStack installation from the same Jenkins instance, so we cannot just override the endpoints globally.

            It would be nice if there was a per-transfer configuration setting, selecting one of several configured providers (which all have their own sets of credentials).

            Show
            simonrichter Simon Richter added a comment - We use both Amazon S3 and CERN's OpenStack installation from the same Jenkins instance, so we cannot just override the endpoints globally. It would be nice if there was a per-transfer configuration setting, selecting one of several configured providers (which all have their own sets of credentials).
            Hide
            cleclerc Cyrille Le Clerc added a comment - - edited

            Simon Richter can't you overwrite the endpoints.json file to add your S3 endpoint? You would have to introduce and "s3.endpoint" I think; you may also have to define an "s3-control.endpoint".

            https://github.com/aws/aws-sdk-java/blob/1.11.587/aws-java-sdk-core/src/main/resources/com/amazonaws/partitions/endpoints.json#L2367

            Show
            cleclerc Cyrille Le Clerc added a comment - - edited Simon Richter can't you overwrite the endpoints.json file to add your S3 endpoint? You would have to introduce and "s3.endpoint" I think; you may also have to define an "s3-control.endpoint". https://github.com/aws/aws-sdk-java/blob/1.11.587/aws-java-sdk-core/src/main/resources/com/amazonaws/partitions/endpoints.json#L2367

              People

              Assignee:
              jimilian Alexander A
              Reporter:
              cleclerc Cyrille Le Clerc
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated: