Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-60407

Launched instances cannot reach public internet (regression)

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • ec2-plugin
    • None
    • Jenkins ver. 2.190.3

      With plugin 1.45 all is working.

      Upgraded to 1.46.1 and although the agents report themselves as available, they have no network access to the outside world. Reverting to 1.45 got them back again.

      The agents are each configured with the same single private subnet to launch into. Investigations showed things like `ping -c 3 google.com` times out.

      Checking out from bitbucket.org was the original fault reported. We have apparently made no changes to the Jenkins master, except to upgrade it to the current LTS from a recent release at the same time as updating the plugins this lunchtime.

          [JENKINS-60407] Launched instances cannot reach public internet (regression)

          Justin Pihony added a comment -

          It would be helpful to document Dominik's solution - it was hard to find otherwise. Adding to the docs would have saved me hours.

          Justin Pihony added a comment - It would be helpful to document Dominik's solution - it was hard to find otherwise. Adding to the docs would have saved me hours.

          I don't see why it should not be able to reach public internet do you have the appropriate routes set up?

          Raihaan Shouhell added a comment - I don't see why it should not be able to reach public internet do you have the appropriate routes set up?

          James Green added a comment -

          raihaan simply upgrading a plugin should not break a behaviour. Downgrading of it proves the culprit. If the upgrade brings changes they should be documented.

          James Green added a comment - raihaan simply upgrading a plugin should not break a behaviour. Downgrading of it proves the culprit. If the upgrade brings changes they should be documented.

          jmkgreen I'd like to understand why it broke. Sure it broke your workflow but not being able to reach public internet and having a public IP are 2 distinct things. I would like to understand why you lose public internet access without a public ip. I can't understand why this change caused this issue.

          Raihaan Shouhell added a comment - jmkgreen I'd like to understand why it broke. Sure it broke your workflow but not being able to reach public internet and having a public IP are 2 distinct things. I would like to understand why you lose public internet access without a public ip. I can't understand why this change caused this issue.

          raihaan I had the exact same issue - lucky me, I was able to compare two EC2 instances (one launched with the old  EC2-plugin version and one with the new one). The only difference I found was the assignement of a public IP. As soon as I assigned a public IP to the instance launched by the new version, it all worked again.

          Dominik Bartholdi added a comment - raihaan I had the exact same issue - lucky me, I was able to compare two EC2 instances (one launched with the old  EC2-plugin version and one with the new one). The only difference I found was the assignement of a public IP. As soon as I assigned a public IP to the instance launched by the new version, it all worked again.

          imod Could I have some details on your setup?

          Do you launch in a vpc? If so is your subnet public or private? What is in that subnets route table?

          Raihaan Shouhell added a comment - imod Could I have some details on your setup? Do you launch in a vpc? If so is your subnet public or private? What is in that subnets route table?

          raihaan sure, I do my best to get you the required details: All I do is done with cloudformation, so it should be reproducible.

          Everytime we install a new version, i create it from scratch: I remove everything and build it up from ground with cloudformation only (no manual steps and no cloudformation updates).

          You can find a stripped down version of the cloudformation templates here: https://gist.github.com/imod/fb702d545dbe77292e8f4796c7804059 

          The templates should contain all the details you need. The 'vpc-cloudformation-template,json' creates the full VPC with route tables, subnet and gateway and can be executed as is, but I had to remove quite a bit from the 'jenkins-cloudformation-template,json' - this one would install Jenkins on a EC2 instance and configure the security groups. 

          I hope this is useful, if you don't get a long with cloudformation, please let me know.

           

          Dominik Bartholdi added a comment - raihaan sure, I do my best to get you the required details: All I do is done with cloudformation, so it should be reproducible. Everytime we install a new version, i create it from scratch: I remove everything and build it up from ground with cloudformation only (no manual steps and no cloudformation updates). You can find a stripped down version of the cloudformation templates here:  https://gist.github.com/imod/fb702d545dbe77292e8f4796c7804059   The templates should contain all the details you need. The 'vpc-cloudformation-template,json' creates the full VPC with route tables, subnet and gateway and can be executed as is, but I had to remove quite a bit from the 'jenkins-cloudformation-template,json' - this one would install Jenkins on a EC2 instance and configure the security groups.  I hope this is useful, if you don't get a long with cloudformation, please let me know.  

          James Green added a comment -

          FWIW our working Jenkins installation (with ec2-plugin:1.45) has instances configured to launch within the same VPC as Jenkins itself, and the "Associate Public IP address" checkbox is not checked. Yet ec2 instances do have public IPs - we just never noticed.

          I am guessing that updated plugin versions now require this checkbox to be checked.

          James Green added a comment - FWIW our working Jenkins installation (with ec2-plugin:1.45) has instances configured to launch within the same VPC as Jenkins itself, and the "Associate Public IP address" checkbox is not checked. Yet ec2 instances do have public IPs - we just never noticed. I am guessing that updated plugin versions now require this checkbox to be checked.

          jmkgreen that sounds like the exact same case then

          Dominik Bartholdi added a comment - jmkgreen that sounds like the exact same case then

          James Green added a comment -

          I have checked the option "Associate Public IP" for each agent and relaunched with ec2-plugin:1.49 - brand new agents are working.

          James Green added a comment - I have checked the option "Associate Public IP" for each agent and relaunched with ec2-plugin:1.49 - brand new agents are working.

            thoulen FABRIZIO MANFREDI
            jmkgreen James Green
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: