Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-51413

Implementation of producer-consumer basic communication

    • GSoC - Coding Phase 1

      Option 1:

      • In KafkaComputerLauncher invoke SlaveComputer#setChannel()
        • Current interface: InputStream in, OutputStream out, TaskListener taskListener, Channel.Listener listener
        • We need need to produce KafkaRemotingOutputStream(KafkaProducer) and KafkaRemotingInputStream(KafkaConsumer)
        • It is not efficient, but should be a good start

      Option 2:

      • Patches in Jenkins core may be needed (unhardcode Channel Builder)
      • CommandTransport for Remoting Kafka. Get Kafka Consumer and Producer connected into it
      • RemotingKafkaChannelBuilder
      • In KafkaComputerLauncher invoke SlaveComputer#setChannel()
        • Current interface: InputStream in, OutputStream out, TaskListener taskListener, Channel.Listener listener
        • We need need to produce KafkaRemotingOutputStream(KafkaProducer) and KafkaRemotingInputStream(KafkaConsumer)
        • It is not efficient, but should be a good start

      Core patch would be needed here:

      • Support passing ChannelBuilder
      public void setChannel(InputStream in, OutputStream out, OutputStream launchLog, Channel.Listener listener) throws IOException, InterruptedException {
              ChannelBuilder cb = new ChannelBuilder(nodeName,threadPoolForRemoting)
                  .withMode(Channel.Mode.NEGOTIATE)
                  .withHeaderStream(launchLog);
      
              for (ChannelConfigurator cc : ChannelConfigurator.all()) {
                  cc.onChannelBuilding(cb,this);
              }
      
              Channel channel = cb.build(in,out);
              setChannel(channel,launchLog,listener);
          }
      

      Option 3. Create KafkaComputer and maybe copy-paste some code?

      • Close to option 2, but without patching the core
      • KafkaCommandTransport as in Phase 2
      • KafkaChannelBuilder as in Option 2, which creates channel with KafkaCommandTransport
      • Creates a new Computer type (Kafka Agent), which is fine
        • "KafkaPermanentAgent" as a name?
        • It can inherit SlaveComputer
        • We need to replace setChannel() implementation by one using KafkaCommandTransport (see below)
          • It overrides old stream-based implementations and throw IOException there
      • Oleg: We need to check that TcpAgentListener won't dare to invoke the agent if somebody tries to connect. Oleg to check it. It is for the next phase, no need to do it now
      • Connection: KafkaComputerLauncher invokes the new setChannel() implementation
      public void setChannel(OutputStream launchLog, Channel.Listener listener) throws IOException, InterruptedException {
              ChannelBuilder cb = new KafkaChannelBuilder(nodeName, threadPoolForRemoting)
                  .withMode(Channel.Mode.NEGOTIATE)
                  .withHeaderStream(launchLog);
      
              for (ChannelConfigurator cc : ChannelConfigurator.all()) {
                  cc.onChannelBuilding(cb,this);
              }
      
              Channel channel = cb.build(in,out);
              setChannel(channel, launchLog, listener);
          }
      

          [JENKINS-51413] Implementation of producer-consumer basic communication

          Oleg Nenashev added a comment -

          Notes:

          • we agreed to start with option 3 for now
          • Benefit: you get some command diagnostics in Kafka

          Oleg Nenashev added a comment - Notes: we agreed to start with option 3 for now Benefit: you get some command diagnostics in Kafka

          Pham Vu Tuan added a comment -

          oleg_nenashev I think for option 3 we still need to include option 1, which passes KafkaInputStream (extends InputStream) and KafkaOutputStream (OutputStream) as parameters for setChannel() method. Am I correct?

          Pham Vu Tuan added a comment - oleg_nenashev I think for option 3 we still need to include option 1, which passes KafkaInputStream (extends InputStream) and KafkaOutputStream (OutputStream) as parameters for setChannel() method. Am I correct?

          Code changed in jenkins
          User: Pham Vu Tuan
          Path:
          .gitignore
          agent/pom.xml
          agent/run.sh
          agent/src/main/java/io/jenkins/plugins/remotingkafka/Agent.java
          agent/src/main/java/io/jenkins/plugins/remotingkafka/Options.java
          commands.sh
          kafka-client-lib/pom.xml
          kafka-client-lib/src/main/java/io/jenkins/plugins/remotingkafka/KafkaConstants.java
          kafka-client-lib/src/main/java/io/jenkins/plugins/remotingkafka/KafkaConsumerPool.java
          kafka-client-lib/src/main/java/io/jenkins/plugins/remotingkafka/KafkaProducerClient.java
          plugin/pom.xml
          plugin/src/main/java/io/jenkins/plugins/remotingkafka/GlobalKafkaConsumerConfiguration.java
          plugin/src/main/java/io/jenkins/plugins/remotingkafka/GlobalKafkaProducerConfiguration.java
          plugin/src/main/java/io/jenkins/plugins/remotingkafka/KafkaComputerLauncher.java
          plugin/src/main/java/io/jenkins/plugins/remotingkafka/commandtransport/KafkaByteArrayCommandTransport.java
          plugin/src/main/java/io/jenkins/plugins/remotingkafka/commandtransport/KafkaByteBufferCommandTransport.java
          plugin/src/main/java/io/jenkins/plugins/remotingkafka/commandtransport/KafkaChunkedCommandTransport.java
          plugin/src/main/java/io/jenkins/plugins/remotingkafka/commandtransport/KafkaClassicCommandTransport.java
          plugin/src/main/resources/index.jelly
          plugin/src/main/resources/io/jenkins/plugins/remotingkafka/GlobalKafkaConsumerConfiguration/config.jelly
          plugin/src/main/resources/io/jenkins/plugins/remotingkafka/GlobalKafkaProducerConfiguration/config.jelly
          plugin/src/main/resources/io/jenkins/plugins/remotingkafka/KafkaComputerLauncher/config.jelly
          plugin/src/main/resources/io/jenkins/plugins/remotingkafka/Messages.properties
          pom.xml
          http://jenkins-ci.org/commit/remoting-kafka-plugin/64e5ce02a2e8d8c01ca82a0cf89ddb59cc87d760
          Log:
          [JENKINS-51413, JENKINS-51414] Implement basic master-agent communication, master command invocation (#2)

          • Implement basic master-agent communication
          • Command transport implementation for Kafka
          • Implement kafka producer and consumer connection pool
          • Reorganize and fix build problem
          • Refactor packagings

          *NOTE:* This service been marked for deprecation: https://developer.github.com/changes/2018-04-25-github-services-deprecation/

          Functionality will be removed from GitHub.com on January 31st, 2019.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Pham Vu Tuan Path: .gitignore agent/pom.xml agent/run.sh agent/src/main/java/io/jenkins/plugins/remotingkafka/Agent.java agent/src/main/java/io/jenkins/plugins/remotingkafka/Options.java commands.sh kafka-client-lib/pom.xml kafka-client-lib/src/main/java/io/jenkins/plugins/remotingkafka/KafkaConstants.java kafka-client-lib/src/main/java/io/jenkins/plugins/remotingkafka/KafkaConsumerPool.java kafka-client-lib/src/main/java/io/jenkins/plugins/remotingkafka/KafkaProducerClient.java plugin/pom.xml plugin/src/main/java/io/jenkins/plugins/remotingkafka/GlobalKafkaConsumerConfiguration.java plugin/src/main/java/io/jenkins/plugins/remotingkafka/GlobalKafkaProducerConfiguration.java plugin/src/main/java/io/jenkins/plugins/remotingkafka/KafkaComputerLauncher.java plugin/src/main/java/io/jenkins/plugins/remotingkafka/commandtransport/KafkaByteArrayCommandTransport.java plugin/src/main/java/io/jenkins/plugins/remotingkafka/commandtransport/KafkaByteBufferCommandTransport.java plugin/src/main/java/io/jenkins/plugins/remotingkafka/commandtransport/KafkaChunkedCommandTransport.java plugin/src/main/java/io/jenkins/plugins/remotingkafka/commandtransport/KafkaClassicCommandTransport.java plugin/src/main/resources/index.jelly plugin/src/main/resources/io/jenkins/plugins/remotingkafka/GlobalKafkaConsumerConfiguration/config.jelly plugin/src/main/resources/io/jenkins/plugins/remotingkafka/GlobalKafkaProducerConfiguration/config.jelly plugin/src/main/resources/io/jenkins/plugins/remotingkafka/KafkaComputerLauncher/config.jelly plugin/src/main/resources/io/jenkins/plugins/remotingkafka/Messages.properties pom.xml http://jenkins-ci.org/commit/remoting-kafka-plugin/64e5ce02a2e8d8c01ca82a0cf89ddb59cc87d760 Log: [JENKINS-51413, JENKINS-51414] Implement basic master-agent communication, master command invocation (#2) Implement basic master-agent communication Command transport implementation for Kafka Implement kafka producer and consumer connection pool Reorganize and fix build problem Refactor packagings * NOTE: * This service been marked for deprecation: https://developer.github.com/changes/2018-04-25-github-services-deprecation/ Functionality will be removed from GitHub.com on January 31st, 2019.

          Pham Vu Tuan added a comment -

          Pham Vu Tuan added a comment - https://github.com/jenkinsci/remoting-kafka-plugin/pull/2

            pvtuan10 Pham Vu Tuan
            pvtuan10 Pham Vu Tuan
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: