I can't duplicate the problem based on your description. That likely means there are differences between my environment and your environment. It likely means that one or more of those differences are important to identify the root cause of the problem.
The description doesn't tell me the configuration you are using. Without that configuration, I can't tell if there are any configuration differences between your environment and mine. The Jenkins support core plugin can generate a detailed report of the configuration of your Jenkins installation, including plugins installed, JDK version that is running, operating system of the controller, operating system of the agents, etc.
You can upload a support bundle as one way of sharing your configuration. You can describe your configuration in detail (Java version, controller operating system, agent operating system, agent java version, etc.) if you'd rather not upload a support bundle.
A Jenkins instance can be run on almost any computer with Java 8 or Java 11 available. Download the "war" file from www.jenkins.io/download and type the command "java -jar jenkins.war". Define a Windows agent, connect the Windows agent, and report if there is a failure in that configuration.
Please provide enough details so that others can duplicate the failure. I'm running Jenkins 2.318 with an agent connected using "Launch agent by connecting it to the controller" from a Windows 10 agent.
Are you able to duplicate the failure if you run a test Jenkins instance and connect agents from those nodes to the test instance?