The Jenkinsfile is cloned to the controller initially so that it can be executed on the controller. If "not a git repository" is being reported from the agent, then there is no git repository on the agent and the Jenkinsfile is not yet available in the agent workspace.
Closing as not reproducible because I assume it is a local issue with your git server or a local issue with the file system on the agent or a local issue with the agent. I've seen intermittent cases of a git server that refused ssh connections because there were too many requests at one time. I'm sure there are other load conditions which may cause the same failure.
I checked the many, many build logs on ci.jenkins.io and on my test server and found no cases of that message in any log file. The message does occur in the source code of GitClientTest.java in the git client plugin, but it is only in the test code and test code is not delivered with the plugin.