Jenkins version - 2.346.2

We are running Jenkins inside a kubernetes cluster (v 1.23.12) and we use dynamic k8s pods as agents configured from cloud templates. 

When the jobs are triggered, the agents showing offline with error  "This agent is offline because Jenkins failed to launch the agent process on it. See log for more details".  The agent pod is not even getting created inside the cluster and Jenkins logs doesn't even show any error logs, nothing. It just stays "agentXXX is provisoned successfully, now we have 4 computers". But we don't even see the agent pod getting created.