I can't duplicate this failure with any version of command line git on any of the platforms I test. I regularly use non-default ssh ports in an ssh URL to a git repository and it works as expected.
I've added a test for this to my private regression test suite.
Can you explore further to identify what's different between your failing installation and my installation that works as expected?
There is a reference to "ssh variant 'simple'" on the git mailing list. If that pull request has been merged, then it indicates you've somehow selected a variant of SSH which does not support port specification. SInce you need port specification, you'll need to reinstall or reconfigure command line git to have the complete SSH implementation, rather than the simple ssh implementation.
As far as I can tell, this is not a git plugin or a git client plugin bug. It is a configuration error in command line git, or a bug in command line git.