I have the correct id_rsa key under my Jenkins user's .ssh folder, and when I log onto the box and run a git clone command, it works fine. However, whenever that slave does a build and tries the exact same git clone command from the same repo, I just get "Permission denied (public key)." 

There's no passphrase on the key, it has the correct permissions, etc. Not sure what's going on...