I would like to configure a job so that some users are only allowed to download artifacts from it but not see anything else, especially not the changes recorded by the SCM plugins (git or svn) and test results. In short, I want the opposite of what not granting the Artifacts permission does, if that is enabled.

When not giving a user the Read permission for the job, they don't see the job at all, despite having the Discover (and Artifact) permission. The Discover permission doesn't seem to do anything on its own for authenticated users.