Once a Pipeline has a polling baseline added to it for a given SCM (shared library, main repo, etc), that baseline cannot be removed directly, and the job will always be built when the SCM is updated.

The only way to remove the baseline is to disable polling for the SCM, build the job, and then restart Jenkins after the build completes. Once that is done, the polling baseline will no longer be present for that SCM and the job will not be built when the SCM is updated.

It should be possible to remove the polling baseline without needing to restart Jenkins.

There are probably some related issues that should be closed as a duplicate of this issue.