This may or may not be a "feature" but the Window installer's "Remove" functionality doesn't remove C:\Program Files\Jenkins, it apparently just stops and removes the service.

When I say Remove, I mean Remove!