I have finished a preliminary analysis of the report, in general, is pretty good but there are some improvement areas (not exhaustive list here)

• Serialization: GitSCM and BuildData provide no tested readResolve methods also unmarshall testing for RemoteConfigConverter and ObjectIdConverter
• Stapler invoked methods: There is some logic in Stapler invoked methods like GitSCM.DescriptorImpl#doGitRemoteCheck, UserRemoteConfig.DescriptorImpl#doCheckdoCheckCredentialsId or diferent repository browser implementations doCheckUrl methods that are no tested (Need to check if is tested on ATH)
• Extension testing: Some extensions like WipeWorkspace and CleanCheckout are lacking coverage on the action method
• Submodules: SubmoduleCombinator#createSubmoduleCombinations and SubmoduleOption
• Some individual lacking areas on other clases (will attach a detailed list)

rarabaolaza is there a correlation between these gaps and the open bugs reported in jira?

Full list of classes lacking coverage attached, please note that I have tried to check only for lacking coverage on significant areas, for example I am completely ignoring if a class is lacking coverage on a getter a toString or a deprecated method

vilacides Checking now for possible correlations

vilacides Some correlations I have found for what I consider the most important cases, please note that not all tickets are open:

• JENKINS-45747 Seems related to lack of readResolve testing
• JENKINS-12267 May be related to lack of testing in ObjectIdConverter#legacyUnmarshall
• JENKINS-44087, JENKINS-19851 May be related to GitSCM#createClient
• There are a bunch of tickets related to submodules (this link shows only open ones) and JENKINS-31586 Directly correlates to lack of covergae in SubmoduleOption#onCheckoutCompleted
• JENKINS-32521 Seems related to lack of ceoverage on InverseBuildChooser
• There is a bunch of tickets related to UserRemoteConfig#doCheckUrl

Please note that not all tickets linked here are still open, but the combination of Unit/Integration testing and PCT on those areas will IMO reduce the risks of regressions or new failures

A very important point here: Even with enough coverage we are lacking confidence with the current set of tests because as markewaite and jglick said there are a lot of possible corner cases related to different git, authentication etc providers.

Just to name a few that the awesome kshultz has mentioned me:

• Gitea
• Gitlab
• Bitbucket Cloud
• Bitbucket Server
• Github
• Github Enterprise

I have created JENKINS-50679 to address this

rarabaolaza I agree with your observation that there are many other cases to test outside the gaps detected by jacoco statement coverage reports. The jacoco statement coverage reports can help you determine areas that are untouched by test automation. They may give some rough indication of some use cases that are not touched by test automation. They don't help you prioritize use cases or identify component interactions.

As an example, I recommend we spend no time on the Submodule combinator. I've never detected anyone using it. It was an interesting experiment created many years ago. I've not seen any bug reports on it, and even if there were a bug report, I would not spend time on it.

I have reviewed the ATH coverage:

• It creates jobs from the UI with credentials and remote locations, (docker container) so I believe it covers GitSCM.DescriptorImpl
• It does not test repository browsers
• It does some testing of submodules
• It tests the InverseBuildStrategy
• It does not perform all these tests with different environments

So, as an initial step, this is my recommendation:

• Create integration tests for migrations from old versions of git plugin to new ones to check that the deserialization properly handles things (readResolve methods) and also for marshaling/unmarshalling
• Select a small subset of ATH tests (the ones that cover P1 cases) and run them for different environments (see above comment about environments) and different git client implementations (git, and JGit for example)
  • Investigate if this tests can be implemented as integration instead of Acceptance ones
  • Ideally also on different platforms like Windows

WDYT markewaite kshultz ?

The integration test of migration from one version to the next sounds great to me.

Can you explain in more detail what you envision testing with the acceptance test harness? I'm a jaded skeptic of Selenium tests for the git plugin because I believe they are testing at the wrong level for the git plugin. I've not found many cases in the git plugin which require Selenium for a valid test.

markewaite What I want is to run some of the test scenarios that are currently on the ATH using different environments like a gitlab server, but I don' t not mean that it has to be done in a selenium test, that is the reason I say that it has to be investigated if that scenarios (a simple ssh checkout of a branch using ssh credentials or a personal access token from a repo for example) can be implemented using integration tests.