i commit through netbeans IDE and not sure what encoding it does use, i presume it uses utf-8, as it would be a wise choice and mercurial's rhodecode UI displays these messages correctly so i think the problem is on Jenkins side and you should definitely use UTF-8 by default.
also mercurial does define encoding it defaults to UTF-8:
UTF-8 strings are used to store most repository metadata. Unlike repository contents, repository metadata is 'owned and managed' by Mercurial and can be made to conform to its rules. In particular, this includes:
commit messages stored in the changelog
user names
tags
branches
The following files are stored in UTF-8:
.hgtags
.hg/branch
.hg/branchheads.cache
.hg/tags.cache
.hg/bookmarks
Unfortunately Mercurial does not define any encoding (for file contents, filenames, or metadata such as user names and comments), so it is impossible for Jenkins to know what encoding was “meant” by the people committing changes, or indeed if they all agree on what encoding to use. My only advice is to use UTF-8 everywhere. Not sure if this is fixable.