-
Bug
-
Resolution: Fixed
-
Major
-
Powered by SuggestiMate
Jenkins email-ext plugin fails to send email, the error is
Failed to create e-mail address for _developer1@xxx.net_
Failed to create e-mail address for _developer2@xxx.net_
Failed to create e-mail address for _developer1@xxx.net_
Failed to create e-mail address for _developer1@xxx.net_
(Names and domain are changed)
Jan 13, 2013 7:15:56 PM hudson.plugins.emailext.ExtendedEmailPublisher addAddressesFromRecipientList
WARNING: Could not create email address.
javax.mail.internet.AddressException: Domain contains illegal character in string ``_developer1@xxx.net_''
at javax.mail.internet.InternetAddress.checkAddress(InternetAddress.java:1269)
at javax.mail.internet.InternetAddress.parse(InternetAddress.java:1091)
at javax.mail.internet.InternetAddress.parse(InternetAddress.java:633)
at javax.mail.internet.InternetAddress.<init>(InternetAddress.java:111)
at hudson.plugins.emailext.EmailRecipientUtils.convertRecipientString(EmailRecipientUtils.java:60)
at hudson.plugins.emailext.ExtendedEmailPublisher.addAddressesFromRecipientList(ExtendedEmailPublisher.java:687)
at hudson.plugins.emailext.ExtendedEmailPublisher.createMail(ExtendedEmailPublisher.java:508)
at hudson.plugins.emailext.ExtendedEmailPublisher.sendMail(ExtendedEmailPublisher.java:297)
at hudson.plugins.emailext.ExtendedEmailPublisher._perform(ExtendedEmailPublisher.java:289)
at hudson.plugins.emailext.ExtendedEmailPublisher.perform(ExtendedEmailPublisher.java:249)
at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:36)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:810)
at hudson.model.AbstractBuild$AbstractBuildExecution.perJan 13, 2013 7:15:56 PM hudson.plugins.emailext.ExtendedEmailPublisher addAddressesFromRecipientList
WARNING: Could not create email address.
javax.mail.internet.AddressException: Domain contains illegal character in string ``_developer1@xxx.net_''
at javax.mail.internet.InternetAddress.checkAddress(InternetAddress.java:1269)
at javax.mail.internet.InternetAddress.parse(InternetAddress.java:1091)
at javax.mail.internet.InternetAddress.parse(InternetAddress.java:633)
at javax.mail.internet.InternetAddress.<init>(InternetAddress.java:111)
at hudson.plugins.emailext.EmailRecipientUtils.convertRecipientString(EmailRecipientUtils.java:60)
at hudson.plugins.emailext.ExtendedEmailPublisher.addAddressesFromRecipientList(ExtendedEmailPublisher.java:687)
at hudson.plugins.emailext.ExtendedEmailPublisher.createMail(ExtendedEmailPublisher.java:508)
at hudson.plugins.emailext.ExtendedEmailPublisher.sendMail(ExtendedEmailPublisher.java:297)
at hudson.plugins.emailext.ExtendedEmailPublisher._perform(ExtendedEmailPublisher.java:289)
at hudson.plugins.emailext.ExtendedEmailPublisher.perform(ExtendedEmailPublisher.java:249)
at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:36)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:810)
at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:785)
at hudson.model.Build$BuildExecution.cleanUp(Build.java:192)
at hudson.model.Run.execute(Run.java:1587)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:236)formAllBuildSteps(AbstractBuild.java:785)
at hudson.model.Build$BuildExecution.cleanUp(Build.java:192)
at hudson.model.Run.execute(Run.java:1587)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:236)
First I thought is the problem is in email-ext plugin, but when I went to "Changes" tab and opened the page
http://host/jenkins/user/Developer%201%20__developer1@xxx.net_/
(hostname changed)
it showed
Jenkins User Id: Developer 1 _developer1@xxx.net_
For unknown reason Jenkins adds a leading and trailing underscore to email address
Environment
Jenkins ver. 1.498
Jenkins Mercurial plugin 1.42
Jenkins Email Extension Plugin 2.25
Mercurial Distributed SCM (version 1.4.3)
Ubuntu 10.04.4 LTS
Java(TM) SE Runtime Environment (build 1.6.0_32-b05)
Java HotSpot(TM) 64-Bit Server VM (build 20.7-b02, mixed mode)
We have a build cluster with a master and two slaves, this problem happened on the slave machine.
When I type hg log on Jenkins job's workspace, then all changesets have
correct email address without underscores, so it is not Mercurial configuration problem.
- is duplicated by
-
JENKINS-15253 javax.mail.internet.AddressException: Local address contains control or whitespace on mercurial commits
-
- Resolved
-
[JENKINS-16332] Leading and trailing underscores are added to commiters email address, unable to send build notification
I should also indicate that my manual access of Jenkins is anonymous. While I've created accounts for all users in SCM, I don't actually login to Jenkins when making changes, manually triggering builds, etc.
We are also using Mercurial and have the same problem. We log in to Jenkins to run jobs.
IT has asked us to turn off email notifications because the bad email addresses are choking our servers.
We are using Mercurial as well, having normal looking commits on that end. Jenkins 1.506 adds underscores to some but not all email addresses.
In one build two out of three email addresses failed. They all look the same in hg, the one big different is that the two failing ones belong to users with Jenkins accounts, the one that works belongs to a user that never created a Jenkins account. The user profiles in the Jenkins installation look fine, the addresses in there seem to be ok. The person that gets the build notification does not have a profile at all.
One thing we noticed is that there are two additional user profiles in the file system that follow the pattern FirstName LastName _email@address_ as folder names. These are about a month old now and belong to other users, both of which have a normal profile as well. I don't know if that is related somehow.
Upgrading to 1.509.1 seems to have resolved this issue for us. The one thing we did notice while looking at this is that it was not actually just adding underscores to the email addresses, it was using the whole Mercurial username with the lesser than/greater than symbols replaced. For example Peter Becker <pbecker@example.com> should have turned into pbecker@example.com but resulted in Peter Becker _pbecker@example.com_ being used as email address.
I believe there may be an issue in MercurialChangeLogParser method MercurialChangeSetList parse(AbstractBuild build, File changelogFile). If I pass in a changelog that looks like the following:
<?xml version='1.0' encoding='UTF-8'?> <changesets> <changeset author='joe.schmo <joe.schmo@example.com>'/> <!-- note that those escape characters are on purpose and not some relic of posting in Jira --> </changesets>
I get a changeset author with the ID and fullname "joe.schmo _joe.schmo@example.com_", which is the error reported in this ticket. I believe the Digester class from Apache Commons is taking the escaped XML characters in the changesets/changeset/@author attribute and rendering them as underscores.
I pushed a unit test showing this to my fork of mercurial-plugin, but not requesting a pull because I don't have a fix yet.
@peterbecker wrote:
In one build two out of three email addresses failed. They all look the same in hg, the one big different is that the two failing ones belong to users with Jenkins accounts, the one that works belongs to a user that never created a Jenkins account.
which I suspect is important, though I still do not know how to reproduce this.
Jesse,
If you provide me with a patched plugin with extra logging I will help you figure out what is going on.
Code changed in jenkins
User: Jesse Glick
Path:
src/test/java/hudson/plugins/mercurial/MercurialChangeLogParserTest.java
http://jenkins-ci.org/commit/mercurial-plugin/319048bbd4de54f8badb19ff57d6475df6b193b1
Log:
JENKINS-16332 Demonstrating that MercurialChangeLogParser is not at fault.
https://github.com/jerrymaloney/mercurial-plugin/commit/037b5b1ac5a3eac18dceb58fa509973d8de40cad
was misleading because it checked getId, whereas getFullName is what is supposed to keep the email address.
On https://github.com/jenkinsci/mercurial-plugin/commit/319048bbd4de54f8badb19ff57d6475df6b193b1 Jesse said:
> getFullName is what is supposed to keep the email address.
It's been a while since I looked at this, but if I recall there are instances where the results of getId are returned to the emailer. From what you say above, this should never happen, so if it is happening that could be the problem. If I get time today I'll try to recreate the situation that leads me to believe this.
I just discovered something important. The auto-created user (from commit logs) has an email address with underscores (same as the user's full name). This is probably why hudson.tasks.MailAddressResolver.resolve(u) returns the same value as the full name.
Who is creating these users from commit logs? Which part of the code is populating the email address for those users? That's where we should be looking!
Who is creating these users from commit logs?
MercurialChangeSet.getAuthor should call User.get("Name <a@ai>") which should result in a User with an id of Name _a@ai_ and a fullName of Name <a@ai>. If it is calling User.get with a value that already contains underscores, that may be a problem in the Mercurial plugin (or the Digester library it uses, as @jerrymaloney claimed). If some other code (which?) is calling User.get with an ID rather than a full name, then that code would presumably be to blame. Or perhaps the User is created correctly but for some reason the mailer or the MailAddressResolver is ignoring the full name and picking up the ID instead.
Jesse,
Okay, so to reiterate (I think you already understand, but just in case): fullName being populated with an incorrect value (it contains underscores) and id is correct (containing underscores). I am trying to create a minimal testcase for you now.
This bug is a pain in the ass. I couldn't reproduce it in a minimal testcase.
1. The next thing I did was scan my build logs. Looking at jobs/name/builds/date/build.xml section <build><culprits><string> is the value supposed to be an email address or user id.
2. If I blow away all past builds, the next time Jenkins builds will it add People based on just the latest commit or will it scan the repository's entire history?
is the value supposed to be an email address or user id
ID, so should use underscores.
will it add People based on just the latest commit or will it scan the repository's entire history?
Neither; the People link shows authors of changesets mentioned in the changelogs of all build records kept in Jenkins.
Neither; the People link shows authors of changesets mentioned in the changelogs of all build records kept in Jenkins.
So I should be able to blow away the build records (e.g. jenkins/jobs/boost/2013-09-02_22-47-27) and then sit back waiting for an invalid user to get created, right? Perhaps we're chasing our tails here, looking for a bug that was fixed in the past but still shows up due to old build logs... By blowing away these records, I'll be able to confirm that the issue exists in the latest version. What do you think?
To anyone who has encountered this bug, be sure to check your $JENKINS_HOME/users/*/config.xml for old user records that might already contain the wrong fullName (or even hudson.tasks.Mailer.UserProperty.emailAddress). If you do, there is nothing wrong with the current Jenkins (or plugin) code: it is just reading bad data. You might have saved the user record in some old version of Jenkins (or with an old buggy plugin TBD) which accidentally used an ID as a full name.
There are other ways the problem might have happened in principle. Say that in one Jenkins session a User record is created with the correct id and fullName, from a changelog for example; but the user configuration file is not saved. Then a hyperlink is generated, or some kind of serialized record refers to this user (a build cause, etc.), which would use only the id. Then in another Jenkins session this link is followed to /jenkins/user/Some%20Body%20_somebody@nowhere.net_/, which calls User.get on the id; and then some change is made to the record (a new API token, etc.). Now a bogus config.xml will be saved.
In either case, correcting the problem for a few users is trivial: go to each user’s configuration screen and fix their email address. Whether Jenkins can defend against this problem in general is another matter; I think it would have to eagerly save config.xml whenever get is called with a fullName which gets translated to a different id.
Jesse,
- My jenkins/users directory does not contain any bogus users, only the ones who manually created their accounts. Does this tell you anything?
- Adding asserts that fail-fast when a bogus config.xml is being created will definitely help us track this to its source.
Code changed in jenkins
User: Jesse Glick
Path:
core/src/main/java/hudson/model/User.java
http://jenkins-ci.org/commit/jenkins/0ee5f00858f47a9f6ae08d48accbc78c6bf16838
Log:
JENKINS-16332 Trying to determine if CanonicalIdResolver is implicated.
Code changed in jenkins
User: Jesse Glick
Path:
core/src/main/java/hudson/model/User.java
http://jenkins-ci.org/commit/jenkins/e07aac61efc6a77fe58dbd99dca28a51504897b8
Log:
JENKINS-16332 More logging at FINE in hudson.model.User.
Running hypothesis: something historically saved a user configuration file with a bogus fullName;
thereafter anyone calling User.get(correctFullName) will get a User with the old fullName.
This would make the bug unreproducible in fresh Jenkins installations.
Compare: https://github.com/jenkinsci/jenkins/compare/b70f4c07f298...e07aac61efc6
Integrated in jenkins_main_trunk #2877
JENKINS-16332 Trying to determine if CanonicalIdResolver is implicated. (Revision 0ee5f00858f47a9f6ae08d48accbc78c6bf16838)
JENKINS-16332 More logging at FINE in hudson.model.User. (Revision e07aac61efc6a77fe58dbd99dca28a51504897b8)
Result = SUCCESS
Jesse Glick : 0ee5f00858f47a9f6ae08d48accbc78c6bf16838
Files :
- core/src/main/java/hudson/model/User.java
Jesse Glick : e07aac61efc6a77fe58dbd99dca28a51504897b8
Files :
- core/src/main/java/hudson/model/User.java
Bad news. I wiped out my build log yesterday and today I found two new users created: one with a proper email address, and one with an invalid one. Meaning: the bug still exists as of 1.530 and I still don't know how to reproduce it.
Here is a regex for validating email addresses (it looks for "FirstName LastName <email@address.com>"):
(.*\s)([^_@][\w.]+@[\w.]+[a-zA-Z0-9]([>\s]|$))
Please commit a runtime in this right place for inbound addresses before they get saved. This should allow us to detect who is trying to save bad addresses.
Code changed in jenkins
User: Jesse Glick
Path:
core/src/main/java/hudson/model/User.java
http://jenkins-ci.org/commit/jenkins/7ca07b5d4b2280d29c1ab92fa83e1391574b1a69
Log:
JENKINS-16332 Further diagnostics. Note that you must enable a hudson.model.User logger at FINE or below to use.
Any reason for using a low logging level? Shouldn't we use a higher logging level in the hopes of getting users to provide more information? Also, I would use a more proactive message: "Suspicious fullName being stored: " + fullName + ". Please report to https://issues.jenkins-ci.org/browse/JENKINS-16332"
Unless it is entirely sure that this case is definitely a sign of a bug, it is better not to spam log files which potentially a large number of messages. People already watching this bug should enable this logger.
Integrated in jenkins_main_trunk #2919
JENKINS-16332 Further diagnostics. Note that you must enable a hudson.model.User logger at FINE or below to use. (Revision 7ca07b5d4b2280d29c1ab92fa83e1391574b1a69)
Result = SUCCESS
Jesse Glick : 7ca07b5d4b2280d29c1ab92fa83e1391574b1a69
Files :
- core/src/main/java/hudson/model/User.java
If it helps I am also seeing this using 1.537 on Windows with RTC as my SCM tool along with the email-ext plugin. Editing the people (these are not real users, they are picked up from SCM changes) to fix the email address seems to fix the issue for a short period of time, but the records are being updated again intermittently.
@jgoodspeed it helps if you can reproduce the problem and are willing to run the logger (FINE in hudson.model.User).
Experiencing similar problem. Unable to send out the emails to the build failed culprits. I went checked out the relevant code in Jenkins, Mercurial plugin and the Mailer plugin. Here is my understanding based on the read source code:
Mercurial states that the author format is FirstName LastName <email@address> ( see http://www.selenic.com/mercurial/hgrc.5.html )
Now the mercurial plugin sees the author part with that format and asks Jenkins to get a user or create one by indirectly invoking https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/model/User.java#L298
During the creation Jenkins will introduce some replacing ( https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/jenkins/model/DefaultUserCanonicalIdResolver.java#L44 ) and all < and > turn into underscores.
And now we've created a Jenkins user FirstName LastName _email@address_. Not sure why we want to do this but probably a good thing for later to pick up the culprits.
Now a build fails and Mailer plugin asks Jenkins to give him the culprits. Jenkins has recorded those users for the build and provides the list of FirstName Lastname _email@address_ users.
Now Mailer asks for the addresses of the users but as those are empty it will use a MailAddressResolver ( https://github.com/jenkinsci/mailer-plugin/blob/master/src/main/java/hudson/tasks/MailAddressResolver.java#L100 - there is a specific subclass for Subversion also but didn't sis Mercurial) and it will do some magic but probably fails because Mailer still tries to send to FirstName Lastname _email@address_.
I see couple of options here
- Mailer plugin(s) to detect the emails better when none is defined
- Mercurial plugin will do the email address extraction itself, as it knows the format and will then set explicitly the email addresses for the users it creates
- Mercurial plugin will create the user with the email address instead of the FirstName LastName <email@address> format. In this case any dark magic that does email address extraction from user id might succeed.
Any holes in my logic? Anyone wants to take crack at it? I'm off to vacation without a laptop
Jesse,
I just reproduced the problem. Unfortunately I didn't have the logger enabled so we missed an opportunity.
I did notice something very interesting though:
1. A build failed and tried to send an email to an auto-generated user with underscores
2. I went into Jenkins -> People and sure enough the user's email address had underscores
3. I rebooted Jenkins and went into Manage Jenkins -> Manage Users and all of a sudden that same user no longer had underscores in the email address.
It's not clear whether the reboot or Manage Users fixed the email address, but that's another clue of what's going on. I've enabled the logger, so the next time this happens I will catch it.
If you figure out how to reproduce it from scratch then I can fix it. I am pretty sure the problem somehow lies with a userId getting passed from one place to another without a User context, and then Jenkins serializing the user record at the wrong time, but I do not know the sequence of events that makes this happen to some people some times.
Jesse,
I triggered a build that failed, and tried emailing the author:
Invalid address: First Last _first.last@gmail.com_
I opened the logger and saw these two lines:
Jan 08, 2014 6:36:06 PM FINE hudson.model.User hudson.model.User$FullNameIdResolver@934e9b mapped First Last _first.last@gmail.com_ to First Last _first.last@gmail.com_ Jan 08, 2014 6:36:06 PM FINE hudson.model.User hudson.model.User$FullNameIdResolver@934e9b mapped First Last _first.last@gmail.com_ to First Last _first.last@gmail.com_
(I've replaced the person's actual name with "First Last" but the rest of the log is identical)
Does this help? Where do we go from here?
@cowwoc so something called
User.get("First Last _first.last@gmail.com_")
which is OK, but there was already a User which had that not only as an id but also as a fullName, which is wrong. The question is how and when that User got created, which this log snippet does not indicate. To learn that, we would need an (earlier) log message starting with
[JENKINS-16332] Suspicious fullName being stored: First Last _first.last@gmail.com_
which includes a stack trace possibly indicating the culprit.
I didn't see such a log entry but I will keep my eye open if this happens again.
The EMail-ext plugin just fixed an interesting bug: https://issues.jenkins-ci.org/browse/JENKINS-20215
Do you think this could have been at fault?
Okay, so to reiterate, I did not see
[JENKINS-16332] Suspicious fullName being stored: First Last _first.last@gmail.com_
in the log file even though the problem occurred. Doesn't that imply that we need to add more logs?
Doesn't that imply that we need to add more logs?
Not necessarily; you might have turned on logging after this user record ($JENKINS_HOME/users/*.xml) was created with the faulty information.
@Jesse,
I enabled this log a few weeks ago, so that's unlikely. That being said, the log did seem pretty short (2 pages long).
- Does Jenkins truncate the log on startup?
- Does it begin logging automatically or do I need to intervene manually every startup?
- Does it log to disk or just to memory?
The log displayed in /log/all is just the last hundred or so messages in memory (since Jenkins was started). Depending on how you launch Jenkins, typically a log file will be written somewhere to disk.
And custom loggers are normally not written to disk at all. I have an unreleased change in the Support Core plugin which writes custom logs to a location inside the Jenkins home dir (rotated after restart). Plan to release that update soonish.
Jesse,
I am now 100% I am seeing:
Jan 08, 2014 6:36:06 PM FINE hudson.model.User hudson.model.User$FullNameIdResolver@934e9b mapped First Last _first.last@gmail.com_ to First Last _first.last@gmail.com_
without:
[JENKINS-16332] Suspicious fullName being stored: First Last _first.last@gmail.com_
ever being logged. It looks like you'll need to add more aggressive logging for us to track this down.
If that never gets logged, and you are still unsure how to reproduce the problem, then I am out of ideas for the moment.
Could this have anything to do with this code? https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/model/AbstractBuild.java#L728 It's using String's instead of User's
Guys,
I've managed to produce a minimal testcase for this bug.
- Extract testcase.zip into $JENKINS_HOME/jobs
- Run the job
You should get this output:
Started by user Gili Tzabari [EnvInject] - Loading node environment variables. Building remotely on MacOS X 10.8 in workspace /Users/builds/.jenkins/workspace/Test [Test] $ /usr/local/bin/hg showconfig paths.default ERROR: Workspace reports paths.default as https://bbs.darktech.org/code/test/ which looks different than https://cowwoc@bitbucket.org/cowwoc/test so falling back to fresh clone rather than incremental update $ /usr/local/bin/hg clone --rev default --noupdate https://cowwoc@bitbucket.org/cowwoc/test /Users/builds/.jenkins/workspace/Test adding changesets adding manifests adding file changes added 19 changesets with 19 changes to 2 files [Test] $ /usr/local/bin/hg update --rev default 2 files updated, 0 files merged, 0 files removed, 0 files unresolved [Test] $ /usr/local/bin/hg log --rev . --template {node} [Test] $ /usr/local/bin/hg log --rev . --template {rev} [Test] $ /usr/local/bin/hg log --rev 6ddb7626f1d05f3e67a4c92fa255ba32a5406020 changeset: 18:6ddb7626f1d0 tag: tip user: Getcho Getchev <getcho.getchev@gmail.com> date: Fri Feb 21 23:40:41 2014 +0200 summary: added line 10 [Test] $ /usr/local/bin/hg log --template "<changeset node='{node}' author='{author|xmlescape}' rev='{rev}' date='{date}'><msg>{desc|xmlescape}</msg><added>{file_adds|stringify|xmlescape}</added><deleted>{file_dels|stringify|xmlescape}</deleted><files>{files|stringify|xmlescape}</files><parents>{parents}</parents></changeset>\n" --rev default:0 --follow --prune 6ddb7626f1d05f3e67a4c92fa255ba32a5406020 --encoding UTF-8 --encodingmode replace [Test] $ /bin/sh -xe /var/folders/ht/zq3sdm6s55j4cv6gvjtkc8vm0000gp/T/hudson4438504154854395225.sh + ./test.sh /var/folders/ht/zq3sdm6s55j4cv6gvjtkc8vm0000gp/T/hudson4438504154854395225.sh: line 2: ./test.sh: No such file or directory Build step 'Execute shell' marked build as failure Invalid address: Getcho Getchev _getcho.getchev@gmail.com_ Invalid address: Gili Tzabari _gili.tzabari@gmail.com_ An attempt to send an e-mail to empty list of recipients, ignored. Finished: FAILURE
I submitted a pull request for "fixing" this.
https://github.com/jenkinsci/mailer-plugin/pull/11
I'm also planning on making a pull request for the Mercurial plugin that would create user accounts a little differently.
Your test case is reproducible but it does not really tell us anything, because the malformed culprits are simply being copied from earlier builds (#2 & #5). What is needed is a test case which demonstrates fresh additions to the culprits list with the malformed names; specifically the call stack of User.<init> with an incorrect fullName. Note that the Mercurial changelog.xml in these builds show the correct text.
@Jesse, I agree it should be calling it with the <>'s, but I think because of this code in AbstractBuild.java, it will call it with the _'s
Still working on reproducing from scratch. I think I may see what the issue is. If I am right, it is a bug in core as I suspected on Dec 19.
OK, finally reproduced. The mentioned line 729 is innocent, and really AbstractBuild as a whole is doing nothing wrong; nor, as far as I can tell, are the Mailer or Mercurial plugins.
The problem (as I surmised on Sep 16) is in User itself. When you run a build that fails, and it has a changelog mentioning at least one user with full name & email address who does not yet have any permanent record in $JENKINS_HOME/users/$id/config.xml, the build stores a culprits field listing the user’s ID, which correctly includes underscores. So long as Jenkins remains running, the loaded User object is fine, and mailing would work. However if you restart Jenkins, and then run a new build that again fails, but with no changelog this time, getCulprits inherits the culprits list from the previous build. This time User.get(id) is called and since there is no persistent record for that user, there is no way of reconstructing the proper fullName—we only have the sanitized ID. Therefore the User is constructed with a bogus fullName and mailing fails.
Looking into a fix.
Jesse, I just like to poke at these until you come up with the correct solution
Code changed in jenkins
User: Jesse Glick
Path:
changelog.html
core/src/main/java/hudson/model/User.java
test/src/test/java/hudson/model/UserTest.java
http://jenkins-ci.org/commit/jenkins/fdda9ca65689423794a8378c1f28faca7dffddf0
Log:
[FIXED JENKINS-16332] Ensure that User records are saved when the fullName is not recoverable from the id.
Otherwise User.get(id).getFullName() after restarting Jenkins will produce the wrong result.
Integrated in jenkins_main_trunk #3192
[FIXED JENKINS-16332] Ensure that User records are saved when the fullName is not recoverable from the id. (Revision fdda9ca65689423794a8378c1f28faca7dffddf0)
Result = SUCCESS
Jesse Glick : fdda9ca65689423794a8378c1f28faca7dffddf0
Files :
- test/src/test/java/hudson/model/UserTest.java
- changelog.html
- core/src/main/java/hudson/model/User.java
Jesse,
Will this fix jobs where malformed culprits were being copied from earlier builds, or do we need to manually "flush" the job history somehow?
This fix will not handle all cases involving historical records. If a new build is run after upgrade whose changelog mentions the given committer’s full name, that name will be properly registered and you should not see the problem for that user again. The main problem I can foresee is a build which was failing at the time of upgrade, and continues to fail without new changes being committed (to that or other jobs developed by the same people), and is using an email trigger configured to repeatedly send mail for every unstable build.
Again there are no “malformed” culprits fields here. The culprits list is correct; the problem only arises when the culprits list contains the only record of a given user in the system. You ought to be able to work around the problem for a given user by visiting that user’s page as hyperlinked from a changelog (logged in as an administrator), and explicitly setting a full name (and/or email address).
I see. So just to make sure I understand correctly:
Until a user makes a new commit, any existing jobs will keep on using the incorrect email address. Once a user makes a new commit, his email address will get corrected system-wide across all jobs. Alternatively, deleting the contents of the "builds" directory will remove all changelogs and force re-evaluation of the email address.
Is that correct?
Until a user makes a new commit, any existing jobs will keep on using the incorrect email address. Once a user makes a new commit, his email address will get corrected system-wide across all jobs.
Right. You can simply look in $JENKINS_HOME/users/ to see whether the user has been defined or not.
deleting the contents of the "builds" directory will remove all changelogs and force re-evaluation of the email address.
No. If you deleted all historical builds, then either the next build has commits from the relevant user(s), in which case their email addresses will be recorded; or it does not, in which case the culprits list for that build will simply be empty and no email will be sent (unless you have an unconditional recipient configured in the job). In the former case, deleting the old build records does not improve your situation at all. In the latter case, deleting them could make the new build’s status be “unstable” due to test failures or whatever, rather than “failed” due to an attempt to send mail to bogus addresses, but the email addresses of users mentioned in those old builds would still be missing.
I've been experiencing this issue as well. I found that it occurs with my build-pipeline. Initially, a clone is made from an Hg repository. This then kicks off a build of the source. If successful, I archive the workspace and then I trigger another job (to run lint). This downstream job uses the saved successful workspace.
I've noticed that when I manually trigger builds within Jenkins, at that point, the email gets screwed up. This occurs with a fresh (1.514) Jenkins build with only the Mercurial and Clone Workspace plugins installed. This occurs whether I do it on the initial build job or the downstream lint job. I'd been doing this (triggering builds manually) to test job configuration. My current workaround is to make a small comment change in my source, commit and push. Both jobs have email user who broke the build as a post-build step.
I didn't experience this bug when experimenting with build configurations using a project on a subversion repository.
To fix the phantom invalid user which Jenkins then recognizes for future SCM changes, I have to delete history of all builds that involve that user (if the build was successful, all jobs builds that it triggered), then restart Jenkins. At this point, the invalid user disappears and I can proceed. The other option is to just clone the jobs, then delete the old ones and rename them back (still requires a restart after this). The downside of both approaches is that you lose a bit (or a lot) of build history.