Friendica 2022.10 seems to be sending malformed email notifications

@Friendica Support
I am the system administrator of the Friendica node at talkabout.cf, and have just updated the node to version 2022.10, along with a system upgrade to Ubuntu 22.04 and an updated web control panel. I just received several email notifications that were garbled in my inbox. I was able to trace the problem to some header lines that have an extra space at the beginning of each line. I am including these redacted headers below. I also leave a couple of lines above and below intact to make these extra leading spaces visible.

Subject: [Friendica:Notify] Name Redacted tagged you (kyle@talkabout.cf)
Precedence: list
 X-Friendica-Host: talkabout.cf
 X-Friendica-Platform: Friendica
 X-Friendica-Version: 2022.10
 List-ID: &lt;notification.talkabout.cf>
 List-Archive: &lt;https://talkabout.cf/notifications/system>
 X-Friendica-Account: &lt;kyle@talkabout.cf>
 References: &lt;RedactedHexID@email.server

Subject: [Friendica:Notify] Name Redacted tagged you (kyle@talkabout.cf)
Precedence: list
 X-Friendica-Host: talkabout.cf
 X-Friendica-Platform: Friendica
 X-Friendica-Version: 2022.10
 List-ID: &lt;notification.talkabout.cf>
 List-Archive: &lt;https://talkabout.cf/notifications/system>
 X-Friendica-Account: &lt;kyle@talkabout.cf>
 References: &lt;RedactedHexID@email.server>
 In-Reply-To: &lt;RedactedHexID@email.server>
 From: Talkabout Federated Social Network | Powered by Friendica &lt;notify@talkabout.cf>
 Reply-To: Talkabout Federated Social Network | Powered by Friendica &lt;notify@talkabout.cf>
 MIME-Version: 1.0
 Content-Type: multipart/alternative; boundary="2-189687198-581818022=:18767"
Message-Id: &lt;RedactedMessageIDSuppliedByServer@email.server>
From: "redacted@email.address" &lt;notify@talkabout.cf>
Date: Sun, 06 Nov 2022 14:33:24 +0100

The message ID of course is added by the mail transfer agent before the message is delivered, but I left that header because it is immediately above the Subject: header, which does not contain a leading space. I did resend the message by copying and pasting the garbled source message into a text editor, removing the leading space from each line and copying and pasting it back into a direct connection to my local SMTP server, sending the message to myself, and I received the correctly decoded message. Did I find a bug in the latest 2022.10 Friendica code, or did something go wrong with the upgrades that I performed prior to the update that somehow didn't affect anything other than these mail headers?

@Friendica Support

Hypolite Petovan

2 years ago •

Thank you for the report. These headers are handled by the third-party library that we are using, PHPMailer. We've been using it for several years now, and you are the first one to report garbled emails so I'm not sure what to say. We don't have direct control over the output and we don't have any insight about your own system. As a result there's little we can do to troubleshoot and potentially fix this issue on either end.

It sounds like there's an overly zealous mail software somewhere as no other admin has had this issue before and we haven't changed the sending software.

Friendica Support reshared this.

Hypolite Petovan

2 years ago •

Ah wait, if you aren't using the phpmailer addon this isn't entirely accurate. We also are using the base mail() PHP function which relies on the current MTA of the server, either sendmail or postfix, etc...

What is your configuration in this regard?

Friendica Support reshared this.

Kyle 🎙 🎶

2 years ago •

I should be using the default mail setup, as I haven't changed anything. But I just found the PHP Mailer SMTP add-on. I can try enabling and configuring that for my local smtp server to see what that does.

Kyle 🎙 🎶

2 years ago •

Thanks very much for the help so far. If I don't use the addon, which is disabled by default, I have the Exim4 mail transfer agent on Ubuntu 22.04. It has a sendmail binary that identifies itself as Exin. It shows version 4.95 #2. Is it Exim that is putting leading spaces on some header lines?

Hypolite Petovan

2 years ago •

I’ve thoroughly checked the core code and I can confirm we don’t add the leading space. However, all the headers that show with a space were supplied as part of the PHP mail() call in Friendica, so I feel like it has to do with Exim. Would you be able to temporarily switch to Postfix to perform a test?

Friendica Support reshared this.

Kyle 🎙 🎶

2 years ago •

Unfortunately I think it may be non-trivial to replace Exim with Postfix on this server, as it is currently running several production mail accounts across multiple domains.

Incidentally, I used the interactive php shell to send one such garbled message back through the php mail function after rewriting the message body to escape the new line characters and rewriting the headers to remove leading spaces and add \r\n to the end of each line as I read in a guide on how to use the function. The message came to my inbox legible and ungarbled. I used the message I'm replying to here, which is already on the public forum, so I won't have to redact anything other than my own email address and my mail server domain. In case it helps, this is how I called the function.

$ php -a
php > mail ('redacted@mail.address', 'Testing php mail', "--9-424522331-136230018=:85663\nContent-Type: text/plain; charset=UTF-8\nContent-Transfer-Encoding: base64\n\nSHlwb2xpdGUgUGV0b3ZhbiByZXBsaWVkIHRvIHlvdSBvbiAiVGhhbmtzIHZlcnkgbXVjaCBmb3Ig\ndGhlIGhlbHAgc28gZmFyLiBJZiBJIGRv

$ php -a
php > mail ('redacted@mail.address', 'Testing php mail', "--9-424522331-136230018=:85663\nContent-Type: text/plain; charset=UTF-8\nContent-Transfer-Encoding: base64\n\nSHlwb2xpdGUgUGV0b3ZhbiByZXBsaWVkIHRvIHlvdSBvbiAiVGhhbmtzIHZlcnkgbXVjaCBmb3Ig\ndGhlIGhlbHAgc28gZmFyLiBJZiBJIGRvbid0IHVzZSB0aGUgYWRkb24sIHdoaeKApiIKCiJJ4oCZ\ndmUgdGhvcm91Z2hseSBjaGVja2VkIHRoZSBjb3JlIGNvZGUgYW5kIEkgY2FuIGNvbmZpcm0gd2Ug\nZG9u4oCZdCBhZOKApiIKCknigJl2ZSB0aG9yb3VnaGx5IGNoZWNrZWQgdGhlIGNvcmUgY29kZSBh\nbmQgSSBjYW4gY29uZmlybSB3ZSBkb27igJl0IGFkZCB0aGUgbGVhZGluZyBzcGFjZS4gSG93ZXZl\nciwgYWxsIHRoZSBoZWFkZXJzIHRoYXQgc2hvdyB3aXRoIGEgc3BhY2Ugd2VyZSBzdXBwbGllZCBh\ncyBwYXJ0IG9mIHRoZSBQSFAgbWFpbCgpIGNhbGwgaW4gRnJpZW5kaWNhLCBzbyBJIGZlZWwgbGlr\nZSBpdCBoYXMgdG8gZG8gd2l0aCBFeGltLiBXb3VsZCB5b3UgYmUgYWJsZSB0byB0ZW1wb3Jhcmls\neSBzd2l0Y2ggdG8gUG9zdGZpeCB0byBwZXJmb3JtIGEgdGVzdD8KClBsZWFzZSB2aXNpdCBodHRw\nczovL3RhbGthYm91dC5jZiB0byB2aWV3IG9yIHJlcGx5IHRvIHRoZSBjb252ZXJzYXRpb24uCmh0\ndHBzOi8vdGFsa2Fib3V0LmNmL25vdGlmeS8xNzgKCnRoYW5rcwpUYWxrYWJvdXQgRmVkZXJhdGVk\nIFNvY2lhbCBOZXR3b3JrIHwgUG93ZXJlZCBieSBGcmllbmRpY2EgQWRtaW5pc3RyYXRvcgo=\n\n--9-424522331-136230018=:85663\nContent-Type: text/html; charset=UTF-8\nContent-Transfer-Encoding: base64\n\nPCFET0NUWVBFIEhUTUwgUFVCTElDICItLy9XM0MvL0RURCBIVE1MIDQuMDEgVHJhbnNpdGlvbmFs\nIC8vRU4iPgo8aHRtbD4KPGhlYWQ+Cgk8dGl0bGU+RnJpZW5kaWNhIE5vdGlmaWNhdGlvbjwvdGl0\nbGU+Cgk8bWV0YSBodHRwLWVxdWl2PSJDb250ZW50LVR5cGUiIGNvbnRlbnQ9InRleHQvaHRtbDtj\naGFyc2V0PXV0Zi04IiAvPgo8L2hlYWQ+Cjxib2R5PgoJPHRhYmxlIHN0eWxlPSJib3JkZXI6MXB4\nIHNvbGlkICNjY2MiPgoJPHRib2R5PgoJCTx0cj4KCQkJPHRkIHN0eWxlPSJiYWNrZ3JvdW5kOiMw\nODQ3Njk7IGNvbG9yOiNGRkZGRkY7IGZvbnQtd2VpZ2h0OmJvbGQ7IGZvbnQtZmFtaWx5OidsdWNp\nZGEgZ3JhbmRlJywgdGFob21hLCB2ZXJkYW5hLGFyaWFsLCBzYW5zLXNlcmlmOyBwYWRkaW5nOiA0\ncHggOHB4OyB2ZXJ0aWNhbC1hbGlnbjogbWlkZGxlOyBmb250LXNpemU6MTZweDsgbGV0dGVyLXNw\nYWNpbmc6IC0wLjAzZW07IHRleHQtYWxpZ246IGxlZnQ7Ij4KCQkJCTxpbWcgc3R5bGU9IndpZHRo\nOjMycHg7aGVpZ2h0OjMycHg7IGZsb2F0OmxlZnQ7IiBzcmM9ImltYWdlcy9mcmllbmRpY2EtMzIu\ncG5nIiBhbHQ9IkZyaWVuZGljYSBCYW5uZXIiPgoJCQkJPGRpdiBzdHlsZT0icGFkZGluZzo3cHg7\nIG1hcmdpbi1sZWZ0OiA1cHg7IGZsb2F0OmxlZnQ7IGZvbnQtc2l6ZToxOHB4O2xldHRlci1zcGFj\naW5nOjFweDsiPkZyaWVuZGljYTwvZGl2PgoJCQkJPGRpdiBzdHlsZT0iY2xlYXI6IGJvdGg7Ij48\nL2Rpdj4KCQkJPC90ZD4KCQk8L3RyPgoJCTx0cj4KCQkJPHRkPgoJCQkJPHRhYmxlPgoJPHRoZWFk\nPgoJCTx0cj4KCQkJPHRkIGNvbHNwYW49IjIiIHN0eWxlPSJwYWRkaW5nLXRvcDoyMnB4OyI+CgkJ\nCQlIeXBvbGl0ZSBQZXRvdmFuIHJlcGxpZWQgdG8geW91IG9uICJUaGFua3MgdmVyeSBtdWNoIGZv\nciB0aGUgaGVscCBzbyBmYXIuIElmIEkgZG9uJ3QgdXNlIHRoZSBhZGRvbiwgd2hp4oCmIgoJCQk8\nL3RkPgoJCTwvdHI+Cgk8L3RoZWFkPgoJPHRib2R5PgoJCQk8dHI+CgkJCTx0ZCBzdHlsZT0icGFk\nZGluZy1sZWZ0OjIycHg7cGFkZGluZy10b3A6MjJweDt3aWR0aDo2MHB4OyIgdmFsaWduPSJ0b3Ai\nIHJvd3NwYW49Mz48YSBocmVmPSJodHRwczovL2ZyaWVuZGljYS5tcnBldG92YW4uY29tL3Byb2Zp\nbGUvaHlwb2xpdGUiPjxpbWcgc3R5bGU9ImJvcmRlcjowcHg7d2lkdGg6NDhweDtoZWlnaHQ6NDhw\neDsiIHNyYz0iaHR0cHM6Ly90YWxrYWJvdXQuY2YvcGhvdG8vNzQ5NDQzMzI2NjYzNjg1ZDllYzI4\nZTkyODcxODUxMzMtNC5wbmc/dHM9MTY2Nzc4NDA5NSI+PC9hPjwvdGQ+CgkJCTx0ZCBzdHlsZT0i\ncGFkZGluZy10b3A6MjJweDsiPjxhIGhyZWY9Imh0dHBzOi8vZnJpZW5kaWNhLm1ycGV0b3Zhbi5j\nb20vcHJvZmlsZS9oeXBvbGl0ZSI+SHlwb2xpdGUgUGV0b3ZhbjwvYT48L3RkPgoJCTwvdHI+CgkJ\nCTx0cj4KCQkJPHRkIHN0eWxlPSJmb250LXdlaWdodDpib2xkO3BhZGRpbmctYm90dG9tOjVweDsi\nPiZxdW90O0nigJl2ZSB0aG9yb3VnaGx5IGNoZWNrZWQgdGhlIGNvcmUgY29kZSBhbmQgSSBjYW4g\nY29uZmlybSB3ZSBkb27igJl0IGFk4oCmJnF1b3Q7PC90ZD4KCQk8L3RyPgoJCTx0cj4KCQkJPHRk\nIHN0eWxlPSJwYWRkaW5nLXJpZ2h0OjIycHg7Ij5J4oCZdmUgdGhvcm91Z2hseSBjaGVja2VkIHRo\nZSBjb3JlIGNvZGUgYW5kIEkgY2FuIGNvbmZpcm0gd2UgZG9u4oCZdCBhZGQgdGhlIGxlYWRpbmcg\nc3BhY2UuIEhvd2V2ZXIsIGFsbCB0aGUgaGVhZGVycyB0aGF0IHNob3cgd2l0aCBhIHNwYWNlIHdl\ncmUgc3VwcGxpZWQgYXMgcGFydCBvZiB0aGUgUEhQIDxjb2RlPm1haWwoKTwvY29kZT4gY2FsbCBp\nbiBGcmllbmRpY2EsIHNvIEkgZmVlbCBsaWtlIGl0IGhhcyB0byBkbyB3aXRoIEV4aW0uIFdvdWxk\nIHlvdSBiZSBhYmxlIHRvIHRlbXBvcmFyaWx5IHN3aXRjaCB0byBQb3N0Zml4IHRvIHBlcmZvcm0g\nYSB0ZXN0PwoJCTwvdGQ+CgkJPC90cj4KCTwvdGJvZHk+Cgk8dGZvb3Q+CgkJPHRyPgoJCQk8dGQg\nY29sc3Bhbj0iMiIgc3R5bGU9InBhZGRpbmctdG9wOjExcHg7Ij4KCQkJCVBsZWFzZSB2aXNpdCA8\nYSBocmVmPSJodHRwczovL3RhbGthYm91dC5jZiI+VGFsa2Fib3V0IEZlZGVyYXRlZCBTb2NpYWwg\nTmV0d29yayB8IFBvd2VyZWQgYnkgRnJpZW5kaWNhPC9hPiB0byB2aWV3IG9yIHJlcGx5IHRvIHRo\nZSBjb252ZXJzYXRpb24uCgkJCTwvdGQ+CgkJPC90cj4KCQk8dHI+CgkJCTx0ZCBjb2xzcGFuPSIy\nIiBzdHlsZT0icGFkZGluZy1ib3R0b206MTFweDsiPgoJCQkJPGEgaHJlZj0iaHR0cHM6Ly90YWxr\nYWJvdXQuY2Yvbm90aWZ5LzE3OCI+aHR0cHM6Ly90YWxrYWJvdXQuY2Yvbm90aWZ5LzE3ODwvYT4K\nCQkJPC90ZD4KCQk8L3RyPgoJCTx0cj4KCQkJPHRkPjwvdGQ+CgkJCTx0ZD4KCQkJCXRoYW5rcwoJ\nCQk8L3RkPgoJCTwvdHI+CgkJPHRyPgoJCQk8dGQ+PC90ZD4KCQkJPHRkPgoJCQkJVGFsa2Fib3V0\nIEZlZGVyYXRlZCBTb2NpYWwgTmV0d29yayB8IFBvd2VyZWQgYnkgRnJpZW5kaWNhIEFkbWluaXN0\ncmF0b3IKCQkJPC90ZD4KCQk8L3RyPgoJPC90Zm9vdD4KPC90YWJsZT4KCgkJCTwvdGQ+CgkJPC90\ncj4KCTwvdGJvZHk+Cgk8L3RhYmxlPgo8L2JvZHk+CjwvaHRtbD4K\n\n--9-424522331-136230018=:85663--\n\n", "Precedence: list\r\nX-Friendica-Host: talkabout.cf\r\nX-Friendica-Platform: Friendica\r\nX-Friendica-Version: 2022.10\r\nList-ID: &lt;notification.talkabout.cf>\r\nList-Archive: &lt;https://talkabout.cf/notifications/system>\r\nX-Friendica-Account: &lt;kyle@talkabout.cf>\r\nReferences: &lt;c162020a-4763-688c-b552-6bb105124076@redacted.mail.server>\r\nIn-Reply-To: &lt;c162020a-4763-688c-b552-6bb105124076@redacted.mail.server>\r\nFrom: Talkabout Federated Social Network | Powered by Friendica &lt;notify@talkabout.cf>\r\nReply-To: Talkabout Federated Social Network | Powered by Friendica &lt;notify@talkabout.cf>\r\nMIME-Version: 1.0\r\nContent-Type: multipart/alternative; boundary=\"9-424522331-136230018=:85663\"\r\nFrom: \"redacted@mail.domain\" &lt;notify@talkabout.cf>");

Steffen K9 🐰

2 years ago from libranet.de •

Just checked the headers of my notification mails. They look ok.

I'm using the phpmailer add-on and my own mail server with postfix.

Peter Vágner

2 years ago •

I haven't a resolution to this but my setup is similar. I am not using phpmailer addon thus using php's builtin mail function for sending out emails. php uses sendmail binary on my system, that's msmtp which in turn uses local instance of haraka smtp server.
Email messages that I am getting into my inbox don't suffer from this.
I can't think of how exim would garble the headers though.

Peter Vágner

2 years ago •

Looking further, these indented headers are put into the same array as the first one, see the file src/Util/EMailer/MailBuilder.php, so it's very unlikelly friendica it-self is the problem.
Are you using other addons that might affect generating or sending emails?

Kyle 🎙 🎶

2 years ago •

Thanks for the suggestion. I didn't suspect any of my enabled addons to be the cause of the problem, but disabling them and reenabling them one at a time is a troubleshooting step that I missed in the process of trying to find out what is going on. I just saw the indented headers, unindented them and resent the message and found it cleared up, so I thought it was traceable to the mailer at that point. Thanks for pointing out the addons. I'll report back here if I find anything, and I will be stumped even further if I find the same problem after disabling everything.

Kyle 🎙 🎶

2 years ago •

I disabled every addon, even backed up my configuration, deleted all the files in my site directory and untarred the archives again. I am currently writing this post with all addons disabled, and a clean site directory, save my config/local.config.php and the database I set there. At this point, I have done everything short of trashing the database and configuration and starting over. Still I have indented headers whenever Friendica sends an email, and no other site using php on this server is indenting headers. At this point, I'm stumped completely.

Hypolite Petovan

2 years ago •

@Kyle 🎙 🎶 I have something you can try. Can you please manually replace the instances of \n by \r\n in src/Object/Email.php on lines 140 and 143 and try again, see if it changes anything?

@Kyle 🎙 🎶

Friendica Support reshared this.

Kyle 🎙 🎶

2 years ago •

@Hypolite Petovan Done. I'm still getting garbled emails, but there has definitely been an improvement. Now most of the Friendica headers are where they should be, but Reply-To, MIME-Version and Content-Type are still indented by one space.

@Hypolite Petovan

Hypolite Petovan

2 years ago •

@Kyle 🎙 🎶 EXCELLENT! I'll have a full patch in the next hour.

@Kyle 🎙 🎶

Friendica Support reshared this.

Hypolite Petovan

2 years ago •

@Kyle 🎙 🎶 See https://github.com/friendica/friendica/pull/12173/files, there's an additional file to edit at src/Util/Emailer.php lines 166 through 168 with the same edit.

And then you should be ok. The big question is whether everyone else will still be ok. 🙈

@Kyle 🎙 🎶

Friendica Support reshared this.

Kyle 🎙 🎶

2 years ago •

@Hypolite Petovan Worked like a charm! I do hope everyone else will be OK, because my emails are coming out perfect now. Thanks very much for the fix.

@Hypolite Petovan

Hypolite Petovan likes this.

Hypolite Petovan

2 years ago •

@Kyle 🎙 🎶 In the PHP documentation, they tell developers to use LF-only as a last resort, so I'm hoping that clients that can support LF will also support CRLF by now.

@Kyle 🎙 🎶

Friendica Support reshared this.

⇧