Zammad: Japanese encoding 7-bit iso-2022-jp is processed incorrectly

Konnichiwa @lsergei - could you please provide a complete example mail that we then can use for our test suite? That would help a lot. Thanks!

Konnichiwa @thorsteneckel, thank you for the fast response.

How you'd like me to forward you a problematic message? Just forwarding it might (and most likely will) replace headers

Usually the mail program has an option to save a raw mail to .eml format. For Apple Mail e.g. it's just File -> Save As. Please make sure to open that file in a regular text editor and remove all sensitive data before uploading it here.

@thorsteneckel thank you for the explanation. Please see the message enclosed.
As GitHub doesn't support uploading of eml files, file extension was changed to txt.

メルマガ解除.txt

Domo arigato!
Issue is verified, we'll update this issue as soon as we're ablet to work on it.

So the root cause of this issue is that the dependency we're using for parsing mails does not support ISO 2022 JP encoded mails. There is an additional gem that brings that support to older versions but is not compatible with the (latest) version we're using.
There is a 6 year old, stale and pending Pull Request on which I added a comment to ask for instructions on how to pick up from there. Waiting for feedback...

Can we please re-evaluate if there possibly is a working solution by now?

There have been no relevant changes in the mail gem and my last comment is therefore (unfortunately) still valid.

Mantas, can you please have a look if there's maybe a solution to work around this without hacking over 7 year old hacked gems? Maybe a conversion would be possible before or so.

Hi @lsergei - the great @mantas found a workaround for the mail gem issue so that we can handle it in our code instead. A sustainable solution in the gem itself would be much nicer but a huge task we can't accomplish right now. However, feel free to test it in the upcoming Zammad 3.4 release in about 2 weeks.

Hi @thorsteneckel and @mantas - thank you, appreciate your efforts! Will test and give you feedback.

Looking forward to new release

Hi. I'm new at zammad.
I think this problem is not fixed at 3.6.x
my zammad server show similar garbled characters.

@hiromayu Would it be possible to receive an original eml file of a email with above encodings that still does fail within Zammad 3.6? You can send this eml to support [at] zammad [dot] com with referencing this issue. We'll have a look then.

Thank you.

Hi MrGeneration. I send eml file to support .

@hiromayu @MrGeneration this problem is not fixed yet indeed. I kind of gave up (have someone to fix those emails manually)

Just as a small follow up:
Thanks to @hiromayu and the sample mail we could pin down the issue.

It's indirectly covering this issue and issue #3096

Hello. Thank you All.
I think this problem is fixed by applying patch.
https://github.com/kuroda/mail-iso-2022-jp.
my Zammad server handle the mail that I had sent to support perfectly.

@hiromayu thank you for the update! May I know how you applied the patch? Just installed the gem as described in github article?

I forgot to tell that I could not distinguish the reason that fixing this problem.
I applied patch and updated zammad server.so one of them or both fixed the problem.
I applied the patch by command.
sudo zammad run gem install mail-iso-2022-jp
and reboot.

Thank you @hiromayu . Just installed the gem but supposedly just install wouldn't fix the issue. May I know what did you change in server.so?

I'm sorry. I didn't do anything to server.so. I just install that patch and update zammad server introduced by ubuntu updater.
if just installing the patch wouldn't work. it's too first to decide problem fixed. I will check my email at several day.

@hiromayu わかりました。I did the same thing and will monitor if the problem is gone.

どもありがとう！

Just an important side note: Keep in mind that this action is not update safe.
So if you really want to stick with the solution and bugfix #3096 is not installed on your system we can't stop you.

However, all bugs that are appearing because of this no longer validate for bug fixing.

@MrGeneration from what @hiromayu suggested it is actually update safe since there is no code modification.

Perhaps closing the ticket is too early, but for today I have no issues after installing said gem.

You did change the enviroment.
This being said, if you don't have a specific source code installation setup, this action is not update safe.

Hey there! To provide some technical background: Installing a gem has no effect on its own. It won't get loaded without further action/code changes. Therefore I'd suggest to remove the installed gem again to restore a vanilla environment. I suspect that the provided fix of #3096 created 7 days ago resolved your issue @hiromayu 🤔

Thank you for all. So, update of Zammad was the reason that fix this problem.
I'm sorry for incorrect information.
I will reinstall Zammad server and check the result. and report it's really fixed at several days after.

Hello. I got another example of this problem.
but, I think small difference before update of zammad.
before update, all of japanese character was garbled. but now subject of mail is correct. only main text of mail is garbled.
What should I do?

Please create a new issue.
This is a complex issue and may appear to be the same but also might be of slightly different nature.