Zammad: LDAP/Exchange UTF-8 Status Code 500

Hi @e311 - could you please provide the upper part of the log, too? This is where the required information is printed out and which is currently missing. Additionally: Could you please describe the steps that you performed to get to the issue? This is currently not clear. Thanks!

Hi,
i will try it.
i open the (in admin --> system -->Integarion) LDAP. I start the configuration All fine.
Domain DN User Password.

I used the standart mapping.

Ad the LDAP Config sreen ist look fine.

I press safe. And i get error message.

After i reload the Page the Frame is emty.

and here is the full log file:

log.txt

Hi @e311 - thanks for the comprehensive writeup! The interesting line in the log is this one:

E, [2018-07-24T11:39:30.491795 #973] ERROR -- : "\xC5" from ASCII-8BIT to UTF-8 (Encoding::UndefinedConversionError)
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.5/lib/active_support/core_ext/object/json.rb:36:in `encode'
...
/opt/zammad/app/controllers/application_controller/renders_models.rb:70:in `model_update_render_item'
/opt/zammad/app/controllers/application_controller/renders_models.rb:66:in `model_update_render'
/opt/zammad/app/controllers/settings_controller.rb:31:in `update'

Which basically says that there are specially encoded chars in the LDAP settings which will get/got stored. I can see those in the log, too.

I think this is sufficiently enough to reproduce the issue. Thanks for your support so far. We will fix the issue.

Please note that the Exchange integration is affected, too: #2152

Hi @e311,

We've been digging into this issue and are pretty confident we understand what's going on, but we need to ask for your help again to be completely certain.

Step 1: Confirm the cause of the bug

Try the following line in your terminal, in the Zammad directory. If our suspicions are correct, then it should fail (_i.e.,_ raise an error and show a traceback):

$ rails r "puts Setting.get('ldap_config').to_json"

If you saw a traceback for the same Encoding::UndefinedConversionError that you got in the initial bug report, then great — onto Step 2! If not, please confirm that you haven't reset your database or altered your LDAP configuration since the last time this bug occurred. If you're still not getting an error, please let us know so that we can continue to investigate.

Step 2: Send us one last bit of crucial information

Run the following command in your terminal:

$ rails r "pp Setting.get('ldap_config').except('bind_pw')" > zammad_2140_debug_output.txt

Then, send the resulting file (zammad_2140_debug_output.txt) to [email protected]. Be sure to mention that the message is for Thorsten Eckel, and for bug #2140.

With this, we should be able to get a fix working right away.

Thanks again for all your help!

This is the First massege

serveradmin@support:/opt/zammad$ rails r "puts Setting.get('ldap_config').to_json"
Traceback (most recent call last):
        4: from bin/rails:3:in <main>'         3: from bin/rails:3:inrequire_relative'
        2: from /opt/zammad/config/boot.rb:3:in <top (required)>'         1: from /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:inrequire'
/usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require': cannot load such file -- bundler/setup (LoadError)

Gesendet: Montag, 06. August 2018 um 11:53 Uhr
Von: "Ryan Lue" notifications@github.com
An: zammad/zammad zammad@noreply.github.com
Cc: e311 falk_wantzen@gmx.de, Mention mention@noreply.github.com
Betreff: Re: [zammad/zammad] LDAP/Exchange UTF-8 Status Code 500 (#2140)

Hi @e311,

We've been digging into this issue and are pretty confident we understand what's going on, but we need to ask for your help again to be completely certain.

Step 1: Confirm the cause of the bug

Try the following line in your terminal, in the Zammad directory. If our suspicions are correct, then it should fail (i.e., raise an error and show a traceback):

$ rails r "puts Setting.get('ldap_config').to_json"

If you saw a traceback for the same Encoding::UndefinedConversionError that you got in the initial bug report, then great — onto Step 2! If not, please confirm that you haven't reset your database or altered your LDAP configuration since the last time this bug occurred. If you're still not getting an error, please let us know so that we can continue to investigate.

Step 2: Send us one last bit of crucial information

Run the following command in your terminal:

$ rails r "pp Setting.get('ldap_config').except('bind_pw')" > zammad_2140_debug_output.txt

Then, send the resulting file (zammad_2140_debug_output.txt) to [email protected]. Be sure to mention that the message is for Thorsten Eckel, and for bug #2140.

With this, we should be able to get a fix working right away.

Thanks again for all your help!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

rails r "pp Setting.get('ldap_config').except('bind_pw')" > zammad_2140_debug_output.txt
prodruce a emty file (i can not upload because it is emty)

Hi @e311 - Please try the following:

zammad run rails r "pp Setting.get('ldap_config').except('bind_pw')" > zammad_2140_debug_output.txt

OR:

bundle exec rails r "pp Setting.get('ldap_config').except('bind_pw')" > zammad_2140_debug_output.txt

but should be the first line.

I try the first Command:
The same as "Serveradmin" with sudo or with out sudo

Bundel my be not installed?

@e311, that's very strange. Let's ignore the second command (bundle exec rails ...) for now — I don't think that's the right path here. Could you maybe try to back up the log file and create a new empty one?

/opt/zammad:# gzip production.log
/opt/zammad:# mv production.log.gz production.log.15.gz
/opt/zammad:# touch production.log
/opt/zammad:# chown zammad.zammad production.log
/opt/zammad:# chmod 0644 production.log
/opt/zammad:# zammad run rails r "pp Setting.get('ldap_config').except('bind_pw')" > zammad_2140_debug_output.txt

Thanks for all your help working through this issue with us.

@rlue we did the commands yesterday with @e311 .

THe log has perfectly fine access rights, I double checked that.
I'll send you the ldap-config output in mattermost, in case you don't have it yet.

Whoops, you totally sent me the log at the start of the day yesterday. Don't know how I missed it!

@e311, I didn't see what I was expecting to see in your debug output. Now that @MrGeneration was able to help you get your zammad run rails r ... commands working, could you try the following again and tell me if it's raising the Encoding::UndefinedConversionError that this issue is for?

$ rails r "puts Setting.get('ldap_config').to_json"

I get the Message "Your Ruby version is 2.5.1, but your Gemfile specified 2.4.4"
I sorry it will not work (i try all the thinks that Google say to me)

Your Ruby & your Elasticsearch version are not compatible with Zammad.
Use Ruby 2.4.4 and Elasticsearch up to 5.6.

@e311 sorry actually, totally my fault. Forgot to add zammad r again. -_-'

I meant

$ zammad run rails r "puts Setting.get('ldap_config').to_json"

Ther is the message :

Neues Textdokument (3).txt

So the output from the last command was exactly what I was expecting. To sum up, it looks like some of the user attributes on your LDAP server have binary data values. Zammad tries to convert these to strings and then pass them around from the backend to the frontend, but since they're not actually strings, we wind up running into some encoding / conversion problems.

Of course, since I can't connect to your LDAP server, I can't be 100% sure that a fix will work until you try it for us. So, I'd like to ask your help on two more points:

1. The last time I asked you to run some commands and email us the output (last Wednesday), the commands I gave you weren't quite right, and didn't exactly give us the information we needed. Here's a script that should get it right this time (download here):

   # Place this script in your Zammad directory and run it with
   #
   #     $ zammad run rails r 2140_bug_report.rb
   #
   # It should create a file named 2140_bug_report.txt.
   # Please send this file to [email protected], and mention that it's for Ryan.
   
   class Hash
    def deep_values
      values.map { |v| v.try(:deep_values) || v }.flatten
    end
   end
   
   string_values = Setting.get('ldap_config').except('bind_pw').deep_values.select { |v| v.is_a?(String) }
   
   File.binwrite('2140_bug_report.txt', Marshal.dump(string_values))
   

   Sorry to make you do it twice.

2. I've written a patch which I think should fix the issue. Basically, anytime it encounters a binary data value in the LDAP sync, it skips over it instead of trying to save it as a string.

   I'd like to confirm that the fix works on your machine before we merge these changes into the main codebase. Here is a bugfixed version of the lib/ldap/user.rb file.

   Download it and copy it to /opt/zammad/lib/ldap/user.rb, then restart Zammad (important!) and run through the entire LDAP configuration wizard again. Since you already have an existing LDAP configuration, that means you'll have to click on the "Change" button at the bottom of the panel:

   

If Step 2 still doesn't fix it for you, please run the script from Step 1 again, and be sure to send us both files.

Thanks again for your help and your patience!

I reboot the hole system, but i can not push the change Button.

befor the patch
2140_bug_report (Step1).txt

after the patch
2140_bug_report(Step2).txt

i also Try a new LDAP configuration. I restore the System befor the LDAP conf. but ist the same result.

I have check the domain functional level. It is 2003 can this be the Problem

I reboot the hole system, but i can not push the change Button.

Thanks for clarifying this. I had not understood the scope of the problem, originally.

i also Try a new LDAP configuration. I restore the System befor the LDAP conf. but ist the same result.

Do you mean you restored the system to before the LDAP conf, successfully ran through the whole wizard, and still wound up with the same problem? Or do you mean you tried to restore the system to before the LDAP conf, but you're still getting this error and can't get to the wizard?

If it's the former, then the patch still needs a little more work; I should have a fix for you tomorrow. If it's the latter, then you can try to reset the LDAP config with

$ RAILS_ENV=production zammad run rails r "Setting.set('ldap_config', {})"

(You can leave out the RAILS_ENV=production part if you're doing this in a development environment.)

I restored a backup, The backup was created before I started the LDAP configuration for the first time.

I try to reset the LDAP con with
$ RAILS_ENV=production zammad run rails r "Setting.set('ldap_config', {})"
i not "reset" the conf, it still works.
It now seems like the whole system is running, without an error message

(Without LDAP)

Okay, so you're saying:

1. Zammad is working,
2. you currently have no LDAP sync configured, and
3. you're not getting an error message on the LDAP sync page anymore?

(To confirm point 2, you can run RAILS_ENV=production zammad run rails r "puts Setting.get('ldap_config')", which should print {} to the console.)

If all of the above is true, and you have copied the patched version of lib/ldap/user.rb that I linked to above, can you try to set up the LDAP sync one more time, and tell me if you still run into the same problem?

Or if I've misunderstood, please let me know what the situation is.

Thanks.

@rlue - JFI: Have a look at https://community.zammad.org/t/unable-to-access-ldap-in-integrations-statuscode-500/1116/5 😍

Sorry for the late reply. Yes everything correct. After the LDAP again configured, I had the same error again.

@rlue - as @e311 already stated there is still something wrong: https://community.zammad.org/t/unable-to-access-ldap-in-integrations-statuscode-500/1116/21

FYI: I've got the same issue and the patched user.rb did'nt work for me, too.
Thanks for your work in advance

Everybody who still has the issue after changing the file and restarting the application please provide the error (including backtrace) from your log/production.log file. Thanks!

@e311, @tbeitter, and everyone else who's currently experiencing this issue — I have _yet another_ debugging script I'd like you to run. I don't want to speak too soon, but this should really be the last one.

Place it in your zammad directory and then run

$ zammad run rails r 2140_improved_bug_report.rb

You will be prompted for your LDAP server URL and login credentials, and then the script will generate a file called 2140_debug_log.txt. This file contains a sample of user attributes from your LDAP server, so if that might mean anything sensitive, do NOT post it publicly. Instead, send it to [email protected], and mention it's for Ryan.

Thank you all for your continued patience as we work through this bug.

Hello,
If i try to run the new script i get this (LDAP is on) no file was created.

serveradmin@support:/opt/zammad$ sudo zammad run rails r 2140_improved_bug_report.rb
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/railties-5.1.5/lib/rails/commands/runner/runner_command.rb:34:in `load': 2140_improved_bug_report.rb:77: invalid multibyte char (UTF-8) (SyntaxError)
        from /opt/zammad/vendor/bundle/ruby/2.4.0/gems/railties-5.1.5/lib/rails/commands/runner/runner_command.rb:34:in `perform'
        from /opt/zammad/vendor/bundle/ruby/2.4.0/gems/thor-0.20.0/lib/thor/command.rb:27:in `run'
        from /opt/zammad/vendor/bundle/ruby/2.4.0/gems/thor-0.20.0/lib/thor/invocation.rb:126:in `invoke_command'
        from /opt/zammad/vendor/bundle/ruby/2.4.0/gems/thor-0.20.0/lib/thor.rb:387:in `dispatch'
        from /opt/zammad/vendor/bundle/ruby/2.4.0/gems/railties-5.1.5/lib/rails/command/base.rb:63:in `perform'
        from /opt/zammad/vendor/bundle/ruby/2.4.0/gems/railties-5.1.5/lib/rails/command.rb:44:in `invoke'
        from /opt/zammad/vendor/bundle/ruby/2.4.0/gems/railties-5.1.5/lib/rails/commands.rb:16:in `<top (required)>'
        from /opt/zammad/bin/rails:4:in `require'
        from /opt/zammad/bin/rails:4:in `<main>'
serveradmin@support:/opt/zammad$

production.log

also have this ldap error... (ubuntu 16.04 installation through repo)

root@zammad01:/opt/zammad# zammad run rails r 2140_improved_bug_report.rb
Please specify a valid ruby command or the path of a script to run.
Run 'bin/rails runner -h' for help.

/opt/zammad/vendor/bundle/ruby/2.4.0/gems/railties-5.1.5/lib/rails/commands/runner/runner_command.rb:37: trailing `_' in number
2140_improved_bug_report.rb
     ^
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/railties-5.1.5/lib/rails/commands/runner/runner_command.rb:37: syntax error, unexpected tIDENTIFIER, expecting end-of-input
2140_improved_bug_report.rb

@redbear-ger, please ensure that 2140_improved_bug_report.rb is in your /opt/zammad directory.

@e311 whoops, that's a weird bug.

I've modified the script by removing some unnecessary multibyte characters from the console output. You can re-download the script and try again, or simply change the smart quotes on line 77 to regular quotes:

# replace “#{default_base_dn}” with '#{default_base_dn}'
printf "Base DN (leave blank for default '#{default_base_dn}'): "

Not 100% sure this will fix it, but I have a hunch.

@rlue, no 2140_improved_bug_report.rb in /opt/zammad

btw: As soon as I delete the LDAP config ( RAILS_ENV=production zammad run rails r "Setting.set('ldap_config', {})" ), the problem disappears.

@redbear-ger thanks for your quick feedback; please download the bug report script, place it in /opt/zammad, and try again.

Even if the problem is 100% gone now, it would help a lot to have your sample data so we can build tests to make sure this doesn't happen again.

The bug is gone but also the ldap integration ;)
Will Download the Report Script tonight

Hello,
I have the same issue and tried all the suggestions above. None of them solve the problem.
So I decided to reproduce the issue on a dev system to give you a debug file without providing sensitive data.

The 2140_improved_bug_report.rb script returned me following text file:
2140_debug_log.txt

Thanks!

Thanks @hublux! For some reason, there are a bunch of places in the file where control characters (_e.g.,_ ^D) are replaced with a literal carat (^) followed by a literal letter (D), and this is causing errors when trying to parse the file. Did you do anything to the file before uploading it, by any chance? (Say, opening it in a text editor, then closing it, and clicking "save changes"?) Or do you suppose that maybe GitHub might have done some kind of processing on it when you uploaded it?

To confirm that the original file is correctly formatted on your machine, please try the following command in your Zammad directory, and verify the output:

$ zammad run rails r "puts Marshal.load(File.read('2140_debug_log.txt')).first.inspect"

# should print the following:
["dn", "CN=Exchange Online-ApplicationAccount,OU=SBSUsers,OU=Users,OU=MyBusiness,DC=ebusiness,DC=local"]

If that's all working as expected, please email the file to [email protected].

Hello,
Thank you for your reply.

Indeed, it looks like some of my actions did change the file when I copied the content out of it.
checking it with cat, it looks a little bit different.

I tried the command and receive the correct output.

I have sent the file to zammad support.

Here is also the new file
2140_debug_log.txt

@hublux, thanks again for your speedy feedback. I think there's something wrong with the debug script I wrote, because this new file isn't quite right either (it's missing a lot of data — 4kB, compared to the 16kB debug log you originally sent).

I've since patched the debug script to (hopefully) make the 2140_debug_log.txt file more robust and safer to transmit without these weird encoding problems. Please download it one more time and try again. (Feel free to include it in your reply on this thread, if it's more convenient for you.)

2140_debug_log.txt
There you have the output with the new script
regards!

Hey everyone,
also chiming in, as the same problem affects me, too - even after updating to the new lib/ldap/user.rb

2140_debug_log.txt

I also started digging around a little bit in the ldap_config-hash and started getting rid of all the !binary contents. But it turned out, that even though I had 4 of these, only two screwed with the encoding.

I had to remove usersmimecertificate and msexchmailboxsecuritydescriptor.

The other two, msexchmasteraccountsid and securityprotocol did not cause any trouble.

Hello,

Just to clarify, I'm not sure if this is obvious since I have not seen anybody mentioning this.
As a LDAP connection I use Active Directory with German as Systemlanguage. So I installed a new system and a seperate Domain. The only difference from the previous system where the issue occured is that on the new system the language is set to U.S English instead of German. With this setup there is no error to be found, everything working as expected. So I took my original zammad system where I had the problem and connected it with newly installed Domain and the problem is gone.

Running following on the working setup give me following output:
sudo zammad run rails r "puts Setting.get('ldap_config').to_json"

workingsetup.txt

If I run the command to check the ldap config with the original setup it fails and receive following error:

Hope this helps you.
Regards!

Thanks @hublux! We've got a fix in the works thanks to your input; it should be merged to develop shortly!

After Updating still get the same error. (But without Exchange-setup, only LDAP-Sync)

Hi @tidet, thanks for the feedback. Please follow the directions in this comment; your sample data will go a long way in helping us resolve this issue conclusively.

Hi,
Just to inform I had the same issue again but the old config did still exist. So please proceed as following if you experince this issue again.

Delete the old config with following command:
sudo RAILS_ENV=production zammad run rails r "Setting.set('ldap_config', {})"

• stop zammad service

• update zammad

• start the zammad service again and try to reconfigure the ldap connection

After this I did not face any issue.

Regards

@rlue should I try solution above or stay at current error for debugging purposes?
E-Mail with logfile is out to support@

@tidet, now that we have your sample LDAP data, feel free to go ahead and try the above. I did find one value which _might_ still cause problems anyway, but it's probably fine. And whether or not the directions above fix it for you, there'll be another patch in the pipeline coming later today.

EDIT: Changed my mind. Since the fix is working for you, I'm going to refrain from modifying the code and adding any unnecessary complexity until it's actually broken for someone.

@hublux Fix worked for me, too.

@hublux fixed as well

we still face this issue. I did was @hublux wrote, deleted the config, updated zammad (2.6.0-1534939663.6d23dae9.stretch now), restarted zammad and configured our ldap connection. Do I have to manually patch code? Thanks in advance.

I was on holiday, sorry for the late reply here and if I missed anything :)

Hi @tbeitter,
If you run this command do you receive any output:
sudo zammad run rails r "puts Setting.get('ldap_config').to_json"

If so check this out:
https://github.com/zammad/zammad/issues/2140#issuecomment-417605384

Regards

Yes, I get an error similar to yours:

"\xC2" from ASCII-8BIT to UTF-8 (Encoding::UndefinedConversionError)

Hi,

I just checked my version of Zammad I updated this morning:

It differs from yours.

Could you please procceed again with these steps:

• Delete LDAP config
• stop zammad service
• update zammad again
• start zammad service

Regards

@hublux that worked for me! Thanks!

Big shout out to @hublux for all your help getting other users sorted out. e07f41ed now includes a database migration that should automatically clear faulty LDAP configurations so that any users who encounter this issue in the future should no longer have to deal with these manual steps.