Zammad: PGP

+1 for S/MIME Support

Okay, I've changed the title. The best would of course if it would offer both things.

I would appreciate to have to possiblity to send and recieve X.509 encrypted mails.

YES, it would be very nice to be able to send and receive X.509 signed or encrypted mails.

So X.509 encrypted mails are S/MIME mails. I personally would like PGP more (also because it is more widespread), but this issue is about both systems. 😃

I'd add encrypted notifications to the wishlist - therefore users need to be able to add there public key to their profile.

FYI: A nice plugin with such functionality exists for redmine: https://github.com/C3S/redmine_openpgp

This gem seems to be not really maintained (last commit ), but there are multiple others. Personally I'd say this gem has the most recent commit.

Today we had the first customer request asking for PGP-Support because she did not want to request her bank details via an unencrypted connection. Therefor +1

+1.
We are working with the IT SEC community and receive tons of PGP encrypted mails we can not open in Zammad, so we have to export & decrypt, it's a hell of a workflow. +1 for supporting PGP and maybe S/MIME Mails in Zammad :)

Nowadays e-mail encryption is a must. The General Data Protection Regulation (GDPR) requires Privacy by Design and by Default and according to the state of the art.

Technically this might be easy to solve by integrating the p≡p engine (pretty easy privacy).

+1
We often receive encrypted mails. Exporting and manually decrypting encrypted tickets is painful, and then we can't even answer to tickets via the ticket system.

+1
Without PGP/GPG I can not migrate from OTRS to Zammad. I have many encrypted tickets with gnupg in my old OTRS.

+1

We would like to GnuPG sign all outgoing ticket emails after a very ugly case of faking our email to rob our customers.

Maybe PeP could be integrated as it is a easy and usable solution for PGP encryption.

I also think it would be a great feature to send signed emails. This would give some degree of confidence for the recipient, that the address is real and can be trusted.

Next step would be to fully encrypt the message.

@martini @monotek we're thinking about tackling the GPG part of this ourselves, in-house. Are there any restrictions with getting this merged, if we do? any functionality you'd like to see or require?

Hi @luto - sounds good! Things that are required for a merge are:

• Tests, preferably RSpec
• Documentation
• Code API documentation
• QAd code via our rubocop and coffeelint configuration

We already looked at some Ruby gems that provide an API to the gpg CLI binary. As far as I remember none of them looked really promising. Please make sure to rely only on maintained/quality dependencies since this is a critical point.
A custom implementation is possible but keep extensibility and maintenance in mind. Don't put all the logic in the module but create a lib class for it. Respect the single responsibility principle.

Functionality should include all variations of signing, verifying and en-/decryption of sending and receiving mails 🤓
Handling incoming messages should be done as early as possible to have the content accessible in other components.
Handling outgoing messages should be done as late as possible to be able to access the content in other components.

There should be a nice admin interface to manage public and private keys.

Feel free to contact us at [email protected] to ask any questions you have and get our support on this. We are more than happy to receive yours 💪 Let's do it the Zammad way 🏎

We also got the first customer that requires such communication, so i also vote for this one. Is there already any progress? I'd also like to help testing a beta.

@martinseener afaik this is not on the roadmap of the Zammad team. We're currently planning to implement this on our own. If you'd like to chip to, so we can get it done faster and you can serve your customer, feel free reach out via the email-address on my github profile.

@thorsteneckel I'm currently prototyping an implementation for receiving mails in two parts:

1. decrypt the mail in Postmaster::PreFilter: set decrypted message as content; throw away original; store used decrypt-key in metadata
2. in Postmaster::PostFilter create an GpgCryptoInfo object, attach it to the Article; store information like the used decrypt-key and signature status permanently

A couple of questions:

• do you think this should be implemented using those filter APIs? Or should this be hardcoded in Channel::EmailParser?
• are there any Postmaster::PreFilter subscribers, which need access to decrypted message content? In this case a hardcoded implementation or a Postmaster::EarlyPreFilter is needed.
• is there any information except for used key to encrypt/decrypt/sign/verify as well as signature status of received mail you'd like to see kept permanently?

Also, a general question: EmailParser or the filters seem to be the perfect place to _decrypt_ mails; can you think of a "perfect" place to encrypt them?

I hope that this issue is the right place to ask implementation questions. If not, please point me to another, preferably public, one :) Thanks!

Hi @luto - I'm currently writing my thoughts about this down. Since it's a rather big and important topic it takes some time. I try to get it done by end of the week. Thanks for your understanding.

Small update: Postmaster::PreFilter filters are ordered. I currently placed mine right after IdentifySender, so I can figure out which gpg keys to try. Knowing that, the filters seem to be the right place to decrypt for me.

@thorsteneckel Thanks for taking the time! Looking forward to your write up. :)

Hi @luto, glad to hear that! I like to have the coordination in this issue as you proposed. So it will be transparent and we can re-use the shared information for a technical documentation. You can refer to this issue in your working branch of your fork and it will get referenced here. Please create a Pull Request after the changes are complete. In the meantime I will check out your branch and see the changes.
We should move the S/MIME functionality to a separate issue since we won't implement it now.

Regarding your questions: You are totally on the right path. I'll write some things down and answer your questions on the way.

General development

In our experience test driven development (with RSpec) works best for this kind of tasks. It's up to you how you approach it but we will definitivly need a comprehensive test suite for the functionality. Therefore I will provide you with a RSpec helper to ease testing and support you on the way the best I can. I will add it to the develop brach in the next days. You should use this branch as your base since it's our working branch, too. It will get merged into master and replace stable in the future.

Postmaster::PreFilter

Postmaster::PreFilters are the right place. You can influence the time of execution by a number prefix in the name of the filter registration. Lower numbers will get executed earlier.
For systems already initialized a migration that adds the setting is needed, too.
I would propose the a name something like app/models/channel/filter/decrypt.rb - but in the end it's up to you.

After the registration is done Zammad will execute the filter for each incomming mail.

PGP message class

Since we will only implement PGP at this point we have to make sure that we won't add obstacles for adding S/MIME later. Additionally it might be necessary in the future to support PGP on other channels than email only. Having this in mind will ensure that the backend and API we design will be easy to test and integrate. Following my proposal (open for discussion):

There should be a class called PgpMessage located in the file lib/pgp_message.rb. This class takes one required argument which will be the (possibly) encrypted message string/mail content. Example:

instance = PgpMessage.new(email_content)

The instance should provide the following interface:

• #signed?
• #verified?(signature) # signature is optional and can be an external signature from e.g. an attachment
• #signature_error
• #crypted?
• #decrytable?
• #decrypted
• #key
• #decryption_error
• #encrypt(public_key(s))
• #sign

PGP interface

To be honest I haven't found any gem that I like to have as a dependency. They all look unmaintained, complex or need C compilation of extensions. It might be worth to try if we can't just access the PGP CLI. What do you think?

PgpMessage integration in the Postmaster::PreFilter for decryption

The PgpMessage class can then be used in the Postmaster::PreFilter to initialize an instance and check the message for relevance. If it's an PGP encrypted message we can procceed to use the other methods to extract the data we need and overwrite the content for body and attachments in the given mail hash.
Additionally we have to store some meta information (as you already stated) in the article that will be created by setting the x-zammad-article-preferences header:

mail[:'x-zammad-article-preferences']['decryption_key'] = instance.key
...

I think the following metadata should be sufficient:

• decryption_key <- key
• decryption_error <- decryption_error (in case of existence)
• encrypted_message <- mail['body']
• signature_status <- verified?
• signature_error <- signature_error (in case of existence)

supported variations of decryptable messages

The PgpMessage class should be able to handle (at least?) the following combinations and variations of incomming PGP messages:

• encrypted
• signed
• encrypted + signed
• inline
• attachment

Do you have enough example mails already?

We have to have a comprehensive list of test cases to cover various cases which can be extended easily in the future.

sending encryted messages

Since Zammad only supports sending HTML emails we only need to cover sending detached PGP mails.

The place where Zammad builds an email from given attributes is in app/models/channel/email_build.rb in the self.build method. This should be extended to use the PgpMessage class to create an instance with the given body attribute and use the encrypted and signed methods to create the PGP encrypted and signed body content and attachments.
For doing this it's required to look up the public keys of the receipients email addresses. How the users can store their key (in their profile) is not clear yet. I'll discuss this internally and let you know.

I think that there should be no option to send unencrypted mails once PGP is configured and the receipient supports it. We need to clarify this but until then this should be the way to go.

Admin interface

This should be isolated from the core functionality and will be described later. I think you'll need our support on this. I will discuss it internally and let you know how we approach it.

Conclusion

This should be sufficient by now and point in the right direction. We probably will reach some spots where we have to realign and cover other cases but let them arise first.
Feel free to ask any questions if they arise.

I'm looking forward to see and review your changes 🤓

Thanks for your support 🚀
Thorsten

To be honest I haven't found any gem that I like to have as a dependency. They all look unmaintained, complex or need C compilation of extensions. It might be worth to try if we can't just access the PGP CLI. What do you think?

The PGP CLI is not stable and should, according to GPG, not be used as an API. I've had success with using ruby-gpgme so far, so that's the route I'd like to follow for now. Given that as far as I know GPG does not have an API, but only C-level libraries, I don't think we'll get away without any C extensions.. except for a re-implementation of GPG in ruby maybe, but I don't know of any.

Thanks for making it clear. I wasn't aware of that. Sounds good to me then. It's a bit concerning that their builds are failing but I will have a look at it.

Hey there! As discussed earlier this issue started as a request for the PGP functionality. Later we added S/MIME, too. But since both get implemented independently and are different technologies we decided to move the S/MIME requirement to the new issue #1961. Feel free to subscribe over there if you are interested in any progress. Discussions should be made over at the community board to avoid noise on the issue tracker. Thanks!

Therefore I will provide you with a RSpec helper to ease testing and support you on the way the best I can. I will add it to the develop brach in the next days.

@thorsteneckel has this landed yet and if yes, where exactly is it? :)

@luto - sorry for the delay - here it is: https://github.com/zammad/zammad/commit/f2bf4cbd029bbfdc79888c188d2a29d1ebc5f27d

You just need to place a *filter_filename*_spec.rb in spec/models/channel/filter/ (as done for models/channel/filter/follow_up_merged.rb and add , type: :channel_filter after your filter class name.
After that you have two helpers available:

filter_parsed(mail_string):
https://github.com/zammad/zammad/blob/f2bf4cbd029bbfdc79888c188d2a29d1ebc5f27d/spec/support/channel_filter.rb#L17-L27

and filter(parsed_mail_hash):
https://github.com/zammad/zammad/blob/f2bf4cbd029bbfdc79888c188d2a29d1ebc5f27d/spec/support/channel_filter.rb#L3-L13

The named parameter channel is optional and should not be needed in your case.

Thanks!
FYI: the branch lives in our fork now. feel free to comment on any of the commits as I go. That should keep the final review easy and short for everyone involved :)

Hi @luto - great to see you work on this with such a pace. I noticed that the code doesn't look like it was checked with rubocop. We use rubocop and coffeelint to ensure the Zammad style guide for both kind of files. There is even a pre-commit filter configuration available if you want to perform the checks automatically. Let me know if I can give you more information on that.

Oh, sure thing. I installed the hook with pre-commit install as well as the matching extension for my editor. The indentation was fixed via filter-branch, so the history looks nicer; all other offenses were fixed by hand. The cop is mostly happy now :)

You might notice a with pythonisms in the code. While I try to emulate the ruby way of doing things as best as I can, I am still a python guy by trade ;) Please just point things out, if they seem backwards to you. Rubocop caught quite a few of them :sweat_smile:

Nice! I just noticed that there are at least one type of changes applied that don't match our rubocop configuration: Using unless instead of negative ifs. So it seems that your rubocop is doing to much here.

Oh the pythonisms is just fine for me. Nothing to complain about so far 🤓

Hi @luto - I just wanted to have a look on the current state of the development but noticed that there wasn't any commit since 24 days. Is there anything I can do?

Is there anything I can do?

fixing up our other internal projects :wink: :grin:
There is no blocker within the bounds of Zammad, just a general lack of time. Work on this should continue next week.

@thorsteneckel I kind of hit a snag. Gpg (or at least gpgme) doesn't make a distinction between "email is encrypted, but can't decrypt" and "email is encrypted and all is well" in most cases. That behavior is not compatible with the encrypted? and decryptable? methods outlined earlier. Right now I've got a number of hacks to detect encrypted, but not decryptable mails, but I can't get it to work in all cases.

What do you think about not making a distinction between decryptable and encrypted in the UI? Except for obvious cases like a missing GPG message header, ofc.

When can we expect the implementation and integration in Zammad? Is the implementation planned for a specific release?

Any news?

I have a feeling that those who could bring forward the development of this plugin have other stuff on their mind but checking the replies on this issue.
If you would like to get their attention on this, you could write an email to them: [email protected] - I have already done that - or send them a friendly tweet @ubernauten

@hatscher sorry for the delay; as @0xErnie guessed I've currently got other stuff to do, but this feature is still on our radar. Please note that I am currently working on this solo, not as part of the Zammad, Inc. team and without any external founding. So this can take a while.

As for blockers, there is one question for @thorsteneckel open here, as well as a private one about how we're going to tackle the UI compontent of this. Feel free to contact him, if you'd like to chip in to move this forward, @hatscher! The main blocker is my lack of time, though.

Sorry for the late response, especially to you @luto ! I can't give this the time and attention it needs currently. I plan to address this in 2 weeks from now.

Any news? We need this feature ...

@hatscher many of us need it :) But after all, everybody who's working on the feature (@luto) does it voluntarily :) If there's real commercial need for it, I guess donating (either some dough or work) would help to drive forward the progress. Unfortunately I don't have the time to contribute, so all I'm left with is hope and patience :P

Talking about contributions: How about submitting the current state of development as PR so other could join if they feel like they could contribute?

Given the sad state of GPG-APIs both in ruby and in general the new GPG rust implementation might be worth a look.

With a grain of salt:

BINDINGS COMING SOON!

Are there any news regarding PGP and S/MIME integration? I would also like to donate an amount because I urgently need this feature.

Is there actually a list with the planned features of the upcoming versions?

Hi @hatscher ,
we also need S/MIME and are in talks with Zammad about the costs. Do we want to talk to each other so that we can maybe share the costs for integration? Maybe just send me an email (firstname at lastname de).

E-Mail ist unterwegs ;-)

We're currently searching for other People who are willing to fund this feature (maybe also together with the S/MIME encryption as a package). Please contact me if you're interested at my forename at lastname .de. I will check and see how many willing people are there so we can just share the costs.

@martinseener Great to hear that you tried to secure funding for this feature. Could you share the progress made or the problems you hit? PGP support is currently making-or-breaking a decision regarding a move to Zammad, so any insight would be appreciated.

Sorry, but at this moment we can't share any information on this topic.
We'll update this issue accordingly as soon as something moves for the Zammad-Core.

Looking at your response time of several months and the previous discussions on this issue, I'm concluding that adding PGP support to Zammad is not a priority for you or will not happen at all. (Just summing it up here for anybody who comes along in the future and doesn't want to read the entire discussion.)

Hey @rixx - you're right. We currently working on tasks based on our priority schema:

1st: support / custom development
These are the people funding Zammad and the features we release. Zammad wouldn't be here today without them. Having them really means a lot to us and encourages that we're on the right track.

2nd: 80% features / issues
The spare time we gain from the support of our customers is spend 100% in the product. It's important for us to be equally fair to any user of our user base. However, as soon as it comes to the details there are more and more decisions we need to make in favor of one party over the other. Therefore we only work on features and issues that are affecting 80% of our user base. This way we can ensure that the majority of our user base profits from the changes we introduce.

3rd: Personal interest
Here at Zammad everyone is free to spend time on tasks that are of personal interest. If you take a second look around you'll notice that we respond to issues and questions that don't fall in the category of funded or most relevant changes. This is because we actually care and enjoy helping people out. However, these are mostly smaller tasks because free time is currently very limited - unfortunately.

Now to get back topic: PGP currently doesn't fall in any of the categories listed above. That's why you're right when saying that it's currently no high priority for us. However, you're wrong when you're saying that PGP won't happen at all. PGP has a comparably high priority for us, it's actually on our internal draft for the public roadmap which includes only the most relevant changes.
So the time for PGP hasn't come yet but surely will. Depending on the priority schema - sooner or later.
What I'm missing from your summary is the fact that the PGP development actually has already been started as a community effort by @luto with our support (thanks again @luto !). Unfortunately the priority shifted and the change is currently stale. However, everyone with the knowledge and capability to pick things up and continue working on it will be supported by us for sure!

There's also news for anyone interested in the overall topic of encrypted communication: The great folks around @martinseener at barzahlen.de funded the development of S/MIME communication (#1961) - which soon will be started.

If you're interested in deepen this discussion or even contribute feel free to open a thread or PM me over at the community board and keep this issue on topic.

Any news about "S/MIME communication" ?

If you have to ask, please at least use the right place: https://github.com/zammad/zammad/issues/1961

Sorry, because of missing feedback in the last months I'm limitting this issue to contributors only.
This is to reduce noise and helps us to concentrade on the issues.

If we have any updates on this, we'll update this issue accordingly.

As you've might seen over at #1961 we've added S/MIME support and will release it with the upcoming version 3.4 of Zammad 🎉 Huge thanks to @martinseener over at barzahlen.de for sponsoring it and therefore making it possible 🙌
Unfortunately we still don't have the required resources to add PGP support as well. We were thinking about testing some things out but haven't had the chance yet. That's why I wanted to give you a short update here.
However the S/MIME integration provides a nearly complete "framework" for handling secure mailing and a nice reference implementation on how things have to be integrated/build. There's still the great initial work done by @luto over at the Uberspace fork. If anyone is willing to take a chance on it we're happy to help with all we can. Just open a thread over at community board and mention me.

We keep you updated as new information is available - promised ✌️