Gitea: Giteabot account was compromised

In the meantime please be careful when downloading our released binaries, especially the windows ones, until further notice. E.g. keep an eye on the size of the binaries, or a double .exe file ending.
We've found out our .exe binaries within release 1.4.2 were altered as well.

We work hard to follow all trails, clean up and get back to daily business as soon as possible.

@daviian Maybe then it is necessary to sign releases?

@Mauladen Thank you for your input. We definitely will discuss this.

Ouch! Yeah, signed binaries would be ideal.

@Mauladen We've rebuilt the binaries for 1.4.2 release just to be sure we provide safe ones.

Or did you mean something else?

This is normal. We retaged 1.4.2 to retrigger CI to rebuild binaries as windows binary for that release was removed and there was added new one malicious one

@daviian, Maybe @Mauladen meant that 1.4.2 release commit without GPG signature, but 1.4.1 was

Still need to edit the list of changes

@axifive commit where not gpg signed by user but github that does it automaticaly (with github key) when the merger is the same the PR author. I wouldn't it consider more safer because if github account were compromised I will been also show as "verified". But we could start to sign tag from now (to be discuss). For information we are, working on signing binary since it is more easy to corrumpt thzan binary as the git commit tree.

You should upload a tarball created by git-archive (in addition to those that github automatically generates) which you sign with signify. You can get an idea on how that works/looks here:

• https://github.com/nicklan/pnmixer/blob/master/MAINTAINING.md#releasing
• https://github.com/nicklan/pnmixer/releases/tag/v0.7.2

Libressl uses the same technology.

Is there anymore information on this? What was the payload of the binary? Are users going to be informed about this through a blog post or anything? Were any of the linux binaries compromised? I'm rather concerned about what these things may have done to my servers. I'm doubly concerned that I only found out about this browsing the issue page by chance vs an official notice on the project page/blog

Is the Gitea 1.4.2 release safe to update to from source code at this point in time?

At this point, it's not clear to me whether only binaries were affected. How did you verify this? Are your branches protected?

shasums differ on binaries I downloaded on 6/4 and 6/7 though they are the same number of bytes:

$ sha256sum gitea-1.4.2-linux-amd64*
e14e54f334ce022d2026d0801da1ed1bf3bcba751b16fb290bae4d10d14482e9  gitea-1.4.2-linux-amd64
c843d462b5edb0d64572b148a0e814e41d069d196c3b3ee491449225e1c39d7d  gitea-1.4.2-linux-amd64.1

$ ls --color=tty -l gitea-1.4.2-linux-amd64*
-rwxr-xr-x 1 matt matt 53674800 Jun  4 20:39 gitea-1.4.2-linux-amd64
-rwxr-xr-x 1 matt matt 53674800 Jun  7 08:18 gitea-1.4.2-linux-amd64.1

Feel like hosting those somewhere so folks can tear them apart?

Uploaded the binaries here: https://send.firefox.com/download/a5dc9a9335/#_k9e4Cu2r4pPNDrsS1T2CA. I'll put them somewhere more permanent if necessary.

Question - is it just the binaries on the website that were affected, or were the repositories also affected? I ask because yesterday I heard about Gitea and cloned and built the v1.4 branch.

Would really like some more info if the source itself is compromised or just the binaries. I run a script to check for updates/build from git nightly and was lucky enough to miss this because of the timing.

Is the current 1.4.2 release verified to be clean? If we know that's tainted we should pull that ASAP and put it somewhere else for analysis, otherwise people will still be downloading that if they don't see this issue. I agree with @CountMurphy, can we add something to the README so it's really visible?

Can we get hashes so we can check if our binaries are affected? My gitea 1.4.1 (linux amd64) looks like this:
$ sha256sum gitea
d8cfa0d39da70497f1f75e519e4fee33e5ee7c0de88919bdfe46a8b0d38af851 gitea

I just downloaded gitea 1.4.1 linux-amd-64 and I got d8cfa0d39da70497f1f75e519e4fee33e5ee7c0de88919bdfe46a8b0d38af851 as the sha256 too

@mikedilger See: https://github.com/go-gitea/gitea/issues/4167#issuecomment-395576229

To make it overwhelmingly clear: nobody knows anything and the only safe hashes are the ones currently found here: https://github.com/go-gitea/gitea/releases

For Gitea 1.4.2 on amd64, that means:

Sorry, I misunderstood https://github.com/go-gitea/gitea/issues/4167#issuecomment-395576229 to mean that both binaries were compromised.

I've edited this post to remove any ambiguities about what we actually know.

Hold on: according to this comment (https://github.com/go-gitea/gitea/issues/4167#issuecomment-395576229), there are two shas e14e54f334ce022d2026d0801da1ed1bf3bcba751b16fb290bae4d10d14482e9 from June 4th and c843d462b5edb0d64572b148a0e814e41d069d196c3b3ee491449225e1c39d7d from June 7th.

Are we saying that both of these are compromised or just one of them? For the record the one on my server is e14e54f334ce022d2026d0801da1ed1bf3bcba751b16fb290bae4d10d14482e9 from June 5th.

@christianbundy, Please, don't make hasty conclusions, wait for a response from the Team member

Please, don't make hasty conclusions, wait for a response from the Team member

Sorry about that, I thought the file hashes would be posted by a team member immediately and didn't realize that the info was coming from someone else who was _also_ in the dark. Is this the correct channel to watch for the latest developments? I've powered off my server and need more information to tell whether I've been infected.

I see your edits crossing out the entry I pointed to. HOWEVER, isn't this too early to draw conclusion from? How do we know for sure e14e54f334ce022d2026d0801da1ed1bf3bcba751b16fb290bae4d10d14482e9 is fine?

We're still missing a couple pieces of information (likely non-exhaustive):

• [ ] Which binaries were compromised and what are their checksums?
• [ ] When did the bot account get compromised?
• [ ] Is the source repository compromised (this might be easy to check if you have a version of the repo cloned from some while back, then you can maybe check diffs or evidence of force pushes).

I got:

# ls -la gitea-1.4.1-linux-amd64
-rwxr-xr-x 1 gitea gitea 53663640 May  3 08:02 gitea-1.4.1-linux-amd64

With sha256sum d8cfa0d39da70497f1f75e519e4fee33e5ee7c0de88919bdfe46a8b0d38af851

and I just downloaded gitea-1.4.2-linux-amd64:

# ls -la gitea-1.4.2-linux-amd64
-rwxr-xr-x 1 root root 53674800 Jun  7 14:18 gitea-1.4.2-linux-amd64

With sha256sum c843d462b5edb0d64572b148a0e814e41d069d196c3b3ee491449225e1c39d7d which matches the provided .sha256 file: gitea-1.4.2-linux-amd64: OK which should be safe. (right?)

[...] We've rebuilt the binaries for 1.4.2 release just to be sure we provide safe ones. [...]

I uploaded gitea-1.4.2-linux-amd64 for analysis here, but it doesn't really tell anything obvious.

Going to watch this issue and keep my gitea installation offline for the time being.

How do we know for sure e14e54f334ce022d2026d0801da1ed1bf3bcba751b16fb290bae4d10d14482e9 is fine?

@shuhaowu I said that c843d462 was fine, not e14e54f3. We know this because that's the file currently being served on GitHub, which they have recently rebuilt.

If 1.4.2 was rebuilt, that should be signed and verified like the other valid releases. Not doing so only adds to the confusion.

@xcolour

Uploaded the binaries here: https://send.firefox.com/download/a5dc9a9335/#_k9e4Cu2r4pPNDrsS1T2CA

The only difference between these two binaries is that about 2000 instances of movabsq $63663754793, %rcx were replaced with movabsq $63663969431, %rcx. I can't say exactly what that changes behaviorally, but I think it's very unlikely that it's enough to be malicious.

Maybe push out a new (signed?) 1.4.3 release with the known-safe code, or would that confuse things more?

@justinclift I like that idea, also a blog post and something in the README about the situation would help clear things up.

@spaghetti2514

On one hand, you're right -- the differences between the _those_ two binaries is easy to see.

On the other hand, the Gitea team hasn't actually posted which binaries were affected, posted any SHA256 hashes, or really said _anything_ that would help us understand the compromise As others have pointed out, we don't have nearly enough info to determine what happened and who's affected.

I'm frustrated to point out that we should probably just assume the worst and wait for diagnostic steps from the core team. With that said, I appreciate you posting the disassembled diff.

Is this what you're looking for? https://github.com/go-gitea/gitea/issues/4167#issuecomment-395579466

Thank you for the update @lafriks !

I have updated issue description with additional info. It was not so bad as it could have been. We wanted to be as transparent as possible so created this issue as soon as we did clean up & fixed everything and as this was first time something like this have happened, so probably we should have given more information faster, sorry for causing confusion.

@lafriks Thanks for the update! Could you post your PGP key that we can use going forward to verify commits?

@4oo4 tags are signed by one of mergers own GPG key, as PR's all merged using Squash&Merge, they are not signed (or maybe signed by GitHub's magic :) )

Releases will be signed by GPG key that will be created before 1.5.0 release, we will publish it on README.md, gitea docs and in blog post.

As a 386 user, I can confirm that the gitea-1.4.1-linux-386 binary currently available on the Releases page matches a version I downloaded on June 4th (cf6344b4).

as PR's all merged using Squash&Merge, they are not signed (or maybe signed by GitHub's magic :) )

Don't use the merge button, that's basically insecure and a random gpg key. Merge locally and push the merge.

@mappu gitea v1.4.1 has not been affected and now v1.4.2 has been rereleased.

You should upload a tarball created by git-archive (in addition to those that github automatically generates) which you sign with signify .

Actually, you do not even have to build the archives and upload it again. You can sign the one GitHub created, because it can be done reproducibly.

Here is a small script for that: https://github.com/rugk/gittools/blob/master/signrelease.sh

@rugk Useful looking script. :smile:

Has any contributing team member got an archive of 1.4.2 prior to breach discovery?

You seem to have left a lot of people in the dark on whether or not this binary has been altered for any Linux releases, which is quite an important piece of information considering:

1) Most deployments will be to Linux stacks

2) Linux stacks are not accustomed to binary trojans and aren't typically suited with the best tools for programmatically detecting suspicious code already running on the system.

While I'd like to believe that this was a simple Windows-targetted attack and nothing more, the fact that they targeted this specific repository just days into a dramatic spike in userbase growth seems indicative to a more well orchestrated effort. I highly doubt that anyone who knew what they were targeting and how to pull off this sort of thing would have just passed up such a ripe opportunity to infect as many hosts as possible.

@stanier dl.gitea.io was not affected and we have verified that no other binaries were tampered with

So that explains why our webproxy refused to download the latest gitea image from hub.docker.com ...

@lafriks So can you confirm that c843d462 and e14e54f3 are both safe? If the CI provided a build with a different signature, then something must have changed in either the source code or the parameters the compiler used for the build. It's a minor technicality by itself, but it was never directly stated here whether or not the adversary had altered the 1.4.2 Linux binaries, only the respective .exe binaries mentioned in the comment by @daviian.

Just to clarify, I'm asking about the GH repo's release page binaries, not dl.gitea.io, I understand nothing was tampered with outside of the GH repo.

Linux stacks are not accustomed to binary trojans and aren't typically suited with the best tools for programmatically detecting suspicious code already running on the system.

Hmmm, don't most package managers (and similar) have provision for verifying at least sha* checksums? Not necessarily just for detecting malware, but as a "lets verify the download wasn't corrupt".

@justinclift yes, but that only applies to binaries installed through the package manager. The issue in question here concerns with a direct download from GitHub, which doesn't have any embedded malware detection to the best of my knowledge.

Ahhh. The term "Linux stacks" threw me off, as I'm more used to direct downloads being simplistic one-off things (eg when prototyping), rather than something an automated solution would do. No worries. :smile:

@stanier the build was redone by drone to clear all. Go doesn't provide same binary each time you build the binary since some variable could change (like build time) so it is normal that the hash differs.
I got all binary during investigation and can provide hashs as soon as I get to my personal computer.

For all, please stop argument on rumors and provide constructive input.
I understand that you worry for your security and we care greatly about it (since we are also at your place as user of gitea). Currently the access were changed and we haven't seen any suspicious activity anymore. We have done this issue to inform as soon as possible to be transparent and be able to receive input from any data usefull to investigate or missed spot as no one is perfect.
We will do a post-mortem to explain what happen and we have definitively think of action to prevent this in the futur.
If this issue go wild, I think we will need to close to keep only usefull and concise information and send the debate to discord where it should go.

To avoid a situation like that in the future we will start to sign all binaries with the next version. I have built a plugin for Drone which you can find at https://github.com/drone-plugins/drone-gpgsign (the documentation for this plugin have to be done) that will sign all binaries, the public key will be uploaded to the download server and to a keyserver, than you will always be able to verify the binaries in a trustable way.

Just to show you an example how this will look and what are the results you can take a look at https://github.dronehippie.de/webhippie/ldap-proxy/49/18 and https://dl.webhippie.de/misc/ldap-proxy/master/, similar files will be uploaded to the Gitea download page and to the GitHub releases.

If you think this plugin is missing something really important, feel free to open an issue on the plugin repository and I can address it.

@sapk Thanks for the clarification. I apologize, it completely slipped my mind that the project was Golang-based-- most issues like this deal with older software so I think my mind jumped to a false relation. My mistake.

I've started to take a look at the install.exe binary that was uploaded: https://grh.am/2018/a-look-at-the-compromised-gitea-release/

It seems it wasn't just Gitea that got hit by this, but also https://github.com/opencompany/www.opencompany.org/releases

Thank you for the clear explanation.

I think this issue is the main reason to go through distro-specific packaging. It brings more attention to the packaging and potential malicious code/binary (especially in case of such a gross attempt). It would be great to have at least Debian/RedHat/Archlinux packages for Gitea : that would prevent many people from running an arbitrary binary on their production server :)

Is PGP-signing the releases enough? Probably. Just make sure to advertise your public key on many different platforms so that compromising for instance your website does not make everybody download wrong keys. (But a Debian package in the backports would be <3)

(Also, reproducible builds ?)

Since you will start signing binary releases, could you also consider signing the Docker images pushed to the registry?

It seems it wasn't just Gitea that got hit by this, but also https://github.com/opencompany/www.opencompany.org/releases

Just emailed their contact people directly, in case they're not looking at their GitHub issues yet.

@graystevens Should the pastebin staff be contacted to kill that pastebin address, or is it better to fully analyse the malware first so it's properly understood?

Thanks everyone.. Would re download and put my server back up

@justinclift I’ll report the paste to PasteBin now - I’ve got a copy of the contents so we can recreate the output for the malware should we need to. Good shout

To be fair, go is now has reproducible build : https://blog.filippo.io/reproducing-go-binaries-byte-by-byte/
That maybe cgo for sqlite and some build env that make them not reproducible.

Just for information, the previous rebuild and un-touched hash list. If you have those hash it means you have the binary before the rebuild we have done to reset the 1.4.2 release and also before the attack. If it is the case, you can stay with them.

156655506d373241fea6b8955acbe6fdc148f06b49b4f6e46d11cadcd00d7316  gitea-1.4.2-darwin-10.6-386
5730c1a471dfc2c055442360d431c950b3a4f266403c5f327c94a68b88e67a10  gitea-1.4.2-darwin-10.6-amd64
8c3cc2127161695dea512176cd4282711e8c57972f333253cc89597dfab002ef  gitea-1.4.2-linux-386
e14e54f334ce022d2026d0801da1ed1bf3bcba751b16fb290bae4d10d14482e9  gitea-1.4.2-linux-amd64
bca75d81c52b9aa57fd05b8f7ce0572abae3e1a008d8ab77840ca08c53975867  gitea-1.4.2-linux-arm-5
3aa791afce2e3450461038e09622fe04f34b891c47db641be50b6cc3f772645e  gitea-1.4.2-linux-arm64
fc73d06621fc23a944a2a8ee3b19fbd8edb972b0cdaefa67248eb885aa4d6240  gitea-1.4.2-linux-arm-6
5b76cd9b84846c09ba3627219a6cad99542a53d8eec71a427a433ab82eaabacf  gitea-1.4.2-linux-arm-7
4fafeabfa069066fd624364b47b5729f8f068545d4cd8a3620774a982937ac16  gitea-1.4.2-linux-mips64le
7d9e0afcd315f0efbfb26cab303b3fee3939fa0e081b0d3f4f480f950d0a9870  gitea-1.4.2-linux-mips64
7f2e3c1163823889bec1fdd34b4135149c83d53b6dc86f186821e51e0f839e53  gitea-1.4.2-linux-mipsle
a1b8338113d4fdb0360582ba18f6fce97722c779493e7d7e07594d78c7c3f003  gitea-1.4.2-linux-mips
82ad50932bd927ead2e7da4e4c575d94f88e0105a82eda4cce1df7c9fc5ba0dc  gitea-1.4.2-windows-4.0-386.exe
86d7332894390ccaedd39be891044d829f54d4cd71fa21fce5dbd9bc3abbce44  gitea-1.4.2-windows-4.0-amd64.exe

The install.exe cleanup pass seems to have missed quite a few repositories:

• go-gitea/gitea-darkmode
• go-gitea/lgtm-cli
• go-gitea/yaml
• go-gitea/bolt
• go-gitea/lgmt-go
• go-gitea/lgmt-docs

Most of these are old and not exactly relevant, but it's probably not a good idea to keep malware around.

@lunny

• https://github.com/lunny/html2md/releases
• https://github.com/lunny/tango/releases
• https://github.com/lunny/nodb/releases

Ugh, thanks. What's the approach you're using to locate these? It seems that whatever approach the GitHub staff have used hasn't really been 100% effective. :frowning:

@jvanraaij @yaggytter thanks.

As after our investigation nothing else was affected and no more information was given by GitHub I think this issue can be closed. Anyone to write blog post about this?

@lafriks I posted this blog post back within a couple of days of this all kicking off - its more of a look at the malware than the issue at hand, but I'm happy to update it if you feel something is worth documenting 👍

I think he wants to write a post-mortem blog post on the gitea blog :)

@graystevens BTW your site notice link is a 404 one. :smile:

As after our investigation nothing else was affected and no more information was given by GitHub ...

As a data point, although GitHub haven't said anything about this in public, privately they've responded (via email) to say effectively "Thanks, we're investigating."

The above comments by @jvanraaij and @yasuokav seemed to help too, as (weirdly, from my PoV) GitHub don't seem to have found those particular instances of the malware prior to that.

For example, all of the repo's listed by @yasuokav here still have the malware: https://github.com/crossoverJie/SSM/issues/36

I'll email GitHub staff again to point it out. They seem to get things taken care of within a few days when contacted directly with an exact list to look at.

This is sad for all the other projects, but at least for Gitea we have solved the issues properly, hopefully... :)

Closing this issue as binaries are now signed, and 2FA is implemented.