Linux Foundation infrastructure including LinuxFoundation.org, Linux.com, and their subdomains are down for maintenance due to a security breach that was discovered on September 8, 2011. The Linux Foundation made this decision in the interest of extreme caution and security best practices. We believe this breach was connected to the intrusion on kernel.org.

We are in the process of restoring services in a secure manner as quickly as possible. As with any intrusion and as a matter of caution, you should consider the passwords and SSH keys that you have used on these sites compromised. If you have reused these passwords on other sites, please change them immediately. We are currently auditing all systems and will update this statement when we have more information.

We apologize for the inconvenience. We are taking this matter seriously and appreciate your patience. The Linux Foundation infrastructure houses a variety of services and programs including Linux.com, Open Printing, Linux Mark, Linux Foundation events and others, but does not include the Linux kernel or its code repositories.

Please contact us at info@linuxfoundation.org with questions about this matter.

The Linux Foundation

According to an unattributed post on the front page of Kernel.org, intruders managed to gain access via a compromised user credential. It's currently unknown how the attacker managed to escalate to root access.

After gaining access, the attacker modified files related to SSH services and added a trojan startup file to the system startup scripts. The trojan was discovered due to an error showing in a system log from a program not actually installed on the server (Xnest).

The status now is that the compromised systems are offline and being restored from pristine backups. All boxes on kernel.org will be getting full re-installs, and analysis is being done of the code to make sure that nothing has been modified. Authorities have also been notified about the breach.

Why it Doesn't Matter Much

Before anyone gets in a tizzy about the compromise, it's worth pointing out that while this is an enormous inconvenience for the kernel folks and site admins it's not going to affect enterprises that run Linux in production.

Even if the attacker managed to compromise the code repository, almost all production servers are running kernels provided by vendors like Red Hat and SUSE. Those kernels were patched, compiled and tested long before this breach. The only way someone might see this is if they're testing the most current kernels or compiling their own. And, again, only if the code was actually compromised – which is considered unlikely at the moment.

There's also the small matter of the source control system used to manage the kernel source. As Jon Corbet writes, "The code for the kernel (and for many other projects) is managed with the "git" source code management system. And git does not allow the code to be modified by third parties without people knowing about it."

Files managed by git have a cryptographic hash associated with them. Every time the file changes, the hash changes. When developers download the files, they'd get a warning from their instance of git that something had been changed.

As Corbet points out in his post, kernel.org may seem like where kernel development is done – but it's not. It's the centralized repository for all the developers who are doing kernel development on their own machines.

Why it Does Matter

Why is it worth reporting? Obviously, the fact that the site hosting the Linux kernel is going to be considered news. But really, any major breach is worth examining since it shows how attackers work and how they might be trying to compromise your systems. Kernel.org has pretty good security, but it just goes to show that a target that has sufficiently motivated attackers may be compromised.

It's not yet known how the attackers managed to gain root access. Once that's known, we'll be sure to report the issue so companies that might be at risk can update immediately.

Disclaimer: Kernel.org is funded by the Linux Foundation, and I do contract work for the Linux Foundation though I am not connected to Kernel.org management in any way.