The Twitter document leak fiasco started with a simple story that personal accounts of Twitter employees were hacked. Twitter CEO Evan Williams commented on that story, saying that Twitter itself was mostly unaffected. No personal accounts were compromised, and “most of the sensitive information was personal rather than company-related,” he said. The individual behind the attacks, known as Hacker Croll, wasn’t happy with that response. Lots of Twitter corporate information was compromised, and he wanted the world to know about it. So he sent us all of the documents that he obtained, some 310 of them, and the story developed from there.
This post isn’t about the confidential information taken from Twitter. It’s about exactly how Hacker Croll was able to get such deep access to Twitter in the first place.
It’s clear that Twitter was completely unaware of how deeply they were affected as a company - when Williams said that most of the information wasn’t company related he believed it. It wasn’t until later that he realized just how much and what kind of information was taken. It included things like financial projections and executive meeting notes that contained highly confidential information.
We’ve already said a lot about all of this and the related “server password = password” story that was discovered by another individual last week. But we’ve got two more stories to tell. The first, this post, is exactly how the hacks took place, based on information gathered from hours of conversations with Hacker Croll. The second is what was happening behind he scenes with Twitter as the story unfolded. We’ll post that later this week.
When the story first broke the true scope of what had taken place and how it occurred was not understood. Various bloggers speculated about the cause of the attack - with some placing the blame on Google while others blaming the rising trend of hosting documents in the cloud.
We immediately informed Twitter of the information we had in our possession (and forwarded it to them), and at the same time reached out to the attacker. With some convincing, the attacker responsible for the intrusion at Twitter began a dialog with us. I spent days communicating with the attacker in an effort to gain insight into how the attack took place, what the true scope of it was and how we could learn from it.
We’ve waited to post exactly what happened until Twitter had time to close all of these security holes.
Some Background
In the security industry there is a generally accepted philosophy that no system or network is completely secure - a competent attacker with enough time, patience and resources will eventually find a way into a target. Some of the more famous information security breaches have relied on nothing more than elementary issues exploited by an attacker with enough time and patience at hand to see their goal through. A classic example is the case of Gary McKinnon, a self-confessed “bumbling computer nerd” who while usually drunk and high on cannabis would spend days randomly dialing or attempting to login to government servers using default passwords. His efforts led to the compromise of almost 100 servers within a number of government departments. After McKinnon spent a number of years trawling through servers looking for evidence of alien life (long story), somebody within the government finally wised up to his activities which lead to not only the arrest and attempted extradition of McKinnon from the United Kingdom, but a massive re-evaluation of the security methods employed to protect government information.
A more recent example is the case of Kendall Myers, who after being recruited to work for the Cuban government by an anonymous stranger they met while on holiday in that country, set out to obtain a high ranking position within the State Department specifically to obtain access to US government secrets. Kendall dedicated his entire life to obtaining state secrets, and up until he was recently caught by the FBI had successfully passed on secret information and internal documents to the Cuban government for 30 years. He relied only on his memory, his education credentials and sheer dedication.
The Twitter Attack: How The Ecosystem Failed
Like other successful attacks, Hacker Croll used the same combination of patience, sheer determination and somewhat elementary methods to gain access to a frightening number of accounts and services related to Twitter and Twitter employees. The list of services affected either directly, or indirectly, are some of the most popular web applications and services in use today - Gmail, Google Apps, GoDaddy, MobileMe, AT&T, Amazon, Hotmail, Paypal and iTunes . Taken individually, most of these services have reasonable security precautions against intrusion. But there are huge weaknesses when they are looked at together, as an ecosystem. Like dominoes, once one fell (Gmail was the first to go), the others all tumbled as well. The end result was chaos, and raises important questions about how private corporate and personal information is managed and secured in a time when the trend is towards more data, applications and entire user identities being hosted on the web and ‘in the cloud’.
“Hacker Croll” is a Frenchman in his early 20’s. He currently resides in a European country and first discovered his interest in web security over two years ago. Currently in between jobs, he has made use of the additional time he now has, along with his acquired skillset, to break into both corporate and personal accounts across the web. His knowledge of web security has been attained through a combination of materials available to the public and from within a tight-knit group of fellow crackers who exchange details of new, and sometimes unknown, techniques and vulnerabilities. Despite the significance and impact a successful attack has, the cracker claims that his primary motivation is a combination of curiosity, exploration and an interest in web security. There is almost a voyeuristic tendency amongst these individuals, as they revel in the thought of gaining privileged access to information about the inner lives of individuals and corporations. The “high” of access and gaining unauthorized knowledge must be big enough to carry a cracker’s motivation through the long hours, days and months of effort it may take to hit the next pot of gold.
For Hacker Croll, his first port of call in setting out to gain access to a target network is to make use of public search engines and public information to build a profile of a company or individual. In the case of the Twitter attacks, this public information allowed him to create a rich catalog of data that included a list of employee names, their associated email addresses and their roles within the company. Information like birth dates, names of pets and other seemingly innocent pieces of data were also found and logged. This dragnet across the millions of pages on the web picked up both work and personal information on each of the names that were discovered. Public information on the web has no concept of, or ability to, distinguish between the work and personal details of a person’s identity - so from the perspective of a cracker on a research mission, having both the business and personal aspects of a target’s digital life intertwined only serves to provide additional potential entry points.
With his target mapped out, Hacker Croll knew that he likely only needed a single entry point in any one of the business or personal accounts in his list in order to penetrate the network and then spread into other accounts and other parts of the business. This is because the web was designed at a time where there was implicit trust between its participants - requiring no central or formal identification mechanism. In order to keep private data private, modern web applications have built out their own systems and policies that require a user to register and then manage their identities separately with each app. The identifier that most applications use is an email address, and it is this common factor that creates a de facto trust relationship between a user’s applications. The second factor is a password: a random string that only the user knows, is unique to each application, and in theory should take even a computer months or years to figure out if it started guessing. These two elements would work well enough for most cases, were it not for what is often the single weakest factor: human habit.
Look at the front page of almost any web application and you will see hints at just how hopeless and helpless we are in managing our digital lives: “forgot my password”, “forgot my username”, “keep me logged in”, “do not keep me logged in”, “forgot my name”, “who am i?”. Features that were designed and built as a compromise since we are often unable to remember and recall a single four-digit PIN number, let alone a unique password for every application we ever sign up for. Each new service that a user signs up for creates a management overhead that collapses quickly into a common dirty habit of using simple passwords, everywhere. At that point, the security of that user’s entire online identity is only as strong as the weakest application they use - which often is to say, very weak.
Now going back to Hacker Croll and his list of Twitter employees and other information. Twitter just happens to be one of a number of a new breed of companies where almost the entire business exists online. Each of these employees, as part of their work, share data with other employees - be it through a feature of a particular application or simply through email. As these users become interwoven, it adds a whole new attack vector whereby the weak point in the chain is no longer just the weakest application - it is the weakest application used by the weakest user. For an attacker such as Hacker Croll looking to exploit the combination of bad user habit, poorly implemented features and users mixing their personal and business data - his chances of success just got exponentially greater. Companies that are heavily web based rely largely on users being able to manage themselves - the odds are not only stacked against Twitter, they are stacked against most companies adopting this model.
Unfortunately for Twitter, Hacker Croll found such a weak point. An employee who has online habits that are probably no different than those of 98% of other web users. It began with the personal Gmail account of this employee. As with most other web applications, the personal edition of Gmail has a password recovery feature that presents a user with a number of challenges to prove their identity so that their password can be reset. It likely wasn’t the first account from a Twitter employee that Hacker Croll had attempted to access - but in the case of this particular account he discovered a kink in the armor that gave him the big first step. On requesting to recover the password, Gmail informed him that an email had been sent to the user’s secondary email account. In an effort to balance usability with security, Gmail offered a hint as to which account the email to reset the password was being sent to, in case the user required a gentle reminder. In this case the obfuscated pointer to the location of the secondary email account was ******@h******.com. The natural best guess was that the secondary email account was hosted at hotmail.com.
At Hotmail, Hacker Croll again attempted the password recovery procedure - making an educated guess of what the username would be based on what he already knew. This is the point where the chain of trust broke down, as the attacker discovered that the account specified as a secondary for Gmail, and hosted at Hotmail was no longer active. This is due to a policy at Hotmail where old and dormant accounts are removed and recycled. He registered the account, re-requested the password recovery feature at Gmail and within a few moments had access to the personal Gmail account of a Twitter employee. The first domino had fallen.
Well designed web applications will never just give a user their password if they forget it, they will force the user to pick a new one. Hacker Croll had access to the account, but with a password he had specified. To not alert the account owner that their account had been compromised, he had to somehow find out what the old Gmail password was and to set it back. He now had a bevy of information at his fingertips, a complete mailbox and control of an email account. It wasn’t long before he found an email that would have looked something like this:
To: Lazy User
From: Super Duper Web Service
Subject: Thank you for signing up to Super Duper Web Service
Dear Lazy User,
Thank you for signing up to Super Duper Web Service. For the benefit of our support department (and anybody else who is reading this), please find your account information below:
username: LazyUser
password: funsticks
To reset your password please follow the link to.. ahh forget it, nobody does this anyway.
Regards,
Super Duper Web Service
Bad human habit #1: Using the same passwords everywhere. We are all guilty of it. Search your own inbox for a password of your own. Hacker Croll reset the password of the Gmail account to the password he found associated with some random web service the user had subscribed to and that sent a confirmation with the password in clear text (and he found the same password more than once). He then waited, to check that the user was still able to access their account. Not too long later there was obvious activity in the email account from the account owner - incoming email read, replies sent and new messages drafted. The account owner never would have noticed that a complete stranger was lurking in the background. The second domino falls.
From here it was easy.
Hacker Croll now sifts through the new set of information he has access to - using the emails from this user’s personal Gmail account to further fill in his information map of his target. He extends his access out to all the other services he finds that this user has signed up for. In some instances, the password is again the same - that led Croll into this user’s work email account, hosted on Google Apps for Domains. It turns out that this employee (and in fact most/all Twitter employees and everyone else) used the same password for their Google Apps email (the Twitter email account) as he did with his personal Gmail account. With other sites, where the original password may not work - he takes advantage of a feature many sites have implemented to help users recover passwords: the notorious “secret question”.
Fork the story here for a moment because there is a real issue here with the “secret question” (from here on abbreviated more appropriately as just “secret ?”). For some strange reason, some sites refer to the “secret ?” as an additional layer of security - when it is often the complete opposite. In the story of Hacker Croll and Twitter, the internal documents that we now all know about were only a few steps away from the first account he gained access to. In addition to that, this attacker, and certainly others just like him, have been able to demonstrate that some of the biggest and most popular applications on the web contain fundamental weaknesses that alone might seem harmless, but in combination with other factors can cause an attacker to completely tear through the accounts of users, even those who maintain good password policy.
This is not the first time that the issue of “secret ?” being used in password recovery systems has been raised. Last September, US Republican Vice Presidential candidate and former governor of Alaska, Sarah Palin, had screenshots of her personal Yahoo mail account published to Wikileaks. A hacker or group known only as ‘Anonymous’ claimed credit for the hack, which was carried out by the attacker making an educated guess in response to the security question used to recover passwords. In early 2005, celebrity Paris Hilton suffered a similar incident when her T-Mobile sidekick account was broken into, and the details of her call log, messages (some with private pictures of Hilton) and contact list were leaked to the media. The culprit, again, was “secret ?”.
Giving the user an option to guess the name of a pet in lieu of actually knowing a password is just dramatically shortening the odds for the attacker. The service is essentially telling the attacker: “we understand that guessing passwords is hard, so let us help you narrow it down from potentially millions of combinations to around a dozen, or even better, if you know how to Google, just one”. The problem is not the concept of having an additional authorization token, such as mothers maiden name, that can be used to authenticate in addition to a password, the problem arises when it is relied on alone, when the answer is stored in the clear in account settings, and when users end up using the same question and answer combination on all of their accounts.
From this point, with a single personal account as a starting point, the intrusion spread like a virus - infecting a number of accounts on a number of different services both inside and outside of Twitter. Once Hacker Croll had access to the employee’s Twitter email account hosted by Google, he was able to download attachments to email that included lots of sensitive information, including more passwords and usernames. He quickly took over the accounts of at least three senior execs, including Evan Williams and Biz Stone. Perusing their email attachments led to lots more sensitive data being downloaded.
He then spidered out and accessed AT&T for phone logs, Amazon for purchasing history, MobileMe for more personal emails and iTunes for full credit card information (iTunes has a security hole that shows credit card information in clear text - we’ve notified Apple but have not heard back, so we won’t publish the still-open exploit now).
Basically, when he was done, Hacker Croll had enough personal and work information on key Twitter executives to make their lives a living hell.
Just to summarize the attack:
1. HC accessed Gmail for a Twitter employee by using the password recovery feature that sends a reset link to a secondary email. In this case the secondary email was an expired Hotmail account, he simply registered it, clicked the link and reset the password. Gmail was then owned.
2. HC then read emails to guess what the original Gmail password was successfully and reset the password so the Twitter employee would not notice the account had changed.
3. HC then used the same password to access the employee’s Twitter email on Google Apps for your domain, getting access to a gold mine of sensitive company information from emails and, particularly, email attachments.
4. HC then used this information along with additional password guesses and resets to take control of other Twitter employee personal and work emails.
5. HC then used the same username/password combinations and password reset features to access AT&T, MobileMe, Amazon and iTunes, among other services. A security hole in iTunes gave HC access to full credit card information in clear text. HC now also had control of Twitter’s domain names at GoDaddy.
6. Even at this point, Twitter had absolutely no idea they had been compromised.
What could have happened next is that Hacker Croll could have used or sold this information for profit. He didn’t do that, and says he never intended to. All he wanted to do, he says, was to highlight the weaknesses in Twitter’s data security policies and get them and other startups to consider more robust security measures.
He also says he’s sorry for causing Twitter so much trouble. We asked Hacker Croll if he had any message he wants to deliver to Twitter, and he sent me the following:
Je tiens à présenter toutes mes excuses au personnel de Twitter. Je trouve que cette société a beaucoup d’avenir devant elle.
J’ai fait cela dans un but non lucratif. La sécurité est un domaine qui me passionne depuis de longues années et je voudrais en faire mon métier. Dans mon quotidien, il m’arrive d’aider des gens à se prémunir contre les dangers de l’internet. Je leur apprend les règles de base.. Par exemple : Faire attention où on clique, les fichiers que l’on télécharge et ce que l’on tape au clavier. S’assurer que l’ordinateur est équipé d’une protection efficace contre les virus, attaques extérieures, spam, phishing… Mettre à jour le système d’exploitation, les logiciels fréquemment utilisés… Penser à utiliser des mots de passe sans aucune similitude entre eux. Penser à les changer régulièrement… Ne jamais stocker d’informations confidentielles sur l’ordinateur…
J’espère que mes interventions répétées auront permis de montrer à quel point il peut être facile à une personne mal intentionnée d’accéder à des informations sensibles sans trop de connaissances.
Hacker Croll.
This roughly translates to:
I would like to offer my personal apology to Twitter. I think this company has a great future ahead of it.
I did not do this to profit from the information. Security is an area that fascinated me for many years and I want to do my job. In my everyday life, I help people to guard against the dangers of the Internet. I learned the basic rules .. For example: Be careful where you click the files that you download and what you type on the keyboard. Ensure that the computer is equipped with effective protection against viruses, external attacks, spam, phishing … Upgrading the operating system, software commonly used … Remember to use passwords without any similarity between them. Remember to change them regularly … Never store confidential information on the computer …
I hope that my intervention will be repeated to show how easy it can be for a malicious person to gain access to sensitive information without too much knowledge.
Croll hacker.
What’s the takeaway from all this? Cloud services are convenient and cheap, and can help a company grow more quickly. But security infrastructure is still nascent. And while any single service can be fairly secure, the important thing is that the ecosystem most certainly is not. Combine the fact that so much personal information about individuals is so easily findable on the web with the reality that most people have merged their work and personal identities and you’ve got the seed of a problem. A single Gmail account falls, and soon the security integrity of an entire startup crumbles. So for a start, reset those passwords and don’t use the same passwords for different services. Don’t use password recovery questions that can easily be answered with a simple web search (an easy solution is to answer those questions falsely). And just in general be paranoid about data security. You may be happy you were.
by Nik Cubrilovic on July 19, 2009
