Showing posts with label hacking. Show all posts
Showing posts with label hacking. Show all posts

Tuesday, March 7, 2017

Wikileaks releases massive trove of CIA documents

Today Wikileaks released a massive new trove of leaks focused on the CIA's IT-based espionage capabilities. Wikileaks has named the document release Vault 7. The trove has just been released this morning, so details remain sketchy, however the included documents appear to contain detailed information related to dozens of malware tools used by the CIA's Center for Cyber Intelligence.

Earlier this morning I heard an NPR report claiming that Wikileaks was redacting the source code associated with these hacking tools. I'm not sure if that is correct; I've found a few files with executable scripts included, but none of the scripts I've found so far are essentially malicious (although they were almost certainly used in the development and packaging of malware). I have found indications that Wikileaks redacted exploit files that were ready for as-is distribution. For example, the files I reviewed in the dump appear to be part of an internal wiki. I reviewed a file list associated with one of the users registered for the wiki (; clicking through the link for a file named '~02.2.3.tmp` - - provided  me with this:

File: ~02.2.3.tmp
MIME: application/x-dosexec; charset=binary
Size: 389632

I have taken significant issue with Wikileaks in the past. My complaints have focused entirely on Wikileaks' unwillingness to remove dangerous (and almost certainly state-sponsored) malicious software from document dumps. The example I cited above is the first time I have ever seen any indication that Wikileaks removed malware from a dump. Unfortunately, this particular editorial decision is of substantially less value then the requests I repeatedly made to Wikileaks to inform their users of the presence of infected files within and older document dump that they continue to publish through the website. The censored malware files in Vault7 were contextually and obviously labelled as malware. The malware I found in earlier Wikileaks dumps included infected document files that were in many cases completely indistinguishable from normal document files and in several cases not detectable for a substantial variety of antivirus platforms.

If you are a journalist or concerned citizen preparing to begin reviewing the Vault 7 document dump, I strongly advise you to take strong security measures prior to beginning your review:

    1. Assume every file in the dump contains a malicious file & govern yourself accordingly. The principle here is similar to the sort of "universal precautions" utilized by medical professionals. This includes files that you may not think of as having the ability to infect your computer with malware, such as text documents, images, spreadsheets and PDFs.

    2. Download & inspect the documents using a computer dedicated to the task. An operating system designed for secure analysis of malware should be used, such as Kali Linux or TAILS. There is compelling evidence that Microsoft provides state-sponsored attackers with backdoors to the Windows OS. After downloading the files, completely disable the internet connectivity for your review computer by disabling (or even disconnecting) any network interfaces.

The inspection of malware is a complex topic that can't be covered in a single post, however the consequences of insecure handling of documents infected with state-sponsored malware are serious - while the advantages of safe handling are substantial. Would you feel comfortable providing a list of your sources to a random government intelligence service? Every reporter I have discussed the issue with feels a strong sense of responsibility for protecting their sources, up to and including a willingness to face incarceration. Securing your IT tools is not as dramatic as saying "No" to a judge threatening you with contempt, but for many sources the threat posed by an intelligence service dwarfs that of a court. Arrest is bad; being "disappeared" is worse.

The average reporter would not defend herself from a finding of contempt of court  - newsrooms invest substantially in legal resources under the calculation that protecting the sources and first amendment rights of journalists serves the both the bottom line & cultural interests of newspapers. Likewise, newsrooms must now consider the expense of an on-staff or consulting systems administrator with a background in security as a cost of doing business. Its not a happy thought, but this is the world we now live in: a world where every communication is spied on, documented, indexed and stored, secretly; and it has been for many years.

So thats the stick. What about the carrot? Malicious software contained within the files is as much a part of the story as the files themselves. Sourcecode comments and filesystem metadata can provide important clues related to the authors of, history behind and justification for distributing data. A thorough investigation of leak files can be the sole opportunity to reveal the true story behind a leak; the alternative, in the absence of communication with the true source of the leak, is to print a summary of a Wikileaks press release supplemented by a Government press release.

Tuesday, November 8, 2016

A nasty pair of MySQL exploits grant attackers system root from any database user

Four days ago I received an email from Dawid Golunski through the list illustrating one of the more brutal pair of security vulnerabilities I have seen recently. Here's how it works.
    The exploit uses a vulnerability within MariaDB, PerconaDB (and/or XtraDB Cluster) and MySQL to, first, gain access to the 'mysql' system user using any mysql user that has CREATE / INSERT / UPDATE permissions. The first part revolves around a race condition when sql generates temporary files as part of the `REPAIR table` command. Then using the mysql system user the second vulnerability grants the attacker root access to the server using a clever hack that takes advantage of mysql_safe's approach to writing to file based error logs. Below I've provided a list of vulnerable server versions. Just about any server using the more recent (unpatched) stable releases of MySQL or MariaDB through CentOS is vulnerable (Percona isn't part of the standard CentOS repositories), with a few of caveats.
    The first caveat is that an unpatched vulnerable server can prevent at least the 2nd exploit by disabling symlinks through /etc/my.cnf using skip-symbolic-links or symbolic-links=0
    The next caveat is that the 2nd exploit also depends on using file-based mysql logging. Using syslog will avoid trouble.
    The third caveat is that for the 1st exploit to work an attacker needs a mysql user and password.
    There is some good news here. The latest stable versions of MariaDB at least disable symbolic links in my.cnf by default (its been a while since I installed MySQL through the repo but I'm fairly sure its disabled here as well). And how would an attacker get a MySQL user anyway?
    Consider that because *any* MySQL user to be used, an un-patched shared server used by a hosting company would depend on the security competency of every one of that c customers to securely handle database authentication. Not only are there a variety of exploits available for obtaining a standard database user, but its depressingly common for web designers to place their connection strings with un-encrypted database username and password into world-readable files. There are a variety of feeds and sites that scan the internet for and compile such files.
    And even without the use of the 2nd exploit, an attacker can still do an enormous amount of damage without server root with only the mysql system user. The attacker will have full access to the MySQL system files. An attacker could easily delete an entire database instance, for example.
    Of course the best part is that this is a vulnerability in MySQL itself. Upgrading MySQL is the scariest, riskiest upgrade there is among standard repo software. A lot of admins compile it from source or install it from a direct RPM (in which cases symlinks are enabled by default). And applications are closely linked with the database version. Even successful upgrades can easily break applications that run on that database as calls used by the application become deprecated. Upgrading applications has substantial costs, whether you develop the application itself or license it. A patch was already in circulation before these exploits were posted, but for all of the reasons listed above, vulnerable databases will be active for years.

Here are the impacted DB versions:

 < 5.5.52
 < 10.1.18
        < 10.0.28

 <= 5.5.51
 <= 5.6.32
 <= 5.7.14

Percona Server
 < 5.5.51-38.2
 < 5.6.32-78-1
 < 5.7.14-8

Percona XtraDB Cluster
 < 5.6.32-25.17
 < 5.7.14-26.17
 < 5.5.41-37.0

Here the first two links below contain a comprehensive breakdown of both exploits with example scripts that you can run to test.

This link includes a video illustrating how a compromise takes place using the example scripts:

Sunday, August 14, 2016

Pandora account compromise warning message

Here is a copy of the email I was sent by Pandora to inform me that my account was compromised kindof but not really and it was totally not their fault.

Pandora account compromise confirmation

This is somewhat old news (I received this email July 6th) but the more copies of this online the better, IMO.

There are a number of things about this email that irritate me. First of all, the email is so incredibly vague that I have absolutely no idea what happened. Someone, somewhere posted my Pandora username (email address?) on the internet along with, presumably, one of the bazillion passwords associated with it. Who posted this information? Why? Where was it taken from? Was it stolen from one of Pandora's infrastructure providers?

If what Pandora implies in the email is true - that the compromise is completely unrelated to Pandora in any way - why are they sending me this email? Does Pandora scour the internet for the email addresses and account names of its many users? If Pandora had no responsibility for this breach and they sent me this message in order to be proactive to protect me - which is great - then why couldn't they be more forthcoming with detailed information? I get that many of Pandora's users are going to be non-technical, but you can include a link to a website with a comprehensive explanation of what happened or simply format the email to begin with a "tl;dr" version, followed by an exhaustive version for nerds.

There are no hard and fast rules for dealing with a compromise, but Pandora's message left me with many more questions than answers.

Sunday, July 31, 2016

Media, "Experts", too quick to assign responsibility for DNC hacks

I'd like to tell you a story. Its a story that doesn't particularly make me look very good. It was at a point in my career where I still had a lot to learn, and like many young people I thought I was smarter than I was. But its a true story and there is an important point to it, so I'm telling it here even at the risk of looking a bit like a schmuck.

To tell the story, we have to go back in time. The year was 2006. There were still movies in the theaters that didn't have a single comic book character in them. George W. Bush was still best known for destroying the middle east and not for his adorable stick-figure self-portraits. No one that worked outside of telecommunications or that didn't wallpaper their house in aluminum foil believed that the NSA was wiretapping everyone and everything. And I had just received a promotion.

I was working within the primary data center of an internet service provider. The company I was working for had a tiered engineering structure and I had just gone from Tier 1 to Tier 2. I would be making more money and accepting more responsibility in return.

A big part of that responsibility was investigating and resolving abuse complaints received by the ISP. Whether a company hosts servers, websites, emails or provides commercial internet service (this company provided all of the above) occasionally someone will do something on your network they aren't supposed to. Sometimes when someone does something naughty on your network, someone from another network notices. Maybe someone downloaded copyrighted material with P2P software and was caught: the copyright holder would send in a DMCA request. Maybe someone's website has been compromised and the hacker has started scanning the entire internet for a specific exploit; the admin of another network notices and sends an email begging to make the scanning stop. Or maybe someone has defrauded the company by using a stolen credit card and fake company details to sign up for a dedicated server, which in turn is used to send spam - one of the many IP reputation services send over an automated email sending examples of the messages. It had become part of my job to read these messages, investigate them where needed and determine how to handle them.

I was really excited about this promotion. When I was younger I had read books like the Cuckoos Egg; now that was going to be my life. But there was a problem: at this point I knew quite a bit about web servers, but not so much about email servers. I knew even less about the even-at-the-time out-of-date and incredibly-proprietary custom qmail cluster that provided an enormous chunk of this company's email. So I started reading.

I read every RFC that referenced the SMTP protocol. Then I read how no one pays any attention to that shit. I read all about qmail. I learned how to read email headers. I learned how to tell when headers were forged and some of the tricks spammers used. I handled my first few dozen cases well and closed them quickly. 

But there was a problem. The cases I came across lacked drama. It wasn't like the Cuckoo's Egg. Although in a few cases I might have been able to find out exactly who was responsible for hacking a server or setting up an illegal spam service, there was nothing I could do with that information. Even in the rare circumstance where the person was actually in the United States, what was I going to do? Call 9-11? Call the State Attorney's Office? Call the FBI or the Secret Service? Despite what you might read in the funny papers, law enforcement is not equipped to investigate or prosecute the vast majority of "cybercrime" cases. Victims have no one to call, local, state and Federal police don't want to be involved unless there is a political or regulatory angle, and the most simple hacking case is almost always a mess of jurisdictional SNAFU's. You think Bernie Fife knows how to get a warrant for those Ukrainian VPN logs? (He doesn't.) The fact is, when you read about a criminal computer crime investigation, you are essentially viewing a photograph of Big Foot. 

But I desperately wanted to be a White Hat Cyber Cop. I wanted to take down a Cyber Porn ring or a bunch of Russian mobsters (Russian Business Network was my Moby Dick). But that just wasn't my job. My job was help fix whatever had been broken, to make sure that my customers were able to safely resume doing business as normal, and to maybe make some recommendations to make the next hack a little harder to pull off without making everyone's life miserable.

One day I came across evidence that two servers owned by the same customer had been the source of a substantial amount of malicious network traffic. Somehow (this was a big network) this had been missed up to this point. It had been going on for months. These servers had been used to break into other servers on other networks; VPN tunnels would then be established and spam would be sent through the tunnels. Most of the time it looked like normal ssl traffic. 

The more I investigated the situation the more I became convinced this customer was not the victim of these attacks, but was responsible for the attacks. There was no smoking gun, but it in my mind everything in my mind pointed to the customer being the Bad Guy. I spoke to the technician who built the pair of servers for the customer, and the tech remembered the customer had a series of very specific, unusual requests for how the disks were supposed to be partitioned and for how the kernel was to be configured that was similar to how I had seen customers setup a server that could be immediately wiped of any incriminating evidence. I checked out the websites hosted on the servers. The main website - I will never forget this - was an incredibly bare-bones CMS selling decorative rocks. Geodes, crystals, that sort of thing. That might not be so weird for someone with a $2 a month webhosting plan, but this guy had multiple dedicated servers; most of the customers getting servers were insurance companies, universities, doctors offices, military contractors. And this guy. Selling rocks.

I sent the customer several warnings about the hacking; I gave him my best estimation of how he could lock down his server and told him he could hire us to secure it for him. The responses were spotty, and the hacking continued. Eventually, I made the case to management to cancel this customer's service. I was able to get them to agree to my assessment and the customer's account was canceled. 

It was almost immediately after that when I realized that I had completely misread the situation.

Sophisticated spammers know how to plan for having their service canceled. Its part of doing business for them. When they sign up for a 1 year contract they know they are only getting a few months of service out of it. Spammers have always been at the forefront of complex unattended installation, continuous data recovery, imaging and virtualization because they have to turn servers up fast and whenever the banhammer comes down they need to already be activating service at another provider. 

When you cancel a spammer's server, they might send an email in asking why they can't reach their host, and when you tell them they've been spamming they will never contact you again. They're prepared, so there is no point in further discussion.

But the customer with the rock website contacted us, and when we told him he had been spamming he was completely devastated. He sent multiple emails. He called everyone at my company he could. It was clear he had no backups, no plan B. The servers were his livelihood. He begged us to reactivate them, at least long enough to make a backup.

I knew I had made a mistake. I was able to work out a compromise in which we built out a new server to replace his two older servers and helped him transfer his data over safely. The story had a happy ending; the customer got a reduced monthly rate, my company got to reduce the power usage in the data center and keep its profit margin the same, and we stopped the hacking. But the happy ending isn't what's important here.

What's important is that I was wrong. When it counted, I was paying more attention to what I wanted to find than I was to what I could find. I made intuitive leaps based on reasoning that didn't support those leaps. I wanted to be Clifford Stoll. I wanted to impress my boss. I wanted to Get the Bad Guys. Perhaps more important than any of these things, I wanted to have The Answer. More compelling than my fantasizes of being a Cyber Cop was my fear of being incompetent. I thought that being competent meant always having the right solution. 

I could have done my job more effectively by taking more time to review the evidence, and spending less time trying to "connect" a handful of dots that didn't lead anywhere meaningful. Although the story had a happy ending, it could just as easily have had a terrible ending. What if the downtime I caused that customer destroyed his business? 

Over the years I have taken this experience to heart. I've become very reluctant to use intuitive leaps to justify troubleshooting or infosec determinations. Although computing provides us with a rare opportunity to work in a forum in which objective decision making is possible. There are right and wrong answers in computing; but there are also situations in which we don't have enough data to determine the difference between them. Its become easier for me to point out when there isn't enough information to resolve a problem (owning my own business has had no small part in this).

Alright, so that's the story. What on earth does all of this have to do with the DNC hacks?

Over the last week or so I've begun getting my hands on and reviewing the emails and attachments from the Democratic National Committee that have been leaked to the public by a shadowy figure(s) named Guccifer 2.0. This hack became international news beginning last month when the controversial "cyberwarfare" company Crowdstrike announced that the DNC had been hacked, and shortly afterward documents from the DNC began being leaked to a variety of different news outlets, from the Smoking Gun to Wikileaks.

From the very beginning of the DNC hack's injection into the news cycle, the blame for the incident has been squarely laid at the feet of Russian intelligence services. The Russian connection was established by Crowdstrike, who had been asked by the DNC to investigate a hack before the leaks began. Crowdstrike CTO Dmitri Alperovitch published a public report of the findings of their investigation, apparently at the behest of the DNC, in which samples of malware were provided that had links to other attacks that had already been attributed to Russian intelligence, like the compromise of the German Bundestag's network discovered earlier this year.

The attribution to Russian intelligence has gained steam over the last few weeks until we reached the point we are at now - where news outlets are now reporting the Russian intelligence attribution as fact. It is primarily this that I take issue with. Please note that it may very well be the case that Russian intelligence is behind all this. My concern is there is not nearly enough evidence to declare that attribution as fact without additional evidence.

Crowdstrike's report does not provide the required evidence to establish the attribution. Although the report provides a malware sample and a list of IP addresses associated with prior Russian intelligence-attributed hacks that Crowdstrike claims to have recovered through their investigation, these samples are provided without any form of context and in a format that makes it impossible for other researchers to attempt to replicate their findings. There is no explanation of how these samples were acquired. This is a bit like if your doctor told you that you have lung cancer, and as evidence offers you a picture of a cancer cell that's been cut out of a medical journal instead of, say, an X-Ray of your chest. The Crowdstrike report is an explanation of Crowdstrike's findings. It is not proof of Crowdstrike's findings.

There are a number of reasons why Crowdstrike would have opted the report in a way that cannot be objectively verified or peer reviewed. The first and foremost reason is that the DNC almost certainly asked them not to provide any information about their network. Another possibility (that is less defensible but I hear repeatedly) is that Crowdstrike would not want to reveal their "sources and methods".

And, to be fair, Crowdstrike provided their findings to two other companies - Fidelis, Mandiant and ThreatConnect - all of whom have apparently confirmed at least some of Crowdstrike's findings.

So I am willing to overlook the fact that Kurtz has a long standing history of making inflammatory accusations that are both demonstrably false and troublingly indicative of someone with little to no understanding of infosec. I am willing to overlook the fact that Crowdstrike's claim to fame was not for its skill in solving complex hacking investigations but for offering so-called "hack-back" retaliation services - a business opportunity that Crowdstrike was able to capture because their methodology was so ethically and legally questionable that no one else in the infosec community would have anything to do with it.

I am even willing to overlook the fact that Crowdstrike has corporate partnerships with the two out of three of "independent" companies that confirmed their findings.

Let's take for granted that Crowdstrike's report is 100% accurate and Russian intelligence services did, in fact, compromise DNC systems.

Even if we take that for granted, it still doesn't mean that the DNC email leaks can be objectively attributed to Russian intelligence. 

Those who have read the Crowdstrike (or Fedelis) reports may notice that there is a lack of any mention of the DNC's email servers or evidence of large-scale file retrieval. Its quite likely that these details were left out as part of the concerns I listed already - that the DNC hopes to profit from security-through-obscurity and prevent even basic information about their network from going public. Reporters eager to demonstrate the Russian connection have relied primarily on the @pwnallthethings Twitter feed, maintained by Matt Tait (who, apropos of nothing, claims to have been "an information security specialist for GCHQ").

Tait's Twitter feed has been used to bridge the gap between the Crowdstrike report and the DNC documents leaks by Guccifer 2.0. Tait's primary contribution was discovering that a number of the documents released by Guccifer 2.0 had been modified, and that the individual who made these changes was using a version of Windows with the Russian Language pack enabled. When reporters and bloggers say that "metadata" within the Guccifer 2.0 documents proves a Russian intelligence connection, this is what they are talking about.

In addition to this finding, journalists relied on retweets from Tait's Twitter account for confirmation of other findings, such as the Bundestag link, as illustrated here:
As I was reading through Tait's tweets and his subsequent blog guest posts, I saw myself 10 years ago, with the rock reseller. The DNC hacks significantly increased Tait's cache on social media, as can be seen here (the hack became public June 14th).

@pwnallthethings follower growth for July 2016
Just to be clear: I'm not alleging some sort of a conspiracy. I didn't accuse the rock seller of being a spammer because I hated him and wanted to get him. I went after him because it was a better story than the truth. It was more interesting than the truth. And there was evidence that confirmed my story, just as there is evidence pointing toward Russian Intelligence being behind the DNC leaks. Its just not enough evidence for us to claim it as a fact (yet).

Tait rejects the claim that his findings are influenced by bias:
Seems reasonable. But the trouble is that everyone is biased. I'm biased. You're biased. If you are human, and you have a subjective point of view of consciousness, you are biased. The way to handle this is not to deny it, but to account for it. I don't think Tait or the journalists who have used his findings as definitive proof that "Russians did it" have a bone to pick with Russia. Its just a damn good story. Who wouldn't want to be part of a spy novel?

Also, I use Tait here because the media has decided to rely on his findings so consistently, but he is not alone in transforming tenuous circumstantial findings into Objective Truth. Some of my personal favorites are:

   - Vice Magazine brought in linguists (I am very much avoiding the use of a hackneyed but still-amusing pun here) to analyze the transcript of an interview between a Vice reporter and Guccifer 2.0. Even the honey-picked quotes provided by Vice made it clear that nothing could be proved from these transcripts other than that Guccifer 2.0 likely used Google Translate, but the article has been used as further "proof" that Guccifer 2.0 is Russian and not Romanian.

   - The version of MS Office used to modify leaked files appears to be cracked. Cracked versions of Office are "popular among Russians and Romanians". Because no one anywhere else in the world pirates Microsoft software (certainly I don't - stop looking at my torrents).
This is just silly, but its taken as gospel by a media that is both hungry to spark a Cyber War and whose reporters frequently have the technical acumen of my 94 year old grandmother.

So before we wrap this post up lets quickly review the fallacies that are used to confirm the Russian Connection:


This is the big one. As I said earlier, I am taking for granted that Crowdstrike's report is God's Own Truth, and that a pair of separate Russian intelligence services hacked the DNC and had access to the DNC's network for up to a year.

Even if we accept that Russian Intelligence hacked the DNC, it does not mean that Russian Intelligence leaked the documents. Let's consider some scenarios.

The number 1 reason why networks and servers are compromised is because those networks / servers are vulnerable to compromise. That's such an obvious statement it comes across as a tautology. But its not, and there are important consequences of this obvious statement. I am regularly called in to help companies that have discovered a breach in their IT infrastructure. Something that often happens is I find evidence of multiple compromises; either the victim is using multiple vulnerable software packages, or multiple parties have taken advantage of the same exploit, or the network was compromised a long time ago by a clever hacker who was able to maintain a presence on the network until some much-less-competent hacker came along and defaced a website or broke something.

One of the most compelling alternate explanations relies on a similar chain of events happening at the DNC. Russian intelligence had compromised the DNC for a long time using the sophisticated techniques described by CrowdStrike. The Russians stayed present in the network for a year in order to accomplish what intelligence services typically want to accomplish - compiling as much information as possible. Then, some knucklehead(s) named Guccifer 2.0 comes along and compromises an email server with the goal of accomplishing some hare-brained political goals known only to him/them. Guccifer 2.0, being a moron, sets off the bells and whistles that cause the DNC to contact CrowdStrike, who in turn discover the Russian intelligence presence.

There's other options. Remember that guy name Edward Snowden? Remember how he worked for a US intelligence agency? Remember how he leaked a bunch of documents to the media? Remember this other person Chelsea Manning? Remember how Chelsea released all of those cables that included detailed intelligence analyses of foreign countries? Remember how those documents had huge political implications in those countries, like maybe sparking the Arab Spring? The point is that leaks within intelligence services happen that aren't necessarily planned by that intelligence service. Those leaks can have devastating impacts on the elections of foreign countries. Here, Guccifer 2.0 is either a Russian intelligence employee or a hacker whose true target was Russian intelligence. Theres a few options within this option - Guccifer 2.0 as working for another nation hoping to influence the US election and increasing US/Russian tensions, Guccifer 2.0 as a Russian intelligence employee who has for whatever reason a *huuuuuuuuuge* (get it?) man-crush on Trump. Some of these options are crazy. But its no more crazy than the explanations of the Putin-Trump Axis of Evil floating through the media.


It sounds silly when its put into words, doesn't it? But this is what the "metadata" and "language analysis" comes down to. Guccifer 2.0 is using Office with Russian language settings. Guccifer 2.0 is chatting the way a Russian would chat. ERGO Guccifer 2.0 is Russian. ERGO Guccifer 2.0 is really Russian Intelligence. I'm not sure how to explain how stupid this is, other than to just point out that, no, not everyone who speaks Russian is a GRU agent. Maybe visit Russia and meet some of them? There are some people who speak Russian who are butchers and bakers and candlestick makers. By golly, there are even people who speak Russian that don't live in Russia at all! I know, your mind is blown, right?


Not every hacker is state-sponsored. Gee whiz, there are even *groups* of hackers who *cooperate* with each other and even *manipulate the media* and *lie about their identity* who are just teenagers somewhere. There is a rich, long standing history of teenagers playing such pranks. Kids have been hacking for longer and frequently using more sophisticated techniques than governments have. Some of the first government "cyber warfare" programs were just field agents who paid kids to hack for them and paid them in drugs. Really.

One of the most recent, well known examples of this is the lulzsec hacking group. lulzsec had a very pointed political agenda and targeted government agencies, law enforcement groups, media companies and others that opposed that agenda. The lulzsec political agenda did not fall into the binary Team Red / Team Blue archetypes that inform what passes for American political commentary, but it was there and it clearly was important to lulzsec and their supporters. Before the indictments began, there were plenty of rumors that lulzsec was state-sponsored.

If you've made it this far - congratulations. You're almost at the end. Let's wrap up.

Some companies tell us that there is evidence the DNC was hacked by Russian intelligence. That evidence hasn't been published. There is different evidence that Russian intelligence is behind the Guccifer 2.0 account. Most of that evidence turns out to be at best incredibly flimsy and circumstantial and at worst utterly irrelevant.

It may very well be the case that Russian intelligence is responsible for the DNC email leaks, but the fact remains that further investigation is required to confirm the identity of Guccifer 2.0. Attributing the attacks to the Russians before such an investigation can occur does an enormous dis-service. The Cold War actually completely sucked. We should avoid repeating that experience based on the flimsy BS that has largely informed the coverage of the DNC hacks up to this point.

Friday, July 29, 2016

Fox News asked for my take on the DNC email dump

I was interviewed yesterday by Fox News science correspondent James Rogers. I was asked for my input on the distribution by Wikileaks of emails leaked from a Democratic National Committee email server earlier this month. The entire article, which includes quotes from a variety of infosec professionals, is now available here.

If anyone is interested I might post my complete conversation with Rogers, where I talk in more detail about how the unlabeled distribution of email attachments from compromised email servers poses unique dangers to journalists, activists and researchers whose job involves reviewing each of those attachments.

This article represents the most attention paid by US media to the significant dangers posed to Wikileaks users by the insecure review methodology in place prior to distribution of these files. Although major newspapers in Europe and the UK published my findings on malware within the GI Files, no major news outlets in the United States published those findings.

Thursday, July 9, 2015

The Florida Local Government Investment Trust website was hacked by a spammer affiliated with ExoClick & Alibaba Group & they haven't told anyone

The Florida Local Government Investment Trust manages money for counties and clerks throughout the state of Florida. They handle bonds that are AAA rated by S&P; pooling assets for municipalities throughout the state to increase their buying power. The Trust was created in 1991.

The Florida Local Government Investment Trust maintains a website based on Wordpress, (I highly recommend that readers do not visit the website from an unsecured browser/computer - preferably using a platform like TAILS). The website contains a description of the Trust, the legislation under which it carries its mandate (Florida Statute 218.415 (16) (a) and 163.01), a list of employees and trustees as well as a series of financial reports covering the last year. The domain is registered to Earl Donaldson, an employee of the Florida Association of Court Clerks. Donaldson's LinkedIn page lists him as a Network Engineer. The website is hosted on a shared hosting server operated by Dreamhost.

Starting no later than March of this year, was compromised. Each document on the site was embedded with links to sales websites that claimed to sell everything from Ralph Lauren merchandise to golf clubs. The links began immediately following a div element titled "footer_column".

All of the links, which included domains registered through a variety of different countries and companies, were hosted on a server in Istanbul by a company called "Sayfa Net", which in turn leases its infrastructure from a host called Radore Hosting. Many of the domains are known spam domains. The domain registrations show classic spam behavior; a single registration would have a registrar in one country, the registrant in another country and would included an email address to a free email service, like gmail. Companies with even the least stringent fraud protection would prevent an automated domain sale under such circumstances. It is very difficult to track down the source of spam using domain registrations in this manner, as those using them are savvy enough to nearly always rely on either a stolen identity or a completely fraudulent identity. More on that soon.

Florida Local Government Trust, Josh Wieder,, spam links, index file
Landing page for demonstrating spam links
I begin by pointing out this specific change to the website because of how obvious it was. Anyone who visited the front page of the website and scrolled down would be able to see this. It would not take any sort of complex security audit to reveal a compromise. It would be obvious that the site has been hacked even to completely non-technical users with no access to the site other than anonymous browsing. I mention this because the site remained defaced for a significant length of time. continues to host malicious files - the site has continued to host malicious files for at least four months, despite efforts to sanitize the site. Adding insult to injury, Google was announcing the site as hacked as early as March 14th:
Florida Local Government Trust, Josh Wieder,, Google Malware service
Google warning message displayed for
In addition to the embedding, over 100 files were uploaded in the root and throughout several subdirectories of Many of these files contained web scripts that forced those who opened them to visit online pharmacies.

There was more to the hack then just embedding bad links in the footer of documents. Above the header of several files, including the landing page index.html, a bit of javascript checked to determine the User Agent string sent by a website visitor and executed one of two scripts based on the reply. Websites can determine what kind of browser someone uses based on the User Agent string (some browsers and savvy users modify the User Agent string to prevent them from being identified using this bit of information).

Florida Local Government Trust, Josh Wieder,, malicious script embed
code embedded in that opened connections to malicious scripts
The gist of the code above is that if your browser matches any of those in a list, you are referred to a CGI script on a website owned by the person or group that hacked the Florida Trust site, who then forwards you to an advertising affiliate network named ExoClick who finally hands over the traffic to a sales page on Alibaba. The upshot of this is that these hackers are a paid affiliate of ExoClick, who is selling the traffic that the hackers steal from Florida Trust (and a number of other websites) to Alibaba. In the world of blackhat and greyhat affiliate web marketing, the method used to hijack a users browser window to gain surreptitious click traffic is referred to as "popunder" or "clickunder". Even under the best circumstances - as when someone is putting popunders on their own website - it is widely considered spam and an unethical programming tactic. Posting such garbage on a hacked site escalates the practice to the realm of the obviously illegal.

Readers will most likely be familiar with Alibaba - their 2014 IPO was the biggest IPO of all time. ExoClick is similarly a heavy hitter in the world of online commerce, though US readers may not be as familiar with them. Based in Spain, ExoClick's affiliate network made the top 500 Alexa list in 2011, an accomplishment they share with the likes of Google, Ebay and Wikipedia.

I realize this is a huge claim. Let's break down the technical details that lead me to this determination.

We start on the landing page. From there, the malicious code in the header of the page sends visitors to one of two websites, both of which are hosted on the same server by IP address One of these two websites - - executes a file called "wat.cgi?13" that forces the user's browser to open a window which sends the users to ExoClick. Exoclick then immediately forwards the traffic to Alibaba. This process occurs in a single request using an iframe:

Florida Local Government Trust, Josh Wieder,, wat.cgi, Alibaba, ExoClick
The content of "wat.cgi?13" that sends users to Alibaba by way of ExoClick
The second website also sends users to Alibaba, but uses a different methodology to do so. This second methodology also appears to cut ExoClick out of the connection. Remember that users get sent to "wat.cgi?13" if their browser matches a pre-specified list. Many browsers place restrictions on the execution of off-domain iframes by default, which explains why two different methods are used. It is unclear whether the hackers are using a different affiliate network to collect payment for this traffic.

With the second method, users are forced to load a javascript file - "click7.js" on a website called Instead of directly opening a new window like "wat.cgi?13" in our last example, this javascript file loads a cookie which in turn forces the launch of yet another website in a new window. This behavior avoids many of the iframes prohibitions mentioned previously. The website loaded in the new window is, but it loads a new file this time - "tijaq.cgi?18".

Florida Local Government Trust, Josh Wieder,, tijaq.cgi, Alibaba
The content of "tijaq.cgi?18"
Notice how this time we go directly to Alibaba's website rather than using ExoClick's website in a referral URL. This may indicate that the hackers are selling this traffic directly to Alibaba, or using an affiliate network other than ExoClick as an intermediary, or ExoClick allows a server-side application to count traffic for reimbursement.

Just to avoid confusion as to the ownership of the sites profiting from this traffic, ,the domain registrations and IP assignments are not obfuscated or consistent with fraud:

$ host has address has address

NetRange: -
NetName:        ALIBABA-US-CDN
OriginAS:       AS45102
Organization: LLC (AL-3)

NetRange: -
NetName:        ALIBABA-US-NET
OriginAS:       AS45102
Organization: LLC (AL-3)

Domain Name:
Registrar WHOIS Server:
Registrar URL:
Updated Date: 2014-10-28T12:38:28-0700
Creation Date: 2006-04-16T11:16:46-0700
Registrar Registration Expiration Date: 2016-04-16T11:16:46-0700
Registrar: MarkMonitor, Inc.
Registrar IANA ID: 292
Registrar Abuse Contact Email:
Registrar Abuse Contact Phone: +1.2083895740
Registrant Name: Timothy Alexander Steinert
Registrant Organization: Hangzhou Alibaba Advertising Co., Ltd.(杭州阿里巴巴广告有限公司)
Registrant Street: No. 699 Wangshang Road , Binjiang District
Registrant City: Hangzhou
Registrant State/Province: Zhejiang
Registrant Postal Code: 310052
Registrant Country: CN
Registrant Phone: +852.22155100
Registrant Phone Ext:
Registrant Fax: +852.22155200
Registrant Email:
Name Server:
Name Server:
Name Server:
Name Server:


$ host has address

Registrar WHOIS Server:
Registrar URL:
Registrar Registration Expiration Date: 2015-09-01T12:21:42Z
Registrar:, LLC
Registrar IANA ID: 146
Registrar Abuse Contact Email:
Registrar Abuse Contact Phone: +1.4806242505
Registrant Name: Benjamin Fonze
Registrant Organization: EXOCLICK, S.L.
Registrant Street: Marina 16-18
Registrant Street: 18B
Registrant City: Barcelona
Registrant State/Province: Barcelona
Registrant Postal Code: 08005
Registrant Country: Spain
Registrant Phone: +34.671646725
Registrant Email:
Name Server: NS1.P23.DYNECT.NET
Name Server: NS2.P23.DYNECT.NET
Name Server: NS3.P23.DYNECT.NET
Name Server: NS4.P23.DYNECT.NET

Note that the Exoclick IP is registered to a company called ISPrime, a hosting provider in New Jersey. I tried to check for a subdelegation, but their RWHOIS times out:

$ whois
Found a referral to


None of this behavior will strike sysadmins or security professionals as particularly unique or not-worthy; this is an almost text-book example of monetizing a website defacement. What is newsworthy about this is the organizations involved, and their reaction.

At some point, the Florida Local Government Investment Trust, the Florida Association of Court Clerks, their hosting provider DreamHost, some third-party tech support or some combination thereof became aware that had been compromised. Remember how I mentioned that over 100 files forwarding visitors to online pharmacies had been uploaded? Originally these files were scattered throughout the web root directory of Someone rounded up all of these files and placed them in a subdirectory called "/burnt/", where they remain right now, and are still indexed by Google:

Florida Local Government Trust, Josh Wieder,, burnt, spam
Malicious files remain hosted on
The webserver parses these files as webscripts. It is not unusual to configure a web server to parse HTML files as PHP or vice versa. It is unusual to parse PDF files in this manner. I was able to execute these files in a browser; the files attempted to save cookies on my computer and redirect me to another server (similar to the behavior described above). To continue to host these files represents a serious professional oversight.

The malicious scripts on the landing page index.html were removed. It makes little sense for the individual or group who hacked to make these changes. Their own websites continue to host malicious scripts forwarding to ExoClick & Alibaba. Removing the malicious forwards from index.html is consistent with restoring a backup version of the file, an action usually performed by the hosting provider (in this case DreamHost) at a customer's request.

To the best of my knowledge, the Securities and Exchange Commission does not explicitly require corporations to disclose so-called "cyber attacks" (as an aside I find it amusing how everyone in government and no one outside of government uses the prefix "cyber-"); however, disclosure of hacking could be required by rules that govern risks and incidents that an "investor would consider important to an investment decision":
Although no existing disclosure requirement explicitly refers to cybersecurity risks and cyber incidents, a number of disclosure requirements may impose an obligation on registrants to disclose such risks and incidents. - Division of Corporation Finance, Securities and Exchange Commission, CF Disclosure Guidance: Topic No. 2
The Florida Trust is an organization that manages millions of dollars of tax payer funds. At the very least, such a substantial security breach of their primary online presence should not be swept under the rug. Preventing a disclosure of these events to Florida tax payers is at best completely unethical. (Florida Statutes §§ 501.171, 282.0041, 282.318(2)(i) also apply to these sorts of disclosures - there is a whole host of regulations that may apply to this sort of thing that I can't explain very well because I am not a lawyer).

Furthermore, this traffic has identified that two very large companies - ExoClick and Alibaba Group - are relying on advertising methodology that is illegal. There is no other reasonable explanation for the malicious files pointing directly at the advertising networks of ExoClick and Alibaba. I realize the gravity of this accusation; and I feel it necessary to clarify it a bit.

I have no evidence that proves Alibaba Group is aware that the traffic they receive from ExoClick is, in essence, stolen from websites like In fact, I find it most likely that Alibaba Group has no idea that what I have described here is occurring. As of this writing, is ranked 59th globally on Alexa, which is a very rough way of demonstrating that it is one of the most frequently visited websites on the planet. Organizations of that scale spend immense amounts of money on advertising, usually with several advertising firms like ExoClick. Identifying, tracking and making sense of the source of all of the traffic that comes pouring in is an incredibly complex task.  Organizations like Google have hired some of the smartest computer engineers alive to tackle that task - and the solutions required frequently terrify people when they learn how invasive such tracking must be to be effective and have lead to class-action lawsuits. So to some degree I sympathize with Alibaba Group.

With that said, the evidence I have uncovered strongly suggests that Alibaba Group money is financing the hackers behind the defacement. Alibaba Group owes the public - and in particular the voters of Florida - in explanation as to why their due diligence has failed to detect this issue before I did. Im just a guy with a computer. It would have been much easier for Alibaba Group to track this sort of activity than it was for me.

ExoClick is in a much less morally ambiguous position. ExoClick is an affiliate advertising network. You sign up for an account and they provide you with a code to embed within your website (or in this case, a series of hacked websites). Every time someone clicks on the code, ExoClick pays you. ExoClick is proud to help their users set up "pop-unders" like we saw on

ExoClick, Josh Wieder, Florida Local Government Investment Trust, affiliate marketing, black hat
ExoClick is proud to ruin your online experience

Under the best of circumstances, this sort of browser behavior has been considered unethical by developers for decades. Its remarkable to see something so contrary to good internet stewardship presented as a normal business practice, as ExoClick does on their website.

For any members of law enforcement that may be reading, it is certain that ExoClick can lead you directly to the individual or group that hacked; they will have a payment history established for and

Florida Local Government Trust, Josh Wieder, ExoClick, payment options, black hat, affiliate advertising, spam
ExoClick's means of transferring funds to "advertisers"
Consider for a moment that any of these payment methods would require bank account information to receive in any significant amount. ExoClick's records could lead to a PayPal account, which would lead to either a real bank account or a stolen bank account.

ExoClick prohibits part of the behavior that the hackers engaged in, specifically this part: "The use of any tools that artificially generate impressions or clicks are not permitted." I think it interesting that the guidelines to do not mention any restrictions on spamvertising or the use of hacking or botnets. The guidelines prohibit publishers from "promoting" hacking, but not actually hacking.

ExoClick, Josh Wieder, publisher guidelines
ExoClick's publisher guidelines; note that the use of hacking & botnets are not prohibited
I should point out here that, as with Alibaba Group, nothing here represents a "smoking gun" that shows that ExoClick deliberately conspired with the hackers. ExoClick's responsibility is more readily apparent than Alibaba's for a few reasons. First, it is almost certain that at some point ExoClick was directly paying the hacker(s). It is much easier to know your contractor - as ExoClick should have - than it is to know your contractor's affiliate - as Alibaba should have. Second, according to ExoClick guidelines, ExoClick employees would have been required to communicate directly with the hacker(s): "New Publishers who reach their minimum payment must contact Customer Services (click “Contact” above and select the Publisher Payments department) to request the activation of the first payment." Finally, all ExoClick would have needed to do to see how awful this affiliate is would have been to put one of the domain names they were billing for through a search engine. is not the only website that was hacked by this group. I have identified several dozen other websites compromised by this same group - many of these sites have been complaining to Wordpress publicly for months that this specific hacker (or group) was using a vulnerability in a Wordpress theme to deface their websites:

Florida Local Government Trust, Josh Wieder,, Wordpress, forum, hacking, spam
Another victim of the / hackers seeks support from Wordpress
I hope that pointing light on this event will compel the Florida Trust to implement greater transparency in their online disaster recovery practices. I hope Alibaba Group will begin to pay closer attention to who they do business with. I hope ExoClick will decide to join the rest of the successful advertising industry in adopting fraud prevention measures. And I hope that law enforcement uses my findings to hold the hackers responsible.

I have additional notes and research available to interested parties upon request. If you feel I have posted something here that is inaccurate or unfair, contact me and let me know how I have made a mistake - if I have printed a factual error the likelihood of me complying with a civil correction request is 100%. 

Sunday, October 12, 2014

NSA Targets Systems Administrators with no Relations to Extremism

The Details

This is a bit of an old story, but I've found to my unpleasant surprise that the issues surrounding the story are not widely understood or known. Here's the gist: leaks from the US intelligence service have explicilty confirmed that the NSA targets systems administrators that have no ties to terrorism or extremist politics. If you are responsible for building and maintaining networks, the NSA will place you under surveillance both personally or professionally; they will hack your email, social network accounts and cell phone. The thinking behind this alarming strategy is that compromising a sysadmin provides root-level access to systems that enable further surveillance; hack an extremist's computer, and you track just that extremist. Hack a sysadmin's computer, and you can track thousands of users who may include extremists among them (its a strategy that is remarkably similar to the targeting of doctors in war zones).

Five years ago such a lead paragraph would be among the most wild-eyed of conspiracy theories. Now, after the Snowden leaks and the work of other sources within the US Intelligence community, the sysadmin targeting scheme has been proven conclusively through supporting documents circulated through a "wiki" style system within the NSA and explained and reported by Ryan Gallagher and Peter Maas of The Intercept. The name of the scheme is I hunt sys admins. The entire document outlining the goals and methods of the I hunt sys admins scheme is available on The Intercept (While I typically publish source documents directly on this website for ease of use, publishing these documents present unique legal concerns that The Intercept is better equipped to handle - I apologize to users for the inconvenience of having to visit a second site to confirm sources but I assure you it is well worth the effort).

There are a few excerpts worth noting explicitly. First and foremost, the document describes that the surveillance typically begins by acquiring the administrator's webmail or Facebook account username. The NSA agent then uses an Agency tool called QUANTUM to inject malware into the admin's account pages. The Intercept has put together a video outlining the QUANTUM tool's capabilities that is worth watching. The existence and capabilities of the tool are themselves also confirmed through extensive NSA documentation. QUANTUM uses a Man-On-The-Side attack to hijack user sessions and redirect traffic to one of the NSA's Tailored Access Operations (TAO) Servers. In this case, the application server used is called FOXACID. The same application is used to compromise Firefox and Tor users (a related program in place at Britain's GCHQ called FLYING PIG offers similar functionality even while using SSL).

QUANTUM has a variety of different uses besides the one outlined above. QUANTUM has a series of plugins that allows NSA agents to take control or IRC networks, compromise DNS queries, run denial of service attacks, corrupt file downloads and replace legitimate file downloads with malware payloads.

The methodology is important as it demonstrates the importance of maintaining operational security even during personal time. These are not attacks that target political or military organizations; they do not even target corporations. They explicitly target individual system administrators.

And there's more.

NSA Agents use the tool Discoroute to retrieve router configurations from passive telnet sessions. NSA documents outline how, rather than use sysadmins to target the corporations they work for, NSA is interested in doing the reverse - using corporate router configurations to target individual sysadmins. For example, using Discoroute, a surveillance agent retrieves the access-list ruleset associated with the router. Using that access-list can reveal home IP addresses that admins use to login to systems remotely. While this may seem to be an egregious security oversight, the access-lists in question are not necessarily for core routers. The access-list could just as easily be retrieved from a PIX; an IP used to allow access to an intranet website.

The I hunt sys admins documents continue by outlining some methods to identify and surveil malicious users. The author of I hunt sys admins references the NSA's access to massive untargeted recordings of SSH sessions. Perhaps we can take some security in that the author apparently does not take it for granted that the NSA can easily decrypt SSH session data. However, quite a bit can be accomplished by analyzing encrypted data. In this instance, I hunt sys admins recommends reviewing the size of SSH login attempts to determine which are successful and which are failed. IP addresses which are recorded failing multiple attempts to large numbers of IPs can safely be identified as belonging to brute force attempters.

Why You Should Care About NSA Surveillance Even if You Do Not Care About NSA Surveillance

This is a website about technology; not politics. Whatever your opinions are about the legitimacy or warrantless surveillance, the actions of the NSA and the other Five Eyes surveillance agencies are having a significant and deleterious impact on the internet and those who build and support it. Additional leaks have demonstrated that NSA provided security firm RSA with $10 million to use the flawed Dual_EC_DRBG random number generator in its unfortunately-named BSAFE cryptographic library, providing a back door to all applications relying on BSAFE. Even more disturbing are confirmations that the NSA has obtained copies of root CA certificates and used them to compromise SSL implemented by major internet services.

But why should we care? I'm not guilty and so I have nothing to hide, as the oft-used rationalization goes. Warrantless surveillance by governments is only one consequence of the actions outlined above. Chief among concerns for the admins targeted by these policies that are unconcerned with government surveillance is that actors other than the Five Eyes nations can easily engage in the same practices as explained in the I hunt sys admins documents; frankly, few if any of the I hunt sys admins guidelines were actually invented by NSA. These are techniques designed by criminals, and criminals have massive incentives to continue innovating those techniques. To protect our privacy from criminals we must follow security best practices, and by following best practices we necessarily protect ourselves against government surveillance as well.

The fact remains that sysadmins will remain a desirable target for those seeking to break into protected systems. Protecting those systems and the users who depend on them is part of our mandate as administrators. Now that we know the extent to which the security environment has changed, the question becomes whether we continue to adapt to the new environment to best protect our applications and users, or whether we disregard our mandate.

Monday, July 22, 2013

Canonical / UbuntuForums.Com Compromised

The Ubuntu Forums maintained by Canonical have been hacked. Canonical has been incredibly transparent and are forwarding all HTTP requests to a statement regarding the attack.

Many people are impacted by this attack. Hopefully, the impact is negligible as people shouldnt be using shared logins. In the real world, though, lots of people need to change their email passwords as quickly as possible.

UbuntuForums.Com / Canonical Hacked, Josh Wieder
Of all the people whose day you could ruin ... Ubuntu?

Wednesday, July 11, 2012

Blogger Traffic Source Spam / StumbleUpon Hacked?

{Update: there is a new bit of Linux malware making the rounds that likes to play games with iframes. Comprehensive descriptions of the exploit are listed below - of particular interest is the write up on Crowdstrike. I don't have enough data to know for sure if the two events are related as nothing I administrate has been compromised, but the iframe mechanism is fairly unique in both cases.
Here is my comment on the Slashdot Article:}

I usually take a quick look at this site's traffic and referral sources following a post. One of the great things about having a circulation close to zero is that any traffic whatsoever represents geometric growth. Traffic is up 100% from 30 days ago and 1000% from 9 days ago! Don't worry about taking notes now - all of this will be included in the prospectus for the soon-to-be-announced-but-inevitable IPO.

It was during one of these regular reviews that I came across something I wasn't used to: traffic referrals from a large and reputable website, stumbleupon.

Usually top traffic sources in my blogger dashboard are reserved for russian advertising affiliates and my mom. Interested, I clicked on the link and was sent here: 

This is more like what I am accustomed to seeing. But why is connecting to blogspot, in particular my blog which is not listed in stumbleupon - in this case ostensibly from an organization called "PaidSocialMediaJobs" by way of twitter. Why did they connect to each of my blogs exactly 14 times? 

Im thinking this is a cheesy way to get some referral traffic. But is that all that is happening? If referral traffic is the point why not just host a server in russia somewhere - why go through an intermediary at all? Without server access I don't have a lot of data to go on.

UPDATE: Another widely viewed site has been compromised. The victim this time is Apartment Ratings, the method a Javascript page that, like before, allows me to control what is loaded in the iframe directly from my browser.

Second Update / Issue Resolved : StumbleUpon appears to have resolved the issue - I can no longer manipulate frames on their domain in the same manner I was able to previously. Kudos to StumbleUpon for a quick fix.

NSA Leak Bust Points to State Surveillance Deal with Printing Firms

Earlier this week a young government contractor named Reality Winner was accused by police of leaking an internal NSA document to news outle...