Showing posts with label DOS. Show all posts
Showing posts with label DOS. Show all posts

Wednesday, January 20, 2016

Microsoft search indexing can be so aggressive that it resembles DoS traffic

As part of my consulting business I have a number of web servers I take care of. This morning, I woke up to receive a particularly crappy message related to one of those servers:

possible DoS attack

Awesome, right? Ever notice how you never get these sorts of messages between the hours of 9 AM and 5 PM, Monday through Friday?

So I tried to SSH into the target server, and was pleased to find I was able to connect. Relieved that this was likely a false alarm, I found this in the Apache logs:

40.77.167.20 - - [19/Jan/2016:19:43:15 -0500] "GET /robots.txt HTTP/1.1" 200 146
40.77.167.20 - - [19/Jan/2016:19:43:15 -0500] "GET /robots.txt HTTP/1.1" 200 146
40.77.167.20 - - [19/Jan/2016:19:43:15 -0500] "GET /robots.txt HTTP/1.1" 200 146
40.77.167.20 - - [19/Jan/2016:19:43:15 -0500] "GET /robots.txt HTTP/1.1" 403 5
40.77.167.20 - - [19/Jan/2016:19:43:15 -0500] "GET /robots.txt HTTP/1.1" 403 5
40.77.167.20 - - [19/Jan/2016:19:43:15 -0500] "GET /css/main.css HTTP/1.1" 403 5

Take a note at the timeframe on these connections: six connections from the same IP address within 1 second, five of which were to the same file. Also note that the initial connections were successful - errors only began occurring because my Apache config blocks suspicious traffic.

You've probably guessed who this IP address belongs to if you read the headline to this article:

NetRange: 40.74.0.0 - 40.125.127.255
NetName: MSFT
Organization: Microsoft Corporation (MSFT)

At first I thought this IP might be part of Microsoft's cloud server system, Azure, or some other product that might be operated by customers. However, that seemed unlikely as this host was going after the robots.txt file and nothing else other than CSS. That is what search engine spiders do. And this IP very much looks like part of Microsoft's search infrastructure:

# host 40.77.167.20
20.167.77.40.in-addr.arpa domain name pointer msnbot-40-77-167-20.search.msn.com.
The day after these weird connections, the same Microsoft IP came back with a more normal traffic pattern:

40.77.167.20 - - [20/Jan/2016:06:53:35 -0500] "GET /robots.txt HTTP/1.1" 200 237
40.77.167.20 - - [20/Jan/2016:06:53:36 -0500] "GET /index.html HTTP/1.1" 301 245

A standard installation of mod_evasive would result in a temporary blacklist for this kindof traffic. It is unclear if this behavior was intentional on the part of Microsoft, or if more rapid requests for files can be expected. The people who make their bread and butter spreading SEO gossip seem to agree that connectivity failures & web server 50* errors can have an impact of search engine rankings. However, such reports should be taken as just that - gossip.

Both Google & Bing report errors encountered during site indexing through their Search Console and Webmaster Tools, but I wasn't able to find anything published by either Bing or Google about how such errors impact search engine placement even in vague terms. Hopefully this was a one-time error on Microsoft's part and not part of a new approach to indexing (fingers crossed).

Tuesday, August 4, 2015

Afternoon Links 8/4/2015

I am a victim of my nostalgia. Yesterday, I revived a years-old post in which I provided bloggees with some of the latest Windows activation keys to update the data for Windows 10. I figured I might as well dredge up another bit I had let fall by the wayside; Weekly links! Exciting, I know.

   - Yahoo's ad network and Microsoft Azure's web hosting service were abused to circulate an enormous flood of malicious software. Malwarebytes is being credited with the discovery - which is a little amusing because Malwarebytes has for had their own issues with security for many years. h/t Washington Post

    - Planned Parenthood and a variety of other related organizations were brought offline by a sustained series of DDoS attacks. In what may or may not have been the work of the same group of individuals, someone has claimed they have hacked Planned Parenthood and retrieved an employee list database of some kind or another.
     AFAIK, this sort of thing is new to the abortion debate in the US - honestly the only political debates where this sort of thing typically comes to the fore are "internet" issues surrounding surveillance, cryptocurrency and the like. The "Culture Wars" are fought in city halls, lobbyist offices and in the bank transfers of PACs rather than through data center Meet Me rooms.
    Personally I am interested in finding out if the DDoS was outsourced or if there is, in fact, a pro-life botnet. Will online hooliganism become a part of the political conversation? h/t Rolling Stone

   - The Electronic Frontier Foundation and Muck Rock have partnered to file a butt-load of FOIA requests in order to provide the public with a better understanding of how biometrics is being used by law enforcement and federal government agencies to provide street level, warrantless surveillance of ordinary Americans. h/t Muck Rock

   - In a strange move, DHS Deputy Secretary Alejandro Mayorkas said that some provisions of the Cybersecurity Information Sharing Act (CISA) “could sweep away important privacy protections” and that proposed legislation “raises privacy and civil liberties concerns.” Apparently Mayorkas found nothing ironic about this statement, while the news outlets who retyped the message for public consumption found it completely normal. h/t Russia Today

NSA Leak Bust Points to State Surveillance Deal with Printing Firms

Earlier this week a young government contractor named Reality Winner was accused by police of leaking an internal NSA document to news outle...