4-30-2011 DDOS ATTACK

[imschur]

The folks that host this server are experiencing a network attack at LAX. Its causing outages of the site(s)

[imschur]

Major System-Wide Outage Posted  (April 30th, 2011 at 2:37 pm PST) by Josh We are investigating the cause and will update this post as soon as we know more.

Update 2:54pm PST: One of our network providers is having issues right now. Our network admins are working on moving all traffic to our two other service providers. We will keep updating this post as soon as we have more information. -Oscar

Update 3:10pm PST: We are having issues with one of our transit links to our LAX datacenter, our network administrators are investigating the issue and we hope to have it resolved as soon as humanly possible. This is causing problems with both external and internal networking – we’ll update this post as soon as we have more information or the issue is resolved. -John

Update 3:30pm PST: Our network engineers are on site at the source of the issue and working on resolving it. More information to follow as it becomes available. -John

Update 3:50pm PST: We are still working feverishly to resolve this issue – some portions of our network are responding better now but there’s more work to be done. -John

Update 4:10pm PST: We are seeing improvement but are still working on this issue. -John

Update 4:40pm PST: Our network engineers rebooted the core router at our LAX facilities but our providers are still flapping. It appears that the LAX facility may be stabilized but we’re experiencing high load on our core routers as a result of all the BGP flapping. We are still investigating the issue but current traffic patterns possibly indicate an attack on our network. -John

Update 5:00pm PST: We are still working to defeat this attack on our network. Nullifying certain source IPs has seemed to help matters, but that attack vectors and profiles are still being introduced and identified. -John

Update 5:25pm PST: Some attacks have been blocked that were inbound and we have identified some that were outbound and blocked those as well. We’re still working on the matter and the network responsiveness is improving but the attacks are still causing BGP flapping. -John

Update 5:50pm PST: We’ve pinned down the sources and nature of the attacks and are in the process of scanning for more attack software as well as filtering bad traffic at the core routers. The issue is not completely resolved yet but we’re making good progress now. More updates to follow. -John

Update 6:15pm PST: In a nutshell we suffered an extremely sophisticated attack. It took a while to get things under control enough to see what was going on and then start not only blocking attack vectors but track down and disable software being used to launch attacks from our network as well. Things are almost under control currently and once they are we’ll get a full and detailed report from our network engineers for you. -John

Update 6:35pm PST: Good news – all inbound attacks appear to now be screened and the CPU levels on our networking equipment is currently stable. We’ve strengthened our defenses at all levels of our network as well – borders are hardened, cores are protected, transit interfaces are stable. That said, we’re remaining vigilant as these attacks are much more advanced than what we’ve seen previously. -John

Update 6:50pm PST: All indications are that everything is under control so we are marking this matter as resolved. We will still provide details on what happened as soon as we’re able to do so. -John

Update 7:12pm PST: It’s back.. we’re on it again.. more updates soon. -Josh

Update 7:31pm PST: It seems that we’re seeing issues with our transit links to our Alchemy datacenter. Our network engineers are working on the issue and we will update this as soon as we have more information. – Oscar

Update: 8:02pm PST: We are seeing some more interruptions in networking, our admins and network engineers are still looking into the issue. – Justin

Update: 8:25pm PST: Our abuse team is getting involved as well at this point.  We have resolved issues on most of our routers, and have found over 500 compromised sites so far, that we are working on fixing.  Overall things should improve over time. -Justin

Update: 8:53pm PST: Things once again seem to be resolved.. we’re going to be cautiously optimistic and say nobody should be seeing any network issues at this time. We’re still going through squashing bad guys here and there, but the system is now sound. We’ll update this post if anything changes. -Josh

Update: 9:14pm PST: I spoke too soon again.. we’re still on it. -Josh

Update 9:41pm PST: Okay, it seems like we were being attacked because of a site we host. We’re investigating, things should be mostly back to normal for nearly everybody but the attack continues.. -Josh

[techmike]

Holy cow.  >:(

[George Wood]

I have worked as a sys admin for a long time. It is sad to see these kinds of attacks continuing to occur. That is not what technology is for. Thanks for the updates.