HoN Service Unavailable 6/21/11 (Problemas con conexión)

[Nonna[a]]

Debido a los últimos ataques recibidos por Heroes Of Newerth en estas semanas decidí cambiarle el nombre a: HoN Service Unavailable, para así mantener informada a la comunidad.

Toda esta información esta sacada desde la página oficial.

DDOS attacks and Network Issues 5/24

Updates:

This thread will be utilized for updates as they are getting lost in the long thread:

Update: 11:3AM

We had the service running for about 3 hours and came under another attack our mitigation specialists are working on fixing the current attack and we should have an update for your shortly.

The systems that we were attempting to use to block the attacks were not sufficient enough and we are going to be implementing a new system over the next several hours, I would expect this would take a MINIMUM of 6 hours before you saw anything and I will update you again with the status of that.

Security Warning:

There's a very good chance it's some of our own players unknowingly attacking us, as we saw with the last DDOS as soon as we starting banning large chunks of ip's that were attacking us we saw numerous threads start to crop up complaining of being unable to connect to the services. If you've installed any purported "map-hacks" "camera distance hacks" or anything else claiming these types of things and you're no longer able to connect to the network it's very possible YOU yourself are attacking the network, please scan your pc for trojans, viruses, etc.

Golden Pebbles Update:

We will be extending the time this is available to account for the downtime we have been experiencing.

2:36 PM Update

We are continuing on implementing a new solution and will provide a further update once we have been able to test the viability of that solution.

6:33 PM Update

Some of the services are slowly being restored, some of you may actually be able to get in and play a game however it is not fully restored 100%, and will not be working for everyone. We are continuing to work on the problem, and yes the boxes are still being attacked and the software is mitigating the attack as we teach it what are valid requests and what aren't.

10:35 PM Update

Yes we know, there are still major issues, yes we are still working on them, unfortunately I cannot give you a firm ETA.

2:55 AM Update

Chat is up, match making will be disabled for a bit.

Logging in: working fairly well

Creating games and joining games: Still buggy working on this before matchmaking is turned back on.

8:18 AM: More Informative Majorish Update

We have been working with Verisign on preventing DDOS traffic into our network, while this has been very successful part of the problem with working with a technology like this is you introduce a whole new network layer and have some pains in getting that ironed out and working perfectly in a complex environment.

We know some of you are having issues authorizing to game servers and match making we are still working on fixing that and will hopefully have those issues ironed out quickly.

12:34PM

Matchmaking is up and functioning for most people, there are still going to be some issues creating public games and some people who end up unintentionally blocked for short periods of time. If you are having issues logging in you should do an ipconfig/ flushdns to make sure you have the latest entries for the servers we are using now.

A traceroute to masterserver.hon.s2games.com should be pointing you to: 199.7.79.170 currently

1:24 PM

Matchmaking is back down we are working on restoring that service along with all the other services.

4:06 PM EST

We are shutting down all servers at the moment to fix an issue. The game will not be playable at all for a short time.

UPDATE

Last night we had our system up and operating with activity as usual. At that time we were seeing hit and miss issues with stats submissions but everything else looked good.

Overnight, as many of the S2 staff working on this attempted to get some sleep, we began to see more issues. While stats were still spotty, the chatserver began to behave incorrectly. We are looking into this issue to get HoN operational again.

I understand that this entire situation has been a complete disaster. I am sure to many of you, this seems completely unacceptable and shows significant incompetence. I can assure you, that is not the case. The HoN system is anything but simple. In an attempt to protect the "wrong" kind of traffic from entering our system, it has caused us to change the way we do a number of things. This is causing the unforseen issues that crop up as we plug one hole, another pops up! Please accept my apologies and realize that we are more stressed about this situation than anyone. We are working as hard (and intelligently) as we can to remedy the issues.

Once things are operating as they should be, we will be running a 2X Coin Event to help ease the pain of the downtime.

UPDATE

HoN is semi-operational again. We are not out of the water just yet. However, we are making progress.

On a more positive note, we have confirmed the identity of the individual responsible for the criminal acts causing HoN's downtime. Even better news is the UK government takes these matters VERY seriously http://www.met.police.uk/pceu/ 10 years in prison for a "flag in a video game" seems pretty silly to me!

We are working with authorities there and expect to have justice soon!

10:05 PM Friday night EST

We are going to be applying a small client side patch to help some of the issues. This should happen soon and after that we will be patching servers. After that we should be much better off.

UPDATE

Will be at least 30 minutes before servers are operational. We will let you know.

1:25 AM

Getting there, doing some tests on the servers to make sure what we put in will work correctly, then we have to patch it and we should be good to go.

Update soon.

1:45 AM

Matchmaking is now online and we know the UI bug of it stuck at 0:00. We will be fixing that when we can, but it does work, so just be patient when you enter the queue.

We are still working on servers.

2:13 AM

Had to turn off Matchmaking due to a Master Server error. Working on it, sorry. No ETA at the moment.

UPDATE

Pub games should be working again. DDOS continues but we are handling it. Server patch AT SOME POINT IN THE NEXT 100 YEARS. PROMISE.

UPDATE

Building and client and server patch to help solve the issues. Sadly, Mac and Linux versions are going to be behind the curve by at least 6 hours due to a porting problem. We are going to get them up over the weekend when we can, but this solution is better than no one being able to play the game.

Working on it.

4:37 AM

We just patched with a load of fixes for clients and servers to stabalize everything and let us get through the weekend easily. Sadly, though, Mac and Linux versions will not be updated until tomorrow sometime due to a problem with porting some of the new code over. I would suggest, if you need to play, using wine/bootcamp/whatever to get a windows version for now. That or you wait until tomorrow.

Sorry, but this is better than zero people being able to play. We will do what we can quickly.

UPDATE

We took MM back down due to spam fail auth issues. Looking into it now through the logs, code, and master server.

Pub games should still work fine I believe.

The following text represents how we feel about the current situation. It was carefully collected by me rolling my face on the keyboard in anger:

H89SAHGF789HGI890u_y#4075864387 890D YA78 36458907589- 7UT894JUHGF89HGF (LOOOOOOOOL)

5:43 AM

Matchmaking is up and should be stable for a while at least. Mac and Linux patches won't be around until tomorrow at Mid Day EST (random guess). Windows should be ok and stable from what we can tell.

There will be a few hickups now and then as we rev back up into full speed, but generally it will be working ok. If you get errors such as "Cannot Auth" or anything related, just try again and you will get through. They should be rare, though.

7:30 PM Saturday

After messing with stuff today, Mac and Linux are finally updated and good to go. We did some backend stuff with the master servers again and everything should be stable for now.

Enjoy the weekend.

UPDATE

Since Tuesday we have spent 100+ hours working on the stability of the HoN service due to vulnerabilities to DDoS. We have made major headway in that regard and are capable of mitigating many more scenarios than previously. The attacks that these criminals continue to aim at HoN are heavily mitigated, even though our service is still not 100% operational.

This morning, while our system was successfully mitigating existing attacks, another attack was aimed at us. This attack is slowing servers capability to authorize clients. Due to this, match making is very spotty and we have chosen to disable it. We are currently attempting to mitigate this attack and close this vulnerability in our system.

I understand that many of you are frustrated with this situation and want to point your anger at S2. It is most definitely our responsibility to create systems to prevent these kinds of attacks. However, the ferocity and length of these CRIMINAL activities is more than we have ever planned for. HoN ran smoothly and near flawlessly for a year of retail. We pride ourselves on our high standards of operation and this is very painful for us to endure. We will work to resolve this issue once and bring HoN back to its peak operating performance.


UPDATE

We have mitigated a lot of the new attacks, so we turned back on matchmaking for now. Enjoy.

UPDTE

Turning it back off for now due to Auth issues.

UPDATE


And back on it goes. We will see how it performs this time


[--> Fuente: http://forums.heroesofnewerth.com/sh...d.php?t=266632 <---]

PROBLEMA SOLUCIONADO

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Login Issues

We are currently seeing an issue with logging in to HoN. I don't have much more information than that at this time. We are investigating the matter and hope to have it resolved soon.

I understand this is frustrating for us all and I plead with you all to be patient as we work through this.


-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

HoN Service Unavailable 6/4

Update:

HoN is currently unavailable. We are aware of and working on the issue. When we have more information, we will update this thread.

More details to come (and hopefully a FIX!!!!!!!). (Maliken)

Update:

We are resolving a number of issues related to network outages at one of our major NoCs. We have WWW back online and limited DB connectivity. Progress is being made... I'll update this thread as more information passes my way. (Maliken)

Update:

Our current situation is not caused by an intentional attack. One of our two main NoCs had significant internal issues. These issues eventually resulted in the corruption of a few very large tables in our database (as well as no accessibility to a crucial SAN drive system).

We are repairing those tables, however, it may a while due to the size of the table and potential damage. I can't give an ETA as there is too much uncertainty.

Just know this, we are working as quickly as possible to restore service. Outside of the potential material impact to S2's business we ALL want to get a few games in ASAP!! :D (Maliken)

Update:

We're still repairing tables. When I have something more to update, I will.

Update:

We have been running the repair operation on our DB for roughly 6 hours. Once the repair is complete we will have to allow time for our replication services to catch up and then we will be live!

Update:

The repair operation is complete. We are now waiting for our replication services to catch up. Due to how long the repair took, this could be a while before we can turn the game back on.

I'm so sorry for the delay and the downtime. We are still waiting word as to what critical element within the NoC caused the outage which caused the corruption.

Update:

We are back in business!!! Thanks for your patience and GAME HoN! :D

Update:

It appears people are having issues logging in. We are unaware at this point what is causing this issue. It may be related to our DDoS protection proxy.

We are currently investigating a number of potential areas of our system the may be causing this. Please be patient.

Update:

Just an update here guys. We are aware that some people are still having issues logging in to HoN. We have been investigating potential problems that may be causing this but have yet to find anything. We will continue to look at this and report back anything we find. We are seeing significant through put though with good numbers today reflecting lots of activity and proper authentication.


[Fuente: http://forums.heroesofnewerth.com/sh....php?t=272851]

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Scheduled Maintenance Thursday 6/9/2011 6 am - 3 pm GMT -5

Update

On Thursday 6/9/2011 from 6:00 am until Noon GMT -5 we will be taking HoN down for maintenance. You will be unable to login and play games at this time. At this time we will be moving our databases and other servers to a more secure and robust environment to hopefully eliminate some of the issues we have been having recently. We thank you for your patience in this maintenance period and will work as fast as possible to have full service restored before noon on Thursday.

Thank you

Update

We have started the maintenance. As of 7am EST, stats will not permanently record. All matches and coin rewards will revert back following the maintenance period.

Regards,
S2KK

Update

Given the current transfer rates of our data we are going to have to extend this window until 3pm GMT -5.

We are working as fast as we can to restore full functionality and appreciate your patience.

Thank you

Update

When this initial maintenance was scheduled we were planning on taking the game down for the entire length of time, we have not had to do that, we will however have to make the game unplayable for about an hour once the database transfer completes. We will update everyone on when this will happen but we are expecting another 2-3 hours 4-5pm EDT (UTC/GMT -4) before that will happen.

Thank you

Update

We expect to take the game down and offline around 6pm GMT -5, we will then begin testing and within an hour or so have full service restored.

Thank you for your patience, I know this has taken much longer than expected but at least we have been able to keep the game up and functioning today during the transition.

No Stats, Coins, etc that have been earned or recorded since 7 am GMT -5 today will be included in your statistics when the new system goes online.

Thank you


-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Maintenance Window June 21st 7:30 AM - 8:30 AM GMT -5, 6/24 Patch Info

Maintenance Window

We will be having scheduled maintenance on June 21st from 7:30 AM - 8:30 AM GMT -5. We are going to be shutting the databases boxes down at this time and restarting them both approximately an hour later for some system configuration changes. Expect any games in progress or games started during that hour window to not record correctly. We do not anticipate this will take longer than one hour.

Patch Info: 6/24/11

There will be no patch on 6/24/11. We are working on features that are simply not ready for release and in effort to increase quality in the upcoming patch we are going to be moving it back a week until 7/1/11. We realize we have had some issues in the past month with our systems and are working hard to ensure every item that is released to the public is in 100% functional state.

Thank you for your patience while we grow and improve HoN.

Update:

We had a few issues during this maintenance window and will be restoring full functionality in the next 30 minutes.

Update:

We are extending the maintenance window until 10:00 AM GMT -5 to ensure all services are 100% functional before turning them back on.

Thank you for your patience

Update:

I'm terribly sorry but we have to again extend this maintenance window until 11:00 AM GMT -5.

It's very disappointing when you are hosting with the purported "Rolls Royce" of hosting companies and something as simple as the 30 minute upgrade they were to perform this morning has turned into a giant bungle on their part. We have been working with their engineers demanding a fix since 8:30 Am GMT -5 and hope to have it back up asap.

Update:

We are extending the maintenance period to 3:00 PM GMT -5 and hope to have it recovered sometime before this time.

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

HoN Downtime 6/21/11

During our scheduled maintenance period this morning we had a catastrophic hardware failure across a number of systems including some of our backup systems. We have the capacity and ability to recover from this failure. However, due to the extent of this failure we cannot provide an ETA on when service will be restored, we will continue to provide you updates as they are available to us.

We apologize, we would like to play just as badly as you would.

Update
We attempted to upgrade the processor on our database boxes. Both of our servers were simultaneously taken offline while the new hardware was being put in place. During this period of time, the file systems on BOTH boxes were destroyed. If this boggles your mind to the point that you can't believe it could happen, you would be in the same exact boat we are! It seems almost statistically impossible, however, it is in fact what happened.

This leaves us with no hardware to operate our DBs from until we can get new servers online. The reason for such a catastrophic situation is that this hardware is not the type that is simply sitting around, even in the largest hosting facilities in the world (in which we host at). We are scrambling to bring our systems back online temporarily so you can play the game through other short term methods while we get new hardware online.


Update
Maliken stated the situation pretty plainly but accurate.

A CPU upgrade was performed at the data center on our databases. A chassis swap was performed and resulted in some manner of data loss on the system. I don't know exactly how this happened. Also note, we are trying to recover that lost data which failed the disc check utility from one of those boxes. We are also actively restoring a backup which any IT professional would know that restoring hundreds of Gigs of data takes time. We are currently racing between two system restores both on site and a different location.


Update
At the same time nothing has really changed. We are not sleeping. Everyone is still here working on stuff, testing everything, re-testing everything, moving data, setting up new boxes, testing more, etc etc.

As Maliken said we had a simple hardware update in our remote datacenter go completely bonkers and destroy our databases. As soon as we found out we had to pick our chins up off the floor then used them to say "ARE YOU KIDDING ME?!"

Ever since we have been doing all we can to figure out where we are with backups, new hardware, syncing domains and making sure all of our systems point to the right places and work correctly. That is what we are doing now.

TL DR - Stuff broke. We are still here fixing it. Nothing changed on your end that you need to know about yet. We will be here all night.

Update
We are finishing up some tests and mass-restarting all of the servers to get some new IP configuration information. We will let you know how things look after all of them have reset and connected to the new database.


Update

We are turning everything back on now. The systems, logins, and such will be coming back up slowly.

Link: http://forums.heroesofnewerth.com/sh...d.php?t=285955

[mexenario]

Justo iba a preguntar eso :P
PD: Yo pensaba que maliken estaba en pleno XxX

[MiGuEl4EvEr]

Me pregunto si seran los mismos. lol

[RafinhO.]

jajaja pa concha escogen al bombardero =D

[Nonna[a]]

Hahah me caigo, ¿ya les dieron su bandera o no? xDDD

[Karasu]

No creo que les den su bandera ahora.
Si ceden a la demanda cualquier indio,japones,chileno,peruano,marciano,etc puede venir y decir "o me dan 40000 gold coins o me tumbo sus servidores".

[Nonna[a]]

Estoy leyendo el post y todos hablan de la bandera India....

Tambien mencionaron a un Plague Rader con Pk, ¿alguien me puede explicar eso?

PD:
*Give them that fckn flag already.
*In America, we don't negotiate with terrorists.

LOOL

[MiGuEl4EvEr]

Tambien mencionaron a un Plague Rader con Pk, ¿alguien me puede explicar eso?


LOOL

Se dice que un S2 jugando TMM baneo a un player porque fue mid con Plague rider y rusheo pk. Lo curioso es que el replay del match ese no se encuentra.

Article por DRD
http://www.dotallyrad.com/2011/05/22...-plague-rider/

[hAisEr]

Con razon . . . .

[Minko]

Alguien sabe xq sacaron la bandera de India :s? no me habia enterado de esto seguro es el motivo por el cual estas ultimas semanas estaba un poco lag la mayoria de juegos

[FoX-FiRe]

jajaja que pendejo banear a alguien por los items que saco xD que pros S2 que pros.

[aRkanO]

hasta cuando estaremos asi ya siento la ansiedad
ya estoy como mi imagen ><

[MiGuEl4EvEr]

Alguien sabe xq sacaron la bandera de India :s? no me habia enterado de esto seguro es el motivo por el cual estas ultimas semanas estaba un poco lag la mayoria de juegos

Una cosa no tiene nada que ver con la otra.
DDoS =/= Lag
Yo creo que nunca llegue a ver la bandera de India.

[limagamers]

Increible Noticia la de la banderilla India pero !! extraño el hon !!:(

[D1p]

No hay HoN? :(

:arrowd: Successful troll is successful.

[Jummpy]

Ya Hay Hon Ntren

[Devil]

Una cosa no tiene nada que ver con la otra.
DDoS =/= Lag
Yo creo que nunca llegue a ver la bandera de India.

Te equivocas. DDoS claro que genera muchisimo lag al punto de desconectar a medio mundo.

En cuanto a Plague Rider, los Admins dijeron que no tuvo nada que ver con que haya sacado PK, sino porque trolleo TODO el juego al punto que se gano el ban. Y el replay se borro automaticamente despues de 30 dias porque sus sistemas no pueden guardar replays por mas tiempo.

Si realmente fue un abuso, porque espero todo ese tiempo para reportar el abuso? Si a cualquier lo banea abusivamente, uno pitea al instante o por ultimo al dia siguiente y no espera que la evidencia desaparezca para recien quejarse.

http://forums.heroesofnewerth.com/sh...d.php?t=264572

[MiGuEl4EvEr]

Me referia al lag que el decia que habia en los juegos. :P Los ataques comenzaron hoy, no tiene nada que ver con los "lags" de semanas anteriores.

Sobre el ban, me parece que hay algo "shady" en todo eso. Ya pasaron casos de rage ban a usuarios por parte de S2 members.

No hay HoN? :(

:arrowd: Successful troll is successful.

Acabo de jugar un game, pero al parecer los servers se volvieron a caer.

[aRkanO]

igual jugue un juego y >< nada sigo sin poder jugar otro :s

[Loser]

a esperar nomas no queda de otra