Starea rețelei Informații despre starea infrastructurii noastre

We are aware about the dallas location down. We reached the datacenter and they updated us that Datacenter is facing power issue due to heavy storm in Dallas.

We are really sorry for the interruption. The DC team is working to back the power online as soon as possible. We will post here once we have more update.

Update 1: Our Dallas facility provider, Prime(formerly Evocative and Carrier-1), has lost utility power. Initially the power load was transferred to battery backup and then generator as designed. However, at some point in time the generators dropped the critical load. Prime has electrical engineers investigating the problem and the solution. We will update you all as things progress.

Update 2: We received following update from our datacenter.

Update 3: All of our servers back online.

Hello,

We apologize for the short notice and inconvenience that this may bring you. Our data center provider is going to stop the colocation service. So, we need to migrate our server to a new location. We have set up a new location and scheduled the migration.

Schedule window: 12-6 AM, 14th-16th September 2022

Our schedule window is 6 hours, However, we expect 2-3 hours of downtime.

If you have any queries, please don't hesitate to contact us.

Regards
ExonHost

Update: The migration didn't go as our scheduled plan. We need to extend the migration schedule until 25th September.

Update2: The migration has been completed.

We are aware of the down. We informed the data center and they are investigating the issue. We are waiting for an update from them.

Once we have more information, we'll update it here.

The servers are up and running fine.

Our Daisy (NYC) server is down at this moment. We informed the DC tech and they are working on it.

It will be back online soon.

Our Los Angeles location is down at this moment. We informed the DC tech and they are working on it.

It will be back online soon.

Affected Servers: Chitra, Lyhcee, Jupiter, Lemon, Peanut and Tulip

12.12 PM: Servers back online.

Our SG1 server is facing a network issue. The data center technician escalated the issue to the senior network admin.

We are waiting for an update from our data center. We are unable to provide an estimated time as we do not have control over Datacenter Network.

We will update here once we get more information from our data center.

Update:

4.15 pm: We did not get any further updates from our data center. We set up a new server and processing backup restore from backup.

7.05 pm: The restoration is still running. We have completed 40% accounts restored. The remaining accounts will take up to 2 hours to complete.

We are facing hardware failure and working to resolve the issue as soon as possible.

Currently, we are checking the data, after that, we'll process with restoration. We don't have any estimated time at this moment.

We apologize for the inconvenience that caused you.

Update: We started the account migration. 32% Already migrated. It will take 2-3 hours to complete.

We are facing a kernel issue on the Nova server. We are working on it.

It should be resolved soon.

Update: The issue has been resolved.

We are aware about Chitra issue. The server is physically up but network is down. We contacted with datacenter and waiting for their update.

We will post further update soon.

We are aware about SG1 issue. We contacted with datacenter and waiting for their update.

We'll post further update here, once we get more information.

Update 1: The server is not booting normal mode. We are trying to fix it. If the fix failed, we have to reinstall the server and need to restore from backup.

Update 2: We are reinstalling the server. We'll proceed the backup restore. It can take 3-6 hours to complete the restore.

Update 3: We thought it will take few hours but for uncircumstance, it's taking longer than expected time. The restoration is running, 20% account restored and rest will be completed soon.

Update 4: 50% Accounts has been restored. Hope rest will be completed within 2 hours.

Update 5: All accounts restored. The server is running fine.

We will do an emergency maintenance in our Bangladeshi datacenter. So, you may face small downtime. Downtime will be roughly 15-30 minutes. 

Maintenance Window:
———————————————
Start date: 12th September, 2021
Start time: 3:00 PM (Bangladesh Time)
End date: 12th September, 2021
End time: 6:00 PM (Bangladesh Time)

Reason for Maintenance:
———————————————
We'll migrate our server on new rack and router will be moved on new rack.

Update: The maintenance has been completed successfully.

Xen2 is down at this moment. We contacted with datacenter and waiting for their update. We'll update soon with more information.

Update at 8.43AM:

Server's console showed a black screen and was unresponsive to keystrokes.
Server has been rebooted and is back at login and pinging.

We apologize for the inconvenience that caused you. We are facing some hardware issue on Jupiter Server. So it's affecting server performance. We are working to resolve it as soon as possible.

We found the issue with network after our investigation. The datacenter changed the switch and network is stable and sites are loading smoothly.

Dear Customers,

The datacenter is down for fire issue in datacenter. We got following update from datacenter.

One of our old generators that have worked for years and was recently load tested had a mechanical failure and caught fire resulting in power being cut to our core routers and fire suppression system controlling the fire. Unfortunately, the fire department opted to cut power to the rest of the building as a precaution even though the power systems were independent. We are currently waiting for an emergency inspector to arrive to give the all-clear so we can bring most of the servers in Ogden back online.   Some servers will have an extended outage as they may require rebuilds due to some water damage. Those builds have a high probability that data is intact. 

Update: All server is up and running.

We are aware about the Jupiter server outage. We contacted with datacenter and waiting for update. Once we get more information. We'll update here.

Update: Server is up now. It was PDU failure and datacenter replaced the PDU.

The datacenter will be migrating from old network to  new network in New York City metro.

Expected Down Time: 2 hours.

Server data will remain intact and untouched. The server will simply be connected to a new switch. The server will remain powered on through the entire migration.

The datacenter will be migrating from old network to  new network in New York City metro.

Expected Down Time: 2 hours.

Server data will remain intact and untouched. The server will simply be connected to a new switch. The server will remain powered on through the entire migration.

We are going reboot the server to update the server software. We'll post the update here.

We are going to reboot the server within next 5 mins.


We rebooted the server and it running fine now.

Dear Customers,

We found both disks in the cherry server is failed. So we need to replace the disk. We've accounts backup and we'll restore accounts from backpu after replace the disks.

Update: Datacenter Tech powered down the server.

Update: Disk Has been replaced and OS reinstalled. Now we are installating softwares. After complete the installation, we'll start the restoration.

Update: We started the data restore. It will take 5-7hrs to complete the restore.

Update: The restoration is still running. 40% completed.

Update: The restoration is still running. 60% completed.

Update: The restoration is still running. 90% completed.

Update: The restoration is completed. If you found any data loss please open support ticket.

We found mysql server is down and we are working to resolve the issue. We'll update the status with further details.

We fixed the MySQL Server issue. It's up now.

We see titan server is down. We contacted with datacenter and waiting for their response. We'll update after getting the response from datacenter.

Update: Datacenter is facing power failure issue in their rack. They are working to fix the issue.

Update: Datacenter is replacing failed PDU in their rack.

Update: Server back online now.

This notification is to inform you of scheduled maintenance that may affect your ExonHost service. Please review the details of this notification to determine how it will affect your service. (all times UTC+6).

Event Type: Server Kernel Upgrad and Reboot
Start Time: 2017-11-20 12:00
End Time: 2017-11-20 12:20
Location(s): Los Angeles
Server: Padma
Client Affecting: Yes

Event Summary: Important maintenance will be performed on one of our server that may affect to your services. The server will be rebooted to upgrade server kernel. So you will face 10-20mins downtime.

Cherry server is facing high load issue. We are working on the server to back normal everything.

We'll update the status soon.

Everything is back normal now. It was ddos attack. We removed the targeted account.

We are aware of server3 down. We contacted with data center and waiting for their update. We'll update the status page when we get more information.

Update: Server is online now after rebooting.

We will be rebooting the server to complete the installation of CloudLinux.

Update: Server back online.

Network is currently down for maintenance.

The Maintenance Duration is : 04-14-2016 8:00 AM (EST) - 04-14-2016 12:00 PM (EST)

We are performing emergency network maintenance during the above window. This could result in a brief network outage and intermittent connectivity issues during the maintenance window.

The scope of work entails upgrading network distribution equipment.

The switch on the cabinet where our server(s) is located has experienced an unexpected power failure. Our data center engineers are currently working on replacing the failed switch with a new one. We should have you back online within the next 30-40min, so please bare with us for a bit longer.

We are aware about the orange server down. Suddently server power got off and we started the server.

Currently FSCK is running. It should be back online after complete the FSCK. We'll update you after back online.

Update: FSCK is finished. Server is back online.

We are aware about the server 6 down. We rebooted the server and server is running a quota check against the file system. It will return online once this is completed. No status output is available to show how far along in the process the server is.

We are are aware about the issue. We are facing high load issue on the server.

Now we rebooted the server, OS is showing error. Our Tech is working to resolve the issue as soon as possibe.

We are going to reboot Xen01 node patch xen hypervisor vulnarabilty. We are expecting 30mins downtime.

We rebooted the server. It should be back within 30mins.

We are facing DDos attack on our srv3. Datacenter nullrouted the server IP.

Update: Attack is still on going at around 5.6 Gbps and 2.3 Million PPS. So datacenter extended the nullroute for next 4hrs.

Update: Nullroute has been lifted. All site is up now.

We are facing kernal panic issue on this server. We are working on this issue.

We are aware about the issue and working to fix the issue.

Update: Server is back online now.

We are aware about server05 issue. We are working to resolve the issue as soon as possible.

Update: Server is now online.

We are facing disk issue on our server 6. We are working to fix the issue as soon as possible.

We are aware about the server6 issue. We are working on thisand we are trying our best to resolve the issue as soon as possible.

Update: The issue has been resolved and all site is loading fine now.

SRV1 server is currently undergoing a DDoS attack, and the main shared IP is nullrouted by the datacenter. We're working to resolve this now.

Update: We've changed our server main IP. So new IP can take 4-6hrs to propagate.

Server2 has suffered a kernel panic and has been rebooted.

The server is currently running a filesystem check and is expected to take around 30mins to complete.

Update: FSCK has completed and the server is back online.

This server is currently under a DDOS attack and the server IP has been nulled. This will be removed in 4 hours.

Update: Datacenter lifted the IP null route. All site is loading fine now.

Server02 is currently down. We've contacted with Datacenter and waiting for response.

We'll post details after back the server online.


Update: server is now online. We'll post update after complete the investigation.

Server01 has been rebooted. It can take 10-30 mins to back online.

Our monitors have alerted us to a DDoS attack against Server 05. We are working on this as quickly as we can and sites should be back up and running shortly.

Update: The server is running fine now.

srv5 is now under a DDoS and it's main IP has been nullrouted for a period of 30 minutes. After this, the nullroute will be removed and sites on the main IP will then be back online.

We see Nginx is down now. So all sites doesn't opening now. We are working to fix Nginx issue.

The issue has been fixed. Please open a support ticket if you are facing any issue.

srv2 is offline currently as it is undergoing a filesystem check (FSCK) process to ensure data integrity. We are watching the fsck process via remote console and will have it back online immediately as soon as we're ensured all data is intact.

Update: FSCK has completed. Server is online and operational.

Server02 and Server06 servers is down now for network issue. Datacenter is working to resolve the issue. Here is the reply from data center.

"The issues being experienced at this time are network related and are being actively troubleshooted by our datacenter technicians."
"At this time, we have no exact ETA but we have our best technicians actively working to resolve the issue. Thank you for your patience."

Server is back online now.

We are seeing unauthorized access on our server. We are investigate the issue now. Typically in this case we will need to reinstall the OS in order to make sure there has been no malicious backdoors or shells left behind. We will update this ticket as more information is gathered in our investigation.

Update: We checked for any obvious signs of malicious files or injections. As nothing has been found so we back server online. We highly recommend you to change your CPanel, FTP, Database password.

We've found some service doesn't running correctly. So we rebooted the server and it can take up to 20 minutes.

Update: It seems now we are facing network issue. We contacted with datacenter and waiting for their response.

Update: Datacenter technician just replied us that there was Network adapter error and they solved the issue. Now server is loading fine.

We have rebooted the server and found physical disk issue, the server is going through a file system check at this time, we'll keep you update.

Update: fsck is still running, we'll keep you udate.

Update: 75% complete.

Update: Server is back online.

It appears our FL datacenter, Hostdime is facing network issue.
Their engineers are aware of the issue and are working on restoring service.
We will update here when server will be back online.
Update: server is back online. The outage was caused by inbound attack on a small portion of their network.

We are aware of performance issues with US8 (LA) and are working to resolve this as quickly as we can. Please bear with us.

Update: As per yesterdays email the server is now going down for a filesystem check.

Update: The check is now being run. We will update with the progress shortly.

Update: The check is currently at 45% complete.

Update: 76% complete

Update: The main fsck has finished and it's performing a quick second check before booting.

Update: The second check is at 60% complete

Now at 88%

The server is now booting

Local quotas are being checked so the server start is taking longer than normal.

The server is now up and running and will take a few minutes to settle.

We have just discovered a potential hacking effort targeting our server. Upon further investigation, it does in fact appear that server has been rooted. For your safeguard, We've taken public facing networking devices offline in order to prepare for our next steps. We will post update shortly.

Update: We are starting restore the server from our offsite backup.

Update: 50% completed.

Update: 80% completed.

The account restoration has completed. If you are facing any issue please submit support ticket.

This server is currently experiencing a small DDoS attack. We're working on mitigating this now.

US8 is currently offline. A reboot has been issued and the server should be back online within the next 20 minutes.

The cause of the downtime will be investigated once the server is up and running.

Update: The server is currently running a filesystem check following the reboot.

Update: The check is at 65% complete

Update: A second check is now running to ensure the filesystem is clean. As the first repaired any issues this should complete much faster.

Update: The second fsck is at 50%

Update: The server is now booting.

Update: Server is now up and running.

Engineers were alerted to a system failure on the us(master) node. Upon investigation, engineers found the system had a kernel panic. Engineers rebooted the node and it is currently running fsck. Server will be online once the fsck is completed. The time required for the fsck process will vary but we expect around 30 minutes for it to complete. We will get back to you in this ticket with further updates shortly.

Server is online now and we are monitoring it. Please note that server will restart after some time when it re-calculates the disk usage quota.

We're currently experiencing downtime on servers US8 due to a network issue within WebNX, our Los Angeles provider.

Onsite techs are working on the issue and we'll provider more updated shortly.

The issue has now been resolved and was traced back to a faulty router.

A full RFO will be posted shortly.

We're currently experiencing packetloss at our Los Angeles location.

Our datacenter provider is aware of the issue and are working on resolving this as quickly as they can.

Our Network engineers are currently working on the node as it isn't responding.

Update: Our engineers have indicated the server was unresponsive at console. Upon reboot the system forced itself into a maintenance shell and is requiring a file system check. We will update the tickets as more information becomes available.

US6 is performing a quick reboot after making some server changes to improve performance.

Update: Server is performing an fsck and is at 40% complete. It should be back up in 30 minutes.

Update: 70% complete

Update: The fsck is complete and the server is booting.

Update: Server is now online.

We're currently aware of a problem with US6 and are working to bring it back online as quickly as we can.

Update: An fsck is currently being run on the filesystem

Update: The fsck is currently 60% complete

Update: It's still running the fsck. As it's taking this long we would assume there will be some filesystem damage. We're going to give it another 4 hours to complete and if it doesn't come back we will swap the RAID card and drives and start restoring from our backups.

Update: We're now swapping the hard drives and RAID card and starting the backup restore.

Update: The restore is now in progress. As anyone that has used R1Soft knows, it doesn't show an ETA for the restore. We're monitoring the bandwidth graphs and will provide a rough ETA once have transferred enough data to work out an average.

Update: So far 11GB has transferred out of around 200GB. We expect the restore to take a few more hours.

Updaet: 60GB restored.

Update: 75GB restored.

Update: It seems there was less data to restore than we thought. The restore completed at 100GB but the new RAID card seems to be incompatible with the restore. We're replacing it again with a card of the same model as previously used.

To clarify, once the RAID card has been swapped we should be back up and running. There is no more data to restore.

Update: The server is having trouble booting due difference in partition sizes between the old and new arrays. Onsite techs are trying to work around this but it's possible we may have to run the restore again.

Update: Unfortunately we're having to run the restore again. We'll post updates for every 25GB restored.

Update: We're at 26GB restored now.

Update: Restore falted earlier and had to be restarted. Since the restart, we are now at 50.5GB and going strong.

Update: 75GB restored.

Update: 100 GB restored.

Update: 125GB restored

Update: The restore is completed and we're making some kernel adjustment to allow it to reboot.

Update: The server is in the middle of performing an fsck - this is normal after a bare metal restore and it should last around 30 minutes

Update: 80% complete.

Update: A second fsck is running and is at 65%

Update: The server is now up and running and we're checking the status of services now.

Update: Litespeed is refusing to start due to the hardware swap. Their licensing system isn't allow the key to be used. We're working around this now.

Update: Litespeed has been started.

Update: We're swapping some cables around to restore network connectivity for dedicated IP customers. Server will be offline for 5 minutes.

There seems to be a Network issue which is causing packetloss on USA server. We have found that it is a syn_flood and will be resolved as soon as possible.

Update: We're blocking the IPs attacking us but this will take a little while to fully process.

cPanel9 is currently offline for a drive swap to resolve recent performance issues - this should take around 20 minutes and the server will be rebooted.

Update - The server is now booting back up again and the RAID is rebuilding.