Affecting System
-
DNS Servives
-
22/09/2020 15:30
- 05/10/2020 22:24
-
Last Updated 05/10/2020 22:24
We are aware that many of our services are showing as down or offline, including web services and e-mails.
We have identified this as DDoS Attack on our customer DNS services.
Our server engineers are currently working on redirecting this attacks and reviewing options to prevent this happening again in the future.
Update @ 19:00 22/09/20: We are still working to mitigate this issue and roll out robust preventative measures, normal service will resume as soon as possible and we thank you for your patience.
Update @ 01:00 23/09/20: The attack has ceased and we've taken some steps to better protect some infrastructure. We will review the situation over the coming hours and make a report in due course. Please accept our apologies for any inconvenience.
Update @ 12:35 23/09/20: The attack has recommenced. We have deployed a secondary NS server as of yesterday evening in a different IP location, however it appears the glue records are taking some time to propagate around the Internet. We are continuing to review and look at options.
Update @ 14:38 23/09/20: We came to the conclusion that the new DNS server built off-network wasn't serving requests fast enough and this was leading to fatal timeouts, despite the hardware being over-specified (on paper). We abandoned this server and built another server in a different location over the last two hours, and changed the DNS glue records for ns1.unlimited.uk.net and ns1.thundercloud.uk again to this new server and are waiting propagation to the wider Internet. ns1.thundercloud.uk appears to be largely serving traffic to the Internet as of this update, and we await ns1.unlimited.uk.net to similarly propagate. Unfortunately, the propagation time of a change of glue records is longer for .uk.net domain a .uk domain. We continue to monitor.
Update @ 15:27 23/09/20: The attack is ongoing, but DNS lookup services are restoring to availability as the new DNS glue records propagates around the Internet. Traffic levels appear nominal for the time of day (excluding attack bandwidth).
Update @ 16:24 23/09/20: The attack is ongoing, but the situation appears to be stable for customers using our standard DNS servers. Customers who have created their own DNS server glue records, can you please contact support via ticket to be told of the new IP's for the ns1.* DNS servers. Where we have access to the registrar record to update these, then we are attempting to do so but this may not be possible in all cases. We will continue to monitor and will update if there is any material change.
Update @ 20:47 23/09/20: The attack is ongoing, but has now also added our main website to the list of targets. We have now put the site behind CloudFlare to restore direct access to the customer support portal. DNS for unlimitedwebhosting.co.uk is now with CloudFlare and is propagating around the Internet, but we expect this to take a few more hours. Service to the cloud servers seems otherwise stable, ns1.unlimited.uk.net and ns1.thundercloud.uk continue to provide service and egress traffic is roughly nominal for this time of the evening.
Update @ 14:18 24/09/20: The attack is ongoing, and we've had to relocate our main website IP again. Otherwise, customer services continue to improve and traffic is largely nominal for the time of day. However, the attack has now spread to involve our group company zFast, and it's DNS servers are similarly being attacked. We will be migrating ns1.liveboxserver.com and ns1.liveboxserver.uk to a new host, and awaiting for DNS propagation. We will be moving zFast also behind CloudFlare however please our primary objective is to protect DNS services and customer websites.
Update @ 14:48 24/09/20: For customers who have created their own DNS/NS records for their domain, then please use 185.53.57.60 in place of 149.255.60.9 and 185.53.57.130 in place of 149.255.60.10. Please don't get confused as to what replaces what! Where possible our engineers we are updating custom name server records from our end to reduce required customer intervention.
Update @ 14:51 24/09/20: Please also note that all our support ticket systems are working via e-mail - if you have any trouble with accessing the Client Portals then just send a request to the support address.
Update @ 12:00 25/09/20: The attack against our prior DNS services ceased around 01:00 and has not yet restarted, so these are now answering queries again - which is helping propagation where it has not already happened. We will not be reverting the change to the ns1.* servers, and most issues which we are dealing with are DNS related. All servers are serving data, and traffic appears to be what would be expected on a Friday. Our apologies for any unanswered live chat and phone calls, please raise any issues via ticket either through the relevant client portal or e-mailing the relevant support address. Engineers are actively triaging and dealing with tickets.
Update @ 12:12 25/09/20: There is a little confusion about the new DNS servers when speaking to people, so can we please confirm / emphasize:
For Unlimited Web Hosting UK, your NS servers should be ns0.thundercloud.uk (149.255.60.1) and ns1.thundercloud.uk (185.53.57.60). If you are using custom name servers based on your domain, then please update these servers to use the IP addresses 149.255.60.1 and 185.53.57.60.
For zFast, Relic Host, No-Wires, 1st DNS and Web Hosting Payments, your NS servers should be ns0.liveboxserver.uk (149.255.60.2) and ns1.liveboxserver.uk (185.53.57.130). If you are using custom name servers based on your domain, then please update these servers to use the IP addresses 149.255.60.2 and 185.53.57.130.
Update @ 15:30 25/09/20: The situation continues to be stable. We will shortly move zfast.co.uk and the other sites to a different location behind CloudFlare for additional security. We anticipate that the domain records will have updated worldwide by now, however if there is any remaining propagation then please wait for this to complete.
Update @ 20:00 25/09/20: The situation continues to be stable and no further DDoS activity has been observed since 01:00. We have completed the move of zFast, Relic Host, No-Wires, 1st DNS and Web Hosting Payments to a new location behind CloudFlare. We will be undertaking background tasks over the weekend, however these should have minimal customer impact. We will observe the situation over the weekend and Monday, and if there is no further activity then we issue a full fault report via e-mail thereafter. We will update this status is there is any significant change.
Update @ 21:00 26/09/20: There has been no recurrence of the attack, and we have continued to fix up some features which were slightly broken due to the infrastructure changes (e.g. VPS control panel, blog redirection, etc). Observed traffic remains nominal for the weekend.
Update @ 22:20 05/10/20: There has been no recurrence of the attack for 10 days, and we are now closing this network status. We will be issuing a fault report in the next two days via e-mail to the customer base. Many thanks for the kind words and supportive tickets over the period, they have been greatly appreciated by all at Unlimited!