OT - Totally off topic - QNAP NAS Deadbolt Ransomware

[Peaklander]

Hi, I am posting this as it may help users of qnap manufactured nas devices (networked hard drive units). I have had a Raid 1 NAS on my home network for many many years. I have used it for central storage of huge amounts of work and personal related stuff. For quite some time I had it enabled to be accessed from the internet and did this so that I could pull and push material from and to it when I was working away. To the best of my knowledge, I set this up following qnap guidelines and features.

Recently I discovered that it had been accessed by malware and the files were all encrypted with something called Deadbolt. Every file except one or two with very unusual filename extensions, was encrypted (about 250K) and so unreadable. As my NAS was running a malware app, the ransom software was seen and removed automatically but not until the encryption had already occurred. In addition, the removal of this meant that I wasn't presented with the ransom demand screen when I logged into the NAS. I spent several hours trying to understand what had happened, realising what I had probably lost and discovering that my backup was also encrypted.

It's hard to over emphasise the sickening feeling I had when I got to this point - encrypted files, no idea what to do next. Slowly i discovered that hundreds of other users were affected, this was at least the second wave of attacks on this manufacturer's platform, but I read enough eventually to find and run some recovery software that produced the ransom page.

Here is the screen, once I had found it, with some of the ransom address scratched-out...

 

Without this page I wouldn't have had the address and so nowhere to pay the fee. I was then faced with the decision, pay the money or find enough of my files in other locations to 'manage'. That took more time and of course soul-searching around why did I do this and why didn't I do that. etc. etc.

Eventually I took actions to secure the NAS, I backed-up the complete drive as it was, with encrypted files. Then I researched bitcoins as that was completely new ground. The decision made, I found a platform that would accept fast pay from the bank and allow an instant purchase of BTC. I finally did this and paid the ransom with a small exchange fee but the whole £sterling to BTC to Ransom address took just a few minutes. I paid the equivalent of £905 in BTC.

Typing in the address was stressful and one mistake in the 42 alphanumerics would have sent my BTC to the wrong place. I then had to find the key for encryption and that was scary and for a hash address / code novice like me it was tricky but after an hour or two, again reading lots online to help, I got the key and nervously entered it into the field and the decryption began.

I'm happy to say that all files have been restored, I have several offline backups and the nas will only be accessed in future through a VPN.

In summary, I had no idea that the settings I had used were insecure, or that there were vulnerabilities in some of the qnap apps that I had installed on the nas. 

I hope that any readers of this post that have a QNAP NAS will take action to ensure it is secure.

[elbekko]

Did you verify that the files were actually encrypted?

But yes, always a bad idea to expose things through the internet without a VPN...

[SPendrey]

Clearly my advice is after the horse has bolted... but I tend to work with three instances of data.  The primary source is local disc, secondary is a NAS type device (in my case, an old networked PC), then thirdly a cloud-based backup.  This way I can always access data anywhere, and I can be ~99% sure I've always got a valid backup somewhere.  If you lose one, you've still got a primary and secondary to fall back on.

The only challenge is data volume in "the cloud", it takes time to migrate there, and takes £'s to store there.

[Peaklander]

Yes the files were encrypted and there’s plenty of discussion out there.

I would rather not get into a “why didn’t you do..” sort of discussion. That isn’t the intention here. I want to raise awareness so that people can sort themselves out before it happens to them.

[pete3000]

There are recent firmware updates for similar NAS script vulnerabilities which have been made available. I have an offline (not connected copy) of my NAS data. as SPendrey says 3 copies.

Your fast access (PC or primary NAS) a slower incremental store e.g QNAP or Truenas, then a removeable drive or media source. All are located apart. I gave up on QNAP commercially when I found the RAID5 array wasn't reporting disc errors until the system was rebooted (after a power failure and expiry of the UPS battery).At this point it decided the array was unrecoverable. A data recovery firm managed to get 90% back.

Most file backup or imaging software has the ability now to detect if large amounts of files are being updated or altered. Some leave a marker file which is a system file created by the backup software,if this is modified in the source the backup won't propagate to the destination.

 

Pete

[BogMonster]

Is this a problem likely to be more widely applicable than just QNAP devices?

I have various vital company documents that I simply cannot afford to use (and my email archive - same reason) which I back up every few weeks to a 2TB portable drive because I have always wondered how secure the NAS drives are and whether they are susceptible to hacking in from outside given they are effectively "always online". Security "professionals" get snobby about the concept of things being isolated via an air gap but if a hacker manages to get into my top drawer here then good luck to them! But there is a lot that is not backed up - mainly photos etc.

Should one block or do anything specific on a firewall to protect a NAS, or how do hackers get in to it, and what is the weak point? Mine isn't set up for remote access (rubbish internet connection) so I guess that is a start.

[Peaklander]

This is what the qnap forum says to prevent further attacks and I believe that this is generic:

Disable or remove any port forward settings in your router that redirect to your NAS
Disable uPnP on your router
Update your NAS to the latest available Firmware

Also there were specific vulnerabilities in one of the qnap NAS apps - Photostation

My problem was compounded by leaving my USB backup drive connected to the NAS rather than only connecting for periodic backups (after first checking for file integrity).

[FridgeFreezer]

22 hours ago, BogMonster said:

Is this a problem likely to be more widely applicable than just QNAP devices?

I have various vital company documents that I simply cannot afford to use (and my email archive - same reason) which I back up every few weeks to a 2TB portable drive because I have always wondered how secure the NAS drives are and whether they are susceptible to hacking in from outside given they are effectively "always online". Security "professionals" get snobby about the concept of things being isolated via an air gap but if a hacker manages to get into my top drawer here then good luck to them! But there is a lot that is not backed up - mainly photos etc.

Should one block or do anything specific on a firewall to protect a NAS, or how do hackers get in to it, and what is the weak point? Mine isn't set up for remote access (rubbish internet connection) so I guess that is a start.

From what I've seen and echoing comments above, nearly all the experts/geeks quote some variant of the "3-2-1 rule"; 3 copies of your data (your "live" data and 2 backup copies) on two different media with one copy off-site for disaster recovery. - it doesn't have to be expensive or complicated (external HDD's are cheap enough & very easy) and you can ignore some aspects depending how you want to do it.

Basically there's "your" copy (stuff on your PC), there's the "live" backup which might be on a NAS or perhaps cloud storage, and ideally a "cold" copy which is not online or connected to anything (a USB HDD is ideal), and these things should also ideally be in different physical locations - no good having 3 copies if they're all in the smouldering remains of your house or all of them got trashed by the same computer virus or hack.

A good & easy system is to have a couple of USB HD's you keep somewhere (EG shed, trusted friend's house, wherever) and every so often run a backup to one of them and go and swap it with the other.

The simple answer to part 2 is that anything connected to the internet / on wifi etc. has to be considered vulnerable to being hacked - doubly so when it's something consumer-grade / PC World sourced or Internet Of Things gadgetry (the running joke is the "S" in "IOT" stands for security). Cheap mass produced consumer gadgets (and yes I include "home" NAS in that) tends to have quite poor firmware, add to that the fact that hackers finding a bug in 1 device can then potentially apply that knowledge to a million of them out there in people's homes, it's a big juicy target.

I use a small HP Microserver running Ubuntu Server as a mini-NAS which lives in my garage, so if the house burns down all my stuff is on that in a separate building. It's online but not exposed to the internet. I really should create a "cold" backup too but that's on the to-do list. Any old PC from the scrap pile with a good size HDD stuffed into it will work nicely as a very cheap NAS, it doesn't need much processing power and there's open-source web interfaces etc. out there that make admin quite easy. You can do it with a Raspberry Pi and USB HD too but it can get a bit untidy.

[Peaklander]

Thanks @FridgeFreezer. The qnap NAS is a large range of devices from single discs to wide racks, all running the same firmware which is frequently updated and easy to do. I chose a dual disc / Raid 1 as I was more worried about disc failure than anything else. In fact after a few years I upgraded to a newer, larger disc and it was amazingly simple to hot-plug one in, let it do what it needed to do and then swap the second one.

The first warning of "eggs in one basket", came two or three years ago when a house on the road took a direct lightening hit. The pulse went across to the telephone pole, took out that junction box, then followed the cable back to the green box at the end of the road and then took-out the individual fibre / copper interface 'cards'. It then travelled to every house and took out the master socket and routers. In ours, with ethernet, it also destroyed the NAS pcb, an Apple TV, an eight way switch and the router-extender.

The funny thing was that although OpenReach sensed a fairly big problem (thirty houses), they wouldn't come into any house to replace the master sockets until the individual ISPs for each property had flagged a problem to them. I ended-up door knocking to tell people to let their ISPs know just how big the 'fault' was and to request a new router whilst they were at it.

In that case I bought a new NAS bare rack and then plugged the two hdds in and all was ok but that got me thinking about cloud storage - which I now use, but not for everything. Also I use Time Machine for my MacBook which I find very good and that is onto a USB HDD, disconnected when not in use.

So, I get it about the backups (and thanks for the link) but I was typical of (qnap) NAS users in that I was running malware software and also followed qnap's instructions on accessing the nas from the www. I thought this was safe. Had I read their forum or had conversations with IT types, maybe I would have been warned sooner. It seems that the malware protection didn't see the malware code quickly enough and the nas was holding the door open. Ironically I don't now need access from the www as I am not travelling for work (well I'm not working either but you get the idea).

[zardos]

Yes to 3-2-1 for me that is PC, Synology NAS and USB hard drive that lives offline in a fire safe so not off site but protected from house burn down scenario (also handy for important paper docs).
Also expose only a VPN port (when I was away), also handy to be able to route all traffic to home to get iplayer in foreign countries, browse the web as in the uk, etc

[pete3000]

On 10/3/2022 at 10:37 PM, BogMonster said:

Is this a problem likely to be more widely applicable than just QNAP devices?

I have various vital company documents that I simply cannot afford to use (and my email archive - same reason) which I back up every few weeks to a 2TB portable drive because I have always wondered how secure the NAS drives are and whether they are susceptible to hacking in from outside given they are effectively "always online". Security "professionals" get snobby about the concept of things being isolated via an air gap but if a hacker manages to get into my top drawer here then good luck to them! But there is a lot that is not backed up - mainly photos etc.

Should one block or do anything specific on a firewall to protect a NAS, or how do hackers get in to it, and what is the weak point? Mine isn't set up for remote access (rubbish internet connection) so I guess that is a start.

For docs and photos you absolutlely have to have, use CD-R's or DVD-R's. Once they are finalised they are read only. Then your only concern is disk physical damage or legacy of hardware (like ls120 or zip drives). Again 2 copies of the CD/DVD's should be made regularly , one in your possesion and one with a trusted relative or the bank. This can be useful in case of fire/theft for proving property ownership for insurance records. I have lost count of the number of stories of "all my photos were on that device when it was lost/stolen/stopped working". I'm usually able to help in the last case.

I don't trust cloud backups at all, or rely on a single copy on spinning disks or flash based storage all can fail.

Pete

[BogMonster]

Thanks Fridge/Pete ... important stuff is backed up 3x already (including my email - I do a manual copy of the PST every few weeks and always have a couple of older copies too) but I've just ordered a 4TB USB HDD which is enough to back up everything on the NAS at the moment, and I'll keep that elsewhere.

Unfortunately cloud based options are a non-starter as we have a fairly tight data cap on our internet here which precludes anything like that (as does a 0.75Mbps upload speed!)

Ah, ZIP drives, I had one of those once ... I think I finally threw it away a couple of years ago!

[FridgeFreezer]

"Cloud" is another name for "someone else's computer that's attached to the internet"

Blu-ray discs are supposed to be very good for backups - they hold a lot and IIRC they're rated for a VERY long shelf life, way beyond CD/DVD.

[reb78]

34 minutes ago, FridgeFreezer said:

"Cloud" is another name for "someone else's computer that's attached to the internet"

Blu-ray discs are supposed to be very good for backups - they hold a lot and IIRC they're rated for a VERY long shelf life, way beyond CD/DVD.

CDs have a shelf life? I would have thought any I create today or in the last 20 years would outlive me.

[pete3000]

9 minutes ago, reb78 said:

CDs have a shelf life? I would have thought any I create today or in the last 20 years would outlive me.

They do, they also can tarnish scratch and burn quite well. That's why i have at least 2 copies refreshed occasionally. The main hidden advantage of burn once media though is that they cannot be overwritten or altered by nasties or accidental means...... which is what i was aluding to above.

[reb78]

I guess the damage aspect is clear, I didnt know that they degraded over time though. Interesting. 

So it sounds like my vinyl collection will outlive the CDs that I havent thrown out or given away!

[FridgeFreezer]

15 minutes ago, reb78 said:

CDs have a shelf life? I would have thought any I create today or in the last 20 years would outlive me.

Everything has a shelf life - especially cheap CD-R and DVD-R media, they can and do deteriorate especially in less than perfect environments (EG hot/humid or cold).

https://www.canada.ca/en/conservation-institute/services/conservation-preservation-publications/canadian-conservation-institute-notes/longevity-recordable-cds-dvds.html

https://www.makeuseof.com/tag/cds-truth-cddvd-longevity-mold-rot/

The difference with digital media is that after a certain amount of errors the data is basically garbage / cannot be decoded, whereas analogue stuff like vinyl or magnetic tape will degrade more and more but still play - the quality/level will get worse but the player doesn't "know" anything is wrong so will still play whatever is there.

[Ed Poore]

I think the only thing that has been proven for long term storage and to be honest is very hard to beat for storage density is magnetic tapes. It can degrade but good environmental storage helps reduce that degradation drastically.

It's not for immediate restoration but if you don't need it recovered quickly then they are hard to beat. A standard LTO tape cartridge will hold 18TB uncompressed (45TB compressed), a stack of DVDs of similar physical space is 153GB. Probably not for everybody but when your backups have to be stored in a secure environment every little space matters.

[pete3000]

4 hours ago, Ed Poore said:

 A standard LTO tape cartridge will hold 18TB uncompressed (45TB compressed), a stack of DVDs of similar physical space is 153GB. Probably not for everybody but when your backups have to be stored in a secure environment every little space matters.

Holy 45TB storage Batman, i guess if you need to store your "special interest" film collection?...... 😎.

[Ed Poore]

On 10/7/2022 at 5:48 PM, pete3000 said:

Holy 45TB storage Batman, i guess if you need to store your "special interest" film collection?...... 😎.

Wasn't film we were storing 

Given we were filling a 24TB raid 0 array in 24h and the only way of getting data quickly off site was to hand carry it in a chopper then storage density was fairly critical.

The hilarious thing was during FAT (Factory Acceptance Testing) the project manager wasn't happy when I told her there wasn't a chance of completing it inside a week as planned. Eventually after criticising me she asked why and I said you forgot about rewinding the tape... One test was fill the tape, error check it, erase it and confirm blank. Fair enough it took almost a day to complete each of the above but she'd completely forgotten the almost 24h it took to rewind the tape in-between each test. It wasn't as if it was slow - sounded like a supercharger whizzing away. They're just very long tapes! 

[FridgeFreezer]

17 hours ago, Ed Poore said:

you forgot about rewinding the tape...

[BogMonster]

On 10/7/2022 at 9:06 AM, Ed Poore said:

I think the only thing that has been proven for long term storage and to be honest is very hard to beat for storage density is magnetic tapes. It can degrade but good environmental storage helps reduce that degradation drastically.

It's not for immediate restoration but if you don't need it recovered quickly then they are hard to beat. A standard LTO tape cartridge will hold 18TB uncompressed (45TB compressed), a stack of DVDs of similar physical space is 153GB. Probably not for everybody but when your backups have to be stored in a secure environment every little space matters.

Seismic data from offshore surveys is stored on magnetic tapes and degradation is an issue over a period of a decade or two, they run the tapes back and forth from time to time to preserve them (my limited understanding is that it's something to do with overprinting from one layer of magnetic tape onto the next) but I know data does get lost particularly if that hasn't been done. The other problem with all old removable media is the significant risk that you suddenly find nobody makes the drives any more....

[Ed Poore]

9 hours ago, BogMonster said:

Seismic data from offshore surveys is stored on magnetic tapes and degradation is an issue over a period of a decade or two, they run the tapes back and forth from time to time to preserve them (my limited understanding is that it's something to do with overprinting from one layer of magnetic tape onto the next) but I know data does get lost particularly if that hasn't been done. The other problem with all old removable media is the significant risk that you suddenly find nobody makes the drives any more....

I do recall that - the difference between magnetic and other forms of media for long term storage is there's a solution for it as you say. CDs, DVDs, flash Drives etc., once they degrade they degrade and there's nothing much you can do about it apart from go into a recovery phase. I.e. it's not prevention.

Not for the average person but if the data is important enough then there'll always be the ability to make a new drive.