Extremely dangerous is probably an over statement. Your seed phrase is the most important thing. If you’re comfortable storing your seed phrase in the safe you should feel comfortable storing your ledger there too.
Maintaining your own keys, if you’re responsible, should make you feel more comfortable than using a 3rd party service.
If you’re paranoid keeping custody of your seed phrase you could split it up. Store half at your safe at home, half in a safety deposit box etc. you can also encrypt it and place portions of it in google drive/Dropbox/iCloud etc. just make sure you encrypt it locally.
Where you store your ledger is important. The pin you set is way less secure than the seed phrase. (It’s much easier to guess the pin than your mnemonic)
If anything I’d be more worried losing my ledger than an encrypted hard drive with my seed phrase on it.