It's actually good. When you really need to access the device through ssh, you can do this temporary removal, which is back after the reboot. But really for security reasons I wouldn't leave ssh open there for the outside network. Remote ssh forwarding works fine, and limits the session to your own machine.