If you have any important data on your USB device, then yes I would eject via the LOCAL menu in ES File Explorer, or via Settings>Other tab>More Settings>Storage>Scroll down to your device and unmount it there.
It is true you'll probably be OK just removing it, as some people do with a Windows PC, but unless you've no data on the storage device that is critical, no point in taking a risk.