usually you can’t rmmod a module in use. even if it’s deleted from the FS it’s still present until reboot, but if the cipher module you are using was in that package and it also got removed from the initrd (which often is rebuilt after performing changes like that), you won’t be able to decrypt the LUKS container after reboot unless you boot another system (e.g. a live USB) that has the cipher module you need.