Yes, crypto holders have to worry.
While China did introduce its central bank digital currency / CBDC it banned crypto transactions and crypto exchanges and prohibited banks holding fiat deposits in exchange for crypto coins.
Banks are the gateways from fiat to crypto and back. Government issued CBDCs have the capability to control every transaction and may take control of the fiat-crypto to/from gateways.
While crypto-currencies will continue to exist, cashing out - from crypto to fiat may may become difficult, or like in case of China - impossible.