1. Put host in maintenance mode.
2. Remove the host from distributed switch
3. Right-click the appropriate host in the inventory pane, and select Remove from Inventory from the pop-up menu

Step 2 might cause issues. Unable to Remove a Host from a vSphere Distributed Switch (vmware.com) The resource ‘Port-ID’ is in use error when removing a host from VDS (2015435) (vmware.com) Better to just Right click the ESXi host from the Inventory and select Connection > Disconnect And after host is disconnected remove from inventory will remove it.

references: Remove a Host from vCenter Server (vmware.com)