adi_v5_swd: Avoid special handling the SELECT cache during connect