This removes the pmap entry when switching away to e.g. drm fb.
Not strictly necessary to remove the mapping — avoiding conflicts (https://github.com/freebsd/drm-kmod/issues/60) is about avoiding *touching* the memory — but it's still nice to clean up.