Function swap_pager_swapoff_object calls vm_pager_unswapped (via swp_pager_force_dirty) for every page that must be unswapped. That means that there's an unneeded check for lock ownership (the caller always owns it), a needless PCTRIE_LOOKUP (the caller has already found it), a call to free one page of swap space only, and a check to see if all blocks are empty, when the caller knows whether that check is useless.
Isolate the essential part, needed however swap_pager_unswapped is invoked, into a smaller function swap_pager_unswapped_acct (suggestions for a better name are welcome). From swapoff_object, invoke swp_pager_update_freerange for each appropriate page, so that there are potentially fewer calls to swp_pager_freeswapspace. Avoid re-fetching swblks and re-scanning them in cases where the object lock has been held since the last fetch and scan.