Starting from version 4.59.2 URE

* Update: Option “Force custom post types to use their own capabilities” replaces default capabilities for the custom taxonomies also.
It takes the slug of the 1st post type associated with such taxonomy (e.g. ‘video’) and builds own capabilities this way: manage_terms->manage_video_terms, edit_terms->edit_video_terms, delete_terms->delete_video_terms, assign_terms->assign_video_terms.

So search between custom capabilities similar to the named above, but with your custom post type inside and grant them to your role.