I would probably use something like this:
:!if [[ $(file --mime %c) =~ 'charset=binary' ]]; then vimdiff -b %b; else vimdiff %f; endif
For this to work, you need appropriate configuration of Vim (this is from sample vimrc or Vim's documentation):
" for handy binary files editing
if &binary
augroup Binary
autocmd!
autocmd BufReadPre * let &bin=1
autocmd BufReadPost * if &bin | %!xxd
autocmd BufReadPost * set ft=xxd | endif
autocmd BufWritePre * if &bin | %!xxd -r
autocmd BufWritePre * endif
autocmd BufWritePost * if &bin | %!xxd
autocmd BufWritePost * set nomod | endif
augroup END
endif