You can adjust 'findprg' option like this:
set findprg='find %s %a -print , -type d \( ! -readable -o ! -executable -o -path "*/.git" \) -prune'
Compared to default value only -o -path "*/.git" is added, which makes find do -prune for paths that match that pattern (the subtree is skipped entirely without descending into it).
See man find for details about the syntax.