You can adjust 'findprg' option like this:
set findprg='find %s %a -print , -type d \( ! -readable -o ! -executable -o -path "*/.git" \) -prune'
Compared to default value only -o -path "*/.git"
is added, which makes find
do -prune
for paths that match that pattern (the subtree is skipped entirely without descending into it).
See man find
for details about the syntax.