Thank you very much, you are awesome!
After your pointing on OR and AND rules I have found it in the documentation (:help vifm-patterns):
To combine several patterns (AND them), make sure you're using one of the
first five forms and write patterns one after another, like this:
<text/plain>{*.vifm}
Mind that if you make a mistake the whole string will be treated as the sixth
form.
|vifm-:filetype|, |vifm-:filextype| and |vifm-:fileviewer| commands accept
comma-separated list of patterns instead of a single pattern, thus effectively
handling OR operation on them: >
<text/plain>{*.vifm},<application/pdf>{*.pdf}
Forms that accept comma-separated lists of patterns also process them as
lists of alternatives.
Now I have two forms of desired configuration:
{*.gz}!{*.tar.gz},
\{*.bz2}!{*.tar.bz2},
\{*.xz}!{*.tar.xz}
" or
/\.(gz|bz2|xz)$/!/\.tar\.(gz|bz2|xz)$/
Both are work well. First one is more readable imho.
To those who still need look behind in POSIX there is verbose workaround, which was worked well until xaizek have answered this question:
/([^r]|[^a]r|[^t]ar|[^\.]tar)\.(gz|bz2|xz)$/
It is convenient to check it with select (select all .gz,.bz2,.xz without .tar before it):
:select /([^r]|[^a]r|[^t]ar|[^\.]tar)\.(gz|bz2|xz)$/