You can set g:vifm_exec_args to something like +'set vifm-=dhistory,...' to run one or more custom commands when Vifm is called from Vim.
More extensible approach could be adding +'source $VIFM/invim.vifm' and have a separate file for plugin-specific adjustments next to your vifmrc.