I'm not sure what xterm-color terminfo entry is for and why it's defined this way, but it uses definitions of functional keys that don't match what XTerm is actually sending:
[~]$ infocmp xterm-color xterm-256color | grep 'kf[34]:'
kf3: '\E[13~', '\EOR'.
kf4: '\E[14~', '\EOS'.
[~]$ Ctrl+V F3: ^[OR
[~]$ Ctrl+V F4: ^[OS
Using xterm or xterm-256color should work fine. There may be a configuration option for XTerm which makes it compatible with xterm-color.