The title kind of says it all. I think I have the initial steps, which is having a mark set for the folder, and sorting the folder based on modification time, but I currently don't see how I can go the the last modified file instead of having to scroll through around the 30 folders I have there (yes, I should also probably clean up :)) and maybe there's an even faster method.
In general you can use } key to skip to the first file.
In this particular case you might want to ignore entry type on sorting and let files and directories interleave like this:
autocmd DirEnter ~/downloads setlocal sort=-mtime,dir
Nice, I'm starting to really like vifm :)