When we count things, the rule it to say “fewer,” and when we measure things, we say “less.” So we have fewer apples, but less distance. Since we measure time, normally we say less.
But you can count time, too, and when that happens, you should use fewer. So the guy in this comic has it wrong.
Just be sure you’re counting units of time, not measuring the time itself.