Every Moment Matters and Every Frame Counts