Both conditions are necessary since input signal rise/fall time affects the delay as well as output loading.
FO4 is generally used as a delay metric because such a load is generally seen in case of tapered buffers driving large loads, and approximately in any logic gate of a logic path sized for minimum delay.
[1] A fan out of 4 is the answer to the canonical problem stated as follows: Given a fixed size inverter, small in comparison to a fixed large load, minimize the delay in driving the large load.
If the load itself is not large, then using a fan out of 4 scaling in successive logic stages does not make sense.
Because scaled technologies are inherently faster (in absolute terms), circuit performance can be more fairly compared using the fan out of 4 as a metric.