Time series modeling is uniquely challenged by the presence ofautocorrelation in both historical and label sequences. Current researchpredominantly focuses on handling autocorrelation within the historicalsequence but often neglects its presence in the label sequence. Specifically,emerging forecast models mainly conform to the direct forecast (DF) paradigm,generating multi-step forecasts under the assumption of conditionalindependence within the label sequence. This assumption disregards the inherentautocorrelation in the label sequence, thereby limiting the performance ofDF-based models. In response to this gap, we introduce the Frequency-enhancedDirect Forecast (FreDF), which bypasses the complexity of label autocorrelationby learning to forecast in the frequency domain. Our experiments demonstratethat FreDF substantially outperforms existing state-of-the-art methodsincluding iTransformer and is compatible with a variety of forecast models.