Forecast horizon and decision contract
The next 15 minutes of load, tomorrow's consumption and next month's capacity are different tasks. Research defines issue frequency, forecast horizon, update time and intended decision. Performance at one horizon does not automatically establish performance at another. The evaluation dataset must reproduce the information available when each forecast would have been issued.
Underprediction and overprediction may have different consequences, such as insufficient capacity versus excess inventory. This asymmetry informs the acceptance metric and comparison method. The decision process also determines whether a point forecast is sufficient or whether an uncertainty interval is needed. The objective remains an operationally useful forecast rather than an isolated score.
Prevent future information from leaking into features
A production total finalized at the end of the day cannot be an input to a morning forecast. Centered rolling averages, normalization fitted on the full dataset and retrospectively corrected meter readings can inflate test results. Each feature needs an availability time as well as an event time, so backtests do not grant the model knowledge it never had.
Training, tuning and final test periods advance chronologically. Evaluation also states the generalization target: predicting tomorrow at an existing site differs from transferring a model to an unseen site. Device or customer identifiers can unintentionally reveal stable patterns. Claims about both settings require separate experiments with appropriate grouping and withheld data.
Rolling-origin evaluation and reference methods
Evaluation uses repeated forecast origins rather than one random split. A model is trained up to a cutoff, predicts the following window and then advances through time. Last-value and seasonal-reference forecasts run on the same windows. Gains are reported relative to those baselines, together with their variation across periods and operating regimes.
For a hypothetical example, reference MAEs of 8, 10 and 12 versus candidate MAEs of 7, 9 and 11 reduce the equally weighted mean from 10 to 9, a 10% relative improvement. This calculation does not provide a confidence interval or prove transfer to a new site. Window sizes, sample counts and dependence between observations must also be reported.
Metrics, intervals and difficult slices
MAE describes absolute error while RMSE weights large errors more heavily. Percentage errors can become unstable near zero actual values. Peak demand, low-load operation and post-maintenance periods require separate inspection. An improved aggregate metric does not demonstrate better results during every critical interval or for every operating group.
Prediction intervals are evaluated for both empirical coverage and width. An interval wide enough to contain every observation may be operationally useless. Assumptions about dependent time-series data are stated, and observed coverage is not converted into a guarantee about future probability. Critical misses are traced back to the underlying data and operating context.
Reconstruct the original forecast issue time
A historical experiment should reconstruct the database as it was known at issue time, not simply use today's corrected records. Later labels and revisions remain unavailable to the simulated forecast. Scaling and feature selection are fitted only within the training window. Without feature-availability timestamps, this reconstruction can be impossible. Data engineering is therefore a prerequisite for valid modeling; a more complex algorithm cannot compensate for missing historical availability evidence.
Results are tabulated by horizon and period with baseline, candidate, sample count and exclusions. If the candidate wins only in some periods, investigate those conditions rather than hiding them inside one improvement percentage. Retraining decisions use the same experimental design. Applications with delayed outcomes also state monitoring delay; forecasts without observed outcomes cannot yet receive an accuracy score.
| Experiment | Measurement | Interpretation boundary |
|---|---|---|
| Historical data snapshot | Feature availability | Later knowledge excluded |
| Rolling test windows | Error by horizon and period | One period is not generalized |
| Distribution shift | Error after outcomes arrive | Shift is not automatically failure |
Drift, retraining and research evidence
Product mix, sensor replacement and operating schedules can invalidate learned relationships. Input distributions, data latency and realized errors are monitored separately. Distribution shift alone does not justify automatic retraining. Once outcomes become available, performance and likely causes are reviewed, and a replacement model must pass the same acceptance process as its predecessor.
Research outputs can include feature-availability records, versioned experiments, window-based comparisons, error slices and deployment criteria. Gokbilge Tech connects model research to data engineering and operational feedback. The model family, timeline and numerical target require assessment of the actual dataset and decision process rather than a generic promise of accuracy.
Technical references
Define your research problem with us.
Discuss your technical requirements →
