Analysis of Least Squares Method Efficiency with L-BFGS-B Optimization for Bilinear Autoregression Parameter Estimation Under Various Noise Distributions

Abstract

The article investigates the influence of various distributions of the updating process on the accuracy of parameter estimates for a bilinear autoregressive model, obtained using the least squares method with the L-BFGS-B optimization algorithm. The research examines the effectiveness of parameter estimation using the least squares method for normal, contaminated normal (Tukey distribution), Student’s t-distribution with various degrees of freedom, Laplace distribution, and uniform distribution. Special attention is given to analyzing the dependence of estimation accuracy on the degrees of freedom in the Student t-distribution and contamination parameters in the Tukey distribution. Calculations of errors and biases of the model parameters are conducted based on extensive computer modeling with the generation of stationary time series. The results show that the accuracy of estimates significantly depends on the type of distribution of the updating process, with the highest sensitivity observed for the parameter associated with the nonlinear term of the model. For the Student’s t-distribution, a nonlinear relationship between estimation errors and the number of degrees of freedom is established, while for the contaminated normal distribution, a critical influence of contamination parameters on the accuracy of estimates is revealed. The obtained results can be used in constructing bilinear time series models under conditions of non-standard residual distributions

Please cite this article in English as:

Goryainov V.B., Masyagin M.M., Semernya V.M. Analysis of least squares method efficiency with L-BFGS-B optimization for bilinear autoregression parameter estimation under various noise distributions. Herald of the Bauman Moscow State Technical University, Series Natural Sciences, 2026, no. 3 (126), pp. 4--23 (in Russ.). EDN: VVLPJN

References

[1] Brockwell P.J., Davis R.A. Introduction to time series and forecasting. Springer, 2002.

[2] Hamilton J.D. Time series analysis. Princeton Univ. Press, 1994.

[3] Gao S., Zhou M., Wang Y., et al. Dendritic neuron model with effective learning algorithms for classification, approximation, and prediction. IEEE Trans. Neural Netw. Learn. Syst., 2019, vol. 30, no. 2, pp. 601--614. DOI: https://doi.org/10.1109/TNNLS.2018.2846646

[4] Huber P.J., Ronchetti E.M. Robust statistics. Wiley, 2009.

[5] Fletcher R. Practical methods of optimization. Wiley, 2013.

[6] Granger C.W.J., Andersen A.P. An introduction to bilinear time series models. Vandenhoeck & Ruprecht, 1978.

[7] Brunner A.D., Hess G.D. Potential problems in estimating bilinear time-series models. J. Econ. Dyn. Control, 1995, vol. 19, no. 4, pp. 663--681.

[8] Storn R., Price K. Differential evolution --- a simple and efficient heuristic for global optimization over continuous spaces. J. Glob. Optim., 1997, vol. 11, no. 4, pp. 341--359. DOI: https://doi.org/10.1023/A:1008202821328

[9] Price K.V., Storn R.M., Lampinen J.A. Differential evolution. Springer, 2005.

[10] Liu D.C., Nocedal J. On the limited-memory BFGS method for large scale optimization. Math. Program., 1989, vol. 45, pp. 503--528. DOI: https://doi.org/10.1007/BF01589116

[11] Riani M., Atkinson A.C., Corbellini A., et al. Robust regression with density power divergence: theory, comparisons, and data analysis. Entropy, 2020, vol. 22, no. 4, art. 399. DOI: https://doi.org/10.3390/e22040399

[12] Zhu C., Byrd R.H., Lu P., et al. Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization. ACM Trans. Math. Softw., 1997, vol. 23, no. 4, pp. 550--560. DOI: https://doi.org/10.1145/279232.279236

[13] Agostinelli C., Valdora M., Yohai V.J. Initial robust estimation in generalized linear models. Comput. Stat. Data Anal., 2019, vol. 134, pp. 144--156. DOI: https://doi.org/10.1016/j.csda.2018.12.010

[14] Yao W., Li L. A new regression model: modal linear regression. Scand. J. Stat., 2014, vol. 41, no. 3, pp. 656--671. DOI: https://doi.org/10.1111/sjos.12054

[15] Brockwell P.J., Davis R.A. Time series: theory and methods. Springer, 1996.

[16] Chen Y., Xie Y., Ma X., et al. Probabilistic forecasting with temporal convolutional neural network. Neurocomputing, 2020, vol. 399, pp. 491--501. DOI: https://doi.org/10.1016/j.neucom.2020.03.011

[17] Billard L. Time series models. In: Biometrics. Vol. 1. UNESCO, 1997.

[18] Sesay S.A.O., Subba Rao T. Frequency-domain estimation of bilinear time series models. J. Time Ser. Anal., 1992, vol. 13, no. 6, pp. 521--545. DOI: https://doi.org/10.1111/j.1467-9892.1992.tb00124.x

[19] Goryainov V.B., Goryainova E.R. Robust estimation in threshold autoregression. Herald of the Bauman Moscow State Technical University, Series Natural Sciences, 2017, no. 6 (75), pp. 19--30 (in Russ.). DOI: http://dx.doi.org/10.18698/1812-3368-2017-6-19-30

[20] Zhu D., Galbraith J.W. Modeling and forecasting expected shortfall with the generalized asymmetric Student-t and asymmetric exponential power distributions. J. Empir. Finance, 2011, vol. 18, no. 4, pp. 765--778. DOI: https://doi.org/10.1016/j.jempfin.2011.05.006

[21] Insolia L., Kenney A., Chiaromonte F., et al. Simultaneous feature selection and outlier detection with optimality guarantees. Biometrics, 2022, vol. 78, no. 4, pp. 1592--1603. DOI: https://doi.org/10.1111/biom.13553