eesti teaduste
akadeemia kirjastus
SINCE 1952
Proceeding cover
of the estonian academy of sciences
ISSN 1736-7530 (Electronic)
ISSN 1736-6046 (Print)
Impact Factor (2020): 1.045

Finite resolution and serial correlations in mass metrology; pp. 163–173

Full article in PDF format | doi: 10.3176/proc.2014.2.06

Viktor Vabson, Riho Vendt, Toomas Kübarsepp, Mart Noorma


In this study the influence of the digital resolution on the properties of a data sample is experimentally determined in mass measurements. A mass comparator with adjustable digital resolution interval, also known as the quantization step size, was used in well controlled repeatable conditions. The same measurement procedure with the resolution differing by up to a factor of 200 was repeatedly carried out, and at least 150 mass differences were recorded with every resolution setting. A clear relationship was observed between the digital resolution and the type of random process characteristics for the data sample. Analysis shows that a white noise process dominates for data sets measured with the smallest digital resolution step from 0.001 mg and up to 0.005 mg. For resolutions from 0.01 mg to 0.2 mg random walk noise is observed for which variance of the sample mean can increase proportionally with sample size. We demonstrate that instrumental resolution is a strictly limiting factor in mass measurements only for the data sets with significant positive correlations, such as those having random walk noise. Otherwise for the white noise process, the smallest possible variance is inversely proportional to averaging time, like in time-frequency metrology, and not limited by the instrumental resolution. Our measurement results show that in this case sample standard deviation of the mean (0.00003 mg) can be more than ten times smaller than that of a single result (0.0005 mg) or the Type B component of the digital resolution (0.00029 mg).


  1. Guide to the Expression of Uncertainty in Measurement. BIPM, IEC, IFCC, ISO, IUPAC, IUPAP, OIML, 1993.

  2. Allan, D. W. Should the classical variance be used as a basic measure in standards metrology? IEEE T. Instrum. Meas., 1987, IM–36(2), 646–654.

  3. Riley, W. J. Handbook of Frequency Stability Analysis. Hamilton Technical Services, 2007.

  4. IEEE Standard Definitions of Physical Quantities for Fundamental Frequency and Time Metrology – Random Instabilities. IEEE Std 1139-2008, 2009.

  5. Witt, T J. Practical methods for treating serial correlations in experimental observations. Eur. Phys. J. Spec. Top., 2009, 172(1), 137–152.

  6. Tang, Y., Solve, S., and Witt, T. J. Allan variance analysis of Josephson voltage standard comparison for data taken at unequal time intervals. IEEE T. Instrum. Meas., 2011, 60(7), 2248–2254.

  7. Witt, T. J. and Fletcher, N. E. Standard deviation of the mean and other time series properties of voltages measured with a digital lock-in amplifier. Metrologia, 2010, 47(5), 616–630.

  8. Witt, T. J. Using the Allan variance and power spectral density to characterize DC nanovoltmeters. IEEE T. Instrum. Meas., 2001, 50(2), 445–448.

  9. Clare, J. F. and White, D. R. Variance in the mean of a sequence of partially correlated measurements. Metrologia, 1990, 27(4), 193–200.

10. Georgakaki, D., Mitsas, C., and Polatoglou, H. Spectral analysis and Allan variance calculation in the case of phase noise. arXiv:1202.6627, 2012.

11. Georgakaki, D., Mitsas, C., and Polatoglou, H. Time series analysis of the response of measurement instruments. arXiv:1202.6599, 2012.

12. Widrow, B. and Kollár, I. Quantization Noise: Roundoff Error in Digital Computation, Signal Processing, Control, and Communications. Cambridge University Press, Cambridge, UK, 2008.

13. ISO/TC 14253-2:2011, Geometrical Product Specifications (GPS) – Inspection by Measurement of Workpieces and Measuring Equipment. Part 2: Guide to Estima­tion of Uncertainty in GPS Measurement in Calibra­tion of Measuring Equipment and in Product Verifica­tion. ISO, 2011.

14. International Recommendation OIML R 111-1, ed. 2004 (E), Weights of classes E1, E2, F1, F2, M1, M1-2, M2, M2-3, M3, Part 1: Metrological and technical require­ments. OIML, Paris, France, 2004.

15. Lira, I. H. and Wöger, W. The evaluation of standard uncertainty in the presence of limited resolution of indicating devices. Meas. Sci. Technol., 1997, 8(4), 441–443.

16. Taraldsen, G. Instrument resolution and measurement accuracy. Metrologia, 2006, 43(6), 539–544.

17. Elster, C. Evaluation of measurement uncertainty in the presence of combined random and analogue-to-digital conversion errors. Meas. Sci. Technol., 2000, 11(9), 1359–1363.

18. Willink, R. On the uncertainty of the mean of digitized measurements. Metrologia, 2007, 44(1), 73–81.

19. Lira, I. Resolution revisited. Metrologia, 2006, 43(3), L14–L17.

20. Lira, I. Assigning a probability density function for the value of a quantity based on discrete data: the resolution problem. Metrologia, 2012, 49(6), 765.

21. Hannig, J., Iyer, H. K., and Wang, C. M. Fiducial approach to uncertainty assessment accounting for error due to instrument resolution. Metrologia, 2007, 44(6), 476–483.

22. Frenkel, R. B. Fiducial inference applied to uncertainty estimation when identical readings are obtained under low instrument resolution. Metrologia, 2009, 46(6), 661–667.

23. Frenkel, R. B. and Kirkup, L. Monte Carlo-based estima­tion of uncertainty owing to limited resolution of digital instruments. Metrologia, 2005, 42(5), L27–L30.

24. Phillips, S. D., Toman, B., and Estler, W. T. Uncertainty due to finite resolution measurements. J. Res. Natl. Inst. Stan., 2008, 113(3), 143.

25. Burr, T., Hamada, M., Cremers, T., Weaver, B., Howell, J., Croft, S., and Vardeman, S. Measurement error models and variance estimation in the presence of rounding error effects. Accredit. Qual. Assur., 2011, 16(7), 347–359.

26. Witkovský, V. and Wimmer, G. Interval estimation of the mean of a normal distribution based on quantized observations. Mathematica Slovaca, 2009, 59(5), 627–645.

27. Burr, T., Croft, S., Hamada, M. S., Vardeman, S., and Weaver, B. Rounding error effects in the presence of underlying measurement error. Accred. Qual. Assur., 2012, 17(5), 485–490.

28. Riley, W. J. and Greenhall, C. A. Power law noise identification using the lag 1 autocorrelation. In Proc. 18th European Frequency and Time Forum (EFTF), 5–7 Apr. 2004, pp. 576–580.

29. Durbin, J. and Watson, G. S. Testing for serial correlation in least squares regression. I. Biometrika, 1950, 37(3–4), 409–428.

30. Durbin, J. and Watson, G. S. Testing for serial correlation in least squares regression. II. Biometrika, 1951, 38(1–2), 159–178.

31. Durbin, J. and Watson, G. S. Testing for serial correlation in least squares regression. III. Biometrika, 1971, 58(1), 1–19.

32. Cordero, R. R., Seckmeyer, G., and Labbe, F. Effect of the resolution on the uncertainty evaluation. Metrologia, 2006, 43(6), L33–L38.

33. Predehl, K., Grosche, G., Raupach, S. M. F., Droste, S., Terra, O., Alnis, J., et al. A 920-Kilometer optical fiber link for frequency metrology at the 19th decimal place. Science, 2012, 336(6080), 441–444.

34. Barnes, J. A. Tables of Bias Functions, B1 and B2, for Variances Based on Finite Samples of Processes with Power Law Spectral Densities. NBS Technical Note 375, Jan. 1969.

35. Vabson, V., Kübarsepp, T., Vendt, R., and Noorma, M. Traceability of mass measurements in Estonia. Measure­ment, 2010, 43(9), 1127–1133.

36. Vendt, R., Vabson, V., Kübarsepp, T., and Noorma, M. Traceability of temperature measurements in Estonia. Proc. Estonian Acad. Sci., 2013, 62, 116–121.

Back to Issue