ESTONIAN ACADEMY
PUBLISHERS
eesti teaduste
akadeemia kirjastus
PUBLISHED
SINCE 1952
 
Proceeding cover
proceedings
of the estonian academy of sciences
ISSN 1736-7530 (Electronic)
ISSN 1736-6046 (Print)
Impact Factor (2022): 0.9
Finite resolution and serial correlations in mass metrology; pp. 163–173
PDF | doi: 10.3176/proc.2014.2.06

Authors
Viktor Vabson, Riho Vendt, Toomas Kübarsepp, Mart Noorma
Abstract

In this study the influence of the digital resolution on the properties of a data sample is experimentally determined in mass measurements. A mass comparator with adjustable digital resolution interval, also known as the quantization step size, was used in well controlled repeatable conditions. The same measurement procedure with the resolution differing by up to a factor of 200 was repeatedly carried out, and at least 150 mass differences were recorded with every resolution setting. A clear relationship was observed between the digital resolution and the type of random process characteristics for the data sample. Analysis shows that a white noise process dominates for data sets measured with the smallest digital resolution step from 0.001 mg and up to 0.005 mg. For resolutions from 0.01 mg to 0.2 mg random walk noise is observed for which variance of the sample mean can increase proportionally with sample size. We demonstrate that instrumental resolution is a strictly limiting factor in mass measurements only for the data sets with significant positive correlations, such as those having random walk noise. Otherwise for the white noise process, the smallest possible variance is inversely proportional to averaging time, like in time-frequency metrology, and not limited by the instrumental resolution. Our measurement results show that in this case sample standard deviation of the mean (0.00003 mg) can be more than ten times smaller than that of a single result (0.0005 mg) or the Type B component of the digital resolution (0.00029 mg).

References

  1. Guide to the Expression of Uncertainty in Measurement. BIPM, IEC, IFCC, ISO, IUPAC, IUPAP, OIML, 1993.

  2. Allan, D. W. Should the classical variance be used as a basic measure in standards metrology? IEEE T. Instrum. Meas., 1987, IM–36(2), 646–654.
http://dx.doi.org/10.1109/TIM.1987.6312761

  3. Riley, W. J. Handbook of Frequency Stability Analysis. Hamilton Technical Services, 2007.

  4. IEEE Standard Definitions of Physical Quantities for Fundamental Frequency and Time Metrology – Random Instabilities. IEEE Std 1139-2008, 2009.

  5. Witt, T J. Practical methods for treating serial correlations in experimental observations. Eur. Phys. J. Spec. Top., 2009, 172(1), 137–152.
http://dx.doi.org/10.1140/epjst/e2009-01047-1

  6. Tang, Y., Solve, S., and Witt, T. J. Allan variance analysis of Josephson voltage standard comparison for data taken at unequal time intervals. IEEE T. Instrum. Meas., 2011, 60(7), 2248–2254.
http://dx.doi.org/10.1109/TIM.2011.2132190

  7. Witt, T. J. and Fletcher, N. E. Standard deviation of the mean and other time series properties of voltages measured with a digital lock-in amplifier. Metrologia, 2010, 47(5), 616–630.
http://dx.doi.org/10.1088/0026-1394/47/5/012

  8. Witt, T. J. Using the Allan variance and power spectral density to characterize DC nanovoltmeters. IEEE T. Instrum. Meas., 2001, 50(2), 445–448.
http://dx.doi.org/10.1109/19.918162

  9. Clare, J. F. and White, D. R. Variance in the mean of a sequence of partially correlated measurements. Metrologia, 1990, 27(4), 193–200.
http://dx.doi.org/10.1088/0026-1394/27/4/003

10. Georgakaki, D., Mitsas, C., and Polatoglou, H. Spectral analysis and Allan variance calculation in the case of phase noise. arXiv:1202.6627, 2012.

11. Georgakaki, D., Mitsas, C., and Polatoglou, H. Time series analysis of the response of measurement instruments. arXiv:1202.6599, 2012.

12. Widrow, B. and Kollár, I. Quantization Noise: Roundoff Error in Digital Computation, Signal Processing, Control, and Communications. Cambridge University Press, Cambridge, UK, 2008.
http://dx.doi.org/10.1017/CBO9780511754661

13. ISO/TC 14253-2:2011, Geometrical Product Specifications (GPS) – Inspection by Measurement of Workpieces and Measuring Equipment. Part 2: Guide to Estima­tion of Uncertainty in GPS Measurement in Calibra­tion of Measuring Equipment and in Product Verifica­tion. ISO, 2011.

14. International Recommendation OIML R 111-1, ed. 2004 (E), Weights of classes E1, E2, F1, F2, M1, M1-2, M2, M2-3, M3, Part 1: Metrological and technical require­ments. OIML, Paris, France, 2004.

15. Lira, I. H. and Wöger, W. The evaluation of standard uncertainty in the presence of limited resolution of indicating devices. Meas. Sci. Technol., 1997, 8(4), 441–443.
http://dx.doi.org/10.1088/0957-0233/8/4/012

16. Taraldsen, G. Instrument resolution and measurement accuracy. Metrologia, 2006, 43(6), 539–544.
http://dx.doi.org/10.1088/0026-1394/43/6/009

17. Elster, C. Evaluation of measurement uncertainty in the presence of combined random and analogue-to-digital conversion errors. Meas. Sci. Technol., 2000, 11(9), 1359–1363.
http://dx.doi.org/10.1088/0957-0233/11/9/315

18. Willink, R. On the uncertainty of the mean of digitized measurements. Metrologia, 2007, 44(1), 73–81.
http://dx.doi.org/10.1088/0026-1394/44/1/011

19. Lira, I. Resolution revisited. Metrologia, 2006, 43(3), L14–L17.
http://dx.doi.org/10.1088/0026-1394/43/3/N03

20. Lira, I. Assigning a probability density function for the value of a quantity based on discrete data: the resolution problem. Metrologia, 2012, 49(6), 765.
http://dx.doi.org/10.1088/0026-1394/49/6/765

21. Hannig, J., Iyer, H. K., and Wang, C. M. Fiducial approach to uncertainty assessment accounting for error due to instrument resolution. Metrologia, 2007, 44(6), 476–483.
http://dx.doi.org/10.1088/0026-1394/44/6/006

22. Frenkel, R. B. Fiducial inference applied to uncertainty estimation when identical readings are obtained under low instrument resolution. Metrologia, 2009, 46(6), 661–667.
http://dx.doi.org/10.1088/0026-1394/46/6/008

23. Frenkel, R. B. and Kirkup, L. Monte Carlo-based estima­tion of uncertainty owing to limited resolution of digital instruments. Metrologia, 2005, 42(5), L27–L30.
http://dx.doi.org/10.1088/0026-1394/42/5/L01

24. Phillips, S. D., Toman, B., and Estler, W. T. Uncertainty due to finite resolution measurements. J. Res. Natl. Inst. Stan., 2008, 113(3), 143.
http://dx.doi.org/10.6028/jres.113.011

25. Burr, T., Hamada, M., Cremers, T., Weaver, B., Howell, J., Croft, S., and Vardeman, S. Measurement error models and variance estimation in the presence of rounding error effects. Accredit. Qual. Assur., 2011, 16(7), 347–359.
http://dx.doi.org/10.1007/s00769-011-0791-0

26. Witkovský, V. and Wimmer, G. Interval estimation of the mean of a normal distribution based on quantized observations. Mathematica Slovaca, 2009, 59(5), 627–645.
http://dx.doi.org/10.2478/s12175-009-0152-1

27. Burr, T., Croft, S., Hamada, M. S., Vardeman, S., and Weaver, B. Rounding error effects in the presence of underlying measurement error. Accred. Qual. Assur., 2012, 17(5), 485–490.
http://dx.doi.org/10.1007/s00769-012-0902-6

28. Riley, W. J. and Greenhall, C. A. Power law noise identification using the lag 1 autocorrelation. In Proc. 18th European Frequency and Time Forum (EFTF), 5–7 Apr. 2004, pp. 576–580.
http://dx.doi.org/10.1049/cp:20040932

29. Durbin, J. and Watson, G. S. Testing for serial correlation in least squares regression. I. Biometrika, 1950, 37(3–4), 409–428.
http://dx.doi.org/10.1093/biomet/37.3-4.409

30. Durbin, J. and Watson, G. S. Testing for serial correlation in least squares regression. II. Biometrika, 1951, 38(1–2), 159–178.
http://dx.doi.org/10.2307/2332325

31. Durbin, J. and Watson, G. S. Testing for serial correlation in least squares regression. III. Biometrika, 1971, 58(1), 1–19.
http://dx.doi.org/10.1093/biomet/58.1.1

32. Cordero, R. R., Seckmeyer, G., and Labbe, F. Effect of the resolution on the uncertainty evaluation. Metrologia, 2006, 43(6), L33–L38.
http://dx.doi.org/10.1088/0026-1394/43/6/N01

33. Predehl, K., Grosche, G., Raupach, S. M. F., Droste, S., Terra, O., Alnis, J., et al. A 920-Kilometer optical fiber link for frequency metrology at the 19th decimal place. Science, 2012, 336(6080), 441–444.
http://dx.doi.org/10.1126/science.1218442

34. Barnes, J. A. Tables of Bias Functions, B1 and B2, for Variances Based on Finite Samples of Processes with Power Law Spectral Densities. NBS Technical Note 375, Jan. 1969.

35. Vabson, V., Kübarsepp, T., Vendt, R., and Noorma, M. Traceability of mass measurements in Estonia. Measure­ment, 2010, 43(9), 1127–1133.
http://dx.doi.org/10.1016/j.measurement.2010.05.002

36. Vendt, R., Vabson, V., Kübarsepp, T., and Noorma, M. Traceability of temperature measurements in Estonia. Proc. Estonian Acad. Sci., 2013, 62, 116–121.
http://dx.doi.org/10.3176/proc.2013.2.05

Back to Issue