In this study the influence of the digital resolution on the properties of a data sample is experimentally determined in mass measurements. A mass comparator with adjustable digital resolution interval, also known as the quantization step size, was used in well controlled repeatable conditions. The same measurement procedure with the resolution differing by up to a factor of 200 was repeatedly carried out, and at least 150 mass differences were recorded with every resolution setting. A clear relationship was observed between the digital resolution and the type of random process characteristics for the data sample. Analysis shows that a white noise process dominates for data sets measured with the smallest digital resolution step from 0.001 mg and up to 0.005 mg. For resolutions from 0.01 mg to 0.2 mg random walk noise is observed for which variance of the sample mean can increase proportionally with sample size. We demonstrate that instrumental resolution is a strictly limiting factor in mass measurements only for the data sets with significant positive correlations, such as those having random walk noise. Otherwise for the white noise process, the smallest possible variance is inversely proportional to averaging time, like in time-frequency metrology, and not limited by the instrumental resolution. Our measurement results show that in this case sample standard deviation of the mean (0.00003 mg) can be more than ten times smaller than that of a single result (0.0005 mg) or the Type B component of the digital resolution (0.00029 mg).
1. Guide to the Expression of Uncertainty in Measurement. BIPM, IEC, IFCC, ISO, IUPAC, IUPAP, OIML, 1993.
2. Allan, D. W. Should the classical variance be used as a basic measure in standards metrology? IEEE T. Instrum. Meas., 1987, IM–36(2), 646–654.
http://dx.doi.org/10.1109/TIM.1987.6312761
3. Riley, W. J. Handbook of Frequency Stability Analysis. Hamilton Technical Services, 2007.
4. IEEE Standard Definitions of Physical Quantities for Fundamental Frequency and Time Metrology – Random Instabilities. IEEE Std 1139-2008, 2009.
5. Witt, T J. Practical methods for treating serial correlations in experimental observations. Eur. Phys. J. Spec. Top., 2009, 172(1), 137–152.
http://dx.doi.org/10.1140/epjst/e2009-01047-1
6. Tang, Y., Solve, S., and Witt, T. J. Allan variance analysis of Josephson voltage standard comparison for data taken at unequal time intervals. IEEE T. Instrum. Meas., 2011, 60(7), 2248–2254.
http://dx.doi.org/10.1109/TIM.2011.2132190
7. Witt, T. J. and Fletcher, N. E. Standard deviation of the mean and other time series properties of voltages measured with a digital lock-in amplifier. Metrologia, 2010, 47(5), 616–630.
http://dx.doi.org/10.1088/0026-1394/47/5/012
8. Witt, T. J. Using the Allan variance and power spectral density to characterize DC nanovoltmeters. IEEE T. Instrum. Meas., 2001, 50(2), 445–448.
http://dx.doi.org/10.1109/19.918162
9. Clare, J. F. and White, D. R. Variance in the mean of a sequence of partially correlated measurements. Metrologia, 1990, 27(4), 193–200.
http://dx.doi.org/10.1088/0026-1394/27/4/003
10. Georgakaki, D., Mitsas, C., and Polatoglou, H. Spectral analysis and Allan variance calculation in the case of phase noise. arXiv:1202.6627, 2012.
11. Georgakaki, D., Mitsas, C., and Polatoglou, H. Time series analysis of the response of measurement instruments. arXiv:1202.6599, 2012.
12. Widrow, B. and Kollár, I. Quantization Noise: Roundoff Error in Digital Computation, Signal Processing, Control, and Communications. Cambridge University Press, Cambridge, UK, 2008.
http://dx.doi.org/10.1017/CBO9780511754661
13. ISO/TC 14253-2:2011, Geometrical Product Specifications (GPS) – Inspection by Measurement of Workpieces and Measuring Equipment. Part 2: Guide to Estimation of Uncertainty in GPS Measurement in Calibration of Measuring Equipment and in Product Verification. ISO, 2011.
14. International Recommendation OIML R 111-1, ed. 2004 (E), Weights of classes E1, E2, F1, F2, M1, M1-2, M2, M2-3, M3, Part 1: Metrological and technical requirements. OIML, Paris, France, 2004.
15. Lira, I. H. and Wöger, W. The evaluation of standard uncertainty in the presence of limited resolution of indicating devices. Meas. Sci. Technol., 1997, 8(4), 441–443.
http://dx.doi.org/10.1088/0957-0233/8/4/012
16. Taraldsen, G. Instrument resolution and measurement accuracy. Metrologia, 2006, 43(6), 539–544.
http://dx.doi.org/10.1088/0026-1394/43/6/009
17. Elster, C. Evaluation of measurement uncertainty in the presence of combined random and analogue-to-digital conversion errors. Meas. Sci. Technol., 2000, 11(9), 1359–1363.
http://dx.doi.org/10.1088/0957-0233/11/9/315
18. Willink, R. On the uncertainty of the mean of digitized measurements. Metrologia, 2007, 44(1), 73–81.
http://dx.doi.org/10.1088/0026-1394/44/1/011
19. Lira, I. Resolution revisited. Metrologia, 2006, 43(3), L14–L17.
http://dx.doi.org/10.1088/0026-1394/43/3/N03
20. Lira, I. Assigning a probability density function for the value of a quantity based on discrete data: the resolution problem. Metrologia, 2012, 49(6), 765.
http://dx.doi.org/10.1088/0026-1394/49/6/765
21. Hannig, J., Iyer, H. K., and Wang, C. M. Fiducial approach to uncertainty assessment accounting for error due to instrument resolution. Metrologia, 2007, 44(6), 476–483.
http://dx.doi.org/10.1088/0026-1394/44/6/006
22. Frenkel, R. B. Fiducial inference applied to uncertainty estimation when identical readings are obtained under low instrument resolution. Metrologia, 2009, 46(6), 661–667.
http://dx.doi.org/10.1088/0026-1394/46/6/008
23. Frenkel, R. B. and Kirkup, L. Monte Carlo-based estimation of uncertainty owing to limited resolution of digital instruments. Metrologia, 2005, 42(5), L27–L30.
http://dx.doi.org/10.1088/0026-1394/42/5/L01
24. Phillips, S. D., Toman, B., and Estler, W. T. Uncertainty due to finite resolution measurements. J. Res. Natl. Inst. Stan., 2008, 113(3), 143.
http://dx.doi.org/10.6028/jres.113.011
25. Burr, T., Hamada, M., Cremers, T., Weaver, B., Howell, J., Croft, S., and Vardeman, S. Measurement error models and variance estimation in the presence of rounding error effects. Accredit. Qual. Assur., 2011, 16(7), 347–359.
http://dx.doi.org/10.1007/s00769-011-0791-0
26. Witkovský, V. and Wimmer, G. Interval estimation of the mean of a normal distribution based on quantized observations. Mathematica Slovaca, 2009, 59(5), 627–645.
http://dx.doi.org/10.2478/s12175-009-0152-1
27. Burr, T., Croft, S., Hamada, M. S., Vardeman, S., and Weaver, B. Rounding error effects in the presence of underlying measurement error. Accred. Qual. Assur., 2012, 17(5), 485–490.
http://dx.doi.org/10.1007/s00769-012-0902-6
28. Riley, W. J. and Greenhall, C. A. Power law noise identification using the lag 1 autocorrelation. In Proc. 18th European Frequency and Time Forum (EFTF), 5–7 Apr. 2004, pp. 576–580.
http://dx.doi.org/10.1049/cp:20040932
29. Durbin, J. and Watson, G. S. Testing for serial correlation in least squares regression. I. Biometrika, 1950, 37(3–4), 409–428.
http://dx.doi.org/10.1093/biomet/37.3-4.409
30. Durbin, J. and Watson, G. S. Testing for serial correlation in least squares regression. II. Biometrika, 1951, 38(1–2), 159–178.
http://dx.doi.org/10.2307/2332325
31. Durbin, J. and Watson, G. S. Testing for serial correlation in least squares regression. III. Biometrika, 1971, 58(1), 1–19.
http://dx.doi.org/10.1093/biomet/58.1.1
32. Cordero, R. R., Seckmeyer, G., and Labbe, F. Effect of the resolution on the uncertainty evaluation. Metrologia, 2006, 43(6), L33–L38.
http://dx.doi.org/10.1088/0026-1394/43/6/N01
33. Predehl, K., Grosche, G., Raupach, S. M. F., Droste, S., Terra, O., Alnis, J., et al. A 920-Kilometer optical fiber link for frequency metrology at the 19th decimal place. Science, 2012, 336(6080), 441–444.
http://dx.doi.org/10.1126/science.1218442
34. Barnes, J. A. Tables of Bias Functions, B1 and B2, for Variances Based on Finite Samples of Processes with Power Law Spectral Densities. NBS Technical Note 375, Jan. 1969.
35. Vabson, V., Kübarsepp, T., Vendt, R., and Noorma, M. Traceability of mass measurements in Estonia. Measurement, 2010, 43(9), 1127–1133.
http://dx.doi.org/10.1016/j.measurement.2010.05.002
http://dx.doi.org/10.3176/proc.2013.2.05
