Floating-point arithmetic is not an efficient way of scaling the ADC values. However, the scaling factor for gain error compensation will be somewhere close to 1, and needs a certain precision to yield good compensated ADC values. Therefore, fixed-point numbers represented by integers can be used.
Since the gain compensation factor certainly never exceeds 2, it can be scaled by a factor of 214 to fit exactly in a signed 16-bit word. In other words, the scaling factor can be represented by two bytes as a 1:14 signed fixed-point number.
The equation for both offset and gain error compensation is shown in Equation 1.
Equation 1:
realvalue = (adcvalue − offset)⋅ gainfactor
When the calculation result is truncated to an ordinary integer later, it is always truncated to the largest integer less than or equal to the result. In order to achieve correct rounding to the nearest integer, 0.5 must be added before truncating. Adding 0.5, scaling the equation by 214 and moving the offset correction outside the parentheses gives Equation 2.
Equation 2:
214⋅realvalue = 214⋅ adcvalue ⋅ gainfactor + 214⋅ 0.5 - 214 ⋅ offset ⋅ gainfactor
Since the gain factor and offset correction value are constants, further optimization can be achieved. In addition, if the result is scaled by 22 , giving a total scale of 216, the upper two bytes of the result equals the truncated integer without the need for a 16-step right-shift.
We introduce some constants and summarize it all in Equation 3.
Equation 3:
factor = 214 ⋅ gainfactor,
correction = 214 ⋅ (0.5 − offset ⋅ gainfactor),
216 ⋅ realvalue = 22 ⋅ ( adcvalue ⋅ factor + correction)
Using this method, the calibration software calculates the constants and stores them in EEPROM memory. At execution time the compensation firmware only needs one integer multiplication, one addition and two left shift operations to correct the ADC values. Using the IAR™ C compiler with highest optimization for speed, this takes 42 CPU cycles.