Unexpected Value in Data Type (Loss Of Precision)
A data type has a physical range start/end value not expected from combining length and raw encoding.
- Create a new linear computation data type with:
- Byte length: 4 byte (32 Bit),
- Raw encoding: “Unsigned,“
- Physical encoding: „IEEE Float (single)“.
- CANdelaStudio shows a physical range end value of 4294967296 (= 2**32).
- Expected at a first glance: 4294967295 (= 2**32 – 1) due to “unsigned” raw encoding and 4 bytes length.
Use a physical encoding with a resolution high enough to cover all values in the raw encoding range.
In the example above: Use „IEEE Float (double)“ as physical encoding. The correct value 4294967295 is shown.
If the resolution of the selected physical encoding is not high enough to resolve the range of possible raw values, raw values must be rounded and an information loss occurs.
In the example above, the physical encoding “IEEE Float (single)” has a reduced precision (8 decimal places). The physical range end value expected by the raw encoding and data type length is 4294967295 (= 2**32 - 1). On conversion to a float with 8 decimal places, it is rounded to 0.42949673 * 1e10 → information is lost.
The physical encoding “IEEE Float (single)” has the same byte length as the data type (4 byte), but some bits must be used for the floating point mantissa. It can thus not cover the full value range of the 4 byte unsigned raw encoding.
Note that factor, divisor and offset of the conversion formula do also affect the computation resolution in principle. But here they are set to 1, 1, and 0 and do not have any impact.