vector.com|Contact
Search tips

apple banana
Find rows that contain at least one of the two words.

+apple +juice
Find rows that contain both words.

+apple macintosh
Find rows that contain the word 'apple', but rank rows higher if they also contain 'macintosh'.

+apple -macintosh
Find rows that contain the word 'apple' but not 'macintosh'.

+apple ~macintosh
Find rows that contain the word 'apple', but if the row also contains the word 'macintosh', rate it lower than if row does not. This is "softer" than a search for '+apple -macintosh', for which the presence of 'macintosh' causes the row not to be returned at all.

+apple +(>turnover >strudel)
Find rows that contain the words 'apple' and 'turnover', or 'apple' and 'strudel' (in any order), but rank 'apple turnover' higher than 'apple strudel'.

apple*
Find rows that contain words such as 'apple', 'apples', 'applesauce', or 'applet'.

"some words"
Find rows that contain the exact phrase 'some words' (for example, rows that contain 'some words of wisdom' but not "some noise words").

By continuing to use this site you agree to the use of cookies. For more information and to find out how to change this click here. Accept Cookies
Please enable cookies in your browser for this website.
Search:
Advanced search

Unexpected Value in Data Type (Loss Of Precision)

Last updated: 2018-03-28
Issue:

A data type has a physical range start/end value not expected from combining length and raw encoding.

Example:

  • Create a new linear computation data type with:
    • Byte length: 4 byte (32 Bit),
    • Raw encoding: “Unsigned,“
    • Physical encoding: „IEEE Float (single)“.
  • CANdelaStudio shows a physical range end value of 4294967296 (= 2**32).
  • Expected at a first glance: 4294967295 (= 2**32 – 1) due to “unsigned” raw encoding and 4 bytes length.

Solution:

Use a physical encoding with a resolution high enough to cover all values in the raw encoding range.


In the example above: Use „IEEE Float (double)“ as physical encoding. The correct value 4294967295 is shown.

Background:

If the resolution of the selected physical encoding is not high enough to resolve the range of possible raw values, raw values must be rounded and an information loss occurs.

In the example above, the physical encoding “IEEE Float (single)” has a reduced precision (8 decimal places). The physical range end value expected by the raw encoding and data type length is 4294967295 (= 2**32 - 1). On conversion to a float with 8 decimal places, it is rounded to 0.42949673 * 1e10 → information is lost.
The physical encoding “IEEE Float (single)” has the same byte length as the data type (4 byte), but some bits must be used for the floating point mantissa. It can thus not cover the full value range of the 4 byte unsigned raw encoding.

Note that factor, divisor and offset of the conversion formula do also affect the computation resolution in principle. But here they are set to 1, 1, and 0 and do not have any impact.

 
Article Options
2018-03-28
Views: 1743
Rate this article: