[cl-json-devel] Encoding DOUBLE-FLOAT
Hi everybody, I flipped through the archives but did not find that question: I get some numbers (DOUBLE-FLOAT) from CLSQL, e.g. 0.0570714084012434D0 If I encode this to JSON I get a "0.0570714084012434D0" but it seems that the notation with "D" is not valid JSON because when I try to decode this string say from R with rjson I get an error. Any help or suggestions? Kind regards, Daniel.
On Wed, Jun 22, 2011 at 8:23 AM, Daniel Brunner <daniel@dbrunner.de> wrote:
I get some numbers (DOUBLE-FLOAT) from CLSQL, e.g.
0.0570714084012434D0
If I encode this to JSON I get a "0.0570714084012434D0" but it seems that the notation with "D" is not valid JSON because when I try to decode this string say from R with rjson I get an error.
This is very strange. I get: (json:encode-json-to-string 0.0570714084012434D0) ⇒ "0.0570714084012434" How exactly do you get your output with D? Yours, - B. Sm.
Hm. I am using exact the same line of code: CL-USER> (json:encode-json-to-string 0.0570714084012434D0) "0.0570714084012434D0" I have cl-json 0.4.0 on Cluzure Common Lisp "Version 1.7-dev-r14406M-trunk (LinuxX8664)" Kind regards, Daniel Am 22.06.2011 11:03, schrieb Boris Smilga:
On Wed, Jun 22, 2011 at 8:23 AM, Daniel Brunner <daniel@dbrunner.de> wrote:
I get some numbers (DOUBLE-FLOAT) from CLSQL, e.g.
0.0570714084012434D0
If I encode this to JSON I get a "0.0570714084012434D0" but it seems that the notation with "D" is not valid JSON because when I try to decode this string say from R with rjson I get an error.
This is very strange. I get:
(json:encode-json-to-string 0.0570714084012434D0) ⇒ "0.0570714084012434"
How exactly do you get your output with D?
Yours, - B. Sm.
On 6/22/11 Jun 22 -11:59 AM, Daniel Brunner wrote:
(json:encode-json-to-string 0.0570714084012434D0)
I have replicated this error.... The bug is here: (defun write-json-number (nr stream) "Write the JSON representation of the number NR to STREAM." (typecase nr (integer (format stream "~d" nr)) (real (format stream "~f" nr)) (t (unencodable-value-error nr 'write-json-number)))) As can be seen by CL-USER> (format t "~f" 0.0570714084012434D0) 0.0570714084012434D0 NIL This suggests that we must avoid using FORMAT in translating floats to strings for JSON, or that there's a recipe for getting FORMAT to shun the D that I don't know about.... Not incredibly well thought out or tested patch: (defun write-json-number (nr stream) "Write the JSON representation of the number NR to STREAM." (typecase nr (integer (format stream "~d" nr)) (real (let* ((raw (format nil "~f" nr)) (dpos (position #\d raw :test 'char-equal))) (princ (subseq raw 0 dpos) stream))) (t (unencodable-value-error nr 'write-json-number)))) This checks for D in the output. I /believe/ that this is the only such character that can creep in, but I am far from being an expert on this. Best, r
I have only barely remembered how to use darcs. Here's a patch file that captures the mod in the last email. R
On 23 Jun 2011, at 01:10, Robert Goldman wrote:
On 6/22/11 Jun 22 -11:59 AM, Daniel Brunner wrote:
(json:encode-json-to-string 0.0570714084012434D0)
I have replicated this error....
The bug is here:
(defun write-json-number (nr stream) "Write the JSON representation of the number NR to STREAM." (typecase nr (integer (format stream "~d" nr)) (real (format stream "~f" nr)) (t (unencodable-value-error nr 'write-json-number))))
As can be seen by
CL-USER> (format t "~f" 0.0570714084012434D0) 0.0570714084012434D0 NIL
I have located it, too. Actually, it's a confusing matter bearing on the interpretation of one passage in the Common Lisp standard. To wit, the standard behaviour of the ~F formatting directive (CLHS, sec. 22.3.3.1) is to print ‘a sequence of digits, containing a single embedded decimal point’. Exponent markers are never mentioned in that section. However, a little further below it tells us that ‘If both w and d [prefix parameters of the directive — B. Sm.] are omitted, then the effect is to print the value using ordinary free- format output; prin1 uses this format for any number whose magnitude is either zero or between 10^-3 (inclusive) and 10^7 (exclusive).’ Now the problem is that the expression ‘ordinary free-format output’ has no well-defined meaning in the standard. One reasonable interpretation is that it means the same as ‘output produced by print- object’. In this case, the implementer is bound by sec. 22.1.3.1.3: ‘If the format of the number does not match that specified by *read- default-float-format*, then the exponent marker for that format and the digit 0 are also printed.’ If we adopt this reading, then CCL does the right thing, and SBCL (which prints no exponent marker) is in violation of the standard. However, if we opt for a different interpretation (what?), then SBCL might end up compliant after all, and CCL might or might not be in violation.
This suggests that we must avoid using FORMAT in translating floats to strings for JSON, or that there's a recipe for getting FORMAT to shun the D that I don't know about....
I would advocate a different approach: to check the type of the float, and to bind *read-default-float-format* to the corresponding format designator: (defun write-json-number (nr stream) "Write the JSON representation of the number NR to STREAM." (typecase nr (integer (format stream "~d" nr)) (real (let ((*read-default-float-format* (typecase nr (long-float 'long-float) (double-float 'double-float) (short-float 'short-float) (t 'single-float)))) (format stream "~f" nr))) (t (unencodable-value-error nr 'write-json-number))))
This checks for D in the output. I /believe/ that this is the only such character that can creep in, but I am far from being an expert on this.
There are also F, L, and S (CLHS, sec. 2.3.2.2). Yours, - B. Sm.
On 6/22/11 Jun 22 -6:25 PM, Boris Smilga wrote:
(defun write-json-number (nr stream) "Write the JSON representation of the number NR to STREAM." (typecase nr (integer (format stream "~d" nr)) (real (let ((*read-default-float-format* (typecase nr (long-float 'long-float) (double-float 'double-float) (short-float 'short-float) (t 'single-float)))) (format stream "~f" nr))) (t (unencodable-value-error nr 'write-json-number))))
I agree that this is better, but when I compile this in Clozure on a 64 bit Intel Mac, I get this warning: ;Compiler warnings for "/Users/rpg/lisp/cl-json/src/encoder.lisp" : ; In WRITE-JSON-NUMBER: Clause (DOUBLE-FLOAT 'DOUBLE-FLOAT) ignored in TYPECASE form - shadowed by (LONG-FLOAT 'LONG-FLOAT) Would it be better to rewrite this from shortest to longest as: (defun write-json-number (nr stream) "Write the JSON representation of the number NR to STREAM." (typecase nr (integer (format stream "~d" nr)) (real (let ((*read-default-float-format* (etypecase nr (short-float 'short-float) (single-float 'single-float) (double-float 'double-float) (long-float 'long-float)))) (format stream "~f" nr))) (t (unencodable-value-error nr 'write-json-number)))) This still yields warnings on Clozure, because SHORT-FLOAT == SINGLE-FLOAT and DOUBLE-FLOAT == LONG-FLOAT (IIUC), but I think that's implementation-dependent, so we can't do away with the intermediates in general, can we? Not sure how to muffle these warnings, or even if it's desirable, since the Clozure implementers might split apart single and short and double and long at some point, right? Best, r
On Thu, Jun 23, 2011 at 1:04 AM, Robert Goldman <rpgoldman@sift.info> wrote:
Would it be better to rewrite this from shortest to longest as:
(defun write-json-number (nr stream) "Write the JSON representation of the number NR to STREAM." (typecase nr (integer (format stream "~d" nr)) (real (let ((*read-default-float-format* (etypecase nr (short-float 'short-float) (single-float 'single-float) (double-float 'double-float) (long-float 'long-float)))) (format stream "~f" nr))) (t (unencodable-value-error nr 'write-json-number))))
What about rationals? They too are a subtype of REAL, but your etypecase doesn't take them into account.
This still yields warnings on Clozure, because SHORT-FLOAT == SINGLE-FLOAT and DOUBLE-FLOAT == LONG-FLOAT (IIUC), but I think that's implementation-dependent, so we can't do away with the intermediates in general, can we?
Not sure how to muffle these warnings, or even if it's desirable, since the Clozure implementers might split apart single and short and double and long at some point, right?
In principle, yes (although in practive such design choices tend to be very stable, undergoing changes only when radically different hardware architectures become available). If these warnings bother you so much, you might just use #-CCL et al. to exclude the TYPECASE clauses that cause them. Yours, - B. Sm.
On 6/23/11 Jun 23 -7:05 AM, Boris Smilga wrote:
On Thu, Jun 23, 2011 at 1:04 AM, Robert Goldman <rpgoldman@sift.info> wrote:
Would it be better to rewrite this from shortest to longest as:
(defun write-json-number (nr stream) "Write the JSON representation of the number NR to STREAM." (typecase nr (integer (format stream "~d" nr)) (real (let ((*read-default-float-format* (etypecase nr (short-float 'short-float) (single-float 'single-float) (double-float 'double-float) (long-float 'long-float)))) (format stream "~f" nr))) (t (unencodable-value-error nr 'write-json-number))))
What about rationals? They too are a subtype of REAL, but your etypecase doesn't take them into account.
Should we check for rationals before we check for reals (i.e., check for integer, rational, real, and then have the error case)? And presumably we should print rationals as if they are floats, correct? So we would have: (defun write-json-number (nr stream) "Write the JSON representation of the number NR to STREAM." (typecase nr (integer (format stream "~d" nr)) (rational (format stream "~f" nr)) (real (let ((*read-default-float-format* (etypecase nr (short-float 'short-float) (single-float 'single-float) (double-float 'double-float) (long-float 'long-float)))) (format stream "~f" nr))) (t (unencodable-value-error nr 'write-json-number))))
This still yields warnings on Clozure, because SHORT-FLOAT == SINGLE-FLOAT and DOUBLE-FLOAT == LONG-FLOAT (IIUC), but I think that's implementation-dependent, so we can't do away with the intermediates in general, can we?
Not sure how to muffle these warnings, or even if it's desirable, since the Clozure implementers might split apart single and short and double and long at some point, right?
In principle, yes (although in practive such design choices tend to be very stable, undergoing changes only when radically different hardware architectures become available). If these warnings bother you so much, you might just use #-CCL et al. to exclude the TYPECASE clauses that cause them.
I think in principle we should try to detect the relationship between the different float subtypes. Presumably we can do this using MOST-POSITIVE-SHORT-FLOAT, MOST-POSITIVE-SINGLE-FLOAT, etc.., etc., etc., and that would be portable... If I get a chance, I will try to do that: looks like it will be a little cumbersome. best, r
On Thu, Jun 23, 2011 at 2:47 PM, Robert Goldman <rpgoldman@sift.info> wrote:
On 6/23/11 Jun 23 -7:05 AM, Boris Smilga wrote:
What about rationals? They too are a subtype of REAL, but your etypecase doesn't take them into account.
Should we check for rationals before we check for reals (i.e., check for integer, rational, real, and then have the error case)? And presumably we should print rationals as if they are floats, correct?
Certainly. Moreover, per CLHS sec. 22.3.3.1, to format a rational with ~F, it is coerced to be a single float. If *read-default-float-format* is bound to something else, we'll get an S marker: (let ((*read-default-float-format* 'double-float)) (format nil "~F" 2/3)) ⇒ "0.6666667S0" ; Actually so in CCL. As we have no way to predict what *read-default-float-format* will be when the user calls encode-json, the invocation of format in the rational clause of typecase has to explicitly bind *read-default-float-format* to 'single-float: (defun write-json-number (nr stream) "Write the JSON representation of the number NR to STREAM." (typecase nr (integer (format stream "~d" nr)) (rational (let ((*read-default-float-format* 'single-float)) (format stream "~f" nr))) (real (let ((*read-default-float-format* (etypecase nr (short-float 'short-float) (single-float 'single-float) (double-float 'double-float) (long-float 'long-float)))) (format stream "~f" nr))) (t (unencodable-value-error nr 'write-json-number)))) Since rational is a subtype of real, we can factor the clause down into etypecase, to make the code more concise: (defun write-json-number (nr stream) "Write the JSON representation of the number NR to STREAM." (typecase nr (integer (format stream "~d" nr)) (real (let ((*read-default-float-format* (etypecase nr (short-float 'short-float) ((single-float rational) 'single-float) (double-float 'double-float) (long-float 'long-float)))) (format stream "~f" nr))) (t (unencodable-value-error nr 'write-json-number)))) This is largely the same thing which I had proposed three messages up the thread. The order of clauses doesn't matter much, because, per the definition of the system class float in CLHS sec. 12.2, ‘any two of [the types short-float, single-float, double-float, and long-float] must be either disjoint types or the same type’.
I think in principle we should try to detect the relationship between the different float subtypes. Presumably we can do this using MOST-POSITIVE-SHORT-FLOAT, MOST-POSITIVE-SINGLE-FLOAT, etc.., etc., etc., and that would be portable... If I get a chance, I will try to do that: looks like it will be a little cumbersome.
Hmmm... I'm afraid I don't really get your idea. The relationship between different float types is laid down quite explicitly in the Standard (should it be called ‘the Most Holy and Exalted Standard’?). Checking the limits won't give us anything, as the format of the number is orthogonal to its magnitude: 3.0D0 is a double, not a single float (in implementations which have distinct double and single floats), even though its numeric value falls inside the interval between least-positive-single-float and most-positive-single-float, and can be represented by a single float (without loss of precision). Pray, what exactly are you trying to accomplish here? Yours, - B. Sm.
On 6/23/11 Jun 23 -5:15 PM, Boris Smilga wrote:
On Thu, Jun 23, 2011 at 2:47 PM, Robert Goldman <rpgoldman@sift.info> wrote:
On 6/23/11 Jun 23 -7:05 AM, Boris Smilga wrote: .....
I think in principle we should try to detect the relationship between the different float subtypes. Presumably we can do this using MOST-POSITIVE-SHORT-FLOAT, MOST-POSITIVE-SINGLE-FLOAT, etc.., etc., etc., and that would be portable... If I get a chance, I will try to do that: looks like it will be a little cumbersome.
Hmmm... I'm afraid I don't really get your idea. The relationship between different float types is laid down quite explicitly in the Standard (should it be called ‘the Most Holy and Exalted Standard’?). Checking the limits won't give us anything, as the format of the number is orthogonal to its magnitude: 3.0D0 is a double, not a single float (in implementations which have distinct double and single floats), even though its numeric value falls inside the interval between least-positive-single-float and most-positive-single-float, and can be represented by a single float (without loss of precision). Pray, what exactly are you trying to accomplish here?
I was going to compare, e.g., MOST-POSITIVE-SHORT-FLOAT, LEAST-NEGATIVE-SHORT-FLOAT and MOST-POSITIVE-SINGLE-FLOAT, LEAST-NEGATIVE-SINGLE-FLOAT (at compile time) to attempt to determine if SHORT-FLOAT and SINGLE-FLOAT are the same on the current implementation and possibly conditionally-compile away the SINGLE-FLOAT type, and similarly for DOUBLE- and LONG-.... I would not be checking the numeric values of the number at run-time, but the bounds on the types at compile time. Would that not work? Thanks for the help, r
On Fri, Jun 24, 2011 at 2:23 AM, Robert Goldman <rpgoldman@sift.info> wrote:
I was going to compare, e.g., MOST-POSITIVE-SHORT-FLOAT, LEAST-NEGATIVE-SHORT-FLOAT and MOST-POSITIVE-SINGLE-FLOAT, LEAST-NEGATIVE-SINGLE-FLOAT (at compile time) to attempt to determine if SHORT-FLOAT and SINGLE-FLOAT are the same on the current implementation and possibly conditionally-compile away the SINGLE-FLOAT type, and similarly for DOUBLE- and LONG-....
I would not be checking the numeric values of the number at run-time, but the bounds on the types at compile time. Would that not work?
Ah, yes, now I see. But then you might just write it straightforwardly as (subtypep 'short-float 'single-float), or vice versa. Since subtypes of float are either disjoint or synonymous, subtypep in this context effectively tests the types for equality. Still, I doubt the intended effect is worth the trouble. Having superfluous clauses in typecase is explicitly allowed: ‘[...] is permissible for more than one clause to specify a matching type’ (CLHS 5.3, definition of TYPECASE et al.) In fact, by trying to get rid of them beforehand we would be (re)doing the compiler's job. E. g., CCL eliminates them right there at macro expansion time: (macroexpand '(typecase foo (short-float #\S) (long-float #\L) (single-float #\F) (double-float #\D))) ⇒ (LET ((#:G4 FOO)) (DECLARE (IGNORABLE #:G4)) (COND ((TYPEP #:G4 'SHORT-FLOAT) NIL #\S) ((TYPEP #:G4 'LONG-FLOAT) NIL #\L))) Are we going to such lengths just in order to get rid of annoying warnings in a single implementation? Yours, - B. Sm.
On 6/25/11 Jun 25 -1:24 AM, Boris Smilga wrote:
On Fri, Jun 24, 2011 at 2:23 AM, Robert Goldman <rpgoldman@sift.info> wrote:
I was going to compare, e.g., MOST-POSITIVE-SHORT-FLOAT, LEAST-NEGATIVE-SHORT-FLOAT and MOST-POSITIVE-SINGLE-FLOAT, LEAST-NEGATIVE-SINGLE-FLOAT (at compile time) to attempt to determine if SHORT-FLOAT and SINGLE-FLOAT are the same on the current implementation and possibly conditionally-compile away the SINGLE-FLOAT type, and similarly for DOUBLE- and LONG-....
I would not be checking the numeric values of the number at run-time, but the bounds on the types at compile time. Would that not work?
Ah, yes, now I see. But then you might just write it straightforwardly as (subtypep 'short-float 'single-float), or vice versa. Since subtypes of float are either disjoint or synonymous, subtypep in this context effectively tests the types for equality.
Still, I doubt the intended effect is worth the trouble. Having superfluous clauses in typecase is explicitly allowed: ‘[...] is permissible for more than one clause to specify a matching type’ (CLHS 5.3, definition of TYPECASE et al.) In fact, by trying to get rid of them beforehand we would be (re)doing the compiler's job. E. g., CCL eliminates them right there at macro expansion time:
(macroexpand '(typecase foo (short-float #\S) (long-float #\L) (single-float #\F) (double-float #\D))) ⇒ (LET ((#:G4 FOO)) (DECLARE (IGNORABLE #:G4)) (COND ((TYPEP #:G4 'SHORT-FLOAT) NIL #\S) ((TYPEP #:G4 'LONG-FLOAT) NIL #\L)))
Are we going to such lengths just in order to get rid of annoying warnings in a single implementation?
Yes. If these were simply style warnings, then I would agree with you. However, they are not --- they are real, bona fide warnings, causing COMPILE-FILE failure: ;Compiler warnings for "home:lisp;cl-json;src;encoder.lisp.newest" : ; In WRITE-JSON-NUMBER: Clause (SINGLE-FLOAT 'SINGLE-FLOAT) ignored in ETYPECASE form - shadowed by (SHORT-FLOAT 'SHORT-FLOAT) . ; In WRITE-JSON-NUMBER: Clause (LONG-FLOAT 'LONG-FLOAT) ignored in ETYPECASE form - shadowed by (DOUBLE-FLOAT 'DOUBLE-FLOAT) . ; Warning: COMPILE-FILE warned while performing #<COMPILE-OP (:FORCE T) #x3020009CEB4D> on #<CL-SOURCE-FILE "cl-json" "src" "encoder">. ; While executing: #<STANDARD-METHOD ASDF:PERFORM (ASDF:COMPILE-OP ASDF:CL-SOURCE-FILE)>, in process repl-thread(10). ; Warning: COMPILE-FILE failed while performing #<COMPILE-OP (:FORCE T) #x3020009CEB4D> on #<CL-SOURCE-FILE "cl-json" "src" "encoder">. ; While executing: #<STANDARD-METHOD ASDF:PERFORM (ASDF:COMPILE-OP ASDF:CL-SOURCE-FILE)>, in process repl-thread(10). So I think these warnings must die. I am inclined to think that these should simply be STYLE-WARNINGs, but CCL does not agree with me. Best, r
On 25 Jun 2011, at 19:01, Robert Goldman wrote:
I am inclined to think that these should simply be STYLE-WARNINGs, but CCL does not agree with me.
CCL seems to be in breach of the standard in this case. I have opened a ticket on their Trac, see http://trac.clozure.com/ccl/ticket/ 872 . A compliant application, I believe, should expect a STYLE- WARNING here; a SIMPLE-WARNING should be considered an implementation quirk. Yours, - B. Sm.
On 25 Jun 2011, at 20:41, Boris Smilga wrote:
CCL seems to be in breach of the standard in this case. I have opened a ticket on their Trac, see http://trac.clozure.com/ccl/ ticket/872 . A compliant application, I believe, should expect a STYLE-WARNING here; a SIMPLE-WARNING should be considered an implementation quirk.
Just to keep you informed: our colleagues over at Clozure had the warning type fixed to some subtype of style-warning. So, we may count on correct behaviour in the repository version and future releases of CCL. Yours, - B. Sm.
On 6/26/11 Jun 26 -4:10 PM, Boris Smilga wrote:
On 25 Jun 2011, at 20:41, Boris Smilga wrote:
CCL seems to be in breach of the standard in this case. I have opened a ticket on their Trac, see http://trac.clozure.com/ccl/ticket/872 . A compliant application, I believe, should expect a STYLE-WARNING here; a SIMPLE-WARNING should be considered an implementation quirk.
Just to keep you informed: our colleagues over at Clozure had the warning type fixed to some subtype of style-warning. So, we may count on correct behaviour in the repository version and future releases of CCL.
I agree that fixing this will be A Good Thing, but even when it is fixed, I don't believe that a library should emit even a STYLE-WARNING, if at all possible. So I think it would be best for us either to keep the expedient I have put in (which I concede is not elegant), or to muffle any warnings about unnecessary typecase branches (since they are compiled away, anyhow). I am not a believer in leaving warnings around, because if one starts to leave benign ones around, one gets in the habit of ignoring them, and ends up missing warnings that indicate real problems.... best, r
On Mon, Jun 27, 2011 at 3:16 PM, Robert Goldman <rpgoldman@sift.info> wrote:
I am not a believer in leaving warnings around, because if one starts to leave benign ones around, one gets in the habit of ignoring them, and ends up missing warnings that indicate real problems....
Actually, that's what the *formal* distinction between style-warning and other kinds of warnings is all about. To quote the standard once again: ‘In general, the question of whether code is faulty or substandard is a subjective decision to be made by the facility processing that code. The intent is that whenever such a facility wishes to complain about code on such subjective grounds, it should use this condition type so that any clients who wish to redirect or muffle superfluous warnings can do so without risking that they will be redirecting or muffling other, more serious warnings.’ This is probably out of scope of this mailing list, though. Yours, - B. Sm.
On 6/27/11 Jun 27 -10:50 AM, Boris Smilga wrote:
On Mon, Jun 27, 2011 at 3:16 PM, Robert Goldman <rpgoldman@sift.info> wrote:
I am not a believer in leaving warnings around, because if one starts to leave benign ones around, one gets in the habit of ignoring them, and ends up missing warnings that indicate real problems....
Actually, that's what the *formal* distinction between style-warning and other kinds of warnings is all about. To quote the standard once again:
‘In general, the question of whether code is faulty or substandard is a subjective decision to be made by the facility processing that code. The intent is that whenever such a facility wishes to complain about code on such subjective grounds, it should use this condition type so that any clients who wish to redirect or muffle superfluous warnings can do so without risking that they will be redirecting or muffling other, more serious warnings.’
This is probably out of scope of this mailing list, though.
Right. That suggests that when the new CCL comes out, we could simply muffle the style warnings in encode-json-nr, and rip out my laborious typecase minimizing #+ and #-'s... Indeed, probably we could take what I've got now and wrap parts of it in #+(and CCL (not CCL-1.7)) then add some #+CCL-1.7 that would take care of muffling the style-warnings from compiling encode-json-nr Cheers, r
participants (3)
-
Boris Smilga -
Daniel Brunner -
Robert Goldman