- cmucl-cvs - mailman3.common-lisp.net

[Git][cmucl/cmucl][issue-367-count-octets-for-encoding] 2 commits: Add test that errors are signaled in string-octet-count
by Raymond Toy (＠rtoy) 13 Jan '25

13 Jan '25

Raymond Toy pushed to branch issue-367-count-octets-for-encoding at cmucl / cmucl Commits: 02be9b8b by Raymond Toy at 2025-01-13T13:19:18-08:00 Add test that errors are signaled in string-octet-count Just test that we signal an error when converting a string to :ascii. - - - - - 06488aeb by Raymond Toy at 2025-01-13T13:25:19-08:00 Add some sanity checks for STRING-TO-OCTETS Since `string-to-octets` can return a buffer of octets whose length is not the same as the returned number of octets, add a check that they are the same. If not, then we `string-octet-count` may not be consistent. Also add a check that the converted number of characters matches the length of the string. - - - - - 1 changed file: - tests/external-formats.lisp Changes: ===================================== tests/external-formats.lisp ===================================== @@ -20,9 +20,22 @@ "Random test string with codepoints below 20000") + (defmacro test-octet-count (string format) - `(assert-equal (length (stream:string-to-octets ,string :external-format ,format)) - (stream::string-octet-count ,string :external-format ,format))) + "Test that STRING-OCTET-COUNT returns the correct number of octets" + ;; We expect STRING-OCTET-COUNT returns the same number of octets + ;; that are produced by STRING-TO-OCTETS. + `(multiple-value-bind (octets count converted) + (stream:string-to-octets ,string :external-format ,format) + ;; While we're at it, make sure that the length of the octet + ;; buffer matches returned count. And make sure we converted all + ;; the characters in the string. + (assert-equal (length octets) count) + (assert-equal (length ,string) converted) + ;; Finally, make sure that STRING-OCTET-COUNT returns the same + ;; number of octets from STRING-TO-OCTETS. + (assert-equal (length octets) + (stream::string-octet-count ,string :external-format ,format)))) (define-test octet-count.iso8859-1 (:tag :octet-count) @@ -32,6 +45,13 @@ (:tag :octet-count) (test-octet-count *test-iso8859-1* :ascii)) +(define-test octet-count.ascii.error + (:tag :octet-count) + (assert-error 'simple-error + (stream::string-octet-count *test-iso8859-1* + :external-format :ascii + :error 'error))) + (define-test octet-count.utf-8 (:tag :octet-count) (test-octet-count *test-unicode* :utf-8)) View it on GitLab: https://gitlab.common-lisp.net/cmucl/cmucl/-/compare/ccd15fcecd9a55e61b22a7… -- View it on GitLab: https://gitlab.common-lisp.net/cmucl/cmucl/-/compare/ccd15fcecd9a55e61b22a7… You're receiving this email because of your account on gitlab.common-lisp.net.

1 0

[Git][cmucl/cmucl][issue-367-count-octets-for-encoding] Add octet-count for mac-roman format with corresponding test
by Raymond Toy (＠rtoy) 13 Jan '25

13 Jan '25

Raymond Toy pushed to branch issue-367-count-octets-for-encoding at cmucl / cmucl Commits: ccd15fce by Raymond Toy at 2025-01-13T12:40:36-08:00 Add octet-count for mac-roman format with corresponding test This also means all the cp12??, koi8-5, mac-greek, mac-icelandic, mac-latin2, and mac-turkish also has octet-count now too since mac-roman is the base for these other formats. - - - - - 2 changed files: - src/pcl/simple-streams/external-formats/mac-roman.lisp - tests/external-formats.lisp Changes: ===================================== src/pcl/simple-streams/external-formats/mac-roman.lisp ===================================== @@ -49,4 +49,19 @@ character and illegal outputs are replaced by a question mark.") (declare (optimize (ext:inhibit-warnings 3))) (funcall ,error "Cannot output codepoint #x~X to MAC-ROMAN stream" ,code)) - #x3F))))))) + #x3F)))))) + () + () + (octet-count (code state error present) + `(if (< ,code 128) + 1 + (let ((,present (get-inverse ,itable ,code))) + (if ,present + 1 + (if ,error + (locally + ;; No warnings about fdefinition + (declare (optimize (ext:inhibit-warnings 3))) + (funcall ,error "Cannot output codepoint #x~X to MAC-ROMAN stream" + ,code)) + 1)))))) ===================================== tests/external-formats.lisp ===================================== @@ -108,5 +108,8 @@ (:tag :octet-count) (test-octet-count *test-iso8859-1* :iso8859-15)) - +(define-test octet-count.mac-roman + (:tag :octet-count) + (test-octet-count *test-iso8859-1* :mac-roman)) + View it on GitLab: https://gitlab.common-lisp.net/cmucl/cmucl/-/commit/ccd15fcecd9a55e61b22a7f… -- View it on GitLab: https://gitlab.common-lisp.net/cmucl/cmucl/-/commit/ccd15fcecd9a55e61b22a7f… You're receiving this email because of your account on gitlab.common-lisp.net.

1 0

[Git][cmucl/cmucl][issue-367-count-octets-for-encoding] 4 commits: Add octet-count method for euc-kr
by Raymond Toy (＠rtoy) 13 Jan '25

13 Jan '25

Raymond Toy pushed to branch issue-367-count-octets-for-encoding at cmucl / cmucl Commits: fe3f7011 by Raymond Toy at 2025-01-13T08:53:49-08:00 Add octet-count method for euc-kr - - - - - f0aea1e8 by Raymond Toy at 2025-01-13T08:54:08-08:00 Add octet-count for iso8859-2 This also adds octet-count for all the other iso8859 formats (except iso8859-1) since they all derive from iso8859-2. - - - - - 260c3806 by Raymond Toy at 2025-01-13T12:09:11-08:00 Oops. Fix bad copy and paste of too much. - - - - - f5e4262b by Raymond Toy at 2025-01-13T12:09:41-08:00 Add tests for euc-kr and iso8859 formats. - - - - - 3 changed files: - src/pcl/simple-streams/external-formats/euc-kr.lisp - src/pcl/simple-streams/external-formats/iso8859-2.lisp - tests/external-formats.lisp Changes: ===================================== src/pcl/simple-streams/external-formats/euc-kr.lisp ===================================== @@ -1007,4 +1007,16 @@ character and illegal outputs are replaced by a question mark.") (t (if ,error (funcall ,error "Cannot output codepoint #x~X to EUC-KR format." ,code) - (,output #X3f)))))))) + (,output #X3f))))))) + () + () + (octet-count (code state error present) + `(if (<= ,code #x7f) + 1 + (let ((,present (get-inverse ,itable ,code))) + (cond (,present + 2) + (t + (if ,error + (funcall ,error "Cannot output codepoint #x~X to EUC-KR format." ,code) + 1))))))) ===================================== src/pcl/simple-streams/external-formats/iso8859-2.lisp ===================================== @@ -47,4 +47,19 @@ character and illegal outputs are replaced by a question mark.") (declare (optimize (ext:inhibit-warnings 3))) (funcall ,error "Cannot output codepoint #x~X to ISO8859-2 stream" ,code)) - #x3F))))))) + #x3F)))))) + () + () + (octet-count (code state error present) + `(if (< ,code 160) + 1 + (let ((,present (get-inverse ,itable ,code))) + (if ,present + 1 + (if ,error + (locally + ;; No warnings about fdefinition + (declare (optimize (ext:inhibit-warnings 3))) + (funcall ,error "Cannot output codepoint #x~X to ISO8859-2 stream" + ,code)) + 1)))))) ===================================== tests/external-formats.lisp ===================================== @@ -60,7 +60,53 @@ (:tag :octet-count) (test-octet-count *test-unicode* :utf-32-le)) +(define-test octet-count.euc-kr + (:tag :octet-count) + (test-octet-count *test-unicode* :euc-kr)) + +(define-test octet-count.iso8859-2 + (:tag :octet-count) + (test-octet-count *test-iso8859-1* :iso8859-2)) + +(define-test octet-count.iso8859-3 + (:tag :octet-count) + (test-octet-count *test-iso8859-1* :iso8859-3)) + +(define-test octet-count.iso8859-4 + (:tag :octet-count) + (test-octet-count *test-iso8859-1* :iso8859-4)) + +(define-test octet-count.iso8859-5 + (:tag :octet-count) + (test-octet-count *test-iso8859-1* :iso8859-5)) + +(define-test octet-count.iso8859-6 + (:tag :octet-count) + (test-octet-count *test-iso8859-1* :iso8859-6)) +(define-test octet-count.iso8859-7 + (:tag :octet-count) + (test-octet-count *test-iso8859-1* :iso8859-7)) + +(define-test octet-count.iso8859-8 + (:tag :octet-count) + (test-octet-count *test-iso8859-1* :iso8859-8)) + +(define-test octet-count.iso8859-10 + (:tag :octet-count) + (test-octet-count *test-iso8859-1* :iso8859-10)) + +(define-test octet-count.iso8859-13 + (:tag :octet-count) + (test-octet-count *test-iso8859-1* :iso8859-13)) + +(define-test octet-count.iso8859-14 + (:tag :octet-count) + (test-octet-count *test-iso8859-1* :iso8859-14)) + +(define-test octet-count.iso8859-15 + (:tag :octet-count) + (test-octet-count *test-iso8859-1* :iso8859-15)) View it on GitLab: https://gitlab.common-lisp.net/cmucl/cmucl/-/compare/865a06b03710020d17e7f9… -- View it on GitLab: https://gitlab.common-lisp.net/cmucl/cmucl/-/compare/865a06b03710020d17e7f9… You're receiving this email because of your account on gitlab.common-lisp.net.

1 0

[Git][cmucl/cmucl][issue-367-count-octets-for-encoding] Make EF-OCTET-COUNT an inline function and add documentation
by Raymond Toy (＠rtoy) 13 Jan '25

13 Jan '25

Raymond Toy pushed to branch issue-367-count-octets-for-encoding at cmucl / cmucl Commits: 865a06b0 by Raymond Toy at 2025-01-12T17:45:18-08:00 Make EF-OCTET-COUNT an inline function and add documentation Like the other EF- functions, make EF-OCTET-COUNT an inline function. Add comments in define-external-format about the OCTET-COUNT slot. - - - - - 1 changed file: - src/code/extfmts.lisp Changes: ===================================== src/code/extfmts.lisp ===================================== @@ -133,7 +133,8 @@ (setf (gethash (ef-name ef) *external-formats*) ef)) (declaim (inline ef-octets-to-code ef-code-to-octets ef-flush-state ef-copy-state - ef-cache ef-min-octets ef-max-octets)) + ef-cache ef-min-octets ef-max-octets + ef-octet-count)) (defun ef-octets-to-code (ef) (efx-octets-to-code (ef-efx ef))) @@ -176,7 +177,7 @@ ;;; DEFINE-EXTERNAL-FORMAT -- Public ;;; ;;; name (&key base min max size documentation) (&rest slots) octets-to-code -;;; code-to-octets flush-state copy-state +;;; code-to-octets flush-state copy-state octet-count ;;; ;;; Define a new external format. If base is specified, then an ;;; external format is defined that is based on a previously defined @@ -238,6 +239,15 @@ ;;; This should probably be a deep copy so that if the original ;;; state is modified, the copy is not. ;;; +;;; octet-count (code state error &rest vars) +;;; Defines a form to determine the number of octets needed to +;;; encode the given CODE using the external format. This is +;;; essentially the same as CODE-TO-OCTETS, except the encoding is +;;; not saved anywhere. ERROR is the same as in CODE-TO-OCTETS. +;;; +;;; This should return one value: the number of octets needed to +;;; encode the given code. +;;; ;;; Note: external-formats work on code-points, not ;;; characters, so that the entire 31 bit ISO-10646 range can be ;;; used internally regardless of the size of a character recognized View it on GitLab: https://gitlab.common-lisp.net/cmucl/cmucl/-/commit/865a06b03710020d17e7f9d… -- View it on GitLab: https://gitlab.common-lisp.net/cmucl/cmucl/-/commit/865a06b03710020d17e7f9d… You're receiving this email because of your account on gitlab.common-lisp.net.

1 0

[Git][cmucl/cmucl][issue-367-count-octets-for-encoding] Include BOM in octet count for UTF-16 and UTF-32
by Raymond Toy (＠rtoy) 12 Jan '25

12 Jan '25

Raymond Toy pushed to branch issue-367-count-octets-for-encoding at cmucl / cmucl Commits: 31a76ff7 by Raymond Toy at 2025-01-12T14:08:01-08:00 Include BOM in octet count for UTF-16 and UTF-32 `string-to-octets` includes the BOM when encoding strings. To be consistent, update `string-octet-count` to include the BOM when computing the number of octets. This is only needed for :UTF-16 and :UTF-32 formats. The other utf-16 and utf-32 formats don't include the BOM. Enable tests for these two formats too. - - - - - 3 changed files: - src/pcl/simple-streams/external-formats/utf-16.lisp - src/pcl/simple-streams/external-formats/utf-32.lisp - tests/external-formats.lisp Changes: ===================================== src/pcl/simple-streams/external-formats/utf-16.lisp ===================================== @@ -158,16 +158,16 @@ Unicode replacement character.") ;; The state is list. Copy it `(copy-list ,state)) (octet-count (code state error) - `(progn - #+nil + `(let ((bom-count 0)) (unless ,state ;; Output BOM - (output #xFEFF) + (setf bom-count 2) (setf ,state t)) - (cond ((< ,code #x10000) - 2) - ((< ,code #x110000) - 4) - (t - ;; Replacement character is 2 octets - 2))))) + (+ bom-count + (cond ((< ,code #x10000) + 2) + ((< ,code #x110000) + 4) + (t + ;; Replacement character is 2 octets + 2)))))) ===================================== src/pcl/simple-streams/external-formats/utf-32.lisp ===================================== @@ -116,11 +116,9 @@ Unicode replacement character.") ;; The state is either NIL or T, so we can just return that. `(progn ,state)) (octet-count (code state error) - `(progn - ;; Should we count the BOM? - #+nil + `(let ((bom-count 0)) (unless ,state - (out #xFEFF) + (setf bom-count 4) (setf ,state t)) (cond ((lisp::surrogatep ,code) (if ,error @@ -130,6 +128,6 @@ Unicode replacement character.") (funcall ,error "Surrogate code #x~4,'0X is illegal for UTF32 output" ,code)) ;; Replacement character is 2 octets - 2)) + (+ 2 bom-count))) (t - 4))))) + (+ 4 bom-count)))))) ===================================== tests/external-formats.lisp ===================================== @@ -36,7 +36,6 @@ (:tag :octet-count) (test-octet-count *test-unicode* :utf-8)) -#+nil (define-test octet-count.utf-16 (:tag :octet-count) (test-octet-count *test-unicode* :utf-16)) @@ -49,7 +48,6 @@ (:tag :octet-count) (test-octet-count *test-unicode* :utf-16-le)) -#+nil (define-test octet-count.utf-32 (:tag :octet-count) (test-octet-count *test-unicode* :utf-32)) View it on GitLab: https://gitlab.common-lisp.net/cmucl/cmucl/-/commit/31a76ff7e674b0dac9f89d9… -- View it on GitLab: https://gitlab.common-lisp.net/cmucl/cmucl/-/commit/31a76ff7e674b0dac9f89d9… You're receiving this email because of your account on gitlab.common-lisp.net.

1 0

[Git][cmucl/cmucl][issue-367-count-octets-for-encoding] Export STRING-OCTET-COUNT
by Raymond Toy (＠rtoy) 12 Jan '25

12 Jan '25

Raymond Toy pushed to branch issue-367-count-octets-for-encoding at cmucl / cmucl Commits: aa6f0565 by Raymond Toy at 2025-01-12T13:49:52-08:00 Export STRING-OCTET-COUNT - - - - - 2 changed files: - src/code/exports.lisp - src/code/extfmts.lisp Changes: ===================================== src/code/exports.lisp ===================================== @@ -1456,7 +1456,8 @@ (name '("STRING-TO-OCTETS" "OCTETS-TO-STRING" "*DEFAULT-EXTERNAL-FORMAT*" "STRING-ENCODE" "STRING-DECODE" "SET-SYSTEM-EXTERNAL-FORMAT" - "LIST-ALL-EXTERNAL-FORMATS" "DESCRIBE-EXTERNAL-FORMAT")) + "LIST-ALL-EXTERNAL-FORMATS" "DESCRIBE-EXTERNAL-FORMAT" + "STRING-OCTET-COUNT")) (intern name "STREAM")) (defpackage "EXTENSIONS" ===================================== src/code/extfmts.lisp ===================================== @@ -19,7 +19,8 @@ string-encode string-decode set-system-external-format +replacement-character-code+ list-all-external-formats - describe-external-format)) + describe-external-format + string-octet-count)) (defvar *default-external-format* :utf-8 View it on GitLab: https://gitlab.common-lisp.net/cmucl/cmucl/-/commit/aa6f05655508dd8f1fae8f8… -- View it on GitLab: https://gitlab.common-lisp.net/cmucl/cmucl/-/commit/aa6f05655508dd8f1fae8f8… You're receiving this email because of your account on gitlab.common-lisp.net.

1 0

[Git][cmucl/cmucl][issue-367-count-octets-for-encoding] Add tests for octet counts
by Raymond Toy (＠rtoy) 12 Jan '25

12 Jan '25

Raymond Toy pushed to branch issue-367-count-octets-for-encoding at cmucl / cmucl Commits: ea8b2abe by Raymond Toy at 2025-01-12T13:46:01-08:00 Add tests for octet counts - - - - - 1 changed file: - + tests/external-formats.lisp Changes: ===================================== tests/external-formats.lisp ===================================== @@ -0,0 +1,68 @@ +;;; Tests for external formats + +(defpackage :external-formats-tests + (:use :cl :lisp-unit)) + +(in-package "EXTERNAL-FORMATS-TESTS") + +(defparameter *test-iso8859-1* + (let ((rs (kernel::make-random-object :state (kernel::init-random-state 27182828)))) + (lisp::codepoints-string + (loop for k from 0 below 1000 + collect (random 256)))) + "Random test string with ISO8859-1 characters") + +(defparameter *test-unicode* + (let ((rs (kernel::make-random-object :state (kernel::init-random-state 27182828)))) + (lisp::codepoints-string + (loop for k from 0 below 1000 + collect (random 20000)))) + "Random test string with codepoints below 20000") + + +(defmacro test-octet-count (string format) + `(assert-equal (length (stream:string-to-octets ,string :external-format ,format)) + (stream::string-octet-count ,string :external-format ,format))) + +(define-test octet-count.iso8859-1 + (:tag :octet-count) + (test-octet-count *test-iso8859-1* :iso8859-1)) + +(define-test octet-count.ascii + (:tag :octet-count) + (test-octet-count *test-iso8859-1* :ascii)) + +(define-test octet-count.utf-8 + (:tag :octet-count) + (test-octet-count *test-unicode* :utf-8)) + +#+nil +(define-test octet-count.utf-16 + (:tag :octet-count) + (test-octet-count *test-unicode* :utf-16)) + +(define-test octet-count.utf-16-be + (:tag :octet-count) + (test-octet-count *test-unicode* :utf-16-be)) + +(define-test octet-count.utf-16-le + (:tag :octet-count) + (test-octet-count *test-unicode* :utf-16-le)) + +#+nil +(define-test octet-count.utf-32 + (:tag :octet-count) + (test-octet-count *test-unicode* :utf-32)) + +(define-test octet-count.utf-32-le + (:tag :octet-count) + (test-octet-count *test-unicode* :utf-32-le)) + +(define-test octet-count.utf-32-le + (:tag :octet-count) + (test-octet-count *test-unicode* :utf-32-le)) + + + + + View it on GitLab: https://gitlab.common-lisp.net/cmucl/cmucl/-/commit/ea8b2abee2ded208d69fdae… -- View it on GitLab: https://gitlab.common-lisp.net/cmucl/cmucl/-/commit/ea8b2abee2ded208d69fdae… You're receiving this email because of your account on gitlab.common-lisp.net.

1 0

[Git][cmucl/cmucl][issue-367-count-octets-for-encoding] 2 commits: Fix typos in octet-count in utf-16 and utf-32
by Raymond Toy (＠rtoy) 12 Jan '25

12 Jan '25

Raymond Toy pushed to branch issue-367-count-octets-for-encoding at cmucl / cmucl Commits: f0298943 by Raymond Toy at 2025-01-12T13:09:13-08:00 Fix typos in octet-count in utf-16 and utf-32 Too much cutting and pasting so that we incorrectly named `octet-count` as `code-to-octets`. Fix the names. - - - - - 171e34ac by Raymond Toy at 2025-01-12T13:25:08-08:00 Fix typo in utf-32-be: Remove extra closing paren - - - - - 6 changed files: - src/pcl/simple-streams/external-formats/utf-16-be.lisp - src/pcl/simple-streams/external-formats/utf-16-le.lisp - src/pcl/simple-streams/external-formats/utf-16.lisp - src/pcl/simple-streams/external-formats/utf-32-be.lisp - src/pcl/simple-streams/external-formats/utf-32-le.lisp - src/pcl/simple-streams/external-formats/utf-32.lisp Changes: ===================================== src/pcl/simple-streams/external-formats/utf-16-be.lisp ===================================== @@ -111,7 +111,7 @@ Unicode replacement character.") ;; The state is either NIL or a codepoint, so nothing really ;; special is needed to copy it. `(progn ,state)) - (code-to-octets (code state error) + (octet-count (code state error) `(cond ((< ,code #x10000) 2) ((< ,code #x110000) ===================================== src/pcl/simple-streams/external-formats/utf-16-le.lisp ===================================== @@ -112,7 +112,7 @@ Unicode replacement character.") ;; The state is either NIL or a codepoint, so nothing really ;; special is needed. `(progn ,state)) - (code-to-octets (code state error) + (octet-count (code state error) `(cond ((< ,code #x10000) 2) ((< ,code #x110000) ===================================== src/pcl/simple-streams/external-formats/utf-16.lisp ===================================== @@ -157,7 +157,7 @@ Unicode replacement character.") (copy-state (state) ;; The state is list. Copy it `(copy-list ,state)) - (code-to-octets (code state error) + (octet-count (code state error) `(progn #+nil (unless ,state ===================================== src/pcl/simple-streams/external-formats/utf-32-be.lisp ===================================== @@ -64,7 +64,7 @@ Unicode replacement character.") (out ,code))))) () () - (code-to-octets (code state error) + (octet-count (code state error) `(cond ((lisp::surrogatep ,code) (if ,error (locally @@ -75,4 +75,4 @@ Unicode replacement character.") ;; Replacement character is 2 octets 2)) (t - 4))))) + 4)))) ===================================== src/pcl/simple-streams/external-formats/utf-32-le.lisp ===================================== @@ -65,7 +65,7 @@ Unicode replacement character.") (out ,code))))) () () - (code-to-octets (code state error) + (octet-count (code state error) `(cond ((lisp::surrogatep ,code) (if ,error (locally ===================================== src/pcl/simple-streams/external-formats/utf-32.lisp ===================================== @@ -115,7 +115,7 @@ Unicode replacement character.") (copy-state (state) ;; The state is either NIL or T, so we can just return that. `(progn ,state)) - (code-to-octets (code state output error i c) + (octet-count (code state error) `(progn ;; Should we count the BOM? #+nil View it on GitLab: https://gitlab.common-lisp.net/cmucl/cmucl/-/compare/83db856a43a3e810cfdce0… -- View it on GitLab: https://gitlab.common-lisp.net/cmucl/cmucl/-/compare/83db856a43a3e810cfdce0… You're receiving this email because of your account on gitlab.common-lisp.net.

1 0

[Git][cmucl/cmucl][issue-367-count-octets-for-encoding] 2 commits: Copy the utf-8, iso8859-1, ascii impl
by Raymond Toy (＠rtoy) 12 Jan '25

12 Jan '25

Raymond Toy pushed to branch issue-367-count-octets-for-encoding at cmucl / cmucl Commits: f5b4ae8a by Raymond Toy at 2025-01-12T11:39:37-08:00 Copy the utf-8, iso8859-1, ascii impl These external formats are defined in extfmts. For completeness, copy these implementations to the corresponding files in src/pcl/simple-streams/external-formats. - - - - - 83db856a by Raymond Toy at 2025-01-12T11:49:48-08:00 Implement octet-count utf-16 and utf-32. Pretty basic implementation, taking care of handling the replacement character if needed. - - - - - 9 changed files: - src/pcl/simple-streams/external-formats/ascii.lisp - src/pcl/simple-streams/external-formats/iso8859-1.lisp - src/pcl/simple-streams/external-formats/utf-16-be.lisp - src/pcl/simple-streams/external-formats/utf-16-le.lisp - src/pcl/simple-streams/external-formats/utf-16.lisp - src/pcl/simple-streams/external-formats/utf-32-be.lisp - src/pcl/simple-streams/external-formats/utf-32-le.lisp - src/pcl/simple-streams/external-formats/utf-32.lisp - src/pcl/simple-streams/external-formats/utf-8.lisp Changes: ===================================== src/pcl/simple-streams/external-formats/ascii.lisp ===================================== @@ -33,5 +33,15 @@ replaced with a question mark.") (declare (optimize (ext:inhibit-warnings 3))) (funcall ,error "Cannot output codepoint #x~X to ASCII stream" ,code)) #x3F) - ,code)))) + ,code))) + () + () + (octet-count (code state error) + `(if (> ,code #x7f) + (if ,error + (locally + (declare (optimize (ext:inhibit-warnings 3))) + (funcall ,error "Cannot output codepoint #x~X to ASCII stream" ,code)) + 1) + 1))) ===================================== src/pcl/simple-streams/external-formats/iso8859-1.lisp ===================================== @@ -31,4 +31,17 @@ character and illegal outputs are replaced by a question mark.") (funcall ,error "Cannot output codepoint #x~X to ISO8859-1 stream" ,code 1)) #x3F) - ,code)))) + ,code))) + () + () + (octet-count (code state error) + `(if (> ,code 255) + (if ,error + (locally + ;; No warnings about fdefinition + (declare (optimize (ext:inhibit-warnings 3))) + (funcall ,error + (intl:gettext "Cannot output codepoint #x~X to ISO8859-1 stream") + ,code 1)) + 1) + 1))) ===================================== src/pcl/simple-streams/external-formats/utf-16-be.lisp ===================================== @@ -110,4 +110,12 @@ Unicode replacement character.") (copy-state (state) ;; The state is either NIL or a codepoint, so nothing really ;; special is needed to copy it. - `(progn ,state))) + `(progn ,state)) + (code-to-octets (code state error) + `(cond ((< ,code #x10000) + 2) + ((< ,code #x110000) + 4) + (t + ;; Replacement character is 2 octets + 2)))) ===================================== src/pcl/simple-streams/external-formats/utf-16-le.lisp ===================================== @@ -111,4 +111,12 @@ Unicode replacement character.") (copy-state (state) ;; The state is either NIL or a codepoint, so nothing really ;; special is needed. - `(progn ,state))) + `(progn ,state)) + (code-to-octets (code state error) + `(cond ((< ,code #x10000) + 2) + ((< ,code #x110000) + 4) + (t + ;; Replacement character is 2 octets + 2)))) ===================================== src/pcl/simple-streams/external-formats/utf-16.lisp ===================================== @@ -156,4 +156,18 @@ Unicode replacement character.") ,c)))))) (copy-state (state) ;; The state is list. Copy it - `(copy-list ,state))) + `(copy-list ,state)) + (code-to-octets (code state error) + `(progn + #+nil + (unless ,state + ;; Output BOM + (output #xFEFF) + (setf ,state t)) + (cond ((< ,code #x10000) + 2) + ((< ,code #x110000) + 4) + (t + ;; Replacement character is 2 octets + 2))))) ===================================== src/pcl/simple-streams/external-formats/utf-32-be.lisp ===================================== @@ -61,4 +61,18 @@ Unicode replacement character.") ,code)) +replacement-character-code+))) (t - (out ,code)))))) + (out ,code))))) + () + () + (code-to-octets (code state error) + `(cond ((lisp::surrogatep ,code) + (if ,error + (locally + ;; No warnings about fdefinition + (declare (optimize (ext:inhibit-warnings 3))) + (funcall ,error "Surrogate code #x~4,'0X is illegal for UTF32 output" + ,code)) + ;; Replacement character is 2 octets + 2)) + (t + 4))))) ===================================== src/pcl/simple-streams/external-formats/utf-32-le.lisp ===================================== @@ -62,4 +62,18 @@ Unicode replacement character.") ,code)) +replacement-character-code+))) (t - (out ,code)))))) + (out ,code))))) + () + () + (code-to-octets (code state error) + `(cond ((lisp::surrogatep ,code) + (if ,error + (locally + ;; No warnings about fdefinition + (declare (optimize (ext:inhibit-warnings 3))) + (funcall ,error "Surrogate code #x~4,'0X is illegal for UTF32 output" + ,code)) + ;; Replacement character is 2 octets + 2)) + (t + 4)))) ===================================== src/pcl/simple-streams/external-formats/utf-32.lisp ===================================== @@ -114,4 +114,22 @@ Unicode replacement character.") nil (copy-state (state) ;; The state is either NIL or T, so we can just return that. - `(progn ,state))) + `(progn ,state)) + (code-to-octets (code state output error i c) + `(progn + ;; Should we count the BOM? + #+nil + (unless ,state + (out #xFEFF) + (setf ,state t)) + (cond ((lisp::surrogatep ,code) + (if ,error + (locally + ;; No warnings about fdefinition + (declare (optimize (ext:inhibit-warnings 3))) + (funcall ,error "Surrogate code #x~4,'0X is illegal for UTF32 output" + ,code)) + ;; Replacement character is 2 octets + 2)) + (t + 4))))) ===================================== src/pcl/simple-streams/external-formats/utf-8.lisp ===================================== @@ -127,4 +127,14 @@ replacement character.") ((< ,code #x800) (utf8 ,code 1)) ((< ,code #x10000) (utf8 ,code 2)) ((< ,code #x110000) (utf8 ,code 3)) - (t (error "How did this happen? Codepoint U+~X is illegal" ,code)))))) + (t (error "How did this happen? Codepoint U+~X is illegal" ,code))))) + () + () + (octet-count (code state error) + `(locally + (declare (optimize (ext:inhibit-warnings 3))) + (cond ((< ,code #x80) 1) + ((< ,code #x800) 2) + ((< ,code #x10000) 3) + ((< ,code #x110000) 4) + (t (error "How did this happen? Codepoint U+~X is illegal" ,code)))))) View it on GitLab: https://gitlab.common-lisp.net/cmucl/cmucl/-/compare/582a9f51db3e6df7b653df… -- View it on GitLab: https://gitlab.common-lisp.net/cmucl/cmucl/-/compare/582a9f51db3e6df7b653df… You're receiving this email because of your account on gitlab.common-lisp.net.

1 0

[Git][cmucl/cmucl][issue-367-count-octets-for-encoding] Fix value of bootstrap var
by Raymond Toy (＠rtoy) 12 Jan '25

12 Jan '25

Raymond Toy pushed to branch issue-367-count-octets-for-encoding at cmucl / cmucl Commits: 582a9f51 by Raymond Toy at 2025-01-11T19:32:54-08:00 Fix value of bootstrap var Forgot to include the "-B" to tell the build.sh what the bootstrap file is. - - - - - 1 changed file: - .gitlab-ci.yml Changes: ===================================== .gitlab-ci.yml ===================================== @@ -1,7 +1,7 @@ variables: download_url: "https://common-lisp.net/project/cmucl/downloads/snapshots/2024/08" version: "2024-08-x86" - bootstrap: "boot-2024-08" + bootstrap: "-B boot-2024-08" stages: View it on GitLab: https://gitlab.common-lisp.net/cmucl/cmucl/-/commit/582a9f51db3e6df7b653df9… -- View it on GitLab: https://gitlab.common-lisp.net/cmucl/cmucl/-/commit/582a9f51db3e6df7b653df9… You're receiving this email because of your account on gitlab.common-lisp.net.

1 0