- cmucl-cvs - mailman3.common-lisp.net

[Git][cmucl/cmucl][issue-375-mkstemp-return-filename] Fix typo in unix-mkdtemp
by Raymond Toy (＠rtoy) 03 Feb '25

03 Feb '25

Raymond Toy pushed to branch issue-375-mkstemp-return-filename at cmucl / cmucl Commits: 144065ee by Raymond Toy at 2025-02-03T10:16:16-08:00 Fix typo in unix-mkdtemp We were returning from `unix-mkstemp` instead of `unix-mkdtemp`. - - - - - 1 changed file: - src/code/unix.lisp Changes: ===================================== src/code/unix.lisp ===================================== @@ -2957,7 +2957,7 @@ returned." ;; Make sure the template is valid. (unless (check-template template) - (return-from unix-mkstemp + (return-from unix-mkdtemp (values nil einval))) (let* ((format (if (eql *filename-encoding* :null) :iso8859-1 View it on GitLab: https://gitlab.common-lisp.net/cmucl/cmucl/-/commit/144065ee00adb189d558ddd… -- View it on GitLab: https://gitlab.common-lisp.net/cmucl/cmucl/-/commit/144065ee00adb189d558ddd… You're receiving this email because of your account on gitlab.common-lisp.net.

1 0

[Git][cmucl/cmucl][issue-375-mkstemp-return-filename] 2 commits: Enforce 6 X's in template for mkstemp/mkdtemp
by Raymond Toy (＠rtoy) 03 Feb '25

03 Feb '25

Raymond Toy pushed to branch issue-375-mkstemp-return-filename at cmucl / cmucl Commits: 3b6909fe by Raymond Toy at 2025-02-03T07:26:09-08:00 Enforce 6 X's in template for mkstemp/mkdtemp Our API enforces the template to have exactly 6 X's at the end of the template. This is only a problem on Darwin which allows 0 or more X's. Also re-enable the tests for too few X's on Darwin and add tests for too many X's. - - - - - 2759e80f by Raymond Toy at 2025-02-03T07:28:50-08:00 Add tests for template that doesn't end in X's. - - - - - 2 changed files: - src/code/unix.lisp - tests/unix.lisp Changes: ===================================== src/code/unix.lisp ===================================== @@ -2904,6 +2904,15 @@ (function (* char)))) c-string)) +(defun check-template (template) + ;; Make sure the template ends with exactly 6 X's and no more. + (let ((last-non-x (position-if-not #'(lambda (c) + (char= c #\X)) + template + :from-end t))) + (and last-non-x + (= last-non-x (- (length template) 7))))) + (defun unix-mkstemp (template) _N"Generates a unique temporary file name from TEMPLATE, and creates and opens the file. On success, the corresponding file descriptor @@ -2911,6 +2920,10 @@ code is returned. The last six characters of the template must be \"XXXXXX\"." + ;; Make sure the template is valid. + (unless (check-template template) + (return-from unix-mkstemp + (values nil einval))) (let* ((format (if (eql *filename-encoding* :null) :iso8859-1 *filename-encoding*)) @@ -2942,6 +2955,10 @@ If the directory cannot be created NIL and the UNIX error code is returned." + ;; Make sure the template is valid. + (unless (check-template template) + (return-from unix-mkstemp + (values nil einval))) (let* ((format (if (eql *filename-encoding* :null) :iso8859-1 *filename-encoding*)) ===================================== tests/unix.lisp ===================================== @@ -47,18 +47,29 @@ (assert-false fd) (assert-true (and (integerp errno) (plusp errno))))) -;; Darwin allows any number of X's (including 0!) in the template but -;; Linux requires exactly 6. Hence skip this test. -#-darwin (define-test mkstemp.bad-template (:tag :issues) (multiple-value-bind (fd errno) (unix::unix-mkstemp "test-") ;; The template doesn't have enough X's so the FD should be NIL, ;; and a positive Unix errno value should be returned. - ;; - ;; Note that Darwin allows any number of X's (including 0!) in the - ;; template but Linux requires exactly 6. + (assert-false fd) + (assert-true (and (integerp errno) (plusp errno))))) + +(define-test mkstemp.bad-template.2 + (:tag :issues) + (multiple-value-bind (fd errno) + (unix::unix-mkstemp "test-XXXXXXX") + ;; The template has too many X's so the FD should be NIL, and a + ;; positive Unix errno value should be returned. + (assert-false fd) + (assert-true (and (integerp errno) (plusp errno))))) + +(define-test mkstemp.bad-template.3 + (:tag :issues) + (multiple-value-bind (fd errno) + (unix::unix-mkstemp "test-XXXXXXa") + ;; The template doesn't end in X's (assert-false fd) (assert-true (and (integerp errno) (plusp errno))))) @@ -100,9 +111,6 @@ (assert-false result) (assert-true (and (integerp errno) (plusp errno))))) -;; Darwin allows any number of X's (including 0!) in the template but -;; Linux requires exactly 6. Hence skip this test. -#-darwin (define-test mkdtemp.bad-template (:tag :issues) (multiple-value-bind (result errno) @@ -111,3 +119,19 @@ (assert-false result) (assert-true (and (integerp errno) (plusp errno))))) +(define-test mkdtemp.bad-template.2 + (:tag :issues) + (multiple-value-bind (result errno) + (unix::unix-mkdtemp "dir-XXXXXXX") + ;; Too many X's in template. + (assert-false result) + (assert-true (and (integerp errno) (plusp errno))))) + +(define-test mkdtemp.bad-template.2 + (:tag :issues) + (multiple-value-bind (result errno) + (unix::unix-mkdtemp "dir-XXXXXXa") + ;; Template doesn't end in X's + (assert-false result) + (assert-true (and (integerp errno) (plusp errno))))) + View it on GitLab: https://gitlab.common-lisp.net/cmucl/cmucl/-/compare/7275edf3ac0d0ab37d1dc7… -- View it on GitLab: https://gitlab.common-lisp.net/cmucl/cmucl/-/compare/7275edf3ac0d0ab37d1dc7… You're receiving this email because of your account on gitlab.common-lisp.net.

1 0

[Git][cmucl/cmucl][issue-375-mkstemp-return-filename] Disable bad template tests on Darwin
by Raymond Toy (＠rtoy) 03 Feb '25

03 Feb '25

Raymond Toy pushed to branch issue-375-mkstemp-return-filename at cmucl / cmucl Commits: 7275edf3 by Raymond Toy at 2025-02-02T16:01:15-08:00 Disable bad template tests on Darwin Darwin bizarrely allows 0(!) or more X's in the template, so our test for a bad template can't work on Darwin. Disable them. - - - - - 1 changed file: - tests/unix.lisp Changes: ===================================== tests/unix.lisp ===================================== @@ -47,6 +47,9 @@ (assert-false fd) (assert-true (and (integerp errno) (plusp errno))))) +;; Darwin allows any number of X's (including 0!) in the template but +;; Linux requires exactly 6. Hence skip this test. +#-darwin (define-test mkstemp.bad-template (:tag :issues) (multiple-value-bind (fd errno) @@ -54,9 +57,8 @@ ;; The template doesn't have enough X's so the FD should be NIL, ;; and a positive Unix errno value should be returned. ;; - ;; Note that Darwin allows any number of X's in the template but - ;; Linux requires exactly 6. Just test with no X's to handle all - ;; OSes. + ;; Note that Darwin allows any number of X's (including 0!) in the + ;; template but Linux requires exactly 6. (assert-false fd) (assert-true (and (integerp errno) (plusp errno))))) @@ -98,6 +100,9 @@ (assert-false result) (assert-true (and (integerp errno) (plusp errno))))) +;; Darwin allows any number of X's (including 0!) in the template but +;; Linux requires exactly 6. Hence skip this test. +#-darwin (define-test mkdtemp.bad-template (:tag :issues) (multiple-value-bind (result errno) View it on GitLab: https://gitlab.common-lisp.net/cmucl/cmucl/-/commit/7275edf3ac0d0ab37d1dc79… -- View it on GitLab: https://gitlab.common-lisp.net/cmucl/cmucl/-/commit/7275edf3ac0d0ab37d1dc79… You're receiving this email because of your account on gitlab.common-lisp.net.

1 0

[Git][cmucl/cmucl][issue-375-mkstemp-return-filename] 3 commits: Simplify mkdtemp and fix error in mkstemp
by Raymond Toy (＠rtoy) 02 Feb '25

02 Feb '25

Raymond Toy pushed to branch issue-375-mkstemp-return-filename at cmucl / cmucl Commits: 544a6787 by Raymond Toy at 2025-02-01T15:59:09-08:00 Simplify mkdtemp and fix error in mkstemp Since mkdtemp returns a pointer to the resulting directory name, we can convert that to the desired lisp string. In mkstemp, we were converting the resulting buffer to a string but we had `:end (1- length)`. It should be `:end length` to get all the characters. - - - - - 0111bb50 by Raymond Toy at 2025-02-01T16:34:20-08:00 Add tests with template with non-ascii char Add a test for mkstemp and mkdtemp where the template name contains a non-ascii char (lower case alpha) to test the we convert the returned name to a Lisp string correctly. - - - - - 6e0378ea by Raymond Toy at 2025-02-01T16:35:27-08:00 Need to convert the return result of mkdtemp Forgot that we need to encode the result to a proper UTF-16 Lisp string. - - - - - 2 changed files: - src/code/unix.lisp - tests/unix.lisp Changes: ===================================== src/code/unix.lisp ===================================== @@ -2930,7 +2930,7 @@ ;; Convert the array of octets in BUFFER back to ;; a Lisp string. (stream:octets-to-string buffer - :end (1- length) + :end length :external-format format)) (sys:vector-sap buffer)))) @@ -2965,6 +2965,4 @@ ;; resulting name. Otherwise, return NIL and the errno. (if (null-alien result) (values nil (unix-errno)) - (values (stream:octets-to-string buffer - :end (1- length) - :external-format format)))))) + (%file->name (cast result c-call:c-string)))))) ===================================== tests/unix.lisp ===================================== @@ -17,6 +17,26 @@ (when fd (unix:unix-unlink name))))) +(define-test mkstemp.name-returned.2 + (:tag :issues) + (let ((unix::*filename-encoding* :utf-8) + fd name) + (unwind-protect + (progn + ;; Temp name starts with a lower case alpha character. + (let* ((template (concatenate 'string (string #\u+3b1) + "test-XXXXXX")) + (x-posn (position #\X template))) + (multiple-value-setq (fd name) + (unix::unix-mkstemp template)) + (assert-true fd) + (assert-false (search "XXXXXX" name) + name) + (assert-true (string= name template :end1 x-posn :end2 x-posn) + name))) + (when fd + (unix:unix-unlink name))))) + (define-test mkstemp.bad-path (:tag :issues) (multiple-value-bind (fd errno) @@ -52,6 +72,25 @@ (when name (unix:unix-rmdir name))))) +(define-test mkdtemp.name-returned.2 + (:tag :issues) + (let ((unix::*filename-encoding* :utf-8) + name) + (unwind-protect + (progn + ;; Temp name starts with a lower case alpha character. + (let* ((template (concatenate 'string (string #\u+3b1) + "dir-XXXXXX")) + (x-posn (position #\X template))) + (setf name (unix::unix-mkdtemp template)) + ;; Verify that the dir name no longer has X's. + (assert-true (stringp name)) + (assert-false (search "XXXXXX" name)) + (assert-true (string= name template :end1 x-posn :end2 x-posn) + name x-posn))) + (when name + (unix:unix-rmdir name))))) + (define-test mkdtemp.bad-path (:tag :issues) (multiple-value-bind (result errno) View it on GitLab: https://gitlab.common-lisp.net/cmucl/cmucl/-/compare/15501db4886ad26a943e1c… -- View it on GitLab: https://gitlab.common-lisp.net/cmucl/cmucl/-/compare/15501db4886ad26a943e1c… You're receiving this email because of your account on gitlab.common-lisp.net.

1 0

[Git][cmucl/cmucl][issue-375-mkstemp-return-filename] Update bad template tests for Darwin
by Raymond Toy (＠rtoy) 01 Feb '25

01 Feb '25

Raymond Toy pushed to branch issue-375-mkstemp-return-filename at cmucl / cmucl Commits: 15501db4 by Raymond Toy at 2025-02-01T09:11:11-08:00 Update bad template tests for Darwin Darwin seems to require at least one X in the template, unlike Linux which requires 6. Update the test for bad templates for both mkstemp and mkdtemp to have no X's in the template. - - - - - 1 changed file: - tests/unix.lisp Changes: ===================================== tests/unix.lisp ===================================== @@ -30,9 +30,13 @@ (define-test mkstemp.bad-template (:tag :issues) (multiple-value-bind (fd errno) - (unix::unix-mkstemp "test-XXXXX") + (unix::unix-mkstemp "test-") ;; The template doesn't have enough X's so the FD should be NIL, ;; and a positive Unix errno value should be returned. + ;; + ;; Note that Darwin allows any number of X's in the template but + ;; Linux requires exactly 6. Just test with no X's to handle all + ;; OSes. (assert-false fd) (assert-true (and (integerp errno) (plusp errno))))) @@ -58,7 +62,8 @@ (define-test mkdtemp.bad-template (:tag :issues) (multiple-value-bind (result errno) - (unix::unix-mkdtemp "dir-XXXXX") + (unix::unix-mkdtemp "dir-") + ;; No X's in template, like for mkstemp.bad-template test. (assert-false result) (assert-true (and (integerp errno) (plusp errno))))) View it on GitLab: https://gitlab.common-lisp.net/cmucl/cmucl/-/commit/15501db4886ad26a943e1c0… -- View it on GitLab: https://gitlab.common-lisp.net/cmucl/cmucl/-/commit/15501db4886ad26a943e1c0… You're receiving this email because of your account on gitlab.common-lisp.net.

1 0

[Git][cmucl/cmucl] Pushed new branch issue-375-mkstemp-return-filename
by Raymond Toy (＠rtoy) 01 Feb '25

01 Feb '25

Raymond Toy pushed new branch issue-375-mkstemp-return-filename at cmucl / cmucl -- View it on GitLab: https://gitlab.common-lisp.net/cmucl/cmucl/-/tree/issue-375-mkstemp-return-… You're receiving this email because of your account on gitlab.common-lisp.net.

1 0

[Git][cmucl/cmucl][master] Update release notes to add issue 367
by Raymond Toy (＠rtoy) 31 Jan '25

31 Jan '25

Raymond Toy pushed to branch master at cmucl / cmucl Commits: 739de261 by Raymond Toy at 2025-01-31T06:57:45-08:00 Update release notes to add issue 367 - - - - - 1 changed file: - src/general-info/release-21f.md Changes: ===================================== src/general-info/release-21f.md ===================================== @@ -113,6 +113,7 @@ public domain. * ~~#361~~ Add herald item to mention where to report issues * ~~#362~~ Simplify "library:" search-list * ~~#364~~ Add interface to `mkdtemp` and `mkstemp` + * ~~#367~~ Add stream:string-count-octets to count octets in a string * ~~#369~~ Improve docstring for `unix::unix-setlocale` * Other changes: * Improvements to the PCL implementation of CLOS: View it on GitLab: https://gitlab.common-lisp.net/cmucl/cmucl/-/commit/739de26192809e9f9a159ab… -- View it on GitLab: https://gitlab.common-lisp.net/cmucl/cmucl/-/commit/739de26192809e9f9a159ab… You're receiving this email because of your account on gitlab.common-lisp.net.

1 0

[Git][cmucl/cmucl][master] 2 commits: Fix #367: Add method to count number of octets to encode a string
by Raymond Toy (＠rtoy) 31 Jan '25

31 Jan '25

Raymond Toy pushed to branch master at cmucl / cmucl Commits: fd391099 by Raymond Toy at 2025-01-31T14:39:40+00:00 Fix #367: Add method to count number of octets to encode a string - - - - - 12e15d8b by Raymond Toy at 2025-01-31T14:39:40+00:00 Merge branch 'issue-367-count-octets-for-encoding' into 'master' Fix #367: Add method to count number of octets to encode a string Closes #367 See merge request cmucl/cmucl!264 - - - - - 18 changed files: - .gitlab-ci.yml - + src/bootfiles/21e/boot-2024-08.lisp - src/code/exports.lisp - src/code/extfmts.lisp - src/i18n/locale/cmucl.pot - src/pcl/simple-streams/external-formats/ascii.lisp - src/pcl/simple-streams/external-formats/euc-kr.lisp - src/pcl/simple-streams/external-formats/iso8859-1.lisp - src/pcl/simple-streams/external-formats/iso8859-2.lisp - src/pcl/simple-streams/external-formats/mac-roman.lisp - src/pcl/simple-streams/external-formats/utf-16-be.lisp - src/pcl/simple-streams/external-formats/utf-16-le.lisp - src/pcl/simple-streams/external-formats/utf-16.lisp - src/pcl/simple-streams/external-formats/utf-32-be.lisp - src/pcl/simple-streams/external-formats/utf-32-le.lisp - src/pcl/simple-streams/external-formats/utf-32.lisp - src/pcl/simple-streams/external-formats/utf-8.lisp - + tests/external-formats.lisp Changes: ===================================== .gitlab-ci.yml ===================================== @@ -1,7 +1,7 @@ variables: download_url: "https://common-lisp.net/project/cmucl/downloads/snapshots/2024/08" version: "2024-08-x86" - bootstrap: "" + bootstrap: "-B boot-2024-08" stages: ===================================== src/bootfiles/21e/boot-2024-08.lisp ===================================== @@ -0,0 +1,10 @@ +;; Bootstrap file for the ef-octet-count changes. Just need to change +;; the value of +ef-max+ + +(in-package "STREAM") + +(handler-bind + ((error (lambda (c) + (declare (ignore c)) + (invoke-restart 'continue)))) + (defconstant +ef-max+ 14)) ===================================== src/code/exports.lisp ===================================== @@ -1456,7 +1456,8 @@ (name '("STRING-TO-OCTETS" "OCTETS-TO-STRING" "*DEFAULT-EXTERNAL-FORMAT*" "STRING-ENCODE" "STRING-DECODE" "SET-SYSTEM-EXTERNAL-FORMAT" - "LIST-ALL-EXTERNAL-FORMATS" "DESCRIBE-EXTERNAL-FORMAT")) + "LIST-ALL-EXTERNAL-FORMATS" "DESCRIBE-EXTERNAL-FORMAT" + "STRING-OCTET-COUNT")) (intern name "STREAM")) (defpackage "EXTENSIONS" ===================================== src/code/extfmts.lisp ===================================== @@ -19,7 +19,8 @@ string-encode string-decode set-system-external-format +replacement-character-code+ list-all-external-formats - describe-external-format)) + describe-external-format + string-octet-count)) (defvar *default-external-format* :utf-8 @@ -52,6 +53,7 @@ flush ; flush state copy-state ; copy state osc ; octets to string, counted + oc ; number of octets to encode string max) ;; Unicode replacement character U+FFFD @@ -96,6 +98,11 @@ (copy-state nil :type (or null function) :read-only t) (cache nil :type (or null simple-vector)) ;; + ;; Function to count the number of octets needed to encode a + ;; codepoint. Basically like code-to-octets, except we return the + ;; number of octets needed instead of the octets themselves. + (octet-count #'%efni :type (or null function) :read-only t) + ;; ;; Minimum number of octets needed to form a codepoint (min 1 :type kernel:index :read-only t) ;; @@ -126,7 +133,8 @@ (setf (gethash (ef-name ef) *external-formats*) ef)) (declaim (inline ef-octets-to-code ef-code-to-octets ef-flush-state ef-copy-state - ef-cache ef-min-octets ef-max-octets)) + ef-cache ef-min-octets ef-max-octets + ef-octet-count)) (defun ef-octets-to-code (ef) (efx-octets-to-code (ef-efx ef))) @@ -143,6 +151,9 @@ (defun ef-cache (ef) (efx-cache (ef-efx ef))) +(defun ef-octet-count (ef) + (efx-octet-count (ef-efx ef))) + (defun ef-min-octets (ef) (efx-min (ef-efx ef))) @@ -166,7 +177,7 @@ ;;; DEFINE-EXTERNAL-FORMAT -- Public ;;; ;;; name (&key base min max size documentation) (&rest slots) octets-to-code -;;; code-to-octets flush-state copy-state +;;; code-to-octets flush-state copy-state octet-count ;;; ;;; Define a new external format. If base is specified, then an ;;; external format is defined that is based on a previously defined @@ -228,6 +239,15 @@ ;;; This should probably be a deep copy so that if the original ;;; state is modified, the copy is not. ;;; +;;; octet-count (code state error &rest vars) +;;; Defines a form to determine the number of octets needed to +;;; encode the given CODE using the external format. This is +;;; essentially the same as CODE-TO-OCTETS, except the encoding is +;;; not saved anywhere. ERROR is the same as in CODE-TO-OCTETS. +;;; +;;; This should return one value: the number of octets needed to +;;; encode the given code. +;;; ;;; Note: external-formats work on code-points, not ;;; characters, so that the entire 31 bit ISO-10646 range can be ;;; used internally regardless of the size of a character recognized @@ -238,7 +258,7 @@ (defmacro define-external-format (name (&key base min max size (documentation "")) (&rest slots) &optional octets-to-code code-to-octets - flush-state copy-state) + flush-state copy-state octet-count) (let* ((tmp (gensym)) (min (or min size 1)) (max (or max size 6)) @@ -282,7 +302,17 @@ (declare (ignorable ,state)) (let (,@',slotb ,@(loop for var in vars collect `(,var (gensym)))) - ,body)))) + ,body))) + (octet-count ((code state error &rest vars) body) + `(lambda (,',tmp ,state ,error) + (declare (ignorable ,state ,error) + (optimize (ext:inhibit-warnings 3))) + (let (,@',slotb + (,code ',code) + ,@(loop for var in vars collect `(,var (gensym)))) + `(let ((,',code (the lisp:codepoint ,,',tmp))) + (declare (ignorable ,',code)) + ,,body))))) (%intern-ef (make-external-format ,name ,(if base `(ef-efx (find-external-format ,(ef-name base))) @@ -291,7 +321,8 @@ :flush-state ,flush-state :copy-state ,copy-state :cache (make-array +ef-max+ - :initial-element nil) + :initial-element nil) + :octet-count ,octet-count :min ,(min min max) :max ,(max min max))) nil @@ -688,7 +719,20 @@ character and illegal outputs are replaced by a question mark.") (intl:gettext "Cannot output codepoint #x~X to ISO8859-1 stream") ,code 1)) #x3F) - ,code)))) + ,code))) + () + () + (octet-count (code state error) + `(if (> ,code 255) + (if ,error + (locally + ;; No warnings about fdefinition + (declare (optimize (ext:inhibit-warnings 3))) + (funcall ,error + (intl:gettext "Cannot output codepoint #x~X to ISO8859-1 stream") + ,code 1)) + 1) + 1))) ;;; OCTETS-TO-CODEPOINT, CODEPOINT-TO-OCTETS -- Semi-Public ;;; @@ -709,6 +753,10 @@ character and illegal outputs are replaced by a question mark.") (let ((ef (find-external-format external-format))) (funcall (ef-code-to-octets ef) code state output error))) +(defmacro count-codepoint-octets (external-format code state &optional error) + (let ((ef (find-external-format external-format))) + (funcall (ef-octet-count ef) code state error))) + (defvar *ef-base* +ef-max+) @@ -878,6 +926,42 @@ character and illegal outputs are replaced by a question mark.") (when f (funcall f state)))) +(defmacro octet-count (external-format char state &optional error) + (let ((nchar (gensym)) + (nstate (gensym)) + (count-it (gensym)) + (ch (gensym))) + `(let ((,nchar ,char) + (,nstate ,state)) + (when (null ,nstate) (setq ,nstate (setf ,state (cons nil nil)))) + (if (lisp::surrogatep (char-code ,nchar) :high) + (setf (car ,nstate) ,nchar) + (flet ((,count-it (,ch) + (count-codepoint-octets ,external-format ,ch (cdr ,nstate) ,error))) + (if (car ,nstate) + (prog1 + (,count-it (if (lisp::surrogatep (char-code ,nchar) :low) + (surrogates-to-codepoint (car ,nstate) ,nchar) + (if ,error + (locally + (declare (optimize (ext:inhibit-warnings 3))) + (funcall ,error + (intl:gettext "Cannot convert invalide surrogate #~x~X to character") + ,nchar)) + +replacement-character-code+))) + (setf (car ,nstate) nil)) + ;; A lone trailing (low surrogate gets replaced with + ;; the replacement character. + (,count-it (if (lisp::surrogatep (char-code ,nchar) :low) + (if ,error + (locally + (declare (optimize (ext:inhibit-warnings 3))) + (funcall ,error + (intl:gettext "Cannot convert lone trailing surrogate #x~X to character") + ,nchar)) + +replacement-character-code+) + (char-code ,nchar))))))))) + (def-ef-macro ef-string-to-octets (extfmt lisp::lisp +ef-max+ +ef-so+) `(lambda (string start end buffer buffer-start buffer-end error bufferp &aux (ptr buffer-start) (state nil) (last-octet buffer-start)) @@ -1071,6 +1155,31 @@ character and illegal outputs are replaced by a question mark.") (values (if stringp string (lisp::shrink-vector string pos)) (- pos s-start) last-octet new-state)))) +(def-ef-macro ef-string-octet-count (extfmt lisp::lisp +ef-max+ +ef-oc+) + `(lambda (string start end error &aux (total 0) (state nil)) + (dotimes (i (- end start) total) + (incf total + (octet-count ,extfmt (schar string (+ start i)) state error))))) + +(defun string-octet-count (string &key (start 0) end (external-format :default) error) + "Compute the number of octets needed to convert String using the + specified External-format. The string is bound by Start (defaulting + to 0) and End (defaulting to the end of the string)." + (let ((composing-format-p + ;; Determine is the external format is a composing format + ;; which we determine by seeing that the name of the format + ;; is a cons. Probably not the best way. + (consp (ef-name (find-external-format external-format))))) + ;; We currently don't know how to get just the number of octets + ;; when a composing external format is used. As a workaround, use + ;; STRING-TO-OCTETS to find the number of octets. + (if composing-format-p + (nth-value 1 + (string-to-octets string :start start :end end + :external-format external-format)) + (lisp::with-array-data ((string string) (start start) (end end)) + (funcall (ef-string-octet-count external-format) + string start end error))))) (def-ef-macro ef-encode (extfmt lisp::lisp +ef-max+ +ef-en+) `(lambda (string start end result error &aux (ptr 0) (state nil)) @@ -1186,10 +1295,11 @@ character and illegal outputs are replaced by a question mark.") (#.+ef-so+ (%ef-string-to-octets ef)) (#.+ef-en+ (%ef-encode ef)) (#.+ef-de+ (%ef-decode ef)) - (#.+ef-osc+ (%ef-octets-to-string-counted ef)))) + (#.+ef-osc+ (%ef-octets-to-string-counted ef)) + (#.+ef-oc+ (%ef-octet-count ef)))) `(setf (aref (ef-cache (find-external-format ,(ef-name ef))) ,slot) ,(subst (ef-name ef) ef - (function-lambda-expression (aref (ef-cache ef) slot)))))) + (function-lambda-expression (aref (ef-cache ef) slot))))))) ;;; Builtin external formats. @@ -1307,7 +1417,17 @@ replacement character.") ((< ,code #x800) (utf8 ,code 1)) ((< ,code #x10000) (utf8 ,code 2)) ((< ,code #x110000) (utf8 ,code 3)) - (t (error "How did this happen? Codepoint U+~X is illegal" ,code)))))) + (t (error "How did this happen? Codepoint U+~X is illegal" ,code))))) + () + () + (octet-count (code state error) + `(locally + (declare (optimize (ext:inhibit-warnings 3))) + (cond ((< ,code #x80) 1) + ((< ,code #x800) 2) + ((< ,code #x10000) 3) + ((< ,code #x110000) 4) + (t (error "How did this happen? Codepoint U+~X is illegal" ,code)))))) (define-external-format :ascii (:size 1 :documentation "US ASCII 7-bit encoding. Illegal input sequences are replaced with @@ -1333,4 +1453,14 @@ replaced with a question mark.") (declare (optimize (ext:inhibit-warnings 3))) (funcall ,error "Cannot output codepoint #x~X to ASCII stream" ,code)) #x3F) - ,code)))) + ,code))) + () + () + (octet-count (code state error) + `(if (> ,code #x7f) + (if ,error + (locally + (declare (optimize (ext:inhibit-warnings 3))) + (funcall ,error "Cannot output codepoint #x~X to ASCII stream" ,code)) + 1) + 1))) ===================================== src/i18n/locale/cmucl.pot ===================================== @@ -9300,6 +9300,13 @@ msgid "" " external format." msgstr "" +#: src/code/extfmts.lisp +msgid "" +"Compute the number of octets needed to convert String using the\n" +" specified External-format. The string is bound by Start (defaulting\n" +" to 0) and End (defaulting to the end of the string)." +msgstr "" + #: src/code/extfmts.lisp msgid "" "Encode the given String using External-Format and return a new\n" ===================================== src/pcl/simple-streams/external-formats/ascii.lisp ===================================== @@ -33,5 +33,15 @@ replaced with a question mark.") (declare (optimize (ext:inhibit-warnings 3))) (funcall ,error "Cannot output codepoint #x~X to ASCII stream" ,code)) #x3F) - ,code)))) + ,code))) + () + () + (octet-count (code state error) + `(if (> ,code #x7f) + (if ,error + (locally + (declare (optimize (ext:inhibit-warnings 3))) + (funcall ,error "Cannot output codepoint #x~X to ASCII stream" ,code)) + 1) + 1))) ===================================== src/pcl/simple-streams/external-formats/euc-kr.lisp ===================================== @@ -1007,4 +1007,16 @@ character and illegal outputs are replaced by a question mark.") (t (if ,error (funcall ,error "Cannot output codepoint #x~X to EUC-KR format." ,code) - (,output #X3f)))))))) + (,output #X3f))))))) + () + () + (octet-count (code state error present) + `(if (<= ,code #x7f) + 1 + (let ((,present (get-inverse ,itable ,code))) + (cond (,present + 2) + (t + (if ,error + (funcall ,error "Cannot output codepoint #x~X to EUC-KR format." ,code) + 1))))))) ===================================== src/pcl/simple-streams/external-formats/iso8859-1.lisp ===================================== @@ -31,4 +31,17 @@ character and illegal outputs are replaced by a question mark.") (funcall ,error "Cannot output codepoint #x~X to ISO8859-1 stream" ,code 1)) #x3F) - ,code)))) + ,code))) + () + () + (octet-count (code state error) + `(if (> ,code 255) + (if ,error + (locally + ;; No warnings about fdefinition + (declare (optimize (ext:inhibit-warnings 3))) + (funcall ,error + (intl:gettext "Cannot output codepoint #x~X to ISO8859-1 stream") + ,code 1)) + 1) + 1))) ===================================== src/pcl/simple-streams/external-formats/iso8859-2.lisp ===================================== @@ -47,4 +47,19 @@ character and illegal outputs are replaced by a question mark.") (declare (optimize (ext:inhibit-warnings 3))) (funcall ,error "Cannot output codepoint #x~X to ISO8859-2 stream" ,code)) - #x3F))))))) + #x3F)))))) + () + () + (octet-count (code state error present) + `(if (< ,code 160) + 1 + (let ((,present (get-inverse ,itable ,code))) + (if ,present + 1 + (if ,error + (locally + ;; No warnings about fdefinition + (declare (optimize (ext:inhibit-warnings 3))) + (funcall ,error "Cannot output codepoint #x~X to ISO8859-2 stream" + ,code)) + 1)))))) ===================================== src/pcl/simple-streams/external-formats/mac-roman.lisp ===================================== @@ -49,4 +49,19 @@ character and illegal outputs are replaced by a question mark.") (declare (optimize (ext:inhibit-warnings 3))) (funcall ,error "Cannot output codepoint #x~X to MAC-ROMAN stream" ,code)) - #x3F))))))) + #x3F)))))) + () + () + (octet-count (code state error present) + `(if (< ,code 128) + 1 + (let ((,present (get-inverse ,itable ,code))) + (if ,present + 1 + (if ,error + (locally + ;; No warnings about fdefinition + (declare (optimize (ext:inhibit-warnings 3))) + (funcall ,error "Cannot output codepoint #x~X to MAC-ROMAN stream" + ,code)) + 1)))))) ===================================== src/pcl/simple-streams/external-formats/utf-16-be.lisp ===================================== @@ -110,4 +110,12 @@ Unicode replacement character.") (copy-state (state) ;; The state is either NIL or a codepoint, so nothing really ;; special is needed to copy it. - `(progn ,state))) + `(progn ,state)) + (octet-count (code state error) + `(cond ((< ,code #x10000) + 2) + ((< ,code #x110000) + 4) + (t + ;; Replacement character is 2 octets + 2)))) ===================================== src/pcl/simple-streams/external-formats/utf-16-le.lisp ===================================== @@ -111,4 +111,12 @@ Unicode replacement character.") (copy-state (state) ;; The state is either NIL or a codepoint, so nothing really ;; special is needed. - `(progn ,state))) + `(progn ,state)) + (octet-count (code state error) + `(cond ((< ,code #x10000) + 2) + ((< ,code #x110000) + 4) + (t + ;; Replacement character is 2 octets + 2)))) ===================================== src/pcl/simple-streams/external-formats/utf-16.lisp ===================================== @@ -156,4 +156,18 @@ Unicode replacement character.") ,c)))))) (copy-state (state) ;; The state is list. Copy it - `(copy-list ,state))) + `(copy-list ,state)) + (octet-count (code state error) + `(let ((bom-count 0)) + (unless ,state + ;; Output BOM + (setf bom-count 2) + (setf ,state t)) + (+ bom-count + (cond ((< ,code #x10000) + 2) + ((< ,code #x110000) + 4) + (t + ;; Replacement character is 2 octets + 2)))))) ===================================== src/pcl/simple-streams/external-formats/utf-32-be.lisp ===================================== @@ -61,4 +61,18 @@ Unicode replacement character.") ,code)) +replacement-character-code+))) (t - (out ,code)))))) + (out ,code))))) + () + () + (octet-count (code state error) + `(cond ((lisp::surrogatep ,code) + (if ,error + (locally + ;; No warnings about fdefinition + (declare (optimize (ext:inhibit-warnings 3))) + (funcall ,error "Surrogate code #x~4,'0X is illegal for UTF32 output" + ,code)) + ;; Replacement character is 2 octets + 2)) + (t + 4)))) ===================================== src/pcl/simple-streams/external-formats/utf-32-le.lisp ===================================== @@ -62,4 +62,18 @@ Unicode replacement character.") ,code)) +replacement-character-code+))) (t - (out ,code)))))) + (out ,code))))) + () + () + (octet-count (code state error) + `(cond ((lisp::surrogatep ,code) + (if ,error + (locally + ;; No warnings about fdefinition + (declare (optimize (ext:inhibit-warnings 3))) + (funcall ,error "Surrogate code #x~4,'0X is illegal for UTF32 output" + ,code)) + ;; Replacement character is 2 octets + 2)) + (t + 4)))) ===================================== src/pcl/simple-streams/external-formats/utf-32.lisp ===================================== @@ -114,4 +114,20 @@ Unicode replacement character.") nil (copy-state (state) ;; The state is either NIL or T, so we can just return that. - `(progn ,state))) + `(progn ,state)) + (octet-count (code state error) + `(let ((bom-count 0)) + (unless ,state + (setf bom-count 4) + (setf ,state t)) + (cond ((lisp::surrogatep ,code) + (if ,error + (locally + ;; No warnings about fdefinition + (declare (optimize (ext:inhibit-warnings 3))) + (funcall ,error "Surrogate code #x~4,'0X is illegal for UTF32 output" + ,code)) + ;; Replacement character is 2 octets + (+ 2 bom-count))) + (t + (+ 4 bom-count)))))) ===================================== src/pcl/simple-streams/external-formats/utf-8.lisp ===================================== @@ -127,4 +127,14 @@ replacement character.") ((< ,code #x800) (utf8 ,code 1)) ((< ,code #x10000) (utf8 ,code 2)) ((< ,code #x110000) (utf8 ,code 3)) - (t (error "How did this happen? Codepoint U+~X is illegal" ,code)))))) + (t (error "How did this happen? Codepoint U+~X is illegal" ,code))))) + () + () + (octet-count (code state error) + `(locally + (declare (optimize (ext:inhibit-warnings 3))) + (cond ((< ,code #x80) 1) + ((< ,code #x800) 2) + ((< ,code #x10000) 3) + ((< ,code #x110000) 4) + (t (error "How did this happen? Codepoint U+~X is illegal" ,code)))))) ===================================== tests/external-formats.lisp ===================================== @@ -0,0 +1,135 @@ +;;; Tests for external formats + +(defpackage :external-formats-tests + (:use :cl :lisp-unit)) + +(in-package "EXTERNAL-FORMATS-TESTS") + +(defparameter *test-iso8859-1* + (let ((rs (kernel::make-random-object :state (kernel::init-random-state 27182828)))) + (lisp::codepoints-string + (loop for k from 0 below 1000 + collect (random 256 rs)))) + "Random test string with ISO8859-1 characters") + +(defparameter *test-unicode* + (let ((rs (kernel::make-random-object :state (kernel::init-random-state 27182828)))) + (lisp::codepoints-string + (loop for k from 0 below 1000 + collect (random 20000 rs)))) + "Random test string with codepoints below 20000") + + + +(defmacro test-octet-count (string format) + "Test that STRING-OCTET-COUNT returns the correct number of octets" + ;; We expect STRING-OCTET-COUNT returns the same number of octets + ;; that are produced by STRING-TO-OCTETS. + `(multiple-value-bind (octets count converted) + (stream:string-to-octets ,string :external-format ,format) + ;; While we're at it, make sure that the length of the octet + ;; buffer matches returned count. And make sure we converted all + ;; the characters in the string. + (assert-equal (length octets) count) + (assert-equal (length ,string) converted) + ;; Finally, make sure that STRING-OCTET-COUNT returns the same + ;; number of octets from STRING-TO-OCTETS. + (assert-equal (length octets) + (stream::string-octet-count ,string :external-format ,format)))) + +(define-test octet-count.iso8859-1 + (:tag :octet-count) + (test-octet-count *test-iso8859-1* :iso8859-1)) + +(define-test octet-count.ascii + (:tag :octet-count) + (test-octet-count *test-iso8859-1* :ascii)) + +(define-test octet-count.ascii.error + (:tag :octet-count) + (assert-error 'simple-error + (stream::string-octet-count *test-iso8859-1* + :external-format :ascii + :error 'error))) + +(define-test octet-count.utf-8 + (:tag :octet-count) + (test-octet-count *test-unicode* :utf-8)) + +(define-test octet-count.utf-16 + (:tag :octet-count) + (test-octet-count *test-unicode* :utf-16)) + +(define-test octet-count.utf-16-be + (:tag :octet-count) + (test-octet-count *test-unicode* :utf-16-be)) + +(define-test octet-count.utf-16-le + (:tag :octet-count) + (test-octet-count *test-unicode* :utf-16-le)) + +(define-test octet-count.utf-32 + (:tag :octet-count) + (test-octet-count *test-unicode* :utf-32)) + +(define-test octet-count.utf-32-le + (:tag :octet-count) + (test-octet-count *test-unicode* :utf-32-le)) + +(define-test octet-count.utf-32-le + (:tag :octet-count) + (test-octet-count *test-unicode* :utf-32-le)) + +(define-test octet-count.euc-kr + (:tag :octet-count) + (test-octet-count *test-unicode* :euc-kr)) + +(define-test octet-count.iso8859-2 + (:tag :octet-count) + (test-octet-count *test-iso8859-1* :iso8859-2)) + +(define-test octet-count.iso8859-3 + (:tag :octet-count) + (test-octet-count *test-iso8859-1* :iso8859-3)) + +(define-test octet-count.iso8859-4 + (:tag :octet-count) + (test-octet-count *test-iso8859-1* :iso8859-4)) + +(define-test octet-count.iso8859-5 + (:tag :octet-count) + (test-octet-count *test-iso8859-1* :iso8859-5)) + +(define-test octet-count.iso8859-6 + (:tag :octet-count) + (test-octet-count *test-iso8859-1* :iso8859-6)) + +(define-test octet-count.iso8859-7 + (:tag :octet-count) + (test-octet-count *test-iso8859-1* :iso8859-7)) + +(define-test octet-count.iso8859-8 + (:tag :octet-count) + (test-octet-count *test-iso8859-1* :iso8859-8)) + +(define-test octet-count.iso8859-10 + (:tag :octet-count) + (test-octet-count *test-iso8859-1* :iso8859-10)) + +(define-test octet-count.iso8859-13 + (:tag :octet-count) + (test-octet-count *test-iso8859-1* :iso8859-13)) + +(define-test octet-count.iso8859-14 + (:tag :octet-count) + (test-octet-count *test-iso8859-1* :iso8859-14)) + +(define-test octet-count.iso8859-15 + (:tag :octet-count) + (test-octet-count *test-iso8859-1* :iso8859-15)) + +(define-test octet-count.mac-roman + (:tag :octet-count) + (test-octet-count *test-iso8859-1* :mac-roman)) + + View it on GitLab: https://gitlab.common-lisp.net/cmucl/cmucl/-/compare/bf461224aaa80f802a5f40… -- View it on GitLab: https://gitlab.common-lisp.net/cmucl/cmucl/-/compare/bf461224aaa80f802a5f40… You're receiving this email because of your account on gitlab.common-lisp.net.

1 0

[Git][cmucl/cmucl][issue-365-add-strerror] 2 commits: Building unix-errno.lisp is independent of of the -b option
by Raymond Toy (＠rtoy) 31 Jan '25

31 Jan '25

Raymond Toy pushed to branch issue-365-add-strerror at cmucl / cmucl Commits: 0628c87a by Raymond Toy at 2025-01-30T17:19:07-08:00 Building unix-errno.lisp is independent of of the -b option Previously, we were setting `GEN_ERROR` if `$BASE` was not set (via the `-b` option). This is incorrect. The choice is independent of the value of the `$BASE` setting. Add a separate test for this. - - - - - fe12bd37 by Raymond Toy at 2025-01-30T17:22:00-08:00 Make code for setting of GEN_ERROR the same as in build.sh - - - - - 2 changed files: - bin/build.sh - bin/cross-build-world.sh Changes: ===================================== bin/build.sh ===================================== @@ -156,8 +156,6 @@ buildit () BUILDWORLD="$TOOLDIR/build-world.sh" BUILD_POT="yes" UPDATE_TRANS= -# Set to "yes" if we auto-generate code/unix-errno.lisp. -GEN_ERRNO= while getopts "123PRGo:b:v:uB:C:Ui:w:O:?" arg do @@ -198,7 +196,6 @@ if [ -z "$BASE" ]; then i86pc) BASE=sol-x86 ;; esac ;; Linux) - GEN_ERRNO=yes BASE=linux ;; # Add support for FreeBSD and NetBSD? Otherwise default to just build. *) BASE=build ;; @@ -207,6 +204,19 @@ fi echo base = $BASE +# Determine if we need to generate unix-errno.lisp + +# Set to "yes" if we auto-generate code/unix-errno.lisp. +GEN_ERRNO= + +case `uname -s` in + # Add more cases as we support more OSes + Linux) + GEN_ERRNO=yes + ;; +esac + + bootfiles_dir=$SRCDIR/bootfiles/$version if [ -n "$bootfiles" ]; then for file in $bootfiles; do ===================================== bin/cross-build-world.sh ===================================== @@ -14,8 +14,6 @@ usage() { MAKE=make INTERACTIVE=nil BREAK="" -# Set to "yes" if we auto-generate code/unix-errno.lisp. -GEN_ERRNO= while getopts "cirlXB:G:" arg do @@ -61,15 +59,14 @@ CROSS="`echo $2 | sed 's:/*$::'`" SCRIPT="$3" LISP="${4:-lisp}" +# Determine if we need to generate unix-errno.lisp +# Set to "yes" if we auto-generate code/unix-errno.lisp. +GEN_ERRNO= case `uname -s` in - Darwin) - ;; - SunOS) - ;; + # Add more cases as we support more OSes Linux) GEN_ERRNO=yes ;; - *) ;; esac if [ -z "$BOOTSTRAP" ]; then View it on GitLab: https://gitlab.common-lisp.net/cmucl/cmucl/-/compare/3cccf95a5ccbe09b22b6fb… -- View it on GitLab: https://gitlab.common-lisp.net/cmucl/cmucl/-/compare/3cccf95a5ccbe09b22b6fb… You're receiving this email because of your account on gitlab.common-lisp.net.

1 0

[Git][cmucl/cmucl][issue-365-add-strerror] Fix typo: Missing closing ";;"
by Raymond Toy (＠rtoy) 31 Jan '25

31 Jan '25

Raymond Toy pushed to branch issue-365-add-strerror at cmucl / cmucl Commits: 3cccf95a by Raymond Toy at 2025-01-30T16:04:32-08:00 Fix typo: Missing closing ";;" - - - - - 1 changed file: - bin/cross-build-world.sh Changes: ===================================== bin/cross-build-world.sh ===================================== @@ -68,6 +68,7 @@ case `uname -s` in ;; Linux) GEN_ERRNO=yes + ;; *) ;; esac View it on GitLab: https://gitlab.common-lisp.net/cmucl/cmucl/-/commit/3cccf95a5ccbe09b22b6fbf… -- View it on GitLab: https://gitlab.common-lisp.net/cmucl/cmucl/-/commit/3cccf95a5ccbe09b22b6fbf… You're receiving this email because of your account on gitlab.common-lisp.net.

1 0