On Tue, 01 Sep 2015 07:25:37 +0200 Daniel Kochmański daniel@turtleware.eu wrote:

btw, it doesn't hang on my machine (runs ~30m already, if it'll fail I'll let you know). Could you provide more details? (OS, *features*, number of cores etc.)

I also have been trying to reproduce it on NetBSD/amd64 with 4 cores i5 with the ECL develop branch HEAD, without really observing it hanging. However, the performance was widely variable with the 4 cores far from used to full capacity. Sometimes a douzen of dots get drawn almost instantly, at other times it takes a few seconds before the next dot. Could these delays be the described "hangs"?

I broke the process after about 20 minutes here.

Thanks,

On Tue, 01 Sep 2015 07:29:00 +0200 Daniel Kochmański daniel@turtleware.eu wrote:

Ok, it failed (just after sending an e-mail, it's good to know that Muprhy's law still works :-)).

A complete hang?

Hm, then I can't reproduce neither of them. Spawning too many threads blows the heap, but it's understandable. I think it might be that i have x86_64 and a new kernel, (maybe it happens only on x86, or linux 3.2 had some bug?).

You can track this issue here: https://gitlab.com/embeddable-common-lisp/ecl/issues/150

Thanks, Daniel

James M. Lawrence writes:

The version with a homemade semaphore is

(defstruct sema (count 0) (lock (mp:make-lock :recursive nil)) (cvar (mp:make-condition-variable)))

(defun inc-sema (sema) (mp:with-lock ((sema-lock sema)) (incf (sema-count sema)) (mp:condition-variable-signal (sema-cvar sema))))

(defun dec-sema (sema) (mp:with-lock ((sema-lock sema)) (loop (cond ((plusp (sema-count sema)) (decf (sema-count sema)) (return)) (t (mp:condition-variable-wait (sema-cvar sema) (sema-lock sema)))))))

(defun test (message-count worker-count) (let ((to-workers (make-sema)) (from-workers (make-sema))) (loop repeat worker-count do (mp:process-run-function "test" (lambda () (loop (dec-sema to-workers) (inc-sema from-workers))))) (loop (loop repeat message-count do (inc-sema to-workers)) (loop repeat message-count do (dec-sema from-workers)) (assert (zerop (sema-count to-workers))) (assert (zerop (sema-count from-workers))) (format t ".") (finish-output))))

(defun run () (test 10000 64))

RUN fails with:

Condition of type: SIMPLE-ERROR Attempted to recursively lock #<lock (nonrecursive) 0a4597f8> which is already owned by #<process "test">

In the previous test case, by "hang" I meant that it hangs indefinitely, as opposed to printing dots in spurts. Both these cases fail within seconds for me, sometimes immediately. They should be compiled. Increasing the number of threads (second argument to TEST) will typically cause a quicker failure in these kinds of stress tests. 4-core machine:

Linux xi 3.2.0-24-generic-pae #39-Ubuntu SMP Mon May 21 18:54:21 UTC 2012 i686 i686 i386 GNU/Linux

(:NEW :LINUX :FORMATTER :ECL-WEAK-HASH :LITTLE-ENDIAN :ECL-READ-WRITE-LOCK :LONG-LONG :UINT64-T :UINT32-T :UINT16-T :RELATIVE-PACKAGE-NAMES :LONG-FLOAT :UNICODE :DFFI :CLOS-STREAMS :CMU-FORMAT :UNIX :ECL-PDE :DLOPEN :CLOS :THREADS :BOEHM-GC :ANSI-CL :COMMON-LISP :IEEE-FLOATING-POINT :PREFIXED-API :FFI :PENTIUM3 :COMMON :ECL)

On Tue, Sep 1, 2015 at 1:04 AM, Daniel Kochmański daniel@turtleware.eu wrote:

...

Hello,

that's probably my fault, sorry. I've migrated bugs manually and probably missed this one (I remember this bug! but can't find anywhere).

I'm adding it to regression tests in repository, thanks! Yes, old reports are unfortunately lost.

As a sienote, please use ecl-devel@common-lisp.net mailing list – I'm closing the old one today. You can subscribe here https://mailman.common-lisp.net/listinfo/ecl-devel . All archives before 2015-08-10 are imported to the new one and gmane stream is redirected (if you use it).

Regards, Daniel

James M. Lawrence writes:

...

Hello, the threading bugs I reported a while ago appear to have not survived the migration from sourceforge, and the old pages are now 404'd. There were a number of test cases, including

(defun test (message-count worker-count) (let ((to-workers (mp:make-semaphore)) (from-workers (mp:make-semaphore))) (loop repeat worker-count do (mp:process-run-function "test" (lambda () (loop (mp:wait-on-semaphore to-workers) (mp:signal-semaphore from-workers))))) (loop (loop repeat message-count do (mp:signal-semaphore to-workers)) (loop repeat message-count do (mp:wait-on-semaphore from-workers)) (assert (zerop (mp:semaphore-count to-workers))) (assert (zerop (mp:semaphore-count from-workers))) (format t ".") (finish-output))))

(defun run () (test 10000 64))

RUN will eventually hang on all versions of ECL I've tried, including the latest. Another test case was a variant of the above using a homemade semaphore. I can rewrite that and other test cases, but before doing so I'd like to know whether the old reports are really lost or have survived in some form.

Best, lmj

---

---

Ecls-list mailing list Ecls-list@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ecls-list

-- Daniel Kochmański | Poznań, Poland ;; aka jackdaniel

"Be the change that you wish to see in the world." - Mahatma Gandhi

James M. Lawrence writes:

On Wed, Sep 2, 2015 at 6:09 AM, Daniel Kochmański daniel@turtleware.eu wrote:

...

Hm, then I can't reproduce neither of them. Spawning too many threads blows the heap, but it's understandable. I think it might be that i have x86_64 and a new kernel, (maybe it happens only on x86, or linux 3.2 had some bug?).

64-bit systems are sometimes better at concealing such threading bugs. The default assumption should be that 64-bit systems just haven't been jiggled to the right tune yet. Over the course of several years, and across several kernels, I haven't seen any version of ECL pass these stress tests. This problem would need to be understood before even thinking about kernel bugs, which seem quite unlikely in this case.

Ok, so I guess I'll just set up 32-bit VM. I hope this will be able to reproduce the problems (next test doesn't fail neither on my x86_64 for my tests). Yeah, I also think that kernel bug is unlikely, but I'll keep that possibility in mind.

I've added your tests to repository and will integrate them soon (infinite loop isn't a good pick for regression tests :-)).

Thanks, Daniel

Moving up the ladder of complexity, the next test case requires a blocking queue and involves repeatedly creating/destroying worker threads. It has three different types of failures; they are sporadic and not determined by the inputs. The following are just example runs and shouldn't be considered representative:

(qtest 0 64) ;=> segfault (qtest 1 64) ; => hang (qtest 10000 64) ; => error "Attempted to recursively lock..."

;;;; raw-queue

(defstruct (raw-queue (:conc-name nil)) (head nil) (tail nil))

(defun push-raw-queue (value queue) (let ((new (cons value nil))) (if (head queue) (setf (cdr (tail queue)) new) (setf (head queue) new)) (setf (tail queue) new)))

(defun pop-raw-queue (queue) (let ((node (head queue))) (if node (multiple-value-prog1 (values (car node) t) (when (null (setf (head queue) (cdr node))) (setf (tail queue) nil)) (setf (car node) nil (cdr node) nil)) (values nil nil))))

;;;; queue

(defstruct queue (impl (make-raw-queue)) (lock (mp:make-lock)) (cvar (mp:make-condition-variable)))

(defun push-queue (object queue) (mp:with-lock ((queue-lock queue)) (push-raw-queue object (queue-impl queue)) (mp:condition-variable-signal (queue-cvar queue))))

(defun pop-queue (queue) (mp:with-lock ((queue-lock queue)) (loop (multiple-value-bind (value presentp) (pop-raw-queue (queue-impl queue)) (if presentp (return value) (mp:condition-variable-wait (queue-cvar queue) (queue-lock queue)))))))

;;;; qtest

(defun qtest (message-count worker-count) (loop (let ((to-workers (make-queue)) (from-workers (make-queue))) (loop repeat worker-count do (mp:process-run-function "test" (lambda () (loop (let ((message (pop-queue to-workers))) (push-queue message from-workers) (unless message (return))))))) (loop repeat message-count do (push-queue t to-workers)) (loop repeat message-count do (pop-queue from-workers)) (loop repeat worker-count do (push-queue nil to-workers)) (loop repeat worker-count do (pop-queue from-workers)) (format t ".") (finish-output))))

On Wed, Sep 2, 2015 at 3:32 PM, Daniel Kochmański

Ok, so I guess I'll just set up 32-bit VM. I hope this will be able to reproduce the problems (next test doesn't fail neither on my x86_64 for my tests). Yeah, I also think that kernel bug is unlikely, but I'll keep that possibility in mind.

I've added your tests to repository and will integrate them soon (infinite loop isn't a good pick for regression tests :-)).

An update: I tried this on a newer kernel and obtained the same results. It also fails on Linux x86_64 running 32-bit Ubuntu 15.10 (kernel version 4.2) inside virtualbox.

http://releases.ubuntu.com/15.10/ubuntu-15.10-desktop-i386.iso

There wasn't an error with only one CPU enabled in virtualbox, but when I enabled all 8 CPUs (for my machine) in virtualbox I received a similarly-intermittent error, though the error text was different. This is for the first test given in this thread. Both 16.0.0 and the latest from git fail in the same way.

lmj@u1510:~$ uname -a Linux u1510 4.2.0-16-generic #19-Ubuntu SMP Thu Oct 8 14:46:51 UTC 2015 i686 i686 i686 GNU/Linux lmj@u1510:~$ ~/usr/stow/ecl-dev/bin/ecl -norc -eval '(load (compile-file "bug.lisp"))' -eval '(run)' ;;; Loading #P"/home/lmj/usr/stow/ecl-dev/lib/ecl-16.1.0/cmp.fas" ;;; ;;; Compiling bug.lisp. ;;; OPTIMIZE levels: Safety=2, Space=0, Speed=3, Debug=0 ;;; ;;; Compiling (DEFUN TEST ...). ;;; Compiling (DEFUN RUN ...). ;;; End of Pass 1. ;;; Emitting code for TEST. ;;; Emitting code for #:G5. ;;; Emitting code for RUN. ;;; Finished compiling bug.lisp. ;;; ;;; Loading "/home/lmj/bug.fas" .......................................................................An error occurred during initialization: Unable to interrupt process #<process test> C library explanation: Interrupted system call..

Condition of type: SIMPLE-ERROR Unable to interrupt process #<process test> C library explanation: Interrupted system call.

Available restarts:

1. (CONTINUE) Ignore initialization errors and continue. 2. (ABORT) Quit ECL unsafely, ignoring all existing threads.

Top level in: #<process TOP-LEVEL>.