Skip to content
Snippets Groups Projects

Remove now-extraneous 'nosync' specs now that loopy got smarter

Merged Remove now-extraneous 'nosync' specs now that loopy got smarter
remove-nosync into master
Merged Andreas Klöckner requested to merge remove-nosync into master
  • :warning: Reset kernel version branch
Edited by Andreas Klöckner

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Andreas Klöckner mentioned in merge request !72 (merged)

    mentioned in merge request !72 (merged)

  • Andreas Klöckner
    Andreas Klöckner @inducer ·
    Author Owner

    cc @mattwala

  • Andreas Klöckner enabled an automatic merge when the pipeline for 6524e96e succeeds

    enabled an automatic merge when the pipeline for 6524e96e succeeds

  • Matt Wala
    Matt Wala @mattwala ·
    Maintainer

    Thanks!

  • Andreas Klöckner canceled the automatic merge

    canceled the automatic merge

  • Andreas Klöckner marked as a Work In Progress

    marked as a Work In Progress

  • Andreas Klöckner changed the description

    changed the description

  • Andreas Klöckner
    Andreas Klöckner @inducer ·
    Author Owner

    Huh. So the weird SIGABRT started happening on Linux, too.

    Example: https://gitlab.tiker.net/inducer/sumpy/-/jobs/44331

    Run with --capture=no gives this:

    Fatal Python error: Cannot recover from stack overflow.                                                                                                                                     
                                                                                                                                                                                             
Current thread 0x00007f3fc15a3700 (most recent call first):                                                                                                                                  
  File "/var/lib/gitlab-runner/builds/569f1b25/4/inducer/sumpy/.miniconda3/envs/testing/lib/python3.5/site-packages/loopy/schedule/__init__.py", line 935 in generate_loop_schedules_internal
  File "/var/lib/gitlab-runner/builds/569f1b25/4/inducer/sumpy/.miniconda3/envs/testing/lib/python3.5/site-packages/loopy/schedule/__init__.py", line 935 in generate_loop_schedules_internal
  File "/var/lib/gitlab-runner/builds/569f1b25/4/inducer/sumpy/.miniconda3/envs/testing/lib/python3.5/site-packages/loopy/schedule/__init__.py", line 935 in generate_loop_schedules_internal
  File "/var/lib/gitlab-runner/builds/569f1b25/4/inducer/sumpy/.miniconda3/envs/testing/lib/python3.5/site-packages/loopy/schedule/__init__.py", line 935 in generate_loop_schedules_internal
  File "/var/lib/gitlab-runner/builds/569f1b25/4/inducer/sumpy/.miniconda3/envs/testing/lib/python3.5/site-packages/loopy/schedule/__init__.py", line 935 in generate_loop_schedules_internal
  File "/var/lib/gitlab-runner/builds/569f1b25/4/inducer/sumpy/.miniconda3/envs/testing/lib/python3.5/site-packages/loopy/schedule/__init__.py", line 935 in generate_loop_schedules_internal
  File "/var/lib/gitlab-runner/builds/569f1b25/4/inducer/sumpy/.miniconda3/envs/testing/lib/python3.5/site-packages/loopy/schedule/__init__.py", line 935 in generate_loop_schedules_internal
  File "/var/lib/gitlab-runner/builds/569f1b25/4/inducer/sumpy/.miniconda3/envs/testing/lib/python3.5/site-packages/loopy/schedule/__init__.py", line 935 in generate_loop_schedules_internal
  File "/var/lib/gitlab-runner/builds/569f1b25/4/inducer/sumpy/.miniconda3/envs/testing/lib/python3.5/site-packages/loopy/schedule/__init__.py", line 935 in generate_loop_schedules_internal
  File "/var/lib/gitlab-runner/builds/569f1b25/4/inducer/sumpy/.miniconda3/envs/testing/lib/python3.5/site-packages/loopy/schedule/__init__.py", line 935 in generate_loop_schedules_internal
  File "/var/lib/gitlab-runner/builds/569f1b25/4/inducer/sumpy/.miniconda3/envs/testing/lib/python3.5/site-packages/loopy/schedule/__init__.py", line 935 in generate_loop_schedules_internal
...
    • I'm not sure why it's intermittent.
    • I'm not sure what we're doing differently now that would trigger this.
    • It seems that in addition to Python's recursion limit, there's also a hard stack-based limit that causes a crash.
    • Found some discussion from sympy though that looks to have been resolved (and may have been something different entirely).
  • Andreas Klöckner
    Andreas Klöckner @inducer ·
    Author Owner

    Maybe a libc update that happened behind our back?

  • Andreas Klöckner
    Andreas Klöckner @inducer ·
    Author Owner

    Maybe a libc update that happened behind our back?

    No evidence of that.

  • Andreas Klöckner
    Andreas Klöckner @inducer ·
    Author Owner 
     /* the macro Py_EnterRecursiveCall() only calls _Py_CheckRecursiveCall()
     if the recursion_depth reaches _Py_CheckRecursionLimit.
     If USE_STACKCHECK, the macro decrements _Py_CheckRecursionLimit
     to guarantee that _Py_CheckRecursiveCall() is regularly called.
     Without USE_STACKCHECK, there is no need for this. */
  int
  _Py_CheckRecursiveCall(const char *where)
  {
      PyThreadState *tstate = PyThreadState_GET();
  
  #ifdef USE_STACKCHECK
      if (PyOS_CheckStack()) {
          --tstate->recursion_depth;
          PyErr_SetString(PyExc_MemoryError, "Stack overflow");
          return -1;
      }
  #endif
      _Py_CheckRecursionLimit = recursion_limit;
      if (tstate->recursion_critical)
          /* Somebody asked that we don't check for recursion. */
          return 0;
      if (tstate->overflowed) {
          if (tstate->recursion_depth > recursion_limit + 50) {
              /* Overflowing while handling an overflow. Give up. */
              Py_FatalError("Cannot recover from stack overflow.");
          }
          return 0;
      }
      if (tstate->recursion_depth > recursion_limit) {
          --tstate->recursion_depth;
          tstate->overflowed = 1;
          PyErr_Format(PyExc_RecursionError,
                       "maximum recursion depth exceeded%s",
                       where);                                                                                                                                                                                                                
          return -1;
      }
      return 0;
  }
  • Andreas Klöckner
    Andreas Klöckner @inducer ·
    Author Owner

    Oh man. This is some stringifier thing again. It's trying to format an error message and then recursively faceplanting. Note the Py_FatalError at #11 in the C stack.

  • Andreas Klöckner
    Andreas Klöckner @inducer ·
    Author Owner

    More informative C stack trace:

    #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007fec2229119a in __GI_abort () at abort.c:89
#2  0x0000000000426c06 in Py_FatalError (msg=msg@entry=0x68db00 "Cannot recover from stack overflow.") at ../Python/pylifecycle.c:1462
#3  0x0000000000550a0d in _Py_CheckRecursiveCall (where=where@entry=0x660f48 " while calling a Python object") at ../Python/ceval.c:686
#4  0x000000000045a17a in _PyObject_FastCallDict (func=func@entry=<type at remote 0x9d2c80>, args=args@entry=0x7fff32b13560, nargs=nargs@entry=1, kwargs=kwargs@entry=0x0) at ../Objects/abstract.c:2305
#5  0x0000000000569fa3 in _PyErr_CreateException (value=<optimized out>, exception=<type at remote 0x9d2c80>) at ../Python/errors.c:65
#6  PyErr_SetObject (value=<optimized out>, exception=<optimized out>) at ../Python/errors.c:97
#7  PyErr_SetNone (exception=<optimized out>) at ../Python/errors.c:149
#8  PyErr_NoMemory () at ../Python/errors.c:478
#9  0x0000000000569e69 in PyErr_FormatV (vargs=0x7fff32b135a0, format=0x66a968 "'%.50s' object has no attribute '%U'", exception=<type at remote 0x9d2c80>) at ../Python/errors.c:845
#10 PyErr_Format (exception=<type at remote 0x9d2c80>, format=format@entry=0x66a968 "'%.50s' object has no attribute '%U'") at ../Python/errors.c:860
#11 0x00000000004c961e in _PyObject_GenericGetAttrWithDict (dict=<optimized out>, name='closed', name@entry=<EncodedFile(buffer=<_io.FileIO at remote 0x7fec1fce6318>, encoding='UTF8') at remote 0x7fec1fcdb0b8>, 
    obj=<EncodedFile(buffer=<_io.FileIO at remote 0x7fec1fce6318>, encoding='UTF8') at remote 0x7fec1fcdb0b8>, obj@entry=0x0) at ../Objects/object.c:1114
#12 PyObject_GenericGetAttr (obj=obj@entry=<EncodedFile(buffer=<_io.FileIO at remote 0x7fec1fce6318>, encoding='UTF8') at remote 0x7fec1fcdb0b8>, name=name@entry='closed') at ../Objects/object.c:1126
#13 0x00000000004e3a89 in slot_tp_getattr_hook (self=<EncodedFile(buffer=<_io.FileIO at remote 0x7fec1fce6318>, encoding='UTF8') at remote 0x7fec1fcdb0b8>, name='closed') at ../Objects/typeobject.c:6264
#14 0x00000000004c86bd in PyObject_GetAttr (name='closed', v=<EncodedFile(buffer=<_io.FileIO at remote 0x7fec1fce6318>, encoding='UTF8') at remote 0x7fec1fcdb0b8>) at ../Objects/object.c:896
#15 PyObject_GetAttrString (v=<EncodedFile(buffer=<_io.FileIO at remote 0x7fec1fce6318>, encoding='UTF8') at remote 0x7fec1fcdb0b8>, name=<optimized out>) at ../Objects/object.c:796
#16 0x000000000042676f in file_is_closed (fobj=<EncodedFile(buffer=<_io.FileIO at remote 0x7fec1fce6318>, encoding='UTF8') at remote 0x7fec1fcdb0b8>) at ../Python/pylifecycle.c:490
#17 flush_std_files () at ../Python/pylifecycle.c:510
#18 0x0000000000426c83 in Py_FatalError (msg=msg@entry=0x68db00 "Cannot recover from stack overflow.") at ../Python/pylifecycle.c:1441
#19 0x0000000000550a0d in _Py_CheckRecursiveCall (where=where@entry=0x660f48 " while calling a Python object") at ../Python/ceval.c:686
#20 0x000000000045a17a in _PyObject_FastCallDict (func=func@entry=<type at remote 0x9d5b40>, args=args@entry=0x0, nargs=nargs@entry=0, kwargs=kwargs@entry=0x0) at ../Objects/abstract.c:2305
#21 0x000000000056a9e2 in _PyErr_CreateException (value=0x0, exception=<type at remote 0x9d5b40>) at ../Python/errors.c:59
#22 PyErr_SetObject (value=0x0, exception=<type at remote 0x9d5b40>) at ../Python/errors.c:97
#23 PyErr_SetNone (exception=<type at remote 0x9d5b40>) at ../Python/errors.c:149
#24 0x000000000047fffc in gen_close (gen=0x7fcbf5c28308, args=<optimized out>) at ../Objects/genobject.c:389
#25 0x00000000004803da in _PyGen_Finalize (self=<generator at remote 0x7fcbf5c28308>) at ../Objects/genobject.c:74
#26 0x00000000004c752b in PyObject_CallFinalizer (self=<generator at remote 0x7fcbf5c28308>) at ../Objects/object.c:297
#27 PyObject_CallFinalizerFromDealloc (self=self@entry=<generator at remote 0x7fcbf5c28308>) at ../Objects/object.c:314
#28 0x000000000047e42c in gen_dealloc (gen=0x7fcbf5c28308) at ../Objects/genobject.c:101
#29 0x0000000000551463 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3584
#30 0x000000000047ff68 in gen_send_ex (closing=1, exc=<optimized out>, arg=<optimized out>, gen=0x7fcbf5c28eb8) at ../Objects/genobject.c:189
#31 gen_close (gen=0x7fcbf5c28eb8, args=<optimized out>) at ../Objects/genobject.c:390
#32 0x00000000004803da in _PyGen_Finalize (self=<generator at remote 0x7fcbf5c28eb8>) at ../Objects/genobject.c:74
#33 0x00000000004c752b in PyObject_CallFinalizer (self=<generator at remote 0x7fcbf5c28eb8>) at ../Objects/object.c:297
#34 PyObject_CallFinalizerFromDealloc (self=self@entry=<generator at remote 0x7fcbf5c28eb8>) at ../Objects/object.c:314
#35 0x000000000047e42c in gen_dealloc (gen=0x7fcbf5c28eb8) at ../Objects/genobject.c:101
#36 0x0000000000551463 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3584
#37 0x000000000047ff68 in gen_send_ex (closing=1, exc=<optimized out>, arg=<optimized out>, gen=0x7fcbf5c282b0) at ../Objects/genobject.c:189
#38 gen_close (gen=0x7fcbf5c282b0, args=<optimized out>) at ../Objects/genobject.c:390
#39 0x00000000004803da in _PyGen_Finalize (self=<generator at remote 0x7fcbf5c282b0>) at ../Objects/genobject.c:74
#40 0x00000000004c752b in PyObject_CallFinalizer (self=<generator at remote 0x7fcbf5c282b0>) at ../Objects/object.c:297
#41 PyObject_CallFinalizerFromDealloc (self=self@entry=<generator at remote 0x7fcbf5c282b0>) at ../Objects/object.c:314
#42 0x000000000047e42c in gen_dealloc (gen=0x7fcbf5c282b0) at ../Objects/genobject.c:101
#43 0x0000000000551463 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3584
  • Andreas Klöckner
    Andreas Klöckner @inducer ·
    Author Owner

    Hmm. That's looping through PyObject_CallFinalizer...

  • Andreas Klöckner
    Andreas Klöckner @inducer ·
    Author Owner

    So there are a few thousand frames of that, and then

    #6418 PyObject_CallFinalizerFromDealloc (self=self@entry=<generator at remote 0x7fcbf574db48>) at ../Objects/object.c:314
#6419 0x000000000047e42c in gen_dealloc (gen=0x7fcbf574db48) at ../Objects/genobject.c:101
#6420 0x0000000000551463 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3584
#6421 0x000000000047ff68 in gen_send_ex (closing=1, exc=<optimized out>, arg=<optimized out>, gen=0x7fcbf574dfc0) at ../Objects/genobject.c:189
#6422 gen_close (gen=0x7fcbf574dfc0, args=<optimized out>) at ../Objects/genobject.c:390
#6423 0x00000000004803da in _PyGen_Finalize (self=<generator at remote 0x7fcbf574dfc0>) at ../Objects/genobject.c:74
#6424 0x00000000004c752b in PyObject_CallFinalizer (self=<generator at remote 0x7fcbf574dfc0>) at ../Objects/object.c:297
#6425 PyObject_CallFinalizerFromDealloc (self=self@entry=<generator at remote 0x7fcbf574dfc0>) at ../Objects/object.c:314
#6426 0x000000000047e42c in gen_dealloc (gen=0x7fcbf574dfc0) at ../Objects/genobject.c:101
#6427 0x0000000000551463 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3584
#6428 0x000000000047ff68 in gen_send_ex (closing=1, exc=<optimized out>, arg=<optimized out>, gen=0x7fcbf574dd58) at ../Objects/genobject.c:189
#6429 gen_close (gen=0x7fcbf574dd58, args=<optimized out>) at ../Objects/genobject.c:390
#6430 0x00000000004803da in _PyGen_Finalize (self=<generator at remote 0x7fcbf574dd58>) at ../Objects/genobject.c:74
#6431 0x00000000004c752b in PyObject_CallFinalizer (self=<generator at remote 0x7fcbf574dd58>) at ../Objects/object.c:297
#6432 PyObject_CallFinalizerFromDealloc (self=self@entry=<generator at remote 0x7fcbf574dd58>) at ../Objects/object.c:314
#6433 0x000000000047e42c in gen_dealloc (gen=0x7fcbf574dd58) at ../Objects/genobject.c:101
#6434 0x0000000000551463 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3584
#6435 0x000000000047ff68 in gen_send_ex (closing=1, exc=<optimized out>, arg=<optimized out>, gen=0x7fcbf57030a0) at ../Objects/genobject.c:189
#6436 gen_close (gen=0x7fcbf57030a0, args=<optimized out>) at ../Objects/genobject.c:390
#6437 0x00000000004803da in _PyGen_Finalize (self=<generator at remote 0x7fcbf57030a0>) at ../Objects/genobject.c:74
#6438 0x0000000000442df0 in finalize_garbage (collectable=0x7fff32b82810) at ../Modules/gcmodule.c:809
#6439 collect (generation=generation@entry=2, n_collected=n_collected@entry=0x7fff32b828d8, n_uncollectable=n_uncollectable@entry=0x7fff32b828e0, nofail=nofail@entry=0) at ../Modules/gcmodule.c:1008
#6440 0x000000000044441a in collect_with_callback (generation=2) at ../Modules/gcmodule.c:1128
#6441 collect_generations () at ../Modules/gcmodule.c:1151
#6442 _PyObject_GC_Alloc (basicsize=<optimized out>, use_calloc=0) at ../Modules/gcmodule.c:1726
#6443 _PyObject_GC_Malloc (basicsize=<optimized out>) at ../Modules/gcmodule.c:1736
#6444 _PyObject_GC_NewVar (tp=tp@entry=0x9e3040 <PyTuple_Type>, nitems=nitems@entry=1) at ../Modules/gcmodule.c:1765
#6445 0x00000000004d68ad in PyTuple_New (size=size@entry=1) at ../Objects/tupleobject.c:114
#6446 0x00000000004e001a in reduce_newobj (obj=obj@entry=<Variable(name='expr_1834', hash_value=-2228923348531945301) at remote 0x7fcbf53bbf28>) at ../Objects/typeobject.c:4250
#6447 0x00000000004e072e in _common_reduce (proto=4, self=<Variable(name='expr_1834', hash_value=-2228923348531945301) at remote 0x7fcbf53bbf28>) at ../Objects/typeobject.c:4335
#6448 object_reduce_ex (self=<Variable(name='expr_1834', hash_value=-2228923348531945301) at remote 0x7fcbf53bbf28>, args=<optimized out>) at ../Objects/typeobject.c:4400
#6449 0x00000000004c4e4b in _PyCFunction_FastCallDict (func_obj=func_obj@entry=<built-in method __reduce_ex__ of Variable object at remote 0x7fcbf53bbf28>, args=args@entry=0x7fff32b82a98, nargs=nargs@entry=1, kwargs=kwargs@entry=0x0)
    at ../Objects/methodobject.c:234
#6450 0x000000000045a1e1 in _PyObject_FastCallDict (func=<built-in method __reduce_ex__ of Variable object at remote 0x7fcbf53bbf28>, args=args@entry=0x7fff32b82a98, nargs=nargs@entry=1, kwargs=kwargs@entry=0x0)
    at ../Objects/abstract.c:2313
#6451 0x00000000005a8ebb in _Pickle_FastCall (func=<optimized out>, obj=<optimized out>) at ../Modules/_pickle.c:349
#6452 0x00000000005b085c in save (self=self@entry=0x7fe8a4061b88, obj=<Variable(name='expr_1834', hash_value=-2228923348531945301) at remote 0x7fcbf53bbf28>, pers_save=pers_save@entry=0) at ../Modules/_pickle.c:3990
#6453 0x00000000005b0946 in store_tuple_elements (len=<optimized out>, t=<optimized out>, self=<optimized out>) at ../Modules/_pickle.c:2360
#6454 save_tuple (obj=(<numpy.float64 at remote 0x7fde5c0e4888>, <Variable(name='expr_1834', hash_value=-2228923348531945301) at remote 0x7fcbf53bbf28>), self=0x7fe8a4061b88) at ../Modules/_pickle.c:2411
#6455 save (self=self@entry=0x7fe8a4061b88, obj=(<numpy.float64 at remote 0x7fde5c0e4888>, <Variable(name='expr_1834', hash_value=-2228923348531945301) at remote 0x7fcbf53bbf28>), pers_save=pers_save@entry=0) at ../Modules/_pickle.c:3921
#6456 0x00000000005b0946 in store_tuple_elements (len=<optimized out>, t=<optimized out>, self=<optimized out>) at ../Modules/_pickle.c:2360
#6457 save_tuple (obj=((<numpy.float64 at remote 0x7fde5c0e4888>, <Variable(name='expr_1834', hash_value=-2228923348531945301) at remote 0x7fcbf53bbf28>),), self=0x7fe8a4061b88) at ../Modules/_pickle.c:2411

#6458 save (self=self@entry=0x7fe8a4061b88, obj=((<numpy.float64 at remote 0x7fde5c0e4888>, <Variable(name='expr_1834', hash_value=-2228923348531945301) at remote 0x7fcbf53bbf28>),), pers_save=pers_save@entry=0)
    at ../Modules/_pickle.c:3921
#6459 0x00000000005b2537 in save_reduce (self=self@entry=0x7fe8a4061b88, 
    args=args@entry=(<function at remote 0x7fec220031e0>, (<type at remote 0x1e84968>,), ((<numpy.float64 at remote 0x7fde5c0e4888>, <Variable(name='expr_1834', hash_value=-2228923348531945301) at remote 0x7fcbf53bbf28>),), None, None), 
    obj=obj@entry=<Product(children=(<numpy.float64 at remote 0x7fde5c0e4888>, <Variable(name='expr_1834', hash_value=-2228923348531945301) at remote 0x7fcbf53bbf28>), hash_value=-8388064068769906100) at remote 0x7fcbf52f27f0>)
    at ../Modules/_pickle.c:3827
#6460 0x00000000005b0516 in save (self=self@entry=0x7fe8a4061b88, 
---Type <return> to continue, or q <return> to quit---
    obj=<Product(children=(<numpy.float64 at remote 0x7fde5c0e4888>, <Variable(name='expr_1834', hash_value=-2228923348531945301) at remote 0x7fcbf53bbf28>), hash_value=-8388064068769906100) at remote 0x7fcbf52f27f0>, 
    pers_save=pers_save@entry=0) at ../Modules/_pickle.c:4031
#6461 0x00000000005b05f0 in store_tuple_elements (len=<optimized out>, t=<optimized out>, self=<optimized out>) at ../Modules/_pickle.c:2360
#6462 save_tuple (
    obj=(<Product(children=(-121645100408832000, <Variable(name='expr_231', hash_value=7029616991540739657) at remote 0x7fcbf50ff358>), hash_value=-7400725364496041712) at remote 0x7fcbf52f24e0>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e47c8>, <Variable(name='expr_1829', hash_value=1348439514963580625) at remote 0x7fcbf53bbd68>), hash_value=-7334194039609706133) at remote 0x7fcbf52f2588>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e4d08>, <Variable(name='expr_1830', hash_value=-5322659217122364627) at remote 0x7fcbf53bbdd8>), hash_value=-629639435305597840) at remote 0x7fcbf52f2630>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e42d0>, <Variable(name='expr_1831', hash_value=-4178123599674749985) at remote 0x7fcbf53bbe48>), hash_value=4831616140040348894) at remote 0x7fcbf52f2240>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e46c0>, <Variable(name='expr_1833', hash_value=-840482050216540515) at remote 0x7fcbf53bbeb8>), hash_value=8455460904441463416) at remote 0x7fcbf52f23c8>, <...(truncated), self=0x7fe8a4061b88) at ../Modules/_pickle.c:2438
#6463 save (self=self@entry=0x7fe8a4061b88, 
    obj=(<Product(children=(-121645100408832000, <Variable(name='expr_231', hash_value=7029616991540739657) at remote 0x7fcbf50ff358>), hash_value=-7400725364496041712) at remote 0x7fcbf52f24e0>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e47c8>, <Variable(name='expr_1829', hash_value=1348439514963580625) at remote 0x7fcbf53bbd68>), hash_value=-7334194039609706133) at remote 0x7fcbf52f2588>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e4d08>, <Variable(name='expr_1830', hash_value=-5322659217122364627) at remote 0x7fcbf53bbdd8>), hash_value=-629639435305597840) at remote 0x7fcbf52f2630>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e42d0>, <Variable(name='expr_1831', hash_value=-4178123599674749985) at remote 0x7fcbf53bbe48>), hash_value=4831616140040348894) at remote 0x7fcbf52f2240>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e46c0>, <Variable(name='expr_1833', hash_value=-840482050216540515) at remote 0x7fcbf53bbeb8>), hash_value=8455460904441463416) at remote 0x7fcbf52f23c8>, <...(truncated), pers_save=pers_save@entry=0) at ../Modules/_pickle.c:3921
#6464 0x00000000005b0946 in store_tuple_elements (len=<optimized out>, t=<optimized out>, self=<optimized out>) at ../Modules/_pickle.c:2360
#6465 save_tuple (
    obj=((<Product(children=(-121645100408832000, <Variable(name='expr_231', hash_value=7029616991540739657) at remote 0x7fcbf50ff358>), hash_value=-7400725364496041712) at remote 0x7fcbf52f24e0>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e47c8>, <Variable(name='expr_1829', hash_value=1348439514963580625) at remote 0x7fcbf53bbd68>), hash_value=-7334194039609706133) at remote 0x7fcbf52f2588>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e4d08>, <Variable(name='expr_1830', hash_value=-5322659217122364627) at remote 0x7fcbf53bbdd8>), hash_value=-629639435305597840) at remote 0x7fcbf52f2630>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e42d0>, <Variable(name='expr_1831', hash_value=-4178123599674749985) at remote 0x7fcbf53bbe48>), hash_value=4831616140040348894) at remote 0x7fcbf52f2240>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e46c0>, <Variable(name='expr_1833', hash_value=-840482050216540515) at remote 0x7fcbf53bbeb8>), hash_value=8455460904441463416) at remote 0x7fcbf52f23c8>, ...(truncated), self=0x7fe8a4061b88) at ../Modules/_pickle.c:2411
#6466 save (self=self@entry=0x7fe8a4061b88, 
    obj=((<Product(children=(-121645100408832000, <Variable(name='expr_231', hash_value=7029616991540739657) at remote 0x7fcbf50ff358>), hash_value=-7400725364496041712) at remote 0x7fcbf52f24e0>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e47c8>, <Variable(name='expr_1829', hash_value=1348439514963580625) at remote 0x7fcbf53bbd68>), hash_value=-7334194039609706133) at remote 0x7fcbf52f2588>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e4d08>, <Variable(name='expr_1830', hash_value=-5322659217122364627) at remote 0x7fcbf53bbdd8>), hash_value=-629639435305597840) at remote 0x7fcbf52f2630>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e42d0>, <Variable(name='expr_1831', hash_value=-4178123599674749985) at remote 0x7fcbf53bbe48>), hash_value=4831616140040348894) at remote 0x7fcbf52f2240>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e46c0>, <Variable(name='expr_1833', hash_value=-840482050216540515) at remote 0x7fcbf53bbeb8>), hash_value=8455460904441463416) at remote 0x7fcbf52f23c8>, ...(truncated), pers_save=pers_save@entry=0) at ../Modules/_pickle.c:3921
#6467 0x00000000005b2537 in save_reduce (self=self@entry=0x7fe8a4061b88, 
    args=args@entry=(<function at remote 0x7fec220031e0>, (<type at remote 0x1e845b8>,), ((<Product(children=(-121645100408832000, <Variable(name='expr_231', hash_value=7029616991540739657) at remote 0x7fcbf50ff358>), hash_value=-7400725364496041712) at remote 0x7fcbf52f24e0>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e47c8>, <Variable(name='expr_1829', hash_value=1348439514963580625) at remote 0x7fcbf53bbd68>), hash_value=-7334194039609706133) at remote 0x7fcbf52f2588>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e4d08>, <Variable(name='expr_1830', hash_value=-5322659217122364627) at remote 0x7fcbf53bbdd8>), hash_value=-629639435305597840) at remote 0x7fcbf52f2630>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e42d0>, <Variable(name='expr_1831', hash_value=-4178123599674749985) at remote 0x7fcbf53bbe48>), hash_value=4831616140040348894) at remote 0x7fcbf52f2240>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e46c0>, <Variable(name='expr_1833', hash_value=-840482050216540515) at remote 0x7fcbf5...(truncated), 
    obj=obj@entry=<Sum(children=(<Product(children=(-121645100408832000, <Variable(name='expr_231', hash_value=7029616991540739657) at remote 0x7fcbf50ff358>), hash_value=-7400725364496041712) at remote 0x7fcbf52f24e0>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e47c8>, <Variable(name='expr_1829', hash_value=1348439514963580625) at remote 0x7fcbf53bbd68>), hash_value=-7334194039609706133) at remote 0x7fcbf52f2588>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e4d08>, <Variable(name='expr_1830', hash_value=-5322659217122364627) at remote 0x7fcbf53bbdd8>), hash_value=-629639435305597840) at remote 0x7fcbf52f2630>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e42d0>, <Variable(name='expr_1831', hash_value=-4178123599674749985) at remote 0x7fcbf53bbe48>), hash_value=4831616140040348894) at remote 0x7fcbf52f2240>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e46c0>, <Variable(name='expr_1833', hash_value=-840482050216540515) at remote 0x7fcbf53bbeb8>), hash_value=8455460904441463416) at remote 0x7f...(truncated)) at ../Modules/_pickle.c:3827
#6468 0x00000000005b0516 in save (self=self@entry=0x7fe8a4061b88, 
    obj=<Sum(children=(<Product(children=(-121645100408832000, <Variable(name='expr_231', hash_value=7029616991540739657) at remote 0x7fcbf50ff358>), hash_value=-7400725364496041712) at remote 0x7fcbf52f24e0>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e47c8>, <Variable(name='expr_1829', hash_value=1348439514963580625) at remote 0x7fcbf53bbd68>), hash_value=-7334194039609706133) at remote 0x7fcbf52f2588>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e4d08>, <Variable(name='expr_1830', hash_value=-5322659217122364627) at remote 0x7fcbf53bbdd8>), hash_value=-629639435305597840) at remote 0x7fcbf52f2630>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e42d0>, <Variable(name='expr_1831', hash_value=-4178123599674749985) at remote 0x7fcbf53bbe48>), hash_value=4831616140040348894) at remote 0x7fcbf52f2240>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e46c0>, <Variable(name='expr_1833', hash_value=-840482050216540515) at remote 0x7fcbf53bbeb8>), hash_value=8455460904441463416) at remote 0x7f...(truncated), pers_save=pers_save@entry=0) at ../Modules/_pickle.c:4031
#6469 0x00000000005b0946 in store_tuple_elements (len=<optimized out>, t=<optimized out>, self=<optimized out>) at ../Modules/_pickle.c:2360
#6470 save_tuple (
    obj=(<Variable(name='expr_1839', hash_value=8661087764752086566) at remote 0x7fcbf53bbc18>, <Sum(children=(<Product(children=(-121645100408832000, <Variable(name='expr_231', hash_value=7029616991540739657) at remote 0x7fcbf50ff358>), hash_value=-7400725364496041712) at remote 0x7fcbf52f24e0>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e47c8>, <Variable(name='expr_1829', hash_value=1348439514963580625) at remote 0x7fcbf53bbd68>), hash_value=-7334194039609706133) at remote 0x7fcbf52f2588>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e4d08>, <Variable(name='expr_1830', hash_value=-5322659217122364627) at remote 0x7fcbf53bbdd8>), hash_value=-629639435305597840) at remote 0x7fcbf52f2630>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e42d0>, <Variable(name='expr_1831', hash_value=-4178123599674749985) at remote 0x7fcbf53bbe48>), hash_value=4831616140040348894) at remote 0x7fcbf52f2240>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e46c0>, <Variable(name='expr_1833', hash_value=-840482...(truncated), self=0x7fe8a4061b88) at ../Modules/_pickle.c:2411
#6471 save (self=self@entry=0x7fe8a4061b88, 
    obj=(<Variable(name='expr_1839', hash_value=8661087764752086566) at remote 0x7fcbf53bbc18>, <Sum(children=(<Product(children=(-121645100408832000, <Variable(name='expr_231', hash_value=7029616991540739657) at remote 0x7fcbf50ff358>), hash_value=-7400725364496041712) at remote 0x7fcbf52f24e0>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e47c8>, <Variable(name='expr_1829', hash_value=1348439514963580625) at remote 0x7fcbf53bbd68>), hash_value=-7334194039609706133) at remote 0x7fcbf52f2588>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e4d08>, <Variable(name='expr_1830', hash_value=-5322659217122364627) at remote 0x7fcbf53bbdd8>), hash_value=-629639435305597840) at remote 0x7fcbf52f2630>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e42d0>, <Variable(name='expr_1831', hash_value=-4178123599674749985) at remote 0x7fcbf53bbe48>), hash_value=4831616140040348894) at remote 0x7fcbf52f2240>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e46c0>, <Variable(name='expr_1833', hash_value=-840482...(truncated), pers_save=pers_save@entry=0) at ../Modules/_pickle.c:3921
#6472 0x00000000005b0946 in store_tuple_elements (len=<optimized out>, t=<optimized out>, self=<optimized out>) at ../Modules/_pickle.c:2360
#6473 save_tuple (
    obj=((<Variable(name='expr_1839', hash_value=8661087764752086566) at remote 0x7fcbf53bbc18>, <Sum(children=(<Product(children=(-121645100408832000, <Variable(name='expr_231', hash_value=7029616991540739657) at remote 0x7fcbf50ff358>), hash_value=-7400725364496041712) at remote 0x7fcbf52f24e0>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e47c8>, <Variable(name='expr_1829', hash_value=1348439514963580625) at remote 0x7fcbf53bbd68>), hash_value=-7334194039609706133) at remote 0x7fcbf52f2588>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e4d08>, <Variable(name='expr_1830', hash_value=-5322659217122364627) at remote 0x7fcbf53bbdd8>), hash_value=-629639435305597840) at remote 0x7fcbf52f2630>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e42d0>, <Variable(name='expr_1831', hash_value=-4178123599674749985) at remote 0x7fcbf53bbe48>), hash_value=4831616140040348894) at remote 0x7fcbf52f2240>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e46c0>, <Variable(name='expr_1833', hash_value=-84048...(truncated), self=0x7fe8a4061b88) at ../Modules/_pickle.c:2411
#6474 save (self=self@entry=0x7fe8a4061b88, 
    obj=((<Variable(name='expr_1839', hash_value=8661087764752086566) at remote 0x7fcbf53bbc18>, <Sum(children=(<Product(children=(-121645100408832000, <Variable(name='expr_231', hash_value=7029616991540739657) at remote 0x7fcbf50ff358>), hash_value=-7400725364496041712) at remote 0x7fcbf52f24e0>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e47c8>, <Variable(name='expr_1829', hash_value=1348439514963580625) at remote 0x7fcbf53bbd68>), hash_value=-7334194039609706133) at remote 0x7fcbf52f2588>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e4d08>, <Variable(name='expr_1830', hash_value=-5322659217122364627) at remote 0x7fcbf53bbdd8>), hash_value=-629639435305597840) at remote 0x7fcbf52f2630>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e42d0>, <Variable(name='expr_1831', hash_value=-4178123599674749985) at remote 0x7fcbf53bbe48>), hash_value=4831616140040348894) at remote 0x7fcbf52f2240>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e46c0>, <Variable(name='expr_1833', hash_value=-84048...(truncated), pers_save=pers_save@entry=0) at ../Modules/_pickle.c:3921
#6475 0x00000000005b2537 in save_reduce (self=self@entry=0x7fe8a4061b88, 
    args=args@entry=(<function at remote 0x7fec220031e0>, (<type at remote 0x1e84968>,), ((<Variable(name='expr_1839', hash_value=8661087764752086566) at remote 0x7fcbf53bbc18>, <Sum(children=(<Product(children=(-121645100408832000, <Variable(name='expr_231', hash_value=7029616991540739657) at remote 0x7fcbf50ff358>), hash_value=-7400725364496041712) at remote 0x7fcbf52f24e0>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e47c8>, <Variable(name='expr_1829', hash_value=1348439514963580625) at remote 0x7fcbf53bbd68>), hash_value=-7334194039609706133) at remote 0x7fcbf52f2588>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e4d08>, <Variable(name='expr_1830', hash_value=-5322659217122364627) at remote 0x7fcbf53bbdd8>), hash_value=-629639435305597840) at remote 0x7fcbf52f2630>, <Product(children=(<numpy.float64 at remote 0x7fde5c0e42d0>, <Variable(name='expr_1831', hash_value=-4178123599674749985) at remote 0x7fcbf53bbe48>), hash_value=4831616140040348894) at remote 0x7fcbf52f2240>, <Product(children=(<numpy.float64 at ...(truncated),
  • Andreas Klöckner
    Andreas Klöckner @inducer ·
    Author Owner

    Hypothesis: If we hit the garbage collector in the middle of pickling, we die. (that would explain why it's intermittent)

    More questions:

    • Why is it recursively GC'ing thousands of generators?
    Edited by Andreas Klöckner
  • Andreas Klöckner mentioned in issue #31

    mentioned in issue #31

  • Matt Wala
    Matt Wala @mattwala ·
    Maintainer
    • Why is it recursively GC'ing thousands of generators?

    If these are generators that are created during pickling, it's possible that they come from the mapper that stringifies the instructions. IIRC, the instructions are also stringified as part of pickling.

  • Andreas Klöckner
    Andreas Klöckner @inducer ·
    Author Owner

    If these are generators that are created during pickling, it's possible that they come from the mapper that stringifies the instructions. IIRC, the instructions are also stringified as part of pickling.

    Tried to find where--didn't find anything. Do you recall where that happens?

  • Matt Wala
    Matt Wala @mattwala ·
    Maintainer

    It happens in LoopyEqKeyBuilder which is called when the lazily unpickling list is pickled.

  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
Please register or sign in to reply