Fix incredibly long test runs

mentioned in merge request !44 (merged)

A guess: I think this pyopencl version uses POCL source caching.

Using /var/lib/gitlab-runner/builds/f31de19a/1/inducer/pytential/.miniconda3/envs/testing/lib/python3.5/site-packages
Searching for pyopencl==2017.1.1
Best match: pyopencl 2017.1.1
Adding pyopencl 2017.1.1 to easy-install.pth file

A guess: I think this pyopencl version uses POCL source caching.

Never mind. It has the same version number but I don't think that's it after having downloaded it.

Interesting observation 2: Porter is under heavy load right now....

That does not fully explain the slowness. Non-Conda pipelines seem to run on porter in half the time.

These are super weird and only appear in Conda runs:

Exception ignored in: <function WeakValueDictionary.__init__.<locals>.remove at 0x7f622bab4ea0>
Traceback (most recent call last):
  File "/var/lib/gitlab-runner/builds/f31de19a/0/inducer/pytential/.miniconda3/envs/testing/lib/python3.5/weakref.py", line 117, in remove
TypeError: 'NoneType' object is not callable
Exception ignored in: <function WeakValueDictionary.__init__.<locals>.remove at 0x7f622bab4ea0>
Traceback (most recent call last):
  File "/var/lib/gitlab-runner/builds/f31de19a/0/inducer/pytential/.miniconda3/envs/testing/lib/python3.5/weakref.py", line 117, in remove
TypeError: 'NoneType' object is not callable
Exception ignored in: <function WeakValueDictionary.__init__.<locals>.remove at 0x7f622bab4ea0>
Traceback (most recent call last):
  File "/var/lib/gitlab-runner/builds/f31de19a/0/inducer/pytential/.miniconda3/envs/testing/lib/python3.5/weakref.py", line 117, in remove
TypeError: 'NoneType' object is not callable
Exception ignored in: <function WeakValueDictionary.__init__.<locals>.remove at 0x7f622bab4ea0>
Traceback (most recent call last):
  File "/var/lib/gitlab-runner/builds/f31de19a/0/inducer/pytential/.miniconda3/envs/testing/lib/python3.5/weakref.py", line 117, in remove
TypeError: 'NoneType' object is not callable

This also doesn't look right:

>>> s.sort_stats("tottime").print_stats(10)
Sat Jul  1 19:34:31 2017    out.prof

         28823445 function calls (27467677 primitive calls) in 147.705 seconds

   Ordered by: internal time
   List reduced from 9126 to 10 due to restriction <10>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    28011   54.461    0.002   54.463    0.002 {built-in method pyopencl._cffi.clobj__delete}
    45519    9.882    0.000    9.883    0.000 {built-in method builtins.compile}
      505    7.202    0.014    7.202    0.014 {built-in method pyopencl._cffi.wait_for_events}
      878    5.384    0.006    5.387    0.006 {built-in method pyopencl._cffi.enqueue_write_buffer}
  1138962    3.032    0.000    3.032    0.000 {method 'match' of '_sre.SRE_Pattern' objects}
   216824    2.681    0.000    2.681    0.000 {method 'reduce' of 'numpy.ufunc' objects}
   108350    2.671    0.000    4.934    0.000 /home/wala1/miniconda3/envs/testing/lib/python3.5/site-packages/numpy/linalg/linalg.py:2014(norm)
      176    2.431    0.014    2.431    0.014 {built-in method pyopencl._cffi.command_queue__finish}
   542786    2.257    0.000    5.089    0.000 /home/wala1/miniconda3/envs/testing/lib/python3.5/site-packages/Mako-1.0.6-py3.5.egg/mako/lexer.py:68(match_reg)
    54340    1.891    0.000    5.804    0.000 /home/wala1/pytential/test/test_global_qbx.py:122(check_disk_undisturbed_by_sources)

^ That was for running test_global_qbx.py on porter. Why is it spending so much time in clobj__delete?

(Warm cache)

More updates: On my machine the warm cache run of test_global_qbx is twice as slow when using Conda compared to using my native environment. I can also reproduce the weakref errors. It happens both with pocl 0.13 and 0.14 in Conda.

mentioned in merge request !48 (merged)

Why is it spending so much time in clobj__delete?

That's pocl's way of saying "I'm actually doing work." Unfortunately, deleting an event waits on the completion of the operation corresponding to the event. (The CL spec does not specify that it should.)

These are super weird and only appear in Conda runs:

Fixed by pyopencl@89a5074d

At this point, the main difference between our dev environments and conda is the version of pyopencl. I'll go ahead and roll a release and push that through to conda. We'll see if that helps. It'll certainly help kill the "compile" item in the profile.

https://github.com/conda-forge/pyopencl-feedstock/pull/14

The new pyopencl is in. I'll retry https://gitlab.tiker.net/inducer/pytential/pipelines/7414 once it finishes. 2017.2 is built: https://anaconda.org/conda-forge/pyopencl/files

Canceled, retrying: https://gitlab.tiker.net/inducer/pytential/builds/26870

Still extremely long. Not sure that helped, at all.

Still extremely long. Not sure that helped, at all.

At least the test is not failing due to timeouts now, so seems like it could have helped a little? I plan to do more profiling of the tests this week.

These are super weird and only appear in Conda runs:

There were actually two issues here--one that I pointed to and another that could conceivably be attributed to this: https://bugs.python.org/issue29519 (a Py 3.5 bug). Not 100% sure though.

It's back to failing due to timeouts. :/

mentioned in merge request !50 (merged)

This was fixed by !63 (merged).

closed

mentioned in commit 9264bdcd

Fix incredibly long test runs

Designs

Child items ...

Activity