Distributed FMM with complete tree structure on each rank (#49)

* Print absolute and relative errors.

* Add stats collection code.

* Add scaling test.

* Generate responsible_boxes_mask on the host. This saves a lot of time
compared with keeping it on the device.

* Revert "Add scaling test."

This reverts commit 3210a175.

* Revert "Add stats collection code."

This reverts commit 501db499.

* Add test script using ConstantOneExpansionWrangler

* Revert "Revert "Add scaling test.""

This reverts commit 903585b9.

* Revert "Print absolute and relative errors."

This reverts commit 514c527f.

* Collect local mask and scan

* Remove unnecessary argument

* Move local weights construction to drive_dfmm

* Improve comment.

* s/butterfly/tree-like/

* Revert "Revert "Revert "Add scaling test."""

This reverts commit 3545bc74.

* Add a comment about efficiency of the mask compressor kernel

* Integrate distributed FMM with compressed list 3

* Dict is more proper than class without methods

* Add workload_weight interface

* Improve work partition to consider m2l, m2p and p2l

* Handle different FMM order by level

* Memorize local tree and traversal build

* Make test script more concise

* Revert memorization to prevent potential deadlock

* Support well_sep_is_n_away flag

* Fix flake8

* Handle list 3, 4 near merged with list 1

* Get well_sep_is_n_away from trav object on root

* Refactor source weight distribution

* put source weight to tree order before distribute_source_weight

* Temporarily use one global command queue for worker process instead of an argument

* Refactor code

* Add tgt_box_mask to local_data

* Add more options for generate_local_travs

* _from_sep_smaller_min_nsources_cumul is not compatible with distributed implementation

* Use Python's standard logging module

* Build one traversal object with different source flags instead of two separate traversals

* Distribute sources and targets without pickle

* Use waitall in tree distribution instead of a loop

* Use nonblocking recv for tree distribution

* Reduce request object

* Use multidimensional array instead of object arrays for local sources/targets

* Move work partition to a separate file

* Refactor ancestor boxes construction

* Refactor source box construction

* Add doc for function src_boxes_mask

* Refactor multipole boxes query

* Refactor local tree building

* Bug fix

* Tweak interface so that partition work is outside local tree build

* Add get_boxes_mask

* Make responsible_box_query an argument

* Update documentation

* Use to_device API

* Refactor code, move distributed implementation into a submodule

* Add more documentation

* Remove mask and scan from local_data to save device memory

* Improve code quality

* Add documentation for FMM calculation

* Integrate test_distributed with pytest

* Integrate constantone test case into test_distributed

* Improve doc

* Add performance model for form_multipole

* Correction for timing API

* Add no_targets option

* Refactor total FMM terms computation

* Count the workload of direct evaluation

* Refactor linear regression, add eval_direct model

* Extend linear regression to multiple variables

* Refactor FMM parameters

* Count m2l operations

* Add script for testing performance model

* Add m2l model

* Add eval_multipoles model

* Add form_locals model

* Add eval_locals model

* Move code around and refactoring

* Integrate performance model into distributed implementation

* Bug fix

* Improve performance model of list 3 and 4

* Add save/load to performance model

* Use robust linear regression

* Allow distributed fmm to use model on disk, bug fix

* Improve logging

* More logging

* Add barriers for accurate timing

* Save and load the perf model to json file

* Add default performance model

* Refactor default perf model

* Refactor direct eval source box counting

* Refactor boxes time prediction

* Tweak interface

* Remove FMM driver argument from perf model

* Remove wrangler factory for performance model

* __add__ for TimingResult

* Add box_target_counts_nonchild kwarg

* Fix syntax error

* Update __add__

* Update uses of TimingResult in the performance model to new interface

* Another fix

* Fixes for reindexing List 4 close

* [WIP] Try to fix CIs on distributed-fmm-global branch

* Use setuptools.find_packages() in setup.py

* Flake8 fixes

* Localize a few more imports in test_distributed.py

* Flake 8 fix

* Make statsmodels optional

* More detailed warning on statsmodels

* Add PY_EXTRA_FLAGS="-m mpi4py.run" to help avoid stuck MPI-based CIs

* Use -m mpi4py.run in spawning subprocesses

* Turn off output capture (for now) on MPI test

* Temporarily add level_nterms to constantone wrangler

* Fix typo

* Add default_perf_model.json to package_data

* Force OpenMP to use 1 thread in test cases

* Add a note for not importing mpi4py.MPI at module level

* Raise runtime error when partition cannot be done

* Add time recording utility

* Bug fix

* Bug fix

* Integrate OpenCL cost model with distributed implementation

* Try disable recv mprobe

* Revert "Try disable recv mprobe"

This reverts commit 4d16f6d0.

* Use new cost model interface

* Remove record_timing argument

* Find boxes in subrange only loop through contrib boxes instead of all boxes

* Try PyOpenCL instead of Loopy for find_boxes_used_by_subrange

* Try pure Python for find_boxes_used_by_subrange

* Try PyOpenCL version again

* Use cost model with calibration parameters

* Add explanation for local_traversal

* Use new cost model interface

* Allow MPI oversubscription in distributed test cases

* Revert "Allow MPI oversubscription in distributed test cases"

This reverts commit 1ae75402.

* Support MPICH for distributed test cases

* Skip distributed test cases for Python<3.5

* Improve doc

* Use new cost model API

* Allow oversubscription in OpenMPI

* Move run_mpi to tools

* Broadcast the complete traversal object to all worker ranks

* Broadcast tree instead of traversal

* Change generate_local_tree interface for accepting a tree instead of a traversal

* Gather source and target indices outside generate_local_tree

* Add a base class DistributedExpansionWrangler

* Fix pylint

* More pylint fix

* Add more documentation

* Log timing for each stage in distributed FMM eval

* Refactor local tree generation

* Placate flake8

* Refactor distributed FMM driver into drive_fmm

* Fix test failure

* Move kernels for generating local trees to methods instead of using namedtuple

* Improve partition interfaces

* Add ImmutableHostDeviceArray

* Add documentation page for the distributed implementation

* Fix doc CI failure

* Address reviewer's comments

* Register pytest markers

* Expansion wranglers: do not store trees

* Remove (unrealized) Wrangler.tree_dependent_info doc

* Add sumpy downstream CI

* Address reviewer's comments

* Accept both host and device array in ImmutableHostDeviceArray

* Fix test failures for tree/wrangler refactor

* Create fmm.py

* More justification for TraversalAndWrangler design

* Placate flake8

* Back out ill-fated TraversalAndWrangler, introduce TreeIndependentDataForWrangler, introduce boxtree.{timing,constant_one}

* Fix pylint/flake8 for tree-indep data for wrangler

* Update the may-hold-tree comment in the wrangler docstring

* Fix incorrect merge of drive_fmm docstring

* Remove *zeros methods from wrangler interface

* Tweak sumpy downstream to use appropriate branch

* Fix downstream Github CI script

* Fix downstream CI script syntax

* Refactor so that FMMLibTreeIndependentDataForWrangler knows kernel but not Helmholtz k

* Adjust downstream pytential CI for wrangler-refactor branch

* Add template_ary arg to finalize_potentials

* Remove global_wrangler from drive_fmm interface

* Mark DistributedExpansionWrangler as an abstract class

* Move distributed logic and MPI communicator to the distributed wrangler

* Remove queue from the distributed wrangler

* Partition boxes on the root rank and distribute the endpoints

* Use PyOpenCL's fancy indexing in favor of custom kernels

* Address reviewer's comments

* placate pylint

* Improve documentation

* Add Github CI job with MPI tests

* Revert "Add Github CI job with MPI tests"

This reverts commit 8b2e392b.

* Placate pylint

* Use MPICH version as default

* Address reviewer's suggestions

* Use code container to control cache lifetime

* Address reviewer's suggestions on distributed partitioning

* Address reviewer's comments on local tree

* Refactor repeated logic of generating local particles

* Make local trees by calling constructors instead of copying

* Continue to address reviewer's comments on local tree generation

* Use the inherited version of Tree to calculate

* Restrict source boxes directly instead of relying on local_box_flags

* Move the logic of modifying target box flags to local tree construction

* Minor cleanups

Co-authored-by: Matt Wala <wala1@illinois.edu>
Co-authored-by: Andreas Klöckner <inform@tiker.net>
Co-authored-by: Hao Gao <haogao@Haos-MacBook-Pro.local>