CSE: Don't pull things out of derivatives.

Now that sumpy has its own CSE, we can eliminate the need for this workaround.