ARTICLE AD BOX
I am a bit confused about what precisely is happing in Polars' query optimization and whether (or how) it is possible to better control optimization.
I am running a large lazy query (hundreds of expressions) on relatively few rows of data. Upon profiling the query execution time with .profile(), I can see that about 2.2 seconds out of 2.5 seconds for the overall execution time is attributable to the initial optimization step. My code for profiling looks about like this:
some_lazy_query.profile()[1]
Since I see very little room for optimizations in my particular query, I therefore next tried to switch off all optimization flags to see how that affects optimization and overall execution time, using the following:
_opts = pl.QueryOptFlags( predicate_pushdown = False, projection_pushdown = False, simplify_expression = False, slice_pushdown = False, comm_subplan_elim = False, comm_subexpr_elim = False, cluster_with_columns = False, check_order_observe = False, fast_projection = False, sort_collapse = False ) some_lazy_query.profile(optimizations=_opts)[1]Optimization time now is down from 2.2 to 2.13 seconds, the overall execution time is about unchanged at 2.5 seconds. I am aware that some of Polars' optimizations cannot be switched off, but am still very surprised to see that switching off all the available flags hardly affected optimization time.
My questions therefore are:
Am I doing this right? Are there any other ways to reduce optimization time that I have overlooked so far? Out of curiosity, what is the optimization step still doing after switching off all the flags? Do you have any other ideas on how to speed up queries where the optimization overhead is huge compared to the actual execution of the query (due to many columns, but only few rows of data)?I am using the latest version of polars (1.40.0), outcomes are similar on slightly earlier verions (e.g. 1.38.1).
I am aware that feature request #25246 will eventually solve the problem for my application, since this will enable pre-running the optimization, but there appears to be little progress on this so far and the feature might take months to years to actually ship.
