Fix RSQL filter including map attributes - ghost joins and unintended cross joins

[vasilchev]

Problem

RSQL queries filtering targets by map attributes (controller attributes, software module metadata) with multiple AND conditions produce catastrophically large intermediate result sets that cause queries to hang indefinitely in production.

Reported query (attribute.sw_1_version=out=(...) and attribute.sw_2_version=out=(...) and attribute.sw_3_version=out=(...)) generated this SQL:

  SELECT DISTINCT COUNT(DISTINCT(t0.id))
  FROM sp_target t0
    LEFT OUTER JOIN sp_target_attributes t2 ON (t2.target = t0.id).  <-- ghost, not used/referenced
    LEFT OUTER JOIN sp_target_attributes t4 ON (t4.target = t0.id)  <-- ghost, not used/referenced
    LEFT OUTER JOIN sp_target_attributes t6 ON (t6.target = t0.id)  <-- ghost, not used/referenced
  , sp_target_attributes t5.                                                                    <-- cross join full table
  , sp_target_attributes t3                                                                     <-- cross join full table
  , sp_target_attributes t1                                                                     <-- cross join full table
  WHERE (t5.attribute_value IS NULL OR t5.attribute_value NOT IN (...))
    AND (t3.attribute_value IS NULL OR t3.attribute_value NOT IN (...))
    AND (t1.attribute_value IS NULL OR t1.attribute_value NOT IN (...))
    AND t5.target = t0.id AND t5.attribute_key = 'sw_1_version'
    AND t3.target = t0.id AND t3.attribute_key = 'sw_2_version'
    AND t1.target = t0.id AND t1.attribute_key = 'sw_3_version'

This affects any map-typed RSQL field: attribute., metadata., or any custom @ElementCollection Map<String,String> field.

[avgustinmm]

Isn't there just that "final MapJoin mapPath = (MapJoin) pathResolver.getPath(attribute);" shall not be called if isNot? - it is not used then and just creates additional unused joins?
Wouldn't now lack on be a problem? Causing inefficiciency?

[vasilchev]

I've initially tested exactly that and output didn't make any sense - it again creates cross table joins (for each key whole table without any join clause).

if (isNot(op)) {
     return compare(comparison, toMapValuePath(pathResolver.getJoinOnInner(attribute, split[1])));
 else {
final MapJoin<?, ?, ?> mapPath = (MapJoin<?, ?, ?>) pathResolver.getPath(attribute);
      return cb.and(equal(mapPath.key(), split[1]), compare(comparison, toMapValuePath(mapPath)));
}

The only improvement is removing the "ghost" joins that are not used/referenced. But crossjoins (whole table without on clause remains)

attribute.key1=out=(00,01,02) and attribute.key2=out=(03,04,05) and attribute.key3=out=(06,07,08)

SELECT * FROM sp_target_attributes t3, sp_target_attributes t2, sp_target t1, sp_target_attributes t0 WHERE ((((((t3.attribute_value IS NULL) OR NOT ((UPPER(t3.attribute_value) IN (?, ?, ?)))) AND ((t2.attribute_value IS NULL) OR NOT ((UPPER(t2.attribute_value) IN (?, ?, ?))))) AND ((t0.attribute_value IS NULL) OR NOT ((UPPER(t0.attribute_value) IN (?, ?, ?))))) AND (t1.tenant = ?)) AND ((((t3.target = t1.id) AND (UPPER(t3.attribute_key) = ?)) AND ((t2.target = t1.id) AND (UPPER(t2.attribute_key) = ?))) AND ((t0.target = t1.id) AND (UPPER(t0.attribute_key) = ?))))