[CALCITE-6451] Improve Nullability Derivation for Intersect and Minus

[xiedeyantu]

Jira Link

CALCITE-6451

Changes Proposed

SetOp overrides deriveRowType() and computes the output row type to be the least restrictive across all inputs here.

So for example given

Input 1: (I64, I64, I64?, I64?)
Input 2: (I64, I64?, I64, I64?)
where ? denotes nullable, the least restrictive output computes:

Output: (I64, I64?, I64?, I64?)
For UNION operations, these nullabilities are accurate.

However for MINUS and INTERSECT there is room for improvement.

MINUS only returns rows from the first input, as such its output nullability should always match that of its first input:

Output: (I64, I64, I64?, I64?)
INTERSECT only returns rows that match across all inputs. If a column is not nullable in any of the inputs, then it is not nullable in the output because no rows can be emitted in which that column is null:

Output: (I64, I64, I64, I64?)

[xiedeyantu]

Related PR #3845.

[mihaibudiu]

you have some checker failures

[mihaibudiu]

Does this work around the problems in the other PR?

[xiedeyantu]

Does this work around the problems in the other PR?

Are you referring to #3845? I noticed that you had approved this PR before, but there were some conflicts. Since it's been a long time, the CI status is no longer visible, and it's unclear if there were other issues back then. I think it's a good PR, so I’m trying to finish it up.

[mihaibudiu]

Yes, the discussion in JIRA was about causing problems with other rules.

[xiedeyantu]

Yes, the discussion in JIRA was about causing problems with other rules.

I didn’t see any discussion in the Jira. Are you referring to the discussion in the original PR? I have resolved the rule conflicts.

[mihaibudiu]

yes, the original PR

[xiedeyantu]

According to this disscusion #3845 (comment) .
I think we don't need to dwell on this issue too much. If we transform INTERSECT into UNION, then we can simply use UNION's type inference. It's like LEFT JOIN (though this example might not be entirely appropriate), which can also change the nullability of columns from the right table. I'm not sure if my understanding is correct.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
94.8% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[xiedeyantu]

@mihaibudiu I'm not sure if you agree with the current simplified processing logic. If you have time, please review this PR to see if there are any other concerns.