Skip to content

refactor: Unify gbq execution as semi-executor#16770

Draft
TrevorBergeron wants to merge 4 commits intomainfrom
task_engine
Draft

refactor: Unify gbq execution as semi-executor#16770
TrevorBergeron wants to merge 4 commits intomainfrom
task_engine

Conversation

@TrevorBergeron
Copy link
Copy Markdown
Contributor

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)

Fixes #<issue_number_goes_here> 🦕

Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the BigFrames execution framework by replacing CacheSpec with TempTableSpec and restructuring BigQueryCachingExecutor to use a recursive _execute_bigquery method. The DirectGbqExecutor was also updated to handle the new ExecutionSpec. Several critical issues were identified in the refactored code, including missing attributes in DirectGbqExecutor (storage_manager, bqstoragereadclient, _labels, and metrics), a missing publisher argument in its constructor call, and an invalid lifetime field usage in TableOutputSpec. Furthermore, the DirectGbqExecutor currently ignores its configured compiler function, which will lead to incorrect behavior when a custom compiler is provided.

Comment thread packages/bigframes/bigframes/session/bq_caching_executor.py
Comment thread packages/bigframes/bigframes/session/bq_caching_executor.py Outdated
Comment thread packages/bigframes/bigframes/session/direct_gbq_execution.py Outdated
return executor.BQTableExecuteResult(
data=result_bq_data,
project_id=self.bqclient.project,
storage_client=self.bqstoragereadclient,
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

DirectGbqExecutor does not have a bqstoragereadclient attribute. This will cause an AttributeError at runtime.

Comment thread packages/bigframes/bigframes/session/direct_gbq_execution.py
Comment thread packages/bigframes/bigframes/session/direct_gbq_execution.py Outdated
Comment thread packages/bigframes/bigframes/session/direct_gbq_execution.py Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant