feat: Add bigframes.pandas.job_history() API to track BigQuery jobs

[shuoweil]

This PR is not ready for review. I need it for colab notebook testing.

This PR introduces a new function bigframes.pandas.job_history() that allows users to retrieve a pandas DataFrame listing the BigQuery jobs initiated by BigFrames in the current Python session. This provides visibility into the underlying BigQuery execution, including query text, resource usage, and job duration, which is invaluable for monitoring and optimization.

Key Changes:

• New API: Added bigframes.pandas.job_history() which returns a local pandas DataFrame containing details of all BigQuery jobs run in the current session.
• Metrics Tracking: Extended ExecutionMetrics and JobMetadata in bigframes/session/metrics.py to store individual job records (metadata only) rather than just aggregate statistics. This avoids holding references to full Job objects to prevent memory leaks.
• Comprehensive Job Support: Updated count_job_stats to handle LoadJobs (data ingestion) in addition to the existing QueryJobs and fast-path RowIterators.
• Detailed Metadata: The history captures:
  • Identifiers: Job ID, Query ID (for fast path), Location, Project.
  • Timings: Creation time, Start time, End time, Duration.
  • Status: Job state (DONE, etc.), Error details.
  • Query Info: SQL text, Destination table, Cache hit status.
  • Resources: Total bytes processed, Slot milliseconds.
  • Load Stats: Input files, Input bytes, Output rows (for load jobs).
  • Links: A direct URL to the job in the Google Cloud Console.
• Loader Integration: Updated bigframes/session/loader.py and internal query paths to ensuring internal management queries (like index uniqueness checks) and data loading operations are correctly recorded.
• Testing: Added tests/unit/session/test_job_history.py to verify metric collection for various job types and the public API. Added a manual test notebook notebooks/dataframes/job_history.ipynb for demonstration.

Usage Example:

    1 import bigframes.pandas as bpd
    2 import pandas as pd
    3
    4 # ... run some bigframes operations ...
    5 df = bpd.read_gbq("SELECT 1")
    6
    7 # Upload some local data (triggers a Load Job)
    8 bpd.read_pandas(pd.DataFrame({'a': [1, 2, 3]}))
    9
   10 # Get a DataFrame of all BQ jobs run in this session
   11 history = bpd.job_history()
   12
   13 # Inspect recent queries, their costs, and durations
   14 print(history[['job_id', 'job_type', 'total_bytes_processed', 'duration_seconds', 'query']])

verified at vs code notebook: screen/8u2yhaRV9iHbDbF

Fixes #<481840739> 🦕

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[TrevorBergeron]

can we add a static factory method to build this from an sdk query job object?

[shuoweil]

Done! Added a from_job classmethod (and from_row_iterator) to handle building the metadata object directly from the jobs.

[TrevorBergeron]

I do worry that at a certain point, storing all query test generated by the session might clog up memory?

[shuoweil]

Good point! To prevent memory bloat during long sessions, I have added truncation so we cap the stored query text strings at a maximum of 1024 characters.

[sycai]

I have the concern of placing job_history under the bigframes.pandas package. We may consider bigframes or session instances as the residing places, mainly because functionalities under bigframes.ml and bigframes.bigquery can also trigger jobs but they do not belong to bigframes.pandas.

[chalmerlowe]

Migration Notice: This library is moving to the google-cloud-python monorepo soon.

We closed this PR due to inactivity to ensure a clean migration. Please re-open this work in the new monorepo once the migration is complete!

[shuoweil]

I have the concern of placing job_history under the bigframes.pandas package. We may consider bigframes or session instances as the residing places, mainly because functionalities under bigframes.ml and bigframes.bigquery can also trigger jobs but they do not belong to bigframes.pandas.

I agree with you. I have fully moved it out of bf.pandas. The API is now renamed to execution_history() to better reflect the broadened abstraction and is directly exposed via the root module (bigframes.execution_history()) and on the Session instance.