Implement batch API with changeset, upsert, and DataFrame integration#129
Open
Implement batch API with changeset, upsert, and DataFrame integration#129
Conversation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
client.batchnamespace -- a deferred-execution batch API that packs multipleDataverse Web API operations into a single
POST $batchHTTP requestclient.batch.dataframenamespace -- pandas DataFrame wrappers for batch operationsclient.records.upsert()andclient.batch.records.upsert()backed by theUpsertMultiplebound action with alternate-key supportbody, causing
400 Bad Requeston the create pathBatch API Design
Implements the Batch API Design spec from @sagebree:
batch.records.*batch.records.upsert(...)batch.tables.*batch.query.sql(...)batch.changeset()batch.execute(continue_on_error=True)batch.dataframe.create/update/deleteDesign constraints enforced:
records.getpaginated overload not supported -- single-record onlytables.createreturns no table metadata on success (HTTP 204)tables.add_columns/tables.remove_columnsdo not flush the picklist cacheclient.flush_cache()not supported in batch (client-side operation)What's included
New:
client.batchAPIbatch.records.create / get / update / delete / upsertbatch.tables.create / get / list / add_columns / remove_columns / deletebatch.tables.list(filter=..., select=...)-- parity withclient.tables.list()from Add filter and select parameters to client.tables.list() #112batch.tables.create_one_to_many_relationship / create_many_to_many_relationship / delete_relationship / get_relationship / create_lookup_fieldbatch.query.sqlbatch.changeset()context manager for transactional (all-or-nothing) operationsexecute(continue_on_error=True)for mixed success/failure batchesBatchResultwith.responses,.succeeded,.failed,.created_ids,.has_errorsNew:
client.batch.dataframeAPIbatch.dataframe.create(table, df)-- DataFrame rows to CreateMultiple batch itembatch.dataframe.update(table, df, id_column)-- DataFrame rows to update batch itemsbatch.dataframe.delete(table, ids_series)-- pandas Series to delete batch itemsExisting: Refactored existing APIs
_build_*/_RawRequestpatternexecute()OData $batch spec compliance
Content-Transfer-Encoding: binaryper partContent-Type: application/httpper partContent-Type: application/json; type=entryfor POST/PATCH bodiesHttpErrorwith parsed Dataverse error details)200,202 Accepted,207 Multi-Status, and400batch response codesReview comment fixes
expectedstatus codes to include202/207for all Dataverse environments_split_multipart/_parse_mime_partreturn type annotations:List[Tuple[Dict[str, str], str]]_build_getto lowercase select column names (consistency with_get_multiple)%20encoding documentation in_build_sqldocstringdatabytes instead ofjsonkwargbatch.records.upsert()raisesTypeError(matchingclient.records.upsert())Testing
Unit tests -- 579 tests passing:
test_batch_operations.py-- BatchRequest, BatchRecordOperations, BatchTableOperations, BatchQueryOperations, ChangeSet, BatchItemResponse, BatchResulttest_batch_serialization.py-- multipart serialization, response parsing, intent resolution, upsert dispatch, batch size limit, content-ID uniqueness, top-level error handlingtest_batch_edge_cases.py-- 40 edge case tests: empty changeset, changeset rollback, content-ID in standalone parts, mixed batch, multiple changesets, batch size limits, top-level errors, continue-on-error, serialization compliance, multipart parsing, content-ID references, intent validationtest_batch_dataframe.py-- 18 tests: DataFrame create/update/delete, validation, NaN handling, empty series, bulk deletetest_odata_internal.py--_build_upsert_multiplebody exclusion, conflict detection, URL/method correctnessE2E tests -- 14 tests passing against live Dataverse (
crm10.dynamics.com):$refcontent-ID)$ref)Examples & docs
examples/advanced/batch.py-- reference examples for all batch operation typesexamples/advanced/walkthrough.py-- batch section added (section 11)examples/basic/functional_testing.py--test_batch_all_operations()covering all operation categories against a live environment