KEMBAR78
FIX: sort strings in UTF-8 encoded byte order by milaGGL · Pull Request #1967 · googleapis/java-firestore · GitHub
Skip to content

Conversation

@milaGGL
Copy link
Contributor

@milaGGL milaGGL commented Jan 6, 2025

Strings should be sorted in UTF-8 encoded byte order. Public document: https://cloud.google.com/firestore/docs/concepts/data-types#data_types

SDK sorts strings using built in comparator method, which sorts lexicographically, and leads to mismatch between server and sdk when special characters are present. This PR fixes the string order mismatches on document field, map key, and document key.

@product-auto-label product-auto-label bot added size: m Pull request size is medium. api: firestore Issues related to the googleapis/java-firestore API. labels Jan 6, 2025
@generated-files-bot
Copy link

Warning: This pull request is touching the following templated files:

  • google-cloud-firestore-admin/src/main/java/com/google/cloud/firestore/v1/FirestoreAdminClient.java
  • google-cloud-firestore-admin/src/main/java/com/google/cloud/firestore/v1/FirestoreAdminSettings.java
  • google-cloud-firestore-admin/src/main/java/com/google/cloud/firestore/v1/package-info.java
  • google-cloud-firestore-admin/src/main/java/com/google/cloud/firestore/v1/stub/FirestoreAdminStub.java
  • google-cloud-firestore-admin/src/main/java/com/google/cloud/firestore/v1/stub/FirestoreAdminStubSettings.java
  • google-cloud-firestore-admin/src/main/java/com/google/cloud/firestore/v1/stub/GrpcFirestoreAdminCallableFactory.java
  • google-cloud-firestore-admin/src/main/java/com/google/cloud/firestore/v1/stub/GrpcFirestoreAdminStub.java
  • google-cloud-firestore-admin/src/test/java/com/google/cloud/firestore/v1/FirestoreAdminClientTest.java
  • google-cloud-firestore-admin/src/test/java/com/google/cloud/firestore/v1/MockFirestoreAdmin.java
  • google-cloud-firestore-admin/src/test/java/com/google/cloud/firestore/v1/MockFirestoreAdminImpl.java
  • google-cloud-firestore/src/main/java/com/google/cloud/firestore/v1/FirestoreClient.java
  • google-cloud-firestore/src/main/java/com/google/cloud/firestore/v1/FirestoreSettings.java
  • google-cloud-firestore/src/main/java/com/google/cloud/firestore/v1/package-info.java
  • google-cloud-firestore/src/main/java/com/google/cloud/firestore/v1/stub/FirestoreStub.java
  • google-cloud-firestore/src/main/java/com/google/cloud/firestore/v1/stub/FirestoreStubSettings.java
  • google-cloud-firestore/src/main/java/com/google/cloud/firestore/v1/stub/GrpcFirestoreCallableFactory.java
  • google-cloud-firestore/src/main/java/com/google/cloud/firestore/v1/stub/GrpcFirestoreStub.java
  • google-cloud-firestore/src/test/java/com/google/cloud/firestore/v1/FirestoreClientTest.java
  • google-cloud-firestore/src/test/java/com/google/cloud/firestore/v1/MockFirestore.java
  • google-cloud-firestore/src/test/java/com/google/cloud/firestore/v1/MockFirestoreImpl.java
  • grpc-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/FirestoreAdminGrpc.java
  • grpc-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/FirestoreGrpc.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/CollectionGroupName.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/CreateIndexRequest.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/CreateIndexRequestOrBuilder.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/DatabaseName.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/DeleteIndexRequest.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/DeleteIndexRequestOrBuilder.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/ExportDocumentsMetadata.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/ExportDocumentsMetadataOrBuilder.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/ExportDocumentsRequest.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/ExportDocumentsRequestOrBuilder.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/ExportDocumentsResponse.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/ExportDocumentsResponseOrBuilder.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/Field.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/FieldName.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/FieldOperationMetadata.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/FieldOperationMetadataOrBuilder.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/FieldOrBuilder.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/FieldProto.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/FirestoreAdminProto.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/GetFieldRequest.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/GetFieldRequestOrBuilder.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/GetIndexRequest.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/GetIndexRequestOrBuilder.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/ImportDocumentsMetadata.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/ImportDocumentsMetadataOrBuilder.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/ImportDocumentsRequest.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/ImportDocumentsRequestOrBuilder.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/Index.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/IndexName.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/IndexOperationMetadata.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/IndexOperationMetadataOrBuilder.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/IndexOrBuilder.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/IndexProto.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/ListFieldsRequest.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/ListFieldsRequestOrBuilder.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/ListFieldsResponse.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/ListFieldsResponseOrBuilder.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/ListIndexesRequest.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/ListIndexesRequestOrBuilder.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/ListIndexesResponse.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/ListIndexesResponseOrBuilder.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/LocationMetadata.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/LocationMetadataOrBuilder.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/LocationProto.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/OperationProto.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/OperationState.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/Progress.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/ProgressOrBuilder.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/UpdateFieldRequest.java
  • proto-google-cloud-firestore-admin-v1/src/main/java/com/google/firestore/admin/v1/UpdateFieldRequestOrBuilder.java
  • proto-google-cloud-firestore-bundle-v1/src/main/java/com/google/firestore/bundle/BundleElement.java
  • proto-google-cloud-firestore-bundle-v1/src/main/java/com/google/firestore/bundle/BundleElementOrBuilder.java
  • proto-google-cloud-firestore-bundle-v1/src/main/java/com/google/firestore/bundle/BundleMetadata.java
  • proto-google-cloud-firestore-bundle-v1/src/main/java/com/google/firestore/bundle/BundleMetadataOrBuilder.java
  • proto-google-cloud-firestore-bundle-v1/src/main/java/com/google/firestore/bundle/BundleProto.java
  • proto-google-cloud-firestore-bundle-v1/src/main/java/com/google/firestore/bundle/BundledDocumentMetadata.java
  • proto-google-cloud-firestore-bundle-v1/src/main/java/com/google/firestore/bundle/BundledDocumentMetadataOrBuilder.java
  • proto-google-cloud-firestore-bundle-v1/src/main/java/com/google/firestore/bundle/BundledQuery.java
  • proto-google-cloud-firestore-bundle-v1/src/main/java/com/google/firestore/bundle/BundledQueryOrBuilder.java
  • proto-google-cloud-firestore-bundle-v1/src/main/java/com/google/firestore/bundle/NamedQuery.java
  • proto-google-cloud-firestore-bundle-v1/src/main/java/com/google/firestore/bundle/NamedQueryOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/ArrayValue.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/ArrayValueOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/BatchGetDocumentsRequest.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/BatchGetDocumentsRequestOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/BatchGetDocumentsResponse.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/BatchGetDocumentsResponseOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/BatchWriteRequest.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/BatchWriteRequestOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/BatchWriteResponse.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/BatchWriteResponseOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/BeginTransactionRequest.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/BeginTransactionRequestOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/BeginTransactionResponse.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/BeginTransactionResponseOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/CommitRequest.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/CommitRequestOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/CommitResponse.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/CommitResponseOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/CommonProto.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/CreateDocumentRequest.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/CreateDocumentRequestOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/Cursor.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/CursorOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/DeleteDocumentRequest.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/DeleteDocumentRequestOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/Document.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/DocumentChange.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/DocumentChangeOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/DocumentDelete.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/DocumentDeleteOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/DocumentMask.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/DocumentMaskOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/DocumentOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/DocumentProto.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/DocumentRemove.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/DocumentRemoveOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/DocumentTransform.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/DocumentTransformOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/ExistenceFilter.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/ExistenceFilterOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/FirestoreProto.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/GetDocumentRequest.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/GetDocumentRequestOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/ListCollectionIdsRequest.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/ListCollectionIdsRequestOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/ListCollectionIdsResponse.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/ListCollectionIdsResponseOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/ListDocumentsRequest.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/ListDocumentsRequestOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/ListDocumentsResponse.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/ListDocumentsResponseOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/ListenRequest.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/ListenRequestOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/ListenResponse.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/ListenResponseOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/MapValue.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/MapValueOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/PartitionQueryRequest.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/PartitionQueryRequestOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/PartitionQueryResponse.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/PartitionQueryResponseOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/Precondition.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/PreconditionOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/QueryProto.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/RollbackRequest.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/RollbackRequestOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/RunQueryRequest.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/RunQueryRequestOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/RunQueryResponse.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/RunQueryResponseOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/StructuredQuery.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/StructuredQueryOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/Target.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/TargetChange.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/TargetChangeOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/TargetOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/TransactionOptions.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/TransactionOptionsOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/UpdateDocumentRequest.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/UpdateDocumentRequestOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/Value.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/ValueOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/Write.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/WriteOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/WriteProto.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/WriteRequest.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/WriteRequestOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/WriteResponse.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/WriteResponseOrBuilder.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/WriteResult.java
  • proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/WriteResultOrBuilder.java

@milaGGL milaGGL changed the title add test sort strings in UTF-8 encoded byte order Jan 6, 2025
@product-auto-label product-auto-label bot added size: l Pull request size is large. and removed size: m Pull request size is medium. labels Jan 7, 2025
@milaGGL milaGGL marked this pull request as ready for review January 10, 2025 15:58
@milaGGL milaGGL requested review from a team as code owners January 10, 2025 15:58
@milaGGL milaGGL changed the title sort strings in UTF-8 encoded byte order FIX: sort strings in UTF-8 encoded byte order Jan 13, 2025
return compareBytes(leftBytes, rightBytes);
}

private static int compareBytes(ByteString leftBytes, ByteString rightBytes) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

compareByteStrings would be a better name.

@product-auto-label product-auto-label bot added size: m Pull request size is medium. and removed size: l Pull request size is large. labels Jan 15, 2025
@milaGGL milaGGL merged commit 4309639 into main Jan 15, 2025
23 checks passed
@milaGGL milaGGL deleted the mila/string-uses-byte-comparison branch January 15, 2025 17:22
dconeybe added a commit that referenced this pull request Jul 7, 2025
The semantics of this logic were originally fixed by #1967, but this fix
caused a material performance degradation, which was then improved by #2021.
The performance was, however, still suboptimal, and this PR further improves the
speed back to close to its original speed and, serendipitously, simplifies the
algorithm too.

This commit effectively ports the following two PRs from the
firebase-android-sdk repository:

- firebase/firebase-android-sdk#7098
- firebase/firebase-android-sdk#7109
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: firestore Issues related to the googleapis/java-firestore API. size: m Pull request size is medium.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants