feat: create manual client for Write API #112

yirutang · 2020-03-19T23:04:14Z

Fixes #111 ☕️

A StreamWriter that provides built-in capabilities to help making bi-di streaming client easy to use:

handle batching of messages;
controlling memory utilization (through flow control);
automatic connection re-establishment;
request cleanup (only keeps write schema on first request in the stream).

Example:
WriteStream writeStream =
client.createWriteStream(
CreateWriteStreamRequest.newBuilder()
.setParent(tableId2)
.setWriteStream(WriteStream.newBuilder().setType(WriteStream.Type.PENDING).build())
.build());
StreamWriter streamWriter = StreamWriter.newBuilder(writeStream.getName()).build();
ApiFuture response =
streamWriter.append(
createAppendRequestComplicateType(writeStream.getName(), new String[] {"aaa"})
.setOffset(Int64Value.of(0L))
.build());
assertEquals(0, response.get().getOffset());
streamWriter.shutdown();
}

First version, test to be developed.

…brary helps to convert a ProtoDescriptorProto out of ProtoDescriptor into a self contained ProtoDescriptorProto, that can be passed into API and reconstructed on the server side. modified: google-cloud-bigquerystorage/pom.xml new file: google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/ProtoSchemaConverter.java new file: google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/ProtoSchemaConverterTest.java new file: google-cloud-bigquerystorage/src/test/proto/test.proto modified: pom.xml

First version, test to be developed.

… into work

modified: google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java modified: google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/FakeBigQueryWriteImpl.java modified: google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/FakeScheduledExecutorService.java modified: google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriterTest.java modified: google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java

modified: google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java

… into work

codecov · 2020-03-20T14:57:17Z

Codecov Report

Merging #112 into master will increase coverage by 1.52%.
The diff coverage is 81.05%.

@@             Coverage Diff              @@
##             master     #112      +/-   ##
============================================
+ Coverage     72.36%   73.88%   +1.52%     
- Complexity      281      319      +38     
============================================
  Files            39       42       +3     
  Lines          1809     2229     +420     
  Branches          4       70      +66     
============================================
+ Hits           1309     1647     +338     
- Misses          496      552      +56     
- Partials          4       30      +26

Impacted Files	Coverage Δ	Complexity Δ
...google/cloud/bigquery/storage/v1alpha2/Waiter.java	`72.54% <72.54%> (ø)`	`14 <14> (?)`
.../cloud/bigquery/storage/v1alpha2/StreamWriter.java	`81.17% <81.17%> (ø)`	`28 <28> (?)`
...igquery/storage/v1alpha2/ProtoSchemaConverter.java	`93.54% <93.54%> (ø)`	`6 <6> (?)`
...igquery/storage/v1beta1/BigQueryStorageClient.java	`68.75% <0%> (-1.84%)`	`16% <0%> (-3%)`
...ery/storage/v1beta1/BaseBigQueryStorageClient.java	`68.75% <0%> (-1.84%)`	`16% <0%> (-3%)`
...d/bigquery/storage/v1beta2/BigQueryReadClient.java	`51.51% <0%> (-1.43%)`	`8% <0%> (-1%)`
.../cloud/bigquery/storage/v1/BigQueryReadClient.java	`51.51% <0%> (-1.43%)`	`8% <0%> (-1%)`
...ud/bigquery/storage/v1/BaseBigQueryReadClient.java	`54.54% <0%> (-1.34%)`	`9% <0%> (-1%)`
...gquery/storage/v1beta2/BaseBigQueryReadClient.java	`54.54% <0%> (-1.34%)`	`9% <0%> (-1%)`
...ery/storage/v1beta2/stub/GrpcBigQueryReadStub.java	`89.74% <0%> (-0.62%)`	`9% <0%> (-1%)`
... and 15 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c4814dc...f942f48. Read the comment docs.

stephaniewang526 · 2020-03-20T15:10:35Z

please take a look at the various build failures.

modified: google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java modified: google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java

…work

…work modified: google-cloud-bigquerystorage/pom.xml modified: google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java modified: google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriterTest.java modified: google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java modified: google-cloud-bigquerystorage/src/test/proto/test.proto modified: google-cloud-bigquerystorage/pom.xml modified: google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java modified: google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriterTest.java modified: google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java modified: google-cloud-bigquerystorage/src/test/proto/test.proto

…work

google-cloud-bigquerystorage/pom.xml

...d-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java

stephaniewang526 · 2020-03-24T19:26:25Z

As a general note, all public methods need to be documented clearly in accompanying Java doc to help developers understand what the inputs, outputs, exceptions, etc are. I see a number of public methods without Java doc. Please make sure to put them in.

...d-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java

...e-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/Waiter.java

...test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java

…gquery/storage/v1alpha2/Waiter.java Co-Authored-By: Stephanie Wang <stephaniewang526@users.noreply.github.com>

…gquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java Co-Authored-By: Stephanie Wang <stephaniewang526@users.noreply.github.com>

yirutang

PTAL

...e-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/Waiter.java

remove dupe gax; add undeclared commons-lang3

98851e9 commit 98851e9 Author: Yiru Tang <yiru@google.com> Date: Thu Apr 2 17:45:37 2020 -0700 feat: create manual client for Write API (#112) * First writeapi manual client First version, test to be developed. * Manual client with e2e * StreamWriter library * ProtoSchema convert library WriteApi client library addition. This library helps to convert a ProtoDescriptorProto out of ProtoDescriptor into a self contained ProtoDescriptorProto, that can be passed into API and reconstructed on the server side. modified: google-cloud-bigquerystorage/pom.xml new file: google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/ProtoSchemaConverter.java new file: google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/ProtoSchemaConverterTest.java new file: google-cloud-bigquerystorage/src/test/proto/test.proto modified: pom.xml * First writeapi manual client First version, test to be developed. * . * incremental development * . modified: google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java modified: google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/FakeBigQueryWriteImpl.java modified: google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/FakeScheduledExecutorService.java modified: google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriterTest.java modified: google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java * . * . * . * feat: create manual java client for writeapi modified: google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java * feat: add integration test final verification. * . * . modified: google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java modified: google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java * . * . * . * . * avoid duplicate count++ and make it easier to read * Update google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/Waiter.java Co-Authored-By: Stephanie Wang <stephaniewang526@users.noreply.github.com> * Update google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java Co-Authored-By: Stephanie Wang <stephaniewang526@users.noreply.github.com> * Update google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java Co-Authored-By: Stephanie Wang <stephaniewang526@users.noreply.github.com> * . * . * . * . * . * dep: add undeclared dep in parent pom * dep: remove duplicate and add undeclared remove dupe gax; add undeclared commons-lang3 Co-authored-by: Stephanie Wang <stephaniewang526@users.noreply.github.com>

…nerator. (#134) * feat: create manual client for Write API (#112) 98851e9 commit 98851e9 Author: Yiru Tang <yiru@google.com> Date: Thu Apr 2 17:45:37 2020 -0700 feat: create manual client for Write API (#112) * First writeapi manual client First version, test to be developed. * Manual client with e2e * StreamWriter library * ProtoSchema convert library WriteApi client library addition. This library helps to convert a ProtoDescriptorProto out of ProtoDescriptor into a self contained ProtoDescriptorProto, that can be passed into API and reconstructed on the server side. modified: google-cloud-bigquerystorage/pom.xml new file: google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/ProtoSchemaConverter.java new file: google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/ProtoSchemaConverterTest.java new file: google-cloud-bigquerystorage/src/test/proto/test.proto modified: pom.xml * First writeapi manual client First version, test to be developed. * . * incremental development * . modified: google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java modified: google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/FakeBigQueryWriteImpl.java modified: google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/FakeScheduledExecutorService.java modified: google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriterTest.java modified: google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java * . * . * . * feat: create manual java client for writeapi modified: google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java * feat: add integration test final verification. * . * . modified: google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java modified: google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java * . * . * . * . * avoid duplicate count++ and make it easier to read * Update google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/Waiter.java Co-Authored-By: Stephanie Wang <stephaniewang526@users.noreply.github.com> * Update google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java Co-Authored-By: Stephanie Wang <stephaniewang526@users.noreply.github.com> * Update google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java Co-Authored-By: Stephanie Wang <stephaniewang526@users.noreply.github.com> * . * . * . * . * . * dep: add undeclared dep in parent pom * dep: remove duplicate and add undeclared remove dupe gax; add undeclared commons-lang3 Co-authored-by: Stephanie Wang <stephaniewang526@users.noreply.github.com> * deps: update dependency io.grpc:grpc-bom to v1.28.1 (#132) 711c5c7 commit 711c5c7 Author: WhiteSource Renovate <bot@renovateapp.com> Date: Fri Apr 3 03:08:41 2020 +0200 deps: update dependency io.grpc:grpc-bom to v1.28.1 (#132) * chore: release 0.127.0 (#122) 45fc919 commit 45fc919 Author: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com> Date: Thu Apr 2 21:49:46 2020 -0400 chore: release 0.127.0 (#122) * updated CHANGELOG.md [ci skip] * updated README.md [ci skip] * updated versions.txt [ci skip] * updated google-cloud-bigquerystorage-bom/pom.xml [ci skip] * updated grpc-google-cloud-bigquerystorage-v1/pom.xml [ci skip] * updated grpc-google-cloud-bigquerystorage-v1alpha2/pom.xml [ci skip] * updated grpc-google-cloud-bigquerystorage-v1beta1/pom.xml [ci skip] * updated grpc-google-cloud-bigquerystorage-v1beta2/pom.xml [ci skip] * updated proto-google-cloud-bigquerystorage-v1/pom.xml [ci skip] * updated proto-google-cloud-bigquerystorage-v1alpha2/pom.xml [ci skip] * updated proto-google-cloud-bigquerystorage-v1beta1/pom.xml [ci skip] * updated proto-google-cloud-bigquerystorage-v1beta2/pom.xml [ci skip] * updated samples/install-without-bom/pom.xml [ci skip] * updated samples/pom.xml [ci skip] * updated samples/snapshot/pom.xml [ci skip] * updated samples/snippets/pom.xml [ci skip] * updated google-cloud-bigquerystorage/pom.xml [ci skip] * updated pom.xml [ci skip] Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>

….0 (googleapis#112) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [com.google.cloud:libraries-bom](https://togithub.com/GoogleCloudPlatform/cloud-opensource-java) | minor | `5.3.0` -> `5.4.0` | --- ### Renovate configuration :date: **Schedule**: At any time (no schedule defined). :vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied. :recycle: **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. :no_bell: **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#googleapis/java-storage-nio).

yirutang added 12 commits February 25, 2020 10:22

First writeapi manual client

66cea17

First version, test to be developed.

Manual client with e2e

4803752

StreamWriter library

2a29c5f

First writeapi manual client

ca67ee6

First version, test to be developed.

.

efec6e2

incremental development

024e6f9

Merge branch 'work' of https://github.com/yirutang/java-bigquerystorage…

d694ad2

… into work

.

b8b6403

.

638959a

.

9147e96

googlebot added the cla: yes This human has signed the Contributor License Agreement. label Mar 19, 2020

yirutang added 2 commits March 19, 2020 16:16

feat: create manual java client for writeapi

eff0a2a

modified: google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java

Merge branch 'work' of https://github.com/yirutang/java-bigquerystorage…

19c5449

… into work

yirutang changed the title ~~Create manual client for Write API~~ feat: create manual client for Write API Mar 19, 2020

yirutang added 2 commits March 19, 2020 16:29

feat: add integration test final verification.

438b864

.

f8624be

stephaniewang526 added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Mar 20, 2020

yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Mar 20, 2020

stephaniewang526 self-requested a review March 20, 2020 17:55

yirutang added 5 commits March 20, 2020 13:54

.

05935b7

modified: google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java modified: google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java

Merge branch 'work' of github.com:yirutang/java-bigquerystorage into …

2050f50

…work

.

4a1e3ce

Merge branch 'work' of github.com:yirutang/java-bigquerystorage into …

9446f5e

…work

stephaniewang526 reviewed Mar 24, 2020

View reviewed changes

.

b16697c

.

4acea18

stephaniewang526 reviewed Mar 24, 2020

View reviewed changes

...d-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java Show resolved Hide resolved

.

2ff57ae

stephaniewang526 requested a review from bigang-li March 26, 2020 18:31

avoid duplicate count++ and make it easier to read

76622fc

stephaniewang526 suggested changes Mar 31, 2020

View reviewed changes

yirutang and others added 5 commits April 1, 2020 10:53

Update google-cloud-bigquerystorage/src/main/java/com/google/cloud/bi…

4a490c4

…gquery/storage/v1alpha2/Waiter.java Co-Authored-By: Stephanie Wang <stephaniewang526@users.noreply.github.com>

Update google-cloud-bigquerystorage/src/test/java/com/google/cloud/bi…

9e21086

…gquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java Co-Authored-By: Stephanie Wang <stephaniewang526@users.noreply.github.com>

Update google-cloud-bigquerystorage/src/test/java/com/google/cloud/bi…

31542b0

…gquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java Co-Authored-By: Stephanie Wang <stephaniewang526@users.noreply.github.com>

.

c1a3f25

.

7f1c8e1

yirutang commented Apr 1, 2020

View reviewed changes

...e-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/Waiter.java Outdated Show resolved Hide resolved

...e-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/Waiter.java Outdated Show resolved Hide resolved

yirutang added 4 commits April 1, 2020 15:10

.

868ae73

.

58608d5

.

a73a72c

.

87b694d

stephaniewang526 approved these changes Apr 2, 2020

View reviewed changes

stephaniewang526 added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Apr 2, 2020

yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Apr 2, 2020

stephaniewang526 added 2 commits April 2, 2020 15:26

dep: add undeclared dep in parent pom

3d148b3

dep: remove duplicate and add undeclared

f942f48

remove dupe gax; add undeclared commons-lang3

stephaniewang526 added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Apr 2, 2020

yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Apr 2, 2020

stephaniewang526 merged commit 98851e9 into googleapis:master Apr 3, 2020

yirutang deleted the work branch June 5, 2020 17:36

feat: create manual client for Write API #112

feat: create manual client for Write API #112

Uh oh!

Conversation

yirutang commented Mar 19, 2020

Uh oh!

codecov bot commented Mar 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

stephaniewang526 commented Mar 20, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

stephaniewang526 commented Mar 24, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yirutang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov bot commented Mar 20, 2020 •

edited

Loading