Streaming data to BQ using Storage Write API not possible
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
Research direction
Start with BigQueryStorage/metadata/descriptor_fix.php and samples/V1/BigQueryWriteClient/append_rows.php, then trace how AppendRowsRequest and ProtoData are constructed. Determine whether the sample can clearly show a supported way to send rows without confusing descriptor setup; done means the streaming path and required data format are documented or the blocking behavior is precisely identified.
Written by the indexing model from the issue text.
Description
Environment details
- OS: Windows/Mac
- PHP version: 8.2/8.3
- Package name and version: google/cloud-bigquery-storage v2.1.3
Steps to reproduce
It seems to be impossible write data because of descriptor_fix.php (though it might be solving other issues).
I couldn't find any other way to use AppendRows other than using Proto-stuffs functionality.
The provided sample does not relieve the pain of explaining how data is sent.
There are mentions of fixes and AppendsRows, mainly by @bshaffer , but I could not find any information on how to move forward without using Proto-stuffs.
Code example
// prerequisites
$bigQueryWriteClient = new BigQueryWriteClient();
$parent = 'pathhere';
// setup the structure
$row = new Struct();
$row->setFields([
'impressions' => (new Value())->setNumberValue(213),
]);
$protoRows = new ProtoRows();
$protoRows->setSerializedRows([$row->serializeToString()]);
// add the data
$protoData = new ProtoData([
'rows' => $protoRows,
]);
// set the schema, which requires the descriptor
$protoData->setWriterSchema(new ProtoSchema([
'proto_descriptor' => new \Google\Protobuf\DescriptorProto(),
]));
// create the append rows request
$appendRowsRequest = new AppendRowsRequest([
'write_stream' => $parent . '/streams/_default',
'proto_rows' => $protoData
]);
// do the writing
$stream = $bigQueryWriteClient->appendRows();
$stream->write($appendRowsRequest);
// finalize
foreach ($stream->closeWriteAndReadAll() as $element) {
printf('Element data: %s' . PHP_EOL, $element->serializeToJsonString());
}
- Dominant language
- PHP
- Stars
- 1.2k
- Forks
- 464
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 103
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from googleapis/google-cloud-php
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
googleapis/google-cloud-php#9730 ·
-
type: feature request
Difficulty 1/5 Under an hour Newbie friendliness 75/100
googleapis/google-cloud-php#9716 · 11 reactions ·
-
Difficulty 3/5 1-2 days Newbie friendliness 65/100
googleapis/google-cloud-php#9725 ·
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
googleapis/google-cloud-php#9675 · 1 comment ·
-
Difficulty 4/5 3-5 days Newbie friendliness 55/100
googleapis/google-cloud-php#9674 · 1 comment ·
All issues in googleapis/google-cloud-php
Similar issues
-
tooling
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
UX
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
ProfessionalWiki/NeoWiki#1525 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
OpenConext/OpenConext-engineblock#2122 ·
-
Bug
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
Automattic/safe-publish#594 ·