ARTICLE AD BOX
I am making cURL request using GuzzleHttp PHP library that returns me a huge list of data I later process. Because of the size and response time, I tried to optimize my side by using stream option enabled. As I am able to process the response by chunks, it saves a lot of memory and I can process the response before the server returns the whole content.
There is a one problem I encounter on to - the response is not complete. It is like a part of the response (depends, sometimes more than half of it, sometimes only the very last bytes) is cut out of and never sent to me. I should note that the correct gzip compressed response is about 15 MB.
I am not allowed to provide a specific endpoint/host I prompt, but basically the following info is requested and received:
GET <censored> HTTP/1.1 Host: <censored> Accept-Encoding: zstd;q=1.0, br;q=0.9, gzip;q=0.8, deflate;q=0.7, *;q=0.1 TE: traileds, gzip;q=0.9, deflate;q=0.7, compress;q=0.7 Accept: application/json User-Agent: GuzzleHttp/7 Authorization: <censored>Guzzle Settings:
[ 'headers' => [ 'Accept-Encoding' => 'zstd;q=1.0, br;q=0.9, gzip;q=0.8, deflate;q=0.7, *;q=0.1', 'TE' => 'trailers, gzip;q=0.8, deflate;q=0.7, compress;q=0.7', 'Accept' => 'application/json', ], 'timeout' => 60 * 45, 'connect_timeout' => 60 * 45, 'cookies' => false, 'stream' => true, 'allow_redirects' => true, 'http_errors' => true, 'version' => 1.1, 'debug' => true, ],Debug (stream enabled):
<GET <censored>> [CONNECT] <GET <censored>> [MIME_TYPE_IS] message: "application/json" <GET <censored>> [PROGRESS] <GET <censored>> [PROGRESS] bytes_transferred: "8192" ...Debug messages about bytes transferred repeats until the very end, increasing the bytes transferred value.
I process the response using something like
// Ensured that the response is valid $buffer = ''; while (!$stream->eof()) { $buffer.=$stream->read(8192); // here, my handler is able to process the buffer and clear its data if processed. No special-relevant logic here. } // For debugging right now: dump($stream->getMetadata()); // here my handler processes the rest of the buffer. Again, no relevant logic.Every dump showed that the stream was correctly finished - no timeout (which I though might happen), and unread byte.
The very same request with compression, same protocol version, headers.. BUT without stream Guzzle option enabled results into valid response body.
What I tried to verify:
the problem occurs on different PHP versions (PHP version not revelant - 7.4; 8.2) occurs on different global networks (does not seems like a specific network problem) occurs on different PHP servers (does not seems like a specific device problem) occurs on different Guzzle versions (7.8.2-7.10.0) sometimes the response body is complete (not usually) increasing timeouts (timeout, connect_timeout, read_timeout) to insane values does not help unsetting stream option or setting it to false (default) always returns complete responsePlease, any suggestions/tips what might be the problem, how to solve it? I really see performance benefits using stream and I would like to use that.
