valyala/fasthttp

fasthttp client. Read body in chunks and stop downloading on condition.

Open

#1065 aperta il 5 ago 2021

Vedi su GitHub
 (8 commenti) (0 reazioni) (0 assegnatari)Go (1755 fork)batch import
help wanted

Metriche repository

Star
 (21.741 star)
Metriche merge PR
 (Merge medio 3g) (44 PR mergiate in 30 g)

Descrizione

For example, I want to stop downloading content if it has wrong mime type. How can I do so? The documentation shows only resp.Body() which gets whole response. Below is how I use standard http library:

resp, err := client.Do(req)
if err != nil {
	return nil, err
}
body := resp.Body
defer body.Close()

checkedMime := false
readBuffer := make([]byte, 1 << 12)
bodyData := make([]byte, 0, typicalSize)
for {
	n, err := body.Read(readBuffer)
	if err != nil && err != io.EOF {
		return nil, err
	}
	bodyData = append(bodyData, readBuffer[0:n]...)
	if !checkedMime && len(bodyData) > 512{			
		mimeType := http.DetectContentType(bodyData)
		if isArchive(mimeType){
			isArchive = true
			break
		}
		if mimeType  != "text/html"{
			isUselessContent = true
			break
		}
		checkedMime = true
	}
	if len(bodyData) > tooBigContentSize {
		tooBigContent = true
		break
	}
	if err == io.EOF {
		break
	}
}
// do something with bodyData, isArchive, isUselessContent, tooBigContent
...

Guida contributor