Original post


I’m writing this web crawler thing that checks a array of urls, finds if a certain url is on a page then crawls that url too. So I tried putting all the code for sending request and getting response in its own function, but something (probably obvious) is wonky with the defer resp.Body.Close() because it throws http2: response body closed. Attempted function below

func httpGet(url string) *http.Response {
    req, _ := http.NewRequest("GET", url, nil)
    req.Header.Set("User-Agent", detRandomStr(userAgents))
    client := torClient()
    resp, err := client.Do(req)
    if err != nil {
        log.Fatal("Error making GET request.", err)
    defer resp.Body.Close()

    if resp.StatusCode != 200 {
        log.Fatalf("status code error: %d %s", resp.StatusCode, resp.Status)
    return resp

I was trying to call this within a loop of urls, then run its resp returned through goquery, find the next url, and so on.