Original post

I try to pull the data of a forum with the logic on this page. In this way, I can pull the data in the pages without any problems. But when I create a multi-threading thread for the For loop, the program will exit some time after pulling the data and cannot pull all the data.
Expample : If there are 300 data in total, it will dig 120 data and end the program.

I waiting your solution suggestions. Thank You For Advance

Colly sample: http://go-colly.org/docs/examples/reddit/

My Codes Steps :

1- Select urls to be scanned from the database
2- Execute “c.Visit (URL)” in for loop
3 – Scrap with “c.OnHtml”

My Codes Sample:

   counter := 0
        resultsLen := len(results)
        var wg sync.WaitGroup
        wg.Add(resultsLen)
        for _, result := range results {
                 func(result primitive.M) {
                        defer wg.Done()
                        counter++
                        
                        mainForumUrl := result["url"].(string)
                        fmt.Println("##### Looking Main Forum Url #####",counter,":",mainForumUrl)
                        c.Visit(mainForumUrl)
                        c.Wait()
                }(result)

        }
        wg.Wait()

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.