When utilizing Go’s great feature of concurrency and goroutines, it is not safe to start too many goroutines at once, say 10000 or 10,000 goroutines running at the moment, these numbers could be too high and not safe as well, could also cause runtime error or panic.
How can we control and limit the total number of goroutines running at the same time? See the following example.
When the waitChan is full(reach the limit of MAX_CONCURRENT_JOBS), the for loop blocks and waiting for one of the running goroutines to be completed, once a goroutine is completed, <-waitChan executed, a position in waitChan is released, then the for loop can continue to start another goroutine.
So the result is that the number of goroutines you will have is at most MAX_CONCURRENT_JOBS at the same time.
package main
import (
"fmt"
"math/rand"
"time"
)
// change this for your situation, 20 or 30, 1,000 or 10,000 may be too high
const MAX_CONCURRENT_JOBS = 2
func main() {
waitChan := make(chan struct{}, MAX_CONCURRENT_JOBS)
count := 0
for {
waitChan <- struct{}{}
count++
go func(count int) {
job(count)
<-waitChan
}(count)
}
}
func job(index int) {
fmt.Println(index, "begin doing something")
time.Sleep(time.Duration(rand.Intn(10) * int(time.Second)))
fmt.Println(index, "done")
}
Output:
2 begin do something
1 begin do something
2 done
3 begin do something
1 done
4 begin do something
3 done
5 begin do something
5 done
6 begin do something
4 done
7 begin do something
6 done
8 begin do something
8 done
9 begin do something