How would you implement a connection pool with health checks? The health checks should run in-band at given intervals, but – crucially – they should not be in sync and check all idle connections at the same time, effectively consuming all idle connections while doing so.
I’ve experimented with a chan-based connection pool, and the channel makes it easy to block (with optional timeout) to wait for a free connection. When done with a connection, it is put back into the channel – great. But I failed to add health checks into the mix here: The check would get a connection from the pool, but that connection might not need checking yet (its interval hasn’t ticked). So I put it back and get another, and another, and another – a kind of busy loop. Not great.
A pool connection can have its own health check, but then
pool.GetConn() would have to iterate over the channel, and see if a connection is available before returning it. It would have to track whether it has looped over all connections, but on completing a single sweep, a connection might have become available again, so it should do another sweep? Not pretty.
I was then thinking to use
map[string]net.Conn, but it makes it trickier to implement a timeout on
pool.GetConn(). It would also have to loop over all items in the map to check for free connections.
Any suggestions on how to approach this?