Introduce a timeout when processing incoming messages

Antonio Scandurra and Nathan Sobo created 3 years ago

We have an hypothesis that the server gets stuck while processing
an incoming message, either because the buffer fills up or because
a handler never completes. This should mitigate that and, once we
add logging, give us some clue as to what is causing it exactly.

Co-Authored-By: Nathan Sobo <nathan@zed.dev>

Change summary

crates/rpc/src/peer.rs | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)

Detailed changes

crates/rpc/src/peer.rs 🔗

@@ -175,8 +175,10 @@ impl Peer {
                             let incoming = incoming.context("received invalid RPC message")?;
                             receive_timeout.set(create_timer(RECEIVE_TIMEOUT).fuse());
                             if let proto::Message::Envelope(incoming) = incoming {
-                                if incoming_tx.send(incoming).await.is_err() {
-                                    return Ok(());
+                                match incoming_tx.send(incoming).timeout(RECEIVE_TIMEOUT).await {
+                                    Some(Ok(_)) => {},
+                                    Some(Err(_)) => return Ok(()),
+                                    None => Err(anyhow!("timed out processing incoming message"))?,
                                 }
                             }
                             break;