What happens when you enqueue a 5000 byte datagram in a 1500 byte MTU?

Datagrams would block sending until:

1. the MTU increases such that a Packet can contain the given datagram (unlikely), or
2. the user removes the datagram from the MTU with retain_datagrams

I had toyed with making the on_transmit implementation smarter, such that it will add whatever datagrams to the packet that it can, even going out-of-order if necessary. For example, if the queue is datagrams of the size [5000, 150, 150, 5000] and an MTU of 1500, datagrams 2 and 3 would eventually be sent, while datagrams 1 and 4 will not until the MTU changes. However I wanted this MR to retain as much of the original behavior of on_transmit as possible, while also correcting the behavior

Datagrams would block sending until the user removes the datagram from the MTU with retain_datagrams

This is a pretty significant change in behavior. Blocking queues is going to cause all sorts of issues and is why the default implementation is the way it is currently. If the head of the queue isn't able to be transmitted with the current MTU then it's unlikely to ever be transmitted.

If you're wanting different behavior you're free to implement your own datagram provider. That's exactly why this functionality is a provider: the default semantics won't make everyone happy.

Fair enough, we'll likely implement our own Sender then.

I would argue that dropping the datagram silently without any warning is not what most would expect. I agree that the default semantics are never going to make everyone happy, however a small warning emitted or an Error bubbled up to the caller seems entirely reasonable and something that I think the vast majority of users would appreciate and would not change the current semantic behavior of the on_transmit function.

If you'd like to keep the current behavior as the default, that is understandable. However there's likely still a bug in the waker behavior. When that too-large-to-send datagram is dropped from the head of the queue, the waker is not woken. Therefore, if the queue is full and someone is waiting to add to the queue, they won't be notified that the queue now has empty space. This will happen whenever the queue is full of too-large-to-send datagrams.

If you'd like, I can modify #2213 to better capture a warning/error as a proposed solution, plus a fix to the waker behavior.

Additionally, if the following is true (which I agree it is):

If the head of the queue isn't able to be transmitted with the current MTU then it's unlikely to ever be transmitted.

Then users should not be able to send_datagram, send_datagram_forced, or poll_send_datagram if the size of the datagram is > than the MTU. Those should all return an error. At least then the application would be able to make some informed decision about what it should do with it's too large datagram, versus just hoping it gets sent, when it never will be.

If you'd like, I can modify #2213 to better capture a warning/error as a proposed solution, plus a fix to the waker behavior.

Yes let's do that. Thanks!

Done, closing PR.