Message delivery issues in EFB Telegram Master channel (comparing to generic IM services)

Different from how usually an IM would work, EFB Telegram Master channel (ETM) strongly rely on Telegram Bot platform. This had made ETM more difficult to deal with messages failed to deliver.

This article is first published on ETM Wiki on 20 April, 2019.

Dealing errors in a generic IM (example)

Below is a simplified flow of message delivery in a generic IM.

A simplified flow of message delivery in a generic IM

1.1.1. Deliver message

If the client failed to send a message to the server, it would keep the message locally and retry when possible, until a receipt is received from the server.

In the case where the server received the message but the receipt is failed to deliver back, the client would continue to try sending the message to server. But since the server keeps a record of all messages with their ID, it could simply identify the request as duplicate, drop it, and redeliver the receipt.

1.1.2. Verify message content

Malformed message content can be detected, and rejected by the server, which can be reflected on the receipt. The client can know that the message is rejected and prompt user to try resend.

1.2.1. Notify client

Similar to 1.1.1., a receipt-based mechanism can be implemented to notifications as well. Even if the notification is delivered without receipt, and missed, the user may still open the app and see the message when message queues are refreshed.

1.2.2. Retrieve message

This is always a stable step. Similar to an HTTP GET request, it can always retry the request until getting what it wants. Parameters like “last message ID” can be used to minimize the data of content during retries.

Dealing errors in ETM

Below is a simplified workflow of ETM message delivery.

A simplified workflow of ETM message delivery

$PlantUML Syntax: @startuml skinparam monochrome reverse skinparam sequenceReferenceAlign center skinparam sequenceMessageAlign center actor User participant “Telegram Bot API” as API participant ETM participant “Slave Channels\n//(via framework)//” as Slave == Send message == User -> API: 2.1. Send message activate API API -> ETM: 2.1.1. Receive message deactivate API activate ETM ETM -> ETM: 2.1.2. Verify & transform activate ETM deactivate ETM ETM -> Slave: 2.1.3. Send message activate Slave ||| Slave –> ETM: Message ID deactivate Slave ETM -> ETM: 2.1.4. Record message activate ETM deactivate ETM ||| deactivate ETM == Receive message == Slave <- : 2.2. Receive message activate Slave Slave -> ETM: 2.2.1. Deliver message deactivate Slave activate ETM ETM -> ETM: 2.2.2. Verify & transform activate ETM deactivate ETM ETM -> API: 2.2.3. Send message activate API API –> ETM: Message ID deactivate API ETM -> ETM: 2.1.4. Record message activate ETM deactivate ETM deactivate ETM @enduml ” usemap=”#plantuml_map”> Error could occur in one or more of the following scenarios: <h3 class=$ 2.1.1. Receive message

Due to the limitation of Telegram Bot API, each message can only be retrieved once. There are no way to request for past messages. Therefore, in some cases, ETM might not even know if a message came, despite the user may see the message has been “successfully sent”.

2.1.2. Verify and transform message

In the case that ETM has to reject message, it can only inform the user by replying the user with a new message informing the failure of delivery. (see 2.2.3 for failure case)

2.1.3. Send message to slave channel

The message might not be delivered due to multiple reasons, probably due to rejection of middleware, slave channel, or failure of delivery to the remote IM. When ETM detects such exception, it will try to inform the user via a reply. (see 2.2.3 for failure case)

2.2.1. Deliver message to ETM

As this portion should be taken care by the framework, middlewares and slave channels, ETM would not, and should not be aware of any missing messages.

This is just listed here for completeness.

2.2.2. Verify and transform

Usually when an unsupported message is received by ETM, it would be delivered in the fallback form (text message) to the user. Thus there should not be any error occuring on this stage.

2.2.3. Sending message to Telegram (incl. any error messages)

This is another crucial scenario in the entire message process. It is (almost) the only way that ETM can pass information to the user. However, ETM cannot do much on this situation other than just retry.

Retry for a fixed number of times
This would not work in some cases. As in the case of internet disruption, all of the consecutive requests would fail until giving up, leading to a misdelivery.
Retry indefinitely
This would work in some cases. However, some users have reported that the same message was flooded to him when “retry indefinitely” strategy is used. This is potentially caused by incomplete response from the server, which has resulted in a detection of failure, despite the message is successfully sent to the server. Furthermore, there is no API from Telegram that can allow a bot to double-check if a previous message is successfully sent.

In general, the main problem of message delivery for a master channel rely on Telegram Bot API is that:

Client is not informed when ETM has difficulty retrieve message from Telegram
Lack of idempotent method of message re-delivery compare to generic IM implementations