From 1f1a2e035066ba20a4d6f5bbbd76d6a311cf725b Mon Sep 17 00:00:00 2001 From: Ulf Norell Date: Fri, 15 May 2020 11:57:49 +0200 Subject: [PATCH 1/2] Start describing internal message flows --- doc/architecture.md | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100644 doc/architecture.md diff --git a/doc/architecture.md b/doc/architecture.md new file mode 100644 index 000000000..d081cef1d --- /dev/null +++ b/doc/architecture.md @@ -0,0 +1,32 @@ +# Server architecture + +This document describes the server architecture and internal message flows. It is work in progress. + +## Incoming client request flow + +Web socket connections from the outside are handled by `emqttd_client` (which is a `gen_server2`), +one process per connection. Incoming packets are parsed into an `#mqtt_packet{}` (by +`emqttd_parser`) and then processed as follows: + +- `emqttd_protocol:received/2` does some validation and calls +- `emqttd_session:publish/2` which in case of QoS 2 has a `gen_server2` to handle acks and retries otherwise + simply calls +- `emqttd_server:publish/1`. This runs the `message.publish` hooks in `emqttd_hooks` and then calls +- `emqttd_pubsub:publish/2`. Here routing magic potentially rewrites the topic and we send a plain Erlang + message `{dispatch, Topic, Msg}` to subscribers of the new topic. In this case the subscriber will be an +- `emqttd_session` (`gen_server2`) managing a session with a VNode emqtt client. This forwards the + message (`emqttd_session:deliver/2`) as `{deliver, Msg}` to the +- `emqttd_client` holding the web socket to the VNode. This calls +- `emqttd_protocol:send/2` which serializes and sends the message over a web socket to +- `emqttc_socket` (receive loop). This parses the packet and forwards it to +- `emqttc` (`gen_fsm`). This has some subscription logic which resolves to sending an Erlang message + `{publish, Topic, Payload}` to +- `n2o_pi` (`gen_server`) which in turn calls +- `n2o_vnode:proc/2`. This decodes the payload (`binary_to_term`) and calls +- `n2o_proto:info/3` which looks up the protocol handlers with `application:get_env(n2o, protocols)`. + These are currently `n2o_nitro`, `n2o_ftp`, and `roster_proto` and calls their `info` function. + The interesting one in this case is +- `roster_proto:info/3` which forwards the message to the appropriate `roster_*` module + (`roster_room`, `roster_friend`, etc) where the interesting stuff happens. + + -- GitLab From 54acd30140888621903d468106b5d27412aacbc6 Mon Sep 17 00:00:00 2001 From: Ulf Norell Date: Fri, 15 May 2020 14:16:08 +0200 Subject: [PATCH 2/2] Also document the way back --- doc/architecture.md | 54 ++++++++++++++++++++++++++++----------------- 1 file changed, 34 insertions(+), 20 deletions(-) diff --git a/doc/architecture.md b/doc/architecture.md index d081cef1d..e1e35c0ae 100644 --- a/doc/architecture.md +++ b/doc/architecture.md @@ -8,25 +8,39 @@ Web socket connections from the outside are handled by `emqttd_client` (which is one process per connection. Incoming packets are parsed into an `#mqtt_packet{}` (by `emqttd_parser`) and then processed as follows: -- `emqttd_protocol:received/2` does some validation and calls -- `emqttd_session:publish/2` which in case of QoS 2 has a `gen_server2` to handle acks and retries otherwise - simply calls -- `emqttd_server:publish/1`. This runs the `message.publish` hooks in `emqttd_hooks` and then calls -- `emqttd_pubsub:publish/2`. Here routing magic potentially rewrites the topic and we send a plain Erlang - message `{dispatch, Topic, Msg}` to subscribers of the new topic. In this case the subscriber will be an -- `emqttd_session` (`gen_server2`) managing a session with a VNode emqtt client. This forwards the - message (`emqttd_session:deliver/2`) as `{deliver, Msg}` to the -- `emqttd_client` holding the web socket to the VNode. This calls -- `emqttd_protocol:send/2` which serializes and sends the message over a web socket to -- `emqttc_socket` (receive loop). This parses the packet and forwards it to -- `emqttc` (`gen_fsm`). This has some subscription logic which resolves to sending an Erlang message - `{publish, Topic, Payload}` to -- `n2o_pi` (`gen_server`) which in turn calls -- `n2o_vnode:proc/2`. This decodes the payload (`binary_to_term`) and calls -- `n2o_proto:info/3` which looks up the protocol handlers with `application:get_env(n2o, protocols)`. - These are currently `n2o_nitro`, `n2o_ftp`, and `roster_proto` and calls their `info` function. - The interesting one in this case is -- `roster_proto:info/3` which forwards the message to the appropriate `roster_*` module - (`roster_room`, `roster_friend`, etc) where the interesting stuff happens. +1. `emqttd_protocol:received/2` does some validation and calls +2. `emqttd_session:publish/2` which in case of QoS 2 has a `gen_server2` to handle acks and retries otherwise + simply calls +3. `emqttd_server:publish/1`. This runs the `message.publish` hooks in `emqttd_hooks` and then calls +4. `emqttd_pubsub:publish/2`. Here routing magic potentially rewrites the topic and we send a plain Erlang + message `{dispatch, Topic, Msg}` to subscribers of the new topic. In this case the subscriber will be an +5. `emqttd_session` (`gen_server2`) managing a session with a VNode emqtt client. This forwards the + message (`emqttd_session:deliver/2`) as `{deliver, Msg}` to the +6. `emqttd_client` holding the web socket to the VNode. This calls +7. `emqttd_protocol:send/2` which serializes and sends the message over a web socket to +8. `emqttc_socket` (receive loop). This parses the packet and forwards it to +9. `emqttc` (`gen_fsm`). This has some subscription logic which resolves to sending an Erlang message + `{publish, Topic, Payload}` to +10. `n2o_pi` (`gen_server`) which in turn calls +11. `n2o_vnode:proc/2`. This decodes the payload (`binary_to_term`) and calls +12. `n2o_proto:info/3` which looks up the protocol handlers with `application:get_env(n2o, protocols)`. + These are currently `n2o_nitro`, `n2o_ftp`, and `roster_proto` and calls their `info` function. + The interesting one in this case is +13. `roster_proto:info/3` which forwards the message to the appropriate `roster_*` module + (`roster_room`, `roster_friend`, etc). +Here is where the actual business logic happens. + +14. The result of `n2o_proto:info/3` is turned into a binary and is sent by +15. `n2o_vnode:send/3` which calls +16. `emqttc:publish/4`. This sends packs up the topic and payload into an `#mqtt_packet` and sends a + `{publish, Packet}` event to +17. the `emqttc` `gen_fsm`, which in turn calls +18. `emqttc_protocol:publish/2`. Here the message in turned into bytes and sent over the socket back to the +19. `emqttd_client`. At this point we follow steps 1-7, execpt that it's the VNode client that is + the sender and the user client that receives the message and sends it back to the user. + +Many requests result in additional messages being sent (for instance, sending a message will send it +to the recipient, as well as replying to the sender). These go through `roster:send/3` which calls +`n2o_vnode:send/3` and proceeds from step 15. -- GitLab