How it is done in proven standard solutions
In an ideal world 2 layers are necessary. This (separation of 2 layers) is something that [modbus.org] [modbus protocol] gets perfectly right. No wonder modbus is the absolute market-leader protocol for the exact task that we are inventing our own for.
application layer is responsible for content dispatch (appPDU: which byte means what?)
- while [http://www.modbus.org/modbus/standMbusLibrary.nsf/fa_libsearch?OpenForm# modbus application protocol] is nice and maps very well to our application (so it is a good idea to browse through the 40 pages to get an idea of commands)
- I think that it's OK for now to have a simplified "sipr" implementation instead of a conformant modbus
- Note: the operations that get/set registers work on 16 bit for modbus, while some of our registers are 8 bit: when we walk towards conformancy, we can treat 2x8 bit config elements as 1 word (some coding inconvenience) or send 16bits over the network but only store the MSB (tiny network overhead)
network layer is responsible for framing and reliable communications (CRC)
- modbus appPDU length is not explicite. It is well defined, but not explicitely at given position inside the appPDU. The intention is clear: the application layer deals with frames and the network layer must provide framing in whatever way is appropriate
- [modbus over serial line] is a very good example of framing without byte stuffing (at least the obligatory RTU, not the optional ASCII), but based on "frame timing" information (blank between transmitted bytes) which might not work on win32 (especially if USB-serial is also involved)
- the most efficient, still win32 friendly protocol: frame marker, explicite frame-length and CRC. Without byte-stuffing the frame-marker and CRC (together) make it easy to find the frame for virtually no added complexity and with very little CPU overhead even if it gets lost and clearly the most performant of all for normal transmission (the "find-frame-by-help-of-CRC method is proven even in gigabit ATM, with the help of frame-marker it just cuts down "worst-case" costs further).
- byte stuffing is also possible of course, with some loss in performance even for the normal (no transmit-error) case
- when packing modbus-like (answer on virtual request) PDU-s in MMC flash (datalogging) the framing based on "frame timing" is a no-go. (but either of above 2 works)
- _crc16_update() in avr/crc16.h seems to be an extremely efficient implementation (very fast, and very small flash) of the (0xA001 polynom) modbus.org compatible CRC.
- It must be possible to leave out CRC, if the carrier (CAN, zipfile, etc...) provides CRC anyway. That is, when another (than serial) network layer is used.
Note that modbus solves some existing problems out of the box, in a standard way:
- besides setting and querying traditional variables (that can be either config or runtime)
- trigger and other event logging (query FIFO command)
modbus shortcomings
- strict predefined allocation of master. When we have a small WBO2+other sensors dongle, a display, an ECU, and a PC to chat, which one should be the master? Any 2 should be able to cooperate without the other 2 present.
- I thought about a simple master election that works on logical level. Even if the master is shut down, the bus continues to work (if otherwise operable physically: unpowered devices don't disturb the RS485 bus) if the slaves detect the loss of master: after 1000 usec silence, all slaves choose a random number t_rand=0..2000 usec and after t_rand time elapsed becomes the master by advertising itself unless another node became master in the meantime.
- there is also a hardware issue: there is a slight recommendation to place "line polarization" (almost like pullup, but symmetrical) into the master. It is probably the best to not install "line polarization" too hard on boards, but leave that to the installer along with "line termination"-s (easy to make in the standard DSUB9)
- no slave-to-slave direct communication (although the RS485 bus would allow)
- slave cannot talk at all unless requested by master
- in the reference 4 units on the RS485 bus with flows: PC=>ECU, sensors=>ECU, ECU=>PC, ECU=>display case the control of flows is non-trivial (unless the ECU-PC link is separate; than the ECU can be the absolute master)
- when 2 ECU-s are interested in the same signal (eg. MAP signals) data, the standard way is to relay data. but it would be possible that the slave ECU cathes the data at the same time as the first ECU
- if I understand right, the master cannot ask another question until reply is received (or timeout)
- this decreases max utilization on the bus (since 115200 is easily possible even with 70m cable, might not be a real problem for most applications)
- different slaves could work on different questions independently
- this could be a place to improve modbus for even higher performance. Collisions must be avoided, though (nontrivial: might not worth the hassle).
- if the questions are simple (such as the case for the typical register-reads) the timeout can be very low and utilization high with short idle-times. Actually, the opposite can be a problem: assuring no sooner reply than the required 3.5 symbol blank-time (315 usec at 115200 baud).
- I don't see a way in modbus to pack multiple commands into a frame
- sigh: the application layer packets can be very short
- the overhead of a network packet might be significant because of CRC, acknowledgement, and possibly routing and bandwidth. It is extremely prohibitive in wireless communications.
- with the incredibly nice CRC it might not be a big issue.
- if the application's PDU-s are made sufficiently big (eg. a set of data sent in one batch, like all 16 ADC values; modbus supports this, see command function code 0x04) it can work efficient without any hacks, even when data goes to MMC flash.
The "wired in CRC", which is the violation of the 2-layer design (application + network, see above) is not a performance issue. It becomes a maintenance issue when we start support for CAN or MMC flash or other data carrier besides serial link. No question that we can solve it than though.
Comments From Alexander Guy:
I think that byte stuffing isn't something that should be overlooked. It's easy to implement, and with it and frame 'flags', frame synchronization becomes trivial. Rather than counting bytes based on the alleged size of the frame, the frame continues until the next flag is hit. If no flag is hit before the maximum buffer size is reached, it can be assumed that the frame was invalid and frame reading can be reset until the next flag.
A rough implementation of an input function is as follows. This is the same HDLC-like framing that PPP uses over 8-bit async serial links:
- \\n
#define BYTE_FLAG 0x7e #define BYTE_ESCAPE 0x7d #define MAX_SIZE 80 struct { char rbuf[MAX_SIZE]; int ridx; enum { INVALID, NORMAL, ESCAPE } rstate; } commstate; void input_byte(char input) { int idx = commstate.ridx; if (input == BYTE_FLAG) { if ((commstate.rstate == NORMAL) && (idx > 0)) { /* XXX - Verify Checksum and Process Message Here */ } commstate.ridx = 0; commstate.rstate = NORMAL; return; } if (commstate.rstate == INVALID) return; if (input == BYTE_ESCAPE) { commstate.rstate = ESCAPE; return; } if (commstate.rstate == ESCAPE) { input ^= 0x20; commstate.rstate = NORMAL; } commstate.rbuf[idx] = input; /* XXX - You Can Update Checksum Here */ idx++; if (idx >= MAX_SIZE) { idx = 0; commstate.rstate = INVALID; } commstate.ridx = idx; return; }
See also
- GenBoard/BinaryProtocol - predecessor (almost same)