On Mac OS X, sending the slip packet byte by byte results in an abysmal
performance. The first byte takes up to 160ms to send, and each next
byte approximately 250ms. The packet to request a data block is
typically 7 bytes large, and therefore takes about 1660ms to send.
Because a dive is transmitted as multiple smaller packets (typically
144 bytes without protocol overhead), downloading a single dive can
easily take several seconds.
However, when sending the entire slip packet at once, the time remains
roughly identical to sending just the first byte. The result is that
the time for sending a packet reduces significantly, proportional to
the length of the packet.
Under the hood, the slip packet is now internally buffered, and the
buffer is send only when the entire packet is complete, or whenever the
buffer gets full. But in practice, the buffer is large enough to always
store an entire packet.
In the original bug report, downloading 57 dives took about 40 minutes.
After applying the patch, that time reduced to only 5 minutes!
The firmware version and serial number are stored in the final block
of each dive. That makes it very tricky to support the devinfo event
correctly. For an efficient implementation of the fingerprint feature,
the devinfo event should be emitted before downloading the manifests
or the dives. Fortunately it turns out it is actually possible to
retrieve the firmware version and serial number independently, using
the special identifier command.
A shutdown command should be send to the device, before the connection
is actually closed. In the absence of this command, the device will
display an error, even if the data transfer itself was successful!
Because the size of a dive isn't known in advance, we use the worst case
value of 0xFFFFFF (or nearly 16MB). However in practice, the real size
is many orders of magnitude smaller, even after the decompression.
Instead of pre-allocating a huge memory buffer, we now start with a much
smaller one, and increase when necessary.
For the predator and petrel manifests, where the size is known in
advance, we continue to pre-allocate the exact amount of memory as
before.
The new Petrel protocol uses a simple data compression scheme to reduce
the transfer times. The data is broken up into blocks of 32 bytes each.
Each block except the first is XOR'ed with the previous block, creating
large runs of zeros due to the similarity of the data. The zeros are
then run-length encoded (RLE) to save space.
This is done in preparation for the implementation of the new Petrel
protocol, which shares the low level communication with the existing
Predator protocol.
