Eagerly anticipating more information on Aeron from Mr. Mechanical Sympathy et. al. "I'm sick of seeing frameworks, our world has become obsessed with it" --Amen brother, Amen.
Nice talk. Takes me back a few decades! On the subject of NAK in a multicast environment and using exponential delay to avoid storms... When working on a stock exchange project many moons ago we layed our own protocol on top of UDP and had the additional requirement that all receivers (1000 trader workstations) had to receive the data at the same time, in this case the network was designed with no active components (to avoid queuing at routers) and all worked fine with similar counters/sliding window as you described however during packet loss (e.g. RF from passing taxi) the naive implementation causes all consumers to NAK and causes a storm and subsequent chaos; our eventual solution was to have a hierarchy of 'watch dogs' where certain machines would be fast responders, other layers of the hierarchy would react slower... a fast reaction by a fast watchdog would cause a retransmission that subsequently appeased the other machines. By creating a hierarchy that reflected the failure modes in the network the storms were avoided.... and as you say very nice to see a room of 1000 machines operating nicely. I'll check out the project... sounds fun.
Exceptional! Fairly dense material, but thanks for showcasing some of the design considerations of the library. You do a great job of not losing the forest from the trees while still providing great jumping off points for people to dive into the technical rigor. Glad to see some of what's powering some pieces of the Akka library underlyingly.
Frameworks v/s Libraries. Thumbs up! Underlining Simplicity and really putting it into practice here. All of this look simple yet effective. Clear cut separation of duties way before in the design phase. A fantastic point on page faults. Not having to mmap an extremely large file is a important problem to solve (without latency jitters). Eager to look at this. Designing keeping hardware in mind from the very start. Peeking into the C++ libraries now! Thank you.
Interesting stuff. Thanks for putting it up. Slightly naive question perhaps. At 26:37 - given that messages can be different sizes, how do you know how far along to move the high water mark?
The message header contains the starting offset in the term plus the length. This allows for exact placement. The full protocol description can be found in the Wiki. github.com/real-logic/Aeron/wiki/Protocol-Specification
Timely solution - keen to get involved and learn more. Thinking of integrating this with golang channels as the API, should be fun to code in, and very performant.
steveoc64 By using the BufferClaim strategy and a codec like SBE then golang channels could be very efficient. Especially as I'm planning on adding IPC support in the next month or so.
Eagerly anticipating more information on Aeron from Mr. Mechanical Sympathy et. al.
"I'm sick of seeing frameworks, our world has become obsessed with it" --Amen brother, Amen.
Nice talk. Takes me back a few decades! On the subject of NAK in a multicast environment and using exponential delay to avoid storms... When working on a stock exchange project many moons ago we layed our own protocol on top of UDP and had the additional requirement that all receivers (1000 trader workstations) had to receive the data at the same time, in this case the network was designed with no active components (to avoid queuing at routers) and all worked fine with similar counters/sliding window as you described however during packet loss (e.g. RF from passing taxi) the naive implementation causes all consumers to NAK and causes a storm and subsequent chaos; our eventual solution was to have a hierarchy of 'watch dogs' where certain machines would be fast responders, other layers of the hierarchy would react slower... a fast reaction by a fast watchdog would cause a retransmission that subsequently appeased the other machines. By creating a hierarchy that reflected the failure modes in the network the storms were avoided.... and as you say very nice to see a room of 1000 machines operating nicely. I'll check out the project... sounds fun.
Exceptional! Fairly dense material, but thanks for showcasing some of the design considerations of the library. You do a great job of not losing the forest from the trees while still providing great jumping off points for people to dive into the technical rigor.
Glad to see some of what's powering some pieces of the Akka library underlyingly.
Frameworks v/s Libraries. Thumbs up!
Underlining Simplicity and really putting it into practice here. All of this look simple yet effective.
Clear cut separation of duties way before in the design phase.
A fantastic point on page faults. Not having to mmap an extremely large file is a important problem to solve (without latency jitters). Eager to look at this.
Designing keeping hardware in mind from the very start.
Peeking into the C++ libraries now! Thank you.
Interesting stuff. Thanks for putting it up. Slightly naive question perhaps. At 26:37 - given that messages can be different sizes, how do you know how far along to move the high water mark?
The message header contains the starting offset in the term plus the length. This allows for exact placement. The full protocol description can be found in the Wiki.
github.com/real-logic/Aeron/wiki/Protocol-Specification
Timely solution - keen to get involved and learn more.
Thinking of integrating this with golang channels as the API, should be fun to code in, and very performant.
steveoc64 By using the BufferClaim strategy and a codec like SBE then golang channels could be very efficient. Especially as I'm planning on adding IPC support in the next month or so.
have you compared performance efficiency of the Java implementation and C++ implementation?
its always nice to hear presentation from Martin
Curious what sort of benchmarks is this platform reaching ? its at version 0.9 now
At 29:00 you mention an algorithm inspired by gene sequencing. I couldn't quite make out what the name of that algorithm is. Could you post it here?
+Alex Leigh Diff
Is there any way to use shared memory/UDS instead of UDP for IPC?
Yes on aeron you can use shared mem for ipc
Will do. Thanks for the link.
Aeron is seriously good. We use it as part of Onyx and it really delivers on the claims.