"Transducers" by Rich Hickey

People keep implementing map, filter and other fundamental algorithms in different contexts - eagerly over collections, over lazy sequences, in parallel, over enumerables/iterables, over observables, over channels/streams etc. In addition to duplication of effort, this yields bloated APIs, and, when implemented in the classic way, often involves the creation of expensive intermediate objects/values that can be difficult to optimize away.
Most problematic is that this approach keeps us from composing core algorithms in a context-independent way which would facilitate reuse and engender greater flexibility.
This talk will describe transducers, a new library feature for Clojure (but of interest to other languages) that emphasizes composable, context-free, intermediate-free notions like 'mapping' and 'filtering' and their concrete reuse across all of the contexts above.
by Rich Hickey (@richhickey) - Cognitect
Rich Hickey, the author of Clojure and designer of Datomic, is a software developer with over 25 years of experience in various domains. Rich has worked on scheduling systems, broadcast automation, audio analysis and fingerprinting, database design, yield management, exit poll systems, and machine listening, in a variety of languages. Rich is CTO of Cognitect, Inc.

[@FreshCodeSoftware]

The audio improves after ~30 seconds.

[@dialecticalmonist3405]

I know it's not his job, but I wish he could give educational talks on many different subjects, because I like the way he thinks and he explains things in such a fundamental and practical way.

[@kpmaynard]

Wow, just when you think it couldn't get any better!!!! Thank you Rich Hickey and Strange Loop for asking us "what are you guys doing?". I would never look at baggage handling the same way again :). This bag is ticking!!!

[@rgiar]

Lectures on Constructive Functional Programming by R.S. Bird -- www.cs.ox.ac.uk/files/3390/PRG69.pdf
A tutorial on the universality and expressiveness of fold by GRAHAM HUTTON --
www.cs.nott.ac.uk/~gmh/fold.pdf

[@WilkerLucio]

It's amazing to see those kind of simplifications, developing really complex and reliable services are getting so much easier because of those. Thank you very much for all people working on those great new stuff.

[@communty]

Oh no wait! There it is! Haha Hickey never disappoints.

[@campyazlo4896]

"If we have something that's about the steps, we can build things that are about the whole job-but not vice-versa."
What a truth.

[@magne6049]

First-principles thinking. 👌🏼

[@skyhester9196]

Rich's comment @19:05 regarding laziness and loop constructs - that foldr and foldl correspond to these major branches - is strangely parallel to the early proliferation of serial ideas and the slow incorporation of laziness in the whole CS field.

[@Kah0ona]

Great talk, like the real world analogy as well, that made it clearer for me.

[@KaliszAd]

Comp is like a treading macro but you are missing the collection/ sequence that is passed through the "thread" of functions. Some things, that are today written as a threading macro are easily converted into a transducer. That makes it run faster the bigger the collection/ sequence that is passed through, because you save on allocations. At least that is my experience. So transducers are in my view the nearest thing to a shell pipeline in the traditional UNIX sense as the input can be very large and it doesn't really matter with most commands (operations).

[@DenisG631]

Wonderful talk, even though at the beginning I've got a little bit confused.
Solved the problem by "layer-of-indirection" :)

[@davidleimbach2153]

You can kind of do this with the plain old Y combinator. If you write all your code in CPS ways you can always have a "step", but it's a bit more explicit. You can weave behaviors into algorithms like memoization and logging into a process with Y, and you can do the same with this. The advantage is you get to write and test them independently then compose them. This has the same advantages.
Great stuff, Rich is super smart and this is pretty danged neat.

[@kpmaynard]

Do you have a link to a talk that explains Y-combinators as clearly as Rich explained transducers? Exciting stuff!!!

[@StrangeLoopConf]

Keith Maynard Maybe you should try this video, also from Strange Loop, from the late great Jim Weirich: www.infoq.com/presentations/Y-Combinator

[@kpmaynard]

Strange Loop Thank you!!!

[@kpmaynard]

Strange Loop Great talk on y-combinator. Now for CPS.

[@ccash3290]

This is what I was looking for.
I really want to replace rxjs with simple functional programming

[@andyfingerhut]

A transcript of this talk is available here: github.com/matthiasn/talk-transcripts/blob/master/Hickey_Rich/Transducers-mostly-text.md

[@elcapitan6126]

python's bi-directional generators / coroutines seem to cover similar ground, although this Transducer formulation is built from plain old functions which I guess has some advantages (no state, easily implemented in most languages). I wonder what Haskellers think of them, are they something Haskellers have "for free" due to laziness?

[@elcapitan6126]

also, i think in general map and filter shouldn't necessarily be defined in terms of operating on sequences (mathematically they operate on domains/types/sets without regard to order). so transducers are generalizing a particular view of how map and filter "behave" in everyday programming languages.

[@evandrix]

Any link to your PDF slides? don't find it on github.com

[@davidcampbell6604]

I agree... programs in the future will not be written in English.

[@communty]

Awww no dictionary defintion to kick it off, bummer.

[@derekfrost8991]

I don't understand why he talks about the type system, maybe this is for non-clojure implementations?? I know typed clojure exists but Hickey doesn't like it usually.

[@kungfooman]

I'm not a LISP/Clojure person but:
(defn mapr [f coll]
(foldr (fn [x r] (cons (f x) r))
() coll))
Is this in JavaScript ES6 I think:
const mapr = (f, coll) => coll.reduceRight((acc, cur) => [f(cur), ...acc], []);
const arr = [1,2,3];
mapr(_ => _ * 2, arr); // [2, 4, 6]

[@HeavyMetalForces93]

This is basically what you get with generic programming techniques for automatic program derivation. You express how data are aggregated together by specifying the base functor of the datatype, and all operations like fold are there for free by initiality. We have been knowing both initial algebra semantics and generic programming for decades, so not really breakthrough material here...

[@ValentinBaca]

His beginning slide of the burrito is literally is about how this is nothing new. "I'm not claiming any novelty here"