Austin Bingham. Transducers in Python. PyCon Belarus

Austin Bingham. Transducers in Python. PyCon Belarus

• 1.
  @sixty_north Understanding Transducers Through Python 1 AustinBingham @austin_bingham Sunday, January 25, 15
• 2.
  237 “Transducers are coming” RichHickeyhttp://blog.cognitect.com/blog/2014/8/6/transducers-are-coming Photo: Howard Lewis Ship under CC-BY Sunday, January 25, 15
• 3.
  Functions which transformreducers What is a transducer? ‣ Reducer (reducing function) • Any function we can pass to reduce(reducer, iterable[, initial]) • (result, input) → result • add(result, input) reduce(add, [1, 2, 3], 0) → 6 ‣ Transducer (transform reducer) • A function which accepts a reducer, and transforms it in some way, and returns a new reducer • ((result, input) → result) → ((result, input) → result) 3 Sunday, January 25, 15
• 4.
  Transducers are afunctional programming technique not restricted to Clojure How does this relate to Python? ‣ Clojure implementation is the prototype/archetype • Heavily uses anonymous functions • Uses overloads on function arity for disparate purposes • Complected with existing approaches ‣ Python implementation is pedagogical (but also useful) • Explicit is better than implicit • Readability counts! • Has all the functional tools we need for transducers • I’m a better Pythonista than Clojurist 4 Sunday, January 25, 15
• 5.
  5 Review functional programming tools in Sunday,January 25, 15
• 6.
  6 my_filter(is_prime, my_map(prime_factors, range(32)) ) iterable sequence sequence Sunday, January 25, 15
• 7.
  7 def my_map(transform, iterable): defmap_reducer(sequence, item): sequence.append(transform(item)) return sequence return reduce(map_reducer, iterable, []) def my_filter(predicate, iterable): def filter_reducer(sequence, item): if predicate(item): sequence.append(item) return sequence return reduce(filter_reducer, iterable, []) Sunday, January 25, 15
• 8.
  7 def my_map(transform, iterable): defmap_reducer(sequence, item): sequence.append(transform(item)) return sequence return reduce(map_reducer, iterable, []) def my_filter(predicate, iterable): def filter_reducer(sequence, item): if predicate(item): sequence.append(item) return sequence return reduce(filter_reducer, iterable, []) reduce() Sunday, January 25, 15
• 9.
  7 def my_map(transform, iterable): defmap_reducer(sequence, item): sequence.append(transform(item)) return sequence return reduce(map_reducer, iterable, []) def my_filter(predicate, iterable): def filter_reducer(sequence, item): if predicate(item): sequence.append(item) return sequence return reduce(filter_reducer, iterable, []) reduce() sequence.append() Sunday, January 25, 15
• 10.
  7 def my_map(transform, iterable): defmap_reducer(sequence, item): sequence.append(transform(item)) return sequence return reduce(map_reducer, iterable, []) def my_filter(predicate, iterable): def filter_reducer(sequence, item): if predicate(item): sequence.append(item) return sequence return reduce(filter_reducer, iterable, []) reduce() sequence.append() Empty list : ‘seed’ must be a mutable sequence Sunday, January 25, 15
• 11.
  8 >>> reduce(make_mapper(square), range(10),[]) [0, 1, 4, 9, 16, 25, 36, 49, 64, 81] >>> reduce(make_filterer(is_prime), range(100), []) [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97] ‣ make_mapper() and make_filterer() are not composable ‣ to square primes we would need to call reduce() twice ‣ this requires we store the intermediate sequence ‣ the reducers returned by make_mapper() and make_filterer() depend on the seed being a mutable sequence e.g. a list Sunday, January 25, 15
• 12.
  9 (defn map ([f] (fn [rf] (fn ([](rf)) ([result] (rf result)) ([result input] (rf result (f input))) ([result input & inputs] (rf result (apply f input inputs)))))) Sunday, January 25, 15
• 13.
  10 (defn map ;;The tranducer factory... ([f] ;; ...accepts a single argument 'f', the transforming function (fn [rf] ;; The transducer function accepts a reducing function 'rf' (fn ;; This is the reducing function returned by the transducer ([] (rf)) ;; 0-arity : Forward to the zero-arity reducing function 'rf' ([result] (rf result)) ;; 1-arity : Forward to the one-arity reducing function 'rf' ([result input] ;; 2-arity : Perform the reduction with one arg to 'f' (rf result (f input))) ([result input & inputs] ;; n-arity : Perform the reduction with multiple args to 'f' (rf result (apply f input inputs)))))) Sunday, January 25, 15
• 14.
  11 (defn map ;;The tranducer factory... ([f] ;; ...accepts a single argument 'f', the transforming function (fn [rf] ;; The transducer function accepts a reducing function 'rf' (fn ;; This is the reducing function returned by the transducer ([] (rf)) ;; 0-arity : Return a 'seed' value obtained from 'rf' ([result] (rf result)) ;; 1-arity : Obtain final result from 'rf' and clean-up ([result input] ;; 2-arity : Perform the reduction with one arg to 'f' (rf result (f input))) ([result input & inputs] ;; n-arity : Perform the reduction with multiple args to 'f' (rf result (apply f input inputs)))))) Sunday, January 25, 15
• 15.
  12 (defn map ;;The tranducer factory... ([f] ;; ...accepts a single argument 'f', the transforming function (fn [rf] ;; The transducer function accepts a reducing function 'rf' (fn ;; This is the reducing function returned by the transducer ([] (rf)) ;; 0-arity : Return a 'seed' value obtained from 'rf' ([result] (rf result)) ;; 1-arity : Obtain final result from 'rf' and clean-up ([result input] ;; 2-arity : Perform the reduction with one arg to 'f' (rf result (f input))) ([result input & inputs] ;; n-arity : Perform the reduction with multiple args to 'f' (rf result (apply f input inputs)))))) To fully implement Clojure’s transducers in Python we also need: ‣ explicit association of the seed value with the reduction operation ‣ support for early termination without reducing the whole series ‣ reduction to a final value and opportunity to clean up state Sunday, January 25, 15
• 16.
  13 class Reducer: def __init__(self,reducer): # Construct from reducing function pass def initial(self): # Return the initial seed value pass # 0-arity def step(self, result, item): # Next step in the reduction pass # 2-arity def complete(self, result): # Produce a final result and clean up pass # 1-arity Sunday, January 25, 15
• 17.
  13 class Reducer: def __init__(self,reducer): # Construct from reducing function pass def initial(self): # Return the initial seed value pass # 0-arity def step(self, result, item): # Next step in the reduction pass # 2-arity def complete(self, result): # Produce a final result and clean up pass # 1-arity new_reducer = Reducer(reducer) Sunday, January 25, 15
• 18.
  13 class Reducer: def __init__(self,reducer): # Construct from reducing function pass def initial(self): # Return the initial seed value pass # 0-arity def step(self, result, item): # Next step in the reduction pass # 2-arity def complete(self, result): # Produce a final result and clean up pass # 1-arity new_reducer = Reducer(reducer) def transducer(reducer): return Reducer(reducer) Sunday, January 25, 15
• 19.
  13 class Reducer: def __init__(self,reducer): # Construct from reducing function pass def initial(self): # Return the initial seed value pass # 0-arity def step(self, result, item): # Next step in the reduction pass # 2-arity def complete(self, result): # Produce a final result and clean up pass # 1-arity new_reducer = Reducer(reducer) def transducer(reducer): return Reducer(reducer) ⇐ two names for two concepts Sunday, January 25, 15
• 20.
  Python Transducer implementations 14 Cognitect https://github.com/cognitect-labs PyPI:transducers • “official” • Python in the style of Clojure • Only eager iterables (step back from regular Python) • Undesirable Python practices such as hiding built-ins • Also Clojure, Javascript, Ruby, etc. Sixty North http://code.sixty-north.com/python- transducers PyPI: transducer • Pythonic • Eager iterables • Lazy iterables • Co-routine based ‘push’ events • Pull-requests welcome! Sunday, January 25, 15
• 21.
  15 Thank you! @sixty_north Austin Bingham @austin_bingham http://sixty-north.com/blog/ Sunday,January 25, 15