pyjamas22_ generic composite in python.pdf

pyjamas22_ generic composite in python.pdf

• 1.
  © BST LABS2022 ALL RIGHTS RESERVED Generic Composite in Python Rationale behind the pycomposite Open Source Library Asher Sterkin SVP Engineering, GM Pyjamas Conf 2022
• 2.
  © BST LABS2022 ALL RIGHTS RESERVED ● A 40-year industry veteran specializing in software architecture and technology ● SVP Engineering @ BST LABS, BlackSwan Technologies ● Former VP Technology @ NDS and Distinguished Engineer @ Cisco ● Initiator and chief technology lead of the Cloud AI Operating System project ● Areas of interest: ○ Serverless Cloud Architecture ○ (Strategic) Domain-Driven Design ○ Promise Theory and Semantic SpaceTimes ○ Complex Adaptive Systems ● Contacts: ○ Linkedin: https://www.linkedin.com/in/asher-sterkin-10a1063 ○ Twitter: https://www.twitter.com/asterkin ○ Email: asher.sterkin@gmail.com ○ Slideshare: https://www.slideshare.net/AsherSterkin ○ Medium: https://asher-sterkin.medium.com/ Who am I?
• 3.
  © BST LABS2022 ALL RIGHTS RESERVED ● Design Patterns ● The “Composite” Design Pattern ● Need for a Generic One ● Design Patterns Density ● Implementation ● More Advanced Case Study ● Things to Remember Table of Contents
• 4.
  © BST LABS2022 ALL RIGHTS RESERVED The Wikipedia definition: “In software engineering, a software design pattern is a general, reusable solution to a commonly occurring problem within a given context in software design.” Quality Without a Name (QWAN): “To seek the timeless way we must first know the quality without a name. There is a central quality which is the root criterion of life and spirit in a man, a town, a building, or a wilderness. This quality is objective and precise, but it cannot be named.” Design Patterns A good collection of Design Patterns in Python: https://github.com/faif/python-patterns Fundamental, though a bit outdated, with examples in C++ and Java Origin of the idea of Design Patterns
• 5.
  © BST LABS2022 ALL RIGHTS RESERVED Whenever we have a tree-like data structure, be it a file system, cloud stack resources, code, or anything composed from smaller parts, the Composite Design Pattern is the first candidate to consider. “Composite” Design Pattern
• 6.
  © BST LABS2022 ALL RIGHTS RESERVED Composite Design Pattern (from Wikipedia) Traditional OO Implementation
• 7.
  © BST LABS2022 ALL RIGHTS RESERVED ● The Composite class has to overload all the abstract interface methods. ● The implementation of Composite methods would most likely be trivial: ○ Go over all child components and invoke the corresponding methods ○ Could be Depth-First or Breadth-First traversal. ○ In most of cases nobody cares which one. ● Outcome of the Composite operation is an aggregation of the child operations. ● Generally, not a big deal if the number of methods is small Limitations of the Traditional OO Implementation
• 8.
  © BST LABS2022 ALL RIGHTS RESERVED I F C services vendors APIs location acquire configure consume operate My Context: Cloud Infrastructure From Python Code service.py service_config.py
• 9.
  © BST LABS2022 ALL RIGHTS RESERVED build_service_parameters() build_service_template_variables() build_service_conditions() build_service_outputs() build_service_resources() build_function_permissions() build_function_resources() build_function_resource_properties() build_function_environment() Core Mechanism: Service Template Builder Composite ResourceTemplateBuilder CompositeResourceTemplateBuilder * ● Every high-level feature (e.g. MySQL Database) has multiple sub-features: ○ Data on rest encryption: yes/no ○ Private network protection: yes/no ○ Interface: (e.g. db_connection vs SQLAlchemy) ○ Allocation: internal/external ○ Access: read-only/read-write ○ User authentication: yes/no ● Each sub-feature might have multiple flavours (e.g. type of encryption) ● Every feature or sub-feature requires one or more cloud resources ● Every service function has to be properly configured to get an access to eachy feature resources ● Every service function implements a particular API, which might bring its own resource feature(s) Service Template Builder Composite API Template Builder Composite API Resource Template Builder Feature Template Builder Composite Feature Resource Template Builder
• 10.
  © BST LABS2022 ALL RIGHTS RESERVED ● The measurement of the amount of design that can be represented as instances of design patterns ● When applied correctly, higher design pattern density implies a higher maturity of the design. ● When applied incorrectly, leads to disastrous over-engineering. Design Pattern Density
• 11.
  © BST LABS2022 ALL RIGHTS RESERVED build_service_parameters() build_service_template_variables() build_service_conditions() build_service_outputs() build_service_resources() build_function_permissions() build_function_resources() build_function_resource_properties() build_function_environment() <<null object>> <<builder>> ResourceTemplateBuilder Putting All Patterns Together * <<composite>> <<iterator>> CompositeResourceTemplateBuilder <<decorator>> composite(cls) <<factory method>> get_builder(config) XyzResourceTemplateBuilder <<specification>> XyzResourceSpec * For each method, define a default implementation that returns an empty structure <<decorator>> ResourceTemplateBuilderDecorator
• 12.
  © BST LABS2022 ALL RIGHTS RESERVED Let’s Go to the Code @composite class CompositeResourceTemplateBuilder(ResourceTemplateBuilder): """ Implements a Composite Design Pattern to support combining multiple ResourceTemplateBuilders. By default, works as a PassThroughGate for underlying builder(s). """ pass class ResourceTemplateBuilderDecorator(ResourceTemplateBuilder): """ Base class for generic (e.g. Storage) ResourceTemplateBuilders selectively delegating specific tasks to platform-dependent builders. By default, works as a StopGate for the underlying builder. :param builder: reference to external platform-specific builder """ def __init__(self, builder: ResourceTemplateBuilder): self._builder = builder
• 13.
  © BST LABS2022 ALL RIGHTS RESERVED composite class decorator def composite(cls: type) -> type: """ Generic class decorator to create a Composite from the original class. Notes: 1. the constructor does not make copy, so do not pass generators, if you plan to invoke more than one operation. 2. it will return always flattened results of any operation. :param cls: original class :return: Composite version of original class """ attrs = { n: _make_method(n, f) for n, f in getmembers(cls, predicate=isfunction) if not n.startswith("_") } attrs["__init__"] = _constructor composite_cls = type(cls.__name__, cls.__bases__, attrs) composite_cls.__iter__ = _make_iterator(composite_cls) return composite_cls
• 14.
  © BST LABS2022 ALL RIGHTS RESERVED _constructor is simple def _constructor(self, *parts) -> None: self._parts = parts
• 15.
  © BST LABS2022 ALL RIGHTS RESERVED _make_method is where the real stuff is done def _make_method(name: str, func: callable) -> callable: > def _make_reduce(m: str, rt: type) -> callable: <-- Python Closure Gotcha > def _make_foreach(m) -> callable: <-- Python Closure Gotcha rt_ = signature(func).return_annotation rt = get_origin(rt_) or rt_ # strip type annotation parameters like tuple[int, ...] if present return _make_foreach(name) if rt is None else _make_reduce(name, rt) # Top-down decomposition in action
• 16.
  © BST LABS2022 ALL RIGHTS RESERVED _make_foreach is boring def _make_foreach(m) -> callable: def _foreach_parts(self, *args, **kwargs) -> callable: <-- Python Closure Gotcha # this is a member function, hence self # self is iterable, concrete functions invoked depth first for obj in self: getattr(obj, m)(*args, **kwargs) return _foreach_parts
• 17.
  © BST LABS2022 ALL RIGHTS RESERVED _make_reduce is more interesting def _make_method(name: str, func: callable) -> callable: def _make_reduce(m: str, rt: type) -> callable: <-- Python Closure Gotcha init_value = rt() combine = add if rt in (int, str, tuple) else always_merger.merge def _reduce_parts(self, *args, **kwargs): # this is a member function, hence self # self is iterable, results come out flattened return reduce( lambda acc, obj: combine(acc, getattr(obj, m)(*args, **kwargs)), self, init_value ) return _reduce_parts
• 18.
  © BST LABS2022 ALL RIGHTS RESERVED _make_iterator is the most tricky def _make_iterator(cls): def _iterator(self): # Simple depth-first composite Iterator # Recursive version did not work for some mysterious reason # This one proved to be more reliable # Credit: https://stackoverflow.com/questions/26145678/implementing-a-depth-first-tree-iterator-in-python stack = deque(self._parts) while stack: # Pop out the first element in the stack part = stack.popleft() if cls == type(part): # The same composite exactly stack.extendleft(reversed(part._parts)) elif isinstance(part, cls) or not isinstance(part, Iterable): yield part # derived classes presumably have overloads else: # Iterable stack.extendleft(reversed(part)) return _iterator
• 19.
  © BST LABS2022 ALL RIGHTS RESERVED <<builder>> K8SManifestBuilder __call__(...) <<builder>> <<template method>> K8SManifestPartBuilder build_manifest_part(...) _build_manifest_part_data(...) __subclasses__() More Advanced Use Case: K8S Manifest Builder * <<composite>> <<iterator>> K8SManifestPartCompositeBuilder <<decorator>> composite(cls) <<factory method>> get_builders() K8SXYZManifestBuilder Even with one function, this approach proved to lead to better modularity and testability
• 20.
  © BST LABS2022 ALL RIGHTS RESERVED I need ‘em all @composite class K8SManifestPartCompositeBuilder(K8SManifestPartBuilder): pass def get_builders(): for m in iter_modules(__path__): # ensure all builders are registered import_module(f'{__package__}.{m.name}') return K8SManifestPartCompositeBuilder( *(sb() for sb in K8SManifestPartBuilder.__subclasses__() if 'K8SManifestPartCompositeBuilder' not in sb.__name__) )
• 21.
  © BST LABS2022 ALL RIGHTS RESERVED ● In this solution I did not implement the Visitor Design Pattern. While it is not hard to implement, I just did not have any real use case for it. If you do, drop me a line ● The current set of aggregation functions is a bit limited reflecting what I needed at the moment. Support for a larger set might come in the future. ● This presentation used a more advanced version of code, currently at the separate branch Concluding Remarks
• 22.
  © BST LABS2022 ALL RIGHTS RESERVED ● Design Patterns are extremely powerful tools. ● Design Patterns work best in concert (high density). ● Composite Design Pattern is suitable for implementing the on/off logic a complex feature ● Also, for implementing a variable list of components you do not want to enumerate explicitly ● Python metaprogramming could make miracles. ● With more power comes more responsibility. ● An unjustified chase after advanced techniques is a sure road to hell. Things to Remember
• 23.
  © BST LABS2022 ALL RIGHTS RESERVED Questions?