KEMBAR78
Hijacking Ruby Syntax in Ruby (RubyConf 2018) | PDF
HIJACKING RUBY SYNTAX
IN RUBY
RubyConf 2018 LA
2018/11/15 (Thu)
@joker1007 & @tagomoris
Self-Intro Joker
▸ id: joker1007
▸ Repro inc. CTO
▸ I’m familiar with Ruby/Rails/Fluentd/ECS/Presto.
▸ This talk is my first speech abroad!!
Satoshi Tagomori (@tagomoris)
Fluentd, MessagePack-Ruby, Norikra, Woothee, ...
Arm Ltd.
Agenda
▸ Ruby Features
▸ binding
▸ Tracepoint
▸ method hook
▸ Hacks
▸ finalist, overrider, abstriker
▸ with_resources, deferral
Ruby Features: Binding
Binding
▸ Context object, includes:
▸ Kernel#binding receiver
▸ local variables
▸ For template engines (?)
▸ Methods:
▸ #receiver, #eval,

#local_variables,

#local_variable_get,

#local_variable_defined?,

#local_variable_set
Binding creates new object per call
Setting variables are ignored on binding
Overwriting existing variable is effective!
Ruby Features: Binding
Binding#local_variable_set
▸ Method to
▸ add a variable only in a binding instance
▸ overwrite values of existing variables in original
context
Ruby Features: TracePoint
TracePoint
▸ Tracing events in VM
▸ and call hook block
▸ For various events:
▸ :line , :raise
▸ :class , :end
▸ :call , :return , :c_call , :c_return , :b_call , :b_return
▸ :thread_begin , :thread_end , :fiber_switch
▸ "We can use TracePoint to gather information specifically for
exceptions:" (from Ruby doc)
▸ This is COMPLETELY WRONG statement...
Ruby Features: TracePoint
TracePoint interrupted the argument (stringified) and return value (upcased)
Ruby Features: TracePoint
TracePoint Methods
▸ Methods:
▸ Control: disable, enable, enabled?
▸ Event what/where: event, defined_class, path, lineno
▸ Method names: method_id, callee_id
▸ Event special: raised_exception, return_value
▸ And: binding
▸ So what?
▸ We can use TracePoint 
▸ to gather information
▸ to overwrite everything
テキスト
Break (10min)
Ruby Features: Refinements
▸ Refinements provide a way to extend a class locally
▸ Useful use case. (Safety monkey patching)
Ruby Features: Refinements
Another use case
▸ Super private method
Ruby Features: Refinements
Ruby Features: Method Hooks
▸ Ruby has six method hooks
▸ Module#method_added
▸ Module#method_removed
▸ Module#method_undefined
▸ BasicObject#singleton_method_added
▸ BasicObject#singleton_method_removed
▸ BasicObject#singleton_method_undefined
Ruby Features: method hooks
Sample: #method_added
Ruby Features: method hooks
Method hook provides a way to implement method modifier.
Hacks: Method modifiers
▸ final (https://github.com/joker1007/finalist)
▸ forbid method override
▸ override (https://github.com/joker1007/overrider)
▸ enforce method has super method
▸ abstract (https://github.com/joker1007/abstriker)
▸ enforce method override
These method modifiers work when Ruby defines class.
It is runtime, but in most case, before main logic.
Hacks: method modifiers
Sample code: finalist
Hacks: method modifiers
Hacks: method modifiers
Sample code: abstriker
Hacks: method modifiers
How to implement method modifiers
I use so many hook methods.
#included, #extended, #method_added, and TracePoint.
And I use Ripper.
In other words, I use the power of many black magics !!
Hacks: method modifiers
Use case of method_added in finalist
Ruby Features: method hooks
MyObject
Finalist
def method_added
def verify_final_method
▸ #method_added checks violations
call
Limitation of Method Hooks
▸ But it is not enough to implement “finalist” actually
▸ Ruby has so many cases of method definition
▸ def or #define_method
▸ #include module
▸ #extend, #prepend
▸ Each case has dedicated hooks
Ruby Features: method hooks
#include changes only chain of method discovering
Foo
Object
Bar
Insert module to hierarchy
It is different from method adding
Class#ancestors displays class-module hierarchy
Ruby Features: method hooks
Hooks in finalist gem
hook method override event by
method_added subclass
singleton_method_added subclass class methods
included included modules
extended extended moduled
And I talk about TracePoint too
overrider and abstriker use TracePoint
▸ #inherited and #included to start TracePoint
Tracepoint in overrider, abstriker
MyObject
Overrider/Abstriker
def included(or inherited)
TracePoint.trace(:end, :c_return, :raise)
call
Tracepoint in overrider, abstriker
TracePoint Hook
Overrider
self.override_methods = [
:method_a,
:method_b,
]
▸ Use Method#super_method method to check method
existence (ex. method(:method_a).super_method)
Why use TracePoint?
▸ In order to verify method existence at the end of class
definition.
▸ Ruby interpreter needs to wait until the end of class
definition to know a method absence.
▸ override and abstract cannot detect violation just when
they are called.
▸ In ruby, The only way to detect the violation is
TracePoint.
Advanced TracePoint: Detect particular class end
Advanced Tracepoint
:end event cannot trace definition by `Class.new`.
Use :c_return and return_value

to detect particular class end
Advanced TracePoint: Ripper combination
Advanced Tracepoint
▸ Ripper:
▸ a standard library, a parser for Ruby code
▸ outputs token list and S-expression
▸ S-expression is similar to AST
▸ has token string and token position
▸ Future option: RubyVM::AbstractSyntaxTree (2.6.0 or later)
Ripper sample:
Advanced Tracepoint
Advanced TracePoint: Ripper combination
Advanced Tracepoint
Detect target Sexp node by TracePoint#lineno
Sexp node type expresses the style of method cal
Ripper empowers TracePoint
▸ Conclusion:
▸ TracePoint detects events and where it occurs
▸ Ripper.sexp provides how methods were called
▸ Other use case
▸ power_assert gem also uses this combination
Advanced Tracepoint
Black Magic is dangerous actually,
but it is very fun,
and it extends Ruby potential
These gems are proof of concepts,
But these are decent practical.
Ruby Quiz
Break (25-30min)
What the difference between:
- #undef_method
- #remove_method
RUBY QUIZ
class Foo
def foo
class Foo
def foo
class Bar < Foo
def foo
class Bar < Foo
def foo
Bar.new.foo()
RUBY QUIZ
class Foo
def foo
class Foo
def foo
class Bar < Foo
class Bar < Foo
Bar.new.foo()
remove_method(:foo)
def foo
NoMethodError
undef_method(:foo)
Hack: with_resources
Add "with" Statement to Ruby
▸ Safe resource allocate/release
▸ Ensure to release resources
▸ at the end of a lexical scope
▸ in reverse order of allocation
▸ Idioms used very frequently
▸ Other languages:
▸ Java:

try-with-resources
▸ Python: with
▸ C#: using
Hack: with_resources
Safe Resource Allocation/Release Statement in Ruby
▸ Open method with blocks
▸ File.open(path){|f| ... }
▸ Ruby way (?)
▸ More indentation
▸ Not implemented sometimes

(e.g., TCPSocket)
Hack: with_resources
with_resources.gem
▸ Safe resource allocation
▸ Top level #with
▸ by Kernel refinement
▸ Resource allocation as lambda
▸ Multi statements to allocate resources
▸ to release first resource

if second resource allocation raises exception
▸ Block to define scope for resources
https://github.com/tagomoris/with_resources
Hack: with_resources
Implementing #with in Ruby
▸ TracePoint
▸ "b_return": pass allocated resources to block arguments
▸ "line": identify allocated resources in lambda
▸ Binding
▸ detect newly defined

local variables

in allocation lambda
▸ Refinements
▸ introduce #with

in top-level without side effects
https://github.com/tagomoris/with_resources
Hack: deferral
Alternative: defer?
▸ Multi-step resource

allocation in a method
▸ Nesting! w/ #with
▸ not so bad
▸ many nesting looks

a bit messy :(
▸ Alternative?
▸ defer in golang
Hack: deferral
Adding #defer to Ruby
▸ Block based #defer
▸ Should work
▸ Requires 1-level

nesting *always*
▸ Defer.start, end (+ nesting)

look too much (than golang)
▸ Abnormal case:

reassigning variables
Hack: deferral
deferral.gem
▸ Safe resource release
▸ Top level #defer
▸ by Kernel refinements
▸ Deferred processing to release resources
▸ at the end of scope (method, block)
▸ or exception raised
Hack: deferral
Implementing "defer" in Ruby
▸ #defer
▸ Enable TracePoint if not yet
▸ Initialize internal stack frame
▸ TracePoint
▸ Monitor method call stack
▸ Get the snapshot of local variables in defer block
▸ Call release blocks at the end of scope
▸ Binding
▸ save/restore local variables of release block
▸ Refinements
▸ introduce #defer in top-level without side effects
stack level 0
stack level 1
The Hard Thing
in Magical World:
Debugging!
"Give yourself to the dark side.
It is the only way you can save
your (Ruby)friends." - Darth vader
Thank You!
@joker1007 & @tagomoris

Hijacking Ruby Syntax in Ruby (RubyConf 2018)

  • 1.
    HIJACKING RUBY SYNTAX INRUBY RubyConf 2018 LA 2018/11/15 (Thu) @joker1007 & @tagomoris
  • 2.
    Self-Intro Joker ▸ id:joker1007 ▸ Repro inc. CTO ▸ I’m familiar with Ruby/Rails/Fluentd/ECS/Presto. ▸ This talk is my first speech abroad!!
  • 3.
    Satoshi Tagomori (@tagomoris) Fluentd,MessagePack-Ruby, Norikra, Woothee, ... Arm Ltd.
  • 5.
    Agenda ▸ Ruby Features ▸binding ▸ Tracepoint ▸ method hook ▸ Hacks ▸ finalist, overrider, abstriker ▸ with_resources, deferral
  • 6.
    Ruby Features: Binding Binding ▸Context object, includes: ▸ Kernel#binding receiver ▸ local variables ▸ For template engines (?) ▸ Methods: ▸ #receiver, #eval,
 #local_variables,
 #local_variable_get,
 #local_variable_defined?,
 #local_variable_set
  • 7.
    Binding creates newobject per call
  • 8.
    Setting variables areignored on binding
  • 9.
  • 10.
    Ruby Features: Binding Binding#local_variable_set ▸Method to ▸ add a variable only in a binding instance ▸ overwrite values of existing variables in original context
  • 11.
    Ruby Features: TracePoint TracePoint ▸Tracing events in VM ▸ and call hook block ▸ For various events: ▸ :line , :raise ▸ :class , :end ▸ :call , :return , :c_call , :c_return , :b_call , :b_return ▸ :thread_begin , :thread_end , :fiber_switch ▸ "We can use TracePoint to gather information specifically for exceptions:" (from Ruby doc) ▸ This is COMPLETELY WRONG statement...
  • 12.
  • 13.
    TracePoint interrupted theargument (stringified) and return value (upcased)
  • 15.
    Ruby Features: TracePoint TracePointMethods ▸ Methods: ▸ Control: disable, enable, enabled? ▸ Event what/where: event, defined_class, path, lineno ▸ Method names: method_id, callee_id ▸ Event special: raised_exception, return_value ▸ And: binding ▸ So what? ▸ We can use TracePoint  ▸ to gather information ▸ to overwrite everything
  • 16.
  • 17.
    Ruby Features: Refinements ▸Refinements provide a way to extend a class locally ▸ Useful use case. (Safety monkey patching) Ruby Features: Refinements
  • 18.
    Another use case ▸Super private method Ruby Features: Refinements
  • 19.
    Ruby Features: MethodHooks ▸ Ruby has six method hooks ▸ Module#method_added ▸ Module#method_removed ▸ Module#method_undefined ▸ BasicObject#singleton_method_added ▸ BasicObject#singleton_method_removed ▸ BasicObject#singleton_method_undefined Ruby Features: method hooks
  • 20.
  • 21.
    Method hook providesa way to implement method modifier.
  • 22.
    Hacks: Method modifiers ▸final (https://github.com/joker1007/finalist) ▸ forbid method override ▸ override (https://github.com/joker1007/overrider) ▸ enforce method has super method ▸ abstract (https://github.com/joker1007/abstriker) ▸ enforce method override These method modifiers work when Ruby defines class. It is runtime, but in most case, before main logic. Hacks: method modifiers
  • 23.
  • 24.
  • 25.
  • 26.
    How to implementmethod modifiers I use so many hook methods. #included, #extended, #method_added, and TracePoint. And I use Ripper. In other words, I use the power of many black magics !! Hacks: method modifiers
  • 27.
    Use case ofmethod_added in finalist Ruby Features: method hooks MyObject Finalist def method_added def verify_final_method ▸ #method_added checks violations call
  • 28.
    Limitation of MethodHooks ▸ But it is not enough to implement “finalist” actually ▸ Ruby has so many cases of method definition ▸ def or #define_method ▸ #include module ▸ #extend, #prepend ▸ Each case has dedicated hooks Ruby Features: method hooks
  • 29.
    #include changes onlychain of method discovering Foo Object Bar Insert module to hierarchy It is different from method adding Class#ancestors displays class-module hierarchy Ruby Features: method hooks
  • 30.
    Hooks in finalistgem hook method override event by method_added subclass singleton_method_added subclass class methods included included modules extended extended moduled
  • 31.
    And I talkabout TracePoint too
  • 32.
    overrider and abstrikeruse TracePoint ▸ #inherited and #included to start TracePoint Tracepoint in overrider, abstriker MyObject Overrider/Abstriker def included(or inherited) TracePoint.trace(:end, :c_return, :raise) call
  • 33.
    Tracepoint in overrider,abstriker TracePoint Hook Overrider self.override_methods = [ :method_a, :method_b, ] ▸ Use Method#super_method method to check method existence (ex. method(:method_a).super_method)
  • 34.
    Why use TracePoint? ▸In order to verify method existence at the end of class definition. ▸ Ruby interpreter needs to wait until the end of class definition to know a method absence. ▸ override and abstract cannot detect violation just when they are called. ▸ In ruby, The only way to detect the violation is TracePoint.
  • 35.
    Advanced TracePoint: Detectparticular class end Advanced Tracepoint :end event cannot trace definition by `Class.new`. Use :c_return and return_value
 to detect particular class end
  • 36.
    Advanced TracePoint: Rippercombination Advanced Tracepoint ▸ Ripper: ▸ a standard library, a parser for Ruby code ▸ outputs token list and S-expression ▸ S-expression is similar to AST ▸ has token string and token position ▸ Future option: RubyVM::AbstractSyntaxTree (2.6.0 or later)
  • 37.
  • 38.
    Advanced TracePoint: Rippercombination Advanced Tracepoint Detect target Sexp node by TracePoint#lineno Sexp node type expresses the style of method cal
  • 39.
    Ripper empowers TracePoint ▸Conclusion: ▸ TracePoint detects events and where it occurs ▸ Ripper.sexp provides how methods were called ▸ Other use case ▸ power_assert gem also uses this combination Advanced Tracepoint
  • 40.
    Black Magic isdangerous actually, but it is very fun, and it extends Ruby potential These gems are proof of concepts, But these are decent practical.
  • 41.
    Ruby Quiz Break (25-30min) Whatthe difference between: - #undef_method - #remove_method
  • 42.
    RUBY QUIZ class Foo deffoo class Foo def foo class Bar < Foo def foo class Bar < Foo def foo Bar.new.foo()
  • 43.
    RUBY QUIZ class Foo deffoo class Foo def foo class Bar < Foo class Bar < Foo Bar.new.foo() remove_method(:foo) def foo NoMethodError undef_method(:foo)
  • 44.
    Hack: with_resources Add "with"Statement to Ruby ▸ Safe resource allocate/release ▸ Ensure to release resources ▸ at the end of a lexical scope ▸ in reverse order of allocation ▸ Idioms used very frequently ▸ Other languages: ▸ Java:
 try-with-resources ▸ Python: with ▸ C#: using
  • 45.
    Hack: with_resources Safe ResourceAllocation/Release Statement in Ruby ▸ Open method with blocks ▸ File.open(path){|f| ... } ▸ Ruby way (?) ▸ More indentation ▸ Not implemented sometimes
 (e.g., TCPSocket)
  • 46.
    Hack: with_resources with_resources.gem ▸ Saferesource allocation ▸ Top level #with ▸ by Kernel refinement ▸ Resource allocation as lambda ▸ Multi statements to allocate resources ▸ to release first resource
 if second resource allocation raises exception ▸ Block to define scope for resources https://github.com/tagomoris/with_resources
  • 47.
    Hack: with_resources Implementing #within Ruby ▸ TracePoint ▸ "b_return": pass allocated resources to block arguments ▸ "line": identify allocated resources in lambda ▸ Binding ▸ detect newly defined
 local variables
 in allocation lambda ▸ Refinements ▸ introduce #with
 in top-level without side effects https://github.com/tagomoris/with_resources
  • 48.
    Hack: deferral Alternative: defer? ▸Multi-step resource
 allocation in a method ▸ Nesting! w/ #with ▸ not so bad ▸ many nesting looks
 a bit messy :( ▸ Alternative? ▸ defer in golang
  • 49.
    Hack: deferral Adding #deferto Ruby ▸ Block based #defer ▸ Should work ▸ Requires 1-level
 nesting *always* ▸ Defer.start, end (+ nesting)
 look too much (than golang) ▸ Abnormal case:
 reassigning variables
  • 50.
    Hack: deferral deferral.gem ▸ Saferesource release ▸ Top level #defer ▸ by Kernel refinements ▸ Deferred processing to release resources ▸ at the end of scope (method, block) ▸ or exception raised
  • 51.
    Hack: deferral Implementing "defer"in Ruby ▸ #defer ▸ Enable TracePoint if not yet ▸ Initialize internal stack frame ▸ TracePoint ▸ Monitor method call stack ▸ Get the snapshot of local variables in defer block ▸ Call release blocks at the end of scope ▸ Binding ▸ save/restore local variables of release block ▸ Refinements ▸ introduce #defer in top-level without side effects stack level 0 stack level 1
  • 52.
    The Hard Thing inMagical World: Debugging!
  • 53.
    "Give yourself tothe dark side. It is the only way you can save your (Ruby)friends." - Darth vader Thank You! @joker1007 & @tagomoris