Downloaded 11 times
![[jLove 2020] Where is my cache architectural patterns for caching microservi...](https://cdn.slidesharecdn.com/ss_thumbnails/whereismycachearchitecturalpatternsforcachingmicroservicesbyexample-201204105143-thumbnail.jpg?width=600ounds&width=560&fit=bounds)
![[DevopsDays India 2019] Where is my cache? Architectural patterns for caching...](https://cdn.slidesharecdn.com/ss_thumbnails/slidesharewhereismycachearchitecturalpatternsforcachingmicroservicesbyexample-190913082107-thumbnail.jpg?width=600ounds&width=560&fit=bounds)
Distributed Locking in Kubernetes
- 1. Distributed Locking inKubernetes Rafał Leszko @RafalLeszko rafalleszko.com Hazelcast
- 2. About me ● Cloud-NativeTeam Lead at Hazelcast ● Worked at Google and CERN ● Author of the book "Continuous Delivery with Docker and Jenkins" ● Trainer and conference speaker ● Live in Kraków, Poland
- 3. About Hazelcast ● DistributedCompany ● Open Source Software ● 140+ Employees ● Products: ○ Hazelcast IMDG ○ Hazelcast Jet ○ Hazelcast Cloud @Hazelcast
- 4. ● Introduction ● DistributedLocking ○ Single-Instance ○ Multi-Instance ○ RedLock ○ Consensus-Based ○ Kubernetes Native ○ Fenced ● Summary Agenda
- 5.
- 6. Reasons for DistributedLocking ● Efficiency ○ prevent executing the same work more than once, e.g., performing some expensive calculation ○ failing lock results in some additional costs or some inconvenience ● Correctness ○ prevent data corruption, data loss, inconsistency ○ failing lock results in some serious problems with the system
- 7.
- 8. Distributed Locking Shared Resource Application 2 Application1 Lock Manager lock()
- 9. Distributed Locking Shared Resource Application 2 Application1 Lock Manager lock() lock held by app 1
- 10. Distributed Locking Shared Resource Application 2 Application1 Lock Manager lock() lock held by app 1 write()
- 11. Distributed Locking Shared Resource Application 2 Application1 Lock Manager lock() lock held by app 1 write() unlock()
- 12. Distributed Locking Shared Resource Application 2 Application1 Lock Manager lock() lock held by app 1 write() unlock() lock()
- 13. Distributed Locking Shared Resource Application 2 Application1 Lock Manager lock() lock held by app 1 write() unlock() lock() lock held by app 2
- 14. Distributed Locking Shared Resource Application 2 Application1 Lock Manager lock() lock held by app 1 write() unlock() lock() lock held by app 2 write()
- 15. Distributed Locking Shared Resource Application 2 Application1 Lock Manager lock held by app 1 lock held by app 2 lock() unlock() write() write() lock() unlock()
- 16.
- 17. Distributed Locking Shared Resource Application 2 Application1 Lock Manager lock() lock held by app 1
- 18. Distributed Locking Shared Resource Application 2 Application1 Lock Manager lock() lock held by app 1
- 19. Distributed Locking Shared Resource Application 2 Application1 Lock Manager lock() lock held by app 1 lease expired
- 20. Distributed Locking Shared Resource Application 2 Application1 Lock Manager lock held by app 1 lock held by app 2 lock() write() lock() unlock() lease expired
- 21.
- 22. Application 1 Single-Instance DistributedLocking Application 2 lock() lock() Hazelcast
- 23. Application 1 Single-Instance DistributedLocking Application 2 lock() lock() Redis
- 24. Application 1 Single-Instance DistributedLocking Application 2 lock() lock() Hazelcast
- 25. Code: Single-Instance DistributedLocking var hazelcast = HazelcastClient. newHazelcastClient(); var lock = hazelcast.getCPSubsystem().getLock( "my-lock"); lock.lock(); writeToSharedResource(); lock.unlock();
- 26. Code: Single-Instance DistributedLocking var config = new ClientConfig(); config.getNetworkConfig().addAddress("hazelcast"); var hazelcast = HazelcastClient.newHazelcastClient(config); var lock = hazelcast.getCPSubsystem().getLock("my-lock"); lock.lock(); writeToSharedResource(); lock.unlock(); Server Application kubectl run hazelcast --image hazelcast/hazelcast --port 5701 --expose
- 27. Single-Instance Distributed Locking Shared Resource Application2 Application 1 Lock Manager lock held by app 1 lock held by app 2 lock() unlock() write() write() lock() unlock()
- 28. Single-Instance Distributed Locking Shared Resource Application2 Application 1 Lock Manager lock held by app 1 lock held by app 2 lock() unlock() write() write() lock() unlock()
- 29. Application 1 Single-Instance DistributedLocking Application 2 lock() lock() Hazelcast
- 30. Summary: Single-Instance DistributedLocking Efficiency Correctness Complexity Performance
- 31. ● Introduction ✔ ●Distributed Locking ○ Single-Instance ✔ ○ Multi-Instance ○ RedLock ○ Consensus-Based ○ Kubernetes Native ○ Fenced ● Summary Agenda
- 32.
- 33. Application 1 Multi-Instance DistributedLocking Application 2 lock() lock() Hazelcast Cluster
- 34. Application 1 Multi-Instance DistributedLocking Application 2 lock() lock() Hazelcast Cluster
- 35. Code: Multi-Instance DistributedLocking var config = new ClientConfig(); config.getNetworkConfig().addAddress("hazelcast"); var hazelcast = HazelcastClient.newHazelcastClient(config); var lock = hazelcast.getCPSubsystem().getLock("my-lock"); lock.lock(); writeToSharedResource(); lock.unlock(); Server Application helm repo add hazelcast https://hazelcast-charts.s3.amazonaws.com/ helm install hazelcast hazelcast/hazelcast
- 36. Application 1 Multi-Instance DistributedLocking Application 2 lock() lock() Hazelcast Cluster
- 37. Application 1 Multi-Instance DistributedLocking Application 2 lock() lock() Hazelcast Cluster(s)
- 38. Summary: Multi-Instance DistributedLocking Efficiency Correctness Complexity Performance
- 39. ● Introduction ✔ ●Distributed Locking ○ Single-Instance ✔ ○ Multi-Instance ✔ ○ RedLock ○ Consensus-Based ○ Kubernetes Native ○ Fenced ● Summary Agenda
- 40.
- 41.
- 42.
- 43.
- 44.
- 45. Code: RedLock var lock= new RedissonRedLock( client("redis-1").getLock("lock1"), client("redis-2").getLock("lock2"), client("redis-3").getLock("lock3") ); lock.lock(); writeToSharedResource(); lock.unlock(); Server Application kubectl run redis-1 --image redis:6.2.1 --port 6379 --expose kubectl run redis-2 --image redis:6.2.1 --port 6379 --expose kubectl run redis-3 --image redis:6.2.1 --port 6379 --expose
- 46.
- 47. RedLock: Issues ● Serverclocks need to be in sync ● Executing lock() and write() is not atomic
- 48. Distributed Locking Shared Resource Application 2 Application1 Lock Manager lock()
- 49. Distributed Locking Shared Resource Application 2 Application1 Lock Manager lock() lock held by app 1
- 50. Distributed Locking Shared Resource Application 2 Application1 Lock Manager lock() lock held by app 1 process blocked
- 51. Distributed Locking Shared Resource Application 2 Application1 Lock Manager lock() lock held by app 1 process blocked lock held by app 2 lock()
- 52. Distributed Locking Shared Resource Application 2 Application1 Lock Manager lock() lock held by app 1 process blocked lock held by app 2 lock() write()
- 53. Distributed Locking Shared Resource Application 2 Application1 Lock Manager lock held by app 1 lock held by app 2 write() write() lock() process blocked lock()
- 54.
- 55. ● Introduction ✔ ●Distributed Locking ○ Single-Instance ✔ ○ Multi-Instance ✔ ○ RedLock ✔ ○ Consensus-Based ○ Kubernetes Native ○ Fenced ● Summary Agenda
- 56.
- 57. Application 1 Consensus-Based DistributedLocking Application 2 lock() lock() Hazelcast Cluster (CP Subsystem Enabled)
- 58. Application 1 Consensus-Based DistributedLocking Application 2 lock() lock() Zookeeper Cluster
- 59. Application 1 Consensus-Based DistributedLocking Application 2 lock() lock() Hazelcast Cluster (CP Subsystem Enabled)
- 60. Code: Consensus-Based DistributedLocking var config = new ClientConfig(); config.getNetworkConfig().addAddress("hazelcast"); var hazelcast = HazelcastClient.newHazelcastClient(config); var lock = hazelcast.getCPSubsystem().getLock("my-lock"); lock.lock(); writeToSharedResource(); lock.unlock(); Server Application helm repo add hazelcast https://hazelcast-charts.s3.amazonaws.com/ helm install hazelcast hazelcast/hazelcast --set hazelcast.yaml.hazelcast.cp-subsystem.cp-member-count=3
- 61. Summary: Consensus-Based DistributedLocking Efficiency Correctness Complexity Performance
- 62. Application 1 Consensus-Based DistributedLocking Application 2 lock() lock() Hazelcast Cluster (CP Subsystem Enabled)
- 63. Application 1 Consensus-Based DistributedLocking Application 2 lock() lock() Zookeeper Cluster
- 64.
- 65. Application 1 Kubernetes NativeDistributed Locking Application 2 lock() lock() Kubernetes Cluster
- 66.
- 68. Libraries for KubernetesNative Distributed Locking ● Lockgate: cross-platform locking library with distributed locks using Kubernetes ● Kube-lock: simple library that implements a distributed lock using annotations on a Kubernetes resource
- 69. Summary: Kubernetes NativeDistributed Locking Efficiency Correctness Complexity Performance
- 70. ● Introduction ✔ ●Distributed Locking ○ Single-Instance ✔ ○ Multi-Instance ✔ ○ RedLock ✔ ○ Consensus-Based ✔ ○ Kubernetes Native ✔ ○ Fenced ● Summary Agenda
- 71.
- 72. Fenced Distributed Locking Shared Resource Application2 Application 1 Lock Manager lock held by app 1 lock held by app 2 lock() write() write() lock() process blocked
- 73. Application Fenced Distributed Locking token= lockAndGetToken() Hazelcast Cluster (CP Subsystem Enabled) Shared Resource token 1 2
- 74. Fenced Distributed Locking Shared Resource Application2 Application 1 Lock Manager
- 75. Fenced Distributed Locking Shared Resource Application2 Application 1 Lock Manager lock()
- 76. Fenced Distributed Locking Shared Resource Application2 Application 1 Lock Manager lock() lock held by app 1 token: 11
- 77. Fenced Distributed Locking Shared Resource Application2 Application 1 Lock Manager lock() lock held by app 1 token: 11 process blocked
- 78. Fenced Distributed Locking Shared Resource Application2 Application 1 Lock Manager lock() lock held by app 1 token: 11 process blocked lock()
- 79. Fenced Distributed Locking Shared Resource Application2 Application 1 Lock Manager lock() lock held by app 1 token: 11 process blocked lock() lock held by app 2 token: 12
- 80. Fenced Distributed Locking Shared Resource Application2 Application 1 Lock Manager lock() lock held by app 1 token: 11 process blocked lock() lock held by app 2 token: 12 write() token: 12
- 81. Fenced Distributed Locking Shared Resource Application2 Application 1 Lock Manager lock() lock held by app 1 token: 11 process blocked lock() lock held by app 2 token: 12 write() token: 12
- 82. Fenced Distributed Locking Shared Resource Application2 Application 1 Lock Manager lock() lock held by app 1 token: 11 process blocked lock() lock held by app 2 token: 12 write() token: 12 write() token: 11
- 83. Fenced Distributed Locking Shared Resource Application2 Application 1 Lock Manager lock held by app 1 lock held by app 2 lock() write() token: 11 write() token: 12 lock() process blocked token: 11 token: 12 rejected
- 84. Code: Fenced DistributedLocking var config = new ClientConfig(); config.getNetworkConfig().addAddress("hazelcast"); var hazelcast = HazelcastClient.newHazelcastClient(config); var lock = hazelcast.getCPSubsystem().getLock("my-lock"); long token = lock.lockAndGetFence(); writetoSharedResource(token); lock.unlock(); Server Application helm repo add hazelcast https://hazelcast-charts.s3.amazonaws.com/ helm install hazelcast hazelcast/hazelcast --set hazelcast.yaml.hazelcast.cp-subsystem.cp-member-count=3
- 85. Summary: Fenced DistributedLocking Efficiency Correctness Complexity Performance
- 86. ● Introduction ✔ ●Distributed Locking ○ Single-Instance ✔ ○ Multi-Instance ✔ ○ RedLock ✔ ○ Consensus-Based ✔ ○ Kubernetes Native ✔ ○ Fenced ✔ ● Summary Agenda
- 87.
- 88.
- 89. Resources ● Code forthis presentation: https://github.com/leszko/distributed-locking ● How to do distributed locking: https://martin.kleppmann.com/2016/02/08/how-to-do-distrib uted-locking.htm ● Distributed Locks are Dead; Long Live Distributed Locks! https://hazelcast.com/blog/long-live-distributed-locks/
- 90.