Download as PDF, PPTX
Docker, Monitoring and SLURM Specific Visualisations
- 1. ss Docker, Monitoring andSLURM Specific Visualisations QNIBTerminal @ work
- 2. ā¢ Docker ina Nutshell ā¢ QNIBx Terminal Monitoring Inventory ā¢ SLURM Autogenerated Dashboards 2 Agenda
- 3. 3 About Me ā¢ ChristianKniep @CQnib, christian@qnib.org
- 4. 4 About Me ā¢ ChristianKniep @CQnib, christian@qnib.org ā¢ >10y Iteration SysAdmin, SysOps, SysEngineer, R&D Engineer DevOps @Locafox (hyper-scale web-service)
- 5. 5 About Me ā¢ ChristianKniep @CQnib, christian@qnib.org ā¢ >10y Iteration SysAdmin, SysOps, SysEngineer, R&D Engineer DevOps @Locafox (hyper-scale web-service) ā¢ Founder of QNIB Solutions Holistic System Management Containerization of SysOps and Workload Consultancy / Software Design & Development
- 6. Docker in aNutshell
- 7. 7 Multiple Guests SERVER SERVER TraditionalVirtualisation Containerisation
- 8. 8 Multiple Guests SERVER HOST Ā KERNEL SERVER HOST Ā KERNEL Traditional Virtualisation Containerisation
- 9. 9 Multiple Guests SERVER HOST Ā KERNEL Userland SERVER HOST Ā KERNEL Userland Traditional Virtualisation Containerisation
- 10. 10 Multiple Guests SERVER HOST Ā KERNEL HYPERVISOR Ā (Type Ā II) Userland SERVER HOST Ā KERNEL Userland Traditional Virtualisation Containerisation
- 11. 11 Multiple Guests SERVER HOST Ā KERNEL HYPERVISOR Ā (Type Ā II) KERNEL Userland KERNEL KERNEL SERVER HOST Ā KERNEL Userland Traditional Virtualisation Containerisation
- 12. 12 Multiple Guests SERVER HOST Ā KERNEL HYPERVISOR Ā (Type Ā II) KERNEL Userland KERNEL KERNEL Userland Userland Userland SERVER HOST Ā KERNEL Userland Traditional Virtualisation Containerisation
- 13. 13 Multiple Guests SERVER HOST Ā KERNEL HYPERVISOR Ā (Type Ā II) KERNEL SERVICE Userland KERNEL KERNEL Userland Userland Userland SERVICE SERVICE SERVER HOST Ā KERNEL Userland Traditional Virtualisation Containerisation
- 14. 14 Multiple Guests SERVER HOST Ā KERNEL HYPERVISOR Ā (Type Ā II) KERNEL SERVICE Userland KERNEL KERNEL Userland Userland Userland SERVICE SERVICE SERVER HOST Ā KERNEL Userland Traditional Virtualisation Containerisation Docker
- 15. 15 Multiple Guests SERVER HOST Ā KERNEL HYPERVISOR Ā (Type Ā II) KERNEL SERVICE Userland KERNEL KERNEL Userland Userland Userland SERVICE SERVICE SERVER HOST Ā KERNEL Userland Userland Ā (#1) Userland Ā (#2) Traditional Virtualisation Containerisation Docker
- 16. 16 Multiple Guests SERVER HOST Ā KERNEL HYPERVISOR Ā (Type Ā II) KERNEL SERVICE Userland KERNEL KERNEL Userland Userland Userland SERVICE SERVICE SERVER HOST Ā KERNEL Userland Userland Ā (#1) Userland Ā (#2) SERVICE SERVICE Traditional Virtualisation Containerisation Docker
- 17. HOST container1 17 Docker Internal View ā¢Containers are āgrouped processesā isolated by Kernel Namespaces (PID, network, mount, ā¦) resource restrictions applicable through CGroups bash ls -l container2 apache container3 mysqld container4 slurmd ssh
- 18. ā¢ 1/2 Day,July 16th @ISC High Performance Deep dive into the talking points How Docker might impact System Operations & HPC Applications Further discussion beyond what I am talking about today 18 Docker Workshop
- 19. ā¢ Full Day,September 28th @ISC Cloud&BigData 19 Docker Workshop #2
- 20.
- 21. ā¢ Framework ofsystem container to spin up stacks SLURM 21 QNIBTerminal
- 22. ā¢ Framework ofsystem container to spin up stacks SLURM 22 QNIBTerminal
- 23. ā¢ Framework ofsystem container to spin up stacks SLURM 23 QNIBTerminal
- 24. ā¢ Framework ofsystem container to spin up stacks SLURM 24 QNIBTerminal 1 2 3
- 25.
- 26. ā¢ Current monitoringsystems do not connect overlaying metrics with log events use/build inventory system to provide connections usually hidden users perspective and scope/context/background 26 QNIBMonitoring
- 27. ā¢ Current monitoringsystems do not connect overlaying metrics with log events use/build inventory system to provide connections usually hidden users perspective and scope/context/background ā¢ QNIBMonitoring provides open metrics system (system / application metrics, log aggregates) log event framework, consuming/processing/visualise events auto discovery / configuration through consul 27 QNIBMonitoring
- 28. 28 QNIBMonitoring ā¢ Logstash (Log/EventMonitoring)
- 29. 29 QNIBMonitoring ā¢ Grafana (PerformanceMonitoring)
- 30.
- 31.
- 32.
- 33.
- 34.
- 35.
- 36. ā¢ Enrich Log/Events ā¢Help visualise connections 36 QNIBInventory
- 37. ā¢ Enrich Log/Events ā¢Help visualise connections ā¢ Build up history 37 QNIBInventory
- 38.
- 39. ā¢ Multiple backgroundshave to be considered Enduser (Engineer, Software Developer, Scientist) Operation Personel Management ā¢ Psychology plays important role Local rationality / context 10.000ft Overview vs. verifying hypothesis vs. Reporting Empower users to extend their domain knowledge by providing toolset 39 Context Sensitive Dashboards
- 40. ā¢ Small SLURMcluster couple of nodes, two user groups, couple of users script & MPI workload 40 Cluster Usecase
- 41. ā¢ Small SLURMcluster couple of nodes, two user groups, couple of users script & MPI workload 41 Cluster Usecase srv backend consul slurmctld slurmctld compute0 slurmd compute<N> slurmd Compute
- 42. ā¢ Small SLURMcluster couple of nodes, two user groups, couple of users script & MPI workload 42 Cluster Usecase srv backend consul slurmctld slurmctld compute0 slurmd compute<N> slurmd Compute
- 43. ā¢ Small SLURMcluster couple of nodes, two user groups, couple of users script & MPI workload 43 Cluster Usecase srv backend consul slurmctld slurmctld compute0 slurmd compute<N> slurmd Compute carbon carbon graphite-api graphite-api Performance grafana grafana
- 44. elasticsearch ā¢ Small SLURMcluster couple of nodes, two user groups, couple of users script & MPI workload 44 Cluster Usecase srv backend consul slurmctld slurmctld compute0 slurmd compute<N> slurmd Compute carbon carbon graphite-api graphite-api Performance grafana grafana Log/Events elasticsearch logger logstash kibana kiabana kopf es-kopf
- 45. elasticsearch ā¢ Small SLURMcluster couple of nodes, two user groups, couple of users script & MPI workload 45 Cluster Usecase srv backend consul slurmctld slurmctld compute0 slurmd compute<N> slurmd Compute carbon carbon graphite-api graphite-api Performance grafana grafana Log/Events elasticsearch logger logstash kibana kiabana kopf es-kopf neo4j neo4j Inventory inventory QINBInv
- 46. elasticsearch ā¢ Small SLURMcluster couple of nodes, two user groups, couple of users script & MPI workload 46 Cluster Usecase srv backend consul slurmctld slurmctld compute0 slurmd compute<N> slurmd Compute carbon carbon graphite-api graphite-api Performance grafana grafana Log/Events elasticsearch logger logstash kibana kiabana kopf es-kopf neo4j neo4j Inventory inventory QINBInv postgres postgres galaxy galaxy galaxy
- 47. ā¢ Live clusterStatus Utilisation per cluster / user / user-group SLA met by SysOps Most common jobs, misbehaving enduser 47 Management Context
- 48. ā¢ Live clusterStatus Utilisation per cluster / user / user-group SLA met by SysOps Most common jobs, misbehaving enduser ā¢ Reports per day / user / job-type / ā¦ 48 Management Context
- 49. ā¢ Live clusterStatus Utilisation per cluster / user / user-group SLA met by SysOps Most common jobs, misbehaving enduser ā¢ Reports per day / user / job-type / ā¦ ā¢ Capacity Planning utilisation over time, comparison of HW generations, global FS capacity 49 Management Context
- 50.
- 51.
- 52. ā¢ Nodes areconnected to Partitions 52 SLURM Inventar
- 53. ā¢ Nodes areconnected to Partitions ā¢ Jobs are connected to both 53 SLURM Inventar
- 54.
- 55. ā¢ Live progressof SLURM job Monitor iteration speed to estimate workload behaviour Get to know job while itās running (instead of postmortem) Introduce application profiling / log events (enhance feedback) 55 Enduser Context
- 56. ā¢ Live progressof SLURM job Monitor iteration speed to estimate workload behaviour Get to know job while itās running (instead of postmortem) Introduce application profiling / log events (enhance feedback) ā¢ Post Mortem Get detailed report after job has finished 56 Enduser Context
- 57. ā¢ Live progressof SLURM job Monitor iteration speed to estimate workload behaviour Get to know job while itās running (instead of postmortem) Introduce application profiling / log events (enhance feedback) ā¢ Post Mortem Get detailed report after job has finished ā¢ MDO jobs depending on outcome and progression submit next iteration(s) 57 Enduser Context
- 58.
- 59.
- 60. ā¢ Live clusterStatus USE method overviews (Utilisation/Saturation/Errors) Anomaly detection (w/ and w/o humans) Spotting abnormal behaviour 60 SysOps Context
- 61. ā¢ Live clusterStatus USE method overviews (Utilisation/Saturation/Errors) Anomaly detection (w/ and w/o humans) Spotting abnormal behaviour ā¢ Drill into monitoring verify hypothesis about incidents/problems correlate events, metrics and inventory 61 SysOps Context
- 62. ā¢ Live clusterStatus USE method overviews (Utilisation/Saturation/Errors) Anomaly detection (w/ and w/o humans) Spotting abnormal behaviour ā¢ Drill into monitoring verify hypothesis about incidents/problems correlate events, metrics and inventory ā¢ Guid through āknown problemsā close feedback loops provide confidence 62 SysOps Context
- 63.
- 64.
- 65.
- 66. 66 Galaxy Use-Cases SLURM Log Events WORKFLOW Metrics Inventory ā¢Model Assess Workflow in Galaxy Easy to grasp (in contrast to Hadoop, Spark, ā¦) Event triggered, Cronjob? Using idle compute resources
- 67. 67 Thank you! ā¢ Contact christian@qnib.org @CQnib,@_qnib ā¢ Web www.qnib.org (blog) doc.qnib.org (Paper) ā¢ Feel freeā¦ ā¦ask questions (now / later) ā¦ask for a Demo