KEMBAR78
Analysis of interrupt latencies in a real-time kernel | PDF
Analysis of
                                                       interrupt latencies
                                                          in a real-time
             Università degli Studi di Trento                 kernel

   Facoltà di Scienze Matematiche Fisiche e Naturali
                                                       Real-time
                                                       operating systems

                                                       Real-time Linux

                                                       CFS
     Analysis of interrupt latencies in a              Recap

              real-time kernel



                   March 26, 2008


Laureando: Gabriele Modena (Mat. 108742)
Relatore: dott. Luigi Palopoli
Correlatore: dott. Luca Abeni
Analysis of
Real-Time Operating Systems (RTOS)                        interrupt latencies
                                                             in a real-time
                                                                 kernel



                                                          Real-time
                                                          operating systems

                                                          Real-time Linux

                                                          CFS

                                                          Recap




Ensure that an operation completes within a fixed amount
of time (deadline) =⇒ predictability.
When dealing with device drivers this implies
    managing race conditions;
    fulfill temporal constraints.
This thesis focuses on shared mixed resources.
Analysis of
Latency                                                    interrupt latencies
                                                              in a real-time
                                                                  kernel



                                                           Real-time
Latency is the measure of the difference between the       operating systems

theoretical (algorithm) and actual (implementation)        Real-time Linux

schedule.                                                  CFS

                                                           Recap
In a RTOS latency:
    should be bounded;
    the bound should not be too high.
The goal of a real-time kernel is to minimize latency by
using algorithms and technology that properly
implements them:

                  preemptible kernel.
Analysis of
Linux as an RTOS                                    interrupt latencies
                                                       in a real-time
                                                           kernel



                                                    Real-time
                                                    operating systems
Linux vanilla:
                                                    Real-time Linux
    since 2.6 preemptible kernel;                   CFS

The rt-preempt patch:                               Recap


    developed by Ingo Molnar;
    makes Linux hard real-time;
    uses the new (2.6.21) high resolution timers;
    preemptible critical sections;
    real-time mutexes;
    threaded interrput handlers.
Analysis of
Benchmarking rt-preempt                                   interrupt latencies
                                                             in a real-time
                                                                 kernel



                                                          Real-time
                                                          operating systems

                                                          Real-time Linux

                                                          CFS

                                                          Recap



Two goals:
    test the actual real-time performance;
    test the performance of I/O scheduling.
We look for a trade-off between latency and throughput.
Analysis of
Cyclictest                                                 interrupt latencies
                                                              in a real-time
                                                                  kernel



                                                           Real-time
                                                           operating systems

                                                           Real-time Linux

                                                           CFS
    high resolution test program written by Thomas         Recap
    Gleixner;
    measure latency;
    schedules periodic threads;
When it comes to real-time applications rt-preempt gives
the expected performance.
Analysis of
Vanilla vs. rt-preempt (1)                                                                                                  interrupt latencies
                                                                                                                               in a real-time
                                                                                                                                   kernel
 Probability of the latency (num. of samples / total samples)




                                                                                                                            Real-time
                                                                                       Preemptible Linux
                                                                                                                            operating systems
                                                                    1
                                                                                                                            Real-time Linux

                                                                                                                            CFS
                                                                   0.1
                                                                                                                            Recap

                                                                  0.01


                                                                 0.001


                                                                0.0001


                                                                1e-05


                                                                1e-06
                                                                         0   1000   2000    3000       4000   5000   6000
                                                                                       Latency (usecs)
Analysis of
Vanilla vs. rt-preempt (2)                                                                                         interrupt latencies
                                                                                                                      in a real-time
                                                                                                                          kernel
 Probability of the latency (num. of samples / total samples)




                                                                                                                   Real-time
                                                                                          RT Linux
                                                                                                                   operating systems
                                                                    1
                                                                                                                   Real-time Linux

                                                                                                                   CFS
                                                                   0.1
                                                                                                                   Recap

                                                                  0.01


                                                                 0.001


                                                                0.0001


                                                                1e-05


                                                                1e-06
                                                                         0   10   20       30       40   50   60
                                                                                       Latency (usecs)
Analysis of
I/O scheduling                                           interrupt latencies
                                                            in a real-time
                                                                kernel



                                                         Real-time
                                                         operating systems

                                                         Real-time Linux

                                                         CFS
    In traditional operating systems interrupt handler   Recap

    have priority above everything.
    rt-preempt =⇒ soft and hard irq handlers are
    threads:
             scheduling I/O ≡ scheduling tasks.
Analysis of
Benchmark network throughput                 interrupt latencies
                                                in a real-time
                                                    kernel



                                             Real-time
                                             operating systems

                                             Real-time Linux

                                             CFS
   netperf;                                  Recap

   UDP traffic;
   750B datagrams;
   schedule the real-time network threads;
   30 minutes execution.
Analysis of
Fixed priority scheduling                                    interrupt latencies
                                                                in a real-time
                                                                    kernel
It is not possibile to obtain low latency and high
throughput;                                                  Real-time
                                                             operating systems

                                                             Real-time Linux

 Net priority   min. lat (µs)    max. lat (µs)   Tp (Mbps)   CFS

      1              20              94            38.9      Recap

     50              22              96            39.9
     80              20             148            77.4
     99              22             164            73.4
      Table: Linux 2.6.24-rc2-rt1, 750B UDP datagrams


    high throughput and high latency;
    low throughput and low latency.
Fixed priority scheduling is not suitable =⇒ we look for
a method to assign CPU quota to tasks.
Analysis of
Completely Fair Scheduler              interrupt latencies
                                          in a real-time
                                              kernel



                                       Real-time
                                       operating systems

                                       Real-time Linux

                                       CFS

                                       Recap
    implemented by Ingo Molnar;
    merged in mainline since 2.6.23;
    weights tasks by ratio;
    group scheduling;
Analysis of
cfs & rt-preempt                                      interrupt latencies
                                                         in a real-time
                                                             kernel

My work
                                                      Real-time
    use CFS and modify the nice system call to set    operating systems

    weights (> 200 µs);                               Real-time Linux

                                                      CFS
    use CFS and tune Control Groups (> 1000 µs).
                                                      Recap
Results

          net   cyc   load   lat (µs)   Tp (Mbps)
           1     1      1     2931        49.9
           2     1      1     3911        67.3
           2     2      1     3721        71.1
           4     2      1     6117        80.9
           1     1      4     2871        35.2
       Table: Scheduling real-time threads with CFS
Analysis of
cfs & rt-preempt                                       interrupt latencies
                                                          in a real-time
                                                              kernel



                                                       Real-time
                                                       operating systems

                                                       Real-time Linux

                                                       CFS
    impact on the I/O throughput =⇒ the ratio system
                                                       Recap
    works;
    high latencies.
Short time possible solutions
    CFS clean up and tuning (in progress);
    EDF scheduling + time limit (Linux 2.6.25).
Analysis of
Overview of the work                                           interrupt latencies
                                                                  in a real-time
Step 1                                                                kernel

    litterature study on real-time computing;
                                                               Real-time
    study of the rt-preempt patch and real-time under Linux;   operating systems

                                                               Real-time Linux
Step 2                                                         CFS

                                                               Recap
    benchmark real-time performance;
    study the relationship between I/O and real-time
    applications;

Step 3
    test the CFS scheduler;
    try custom and standard solutions;
Future work
    results presented at the RTAS 2008 conference
    (http://www.rtas.org/ );
    tuning CFS and testing new scheduling policies.
Analysis of
Questions                   interrupt latencies
                               in a real-time
                                   kernel



                            Real-time
                            operating systems

                            Real-time Linux

                            CFS

                            Recap



            Any question?

Analysis of interrupt latencies in a real-time kernel

  • 1.
    Analysis of interrupt latencies in a real-time Università degli Studi di Trento kernel Facoltà di Scienze Matematiche Fisiche e Naturali Real-time operating systems Real-time Linux CFS Analysis of interrupt latencies in a Recap real-time kernel March 26, 2008 Laureando: Gabriele Modena (Mat. 108742) Relatore: dott. Luigi Palopoli Correlatore: dott. Luca Abeni
  • 2.
    Analysis of Real-Time OperatingSystems (RTOS) interrupt latencies in a real-time kernel Real-time operating systems Real-time Linux CFS Recap Ensure that an operation completes within a fixed amount of time (deadline) =⇒ predictability. When dealing with device drivers this implies managing race conditions; fulfill temporal constraints. This thesis focuses on shared mixed resources.
  • 3.
    Analysis of Latency interrupt latencies in a real-time kernel Real-time Latency is the measure of the difference between the operating systems theoretical (algorithm) and actual (implementation) Real-time Linux schedule. CFS Recap In a RTOS latency: should be bounded; the bound should not be too high. The goal of a real-time kernel is to minimize latency by using algorithms and technology that properly implements them: preemptible kernel.
  • 4.
    Analysis of Linux asan RTOS interrupt latencies in a real-time kernel Real-time operating systems Linux vanilla: Real-time Linux since 2.6 preemptible kernel; CFS The rt-preempt patch: Recap developed by Ingo Molnar; makes Linux hard real-time; uses the new (2.6.21) high resolution timers; preemptible critical sections; real-time mutexes; threaded interrput handlers.
  • 5.
    Analysis of Benchmarking rt-preempt interrupt latencies in a real-time kernel Real-time operating systems Real-time Linux CFS Recap Two goals: test the actual real-time performance; test the performance of I/O scheduling. We look for a trade-off between latency and throughput.
  • 6.
    Analysis of Cyclictest interrupt latencies in a real-time kernel Real-time operating systems Real-time Linux CFS high resolution test program written by Thomas Recap Gleixner; measure latency; schedules periodic threads; When it comes to real-time applications rt-preempt gives the expected performance.
  • 7.
    Analysis of Vanilla vs.rt-preempt (1) interrupt latencies in a real-time kernel Probability of the latency (num. of samples / total samples) Real-time Preemptible Linux operating systems 1 Real-time Linux CFS 0.1 Recap 0.01 0.001 0.0001 1e-05 1e-06 0 1000 2000 3000 4000 5000 6000 Latency (usecs)
  • 8.
    Analysis of Vanilla vs.rt-preempt (2) interrupt latencies in a real-time kernel Probability of the latency (num. of samples / total samples) Real-time RT Linux operating systems 1 Real-time Linux CFS 0.1 Recap 0.01 0.001 0.0001 1e-05 1e-06 0 10 20 30 40 50 60 Latency (usecs)
  • 9.
    Analysis of I/O scheduling interrupt latencies in a real-time kernel Real-time operating systems Real-time Linux CFS In traditional operating systems interrupt handler Recap have priority above everything. rt-preempt =⇒ soft and hard irq handlers are threads: scheduling I/O ≡ scheduling tasks.
  • 10.
    Analysis of Benchmark networkthroughput interrupt latencies in a real-time kernel Real-time operating systems Real-time Linux CFS netperf; Recap UDP traffic; 750B datagrams; schedule the real-time network threads; 30 minutes execution.
  • 11.
    Analysis of Fixed priorityscheduling interrupt latencies in a real-time kernel It is not possibile to obtain low latency and high throughput; Real-time operating systems Real-time Linux Net priority min. lat (µs) max. lat (µs) Tp (Mbps) CFS 1 20 94 38.9 Recap 50 22 96 39.9 80 20 148 77.4 99 22 164 73.4 Table: Linux 2.6.24-rc2-rt1, 750B UDP datagrams high throughput and high latency; low throughput and low latency. Fixed priority scheduling is not suitable =⇒ we look for a method to assign CPU quota to tasks.
  • 12.
    Analysis of Completely FairScheduler interrupt latencies in a real-time kernel Real-time operating systems Real-time Linux CFS Recap implemented by Ingo Molnar; merged in mainline since 2.6.23; weights tasks by ratio; group scheduling;
  • 13.
    Analysis of cfs &rt-preempt interrupt latencies in a real-time kernel My work Real-time use CFS and modify the nice system call to set operating systems weights (> 200 µs); Real-time Linux CFS use CFS and tune Control Groups (> 1000 µs). Recap Results net cyc load lat (µs) Tp (Mbps) 1 1 1 2931 49.9 2 1 1 3911 67.3 2 2 1 3721 71.1 4 2 1 6117 80.9 1 1 4 2871 35.2 Table: Scheduling real-time threads with CFS
  • 14.
    Analysis of cfs &rt-preempt interrupt latencies in a real-time kernel Real-time operating systems Real-time Linux CFS impact on the I/O throughput =⇒ the ratio system Recap works; high latencies. Short time possible solutions CFS clean up and tuning (in progress); EDF scheduling + time limit (Linux 2.6.25).
  • 15.
    Analysis of Overview ofthe work interrupt latencies in a real-time Step 1 kernel litterature study on real-time computing; Real-time study of the rt-preempt patch and real-time under Linux; operating systems Real-time Linux Step 2 CFS Recap benchmark real-time performance; study the relationship between I/O and real-time applications; Step 3 test the CFS scheduler; try custom and standard solutions; Future work results presented at the RTAS 2008 conference (http://www.rtas.org/ ); tuning CFS and testing new scheduling policies.
  • 16.
    Analysis of Questions interrupt latencies in a real-time kernel Real-time operating systems Real-time Linux CFS Recap Any question?