_________________________________________________________________
IRQTUNE -- A LINUX X86 IRQ PRIORITY OPTIMIZER
Copyright 1996 by Craig Estey.
This FAQ is revision 0.2
Last updated: Mon Aug 26 04:53:01 PDT 1996
See the Changes section at the bottom of this document.
irqtune changes the IRQ priority of devices to allow devices that
require high priority and fast service (e.g. serial ports, modems) to
have it.
With irqtune, a 3X speedup of serial/modem throughput is possible.
_________________________________________________________________
WHERE DO I GET IRQTUNE?
irqtune is free software under the terms and conditions of the GNU
Public License. See the file COPYING, included in the distribution,
for details.
* It's available via anonymous ftp from:
ftp://www.best.com/pub/cae/irqtune.tgz (gzipped tar archive)
* The author is Craig Estey, (cae@best.com).
* This FAQ file is available both in html and text versions included
in the tar file. This FAQ is also available online via
http://www.best.com/~cae/irqtune
_________________________________________________________________
HOW DO I KNOW IF I NEED IRQTUNE?
You are running Linux on an x86 PC, other architectures to be
implemented later--Sorry.
You probably need irqtune, if you are experiencing any of the
following:
* SLIP/PPP transfers seem slow. For example, using a 28.8 (or
better) modem, the effective throughput is approximately 700
bytes/second instead of the expected 2500 bytes/second.
* A running SLIP/PPP dies at random times.
* Serial connections are slow or drop data.
* Netscape hangs mysteriously or stalls when trying to access a web
page.
* Equivalent serial/PPP programs under Windoze run much faster than
under Linux.
* Disk accesses seem to interfere with SLIP/PPP.
* Interrupt handlers for specialized, time critical devices don't
get control when they need to.
_________________________________________________________________
WHAT IS ACTUALLY HAPPENING TO CAUSE THESE PROBLEMS?
When the PC boots Linux, the timer is given, by default, the highest
IRQ priority in the system (it's IRQ 0 and thus, priority 0). On a
standard configuration, the serial ports are priority 11 and 12!!!
This means that 10 other devices have higher priority.
Q: So what does IRQ priority do?
When multiple devices are in contention to interrupt the CPU, their
priority decides which interrupts will occur in what order.
Q: When does this contention occur?
After an arbitrary period of having interrupts disabled (e.g after a
cli), at the point where they're reenabled (sti). This can happen in
several places:
* In an ISR that runs with interrupts locked, it happens in the
epilog, just before attempting to execute the bottom-half.
* The bottom-half itself may do a lock and unlock.
* When a task that enters the kernel to do a system call, the system
call handler may lock and unlock interrupts briefly.
* Almost anywhere in the kernel are brief periods where it does a
cli to lock interrupts and then an sti to unlock them again.
Q: If there are multiple interrupts now pending,which one gets the
service, the serial or some other?
In the default configuration, the serial ISR will usually lose as it's
priority 11.
_________________________________________________________________
HOW DOES IRQTUNE HELP THIS?
irqtune gives priority 0 to the whatever device you specify. If we
specify a serial device, irqtune guarantees that the serial ISR gets
control whenever a contention occurs.
_________________________________________________________________
WHY DOES THE SERIAL INTERRUPT SERVICE REQUIRE THE HIGHEST PRIORITY?
Q: Why does the serial device merit such special treatment?
Serial devices are somewhat unique. Even though they have one of the
slowest data rates (relative to a disk), they are the largest consumer
of interrupts and are extremely sensitive to interrupt latency (the
time from when a device yanks the IRQ line until its ISR is executed).
Q: Could you give a concrete example of this?
* For a modem running at 33.6, we can have a peak maximum (with
compression) of 6700 bytes/second. The serial driver programs the
silo to interrupt after 8 bytes, leaving a latency window of 8
bytes. This means that when the serial port yanks its IRQ line, it
can still absorb 8 more characters before its buffer will overflow
and data will be dropped
* In terms of time, this means that the maximum that the serial ISR
may be delayed is (8 / 6700) seconds or 1194 microseconds. It also
means that the serial ISR will require 838 interrupts/second.
* Currently there are 52 devices that install their ISR with the
SA_INTERRUPT option. This means that they wish to run with
interrupts disabled. I have not looked at all the devices, but
just assume for the moment that all will run with interrupts
locked for an arbitrary period.
* Assume we've got 4 of them pending (remember, at higher priority).
Assume that they will lock interrupts individually for 320, 300,
190, and 500 microseconds, respectively. Assume that the serial
ISR also wants an interrupt.
* This means that the serial ISR will have to wait for (320 + 300 +
190 + 500) or 1310us for these other higher priority ISR's to run
to completion. This is greater than the maximum of 1194 us.
* With internal modems, we probably just see start/stop behavior
causing a slowdown. With an external modem, data will be dropped.
This will cause PPP to see a CRC error and request transmission.
Remember, the entire packet must be retransmitted, which means we
just wasted 296-1500 bytes (depending on the selected MRU). With
an MRU of 1500 bytes, we just wasted 23.4% of the bandwidth in a
given second--all for the loss of a single byte!
* Worse yet, consider a 115 Kbps ISDN card that is masquerading as a
serial device (Yes, I know it should be a DMA device). In this
case, the maximum delay the serial ISR can wait is 695 us. And
that's just one ISDN channel. Try adding the other and, well,
golly :-).
Q: In this example, how would boosting serial IRQ priority help?
* If the serial IRQ had priority 0, it would get in before the
others. It could also re-interrupt after any individual lockout
window. Thus, the maximum it would be forced to wait would be
largest individual time, not the summation of all of these times.
In this example, this means 500 us. which is now within the 1194
us. maximum for the serial ISR.
* It has been my experience that the serial device must always win
these battles for contention. When serial devices don't get what
they need, when they need it, they slow horribly or drop data
outright.
_________________________________________________________________
DOESN'T THIS HURT THE PERFORMANCE OF OTHER DEVICES?
Not really. In actual practice, most devices don't even notice the
difference. Most other devices (e.g. disks, tape, ethernet) are DMA
devices. The DMA does most of the work, thus greatly reducing their
need for interrupts. If the device allows a request queue, it may
function autonomously on several requests, producing only one
interrupt for the entire batch.
Furthermore, serial interrupt services are, themselves, very fast.
They slam their data as quickly as possible and get out ASAP. No fancy
calculations, just the minimum, mindless data transfer. Almost
everything else is handled later, in the bottom-half with interrupts
enabled. In fact, a serial ISR may have to re-interrupt it's own
bottom-half several times.
Those devices that do experience some slight slowdown are more likely
to have long interrupt disable windows themselves. Having several
smaller cli/sti windows is much better than one large cli/sti
window--It's just harder to program.
Q: But suppose I really want both fast serial and fast disk?
Ultimately, it's a bit of a compromise. Which is better:
* Reliable serial and slightly slower disk.
* Slightly faster disk and unreliable serial/modem support.
When paying an ISP for Internet access in $$$/hour, it's an easy
decision :-).
_________________________________________________________________
ISN'T THIS IRQ PRIORITY THING A BIT OF A NEW IDEA?
No. It's actually an old idea. I've been doing device drivers since
1977 and Unix kernel work since 1981. I've personally written 8 serial
drivers have used this many times commercially. Giving the serial
device the highest priority is actually standard practice in many
systems. With a 4Mhz CPU, these problems used to occur at 1200 baud
:-)
_________________________________________________________________
HOW DO I INSTALL IRQTUNE?
* Unpack the tar file:
gzip -d
It will create a directory, irqtune in
whatever is the current directory.
Go to that directory:
cd ./irqtune
To install irqtune, enter:
make install
This will install the files:
/sbin/irqtune
/sbin/irqtune_mod.o
_________________________________________________________________
HOW DO I USE IRQTUNE? DON'T I HAVE TO REBUILD MY KERNEL?
No, you do not have to rebuild your kernel. irqtune uses insmod and
rmmod to dynamically load and unload a kernel module. But you are
correct in sensing that irqtune is a kernel patch.
Q: Ok, if it's a kernel patch, why not just issue a kernel patch like
everybody else does (e.g. diff -u output)?
irqtune will work even if you don't have the kernel source loaded. It
uses insmod to load the patch, invoke it, and then unload it. The IRQ
priority changes will last so long as the kernel is booted.
Q: How do I invoke it?
irqtune takes two arguments optional arguments:
* irqtune master slave
The only caveat is that you must specify the full pathname, evenif
irqtune is placed in a directory that is in $PATH. This is required
because irqtune uses argv[0] to locate its irqtune_mod.o file.
The default is 3 14 which will work for many standard configurations.
More on this later.
Q: Could I do this from my /etc/rc.d/rc.local file?
Yes. Just add a /sbin/irqtune line to this file and you're in
business. You may also issue another irqtune command at any time.
Q: But aren't kernel patches dependent on the particular revision of
the kernel? What if my kernel revision is different from the kernel
revision that you built it on?
irqtune is 99.44% kernel revision independent. It is built using ELF
binaries, so your insmod must understand them. Also, make sure that
you have the correct insmod for your kernel. But that's really about
it.
Q: But what if I don't have ELF binary support, how can I still do
things?
Well, I'd recommend that you upgrade your kernel as ELF binaries are
cool :-) But if you insist, you'll just have to recompile irqtune.
Just be sure that /usr/src/linux/include is installed. The exact
procedure for building a.out binaries can vary with compiler revision,
so be sure to check your documentation on this (You may need to add a
parameter or two).
* Then, just type:
make clean
make sbin
make install
That should do the trick. But, if it doesn't, look at revisionnumbers
on insmod, gcc, ld, and the kernel for incompatibilities. Upgrade as
necessary.
_________________________________________________________________
WHAT ABOUT MY NON-STANDARD HARDWARE CONFIGURATION?
irqtune defaults for a standard IRQ configuration. It assumes that the
highest priority device should be on IRQ 3. This is normally the first
serial port on standard configurations, which is what you want.
Q: How do I determine what my IRQ configuration is?
Just type cat /proc/interrupts and you'll get something like:
0: 8578913 timer
1: 109547 keyboard
2: 0 + cascade
3: 86470 + serial
4: 197648 + serial
13: 1 math error
14: 93123 + Ux4F
Note that /proc/interrupts only reports on active devices. So to scope
out the serial IRQ's ideally you'd have X Windows up with your serial
mouse and be connected via PPP to the net.
Q: OK, I've got the output from /proc/interrupts, what do I do with
it?
The leftmost number is the IRQ number. The rightmost column is the
internal device name (not to be confused with /dev names). In the
above case, the two serial ports are on IRQ 3 and IRQ 4. Just use the
lower number, in this case 3:
* /sbin/irqtune 3
This sets IRQ 3 to the highest priority. In fact, before we invoked
irqtune, the IRQ number was also its priority:
IRQ PRIOR
0 0
1 1
2 2
3 3
4 4
5 5
6 6
7 7
After this command, the IRQ priorities are now:
IRQ PRIOR
0 5
1 6
2 7
3 0
4 1
5 2
6 3
7 4
Q: BTW, What's the cascade device I saw in the output of
/proc/interrupts?
Glad you asked. There are actually two interrupt controllers, a master
and a slave. The slave is cascaded to the master via its IRQ 2. The
master controls IRQ's 0-7 and the slave controls IRQ's 8-15.
You actually may select two high IRQ priorities, one for the master
and one for the slave. irqtune defaults the slave to IRQ 14, which is
normally the disk controller.
Although the normal notation is to refer to IRQ's as 0-15, it may be
easier to understand if we refer to the master IRQ's as M0-M7 and the
slave IRQ's as S0-S7.
Q: But I've also got an Ethernet controller on IRQ 12. What about
that?
In other words, your configuration might look something like this:
0: 8578913 timer
1: 109547 keyboard
2: 0 + cascade
3: 86470 + serial
4: 197648 + serial
12: 17968 + eth
13: 1 math error
14: 93123 + Ux4F
In this case, you might want to use:
* /sbin/irqtune 3 12
because you want your ethernet card to have a higher priority than the
diskcontroller. Actually if you did have this configuration, setting 3
14 (the default) would make the ethernet card, the lowest priority
device in the system.
In our new notation IRQ 12 is S4, and the resulting priority would be:
IRQ M/S PRIOR
0 M0 5
1 M1 6
2 M2 7
3 M3 0
4 M4 1
5 M5 2
6 M6 3
7 M7 4
8 S0 12
9 S1 13
10 S2 14
11 S3 15
12 S4 8
13 S5 9
14 S6 10
15 S7 11
Q: Suppose I also had a serial multiplexer card on IRQ 11?
Once again, your configuration might look something like this:
0: 8578913 timer
1: 109547 keyboard
2: 0 + cascade
3: 86470 + serial
4: 197648 + serial
11: 197648 + sermux
12: 17968 + eth
13: 1 math error
14: 93123 + Ux4F
This configuration is a bit tricky because now we've got a serial
device on the slave controller. It would be much better to put all
serial cards on the master controller. Things would stay much simpler.
In this case you would want to use:
* /sbin/irqtune 2 11
The resulting priorities would be more complex and would result in
somethinglike:
IRQ M/S PRIOR
0 M0 13
1 M1 14
2 M2 0
3 M3 8
4 M4 9
5 M5 10
6 M6 11
7 M7 12
8 S0 5
9 S1 6
10 S2 7
11 S3 0
12 S4 1
13 S5 2
14 S6 3
15 S7 4
The reason things would be better if all serial devices were on the
master is that now you have serial devices at priorities 0, 8, and
9.
Q: So what's wrong with that?
Well, we boosted the priority of the serial multiplexer at the expense
of the regular serial ports. The only way to allow all serial ports
equally high priority is to group them on consecutive IRQ's and set
the high priority for the lowest of those IRQ's.
_________________________________________________________________
HOW CAN I TELL IF IRQTUNE ACTUALLY DID ANYTHING FOR ME?
Well, first off, if PPP/SLIP was dying mysteriously, it will probably
be more reliable.
Secondly, run without it and get a feel for the transfer rate:
* Hit many favorite web sites and note the transfer rates in
bytes/second. Make life easy. Netscape is at least one browser
that reports transfer rates in bytes/second in the status line.
* FTP reports the transfer time of a file in bytes/second. Download
(or upload) a few files (300K or greater to smooth out the
benchmark) and note the transfer rates.
* Try several things of varying duration, different times of day,
different sites to accomodate variations in network loading. Don't
stop until there is an average set of numbers that are more or
less repeatable.
Repeat this using irqtune and note the transfer times again. NOTE:
IRQTUNE just won't quit--if you want to test in the original mode
again, reboot the system first.
_________________________________________________________________
WHAT IF I STILL DON'T SEE ANY REAL IMPROVEMENT?
It's a matter of probability. Performance measurement is as much art
as science.
* First, verify that there are no more obvious problems like bad
config, bad cabling, etc.
* We're much more likely to see improvement on a DX2/66 than a
Pentium/166. With the 166, we may be overpowering the problem.
We'll still have a problem, we just won't notice it as much
because the Pentium has the extra speed to burn. A badly tuned
Ferrari may still outperform a well-tuned VW :-)
* System loading is very light. The problems that irqtune will fix
are more likely to happen when more devices and more work are
added.
* What problem was occurring before? Which of the symptoms listed
earlier is happening? If performance was 2500 bytes/second before
using irqtune, we're less likely to notice the smaller jump to,
say, 2800.
* Verify that irqtune got the correct IRQ numbers for the specific
configuration. For example, suppose our primary serial port is on
IRQ 3, but we gave irqtune the value 4. We just made the serial
device on IRQ 3 into priority 14, not priority 0.
* We may have a rogue interrupt service that disables interrupts for
something outrageous, say 2 ms. This is singularly longer than the
1194 us. in the earlier example. irqtune will still help, but the
real solution here is to reduce interrupt lockout times in the
other device below the 1194 us. threshold.
_________________________________________________________________
WHAT ABOUT OTHER REMEDIES I'VE HEARD ABOUT?
Q: What about disabling Van Jacobsen header compression?
This reduces the amount of bottom-half processing the system has to do
at the expense of larger packets being sent.
Q: What about using ``hdparm -u'' to set the interrupt-unmask flag in
hard disk driver?
It suffers from the same problem as the bottom-half routine. The disk
controller typically uses IRQ 14 and 15. While the slave interrupt
controller would probably allow preemption, the master (on IRQ 2)
would not because the priority of all slave devices is higher than the
serial IRQ priority.
Q: What about adjusting the MRU/MTU numbers?
This will have less of an effect now. In fact, we normally reduced the
MRU to a minimum (296) to reduce the bottom-half processing and
<flip-buffer> latency at the expense of adding extra overhead bytes
due to the reduced packet size. We may now actually be able to
increase the MRU to regain the efficiency.
Beware: Do this slowly as the optimal may not be 1500. The flip buffer
in the serial/tty drivers is only 512 bytes.
Q: What about going to newer kernel revisions?
Although irqtune will work surprisingly well with just about any
kernel revision, the low level IRQ handlers and device drivers have
been vastly improved in the 2.0.X kernels. This will only improve
irqtune's effect. In fact, 2.0.X and irqtune actually complement one
another.
_________________________________________________________________
DIDN'T YOU GIVE ANOTHER EXPLANATION BEFORE, INVOLVING BOTTOM-HALF ROUTINES?
Yes.
* My error was that I had looked at all the existing IRQ code in
Linux, when I was first investigating the serial problem. I spaced
out, misread the code, and concluded the EOI was done in the
epilog. I also was looking at the slow interrupt prolog/epilog.
* Anyway, I then did measurements and verified that the serial ISR
never got control at an effective rate higher than 1000 bytes/sec.
* It was clear that something was holding up the serial ISR.
* This pointed to a priority problem.
* I coded up irqtune and tried it.
* irqtune worked. I went from 700 bytes/second to 2500 bytes/second.
* I was so happy surfing at 3X my former speed, that I didn't bother
to try to track down the culprit exactly as I had used this
priority fix before.
* After running it for about a month, I decided to package it up so
others could use it. In my haste to get it out, I wrote the
explanation too quickly. I also wanted to get some feedback from
various configurations and change things accordingly. It was a
case of round up the usual suspects.
* In fact, now I'll be adding kernel tracing that will give some
real world numbers to back things up. When I have completed this
analysis, I'll add them to this FAQ.
Many thanks to those that pointed this out.
_________________________________________________________________
CHANGES
FAQ 0.2 Changes:
* Major rewrite and expansion of the problem explanation section
* More thorough explanation of how and why irqtune works
* Explanation of why serial devices must be highest priority
* Impact on other devices
* Cleaner and better installation instructions
* Better benchmarking section
* Problem resolution section
* Explanation of my prior misread on the EOI thing