The problem here is that savecore needs to use a kernel file to build the dump image. By default, savecore uses the booting kernel. If you are booting off a different kernel after a panic, you must run savecore manually to tell it where to find the proper kernel file. Interrupt the boot during the initial countdown, and boot into single-user mode with this command: ………………………………………………………………………………………. ok boot -s ………………………………………………………………………………………. When the system gives you a command prompt, fsck your system first. After a panic, the disks are almost always dirty: ………………………………………………………………………………………. # fsck -p ………………………………………………………………………………………. (This can take several minutes on a modern (huge) disk.) Once fsck finishes, mount the filesystem where you keep your kernel core files: ………………………………………………………………………………………. # mount /var ………………………………………………………………………………………. Finally, save your kernel core using the proper kernel file, telling savecore which kernel file to use with the -N flag. If your panicked kernel is /kernel.bad, use something like this: ………………………………………………………………………………………. # savecore -N /kernel.bad /var/crash ………………………………………………………………………………………. You can, of course, use additional savecore options like -v and -z in a manual core dump. Using the Dump If you’re a kernel developer, this is where you stop listening to me and rely upon your own debugging experience. If you’re a new systems administrator, though, you probably don’t know enough about C and kernel internals to have any real hope of debugging a complicated kernel issue. As such, we’ll focus on extracting enough information to give a developer a good shot at identifying the problem. If you look at /var/crash after a dumped panic, you’ll see the files kernel.0 and vmcore.0. (Each subsequent crash dump will get a consecutively higher number, such as kernel.1 and vmcore.1.) The vmcore.0 file is the actual memory dump, while the kernel.0 file is a copy of the crashed kernel. The kernel.0 file isn’t useful for what we’re doing, but keep it just in case. The vmcore.0 file is vital. Once you actually have a crash, you might copy your debugging kernel to /var/crash/kernel.debug.0 to keep dumps in sync with their kernels. 457

Note: If you are looking for good and high quality web space to host and run your application check Lunarwebhost Linux Web Hosting services

Dumpon Now it’s time to tell the system where to write the core dump this location is the dumpdev. FreeBSD uses the swap partition as the dump device, which is why it has to be slightly larger than your physical memory. (You can use a UFS partition, but after the crash it won’t be a usable UFS partition any more!) You can get the device name from /etc/fstab. Look for a line with a FSType entry of swap; the first entry in that line is the physical device name. For example, on my laptop, my swap field in /etc/fstab looks like this: ………………………………………………………………………………………. /dev/ad0s4b none swap sw 0 0 ………………………………………………………………………………………. Tell the system to use a dump device with dumpon(8), which must be set each time the system boots. Of course, as you might guess, there’s an rc.conf switch for this. My swap partition is /dev/ad0s4b, so I specify this as the dump device in /etc/rc.conf: ………………………………………………………………………………………. dumpdev=”/dev/ad0s4b” ………………………………………………………………………………………. Savecore Next, tell your system where to save the dump after the reboot using savecore(8). You can change the default, /var/crash, with rc.conf’s dumpdir setting. (This directory must exist; savecore will not create it!) As you become more experienced in saving panics, you may find that you need to adjust the core-saving behavior. Read savecore(8), and set any appropriate options in savecore_flags in /etc/rc. One popular flag is -z, which compresses the core file and can save some disk space. Savecore is now smart enough to automatically eliminate unused memory from the dump, which can save a lot of room. Upon a Crash If you’re in front of your computer when it crashes, you’ll see the panic message. If the system is set to reboot automatically, numbers will start to flow by, counting the megs of memory being dumped. Finally, the computer will reboot, fsck will run, and you can watch savecore copy the memory dump from swap to a file. If your system doesn’t reboot automatically, you’ll need to enter two commands after the panic, at the debugger prompt: panic to sync the disks and continue to start the reboot process. (FreeBSD supports many other debugging options, but you have to know how to use the kernel debugger to make use of them.) Dumps and Bad Kernels Some kernels just crash and die during boot, or won’t stay up long enough to fix a problem. In that case, you need to boot with a different kernel. 456

Note: If you are looking for good and high quality web space to host and run your application check Lunarwebhost PHP Web Hosting services

………………………………………………………………………………………. options DDB makeoptions DEBUG=-g ………………………………………………………………………………………. The DDB option installs the DDB kernel debugger. (This isn’t strictly necessary, but it can be helpful and it doesn’t take up that much room.) The makeoptions you set here tell the system to build a debugging kernel. Post-Panic Behavior When configuring your system, you’ll need to decide how you want the system to behave after a panic. Do you want the computer to reboot automatically, or do you want it to stay at the panic screen until you manually trigger a reboot? If the system is at a remote location, you’ll almost certainly want the computer to reboot automatically, but if you’re at the console debugging kernel changes, you might want it to wait for you to tell it to reboot. To reboot automatically, include the kernel option DDB_UNATTENDED: ………………………………………………………………………………………. options DDB_UNATTENDED ………………………………………………………………………………………. If you don’t include this option, the system will wait for you to tell it to reboot. kernel.debug Once you have the kernel configured the way you want, do the usual dance (described in Chapter 4) to configure and install it. Once you’ve installed your new kernel, you’ll find a file in the kernel compile directory called kernel.debug. This is your kernel with symbols; save it somewhere. The next time you upgrade your system or customize the kernel, this debugging kernel will be overwritten by a new debugging kernel. If you’ve built a kernel just for testing, you want to be sure that you have your known-to-be-good debugging kernel available. One of the frequent causes of a failed debugging process is losing the debugging kernel and trying to debug a crashed kernel with a different kernel.debug. This won’t work. I generally copy kernel.debug to /var/crash/kernel.debug.date, so I can tell when a particular debug kernel was built. This lets me date-match the current kernel to a debugging kernel, and also tells me when a kernel.debug is old enough that I can delete it. With any luck, you’ll never need these debugging kernels, though personally, I’ve found my luck to be unreliable. Debugging kernels take little disk space and provide quick answers when trouble hits, so I strongly suggest using them. 455

Note: If you are looking for good and high quality web space to host and run your application check Lunarwebhost Linux Web Hosting services

panic happens. Set it up when you install the server. That way, you’ll get a backtrace automatically if it ever crashes. This might seem like a novel idea, and it certainly isn’t emphasized in the FreeBSD documentation, but it makes sense to be ready for disaster. If it never happens, well, you don’t have anything to complain about. If you get a panic, you’re ready and you’ll be able to present the FreeBSD folks with a complete debugging dump the second a problem appears. Prerequisites prepare for a kernel panic, you need to have the system source code installed. You’ll also need one (or more) swap partitions that is at least 1MB larger than your physical memory, and preferably twice as large as your RAM. If you have 512MB of RAM, for example, you need a swap partition that is 513MB or larger, with 1024MB being preferable. (On a server, you should certainly have multiple swap partitions on multiple drives!) If your swap partition isn’t large enough, you’ll have to either add another hard drive with an adequate swap partition, or reinstall. (While having a /var partition with at least that much disk space free is helpful, it isn’t necessary.) If you followed the installation suggestions in the beginning of the book, you’re all set. Crash Dump Process The kernel crash-capturing process works somewhat like this. If a properly configured system crashes, it will save a core dump of the system memory. You can’t save it to a file, because the crashed kernel doesn’t know about files; it only knows about partitions. The simplest place to write this dump is to the swap partition, and the dump is placed as close to the end of the swap partition as possible. Once the crashing system saves the core to swap, it reboots the computer. During the reboot, /etc/rc enables the swap partition. It then (probably) runs fsck on the crashed disks. It has to enable swapping before running fsck, because fsck might need to use swap space. Hopefully, you have enough swap space that fsck can get everything it needs without overwriting the dump file lurking in your swap partition. Once the system has a place where it can save a core dump, it checks the swap partition for a dump. Upon finding a core dump, savecore copies the dump from swap to the proper file, clears the dump from swap, and lets the reboot proceed. You now have a kernel core file, and can use that to get a backtrace. The Debugging Kernel The standard FreeBSD kernel install removes all the debugging information from the kernel before installing it, including symbols, which provide a map between the machine code and the source code. Such a map can be larger than the actual program, and nobody wants to run a kernel that’s three times larger than it has to be! However, we need this map, and other debugging information, to diagnose what went wrong in the crash. This map also includes a complete list of source-code line numbers, so the developer can learn exactly where a problem occurred. Without this information, the developer is stuck trying to map a kernel core to the source code by hand, which is somewhat like trying to assemble a million-piece puzzle without a picture, or even knowing that you have all the pieces. Overall, this is an ugly job. It’s even uglier when you consider that the developer who needs to do the work is a volunteer. That’s why your debugging kernel should include its symbols. To keep the symbols, add these lines to your kernel configuration: 454

Note: If you are looking for good and high quality web space to host and run your application check Lunarwebhost PHP Web Hosting services

code segment = base 0×0, limit 0xfffff, type 0×1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL=0 current process = 5 (syncer) interrupt mask = bio trap number = 12 panic: page fault ………………………………………………………………………………………. If you’re an inexperienced sysadmin, messages like this can turn your blood cold, but don’t fret yet. FreeBSD sometimes gives somewhat friendly messages that describe what’s wrong, which give you a specific place to start looking, or at least a term to Google. I’ve seen panics that give very specific instructions on kernel options that should be set to prevent their recurrence. Other panic messages, like this one, are much more puzzling. The only word that looks even vaguely familiar in this panic message is the fourth line from the bottom, where we see that the current process is something called “syncer”. Most people don’t know what the syncer is, and most of those who recognize it know better than to try to fix it. The mysterious panic is among the worst situations you can have in FreeBSD. Responding to a Panic If you get a system panic, the first thing to do is get a copy of the panic message. Since FreeBSD is no longer running at this point, the standard UNIX commands will not work the system won’t let you SSH in or out, and even simple commands like script(1) will not work. The console might be utterly locked up, or it could be in a debugger. In either event, you need the error message. The first time I received an error message like the preceding one, I scrambled for paper and pen. Eventually I found an old envelope and a broken stub of pencil, and crawled between the server rack and the rough brick wall. I balanced the six-inch black-and-white monitor that I’d dragged back there in one hand, while with my other hand I held the old envelope against the wall. Apparently I had a third hand to copy the panic message to the envelope, because it somehow got there. Finally, scraped and cramped, I slithered back out of the rack and victoriously typed the whole mess into an email. Surely the crack FreeBSD developers would be able to look at this garbage and tell me exactly what had happened. After all of this struggle, the initial response was quite frustrating: “Can you send a backtrace?” I’ve seen many, many messages to a FreeBSD mailing list reporting problems like this, and they always get this same response. Most of the people who send these messages are never heard from again, and I understand exactly how they feel. When you’ve been dealing with a server that crashes, or (worse) keeps crashing, the last thing you want to do is reconfigure it. The problem with the panic message on my envelope was that it only gave a tiny scrap of the story. It was so vague, in fact, that it was like describing a stolen car as “red, with a scratch on the fender.”If you don’t give the car’s make, model, and VIN number or license plate, you cannot expect the police to make much headway. Similarly, without more information from your crashing kernel, the FreeBSD developers can’t catch the criminal code. There’s a simple way around this problem, however: Set up your server to handle a panic before the 453

Note: If you are looking for good and high quality web space to host and run your application check Lunarwebhost Linux Web Hosting services

Chapter 20: System Crashes and Panics One of the nice things about FreeBSD is its stability; the only Blue Screen of Death is a screensaver. In fact, it was almost a year before I realized that a FreeBSD machine could crash for reasons other than bad hardware. FreeBSD can crash, or panic, but it allows you to recover from a panic fairly easily, so don’t, er, panic. You can even connect to the console remotely when the system is completely locked up and nonresponsive, and force a reboot. FreeBSD provides the tools you need to discover exactly what happened, and gives you extensive debugging information about the panic. (Even if you don’t know what to do with this information, you can submit a problem report and discuss the issue with the FreeBSD development team.) What Causes Panics? When does a system panic? Well, panicking is a choice that the kernel makes. If the system reaches a condition that it doesn’t know how to handle, or if it fails its own internal consistency checks, it will panic. If you’re using FreeBSD as a desktop, you can panic it by doing a variety of things, most of which are hardware related. For example, in older FreeBSDs, you could panic a laptop by pulling out certain PCMCIA cards without shutting the card down first. (That’s, fortunately, no longer the case.) Now, to panic a system, you pretty much have to do something wrong while logged in as root (or be running-current). It’s much more difficult to panic a server. Panics generally occur only if you have configured your kernel improperly, if you exhaust your system resources, or if you’ve tickled a previously unknown FreeBSD bug. The first two are pretty straightforward to fix, and are discussed in Chapters 4 and 18. The last is the most disturbing. FreeBSD is complex, and not even its royal-blood lineage or open-source development process can protect it from bugs. (Some people argue that no software is bug-free.) Fortunately, that heritage and development process give you the information to debug FreeBSD yourself, and the tools to provide the information necessary for other people to debug it. You might begin with a cryptic error code, but you’ll quickly learn that it means something to someone. Note If you’re not a programmer, don’t worry. By preparing your system to debug panics, you’ll help someone else to fix the problem. Hopefully, you’ll never need to debug your system, but if you do, you’ll be glad you have it set up properly! What Does a Panic Look Like? When a system panics, it stops running all programs. Instead of the usual console messages, the console displays a message much like this one: ………………………………………………………………………………………. Fatal trap 12: page fault while in kernel mode fault virtual address = 0×80c0153a fault code / supervisor write, page not present instruction pointer = 0×8:0xc015aa84 stack pointer = 0×10:0xc7377e7c frame pointer = 0×10:0xc7377e80 452

Note: If you are looking for good and high quality web space to host and run your application check Lunarwebhost Unix Web Hosting services

by a huge amount. 451

Note: If you are looking for good and high quality web space to host and run your application check Lunarwebhost Cheap Web Hosting services

………………………………………………………………………………………. Network Traffic Just for completeness, let’s look at a sample of how to measure network traffic. This assumes a 10Mbps network connection; if you’re using 100BaseT, multiply MaxBytes by 10. ………………………………………………………………………………………. Title[myhost.AbsoluteBSD.com.traffic]: myhost.AbsoluteBSD.com network traffic MaxBytes[myhost.AbsoluteBSD.com.traffic]: 125000 WithPeak[myhost.AbsoluteBSD.com.traffic]: wmy Target[myhost.AbsoluteBSD.com.traffic]: public@myhost.AbsoluteBSD.com Options[myhost.AbsoluteBSD.com.traffic]: growright, bits PageTop[myhost.AbsoluteBSD.com.traffic]:

Network traffic on myhost.AbsoluteBSD.com

………………………………………………………………………………………. Monitoring Non-BSD Systems MRTG can monitor any system that uses SNMP; it is a standard spoken by almost every network-equipment and operating-system vendor. To do so, all you need is the list of MIBs that the product supports and what they mean. Tables of MIB interpretations simple text files containing tables of how to read the vendor’s MIB tree are generally available from vendors. Each file starts something like this: ………………………………………………………………………………………. IPV6-MIB DEFINITIONS ::= BEGIN IMPORTS MODULE-IDENTITY, OBJECT-TYPE, NOTIFICATION-TYPE, mib-2, Counter32, Unsigned32, Integer32, Gauge32 FROM SNMPv2-SMI DisplayString, PhysAddress, TruthValue, TimeStamp, VariablePointer, RowPointer FROM SNMPv2-TC MODULE-COMPLIANCE, OBJECT-GROUP, NOTIFICATION-GROUP FROM SNMPv2-CONF Ipv6IfIndex, Ipv6Address, Ipv6AddressPrefix, Ipv6AddressIfIdentifier, Ipv6IfIndexOrZero FROM IPV6-TC; ………………………………………………………………………………………. Copy these definition files to a directory on your system. Next, set the environment variable MIBDIRS to point to that directory, and snmptranslate will recognize them. You can even install SNMP on your NT systems; this makes them easy to monitor, at a fraction of the cost of commercial systems. The only difference between MRTG and a commercial system is that you must know what you’re doing to use MRTG. I highly recommend Garth Williams “SNMP for the Public Community” site (http://www.wtcs.org/snmp4tpc/) for SNMP on other platforms. Be warned in advance: To call Microsoft’s implementation of SNMP “skeletal” would leave you without an adequate description of its error messages. Windows 2000 and XP are better than NT, but not 450

Note: If you are looking for good and high quality web space to host and run your application check Lunarwebhost Cheap Web Hosting services

LegendO[myhost.AbsoluteBSD.com.cpu]: System CPU Legend1[myhost.AbsoluteBSD.com.cpu]: User Processor Usage Legend2[myhost.AbsoluteBSD.com.cpu]: System Processor Usage Legend3[myhost.AbsoluteBSD.com.cpu]: Maximal 5 Minute User Processor Usage Legend4[myhost.AbsoluteBSD.com.cpu]: Maximal 5 Minute System Processor Usage Options[myhost.AbsoluteBSD.com.cpu]: growright, gauge PageTop[myhost.AbsoluteBSD.com.cpu]:

CPU usage for myhost.AbsoluteBSD.com

………………………………………………………………………………………. Swap Usage Before you can measure swap space, you’ll need to know how much swap your system has. You can get that information by doing an snmpwalk to get the total swap available, or snmpwalk hostname community .1.3.6.1.4.1.2021.4.3. Put that value in the MaxBytes space, and under the PageTop, so it will show up on your graph. ………………………………………………………………………………………. Title[myhost.AbsoluteBSD.com.swap]: myhost.AbsoluteBSD.com Available Swap YLegend[myhost.AbsoluteBSD.com.swap]: swap WithPeak[myhost.AbsoluteBSD.com.swap]: wmy MaxBytes[myhost.AbsoluteBSD.com.swap]: 128000 Target[myhost.AbsoluteBSD.com.swap]: .1.3.6.1.4.1.2021.4.4.0&.1.3.6.1.4.1.2021.4.4.0:public@myhost.AbsoluteBSD.com ShortLegend[myhost.AbsoluteBSD.com.swap]: Swap LegendI[myhost.AbsoluteBSD.com.swap]: Available Swap LegendO[myhost.AbsoluteBSD.com.swap]: Available Swap Legend1[myhost.AbsoluteBSD.com.swap]: Available Swap Legend2[myhost.AbsoluteBSD.com.swap]: Available Swap Legend3[myhost.AbsoluteBSD.com.swap]: Maximal 5 Minute Available Swap Legend4[myhost.AbsoluteBSD.com.swap]: Maximal 5 Minute Available Swap Options[myhost.AbsoluteBSD.com.swap]: growright, gauge PageTop[myhost.AbsoluteBSD.com.swap]:

Available Swap (out of 128M) for myhost.AbsoluteBSD.com

………………………………………………………………………………………. Disk Partition Measuring the amount of space used on a disk partition is similarly easy, though you’ll need to know the MIB index for the disk partition first. (See the examples in “Index Numbers” earlier in this chapter.) Be sure to give the correct size in the PageTop space. ………………………………………………………………………………………. Target[myhost.AbsoluteBSD.com.root]:.1.3.6.1.4.1.2021.9.1.9.1&.1.3.6.1.4.1.2021.9.1. 9.1:public@myhost.AbsoluteBSD.com Title[myhost.AbsoluteBSD.com.root]: Myhost.AbsoluteBSD.com Root partition % used MaxBytes[myhost.AbsoluteBSD.com.root]: 100 WithPeak[myhost.AbsoluteBSD.com.root]: wmy Suppress[myhost.AbsoluteBSD.com.root]: y LegendI[myhost.AbsoluteBSD.com.root]: % used Legend1[myhost.AbsoluteBSD.com.root]: % used YLegend[myhost.AbsoluteBSD.com.root]: percent used ShortLegend[myhost.AbsoluteBSD.com.root]: used Options[myhost.AbsoluteBSD.com.root]: gauge, growright PageTop[myhost.AbsoluteBSD.com.root]:

myhost Root partition (% of 128M) used

449

Note: If you are looking for good and high quality web space to host and run your application check Lunarwebhost Cheap Web Hosting services

Options[label]: option-names Options allow you to handle special cases. All options are specified on one line, after the target. Some good examples follow: growright By default, MRTG draws graphs from right to left. Use this option to make it draw graphs from left to right. bits This changes the graphs from measuring bytes to bits. Bits are not only more impressive, but they may also be more accurate, depending on what you’re measuring. For example, many people measure bandwidth in bits, not kilobytes. gauge SNMP generally retains information in counter form. MRTG subtracts the previous reading from the current reading to get the change in the last five minutes. Use the gauge option for SNMP MIBs that don’t change, such as disk capacity. unknaszero When a target is not reachable for any reason (including power failure or network problems), the system will assume the last known value for the charts. Whether or not this is more accurate is a matter of some controversy. This option causes MRTG to assume a value of zero when it cannot reach a target. MRTG Index Page The last bit of configuration we’ll need to do is to create a single HTML index page for our MRTG setup. MRTG includes an indexmaker tool that automagically does this for us. Just run this: ………………………………………………………………………………………. # indexmaker mrtg.cfg > index.html ………………………………………………………………………………………. You’ll probably want to edit this, but it’s a nice starting place. Sample MRTG Configurations This section contains some sample MRTG configurations. To use them, insert the correct hostname and community name. Also, note that the Target statements are supposed to be all on one line; the printed page simply isn’t wide enough to handle this at a type size you can see without a magnifying glass. In each example, you will need to edit the Target statement to give the proper hostname and community name, and you’ll probably want to edit the PageTop setting so your graph displays the correct information for your host. User and System CPU Usage This first sample compares user CPU usage with system CPU usage as percentages. (Note that we use the gauge option.) ………………………………………………………………………………………. Title[myhost.AbsoluteBSD.com.cpu]: myhost.AbsoluteBSD.com CPU usage YLegend[myhost.AbsoluteBSD.com.cpu]: CPU usage WithPeak[myhost.AbsoluteBSD.com.cpu]: wmy MaxBytes[myhost.AbsoluteBSD.com.cpu]: 100 Target[myhost.AbsoluteBSD.com.cpu]:.1.3.6.1.4.1.2021.11.9.0&.1.3.6.1.4.1.2021.11.10. 0:public@myhost.AbsoluteBSD.com ShortLegend[myhost.AbsoluteBSD.com.cpu]: CPU LegendI[myhost.AbsoluteBSD.com.cpu]: User CPU 448

Note: If you are looking for good and high quality web space to host and run your application check Lunarwebhost Cheap Web Hosting services