Enthused by how Pugs impressed their professor in the last class, Shweta wanted to do so too. And there was soon an opportunity: finding out where the output of printk had gone. So, as soon as she entered the lab, she grabbed the best system, logged in, and began work. Knowing her professor well, she realised that he would have dropped a hint about the possible solution in the previous class itself. Going over what had been taught, she remembered the error output demonstration from insmod vfat.ko
— running dmesg | tail
. She immediately tried that, and found the printk
output there.
But how did it come to be here? A tap on her shoulder roused her from her thoughts. “Shall we go for a coffee?” proposed Pugs.
“But I need to –“.
“I know what you’re thinking about,” interrupted Pugs. “Let’s go, I’ll explain you all about dmesg.”
Kernel message logging
Over coffee, Pugs began his explanation.
As far as parameters are concerned, printf
and printk
are the same, except that when programming for the kernel, we don’t bother about the float formats %f
, %lf
and the like. However, unlike printf
, printk
is not designed to dump its output to some console.
In fact, it cannot do so; it is something in the background, and executes like a library, only when triggered either from hardware-space or user-space. All printk
calls put their output into the (log) ring buffer of the kernel. Then, the syslog
daemon running in user-space picks them up for final processing and redirection to various devices, as configured in the configuration file /etc/syslog.conf
.
You must have observed the out-of-place macro KERN_INFO
, in the printk
calls, in the last article. That is actually a constant string, which gets concatenated with the format string after it, into a single string. Note that there is no comma (,) between them; they are not two separate arguments. There are eight such macros defined in linux/kernel.h
in the kernel source, namely:
#define KERN_EMERG "<0>" /* system is unusable */ #define KERN_ALERT "<1>" /* action must be taken immediately */ #define KERN_CRIT "<2>" /* critical conditions */ #define KERN_ERR "<3>" /* error conditions */ #define KERN_WARNING "<4>" /* warning conditions */ #define KERN_NOTICE "<5>" /* normal but significant condition */ #define KERN_INFO "<6>" /* informational */ #define KERN_DEBUG "<7>" /* debug-level messages */
Now depending on these log levels (i.e., the first three characters in the format string), the syslog
user-space daemon redirects the corresponding messages to their configured locations. A typical destination is the log file /var/log/messages
, for all log levels. Hence, all the printk
outputs are, by default, in that file. However, they can be configured differently — to a serial port (like /dev/ttyS0
), for instance, or to all consoles, like what typically happens for KERN_EMERG
.
Now, /var/log/messages
is buffered, and contains messages not only from the kernel, but also from various daemons running in user-space. Moreover, this file is often not readable by a normal user. Hence, a user-space utility, dmesg
, is provided to directly parse the kernel ring buffer, and dump it to standard output. Figure 1 shows snippets from the two.
Kernel-specific GCC extensions
Shweta, frustrated since she could no longer show off as having discovered all these on her own, retorted, “Since you have explained all about printing in the kernel, why don’t you also tell me about the weird C in the driver as well — the special keywords __init
, __exit
, etc.”
These are not special keywords. Kernel C is not “weird C”, but just standard C with some additional extensions from the C compiler, GCC. Macros __init
and __exit
are just two of these extensions. However, these do not have any relevance in case we are using them for a dynamically loadable driver, but only when the same code gets built into the kernel. All functions marked with __init
get placed inside the init
section of the kernel image automatically, by GCC, during kernel compilation; and all functions marked with __exit
are placed in the exit
section of the kernel image.
What is the benefit of this? All functions with __init
are supposed to be executed only once during bootup (and not executed again till the next bootup). So, once they are executed during bootup, the kernel frees up RAM by removing them (by freeing the init
section). Similarly, all functions in the exit
section are supposed to be called during system shutdown.
Now, if the system is shutting down anyway, why do you need to do any cleaning up? Hence, the exit
section is not even loaded into the kernel — another cool optimisation. This is a beautiful example of how the kernel and GCC work hand-in-hand to achieve a lot of optimisation, and many other tricks that we will see as we go along. And that is why the Linux kernel can only be compiled using GCC-based compilers — a closely knit bond.
The kernel function’s return guidelines
While returning from coffee, Pugs kept praising OSS and the community that’s grown around it. Do you know why different individuals are able to come together and contribute excellently without any conflicts, and in a project as huge as Linux, at that? There are many reasons, but most important amongst them is that they all follow and abide by inherent coding guidelines.
Take, for example, the kernel programming guideline for returning values from a function. Any kernel function needing error handling, typically returns an integer-like type — and the return value again follows a guideline. For an error, we return a negative number: a minus sign appended with a macro that is available through the kernel header linux/errno.h
, that includes the various error number headers under the kernel sources — namely, asm/errno.h
, asm-generic/errno.h
, asm-generic/errno-base.h
.
For success, zero is the most common return value, unless there is some additional information to be provided. In that case, a positive value is returned, the value indicating the information, such as the number of bytes transferred by the function.
Kernel C = pure C
Once back in the lab, Shweta remembered their professor mentioning that no /usr/include
headers can be used for kernel programming. But Pugs had said that kernel C is just standard C with some GCC extensions. Why this conflict?
Actually this is not a conflict. Standard C is pure C — just the language. The headers are not part of it. Those are part of the standard libraries built in for C programmers, based on the concept of reusing code.
Does that mean that all standard libraries, and hence, all ANSI standard functions, are not part of “pure” C? Yes, that’s right. Then, was it really tough coding the kernel?
Well, not for this reason. In reality, kernel developers have evolved their own set of required functions, which are all part of the kernel code. The printk
function is just one of them. Similarly, many string functions, memory functions, and more, are all part of the kernel source, under various directories like kernel
, ipc
, lib
, and so on, along with the corresponding headers under the include/linux
directory.
“Oh yes! That is why we need to have the kernel source to build a driver,” agreed Shweta.
“If not the complete source, at least the headers are a must. And that is why we have separate packages to install the complete kernel source, or just the kernel headers,” added Pugs.
“In the lab, all the sources are set up. But if I want to try out drivers on my Linux system in my hostel room, how do I go about it?” asked Shweta.
“Our lab has Fedora, where the kernel sources are typically installed under /usr/src/kernels/<kernel-version>
, unlike the standard /usr/src/linux
. Lab administrators must have installed it using the command-line yum install kernel-devel
. I use Mandriva, and installed the kernel sources using urpmi kernel-source
,” replied Pugs.
“But I have Ubuntu,” Shweta said.
“Okay! For that, just use apt-get
utility to fetch the source — possibly apt-get install linux-source
,” replied Pugs.
Summing up
The lab session was almost over when Shweta suddenly asked, out of curiosity, “Hey Pugs, what’s the next topic we are going to learn in our Linux device drivers class?”
“Hmm… most probably character drivers,” threw back Pugs.
With this information, Shweta hurriedly packed her bag and headed towards her room to set up the kernel sources, and try out the next driver on her own. “In case you get stuck, just give me a call,” smiled Pugs.
Dude you are a rock start…
awesome post man!!
Thanks dude :)
that’s great pugs !!! just have few things to clear. 1. I don’t have the /var/log.messages folder in my ubuntu 12.04. Where I find the same in ubuntu? 2. Currently I run on 3.2.0-27-generic-pae (uname -r) kernel. I want to know what is this generic-pae after the kernel version. Second thing when I went to /usr/src directory I din’t find the linux folder which contains the whole source code. But I found only the header file as the name linux-headers-3.2.0-27-generic-pae folder. Now I wanted to install the source code. So as per your advice is used “sudo apt-get install linux-source”. It downloaded and installed the code with this folder name “linux-source-3.2.0”. But here I only got the 3.2.0 but not 3.2.0-27. What is the 27 for? Should not I use the same source, the kernel I am running for development? I am sorry asked so many questions in a messy manner. Sorry for my language….
It is /var/log/messages & is a file.
The downloaded kernel source version is same as the running kernel, except the EXTRAVERSION which is updated by the disro guys – in your case by ubuntu
Regards
Pugs
there is no messages file or folder in ubuntu 12.04
/var/log/messages
Check out, if you have /etc/syslog.conf file. If yes, open that & check the log file’s name.
I don’t have /etc/syslog.conf file
Check out the “SEE ALSO” section at the bottom of ‘man 3 syslog’, for some configuration file name. That should be the one, you should look for in your system.
Thank for reply….
instead of syslog.conf i found rsyslog.conf … both are same?
Okay, so what the log file name mentioned there?
http://bdup.wordpress.com/2012/03/15/where-is-varlogmessages-on-ubuntu-12-04-b1-precise/
Thanks for that info.
Thanks for sharing this info.
Why don’t you use /var/log/syslog in ubuntu …. I use it. It is same as /var/log/messages.
1. Use /var/log/syslog in ubuntu 12.04.
2. “generic-pae” kernel is ubuntu’s customized kernel with PAE (Physical Address Extension) enabled for generic purpose i.e for Desktop, Laptop but not for server (ubuntu server kernels comes with different server-oriented optimizations enabled). PAE enables a 32-bit kernel to support more than 4 GiB of RAM.
3. Yes, you can use 3.2.0 source for your development but in that case you need to build the kernel and reboot into that new kernel. Forget it. Just to build drivers, you can use “linux-headers-3.2.0-27-generic-pae” . e.g,
$ make -C /usr/src/linux-headers-3.2.0-27-generic-pae -M `pwd` modules
Excellent part explained more about basics. Enjoyed reading :)
Thanks for reading & appreciating.
Congratulations on this great article series! It explains so many things that the standard reference books only mention in passing, e.g. kernel-specific GCC extensions. Why don’t you expand the article series into a book? I would buy it immediately! Keep up the good work.
Thanks for the appreciation and the motivation for converting these articles into a book. Hmmm! That seems to be something to think about.
You are a true teacher man. You answered every arising question in my mind through your article as if you were sitting next beside me reading my mind. Great.
Thanks for your high appreciation.
Thank you Sir. Really really great work :-)
Thanks for your appreciation.
how to install kernel in fedora 19 and which tar file best to download?
how to install kernel in fedora 19 and which tar file best to download?
That’s one of the most cumbersome thing to do – installing kernel in fedora. After you download the tar, you need to apply lot of patches, in particular order to get the closest possible source code they have built & supplied for the distro. If you are interested in playing with the kernel & related stuff, I suggest you to change your distro to say Mageia, Ubuntu, etc.
if i install opensuse 12.3 which kernel kernel version should i download?n Is it workable in VMWARE?
Typically, all these should work with vmware. Moreover, most distros come with the kernel sources package of the same kernel version as the one running under that distro. So you just install that, and start playing with that first. Once comfortable with that, then download any other later kernel version to upgrade.
thnx sir for reply
The author+girlfriend dialogue to explain device drivers seems to be taken by the author as “cool” but looks very stupid as you read. The author is striving hard to impress and build up the interest of the reader through this technique, but pathetically fails! To try something like this one needs to be a prolific writer, otherwise it usually turns out to be lame as this one. This stupid effort of being “cool” is actually a distraction to the reader. The editor of this site moderating the articles seriously needs some help — read LinuxJournal if you want to know about maintaining standards. The whole series is spoiled because of this stupidity! If you want to take this as a constructive feedback, stop striving to act “cool” or to be “Erich Segal” and better focus on what you want to say on the technical plane.
Help yourself, or you are never going to have quality audience!
And what’s a quality audience by your standards ? Read all the comments bellow and then read your comment again, and see which writing looks more stupid.
Hi,sir Great explanation every bit of doubt clarified.
Thanks a million.
expect more articles in device drivers like core drivers i2c ,spi would be a great help sir.
Thanks for reading & appreciating the article. Though not yet written any on I2C & SPI, there are many others, you may read at http://sysplay.in/blog/linux-device-drivers/
Hi
I am using linux arch linux. I have installed the linux headers using pacman,
Can you please guide me in knowing where the headers will be installed as I dont find it in usr/src/
The headers are in /lib/modules/`uname -r`/build
Superb explanation…