Sunday, May 21, 2006
Modifying a Laptop BIOS for Fun and Profit
NOTE: My blog as moved. Please visit my new blog where I talk about audio apps, diy synthesizers, and CNCs. You can also follow me on twitter.
In today's hack, we will analyze and modify a laptop BIOS image for fun and profit. Note that this is stupidly dangerous: a bad flash and we'll end up with a doorstop, or we'll have to remove the chip and reflash it manually. But in either case, the outcome can be fun.
# strings bios.com | more
BIOS
V8.10 3020CT
BM&#
wwww
wwwwww
www"www
wwwwp
wwwww
wwwww
wwwww
=PAu
=THu
GG&;E
_^Yt
... (output truncated)
This looks at least somewhat promising: there are ASCII strings in the file. Often files are compressed, encrypted, or otherwise modified to hide the original data. So, what do we see:
BIOS - probably in the file header to identify itself as, well, a BIOS.
v8.10 3020ct - BIOS/laptop version string
From a first glance, we don't see anything else that is readable. However, let's think about a couple things:
What we want to do now, is extract the payload into it's own file for modifications (using Paint). We need two peices of information: where does this image start (the offset), and it's size. Using our trusty hex editor, we find the BM&# magic appears at 0x100 (it's pretty obvious, we don't even have to search for it). For the size, let's just extract too much information, since another hidden fact of almost all image editing software is that they ignore data that comes after the end of a valid image (they read the data they need, and that is it). So we'll use a large number, such as 0x5900. We use 0x5900 since, if you look at the format of the bios file, around 0x6000 there is an obvious change in structure (a large series of null bytes all of a sudden into a stream of bytes with no apparent structure). We use the following C program to extract the data:
(save as ripimg.c)
#include <string.h>
// these are just guesses ;)
#define IMG_START 0x100
#define IMG_SIZE 0x5900
typedef unsigned char uint8_t;
int main(int argc, char **argv)
{
if (argc != 3) {
printf("Usage: %s <bios file> <output image>\n", argv[0]);
}
uint8_t *buf;
FILE *in = fopen(argv[1], "rb");
FILE *out = fopen(argv[2], "wb");
buf = (uint8_t *) malloc(sizeof(uint8_t)*4000);
fseek(in, IMG_START, SEEK_SET);
int done = IMG_SIZE;
while (done > 4000) {
fread(buf, sizeof(uint8_t), 4000, in);
fwrite(buf, sizeof(uint8_t), 4000, out);
done-= 4000;
}
if (done) {
fread(buf, sizeof(uint8_t), done, in);
fwrite(buf, sizeof(uint8_t), done, out);
}
fclose(in);
fclose(out);
}
To compile, we run:
gcc -o ripimg ripimg.c
And hope it compiles cleanly. Now, let's run this on our bios file. But first, let's make a backup: the last thing we want to do is inadvertently change a byte of the file, leading to a bad flash. So at every point we will diff our file in use against a clean backup to ensure we havn't mistakenly modified the bios. To run it, we execute:
./ripimg biosfile img.out
Which writes img.out to the same directory. Opening img.out in an image editor, we find out.. SUCCESS! We've extracted the file!
#include <stdio.h>
#include <string.h>
typedef unsigned char uint8_t;
// make no mistakes ;)
#define FILE_SIZE 8998
#define IMG_START 0x100
int main(int argc, char **argv)
{
uint8_t *buf;
FILE *in;
FILE *out;
int check;
buf = (uint8_t *) malloc(sizeof(uint8_t)*FILE_SIZE);
out = fopen(argv[1], "r+");
in = fopen(argv[2], "r");
fseek(out, IMG_START, SEEK_SET);
check = fread(buf, sizeof(uint8_t), FILE_SIZE, in);
if (check != FILE_SIZE) {
printf("Error reading image\n");
exit(1);
fclose(in); fclose(out);
}
fwrite(buf, sizeof(uint8_t), FILE_SIZE, out);
fclose(in);
fclose(out);
}
To run this, we execute it by:
./injectimg biosfile edited.bmp
The file 'biosfile' now has the edited image embedded.
#include <stdio.h>
#include <string.h>
typedef unsigned char uint8_t;
#define FILE_SIZE 8998
#define IMG_START 0x100
int main(int argc, char **argv)
{
FILE *fp1;
FILE *fp2;
uint8_t buf1, buf2;
int count = 0;
int diff;
fp1 = fopen(argv[1], "r");
fp2 = fopen(argv[2], "r");
while (!feof(fp1)) {
fread(&buf1, sizeof(uint8_t), 1, fp1);
fread(&buf2, sizeof(uint8_t), 1, fp2);
if (buf1 != buf2) {
if (count > IMG_START && count < IMG_START +FILE_SIZE) {
} else {
diff = ftell(fp1);
printf("Difference at %08x!!\n", diff);
}
}
count++;
}
fclose(fp1);
fclose(fp2);
}
To run, just pass as arguments the name of our original bios file, and the modified file. If the program does not output any text then it has run successfully and we only modified the image portion. Otherwise STOP, and restart the tutorial. The last thing to do is to ensure that our checkimg program did not itself modify our image, so we run 'diff' to make that assurance:
# diff biosfile biosfile-2
..
...
..
Oh, dang, it doesn't work. CRC failure. Toshiba's one step ahead of us.
The reason Attempt 2 failed is because, prior to loading the BIOS and calculating the CRC, the application detects that we are in Virtual Mode, assumes we're running it in a DOS box in Windows, and barfs because of that. We could get around this check by editing the application, but with that level of understanding we'd be able to just remove the CRC check altogether - which is what we're going to do.
On your other computer, open chgbiosa.exe in HT, and change the mode to "mz/image". Our cursor is now at the starting point of execution for the program. On the laptop, run "debug chgbiosa.exe".
We will manually step through execution of the flashing program using the "proceed" command in DEBUG, shortcut "p". The "proceed" command is like GDB's "next" command, in that it steps over subroutines as opposed to into them. The application changes position of the cursor, so be ready. Keep going until we're executing the line:
0E53:46BA E857FD CALL 4414
Note: Details aside, there is an offset difference of 0x200 between DEBUG.COM and HT's instruction addresses. It has to do with virtual addresses and segments in 16-bit mode.
Hit p again on this line, and in the middle of the screen you see "Input BIOS file name:". Obviously 4414 is the function that prints the prompt and asks for a filename, so enter our bios name.
It's a good idea to hit enter a number of times to push the cursor to the bottom of the screen whenever the application repositions it, effectively eliminating the annoying double text.
Anyways, we keep stepping until the line:
0E53:46DD E8F9C9 CALL 12D9
Hit 'p' once more, and we see the error printed out "CRC Error". So we've found the subroutine that checks the CRC! Let's now use the 'trace' command to step into that subroutine and see what's going on. So, exit the debug session, and again execute "debug chgbiosa.exe". To save time we can execute right up to that call into the CRC subroutine, by giving the command:
-G 46DD
That should position us at the line that calls the CRC routine. Now use the 't' command to enter the subroutine. You should be looking at the code at offset 12d9 in HT - this is the start of the CRC routine, and where we're currently positioned in DEBUG. Browse the code until you see the two 'ret' commands at 1448 and 144a. You should be able to see a pattern with the code up until that point: we test a word in memory against specific bits, and if it's not zero we iterate over a portion of the loaded BIOS file. That's all that happens until 0x1437 which is different as you see below:
00001430 ac lodsb
00001431 e81700 call 0x144b
00001434 59 pop cx
00001435 e2f8 loop 0x142f <--- last looping command
00001437 1f pop ds
00001438 3b1e7801 cmp bx, [0178] <---- compare BX against a word in memory - this is "test
computed vs stored" CRC values
0000143c 7503 jnz 0x1441 <---- if different, jump to 0x1441
0000143e eb09 jmp 0x1449 <---- else 0x1449
00001440 90 nop
00001441 bef402 mov si, 0x2f4
00001444 e869f2 call 0x6b0 <---- print "CRC Error"
00001447 f9 stc <---- set the carry flag to indicate CRC failure
00001448 c3 ret <---- return
00001449 f8 clc <---- clear carry flag to indicate success
0000144a c3 ret <---- return
All we're to do now, is modify the flow of execution so that we never return failure. So we modify the two bytes at 143c to two nop's, effectively forcing the flow of execution to return success. In HT, hit F4 to switch to Edit mode, and change the two bytes at 143c to both be equal to 0x90. The resulting assembly should look like the following:
00001437 1f pop ds
00001438 3b1e7801 cmp bx, [0178]
0000143c 90 nop <-- changed to NOP's
0000143d 90 nop
0000143e eb09 jmp 0x1449
00001440 90 nop
00001441 bef402 mov si, 0x2f4
00001444 e869f2 call 0x6b0
00001447 f9 stc
00001448 c3 ret
00001449 f8 clc
0000144a c3 ret
Hit F4 to save, copy the new application to the laptop, and run (not in DEBUG!)! Success? You bet!
Read Post..
In today's hack, we will analyze and modify a laptop BIOS image for fun and profit. Note that this is stupidly dangerous: a bad flash and we'll end up with a doorstop, or we'll have to remove the chip and reflash it manually. But in either case, the outcome can be fun.
Technologies used/Requirements
- Laptop (I'm using a Toshiba Portege 3020ct)
- hex editor (I use ht)
- simple image editor (I used MS Paint - I had a reason!!)
- DEBUG.COM from The FreeDOS Project
- Linux
DISCLAIMER #2
In some comments it appears people are thinking this is a generic method to modify any bios. It's not. It's the steps I had to take to modify mine. Yours will be different. Try using the same steps as me, but don't be suprised if you run into horrible problems such as compressed data. And again, don't do it unless your willing to brick it. I had a little faith since I did every step of the hack myself, but to you I'm just someone writing whatever he wants on a blog. Good luck :).DISCLAIMER
Don't follow this hack! You could destroy your laptop! Only risk what you can afford to lose. As well, don't be fooled by the success of my hack.. There are probably BIOS's out there that do their CRC check internally on every boot, and if the value is incorrect it won't boot up.. and this is something you'd only learn of after flashing the bad image! As well, many BIOS images when downloaded are probably compressed, leading to a much harder chase for something as trivial as changing the splashscreen. You've been warned.Overview
When I turn on my laptop, I'm greeted with a splashscreen advertising TOSHIBA. I'd like to change it to something custom to show off my elite-ness, so I've obtained a BIOS image from the Toshiba support page (available from here.Finding the Picture
The BIOS image represents a binary file that is completely foreign to us: we have no knowledge at this time of it's structure. A good first step in file format analysis is the most simplest: strings. strings is a command line utility that analyzes a byte stream and prints to stdout any ASCII strings it finds of length 4 or more characters. So, we run strings:# strings bios.com | more
BIOS
V8.10 3020CT
BM&amp;amp;amp;amp;amp;amp;amp;amp;amp;#
wwww
wwwwww
www"www
wwwwp
wwwww
wwwww
wwwww
=PAu
=THu
GG&;E
_^Yt
... (output truncated)
This looks at least somewhat promising: there are ASCII strings in the file. Often files are compressed, encrypted, or otherwise modified to hide the original data. So, what do we see:
BIOS - probably in the file header to identify itself as, well, a BIOS.
v8.10 3020ct - BIOS/laptop version string
From a first glance, we don't see anything else that is readable. However, let's think about a couple things:
- The splashscreen is displayed immediately upon boot
- It's embedded in the BIOS, not a lot of room for image decompression library or anything similiar
What we want to do now, is extract the payload into it's own file for modifications (using Paint). We need two peices of information: where does this image start (the offset), and it's size. Using our trusty hex editor, we find the BM&# magic appears at 0x100 (it's pretty obvious, we don't even have to search for it). For the size, let's just extract too much information, since another hidden fact of almost all image editing software is that they ignore data that comes after the end of a valid image (they read the data they need, and that is it). So we'll use a large number, such as 0x5900. We use 0x5900 since, if you look at the format of the bios file, around 0x6000 there is an obvious change in structure (a large series of null bytes all of a sudden into a stream of bytes with no apparent structure). We use the following C program to extract the data:
(save as ripimg.c)
#include <string.h>
// these are just guesses ;)
#define IMG_START 0x100
#define IMG_SIZE 0x5900
typedef unsigned char uint8_t;
int main(int argc, char **argv)
{
if (argc != 3) {
printf("Usage: %s <bios file> <output image>\n", argv[0]);
}
uint8_t *buf;
FILE *in = fopen(argv[1], "rb");
FILE *out = fopen(argv[2], "wb");
buf = (uint8_t *) malloc(sizeof(uint8_t)*4000);
fseek(in, IMG_START, SEEK_SET);
int done = IMG_SIZE;
while (done > 4000) {
fread(buf, sizeof(uint8_t), 4000, in);
fwrite(buf, sizeof(uint8_t), 4000, out);
done-= 4000;
}
if (done) {
fread(buf, sizeof(uint8_t), done, in);
fwrite(buf, sizeof(uint8_t), done, out);
}
fclose(in);
fclose(out);
}
To compile, we run:
gcc -o ripimg ripimg.c
And hope it compiles cleanly. Now, let's run this on our bios file. But first, let's make a backup: the last thing we want to do is inadvertently change a byte of the file, leading to a bad flash. So at every point we will diff our file in use against a clean backup to ensure we havn't mistakenly modified the bios. To run it, we execute:
./ripimg biosfile img.out
Which writes img.out to the same directory. Opening img.out in an image editor, we find out.. SUCCESS! We've extracted the file!
Modifying the image
The reason I used Paint to modify the image is that I wanted to ensure that the file format did not change in any way, shape, or form. I don't expect that the bios would be able to parse the embedded bitmap and act appropriately if a bitmap of anything but the exact same specifications were to appear. As well, we need the bitmap to be the exact same size as the embedded bitmap so that our modification does not overwrite any of the other data stored in the BIOS. So, we open and edit the bitmap to our liking within Paint, and save the final image as edited.bmp. Looking at edited.bmp, we see it's size is only 8998 bytes. That's because we had originally ripped out too much data to ensure we had read the entire bitmap.Injecting the image
Injecting is basically the inverse operation to ripping the image from the BIOS file. The biggest difference is that we have to be much more careful with respect to what we're doing, since we ARE modifying the bios image. Once we're done, we'll do some tests to (hopefully) ensure that we have changed the bitmap and only the bitmap. The injection code is as follows:#include <stdio.h>
#include <string.h>
typedef unsigned char uint8_t;
// make no mistakes ;)
#define FILE_SIZE 8998
#define IMG_START 0x100
int main(int argc, char **argv)
{
uint8_t *buf;
FILE *in;
FILE *out;
int check;
buf = (uint8_t *) malloc(sizeof(uint8_t)*FILE_SIZE);
out = fopen(argv[1], "r+");
in = fopen(argv[2], "r");
fseek(out, IMG_START, SEEK_SET);
check = fread(buf, sizeof(uint8_t), FILE_SIZE, in);
if (check != FILE_SIZE) {
printf("Error reading image\n");
exit(1);
fclose(in); fclose(out);
}
fwrite(buf, sizeof(uint8_t), FILE_SIZE, out);
fclose(in);
fclose(out);
}
To run this, we execute it by:
./injectimg biosfile edited.bmp
The file 'biosfile' now has the edited image embedded.
Verifying the file
This is a laptop we're talking about, and we're risking it all just to change a splashscreen. That's sort of stupid. So let's be as safe as possible. Make another backup of this bios image (biosfile-2, let's call it). Now, we'll run a program to verify that the we modified no bytes other than those starting at 0x100 and ending at 0x100 + 8998 (the bitmap size). The code is as follows:#include <stdio.h>
#include <string.h>
typedef unsigned char uint8_t;
#define FILE_SIZE 8998
#define IMG_START 0x100
int main(int argc, char **argv)
{
FILE *fp1;
FILE *fp2;
uint8_t buf1, buf2;
int count = 0;
int diff;
fp1 = fopen(argv[1], "r");
fp2 = fopen(argv[2], "r");
while (!feof(fp1)) {
fread(&buf1, sizeof(uint8_t), 1, fp1);
fread(&buf2, sizeof(uint8_t), 1, fp2);
if (buf1 != buf2) {
if (count > IMG_START && count < IMG_START +FILE_SIZE) {
} else {
diff = ftell(fp1);
printf("Difference at %08x!!\n", diff);
}
}
count++;
}
fclose(fp1);
fclose(fp2);
}
To run, just pass as arguments the name of our original bios file, and the modified file. If the program does not output any text then it has run successfully and we only modified the image portion. Otherwise STOP, and restart the tutorial. The last thing to do is to ensure that our checkimg program did not itself modify our image, so we run 'diff' to make that assurance:
# diff biosfile biosfile-2
Flashing the Image
Alright! We've successfully modified the BIOS file, let's flash it and see the results! Enter a native DOS command prompt, run the chgbiosa.exe utility, and provide the filename of our newly modified BIOS file and.....
...
..
Oh, dang, it doesn't work. CRC failure. Toshiba's one step ahead of us.
File CRC
If we look at the bios file, it appears that the first 16 bytes or so is a header that does not get flashed. Assumably the CRC checksum is embedded in this header. Our problem is not finding the checksum in the BIOS, but rather.. how do we compute the proper checksum? There are a number of options, and I tried a number of different ideas before succeeding. I'll outline them below:Attempt 1
The use of the term "CRC" makes me believe that maybe they're using a standard CRC16 checksum algorithm. If we assume this is true, the only thing we need to know is the start/ending offsets for the portion that is CRC'd. I'm not going to walk you through this because it failed :), but what I did was I found the source code to a CRC routine, and wrote a brute force program that attempted to sum different portions of the valid bios file to obtain the required CRC magic. Like I said, it didn't work. After a couple days running 24/7 I decided to go for a nicer, smoother hack.Attempt 2
Attempt 2 also failed, but if it weren't for the target being a BIOS flashing utility it would have worked. Attempt 2 was to run the application inside DOSEMU, and to dump the entire memory region of the application before it exits (and after it has computed the CRC). Doing so with a valid BIOS file would return at least two positions in memory that would contain the correct 16 bit CRC value (the calculated value and the value loaded from the file). Then we'd do it with our modified file, look at the same offsets, and learn what the proper CRC value is for our modified BIOS.The reason Attempt 2 failed is because, prior to loading the BIOS and calculating the CRC, the application detects that we are in Virtual Mode, assumes we're running it in a DOS box in Windows, and barfs because of that. We could get around this check by editing the application, but with that level of understanding we'd be able to just remove the CRC check altogether - which is what we're going to do.
Attempt 3 - The Successful Attempt
Clever introductions aside, the program that saves the day is the version of DEBUG.COM that comes with FreeDOS. Using it, we can interactively debug the BIOS flashing utility from a native DOS shell (thus satisfying the application's VM checks). By interactively debugging on the laptop, while watching the assembly source on another computer, we should be able to determine where the application makes the fork in execution based on CRC pass/fail. So let's begin.Environment
Assumably, you aren't currently in a native DOS command shell on your laptop. Before rebooting, copy the files you need to a directory on your MSDOS partition. You will need:- the modified BIOS file
- the chgbiosa.exe application
- debug.com, from the FreeDOS project.
On your other computer, open chgbiosa.exe in HT, and change the mode to "mz/image". Our cursor is now at the starting point of execution for the program. On the laptop, run "debug chgbiosa.exe".
We will manually step through execution of the flashing program using the "proceed" command in DEBUG, shortcut "p". The "proceed" command is like GDB's "next" command, in that it steps over subroutines as opposed to into them. The application changes position of the cursor, so be ready. Keep going until we're executing the line:
0E53:46BA E857FD CALL 4414
Note: Details aside, there is an offset difference of 0x200 between DEBUG.COM and HT's instruction addresses. It has to do with virtual addresses and segments in 16-bit mode.
Hit p again on this line, and in the middle of the screen you see "Input BIOS file name:". Obviously 4414 is the function that prints the prompt and asks for a filename, so enter our bios name.
It's a good idea to hit enter a number of times to push the cursor to the bottom of the screen whenever the application repositions it, effectively eliminating the annoying double text.
Anyways, we keep stepping until the line:
0E53:46DD E8F9C9 CALL 12D9
Hit 'p' once more, and we see the error printed out "CRC Error". So we've found the subroutine that checks the CRC! Let's now use the 'trace' command to step into that subroutine and see what's going on. So, exit the debug session, and again execute "debug chgbiosa.exe". To save time we can execute right up to that call into the CRC subroutine, by giving the command:
-G 46DD
That should position us at the line that calls the CRC routine. Now use the 't' command to enter the subroutine. You should be looking at the code at offset 12d9 in HT - this is the start of the CRC routine, and where we're currently positioned in DEBUG. Browse the code until you see the two 'ret' commands at 1448 and 144a. You should be able to see a pattern with the code up until that point: we test a word in memory against specific bits, and if it's not zero we iterate over a portion of the loaded BIOS file. That's all that happens until 0x1437 which is different as you see below:
00001430 ac lodsb
00001431 e81700 call 0x144b
00001434 59 pop cx
00001435 e2f8 loop 0x142f <--- last looping command
00001437 1f pop ds
00001438 3b1e7801 cmp bx, [0178] <---- compare BX against a word in memory - this is "test
computed vs stored" CRC values
0000143c 7503 jnz 0x1441 <---- if different, jump to 0x1441
0000143e eb09 jmp 0x1449 <---- else 0x1449
00001440 90 nop
00001441 bef402 mov si, 0x2f4
00001444 e869f2 call 0x6b0 <---- print "CRC Error"
00001447 f9 stc <---- set the carry flag to indicate CRC failure
00001448 c3 ret <---- return
00001449 f8 clc <---- clear carry flag to indicate success
0000144a c3 ret <---- return
All we're to do now, is modify the flow of execution so that we never return failure. So we modify the two bytes at 143c to two nop's, effectively forcing the flow of execution to return success. In HT, hit F4 to switch to Edit mode, and change the two bytes at 143c to both be equal to 0x90. The resulting assembly should look like the following:
00001437 1f pop ds
00001438 3b1e7801 cmp bx, [0178]
0000143c 90 nop <-- changed to NOP's
0000143d 90 nop
0000143e eb09 jmp 0x1449
00001440 90 nop
00001441 bef402 mov si, 0x2f4
00001444 e869f2 call 0x6b0
00001447 f9 stc
00001448 c3 ret
00001449 f8 clc
0000144a c3 ret
Hit F4 to save, copy the new application to the laptop, and run (not in DEBUG!)! Success? You bet!
Finale
My heart definitely stopped during the reboot cycle after reflashing, but it certainly was rewarding to see my new custom bootscreen appear. I hope you've enjoyed the hack, and hopefully learnt something about reverse engineering. Here's my result, from ~half an hour of MS Paint-work:Read Post..
Friday, April 14, 2006
Microhack: XScreensaver as root
Not all hacks are life-changing applications. Sometimes the best hacks are those that are two lines long. Or, at least, it's something you've never considered before. That's why I will sometimes post microhacks - short, to the point, tweaks that give a nice result.
The first thing we need to do is to turn off our current wallpaper. If the script has any calls to Esetroot, or fbsetbg, or similiar, comment them out. You may also need to open your style config file (often found in /usr/share/fluxbox or /usr/share/commonbox) and comment out the call to fbsetbg or similiar.
As a final step, add the line below to the startup file:
/path/to/screensaver -root &
Restart and we're done!
Read Post..
Screensaver on X11 RootThis isn't a big deal - it's built into the xscreensaver package. All you have to do is add it into the X11 initialization scripts to run instead of a given background. In this example, if it matters, we're using fluxbox on Gentoo. If you're using a window manager that provides icons on the root window this may not work since the window manager is already overlaying the root window with it's own graphics.xscreensaver packagexscreensaver is a collection of screensavers for X11. A neat feature that most people ignore is that you can run them on the root window (the background of your desktop). Just, once the package is installed, run them with the argument '-root'. For example, if the package binaries reside in /usr/lib/misc/xscreensaver, then run:/usr/lib/misc/xscreensaver/anemone -root |
Running on Startup
Users running fluxbox most likely have a ~/.fluxbox directory that stores their user preferences. Open the file "startup" that exists in that directory - it is a script that is run during fluxbox's initialization.The first thing we need to do is to turn off our current wallpaper. If the script has any calls to Esetroot, or fbsetbg, or similiar, comment them out. You may also need to open your style config file (often found in /usr/share/fluxbox or /usr/share/commonbox) and comment out the call to fbsetbg or similiar.
As a final step, add the line below to the startup file:
/path/to/screensaver -root &
Restart and we're done!
Read Post..
Wednesday, April 12, 2006
MobiGM, First edition
NOTE: My blog as moved. Please visit my new blog where I talk about DIY electronics, audio apps and CNC machines. You can also follow me on twitter.
There is a lot of potential to add power to this application. With a little knowledge of PERL, you could create custom filters (including things impossible with GMail filters such as "only forward during xx hours of the day") and reactions to receiving emails.
Let's begin.
The first thing the script needs to do, is receive the atom feed for your GMail inbox. We will use GNU wget to obtain the feed. In it's most basic form, wget's syntax is:
wget <url>
However, we need to specify a few additional parameters, namely HTTP authentication username and password. As well, we want to force wget to always download to the same filename. It's normal action is to not overwrite files that already exist, so we have to override it. So, the syntax becomes:
wget -O <output file> <url> --http-user=<username> --http-password=<password>
For GMail, the parameters are as follows:
URL: https://mail.google.com/gmail/feed/atom
http-user : your.e.mail@gmail.com
http-password: your gmail password
Note that since the atom feed is on an https server, we are not passing our username/password in the clear.
For the article, let's force wget to save into a file called 'atomfeed'. Our command is now:
wget -O atomfeed https://mail.google.com/gmail/feed/atom --http-user=gmail.user@gmail.com --http-password=GMailPassword
Now that we have the atom feed, it is time to parse it.
Atom feeds are conceptually the same as RSS feeds, in case you have not heard of both. They are XML documents that contain a summary of information: ie, blog entries, news entries, or in this case, your unread messages in your inbox. The structure of the GMail atom feed is as follows:
#!/usr/bin/perl -w
use XML::Parser;
use Net::SMTP;
open(ATOM, "atomfeed") or die ("Failed to get atomfeed");
my $page;
#load the atom feed
while ($line =) {
$page = $page . $line;
}
close(ATOM);
my $curID = "";
my $curTitle = "";
my $curContents = "";
my $curAuthor = "";
my $curAuthorEmail = "";
my $elementtext;
my @context;
my @oldIDs;
my @newIDs;
my $parser = new XML::Parser();
The code's pretty self explanitory. We load the atom feed into the $page variable. For use with the parsing functions, we initialize five variables ($curID. $curTitle, $curContents, $curAuthor, $curAuthorEmail) used to keep track of the entry currently being parsed. As well we create variables to keep track of our parsing state ($elementtext and @context). Ignore the ID-related variables, we'll deal with them later.
Our next step is to use XML::Parser to parse the atom feed. This is accomplished by the following code:
$parser->setHandlers(
Start => \&startTag,
Char => \&parseTag,
End => \&endTag, );
$parser->parse($page);
What we've done is told the XML parser to use three supplied subroutines (passed by reference) as callbacks during parsing. The startTag routine will be called whenever the parser encounters a new tag. Likewise, the parseTag routine is called as we encounter text in each tag, and the endTag routine is called whenever the parser encounters the end of a tag. The final line of the snippet executes the parser, parsing the atom feed using our supplied functions.
We have to supply these subrountines, since they detail how we deal with the data we're parsing. Put these at the bottom of the perl script since they're separate subrountines.
sub startTag {
my ($p, $tag, %atts) = @_;
push @context, $tag;
}
sub parseTag {
my ($p, $text) = @_;
return if ($text !~ /\S/);
$text =~ s/^\s*//;
$text =~ s/\s*^//;
$elementtext .= $text;
}
sub endTag {
my ($p, $tag) = @_;
$parentElem = $context[-2];
if ($tag eq "id" && $parentElem eq "entry") {
$curID = $elementtext;
} elsif ($tag eq "title" && $parentElem eq "entry") {
$curTitle = $elementtext;
} elsif ($tag eq "summary" && $parentElem eq "entry") {
$curContents = $elementtext;
} elsif ($tag eq "email" && $parentElem eq "author") {
$curAuthorEmail = $elementtext;
} elsif ($tag eq "name" && $parentElem eq "author") {
$curAuthor = $elementtext;
} elsif ($tag eq "entry") {
push @newIDs, $curID;
my $old = 0;
sendMail("Subject:$curTitle\n\nFrom:$curAuthorEmail [$curAuthor]\n$curContents");
}
$elementtext = "";
pop @context;
}
The startTag and parseTag subroutines are trivial. We're using the @context array as a stack to keep track of our state within the parser. Each time we enter a tag and begin to parse it's children or data, we push the element name onto the stack. We use this in the endTag subroutine to determine what to do with the parsed text. The parseTag subroutine, which is run during the parsing of data nodes, appends whatever data is supplied to the routine to the $elementtext variable (which will be used in endTag.
The endTag subroutine looks at the current context in the node tree and saves the current $elementtext to the proper variable. Whenever we encounter the end of an entry tag, we send the email to the server. This is a dirty solution: a nicer solution would be to save all the entry data into a data structure and later deal with each entry saved (an exercise for the reader ;) ). The final thing the procedure does is pop the finished tag off the @context stack and clear the $elementtext buffer.
Note that if you want to change the format of the sent email, change the formatting in the sendMail function call, above.
sub sendMail {
my @message;
push @message, $_[0];
$smtp2 = Net::SMTP->new('smtp.server.ca');
$smtp2->mail('source.email@isp.ca');
$smtp2->to('cellphoneemail@provider.com');
$smtp2->data();
$smtp2->datasend(@message);
$smtp2->dataend();
$smtp2->quit;
}
sendMail just uses the Net::SMTP module to send an email. You need to provide a SMTP server, source email address, and your cellphone's email address.
The first step is to load all the old ID's into an array. Place the code snippet below after the variable declaration (first snippet) and before the parser initialization (second snippet)
#load the previous inbox ID's
open(OLDID, "ids.old");
while ($line =) {
chomp $line;
push @oldIDs, $line;
}
close(OLDID);
We open the file 'ids.old' (assumably saved by a previous run of the script), and load each line of the file into an array. We will modify endTag to look through this array before determining whether or not it should send the email address, by changing the function's code as follows:
sub endTag {
my ($p, $tag) = @_;
$parentElem = $context[-2];
if ($tag eq "id" &&amp; $parentElem eq "entry") {
$curID = $elementtext;
} elsif ($tag eq "title" &&amp; $parentElem eq "entry") {
$curTitle = $elementtext;
} elsif ($tag eq "summary" && $parentElem eq "entry") {
$curContents = $elementtext;
} elsif ($tag eq "email" &&amp; $parentElem eq "author") {
$curAuthorEmail = $elementtext;
} elsif ($tag eq "name" &&amp; $parentElem eq "author") {
$curAuthor = $elementtext;
} elsif ($tag eq "entry") {
push @newIDs, $curID;
my $old = 0;
#was email in oldids?
foreach $id (@oldIDs) {
if ($curID eq $id) {
$old = 1;
}
}
if ($old == 0) {
sendMail("Subject:$curTitle\n\nFrom:$curAuthorEmail [$curAuthor]\n$curContents");
}
}
$elementtext = "";
pop @context;
}
As you can see, we are now comparing each entry's id to the loaded list of old ID's. Only if the id is not in the list do we notify the cell phone of the waiting message. As well, we are saving every message ID (old or new) in the array @newIDs.
The final thing to do is to re-write the 'ids.old' file to contain the message ID for all emails in our current feed. This code is placed after the $parser->parse line, making it the last thing that happens in our main routine (and immediately above our subroutine code).
open(NEWID, ">ids.old");
foreach $id (@newIDs) {
print NEWID $id . "\n";
}
close(NEWID);
#!/bin/sh
wget -O atomfeed https://mail.google.com/gmail/feed/atom --http-user=gmail.user@gmail.com --http-password=gmailPassword
./mail.pl
Test this out. If it doesn't work, try adding debugging statements in the code to see where/why it fails.
host user # crontab -e
Then add a line to the crontab file such as:
0,15,30,45 * * * * /home/th0mas/mobigm/run.sh
This tells the server to run the script 0, 15, 30, and 45 minutes after every hour.
http://th0mas.xbox-scene.com/tuts/mobigm-1.tgz
Read Post..
What is it?
| MobiGM is a small utility that periodically checks your GMail inbox, and sends updates to a mobile phone via email. Certainly, you could just set up a couple filters with GMail's powerful filtering system to forward messages to your phone and be happy with that, but MobiGM allows you to dynamically react to what appears in your mailbox (which we will exploit in future projects). |
Technologies used/Requirements:
How does it work?
GMail users can subscribe to an atom feed that contains unread emails in their inbox. We will write a simple script to download the atom feed, parse it, figure out if any emails are new, and forward the new emails to our cell phone address. Note that there are still some pay-as-you-go cellphone providers that do not charge for receiving text messages (even from email), so this can be a very inexpensive way to keep track of your inbox on the go.There is a lot of potential to add power to this application. With a little knowledge of PERL, you could create custom filters (including things impossible with GMail filters such as "only forward during xx hours of the day") and reactions to receiving emails.
Let's begin.
Receiving the feed
Note: You could do this also in PERL. I'm doing it in wget.The first thing the script needs to do, is receive the atom feed for your GMail inbox. We will use GNU wget to obtain the feed. In it's most basic form, wget's syntax is:
wget <url>
However, we need to specify a few additional parameters, namely HTTP authentication username and password. As well, we want to force wget to always download to the same filename. It's normal action is to not overwrite files that already exist, so we have to override it. So, the syntax becomes:
wget -O <output file> <url> --http-user=<username> --http-password=<password>
For GMail, the parameters are as follows:
URL: https://mail.google.com/gmail/feed/atom
http-user : your.e.mail@gmail.com
http-password: your gmail password
Note that since the atom feed is on an https server, we are not passing our username/password in the clear.
For the article, let's force wget to save into a file called 'atomfeed'. Our command is now:
wget -O atomfeed https://mail.google.com/gmail/feed/atom --http-user=gmail.user@gmail.com --http-password=GMailPassword
Now that we have the atom feed, it is time to parse it.
Parsing the Atom feed
We will use a short PERL script to parse the atom feed, and forward pertinent messages to the cellphone.Atom feeds are conceptually the same as RSS feeds, in case you have not heard of both. They are XML documents that contain a summary of information: ie, blog entries, news entries, or in this case, your unread messages in your inbox. The structure of the GMail atom feed is as follows:
- feed - top level tag
- fullcount - Number of unread emails
- entry - tag for each unread email
- title - Email subject line
- id - Unique identifier for each inbox entry
- summary - Short snippet of email contents
- author - Parent tag for source's information
- email - Author's email
- name - Author's name
#!/usr/bin/perl -w
use XML::Parser;
use Net::SMTP;
open(ATOM, "atomfeed") or die ("Failed to get atomfeed");
my $page;
#load the atom feed
while ($line =
$page = $page . $line;
}
close(ATOM);
my $curID = "";
my $curTitle = "";
my $curContents = "";
my $curAuthor = "";
my $curAuthorEmail = "";
my $elementtext;
my @context;
my @oldIDs;
my @newIDs;
my $parser = new XML::Parser();
The code's pretty self explanitory. We load the atom feed into the $page variable. For use with the parsing functions, we initialize five variables ($curID. $curTitle, $curContents, $curAuthor, $curAuthorEmail) used to keep track of the entry currently being parsed. As well we create variables to keep track of our parsing state ($elementtext and @context). Ignore the ID-related variables, we'll deal with them later.
Our next step is to use XML::Parser to parse the atom feed. This is accomplished by the following code:
$parser->setHandlers(
Start => \&startTag,
Char => \&parseTag,
End => \&endTag, );
$parser->parse($page);
What we've done is told the XML parser to use three supplied subroutines (passed by reference) as callbacks during parsing. The startTag routine will be called whenever the parser encounters a new tag. Likewise, the parseTag routine is called as we encounter text in each tag, and the endTag routine is called whenever the parser encounters the end of a tag. The final line of the snippet executes the parser, parsing the atom feed using our supplied functions.
We have to supply these subrountines, since they detail how we deal with the data we're parsing. Put these at the bottom of the perl script since they're separate subrountines.
sub startTag {
my ($p, $tag, %atts) = @_;
push @context, $tag;
}
sub parseTag {
my ($p, $text) = @_;
return if ($text !~ /\S/);
$text =~ s/^\s*//;
$text =~ s/\s*^//;
$elementtext .= $text;
}
sub endTag {
my ($p, $tag) = @_;
$parentElem = $context[-2];
if ($tag eq "id" && $parentElem eq "entry") {
$curID = $elementtext;
} elsif ($tag eq "title" && $parentElem eq "entry") {
$curTitle = $elementtext;
} elsif ($tag eq "summary" && $parentElem eq "entry") {
$curContents = $elementtext;
} elsif ($tag eq "email" && $parentElem eq "author") {
$curAuthorEmail = $elementtext;
} elsif ($tag eq "name" && $parentElem eq "author") {
$curAuthor = $elementtext;
} elsif ($tag eq "entry") {
push @newIDs, $curID;
my $old = 0;
sendMail("Subject:$curTitle\n\nFrom:$curAuthorEmail [$curAuthor]\n$curContents");
}
$elementtext = "";
pop @context;
}
The startTag and parseTag subroutines are trivial. We're using the @context array as a stack to keep track of our state within the parser. Each time we enter a tag and begin to parse it's children or data, we push the element name onto the stack. We use this in the endTag subroutine to determine what to do with the parsed text. The parseTag subroutine, which is run during the parsing of data nodes, appends whatever data is supplied to the routine to the $elementtext variable (which will be used in endTag.
The endTag subroutine looks at the current context in the node tree and saves the current $elementtext to the proper variable. Whenever we encounter the end of an entry tag, we send the email to the server. This is a dirty solution: a nicer solution would be to save all the entry data into a data structure and later deal with each entry saved (an exercise for the reader ;) ). The final thing the procedure does is pop the finished tag off the @context stack and clear the $elementtext buffer.
Note that if you want to change the format of the sent email, change the formatting in the sendMail function call, above.
Sending the Email
As shown in endTag, we use sendMail to send the email to the cellphone. The source code is as follows:sub sendMail {
my @message;
push @message, $_[0];
$smtp2 = Net::SMTP->new('smtp.server.ca');
$smtp2->mail('source.email@isp.ca');
$smtp2->to('cellphoneemail@provider.com');
$smtp2->data();
$smtp2->datasend(@message);
$smtp2->dataend();
$smtp2->quit;
}
sendMail just uses the Net::SMTP module to send an email. You need to provide a SMTP server, source email address, and your cellphone's email address.
What about duplicates?!?!
There's a major flaw with our current solution: every time our perlscript runs, it will send an email for all unread messages in our mailbox! That means that, until we read the messages, we will receive text messages every n minutes for the same messages. Obviously this is unsuitable!Solution
We will save the ID's of messages we have notified the cellphone of in a file. Before emailing the cellphone, we will compare with a saved list of ID's for messages that we have already sent to the cellphone. That way, the phone only receives notification of new emails.The first step is to load all the old ID's into an array. Place the code snippet below after the variable declaration (first snippet) and before the parser initialization (second snippet)
#load the previous inbox ID's
open(OLDID, "ids.old");
while ($line =
chomp $line;
push @oldIDs, $line;
}
close(OLDID);
We open the file 'ids.old' (assumably saved by a previous run of the script), and load each line of the file into an array. We will modify endTag to look through this array before determining whether or not it should send the email address, by changing the function's code as follows:
sub endTag {
my ($p, $tag) = @_;
$parentElem = $context[-2];
if ($tag eq "id" &&amp; $parentElem eq "entry") {
$curID = $elementtext;
} elsif ($tag eq "title" &&amp; $parentElem eq "entry") {
$curTitle = $elementtext;
} elsif ($tag eq "summary" && $parentElem eq "entry") {
$curContents = $elementtext;
} elsif ($tag eq "email" &&amp; $parentElem eq "author") {
$curAuthorEmail = $elementtext;
} elsif ($tag eq "name" &&amp; $parentElem eq "author") {
$curAuthor = $elementtext;
} elsif ($tag eq "entry") {
push @newIDs, $curID;
my $old = 0;
#was email in oldids?
foreach $id (@oldIDs) {
if ($curID eq $id) {
$old = 1;
}
}
if ($old == 0) {
sendMail("Subject:$curTitle\n\nFrom:$curAuthorEmail [$curAuthor]\n$curContents");
}
}
$elementtext = "";
pop @context;
}
As you can see, we are now comparing each entry's id to the loaded list of old ID's. Only if the id is not in the list do we notify the cell phone of the waiting message. As well, we are saving every message ID (old or new) in the array @newIDs.
The final thing to do is to re-write the 'ids.old' file to contain the message ID for all emails in our current feed. This code is placed after the $parser->parse line, making it the last thing that happens in our main routine (and immediately above our subroutine code).
open(NEWID, ">ids.old");
foreach $id (@newIDs) {
print NEWID $id . "\n";
}
close(NEWID);
Putting it all together
We want a single command to run to complete both functions (download and parse the atom feed). So, save the perlscript as 'mail.pl' and make a small shell script called run.sh:#!/bin/sh
wget -O atomfeed https://mail.google.com/gmail/feed/atom --http-user=gmail.user@gmail.com --http-password=gmailPassword
./mail.pl
Test this out. If it doesn't work, try adding debugging statements in the code to see where/why it fails.
Automation
The last thing to do is to automate the script so it runs periodically. For this you need to be running a cron daemon. There's a large variety of daemons available, each with it's own method of configuration. For Gentoo users, we can do the following (as root):host user # crontab -e
Then add a line to the crontab file such as:
0,15,30,45 * * * * /home/th0mas/mobigm/run.sh
This tells the server to run the script 0, 15, 30, and 45 minutes after every hour.
Ideas
It'd be easy to set up custom filters, for example you could compare the author's name to a whitelist, or ensure that it is currently a specific time of day (or do that automation with the crontab). Or you could modify the code to look at other RSS or atom feeds to notify your cellphone of a new blog update.Download
You can download a copy of the files we've created in this entry here:http://th0mas.xbox-scene.com/tuts/mobigm-1.tgz
Feedback
This is the first tutorial I've written in awhile. Please let me know if I should expand on detail in places, or be more terse in others. I'll be reading the comments.Read Post..
Tuesday, April 11, 2006
F1rst P0st!!!11
Welcome to Open-Source Fun.
What's this blog's purpose?
Open source is a powerful concept. However, common users do not take advantage of it's versatility: they just install the latest snapshot and begin using the application. The purpose of this blog is to document original projects and guides that highlight the power of open source, and how you can harness open source technology to solve common (and not so common) problems.
Read Post..
What's this blog's purpose?
Open source is a powerful concept. However, common users do not take advantage of it's versatility: they just install the latest snapshot and begin using the application. The purpose of this blog is to document original projects and guides that highlight the power of open source, and how you can harness open source technology to solve common (and not so common) problems.
Read Post..
