Last news : A Windows interface for MBGrabber written by Shade. See addition section below.

A few weeks ago (actually during december 99) I wrote a little program that helped me to archive interesting threads on InsideTheWeb message boards. Altough this was quite easier than manually copy and paste every post in the thread, it was still complicated to archive many threads or even a whole message board.
So I thought a step further and started to write what I now call the MBGrabber. This new version offer the following improvements :

• Easy archiving of a whole message board
• Easy archiving of a single thread
• Download only newer posts since the last archiving
• Create a update.htm file which contain a link to all the new posts
• Support proxy connection

http://www.InsideTheWeb.com/messageboard/mbs.cgi?acct=mbnnnnnn&MyNum=ppppppppp&P=Yes&TL=ttttttttt

Where mbnnnnnn is the board ID. ppppppppp is the current message ID. ttttttttt is the current thread ID.

archiving a whole message board

The command line argument /M followed by a message board descriptor (mbnnnnnn) will force MBGrabber to retrieve every post in the specified mb. A file named mbnnnn.htm will be created. This file will hold the whole mb thread hierarchy.
For each threads, a file named th_ttttttttt.htm is created. This file will hold the thread hierarchy as well as every post in this thread.
When a th_ttttttttt.htm file already exist in the current directory, MBGrabber will only download and update new posts in this thread.

archiving a single thread

The command line argument /T followed by a thread descriptor (ttttttttt) will force MBGrabber to retrieve only the specified thread in the current mb.
This is usefull if from, times to times, you'd like to keep a trace of a very interesting thread without bothering to retrieve the whole mb. Just note the thread ID and pass it as argument to MBGrabber.

Download only newer posts

By default, MBGrabber first check if a post is not already present in the th_ttttttttt.htm file before downloading it. This prevent downloading already archived posts.
The command line argument /A force MBGrabber to download every post of every thread without checking against already existing ones. This can be usefull in the case posts were deleted from the main board and you like to remove them from your archive too. Indeed, MBGrabber can't remove by itself no more existing posts. from a archived thread file.

Create an update file

The command line argument /U force MBGrabber to generate a file called update.htm. This file will be filled with links to every new post since the last archiving.
This should allow a user to quickly browse through all new post since the last archiving of the mb (or thread).

Support proxy connection

The command line argument /P should enable MBGrabber to work through a proxy server. If you know you're working through a proxy or want to use a proxy, just pass your proxy server name and port to mbgrabber through the /P parameter. Default port is 80.

Usage :

MBGrabber.exe /Mmbnnnnnn [/Tttttttttt] [/A] [/U] [/P proxyserver[:port]] [/V] [/D]

Where :

• /M mbnnnnnn indicates the mb you want to archive (ex: mb950700)
• /T ttttttttt indicates the optional single thread to retrieve (ex: 948348869)
• /A force retrieving of all posts (newer and older)
• /U generates a update.htm file
• /P allow proxy use
• /V turn verbose mode off
• /D turn debug mode on

Sample grabbed pages :

If you'd like to see the result you may expect from MBGrabber, just look at the following sample pages generated automagically by MBGrabber :

• mb850700.htm : The main page generated when retrieving a whole mb (Shortened manually to only 3 threads).
  This page is generated when retrieving a whole mb (no /T paramter).
  For example, try MBGrabber /Mmb850700
• th_948348869.htm, th_948485893.htm, th_948232112.htm : The seperated thread file of the above board.
  Those file are generated when archiving a whole mb or a single thread.
  For example, try MBGrabber /Mmb850700 /T 948348869
• update.htm : A sample update file generated when parameter /U is present.
  Try for example MBGrabber /Mmb850700 /U

Oops, MBGrabber crashed when you used it ? Well, I told you this may happen :(
Please send me a little mail describing the problem (which board, which thread, ...) and I'll do my best to correct this bug. Want to correct the bug yourself ? Want to improve MBGrabber ? Want to adapt it for other message boards providers ? In one word : want to get the source code ?
No problem, get it here. But please, keep me informed of everything you correct, modify, improve ... so everybody else can enjoy it too. Thanks in advance !

History :

To do :

Please send me any comments about things you'd like MBGrabber do.

• Even more catching the many exceptions that may arise if something is wrong. For example, if you start MBGrabber with a worng mb identificator, it will more than certainly crash because it's excepting a main thread page and get something more different.
• A windows front end for MBGrabber. Not for me, I'm not good at it :(
  But hopefully Shade came to my rescue and wrote a very nice one.
• Find a way to 'automagically' allow the grabber to adapt itself whenever the message board page layout change.
• rewrite mbgrabber as a perl script (or a PHP3 ... hehe :)

Addition/related projects :

• Shade wrote a very nice visual front end running under Windows for the MBGrabber. Feel free to download it (158Kb). You'll also need MBGrabber itself in order to use it.
• You can have a look at the original Thread Grabber project that lead to MBGrabber. This project is stopped for obvious reasons :)

Page created by Laurent