After having once collected all the post of a thread in order to archive them
, I thought it should be possible to automatize this task.
So I started thinking about writting a little program that would help me (or
anyone else doing the same).
What should that program do ?
Well, For each post in a thread, it should extract the post message and write
it to a file.
How do that ? If you look at a post in a thread, you'll see that below the
post itself we can find the whole thread structure with links to other post
in thread.
So, what the program do is first download a post in a thread, parse the
thread structure, and download each subsequent post in order to extract the
message.
During the parsing process the program also copy the thread structure to
the destination file while redirecting the links to anchors (#post_1, #post_2
, ...). In this way, not only the thread will be archived, but it will also
be 'browsable'.
During the message extraction of each post, a local anchor is added along
with a 'back to top' link. This will again improve 'browsability'.
Thus, not only is thread grabber a way to easily archive threads, but it also
allow you to browse them offline.
Of course, you won't find any banners nor any other useless stuff in the
generated file. Who will complain ?
ThreadGrabber is designed to work specifically with InsideTheWeb message
boards.
Although I did not tested it very intensively, ThreadG seems to work well
with many InsideTheWeb Messageboard. There are however a few limitations
regarding the content of the posts beeing grabbed. This is due to the way
actual message and threads are located inside an htm page. If you look at
the source of such a page you'll notice that there is none clear indication
of where the actual post message start or end. Neither is there any clear
indication of where the 'message thread' section start or end. Due to this,
I had to figure out some 'unique' way to devise those position in the page
source.
For example, looking at the page source you'll see that just before the
actual message there is the following tags :
<TABLE WIDTH="100%"><TR><TD>
I use this to determine the starting position of the message.
In the same way you can notice that just after the message you have the
following tag :
<hr size="1">
Once again I use this to locate the end of the message.
It would have been easier if InsideTheWeb people had used some meaningfull
comments like :
<!--Start of message-->
......
<!--End of message-->
But that's probably asking for too much :(
Anyway, due to those facts, it is possible to 'fool' ThreadGrabber by, for example, inserting a <hr size="1"> tag in the body of a message. In this case, the part of the message below this tag will be lost.
Also, the same limitation apply to the 'thread extraction' process. However
this apply only for the Starting page, as thread structure is analyzed only
for the first post of the thread.
I won't describe those limitations in this document, but you can easily
figure them out by studying both the source code of Thread Grabber and a
sample post htm page.
Btw, if InsideTheWeb people ever read this, maybe they would be kind enough to revise their pages layout in order to include the above mentionned 'separation comment's and also in order to correct some little 'buging' details. For example, tag appear either in uppercase or lower case. Same for the tag. All upper or all lower case would be easier to parse.
Adaptation for other messageboard provider should be easy to do. Source code is available for free. It's up to you to suit it to your needs. This program is written in C langage. It would probably be better to rewrite it in perl or even java, but at this time I'm not good enough at none of these :( ... I'm probably not very good in C neither, but well, the program seems to work fine :)
Usage :
ThreadG.exe FirstPage [DestFile]
FirstPage is the path+filename part of the url of the first post in a thread.
Please don't try to 'archive' a whole messageboard, it won't work :(. In
other words, always use as FirstPage the URL of the first post of a thread.
For example if you have the following url :
http://www.InsideTheWeb.com/messageboard/mbs.cgi?acct=mb850700&MyNum=944636118&P=No&TL=944636118
Then FirstPage should be :
/messageboard/mbs.cgi?acct=mb850700&MyNum=944636118&P=No&TL=944636118
DestFile is the name of the file which will be created. This should be a .htm.
If no DestFile is specified then the program will create a file name based
on the FirstPage argument.
For the same example as above, if no DestFile is given, the filename will
be :th_944636118.htm
Some sample thread grabbed :