{"id":2410,"date":"2010-06-26T12:07:00","date_gmt":"2010-06-26T12:07:00","guid":{"rendered":"http:\/\/www.b.shuttle.de\/hayek\/Hayek\/Jochen\/wp\/blog-en\/2010\/06\/26\/my-approach-to-http-scripting-web-harvesting-page-scraping-and-all-that\/"},"modified":"2010-06-26T12:07:00","modified_gmt":"2010-06-26T12:07:00","slug":"my-approach-to-http-scripting-web-harvesting-page-scraping-and-all-that","status":"publish","type":"post","link":"https:\/\/wp.jochen.hayek.name\/blog-en\/2010\/06\/26\/my-approach-to-http-scripting-web-harvesting-page-scraping-and-all-that\/","title":{"rendered":"my approach to HTTP scripting, web harvesting, page scraping, and all that"},"content":{"rendered":"<ul>\n<li>use LiveHTTPheaders in Firefox<\/li>\n<li>resp. ieHTTPHeaders in IE for extracting the relevant HTTP traffic;<\/li>\n<li>run my script <span><span>scan_HTTP_log_file.pl<\/span><\/span> on that and create a raw perl script (raw because e.g. it doesn&#8217;t know, that it includes <i>session ID<\/i>s etc. as constant literals),<\/li>\n<li>that makes use of my <span><span>JHwis<\/span><\/span> toolkit around <span><span>libcurl<\/span><\/span>, it also includes a truely working <i>cookie machine<\/i>.<\/li>\n<li>on the Curl web-site you can also find <span><span>formfind.pl<\/span><\/span>, that helps a lot in finding all the &#8220;<span><span>input<\/span><\/span>&#8221; tags.<\/li>\n<li>now you have a raw perl script, that you can enrich will all necessary condition handling \u2013 there is all kinds of stuff in that code (<b>like in any code<\/b> generated from visiting a web-site), that you need to replace by dedicated handling, like session IDs and similar fields and URL ingredients; associate coordinate names with the resp. image names.<\/li>\n<li>on CPAN lives a module by the name of\u00a0<span>HTML::TableExtract<\/span>\u00a0for extracting HTML tables;<br \/>I wrapped it up a little, so that I can make use of it on a command line.<\/li>\n<li>that command line utility supplies me with all necessary options for related tasks regarding navigating through HTML and all its tables, just what <span>HTML::TableExtract<\/span> actually does.<\/li>\n<li>I love writing &#8220;small&#8221; utilities in perl resp. ruby, that I wrap up bash resp. zsh scripts.<\/li>\n<\/ul>\n<p>Years ago (I guess the situation is not so much different nowadays) <span><span>libcurl<\/span><\/span> was just so much more powerful than <span><span>LWP<\/span><\/span>, that I simply had to got for <span><span>libcurl<\/span><\/span>. Read <span>&#8220;<i><a href=\"http:\/\/curl.haxx.se\/docs\/httpscripting.html\">Using cURL to automate HTTP job<\/a><\/i>&#8221; <span>aka &#8220;<\/span><\/span><a href=\"http:\/\/curl.haxx.se\/docs\/httpscripting.html\"><i>The Art Of Scripting HTTP Requests Using Curl<\/i><\/a><span><span>&#8220;, that&#8217;s IMHO <b>the major evergreen<\/b> in that area. libcurl does a lot more than just simple PUT, GET and that sort of thing.<\/span><\/span><br \/><span><span>All that makes my swiss army knife of web harvesting and page scraping. <\/span><\/span><br \/><span><span>Reality is seriously more challenging than text book examples, trust me!<\/span><\/span><br \/><span><span>Right, I could make all of that open source. I just recently started my open sourcerer career, after <a href=\"http:\/\/szabgab.com\/\">SzabGab<\/a> had stayed in my place for a couple of days around <a href=\"http:\/\/linuxtag.org\/2010\">LinuxTag.org\/2010<\/a> at Berlin.<\/span><\/span><\/p>\n<p><span><span>I should also mention <a href=\"http:\/\/daniel.haxx.se\/\">Daniel Stenberg<\/a>, the father of <span><span>curl<\/span><\/span>. IMO without his great work the art of HTTP scripting would not stand, where it stands today.<\/span><\/span><\/p>\n<p><span><span>Right: last not least: no, I am not into dealing with AJAX and all that. For the last couple of years my approach has been: with the toolset I described above I can still manage &#8220;all&#8221; tasks without caring for AJAX. It does not matter enough.<\/span><\/span>\t\t\t\t<\/p>\n","protected":false},"excerpt":{"rendered":"<p>use LiveHTTPheaders in Firefox resp. ieHTTPHeaders in IE for extracting the relevant HTTP traffic; run my script scan_HTTP_log_file.pl on that and create a raw perl script (raw because e.g. it doesn&#8217;t know, that it includes session IDs etc. as constant literals), that makes use of my JHwis toolkit around libcurl, it also includes a truely [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_crdt_document":"","jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"_share_on_mastodon":"0"},"categories":[666],"tags":[],"class_list":["post-2410","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"share_on_mastodon":{"url":"","error":""},"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/paO0kP-CS","jetpack_likes_enabled":true,"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/wp.jochen.hayek.name\/blog-en\/wp-json\/wp\/v2\/posts\/2410","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wp.jochen.hayek.name\/blog-en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/wp.jochen.hayek.name\/blog-en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/wp.jochen.hayek.name\/blog-en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/wp.jochen.hayek.name\/blog-en\/wp-json\/wp\/v2\/comments?post=2410"}],"version-history":[{"count":0,"href":"https:\/\/wp.jochen.hayek.name\/blog-en\/wp-json\/wp\/v2\/posts\/2410\/revisions"}],"wp:attachment":[{"href":"https:\/\/wp.jochen.hayek.name\/blog-en\/wp-json\/wp\/v2\/media?parent=2410"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/wp.jochen.hayek.name\/blog-en\/wp-json\/wp\/v2\/categories?post=2410"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/wp.jochen.hayek.name\/blog-en\/wp-json\/wp\/v2\/tags?post=2410"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}