{"id":456,"date":"2012-09-18T15:21:00","date_gmt":"2012-09-18T15:21:00","guid":{"rendered":"http:\/\/www.b.shuttle.de\/hayek\/Hayek\/Jochen\/wp\/blog-en\/2012\/09\/18\/curl-hangs-talking-to-a-web-site-through-https-actually-a-tsl-version-issue\/"},"modified":"2012-09-18T15:21:00","modified_gmt":"2012-09-18T15:21:00","slug":"curl-hangs-talking-to-a-web-site-through-https-actually-a-tsl-version-issue","status":"publish","type":"post","link":"https:\/\/wp.jochen.hayek.name\/blog-en\/2012\/09\/18\/curl-hangs-talking-to-a-web-site-through-https-actually-a-tsl-version-issue\/","title":{"rendered":"curl hangs talking to a web-site through https \u2013 actually a TSL version issue"},"content":{"rendered":"<p>\t\t\t\t<b><span>Solved.<\/span><\/b><\/p>\n<p>Known problems (possibly) related to this:<\/p>\n<ul>\n<li><a href=\"http:\/\/drupal.org\/node\/1506312\">http:\/\/drupal.org\/node\/1506312<\/a>\u00a0(not solved there so far (apparently))<\/li>\n<li><a href=\"http:\/\/bugs.debian.org\/cgi-bin\/bugreport.cgi?bug=658276\">http:\/\/bugs.debian.org\/cgi-bin\/bugreport.cgi?bug=658276<\/a>\u00a0\u2013 yes, definitely the same!!!\u00a0<\/li>\n<li>\u2026<\/li>\n<\/ul>\n<div>\nThis SSL \/ TLS problem seems to appear in March 2012; all the &#8220;<i>before 2012<\/i>&#8221; problems, that sound similar, are <b>not<\/b> related to this issue resp. they do not have the exact same reason.<\/p>\n<p>The solution is described almost at the end down here (&#8220;<i>Update 2012-09-19 \/ 1<\/i>&#8220;). Skip to there, if you are in a hurry!<\/p>\n<p><\/div>\n<p>\nWith my recent openSUSE upgrade \/ migration (from 12.1 <a href=\"https:\/\/www.bibleserver.com\/text\/ESV\/Tobit12%3A2\" class=\"bibleserver extern\" target=\"_blank\" rel=\"noopener noreferrer\">to 12.2<\/a>) came a new curl (and of course libcurl).<\/p>\n<p>My bank statement scraper in Perl makes use of libcurl,\u00a0and now it does no longer read the HTML for the bank&#8217;s web-site.<\/p>\n<p>curl and libcurl always come together,\u00a0and I tried the rough equivalent of the libcurl access in question with curl on the command line:<\/p>\n<div>\n<span>$ curl &#8211;verbose &#8216;https:\/\/banking.postbank.de\/rai\/login&#8217;<\/span><br \/><span>* About to connect() to banking.postbank.de port 443 (#0)<\/span><br \/><span>* \u00a0 Trying 62.153.105.15&#8230;<\/span><br \/><span>* connected<\/span><br \/><span>* Connected to banking.postbank.de (62.153.105.15) port 443 (#0)<\/span><br \/><span>* successfully set certificate verify locations:<\/span><br \/><span>* \u00a0 CAfile: none<\/span><br \/><span>\u00a0 CApath: \/etc\/ssl\/certs\/<\/span><br \/><span>* SSLv3, TLS handshake, Client hello (1):<\/span><br \/><span>* Unknown SSL protocol error in connection to banking.postbank.de:443\u00a0<\/span><br \/><span>* Closing connection #0<\/span><br \/><span>curl: (35) Unknown SSL protocol error in connection to banking.postbank.de:443\u00a0<\/span><\/div>\n<p>\nThat new curl (7.25.0) was compiled against OpenSSL\/1.0.1c.<\/p>\n<p>So how did I proceed in order to find the reasons for my problem?<\/p>\n<p>A web page on <a href=\"http:\/\/curl.haxx.se\/\">curl.haxx.se<\/a> (<a href=\"http:\/\/curl.haxx.se\/docs\/sslcerts.html\">http:\/\/curl.haxx.se\/docs\/sslcerts.html<\/a>) teaches me,\u00a0that I should try this, in order to find out, whether the problem is with openssl resp. where it is:<br \/><span><br \/><\/span> <span>$ openssl s_client -connect <i>banking.postbank.de<\/i>:443<\/span><\/p>\n<p>I am quite sure, it must have worked once, so when (&#8220;with which release?&#8221;) did the problem start?<\/p>\n<p>Alright, I am doing a binary search on the &#8220;recent&#8221; releases of openssl:<\/p>\n<p>0.9.8x, 1.0.0, 1.0.0j, 1.0.1, 1.0.1c<\/p>\n<p>The latest one, that does not break my request <a href=\"https:\/\/www.bibleserver.com\/text\/ESV\/Isaiah1\" class=\"bibleserver extern\" target=\"_blank\" rel=\"noopener noreferrer\">is 1.0.0j<\/a>,<br \/>\nthe first one, that breaks my request <a href=\"https:\/\/www.bibleserver.com\/text\/ESV\/Isaiah1-1\" class=\"bibleserver extern\" target=\"_blank\" rel=\"noopener noreferrer\">is 1.0.1<\/a> (I skipped the betas),<br \/>\nand it looks like this (&#8220;SSL handshake <b>has read 0 bytes<\/b> and written &#8230;&#8221;):<\/p>\n<p><span>$ openssl s_client -connect banking.postbank.de:443<\/span><br \/><span>CONNECTED(00000003)<\/span><br \/><span>write:errno=104<\/span><br \/><span>&#8212;<\/span><br \/><span>no peer certificate available<\/span><br \/><span>&#8212;<\/span><br \/><span>No client certificate CA names sent<\/span><br \/><span>&#8212;<\/span><br \/><span>SSL handshake has read 0 bytes and written 321 bytes<\/span><br \/><span>&#8212;<\/span><br \/><span>New, (NONE), Cipher is (NONE)<\/span><br \/><span>Secure Renegotiation IS NOT supported<\/span><br \/><span>Compression: NONE<\/span><br \/><span>Expansion: NONE<\/span><br \/><span>&#8212;<\/span><\/p>\n<p>\nI am posting my problem with a few more details to the &#8220;<i>openssl-users<\/i>&#8221; mailing list (see the thread on Google Groups [<a href=\"https:\/\/groups.google.com\/forum\/?fromgroups=#!topic\/mailing.openssl.users\/PIjCBj22CLE\">Link<\/a>]),\u00a0now I am waiting for responses.<\/p>\n<p>In the meantime I am going to &#8220;<i>brew<\/i>&#8221; my own curl.<br \/>\nCompiling against openssl-1.0.0j, the last release, that successfully talks to that bank&#8217;s web-site, fails.<br \/>\nCompiling against openssl-1.0.0, another release, that successfully talks to that bank&#8217;s web-site, does not yet build a curl, that uses openssl-1.0.0, but instead it uses openssl-1.0.1c nevertheless.<\/p>\n<div>\nI will keep you updated.<\/p>\n<p>Update 2012-09-19 \/ 0:<br \/>\nMy attempts to brew my own openssl were sort of successful. I wasn&#8217;t actually quite use, which way my curl brewing would prefer:<\/p>\n<ul>\n<li><span>$ .\/config &#8211;prefix=\/usr\/local &#8211;openssldir=\/usr\/local\/openssl<\/span><\/li>\n<li><span>$ .\/config &#8211;openssldir=\/usr\/local\/openssl<\/span><\/li>\n<\/ul>\n<p>\nMy attempts to brew my own curl, that in turn uses my own openssl were a terrible mess.<br \/>\nI gave up \u2013 but only after receiving the successful hint, reported below.<\/p>\n<p>I actually started with brewing openssl into <span>\/usr\/local\/openssl-u.v.w<\/span> and curl into <span>\/usr\/local\/curl-x.y.z<\/span>, but that did not lead to the success, that I had expected.<\/p>\n<p>But then, I also did not the expect the rather quick reply from the openssl-users mailing list, that directly led to my problem fix.<\/p>\n<p>Update 2012-09-19 \/ 1:<br \/>\nDr Stephen N. Henson (&#8220;OpenSSL project core developer&#8221;) gave me a rather precious hint on the openssl-users mailing list [<a href=\"http:\/\/www.mail-archive.com\/openssl-users@openssl.org\/msg68653.html\">Link<\/a>]:<\/p>\n<blockquote><p>\n<i>This is a problem with the server. OpenSSL 1.0.1 is the first release to\u00a0support TLS version 1.2 and some servers &#8220;hang&#8221; when connecting. The option\u00a0-no_tls1_2 or -tls1 should allow you to connect again.<\/i><\/p><\/blockquote>\n<p>I did as advised \u2013\u2013 success:<\/p>\n<p><span>$ openssl s_client -no_tls1_2 -connect banking.postbank.de:443<\/span><br \/><span>$ openssl s_client -tls1 \u00a0 \u00a0 \u00a0-connect banking.postbank.de:443<\/span><\/p>\n<div>\n<\/div>\n<p>curl and libcurl do have their related options in order to makes use of this:<\/p>\n<p><span>shell $ curl &#8211;verbose <b>&#8211;tlsv1<\/b> &#8216;https:\/\/banking.postbank.de\/rai\/login&#8217;<\/span><br \/><span><br \/><\/span><br \/>\n<span>perl: $h-&gt;setopt(CURLOPT_SSLVERSION,CURL_SSLVERSION_TLSv1);<\/span><\/p>\n<p>I am very relieved.<\/p>\n<p>Update 2012-09-19 \/ 2:<br \/>\nNow that my problem is solved, I wonder, why curl+openssl don&#8217;t negotiate the right TLS version with the https server, just as we might assume web browsers do. I guess, that&#8217;s because curl and openssl are developers&#8217; tools, so people using curl and openssl are expected to be able to handle this sort of thing. Maybe you are experiencing this problem as well, now you are reading my article, and maybe this saves you a lot of time.<\/div>\n","protected":false},"excerpt":{"rendered":"<p>Solved. Known problems (possibly) related to this: http:\/\/drupal.org\/node\/1506312\u00a0(not solved there so far (apparently)) http:\/\/bugs.debian.org\/cgi-bin\/bugreport.cgi?bug=658276\u00a0\u2013 yes, definitely the same!!!\u00a0 \u2026 This SSL \/ TLS problem seems to appear in March 2012; all the &#8220;before 2012&#8221; problems, that sound similar, are not related to this issue resp. they do not have the exact same reason. The solution [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_crdt_document":"","jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"_share_on_mastodon":"0"},"categories":[666],"tags":[],"class_list":["post-456","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"share_on_mastodon":{"url":"","error":""},"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/paO0kP-7m","jetpack_likes_enabled":true,"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/wp.jochen.hayek.name\/blog-en\/wp-json\/wp\/v2\/posts\/456","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wp.jochen.hayek.name\/blog-en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/wp.jochen.hayek.name\/blog-en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/wp.jochen.hayek.name\/blog-en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/wp.jochen.hayek.name\/blog-en\/wp-json\/wp\/v2\/comments?post=456"}],"version-history":[{"count":0,"href":"https:\/\/wp.jochen.hayek.name\/blog-en\/wp-json\/wp\/v2\/posts\/456\/revisions"}],"wp:attachment":[{"href":"https:\/\/wp.jochen.hayek.name\/blog-en\/wp-json\/wp\/v2\/media?parent=456"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/wp.jochen.hayek.name\/blog-en\/wp-json\/wp\/v2\/categories?post=456"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/wp.jochen.hayek.name\/blog-en\/wp-json\/wp\/v2\/tags?post=456"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}