Solved.
Known problems (possibly) related to this:
- http://drupal.org/node/1506312 (not solved there so far (apparently))
- http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=658276 – yes, definitely the same!!!
- …
The solution is described almost at the end down here (“Update 2012-09-19 / 1“). Skip to there, if you are in a hurry!
With my recent openSUSE upgrade / migration (from 12.1 to 12.2) came a new curl (and of course libcurl).
My bank statement scraper in Perl makes use of libcurl, and now it does no longer read the HTML for the bank’s web-site.
curl and libcurl always come together, and I tried the rough equivalent of the libcurl access in question with curl on the command line:
* About to connect() to banking.postbank.de port 443 (#0)
* Trying 62.153.105.15…
* connected
* Connected to banking.postbank.de (62.153.105.15) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs/
* SSLv3, TLS handshake, Client hello (1):
* Unknown SSL protocol error in connection to banking.postbank.de:443
* Closing connection #0
curl: (35) Unknown SSL protocol error in connection to banking.postbank.de:443
That new curl (7.25.0) was compiled against OpenSSL/1.0.1c.
So how did I proceed in order to find the reasons for my problem?
A web page on curl.haxx.se (http://curl.haxx.se/docs/sslcerts.html) teaches me, that I should try this, in order to find out, whether the problem is with openssl resp. where it is:
$ openssl s_client -connect banking.postbank.de:443
I am quite sure, it must have worked once, so when (“with which release?”) did the problem start?
Alright, I am doing a binary search on the “recent” releases of openssl:
0.9.8x, 1.0.0, 1.0.0j, 1.0.1, 1.0.1c
The latest one, that does not break my request is 1.0.0j,
the first one, that breaks my request is 1.0.1 (I skipped the betas),
and it looks like this (“SSL handshake has read 0 bytes and written …”):
$ openssl s_client -connect banking.postbank.de:443
CONNECTED(00000003)
write:errno=104
—
no peer certificate available
—
No client certificate CA names sent
—
SSL handshake has read 0 bytes and written 321 bytes
—
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
—
I am posting my problem with a few more details to the “openssl-users” mailing list (see the thread on Google Groups [Link]), now I am waiting for responses.
In the meantime I am going to “brew” my own curl.
Compiling against openssl-1.0.0j, the last release, that successfully talks to that bank’s web-site, fails.
Compiling against openssl-1.0.0, another release, that successfully talks to that bank’s web-site, does not yet build a curl, that uses openssl-1.0.0, but instead it uses openssl-1.0.1c nevertheless.
Update 2012-09-19 / 0:
My attempts to brew my own openssl were sort of successful. I wasn’t actually quite use, which way my curl brewing would prefer:
- $ ./config –prefix=/usr/local –openssldir=/usr/local/openssl
- $ ./config –openssldir=/usr/local/openssl
My attempts to brew my own curl, that in turn uses my own openssl were a terrible mess.
I gave up – but only after receiving the successful hint, reported below.
I actually started with brewing openssl into /usr/local/openssl-u.v.w and curl into /usr/local/curl-x.y.z, but that did not lead to the success, that I had expected.
But then, I also did not the expect the rather quick reply from the openssl-users mailing list, that directly led to my problem fix.
Update 2012-09-19 / 1:
Dr Stephen N. Henson (“OpenSSL project core developer”) gave me a rather precious hint on the openssl-users mailing list [Link]:
This is a problem with the server. OpenSSL 1.0.1 is the first release to support TLS version 1.2 and some servers “hang” when connecting. The option -no_tls1_2 or -tls1 should allow you to connect again.
I did as advised –– success:
$ openssl s_client -no_tls1_2 -connect banking.postbank.de:443
$ openssl s_client -tls1 -connect banking.postbank.de:443
curl and libcurl do have their related options in order to makes use of this:
shell $ curl –verbose –tlsv1 ‘https://banking.postbank.de/rai/login’
perl: $h->setopt(CURLOPT_SSLVERSION,CURL_SSLVERSION_TLSv1);
I am very relieved.
Update 2012-09-19 / 2:
Now that my problem is solved, I wonder, why curl+openssl don’t negotiate the right TLS version with the https server, just as we might assume web browsers do. I guess, that’s because curl and openssl are developers’ tools, so people using curl and openssl are expected to be able to handle this sort of thing. Maybe you are experiencing this problem as well, now you are reading my article, and maybe this saves you a lot of time.
Leave a Reply