Date: 2017-04-07
Updated: 2018-09-11
Author: Erik Leth Danielsen
2018-10-15 There is an unresolved issue i have to find a solution for, they have changed the way the page are loading.
you get a javascript doc you have to run, it reloade to the page after 30ms.
The problem is you have to have a javascript engine, so maby i'm going to make it in nodejs, and even in nodejs it's not easy you have to wate 30ms before grabbing the page.

Https WebClient OpenSSL VB.NET
Get Digikey Prices over SSL TLS1.2 Framework 2.0

Problem:
I had made function there got the prices from Digikey homepage over http so i could calculate a product price it used
System.Net.WebClient.DownloadString()
But Digikey changed their homepage to Https TLS1.2
Then i had to have .Net Framework 4.6, but none have that, so i want to make a solution there would run under Framework2.0

The task is simple just get the price
https://www.digikey.com/products/en?keywords=311-10.0KCRCT-ND

Test Software

---

Solution:
I search the www. and found openssl.org
and looked after a .net version and found OpenSSL.NET
But there was no example on how to use it, it took me a week to figure it out.

Download:
'Added 171129 Call this once to get session id
'Added 180911 Changed getElementById("product-dollars") to getElementByClassName("product-dollars")
https://www.digikey.com/i10c@p1/client/nv-8.0.6/monitor.html_streaming/instart.js
Complete Code VB.Net
To Run it \WindowsApplication1\bin\Release\WindowsApplication1.exe

To view code open \MeldcoHttpReqTest.sln in .net

To use SSL dll in a app make a ref to
\bin\Release\ManagedOpenSsl.dll
and place these dlls in the work dir
\bin\Release\libeay32.dll
\bin\Release\ssleay32.dll

Example of how it works:
1.Get Https Autorisation
stream.AuthenticateAsClient(address.Host)

2.Send Request
str1 = ""
str1 += "GET " & address.PathAndQuery & " HTTP/1.1" & vbCrLf '"/products/en?keywords=311-10.0KCRCT-ND"
str1 += "Host: " & address.Host & vbCrLf '"www.digikey.com"
str1 += "User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:51.0) Gecko/20100101 Firefox/51.0" & vbCrLf
str1 += "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" & vbCrLf
str1 += "Accept-Language: da,en-US;q=0.7,en;q=0.3" & vbCrLf
str1 += "Accept-Encoding: gzip, deflate" & vbCrLf 'Req gzip
str1 += "Connection: keep-alive" & vbCrLf
str1 += "Upgrade-Insecure-Requests: 1" & vbCrLf
str1 += vbCrLf
stream.write(str1)

3.Receive Response Header one byte a time until (vbCrLf & vbCrLf)
"HTTP/1.1 200 OK" & vbCrLf
"Server: instart/nginx" & vbCrLf
"Date: Mon, 17 Apr 2017 07:23:50 GMT" & vbCrLf
"Content-Type: text/html; charset=utf-8" & vbCrLf
"Content-Length: 100" & vbCrLf
"Transfer-Encoding: chunked" & vbCrLf
"Connection: keep-alive" & vbCrLf
"Timing-Allow-Origin: *" & vbCrLf
"X-AspNet-Version: 4.0.30319" & vbCrLf
"P3P: CP="PHY DEM ONL STA PUR NAV COM INT UNI OUR CUR ADM DEV TAI IDC COR BUS DSP"" & vbCrLf
"dynaTrace: PT=22681862;PA=-354454881;SP=www.digikey.com;PS=-560419380" & vbCrLf
"X-dynaTrace-JS-Agent: true" & vbCrLf
"X-Powered-By: ASP.NET" & vbCrLf
"Content-Encoding: gzip" & vbCrLf
"X-Instart-Streaming: HtmlStreaming:HIT;InstantLoad:SKIP,optimization_disabled" & vbCrLf
"X-dynaTrace: PT=22681862;PA=-354454881;SP=www.digikey.com;PS=-560419380 & vbCrLf
"Expires: -1" & vbCrLf
"Vary: User-Agent,Accept-Encoding" & vbCrLf
"Pragma: no-cache" & vbCrLf
"X-AspNetMvc-Version: 5.0" & vbCrLf
"Cache-Control: no-cache, no-store" & vbCrLf
"X-Instart-Request-ID: 533490969632955854:NZF01-CPVNPPRY05:1492413830:766" & vbCrLf
vbCrLf

4.If Transfer-Encoding: chunked, Receive Chunks
"6" & vbCrLf
"Meldco" & vbCrLf
"3" & vbCrLf
".dk" & vbCrLf
"A" & vbCrLf
"WebClient1" & vbCrLf
"0" & vbCrLf
vbCrLf

4.Else Receive Content-Length:

5.If Content-Encoding: gzip
UnZip Data

6. You are done, Then you just have to find the price in the Html String.

Keywords:
Https
HTTP 1.1
GZip
Transfer-Encoding: chunked

Links:
openssl.org
OpenSSL.NET
IIS Dynamic Compression GZip
Chunked transfer
Transfer-Encoding: chunked
OpenSSL Wiki