I have a similar problem, and it seems that there could be a problem with certificate itself.
Here's the path that worked for me:
makecert.exe -r -a sha1 -n CN=localhost -sky exchange -pe -b 01/01/2000 -e 01/01/2050 -ss my -sr localmachine
then look up certificate thumbprint, copy it to the clipboard and remove spaces. This will be a parameter after -h in the next command:
HttpCfg.exe set ssl -i 0.0.0.0:801 -h 35c65fd4853f49552471d2226e03dd10b7a11755
then run a service host on https://localhost:801/ and it works perfectly.
what I cannot make work is for https to run on self-generated certificate. Here's the code I run to generate one (error handling taken out for clarity):
LPCTSTR pszX500 = subject;
DWORD cbEncoded = 0;
CertStrToName(X509_ASN_ENCODING, pszX500, CERT_X500_NAME_STR, NULL, pbEncoded, &cbEncoded, NULL);
pbEncoded = (BYTE *)malloc(cbEncoded);
CertStrToName(X509_ASN_ENCODING, pszX500, CERT_X500_NAME_STR, NULL, pbEncoded, &cbEncoded, NULL);
// Prepare certificate Subject for self-signed certificate
CERT_NAME_BLOB SubjectIssuerBlob;
memset(&SubjectIssuerBlob, 0, sizeof(SubjectIssuerBlob));
SubjectIssuerBlob.cbData = cbEncoded;
SubjectIssuerBlob.pbData = pbEncoded;
// Prepare key provider structure for self-signed certificate
CRYPT_KEY_PROV_INFO KeyProvInfo;
memset(&KeyProvInfo, 0, sizeof(KeyProvInfo));
KeyProvInfo.pwszContainerName = _T("my-container");
KeyProvInfo.pwszProvName = NULL;
KeyProvInfo.dwProvType = PROV_RSA_FULL;
KeyProvInfo.dwFlags = CRYPT_MACHINE_KEYSET;
KeyProvInfo.cProvParam = 0;
KeyProvInfo.rgProvParam = NULL;
KeyProvInfo.dwKeySpec = AT_SIGNATURE;
// Prepare algorithm structure for self-signed certificate
CRYPT_ALGORITHM_IDENTIFIER SignatureAlgorithm;
memset(&SignatureAlgorithm, 0, sizeof(SignatureAlgorithm));
SignatureAlgorithm.pszObjId = szOID_RSA_SHA1RSA;
// Prepare Expiration date for self-signed certificate
SYSTEMTIME EndTime;
GetSystemTime(&EndTime);
EndTime.wYear += 5;
// Create self-signed certificate
pCertContext = CertCreateSelfSignCertificate(NULL, &SubjectIssuerBlob, 0, &KeyProvInfo, &SignatureAlgorithm, 0, &EndTime, 0);
hStore = CertOpenStore(CERT_STORE_PROV_SYSTEM, 0, 0, CERT_SYSTEM_STORE_LOCAL_MACHINE, L"MY");
CertAddCertificateContextToStore(hStore, pCertContext, CERT_STORE_ADD_REPLACE_EXISTING, 0);
Certificate shows fine and it has a working private key, but https will timeout as if thumbprint was never registered. If anyone knows why - plz comment
EDIT1: After some playing around, I have found the initialization for CertCreateSelfSignCertificate which generates proper certificate:
CRYPT_KEY_PROV_INFO KeyProvInfo;
memset(&KeyProvInfo, 0, sizeof(KeyProvInfo));
KeyProvInfo.pwszContainerName = _T("my-container");
KeyProvInfo.pwszProvName = _T("Microsoft RSA SChannel Cryptographic Provider");
KeyProvInfo.dwProvType = PROV_RSA_SCHANNEL;
KeyProvInfo.dwFlags = CRYPT_MACHINE_KEYSET;
KeyProvInfo.cProvParam = 0;
KeyProvInfo.rgProvParam = NULL;
KeyProvInfo.dwKeySpec = AT_KEYEXCHANGE;