httpwebrequest - Fill Form C# & Post Error

Question

I took an example of filling in a form in C# from another question, but when I run the code I just get the page back again (as if i didn't submit anything). I tried manually going to the URL and added ?variable=value&variable2=value2 but that didn't seem to pre-populate the form, not sure if that is why this isn't working.

private void button2_Click(object sender, EventArgs e)
{
    var encoding = new ASCIIEncoding();
    var postData = "appid=001";
    postData += ("&[email protected]");
    postData += ("&receipt=testing");
    postData += ("&machineid=219481142226.1");
    byte[] data = encoding.GetBytes(postData);

    var myRequest =
       (HttpWebRequest)WebRequest.Create("http://www.example.com/licensing/check.php");
    myRequest.Method = "POST";
    myRequest.ContentType = "application/x-www-form-urlencoded";
    myRequest.ContentLength = data.Length;
    var newStream = myRequest.GetRequestStream();
    newStream.Write(data, 0, data.Length);
    newStream.Close();

    var response = myRequest.GetResponse();
    var responseStream = response.GetResponseStream();
    var responseReader = new StreamReader(responseStream);
    label2.Text = responseReader.ReadToEnd();
}

Answer 1

To submit a form programmatically, the general idea is that you want to impersonate a browser. To do this, you'll need to find the right combination of URL, HTTP headers, and POST data to satisfy the server.

The easiest way to figure out what the server needs is to use Fiddler, FireBug, or another tool which lets you examine exactly what the browser is sending over the wire. Then you can experiment in your code by adding or removing headers, changing POST data, etc. until the server accepts the request.

Here's a few gotchas you may run into:

• first, make sure you're submitting the form to the right target URL. Many forms don't post to themselves but will post to a different URL
• next, the form might be checking for a session cookie or authentication cookie, meaning you'll need to make one request (to pick up the cookie) and then make a subsequent cookied request to submit the form.
• the form may have hidden fields you forgot to fill in. use Fiddler or Firebug to look at the form fields submitted when you fill in the form manually in the browser, and make sure to include the same fields in your code
• depending on the server implementation, you may need to encode the @ character as %40

There may be other challenges too, but those above are the most likely. To see the full list, take a look at my answer to another screen-scraping question.

BTW, the code you're using to submit the form is much harder and verbose than needed. Instead you can use WebClient.UploadValues() and accomplish the same thing with less code and with the encoding done automatically for you. Like this:

NameValueCollection postData = new NameValueCollection();
postData.Add ("appid","001");
postData.Add ("email","[email protected]");
postData.Add ("receipt","testing");
postData.Add ("machineid","219481142226.1");
postData.Add ("checkit","checkit");

WebClient wc = new WebClient();
byte[] results = wc.UploadValues (
                       "http://www.example.com/licensing/check.php", 
                       postData);
label2.Text = Encoding.ASCII.GetString(results); 

UPDATE:

Given our discussion in the comments, the problem you're running into is one of the causes I originally noted above:

the form might be checking for a session cookie or authentication cookie, meaning you'll need to make one request (to pick up the cookie) and then make a subsequent cookied request to submit the form.

On a server that uses cookies for session tracking or authentication, if a request shows up without a cookie attached, the server will usually redirect to the same URL. The redirect will contain a Set-Cookie header, meaning when the redirected URL is re-requested, it will have a cookie attached by the client. This approach breaks if the first request is a form POST, because the server and/or the client isn't handling redirection of the POST.

The fix is, as I originally described, make an initial GET request to pick up the cookie, then make your POST as a second request, passing back the cookie.

Like this:

using System;

public class CookieAwareWebClient : System.Net.WebClient
{
    private System.Net.CookieContainer Cookies = new System.Net.CookieContainer();

    protected override System.Net.WebRequest GetWebRequest(Uri address)
    {
        System.Net.WebRequest request = base.GetWebRequest(address);
        if (request is System.Net.HttpWebRequest)
        {
            var hwr = request as System.Net.HttpWebRequest;
            hwr.CookieContainer = Cookies;
        }
        return request;
    }
} 

class Program
{
    static void Main(string[] args)
    {
        var postData = new System.Collections.Specialized.NameValueCollection();
        postData.Add("appid", "001");
        postData.Add("email", "[email protected]");
        postData.Add("receipt", "testing");
        postData.Add("machineid", "219481142226.1");
        postData.Add("checkit","checkit");

        var wc = new CookieAwareWebClient();
        string url = "http://www.example.com/licensing/check.php";

        // visit the page once to get the cookie attached to this session. 
        // PHP will redirect the request to ensure that the cookie is attached
        wc.DownloadString(url);

        // now that we have a valid session cookie, upload the form data
        byte[] results = wc.UploadValues(url, postData);
        string text = System.Text.Encoding.ASCII.GetString(results);
        Console.WriteLine(text);
    }
}