Oct 4, 2009

Using Python to perform Http Request

This summer I was turned on to Python. It's an amazing programming language in my opinion. Python modules or libraries are used just as with other programming languages as helper applications you can import into your programs.
The first package I looked at to attempt to retrieve info from a web page was the "Urllib" package. The urllib package provides a heigh level interface for writing clients that need to interact with HTTP servers, FTP servers, and local files. To retrieve info from a web page is:

try:
import urllib

#python 2
#"u" is a variable we created

u = urlopen("google.com")
data = u.read()

Amazing there is very little code needed to make this work. But there are a few things I found about the above code that I didn't like. The main problem was, I needed to tell the program the format I wanted the data returned. And another issue I found was that we never closed the connection.

I looked at a number of functions in this module, but I didn't really like any of the methods I found. I looked through the book "Python Essential Reference, Developer's Library" but I didn't see what I really wanted. So I started looking into different libraries to see what functionality existed. Of course searching the web I found a library that worked without any problems
(http://stackoverflow.com/questions/843392/python-get-http-headers-from-urllib-call).

I downloaded the "http library" which for some reason my text didn't mention. And I tried the following code:

import httplib
conn = httplib.HTTPConnection("www.python.org")
conn.request("GET", "/index.html")
r1 = conn.getresponse()
print r1.status, r1.reason
>>>200 OK # program printed this response
data1 = r1.read()
conn.request("GET", "/parrot.spam")
r2 = conn.getresponse()
print r2.status, r2.reason
404 Not Found
data2 = r2.read()
conn.close()

The above code provided me with all the headers from the web pages I used to test this code.


3 comments:

  1. I don't know anything about python, so that is interesting. It looks quite a bit different!

    ReplyDelete
  2. Always interesting to learn a little more about another software language. Just so many to learn is hard to be really proficient in them all, but maybe after this class i dable a little more in python. But i would probably want to research the market demend for Python, i had thought it was losing ground to some of the other languages that seem to be more in currently in-style. But thanks for showing another option to use, and some great resource links.

    ReplyDelete
  3. This is an interesting post considering the fact that I haven't really heard of Python until 5 minutes ago. What kind of language is it tho? What are its main uses?

    ReplyDelete