Author: John J. Lee <jjl@@pobox.com>
Keywords: form,HTML,Python,web,client,client-side
Date: @(time.strftime(

@# This file is processed by EmPy @# http://wwwsearch.sf.net/bits/colorize.py @{ from colorize import colorize import time import release last_modified = release.svn_id_to_time("$Id: README.html.in 56674 2008-07-19 13:44:41Z jjlee $") try: base except NameError: base = False }
@[if base]@[end if]

# ClientForm

ClientForm is a Python module for handling HTML forms on the client side, useful for parsing HTML forms, filling them in and returning the completed forms to the server. It developed from a port of Gisle Aas' Perl module `HTML::Form`, from the libwww-perl library, but the interface is not the same.

Simple working example: @{colorize("".join(open("examples/simple.py").readlines()[2:]))}

A more complicated working example (Note: this example makes use of the ClientForm 0.2 API; refer to the README.html file in the latest 0.1 release for the corresponding code for that version.): @{colorize("".join(open("examples/example.py").readlines()[2:]))}

All of the standard control types are supported: `TEXT`, `PASSWORD`, `HIDDEN`, `TEXTAREA`, `ISINDEX`, `RESET`, `BUTTON` (`INPUT TYPE=BUTTON` and the various `BUTTON` types), `SUBMIT`, `IMAGE`, `RADIO`, `CHECKBOX`, `SELECT`/`OPTION` and `FILE` (for file upload). Both standard form encodings (`application/x-www-form-urlencoded` and `multipart/form-data`) are supported.

The module is designed for testing and automation of web interfaces, not for implementing interactive user agents.

Security note: Remember that any passwords you store in `HTMLForm` instances will be saved to disk in the clear if you pickle them (directly or indirectly). The simplest solution to this is to avoid pickling `HTMLForm` objects. You could also pickle before filling in any password, or just set the password to `""` before pickling.

Python 2.0 or above is required. To run the tests, you need the `unittest` module (from PyUnit). `unittest` is a standard library module with Python 2.1 and above.

For full documentation, see the docstrings in ClientForm.py.

Note: this page describes the 0.2 (stable release) interface. See here for the old 0.1 interface.

## Parsers

ClientForm contains two parsers. See the FAQ entry on XHTML for details.

mxTidy or µTidylib can be useful for dealing with bad HTML.

I think it would be nice to have an implementation of ClientForm based on BeautifulSoup (i.e. all methods and attributes implemented using the BeautifulSoup API), since that module does tolerant HTML parsing with a nice API for doing non-forms stuff. (I'm not about to do this, though. For anybody interested in doing this, note that the ClientForm tests would need making constructor-independent first.)

## Backwards-compatibility mode

ClientForm 0.2 includes three minor backwards-incompatible interface changes from version 0.1.

To make upgrading from 0.1 easier, and to allow me to stop supporting version 0.1 sooner, version 0.2 contains support for operating in a backwards-compatible mode, under which code written for 0.1 should work without modification. This is done on a per-`HTMLForm` basis via the `.backwards_compat` attribute, but for convenience the ParseResponse() and ParseFile() factory functions accept `backwards_compat` arguments. These backwards-compatibility features will be removed in version 0.3. The default is to operate in backwards-compatible mode. To run with backwards compatible mode turned OFF (strongly recommended): @{colorize(r""" from urllib2 import urlopen from ClientForm import ParseResponse forms = ParseResponse(urlopen("http://example.com/"), backwards_compat=False) # ... """)}

The backwards-incompatible changes are:

- Ambiguous specification of controls or items now results in AmbiguityError. If you want the old behaviour, explicitly pass `nr=0` to indicate you want the first matching control or item.

- Item label matching is now done by substring, not by strict string-equality (but note leading and trailing space is always stripped). (Control label matching is always done by substring.)

- Handling of disabled list items has changed. First, note that handling of disabled list items in 0.1 (and in 0.2's backwards-compatibility mode!) is buggy: disabled items are successful (ie. disabled item names are sent back to the server). As a result, there was no distinction to be made between successful items and selected items. In 0.2, the bug is fixed, so this is no longer the case, and it is important to note that list controls' `.value` attribute contains only the successful item names; items that are selected but not successful (because disabled) are not included in `.value`. Second, disabled list items may no longer be deselected: AttributeError is raised in 0.2, whereas deselection was allowed in 0.1. The bug in 0.1 and in 0.2's backwards-compatibility mode will not be fixed, to preserve compatibility and to encourage people to upgrade to the new 0.2 `backwards_compat=False` behaviour.

## Credits

Apart from Gisle Aas for allowing the original port from libwww-perl, particular credit is due to Gary Poster and Benji York, and their employer, Zope Corporation, for their contributions which led to ClientForm 0.2 being released. Thanks also to the many people who have contributed bug reports.

## Download

For installation instructions, see the INSTALL.txt file included in the distribution.

Stable release There have been three fairly minor backwards-incompatible interface changes since version 0.1 (see above), but by default the code operates in a backwards-compatible mode so that code written for 0.1 should work without changes.

0.2 includes better support for labels, and a simpler interface (all the old methods are still there, but some have been deprecated and a few added).

@{version = "0.2.10"}
- ClientForm-@(version).tar.gz
- ClientForm-@(version).zip
- Change Log (included in distribution)
- Older releases.

Old release No longer maintained. I recommend upgrading from 0.1 to 0.2.

There were many interface changes between 0.0 and 0.1, so you should take care if upgrading old code from 0.0.

0.1 includes `FILE` control support for file upload, handling of disabled list items, and a redesigned interface.

@{version = "0.1.17"} @{win_version = release.win_version(version)}
- ClientForm-@(version).tar.gz
- ClientForm-@(win_version).zip
- Change Log (included in distribution)
- Older releases.

Ancient release No longer maintained. You don't want this.

@{version = "0.0.16"} @{win_version = release.win_version(version)}
- ClientForm-@(version).tar.gz
- ClientForm-@(win_version).zip
- Change Log (included in distribution)
- Older releases.

## Subversion

The Subversion (SVN) trunk is http://codespeak.net/svn/wwwsearch/ClientForm/trunk, so to check out the source:

    svn co http://codespeak.net/svn/wwwsearch/ClientForm/trunk ClientForm

## FAQs

- Doesn't the standard Python library module, `cgi`, do this?

  No: the `cgi` module does the server end of the job. It doesn't know how to parse or fill in a form or how to send it back to the server.

- Which version of Python do I need?

  2.0 or above (ClientForm 0.2; version 0.1 requires Python 1.5.2 or above).

- Is `urllib2` required?

  No.

- How do I use it without `urllib2`?

  Use `.click_request_data()` instead of `.click()`.

- Which `urllib2` do I need?

  You don't. It's convenient, though. If you have Python 2.0, you need to upgrade to the version from Python 2.1 (available from www.python.org). Otherwise, you're OK.

- Which license?

  ClientForm is dual-licensed: you may pick either the BSD license, or the ZPL 2.1 (both are included in the distribution).

- Is XHTML supported?

  Yes. You must pass `form_parser_class=ClientForm.XHTMLCompatibleFormParser` to `ParseResponse()` / `ParseFile()`. Note this parser is less tolerant of bad HTML than the default, `ClientForm.FormParser`

- How do I figure out what control names and values to use?

  `print form` is usually all you need. In your code, things like the `HTMLForm.items` attribute of `HTMLForm` instances can be useful to inspect forms at runtime. Note that it's possible to use item labels instead of item names, which can be useful — use the `by_label` arguments to the various methods, and the `.get_value_by_label()` / `.set_value_by_label()` methods on `ListControl`.

- What do those `'*'` characters mean in the string representations of list controls?

  A `*` next to an item means that item is selected.

- What do those parentheses (round brackets) mean in the string representations of list controls?

  Parentheses `(foo)` around an item mean that item is disabled.

- Why doesn't <some control> turn up in the data returned by `.click*()` when that control has non-`None` value?

  Either the control is disabled, or it is not successful for some other reason. 'Successful' (see HTML 4 specification) means that the control will cause data to get sent to the server.

- Why does ClientForm not follow the HTML 4.0 / RFC 1866 standards for `RADIO` and multiple-selection `SELECT` controls?

  Because by default, it follows browser behaviour when setting the initially-selected items in list controls that have no items explicitly selected in the HTML. Use the `select_default` argument to `ParseResponse` if you want to follow the RFC 1866 rules instead. Note that browser behaviour violates the HTML 4.01 specification in the case of `RADIO` controls.

- Why does `.click()`ing on a button not work for me?
  - Clicking on a `RESET` button doesn't do anything, by design - this is a library for web automation, not an interactive browser. Even in an interactive browser, clicking on `RESET` sends nothing to the server, so there is little point in having `.click()` do anything special here.
  - Clicking on a `BUTTON TYPE=BUTTON` doesn't do anything either, also by design. This time, the reason is that that `BUTTON` is only in the HTML standard so that one can attach callbacks to its events. The callbacks are functions in `SCRIPT` elements (such as Javascript) embedded in the HTML, and their execution may result in information getting sent back to the server. ClientForm, however, knows nothing about these callbacks, so it can't do anything useful with a click on a `BUTTON` whose type is `BUTTON`.
  - Generally, embedded script may be messing things up in all kinds of ways. See the answer to the next question.
- Embedded script is messing up my form filling. What do I do?

  See the General FAQs page and the next FAQ entry for what to do about this.

- How do I change `INPUT TYPE=HIDDEN` field values (for example, to emulate the effect of JavaScript code)?

  As with any control, set the control's `readonly` attribute false.

  @{colorize(r""" form.find_control("foo").readonly = False # allow changing .value of control foo form.set_all_readonly(False) # allow changing the .value of all controls """)}

- I'm having trouble debugging my code.

  The ClientCookie package makes it easy to get `.seek()`able response objects, which is convenient for debugging. See also here for few relevant tips. Also see General FAQs.

- I have a control containing a list of integers. How do I select the one whose value is nearest to the one I want?

  @{colorize(r""" import bisect def closest_int_value(form, ctrl_name, value): values = map(int, [item.name for item in form.find_control(ctrl_name).items]) return str(values[bisect.bisect(values, value) - 1]) form["distance"] = [closest_int_value(form, "distance", 23)] """)}

- Where can I find out more about the HTML and HTTP standards?
  - W3C HTML 4.01 Specification.
  - RFC 1866 - the HTML 2.0 standard.
  - RFC 1867 - Form-based file upload.
  - RFC 2616 - HTTP 1.1 Specification.

I prefer questions and comments to be sent to the mailing list rather than direct to me.

John J. Lee, @(time.strftime("%B %Y", last_modified)).