Web Runtime Widget Tutorial – Part 3: Requesting data from Digg and displaying the results

Grab the code!

In this third part of the tutorial we continue the evolution of our widget from a simple Hello World example into a Digg Client widget. We’ve already put together a basic but functional user interface and populated it with some hardcoded data. Now it’s time to hit the web in search of some dynamic content.

In case that sounds like it might be difficult and you’re reaching for your browser’s back button, I’ve got “before and after” screenshots to show you what you get if you stick with this. This is what we had at the end of part 2 of the tutorial:

We had a TextField and a FormButton that didn’t really do anything useful, plus three hardcoded NavigationButtons masquerading as content.

Well, with a bit more code, we end up with this:

There’s no hardcoded trickery going on in the screenshot above. That’s genuine content retrieved from Digg.

Digg List Stories API

Digg publish a number of APIs that allow users & partners to interact programmatically with Digg services. I encourage you to read more about the APIs on offer and in particular I’d like to draw your attention to the API License Agreement.

We will be making use of just one of Digg’s APIs, the List Stories API. As you’ll see if you read the documentation provided by Digg, the API is quite extensive and supports responses in several formats, including XML and JSON.

The  request we will send to Digg will be a variant of this form:

http://services.digg.com/stories/topic/sometopic?count=20&appkey=http%3A%2F%2Fgavinmeiklejohn.com&type=xml 

where “sometopic” will be replaced with the topic entered in our widget’s TextField.

I should also draw your attention to the appkey in the above request. Digg require each request sent to them to contain an application key. Digg do not currently issue keys or use keys for authentication, they only monitor application keys for statistical purposes. You may therefore use the key shown in the example request (and used in the code we’ll see later on), or you can define your own. Just make sure that the key you use is valid according to Digg’s application keys rules.

Digg XML Response Example

The XML responses issued by Digg are presented in Digg’s own custom XML format.

You can see an example of the type of XML data our widget should receive by looking at the example XML response on the List Stories API page referenced earlier.

Briefly, the response consists of a <stories> container, within which will be 0..n <story> elements. (In our case, n will be limited to the value of count in the request we send to Digg.) Each <story> contains a number of attributes and further elements that provide information on a given story, such as the title, a short description, thumbnail associated with the story, number of “diggs” a story has and more.

New file

Technically we could write all of the new code we’re going to need in our existing diggclient.js file and our widget would work. However, if we did that the code would be a mess and your widget probably wouldn’t stay working for very much longer if you wanted to add more functionality to it later on. We need to structure our widget with just as much thought as we would structure any other piece of software we might write.

So we’ll create a new file to contain some of the new JavaScript we need to write – we’ll use the new file to handle our web request to Digg.

To add a new JavaScript file to your project, right click on your project in Aptana Studio, select “New” from the context menu, then select “JavaScript File” from the “New” submenu, as shown in the following image:

This will bring up Aptana’s New JavaScript File dialog. The container should be correctly specified as your project and a default file name of “new_file.js” is offered by Aptana. 

Change the file name to “diggweb.js” and click the Finish button.

Your new file should then be created, added to the list of files in your project and opened in the main editor pane. Hopefully you will see something similar to this:

We’ve just added a new file to our project in Aptana. However, only Aptana understands this. If you wrote code in the new file and tried to call that code or use it in some way from code in diggclient.js, typically nothing much would happen. Why not? Because there is currently nothing in our widget that includes our new JavaScript file at runtime.

There are two ways to resolve this. You can either write a small function to dynamically add a <script> tag to your widget’s DOM document, defining the src attribute of the <script> tag to point to your new file. Or, more simply, you can achieve the same result by modifying your project’s diggclient.html file by hand.

We’ll use the simpler approach as we’ve only got one file to add. Modify your diggclient.html file to match the listing below, by adding the line shown in bold:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
    <head>
        <title></title>
        <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />       
        <script type="text/javascript" src="diggclient.js"></script>
        <script type="text/javascript" src="diggweb.js"></script> 
        <script type="text/javascript" src="WRTKit/WRTKit.js"></script>
        <link rel="stylesheet" href="diggclient.css" type="text/css">
        <META NAME="Generator" CONTENT="Nokia WRT plug-in for Aptana Studio 2.3.0" />
    </head>
    <body onload="init()">
    </body>
</html>

Requesting data from Digg

Let’s get on and write the code to request data from Digg.

In common with virtually all modern browser environments, Nokia WRT supports the XMLHttpRequest API. This API greatly simplifies the work required by web application developers to issue requests to HTTP servers and process any data returned by the server.

XMLHttpRequest does have some shortcomings in so much as it is not an official standard and the implementation of the API tends to vary slightly from browser to browser and from runtime to runtime. More significant though is the difference in security policies between different environments which can make the job of issuing an HTTP request a bit tricky.

The good news is that WRTKit builds on the XMLHttpRequest API to offer an Ajax object, which deals with the differences in underlying API implementation & security policies between browser environments. The Ajax object is a very thin wrapper over the basic XMLHttpRequest but makes the task of issuing a request to an HTTP server about as simple as it can get.

We’ll build another thin wrapper over WRTKit’s Ajax object to help us separate the web request code from the main UI code thereby keeping the UI code as straightforward as possible.

DiggWeb object constructor

We’ll call our web request object DiggWeb. It doesn’t need to do anything special when we create it, so the constructor is just an empty function:

DiggWeb properties

Our DiggWeb object needs just two properties. One to hold a reference to the WRTKit Ajax request object we’re wrapping and one to hold a reference to a callback function that we’ll call when our HTTP request completes. The callback function will be implemented in diggclient.js later in this part of the tutorial.

What does DiggWeb.prototype mean?

If you’re new to JavaScript you might be wondering about the syntax I’ve used to add properties to our DiggWeb object, particularly if you normally speak C++, C# or other class based languages. JavaScript doesn’t support classes. There is no distinction between an object and the definition of that object – everything in JavaScript is an object.

Any function in JavaScript can be a constructor. All that distinguishes a constructor in JavaScript from another function is the usage of the new operator before calling your constructor function. This also implies that the role of a constructor is temporary – it is possible to call a “constructor” function on an already existing object.

No classes? Any function can be a constructor? Surely this is chaos! No, there is order and control in the way JavaScript works.

Every JavaScript object has a prototype property which itself is a collection of properties inherited from a base object. At the top of the inheritance tree is the in-built JavaScript Object data type, from which every JavaScript object inherits.

Any object can extend it’s set of properties by adding new properties to it’s prototype property. These new properties can be data properties or new functions, which provides a powerful mechanism to construct complex objects and object inheritance trees.

So, a line of JavaScript such as:

DiggWeb.prototype.ajaxRequest = null;

is saying, add a new property called ajaxRequest to the set of properties contained by DiggWeb’s prototype property.

If you’d like to read more about the object-oriented aspects of JavaScript I suggest you start with the following links:

• Mozilla Developer Center
• Mike Koss’s paper on Object Oriented Programming in JavaScript

If you want to go further or if you prefer books, I recommend Douglas Crockford’s “JavaScript: The Good Parts”

Issuing a HTTP request

In addition to two properties, our DiggWeb object needs just two functions. One to issue a HTTP request and one to handle changes in the state of the request. Let’s start looking at how we actually issue a HTTP request.

We start by defining a function called requestFromDigg that takes two parameters.

• topic – this will be the text entered by the user in the main view and will be inserted in the correct place in the request URL we construct
• callback – this will be a reference to a function in diggclient.js that is to be called when the HTTP request completes.

Next, we create a WRTKit Ajax object and assign it to our DiggWeb object’s ajaxRequest property. The reference to the callers callback function is stored in our DiggWeb object’s property of the same name.

The Ajax object that will manage the actual HTTP request to Digg for us also requires a callback function. We handle this in two steps. Firstly, we provide an implementation of XMLHttpRequest’s onreadystatechange() function, like this:

What we’ve done here is provide a simple implementation for XMLHttpRequest.onreadystatechange() that will call the function readyStateChange() on our DiggWeb object. readyStateChange() is the second of the two functions we will add to our DiggWeb object.

Why have we written self.readyStateChange() rather than this.readyStateChange()? Because when the function XMLHttpRequest.onreadystatechange() runs, this would refer to the XMLHttpRequest object, not our DiggWeb object. We want the function readyStateChange() in our DiggWeb object to be called, so we need to take care to be explicit about which object we’re referring to.

We’re just about ready to issue our request to Digg for some data to show in our UI. We just need to construct the request URL, like this:

This is just simple string concatenation to build a URL of the form required to request a list of (up to) 20 stories about a specific topic from Digg. Of course, topic is the parameter passed from the caller and will, when we’ve plumbed all of the code together, correspond to the text entered by the user in our widget’s main view.

Finally we can actually transmit our request out on to the web:

As with typical browsing requests, we’re issuing an HTTP “GET” command, hence the first parameter to the open() function. The second parameter is our string containing the URL to request some stories, with the third parameter indicating that we want the request to be performed asynchronously.

By performing our web request asynchronously we can keep the UI responsive, possibly displaying a progress indication, or if our widget was more powerful, possibly allowing the user to do some other task while the request is processed and data returned from Digg.

The call to open() prepares the XMLHttpRequest object to send the request. Nothing actually gets sent until we call send(). We pass null to send() because we’ve already defined all the data that needs to be sent in the open() function.

All we need to do now is write some code to handle the request state changes and ultimately callback to our caller’s code to tell them when the request is complete.

Dealing with request state changes

XMLHttpRequest has an internal readyState property. Every time the readyState property changes in value, the readystatechange event is fired.

The readyState property can have one of the following values:

readyState value	Meaning
0	The XMLHttpRequest object has not yet been used to send any request.
1	The open() function has been called, but no data has yet been sent.
2	The send() function has been called and a request has been sent, but no response has been received yet.
3	Data is now being received by the XMLHttpRequest object.
4	This is the final state which indicates that the request is complete. If data was being returned, it also indicates that all data has now been received by the XMLHttpRequest object.

The readystatechange event is fired once on every state transition, but can also be called multiple times during state 3 if there is a substantial amount of data to be received.

It is the readystatechange event that XMLHttpRequest’s onreadystatechange() function is being called to process. We’ve already seen that we delegate that event handling to a method of our DiggWeb object called readyStateChange().

In our widget we’re only really interested in knowing about completion of our request. We are quite happy to ignore all other states. So we start our event handler by checking the value of the readyState property; if it’s anything other than 4, we return from the function and don’t do any further processing.

If the value of the readyState property is 4, our request is complete and we should call our client’s callback function:

Recall that we stored a reference to the client’s callback function in our DiggWeb object’s callback property.

We use the callback property to allow us to call the client’s callback function, passing two parameters. The first parameter we pass to the callback function is the numerical HTTP status code.

The second parameter we pass to the callback function varies depending on whether the request completed correctly, or with an error.

If all went well and we have a HTTP status of 200, we pass XMLHttpRequest’s responseXML property as the second parameter to the client’s callback function. XMLHttpRequest detects if valid XML is returned as a result of a request; if so, it stores a XML tree in the responseXML property, ready for parsing.

If something unexpected happened and we did not get a HTTP status of 200, we pass XMLHttpRequest’s statusText property as the second parameter to the client’s callback function. The statusText property contains a human readable textual version of the HTTP status code, such as “Not found” for a status code of 404.

Displaying the data in our widget’s UI

Instead of going through every line of code in diggclient.js as we have done in parts 1 & 2 of the tutorial, I’ll just focus on the new and modified code we need to plumb the XML data received from Digg into our widget’s UI.

More objects, more variables

Let’s start by looking at the new variables we require this time.

Now that we’re getting dynamic data from Digg to display in our UI, we need to allow for the case where there are no stories to show. The most likely reason for this would be that the topic entered by the user does not exist. We’ll cater for this by defining a new Label control that will be shown in the main view, as necessary.

The variable noStoriesLabel will hold a reference to this new Label.

We also need one variable to reference our DiggWeb HTTP request object and another variable to reference an array of stories received from Digg, the length of the array being determined by the number of stories we receive. Here are the declarations for these two variables:

The other variables we require are as defined in part 2 of the tutorial.

Initialisation

Our init() function remains almost identical to the code used in part 2 of the tutorial. We just need to remove the code that temporarily added 3 NavigationButtons to the main view and replace it with code that adds a Label to show that there are no stories available yet.

Remove the following code from your init() function…

  and replace it with the following code:

Here we’ve created a new Label control and assigned it to our new noStoriesLabel variable. The Label is constructed in the familiar way, but here we pass null as the second parameter to the Label constructor, signalling that we do not require a caption associated with this Label. Once constructed, the Label is added to our widget’s main view.

We will still use NavigationButton controls to represent each story returned from Digg, but we’ll create the NavigationButton controls as and when they are needed and we’ll also dynamically swap between displaying NavigationButtons when there is data to show and the “No stories available” Label when there is not.

Button Event Handlers

In part 2 of our tutorial we had two button event handler functions:

• diggButtonClicked() that handled the event fired by the user clicking on the main view’s “Digg it” button.
• storyClicked() that handled the event fired by the user clicking on one of the NavigationButtons representing stories in the main view

Of these event handlers, only the implementation of diggButtonClicked() changes in this part of the tutorial. The new implementation looks like this:

Compared to the implementation in part 2 of the tutorial, it’s only the code within the else clause that is different. Now, instead of just displaying a notification showing the user the text they had entered into the TextField control, we want to do something useful and issue a request to Digg for some stories on the topic entered by the user.

We do this by creating a new Array and assigning the resulting array to our storyArray variable. Array is an inbuilt JavaScript object that allows easy creation, manipulation and removal of data in an array. Next we create one of our own DiggWeb objects and assign that to the webRequest variable.

Finally, we use our DiggWeb object to issue a request to Digg for some stories. As we know, because we wrote the DiggWeb object, we issue a request by calling the requestFromDigg() function of our DiggWeb object. We also know that we need to pass two parameters to requestFromDigg():

• the topic on which we want some stories
• a reference to the function we want to be called when the request completes

Therefore, we pass in the topic variable which holds the text the user entered in the TextField control and a reference to a function called handleDiggResponse(), the implementation of which we’ll look at next.

Handling data returned from Digg

You might like to have a look at the example XML data provided by Digg as you follow the implementation of this function.

Also, we’re heavily reliant on the DOM APIs to help process the XML, so if you’re not familiar with them, you might like to take a look at the material on this at the Mozilla Developer Center.

The first thing we do is check the completion status of the request, which is an HTTP status code. We’re looking for a status of 200 which means the request completed correctly.

If we’ve got an HTTP status of 200, what we want next are all the <story> elements in the XML returned from Digg.

We get all of the <story> elements by calling the DOM API function getElementsByTagName() which returns an array of all elements tagged with a specific string, starting from the root of the XML tree we call the function on.

Now we need to dissect each “story” element in turn and pull out the data we’re interested in.

We use a for loop to examine each story element in turn. The length property of the array of story elements returned by getElementsByTagName() gives us the number of stories we have received from Digg.

The first thing we need to do when we start processing a story is create some variables in which we can store the parts of the story we are interested in. That’s the purpose of lines 224 – 228. Once we’ve finished getting hold of the individual parts of a story, we’ll use JavaScript Object Notation (JSON) to combine the parts into a single object and add it to our array of stories.

Note: A good introduction to JSON can be found on the msdn site. You don’t have to speak .NET to get something out of the article. Alternatively, if you haven’t already decided to take a look at “JavaScript: The Good Parts”, JSON is another reason to do so.

Some of the data we’re interested in is contained within the attributes of each story element, so we again use the DOM API, this time to gain access to the attributes for the current story.

Having obtained the attributes for the current story, we store the values of the “link” and “diggs” attributes in their namesake variables.

Although there are several other attributes we could look at, I’m more interested in the data to be found in the child nodes of each story element. So let’s look at how we examine the child nodes:

We use the DOM API once more to retrieve an array of child nodes for the current story. Having got the array of child nodes, we use another for loop to allow us to examine each child node in turn, looking for the data that is of interest to us.

What we want are the parts of the story that have real meaning to the user of our widget – so we’re interested in the title & description of our story together with the thumbnail image associated with the story, if there is one.

In the case of the “title” and “description” child nodes, the actual data, if present, is the value of the first child node of the “title” and “description” child nodes respectively. It is legal for a child node to exist but for there to be no associated value, which is why we need to check the length of the second-level child nodes array rather than assuming we can access the value of the “title” and “description”. When we get the data values we’re interested in, we again store them in their namesake variables, defined earlier in this function.

In the case of the “thumbnail” child node, the data we’re interested in is the URL of the thumbnail image, which is stored in the src attribute. So, just as we did for the story attributes earlier, we get hold of the attributes of the “thumbnail” child node and then retrieve the value of the src attribute. We store the value of the thumbnail’s src attribute in the thumbnail variable defined earlier in this function.

We use JSON to combine the variables declared earlier into a single object which we store in our array of stories by calling the push() function on our storyArray.

JSON provides us with a useful mechanism to create simple data objects on the fly which we can then add to our storyArray.

To construct an object using JSON you write a statement enclosed in braces { }. The JSON statement consists of a comma separated list of property names and values, with a colon used to separate each property name from it’s value.

So our JSON statement creates an object with properties named title, description, diggs, thumbnail and link, giving each of these properties the value of the appropriate data extracted from the XML received from Digg.

The call to push() on our storyArray then adds the object created by our JSON statement to our array of stories. As we’ll see shortly, when we come to access the objects in our storyArray, we can directly reference the properties title, description, diggs, thumbnail and link, even though there is no object defined anywhere in our code that has such properties.

I encourage you to read more about JSON and find out what it can do to make your JavaScript development both easier & more powerful.

Note that the closing brace shown on line 257 above is the closing brace of the outer for loop of our story processing code.

All we need to do now is deal with the case when the request has failed in some way:

Recall from the implementation of our DiggWeb object that we return the XMLHttpRequest’s xmlResponse property if the request completes correctly, but we return the statusText property if there was a problem which we can use to conveniently present a notification to the user informing them why the request failed.

The code that presents the notification is conditional – we don’t show a notification if the response is 404, which means, of course, that the URL was not found.

By convention, Digg will return a 404 status if the topic we are looking for does not exist. We’ve already said that we will show a textual label in the main view to indicate there are no stories related to the topic entered by the user in this case.

If we also showed the “Not found” notification we’d be duplicating information, and as “Not found” is potentially the most frequent error we are likely to encounter, it is better we deal with this case more gracefully in our widget’s UI.

The last thing we do in our handleDiggResponse() function is call another new function, refreshMainView(), passing it the storyArray. This new function will handle updating the controls visible in the main view to reflect the data received from Digg, including displaying the “No stories available” label if necessary.

Updating the main view

To display the stories received from Digg in our widget’s main view, we need to perform two tasks:

1. Remove any current content from the view that is no longer relevant.
2. Add the updated content.

Removing old content

Only two controls in the main view are always relevant – the TextField and the FormButton, so we’ll write code to remove everything apart from these controls from the main view:

We start this function by retrieving an array of the controls in the main view, which we get by calling the function getControls() on the mainView object.

A for loop is then used, which starts with the last control and finishes with the third control which was added to the main view.

Within the for loop we delete each control by calling removeControl() on the mainView object, passing the element of the control array to be deleted. When the for loop exits, the only controls left in the main view will be the TextField and the FormButton.

Note that this code is dependent on the fact that the ListView maintains an array of controls, listed by addition / insertion order. We know therefore that curControls[0] is the TextField control and that curControls[1] is the FormButton control.

Check your references

Beware! If you reversed the direction of the for loop in the code above you would not get the desired result. The call to ListView.getControls() returns a reference to the array of controls contained by the ListView control. When a control is removed from the internal array by calling ListView.removeControl(), the result is visible via the reference we got before we started removing content. Therefore, by advancing the index in the for loop, we effectively skip a possible candidate for deletion. You can of course write code to deal with this, and looping through the array in reverse order is one such mechanism.

Adding new content

The array of stories we received from Digg was passed as the parameter, newStories, to this function. So all we need to do to add fresh content to the main view is construct a NavigationButton for each story, using relevant data from each item in our array of stories.

We start by checking the length of the newStories array because we only want to add NavigationButtons if we’ve got at least one story. For each story, we create a NavigationButton and associate the storyClicked() event handler with it before adding the button to the main view. Notice that the code to do this is very similar to the code we used to add temporary NavigationButtons to the main view in the init() function in part 2 of the tutorial.

The data used to construct each NavigationButton is extracted from the appropriate item in the newStories array.  Notice that we again use a numerical index as the id of the NavigationButton controls and use the title property of our JSON story object as the text on the face of each button. What is interesting is that instead of passing the hardcoded “icon.png” as the image for each button, we use the thumbnail property. This works because behind the scenes the text passed as the third parameter to the NavigationButton constructor becomes the src attribute of an HTML <img> element.

All that’s left to do now is handle the case where there are no stories available. We’ve already created a new Label control and stored a reference to the control in a variable called noStoriesLabel. So we can quite simply deal with this case by adding our noStoriesLabel control to the main view, in lieu of more interesting content:

Updating the detail view

We now need to complete part 3 of our tutorial by updating the code that populates the detail view with content. In part 2 we used the text from the NavigationButton in the main view that corresponded to the story we wanted to read as the title of the story in the detail view. The other data in the detail view was completely hardcoded.

Now we can do a little better by using some of the data we’ve got stored in our story array.

Here’s the new implementation of populateDetailView():

The parameter passed to populateDetailView() is still the object which fired the ActionPerformed event, as it was in part 2 of the tutorial. This allows us to get the id of the control which fired the event, which corresponds to the index in our storyArray of the relevant story object.

Given this, we can now populate the three labels in the detail view with relevant content received from Digg.

Additional Notes

You might be wondering why we requested a XML response from Digg when we could have asked for a JSON request, particularly as we then used JSON to coalesce the parts of the XML we were interested in to a single object.

Again I suggest that by using & processing XML to obtain data from Digg and then using a very small part of JSON’s capabilities to simplify the creation of data objects, we’ve been able to cover more in a single tutorial.

We could have formed our request URL to Digg differently and implemented a function to handle a JSON response. Or, sticking with the XML response, we could have created another JavaScript object, say something like DiggStory which might have simply consisted of some data properties. We could then created these new DiggStory objects, populating them with data received from Digg, before adding them to our stories array.

I encourage you to use the code of this tutorial as a basis for your own development and experimentation. Please feel free to re-implement DiggWeb to receive JSON requests or whatever you wish to try!

Coming next…

This concludes the third part of our tutorial on creating a Digg client with Nokia WRT & Aptana Studio. We now have a functional widget which uses HTTP requests to retrieve content from Digg and present it to the user.

Except, there are a few problems:

• You don’t know whether your request has completed or not, until the UI refreshes with new content. But if there were no stories returned twice in a row, you get no visual indication at all because the UI doesn’t change and keeps saying there are no stories available!
• The detail view only shows a short description of the story, there’s no way to access the full story!
• And what’s with the “Hello” icon? Can’t we have something better?

In the next part of this tutorial we will tidy up some of the rough edges in our widget, introduce usage of CSS in WRT widget development and maybe I’ll even get my digital crayons out and create a different icon for our widget!

Part 4>