Franchu’s lair

This post is an english translation of my previous post Como hacer un mashup y no morir en el intento… as kindly requested by D. Canos y T. Forza on behalf of the other students of the J2EE programming with passion course by Sang Shin.

Before attending the Google Developer Day in Madrid, I wanted to play around with the Google Maps API in order to see if it was difficult to use and get a feeling of how difficult it would be the Google Maps API workshop I was going to attend. That is what motivated me to create a little mashup that would display over a map of Spain the status of the dams and see where we have water and where we are not doing that well.

The first thing that you need in order to do a mashup up is at least one data source. It is getting more common to get the data in a syndicated fromat (RSS, Atom) or to be able to access them through APIs that expose the data in XML or JSON. Nevertheless, it is also possible that you have to face the case in which the information that you want to use in your mashup is available in a website but there is no easy way to extract it. That was my case and the solution is to create an RSS feed through web scrapping.

How to create a feed from a website

You can always hack, in a language you are familiar with (I’d go for python), a web scrapper that would extract the information you are interested in and returns a feed, but that might not be a trivial thing to do for everybody. Fortunately, there is an easier way, that uses a GUI and that is freely hosted by a third party provider that does exactly what we are trying to do. As this mashup was just a proof-of-concept exercise, I didn’t feel like hacking my own web scrapper and it also provided me with an opportunity to discover how Openkapow worked in order to create my feed. In a couple of minutes I had my feed up and running, publicly accesible and created from the data available in a page that gathers all the information about spanish dams extracted from the website of the Spanish Ministry of the Environment. I could have web scrapped directly the original source, but the structure of the website made it more complex for the web scrapper.

In order to create the web scrapper you need to download the tool that is available for downloading directly from the openkapow website. The only downside of this tool is that is available for Windows and Linux, leaving us the poor Mac users a little bit out of the game, but we can always find ways to get around this. You can find the robot that I have created by downloading it from the openkapow website.

Once you publish it, the feed is available to be used. I have put an update rate of 1 day for the robot, in order not to generate useless traffic as the contents are updated once a week. Moreover, in order for the feed not to load directly from the openkapow website (I’ve had some service downtime problems), I use FeedBurner as a feed cache.

Feed manipulation and relevant data extraction

As the feed contains a lot of informatino that I am not interested in displaying in the map in this first experiment (eg. global data of the status of the spanish dams, or regional data), I need to manipulate the data before using them in the map. In order to do this I use an online tool called Yahoo! Pipes that allows the manipulation of different data sources and output a RSS feed, a JSON object, a XML document, …

You can see the pipe that I have created. The nice thing of this tool is that it allows you to see how you have created the pipe, and learn really fast how to do useful things.

Basically what my pipe does is to filter out the items I am not interested in, and extract the values I will use as variables in my mashup (water level, coordinates of the dam, name, …) and output it in the feed.

Showing it in the map

After loading the JSON object directly from the pipe output, displaying them in the map is almost trivial. As a function of the water level of the dam I decided to use a different marker icon, and as there are lots of markers to be displayed it is better to use a marker manager. You can take a look at the source code that I use to generate the map. Instead of the php include, I could have used the Google AJAX feed API, but I didn’t have time to start playing with another API

The final result can be seen in my other blog entry: “Status of the Spanish Dams”

If you have any questions, do not hesitate to ask them in the comments and I will do my best to answer them

Antes de ir al Google Developer Day quería jugar con la API de Google Maps para ver si era complicada de usar y ver cómo de dificil me iba a resultar el taller al que iba a asistir. Es por eso que decidí hacer un pequeño mashup que mostrase sobre un mapa de la peninsula ibérica el estado de los embalses españoles.

Lo primero que se necesita para hacer un mashup es una o varias fuentes de datos. Cada vez es más común disponer de esos datos en formatos sindicados (RSS, Atom) o incluso mediante APIs que exponen los datos en formato XML o JSON. Sin embargo, yo no encontré ningún sitio que me proporcionase esta información en un feed, así que tuve que crearmelo

Creación de feeds a partir de webs

Siempre queda la posibilidad de programarse en algún lenguaje un web scrapper que extraiga la información que queramos y nos la devuelva como un feed. Por suerte, existe una forma más sencilla, gráfica y alojada por terceros que nos permite hacer precisamente esto. Como no tenía ganas de ponerme a programar y quería ver cómo de complicado era de usar la solución gráfica, me decanté por usar Openkapow para crear mi feed. En un par de minutos tenía mi feed creado a partir de los datos de una página que agrega los datos de la página del Ministerio de Medio Ambiente.

Para crearlo, es necesario emplear la herramienta que se descarga de la página de openkapow. La única pega es que solo existen versiones para Windows y Linux. Podeis ver el robot que he creado descargandolo de la página de openkapow.

Una vez publicado, ya está listo para utilizarse. Le he puesto una frecuencia de actualización de 1 día para no generar tráfico inecesario al servidor del que extraigo los datos. Además, para que el feed no se cargue directamente de la página de openkapow (he tenido problemas de continuidad de servicio), utilizo FeedBurner como cache del feed.

Manipulación del feed y extracción de datos relevantes

Puesto que el feed contiene mucha información que no estoy interesado en mostrar en el mapa en una primera instancia (p.ej: datos globales y datos de cada cuenca), necesito manipular los datos del feed antes de usarlos en el mapa. Para ello utilizo otra herramienta online llamada Yahoo! Pipes que permite manipular diferentes fuentes de datos y sacar un feed en RSS, un objeto JSON, XML…

Podeis ver el pipe que me he creado. Lo bueno que tiene esta herramienta es que permite ver cómo se ha creado cada pipe, y aprender de una forma muy rápida cómo hacer cosas útiles.

A grandes rasgos el pipe lo que hace es filtrar los items que me interesan, y extraer los valores que luego utilizaré como variables (nivel de llenado, coordenadas del embalse, nombre del pantano, …) y sacarlo en el feed.

Presentación en el mapa

Una vez cargado el objeto JSON proporcionado por mi pipe, representarlo en el mapa es casi trivial. En función del nivel de llenado se pone un icono diferente al marker, y se usa un gestor de markers para su representación puesto que el número de pantanos es muy elevado. Podeis echar un vistazo al código fuente donde genero el mapa. En vez del php include, se podría haber usado la API de AJAX feed de Google, pero no tenía tiempo de ponerme a jugar con una API más

El resultado lo podeis ver en mi anterior entrada: “Estado de los embalses españoles“

Last week I needed to create some iCal feeds populating the list of events from a database in a Tomcat + MySQL environment.

The first impulse was to create a JSP file that complies with the iCal format defined in the RFC2445 but in the end it resulted in an ugly and unmaintainable code. So the idea died on the spot and another solution was to be found.

Fortunately, there is a nice library called iCal4j developed by Ben Fortuna that makes it very easy to create iCal compatible files from a Java application. Then, the natural solution would have been to create a servlet that would output the iCal file, but the code that I was developing was to be maintained by many people that are not necessarily comfortable coding in Java.

That is why, I started a new project iCal-taglib, that provides a JSP taglib wrapper around the functionality implemented in the library iCal4j. There is still a lot of functionality to be implemented, but at least now, creating a calendar from any JSP page is as easy as:

<ical:calendar>
<ical:event>
<ical:summary>Test</ical:summary>
<ical:startdate>2007-04-10</ical:startdate>
</ical:event>
</ical:calendar>

to create the iCal compatible text:

BEGIN:VCALENDAR
PRODID:-//Miguel Gil Biraud//ical-taglib 0.1//based on iCal4j 1.0//EN//
VERSION:2.0
METHOD:PUBLISH
CALSCALE:GREGORIAN
BEGIN:VEVENT
DTSTAMP:20070410T183339Z
UID:detczmfg5tgah0ldwfkf8@franchu.net
SUMMARY:Test
DTSTART;VALUE=DATE:20070410
END:VEVENT
END:VCALENDAR

In order to use it, you need to download the iCal4j files and the ical-taglib files and place them in the WEB-INF/lib folder of you application server.

Much development is still needed, but the taglib implements so far the basic functionality needed to create simple calendars that are accepted by the main calendaring software available in the market.