How CartoDB Powers Data Journalism

From its beginning, CartoDB been a powerful tool for journalists because it’s easy, flexible, scalable and open source. With this simple-to-use tool journalists everywhere are discovering the immense power data visualization gives them for research and telling stories.Political Moneyball by The WSJToday more than 150 media organizations, many of the world’s most respected and widely-followed, use CartoDB in their online reporting.

Journalist Alexandre Léchenet at lemonde.fr has used CartoDB to illustrate reports ranging from the housing travails of Roma groups in France to the different ways French cities have adjusted their students academic year. L’chenet explains why he likes CartoDB:

“I love CartoDB most because it’s a simple and clean way to rapidly highlight geographical data. Geocoding is easy. The ability to really control the markers and colors with something as simple as CartoCSS makes it the best friend of cartography for data-journalism. And it’s open-source!”

Daniel Bramatti and his team from Brazil’s Estadao Dados uses CartoDB a lot in their data blog. He says:

“Last year CartoDB was our platform of choice to visualize election maps. We like CartoDB because it is a very powerful tool, yet very easy to use. It just takes a few clicks and we have a beautiful and fully customizable map to publish in the web or to export to our infographics department.”

And this is what Simon Rogers — formerly of The Guardian, now working at Twitter, says about the platform:

“CartoDB is unique in combining ambitious, elegant mapping with the accessibility of free tools. If you want to show complex geo data - in an accessible, shareable way - for most people CartoDB will be the best solution.”

Repeatedly journalists are expressing delight at how the simple user interface of CartoDB makes their work easier, as well as, in some cases, more imaginative. Some stories are now more possible to tell in a way that an audience will immediately understand.

Last year the Online News Association (ONA) recognized CartoDB’s contribution to the practice of online journalism by naming it a finalist for an award in Technical Innovation in the Service of Digital Journalism. This February, the public named CartoDB a winner in the Publishing Startup Showcase at the O’Reilly Tools of Change for Publishing Conference>.

 

You don’t want your maps to be like others

Now let’s get into the specifics of why CartoDB-generated maps are especially good for data journalists. First, using CartoDB, you can make gorgeous maps in your own distinctive style in at least two ways.

On the one hand, you have the basemaps. You can choose between the existing styles or just add your own. On the other, you have the wizards. These let you style your maps in lots of different ways without having to write a single line of code. Intensity maps, choropleth maps or bubble maps are just some of the wizards implemented in the UI.Map wizards in the CartoDB UIThe idea with all of these methods is that they give instant data visualization power to users with zero coding skills. Of course, users who want, can always use the wizard results as a starting point and modify further for even more map customization.

 

Merge your datasets and use our filters to discover new relations

One of the most popular features on CartoDB is the Visual Merge. By using this wizard, you can merge two existing tables into a single one. This function is really helpful when creating thematic maps, since it will save you from creating complex SQL queries.

Once you have your data ready, you can analyze and explore it using our brand new filters. These will give you a quick view of how your data looks and allow you to explore it with a higher level of detail. You can even use them as a SQL learning tool, since the result of the filters is plain SQL and can be always edited in your SQL window.New brand filters in the CartoDB UIThis feature and the new ones coming in the 2.1 version will make your maps an awesome piece of information and storytelling.

 

Your maps always available

We know that media organizations have very specific requirements when working with maps. In most cases, the maps you create will have huge traffic during the first days, but then, the visit counter will go down. So having a platform that scales is important, as is a pricing model that won’t ruin you.

CartoDB uses different Content Delivery Networks in order to keep your map tiles always available and distribute them as fast as possible in the different regions. So data handling, of any volume, is efficient. At the same time, the pricing model is based in two things:  amount of data and number of mapviews. This basically means you only pay for what you use. And if your maps are viewed much more than anticipated you will pay more only if you exceed the number of mapviews included in your plan.

 

Meet @CartoDB this weekend at Barcelona during the first Open Knowledge Foundation Data Journalism Conference.

For an in-depth illustration of all these data journalism features come meet us this weekend in Barcelona! At the first Data Journalism Conference, organized by the Open Knowledge Foundation, @saleiva will be giving a presentation on Visualizing Data that we promise will tickle your minds (Friday 24th, at 20:00). Join @CartoDB this Saturday the 25th at 18:30, for a hands-on session. Check out the whole event program here.

And we’ll also be giving away a free annual John Snow subscription to the best map application developed during the hackathon, so…. Good luck mapping your data! We can’t wait to see what you come up with.

PGCon: The-most-advanced-database-in-the-world Conference

image

Did you know that CartoDB runs atop PostgreSQL, the most advanced database in the world? Which also happens to be open source? And the fastest growing database in the market?

We actually contribute extensively to our favorite extension of it, PostGIS.

A lot of what is possible in CartoDB is thanks to this flexibility and extensibility of PostgreSQL. (And it still amazes us, all the cool things that can be done in just good, old, plain SQL (size sorting, distance queries, or digging into data).

PostgreSQL has some of the most advanced and replicated technologies, with extremely powerful extensions and analysis SQL. But even more importantly for us at CartoDB, with PostGIS, PostgreSQL offers exceptional geospatial support. PostGIS performs better and with superior functionalities than Oracle, MongoDB, SQL Server, and well, any other database out there. And the fact that PostGIS is just an extension to PostgreSQL demonstrates how extensible and powerful that database is.

Every year the PostgreSQL community gathers together to discuss the future of our favorite database and share ideas. This event is called PGCon, and it happens in Ottawa this week! Our CEO Javier de la Torre will be there to hear from others and share some of CartoDB’s experiences.

Or,  if you ever want to talk about maps and PostGIS in general please ping us. We’d love to show you CartoDB, talk about maps with millions of records, hosted PostgreSQL services, etc.

But hopefully — see you in Ottawa!

 

The open source geospatial community comes together again at FOSS4GNA

image

The open source geospatial community has some of the best people and the most exciting ideas. That’s why we are always excited to take part in one of a few conferences that bring so many of those great people and amazing ideas into the same place for a few days. This Wednesday marks the start of FOSS4G-NA, a fairly new meeting meant to give the North American FOSS4G community an additional and slightly easier way to get together outside of the big FOSS4G conference. It has been a great addition.

Right now, we are putting the finishing touches on a couple of presentations we will be giving at the conference. The first presentation is a lightning talk titled, The future of geospatial data formats at CartoDB. In that talk, we want to show you some of the ways we are using novel formats and the modern web to push geospatial data visualization. The second is titled, Building Geospatial Applications that can read and write data to CartoDB without proxy. In that presentation, we are showing how CartoDB can be used without any additional backend infrastructure to do some interesting things on the web.

If you can’t make it, be sure to follow the conversation on Twitter. If you can make it, find us (@CartoDB or @andrewxhill) and let us know what you are up to using CartoDB or ask us any questions you are dying to get answered. Looking forward to seeing everyone there!

Make maps from data you collect in Google Forms

While many CartoDB users arrive to the service with data on hand others look to use CartoDB to host and map data from ongoing collection. For those users, we offer a number of useful client libraries and tutorials for using our APIs. For businesses, scientists, and students that still want a little easier way to collect data, we thought we would put together this tutorial covering how to collect data with Google Forms and have it inserted directly into a CartoDB table.

In our example, we create a simple form to collect people’s favorite color and their location in the world. Using Google Forms plus a small Google App Script, we then insert those results into our CartoDB table where we have a live map that shows the latest and greatest results to the world. Be sure to fill the form out yourself and see your vote show up on the map!

We have included all the code plus the screencast for you to create your own forms for data collection. It only takes about 5 minutes! Have fun and remember to share the cool things you come up with!

The versatility of retreiving and rendering geospatial data with CartoDB

We have been discussing a lot lately how we can summarize CartoDB in a short and sweet sentence. A lot of adjectives have been tested. One of the options we have liked the most is, CartoDB allows you to render your data on a map. As simple as the phrase sounds, it packs a lot of meaning. Let us explain…


How CartoDB serves geospatial data, the common

At its core, CartoDB gets the data from a database and renders it to tiles. Tiles are a clever solution where images of data instead of complete datasets are transferred over the internet, saving a lot of bandwidth. They also are created on a regular grid that makes them perfect for caching. Once delivered to the browser, those tiles are rendered by any client side API of your liking (e.g. Leaflet, Google Maps, OpenLayers, ModestMaps). Besides being a clever solution to displaying large-scale data on the web, rendering tiles is fast and compatible with old browsers and mobile devices.

Tiles are not a magic bullet for maps on the web. Most obviously, when we don’t have a lot of data there isn’t much gain from tiles versus sending the real data itself. For example, if you need to render 100 points, you will need 100*4*2 (4 bytes per float, latitude and longitude), 800 bytes. It is very likely that the tile or tiles you would need to render these points would take much more than 800 bytes. You also get all the benefits of dynamically changing your data (e.g. adding, removing, or moving points) and you can add things like hover effects that can not be done with tiles alone.

Luckily, CartoDB is way more than just tiles.


How CartoDB serves geospatial data, the non-tile way

CartoDB has a nice API to access to the data, the SQL API. It takes a SQL statement and returns the resulting data. You could use it for simple things like, get the points in your table, to complex things like we do with Torque. Torque is built around the idea that a powerful SQL function can be run to turn any CartoDB table with temporal geometry into a moving visualization in the browser. That can’t be done with tiles.

The SQL API also gives developers a few options for data formats, those include a flat JSON format, TopoJSON and GeoJSON. The GeoJSON format is meant for us mappers, it allows you to transfer geographic information (including metadata) using JSON as container, but in a predictable schema that many of your client side mapping libraries already know. Sounds good, eh?

Getting the data from your cartodb account to your JavaScript application can be done really easily using CartoDB.js. Here is an example of the JavaScript API:


Rendering vector data using Leaflet

Using Leaflet as an example, we render that GeoJSON data on our map with just two more lines of code:

You can see it running here.


Rendering vector data using Google Maps

Google Maps does not support GeoJSON natively but there is a library that fills the void, GeoJSON-to-Google-Maps. The library works in a different way than Leaflet and takes a few more lines of code. You can see a simple example with source code here.


CartoDB is built for your dynamic data

Okay, that was fun but the best thing is yet to come. CartoDB is a dynamic service and that means you have some powerful flexibility in how you retrieve your data for the web. You could execute queries to only get a subset of your data. Or you could query for only the latests data inserted into your tables. You could optimize your maps by changing the quality of the data depending the device your viewers are using. Or you could animate your data if it is linestrings or points with timestamps. This is where the SQL API really gets powerful.

Filtering the data

Imagine you want to get the countries of a particular size, say 1,000,000 Sq Km. Here, we’ll use the SQL API with a little bit of geospatial filtering (ST_Area is going to return meters, so we divide by 1 Sq Km):

Here, we can query for only the points close to known location, specifically the wifi hotspots nearest a viewer.

See the complete example with custom point markers.

(Click anywhere in Manhattan below to see the closest WiFi locations)


Simplify geometries

Sometimes geometries, such as country borders, are really complex, meaning they also make for large files to transfer over the web. We can fix this easily with CartoDB and on-demand geometry simplification over the SQL API:

This can be really useful for developing mobile applications, where data transferred is a very important consideration.


Hover effects

Now that we are rendering data directly in the browser, hover effects are as simple as changing the style of the target:

See the hover example for a full example.


Advanced usage

Apart of these simple examples you can go a lot further and do animations, add effects using D3, draw on canvas, or integrate your geospatial data with other web technologies. For inspiration, take a look at this animated visualization showing the earthquakes using D3 or Torque, our library to create animations using canvas.


Conclusions

Now you’ve seen how CartoDB can dynamically filter data based on user actions and can provide vector layers to display right on Leaflet and Google Maps.  This is one of the key advantages of CartoDB, it allows you to show your dynamic data. Not only based on changes to your request, but as soon as you add or update data in your table, you will see it realtime on the map. This is true whether you choose to to use vector data or tiles! And not even that, CartoDB provides all the infraestructure to support millions of map views, see eRepublik’s dynamic maps use case.

If you have ever been curious about why you would choose CartoDB, we hope this helps you make the decision!

Did you know you can do dynamic graphs with CartoDB? The Policy Climate Interactive project

Even though our blog gives a lot of attention to maps, CartoDB is a great tool for a lot more than just maps. We have seen in the past how the CartoDB APIs can do all sorts of dynamic queries to CartoDB hosted data. While dynamic queries CartoDB can be geospatial in nature, even returning GeoJSON formatted results, we haven’t spent much time highlighting the fact that they don’t have to be geospatial. That is why we are excited by the latest project released by the Climate Policy Initiative, the Policy Climate Interactive.

The Policy Climate Interactive is an online tool, focused on the key economies driving both climate change and climate policy in the world. The report highlights both the good and the bad through the use of interactive data visualization driven by the CartoDB APIs. No other platform available gives the flexibility to dynamically query hosted data in a way that you can create client side visualizations so simply.

In this case, the D3 framework was used to build some really beautiful graphs of the data underlying global climate policy. We were really impressed to see how beautifully CartoDB works with D3 for doing these data visualizations without any maps! This project highlights how powerful CartoDB can be even for non-map based data visualization. Take a look through some of the results,

The way the project uses the D3 library and data coming directly from CartoDB is cool for many reasons. A clear one is the data curation and management flexibility it give the project administrators. Administrators can easily add and modify the source data on CartoDB and the results will immediately appear in the graphs online! CartoDB also gives users the ability to create download links for particular subsets of data, by adding a link to the API query with and indicating the export format (e.g. CSV, Excel, etc).

We are excited to share this project and are honored to have CartoDB be part of such an important project. Congratulations to [Climate Policy Initiative](http://climatepolicyinitiative.org/) for delivering such a forward thinking website and data publishing platform.


eRepublik Brings CartoDB’s Dynamic Mapping to Online Gaming

For years we at CartoDB have been dedicated to visualizing geospecific data to gain analytic insight and tell compelling stories. Our real-time rendering and time-lapse technologies have helped illustrate phenomena all over the earth from Barcelona traffic to Kenyan elephant movements to the history of meteorite strikes on the planet. Now for the first time CartoDB’s power of dynamic mapping is also enhancing a virtual world with massively multiplayer online game eRepublik.

eRepublik is a popular free-to-play strategy game with hundreds of thousands of users globally. Players in eRepublik’s New World claim allegiance to a specific country, for which they build companies, develop diplomatic strategies, and fight on battlefields for more territory. While maps are an especially apt visualization for this kind of gaming application, eRepublik had a hard time to provide them for so many players with any useful accuracy. The game’s very popularity and the volume of rapidly changing content posed the greatest obstacle to successful mapping. CartoDB, however, gives eRepublik a means of producing accurate maps for their many users for a fraction of the cost charged by other SaaS mapping companies.

CartoDB, unlike other mapping platforms, is in fact designed to handle dynamic content at large scales efficiently. With our API eRepublik no longer needs to generate new maps after each player’s action. Instead, as the game progresses eRepublik simply modifies the map data, and the platform renders the changes automatically. Map content gets updated at least once every twenty minutes and players are able to see a visual representation of the game’s current state with practically real-time accuracy. CartoDB handles approximately two million map views in eRepublik daily.

We’re pleased at CartoDB to share this use of our platform by eRepublik both because we’re proud of how it showcases our exceptional dynamic rendering and scaling capabilities, and because for us it’s new territory. Games like eRepublik demonstrate that not every community organizes around real space, and geospatial data is not the only kind that produces interesting map stories.

Let us know if you have more ideas for alternative-space uses of CartoDB, and as always, don’t be shy about sharing your own map stories!

Guest Post: TeamUP wins third prize in Urban Data Challenge

Today we here from Chong Zi Xin and a team of hackers that took part in the recent, Urban Data Challenge. The team reached out a few months ago about using CartoDB for their challenge entry and the other day we found that their entry had won 3rd place. We were really impressed with some of the maps they produced, including perspective views with integrated bar charts (wow!). 

The Urban Data Challenge is a competition that aims to improve transportation through the visualization of urban data sets by drawing meaningful insights. Public transport data such as trams, buses, bicycles and pedestrians from Zurich, Geneva and San Francisco is provided for participants to merge and compare mobility data sets from these three cities.

image

The project, titled “A City’s Heartbeat”, done by core members of three start-ups, won third prize in the Urban Data Challenge. The project utilizes Geneva’s trams and their passenger flow information to help improve urban mobility, tourism experience as well as businesses (such as cafes near tram stops) to make better decisions.

Our first challenge, is to render big, time-series data of tram movements over a two day period. This temporal mapping was achieved by using the CartoDB SQL API and reusing code from their Torque examples. The basic idea is to treat the database table as a cube, while only downloading and visualizing a few slices of the data at a time. It worked beautifully. 

image

image

Develop geo apps and maps at NYC BigApps 2013 with CartoDB

image

This year’s BigApps competition is kicking off right now in New York City. The competition challenges developers and designers to create applications that help the city tackle big issues, help make people’s lives a little better, and help people engage in the city around them.

We have been supporting the competition for a few years now, and we have seen cool applications like Scene Near Me developed using CartoDB. For this years challenge, the City has opened some great new datasets and has even expanded the rules of what you can use to participate on the challenge. As part of that expansion, CartoDB is now one of the official APIs and developing your app using CartoDB qualifies you to enter into the competition. We would love to see one of you win any of the great prices.

We are at the NYC BigApps 2013 Expo and Hackathon Weekend at eBay presenting our API and helping developers with anything geo and maps. If you are around, Andrew Hill can give your team a free upgrade to a Magellan account on CartoDB. 

We can imagine lots of different applications that can be developed for the great city of New York.  There are some areas in particular where we think CartoDB can really help participants. If your application has:

  • Maps that are interactive and that are dynamic
  • You need to find what is the closest subway, park, museum or whatever to a location.
  • You what to intersect location data from phones with any other dataset, in real-time.
  • You need a backend to store your data and be able to query on it live.
  • You need to analyze multiple datasets, merging them via location. For example what neighborhoods are more “bike friendly”.

We are really excited on this challenge and we look forward to help all participants with their GEO needs! Good luck everybody.

Guest Post: Michael Keller from Newsbeast Labs

As some of you may already know, Newsweek / The Daily Beast has been using CartoDB for some time now, and as such today’s blog post comes from Michael Keller of Newsbeast labs. We’d also like to take the opportunity thank Michael for his amazing contributions to the CartoDB community. Thanks!  

A number of recent stories at the Daily Beast have had some kind of mapping component. We use them often to let people see how a national topic affects readers’ local areas.

image

I have been reusing code from former projects and so it was about time I standardized them into reusable templates with Leaflet.js. I released them on Github this week. 

I made three categories: basic map with hover states, hover states + hover infowindow, and all of that with templated infowindows using Underscore.js.

In each of these categories you’ll see a template for a point map, a polygon map, and a map with both points and polygons.

Some features:

• On point + polygon templates, the polygon hover state turns off when you hover over a point.

• Hover windows follow the mouse and respect the boundaries of the map-canvas. I find it most useful to have hover windows close to the mouse so your eye doesn’t have to leave that map region to see that region’s details

image

• Templates with Underscore.js hover windows include sample formatHelper functions to act as a formatting layer between your data values and how you want them to display. For instance, you could store all your feature attributes as boolean variables and run them through various formatHelpers functions to return nice display strings.

• The hover states work by storing a simplified GeoJSON representation of that feature as a feature attribute. On featureOver, that GeoJSON is plotted as a vector using Leaflet.js.

• Point + polygon templates add a secondary style class to hover windows when hovering over points to differentiate from polygons.

If you have any questions, I’m at @mhkeller. If you have improvements, pull requests at http://github.com/mhkeller/cartodb-templates