KEMBAR78
Consuming open and linked data with open source tools | PDF
5/31/2011




Consuming Open and
Linked Data with Open
    Source Tools
Or, how to do something useful
     with all this free data



    Jo Cook, Geospatial Consultant, Astun Technology
Talk Outline

 ● The data and the distribution technologies
   explained
 ● The software stack explained
 ● Actually getting some data and doing something
   with it
 ● Gotchas
 ● Further resources and links
The Data and Technologies
The Data and Technologies
The Data and Technologies




           Point in Latitude/Longitude format (WGS84)
The Data and Technologies
The Data and Technologies
The Data and Technologies
PREFIX naptan: <http://transport.data.gov.uk/def/naptan/>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
PREFIX geo: <http://www.w3.org/2003/01/geo/wgs84_pos#>
SELECT DISTINCT ?stopPoint_airport ?name ?iataRef ?lat ?long
WHERE {
  ?item a naptan:StopArea ;
   naptan:administrativeArea <http://transport.data.gov.uk/id/administrative-
area/145> ;
   skos:prefLabel ?name ;
   geo:lat ?lat ;
   geo:long ?long ;
   naptan:stopPoint [ naptan:airport ?stopPoint_airport ] .
  ?stopPoint_airport naptan:iataRef ?iataRef .
}
The Data and Technologies
The Data and Technologies
The Data and Technologies




http://wallpapertree.com/wallpaper/Sad-Kitten/
The Data and Technologies
The Data and Technologies
The Data and Technologies
The Data and Technologies




http://freeimagesarchive.com
The Data and Technologies
The Data and Technologies




1m4ge.blogspot.com
The Data and Technologies
The Data and Technologies
Actually doing something with the data
Actually doing something with the data
Gotchas




smashingmagazine.com
Gotchas
Mitigation



                    Keep your data here so you're
                    not relying directly on the service
                    for updates to customers




(Maybe even
keep it here too)
Conclusions
A dash of all these, some readily
available python libraries, and
some relatively simple scripting
gets you useful data for your own
applications...

... and by using PostgreSQL to
store your data, you can provide
a buffer in case of api changes or
outages...

... and even do some nifty
manipulation to analyse and
present the data differently!
Conclusions




        Simples!
Conclusions
Not *that* simple, but...
Resources
Linked Data Resources and Example Queries

Best API for transport data: http://openuplabs.tso.co.uk/
O'Reilly Book: http://www.learningsparql.com/
Blog: http://talis-systems.com/blog/
Airport Query: http://getthedata.org/questions/137/where-can-i-get-a-list-of-all-airports-codes-names-locations
Beginner Guide: http://www.pezholio.co.uk/2011/01/a-beginners-guide-to-sparqling-linked-data-part-1/

NHS Choices- RSS

http://www.nhs.uk/servicedirectories/Pages/ServiceSearch.aspx

Police Data- JSON/REST

API: http://policeapi2.rkh.co.uk/api/docs/
CSV/SHP: http://www.police.uk/data

Basic Software Resources

Mapserver dynamic charting: http://mapserver.org/output/dynamic_charting.html
Python SparQLWrapper: http://sparql-wrapper.sourceforge.net/
Python SimpleJSON: https://github.com/simplejson/simplejson
Python PygreSQL: http://www.pygresql.org/
Loader: https://github.com/AstunTechnology/Loader
Thank You!




         www.archaeogeek.com
         @archaeogeek
         jocook@astuntechnology.com

Consuming open and linked data with open source tools

  • 1.
    5/31/2011 Consuming Open and LinkedData with Open Source Tools Or, how to do something useful with all this free data Jo Cook, Geospatial Consultant, Astun Technology
  • 2.
    Talk Outline ●The data and the distribution technologies explained ● The software stack explained ● Actually getting some data and doing something with it ● Gotchas ● Further resources and links
  • 3.
    The Data andTechnologies
  • 4.
    The Data andTechnologies
  • 5.
    The Data andTechnologies Point in Latitude/Longitude format (WGS84)
  • 6.
    The Data andTechnologies
  • 7.
    The Data andTechnologies
  • 8.
    The Data andTechnologies PREFIX naptan: <http://transport.data.gov.uk/def/naptan/> PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> PREFIX skos: <http://www.w3.org/2004/02/skos/core#> PREFIX geo: <http://www.w3.org/2003/01/geo/wgs84_pos#> SELECT DISTINCT ?stopPoint_airport ?name ?iataRef ?lat ?long WHERE { ?item a naptan:StopArea ; naptan:administrativeArea <http://transport.data.gov.uk/id/administrative- area/145> ; skos:prefLabel ?name ; geo:lat ?lat ; geo:long ?long ; naptan:stopPoint [ naptan:airport ?stopPoint_airport ] . ?stopPoint_airport naptan:iataRef ?iataRef . }
  • 9.
    The Data andTechnologies
  • 10.
    The Data andTechnologies
  • 11.
    The Data andTechnologies http://wallpapertree.com/wallpaper/Sad-Kitten/
  • 12.
    The Data andTechnologies
  • 13.
    The Data andTechnologies
  • 14.
    The Data andTechnologies
  • 15.
    The Data andTechnologies http://freeimagesarchive.com
  • 16.
    The Data andTechnologies
  • 17.
    The Data andTechnologies 1m4ge.blogspot.com
  • 18.
    The Data andTechnologies
  • 19.
    The Data andTechnologies
  • 20.
  • 21.
  • 22.
  • 23.
  • 24.
    Mitigation Keep your data here so you're not relying directly on the service for updates to customers (Maybe even keep it here too)
  • 25.
    Conclusions A dash ofall these, some readily available python libraries, and some relatively simple scripting gets you useful data for your own applications... ... and by using PostgreSQL to store your data, you can provide a buffer in case of api changes or outages... ... and even do some nifty manipulation to analyse and present the data differently!
  • 26.
    Conclusions Simples!
  • 27.
  • 28.
    Resources Linked Data Resourcesand Example Queries Best API for transport data: http://openuplabs.tso.co.uk/ O'Reilly Book: http://www.learningsparql.com/ Blog: http://talis-systems.com/blog/ Airport Query: http://getthedata.org/questions/137/where-can-i-get-a-list-of-all-airports-codes-names-locations Beginner Guide: http://www.pezholio.co.uk/2011/01/a-beginners-guide-to-sparqling-linked-data-part-1/ NHS Choices- RSS http://www.nhs.uk/servicedirectories/Pages/ServiceSearch.aspx Police Data- JSON/REST API: http://policeapi2.rkh.co.uk/api/docs/ CSV/SHP: http://www.police.uk/data Basic Software Resources Mapserver dynamic charting: http://mapserver.org/output/dynamic_charting.html Python SparQLWrapper: http://sparql-wrapper.sourceforge.net/ Python SimpleJSON: https://github.com/simplejson/simplejson Python PygreSQL: http://www.pygresql.org/ Loader: https://github.com/AstunTechnology/Loader
  • 29.
    Thank You! www.archaeogeek.com @archaeogeek jocook@astuntechnology.com