Forum posts to 'Developing the site '

Re: Browsing by data format.

Thu, 19 Aug 2010 14:42:13 +1200

Strongly agreed!

Posted to: Browsing by data format. Show Thread | Post Reply

User survey

Anthony Hawkins — Fri, 26 Mar 2010 09:43:08 +1300

Please help us improve the site by taking our user survey. Closes 15 April 2010.

http://www.surveygizmo.com/s/266272/data-govt-user-survey

Posted to: User survey Show Thread | Post Reply

Re: Filter for geographic data

Tue, 09 Feb 2010 17:28:57 +1300

+1 for ability to geographically filter the data.

Posted to: Filter for geographic data Show Thread | Post Reply

Re: I don't seem to be able to see the bit where I can actually get to the data

Tue, 09 Feb 2010 17:27:30 +1300

This is a good first step, but not overly useful. Google already does a "pretty good job".

I'm sitting on the edge of my seat in anticipation. When are we going to have an API that we can directly query the data???

Posted to: I don't seem to be able to see the bit where I can actually get to the data Show Thread | Post Reply

Browsing by data format.

Sun, 06 Dec 2009 13:06:21 +1300

I think that browsing by data type format would be very useful.

I am currently only interested in datasets that are provided in KML format. I can't see how I can limit this at the the moment which means I have to scroll through everything!.

Unless you can provide me with a list ;-)

Cheers,
Kim.

Posted to: Browsing by data format. Show Thread | Post Reply

Re: Guidance on creating datasets

Anthony Hawkins — Thu, 26 Nov 2009 08:42:05 +1300

Hi Kimeros

As part of this work, we'll be contributing to guidance on that for departments. Which is to say the guidance doesn't exist yet ... It will be created as part of cross-govt work, and we consider it high priority. However -

Data.govt.nz indexes: CSV, Spreadsheets, XML, geospatial formats (increasingly - working with LINZ on this to get it right) and HTML. We index HTML because the benefits (data importance, ‘pastability’), as this stage, outweigh the disadvantages. We let people know where there's an API. We don’t index PDFs.

When we talk to agencies we ask them to release or re-format their data in CSV and, if they want, as spreadsheets. Where they are providing HTML from databases, we recommend enabling users to export it in open formats. Providing feeds also important.

Have found that the issues aren't so much technical as 'business': establishing who in the agency owns the data, what status it has (offical, advisory, caveat-heavy or good-to-go), and sometimes who in the business pays for conversion, release, updating, etc. That's just for converting data, not releasing new data. The issues are often the same as those for getting agencies to use web standards.

Other info:

The site currently includes, or will include, the following broad types:

* “Reference” datasets and quick-changing datasets. Reference: eg, region definitions, political borders. Quick-changing: eg, teacher registrations, real-time river flows, traffic reports.
* Statistical and non-statistical. Statistical: eg, pinus radiate logging prices. Non-statistical: eg, threatened environments classification.
* "Authoritative" and non-authoritative data. Authoritative: eg, Current account balance. Non-authoritative: eg, traffic webcams (API)

Then there's the UK (RDFa etc) ...

http://cloudofdata.com/2009/07/talking-with-mark-birbeck-about-rdfa-and-its-use-in-government/

Be good to get yr thoughts on that.

Posted to: Guidance on creating datasets Show Thread | Post Reply

Guidance on creating datasets

Tue, 24 Nov 2009 14:58:23 +1300

Hi,

I haven't been able to find Guidance on this website for Government Departments around:

Best practice for creating datasets and the recommended formats.

Could you please direct me to this information if it is available or consider providing it if it is not.

Regards,
Kimeros.

Posted to: Guidance on creating datasets Show Thread | Post Reply

Re: I don't seem to be able to see the bit where I can actually get to the data

Anthony Hawkins — Thu, 19 Nov 2009 12:20:46 +1300

Hi Kuraki. Google's good, but there's a lot of stuff on our site that you'll find only after many results pages' worth of searching. We also check it with agencies where we need to, add metadata, license info etc. It's specialist site, Google's not. You'll be pleased to hear the site was built in-house, with min spending. Cheers

Posted to: I don't seem to be able to see the bit where I can actually get to the data Show Thread | Post Reply

Re: Separation of time-series from reference data?

Anthony Hawkins — Thu, 19 Nov 2009 12:14:53 +1300

Hi Bruce - good idea. We're reviewing metadata at the moment and will include this suggestion on the list.

Posted to: Separation of time-series from reference data? Show Thread | Post Reply

Re: I don't seem to be able to see the bit where I can actually get to the data

Wed, 18 Nov 2009 21:39:19 +1300

This site is useless and a waste of tax payer's money. I believe that all of this links can be googled. Did they just do this so that they can do something?

Posted to: I don't seem to be able to see the bit where I can actually get to the data Show Thread | Post Reply

Re: I don't seem to be able to see the bit where I can actually get to the data

Anthony Hawkins — Fri, 13 Nov 2009 10:45:19 +1300

Hi there, our definite preference is for open formats. As you say though, its about progress at this stage. Other parts of the public service are looking at standards (or at least coherent guidance) in this area, and that's a priority. We've been concentrating on making the data findable, in the first instance.

Where we index a non-open format dataset (or HTML dataset), we contact the owner agency and ask it be converted to something better.

Re linking to referring HTML pages rather than direct to the datasets, we need to let people see the licencing info and sometimes data disclaimer info. Possibly later this could be included in XML feeds or similar, removing the need, but at this stage we need to do this. Cheers

Posted to: I don't seem to be able to see the bit where I can actually get to the data Show Thread | Post Reply

Re: I don't seem to be able to see the bit where I can actually get to the data

Thu, 12 Nov 2009 15:38:21 +1300

Data formats are also crucial xls is "good step forward" but still requires excel installed to read it properly.

Promoting XML or any other open standards would be appreciated

The other issue we have run into are govt sites that have data that you can not get to from a URL but require button clicks.. also not good for data mining

And requires us to write tedious documents about how to reach the data

Posted to: I don't seem to be able to see the bit where I can actually get to the data Show Thread | Post Reply

Separation of time-series from reference data?

Thu, 12 Nov 2009 09:12:16 +1300

Disclaimer: novice opinion.

When looking at the data sets I was most interested to find the geographic and other snapshot reference data sets. This is "slow to change" stuff like region definitions, political borders, populations, hell even names of all the cities, regions, etc in NZ. I know some of this is available for $$$ but it is basic reference data useful for all kinds of NZ apps.

1. Should I expect this kind of data here?
2. Is there a name for "hard" data versus statistical reports or am I inventing this difference?
3. Any plans to move to a static URL redirection scheme so web-apps can periodically use the "latest" version of data?

Don't get me wrong though, I'm very happy to see what is here already.

Cheers,
Bruce

Posted to: Separation of time-series from reference data? Show Thread | Post Reply

Re: Filter for geographic data

Anthony Hawkins — Fri, 06 Nov 2009 14:10:42 +1300

Hi Ed - geo's not something we've catered for in detail with the launch, but we're working on this with other agencies now. In the meantime, searches on "geo" and "land" will return results.

Posted to: Filter for geographic data Show Thread | Post Reply

Filter for geographic data

Fri, 06 Nov 2009 12:48:42 +1300

Some kind of cross-site filter or category for showing only entries with geographic data types would be very handy.

Posted to: Filter for geographic data Show Thread | Post Reply

Re: I don't seem to be able to see the bit where I can actually get to the data

Fri, 06 Nov 2009 08:41:33 +1300

Mandating is not something we can decide, though since launch we've had a number of agencies asking what we'd like and how they can help. Standardising formats and creating solid internal processes about data release (including dealing with URL persistence) is the key. We're beginning to look at this now with other agencies and hopefully non-govt people.

Posted to: I don't seem to be able to see the bit where I can actually get to the data Show Thread | Post Reply

Re: should this site valiidate

Fri, 06 Nov 2009 08:23:27 +1300

It certainly should validate. Apologies - onto this now.

Posted to: should this site valiidate Show Thread | Post Reply

Re: I don't seem to be able to see the bit where I can actually get to the data

Thu, 05 Nov 2009 23:04:11 +1300

I guess there is either a whole of government approach or not

If this is data.govt.nz surely you should be mandating agencies to do it right. Is thia not how the nz govt standards were done

However the nz govt standards illustrate how mandating don't really help get a good result - even if some agencies complied there has still been shoddy practice.

Thinking notably DIA and Commerce both have really bad sytems that create both ugly URIs and keep changing URI stucture without adequate redirection - to trust a URL we need persistence.

So i dont really care what you are asking agencies/departments - I would rather see what you are saying should happen.

Posted to: I don't seem to be able to see the bit where I can actually get to the data Show Thread | Post Reply

Re: Download of Metadata

Rowan — Thu, 05 Nov 2009 10:39:46 +1300

Hi there - you can get take a feed of most of it from the "Browse" page. Currently working on expanding the fields in the feed, I'll advise progress soon.

Posted to: Download of Metadata Show Thread | Post Reply

Re: I don't seem to be able to see the bit where I can actually get to the data

Anthony Hawkins — Thu, 05 Nov 2009 08:10:18 +1300

Hi there. At the moment the records link to a HTML pages, rather than directly to the data. That's usually when those pages contain an index to multiple datasets, or when there's some important caveat included about the data (or any info at all - our summaries for the record summaries are short ). Documenting APIs and other ways into the data are things we're definitely looking at. We're also encouraging depts to provide things like auto-updating xml sitemaps. That kind of discussion is a big part of the project, but in the meantime, we've erred on the side of html.

Cheers

Posted to: I don't seem to be able to see the bit where I can actually get to the data Show Thread | Post Reply

I don't seem to be able to see the bit where I can actually get to the data

savagehardy — Wed, 04 Nov 2009 22:59:28 +1300

This just seems to be a list of some websites that have a spreadsheet or similar

How do I query the data - is there an open search or similar method i can get stuff

Are there documented APIs

or is it just about downloading spreadsheets

A link like this does not seem to be data
http://data.govt.nz/dataset/show/633
just returns a web page even though you say Format : Spreadsheet
http://www.med.govt.nz/templates/ContentTopicSummary____20511.aspx

10 clicks later i may find a spreadsheet
http://www.med.govt.nz/upload/37417/coal_table_1.xls

should this level of spreadsheet not be catalogued?

or maybe my idea of data is confused...

Posted to: I don't seem to be able to see the bit where I can actually get to the data Show Thread | Post Reply

should this site valiidate

savagehardy — Wed, 04 Nov 2009 22:49:23 +1300

http://validator.w3.org/check?verbose=1&uri=http%3A%2F%2Fdata.govt.nz%2F

51 Errors, 1 warning(s)

http://jigsaw.w3.org/css-validator/validator?profile=css21&warning=0&uri=http%3A%2F%2Fdata.govt.nz%2F

Posted to: should this site valiidate Show Thread | Post Reply

Download of Metadata

barnaclebarnes — Wed, 04 Nov 2009 14:18:49 +1300

Can you provide a download of this metadata? I would like to include it in the Open Data Catalogue as an authoritative source.

Thanks,
Glen

Posted to: Download of Metadata Show Thread | Post Reply