Jun 092015
At lunch. Sargents Cafe. Return of the "Kangaroos"

At lunch. Sargents Cafe. Return of the “Kangaroos”. Flickr: https://www.flickr.com/photos/state-records-nsw/18431553628/

State Records NSW is joining other Australian archives and libraries for a WW1-themed hackathon to be held at the State Library of NSW in conjunction with GovHack 2015 (3-5 July).

As the New South Wales State Government Archive, State Records NSW holds a number of datasets relating to the First World War, particularly relating to the home front and to the involvement of government employees in the war.

The datasets we’re making available for the hackathon are described in this post. Staff from State Records NSW will be at the State Library ready to assist teams understand and use the data. Hope to see you there!

Continue reading »

Jul 102014

This weekend (11-13 July 2014) State Records is returning for another GovHack.

We’ll be bringing with us two new sets of data:

  • approx 140,000 entries from our convict indexes (great for graphing and viz.)
  • nearly 1000 high res digitised sketches and plans from the NSW Surveyor General (a fantastic resource for a maps mashup… maybe with the National Map?)

Surveyor General sketch

Details of all of our datasets are available here.

This year we’ll be offering a prize to the best GovHack entrant using State Records NSW data (details when the prizes are announced).

If you are participating in GovHack this year, and are interested in working with our data, please get in touch with me (Richard Lehane) or Danny Archer during the event.


May 242013

State Records NSW is excited to be a part of Govhack 2013 on 31 May to 2 June. We’re offering three great datasets for developers to work with: check out the Datasets page.


State Records NSW’s catalogue API

With over a million entries, http://search.records.nsw.gov.au is our largest dataset. The entire catalogue, as well as its search functionality, are accessible via calls to a Rails-style web API described here: http://search.records.nsw.gov.au/usage. The record series and items entities describe the holdings of the State Archives collection and are, in their own right, a fantastic historical resource. Those descriptions are linked to entities representing the agencies and people who made the records and the business functions and activities that informed their work. These contextual entities give a rich picture of government in New South Wales and its activities right back to the colonial period.

OpenGov API

State Records NSW manages the OpenGov NSW website. This website is a repository for information published by NSW Government agencies, including Annual Reports and open access information released under the Government Information (Public Access) Act 2009 (GIPA Act). The site currently contains over 2000 publications. Metadata for those publications, links to the PDFs, and their extracted full text contents are accessible via calls to a web API described here: https://www.opengov.nsw.gov.au/api.

Soldier settlement indexes

Specially released for the Govhack 2013 event, these indexes were created as part of State Records NSW’s Volunteer Program, and are finding aids for series of records that relate to New South Wales’s soldier settlement scheme for discharged soldiers who served in World War I. To get a picture of soldier settlement, check out the Wikipedia article: Soldier Settlement (Australia). The indexes have the names and locations of returning soldiers and would be a great resource for a Centenary of Anzac project. They are also very suited to geolocation and visualisation. The indexes are in a CSV format and are available on the Datasets page.

Interested in using State Records NSW’s data during GovHack 2013?

I’ll be at the Sydney event: please grab me () if you are interested in working with any of our datasets. If you are at one of the other locations, you can reach me on the weekend at @richardlehane.

Oct 112011

State Records NSW is now inviting our regular users to trial http://api.records.nsw.gov.au as a new search tool for accessing the State Archives collection.

If you have tried this new search tool, and have feedback to give, we would love to hear it. We are actively developing the tool and would like to make it as useful and as intuitive as possible. So please post any feedback you have as comments to this blog post.

From time to time we’ll post project updates to this blog. Any posts of particular interest for regular users using http://api.records.nsw.gov.au as a search tool are being marked with the “Regular users” category (in the right-hand column).

So far there have been posts on:

So what’s the whole API thing about anyway?

On this site, and in other places, you may find that the new search tool is also being described as an API, or application programming interface. This is because http://api.records.nsw.gov.au isn’t just a search tool, it is also an interface for making the raw data underlying the catalogue accessible, particularly for re-use by developers.

It’s a bit like toy trucks. If most online catalogues are toy trucks that you can play with, but only using the features built-in by the manufacturer, then http://api.records.nsw.gov.au is a toy truck built from lego bricks.

Because it is an API, you can take the search tool apart and use its “bricks” (i.e. XML or JSON versions of the search results and entities) to create other things (such as this mashup of ministries entities), mix it with other sources of data (e.g. to create federated search portals), or even upload your own data (by creating applications that automatically tag or add comments to items in the catalogue).

Lego truck, by monkeyc.net (flickr)

This “API approach” also has a lot of value for State Records because it means we can make better use of our own data (for example, it makes it much easier for us to contemplate creating new tools like mobile phone applications that integrate with the catalogue).

That said, if you just want a toy truck (a simple but powerful search tool), and don’t want to worry about all this API business, that’s OK, because, at the end of the day, it is a toy truck too!



Aug 252011

Opening Hours, by bbodien (flickr)

State Records’ API used to have the odd distinction of being one of the few web services with opening hours (Mon-Fri 9am to 5pm, open weekends too). This wasn’t anything intentional, just a compromise we’d had to make in order to connect the API to our online catalogue’s live database which required a network link that is unfortunately routinely shut down.

Anyway, thanks to the hard work of a number of our staff (Nott, Damien and Ninh), I am pleased to announce that the API is now available 24/7, like any proper web service should be.

This will enable:

  • after hours research
  • late night hacking
  • and overseas visitors!

It also marks a point at which the API can be considered to offer a reliable and stable service on top of which other stuff can be built. Of course, this doesn’t mean we will stop experimenting & in fact we’ve recently added a lot of cool new read-write functionality (tags and comments) that we’re busily documenting and will properly announce soon.



Aug 192011

The API is currently (Fri 19 August) undergoing maintenance and won’t be available again until Monday (22 August).

The good news

Since launching we’ve been forced to close down at 5pm each evening as a result of a regular, scheduled shutdown of our network connection. This problem is being addressed by today’s maintenance and (fingers crossed) from next week the API will be accessible fulltime. This will enable more widespread use of the API and will provide the stability required by developers wishing to develop against it.

Aug 082011

Zotero is a free tool that helps you collect and manage research notes and references. It has many useful features including the automatic creation of bibliographies and footnotes, online back-up and syncing, and search and tagging of your notes and references. Zotero also integrates with many different websites (such as library catalogues, online journals or newspapers, and reference sites like Wikipedia) to automatically record appropriate citations when you are doing online research.

State Records’ new API (http://api.records.nsw.gov.au) supports the automatic capture of series and item citations by Zotero.

To try this out:

  1. download Zotero. If you use Mozilla Firefox as your browser, you can install it as a browser plug-in. Otherwise, install the standalone version.
  2. navigate to the series or item in the API that you would like to cite. E.g. http://api.records.nsw.gov.au/series/1
  3. click the scroll icon in your browser’s address bar to automatically capture the citation.

Add series to ZoteroTo check that the citation has been correctly added, bring up the Zotero screen (Ctrl-Alt-Z) and you should have a new item in your Zotero library. You can append your research notes or attach digital images to this item.

In Zotero

Jul 072011

Relationships are key to the way that State Records NSW describes archives. Descriptions of individual items and series depend on their links to each other, to the agencies or people that created them, and to their role in government business (functions and activities), for their full meaning.

I am therefore very pleased to announce that relationships between entities are now included in State Records NSW’s API.

Relationships are visible when you visit pages for individual entities in the API, e.g.:

Entity relationships in the API

And of course, because this is an API, developers can access these relationships (in multiple formats such as XML and JSON) through logical URLs such as http://api.records.nsw.gov.au/series/1/persons.xml. For full details, see the documentation: http://api.records.nsw.gov.au/usage.

Many thanks to Wisanu Promthong (aka Nott), State Records NSW’s new Systems developer, Digital Archives, for implementing these additions to the API.

Jun 272011

I am excited to announce the addition of a new ‘Open archival data’ link (on the right of this page): Archival Data – Public Record Office Victoria.

As part of its project to release raw archival data for re-use, Public Record Office Victoria (PROV) is making agency and government function descriptions available for download as XML (agency data in the EAC-CPF format, function data in a format based on EAC-CPF). PROV is also considering future steps such as the publication of series and items data and the development of an API.

State Records NSW’s agency and function data is available in XML from the datasets page and through the new API. There is fantastic scope for combining this data with PROV’s (and indeed with similar data released by the National Archives of Australia). By doing so, what can we learn about Australian administrative history? How do the functions of Victorian and New South Wales government compare? Get mashing!

Jun 232011

State Records NSW’s new API is designed primarily as a framework to allow the development of new web services (both internally by State Records staff and by external developers). Nevertheless the creation of the API has provided us with an opportunity to experiment with new ways of presenting collection search results and this aspect of the project may be of interest to all researchers using the collection.

In this post I describe key features of the API’s collection search and also some of the more advanced functionality you can access ‘under the hood’. To try the collection search yourself, go to: http://api.records.nsw.gov.au.

An example

Example search results using the new API

What, why and who

Rather than presenting search results as a simple list, the API’s collection search provides a structured view, clustering results according to three questions:

  • what records (both record series and individual items) relate to the query?
  • why might records relating to the query have been created by Government (Government functions and activities)?
  • who in Government (agencies and people) might have created records relating to the query?

(For those interested in archival theory, this three-part division matches Australian archivist Chris Hurley’s conception of archival description as comprising three essential types of entity: documents, deeds and doers.)

Simple search, but not too simple

The new search box might look ‘simple’ but sophisticated searching of the catalogue is still possible.

Swamped with too many hits? Use the two ‘filters’ in the right-hand column of the results page to drill down to more relevant results. The date filter narrows results by date range. The series filter allows you to see at a glance the key record series relating to your query and narrow your results to particular series.

If you are a ‘power user’ you can include these filters in your initial search by adding the following special keywords to your query:

  • entities:[Item,Series,Function,Activity,Person,Agency]
  • series:[series id number]
  • from:[year]
  • to:[year]

For example, the following query will just return record items dating between 1900 and 1950:

Custom search

Incorporating the new collection search into your browser

Fallen in love with the new collection search? If you use a modern version of firefox or internet explorer, you can take it with you anywhere you go on the internet by including it amongst the ‘search providers’ in your browser’s search box (next to the main address box). This will allow you to quickly search State Records’ collection wherever you happen to be browsing. To do this:

  1. go to http://api.records.nsw.gov.au
  2. if you are on internet explorer, do this:

Or, if you are on firefox do this:

Got suggestions?

The API’s collection search is, like the API itself, still in an experimental mode. If you have any suggestions for how it might be improved we would love to hear them (and we might try to implement them). Please post your ideas as comments to this post.