Open government data site

The Digital Engagement team are inviting developers to show government how to get the (forthcoming) UK government data site right. For now you need to apply to preview and contribute ideas via a Google Group.

With over 1000 existing data sets, from 7 departments (brought together in re-useable form for the first time) and community resources, we want developers to work with us to use the data to create great applications; give us feedback on the early operational community; and tell us how to develop what we have into a single point of access for government-held public data.

Down Under data.australia.gov.au has just launched.

US Data.gov launched in May. Its goal is nicely put:

A primary goal of Data.gov is to improve access to Federal data and expand creative use of those data beyond the walls of government by encouraging innovative ideas (e.g., web applications). Data.gov strives to make government more transparent and is committed to creating an unprecedented level of openness in Government. The openness derived from Data.gov will strengthen our Nation’s democracy and promote efficiency and effectiveness in Government.

OPSI releases legislation API

OPSI is some way down the road in developing a new Single Legislation Service (SLS) which will combine legislation from the OPSI website and the SLD. As I understand it the SLS will effectively combine and integrate the following legislation data resources:

• the “as enacted” versions of legislation from OPSI, immediately on enactment
• the revised versions of legislation from the SLD, as and when available, complete with all versioning and annotation information
• the tables of effects data maintained by the SLD which will link past legislative provisions to relevant amending provisions
• the explanatory notes, integrated with the relevant legislative provisions

Key to the utility of the service and opening up the information has been the development of a permanent URI scheme for addressing legislation, legislation fragments, versions and related resources on the new site. Say Goodbye to link rot and Hello to an intuitive format with which any external individual or website can reliably construct an address to any legislative resource. This is not just a technical nicety, but a fundamental improvement that will open up UK legislation for public consumption.

The API is now documented on the site www.legislation.gov.uk.

Pilot Project development under way

I’m pleased to report that last week we received our first funding commitment – from OPSI – sufficient to proceed in earnest with the first phase of development of the FreeLegalWeb Pilot Project.

At the core of FLW will be what we call the Citator – essentially linked data tables identifying and connecting everything we know about the law: both primary resources such as legislation, cases and other official documents, and secondary law resources such as publications and articles. Using the Citator we will be able to request data both internally from FLW and externally from the web via an API which will be made publicly available so that any external website can leverage that information.

In the UK OPSI is the key facilitator of public sector information policy, setting standards and providing a practical framework of best practice for opening up and encouraging the use of PSI. It also has direct responsibility for the publication (inter alia) of legislation and is advanced in its development of a new legislation website which will provide an API enabling direct addressing of legislation data and resources. Via the Citator FLW will both exploit the facilities the new legislation website offers and also help contribute to OPSI’s PSI policy agenda. On this basis OPSI is part-financing the development of the Citator.

A big thanks to John Sheridan at OPSI who has been a leading FLW supporter from the outset. Indeed it was he who first encouraged me to go public with the initiative last autumn.

We have some way to go in securing sufficient funds to commit to development of the full Pilot Project but are nevertheless proceeding with other aspects of development in the expectation that some of our other current funding prospects will come good.

POI Task Force report

The Power of Information Task Force has just published its final report describing the key actions that government can take in the short to medium term significantly to improve its use of digital technologies.

Of particular relevance to FreeLegalWeb is the section on modernising data publishing and reuse.

Improving access to legislation

I’ll refer again to Lord Justice Toulson’s comments (at [2008] EWCA Crim 2467) that:

It is a serious state of affairs when the relevant legislation is not accessible, the Government’s own public information website (OPSI) is incomplete and the [court is unintentionally misled] as to the relevant Regulations in force. [This is a] problem of substantial constitutional importance.

He was echoing – and specifically referred to – the Law Commission’s recommendations in its 2006 report on Post-Legislative Scrutiny (Law Com 302,  at para 4.11):

We are concerned when information so fundamental to a democracy is difficult to identify, obtain and understand, and is frequently out of date.

How do we fix this?

There are three ways to address the problem – and all three need to be pursued with equal vigour.

Firstly, let’s improve access to existing legislation. This is the task OPSI have and it is one which FreeLegalWeb is addressing. But this is applying sticking plaster to the wounds.

So, let’s improve legislation via new legislation! The Leader of the House of Commons is on the case, describing the several ways in which the Government is simplifying and consolidating legislation. But that does not address the huge corpus of remaining existing fragmented legislation.

The third way we can improve access to legislation will be by far the most effective and it does not appear yet to have attracted any public discussion – and that is to improve the legislative drafting process.

Legislation is drafted by the Office of Parliamentary Counsel and it is therefore to the OPC that we should look for the answer. To do their job in preparing amending legislation OPC need to work from a consolidated version of the current legislation. Why cannot that be made available to us? (Do LexisNexis own it?!) And if, in drafting amending legislation, they are updating existing legislation, why cannot they apply technology to automate the process?

In 2004 Tim Arnold-Moore of the Royal Melbourne Institute of Technology published an excellent paper on Automating consolidation of amendments to legislation in common law and civil jurisdictions which gives a full background to drafting practices in commonwealth jurisdictions and discusses the reasons for extending the traditional role of the official drafting office to the management of the legislative repository:

They have historically maintained consolidations of legislation for their own use to facilitate the drafting of amending legislation. Rather than duplicating the effort to consolidate the amendments, the public resources used to maintain this repository should be for the benefit of all users of legislative material. If these versions are considered authoritative enough to be used as the basis for amending legislation, they should be considered sufficiently reliable for the government to endorse as authoritative. The cost of delivering these consolidations in a timely fashion to the public is a small proportion of the cost of manually maintaining these consolidations and there is no excuse for governments failing in this respect. Providing older consolidations is simply a matter of retaining old versions and providing a suitable mechanism to search and browse them.

The article also describes how the “redlining” of amendments on the consolidated master version of the legislation can be used automatically to generate the amending legislation. This stuff is happening down under.

Tasmania, with its automated consolidation system, usually has consolidated legislation available on the day the amendments commence if not before. Contrast this with the vast resources of the US Congress where updated versions of the US Code are made available as long as 2 years after the amendment becomes law.

Sadly the UK lags even further behind, with the SLD only substantially up to date to 2002.

See also AustLII’s Point-in-Time legislation system: A generic PiT system for presenting legislation.

Permanent URIs – a small(ish) step, big dividends

Jeni Tennison has provided on her blog a very helpful explanation of URIs (Uniform Resource Identifiers) and why the simple step of establishing permanent identitifier URIs for public sector data resources will be so helpful in opening those resources up for public consumption.

Identifier URIs are what are really important when it comes to opening up your data. They shift the focus from the documents that you serve to the resources that they are about. By assigning URIs to resources, you enable other people to talk about them. Even if that’s all you do, you have done good.

Identifier URIs don’t identify particular documents or particular views or versions of the information; they identify the idea. They can then be used to direct the user to particular documents or views wherever they may be. So, no matter how much the structure of the website changes over time, the URI will remain a reliable, permanent means of addressing the information.

For a law resource a suitable identifier URI can, typically, be constructed from its citation. Jeni and others have recently been responsible for settling for OPSI a URI scheme for legislation and we can look forward to publication of that scheme very soon. So, for example, ukpga/2008/32 will be the identifier for (UK Public General Act) 2008 c. 32. That simple, reliable, permanent address can then be used by OPSI to direct us to a particular view of the information, for example the web document for the Act.

A small step, but a huge boon “for anyone to re-present, mash-up, analyse and generally do whatever they want to do”.

Now wouldn’t it be good if we could look forward to similar schemes being adopted by the courts? Unfortunately there are a host of problems there, not least that the High Court does not publish its own judgments. And although the systems of standard neutral citation adopted in recent years by the higher courts in all jurisdictions and in some other courts and tribunals provide an obvious basis for identifier URIs, settling sensible schemes for older judgments and in other courts requires a good deal more thought.

Although only the courts can settle URIs for their own resources, we must do what we can to inform and influence that process.

Ignorance of the law

I referred earlier to Lord Justice Toulson’s comments in R v Chambers [2008] EWCA Crim 2467 which I felt highlighted the need for FreeLegalWeb. His comments in full:

64.  … It is a maxim that ignorance of the law is no excuse, but it is profoundly unsatisfactory if the law itself is not practically accessible. To a worryingly large extent, statutory law is not practically accessible today, even to the courts whose constitutional duty it is to interpret and enforce it. There are four principal reasons.

65. First, the majority of legislation is secondary legislation.

66. Secondly, the volume of legislation has increased very greatly over the last 40 years …

67. Thirdly, on many subjects the legislation cannot be found in a single place, but in a patchwork of primary and secondary legislation.

68. Fourthly, there is no comprehensive statute law database with hyperlinks which would enable an intelligent person, by using a search engine, to find out all the legislation on a particular topic. This means that the courts are in many cases unable to discover what the law is, or was at the date with which the court is concerned, and are entirely dependent on the parties for being able to inform them what were the relevant statutory provisions which the court has to apply. This lamentable state of affairs has been raised by responsible bodies on many occasions, including the House of Lords Committee on the Merits of Secondary Legislation. In its Report on Post-Legislative Scrutiny BAILII: [2006] EWLC 302, under the heading “Access to legislation and consolidation”, the Law Commission stated:

“4.11. One theme related to delegated legislation, on which a number of consultees commented, was access to legislation. The joint response of the Children’s Legal Centre and National Children’s Bureau addressed the problem that despite their familiarity with the broader legal framework, they still found access to be a real problem:

‘The lack of access to statutes with appropriate links to the regulations and guidance which are currently in force must be a cause of serious inconvenience to anyone who does not have access to specialist services. We are concerned when information so fundamental to a democracy is difficult to identify, obtain and understand, and is frequently out of date. It is frequently the case that secondary legislation and guidance are overlooked in the process of scrutiny, although their impact on the day-to-day operation of the law is as significant as the primary statute.’

4.12. The joint response stated that experience of practice in childcare suggests that many injustices are the result not of failure to comply with the statute, but of failure to know about, understand or access secondary legislation.”

69. The problem is not confined to secondary legislation relating to childcare. It affects many other areas of law of great impact on the ordinary citizen, such as social security benefits.

70. The Law Commission concluded this section of its report as follows:

“4.15. It is also important that all related statutory provisions, whether primary or secondary, should be capable of being readily accessed together. We are aware of the work being undertaken on the Statute Law Database and recognise that public access to that resource is a step in the right direction. We recommend that steps should be taken to ensure that the related provisions of primary and secondary legislation should be capable of being accessed in a coherent fashion by a straightforward and freely available electronic search.”

71. The Government’s response to that recommendation was presented to Parliament in March 2008, CM 7320. It stated as follows:

“35. Her Majesty’s Stationery Office (HMSO) and the Statutory Publications Office (SPO), which produces the Statute Law Database, are to work together to create a single, powerful and free to access online legislation service. The launch of the SLD has been a milestone in government’s online legislation publishing.

36. Over the last two years HMSO, via the OPSI website (www.opsi.gov.uk) has embarked on wide ranging improvements to how legislation is published online, taking account of key usability features for layout and navigation. This work is being undertaken as part of ‘The Transforming Legislation Publishing Programme’. The aim has been to present legislation in the most accessible and usable way [my emphasis], whilst maintaining the traditional strengths of immediacy and accuracy. One of the benefits is that it affords the opportunity to provide links to related information. Initially these links will be to the Explanatory Note for Acts or the Explanatory Memorandum for Statutory Instruments. Alongside this is also published an ATOM feed for the piece of legislation. This provides visitors with an easy way to keep up to date with subsequent additions to the website, like the addition of Explanatory Notes for an Act, and also the enacting or making of other related legislation such as Commencement Orders or, longer term, amending legislation. In future HMSO will be adding explicit links to Commencement Orders, and where legislation implements an EU Directive, a link also to that Directive.

37. HMSO/OPSI and SPO will continue to work together and with government’s online legislation visitors, to improve the service and ensure that UK legislation is available in a high quality and straight forward terms, with a freely available and powerful search.”

72. The aim is laudable, indeed imperative, but there is a long way to go and meanwhile the volume of legislation advances apace. It is a serious state of affairs when the relevant legislation is not accessible, the Government’s own public information website (OPSI) is incomplete and the prosecution in an excise case unintentionally misleads the court as to the relevant Regulations in force. Although the problem has in this case arisen in an excise context, it is part of a wider problem of substantial constitutional importance.

Now you may say this is a problem for Parliament to resolve via legislative reform. I’d go with that, but it won’t happen soon enough and wouldn’t have retrospective effect. Or you may say it is for Government to resolve by extending the SLD consolidation work to cover secondary legislation. I say, pigs might fly: it’s not on the cards and, if it were, would optimistically take another ten years, or – more likely – never be finished. So, in practical terms, we are left with OPSI improving access to the relevant as-enacted secondary legislation. OPSI is working hard to improve matters. But it doesn’t have all the answers, and it know this. The Power of Information agenda acknowledges that government shouldn’t try to be Big Brother, but provide better access to government information which will encourage innovative services to be developed which support the government’s own efforts. In respect of better access to the law, FreeLegalWeb has stepped forward and our award in the ShowUsABetterWay competition is recognition that – with support and encouragement – we can make a difference. There is no magic bullet that will solve the burgeoning secondary legislation problem, but we’ll be working with OPSI and the POI Task Force to improve matters substantially.

More accessible consultation papers

Props to Harry Metcalfe for bending Lord Norton’s ear:

In a recent post, I identified problems with ensuring that government consultations are accessible to interested members of the public. Consultation papers are sent to established groups on a Department’s mailing list but otherwise often just placed on the Department’s website as a means of reaching the public.

As a result of the post, I received some extremely helpful responses, not least from MJ Ray, Dave Briggs and Harry Metcalfe. I have since had a very useful meeting with Harry Metcalfe, who is responsible for the tellthemwhatyouthink.org website. I now have a much clearer idea of what Government should be undertaking in order to make its consultation papers more accessible to the public. I shall be tabling some parliamentary questions this week in order to pursue the issue.

… this blog has certainly proved its worth.

Help Free Our Bills

Tom Steinberg and all the mySociety team are asking you nicely to write to your MP to help Free Our Bills:

Today we just want you to do one thing to help TheyWorkForYou’s campaign to get Parliament to publish bills in a better way, if you’d be so kind.

Please go to WriteToThem and ask your MP to sign Early Day Motion (EDM) 2141.

The more MPs that sign the more the house authorities will realise how this is an issue of wide public interest, and that there are thousands of people behind Free Our Bills.

Thanks very much, and do let us know if your MP replies with anything interesting!

All to the good of the FreeLegalWeb too of course. So I’ve written; you write too.

Who owns the law?

In Who Owns the Law? in the New York Times, Noam Cohen investigates why, though US laws and judicial opinions are public domain, a variety of organisations — from trade groups and legal publishers to the government itself — claim copyright in it by virtue of the “accoutrements” surrounding the public material. “So while the laws and court decisions themselves may be in the public domain, the same is not necessarily true for the organizational system that renders them intelligible or the supporting materials that put them into context.”

The position is similar here. Legislation and judgments are Crown copyright, though OPSI licences their use on a generous open basis via its Click-Use licence. But if a private publisher or the government itself adds some value, whether just presentational, that added value is the copyright of the secondary publisher.

We had that argument over the Statute Law Database where the government’s position was initially that its consolidation and annotation of the legislation was a value-added service. In the end – hurrah – it was nevertheless declared “open”. But don’t expect LexisNexis or Thomson or ICLR or TSO, or even BAILII, to follow suit with respect to their added value.

If the government publish the information in the most open way possible, no-one will have a stranglehold over the law. The government now embraces this argument … and here we are to make the most of it.