Category Archives: Open Data

LMI for All API released

Posted on by
Reply

I have written periodic updates on the work we have been doing for the UKCES on open data, developing an open API to provide access to Labour Market Information. Although the APi is specifically targeted towards careers guidance organisations and towards end users looking for data to help in careers choices, in the longer term it may be of interest to others involved in labour market analysis and planning and for those working in economic, education and social planning.

The project has had to overcome a number of barriers, especially around the issues of disclosure, confidentiality and statistical reliability. The first public release of the API is now available. The following text is based on an email sent to interested individuals and organisations. Get in touch if you would like more information or would like to develop applications based on the API.

The screenshot above is of one of the ten applications developed at a hack day organised by one of our partners in the project, Rewired State. You can see all ten on their website.

The first pilot release of LMI for All is now available and to send you some details about this. Although this is a pilot version, it is fully functional and it would be great if you could test it as a pilot and let us know what is working well and what needs to be improved.

The main LMI for All site is at http://www.lmiforall.org.uk/.  This contains information about LMI for All and how it can be used.

The APi web explorer for developers can be accessed at http://api.lmiforall.org.uk/.  The APi is currently open for you to test and explore the potential for  development. If you wish to deploy the APi in your web site or application please email us at graham10 [at] mac [dot] com and we will supply you with an APi key.

For technical details and details about the data go to our wiki at http://collab.lmiforall.org.uk/.  This includes all the documentation including details about what data LMI for All includes and how this can be used.  There is also a frequently asked questions section.

Ongoing feedback from your organisation is an important part of the ongoing development of this data tool because we want to ensure that future improvements to LMI for All are based on feedback from people who have used it. To enable us to integrate this feedback into the development process, if you use LMI for All we will want to contact you about every four to six months to ask how things are progressing with the data tool. Additionally, to help with the promotion and roll out of LMI for All towards the end of the development period (second half of 2014), we may ask you for your permission to showcase particular LMI applications that your organisation chooses to develop.

If you have any questions, or need any further help, please use the FAQ space initially. However, if you have any specific questions which cannot be answered here, please use the LMI for All email address lmiforall [at] ukces [dot] org [dot] uk.

 

LMI for All – coming soon

Posted on by
Reply

A quick and overdue update on the Labour Market Information for All project, which we are developing together with Raycom, the University of Warwick and Rewired State and  is sponsored by the UK Commission for Employment and Skills (UKCES).

LMI for All will provide an online data portal bringing together existing sources of labour market information (LMI) that can inform people’s decisions about their careers.  The database will contain robust LMI from national surveys and data sources providing a common and consistent baseline to use alongside less formal sources of intelligence. Due for release at the end of May 2013, access to the database will be through an open API. the results of queries can then be embedded by developers in their own web sites of apps. We will also provide a code library to assist developers.

The project builds on the commitment by the UK government to open data. despite this, it is not simple. As the Open Data White Paper (HM Government, 2012)highlights,  data gathered by the public sector is not always readily accessible. Quality of the data, intermittent publication and a lack of common standards are also barriers. A commitment is given to change the culture of organisations, to bring about change: ‘This must change and one of the barriers to change is cultural’ (p. 18).

We have talked to a considerable number of data providers including government bodies. It is striking that all have been cooperative and wishing to help us in providing access to data. However, the devil is in the detail.

Much of the data publicly collected, is done so on the condition that is is non disclosive e.e. that it is impossible to find out who submitted that data. And of course the lower the level of aggregation, the easier it is to identify where the data is coming from. And the more the data is linked, the more risk there s of chong qi cheng bao disclosure.

We have developed ways of getting round this using both statistical methods (e.g. estimation) and technical approaches (data aggregation). But it remains a lot of work preparing the data for uploading to our database. And I guess that level of work will discourage others from utilising the potential of open data. It may explain why, transport excluded, their remain limited applications built on the open data movement in the UK.

It may suggest that the model we are working on, of a publicly funded project providing access to data, and then providing tools to build applications on top of that data, could provide a model for providing access to public data.

In the meantime if you are interested in using our API and developing your own applications for careers guidance and support, please get in touch.

 

Anonymising open data

Posted on by
Reply

Here is the next in our occasional series about open and linked data. I wrote in a previous post that we are worki8ngt on developing an application for visualising Labour market Information for use in careers guidance.

One of the major issues we face is the anonymity of the data. fairly obviously, the mo0re sources of data are linked, the more possible it may become to identify people through the data. The UK information Commissioner’s Office has recently published a code of practice on “Anonymisation: managing data protection risk” and set up an Anonymisation Network. In the foreword to the code of practice they say:

The UK is putting more and more data into the public domain.

The government’s open data agenda allows us to find out more than ever about the performance of public bodies. We can piece together a picture that gives us a far better understanding of how our society operates and how things could be improved. However, there is also a risk that we will be able to piece together a picture of individuals’ private lives too. With ever increasing amounts of personal information in the public domain, it is important that organisations have a structured and methodical approach to assessing the risks.

The key points about the code are listed as:

  • Data protection law does not apply to data rendered anonymous in such a way that the data subject is no longer identifiable. Fewer legal restrictions apply to anonymised data.
  • The anonymisation of personal data is possible and can help service society’s information needs in a privacy-friendly way.
  • The code will help all organisations that need to anonymise personal data, for whatever purpose.
  • The code will help you to identify the issues you need to consider to ensure the anonymisation of personal data is effective.
  • The code focuses on the legal tests required in the Data Protection Act
Particularly useful are the Appendices which presents a list of key anonymisation techniques, examples and case studies and a discussion of the advantages and disadvantages of each. These include:
  • Partial data removal
  • Data quarantining
  • Pseudonymisation
  • Aggregation
  • Derived data items and banding
The report is well worth reading for anyone interested in open and linked data – even if you are not from the UK. Note for some reason files are downloading with an ashx suffix. But if you just change this locally to pdf they will  open fine.

Open data and Careers Choices

Posted on by
Reply

A number of readers have asked me about our ongoing work on using data for careers guidance. I am happy to say that after our initial ‘proof of process’ or prototype project undertaken for the UK Commission for Employment and Skills (UKCES), we have been awarded a new contract as part of a consortium to develop a database and open APi. The project is called LMI4All and we will work with colleagues from the University of Warwick and Raycom.

The database will draw on various sources of labour market data including the Office of National Statistics (ONS) Labour Force Survey (LFS) and the Annual survey of Hours and Earnings (ASHE). Although we will be developing some sample clients and will be organising a hackday and a modding day with external developers, it is hoped that the availability of an open API will encourage other organisations and developers to design and develop their own apps.

Despite the support for open data at a policy level in the UK and the launch of a series of measures to support the development of an open data community, projects such as this face a number of barriers. In the coming weeks, I will write a short series of articles looking at some of these issues.

In the meantime, here is an extract from the UKCES Briefing Paper about the project. You can download the full press release (PDF) at the bottom of this post. And if you would like to be informed about progress with the project, or better still are interested in being involved as a tester or early adapter, please get in touch.

What is LMI for All?

LMI for All is a data tool that the UK Commission for Employment and Skills is developing to bring together existing sources of labour market information (LMI) that can inform people’s decisions about their careers.

The outcome won’t be a new website for individuals to access but a tool that seeks to make the data freely available and to encourage open use by applications and websites which can bring the data to life for varying audiences.

At heart this is an open data project, which will support the wider government agenda to encourage use and re-use of government data sets.

What will the benefits be?

The data tool will put people in touch with some of the most robust LMI from our national surveys/sources therefore providing a common and consistent baseline for people to use alongside wider intelligence.

The data tool will have an access layer which will include guidance for developers about what the different data sources mean and how they can be used without compromising quality or confidentiality. This will help ensure that data is used appropriately and encourage the use of data in a form that suits a non-technical audience.

What LMI sources will be included?

The data tool will include LMI that can answer the questions people commonly ask when thinking about their careers, including ‘what do people get paid?’ and ‘what type of person does that job?’. It will include data about characteristics of people who work in different occupations, what qualifications they have, how much they get paid, and allow people to make comparisons across different jobs.

The first release of the data tool will include information from the Labour Force Survey and the Annual Survey of Hours and Earnings. We will be consulting with other organisations that own data during the project to extend the range of LMI available through the data tool.

LMI for All Briefing Paper

Open and Linked Data and Mediation

Posted on by
Reply

There has been an explosion of interest in Open Data and the potential for linking data to produce new social apps. Yet despite all this attentions, and the growing access to data in some countries such as the UK, the development of new apps has been less than impressive.

Rather than full apps, probably the main use has been the development of interactive visualizations allowing users to explore different data sets and quick visualisations of different data sets. The Guardian newspaper data blog has led the way in the UK and in particular has shown the value of open journalism such as in this discussion on how they got the colours of the maps right.

But the development of more advanced apps has been slower. Probably the biggest take off has been around transport allowing real time timetable tracking etc. But even here the problem of the social purpose and use of data apps is an issue. take this compelling app from the German newspaper Suddeutsche . Its hows graphic representations of train journeys in Germany, providing information on each train’s itinerary and the details of any delay. There is also an interactive timeline, allowing you to watch previous days’ travel play out. Its fun. But I can’t really see that it is much use! Or take this app – available in various forms – using crowd sourced data to find the nearest post box in the UK. Do we really need it? Why not just ask somebody / anybody?

In education there are a number of apps for finding schools etc. But there is little use of open and linked data for learning.

We have been working with a number of organisations to produce open and linked data apps for use in careers guidance. There are now three iterations of what we variously call a TEBO (Technologically Enhanced Boundary Object) or Careers dashboard.

The first was a quick demonstrator which we built to see how it might work. The second works through an API to the Careers Wales beta web site. And the third – more technically advanced – iteration is a database and API developed for UKCES which is not publicly available at present.

One of issues being raised in this work is mediation. In general government / agencies seem to regard data as just standing on its own. Within the TEBO concept we always stressed the need for social mediation and had ideas for a number of ways in which this might happen using social software e.g Question and Answer applications.

In fact mediation takes place at a series of levels – including the selection of data originally collected, and the way data is selected for use and display within an application. Different people will need different apps for interrogating the same data. For instance our Careers Dashboard may have potential interest and use for:

  • Young people thinking about career choices;
  •  Young people applying to further or higher education, seeking an apprenticeship or employment;
  • Adults who are newly unemployed;
  •  Long term unemployed adults;
  •  Adults considering re-entering education and training (e.g. women returners);
  • Adults thinking about a change in career direction (e.g. mid-career changers);
  • Parents and carers supporting young people wishing to enter further education, vocational training or employment
  • Career professionals – careers teachers, careers advisers and subject teachers; and
  • Various others (e.g. educational planners and policymakers, professionals preparing funding applications, researchers).

However, mediation seems to be commonly understood as intervention and then posed as a dichotomy between non intervention or intervention or to put it another way – let end users access to data or only let professionals access to data. This seems to me a misunderstanding of both the potentials and limitations of the data but of the potentially rich ways in which mediation happens and the ways in which technologically can be used in such processes.

It would be interesting to look at mediation within physical communities and through extended web and social media based communities. It would also be interesting to link mediation to the potential quality of careers interventions (i.e. after mediation takes place.)

More to follow…..