Map of school enrolment

Our Data Diva Michael Bauer spent his last week in Dar Es Salaam working with the Ministry of Education, the Examination Council, and the National Bureau of Statistics on joining their data efforts.

As in many other countries, the World Bank is a driving force behind the Open Government Data program in Tanzania, both funding the initiative and making sure government employees have the skills to provide high-quality data. As part of this, they have reached out to School of Data to work with and train ministry workers. I spent the last week in Tanzania helping different sources of educational data to understand what is needed to easily join the data they collect and what is possible if they do so.

Three institutions collect education data in Tanzania. The Ministry of Education collects general statistics on such things as school enrollment, infrastructure, and teachers in schools; the Examination Council (NECTA) collects data on the outcomes of primary and secondary school standardized tests; and finally, the National Bureau of Statistics collects the location of various landmarks including markets, religious sites, and schools while preparing censuses. Until now, these different sets of data have been living in departmental silos. NECTA publishes the test results on a per-school level, the ministry only publishes spreadsheets full of barely usable pivot tables, and the census geodata has not been released at all. For the first time, we brought these data sources together in a two-day workshop to make their data inter-operable.

If the data is present, one might think we could simply use it and bring it together. Easier said than done. Since nobody had previously thought of using their data together with someone else’s before, a clear way of joining the datasets, such as a unique identifier for each school, was lacking. The Ministry of Education, who in theory should know about every school that exists, pushed hard for having their registration numbers used as unique identifiers. Since this was fine for everyone else, we agreed on using them. First question: where do we get them? Oh, uhhm…

There is a database used for the statistics created in NECTA’s aforementioned pivot table madness. A quick look at the data led everyone to doubt its quality. Instead of a consistent number format, registration numbers were all over the place and needed to be cleaned up. I introduced the participants to OpenRefine for this job. Using Refine’s duplicate facet, we found that some registration numbers were used twice, some schools were entered twice, and so on. We also discovered 19 different ways of writing Dar Es Salaam and unified them using the OpenRefine cluster feature—but we didn’t trust the list. On the second day of the workshop, we got our hands on a soft copy (and the books) of the school registry. More dirty data madness.

After seeing the data, I thought of a new way to join these datasets up: they all contained the names of the schools (although these were written differently) and the names of the region, district, and ward the schools were in. Fuzzy matching for the win! One nice feature Refine supports is reconciliation: A way of looking up entries against a database (e.g. companies in opencorporates). I decided to use the reconciliation service to look up schools in a CSV file using fuzzy matching. Fuzzy matching is handy whenever things might be written differently (e.g. due to typos etc.). Various algorithm help you to figure out which entry is closest to what you’ve got.

I went to work and started implementing a reconciliation service that can work on a CSV file, in our case a list of school names with registration numbers, districts, regions, and wards. I built a small reconciliation API around a fuzzy matching library I wrote in Clojure a while back (mainly to learn more about Clojure and fuzzy matching).

But we needed a canonical list to work from—so we first combined the two lists, until on the third day NECTA produced a list of registration numbers from schools signing up for tests. We matched all three of them and created a list of schools we trust, meaning they had the same registration number in all three lists. This contained a little less than half of the schools that allegedly exist. We then used this list to get registration numbers into all data that didn’t have them yet, mainly the NECTA results and the geodata. This took two more packed days working with NECTA and the Ministry of Education. Finally we had a list of around 800 secondary schools where we had locations (the data of the NBS does not yet contain all the regions), test results, and general statistics. Now it was all a matter of VLOOKUPs (or cell.cross in Refine), and we could produce a map showing the data.

After an intensive week, I left a group of government clerks that now had an air of breaking for new borders. We’ll continue to work together, getting more and more data in and making it useable inside and outside its institutions. Another result of the exercise is reconcile-csv, the fuzzy matching reconciliation service developed to be able to join messy datasets like the ones on hand.

Photo credit: UNDP

As shown in the Open Data Index, Moldova made impressive progress in opening up government information, ranking 12th out of 70 countries assessed in the index. This is largely thanks to the e-Government Centre which led the country through a process of e-transformation, including the adoption of many e-services, the implementation of an open data portal with over 700 data sets, and joining the Open Government Partnership (OGP) in 2012. However, Moldova has a long way ahead in terms turning data into knowledge by stimulating the demand and feedback from civil society, mass media and businesses.

Last week I was invited to run a small workshop with both government and civil society representatives and to discuss how they can build on existing work and make Moldova a true leader of open data in the region and beyond.

One of the first things we did was to map challenges and opportunities of open data in Moldova through a participatory exercise. While participants recognized and truly believed in the opportunities of open data, they also raised numerous challenges on the entire spectrum from supply to demand.

##Opportunities:

• More informed and active citizenry and mass media
• Better policies through public participation and oversight
• Improved internal mechanisms within the government
• Improved government image and more trust in public institution
• Supports the fight against corruption
• Stimulate business and innovation

##Challenges:

At the level of technical infrastructure

• High cost of digitization
• Most data is not available in electronic format
• Data is outdated or irrelevant
• Lack of proper IT infrastructure in some institution
• Varied capacity and efficiency to process data among institutions

At the institutional, administrative level

• Weak or vague regulations on what data to open and how to do it
• Lack of political will in some ministries
• Fear of exposing corruption or wrong doing
• Lack of human resources and capacity to elaborate and open data
• Lack of proper collaboration mechanism among institutions
• Lack of mechanisms to award those who perform well and sanction those who lag behind

At the level of demand

• Lack of well articulated demand from civil society
• Lack of culture of opening and consuming data
• Often data is mis-interpreted
• Limited access to technologies
• Low capacity to make use of open data (analysis, providing feedback, etc.)

What I learned

Undoubtedly, there is a window of opportunity now in Moldova – there are numerous leaders from both government and civil society who have extensive experience and passion for open government and are eager to learn more and collaborate.

• Civil society organisations, journalists and businesses need to keep using the data available and push the government to release more data that is truly open: data which can be freely used, reused, and redistributed, by anyone, anywhere, for any purpose (see http://opendefinition.org/)
• The government needs to keep working to improve internal processes and workflows and invest in training and tools to ensure ease of collection and release of open data in open and useful formats.
• There is a strong need of conveners who can bring together and stimulate collaboration between government institutions, CSOs, journalists, tech enthusiasts, businesses, etc. In particular, collaboration should be stimulated on specific sectors such as education, environment, etc.
• There is a thirst for international examples and best practices: from successes and examples of innovative uses of open data to stimulate the demand through to examples of internal processes, policies and workflows used by other governments.

Our data diva Michael recently traveled to Kenya to explore development aid and more. Here’s his report back.

The Kiswahili word for government – serikali, is (so is said) derived from siri-kali: the words for secret and danger. Thus, government is perceived to be a dangerous secret – the Kenyan government for sure lived up to this reputation in the past: uncovering it’s secrets could have drastic consequences. Recently, the government has worked on increasing transparency and started an Open Data initiative. A Freedom of Information act is slowly on it’s way. While the personally most engaging where the days I tried to get a flight out of Nairobi (I was originally scheduled the day of the airport fire), my presence in Kenya before brought me further than negotiating with airline staff. The latter resulted in being flagged as “angry passenger” in my PNR record, the first a set of new friends and re-strenghened relationships.

The reason for my visit to Kenya was a workshop with Development Initiatives – who trained a group of 30 activists from CSOs on Aid and Budget Transparency. The workshop had a strong focus on campaign and advocacy strategies but also involved some data work. I was there to guide the participants through a data scouting excercise (imagine the first bits of a data expedition: brainstorming ideas and working out what data are needed to answer them). Since the workshop had a budget focus, the participants were further introduced to visualizing data with OpenSpending – feeding back into the documentation around the project. In the scouting excercise participants were free to brainstorm questions around the topic or even leave the topic completely: one of the groups decided to focus on diamond mining in Sierra Leone. Participants brought up crucial questions around where money is spent and how – diving into corruption, need and where projects are conducted. We took a quick look about the water supply to Kenyas population, where only a fraction of people do have access to a clean-water tap at home, corruption, especially related to service delivery and tried to find data and information on government run sanitation projects. The groups investigating mining in Sierra Leone took a deep dive into the EITI process to see how revenues are distributed – only to find out the country is currently listed as suspended from the process (due to failure to comply to the minimum requirements).

After intense and cold days at the workshop in Limuru (said to be one of the coldest places in Kenya), I spent some days in Nairobi meeting new and old acquaintances. Back in Nairobi it was time for a small data expedition with the Data Dredger team of Internews Kenya. We started from data they had collected on Malaria in Kenya. Three teams quickly formed and explored questions like: is there a relationship between income and malaria, is money better spent on prevention, rather than curing people and having them sick and is malaria better diagnosed and treated in some areas of Kenya than in others. The results were quite interesting: the team investigating income related malaria incidence discovered that younger children are more often sleeping under mosquito nets than older ones – attributed partly to the habit of passing nets down to younger siblings and the particular vulnerability of little children. They also discovered an interesting outlier in mosquito net use of women: women in the 4th income quintile (a quite wealthy quintile) were clearly using nets less often than expected by the otherwise quite linear relationship between net usage and income category. However, in pregnant women, they used nets more often than would have been expected. A fact, that left the team curious. A possible lead is the number of children per woman in higher income families: there is an inverse correlation between wealth and children per family. Thus it could simply signify that children were perceived more important and thus pregnant women received more care.

The day continued with a visit to the iHub – a technology accellerator in Nairobi, started by people around Ushahidi and now a major institution in Nairobi. A year ago they started a research venture and started researching the impact of technology in various areas in Africa. They also are evaluating the Code for Kenya project, organized by our friends at the Open Institute. I met them next in the Nairobi Hacks/Hackers meetup. We talked a lot about how to scrape information from the Kenya Gazette – the official government bulletin. We experimented with Natural Entity Recognition using the ActivityAPI – a small rudimentary scraper was the result. In my planning for meetings, I had further arranged a meeting with Creative Commons in Kenya the next day – only to discover I met the publisher of the online version of the Kenya Gazette – the Kenya Law Report. Before the report was published online it was very hard for both lawyers and citizens to learn about high-court rulings, changes to the law etc. The Law Report started collecting all this information and putting it online. While there is an ongoing initiative to improve the website with a better searchable cross-referenced database – the website as it is already helped to greatly increase the transparency of legal procedings. A quick meeting with the Kenya ICT board – the project leaders for the Government Open Data program concluded the intense series of meetings in Kenya.

Last night, Michael Bauer carried out a School of Data workshop in Santiago, Chile, kindly organised by the HacksHackers Santiago network. The workshop, based around Money, Politics and Education, attracted over 80 people to the School of Journalism at the University of Diego Portales in Santiago, and was a great success!

Credits: James Florio. Poderomedia

First, they looked at Ministers of Education in Chile, getting the data from a table in a Wikipedia page into a Google spreadsheet. Then (getting a little more complicated) – at the amount of money earned by people working in the Chilean President’s office, through the government portal.

And finally, they learned how to get data out of multiple tables from the same website all at once, again from the Government Portal, using Google Refine. By the end, the data was all prepared and ready to upload on to the OpenData Latino America portal, at the upcoming “Scrapaton”, to be held on June 29th in Santiago. You can see the instructions they followed online here.

Big thanks go to ICFJ Knight Fellow Miguel Paz, organiser of the Hacks Hackers Santiago network, and everyone else who organised such a great group of people to come together, including live streaming of the event. You can see photos of the event online too.

For those who attended and want to know more about the School of Data, sign up to the School of Data Latin America announcement list. And if you’re interested in finding out more about the Open Knowledge Foundation’s international activites and/or setting up a Local Group in Chile, drop Zara an email (zara.rahman[at]okfn.org)  or leave your details on this Google form.

Do you live in Latin America? Hungry for some School of Data materials in Spanish? We have some good news for you: The School of Data is going to come to you!

Image CC-BY-SA Eric Fisher

While our friends at Social-TIC are working hard on translating School of Data materials in Spanish, Michael Bauer and Zara Rahman are going to visit La Paz (Bolivia), Santiago (Chile), Buenos Aires (Argentina) and Montevideo (Uruguay).

Michael kicks his tour off with the first DataBootcamp in Latin America in Bolivia, he’s then joined by Zara in Santiago, where there will be a Workshop on Scraping on Monday June 17th. They will also shortly present at the Data Tuesday the next day. They will continue their trip to Argentina with a Workshop on June 20th and finish their tour at AbreLatam – the open Latin America unconference.

During the time they will be available to meet, scheme and plot. If you’re interested meeting them: Contact us at schoolofdata [at] okfn.org.

This week Michael Bauer has been traveling to give workshops in Prague, Barcelona and Accra. Training Data Skills with Civil Society Organizations and Journalists. He reports:

With the coming of spring in Europe not only nature awakes. Suddenly the world is buzzing with workshops and interesting events. This means a lot of traveling and meeting new, interesting people. This week I spent time with people from Civil Society Organizations in Prague and met data journalists in Barcelona.

The Otakar Motejl Fund together with the School of Data organized a two day hands-on Workshop, similar to our previous Workshop in Berlin. In two days Anna Kuliberda of Techsoup and me worked with the participants on integrating evidence into their campaigns and on optimizing their workflows with data. Scouting Data and Projects of the participants on the first day showed intense interest in how to find good issues for data driven campaigns and how to clean and merge datasets without headache. This, together with an insights into community building delivered by Anna, were the focus of our second day.

After an intense two days I set out for Barcelona for the first Journadas de Periodismo de Datos organized by the Spanish chapter of the Open Knowledge Foundation. While the first day was reserved for overview and introduction talks, the second and third were reserved for workshops and a hackathon. Attendees of the workshops were taught how to scrape pdfs and the web, to use open refine and basic analysis using spreadsheets. Interest in Data Driven Journalism is high in Barcelona and this will probably start an annual event in Spain.

Leaving Barcelona the Journey took me (back) to Accra, Ghana. The second Data Bootcamp, exploring the impact of extractive industries, was happening. Data Bootcamps are an Initiative of the Worldbank Institute and the African Media Initiative to promote digital literacy and build communities around open data programs. The three day events are filled with hands-on introduction to tools. The participants pitch projects, they want to work on during the day and at the end of the event winning projects are chosen. Since this was the second Data Bootcamp in Ghana, the previous winner Where My Money Dey? – a project examining the use of Oil generated Revenues – was demoed. After three exhausting Days 8 Teams pitched their Ideas and it was hard to choose a winner. After all two projects peaked out: “Follow the Issue” – a project aiming to turn news items into actionable issues and allowing communities of engaged people to form and organize around them – and “Ghost Busters” – having a systematic look into payrolls and accounting information (available seperately) to identify people being paid without formal employment: a frequent issue in African countries. These projects received initial seed funding and will receive further support to build their prototypes.

The Ghost Busters team explaining their strategy to identify ghost records on payroll

Interested in connecting with the international Data Journalism Community? Join the Data Journalism Mailing List. To stay in touch about