On February 21 (International Open Data Day), Bantay.ph, a platform that uses technology in mobilizing citizens to demand good governance, will host the very first citizen-initiated “Data Jam.”

The workshop was mainly led by Ali Rebaie, a Senior School of Data fellow, and Bahia Halawi, a data scientist at Data Aurora, along with the data community team assistants; Zayna Ayyad, Noor Latif and Hsein Kassab. The aim of the workshop was to give the students an introduction to the world of open data and data journalism, in particular, through tutorials on open source tools and methods used in this field. Moreover, we wanted to put students on track regarding the use of data.

On the first day, the students were introduced to data journalism, from a theoretical approach, in particular, the data pipeline which outlined the different phases in any data visualization project: find, get, verify, clean, analyze and present. After that, students were being technically involved in scraping and cleaning data using tools such as open refine and Tabula.

Day two was all about mapping, from mapping best practices to mapping formats and shapes. Students were first exposed to different types of maps and design styles that served the purpose of each map. Moreover, best mappings techniques and visualizations were emphasized to explain their relative serving purpose. Eventually, participants became able to differentiate between the dot maps and the choropleth maps as well as many others. Then they used twitter data that contained geolocations to contrast varying tweeting zones by placing these tweets at their origins on cartodb. Similarly, they created other maps using QGIS and Tilemill. The mapping exercises were really fun and students were very happy to create their own maps without a single line of code.

On the third day, Bahia gave a lecture on network analysis, some important mathematical notions needed for working with graphs as well as possible uses and case studies related to this field. Meanwhile, Ali was unveiling different open data portals to provide the students with more resources and data sets. After these topics were emphasized, a technical demonstration on the use of network analysis tool to analyze two topics was performed. Students were analyzing climate change and later, the AUB media group on Facebook was also analyzed and we had its graph drawn. It was very cool to find out that one of the top influencers in that network was among the students taking the training. Students were also taught to do the same analysis for their own friends’ lists. Facebook data was being collected and the visualizations were being drawn in a network visualization tool.

After completing the interactive types of visualizations, the fourth day was about static ones, mainly, infographics. Each student had the chance to extract the information needed for an interesting topic to transform it into a visual piece.  Bahia was working around with students, teaching them how to refine the data so that it becomes simple and short, thus usable for building the infographic design. Later, Yousif, a senior creative designer at Data Aurora, trained the students on the use of Photoshop and illustrator, two of the tools commonly used by infographic designers. At the end of the session, each student submitted a well done infographic of which some are posted below.

After the workshop Zayna had small talks with the students to get their feedback and here she quoted some of their opinions:

“It should be a full course, the performance and content was good but at some point, some data journalism tools need to be more mature and user-friendly to reduce the time needed to create a story,” said Jad Melki, Director of media studies program at AUB, “it was great overall.”

“It’s really good but the technical parts need a lot of time. We learned about new apps. Mapping, definitely I will try to learn more about it,” said Carla Sertin, a media student.

“It was great we got introduced to new stuff. Mapping, I loved it and found it very useful for me,” said Ellen Francis, civil engineering student. “The workshop was a motivation for me to work more on this,” she added, “it would work as a one semester long course.”

Azza El Masri, a media student, is interested in doing MA in data journalism. “I like it I expected it to be a bit harder, I would prefer more advanced stuff in scraping,” she added.

Exploding book in the pulpit of the Algemene Doopsgezinde Sociëteit in Spui, Amsterdam where some of the sprint took place

Last week I attended the 7th annual Winter School at Amsterdam University. Run by the Digital Methods Initiative, it took the form of a data sprint in which students joined professional developers and designers to answer research questions using social media data.

The DMI group at Amsterdam have developed and collated a suite of easy-to-use tools specifically for this kind of research. They are well worth checking out for anyone interested in this field and they cover a range of techniques from web scraping to list triangulation, and can be found online here.

I joined a group looking at bias across three APIs through which you can acquire Twitter data: the Search API, the Stream API and the proprietary Firehose endpoint – generally regarded as the most complete source of Twitter data. We had three sets captured from the three separate APIs for a critical period between 7th and 15th October 2014 when the Hong Kong protests were taking place.

Other groups took on a range of tasks from mapping the open data revolution to tracking the global climate change debate. All projects deployed a range of data wrangling techniques to answer these complex social, political and cultural phenomena.

A few things I learned:

• Anyone wanting to use social media data to answer research questions about society and culture needs more than just spreadsheet skills. These datasets are generally larger than what Excel can comfortably handle, so basic database skills are a massive help.
• Off-the-shelf tools for data analysis are brilliant, but often one needs to tweak lines of enquiry to your specific research question. Having some knowledge of programming means that you can take a much more flexible approach then when relying on the GUI tools.
• Working in such a collaborative fast-paced environment meant that reproducibility (ie. where different parts of the team would re-use scripts and code developed by other parts of the team) was essential, alongside creating documentation on the fly. We found iPython notebooks especially useful for this, whereas analytical steps taken in Excel were harder to reproduce.
• Free Twitter data – like that which can be acquired from the Search and Stream API – is still good, and sometimes better than that which you get through the proprietary APIs. When investigating online reactions to contentious and controversial events – such as the Hong Kong protests – tweets will inevitably be removed both by users and Twitter. If you want to get the full story, it’s far better to scape data as it comes in through the streaming API.
• We’ve written about it before on this blog but the Pandas module for Python is brilliant for data wrangling and analysis and well worth getting to know if you plan on working with big datasets. It’s quick, flexible and powerful.
• Nothing beats hands-on learning when it comes to technical skills. Having a motivating research question and some real life data is the best way to learn how to use the multitude of tools now at any budding data wranglers disposal. I learnt more in a week than I could have in months reading about tools and languages in the abstract!

For those interested in attending a DMI school in the future – take a look at the summer school coming later in 2015.

I had the pleasure of running a couple of School of Data related sessions, too – one short skillshare running through the ‘data pipeline’, and a longer session building out a ‘follow the money’ focused data pipeline, focused mainly on gathering various data sources on topics in this field. The pipeline, in its rough format, is online here, and I’ll publish it in a more accessible format on the School of Data site soon too.

The value of asking questions

These sessions made me think about how data literacy skills could be developed within this community, and what is really needed to support and further the work of Follow the Money initiatives. Pragmatically speaking, for technology and data to be engaged and used successfully to further people’s work, not everyone in that room needs to be a superstar data wrangler or developer. What they do need, though, is to know where the people with technical expertise are, and to be able to ask them for assistance.

In the ‘thanks’ at the end of the workshop, lots of us mentioned that being in a space where, as our facilitator Allen Gunn said, ‘asking a question is considered to be a heroic act of leadership’ rather than a signal of a lack of knowledge. It was obvious that we valued most the patience and understanding of those around us who have higher levels of knowledge in a certain field, be that topical expertise, or technical; and that for many, the opportunity to ask these technical questions comes far too rarely.

This made me think about the value of the School of Data community – in my follow up emails from the workshop, I’ve been connecting people from various countries and contexts to former fellows who are based near them, or people running local groups in neighbouring countries, who can help them in person as well as online with their data-related queries. From past experience of seeing how well our data trainers and community members work with civil society groups with lower levels of data literacy, I’m optimistic that this will work out well – whether it be simply exchanging a few emails, or working with the community members or us at School of Data central to commission actual in person trainings.

Data wrangling + topical expertise = effective data-driven campaigning

As I mentioned, these connections provide a somewhat pragmatic solution to a need for better use of data among the community. Ideally, however, we would have people based within these organisations for long term support, who have both topical expertise and data wrangling skills.

And from what I heard, the need for this skillset will become extremely pronounced in the coming years; various directives and new laws regarding data availability and transparency sitting at different points of the money trail will be coming into force over the next couple of years, and they will bring with them a deluge of data. For example, data on extractives following Section 1504 of the Dodd Frank Reform, and company data following the EU Accounting and Transparency directives. What stories lie within that data, and how can we uncover them?

Many of the people and organisations represented at the Follow the Money workshop have been instrumental in campaigning for those transparency directives; but how many of those organisations possess in-house ability to actually process and use that data? Effectively, the next round of campaigning should be based on stories that come out of that hard-fought for data – but for that to happen, we need to start preparing now, by building data and technical skills among our communities.

Laying the groundwork for data storytelling

So, how can we start doing this? It could be through providing support for current employees of organisations to attend data expeditions or data skills courses on an ongoing basis; not just one off workshops, but people learning skills that are clearly relevant to their work, and having regular refresher courses to keep it relevant and in their minds. Or, (apologies for the blatant self-promotion here!) – it could be through supporting topical School of Data fellows to be based within the community and provide ongoing support, focusing on a specific topic – like extractives, or corporate money flows, for example.

Our experiences from the 2014 fellowships have led us to believe that the fellowship scheme is a sustainable and successful method of building up capacity both in terms of finding and supporting data storytellers and trainers (the Fellows), and equipping them with the skills they need to provide ongoing support to organisations based in their area, with whom they share their skills. Last year, the fellows carried out activities ranging from regular workshops with local organisations, to data clinics and expeditions for newcomers to get hands on with data, to simply being present within organisations as in-house support.

From what I saw last week, a lot of organisations within the Follow the Money network could do with this support. The earlier we start developing this capacity, the better equipped we will be as a community to start delving into the avalanche of data that is soon to come our way.

If you want to find out more about the Fellowship scheme, see the section ‘Fellowship Programme’ on our 2014 Annual Report, and if you’d like to talk about supporting a fellow through our upcoming 2015 scheme, get in touch with me on zara.rahman [at] okfn.org

The first trainings took place on September 9 and 11. Before the events, we sent out a questionnaire to get an idea of what skills participants already have and what they are interested in learning. It turned out from the responses that tools for data analysis and visualization would be the most useful for their daily work, so we decided to cover formatting tricks, functions, pivot tables and visualization with charts. To make the work more effective, we separated a beginner and an advanced group, and customized the material to the different skill levels, covering basic formatting and functions in the former and advanced formatting, pivot tables and charts in the latter group. We have worked with real world data on school performance and on migration, published by the Office of Education and the Hungarian Central Statistics Office.

A slight piquancy is added to the events by the fact that we were holding the workshops during the hours when the Hungarian police saw fit to raid the offices of NGOs distributing Norway Grants.

On the theme of open data, there was a very interesting discussion on how the open data movement can help in strengthening digital democracy in Indonesia. One of the examples shown is the story of KawalPemilu , a platform for voter count verification which uses crowdsourcing; it was created by just five poeple, and yet played a pivotal role in Indonesia’s 2014 presidential election as a checking mechanism for the voter count. The platform allows citizens of Indonesia to see and verify the voter count of the election and to check if anything is amiss. It stood as a strong example of citizen participation in the Indonesian elections.

Regarding the state of open data more generally in Indonesia, there was an overall acknowledgement that the movement is still in early stages. There is still a lot of need to raise awareness regarding the importance and usage of it, as well as skills around how to actually work with the data. Also notable in the discussion is the importance of collaboration between the various open data actors from both the supply and demand side such as government, CSO, and citizens.

Meetups like these provide a great platform for these actors in the open data space to actually connect and collaborate with each other – having these more frequently would strengthen the movement as a whole.

Just like we have seen in Nigeria, NGOs and activist seem not familiar with data pipelines or what we refer to as the data management processes, likewise basic tools that can be used in analyzing data. Unlike Nigeria, the transparency and accountability [T&A] movement in Ghana is coordinated under the STARGHANA project. Thus creating an ecosystem of groups working in the T&A component of the Open Data movement. “Two years ago, I was part of a team that initiated the SMS reporting on service delivery in the health sector, however, I am not sure how much the system is working anymore” explained Joseph Senyo, National Director of Programmes, Community and Family Aid Foundation

Participants at the Open Data event in Accra Ghana

While going through finding data, it was interesting to know that Nigeria has more datasets available online than Ghana, as most of the participants couldn’t figure out where to find the budget data of the country, although some mentioned the ministry of finance, but surprisingly we couldn’t get budget data from this website. Nevertheless, the country national statistics online portal is a one – stop shop for datasets in the country, and only one of the participants knew this existed. Analyzing using Microsoft Excel, and Google Spreadsheets was an eye – opener for participants, as most of them requested to know how this can be applicable in their various works.

While it was important to drive this conversation forward, outside the training sessions, the participants were already thinking about a 3 –day event that could bring together government, NGOs and other activist in the coming year. But, our trip to Accra would not have been complete without taking some time at the iSpace (it was a women in technology day, and we had ladies) and the LaBadi Beach – it is known that trainings can also be complemented with ice breakers on the beach – and same we did, and fortunately for us – it was the reggae night.

Getting instant feedbacks from participants

“We would have like to have more days of training, as the little minutes I spent was quite educative, especially the use of analysis tools, thus making me to know how important data is to my various monitoring and evaluation work” said Mensah Ileom of Inspire Africa. Actually, I have seen more NGO participants looking towards how data gathering can also help them in monitoring and evaluation, aside using it for advocacy, and monitoring service delivery.

The purpose of this event is twofold. The first is to introduce the Code for Bandung movement to the people of Bandung and the second is to give an insight how the people of Bandung can use data especially open data for civic technology. The second purpose is very important because right now awareness regarding the usage of data for civic tech is still low, even in the technology community. There are currently many effort in the field of data based decision for business but for things like civic tech based on data is still pretty much undeveloped.

In the event, we talked about things such as: what is data and open data; why those are important; examples of civic tech apps using data that is publicly available; etc. One example of such project was the water height in the city water canal which gives early warning information for flood alert. Besides civic tech usage, we also discussed about making a start-up with publicly available data, and continued with listing and searching what kind of data actually can be provided by the government of Bandung and can be used to create a useful solution.

The event covered some interesting topics. It is a nice and refreshing perspective to see how data can used for social and civic goals from the perspective of the technology people. It brings to mind that sometimes the most creative uses of data can be achieved by combining the perspective of many different people from different backgrounds, such as from the government, technology, and advocacy.

What participants said about the 2 – day event

In a 2 – day event on Friday, November 28 and Saturday 29, 2014 at the African University of Science and Technology, that was meant to raise the awareness on how NGOs can use available data to monitor service delivery in the health sector; empower journalist on using data for creating compelling stories that can cause change; and in all create a platform (on-the training) that can be used to monitor service delivery in the health sector. “We will be most interested in how citizens turned professionals like you all here, can take up stories from the data that will be curated during this event, in asking government questions about inputs in the health sector, and other sectors as well” said Christine K, Country Director of Heinrich Boell Stiftung Nigeria, during her keynote at the event.

In the minds of many participants was how we fit into this new world of Open Data with a party at the end. Did you ever wonder why the party? Well to clear the air, we started the “party” helping participants to know what data will mean to us, they as participants, and what it can change in the life of that curious woman that walks 30km from Keta to Goronyo to join an antenatal care program; what it meant for that hardworking man to transit from Potiskum to Kaduna before he can get a Hepatitis C viral load test, even though he had to borrow the 23, 000 Naira meant for this test. Yes, available and structured data can create a great story out of this recurring event.” If you are still looking for what could then happen from the gathering of these 102 participants – it’s all written in gold here, even though these are still stories in the making, but we can do much more” exclaimed Anas Sani Anka of the Nigeria Television Authority in Gusau, Zamfara

Adam Talsma of Reboot sharing skills that can make data matter to people on ground

Going through the data pipeline (data sourcing, collection, collation, analysis, reporting and use) surprisingly, we got this shock again! Only 2% of the participants knew where to quickly find the available data of the federal government budget in Nigeria. Whilst data pipelines was meant to guide participants through the data management processes (in a participatory manner) it was another opportunity to share where the available data are online in the country, and how they can be used in advocacy and storytelling to start conversation around transparency and accountability; and also in exchanging feedbacks between the people and government.

Leading the skill share session was Adam Talsma of Reboot taking participants through using formhub and textit and Michael Egbe of eHealth Africa introducing participants to how they are mapping Nigeria using Open Street Maps. The storytelling sessions had Tina Armstrong, an award winning data journalist that is interested in telling stories of vulnerable communities using data; Joshua Olufemi shared skills and tools that has made Premium Times the best online investigative media in the country; while the session was rounded up by Ledum of Right to Know, showing participants how to enact the Freedom of Information Act in getting data from the government.

Joshua Olufemi of Premium Times Nigeria sharing skills on telling stories with data

The high point of the first day was the, I want to learn, and I want to teach session – a remix of the School of Data Summer Camp World Cafe and Skill Share Session. “Learning particular skills in 10 minutes can be mind blowing and something I will not want to forget in a long time, I only hope we could have had more time other than the 30 minutes for the 10 min/skill session” narrated Michael Saanu of Africa Hope Foundation. Amongst skills that were taught is using Microsoft Excel for analysis, creating Survey form using Google Form, collaboration techniques with the Google Drive, writing funding proposals, community building, using Twitter and Facebook for advocacy, data scraping using Tabula amongst others. After this session, it was clear that participants wanted to be part of all the sessions, but they were only limited to three, as the night crept in faster than we expected – what an energetic way to end the first day!

Participants using sticky notes to chose what to learn and what to teach

Kick starting day 2, with the sun and expectations so high was lessons from participants, and an ice breaker on the power of around leadership. This day was dedicated to Open Street Maps and Data Sprint on Funds meant for inputs in the health sector. Moving from scraping the data from the budget office to visualizing it, and creating a monitoring instrument amongst the participants. Working through the available health facility data for Goronyo on NMIS data, we found out that some Goronyo data were not accurate –  So if we can’t use that, how do we get the government health facility data – most participants of this group concluded that the dhis2 data could be more reliable but its usage still remains difficult! Anyone wants to help in getting Goronyo health facility geo-referenced data? Please comment here. Not giving up, Sidney Bamidele of eHealth Africa trained participants on how to add, and edit points on open street maps and how to create task managers on HOTOSM.

Sidney Bamidele of eHealth Africa training participants on using Open Street Maps

Nevertheless, the data sprint with music, and drinks took the whole day, and I couldn’t stop hearing – OMG! So 20 million was budgeted for the construction of this health facility in my LGA, how come it is still at this state, I think we need to go and ask”; “I have found that so many time, descriptions of budget data has been duplicated – and how do we stop this”. As it has always been, only one sprinter had an apple laptop out of the 50 laptops on the tables; Most of the participants agreed that only 30% of Nigerians own a smart phone, so how many will used it, and how many will use an android or that new android app you are about to make? Maybe the feature of mobile activism in the country still lies in feature phones. These and many are conversations that always ensue during training and data sprint sessions I have facilitated. At the end what did we make – an Ushahidi Crowdmap instance of where funds for health input will go? a first step in starting a conversation around monitoring service delivery in that sector.

Participants during the Mapping and Data Sprint

What next? in the words of the Hamzat Lawal, the Chief Executive of Connected Development [CODE], it is important that we brace up, and start using the data on this platform in asking questions directed not only to the government on if budget data description got to citizens it was meant for, but also to citizens it was meant for – on facility and health input usage and quality. As a School of Data Fellow, I have learnt that citizens need basic tools and skills to hold government accountable. As a monitoring and evaluation expert, I can see that in few years, lots of data will be released (even though most wouldn’t be responsible), but how citizens will identify and use the reliable ones remain a herculean task. As a human being, I learned how hardworking and brave my colleagues and participants are. At no time did I feel that facilitating data trainings was futile. Ultimately, what I really learned about data, or open data, or available data is that the NGOs, journalist, activist and governments still need more capacity building around this phenomenon.

Pictures from This event are on Flickr

The skills shared are mainly the detailed and technical version of the data pipeline concept of the School of Data:

• Finding & getting data: Google advanced search, use of data portals, using Tabula to get data from PDFs
• Cleaning data: data cleaning principles, using Open Refine to clean up messy data
• Analyzing data: Excel, Tableau
• Visualizing data: visualization principles, Infogr.am, Piktochart, etc
• And a lot of other data related stuff: spatial data, examples of advocacy using data

The event is more focused on discussions around each topic. The School of Data Fellow first gives a short presentation about the topic and continues with technical hands sessions or with a discussion based on a case study, depending on the topic. With these kinds of knowledge sharing, hopefully that CSOs such as PWYP Indonesia will have a more hands on experience on working with data.

Right now the knowledge sharing has been done about 10 times, with some encouraging results. PWYP Indonesia began to use data more thoroughly as the underlying base for some of their advocacy program and have created some infographics in order to better communicate their data better. Hopefully, knowledge sharing like this can continue and better support CSOs in using data for their advocacy uses.