Reinforcement Learning and why you should care.

Amidst all the complexities of life, have you ever stopped to appreciate the simple things? I hope you have, its a refreshing perspective. One thing we definitely under appreciate is learning. How exactly do we learn to read, write, walk, speak. Unlike other animals, our brains are not completely wired up at birth. We are not given any manuals, let alone the ability to understand manuals at birth, yet in less than a decade we fully master these tasks. There is only one valid explanation, reinforcement learning.







Reinforcement learning(RL) is a way of learning without a teacher, in other words learning by doing. Its a proactive way of learning where you perform actions first and subsequent rewards shape future actions. So, when you are learning to walk, if you move the wrong way, you fall (negative feedback) making you less likely to make the same move again. On the other hand, when you make a winning move, reward centers of the brain are activated with Dopamine (positive feedback), making you more likely to repeat the same move. Repeating these actions and making mistakes over time is what makes you better.


In an effort to make computers intelligent, reinforcement learning has also found its way into Machine Learning. It is especially useful where there is no data to train models. If only we could explain how we walk or solve problems, we could easily train robots to do the same, but we don't, so Reinforcement Learning is used instead. It has been successful so far, it has managed to train robots on complex manipulation tasks, train computers to beat humans at Go and play ATARI games among many other tasks. It surely is an important Machine Learning technique.

Although it is high performing, reinforcement learning in computers isn't as simple as it happens in the brain. A lot of conscious effort has to be made to determine the appropriate states, the actions and rewards for those actions. How many dimensions (factors) are required to represent a state? too few and it doesn't capture the problem, too many it becomes intractable. Current research is working to solving this problem.

Albeit its complexity, its interesting and somehow alarming to know that computers can be trained in the same way humans are to be even better than humans. So, perhaps they will one day have a conscious, pass the Turing test and change the course of civilization. Either way, we should expect much more Intelligence in the near future.

In case you are wondering, I'm currently reading The Brain by David Eagleman. It's explains fascinating facts about the brain. I highly recommend it. Have a great week ahead!

What you probably missed out on ..#PyConZA

Here is a very delayed post on PyconZA which took place on 6th to 7th October. I wish I wrote this sooner but between dealing with somewhat-of-a-personal-crisis, UCT protests and my final year project, blogging got the back seat for a while. That being  said, here we go. I got the opportunity to attend PyConZA, thanks to the conference organizers I got a sponsored ticket. PyconZA conference for python enthusiasts in South Africa to learn from each other. It has been taking place in Johannesburg , luckily it happened in CapeTown this year. The venue was the River Club, which as you can tell from the pictures was a great venue.

the river view

The talks were not only interesting but some were mind blowing. One of my previous lecturers gave a 5 minute introduction on radio astronomy. I learned about about using python for Micro controllers,  for story telling, for serving underprivileged communities , juggling tasks , Machine Learning among many other applications. I know Python is very versatile, but its a lot more fascinating when you actually see the applications. I also enjoyed the panel of teaching python for many reasons: first, the question of increasing accessibility to women was thoroughly discussed, second, one of my lecturers was also on a panelist and gave interesting points on Computer Science Education, something I have taken interest in lately or yeah and I also volunteered to monitor the mic. My CV is getting beefed with such volunteer experiences ..:).

A mobile retail app by the startup Timbuktu

Micropython for microcontrollers

Python for creative writing

 Thanks to amazing sponsors, we got lots of goodies especially Takealot vouchers.

Bags of Swag!!

Conferences are more about the people, it has never been truer. I got to connect with fellow East Africans, most of whom were giving talks. This got me all the more inspired to start start speaking at conferences. I also reconnected with colleagues at j5 International where I interned last summer, fellow students and alumni.

j5 founder

I had such a fantastic time at Pycon, I highly recommend it to anyone, even if Python is not your first language. I learnt a lot more about programming in general and open source. If you missed it, videos are now online. Till next time!

PS; You might also be interested in Pycon Namibia and Pycon Zimbabwe happening soon!

What you missed out on ...#ICFP2016

I got the opportunity to attend ICFP ( International  Conference on Functional Programming) that took place in Nara from 18th to 24th September 2016. I got the scholarship from ACM-Women and I also signed up to be a student volunteer, which meant I got free registration in exchange for helping out in some handy and administrative tasks. Registration costs $400 , so it is a great deal!
Fun fact: I was the only attendee from Africa.

Day One:
The first day was for workshops. There were four different workshops running in parallel, the Type-Driven Development workshop(TyDe), Scheme and Functional Programming workshop, Higher-Order Programming with Effects and Programming Languages Mentoring Workshop (PLMW). I attended TyDe to learn more about APL and PLMW. PLMW is geared to senior undergrad and graduate students and is perfect for beginners. I learnt not only about functional languages but also about research, how to write and present papers. At the end of the day we, student volunteers had Japanese style dinner.

Told you I was the only African ..;)

Student volunteers dinner

Day Two:
This was when the official conference began. We had a keynote address about Tensor flow by Martin Abadi from Google Research. I thoroughly enjoyed it due to my interest in Machine Learning. The rest of the talks were great, some of which were over my head. I mentioned this in a conversation with a proffesor and he assured me that I wasn't the only one. I enoyed the presentation on the Clojent, a new functional language for verification.  At the end of the day we had the welcome reception and poster show for the ACM SRC competition.

Keynote

Some form of rice and fish for lunch

conference venue

Conference venue

Day Three: (Conference Report, Banquet)

I attended most of the talks, not that I understood it all, but I got the terms to google. I particularly enjoyed the talk about Computational Geometry and Higher Order Ghost State.Being a student volunteer, I had a session assistant duty where my job was basically to hold the mike as people ( academic superstars like Jeremy Gibbons and John Hughes) asked questions. This will be definitely go on my CV. 

Later we had report from the conference chairs and icfp contest. I am motivated to participate in the contest as it is open to everyone, not necessarily students. The winning team has been the winner three times in a row! #Goals. It was here that the conference chairs announced the demographics of the attendees. In previous years, attendees mainly came from Europe, America and Asia-Pacific. This year there is one attendee from Africa, you can guess who that is. I met two other Africans , from Egypt and Rwanda respectively, but they were affiliated with Universities in America, so yeah, I represented the whole continent. 

The day was concluded by the banquet at Hotel Nikko where prizes were awarded to SIGPLAN (Special Interest Group on Programming Languages) members in various categories (service, most influential papers etc). 

but first..selfies

Banquet

Buddha deer welcomes you to Nara

Day Four: (Final day of ICFP )

This was the final day of the conference. I enjoyed the sessions on Linear Functional Programming and Context-Free Session Types. It was winded by SRC awards and ICFP 2017 presentation. A friend I met won an SRC award, I was very inspired. It was also announced the next ICFP will take place in Oxford, having been there before, I can't help but hope to attend (or even present)it too.

Day Five: Workshops and Industry reception

There were a couple of workshops on Haskell, ML family and functional high performance computing and commercial users of FP. I attended a little bit of everything as they were all very interesting to me. I learnt of WebAssembly the functional language that is anticipated to replace JavaScript on the web and programming APL on the GPU.

This was a dinner prepared by the companies that sponsored the conference, these include Ahrefs, Jane Street, Mozilla, Ambiata, Facebook Galois among others. I was glad to learn that many companies were using functional languages (particularly Haskell and Oocaml) for their machine learning. Oh and the food was great.

yummy!

Industry reception

Day Six: 

 There was the Haskell Symposium, Erlang workshop and Oocaml workshop. I attended a few sessions and did some sightseeing of Nara. I also got to talk with graduate students and faculty who gave me useful advice on research.

Day seven: Functional Art Modelling and Design
The last day had workshops on Commercial use of functional programming, Haskell Implementors and Functional Art. I went to the functional art workshop (it sounded as cool as it sounds now). I saw demos of how functional languages are used to make music, paintings and are applied on the Arduino. Thereafter I went to the performance evening where performers played live music by coding it in functional languages. I'm still fascinated.

Making music with functional languages

Being an undergrad , relatively new to programming languages in general and this being my first research conference I had a lot to learn. The people were very open and approachable. Nobody would laugh, if you admitted you don't know X,Y or Z, as a matter of fact they liked the opportunity to spread the word. Its probably because functional programming is unpopular and thus the community has one mission , to spread the word -egos aside.


Spending the week with academic superstars was definitely the highlight of the conference. The atmosphere was intellectually stimulating. Since I couldn't sleep, I worked on my thesis at night and I was surprisingly very productive. Being a final year student, it opened my eyes to the joys and reality of research. I'm still looking interesting topics and universities but post grad will definitely happen down the road.

I'm sad that there weren't many Africans. I now see why they say Africa is being left behind. I guess the reason could be that in Africa we have other problems where our research is focused on (like on ICT4D Information and Communication Technology for Development at UCT) that theoretical research (like Functional and Programming Languages) is not being done.

Otherwise it was a great conference. Thanks to ACM-W, I now see myself as a researcher.
PS: I also did some touring, read here for more details. Also if you are a women in tech, apply for the ACM-W scholarship, its worth it!