Positively Precise Predictions

I have been able to run a few more experiments on my local machine rather than SSH-ing into a very remote (located in the Far East) server.

I installed GNU Octave, which is essentially a free knock-off version of MATLAB. It incorporates much of MATLAB's non-cryptic syntax, making it possible to run complete scripts from .m files.

Will I be able to predict the future?

I previously discussed how I was able to perform interpolation with my data. I would take every n-th point from the initial data set and use this set as the training set. I would then proceed by taking every m-th point (where m < n) and place these into another set, such that the resulting set is larger than the training set and includes a few points 'in between' training examples. This resulting set will be the test set.

I displayed that interpolation could be done reasonable accurately. The next obvious area to investigate is to look at how GP regression performs for extrapolation. I will fix the amount of data points I would like to predict and vary the size of the training set. This makes it possible to observe the behaviour as the program encounters more data points to train on, and whether this increases the accuracy of predictions.

I made sure to use a method of choosing which points to use in the training or test set which makes it easier to check the accuracy of predictions. I simply look at the last P points, starting from the last observed data point that I have, and have that as the test set. This set will be fixed. Now, I look at the Q points that are before the P unseen testing points and use this as the training set. So, the larger the Q, the further back in history we are looking at and thus the more comprehensive the training set will be for the program. This also means that we are further away from the testing points, so predictions may actually become less accurate as we continue increasing Q.

After some trial and error, I chose the length scale and signal variance parameters to be 14 and 10 respectively. It would be interesting to see if there is a way in which these hyperparameters can be learnt via GP, as trial and error is far from a scientific approach.

Below are the results. As always, in the diagrams the red curve shows the predictions given the values of the 18 parameters corresponding to each observed PM2.5 value, which are shown as the blue crosses. The aim is to see how close we can get the red curve to 'fit' the blue observations to see how accurate we are. The size of the training set varies from 10 to 1500.

We can see that within the 1000-1500 range for the training set size, we have learnt a reasonably accurate curve to fit the data, which is what we wanted.

What else needs to be looked into?

As mentioned, I need to find a way to learnt the signal variance and length scale parameters. In addition I am currently learning the form of the mean function hyperparameter as I assume a priori that it is unknown. It may be a better idea to use the mean value of PM2.5, a positive real number, to act as a constant mean function, and see how this affects predictions. I currently limit the number of training examples to 1500 in this case, as after this point, Octave begins to struggle with the computations. It would be helpful to increase this number further to see how this affects the accuracy of predictions.

Leaving for London

After the weekend in Shanghai, it was time to return to London.

Pre-flight

Monday (24/08) was literally my last day at the Tsinghua university campus. It was an emotional day, but the way I spent it was a nice way to wrap up the incredible eight weeks I spent here. 

In the morning I went to the local supermarket to buy as many strange dried (for ease of transport) China-specific foods as possible, for example, green tea flavoured Oreos, green tea flavoured bread, green tea, etc. 

While having lunch, I stumbled upon someone from my floor in the FIT building. We had a great conversation about my time in China as per usual. He had lived in China for almost eight years so he was completely fluent in Mandarin on a professional level. It made me slightly jealous how he was able to have loud, energetic conversations with locals across the street! We also discussed his research, which was apparently about analysing the genre of different Chinese operas. I was offered to make contributions to and proofread his paper, since it was returned to him after submission with a large number of changes to be made.

Returning back to my dorm, I finally had the chance to meet who would be living in the room next to mine. He was a French-born Chinese student, who was at Tsinghua for the first time. It was a shame that I was only able to meet him on the day that I was due to leave the country, but I told him everything useful that I learnt during my time here and we exchanged WeChats (how else do you describe the process of scanning someone else's WeChat QR code?). I also offered him some green tea.

I spent most of the afternoon packing, and after a few hours I decided I should bring my big camera out one last time to take images of the campus. The weather just happened to be great on this day, with very little pollution to worry about.

Like almost every other time I left my room, I was asked if I could take some pictures with some random girls from the university, "since I was foreign". 

In the evening I watched the freshmen who were undergoing their compulsory military training. Speaking to a student who I spotted cruising around on his skateboard, I was told that the training was only compulsory for students originally from mainland China, and not for foreign students. Although this is quite a controversial opinion, I honestly think that making it compulsory for foreigners may actually help to give them a taste of Chinese culture upon arrival to the university. Although one of the procedure's primary purposes is for discipline, the students end up developing strong relationships with others undergoing the training.

I stole the guy's skateboard for this picture.

Departure

After dinner, I realised that after all my interactions in the day, my flight was in four hours and that I had not checked-out or finished packing. Not good.

By some unknown force of nature, I somehow managed to pack everything I needed in my suitcase, breaking my Poundland-purchased scales in the process so I had no idea whether my luggage was within the limit. I checked out, received my deposit, returned my keys, and hopped into a taxi to the Beijing Capital Airport at Terminal 3, arriving with just under two hours before my flight.

While queueing up for the baggage check-in, I met a cool German-born Indian guy who was studying at the Beijing Institute of Technology for a Summer school course in the Mandarin language. We discussed our many difficult experiences during our stay in China, with him having an especially hard time since he was required to renew his X1 visa, but he could only do this in his home country, which was why he was getting this plane to fly back to Germany.

Flight

The journey back was difficult to say the least. I really underestimated the distance between China and the UK and the exhaustion that would result from making the 20+ hour ride. I managed to watch a few films (unfortunately not in first class this time!) and was met by my sister before catching the final bus home in London, which I am of course hugely grateful for. 

Reflection

It really was hard for me to leave China, after such a challenging but overall positive experience. I learnt a truly great deal and would happily return to China a second time. I feel that even after two months, I had not integrated as much as I wanted to, but I was nevertheless at a comfortable level with dealing with locals even with my very limited Mandarin.

I feel that for me to return, it must be for the purpose of work, as China is such a strongly competitive and powerful country. Arriving in the country for sightseeing but not contributing in any way with purpose seems like a complete waste of time. Being at the centre of a place like this reinforced the fact that anything that I do in the UK, someone somewhere in China has probably done it already and to a much higher level. Living in the Far East where the sun rises seven hours earlier may have something to do with this! I learnt that this was something even told to students in high school about those living in Western compared to Eastern China. 

I now know that in terms of looking at my own progression, it isn't enough to just look at myself and my surrounding country (UK). I need to aim to be competitive on an international rather than domestic level. The Chinese really do struggle hard for success and are simply on another level to the UK from what I saw. China is a superpower as it is rightfully described by many. The experience has also strongly motivated me to study Mandarin in the coming academic year, as I believe it would make the it far more easier to get things done if I were to return to China, and of course due to the overall importance of knowing the language.

The Showers of Shanghai

I bit the bullet and bought bullet train tickets to Shanghai for a weekend. I was there between 22/08 and 23/08, staying at the Shanghai Rock & Wood International Youth Hostel (which I thoroughly recommend to visitors of Shanghai!).

Preparation

Looking back, I do not think it was a wise decision to do with so little planning, lack of time, and money, but I do believe it was a great opportunity to be at the heart of the biggest city by population in the world, somewhere I may not be able to visit again in the future.

I hopped on the subway on Friday (21/08) to Beijing South Railway station just after I bought the tickets and spent about 30 minutes packing and leaving the lab on very short notice.

After getting past the ridiculous amount of security at the station and waiting in endless queues, I was told that the train for the ticket I bought did not exist and that I should go and wait in another queue to talk to someone at the ticket office to change my ticket.

After a long wait in the queue, I was told that it was not possible to exchange my ticket and that the (very expensive) one I bought was now useless, as after all this time waiting, it was now after the departure time of the train that was meant to leave.


I still had a return ticket, so I thought I may as well buy another ticket to Shanghai, since I was given the impression that there genuinely wasn't anyone in this station who cared the slightest about my circumstance and how much I pay for the ticket.

After wasting my entire evening not travelling to Shanghai, I bought another expensive ticket for 7am the next day (which of course was done within seconds without any problems).

I had a genuinely great conversation with the person I sat next to for the next 5 hours of the bullet train journey. We spoke about the many differences between the cultures of the West and the East, and my experience of the Far East for the past eight weeks. She was a devout Christian, and was the first local (apart from in church) to ask me about my faith and how I eventually became a Christian. She also shared her story. She went to a church which held many similarities to my own local church in London. Even though this was my last weekend in China, this unexpected conversation really opened my eyes, letting me know that whichever country in the world I'm in and whatever train ride I happen to be on, God's work can still be seen!

Arrival

I successfully entered Shanghai soil at around 1pm on Saturday and checked in at my hostel.

After buying a tourist map, I spoke to an incredibly cool Korean guy who let me know of all the things I could do within the short time I would spend in the city. Everything seemed to be on line 2 of the subway, which was helpful as the hostel was about a 5 minute walk from it.

I immediately unpacked, and left to go to the famous People's Square, located in a very central part of the city in a attempt to visit the Shanghai museum. It was closed.

Seeing the Oriental Pearl Tower in the distance, I walked towards it to get a look at the famous futuristic Shanghai skyline. It was a great view at the time (late afternoon), but looked a lot more magnificent at night when the city is completely lit up. It gave me a chance to practice night photography, and see how different shutter speeds affected the resulting image.

I walked around the area near Yuyuan Garden (which was also closed), and was able to get a feel of the older and more traditional Chinese-looking parts of the city. I was able to find a restaurant of Shanghai cuisine, only to be told by a local that it was "Shanghai cuisine for tourists and so isn't that delicious"! I didn't mind, I hadn't eaten properly for the whole day and needed dinner.

An odd sight, having the futuristic Shanghai tower in the background

The next day I left the hostel early to visit the Shanghai Science and Technology Museum. After getting there, standing in a gigantic queue in the pouring rain for about 30 minutes only to make a few small steps, I decided it wasn't worth my time.

I got the subway to Lujiazui, to get a close up view of the huge buildings which make up the incredible skyline. With the cloudy weather it was interesting to see how the towers literally rose up above the moving clouds. There was a walkway which enabled visitors to walk around at an elevated level and have a look at each of the towers up close. Although the resulting images looked cool, the rainy weather made it difficult to point my camera up to take photos! My left hand was used here as a makeshift lens hood.

Departure

I hopped back on the subway to East Nanjing Road, the main shopping street of Shanghai, in another attempt to try and get some real Shanghai food. In broken Chinese I asked a local about my request and he pointed me to a small restaurant. It genuinely wasn't a place for tourists, as nobody in the restaurant could speak any English - a good sign! It was such a simple dish - freshly made fried dumplings with vinegar on the side, but it made for  a very delicious lunch. I was able to (again with my broken Chinese) have a small conversation with some locals around the table also having lunch, and at this point I felt a real sense of belonging.

After this, I returned back to the hostel, re-packed, and caught my return bullet train to Beijing.

The guy sitting next to me happened to have worked for high-speed railways for the past 13 years.

Results of Regression

It has been about two weeks since my last blog post. Not good.

Interpolation

In terms of research, I have managed to produce some reasonable results in terms of PM2.5 prediction. I performed a variety of experiments to see how accurately predictions can be made.

The first of which is checking the interpolation ability of GP regression. I first looked at each 100th observed data point to train on, and used a larger set of every 10th data point to test on. We want to see whether the ten unseen points between each training example produce reasonable results. The hyperparameter which was varied between each experiment was the 'length scale', while keeping the covariance function as the squared exponential. It practically determines how strongly the predicted curve will try to fit the data. Theoretically, it is a measure of how 'close' two points in the training set need to be for them to be seen as 'significant' and be assigned a greater weight. This behaviour is similar to what is seen in weighted linear regression.

Here are some results.

We can see from these result that the larger the length scale is, the less likely we will encounter problems with overfitting the data, but the less accurate the data seems to be from what is expected.

Extrapolation

An alternative experiment to the above is to try to extrapolate from the data and predict future values given a few sequential data points to train on. I happen to find these results the most interesting so far.

Unfortunately I am not able to present the findings here because the SSH connection to the server at Tsinghua is horrendously slow. As soon as I get a better connection I will try to upload the images.

Meat and Matrices

What have I been up to?

I will write about my experiences in the past few days - in particular, Wednesday and Thursday.

On Wednesday, I experimented more with MATLAB, forgoing my code for implementing GP regression and instead lookeding further into the GPML package, which contains many helpful features for the purpose of GP regression. It is also very fast, which helps. I spent most of the day trying to get my head around the documentation provided online and in the source code.

Due to tradition adopted from university, every Wednesday is a 'sports night', which was initially designed for members of different sports clubs and societies at university to get together and mingle, but nowadays it is a time for anyone from anywhere to get together and intoxicate themselves. Therefore, we headed to the 'Lush' bar in Wudaokou (五道口), a popular student hangout, to attend a pub quiz. We ended up coming third, winning beer, alongside more beer offered in another bar further down the street for all quiz participants.

On Thursday, I invested more into the GPML package, learning more about its power and how it can help me in my task. From my data set of 'training examples' (a 5022 x 19 matix) I narrowed it down to just the past 24/48 hours to learn from, which led to much quicker computation times and allowed me to actually see some helpful results. I generated a test set to be just like the training set but with a little bit of normally-distributed noise added to make it a bit more realistic, but will allow me to see how close the predictions for the noisy test data is to the corresponding training examples.

The figure below shows the results. The red curve shows predictions, the blue crosses are the training examples target values, and the region between the two blue curves is a 95% confidence interval. GP regression is very handy in that for each prediction, it returns how uncertain it is about such prediction.

Results with my test data = training examples + a tiny bit of noise

Pretty accurate, I thought. All was well and good until I changed my training set to be the data points given at every 100th hour. I modified my test set to be the original data, and wanted to see how well the GP could train itself to be able to replicate the original data. I noticed that at every point that wasn't exactly a training example, the algorithm spat out a big fat zero. Not good. I received a WeChat message that we were meeting to get dinner at 5:30pm so I had to leave this problem until tomorrow.

Dinner was amazing. A friend of mine had a relative who would be studying in London next semester, so naturally the Imperial lot alongside his extended family members ate out to celebrate. It was a traditional Chinese 'hotpot', where a huge array of meats and vegetables are thrown into a large steaming brew located in the middle of the dining table. It was a great chance to relax and meet some of my friends extended family (and extended friends).

What will I change for tomorrow?

I will find out what is wrong with the GP code and why. 

I had a nice discussion with somebody from the dinner party about my experience in Beijing. It was nice to reflect on how much I have learned from the difficult situations forced upon me since arriving, but which have helped me immensely during my trip. It was a reminder to ensure that my final week in Beijing is not a wasted one.

Back in Beijing

What happened in the last few days?

Apologies. I arrived back in Beijing on Sunday afternoon, after Zixiao's father kindly drove me to Zibo train station, helping me for my trip back to Beijing. My trip wasn't too bad, as I remembered to pack food for the train. I had a truly successful and highly enjoyable stay in Zibo.

When the sun set, I immediately left my apartment to try out my new camera (a Canon Rebel T1i) in low light conditions - I think it performed reasonably well. Here is a picture of my bike just outside the main building of Tsinghua University.

It was 7pm at the time, but the floor was still super hot and I was covered in sweat.

On Monday I went back to the lab, amazingly managing to leave the apartment before the overzealous room-cleaning team kicked me out. 

I spent most of my day trying to figure out how I could speed up an experiment. I changed my code for GP regression to handle multidimensional input, changing the kernel function to compute the difference between two vectors in the input matrix, and taking the inner product between itself, i.e. <x1 - x2, x1 - x2>, to represent how "close" two data points are. I tested out my code for just one data point and realised that waiting for minutes for just one prediction really isn't practical.

I wanted to take a picture of the sunset because the weather outside was absolutely beautiful with clear blue skies for the entire day, but I ended up finding one of my friends during dinner and talking with him for just a little too long, completely missing it. I ended up with a shot much worse than I hoped for. The pollution today (Tuesday) is a lot higher, so it's likely I won't be able to see such a nice sunset again.

Probably the last time I'll see these mountains in the east before leaving Beijing.

What should I change for tomorrow?

Plan my day better! Spending too long on a problem without progress kills productivity. Delegating a specific, shorter amount of time for one purpose and focusing all attention on it is far better than spreading yourself thin and trying to do many things at once. This is partly responsible for my failure to write a blog post yesterday. Also, I should really learn more of the settings of my new camera - it appears to really pack a punch in terms of functionality and power.

Pampering Parents

What did I get up to?

It was an incredible day in Zibo (淄博) - a recurring theme of today was the presence of amazing food and a massive amount of it.

The day started with a very Chinese breakfast, including tea-boiled eggs, rice porridge (粥 - "jook"), meat pie and some cakes. Myself and Zixiao left to explore the surrounding area and visit his local park, which was absolutely beautiful and picturesque.

At noon, we met with one of Zixiao's good friends from high school - "Mr. Liu". After walking around a bit more, we eventually found another one of his friends and we entered a Chinese buffet for lunch.

It was a huge contrast to the "buffet" of the sort found in London. It was a barbecue buffet, where the food was initially raw and we were expected to cook it on hobs incorporated in our tables. It led to weird and wonderful combinations of meat and seafood-heavy dishes.

After lunch we caught a bus to visit the History Museum of Qi Kingdom. The Qi state during the time of the Zhou dynasty in China had the present day Zibo as its capital (it was called Linzi at the time). It was inspiring to see how important the city was and still is to the present day.

We (Zixiao, Mr. Liu and I) then headed back to Zixiao's home, where we had an incredible meal professionally prepared by his parents. I really was in awe at the quality and sheer volume of the food which was consumed. Oysters were eaten, along with shrimps, home-made sausage, tofu, and squid among many, many others. I was constantly being prodded to eat more and try new foods i have never come across before. To wash everything down, Chinese white wine (apparently with over 50% alcohol content) was consumed along with the traditional Tsingtao beer and many more cups of green tea. I was completely full afterwards.

This has by far been one my best days since touching down on Chinese soil. I really did feel at home with the Wang family (if not slightly overwhelmed by the kindness and of course, the food). Watching Zixiao interact with locals, his friends and his family brought me great joy, as this is of course something I had never seen much of in halls in London.

The overall response from the locals of the city was extremely positive. With many people not having seen a foreigner before, there was the occasional fits of giggles from girls and requests to take photos, but smiles all round. I would love to spend more time here - I just wish my train back to Beijing wasn't at 10am tomorrow morning!

How has this helped me?

I have learned the great importance of maintaining good relations with my close friends. Zixiao was a truly incredible host, with himself, his parents and his friends bending over backwards for me countless times to make my experience just that little bit more pleasant. It made me seriously reconsider how I normally treat my close friends, and ask myself whether it is on a level of kindness anywhere near what Zixiao displayed to me today.