AI Recommendation System and the Visualization in a Cloud-based Collabration App

AI & Data Visualization & Design Process

My Role

I was responsible for leading the project planning, requirement definition, feature and functional design, user interface and prototype design, and patent application.

Background and motivation

IDC conducted a global survey to understand information worker and IT professional's needs and challenges in North America, Europe and east Asia. They found that Employees spend over 8 hours a week searching and gathering information [1]. McKinsey global institute conducted a survey about social technology and productivity across more than 20 countries and 30 industries. Thet also found the similar resutls, especially, by improving searching and gathering information could help increase 30-35% productivity [2]. People spent not actually completing projects, but rather sifting through documents and other company knowledge in order to complete them.

Since 2019, many employees have switched to fully remotr work due to the pandemic lockdown. A survey published in this year from 704 researches in University in Europe who work from home reported that almost 50% researchers who work from home found that their work efficiency was decreased [3]. Last year, a research from Forbes [4] stated that more than 75% of employees admit their productivity has plummeted due to distractions related to working from home and the pandemic. Many people had to work overtime than regular hours, some because they were virtual baby sitting or schooling. Some because they couldn’t complete the teamwork across different time zones.

From these data, we might presume that most people wanted to go back to office immediately, however, a survey conducted last year about Canadian workforce of the future across all industries and provinces revealed that only 20% employees say they want to go back to their workplace full time [5]. Besides, they also found that almost all industry sectors assess that the necessary tools and technology are decreased than pre-pandemic. Public sector workers in education, government and healthcare report a high decreased access to the tools they need to do their jobs.

We've known that many companies have utilized AI recommendation system for improving the information searching efficiency. Particularly, Netflix, Amazon and YouTube got significant success in profits by implementing their recommendation system. However, many current recommendation systems display the results with very limited explanation and interaction to the users. People don't understand how and why the recommendation was generated and what would happen on the next recommendation.

From the research I did above, we aimed to improve collabration apps' effciency of communication during the work. I concluded three questions/goals and the solutions, correspondingly.

The Proposed Solution and Architecture

For achieving the goals, I proposed that the best way is to using a AI-powered recommendation system when users searching information. I also proposed that the system should at least have 3 different functional modules to cover the most needed use cases. At last, we should make the system more visible, understandable and interactive.

Taking the communication module for example, my propsed system architecture generally consists of three parts, i.e., feature corpus, AI algorithm engine and interactive user interface. The offline corpus genarates the static user profile model as the ice breaker. The online corpus on the cloud modifies the user profile dynamically.

User Interface Design

I designed two fundamental UIs for two different scenarios based on the system architecture. The first one is to pre-define the rules(recommendation tags for the offline corpus when the users log in at the first time). It allows the users to select recommendation ranking tags among several features from the data lake, and prioritize them based on their preferences.

The second design is for visualizing the recommendations and providing multiple interactions for updating user behaviors of the online corpus. It displays the pre-defined rules and their results. Users could select the results from the individual recommendations or group recommendations, refresh the ranked recommendations, swap the rule orders or edit the pre-defined rules.

The Demo

Visualization of Recommendation System on a collaboration app from Heather Q on Vimeo.

References

1. https://warekennis.nl/wp-content/uploads/2013/11/bridging-the-information-worker-productivity-gap.pdf
2. https://www.mckinsey.com/industries/technology-media-and-telecommunications/our-insights/the-social-economy
3. Aczel, B., Kovacs, M., van der Lippe, T., & Szaszi, B. (2021). Researchers working from home: Benefits and challenges. PloS one, 16(3), e0249127.
4. https://www.forbes.com/sites/vickyvalet/2020/03/12/working-from-home-during-the-coronavirus-pandemic-what-you-need-to-know/?sh=1693a6721421
5. https://www.pwc.com/ca/en/today-s-issues/upskilling/canadian-pulse-survey.html
6. https://www.researchgate.net/publication/303953909_Evolution_of_Recommender_Systems_from_Ancient_Times_to_Modern_Era_A_Survey
7. https://www.pwc.com/us/en/tech-effect/ai-analytics/ai-predictions.html
8. He, C., Parra, D., & Verbert, K. (2016). Interactive recommender systems: A survey of the state of the art and future research challenges and opportunities. Expert Systems with Applications, 56, 9-27.

Competitor Analysis on B2B products

Usability Testing & Design Process

My Role

I was responsible for the entire project indluing backgroung research, analysis, test conducting and reporting.

Project Goals

The purpose of competitive analysis is to find out the differences between other products, understand the strength of our own and the weakness of the competiors, thus suggest improvements on product design and development, and tackle strategies for product branding and sales.

Pain Points

Product Comparison

1. Feature Comparison

2. Software Architecture Comparison

3. Software Updates, Iterations and User Reviews

User Testing Comparison

Viewpoint direction in VR

VR & Empirical study

My Role

I was responsible for the experimental design, building the scene/environment, and developing the software(interative machanism and test beds).

Introduction

When we're in a virtual cinematic environment or a casual 3D/VR game, it seems a bit arbitrary to complete the goals, although there was somewhat storyline embedded in advance. Some techniques such as naratives are used for the storytelling but it highly affects the immersion and presence, especially in VR. We decided to utilize certain stimuli to redirect the user's attention to complete a full and fluid VR narative.

Methodology

Since our basic methodology is to use various aids in a few immersive scenarios presenting a story, the auditory and visual senses are likely the most common aids used in virtual or augemanted environment. The scenes would have a few objects with auditory or visual aids presenting in an order continuesly, and ask the users to find the right object. Hence, we need to implement gazing and hearing mechanism as the input techniques for users to receive the aids, implement pointing mechanism as the output according to the hardware to measure the performance and compare the results, also, we need to develope a test bed including the data collection and log file setup.

1. Hardware

HTC Vive Pro Eye

HTC Vive Pro Eye is HTC'S latest head-mounted display with dual-OLED and a resolution of 2880x1600 pixels. It features the eye-tracking techinique in both eyes that enables various eye-based activities in VR, such as mirroring what the users see, rendering the fovea's resolution, and even controlling the interaction by eyes. We selected HTC Vive Pro Eye as our prime hardware as we could collect accurate data from eyes that other HMDs can't provide.

Google Cardboard

The Google Cardboard has fold-out cardboard viewer into which a smartphone is inserted. The platform was intended as a low-cost system to encourage interest and development in VR applications. The reason we selected Google Cardboard as the secondary hardware is due to the pandemic lockdown. It was difficult to invite participants to the lab to tyr on HTC Vive Pro Eye personally. We mailed out the Cardboard to all participants and provide instructions on how to install the experiment software.

2. Software and Experimental design

I used Unity and c# as the software development tools.

a. Independent variables

1. Input type: auditory and visual

2. Aid type: diegetic and non-diegetic

The auditory and visual inputs are the stimuli enabled both with sounds and visual changes, associated with appropriate objects. All the independent variables form the experiment with 2x2 within-subject design. Each combination is one condition. The combination is randomized by the software. See the object list below.

The diegetic and non-diegetic features occurs within or without the context of the story. Take the sound for example, diegetuc sound is sound that comes from the setting of the story, such as the utensils clattering in the background, or music coming from a piano being played in the story. Non-diegetic sound is sound that comes from our world, such as the film music or voice-over narration.

b. Dependent variables

Error rates, task completion time, immersive evaluation and cybersickness evaluation.

The process

Due to COVID-19 pandemic, the lab has been closed and in-person user study is forbidden. We had to choose Google Cardboard as the first phase experiment. The experiment sessions are totally remote, the participants take instruction via on-line meeting with the moderators. Therefore, before conducting, we had several steps of preparation. First, we recruited participants who had large-screen mobile phone and were able to install application apk. Then we bought many Google Cardboards and mailed them to the participants. Upon arriving, we also taught them how to assemble the cardboard and install the application. During testing, we asked them to "think aloud" and complete the questionnaires after each condition. The software recorded all user interactions in the log file. After done with the test, we asked the participants to find the file in the mobile phone and send us.

The results

The experiment is still on-going. More details will be revealed later.

Redesign a Recipe App

Mobile App & Usability Testing

Introduction of the App

This recipe app was designed for helping end users find and follow recipes when using a specific cooking appliance. All the recipes were customized to the cooking appliance and could be remotely controlled by the app and voice via Google and Amazon IoT speakers as the future features. The goal of this redesign project was to find all usability issues, make changes thus get better prepared for further development of more features.

My Role

I was responsible for the entire redesign project including planning and executing proper evaluations, analyzing the results, compiling the reports and redesigning the UIs.

The Process

The process was divided into 2 main parts, usability testing and UI improvements. To locate the current issues, a usability testing and a post-test interview were conducted. I recruited 10 participants from the local community who had different levels of experience of the appliance product and the app.Usability issues were found and addressed. In the UI improvements stage, some small user studies such as card sorting, wizard of Oz and A/B tests were conducted to validate the results of the usability testing and UI design directions.

Usability Testing

4 major modules of the app were testing that covered 9 scenarios and 24 tasks, and evaluated by several metrics such as success/error rates, task time, Jacob Nielsen’s 10 heuristics, system usability scale (SUS), Whitney Quesenbery’s 5Es, severity rankings, etc.

According to Nielsen’s heuristics, I found that the below aspects were rated in lower scores:

• User control and freedom
• Recognition rather than recall
• Help users recognize, diagnose, and recover from errors

From locating the issues based on each screen, there were 4 major objectives for the redesign:

1. Simplify the information architecture
2. Unify screen elements including colors, fonts and icons
3. Refine the wording
4. Simplify the interactions
5. Make the workflow smoother and clearer

Here're a few examples.

1. Color Design for text labels

In the app, there were 3 different gray colours that all represented the minor information of texts and they were clickable. This inconsistency confused the users that the darker text might be clickable but the lighter text was unclickable.

Solution

These 3 different colors should be unified into one color. I chose the darker gray to distinguish from unclickable gray color.

2. Home Screen's Architecture and Wording

In the usability testing, all the participants found it was difficult to navigate on the home screen: the user guide section was split into 2 parts for reinforcing the branding value and was entitled with the company's name according to the needs of marketing; the different sections of recipes were not well identified and named, participants didn't understand the "featured recipe" and didn't show any interests on the "newest recipe".

Solution

Redesigned the layout of the home screen after a few iterations and A/B tests. See below.

3. Icon and Its Interactions

There were 3 different colors of checkmarks represented different meanings in the App. In the usability testing, all the participants were very confused when multiple items were selected, especilly after adding a few items to the "Grocery" list, 3 different colors of checkmarks were all displayed on the screen. Users were hesitated to select items. They could not undo the operations correctly and foresee the potencial errors.

Solution

To make the screen simpler and clearer, I removed all the gray checkmarks that reprents "not checked item". To distinguish the confusion of two different interactions "check" and "add to the grocery", I split them into 2 different places and icons. A shopping cart icon was added to represent the current numbers of the items on the grocery list.

See the comparion below.

Result

After a few iterations of the redesign, the SUS(system usability scale) score was greatly increased from 39.5% to 76%.

Design Data Visualization for a Reading Log Application

Data Visualization & Design Process

Introduction/Background

This data visualization has been designed for being embedded in a SaaS system to show users' reading logs. The reading log was a part of personal management system to self-manage user's time and resources for both screens and off-line activities. The objectives of this project was to design a novel visualization of several types of reading activites that should be different from the traditional data charts.

My Role

I was responsible for the entire design project, deliver the high-fidelity prototype and collaborate the software development.

The Requirements and Constrains

The stakeholders requirements were to visualize the reading sources/types and sentiment based on the time, describe the user's reading routine easily in a fresh manner. The data should be summarized and displayed in periods such as day, week, month and/or year. The reading sources and sentiment would be categorized into segments in the furture (not the first version). The major requirements are listed below:

1. Present the time.
2. Present the reason and the source of users' reading based on user's inputs.
3. Present the sentiment of each reading based on user's inputs.
4. Allow the filtering based on the time.
5. Allow the filtering based on the reading source.
6. Allow the filtering based on the sentiment.
7. The primary time window could be a day or a week.

There were also a few constrains from the requirements.

1. Do not use any traditional data charts that could be derived from Excel, Tableau, etc.
2. The visualization would be embedded into part of SaaS dashboard, the space for the visualization was limited.
3. Need to consider potencial design features for acommodating more detailed features in the future.

The Process

Low-fidelity Prototype

Designing data visualization is similar to app design that should follow some guidelines and stardards. For example, the first step was to identify data types, marks, channels and visual encodings.

Some low-fidelity prototypes by paper and pen

A High-fidelity prototype based on above sketches and user testing (A/B tests, interviews and surveys)

Design Voice UI for a Senoir's Painting App

IoT & VUI & Design Process & Empirical Study

My Role

This was a consulting project of a future painting app using voice recognition technology. The project's goal was to investigate and evaluate the feasibility of the app. The app was defined as a therapy app for seniors according to the skateholders' initial requirements. I was responsible to identify the user needs and design the effective features, functionalities, architecture, interactions, and future user study and evaluations for UI design and software development.

Introduction

Nowadays, drawing and painting is highly recommended to seniors as an art therapy. Seniors living with a number of ailments may benefit from art therapy. This is a targeted use of painting to help improve the memory, reduce stress and alleviate pain in a setting that is fun, casual and social. Some researches show that painting, drawing and sculpting in old age may protect against dementia. Meanwhile, after working hard for the family for some decades, many seniors would like to look for a long term hobby that they might want to pursue in their younger ages but give up due to busy and fast-paced working for their careers and money. However, there are many constrains against painting by seniors due to their health problems. First, to hold the drawing pencil, it needs the fingers bend their joint bones properly to draw accurately. To grasp the painting brush, it requires the muscles support the arm in the mid air and work together with the wrist and fingers. The seniors may not be able to hold the brush for a long time because of muscle atrophy or arthritis. Many painting stands are designed for standing upright, they might also be very large that requires the artist to bend the waist and knees. At this moment, the seniors would not be able to perform such actions, especially for those with mobility issues. Additionally, the painting activity also requires the artist to be very concentrated and remember her initial creative idea since the painting work would last for many times or days. Therefore, although many seniors have the needs on drawing for different purposes, the painting activities require very much efforts on mental and physical health conditions.

As the result, the regular means of painting activities by hands could not fulfill the special needs from senior citizens. We know that some artists with disability utilized mouth or foot to paint, but that cannot solve the problem for seniors because they would also cause very high physical fatigue, besides, mouth or foot painting needs training. And some neurofeedback technologies, for example, using EGG signals [1, 2] to paint requires expensive devices, highly trained researcher and users, and technical contains in terms of accuracy, which cannot be widely used for general public. Inspired by the neurofeedback technologies, voice that can perform orders from the mouth directly, is a more reliable way to control painting than EGG signals. Voice has been utilized as an alternative to control IoT products like smart lights, bulbs, TVs, etc. since the past few years. By speaking out the orders and actions that the seniors want, then convert the orders into output devices to complete the tasks, it needs much less physical efforts and possibly less mental focus if is is designed properly. The seniors have no need to stand, walk, lift the arms or even bend the fingers. Thus, from connecting the needs of painting and problems of seniors, the voice recognition would be a solution for solving them.

Technology Affordance

Back in 1988, Froessl [3] invented a method to manipulate image by speech signals. In 1989, Porter et al. [4] invented a more explicit voice recognition system based on previous inventions about a a text locating system recognizes spoken utterances, uses the recognized words as a search string, and searches text for words matching that search string. In early 21 centuries, Wright et al. [5] utilized spoken commands to receive data and manipulate the actions on computers. Then, many other researchers like Erell et al. [6] optimized the voice recognition technology like synthesized speech fragment for independent input devices. Empirical studies [7] started to investigate the efficiency between voice and other modalities like hand-writing and type-writing in 1970s. Cohen et al. [8] compared naturual language interaction with GUI and direct manipulation and found that applications requiring speedy user input of complex descriptions favoured spoken natural language communication. Thanks to these pioneer studies, many applications about voice recognition have been developed and evaluated since then. Igarashi et al. [9] proposed an application about using nonverbal voice input like pitch and volume for directing control interactions. Then, Perera et al. [10] reported their experiments and results to use sound volume and a few sound patterns to control the drawing for upper limb disabilities. They believed that paralinguistic voice can assisting artists with upper limb disabilities to perform tasks like creating visual art. Based on the academical accomplishments, the industry started to develop voice assistant applications from Apple Siri, to Amazon Alexa and Google. Especially for Alexa, which was just released in 2014, announced in April 2019, Amazon had over 90,000 functions ("skills") available for users to download on their Alexa-enabled devices. As we have already known from the literatures about the mechanism of how the voice recognition system process inputs and outputs. Let us take an example of Alexa to see how voice was converted to actual applications. In Amazon Alexa, a skill is an entire function that includes intents, slots, and utterances.

An intent represents an action that fulfills a user's spoken request. Intents can optionally have arguments called slots. The sample utterances are a set of likely spoken phrases mapped to the intents. When you create a new custom intent, you provide a name and a list of utterances that users would say to invoke this intent. Then you identify slots for each intents. A skill can have many intents as user actions like add, delete, select, paint, erase, etc. An intent also can have many slots like the time, length, number, etc.

In addition to building the skills, Alexa has utilized AI to learn natural language understanding, natural language generation, context modelling, common-sense reasoning and dialog planning.

Initial Design

The picture below shows the system architecture of the painting application by voice recognition with Amazon Alexa. The reason I chose to use Alexa because it has been a very mature ecosystem that provides many output devices and Amazon Web Service (AWS) including the cloud. The output devices can be a tablet, a computer, a smart TV or even motor-based mechanical arms that can draw pictures on more traditional materials like paper according to users’ needs. Regarding to the possible issues of visual impairment, I added an extra input, that is a camera that can assist the voice input to amend the orders when the seniors are copy painting or natural sketching. The image processing could allow the users to learn the dimensions of objects and understand better the picture structure. All the data will be sent to Amazon cloud for storing and machine learning.

Before designing the Alexa skills, we should understand how to describe an object in painting language. For example, the basic elements in painting are dots, lines and shapes. When we draw a dot, we would first locate it on the canvas, then decide the size (or the diameter) of the dot, at last we would decide the color. If drawing a line, we would decide the beginning and the end of the line, the degrees or the slope, the length, then the color. And when drawing the shapes, we would select among a circle, an oval, a rectangle, a square, a star, etc., then the diameter, the color of the border and the inner part, and even the shadow, transparency, reflection patterns. After finishing the structure of the picture, we would decide how to color the entire picture and make refinement by combining some shapes, erasing some lines, etc. Therefore, this Alexa Skill named “Painting Skill” could have several intents like “draw”, “fill color”, “delete”, “erase”, “expand”, etc. Each intent might have several slots like element types, locations or coordinates, size, diameter, and patterns. The element types may have parameters like “line”, “circle”, “rectangle”, “star”, etc. We can also add some templates for several common objects like trees, flowers, birds, etc. and provide a pattern source of database for each template.

When considering the real users, we need to divide the user needs based on some segments. For instance, the seniors citizens would have different levels of painting skills regarding to novice and experienced users. They might also want to learn or utilize different levels of difficulties of painting skills depending on the scenarios. For example, if a novice senior who has never painted before, she might ask Alexa to put the objects by words “left, right or top”, then ask the Alexa to move left or up to make changes. However, the experienced users may have more accurate control on the painting brush or the cursor, and might be able to use four quadrants to locate.

For the seniors who have visual impairment, the computer vision from the camera could tell them the general dimensions and locations of the objects they want to paint, or take pictures of what they have been drawing in process thus tell them what they have so far.

Design experiment and evaluations

Before designing the explicit low and high-fidelity prototypes, we need to assess if this initial design can solve the problem at the beginning. We could evaluate from the objective and subjective point of view as listed below:

1. What is the performance of voice recognition on painting for senior?
2. What is the satisfaction, immersion and frustration?

Experimental design

Independent variables:

Inputs: voice drawing, hand-drawing, computer graphic drawing.

Dependent variables:

Objective: Task completion time, errors made in the process like the usages of “erase” and “delete”, heart rate before and after, blood pressure before and after.

Subjective: SUS questionnaire to measure usability, learnability and consistency, satisfaction and fatigues questionnaire.

We should also conduct a longitudinal study to evaluate the changes the participants made after repeating the sessions in a long term. The seniors have longer learning curve because they have worse memories and slower reactions than adults so a longitudinal study could be better to understand the system effectiveness.

References

1. Münßinger, J. I., Halder, S., Kleih, S. C., Furdea, A., Raco, V., Hösle, A., & Kubler, A. (2010). Brain painting: first evaluation of a new brain–computer interface application with ALS-patients and healthy volunteers. Frontiers in neuroscience, 4, 182.
2. Zickler, C., Halder, S., Kleih, S. C., Herbert, C., & Kübler, A. (2013). Brain painting: usability testing according to the user-centered design in end users with severe motor paralysis. Artificial intelligence in medicine, 59(2), 99-110.
3. Froessl, H. (1988). U.S. Patent No. 4,726,065. Washington, DC: U.S. Patent and Trademark Office.
4. Porter, E. W. (1989). U.S. Patent No. 4,829,576. Washington, DC: U.S. Patent and Trademark Office.
5. Wright, B. D., Dubach, J., Parmenter, D. W., Gold, A., Young, J. H., & Gould, J. M. (2003). U.S. Patent No. 6,601,027. Washington, DC: U.S. Patent and Trademark Office.
6. Erell, A., & Melzer, E. (2005). U.S. Patent Application No. 10/857,848.
7. Ochsman,R.B.&Chapanis,A.(1974)Int.J.Man-Mach.Stud. 6,579-620.
8. Cohen, P. R., & Oviatt, S. L. (1995). The role of voice input for human-machine communication. proceedings of the National Academy of Sciences, 92(22), 9921-9927.
9. Igarashi, T., & Hughes, J. F. (2001, November). Voice as sound: using non-verbal voice input for interactive control. In Proceedings of the 14th annual ACM symposium on User interface software and technology (pp. 155-156). ACM.
10. Perera, D., Eales, R. J., & Blashki, K. (2009). Supporting the creative drive: investigating paralinguistic voice as a mode of interaction for artists with upper limb disabilities. Universal Access in the Information Society, 8(2), 77-88.

Sentiment Analysis of Products on Amazon by Using Deep Learning

AI - Machine Learning - Deep Learning

My Role

I was responsible for using an effective methodology to investigate the advantages and disadvantages of competitors’ products on Amazon. The results would help the marketing define the long-term roadmaps and short-term strategies, help the high management and product team to develop new products.

The project methodology and technical details are demonstrated below, however, the explicit data and results about the product and their competitions are moved for the purpose of business privacy protection.

Introduction

As e-commerce has been more and more popular since this couple of decades and many consumers’ on-line shopping requirements have been raising, e-commerce platforms such as Amazon and eBay have accomplished great success and opportunities. Meanwhile, the competition of product sales and marketing in the industry is fiercer, especially for consuming products. In this background, the necessity to understand the customers’ requirements will be more and more important before improving the product quality and cutting down the product price. Among the marketing research approaches, listening to user’s voice, e.g., researching on users’ comments and complains on that product would help the product manager, designer and other stakeholders to understand user’s real needs, the products’ pain points from a large scale more efficiently and accurately, rather than usability testing that has very small participant sample, or survey that could be more time and cost-consuming.

Sentiment analysis is also called opinion mining or emotion AI. It identifies, extracts and quantifies affective states and subjective information systematically by deploying techniques such as statistics, language processing, text analysis, computational linguistics, and biometrics. It is a classification process for analyzing textual data about the emotional statement such as positive, negative and neutral. Sentiment Analysis is also one of the more advanced tasks in natural language processing. The ultimate goal of this task is to make the computer understand the human emotional world.

Latent Dirichlet Allocation (LDA) is a topic model that can present the topic in each document in the document set based on the probability distribution. It is one of the methods of sentiment analysis. It can read the general tendency in certain topics that could be classified as regards positive and negative sentiment. The advantage of LDA model is that there is no need to manually label the training set during training, all that is needed is the document set and the number k of specified topics. In addition, another advantage of LDA is that for each topic, some words can be found to describe it.

Solution and Technical Affordance

The picture below shows the process in this project. First, I found the product review from Amazon by data crawling programs in Python or a few existing tools. For conducting sentiment analysis, the reviews were separate into 3 categories, that were positive, negative and neutral. The “4 Star” and “5 Star” reviews were merged together as positive reviews, while “1 Star” and “2 Star” reviews were merged together as negative reviews, and the “3 Star” reviews were neutral. After the essential data formatting, I deleted unnecessary punctuations, applied tokenization, cleaning, removing stop words, stemming and lemmatization to pre-process the text. I applied LDA to conduct feature extraction, topic modeling and vectoring, then applied Word2Vec to obtain the word vectors. After those, I combined two kinds of vector matrix and imported the combination into convolutional neural network (CNN). At last, I calculated the accuracy, precision, recall and F-measure for the model with different classifications. And demonstrated the results of the sentiment analysis based on the top 10 topics in each product categories.

In this project, there were 3 major technical barriers. The first was topic modelling and clustering by applying LDA model. Topic model is a statistical model used to discover abstract topics in a series of documents. Intuitively speaking, if an article has a central idea, there must be some specific words that appear more frequently. However, in reality, an article usually contains multiple topics, and the proportion of each topic is different. For example, 10% of an article is related to Apple Inc., and 90% is related to Microsoft Corporation, so it is related to Microsoft Keywords should appear 9 times more often than Apple keywords. The topic model is an automatic analysis of each document, counting the words in the document, and judging which topics the current document contains and how much each topic accounts for according to the statistical information.

The second was Word2Vec, a toolkit for obtaining word vectors that Google open sourced in 2013. It is simple and efficient and has attracted a lot of attention. Word2vec can quickly and efficiently express a word into a vector through an optimized training model based on a given corpus, providing a new tool for applied research in the field of natural language processing. The word2vec word vector can better express the similarity and analogous relationship between different words. It has been widely used in natural language processing tasks since it was proposed, and it has become a prerequisite for a powerful natural language processing task.

Third, I applied the CNN model similar to Kim’s structure proposed for text analysis [1]. He utilized filters with different lengths to convolve the text matrix. The width of the filter is equal to the length of the word vector. Then he used max-pooling to operate on the vectors extracted by each filter, therefore each filter reflects to a number. If stitching these filters together, we could get a vector that represents the whole sentence. The final predictions are all based on this sentence. This model has served as a classic model for a few years and has been evaluated effectively in many research papers [2].

Evaluation and Results

When doing classification in machine learning, we utilize some metrics to evaluate the advantages and disadvantages of algorithms, such as operational analysis, efficiency analysis, etc. The major functional analysis includes the accuracy, loss, precision, recall, F1, AUC, ROC curve, etc.

The comparison table with actual values of training and testing sets are shown below:

In the result, the polarity of sentiments based on the top 10 topics in each product categorie, especially the competitors’ products were analyzed. All the word clouds and their sentiments of each product were also visualized to reveal their great features and pain points. The results benefited many stakeholders in the company including the marketing team and the product team.

References

1. Kim, Yoon. "Convolutional neural networks for sentence classification." arXiv preprint arXiv:1408.5882 (2014).
2. [1] Sun, Fanke, and Heping Chen. "Feature extension for chinese short text classification based on LDA and word2vec." In 2018 13th IEEE Conference on Industrial Electronics and Applications (ICIEA), pp. 1189-1194. IEEE, 2018.

Eye-Based Selection and Navigation in Virtual Reality

VR/AR & Empirical Study

Introduction & Motivation

FOVE HMD was the first commercial VR handset with eye-tracking on January 2017. In order to evaluate the efficiency of the eye-tracking selecting performance in FOVE, I conducted an experiment based on the international standard, ISO 9241-9, which utilizes Fitts’ law to evaluate pointing devices. I compared three different selection techniques using the FOVE HMD: 1) eye-based selection without head-tracking, which I dub eye-only selection, 2) head-based selection without eye-tracking, dubbed head-only selection, and 3) eye-tracking and head-tracking enabled at the same time, henceforth eye+head selection. I compared these selection techniques across several different combinations of target size and depth, based on past work in 3D selection.

My Role

I was the first author of this study. I researched many related works, designed the experiment, developed the software, conducted the experiment and analyzed the data. My supervisor provided me with FOVE HMD, along with many ideas of experiment design and advice in the whole research project.

My Challenge

I encountered many challenges in this project. The first was that was my first time to use FOVE, a VR HMD with eye-tracker. Before start developing, I tried out several traditional ye-trackers including Tobii, Eyetribe. I downloaded FOVE's plugin and examples then tried to change the examples in Unity and C#. The second challenge was that no studies had used 3D Fitts' law to implement and measure eye-based and head-based selections. After several versions of the software, I was finally able to get the right result.

Results

We found that the head-only selection offered the fastest selection times and the best accuracy. Moreover, it was strongly preferred by participants. The combination of eye-tracking and head-based selection (our eye+head input method) performed roughly between the other two, failing to leverage the benefits of each. The results indicate that, at least for the time being and in the absence of more precise eye trackers with better calibration methods, head-only selection is likely to continue to dominate VR interaction.

This paper was published and presented in ACM SUI (Brighton, UK), and was elected in “Highlights of SUI”. I was then invited to present it in ACM ISS’17. This paper has been highly cited for many times by formal journals and conferences since Nov 2017.

The Follow-up Study: Navigation by Eyes in VR

After the selection study, I developed and conducted the navigation study in VR by eye-based technique. A benefit of eye-based navigation in VR is that it may require less physical effort compared to other input devices (e.g., wands) to explore the envoronment and move between locations, especially in large three-dimensional spaces.

This paper was published by the 6th ACM Symposium on Spatial User Interaction (SUI 2018), in Berlin, Germany. It was also chosen as one of the top papers and invited to present my work as a poster in UIST 2018 (31st ACM User Interface Software and Technology Symposium).

Smart Safety Helmet and Wristbands for Cyclists

IoT & Empirical Study

Introduction & Motivation

I found many cyclists have been suffering injuries on the roads in Canada every year. Cyclists are probably pedaling in a dangerous situation but they have not realized it. Because I was hit by a motorcycle when riding a bike and then suffered two surgeries many years ago, I was strongly worried about the situation that cyclists were facing.

From previous researches, major reasons behind bicycle collisions include inattentive riding/driving, riding too close to motor vehicles, lane merging, and riding too fast. Therefore, this year, I proposed the design of a smart safety helmet that alerts cyclists from potentially dangerous situations. This smart safety helmet prototype was linked to two accompanying wristbands. When the smart safety helmet senses an approaching object, the safety wristbands alert the user in response.

My Role

I was the idea initiator, prototype designer, hardware and software developer, statistics analysis and coordinator of user research and experiment hosting. Our team had 4 members.

My Challenge and Design Process

It was the first time that I used Arduino to develop prototypes. I took this responsibility because the other three team members didn't have the electrical engineering background. I researched and finally developed the entire prototype system successfully.

The other challenge was that we were mainly approaching the project from a human-computer interaction perspective to evaluate the effectiveness of different types of feedback in a screen-free interface. We brainstormed and discussed, finally confirmed that we would test haptic, auditory, and visual feedback to the cyclist through the smart wristbands for exploring the most effective notification of awareness of direction and cyclist's turning decision.

Results

We discovered that the visual notification should be the most effective, and the combination of audio and visual might be a good alternative. The haptic notification could cause misunderstanding and neglect.

This study was published as a poster presented in the First Annual Symposium of CLUE in Ottawa, Canada.