Skip to main content

THE 2016 WAGS/PROQUEST INNOVATION IN TECHNOLOGY AWARD:

Morgan Dixon, University of Washington, Seattle

Pixel-Based Reverse Engineering of Graphical Interfaces

Research Summary
It is incredibly difficult to make computers easy to use. In fact, most of the time and money needed to build a typical application is dedicated to the user interface, the part of the application that includes the touch screen or keyboard, the graphical display, and the logic that responds to our commands. Unfortunately, this logic needs to run on many different devices and serve a wide variety of people with complex needs. For example, applications may be required to run on the desktop, on mobile touchscreen devices, both for people with motor impairments and able-bodied users, and in multiple languages. As a result, most of today’s interfaces require an immense amount of resources to address the increasingly complex ecosystem of devices and users.
My dissertation focuses on unlocking existing user interfaces to overcome these challenges. Similar to how Wikipedia opened encyclopedia articles for anyone to edit, I envision a similar democratization of user interfaces, where anybody can modify the graphical interface of any application. Any single company or group of developers cannot single-handedly address people’s wide variety of needs, but unlocking existing interfaces could diffuse the cost and effort across many people.
As a first step towards realizing this vision, my dissertation explores how to modify existing user interfaces from their pixels, the tiny dots that make up an image on a screen. By reverse engineering the pixels in existing graphical interfaces, we have opened them up for people to change and improve, without any compliance from the creators. For example, we used our methods to modify popular applications like Adobe Photoshop, Apple iTunes, Facebook, Gmail, and Microsoft Office. We did not have access to the source code of these applications, but we modified them in a variety of ways, such as customizing for people with disabilities, improving their touchscreen behavior on mobile devices, and translating their text to different languages.
Our pixel-based methods, implemented in a system called Prefab, can also be used to transition promising research out of the lab and into the wild. For example, we used Prefab to realize the Bubble Cursor, one of the fastest techniques used for pointing a mouse or finger on a computer or mobile device. Target-aware techniques like this make pointing and clicking easier, especially for people with motor impairments. Unfortunately, these improvements are difficult to deploy because current toolkits do not support them. As a result, the Bubble Cursor was invented 10 years ago, but could not be deployed in practice. Prefab allows us to implement techniques like these as general enhancements over the entire desktop.
Prefab also can be used to transform existing desktop interfaces into mobile touchscreen interfaces. This is important because new hybrid devices – such as the Microsoft Surface – include a touchscreen and a mouse. However, these hybrids currently require manual implementation of two interfaces: one for the mouse and one for touch. Instead, Prefab can be used to automatically monitor and manipulate existing desktop widgets – the small applications that run a device’s basic functions such as a clock or calendar - and overlay them with new touch-screen-based sliding widgets. Prefab’s models can also be used to intelligently render the sliding widgets with an appearance that matches the style of the original interface.
These examples demonstrate some basic capabilities described in my dissertation. Ultimately, I see this as a first step toward unlocking existing user interface tools so that developers can explore new ways to build interfaces inside the current ecosystem. For example, a human-computer interaction researcher developing a new interaction technique might evaluate it in several real-world applications. A practitioner or hobbyist who sees the researcher’s prototype might add the technique to several favorite applications. Web communities might organize around causes, such as translating interfaces into new languages, improving the accessibility of applications or updating interfaces to better support ink, gesture, speech and other advanced interactions. I envision a future where the creation of software is a disseminated community-driven effort, so we can help the any person in any facet of their life

 

Research SummaryIt is incredibly difficult to make computers easy to use. In fact, most of the time and money needed to build a typical application is dedicated to the user interface, the part of the application that includes the touch screen or keyboard, the graphical display, and the logic that responds to our commands. Unfortunately, this logic needs to run on many different devices and serve a wide variety of people with complex needs. For example, applications may be required to run on the desktop, on mobile touchscreen devices, both for people with motor impairments and able-bodied users, and in multiple languages. As a result, most of today’s interfaces require an immense amount of resources to address the increasingly complex ecosystem of devices and users.

My dissertation focuses on unlocking existing user interfaces to overcome these challenges. Similar to how Wikipedia opened encyclopedia articles for anyone to edit, I envision a similar democratization of user interfaces, where anybody can modify the graphical interface of any application. Any single company or group of developers cannot single-handedly address people’s wide variety of needs, but unlocking existing interfaces could diffuse the cost and effort across many people.

As a first step towards realizing this vision, my dissertation explores how to modify existing user interfaces from their pixels, the tiny dots that make up an image on a screen. By reverse engineering the pixels in existing graphical interfaces, we have opened them up for people to change and improve, without any compliance from the creators. For example, we used our methods to modify popular applications like Adobe Photoshop, Apple iTunes, Facebook, Gmail, and Microsoft Office. We did not have access to the source code of these applications, but we modified them in a variety of ways, such as customizing for people with disabilities, improving their touchscreen behavior on mobile devices, and translating their text to different languages.

Our pixel-based methods, implemented in a system called Prefab, can also be used to transition promising research out of the lab and into the wild. For example, we used Prefab to realize the Bubble Cursor, one of the fastest techniques used for pointing a mouse or finger on a computer or mobile device. Target-aware techniques like this make pointing and clicking easier, especially for people with motor impairments. Unfortunately, these improvements are difficult to deploy because current toolkits do not support them. As a result, the Bubble Cursor was invented 10 years ago, but could not be deployed in practice. Prefab allows us to implement techniques like these as general enhancements over the entire desktop.

Prefab also can be used to transform existing desktop interfaces into mobile touchscreen interfaces. This is important because new hybrid devices – such as the Microsoft Surface – include a touchscreen and a mouse. However, these hybrids currently require manual implementation of two interfaces: one for the mouse and one for touch. Instead, Prefab can be used to automatically monitor and manipulate existing desktop widgets – the small applications that run a device’s basic functions such as a clock or calendar - and overlay them with new touch-screen-based sliding widgets. Prefab’s models can also be used to intelligently render the sliding widgets with an appearance that matches the style of the original interface.

These examples demonstrate some basic capabilities described in my dissertation. Ultimately, I see this as a first step toward unlocking existing user interface tools so that developers can explore new ways to build interfaces inside the current ecosystem. For example, a human-computer interaction researcher developing a new interaction technique might evaluate it in several real-world applications. A practitioner or hobbyist who sees the researcher’s prototype might add the technique to several favorite applications. Web communities might organize around causes, such as translating interfaces into new languages, improving the accessibility of applications or updating interfaces to better support ink, gesture, speech and other advanced interactions. I envision a future where the creation of software is a disseminated community-driven effort, so we can help the any person in any facet of their life.

 

HUMANITIES, SOCIAL SCIENCES, EDUCATION AND BUSINESS:

Maegan Hough, University of Victoria

Personal Recollections and Civic Responsibilities: Dispute Resolution and the Indian Residential Schools Legacy

I call on all Canadians – elders and youth, Aboriginal or not – to commit to reconciliation and breaking down the wall of indifference. This is not just a dream, it is a collective responsibility. – Rt. Hon. Michaëlle Jean, former Governor General of Canada, Truth and Reconciliation Commission of Canada Honorary Witness.1

Do present-day Canadians bear any responsibility for the actions of past Canadians?  Saying “no” effectively absolves Canadians from any responsibility to correct any lasting effects of harmful actions. Saying “yes” means challenging the assumptions underpinning our social, political and legal institutions. Saying “no” ignores the reality of tens of thousands of Canadians.  Saying “yes” requires Canadians to re- think who they are as a people. This is the dilemma I found myself in when I entered the Law and Society Masters of Law program at the University of Victoria Faculty of Law.

My research was based on my personal experience attending Independent Assessment Process (IAP) hearings as part of the Indian Residential Schools Settlement Agreement while working for the Department of Justice Canada in Whitehorse, Yukon from 2009-2012. The Settlement Agreement is the biggest class action settlement in Canadian history and involved approximately 80,000 survivors of Indian Residential Schools. At IAP hearings I met with survivors who suffered sexual or severe physical abuse at a Residential School, I listened to them tell stories about their childhoods, their experiences at Residential School, and of the effect of the schools and the abuse they suffered on their lives. My title was “Canada’s Representative”. My role was to make submissions to the impartial adjudicator about the appropriate categories of abuse and related harms and the compensation that should be given the survivor.

I was deeply affected by this process. I quickly realized the stories the survivors told and the harms they described were often more about loss of language and culture and family ties than about the sexual and severe physical abuse they suffered.  But the process, the IAP, was focused on the specific acts of abuse that would, under different circumstances, form the basis of a criminal charge such as sexual assault. Consequently I began to question the value of the IAP and the ability of any legal process to address historical wrongs, especially those, such as loss of language and culture, which fall outside of accepted categories within criminal and tort-law. Perhaps more importantly I began to question my role, as a Canadian citizen, in the creation of the Indian Residential Schools (IRS) system and my role in its aftermath.

My research explores the legal, social, and political dispute resolution mechanisms available in Canada to address harms as they have been applied to the IRS.legacy. I use auto-ethnography, a form of writing that uses my autobiography as a way to explore and explain legal and political theory, to address the role non-Indigenous Canadians played in the creation of the IRS policy, the way it was carried-out, and the way individual citizens and our government has dealt with the harms caused. Central to my need to explore these questions were the interactions I had with coworkers, family members, friends and strangers on the subject of residential schools and its aftermath, including the IAP. Some were sympathetic to survivors and the compensation process, many were unaware, and just as many were hostile to the idea of compensation or reparations either from a historical perspective (don’t they know we won the war?) or from a legal perspective (these claims would never be proven in court so I don’t see why we should be paying these people anything). As a young lawyer working in a city far away from my family and normal support structure, I struggled with how to respond to these comments that differed so greatly from my own opinions, and with how to carry out my job as Canada’s Representative when my assessment of the needs of the survivors I met was so different from what the IAP offered them. I was confronted by my country’s history and found myself incapable of a meaningful response in either legal or social terms. My research is one of only a handful of academic projects that attempts to address historical injustices between Indigenous and non-Indigenous Canadians from a non- Indigenous perspective.

My first challenge was to the once dominant narrative that the Residential Schools were created and run by well-meaning people, but with tragic consequences for some students who suffered sexual abuse. In fact, the Residential Schools were one tool of colonialism, separating Indigenous children from their families in order to assimilate them into a “civilized” way of life. Once this is understood the category of harms caused by the residential schools policy expands from a few instances of abuse to large-scale harm to Indigenous language, culture, and family ties.

My second challenge was to the legal framework of crime and tort that has been the predominant mode of dealing with these harms. To date most of the standard euro- Canadian legal processes have been applied to the harms stemming from the residential schools policy. Unfortunately the narrowly framed crime of assault or the tort of vicarious liability have not been able to capture the full range of harms stemming from the residential schools policy. Assumptions of what constitutes a “harm” embedded within those processes make them incapable of recognizing the full range of harms. My research demonstrates similar limitations in the several political and social processes that have also attempted to address the harms of the residential schools policy. The Canadian Government has applied thirteen measures in all – political apologies, criminal trials, civil litigation, commemoration, a Truth and Reconciliation Commission, and compensation, among others – to the harms of the residential schools policy. Despite all of these measures my own experience was that no one process could address the massive historical and inter-generational scope of the harms caused, and that the effect of the combination of the thirteen processes was impossible for individuals to comprehend.

The result of this analysis confirmed what I had been feeling while acting as Canada’s Representative. I could not place myself, as a Canadian, in those processes. If the criminal law was the most appropriate process, I had to be personally guilty for the loss of a language. If civil litigation was really the best choice, I was liable for all the damage caused by my government’s one-hundred year old decisions and would spend therest of my life paying compensation. I could not accept either of those conclusions and neither could most of the people around me. The temptation to disengage with the questions of harm and reparations was just too strong. I needed a different framework to engage with my country’s history, and my own present role as Canada’s Representative.

I found this framework in the concept of “responsibility”. Instead of being guilty or liable for the harmful acts, I could recognize that these acts, above all the creation of the racist IRS policy, were done for my benefit as a future Canadian and that, as an intended “beneficiary” of those policies, it was my responsibility to correct any decisions I did not agree with. I propose that Canadians, as a society, need to reframe and restart our discussion about harms and reparations using this framework of “responsibility”.

Throughout the thesis I return to my auto-ethnographic journey through Canadian history and the IRS legacy. As a non-Indigenous Canadian I needed to place myself within the problem in a way that both recognized the harms, and recognized my distance from the creation of the IRS policy. Finding those concepts of “beneficiary” and “responsibility” allowed me to engage with the harms of the IRS legacy with both humility and purpose. I end my thesis with a discussion of several social and political processes that could allow all Canadians to engage with the IRS legacy. Without this engagement no attempt at reparations for the harms of the IRS legacy will be complete.

 

1 Quoted in Truth and Reconciliation Commission of Canada, British Columbia National Event Program (18- 21 September 2013) at 35

Biological Sciences, Mathematics and Physical Sciences, Life Sciences, and Engineering:

Cameron Jack, Oregon State University

Colony Level Infection of Honey Bee Gut Pathogen, Nosema ceranae and Role of Pollen Nutrition in Nosema cerenae Infection and Bee Survival

Honey bees (Apis mellifera L.) are arguably the most important pollinator in agricultural systems with an estimated economic value of pollination at $17 billion per year in the United States. However, recent honey bee declines in North America, Europe, and several other countries have been repo1ied to be above the normal expected losses, drawing attention to the vulnerability of worldwide agriculture to pollinator loss. One of the main suspects thought to be involved in bee decline is the fungal pathogen Nosema ceranae.

 Nosema ceranae spores reproduce within the midgut epithelial cells, severely damaging the honey bee digestive tract (the main site of nutrient absorption). As a result, infected honey bees commonly demonstrate pronounced hunger and are less likely to share food with nestmates. Furthermore, Nosema ceranae disrupts the metabolism of proteins in the midgut and causes further energetic stress. The disruption of protein metabolism is also manifested in the atrophy of hypopharyngeal glands, which are necessary for the rearing of brood, thereby affecting the health of the colony. Nosema ceranae infected colonies have also been observed to exhibit higher winter mortality rate than non-infected colonies. This may be due to the premature death of individual bees during the winter, which reduces colony population and leads to poor thermoregulation and decreased honey production.

 Like other animals, bees require adequate nutrition to thrive. Honey bees get the vast majority of their required protein, lipids, vitamins and minerals from pollen. Since honey bees can forage for pollen within a 3-mile radius of the hive, land management practices ave an enormous impact on how honey bees obtain food and what is available. As current farming practices are largely dependent upon monoculture cropping, a rapid decline in pollen diversity in agricultural areas has narrowed the range of floral resources available to bees, negatively affecting colony survival. Reduced pollen diversity available to honey bees may negatively impact brood production, gland development, reduce individual bee weight and shorten a honey bee's life span. Previous studies have shown that a direct result of nutritional deficiency causes a decrease in the resistance threshold of bees to bacteria, viruses, and pesticides.

 An antibiotic has been found to be effective for Nosema ceranae suppression, but because there are no outward signs of Nosema ceranae infection, many beekeepers have felt compelled to treat their colonies prophylactically. Prophylactic treatment has significantly increased hive management costs and can potentially reduce the effectiveness of the treatment. In order to properly employ preventative measures to keep colonies from succumbing to Nosema ceranae infection as well as preventing the overuse of the treatment, a more reliable Nosema ceranae sampling protocol is needed to provide beekeepers with a realistic view of colony infection levels. Further, several Oregon beekeepers have claimed that supplemental feeding of protein to their colonies has increased colony survival and helped reduce their reliance on the treatment.

 Since the role of pollen nutrition in Nosema ceranae infection and survival of infected bees is largely unknown, I sought to elucidate our understanding in this area