FPGA-Based Communication Portal for the Speech Impaired and the Physically Challenged

Engr. Antoniette P. Mondigo Adviser University of San Carlos Lloyd Matias M. Arbiol Researcher University of San Carlos lloyd_arbiol@yahoo.com Cathy Angelee B. Sarile Researcher University of San Carlos tatie.sarile@yahoo.com Gibson Nigel S. Tan Researcher University of San Carlos gibsonigeltan@yahoo.com Christopher Eduard I. Villondo Researcher University of San Carlos cevillondo@gmail.com Michelle D. Ybaez Researcher University of San Carlos ybanez.michelle_david@yahoo.com.ph

Abstract
As FPGA (Field-Programmable Gate Array) devices progressed both in terms of resources and performance, the latest FPGAs have come to provide "platform" solutions that are easily customizable for system data processing applications. As platform solutions are becoming more and more important, hardware designers are coming up with easy-to-use design development tools. The project aims to build a communication portal for the speech impaired and the physically challenged. The project is capable of suggesting words from the inputted letters to make efficient communication. The system lets the user choose what type of input he desires to use by pressing its corresponding button on the board. The system then enables the necessary modules that are related to the type of input chosen. If the user chooses to use the keyboard, he can type the desired input. If the user chooses the sensor or button input, a virtual keyboard appears on the screen and the user simply activates the sensor or button when the highlight is at a certain character and words are suggested as he is choosing letters. Communication portals make use of hardware and software components to be effective in delivering its purpose. The system is made up of several components that control the hardware and software modules like

2012 Engr. Antoniette P. Mondigo, Lloyd Matias M. Arbiol, Cathy Angelee B. Sarile, Gibson Nigel S. Tan, Christopher Eduard I. Villondo, and Michelle D. YbaezAll Rights Reserved 2012

the LCD, SVGA port, transmission of data from SD card to the system, etc. The output of this project is a pioneer implementation of hardware design using the Altium NanoBoard 3000 with Spartan3AN user FPGA, a recently acquired board by the University of San Carlos and it may be used as a reference for future project designs.

Introduction
Communication is a vital key in human interaction. Over the years, different methods and techniques have been developed in order to relay thoughts, feelings, and ideas to others. Among these, speech is the most commonly used since it is learned and developed as early as the infancy stage of human life. However, factors such as congenital defects and accidents or disorders that led to speech impairment caused some people to have difficulty in communicating through speech. To solve this problem, alternative means of communication have been used, such as simple gestures, sign language, lip reading, or alphabet boards. Nevertheless, for those with physical disabilities who also may not be able to communicate effectively using speech, other communication systems aiding people with this kind of impairment exist. Electronic or computer-based equipment are commercially available with emerging technologies, mostly geared towards software implementation of such. Speech devices are one of the most common communication equipments being used today. These are built adaptive especially for people with impairments and have difficulties in communication. Most of these devices are in forms of computers; specifically computer generated output

speech devices with alternative input devices such as alternative keyboards, ability switches, head pointers, joysticks, eye trackers, and even mind controllers. With this problem, the researchers aim to design and develop a communication portal for the speech impaired and the physically challenged. The user simply selects individual characters to form words, phrases, or a sentence out of a display of alphanumeric characters displayed on a monitor. Most commonly selected words was identified and suggested to make the system more adaptive. Character selection was done using a standard PS/2 keyboard, a button press or an alternative method such as an IR sensor interface, tilt switch, reed switch or a foot switch were developed for the paralyzed or physically challenged users. The proposed project was developed through VHDL (Very High Speed Integrated Circuit Hardware Description Language) and was implemented in an FPGA (FieldProgrammable Gate Array) chip.

Methodology

This section of the study illustrated the physical overview, input and output modules employed in the implementation of the whole project system. The major

2012 Engr. Antoniette P. Mondigo, Lloyd Matias M. Arbiol, Cathy Angelee B. Sarile, Gibson Nigel S. Tan, Christopher Eduard I. Villondo, and Michelle D. YbaezAll Rights Reserved 2012

components that are used in this study are the following: PS/2 Keyboard, Button, and the 4 Sensors for the input devices; VGA Monitor and buzzer as its output devices and the SD Card as its memory component. USING THE KEYBOARD A User Interface with a full-screen text area will be displayed on the monitor when the input device chosen was the keyboard. The user can type letters, and numbers which will be directly displayed on the provided text area in the monitor. Other keys which can be used were Enter, Space, Backspace, and Esc Keys. The Esc Key served as a clear display function for this system. The main User Interface for the keyboard is shown in the figure below. USING THE BUTTON OR SENSORS The infrared sensors, reed switch, tilt switch and foot switch are the sensors that are used. If either button or sensor was chosen as an input device, a User Interface with a display of alphanumeric characters, word suggestion panel, and a text area will be displayed on the monitor. A text area with alphanumeric characters, word suggestion panel and a text area were displayed on the monitor. The alphanumeric character display is divided into 4 lines which contain 12 characters including the special character operations (Space, Backspace, Clear Display, and End functions). Selection of characters was done first by highlighting each line one by one with the default delay. When the button is pressed or the sensor was triggered, the highlighted line will be selected. Each character on that line was highlighted one at a time with the same delay time. Other onboard buttons

are used for incrementing and decrementing the delay. Character selection was done by a button press or by triggering the sensor. A cancel or exit option was also displayed at the end of each line which will be selected whenever the user wishes to select other characters which were not found on that specific line. Once a character is selected, the system will search for a word starting with that character in the word bank. Search algorithm using hash tables and bloom filter is also applied. If found, the searched word will be displayed and highlighted on the word suggestion panel for three (3) seconds. If the user selected that word using either the button or the sensor, it will be automatically displayed on the text area. If the user failed in selecting that word within the given time, the system will assume it was not the intended word. However, the word suggestion panel was automatically updated if the user types in another letter which pertains to the words stored in the bank. The figure below illustrated the main User Interface for the button and sensor.

2012 Engr. Antoniette P. Mondigo, Lloyd Matias M. Arbiol, Cathy Angelee B. Sarile, Gibson Nigel S. Tan, Christopher Eduard I. Villondo, and Michelle D. YbaezAll Rights Reserved 2012

An external buzzer was also interfaced into the system. The user can trigger the chosen user input, either the button or sensor, to indicate that the user started character selection. The highlighting delay was displayed using the TFT LCD of the Altium NanoBoard 3000. The mentioned delay can be set from 1 to 10 seconds.

Data Gathering
To ensure that the system would work as expected, a plan of action was prepared to test the resulting project. There were two phases to test the project. One was to do a modular testing of each of the modules and the other, a fully integrated system testing. In doing a modular test, the different modules were tested individually. The different modules underwent a test bench simulation to check whether the expected outputs correspond to the correct inputs. The simulation was used to verify early on if something is wrong with a particular module. After the simulation of the module, it would also be implemented as a stand-alone system using the Altium Designer Summer 09s virtual I/O devices. This action verifies the validity of the simulation, whether an error exists between the theoretical and actual results. After a test of the individual modules, the system should also be able to run as whole. The modules were integrated. The integrated system was then tested for errors and existing bugs. If any should be found, a system fix shall be done to try to eliminate them. A system test shall be redone until a satisfactory system with the least problems is produced. Moreover, when the system operates well enough, the system was tested by five (5) test users to

ensure and verify the functionality of the system. The main objective of the testing was to measure the time a user can input a particular word using the different input devices. The system was evaluated by five test users. Two (2) tests were employed by the researchers. The first test involved keying in the words ATE, HELP, HOLLOW, POSITION and BEAUTIFUL. Five (5) input devices namely the tilt switch, foot switch, infrared sensor, reed switch and button were used in keying in the words. Each word has an ideal time in which a user can finish keying it in. With this, the researchers opted to record the time it takes for the user to actually key in the words. The actual time will then be compared with ideal time to determine whether or not the user had difficulty using the device based on the time intervals gathered. The ideal time can be calculated by considering that each highlight of a character or row is considered one count based on the highlight delay being selected. Based on the criteria for ideal time calculation, we can therefore get the ideal time of any word.

Presentation and Analysis of Data

The ideal time of the five words being used as test cases using the one (1) second delay are shown below. WORD ATE HELP HOLLOW POSITION BEAUTIFUL IDEAL TIME 35s 32s 43s 90s 94s

2012 Engr. Antoniette P. Mondigo, Lloyd Matias M. Arbiol, Cathy Angelee B. Sarile, Gibson Nigel S. Tan, Christopher Eduard I. Villondo, and Michelle D. YbaezAll Rights Reserved 2012

The researchers then tabulate the actual time the user keyed in the data using the various sensors/switches. The researchers were unable to acquire the actual time when using the IR sensor because the users found it difficult to key in the words. The IR sensor became very erratic, becoming very sensitive to environmental light that it cannot be able to give a good output. This is why data cannot be obtained for the IR Sensor. Initial testing done by the researcher also delivers the same output. The graph illustrates the real result of selecting characters to form words. It could be suggested that the switches were able to perform their function of allowing users to select the desired characters from the selection screen. By inspecting the graphs, some of the input devices took longer than the ideal time. One factor that affected this is that the switch was not very sensitive to the trigger of the user. This is one idea pointed out by a user during the testing. Another factor is that the user was still getting the hang of using the input devices. Most values for the foot switch and the reed switch return a favorable result, in the sense that they do not differ so much from the ideal time. The next test involves acquiring feedback from the test user in the form of three (3) questions. These questions will be used as an end-user recommendation for the design project. The first question asks about the difficulties encountered while keying in the words. Most of the testers complaints were about the switches and the character counting. The switches required much effort to operate. After long usage of the foot switch, the user felt discomfort due to the force exerted by simply pedaling down the switch. This complaint is understandable considering that the system and the

Computing the percent performance of a user in keying in a word After gathering the time durations for the five (5) test users, the researchers took the average time to key in each word using the different input devices excluding the IR sensor. Equation 4.2 illustrates how to achieve the percent performance per device per word. If average time duration took longer than that of the ideal time then the user performed slower using that specific device. Otherwise, the user performed either as expected or better when using that specific device. The figure below shows the average time the users were able to key in the words in percentage values.

2012 Engr. Antoniette P. Mondigo, Lloyd Matias M. Arbiol, Cathy Angelee B. Sarile, Gibson Nigel S. Tan, Christopher Eduard I. Villondo, and Michelle D. YbaezAll Rights Reserved 2012

attached sensors/button are prototypes. There are also occasions when the user enters a character highlight sequence the highlight would be ahead of the actual letter being highlighted. However, once the user exits the character highlight sequence the row/character highlight would be functional again. The researchers have yet to remedy this problem. Future generations who opt to use this document as a reference should take note of this problem and find alternatives that deviate from encountering such problem. The second question asks about the user recommendations or remedies for the difficulties encountered. Majority of the users recommended a better switch so that the data acquired would be accurate. The researchers also agree with this recommendation and are in line with the recommendation being presented in the next chapter. Also, one (1) user suggested fixing the character positioning or the layout of the virtual keyboard. This was in congruence with the recommendation for a faster highlighting sequence. One (1) second is too long of a time for a person to wait to travel from one (1) letter to the next. This can be considered however the test users were in perfect health. Situations and solutions can vary based on the type of user. Changing the keyboard layout can be feasible if the Altium NanoBoard is capable of higher resolutions than that of its current configured hardware. The last question asks if whether or not the users were able to view their highlighted word. For this part, all users were able to view the keyed in words but needed to use the word suggestion often to get used to the feature.

The table above illustrates the device utilization summary of the entire project. As shown, the number of slices consumed is a critical factor in implementing the design project. Also, the number of BRAMs consumed takes a significant toll in designing projects that involves soft-core microprocessors. The consumed BRAMs relate to the code memory consumed along with the kinds of peripherals that interfaces with the microprocessor (i.e. virtual terminal, general I/O ports, SD Card, etc.). The previous design exceeded the maximum number of resources, specifically by 12% of the slices. This left the researchers to embrace the alternative of using two boards. However, the researchers were able to optimize the design to fit on a single board. The design projects slice usage was condensed from 112% to a smaller amount of 91%.

2012 Engr. Antoniette P. Mondigo, Lloyd Matias M. Arbiol, Cathy Angelee B. Sarile, Gibson Nigel S. Tan, Christopher Eduard I. Villondo, and Michelle D. YbaezAll Rights Reserved 2012

Summary
The design project was to be able to design and develop a communication tool that may aide the speech impaired and the physically challenged. With this, users who are incapable of vocal communication will be able to communicate by simply typing using the keyboard or by selecting individual characters to form words, phrases, or a sentence out of a display of alphanumeric characters displayed on a monitor. For the system to be more adaptive, word suggestion was implemented. Different modules were also implemented. Through simulation, it was observed that these system modules were functional. The system modules were developed using Very High Speed Integrated Circuit Hardware Description Language (VHDL). These were then individually implemented and tested using the Altium NanoBoard 3000. System testing was done by collecting data from different test users. In analyzing the different modules, the interfaced input and output devices worked properly. The implementation of the SD Card as embedded software in a soft-core microprocessor, particularly the TSK3000, was effective in storing and retrieving data. However, various problems arose during the integration. One of which is the time it takes for synthesizing and completing partially integrated modules. Resource management posed as the biggest problem but was overcome by the proponents through integration of various modules into one single module. An input buffer was also included here to solve the synchronous input problem posed by the inputs used in the design. The system prototype is made out of transparent acrylic material. Mounted to it is the Altium

NanoBoard 3000 with the embedded FPGA project. Though enclosed in the prototype casing, the system however, is still subject to physical laws. It may not be strong enough to withstand being dropped or falling over. It is still ideal to handle with care.

Conclusion
Each module of the system was developed and completed. Testing was done modularly by the researchers and the final system was tested with the help of selected test users. All this was done using the available resources of the Altium NanoBoard. On that basis, the following have been achieved: Different VHDL modules were designed and developed using the Altium NanoBoard 3000. These modules were implemented as shown in the data presentation. The keyboard, buttons and sensors were interfaced to form words or phrases. Only the alphanumeric and special characters such as the space and backspace are usable for keyboard input and character selection. The sensors worked well with the exception of the IR sensor that became very erratic as it was very sensitive to environmental light. The external button was also found to debounce and may sometimes input characters multiple times. The SD Card was utilized for non-volatile storage and retrieval of words. The digital instruments of the Altium Designer ensured that the SD Card was indeed able to do these. Word storage and retrieval of only up to nine (9) characters was implemented on the embedded microcontroller. Searching algorithm was made

2012 Engr. Antoniette P. Mondigo, Lloyd Matias M. Arbiol, Cathy Angelee B. Sarile, Gibson Nigel S. Tan, Christopher Eduard I. Villondo, and Michelle D. YbaezAll Rights Reserved 2012

optimal by using the Bloom Filter and the Trie data structures. Some problems however occurred throughout the course of the project. This included the financial incapability of the researchers to obtain a license for a newer version of the Altium Designer. This newer version supposedly allows the use of the SD Mode to access the SD Card. To solve the situation, the SPI mode was used to access the SD Card. The trade-off for this however, is slower access time. Combining all the developed modules became a very hard obstacle for the researchers. The FPGAs available resources were not enough to cater to the initial design of the system. A module was redesigned to reduce the ROM space it was taking up. The system was finally implemented on a single board, however, it was not the same as the expected output. Noticeable glitches were found as characters were displayed on the SVGA. This was due to the exhaustive writing and reading of the SRAM. The final system was still able to store and suggest words of up to nine characters, but despite these mishaps, a communication portal was still evident taking up 91% of the FPGA slices based on the Design Summary shown previously.

Recommendations
Based on the data gathered and the conclusion, the researchers of this project recommend the following to improve the design of the system: Use embedded systems in addition to VHDL to implement some of the modules, such as the LCD and the SVGA. This may help reduce the use of FPGA resources. Instead of using the SPI mode to access the SD Card, the SD mode is recommended because it is faster than the SPI mode. This was not implemented because it required the use of newer versions of the Altium Designer. The license keys to the former were financially unobtainable for the researchers. Make the system also compatible to SDHC and SDXC cards because these are the newer and advanced SD Cards available in the market. Again, the drivers supported by the Altium Designer 9, which was the one used in this project, did not support higher SD card variants. In addition to the buttons and sensors used, the researchers recommend that researches be done on other kinds of buttons and sensors that can complement the system. It is recommended that they are stable enough to deliver the users desired characters.

2012 Engr. Antoniette P. Mondigo, Lloyd Matias M. Arbiol, Cathy Angelee B. Sarile, Gibson Nigel S. Tan, Christopher Eduard I. Villondo, and Michelle D. YbaezAll Rights Reserved 2012