Reporting Bugs
From skylightwiki
This page applies to Version Beta1 (Build numbers v1.0.0709.13.1137 (ALPHA) to ...) of the application (Sept 13 2007-...)
Reporting Bugs in the Software
Bugs will be tracked with http://mantis.ctlt.wsu.edu. You will security warning, just click through
Intro to Mantis
- You can log into Mantis with the same account that you use to log into Skylight and this wiki.
- Click OK to get past security warnings (we don't pay for certificates for internal tools)
- Mantis is being used to track several projects at WSU, so in the upper right of the screen, switch to "Skylight" project
- In the toolbar, "Report Issue" can be used to pull up a form for reporting problems.
- "View Issues" will give an overview of things in the system and their status
Reporting Documentation Errors
You are encouraged to make small corrections to the documentation in situ (its a wiki, be bold), but you may prefer to report problems with documentation as described below
Log into the wiki with your Skylight Username and password and edit the appropriate wiki page(s). Near where the problem occurs in the text, or at the bottom of the page (which ever seems better) summarize the problem with this template tool:
You would type:
{{Template:Bug Noted| Your problem summary here }}
For example:
{{Template:Bug Noted|You have spelled "Toledeo" wrong on this page}}
You can make your description fairly long right in the bug note, or you can add other information as needed after the }} of the template.
Recording Feature Ideas
Login (link upper right) with your WSU Friend ID& password, and then edit this page. Follow the pattern of those before you, adding a new item with at title surrounded by 4 equals (=) signs and then a body.
Folders
We must be able to nest folders. There will be just too many surveys, and even groups of surveys, to which authors can be given access. I'd prefer to give all users access to a folder called "Flashlight" for example, in which there were a number of folders (and perhaps some folders inside those), in order to make exploration easier.
Sharing folders also creates the need for folder names that alert the author to the fact that the folder was created by someone else. if you create a folder called "course feedback forms" and later make that available to a colleague, that colleague may forget (or may not even know) whose folder it really is. My solution: automatically stamp a folder with the name of its creator (maybe the stamp isn't visible to the creator, just to anyone else who can see it) and include more information about the folder's origin in "properties" (name of 'business entity', date of creation).
Reordering question groups and pages
It's essential to do this more easily. (Now the only way to do it is to start a new survey, and copy questions from the older survey but in a different order.)
Managing Multi-Author Matrix surveys
Both the authors and contributing authors of matrix surveys will need tools to help manage the length of the response forms. What sort of tools might they be? A graph showing authors and contributing authors the current number of pages, questions, or question elements for the distribution of forms (from shortest to longest)? A messaging utility to help authors and contributing authors communicate? The ability for authors to create a warning or even a mandatory stop when the longest form exceeds some threshold of questions or question elements?
Display of meta-tags needed for matrix surveys
It would be much easier to create matrix surveys if one could display all the keys and keyvalues created thus far for this particular survey.
For example, today I returned to a matrix survey we use regularly for evaluating workshops, in order to generate a new respondent pool. In other to find the metadata definitions I would need to define the new pool, I first had to go to the questions and read carefully through them, looking for all the metadata. I wanted to be able to pop up a list of defined metadata while in the respondent pool interface.
Being able to see a list of existing metadata defined by questions or respondent pools is even more urgent for multi-authored matrix surveys. Authors can't risk accidentally using the same metatags for different purposes; they need to see updated lists of metatags in use in their surveys, with links that will permit them to understand how that metatag is used/defined.
Even when I work alone, I can easily lose track of the keys and keyvalues in my survey. Even if I create an 'outside' document where I keep track of them, I might make a mistake in entering one of them (e.g., writing "Activity" as a key in one place and accidentally writing "Activities" in another place).
To sum up, Skylight needs a command that allows the author of a matrix to automatically display and sort all key/value combinations, together with a link or pointer to where that key/value was entered. Ideally the list should be clickable so that if I discover that I've accidentally created two different versions of what I intended as the same metadata ('activity' in one place, 'activities' in another), I can go directly to one of them, and fix it.
Enable 'batch' commands
When doing a lot of copying or deleting (e.g., copying 7 items from an item bank into a survey; or copying the item bank and then deleting the 25 items you don't need) or executing certain other commands (e.g., changing 6 surveys to "read only" (we need a read only property; more on that in another post), it is very, very important to a) click boxes beside all the objects or files on which a command is to be executed, and then b) execute the command on all of the simultaneously.
It would be my preference to have this capability in place before we roll out the system, but let's talk first if that would have an impact on the roll out date. I think this one thing will make the system much easier to use and administer.
Making Flashlight Online more intuitive
When we go from the draft interface to the real interface, it's very important that actions such as creating a matrix survey, and creating a URL for survey administration, be made more intuitive. In the draft interface, there's no way to guess how to do either of those functions. Someone has to train you. Which means, if there's a wait of some months between your first survey and your second, you're very likely to forget how to do those things.
Visible feedback about progress of execution
When uploading a large number of respondents, whether by pasting them into a respondent pool or uploading a spreadsheet, the system should quickly give a visual sign that the upload is in process. Otherwise users are tempted to click "add" a second time, which can then upload the respondents twice.
Novice interface
Nils suggested some time ago the possibility of a novice interface that would hide some features that novices aren't likely to need at first. We do need to think about some basic functions that are too disguised by the current interface, e.g., 'starting' a survey.
Dates of survey creation and modification visible in Survey List
when you have lots of surveys in your folders, you sometimes want to find a survey quickly that you created recently. In the "Survey List" you ought to be able to see, and sort by, each of those dates.
Location of "add" buttons and screen display after a 'save'
Two features of the system that are, together, time-wasting: 1. The buttons for adding and editing are only at the top of that block. So, if you need to add a new question group, you first need to scroll to the top of the page, even if you've been working at the very bottom of a long survey, and 2. Once you add the new question group or page, the screen display returns you to the top of the page. So you now have to scroll back down to where you were. Anyway to change the interface so that less scrolling is needed when adding elements to a long survey?
"Move" of a survey versus "copy/paste" of a survey
For the purposes of managing one's surveys and data, there's a big difference between "move" of a survey from one folder to another, versus copying/pasting the survey to a new folder and then deleting the original. Consider a survey that's already started, with data collected. Now you want to reorganize your surveys by moving some of them to different folders. But currently you can't do move a survey with data to a new folder.
In Flashlight Online 1.0, a survey with data could not be deleted. Is that also the case in FLO 2.0?
"Outline view" of survey
A Flashlight Online 2.0 survey is organized hierarchically - question groups, pages, questions, question elements, just as a Word document might be organized with four levels of headings, or a set of files might be listed four levels deep. In the latter two cases, there is usually more than one way to compress your view of the document (often by clicking a plus sign to expand a section, or a minus sign to compress it), or to expand or compress your whole view (think "outline view" in Word or PowerPoint) so that all you see are the section headings. Especially for matrix surveys, I think we'd benefit from having flexible ways of viewing survey content, and perhaps of doing editing at that level - displaying only question groups, for example, and then deleting an entire group, or moving a group up or down, while in that "view."
Question Group, Page, and question numbering
Authors need to be able to see number of question groups (by respondent pool), number of pages (by respondent pool), and perhaps also number of questions (by respondent pool). is "number of choices" the key measure for each respondent? number of clicks? Should we add a field to each question - estimated time to respond to this element. if we add this, how do we make it intuitive? have the field with an automated default for number of seconds for that element, with freedom for the author to change that value?
*Controlling the size of the open response field in multiple choice questions, or making the field larger
I love the ability to simply add an open text field for respondents with each leaf of a block of questions. But the fields are too small for my needs. I think they subtly discourage people from writing more than a phrase. Ideally i should be able to control the size of these fields, just as I do for the open response items. Second choice: at least make the field extend to the right edge of the screen.
Item Banks
I'm exploring ways to manage the Current Student Inventory (CSI) and I’ve run into a challenge. I’ve begun by creating a small item bank containing active learning questions from the CSI. I’ve filed the item bank in TLT webinar examples.
Take a look at a question like 0397S018. It’s indexed as active learning (A) and creating high expectations (E)
In your opinion, to what extent were each of the following given priority in this course...
This use of meta data is a strength of Silhouette: we can have individual lines in a block of questions indexed by two subscales in FLO1, AND accessible via the table of contents, AND the index. So where do we put an item like this in Flashlight Online 2.0 so that it’s easy to find and use?
At this moment, the only (inelegant) solution I can see is: a) use a subscale like active learning to bring up all those items in FLO1, break them into smaller item banks, and put each small item bank into FLO2 (e.g., “active learning, no specific technology, priority”) b) then use another such subscale and do it again (e.g., raising expectations, no specific technology, priority) (entering some items a second time) c) then use the table of contents, break it down into small item banks, and enter those, too (entering some items a third time)
After finishing this procedure, a couple hundred CSI items will have been put into FLO2 three times, and we’ll have a huge collection of small item banks (50-100, each with 5-15 items).
I’m also not clear about how to make these and other Flashlight templates and item banks easily visible to all users. As we collect classroom assessment instruments and student course evaluation resources from subscribers, review them and want to make some of them visible to other users, this challenge will grow. I hope we can come up with a better solution than we have in Flashlight Online 1.0 (the list of templates, with a view option, complemented with this web page: http://www.tltgroup.org/Flashlight/FLO/Templates.htm
More on item banks
We need better tools for managing item banks of more than a few items. The problem is that authors can act on only one question at a time. Imagine an item bank of, say, 30 items of which the author wants to use, say, 8 items. Right now the author has only two options: copy the item bank and delete 22 items, one at a time. Or copy paste 8 items, one at a time, from one survey into another. Most systems allow a strategy for designating many members of a set, and then performing the same action (e.g., delete, copy/paste) on all the items of the set simultaneously. Our new interface should allow that, too.
*Offering a set of folders to which new users can request access
It would be great if new users logging in for the first time could be given automatic access to a number of folders (e.g., if the new author is a physics faculty member, the folders might deal with surveys and feedback in science teaching). It would also be good to offer access (grayed out folder?) where consent of the manager (?) of that folder is required
Using tables as a mode of data entry; using tag clouds for reporting text entries in quick report
Two different but related ideas for future versions.
1. Give authors the ability to create simple (e.g., 3x3) tables for respondents to enter numbers or text in each cell.
2. For items (like that) where respondents enter text, provide tag clouds as the way/one way to see data in a quick report.
Suspending rather than deleting author accounts
Because faculty and administrators are mobile and may move from a Flashlight Online institution to another institution, I'd like to have the option of allowing John Smith's account to be reactivated after moving to another institution. We should ask institutions about this, though. Intellectual property issues?
Time log for survey response (option)
A user has requested the option of getting a record of how long a respondent took to take a survey. this would be an aid to 'cleaning data' (if all the same options were picked throughout, and the 20 item survey only took 30 seconds to complete, the author might choose to delete that record from the analysis, for example). Informed consent forms require you to estimate the time for the effort required in a study. This would provide a measure.
Ability to print surveys in some kind of routinized way
Some institutions will want to offer a hybrid model: paper surveys for faculty who insist on them. How easy/hard would it be to manage a print job for, say, 10% of all sections of a large university? to print out in formats that allow 'scantron' data entry?
Filter for surveys that receive less than a certain number of percentage of responses, relative to a target
this may already be a feature of the reporting routines (something you can do easily with Excel, across all reports?). I'm sure some programs will want to hide results from classes where only a minimal number of students respond. I don't see this a crucial feature, but it would be nice to offer if it's easy to provide and manage. One of our contacts at Carnegie Mellon mentioned the desire for this.
New question element: fill in the blank for each answer option
Fatimah Taherbhai of Clayton College suggested a question type that could produce a question like this:
How many of each of the following types of animal are on your farm?
___ cows
___ pigs
___ chickens
___ ducks
___ (other type of animal: _____ )
___ (other type of animal: ______ )
Numbering questions, or some other way to easily identify questions
Numbered questions makes it easier to ask a pretester for feedback on a survey (e.g., question 17 is vague and could use another answer option"). can we give authors the option of autonumbering of question elements for this purpose? or is there some other way to make it easier for pretesters to reference of attach their comments to particular questions?
Custom Report - Replace All Content (possible bug?)
Currently, only the data on the "Data" worksheet is replaced when using a Custom Report. It would be beneficial to have the content of all sheets replaced (esp. the Summary tab that contains the "Report created" field). In addition to this, additional Respondent Pools/Respondents might be added to a survey, after a custom report has been created -- thus the importance of replacing the data on those other tabs too...
Bugs not yet moved to Mantis
*can't delete an item that had been copied to another location in the survey
In this survey: ...SurveyID=61e522a8-6611-4ff7-9111-4acf16cf14d0 (removed URL pointing to MOCHA, databases are now split 2/22/08) I wanted to move the block format question near the bottom to a page of its own. i was able to copy it to a new page at the bottom, but I was never able to delete the original. I know how to use the delete command and have deleted other items, but I could not delete this one. I have no clue why: because it's block format? because it had been copied? some other reason entirely?
*Nonfunctional buttons that aren't grayed out
This is annoying rather than a serious bug. There are a number of buttons ('preview' for new question elements, reorder questions, search, and so on) that don't yet work (which is OK) but they aren't grayed out. Other buttons that don't work ARE grayed out, so a user has a right to suppose that these buttons would work.
"Leaf comments" button in quick reports doesn't work (Mocha)
I click the button but nothing happens.
Returning to survey manager, proper folder
I start from Survey Manager, pick a folder, pick a survey within the folder and, later, return to survey manager because I want to see a different survey in same folder. But the survey manager always returns me to the top folder in the list (which, in this case, isn't een a folder I use). So I need to take what feels like an unnecessary step to find my folder and click it. Shouldn't the default be to come to the folder in the manager that I've been using?
Canceling or Deleting a question creates an "Untitled Question"
"Canceling" or "Deleting" a question (in this case, a radio button-type question) produces an "Untitled" question that can only be removed by clicking the Delete Question button.