[p2pu-dev] PD/Course Assessment/Metrics Mockups

Sun Jul 31 16:34:01 UTC 2011

Thank you for your feedback :) There's no need to apologize for giving me
more to think about. That's a good thing!

Regarding being able to focus on a certain task's activity: very good idea.
Right now, a task is a page on a course so it could be limited by that much
like how web analytics can be on a certain url pattern. I think there are
going to be different types of pages in the future: tasks, non-tasks. If so,
that can help. Example: "Show me all tasks" vs all pages that are under the
course like the "full description." If not, it could be a drill-down model.

And, really? In-line graphics didn't work? Aw :(

On Sun, Jul 31, 2011 at 7:30 AM, Pippa Buchanan <Pippa.Buchanan at gmail.com>wrote:

> This looks really great and is going to be so useful to help understand
> what's going on in a group.
>
>
> *Question: *Should the organizers be listed in the charts?
> Organiser involvement: yes, you should be able to toggle (all) organisers
> metrics off or include them.
>
>
> *Question:* Should the public be able to see these? Their view would not
> show participants' names, of course.
>
> Public viewing? I guess the question is how will this be useful to someone
> outside of the group?
> Maybe? Use case: I'm creating a course and want to find out how to create
> amazing tasks. If I could look at charts course by course and identify what
> the most popular / useful tasks are across different groups, I can use that
> information to improve my own tasks.
>
> If I were a researcher I'd like to have access to this data, but I'd use it
> in different ways.
>
>
> At the moment the focus is on activity over time. Which is useful,
> particularly when logging a course with a specific time period - it lets you
> identify where attention begins to flag. You can also track whether certain
> participants are always first to respond etc.
>
> It will also help organisers identify whether their continual involvement
> is useful for the group ("oh, i see - this was the week I had a cold and
> didn't have much energy to spare, and after that activity drops off, but it
> increases after I contributed 40% more time".
>
> If we can track how much time a specific user is logged in we'll also be
> able to use that to better identify how much time is needed to successfully
> participate at both a general learner and organiser specific levels. (On
> average learners spend x hours a week, and participants dedicate y hours per
> month)
>
> As a participant in a group I'd also like to see my own involvement data
> over time and to compare how I'm participating in multiple groups and to get
> feedback on my own time management.
>
> *By Task v By Time*
>
> What I personally would find very interesting is data about the tasks - I'm
> not sure if this is something that is planned.
>
> Which tasks have the most activity? What is the activity over time for a
> specific task?
>
> Which tasks are *edited* the most, and could we see if after specific
> edits the types and frequency of responses improve? (eg. I add a "How to
> respond" section in my task description - suddenly its explained more
> clearly, is easier to respond to and more comments are added).
>
> This will allow us to identify how useful a task is - are people answering
> it because it's easier than the others, or is there something in the
> structure and wording that attracts participants? Do people respond in
> bursts to tasks? What order do they attempt the tasks in? If a task has a
> deadline, what is the pattern of submission - who is responding first etc?
> Who is responding multiple times in a task - are they being lovely and
> sociable or are they in fact trolling the group?
>
>
> Sorry if i've given you more to think about Jessica! I think this is really
> interesting and positive step forward for the project.
>
> Pippa
>
>
>
>
>
> On 31 July 2011 04:03, Jessica Ledbetter <jessica at jessicaledbetter.com>wrote:
>
>> Greetings all (and especially Karen):
>>
>> Attached are some mockups of some course metrics' graphs/charts. I'd love
>> to get feedback from everyone by Wednesday. If that's not enough time, let
>> me know.  (Karen asked about getting a way to assess courses into P2PU and
>> these are the first draft attempts at what it could look like.)
>>
>> We start by going to the course's admin page. Right now it's a button
>> called "Edit Study Group" (or "Edit Course" or .. etc.) but I propose that
>> that becomes "Administer Study Group" or "Study Group Admin" to better
>> encompass the addition of metrics (and maybe some other stuff that's in my
>> ideabox). Also, we'd have a link to them on the study group's sidebar in the
>> box where Updates, Organizers, and Participant count are.
>>
>> Not obvious by this screenshot is that I also widened the box that this is
>> in so that it's the same width as the P2PU header. More real estate for all
>> this will be a very good thing :) In my experimentation, I wanted even more
>> room!
>>
>> The first chart on the metric page is* "Overall Activity"*:
>>
>> [image: projectAdmin.png]
>>
>> *1:* Can download this graph as a graphic. This will come in handy when
>> the graphs get too big for the real estate available or for inclusion in
>> presentations/reports.
>>
>> *2: *When mousing over points on the chart, hint text pops up with more
>> information. The line in this is an average of the 3 participants.
>>
>> *3: *Specifically for this chart, I was trying to show the "health" of
>> the course. Missing are Task Edits, Task Creations, Page Views by
>> Participants, Page Views by Non-Participants (this would have to be
>> somewhere else). I can add those here too, of course. I'm not sure if Time
>> on Page fits well with these other data points but I put it here since it
>> was requested. Maybe another chart that is more web analytics like the page
>> views and time could be together? I have a thought on "health" that might be
>> better in its own email. However, I talk about it below after all the
>> mockups ("Health Formula") :)
>>
>> *Overall:* This chart, and others, could become difficult to use with a
>> lot of students in the course. But I can stretch it out so that you can
>> scroll over to see it all or have a sort of zoom in feature so you can see
>> the overall lines and zoom in on certain areas.
>>
>> Also, I can make it so that when you click a bar that you "drill down" to
>> see more information like *when *the activity occurred for the person.
>>
>> *Next is "Activity Over Time"* which is a more specific metric for health
>> reporting.
>>
>> [image: chartActivityOverTime.png]
>>
>> Activities listed right now are: comments, task edits, participants' last
>> log in time, and how many tasks were added. We can add page views in phase
>> 2. We could also have a running "total" showing as well. Requested was the
>> ability to see by day, week, and total. Also, I can give this as a table.
>>
>> Next is* "Participant Comments."*
>>
>> [image: chartParticipantComments.png]
>>
>> Here we have a "total" for the day plus what each person did to contribute
>> to this total. There would be a chart like this for all activity types so
>> that you can see per day and total. This one is from the view of a
>> non-organizer. The participants' names are not shown but the logged in user
>> will be able to see his or her own place compared to others. If the viewer
>> is an organizer, then he or she can see all names.
>> *
>> Question: *Should the organizers be listed in the charts? I think they
>> should but just wanted to double check. You can also click a name and toggle
>> off whether or not the person's data is shown on the chart.
>>
>> *Question:* Should the public be able to see these? Their view would not
>> show participants' names, of course.
>>
>> *Roadmap:*
>>
>>    - Get feedback :) Update based on that.
>>
>>
>>    - Phase 1 would be:
>>
>>
>>    - Comments
>>    - Task edits
>>    - Task creations
>>    - Privacy policy update by John
>>
>>
>>    - Phase 2 would be:
>>
>>
>>    - Page views
>>    - Time on page
>>    - Last login
>>
>>
>> *Health Formula:*
>>
>> As I mentioned above, I have an idea about the "health" measurement. What
>> if we came up with a formula that helps us see at a glance how "healthy" a
>> course is?
>>
>> What about putting more weight on comments and logged in user page views
>> vs non logged in user page views. Task creation is important but a
>> workshop-style class could have 3 tasks vs a more traditional course could
>> have many. So this would probably be more focused on a course/study group vs
>> a dashboard of all courses.
>>
>> *Perhaps:*
>>
>> Health = Number of Task Comments * 20  + Number of Wall Comments * 10 +
>> Number of Tasks Created * 5 + Number of Page Views by Logged In Users * 5 +
>> Number of Page Views by Non Logged In Users + Task Edits + Amount of Time in
>> Seconds Logged In Users Spent on Page * 3 [This could be very faulty though]
>> + Amount of Time in Seconds Non Participants Spent on Page [Still faulty but
>> would be great if it did work :) ]
>>
>> Once we have badges in place, we could add in how many "complete a
>> task"/"get a badge."
>>
>> We could also somehow leave it up to the organizer what the "health
>> formula."
>>
>> *And then:
>>
>> *This was longer than I meant but hopefully this shows what I'm thinking
>> and makes sure I'm covering everything :) Please let me know if you have any
>> questions!
>>
>> I'm looking forward to the feedback and really looking forward to coding
>> this up! I think this will benefit many organizers.
>>
>> By the way, the chart maker is http://www.highcharts.com. It's free for
>> non-profits and I think it fits in with our license. If not, I'll need to
>> look for something else.
>>
>> Thank you!
>> Jessica
>>
>> --
>> Jessica Ledbetter
>> http://jessicaledbetter.com
>>
>>
>>
>> _______________________________________________
>> p2pu-dev mailing list
>> p2pu-dev at lists.p2pu.org
>> http://lists.p2pu.org/mailman/listinfo/p2pu-dev
>>
>>
>
> _______________________________________________
> p2pu-dev mailing list
> p2pu-dev at lists.p2pu.org
> http://lists.p2pu.org/mailman/listinfo/p2pu-dev
>
>

-- 
Jessica Ledbetter
http://jessicaledbetter.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.p2pu.org/pipermail/p2pu-dev/attachments/20110731/403b8157/attachment.html>