[p2pu-dev] PD/Course Assessment/Metrics Mockups

Sun Jul 31 11:30:18 UTC 2011

This looks really great and is going to be so useful to help understand
what's going on in a group.

*Question: *Should the organizers be listed in the charts?
Organiser involvement: yes, you should be able to toggle (all) organisers
metrics off or include them.

*Question:* Should the public be able to see these? Their view would not
show participants' names, of course.

Public viewing? I guess the question is how will this be useful to someone
outside of the group?
Maybe? Use case: I'm creating a course and want to find out how to create
amazing tasks. If I could look at charts course by course and identify what
the most popular / useful tasks are across different groups, I can use that
information to improve my own tasks.

If I were a researcher I'd like to have access to this data, but I'd use it
in different ways.

At the moment the focus is on activity over time. Which is useful,
particularly when logging a course with a specific time period - it lets you
identify where attention begins to flag. You can also track whether certain
participants are always first to respond etc.

It will also help organisers identify whether their continual involvement is
useful for the group ("oh, i see - this was the week I had a cold and didn't
have much energy to spare, and after that activity drops off, but it
increases after I contributed 40% more time".

If we can track how much time a specific user is logged in we'll also be
able to use that to better identify how much time is needed to successfully
participate at both a general learner and organiser specific levels. (On
average learners spend x hours a week, and participants dedicate y hours per
month)

As a participant in a group I'd also like to see my own involvement data
over time and to compare how I'm participating in multiple groups and to get
feedback on my own time management.

*By Task v By Time*

What I personally would find very interesting is data about the tasks - I'm
not sure if this is something that is planned.

Which tasks have the most activity? What is the activity over time for a
specific task?

Which tasks are *edited* the most, and could we see if after specific edits
the types and frequency of responses improve? (eg. I add a "How to respond"
section in my task description - suddenly its explained more clearly, is
easier to respond to and more comments are added).

This will allow us to identify how useful a task is - are people answering
it because it's easier than the others, or is there something in the
structure and wording that attracts participants? Do people respond in
bursts to tasks? What order do they attempt the tasks in? If a task has a
deadline, what is the pattern of submission - who is responding first etc?
Who is responding multiple times in a task - are they being lovely and
sociable or are they in fact trolling the group?

Sorry if i've given you more to think about Jessica! I think this is really
interesting and positive step forward for the project.

Pippa

On 31 July 2011 04:03, Jessica Ledbetter <jessica at jessicaledbetter.com>wrote:

> Greetings all (and especially Karen):
>
> Attached are some mockups of some course metrics' graphs/charts. I'd love
> to get feedback from everyone by Wednesday. If that's not enough time, let
> me know.  (Karen asked about getting a way to assess courses into P2PU and
> these are the first draft attempts at what it could look like.)
>
> We start by going to the course's admin page. Right now it's a button
> called "Edit Study Group" (or "Edit Course" or .. etc.) but I propose that
> that becomes "Administer Study Group" or "Study Group Admin" to better
> encompass the addition of metrics (and maybe some other stuff that's in my
> ideabox). Also, we'd have a link to them on the study group's sidebar in the
> box where Updates, Organizers, and Participant count are.
>
> Not obvious by this screenshot is that I also widened the box that this is
> in so that it's the same width as the P2PU header. More real estate for all
> this will be a very good thing :) In my experimentation, I wanted even more
> room!
>
> The first chart on the metric page is* "Overall Activity"*:
>
> [image: projectAdmin.png]
>
> *1:* Can download this graph as a graphic. This will come in handy when
> the graphs get too big for the real estate available or for inclusion in
> presentations/reports.
>
> *2: *When mousing over points on the chart, hint text pops up with more
> information. The line in this is an average of the 3 participants.
>
> *3: *Specifically for this chart, I was trying to show the "health" of the
> course. Missing are Task Edits, Task Creations, Page Views by Participants,
> Page Views by Non-Participants (this would have to be somewhere else). I can
> add those here too, of course. I'm not sure if Time on Page fits well with
> these other data points but I put it here since it was requested. Maybe
> another chart that is more web analytics like the page views and time could
> be together? I have a thought on "health" that might be better in its own
> email. However, I talk about it below after all the mockups ("Health
> Formula") :)
>
> *Overall:* This chart, and others, could become difficult to use with a
> lot of students in the course. But I can stretch it out so that you can
> scroll over to see it all or have a sort of zoom in feature so you can see
> the overall lines and zoom in on certain areas.
>
> Also, I can make it so that when you click a bar that you "drill down" to
> see more information like *when *the activity occurred for the person.
>
> *Next is "Activity Over Time"* which is a more specific metric for health
> reporting.
>
> [image: chartActivityOverTime.png]
>
> Activities listed right now are: comments, task edits, participants' last
> log in time, and how many tasks were added. We can add page views in phase
> 2. We could also have a running "total" showing as well. Requested was the
> ability to see by day, week, and total. Also, I can give this as a table.
>
> Next is* "Participant Comments."*
>
> [image: chartParticipantComments.png]
>
> Here we have a "total" for the day plus what each person did to contribute
> to this total. There would be a chart like this for all activity types so
> that you can see per day and total. This one is from the view of a
> non-organizer. The participants' names are not shown but the logged in user
> will be able to see his or her own place compared to others. If the viewer
> is an organizer, then he or she can see all names.
> *
> Question: *Should the organizers be listed in the charts? I think they
> should but just wanted to double check. You can also click a name and toggle
> off whether or not the person's data is shown on the chart.
>
> *Question:* Should the public be able to see these? Their view would not
> show participants' names, of course.
>
> *Roadmap:*
>
>    - Get feedback :) Update based on that.
>
>
>    - Phase 1 would be:
>
>
>    - Comments
>    - Task edits
>    - Task creations
>    - Privacy policy update by John
>
>
>    - Phase 2 would be:
>
>
>    - Page views
>    - Time on page
>    - Last login
>
>
> *Health Formula:*
>
> As I mentioned above, I have an idea about the "health" measurement. What
> if we came up with a formula that helps us see at a glance how "healthy" a
> course is?
>
> What about putting more weight on comments and logged in user page views vs
> non logged in user page views. Task creation is important but a
> workshop-style class could have 3 tasks vs a more traditional course could
> have many. So this would probably be more focused on a course/study group vs
> a dashboard of all courses.
>
> *Perhaps:*
>
> Health = Number of Task Comments * 20  + Number of Wall Comments * 10 +
> Number of Tasks Created * 5 + Number of Page Views by Logged In Users * 5 +
> Number of Page Views by Non Logged In Users + Task Edits + Amount of Time in
> Seconds Logged In Users Spent on Page * 3 [This could be very faulty though]
> + Amount of Time in Seconds Non Participants Spent on Page [Still faulty but
> would be great if it did work :) ]
>
> Once we have badges in place, we could add in how many "complete a
> task"/"get a badge."
>
> We could also somehow leave it up to the organizer what the "health
> formula."
>
> *And then:
>
> *This was longer than I meant but hopefully this shows what I'm thinking
> and makes sure I'm covering everything :) Please let me know if you have any
> questions!
>
> I'm looking forward to the feedback and really looking forward to coding
> this up! I think this will benefit many organizers.
>
> By the way, the chart maker is http://www.highcharts.com. It's free for
> non-profits and I think it fits in with our license. If not, I'll need to
> look for something else.
>
> Thank you!
> Jessica
>
> --
> Jessica Ledbetter
> http://jessicaledbetter.com
>
>
>
> _______________________________________________
> p2pu-dev mailing list
> p2pu-dev at lists.p2pu.org
> http://lists.p2pu.org/mailman/listinfo/p2pu-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.p2pu.org/pipermail/p2pu-dev/attachments/20110731/0c77adfa/attachment.html>