[p2pu-dev] PD/Course Assessment/Metrics Mockups

Mon Aug 1 17:28:16 UTC 2011

Jessica this is excellent work.

ASAIK the PD folks came to a rough consensus that this data should be public
by individual users remain anonymous. I agree with them and think it's a
safe middle route to follow, and if strong feedback occurs we can make
changes to the process (
http://www.p2pu.org/en/groups/pd-on-p2pu-orientation/).

It would be great to involve the research group in integrating some of
Stian's ideas into future iterations of metrics if the some sort of research
group revival occurs.

ALISON
p2pu.org/en/alison/

On Sun, Jul 31, 2011 at 9:34 AM, Jessica Ledbetter <
jessica at jessicaledbetter.com> wrote:

> Thank you for your feedback :) There's no need to apologize for giving me
> more to think about. That's a good thing!
>
> Regarding being able to focus on a certain task's activity: very good idea.
> Right now, a task is a page on a course so it could be limited by that much
> like how web analytics can be on a certain url pattern. I think there are
> going to be different types of pages in the future: tasks, non-tasks. If so,
> that can help. Example: "Show me all tasks" vs all pages that are under the
> course like the "full description." If not, it could be a drill-down model.
>
> And, really? In-line graphics didn't work? Aw :(
>
>
> On Sun, Jul 31, 2011 at 7:30 AM, Pippa Buchanan <Pippa.Buchanan at gmail.com>wrote:
>
>> This looks really great and is going to be so useful to help understand
>> what's going on in a group.
>>
>>
>> *Question: *Should the organizers be listed in the charts?
>> Organiser involvement: yes, you should be able to toggle (all) organisers
>> metrics off or include them.
>>
>>
>> *Question:* Should the public be able to see these? Their view would not
>> show participants' names, of course.
>>
>> Public viewing? I guess the question is how will this be useful to someone
>> outside of the group?
>> Maybe? Use case: I'm creating a course and want to find out how to create
>> amazing tasks. If I could look at charts course by course and identify what
>> the most popular / useful tasks are across different groups, I can use that
>> information to improve my own tasks.
>>
>> If I were a researcher I'd like to have access to this data, but I'd use
>> it in different ways.
>>
>>
>> At the moment the focus is on activity over time. Which is useful,
>> particularly when logging a course with a specific time period - it lets you
>> identify where attention begins to flag. You can also track whether certain
>> participants are always first to respond etc.
>>
>> It will also help organisers identify whether their continual involvement
>> is useful for the group ("oh, i see - this was the week I had a cold and
>> didn't have much energy to spare, and after that activity drops off, but it
>> increases after I contributed 40% more time".
>>
>> If we can track how much time a specific user is logged in we'll also be
>> able to use that to better identify how much time is needed to successfully
>> participate at both a general learner and organiser specific levels. (On
>> average learners spend x hours a week, and participants dedicate y hours per
>> month)
>>
>> As a participant in a group I'd also like to see my own involvement data
>> over time and to compare how I'm participating in multiple groups and to get
>> feedback on my own time management.
>>
>> *By Task v By Time*
>>
>> What I personally would find very interesting is data about the tasks -
>> I'm not sure if this is something that is planned.
>>
>> Which tasks have the most activity? What is the activity over time for a
>> specific task?
>>
>> Which tasks are *edited* the most, and could we see if after specific
>> edits the types and frequency of responses improve? (eg. I add a "How to
>> respond" section in my task description - suddenly its explained more
>> clearly, is easier to respond to and more comments are added).
>>
>> This will allow us to identify how useful a task is - are people answering
>> it because it's easier than the others, or is there something in the
>> structure and wording that attracts participants? Do people respond in
>> bursts to tasks? What order do they attempt the tasks in? If a task has a
>> deadline, what is the pattern of submission - who is responding first etc?
>> Who is responding multiple times in a task - are they being lovely and
>> sociable or are they in fact trolling the group?
>>
>>
>> Sorry if i've given you more to think about Jessica! I think this is
>> really interesting and positive step forward for the project.
>>
>> Pippa
>>
>>
>>
>>
>>
>> On 31 July 2011 04:03, Jessica Ledbetter <jessica at jessicaledbetter.com>wrote:
>>
>>> Greetings all (and especially Karen):
>>>
>>> Attached are some mockups of some course metrics' graphs/charts. I'd love
>>> to get feedback from everyone by Wednesday. If that's not enough time, let
>>> me know.  (Karen asked about getting a way to assess courses into P2PU and
>>> these are the first draft attempts at what it could look like.)
>>>
>>> We start by going to the course's admin page. Right now it's a button
>>> called "Edit Study Group" (or "Edit Course" or .. etc.) but I propose that
>>> that becomes "Administer Study Group" or "Study Group Admin" to better
>>> encompass the addition of metrics (and maybe some other stuff that's in my
>>> ideabox). Also, we'd have a link to them on the study group's sidebar in the
>>> box where Updates, Organizers, and Participant count are.
>>>
>>> Not obvious by this screenshot is that I also widened the box that this
>>> is in so that it's the same width as the P2PU header. More real estate for
>>> all this will be a very good thing :) In my experimentation, I wanted even
>>> more room!
>>>
>>> The first chart on the metric page is* "Overall Activity"*:
>>>
>>> [image: projectAdmin.png]
>>>
>>> *1:* Can download this graph as a graphic. This will come in handy when
>>> the graphs get too big for the real estate available or for inclusion in
>>> presentations/reports.
>>>
>>> *2: *When mousing over points on the chart, hint text pops up with more
>>> information. The line in this is an average of the 3 participants.
>>>
>>> *3: *Specifically for this chart, I was trying to show the "health" of
>>> the course. Missing are Task Edits, Task Creations, Page Views by
>>> Participants, Page Views by Non-Participants (this would have to be
>>> somewhere else). I can add those here too, of course. I'm not sure if Time
>>> on Page fits well with these other data points but I put it here since it
>>> was requested. Maybe another chart that is more web analytics like the page
>>> views and time could be together? I have a thought on "health" that might be
>>> better in its own email. However, I talk about it below after all the
>>> mockups ("Health Formula") :)
>>>
>>> *Overall:* This chart, and others, could become difficult to use with a
>>> lot of students in the course. But I can stretch it out so that you can
>>> scroll over to see it all or have a sort of zoom in feature so you can see
>>> the overall lines and zoom in on certain areas.
>>>
>>> Also, I can make it so that when you click a bar that you "drill down" to
>>> see more information like *when *the activity occurred for the person.
>>>
>>> *Next is "Activity Over Time"* which is a more specific metric for
>>> health reporting.
>>>
>>> [image: chartActivityOverTime.png]
>>>
>>> Activities listed right now are: comments, task edits, participants' last
>>> log in time, and how many tasks were added. We can add page views in phase
>>> 2. We could also have a running "total" showing as well. Requested was the
>>> ability to see by day, week, and total. Also, I can give this as a table.
>>>
>>> Next is* "Participant Comments."*
>>>
>>> [image: chartParticipantComments.png]
>>>
>>> Here we have a "total" for the day plus what each person did to
>>> contribute to this total. There would be a chart like this for all activity
>>> types so that you can see per day and total. This one is from the view of a
>>> non-organizer. The participants' names are not shown but the logged in user
>>> will be able to see his or her own place compared to others. If the viewer
>>> is an organizer, then he or she can see all names.
>>> *
>>> Question: *Should the organizers be listed in the charts? I think they
>>> should but just wanted to double check. You can also click a name and toggle
>>> off whether or not the person's data is shown on the chart.
>>>
>>> *Question:* Should the public be able to see these? Their view would not
>>> show participants' names, of course.
>>>
>>> *Roadmap:*
>>>
>>>    - Get feedback :) Update based on that.
>>>
>>>
>>>    - Phase 1 would be:
>>>
>>>
>>>    - Comments
>>>    - Task edits
>>>    - Task creations
>>>    - Privacy policy update by John
>>>
>>>
>>>    - Phase 2 would be:
>>>
>>>
>>>    - Page views
>>>    - Time on page
>>>    - Last login
>>>
>>>
>>> *Health Formula:*
>>>
>>> As I mentioned above, I have an idea about the "health" measurement. What
>>> if we came up with a formula that helps us see at a glance how "healthy" a
>>> course is?
>>>
>>> What about putting more weight on comments and logged in user page views
>>> vs non logged in user page views. Task creation is important but a
>>> workshop-style class could have 3 tasks vs a more traditional course could
>>> have many. So this would probably be more focused on a course/study group vs
>>> a dashboard of all courses.
>>>
>>> *Perhaps:*
>>>
>>> Health = Number of Task Comments * 20  + Number of Wall Comments * 10 +
>>> Number of Tasks Created * 5 + Number of Page Views by Logged In Users * 5 +
>>> Number of Page Views by Non Logged In Users + Task Edits + Amount of Time in
>>> Seconds Logged In Users Spent on Page * 3 [This could be very faulty though]
>>> + Amount of Time in Seconds Non Participants Spent on Page [Still faulty but
>>> would be great if it did work :) ]
>>>
>>> Once we have badges in place, we could add in how many "complete a
>>> task"/"get a badge."
>>>
>>> We could also somehow leave it up to the organizer what the "health
>>> formula."
>>>
>>> *And then:
>>>
>>> *This was longer than I meant but hopefully this shows what I'm thinking
>>> and makes sure I'm covering everything :) Please let me know if you have any
>>> questions!
>>>
>>> I'm looking forward to the feedback and really looking forward to coding
>>> this up! I think this will benefit many organizers.
>>>
>>> By the way, the chart maker is http://www.highcharts.com. It's free for
>>> non-profits and I think it fits in with our license. If not, I'll need to
>>> look for something else.
>>>
>>> Thank you!
>>> Jessica
>>>
>>> --
>>> Jessica Ledbetter
>>> http://jessicaledbetter.com
>>>
>>>
>>>
>>> _______________________________________________
>>> p2pu-dev mailing list
>>> p2pu-dev at lists.p2pu.org
>>> http://lists.p2pu.org/mailman/listinfo/p2pu-dev
>>>
>>>
>>
>> _______________________________________________
>> p2pu-dev mailing list
>> p2pu-dev at lists.p2pu.org
>> http://lists.p2pu.org/mailman/listinfo/p2pu-dev
>>
>>
>
>
> --
> Jessica Ledbetter
> http://jessicaledbetter.com
>
>
>
> _______________________________________________
> p2pu-dev mailing list
> p2pu-dev at lists.p2pu.org
> http://lists.p2pu.org/mailman/listinfo/p2pu-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.p2pu.org/pipermail/p2pu-dev/attachments/20110801/582a8c0e/attachment.html>