[p2pu-dev] Two way communication for notifications and messaging

[Stian Håklev]

>From my limited understanding we'd have to handle the post separately from
the comment forms, and generate the tokens etc ourselves - I think the only
thing cloudmailin does is take the mail server out of the equation. Whether
that's a big gain or not, I don't know, since I've never tried to maintain
a mail server which receives thousands of incoming mails per day.

Stian

On Mon, Jun 4, 2012 at 4:58 PM, Jos Flores <josmasflores at gmail.com> wrote:

> Stian, I understand the workflow but my question is: where do
> cloudmailin come in in that workflow? as far as I know mail now is
> handled by sendgrid; does that have to change? who creates the token?
> cloudmailin? lernanta in the codebase?
> The example they give is that it is just like a POST in a form in your
> site: in lernanta to do a POST in a reply form you have to be
> authenticated, so it's not a straight forward, open POST request.
>
> cheers,
> José
>
> On 4 June 2012 21:17, Stian Håklev <shaklev at gmail.com> wrote:
> > Well, I guess every single message is sent out with a unique token, like
> >
> > reply-34093049304 at p2pu.org
> >
> > the combination of this token, and the sender e-mail address should be
> > enough authentication?
> >
> > Or do you mean how do we know that the HTTP posts from cloudmailin
> actually
> > come from cloudmailin?
> >
> > Apart from the possible vicissitudes of handling email ourselves (which I
> > have no experience with, but it always seems more complex than what you'd
> > think - although perhaps that's more for sending email than for
> receiving),
> > it really doesn't seem all that difficult to me.
> >
> > For each notification e-mail, generate a unique ID, and store that ID and
> > the e-mail addresses that received the notification in a database. When
> you
> > receive a reply, look up the ID, check that e-mail match, and look up
> where
> > to append the reply.
> >
> > Stian
> >
> >
> > On Mon, Jun 4, 2012 at 4:13 PM, Jos Flores <josmasflores at gmail.com>
> wrote:
> >>
> >> How does this handle authorisation though? I see they do have docs but
> >> I'm not following... Would it be related to the email account the user
> >> signed up on the site with?
> >> http://docs.cloudmailin.com/features/callbacks/#authorization-callbacks
> >>
> >> They even have a Django app all ready:
> >> https://github.com/CloudMailin/django-cloudmailin      ...tempting
> >> indeed! :)
> >>
> >> cheers,
> >> José
> >>
> >>
> >> On 4 June 2012 18:51, Jessy Kate Schingler <jessy at jessykate.com> wrote:
> >> > that's cool dirk i've not heard of cloudmailin!
> >> >
> >> > looking at their site it seems like a pretty neat service in general,
> >> > but
> >> > i;m not sure how much it would buy us in terms of reducing the work we
> >> > have
> >> > to do on the server side. doing it ourselves i guess we would write a
> >> > script
> >> > that parses incoming mails in the postfix maildir or exim equivalent
> >> > (which
> >> > cloudmailin would take care of). but we still have to disambiguate
> >> > responses
> >> > into specific accounts/discussion threads/models.
> >> >
> >> > anyway, my sense is that it wouldn't save us a ton of effort, and
> would
> >> > increase dependencies on external services, but curious what others
> >> > think.
> >> >
> >> > other thoughts?
> >> >
> >> >
> >> > On Mon, Jun 4, 2012 at 8:53 AM, Stian Håklev <shaklev at gmail.com>
> wrote:
> >> >>
> >> >> To get a sense of the traffic levels, how many messages are posted
> >> >> every
> >> >> month on P2PU right now? (Of course, we'd then have to estimate
> roughly
> >> >> what
> >> >> percentage of those messages are likely to be posted through email
> >> >> rather
> >> >> than online).
> >> >>
> >> >> Stian
> >> >>
> >> >>
> >> >> On Mon, Jun 4, 2012 at 11:22 AM, Dirk Uys <dirk at p2pu.org> wrote:
> >> >>>
> >> >>> Googling around a bit I found http://www.cloudmailin.com/
> >> >>>
> >> >>> They turn your email into an http post to your website. I like the
> >> >>> idea, it keeps the email stuff separate from the messaging logic.
> >> >>>
> >> >>> They also have special pricing options for non-profits.
> >> >>>
> >> >>> The alternative would be to setup our own mail server and use some
> >> >>> python bindings for mail and a celery task. There's poplib and
> imaplib
> >> >>> for python.
> >> >>>
> >> >>> Any thoughts
> >> >>>
> >> >>> Cheers
> >> >>> d
> >> >>>
> >> >>> On Sat, Jun 2, 2012 at 1:02 AM, Stian Håklev <shaklev at gmail.com>
> >> >>> wrote:
> >> >>> > A lot of emails disregard anything after a + in an email user
> name,
> >> >>> > like
> >> >>> > GMail which will send all mail to shaklev+test at gmail.com to
> >> >>> > shaklev at gmail.com. So if the server receives all mail to
> reply+dfsfd
> >> >>> > to
> >> >>> > the
> >> >>> > reply address, and the reply address is able to strip this out of
> >> >>> > the
> >> >>> > header, and then know where to put it, based on the unique ID, we
> >> >>> > should be
> >> >>> > good to go.
> >> >>> >
> >> >>> > I am sure there are services and APIs out there for doing this as
> >> >>> > well,
> >> >>> > but
> >> >>> > they are probably expensive for the kind of volume (although we'd
> >> >>> > have
> >> >>> > a lot
> >> >>> > less replies by email than we have email notifications).
> >> >>> >
> >> >>> > For such a common problem, you'd think there'd be a Django module
> or
> >> >>> > at
> >> >>> > least a Python implementation out there somewhere as well. But I
> >> >>> > don't
> >> >>> > know
> >> >>> > how much of the challenge is in the Lernanta codebase, and how
> much
> >> >>> > is
> >> >>> > the
> >> >>> > email server etc.
> >> >>> >
> >> >>> > Stian
> >> >>> >
> >> >>> >
> >> >>> > On Fri, Jun 1, 2012 at 1:44 PM, Jessy Kate Schingler
> >> >>> > <jessy at jessykate.com>
> >> >>> > wrote:
> >> >>> >>
> >> >>> >> really cool dirk.
> >> >>> >>
> >> >>> >> i dont have a particular mail server recommendation though have
> >> >>> >> personally
> >> >>> >> done more with postfix than exim.
> >> >>> >>
> >> >>> >> i just did a quick search for "comments reply by email" and
> noticed
> >> >>> >> that
> >> >>> >> wordpress implemented this last year. obviously they're php but
> it
> >> >>> >> might be
> >> >>> >> interested to take a look at the WP codebase to see how they
> >> >>> >> implemented it.
> >> >>> >>
> >> >>> >> looking at facebook notifications, they use emails structured
> like
> >> >>> >> so:
> >> >>> >> update+ohlp1evf at facebookmail.com
> >> >>> >>
> >> >>> >> so i guess update@ is their updates email address and the hash
> >> >>> >> let's
> >> >>> >> them
> >> >>> >> disambiguate to a specific conversation thread via some kind of
> >> >>> >> script
> >> >>> >> internally. i think this would be a fun/interesting thing to
> >> >>> >> contribute to
> >> >>> >> if it gets going, but know my time will be somewhat limited this
> >> >>> >> summer with
> >> >>> >> internship work.
> >> >>> >>
> >> >>> >> jessy
> >> >>> >>
> >> >>> >>
> >> >>> >> On Fri, Jun 1, 2012 at 6:26 AM, Dirk Uys <dirk at p2pu.org> wrote:
> >> >>> >>>
> >> >>> >>> Thanks, having a look at ack atm!
> >> >>> >>>
> >> >>> >>> On Fri, Jun 1, 2012 at 3:07 PM, Jos Flores
> >> >>> >>> <josmasflores at gmail.com>
> >> >>> >>> wrote:
> >> >>> >>> > Dirk, thanks for the walkthrough, it does make sense.
> >> >>> >>> >
> >> >>> >>> > I've noticed you recommend grep... have you tried ack? it's
> >> >>> >>> > really
> >> >>> >>> > cool!
> >> >>> >>> > I use grep very little since I discovered it :)
> >> >>> >>> >
> >> >>> >>> > cheers,
> >> >>> >>> > José
> >> >>> >>> >
> >> >>> >>> > On 1 June 2012 09:57, Dirk Uys <dirk at p2pu.org> wrote:
> >> >>> >>> >> No problem. At the moment the notification code isn't in one
> >> >>> >>> >> place
> >> >>> >>> >> (something we can work on).
> >> >>> >>> >>
> >> >>> >>> >> I'm going to start explaining from the point where the email
> is
> >> >>> >>> >> sent
> >> >>> >>> >> to the user and work my way backwards to where notifications
> >> >>> >>> >> are
> >> >>> >>> >> generated. Please tell me if this ends up being confusing
> >> >>> >>> >>
> >> >>> >>> >> Emails are sent to a user by calling
> >> >>> >>> >> django.contrib.auth.models.User.email_user
> >> >>> >>> >>
> >> >>> >>> >>
> >> >>> >>> >>
> >> >>> >>> >> (
> https://docs.djangoproject.com/en/dev/topics/auth/#django.contrib.auth.models.User.email_user
> )
> >> >>> >>> >>
> >> >>> >>> >> email_user() is called by an async celery task
> SendNotification
> >> >>> >>> >>
> >> >>> >>> >>
> >> >>> >>> >>
> >> >>> >>> >> (
> https://github.com/p2pu/lernanta/blob/master/lernanta/apps/users/tasks.py#L25
> )
> >> >>> >>> >>
> >> >>> >>> >> SendNotification tasks can be created in multiple places:
> >> >>> >>> >>
> >> >>> >>> >> There is a signal
> >> >>> >>> >>
> >> >>> >>> >>
> >> >>> >>> >>
> >> >>> >>> >> (
> https://github.com/p2pu/lernanta/blob/master/lernanta/apps/content/models.py#L245
> )
> >> >>> >>> >> that fires when tasks are created or modified. This signal
> then
> >> >>> >>> >> calls
> >> >>> >>> >> a function called send_email_notification
> >> >>> >>> >>
> >> >>> >>> >>
> >> >>> >>> >>
> >> >>> >>> >> (
> https://github.com/p2pu/lernanta/blob/master/lernanta/apps/content/models.py#L213
> )
> >> >>> >>> >> that creates SendNotifications tasks.
> >> >>> >>> >>
> >> >>> >>> >> Another example is sent_creation_notification
> >> >>> >>> >>
> >> >>> >>> >>
> >> >>> >>> >>
> >> >>> >>> >> (
> https://github.com/p2pu/lernanta/blob/master/lernanta/apps/projects/models.py#L270
> )
> >> >>> >>> >> thats part of projects.models.Project. This function is
> called
> >> >>> >>> >> from
> >> >>> >>> >> the create function
> >> >>> >>> >>
> >> >>> >>> >>
> >> >>> >>> >>
> >> >>> >>> >> (
> https://github.com/p2pu/lernanta/blob/master/lernanta/apps/projects/models.py#L270/
> )
> >> >>> >>> >> that in turn gets called in the create view
> >> >>> >>> >>
> >> >>> >>> >>
> >> >>> >>> >>
> >> >>> >>> >> (
> https://github.com/p2pu/lernanta/blob/master/lernanta/apps/projects/views.py#L189
> )
> >> >>> >>> >>
> >> >>> >>> >> I'm not sure at what other places SendNotification tasks are
> >> >>> >>> >> created,
> >> >>> >>> >> but you can grep to code base to find out. grep -r
> >> >>> >>> >> SendNotification *
> >> >>> >>> >>
> >> >>> >>> >> The fact that the places where SendNotification tasks are
> >> >>> >>> >> created
> >> >>> >>> >> are
> >> >>> >>> >> spread all over isn't a problem in itself, but at the moment
> >> >>> >>> >> checks
> >> >>> >>> >> for notification preference also seem spread out a bit. For
> >> >>> >>> >> example
> >> >>> >>> >> task updates
> >> >>> >>> >>
> >> >>> >>> >>
> >> >>> >>> >> (
> https://github.com/p2pu/lernanta/blob/master/lernanta/apps/content/models.py#L234
> ).
> >> >>> >>> >> It also makes it difficult to implement batching of
> >> >>> >>> >> notifications.
> >> >>> >>> >>
> >> >>> >>> >> Than being said, I think that notification may be one of the
> >> >>> >>> >> first
> >> >>> >>> >> parts of Lernanta that we can put into a separate module
> >> >>> >>> >> without
> >> >>> >>> >> too
> >> >>> >>> >> much surgery on the whole code base.
> >> >>> >>> >>
> >> >>> >>> >> Cheers
> >> >>> >>> >> d
> >> >>> >>> >>
> >> >>> >>> >> On Fri, Jun 1, 2012 at 10:28 AM, Jos Flores
> >> >>> >>> >> <josmasflores at gmail.com>
> >> >>> >>> >> wrote:
> >> >>> >>> >>> Hey Dirk,
> >> >>> >>> >>>
> >> >>> >>> >>> I know nothing about the topic but curious about where the
> >> >>> >>> >>> notifications code is and how it works... could you point me
> >> >>> >>> >>> towards
> >> >>> >>> >>> it?
> >> >>> >>> >>>
> >> >>> >>> >>> cheers,
> >> >>> >>> >>> José
> >> >>> >>> >>>
> >> >>> >>> >>>
> >> >>> >>> >>> On 1 June 2012 09:01, Dirk Uys <dirk at p2pu.org> wrote:
> >> >>> >>> >>>> Hi
> >> >>> >>> >>>>
> >> >>> >>> >>>> I've been thinking a little bit about two way interaction
> for
> >> >>> >>> >>>> messaging and notifications. On the software side of
> things I
> >> >>> >>> >>>> don't
> >> >>> >>> >>>> foresee too much trouble, but on the config side of things
> >> >>> >>> >>>> I'm a
> >> >>> >>> >>>> little uncertain.
> >> >>> >>> >>>>
> >> >>> >>> >>>> Our primary MX record points to google, then @
> lists.p2pu.org
> >> >>> >>> >>>> points
> >> >>> >>> >>>> to
> >> >>> >>> >>>> the service we use for the mailing list. I guess we could
> use
> >> >>> >>> >>>> something like @site.p2pu.org? I see that github uses
> >> >>> >>> >>>> @reply.github.com with something that looks like a one
> time
> >> >>> >>> >>>> token.
> >> >>> >>> >>>>
> >> >>> >>> >>>> Is there any specific mail server that anyone would
> suggest?
> >> >>> >>> >>>> I
> >> >>> >>> >>>> have
> >> >>> >>> >>>> always skipped the talk about mail servers when reading
> linux
> >> >>> >>> >>>> sysadmin
> >> >>> >>> >>>> docs :)
> >> >>> >>> >>>>
> >> >>> >>> >>>> Cheers
> >> >>> >>> >>>> d
> >> >>> >>> >>>>
> >> >>> >>> >>>> ps. I apologize for the thinking as I type email
> >> >>> >>> >>>> _______________________________________________
> >> >>> >>> >>>> p2pu-dev mailing list
> >> >>> >>> >>>> p2pu-dev at lists.p2pu.org
> >> >>> >>> >>>> http://lists.p2pu.org/mailman/listinfo/p2pu-dev
> >> >>> >>> >>> _______________________________________________
> >> >>> >>> >>> p2pu-dev mailing list
> >> >>> >>> >>> p2pu-dev at lists.p2pu.org
> >> >>> >>> >>> http://lists.p2pu.org/mailman/listinfo/p2pu-dev
> >> >>> >>> >> _______________________________________________
> >> >>> >>> >> p2pu-dev mailing list
> >> >>> >>> >> p2pu-dev at lists.p2pu.org
> >> >>> >>> >> http://lists.p2pu.org/mailman/listinfo/p2pu-dev
> >> >>> >>> > _______________________________________________
> >> >>> >>> > p2pu-dev mailing list
> >> >>> >>> > p2pu-dev at lists.p2pu.org
> >> >>> >>> > http://lists.p2pu.org/mailman/listinfo/p2pu-dev
> >> >>> >>> _______________________________________________
> >> >>> >>> p2pu-dev mailing list
> >> >>> >>> p2pu-dev at lists.p2pu.org
> >> >>> >>> http://lists.p2pu.org/mailman/listinfo/p2pu-dev
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >> --
> >> >>> >> Jessy
> >> >>> >> http://jessykate.com
> >> >>> >>
> >> >>> >>
> >> >>> >> _______________________________________________
> >> >>> >> p2pu-dev mailing list
> >> >>> >> p2pu-dev at lists.p2pu.org
> >> >>> >> http://lists.p2pu.org/mailman/listinfo/p2pu-dev
> >> >>> >>
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> > --
> >> >>> > http://reganmian.net/blog -- Random Stuff that Matters
> >> >>> >
> >> >>> >
> >> >>> > _______________________________________________
> >> >>> > p2pu-dev mailing list
> >> >>> > p2pu-dev at lists.p2pu.org
> >> >>> > http://lists.p2pu.org/mailman/listinfo/p2pu-dev
> >> >>> >
> >> >>> _______________________________________________
> >> >>> p2pu-dev mailing list
> >> >>> p2pu-dev at lists.p2pu.org
> >> >>> http://lists.p2pu.org/mailman/listinfo/p2pu-dev
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> http://reganmian.net/blog -- Random Stuff that Matters
> >> >>
> >> >>
> >> >> _______________________________________________
> >> >> p2pu-dev mailing list
> >> >> p2pu-dev at lists.p2pu.org
> >> >> http://lists.p2pu.org/mailman/listinfo/p2pu-dev
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Jessy
> >> > http://jessykate.com
> >> >
> >> >
> >> > _______________________________________________
> >> > p2pu-dev mailing list
> >> > p2pu-dev at lists.p2pu.org
> >> > http://lists.p2pu.org/mailman/listinfo/p2pu-dev
> >> >
> >> _______________________________________________
> >> p2pu-dev mailing list
> >> p2pu-dev at lists.p2pu.org
> >> http://lists.p2pu.org/mailman/listinfo/p2pu-dev
> >
> >
> >
> >
> > --
> > http://reganmian.net/blog -- Random Stuff that Matters
> >
> >
> > _______________________________________________
> > p2pu-dev mailing list
> > p2pu-dev at lists.p2pu.org
> > http://lists.p2pu.org/mailman/listinfo/p2pu-dev
> >
> _______________________________________________
> p2pu-dev mailing list
> p2pu-dev at lists.p2pu.org
> http://lists.p2pu.org/mailman/listinfo/p2pu-dev
>



-- 
http://reganmian.net/blog -- Random Stuff that Matters
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.p2pu.org/pipermail/p2pu-dev/attachments/20120604/625544f4/attachment-0001.html>