[p2pu-dev] robots.txt

[Dirk Uys]

Hi

Currently we don't have a robots.txt on p2pu.org. If you go to
http://p2pu.org/robots.txt you get redirected to
archive.p2pu.org/robots.txt which is the robots.txt for the old drupal
site!

I propose that we use django-robots to handle this for us?
http://pypi.python.org/pypi/django-robots

Does anyone have any experience/suggestions regarding this?

Also, I noticed that we use django-wellknown. Are we currently using this?

Another thing to consider is using the django sitemap contrib app?
http://docs.djangoproject.com/en/dev/ref/contrib/sitemaps/

Cheers
Dirk