SOLVED Django Help.... Custom storage?

Stratus_ss · Nov 19, 2014

I am hoping someone can help me out with some Django stuff. By default there are a few things in Django's stdlib Storage that are causing me grief. The main one is that it is stripping anything that is not alpha-numeric. Specifically there is a text.py which has a get_valid_filename(s) method which I believe is the one doing this function... it reads as follows:

Code:

>>> def get_valid_filename(s):
...     """
...     Returns the given string converted to a string that can be used for a clean
...     filename. Specifically, leading and trailing spaces are removed; other
...     spaces are converted to underscores; and anything that is not a unicode
...     alphanumeric, dash, underscore, or dot, is removed.
...     >>> get_valid_filename("john's portrait in 2004.jpg")
...     'johns_portrait_in_2004.jpg'
...     """
...     s = force_text(s).strip().replace(' ', '_')
...     return re.sub(r'(?u)[^-\w.]', '', s)

When I went to the #django IRC, they told me the best way to handle this is to write a custom storage see django docs. However I am not sure how to implement this. I have subclassed/overridden my own classes/methods but I dont really know how to do this with stdlib.

Do I have to reimplement everything (i.e. de-construct what calls are made one and redirect them to my override)?

Can anyone help me out?

EDIT: I have confirmed if I actually change this

Code:

return re.sub(r'(?u)[^-\w.]', '', s)

To this

Code:

return(s)

I get the expected result, but I know that editing the stdlib isnt the portable way of doing things

pcgamer4life · Nov 24, 2014

I'm not familiar with the language at all, but shouldn't you just be able to create a custom storage class that will inherit the methods defined in stdlib, and then override the get_valid_filename method?

Stratus_ss · Nov 25, 2014

pcgamer4life said:
I'm not familiar with the language at all, but shouldn't you just be able to create a custom storage class that will inherit the methods defined in stdlib, and then override the get_valid_filename method?

The problem I am struggling with is conceptually how do I get the program to use the custom storage without intimately knowing what calls are being made to redirect them. Since this is a stdlib, there is a lot of "magic" happening that I dont necessarily know what its doing precisely. I guess what I am struggling with is I don't know HOW to make sure its using my storage method and not the stdlib one

pcgamer4life · Nov 25, 2014

I found this snippet on the web, it should work similar to this, I would suspect. It should be the same as inheritance and overriding methods in any other language, just syntactically different. Import whatever library the base storage class is in, and then define a new class that inherits from whatever storage class you're used to working with. Your new class will contain all the methods of the parent storage class by default. Then, it's just a matter of overriding the get_valid_filename() method and replacing with your little bit of code above. Finally, modify your personal code to instantiate your new storage class rather than the one out of the box. Now whenever you invoke that method of your storage class, it'll execute your bit of code rather than the one defined in stdlib.

Does that help, or do you already know this and I'm missing the point of what you're trying to do?

Code:

import os
from django.db import models
 
 
class ModelClass (models.Model):
    name = models.CharField(max_length=100)
    email = models.EmailField()
    hash = models.CharField(primary_key=True, max_length=32)
 
    def create_hash(self):
        return os.urandom(32).encode('hex')
 
    #Overriding
    def save(self, *args, **kwargs):
        #check if the row with this hash already exists.
        if not self.pk:
            self.hash = self.create_hash() 
        self.my_stuff = 'something I want to save in that field'
        super(ModelClass, self).save(*args, **kwargs)

pcarini · Nov 25, 2014

Every place I've seen in the Django API that uses Storage allows you to inject your own custom Storage class rather than needing to replace the stdlib Storage. In what context are you using it?

pcarini · Nov 25, 2014

This (untested) idea should work.

In a module named custom_storage:

Code:

from django.core.files.storage import FileSystemStorage


class CustomStorage(FileSystemStorage):
    def get_valid_filename(s):
        return s


my_storage = CustomStorage()

In your models file:

Code:

from django.db import models
from custom_storage import my_storage


class MyModel(models.Model):
    foo = models.FileField(storage=my_storage)

Stratus_ss · Nov 30, 2014

Thanks... Sorry I have been wrapped up in work the last several days I will definitely give this a shot. I appreciate the eyes

I think i wasn't understanding what I was reading. I understand about inheritance, what I didnt understand was how to replace storage with your own. That brief example (which I must have missed) makes it a lot more clear

Stratus_ss · Nov 30, 2014

That didn't seem to work. As a test, this is what I did in my models.py

Code:

from django.db import models
from django.core.files.storage import FileSystemStorage
import os

class CustomStorage(FileSystemStorage):
    def get_valid_filename(s):
        print(s)
        return s


my_storage = CustomStorage()


#This file contains all of the database work. It tracks which tables are created, what the field types are etc.
#Because we require a different upload location from each project/environment, a subclass overriding those specific
#traits is required


class GenericMixin(models.Model):
    """This mixin holds the fields that are required by all models.
    The only difference between the models (aside from being stored 
    in different tables), is the upload location"""

    slug = models.SlugField(max_length=50, blank=True)
    project = models.CharField(max_length=50, blank=True)
    user_name =  models.CharField(max_length=50, blank=True)
    date =  models.DateTimeField(auto_now=True)
    version =  models.CharField(max_length=50, blank=True)
    #This is to attempt to keep an audit trail using "logical" deletes instead of removing the record
    visible = models.CharField(max_length=3, blank=True)
    currently_deployed = models.CharField(max_length=3, blank=True)
    file_name = models.CharField(max_length=100, blank=True)
    def __init__(self, *args, **kwargs):
        self.project = None
        self.user_name = None
        self.version = None
        self.visible = None
        self.currently_deployed = None
        self.file_name = None
        super(GenericMixin, self).__init__(*args, **kwargs)

    class Meta:
        """This is what allows the model to be subclassed while retaining all fields specified above"""
        abstract = True

    def __unicode__(self):
        """This is what is returned for the audit section"""
        return u'%s | %s | %s | %s | %s' % (self.file_name, self.project, self.user_name, self.date, self.version)

    def save(self, *args, **kwargs):
        self.slug = self.file.name
        super(GenericMixin, self).save(*args, **kwargs)

    def delete(self, *args, **kwargs):
        """delete -- Remove to leave file."""
        self.file.delete(False)
        super(GenericMixin, self).delete(*args, **kwargs)  



class LN_CHRYSLER_DEV_Upload(GenericMixin):

    upload_location = '/media/LN_Chrysler_DEV'
    file = models.FileField(storage=my_storage, upload_to=upload_location)

Files are still having the hashes stripped

Stratus_ss · Nov 30, 2014

Here is what is stored in the database

Code:

| id | slug              | project      | user_name     | date                | version | visible | currently_deployed | file_name         | file                                    |
+----+-------------------+--------------+---------------+---------------------+---------+---------+--------------------+-------------------+-----------------------------------------+
|  1 | hostc#bmo.war     | chrysler_dev | AnonymousUser | 2014-11-30 15:42:59 |         | yes     |                    | hostc#bmo.war     | /media/LN_Chrysler_DEV/hostc#bmo.war    |
|  2 | hostc#sni.war     | chrysler_dev | AnonymousUser | 2014-11-30 15:43:34 |         | yes     |                    | hostc#sni.war     | /media/LN_Chrysler_DEV/hostcsni.war     |
|  3 | hostc#help-me.war | chrysler_dev | AnonymousUser | 2014-11-30 15:45:54 |         | yes     |                    | hostc#help-me.war | /media/LN_Chrysler_DEV/hostchelp-me.war |

Line 1 is with the stdlib changed. Line 2 is with stdlib unmodified. Line 3 is with the custom storage implemented, stdlib unmodified

Stratus_ss · Dec 17, 2014

Just thought I would follow up on this. It turns out that overriding get_valid_filename is to far along the stack. By the time this is called, there is already a previous method of get_valid_name which calls get_valid_filename. Therefore in order to solve my problem I do something like the following psuedo code

Code:

from django.core.files.storage import Storage

def my_filename_handler(self, name)
    old_funct(self, name)
    return(name)

old_funct = Storage.get_valid_name
Storage.get_valid_name = my_filename_handler

pcgamer4life · Dec 17, 2014

Glad to see you got it all worked out!

SOLVED Django Help.... Custom storage?

Stratus_ss

Overclockix Snake Charming Senior, Alt OS Content

pcgamer4life

Member

Stratus_ss

Overclockix Snake Charming Senior, Alt OS Content

pcgamer4life

Member

pcarini

Member

pcarini

Member

Stratus_ss

Overclockix Snake Charming Senior, Alt OS Content

Stratus_ss

Overclockix Snake Charming Senior, Alt OS Content

Stratus_ss

Overclockix Snake Charming Senior, Alt OS Content

Stratus_ss

Overclockix Snake Charming Senior, Alt OS Content

pcgamer4life

Member

Similar threads