Is Cleaning Up Orphaned Uploaded Files Necessary?
luxlogica
New
With the 'Advanced Editor' it's quite easy for users to upload images and files to their hearts' content. But I've found that even after a short time after setting the forum up, my handful of users have already uploaded several files that are not being used anywhere. It seems they sometimes upload the wrong image/file, and once they see it in their post, they realise they don't need it, and delete it. But deleting it from the post doesn't actually delete the file from the server - it's still there, in the 'editor' sub-directory...
As I'm new to Vanilla, I'm not sure whether this is something I should be concerned about. Do I need to do anything to clean up these old uploaded, orphaned files, that are not being used anywhere? Is there a task that Vanilla automatically runs periodically in the background that already does this? Do I need to install a plugin?
Any guidance would be appreciated!
Comments
Not of the same nature but also important to this matter: The current Advanced Editor as well as the FileUpload do not resize the uploded file or compress them again. Therefore i run a script to resize and recompress files on the server continously.
That helps regarding lage speed and user site speed a lot.
A CleanUp Method against the database would be great.
For what it's worth, neither Vanilla nor I personally make any effort to clean them up currently.
You mean, not even manually? Doesn't that sooner or later become a problem? - or am I overreacting?
Nope. I mean, if you're on a 1 GB GoDaddy plan, sure, it's possibly a problem (at that point isn't just running a forum a problem?). But on most mid-grade or higher hosting plans, disk space is about the last thing to worry about these days. We use a CDN and frankly just don't care if 1% of the files are junk.
It's good to know that it's not a problem, practically. There are, however, a number of reasons why this would be considered bad practice - companies usually don't want unused files of unknown content hanging around their servers. It would still be very useful if there were some kind of 'cleanup orphaned files' task/filter/plugin that could be applied...
The Advanced Editor renames files - seemingly based on their checksum - and places them in hexadecimal-titled directories, like git repositories do. Therefore, trying to do this cleanup by hand is a real chore: it's impossible to know where or what a file is, unless you actually open each directory, and each file, one by one. An automated process is really, really needed.
@luxlogica: I agree with you. Uploads can specifically be a problem as users often (with even jumping the board rules) upload copyrighted material. Having these things cleaned up as well if not in use anymore is also a plus to every forum owner.
@phreak indeed, in the past I've had clients with disgruntled ex-employees who maliciously posted very damaging content to the company's forum -ie, confidential files and papers, photos, etc. Another example that comes to mind, from a few years ago, was a gaming site, where many of the users were 'amateur hackers', who kept trying to upload viruses and little malware scripts (often disguised as pdf files or images).
In both sites it was important for us to be able to view the uploaded files, and immediately delete them, and it would have been disastrous to just leave them hanging around.
The site I've just setup with Vanilla is a low-risk one, with very few users, in a wonderfully friendly and cooperative environment - so I don't foresee any problems there. But if there is no way to cleanup old, orphaned uploads, then this will be a deal-breaker... :-(
I'm extremely surprised.
@luxlogica
If it matters so much, you could do it via mySQL or via an ftp program like Filezilla, although I don't know how you would be able to tell if a file was 'orphaned' or no longer used/needed.
This would be a nice core feature to have. Something like phpBB, vBulletin and IPB have via their APC (I use all three and Burning Board as well).
@subdreamer
Well, since Linc has said Vanilla don't bother about it, it would probably need to be a plugin commissioned for the open source version, rather than a core feature.
How many people want it, and how much would they pay to make it happen?
(I'm not touting, as I wouldn't have the first clue how to do it!)
@whu606 said:
I don't really need it to be honest. I'm just saying it would be a nice feature to have (like the aforementioned forums). I plan to use Vanilla in a private close-knit close knit community so rogue attachments shouldn't be an issue. If the developers don't want/need it then, yes, you (as in we) could commission someone to make it. I've donate to things that I don't even use/need just support developers who selflessly create add-ons for communities such as this one (you all know who you are!) and wouldn't mind giving a small donation to motivate such awesome individual. I need coffee.
No moreso than how surprised I am this is the feature (or lack thereof) being fixated on today. No one cares for 7 years, then suddenly it's why someone says they can never use your software because OF COURSE it's the most important thing. C'est la vie.
@Linc Sorry for being "surprised" that in 7 years, as you put it, no one - including all the developers involved with the project - noticed this seemingly glaring security hole. If for you this is just a minor 'feature' that is not worth any attention, then indeed, c'est la vie.
My surprise remains: this seems to be a pretty major, gaping hole in the software's security. I believed that Vanilla had been widely tested and used by major enterprise, in a variety of settings, hence my surprise that this had never been picked up before - specially considering that I picked it up so quickly, being a total newbie.
And that was the reason why I posted this question in the first place. I thought I had certainly missed something obvious, or had misunderstood something about the way the software worked. Apparently not. It's a shame.
I will keep on checking, to see whether this ever gets updated, or improved, but until then, I'm afraid that this truly is a deal-breaker for all but a very small percentage of my clients. I can do nothing but envy you, who can instal forum software in such trustworthy - and trusting - communities...
@luxlogica
You seem to be moving the goalposts.
Your original question was
Is Cleaning Up Orphaned Uploaded Files Necessary?
Now you have changed it to
Orphaned files are a significant security risk, why aren't you doing something about it?
I can't see how a file is more or less of a risk if they are no longer being used.
If you have forums where you think people will upload maliciously, then that is a separate issue, and nothing to do with the OP.
You don't have to allow uploading files, and can limit the type of files permitted for upload.
Vanilla gives you locks and bolts to protect your forum; if you want to open the door and invite people in, then expect them to provide a security guard to deal with your decisions, I think you are being unreasonable.
If it is so vital to you, and you have a large number of clients, why not commission a plugin and donate it to the Open Source community?
An "orphaned" file is a file that is not used anywhere. Since a one time uploaded file could be referenced in more than one place, you would have to search at least
Discussion
Comment
ActivityComment
Message
Category
ConversationMessage
for occurances of all entries in table Media, column Path. But a) I'm not sure that the list above is complete, b) there might be some more tables, added by plugins c) even some Vanilla files are referenced in the config and not in any db table, so that you should better not delete them.
I wouldn't write a plugin like that, because you can never be sure that you wouldn't delete a still needed file.
When it comes to security: you wouldn't say that "orphaned" pictures are a security risk, would you? You can restrict which file types can be uploaded. And you can (and should!) make files in the upload folder non-executable. So if there is still a security risk, I really and honestly would be interested what that can be.
Just one last thought. If you on't think you can live without cleaning up those files, I think you could do it "manually". You would need a dump of your database and should concat that with the config.php. Afterwards get all filenames with a select "Path" from GDN_Media. It should be safe to delete all files with filenames that cannot be found in the dump.
That actually something I do on some hosts occasionally to save some space.
grepyourdomain.com/uploads/.* from the dumpsortboth and thendiffAnd you got a list of files to delete. All this can be done from the shell.
My themes: pure | minusbaseline - My plugins: CSSedit | HTMLedit | InfiniteScroll | BirthdayModule | [all] - PM me about customizations
VanillaSkins.com - Plugins, Themes and Graphics for Vanillaforums OS
We noticed the fact of it long ago, and it's not a security hole.
Hi all, just for my understanding I have a question.
What I found out myself:
When I add an image to a post and later remove that image by editing the post, the image itself wont be deleted.
It is stored in the folder "./forum/uploads/editor" and also a reference in table "GDN_Media".
My question:
There still is no possibility to delete those orphaned images, right?