Oh, and I love TTH. Best thing since sliced bread.
Also, thought up some other functions that would be nice. I will post those later and seperately, in the interest of brevity.
Wow, all that writing and I still haven't even started my actual post. How do I turn off verbose output on myself?
I have been trying to clean up my queue recently. Over time my queue has grown rather large, which I am certain is symptom unique to me among DC++ users (cough cough). My first thought was to search through the queue and remove duplicate or otherwise unwanted files. That's when I first realized that I was going to have some trouble with my self-appointed task.
The short of it is this: I would humbly request file management features. The ideal solution would be the ability to map the queue as a virtual drive and then manage it with standard file managers (think WebDAV/Davenport or mounting a virtual device in Linux). But ideals are not often practical solutions, so I will describe some informal use case examples and sketch out some (simple?/easy?) feature requests to address them.
1. Searching the Queue
My first idea was to search the queue to find files to remove. I open up the Download Queue tab and ... no search function. Small problem, that.
Since there was no search available I went to my handy dandy text editor and found I could search the queue.xml easily enough. Moving between the editor and DC++ I could make some simple modifications but found it very time consuming. So I decided to modify the queue.xml file directly but found additional issues there.
XML ignores blank spaces in data files and therefore writes each record across multiple lines to make data structures formatted for easy reading by human eyes. Or something. While it is a great feature of XML it made doing a search and replace/delete of whole XML records difficult to accomplish in the text editor without creating a malformed XML file. Not insurmountable but worth mentioning as a difficulty. The real problem was with what happened post editing.
The queue.xml file appears to be read by DC++ only at startup and is subsequently written to as the queue is modified. Any changes I make to the queue.xml while DC++ is running would be overwritten by the blind writes. Making changes to queue.xml would have to occur after I had exited from DC++ and all final commits had been written by it to the file. Once that happened I could modify the file and start DC++ to load changes. (I won't go into using a copy of queue.xml while DC++ is running as the problems inherent in that are obvious.) Stopping and starting DC++ whenever I wanted to edit the Download Queue seemed a bit inelegant a solution so, except for major revisions, I think I will avoid that as a fix.
Any way I go about this searching seems a bit of a kludge.
So the requests here are:
A: A search function under the Download Queue tab. The search should have the same elements as are found in the dialog box for adding ADL Searches.
B: A filter function similar to ADL Search for the Download Queue, if it could retain the directory structure.
2. Looking for Duplicates within the Download Queue tab
Since searching and modifying the queue externally is impractical I turned my attention to hunting via the browsable interface. The easiest thing to do was go into directories containing many files and sort according to a variety of attributes (name, size, TTH) to find exact or probable duplicate files. This helped and was an easy quick fix, but the limitations were apparent.
As I could only filter within a single directory I could not find duplicates when they existed outside of the directory of the original file. Unfortunately this appears to be the majority case for duplicate files.
While the TTH sort made it very easy to spot exact duplicates small changes made to a file would make it impossible to use TTH. By small changes I mean changes which do not affect the function of a file but change file size or signature values. Examples would be ID3, IPTC or information files added to compressed archives.
The requests here are:
A: An automated duplicate file finder to sort and compare files by multiple attributes such as filename, file size and (especially) TTH. It should handle duplicates discovery automatically and display them for the user to make a final decision on disposition.
B: Like in example 1, request B; A filter function similar to ADL Search for the Download Queue. However it should display the filtered files as if they were in a single directory so that sorting can be done by hand to find similar but not duplicate files (ie: where ID3 or metadata is changed but the data load is the same).
3. Looking for Duplicates from Outside the Download Queue tab
Then it dawned on me that I would miss a whole class of duplicate files if I focused only on those found in the Download Queue. The class I thought of were those files which I currently had or had already discarded. Two sources for files to be removed from the Download Queue sprang immediately to mind, those files that I had downloaded and those files which were on my system.
Since I knew the search and filter functions available weren't going to meet my needs I decided to do a quick script to match TTH values from the two sources I noted against queue.xml (after exiting DC++). Two things became apparent after a little searching. First, there are no TTH values in the Download.log file. Second, there are no easily discovered ways to gather TTH values for local files. I did find a way around the second problem, kinda.
I looked around for integrity checkers that I could use to create file hashes and none seemed to come up with a good test hash. Looking into it I realized that I didn't need a simple Tiger hash, but one that did a THEX (apparently a Tiger/1024 hash would be just as well). I asked Gojomo and found out that Bitzi would return the needed hash value but for this use should have submission to the Bitzi DB supressed (bitcollider -p). Yay! This is still a problem as bitcollider is suboptimal for scanning large numbers of files like integrity checkers do (fsum, md5sum, tripwire, etc.).
Also, to deal with query to and from XML I found a tool called XML Startlet that looks ideal, so far.
As I prepare to put some time into this I thought I would write up these requests. I think most of the function I am requesting already exists in DC++ and could be adapted to the particular tasks I mentioned.
The requests are:
A: First and foremost, please add a TTH argument to the download.log configuration format. This one little change would make a world of difference when clearing detritus from the Download Queue, either by additional functions in future DC++ versions or with external modification of the queue.xml. Another option would be to have an alternative download log file that could be written as an XML file for easier parsing.
B: Allow creation of TTH hashes of any files on a computer, not just those shared. It would seem easiest to add a File Management tab with the same range of functions as the Sharing area in configuration. If you implement it this way please let the hashes be written to an XML file accessible by external applications. (But an external implementation of a command line THEX/TTH tool would be a powerful tool for many uses, even outside of DC++.)
C: Add the ability to query TTH values from the Download Queue against TTH values in the Download.log file and those files stored locally on the computer/listed in the File Management area (or from a user maintained XML file of file names, sizes and TTH values).
Would asking for the inclusion of a complete file manager/explorer type interface be too much? It would be nice.
And for Christmas I would like ...If you have any questions about any of these requests or want to chat about design philosophy, or whatever, please feel free to post, email or IM me. When I finish doing whatever it is I end up doing I will try to get it into a final version for others to use and make it available.
Dan
.s
/s
.damnit!