Offline Searches

Archived discussion about features (predating the use of Bugzilla as a bug and feature tracker)

Moderator: Moderators

Locked
mai9
Posts: 111
Joined: 2003-04-16 23:02

Offline Searches

Post by mai9 » 2003-04-21 23:21

The a feature I am missing most is Offline Searches. This will search on downloaded filelists instead of using the hub network.

Why I am looking for this:

- Can search on users that are offline (the most important reason).

- No need to be connected (obvious).

- Can have all results (normal searches are quite buggy)

This feature and the possibility to download users' filelists every week would make my dc++ experience much pleasant. :D

The hub I participate most has 300 active users (people connected since last week), but only 150 are connected at the same time. So I am searching across half of what the hub offers, and there's no way to search on the other half unless I leave the search opened for a week. :evil:

Let's see if some developer likes this two ideas. :roll:

GargoyleMT
DC++ Contributor
Posts: 3212
Joined: 2003-01-07 21:46
Location: .pa.us

Re: Offline Searches

Post by GargoyleMT » 2003-04-22 08:06

mai9 wrote:Let's see if some developer likes this two ideas. :roll:
:)

Like... yes. Motivated enough to code... no. (I only speak for myself.) This has been brought up before, and I've always liked the idea of searching file lists.

mai9
Posts: 111
Joined: 2003-04-16 23:02

Post by mai9 » 2003-04-23 06:40

I see you prefer forums! :D

So, this idea has come before....

And what kind of motivation do you need? You even like the idea of searching filelists!

GargoyleMT
DC++ Contributor
Posts: 3212
Joined: 2003-01-07 21:46
Location: .pa.us

Post by GargoyleMT » 2003-04-23 08:01

What type of motivation do I need? An elegant solution in my head. :-D

The problem is that BZ2 decompression, or at least opening file lists though the main menu is slow on my system - an XP 1900+. Maybe it's just creating the tree or the data structure in memory, but that's colored my view of how fast offline searches would be. I think it might be better to pre-separate the files or tokens into a format that's easier to search (instead of just opening up every file list sequentially), but although that seems like The Right Way, it's also complicated.

I'd much prefer to tackle a problem that I can actually solve, like making an Upload Queue (which is also more of a pressing need for me, since I upload 3x more than I download, and I don't get much rotation on my 10-16kb upstream; I'd prefer if people didn't have to go days without getting at least some of their files serviced).

Moch
Posts: 71
Joined: 2003-03-21 22:29

Post by Moch » 2003-04-23 13:25

GargoyleMT,

I was just wondering.... how are you going to enforce an upload queue without modifying everyone elses client? *L*

I can't really think of an elegant way.. but if you had, please let me know, I'm intreasted. The only thing I can think of, but this isn't elegant and this isn't really a queue, is to make it so after a person is done with one file they can't download anymore from you for some kind of time delay... an 30 minutes to an hour perhaps.... or maybe it would be a sliding time frame depending on how long (or much) they have downloaded from you.

Err... ok... carry on with the thread, sorry to interupt ;)

~Moch

GargoyleMT
DC++ Contributor
Posts: 3212
Joined: 2003-01-07 21:46
Location: .pa.us

Post by GargoyleMT » 2003-04-23 13:40

Moch wrote:I can't really think of an elegant way.. but if you had, please let me know, I'm intreasted. The only thing I can think of, but this isn't elegant and this isn't really a queue, is to make it so after a person is done with one file they can't download anymore from you for some kind of time delay... an 30 minutes to an hour perhaps.... or maybe it would be a sliding time frame depending on how long (or much) they have downloaded from you.
Well, unless I'm missing something (could easily be...), all you need is a data structure (_STL::deque) and some modifications in UploadManager::prepareFile. (BCDC has a version already... it's just not as full featured as the one in my head.)

Add everyone to the back of a queue, and (unless they're asking for a file list or small file) only service people off the front of that list. Once a slot gets opened up, I think sending a CTM through the hub should suffice as a notification scheme. I haven't tried this, and BC does it a different way in his client - he correlates nicks to IPs and initiates the connection directly.

But your point is that a queue isn't efficient unless it actually rotates uploaders, I believe. I'm just going to set a hard limit of so many megabytes (configurable, with a reasonable lower limit in the SettingsManager) and keep track when a user goes over and give them the "No Free Slots" message and put them at the back of the queue.

I think a $Supports QueuePosition or QueueRating or something might be in order too, so a queue-aware client can give some feedback.

Once I get some working code, let me know if you want to try it. I'm not going to press ahead if it turns out to have little benefit to DC in general, but I'd really like rotation among uploaders for my own personal use. And users migrating from some other P2P applications think this is a missing feature in DC++.

Moch
Posts: 71
Joined: 2003-03-21 22:29

Post by Moch » 2003-04-23 14:02

Ok, now perhaps I am missing something....

Sending a CTM is nice and dandy, but if a client has been waiting several minutes for a slot and it how is it going to remmeber that CTM is for a file it requested? I am just starting on the downloads part of my client... but I didn't think a client was suppose to accept any file from any client that just connected to it, it needs to be... err expecting it. I see how this method may work on Passive Clients, but Active Clients won't be expecting a CTM right?

Also there is an issue with, what if the user isn't in the Hub? Are they going to be kept on the Queue and just skipped until they come back in? What if a user has gotten the requested file they are in Queue for by someone else in that time? What if the Client has canceled the download in their Queue and then has requested a diffrent file? Will it reset them in the Queue? and if it does.... How will you know if they still want the file they previously requested first or not?

Also... using CTM would only work for active Clients (the one sending the CTM). For an elegant solution don't we need something that works for Passive as well?

Perhaps, I am missing something, but it still seems to me, you will need to expect both clients to support the Queue ability (thus your implication by mentioning the $Support) I agree in an ideal world that would be elegant..... What about in the mean time? If both clients don't support the queue I really dont see how the way you mentioned will work, for the reasons I stated (if they are invalid, please, shoot them down!).

Perhaps at the very least the round robin idea would be a good option to apply to the clients that don't support the queue feature.. hmmm

Looking forward to your response,
~Moch

GargoyleMT
DC++ Contributor
Posts: 3212
Joined: 2003-01-07 21:46
Location: .pa.us

Post by GargoyleMT » 2003-04-23 14:22

Moch wrote:Sending a CTM is nice and dandy, but if a client has been waiting several minutes for a slot and it how is it going to remmeber that CTM is for a file it requested? I am just starting on the downloads part of my client... but I didn't think a client was suppose to accept any file from any client that just connected to it, it needs to be... err expecting it. I see how this method may work on Passive Clients, but Active Clients won't be expecting a CTM right?
Well, here's where my understanding breaks down. DC++ should handle it correctly, I believe. It depends on how stateful the client is... or that's my hypothesis until I have some code working enough to test it against all the clients I can get my hands on. (I have windows, Linux, and OSX, so I should be covered.) But, honestly, if NMDC is as stateless as some of the detection schemes (it will respond to a $Lock anytime with $Key) indicate it to be, it should be fine. And if DC++ works, any other clients that work are just gravy. :)
Moch wrote:Also there is an issue with, what if the user isn't in the Hub? Are they going to be kept on the Queue and just skipped until they come back in? What if a user has gotten the requested file they are in Queue for by someone else in that time? What if the Client has canceled the download in their Queue and then has requested a diffrent file? Will it reset them in the Queue? and if it does.... How will you know if they still want the file they previously requested first or not?
These are all excellent concerns. My implementation is strictly user nick based, and doesn't keep track of files or lock users into downloading only specific files. They will need to check in every so often to maintain their position. If they hammer by asking xx times in yy minutes, they'll be flagged as banned, and will have a temporary ban for ww minutes. This is all subject to change. But if they don't check in (have left the hub) they'll get a little bit of grace period, but will basically lose out, just as they would in the current system.
Moch wrote:Also... using CTM would only work for active Clients (the one sending the CTM). For an elegant solution don't we need something that works for Passive as well?
Yeah, BC handles passive correctly, I'll probably do that in the same way... I just don't like keeping nick/IP correlations around, if there's a way to do it otherwise.
Moch wrote:Perhaps, I am missing something, but it still seems to me, you will need to expect both clients to support the Queue ability (thus your implication by mentioning the $Support) I agree in an ideal world that would be elegant..... What about in the mean time? If both clients don't support the queue I really dont see how the way you mentioned will work, for the reasons I stated (if they are invalid, please, shoot them down!).
Well, take a look at BCDC, it works pretty much as the above, and BlackClaw says it does work. Check out some of sarf's posts on the issue too, he usually does a good job of convincing me that something is indeed possible. :)

My $Supports would just give a piece of information that could be displayed instead of "No Free Slots" - to give the remote user an idea of what rank they are in line.

Note, this is the way I think it should work, there are other ideas, including a "Slot Memory" thread from not too far back, which advocated the queue being persistent across sessions and locking users into downloading only files they had requested - and round-robining those.

mai9
Posts: 111
Joined: 2003-04-16 23:02

Post by mai9 » 2003-05-03 09:59

GargoyleMT wrote:I'd much prefer to tackle [...] an Upload Queue
Reading this thread I see it has more supporters :roll:

Locked